Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 5.775150707s FAIL (2025-07-19 05:27:31 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 4855s) [upload=4769]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 04:51:09.109709627 (6h0m15.482288982s) [upload=4769]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 04:51:09.556145753 (6h0m15.928725108s) [seg_pos={0 0}, upload=4769]: Begin upload segment... Jul 19 04:51:09.556194817 (6h0m15.928774172s) [seg_pos={0 0}, upload=4769]: Done begin upload segment. Jul 19 04:51:09.556223893 (6h0m15.928803248s) [noise=true , address=79.127.219.38:4023 , piece_id=PTPOZGLYV4MZTOFZNN3NCGYL7DFAST4PXKEUIVTMFL7HV4GZTCLQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556252197 (6h0m15.928831552s) [noise=true , address=45.91.95.4:4011 , piece_id=ZMBAS6KKHYVIRE6OBL2NFVCWMM4TKQ7J64BD3SYXIK54XCEDI2XA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556579383 (6h0m15.929158738s) [noise=true , address=45.91.116.6:4007 , piece_id=A2AFVTZBXSGBOE7XIM3B5HRN4STITFBXO6MZRJ365URNAAOZ3PSQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556854469 (6h0m15.929433824s) [noise=true , address=104.36.82.19:4423 , piece_id=PIJUHDUND36HQL67I33D7A3EULMJHS7N55MN4OFQJ4OWNQ42O47A, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557094107 (6h0m15.929673462s) [noise=true , address=45.91.116.9:4264 , piece_id=GHM5BKZKMT2QOHHBOS4FSPOWEWSFQRGRVVR4GSNDPHVIFRZHQL2A, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557336381 (6h0m15.929915736s) [noise=true , address=79.127.205.235:4660 , piece_id=UOUOGE2VP7PX7EMRZLPMCWFCYEZERM3DQMCQ6NG4J2K2DEIZR63Q, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557557885 (6h0m15.93013724s) [noise=true , address=79.127.205.246:4280 , piece_id=FLBOZI3UGTQXAQBLZ2F45Z3OEYL65F5PQBXRF2DQMZ7NGF7VDRPA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557762296 (6h0m15.930341651s) [noise=true , address=79.127.205.233:4622 , piece_id=6WWNYQTH5MGAJOEB7BXKZDJOMCAZXDW66J3OEANGDNJDNUMLZM7Q, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558014689 (6h0m15.930594044s) [noise=true , address=64.49.13.138:4482 , piece_id=MAW2M4US2OQNOS3WI67AA7NIBTJDE5T2JQTXKFPBLGVGXKVCFGTQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558215974 (6h0m15.930795319s) [noise=true , address=77.90.31.10:4546 , piece_id=R6LCA5WZOEBDURL5S3MEK22YPVYXX6EXSS4J4W57543HELYXOKZA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558409364 (6h0m15.930988719s) [noise=true , address=45.91.95.5:4019 , piece_id=YTSBCN2LPHEMRRAZIHJGVB7NBOTRXVEHD2TWUCTTLBNFFYEWWDKA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558623504 (6h0m15.931202859s) [noise=true , address=79.127.201.215:4357 , piece_id=75XWAG2VQWEYCWN2Q3WRIZSBI7LDOISZJPXQLQ2GIPNEN34VLSUQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558805162 (6h0m15.931384517s) [noise=true , address=45.91.116.14:4219 , piece_id=PDW6JPRHCVMHHOWIHT6JKV4CTSC3JY6NDKDVFZSO47OYNPZ7PGGA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558999373 (6h0m15.931578728s) [noise=true , address=77.90.31.11:4564 , piece_id=L2U7JFJRDQOFA2AYVAGTI4ZW4VABCXKMXWBIUWQO2QNNUUBJ74BA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559184768 (6h0m15.931764123s) [noise=true , address=77.90.31.16:4768 , piece_id=YNLSONFM6COZEVF6ZNQR2EA6V4LSQFZJONIXR4AFTNNMFGPV57JA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559368719 (6h0m15.931948074s) [noise=true , address=79.127.220.98:4432 , piece_id=2SUTFQ7FVDPAM2H4W3OLW6AVH3VJ33IJFE4L52SA3ET5LGBMOMPQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559549706 (6h0m15.932129061s) [noise=true , address=77.90.31.18:4813 , piece_id=VDOT7BB4O7XYJHW6BFTAYQZZYLZU5PAUXVJGSXJ7FDI4TH425COA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559727836 (6h0m15.932307191s) [noise=true , address=79.127.219.51:4211 , piece_id=HO5PLWBVU5YAYNYMN352NA73SM4WBJV4WZ7XLSQTCQ76UAL2TOJA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:10.663359087 (6h0m17.035938442s) [upload=4769]: Next returned nil segment Jul 19 04:51:10.663378664 (6h0m17.035958019s) [upload=4769]: Waiting for error group managing segments... Jul 19 04:51:10.962945479 (6h0m17.335524834s) [noise=true , address=79.127.205.233:4622 , piece_id=6WWNYQTH5MGAJOEB7BXKZDJOMCAZXDW66J3OEANGDNJDNUMLZM7Q, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:10.967689807 (6h0m17.340269152s) [noise=true , address=79.127.205.235:4660 , piece_id=UOUOGE2VP7PX7EMRZLPMCWFCYEZERM3DQMCQ6NG4J2K2DEIZR63Q, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:10.976968631 (6h0m17.349547986s) [noise=true , address=64.49.13.138:4482 , piece_id=MAW2M4US2OQNOS3WI67AA7NIBTJDE5T2JQTXKFPBLGVGXKVCFGTQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.048750700 (6h0m17.421330055s) [noise=true , address=79.127.205.246:4280 , piece_id=FLBOZI3UGTQXAQBLZ2F45Z3OEYL65F5PQBXRF2DQMZ7NGF7VDRPA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.221642403 (6h0m17.594221758s) [noise=true , address=79.127.220.98:4432 , piece_id=2SUTFQ7FVDPAM2H4W3OLW6AVH3VJ33IJFE4L52SA3ET5LGBMOMPQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.317070435 (6h0m17.68964979s) [noise=true , address=45.91.95.4:4011 , piece_id=ZMBAS6KKHYVIRE6OBL2NFVCWMM4TKQ7J64BD3SYXIK54XCEDI2XA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.324002660 (6h0m17.696582015s) [noise=true , address=45.91.95.5:4019 , piece_id=YTSBCN2LPHEMRRAZIHJGVB7NBOTRXVEHD2TWUCTTLBNFFYEWWDKA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.460976317 (6h0m17.833555672s) [noise=true , address=79.127.201.215:4357 , piece_id=75XWAG2VQWEYCWN2Q3WRIZSBI7LDOISZJPXQLQ2GIPNEN34VLSUQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.462584603 (6h0m17.835163958s) [noise=true , address=45.91.116.6:4007 , piece_id=A2AFVTZBXSGBOE7XIM3B5HRN4STITFBXO6MZRJ365URNAAOZ3PSQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.490392973 (6h0m17.862972328s) [noise=true , address=45.91.116.9:4264 , piece_id=GHM5BKZKMT2QOHHBOS4FSPOWEWSFQRGRVVR4GSNDPHVIFRZHQL2A, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.502040780 (6h0m17.874620135s) [noise=true , address=45.91.116.14:4219 , piece_id=PDW6JPRHCVMHHOWIHT6JKV4CTSC3JY6NDKDVFZSO47OYNPZ7PGGA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.600434096 (6h0m17.973013451s) [noise=true , address=79.127.219.51:4211 , piece_id=HO5PLWBVU5YAYNYMN352NA73SM4WBJV4WZ7XLSQTCQ76UAL2TOJA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.609008984 (6h0m17.981588339s) [noise=true , address=104.36.82.19:4423 , piece_id=PIJUHDUND36HQL67I33D7A3EULMJHS7N55MN4OFQJ4OWNQ42O47A, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.620825273 (6h0m17.993404628s) [noise=true , address=79.127.219.38:4023 , piece_id=PTPOZGLYV4MZTOFZNN3NCGYL7DFAST4PXKEUIVTMFL7HV4GZTCLQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.620838659 (6h0m17.993418014s) [seg_pos={0 0}, upload=4769]: Segment reached optimal threshold of 14 pieces. Jul 19 04:51:11.621067807 (6h0m17.993647163s) [noise=true , address=77.90.31.11:4564 , piece_id=L2U7JFJRDQOFA2AYVAGTI4ZW4VABCXKMXWBIUWQO2QNNUUBJ74BA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection; write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection Jul 19 04:51:11.621134234 (6h0m17.993713589s) [noise=true , address=77.90.31.10:4546 , piece_id=R6LCA5WZOEBDURL5S3MEK22YPVYXX6EXSS4J4W57543HELYXOKZA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK): protocol: read tcp 188.166.225.170:58476->77.90.31.10:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621174191 (6h0m17.993753546s) [noise=true , address=77.90.31.16:4768 , piece_id=YNLSONFM6COZEVF6ZNQR2EA6V4LSQFZJONIXR4AFTNNMFGPV57JA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UchewmtxBrtSZSpVAJFdZmcTsQVQxkjK9re6cgUGqDhyonQeF): protocol: read tcp 188.166.225.170:56326->77.90.31.16:4768: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621216842 (6h0m17.993796197s) [noise=true , address=77.90.31.18:4813 , piece_id=VDOT7BB4O7XYJHW6BFTAYQZZYLZU5PAUXVJGSXJ7FDI4TH425COA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128WthA1x9wp1dV1vkpGnoveZQpU9CSgDCZfw5GMQuVpCNSVAM5): protocol: read tcp 188.166.225.170:49704->77.90.31.18:4813: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621234917 (6h0m17.993814272s) [seg_pos={0 0}, upload=4769]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 19 04:51:11.621301444 (6h0m17.993880799s) [upload=4769]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 19 04:51:11.621305402 (6h0m17.993884758s) [upload=4769]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 19 04:51:11.923136970 (6h0m18.295716325s) [upload=4769]: Done uploading segments... Jul 19 04:51:11.923156627 (6h0m18.295735972s) [upload=4769]: Upload finished. err: <nil> Duration: 5.775188129s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [.....XX...X...X....X..X....XXXXX.X.X.....XX.X...............] duration: 21.548592298s FAIL (2025-07-19 05:27:17 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== gment. successful: 13 optimal: 13 errs: <nil> Jul 19 05:14:35.936262577 (6h23m42.308841932s) [upload=5075]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 19 05:14:35.936281884 (6h23m42.308861239s) [upload=5075]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 19 05:14:36.234171969 (6h23m42.606751314s) [upload=5075]: Done uploading segments... Jul 19 05:14:36.234192037 (6h23m42.606771382s) [upload=5075]: Upload finished. err: <nil> Jul 19 05:14:55.587986728 (6h24m1.960566093s) [upload=5081]: Starting upload Jul 19 05:14:55.588051823 (6h24m1.960631178s) [upload=5081]: Done starting upload Jul 19 05:14:55.588146724 (6h24m1.960726079s) [upload=5081]: Uploading segments... Jul 19 05:14:55.588391873 (6h24m1.960971228s) [upload=5081]: Got next segment. Inline: false Jul 19 05:14:55.588649837 (6h24m1.961229193s) [upload=5081]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 05:14:55.588655057 (6h24m1.961234402s) [upload=5081]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 05:14:55.823387692 (6h24m2.195967047s) [seg_pos={0 0}, upload=5081]: Begin upload segment... Jul 19 05:14:55.823450572 (6h24m2.196029927s) [seg_pos={0 0}, upload=5081]: Done begin upload segment. Jul 19 05:14:55.823481952 (6h24m2.196061307s) [noise=true , address=77.90.31.17:4772 , piece_id=B544E3F2UC3UVE2ITUQAWGVDMSPS2HFIVHXZ2OLWB3T25AHMFAVA, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.823491401 (6h24m2.196070756s) [noise=true , address=45.91.116.10:4082 , piece_id=HZDMC4J5O5UHVDH5SOQBKAFPN6RJUHIWKEZ4OPVMW2HSN7NDZMZA, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.823923789 (6h24m2.196503144s) [noise=true , address=45.91.116.9:4262 , piece_id=FRIATDM7LPPHXYAMUISAYRTQWAKSTB4EYUANHGEMQHM6JQADDTRQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.823949737 (6h24m2.196529092s) [noise=true , address=45.91.116.12:4227 , piece_id=QB7YKZFXWMRB3VQ3HSFMMMLLX5YFWOXG4Y4GJGGYT6HDV4FQ65VA, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.824310158 (6h24m2.196889513s) [noise=true , address=45.91.116.4:4937 , piece_id=V2IOOA7DVCGZBH3D43ROYHT6ZECPY6GTC5M7NFO2RWPQB6EYGALA, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.824403045 (6h24m2.19698239s) [noise=true , address=77.90.31.11:4555 , piece_id=YICA5IVHSCC3RPC5FRKD26BSMY6N5WJQA5LCABJSRJYL6BZJYL6A, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.824651210 (6h24m2.197230565s) [noise=true , address=77.90.31.13:4627 , piece_id=E7GSSBZ653K2AA4TP6A4BRT5BDZCBODNRAZM3P6SYJXJFQU2BACQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.824716334 (6h24m2.197295689s) [noise=true , address=77.90.31.16:4734 , piece_id=YUW6MWSUDNLAC2VHLBT6H4T27YTNJ3RDK4BSGE7OZQCO3F2LBH4A, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.824960491 (6h24m2.197539856s) [noise=true , address=45.91.116.7:4010 , piece_id=BH76RM4NXGEAI2J2424ALK4PFSXUTBCYOH777BUOIDW7BW2IJUJA, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.824977494 (6h24m2.197556849s) [noise=true , address=45.91.116.14:4190 , piece_id=RUU2MICSAWOECAAYNOBS75SE2YKSUBMSSFKLPON7FWK77UNJEYGQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.825213285 (6h24m2.19779264s) [noise=true , address=45.91.116.8:4047 , piece_id=6UOFOV4VCVT6DB7D7Q23REAX7VZPAW5PSG5PLA54LISOUWMPQBBQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.825271547 (6h24m2.197850902s) [noise=true , address=77.90.31.18:4806 , piece_id=67PXFJMJB6XWNU6JMOKPO4BXKDD2Z225ZKTB7DZEA46OMSSXHHMQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.825445860 (6h24m2.198025215s) [noise=true , address=45.91.116.7:4009 , piece_id=HH5I32ZYT5LVXHDLBVSAVX6T23MJ6RGNPNAWPIQAREJ4ELPFJGYA, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.825568244 (6h24m2.198147599s) [noise=true , address=77.90.31.15:4697 , piece_id=W62ARDAVLI64P6VYSPVFRTDKB5H2WXTSCF2VKP3UN7JKGBCUEZTQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.825902985 (6h24m2.19848234s) [noise=true , address=45.91.116.5:4299 , piece_id=JQLK5LIMCMW6R2UZEJZSG6QAP2M65XBVE6JFEXY5KDWBVNDG2VCQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.826216495 (6h24m2.19879585s) [noise=true , address=77.90.31.14:4663 , piece_id=LNTFUWB6LUUFD2EB536ZHBSNVVXYZPOOB4DUZB2M6BXSSFEL3UMQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:58.633690556 (6h24m5.006269911s) [upload=5081]: Next returned nil segment Jul 19 05:14:58.633708100 (6h24m5.006287445s) [upload=5081]: Waiting for error group managing segments... Jul 19 05:14:58.915966948 (6h24m5.288546293s) [noise=true , address=45.91.116.7:4009 , piece_id=HH5I32ZYT5LVXHDLBVSAVX6T23MJ6RGNPNAWPIQAREJ4ELPFJGYA, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:58.916093541 (6h24m5.288672896s) [noise=true , address=45.91.116.12:4227 , piece_id=QB7YKZFXWMRB3VQ3HSFMMMLLX5YFWOXG4Y4GJGGYT6HDV4FQ65VA, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:58.923615462 (6h24m5.296194817s) [noise=true , address=45.91.116.14:4190 , piece_id=RUU2MICSAWOECAAYNOBS75SE2YKSUBMSSFKLPON7FWK77UNJEYGQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:58.928612934 (6h24m5.301192289s) [noise=true , address=45.91.116.7:4010 , piece_id=BH76RM4NXGEAI2J2424ALK4PFSXUTBCYOH777BUOIDW7BW2IJUJA, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:58.990915468 (6h24m5.363494833s) [noise=true , address=45.91.116.4:4937 , piece_id=V2IOOA7DVCGZBH3D43ROYHT6ZECPY6GTC5M7NFO2RWPQB6EYGALA, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:58.992136464 (6h24m5.364715819s) [noise=true , address=45.91.116.8:4047 , piece_id=6UOFOV4VCVT6DB7D7Q23REAX7VZPAW5PSG5PLA54LISOUWMPQBBQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:58.997711812 (6h24m5.370291157s) [noise=true , address=45.91.116.9:4262 , piece_id=FRIATDM7LPPHXYAMUISAYRTQWAKSTB4EYUANHGEMQHM6JQADDTRQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:59.010548391 (6h24m5.383127746s) [noise=true , address=45.91.116.10:4082 , piece_id=HZDMC4J5O5UHVDH5SOQBKAFPN6RJUHIWKEZ4OPVMW2HSN7NDZMZA, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:59.039607621 (6h24m5.412186996s) [noise=true , address=45.91.116.5:4299 , piece_id=JQLK5LIMCMW6R2UZEJZSG6QAP2M65XBVE6JFEXY5KDWBVNDG2VCQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:15:19.998482215 (6h24m26.37106157s) [noise=true , address=77.90.31.14:4663 , piece_id=LNTFUWB6LUUFD2EB536ZHBSNVVXYZPOOB4DUZB2M6BXSSFEL3UMQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:15:21.170089451 (6h24m27.542668806s) [noise=true , address=77.90.31.13:4627 , piece_id=E7GSSBZ653K2AA4TP6A4BRT5BDZCBODNRAZM3P6SYJXJFQU2BACQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:15:24.941469148 (6h24m31.314048503s) [noise=true , address=77.90.31.15:4697 , piece_id=W62ARDAVLI64P6VYSPVFRTDKB5H2WXTSCF2VKP3UN7JKGBCUEZTQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: expected piece hash; context deadline exceeded; EOF Jul 19 05:15:24.941530766 (6h24m31.314110121s) [noise=true , address=77.90.31.17:4772 , piece_id=B544E3F2UC3UVE2ITUQAWGVDMSPS2HFIVHXZ2OLWB3T25AHMFAVA, seg_pos={0 0}, upload=5081]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:44456->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:44456->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44456->77.90.31.17:4772: use of closed network connection Jul 19 05:15:24.941543591 (6h24m31.314122946s) [noise=true , address=77.90.31.16:4734 , piece_id=YUW6MWSUDNLAC2VHLBT6H4T27YTNJ3RDK4BSGE7OZQCO3F2LBH4A, seg_pos={0 0}, upload=5081]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Jul 19 05:15:24.941614786 (6h24m31.314194141s) [noise=true , address=77.90.31.11:4555 , piece_id=YICA5IVHSCC3RPC5FRKD26BSMY6N5WJQA5LCABJSRJYL6BZJYL6A, seg_pos={0 0}, upload=5081]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: write tcp 188.166.225.170:37392->77.90.31.11:4555: use of closed network connection; write tcp 188.166.225.170:37392->77.90.31.11:4555: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37392->77.90.31.11:4555: use of closed network connection Jul 19 05:15:24.941641888 (6h24m31.314221243s) [noise=true , address=77.90.31.18:4806 , piece_id=67PXFJMJB6XWNU6JMOKPO4BXKDD2Z225ZKTB7DZEA46OMSSXHHMQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:47114->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:47114->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47114->77.90.31.18:4806: use of closed network connection Jul 19 05:15:24.941663099 (6h24m31.314242454s) [seg_pos={0 0}, upload=5081]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 19 05:15:25.179949476 (6h24m31.552528831s) [upload=5081]: Done uploading segments... Jul 19 05:15:25.179975797 (6h24m31.552555152s) [upload=5081]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 21.548618537s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [......XX...X...X....X..X....XXXXX.X.X.....XX.X..............] duration: 19.163509266s FAIL (2025-07-19 05:26:31 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== gment. successful: 13 optimal: 13 errs: <nil> Jul 19 05:14:35.936262577 (6h23m42.308841932s) [upload=5075]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 19 05:14:35.936281884 (6h23m42.308861239s) [upload=5075]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 19 05:14:36.234171969 (6h23m42.606751314s) [upload=5075]: Done uploading segments... Jul 19 05:14:36.234192037 (6h23m42.606771382s) [upload=5075]: Upload finished. err: <nil> Jul 19 05:14:55.587986728 (6h24m1.960566093s) [upload=5081]: Starting upload Jul 19 05:14:55.588051823 (6h24m1.960631178s) [upload=5081]: Done starting upload Jul 19 05:14:55.588146724 (6h24m1.960726079s) [upload=5081]: Uploading segments... Jul 19 05:14:55.588391873 (6h24m1.960971228s) [upload=5081]: Got next segment. Inline: false Jul 19 05:14:55.588649837 (6h24m1.961229193s) [upload=5081]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 05:14:55.588655057 (6h24m1.961234402s) [upload=5081]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 05:14:55.823387692 (6h24m2.195967047s) [seg_pos={0 0}, upload=5081]: Begin upload segment... Jul 19 05:14:55.823450572 (6h24m2.196029927s) [seg_pos={0 0}, upload=5081]: Done begin upload segment. Jul 19 05:14:55.823481952 (6h24m2.196061307s) [noise=true , address=77.90.31.17:4772 , piece_id=B544E3F2UC3UVE2ITUQAWGVDMSPS2HFIVHXZ2OLWB3T25AHMFAVA, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.823491401 (6h24m2.196070756s) [noise=true , address=45.91.116.10:4082 , piece_id=HZDMC4J5O5UHVDH5SOQBKAFPN6RJUHIWKEZ4OPVMW2HSN7NDZMZA, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.823923789 (6h24m2.196503144s) [noise=true , address=45.91.116.9:4262 , piece_id=FRIATDM7LPPHXYAMUISAYRTQWAKSTB4EYUANHGEMQHM6JQADDTRQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.823949737 (6h24m2.196529092s) [noise=true , address=45.91.116.12:4227 , piece_id=QB7YKZFXWMRB3VQ3HSFMMMLLX5YFWOXG4Y4GJGGYT6HDV4FQ65VA, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.824310158 (6h24m2.196889513s) [noise=true , address=45.91.116.4:4937 , piece_id=V2IOOA7DVCGZBH3D43ROYHT6ZECPY6GTC5M7NFO2RWPQB6EYGALA, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.824403045 (6h24m2.19698239s) [noise=true , address=77.90.31.11:4555 , piece_id=YICA5IVHSCC3RPC5FRKD26BSMY6N5WJQA5LCABJSRJYL6BZJYL6A, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.824651210 (6h24m2.197230565s) [noise=true , address=77.90.31.13:4627 , piece_id=E7GSSBZ653K2AA4TP6A4BRT5BDZCBODNRAZM3P6SYJXJFQU2BACQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.824716334 (6h24m2.197295689s) [noise=true , address=77.90.31.16:4734 , piece_id=YUW6MWSUDNLAC2VHLBT6H4T27YTNJ3RDK4BSGE7OZQCO3F2LBH4A, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.824960491 (6h24m2.197539856s) [noise=true , address=45.91.116.7:4010 , piece_id=BH76RM4NXGEAI2J2424ALK4PFSXUTBCYOH777BUOIDW7BW2IJUJA, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.824977494 (6h24m2.197556849s) [noise=true , address=45.91.116.14:4190 , piece_id=RUU2MICSAWOECAAYNOBS75SE2YKSUBMSSFKLPON7FWK77UNJEYGQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.825213285 (6h24m2.19779264s) [noise=true , address=45.91.116.8:4047 , piece_id=6UOFOV4VCVT6DB7D7Q23REAX7VZPAW5PSG5PLA54LISOUWMPQBBQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.825271547 (6h24m2.197850902s) [noise=true , address=77.90.31.18:4806 , piece_id=67PXFJMJB6XWNU6JMOKPO4BXKDD2Z225ZKTB7DZEA46OMSSXHHMQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.825445860 (6h24m2.198025215s) [noise=true , address=45.91.116.7:4009 , piece_id=HH5I32ZYT5LVXHDLBVSAVX6T23MJ6RGNPNAWPIQAREJ4ELPFJGYA, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.825568244 (6h24m2.198147599s) [noise=true , address=77.90.31.15:4697 , piece_id=W62ARDAVLI64P6VYSPVFRTDKB5H2WXTSCF2VKP3UN7JKGBCUEZTQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.825902985 (6h24m2.19848234s) [noise=true , address=45.91.116.5:4299 , piece_id=JQLK5LIMCMW6R2UZEJZSG6QAP2M65XBVE6JFEXY5KDWBVNDG2VCQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.826216495 (6h24m2.19879585s) [noise=true , address=77.90.31.14:4663 , piece_id=LNTFUWB6LUUFD2EB536ZHBSNVVXYZPOOB4DUZB2M6BXSSFEL3UMQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:58.633690556 (6h24m5.006269911s) [upload=5081]: Next returned nil segment Jul 19 05:14:58.633708100 (6h24m5.006287445s) [upload=5081]: Waiting for error group managing segments... Jul 19 05:14:58.915966948 (6h24m5.288546293s) [noise=true , address=45.91.116.7:4009 , piece_id=HH5I32ZYT5LVXHDLBVSAVX6T23MJ6RGNPNAWPIQAREJ4ELPFJGYA, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:58.916093541 (6h24m5.288672896s) [noise=true , address=45.91.116.12:4227 , piece_id=QB7YKZFXWMRB3VQ3HSFMMMLLX5YFWOXG4Y4GJGGYT6HDV4FQ65VA, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:58.923615462 (6h24m5.296194817s) [noise=true , address=45.91.116.14:4190 , piece_id=RUU2MICSAWOECAAYNOBS75SE2YKSUBMSSFKLPON7FWK77UNJEYGQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:58.928612934 (6h24m5.301192289s) [noise=true , address=45.91.116.7:4010 , piece_id=BH76RM4NXGEAI2J2424ALK4PFSXUTBCYOH777BUOIDW7BW2IJUJA, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:58.990915468 (6h24m5.363494833s) [noise=true , address=45.91.116.4:4937 , piece_id=V2IOOA7DVCGZBH3D43ROYHT6ZECPY6GTC5M7NFO2RWPQB6EYGALA, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:58.992136464 (6h24m5.364715819s) [noise=true , address=45.91.116.8:4047 , piece_id=6UOFOV4VCVT6DB7D7Q23REAX7VZPAW5PSG5PLA54LISOUWMPQBBQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:58.997711812 (6h24m5.370291157s) [noise=true , address=45.91.116.9:4262 , piece_id=FRIATDM7LPPHXYAMUISAYRTQWAKSTB4EYUANHGEMQHM6JQADDTRQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:59.010548391 (6h24m5.383127746s) [noise=true , address=45.91.116.10:4082 , piece_id=HZDMC4J5O5UHVDH5SOQBKAFPN6RJUHIWKEZ4OPVMW2HSN7NDZMZA, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:59.039607621 (6h24m5.412186996s) [noise=true , address=45.91.116.5:4299 , piece_id=JQLK5LIMCMW6R2UZEJZSG6QAP2M65XBVE6JFEXY5KDWBVNDG2VCQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:15:19.998482215 (6h24m26.37106157s) [noise=true , address=77.90.31.14:4663 , piece_id=LNTFUWB6LUUFD2EB536ZHBSNVVXYZPOOB4DUZB2M6BXSSFEL3UMQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:15:21.170089451 (6h24m27.542668806s) [noise=true , address=77.90.31.13:4627 , piece_id=E7GSSBZ653K2AA4TP6A4BRT5BDZCBODNRAZM3P6SYJXJFQU2BACQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:15:24.941469148 (6h24m31.314048503s) [noise=true , address=77.90.31.15:4697 , piece_id=W62ARDAVLI64P6VYSPVFRTDKB5H2WXTSCF2VKP3UN7JKGBCUEZTQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: expected piece hash; context deadline exceeded; EOF Jul 19 05:15:24.941530766 (6h24m31.314110121s) [noise=true , address=77.90.31.17:4772 , piece_id=B544E3F2UC3UVE2ITUQAWGVDMSPS2HFIVHXZ2OLWB3T25AHMFAVA, seg_pos={0 0}, upload=5081]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:44456->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:44456->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44456->77.90.31.17:4772: use of closed network connection Jul 19 05:15:24.941543591 (6h24m31.314122946s) [noise=true , address=77.90.31.16:4734 , piece_id=YUW6MWSUDNLAC2VHLBT6H4T27YTNJ3RDK4BSGE7OZQCO3F2LBH4A, seg_pos={0 0}, upload=5081]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Jul 19 05:15:24.941614786 (6h24m31.314194141s) [noise=true , address=77.90.31.11:4555 , piece_id=YICA5IVHSCC3RPC5FRKD26BSMY6N5WJQA5LCABJSRJYL6BZJYL6A, seg_pos={0 0}, upload=5081]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: write tcp 188.166.225.170:37392->77.90.31.11:4555: use of closed network connection; write tcp 188.166.225.170:37392->77.90.31.11:4555: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37392->77.90.31.11:4555: use of closed network connection Jul 19 05:15:24.941641888 (6h24m31.314221243s) [noise=true , address=77.90.31.18:4806 , piece_id=67PXFJMJB6XWNU6JMOKPO4BXKDD2Z225ZKTB7DZEA46OMSSXHHMQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:47114->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:47114->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47114->77.90.31.18:4806: use of closed network connection Jul 19 05:15:24.941663099 (6h24m31.314242454s) [seg_pos={0 0}, upload=5081]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 19 05:15:25.179949476 (6h24m31.552528831s) [upload=5081]: Done uploading segments... Jul 19 05:15:25.179975797 (6h24m31.552555152s) [upload=5081]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 19.163540126s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 8.879516978s FAIL (2025-07-19 05:26:28 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 4855s) [upload=4769]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 04:51:09.109709627 (6h0m15.482288982s) [upload=4769]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 04:51:09.556145753 (6h0m15.928725108s) [seg_pos={0 0}, upload=4769]: Begin upload segment... Jul 19 04:51:09.556194817 (6h0m15.928774172s) [seg_pos={0 0}, upload=4769]: Done begin upload segment. Jul 19 04:51:09.556223893 (6h0m15.928803248s) [noise=true , address=79.127.219.38:4023 , piece_id=PTPOZGLYV4MZTOFZNN3NCGYL7DFAST4PXKEUIVTMFL7HV4GZTCLQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556252197 (6h0m15.928831552s) [noise=true , address=45.91.95.4:4011 , piece_id=ZMBAS6KKHYVIRE6OBL2NFVCWMM4TKQ7J64BD3SYXIK54XCEDI2XA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556579383 (6h0m15.929158738s) [noise=true , address=45.91.116.6:4007 , piece_id=A2AFVTZBXSGBOE7XIM3B5HRN4STITFBXO6MZRJ365URNAAOZ3PSQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556854469 (6h0m15.929433824s) [noise=true , address=104.36.82.19:4423 , piece_id=PIJUHDUND36HQL67I33D7A3EULMJHS7N55MN4OFQJ4OWNQ42O47A, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557094107 (6h0m15.929673462s) [noise=true , address=45.91.116.9:4264 , piece_id=GHM5BKZKMT2QOHHBOS4FSPOWEWSFQRGRVVR4GSNDPHVIFRZHQL2A, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557336381 (6h0m15.929915736s) [noise=true , address=79.127.205.235:4660 , piece_id=UOUOGE2VP7PX7EMRZLPMCWFCYEZERM3DQMCQ6NG4J2K2DEIZR63Q, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557557885 (6h0m15.93013724s) [noise=true , address=79.127.205.246:4280 , piece_id=FLBOZI3UGTQXAQBLZ2F45Z3OEYL65F5PQBXRF2DQMZ7NGF7VDRPA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557762296 (6h0m15.930341651s) [noise=true , address=79.127.205.233:4622 , piece_id=6WWNYQTH5MGAJOEB7BXKZDJOMCAZXDW66J3OEANGDNJDNUMLZM7Q, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558014689 (6h0m15.930594044s) [noise=true , address=64.49.13.138:4482 , piece_id=MAW2M4US2OQNOS3WI67AA7NIBTJDE5T2JQTXKFPBLGVGXKVCFGTQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558215974 (6h0m15.930795319s) [noise=true , address=77.90.31.10:4546 , piece_id=R6LCA5WZOEBDURL5S3MEK22YPVYXX6EXSS4J4W57543HELYXOKZA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558409364 (6h0m15.930988719s) [noise=true , address=45.91.95.5:4019 , piece_id=YTSBCN2LPHEMRRAZIHJGVB7NBOTRXVEHD2TWUCTTLBNFFYEWWDKA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558623504 (6h0m15.931202859s) [noise=true , address=79.127.201.215:4357 , piece_id=75XWAG2VQWEYCWN2Q3WRIZSBI7LDOISZJPXQLQ2GIPNEN34VLSUQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558805162 (6h0m15.931384517s) [noise=true , address=45.91.116.14:4219 , piece_id=PDW6JPRHCVMHHOWIHT6JKV4CTSC3JY6NDKDVFZSO47OYNPZ7PGGA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558999373 (6h0m15.931578728s) [noise=true , address=77.90.31.11:4564 , piece_id=L2U7JFJRDQOFA2AYVAGTI4ZW4VABCXKMXWBIUWQO2QNNUUBJ74BA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559184768 (6h0m15.931764123s) [noise=true , address=77.90.31.16:4768 , piece_id=YNLSONFM6COZEVF6ZNQR2EA6V4LSQFZJONIXR4AFTNNMFGPV57JA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559368719 (6h0m15.931948074s) [noise=true , address=79.127.220.98:4432 , piece_id=2SUTFQ7FVDPAM2H4W3OLW6AVH3VJ33IJFE4L52SA3ET5LGBMOMPQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559549706 (6h0m15.932129061s) [noise=true , address=77.90.31.18:4813 , piece_id=VDOT7BB4O7XYJHW6BFTAYQZZYLZU5PAUXVJGSXJ7FDI4TH425COA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559727836 (6h0m15.932307191s) [noise=true , address=79.127.219.51:4211 , piece_id=HO5PLWBVU5YAYNYMN352NA73SM4WBJV4WZ7XLSQTCQ76UAL2TOJA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:10.663359087 (6h0m17.035938442s) [upload=4769]: Next returned nil segment Jul 19 04:51:10.663378664 (6h0m17.035958019s) [upload=4769]: Waiting for error group managing segments... Jul 19 04:51:10.962945479 (6h0m17.335524834s) [noise=true , address=79.127.205.233:4622 , piece_id=6WWNYQTH5MGAJOEB7BXKZDJOMCAZXDW66J3OEANGDNJDNUMLZM7Q, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:10.967689807 (6h0m17.340269152s) [noise=true , address=79.127.205.235:4660 , piece_id=UOUOGE2VP7PX7EMRZLPMCWFCYEZERM3DQMCQ6NG4J2K2DEIZR63Q, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:10.976968631 (6h0m17.349547986s) [noise=true , address=64.49.13.138:4482 , piece_id=MAW2M4US2OQNOS3WI67AA7NIBTJDE5T2JQTXKFPBLGVGXKVCFGTQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.048750700 (6h0m17.421330055s) [noise=true , address=79.127.205.246:4280 , piece_id=FLBOZI3UGTQXAQBLZ2F45Z3OEYL65F5PQBXRF2DQMZ7NGF7VDRPA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.221642403 (6h0m17.594221758s) [noise=true , address=79.127.220.98:4432 , piece_id=2SUTFQ7FVDPAM2H4W3OLW6AVH3VJ33IJFE4L52SA3ET5LGBMOMPQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.317070435 (6h0m17.68964979s) [noise=true , address=45.91.95.4:4011 , piece_id=ZMBAS6KKHYVIRE6OBL2NFVCWMM4TKQ7J64BD3SYXIK54XCEDI2XA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.324002660 (6h0m17.696582015s) [noise=true , address=45.91.95.5:4019 , piece_id=YTSBCN2LPHEMRRAZIHJGVB7NBOTRXVEHD2TWUCTTLBNFFYEWWDKA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.460976317 (6h0m17.833555672s) [noise=true , address=79.127.201.215:4357 , piece_id=75XWAG2VQWEYCWN2Q3WRIZSBI7LDOISZJPXQLQ2GIPNEN34VLSUQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.462584603 (6h0m17.835163958s) [noise=true , address=45.91.116.6:4007 , piece_id=A2AFVTZBXSGBOE7XIM3B5HRN4STITFBXO6MZRJ365URNAAOZ3PSQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.490392973 (6h0m17.862972328s) [noise=true , address=45.91.116.9:4264 , piece_id=GHM5BKZKMT2QOHHBOS4FSPOWEWSFQRGRVVR4GSNDPHVIFRZHQL2A, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.502040780 (6h0m17.874620135s) [noise=true , address=45.91.116.14:4219 , piece_id=PDW6JPRHCVMHHOWIHT6JKV4CTSC3JY6NDKDVFZSO47OYNPZ7PGGA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.600434096 (6h0m17.973013451s) [noise=true , address=79.127.219.51:4211 , piece_id=HO5PLWBVU5YAYNYMN352NA73SM4WBJV4WZ7XLSQTCQ76UAL2TOJA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.609008984 (6h0m17.981588339s) [noise=true , address=104.36.82.19:4423 , piece_id=PIJUHDUND36HQL67I33D7A3EULMJHS7N55MN4OFQJ4OWNQ42O47A, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.620825273 (6h0m17.993404628s) [noise=true , address=79.127.219.38:4023 , piece_id=PTPOZGLYV4MZTOFZNN3NCGYL7DFAST4PXKEUIVTMFL7HV4GZTCLQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.620838659 (6h0m17.993418014s) [seg_pos={0 0}, upload=4769]: Segment reached optimal threshold of 14 pieces. Jul 19 04:51:11.621067807 (6h0m17.993647163s) [noise=true , address=77.90.31.11:4564 , piece_id=L2U7JFJRDQOFA2AYVAGTI4ZW4VABCXKMXWBIUWQO2QNNUUBJ74BA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection; write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection Jul 19 04:51:11.621134234 (6h0m17.993713589s) [noise=true , address=77.90.31.10:4546 , piece_id=R6LCA5WZOEBDURL5S3MEK22YPVYXX6EXSS4J4W57543HELYXOKZA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK): protocol: read tcp 188.166.225.170:58476->77.90.31.10:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621174191 (6h0m17.993753546s) [noise=true , address=77.90.31.16:4768 , piece_id=YNLSONFM6COZEVF6ZNQR2EA6V4LSQFZJONIXR4AFTNNMFGPV57JA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UchewmtxBrtSZSpVAJFdZmcTsQVQxkjK9re6cgUGqDhyonQeF): protocol: read tcp 188.166.225.170:56326->77.90.31.16:4768: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621216842 (6h0m17.993796197s) [noise=true , address=77.90.31.18:4813 , piece_id=VDOT7BB4O7XYJHW6BFTAYQZZYLZU5PAUXVJGSXJ7FDI4TH425COA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128WthA1x9wp1dV1vkpGnoveZQpU9CSgDCZfw5GMQuVpCNSVAM5): protocol: read tcp 188.166.225.170:49704->77.90.31.18:4813: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621234917 (6h0m17.993814272s) [seg_pos={0 0}, upload=4769]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 19 04:51:11.621301444 (6h0m17.993880799s) [upload=4769]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 19 04:51:11.621305402 (6h0m17.993884758s) [upload=4769]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 19 04:51:11.923136970 (6h0m18.295716325s) [upload=4769]: Done uploading segments... Jul 19 04:51:11.923156627 (6h0m18.295735972s) [upload=4769]: Upload finished. err: <nil> Duration: 8.879546384s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 22.819841469s FAIL (2025-07-19 05:25:28 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 4855s) [upload=4769]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 04:51:09.109709627 (6h0m15.482288982s) [upload=4769]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 04:51:09.556145753 (6h0m15.928725108s) [seg_pos={0 0}, upload=4769]: Begin upload segment... Jul 19 04:51:09.556194817 (6h0m15.928774172s) [seg_pos={0 0}, upload=4769]: Done begin upload segment. Jul 19 04:51:09.556223893 (6h0m15.928803248s) [noise=true , address=79.127.219.38:4023 , piece_id=PTPOZGLYV4MZTOFZNN3NCGYL7DFAST4PXKEUIVTMFL7HV4GZTCLQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556252197 (6h0m15.928831552s) [noise=true , address=45.91.95.4:4011 , piece_id=ZMBAS6KKHYVIRE6OBL2NFVCWMM4TKQ7J64BD3SYXIK54XCEDI2XA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556579383 (6h0m15.929158738s) [noise=true , address=45.91.116.6:4007 , piece_id=A2AFVTZBXSGBOE7XIM3B5HRN4STITFBXO6MZRJ365URNAAOZ3PSQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556854469 (6h0m15.929433824s) [noise=true , address=104.36.82.19:4423 , piece_id=PIJUHDUND36HQL67I33D7A3EULMJHS7N55MN4OFQJ4OWNQ42O47A, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557094107 (6h0m15.929673462s) [noise=true , address=45.91.116.9:4264 , piece_id=GHM5BKZKMT2QOHHBOS4FSPOWEWSFQRGRVVR4GSNDPHVIFRZHQL2A, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557336381 (6h0m15.929915736s) [noise=true , address=79.127.205.235:4660 , piece_id=UOUOGE2VP7PX7EMRZLPMCWFCYEZERM3DQMCQ6NG4J2K2DEIZR63Q, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557557885 (6h0m15.93013724s) [noise=true , address=79.127.205.246:4280 , piece_id=FLBOZI3UGTQXAQBLZ2F45Z3OEYL65F5PQBXRF2DQMZ7NGF7VDRPA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557762296 (6h0m15.930341651s) [noise=true , address=79.127.205.233:4622 , piece_id=6WWNYQTH5MGAJOEB7BXKZDJOMCAZXDW66J3OEANGDNJDNUMLZM7Q, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558014689 (6h0m15.930594044s) [noise=true , address=64.49.13.138:4482 , piece_id=MAW2M4US2OQNOS3WI67AA7NIBTJDE5T2JQTXKFPBLGVGXKVCFGTQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558215974 (6h0m15.930795319s) [noise=true , address=77.90.31.10:4546 , piece_id=R6LCA5WZOEBDURL5S3MEK22YPVYXX6EXSS4J4W57543HELYXOKZA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558409364 (6h0m15.930988719s) [noise=true , address=45.91.95.5:4019 , piece_id=YTSBCN2LPHEMRRAZIHJGVB7NBOTRXVEHD2TWUCTTLBNFFYEWWDKA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558623504 (6h0m15.931202859s) [noise=true , address=79.127.201.215:4357 , piece_id=75XWAG2VQWEYCWN2Q3WRIZSBI7LDOISZJPXQLQ2GIPNEN34VLSUQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558805162 (6h0m15.931384517s) [noise=true , address=45.91.116.14:4219 , piece_id=PDW6JPRHCVMHHOWIHT6JKV4CTSC3JY6NDKDVFZSO47OYNPZ7PGGA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558999373 (6h0m15.931578728s) [noise=true , address=77.90.31.11:4564 , piece_id=L2U7JFJRDQOFA2AYVAGTI4ZW4VABCXKMXWBIUWQO2QNNUUBJ74BA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559184768 (6h0m15.931764123s) [noise=true , address=77.90.31.16:4768 , piece_id=YNLSONFM6COZEVF6ZNQR2EA6V4LSQFZJONIXR4AFTNNMFGPV57JA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559368719 (6h0m15.931948074s) [noise=true , address=79.127.220.98:4432 , piece_id=2SUTFQ7FVDPAM2H4W3OLW6AVH3VJ33IJFE4L52SA3ET5LGBMOMPQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559549706 (6h0m15.932129061s) [noise=true , address=77.90.31.18:4813 , piece_id=VDOT7BB4O7XYJHW6BFTAYQZZYLZU5PAUXVJGSXJ7FDI4TH425COA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559727836 (6h0m15.932307191s) [noise=true , address=79.127.219.51:4211 , piece_id=HO5PLWBVU5YAYNYMN352NA73SM4WBJV4WZ7XLSQTCQ76UAL2TOJA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:10.663359087 (6h0m17.035938442s) [upload=4769]: Next returned nil segment Jul 19 04:51:10.663378664 (6h0m17.035958019s) [upload=4769]: Waiting for error group managing segments... Jul 19 04:51:10.962945479 (6h0m17.335524834s) [noise=true , address=79.127.205.233:4622 , piece_id=6WWNYQTH5MGAJOEB7BXKZDJOMCAZXDW66J3OEANGDNJDNUMLZM7Q, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:10.967689807 (6h0m17.340269152s) [noise=true , address=79.127.205.235:4660 , piece_id=UOUOGE2VP7PX7EMRZLPMCWFCYEZERM3DQMCQ6NG4J2K2DEIZR63Q, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:10.976968631 (6h0m17.349547986s) [noise=true , address=64.49.13.138:4482 , piece_id=MAW2M4US2OQNOS3WI67AA7NIBTJDE5T2JQTXKFPBLGVGXKVCFGTQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.048750700 (6h0m17.421330055s) [noise=true , address=79.127.205.246:4280 , piece_id=FLBOZI3UGTQXAQBLZ2F45Z3OEYL65F5PQBXRF2DQMZ7NGF7VDRPA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.221642403 (6h0m17.594221758s) [noise=true , address=79.127.220.98:4432 , piece_id=2SUTFQ7FVDPAM2H4W3OLW6AVH3VJ33IJFE4L52SA3ET5LGBMOMPQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.317070435 (6h0m17.68964979s) [noise=true , address=45.91.95.4:4011 , piece_id=ZMBAS6KKHYVIRE6OBL2NFVCWMM4TKQ7J64BD3SYXIK54XCEDI2XA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.324002660 (6h0m17.696582015s) [noise=true , address=45.91.95.5:4019 , piece_id=YTSBCN2LPHEMRRAZIHJGVB7NBOTRXVEHD2TWUCTTLBNFFYEWWDKA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.460976317 (6h0m17.833555672s) [noise=true , address=79.127.201.215:4357 , piece_id=75XWAG2VQWEYCWN2Q3WRIZSBI7LDOISZJPXQLQ2GIPNEN34VLSUQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.462584603 (6h0m17.835163958s) [noise=true , address=45.91.116.6:4007 , piece_id=A2AFVTZBXSGBOE7XIM3B5HRN4STITFBXO6MZRJ365URNAAOZ3PSQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.490392973 (6h0m17.862972328s) [noise=true , address=45.91.116.9:4264 , piece_id=GHM5BKZKMT2QOHHBOS4FSPOWEWSFQRGRVVR4GSNDPHVIFRZHQL2A, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.502040780 (6h0m17.874620135s) [noise=true , address=45.91.116.14:4219 , piece_id=PDW6JPRHCVMHHOWIHT6JKV4CTSC3JY6NDKDVFZSO47OYNPZ7PGGA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.600434096 (6h0m17.973013451s) [noise=true , address=79.127.219.51:4211 , piece_id=HO5PLWBVU5YAYNYMN352NA73SM4WBJV4WZ7XLSQTCQ76UAL2TOJA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.609008984 (6h0m17.981588339s) [noise=true , address=104.36.82.19:4423 , piece_id=PIJUHDUND36HQL67I33D7A3EULMJHS7N55MN4OFQJ4OWNQ42O47A, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.620825273 (6h0m17.993404628s) [noise=true , address=79.127.219.38:4023 , piece_id=PTPOZGLYV4MZTOFZNN3NCGYL7DFAST4PXKEUIVTMFL7HV4GZTCLQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.620838659 (6h0m17.993418014s) [seg_pos={0 0}, upload=4769]: Segment reached optimal threshold of 14 pieces. Jul 19 04:51:11.621067807 (6h0m17.993647163s) [noise=true , address=77.90.31.11:4564 , piece_id=L2U7JFJRDQOFA2AYVAGTI4ZW4VABCXKMXWBIUWQO2QNNUUBJ74BA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection; write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection Jul 19 04:51:11.621134234 (6h0m17.993713589s) [noise=true , address=77.90.31.10:4546 , piece_id=R6LCA5WZOEBDURL5S3MEK22YPVYXX6EXSS4J4W57543HELYXOKZA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK): protocol: read tcp 188.166.225.170:58476->77.90.31.10:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621174191 (6h0m17.993753546s) [noise=true , address=77.90.31.16:4768 , piece_id=YNLSONFM6COZEVF6ZNQR2EA6V4LSQFZJONIXR4AFTNNMFGPV57JA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UchewmtxBrtSZSpVAJFdZmcTsQVQxkjK9re6cgUGqDhyonQeF): protocol: read tcp 188.166.225.170:56326->77.90.31.16:4768: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621216842 (6h0m17.993796197s) [noise=true , address=77.90.31.18:4813 , piece_id=VDOT7BB4O7XYJHW6BFTAYQZZYLZU5PAUXVJGSXJ7FDI4TH425COA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128WthA1x9wp1dV1vkpGnoveZQpU9CSgDCZfw5GMQuVpCNSVAM5): protocol: read tcp 188.166.225.170:49704->77.90.31.18:4813: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621234917 (6h0m17.993814272s) [seg_pos={0 0}, upload=4769]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 19 04:51:11.621301444 (6h0m17.993880799s) [upload=4769]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 19 04:51:11.621305402 (6h0m17.993884758s) [upload=4769]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 19 04:51:11.923136970 (6h0m18.295716325s) [upload=4769]: Done uploading segments... Jul 19 04:51:11.923156627 (6h0m18.295735972s) [upload=4769]: Upload finished. err: <nil> Duration: 22.819860276s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [.......XX...X...X....X..X....XXXXX.X.X.....XX.X.............] duration: 5.868002584s FAIL (2025-07-19 05:25:06 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== gment. successful: 13 optimal: 13 errs: <nil> Jul 19 05:14:35.936262577 (6h23m42.308841932s) [upload=5075]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 19 05:14:35.936281884 (6h23m42.308861239s) [upload=5075]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 19 05:14:36.234171969 (6h23m42.606751314s) [upload=5075]: Done uploading segments... Jul 19 05:14:36.234192037 (6h23m42.606771382s) [upload=5075]: Upload finished. err: <nil> Jul 19 05:14:55.587986728 (6h24m1.960566093s) [upload=5081]: Starting upload Jul 19 05:14:55.588051823 (6h24m1.960631178s) [upload=5081]: Done starting upload Jul 19 05:14:55.588146724 (6h24m1.960726079s) [upload=5081]: Uploading segments... Jul 19 05:14:55.588391873 (6h24m1.960971228s) [upload=5081]: Got next segment. Inline: false Jul 19 05:14:55.588649837 (6h24m1.961229193s) [upload=5081]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 05:14:55.588655057 (6h24m1.961234402s) [upload=5081]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 05:14:55.823387692 (6h24m2.195967047s) [seg_pos={0 0}, upload=5081]: Begin upload segment... Jul 19 05:14:55.823450572 (6h24m2.196029927s) [seg_pos={0 0}, upload=5081]: Done begin upload segment. Jul 19 05:14:55.823481952 (6h24m2.196061307s) [noise=true , address=77.90.31.17:4772 , piece_id=B544E3F2UC3UVE2ITUQAWGVDMSPS2HFIVHXZ2OLWB3T25AHMFAVA, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.823491401 (6h24m2.196070756s) [noise=true , address=45.91.116.10:4082 , piece_id=HZDMC4J5O5UHVDH5SOQBKAFPN6RJUHIWKEZ4OPVMW2HSN7NDZMZA, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.823923789 (6h24m2.196503144s) [noise=true , address=45.91.116.9:4262 , piece_id=FRIATDM7LPPHXYAMUISAYRTQWAKSTB4EYUANHGEMQHM6JQADDTRQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.823949737 (6h24m2.196529092s) [noise=true , address=45.91.116.12:4227 , piece_id=QB7YKZFXWMRB3VQ3HSFMMMLLX5YFWOXG4Y4GJGGYT6HDV4FQ65VA, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.824310158 (6h24m2.196889513s) [noise=true , address=45.91.116.4:4937 , piece_id=V2IOOA7DVCGZBH3D43ROYHT6ZECPY6GTC5M7NFO2RWPQB6EYGALA, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.824403045 (6h24m2.19698239s) [noise=true , address=77.90.31.11:4555 , piece_id=YICA5IVHSCC3RPC5FRKD26BSMY6N5WJQA5LCABJSRJYL6BZJYL6A, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.824651210 (6h24m2.197230565s) [noise=true , address=77.90.31.13:4627 , piece_id=E7GSSBZ653K2AA4TP6A4BRT5BDZCBODNRAZM3P6SYJXJFQU2BACQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.824716334 (6h24m2.197295689s) [noise=true , address=77.90.31.16:4734 , piece_id=YUW6MWSUDNLAC2VHLBT6H4T27YTNJ3RDK4BSGE7OZQCO3F2LBH4A, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.824960491 (6h24m2.197539856s) [noise=true , address=45.91.116.7:4010 , piece_id=BH76RM4NXGEAI2J2424ALK4PFSXUTBCYOH777BUOIDW7BW2IJUJA, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.824977494 (6h24m2.197556849s) [noise=true , address=45.91.116.14:4190 , piece_id=RUU2MICSAWOECAAYNOBS75SE2YKSUBMSSFKLPON7FWK77UNJEYGQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.825213285 (6h24m2.19779264s) [noise=true , address=45.91.116.8:4047 , piece_id=6UOFOV4VCVT6DB7D7Q23REAX7VZPAW5PSG5PLA54LISOUWMPQBBQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.825271547 (6h24m2.197850902s) [noise=true , address=77.90.31.18:4806 , piece_id=67PXFJMJB6XWNU6JMOKPO4BXKDD2Z225ZKTB7DZEA46OMSSXHHMQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.825445860 (6h24m2.198025215s) [noise=true , address=45.91.116.7:4009 , piece_id=HH5I32ZYT5LVXHDLBVSAVX6T23MJ6RGNPNAWPIQAREJ4ELPFJGYA, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.825568244 (6h24m2.198147599s) [noise=true , address=77.90.31.15:4697 , piece_id=W62ARDAVLI64P6VYSPVFRTDKB5H2WXTSCF2VKP3UN7JKGBCUEZTQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.825902985 (6h24m2.19848234s) [noise=true , address=45.91.116.5:4299 , piece_id=JQLK5LIMCMW6R2UZEJZSG6QAP2M65XBVE6JFEXY5KDWBVNDG2VCQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.826216495 (6h24m2.19879585s) [noise=true , address=77.90.31.14:4663 , piece_id=LNTFUWB6LUUFD2EB536ZHBSNVVXYZPOOB4DUZB2M6BXSSFEL3UMQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:58.633690556 (6h24m5.006269911s) [upload=5081]: Next returned nil segment Jul 19 05:14:58.633708100 (6h24m5.006287445s) [upload=5081]: Waiting for error group managing segments... Jul 19 05:14:58.915966948 (6h24m5.288546293s) [noise=true , address=45.91.116.7:4009 , piece_id=HH5I32ZYT5LVXHDLBVSAVX6T23MJ6RGNPNAWPIQAREJ4ELPFJGYA, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:58.916093541 (6h24m5.288672896s) [noise=true , address=45.91.116.12:4227 , piece_id=QB7YKZFXWMRB3VQ3HSFMMMLLX5YFWOXG4Y4GJGGYT6HDV4FQ65VA, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:58.923615462 (6h24m5.296194817s) [noise=true , address=45.91.116.14:4190 , piece_id=RUU2MICSAWOECAAYNOBS75SE2YKSUBMSSFKLPON7FWK77UNJEYGQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:58.928612934 (6h24m5.301192289s) [noise=true , address=45.91.116.7:4010 , piece_id=BH76RM4NXGEAI2J2424ALK4PFSXUTBCYOH777BUOIDW7BW2IJUJA, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:58.990915468 (6h24m5.363494833s) [noise=true , address=45.91.116.4:4937 , piece_id=V2IOOA7DVCGZBH3D43ROYHT6ZECPY6GTC5M7NFO2RWPQB6EYGALA, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:58.992136464 (6h24m5.364715819s) [noise=true , address=45.91.116.8:4047 , piece_id=6UOFOV4VCVT6DB7D7Q23REAX7VZPAW5PSG5PLA54LISOUWMPQBBQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:58.997711812 (6h24m5.370291157s) [noise=true , address=45.91.116.9:4262 , piece_id=FRIATDM7LPPHXYAMUISAYRTQWAKSTB4EYUANHGEMQHM6JQADDTRQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:59.010548391 (6h24m5.383127746s) [noise=true , address=45.91.116.10:4082 , piece_id=HZDMC4J5O5UHVDH5SOQBKAFPN6RJUHIWKEZ4OPVMW2HSN7NDZMZA, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:59.039607621 (6h24m5.412186996s) [noise=true , address=45.91.116.5:4299 , piece_id=JQLK5LIMCMW6R2UZEJZSG6QAP2M65XBVE6JFEXY5KDWBVNDG2VCQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:15:19.998482215 (6h24m26.37106157s) [noise=true , address=77.90.31.14:4663 , piece_id=LNTFUWB6LUUFD2EB536ZHBSNVVXYZPOOB4DUZB2M6BXSSFEL3UMQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:15:21.170089451 (6h24m27.542668806s) [noise=true , address=77.90.31.13:4627 , piece_id=E7GSSBZ653K2AA4TP6A4BRT5BDZCBODNRAZM3P6SYJXJFQU2BACQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:15:24.941469148 (6h24m31.314048503s) [noise=true , address=77.90.31.15:4697 , piece_id=W62ARDAVLI64P6VYSPVFRTDKB5H2WXTSCF2VKP3UN7JKGBCUEZTQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: expected piece hash; context deadline exceeded; EOF Jul 19 05:15:24.941530766 (6h24m31.314110121s) [noise=true , address=77.90.31.17:4772 , piece_id=B544E3F2UC3UVE2ITUQAWGVDMSPS2HFIVHXZ2OLWB3T25AHMFAVA, seg_pos={0 0}, upload=5081]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:44456->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:44456->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44456->77.90.31.17:4772: use of closed network connection Jul 19 05:15:24.941543591 (6h24m31.314122946s) [noise=true , address=77.90.31.16:4734 , piece_id=YUW6MWSUDNLAC2VHLBT6H4T27YTNJ3RDK4BSGE7OZQCO3F2LBH4A, seg_pos={0 0}, upload=5081]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Jul 19 05:15:24.941614786 (6h24m31.314194141s) [noise=true , address=77.90.31.11:4555 , piece_id=YICA5IVHSCC3RPC5FRKD26BSMY6N5WJQA5LCABJSRJYL6BZJYL6A, seg_pos={0 0}, upload=5081]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: write tcp 188.166.225.170:37392->77.90.31.11:4555: use of closed network connection; write tcp 188.166.225.170:37392->77.90.31.11:4555: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37392->77.90.31.11:4555: use of closed network connection Jul 19 05:15:24.941641888 (6h24m31.314221243s) [noise=true , address=77.90.31.18:4806 , piece_id=67PXFJMJB6XWNU6JMOKPO4BXKDD2Z225ZKTB7DZEA46OMSSXHHMQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:47114->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:47114->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47114->77.90.31.18:4806: use of closed network connection Jul 19 05:15:24.941663099 (6h24m31.314242454s) [seg_pos={0 0}, upload=5081]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 19 05:15:25.179949476 (6h24m31.552528831s) [upload=5081]: Done uploading segments... Jul 19 05:15:25.179975797 (6h24m31.552555152s) [upload=5081]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.868029747s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 5.591685072s FAIL (2025-07-19 05:24:11 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 4855s) [upload=4769]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 04:51:09.109709627 (6h0m15.482288982s) [upload=4769]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 04:51:09.556145753 (6h0m15.928725108s) [seg_pos={0 0}, upload=4769]: Begin upload segment... Jul 19 04:51:09.556194817 (6h0m15.928774172s) [seg_pos={0 0}, upload=4769]: Done begin upload segment. Jul 19 04:51:09.556223893 (6h0m15.928803248s) [noise=true , address=79.127.219.38:4023 , piece_id=PTPOZGLYV4MZTOFZNN3NCGYL7DFAST4PXKEUIVTMFL7HV4GZTCLQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556252197 (6h0m15.928831552s) [noise=true , address=45.91.95.4:4011 , piece_id=ZMBAS6KKHYVIRE6OBL2NFVCWMM4TKQ7J64BD3SYXIK54XCEDI2XA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556579383 (6h0m15.929158738s) [noise=true , address=45.91.116.6:4007 , piece_id=A2AFVTZBXSGBOE7XIM3B5HRN4STITFBXO6MZRJ365URNAAOZ3PSQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556854469 (6h0m15.929433824s) [noise=true , address=104.36.82.19:4423 , piece_id=PIJUHDUND36HQL67I33D7A3EULMJHS7N55MN4OFQJ4OWNQ42O47A, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557094107 (6h0m15.929673462s) [noise=true , address=45.91.116.9:4264 , piece_id=GHM5BKZKMT2QOHHBOS4FSPOWEWSFQRGRVVR4GSNDPHVIFRZHQL2A, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557336381 (6h0m15.929915736s) [noise=true , address=79.127.205.235:4660 , piece_id=UOUOGE2VP7PX7EMRZLPMCWFCYEZERM3DQMCQ6NG4J2K2DEIZR63Q, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557557885 (6h0m15.93013724s) [noise=true , address=79.127.205.246:4280 , piece_id=FLBOZI3UGTQXAQBLZ2F45Z3OEYL65F5PQBXRF2DQMZ7NGF7VDRPA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557762296 (6h0m15.930341651s) [noise=true , address=79.127.205.233:4622 , piece_id=6WWNYQTH5MGAJOEB7BXKZDJOMCAZXDW66J3OEANGDNJDNUMLZM7Q, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558014689 (6h0m15.930594044s) [noise=true , address=64.49.13.138:4482 , piece_id=MAW2M4US2OQNOS3WI67AA7NIBTJDE5T2JQTXKFPBLGVGXKVCFGTQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558215974 (6h0m15.930795319s) [noise=true , address=77.90.31.10:4546 , piece_id=R6LCA5WZOEBDURL5S3MEK22YPVYXX6EXSS4J4W57543HELYXOKZA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558409364 (6h0m15.930988719s) [noise=true , address=45.91.95.5:4019 , piece_id=YTSBCN2LPHEMRRAZIHJGVB7NBOTRXVEHD2TWUCTTLBNFFYEWWDKA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558623504 (6h0m15.931202859s) [noise=true , address=79.127.201.215:4357 , piece_id=75XWAG2VQWEYCWN2Q3WRIZSBI7LDOISZJPXQLQ2GIPNEN34VLSUQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558805162 (6h0m15.931384517s) [noise=true , address=45.91.116.14:4219 , piece_id=PDW6JPRHCVMHHOWIHT6JKV4CTSC3JY6NDKDVFZSO47OYNPZ7PGGA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558999373 (6h0m15.931578728s) [noise=true , address=77.90.31.11:4564 , piece_id=L2U7JFJRDQOFA2AYVAGTI4ZW4VABCXKMXWBIUWQO2QNNUUBJ74BA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559184768 (6h0m15.931764123s) [noise=true , address=77.90.31.16:4768 , piece_id=YNLSONFM6COZEVF6ZNQR2EA6V4LSQFZJONIXR4AFTNNMFGPV57JA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559368719 (6h0m15.931948074s) [noise=true , address=79.127.220.98:4432 , piece_id=2SUTFQ7FVDPAM2H4W3OLW6AVH3VJ33IJFE4L52SA3ET5LGBMOMPQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559549706 (6h0m15.932129061s) [noise=true , address=77.90.31.18:4813 , piece_id=VDOT7BB4O7XYJHW6BFTAYQZZYLZU5PAUXVJGSXJ7FDI4TH425COA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559727836 (6h0m15.932307191s) [noise=true , address=79.127.219.51:4211 , piece_id=HO5PLWBVU5YAYNYMN352NA73SM4WBJV4WZ7XLSQTCQ76UAL2TOJA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:10.663359087 (6h0m17.035938442s) [upload=4769]: Next returned nil segment Jul 19 04:51:10.663378664 (6h0m17.035958019s) [upload=4769]: Waiting for error group managing segments... Jul 19 04:51:10.962945479 (6h0m17.335524834s) [noise=true , address=79.127.205.233:4622 , piece_id=6WWNYQTH5MGAJOEB7BXKZDJOMCAZXDW66J3OEANGDNJDNUMLZM7Q, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:10.967689807 (6h0m17.340269152s) [noise=true , address=79.127.205.235:4660 , piece_id=UOUOGE2VP7PX7EMRZLPMCWFCYEZERM3DQMCQ6NG4J2K2DEIZR63Q, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:10.976968631 (6h0m17.349547986s) [noise=true , address=64.49.13.138:4482 , piece_id=MAW2M4US2OQNOS3WI67AA7NIBTJDE5T2JQTXKFPBLGVGXKVCFGTQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.048750700 (6h0m17.421330055s) [noise=true , address=79.127.205.246:4280 , piece_id=FLBOZI3UGTQXAQBLZ2F45Z3OEYL65F5PQBXRF2DQMZ7NGF7VDRPA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.221642403 (6h0m17.594221758s) [noise=true , address=79.127.220.98:4432 , piece_id=2SUTFQ7FVDPAM2H4W3OLW6AVH3VJ33IJFE4L52SA3ET5LGBMOMPQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.317070435 (6h0m17.68964979s) [noise=true , address=45.91.95.4:4011 , piece_id=ZMBAS6KKHYVIRE6OBL2NFVCWMM4TKQ7J64BD3SYXIK54XCEDI2XA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.324002660 (6h0m17.696582015s) [noise=true , address=45.91.95.5:4019 , piece_id=YTSBCN2LPHEMRRAZIHJGVB7NBOTRXVEHD2TWUCTTLBNFFYEWWDKA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.460976317 (6h0m17.833555672s) [noise=true , address=79.127.201.215:4357 , piece_id=75XWAG2VQWEYCWN2Q3WRIZSBI7LDOISZJPXQLQ2GIPNEN34VLSUQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.462584603 (6h0m17.835163958s) [noise=true , address=45.91.116.6:4007 , piece_id=A2AFVTZBXSGBOE7XIM3B5HRN4STITFBXO6MZRJ365URNAAOZ3PSQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.490392973 (6h0m17.862972328s) [noise=true , address=45.91.116.9:4264 , piece_id=GHM5BKZKMT2QOHHBOS4FSPOWEWSFQRGRVVR4GSNDPHVIFRZHQL2A, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.502040780 (6h0m17.874620135s) [noise=true , address=45.91.116.14:4219 , piece_id=PDW6JPRHCVMHHOWIHT6JKV4CTSC3JY6NDKDVFZSO47OYNPZ7PGGA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.600434096 (6h0m17.973013451s) [noise=true , address=79.127.219.51:4211 , piece_id=HO5PLWBVU5YAYNYMN352NA73SM4WBJV4WZ7XLSQTCQ76UAL2TOJA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.609008984 (6h0m17.981588339s) [noise=true , address=104.36.82.19:4423 , piece_id=PIJUHDUND36HQL67I33D7A3EULMJHS7N55MN4OFQJ4OWNQ42O47A, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.620825273 (6h0m17.993404628s) [noise=true , address=79.127.219.38:4023 , piece_id=PTPOZGLYV4MZTOFZNN3NCGYL7DFAST4PXKEUIVTMFL7HV4GZTCLQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.620838659 (6h0m17.993418014s) [seg_pos={0 0}, upload=4769]: Segment reached optimal threshold of 14 pieces. Jul 19 04:51:11.621067807 (6h0m17.993647163s) [noise=true , address=77.90.31.11:4564 , piece_id=L2U7JFJRDQOFA2AYVAGTI4ZW4VABCXKMXWBIUWQO2QNNUUBJ74BA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection; write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection Jul 19 04:51:11.621134234 (6h0m17.993713589s) [noise=true , address=77.90.31.10:4546 , piece_id=R6LCA5WZOEBDURL5S3MEK22YPVYXX6EXSS4J4W57543HELYXOKZA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK): protocol: read tcp 188.166.225.170:58476->77.90.31.10:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621174191 (6h0m17.993753546s) [noise=true , address=77.90.31.16:4768 , piece_id=YNLSONFM6COZEVF6ZNQR2EA6V4LSQFZJONIXR4AFTNNMFGPV57JA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UchewmtxBrtSZSpVAJFdZmcTsQVQxkjK9re6cgUGqDhyonQeF): protocol: read tcp 188.166.225.170:56326->77.90.31.16:4768: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621216842 (6h0m17.993796197s) [noise=true , address=77.90.31.18:4813 , piece_id=VDOT7BB4O7XYJHW6BFTAYQZZYLZU5PAUXVJGSXJ7FDI4TH425COA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128WthA1x9wp1dV1vkpGnoveZQpU9CSgDCZfw5GMQuVpCNSVAM5): protocol: read tcp 188.166.225.170:49704->77.90.31.18:4813: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621234917 (6h0m17.993814272s) [seg_pos={0 0}, upload=4769]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 19 04:51:11.621301444 (6h0m17.993880799s) [upload=4769]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 19 04:51:11.621305402 (6h0m17.993884758s) [upload=4769]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 19 04:51:11.923136970 (6h0m18.295716325s) [upload=4769]: Done uploading segments... Jul 19 04:51:11.923156627 (6h0m18.295735972s) [upload=4769]: Upload finished. err: <nil> Duration: 5.591745256s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [........XX...X...X....X..X....XXXXX.X.X.....XX.X............] duration: 5.859974112s FAIL (2025-07-19 05:23:55 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== gment. successful: 13 optimal: 13 errs: <nil> Jul 19 05:14:35.936262577 (6h23m42.308841932s) [upload=5075]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 19 05:14:35.936281884 (6h23m42.308861239s) [upload=5075]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 19 05:14:36.234171969 (6h23m42.606751314s) [upload=5075]: Done uploading segments... Jul 19 05:14:36.234192037 (6h23m42.606771382s) [upload=5075]: Upload finished. err: <nil> Jul 19 05:14:55.587986728 (6h24m1.960566093s) [upload=5081]: Starting upload Jul 19 05:14:55.588051823 (6h24m1.960631178s) [upload=5081]: Done starting upload Jul 19 05:14:55.588146724 (6h24m1.960726079s) [upload=5081]: Uploading segments... Jul 19 05:14:55.588391873 (6h24m1.960971228s) [upload=5081]: Got next segment. Inline: false Jul 19 05:14:55.588649837 (6h24m1.961229193s) [upload=5081]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 05:14:55.588655057 (6h24m1.961234402s) [upload=5081]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 05:14:55.823387692 (6h24m2.195967047s) [seg_pos={0 0}, upload=5081]: Begin upload segment... Jul 19 05:14:55.823450572 (6h24m2.196029927s) [seg_pos={0 0}, upload=5081]: Done begin upload segment. Jul 19 05:14:55.823481952 (6h24m2.196061307s) [noise=true , address=77.90.31.17:4772 , piece_id=B544E3F2UC3UVE2ITUQAWGVDMSPS2HFIVHXZ2OLWB3T25AHMFAVA, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.823491401 (6h24m2.196070756s) [noise=true , address=45.91.116.10:4082 , piece_id=HZDMC4J5O5UHVDH5SOQBKAFPN6RJUHIWKEZ4OPVMW2HSN7NDZMZA, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.823923789 (6h24m2.196503144s) [noise=true , address=45.91.116.9:4262 , piece_id=FRIATDM7LPPHXYAMUISAYRTQWAKSTB4EYUANHGEMQHM6JQADDTRQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.823949737 (6h24m2.196529092s) [noise=true , address=45.91.116.12:4227 , piece_id=QB7YKZFXWMRB3VQ3HSFMMMLLX5YFWOXG4Y4GJGGYT6HDV4FQ65VA, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.824310158 (6h24m2.196889513s) [noise=true , address=45.91.116.4:4937 , piece_id=V2IOOA7DVCGZBH3D43ROYHT6ZECPY6GTC5M7NFO2RWPQB6EYGALA, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.824403045 (6h24m2.19698239s) [noise=true , address=77.90.31.11:4555 , piece_id=YICA5IVHSCC3RPC5FRKD26BSMY6N5WJQA5LCABJSRJYL6BZJYL6A, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.824651210 (6h24m2.197230565s) [noise=true , address=77.90.31.13:4627 , piece_id=E7GSSBZ653K2AA4TP6A4BRT5BDZCBODNRAZM3P6SYJXJFQU2BACQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.824716334 (6h24m2.197295689s) [noise=true , address=77.90.31.16:4734 , piece_id=YUW6MWSUDNLAC2VHLBT6H4T27YTNJ3RDK4BSGE7OZQCO3F2LBH4A, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.824960491 (6h24m2.197539856s) [noise=true , address=45.91.116.7:4010 , piece_id=BH76RM4NXGEAI2J2424ALK4PFSXUTBCYOH777BUOIDW7BW2IJUJA, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.824977494 (6h24m2.197556849s) [noise=true , address=45.91.116.14:4190 , piece_id=RUU2MICSAWOECAAYNOBS75SE2YKSUBMSSFKLPON7FWK77UNJEYGQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.825213285 (6h24m2.19779264s) [noise=true , address=45.91.116.8:4047 , piece_id=6UOFOV4VCVT6DB7D7Q23REAX7VZPAW5PSG5PLA54LISOUWMPQBBQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.825271547 (6h24m2.197850902s) [noise=true , address=77.90.31.18:4806 , piece_id=67PXFJMJB6XWNU6JMOKPO4BXKDD2Z225ZKTB7DZEA46OMSSXHHMQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.825445860 (6h24m2.198025215s) [noise=true , address=45.91.116.7:4009 , piece_id=HH5I32ZYT5LVXHDLBVSAVX6T23MJ6RGNPNAWPIQAREJ4ELPFJGYA, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.825568244 (6h24m2.198147599s) [noise=true , address=77.90.31.15:4697 , piece_id=W62ARDAVLI64P6VYSPVFRTDKB5H2WXTSCF2VKP3UN7JKGBCUEZTQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.825902985 (6h24m2.19848234s) [noise=true , address=45.91.116.5:4299 , piece_id=JQLK5LIMCMW6R2UZEJZSG6QAP2M65XBVE6JFEXY5KDWBVNDG2VCQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.826216495 (6h24m2.19879585s) [noise=true , address=77.90.31.14:4663 , piece_id=LNTFUWB6LUUFD2EB536ZHBSNVVXYZPOOB4DUZB2M6BXSSFEL3UMQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:58.633690556 (6h24m5.006269911s) [upload=5081]: Next returned nil segment Jul 19 05:14:58.633708100 (6h24m5.006287445s) [upload=5081]: Waiting for error group managing segments... Jul 19 05:14:58.915966948 (6h24m5.288546293s) [noise=true , address=45.91.116.7:4009 , piece_id=HH5I32ZYT5LVXHDLBVSAVX6T23MJ6RGNPNAWPIQAREJ4ELPFJGYA, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:58.916093541 (6h24m5.288672896s) [noise=true , address=45.91.116.12:4227 , piece_id=QB7YKZFXWMRB3VQ3HSFMMMLLX5YFWOXG4Y4GJGGYT6HDV4FQ65VA, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:58.923615462 (6h24m5.296194817s) [noise=true , address=45.91.116.14:4190 , piece_id=RUU2MICSAWOECAAYNOBS75SE2YKSUBMSSFKLPON7FWK77UNJEYGQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:58.928612934 (6h24m5.301192289s) [noise=true , address=45.91.116.7:4010 , piece_id=BH76RM4NXGEAI2J2424ALK4PFSXUTBCYOH777BUOIDW7BW2IJUJA, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:58.990915468 (6h24m5.363494833s) [noise=true , address=45.91.116.4:4937 , piece_id=V2IOOA7DVCGZBH3D43ROYHT6ZECPY6GTC5M7NFO2RWPQB6EYGALA, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:58.992136464 (6h24m5.364715819s) [noise=true , address=45.91.116.8:4047 , piece_id=6UOFOV4VCVT6DB7D7Q23REAX7VZPAW5PSG5PLA54LISOUWMPQBBQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:58.997711812 (6h24m5.370291157s) [noise=true , address=45.91.116.9:4262 , piece_id=FRIATDM7LPPHXYAMUISAYRTQWAKSTB4EYUANHGEMQHM6JQADDTRQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:59.010548391 (6h24m5.383127746s) [noise=true , address=45.91.116.10:4082 , piece_id=HZDMC4J5O5UHVDH5SOQBKAFPN6RJUHIWKEZ4OPVMW2HSN7NDZMZA, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:59.039607621 (6h24m5.412186996s) [noise=true , address=45.91.116.5:4299 , piece_id=JQLK5LIMCMW6R2UZEJZSG6QAP2M65XBVE6JFEXY5KDWBVNDG2VCQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:15:19.998482215 (6h24m26.37106157s) [noise=true , address=77.90.31.14:4663 , piece_id=LNTFUWB6LUUFD2EB536ZHBSNVVXYZPOOB4DUZB2M6BXSSFEL3UMQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:15:21.170089451 (6h24m27.542668806s) [noise=true , address=77.90.31.13:4627 , piece_id=E7GSSBZ653K2AA4TP6A4BRT5BDZCBODNRAZM3P6SYJXJFQU2BACQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:15:24.941469148 (6h24m31.314048503s) [noise=true , address=77.90.31.15:4697 , piece_id=W62ARDAVLI64P6VYSPVFRTDKB5H2WXTSCF2VKP3UN7JKGBCUEZTQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: expected piece hash; context deadline exceeded; EOF Jul 19 05:15:24.941530766 (6h24m31.314110121s) [noise=true , address=77.90.31.17:4772 , piece_id=B544E3F2UC3UVE2ITUQAWGVDMSPS2HFIVHXZ2OLWB3T25AHMFAVA, seg_pos={0 0}, upload=5081]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:44456->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:44456->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44456->77.90.31.17:4772: use of closed network connection Jul 19 05:15:24.941543591 (6h24m31.314122946s) [noise=true , address=77.90.31.16:4734 , piece_id=YUW6MWSUDNLAC2VHLBT6H4T27YTNJ3RDK4BSGE7OZQCO3F2LBH4A, seg_pos={0 0}, upload=5081]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Jul 19 05:15:24.941614786 (6h24m31.314194141s) [noise=true , address=77.90.31.11:4555 , piece_id=YICA5IVHSCC3RPC5FRKD26BSMY6N5WJQA5LCABJSRJYL6BZJYL6A, seg_pos={0 0}, upload=5081]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: write tcp 188.166.225.170:37392->77.90.31.11:4555: use of closed network connection; write tcp 188.166.225.170:37392->77.90.31.11:4555: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37392->77.90.31.11:4555: use of closed network connection Jul 19 05:15:24.941641888 (6h24m31.314221243s) [noise=true , address=77.90.31.18:4806 , piece_id=67PXFJMJB6XWNU6JMOKPO4BXKDD2Z225ZKTB7DZEA46OMSSXHHMQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:47114->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:47114->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47114->77.90.31.18:4806: use of closed network connection Jul 19 05:15:24.941663099 (6h24m31.314242454s) [seg_pos={0 0}, upload=5081]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 19 05:15:25.179949476 (6h24m31.552528831s) [upload=5081]: Done uploading segments... Jul 19 05:15:25.179975797 (6h24m31.552555152s) [upload=5081]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.860018027s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [.........XX...X...X....X..X....XXXXX.X.X.....XX.X...........] duration: 5.789282823s FAIL (2025-07-19 05:23:28 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== gment. successful: 13 optimal: 13 errs: <nil> Jul 19 05:14:35.936262577 (6h23m42.308841932s) [upload=5075]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 19 05:14:35.936281884 (6h23m42.308861239s) [upload=5075]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 19 05:14:36.234171969 (6h23m42.606751314s) [upload=5075]: Done uploading segments... Jul 19 05:14:36.234192037 (6h23m42.606771382s) [upload=5075]: Upload finished. err: <nil> Jul 19 05:14:55.587986728 (6h24m1.960566093s) [upload=5081]: Starting upload Jul 19 05:14:55.588051823 (6h24m1.960631178s) [upload=5081]: Done starting upload Jul 19 05:14:55.588146724 (6h24m1.960726079s) [upload=5081]: Uploading segments... Jul 19 05:14:55.588391873 (6h24m1.960971228s) [upload=5081]: Got next segment. Inline: false Jul 19 05:14:55.588649837 (6h24m1.961229193s) [upload=5081]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 05:14:55.588655057 (6h24m1.961234402s) [upload=5081]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 05:14:55.823387692 (6h24m2.195967047s) [seg_pos={0 0}, upload=5081]: Begin upload segment... Jul 19 05:14:55.823450572 (6h24m2.196029927s) [seg_pos={0 0}, upload=5081]: Done begin upload segment. Jul 19 05:14:55.823481952 (6h24m2.196061307s) [noise=true , address=77.90.31.17:4772 , piece_id=B544E3F2UC3UVE2ITUQAWGVDMSPS2HFIVHXZ2OLWB3T25AHMFAVA, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.823491401 (6h24m2.196070756s) [noise=true , address=45.91.116.10:4082 , piece_id=HZDMC4J5O5UHVDH5SOQBKAFPN6RJUHIWKEZ4OPVMW2HSN7NDZMZA, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.823923789 (6h24m2.196503144s) [noise=true , address=45.91.116.9:4262 , piece_id=FRIATDM7LPPHXYAMUISAYRTQWAKSTB4EYUANHGEMQHM6JQADDTRQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.823949737 (6h24m2.196529092s) [noise=true , address=45.91.116.12:4227 , piece_id=QB7YKZFXWMRB3VQ3HSFMMMLLX5YFWOXG4Y4GJGGYT6HDV4FQ65VA, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.824310158 (6h24m2.196889513s) [noise=true , address=45.91.116.4:4937 , piece_id=V2IOOA7DVCGZBH3D43ROYHT6ZECPY6GTC5M7NFO2RWPQB6EYGALA, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.824403045 (6h24m2.19698239s) [noise=true , address=77.90.31.11:4555 , piece_id=YICA5IVHSCC3RPC5FRKD26BSMY6N5WJQA5LCABJSRJYL6BZJYL6A, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.824651210 (6h24m2.197230565s) [noise=true , address=77.90.31.13:4627 , piece_id=E7GSSBZ653K2AA4TP6A4BRT5BDZCBODNRAZM3P6SYJXJFQU2BACQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.824716334 (6h24m2.197295689s) [noise=true , address=77.90.31.16:4734 , piece_id=YUW6MWSUDNLAC2VHLBT6H4T27YTNJ3RDK4BSGE7OZQCO3F2LBH4A, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.824960491 (6h24m2.197539856s) [noise=true , address=45.91.116.7:4010 , piece_id=BH76RM4NXGEAI2J2424ALK4PFSXUTBCYOH777BUOIDW7BW2IJUJA, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.824977494 (6h24m2.197556849s) [noise=true , address=45.91.116.14:4190 , piece_id=RUU2MICSAWOECAAYNOBS75SE2YKSUBMSSFKLPON7FWK77UNJEYGQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.825213285 (6h24m2.19779264s) [noise=true , address=45.91.116.8:4047 , piece_id=6UOFOV4VCVT6DB7D7Q23REAX7VZPAW5PSG5PLA54LISOUWMPQBBQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.825271547 (6h24m2.197850902s) [noise=true , address=77.90.31.18:4806 , piece_id=67PXFJMJB6XWNU6JMOKPO4BXKDD2Z225ZKTB7DZEA46OMSSXHHMQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.825445860 (6h24m2.198025215s) [noise=true , address=45.91.116.7:4009 , piece_id=HH5I32ZYT5LVXHDLBVSAVX6T23MJ6RGNPNAWPIQAREJ4ELPFJGYA, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.825568244 (6h24m2.198147599s) [noise=true , address=77.90.31.15:4697 , piece_id=W62ARDAVLI64P6VYSPVFRTDKB5H2WXTSCF2VKP3UN7JKGBCUEZTQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.825902985 (6h24m2.19848234s) [noise=true , address=45.91.116.5:4299 , piece_id=JQLK5LIMCMW6R2UZEJZSG6QAP2M65XBVE6JFEXY5KDWBVNDG2VCQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.826216495 (6h24m2.19879585s) [noise=true , address=77.90.31.14:4663 , piece_id=LNTFUWB6LUUFD2EB536ZHBSNVVXYZPOOB4DUZB2M6BXSSFEL3UMQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:58.633690556 (6h24m5.006269911s) [upload=5081]: Next returned nil segment Jul 19 05:14:58.633708100 (6h24m5.006287445s) [upload=5081]: Waiting for error group managing segments... Jul 19 05:14:58.915966948 (6h24m5.288546293s) [noise=true , address=45.91.116.7:4009 , piece_id=HH5I32ZYT5LVXHDLBVSAVX6T23MJ6RGNPNAWPIQAREJ4ELPFJGYA, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:58.916093541 (6h24m5.288672896s) [noise=true , address=45.91.116.12:4227 , piece_id=QB7YKZFXWMRB3VQ3HSFMMMLLX5YFWOXG4Y4GJGGYT6HDV4FQ65VA, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:58.923615462 (6h24m5.296194817s) [noise=true , address=45.91.116.14:4190 , piece_id=RUU2MICSAWOECAAYNOBS75SE2YKSUBMSSFKLPON7FWK77UNJEYGQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:58.928612934 (6h24m5.301192289s) [noise=true , address=45.91.116.7:4010 , piece_id=BH76RM4NXGEAI2J2424ALK4PFSXUTBCYOH777BUOIDW7BW2IJUJA, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:58.990915468 (6h24m5.363494833s) [noise=true , address=45.91.116.4:4937 , piece_id=V2IOOA7DVCGZBH3D43ROYHT6ZECPY6GTC5M7NFO2RWPQB6EYGALA, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:58.992136464 (6h24m5.364715819s) [noise=true , address=45.91.116.8:4047 , piece_id=6UOFOV4VCVT6DB7D7Q23REAX7VZPAW5PSG5PLA54LISOUWMPQBBQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:58.997711812 (6h24m5.370291157s) [noise=true , address=45.91.116.9:4262 , piece_id=FRIATDM7LPPHXYAMUISAYRTQWAKSTB4EYUANHGEMQHM6JQADDTRQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:59.010548391 (6h24m5.383127746s) [noise=true , address=45.91.116.10:4082 , piece_id=HZDMC4J5O5UHVDH5SOQBKAFPN6RJUHIWKEZ4OPVMW2HSN7NDZMZA, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:59.039607621 (6h24m5.412186996s) [noise=true , address=45.91.116.5:4299 , piece_id=JQLK5LIMCMW6R2UZEJZSG6QAP2M65XBVE6JFEXY5KDWBVNDG2VCQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:15:19.998482215 (6h24m26.37106157s) [noise=true , address=77.90.31.14:4663 , piece_id=LNTFUWB6LUUFD2EB536ZHBSNVVXYZPOOB4DUZB2M6BXSSFEL3UMQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:15:21.170089451 (6h24m27.542668806s) [noise=true , address=77.90.31.13:4627 , piece_id=E7GSSBZ653K2AA4TP6A4BRT5BDZCBODNRAZM3P6SYJXJFQU2BACQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:15:24.941469148 (6h24m31.314048503s) [noise=true , address=77.90.31.15:4697 , piece_id=W62ARDAVLI64P6VYSPVFRTDKB5H2WXTSCF2VKP3UN7JKGBCUEZTQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: expected piece hash; context deadline exceeded; EOF Jul 19 05:15:24.941530766 (6h24m31.314110121s) [noise=true , address=77.90.31.17:4772 , piece_id=B544E3F2UC3UVE2ITUQAWGVDMSPS2HFIVHXZ2OLWB3T25AHMFAVA, seg_pos={0 0}, upload=5081]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:44456->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:44456->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44456->77.90.31.17:4772: use of closed network connection Jul 19 05:15:24.941543591 (6h24m31.314122946s) [noise=true , address=77.90.31.16:4734 , piece_id=YUW6MWSUDNLAC2VHLBT6H4T27YTNJ3RDK4BSGE7OZQCO3F2LBH4A, seg_pos={0 0}, upload=5081]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Jul 19 05:15:24.941614786 (6h24m31.314194141s) [noise=true , address=77.90.31.11:4555 , piece_id=YICA5IVHSCC3RPC5FRKD26BSMY6N5WJQA5LCABJSRJYL6BZJYL6A, seg_pos={0 0}, upload=5081]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: write tcp 188.166.225.170:37392->77.90.31.11:4555: use of closed network connection; write tcp 188.166.225.170:37392->77.90.31.11:4555: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37392->77.90.31.11:4555: use of closed network connection Jul 19 05:15:24.941641888 (6h24m31.314221243s) [noise=true , address=77.90.31.18:4806 , piece_id=67PXFJMJB6XWNU6JMOKPO4BXKDD2Z225ZKTB7DZEA46OMSSXHHMQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:47114->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:47114->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47114->77.90.31.18:4806: use of closed network connection Jul 19 05:15:24.941663099 (6h24m31.314242454s) [seg_pos={0 0}, upload=5081]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 19 05:15:25.179949476 (6h24m31.552528831s) [upload=5081]: Done uploading segments... Jul 19 05:15:25.179975797 (6h24m31.552555152s) [upload=5081]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.789307011s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 6.582073845s FAIL (2025-07-19 05:23:13 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 4855s) [upload=4769]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 04:51:09.109709627 (6h0m15.482288982s) [upload=4769]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 04:51:09.556145753 (6h0m15.928725108s) [seg_pos={0 0}, upload=4769]: Begin upload segment... Jul 19 04:51:09.556194817 (6h0m15.928774172s) [seg_pos={0 0}, upload=4769]: Done begin upload segment. Jul 19 04:51:09.556223893 (6h0m15.928803248s) [noise=true , address=79.127.219.38:4023 , piece_id=PTPOZGLYV4MZTOFZNN3NCGYL7DFAST4PXKEUIVTMFL7HV4GZTCLQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556252197 (6h0m15.928831552s) [noise=true , address=45.91.95.4:4011 , piece_id=ZMBAS6KKHYVIRE6OBL2NFVCWMM4TKQ7J64BD3SYXIK54XCEDI2XA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556579383 (6h0m15.929158738s) [noise=true , address=45.91.116.6:4007 , piece_id=A2AFVTZBXSGBOE7XIM3B5HRN4STITFBXO6MZRJ365URNAAOZ3PSQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556854469 (6h0m15.929433824s) [noise=true , address=104.36.82.19:4423 , piece_id=PIJUHDUND36HQL67I33D7A3EULMJHS7N55MN4OFQJ4OWNQ42O47A, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557094107 (6h0m15.929673462s) [noise=true , address=45.91.116.9:4264 , piece_id=GHM5BKZKMT2QOHHBOS4FSPOWEWSFQRGRVVR4GSNDPHVIFRZHQL2A, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557336381 (6h0m15.929915736s) [noise=true , address=79.127.205.235:4660 , piece_id=UOUOGE2VP7PX7EMRZLPMCWFCYEZERM3DQMCQ6NG4J2K2DEIZR63Q, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557557885 (6h0m15.93013724s) [noise=true , address=79.127.205.246:4280 , piece_id=FLBOZI3UGTQXAQBLZ2F45Z3OEYL65F5PQBXRF2DQMZ7NGF7VDRPA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557762296 (6h0m15.930341651s) [noise=true , address=79.127.205.233:4622 , piece_id=6WWNYQTH5MGAJOEB7BXKZDJOMCAZXDW66J3OEANGDNJDNUMLZM7Q, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558014689 (6h0m15.930594044s) [noise=true , address=64.49.13.138:4482 , piece_id=MAW2M4US2OQNOS3WI67AA7NIBTJDE5T2JQTXKFPBLGVGXKVCFGTQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558215974 (6h0m15.930795319s) [noise=true , address=77.90.31.10:4546 , piece_id=R6LCA5WZOEBDURL5S3MEK22YPVYXX6EXSS4J4W57543HELYXOKZA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558409364 (6h0m15.930988719s) [noise=true , address=45.91.95.5:4019 , piece_id=YTSBCN2LPHEMRRAZIHJGVB7NBOTRXVEHD2TWUCTTLBNFFYEWWDKA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558623504 (6h0m15.931202859s) [noise=true , address=79.127.201.215:4357 , piece_id=75XWAG2VQWEYCWN2Q3WRIZSBI7LDOISZJPXQLQ2GIPNEN34VLSUQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558805162 (6h0m15.931384517s) [noise=true , address=45.91.116.14:4219 , piece_id=PDW6JPRHCVMHHOWIHT6JKV4CTSC3JY6NDKDVFZSO47OYNPZ7PGGA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558999373 (6h0m15.931578728s) [noise=true , address=77.90.31.11:4564 , piece_id=L2U7JFJRDQOFA2AYVAGTI4ZW4VABCXKMXWBIUWQO2QNNUUBJ74BA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559184768 (6h0m15.931764123s) [noise=true , address=77.90.31.16:4768 , piece_id=YNLSONFM6COZEVF6ZNQR2EA6V4LSQFZJONIXR4AFTNNMFGPV57JA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559368719 (6h0m15.931948074s) [noise=true , address=79.127.220.98:4432 , piece_id=2SUTFQ7FVDPAM2H4W3OLW6AVH3VJ33IJFE4L52SA3ET5LGBMOMPQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559549706 (6h0m15.932129061s) [noise=true , address=77.90.31.18:4813 , piece_id=VDOT7BB4O7XYJHW6BFTAYQZZYLZU5PAUXVJGSXJ7FDI4TH425COA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559727836 (6h0m15.932307191s) [noise=true , address=79.127.219.51:4211 , piece_id=HO5PLWBVU5YAYNYMN352NA73SM4WBJV4WZ7XLSQTCQ76UAL2TOJA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:10.663359087 (6h0m17.035938442s) [upload=4769]: Next returned nil segment Jul 19 04:51:10.663378664 (6h0m17.035958019s) [upload=4769]: Waiting for error group managing segments... Jul 19 04:51:10.962945479 (6h0m17.335524834s) [noise=true , address=79.127.205.233:4622 , piece_id=6WWNYQTH5MGAJOEB7BXKZDJOMCAZXDW66J3OEANGDNJDNUMLZM7Q, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:10.967689807 (6h0m17.340269152s) [noise=true , address=79.127.205.235:4660 , piece_id=UOUOGE2VP7PX7EMRZLPMCWFCYEZERM3DQMCQ6NG4J2K2DEIZR63Q, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:10.976968631 (6h0m17.349547986s) [noise=true , address=64.49.13.138:4482 , piece_id=MAW2M4US2OQNOS3WI67AA7NIBTJDE5T2JQTXKFPBLGVGXKVCFGTQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.048750700 (6h0m17.421330055s) [noise=true , address=79.127.205.246:4280 , piece_id=FLBOZI3UGTQXAQBLZ2F45Z3OEYL65F5PQBXRF2DQMZ7NGF7VDRPA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.221642403 (6h0m17.594221758s) [noise=true , address=79.127.220.98:4432 , piece_id=2SUTFQ7FVDPAM2H4W3OLW6AVH3VJ33IJFE4L52SA3ET5LGBMOMPQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.317070435 (6h0m17.68964979s) [noise=true , address=45.91.95.4:4011 , piece_id=ZMBAS6KKHYVIRE6OBL2NFVCWMM4TKQ7J64BD3SYXIK54XCEDI2XA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.324002660 (6h0m17.696582015s) [noise=true , address=45.91.95.5:4019 , piece_id=YTSBCN2LPHEMRRAZIHJGVB7NBOTRXVEHD2TWUCTTLBNFFYEWWDKA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.460976317 (6h0m17.833555672s) [noise=true , address=79.127.201.215:4357 , piece_id=75XWAG2VQWEYCWN2Q3WRIZSBI7LDOISZJPXQLQ2GIPNEN34VLSUQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.462584603 (6h0m17.835163958s) [noise=true , address=45.91.116.6:4007 , piece_id=A2AFVTZBXSGBOE7XIM3B5HRN4STITFBXO6MZRJ365URNAAOZ3PSQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.490392973 (6h0m17.862972328s) [noise=true , address=45.91.116.9:4264 , piece_id=GHM5BKZKMT2QOHHBOS4FSPOWEWSFQRGRVVR4GSNDPHVIFRZHQL2A, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.502040780 (6h0m17.874620135s) [noise=true , address=45.91.116.14:4219 , piece_id=PDW6JPRHCVMHHOWIHT6JKV4CTSC3JY6NDKDVFZSO47OYNPZ7PGGA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.600434096 (6h0m17.973013451s) [noise=true , address=79.127.219.51:4211 , piece_id=HO5PLWBVU5YAYNYMN352NA73SM4WBJV4WZ7XLSQTCQ76UAL2TOJA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.609008984 (6h0m17.981588339s) [noise=true , address=104.36.82.19:4423 , piece_id=PIJUHDUND36HQL67I33D7A3EULMJHS7N55MN4OFQJ4OWNQ42O47A, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.620825273 (6h0m17.993404628s) [noise=true , address=79.127.219.38:4023 , piece_id=PTPOZGLYV4MZTOFZNN3NCGYL7DFAST4PXKEUIVTMFL7HV4GZTCLQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.620838659 (6h0m17.993418014s) [seg_pos={0 0}, upload=4769]: Segment reached optimal threshold of 14 pieces. Jul 19 04:51:11.621067807 (6h0m17.993647163s) [noise=true , address=77.90.31.11:4564 , piece_id=L2U7JFJRDQOFA2AYVAGTI4ZW4VABCXKMXWBIUWQO2QNNUUBJ74BA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection; write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection Jul 19 04:51:11.621134234 (6h0m17.993713589s) [noise=true , address=77.90.31.10:4546 , piece_id=R6LCA5WZOEBDURL5S3MEK22YPVYXX6EXSS4J4W57543HELYXOKZA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK): protocol: read tcp 188.166.225.170:58476->77.90.31.10:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621174191 (6h0m17.993753546s) [noise=true , address=77.90.31.16:4768 , piece_id=YNLSONFM6COZEVF6ZNQR2EA6V4LSQFZJONIXR4AFTNNMFGPV57JA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UchewmtxBrtSZSpVAJFdZmcTsQVQxkjK9re6cgUGqDhyonQeF): protocol: read tcp 188.166.225.170:56326->77.90.31.16:4768: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621216842 (6h0m17.993796197s) [noise=true , address=77.90.31.18:4813 , piece_id=VDOT7BB4O7XYJHW6BFTAYQZZYLZU5PAUXVJGSXJ7FDI4TH425COA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128WthA1x9wp1dV1vkpGnoveZQpU9CSgDCZfw5GMQuVpCNSVAM5): protocol: read tcp 188.166.225.170:49704->77.90.31.18:4813: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621234917 (6h0m17.993814272s) [seg_pos={0 0}, upload=4769]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 19 04:51:11.621301444 (6h0m17.993880799s) [upload=4769]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 19 04:51:11.621305402 (6h0m17.993884758s) [upload=4769]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 19 04:51:11.923136970 (6h0m18.295716325s) [upload=4769]: Done uploading segments... Jul 19 04:51:11.923156627 (6h0m18.295735972s) [upload=4769]: Upload finished. err: <nil> Duration: 6.582150151s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [..........XX...X...X....X..X....XXXXX.X.X.....XX.X..........] duration: 8.383972712s FAIL (2025-07-19 05:23:00 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== gment. successful: 13 optimal: 13 errs: <nil> Jul 19 05:14:35.936262577 (6h23m42.308841932s) [upload=5075]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 19 05:14:35.936281884 (6h23m42.308861239s) [upload=5075]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 19 05:14:36.234171969 (6h23m42.606751314s) [upload=5075]: Done uploading segments... Jul 19 05:14:36.234192037 (6h23m42.606771382s) [upload=5075]: Upload finished. err: <nil> Jul 19 05:14:55.587986728 (6h24m1.960566093s) [upload=5081]: Starting upload Jul 19 05:14:55.588051823 (6h24m1.960631178s) [upload=5081]: Done starting upload Jul 19 05:14:55.588146724 (6h24m1.960726079s) [upload=5081]: Uploading segments... Jul 19 05:14:55.588391873 (6h24m1.960971228s) [upload=5081]: Got next segment. Inline: false Jul 19 05:14:55.588649837 (6h24m1.961229193s) [upload=5081]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 05:14:55.588655057 (6h24m1.961234402s) [upload=5081]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 05:14:55.823387692 (6h24m2.195967047s) [seg_pos={0 0}, upload=5081]: Begin upload segment... Jul 19 05:14:55.823450572 (6h24m2.196029927s) [seg_pos={0 0}, upload=5081]: Done begin upload segment. Jul 19 05:14:55.823481952 (6h24m2.196061307s) [noise=true , address=77.90.31.17:4772 , piece_id=B544E3F2UC3UVE2ITUQAWGVDMSPS2HFIVHXZ2OLWB3T25AHMFAVA, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.823491401 (6h24m2.196070756s) [noise=true , address=45.91.116.10:4082 , piece_id=HZDMC4J5O5UHVDH5SOQBKAFPN6RJUHIWKEZ4OPVMW2HSN7NDZMZA, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.823923789 (6h24m2.196503144s) [noise=true , address=45.91.116.9:4262 , piece_id=FRIATDM7LPPHXYAMUISAYRTQWAKSTB4EYUANHGEMQHM6JQADDTRQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.823949737 (6h24m2.196529092s) [noise=true , address=45.91.116.12:4227 , piece_id=QB7YKZFXWMRB3VQ3HSFMMMLLX5YFWOXG4Y4GJGGYT6HDV4FQ65VA, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.824310158 (6h24m2.196889513s) [noise=true , address=45.91.116.4:4937 , piece_id=V2IOOA7DVCGZBH3D43ROYHT6ZECPY6GTC5M7NFO2RWPQB6EYGALA, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.824403045 (6h24m2.19698239s) [noise=true , address=77.90.31.11:4555 , piece_id=YICA5IVHSCC3RPC5FRKD26BSMY6N5WJQA5LCABJSRJYL6BZJYL6A, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.824651210 (6h24m2.197230565s) [noise=true , address=77.90.31.13:4627 , piece_id=E7GSSBZ653K2AA4TP6A4BRT5BDZCBODNRAZM3P6SYJXJFQU2BACQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.824716334 (6h24m2.197295689s) [noise=true , address=77.90.31.16:4734 , piece_id=YUW6MWSUDNLAC2VHLBT6H4T27YTNJ3RDK4BSGE7OZQCO3F2LBH4A, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.824960491 (6h24m2.197539856s) [noise=true , address=45.91.116.7:4010 , piece_id=BH76RM4NXGEAI2J2424ALK4PFSXUTBCYOH777BUOIDW7BW2IJUJA, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.824977494 (6h24m2.197556849s) [noise=true , address=45.91.116.14:4190 , piece_id=RUU2MICSAWOECAAYNOBS75SE2YKSUBMSSFKLPON7FWK77UNJEYGQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.825213285 (6h24m2.19779264s) [noise=true , address=45.91.116.8:4047 , piece_id=6UOFOV4VCVT6DB7D7Q23REAX7VZPAW5PSG5PLA54LISOUWMPQBBQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.825271547 (6h24m2.197850902s) [noise=true , address=77.90.31.18:4806 , piece_id=67PXFJMJB6XWNU6JMOKPO4BXKDD2Z225ZKTB7DZEA46OMSSXHHMQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.825445860 (6h24m2.198025215s) [noise=true , address=45.91.116.7:4009 , piece_id=HH5I32ZYT5LVXHDLBVSAVX6T23MJ6RGNPNAWPIQAREJ4ELPFJGYA, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.825568244 (6h24m2.198147599s) [noise=true , address=77.90.31.15:4697 , piece_id=W62ARDAVLI64P6VYSPVFRTDKB5H2WXTSCF2VKP3UN7JKGBCUEZTQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.825902985 (6h24m2.19848234s) [noise=true , address=45.91.116.5:4299 , piece_id=JQLK5LIMCMW6R2UZEJZSG6QAP2M65XBVE6JFEXY5KDWBVNDG2VCQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.826216495 (6h24m2.19879585s) [noise=true , address=77.90.31.14:4663 , piece_id=LNTFUWB6LUUFD2EB536ZHBSNVVXYZPOOB4DUZB2M6BXSSFEL3UMQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:58.633690556 (6h24m5.006269911s) [upload=5081]: Next returned nil segment Jul 19 05:14:58.633708100 (6h24m5.006287445s) [upload=5081]: Waiting for error group managing segments... Jul 19 05:14:58.915966948 (6h24m5.288546293s) [noise=true , address=45.91.116.7:4009 , piece_id=HH5I32ZYT5LVXHDLBVSAVX6T23MJ6RGNPNAWPIQAREJ4ELPFJGYA, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:58.916093541 (6h24m5.288672896s) [noise=true , address=45.91.116.12:4227 , piece_id=QB7YKZFXWMRB3VQ3HSFMMMLLX5YFWOXG4Y4GJGGYT6HDV4FQ65VA, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:58.923615462 (6h24m5.296194817s) [noise=true , address=45.91.116.14:4190 , piece_id=RUU2MICSAWOECAAYNOBS75SE2YKSUBMSSFKLPON7FWK77UNJEYGQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:58.928612934 (6h24m5.301192289s) [noise=true , address=45.91.116.7:4010 , piece_id=BH76RM4NXGEAI2J2424ALK4PFSXUTBCYOH777BUOIDW7BW2IJUJA, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:58.990915468 (6h24m5.363494833s) [noise=true , address=45.91.116.4:4937 , piece_id=V2IOOA7DVCGZBH3D43ROYHT6ZECPY6GTC5M7NFO2RWPQB6EYGALA, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:58.992136464 (6h24m5.364715819s) [noise=true , address=45.91.116.8:4047 , piece_id=6UOFOV4VCVT6DB7D7Q23REAX7VZPAW5PSG5PLA54LISOUWMPQBBQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:58.997711812 (6h24m5.370291157s) [noise=true , address=45.91.116.9:4262 , piece_id=FRIATDM7LPPHXYAMUISAYRTQWAKSTB4EYUANHGEMQHM6JQADDTRQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:59.010548391 (6h24m5.383127746s) [noise=true , address=45.91.116.10:4082 , piece_id=HZDMC4J5O5UHVDH5SOQBKAFPN6RJUHIWKEZ4OPVMW2HSN7NDZMZA, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:59.039607621 (6h24m5.412186996s) [noise=true , address=45.91.116.5:4299 , piece_id=JQLK5LIMCMW6R2UZEJZSG6QAP2M65XBVE6JFEXY5KDWBVNDG2VCQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:15:19.998482215 (6h24m26.37106157s) [noise=true , address=77.90.31.14:4663 , piece_id=LNTFUWB6LUUFD2EB536ZHBSNVVXYZPOOB4DUZB2M6BXSSFEL3UMQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:15:21.170089451 (6h24m27.542668806s) [noise=true , address=77.90.31.13:4627 , piece_id=E7GSSBZ653K2AA4TP6A4BRT5BDZCBODNRAZM3P6SYJXJFQU2BACQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:15:24.941469148 (6h24m31.314048503s) [noise=true , address=77.90.31.15:4697 , piece_id=W62ARDAVLI64P6VYSPVFRTDKB5H2WXTSCF2VKP3UN7JKGBCUEZTQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: expected piece hash; context deadline exceeded; EOF Jul 19 05:15:24.941530766 (6h24m31.314110121s) [noise=true , address=77.90.31.17:4772 , piece_id=B544E3F2UC3UVE2ITUQAWGVDMSPS2HFIVHXZ2OLWB3T25AHMFAVA, seg_pos={0 0}, upload=5081]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:44456->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:44456->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44456->77.90.31.17:4772: use of closed network connection Jul 19 05:15:24.941543591 (6h24m31.314122946s) [noise=true , address=77.90.31.16:4734 , piece_id=YUW6MWSUDNLAC2VHLBT6H4T27YTNJ3RDK4BSGE7OZQCO3F2LBH4A, seg_pos={0 0}, upload=5081]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Jul 19 05:15:24.941614786 (6h24m31.314194141s) [noise=true , address=77.90.31.11:4555 , piece_id=YICA5IVHSCC3RPC5FRKD26BSMY6N5WJQA5LCABJSRJYL6BZJYL6A, seg_pos={0 0}, upload=5081]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: write tcp 188.166.225.170:37392->77.90.31.11:4555: use of closed network connection; write tcp 188.166.225.170:37392->77.90.31.11:4555: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37392->77.90.31.11:4555: use of closed network connection Jul 19 05:15:24.941641888 (6h24m31.314221243s) [noise=true , address=77.90.31.18:4806 , piece_id=67PXFJMJB6XWNU6JMOKPO4BXKDD2Z225ZKTB7DZEA46OMSSXHHMQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:47114->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:47114->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47114->77.90.31.18:4806: use of closed network connection Jul 19 05:15:24.941663099 (6h24m31.314242454s) [seg_pos={0 0}, upload=5081]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 19 05:15:25.179949476 (6h24m31.552528831s) [upload=5081]: Done uploading segments... Jul 19 05:15:25.179975797 (6h24m31.552555152s) [upload=5081]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 8.383991227s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [...........XX...X...X....X..X....XXXXX.X.X.....XX.X.........] duration: 26.001033642s FAIL (2025-07-19 05:22:51 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== gment. successful: 13 optimal: 13 errs: <nil> Jul 19 05:14:35.936262577 (6h23m42.308841932s) [upload=5075]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 19 05:14:35.936281884 (6h23m42.308861239s) [upload=5075]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 19 05:14:36.234171969 (6h23m42.606751314s) [upload=5075]: Done uploading segments... Jul 19 05:14:36.234192037 (6h23m42.606771382s) [upload=5075]: Upload finished. err: <nil> Jul 19 05:14:55.587986728 (6h24m1.960566093s) [upload=5081]: Starting upload Jul 19 05:14:55.588051823 (6h24m1.960631178s) [upload=5081]: Done starting upload Jul 19 05:14:55.588146724 (6h24m1.960726079s) [upload=5081]: Uploading segments... Jul 19 05:14:55.588391873 (6h24m1.960971228s) [upload=5081]: Got next segment. Inline: false Jul 19 05:14:55.588649837 (6h24m1.961229193s) [upload=5081]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 05:14:55.588655057 (6h24m1.961234402s) [upload=5081]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 05:14:55.823387692 (6h24m2.195967047s) [seg_pos={0 0}, upload=5081]: Begin upload segment... Jul 19 05:14:55.823450572 (6h24m2.196029927s) [seg_pos={0 0}, upload=5081]: Done begin upload segment. Jul 19 05:14:55.823481952 (6h24m2.196061307s) [noise=true , address=77.90.31.17:4772 , piece_id=B544E3F2UC3UVE2ITUQAWGVDMSPS2HFIVHXZ2OLWB3T25AHMFAVA, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.823491401 (6h24m2.196070756s) [noise=true , address=45.91.116.10:4082 , piece_id=HZDMC4J5O5UHVDH5SOQBKAFPN6RJUHIWKEZ4OPVMW2HSN7NDZMZA, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.823923789 (6h24m2.196503144s) [noise=true , address=45.91.116.9:4262 , piece_id=FRIATDM7LPPHXYAMUISAYRTQWAKSTB4EYUANHGEMQHM6JQADDTRQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.823949737 (6h24m2.196529092s) [noise=true , address=45.91.116.12:4227 , piece_id=QB7YKZFXWMRB3VQ3HSFMMMLLX5YFWOXG4Y4GJGGYT6HDV4FQ65VA, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.824310158 (6h24m2.196889513s) [noise=true , address=45.91.116.4:4937 , piece_id=V2IOOA7DVCGZBH3D43ROYHT6ZECPY6GTC5M7NFO2RWPQB6EYGALA, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.824403045 (6h24m2.19698239s) [noise=true , address=77.90.31.11:4555 , piece_id=YICA5IVHSCC3RPC5FRKD26BSMY6N5WJQA5LCABJSRJYL6BZJYL6A, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.824651210 (6h24m2.197230565s) [noise=true , address=77.90.31.13:4627 , piece_id=E7GSSBZ653K2AA4TP6A4BRT5BDZCBODNRAZM3P6SYJXJFQU2BACQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.824716334 (6h24m2.197295689s) [noise=true , address=77.90.31.16:4734 , piece_id=YUW6MWSUDNLAC2VHLBT6H4T27YTNJ3RDK4BSGE7OZQCO3F2LBH4A, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.824960491 (6h24m2.197539856s) [noise=true , address=45.91.116.7:4010 , piece_id=BH76RM4NXGEAI2J2424ALK4PFSXUTBCYOH777BUOIDW7BW2IJUJA, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.824977494 (6h24m2.197556849s) [noise=true , address=45.91.116.14:4190 , piece_id=RUU2MICSAWOECAAYNOBS75SE2YKSUBMSSFKLPON7FWK77UNJEYGQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.825213285 (6h24m2.19779264s) [noise=true , address=45.91.116.8:4047 , piece_id=6UOFOV4VCVT6DB7D7Q23REAX7VZPAW5PSG5PLA54LISOUWMPQBBQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.825271547 (6h24m2.197850902s) [noise=true , address=77.90.31.18:4806 , piece_id=67PXFJMJB6XWNU6JMOKPO4BXKDD2Z225ZKTB7DZEA46OMSSXHHMQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.825445860 (6h24m2.198025215s) [noise=true , address=45.91.116.7:4009 , piece_id=HH5I32ZYT5LVXHDLBVSAVX6T23MJ6RGNPNAWPIQAREJ4ELPFJGYA, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.825568244 (6h24m2.198147599s) [noise=true , address=77.90.31.15:4697 , piece_id=W62ARDAVLI64P6VYSPVFRTDKB5H2WXTSCF2VKP3UN7JKGBCUEZTQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.825902985 (6h24m2.19848234s) [noise=true , address=45.91.116.5:4299 , piece_id=JQLK5LIMCMW6R2UZEJZSG6QAP2M65XBVE6JFEXY5KDWBVNDG2VCQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.826216495 (6h24m2.19879585s) [noise=true , address=77.90.31.14:4663 , piece_id=LNTFUWB6LUUFD2EB536ZHBSNVVXYZPOOB4DUZB2M6BXSSFEL3UMQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:58.633690556 (6h24m5.006269911s) [upload=5081]: Next returned nil segment Jul 19 05:14:58.633708100 (6h24m5.006287445s) [upload=5081]: Waiting for error group managing segments... Jul 19 05:14:58.915966948 (6h24m5.288546293s) [noise=true , address=45.91.116.7:4009 , piece_id=HH5I32ZYT5LVXHDLBVSAVX6T23MJ6RGNPNAWPIQAREJ4ELPFJGYA, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:58.916093541 (6h24m5.288672896s) [noise=true , address=45.91.116.12:4227 , piece_id=QB7YKZFXWMRB3VQ3HSFMMMLLX5YFWOXG4Y4GJGGYT6HDV4FQ65VA, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:58.923615462 (6h24m5.296194817s) [noise=true , address=45.91.116.14:4190 , piece_id=RUU2MICSAWOECAAYNOBS75SE2YKSUBMSSFKLPON7FWK77UNJEYGQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:58.928612934 (6h24m5.301192289s) [noise=true , address=45.91.116.7:4010 , piece_id=BH76RM4NXGEAI2J2424ALK4PFSXUTBCYOH777BUOIDW7BW2IJUJA, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:58.990915468 (6h24m5.363494833s) [noise=true , address=45.91.116.4:4937 , piece_id=V2IOOA7DVCGZBH3D43ROYHT6ZECPY6GTC5M7NFO2RWPQB6EYGALA, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:58.992136464 (6h24m5.364715819s) [noise=true , address=45.91.116.8:4047 , piece_id=6UOFOV4VCVT6DB7D7Q23REAX7VZPAW5PSG5PLA54LISOUWMPQBBQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:58.997711812 (6h24m5.370291157s) [noise=true , address=45.91.116.9:4262 , piece_id=FRIATDM7LPPHXYAMUISAYRTQWAKSTB4EYUANHGEMQHM6JQADDTRQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:59.010548391 (6h24m5.383127746s) [noise=true , address=45.91.116.10:4082 , piece_id=HZDMC4J5O5UHVDH5SOQBKAFPN6RJUHIWKEZ4OPVMW2HSN7NDZMZA, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:59.039607621 (6h24m5.412186996s) [noise=true , address=45.91.116.5:4299 , piece_id=JQLK5LIMCMW6R2UZEJZSG6QAP2M65XBVE6JFEXY5KDWBVNDG2VCQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:15:19.998482215 (6h24m26.37106157s) [noise=true , address=77.90.31.14:4663 , piece_id=LNTFUWB6LUUFD2EB536ZHBSNVVXYZPOOB4DUZB2M6BXSSFEL3UMQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:15:21.170089451 (6h24m27.542668806s) [noise=true , address=77.90.31.13:4627 , piece_id=E7GSSBZ653K2AA4TP6A4BRT5BDZCBODNRAZM3P6SYJXJFQU2BACQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:15:24.941469148 (6h24m31.314048503s) [noise=true , address=77.90.31.15:4697 , piece_id=W62ARDAVLI64P6VYSPVFRTDKB5H2WXTSCF2VKP3UN7JKGBCUEZTQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: expected piece hash; context deadline exceeded; EOF Jul 19 05:15:24.941530766 (6h24m31.314110121s) [noise=true , address=77.90.31.17:4772 , piece_id=B544E3F2UC3UVE2ITUQAWGVDMSPS2HFIVHXZ2OLWB3T25AHMFAVA, seg_pos={0 0}, upload=5081]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:44456->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:44456->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44456->77.90.31.17:4772: use of closed network connection Jul 19 05:15:24.941543591 (6h24m31.314122946s) [noise=true , address=77.90.31.16:4734 , piece_id=YUW6MWSUDNLAC2VHLBT6H4T27YTNJ3RDK4BSGE7OZQCO3F2LBH4A, seg_pos={0 0}, upload=5081]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Jul 19 05:15:24.941614786 (6h24m31.314194141s) [noise=true , address=77.90.31.11:4555 , piece_id=YICA5IVHSCC3RPC5FRKD26BSMY6N5WJQA5LCABJSRJYL6BZJYL6A, seg_pos={0 0}, upload=5081]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: write tcp 188.166.225.170:37392->77.90.31.11:4555: use of closed network connection; write tcp 188.166.225.170:37392->77.90.31.11:4555: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37392->77.90.31.11:4555: use of closed network connection Jul 19 05:15:24.941641888 (6h24m31.314221243s) [noise=true , address=77.90.31.18:4806 , piece_id=67PXFJMJB6XWNU6JMOKPO4BXKDD2Z225ZKTB7DZEA46OMSSXHHMQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:47114->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:47114->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47114->77.90.31.18:4806: use of closed network connection Jul 19 05:15:24.941663099 (6h24m31.314242454s) [seg_pos={0 0}, upload=5081]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 19 05:15:25.179949476 (6h24m31.552528831s) [upload=5081]: Done uploading segments... Jul 19 05:15:25.179975797 (6h24m31.552555152s) [upload=5081]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 26.001064049s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 12.191557058s FAIL (2025-07-19 05:22:27 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 4855s) [upload=4769]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 04:51:09.109709627 (6h0m15.482288982s) [upload=4769]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 04:51:09.556145753 (6h0m15.928725108s) [seg_pos={0 0}, upload=4769]: Begin upload segment... Jul 19 04:51:09.556194817 (6h0m15.928774172s) [seg_pos={0 0}, upload=4769]: Done begin upload segment. Jul 19 04:51:09.556223893 (6h0m15.928803248s) [noise=true , address=79.127.219.38:4023 , piece_id=PTPOZGLYV4MZTOFZNN3NCGYL7DFAST4PXKEUIVTMFL7HV4GZTCLQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556252197 (6h0m15.928831552s) [noise=true , address=45.91.95.4:4011 , piece_id=ZMBAS6KKHYVIRE6OBL2NFVCWMM4TKQ7J64BD3SYXIK54XCEDI2XA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556579383 (6h0m15.929158738s) [noise=true , address=45.91.116.6:4007 , piece_id=A2AFVTZBXSGBOE7XIM3B5HRN4STITFBXO6MZRJ365URNAAOZ3PSQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556854469 (6h0m15.929433824s) [noise=true , address=104.36.82.19:4423 , piece_id=PIJUHDUND36HQL67I33D7A3EULMJHS7N55MN4OFQJ4OWNQ42O47A, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557094107 (6h0m15.929673462s) [noise=true , address=45.91.116.9:4264 , piece_id=GHM5BKZKMT2QOHHBOS4FSPOWEWSFQRGRVVR4GSNDPHVIFRZHQL2A, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557336381 (6h0m15.929915736s) [noise=true , address=79.127.205.235:4660 , piece_id=UOUOGE2VP7PX7EMRZLPMCWFCYEZERM3DQMCQ6NG4J2K2DEIZR63Q, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557557885 (6h0m15.93013724s) [noise=true , address=79.127.205.246:4280 , piece_id=FLBOZI3UGTQXAQBLZ2F45Z3OEYL65F5PQBXRF2DQMZ7NGF7VDRPA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557762296 (6h0m15.930341651s) [noise=true , address=79.127.205.233:4622 , piece_id=6WWNYQTH5MGAJOEB7BXKZDJOMCAZXDW66J3OEANGDNJDNUMLZM7Q, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558014689 (6h0m15.930594044s) [noise=true , address=64.49.13.138:4482 , piece_id=MAW2M4US2OQNOS3WI67AA7NIBTJDE5T2JQTXKFPBLGVGXKVCFGTQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558215974 (6h0m15.930795319s) [noise=true , address=77.90.31.10:4546 , piece_id=R6LCA5WZOEBDURL5S3MEK22YPVYXX6EXSS4J4W57543HELYXOKZA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558409364 (6h0m15.930988719s) [noise=true , address=45.91.95.5:4019 , piece_id=YTSBCN2LPHEMRRAZIHJGVB7NBOTRXVEHD2TWUCTTLBNFFYEWWDKA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558623504 (6h0m15.931202859s) [noise=true , address=79.127.201.215:4357 , piece_id=75XWAG2VQWEYCWN2Q3WRIZSBI7LDOISZJPXQLQ2GIPNEN34VLSUQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558805162 (6h0m15.931384517s) [noise=true , address=45.91.116.14:4219 , piece_id=PDW6JPRHCVMHHOWIHT6JKV4CTSC3JY6NDKDVFZSO47OYNPZ7PGGA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558999373 (6h0m15.931578728s) [noise=true , address=77.90.31.11:4564 , piece_id=L2U7JFJRDQOFA2AYVAGTI4ZW4VABCXKMXWBIUWQO2QNNUUBJ74BA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559184768 (6h0m15.931764123s) [noise=true , address=77.90.31.16:4768 , piece_id=YNLSONFM6COZEVF6ZNQR2EA6V4LSQFZJONIXR4AFTNNMFGPV57JA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559368719 (6h0m15.931948074s) [noise=true , address=79.127.220.98:4432 , piece_id=2SUTFQ7FVDPAM2H4W3OLW6AVH3VJ33IJFE4L52SA3ET5LGBMOMPQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559549706 (6h0m15.932129061s) [noise=true , address=77.90.31.18:4813 , piece_id=VDOT7BB4O7XYJHW6BFTAYQZZYLZU5PAUXVJGSXJ7FDI4TH425COA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559727836 (6h0m15.932307191s) [noise=true , address=79.127.219.51:4211 , piece_id=HO5PLWBVU5YAYNYMN352NA73SM4WBJV4WZ7XLSQTCQ76UAL2TOJA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:10.663359087 (6h0m17.035938442s) [upload=4769]: Next returned nil segment Jul 19 04:51:10.663378664 (6h0m17.035958019s) [upload=4769]: Waiting for error group managing segments... Jul 19 04:51:10.962945479 (6h0m17.335524834s) [noise=true , address=79.127.205.233:4622 , piece_id=6WWNYQTH5MGAJOEB7BXKZDJOMCAZXDW66J3OEANGDNJDNUMLZM7Q, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:10.967689807 (6h0m17.340269152s) [noise=true , address=79.127.205.235:4660 , piece_id=UOUOGE2VP7PX7EMRZLPMCWFCYEZERM3DQMCQ6NG4J2K2DEIZR63Q, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:10.976968631 (6h0m17.349547986s) [noise=true , address=64.49.13.138:4482 , piece_id=MAW2M4US2OQNOS3WI67AA7NIBTJDE5T2JQTXKFPBLGVGXKVCFGTQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.048750700 (6h0m17.421330055s) [noise=true , address=79.127.205.246:4280 , piece_id=FLBOZI3UGTQXAQBLZ2F45Z3OEYL65F5PQBXRF2DQMZ7NGF7VDRPA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.221642403 (6h0m17.594221758s) [noise=true , address=79.127.220.98:4432 , piece_id=2SUTFQ7FVDPAM2H4W3OLW6AVH3VJ33IJFE4L52SA3ET5LGBMOMPQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.317070435 (6h0m17.68964979s) [noise=true , address=45.91.95.4:4011 , piece_id=ZMBAS6KKHYVIRE6OBL2NFVCWMM4TKQ7J64BD3SYXIK54XCEDI2XA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.324002660 (6h0m17.696582015s) [noise=true , address=45.91.95.5:4019 , piece_id=YTSBCN2LPHEMRRAZIHJGVB7NBOTRXVEHD2TWUCTTLBNFFYEWWDKA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.460976317 (6h0m17.833555672s) [noise=true , address=79.127.201.215:4357 , piece_id=75XWAG2VQWEYCWN2Q3WRIZSBI7LDOISZJPXQLQ2GIPNEN34VLSUQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.462584603 (6h0m17.835163958s) [noise=true , address=45.91.116.6:4007 , piece_id=A2AFVTZBXSGBOE7XIM3B5HRN4STITFBXO6MZRJ365URNAAOZ3PSQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.490392973 (6h0m17.862972328s) [noise=true , address=45.91.116.9:4264 , piece_id=GHM5BKZKMT2QOHHBOS4FSPOWEWSFQRGRVVR4GSNDPHVIFRZHQL2A, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.502040780 (6h0m17.874620135s) [noise=true , address=45.91.116.14:4219 , piece_id=PDW6JPRHCVMHHOWIHT6JKV4CTSC3JY6NDKDVFZSO47OYNPZ7PGGA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.600434096 (6h0m17.973013451s) [noise=true , address=79.127.219.51:4211 , piece_id=HO5PLWBVU5YAYNYMN352NA73SM4WBJV4WZ7XLSQTCQ76UAL2TOJA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.609008984 (6h0m17.981588339s) [noise=true , address=104.36.82.19:4423 , piece_id=PIJUHDUND36HQL67I33D7A3EULMJHS7N55MN4OFQJ4OWNQ42O47A, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.620825273 (6h0m17.993404628s) [noise=true , address=79.127.219.38:4023 , piece_id=PTPOZGLYV4MZTOFZNN3NCGYL7DFAST4PXKEUIVTMFL7HV4GZTCLQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.620838659 (6h0m17.993418014s) [seg_pos={0 0}, upload=4769]: Segment reached optimal threshold of 14 pieces. Jul 19 04:51:11.621067807 (6h0m17.993647163s) [noise=true , address=77.90.31.11:4564 , piece_id=L2U7JFJRDQOFA2AYVAGTI4ZW4VABCXKMXWBIUWQO2QNNUUBJ74BA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection; write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection Jul 19 04:51:11.621134234 (6h0m17.993713589s) [noise=true , address=77.90.31.10:4546 , piece_id=R6LCA5WZOEBDURL5S3MEK22YPVYXX6EXSS4J4W57543HELYXOKZA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK): protocol: read tcp 188.166.225.170:58476->77.90.31.10:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621174191 (6h0m17.993753546s) [noise=true , address=77.90.31.16:4768 , piece_id=YNLSONFM6COZEVF6ZNQR2EA6V4LSQFZJONIXR4AFTNNMFGPV57JA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UchewmtxBrtSZSpVAJFdZmcTsQVQxkjK9re6cgUGqDhyonQeF): protocol: read tcp 188.166.225.170:56326->77.90.31.16:4768: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621216842 (6h0m17.993796197s) [noise=true , address=77.90.31.18:4813 , piece_id=VDOT7BB4O7XYJHW6BFTAYQZZYLZU5PAUXVJGSXJ7FDI4TH425COA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128WthA1x9wp1dV1vkpGnoveZQpU9CSgDCZfw5GMQuVpCNSVAM5): protocol: read tcp 188.166.225.170:49704->77.90.31.18:4813: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621234917 (6h0m17.993814272s) [seg_pos={0 0}, upload=4769]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 19 04:51:11.621301444 (6h0m17.993880799s) [upload=4769]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 19 04:51:11.621305402 (6h0m17.993884758s) [upload=4769]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 19 04:51:11.923136970 (6h0m18.295716325s) [upload=4769]: Done uploading segments... Jul 19 04:51:11.923156627 (6h0m18.295735972s) [upload=4769]: Upload finished. err: <nil> Duration: 12.191575272s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 7.429823957s FAIL (2025-07-19 05:21:55 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 4855s) [upload=4769]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 04:51:09.109709627 (6h0m15.482288982s) [upload=4769]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 04:51:09.556145753 (6h0m15.928725108s) [seg_pos={0 0}, upload=4769]: Begin upload segment... Jul 19 04:51:09.556194817 (6h0m15.928774172s) [seg_pos={0 0}, upload=4769]: Done begin upload segment. Jul 19 04:51:09.556223893 (6h0m15.928803248s) [noise=true , address=79.127.219.38:4023 , piece_id=PTPOZGLYV4MZTOFZNN3NCGYL7DFAST4PXKEUIVTMFL7HV4GZTCLQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556252197 (6h0m15.928831552s) [noise=true , address=45.91.95.4:4011 , piece_id=ZMBAS6KKHYVIRE6OBL2NFVCWMM4TKQ7J64BD3SYXIK54XCEDI2XA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556579383 (6h0m15.929158738s) [noise=true , address=45.91.116.6:4007 , piece_id=A2AFVTZBXSGBOE7XIM3B5HRN4STITFBXO6MZRJ365URNAAOZ3PSQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556854469 (6h0m15.929433824s) [noise=true , address=104.36.82.19:4423 , piece_id=PIJUHDUND36HQL67I33D7A3EULMJHS7N55MN4OFQJ4OWNQ42O47A, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557094107 (6h0m15.929673462s) [noise=true , address=45.91.116.9:4264 , piece_id=GHM5BKZKMT2QOHHBOS4FSPOWEWSFQRGRVVR4GSNDPHVIFRZHQL2A, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557336381 (6h0m15.929915736s) [noise=true , address=79.127.205.235:4660 , piece_id=UOUOGE2VP7PX7EMRZLPMCWFCYEZERM3DQMCQ6NG4J2K2DEIZR63Q, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557557885 (6h0m15.93013724s) [noise=true , address=79.127.205.246:4280 , piece_id=FLBOZI3UGTQXAQBLZ2F45Z3OEYL65F5PQBXRF2DQMZ7NGF7VDRPA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557762296 (6h0m15.930341651s) [noise=true , address=79.127.205.233:4622 , piece_id=6WWNYQTH5MGAJOEB7BXKZDJOMCAZXDW66J3OEANGDNJDNUMLZM7Q, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558014689 (6h0m15.930594044s) [noise=true , address=64.49.13.138:4482 , piece_id=MAW2M4US2OQNOS3WI67AA7NIBTJDE5T2JQTXKFPBLGVGXKVCFGTQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558215974 (6h0m15.930795319s) [noise=true , address=77.90.31.10:4546 , piece_id=R6LCA5WZOEBDURL5S3MEK22YPVYXX6EXSS4J4W57543HELYXOKZA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558409364 (6h0m15.930988719s) [noise=true , address=45.91.95.5:4019 , piece_id=YTSBCN2LPHEMRRAZIHJGVB7NBOTRXVEHD2TWUCTTLBNFFYEWWDKA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558623504 (6h0m15.931202859s) [noise=true , address=79.127.201.215:4357 , piece_id=75XWAG2VQWEYCWN2Q3WRIZSBI7LDOISZJPXQLQ2GIPNEN34VLSUQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558805162 (6h0m15.931384517s) [noise=true , address=45.91.116.14:4219 , piece_id=PDW6JPRHCVMHHOWIHT6JKV4CTSC3JY6NDKDVFZSO47OYNPZ7PGGA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558999373 (6h0m15.931578728s) [noise=true , address=77.90.31.11:4564 , piece_id=L2U7JFJRDQOFA2AYVAGTI4ZW4VABCXKMXWBIUWQO2QNNUUBJ74BA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559184768 (6h0m15.931764123s) [noise=true , address=77.90.31.16:4768 , piece_id=YNLSONFM6COZEVF6ZNQR2EA6V4LSQFZJONIXR4AFTNNMFGPV57JA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559368719 (6h0m15.931948074s) [noise=true , address=79.127.220.98:4432 , piece_id=2SUTFQ7FVDPAM2H4W3OLW6AVH3VJ33IJFE4L52SA3ET5LGBMOMPQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559549706 (6h0m15.932129061s) [noise=true , address=77.90.31.18:4813 , piece_id=VDOT7BB4O7XYJHW6BFTAYQZZYLZU5PAUXVJGSXJ7FDI4TH425COA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559727836 (6h0m15.932307191s) [noise=true , address=79.127.219.51:4211 , piece_id=HO5PLWBVU5YAYNYMN352NA73SM4WBJV4WZ7XLSQTCQ76UAL2TOJA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:10.663359087 (6h0m17.035938442s) [upload=4769]: Next returned nil segment Jul 19 04:51:10.663378664 (6h0m17.035958019s) [upload=4769]: Waiting for error group managing segments... Jul 19 04:51:10.962945479 (6h0m17.335524834s) [noise=true , address=79.127.205.233:4622 , piece_id=6WWNYQTH5MGAJOEB7BXKZDJOMCAZXDW66J3OEANGDNJDNUMLZM7Q, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:10.967689807 (6h0m17.340269152s) [noise=true , address=79.127.205.235:4660 , piece_id=UOUOGE2VP7PX7EMRZLPMCWFCYEZERM3DQMCQ6NG4J2K2DEIZR63Q, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:10.976968631 (6h0m17.349547986s) [noise=true , address=64.49.13.138:4482 , piece_id=MAW2M4US2OQNOS3WI67AA7NIBTJDE5T2JQTXKFPBLGVGXKVCFGTQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.048750700 (6h0m17.421330055s) [noise=true , address=79.127.205.246:4280 , piece_id=FLBOZI3UGTQXAQBLZ2F45Z3OEYL65F5PQBXRF2DQMZ7NGF7VDRPA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.221642403 (6h0m17.594221758s) [noise=true , address=79.127.220.98:4432 , piece_id=2SUTFQ7FVDPAM2H4W3OLW6AVH3VJ33IJFE4L52SA3ET5LGBMOMPQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.317070435 (6h0m17.68964979s) [noise=true , address=45.91.95.4:4011 , piece_id=ZMBAS6KKHYVIRE6OBL2NFVCWMM4TKQ7J64BD3SYXIK54XCEDI2XA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.324002660 (6h0m17.696582015s) [noise=true , address=45.91.95.5:4019 , piece_id=YTSBCN2LPHEMRRAZIHJGVB7NBOTRXVEHD2TWUCTTLBNFFYEWWDKA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.460976317 (6h0m17.833555672s) [noise=true , address=79.127.201.215:4357 , piece_id=75XWAG2VQWEYCWN2Q3WRIZSBI7LDOISZJPXQLQ2GIPNEN34VLSUQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.462584603 (6h0m17.835163958s) [noise=true , address=45.91.116.6:4007 , piece_id=A2AFVTZBXSGBOE7XIM3B5HRN4STITFBXO6MZRJ365URNAAOZ3PSQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.490392973 (6h0m17.862972328s) [noise=true , address=45.91.116.9:4264 , piece_id=GHM5BKZKMT2QOHHBOS4FSPOWEWSFQRGRVVR4GSNDPHVIFRZHQL2A, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.502040780 (6h0m17.874620135s) [noise=true , address=45.91.116.14:4219 , piece_id=PDW6JPRHCVMHHOWIHT6JKV4CTSC3JY6NDKDVFZSO47OYNPZ7PGGA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.600434096 (6h0m17.973013451s) [noise=true , address=79.127.219.51:4211 , piece_id=HO5PLWBVU5YAYNYMN352NA73SM4WBJV4WZ7XLSQTCQ76UAL2TOJA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.609008984 (6h0m17.981588339s) [noise=true , address=104.36.82.19:4423 , piece_id=PIJUHDUND36HQL67I33D7A3EULMJHS7N55MN4OFQJ4OWNQ42O47A, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.620825273 (6h0m17.993404628s) [noise=true , address=79.127.219.38:4023 , piece_id=PTPOZGLYV4MZTOFZNN3NCGYL7DFAST4PXKEUIVTMFL7HV4GZTCLQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.620838659 (6h0m17.993418014s) [seg_pos={0 0}, upload=4769]: Segment reached optimal threshold of 14 pieces. Jul 19 04:51:11.621067807 (6h0m17.993647163s) [noise=true , address=77.90.31.11:4564 , piece_id=L2U7JFJRDQOFA2AYVAGTI4ZW4VABCXKMXWBIUWQO2QNNUUBJ74BA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection; write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection Jul 19 04:51:11.621134234 (6h0m17.993713589s) [noise=true , address=77.90.31.10:4546 , piece_id=R6LCA5WZOEBDURL5S3MEK22YPVYXX6EXSS4J4W57543HELYXOKZA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK): protocol: read tcp 188.166.225.170:58476->77.90.31.10:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621174191 (6h0m17.993753546s) [noise=true , address=77.90.31.16:4768 , piece_id=YNLSONFM6COZEVF6ZNQR2EA6V4LSQFZJONIXR4AFTNNMFGPV57JA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UchewmtxBrtSZSpVAJFdZmcTsQVQxkjK9re6cgUGqDhyonQeF): protocol: read tcp 188.166.225.170:56326->77.90.31.16:4768: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621216842 (6h0m17.993796197s) [noise=true , address=77.90.31.18:4813 , piece_id=VDOT7BB4O7XYJHW6BFTAYQZZYLZU5PAUXVJGSXJ7FDI4TH425COA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128WthA1x9wp1dV1vkpGnoveZQpU9CSgDCZfw5GMQuVpCNSVAM5): protocol: read tcp 188.166.225.170:49704->77.90.31.18:4813: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621234917 (6h0m17.993814272s) [seg_pos={0 0}, upload=4769]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 19 04:51:11.621301444 (6h0m17.993880799s) [upload=4769]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 19 04:51:11.621305402 (6h0m17.993884758s) [upload=4769]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 19 04:51:11.923136970 (6h0m18.295716325s) [upload=4769]: Done uploading segments... Jul 19 04:51:11.923156627 (6h0m18.295735972s) [upload=4769]: Upload finished. err: <nil> Duration: 7.429847582s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [............XX...X...X....X..X....XXXXX.X.X.....XX.X........] duration: 5.793107115s FAIL (2025-07-19 05:21:40 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== gment. successful: 13 optimal: 13 errs: <nil> Jul 19 05:14:35.936262577 (6h23m42.308841932s) [upload=5075]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 19 05:14:35.936281884 (6h23m42.308861239s) [upload=5075]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 19 05:14:36.234171969 (6h23m42.606751314s) [upload=5075]: Done uploading segments... Jul 19 05:14:36.234192037 (6h23m42.606771382s) [upload=5075]: Upload finished. err: <nil> Jul 19 05:14:55.587986728 (6h24m1.960566093s) [upload=5081]: Starting upload Jul 19 05:14:55.588051823 (6h24m1.960631178s) [upload=5081]: Done starting upload Jul 19 05:14:55.588146724 (6h24m1.960726079s) [upload=5081]: Uploading segments... Jul 19 05:14:55.588391873 (6h24m1.960971228s) [upload=5081]: Got next segment. Inline: false Jul 19 05:14:55.588649837 (6h24m1.961229193s) [upload=5081]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 05:14:55.588655057 (6h24m1.961234402s) [upload=5081]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 05:14:55.823387692 (6h24m2.195967047s) [seg_pos={0 0}, upload=5081]: Begin upload segment... Jul 19 05:14:55.823450572 (6h24m2.196029927s) [seg_pos={0 0}, upload=5081]: Done begin upload segment. Jul 19 05:14:55.823481952 (6h24m2.196061307s) [noise=true , address=77.90.31.17:4772 , piece_id=B544E3F2UC3UVE2ITUQAWGVDMSPS2HFIVHXZ2OLWB3T25AHMFAVA, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.823491401 (6h24m2.196070756s) [noise=true , address=45.91.116.10:4082 , piece_id=HZDMC4J5O5UHVDH5SOQBKAFPN6RJUHIWKEZ4OPVMW2HSN7NDZMZA, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.823923789 (6h24m2.196503144s) [noise=true , address=45.91.116.9:4262 , piece_id=FRIATDM7LPPHXYAMUISAYRTQWAKSTB4EYUANHGEMQHM6JQADDTRQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.823949737 (6h24m2.196529092s) [noise=true , address=45.91.116.12:4227 , piece_id=QB7YKZFXWMRB3VQ3HSFMMMLLX5YFWOXG4Y4GJGGYT6HDV4FQ65VA, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.824310158 (6h24m2.196889513s) [noise=true , address=45.91.116.4:4937 , piece_id=V2IOOA7DVCGZBH3D43ROYHT6ZECPY6GTC5M7NFO2RWPQB6EYGALA, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.824403045 (6h24m2.19698239s) [noise=true , address=77.90.31.11:4555 , piece_id=YICA5IVHSCC3RPC5FRKD26BSMY6N5WJQA5LCABJSRJYL6BZJYL6A, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.824651210 (6h24m2.197230565s) [noise=true , address=77.90.31.13:4627 , piece_id=E7GSSBZ653K2AA4TP6A4BRT5BDZCBODNRAZM3P6SYJXJFQU2BACQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.824716334 (6h24m2.197295689s) [noise=true , address=77.90.31.16:4734 , piece_id=YUW6MWSUDNLAC2VHLBT6H4T27YTNJ3RDK4BSGE7OZQCO3F2LBH4A, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.824960491 (6h24m2.197539856s) [noise=true , address=45.91.116.7:4010 , piece_id=BH76RM4NXGEAI2J2424ALK4PFSXUTBCYOH777BUOIDW7BW2IJUJA, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.824977494 (6h24m2.197556849s) [noise=true , address=45.91.116.14:4190 , piece_id=RUU2MICSAWOECAAYNOBS75SE2YKSUBMSSFKLPON7FWK77UNJEYGQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.825213285 (6h24m2.19779264s) [noise=true , address=45.91.116.8:4047 , piece_id=6UOFOV4VCVT6DB7D7Q23REAX7VZPAW5PSG5PLA54LISOUWMPQBBQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.825271547 (6h24m2.197850902s) [noise=true , address=77.90.31.18:4806 , piece_id=67PXFJMJB6XWNU6JMOKPO4BXKDD2Z225ZKTB7DZEA46OMSSXHHMQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.825445860 (6h24m2.198025215s) [noise=true , address=45.91.116.7:4009 , piece_id=HH5I32ZYT5LVXHDLBVSAVX6T23MJ6RGNPNAWPIQAREJ4ELPFJGYA, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.825568244 (6h24m2.198147599s) [noise=true , address=77.90.31.15:4697 , piece_id=W62ARDAVLI64P6VYSPVFRTDKB5H2WXTSCF2VKP3UN7JKGBCUEZTQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.825902985 (6h24m2.19848234s) [noise=true , address=45.91.116.5:4299 , piece_id=JQLK5LIMCMW6R2UZEJZSG6QAP2M65XBVE6JFEXY5KDWBVNDG2VCQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.826216495 (6h24m2.19879585s) [noise=true , address=77.90.31.14:4663 , piece_id=LNTFUWB6LUUFD2EB536ZHBSNVVXYZPOOB4DUZB2M6BXSSFEL3UMQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:58.633690556 (6h24m5.006269911s) [upload=5081]: Next returned nil segment Jul 19 05:14:58.633708100 (6h24m5.006287445s) [upload=5081]: Waiting for error group managing segments... Jul 19 05:14:58.915966948 (6h24m5.288546293s) [noise=true , address=45.91.116.7:4009 , piece_id=HH5I32ZYT5LVXHDLBVSAVX6T23MJ6RGNPNAWPIQAREJ4ELPFJGYA, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:58.916093541 (6h24m5.288672896s) [noise=true , address=45.91.116.12:4227 , piece_id=QB7YKZFXWMRB3VQ3HSFMMMLLX5YFWOXG4Y4GJGGYT6HDV4FQ65VA, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:58.923615462 (6h24m5.296194817s) [noise=true , address=45.91.116.14:4190 , piece_id=RUU2MICSAWOECAAYNOBS75SE2YKSUBMSSFKLPON7FWK77UNJEYGQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:58.928612934 (6h24m5.301192289s) [noise=true , address=45.91.116.7:4010 , piece_id=BH76RM4NXGEAI2J2424ALK4PFSXUTBCYOH777BUOIDW7BW2IJUJA, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:58.990915468 (6h24m5.363494833s) [noise=true , address=45.91.116.4:4937 , piece_id=V2IOOA7DVCGZBH3D43ROYHT6ZECPY6GTC5M7NFO2RWPQB6EYGALA, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:58.992136464 (6h24m5.364715819s) [noise=true , address=45.91.116.8:4047 , piece_id=6UOFOV4VCVT6DB7D7Q23REAX7VZPAW5PSG5PLA54LISOUWMPQBBQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:58.997711812 (6h24m5.370291157s) [noise=true , address=45.91.116.9:4262 , piece_id=FRIATDM7LPPHXYAMUISAYRTQWAKSTB4EYUANHGEMQHM6JQADDTRQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:59.010548391 (6h24m5.383127746s) [noise=true , address=45.91.116.10:4082 , piece_id=HZDMC4J5O5UHVDH5SOQBKAFPN6RJUHIWKEZ4OPVMW2HSN7NDZMZA, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:59.039607621 (6h24m5.412186996s) [noise=true , address=45.91.116.5:4299 , piece_id=JQLK5LIMCMW6R2UZEJZSG6QAP2M65XBVE6JFEXY5KDWBVNDG2VCQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:15:19.998482215 (6h24m26.37106157s) [noise=true , address=77.90.31.14:4663 , piece_id=LNTFUWB6LUUFD2EB536ZHBSNVVXYZPOOB4DUZB2M6BXSSFEL3UMQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:15:21.170089451 (6h24m27.542668806s) [noise=true , address=77.90.31.13:4627 , piece_id=E7GSSBZ653K2AA4TP6A4BRT5BDZCBODNRAZM3P6SYJXJFQU2BACQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:15:24.941469148 (6h24m31.314048503s) [noise=true , address=77.90.31.15:4697 , piece_id=W62ARDAVLI64P6VYSPVFRTDKB5H2WXTSCF2VKP3UN7JKGBCUEZTQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: expected piece hash; context deadline exceeded; EOF Jul 19 05:15:24.941530766 (6h24m31.314110121s) [noise=true , address=77.90.31.17:4772 , piece_id=B544E3F2UC3UVE2ITUQAWGVDMSPS2HFIVHXZ2OLWB3T25AHMFAVA, seg_pos={0 0}, upload=5081]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:44456->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:44456->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44456->77.90.31.17:4772: use of closed network connection Jul 19 05:15:24.941543591 (6h24m31.314122946s) [noise=true , address=77.90.31.16:4734 , piece_id=YUW6MWSUDNLAC2VHLBT6H4T27YTNJ3RDK4BSGE7OZQCO3F2LBH4A, seg_pos={0 0}, upload=5081]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Jul 19 05:15:24.941614786 (6h24m31.314194141s) [noise=true , address=77.90.31.11:4555 , piece_id=YICA5IVHSCC3RPC5FRKD26BSMY6N5WJQA5LCABJSRJYL6BZJYL6A, seg_pos={0 0}, upload=5081]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: write tcp 188.166.225.170:37392->77.90.31.11:4555: use of closed network connection; write tcp 188.166.225.170:37392->77.90.31.11:4555: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37392->77.90.31.11:4555: use of closed network connection Jul 19 05:15:24.941641888 (6h24m31.314221243s) [noise=true , address=77.90.31.18:4806 , piece_id=67PXFJMJB6XWNU6JMOKPO4BXKDD2Z225ZKTB7DZEA46OMSSXHHMQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:47114->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:47114->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47114->77.90.31.18:4806: use of closed network connection Jul 19 05:15:24.941663099 (6h24m31.314242454s) [seg_pos={0 0}, upload=5081]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 19 05:15:25.179949476 (6h24m31.552528831s) [upload=5081]: Done uploading segments... Jul 19 05:15:25.179975797 (6h24m31.552555152s) [upload=5081]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.793136761s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [.............XX...X...X....X..X....XXXXX.X.X.....XX.X.......] duration: 20.829346528s FAIL (2025-07-19 05:20:57 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== gment. successful: 13 optimal: 13 errs: <nil> Jul 19 05:14:35.936262577 (6h23m42.308841932s) [upload=5075]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 19 05:14:35.936281884 (6h23m42.308861239s) [upload=5075]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 19 05:14:36.234171969 (6h23m42.606751314s) [upload=5075]: Done uploading segments... Jul 19 05:14:36.234192037 (6h23m42.606771382s) [upload=5075]: Upload finished. err: <nil> Jul 19 05:14:55.587986728 (6h24m1.960566093s) [upload=5081]: Starting upload Jul 19 05:14:55.588051823 (6h24m1.960631178s) [upload=5081]: Done starting upload Jul 19 05:14:55.588146724 (6h24m1.960726079s) [upload=5081]: Uploading segments... Jul 19 05:14:55.588391873 (6h24m1.960971228s) [upload=5081]: Got next segment. Inline: false Jul 19 05:14:55.588649837 (6h24m1.961229193s) [upload=5081]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 05:14:55.588655057 (6h24m1.961234402s) [upload=5081]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 05:14:55.823387692 (6h24m2.195967047s) [seg_pos={0 0}, upload=5081]: Begin upload segment... Jul 19 05:14:55.823450572 (6h24m2.196029927s) [seg_pos={0 0}, upload=5081]: Done begin upload segment. Jul 19 05:14:55.823481952 (6h24m2.196061307s) [noise=true , address=77.90.31.17:4772 , piece_id=B544E3F2UC3UVE2ITUQAWGVDMSPS2HFIVHXZ2OLWB3T25AHMFAVA, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.823491401 (6h24m2.196070756s) [noise=true , address=45.91.116.10:4082 , piece_id=HZDMC4J5O5UHVDH5SOQBKAFPN6RJUHIWKEZ4OPVMW2HSN7NDZMZA, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.823923789 (6h24m2.196503144s) [noise=true , address=45.91.116.9:4262 , piece_id=FRIATDM7LPPHXYAMUISAYRTQWAKSTB4EYUANHGEMQHM6JQADDTRQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.823949737 (6h24m2.196529092s) [noise=true , address=45.91.116.12:4227 , piece_id=QB7YKZFXWMRB3VQ3HSFMMMLLX5YFWOXG4Y4GJGGYT6HDV4FQ65VA, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.824310158 (6h24m2.196889513s) [noise=true , address=45.91.116.4:4937 , piece_id=V2IOOA7DVCGZBH3D43ROYHT6ZECPY6GTC5M7NFO2RWPQB6EYGALA, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.824403045 (6h24m2.19698239s) [noise=true , address=77.90.31.11:4555 , piece_id=YICA5IVHSCC3RPC5FRKD26BSMY6N5WJQA5LCABJSRJYL6BZJYL6A, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.824651210 (6h24m2.197230565s) [noise=true , address=77.90.31.13:4627 , piece_id=E7GSSBZ653K2AA4TP6A4BRT5BDZCBODNRAZM3P6SYJXJFQU2BACQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.824716334 (6h24m2.197295689s) [noise=true , address=77.90.31.16:4734 , piece_id=YUW6MWSUDNLAC2VHLBT6H4T27YTNJ3RDK4BSGE7OZQCO3F2LBH4A, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.824960491 (6h24m2.197539856s) [noise=true , address=45.91.116.7:4010 , piece_id=BH76RM4NXGEAI2J2424ALK4PFSXUTBCYOH777BUOIDW7BW2IJUJA, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.824977494 (6h24m2.197556849s) [noise=true , address=45.91.116.14:4190 , piece_id=RUU2MICSAWOECAAYNOBS75SE2YKSUBMSSFKLPON7FWK77UNJEYGQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.825213285 (6h24m2.19779264s) [noise=true , address=45.91.116.8:4047 , piece_id=6UOFOV4VCVT6DB7D7Q23REAX7VZPAW5PSG5PLA54LISOUWMPQBBQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.825271547 (6h24m2.197850902s) [noise=true , address=77.90.31.18:4806 , piece_id=67PXFJMJB6XWNU6JMOKPO4BXKDD2Z225ZKTB7DZEA46OMSSXHHMQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.825445860 (6h24m2.198025215s) [noise=true , address=45.91.116.7:4009 , piece_id=HH5I32ZYT5LVXHDLBVSAVX6T23MJ6RGNPNAWPIQAREJ4ELPFJGYA, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.825568244 (6h24m2.198147599s) [noise=true , address=77.90.31.15:4697 , piece_id=W62ARDAVLI64P6VYSPVFRTDKB5H2WXTSCF2VKP3UN7JKGBCUEZTQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.825902985 (6h24m2.19848234s) [noise=true , address=45.91.116.5:4299 , piece_id=JQLK5LIMCMW6R2UZEJZSG6QAP2M65XBVE6JFEXY5KDWBVNDG2VCQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.826216495 (6h24m2.19879585s) [noise=true , address=77.90.31.14:4663 , piece_id=LNTFUWB6LUUFD2EB536ZHBSNVVXYZPOOB4DUZB2M6BXSSFEL3UMQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:58.633690556 (6h24m5.006269911s) [upload=5081]: Next returned nil segment Jul 19 05:14:58.633708100 (6h24m5.006287445s) [upload=5081]: Waiting for error group managing segments... Jul 19 05:14:58.915966948 (6h24m5.288546293s) [noise=true , address=45.91.116.7:4009 , piece_id=HH5I32ZYT5LVXHDLBVSAVX6T23MJ6RGNPNAWPIQAREJ4ELPFJGYA, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:58.916093541 (6h24m5.288672896s) [noise=true , address=45.91.116.12:4227 , piece_id=QB7YKZFXWMRB3VQ3HSFMMMLLX5YFWOXG4Y4GJGGYT6HDV4FQ65VA, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:58.923615462 (6h24m5.296194817s) [noise=true , address=45.91.116.14:4190 , piece_id=RUU2MICSAWOECAAYNOBS75SE2YKSUBMSSFKLPON7FWK77UNJEYGQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:58.928612934 (6h24m5.301192289s) [noise=true , address=45.91.116.7:4010 , piece_id=BH76RM4NXGEAI2J2424ALK4PFSXUTBCYOH777BUOIDW7BW2IJUJA, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:58.990915468 (6h24m5.363494833s) [noise=true , address=45.91.116.4:4937 , piece_id=V2IOOA7DVCGZBH3D43ROYHT6ZECPY6GTC5M7NFO2RWPQB6EYGALA, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:58.992136464 (6h24m5.364715819s) [noise=true , address=45.91.116.8:4047 , piece_id=6UOFOV4VCVT6DB7D7Q23REAX7VZPAW5PSG5PLA54LISOUWMPQBBQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:58.997711812 (6h24m5.370291157s) [noise=true , address=45.91.116.9:4262 , piece_id=FRIATDM7LPPHXYAMUISAYRTQWAKSTB4EYUANHGEMQHM6JQADDTRQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:59.010548391 (6h24m5.383127746s) [noise=true , address=45.91.116.10:4082 , piece_id=HZDMC4J5O5UHVDH5SOQBKAFPN6RJUHIWKEZ4OPVMW2HSN7NDZMZA, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:59.039607621 (6h24m5.412186996s) [noise=true , address=45.91.116.5:4299 , piece_id=JQLK5LIMCMW6R2UZEJZSG6QAP2M65XBVE6JFEXY5KDWBVNDG2VCQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:15:19.998482215 (6h24m26.37106157s) [noise=true , address=77.90.31.14:4663 , piece_id=LNTFUWB6LUUFD2EB536ZHBSNVVXYZPOOB4DUZB2M6BXSSFEL3UMQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:15:21.170089451 (6h24m27.542668806s) [noise=true , address=77.90.31.13:4627 , piece_id=E7GSSBZ653K2AA4TP6A4BRT5BDZCBODNRAZM3P6SYJXJFQU2BACQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:15:24.941469148 (6h24m31.314048503s) [noise=true , address=77.90.31.15:4697 , piece_id=W62ARDAVLI64P6VYSPVFRTDKB5H2WXTSCF2VKP3UN7JKGBCUEZTQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: expected piece hash; context deadline exceeded; EOF Jul 19 05:15:24.941530766 (6h24m31.314110121s) [noise=true , address=77.90.31.17:4772 , piece_id=B544E3F2UC3UVE2ITUQAWGVDMSPS2HFIVHXZ2OLWB3T25AHMFAVA, seg_pos={0 0}, upload=5081]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:44456->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:44456->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44456->77.90.31.17:4772: use of closed network connection Jul 19 05:15:24.941543591 (6h24m31.314122946s) [noise=true , address=77.90.31.16:4734 , piece_id=YUW6MWSUDNLAC2VHLBT6H4T27YTNJ3RDK4BSGE7OZQCO3F2LBH4A, seg_pos={0 0}, upload=5081]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Jul 19 05:15:24.941614786 (6h24m31.314194141s) [noise=true , address=77.90.31.11:4555 , piece_id=YICA5IVHSCC3RPC5FRKD26BSMY6N5WJQA5LCABJSRJYL6BZJYL6A, seg_pos={0 0}, upload=5081]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: write tcp 188.166.225.170:37392->77.90.31.11:4555: use of closed network connection; write tcp 188.166.225.170:37392->77.90.31.11:4555: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37392->77.90.31.11:4555: use of closed network connection Jul 19 05:15:24.941641888 (6h24m31.314221243s) [noise=true , address=77.90.31.18:4806 , piece_id=67PXFJMJB6XWNU6JMOKPO4BXKDD2Z225ZKTB7DZEA46OMSSXHHMQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:47114->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:47114->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47114->77.90.31.18:4806: use of closed network connection Jul 19 05:15:24.941663099 (6h24m31.314242454s) [seg_pos={0 0}, upload=5081]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 19 05:15:25.179949476 (6h24m31.552528831s) [upload=5081]: Done uploading segments... Jul 19 05:15:25.179975797 (6h24m31.552555152s) [upload=5081]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 20.829366946s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 5.611042922s FAIL (2025-07-19 05:20:43 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 4855s) [upload=4769]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 04:51:09.109709627 (6h0m15.482288982s) [upload=4769]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 04:51:09.556145753 (6h0m15.928725108s) [seg_pos={0 0}, upload=4769]: Begin upload segment... Jul 19 04:51:09.556194817 (6h0m15.928774172s) [seg_pos={0 0}, upload=4769]: Done begin upload segment. Jul 19 04:51:09.556223893 (6h0m15.928803248s) [noise=true , address=79.127.219.38:4023 , piece_id=PTPOZGLYV4MZTOFZNN3NCGYL7DFAST4PXKEUIVTMFL7HV4GZTCLQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556252197 (6h0m15.928831552s) [noise=true , address=45.91.95.4:4011 , piece_id=ZMBAS6KKHYVIRE6OBL2NFVCWMM4TKQ7J64BD3SYXIK54XCEDI2XA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556579383 (6h0m15.929158738s) [noise=true , address=45.91.116.6:4007 , piece_id=A2AFVTZBXSGBOE7XIM3B5HRN4STITFBXO6MZRJ365URNAAOZ3PSQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556854469 (6h0m15.929433824s) [noise=true , address=104.36.82.19:4423 , piece_id=PIJUHDUND36HQL67I33D7A3EULMJHS7N55MN4OFQJ4OWNQ42O47A, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557094107 (6h0m15.929673462s) [noise=true , address=45.91.116.9:4264 , piece_id=GHM5BKZKMT2QOHHBOS4FSPOWEWSFQRGRVVR4GSNDPHVIFRZHQL2A, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557336381 (6h0m15.929915736s) [noise=true , address=79.127.205.235:4660 , piece_id=UOUOGE2VP7PX7EMRZLPMCWFCYEZERM3DQMCQ6NG4J2K2DEIZR63Q, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557557885 (6h0m15.93013724s) [noise=true , address=79.127.205.246:4280 , piece_id=FLBOZI3UGTQXAQBLZ2F45Z3OEYL65F5PQBXRF2DQMZ7NGF7VDRPA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557762296 (6h0m15.930341651s) [noise=true , address=79.127.205.233:4622 , piece_id=6WWNYQTH5MGAJOEB7BXKZDJOMCAZXDW66J3OEANGDNJDNUMLZM7Q, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558014689 (6h0m15.930594044s) [noise=true , address=64.49.13.138:4482 , piece_id=MAW2M4US2OQNOS3WI67AA7NIBTJDE5T2JQTXKFPBLGVGXKVCFGTQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558215974 (6h0m15.930795319s) [noise=true , address=77.90.31.10:4546 , piece_id=R6LCA5WZOEBDURL5S3MEK22YPVYXX6EXSS4J4W57543HELYXOKZA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558409364 (6h0m15.930988719s) [noise=true , address=45.91.95.5:4019 , piece_id=YTSBCN2LPHEMRRAZIHJGVB7NBOTRXVEHD2TWUCTTLBNFFYEWWDKA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558623504 (6h0m15.931202859s) [noise=true , address=79.127.201.215:4357 , piece_id=75XWAG2VQWEYCWN2Q3WRIZSBI7LDOISZJPXQLQ2GIPNEN34VLSUQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558805162 (6h0m15.931384517s) [noise=true , address=45.91.116.14:4219 , piece_id=PDW6JPRHCVMHHOWIHT6JKV4CTSC3JY6NDKDVFZSO47OYNPZ7PGGA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558999373 (6h0m15.931578728s) [noise=true , address=77.90.31.11:4564 , piece_id=L2U7JFJRDQOFA2AYVAGTI4ZW4VABCXKMXWBIUWQO2QNNUUBJ74BA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559184768 (6h0m15.931764123s) [noise=true , address=77.90.31.16:4768 , piece_id=YNLSONFM6COZEVF6ZNQR2EA6V4LSQFZJONIXR4AFTNNMFGPV57JA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559368719 (6h0m15.931948074s) [noise=true , address=79.127.220.98:4432 , piece_id=2SUTFQ7FVDPAM2H4W3OLW6AVH3VJ33IJFE4L52SA3ET5LGBMOMPQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559549706 (6h0m15.932129061s) [noise=true , address=77.90.31.18:4813 , piece_id=VDOT7BB4O7XYJHW6BFTAYQZZYLZU5PAUXVJGSXJ7FDI4TH425COA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559727836 (6h0m15.932307191s) [noise=true , address=79.127.219.51:4211 , piece_id=HO5PLWBVU5YAYNYMN352NA73SM4WBJV4WZ7XLSQTCQ76UAL2TOJA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:10.663359087 (6h0m17.035938442s) [upload=4769]: Next returned nil segment Jul 19 04:51:10.663378664 (6h0m17.035958019s) [upload=4769]: Waiting for error group managing segments... Jul 19 04:51:10.962945479 (6h0m17.335524834s) [noise=true , address=79.127.205.233:4622 , piece_id=6WWNYQTH5MGAJOEB7BXKZDJOMCAZXDW66J3OEANGDNJDNUMLZM7Q, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:10.967689807 (6h0m17.340269152s) [noise=true , address=79.127.205.235:4660 , piece_id=UOUOGE2VP7PX7EMRZLPMCWFCYEZERM3DQMCQ6NG4J2K2DEIZR63Q, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:10.976968631 (6h0m17.349547986s) [noise=true , address=64.49.13.138:4482 , piece_id=MAW2M4US2OQNOS3WI67AA7NIBTJDE5T2JQTXKFPBLGVGXKVCFGTQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.048750700 (6h0m17.421330055s) [noise=true , address=79.127.205.246:4280 , piece_id=FLBOZI3UGTQXAQBLZ2F45Z3OEYL65F5PQBXRF2DQMZ7NGF7VDRPA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.221642403 (6h0m17.594221758s) [noise=true , address=79.127.220.98:4432 , piece_id=2SUTFQ7FVDPAM2H4W3OLW6AVH3VJ33IJFE4L52SA3ET5LGBMOMPQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.317070435 (6h0m17.68964979s) [noise=true , address=45.91.95.4:4011 , piece_id=ZMBAS6KKHYVIRE6OBL2NFVCWMM4TKQ7J64BD3SYXIK54XCEDI2XA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.324002660 (6h0m17.696582015s) [noise=true , address=45.91.95.5:4019 , piece_id=YTSBCN2LPHEMRRAZIHJGVB7NBOTRXVEHD2TWUCTTLBNFFYEWWDKA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.460976317 (6h0m17.833555672s) [noise=true , address=79.127.201.215:4357 , piece_id=75XWAG2VQWEYCWN2Q3WRIZSBI7LDOISZJPXQLQ2GIPNEN34VLSUQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.462584603 (6h0m17.835163958s) [noise=true , address=45.91.116.6:4007 , piece_id=A2AFVTZBXSGBOE7XIM3B5HRN4STITFBXO6MZRJ365URNAAOZ3PSQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.490392973 (6h0m17.862972328s) [noise=true , address=45.91.116.9:4264 , piece_id=GHM5BKZKMT2QOHHBOS4FSPOWEWSFQRGRVVR4GSNDPHVIFRZHQL2A, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.502040780 (6h0m17.874620135s) [noise=true , address=45.91.116.14:4219 , piece_id=PDW6JPRHCVMHHOWIHT6JKV4CTSC3JY6NDKDVFZSO47OYNPZ7PGGA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.600434096 (6h0m17.973013451s) [noise=true , address=79.127.219.51:4211 , piece_id=HO5PLWBVU5YAYNYMN352NA73SM4WBJV4WZ7XLSQTCQ76UAL2TOJA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.609008984 (6h0m17.981588339s) [noise=true , address=104.36.82.19:4423 , piece_id=PIJUHDUND36HQL67I33D7A3EULMJHS7N55MN4OFQJ4OWNQ42O47A, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.620825273 (6h0m17.993404628s) [noise=true , address=79.127.219.38:4023 , piece_id=PTPOZGLYV4MZTOFZNN3NCGYL7DFAST4PXKEUIVTMFL7HV4GZTCLQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.620838659 (6h0m17.993418014s) [seg_pos={0 0}, upload=4769]: Segment reached optimal threshold of 14 pieces. Jul 19 04:51:11.621067807 (6h0m17.993647163s) [noise=true , address=77.90.31.11:4564 , piece_id=L2U7JFJRDQOFA2AYVAGTI4ZW4VABCXKMXWBIUWQO2QNNUUBJ74BA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection; write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection Jul 19 04:51:11.621134234 (6h0m17.993713589s) [noise=true , address=77.90.31.10:4546 , piece_id=R6LCA5WZOEBDURL5S3MEK22YPVYXX6EXSS4J4W57543HELYXOKZA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK): protocol: read tcp 188.166.225.170:58476->77.90.31.10:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621174191 (6h0m17.993753546s) [noise=true , address=77.90.31.16:4768 , piece_id=YNLSONFM6COZEVF6ZNQR2EA6V4LSQFZJONIXR4AFTNNMFGPV57JA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UchewmtxBrtSZSpVAJFdZmcTsQVQxkjK9re6cgUGqDhyonQeF): protocol: read tcp 188.166.225.170:56326->77.90.31.16:4768: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621216842 (6h0m17.993796197s) [noise=true , address=77.90.31.18:4813 , piece_id=VDOT7BB4O7XYJHW6BFTAYQZZYLZU5PAUXVJGSXJ7FDI4TH425COA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128WthA1x9wp1dV1vkpGnoveZQpU9CSgDCZfw5GMQuVpCNSVAM5): protocol: read tcp 188.166.225.170:49704->77.90.31.18:4813: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621234917 (6h0m17.993814272s) [seg_pos={0 0}, upload=4769]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 19 04:51:11.621301444 (6h0m17.993880799s) [upload=4769]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 19 04:51:11.621305402 (6h0m17.993884758s) [upload=4769]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 19 04:51:11.923136970 (6h0m18.295716325s) [upload=4769]: Done uploading segments... Jul 19 04:51:11.923156627 (6h0m18.295735972s) [upload=4769]: Upload finished. err: <nil> Duration: 5.61107784s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 5.574564534s FAIL (2025-07-19 05:19:55 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 4855s) [upload=4769]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 04:51:09.109709627 (6h0m15.482288982s) [upload=4769]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 04:51:09.556145753 (6h0m15.928725108s) [seg_pos={0 0}, upload=4769]: Begin upload segment... Jul 19 04:51:09.556194817 (6h0m15.928774172s) [seg_pos={0 0}, upload=4769]: Done begin upload segment. Jul 19 04:51:09.556223893 (6h0m15.928803248s) [noise=true , address=79.127.219.38:4023 , piece_id=PTPOZGLYV4MZTOFZNN3NCGYL7DFAST4PXKEUIVTMFL7HV4GZTCLQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556252197 (6h0m15.928831552s) [noise=true , address=45.91.95.4:4011 , piece_id=ZMBAS6KKHYVIRE6OBL2NFVCWMM4TKQ7J64BD3SYXIK54XCEDI2XA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556579383 (6h0m15.929158738s) [noise=true , address=45.91.116.6:4007 , piece_id=A2AFVTZBXSGBOE7XIM3B5HRN4STITFBXO6MZRJ365URNAAOZ3PSQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556854469 (6h0m15.929433824s) [noise=true , address=104.36.82.19:4423 , piece_id=PIJUHDUND36HQL67I33D7A3EULMJHS7N55MN4OFQJ4OWNQ42O47A, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557094107 (6h0m15.929673462s) [noise=true , address=45.91.116.9:4264 , piece_id=GHM5BKZKMT2QOHHBOS4FSPOWEWSFQRGRVVR4GSNDPHVIFRZHQL2A, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557336381 (6h0m15.929915736s) [noise=true , address=79.127.205.235:4660 , piece_id=UOUOGE2VP7PX7EMRZLPMCWFCYEZERM3DQMCQ6NG4J2K2DEIZR63Q, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557557885 (6h0m15.93013724s) [noise=true , address=79.127.205.246:4280 , piece_id=FLBOZI3UGTQXAQBLZ2F45Z3OEYL65F5PQBXRF2DQMZ7NGF7VDRPA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557762296 (6h0m15.930341651s) [noise=true , address=79.127.205.233:4622 , piece_id=6WWNYQTH5MGAJOEB7BXKZDJOMCAZXDW66J3OEANGDNJDNUMLZM7Q, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558014689 (6h0m15.930594044s) [noise=true , address=64.49.13.138:4482 , piece_id=MAW2M4US2OQNOS3WI67AA7NIBTJDE5T2JQTXKFPBLGVGXKVCFGTQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558215974 (6h0m15.930795319s) [noise=true , address=77.90.31.10:4546 , piece_id=R6LCA5WZOEBDURL5S3MEK22YPVYXX6EXSS4J4W57543HELYXOKZA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558409364 (6h0m15.930988719s) [noise=true , address=45.91.95.5:4019 , piece_id=YTSBCN2LPHEMRRAZIHJGVB7NBOTRXVEHD2TWUCTTLBNFFYEWWDKA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558623504 (6h0m15.931202859s) [noise=true , address=79.127.201.215:4357 , piece_id=75XWAG2VQWEYCWN2Q3WRIZSBI7LDOISZJPXQLQ2GIPNEN34VLSUQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558805162 (6h0m15.931384517s) [noise=true , address=45.91.116.14:4219 , piece_id=PDW6JPRHCVMHHOWIHT6JKV4CTSC3JY6NDKDVFZSO47OYNPZ7PGGA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558999373 (6h0m15.931578728s) [noise=true , address=77.90.31.11:4564 , piece_id=L2U7JFJRDQOFA2AYVAGTI4ZW4VABCXKMXWBIUWQO2QNNUUBJ74BA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559184768 (6h0m15.931764123s) [noise=true , address=77.90.31.16:4768 , piece_id=YNLSONFM6COZEVF6ZNQR2EA6V4LSQFZJONIXR4AFTNNMFGPV57JA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559368719 (6h0m15.931948074s) [noise=true , address=79.127.220.98:4432 , piece_id=2SUTFQ7FVDPAM2H4W3OLW6AVH3VJ33IJFE4L52SA3ET5LGBMOMPQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559549706 (6h0m15.932129061s) [noise=true , address=77.90.31.18:4813 , piece_id=VDOT7BB4O7XYJHW6BFTAYQZZYLZU5PAUXVJGSXJ7FDI4TH425COA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559727836 (6h0m15.932307191s) [noise=true , address=79.127.219.51:4211 , piece_id=HO5PLWBVU5YAYNYMN352NA73SM4WBJV4WZ7XLSQTCQ76UAL2TOJA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:10.663359087 (6h0m17.035938442s) [upload=4769]: Next returned nil segment Jul 19 04:51:10.663378664 (6h0m17.035958019s) [upload=4769]: Waiting for error group managing segments... Jul 19 04:51:10.962945479 (6h0m17.335524834s) [noise=true , address=79.127.205.233:4622 , piece_id=6WWNYQTH5MGAJOEB7BXKZDJOMCAZXDW66J3OEANGDNJDNUMLZM7Q, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:10.967689807 (6h0m17.340269152s) [noise=true , address=79.127.205.235:4660 , piece_id=UOUOGE2VP7PX7EMRZLPMCWFCYEZERM3DQMCQ6NG4J2K2DEIZR63Q, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:10.976968631 (6h0m17.349547986s) [noise=true , address=64.49.13.138:4482 , piece_id=MAW2M4US2OQNOS3WI67AA7NIBTJDE5T2JQTXKFPBLGVGXKVCFGTQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.048750700 (6h0m17.421330055s) [noise=true , address=79.127.205.246:4280 , piece_id=FLBOZI3UGTQXAQBLZ2F45Z3OEYL65F5PQBXRF2DQMZ7NGF7VDRPA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.221642403 (6h0m17.594221758s) [noise=true , address=79.127.220.98:4432 , piece_id=2SUTFQ7FVDPAM2H4W3OLW6AVH3VJ33IJFE4L52SA3ET5LGBMOMPQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.317070435 (6h0m17.68964979s) [noise=true , address=45.91.95.4:4011 , piece_id=ZMBAS6KKHYVIRE6OBL2NFVCWMM4TKQ7J64BD3SYXIK54XCEDI2XA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.324002660 (6h0m17.696582015s) [noise=true , address=45.91.95.5:4019 , piece_id=YTSBCN2LPHEMRRAZIHJGVB7NBOTRXVEHD2TWUCTTLBNFFYEWWDKA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.460976317 (6h0m17.833555672s) [noise=true , address=79.127.201.215:4357 , piece_id=75XWAG2VQWEYCWN2Q3WRIZSBI7LDOISZJPXQLQ2GIPNEN34VLSUQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.462584603 (6h0m17.835163958s) [noise=true , address=45.91.116.6:4007 , piece_id=A2AFVTZBXSGBOE7XIM3B5HRN4STITFBXO6MZRJ365URNAAOZ3PSQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.490392973 (6h0m17.862972328s) [noise=true , address=45.91.116.9:4264 , piece_id=GHM5BKZKMT2QOHHBOS4FSPOWEWSFQRGRVVR4GSNDPHVIFRZHQL2A, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.502040780 (6h0m17.874620135s) [noise=true , address=45.91.116.14:4219 , piece_id=PDW6JPRHCVMHHOWIHT6JKV4CTSC3JY6NDKDVFZSO47OYNPZ7PGGA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.600434096 (6h0m17.973013451s) [noise=true , address=79.127.219.51:4211 , piece_id=HO5PLWBVU5YAYNYMN352NA73SM4WBJV4WZ7XLSQTCQ76UAL2TOJA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.609008984 (6h0m17.981588339s) [noise=true , address=104.36.82.19:4423 , piece_id=PIJUHDUND36HQL67I33D7A3EULMJHS7N55MN4OFQJ4OWNQ42O47A, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.620825273 (6h0m17.993404628s) [noise=true , address=79.127.219.38:4023 , piece_id=PTPOZGLYV4MZTOFZNN3NCGYL7DFAST4PXKEUIVTMFL7HV4GZTCLQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.620838659 (6h0m17.993418014s) [seg_pos={0 0}, upload=4769]: Segment reached optimal threshold of 14 pieces. Jul 19 04:51:11.621067807 (6h0m17.993647163s) [noise=true , address=77.90.31.11:4564 , piece_id=L2U7JFJRDQOFA2AYVAGTI4ZW4VABCXKMXWBIUWQO2QNNUUBJ74BA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection; write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection Jul 19 04:51:11.621134234 (6h0m17.993713589s) [noise=true , address=77.90.31.10:4546 , piece_id=R6LCA5WZOEBDURL5S3MEK22YPVYXX6EXSS4J4W57543HELYXOKZA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK): protocol: read tcp 188.166.225.170:58476->77.90.31.10:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621174191 (6h0m17.993753546s) [noise=true , address=77.90.31.16:4768 , piece_id=YNLSONFM6COZEVF6ZNQR2EA6V4LSQFZJONIXR4AFTNNMFGPV57JA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UchewmtxBrtSZSpVAJFdZmcTsQVQxkjK9re6cgUGqDhyonQeF): protocol: read tcp 188.166.225.170:56326->77.90.31.16:4768: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621216842 (6h0m17.993796197s) [noise=true , address=77.90.31.18:4813 , piece_id=VDOT7BB4O7XYJHW6BFTAYQZZYLZU5PAUXVJGSXJ7FDI4TH425COA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128WthA1x9wp1dV1vkpGnoveZQpU9CSgDCZfw5GMQuVpCNSVAM5): protocol: read tcp 188.166.225.170:49704->77.90.31.18:4813: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621234917 (6h0m17.993814272s) [seg_pos={0 0}, upload=4769]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 19 04:51:11.621301444 (6h0m17.993880799s) [upload=4769]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 19 04:51:11.621305402 (6h0m17.993884758s) [upload=4769]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 19 04:51:11.923136970 (6h0m18.295716325s) [upload=4769]: Done uploading segments... Jul 19 04:51:11.923156627 (6h0m18.295735972s) [upload=4769]: Upload finished. err: <nil> Duration: 5.574589191s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [..............XX...X...X....X..X....XXXXX.X.X.....XX.X......] duration: 5.864135854s FAIL (2025-07-19 05:19:45 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== gment. successful: 13 optimal: 13 errs: <nil> Jul 19 05:14:35.936262577 (6h23m42.308841932s) [upload=5075]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 19 05:14:35.936281884 (6h23m42.308861239s) [upload=5075]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 19 05:14:36.234171969 (6h23m42.606751314s) [upload=5075]: Done uploading segments... Jul 19 05:14:36.234192037 (6h23m42.606771382s) [upload=5075]: Upload finished. err: <nil> Jul 19 05:14:55.587986728 (6h24m1.960566093s) [upload=5081]: Starting upload Jul 19 05:14:55.588051823 (6h24m1.960631178s) [upload=5081]: Done starting upload Jul 19 05:14:55.588146724 (6h24m1.960726079s) [upload=5081]: Uploading segments... Jul 19 05:14:55.588391873 (6h24m1.960971228s) [upload=5081]: Got next segment. Inline: false Jul 19 05:14:55.588649837 (6h24m1.961229193s) [upload=5081]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 05:14:55.588655057 (6h24m1.961234402s) [upload=5081]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 05:14:55.823387692 (6h24m2.195967047s) [seg_pos={0 0}, upload=5081]: Begin upload segment... Jul 19 05:14:55.823450572 (6h24m2.196029927s) [seg_pos={0 0}, upload=5081]: Done begin upload segment. Jul 19 05:14:55.823481952 (6h24m2.196061307s) [noise=true , address=77.90.31.17:4772 , piece_id=B544E3F2UC3UVE2ITUQAWGVDMSPS2HFIVHXZ2OLWB3T25AHMFAVA, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.823491401 (6h24m2.196070756s) [noise=true , address=45.91.116.10:4082 , piece_id=HZDMC4J5O5UHVDH5SOQBKAFPN6RJUHIWKEZ4OPVMW2HSN7NDZMZA, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.823923789 (6h24m2.196503144s) [noise=true , address=45.91.116.9:4262 , piece_id=FRIATDM7LPPHXYAMUISAYRTQWAKSTB4EYUANHGEMQHM6JQADDTRQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.823949737 (6h24m2.196529092s) [noise=true , address=45.91.116.12:4227 , piece_id=QB7YKZFXWMRB3VQ3HSFMMMLLX5YFWOXG4Y4GJGGYT6HDV4FQ65VA, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.824310158 (6h24m2.196889513s) [noise=true , address=45.91.116.4:4937 , piece_id=V2IOOA7DVCGZBH3D43ROYHT6ZECPY6GTC5M7NFO2RWPQB6EYGALA, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.824403045 (6h24m2.19698239s) [noise=true , address=77.90.31.11:4555 , piece_id=YICA5IVHSCC3RPC5FRKD26BSMY6N5WJQA5LCABJSRJYL6BZJYL6A, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.824651210 (6h24m2.197230565s) [noise=true , address=77.90.31.13:4627 , piece_id=E7GSSBZ653K2AA4TP6A4BRT5BDZCBODNRAZM3P6SYJXJFQU2BACQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.824716334 (6h24m2.197295689s) [noise=true , address=77.90.31.16:4734 , piece_id=YUW6MWSUDNLAC2VHLBT6H4T27YTNJ3RDK4BSGE7OZQCO3F2LBH4A, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.824960491 (6h24m2.197539856s) [noise=true , address=45.91.116.7:4010 , piece_id=BH76RM4NXGEAI2J2424ALK4PFSXUTBCYOH777BUOIDW7BW2IJUJA, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.824977494 (6h24m2.197556849s) [noise=true , address=45.91.116.14:4190 , piece_id=RUU2MICSAWOECAAYNOBS75SE2YKSUBMSSFKLPON7FWK77UNJEYGQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.825213285 (6h24m2.19779264s) [noise=true , address=45.91.116.8:4047 , piece_id=6UOFOV4VCVT6DB7D7Q23REAX7VZPAW5PSG5PLA54LISOUWMPQBBQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.825271547 (6h24m2.197850902s) [noise=true , address=77.90.31.18:4806 , piece_id=67PXFJMJB6XWNU6JMOKPO4BXKDD2Z225ZKTB7DZEA46OMSSXHHMQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.825445860 (6h24m2.198025215s) [noise=true , address=45.91.116.7:4009 , piece_id=HH5I32ZYT5LVXHDLBVSAVX6T23MJ6RGNPNAWPIQAREJ4ELPFJGYA, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.825568244 (6h24m2.198147599s) [noise=true , address=77.90.31.15:4697 , piece_id=W62ARDAVLI64P6VYSPVFRTDKB5H2WXTSCF2VKP3UN7JKGBCUEZTQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.825902985 (6h24m2.19848234s) [noise=true , address=45.91.116.5:4299 , piece_id=JQLK5LIMCMW6R2UZEJZSG6QAP2M65XBVE6JFEXY5KDWBVNDG2VCQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.826216495 (6h24m2.19879585s) [noise=true , address=77.90.31.14:4663 , piece_id=LNTFUWB6LUUFD2EB536ZHBSNVVXYZPOOB4DUZB2M6BXSSFEL3UMQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:58.633690556 (6h24m5.006269911s) [upload=5081]: Next returned nil segment Jul 19 05:14:58.633708100 (6h24m5.006287445s) [upload=5081]: Waiting for error group managing segments... Jul 19 05:14:58.915966948 (6h24m5.288546293s) [noise=true , address=45.91.116.7:4009 , piece_id=HH5I32ZYT5LVXHDLBVSAVX6T23MJ6RGNPNAWPIQAREJ4ELPFJGYA, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:58.916093541 (6h24m5.288672896s) [noise=true , address=45.91.116.12:4227 , piece_id=QB7YKZFXWMRB3VQ3HSFMMMLLX5YFWOXG4Y4GJGGYT6HDV4FQ65VA, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:58.923615462 (6h24m5.296194817s) [noise=true , address=45.91.116.14:4190 , piece_id=RUU2MICSAWOECAAYNOBS75SE2YKSUBMSSFKLPON7FWK77UNJEYGQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:58.928612934 (6h24m5.301192289s) [noise=true , address=45.91.116.7:4010 , piece_id=BH76RM4NXGEAI2J2424ALK4PFSXUTBCYOH777BUOIDW7BW2IJUJA, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:58.990915468 (6h24m5.363494833s) [noise=true , address=45.91.116.4:4937 , piece_id=V2IOOA7DVCGZBH3D43ROYHT6ZECPY6GTC5M7NFO2RWPQB6EYGALA, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:58.992136464 (6h24m5.364715819s) [noise=true , address=45.91.116.8:4047 , piece_id=6UOFOV4VCVT6DB7D7Q23REAX7VZPAW5PSG5PLA54LISOUWMPQBBQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:58.997711812 (6h24m5.370291157s) [noise=true , address=45.91.116.9:4262 , piece_id=FRIATDM7LPPHXYAMUISAYRTQWAKSTB4EYUANHGEMQHM6JQADDTRQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:59.010548391 (6h24m5.383127746s) [noise=true , address=45.91.116.10:4082 , piece_id=HZDMC4J5O5UHVDH5SOQBKAFPN6RJUHIWKEZ4OPVMW2HSN7NDZMZA, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:59.039607621 (6h24m5.412186996s) [noise=true , address=45.91.116.5:4299 , piece_id=JQLK5LIMCMW6R2UZEJZSG6QAP2M65XBVE6JFEXY5KDWBVNDG2VCQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:15:19.998482215 (6h24m26.37106157s) [noise=true , address=77.90.31.14:4663 , piece_id=LNTFUWB6LUUFD2EB536ZHBSNVVXYZPOOB4DUZB2M6BXSSFEL3UMQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:15:21.170089451 (6h24m27.542668806s) [noise=true , address=77.90.31.13:4627 , piece_id=E7GSSBZ653K2AA4TP6A4BRT5BDZCBODNRAZM3P6SYJXJFQU2BACQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:15:24.941469148 (6h24m31.314048503s) [noise=true , address=77.90.31.15:4697 , piece_id=W62ARDAVLI64P6VYSPVFRTDKB5H2WXTSCF2VKP3UN7JKGBCUEZTQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: expected piece hash; context deadline exceeded; EOF Jul 19 05:15:24.941530766 (6h24m31.314110121s) [noise=true , address=77.90.31.17:4772 , piece_id=B544E3F2UC3UVE2ITUQAWGVDMSPS2HFIVHXZ2OLWB3T25AHMFAVA, seg_pos={0 0}, upload=5081]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:44456->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:44456->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44456->77.90.31.17:4772: use of closed network connection Jul 19 05:15:24.941543591 (6h24m31.314122946s) [noise=true , address=77.90.31.16:4734 , piece_id=YUW6MWSUDNLAC2VHLBT6H4T27YTNJ3RDK4BSGE7OZQCO3F2LBH4A, seg_pos={0 0}, upload=5081]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Jul 19 05:15:24.941614786 (6h24m31.314194141s) [noise=true , address=77.90.31.11:4555 , piece_id=YICA5IVHSCC3RPC5FRKD26BSMY6N5WJQA5LCABJSRJYL6BZJYL6A, seg_pos={0 0}, upload=5081]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: write tcp 188.166.225.170:37392->77.90.31.11:4555: use of closed network connection; write tcp 188.166.225.170:37392->77.90.31.11:4555: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37392->77.90.31.11:4555: use of closed network connection Jul 19 05:15:24.941641888 (6h24m31.314221243s) [noise=true , address=77.90.31.18:4806 , piece_id=67PXFJMJB6XWNU6JMOKPO4BXKDD2Z225ZKTB7DZEA46OMSSXHHMQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:47114->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:47114->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47114->77.90.31.18:4806: use of closed network connection Jul 19 05:15:24.941663099 (6h24m31.314242454s) [seg_pos={0 0}, upload=5081]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 19 05:15:25.179949476 (6h24m31.552528831s) [upload=5081]: Done uploading segments... Jul 19 05:15:25.179975797 (6h24m31.552555152s) [upload=5081]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.864150953s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [...............XX...X...X....X..X....XXXXX.X.X.....XX.X.....] duration: 29.643075808s FAIL (2025-07-19 05:19:38 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== gment. successful: 13 optimal: 13 errs: <nil> Jul 19 05:14:35.936262577 (6h23m42.308841932s) [upload=5075]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 19 05:14:35.936281884 (6h23m42.308861239s) [upload=5075]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 19 05:14:36.234171969 (6h23m42.606751314s) [upload=5075]: Done uploading segments... Jul 19 05:14:36.234192037 (6h23m42.606771382s) [upload=5075]: Upload finished. err: <nil> Jul 19 05:14:55.587986728 (6h24m1.960566093s) [upload=5081]: Starting upload Jul 19 05:14:55.588051823 (6h24m1.960631178s) [upload=5081]: Done starting upload Jul 19 05:14:55.588146724 (6h24m1.960726079s) [upload=5081]: Uploading segments... Jul 19 05:14:55.588391873 (6h24m1.960971228s) [upload=5081]: Got next segment. Inline: false Jul 19 05:14:55.588649837 (6h24m1.961229193s) [upload=5081]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 05:14:55.588655057 (6h24m1.961234402s) [upload=5081]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 05:14:55.823387692 (6h24m2.195967047s) [seg_pos={0 0}, upload=5081]: Begin upload segment... Jul 19 05:14:55.823450572 (6h24m2.196029927s) [seg_pos={0 0}, upload=5081]: Done begin upload segment. Jul 19 05:14:55.823481952 (6h24m2.196061307s) [noise=true , address=77.90.31.17:4772 , piece_id=B544E3F2UC3UVE2ITUQAWGVDMSPS2HFIVHXZ2OLWB3T25AHMFAVA, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.823491401 (6h24m2.196070756s) [noise=true , address=45.91.116.10:4082 , piece_id=HZDMC4J5O5UHVDH5SOQBKAFPN6RJUHIWKEZ4OPVMW2HSN7NDZMZA, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.823923789 (6h24m2.196503144s) [noise=true , address=45.91.116.9:4262 , piece_id=FRIATDM7LPPHXYAMUISAYRTQWAKSTB4EYUANHGEMQHM6JQADDTRQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.823949737 (6h24m2.196529092s) [noise=true , address=45.91.116.12:4227 , piece_id=QB7YKZFXWMRB3VQ3HSFMMMLLX5YFWOXG4Y4GJGGYT6HDV4FQ65VA, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.824310158 (6h24m2.196889513s) [noise=true , address=45.91.116.4:4937 , piece_id=V2IOOA7DVCGZBH3D43ROYHT6ZECPY6GTC5M7NFO2RWPQB6EYGALA, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.824403045 (6h24m2.19698239s) [noise=true , address=77.90.31.11:4555 , piece_id=YICA5IVHSCC3RPC5FRKD26BSMY6N5WJQA5LCABJSRJYL6BZJYL6A, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.824651210 (6h24m2.197230565s) [noise=true , address=77.90.31.13:4627 , piece_id=E7GSSBZ653K2AA4TP6A4BRT5BDZCBODNRAZM3P6SYJXJFQU2BACQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.824716334 (6h24m2.197295689s) [noise=true , address=77.90.31.16:4734 , piece_id=YUW6MWSUDNLAC2VHLBT6H4T27YTNJ3RDK4BSGE7OZQCO3F2LBH4A, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.824960491 (6h24m2.197539856s) [noise=true , address=45.91.116.7:4010 , piece_id=BH76RM4NXGEAI2J2424ALK4PFSXUTBCYOH777BUOIDW7BW2IJUJA, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.824977494 (6h24m2.197556849s) [noise=true , address=45.91.116.14:4190 , piece_id=RUU2MICSAWOECAAYNOBS75SE2YKSUBMSSFKLPON7FWK77UNJEYGQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.825213285 (6h24m2.19779264s) [noise=true , address=45.91.116.8:4047 , piece_id=6UOFOV4VCVT6DB7D7Q23REAX7VZPAW5PSG5PLA54LISOUWMPQBBQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.825271547 (6h24m2.197850902s) [noise=true , address=77.90.31.18:4806 , piece_id=67PXFJMJB6XWNU6JMOKPO4BXKDD2Z225ZKTB7DZEA46OMSSXHHMQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.825445860 (6h24m2.198025215s) [noise=true , address=45.91.116.7:4009 , piece_id=HH5I32ZYT5LVXHDLBVSAVX6T23MJ6RGNPNAWPIQAREJ4ELPFJGYA, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.825568244 (6h24m2.198147599s) [noise=true , address=77.90.31.15:4697 , piece_id=W62ARDAVLI64P6VYSPVFRTDKB5H2WXTSCF2VKP3UN7JKGBCUEZTQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.825902985 (6h24m2.19848234s) [noise=true , address=45.91.116.5:4299 , piece_id=JQLK5LIMCMW6R2UZEJZSG6QAP2M65XBVE6JFEXY5KDWBVNDG2VCQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.826216495 (6h24m2.19879585s) [noise=true , address=77.90.31.14:4663 , piece_id=LNTFUWB6LUUFD2EB536ZHBSNVVXYZPOOB4DUZB2M6BXSSFEL3UMQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:58.633690556 (6h24m5.006269911s) [upload=5081]: Next returned nil segment Jul 19 05:14:58.633708100 (6h24m5.006287445s) [upload=5081]: Waiting for error group managing segments... Jul 19 05:14:58.915966948 (6h24m5.288546293s) [noise=true , address=45.91.116.7:4009 , piece_id=HH5I32ZYT5LVXHDLBVSAVX6T23MJ6RGNPNAWPIQAREJ4ELPFJGYA, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:58.916093541 (6h24m5.288672896s) [noise=true , address=45.91.116.12:4227 , piece_id=QB7YKZFXWMRB3VQ3HSFMMMLLX5YFWOXG4Y4GJGGYT6HDV4FQ65VA, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:58.923615462 (6h24m5.296194817s) [noise=true , address=45.91.116.14:4190 , piece_id=RUU2MICSAWOECAAYNOBS75SE2YKSUBMSSFKLPON7FWK77UNJEYGQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:58.928612934 (6h24m5.301192289s) [noise=true , address=45.91.116.7:4010 , piece_id=BH76RM4NXGEAI2J2424ALK4PFSXUTBCYOH777BUOIDW7BW2IJUJA, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:58.990915468 (6h24m5.363494833s) [noise=true , address=45.91.116.4:4937 , piece_id=V2IOOA7DVCGZBH3D43ROYHT6ZECPY6GTC5M7NFO2RWPQB6EYGALA, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:58.992136464 (6h24m5.364715819s) [noise=true , address=45.91.116.8:4047 , piece_id=6UOFOV4VCVT6DB7D7Q23REAX7VZPAW5PSG5PLA54LISOUWMPQBBQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:58.997711812 (6h24m5.370291157s) [noise=true , address=45.91.116.9:4262 , piece_id=FRIATDM7LPPHXYAMUISAYRTQWAKSTB4EYUANHGEMQHM6JQADDTRQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:59.010548391 (6h24m5.383127746s) [noise=true , address=45.91.116.10:4082 , piece_id=HZDMC4J5O5UHVDH5SOQBKAFPN6RJUHIWKEZ4OPVMW2HSN7NDZMZA, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:59.039607621 (6h24m5.412186996s) [noise=true , address=45.91.116.5:4299 , piece_id=JQLK5LIMCMW6R2UZEJZSG6QAP2M65XBVE6JFEXY5KDWBVNDG2VCQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:15:19.998482215 (6h24m26.37106157s) [noise=true , address=77.90.31.14:4663 , piece_id=LNTFUWB6LUUFD2EB536ZHBSNVVXYZPOOB4DUZB2M6BXSSFEL3UMQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:15:21.170089451 (6h24m27.542668806s) [noise=true , address=77.90.31.13:4627 , piece_id=E7GSSBZ653K2AA4TP6A4BRT5BDZCBODNRAZM3P6SYJXJFQU2BACQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:15:24.941469148 (6h24m31.314048503s) [noise=true , address=77.90.31.15:4697 , piece_id=W62ARDAVLI64P6VYSPVFRTDKB5H2WXTSCF2VKP3UN7JKGBCUEZTQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: expected piece hash; context deadline exceeded; EOF Jul 19 05:15:24.941530766 (6h24m31.314110121s) [noise=true , address=77.90.31.17:4772 , piece_id=B544E3F2UC3UVE2ITUQAWGVDMSPS2HFIVHXZ2OLWB3T25AHMFAVA, seg_pos={0 0}, upload=5081]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:44456->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:44456->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44456->77.90.31.17:4772: use of closed network connection Jul 19 05:15:24.941543591 (6h24m31.314122946s) [noise=true , address=77.90.31.16:4734 , piece_id=YUW6MWSUDNLAC2VHLBT6H4T27YTNJ3RDK4BSGE7OZQCO3F2LBH4A, seg_pos={0 0}, upload=5081]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Jul 19 05:15:24.941614786 (6h24m31.314194141s) [noise=true , address=77.90.31.11:4555 , piece_id=YICA5IVHSCC3RPC5FRKD26BSMY6N5WJQA5LCABJSRJYL6BZJYL6A, seg_pos={0 0}, upload=5081]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: write tcp 188.166.225.170:37392->77.90.31.11:4555: use of closed network connection; write tcp 188.166.225.170:37392->77.90.31.11:4555: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37392->77.90.31.11:4555: use of closed network connection Jul 19 05:15:24.941641888 (6h24m31.314221243s) [noise=true , address=77.90.31.18:4806 , piece_id=67PXFJMJB6XWNU6JMOKPO4BXKDD2Z225ZKTB7DZEA46OMSSXHHMQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:47114->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:47114->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47114->77.90.31.18:4806: use of closed network connection Jul 19 05:15:24.941663099 (6h24m31.314242454s) [seg_pos={0 0}, upload=5081]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 19 05:15:25.179949476 (6h24m31.552528831s) [upload=5081]: Done uploading segments... Jul 19 05:15:25.179975797 (6h24m31.552555152s) [upload=5081]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 29.643101477s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 5.82525709s FAIL (2025-07-19 05:19:00 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 4855s) [upload=4769]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 04:51:09.109709627 (6h0m15.482288982s) [upload=4769]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 04:51:09.556145753 (6h0m15.928725108s) [seg_pos={0 0}, upload=4769]: Begin upload segment... Jul 19 04:51:09.556194817 (6h0m15.928774172s) [seg_pos={0 0}, upload=4769]: Done begin upload segment. Jul 19 04:51:09.556223893 (6h0m15.928803248s) [noise=true , address=79.127.219.38:4023 , piece_id=PTPOZGLYV4MZTOFZNN3NCGYL7DFAST4PXKEUIVTMFL7HV4GZTCLQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556252197 (6h0m15.928831552s) [noise=true , address=45.91.95.4:4011 , piece_id=ZMBAS6KKHYVIRE6OBL2NFVCWMM4TKQ7J64BD3SYXIK54XCEDI2XA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556579383 (6h0m15.929158738s) [noise=true , address=45.91.116.6:4007 , piece_id=A2AFVTZBXSGBOE7XIM3B5HRN4STITFBXO6MZRJ365URNAAOZ3PSQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556854469 (6h0m15.929433824s) [noise=true , address=104.36.82.19:4423 , piece_id=PIJUHDUND36HQL67I33D7A3EULMJHS7N55MN4OFQJ4OWNQ42O47A, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557094107 (6h0m15.929673462s) [noise=true , address=45.91.116.9:4264 , piece_id=GHM5BKZKMT2QOHHBOS4FSPOWEWSFQRGRVVR4GSNDPHVIFRZHQL2A, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557336381 (6h0m15.929915736s) [noise=true , address=79.127.205.235:4660 , piece_id=UOUOGE2VP7PX7EMRZLPMCWFCYEZERM3DQMCQ6NG4J2K2DEIZR63Q, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557557885 (6h0m15.93013724s) [noise=true , address=79.127.205.246:4280 , piece_id=FLBOZI3UGTQXAQBLZ2F45Z3OEYL65F5PQBXRF2DQMZ7NGF7VDRPA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557762296 (6h0m15.930341651s) [noise=true , address=79.127.205.233:4622 , piece_id=6WWNYQTH5MGAJOEB7BXKZDJOMCAZXDW66J3OEANGDNJDNUMLZM7Q, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558014689 (6h0m15.930594044s) [noise=true , address=64.49.13.138:4482 , piece_id=MAW2M4US2OQNOS3WI67AA7NIBTJDE5T2JQTXKFPBLGVGXKVCFGTQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558215974 (6h0m15.930795319s) [noise=true , address=77.90.31.10:4546 , piece_id=R6LCA5WZOEBDURL5S3MEK22YPVYXX6EXSS4J4W57543HELYXOKZA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558409364 (6h0m15.930988719s) [noise=true , address=45.91.95.5:4019 , piece_id=YTSBCN2LPHEMRRAZIHJGVB7NBOTRXVEHD2TWUCTTLBNFFYEWWDKA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558623504 (6h0m15.931202859s) [noise=true , address=79.127.201.215:4357 , piece_id=75XWAG2VQWEYCWN2Q3WRIZSBI7LDOISZJPXQLQ2GIPNEN34VLSUQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558805162 (6h0m15.931384517s) [noise=true , address=45.91.116.14:4219 , piece_id=PDW6JPRHCVMHHOWIHT6JKV4CTSC3JY6NDKDVFZSO47OYNPZ7PGGA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558999373 (6h0m15.931578728s) [noise=true , address=77.90.31.11:4564 , piece_id=L2U7JFJRDQOFA2AYVAGTI4ZW4VABCXKMXWBIUWQO2QNNUUBJ74BA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559184768 (6h0m15.931764123s) [noise=true , address=77.90.31.16:4768 , piece_id=YNLSONFM6COZEVF6ZNQR2EA6V4LSQFZJONIXR4AFTNNMFGPV57JA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559368719 (6h0m15.931948074s) [noise=true , address=79.127.220.98:4432 , piece_id=2SUTFQ7FVDPAM2H4W3OLW6AVH3VJ33IJFE4L52SA3ET5LGBMOMPQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559549706 (6h0m15.932129061s) [noise=true , address=77.90.31.18:4813 , piece_id=VDOT7BB4O7XYJHW6BFTAYQZZYLZU5PAUXVJGSXJ7FDI4TH425COA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559727836 (6h0m15.932307191s) [noise=true , address=79.127.219.51:4211 , piece_id=HO5PLWBVU5YAYNYMN352NA73SM4WBJV4WZ7XLSQTCQ76UAL2TOJA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:10.663359087 (6h0m17.035938442s) [upload=4769]: Next returned nil segment Jul 19 04:51:10.663378664 (6h0m17.035958019s) [upload=4769]: Waiting for error group managing segments... Jul 19 04:51:10.962945479 (6h0m17.335524834s) [noise=true , address=79.127.205.233:4622 , piece_id=6WWNYQTH5MGAJOEB7BXKZDJOMCAZXDW66J3OEANGDNJDNUMLZM7Q, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:10.967689807 (6h0m17.340269152s) [noise=true , address=79.127.205.235:4660 , piece_id=UOUOGE2VP7PX7EMRZLPMCWFCYEZERM3DQMCQ6NG4J2K2DEIZR63Q, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:10.976968631 (6h0m17.349547986s) [noise=true , address=64.49.13.138:4482 , piece_id=MAW2M4US2OQNOS3WI67AA7NIBTJDE5T2JQTXKFPBLGVGXKVCFGTQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.048750700 (6h0m17.421330055s) [noise=true , address=79.127.205.246:4280 , piece_id=FLBOZI3UGTQXAQBLZ2F45Z3OEYL65F5PQBXRF2DQMZ7NGF7VDRPA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.221642403 (6h0m17.594221758s) [noise=true , address=79.127.220.98:4432 , piece_id=2SUTFQ7FVDPAM2H4W3OLW6AVH3VJ33IJFE4L52SA3ET5LGBMOMPQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.317070435 (6h0m17.68964979s) [noise=true , address=45.91.95.4:4011 , piece_id=ZMBAS6KKHYVIRE6OBL2NFVCWMM4TKQ7J64BD3SYXIK54XCEDI2XA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.324002660 (6h0m17.696582015s) [noise=true , address=45.91.95.5:4019 , piece_id=YTSBCN2LPHEMRRAZIHJGVB7NBOTRXVEHD2TWUCTTLBNFFYEWWDKA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.460976317 (6h0m17.833555672s) [noise=true , address=79.127.201.215:4357 , piece_id=75XWAG2VQWEYCWN2Q3WRIZSBI7LDOISZJPXQLQ2GIPNEN34VLSUQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.462584603 (6h0m17.835163958s) [noise=true , address=45.91.116.6:4007 , piece_id=A2AFVTZBXSGBOE7XIM3B5HRN4STITFBXO6MZRJ365URNAAOZ3PSQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.490392973 (6h0m17.862972328s) [noise=true , address=45.91.116.9:4264 , piece_id=GHM5BKZKMT2QOHHBOS4FSPOWEWSFQRGRVVR4GSNDPHVIFRZHQL2A, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.502040780 (6h0m17.874620135s) [noise=true , address=45.91.116.14:4219 , piece_id=PDW6JPRHCVMHHOWIHT6JKV4CTSC3JY6NDKDVFZSO47OYNPZ7PGGA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.600434096 (6h0m17.973013451s) [noise=true , address=79.127.219.51:4211 , piece_id=HO5PLWBVU5YAYNYMN352NA73SM4WBJV4WZ7XLSQTCQ76UAL2TOJA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.609008984 (6h0m17.981588339s) [noise=true , address=104.36.82.19:4423 , piece_id=PIJUHDUND36HQL67I33D7A3EULMJHS7N55MN4OFQJ4OWNQ42O47A, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.620825273 (6h0m17.993404628s) [noise=true , address=79.127.219.38:4023 , piece_id=PTPOZGLYV4MZTOFZNN3NCGYL7DFAST4PXKEUIVTMFL7HV4GZTCLQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.620838659 (6h0m17.993418014s) [seg_pos={0 0}, upload=4769]: Segment reached optimal threshold of 14 pieces. Jul 19 04:51:11.621067807 (6h0m17.993647163s) [noise=true , address=77.90.31.11:4564 , piece_id=L2U7JFJRDQOFA2AYVAGTI4ZW4VABCXKMXWBIUWQO2QNNUUBJ74BA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection; write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection Jul 19 04:51:11.621134234 (6h0m17.993713589s) [noise=true , address=77.90.31.10:4546 , piece_id=R6LCA5WZOEBDURL5S3MEK22YPVYXX6EXSS4J4W57543HELYXOKZA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK): protocol: read tcp 188.166.225.170:58476->77.90.31.10:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621174191 (6h0m17.993753546s) [noise=true , address=77.90.31.16:4768 , piece_id=YNLSONFM6COZEVF6ZNQR2EA6V4LSQFZJONIXR4AFTNNMFGPV57JA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UchewmtxBrtSZSpVAJFdZmcTsQVQxkjK9re6cgUGqDhyonQeF): protocol: read tcp 188.166.225.170:56326->77.90.31.16:4768: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621216842 (6h0m17.993796197s) [noise=true , address=77.90.31.18:4813 , piece_id=VDOT7BB4O7XYJHW6BFTAYQZZYLZU5PAUXVJGSXJ7FDI4TH425COA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128WthA1x9wp1dV1vkpGnoveZQpU9CSgDCZfw5GMQuVpCNSVAM5): protocol: read tcp 188.166.225.170:49704->77.90.31.18:4813: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621234917 (6h0m17.993814272s) [seg_pos={0 0}, upload=4769]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 19 04:51:11.621301444 (6h0m17.993880799s) [upload=4769]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 19 04:51:11.621305402 (6h0m17.993884758s) [upload=4769]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 19 04:51:11.923136970 (6h0m18.295716325s) [upload=4769]: Done uploading segments... Jul 19 04:51:11.923156627 (6h0m18.295735972s) [upload=4769]: Upload finished. err: <nil> Duration: 5.825278972s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 6.614340519s FAIL (2025-07-19 05:18:35 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 4855s) [upload=4769]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 04:51:09.109709627 (6h0m15.482288982s) [upload=4769]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 04:51:09.556145753 (6h0m15.928725108s) [seg_pos={0 0}, upload=4769]: Begin upload segment... Jul 19 04:51:09.556194817 (6h0m15.928774172s) [seg_pos={0 0}, upload=4769]: Done begin upload segment. Jul 19 04:51:09.556223893 (6h0m15.928803248s) [noise=true , address=79.127.219.38:4023 , piece_id=PTPOZGLYV4MZTOFZNN3NCGYL7DFAST4PXKEUIVTMFL7HV4GZTCLQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556252197 (6h0m15.928831552s) [noise=true , address=45.91.95.4:4011 , piece_id=ZMBAS6KKHYVIRE6OBL2NFVCWMM4TKQ7J64BD3SYXIK54XCEDI2XA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556579383 (6h0m15.929158738s) [noise=true , address=45.91.116.6:4007 , piece_id=A2AFVTZBXSGBOE7XIM3B5HRN4STITFBXO6MZRJ365URNAAOZ3PSQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556854469 (6h0m15.929433824s) [noise=true , address=104.36.82.19:4423 , piece_id=PIJUHDUND36HQL67I33D7A3EULMJHS7N55MN4OFQJ4OWNQ42O47A, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557094107 (6h0m15.929673462s) [noise=true , address=45.91.116.9:4264 , piece_id=GHM5BKZKMT2QOHHBOS4FSPOWEWSFQRGRVVR4GSNDPHVIFRZHQL2A, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557336381 (6h0m15.929915736s) [noise=true , address=79.127.205.235:4660 , piece_id=UOUOGE2VP7PX7EMRZLPMCWFCYEZERM3DQMCQ6NG4J2K2DEIZR63Q, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557557885 (6h0m15.93013724s) [noise=true , address=79.127.205.246:4280 , piece_id=FLBOZI3UGTQXAQBLZ2F45Z3OEYL65F5PQBXRF2DQMZ7NGF7VDRPA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557762296 (6h0m15.930341651s) [noise=true , address=79.127.205.233:4622 , piece_id=6WWNYQTH5MGAJOEB7BXKZDJOMCAZXDW66J3OEANGDNJDNUMLZM7Q, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558014689 (6h0m15.930594044s) [noise=true , address=64.49.13.138:4482 , piece_id=MAW2M4US2OQNOS3WI67AA7NIBTJDE5T2JQTXKFPBLGVGXKVCFGTQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558215974 (6h0m15.930795319s) [noise=true , address=77.90.31.10:4546 , piece_id=R6LCA5WZOEBDURL5S3MEK22YPVYXX6EXSS4J4W57543HELYXOKZA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558409364 (6h0m15.930988719s) [noise=true , address=45.91.95.5:4019 , piece_id=YTSBCN2LPHEMRRAZIHJGVB7NBOTRXVEHD2TWUCTTLBNFFYEWWDKA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558623504 (6h0m15.931202859s) [noise=true , address=79.127.201.215:4357 , piece_id=75XWAG2VQWEYCWN2Q3WRIZSBI7LDOISZJPXQLQ2GIPNEN34VLSUQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558805162 (6h0m15.931384517s) [noise=true , address=45.91.116.14:4219 , piece_id=PDW6JPRHCVMHHOWIHT6JKV4CTSC3JY6NDKDVFZSO47OYNPZ7PGGA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558999373 (6h0m15.931578728s) [noise=true , address=77.90.31.11:4564 , piece_id=L2U7JFJRDQOFA2AYVAGTI4ZW4VABCXKMXWBIUWQO2QNNUUBJ74BA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559184768 (6h0m15.931764123s) [noise=true , address=77.90.31.16:4768 , piece_id=YNLSONFM6COZEVF6ZNQR2EA6V4LSQFZJONIXR4AFTNNMFGPV57JA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559368719 (6h0m15.931948074s) [noise=true , address=79.127.220.98:4432 , piece_id=2SUTFQ7FVDPAM2H4W3OLW6AVH3VJ33IJFE4L52SA3ET5LGBMOMPQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559549706 (6h0m15.932129061s) [noise=true , address=77.90.31.18:4813 , piece_id=VDOT7BB4O7XYJHW6BFTAYQZZYLZU5PAUXVJGSXJ7FDI4TH425COA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559727836 (6h0m15.932307191s) [noise=true , address=79.127.219.51:4211 , piece_id=HO5PLWBVU5YAYNYMN352NA73SM4WBJV4WZ7XLSQTCQ76UAL2TOJA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:10.663359087 (6h0m17.035938442s) [upload=4769]: Next returned nil segment Jul 19 04:51:10.663378664 (6h0m17.035958019s) [upload=4769]: Waiting for error group managing segments... Jul 19 04:51:10.962945479 (6h0m17.335524834s) [noise=true , address=79.127.205.233:4622 , piece_id=6WWNYQTH5MGAJOEB7BXKZDJOMCAZXDW66J3OEANGDNJDNUMLZM7Q, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:10.967689807 (6h0m17.340269152s) [noise=true , address=79.127.205.235:4660 , piece_id=UOUOGE2VP7PX7EMRZLPMCWFCYEZERM3DQMCQ6NG4J2K2DEIZR63Q, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:10.976968631 (6h0m17.349547986s) [noise=true , address=64.49.13.138:4482 , piece_id=MAW2M4US2OQNOS3WI67AA7NIBTJDE5T2JQTXKFPBLGVGXKVCFGTQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.048750700 (6h0m17.421330055s) [noise=true , address=79.127.205.246:4280 , piece_id=FLBOZI3UGTQXAQBLZ2F45Z3OEYL65F5PQBXRF2DQMZ7NGF7VDRPA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.221642403 (6h0m17.594221758s) [noise=true , address=79.127.220.98:4432 , piece_id=2SUTFQ7FVDPAM2H4W3OLW6AVH3VJ33IJFE4L52SA3ET5LGBMOMPQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.317070435 (6h0m17.68964979s) [noise=true , address=45.91.95.4:4011 , piece_id=ZMBAS6KKHYVIRE6OBL2NFVCWMM4TKQ7J64BD3SYXIK54XCEDI2XA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.324002660 (6h0m17.696582015s) [noise=true , address=45.91.95.5:4019 , piece_id=YTSBCN2LPHEMRRAZIHJGVB7NBOTRXVEHD2TWUCTTLBNFFYEWWDKA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.460976317 (6h0m17.833555672s) [noise=true , address=79.127.201.215:4357 , piece_id=75XWAG2VQWEYCWN2Q3WRIZSBI7LDOISZJPXQLQ2GIPNEN34VLSUQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.462584603 (6h0m17.835163958s) [noise=true , address=45.91.116.6:4007 , piece_id=A2AFVTZBXSGBOE7XIM3B5HRN4STITFBXO6MZRJ365URNAAOZ3PSQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.490392973 (6h0m17.862972328s) [noise=true , address=45.91.116.9:4264 , piece_id=GHM5BKZKMT2QOHHBOS4FSPOWEWSFQRGRVVR4GSNDPHVIFRZHQL2A, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.502040780 (6h0m17.874620135s) [noise=true , address=45.91.116.14:4219 , piece_id=PDW6JPRHCVMHHOWIHT6JKV4CTSC3JY6NDKDVFZSO47OYNPZ7PGGA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.600434096 (6h0m17.973013451s) [noise=true , address=79.127.219.51:4211 , piece_id=HO5PLWBVU5YAYNYMN352NA73SM4WBJV4WZ7XLSQTCQ76UAL2TOJA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.609008984 (6h0m17.981588339s) [noise=true , address=104.36.82.19:4423 , piece_id=PIJUHDUND36HQL67I33D7A3EULMJHS7N55MN4OFQJ4OWNQ42O47A, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.620825273 (6h0m17.993404628s) [noise=true , address=79.127.219.38:4023 , piece_id=PTPOZGLYV4MZTOFZNN3NCGYL7DFAST4PXKEUIVTMFL7HV4GZTCLQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.620838659 (6h0m17.993418014s) [seg_pos={0 0}, upload=4769]: Segment reached optimal threshold of 14 pieces. Jul 19 04:51:11.621067807 (6h0m17.993647163s) [noise=true , address=77.90.31.11:4564 , piece_id=L2U7JFJRDQOFA2AYVAGTI4ZW4VABCXKMXWBIUWQO2QNNUUBJ74BA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection; write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection Jul 19 04:51:11.621134234 (6h0m17.993713589s) [noise=true , address=77.90.31.10:4546 , piece_id=R6LCA5WZOEBDURL5S3MEK22YPVYXX6EXSS4J4W57543HELYXOKZA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK): protocol: read tcp 188.166.225.170:58476->77.90.31.10:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621174191 (6h0m17.993753546s) [noise=true , address=77.90.31.16:4768 , piece_id=YNLSONFM6COZEVF6ZNQR2EA6V4LSQFZJONIXR4AFTNNMFGPV57JA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UchewmtxBrtSZSpVAJFdZmcTsQVQxkjK9re6cgUGqDhyonQeF): protocol: read tcp 188.166.225.170:56326->77.90.31.16:4768: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621216842 (6h0m17.993796197s) [noise=true , address=77.90.31.18:4813 , piece_id=VDOT7BB4O7XYJHW6BFTAYQZZYLZU5PAUXVJGSXJ7FDI4TH425COA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128WthA1x9wp1dV1vkpGnoveZQpU9CSgDCZfw5GMQuVpCNSVAM5): protocol: read tcp 188.166.225.170:49704->77.90.31.18:4813: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621234917 (6h0m17.993814272s) [seg_pos={0 0}, upload=4769]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 19 04:51:11.621301444 (6h0m17.993880799s) [upload=4769]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 19 04:51:11.621305402 (6h0m17.993884758s) [upload=4769]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 19 04:51:11.923136970 (6h0m18.295716325s) [upload=4769]: Done uploading segments... Jul 19 04:51:11.923156627 (6h0m18.295735972s) [upload=4769]: Upload finished. err: <nil> Duration: 6.614363913s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [................XX...X...X....X..X....XXXXX.X.X.....XX.X....] duration: 6.943225083s FAIL (2025-07-19 05:18:30 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== gment. successful: 13 optimal: 13 errs: <nil> Jul 19 05:14:35.936262577 (6h23m42.308841932s) [upload=5075]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 19 05:14:35.936281884 (6h23m42.308861239s) [upload=5075]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 19 05:14:36.234171969 (6h23m42.606751314s) [upload=5075]: Done uploading segments... Jul 19 05:14:36.234192037 (6h23m42.606771382s) [upload=5075]: Upload finished. err: <nil> Jul 19 05:14:55.587986728 (6h24m1.960566093s) [upload=5081]: Starting upload Jul 19 05:14:55.588051823 (6h24m1.960631178s) [upload=5081]: Done starting upload Jul 19 05:14:55.588146724 (6h24m1.960726079s) [upload=5081]: Uploading segments... Jul 19 05:14:55.588391873 (6h24m1.960971228s) [upload=5081]: Got next segment. Inline: false Jul 19 05:14:55.588649837 (6h24m1.961229193s) [upload=5081]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 05:14:55.588655057 (6h24m1.961234402s) [upload=5081]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 05:14:55.823387692 (6h24m2.195967047s) [seg_pos={0 0}, upload=5081]: Begin upload segment... Jul 19 05:14:55.823450572 (6h24m2.196029927s) [seg_pos={0 0}, upload=5081]: Done begin upload segment. Jul 19 05:14:55.823481952 (6h24m2.196061307s) [noise=true , address=77.90.31.17:4772 , piece_id=B544E3F2UC3UVE2ITUQAWGVDMSPS2HFIVHXZ2OLWB3T25AHMFAVA, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.823491401 (6h24m2.196070756s) [noise=true , address=45.91.116.10:4082 , piece_id=HZDMC4J5O5UHVDH5SOQBKAFPN6RJUHIWKEZ4OPVMW2HSN7NDZMZA, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.823923789 (6h24m2.196503144s) [noise=true , address=45.91.116.9:4262 , piece_id=FRIATDM7LPPHXYAMUISAYRTQWAKSTB4EYUANHGEMQHM6JQADDTRQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.823949737 (6h24m2.196529092s) [noise=true , address=45.91.116.12:4227 , piece_id=QB7YKZFXWMRB3VQ3HSFMMMLLX5YFWOXG4Y4GJGGYT6HDV4FQ65VA, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.824310158 (6h24m2.196889513s) [noise=true , address=45.91.116.4:4937 , piece_id=V2IOOA7DVCGZBH3D43ROYHT6ZECPY6GTC5M7NFO2RWPQB6EYGALA, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.824403045 (6h24m2.19698239s) [noise=true , address=77.90.31.11:4555 , piece_id=YICA5IVHSCC3RPC5FRKD26BSMY6N5WJQA5LCABJSRJYL6BZJYL6A, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.824651210 (6h24m2.197230565s) [noise=true , address=77.90.31.13:4627 , piece_id=E7GSSBZ653K2AA4TP6A4BRT5BDZCBODNRAZM3P6SYJXJFQU2BACQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.824716334 (6h24m2.197295689s) [noise=true , address=77.90.31.16:4734 , piece_id=YUW6MWSUDNLAC2VHLBT6H4T27YTNJ3RDK4BSGE7OZQCO3F2LBH4A, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.824960491 (6h24m2.197539856s) [noise=true , address=45.91.116.7:4010 , piece_id=BH76RM4NXGEAI2J2424ALK4PFSXUTBCYOH777BUOIDW7BW2IJUJA, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.824977494 (6h24m2.197556849s) [noise=true , address=45.91.116.14:4190 , piece_id=RUU2MICSAWOECAAYNOBS75SE2YKSUBMSSFKLPON7FWK77UNJEYGQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.825213285 (6h24m2.19779264s) [noise=true , address=45.91.116.8:4047 , piece_id=6UOFOV4VCVT6DB7D7Q23REAX7VZPAW5PSG5PLA54LISOUWMPQBBQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.825271547 (6h24m2.197850902s) [noise=true , address=77.90.31.18:4806 , piece_id=67PXFJMJB6XWNU6JMOKPO4BXKDD2Z225ZKTB7DZEA46OMSSXHHMQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.825445860 (6h24m2.198025215s) [noise=true , address=45.91.116.7:4009 , piece_id=HH5I32ZYT5LVXHDLBVSAVX6T23MJ6RGNPNAWPIQAREJ4ELPFJGYA, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.825568244 (6h24m2.198147599s) [noise=true , address=77.90.31.15:4697 , piece_id=W62ARDAVLI64P6VYSPVFRTDKB5H2WXTSCF2VKP3UN7JKGBCUEZTQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.825902985 (6h24m2.19848234s) [noise=true , address=45.91.116.5:4299 , piece_id=JQLK5LIMCMW6R2UZEJZSG6QAP2M65XBVE6JFEXY5KDWBVNDG2VCQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.826216495 (6h24m2.19879585s) [noise=true , address=77.90.31.14:4663 , piece_id=LNTFUWB6LUUFD2EB536ZHBSNVVXYZPOOB4DUZB2M6BXSSFEL3UMQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:58.633690556 (6h24m5.006269911s) [upload=5081]: Next returned nil segment Jul 19 05:14:58.633708100 (6h24m5.006287445s) [upload=5081]: Waiting for error group managing segments... Jul 19 05:14:58.915966948 (6h24m5.288546293s) [noise=true , address=45.91.116.7:4009 , piece_id=HH5I32ZYT5LVXHDLBVSAVX6T23MJ6RGNPNAWPIQAREJ4ELPFJGYA, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:58.916093541 (6h24m5.288672896s) [noise=true , address=45.91.116.12:4227 , piece_id=QB7YKZFXWMRB3VQ3HSFMMMLLX5YFWOXG4Y4GJGGYT6HDV4FQ65VA, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:58.923615462 (6h24m5.296194817s) [noise=true , address=45.91.116.14:4190 , piece_id=RUU2MICSAWOECAAYNOBS75SE2YKSUBMSSFKLPON7FWK77UNJEYGQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:58.928612934 (6h24m5.301192289s) [noise=true , address=45.91.116.7:4010 , piece_id=BH76RM4NXGEAI2J2424ALK4PFSXUTBCYOH777BUOIDW7BW2IJUJA, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:58.990915468 (6h24m5.363494833s) [noise=true , address=45.91.116.4:4937 , piece_id=V2IOOA7DVCGZBH3D43ROYHT6ZECPY6GTC5M7NFO2RWPQB6EYGALA, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:58.992136464 (6h24m5.364715819s) [noise=true , address=45.91.116.8:4047 , piece_id=6UOFOV4VCVT6DB7D7Q23REAX7VZPAW5PSG5PLA54LISOUWMPQBBQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:58.997711812 (6h24m5.370291157s) [noise=true , address=45.91.116.9:4262 , piece_id=FRIATDM7LPPHXYAMUISAYRTQWAKSTB4EYUANHGEMQHM6JQADDTRQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:59.010548391 (6h24m5.383127746s) [noise=true , address=45.91.116.10:4082 , piece_id=HZDMC4J5O5UHVDH5SOQBKAFPN6RJUHIWKEZ4OPVMW2HSN7NDZMZA, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:59.039607621 (6h24m5.412186996s) [noise=true , address=45.91.116.5:4299 , piece_id=JQLK5LIMCMW6R2UZEJZSG6QAP2M65XBVE6JFEXY5KDWBVNDG2VCQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:15:19.998482215 (6h24m26.37106157s) [noise=true , address=77.90.31.14:4663 , piece_id=LNTFUWB6LUUFD2EB536ZHBSNVVXYZPOOB4DUZB2M6BXSSFEL3UMQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:15:21.170089451 (6h24m27.542668806s) [noise=true , address=77.90.31.13:4627 , piece_id=E7GSSBZ653K2AA4TP6A4BRT5BDZCBODNRAZM3P6SYJXJFQU2BACQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:15:24.941469148 (6h24m31.314048503s) [noise=true , address=77.90.31.15:4697 , piece_id=W62ARDAVLI64P6VYSPVFRTDKB5H2WXTSCF2VKP3UN7JKGBCUEZTQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: expected piece hash; context deadline exceeded; EOF Jul 19 05:15:24.941530766 (6h24m31.314110121s) [noise=true , address=77.90.31.17:4772 , piece_id=B544E3F2UC3UVE2ITUQAWGVDMSPS2HFIVHXZ2OLWB3T25AHMFAVA, seg_pos={0 0}, upload=5081]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:44456->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:44456->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44456->77.90.31.17:4772: use of closed network connection Jul 19 05:15:24.941543591 (6h24m31.314122946s) [noise=true , address=77.90.31.16:4734 , piece_id=YUW6MWSUDNLAC2VHLBT6H4T27YTNJ3RDK4BSGE7OZQCO3F2LBH4A, seg_pos={0 0}, upload=5081]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Jul 19 05:15:24.941614786 (6h24m31.314194141s) [noise=true , address=77.90.31.11:4555 , piece_id=YICA5IVHSCC3RPC5FRKD26BSMY6N5WJQA5LCABJSRJYL6BZJYL6A, seg_pos={0 0}, upload=5081]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: write tcp 188.166.225.170:37392->77.90.31.11:4555: use of closed network connection; write tcp 188.166.225.170:37392->77.90.31.11:4555: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37392->77.90.31.11:4555: use of closed network connection Jul 19 05:15:24.941641888 (6h24m31.314221243s) [noise=true , address=77.90.31.18:4806 , piece_id=67PXFJMJB6XWNU6JMOKPO4BXKDD2Z225ZKTB7DZEA46OMSSXHHMQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:47114->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:47114->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47114->77.90.31.18:4806: use of closed network connection Jul 19 05:15:24.941663099 (6h24m31.314242454s) [seg_pos={0 0}, upload=5081]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 19 05:15:25.179949476 (6h24m31.552528831s) [upload=5081]: Done uploading segments... Jul 19 05:15:25.179975797 (6h24m31.552555152s) [upload=5081]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.943256343s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 5.695963941s FAIL (2025-07-19 05:18:10 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 4855s) [upload=4769]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 04:51:09.109709627 (6h0m15.482288982s) [upload=4769]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 04:51:09.556145753 (6h0m15.928725108s) [seg_pos={0 0}, upload=4769]: Begin upload segment... Jul 19 04:51:09.556194817 (6h0m15.928774172s) [seg_pos={0 0}, upload=4769]: Done begin upload segment. Jul 19 04:51:09.556223893 (6h0m15.928803248s) [noise=true , address=79.127.219.38:4023 , piece_id=PTPOZGLYV4MZTOFZNN3NCGYL7DFAST4PXKEUIVTMFL7HV4GZTCLQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556252197 (6h0m15.928831552s) [noise=true , address=45.91.95.4:4011 , piece_id=ZMBAS6KKHYVIRE6OBL2NFVCWMM4TKQ7J64BD3SYXIK54XCEDI2XA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556579383 (6h0m15.929158738s) [noise=true , address=45.91.116.6:4007 , piece_id=A2AFVTZBXSGBOE7XIM3B5HRN4STITFBXO6MZRJ365URNAAOZ3PSQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556854469 (6h0m15.929433824s) [noise=true , address=104.36.82.19:4423 , piece_id=PIJUHDUND36HQL67I33D7A3EULMJHS7N55MN4OFQJ4OWNQ42O47A, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557094107 (6h0m15.929673462s) [noise=true , address=45.91.116.9:4264 , piece_id=GHM5BKZKMT2QOHHBOS4FSPOWEWSFQRGRVVR4GSNDPHVIFRZHQL2A, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557336381 (6h0m15.929915736s) [noise=true , address=79.127.205.235:4660 , piece_id=UOUOGE2VP7PX7EMRZLPMCWFCYEZERM3DQMCQ6NG4J2K2DEIZR63Q, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557557885 (6h0m15.93013724s) [noise=true , address=79.127.205.246:4280 , piece_id=FLBOZI3UGTQXAQBLZ2F45Z3OEYL65F5PQBXRF2DQMZ7NGF7VDRPA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557762296 (6h0m15.930341651s) [noise=true , address=79.127.205.233:4622 , piece_id=6WWNYQTH5MGAJOEB7BXKZDJOMCAZXDW66J3OEANGDNJDNUMLZM7Q, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558014689 (6h0m15.930594044s) [noise=true , address=64.49.13.138:4482 , piece_id=MAW2M4US2OQNOS3WI67AA7NIBTJDE5T2JQTXKFPBLGVGXKVCFGTQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558215974 (6h0m15.930795319s) [noise=true , address=77.90.31.10:4546 , piece_id=R6LCA5WZOEBDURL5S3MEK22YPVYXX6EXSS4J4W57543HELYXOKZA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558409364 (6h0m15.930988719s) [noise=true , address=45.91.95.5:4019 , piece_id=YTSBCN2LPHEMRRAZIHJGVB7NBOTRXVEHD2TWUCTTLBNFFYEWWDKA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558623504 (6h0m15.931202859s) [noise=true , address=79.127.201.215:4357 , piece_id=75XWAG2VQWEYCWN2Q3WRIZSBI7LDOISZJPXQLQ2GIPNEN34VLSUQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558805162 (6h0m15.931384517s) [noise=true , address=45.91.116.14:4219 , piece_id=PDW6JPRHCVMHHOWIHT6JKV4CTSC3JY6NDKDVFZSO47OYNPZ7PGGA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558999373 (6h0m15.931578728s) [noise=true , address=77.90.31.11:4564 , piece_id=L2U7JFJRDQOFA2AYVAGTI4ZW4VABCXKMXWBIUWQO2QNNUUBJ74BA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559184768 (6h0m15.931764123s) [noise=true , address=77.90.31.16:4768 , piece_id=YNLSONFM6COZEVF6ZNQR2EA6V4LSQFZJONIXR4AFTNNMFGPV57JA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559368719 (6h0m15.931948074s) [noise=true , address=79.127.220.98:4432 , piece_id=2SUTFQ7FVDPAM2H4W3OLW6AVH3VJ33IJFE4L52SA3ET5LGBMOMPQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559549706 (6h0m15.932129061s) [noise=true , address=77.90.31.18:4813 , piece_id=VDOT7BB4O7XYJHW6BFTAYQZZYLZU5PAUXVJGSXJ7FDI4TH425COA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559727836 (6h0m15.932307191s) [noise=true , address=79.127.219.51:4211 , piece_id=HO5PLWBVU5YAYNYMN352NA73SM4WBJV4WZ7XLSQTCQ76UAL2TOJA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:10.663359087 (6h0m17.035938442s) [upload=4769]: Next returned nil segment Jul 19 04:51:10.663378664 (6h0m17.035958019s) [upload=4769]: Waiting for error group managing segments... Jul 19 04:51:10.962945479 (6h0m17.335524834s) [noise=true , address=79.127.205.233:4622 , piece_id=6WWNYQTH5MGAJOEB7BXKZDJOMCAZXDW66J3OEANGDNJDNUMLZM7Q, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:10.967689807 (6h0m17.340269152s) [noise=true , address=79.127.205.235:4660 , piece_id=UOUOGE2VP7PX7EMRZLPMCWFCYEZERM3DQMCQ6NG4J2K2DEIZR63Q, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:10.976968631 (6h0m17.349547986s) [noise=true , address=64.49.13.138:4482 , piece_id=MAW2M4US2OQNOS3WI67AA7NIBTJDE5T2JQTXKFPBLGVGXKVCFGTQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.048750700 (6h0m17.421330055s) [noise=true , address=79.127.205.246:4280 , piece_id=FLBOZI3UGTQXAQBLZ2F45Z3OEYL65F5PQBXRF2DQMZ7NGF7VDRPA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.221642403 (6h0m17.594221758s) [noise=true , address=79.127.220.98:4432 , piece_id=2SUTFQ7FVDPAM2H4W3OLW6AVH3VJ33IJFE4L52SA3ET5LGBMOMPQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.317070435 (6h0m17.68964979s) [noise=true , address=45.91.95.4:4011 , piece_id=ZMBAS6KKHYVIRE6OBL2NFVCWMM4TKQ7J64BD3SYXIK54XCEDI2XA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.324002660 (6h0m17.696582015s) [noise=true , address=45.91.95.5:4019 , piece_id=YTSBCN2LPHEMRRAZIHJGVB7NBOTRXVEHD2TWUCTTLBNFFYEWWDKA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.460976317 (6h0m17.833555672s) [noise=true , address=79.127.201.215:4357 , piece_id=75XWAG2VQWEYCWN2Q3WRIZSBI7LDOISZJPXQLQ2GIPNEN34VLSUQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.462584603 (6h0m17.835163958s) [noise=true , address=45.91.116.6:4007 , piece_id=A2AFVTZBXSGBOE7XIM3B5HRN4STITFBXO6MZRJ365URNAAOZ3PSQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.490392973 (6h0m17.862972328s) [noise=true , address=45.91.116.9:4264 , piece_id=GHM5BKZKMT2QOHHBOS4FSPOWEWSFQRGRVVR4GSNDPHVIFRZHQL2A, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.502040780 (6h0m17.874620135s) [noise=true , address=45.91.116.14:4219 , piece_id=PDW6JPRHCVMHHOWIHT6JKV4CTSC3JY6NDKDVFZSO47OYNPZ7PGGA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.600434096 (6h0m17.973013451s) [noise=true , address=79.127.219.51:4211 , piece_id=HO5PLWBVU5YAYNYMN352NA73SM4WBJV4WZ7XLSQTCQ76UAL2TOJA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.609008984 (6h0m17.981588339s) [noise=true , address=104.36.82.19:4423 , piece_id=PIJUHDUND36HQL67I33D7A3EULMJHS7N55MN4OFQJ4OWNQ42O47A, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.620825273 (6h0m17.993404628s) [noise=true , address=79.127.219.38:4023 , piece_id=PTPOZGLYV4MZTOFZNN3NCGYL7DFAST4PXKEUIVTMFL7HV4GZTCLQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.620838659 (6h0m17.993418014s) [seg_pos={0 0}, upload=4769]: Segment reached optimal threshold of 14 pieces. Jul 19 04:51:11.621067807 (6h0m17.993647163s) [noise=true , address=77.90.31.11:4564 , piece_id=L2U7JFJRDQOFA2AYVAGTI4ZW4VABCXKMXWBIUWQO2QNNUUBJ74BA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection; write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection Jul 19 04:51:11.621134234 (6h0m17.993713589s) [noise=true , address=77.90.31.10:4546 , piece_id=R6LCA5WZOEBDURL5S3MEK22YPVYXX6EXSS4J4W57543HELYXOKZA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK): protocol: read tcp 188.166.225.170:58476->77.90.31.10:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621174191 (6h0m17.993753546s) [noise=true , address=77.90.31.16:4768 , piece_id=YNLSONFM6COZEVF6ZNQR2EA6V4LSQFZJONIXR4AFTNNMFGPV57JA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UchewmtxBrtSZSpVAJFdZmcTsQVQxkjK9re6cgUGqDhyonQeF): protocol: read tcp 188.166.225.170:56326->77.90.31.16:4768: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621216842 (6h0m17.993796197s) [noise=true , address=77.90.31.18:4813 , piece_id=VDOT7BB4O7XYJHW6BFTAYQZZYLZU5PAUXVJGSXJ7FDI4TH425COA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128WthA1x9wp1dV1vkpGnoveZQpU9CSgDCZfw5GMQuVpCNSVAM5): protocol: read tcp 188.166.225.170:49704->77.90.31.18:4813: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621234917 (6h0m17.993814272s) [seg_pos={0 0}, upload=4769]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 19 04:51:11.621301444 (6h0m17.993880799s) [upload=4769]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 19 04:51:11.621305402 (6h0m17.993884758s) [upload=4769]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 19 04:51:11.923136970 (6h0m18.295716325s) [upload=4769]: Done uploading segments... Jul 19 04:51:11.923156627 (6h0m18.295735972s) [upload=4769]: Upload finished. err: <nil> Duration: 5.695988086s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [.................XX...X...X....X..X....XXXXX.X.X.....XX.X...] duration: 5.861284015s FAIL (2025-07-19 05:17:15 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== gment. successful: 13 optimal: 13 errs: <nil> Jul 19 05:14:35.936262577 (6h23m42.308841932s) [upload=5075]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 19 05:14:35.936281884 (6h23m42.308861239s) [upload=5075]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 19 05:14:36.234171969 (6h23m42.606751314s) [upload=5075]: Done uploading segments... Jul 19 05:14:36.234192037 (6h23m42.606771382s) [upload=5075]: Upload finished. err: <nil> Jul 19 05:14:55.587986728 (6h24m1.960566093s) [upload=5081]: Starting upload Jul 19 05:14:55.588051823 (6h24m1.960631178s) [upload=5081]: Done starting upload Jul 19 05:14:55.588146724 (6h24m1.960726079s) [upload=5081]: Uploading segments... Jul 19 05:14:55.588391873 (6h24m1.960971228s) [upload=5081]: Got next segment. Inline: false Jul 19 05:14:55.588649837 (6h24m1.961229193s) [upload=5081]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 05:14:55.588655057 (6h24m1.961234402s) [upload=5081]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 05:14:55.823387692 (6h24m2.195967047s) [seg_pos={0 0}, upload=5081]: Begin upload segment... Jul 19 05:14:55.823450572 (6h24m2.196029927s) [seg_pos={0 0}, upload=5081]: Done begin upload segment. Jul 19 05:14:55.823481952 (6h24m2.196061307s) [noise=true , address=77.90.31.17:4772 , piece_id=B544E3F2UC3UVE2ITUQAWGVDMSPS2HFIVHXZ2OLWB3T25AHMFAVA, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.823491401 (6h24m2.196070756s) [noise=true , address=45.91.116.10:4082 , piece_id=HZDMC4J5O5UHVDH5SOQBKAFPN6RJUHIWKEZ4OPVMW2HSN7NDZMZA, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.823923789 (6h24m2.196503144s) [noise=true , address=45.91.116.9:4262 , piece_id=FRIATDM7LPPHXYAMUISAYRTQWAKSTB4EYUANHGEMQHM6JQADDTRQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.823949737 (6h24m2.196529092s) [noise=true , address=45.91.116.12:4227 , piece_id=QB7YKZFXWMRB3VQ3HSFMMMLLX5YFWOXG4Y4GJGGYT6HDV4FQ65VA, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.824310158 (6h24m2.196889513s) [noise=true , address=45.91.116.4:4937 , piece_id=V2IOOA7DVCGZBH3D43ROYHT6ZECPY6GTC5M7NFO2RWPQB6EYGALA, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.824403045 (6h24m2.19698239s) [noise=true , address=77.90.31.11:4555 , piece_id=YICA5IVHSCC3RPC5FRKD26BSMY6N5WJQA5LCABJSRJYL6BZJYL6A, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.824651210 (6h24m2.197230565s) [noise=true , address=77.90.31.13:4627 , piece_id=E7GSSBZ653K2AA4TP6A4BRT5BDZCBODNRAZM3P6SYJXJFQU2BACQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.824716334 (6h24m2.197295689s) [noise=true , address=77.90.31.16:4734 , piece_id=YUW6MWSUDNLAC2VHLBT6H4T27YTNJ3RDK4BSGE7OZQCO3F2LBH4A, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.824960491 (6h24m2.197539856s) [noise=true , address=45.91.116.7:4010 , piece_id=BH76RM4NXGEAI2J2424ALK4PFSXUTBCYOH777BUOIDW7BW2IJUJA, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.824977494 (6h24m2.197556849s) [noise=true , address=45.91.116.14:4190 , piece_id=RUU2MICSAWOECAAYNOBS75SE2YKSUBMSSFKLPON7FWK77UNJEYGQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.825213285 (6h24m2.19779264s) [noise=true , address=45.91.116.8:4047 , piece_id=6UOFOV4VCVT6DB7D7Q23REAX7VZPAW5PSG5PLA54LISOUWMPQBBQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.825271547 (6h24m2.197850902s) [noise=true , address=77.90.31.18:4806 , piece_id=67PXFJMJB6XWNU6JMOKPO4BXKDD2Z225ZKTB7DZEA46OMSSXHHMQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.825445860 (6h24m2.198025215s) [noise=true , address=45.91.116.7:4009 , piece_id=HH5I32ZYT5LVXHDLBVSAVX6T23MJ6RGNPNAWPIQAREJ4ELPFJGYA, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.825568244 (6h24m2.198147599s) [noise=true , address=77.90.31.15:4697 , piece_id=W62ARDAVLI64P6VYSPVFRTDKB5H2WXTSCF2VKP3UN7JKGBCUEZTQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.825902985 (6h24m2.19848234s) [noise=true , address=45.91.116.5:4299 , piece_id=JQLK5LIMCMW6R2UZEJZSG6QAP2M65XBVE6JFEXY5KDWBVNDG2VCQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.826216495 (6h24m2.19879585s) [noise=true , address=77.90.31.14:4663 , piece_id=LNTFUWB6LUUFD2EB536ZHBSNVVXYZPOOB4DUZB2M6BXSSFEL3UMQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:58.633690556 (6h24m5.006269911s) [upload=5081]: Next returned nil segment Jul 19 05:14:58.633708100 (6h24m5.006287445s) [upload=5081]: Waiting for error group managing segments... Jul 19 05:14:58.915966948 (6h24m5.288546293s) [noise=true , address=45.91.116.7:4009 , piece_id=HH5I32ZYT5LVXHDLBVSAVX6T23MJ6RGNPNAWPIQAREJ4ELPFJGYA, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:58.916093541 (6h24m5.288672896s) [noise=true , address=45.91.116.12:4227 , piece_id=QB7YKZFXWMRB3VQ3HSFMMMLLX5YFWOXG4Y4GJGGYT6HDV4FQ65VA, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:58.923615462 (6h24m5.296194817s) [noise=true , address=45.91.116.14:4190 , piece_id=RUU2MICSAWOECAAYNOBS75SE2YKSUBMSSFKLPON7FWK77UNJEYGQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:58.928612934 (6h24m5.301192289s) [noise=true , address=45.91.116.7:4010 , piece_id=BH76RM4NXGEAI2J2424ALK4PFSXUTBCYOH777BUOIDW7BW2IJUJA, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:58.990915468 (6h24m5.363494833s) [noise=true , address=45.91.116.4:4937 , piece_id=V2IOOA7DVCGZBH3D43ROYHT6ZECPY6GTC5M7NFO2RWPQB6EYGALA, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:58.992136464 (6h24m5.364715819s) [noise=true , address=45.91.116.8:4047 , piece_id=6UOFOV4VCVT6DB7D7Q23REAX7VZPAW5PSG5PLA54LISOUWMPQBBQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:58.997711812 (6h24m5.370291157s) [noise=true , address=45.91.116.9:4262 , piece_id=FRIATDM7LPPHXYAMUISAYRTQWAKSTB4EYUANHGEMQHM6JQADDTRQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:59.010548391 (6h24m5.383127746s) [noise=true , address=45.91.116.10:4082 , piece_id=HZDMC4J5O5UHVDH5SOQBKAFPN6RJUHIWKEZ4OPVMW2HSN7NDZMZA, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:59.039607621 (6h24m5.412186996s) [noise=true , address=45.91.116.5:4299 , piece_id=JQLK5LIMCMW6R2UZEJZSG6QAP2M65XBVE6JFEXY5KDWBVNDG2VCQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:15:19.998482215 (6h24m26.37106157s) [noise=true , address=77.90.31.14:4663 , piece_id=LNTFUWB6LUUFD2EB536ZHBSNVVXYZPOOB4DUZB2M6BXSSFEL3UMQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:15:21.170089451 (6h24m27.542668806s) [noise=true , address=77.90.31.13:4627 , piece_id=E7GSSBZ653K2AA4TP6A4BRT5BDZCBODNRAZM3P6SYJXJFQU2BACQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:15:24.941469148 (6h24m31.314048503s) [noise=true , address=77.90.31.15:4697 , piece_id=W62ARDAVLI64P6VYSPVFRTDKB5H2WXTSCF2VKP3UN7JKGBCUEZTQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: expected piece hash; context deadline exceeded; EOF Jul 19 05:15:24.941530766 (6h24m31.314110121s) [noise=true , address=77.90.31.17:4772 , piece_id=B544E3F2UC3UVE2ITUQAWGVDMSPS2HFIVHXZ2OLWB3T25AHMFAVA, seg_pos={0 0}, upload=5081]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:44456->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:44456->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44456->77.90.31.17:4772: use of closed network connection Jul 19 05:15:24.941543591 (6h24m31.314122946s) [noise=true , address=77.90.31.16:4734 , piece_id=YUW6MWSUDNLAC2VHLBT6H4T27YTNJ3RDK4BSGE7OZQCO3F2LBH4A, seg_pos={0 0}, upload=5081]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Jul 19 05:15:24.941614786 (6h24m31.314194141s) [noise=true , address=77.90.31.11:4555 , piece_id=YICA5IVHSCC3RPC5FRKD26BSMY6N5WJQA5LCABJSRJYL6BZJYL6A, seg_pos={0 0}, upload=5081]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: write tcp 188.166.225.170:37392->77.90.31.11:4555: use of closed network connection; write tcp 188.166.225.170:37392->77.90.31.11:4555: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37392->77.90.31.11:4555: use of closed network connection Jul 19 05:15:24.941641888 (6h24m31.314221243s) [noise=true , address=77.90.31.18:4806 , piece_id=67PXFJMJB6XWNU6JMOKPO4BXKDD2Z225ZKTB7DZEA46OMSSXHHMQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:47114->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:47114->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47114->77.90.31.18:4806: use of closed network connection Jul 19 05:15:24.941663099 (6h24m31.314242454s) [seg_pos={0 0}, upload=5081]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 19 05:15:25.179949476 (6h24m31.552528831s) [upload=5081]: Done uploading segments... Jul 19 05:15:25.179975797 (6h24m31.552555152s) [upload=5081]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.861307069s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 5.611207288s FAIL (2025-07-19 05:16:55 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 4855s) [upload=4769]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 04:51:09.109709627 (6h0m15.482288982s) [upload=4769]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 04:51:09.556145753 (6h0m15.928725108s) [seg_pos={0 0}, upload=4769]: Begin upload segment... Jul 19 04:51:09.556194817 (6h0m15.928774172s) [seg_pos={0 0}, upload=4769]: Done begin upload segment. Jul 19 04:51:09.556223893 (6h0m15.928803248s) [noise=true , address=79.127.219.38:4023 , piece_id=PTPOZGLYV4MZTOFZNN3NCGYL7DFAST4PXKEUIVTMFL7HV4GZTCLQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556252197 (6h0m15.928831552s) [noise=true , address=45.91.95.4:4011 , piece_id=ZMBAS6KKHYVIRE6OBL2NFVCWMM4TKQ7J64BD3SYXIK54XCEDI2XA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556579383 (6h0m15.929158738s) [noise=true , address=45.91.116.6:4007 , piece_id=A2AFVTZBXSGBOE7XIM3B5HRN4STITFBXO6MZRJ365URNAAOZ3PSQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556854469 (6h0m15.929433824s) [noise=true , address=104.36.82.19:4423 , piece_id=PIJUHDUND36HQL67I33D7A3EULMJHS7N55MN4OFQJ4OWNQ42O47A, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557094107 (6h0m15.929673462s) [noise=true , address=45.91.116.9:4264 , piece_id=GHM5BKZKMT2QOHHBOS4FSPOWEWSFQRGRVVR4GSNDPHVIFRZHQL2A, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557336381 (6h0m15.929915736s) [noise=true , address=79.127.205.235:4660 , piece_id=UOUOGE2VP7PX7EMRZLPMCWFCYEZERM3DQMCQ6NG4J2K2DEIZR63Q, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557557885 (6h0m15.93013724s) [noise=true , address=79.127.205.246:4280 , piece_id=FLBOZI3UGTQXAQBLZ2F45Z3OEYL65F5PQBXRF2DQMZ7NGF7VDRPA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557762296 (6h0m15.930341651s) [noise=true , address=79.127.205.233:4622 , piece_id=6WWNYQTH5MGAJOEB7BXKZDJOMCAZXDW66J3OEANGDNJDNUMLZM7Q, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558014689 (6h0m15.930594044s) [noise=true , address=64.49.13.138:4482 , piece_id=MAW2M4US2OQNOS3WI67AA7NIBTJDE5T2JQTXKFPBLGVGXKVCFGTQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558215974 (6h0m15.930795319s) [noise=true , address=77.90.31.10:4546 , piece_id=R6LCA5WZOEBDURL5S3MEK22YPVYXX6EXSS4J4W57543HELYXOKZA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558409364 (6h0m15.930988719s) [noise=true , address=45.91.95.5:4019 , piece_id=YTSBCN2LPHEMRRAZIHJGVB7NBOTRXVEHD2TWUCTTLBNFFYEWWDKA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558623504 (6h0m15.931202859s) [noise=true , address=79.127.201.215:4357 , piece_id=75XWAG2VQWEYCWN2Q3WRIZSBI7LDOISZJPXQLQ2GIPNEN34VLSUQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558805162 (6h0m15.931384517s) [noise=true , address=45.91.116.14:4219 , piece_id=PDW6JPRHCVMHHOWIHT6JKV4CTSC3JY6NDKDVFZSO47OYNPZ7PGGA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558999373 (6h0m15.931578728s) [noise=true , address=77.90.31.11:4564 , piece_id=L2U7JFJRDQOFA2AYVAGTI4ZW4VABCXKMXWBIUWQO2QNNUUBJ74BA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559184768 (6h0m15.931764123s) [noise=true , address=77.90.31.16:4768 , piece_id=YNLSONFM6COZEVF6ZNQR2EA6V4LSQFZJONIXR4AFTNNMFGPV57JA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559368719 (6h0m15.931948074s) [noise=true , address=79.127.220.98:4432 , piece_id=2SUTFQ7FVDPAM2H4W3OLW6AVH3VJ33IJFE4L52SA3ET5LGBMOMPQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559549706 (6h0m15.932129061s) [noise=true , address=77.90.31.18:4813 , piece_id=VDOT7BB4O7XYJHW6BFTAYQZZYLZU5PAUXVJGSXJ7FDI4TH425COA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559727836 (6h0m15.932307191s) [noise=true , address=79.127.219.51:4211 , piece_id=HO5PLWBVU5YAYNYMN352NA73SM4WBJV4WZ7XLSQTCQ76UAL2TOJA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:10.663359087 (6h0m17.035938442s) [upload=4769]: Next returned nil segment Jul 19 04:51:10.663378664 (6h0m17.035958019s) [upload=4769]: Waiting for error group managing segments... Jul 19 04:51:10.962945479 (6h0m17.335524834s) [noise=true , address=79.127.205.233:4622 , piece_id=6WWNYQTH5MGAJOEB7BXKZDJOMCAZXDW66J3OEANGDNJDNUMLZM7Q, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:10.967689807 (6h0m17.340269152s) [noise=true , address=79.127.205.235:4660 , piece_id=UOUOGE2VP7PX7EMRZLPMCWFCYEZERM3DQMCQ6NG4J2K2DEIZR63Q, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:10.976968631 (6h0m17.349547986s) [noise=true , address=64.49.13.138:4482 , piece_id=MAW2M4US2OQNOS3WI67AA7NIBTJDE5T2JQTXKFPBLGVGXKVCFGTQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.048750700 (6h0m17.421330055s) [noise=true , address=79.127.205.246:4280 , piece_id=FLBOZI3UGTQXAQBLZ2F45Z3OEYL65F5PQBXRF2DQMZ7NGF7VDRPA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.221642403 (6h0m17.594221758s) [noise=true , address=79.127.220.98:4432 , piece_id=2SUTFQ7FVDPAM2H4W3OLW6AVH3VJ33IJFE4L52SA3ET5LGBMOMPQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.317070435 (6h0m17.68964979s) [noise=true , address=45.91.95.4:4011 , piece_id=ZMBAS6KKHYVIRE6OBL2NFVCWMM4TKQ7J64BD3SYXIK54XCEDI2XA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.324002660 (6h0m17.696582015s) [noise=true , address=45.91.95.5:4019 , piece_id=YTSBCN2LPHEMRRAZIHJGVB7NBOTRXVEHD2TWUCTTLBNFFYEWWDKA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.460976317 (6h0m17.833555672s) [noise=true , address=79.127.201.215:4357 , piece_id=75XWAG2VQWEYCWN2Q3WRIZSBI7LDOISZJPXQLQ2GIPNEN34VLSUQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.462584603 (6h0m17.835163958s) [noise=true , address=45.91.116.6:4007 , piece_id=A2AFVTZBXSGBOE7XIM3B5HRN4STITFBXO6MZRJ365URNAAOZ3PSQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.490392973 (6h0m17.862972328s) [noise=true , address=45.91.116.9:4264 , piece_id=GHM5BKZKMT2QOHHBOS4FSPOWEWSFQRGRVVR4GSNDPHVIFRZHQL2A, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.502040780 (6h0m17.874620135s) [noise=true , address=45.91.116.14:4219 , piece_id=PDW6JPRHCVMHHOWIHT6JKV4CTSC3JY6NDKDVFZSO47OYNPZ7PGGA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.600434096 (6h0m17.973013451s) [noise=true , address=79.127.219.51:4211 , piece_id=HO5PLWBVU5YAYNYMN352NA73SM4WBJV4WZ7XLSQTCQ76UAL2TOJA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.609008984 (6h0m17.981588339s) [noise=true , address=104.36.82.19:4423 , piece_id=PIJUHDUND36HQL67I33D7A3EULMJHS7N55MN4OFQJ4OWNQ42O47A, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.620825273 (6h0m17.993404628s) [noise=true , address=79.127.219.38:4023 , piece_id=PTPOZGLYV4MZTOFZNN3NCGYL7DFAST4PXKEUIVTMFL7HV4GZTCLQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.620838659 (6h0m17.993418014s) [seg_pos={0 0}, upload=4769]: Segment reached optimal threshold of 14 pieces. Jul 19 04:51:11.621067807 (6h0m17.993647163s) [noise=true , address=77.90.31.11:4564 , piece_id=L2U7JFJRDQOFA2AYVAGTI4ZW4VABCXKMXWBIUWQO2QNNUUBJ74BA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection; write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection Jul 19 04:51:11.621134234 (6h0m17.993713589s) [noise=true , address=77.90.31.10:4546 , piece_id=R6LCA5WZOEBDURL5S3MEK22YPVYXX6EXSS4J4W57543HELYXOKZA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK): protocol: read tcp 188.166.225.170:58476->77.90.31.10:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621174191 (6h0m17.993753546s) [noise=true , address=77.90.31.16:4768 , piece_id=YNLSONFM6COZEVF6ZNQR2EA6V4LSQFZJONIXR4AFTNNMFGPV57JA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UchewmtxBrtSZSpVAJFdZmcTsQVQxkjK9re6cgUGqDhyonQeF): protocol: read tcp 188.166.225.170:56326->77.90.31.16:4768: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621216842 (6h0m17.993796197s) [noise=true , address=77.90.31.18:4813 , piece_id=VDOT7BB4O7XYJHW6BFTAYQZZYLZU5PAUXVJGSXJ7FDI4TH425COA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128WthA1x9wp1dV1vkpGnoveZQpU9CSgDCZfw5GMQuVpCNSVAM5): protocol: read tcp 188.166.225.170:49704->77.90.31.18:4813: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621234917 (6h0m17.993814272s) [seg_pos={0 0}, upload=4769]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 19 04:51:11.621301444 (6h0m17.993880799s) [upload=4769]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 19 04:51:11.621305402 (6h0m17.993884758s) [upload=4769]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 19 04:51:11.923136970 (6h0m18.295716325s) [upload=4769]: Done uploading segments... Jul 19 04:51:11.923156627 (6h0m18.295735972s) [upload=4769]: Upload finished. err: <nil> Duration: 5.611237685s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [..................XX...X...X....X..X....XXXXX.X.X.....XX.X..] duration: 6.933027984s FAIL (2025-07-19 05:16:39 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== gment. successful: 13 optimal: 13 errs: <nil> Jul 19 05:14:35.936262577 (6h23m42.308841932s) [upload=5075]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 19 05:14:35.936281884 (6h23m42.308861239s) [upload=5075]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 19 05:14:36.234171969 (6h23m42.606751314s) [upload=5075]: Done uploading segments... Jul 19 05:14:36.234192037 (6h23m42.606771382s) [upload=5075]: Upload finished. err: <nil> Jul 19 05:14:55.587986728 (6h24m1.960566093s) [upload=5081]: Starting upload Jul 19 05:14:55.588051823 (6h24m1.960631178s) [upload=5081]: Done starting upload Jul 19 05:14:55.588146724 (6h24m1.960726079s) [upload=5081]: Uploading segments... Jul 19 05:14:55.588391873 (6h24m1.960971228s) [upload=5081]: Got next segment. Inline: false Jul 19 05:14:55.588649837 (6h24m1.961229193s) [upload=5081]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 05:14:55.588655057 (6h24m1.961234402s) [upload=5081]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 05:14:55.823387692 (6h24m2.195967047s) [seg_pos={0 0}, upload=5081]: Begin upload segment... Jul 19 05:14:55.823450572 (6h24m2.196029927s) [seg_pos={0 0}, upload=5081]: Done begin upload segment. Jul 19 05:14:55.823481952 (6h24m2.196061307s) [noise=true , address=77.90.31.17:4772 , piece_id=B544E3F2UC3UVE2ITUQAWGVDMSPS2HFIVHXZ2OLWB3T25AHMFAVA, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.823491401 (6h24m2.196070756s) [noise=true , address=45.91.116.10:4082 , piece_id=HZDMC4J5O5UHVDH5SOQBKAFPN6RJUHIWKEZ4OPVMW2HSN7NDZMZA, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.823923789 (6h24m2.196503144s) [noise=true , address=45.91.116.9:4262 , piece_id=FRIATDM7LPPHXYAMUISAYRTQWAKSTB4EYUANHGEMQHM6JQADDTRQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.823949737 (6h24m2.196529092s) [noise=true , address=45.91.116.12:4227 , piece_id=QB7YKZFXWMRB3VQ3HSFMMMLLX5YFWOXG4Y4GJGGYT6HDV4FQ65VA, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.824310158 (6h24m2.196889513s) [noise=true , address=45.91.116.4:4937 , piece_id=V2IOOA7DVCGZBH3D43ROYHT6ZECPY6GTC5M7NFO2RWPQB6EYGALA, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.824403045 (6h24m2.19698239s) [noise=true , address=77.90.31.11:4555 , piece_id=YICA5IVHSCC3RPC5FRKD26BSMY6N5WJQA5LCABJSRJYL6BZJYL6A, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.824651210 (6h24m2.197230565s) [noise=true , address=77.90.31.13:4627 , piece_id=E7GSSBZ653K2AA4TP6A4BRT5BDZCBODNRAZM3P6SYJXJFQU2BACQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.824716334 (6h24m2.197295689s) [noise=true , address=77.90.31.16:4734 , piece_id=YUW6MWSUDNLAC2VHLBT6H4T27YTNJ3RDK4BSGE7OZQCO3F2LBH4A, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.824960491 (6h24m2.197539856s) [noise=true , address=45.91.116.7:4010 , piece_id=BH76RM4NXGEAI2J2424ALK4PFSXUTBCYOH777BUOIDW7BW2IJUJA, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.824977494 (6h24m2.197556849s) [noise=true , address=45.91.116.14:4190 , piece_id=RUU2MICSAWOECAAYNOBS75SE2YKSUBMSSFKLPON7FWK77UNJEYGQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.825213285 (6h24m2.19779264s) [noise=true , address=45.91.116.8:4047 , piece_id=6UOFOV4VCVT6DB7D7Q23REAX7VZPAW5PSG5PLA54LISOUWMPQBBQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.825271547 (6h24m2.197850902s) [noise=true , address=77.90.31.18:4806 , piece_id=67PXFJMJB6XWNU6JMOKPO4BXKDD2Z225ZKTB7DZEA46OMSSXHHMQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.825445860 (6h24m2.198025215s) [noise=true , address=45.91.116.7:4009 , piece_id=HH5I32ZYT5LVXHDLBVSAVX6T23MJ6RGNPNAWPIQAREJ4ELPFJGYA, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.825568244 (6h24m2.198147599s) [noise=true , address=77.90.31.15:4697 , piece_id=W62ARDAVLI64P6VYSPVFRTDKB5H2WXTSCF2VKP3UN7JKGBCUEZTQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.825902985 (6h24m2.19848234s) [noise=true , address=45.91.116.5:4299 , piece_id=JQLK5LIMCMW6R2UZEJZSG6QAP2M65XBVE6JFEXY5KDWBVNDG2VCQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.826216495 (6h24m2.19879585s) [noise=true , address=77.90.31.14:4663 , piece_id=LNTFUWB6LUUFD2EB536ZHBSNVVXYZPOOB4DUZB2M6BXSSFEL3UMQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:58.633690556 (6h24m5.006269911s) [upload=5081]: Next returned nil segment Jul 19 05:14:58.633708100 (6h24m5.006287445s) [upload=5081]: Waiting for error group managing segments... Jul 19 05:14:58.915966948 (6h24m5.288546293s) [noise=true , address=45.91.116.7:4009 , piece_id=HH5I32ZYT5LVXHDLBVSAVX6T23MJ6RGNPNAWPIQAREJ4ELPFJGYA, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:58.916093541 (6h24m5.288672896s) [noise=true , address=45.91.116.12:4227 , piece_id=QB7YKZFXWMRB3VQ3HSFMMMLLX5YFWOXG4Y4GJGGYT6HDV4FQ65VA, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:58.923615462 (6h24m5.296194817s) [noise=true , address=45.91.116.14:4190 , piece_id=RUU2MICSAWOECAAYNOBS75SE2YKSUBMSSFKLPON7FWK77UNJEYGQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:58.928612934 (6h24m5.301192289s) [noise=true , address=45.91.116.7:4010 , piece_id=BH76RM4NXGEAI2J2424ALK4PFSXUTBCYOH777BUOIDW7BW2IJUJA, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:58.990915468 (6h24m5.363494833s) [noise=true , address=45.91.116.4:4937 , piece_id=V2IOOA7DVCGZBH3D43ROYHT6ZECPY6GTC5M7NFO2RWPQB6EYGALA, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:58.992136464 (6h24m5.364715819s) [noise=true , address=45.91.116.8:4047 , piece_id=6UOFOV4VCVT6DB7D7Q23REAX7VZPAW5PSG5PLA54LISOUWMPQBBQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:58.997711812 (6h24m5.370291157s) [noise=true , address=45.91.116.9:4262 , piece_id=FRIATDM7LPPHXYAMUISAYRTQWAKSTB4EYUANHGEMQHM6JQADDTRQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:59.010548391 (6h24m5.383127746s) [noise=true , address=45.91.116.10:4082 , piece_id=HZDMC4J5O5UHVDH5SOQBKAFPN6RJUHIWKEZ4OPVMW2HSN7NDZMZA, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:59.039607621 (6h24m5.412186996s) [noise=true , address=45.91.116.5:4299 , piece_id=JQLK5LIMCMW6R2UZEJZSG6QAP2M65XBVE6JFEXY5KDWBVNDG2VCQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:15:19.998482215 (6h24m26.37106157s) [noise=true , address=77.90.31.14:4663 , piece_id=LNTFUWB6LUUFD2EB536ZHBSNVVXYZPOOB4DUZB2M6BXSSFEL3UMQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:15:21.170089451 (6h24m27.542668806s) [noise=true , address=77.90.31.13:4627 , piece_id=E7GSSBZ653K2AA4TP6A4BRT5BDZCBODNRAZM3P6SYJXJFQU2BACQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:15:24.941469148 (6h24m31.314048503s) [noise=true , address=77.90.31.15:4697 , piece_id=W62ARDAVLI64P6VYSPVFRTDKB5H2WXTSCF2VKP3UN7JKGBCUEZTQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: expected piece hash; context deadline exceeded; EOF Jul 19 05:15:24.941530766 (6h24m31.314110121s) [noise=true , address=77.90.31.17:4772 , piece_id=B544E3F2UC3UVE2ITUQAWGVDMSPS2HFIVHXZ2OLWB3T25AHMFAVA, seg_pos={0 0}, upload=5081]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:44456->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:44456->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44456->77.90.31.17:4772: use of closed network connection Jul 19 05:15:24.941543591 (6h24m31.314122946s) [noise=true , address=77.90.31.16:4734 , piece_id=YUW6MWSUDNLAC2VHLBT6H4T27YTNJ3RDK4BSGE7OZQCO3F2LBH4A, seg_pos={0 0}, upload=5081]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Jul 19 05:15:24.941614786 (6h24m31.314194141s) [noise=true , address=77.90.31.11:4555 , piece_id=YICA5IVHSCC3RPC5FRKD26BSMY6N5WJQA5LCABJSRJYL6BZJYL6A, seg_pos={0 0}, upload=5081]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: write tcp 188.166.225.170:37392->77.90.31.11:4555: use of closed network connection; write tcp 188.166.225.170:37392->77.90.31.11:4555: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37392->77.90.31.11:4555: use of closed network connection Jul 19 05:15:24.941641888 (6h24m31.314221243s) [noise=true , address=77.90.31.18:4806 , piece_id=67PXFJMJB6XWNU6JMOKPO4BXKDD2Z225ZKTB7DZEA46OMSSXHHMQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:47114->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:47114->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47114->77.90.31.18:4806: use of closed network connection Jul 19 05:15:24.941663099 (6h24m31.314242454s) [seg_pos={0 0}, upload=5081]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 19 05:15:25.179949476 (6h24m31.552528831s) [upload=5081]: Done uploading segments... Jul 19 05:15:25.179975797 (6h24m31.552555152s) [upload=5081]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.933058504s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [...................XX...X...X....X..X....XXXXX.X.X.....XX.X.] duration: 19.002634454s FAIL (2025-07-19 05:16:20 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== gment. successful: 13 optimal: 13 errs: <nil> Jul 19 05:14:35.936262577 (6h23m42.308841932s) [upload=5075]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 19 05:14:35.936281884 (6h23m42.308861239s) [upload=5075]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 19 05:14:36.234171969 (6h23m42.606751314s) [upload=5075]: Done uploading segments... Jul 19 05:14:36.234192037 (6h23m42.606771382s) [upload=5075]: Upload finished. err: <nil> Jul 19 05:14:55.587986728 (6h24m1.960566093s) [upload=5081]: Starting upload Jul 19 05:14:55.588051823 (6h24m1.960631178s) [upload=5081]: Done starting upload Jul 19 05:14:55.588146724 (6h24m1.960726079s) [upload=5081]: Uploading segments... Jul 19 05:14:55.588391873 (6h24m1.960971228s) [upload=5081]: Got next segment. Inline: false Jul 19 05:14:55.588649837 (6h24m1.961229193s) [upload=5081]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 05:14:55.588655057 (6h24m1.961234402s) [upload=5081]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 05:14:55.823387692 (6h24m2.195967047s) [seg_pos={0 0}, upload=5081]: Begin upload segment... Jul 19 05:14:55.823450572 (6h24m2.196029927s) [seg_pos={0 0}, upload=5081]: Done begin upload segment. Jul 19 05:14:55.823481952 (6h24m2.196061307s) [noise=true , address=77.90.31.17:4772 , piece_id=B544E3F2UC3UVE2ITUQAWGVDMSPS2HFIVHXZ2OLWB3T25AHMFAVA, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.823491401 (6h24m2.196070756s) [noise=true , address=45.91.116.10:4082 , piece_id=HZDMC4J5O5UHVDH5SOQBKAFPN6RJUHIWKEZ4OPVMW2HSN7NDZMZA, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.823923789 (6h24m2.196503144s) [noise=true , address=45.91.116.9:4262 , piece_id=FRIATDM7LPPHXYAMUISAYRTQWAKSTB4EYUANHGEMQHM6JQADDTRQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.823949737 (6h24m2.196529092s) [noise=true , address=45.91.116.12:4227 , piece_id=QB7YKZFXWMRB3VQ3HSFMMMLLX5YFWOXG4Y4GJGGYT6HDV4FQ65VA, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.824310158 (6h24m2.196889513s) [noise=true , address=45.91.116.4:4937 , piece_id=V2IOOA7DVCGZBH3D43ROYHT6ZECPY6GTC5M7NFO2RWPQB6EYGALA, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.824403045 (6h24m2.19698239s) [noise=true , address=77.90.31.11:4555 , piece_id=YICA5IVHSCC3RPC5FRKD26BSMY6N5WJQA5LCABJSRJYL6BZJYL6A, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.824651210 (6h24m2.197230565s) [noise=true , address=77.90.31.13:4627 , piece_id=E7GSSBZ653K2AA4TP6A4BRT5BDZCBODNRAZM3P6SYJXJFQU2BACQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.824716334 (6h24m2.197295689s) [noise=true , address=77.90.31.16:4734 , piece_id=YUW6MWSUDNLAC2VHLBT6H4T27YTNJ3RDK4BSGE7OZQCO3F2LBH4A, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.824960491 (6h24m2.197539856s) [noise=true , address=45.91.116.7:4010 , piece_id=BH76RM4NXGEAI2J2424ALK4PFSXUTBCYOH777BUOIDW7BW2IJUJA, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.824977494 (6h24m2.197556849s) [noise=true , address=45.91.116.14:4190 , piece_id=RUU2MICSAWOECAAYNOBS75SE2YKSUBMSSFKLPON7FWK77UNJEYGQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.825213285 (6h24m2.19779264s) [noise=true , address=45.91.116.8:4047 , piece_id=6UOFOV4VCVT6DB7D7Q23REAX7VZPAW5PSG5PLA54LISOUWMPQBBQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.825271547 (6h24m2.197850902s) [noise=true , address=77.90.31.18:4806 , piece_id=67PXFJMJB6XWNU6JMOKPO4BXKDD2Z225ZKTB7DZEA46OMSSXHHMQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.825445860 (6h24m2.198025215s) [noise=true , address=45.91.116.7:4009 , piece_id=HH5I32ZYT5LVXHDLBVSAVX6T23MJ6RGNPNAWPIQAREJ4ELPFJGYA, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.825568244 (6h24m2.198147599s) [noise=true , address=77.90.31.15:4697 , piece_id=W62ARDAVLI64P6VYSPVFRTDKB5H2WXTSCF2VKP3UN7JKGBCUEZTQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.825902985 (6h24m2.19848234s) [noise=true , address=45.91.116.5:4299 , piece_id=JQLK5LIMCMW6R2UZEJZSG6QAP2M65XBVE6JFEXY5KDWBVNDG2VCQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.826216495 (6h24m2.19879585s) [noise=true , address=77.90.31.14:4663 , piece_id=LNTFUWB6LUUFD2EB536ZHBSNVVXYZPOOB4DUZB2M6BXSSFEL3UMQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:58.633690556 (6h24m5.006269911s) [upload=5081]: Next returned nil segment Jul 19 05:14:58.633708100 (6h24m5.006287445s) [upload=5081]: Waiting for error group managing segments... Jul 19 05:14:58.915966948 (6h24m5.288546293s) [noise=true , address=45.91.116.7:4009 , piece_id=HH5I32ZYT5LVXHDLBVSAVX6T23MJ6RGNPNAWPIQAREJ4ELPFJGYA, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:58.916093541 (6h24m5.288672896s) [noise=true , address=45.91.116.12:4227 , piece_id=QB7YKZFXWMRB3VQ3HSFMMMLLX5YFWOXG4Y4GJGGYT6HDV4FQ65VA, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:58.923615462 (6h24m5.296194817s) [noise=true , address=45.91.116.14:4190 , piece_id=RUU2MICSAWOECAAYNOBS75SE2YKSUBMSSFKLPON7FWK77UNJEYGQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:58.928612934 (6h24m5.301192289s) [noise=true , address=45.91.116.7:4010 , piece_id=BH76RM4NXGEAI2J2424ALK4PFSXUTBCYOH777BUOIDW7BW2IJUJA, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:58.990915468 (6h24m5.363494833s) [noise=true , address=45.91.116.4:4937 , piece_id=V2IOOA7DVCGZBH3D43ROYHT6ZECPY6GTC5M7NFO2RWPQB6EYGALA, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:58.992136464 (6h24m5.364715819s) [noise=true , address=45.91.116.8:4047 , piece_id=6UOFOV4VCVT6DB7D7Q23REAX7VZPAW5PSG5PLA54LISOUWMPQBBQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:58.997711812 (6h24m5.370291157s) [noise=true , address=45.91.116.9:4262 , piece_id=FRIATDM7LPPHXYAMUISAYRTQWAKSTB4EYUANHGEMQHM6JQADDTRQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:59.010548391 (6h24m5.383127746s) [noise=true , address=45.91.116.10:4082 , piece_id=HZDMC4J5O5UHVDH5SOQBKAFPN6RJUHIWKEZ4OPVMW2HSN7NDZMZA, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:59.039607621 (6h24m5.412186996s) [noise=true , address=45.91.116.5:4299 , piece_id=JQLK5LIMCMW6R2UZEJZSG6QAP2M65XBVE6JFEXY5KDWBVNDG2VCQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:15:19.998482215 (6h24m26.37106157s) [noise=true , address=77.90.31.14:4663 , piece_id=LNTFUWB6LUUFD2EB536ZHBSNVVXYZPOOB4DUZB2M6BXSSFEL3UMQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:15:21.170089451 (6h24m27.542668806s) [noise=true , address=77.90.31.13:4627 , piece_id=E7GSSBZ653K2AA4TP6A4BRT5BDZCBODNRAZM3P6SYJXJFQU2BACQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:15:24.941469148 (6h24m31.314048503s) [noise=true , address=77.90.31.15:4697 , piece_id=W62ARDAVLI64P6VYSPVFRTDKB5H2WXTSCF2VKP3UN7JKGBCUEZTQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: expected piece hash; context deadline exceeded; EOF Jul 19 05:15:24.941530766 (6h24m31.314110121s) [noise=true , address=77.90.31.17:4772 , piece_id=B544E3F2UC3UVE2ITUQAWGVDMSPS2HFIVHXZ2OLWB3T25AHMFAVA, seg_pos={0 0}, upload=5081]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:44456->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:44456->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44456->77.90.31.17:4772: use of closed network connection Jul 19 05:15:24.941543591 (6h24m31.314122946s) [noise=true , address=77.90.31.16:4734 , piece_id=YUW6MWSUDNLAC2VHLBT6H4T27YTNJ3RDK4BSGE7OZQCO3F2LBH4A, seg_pos={0 0}, upload=5081]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Jul 19 05:15:24.941614786 (6h24m31.314194141s) [noise=true , address=77.90.31.11:4555 , piece_id=YICA5IVHSCC3RPC5FRKD26BSMY6N5WJQA5LCABJSRJYL6BZJYL6A, seg_pos={0 0}, upload=5081]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: write tcp 188.166.225.170:37392->77.90.31.11:4555: use of closed network connection; write tcp 188.166.225.170:37392->77.90.31.11:4555: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37392->77.90.31.11:4555: use of closed network connection Jul 19 05:15:24.941641888 (6h24m31.314221243s) [noise=true , address=77.90.31.18:4806 , piece_id=67PXFJMJB6XWNU6JMOKPO4BXKDD2Z225ZKTB7DZEA46OMSSXHHMQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:47114->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:47114->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47114->77.90.31.18:4806: use of closed network connection Jul 19 05:15:24.941663099 (6h24m31.314242454s) [seg_pos={0 0}, upload=5081]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 19 05:15:25.179949476 (6h24m31.552528831s) [upload=5081]: Done uploading segments... Jul 19 05:15:25.179975797 (6h24m31.552555152s) [upload=5081]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 19.002654132s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 15.68235027s FAIL (2025-07-19 05:16:20 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 4855s) [upload=4769]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 04:51:09.109709627 (6h0m15.482288982s) [upload=4769]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 04:51:09.556145753 (6h0m15.928725108s) [seg_pos={0 0}, upload=4769]: Begin upload segment... Jul 19 04:51:09.556194817 (6h0m15.928774172s) [seg_pos={0 0}, upload=4769]: Done begin upload segment. Jul 19 04:51:09.556223893 (6h0m15.928803248s) [noise=true , address=79.127.219.38:4023 , piece_id=PTPOZGLYV4MZTOFZNN3NCGYL7DFAST4PXKEUIVTMFL7HV4GZTCLQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556252197 (6h0m15.928831552s) [noise=true , address=45.91.95.4:4011 , piece_id=ZMBAS6KKHYVIRE6OBL2NFVCWMM4TKQ7J64BD3SYXIK54XCEDI2XA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556579383 (6h0m15.929158738s) [noise=true , address=45.91.116.6:4007 , piece_id=A2AFVTZBXSGBOE7XIM3B5HRN4STITFBXO6MZRJ365URNAAOZ3PSQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556854469 (6h0m15.929433824s) [noise=true , address=104.36.82.19:4423 , piece_id=PIJUHDUND36HQL67I33D7A3EULMJHS7N55MN4OFQJ4OWNQ42O47A, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557094107 (6h0m15.929673462s) [noise=true , address=45.91.116.9:4264 , piece_id=GHM5BKZKMT2QOHHBOS4FSPOWEWSFQRGRVVR4GSNDPHVIFRZHQL2A, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557336381 (6h0m15.929915736s) [noise=true , address=79.127.205.235:4660 , piece_id=UOUOGE2VP7PX7EMRZLPMCWFCYEZERM3DQMCQ6NG4J2K2DEIZR63Q, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557557885 (6h0m15.93013724s) [noise=true , address=79.127.205.246:4280 , piece_id=FLBOZI3UGTQXAQBLZ2F45Z3OEYL65F5PQBXRF2DQMZ7NGF7VDRPA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557762296 (6h0m15.930341651s) [noise=true , address=79.127.205.233:4622 , piece_id=6WWNYQTH5MGAJOEB7BXKZDJOMCAZXDW66J3OEANGDNJDNUMLZM7Q, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558014689 (6h0m15.930594044s) [noise=true , address=64.49.13.138:4482 , piece_id=MAW2M4US2OQNOS3WI67AA7NIBTJDE5T2JQTXKFPBLGVGXKVCFGTQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558215974 (6h0m15.930795319s) [noise=true , address=77.90.31.10:4546 , piece_id=R6LCA5WZOEBDURL5S3MEK22YPVYXX6EXSS4J4W57543HELYXOKZA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558409364 (6h0m15.930988719s) [noise=true , address=45.91.95.5:4019 , piece_id=YTSBCN2LPHEMRRAZIHJGVB7NBOTRXVEHD2TWUCTTLBNFFYEWWDKA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558623504 (6h0m15.931202859s) [noise=true , address=79.127.201.215:4357 , piece_id=75XWAG2VQWEYCWN2Q3WRIZSBI7LDOISZJPXQLQ2GIPNEN34VLSUQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558805162 (6h0m15.931384517s) [noise=true , address=45.91.116.14:4219 , piece_id=PDW6JPRHCVMHHOWIHT6JKV4CTSC3JY6NDKDVFZSO47OYNPZ7PGGA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558999373 (6h0m15.931578728s) [noise=true , address=77.90.31.11:4564 , piece_id=L2U7JFJRDQOFA2AYVAGTI4ZW4VABCXKMXWBIUWQO2QNNUUBJ74BA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559184768 (6h0m15.931764123s) [noise=true , address=77.90.31.16:4768 , piece_id=YNLSONFM6COZEVF6ZNQR2EA6V4LSQFZJONIXR4AFTNNMFGPV57JA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559368719 (6h0m15.931948074s) [noise=true , address=79.127.220.98:4432 , piece_id=2SUTFQ7FVDPAM2H4W3OLW6AVH3VJ33IJFE4L52SA3ET5LGBMOMPQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559549706 (6h0m15.932129061s) [noise=true , address=77.90.31.18:4813 , piece_id=VDOT7BB4O7XYJHW6BFTAYQZZYLZU5PAUXVJGSXJ7FDI4TH425COA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559727836 (6h0m15.932307191s) [noise=true , address=79.127.219.51:4211 , piece_id=HO5PLWBVU5YAYNYMN352NA73SM4WBJV4WZ7XLSQTCQ76UAL2TOJA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:10.663359087 (6h0m17.035938442s) [upload=4769]: Next returned nil segment Jul 19 04:51:10.663378664 (6h0m17.035958019s) [upload=4769]: Waiting for error group managing segments... Jul 19 04:51:10.962945479 (6h0m17.335524834s) [noise=true , address=79.127.205.233:4622 , piece_id=6WWNYQTH5MGAJOEB7BXKZDJOMCAZXDW66J3OEANGDNJDNUMLZM7Q, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:10.967689807 (6h0m17.340269152s) [noise=true , address=79.127.205.235:4660 , piece_id=UOUOGE2VP7PX7EMRZLPMCWFCYEZERM3DQMCQ6NG4J2K2DEIZR63Q, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:10.976968631 (6h0m17.349547986s) [noise=true , address=64.49.13.138:4482 , piece_id=MAW2M4US2OQNOS3WI67AA7NIBTJDE5T2JQTXKFPBLGVGXKVCFGTQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.048750700 (6h0m17.421330055s) [noise=true , address=79.127.205.246:4280 , piece_id=FLBOZI3UGTQXAQBLZ2F45Z3OEYL65F5PQBXRF2DQMZ7NGF7VDRPA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.221642403 (6h0m17.594221758s) [noise=true , address=79.127.220.98:4432 , piece_id=2SUTFQ7FVDPAM2H4W3OLW6AVH3VJ33IJFE4L52SA3ET5LGBMOMPQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.317070435 (6h0m17.68964979s) [noise=true , address=45.91.95.4:4011 , piece_id=ZMBAS6KKHYVIRE6OBL2NFVCWMM4TKQ7J64BD3SYXIK54XCEDI2XA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.324002660 (6h0m17.696582015s) [noise=true , address=45.91.95.5:4019 , piece_id=YTSBCN2LPHEMRRAZIHJGVB7NBOTRXVEHD2TWUCTTLBNFFYEWWDKA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.460976317 (6h0m17.833555672s) [noise=true , address=79.127.201.215:4357 , piece_id=75XWAG2VQWEYCWN2Q3WRIZSBI7LDOISZJPXQLQ2GIPNEN34VLSUQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.462584603 (6h0m17.835163958s) [noise=true , address=45.91.116.6:4007 , piece_id=A2AFVTZBXSGBOE7XIM3B5HRN4STITFBXO6MZRJ365URNAAOZ3PSQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.490392973 (6h0m17.862972328s) [noise=true , address=45.91.116.9:4264 , piece_id=GHM5BKZKMT2QOHHBOS4FSPOWEWSFQRGRVVR4GSNDPHVIFRZHQL2A, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.502040780 (6h0m17.874620135s) [noise=true , address=45.91.116.14:4219 , piece_id=PDW6JPRHCVMHHOWIHT6JKV4CTSC3JY6NDKDVFZSO47OYNPZ7PGGA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.600434096 (6h0m17.973013451s) [noise=true , address=79.127.219.51:4211 , piece_id=HO5PLWBVU5YAYNYMN352NA73SM4WBJV4WZ7XLSQTCQ76UAL2TOJA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.609008984 (6h0m17.981588339s) [noise=true , address=104.36.82.19:4423 , piece_id=PIJUHDUND36HQL67I33D7A3EULMJHS7N55MN4OFQJ4OWNQ42O47A, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.620825273 (6h0m17.993404628s) [noise=true , address=79.127.219.38:4023 , piece_id=PTPOZGLYV4MZTOFZNN3NCGYL7DFAST4PXKEUIVTMFL7HV4GZTCLQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.620838659 (6h0m17.993418014s) [seg_pos={0 0}, upload=4769]: Segment reached optimal threshold of 14 pieces. Jul 19 04:51:11.621067807 (6h0m17.993647163s) [noise=true , address=77.90.31.11:4564 , piece_id=L2U7JFJRDQOFA2AYVAGTI4ZW4VABCXKMXWBIUWQO2QNNUUBJ74BA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection; write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection Jul 19 04:51:11.621134234 (6h0m17.993713589s) [noise=true , address=77.90.31.10:4546 , piece_id=R6LCA5WZOEBDURL5S3MEK22YPVYXX6EXSS4J4W57543HELYXOKZA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK): protocol: read tcp 188.166.225.170:58476->77.90.31.10:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621174191 (6h0m17.993753546s) [noise=true , address=77.90.31.16:4768 , piece_id=YNLSONFM6COZEVF6ZNQR2EA6V4LSQFZJONIXR4AFTNNMFGPV57JA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UchewmtxBrtSZSpVAJFdZmcTsQVQxkjK9re6cgUGqDhyonQeF): protocol: read tcp 188.166.225.170:56326->77.90.31.16:4768: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621216842 (6h0m17.993796197s) [noise=true , address=77.90.31.18:4813 , piece_id=VDOT7BB4O7XYJHW6BFTAYQZZYLZU5PAUXVJGSXJ7FDI4TH425COA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128WthA1x9wp1dV1vkpGnoveZQpU9CSgDCZfw5GMQuVpCNSVAM5): protocol: read tcp 188.166.225.170:49704->77.90.31.18:4813: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621234917 (6h0m17.993814272s) [seg_pos={0 0}, upload=4769]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 19 04:51:11.621301444 (6h0m17.993880799s) [upload=4769]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 19 04:51:11.621305402 (6h0m17.993884758s) [upload=4769]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 19 04:51:11.923136970 (6h0m18.295716325s) [upload=4769]: Done uploading segments... Jul 19 04:51:11.923156627 (6h0m18.295735972s) [upload=4769]: Upload finished. err: <nil> Duration: 15.682377171s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 5.54229691s FAIL (2025-07-19 05:15:51 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 4855s) [upload=4769]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 04:51:09.109709627 (6h0m15.482288982s) [upload=4769]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 04:51:09.556145753 (6h0m15.928725108s) [seg_pos={0 0}, upload=4769]: Begin upload segment... Jul 19 04:51:09.556194817 (6h0m15.928774172s) [seg_pos={0 0}, upload=4769]: Done begin upload segment. Jul 19 04:51:09.556223893 (6h0m15.928803248s) [noise=true , address=79.127.219.38:4023 , piece_id=PTPOZGLYV4MZTOFZNN3NCGYL7DFAST4PXKEUIVTMFL7HV4GZTCLQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556252197 (6h0m15.928831552s) [noise=true , address=45.91.95.4:4011 , piece_id=ZMBAS6KKHYVIRE6OBL2NFVCWMM4TKQ7J64BD3SYXIK54XCEDI2XA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556579383 (6h0m15.929158738s) [noise=true , address=45.91.116.6:4007 , piece_id=A2AFVTZBXSGBOE7XIM3B5HRN4STITFBXO6MZRJ365URNAAOZ3PSQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556854469 (6h0m15.929433824s) [noise=true , address=104.36.82.19:4423 , piece_id=PIJUHDUND36HQL67I33D7A3EULMJHS7N55MN4OFQJ4OWNQ42O47A, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557094107 (6h0m15.929673462s) [noise=true , address=45.91.116.9:4264 , piece_id=GHM5BKZKMT2QOHHBOS4FSPOWEWSFQRGRVVR4GSNDPHVIFRZHQL2A, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557336381 (6h0m15.929915736s) [noise=true , address=79.127.205.235:4660 , piece_id=UOUOGE2VP7PX7EMRZLPMCWFCYEZERM3DQMCQ6NG4J2K2DEIZR63Q, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557557885 (6h0m15.93013724s) [noise=true , address=79.127.205.246:4280 , piece_id=FLBOZI3UGTQXAQBLZ2F45Z3OEYL65F5PQBXRF2DQMZ7NGF7VDRPA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557762296 (6h0m15.930341651s) [noise=true , address=79.127.205.233:4622 , piece_id=6WWNYQTH5MGAJOEB7BXKZDJOMCAZXDW66J3OEANGDNJDNUMLZM7Q, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558014689 (6h0m15.930594044s) [noise=true , address=64.49.13.138:4482 , piece_id=MAW2M4US2OQNOS3WI67AA7NIBTJDE5T2JQTXKFPBLGVGXKVCFGTQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558215974 (6h0m15.930795319s) [noise=true , address=77.90.31.10:4546 , piece_id=R6LCA5WZOEBDURL5S3MEK22YPVYXX6EXSS4J4W57543HELYXOKZA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558409364 (6h0m15.930988719s) [noise=true , address=45.91.95.5:4019 , piece_id=YTSBCN2LPHEMRRAZIHJGVB7NBOTRXVEHD2TWUCTTLBNFFYEWWDKA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558623504 (6h0m15.931202859s) [noise=true , address=79.127.201.215:4357 , piece_id=75XWAG2VQWEYCWN2Q3WRIZSBI7LDOISZJPXQLQ2GIPNEN34VLSUQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558805162 (6h0m15.931384517s) [noise=true , address=45.91.116.14:4219 , piece_id=PDW6JPRHCVMHHOWIHT6JKV4CTSC3JY6NDKDVFZSO47OYNPZ7PGGA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558999373 (6h0m15.931578728s) [noise=true , address=77.90.31.11:4564 , piece_id=L2U7JFJRDQOFA2AYVAGTI4ZW4VABCXKMXWBIUWQO2QNNUUBJ74BA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559184768 (6h0m15.931764123s) [noise=true , address=77.90.31.16:4768 , piece_id=YNLSONFM6COZEVF6ZNQR2EA6V4LSQFZJONIXR4AFTNNMFGPV57JA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559368719 (6h0m15.931948074s) [noise=true , address=79.127.220.98:4432 , piece_id=2SUTFQ7FVDPAM2H4W3OLW6AVH3VJ33IJFE4L52SA3ET5LGBMOMPQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559549706 (6h0m15.932129061s) [noise=true , address=77.90.31.18:4813 , piece_id=VDOT7BB4O7XYJHW6BFTAYQZZYLZU5PAUXVJGSXJ7FDI4TH425COA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559727836 (6h0m15.932307191s) [noise=true , address=79.127.219.51:4211 , piece_id=HO5PLWBVU5YAYNYMN352NA73SM4WBJV4WZ7XLSQTCQ76UAL2TOJA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:10.663359087 (6h0m17.035938442s) [upload=4769]: Next returned nil segment Jul 19 04:51:10.663378664 (6h0m17.035958019s) [upload=4769]: Waiting for error group managing segments... Jul 19 04:51:10.962945479 (6h0m17.335524834s) [noise=true , address=79.127.205.233:4622 , piece_id=6WWNYQTH5MGAJOEB7BXKZDJOMCAZXDW66J3OEANGDNJDNUMLZM7Q, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:10.967689807 (6h0m17.340269152s) [noise=true , address=79.127.205.235:4660 , piece_id=UOUOGE2VP7PX7EMRZLPMCWFCYEZERM3DQMCQ6NG4J2K2DEIZR63Q, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:10.976968631 (6h0m17.349547986s) [noise=true , address=64.49.13.138:4482 , piece_id=MAW2M4US2OQNOS3WI67AA7NIBTJDE5T2JQTXKFPBLGVGXKVCFGTQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.048750700 (6h0m17.421330055s) [noise=true , address=79.127.205.246:4280 , piece_id=FLBOZI3UGTQXAQBLZ2F45Z3OEYL65F5PQBXRF2DQMZ7NGF7VDRPA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.221642403 (6h0m17.594221758s) [noise=true , address=79.127.220.98:4432 , piece_id=2SUTFQ7FVDPAM2H4W3OLW6AVH3VJ33IJFE4L52SA3ET5LGBMOMPQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.317070435 (6h0m17.68964979s) [noise=true , address=45.91.95.4:4011 , piece_id=ZMBAS6KKHYVIRE6OBL2NFVCWMM4TKQ7J64BD3SYXIK54XCEDI2XA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.324002660 (6h0m17.696582015s) [noise=true , address=45.91.95.5:4019 , piece_id=YTSBCN2LPHEMRRAZIHJGVB7NBOTRXVEHD2TWUCTTLBNFFYEWWDKA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.460976317 (6h0m17.833555672s) [noise=true , address=79.127.201.215:4357 , piece_id=75XWAG2VQWEYCWN2Q3WRIZSBI7LDOISZJPXQLQ2GIPNEN34VLSUQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.462584603 (6h0m17.835163958s) [noise=true , address=45.91.116.6:4007 , piece_id=A2AFVTZBXSGBOE7XIM3B5HRN4STITFBXO6MZRJ365URNAAOZ3PSQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.490392973 (6h0m17.862972328s) [noise=true , address=45.91.116.9:4264 , piece_id=GHM5BKZKMT2QOHHBOS4FSPOWEWSFQRGRVVR4GSNDPHVIFRZHQL2A, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.502040780 (6h0m17.874620135s) [noise=true , address=45.91.116.14:4219 , piece_id=PDW6JPRHCVMHHOWIHT6JKV4CTSC3JY6NDKDVFZSO47OYNPZ7PGGA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.600434096 (6h0m17.973013451s) [noise=true , address=79.127.219.51:4211 , piece_id=HO5PLWBVU5YAYNYMN352NA73SM4WBJV4WZ7XLSQTCQ76UAL2TOJA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.609008984 (6h0m17.981588339s) [noise=true , address=104.36.82.19:4423 , piece_id=PIJUHDUND36HQL67I33D7A3EULMJHS7N55MN4OFQJ4OWNQ42O47A, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.620825273 (6h0m17.993404628s) [noise=true , address=79.127.219.38:4023 , piece_id=PTPOZGLYV4MZTOFZNN3NCGYL7DFAST4PXKEUIVTMFL7HV4GZTCLQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.620838659 (6h0m17.993418014s) [seg_pos={0 0}, upload=4769]: Segment reached optimal threshold of 14 pieces. Jul 19 04:51:11.621067807 (6h0m17.993647163s) [noise=true , address=77.90.31.11:4564 , piece_id=L2U7JFJRDQOFA2AYVAGTI4ZW4VABCXKMXWBIUWQO2QNNUUBJ74BA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection; write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection Jul 19 04:51:11.621134234 (6h0m17.993713589s) [noise=true , address=77.90.31.10:4546 , piece_id=R6LCA5WZOEBDURL5S3MEK22YPVYXX6EXSS4J4W57543HELYXOKZA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK): protocol: read tcp 188.166.225.170:58476->77.90.31.10:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621174191 (6h0m17.993753546s) [noise=true , address=77.90.31.16:4768 , piece_id=YNLSONFM6COZEVF6ZNQR2EA6V4LSQFZJONIXR4AFTNNMFGPV57JA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UchewmtxBrtSZSpVAJFdZmcTsQVQxkjK9re6cgUGqDhyonQeF): protocol: read tcp 188.166.225.170:56326->77.90.31.16:4768: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621216842 (6h0m17.993796197s) [noise=true , address=77.90.31.18:4813 , piece_id=VDOT7BB4O7XYJHW6BFTAYQZZYLZU5PAUXVJGSXJ7FDI4TH425COA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128WthA1x9wp1dV1vkpGnoveZQpU9CSgDCZfw5GMQuVpCNSVAM5): protocol: read tcp 188.166.225.170:49704->77.90.31.18:4813: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621234917 (6h0m17.993814272s) [seg_pos={0 0}, upload=4769]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 19 04:51:11.621301444 (6h0m17.993880799s) [upload=4769]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 19 04:51:11.621305402 (6h0m17.993884758s) [upload=4769]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 19 04:51:11.923136970 (6h0m18.295716325s) [upload=4769]: Done uploading segments... Jul 19 04:51:11.923156627 (6h0m18.295735972s) [upload=4769]: Upload finished. err: <nil> Duration: 5.542319122s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [....................XX...X...X....X..X....XXXXX.X.X.....XX.X] duration: 30.23968037s FAIL (2025-07-19 05:15:25 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== gment. successful: 13 optimal: 13 errs: <nil> Jul 19 05:14:35.936262577 (6h23m42.308841932s) [upload=5075]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 19 05:14:35.936281884 (6h23m42.308861239s) [upload=5075]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 19 05:14:36.234171969 (6h23m42.606751314s) [upload=5075]: Done uploading segments... Jul 19 05:14:36.234192037 (6h23m42.606771382s) [upload=5075]: Upload finished. err: <nil> Jul 19 05:14:55.587986728 (6h24m1.960566093s) [upload=5081]: Starting upload Jul 19 05:14:55.588051823 (6h24m1.960631178s) [upload=5081]: Done starting upload Jul 19 05:14:55.588146724 (6h24m1.960726079s) [upload=5081]: Uploading segments... Jul 19 05:14:55.588391873 (6h24m1.960971228s) [upload=5081]: Got next segment. Inline: false Jul 19 05:14:55.588649837 (6h24m1.961229193s) [upload=5081]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 05:14:55.588655057 (6h24m1.961234402s) [upload=5081]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 05:14:55.823387692 (6h24m2.195967047s) [seg_pos={0 0}, upload=5081]: Begin upload segment... Jul 19 05:14:55.823450572 (6h24m2.196029927s) [seg_pos={0 0}, upload=5081]: Done begin upload segment. Jul 19 05:14:55.823481952 (6h24m2.196061307s) [noise=true , address=77.90.31.17:4772 , piece_id=B544E3F2UC3UVE2ITUQAWGVDMSPS2HFIVHXZ2OLWB3T25AHMFAVA, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.823491401 (6h24m2.196070756s) [noise=true , address=45.91.116.10:4082 , piece_id=HZDMC4J5O5UHVDH5SOQBKAFPN6RJUHIWKEZ4OPVMW2HSN7NDZMZA, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.823923789 (6h24m2.196503144s) [noise=true , address=45.91.116.9:4262 , piece_id=FRIATDM7LPPHXYAMUISAYRTQWAKSTB4EYUANHGEMQHM6JQADDTRQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.823949737 (6h24m2.196529092s) [noise=true , address=45.91.116.12:4227 , piece_id=QB7YKZFXWMRB3VQ3HSFMMMLLX5YFWOXG4Y4GJGGYT6HDV4FQ65VA, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.824310158 (6h24m2.196889513s) [noise=true , address=45.91.116.4:4937 , piece_id=V2IOOA7DVCGZBH3D43ROYHT6ZECPY6GTC5M7NFO2RWPQB6EYGALA, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.824403045 (6h24m2.19698239s) [noise=true , address=77.90.31.11:4555 , piece_id=YICA5IVHSCC3RPC5FRKD26BSMY6N5WJQA5LCABJSRJYL6BZJYL6A, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.824651210 (6h24m2.197230565s) [noise=true , address=77.90.31.13:4627 , piece_id=E7GSSBZ653K2AA4TP6A4BRT5BDZCBODNRAZM3P6SYJXJFQU2BACQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.824716334 (6h24m2.197295689s) [noise=true , address=77.90.31.16:4734 , piece_id=YUW6MWSUDNLAC2VHLBT6H4T27YTNJ3RDK4BSGE7OZQCO3F2LBH4A, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.824960491 (6h24m2.197539856s) [noise=true , address=45.91.116.7:4010 , piece_id=BH76RM4NXGEAI2J2424ALK4PFSXUTBCYOH777BUOIDW7BW2IJUJA, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.824977494 (6h24m2.197556849s) [noise=true , address=45.91.116.14:4190 , piece_id=RUU2MICSAWOECAAYNOBS75SE2YKSUBMSSFKLPON7FWK77UNJEYGQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.825213285 (6h24m2.19779264s) [noise=true , address=45.91.116.8:4047 , piece_id=6UOFOV4VCVT6DB7D7Q23REAX7VZPAW5PSG5PLA54LISOUWMPQBBQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.825271547 (6h24m2.197850902s) [noise=true , address=77.90.31.18:4806 , piece_id=67PXFJMJB6XWNU6JMOKPO4BXKDD2Z225ZKTB7DZEA46OMSSXHHMQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.825445860 (6h24m2.198025215s) [noise=true , address=45.91.116.7:4009 , piece_id=HH5I32ZYT5LVXHDLBVSAVX6T23MJ6RGNPNAWPIQAREJ4ELPFJGYA, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.825568244 (6h24m2.198147599s) [noise=true , address=77.90.31.15:4697 , piece_id=W62ARDAVLI64P6VYSPVFRTDKB5H2WXTSCF2VKP3UN7JKGBCUEZTQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.825902985 (6h24m2.19848234s) [noise=true , address=45.91.116.5:4299 , piece_id=JQLK5LIMCMW6R2UZEJZSG6QAP2M65XBVE6JFEXY5KDWBVNDG2VCQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:55.826216495 (6h24m2.19879585s) [noise=true , address=77.90.31.14:4663 , piece_id=LNTFUWB6LUUFD2EB536ZHBSNVVXYZPOOB4DUZB2M6BXSSFEL3UMQ, seg_pos={0 0}, upload=5081]: Uploading piece... Jul 19 05:14:58.633690556 (6h24m5.006269911s) [upload=5081]: Next returned nil segment Jul 19 05:14:58.633708100 (6h24m5.006287445s) [upload=5081]: Waiting for error group managing segments... Jul 19 05:14:58.915966948 (6h24m5.288546293s) [noise=true , address=45.91.116.7:4009 , piece_id=HH5I32ZYT5LVXHDLBVSAVX6T23MJ6RGNPNAWPIQAREJ4ELPFJGYA, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:58.916093541 (6h24m5.288672896s) [noise=true , address=45.91.116.12:4227 , piece_id=QB7YKZFXWMRB3VQ3HSFMMMLLX5YFWOXG4Y4GJGGYT6HDV4FQ65VA, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:58.923615462 (6h24m5.296194817s) [noise=true , address=45.91.116.14:4190 , piece_id=RUU2MICSAWOECAAYNOBS75SE2YKSUBMSSFKLPON7FWK77UNJEYGQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:58.928612934 (6h24m5.301192289s) [noise=true , address=45.91.116.7:4010 , piece_id=BH76RM4NXGEAI2J2424ALK4PFSXUTBCYOH777BUOIDW7BW2IJUJA, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:58.990915468 (6h24m5.363494833s) [noise=true , address=45.91.116.4:4937 , piece_id=V2IOOA7DVCGZBH3D43ROYHT6ZECPY6GTC5M7NFO2RWPQB6EYGALA, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:58.992136464 (6h24m5.364715819s) [noise=true , address=45.91.116.8:4047 , piece_id=6UOFOV4VCVT6DB7D7Q23REAX7VZPAW5PSG5PLA54LISOUWMPQBBQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:58.997711812 (6h24m5.370291157s) [noise=true , address=45.91.116.9:4262 , piece_id=FRIATDM7LPPHXYAMUISAYRTQWAKSTB4EYUANHGEMQHM6JQADDTRQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:59.010548391 (6h24m5.383127746s) [noise=true , address=45.91.116.10:4082 , piece_id=HZDMC4J5O5UHVDH5SOQBKAFPN6RJUHIWKEZ4OPVMW2HSN7NDZMZA, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:14:59.039607621 (6h24m5.412186996s) [noise=true , address=45.91.116.5:4299 , piece_id=JQLK5LIMCMW6R2UZEJZSG6QAP2M65XBVE6JFEXY5KDWBVNDG2VCQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:15:19.998482215 (6h24m26.37106157s) [noise=true , address=77.90.31.14:4663 , piece_id=LNTFUWB6LUUFD2EB536ZHBSNVVXYZPOOB4DUZB2M6BXSSFEL3UMQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:15:21.170089451 (6h24m27.542668806s) [noise=true , address=77.90.31.13:4627 , piece_id=E7GSSBZ653K2AA4TP6A4BRT5BDZCBODNRAZM3P6SYJXJFQU2BACQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: <nil> Jul 19 05:15:24.941469148 (6h24m31.314048503s) [noise=true , address=77.90.31.15:4697 , piece_id=W62ARDAVLI64P6VYSPVFRTDKB5H2WXTSCF2VKP3UN7JKGBCUEZTQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: expected piece hash; context deadline exceeded; EOF Jul 19 05:15:24.941530766 (6h24m31.314110121s) [noise=true , address=77.90.31.17:4772 , piece_id=B544E3F2UC3UVE2ITUQAWGVDMSPS2HFIVHXZ2OLWB3T25AHMFAVA, seg_pos={0 0}, upload=5081]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:44456->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:44456->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44456->77.90.31.17:4772: use of closed network connection Jul 19 05:15:24.941543591 (6h24m31.314122946s) [noise=true , address=77.90.31.16:4734 , piece_id=YUW6MWSUDNLAC2VHLBT6H4T27YTNJ3RDK4BSGE7OZQCO3F2LBH4A, seg_pos={0 0}, upload=5081]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Jul 19 05:15:24.941614786 (6h24m31.314194141s) [noise=true , address=77.90.31.11:4555 , piece_id=YICA5IVHSCC3RPC5FRKD26BSMY6N5WJQA5LCABJSRJYL6BZJYL6A, seg_pos={0 0}, upload=5081]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: write tcp 188.166.225.170:37392->77.90.31.11:4555: use of closed network connection; write tcp 188.166.225.170:37392->77.90.31.11:4555: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37392->77.90.31.11:4555: use of closed network connection Jul 19 05:15:24.941641888 (6h24m31.314221243s) [noise=true , address=77.90.31.18:4806 , piece_id=67PXFJMJB6XWNU6JMOKPO4BXKDD2Z225ZKTB7DZEA46OMSSXHHMQ, seg_pos={0 0}, upload=5081]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: write tcp 188.166.225.170:47114->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:47114->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47114->77.90.31.18:4806: use of closed network connection Jul 19 05:15:24.941663099 (6h24m31.314242454s) [seg_pos={0 0}, upload=5081]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 19 05:15:25.179949476 (6h24m31.552528831s) [upload=5081]: Done uploading segments... Jul 19 05:15:25.179975797 (6h24m31.552555152s) [upload=5081]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.239699457s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 21.867266141s FAIL (2025-07-19 05:15:10 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 4855s) [upload=4769]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 04:51:09.109709627 (6h0m15.482288982s) [upload=4769]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 04:51:09.556145753 (6h0m15.928725108s) [seg_pos={0 0}, upload=4769]: Begin upload segment... Jul 19 04:51:09.556194817 (6h0m15.928774172s) [seg_pos={0 0}, upload=4769]: Done begin upload segment. Jul 19 04:51:09.556223893 (6h0m15.928803248s) [noise=true , address=79.127.219.38:4023 , piece_id=PTPOZGLYV4MZTOFZNN3NCGYL7DFAST4PXKEUIVTMFL7HV4GZTCLQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556252197 (6h0m15.928831552s) [noise=true , address=45.91.95.4:4011 , piece_id=ZMBAS6KKHYVIRE6OBL2NFVCWMM4TKQ7J64BD3SYXIK54XCEDI2XA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556579383 (6h0m15.929158738s) [noise=true , address=45.91.116.6:4007 , piece_id=A2AFVTZBXSGBOE7XIM3B5HRN4STITFBXO6MZRJ365URNAAOZ3PSQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556854469 (6h0m15.929433824s) [noise=true , address=104.36.82.19:4423 , piece_id=PIJUHDUND36HQL67I33D7A3EULMJHS7N55MN4OFQJ4OWNQ42O47A, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557094107 (6h0m15.929673462s) [noise=true , address=45.91.116.9:4264 , piece_id=GHM5BKZKMT2QOHHBOS4FSPOWEWSFQRGRVVR4GSNDPHVIFRZHQL2A, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557336381 (6h0m15.929915736s) [noise=true , address=79.127.205.235:4660 , piece_id=UOUOGE2VP7PX7EMRZLPMCWFCYEZERM3DQMCQ6NG4J2K2DEIZR63Q, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557557885 (6h0m15.93013724s) [noise=true , address=79.127.205.246:4280 , piece_id=FLBOZI3UGTQXAQBLZ2F45Z3OEYL65F5PQBXRF2DQMZ7NGF7VDRPA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557762296 (6h0m15.930341651s) [noise=true , address=79.127.205.233:4622 , piece_id=6WWNYQTH5MGAJOEB7BXKZDJOMCAZXDW66J3OEANGDNJDNUMLZM7Q, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558014689 (6h0m15.930594044s) [noise=true , address=64.49.13.138:4482 , piece_id=MAW2M4US2OQNOS3WI67AA7NIBTJDE5T2JQTXKFPBLGVGXKVCFGTQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558215974 (6h0m15.930795319s) [noise=true , address=77.90.31.10:4546 , piece_id=R6LCA5WZOEBDURL5S3MEK22YPVYXX6EXSS4J4W57543HELYXOKZA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558409364 (6h0m15.930988719s) [noise=true , address=45.91.95.5:4019 , piece_id=YTSBCN2LPHEMRRAZIHJGVB7NBOTRXVEHD2TWUCTTLBNFFYEWWDKA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558623504 (6h0m15.931202859s) [noise=true , address=79.127.201.215:4357 , piece_id=75XWAG2VQWEYCWN2Q3WRIZSBI7LDOISZJPXQLQ2GIPNEN34VLSUQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558805162 (6h0m15.931384517s) [noise=true , address=45.91.116.14:4219 , piece_id=PDW6JPRHCVMHHOWIHT6JKV4CTSC3JY6NDKDVFZSO47OYNPZ7PGGA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558999373 (6h0m15.931578728s) [noise=true , address=77.90.31.11:4564 , piece_id=L2U7JFJRDQOFA2AYVAGTI4ZW4VABCXKMXWBIUWQO2QNNUUBJ74BA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559184768 (6h0m15.931764123s) [noise=true , address=77.90.31.16:4768 , piece_id=YNLSONFM6COZEVF6ZNQR2EA6V4LSQFZJONIXR4AFTNNMFGPV57JA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559368719 (6h0m15.931948074s) [noise=true , address=79.127.220.98:4432 , piece_id=2SUTFQ7FVDPAM2H4W3OLW6AVH3VJ33IJFE4L52SA3ET5LGBMOMPQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559549706 (6h0m15.932129061s) [noise=true , address=77.90.31.18:4813 , piece_id=VDOT7BB4O7XYJHW6BFTAYQZZYLZU5PAUXVJGSXJ7FDI4TH425COA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559727836 (6h0m15.932307191s) [noise=true , address=79.127.219.51:4211 , piece_id=HO5PLWBVU5YAYNYMN352NA73SM4WBJV4WZ7XLSQTCQ76UAL2TOJA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:10.663359087 (6h0m17.035938442s) [upload=4769]: Next returned nil segment Jul 19 04:51:10.663378664 (6h0m17.035958019s) [upload=4769]: Waiting for error group managing segments... Jul 19 04:51:10.962945479 (6h0m17.335524834s) [noise=true , address=79.127.205.233:4622 , piece_id=6WWNYQTH5MGAJOEB7BXKZDJOMCAZXDW66J3OEANGDNJDNUMLZM7Q, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:10.967689807 (6h0m17.340269152s) [noise=true , address=79.127.205.235:4660 , piece_id=UOUOGE2VP7PX7EMRZLPMCWFCYEZERM3DQMCQ6NG4J2K2DEIZR63Q, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:10.976968631 (6h0m17.349547986s) [noise=true , address=64.49.13.138:4482 , piece_id=MAW2M4US2OQNOS3WI67AA7NIBTJDE5T2JQTXKFPBLGVGXKVCFGTQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.048750700 (6h0m17.421330055s) [noise=true , address=79.127.205.246:4280 , piece_id=FLBOZI3UGTQXAQBLZ2F45Z3OEYL65F5PQBXRF2DQMZ7NGF7VDRPA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.221642403 (6h0m17.594221758s) [noise=true , address=79.127.220.98:4432 , piece_id=2SUTFQ7FVDPAM2H4W3OLW6AVH3VJ33IJFE4L52SA3ET5LGBMOMPQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.317070435 (6h0m17.68964979s) [noise=true , address=45.91.95.4:4011 , piece_id=ZMBAS6KKHYVIRE6OBL2NFVCWMM4TKQ7J64BD3SYXIK54XCEDI2XA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.324002660 (6h0m17.696582015s) [noise=true , address=45.91.95.5:4019 , piece_id=YTSBCN2LPHEMRRAZIHJGVB7NBOTRXVEHD2TWUCTTLBNFFYEWWDKA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.460976317 (6h0m17.833555672s) [noise=true , address=79.127.201.215:4357 , piece_id=75XWAG2VQWEYCWN2Q3WRIZSBI7LDOISZJPXQLQ2GIPNEN34VLSUQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.462584603 (6h0m17.835163958s) [noise=true , address=45.91.116.6:4007 , piece_id=A2AFVTZBXSGBOE7XIM3B5HRN4STITFBXO6MZRJ365URNAAOZ3PSQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.490392973 (6h0m17.862972328s) [noise=true , address=45.91.116.9:4264 , piece_id=GHM5BKZKMT2QOHHBOS4FSPOWEWSFQRGRVVR4GSNDPHVIFRZHQL2A, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.502040780 (6h0m17.874620135s) [noise=true , address=45.91.116.14:4219 , piece_id=PDW6JPRHCVMHHOWIHT6JKV4CTSC3JY6NDKDVFZSO47OYNPZ7PGGA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.600434096 (6h0m17.973013451s) [noise=true , address=79.127.219.51:4211 , piece_id=HO5PLWBVU5YAYNYMN352NA73SM4WBJV4WZ7XLSQTCQ76UAL2TOJA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.609008984 (6h0m17.981588339s) [noise=true , address=104.36.82.19:4423 , piece_id=PIJUHDUND36HQL67I33D7A3EULMJHS7N55MN4OFQJ4OWNQ42O47A, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.620825273 (6h0m17.993404628s) [noise=true , address=79.127.219.38:4023 , piece_id=PTPOZGLYV4MZTOFZNN3NCGYL7DFAST4PXKEUIVTMFL7HV4GZTCLQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.620838659 (6h0m17.993418014s) [seg_pos={0 0}, upload=4769]: Segment reached optimal threshold of 14 pieces. Jul 19 04:51:11.621067807 (6h0m17.993647163s) [noise=true , address=77.90.31.11:4564 , piece_id=L2U7JFJRDQOFA2AYVAGTI4ZW4VABCXKMXWBIUWQO2QNNUUBJ74BA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection; write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection Jul 19 04:51:11.621134234 (6h0m17.993713589s) [noise=true , address=77.90.31.10:4546 , piece_id=R6LCA5WZOEBDURL5S3MEK22YPVYXX6EXSS4J4W57543HELYXOKZA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK): protocol: read tcp 188.166.225.170:58476->77.90.31.10:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621174191 (6h0m17.993753546s) [noise=true , address=77.90.31.16:4768 , piece_id=YNLSONFM6COZEVF6ZNQR2EA6V4LSQFZJONIXR4AFTNNMFGPV57JA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UchewmtxBrtSZSpVAJFdZmcTsQVQxkjK9re6cgUGqDhyonQeF): protocol: read tcp 188.166.225.170:56326->77.90.31.16:4768: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621216842 (6h0m17.993796197s) [noise=true , address=77.90.31.18:4813 , piece_id=VDOT7BB4O7XYJHW6BFTAYQZZYLZU5PAUXVJGSXJ7FDI4TH425COA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128WthA1x9wp1dV1vkpGnoveZQpU9CSgDCZfw5GMQuVpCNSVAM5): protocol: read tcp 188.166.225.170:49704->77.90.31.18:4813: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621234917 (6h0m17.993814272s) [seg_pos={0 0}, upload=4769]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 19 04:51:11.621301444 (6h0m17.993880799s) [upload=4769]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 19 04:51:11.621305402 (6h0m17.993884758s) [upload=4769]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 19 04:51:11.923136970 (6h0m18.295716325s) [upload=4769]: Done uploading segments... Jul 19 04:51:11.923156627 (6h0m18.295735972s) [upload=4769]: Upload finished. err: <nil> Duration: 21.867295738s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.....................XX...X...X....X..X....XXXXX.X.X.....XX.] duration: 5.929817875s FAIL (2025-07-19 05:14:38 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== network connection; write tcp 188.166.225.170:39440->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39440->77.90.31.17:4771: use of closed network connection Jul 19 05:13:30.499725691 (6h22m36.872305046s) [seg_pos={0 0}, upload=5061]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 19 05:13:30.730093449 (6h22m37.102672804s) [upload=5061]: Done uploading segments... Jul 19 05:13:30.730128235 (6h22m37.10270759s) [upload=5061]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Jul 19 05:13:31.615702822 (6h22m37.988282217s) [upload=5064]: Starting upload Jul 19 05:13:31.615841878 (6h22m37.988421233s) [upload=5064]: Done starting upload Jul 19 05:13:31.615921711 (6h22m37.988501106s) [upload=5064]: Uploading segments... Jul 19 05:13:31.616305465 (6h22m37.98888485s) [upload=5064]: Got next segment. Inline: false Jul 19 05:13:31.616375579 (6h22m37.988954964s) [upload=5064]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 05:13:31.616384025 (6h22m37.988963421s) [upload=5064]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 05:13:31.872093568 (6h22m38.244672923s) [seg_pos={0 0}, upload=5064]: Begin upload segment... Jul 19 05:13:31.872131390 (6h22m38.244710735s) [seg_pos={0 0}, upload=5064]: Done begin upload segment. Jul 19 05:13:31.872182527 (6h22m38.244761883s) [noise=true , address=77.90.31.14:4662 , piece_id=7GKZYRTV7JJFMKEE3HHZKGRBBK22UHWS6HF3MGGNPHQBMVW5UZ7Q, seg_pos={0 0}, upload=5064]: Uploading piece... Jul 19 05:13:31.872276428 (6h22m38.244855823s) [noise=true , address=45.91.116.10:4081 , piece_id=YDEAU72ZQKBHGX4R2BOOLJ6EDRCAJGZ7JLJV76A3QA3P7E537XOQ, seg_pos={0 0}, upload=5064]: Uploading piece... Jul 19 05:13:31.872562415 (6h22m38.24514177s) [noise=true , address=45.91.116.9:4261 , piece_id=WUVMOJPZX7L2YJP3DHMOGYO2RESZ6PL7XLP3E5PT2WHM5XBPDTXQ, seg_pos={0 0}, upload=5064]: Uploading piece... Jul 19 05:13:31.872861657 (6h22m38.245441012s) [noise=true , address=45.91.116.6:4975 , piece_id=ZWUNYTCKQXERWQTQJCOB4VV65JROS3CCHODQ5NZ77P5NK7P22UOQ, seg_pos={0 0}, upload=5064]: Uploading piece... Jul 19 05:13:31.873164256 (6h22m38.245743611s) [noise=true , address=77.90.31.17:4772 , piece_id=YMOC4YRTBQGXIPSBQRO5NGIIVRIZOB6GAUHRBSEYNUEYC3PEC6AQ, seg_pos={0 0}, upload=5064]: Uploading piece... Jul 19 05:13:31.873213661 (6h22m38.245793026s) [noise=true , address=45.91.116.4:4937 , piece_id=PLHQNIOBAOMV7LGCSZG4WI4WNVOJFIIZN634ISJEK2TAA2OYHJ5Q, seg_pos={0 0}, upload=5064]: Uploading piece... Jul 19 05:13:31.873502404 (6h22m38.246081759s) [noise=true , address=77.90.31.15:4698 , piece_id=FMKNJIQXLX4DJBMLA22ZHYXNL4XJS6GL5XFPO4I2MCSADEUSEKUQ, seg_pos={0 0}, upload=5064]: Uploading piece... Jul 19 05:13:31.873787228 (6h22m38.246366583s) [noise=true , address=45.91.116.5:4299 , piece_id=KMRMQCO7DEUVM6O3U5SKWRWU53AYEPKVAV5DGF5OPMUPSDR5UFSQ, seg_pos={0 0}, upload=5064]: Uploading piece... Jul 19 05:13:31.873831894 (6h22m38.246411249s) [noise=true , address=77.90.31.18:4806 , piece_id=C6PH5INGJYAGWOMZXHD4YCIEBKJ3ENPPSDU2TRI4YXUUJZBNGNLA, seg_pos={0 0}, upload=5064]: Uploading piece... Jul 19 05:13:31.874090849 (6h22m38.246670204s) [noise=true , address=104.36.82.4:4801 , piece_id=H4RIRSEKFP2PQSASUHXTBXGYGO6V25VQSJTM5DAAXQPT2VULJWHA, seg_pos={0 0}, upload=5064]: Uploading piece... Jul 19 05:13:31.874289600 (6h22m38.246868955s) [noise=true , address=45.91.116.8:4045 , piece_id=MVFBXMEJAW4GTWCHM2JLDGH53PS7W7ITLPQOGI7RJ3OWONAWI66Q, seg_pos={0 0}, upload=5064]: Uploading piece... Jul 19 05:13:31.874332071 (6h22m38.246911426s) [noise=true , address=45.91.116.13:4155 , piece_id=OOHUAZ76KC472TE56LLLOKNBTPX7MXO54GUICIIFU5COLN22NFBA, seg_pos={0 0}, upload=5064]: Uploading piece... Jul 19 05:13:31.874767705 (6h22m38.24734706s) [noise=true , address=45.91.116.6:4974 , piece_id=3EQ7MFBZLAOV3YYUZPDZRVEGQN6RXIJSTEL2AQOBW6ZM3XMD7IYA, seg_pos={0 0}, upload=5064]: Uploading piece... Jul 19 05:13:31.875296988 (6h22m38.247876343s) [noise=true , address=104.36.82.5:4840 , piece_id=WTZIC6PVZX3GFGQ3TAI5MS6ZONYRQOHL5B4GTYDGOM337RP5AJUA, seg_pos={0 0}, upload=5064]: Uploading piece... Jul 19 05:13:31.875854935 (6h22m38.24843429s) [noise=true , address=77.90.31.16:4735 , piece_id=XIIFNPKJQXW2BE3C3WNVVCSKTFK6N2HP7YTYQMRV5KB6VZITTTXQ, seg_pos={0 0}, upload=5064]: Uploading piece... Jul 19 05:13:31.876996680 (6h22m38.249576035s) [noise=true , address=77.90.31.14:4661 , piece_id=HF4PBASCVUG37QJ647CJCYC567ECPWV2QUFC2QVVO46XJXZ4HTCA, seg_pos={0 0}, upload=5064]: Uploading piece... Jul 19 05:13:33.562404160 (6h22m39.934983515s) [upload=5064]: Next returned nil segment Jul 19 05:13:33.562420322 (6h22m39.934999677s) [upload=5064]: Waiting for error group managing segments... Jul 19 05:13:33.913312323 (6h22m40.285891678s) [noise=true , address=45.91.116.9:4261 , piece_id=WUVMOJPZX7L2YJP3DHMOGYO2RESZ6PL7XLP3E5PT2WHM5XBPDTXQ, seg_pos={0 0}, upload=5064]: Done uploading piece. err: <nil> Jul 19 05:13:33.919093626 (6h22m40.291672981s) [noise=true , address=45.91.116.6:4975 , piece_id=ZWUNYTCKQXERWQTQJCOB4VV65JROS3CCHODQ5NZ77P5NK7P22UOQ, seg_pos={0 0}, upload=5064]: Done uploading piece. err: <nil> Jul 19 05:13:33.927887905 (6h22m40.30046726s) [noise=true , address=45.91.116.5:4299 , piece_id=KMRMQCO7DEUVM6O3U5SKWRWU53AYEPKVAV5DGF5OPMUPSDR5UFSQ, seg_pos={0 0}, upload=5064]: Done uploading piece. err: <nil> Jul 19 05:13:33.931433800 (6h22m40.304013155s) [noise=true , address=45.91.116.10:4081 , piece_id=YDEAU72ZQKBHGX4R2BOOLJ6EDRCAJGZ7JLJV76A3QA3P7E537XOQ, seg_pos={0 0}, upload=5064]: Done uploading piece. err: <nil> Jul 19 05:13:33.935178076 (6h22m40.307757431s) [noise=true , address=45.91.116.6:4974 , piece_id=3EQ7MFBZLAOV3YYUZPDZRVEGQN6RXIJSTEL2AQOBW6ZM3XMD7IYA, seg_pos={0 0}, upload=5064]: Done uploading piece. err: <nil> Jul 19 05:13:33.945748795 (6h22m40.31832815s) [noise=true , address=45.91.116.8:4045 , piece_id=MVFBXMEJAW4GTWCHM2JLDGH53PS7W7ITLPQOGI7RJ3OWONAWI66Q, seg_pos={0 0}, upload=5064]: Done uploading piece. err: <nil> Jul 19 05:13:33.947812284 (6h22m40.320391639s) [noise=true , address=45.91.116.4:4937 , piece_id=PLHQNIOBAOMV7LGCSZG4WI4WNVOJFIIZN634ISJEK2TAA2OYHJ5Q, seg_pos={0 0}, upload=5064]: Done uploading piece. err: <nil> Jul 19 05:13:34.080127952 (6h22m40.452707307s) [noise=true , address=45.91.116.13:4155 , piece_id=OOHUAZ76KC472TE56LLLOKNBTPX7MXO54GUICIIFU5COLN22NFBA, seg_pos={0 0}, upload=5064]: Done uploading piece. err: <nil> Jul 19 05:13:34.082690075 (6h22m40.45526943s) [noise=true , address=104.36.82.5:4840 , piece_id=WTZIC6PVZX3GFGQ3TAI5MS6ZONYRQOHL5B4GTYDGOM337RP5AJUA, seg_pos={0 0}, upload=5064]: Done uploading piece. err: <nil> Jul 19 05:13:34.087918400 (6h22m40.460497755s) [noise=true , address=104.36.82.4:4801 , piece_id=H4RIRSEKFP2PQSASUHXTBXGYGO6V25VQSJTM5DAAXQPT2VULJWHA, seg_pos={0 0}, upload=5064]: Done uploading piece. err: <nil> Jul 19 05:13:51.594432847 (6h22m57.967012202s) [noise=true , address=77.90.31.15:4698 , piece_id=FMKNJIQXLX4DJBMLA22ZHYXNL4XJS6GL5XFPO4I2MCSADEUSEKUQ, seg_pos={0 0}, upload=5064]: Done uploading piece. err: <nil> Jul 19 05:13:55.163396647 (6h23m1.535976002s) [noise=true , address=77.90.31.14:4661 , piece_id=HF4PBASCVUG37QJ647CJCYC567ECPWV2QUFC2QVVO46XJXZ4HTCA, seg_pos={0 0}, upload=5064]: Done uploading piece. err: <nil> Jul 19 05:14:00.966255878 (6h23m7.338835233s) [noise=true , address=77.90.31.18:4806 , piece_id=C6PH5INGJYAGWOMZXHD4YCIEBKJ3ENPPSDU2TRI4YXUUJZBNGNLA, seg_pos={0 0}, upload=5064]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: expected piece hash; context deadline exceeded; EOF Jul 19 05:14:00.966369235 (6h23m7.33894859s) [noise=true , address=77.90.31.14:4662 , piece_id=7GKZYRTV7JJFMKEE3HHZKGRBBK22UHWS6HF3MGGNPHQBMVW5UZ7Q, seg_pos={0 0}, upload=5064]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: write tcp 188.166.225.170:56328->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:56328->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56328->77.90.31.14:4662: use of closed network connection Jul 19 05:14:00.966401958 (6h23m7.338981313s) [noise=true , address=77.90.31.16:4735 , piece_id=XIIFNPKJQXW2BE3C3WNVVCSKTFK6N2HP7YTYQMRV5KB6VZITTTXQ, seg_pos={0 0}, upload=5064]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:58392->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:58392->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58392->77.90.31.16:4735: use of closed network connection Jul 19 05:14:00.966414441 (6h23m7.338993797s) [noise=true , address=77.90.31.17:4772 , piece_id=YMOC4YRTBQGXIPSBQRO5NGIIVRIZOB6GAUHRBSEYNUEYC3PEC6AQ, seg_pos={0 0}, upload=5064]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:50432->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:50432->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50432->77.90.31.17:4772: use of closed network connection Jul 19 05:14:00.966441754 (6h23m7.339021109s) [seg_pos={0 0}, upload=5064]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 19 05:14:01.197829382 (6h23m7.570408747s) [upload=5064]: Done uploading segments... Jul 19 05:14:01.197859059 (6h23m7.570438424s) [upload=5064]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.929843394s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 16.814865708s FAIL (2025-07-19 05:14:03 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 4855s) [upload=4769]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 04:51:09.109709627 (6h0m15.482288982s) [upload=4769]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 04:51:09.556145753 (6h0m15.928725108s) [seg_pos={0 0}, upload=4769]: Begin upload segment... Jul 19 04:51:09.556194817 (6h0m15.928774172s) [seg_pos={0 0}, upload=4769]: Done begin upload segment. Jul 19 04:51:09.556223893 (6h0m15.928803248s) [noise=true , address=79.127.219.38:4023 , piece_id=PTPOZGLYV4MZTOFZNN3NCGYL7DFAST4PXKEUIVTMFL7HV4GZTCLQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556252197 (6h0m15.928831552s) [noise=true , address=45.91.95.4:4011 , piece_id=ZMBAS6KKHYVIRE6OBL2NFVCWMM4TKQ7J64BD3SYXIK54XCEDI2XA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556579383 (6h0m15.929158738s) [noise=true , address=45.91.116.6:4007 , piece_id=A2AFVTZBXSGBOE7XIM3B5HRN4STITFBXO6MZRJ365URNAAOZ3PSQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556854469 (6h0m15.929433824s) [noise=true , address=104.36.82.19:4423 , piece_id=PIJUHDUND36HQL67I33D7A3EULMJHS7N55MN4OFQJ4OWNQ42O47A, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557094107 (6h0m15.929673462s) [noise=true , address=45.91.116.9:4264 , piece_id=GHM5BKZKMT2QOHHBOS4FSPOWEWSFQRGRVVR4GSNDPHVIFRZHQL2A, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557336381 (6h0m15.929915736s) [noise=true , address=79.127.205.235:4660 , piece_id=UOUOGE2VP7PX7EMRZLPMCWFCYEZERM3DQMCQ6NG4J2K2DEIZR63Q, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557557885 (6h0m15.93013724s) [noise=true , address=79.127.205.246:4280 , piece_id=FLBOZI3UGTQXAQBLZ2F45Z3OEYL65F5PQBXRF2DQMZ7NGF7VDRPA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557762296 (6h0m15.930341651s) [noise=true , address=79.127.205.233:4622 , piece_id=6WWNYQTH5MGAJOEB7BXKZDJOMCAZXDW66J3OEANGDNJDNUMLZM7Q, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558014689 (6h0m15.930594044s) [noise=true , address=64.49.13.138:4482 , piece_id=MAW2M4US2OQNOS3WI67AA7NIBTJDE5T2JQTXKFPBLGVGXKVCFGTQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558215974 (6h0m15.930795319s) [noise=true , address=77.90.31.10:4546 , piece_id=R6LCA5WZOEBDURL5S3MEK22YPVYXX6EXSS4J4W57543HELYXOKZA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558409364 (6h0m15.930988719s) [noise=true , address=45.91.95.5:4019 , piece_id=YTSBCN2LPHEMRRAZIHJGVB7NBOTRXVEHD2TWUCTTLBNFFYEWWDKA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558623504 (6h0m15.931202859s) [noise=true , address=79.127.201.215:4357 , piece_id=75XWAG2VQWEYCWN2Q3WRIZSBI7LDOISZJPXQLQ2GIPNEN34VLSUQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558805162 (6h0m15.931384517s) [noise=true , address=45.91.116.14:4219 , piece_id=PDW6JPRHCVMHHOWIHT6JKV4CTSC3JY6NDKDVFZSO47OYNPZ7PGGA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558999373 (6h0m15.931578728s) [noise=true , address=77.90.31.11:4564 , piece_id=L2U7JFJRDQOFA2AYVAGTI4ZW4VABCXKMXWBIUWQO2QNNUUBJ74BA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559184768 (6h0m15.931764123s) [noise=true , address=77.90.31.16:4768 , piece_id=YNLSONFM6COZEVF6ZNQR2EA6V4LSQFZJONIXR4AFTNNMFGPV57JA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559368719 (6h0m15.931948074s) [noise=true , address=79.127.220.98:4432 , piece_id=2SUTFQ7FVDPAM2H4W3OLW6AVH3VJ33IJFE4L52SA3ET5LGBMOMPQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559549706 (6h0m15.932129061s) [noise=true , address=77.90.31.18:4813 , piece_id=VDOT7BB4O7XYJHW6BFTAYQZZYLZU5PAUXVJGSXJ7FDI4TH425COA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559727836 (6h0m15.932307191s) [noise=true , address=79.127.219.51:4211 , piece_id=HO5PLWBVU5YAYNYMN352NA73SM4WBJV4WZ7XLSQTCQ76UAL2TOJA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:10.663359087 (6h0m17.035938442s) [upload=4769]: Next returned nil segment Jul 19 04:51:10.663378664 (6h0m17.035958019s) [upload=4769]: Waiting for error group managing segments... Jul 19 04:51:10.962945479 (6h0m17.335524834s) [noise=true , address=79.127.205.233:4622 , piece_id=6WWNYQTH5MGAJOEB7BXKZDJOMCAZXDW66J3OEANGDNJDNUMLZM7Q, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:10.967689807 (6h0m17.340269152s) [noise=true , address=79.127.205.235:4660 , piece_id=UOUOGE2VP7PX7EMRZLPMCWFCYEZERM3DQMCQ6NG4J2K2DEIZR63Q, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:10.976968631 (6h0m17.349547986s) [noise=true , address=64.49.13.138:4482 , piece_id=MAW2M4US2OQNOS3WI67AA7NIBTJDE5T2JQTXKFPBLGVGXKVCFGTQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.048750700 (6h0m17.421330055s) [noise=true , address=79.127.205.246:4280 , piece_id=FLBOZI3UGTQXAQBLZ2F45Z3OEYL65F5PQBXRF2DQMZ7NGF7VDRPA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.221642403 (6h0m17.594221758s) [noise=true , address=79.127.220.98:4432 , piece_id=2SUTFQ7FVDPAM2H4W3OLW6AVH3VJ33IJFE4L52SA3ET5LGBMOMPQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.317070435 (6h0m17.68964979s) [noise=true , address=45.91.95.4:4011 , piece_id=ZMBAS6KKHYVIRE6OBL2NFVCWMM4TKQ7J64BD3SYXIK54XCEDI2XA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.324002660 (6h0m17.696582015s) [noise=true , address=45.91.95.5:4019 , piece_id=YTSBCN2LPHEMRRAZIHJGVB7NBOTRXVEHD2TWUCTTLBNFFYEWWDKA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.460976317 (6h0m17.833555672s) [noise=true , address=79.127.201.215:4357 , piece_id=75XWAG2VQWEYCWN2Q3WRIZSBI7LDOISZJPXQLQ2GIPNEN34VLSUQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.462584603 (6h0m17.835163958s) [noise=true , address=45.91.116.6:4007 , piece_id=A2AFVTZBXSGBOE7XIM3B5HRN4STITFBXO6MZRJ365URNAAOZ3PSQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.490392973 (6h0m17.862972328s) [noise=true , address=45.91.116.9:4264 , piece_id=GHM5BKZKMT2QOHHBOS4FSPOWEWSFQRGRVVR4GSNDPHVIFRZHQL2A, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.502040780 (6h0m17.874620135s) [noise=true , address=45.91.116.14:4219 , piece_id=PDW6JPRHCVMHHOWIHT6JKV4CTSC3JY6NDKDVFZSO47OYNPZ7PGGA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.600434096 (6h0m17.973013451s) [noise=true , address=79.127.219.51:4211 , piece_id=HO5PLWBVU5YAYNYMN352NA73SM4WBJV4WZ7XLSQTCQ76UAL2TOJA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.609008984 (6h0m17.981588339s) [noise=true , address=104.36.82.19:4423 , piece_id=PIJUHDUND36HQL67I33D7A3EULMJHS7N55MN4OFQJ4OWNQ42O47A, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.620825273 (6h0m17.993404628s) [noise=true , address=79.127.219.38:4023 , piece_id=PTPOZGLYV4MZTOFZNN3NCGYL7DFAST4PXKEUIVTMFL7HV4GZTCLQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.620838659 (6h0m17.993418014s) [seg_pos={0 0}, upload=4769]: Segment reached optimal threshold of 14 pieces. Jul 19 04:51:11.621067807 (6h0m17.993647163s) [noise=true , address=77.90.31.11:4564 , piece_id=L2U7JFJRDQOFA2AYVAGTI4ZW4VABCXKMXWBIUWQO2QNNUUBJ74BA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection; write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection Jul 19 04:51:11.621134234 (6h0m17.993713589s) [noise=true , address=77.90.31.10:4546 , piece_id=R6LCA5WZOEBDURL5S3MEK22YPVYXX6EXSS4J4W57543HELYXOKZA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK): protocol: read tcp 188.166.225.170:58476->77.90.31.10:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621174191 (6h0m17.993753546s) [noise=true , address=77.90.31.16:4768 , piece_id=YNLSONFM6COZEVF6ZNQR2EA6V4LSQFZJONIXR4AFTNNMFGPV57JA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UchewmtxBrtSZSpVAJFdZmcTsQVQxkjK9re6cgUGqDhyonQeF): protocol: read tcp 188.166.225.170:56326->77.90.31.16:4768: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621216842 (6h0m17.993796197s) [noise=true , address=77.90.31.18:4813 , piece_id=VDOT7BB4O7XYJHW6BFTAYQZZYLZU5PAUXVJGSXJ7FDI4TH425COA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128WthA1x9wp1dV1vkpGnoveZQpU9CSgDCZfw5GMQuVpCNSVAM5): protocol: read tcp 188.166.225.170:49704->77.90.31.18:4813: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621234917 (6h0m17.993814272s) [seg_pos={0 0}, upload=4769]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 19 04:51:11.621301444 (6h0m17.993880799s) [upload=4769]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 19 04:51:11.621305402 (6h0m17.993884758s) [upload=4769]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 19 04:51:11.923136970 (6h0m18.295716325s) [upload=4769]: Done uploading segments... Jul 19 04:51:11.923156627 (6h0m18.295735972s) [upload=4769]: Upload finished. err: <nil> Duration: 16.814888079s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [......................XX...X...X....X..X....XXXXX.X.X.....XX] duration: 30.233891703s FAIL (2025-07-19 05:14:01 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== network connection; write tcp 188.166.225.170:39440->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39440->77.90.31.17:4771: use of closed network connection Jul 19 05:13:30.499725691 (6h22m36.872305046s) [seg_pos={0 0}, upload=5061]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 19 05:13:30.730093449 (6h22m37.102672804s) [upload=5061]: Done uploading segments... Jul 19 05:13:30.730128235 (6h22m37.10270759s) [upload=5061]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Jul 19 05:13:31.615702822 (6h22m37.988282217s) [upload=5064]: Starting upload Jul 19 05:13:31.615841878 (6h22m37.988421233s) [upload=5064]: Done starting upload Jul 19 05:13:31.615921711 (6h22m37.988501106s) [upload=5064]: Uploading segments... Jul 19 05:13:31.616305465 (6h22m37.98888485s) [upload=5064]: Got next segment. Inline: false Jul 19 05:13:31.616375579 (6h22m37.988954964s) [upload=5064]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 05:13:31.616384025 (6h22m37.988963421s) [upload=5064]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 05:13:31.872093568 (6h22m38.244672923s) [seg_pos={0 0}, upload=5064]: Begin upload segment... Jul 19 05:13:31.872131390 (6h22m38.244710735s) [seg_pos={0 0}, upload=5064]: Done begin upload segment. Jul 19 05:13:31.872182527 (6h22m38.244761883s) [noise=true , address=77.90.31.14:4662 , piece_id=7GKZYRTV7JJFMKEE3HHZKGRBBK22UHWS6HF3MGGNPHQBMVW5UZ7Q, seg_pos={0 0}, upload=5064]: Uploading piece... Jul 19 05:13:31.872276428 (6h22m38.244855823s) [noise=true , address=45.91.116.10:4081 , piece_id=YDEAU72ZQKBHGX4R2BOOLJ6EDRCAJGZ7JLJV76A3QA3P7E537XOQ, seg_pos={0 0}, upload=5064]: Uploading piece... Jul 19 05:13:31.872562415 (6h22m38.24514177s) [noise=true , address=45.91.116.9:4261 , piece_id=WUVMOJPZX7L2YJP3DHMOGYO2RESZ6PL7XLP3E5PT2WHM5XBPDTXQ, seg_pos={0 0}, upload=5064]: Uploading piece... Jul 19 05:13:31.872861657 (6h22m38.245441012s) [noise=true , address=45.91.116.6:4975 , piece_id=ZWUNYTCKQXERWQTQJCOB4VV65JROS3CCHODQ5NZ77P5NK7P22UOQ, seg_pos={0 0}, upload=5064]: Uploading piece... Jul 19 05:13:31.873164256 (6h22m38.245743611s) [noise=true , address=77.90.31.17:4772 , piece_id=YMOC4YRTBQGXIPSBQRO5NGIIVRIZOB6GAUHRBSEYNUEYC3PEC6AQ, seg_pos={0 0}, upload=5064]: Uploading piece... Jul 19 05:13:31.873213661 (6h22m38.245793026s) [noise=true , address=45.91.116.4:4937 , piece_id=PLHQNIOBAOMV7LGCSZG4WI4WNVOJFIIZN634ISJEK2TAA2OYHJ5Q, seg_pos={0 0}, upload=5064]: Uploading piece... Jul 19 05:13:31.873502404 (6h22m38.246081759s) [noise=true , address=77.90.31.15:4698 , piece_id=FMKNJIQXLX4DJBMLA22ZHYXNL4XJS6GL5XFPO4I2MCSADEUSEKUQ, seg_pos={0 0}, upload=5064]: Uploading piece... Jul 19 05:13:31.873787228 (6h22m38.246366583s) [noise=true , address=45.91.116.5:4299 , piece_id=KMRMQCO7DEUVM6O3U5SKWRWU53AYEPKVAV5DGF5OPMUPSDR5UFSQ, seg_pos={0 0}, upload=5064]: Uploading piece... Jul 19 05:13:31.873831894 (6h22m38.246411249s) [noise=true , address=77.90.31.18:4806 , piece_id=C6PH5INGJYAGWOMZXHD4YCIEBKJ3ENPPSDU2TRI4YXUUJZBNGNLA, seg_pos={0 0}, upload=5064]: Uploading piece... Jul 19 05:13:31.874090849 (6h22m38.246670204s) [noise=true , address=104.36.82.4:4801 , piece_id=H4RIRSEKFP2PQSASUHXTBXGYGO6V25VQSJTM5DAAXQPT2VULJWHA, seg_pos={0 0}, upload=5064]: Uploading piece... Jul 19 05:13:31.874289600 (6h22m38.246868955s) [noise=true , address=45.91.116.8:4045 , piece_id=MVFBXMEJAW4GTWCHM2JLDGH53PS7W7ITLPQOGI7RJ3OWONAWI66Q, seg_pos={0 0}, upload=5064]: Uploading piece... Jul 19 05:13:31.874332071 (6h22m38.246911426s) [noise=true , address=45.91.116.13:4155 , piece_id=OOHUAZ76KC472TE56LLLOKNBTPX7MXO54GUICIIFU5COLN22NFBA, seg_pos={0 0}, upload=5064]: Uploading piece... Jul 19 05:13:31.874767705 (6h22m38.24734706s) [noise=true , address=45.91.116.6:4974 , piece_id=3EQ7MFBZLAOV3YYUZPDZRVEGQN6RXIJSTEL2AQOBW6ZM3XMD7IYA, seg_pos={0 0}, upload=5064]: Uploading piece... Jul 19 05:13:31.875296988 (6h22m38.247876343s) [noise=true , address=104.36.82.5:4840 , piece_id=WTZIC6PVZX3GFGQ3TAI5MS6ZONYRQOHL5B4GTYDGOM337RP5AJUA, seg_pos={0 0}, upload=5064]: Uploading piece... Jul 19 05:13:31.875854935 (6h22m38.24843429s) [noise=true , address=77.90.31.16:4735 , piece_id=XIIFNPKJQXW2BE3C3WNVVCSKTFK6N2HP7YTYQMRV5KB6VZITTTXQ, seg_pos={0 0}, upload=5064]: Uploading piece... Jul 19 05:13:31.876996680 (6h22m38.249576035s) [noise=true , address=77.90.31.14:4661 , piece_id=HF4PBASCVUG37QJ647CJCYC567ECPWV2QUFC2QVVO46XJXZ4HTCA, seg_pos={0 0}, upload=5064]: Uploading piece... Jul 19 05:13:33.562404160 (6h22m39.934983515s) [upload=5064]: Next returned nil segment Jul 19 05:13:33.562420322 (6h22m39.934999677s) [upload=5064]: Waiting for error group managing segments... Jul 19 05:13:33.913312323 (6h22m40.285891678s) [noise=true , address=45.91.116.9:4261 , piece_id=WUVMOJPZX7L2YJP3DHMOGYO2RESZ6PL7XLP3E5PT2WHM5XBPDTXQ, seg_pos={0 0}, upload=5064]: Done uploading piece. err: <nil> Jul 19 05:13:33.919093626 (6h22m40.291672981s) [noise=true , address=45.91.116.6:4975 , piece_id=ZWUNYTCKQXERWQTQJCOB4VV65JROS3CCHODQ5NZ77P5NK7P22UOQ, seg_pos={0 0}, upload=5064]: Done uploading piece. err: <nil> Jul 19 05:13:33.927887905 (6h22m40.30046726s) [noise=true , address=45.91.116.5:4299 , piece_id=KMRMQCO7DEUVM6O3U5SKWRWU53AYEPKVAV5DGF5OPMUPSDR5UFSQ, seg_pos={0 0}, upload=5064]: Done uploading piece. err: <nil> Jul 19 05:13:33.931433800 (6h22m40.304013155s) [noise=true , address=45.91.116.10:4081 , piece_id=YDEAU72ZQKBHGX4R2BOOLJ6EDRCAJGZ7JLJV76A3QA3P7E537XOQ, seg_pos={0 0}, upload=5064]: Done uploading piece. err: <nil> Jul 19 05:13:33.935178076 (6h22m40.307757431s) [noise=true , address=45.91.116.6:4974 , piece_id=3EQ7MFBZLAOV3YYUZPDZRVEGQN6RXIJSTEL2AQOBW6ZM3XMD7IYA, seg_pos={0 0}, upload=5064]: Done uploading piece. err: <nil> Jul 19 05:13:33.945748795 (6h22m40.31832815s) [noise=true , address=45.91.116.8:4045 , piece_id=MVFBXMEJAW4GTWCHM2JLDGH53PS7W7ITLPQOGI7RJ3OWONAWI66Q, seg_pos={0 0}, upload=5064]: Done uploading piece. err: <nil> Jul 19 05:13:33.947812284 (6h22m40.320391639s) [noise=true , address=45.91.116.4:4937 , piece_id=PLHQNIOBAOMV7LGCSZG4WI4WNVOJFIIZN634ISJEK2TAA2OYHJ5Q, seg_pos={0 0}, upload=5064]: Done uploading piece. err: <nil> Jul 19 05:13:34.080127952 (6h22m40.452707307s) [noise=true , address=45.91.116.13:4155 , piece_id=OOHUAZ76KC472TE56LLLOKNBTPX7MXO54GUICIIFU5COLN22NFBA, seg_pos={0 0}, upload=5064]: Done uploading piece. err: <nil> Jul 19 05:13:34.082690075 (6h22m40.45526943s) [noise=true , address=104.36.82.5:4840 , piece_id=WTZIC6PVZX3GFGQ3TAI5MS6ZONYRQOHL5B4GTYDGOM337RP5AJUA, seg_pos={0 0}, upload=5064]: Done uploading piece. err: <nil> Jul 19 05:13:34.087918400 (6h22m40.460497755s) [noise=true , address=104.36.82.4:4801 , piece_id=H4RIRSEKFP2PQSASUHXTBXGYGO6V25VQSJTM5DAAXQPT2VULJWHA, seg_pos={0 0}, upload=5064]: Done uploading piece. err: <nil> Jul 19 05:13:51.594432847 (6h22m57.967012202s) [noise=true , address=77.90.31.15:4698 , piece_id=FMKNJIQXLX4DJBMLA22ZHYXNL4XJS6GL5XFPO4I2MCSADEUSEKUQ, seg_pos={0 0}, upload=5064]: Done uploading piece. err: <nil> Jul 19 05:13:55.163396647 (6h23m1.535976002s) [noise=true , address=77.90.31.14:4661 , piece_id=HF4PBASCVUG37QJ647CJCYC567ECPWV2QUFC2QVVO46XJXZ4HTCA, seg_pos={0 0}, upload=5064]: Done uploading piece. err: <nil> Jul 19 05:14:00.966255878 (6h23m7.338835233s) [noise=true , address=77.90.31.18:4806 , piece_id=C6PH5INGJYAGWOMZXHD4YCIEBKJ3ENPPSDU2TRI4YXUUJZBNGNLA, seg_pos={0 0}, upload=5064]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: expected piece hash; context deadline exceeded; EOF Jul 19 05:14:00.966369235 (6h23m7.33894859s) [noise=true , address=77.90.31.14:4662 , piece_id=7GKZYRTV7JJFMKEE3HHZKGRBBK22UHWS6HF3MGGNPHQBMVW5UZ7Q, seg_pos={0 0}, upload=5064]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: write tcp 188.166.225.170:56328->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:56328->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56328->77.90.31.14:4662: use of closed network connection Jul 19 05:14:00.966401958 (6h23m7.338981313s) [noise=true , address=77.90.31.16:4735 , piece_id=XIIFNPKJQXW2BE3C3WNVVCSKTFK6N2HP7YTYQMRV5KB6VZITTTXQ, seg_pos={0 0}, upload=5064]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:58392->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:58392->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58392->77.90.31.16:4735: use of closed network connection Jul 19 05:14:00.966414441 (6h23m7.338993797s) [noise=true , address=77.90.31.17:4772 , piece_id=YMOC4YRTBQGXIPSBQRO5NGIIVRIZOB6GAUHRBSEYNUEYC3PEC6AQ, seg_pos={0 0}, upload=5064]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:50432->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:50432->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50432->77.90.31.17:4772: use of closed network connection Jul 19 05:14:00.966441754 (6h23m7.339021109s) [seg_pos={0 0}, upload=5064]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 19 05:14:01.197829382 (6h23m7.570408747s) [upload=5064]: Done uploading segments... Jul 19 05:14:01.197859059 (6h23m7.570438424s) [upload=5064]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.233906382s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [.......................XX...X...X....X..X....XXXXX.X.X.....X] duration: 30.231991163s FAIL (2025-07-19 05:13:30 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 8:4805 , piece_id=WNU3XPTJXHVMLS6VC5EO6K36RMDYVIU35J6EQ2TVRUE4JPZP3PPA, seg_pos={0 0}, upload=5046]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF Jul 19 05:12:11.346253802 (6h21m17.718833157s) [seg_pos={0 0}, upload=5046]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 19 05:12:11.346346349 (6h21m17.718925714s) [upload=5046]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 19 05:12:11.346352841 (6h21m17.718932196s) [upload=5046]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 19 05:12:11.658573508 (6h21m18.031152863s) [upload=5046]: Done uploading segments... Jul 19 05:12:11.658594047 (6h21m18.031173402s) [upload=5046]: Upload finished. err: <nil> Jul 19 05:13:01.140644889 (6h22m7.513224244s) [upload=5061]: Starting upload Jul 19 05:13:01.140727768 (6h22m7.513307123s) [upload=5061]: Done starting upload Jul 19 05:13:01.140851815 (6h22m7.51343117s) [upload=5061]: Uploading segments... Jul 19 05:13:01.140949472 (6h22m7.513528827s) [upload=5061]: Got next segment. Inline: false Jul 19 05:13:01.140984889 (6h22m7.513564244s) [upload=5061]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 05:13:01.140997755 (6h22m7.51357711s) [upload=5061]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 05:13:01.393965340 (6h22m7.766544695s) [seg_pos={0 0}, upload=5061]: Begin upload segment... Jul 19 05:13:01.394009123 (6h22m7.766588478s) [seg_pos={0 0}, upload=5061]: Done begin upload segment. Jul 19 05:13:01.394033981 (6h22m7.766613336s) [noise=true , address=104.36.82.15:4258 , piece_id=K2MYX5D4J6HMHBWJDU6RYKUQEWRZ5EMRX65P7IOLFWI42LI5CIGA, seg_pos={0 0}, upload=5061]: Uploading piece... Jul 19 05:13:01.394103294 (6h22m7.766682659s) [noise=true , address=45.91.116.7:4010 , piece_id=ZQUQBEZ2MYZ76MQROXFMDDT7XRQIRZ6GI6SDFNIQAPRBZV6AB7OA, seg_pos={0 0}, upload=5061]: Uploading piece... Jul 19 05:13:01.394446921 (6h22m7.767026276s) [noise=true , address=45.91.116.12:4226 , piece_id=DY2H2SJOH6BMVAKSRM2HDQCERKJ7BWTW6JXNONU5QMOXKQXXT6MA, seg_pos={0 0}, upload=5061]: Uploading piece... Jul 19 05:13:01.394633318 (6h22m7.767212673s) [noise=true , address=77.90.31.11:4554 , piece_id=ZYOAIZOTTSX2YFVHNSTFO2WJN62NU2ZZMHTO37ZT3X3J36HQE5RA, seg_pos={0 0}, upload=5061]: Uploading piece... Jul 19 05:13:01.394920708 (6h22m7.767500063s) [noise=true , address=77.90.31.17:4771 , piece_id=MUDT6RGUD2ZRHJWCY5OSRAMV5D5AP4MKIZKTFMRXDHSG3YNYPN3Q, seg_pos={0 0}, upload=5061]: Uploading piece... Jul 19 05:13:01.395002735 (6h22m7.76758209s) [noise=true , address=45.91.116.4:4940 , piece_id=GMKBHSP3J3Q52HKK5MAT474IPPLUG3WMPEP7LQ4KM2KB5XZY4CFQ, seg_pos={0 0}, upload=5061]: Uploading piece... Jul 19 05:13:01.395292549 (6h22m7.767871914s) [noise=true , address=45.91.116.12:4227 , piece_id=PTMQUYSFEROMWZHGAIRDMVF3VOUHYNWRTC7NEAHAG5YO6ATGIFJA, seg_pos={0 0}, upload=5061]: Uploading piece... Jul 19 05:13:01.395653711 (6h22m7.768233066s) [noise=true , address=45.91.116.14:4191 , piece_id=QUNQC5RP4FD37YMTP7PYHAL6SSYEJZJMGWE5TGN2THFYKEDNAVJQ, seg_pos={0 0}, upload=5061]: Uploading piece... Jul 19 05:13:01.395676564 (6h22m7.768255919s) [noise=true , address=77.90.31.14:4661 , piece_id=LNPTMD7K3DDMX5N7HM5H4IL4UFAY4N3ORTSZEOGG4C4IVTT6EUJQ, seg_pos={0 0}, upload=5061]: Uploading piece... Jul 19 05:13:01.395895213 (6h22m7.768474568s) [noise=true , address=45.91.116.6:4974 , piece_id=3P5UOXZ2JCBDUR7UQ7SXIJAABU26NF5EWNEL2E77UC5SMI5OYCCQ, seg_pos={0 0}, upload=5061]: Uploading piece... Jul 19 05:13:01.395960017 (6h22m7.768539372s) [noise=true , address=45.91.116.9:4263 , piece_id=BXWDBJIAXEYU2P6POCAB52HI7KPA6HA5SAQNQS2E4CRNLS2LOJ4Q, seg_pos={0 0}, upload=5061]: Uploading piece... Jul 19 05:13:01.396225696 (6h22m7.768805051s) [noise=true , address=77.90.31.10:4520 , piece_id=7DIMJKGHXPP3DMQDFODZDJRGCOVHR26OXFMHRQ4JFR3MBN34D5DQ, seg_pos={0 0}, upload=5061]: Uploading piece... Jul 19 05:13:01.396265582 (6h22m7.768844937s) [noise=true , address=104.36.82.11:4114 , piece_id=MFXVM2QAOQFAFPIEXBHAF7S6WA4TM5MR7DQUEAYJFTVCVCM3E4GQ, seg_pos={0 0}, upload=5061]: Uploading piece... Jul 19 05:13:01.396494310 (6h22m7.769073665s) [noise=true , address=45.91.116.10:4081 , piece_id=F5CLAYNS7J3DODS6XSEUXPZXXLU22EJ7SEFMJ6JE2GHMJ2MZHI6A, seg_pos={0 0}, upload=5061]: Uploading piece... Jul 19 05:13:01.396567389 (6h22m7.769146744s) [noise=true , address=45.91.116.7:4011 , piece_id=KORMU6LCNH3TU64FG7JGZB7KRYZ33XK6GACS5IBWK2MXIU4XLPRA, seg_pos={0 0}, upload=5061]: Uploading piece... Jul 19 05:13:01.396754327 (6h22m7.769333682s) [noise=true , address=45.91.116.11:4117 , piece_id=BBCY4YWMDEX25J3M466YPZ33C7OKZOT77MDTB6I62VCFHUN3KX6A, seg_pos={0 0}, upload=5061]: Uploading piece... Jul 19 05:13:03.153545360 (6h22m9.526124715s) [upload=5061]: Next returned nil segment Jul 19 05:13:03.153560329 (6h22m9.526139674s) [upload=5061]: Waiting for error group managing segments... Jul 19 05:13:03.443194182 (6h22m9.815773537s) [noise=true , address=45.91.116.9:4263 , piece_id=BXWDBJIAXEYU2P6POCAB52HI7KPA6HA5SAQNQS2E4CRNLS2LOJ4Q, seg_pos={0 0}, upload=5061]: Done uploading piece. err: <nil> Jul 19 05:13:03.450560849 (6h22m9.823140204s) [noise=true , address=45.91.116.4:4940 , piece_id=GMKBHSP3J3Q52HKK5MAT474IPPLUG3WMPEP7LQ4KM2KB5XZY4CFQ, seg_pos={0 0}, upload=5061]: Done uploading piece. err: <nil> Jul 19 05:13:03.453309278 (6h22m9.825888633s) [noise=true , address=45.91.116.12:4226 , piece_id=DY2H2SJOH6BMVAKSRM2HDQCERKJ7BWTW6JXNONU5QMOXKQXXT6MA, seg_pos={0 0}, upload=5061]: Done uploading piece. err: <nil> Jul 19 05:13:03.460095565 (6h22m9.83267492s) [noise=true , address=45.91.116.6:4974 , piece_id=3P5UOXZ2JCBDUR7UQ7SXIJAABU26NF5EWNEL2E77UC5SMI5OYCCQ, seg_pos={0 0}, upload=5061]: Done uploading piece. err: <nil> Jul 19 05:13:03.468362225 (6h22m9.84094158s) [noise=true , address=45.91.116.10:4081 , piece_id=F5CLAYNS7J3DODS6XSEUXPZXXLU22EJ7SEFMJ6JE2GHMJ2MZHI6A, seg_pos={0 0}, upload=5061]: Done uploading piece. err: <nil> Jul 19 05:13:03.470190894 (6h22m9.84277025s) [noise=true , address=45.91.116.7:4011 , piece_id=KORMU6LCNH3TU64FG7JGZB7KRYZ33XK6GACS5IBWK2MXIU4XLPRA, seg_pos={0 0}, upload=5061]: Done uploading piece. err: <nil> Jul 19 05:13:03.546087355 (6h22m9.91866671s) [noise=true , address=45.91.116.12:4227 , piece_id=PTMQUYSFEROMWZHGAIRDMVF3VOUHYNWRTC7NEAHAG5YO6ATGIFJA, seg_pos={0 0}, upload=5061]: Done uploading piece. err: <nil> Jul 19 05:13:03.553001787 (6h22m9.925581142s) [noise=true , address=45.91.116.7:4010 , piece_id=ZQUQBEZ2MYZ76MQROXFMDDT7XRQIRZ6GI6SDFNIQAPRBZV6AB7OA, seg_pos={0 0}, upload=5061]: Done uploading piece. err: <nil> Jul 19 05:13:03.586073515 (6h22m9.95865287s) [noise=true , address=104.36.82.11:4114 , piece_id=MFXVM2QAOQFAFPIEXBHAF7S6WA4TM5MR7DQUEAYJFTVCVCM3E4GQ, seg_pos={0 0}, upload=5061]: Done uploading piece. err: <nil> Jul 19 05:13:03.608633908 (6h22m9.981213263s) [noise=true , address=45.91.116.11:4117 , piece_id=BBCY4YWMDEX25J3M466YPZ33C7OKZOT77MDTB6I62VCFHUN3KX6A, seg_pos={0 0}, upload=5061]: Done uploading piece. err: <nil> Jul 19 05:13:03.613407333 (6h22m9.985986698s) [noise=true , address=45.91.116.14:4191 , piece_id=QUNQC5RP4FD37YMTP7PYHAL6SSYEJZJMGWE5TGN2THFYKEDNAVJQ, seg_pos={0 0}, upload=5061]: Done uploading piece. err: <nil> Jul 19 05:13:03.621088622 (6h22m9.993667977s) [noise=true , address=104.36.82.15:4258 , piece_id=K2MYX5D4J6HMHBWJDU6RYKUQEWRZ5EMRX65P7IOLFWI42LI5CIGA, seg_pos={0 0}, upload=5061]: Done uploading piece. err: <nil> Jul 19 05:13:30.499554093 (6h22m36.872133438s) [noise=true , address=77.90.31.14:4661 , piece_id=LNPTMD7K3DDMX5N7HM5H4IL4UFAY4N3ORTSZEOGG4C4IVTT6EUJQ, seg_pos={0 0}, upload=5061]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: expected piece hash; context deadline exceeded; EOF Jul 19 05:13:30.499588759 (6h22m36.872168114s) [noise=true , address=77.90.31.10:4520 , piece_id=7DIMJKGHXPP3DMQDFODZDJRGCOVHR26OXFMHRQ4JFR3MBN34D5DQ, seg_pos={0 0}, upload=5061]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: expected piece hash; context deadline exceeded; EOF Jul 19 05:13:30.499668452 (6h22m36.872247807s) [noise=true , address=77.90.31.11:4554 , piece_id=ZYOAIZOTTSX2YFVHNSTFO2WJN62NU2ZZMHTO37ZT3X3J36HQE5RA, seg_pos={0 0}, upload=5061]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:42192->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:42192->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42192->77.90.31.11:4554: use of closed network connection Jul 19 05:13:30.499708157 (6h22m36.872287512s) [noise=true , address=77.90.31.17:4771 , piece_id=MUDT6RGUD2ZRHJWCY5OSRAMV5D5AP4MKIZKTFMRXDHSG3YNYPN3Q, seg_pos={0 0}, upload=5061]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 188.166.225.170:39440->77.90.31.17:4771: use of closed network connection; write tcp 188.166.225.170:39440->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39440->77.90.31.17:4771: use of closed network connection Jul 19 05:13:30.499725691 (6h22m36.872305046s) [seg_pos={0 0}, upload=5061]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 19 05:13:30.730093449 (6h22m37.102672804s) [upload=5061]: Done uploading segments... Jul 19 05:13:30.730128235 (6h22m37.10270759s) [upload=5061]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.232003126s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 9.430533751s FAIL (2025-07-19 05:12:44 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 4855s) [upload=4769]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 04:51:09.109709627 (6h0m15.482288982s) [upload=4769]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 04:51:09.556145753 (6h0m15.928725108s) [seg_pos={0 0}, upload=4769]: Begin upload segment... Jul 19 04:51:09.556194817 (6h0m15.928774172s) [seg_pos={0 0}, upload=4769]: Done begin upload segment. Jul 19 04:51:09.556223893 (6h0m15.928803248s) [noise=true , address=79.127.219.38:4023 , piece_id=PTPOZGLYV4MZTOFZNN3NCGYL7DFAST4PXKEUIVTMFL7HV4GZTCLQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556252197 (6h0m15.928831552s) [noise=true , address=45.91.95.4:4011 , piece_id=ZMBAS6KKHYVIRE6OBL2NFVCWMM4TKQ7J64BD3SYXIK54XCEDI2XA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556579383 (6h0m15.929158738s) [noise=true , address=45.91.116.6:4007 , piece_id=A2AFVTZBXSGBOE7XIM3B5HRN4STITFBXO6MZRJ365URNAAOZ3PSQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556854469 (6h0m15.929433824s) [noise=true , address=104.36.82.19:4423 , piece_id=PIJUHDUND36HQL67I33D7A3EULMJHS7N55MN4OFQJ4OWNQ42O47A, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557094107 (6h0m15.929673462s) [noise=true , address=45.91.116.9:4264 , piece_id=GHM5BKZKMT2QOHHBOS4FSPOWEWSFQRGRVVR4GSNDPHVIFRZHQL2A, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557336381 (6h0m15.929915736s) [noise=true , address=79.127.205.235:4660 , piece_id=UOUOGE2VP7PX7EMRZLPMCWFCYEZERM3DQMCQ6NG4J2K2DEIZR63Q, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557557885 (6h0m15.93013724s) [noise=true , address=79.127.205.246:4280 , piece_id=FLBOZI3UGTQXAQBLZ2F45Z3OEYL65F5PQBXRF2DQMZ7NGF7VDRPA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557762296 (6h0m15.930341651s) [noise=true , address=79.127.205.233:4622 , piece_id=6WWNYQTH5MGAJOEB7BXKZDJOMCAZXDW66J3OEANGDNJDNUMLZM7Q, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558014689 (6h0m15.930594044s) [noise=true , address=64.49.13.138:4482 , piece_id=MAW2M4US2OQNOS3WI67AA7NIBTJDE5T2JQTXKFPBLGVGXKVCFGTQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558215974 (6h0m15.930795319s) [noise=true , address=77.90.31.10:4546 , piece_id=R6LCA5WZOEBDURL5S3MEK22YPVYXX6EXSS4J4W57543HELYXOKZA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558409364 (6h0m15.930988719s) [noise=true , address=45.91.95.5:4019 , piece_id=YTSBCN2LPHEMRRAZIHJGVB7NBOTRXVEHD2TWUCTTLBNFFYEWWDKA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558623504 (6h0m15.931202859s) [noise=true , address=79.127.201.215:4357 , piece_id=75XWAG2VQWEYCWN2Q3WRIZSBI7LDOISZJPXQLQ2GIPNEN34VLSUQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558805162 (6h0m15.931384517s) [noise=true , address=45.91.116.14:4219 , piece_id=PDW6JPRHCVMHHOWIHT6JKV4CTSC3JY6NDKDVFZSO47OYNPZ7PGGA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558999373 (6h0m15.931578728s) [noise=true , address=77.90.31.11:4564 , piece_id=L2U7JFJRDQOFA2AYVAGTI4ZW4VABCXKMXWBIUWQO2QNNUUBJ74BA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559184768 (6h0m15.931764123s) [noise=true , address=77.90.31.16:4768 , piece_id=YNLSONFM6COZEVF6ZNQR2EA6V4LSQFZJONIXR4AFTNNMFGPV57JA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559368719 (6h0m15.931948074s) [noise=true , address=79.127.220.98:4432 , piece_id=2SUTFQ7FVDPAM2H4W3OLW6AVH3VJ33IJFE4L52SA3ET5LGBMOMPQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559549706 (6h0m15.932129061s) [noise=true , address=77.90.31.18:4813 , piece_id=VDOT7BB4O7XYJHW6BFTAYQZZYLZU5PAUXVJGSXJ7FDI4TH425COA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559727836 (6h0m15.932307191s) [noise=true , address=79.127.219.51:4211 , piece_id=HO5PLWBVU5YAYNYMN352NA73SM4WBJV4WZ7XLSQTCQ76UAL2TOJA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:10.663359087 (6h0m17.035938442s) [upload=4769]: Next returned nil segment Jul 19 04:51:10.663378664 (6h0m17.035958019s) [upload=4769]: Waiting for error group managing segments... Jul 19 04:51:10.962945479 (6h0m17.335524834s) [noise=true , address=79.127.205.233:4622 , piece_id=6WWNYQTH5MGAJOEB7BXKZDJOMCAZXDW66J3OEANGDNJDNUMLZM7Q, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:10.967689807 (6h0m17.340269152s) [noise=true , address=79.127.205.235:4660 , piece_id=UOUOGE2VP7PX7EMRZLPMCWFCYEZERM3DQMCQ6NG4J2K2DEIZR63Q, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:10.976968631 (6h0m17.349547986s) [noise=true , address=64.49.13.138:4482 , piece_id=MAW2M4US2OQNOS3WI67AA7NIBTJDE5T2JQTXKFPBLGVGXKVCFGTQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.048750700 (6h0m17.421330055s) [noise=true , address=79.127.205.246:4280 , piece_id=FLBOZI3UGTQXAQBLZ2F45Z3OEYL65F5PQBXRF2DQMZ7NGF7VDRPA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.221642403 (6h0m17.594221758s) [noise=true , address=79.127.220.98:4432 , piece_id=2SUTFQ7FVDPAM2H4W3OLW6AVH3VJ33IJFE4L52SA3ET5LGBMOMPQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.317070435 (6h0m17.68964979s) [noise=true , address=45.91.95.4:4011 , piece_id=ZMBAS6KKHYVIRE6OBL2NFVCWMM4TKQ7J64BD3SYXIK54XCEDI2XA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.324002660 (6h0m17.696582015s) [noise=true , address=45.91.95.5:4019 , piece_id=YTSBCN2LPHEMRRAZIHJGVB7NBOTRXVEHD2TWUCTTLBNFFYEWWDKA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.460976317 (6h0m17.833555672s) [noise=true , address=79.127.201.215:4357 , piece_id=75XWAG2VQWEYCWN2Q3WRIZSBI7LDOISZJPXQLQ2GIPNEN34VLSUQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.462584603 (6h0m17.835163958s) [noise=true , address=45.91.116.6:4007 , piece_id=A2AFVTZBXSGBOE7XIM3B5HRN4STITFBXO6MZRJ365URNAAOZ3PSQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.490392973 (6h0m17.862972328s) [noise=true , address=45.91.116.9:4264 , piece_id=GHM5BKZKMT2QOHHBOS4FSPOWEWSFQRGRVVR4GSNDPHVIFRZHQL2A, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.502040780 (6h0m17.874620135s) [noise=true , address=45.91.116.14:4219 , piece_id=PDW6JPRHCVMHHOWIHT6JKV4CTSC3JY6NDKDVFZSO47OYNPZ7PGGA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.600434096 (6h0m17.973013451s) [noise=true , address=79.127.219.51:4211 , piece_id=HO5PLWBVU5YAYNYMN352NA73SM4WBJV4WZ7XLSQTCQ76UAL2TOJA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.609008984 (6h0m17.981588339s) [noise=true , address=104.36.82.19:4423 , piece_id=PIJUHDUND36HQL67I33D7A3EULMJHS7N55MN4OFQJ4OWNQ42O47A, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.620825273 (6h0m17.993404628s) [noise=true , address=79.127.219.38:4023 , piece_id=PTPOZGLYV4MZTOFZNN3NCGYL7DFAST4PXKEUIVTMFL7HV4GZTCLQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.620838659 (6h0m17.993418014s) [seg_pos={0 0}, upload=4769]: Segment reached optimal threshold of 14 pieces. Jul 19 04:51:11.621067807 (6h0m17.993647163s) [noise=true , address=77.90.31.11:4564 , piece_id=L2U7JFJRDQOFA2AYVAGTI4ZW4VABCXKMXWBIUWQO2QNNUUBJ74BA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection; write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection Jul 19 04:51:11.621134234 (6h0m17.993713589s) [noise=true , address=77.90.31.10:4546 , piece_id=R6LCA5WZOEBDURL5S3MEK22YPVYXX6EXSS4J4W57543HELYXOKZA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK): protocol: read tcp 188.166.225.170:58476->77.90.31.10:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621174191 (6h0m17.993753546s) [noise=true , address=77.90.31.16:4768 , piece_id=YNLSONFM6COZEVF6ZNQR2EA6V4LSQFZJONIXR4AFTNNMFGPV57JA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UchewmtxBrtSZSpVAJFdZmcTsQVQxkjK9re6cgUGqDhyonQeF): protocol: read tcp 188.166.225.170:56326->77.90.31.16:4768: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621216842 (6h0m17.993796197s) [noise=true , address=77.90.31.18:4813 , piece_id=VDOT7BB4O7XYJHW6BFTAYQZZYLZU5PAUXVJGSXJ7FDI4TH425COA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128WthA1x9wp1dV1vkpGnoveZQpU9CSgDCZfw5GMQuVpCNSVAM5): protocol: read tcp 188.166.225.170:49704->77.90.31.18:4813: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621234917 (6h0m17.993814272s) [seg_pos={0 0}, upload=4769]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 19 04:51:11.621301444 (6h0m17.993880799s) [upload=4769]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 19 04:51:11.621305402 (6h0m17.993884758s) [upload=4769]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 19 04:51:11.923136970 (6h0m18.295716325s) [upload=4769]: Done uploading segments... Jul 19 04:51:11.923156627 (6h0m18.295735972s) [upload=4769]: Upload finished. err: <nil> Duration: 9.430565483s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 8.368369355s FAIL (2025-07-19 05:12:20 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 4855s) [upload=4769]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 04:51:09.109709627 (6h0m15.482288982s) [upload=4769]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 04:51:09.556145753 (6h0m15.928725108s) [seg_pos={0 0}, upload=4769]: Begin upload segment... Jul 19 04:51:09.556194817 (6h0m15.928774172s) [seg_pos={0 0}, upload=4769]: Done begin upload segment. Jul 19 04:51:09.556223893 (6h0m15.928803248s) [noise=true , address=79.127.219.38:4023 , piece_id=PTPOZGLYV4MZTOFZNN3NCGYL7DFAST4PXKEUIVTMFL7HV4GZTCLQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556252197 (6h0m15.928831552s) [noise=true , address=45.91.95.4:4011 , piece_id=ZMBAS6KKHYVIRE6OBL2NFVCWMM4TKQ7J64BD3SYXIK54XCEDI2XA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556579383 (6h0m15.929158738s) [noise=true , address=45.91.116.6:4007 , piece_id=A2AFVTZBXSGBOE7XIM3B5HRN4STITFBXO6MZRJ365URNAAOZ3PSQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556854469 (6h0m15.929433824s) [noise=true , address=104.36.82.19:4423 , piece_id=PIJUHDUND36HQL67I33D7A3EULMJHS7N55MN4OFQJ4OWNQ42O47A, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557094107 (6h0m15.929673462s) [noise=true , address=45.91.116.9:4264 , piece_id=GHM5BKZKMT2QOHHBOS4FSPOWEWSFQRGRVVR4GSNDPHVIFRZHQL2A, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557336381 (6h0m15.929915736s) [noise=true , address=79.127.205.235:4660 , piece_id=UOUOGE2VP7PX7EMRZLPMCWFCYEZERM3DQMCQ6NG4J2K2DEIZR63Q, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557557885 (6h0m15.93013724s) [noise=true , address=79.127.205.246:4280 , piece_id=FLBOZI3UGTQXAQBLZ2F45Z3OEYL65F5PQBXRF2DQMZ7NGF7VDRPA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557762296 (6h0m15.930341651s) [noise=true , address=79.127.205.233:4622 , piece_id=6WWNYQTH5MGAJOEB7BXKZDJOMCAZXDW66J3OEANGDNJDNUMLZM7Q, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558014689 (6h0m15.930594044s) [noise=true , address=64.49.13.138:4482 , piece_id=MAW2M4US2OQNOS3WI67AA7NIBTJDE5T2JQTXKFPBLGVGXKVCFGTQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558215974 (6h0m15.930795319s) [noise=true , address=77.90.31.10:4546 , piece_id=R6LCA5WZOEBDURL5S3MEK22YPVYXX6EXSS4J4W57543HELYXOKZA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558409364 (6h0m15.930988719s) [noise=true , address=45.91.95.5:4019 , piece_id=YTSBCN2LPHEMRRAZIHJGVB7NBOTRXVEHD2TWUCTTLBNFFYEWWDKA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558623504 (6h0m15.931202859s) [noise=true , address=79.127.201.215:4357 , piece_id=75XWAG2VQWEYCWN2Q3WRIZSBI7LDOISZJPXQLQ2GIPNEN34VLSUQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558805162 (6h0m15.931384517s) [noise=true , address=45.91.116.14:4219 , piece_id=PDW6JPRHCVMHHOWIHT6JKV4CTSC3JY6NDKDVFZSO47OYNPZ7PGGA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558999373 (6h0m15.931578728s) [noise=true , address=77.90.31.11:4564 , piece_id=L2U7JFJRDQOFA2AYVAGTI4ZW4VABCXKMXWBIUWQO2QNNUUBJ74BA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559184768 (6h0m15.931764123s) [noise=true , address=77.90.31.16:4768 , piece_id=YNLSONFM6COZEVF6ZNQR2EA6V4LSQFZJONIXR4AFTNNMFGPV57JA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559368719 (6h0m15.931948074s) [noise=true , address=79.127.220.98:4432 , piece_id=2SUTFQ7FVDPAM2H4W3OLW6AVH3VJ33IJFE4L52SA3ET5LGBMOMPQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559549706 (6h0m15.932129061s) [noise=true , address=77.90.31.18:4813 , piece_id=VDOT7BB4O7XYJHW6BFTAYQZZYLZU5PAUXVJGSXJ7FDI4TH425COA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559727836 (6h0m15.932307191s) [noise=true , address=79.127.219.51:4211 , piece_id=HO5PLWBVU5YAYNYMN352NA73SM4WBJV4WZ7XLSQTCQ76UAL2TOJA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:10.663359087 (6h0m17.035938442s) [upload=4769]: Next returned nil segment Jul 19 04:51:10.663378664 (6h0m17.035958019s) [upload=4769]: Waiting for error group managing segments... Jul 19 04:51:10.962945479 (6h0m17.335524834s) [noise=true , address=79.127.205.233:4622 , piece_id=6WWNYQTH5MGAJOEB7BXKZDJOMCAZXDW66J3OEANGDNJDNUMLZM7Q, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:10.967689807 (6h0m17.340269152s) [noise=true , address=79.127.205.235:4660 , piece_id=UOUOGE2VP7PX7EMRZLPMCWFCYEZERM3DQMCQ6NG4J2K2DEIZR63Q, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:10.976968631 (6h0m17.349547986s) [noise=true , address=64.49.13.138:4482 , piece_id=MAW2M4US2OQNOS3WI67AA7NIBTJDE5T2JQTXKFPBLGVGXKVCFGTQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.048750700 (6h0m17.421330055s) [noise=true , address=79.127.205.246:4280 , piece_id=FLBOZI3UGTQXAQBLZ2F45Z3OEYL65F5PQBXRF2DQMZ7NGF7VDRPA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.221642403 (6h0m17.594221758s) [noise=true , address=79.127.220.98:4432 , piece_id=2SUTFQ7FVDPAM2H4W3OLW6AVH3VJ33IJFE4L52SA3ET5LGBMOMPQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.317070435 (6h0m17.68964979s) [noise=true , address=45.91.95.4:4011 , piece_id=ZMBAS6KKHYVIRE6OBL2NFVCWMM4TKQ7J64BD3SYXIK54XCEDI2XA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.324002660 (6h0m17.696582015s) [noise=true , address=45.91.95.5:4019 , piece_id=YTSBCN2LPHEMRRAZIHJGVB7NBOTRXVEHD2TWUCTTLBNFFYEWWDKA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.460976317 (6h0m17.833555672s) [noise=true , address=79.127.201.215:4357 , piece_id=75XWAG2VQWEYCWN2Q3WRIZSBI7LDOISZJPXQLQ2GIPNEN34VLSUQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.462584603 (6h0m17.835163958s) [noise=true , address=45.91.116.6:4007 , piece_id=A2AFVTZBXSGBOE7XIM3B5HRN4STITFBXO6MZRJ365URNAAOZ3PSQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.490392973 (6h0m17.862972328s) [noise=true , address=45.91.116.9:4264 , piece_id=GHM5BKZKMT2QOHHBOS4FSPOWEWSFQRGRVVR4GSNDPHVIFRZHQL2A, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.502040780 (6h0m17.874620135s) [noise=true , address=45.91.116.14:4219 , piece_id=PDW6JPRHCVMHHOWIHT6JKV4CTSC3JY6NDKDVFZSO47OYNPZ7PGGA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.600434096 (6h0m17.973013451s) [noise=true , address=79.127.219.51:4211 , piece_id=HO5PLWBVU5YAYNYMN352NA73SM4WBJV4WZ7XLSQTCQ76UAL2TOJA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.609008984 (6h0m17.981588339s) [noise=true , address=104.36.82.19:4423 , piece_id=PIJUHDUND36HQL67I33D7A3EULMJHS7N55MN4OFQJ4OWNQ42O47A, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.620825273 (6h0m17.993404628s) [noise=true , address=79.127.219.38:4023 , piece_id=PTPOZGLYV4MZTOFZNN3NCGYL7DFAST4PXKEUIVTMFL7HV4GZTCLQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.620838659 (6h0m17.993418014s) [seg_pos={0 0}, upload=4769]: Segment reached optimal threshold of 14 pieces. Jul 19 04:51:11.621067807 (6h0m17.993647163s) [noise=true , address=77.90.31.11:4564 , piece_id=L2U7JFJRDQOFA2AYVAGTI4ZW4VABCXKMXWBIUWQO2QNNUUBJ74BA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection; write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection Jul 19 04:51:11.621134234 (6h0m17.993713589s) [noise=true , address=77.90.31.10:4546 , piece_id=R6LCA5WZOEBDURL5S3MEK22YPVYXX6EXSS4J4W57543HELYXOKZA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK): protocol: read tcp 188.166.225.170:58476->77.90.31.10:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621174191 (6h0m17.993753546s) [noise=true , address=77.90.31.16:4768 , piece_id=YNLSONFM6COZEVF6ZNQR2EA6V4LSQFZJONIXR4AFTNNMFGPV57JA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UchewmtxBrtSZSpVAJFdZmcTsQVQxkjK9re6cgUGqDhyonQeF): protocol: read tcp 188.166.225.170:56326->77.90.31.16:4768: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621216842 (6h0m17.993796197s) [noise=true , address=77.90.31.18:4813 , piece_id=VDOT7BB4O7XYJHW6BFTAYQZZYLZU5PAUXVJGSXJ7FDI4TH425COA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128WthA1x9wp1dV1vkpGnoveZQpU9CSgDCZfw5GMQuVpCNSVAM5): protocol: read tcp 188.166.225.170:49704->77.90.31.18:4813: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621234917 (6h0m17.993814272s) [seg_pos={0 0}, upload=4769]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 19 04:51:11.621301444 (6h0m17.993880799s) [upload=4769]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 19 04:51:11.621305402 (6h0m17.993884758s) [upload=4769]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 19 04:51:11.923136970 (6h0m18.295716325s) [upload=4769]: Done uploading segments... Jul 19 04:51:11.923156627 (6h0m18.295735972s) [upload=4769]: Upload finished. err: <nil> Duration: 8.368400405s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [........................XX...X...X....X..X....XXXXX.X.X.....] duration: 7.879249541s FAIL (2025-07-19 05:12:14 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== : <nil> Jul 19 05:08:29.800544823 (6h17m36.173124169s) [upload=4997]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 19 05:08:29.800548410 (6h17m36.173127765s) [upload=4997]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 19 05:08:30.148027600 (6h17m36.520606955s) [upload=4997]: Done uploading segments... Jul 19 05:08:30.148148401 (6h17m36.520727746s) [upload=4997]: Upload finished. err: <nil> Jul 19 05:08:48.807983386 (6h17m55.180562741s) [upload=5005]: Starting upload Jul 19 05:08:48.808044453 (6h17m55.180623808s) [upload=5005]: Done starting upload Jul 19 05:08:48.808175243 (6h17m55.180754598s) [upload=5005]: Uploading segments... Jul 19 05:08:48.808334198 (6h17m55.180913553s) [upload=5005]: Got next segment. Inline: false Jul 19 05:08:48.808392509 (6h17m55.180971874s) [upload=5005]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 05:08:48.808399823 (6h17m55.180979188s) [upload=5005]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 05:08:49.075515233 (6h17m55.448094578s) [seg_pos={0 0}, upload=5005]: Begin upload segment... Jul 19 05:08:49.075564547 (6h17m55.448143902s) [seg_pos={0 0}, upload=5005]: Done begin upload segment. Jul 19 05:08:49.075602881 (6h17m55.448182236s) [noise=true , address=77.90.31.11:4555 , piece_id=QGRXVFSKXDCLKSSSXEK4QYZEETWQGFZMQQBSTXNMAHY45RA6MAVQ, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.075612279 (6h17m55.448191634s) [noise=true , address=104.36.82.13:4186 , piece_id=PPK6XEQYJ3XV27XUJYKUL2MYAR5KUYKF5AHJS5EBXOK7PZWWM66Q, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.075964213 (6h17m55.448543568s) [noise=true , address=77.90.31.12:4592 , piece_id=HS5YQKPTPUVHFZH3VZ7X2LAO2SVS6JZOTHTLJ6T47G3CLTZTR7UQ, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.076122025 (6h17m55.44870137s) [noise=true , address=104.36.82.12:4150 , piece_id=GQ4GQQZHK3Y4KVVB2IUU4LEEBR4OJ2YSS5Z5DF7WHHDCYYSSGRYA, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.076272903 (6h17m55.448852258s) [noise=true , address=45.91.116.13:4154 , piece_id=OJO7VBXGBLV2SG2AQ6BPBP57SL75WV4UZSWC3O5HXD7ROBICSYGA, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.076533412 (6h17m55.449112767s) [noise=true , address=104.36.82.2:4732 , piece_id=DBPJH5FSNFRPIBLP7HWWJFVVXYC47J7UGPCXFL7L3BD6BGFSJDFA, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.076794962 (6h17m55.449374317s) [noise=true , address=45.91.116.9:4263 , piece_id=4L3UWFKV3AC7F4RPS7GJFXOO6DJNCPMFB5NWGJH6HXAFBGLYAMYA, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.077044971 (6h17m55.449624326s) [noise=true , address=104.36.82.16:4293 , piece_id=2NA7I27WVAWAUWDCZVYBJLNXGWIDNJMKZU7YFJZSXCQMCHKLCGXA, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.077296953 (6h17m55.449876308s) [noise=true , address=77.90.31.15:4698 , piece_id=HNTVTQMVXDUMK4CMHKVUZ7DHM3MZL56TGXFR4FFFV2V3DCTLAVZQ, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.077501123 (6h17m55.450080468s) [noise=true , address=104.36.82.9:4007 , piece_id=R2IPGORJ3Z4FFCC52P5WMXYIA6O5V7DDZHHL4X64XV6J36SIWSBQ, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.077695656 (6h17m55.450275001s) [noise=true , address=45.91.116.8:4048 , piece_id=QKXGR6YZSYTQIUXV54DRYQNWY352UFS2Z2VXWFJTD7YJYJZ24NCQ, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.077994558 (6h17m55.450573913s) [noise=true , address=104.36.82.14:4222 , piece_id=4F6ZDS2KMQNALT2YV6D3TAZLIBFUFBM5NNCBGLSE6L6PR5QMGPVA, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.078213187 (6h17m55.450792542s) [noise=true , address=45.91.116.4:4939 , piece_id=J2BIIQITH7R54K4ENE6AKU7KCOUAGYBOTKX6VF6FY7FNQDMNNFCA, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.078412077 (6h17m55.450991432s) [noise=true , address=77.90.31.13:4626 , piece_id=2QAZAYJWKJXTPY475RLOSPV3ZZTMB6OZLNYAKD7FCFV7T4WPC5ZQ, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.079649705 (6h17m55.45222906s) [noise=true , address=45.91.116.9:4262 , piece_id=EXWTIYPESLJQK3CDK732WC63OZ7OJOQAUCBPGO3ZU54UBFUJ3TSA, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.080903344 (6h17m55.453482699s) [noise=true , address=77.90.31.13:4628 , piece_id=OAGGFQO2CORAZQV6ND3EERSICEOSHKK73Y3PUWLT45ZUWEWDHOFQ, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:50.857952826 (6h17m57.230532181s) [upload=5005]: Next returned nil segment Jul 19 05:08:50.857969477 (6h17m57.230548832s) [upload=5005]: Waiting for error group managing segments... Jul 19 05:08:51.139405061 (6h17m57.511984416s) [noise=true , address=104.36.82.13:4186 , piece_id=PPK6XEQYJ3XV27XUJYKUL2MYAR5KUYKF5AHJS5EBXOK7PZWWM66Q, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:51.143987831 (6h17m57.516567176s) [noise=true , address=45.91.116.8:4048 , piece_id=QKXGR6YZSYTQIUXV54DRYQNWY352UFS2Z2VXWFJTD7YJYJZ24NCQ, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:51.148258443 (6h17m57.520837798s) [noise=true , address=104.36.82.16:4293 , piece_id=2NA7I27WVAWAUWDCZVYBJLNXGWIDNJMKZU7YFJZSXCQMCHKLCGXA, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:51.156796512 (6h17m57.529375867s) [noise=true , address=45.91.116.13:4154 , piece_id=OJO7VBXGBLV2SG2AQ6BPBP57SL75WV4UZSWC3O5HXD7ROBICSYGA, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:51.193908742 (6h17m57.566488098s) [noise=true , address=104.36.82.14:4222 , piece_id=4F6ZDS2KMQNALT2YV6D3TAZLIBFUFBM5NNCBGLSE6L6PR5QMGPVA, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:51.217498736 (6h17m57.590078091s) [noise=true , address=45.91.116.4:4939 , piece_id=J2BIIQITH7R54K4ENE6AKU7KCOUAGYBOTKX6VF6FY7FNQDMNNFCA, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:51.218585386 (6h17m57.591164741s) [noise=true , address=45.91.116.9:4263 , piece_id=4L3UWFKV3AC7F4RPS7GJFXOO6DJNCPMFB5NWGJH6HXAFBGLYAMYA, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:51.290646027 (6h17m57.663225392s) [noise=true , address=104.36.82.2:4732 , piece_id=DBPJH5FSNFRPIBLP7HWWJFVVXYC47J7UGPCXFL7L3BD6BGFSJDFA, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:51.309329961 (6h17m57.681909316s) [noise=true , address=104.36.82.12:4150 , piece_id=GQ4GQQZHK3Y4KVVB2IUU4LEEBR4OJ2YSS5Z5DF7WHHDCYYSSGRYA, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:51.317895332 (6h17m57.690474687s) [noise=true , address=104.36.82.9:4007 , piece_id=R2IPGORJ3Z4FFCC52P5WMXYIA6O5V7DDZHHL4X64XV6J36SIWSBQ, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:52.525479047 (6h17m58.898058402s) [noise=true , address=45.91.116.9:4262 , piece_id=EXWTIYPESLJQK3CDK732WC63OZ7OJOQAUCBPGO3ZU54UBFUJ3TSA, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:09:18.157484198 (6h18m24.530063553s) [noise=true , address=77.90.31.13:4628 , piece_id=OAGGFQO2CORAZQV6ND3EERSICEOSHKK73Y3PUWLT45ZUWEWDHOFQ, seg_pos={0 0}, upload=5005]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: expected piece hash; context deadline exceeded; EOF Jul 19 05:09:18.157617684 (6h18m24.530197039s) [noise=true , address=77.90.31.15:4698 , piece_id=HNTVTQMVXDUMK4CMHKVUZ7DHM3MZL56TGXFR4FFFV2V3DCTLAVZQ, seg_pos={0 0}, upload=5005]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF Jul 19 05:09:18.157744796 (6h18m24.530324151s) [noise=true , address=77.90.31.11:4555 , piece_id=QGRXVFSKXDCLKSSSXEK4QYZEETWQGFZMQQBSTXNMAHY45RA6MAVQ, seg_pos={0 0}, upload=5005]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: write tcp 188.166.225.170:38288->77.90.31.11:4555: use of closed network connection; write tcp 188.166.225.170:38288->77.90.31.11:4555: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38288->77.90.31.11:4555: use of closed network connection Jul 19 05:09:18.157785745 (6h18m24.5303651s) [noise=true , address=77.90.31.13:4626 , piece_id=2QAZAYJWKJXTPY475RLOSPV3ZZTMB6OZLNYAKD7FCFV7T4WPC5ZQ, seg_pos={0 0}, upload=5005]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:58846->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:58846->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58846->77.90.31.13:4626: use of closed network connection Jul 19 05:09:18.157859036 (6h18m24.530438391s) [noise=true , address=77.90.31.12:4592 , piece_id=HS5YQKPTPUVHFZH3VZ7X2LAO2SVS6JZOTHTLJ6T47G3CLTZTR7UQ, seg_pos={0 0}, upload=5005]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:38220->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:38220->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38220->77.90.31.12:4592: use of closed network connection Jul 19 05:09:18.157898821 (6h18m24.530478176s) [seg_pos={0 0}, upload=5005]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 19 05:09:18.389974737 (6h18m24.762554092s) [upload=5005]: Done uploading segments... Jul 19 05:09:18.389994916 (6h18m24.762574271s) [upload=5005]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 7.87927533s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.........................XX...X...X....X..X....XXXXX.X.X....] duration: 5.780683075s FAIL (2025-07-19 05:11:42 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== : <nil> Jul 19 05:08:29.800544823 (6h17m36.173124169s) [upload=4997]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 19 05:08:29.800548410 (6h17m36.173127765s) [upload=4997]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 19 05:08:30.148027600 (6h17m36.520606955s) [upload=4997]: Done uploading segments... Jul 19 05:08:30.148148401 (6h17m36.520727746s) [upload=4997]: Upload finished. err: <nil> Jul 19 05:08:48.807983386 (6h17m55.180562741s) [upload=5005]: Starting upload Jul 19 05:08:48.808044453 (6h17m55.180623808s) [upload=5005]: Done starting upload Jul 19 05:08:48.808175243 (6h17m55.180754598s) [upload=5005]: Uploading segments... Jul 19 05:08:48.808334198 (6h17m55.180913553s) [upload=5005]: Got next segment. Inline: false Jul 19 05:08:48.808392509 (6h17m55.180971874s) [upload=5005]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 05:08:48.808399823 (6h17m55.180979188s) [upload=5005]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 05:08:49.075515233 (6h17m55.448094578s) [seg_pos={0 0}, upload=5005]: Begin upload segment... Jul 19 05:08:49.075564547 (6h17m55.448143902s) [seg_pos={0 0}, upload=5005]: Done begin upload segment. Jul 19 05:08:49.075602881 (6h17m55.448182236s) [noise=true , address=77.90.31.11:4555 , piece_id=QGRXVFSKXDCLKSSSXEK4QYZEETWQGFZMQQBSTXNMAHY45RA6MAVQ, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.075612279 (6h17m55.448191634s) [noise=true , address=104.36.82.13:4186 , piece_id=PPK6XEQYJ3XV27XUJYKUL2MYAR5KUYKF5AHJS5EBXOK7PZWWM66Q, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.075964213 (6h17m55.448543568s) [noise=true , address=77.90.31.12:4592 , piece_id=HS5YQKPTPUVHFZH3VZ7X2LAO2SVS6JZOTHTLJ6T47G3CLTZTR7UQ, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.076122025 (6h17m55.44870137s) [noise=true , address=104.36.82.12:4150 , piece_id=GQ4GQQZHK3Y4KVVB2IUU4LEEBR4OJ2YSS5Z5DF7WHHDCYYSSGRYA, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.076272903 (6h17m55.448852258s) [noise=true , address=45.91.116.13:4154 , piece_id=OJO7VBXGBLV2SG2AQ6BPBP57SL75WV4UZSWC3O5HXD7ROBICSYGA, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.076533412 (6h17m55.449112767s) [noise=true , address=104.36.82.2:4732 , piece_id=DBPJH5FSNFRPIBLP7HWWJFVVXYC47J7UGPCXFL7L3BD6BGFSJDFA, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.076794962 (6h17m55.449374317s) [noise=true , address=45.91.116.9:4263 , piece_id=4L3UWFKV3AC7F4RPS7GJFXOO6DJNCPMFB5NWGJH6HXAFBGLYAMYA, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.077044971 (6h17m55.449624326s) [noise=true , address=104.36.82.16:4293 , piece_id=2NA7I27WVAWAUWDCZVYBJLNXGWIDNJMKZU7YFJZSXCQMCHKLCGXA, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.077296953 (6h17m55.449876308s) [noise=true , address=77.90.31.15:4698 , piece_id=HNTVTQMVXDUMK4CMHKVUZ7DHM3MZL56TGXFR4FFFV2V3DCTLAVZQ, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.077501123 (6h17m55.450080468s) [noise=true , address=104.36.82.9:4007 , piece_id=R2IPGORJ3Z4FFCC52P5WMXYIA6O5V7DDZHHL4X64XV6J36SIWSBQ, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.077695656 (6h17m55.450275001s) [noise=true , address=45.91.116.8:4048 , piece_id=QKXGR6YZSYTQIUXV54DRYQNWY352UFS2Z2VXWFJTD7YJYJZ24NCQ, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.077994558 (6h17m55.450573913s) [noise=true , address=104.36.82.14:4222 , piece_id=4F6ZDS2KMQNALT2YV6D3TAZLIBFUFBM5NNCBGLSE6L6PR5QMGPVA, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.078213187 (6h17m55.450792542s) [noise=true , address=45.91.116.4:4939 , piece_id=J2BIIQITH7R54K4ENE6AKU7KCOUAGYBOTKX6VF6FY7FNQDMNNFCA, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.078412077 (6h17m55.450991432s) [noise=true , address=77.90.31.13:4626 , piece_id=2QAZAYJWKJXTPY475RLOSPV3ZZTMB6OZLNYAKD7FCFV7T4WPC5ZQ, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.079649705 (6h17m55.45222906s) [noise=true , address=45.91.116.9:4262 , piece_id=EXWTIYPESLJQK3CDK732WC63OZ7OJOQAUCBPGO3ZU54UBFUJ3TSA, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.080903344 (6h17m55.453482699s) [noise=true , address=77.90.31.13:4628 , piece_id=OAGGFQO2CORAZQV6ND3EERSICEOSHKK73Y3PUWLT45ZUWEWDHOFQ, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:50.857952826 (6h17m57.230532181s) [upload=5005]: Next returned nil segment Jul 19 05:08:50.857969477 (6h17m57.230548832s) [upload=5005]: Waiting for error group managing segments... Jul 19 05:08:51.139405061 (6h17m57.511984416s) [noise=true , address=104.36.82.13:4186 , piece_id=PPK6XEQYJ3XV27XUJYKUL2MYAR5KUYKF5AHJS5EBXOK7PZWWM66Q, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:51.143987831 (6h17m57.516567176s) [noise=true , address=45.91.116.8:4048 , piece_id=QKXGR6YZSYTQIUXV54DRYQNWY352UFS2Z2VXWFJTD7YJYJZ24NCQ, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:51.148258443 (6h17m57.520837798s) [noise=true , address=104.36.82.16:4293 , piece_id=2NA7I27WVAWAUWDCZVYBJLNXGWIDNJMKZU7YFJZSXCQMCHKLCGXA, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:51.156796512 (6h17m57.529375867s) [noise=true , address=45.91.116.13:4154 , piece_id=OJO7VBXGBLV2SG2AQ6BPBP57SL75WV4UZSWC3O5HXD7ROBICSYGA, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:51.193908742 (6h17m57.566488098s) [noise=true , address=104.36.82.14:4222 , piece_id=4F6ZDS2KMQNALT2YV6D3TAZLIBFUFBM5NNCBGLSE6L6PR5QMGPVA, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:51.217498736 (6h17m57.590078091s) [noise=true , address=45.91.116.4:4939 , piece_id=J2BIIQITH7R54K4ENE6AKU7KCOUAGYBOTKX6VF6FY7FNQDMNNFCA, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:51.218585386 (6h17m57.591164741s) [noise=true , address=45.91.116.9:4263 , piece_id=4L3UWFKV3AC7F4RPS7GJFXOO6DJNCPMFB5NWGJH6HXAFBGLYAMYA, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:51.290646027 (6h17m57.663225392s) [noise=true , address=104.36.82.2:4732 , piece_id=DBPJH5FSNFRPIBLP7HWWJFVVXYC47J7UGPCXFL7L3BD6BGFSJDFA, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:51.309329961 (6h17m57.681909316s) [noise=true , address=104.36.82.12:4150 , piece_id=GQ4GQQZHK3Y4KVVB2IUU4LEEBR4OJ2YSS5Z5DF7WHHDCYYSSGRYA, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:51.317895332 (6h17m57.690474687s) [noise=true , address=104.36.82.9:4007 , piece_id=R2IPGORJ3Z4FFCC52P5WMXYIA6O5V7DDZHHL4X64XV6J36SIWSBQ, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:52.525479047 (6h17m58.898058402s) [noise=true , address=45.91.116.9:4262 , piece_id=EXWTIYPESLJQK3CDK732WC63OZ7OJOQAUCBPGO3ZU54UBFUJ3TSA, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:09:18.157484198 (6h18m24.530063553s) [noise=true , address=77.90.31.13:4628 , piece_id=OAGGFQO2CORAZQV6ND3EERSICEOSHKK73Y3PUWLT45ZUWEWDHOFQ, seg_pos={0 0}, upload=5005]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: expected piece hash; context deadline exceeded; EOF Jul 19 05:09:18.157617684 (6h18m24.530197039s) [noise=true , address=77.90.31.15:4698 , piece_id=HNTVTQMVXDUMK4CMHKVUZ7DHM3MZL56TGXFR4FFFV2V3DCTLAVZQ, seg_pos={0 0}, upload=5005]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF Jul 19 05:09:18.157744796 (6h18m24.530324151s) [noise=true , address=77.90.31.11:4555 , piece_id=QGRXVFSKXDCLKSSSXEK4QYZEETWQGFZMQQBSTXNMAHY45RA6MAVQ, seg_pos={0 0}, upload=5005]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: write tcp 188.166.225.170:38288->77.90.31.11:4555: use of closed network connection; write tcp 188.166.225.170:38288->77.90.31.11:4555: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38288->77.90.31.11:4555: use of closed network connection Jul 19 05:09:18.157785745 (6h18m24.5303651s) [noise=true , address=77.90.31.13:4626 , piece_id=2QAZAYJWKJXTPY475RLOSPV3ZZTMB6OZLNYAKD7FCFV7T4WPC5ZQ, seg_pos={0 0}, upload=5005]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:58846->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:58846->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58846->77.90.31.13:4626: use of closed network connection Jul 19 05:09:18.157859036 (6h18m24.530438391s) [noise=true , address=77.90.31.12:4592 , piece_id=HS5YQKPTPUVHFZH3VZ7X2LAO2SVS6JZOTHTLJ6T47G3CLTZTR7UQ, seg_pos={0 0}, upload=5005]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:38220->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:38220->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38220->77.90.31.12:4592: use of closed network connection Jul 19 05:09:18.157898821 (6h18m24.530478176s) [seg_pos={0 0}, upload=5005]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 19 05:09:18.389974737 (6h18m24.762554092s) [upload=5005]: Done uploading segments... Jul 19 05:09:18.389994916 (6h18m24.762574271s) [upload=5005]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.780713294s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 5.508858139s FAIL (2025-07-19 05:11:28 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 4855s) [upload=4769]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 04:51:09.109709627 (6h0m15.482288982s) [upload=4769]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 04:51:09.556145753 (6h0m15.928725108s) [seg_pos={0 0}, upload=4769]: Begin upload segment... Jul 19 04:51:09.556194817 (6h0m15.928774172s) [seg_pos={0 0}, upload=4769]: Done begin upload segment. Jul 19 04:51:09.556223893 (6h0m15.928803248s) [noise=true , address=79.127.219.38:4023 , piece_id=PTPOZGLYV4MZTOFZNN3NCGYL7DFAST4PXKEUIVTMFL7HV4GZTCLQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556252197 (6h0m15.928831552s) [noise=true , address=45.91.95.4:4011 , piece_id=ZMBAS6KKHYVIRE6OBL2NFVCWMM4TKQ7J64BD3SYXIK54XCEDI2XA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556579383 (6h0m15.929158738s) [noise=true , address=45.91.116.6:4007 , piece_id=A2AFVTZBXSGBOE7XIM3B5HRN4STITFBXO6MZRJ365URNAAOZ3PSQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556854469 (6h0m15.929433824s) [noise=true , address=104.36.82.19:4423 , piece_id=PIJUHDUND36HQL67I33D7A3EULMJHS7N55MN4OFQJ4OWNQ42O47A, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557094107 (6h0m15.929673462s) [noise=true , address=45.91.116.9:4264 , piece_id=GHM5BKZKMT2QOHHBOS4FSPOWEWSFQRGRVVR4GSNDPHVIFRZHQL2A, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557336381 (6h0m15.929915736s) [noise=true , address=79.127.205.235:4660 , piece_id=UOUOGE2VP7PX7EMRZLPMCWFCYEZERM3DQMCQ6NG4J2K2DEIZR63Q, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557557885 (6h0m15.93013724s) [noise=true , address=79.127.205.246:4280 , piece_id=FLBOZI3UGTQXAQBLZ2F45Z3OEYL65F5PQBXRF2DQMZ7NGF7VDRPA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557762296 (6h0m15.930341651s) [noise=true , address=79.127.205.233:4622 , piece_id=6WWNYQTH5MGAJOEB7BXKZDJOMCAZXDW66J3OEANGDNJDNUMLZM7Q, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558014689 (6h0m15.930594044s) [noise=true , address=64.49.13.138:4482 , piece_id=MAW2M4US2OQNOS3WI67AA7NIBTJDE5T2JQTXKFPBLGVGXKVCFGTQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558215974 (6h0m15.930795319s) [noise=true , address=77.90.31.10:4546 , piece_id=R6LCA5WZOEBDURL5S3MEK22YPVYXX6EXSS4J4W57543HELYXOKZA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558409364 (6h0m15.930988719s) [noise=true , address=45.91.95.5:4019 , piece_id=YTSBCN2LPHEMRRAZIHJGVB7NBOTRXVEHD2TWUCTTLBNFFYEWWDKA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558623504 (6h0m15.931202859s) [noise=true , address=79.127.201.215:4357 , piece_id=75XWAG2VQWEYCWN2Q3WRIZSBI7LDOISZJPXQLQ2GIPNEN34VLSUQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558805162 (6h0m15.931384517s) [noise=true , address=45.91.116.14:4219 , piece_id=PDW6JPRHCVMHHOWIHT6JKV4CTSC3JY6NDKDVFZSO47OYNPZ7PGGA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558999373 (6h0m15.931578728s) [noise=true , address=77.90.31.11:4564 , piece_id=L2U7JFJRDQOFA2AYVAGTI4ZW4VABCXKMXWBIUWQO2QNNUUBJ74BA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559184768 (6h0m15.931764123s) [noise=true , address=77.90.31.16:4768 , piece_id=YNLSONFM6COZEVF6ZNQR2EA6V4LSQFZJONIXR4AFTNNMFGPV57JA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559368719 (6h0m15.931948074s) [noise=true , address=79.127.220.98:4432 , piece_id=2SUTFQ7FVDPAM2H4W3OLW6AVH3VJ33IJFE4L52SA3ET5LGBMOMPQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559549706 (6h0m15.932129061s) [noise=true , address=77.90.31.18:4813 , piece_id=VDOT7BB4O7XYJHW6BFTAYQZZYLZU5PAUXVJGSXJ7FDI4TH425COA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559727836 (6h0m15.932307191s) [noise=true , address=79.127.219.51:4211 , piece_id=HO5PLWBVU5YAYNYMN352NA73SM4WBJV4WZ7XLSQTCQ76UAL2TOJA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:10.663359087 (6h0m17.035938442s) [upload=4769]: Next returned nil segment Jul 19 04:51:10.663378664 (6h0m17.035958019s) [upload=4769]: Waiting for error group managing segments... Jul 19 04:51:10.962945479 (6h0m17.335524834s) [noise=true , address=79.127.205.233:4622 , piece_id=6WWNYQTH5MGAJOEB7BXKZDJOMCAZXDW66J3OEANGDNJDNUMLZM7Q, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:10.967689807 (6h0m17.340269152s) [noise=true , address=79.127.205.235:4660 , piece_id=UOUOGE2VP7PX7EMRZLPMCWFCYEZERM3DQMCQ6NG4J2K2DEIZR63Q, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:10.976968631 (6h0m17.349547986s) [noise=true , address=64.49.13.138:4482 , piece_id=MAW2M4US2OQNOS3WI67AA7NIBTJDE5T2JQTXKFPBLGVGXKVCFGTQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.048750700 (6h0m17.421330055s) [noise=true , address=79.127.205.246:4280 , piece_id=FLBOZI3UGTQXAQBLZ2F45Z3OEYL65F5PQBXRF2DQMZ7NGF7VDRPA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.221642403 (6h0m17.594221758s) [noise=true , address=79.127.220.98:4432 , piece_id=2SUTFQ7FVDPAM2H4W3OLW6AVH3VJ33IJFE4L52SA3ET5LGBMOMPQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.317070435 (6h0m17.68964979s) [noise=true , address=45.91.95.4:4011 , piece_id=ZMBAS6KKHYVIRE6OBL2NFVCWMM4TKQ7J64BD3SYXIK54XCEDI2XA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.324002660 (6h0m17.696582015s) [noise=true , address=45.91.95.5:4019 , piece_id=YTSBCN2LPHEMRRAZIHJGVB7NBOTRXVEHD2TWUCTTLBNFFYEWWDKA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.460976317 (6h0m17.833555672s) [noise=true , address=79.127.201.215:4357 , piece_id=75XWAG2VQWEYCWN2Q3WRIZSBI7LDOISZJPXQLQ2GIPNEN34VLSUQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.462584603 (6h0m17.835163958s) [noise=true , address=45.91.116.6:4007 , piece_id=A2AFVTZBXSGBOE7XIM3B5HRN4STITFBXO6MZRJ365URNAAOZ3PSQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.490392973 (6h0m17.862972328s) [noise=true , address=45.91.116.9:4264 , piece_id=GHM5BKZKMT2QOHHBOS4FSPOWEWSFQRGRVVR4GSNDPHVIFRZHQL2A, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.502040780 (6h0m17.874620135s) [noise=true , address=45.91.116.14:4219 , piece_id=PDW6JPRHCVMHHOWIHT6JKV4CTSC3JY6NDKDVFZSO47OYNPZ7PGGA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.600434096 (6h0m17.973013451s) [noise=true , address=79.127.219.51:4211 , piece_id=HO5PLWBVU5YAYNYMN352NA73SM4WBJV4WZ7XLSQTCQ76UAL2TOJA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.609008984 (6h0m17.981588339s) [noise=true , address=104.36.82.19:4423 , piece_id=PIJUHDUND36HQL67I33D7A3EULMJHS7N55MN4OFQJ4OWNQ42O47A, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.620825273 (6h0m17.993404628s) [noise=true , address=79.127.219.38:4023 , piece_id=PTPOZGLYV4MZTOFZNN3NCGYL7DFAST4PXKEUIVTMFL7HV4GZTCLQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.620838659 (6h0m17.993418014s) [seg_pos={0 0}, upload=4769]: Segment reached optimal threshold of 14 pieces. Jul 19 04:51:11.621067807 (6h0m17.993647163s) [noise=true , address=77.90.31.11:4564 , piece_id=L2U7JFJRDQOFA2AYVAGTI4ZW4VABCXKMXWBIUWQO2QNNUUBJ74BA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection; write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection Jul 19 04:51:11.621134234 (6h0m17.993713589s) [noise=true , address=77.90.31.10:4546 , piece_id=R6LCA5WZOEBDURL5S3MEK22YPVYXX6EXSS4J4W57543HELYXOKZA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK): protocol: read tcp 188.166.225.170:58476->77.90.31.10:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621174191 (6h0m17.993753546s) [noise=true , address=77.90.31.16:4768 , piece_id=YNLSONFM6COZEVF6ZNQR2EA6V4LSQFZJONIXR4AFTNNMFGPV57JA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UchewmtxBrtSZSpVAJFdZmcTsQVQxkjK9re6cgUGqDhyonQeF): protocol: read tcp 188.166.225.170:56326->77.90.31.16:4768: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621216842 (6h0m17.993796197s) [noise=true , address=77.90.31.18:4813 , piece_id=VDOT7BB4O7XYJHW6BFTAYQZZYLZU5PAUXVJGSXJ7FDI4TH425COA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128WthA1x9wp1dV1vkpGnoveZQpU9CSgDCZfw5GMQuVpCNSVAM5): protocol: read tcp 188.166.225.170:49704->77.90.31.18:4813: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621234917 (6h0m17.993814272s) [seg_pos={0 0}, upload=4769]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 19 04:51:11.621301444 (6h0m17.993880799s) [upload=4769]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 19 04:51:11.621305402 (6h0m17.993884758s) [upload=4769]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 19 04:51:11.923136970 (6h0m18.295716325s) [upload=4769]: Done uploading segments... Jul 19 04:51:11.923156627 (6h0m18.295735972s) [upload=4769]: Upload finished. err: <nil> Duration: 5.508886104s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [..........................XX...X...X....X..X....XXXXX.X.X...] duration: 5.800191955s FAIL (2025-07-19 05:11:12 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== : <nil> Jul 19 05:08:29.800544823 (6h17m36.173124169s) [upload=4997]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 19 05:08:29.800548410 (6h17m36.173127765s) [upload=4997]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 19 05:08:30.148027600 (6h17m36.520606955s) [upload=4997]: Done uploading segments... Jul 19 05:08:30.148148401 (6h17m36.520727746s) [upload=4997]: Upload finished. err: <nil> Jul 19 05:08:48.807983386 (6h17m55.180562741s) [upload=5005]: Starting upload Jul 19 05:08:48.808044453 (6h17m55.180623808s) [upload=5005]: Done starting upload Jul 19 05:08:48.808175243 (6h17m55.180754598s) [upload=5005]: Uploading segments... Jul 19 05:08:48.808334198 (6h17m55.180913553s) [upload=5005]: Got next segment. Inline: false Jul 19 05:08:48.808392509 (6h17m55.180971874s) [upload=5005]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 05:08:48.808399823 (6h17m55.180979188s) [upload=5005]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 05:08:49.075515233 (6h17m55.448094578s) [seg_pos={0 0}, upload=5005]: Begin upload segment... Jul 19 05:08:49.075564547 (6h17m55.448143902s) [seg_pos={0 0}, upload=5005]: Done begin upload segment. Jul 19 05:08:49.075602881 (6h17m55.448182236s) [noise=true , address=77.90.31.11:4555 , piece_id=QGRXVFSKXDCLKSSSXEK4QYZEETWQGFZMQQBSTXNMAHY45RA6MAVQ, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.075612279 (6h17m55.448191634s) [noise=true , address=104.36.82.13:4186 , piece_id=PPK6XEQYJ3XV27XUJYKUL2MYAR5KUYKF5AHJS5EBXOK7PZWWM66Q, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.075964213 (6h17m55.448543568s) [noise=true , address=77.90.31.12:4592 , piece_id=HS5YQKPTPUVHFZH3VZ7X2LAO2SVS6JZOTHTLJ6T47G3CLTZTR7UQ, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.076122025 (6h17m55.44870137s) [noise=true , address=104.36.82.12:4150 , piece_id=GQ4GQQZHK3Y4KVVB2IUU4LEEBR4OJ2YSS5Z5DF7WHHDCYYSSGRYA, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.076272903 (6h17m55.448852258s) [noise=true , address=45.91.116.13:4154 , piece_id=OJO7VBXGBLV2SG2AQ6BPBP57SL75WV4UZSWC3O5HXD7ROBICSYGA, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.076533412 (6h17m55.449112767s) [noise=true , address=104.36.82.2:4732 , piece_id=DBPJH5FSNFRPIBLP7HWWJFVVXYC47J7UGPCXFL7L3BD6BGFSJDFA, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.076794962 (6h17m55.449374317s) [noise=true , address=45.91.116.9:4263 , piece_id=4L3UWFKV3AC7F4RPS7GJFXOO6DJNCPMFB5NWGJH6HXAFBGLYAMYA, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.077044971 (6h17m55.449624326s) [noise=true , address=104.36.82.16:4293 , piece_id=2NA7I27WVAWAUWDCZVYBJLNXGWIDNJMKZU7YFJZSXCQMCHKLCGXA, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.077296953 (6h17m55.449876308s) [noise=true , address=77.90.31.15:4698 , piece_id=HNTVTQMVXDUMK4CMHKVUZ7DHM3MZL56TGXFR4FFFV2V3DCTLAVZQ, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.077501123 (6h17m55.450080468s) [noise=true , address=104.36.82.9:4007 , piece_id=R2IPGORJ3Z4FFCC52P5WMXYIA6O5V7DDZHHL4X64XV6J36SIWSBQ, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.077695656 (6h17m55.450275001s) [noise=true , address=45.91.116.8:4048 , piece_id=QKXGR6YZSYTQIUXV54DRYQNWY352UFS2Z2VXWFJTD7YJYJZ24NCQ, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.077994558 (6h17m55.450573913s) [noise=true , address=104.36.82.14:4222 , piece_id=4F6ZDS2KMQNALT2YV6D3TAZLIBFUFBM5NNCBGLSE6L6PR5QMGPVA, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.078213187 (6h17m55.450792542s) [noise=true , address=45.91.116.4:4939 , piece_id=J2BIIQITH7R54K4ENE6AKU7KCOUAGYBOTKX6VF6FY7FNQDMNNFCA, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.078412077 (6h17m55.450991432s) [noise=true , address=77.90.31.13:4626 , piece_id=2QAZAYJWKJXTPY475RLOSPV3ZZTMB6OZLNYAKD7FCFV7T4WPC5ZQ, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.079649705 (6h17m55.45222906s) [noise=true , address=45.91.116.9:4262 , piece_id=EXWTIYPESLJQK3CDK732WC63OZ7OJOQAUCBPGO3ZU54UBFUJ3TSA, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.080903344 (6h17m55.453482699s) [noise=true , address=77.90.31.13:4628 , piece_id=OAGGFQO2CORAZQV6ND3EERSICEOSHKK73Y3PUWLT45ZUWEWDHOFQ, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:50.857952826 (6h17m57.230532181s) [upload=5005]: Next returned nil segment Jul 19 05:08:50.857969477 (6h17m57.230548832s) [upload=5005]: Waiting for error group managing segments... Jul 19 05:08:51.139405061 (6h17m57.511984416s) [noise=true , address=104.36.82.13:4186 , piece_id=PPK6XEQYJ3XV27XUJYKUL2MYAR5KUYKF5AHJS5EBXOK7PZWWM66Q, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:51.143987831 (6h17m57.516567176s) [noise=true , address=45.91.116.8:4048 , piece_id=QKXGR6YZSYTQIUXV54DRYQNWY352UFS2Z2VXWFJTD7YJYJZ24NCQ, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:51.148258443 (6h17m57.520837798s) [noise=true , address=104.36.82.16:4293 , piece_id=2NA7I27WVAWAUWDCZVYBJLNXGWIDNJMKZU7YFJZSXCQMCHKLCGXA, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:51.156796512 (6h17m57.529375867s) [noise=true , address=45.91.116.13:4154 , piece_id=OJO7VBXGBLV2SG2AQ6BPBP57SL75WV4UZSWC3O5HXD7ROBICSYGA, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:51.193908742 (6h17m57.566488098s) [noise=true , address=104.36.82.14:4222 , piece_id=4F6ZDS2KMQNALT2YV6D3TAZLIBFUFBM5NNCBGLSE6L6PR5QMGPVA, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:51.217498736 (6h17m57.590078091s) [noise=true , address=45.91.116.4:4939 , piece_id=J2BIIQITH7R54K4ENE6AKU7KCOUAGYBOTKX6VF6FY7FNQDMNNFCA, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:51.218585386 (6h17m57.591164741s) [noise=true , address=45.91.116.9:4263 , piece_id=4L3UWFKV3AC7F4RPS7GJFXOO6DJNCPMFB5NWGJH6HXAFBGLYAMYA, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:51.290646027 (6h17m57.663225392s) [noise=true , address=104.36.82.2:4732 , piece_id=DBPJH5FSNFRPIBLP7HWWJFVVXYC47J7UGPCXFL7L3BD6BGFSJDFA, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:51.309329961 (6h17m57.681909316s) [noise=true , address=104.36.82.12:4150 , piece_id=GQ4GQQZHK3Y4KVVB2IUU4LEEBR4OJ2YSS5Z5DF7WHHDCYYSSGRYA, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:51.317895332 (6h17m57.690474687s) [noise=true , address=104.36.82.9:4007 , piece_id=R2IPGORJ3Z4FFCC52P5WMXYIA6O5V7DDZHHL4X64XV6J36SIWSBQ, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:52.525479047 (6h17m58.898058402s) [noise=true , address=45.91.116.9:4262 , piece_id=EXWTIYPESLJQK3CDK732WC63OZ7OJOQAUCBPGO3ZU54UBFUJ3TSA, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:09:18.157484198 (6h18m24.530063553s) [noise=true , address=77.90.31.13:4628 , piece_id=OAGGFQO2CORAZQV6ND3EERSICEOSHKK73Y3PUWLT45ZUWEWDHOFQ, seg_pos={0 0}, upload=5005]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: expected piece hash; context deadline exceeded; EOF Jul 19 05:09:18.157617684 (6h18m24.530197039s) [noise=true , address=77.90.31.15:4698 , piece_id=HNTVTQMVXDUMK4CMHKVUZ7DHM3MZL56TGXFR4FFFV2V3DCTLAVZQ, seg_pos={0 0}, upload=5005]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF Jul 19 05:09:18.157744796 (6h18m24.530324151s) [noise=true , address=77.90.31.11:4555 , piece_id=QGRXVFSKXDCLKSSSXEK4QYZEETWQGFZMQQBSTXNMAHY45RA6MAVQ, seg_pos={0 0}, upload=5005]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: write tcp 188.166.225.170:38288->77.90.31.11:4555: use of closed network connection; write tcp 188.166.225.170:38288->77.90.31.11:4555: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38288->77.90.31.11:4555: use of closed network connection Jul 19 05:09:18.157785745 (6h18m24.5303651s) [noise=true , address=77.90.31.13:4626 , piece_id=2QAZAYJWKJXTPY475RLOSPV3ZZTMB6OZLNYAKD7FCFV7T4WPC5ZQ, seg_pos={0 0}, upload=5005]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:58846->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:58846->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58846->77.90.31.13:4626: use of closed network connection Jul 19 05:09:18.157859036 (6h18m24.530438391s) [noise=true , address=77.90.31.12:4592 , piece_id=HS5YQKPTPUVHFZH3VZ7X2LAO2SVS6JZOTHTLJ6T47G3CLTZTR7UQ, seg_pos={0 0}, upload=5005]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:38220->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:38220->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38220->77.90.31.12:4592: use of closed network connection Jul 19 05:09:18.157898821 (6h18m24.530478176s) [seg_pos={0 0}, upload=5005]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 19 05:09:18.389974737 (6h18m24.762554092s) [upload=5005]: Done uploading segments... Jul 19 05:09:18.389994916 (6h18m24.762574271s) [upload=5005]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.800217705s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 7.123818105s FAIL (2025-07-19 05:10:26 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 4855s) [upload=4769]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 04:51:09.109709627 (6h0m15.482288982s) [upload=4769]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 04:51:09.556145753 (6h0m15.928725108s) [seg_pos={0 0}, upload=4769]: Begin upload segment... Jul 19 04:51:09.556194817 (6h0m15.928774172s) [seg_pos={0 0}, upload=4769]: Done begin upload segment. Jul 19 04:51:09.556223893 (6h0m15.928803248s) [noise=true , address=79.127.219.38:4023 , piece_id=PTPOZGLYV4MZTOFZNN3NCGYL7DFAST4PXKEUIVTMFL7HV4GZTCLQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556252197 (6h0m15.928831552s) [noise=true , address=45.91.95.4:4011 , piece_id=ZMBAS6KKHYVIRE6OBL2NFVCWMM4TKQ7J64BD3SYXIK54XCEDI2XA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556579383 (6h0m15.929158738s) [noise=true , address=45.91.116.6:4007 , piece_id=A2AFVTZBXSGBOE7XIM3B5HRN4STITFBXO6MZRJ365URNAAOZ3PSQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556854469 (6h0m15.929433824s) [noise=true , address=104.36.82.19:4423 , piece_id=PIJUHDUND36HQL67I33D7A3EULMJHS7N55MN4OFQJ4OWNQ42O47A, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557094107 (6h0m15.929673462s) [noise=true , address=45.91.116.9:4264 , piece_id=GHM5BKZKMT2QOHHBOS4FSPOWEWSFQRGRVVR4GSNDPHVIFRZHQL2A, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557336381 (6h0m15.929915736s) [noise=true , address=79.127.205.235:4660 , piece_id=UOUOGE2VP7PX7EMRZLPMCWFCYEZERM3DQMCQ6NG4J2K2DEIZR63Q, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557557885 (6h0m15.93013724s) [noise=true , address=79.127.205.246:4280 , piece_id=FLBOZI3UGTQXAQBLZ2F45Z3OEYL65F5PQBXRF2DQMZ7NGF7VDRPA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557762296 (6h0m15.930341651s) [noise=true , address=79.127.205.233:4622 , piece_id=6WWNYQTH5MGAJOEB7BXKZDJOMCAZXDW66J3OEANGDNJDNUMLZM7Q, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558014689 (6h0m15.930594044s) [noise=true , address=64.49.13.138:4482 , piece_id=MAW2M4US2OQNOS3WI67AA7NIBTJDE5T2JQTXKFPBLGVGXKVCFGTQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558215974 (6h0m15.930795319s) [noise=true , address=77.90.31.10:4546 , piece_id=R6LCA5WZOEBDURL5S3MEK22YPVYXX6EXSS4J4W57543HELYXOKZA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558409364 (6h0m15.930988719s) [noise=true , address=45.91.95.5:4019 , piece_id=YTSBCN2LPHEMRRAZIHJGVB7NBOTRXVEHD2TWUCTTLBNFFYEWWDKA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558623504 (6h0m15.931202859s) [noise=true , address=79.127.201.215:4357 , piece_id=75XWAG2VQWEYCWN2Q3WRIZSBI7LDOISZJPXQLQ2GIPNEN34VLSUQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558805162 (6h0m15.931384517s) [noise=true , address=45.91.116.14:4219 , piece_id=PDW6JPRHCVMHHOWIHT6JKV4CTSC3JY6NDKDVFZSO47OYNPZ7PGGA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558999373 (6h0m15.931578728s) [noise=true , address=77.90.31.11:4564 , piece_id=L2U7JFJRDQOFA2AYVAGTI4ZW4VABCXKMXWBIUWQO2QNNUUBJ74BA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559184768 (6h0m15.931764123s) [noise=true , address=77.90.31.16:4768 , piece_id=YNLSONFM6COZEVF6ZNQR2EA6V4LSQFZJONIXR4AFTNNMFGPV57JA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559368719 (6h0m15.931948074s) [noise=true , address=79.127.220.98:4432 , piece_id=2SUTFQ7FVDPAM2H4W3OLW6AVH3VJ33IJFE4L52SA3ET5LGBMOMPQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559549706 (6h0m15.932129061s) [noise=true , address=77.90.31.18:4813 , piece_id=VDOT7BB4O7XYJHW6BFTAYQZZYLZU5PAUXVJGSXJ7FDI4TH425COA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559727836 (6h0m15.932307191s) [noise=true , address=79.127.219.51:4211 , piece_id=HO5PLWBVU5YAYNYMN352NA73SM4WBJV4WZ7XLSQTCQ76UAL2TOJA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:10.663359087 (6h0m17.035938442s) [upload=4769]: Next returned nil segment Jul 19 04:51:10.663378664 (6h0m17.035958019s) [upload=4769]: Waiting for error group managing segments... Jul 19 04:51:10.962945479 (6h0m17.335524834s) [noise=true , address=79.127.205.233:4622 , piece_id=6WWNYQTH5MGAJOEB7BXKZDJOMCAZXDW66J3OEANGDNJDNUMLZM7Q, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:10.967689807 (6h0m17.340269152s) [noise=true , address=79.127.205.235:4660 , piece_id=UOUOGE2VP7PX7EMRZLPMCWFCYEZERM3DQMCQ6NG4J2K2DEIZR63Q, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:10.976968631 (6h0m17.349547986s) [noise=true , address=64.49.13.138:4482 , piece_id=MAW2M4US2OQNOS3WI67AA7NIBTJDE5T2JQTXKFPBLGVGXKVCFGTQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.048750700 (6h0m17.421330055s) [noise=true , address=79.127.205.246:4280 , piece_id=FLBOZI3UGTQXAQBLZ2F45Z3OEYL65F5PQBXRF2DQMZ7NGF7VDRPA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.221642403 (6h0m17.594221758s) [noise=true , address=79.127.220.98:4432 , piece_id=2SUTFQ7FVDPAM2H4W3OLW6AVH3VJ33IJFE4L52SA3ET5LGBMOMPQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.317070435 (6h0m17.68964979s) [noise=true , address=45.91.95.4:4011 , piece_id=ZMBAS6KKHYVIRE6OBL2NFVCWMM4TKQ7J64BD3SYXIK54XCEDI2XA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.324002660 (6h0m17.696582015s) [noise=true , address=45.91.95.5:4019 , piece_id=YTSBCN2LPHEMRRAZIHJGVB7NBOTRXVEHD2TWUCTTLBNFFYEWWDKA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.460976317 (6h0m17.833555672s) [noise=true , address=79.127.201.215:4357 , piece_id=75XWAG2VQWEYCWN2Q3WRIZSBI7LDOISZJPXQLQ2GIPNEN34VLSUQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.462584603 (6h0m17.835163958s) [noise=true , address=45.91.116.6:4007 , piece_id=A2AFVTZBXSGBOE7XIM3B5HRN4STITFBXO6MZRJ365URNAAOZ3PSQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.490392973 (6h0m17.862972328s) [noise=true , address=45.91.116.9:4264 , piece_id=GHM5BKZKMT2QOHHBOS4FSPOWEWSFQRGRVVR4GSNDPHVIFRZHQL2A, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.502040780 (6h0m17.874620135s) [noise=true , address=45.91.116.14:4219 , piece_id=PDW6JPRHCVMHHOWIHT6JKV4CTSC3JY6NDKDVFZSO47OYNPZ7PGGA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.600434096 (6h0m17.973013451s) [noise=true , address=79.127.219.51:4211 , piece_id=HO5PLWBVU5YAYNYMN352NA73SM4WBJV4WZ7XLSQTCQ76UAL2TOJA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.609008984 (6h0m17.981588339s) [noise=true , address=104.36.82.19:4423 , piece_id=PIJUHDUND36HQL67I33D7A3EULMJHS7N55MN4OFQJ4OWNQ42O47A, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.620825273 (6h0m17.993404628s) [noise=true , address=79.127.219.38:4023 , piece_id=PTPOZGLYV4MZTOFZNN3NCGYL7DFAST4PXKEUIVTMFL7HV4GZTCLQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.620838659 (6h0m17.993418014s) [seg_pos={0 0}, upload=4769]: Segment reached optimal threshold of 14 pieces. Jul 19 04:51:11.621067807 (6h0m17.993647163s) [noise=true , address=77.90.31.11:4564 , piece_id=L2U7JFJRDQOFA2AYVAGTI4ZW4VABCXKMXWBIUWQO2QNNUUBJ74BA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection; write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection Jul 19 04:51:11.621134234 (6h0m17.993713589s) [noise=true , address=77.90.31.10:4546 , piece_id=R6LCA5WZOEBDURL5S3MEK22YPVYXX6EXSS4J4W57543HELYXOKZA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK): protocol: read tcp 188.166.225.170:58476->77.90.31.10:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621174191 (6h0m17.993753546s) [noise=true , address=77.90.31.16:4768 , piece_id=YNLSONFM6COZEVF6ZNQR2EA6V4LSQFZJONIXR4AFTNNMFGPV57JA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UchewmtxBrtSZSpVAJFdZmcTsQVQxkjK9re6cgUGqDhyonQeF): protocol: read tcp 188.166.225.170:56326->77.90.31.16:4768: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621216842 (6h0m17.993796197s) [noise=true , address=77.90.31.18:4813 , piece_id=VDOT7BB4O7XYJHW6BFTAYQZZYLZU5PAUXVJGSXJ7FDI4TH425COA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128WthA1x9wp1dV1vkpGnoveZQpU9CSgDCZfw5GMQuVpCNSVAM5): protocol: read tcp 188.166.225.170:49704->77.90.31.18:4813: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621234917 (6h0m17.993814272s) [seg_pos={0 0}, upload=4769]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 19 04:51:11.621301444 (6h0m17.993880799s) [upload=4769]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 19 04:51:11.621305402 (6h0m17.993884758s) [upload=4769]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 19 04:51:11.923136970 (6h0m18.295716325s) [upload=4769]: Done uploading segments... Jul 19 04:51:11.923156627 (6h0m18.295735972s) [upload=4769]: Upload finished. err: <nil> Duration: 7.123836702s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [...........................XX...X...X....X..X....XXXXX.X.X..] duration: 15.087940088s FAIL (2025-07-19 05:10:21 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== : <nil> Jul 19 05:08:29.800544823 (6h17m36.173124169s) [upload=4997]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 19 05:08:29.800548410 (6h17m36.173127765s) [upload=4997]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 19 05:08:30.148027600 (6h17m36.520606955s) [upload=4997]: Done uploading segments... Jul 19 05:08:30.148148401 (6h17m36.520727746s) [upload=4997]: Upload finished. err: <nil> Jul 19 05:08:48.807983386 (6h17m55.180562741s) [upload=5005]: Starting upload Jul 19 05:08:48.808044453 (6h17m55.180623808s) [upload=5005]: Done starting upload Jul 19 05:08:48.808175243 (6h17m55.180754598s) [upload=5005]: Uploading segments... Jul 19 05:08:48.808334198 (6h17m55.180913553s) [upload=5005]: Got next segment. Inline: false Jul 19 05:08:48.808392509 (6h17m55.180971874s) [upload=5005]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 05:08:48.808399823 (6h17m55.180979188s) [upload=5005]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 05:08:49.075515233 (6h17m55.448094578s) [seg_pos={0 0}, upload=5005]: Begin upload segment... Jul 19 05:08:49.075564547 (6h17m55.448143902s) [seg_pos={0 0}, upload=5005]: Done begin upload segment. Jul 19 05:08:49.075602881 (6h17m55.448182236s) [noise=true , address=77.90.31.11:4555 , piece_id=QGRXVFSKXDCLKSSSXEK4QYZEETWQGFZMQQBSTXNMAHY45RA6MAVQ, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.075612279 (6h17m55.448191634s) [noise=true , address=104.36.82.13:4186 , piece_id=PPK6XEQYJ3XV27XUJYKUL2MYAR5KUYKF5AHJS5EBXOK7PZWWM66Q, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.075964213 (6h17m55.448543568s) [noise=true , address=77.90.31.12:4592 , piece_id=HS5YQKPTPUVHFZH3VZ7X2LAO2SVS6JZOTHTLJ6T47G3CLTZTR7UQ, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.076122025 (6h17m55.44870137s) [noise=true , address=104.36.82.12:4150 , piece_id=GQ4GQQZHK3Y4KVVB2IUU4LEEBR4OJ2YSS5Z5DF7WHHDCYYSSGRYA, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.076272903 (6h17m55.448852258s) [noise=true , address=45.91.116.13:4154 , piece_id=OJO7VBXGBLV2SG2AQ6BPBP57SL75WV4UZSWC3O5HXD7ROBICSYGA, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.076533412 (6h17m55.449112767s) [noise=true , address=104.36.82.2:4732 , piece_id=DBPJH5FSNFRPIBLP7HWWJFVVXYC47J7UGPCXFL7L3BD6BGFSJDFA, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.076794962 (6h17m55.449374317s) [noise=true , address=45.91.116.9:4263 , piece_id=4L3UWFKV3AC7F4RPS7GJFXOO6DJNCPMFB5NWGJH6HXAFBGLYAMYA, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.077044971 (6h17m55.449624326s) [noise=true , address=104.36.82.16:4293 , piece_id=2NA7I27WVAWAUWDCZVYBJLNXGWIDNJMKZU7YFJZSXCQMCHKLCGXA, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.077296953 (6h17m55.449876308s) [noise=true , address=77.90.31.15:4698 , piece_id=HNTVTQMVXDUMK4CMHKVUZ7DHM3MZL56TGXFR4FFFV2V3DCTLAVZQ, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.077501123 (6h17m55.450080468s) [noise=true , address=104.36.82.9:4007 , piece_id=R2IPGORJ3Z4FFCC52P5WMXYIA6O5V7DDZHHL4X64XV6J36SIWSBQ, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.077695656 (6h17m55.450275001s) [noise=true , address=45.91.116.8:4048 , piece_id=QKXGR6YZSYTQIUXV54DRYQNWY352UFS2Z2VXWFJTD7YJYJZ24NCQ, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.077994558 (6h17m55.450573913s) [noise=true , address=104.36.82.14:4222 , piece_id=4F6ZDS2KMQNALT2YV6D3TAZLIBFUFBM5NNCBGLSE6L6PR5QMGPVA, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.078213187 (6h17m55.450792542s) [noise=true , address=45.91.116.4:4939 , piece_id=J2BIIQITH7R54K4ENE6AKU7KCOUAGYBOTKX6VF6FY7FNQDMNNFCA, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.078412077 (6h17m55.450991432s) [noise=true , address=77.90.31.13:4626 , piece_id=2QAZAYJWKJXTPY475RLOSPV3ZZTMB6OZLNYAKD7FCFV7T4WPC5ZQ, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.079649705 (6h17m55.45222906s) [noise=true , address=45.91.116.9:4262 , piece_id=EXWTIYPESLJQK3CDK732WC63OZ7OJOQAUCBPGO3ZU54UBFUJ3TSA, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.080903344 (6h17m55.453482699s) [noise=true , address=77.90.31.13:4628 , piece_id=OAGGFQO2CORAZQV6ND3EERSICEOSHKK73Y3PUWLT45ZUWEWDHOFQ, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:50.857952826 (6h17m57.230532181s) [upload=5005]: Next returned nil segment Jul 19 05:08:50.857969477 (6h17m57.230548832s) [upload=5005]: Waiting for error group managing segments... Jul 19 05:08:51.139405061 (6h17m57.511984416s) [noise=true , address=104.36.82.13:4186 , piece_id=PPK6XEQYJ3XV27XUJYKUL2MYAR5KUYKF5AHJS5EBXOK7PZWWM66Q, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:51.143987831 (6h17m57.516567176s) [noise=true , address=45.91.116.8:4048 , piece_id=QKXGR6YZSYTQIUXV54DRYQNWY352UFS2Z2VXWFJTD7YJYJZ24NCQ, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:51.148258443 (6h17m57.520837798s) [noise=true , address=104.36.82.16:4293 , piece_id=2NA7I27WVAWAUWDCZVYBJLNXGWIDNJMKZU7YFJZSXCQMCHKLCGXA, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:51.156796512 (6h17m57.529375867s) [noise=true , address=45.91.116.13:4154 , piece_id=OJO7VBXGBLV2SG2AQ6BPBP57SL75WV4UZSWC3O5HXD7ROBICSYGA, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:51.193908742 (6h17m57.566488098s) [noise=true , address=104.36.82.14:4222 , piece_id=4F6ZDS2KMQNALT2YV6D3TAZLIBFUFBM5NNCBGLSE6L6PR5QMGPVA, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:51.217498736 (6h17m57.590078091s) [noise=true , address=45.91.116.4:4939 , piece_id=J2BIIQITH7R54K4ENE6AKU7KCOUAGYBOTKX6VF6FY7FNQDMNNFCA, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:51.218585386 (6h17m57.591164741s) [noise=true , address=45.91.116.9:4263 , piece_id=4L3UWFKV3AC7F4RPS7GJFXOO6DJNCPMFB5NWGJH6HXAFBGLYAMYA, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:51.290646027 (6h17m57.663225392s) [noise=true , address=104.36.82.2:4732 , piece_id=DBPJH5FSNFRPIBLP7HWWJFVVXYC47J7UGPCXFL7L3BD6BGFSJDFA, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:51.309329961 (6h17m57.681909316s) [noise=true , address=104.36.82.12:4150 , piece_id=GQ4GQQZHK3Y4KVVB2IUU4LEEBR4OJ2YSS5Z5DF7WHHDCYYSSGRYA, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:51.317895332 (6h17m57.690474687s) [noise=true , address=104.36.82.9:4007 , piece_id=R2IPGORJ3Z4FFCC52P5WMXYIA6O5V7DDZHHL4X64XV6J36SIWSBQ, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:52.525479047 (6h17m58.898058402s) [noise=true , address=45.91.116.9:4262 , piece_id=EXWTIYPESLJQK3CDK732WC63OZ7OJOQAUCBPGO3ZU54UBFUJ3TSA, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:09:18.157484198 (6h18m24.530063553s) [noise=true , address=77.90.31.13:4628 , piece_id=OAGGFQO2CORAZQV6ND3EERSICEOSHKK73Y3PUWLT45ZUWEWDHOFQ, seg_pos={0 0}, upload=5005]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: expected piece hash; context deadline exceeded; EOF Jul 19 05:09:18.157617684 (6h18m24.530197039s) [noise=true , address=77.90.31.15:4698 , piece_id=HNTVTQMVXDUMK4CMHKVUZ7DHM3MZL56TGXFR4FFFV2V3DCTLAVZQ, seg_pos={0 0}, upload=5005]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF Jul 19 05:09:18.157744796 (6h18m24.530324151s) [noise=true , address=77.90.31.11:4555 , piece_id=QGRXVFSKXDCLKSSSXEK4QYZEETWQGFZMQQBSTXNMAHY45RA6MAVQ, seg_pos={0 0}, upload=5005]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: write tcp 188.166.225.170:38288->77.90.31.11:4555: use of closed network connection; write tcp 188.166.225.170:38288->77.90.31.11:4555: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38288->77.90.31.11:4555: use of closed network connection Jul 19 05:09:18.157785745 (6h18m24.5303651s) [noise=true , address=77.90.31.13:4626 , piece_id=2QAZAYJWKJXTPY475RLOSPV3ZZTMB6OZLNYAKD7FCFV7T4WPC5ZQ, seg_pos={0 0}, upload=5005]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:58846->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:58846->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58846->77.90.31.13:4626: use of closed network connection Jul 19 05:09:18.157859036 (6h18m24.530438391s) [noise=true , address=77.90.31.12:4592 , piece_id=HS5YQKPTPUVHFZH3VZ7X2LAO2SVS6JZOTHTLJ6T47G3CLTZTR7UQ, seg_pos={0 0}, upload=5005]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:38220->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:38220->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38220->77.90.31.12:4592: use of closed network connection Jul 19 05:09:18.157898821 (6h18m24.530478176s) [seg_pos={0 0}, upload=5005]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 19 05:09:18.389974737 (6h18m24.762554092s) [upload=5005]: Done uploading segments... Jul 19 05:09:18.389994916 (6h18m24.762574271s) [upload=5005]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 15.087967561s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 5.946452522s FAIL (2025-07-19 05:09:39 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 4855s) [upload=4769]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 04:51:09.109709627 (6h0m15.482288982s) [upload=4769]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 04:51:09.556145753 (6h0m15.928725108s) [seg_pos={0 0}, upload=4769]: Begin upload segment... Jul 19 04:51:09.556194817 (6h0m15.928774172s) [seg_pos={0 0}, upload=4769]: Done begin upload segment. Jul 19 04:51:09.556223893 (6h0m15.928803248s) [noise=true , address=79.127.219.38:4023 , piece_id=PTPOZGLYV4MZTOFZNN3NCGYL7DFAST4PXKEUIVTMFL7HV4GZTCLQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556252197 (6h0m15.928831552s) [noise=true , address=45.91.95.4:4011 , piece_id=ZMBAS6KKHYVIRE6OBL2NFVCWMM4TKQ7J64BD3SYXIK54XCEDI2XA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556579383 (6h0m15.929158738s) [noise=true , address=45.91.116.6:4007 , piece_id=A2AFVTZBXSGBOE7XIM3B5HRN4STITFBXO6MZRJ365URNAAOZ3PSQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556854469 (6h0m15.929433824s) [noise=true , address=104.36.82.19:4423 , piece_id=PIJUHDUND36HQL67I33D7A3EULMJHS7N55MN4OFQJ4OWNQ42O47A, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557094107 (6h0m15.929673462s) [noise=true , address=45.91.116.9:4264 , piece_id=GHM5BKZKMT2QOHHBOS4FSPOWEWSFQRGRVVR4GSNDPHVIFRZHQL2A, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557336381 (6h0m15.929915736s) [noise=true , address=79.127.205.235:4660 , piece_id=UOUOGE2VP7PX7EMRZLPMCWFCYEZERM3DQMCQ6NG4J2K2DEIZR63Q, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557557885 (6h0m15.93013724s) [noise=true , address=79.127.205.246:4280 , piece_id=FLBOZI3UGTQXAQBLZ2F45Z3OEYL65F5PQBXRF2DQMZ7NGF7VDRPA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557762296 (6h0m15.930341651s) [noise=true , address=79.127.205.233:4622 , piece_id=6WWNYQTH5MGAJOEB7BXKZDJOMCAZXDW66J3OEANGDNJDNUMLZM7Q, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558014689 (6h0m15.930594044s) [noise=true , address=64.49.13.138:4482 , piece_id=MAW2M4US2OQNOS3WI67AA7NIBTJDE5T2JQTXKFPBLGVGXKVCFGTQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558215974 (6h0m15.930795319s) [noise=true , address=77.90.31.10:4546 , piece_id=R6LCA5WZOEBDURL5S3MEK22YPVYXX6EXSS4J4W57543HELYXOKZA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558409364 (6h0m15.930988719s) [noise=true , address=45.91.95.5:4019 , piece_id=YTSBCN2LPHEMRRAZIHJGVB7NBOTRXVEHD2TWUCTTLBNFFYEWWDKA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558623504 (6h0m15.931202859s) [noise=true , address=79.127.201.215:4357 , piece_id=75XWAG2VQWEYCWN2Q3WRIZSBI7LDOISZJPXQLQ2GIPNEN34VLSUQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558805162 (6h0m15.931384517s) [noise=true , address=45.91.116.14:4219 , piece_id=PDW6JPRHCVMHHOWIHT6JKV4CTSC3JY6NDKDVFZSO47OYNPZ7PGGA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558999373 (6h0m15.931578728s) [noise=true , address=77.90.31.11:4564 , piece_id=L2U7JFJRDQOFA2AYVAGTI4ZW4VABCXKMXWBIUWQO2QNNUUBJ74BA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559184768 (6h0m15.931764123s) [noise=true , address=77.90.31.16:4768 , piece_id=YNLSONFM6COZEVF6ZNQR2EA6V4LSQFZJONIXR4AFTNNMFGPV57JA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559368719 (6h0m15.931948074s) [noise=true , address=79.127.220.98:4432 , piece_id=2SUTFQ7FVDPAM2H4W3OLW6AVH3VJ33IJFE4L52SA3ET5LGBMOMPQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559549706 (6h0m15.932129061s) [noise=true , address=77.90.31.18:4813 , piece_id=VDOT7BB4O7XYJHW6BFTAYQZZYLZU5PAUXVJGSXJ7FDI4TH425COA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559727836 (6h0m15.932307191s) [noise=true , address=79.127.219.51:4211 , piece_id=HO5PLWBVU5YAYNYMN352NA73SM4WBJV4WZ7XLSQTCQ76UAL2TOJA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:10.663359087 (6h0m17.035938442s) [upload=4769]: Next returned nil segment Jul 19 04:51:10.663378664 (6h0m17.035958019s) [upload=4769]: Waiting for error group managing segments... Jul 19 04:51:10.962945479 (6h0m17.335524834s) [noise=true , address=79.127.205.233:4622 , piece_id=6WWNYQTH5MGAJOEB7BXKZDJOMCAZXDW66J3OEANGDNJDNUMLZM7Q, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:10.967689807 (6h0m17.340269152s) [noise=true , address=79.127.205.235:4660 , piece_id=UOUOGE2VP7PX7EMRZLPMCWFCYEZERM3DQMCQ6NG4J2K2DEIZR63Q, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:10.976968631 (6h0m17.349547986s) [noise=true , address=64.49.13.138:4482 , piece_id=MAW2M4US2OQNOS3WI67AA7NIBTJDE5T2JQTXKFPBLGVGXKVCFGTQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.048750700 (6h0m17.421330055s) [noise=true , address=79.127.205.246:4280 , piece_id=FLBOZI3UGTQXAQBLZ2F45Z3OEYL65F5PQBXRF2DQMZ7NGF7VDRPA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.221642403 (6h0m17.594221758s) [noise=true , address=79.127.220.98:4432 , piece_id=2SUTFQ7FVDPAM2H4W3OLW6AVH3VJ33IJFE4L52SA3ET5LGBMOMPQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.317070435 (6h0m17.68964979s) [noise=true , address=45.91.95.4:4011 , piece_id=ZMBAS6KKHYVIRE6OBL2NFVCWMM4TKQ7J64BD3SYXIK54XCEDI2XA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.324002660 (6h0m17.696582015s) [noise=true , address=45.91.95.5:4019 , piece_id=YTSBCN2LPHEMRRAZIHJGVB7NBOTRXVEHD2TWUCTTLBNFFYEWWDKA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.460976317 (6h0m17.833555672s) [noise=true , address=79.127.201.215:4357 , piece_id=75XWAG2VQWEYCWN2Q3WRIZSBI7LDOISZJPXQLQ2GIPNEN34VLSUQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.462584603 (6h0m17.835163958s) [noise=true , address=45.91.116.6:4007 , piece_id=A2AFVTZBXSGBOE7XIM3B5HRN4STITFBXO6MZRJ365URNAAOZ3PSQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.490392973 (6h0m17.862972328s) [noise=true , address=45.91.116.9:4264 , piece_id=GHM5BKZKMT2QOHHBOS4FSPOWEWSFQRGRVVR4GSNDPHVIFRZHQL2A, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.502040780 (6h0m17.874620135s) [noise=true , address=45.91.116.14:4219 , piece_id=PDW6JPRHCVMHHOWIHT6JKV4CTSC3JY6NDKDVFZSO47OYNPZ7PGGA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.600434096 (6h0m17.973013451s) [noise=true , address=79.127.219.51:4211 , piece_id=HO5PLWBVU5YAYNYMN352NA73SM4WBJV4WZ7XLSQTCQ76UAL2TOJA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.609008984 (6h0m17.981588339s) [noise=true , address=104.36.82.19:4423 , piece_id=PIJUHDUND36HQL67I33D7A3EULMJHS7N55MN4OFQJ4OWNQ42O47A, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.620825273 (6h0m17.993404628s) [noise=true , address=79.127.219.38:4023 , piece_id=PTPOZGLYV4MZTOFZNN3NCGYL7DFAST4PXKEUIVTMFL7HV4GZTCLQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.620838659 (6h0m17.993418014s) [seg_pos={0 0}, upload=4769]: Segment reached optimal threshold of 14 pieces. Jul 19 04:51:11.621067807 (6h0m17.993647163s) [noise=true , address=77.90.31.11:4564 , piece_id=L2U7JFJRDQOFA2AYVAGTI4ZW4VABCXKMXWBIUWQO2QNNUUBJ74BA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection; write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection Jul 19 04:51:11.621134234 (6h0m17.993713589s) [noise=true , address=77.90.31.10:4546 , piece_id=R6LCA5WZOEBDURL5S3MEK22YPVYXX6EXSS4J4W57543HELYXOKZA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK): protocol: read tcp 188.166.225.170:58476->77.90.31.10:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621174191 (6h0m17.993753546s) [noise=true , address=77.90.31.16:4768 , piece_id=YNLSONFM6COZEVF6ZNQR2EA6V4LSQFZJONIXR4AFTNNMFGPV57JA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UchewmtxBrtSZSpVAJFdZmcTsQVQxkjK9re6cgUGqDhyonQeF): protocol: read tcp 188.166.225.170:56326->77.90.31.16:4768: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621216842 (6h0m17.993796197s) [noise=true , address=77.90.31.18:4813 , piece_id=VDOT7BB4O7XYJHW6BFTAYQZZYLZU5PAUXVJGSXJ7FDI4TH425COA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128WthA1x9wp1dV1vkpGnoveZQpU9CSgDCZfw5GMQuVpCNSVAM5): protocol: read tcp 188.166.225.170:49704->77.90.31.18:4813: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621234917 (6h0m17.993814272s) [seg_pos={0 0}, upload=4769]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 19 04:51:11.621301444 (6h0m17.993880799s) [upload=4769]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 19 04:51:11.621305402 (6h0m17.993884758s) [upload=4769]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 19 04:51:11.923136970 (6h0m18.295716325s) [upload=4769]: Done uploading segments... Jul 19 04:51:11.923156627 (6h0m18.295735972s) [upload=4769]: Upload finished. err: <nil> Duration: 5.946481658s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [............................XX...X...X....X..X....XXXXX.X.X.] duration: 5.899609796s FAIL (2025-07-19 05:09:25 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== : <nil> Jul 19 05:08:29.800544823 (6h17m36.173124169s) [upload=4997]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 19 05:08:29.800548410 (6h17m36.173127765s) [upload=4997]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 19 05:08:30.148027600 (6h17m36.520606955s) [upload=4997]: Done uploading segments... Jul 19 05:08:30.148148401 (6h17m36.520727746s) [upload=4997]: Upload finished. err: <nil> Jul 19 05:08:48.807983386 (6h17m55.180562741s) [upload=5005]: Starting upload Jul 19 05:08:48.808044453 (6h17m55.180623808s) [upload=5005]: Done starting upload Jul 19 05:08:48.808175243 (6h17m55.180754598s) [upload=5005]: Uploading segments... Jul 19 05:08:48.808334198 (6h17m55.180913553s) [upload=5005]: Got next segment. Inline: false Jul 19 05:08:48.808392509 (6h17m55.180971874s) [upload=5005]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 05:08:48.808399823 (6h17m55.180979188s) [upload=5005]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 05:08:49.075515233 (6h17m55.448094578s) [seg_pos={0 0}, upload=5005]: Begin upload segment... Jul 19 05:08:49.075564547 (6h17m55.448143902s) [seg_pos={0 0}, upload=5005]: Done begin upload segment. Jul 19 05:08:49.075602881 (6h17m55.448182236s) [noise=true , address=77.90.31.11:4555 , piece_id=QGRXVFSKXDCLKSSSXEK4QYZEETWQGFZMQQBSTXNMAHY45RA6MAVQ, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.075612279 (6h17m55.448191634s) [noise=true , address=104.36.82.13:4186 , piece_id=PPK6XEQYJ3XV27XUJYKUL2MYAR5KUYKF5AHJS5EBXOK7PZWWM66Q, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.075964213 (6h17m55.448543568s) [noise=true , address=77.90.31.12:4592 , piece_id=HS5YQKPTPUVHFZH3VZ7X2LAO2SVS6JZOTHTLJ6T47G3CLTZTR7UQ, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.076122025 (6h17m55.44870137s) [noise=true , address=104.36.82.12:4150 , piece_id=GQ4GQQZHK3Y4KVVB2IUU4LEEBR4OJ2YSS5Z5DF7WHHDCYYSSGRYA, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.076272903 (6h17m55.448852258s) [noise=true , address=45.91.116.13:4154 , piece_id=OJO7VBXGBLV2SG2AQ6BPBP57SL75WV4UZSWC3O5HXD7ROBICSYGA, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.076533412 (6h17m55.449112767s) [noise=true , address=104.36.82.2:4732 , piece_id=DBPJH5FSNFRPIBLP7HWWJFVVXYC47J7UGPCXFL7L3BD6BGFSJDFA, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.076794962 (6h17m55.449374317s) [noise=true , address=45.91.116.9:4263 , piece_id=4L3UWFKV3AC7F4RPS7GJFXOO6DJNCPMFB5NWGJH6HXAFBGLYAMYA, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.077044971 (6h17m55.449624326s) [noise=true , address=104.36.82.16:4293 , piece_id=2NA7I27WVAWAUWDCZVYBJLNXGWIDNJMKZU7YFJZSXCQMCHKLCGXA, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.077296953 (6h17m55.449876308s) [noise=true , address=77.90.31.15:4698 , piece_id=HNTVTQMVXDUMK4CMHKVUZ7DHM3MZL56TGXFR4FFFV2V3DCTLAVZQ, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.077501123 (6h17m55.450080468s) [noise=true , address=104.36.82.9:4007 , piece_id=R2IPGORJ3Z4FFCC52P5WMXYIA6O5V7DDZHHL4X64XV6J36SIWSBQ, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.077695656 (6h17m55.450275001s) [noise=true , address=45.91.116.8:4048 , piece_id=QKXGR6YZSYTQIUXV54DRYQNWY352UFS2Z2VXWFJTD7YJYJZ24NCQ, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.077994558 (6h17m55.450573913s) [noise=true , address=104.36.82.14:4222 , piece_id=4F6ZDS2KMQNALT2YV6D3TAZLIBFUFBM5NNCBGLSE6L6PR5QMGPVA, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.078213187 (6h17m55.450792542s) [noise=true , address=45.91.116.4:4939 , piece_id=J2BIIQITH7R54K4ENE6AKU7KCOUAGYBOTKX6VF6FY7FNQDMNNFCA, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.078412077 (6h17m55.450991432s) [noise=true , address=77.90.31.13:4626 , piece_id=2QAZAYJWKJXTPY475RLOSPV3ZZTMB6OZLNYAKD7FCFV7T4WPC5ZQ, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.079649705 (6h17m55.45222906s) [noise=true , address=45.91.116.9:4262 , piece_id=EXWTIYPESLJQK3CDK732WC63OZ7OJOQAUCBPGO3ZU54UBFUJ3TSA, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.080903344 (6h17m55.453482699s) [noise=true , address=77.90.31.13:4628 , piece_id=OAGGFQO2CORAZQV6ND3EERSICEOSHKK73Y3PUWLT45ZUWEWDHOFQ, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:50.857952826 (6h17m57.230532181s) [upload=5005]: Next returned nil segment Jul 19 05:08:50.857969477 (6h17m57.230548832s) [upload=5005]: Waiting for error group managing segments... Jul 19 05:08:51.139405061 (6h17m57.511984416s) [noise=true , address=104.36.82.13:4186 , piece_id=PPK6XEQYJ3XV27XUJYKUL2MYAR5KUYKF5AHJS5EBXOK7PZWWM66Q, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:51.143987831 (6h17m57.516567176s) [noise=true , address=45.91.116.8:4048 , piece_id=QKXGR6YZSYTQIUXV54DRYQNWY352UFS2Z2VXWFJTD7YJYJZ24NCQ, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:51.148258443 (6h17m57.520837798s) [noise=true , address=104.36.82.16:4293 , piece_id=2NA7I27WVAWAUWDCZVYBJLNXGWIDNJMKZU7YFJZSXCQMCHKLCGXA, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:51.156796512 (6h17m57.529375867s) [noise=true , address=45.91.116.13:4154 , piece_id=OJO7VBXGBLV2SG2AQ6BPBP57SL75WV4UZSWC3O5HXD7ROBICSYGA, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:51.193908742 (6h17m57.566488098s) [noise=true , address=104.36.82.14:4222 , piece_id=4F6ZDS2KMQNALT2YV6D3TAZLIBFUFBM5NNCBGLSE6L6PR5QMGPVA, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:51.217498736 (6h17m57.590078091s) [noise=true , address=45.91.116.4:4939 , piece_id=J2BIIQITH7R54K4ENE6AKU7KCOUAGYBOTKX6VF6FY7FNQDMNNFCA, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:51.218585386 (6h17m57.591164741s) [noise=true , address=45.91.116.9:4263 , piece_id=4L3UWFKV3AC7F4RPS7GJFXOO6DJNCPMFB5NWGJH6HXAFBGLYAMYA, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:51.290646027 (6h17m57.663225392s) [noise=true , address=104.36.82.2:4732 , piece_id=DBPJH5FSNFRPIBLP7HWWJFVVXYC47J7UGPCXFL7L3BD6BGFSJDFA, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:51.309329961 (6h17m57.681909316s) [noise=true , address=104.36.82.12:4150 , piece_id=GQ4GQQZHK3Y4KVVB2IUU4LEEBR4OJ2YSS5Z5DF7WHHDCYYSSGRYA, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:51.317895332 (6h17m57.690474687s) [noise=true , address=104.36.82.9:4007 , piece_id=R2IPGORJ3Z4FFCC52P5WMXYIA6O5V7DDZHHL4X64XV6J36SIWSBQ, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:52.525479047 (6h17m58.898058402s) [noise=true , address=45.91.116.9:4262 , piece_id=EXWTIYPESLJQK3CDK732WC63OZ7OJOQAUCBPGO3ZU54UBFUJ3TSA, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:09:18.157484198 (6h18m24.530063553s) [noise=true , address=77.90.31.13:4628 , piece_id=OAGGFQO2CORAZQV6ND3EERSICEOSHKK73Y3PUWLT45ZUWEWDHOFQ, seg_pos={0 0}, upload=5005]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: expected piece hash; context deadline exceeded; EOF Jul 19 05:09:18.157617684 (6h18m24.530197039s) [noise=true , address=77.90.31.15:4698 , piece_id=HNTVTQMVXDUMK4CMHKVUZ7DHM3MZL56TGXFR4FFFV2V3DCTLAVZQ, seg_pos={0 0}, upload=5005]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF Jul 19 05:09:18.157744796 (6h18m24.530324151s) [noise=true , address=77.90.31.11:4555 , piece_id=QGRXVFSKXDCLKSSSXEK4QYZEETWQGFZMQQBSTXNMAHY45RA6MAVQ, seg_pos={0 0}, upload=5005]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: write tcp 188.166.225.170:38288->77.90.31.11:4555: use of closed network connection; write tcp 188.166.225.170:38288->77.90.31.11:4555: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38288->77.90.31.11:4555: use of closed network connection Jul 19 05:09:18.157785745 (6h18m24.5303651s) [noise=true , address=77.90.31.13:4626 , piece_id=2QAZAYJWKJXTPY475RLOSPV3ZZTMB6OZLNYAKD7FCFV7T4WPC5ZQ, seg_pos={0 0}, upload=5005]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:58846->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:58846->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58846->77.90.31.13:4626: use of closed network connection Jul 19 05:09:18.157859036 (6h18m24.530438391s) [noise=true , address=77.90.31.12:4592 , piece_id=HS5YQKPTPUVHFZH3VZ7X2LAO2SVS6JZOTHTLJ6T47G3CLTZTR7UQ, seg_pos={0 0}, upload=5005]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:38220->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:38220->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38220->77.90.31.12:4592: use of closed network connection Jul 19 05:09:18.157898821 (6h18m24.530478176s) [seg_pos={0 0}, upload=5005]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 19 05:09:18.389974737 (6h18m24.762554092s) [upload=5005]: Done uploading segments... Jul 19 05:09:18.389994916 (6h18m24.762574271s) [upload=5005]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.899642679s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 27.567721207s FAIL (2025-07-19 05:09:19 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 4855s) [upload=4769]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 04:51:09.109709627 (6h0m15.482288982s) [upload=4769]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 04:51:09.556145753 (6h0m15.928725108s) [seg_pos={0 0}, upload=4769]: Begin upload segment... Jul 19 04:51:09.556194817 (6h0m15.928774172s) [seg_pos={0 0}, upload=4769]: Done begin upload segment. Jul 19 04:51:09.556223893 (6h0m15.928803248s) [noise=true , address=79.127.219.38:4023 , piece_id=PTPOZGLYV4MZTOFZNN3NCGYL7DFAST4PXKEUIVTMFL7HV4GZTCLQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556252197 (6h0m15.928831552s) [noise=true , address=45.91.95.4:4011 , piece_id=ZMBAS6KKHYVIRE6OBL2NFVCWMM4TKQ7J64BD3SYXIK54XCEDI2XA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556579383 (6h0m15.929158738s) [noise=true , address=45.91.116.6:4007 , piece_id=A2AFVTZBXSGBOE7XIM3B5HRN4STITFBXO6MZRJ365URNAAOZ3PSQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556854469 (6h0m15.929433824s) [noise=true , address=104.36.82.19:4423 , piece_id=PIJUHDUND36HQL67I33D7A3EULMJHS7N55MN4OFQJ4OWNQ42O47A, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557094107 (6h0m15.929673462s) [noise=true , address=45.91.116.9:4264 , piece_id=GHM5BKZKMT2QOHHBOS4FSPOWEWSFQRGRVVR4GSNDPHVIFRZHQL2A, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557336381 (6h0m15.929915736s) [noise=true , address=79.127.205.235:4660 , piece_id=UOUOGE2VP7PX7EMRZLPMCWFCYEZERM3DQMCQ6NG4J2K2DEIZR63Q, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557557885 (6h0m15.93013724s) [noise=true , address=79.127.205.246:4280 , piece_id=FLBOZI3UGTQXAQBLZ2F45Z3OEYL65F5PQBXRF2DQMZ7NGF7VDRPA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557762296 (6h0m15.930341651s) [noise=true , address=79.127.205.233:4622 , piece_id=6WWNYQTH5MGAJOEB7BXKZDJOMCAZXDW66J3OEANGDNJDNUMLZM7Q, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558014689 (6h0m15.930594044s) [noise=true , address=64.49.13.138:4482 , piece_id=MAW2M4US2OQNOS3WI67AA7NIBTJDE5T2JQTXKFPBLGVGXKVCFGTQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558215974 (6h0m15.930795319s) [noise=true , address=77.90.31.10:4546 , piece_id=R6LCA5WZOEBDURL5S3MEK22YPVYXX6EXSS4J4W57543HELYXOKZA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558409364 (6h0m15.930988719s) [noise=true , address=45.91.95.5:4019 , piece_id=YTSBCN2LPHEMRRAZIHJGVB7NBOTRXVEHD2TWUCTTLBNFFYEWWDKA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558623504 (6h0m15.931202859s) [noise=true , address=79.127.201.215:4357 , piece_id=75XWAG2VQWEYCWN2Q3WRIZSBI7LDOISZJPXQLQ2GIPNEN34VLSUQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558805162 (6h0m15.931384517s) [noise=true , address=45.91.116.14:4219 , piece_id=PDW6JPRHCVMHHOWIHT6JKV4CTSC3JY6NDKDVFZSO47OYNPZ7PGGA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558999373 (6h0m15.931578728s) [noise=true , address=77.90.31.11:4564 , piece_id=L2U7JFJRDQOFA2AYVAGTI4ZW4VABCXKMXWBIUWQO2QNNUUBJ74BA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559184768 (6h0m15.931764123s) [noise=true , address=77.90.31.16:4768 , piece_id=YNLSONFM6COZEVF6ZNQR2EA6V4LSQFZJONIXR4AFTNNMFGPV57JA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559368719 (6h0m15.931948074s) [noise=true , address=79.127.220.98:4432 , piece_id=2SUTFQ7FVDPAM2H4W3OLW6AVH3VJ33IJFE4L52SA3ET5LGBMOMPQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559549706 (6h0m15.932129061s) [noise=true , address=77.90.31.18:4813 , piece_id=VDOT7BB4O7XYJHW6BFTAYQZZYLZU5PAUXVJGSXJ7FDI4TH425COA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559727836 (6h0m15.932307191s) [noise=true , address=79.127.219.51:4211 , piece_id=HO5PLWBVU5YAYNYMN352NA73SM4WBJV4WZ7XLSQTCQ76UAL2TOJA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:10.663359087 (6h0m17.035938442s) [upload=4769]: Next returned nil segment Jul 19 04:51:10.663378664 (6h0m17.035958019s) [upload=4769]: Waiting for error group managing segments... Jul 19 04:51:10.962945479 (6h0m17.335524834s) [noise=true , address=79.127.205.233:4622 , piece_id=6WWNYQTH5MGAJOEB7BXKZDJOMCAZXDW66J3OEANGDNJDNUMLZM7Q, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:10.967689807 (6h0m17.340269152s) [noise=true , address=79.127.205.235:4660 , piece_id=UOUOGE2VP7PX7EMRZLPMCWFCYEZERM3DQMCQ6NG4J2K2DEIZR63Q, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:10.976968631 (6h0m17.349547986s) [noise=true , address=64.49.13.138:4482 , piece_id=MAW2M4US2OQNOS3WI67AA7NIBTJDE5T2JQTXKFPBLGVGXKVCFGTQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.048750700 (6h0m17.421330055s) [noise=true , address=79.127.205.246:4280 , piece_id=FLBOZI3UGTQXAQBLZ2F45Z3OEYL65F5PQBXRF2DQMZ7NGF7VDRPA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.221642403 (6h0m17.594221758s) [noise=true , address=79.127.220.98:4432 , piece_id=2SUTFQ7FVDPAM2H4W3OLW6AVH3VJ33IJFE4L52SA3ET5LGBMOMPQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.317070435 (6h0m17.68964979s) [noise=true , address=45.91.95.4:4011 , piece_id=ZMBAS6KKHYVIRE6OBL2NFVCWMM4TKQ7J64BD3SYXIK54XCEDI2XA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.324002660 (6h0m17.696582015s) [noise=true , address=45.91.95.5:4019 , piece_id=YTSBCN2LPHEMRRAZIHJGVB7NBOTRXVEHD2TWUCTTLBNFFYEWWDKA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.460976317 (6h0m17.833555672s) [noise=true , address=79.127.201.215:4357 , piece_id=75XWAG2VQWEYCWN2Q3WRIZSBI7LDOISZJPXQLQ2GIPNEN34VLSUQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.462584603 (6h0m17.835163958s) [noise=true , address=45.91.116.6:4007 , piece_id=A2AFVTZBXSGBOE7XIM3B5HRN4STITFBXO6MZRJ365URNAAOZ3PSQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.490392973 (6h0m17.862972328s) [noise=true , address=45.91.116.9:4264 , piece_id=GHM5BKZKMT2QOHHBOS4FSPOWEWSFQRGRVVR4GSNDPHVIFRZHQL2A, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.502040780 (6h0m17.874620135s) [noise=true , address=45.91.116.14:4219 , piece_id=PDW6JPRHCVMHHOWIHT6JKV4CTSC3JY6NDKDVFZSO47OYNPZ7PGGA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.600434096 (6h0m17.973013451s) [noise=true , address=79.127.219.51:4211 , piece_id=HO5PLWBVU5YAYNYMN352NA73SM4WBJV4WZ7XLSQTCQ76UAL2TOJA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.609008984 (6h0m17.981588339s) [noise=true , address=104.36.82.19:4423 , piece_id=PIJUHDUND36HQL67I33D7A3EULMJHS7N55MN4OFQJ4OWNQ42O47A, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.620825273 (6h0m17.993404628s) [noise=true , address=79.127.219.38:4023 , piece_id=PTPOZGLYV4MZTOFZNN3NCGYL7DFAST4PXKEUIVTMFL7HV4GZTCLQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.620838659 (6h0m17.993418014s) [seg_pos={0 0}, upload=4769]: Segment reached optimal threshold of 14 pieces. Jul 19 04:51:11.621067807 (6h0m17.993647163s) [noise=true , address=77.90.31.11:4564 , piece_id=L2U7JFJRDQOFA2AYVAGTI4ZW4VABCXKMXWBIUWQO2QNNUUBJ74BA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection; write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection Jul 19 04:51:11.621134234 (6h0m17.993713589s) [noise=true , address=77.90.31.10:4546 , piece_id=R6LCA5WZOEBDURL5S3MEK22YPVYXX6EXSS4J4W57543HELYXOKZA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK): protocol: read tcp 188.166.225.170:58476->77.90.31.10:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621174191 (6h0m17.993753546s) [noise=true , address=77.90.31.16:4768 , piece_id=YNLSONFM6COZEVF6ZNQR2EA6V4LSQFZJONIXR4AFTNNMFGPV57JA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UchewmtxBrtSZSpVAJFdZmcTsQVQxkjK9re6cgUGqDhyonQeF): protocol: read tcp 188.166.225.170:56326->77.90.31.16:4768: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621216842 (6h0m17.993796197s) [noise=true , address=77.90.31.18:4813 , piece_id=VDOT7BB4O7XYJHW6BFTAYQZZYLZU5PAUXVJGSXJ7FDI4TH425COA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128WthA1x9wp1dV1vkpGnoveZQpU9CSgDCZfw5GMQuVpCNSVAM5): protocol: read tcp 188.166.225.170:49704->77.90.31.18:4813: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621234917 (6h0m17.993814272s) [seg_pos={0 0}, upload=4769]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 19 04:51:11.621301444 (6h0m17.993880799s) [upload=4769]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 19 04:51:11.621305402 (6h0m17.993884758s) [upload=4769]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 19 04:51:11.923136970 (6h0m18.295716325s) [upload=4769]: Done uploading segments... Jul 19 04:51:11.923156627 (6h0m18.295735972s) [upload=4769]: Upload finished. err: <nil> Duration: 27.567740253s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.............................XX...X...X....X..X....XXXXX.X.X] duration: 30.233523858s FAIL (2025-07-19 05:09:18 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== : <nil> Jul 19 05:08:29.800544823 (6h17m36.173124169s) [upload=4997]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 19 05:08:29.800548410 (6h17m36.173127765s) [upload=4997]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 19 05:08:30.148027600 (6h17m36.520606955s) [upload=4997]: Done uploading segments... Jul 19 05:08:30.148148401 (6h17m36.520727746s) [upload=4997]: Upload finished. err: <nil> Jul 19 05:08:48.807983386 (6h17m55.180562741s) [upload=5005]: Starting upload Jul 19 05:08:48.808044453 (6h17m55.180623808s) [upload=5005]: Done starting upload Jul 19 05:08:48.808175243 (6h17m55.180754598s) [upload=5005]: Uploading segments... Jul 19 05:08:48.808334198 (6h17m55.180913553s) [upload=5005]: Got next segment. Inline: false Jul 19 05:08:48.808392509 (6h17m55.180971874s) [upload=5005]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 05:08:48.808399823 (6h17m55.180979188s) [upload=5005]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 05:08:49.075515233 (6h17m55.448094578s) [seg_pos={0 0}, upload=5005]: Begin upload segment... Jul 19 05:08:49.075564547 (6h17m55.448143902s) [seg_pos={0 0}, upload=5005]: Done begin upload segment. Jul 19 05:08:49.075602881 (6h17m55.448182236s) [noise=true , address=77.90.31.11:4555 , piece_id=QGRXVFSKXDCLKSSSXEK4QYZEETWQGFZMQQBSTXNMAHY45RA6MAVQ, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.075612279 (6h17m55.448191634s) [noise=true , address=104.36.82.13:4186 , piece_id=PPK6XEQYJ3XV27XUJYKUL2MYAR5KUYKF5AHJS5EBXOK7PZWWM66Q, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.075964213 (6h17m55.448543568s) [noise=true , address=77.90.31.12:4592 , piece_id=HS5YQKPTPUVHFZH3VZ7X2LAO2SVS6JZOTHTLJ6T47G3CLTZTR7UQ, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.076122025 (6h17m55.44870137s) [noise=true , address=104.36.82.12:4150 , piece_id=GQ4GQQZHK3Y4KVVB2IUU4LEEBR4OJ2YSS5Z5DF7WHHDCYYSSGRYA, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.076272903 (6h17m55.448852258s) [noise=true , address=45.91.116.13:4154 , piece_id=OJO7VBXGBLV2SG2AQ6BPBP57SL75WV4UZSWC3O5HXD7ROBICSYGA, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.076533412 (6h17m55.449112767s) [noise=true , address=104.36.82.2:4732 , piece_id=DBPJH5FSNFRPIBLP7HWWJFVVXYC47J7UGPCXFL7L3BD6BGFSJDFA, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.076794962 (6h17m55.449374317s) [noise=true , address=45.91.116.9:4263 , piece_id=4L3UWFKV3AC7F4RPS7GJFXOO6DJNCPMFB5NWGJH6HXAFBGLYAMYA, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.077044971 (6h17m55.449624326s) [noise=true , address=104.36.82.16:4293 , piece_id=2NA7I27WVAWAUWDCZVYBJLNXGWIDNJMKZU7YFJZSXCQMCHKLCGXA, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.077296953 (6h17m55.449876308s) [noise=true , address=77.90.31.15:4698 , piece_id=HNTVTQMVXDUMK4CMHKVUZ7DHM3MZL56TGXFR4FFFV2V3DCTLAVZQ, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.077501123 (6h17m55.450080468s) [noise=true , address=104.36.82.9:4007 , piece_id=R2IPGORJ3Z4FFCC52P5WMXYIA6O5V7DDZHHL4X64XV6J36SIWSBQ, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.077695656 (6h17m55.450275001s) [noise=true , address=45.91.116.8:4048 , piece_id=QKXGR6YZSYTQIUXV54DRYQNWY352UFS2Z2VXWFJTD7YJYJZ24NCQ, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.077994558 (6h17m55.450573913s) [noise=true , address=104.36.82.14:4222 , piece_id=4F6ZDS2KMQNALT2YV6D3TAZLIBFUFBM5NNCBGLSE6L6PR5QMGPVA, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.078213187 (6h17m55.450792542s) [noise=true , address=45.91.116.4:4939 , piece_id=J2BIIQITH7R54K4ENE6AKU7KCOUAGYBOTKX6VF6FY7FNQDMNNFCA, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.078412077 (6h17m55.450991432s) [noise=true , address=77.90.31.13:4626 , piece_id=2QAZAYJWKJXTPY475RLOSPV3ZZTMB6OZLNYAKD7FCFV7T4WPC5ZQ, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.079649705 (6h17m55.45222906s) [noise=true , address=45.91.116.9:4262 , piece_id=EXWTIYPESLJQK3CDK732WC63OZ7OJOQAUCBPGO3ZU54UBFUJ3TSA, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.080903344 (6h17m55.453482699s) [noise=true , address=77.90.31.13:4628 , piece_id=OAGGFQO2CORAZQV6ND3EERSICEOSHKK73Y3PUWLT45ZUWEWDHOFQ, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:50.857952826 (6h17m57.230532181s) [upload=5005]: Next returned nil segment Jul 19 05:08:50.857969477 (6h17m57.230548832s) [upload=5005]: Waiting for error group managing segments... Jul 19 05:08:51.139405061 (6h17m57.511984416s) [noise=true , address=104.36.82.13:4186 , piece_id=PPK6XEQYJ3XV27XUJYKUL2MYAR5KUYKF5AHJS5EBXOK7PZWWM66Q, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:51.143987831 (6h17m57.516567176s) [noise=true , address=45.91.116.8:4048 , piece_id=QKXGR6YZSYTQIUXV54DRYQNWY352UFS2Z2VXWFJTD7YJYJZ24NCQ, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:51.148258443 (6h17m57.520837798s) [noise=true , address=104.36.82.16:4293 , piece_id=2NA7I27WVAWAUWDCZVYBJLNXGWIDNJMKZU7YFJZSXCQMCHKLCGXA, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:51.156796512 (6h17m57.529375867s) [noise=true , address=45.91.116.13:4154 , piece_id=OJO7VBXGBLV2SG2AQ6BPBP57SL75WV4UZSWC3O5HXD7ROBICSYGA, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:51.193908742 (6h17m57.566488098s) [noise=true , address=104.36.82.14:4222 , piece_id=4F6ZDS2KMQNALT2YV6D3TAZLIBFUFBM5NNCBGLSE6L6PR5QMGPVA, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:51.217498736 (6h17m57.590078091s) [noise=true , address=45.91.116.4:4939 , piece_id=J2BIIQITH7R54K4ENE6AKU7KCOUAGYBOTKX6VF6FY7FNQDMNNFCA, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:51.218585386 (6h17m57.591164741s) [noise=true , address=45.91.116.9:4263 , piece_id=4L3UWFKV3AC7F4RPS7GJFXOO6DJNCPMFB5NWGJH6HXAFBGLYAMYA, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:51.290646027 (6h17m57.663225392s) [noise=true , address=104.36.82.2:4732 , piece_id=DBPJH5FSNFRPIBLP7HWWJFVVXYC47J7UGPCXFL7L3BD6BGFSJDFA, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:51.309329961 (6h17m57.681909316s) [noise=true , address=104.36.82.12:4150 , piece_id=GQ4GQQZHK3Y4KVVB2IUU4LEEBR4OJ2YSS5Z5DF7WHHDCYYSSGRYA, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:51.317895332 (6h17m57.690474687s) [noise=true , address=104.36.82.9:4007 , piece_id=R2IPGORJ3Z4FFCC52P5WMXYIA6O5V7DDZHHL4X64XV6J36SIWSBQ, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:52.525479047 (6h17m58.898058402s) [noise=true , address=45.91.116.9:4262 , piece_id=EXWTIYPESLJQK3CDK732WC63OZ7OJOQAUCBPGO3ZU54UBFUJ3TSA, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:09:18.157484198 (6h18m24.530063553s) [noise=true , address=77.90.31.13:4628 , piece_id=OAGGFQO2CORAZQV6ND3EERSICEOSHKK73Y3PUWLT45ZUWEWDHOFQ, seg_pos={0 0}, upload=5005]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: expected piece hash; context deadline exceeded; EOF Jul 19 05:09:18.157617684 (6h18m24.530197039s) [noise=true , address=77.90.31.15:4698 , piece_id=HNTVTQMVXDUMK4CMHKVUZ7DHM3MZL56TGXFR4FFFV2V3DCTLAVZQ, seg_pos={0 0}, upload=5005]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF Jul 19 05:09:18.157744796 (6h18m24.530324151s) [noise=true , address=77.90.31.11:4555 , piece_id=QGRXVFSKXDCLKSSSXEK4QYZEETWQGFZMQQBSTXNMAHY45RA6MAVQ, seg_pos={0 0}, upload=5005]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: write tcp 188.166.225.170:38288->77.90.31.11:4555: use of closed network connection; write tcp 188.166.225.170:38288->77.90.31.11:4555: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38288->77.90.31.11:4555: use of closed network connection Jul 19 05:09:18.157785745 (6h18m24.5303651s) [noise=true , address=77.90.31.13:4626 , piece_id=2QAZAYJWKJXTPY475RLOSPV3ZZTMB6OZLNYAKD7FCFV7T4WPC5ZQ, seg_pos={0 0}, upload=5005]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:58846->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:58846->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58846->77.90.31.13:4626: use of closed network connection Jul 19 05:09:18.157859036 (6h18m24.530438391s) [noise=true , address=77.90.31.12:4592 , piece_id=HS5YQKPTPUVHFZH3VZ7X2LAO2SVS6JZOTHTLJ6T47G3CLTZTR7UQ, seg_pos={0 0}, upload=5005]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:38220->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:38220->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38220->77.90.31.12:4592: use of closed network connection Jul 19 05:09:18.157898821 (6h18m24.530478176s) [seg_pos={0 0}, upload=5005]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 19 05:09:18.389974737 (6h18m24.762554092s) [upload=5005]: Done uploading segments... Jul 19 05:09:18.389994916 (6h18m24.762574271s) [upload=5005]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.233549267s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [..............................XX...X...X....X..X....XXXXX.X.] duration: 22.433473572s FAIL (2025-07-19 05:08:32 UTC) orbiter warning: ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , upload=4978]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: read tcp 188.166.225.170:56432->77.90.31.21:4879: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 05:06:53.739102075 (6h16m0.11168143s) [seg_pos={0 0}, upload=4978]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 19 05:06:53.739197288 (6h16m0.111776673s) [upload=4978]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 19 05:06:53.739203930 (6h16m0.111783286s) [upload=4978]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 19 05:06:54.046241769 (6h16m0.418821124s) [upload=4978]: Done uploading segments... Jul 19 05:06:54.046260585 (6h16m0.41883994s) [upload=4978]: Upload finished. err: <nil> Jul 19 05:07:18.967786786 (6h16m25.340366131s) [upload=4987]: Starting upload Jul 19 05:07:18.967842954 (6h16m25.340422309s) [upload=4987]: Done starting upload Jul 19 05:07:18.967928819 (6h16m25.340508174s) [upload=4987]: Uploading segments... Jul 19 05:07:18.968231738 (6h16m25.340811094s) [upload=4987]: Got next segment. Inline: false Jul 19 05:07:18.968276884 (6h16m25.340856239s) [upload=4987]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 05:07:18.968282466 (6h16m25.340861821s) [upload=4987]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 05:07:19.221977372 (6h16m25.594556728s) [seg_pos={0 0}, upload=4987]: Begin upload segment... Jul 19 05:07:19.222023461 (6h16m25.594602816s) [seg_pos={0 0}, upload=4987]: Done begin upload segment. Jul 19 05:07:19.222042989 (6h16m25.594622334s) [noise=true , address=77.90.31.15:4699 , piece_id=AGUDMA4LGNU3DSZB42QLS7M4JEZ6N5O7PNOAUOMLIQJEGISI3W5Q, seg_pos={0 0}, upload=4987]: Uploading piece... Jul 19 05:07:19.222313176 (6h16m25.594892531s) [noise=true , address=77.90.31.15:4697 , piece_id=544MNCQQNU6A75WX2WIJQ6A3F4N4GCCSSX3C4UTT7WHH77QOCUHA, seg_pos={0 0}, upload=4987]: Uploading piece... Jul 19 05:07:19.222568183 (6h16m25.595147538s) [noise=true , address=104.36.82.4:4801 , piece_id=3O2Q25TEDWVGN5PKMUMVK2TGYGXKRDWPV3ZY32JL7HQCDO3FB6KA, seg_pos={0 0}, upload=4987]: Uploading piece... Jul 19 05:07:19.222804205 (6h16m25.59538356s) [noise=true , address=77.90.31.14:4663 , piece_id=6XQO4MRE2VYQHQAQR3BGOLYFPFDR3GNOQNO4NHNZJCQKGWVO6JAQ, seg_pos={0 0}, upload=4987]: Uploading piece... Jul 19 05:07:19.223029146 (6h16m25.595608501s) [noise=true , address=77.90.31.10:4520 , piece_id=FOH2YFXGUQ772SNJXDP7ZR7CF5LFZ3LGYKCIEOEM4LJHPSMRMYHQ, seg_pos={0 0}, upload=4987]: Uploading piece... Jul 19 05:07:19.223246592 (6h16m25.595825947s) [noise=true , address=45.91.116.6:4974 , piece_id=U3Z2FOR3EZI3P2MCADPDJL6JKKUSQODAYCD263AM22E2WMRIFX4A, seg_pos={0 0}, upload=4987]: Uploading piece... Jul 19 05:07:19.223512381 (6h16m25.596091736s) [noise=true , address=45.91.116.6:4973 , piece_id=HWJBA2OCLJKSN64HSRK7RULJHIQPVKIT3GWPK64Y32VUUYFEUTMQ, seg_pos={0 0}, upload=4987]: Uploading piece... Jul 19 05:07:19.223703747 (6h16m25.596283102s) [noise=true , address=77.90.31.17:4772 , piece_id=VP3JS4QYDXJ6R35MFRUK2XULQLGJJ23BSMM7IRWMVR3IISVEBLUQ, seg_pos={0 0}, upload=4987]: Uploading piece... Jul 19 05:07:19.223956139 (6h16m25.596535495s) [noise=true , address=77.90.31.12:4592 , piece_id=36WYINRSYTUIPPO76JUD7HAO6SX4NW25G37XPOU3YVT2QLACABIA, seg_pos={0 0}, upload=4987]: Uploading piece... Jul 19 05:07:19.224198243 (6h16m25.596777598s) [noise=true , address=104.36.82.5:4837 , piece_id=7YZBYKPWKCGARRJLE7WK3YRR536QC7BZZFUHYGCGQRTHRX3PTSPQ, seg_pos={0 0}, upload=4987]: Uploading piece... Jul 19 05:07:19.224392975 (6h16m25.59697233s) [noise=true , address=77.90.31.21:4878 , piece_id=PSLYCIRGFJ6RTL65HOEO3S2JDC737V5MJ6R3PGJJ3ODMDPNIHWCA, seg_pos={0 0}, upload=4987]: Uploading piece... Jul 19 05:07:19.224586385 (6h16m25.59716574s) [noise=true , address=77.90.31.18:4807 , piece_id=3S6MGZJ2IC7UVNLPWKJCPBCY23B2DLGIJ6ANTNPNB4UKDIW45Y7Q, seg_pos={0 0}, upload=4987]: Uploading piece... Jul 19 05:07:19.224772702 (6h16m25.597352057s) [noise=true , address=45.91.116.13:4154 , piece_id=JFXGESP7N4C4ZQ7TWKTOQVVKQ3APV6Y3C6NBN7OIKXYJMJ5RPHWA, seg_pos={0 0}, upload=4987]: Uploading piece... Jul 19 05:07:19.225006720 (6h16m25.597586075s) [noise=true , address=104.36.82.7:4909 , piece_id=YCN3RDTWQZLZNYCVOQ334JY2XZJFAXDXZ5CZBWOTN4I2XI5U3FMA, seg_pos={0 0}, upload=4987]: Uploading piece... Jul 19 05:07:19.225222964 (6h16m25.597802319s) [noise=true , address=45.91.116.9:4261 , piece_id=VGBFN7ZKPHS5S4SDWUHEKY6F7LS6JTVRY3ITUAGXHYWNWQMLJOIA, seg_pos={0 0}, upload=4987]: Uploading piece... Jul 19 05:07:19.225416364 (6h16m25.597995719s) [noise=true , address=45.91.116.4:4939 , piece_id=BLQFFPUNAGDF7FCF2JPLBPPUAEWI7FMG4DKD7TWRMPQTYX3LEUAQ, seg_pos={0 0}, upload=4987]: Uploading piece... Jul 19 05:07:20.980351225 (6h16m27.35293058s) [upload=4987]: Next returned nil segment Jul 19 05:07:20.980380271 (6h16m27.352959626s) [upload=4987]: Waiting for error group managing segments... Jul 19 05:07:21.267793974 (6h16m27.640373329s) [noise=true , address=104.36.82.4:4801 , piece_id=3O2Q25TEDWVGN5PKMUMVK2TGYGXKRDWPV3ZY32JL7HQCDO3FB6KA, seg_pos={0 0}, upload=4987]: Done uploading piece. err: <nil> Jul 19 05:07:21.285583227 (6h16m27.658162583s) [noise=true , address=45.91.116.13:4154 , piece_id=JFXGESP7N4C4ZQ7TWKTOQVVKQ3APV6Y3C6NBN7OIKXYJMJ5RPHWA, seg_pos={0 0}, upload=4987]: Done uploading piece. err: <nil> Jul 19 05:07:21.286980110 (6h16m27.659559465s) [noise=true , address=45.91.116.9:4261 , piece_id=VGBFN7ZKPHS5S4SDWUHEKY6F7LS6JTVRY3ITUAGXHYWNWQMLJOIA, seg_pos={0 0}, upload=4987]: Done uploading piece. err: <nil> Jul 19 05:07:21.359982162 (6h16m27.732561517s) [noise=true , address=104.36.82.5:4837 , piece_id=7YZBYKPWKCGARRJLE7WK3YRR536QC7BZZFUHYGCGQRTHRX3PTSPQ, seg_pos={0 0}, upload=4987]: Done uploading piece. err: <nil> Jul 19 05:07:21.363819385 (6h16m27.73639874s) [noise=true , address=45.91.116.6:4974 , piece_id=U3Z2FOR3EZI3P2MCADPDJL6JKKUSQODAYCD263AM22E2WMRIFX4A, seg_pos={0 0}, upload=4987]: Done uploading piece. err: <nil> Jul 19 05:07:21.372194202 (6h16m27.744773557s) [noise=true , address=45.91.116.6:4973 , piece_id=HWJBA2OCLJKSN64HSRK7RULJHIQPVKIT3GWPK64Y32VUUYFEUTMQ, seg_pos={0 0}, upload=4987]: Done uploading piece. err: <nil> Jul 19 05:07:21.375296359 (6h16m27.747875714s) [noise=true , address=104.36.82.7:4909 , piece_id=YCN3RDTWQZLZNYCVOQ334JY2XZJFAXDXZ5CZBWOTN4I2XI5U3FMA, seg_pos={0 0}, upload=4987]: Done uploading piece. err: <nil> Jul 19 05:07:21.381291291 (6h16m27.753870646s) [noise=true , address=45.91.116.4:4939 , piece_id=BLQFFPUNAGDF7FCF2JPLBPPUAEWI7FMG4DKD7TWRMPQTYX3LEUAQ, seg_pos={0 0}, upload=4987]: Done uploading piece. err: <nil> Jul 19 05:07:32.446172843 (6h16m38.818752198s) [noise=true , address=77.90.31.17:4772 , piece_id=VP3JS4QYDXJ6R35MFRUK2XULQLGJJ23BSMM7IRWMVR3IISVEBLUQ, seg_pos={0 0}, upload=4987]: Done uploading piece. err: <nil> Jul 19 05:07:37.950304448 (6h16m44.322883803s) [noise=true , address=77.90.31.10:4520 , piece_id=FOH2YFXGUQ772SNJXDP7ZR7CF5LFZ3LGYKCIEOEM4LJHPSMRMYHQ, seg_pos={0 0}, upload=4987]: Done uploading piece. err: <nil> Jul 19 05:07:38.051458593 (6h16m44.424037949s) [noise=true , address=77.90.31.15:4697 , piece_id=544MNCQQNU6A75WX2WIJQ6A3F4N4GCCSSX3C4UTT7WHH77QOCUHA, seg_pos={0 0}, upload=4987]: Done uploading piece. err: <nil> Jul 19 05:07:38.236231036 (6h16m44.608810401s) [noise=true , address=77.90.31.14:4663 , piece_id=6XQO4MRE2VYQHQAQR3BGOLYFPFDR3GNOQNO4NHNZJCQKGWVO6JAQ, seg_pos={0 0}, upload=4987]: Done uploading piece. err: <nil> Jul 19 05:07:48.143474611 (6h16m54.516053967s) [noise=true , address=77.90.31.21:4878 , piece_id=PSLYCIRGFJ6RTL65HOEO3S2JDC737V5MJ6R3PGJJ3ODMDPNIHWCA, seg_pos={0 0}, upload=4987]: Done uploading piece. err: <nil> Jul 19 05:07:48.143499839 (6h16m54.516079194s) [seg_pos={0 0}, upload=4987]: Segment reached optimal threshold of 13 pieces. Jul 19 05:07:48.143654295 (6h16m54.51623364s) [noise=true , address=77.90.31.18:4807 , piece_id=3S6MGZJ2IC7UVNLPWKJCPBCY23B2DLGIJ6ANTNPNB4UKDIW45Y7Q, seg_pos={0 0}, upload=4987]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: expected piece hash; context canceled; EOF Jul 19 05:07:48.143694843 (6h16m54.516274198s) [noise=true , address=77.90.31.12:4592 , piece_id=36WYINRSYTUIPPO76JUD7HAO6SX4NW25G37XPOU3YVT2QLACABIA, seg_pos={0 0}, upload=4987]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3): protocol: expected piece hash; context canceled; EOF Jul 19 05:07:48.143734959 (6h16m54.516314314s) [noise=true , address=77.90.31.15:4699 , piece_id=AGUDMA4LGNU3DSZB42QLS7M4JEZ6N5O7PNOAUOMLIQJEGISI3W5Q, seg_pos={0 0}, upload=4987]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M): protocol: write tcp 188.166.225.170:57034->77.90.31.15:4699: use of closed network connection; write tcp 188.166.225.170:57034->77.90.31.15:4699: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57034->77.90.31.15:4699: use of closed network connection Jul 19 05:07:48.143753064 (6h16m54.516332419s) [seg_pos={0 0}, upload=4987]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 19 05:07:48.143813039 (6h16m54.516392384s) [upload=4987]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 19 05:07:48.143816736 (6h16m54.516396091s) [upload=4987]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 19 05:07:48.532152020 (6h16m54.904731375s) [upload=4987]: Done uploading segments... Jul 19 05:07:48.532172109 (6h16m54.904751464s) [upload=4987]: Upload finished. err: metaclient: context deadline exceeded Duration: 22.433497338s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 11.956095934s FAIL (2025-07-19 05:08:02 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 4855s) [upload=4769]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 04:51:09.109709627 (6h0m15.482288982s) [upload=4769]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 04:51:09.556145753 (6h0m15.928725108s) [seg_pos={0 0}, upload=4769]: Begin upload segment... Jul 19 04:51:09.556194817 (6h0m15.928774172s) [seg_pos={0 0}, upload=4769]: Done begin upload segment. Jul 19 04:51:09.556223893 (6h0m15.928803248s) [noise=true , address=79.127.219.38:4023 , piece_id=PTPOZGLYV4MZTOFZNN3NCGYL7DFAST4PXKEUIVTMFL7HV4GZTCLQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556252197 (6h0m15.928831552s) [noise=true , address=45.91.95.4:4011 , piece_id=ZMBAS6KKHYVIRE6OBL2NFVCWMM4TKQ7J64BD3SYXIK54XCEDI2XA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556579383 (6h0m15.929158738s) [noise=true , address=45.91.116.6:4007 , piece_id=A2AFVTZBXSGBOE7XIM3B5HRN4STITFBXO6MZRJ365URNAAOZ3PSQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556854469 (6h0m15.929433824s) [noise=true , address=104.36.82.19:4423 , piece_id=PIJUHDUND36HQL67I33D7A3EULMJHS7N55MN4OFQJ4OWNQ42O47A, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557094107 (6h0m15.929673462s) [noise=true , address=45.91.116.9:4264 , piece_id=GHM5BKZKMT2QOHHBOS4FSPOWEWSFQRGRVVR4GSNDPHVIFRZHQL2A, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557336381 (6h0m15.929915736s) [noise=true , address=79.127.205.235:4660 , piece_id=UOUOGE2VP7PX7EMRZLPMCWFCYEZERM3DQMCQ6NG4J2K2DEIZR63Q, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557557885 (6h0m15.93013724s) [noise=true , address=79.127.205.246:4280 , piece_id=FLBOZI3UGTQXAQBLZ2F45Z3OEYL65F5PQBXRF2DQMZ7NGF7VDRPA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557762296 (6h0m15.930341651s) [noise=true , address=79.127.205.233:4622 , piece_id=6WWNYQTH5MGAJOEB7BXKZDJOMCAZXDW66J3OEANGDNJDNUMLZM7Q, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558014689 (6h0m15.930594044s) [noise=true , address=64.49.13.138:4482 , piece_id=MAW2M4US2OQNOS3WI67AA7NIBTJDE5T2JQTXKFPBLGVGXKVCFGTQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558215974 (6h0m15.930795319s) [noise=true , address=77.90.31.10:4546 , piece_id=R6LCA5WZOEBDURL5S3MEK22YPVYXX6EXSS4J4W57543HELYXOKZA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558409364 (6h0m15.930988719s) [noise=true , address=45.91.95.5:4019 , piece_id=YTSBCN2LPHEMRRAZIHJGVB7NBOTRXVEHD2TWUCTTLBNFFYEWWDKA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558623504 (6h0m15.931202859s) [noise=true , address=79.127.201.215:4357 , piece_id=75XWAG2VQWEYCWN2Q3WRIZSBI7LDOISZJPXQLQ2GIPNEN34VLSUQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558805162 (6h0m15.931384517s) [noise=true , address=45.91.116.14:4219 , piece_id=PDW6JPRHCVMHHOWIHT6JKV4CTSC3JY6NDKDVFZSO47OYNPZ7PGGA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558999373 (6h0m15.931578728s) [noise=true , address=77.90.31.11:4564 , piece_id=L2U7JFJRDQOFA2AYVAGTI4ZW4VABCXKMXWBIUWQO2QNNUUBJ74BA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559184768 (6h0m15.931764123s) [noise=true , address=77.90.31.16:4768 , piece_id=YNLSONFM6COZEVF6ZNQR2EA6V4LSQFZJONIXR4AFTNNMFGPV57JA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559368719 (6h0m15.931948074s) [noise=true , address=79.127.220.98:4432 , piece_id=2SUTFQ7FVDPAM2H4W3OLW6AVH3VJ33IJFE4L52SA3ET5LGBMOMPQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559549706 (6h0m15.932129061s) [noise=true , address=77.90.31.18:4813 , piece_id=VDOT7BB4O7XYJHW6BFTAYQZZYLZU5PAUXVJGSXJ7FDI4TH425COA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559727836 (6h0m15.932307191s) [noise=true , address=79.127.219.51:4211 , piece_id=HO5PLWBVU5YAYNYMN352NA73SM4WBJV4WZ7XLSQTCQ76UAL2TOJA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:10.663359087 (6h0m17.035938442s) [upload=4769]: Next returned nil segment Jul 19 04:51:10.663378664 (6h0m17.035958019s) [upload=4769]: Waiting for error group managing segments... Jul 19 04:51:10.962945479 (6h0m17.335524834s) [noise=true , address=79.127.205.233:4622 , piece_id=6WWNYQTH5MGAJOEB7BXKZDJOMCAZXDW66J3OEANGDNJDNUMLZM7Q, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:10.967689807 (6h0m17.340269152s) [noise=true , address=79.127.205.235:4660 , piece_id=UOUOGE2VP7PX7EMRZLPMCWFCYEZERM3DQMCQ6NG4J2K2DEIZR63Q, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:10.976968631 (6h0m17.349547986s) [noise=true , address=64.49.13.138:4482 , piece_id=MAW2M4US2OQNOS3WI67AA7NIBTJDE5T2JQTXKFPBLGVGXKVCFGTQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.048750700 (6h0m17.421330055s) [noise=true , address=79.127.205.246:4280 , piece_id=FLBOZI3UGTQXAQBLZ2F45Z3OEYL65F5PQBXRF2DQMZ7NGF7VDRPA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.221642403 (6h0m17.594221758s) [noise=true , address=79.127.220.98:4432 , piece_id=2SUTFQ7FVDPAM2H4W3OLW6AVH3VJ33IJFE4L52SA3ET5LGBMOMPQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.317070435 (6h0m17.68964979s) [noise=true , address=45.91.95.4:4011 , piece_id=ZMBAS6KKHYVIRE6OBL2NFVCWMM4TKQ7J64BD3SYXIK54XCEDI2XA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.324002660 (6h0m17.696582015s) [noise=true , address=45.91.95.5:4019 , piece_id=YTSBCN2LPHEMRRAZIHJGVB7NBOTRXVEHD2TWUCTTLBNFFYEWWDKA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.460976317 (6h0m17.833555672s) [noise=true , address=79.127.201.215:4357 , piece_id=75XWAG2VQWEYCWN2Q3WRIZSBI7LDOISZJPXQLQ2GIPNEN34VLSUQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.462584603 (6h0m17.835163958s) [noise=true , address=45.91.116.6:4007 , piece_id=A2AFVTZBXSGBOE7XIM3B5HRN4STITFBXO6MZRJ365URNAAOZ3PSQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.490392973 (6h0m17.862972328s) [noise=true , address=45.91.116.9:4264 , piece_id=GHM5BKZKMT2QOHHBOS4FSPOWEWSFQRGRVVR4GSNDPHVIFRZHQL2A, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.502040780 (6h0m17.874620135s) [noise=true , address=45.91.116.14:4219 , piece_id=PDW6JPRHCVMHHOWIHT6JKV4CTSC3JY6NDKDVFZSO47OYNPZ7PGGA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.600434096 (6h0m17.973013451s) [noise=true , address=79.127.219.51:4211 , piece_id=HO5PLWBVU5YAYNYMN352NA73SM4WBJV4WZ7XLSQTCQ76UAL2TOJA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.609008984 (6h0m17.981588339s) [noise=true , address=104.36.82.19:4423 , piece_id=PIJUHDUND36HQL67I33D7A3EULMJHS7N55MN4OFQJ4OWNQ42O47A, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.620825273 (6h0m17.993404628s) [noise=true , address=79.127.219.38:4023 , piece_id=PTPOZGLYV4MZTOFZNN3NCGYL7DFAST4PXKEUIVTMFL7HV4GZTCLQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.620838659 (6h0m17.993418014s) [seg_pos={0 0}, upload=4769]: Segment reached optimal threshold of 14 pieces. Jul 19 04:51:11.621067807 (6h0m17.993647163s) [noise=true , address=77.90.31.11:4564 , piece_id=L2U7JFJRDQOFA2AYVAGTI4ZW4VABCXKMXWBIUWQO2QNNUUBJ74BA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection; write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection Jul 19 04:51:11.621134234 (6h0m17.993713589s) [noise=true , address=77.90.31.10:4546 , piece_id=R6LCA5WZOEBDURL5S3MEK22YPVYXX6EXSS4J4W57543HELYXOKZA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK): protocol: read tcp 188.166.225.170:58476->77.90.31.10:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621174191 (6h0m17.993753546s) [noise=true , address=77.90.31.16:4768 , piece_id=YNLSONFM6COZEVF6ZNQR2EA6V4LSQFZJONIXR4AFTNNMFGPV57JA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UchewmtxBrtSZSpVAJFdZmcTsQVQxkjK9re6cgUGqDhyonQeF): protocol: read tcp 188.166.225.170:56326->77.90.31.16:4768: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621216842 (6h0m17.993796197s) [noise=true , address=77.90.31.18:4813 , piece_id=VDOT7BB4O7XYJHW6BFTAYQZZYLZU5PAUXVJGSXJ7FDI4TH425COA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128WthA1x9wp1dV1vkpGnoveZQpU9CSgDCZfw5GMQuVpCNSVAM5): protocol: read tcp 188.166.225.170:49704->77.90.31.18:4813: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621234917 (6h0m17.993814272s) [seg_pos={0 0}, upload=4769]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 19 04:51:11.621301444 (6h0m17.993880799s) [upload=4769]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 19 04:51:11.621305402 (6h0m17.993884758s) [upload=4769]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 19 04:51:11.923136970 (6h0m18.295716325s) [upload=4769]: Done uploading segments... Jul 19 04:51:11.923156627 (6h0m18.295735972s) [upload=4769]: Upload finished. err: <nil> Duration: 11.956122926s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)