Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 9.430533751s FAIL (2025-07-19 05:12:44 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 4855s) [upload=4769]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 04:51:09.109709627 (6h0m15.482288982s) [upload=4769]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 04:51:09.556145753 (6h0m15.928725108s) [seg_pos={0 0}, upload=4769]: Begin upload segment... Jul 19 04:51:09.556194817 (6h0m15.928774172s) [seg_pos={0 0}, upload=4769]: Done begin upload segment. Jul 19 04:51:09.556223893 (6h0m15.928803248s) [noise=true , address=79.127.219.38:4023 , piece_id=PTPOZGLYV4MZTOFZNN3NCGYL7DFAST4PXKEUIVTMFL7HV4GZTCLQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556252197 (6h0m15.928831552s) [noise=true , address=45.91.95.4:4011 , piece_id=ZMBAS6KKHYVIRE6OBL2NFVCWMM4TKQ7J64BD3SYXIK54XCEDI2XA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556579383 (6h0m15.929158738s) [noise=true , address=45.91.116.6:4007 , piece_id=A2AFVTZBXSGBOE7XIM3B5HRN4STITFBXO6MZRJ365URNAAOZ3PSQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556854469 (6h0m15.929433824s) [noise=true , address=104.36.82.19:4423 , piece_id=PIJUHDUND36HQL67I33D7A3EULMJHS7N55MN4OFQJ4OWNQ42O47A, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557094107 (6h0m15.929673462s) [noise=true , address=45.91.116.9:4264 , piece_id=GHM5BKZKMT2QOHHBOS4FSPOWEWSFQRGRVVR4GSNDPHVIFRZHQL2A, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557336381 (6h0m15.929915736s) [noise=true , address=79.127.205.235:4660 , piece_id=UOUOGE2VP7PX7EMRZLPMCWFCYEZERM3DQMCQ6NG4J2K2DEIZR63Q, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557557885 (6h0m15.93013724s) [noise=true , address=79.127.205.246:4280 , piece_id=FLBOZI3UGTQXAQBLZ2F45Z3OEYL65F5PQBXRF2DQMZ7NGF7VDRPA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557762296 (6h0m15.930341651s) [noise=true , address=79.127.205.233:4622 , piece_id=6WWNYQTH5MGAJOEB7BXKZDJOMCAZXDW66J3OEANGDNJDNUMLZM7Q, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558014689 (6h0m15.930594044s) [noise=true , address=64.49.13.138:4482 , piece_id=MAW2M4US2OQNOS3WI67AA7NIBTJDE5T2JQTXKFPBLGVGXKVCFGTQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558215974 (6h0m15.930795319s) [noise=true , address=77.90.31.10:4546 , piece_id=R6LCA5WZOEBDURL5S3MEK22YPVYXX6EXSS4J4W57543HELYXOKZA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558409364 (6h0m15.930988719s) [noise=true , address=45.91.95.5:4019 , piece_id=YTSBCN2LPHEMRRAZIHJGVB7NBOTRXVEHD2TWUCTTLBNFFYEWWDKA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558623504 (6h0m15.931202859s) [noise=true , address=79.127.201.215:4357 , piece_id=75XWAG2VQWEYCWN2Q3WRIZSBI7LDOISZJPXQLQ2GIPNEN34VLSUQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558805162 (6h0m15.931384517s) [noise=true , address=45.91.116.14:4219 , piece_id=PDW6JPRHCVMHHOWIHT6JKV4CTSC3JY6NDKDVFZSO47OYNPZ7PGGA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558999373 (6h0m15.931578728s) [noise=true , address=77.90.31.11:4564 , piece_id=L2U7JFJRDQOFA2AYVAGTI4ZW4VABCXKMXWBIUWQO2QNNUUBJ74BA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559184768 (6h0m15.931764123s) [noise=true , address=77.90.31.16:4768 , piece_id=YNLSONFM6COZEVF6ZNQR2EA6V4LSQFZJONIXR4AFTNNMFGPV57JA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559368719 (6h0m15.931948074s) [noise=true , address=79.127.220.98:4432 , piece_id=2SUTFQ7FVDPAM2H4W3OLW6AVH3VJ33IJFE4L52SA3ET5LGBMOMPQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559549706 (6h0m15.932129061s) [noise=true , address=77.90.31.18:4813 , piece_id=VDOT7BB4O7XYJHW6BFTAYQZZYLZU5PAUXVJGSXJ7FDI4TH425COA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559727836 (6h0m15.932307191s) [noise=true , address=79.127.219.51:4211 , piece_id=HO5PLWBVU5YAYNYMN352NA73SM4WBJV4WZ7XLSQTCQ76UAL2TOJA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:10.663359087 (6h0m17.035938442s) [upload=4769]: Next returned nil segment Jul 19 04:51:10.663378664 (6h0m17.035958019s) [upload=4769]: Waiting for error group managing segments... Jul 19 04:51:10.962945479 (6h0m17.335524834s) [noise=true , address=79.127.205.233:4622 , piece_id=6WWNYQTH5MGAJOEB7BXKZDJOMCAZXDW66J3OEANGDNJDNUMLZM7Q, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:10.967689807 (6h0m17.340269152s) [noise=true , address=79.127.205.235:4660 , piece_id=UOUOGE2VP7PX7EMRZLPMCWFCYEZERM3DQMCQ6NG4J2K2DEIZR63Q, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:10.976968631 (6h0m17.349547986s) [noise=true , address=64.49.13.138:4482 , piece_id=MAW2M4US2OQNOS3WI67AA7NIBTJDE5T2JQTXKFPBLGVGXKVCFGTQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.048750700 (6h0m17.421330055s) [noise=true , address=79.127.205.246:4280 , piece_id=FLBOZI3UGTQXAQBLZ2F45Z3OEYL65F5PQBXRF2DQMZ7NGF7VDRPA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.221642403 (6h0m17.594221758s) [noise=true , address=79.127.220.98:4432 , piece_id=2SUTFQ7FVDPAM2H4W3OLW6AVH3VJ33IJFE4L52SA3ET5LGBMOMPQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.317070435 (6h0m17.68964979s) [noise=true , address=45.91.95.4:4011 , piece_id=ZMBAS6KKHYVIRE6OBL2NFVCWMM4TKQ7J64BD3SYXIK54XCEDI2XA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.324002660 (6h0m17.696582015s) [noise=true , address=45.91.95.5:4019 , piece_id=YTSBCN2LPHEMRRAZIHJGVB7NBOTRXVEHD2TWUCTTLBNFFYEWWDKA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.460976317 (6h0m17.833555672s) [noise=true , address=79.127.201.215:4357 , piece_id=75XWAG2VQWEYCWN2Q3WRIZSBI7LDOISZJPXQLQ2GIPNEN34VLSUQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.462584603 (6h0m17.835163958s) [noise=true , address=45.91.116.6:4007 , piece_id=A2AFVTZBXSGBOE7XIM3B5HRN4STITFBXO6MZRJ365URNAAOZ3PSQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.490392973 (6h0m17.862972328s) [noise=true , address=45.91.116.9:4264 , piece_id=GHM5BKZKMT2QOHHBOS4FSPOWEWSFQRGRVVR4GSNDPHVIFRZHQL2A, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.502040780 (6h0m17.874620135s) [noise=true , address=45.91.116.14:4219 , piece_id=PDW6JPRHCVMHHOWIHT6JKV4CTSC3JY6NDKDVFZSO47OYNPZ7PGGA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.600434096 (6h0m17.973013451s) [noise=true , address=79.127.219.51:4211 , piece_id=HO5PLWBVU5YAYNYMN352NA73SM4WBJV4WZ7XLSQTCQ76UAL2TOJA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.609008984 (6h0m17.981588339s) [noise=true , address=104.36.82.19:4423 , piece_id=PIJUHDUND36HQL67I33D7A3EULMJHS7N55MN4OFQJ4OWNQ42O47A, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.620825273 (6h0m17.993404628s) [noise=true , address=79.127.219.38:4023 , piece_id=PTPOZGLYV4MZTOFZNN3NCGYL7DFAST4PXKEUIVTMFL7HV4GZTCLQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.620838659 (6h0m17.993418014s) [seg_pos={0 0}, upload=4769]: Segment reached optimal threshold of 14 pieces. Jul 19 04:51:11.621067807 (6h0m17.993647163s) [noise=true , address=77.90.31.11:4564 , piece_id=L2U7JFJRDQOFA2AYVAGTI4ZW4VABCXKMXWBIUWQO2QNNUUBJ74BA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection; write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection Jul 19 04:51:11.621134234 (6h0m17.993713589s) [noise=true , address=77.90.31.10:4546 , piece_id=R6LCA5WZOEBDURL5S3MEK22YPVYXX6EXSS4J4W57543HELYXOKZA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK): protocol: read tcp 188.166.225.170:58476->77.90.31.10:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621174191 (6h0m17.993753546s) [noise=true , address=77.90.31.16:4768 , piece_id=YNLSONFM6COZEVF6ZNQR2EA6V4LSQFZJONIXR4AFTNNMFGPV57JA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UchewmtxBrtSZSpVAJFdZmcTsQVQxkjK9re6cgUGqDhyonQeF): protocol: read tcp 188.166.225.170:56326->77.90.31.16:4768: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621216842 (6h0m17.993796197s) [noise=true , address=77.90.31.18:4813 , piece_id=VDOT7BB4O7XYJHW6BFTAYQZZYLZU5PAUXVJGSXJ7FDI4TH425COA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128WthA1x9wp1dV1vkpGnoveZQpU9CSgDCZfw5GMQuVpCNSVAM5): protocol: read tcp 188.166.225.170:49704->77.90.31.18:4813: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621234917 (6h0m17.993814272s) [seg_pos={0 0}, upload=4769]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 19 04:51:11.621301444 (6h0m17.993880799s) [upload=4769]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 19 04:51:11.621305402 (6h0m17.993884758s) [upload=4769]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 19 04:51:11.923136970 (6h0m18.295716325s) [upload=4769]: Done uploading segments... Jul 19 04:51:11.923156627 (6h0m18.295735972s) [upload=4769]: Upload finished. err: <nil> Duration: 9.430565483s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 8.368369355s FAIL (2025-07-19 05:12:20 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 4855s) [upload=4769]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 04:51:09.109709627 (6h0m15.482288982s) [upload=4769]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 04:51:09.556145753 (6h0m15.928725108s) [seg_pos={0 0}, upload=4769]: Begin upload segment... Jul 19 04:51:09.556194817 (6h0m15.928774172s) [seg_pos={0 0}, upload=4769]: Done begin upload segment. Jul 19 04:51:09.556223893 (6h0m15.928803248s) [noise=true , address=79.127.219.38:4023 , piece_id=PTPOZGLYV4MZTOFZNN3NCGYL7DFAST4PXKEUIVTMFL7HV4GZTCLQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556252197 (6h0m15.928831552s) [noise=true , address=45.91.95.4:4011 , piece_id=ZMBAS6KKHYVIRE6OBL2NFVCWMM4TKQ7J64BD3SYXIK54XCEDI2XA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556579383 (6h0m15.929158738s) [noise=true , address=45.91.116.6:4007 , piece_id=A2AFVTZBXSGBOE7XIM3B5HRN4STITFBXO6MZRJ365URNAAOZ3PSQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556854469 (6h0m15.929433824s) [noise=true , address=104.36.82.19:4423 , piece_id=PIJUHDUND36HQL67I33D7A3EULMJHS7N55MN4OFQJ4OWNQ42O47A, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557094107 (6h0m15.929673462s) [noise=true , address=45.91.116.9:4264 , piece_id=GHM5BKZKMT2QOHHBOS4FSPOWEWSFQRGRVVR4GSNDPHVIFRZHQL2A, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557336381 (6h0m15.929915736s) [noise=true , address=79.127.205.235:4660 , piece_id=UOUOGE2VP7PX7EMRZLPMCWFCYEZERM3DQMCQ6NG4J2K2DEIZR63Q, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557557885 (6h0m15.93013724s) [noise=true , address=79.127.205.246:4280 , piece_id=FLBOZI3UGTQXAQBLZ2F45Z3OEYL65F5PQBXRF2DQMZ7NGF7VDRPA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557762296 (6h0m15.930341651s) [noise=true , address=79.127.205.233:4622 , piece_id=6WWNYQTH5MGAJOEB7BXKZDJOMCAZXDW66J3OEANGDNJDNUMLZM7Q, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558014689 (6h0m15.930594044s) [noise=true , address=64.49.13.138:4482 , piece_id=MAW2M4US2OQNOS3WI67AA7NIBTJDE5T2JQTXKFPBLGVGXKVCFGTQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558215974 (6h0m15.930795319s) [noise=true , address=77.90.31.10:4546 , piece_id=R6LCA5WZOEBDURL5S3MEK22YPVYXX6EXSS4J4W57543HELYXOKZA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558409364 (6h0m15.930988719s) [noise=true , address=45.91.95.5:4019 , piece_id=YTSBCN2LPHEMRRAZIHJGVB7NBOTRXVEHD2TWUCTTLBNFFYEWWDKA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558623504 (6h0m15.931202859s) [noise=true , address=79.127.201.215:4357 , piece_id=75XWAG2VQWEYCWN2Q3WRIZSBI7LDOISZJPXQLQ2GIPNEN34VLSUQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558805162 (6h0m15.931384517s) [noise=true , address=45.91.116.14:4219 , piece_id=PDW6JPRHCVMHHOWIHT6JKV4CTSC3JY6NDKDVFZSO47OYNPZ7PGGA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558999373 (6h0m15.931578728s) [noise=true , address=77.90.31.11:4564 , piece_id=L2U7JFJRDQOFA2AYVAGTI4ZW4VABCXKMXWBIUWQO2QNNUUBJ74BA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559184768 (6h0m15.931764123s) [noise=true , address=77.90.31.16:4768 , piece_id=YNLSONFM6COZEVF6ZNQR2EA6V4LSQFZJONIXR4AFTNNMFGPV57JA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559368719 (6h0m15.931948074s) [noise=true , address=79.127.220.98:4432 , piece_id=2SUTFQ7FVDPAM2H4W3OLW6AVH3VJ33IJFE4L52SA3ET5LGBMOMPQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559549706 (6h0m15.932129061s) [noise=true , address=77.90.31.18:4813 , piece_id=VDOT7BB4O7XYJHW6BFTAYQZZYLZU5PAUXVJGSXJ7FDI4TH425COA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559727836 (6h0m15.932307191s) [noise=true , address=79.127.219.51:4211 , piece_id=HO5PLWBVU5YAYNYMN352NA73SM4WBJV4WZ7XLSQTCQ76UAL2TOJA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:10.663359087 (6h0m17.035938442s) [upload=4769]: Next returned nil segment Jul 19 04:51:10.663378664 (6h0m17.035958019s) [upload=4769]: Waiting for error group managing segments... Jul 19 04:51:10.962945479 (6h0m17.335524834s) [noise=true , address=79.127.205.233:4622 , piece_id=6WWNYQTH5MGAJOEB7BXKZDJOMCAZXDW66J3OEANGDNJDNUMLZM7Q, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:10.967689807 (6h0m17.340269152s) [noise=true , address=79.127.205.235:4660 , piece_id=UOUOGE2VP7PX7EMRZLPMCWFCYEZERM3DQMCQ6NG4J2K2DEIZR63Q, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:10.976968631 (6h0m17.349547986s) [noise=true , address=64.49.13.138:4482 , piece_id=MAW2M4US2OQNOS3WI67AA7NIBTJDE5T2JQTXKFPBLGVGXKVCFGTQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.048750700 (6h0m17.421330055s) [noise=true , address=79.127.205.246:4280 , piece_id=FLBOZI3UGTQXAQBLZ2F45Z3OEYL65F5PQBXRF2DQMZ7NGF7VDRPA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.221642403 (6h0m17.594221758s) [noise=true , address=79.127.220.98:4432 , piece_id=2SUTFQ7FVDPAM2H4W3OLW6AVH3VJ33IJFE4L52SA3ET5LGBMOMPQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.317070435 (6h0m17.68964979s) [noise=true , address=45.91.95.4:4011 , piece_id=ZMBAS6KKHYVIRE6OBL2NFVCWMM4TKQ7J64BD3SYXIK54XCEDI2XA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.324002660 (6h0m17.696582015s) [noise=true , address=45.91.95.5:4019 , piece_id=YTSBCN2LPHEMRRAZIHJGVB7NBOTRXVEHD2TWUCTTLBNFFYEWWDKA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.460976317 (6h0m17.833555672s) [noise=true , address=79.127.201.215:4357 , piece_id=75XWAG2VQWEYCWN2Q3WRIZSBI7LDOISZJPXQLQ2GIPNEN34VLSUQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.462584603 (6h0m17.835163958s) [noise=true , address=45.91.116.6:4007 , piece_id=A2AFVTZBXSGBOE7XIM3B5HRN4STITFBXO6MZRJ365URNAAOZ3PSQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.490392973 (6h0m17.862972328s) [noise=true , address=45.91.116.9:4264 , piece_id=GHM5BKZKMT2QOHHBOS4FSPOWEWSFQRGRVVR4GSNDPHVIFRZHQL2A, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.502040780 (6h0m17.874620135s) [noise=true , address=45.91.116.14:4219 , piece_id=PDW6JPRHCVMHHOWIHT6JKV4CTSC3JY6NDKDVFZSO47OYNPZ7PGGA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.600434096 (6h0m17.973013451s) [noise=true , address=79.127.219.51:4211 , piece_id=HO5PLWBVU5YAYNYMN352NA73SM4WBJV4WZ7XLSQTCQ76UAL2TOJA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.609008984 (6h0m17.981588339s) [noise=true , address=104.36.82.19:4423 , piece_id=PIJUHDUND36HQL67I33D7A3EULMJHS7N55MN4OFQJ4OWNQ42O47A, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.620825273 (6h0m17.993404628s) [noise=true , address=79.127.219.38:4023 , piece_id=PTPOZGLYV4MZTOFZNN3NCGYL7DFAST4PXKEUIVTMFL7HV4GZTCLQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.620838659 (6h0m17.993418014s) [seg_pos={0 0}, upload=4769]: Segment reached optimal threshold of 14 pieces. Jul 19 04:51:11.621067807 (6h0m17.993647163s) [noise=true , address=77.90.31.11:4564 , piece_id=L2U7JFJRDQOFA2AYVAGTI4ZW4VABCXKMXWBIUWQO2QNNUUBJ74BA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection; write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection Jul 19 04:51:11.621134234 (6h0m17.993713589s) [noise=true , address=77.90.31.10:4546 , piece_id=R6LCA5WZOEBDURL5S3MEK22YPVYXX6EXSS4J4W57543HELYXOKZA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK): protocol: read tcp 188.166.225.170:58476->77.90.31.10:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621174191 (6h0m17.993753546s) [noise=true , address=77.90.31.16:4768 , piece_id=YNLSONFM6COZEVF6ZNQR2EA6V4LSQFZJONIXR4AFTNNMFGPV57JA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UchewmtxBrtSZSpVAJFdZmcTsQVQxkjK9re6cgUGqDhyonQeF): protocol: read tcp 188.166.225.170:56326->77.90.31.16:4768: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621216842 (6h0m17.993796197s) [noise=true , address=77.90.31.18:4813 , piece_id=VDOT7BB4O7XYJHW6BFTAYQZZYLZU5PAUXVJGSXJ7FDI4TH425COA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128WthA1x9wp1dV1vkpGnoveZQpU9CSgDCZfw5GMQuVpCNSVAM5): protocol: read tcp 188.166.225.170:49704->77.90.31.18:4813: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621234917 (6h0m17.993814272s) [seg_pos={0 0}, upload=4769]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 19 04:51:11.621301444 (6h0m17.993880799s) [upload=4769]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 19 04:51:11.621305402 (6h0m17.993884758s) [upload=4769]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 19 04:51:11.923136970 (6h0m18.295716325s) [upload=4769]: Done uploading segments... Jul 19 04:51:11.923156627 (6h0m18.295735972s) [upload=4769]: Upload finished. err: <nil> Duration: 8.368400405s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [........................XX...X...X....X..X....XXXXX.X.X.....] duration: 7.879249541s FAIL (2025-07-19 05:12:14 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== : <nil> Jul 19 05:08:29.800544823 (6h17m36.173124169s) [upload=4997]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 19 05:08:29.800548410 (6h17m36.173127765s) [upload=4997]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 19 05:08:30.148027600 (6h17m36.520606955s) [upload=4997]: Done uploading segments... Jul 19 05:08:30.148148401 (6h17m36.520727746s) [upload=4997]: Upload finished. err: <nil> Jul 19 05:08:48.807983386 (6h17m55.180562741s) [upload=5005]: Starting upload Jul 19 05:08:48.808044453 (6h17m55.180623808s) [upload=5005]: Done starting upload Jul 19 05:08:48.808175243 (6h17m55.180754598s) [upload=5005]: Uploading segments... Jul 19 05:08:48.808334198 (6h17m55.180913553s) [upload=5005]: Got next segment. Inline: false Jul 19 05:08:48.808392509 (6h17m55.180971874s) [upload=5005]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 05:08:48.808399823 (6h17m55.180979188s) [upload=5005]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 05:08:49.075515233 (6h17m55.448094578s) [seg_pos={0 0}, upload=5005]: Begin upload segment... Jul 19 05:08:49.075564547 (6h17m55.448143902s) [seg_pos={0 0}, upload=5005]: Done begin upload segment. Jul 19 05:08:49.075602881 (6h17m55.448182236s) [noise=true , address=77.90.31.11:4555 , piece_id=QGRXVFSKXDCLKSSSXEK4QYZEETWQGFZMQQBSTXNMAHY45RA6MAVQ, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.075612279 (6h17m55.448191634s) [noise=true , address=104.36.82.13:4186 , piece_id=PPK6XEQYJ3XV27XUJYKUL2MYAR5KUYKF5AHJS5EBXOK7PZWWM66Q, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.075964213 (6h17m55.448543568s) [noise=true , address=77.90.31.12:4592 , piece_id=HS5YQKPTPUVHFZH3VZ7X2LAO2SVS6JZOTHTLJ6T47G3CLTZTR7UQ, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.076122025 (6h17m55.44870137s) [noise=true , address=104.36.82.12:4150 , piece_id=GQ4GQQZHK3Y4KVVB2IUU4LEEBR4OJ2YSS5Z5DF7WHHDCYYSSGRYA, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.076272903 (6h17m55.448852258s) [noise=true , address=45.91.116.13:4154 , piece_id=OJO7VBXGBLV2SG2AQ6BPBP57SL75WV4UZSWC3O5HXD7ROBICSYGA, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.076533412 (6h17m55.449112767s) [noise=true , address=104.36.82.2:4732 , piece_id=DBPJH5FSNFRPIBLP7HWWJFVVXYC47J7UGPCXFL7L3BD6BGFSJDFA, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.076794962 (6h17m55.449374317s) [noise=true , address=45.91.116.9:4263 , piece_id=4L3UWFKV3AC7F4RPS7GJFXOO6DJNCPMFB5NWGJH6HXAFBGLYAMYA, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.077044971 (6h17m55.449624326s) [noise=true , address=104.36.82.16:4293 , piece_id=2NA7I27WVAWAUWDCZVYBJLNXGWIDNJMKZU7YFJZSXCQMCHKLCGXA, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.077296953 (6h17m55.449876308s) [noise=true , address=77.90.31.15:4698 , piece_id=HNTVTQMVXDUMK4CMHKVUZ7DHM3MZL56TGXFR4FFFV2V3DCTLAVZQ, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.077501123 (6h17m55.450080468s) [noise=true , address=104.36.82.9:4007 , piece_id=R2IPGORJ3Z4FFCC52P5WMXYIA6O5V7DDZHHL4X64XV6J36SIWSBQ, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.077695656 (6h17m55.450275001s) [noise=true , address=45.91.116.8:4048 , piece_id=QKXGR6YZSYTQIUXV54DRYQNWY352UFS2Z2VXWFJTD7YJYJZ24NCQ, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.077994558 (6h17m55.450573913s) [noise=true , address=104.36.82.14:4222 , piece_id=4F6ZDS2KMQNALT2YV6D3TAZLIBFUFBM5NNCBGLSE6L6PR5QMGPVA, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.078213187 (6h17m55.450792542s) [noise=true , address=45.91.116.4:4939 , piece_id=J2BIIQITH7R54K4ENE6AKU7KCOUAGYBOTKX6VF6FY7FNQDMNNFCA, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.078412077 (6h17m55.450991432s) [noise=true , address=77.90.31.13:4626 , piece_id=2QAZAYJWKJXTPY475RLOSPV3ZZTMB6OZLNYAKD7FCFV7T4WPC5ZQ, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.079649705 (6h17m55.45222906s) [noise=true , address=45.91.116.9:4262 , piece_id=EXWTIYPESLJQK3CDK732WC63OZ7OJOQAUCBPGO3ZU54UBFUJ3TSA, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.080903344 (6h17m55.453482699s) [noise=true , address=77.90.31.13:4628 , piece_id=OAGGFQO2CORAZQV6ND3EERSICEOSHKK73Y3PUWLT45ZUWEWDHOFQ, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:50.857952826 (6h17m57.230532181s) [upload=5005]: Next returned nil segment Jul 19 05:08:50.857969477 (6h17m57.230548832s) [upload=5005]: Waiting for error group managing segments... Jul 19 05:08:51.139405061 (6h17m57.511984416s) [noise=true , address=104.36.82.13:4186 , piece_id=PPK6XEQYJ3XV27XUJYKUL2MYAR5KUYKF5AHJS5EBXOK7PZWWM66Q, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:51.143987831 (6h17m57.516567176s) [noise=true , address=45.91.116.8:4048 , piece_id=QKXGR6YZSYTQIUXV54DRYQNWY352UFS2Z2VXWFJTD7YJYJZ24NCQ, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:51.148258443 (6h17m57.520837798s) [noise=true , address=104.36.82.16:4293 , piece_id=2NA7I27WVAWAUWDCZVYBJLNXGWIDNJMKZU7YFJZSXCQMCHKLCGXA, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:51.156796512 (6h17m57.529375867s) [noise=true , address=45.91.116.13:4154 , piece_id=OJO7VBXGBLV2SG2AQ6BPBP57SL75WV4UZSWC3O5HXD7ROBICSYGA, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:51.193908742 (6h17m57.566488098s) [noise=true , address=104.36.82.14:4222 , piece_id=4F6ZDS2KMQNALT2YV6D3TAZLIBFUFBM5NNCBGLSE6L6PR5QMGPVA, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:51.217498736 (6h17m57.590078091s) [noise=true , address=45.91.116.4:4939 , piece_id=J2BIIQITH7R54K4ENE6AKU7KCOUAGYBOTKX6VF6FY7FNQDMNNFCA, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:51.218585386 (6h17m57.591164741s) [noise=true , address=45.91.116.9:4263 , piece_id=4L3UWFKV3AC7F4RPS7GJFXOO6DJNCPMFB5NWGJH6HXAFBGLYAMYA, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:51.290646027 (6h17m57.663225392s) [noise=true , address=104.36.82.2:4732 , piece_id=DBPJH5FSNFRPIBLP7HWWJFVVXYC47J7UGPCXFL7L3BD6BGFSJDFA, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:51.309329961 (6h17m57.681909316s) [noise=true , address=104.36.82.12:4150 , piece_id=GQ4GQQZHK3Y4KVVB2IUU4LEEBR4OJ2YSS5Z5DF7WHHDCYYSSGRYA, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:51.317895332 (6h17m57.690474687s) [noise=true , address=104.36.82.9:4007 , piece_id=R2IPGORJ3Z4FFCC52P5WMXYIA6O5V7DDZHHL4X64XV6J36SIWSBQ, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:52.525479047 (6h17m58.898058402s) [noise=true , address=45.91.116.9:4262 , piece_id=EXWTIYPESLJQK3CDK732WC63OZ7OJOQAUCBPGO3ZU54UBFUJ3TSA, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:09:18.157484198 (6h18m24.530063553s) [noise=true , address=77.90.31.13:4628 , piece_id=OAGGFQO2CORAZQV6ND3EERSICEOSHKK73Y3PUWLT45ZUWEWDHOFQ, seg_pos={0 0}, upload=5005]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: expected piece hash; context deadline exceeded; EOF Jul 19 05:09:18.157617684 (6h18m24.530197039s) [noise=true , address=77.90.31.15:4698 , piece_id=HNTVTQMVXDUMK4CMHKVUZ7DHM3MZL56TGXFR4FFFV2V3DCTLAVZQ, seg_pos={0 0}, upload=5005]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF Jul 19 05:09:18.157744796 (6h18m24.530324151s) [noise=true , address=77.90.31.11:4555 , piece_id=QGRXVFSKXDCLKSSSXEK4QYZEETWQGFZMQQBSTXNMAHY45RA6MAVQ, seg_pos={0 0}, upload=5005]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: write tcp 188.166.225.170:38288->77.90.31.11:4555: use of closed network connection; write tcp 188.166.225.170:38288->77.90.31.11:4555: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38288->77.90.31.11:4555: use of closed network connection Jul 19 05:09:18.157785745 (6h18m24.5303651s) [noise=true , address=77.90.31.13:4626 , piece_id=2QAZAYJWKJXTPY475RLOSPV3ZZTMB6OZLNYAKD7FCFV7T4WPC5ZQ, seg_pos={0 0}, upload=5005]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:58846->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:58846->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58846->77.90.31.13:4626: use of closed network connection Jul 19 05:09:18.157859036 (6h18m24.530438391s) [noise=true , address=77.90.31.12:4592 , piece_id=HS5YQKPTPUVHFZH3VZ7X2LAO2SVS6JZOTHTLJ6T47G3CLTZTR7UQ, seg_pos={0 0}, upload=5005]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:38220->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:38220->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38220->77.90.31.12:4592: use of closed network connection Jul 19 05:09:18.157898821 (6h18m24.530478176s) [seg_pos={0 0}, upload=5005]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 19 05:09:18.389974737 (6h18m24.762554092s) [upload=5005]: Done uploading segments... Jul 19 05:09:18.389994916 (6h18m24.762574271s) [upload=5005]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 7.87927533s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.........................XX...X...X....X..X....XXXXX.X.X....] duration: 5.780683075s FAIL (2025-07-19 05:11:42 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== : <nil> Jul 19 05:08:29.800544823 (6h17m36.173124169s) [upload=4997]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 19 05:08:29.800548410 (6h17m36.173127765s) [upload=4997]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 19 05:08:30.148027600 (6h17m36.520606955s) [upload=4997]: Done uploading segments... Jul 19 05:08:30.148148401 (6h17m36.520727746s) [upload=4997]: Upload finished. err: <nil> Jul 19 05:08:48.807983386 (6h17m55.180562741s) [upload=5005]: Starting upload Jul 19 05:08:48.808044453 (6h17m55.180623808s) [upload=5005]: Done starting upload Jul 19 05:08:48.808175243 (6h17m55.180754598s) [upload=5005]: Uploading segments... Jul 19 05:08:48.808334198 (6h17m55.180913553s) [upload=5005]: Got next segment. Inline: false Jul 19 05:08:48.808392509 (6h17m55.180971874s) [upload=5005]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 05:08:48.808399823 (6h17m55.180979188s) [upload=5005]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 05:08:49.075515233 (6h17m55.448094578s) [seg_pos={0 0}, upload=5005]: Begin upload segment... Jul 19 05:08:49.075564547 (6h17m55.448143902s) [seg_pos={0 0}, upload=5005]: Done begin upload segment. Jul 19 05:08:49.075602881 (6h17m55.448182236s) [noise=true , address=77.90.31.11:4555 , piece_id=QGRXVFSKXDCLKSSSXEK4QYZEETWQGFZMQQBSTXNMAHY45RA6MAVQ, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.075612279 (6h17m55.448191634s) [noise=true , address=104.36.82.13:4186 , piece_id=PPK6XEQYJ3XV27XUJYKUL2MYAR5KUYKF5AHJS5EBXOK7PZWWM66Q, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.075964213 (6h17m55.448543568s) [noise=true , address=77.90.31.12:4592 , piece_id=HS5YQKPTPUVHFZH3VZ7X2LAO2SVS6JZOTHTLJ6T47G3CLTZTR7UQ, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.076122025 (6h17m55.44870137s) [noise=true , address=104.36.82.12:4150 , piece_id=GQ4GQQZHK3Y4KVVB2IUU4LEEBR4OJ2YSS5Z5DF7WHHDCYYSSGRYA, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.076272903 (6h17m55.448852258s) [noise=true , address=45.91.116.13:4154 , piece_id=OJO7VBXGBLV2SG2AQ6BPBP57SL75WV4UZSWC3O5HXD7ROBICSYGA, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.076533412 (6h17m55.449112767s) [noise=true , address=104.36.82.2:4732 , piece_id=DBPJH5FSNFRPIBLP7HWWJFVVXYC47J7UGPCXFL7L3BD6BGFSJDFA, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.076794962 (6h17m55.449374317s) [noise=true , address=45.91.116.9:4263 , piece_id=4L3UWFKV3AC7F4RPS7GJFXOO6DJNCPMFB5NWGJH6HXAFBGLYAMYA, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.077044971 (6h17m55.449624326s) [noise=true , address=104.36.82.16:4293 , piece_id=2NA7I27WVAWAUWDCZVYBJLNXGWIDNJMKZU7YFJZSXCQMCHKLCGXA, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.077296953 (6h17m55.449876308s) [noise=true , address=77.90.31.15:4698 , piece_id=HNTVTQMVXDUMK4CMHKVUZ7DHM3MZL56TGXFR4FFFV2V3DCTLAVZQ, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.077501123 (6h17m55.450080468s) [noise=true , address=104.36.82.9:4007 , piece_id=R2IPGORJ3Z4FFCC52P5WMXYIA6O5V7DDZHHL4X64XV6J36SIWSBQ, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.077695656 (6h17m55.450275001s) [noise=true , address=45.91.116.8:4048 , piece_id=QKXGR6YZSYTQIUXV54DRYQNWY352UFS2Z2VXWFJTD7YJYJZ24NCQ, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.077994558 (6h17m55.450573913s) [noise=true , address=104.36.82.14:4222 , piece_id=4F6ZDS2KMQNALT2YV6D3TAZLIBFUFBM5NNCBGLSE6L6PR5QMGPVA, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.078213187 (6h17m55.450792542s) [noise=true , address=45.91.116.4:4939 , piece_id=J2BIIQITH7R54K4ENE6AKU7KCOUAGYBOTKX6VF6FY7FNQDMNNFCA, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.078412077 (6h17m55.450991432s) [noise=true , address=77.90.31.13:4626 , piece_id=2QAZAYJWKJXTPY475RLOSPV3ZZTMB6OZLNYAKD7FCFV7T4WPC5ZQ, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.079649705 (6h17m55.45222906s) [noise=true , address=45.91.116.9:4262 , piece_id=EXWTIYPESLJQK3CDK732WC63OZ7OJOQAUCBPGO3ZU54UBFUJ3TSA, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.080903344 (6h17m55.453482699s) [noise=true , address=77.90.31.13:4628 , piece_id=OAGGFQO2CORAZQV6ND3EERSICEOSHKK73Y3PUWLT45ZUWEWDHOFQ, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:50.857952826 (6h17m57.230532181s) [upload=5005]: Next returned nil segment Jul 19 05:08:50.857969477 (6h17m57.230548832s) [upload=5005]: Waiting for error group managing segments... Jul 19 05:08:51.139405061 (6h17m57.511984416s) [noise=true , address=104.36.82.13:4186 , piece_id=PPK6XEQYJ3XV27XUJYKUL2MYAR5KUYKF5AHJS5EBXOK7PZWWM66Q, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:51.143987831 (6h17m57.516567176s) [noise=true , address=45.91.116.8:4048 , piece_id=QKXGR6YZSYTQIUXV54DRYQNWY352UFS2Z2VXWFJTD7YJYJZ24NCQ, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:51.148258443 (6h17m57.520837798s) [noise=true , address=104.36.82.16:4293 , piece_id=2NA7I27WVAWAUWDCZVYBJLNXGWIDNJMKZU7YFJZSXCQMCHKLCGXA, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:51.156796512 (6h17m57.529375867s) [noise=true , address=45.91.116.13:4154 , piece_id=OJO7VBXGBLV2SG2AQ6BPBP57SL75WV4UZSWC3O5HXD7ROBICSYGA, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:51.193908742 (6h17m57.566488098s) [noise=true , address=104.36.82.14:4222 , piece_id=4F6ZDS2KMQNALT2YV6D3TAZLIBFUFBM5NNCBGLSE6L6PR5QMGPVA, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:51.217498736 (6h17m57.590078091s) [noise=true , address=45.91.116.4:4939 , piece_id=J2BIIQITH7R54K4ENE6AKU7KCOUAGYBOTKX6VF6FY7FNQDMNNFCA, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:51.218585386 (6h17m57.591164741s) [noise=true , address=45.91.116.9:4263 , piece_id=4L3UWFKV3AC7F4RPS7GJFXOO6DJNCPMFB5NWGJH6HXAFBGLYAMYA, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:51.290646027 (6h17m57.663225392s) [noise=true , address=104.36.82.2:4732 , piece_id=DBPJH5FSNFRPIBLP7HWWJFVVXYC47J7UGPCXFL7L3BD6BGFSJDFA, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:51.309329961 (6h17m57.681909316s) [noise=true , address=104.36.82.12:4150 , piece_id=GQ4GQQZHK3Y4KVVB2IUU4LEEBR4OJ2YSS5Z5DF7WHHDCYYSSGRYA, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:51.317895332 (6h17m57.690474687s) [noise=true , address=104.36.82.9:4007 , piece_id=R2IPGORJ3Z4FFCC52P5WMXYIA6O5V7DDZHHL4X64XV6J36SIWSBQ, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:52.525479047 (6h17m58.898058402s) [noise=true , address=45.91.116.9:4262 , piece_id=EXWTIYPESLJQK3CDK732WC63OZ7OJOQAUCBPGO3ZU54UBFUJ3TSA, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:09:18.157484198 (6h18m24.530063553s) [noise=true , address=77.90.31.13:4628 , piece_id=OAGGFQO2CORAZQV6ND3EERSICEOSHKK73Y3PUWLT45ZUWEWDHOFQ, seg_pos={0 0}, upload=5005]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: expected piece hash; context deadline exceeded; EOF Jul 19 05:09:18.157617684 (6h18m24.530197039s) [noise=true , address=77.90.31.15:4698 , piece_id=HNTVTQMVXDUMK4CMHKVUZ7DHM3MZL56TGXFR4FFFV2V3DCTLAVZQ, seg_pos={0 0}, upload=5005]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF Jul 19 05:09:18.157744796 (6h18m24.530324151s) [noise=true , address=77.90.31.11:4555 , piece_id=QGRXVFSKXDCLKSSSXEK4QYZEETWQGFZMQQBSTXNMAHY45RA6MAVQ, seg_pos={0 0}, upload=5005]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: write tcp 188.166.225.170:38288->77.90.31.11:4555: use of closed network connection; write tcp 188.166.225.170:38288->77.90.31.11:4555: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38288->77.90.31.11:4555: use of closed network connection Jul 19 05:09:18.157785745 (6h18m24.5303651s) [noise=true , address=77.90.31.13:4626 , piece_id=2QAZAYJWKJXTPY475RLOSPV3ZZTMB6OZLNYAKD7FCFV7T4WPC5ZQ, seg_pos={0 0}, upload=5005]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:58846->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:58846->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58846->77.90.31.13:4626: use of closed network connection Jul 19 05:09:18.157859036 (6h18m24.530438391s) [noise=true , address=77.90.31.12:4592 , piece_id=HS5YQKPTPUVHFZH3VZ7X2LAO2SVS6JZOTHTLJ6T47G3CLTZTR7UQ, seg_pos={0 0}, upload=5005]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:38220->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:38220->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38220->77.90.31.12:4592: use of closed network connection Jul 19 05:09:18.157898821 (6h18m24.530478176s) [seg_pos={0 0}, upload=5005]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 19 05:09:18.389974737 (6h18m24.762554092s) [upload=5005]: Done uploading segments... Jul 19 05:09:18.389994916 (6h18m24.762574271s) [upload=5005]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.780713294s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 5.508858139s FAIL (2025-07-19 05:11:28 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 4855s) [upload=4769]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 04:51:09.109709627 (6h0m15.482288982s) [upload=4769]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 04:51:09.556145753 (6h0m15.928725108s) [seg_pos={0 0}, upload=4769]: Begin upload segment... Jul 19 04:51:09.556194817 (6h0m15.928774172s) [seg_pos={0 0}, upload=4769]: Done begin upload segment. Jul 19 04:51:09.556223893 (6h0m15.928803248s) [noise=true , address=79.127.219.38:4023 , piece_id=PTPOZGLYV4MZTOFZNN3NCGYL7DFAST4PXKEUIVTMFL7HV4GZTCLQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556252197 (6h0m15.928831552s) [noise=true , address=45.91.95.4:4011 , piece_id=ZMBAS6KKHYVIRE6OBL2NFVCWMM4TKQ7J64BD3SYXIK54XCEDI2XA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556579383 (6h0m15.929158738s) [noise=true , address=45.91.116.6:4007 , piece_id=A2AFVTZBXSGBOE7XIM3B5HRN4STITFBXO6MZRJ365URNAAOZ3PSQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556854469 (6h0m15.929433824s) [noise=true , address=104.36.82.19:4423 , piece_id=PIJUHDUND36HQL67I33D7A3EULMJHS7N55MN4OFQJ4OWNQ42O47A, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557094107 (6h0m15.929673462s) [noise=true , address=45.91.116.9:4264 , piece_id=GHM5BKZKMT2QOHHBOS4FSPOWEWSFQRGRVVR4GSNDPHVIFRZHQL2A, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557336381 (6h0m15.929915736s) [noise=true , address=79.127.205.235:4660 , piece_id=UOUOGE2VP7PX7EMRZLPMCWFCYEZERM3DQMCQ6NG4J2K2DEIZR63Q, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557557885 (6h0m15.93013724s) [noise=true , address=79.127.205.246:4280 , piece_id=FLBOZI3UGTQXAQBLZ2F45Z3OEYL65F5PQBXRF2DQMZ7NGF7VDRPA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557762296 (6h0m15.930341651s) [noise=true , address=79.127.205.233:4622 , piece_id=6WWNYQTH5MGAJOEB7BXKZDJOMCAZXDW66J3OEANGDNJDNUMLZM7Q, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558014689 (6h0m15.930594044s) [noise=true , address=64.49.13.138:4482 , piece_id=MAW2M4US2OQNOS3WI67AA7NIBTJDE5T2JQTXKFPBLGVGXKVCFGTQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558215974 (6h0m15.930795319s) [noise=true , address=77.90.31.10:4546 , piece_id=R6LCA5WZOEBDURL5S3MEK22YPVYXX6EXSS4J4W57543HELYXOKZA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558409364 (6h0m15.930988719s) [noise=true , address=45.91.95.5:4019 , piece_id=YTSBCN2LPHEMRRAZIHJGVB7NBOTRXVEHD2TWUCTTLBNFFYEWWDKA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558623504 (6h0m15.931202859s) [noise=true , address=79.127.201.215:4357 , piece_id=75XWAG2VQWEYCWN2Q3WRIZSBI7LDOISZJPXQLQ2GIPNEN34VLSUQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558805162 (6h0m15.931384517s) [noise=true , address=45.91.116.14:4219 , piece_id=PDW6JPRHCVMHHOWIHT6JKV4CTSC3JY6NDKDVFZSO47OYNPZ7PGGA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558999373 (6h0m15.931578728s) [noise=true , address=77.90.31.11:4564 , piece_id=L2U7JFJRDQOFA2AYVAGTI4ZW4VABCXKMXWBIUWQO2QNNUUBJ74BA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559184768 (6h0m15.931764123s) [noise=true , address=77.90.31.16:4768 , piece_id=YNLSONFM6COZEVF6ZNQR2EA6V4LSQFZJONIXR4AFTNNMFGPV57JA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559368719 (6h0m15.931948074s) [noise=true , address=79.127.220.98:4432 , piece_id=2SUTFQ7FVDPAM2H4W3OLW6AVH3VJ33IJFE4L52SA3ET5LGBMOMPQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559549706 (6h0m15.932129061s) [noise=true , address=77.90.31.18:4813 , piece_id=VDOT7BB4O7XYJHW6BFTAYQZZYLZU5PAUXVJGSXJ7FDI4TH425COA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559727836 (6h0m15.932307191s) [noise=true , address=79.127.219.51:4211 , piece_id=HO5PLWBVU5YAYNYMN352NA73SM4WBJV4WZ7XLSQTCQ76UAL2TOJA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:10.663359087 (6h0m17.035938442s) [upload=4769]: Next returned nil segment Jul 19 04:51:10.663378664 (6h0m17.035958019s) [upload=4769]: Waiting for error group managing segments... Jul 19 04:51:10.962945479 (6h0m17.335524834s) [noise=true , address=79.127.205.233:4622 , piece_id=6WWNYQTH5MGAJOEB7BXKZDJOMCAZXDW66J3OEANGDNJDNUMLZM7Q, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:10.967689807 (6h0m17.340269152s) [noise=true , address=79.127.205.235:4660 , piece_id=UOUOGE2VP7PX7EMRZLPMCWFCYEZERM3DQMCQ6NG4J2K2DEIZR63Q, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:10.976968631 (6h0m17.349547986s) [noise=true , address=64.49.13.138:4482 , piece_id=MAW2M4US2OQNOS3WI67AA7NIBTJDE5T2JQTXKFPBLGVGXKVCFGTQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.048750700 (6h0m17.421330055s) [noise=true , address=79.127.205.246:4280 , piece_id=FLBOZI3UGTQXAQBLZ2F45Z3OEYL65F5PQBXRF2DQMZ7NGF7VDRPA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.221642403 (6h0m17.594221758s) [noise=true , address=79.127.220.98:4432 , piece_id=2SUTFQ7FVDPAM2H4W3OLW6AVH3VJ33IJFE4L52SA3ET5LGBMOMPQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.317070435 (6h0m17.68964979s) [noise=true , address=45.91.95.4:4011 , piece_id=ZMBAS6KKHYVIRE6OBL2NFVCWMM4TKQ7J64BD3SYXIK54XCEDI2XA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.324002660 (6h0m17.696582015s) [noise=true , address=45.91.95.5:4019 , piece_id=YTSBCN2LPHEMRRAZIHJGVB7NBOTRXVEHD2TWUCTTLBNFFYEWWDKA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.460976317 (6h0m17.833555672s) [noise=true , address=79.127.201.215:4357 , piece_id=75XWAG2VQWEYCWN2Q3WRIZSBI7LDOISZJPXQLQ2GIPNEN34VLSUQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.462584603 (6h0m17.835163958s) [noise=true , address=45.91.116.6:4007 , piece_id=A2AFVTZBXSGBOE7XIM3B5HRN4STITFBXO6MZRJ365URNAAOZ3PSQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.490392973 (6h0m17.862972328s) [noise=true , address=45.91.116.9:4264 , piece_id=GHM5BKZKMT2QOHHBOS4FSPOWEWSFQRGRVVR4GSNDPHVIFRZHQL2A, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.502040780 (6h0m17.874620135s) [noise=true , address=45.91.116.14:4219 , piece_id=PDW6JPRHCVMHHOWIHT6JKV4CTSC3JY6NDKDVFZSO47OYNPZ7PGGA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.600434096 (6h0m17.973013451s) [noise=true , address=79.127.219.51:4211 , piece_id=HO5PLWBVU5YAYNYMN352NA73SM4WBJV4WZ7XLSQTCQ76UAL2TOJA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.609008984 (6h0m17.981588339s) [noise=true , address=104.36.82.19:4423 , piece_id=PIJUHDUND36HQL67I33D7A3EULMJHS7N55MN4OFQJ4OWNQ42O47A, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.620825273 (6h0m17.993404628s) [noise=true , address=79.127.219.38:4023 , piece_id=PTPOZGLYV4MZTOFZNN3NCGYL7DFAST4PXKEUIVTMFL7HV4GZTCLQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.620838659 (6h0m17.993418014s) [seg_pos={0 0}, upload=4769]: Segment reached optimal threshold of 14 pieces. Jul 19 04:51:11.621067807 (6h0m17.993647163s) [noise=true , address=77.90.31.11:4564 , piece_id=L2U7JFJRDQOFA2AYVAGTI4ZW4VABCXKMXWBIUWQO2QNNUUBJ74BA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection; write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection Jul 19 04:51:11.621134234 (6h0m17.993713589s) [noise=true , address=77.90.31.10:4546 , piece_id=R6LCA5WZOEBDURL5S3MEK22YPVYXX6EXSS4J4W57543HELYXOKZA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK): protocol: read tcp 188.166.225.170:58476->77.90.31.10:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621174191 (6h0m17.993753546s) [noise=true , address=77.90.31.16:4768 , piece_id=YNLSONFM6COZEVF6ZNQR2EA6V4LSQFZJONIXR4AFTNNMFGPV57JA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UchewmtxBrtSZSpVAJFdZmcTsQVQxkjK9re6cgUGqDhyonQeF): protocol: read tcp 188.166.225.170:56326->77.90.31.16:4768: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621216842 (6h0m17.993796197s) [noise=true , address=77.90.31.18:4813 , piece_id=VDOT7BB4O7XYJHW6BFTAYQZZYLZU5PAUXVJGSXJ7FDI4TH425COA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128WthA1x9wp1dV1vkpGnoveZQpU9CSgDCZfw5GMQuVpCNSVAM5): protocol: read tcp 188.166.225.170:49704->77.90.31.18:4813: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621234917 (6h0m17.993814272s) [seg_pos={0 0}, upload=4769]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 19 04:51:11.621301444 (6h0m17.993880799s) [upload=4769]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 19 04:51:11.621305402 (6h0m17.993884758s) [upload=4769]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 19 04:51:11.923136970 (6h0m18.295716325s) [upload=4769]: Done uploading segments... Jul 19 04:51:11.923156627 (6h0m18.295735972s) [upload=4769]: Upload finished. err: <nil> Duration: 5.508886104s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [..........................XX...X...X....X..X....XXXXX.X.X...] duration: 5.800191955s FAIL (2025-07-19 05:11:12 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== : <nil> Jul 19 05:08:29.800544823 (6h17m36.173124169s) [upload=4997]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 19 05:08:29.800548410 (6h17m36.173127765s) [upload=4997]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 19 05:08:30.148027600 (6h17m36.520606955s) [upload=4997]: Done uploading segments... Jul 19 05:08:30.148148401 (6h17m36.520727746s) [upload=4997]: Upload finished. err: <nil> Jul 19 05:08:48.807983386 (6h17m55.180562741s) [upload=5005]: Starting upload Jul 19 05:08:48.808044453 (6h17m55.180623808s) [upload=5005]: Done starting upload Jul 19 05:08:48.808175243 (6h17m55.180754598s) [upload=5005]: Uploading segments... Jul 19 05:08:48.808334198 (6h17m55.180913553s) [upload=5005]: Got next segment. Inline: false Jul 19 05:08:48.808392509 (6h17m55.180971874s) [upload=5005]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 05:08:48.808399823 (6h17m55.180979188s) [upload=5005]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 05:08:49.075515233 (6h17m55.448094578s) [seg_pos={0 0}, upload=5005]: Begin upload segment... Jul 19 05:08:49.075564547 (6h17m55.448143902s) [seg_pos={0 0}, upload=5005]: Done begin upload segment. Jul 19 05:08:49.075602881 (6h17m55.448182236s) [noise=true , address=77.90.31.11:4555 , piece_id=QGRXVFSKXDCLKSSSXEK4QYZEETWQGFZMQQBSTXNMAHY45RA6MAVQ, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.075612279 (6h17m55.448191634s) [noise=true , address=104.36.82.13:4186 , piece_id=PPK6XEQYJ3XV27XUJYKUL2MYAR5KUYKF5AHJS5EBXOK7PZWWM66Q, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.075964213 (6h17m55.448543568s) [noise=true , address=77.90.31.12:4592 , piece_id=HS5YQKPTPUVHFZH3VZ7X2LAO2SVS6JZOTHTLJ6T47G3CLTZTR7UQ, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.076122025 (6h17m55.44870137s) [noise=true , address=104.36.82.12:4150 , piece_id=GQ4GQQZHK3Y4KVVB2IUU4LEEBR4OJ2YSS5Z5DF7WHHDCYYSSGRYA, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.076272903 (6h17m55.448852258s) [noise=true , address=45.91.116.13:4154 , piece_id=OJO7VBXGBLV2SG2AQ6BPBP57SL75WV4UZSWC3O5HXD7ROBICSYGA, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.076533412 (6h17m55.449112767s) [noise=true , address=104.36.82.2:4732 , piece_id=DBPJH5FSNFRPIBLP7HWWJFVVXYC47J7UGPCXFL7L3BD6BGFSJDFA, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.076794962 (6h17m55.449374317s) [noise=true , address=45.91.116.9:4263 , piece_id=4L3UWFKV3AC7F4RPS7GJFXOO6DJNCPMFB5NWGJH6HXAFBGLYAMYA, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.077044971 (6h17m55.449624326s) [noise=true , address=104.36.82.16:4293 , piece_id=2NA7I27WVAWAUWDCZVYBJLNXGWIDNJMKZU7YFJZSXCQMCHKLCGXA, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.077296953 (6h17m55.449876308s) [noise=true , address=77.90.31.15:4698 , piece_id=HNTVTQMVXDUMK4CMHKVUZ7DHM3MZL56TGXFR4FFFV2V3DCTLAVZQ, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.077501123 (6h17m55.450080468s) [noise=true , address=104.36.82.9:4007 , piece_id=R2IPGORJ3Z4FFCC52P5WMXYIA6O5V7DDZHHL4X64XV6J36SIWSBQ, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.077695656 (6h17m55.450275001s) [noise=true , address=45.91.116.8:4048 , piece_id=QKXGR6YZSYTQIUXV54DRYQNWY352UFS2Z2VXWFJTD7YJYJZ24NCQ, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.077994558 (6h17m55.450573913s) [noise=true , address=104.36.82.14:4222 , piece_id=4F6ZDS2KMQNALT2YV6D3TAZLIBFUFBM5NNCBGLSE6L6PR5QMGPVA, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.078213187 (6h17m55.450792542s) [noise=true , address=45.91.116.4:4939 , piece_id=J2BIIQITH7R54K4ENE6AKU7KCOUAGYBOTKX6VF6FY7FNQDMNNFCA, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.078412077 (6h17m55.450991432s) [noise=true , address=77.90.31.13:4626 , piece_id=2QAZAYJWKJXTPY475RLOSPV3ZZTMB6OZLNYAKD7FCFV7T4WPC5ZQ, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.079649705 (6h17m55.45222906s) [noise=true , address=45.91.116.9:4262 , piece_id=EXWTIYPESLJQK3CDK732WC63OZ7OJOQAUCBPGO3ZU54UBFUJ3TSA, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.080903344 (6h17m55.453482699s) [noise=true , address=77.90.31.13:4628 , piece_id=OAGGFQO2CORAZQV6ND3EERSICEOSHKK73Y3PUWLT45ZUWEWDHOFQ, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:50.857952826 (6h17m57.230532181s) [upload=5005]: Next returned nil segment Jul 19 05:08:50.857969477 (6h17m57.230548832s) [upload=5005]: Waiting for error group managing segments... Jul 19 05:08:51.139405061 (6h17m57.511984416s) [noise=true , address=104.36.82.13:4186 , piece_id=PPK6XEQYJ3XV27XUJYKUL2MYAR5KUYKF5AHJS5EBXOK7PZWWM66Q, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:51.143987831 (6h17m57.516567176s) [noise=true , address=45.91.116.8:4048 , piece_id=QKXGR6YZSYTQIUXV54DRYQNWY352UFS2Z2VXWFJTD7YJYJZ24NCQ, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:51.148258443 (6h17m57.520837798s) [noise=true , address=104.36.82.16:4293 , piece_id=2NA7I27WVAWAUWDCZVYBJLNXGWIDNJMKZU7YFJZSXCQMCHKLCGXA, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:51.156796512 (6h17m57.529375867s) [noise=true , address=45.91.116.13:4154 , piece_id=OJO7VBXGBLV2SG2AQ6BPBP57SL75WV4UZSWC3O5HXD7ROBICSYGA, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:51.193908742 (6h17m57.566488098s) [noise=true , address=104.36.82.14:4222 , piece_id=4F6ZDS2KMQNALT2YV6D3TAZLIBFUFBM5NNCBGLSE6L6PR5QMGPVA, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:51.217498736 (6h17m57.590078091s) [noise=true , address=45.91.116.4:4939 , piece_id=J2BIIQITH7R54K4ENE6AKU7KCOUAGYBOTKX6VF6FY7FNQDMNNFCA, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:51.218585386 (6h17m57.591164741s) [noise=true , address=45.91.116.9:4263 , piece_id=4L3UWFKV3AC7F4RPS7GJFXOO6DJNCPMFB5NWGJH6HXAFBGLYAMYA, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:51.290646027 (6h17m57.663225392s) [noise=true , address=104.36.82.2:4732 , piece_id=DBPJH5FSNFRPIBLP7HWWJFVVXYC47J7UGPCXFL7L3BD6BGFSJDFA, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:51.309329961 (6h17m57.681909316s) [noise=true , address=104.36.82.12:4150 , piece_id=GQ4GQQZHK3Y4KVVB2IUU4LEEBR4OJ2YSS5Z5DF7WHHDCYYSSGRYA, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:51.317895332 (6h17m57.690474687s) [noise=true , address=104.36.82.9:4007 , piece_id=R2IPGORJ3Z4FFCC52P5WMXYIA6O5V7DDZHHL4X64XV6J36SIWSBQ, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:52.525479047 (6h17m58.898058402s) [noise=true , address=45.91.116.9:4262 , piece_id=EXWTIYPESLJQK3CDK732WC63OZ7OJOQAUCBPGO3ZU54UBFUJ3TSA, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:09:18.157484198 (6h18m24.530063553s) [noise=true , address=77.90.31.13:4628 , piece_id=OAGGFQO2CORAZQV6ND3EERSICEOSHKK73Y3PUWLT45ZUWEWDHOFQ, seg_pos={0 0}, upload=5005]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: expected piece hash; context deadline exceeded; EOF Jul 19 05:09:18.157617684 (6h18m24.530197039s) [noise=true , address=77.90.31.15:4698 , piece_id=HNTVTQMVXDUMK4CMHKVUZ7DHM3MZL56TGXFR4FFFV2V3DCTLAVZQ, seg_pos={0 0}, upload=5005]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF Jul 19 05:09:18.157744796 (6h18m24.530324151s) [noise=true , address=77.90.31.11:4555 , piece_id=QGRXVFSKXDCLKSSSXEK4QYZEETWQGFZMQQBSTXNMAHY45RA6MAVQ, seg_pos={0 0}, upload=5005]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: write tcp 188.166.225.170:38288->77.90.31.11:4555: use of closed network connection; write tcp 188.166.225.170:38288->77.90.31.11:4555: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38288->77.90.31.11:4555: use of closed network connection Jul 19 05:09:18.157785745 (6h18m24.5303651s) [noise=true , address=77.90.31.13:4626 , piece_id=2QAZAYJWKJXTPY475RLOSPV3ZZTMB6OZLNYAKD7FCFV7T4WPC5ZQ, seg_pos={0 0}, upload=5005]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:58846->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:58846->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58846->77.90.31.13:4626: use of closed network connection Jul 19 05:09:18.157859036 (6h18m24.530438391s) [noise=true , address=77.90.31.12:4592 , piece_id=HS5YQKPTPUVHFZH3VZ7X2LAO2SVS6JZOTHTLJ6T47G3CLTZTR7UQ, seg_pos={0 0}, upload=5005]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:38220->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:38220->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38220->77.90.31.12:4592: use of closed network connection Jul 19 05:09:18.157898821 (6h18m24.530478176s) [seg_pos={0 0}, upload=5005]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 19 05:09:18.389974737 (6h18m24.762554092s) [upload=5005]: Done uploading segments... Jul 19 05:09:18.389994916 (6h18m24.762574271s) [upload=5005]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.800217705s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 7.123818105s FAIL (2025-07-19 05:10:26 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 4855s) [upload=4769]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 04:51:09.109709627 (6h0m15.482288982s) [upload=4769]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 04:51:09.556145753 (6h0m15.928725108s) [seg_pos={0 0}, upload=4769]: Begin upload segment... Jul 19 04:51:09.556194817 (6h0m15.928774172s) [seg_pos={0 0}, upload=4769]: Done begin upload segment. Jul 19 04:51:09.556223893 (6h0m15.928803248s) [noise=true , address=79.127.219.38:4023 , piece_id=PTPOZGLYV4MZTOFZNN3NCGYL7DFAST4PXKEUIVTMFL7HV4GZTCLQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556252197 (6h0m15.928831552s) [noise=true , address=45.91.95.4:4011 , piece_id=ZMBAS6KKHYVIRE6OBL2NFVCWMM4TKQ7J64BD3SYXIK54XCEDI2XA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556579383 (6h0m15.929158738s) [noise=true , address=45.91.116.6:4007 , piece_id=A2AFVTZBXSGBOE7XIM3B5HRN4STITFBXO6MZRJ365URNAAOZ3PSQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556854469 (6h0m15.929433824s) [noise=true , address=104.36.82.19:4423 , piece_id=PIJUHDUND36HQL67I33D7A3EULMJHS7N55MN4OFQJ4OWNQ42O47A, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557094107 (6h0m15.929673462s) [noise=true , address=45.91.116.9:4264 , piece_id=GHM5BKZKMT2QOHHBOS4FSPOWEWSFQRGRVVR4GSNDPHVIFRZHQL2A, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557336381 (6h0m15.929915736s) [noise=true , address=79.127.205.235:4660 , piece_id=UOUOGE2VP7PX7EMRZLPMCWFCYEZERM3DQMCQ6NG4J2K2DEIZR63Q, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557557885 (6h0m15.93013724s) [noise=true , address=79.127.205.246:4280 , piece_id=FLBOZI3UGTQXAQBLZ2F45Z3OEYL65F5PQBXRF2DQMZ7NGF7VDRPA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557762296 (6h0m15.930341651s) [noise=true , address=79.127.205.233:4622 , piece_id=6WWNYQTH5MGAJOEB7BXKZDJOMCAZXDW66J3OEANGDNJDNUMLZM7Q, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558014689 (6h0m15.930594044s) [noise=true , address=64.49.13.138:4482 , piece_id=MAW2M4US2OQNOS3WI67AA7NIBTJDE5T2JQTXKFPBLGVGXKVCFGTQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558215974 (6h0m15.930795319s) [noise=true , address=77.90.31.10:4546 , piece_id=R6LCA5WZOEBDURL5S3MEK22YPVYXX6EXSS4J4W57543HELYXOKZA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558409364 (6h0m15.930988719s) [noise=true , address=45.91.95.5:4019 , piece_id=YTSBCN2LPHEMRRAZIHJGVB7NBOTRXVEHD2TWUCTTLBNFFYEWWDKA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558623504 (6h0m15.931202859s) [noise=true , address=79.127.201.215:4357 , piece_id=75XWAG2VQWEYCWN2Q3WRIZSBI7LDOISZJPXQLQ2GIPNEN34VLSUQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558805162 (6h0m15.931384517s) [noise=true , address=45.91.116.14:4219 , piece_id=PDW6JPRHCVMHHOWIHT6JKV4CTSC3JY6NDKDVFZSO47OYNPZ7PGGA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558999373 (6h0m15.931578728s) [noise=true , address=77.90.31.11:4564 , piece_id=L2U7JFJRDQOFA2AYVAGTI4ZW4VABCXKMXWBIUWQO2QNNUUBJ74BA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559184768 (6h0m15.931764123s) [noise=true , address=77.90.31.16:4768 , piece_id=YNLSONFM6COZEVF6ZNQR2EA6V4LSQFZJONIXR4AFTNNMFGPV57JA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559368719 (6h0m15.931948074s) [noise=true , address=79.127.220.98:4432 , piece_id=2SUTFQ7FVDPAM2H4W3OLW6AVH3VJ33IJFE4L52SA3ET5LGBMOMPQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559549706 (6h0m15.932129061s) [noise=true , address=77.90.31.18:4813 , piece_id=VDOT7BB4O7XYJHW6BFTAYQZZYLZU5PAUXVJGSXJ7FDI4TH425COA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559727836 (6h0m15.932307191s) [noise=true , address=79.127.219.51:4211 , piece_id=HO5PLWBVU5YAYNYMN352NA73SM4WBJV4WZ7XLSQTCQ76UAL2TOJA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:10.663359087 (6h0m17.035938442s) [upload=4769]: Next returned nil segment Jul 19 04:51:10.663378664 (6h0m17.035958019s) [upload=4769]: Waiting for error group managing segments... Jul 19 04:51:10.962945479 (6h0m17.335524834s) [noise=true , address=79.127.205.233:4622 , piece_id=6WWNYQTH5MGAJOEB7BXKZDJOMCAZXDW66J3OEANGDNJDNUMLZM7Q, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:10.967689807 (6h0m17.340269152s) [noise=true , address=79.127.205.235:4660 , piece_id=UOUOGE2VP7PX7EMRZLPMCWFCYEZERM3DQMCQ6NG4J2K2DEIZR63Q, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:10.976968631 (6h0m17.349547986s) [noise=true , address=64.49.13.138:4482 , piece_id=MAW2M4US2OQNOS3WI67AA7NIBTJDE5T2JQTXKFPBLGVGXKVCFGTQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.048750700 (6h0m17.421330055s) [noise=true , address=79.127.205.246:4280 , piece_id=FLBOZI3UGTQXAQBLZ2F45Z3OEYL65F5PQBXRF2DQMZ7NGF7VDRPA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.221642403 (6h0m17.594221758s) [noise=true , address=79.127.220.98:4432 , piece_id=2SUTFQ7FVDPAM2H4W3OLW6AVH3VJ33IJFE4L52SA3ET5LGBMOMPQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.317070435 (6h0m17.68964979s) [noise=true , address=45.91.95.4:4011 , piece_id=ZMBAS6KKHYVIRE6OBL2NFVCWMM4TKQ7J64BD3SYXIK54XCEDI2XA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.324002660 (6h0m17.696582015s) [noise=true , address=45.91.95.5:4019 , piece_id=YTSBCN2LPHEMRRAZIHJGVB7NBOTRXVEHD2TWUCTTLBNFFYEWWDKA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.460976317 (6h0m17.833555672s) [noise=true , address=79.127.201.215:4357 , piece_id=75XWAG2VQWEYCWN2Q3WRIZSBI7LDOISZJPXQLQ2GIPNEN34VLSUQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.462584603 (6h0m17.835163958s) [noise=true , address=45.91.116.6:4007 , piece_id=A2AFVTZBXSGBOE7XIM3B5HRN4STITFBXO6MZRJ365URNAAOZ3PSQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.490392973 (6h0m17.862972328s) [noise=true , address=45.91.116.9:4264 , piece_id=GHM5BKZKMT2QOHHBOS4FSPOWEWSFQRGRVVR4GSNDPHVIFRZHQL2A, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.502040780 (6h0m17.874620135s) [noise=true , address=45.91.116.14:4219 , piece_id=PDW6JPRHCVMHHOWIHT6JKV4CTSC3JY6NDKDVFZSO47OYNPZ7PGGA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.600434096 (6h0m17.973013451s) [noise=true , address=79.127.219.51:4211 , piece_id=HO5PLWBVU5YAYNYMN352NA73SM4WBJV4WZ7XLSQTCQ76UAL2TOJA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.609008984 (6h0m17.981588339s) [noise=true , address=104.36.82.19:4423 , piece_id=PIJUHDUND36HQL67I33D7A3EULMJHS7N55MN4OFQJ4OWNQ42O47A, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.620825273 (6h0m17.993404628s) [noise=true , address=79.127.219.38:4023 , piece_id=PTPOZGLYV4MZTOFZNN3NCGYL7DFAST4PXKEUIVTMFL7HV4GZTCLQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.620838659 (6h0m17.993418014s) [seg_pos={0 0}, upload=4769]: Segment reached optimal threshold of 14 pieces. Jul 19 04:51:11.621067807 (6h0m17.993647163s) [noise=true , address=77.90.31.11:4564 , piece_id=L2U7JFJRDQOFA2AYVAGTI4ZW4VABCXKMXWBIUWQO2QNNUUBJ74BA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection; write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection Jul 19 04:51:11.621134234 (6h0m17.993713589s) [noise=true , address=77.90.31.10:4546 , piece_id=R6LCA5WZOEBDURL5S3MEK22YPVYXX6EXSS4J4W57543HELYXOKZA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK): protocol: read tcp 188.166.225.170:58476->77.90.31.10:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621174191 (6h0m17.993753546s) [noise=true , address=77.90.31.16:4768 , piece_id=YNLSONFM6COZEVF6ZNQR2EA6V4LSQFZJONIXR4AFTNNMFGPV57JA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UchewmtxBrtSZSpVAJFdZmcTsQVQxkjK9re6cgUGqDhyonQeF): protocol: read tcp 188.166.225.170:56326->77.90.31.16:4768: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621216842 (6h0m17.993796197s) [noise=true , address=77.90.31.18:4813 , piece_id=VDOT7BB4O7XYJHW6BFTAYQZZYLZU5PAUXVJGSXJ7FDI4TH425COA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128WthA1x9wp1dV1vkpGnoveZQpU9CSgDCZfw5GMQuVpCNSVAM5): protocol: read tcp 188.166.225.170:49704->77.90.31.18:4813: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621234917 (6h0m17.993814272s) [seg_pos={0 0}, upload=4769]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 19 04:51:11.621301444 (6h0m17.993880799s) [upload=4769]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 19 04:51:11.621305402 (6h0m17.993884758s) [upload=4769]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 19 04:51:11.923136970 (6h0m18.295716325s) [upload=4769]: Done uploading segments... Jul 19 04:51:11.923156627 (6h0m18.295735972s) [upload=4769]: Upload finished. err: <nil> Duration: 7.123836702s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [...........................XX...X...X....X..X....XXXXX.X.X..] duration: 15.087940088s FAIL (2025-07-19 05:10:21 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== : <nil> Jul 19 05:08:29.800544823 (6h17m36.173124169s) [upload=4997]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 19 05:08:29.800548410 (6h17m36.173127765s) [upload=4997]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 19 05:08:30.148027600 (6h17m36.520606955s) [upload=4997]: Done uploading segments... Jul 19 05:08:30.148148401 (6h17m36.520727746s) [upload=4997]: Upload finished. err: <nil> Jul 19 05:08:48.807983386 (6h17m55.180562741s) [upload=5005]: Starting upload Jul 19 05:08:48.808044453 (6h17m55.180623808s) [upload=5005]: Done starting upload Jul 19 05:08:48.808175243 (6h17m55.180754598s) [upload=5005]: Uploading segments... Jul 19 05:08:48.808334198 (6h17m55.180913553s) [upload=5005]: Got next segment. Inline: false Jul 19 05:08:48.808392509 (6h17m55.180971874s) [upload=5005]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 05:08:48.808399823 (6h17m55.180979188s) [upload=5005]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 05:08:49.075515233 (6h17m55.448094578s) [seg_pos={0 0}, upload=5005]: Begin upload segment... Jul 19 05:08:49.075564547 (6h17m55.448143902s) [seg_pos={0 0}, upload=5005]: Done begin upload segment. Jul 19 05:08:49.075602881 (6h17m55.448182236s) [noise=true , address=77.90.31.11:4555 , piece_id=QGRXVFSKXDCLKSSSXEK4QYZEETWQGFZMQQBSTXNMAHY45RA6MAVQ, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.075612279 (6h17m55.448191634s) [noise=true , address=104.36.82.13:4186 , piece_id=PPK6XEQYJ3XV27XUJYKUL2MYAR5KUYKF5AHJS5EBXOK7PZWWM66Q, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.075964213 (6h17m55.448543568s) [noise=true , address=77.90.31.12:4592 , piece_id=HS5YQKPTPUVHFZH3VZ7X2LAO2SVS6JZOTHTLJ6T47G3CLTZTR7UQ, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.076122025 (6h17m55.44870137s) [noise=true , address=104.36.82.12:4150 , piece_id=GQ4GQQZHK3Y4KVVB2IUU4LEEBR4OJ2YSS5Z5DF7WHHDCYYSSGRYA, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.076272903 (6h17m55.448852258s) [noise=true , address=45.91.116.13:4154 , piece_id=OJO7VBXGBLV2SG2AQ6BPBP57SL75WV4UZSWC3O5HXD7ROBICSYGA, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.076533412 (6h17m55.449112767s) [noise=true , address=104.36.82.2:4732 , piece_id=DBPJH5FSNFRPIBLP7HWWJFVVXYC47J7UGPCXFL7L3BD6BGFSJDFA, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.076794962 (6h17m55.449374317s) [noise=true , address=45.91.116.9:4263 , piece_id=4L3UWFKV3AC7F4RPS7GJFXOO6DJNCPMFB5NWGJH6HXAFBGLYAMYA, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.077044971 (6h17m55.449624326s) [noise=true , address=104.36.82.16:4293 , piece_id=2NA7I27WVAWAUWDCZVYBJLNXGWIDNJMKZU7YFJZSXCQMCHKLCGXA, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.077296953 (6h17m55.449876308s) [noise=true , address=77.90.31.15:4698 , piece_id=HNTVTQMVXDUMK4CMHKVUZ7DHM3MZL56TGXFR4FFFV2V3DCTLAVZQ, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.077501123 (6h17m55.450080468s) [noise=true , address=104.36.82.9:4007 , piece_id=R2IPGORJ3Z4FFCC52P5WMXYIA6O5V7DDZHHL4X64XV6J36SIWSBQ, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.077695656 (6h17m55.450275001s) [noise=true , address=45.91.116.8:4048 , piece_id=QKXGR6YZSYTQIUXV54DRYQNWY352UFS2Z2VXWFJTD7YJYJZ24NCQ, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.077994558 (6h17m55.450573913s) [noise=true , address=104.36.82.14:4222 , piece_id=4F6ZDS2KMQNALT2YV6D3TAZLIBFUFBM5NNCBGLSE6L6PR5QMGPVA, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.078213187 (6h17m55.450792542s) [noise=true , address=45.91.116.4:4939 , piece_id=J2BIIQITH7R54K4ENE6AKU7KCOUAGYBOTKX6VF6FY7FNQDMNNFCA, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.078412077 (6h17m55.450991432s) [noise=true , address=77.90.31.13:4626 , piece_id=2QAZAYJWKJXTPY475RLOSPV3ZZTMB6OZLNYAKD7FCFV7T4WPC5ZQ, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.079649705 (6h17m55.45222906s) [noise=true , address=45.91.116.9:4262 , piece_id=EXWTIYPESLJQK3CDK732WC63OZ7OJOQAUCBPGO3ZU54UBFUJ3TSA, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.080903344 (6h17m55.453482699s) [noise=true , address=77.90.31.13:4628 , piece_id=OAGGFQO2CORAZQV6ND3EERSICEOSHKK73Y3PUWLT45ZUWEWDHOFQ, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:50.857952826 (6h17m57.230532181s) [upload=5005]: Next returned nil segment Jul 19 05:08:50.857969477 (6h17m57.230548832s) [upload=5005]: Waiting for error group managing segments... Jul 19 05:08:51.139405061 (6h17m57.511984416s) [noise=true , address=104.36.82.13:4186 , piece_id=PPK6XEQYJ3XV27XUJYKUL2MYAR5KUYKF5AHJS5EBXOK7PZWWM66Q, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:51.143987831 (6h17m57.516567176s) [noise=true , address=45.91.116.8:4048 , piece_id=QKXGR6YZSYTQIUXV54DRYQNWY352UFS2Z2VXWFJTD7YJYJZ24NCQ, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:51.148258443 (6h17m57.520837798s) [noise=true , address=104.36.82.16:4293 , piece_id=2NA7I27WVAWAUWDCZVYBJLNXGWIDNJMKZU7YFJZSXCQMCHKLCGXA, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:51.156796512 (6h17m57.529375867s) [noise=true , address=45.91.116.13:4154 , piece_id=OJO7VBXGBLV2SG2AQ6BPBP57SL75WV4UZSWC3O5HXD7ROBICSYGA, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:51.193908742 (6h17m57.566488098s) [noise=true , address=104.36.82.14:4222 , piece_id=4F6ZDS2KMQNALT2YV6D3TAZLIBFUFBM5NNCBGLSE6L6PR5QMGPVA, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:51.217498736 (6h17m57.590078091s) [noise=true , address=45.91.116.4:4939 , piece_id=J2BIIQITH7R54K4ENE6AKU7KCOUAGYBOTKX6VF6FY7FNQDMNNFCA, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:51.218585386 (6h17m57.591164741s) [noise=true , address=45.91.116.9:4263 , piece_id=4L3UWFKV3AC7F4RPS7GJFXOO6DJNCPMFB5NWGJH6HXAFBGLYAMYA, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:51.290646027 (6h17m57.663225392s) [noise=true , address=104.36.82.2:4732 , piece_id=DBPJH5FSNFRPIBLP7HWWJFVVXYC47J7UGPCXFL7L3BD6BGFSJDFA, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:51.309329961 (6h17m57.681909316s) [noise=true , address=104.36.82.12:4150 , piece_id=GQ4GQQZHK3Y4KVVB2IUU4LEEBR4OJ2YSS5Z5DF7WHHDCYYSSGRYA, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:51.317895332 (6h17m57.690474687s) [noise=true , address=104.36.82.9:4007 , piece_id=R2IPGORJ3Z4FFCC52P5WMXYIA6O5V7DDZHHL4X64XV6J36SIWSBQ, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:52.525479047 (6h17m58.898058402s) [noise=true , address=45.91.116.9:4262 , piece_id=EXWTIYPESLJQK3CDK732WC63OZ7OJOQAUCBPGO3ZU54UBFUJ3TSA, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:09:18.157484198 (6h18m24.530063553s) [noise=true , address=77.90.31.13:4628 , piece_id=OAGGFQO2CORAZQV6ND3EERSICEOSHKK73Y3PUWLT45ZUWEWDHOFQ, seg_pos={0 0}, upload=5005]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: expected piece hash; context deadline exceeded; EOF Jul 19 05:09:18.157617684 (6h18m24.530197039s) [noise=true , address=77.90.31.15:4698 , piece_id=HNTVTQMVXDUMK4CMHKVUZ7DHM3MZL56TGXFR4FFFV2V3DCTLAVZQ, seg_pos={0 0}, upload=5005]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF Jul 19 05:09:18.157744796 (6h18m24.530324151s) [noise=true , address=77.90.31.11:4555 , piece_id=QGRXVFSKXDCLKSSSXEK4QYZEETWQGFZMQQBSTXNMAHY45RA6MAVQ, seg_pos={0 0}, upload=5005]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: write tcp 188.166.225.170:38288->77.90.31.11:4555: use of closed network connection; write tcp 188.166.225.170:38288->77.90.31.11:4555: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38288->77.90.31.11:4555: use of closed network connection Jul 19 05:09:18.157785745 (6h18m24.5303651s) [noise=true , address=77.90.31.13:4626 , piece_id=2QAZAYJWKJXTPY475RLOSPV3ZZTMB6OZLNYAKD7FCFV7T4WPC5ZQ, seg_pos={0 0}, upload=5005]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:58846->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:58846->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58846->77.90.31.13:4626: use of closed network connection Jul 19 05:09:18.157859036 (6h18m24.530438391s) [noise=true , address=77.90.31.12:4592 , piece_id=HS5YQKPTPUVHFZH3VZ7X2LAO2SVS6JZOTHTLJ6T47G3CLTZTR7UQ, seg_pos={0 0}, upload=5005]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:38220->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:38220->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38220->77.90.31.12:4592: use of closed network connection Jul 19 05:09:18.157898821 (6h18m24.530478176s) [seg_pos={0 0}, upload=5005]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 19 05:09:18.389974737 (6h18m24.762554092s) [upload=5005]: Done uploading segments... Jul 19 05:09:18.389994916 (6h18m24.762574271s) [upload=5005]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 15.087967561s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 5.946452522s FAIL (2025-07-19 05:09:39 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 4855s) [upload=4769]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 04:51:09.109709627 (6h0m15.482288982s) [upload=4769]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 04:51:09.556145753 (6h0m15.928725108s) [seg_pos={0 0}, upload=4769]: Begin upload segment... Jul 19 04:51:09.556194817 (6h0m15.928774172s) [seg_pos={0 0}, upload=4769]: Done begin upload segment. Jul 19 04:51:09.556223893 (6h0m15.928803248s) [noise=true , address=79.127.219.38:4023 , piece_id=PTPOZGLYV4MZTOFZNN3NCGYL7DFAST4PXKEUIVTMFL7HV4GZTCLQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556252197 (6h0m15.928831552s) [noise=true , address=45.91.95.4:4011 , piece_id=ZMBAS6KKHYVIRE6OBL2NFVCWMM4TKQ7J64BD3SYXIK54XCEDI2XA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556579383 (6h0m15.929158738s) [noise=true , address=45.91.116.6:4007 , piece_id=A2AFVTZBXSGBOE7XIM3B5HRN4STITFBXO6MZRJ365URNAAOZ3PSQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556854469 (6h0m15.929433824s) [noise=true , address=104.36.82.19:4423 , piece_id=PIJUHDUND36HQL67I33D7A3EULMJHS7N55MN4OFQJ4OWNQ42O47A, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557094107 (6h0m15.929673462s) [noise=true , address=45.91.116.9:4264 , piece_id=GHM5BKZKMT2QOHHBOS4FSPOWEWSFQRGRVVR4GSNDPHVIFRZHQL2A, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557336381 (6h0m15.929915736s) [noise=true , address=79.127.205.235:4660 , piece_id=UOUOGE2VP7PX7EMRZLPMCWFCYEZERM3DQMCQ6NG4J2K2DEIZR63Q, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557557885 (6h0m15.93013724s) [noise=true , address=79.127.205.246:4280 , piece_id=FLBOZI3UGTQXAQBLZ2F45Z3OEYL65F5PQBXRF2DQMZ7NGF7VDRPA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557762296 (6h0m15.930341651s) [noise=true , address=79.127.205.233:4622 , piece_id=6WWNYQTH5MGAJOEB7BXKZDJOMCAZXDW66J3OEANGDNJDNUMLZM7Q, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558014689 (6h0m15.930594044s) [noise=true , address=64.49.13.138:4482 , piece_id=MAW2M4US2OQNOS3WI67AA7NIBTJDE5T2JQTXKFPBLGVGXKVCFGTQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558215974 (6h0m15.930795319s) [noise=true , address=77.90.31.10:4546 , piece_id=R6LCA5WZOEBDURL5S3MEK22YPVYXX6EXSS4J4W57543HELYXOKZA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558409364 (6h0m15.930988719s) [noise=true , address=45.91.95.5:4019 , piece_id=YTSBCN2LPHEMRRAZIHJGVB7NBOTRXVEHD2TWUCTTLBNFFYEWWDKA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558623504 (6h0m15.931202859s) [noise=true , address=79.127.201.215:4357 , piece_id=75XWAG2VQWEYCWN2Q3WRIZSBI7LDOISZJPXQLQ2GIPNEN34VLSUQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558805162 (6h0m15.931384517s) [noise=true , address=45.91.116.14:4219 , piece_id=PDW6JPRHCVMHHOWIHT6JKV4CTSC3JY6NDKDVFZSO47OYNPZ7PGGA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558999373 (6h0m15.931578728s) [noise=true , address=77.90.31.11:4564 , piece_id=L2U7JFJRDQOFA2AYVAGTI4ZW4VABCXKMXWBIUWQO2QNNUUBJ74BA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559184768 (6h0m15.931764123s) [noise=true , address=77.90.31.16:4768 , piece_id=YNLSONFM6COZEVF6ZNQR2EA6V4LSQFZJONIXR4AFTNNMFGPV57JA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559368719 (6h0m15.931948074s) [noise=true , address=79.127.220.98:4432 , piece_id=2SUTFQ7FVDPAM2H4W3OLW6AVH3VJ33IJFE4L52SA3ET5LGBMOMPQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559549706 (6h0m15.932129061s) [noise=true , address=77.90.31.18:4813 , piece_id=VDOT7BB4O7XYJHW6BFTAYQZZYLZU5PAUXVJGSXJ7FDI4TH425COA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559727836 (6h0m15.932307191s) [noise=true , address=79.127.219.51:4211 , piece_id=HO5PLWBVU5YAYNYMN352NA73SM4WBJV4WZ7XLSQTCQ76UAL2TOJA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:10.663359087 (6h0m17.035938442s) [upload=4769]: Next returned nil segment Jul 19 04:51:10.663378664 (6h0m17.035958019s) [upload=4769]: Waiting for error group managing segments... Jul 19 04:51:10.962945479 (6h0m17.335524834s) [noise=true , address=79.127.205.233:4622 , piece_id=6WWNYQTH5MGAJOEB7BXKZDJOMCAZXDW66J3OEANGDNJDNUMLZM7Q, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:10.967689807 (6h0m17.340269152s) [noise=true , address=79.127.205.235:4660 , piece_id=UOUOGE2VP7PX7EMRZLPMCWFCYEZERM3DQMCQ6NG4J2K2DEIZR63Q, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:10.976968631 (6h0m17.349547986s) [noise=true , address=64.49.13.138:4482 , piece_id=MAW2M4US2OQNOS3WI67AA7NIBTJDE5T2JQTXKFPBLGVGXKVCFGTQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.048750700 (6h0m17.421330055s) [noise=true , address=79.127.205.246:4280 , piece_id=FLBOZI3UGTQXAQBLZ2F45Z3OEYL65F5PQBXRF2DQMZ7NGF7VDRPA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.221642403 (6h0m17.594221758s) [noise=true , address=79.127.220.98:4432 , piece_id=2SUTFQ7FVDPAM2H4W3OLW6AVH3VJ33IJFE4L52SA3ET5LGBMOMPQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.317070435 (6h0m17.68964979s) [noise=true , address=45.91.95.4:4011 , piece_id=ZMBAS6KKHYVIRE6OBL2NFVCWMM4TKQ7J64BD3SYXIK54XCEDI2XA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.324002660 (6h0m17.696582015s) [noise=true , address=45.91.95.5:4019 , piece_id=YTSBCN2LPHEMRRAZIHJGVB7NBOTRXVEHD2TWUCTTLBNFFYEWWDKA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.460976317 (6h0m17.833555672s) [noise=true , address=79.127.201.215:4357 , piece_id=75XWAG2VQWEYCWN2Q3WRIZSBI7LDOISZJPXQLQ2GIPNEN34VLSUQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.462584603 (6h0m17.835163958s) [noise=true , address=45.91.116.6:4007 , piece_id=A2AFVTZBXSGBOE7XIM3B5HRN4STITFBXO6MZRJ365URNAAOZ3PSQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.490392973 (6h0m17.862972328s) [noise=true , address=45.91.116.9:4264 , piece_id=GHM5BKZKMT2QOHHBOS4FSPOWEWSFQRGRVVR4GSNDPHVIFRZHQL2A, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.502040780 (6h0m17.874620135s) [noise=true , address=45.91.116.14:4219 , piece_id=PDW6JPRHCVMHHOWIHT6JKV4CTSC3JY6NDKDVFZSO47OYNPZ7PGGA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.600434096 (6h0m17.973013451s) [noise=true , address=79.127.219.51:4211 , piece_id=HO5PLWBVU5YAYNYMN352NA73SM4WBJV4WZ7XLSQTCQ76UAL2TOJA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.609008984 (6h0m17.981588339s) [noise=true , address=104.36.82.19:4423 , piece_id=PIJUHDUND36HQL67I33D7A3EULMJHS7N55MN4OFQJ4OWNQ42O47A, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.620825273 (6h0m17.993404628s) [noise=true , address=79.127.219.38:4023 , piece_id=PTPOZGLYV4MZTOFZNN3NCGYL7DFAST4PXKEUIVTMFL7HV4GZTCLQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.620838659 (6h0m17.993418014s) [seg_pos={0 0}, upload=4769]: Segment reached optimal threshold of 14 pieces. Jul 19 04:51:11.621067807 (6h0m17.993647163s) [noise=true , address=77.90.31.11:4564 , piece_id=L2U7JFJRDQOFA2AYVAGTI4ZW4VABCXKMXWBIUWQO2QNNUUBJ74BA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection; write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection Jul 19 04:51:11.621134234 (6h0m17.993713589s) [noise=true , address=77.90.31.10:4546 , piece_id=R6LCA5WZOEBDURL5S3MEK22YPVYXX6EXSS4J4W57543HELYXOKZA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK): protocol: read tcp 188.166.225.170:58476->77.90.31.10:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621174191 (6h0m17.993753546s) [noise=true , address=77.90.31.16:4768 , piece_id=YNLSONFM6COZEVF6ZNQR2EA6V4LSQFZJONIXR4AFTNNMFGPV57JA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UchewmtxBrtSZSpVAJFdZmcTsQVQxkjK9re6cgUGqDhyonQeF): protocol: read tcp 188.166.225.170:56326->77.90.31.16:4768: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621216842 (6h0m17.993796197s) [noise=true , address=77.90.31.18:4813 , piece_id=VDOT7BB4O7XYJHW6BFTAYQZZYLZU5PAUXVJGSXJ7FDI4TH425COA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128WthA1x9wp1dV1vkpGnoveZQpU9CSgDCZfw5GMQuVpCNSVAM5): protocol: read tcp 188.166.225.170:49704->77.90.31.18:4813: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621234917 (6h0m17.993814272s) [seg_pos={0 0}, upload=4769]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 19 04:51:11.621301444 (6h0m17.993880799s) [upload=4769]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 19 04:51:11.621305402 (6h0m17.993884758s) [upload=4769]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 19 04:51:11.923136970 (6h0m18.295716325s) [upload=4769]: Done uploading segments... Jul 19 04:51:11.923156627 (6h0m18.295735972s) [upload=4769]: Upload finished. err: <nil> Duration: 5.946481658s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [............................XX...X...X....X..X....XXXXX.X.X.] duration: 5.899609796s FAIL (2025-07-19 05:09:25 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== : <nil> Jul 19 05:08:29.800544823 (6h17m36.173124169s) [upload=4997]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 19 05:08:29.800548410 (6h17m36.173127765s) [upload=4997]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 19 05:08:30.148027600 (6h17m36.520606955s) [upload=4997]: Done uploading segments... Jul 19 05:08:30.148148401 (6h17m36.520727746s) [upload=4997]: Upload finished. err: <nil> Jul 19 05:08:48.807983386 (6h17m55.180562741s) [upload=5005]: Starting upload Jul 19 05:08:48.808044453 (6h17m55.180623808s) [upload=5005]: Done starting upload Jul 19 05:08:48.808175243 (6h17m55.180754598s) [upload=5005]: Uploading segments... Jul 19 05:08:48.808334198 (6h17m55.180913553s) [upload=5005]: Got next segment. Inline: false Jul 19 05:08:48.808392509 (6h17m55.180971874s) [upload=5005]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 05:08:48.808399823 (6h17m55.180979188s) [upload=5005]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 05:08:49.075515233 (6h17m55.448094578s) [seg_pos={0 0}, upload=5005]: Begin upload segment... Jul 19 05:08:49.075564547 (6h17m55.448143902s) [seg_pos={0 0}, upload=5005]: Done begin upload segment. Jul 19 05:08:49.075602881 (6h17m55.448182236s) [noise=true , address=77.90.31.11:4555 , piece_id=QGRXVFSKXDCLKSSSXEK4QYZEETWQGFZMQQBSTXNMAHY45RA6MAVQ, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.075612279 (6h17m55.448191634s) [noise=true , address=104.36.82.13:4186 , piece_id=PPK6XEQYJ3XV27XUJYKUL2MYAR5KUYKF5AHJS5EBXOK7PZWWM66Q, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.075964213 (6h17m55.448543568s) [noise=true , address=77.90.31.12:4592 , piece_id=HS5YQKPTPUVHFZH3VZ7X2LAO2SVS6JZOTHTLJ6T47G3CLTZTR7UQ, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.076122025 (6h17m55.44870137s) [noise=true , address=104.36.82.12:4150 , piece_id=GQ4GQQZHK3Y4KVVB2IUU4LEEBR4OJ2YSS5Z5DF7WHHDCYYSSGRYA, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.076272903 (6h17m55.448852258s) [noise=true , address=45.91.116.13:4154 , piece_id=OJO7VBXGBLV2SG2AQ6BPBP57SL75WV4UZSWC3O5HXD7ROBICSYGA, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.076533412 (6h17m55.449112767s) [noise=true , address=104.36.82.2:4732 , piece_id=DBPJH5FSNFRPIBLP7HWWJFVVXYC47J7UGPCXFL7L3BD6BGFSJDFA, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.076794962 (6h17m55.449374317s) [noise=true , address=45.91.116.9:4263 , piece_id=4L3UWFKV3AC7F4RPS7GJFXOO6DJNCPMFB5NWGJH6HXAFBGLYAMYA, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.077044971 (6h17m55.449624326s) [noise=true , address=104.36.82.16:4293 , piece_id=2NA7I27WVAWAUWDCZVYBJLNXGWIDNJMKZU7YFJZSXCQMCHKLCGXA, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.077296953 (6h17m55.449876308s) [noise=true , address=77.90.31.15:4698 , piece_id=HNTVTQMVXDUMK4CMHKVUZ7DHM3MZL56TGXFR4FFFV2V3DCTLAVZQ, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.077501123 (6h17m55.450080468s) [noise=true , address=104.36.82.9:4007 , piece_id=R2IPGORJ3Z4FFCC52P5WMXYIA6O5V7DDZHHL4X64XV6J36SIWSBQ, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.077695656 (6h17m55.450275001s) [noise=true , address=45.91.116.8:4048 , piece_id=QKXGR6YZSYTQIUXV54DRYQNWY352UFS2Z2VXWFJTD7YJYJZ24NCQ, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.077994558 (6h17m55.450573913s) [noise=true , address=104.36.82.14:4222 , piece_id=4F6ZDS2KMQNALT2YV6D3TAZLIBFUFBM5NNCBGLSE6L6PR5QMGPVA, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.078213187 (6h17m55.450792542s) [noise=true , address=45.91.116.4:4939 , piece_id=J2BIIQITH7R54K4ENE6AKU7KCOUAGYBOTKX6VF6FY7FNQDMNNFCA, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.078412077 (6h17m55.450991432s) [noise=true , address=77.90.31.13:4626 , piece_id=2QAZAYJWKJXTPY475RLOSPV3ZZTMB6OZLNYAKD7FCFV7T4WPC5ZQ, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.079649705 (6h17m55.45222906s) [noise=true , address=45.91.116.9:4262 , piece_id=EXWTIYPESLJQK3CDK732WC63OZ7OJOQAUCBPGO3ZU54UBFUJ3TSA, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.080903344 (6h17m55.453482699s) [noise=true , address=77.90.31.13:4628 , piece_id=OAGGFQO2CORAZQV6ND3EERSICEOSHKK73Y3PUWLT45ZUWEWDHOFQ, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:50.857952826 (6h17m57.230532181s) [upload=5005]: Next returned nil segment Jul 19 05:08:50.857969477 (6h17m57.230548832s) [upload=5005]: Waiting for error group managing segments... Jul 19 05:08:51.139405061 (6h17m57.511984416s) [noise=true , address=104.36.82.13:4186 , piece_id=PPK6XEQYJ3XV27XUJYKUL2MYAR5KUYKF5AHJS5EBXOK7PZWWM66Q, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:51.143987831 (6h17m57.516567176s) [noise=true , address=45.91.116.8:4048 , piece_id=QKXGR6YZSYTQIUXV54DRYQNWY352UFS2Z2VXWFJTD7YJYJZ24NCQ, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:51.148258443 (6h17m57.520837798s) [noise=true , address=104.36.82.16:4293 , piece_id=2NA7I27WVAWAUWDCZVYBJLNXGWIDNJMKZU7YFJZSXCQMCHKLCGXA, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:51.156796512 (6h17m57.529375867s) [noise=true , address=45.91.116.13:4154 , piece_id=OJO7VBXGBLV2SG2AQ6BPBP57SL75WV4UZSWC3O5HXD7ROBICSYGA, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:51.193908742 (6h17m57.566488098s) [noise=true , address=104.36.82.14:4222 , piece_id=4F6ZDS2KMQNALT2YV6D3TAZLIBFUFBM5NNCBGLSE6L6PR5QMGPVA, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:51.217498736 (6h17m57.590078091s) [noise=true , address=45.91.116.4:4939 , piece_id=J2BIIQITH7R54K4ENE6AKU7KCOUAGYBOTKX6VF6FY7FNQDMNNFCA, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:51.218585386 (6h17m57.591164741s) [noise=true , address=45.91.116.9:4263 , piece_id=4L3UWFKV3AC7F4RPS7GJFXOO6DJNCPMFB5NWGJH6HXAFBGLYAMYA, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:51.290646027 (6h17m57.663225392s) [noise=true , address=104.36.82.2:4732 , piece_id=DBPJH5FSNFRPIBLP7HWWJFVVXYC47J7UGPCXFL7L3BD6BGFSJDFA, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:51.309329961 (6h17m57.681909316s) [noise=true , address=104.36.82.12:4150 , piece_id=GQ4GQQZHK3Y4KVVB2IUU4LEEBR4OJ2YSS5Z5DF7WHHDCYYSSGRYA, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:51.317895332 (6h17m57.690474687s) [noise=true , address=104.36.82.9:4007 , piece_id=R2IPGORJ3Z4FFCC52P5WMXYIA6O5V7DDZHHL4X64XV6J36SIWSBQ, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:52.525479047 (6h17m58.898058402s) [noise=true , address=45.91.116.9:4262 , piece_id=EXWTIYPESLJQK3CDK732WC63OZ7OJOQAUCBPGO3ZU54UBFUJ3TSA, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:09:18.157484198 (6h18m24.530063553s) [noise=true , address=77.90.31.13:4628 , piece_id=OAGGFQO2CORAZQV6ND3EERSICEOSHKK73Y3PUWLT45ZUWEWDHOFQ, seg_pos={0 0}, upload=5005]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: expected piece hash; context deadline exceeded; EOF Jul 19 05:09:18.157617684 (6h18m24.530197039s) [noise=true , address=77.90.31.15:4698 , piece_id=HNTVTQMVXDUMK4CMHKVUZ7DHM3MZL56TGXFR4FFFV2V3DCTLAVZQ, seg_pos={0 0}, upload=5005]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF Jul 19 05:09:18.157744796 (6h18m24.530324151s) [noise=true , address=77.90.31.11:4555 , piece_id=QGRXVFSKXDCLKSSSXEK4QYZEETWQGFZMQQBSTXNMAHY45RA6MAVQ, seg_pos={0 0}, upload=5005]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: write tcp 188.166.225.170:38288->77.90.31.11:4555: use of closed network connection; write tcp 188.166.225.170:38288->77.90.31.11:4555: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38288->77.90.31.11:4555: use of closed network connection Jul 19 05:09:18.157785745 (6h18m24.5303651s) [noise=true , address=77.90.31.13:4626 , piece_id=2QAZAYJWKJXTPY475RLOSPV3ZZTMB6OZLNYAKD7FCFV7T4WPC5ZQ, seg_pos={0 0}, upload=5005]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:58846->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:58846->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58846->77.90.31.13:4626: use of closed network connection Jul 19 05:09:18.157859036 (6h18m24.530438391s) [noise=true , address=77.90.31.12:4592 , piece_id=HS5YQKPTPUVHFZH3VZ7X2LAO2SVS6JZOTHTLJ6T47G3CLTZTR7UQ, seg_pos={0 0}, upload=5005]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:38220->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:38220->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38220->77.90.31.12:4592: use of closed network connection Jul 19 05:09:18.157898821 (6h18m24.530478176s) [seg_pos={0 0}, upload=5005]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 19 05:09:18.389974737 (6h18m24.762554092s) [upload=5005]: Done uploading segments... Jul 19 05:09:18.389994916 (6h18m24.762574271s) [upload=5005]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.899642679s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 27.567721207s FAIL (2025-07-19 05:09:19 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 4855s) [upload=4769]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 04:51:09.109709627 (6h0m15.482288982s) [upload=4769]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 04:51:09.556145753 (6h0m15.928725108s) [seg_pos={0 0}, upload=4769]: Begin upload segment... Jul 19 04:51:09.556194817 (6h0m15.928774172s) [seg_pos={0 0}, upload=4769]: Done begin upload segment. Jul 19 04:51:09.556223893 (6h0m15.928803248s) [noise=true , address=79.127.219.38:4023 , piece_id=PTPOZGLYV4MZTOFZNN3NCGYL7DFAST4PXKEUIVTMFL7HV4GZTCLQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556252197 (6h0m15.928831552s) [noise=true , address=45.91.95.4:4011 , piece_id=ZMBAS6KKHYVIRE6OBL2NFVCWMM4TKQ7J64BD3SYXIK54XCEDI2XA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556579383 (6h0m15.929158738s) [noise=true , address=45.91.116.6:4007 , piece_id=A2AFVTZBXSGBOE7XIM3B5HRN4STITFBXO6MZRJ365URNAAOZ3PSQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556854469 (6h0m15.929433824s) [noise=true , address=104.36.82.19:4423 , piece_id=PIJUHDUND36HQL67I33D7A3EULMJHS7N55MN4OFQJ4OWNQ42O47A, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557094107 (6h0m15.929673462s) [noise=true , address=45.91.116.9:4264 , piece_id=GHM5BKZKMT2QOHHBOS4FSPOWEWSFQRGRVVR4GSNDPHVIFRZHQL2A, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557336381 (6h0m15.929915736s) [noise=true , address=79.127.205.235:4660 , piece_id=UOUOGE2VP7PX7EMRZLPMCWFCYEZERM3DQMCQ6NG4J2K2DEIZR63Q, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557557885 (6h0m15.93013724s) [noise=true , address=79.127.205.246:4280 , piece_id=FLBOZI3UGTQXAQBLZ2F45Z3OEYL65F5PQBXRF2DQMZ7NGF7VDRPA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557762296 (6h0m15.930341651s) [noise=true , address=79.127.205.233:4622 , piece_id=6WWNYQTH5MGAJOEB7BXKZDJOMCAZXDW66J3OEANGDNJDNUMLZM7Q, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558014689 (6h0m15.930594044s) [noise=true , address=64.49.13.138:4482 , piece_id=MAW2M4US2OQNOS3WI67AA7NIBTJDE5T2JQTXKFPBLGVGXKVCFGTQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558215974 (6h0m15.930795319s) [noise=true , address=77.90.31.10:4546 , piece_id=R6LCA5WZOEBDURL5S3MEK22YPVYXX6EXSS4J4W57543HELYXOKZA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558409364 (6h0m15.930988719s) [noise=true , address=45.91.95.5:4019 , piece_id=YTSBCN2LPHEMRRAZIHJGVB7NBOTRXVEHD2TWUCTTLBNFFYEWWDKA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558623504 (6h0m15.931202859s) [noise=true , address=79.127.201.215:4357 , piece_id=75XWAG2VQWEYCWN2Q3WRIZSBI7LDOISZJPXQLQ2GIPNEN34VLSUQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558805162 (6h0m15.931384517s) [noise=true , address=45.91.116.14:4219 , piece_id=PDW6JPRHCVMHHOWIHT6JKV4CTSC3JY6NDKDVFZSO47OYNPZ7PGGA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558999373 (6h0m15.931578728s) [noise=true , address=77.90.31.11:4564 , piece_id=L2U7JFJRDQOFA2AYVAGTI4ZW4VABCXKMXWBIUWQO2QNNUUBJ74BA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559184768 (6h0m15.931764123s) [noise=true , address=77.90.31.16:4768 , piece_id=YNLSONFM6COZEVF6ZNQR2EA6V4LSQFZJONIXR4AFTNNMFGPV57JA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559368719 (6h0m15.931948074s) [noise=true , address=79.127.220.98:4432 , piece_id=2SUTFQ7FVDPAM2H4W3OLW6AVH3VJ33IJFE4L52SA3ET5LGBMOMPQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559549706 (6h0m15.932129061s) [noise=true , address=77.90.31.18:4813 , piece_id=VDOT7BB4O7XYJHW6BFTAYQZZYLZU5PAUXVJGSXJ7FDI4TH425COA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559727836 (6h0m15.932307191s) [noise=true , address=79.127.219.51:4211 , piece_id=HO5PLWBVU5YAYNYMN352NA73SM4WBJV4WZ7XLSQTCQ76UAL2TOJA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:10.663359087 (6h0m17.035938442s) [upload=4769]: Next returned nil segment Jul 19 04:51:10.663378664 (6h0m17.035958019s) [upload=4769]: Waiting for error group managing segments... Jul 19 04:51:10.962945479 (6h0m17.335524834s) [noise=true , address=79.127.205.233:4622 , piece_id=6WWNYQTH5MGAJOEB7BXKZDJOMCAZXDW66J3OEANGDNJDNUMLZM7Q, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:10.967689807 (6h0m17.340269152s) [noise=true , address=79.127.205.235:4660 , piece_id=UOUOGE2VP7PX7EMRZLPMCWFCYEZERM3DQMCQ6NG4J2K2DEIZR63Q, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:10.976968631 (6h0m17.349547986s) [noise=true , address=64.49.13.138:4482 , piece_id=MAW2M4US2OQNOS3WI67AA7NIBTJDE5T2JQTXKFPBLGVGXKVCFGTQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.048750700 (6h0m17.421330055s) [noise=true , address=79.127.205.246:4280 , piece_id=FLBOZI3UGTQXAQBLZ2F45Z3OEYL65F5PQBXRF2DQMZ7NGF7VDRPA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.221642403 (6h0m17.594221758s) [noise=true , address=79.127.220.98:4432 , piece_id=2SUTFQ7FVDPAM2H4W3OLW6AVH3VJ33IJFE4L52SA3ET5LGBMOMPQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.317070435 (6h0m17.68964979s) [noise=true , address=45.91.95.4:4011 , piece_id=ZMBAS6KKHYVIRE6OBL2NFVCWMM4TKQ7J64BD3SYXIK54XCEDI2XA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.324002660 (6h0m17.696582015s) [noise=true , address=45.91.95.5:4019 , piece_id=YTSBCN2LPHEMRRAZIHJGVB7NBOTRXVEHD2TWUCTTLBNFFYEWWDKA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.460976317 (6h0m17.833555672s) [noise=true , address=79.127.201.215:4357 , piece_id=75XWAG2VQWEYCWN2Q3WRIZSBI7LDOISZJPXQLQ2GIPNEN34VLSUQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.462584603 (6h0m17.835163958s) [noise=true , address=45.91.116.6:4007 , piece_id=A2AFVTZBXSGBOE7XIM3B5HRN4STITFBXO6MZRJ365URNAAOZ3PSQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.490392973 (6h0m17.862972328s) [noise=true , address=45.91.116.9:4264 , piece_id=GHM5BKZKMT2QOHHBOS4FSPOWEWSFQRGRVVR4GSNDPHVIFRZHQL2A, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.502040780 (6h0m17.874620135s) [noise=true , address=45.91.116.14:4219 , piece_id=PDW6JPRHCVMHHOWIHT6JKV4CTSC3JY6NDKDVFZSO47OYNPZ7PGGA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.600434096 (6h0m17.973013451s) [noise=true , address=79.127.219.51:4211 , piece_id=HO5PLWBVU5YAYNYMN352NA73SM4WBJV4WZ7XLSQTCQ76UAL2TOJA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.609008984 (6h0m17.981588339s) [noise=true , address=104.36.82.19:4423 , piece_id=PIJUHDUND36HQL67I33D7A3EULMJHS7N55MN4OFQJ4OWNQ42O47A, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.620825273 (6h0m17.993404628s) [noise=true , address=79.127.219.38:4023 , piece_id=PTPOZGLYV4MZTOFZNN3NCGYL7DFAST4PXKEUIVTMFL7HV4GZTCLQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.620838659 (6h0m17.993418014s) [seg_pos={0 0}, upload=4769]: Segment reached optimal threshold of 14 pieces. Jul 19 04:51:11.621067807 (6h0m17.993647163s) [noise=true , address=77.90.31.11:4564 , piece_id=L2U7JFJRDQOFA2AYVAGTI4ZW4VABCXKMXWBIUWQO2QNNUUBJ74BA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection; write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection Jul 19 04:51:11.621134234 (6h0m17.993713589s) [noise=true , address=77.90.31.10:4546 , piece_id=R6LCA5WZOEBDURL5S3MEK22YPVYXX6EXSS4J4W57543HELYXOKZA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK): protocol: read tcp 188.166.225.170:58476->77.90.31.10:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621174191 (6h0m17.993753546s) [noise=true , address=77.90.31.16:4768 , piece_id=YNLSONFM6COZEVF6ZNQR2EA6V4LSQFZJONIXR4AFTNNMFGPV57JA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UchewmtxBrtSZSpVAJFdZmcTsQVQxkjK9re6cgUGqDhyonQeF): protocol: read tcp 188.166.225.170:56326->77.90.31.16:4768: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621216842 (6h0m17.993796197s) [noise=true , address=77.90.31.18:4813 , piece_id=VDOT7BB4O7XYJHW6BFTAYQZZYLZU5PAUXVJGSXJ7FDI4TH425COA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128WthA1x9wp1dV1vkpGnoveZQpU9CSgDCZfw5GMQuVpCNSVAM5): protocol: read tcp 188.166.225.170:49704->77.90.31.18:4813: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621234917 (6h0m17.993814272s) [seg_pos={0 0}, upload=4769]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 19 04:51:11.621301444 (6h0m17.993880799s) [upload=4769]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 19 04:51:11.621305402 (6h0m17.993884758s) [upload=4769]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 19 04:51:11.923136970 (6h0m18.295716325s) [upload=4769]: Done uploading segments... Jul 19 04:51:11.923156627 (6h0m18.295735972s) [upload=4769]: Upload finished. err: <nil> Duration: 27.567740253s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.............................XX...X...X....X..X....XXXXX.X.X] duration: 30.233523858s FAIL (2025-07-19 05:09:18 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== : <nil> Jul 19 05:08:29.800544823 (6h17m36.173124169s) [upload=4997]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 19 05:08:29.800548410 (6h17m36.173127765s) [upload=4997]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 19 05:08:30.148027600 (6h17m36.520606955s) [upload=4997]: Done uploading segments... Jul 19 05:08:30.148148401 (6h17m36.520727746s) [upload=4997]: Upload finished. err: <nil> Jul 19 05:08:48.807983386 (6h17m55.180562741s) [upload=5005]: Starting upload Jul 19 05:08:48.808044453 (6h17m55.180623808s) [upload=5005]: Done starting upload Jul 19 05:08:48.808175243 (6h17m55.180754598s) [upload=5005]: Uploading segments... Jul 19 05:08:48.808334198 (6h17m55.180913553s) [upload=5005]: Got next segment. Inline: false Jul 19 05:08:48.808392509 (6h17m55.180971874s) [upload=5005]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 05:08:48.808399823 (6h17m55.180979188s) [upload=5005]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 05:08:49.075515233 (6h17m55.448094578s) [seg_pos={0 0}, upload=5005]: Begin upload segment... Jul 19 05:08:49.075564547 (6h17m55.448143902s) [seg_pos={0 0}, upload=5005]: Done begin upload segment. Jul 19 05:08:49.075602881 (6h17m55.448182236s) [noise=true , address=77.90.31.11:4555 , piece_id=QGRXVFSKXDCLKSSSXEK4QYZEETWQGFZMQQBSTXNMAHY45RA6MAVQ, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.075612279 (6h17m55.448191634s) [noise=true , address=104.36.82.13:4186 , piece_id=PPK6XEQYJ3XV27XUJYKUL2MYAR5KUYKF5AHJS5EBXOK7PZWWM66Q, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.075964213 (6h17m55.448543568s) [noise=true , address=77.90.31.12:4592 , piece_id=HS5YQKPTPUVHFZH3VZ7X2LAO2SVS6JZOTHTLJ6T47G3CLTZTR7UQ, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.076122025 (6h17m55.44870137s) [noise=true , address=104.36.82.12:4150 , piece_id=GQ4GQQZHK3Y4KVVB2IUU4LEEBR4OJ2YSS5Z5DF7WHHDCYYSSGRYA, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.076272903 (6h17m55.448852258s) [noise=true , address=45.91.116.13:4154 , piece_id=OJO7VBXGBLV2SG2AQ6BPBP57SL75WV4UZSWC3O5HXD7ROBICSYGA, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.076533412 (6h17m55.449112767s) [noise=true , address=104.36.82.2:4732 , piece_id=DBPJH5FSNFRPIBLP7HWWJFVVXYC47J7UGPCXFL7L3BD6BGFSJDFA, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.076794962 (6h17m55.449374317s) [noise=true , address=45.91.116.9:4263 , piece_id=4L3UWFKV3AC7F4RPS7GJFXOO6DJNCPMFB5NWGJH6HXAFBGLYAMYA, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.077044971 (6h17m55.449624326s) [noise=true , address=104.36.82.16:4293 , piece_id=2NA7I27WVAWAUWDCZVYBJLNXGWIDNJMKZU7YFJZSXCQMCHKLCGXA, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.077296953 (6h17m55.449876308s) [noise=true , address=77.90.31.15:4698 , piece_id=HNTVTQMVXDUMK4CMHKVUZ7DHM3MZL56TGXFR4FFFV2V3DCTLAVZQ, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.077501123 (6h17m55.450080468s) [noise=true , address=104.36.82.9:4007 , piece_id=R2IPGORJ3Z4FFCC52P5WMXYIA6O5V7DDZHHL4X64XV6J36SIWSBQ, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.077695656 (6h17m55.450275001s) [noise=true , address=45.91.116.8:4048 , piece_id=QKXGR6YZSYTQIUXV54DRYQNWY352UFS2Z2VXWFJTD7YJYJZ24NCQ, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.077994558 (6h17m55.450573913s) [noise=true , address=104.36.82.14:4222 , piece_id=4F6ZDS2KMQNALT2YV6D3TAZLIBFUFBM5NNCBGLSE6L6PR5QMGPVA, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.078213187 (6h17m55.450792542s) [noise=true , address=45.91.116.4:4939 , piece_id=J2BIIQITH7R54K4ENE6AKU7KCOUAGYBOTKX6VF6FY7FNQDMNNFCA, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.078412077 (6h17m55.450991432s) [noise=true , address=77.90.31.13:4626 , piece_id=2QAZAYJWKJXTPY475RLOSPV3ZZTMB6OZLNYAKD7FCFV7T4WPC5ZQ, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.079649705 (6h17m55.45222906s) [noise=true , address=45.91.116.9:4262 , piece_id=EXWTIYPESLJQK3CDK732WC63OZ7OJOQAUCBPGO3ZU54UBFUJ3TSA, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:49.080903344 (6h17m55.453482699s) [noise=true , address=77.90.31.13:4628 , piece_id=OAGGFQO2CORAZQV6ND3EERSICEOSHKK73Y3PUWLT45ZUWEWDHOFQ, seg_pos={0 0}, upload=5005]: Uploading piece... Jul 19 05:08:50.857952826 (6h17m57.230532181s) [upload=5005]: Next returned nil segment Jul 19 05:08:50.857969477 (6h17m57.230548832s) [upload=5005]: Waiting for error group managing segments... Jul 19 05:08:51.139405061 (6h17m57.511984416s) [noise=true , address=104.36.82.13:4186 , piece_id=PPK6XEQYJ3XV27XUJYKUL2MYAR5KUYKF5AHJS5EBXOK7PZWWM66Q, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:51.143987831 (6h17m57.516567176s) [noise=true , address=45.91.116.8:4048 , piece_id=QKXGR6YZSYTQIUXV54DRYQNWY352UFS2Z2VXWFJTD7YJYJZ24NCQ, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:51.148258443 (6h17m57.520837798s) [noise=true , address=104.36.82.16:4293 , piece_id=2NA7I27WVAWAUWDCZVYBJLNXGWIDNJMKZU7YFJZSXCQMCHKLCGXA, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:51.156796512 (6h17m57.529375867s) [noise=true , address=45.91.116.13:4154 , piece_id=OJO7VBXGBLV2SG2AQ6BPBP57SL75WV4UZSWC3O5HXD7ROBICSYGA, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:51.193908742 (6h17m57.566488098s) [noise=true , address=104.36.82.14:4222 , piece_id=4F6ZDS2KMQNALT2YV6D3TAZLIBFUFBM5NNCBGLSE6L6PR5QMGPVA, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:51.217498736 (6h17m57.590078091s) [noise=true , address=45.91.116.4:4939 , piece_id=J2BIIQITH7R54K4ENE6AKU7KCOUAGYBOTKX6VF6FY7FNQDMNNFCA, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:51.218585386 (6h17m57.591164741s) [noise=true , address=45.91.116.9:4263 , piece_id=4L3UWFKV3AC7F4RPS7GJFXOO6DJNCPMFB5NWGJH6HXAFBGLYAMYA, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:51.290646027 (6h17m57.663225392s) [noise=true , address=104.36.82.2:4732 , piece_id=DBPJH5FSNFRPIBLP7HWWJFVVXYC47J7UGPCXFL7L3BD6BGFSJDFA, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:51.309329961 (6h17m57.681909316s) [noise=true , address=104.36.82.12:4150 , piece_id=GQ4GQQZHK3Y4KVVB2IUU4LEEBR4OJ2YSS5Z5DF7WHHDCYYSSGRYA, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:51.317895332 (6h17m57.690474687s) [noise=true , address=104.36.82.9:4007 , piece_id=R2IPGORJ3Z4FFCC52P5WMXYIA6O5V7DDZHHL4X64XV6J36SIWSBQ, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:08:52.525479047 (6h17m58.898058402s) [noise=true , address=45.91.116.9:4262 , piece_id=EXWTIYPESLJQK3CDK732WC63OZ7OJOQAUCBPGO3ZU54UBFUJ3TSA, seg_pos={0 0}, upload=5005]: Done uploading piece. err: <nil> Jul 19 05:09:18.157484198 (6h18m24.530063553s) [noise=true , address=77.90.31.13:4628 , piece_id=OAGGFQO2CORAZQV6ND3EERSICEOSHKK73Y3PUWLT45ZUWEWDHOFQ, seg_pos={0 0}, upload=5005]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: expected piece hash; context deadline exceeded; EOF Jul 19 05:09:18.157617684 (6h18m24.530197039s) [noise=true , address=77.90.31.15:4698 , piece_id=HNTVTQMVXDUMK4CMHKVUZ7DHM3MZL56TGXFR4FFFV2V3DCTLAVZQ, seg_pos={0 0}, upload=5005]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: expected piece hash; context deadline exceeded; EOF Jul 19 05:09:18.157744796 (6h18m24.530324151s) [noise=true , address=77.90.31.11:4555 , piece_id=QGRXVFSKXDCLKSSSXEK4QYZEETWQGFZMQQBSTXNMAHY45RA6MAVQ, seg_pos={0 0}, upload=5005]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: write tcp 188.166.225.170:38288->77.90.31.11:4555: use of closed network connection; write tcp 188.166.225.170:38288->77.90.31.11:4555: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38288->77.90.31.11:4555: use of closed network connection Jul 19 05:09:18.157785745 (6h18m24.5303651s) [noise=true , address=77.90.31.13:4626 , piece_id=2QAZAYJWKJXTPY475RLOSPV3ZZTMB6OZLNYAKD7FCFV7T4WPC5ZQ, seg_pos={0 0}, upload=5005]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:58846->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:58846->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58846->77.90.31.13:4626: use of closed network connection Jul 19 05:09:18.157859036 (6h18m24.530438391s) [noise=true , address=77.90.31.12:4592 , piece_id=HS5YQKPTPUVHFZH3VZ7X2LAO2SVS6JZOTHTLJ6T47G3CLTZTR7UQ, seg_pos={0 0}, upload=5005]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:38220->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:38220->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38220->77.90.31.12:4592: use of closed network connection Jul 19 05:09:18.157898821 (6h18m24.530478176s) [seg_pos={0 0}, upload=5005]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 19 05:09:18.389974737 (6h18m24.762554092s) [upload=5005]: Done uploading segments... Jul 19 05:09:18.389994916 (6h18m24.762574271s) [upload=5005]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.233549267s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [..............................XX...X...X....X..X....XXXXX.X.] duration: 22.433473572s FAIL (2025-07-19 05:08:32 UTC) orbiter warning: ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , upload=4978]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: read tcp 188.166.225.170:56432->77.90.31.21:4879: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 05:06:53.739102075 (6h16m0.11168143s) [seg_pos={0 0}, upload=4978]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 19 05:06:53.739197288 (6h16m0.111776673s) [upload=4978]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 19 05:06:53.739203930 (6h16m0.111783286s) [upload=4978]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 19 05:06:54.046241769 (6h16m0.418821124s) [upload=4978]: Done uploading segments... Jul 19 05:06:54.046260585 (6h16m0.41883994s) [upload=4978]: Upload finished. err: <nil> Jul 19 05:07:18.967786786 (6h16m25.340366131s) [upload=4987]: Starting upload Jul 19 05:07:18.967842954 (6h16m25.340422309s) [upload=4987]: Done starting upload Jul 19 05:07:18.967928819 (6h16m25.340508174s) [upload=4987]: Uploading segments... Jul 19 05:07:18.968231738 (6h16m25.340811094s) [upload=4987]: Got next segment. Inline: false Jul 19 05:07:18.968276884 (6h16m25.340856239s) [upload=4987]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 05:07:18.968282466 (6h16m25.340861821s) [upload=4987]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 05:07:19.221977372 (6h16m25.594556728s) [seg_pos={0 0}, upload=4987]: Begin upload segment... Jul 19 05:07:19.222023461 (6h16m25.594602816s) [seg_pos={0 0}, upload=4987]: Done begin upload segment. Jul 19 05:07:19.222042989 (6h16m25.594622334s) [noise=true , address=77.90.31.15:4699 , piece_id=AGUDMA4LGNU3DSZB42QLS7M4JEZ6N5O7PNOAUOMLIQJEGISI3W5Q, seg_pos={0 0}, upload=4987]: Uploading piece... Jul 19 05:07:19.222313176 (6h16m25.594892531s) [noise=true , address=77.90.31.15:4697 , piece_id=544MNCQQNU6A75WX2WIJQ6A3F4N4GCCSSX3C4UTT7WHH77QOCUHA, seg_pos={0 0}, upload=4987]: Uploading piece... Jul 19 05:07:19.222568183 (6h16m25.595147538s) [noise=true , address=104.36.82.4:4801 , piece_id=3O2Q25TEDWVGN5PKMUMVK2TGYGXKRDWPV3ZY32JL7HQCDO3FB6KA, seg_pos={0 0}, upload=4987]: Uploading piece... Jul 19 05:07:19.222804205 (6h16m25.59538356s) [noise=true , address=77.90.31.14:4663 , piece_id=6XQO4MRE2VYQHQAQR3BGOLYFPFDR3GNOQNO4NHNZJCQKGWVO6JAQ, seg_pos={0 0}, upload=4987]: Uploading piece... Jul 19 05:07:19.223029146 (6h16m25.595608501s) [noise=true , address=77.90.31.10:4520 , piece_id=FOH2YFXGUQ772SNJXDP7ZR7CF5LFZ3LGYKCIEOEM4LJHPSMRMYHQ, seg_pos={0 0}, upload=4987]: Uploading piece... Jul 19 05:07:19.223246592 (6h16m25.595825947s) [noise=true , address=45.91.116.6:4974 , piece_id=U3Z2FOR3EZI3P2MCADPDJL6JKKUSQODAYCD263AM22E2WMRIFX4A, seg_pos={0 0}, upload=4987]: Uploading piece... Jul 19 05:07:19.223512381 (6h16m25.596091736s) [noise=true , address=45.91.116.6:4973 , piece_id=HWJBA2OCLJKSN64HSRK7RULJHIQPVKIT3GWPK64Y32VUUYFEUTMQ, seg_pos={0 0}, upload=4987]: Uploading piece... Jul 19 05:07:19.223703747 (6h16m25.596283102s) [noise=true , address=77.90.31.17:4772 , piece_id=VP3JS4QYDXJ6R35MFRUK2XULQLGJJ23BSMM7IRWMVR3IISVEBLUQ, seg_pos={0 0}, upload=4987]: Uploading piece... Jul 19 05:07:19.223956139 (6h16m25.596535495s) [noise=true , address=77.90.31.12:4592 , piece_id=36WYINRSYTUIPPO76JUD7HAO6SX4NW25G37XPOU3YVT2QLACABIA, seg_pos={0 0}, upload=4987]: Uploading piece... Jul 19 05:07:19.224198243 (6h16m25.596777598s) [noise=true , address=104.36.82.5:4837 , piece_id=7YZBYKPWKCGARRJLE7WK3YRR536QC7BZZFUHYGCGQRTHRX3PTSPQ, seg_pos={0 0}, upload=4987]: Uploading piece... Jul 19 05:07:19.224392975 (6h16m25.59697233s) [noise=true , address=77.90.31.21:4878 , piece_id=PSLYCIRGFJ6RTL65HOEO3S2JDC737V5MJ6R3PGJJ3ODMDPNIHWCA, seg_pos={0 0}, upload=4987]: Uploading piece... Jul 19 05:07:19.224586385 (6h16m25.59716574s) [noise=true , address=77.90.31.18:4807 , piece_id=3S6MGZJ2IC7UVNLPWKJCPBCY23B2DLGIJ6ANTNPNB4UKDIW45Y7Q, seg_pos={0 0}, upload=4987]: Uploading piece... Jul 19 05:07:19.224772702 (6h16m25.597352057s) [noise=true , address=45.91.116.13:4154 , piece_id=JFXGESP7N4C4ZQ7TWKTOQVVKQ3APV6Y3C6NBN7OIKXYJMJ5RPHWA, seg_pos={0 0}, upload=4987]: Uploading piece... Jul 19 05:07:19.225006720 (6h16m25.597586075s) [noise=true , address=104.36.82.7:4909 , piece_id=YCN3RDTWQZLZNYCVOQ334JY2XZJFAXDXZ5CZBWOTN4I2XI5U3FMA, seg_pos={0 0}, upload=4987]: Uploading piece... Jul 19 05:07:19.225222964 (6h16m25.597802319s) [noise=true , address=45.91.116.9:4261 , piece_id=VGBFN7ZKPHS5S4SDWUHEKY6F7LS6JTVRY3ITUAGXHYWNWQMLJOIA, seg_pos={0 0}, upload=4987]: Uploading piece... Jul 19 05:07:19.225416364 (6h16m25.597995719s) [noise=true , address=45.91.116.4:4939 , piece_id=BLQFFPUNAGDF7FCF2JPLBPPUAEWI7FMG4DKD7TWRMPQTYX3LEUAQ, seg_pos={0 0}, upload=4987]: Uploading piece... Jul 19 05:07:20.980351225 (6h16m27.35293058s) [upload=4987]: Next returned nil segment Jul 19 05:07:20.980380271 (6h16m27.352959626s) [upload=4987]: Waiting for error group managing segments... Jul 19 05:07:21.267793974 (6h16m27.640373329s) [noise=true , address=104.36.82.4:4801 , piece_id=3O2Q25TEDWVGN5PKMUMVK2TGYGXKRDWPV3ZY32JL7HQCDO3FB6KA, seg_pos={0 0}, upload=4987]: Done uploading piece. err: <nil> Jul 19 05:07:21.285583227 (6h16m27.658162583s) [noise=true , address=45.91.116.13:4154 , piece_id=JFXGESP7N4C4ZQ7TWKTOQVVKQ3APV6Y3C6NBN7OIKXYJMJ5RPHWA, seg_pos={0 0}, upload=4987]: Done uploading piece. err: <nil> Jul 19 05:07:21.286980110 (6h16m27.659559465s) [noise=true , address=45.91.116.9:4261 , piece_id=VGBFN7ZKPHS5S4SDWUHEKY6F7LS6JTVRY3ITUAGXHYWNWQMLJOIA, seg_pos={0 0}, upload=4987]: Done uploading piece. err: <nil> Jul 19 05:07:21.359982162 (6h16m27.732561517s) [noise=true , address=104.36.82.5:4837 , piece_id=7YZBYKPWKCGARRJLE7WK3YRR536QC7BZZFUHYGCGQRTHRX3PTSPQ, seg_pos={0 0}, upload=4987]: Done uploading piece. err: <nil> Jul 19 05:07:21.363819385 (6h16m27.73639874s) [noise=true , address=45.91.116.6:4974 , piece_id=U3Z2FOR3EZI3P2MCADPDJL6JKKUSQODAYCD263AM22E2WMRIFX4A, seg_pos={0 0}, upload=4987]: Done uploading piece. err: <nil> Jul 19 05:07:21.372194202 (6h16m27.744773557s) [noise=true , address=45.91.116.6:4973 , piece_id=HWJBA2OCLJKSN64HSRK7RULJHIQPVKIT3GWPK64Y32VUUYFEUTMQ, seg_pos={0 0}, upload=4987]: Done uploading piece. err: <nil> Jul 19 05:07:21.375296359 (6h16m27.747875714s) [noise=true , address=104.36.82.7:4909 , piece_id=YCN3RDTWQZLZNYCVOQ334JY2XZJFAXDXZ5CZBWOTN4I2XI5U3FMA, seg_pos={0 0}, upload=4987]: Done uploading piece. err: <nil> Jul 19 05:07:21.381291291 (6h16m27.753870646s) [noise=true , address=45.91.116.4:4939 , piece_id=BLQFFPUNAGDF7FCF2JPLBPPUAEWI7FMG4DKD7TWRMPQTYX3LEUAQ, seg_pos={0 0}, upload=4987]: Done uploading piece. err: <nil> Jul 19 05:07:32.446172843 (6h16m38.818752198s) [noise=true , address=77.90.31.17:4772 , piece_id=VP3JS4QYDXJ6R35MFRUK2XULQLGJJ23BSMM7IRWMVR3IISVEBLUQ, seg_pos={0 0}, upload=4987]: Done uploading piece. err: <nil> Jul 19 05:07:37.950304448 (6h16m44.322883803s) [noise=true , address=77.90.31.10:4520 , piece_id=FOH2YFXGUQ772SNJXDP7ZR7CF5LFZ3LGYKCIEOEM4LJHPSMRMYHQ, seg_pos={0 0}, upload=4987]: Done uploading piece. err: <nil> Jul 19 05:07:38.051458593 (6h16m44.424037949s) [noise=true , address=77.90.31.15:4697 , piece_id=544MNCQQNU6A75WX2WIJQ6A3F4N4GCCSSX3C4UTT7WHH77QOCUHA, seg_pos={0 0}, upload=4987]: Done uploading piece. err: <nil> Jul 19 05:07:38.236231036 (6h16m44.608810401s) [noise=true , address=77.90.31.14:4663 , piece_id=6XQO4MRE2VYQHQAQR3BGOLYFPFDR3GNOQNO4NHNZJCQKGWVO6JAQ, seg_pos={0 0}, upload=4987]: Done uploading piece. err: <nil> Jul 19 05:07:48.143474611 (6h16m54.516053967s) [noise=true , address=77.90.31.21:4878 , piece_id=PSLYCIRGFJ6RTL65HOEO3S2JDC737V5MJ6R3PGJJ3ODMDPNIHWCA, seg_pos={0 0}, upload=4987]: Done uploading piece. err: <nil> Jul 19 05:07:48.143499839 (6h16m54.516079194s) [seg_pos={0 0}, upload=4987]: Segment reached optimal threshold of 13 pieces. Jul 19 05:07:48.143654295 (6h16m54.51623364s) [noise=true , address=77.90.31.18:4807 , piece_id=3S6MGZJ2IC7UVNLPWKJCPBCY23B2DLGIJ6ANTNPNB4UKDIW45Y7Q, seg_pos={0 0}, upload=4987]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: expected piece hash; context canceled; EOF Jul 19 05:07:48.143694843 (6h16m54.516274198s) [noise=true , address=77.90.31.12:4592 , piece_id=36WYINRSYTUIPPO76JUD7HAO6SX4NW25G37XPOU3YVT2QLACABIA, seg_pos={0 0}, upload=4987]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3): protocol: expected piece hash; context canceled; EOF Jul 19 05:07:48.143734959 (6h16m54.516314314s) [noise=true , address=77.90.31.15:4699 , piece_id=AGUDMA4LGNU3DSZB42QLS7M4JEZ6N5O7PNOAUOMLIQJEGISI3W5Q, seg_pos={0 0}, upload=4987]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M): protocol: write tcp 188.166.225.170:57034->77.90.31.15:4699: use of closed network connection; write tcp 188.166.225.170:57034->77.90.31.15:4699: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57034->77.90.31.15:4699: use of closed network connection Jul 19 05:07:48.143753064 (6h16m54.516332419s) [seg_pos={0 0}, upload=4987]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 19 05:07:48.143813039 (6h16m54.516392384s) [upload=4987]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 19 05:07:48.143816736 (6h16m54.516396091s) [upload=4987]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 19 05:07:48.532152020 (6h16m54.904731375s) [upload=4987]: Done uploading segments... Jul 19 05:07:48.532172109 (6h16m54.904751464s) [upload=4987]: Upload finished. err: metaclient: context deadline exceeded Duration: 22.433497338s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 11.956095934s FAIL (2025-07-19 05:08:02 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 4855s) [upload=4769]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 04:51:09.109709627 (6h0m15.482288982s) [upload=4769]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 04:51:09.556145753 (6h0m15.928725108s) [seg_pos={0 0}, upload=4769]: Begin upload segment... Jul 19 04:51:09.556194817 (6h0m15.928774172s) [seg_pos={0 0}, upload=4769]: Done begin upload segment. Jul 19 04:51:09.556223893 (6h0m15.928803248s) [noise=true , address=79.127.219.38:4023 , piece_id=PTPOZGLYV4MZTOFZNN3NCGYL7DFAST4PXKEUIVTMFL7HV4GZTCLQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556252197 (6h0m15.928831552s) [noise=true , address=45.91.95.4:4011 , piece_id=ZMBAS6KKHYVIRE6OBL2NFVCWMM4TKQ7J64BD3SYXIK54XCEDI2XA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556579383 (6h0m15.929158738s) [noise=true , address=45.91.116.6:4007 , piece_id=A2AFVTZBXSGBOE7XIM3B5HRN4STITFBXO6MZRJ365URNAAOZ3PSQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556854469 (6h0m15.929433824s) [noise=true , address=104.36.82.19:4423 , piece_id=PIJUHDUND36HQL67I33D7A3EULMJHS7N55MN4OFQJ4OWNQ42O47A, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557094107 (6h0m15.929673462s) [noise=true , address=45.91.116.9:4264 , piece_id=GHM5BKZKMT2QOHHBOS4FSPOWEWSFQRGRVVR4GSNDPHVIFRZHQL2A, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557336381 (6h0m15.929915736s) [noise=true , address=79.127.205.235:4660 , piece_id=UOUOGE2VP7PX7EMRZLPMCWFCYEZERM3DQMCQ6NG4J2K2DEIZR63Q, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557557885 (6h0m15.93013724s) [noise=true , address=79.127.205.246:4280 , piece_id=FLBOZI3UGTQXAQBLZ2F45Z3OEYL65F5PQBXRF2DQMZ7NGF7VDRPA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557762296 (6h0m15.930341651s) [noise=true , address=79.127.205.233:4622 , piece_id=6WWNYQTH5MGAJOEB7BXKZDJOMCAZXDW66J3OEANGDNJDNUMLZM7Q, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558014689 (6h0m15.930594044s) [noise=true , address=64.49.13.138:4482 , piece_id=MAW2M4US2OQNOS3WI67AA7NIBTJDE5T2JQTXKFPBLGVGXKVCFGTQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558215974 (6h0m15.930795319s) [noise=true , address=77.90.31.10:4546 , piece_id=R6LCA5WZOEBDURL5S3MEK22YPVYXX6EXSS4J4W57543HELYXOKZA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558409364 (6h0m15.930988719s) [noise=true , address=45.91.95.5:4019 , piece_id=YTSBCN2LPHEMRRAZIHJGVB7NBOTRXVEHD2TWUCTTLBNFFYEWWDKA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558623504 (6h0m15.931202859s) [noise=true , address=79.127.201.215:4357 , piece_id=75XWAG2VQWEYCWN2Q3WRIZSBI7LDOISZJPXQLQ2GIPNEN34VLSUQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558805162 (6h0m15.931384517s) [noise=true , address=45.91.116.14:4219 , piece_id=PDW6JPRHCVMHHOWIHT6JKV4CTSC3JY6NDKDVFZSO47OYNPZ7PGGA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558999373 (6h0m15.931578728s) [noise=true , address=77.90.31.11:4564 , piece_id=L2U7JFJRDQOFA2AYVAGTI4ZW4VABCXKMXWBIUWQO2QNNUUBJ74BA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559184768 (6h0m15.931764123s) [noise=true , address=77.90.31.16:4768 , piece_id=YNLSONFM6COZEVF6ZNQR2EA6V4LSQFZJONIXR4AFTNNMFGPV57JA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559368719 (6h0m15.931948074s) [noise=true , address=79.127.220.98:4432 , piece_id=2SUTFQ7FVDPAM2H4W3OLW6AVH3VJ33IJFE4L52SA3ET5LGBMOMPQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559549706 (6h0m15.932129061s) [noise=true , address=77.90.31.18:4813 , piece_id=VDOT7BB4O7XYJHW6BFTAYQZZYLZU5PAUXVJGSXJ7FDI4TH425COA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559727836 (6h0m15.932307191s) [noise=true , address=79.127.219.51:4211 , piece_id=HO5PLWBVU5YAYNYMN352NA73SM4WBJV4WZ7XLSQTCQ76UAL2TOJA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:10.663359087 (6h0m17.035938442s) [upload=4769]: Next returned nil segment Jul 19 04:51:10.663378664 (6h0m17.035958019s) [upload=4769]: Waiting for error group managing segments... Jul 19 04:51:10.962945479 (6h0m17.335524834s) [noise=true , address=79.127.205.233:4622 , piece_id=6WWNYQTH5MGAJOEB7BXKZDJOMCAZXDW66J3OEANGDNJDNUMLZM7Q, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:10.967689807 (6h0m17.340269152s) [noise=true , address=79.127.205.235:4660 , piece_id=UOUOGE2VP7PX7EMRZLPMCWFCYEZERM3DQMCQ6NG4J2K2DEIZR63Q, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:10.976968631 (6h0m17.349547986s) [noise=true , address=64.49.13.138:4482 , piece_id=MAW2M4US2OQNOS3WI67AA7NIBTJDE5T2JQTXKFPBLGVGXKVCFGTQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.048750700 (6h0m17.421330055s) [noise=true , address=79.127.205.246:4280 , piece_id=FLBOZI3UGTQXAQBLZ2F45Z3OEYL65F5PQBXRF2DQMZ7NGF7VDRPA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.221642403 (6h0m17.594221758s) [noise=true , address=79.127.220.98:4432 , piece_id=2SUTFQ7FVDPAM2H4W3OLW6AVH3VJ33IJFE4L52SA3ET5LGBMOMPQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.317070435 (6h0m17.68964979s) [noise=true , address=45.91.95.4:4011 , piece_id=ZMBAS6KKHYVIRE6OBL2NFVCWMM4TKQ7J64BD3SYXIK54XCEDI2XA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.324002660 (6h0m17.696582015s) [noise=true , address=45.91.95.5:4019 , piece_id=YTSBCN2LPHEMRRAZIHJGVB7NBOTRXVEHD2TWUCTTLBNFFYEWWDKA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.460976317 (6h0m17.833555672s) [noise=true , address=79.127.201.215:4357 , piece_id=75XWAG2VQWEYCWN2Q3WRIZSBI7LDOISZJPXQLQ2GIPNEN34VLSUQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.462584603 (6h0m17.835163958s) [noise=true , address=45.91.116.6:4007 , piece_id=A2AFVTZBXSGBOE7XIM3B5HRN4STITFBXO6MZRJ365URNAAOZ3PSQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.490392973 (6h0m17.862972328s) [noise=true , address=45.91.116.9:4264 , piece_id=GHM5BKZKMT2QOHHBOS4FSPOWEWSFQRGRVVR4GSNDPHVIFRZHQL2A, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.502040780 (6h0m17.874620135s) [noise=true , address=45.91.116.14:4219 , piece_id=PDW6JPRHCVMHHOWIHT6JKV4CTSC3JY6NDKDVFZSO47OYNPZ7PGGA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.600434096 (6h0m17.973013451s) [noise=true , address=79.127.219.51:4211 , piece_id=HO5PLWBVU5YAYNYMN352NA73SM4WBJV4WZ7XLSQTCQ76UAL2TOJA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.609008984 (6h0m17.981588339s) [noise=true , address=104.36.82.19:4423 , piece_id=PIJUHDUND36HQL67I33D7A3EULMJHS7N55MN4OFQJ4OWNQ42O47A, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.620825273 (6h0m17.993404628s) [noise=true , address=79.127.219.38:4023 , piece_id=PTPOZGLYV4MZTOFZNN3NCGYL7DFAST4PXKEUIVTMFL7HV4GZTCLQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.620838659 (6h0m17.993418014s) [seg_pos={0 0}, upload=4769]: Segment reached optimal threshold of 14 pieces. Jul 19 04:51:11.621067807 (6h0m17.993647163s) [noise=true , address=77.90.31.11:4564 , piece_id=L2U7JFJRDQOFA2AYVAGTI4ZW4VABCXKMXWBIUWQO2QNNUUBJ74BA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection; write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection Jul 19 04:51:11.621134234 (6h0m17.993713589s) [noise=true , address=77.90.31.10:4546 , piece_id=R6LCA5WZOEBDURL5S3MEK22YPVYXX6EXSS4J4W57543HELYXOKZA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK): protocol: read tcp 188.166.225.170:58476->77.90.31.10:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621174191 (6h0m17.993753546s) [noise=true , address=77.90.31.16:4768 , piece_id=YNLSONFM6COZEVF6ZNQR2EA6V4LSQFZJONIXR4AFTNNMFGPV57JA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UchewmtxBrtSZSpVAJFdZmcTsQVQxkjK9re6cgUGqDhyonQeF): protocol: read tcp 188.166.225.170:56326->77.90.31.16:4768: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621216842 (6h0m17.993796197s) [noise=true , address=77.90.31.18:4813 , piece_id=VDOT7BB4O7XYJHW6BFTAYQZZYLZU5PAUXVJGSXJ7FDI4TH425COA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128WthA1x9wp1dV1vkpGnoveZQpU9CSgDCZfw5GMQuVpCNSVAM5): protocol: read tcp 188.166.225.170:49704->77.90.31.18:4813: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621234917 (6h0m17.993814272s) [seg_pos={0 0}, upload=4769]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 19 04:51:11.621301444 (6h0m17.993880799s) [upload=4769]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 19 04:51:11.621305402 (6h0m17.993884758s) [upload=4769]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 19 04:51:11.923136970 (6h0m18.295716325s) [upload=4769]: Done uploading segments... Jul 19 04:51:11.923156627 (6h0m18.295735972s) [upload=4769]: Upload finished. err: <nil> Duration: 11.956122926s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [...............................XX...X...X....X..X....XXXXX.X] duration: 30.226194444s FAIL (2025-07-19 05:07:48 UTC) orbiter warning: ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , upload=4978]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: read tcp 188.166.225.170:56432->77.90.31.21:4879: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 05:06:53.739102075 (6h16m0.11168143s) [seg_pos={0 0}, upload=4978]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 19 05:06:53.739197288 (6h16m0.111776673s) [upload=4978]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 19 05:06:53.739203930 (6h16m0.111783286s) [upload=4978]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 19 05:06:54.046241769 (6h16m0.418821124s) [upload=4978]: Done uploading segments... Jul 19 05:06:54.046260585 (6h16m0.41883994s) [upload=4978]: Upload finished. err: <nil> Jul 19 05:07:18.967786786 (6h16m25.340366131s) [upload=4987]: Starting upload Jul 19 05:07:18.967842954 (6h16m25.340422309s) [upload=4987]: Done starting upload Jul 19 05:07:18.967928819 (6h16m25.340508174s) [upload=4987]: Uploading segments... Jul 19 05:07:18.968231738 (6h16m25.340811094s) [upload=4987]: Got next segment. Inline: false Jul 19 05:07:18.968276884 (6h16m25.340856239s) [upload=4987]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 05:07:18.968282466 (6h16m25.340861821s) [upload=4987]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 05:07:19.221977372 (6h16m25.594556728s) [seg_pos={0 0}, upload=4987]: Begin upload segment... Jul 19 05:07:19.222023461 (6h16m25.594602816s) [seg_pos={0 0}, upload=4987]: Done begin upload segment. Jul 19 05:07:19.222042989 (6h16m25.594622334s) [noise=true , address=77.90.31.15:4699 , piece_id=AGUDMA4LGNU3DSZB42QLS7M4JEZ6N5O7PNOAUOMLIQJEGISI3W5Q, seg_pos={0 0}, upload=4987]: Uploading piece... Jul 19 05:07:19.222313176 (6h16m25.594892531s) [noise=true , address=77.90.31.15:4697 , piece_id=544MNCQQNU6A75WX2WIJQ6A3F4N4GCCSSX3C4UTT7WHH77QOCUHA, seg_pos={0 0}, upload=4987]: Uploading piece... Jul 19 05:07:19.222568183 (6h16m25.595147538s) [noise=true , address=104.36.82.4:4801 , piece_id=3O2Q25TEDWVGN5PKMUMVK2TGYGXKRDWPV3ZY32JL7HQCDO3FB6KA, seg_pos={0 0}, upload=4987]: Uploading piece... Jul 19 05:07:19.222804205 (6h16m25.59538356s) [noise=true , address=77.90.31.14:4663 , piece_id=6XQO4MRE2VYQHQAQR3BGOLYFPFDR3GNOQNO4NHNZJCQKGWVO6JAQ, seg_pos={0 0}, upload=4987]: Uploading piece... Jul 19 05:07:19.223029146 (6h16m25.595608501s) [noise=true , address=77.90.31.10:4520 , piece_id=FOH2YFXGUQ772SNJXDP7ZR7CF5LFZ3LGYKCIEOEM4LJHPSMRMYHQ, seg_pos={0 0}, upload=4987]: Uploading piece... Jul 19 05:07:19.223246592 (6h16m25.595825947s) [noise=true , address=45.91.116.6:4974 , piece_id=U3Z2FOR3EZI3P2MCADPDJL6JKKUSQODAYCD263AM22E2WMRIFX4A, seg_pos={0 0}, upload=4987]: Uploading piece... Jul 19 05:07:19.223512381 (6h16m25.596091736s) [noise=true , address=45.91.116.6:4973 , piece_id=HWJBA2OCLJKSN64HSRK7RULJHIQPVKIT3GWPK64Y32VUUYFEUTMQ, seg_pos={0 0}, upload=4987]: Uploading piece... Jul 19 05:07:19.223703747 (6h16m25.596283102s) [noise=true , address=77.90.31.17:4772 , piece_id=VP3JS4QYDXJ6R35MFRUK2XULQLGJJ23BSMM7IRWMVR3IISVEBLUQ, seg_pos={0 0}, upload=4987]: Uploading piece... Jul 19 05:07:19.223956139 (6h16m25.596535495s) [noise=true , address=77.90.31.12:4592 , piece_id=36WYINRSYTUIPPO76JUD7HAO6SX4NW25G37XPOU3YVT2QLACABIA, seg_pos={0 0}, upload=4987]: Uploading piece... Jul 19 05:07:19.224198243 (6h16m25.596777598s) [noise=true , address=104.36.82.5:4837 , piece_id=7YZBYKPWKCGARRJLE7WK3YRR536QC7BZZFUHYGCGQRTHRX3PTSPQ, seg_pos={0 0}, upload=4987]: Uploading piece... Jul 19 05:07:19.224392975 (6h16m25.59697233s) [noise=true , address=77.90.31.21:4878 , piece_id=PSLYCIRGFJ6RTL65HOEO3S2JDC737V5MJ6R3PGJJ3ODMDPNIHWCA, seg_pos={0 0}, upload=4987]: Uploading piece... Jul 19 05:07:19.224586385 (6h16m25.59716574s) [noise=true , address=77.90.31.18:4807 , piece_id=3S6MGZJ2IC7UVNLPWKJCPBCY23B2DLGIJ6ANTNPNB4UKDIW45Y7Q, seg_pos={0 0}, upload=4987]: Uploading piece... Jul 19 05:07:19.224772702 (6h16m25.597352057s) [noise=true , address=45.91.116.13:4154 , piece_id=JFXGESP7N4C4ZQ7TWKTOQVVKQ3APV6Y3C6NBN7OIKXYJMJ5RPHWA, seg_pos={0 0}, upload=4987]: Uploading piece... Jul 19 05:07:19.225006720 (6h16m25.597586075s) [noise=true , address=104.36.82.7:4909 , piece_id=YCN3RDTWQZLZNYCVOQ334JY2XZJFAXDXZ5CZBWOTN4I2XI5U3FMA, seg_pos={0 0}, upload=4987]: Uploading piece... Jul 19 05:07:19.225222964 (6h16m25.597802319s) [noise=true , address=45.91.116.9:4261 , piece_id=VGBFN7ZKPHS5S4SDWUHEKY6F7LS6JTVRY3ITUAGXHYWNWQMLJOIA, seg_pos={0 0}, upload=4987]: Uploading piece... Jul 19 05:07:19.225416364 (6h16m25.597995719s) [noise=true , address=45.91.116.4:4939 , piece_id=BLQFFPUNAGDF7FCF2JPLBPPUAEWI7FMG4DKD7TWRMPQTYX3LEUAQ, seg_pos={0 0}, upload=4987]: Uploading piece... Jul 19 05:07:20.980351225 (6h16m27.35293058s) [upload=4987]: Next returned nil segment Jul 19 05:07:20.980380271 (6h16m27.352959626s) [upload=4987]: Waiting for error group managing segments... Jul 19 05:07:21.267793974 (6h16m27.640373329s) [noise=true , address=104.36.82.4:4801 , piece_id=3O2Q25TEDWVGN5PKMUMVK2TGYGXKRDWPV3ZY32JL7HQCDO3FB6KA, seg_pos={0 0}, upload=4987]: Done uploading piece. err: <nil> Jul 19 05:07:21.285583227 (6h16m27.658162583s) [noise=true , address=45.91.116.13:4154 , piece_id=JFXGESP7N4C4ZQ7TWKTOQVVKQ3APV6Y3C6NBN7OIKXYJMJ5RPHWA, seg_pos={0 0}, upload=4987]: Done uploading piece. err: <nil> Jul 19 05:07:21.286980110 (6h16m27.659559465s) [noise=true , address=45.91.116.9:4261 , piece_id=VGBFN7ZKPHS5S4SDWUHEKY6F7LS6JTVRY3ITUAGXHYWNWQMLJOIA, seg_pos={0 0}, upload=4987]: Done uploading piece. err: <nil> Jul 19 05:07:21.359982162 (6h16m27.732561517s) [noise=true , address=104.36.82.5:4837 , piece_id=7YZBYKPWKCGARRJLE7WK3YRR536QC7BZZFUHYGCGQRTHRX3PTSPQ, seg_pos={0 0}, upload=4987]: Done uploading piece. err: <nil> Jul 19 05:07:21.363819385 (6h16m27.73639874s) [noise=true , address=45.91.116.6:4974 , piece_id=U3Z2FOR3EZI3P2MCADPDJL6JKKUSQODAYCD263AM22E2WMRIFX4A, seg_pos={0 0}, upload=4987]: Done uploading piece. err: <nil> Jul 19 05:07:21.372194202 (6h16m27.744773557s) [noise=true , address=45.91.116.6:4973 , piece_id=HWJBA2OCLJKSN64HSRK7RULJHIQPVKIT3GWPK64Y32VUUYFEUTMQ, seg_pos={0 0}, upload=4987]: Done uploading piece. err: <nil> Jul 19 05:07:21.375296359 (6h16m27.747875714s) [noise=true , address=104.36.82.7:4909 , piece_id=YCN3RDTWQZLZNYCVOQ334JY2XZJFAXDXZ5CZBWOTN4I2XI5U3FMA, seg_pos={0 0}, upload=4987]: Done uploading piece. err: <nil> Jul 19 05:07:21.381291291 (6h16m27.753870646s) [noise=true , address=45.91.116.4:4939 , piece_id=BLQFFPUNAGDF7FCF2JPLBPPUAEWI7FMG4DKD7TWRMPQTYX3LEUAQ, seg_pos={0 0}, upload=4987]: Done uploading piece. err: <nil> Jul 19 05:07:32.446172843 (6h16m38.818752198s) [noise=true , address=77.90.31.17:4772 , piece_id=VP3JS4QYDXJ6R35MFRUK2XULQLGJJ23BSMM7IRWMVR3IISVEBLUQ, seg_pos={0 0}, upload=4987]: Done uploading piece. err: <nil> Jul 19 05:07:37.950304448 (6h16m44.322883803s) [noise=true , address=77.90.31.10:4520 , piece_id=FOH2YFXGUQ772SNJXDP7ZR7CF5LFZ3LGYKCIEOEM4LJHPSMRMYHQ, seg_pos={0 0}, upload=4987]: Done uploading piece. err: <nil> Jul 19 05:07:38.051458593 (6h16m44.424037949s) [noise=true , address=77.90.31.15:4697 , piece_id=544MNCQQNU6A75WX2WIJQ6A3F4N4GCCSSX3C4UTT7WHH77QOCUHA, seg_pos={0 0}, upload=4987]: Done uploading piece. err: <nil> Jul 19 05:07:38.236231036 (6h16m44.608810401s) [noise=true , address=77.90.31.14:4663 , piece_id=6XQO4MRE2VYQHQAQR3BGOLYFPFDR3GNOQNO4NHNZJCQKGWVO6JAQ, seg_pos={0 0}, upload=4987]: Done uploading piece. err: <nil> Jul 19 05:07:48.143474611 (6h16m54.516053967s) [noise=true , address=77.90.31.21:4878 , piece_id=PSLYCIRGFJ6RTL65HOEO3S2JDC737V5MJ6R3PGJJ3ODMDPNIHWCA, seg_pos={0 0}, upload=4987]: Done uploading piece. err: <nil> Jul 19 05:07:48.143499839 (6h16m54.516079194s) [seg_pos={0 0}, upload=4987]: Segment reached optimal threshold of 13 pieces. Jul 19 05:07:48.143654295 (6h16m54.51623364s) [noise=true , address=77.90.31.18:4807 , piece_id=3S6MGZJ2IC7UVNLPWKJCPBCY23B2DLGIJ6ANTNPNB4UKDIW45Y7Q, seg_pos={0 0}, upload=4987]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS): protocol: expected piece hash; context canceled; EOF Jul 19 05:07:48.143694843 (6h16m54.516274198s) [noise=true , address=77.90.31.12:4592 , piece_id=36WYINRSYTUIPPO76JUD7HAO6SX4NW25G37XPOU3YVT2QLACABIA, seg_pos={0 0}, upload=4987]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3): protocol: expected piece hash; context canceled; EOF Jul 19 05:07:48.143734959 (6h16m54.516314314s) [noise=true , address=77.90.31.15:4699 , piece_id=AGUDMA4LGNU3DSZB42QLS7M4JEZ6N5O7PNOAUOMLIQJEGISI3W5Q, seg_pos={0 0}, upload=4987]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M): protocol: write tcp 188.166.225.170:57034->77.90.31.15:4699: use of closed network connection; write tcp 188.166.225.170:57034->77.90.31.15:4699: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57034->77.90.31.15:4699: use of closed network connection Jul 19 05:07:48.143753064 (6h16m54.516332419s) [seg_pos={0 0}, upload=4987]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 19 05:07:48.143813039 (6h16m54.516392384s) [upload=4987]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 19 05:07:48.143816736 (6h16m54.516396091s) [upload=4987]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 19 05:07:48.532152020 (6h16m54.904731375s) [upload=4987]: Done uploading segments... Jul 19 05:07:48.532172109 (6h16m54.904751464s) [upload=4987]: Upload finished. err: metaclient: context deadline exceeded Duration: 30.226217619s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 13.318795052s FAIL (2025-07-19 05:07:23 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 4855s) [upload=4769]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 04:51:09.109709627 (6h0m15.482288982s) [upload=4769]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 04:51:09.556145753 (6h0m15.928725108s) [seg_pos={0 0}, upload=4769]: Begin upload segment... Jul 19 04:51:09.556194817 (6h0m15.928774172s) [seg_pos={0 0}, upload=4769]: Done begin upload segment. Jul 19 04:51:09.556223893 (6h0m15.928803248s) [noise=true , address=79.127.219.38:4023 , piece_id=PTPOZGLYV4MZTOFZNN3NCGYL7DFAST4PXKEUIVTMFL7HV4GZTCLQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556252197 (6h0m15.928831552s) [noise=true , address=45.91.95.4:4011 , piece_id=ZMBAS6KKHYVIRE6OBL2NFVCWMM4TKQ7J64BD3SYXIK54XCEDI2XA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556579383 (6h0m15.929158738s) [noise=true , address=45.91.116.6:4007 , piece_id=A2AFVTZBXSGBOE7XIM3B5HRN4STITFBXO6MZRJ365URNAAOZ3PSQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556854469 (6h0m15.929433824s) [noise=true , address=104.36.82.19:4423 , piece_id=PIJUHDUND36HQL67I33D7A3EULMJHS7N55MN4OFQJ4OWNQ42O47A, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557094107 (6h0m15.929673462s) [noise=true , address=45.91.116.9:4264 , piece_id=GHM5BKZKMT2QOHHBOS4FSPOWEWSFQRGRVVR4GSNDPHVIFRZHQL2A, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557336381 (6h0m15.929915736s) [noise=true , address=79.127.205.235:4660 , piece_id=UOUOGE2VP7PX7EMRZLPMCWFCYEZERM3DQMCQ6NG4J2K2DEIZR63Q, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557557885 (6h0m15.93013724s) [noise=true , address=79.127.205.246:4280 , piece_id=FLBOZI3UGTQXAQBLZ2F45Z3OEYL65F5PQBXRF2DQMZ7NGF7VDRPA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557762296 (6h0m15.930341651s) [noise=true , address=79.127.205.233:4622 , piece_id=6WWNYQTH5MGAJOEB7BXKZDJOMCAZXDW66J3OEANGDNJDNUMLZM7Q, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558014689 (6h0m15.930594044s) [noise=true , address=64.49.13.138:4482 , piece_id=MAW2M4US2OQNOS3WI67AA7NIBTJDE5T2JQTXKFPBLGVGXKVCFGTQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558215974 (6h0m15.930795319s) [noise=true , address=77.90.31.10:4546 , piece_id=R6LCA5WZOEBDURL5S3MEK22YPVYXX6EXSS4J4W57543HELYXOKZA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558409364 (6h0m15.930988719s) [noise=true , address=45.91.95.5:4019 , piece_id=YTSBCN2LPHEMRRAZIHJGVB7NBOTRXVEHD2TWUCTTLBNFFYEWWDKA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558623504 (6h0m15.931202859s) [noise=true , address=79.127.201.215:4357 , piece_id=75XWAG2VQWEYCWN2Q3WRIZSBI7LDOISZJPXQLQ2GIPNEN34VLSUQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558805162 (6h0m15.931384517s) [noise=true , address=45.91.116.14:4219 , piece_id=PDW6JPRHCVMHHOWIHT6JKV4CTSC3JY6NDKDVFZSO47OYNPZ7PGGA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558999373 (6h0m15.931578728s) [noise=true , address=77.90.31.11:4564 , piece_id=L2U7JFJRDQOFA2AYVAGTI4ZW4VABCXKMXWBIUWQO2QNNUUBJ74BA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559184768 (6h0m15.931764123s) [noise=true , address=77.90.31.16:4768 , piece_id=YNLSONFM6COZEVF6ZNQR2EA6V4LSQFZJONIXR4AFTNNMFGPV57JA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559368719 (6h0m15.931948074s) [noise=true , address=79.127.220.98:4432 , piece_id=2SUTFQ7FVDPAM2H4W3OLW6AVH3VJ33IJFE4L52SA3ET5LGBMOMPQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559549706 (6h0m15.932129061s) [noise=true , address=77.90.31.18:4813 , piece_id=VDOT7BB4O7XYJHW6BFTAYQZZYLZU5PAUXVJGSXJ7FDI4TH425COA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559727836 (6h0m15.932307191s) [noise=true , address=79.127.219.51:4211 , piece_id=HO5PLWBVU5YAYNYMN352NA73SM4WBJV4WZ7XLSQTCQ76UAL2TOJA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:10.663359087 (6h0m17.035938442s) [upload=4769]: Next returned nil segment Jul 19 04:51:10.663378664 (6h0m17.035958019s) [upload=4769]: Waiting for error group managing segments... Jul 19 04:51:10.962945479 (6h0m17.335524834s) [noise=true , address=79.127.205.233:4622 , piece_id=6WWNYQTH5MGAJOEB7BXKZDJOMCAZXDW66J3OEANGDNJDNUMLZM7Q, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:10.967689807 (6h0m17.340269152s) [noise=true , address=79.127.205.235:4660 , piece_id=UOUOGE2VP7PX7EMRZLPMCWFCYEZERM3DQMCQ6NG4J2K2DEIZR63Q, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:10.976968631 (6h0m17.349547986s) [noise=true , address=64.49.13.138:4482 , piece_id=MAW2M4US2OQNOS3WI67AA7NIBTJDE5T2JQTXKFPBLGVGXKVCFGTQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.048750700 (6h0m17.421330055s) [noise=true , address=79.127.205.246:4280 , piece_id=FLBOZI3UGTQXAQBLZ2F45Z3OEYL65F5PQBXRF2DQMZ7NGF7VDRPA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.221642403 (6h0m17.594221758s) [noise=true , address=79.127.220.98:4432 , piece_id=2SUTFQ7FVDPAM2H4W3OLW6AVH3VJ33IJFE4L52SA3ET5LGBMOMPQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.317070435 (6h0m17.68964979s) [noise=true , address=45.91.95.4:4011 , piece_id=ZMBAS6KKHYVIRE6OBL2NFVCWMM4TKQ7J64BD3SYXIK54XCEDI2XA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.324002660 (6h0m17.696582015s) [noise=true , address=45.91.95.5:4019 , piece_id=YTSBCN2LPHEMRRAZIHJGVB7NBOTRXVEHD2TWUCTTLBNFFYEWWDKA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.460976317 (6h0m17.833555672s) [noise=true , address=79.127.201.215:4357 , piece_id=75XWAG2VQWEYCWN2Q3WRIZSBI7LDOISZJPXQLQ2GIPNEN34VLSUQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.462584603 (6h0m17.835163958s) [noise=true , address=45.91.116.6:4007 , piece_id=A2AFVTZBXSGBOE7XIM3B5HRN4STITFBXO6MZRJ365URNAAOZ3PSQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.490392973 (6h0m17.862972328s) [noise=true , address=45.91.116.9:4264 , piece_id=GHM5BKZKMT2QOHHBOS4FSPOWEWSFQRGRVVR4GSNDPHVIFRZHQL2A, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.502040780 (6h0m17.874620135s) [noise=true , address=45.91.116.14:4219 , piece_id=PDW6JPRHCVMHHOWIHT6JKV4CTSC3JY6NDKDVFZSO47OYNPZ7PGGA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.600434096 (6h0m17.973013451s) [noise=true , address=79.127.219.51:4211 , piece_id=HO5PLWBVU5YAYNYMN352NA73SM4WBJV4WZ7XLSQTCQ76UAL2TOJA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.609008984 (6h0m17.981588339s) [noise=true , address=104.36.82.19:4423 , piece_id=PIJUHDUND36HQL67I33D7A3EULMJHS7N55MN4OFQJ4OWNQ42O47A, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.620825273 (6h0m17.993404628s) [noise=true , address=79.127.219.38:4023 , piece_id=PTPOZGLYV4MZTOFZNN3NCGYL7DFAST4PXKEUIVTMFL7HV4GZTCLQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.620838659 (6h0m17.993418014s) [seg_pos={0 0}, upload=4769]: Segment reached optimal threshold of 14 pieces. Jul 19 04:51:11.621067807 (6h0m17.993647163s) [noise=true , address=77.90.31.11:4564 , piece_id=L2U7JFJRDQOFA2AYVAGTI4ZW4VABCXKMXWBIUWQO2QNNUUBJ74BA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection; write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection Jul 19 04:51:11.621134234 (6h0m17.993713589s) [noise=true , address=77.90.31.10:4546 , piece_id=R6LCA5WZOEBDURL5S3MEK22YPVYXX6EXSS4J4W57543HELYXOKZA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK): protocol: read tcp 188.166.225.170:58476->77.90.31.10:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621174191 (6h0m17.993753546s) [noise=true , address=77.90.31.16:4768 , piece_id=YNLSONFM6COZEVF6ZNQR2EA6V4LSQFZJONIXR4AFTNNMFGPV57JA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UchewmtxBrtSZSpVAJFdZmcTsQVQxkjK9re6cgUGqDhyonQeF): protocol: read tcp 188.166.225.170:56326->77.90.31.16:4768: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621216842 (6h0m17.993796197s) [noise=true , address=77.90.31.18:4813 , piece_id=VDOT7BB4O7XYJHW6BFTAYQZZYLZU5PAUXVJGSXJ7FDI4TH425COA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128WthA1x9wp1dV1vkpGnoveZQpU9CSgDCZfw5GMQuVpCNSVAM5): protocol: read tcp 188.166.225.170:49704->77.90.31.18:4813: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621234917 (6h0m17.993814272s) [seg_pos={0 0}, upload=4769]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 19 04:51:11.621301444 (6h0m17.993880799s) [upload=4769]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 19 04:51:11.621305402 (6h0m17.993884758s) [upload=4769]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 19 04:51:11.923136970 (6h0m18.295716325s) [upload=4769]: Done uploading segments... Jul 19 04:51:11.923156627 (6h0m18.295735972s) [upload=4769]: Upload finished. err: <nil> Duration: 13.318817465s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 24.940191671s FAIL (2025-07-19 05:07:09 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 4855s) [upload=4769]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 04:51:09.109709627 (6h0m15.482288982s) [upload=4769]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 04:51:09.556145753 (6h0m15.928725108s) [seg_pos={0 0}, upload=4769]: Begin upload segment... Jul 19 04:51:09.556194817 (6h0m15.928774172s) [seg_pos={0 0}, upload=4769]: Done begin upload segment. Jul 19 04:51:09.556223893 (6h0m15.928803248s) [noise=true , address=79.127.219.38:4023 , piece_id=PTPOZGLYV4MZTOFZNN3NCGYL7DFAST4PXKEUIVTMFL7HV4GZTCLQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556252197 (6h0m15.928831552s) [noise=true , address=45.91.95.4:4011 , piece_id=ZMBAS6KKHYVIRE6OBL2NFVCWMM4TKQ7J64BD3SYXIK54XCEDI2XA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556579383 (6h0m15.929158738s) [noise=true , address=45.91.116.6:4007 , piece_id=A2AFVTZBXSGBOE7XIM3B5HRN4STITFBXO6MZRJ365URNAAOZ3PSQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556854469 (6h0m15.929433824s) [noise=true , address=104.36.82.19:4423 , piece_id=PIJUHDUND36HQL67I33D7A3EULMJHS7N55MN4OFQJ4OWNQ42O47A, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557094107 (6h0m15.929673462s) [noise=true , address=45.91.116.9:4264 , piece_id=GHM5BKZKMT2QOHHBOS4FSPOWEWSFQRGRVVR4GSNDPHVIFRZHQL2A, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557336381 (6h0m15.929915736s) [noise=true , address=79.127.205.235:4660 , piece_id=UOUOGE2VP7PX7EMRZLPMCWFCYEZERM3DQMCQ6NG4J2K2DEIZR63Q, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557557885 (6h0m15.93013724s) [noise=true , address=79.127.205.246:4280 , piece_id=FLBOZI3UGTQXAQBLZ2F45Z3OEYL65F5PQBXRF2DQMZ7NGF7VDRPA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557762296 (6h0m15.930341651s) [noise=true , address=79.127.205.233:4622 , piece_id=6WWNYQTH5MGAJOEB7BXKZDJOMCAZXDW66J3OEANGDNJDNUMLZM7Q, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558014689 (6h0m15.930594044s) [noise=true , address=64.49.13.138:4482 , piece_id=MAW2M4US2OQNOS3WI67AA7NIBTJDE5T2JQTXKFPBLGVGXKVCFGTQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558215974 (6h0m15.930795319s) [noise=true , address=77.90.31.10:4546 , piece_id=R6LCA5WZOEBDURL5S3MEK22YPVYXX6EXSS4J4W57543HELYXOKZA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558409364 (6h0m15.930988719s) [noise=true , address=45.91.95.5:4019 , piece_id=YTSBCN2LPHEMRRAZIHJGVB7NBOTRXVEHD2TWUCTTLBNFFYEWWDKA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558623504 (6h0m15.931202859s) [noise=true , address=79.127.201.215:4357 , piece_id=75XWAG2VQWEYCWN2Q3WRIZSBI7LDOISZJPXQLQ2GIPNEN34VLSUQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558805162 (6h0m15.931384517s) [noise=true , address=45.91.116.14:4219 , piece_id=PDW6JPRHCVMHHOWIHT6JKV4CTSC3JY6NDKDVFZSO47OYNPZ7PGGA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558999373 (6h0m15.931578728s) [noise=true , address=77.90.31.11:4564 , piece_id=L2U7JFJRDQOFA2AYVAGTI4ZW4VABCXKMXWBIUWQO2QNNUUBJ74BA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559184768 (6h0m15.931764123s) [noise=true , address=77.90.31.16:4768 , piece_id=YNLSONFM6COZEVF6ZNQR2EA6V4LSQFZJONIXR4AFTNNMFGPV57JA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559368719 (6h0m15.931948074s) [noise=true , address=79.127.220.98:4432 , piece_id=2SUTFQ7FVDPAM2H4W3OLW6AVH3VJ33IJFE4L52SA3ET5LGBMOMPQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559549706 (6h0m15.932129061s) [noise=true , address=77.90.31.18:4813 , piece_id=VDOT7BB4O7XYJHW6BFTAYQZZYLZU5PAUXVJGSXJ7FDI4TH425COA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559727836 (6h0m15.932307191s) [noise=true , address=79.127.219.51:4211 , piece_id=HO5PLWBVU5YAYNYMN352NA73SM4WBJV4WZ7XLSQTCQ76UAL2TOJA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:10.663359087 (6h0m17.035938442s) [upload=4769]: Next returned nil segment Jul 19 04:51:10.663378664 (6h0m17.035958019s) [upload=4769]: Waiting for error group managing segments... Jul 19 04:51:10.962945479 (6h0m17.335524834s) [noise=true , address=79.127.205.233:4622 , piece_id=6WWNYQTH5MGAJOEB7BXKZDJOMCAZXDW66J3OEANGDNJDNUMLZM7Q, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:10.967689807 (6h0m17.340269152s) [noise=true , address=79.127.205.235:4660 , piece_id=UOUOGE2VP7PX7EMRZLPMCWFCYEZERM3DQMCQ6NG4J2K2DEIZR63Q, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:10.976968631 (6h0m17.349547986s) [noise=true , address=64.49.13.138:4482 , piece_id=MAW2M4US2OQNOS3WI67AA7NIBTJDE5T2JQTXKFPBLGVGXKVCFGTQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.048750700 (6h0m17.421330055s) [noise=true , address=79.127.205.246:4280 , piece_id=FLBOZI3UGTQXAQBLZ2F45Z3OEYL65F5PQBXRF2DQMZ7NGF7VDRPA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.221642403 (6h0m17.594221758s) [noise=true , address=79.127.220.98:4432 , piece_id=2SUTFQ7FVDPAM2H4W3OLW6AVH3VJ33IJFE4L52SA3ET5LGBMOMPQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.317070435 (6h0m17.68964979s) [noise=true , address=45.91.95.4:4011 , piece_id=ZMBAS6KKHYVIRE6OBL2NFVCWMM4TKQ7J64BD3SYXIK54XCEDI2XA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.324002660 (6h0m17.696582015s) [noise=true , address=45.91.95.5:4019 , piece_id=YTSBCN2LPHEMRRAZIHJGVB7NBOTRXVEHD2TWUCTTLBNFFYEWWDKA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.460976317 (6h0m17.833555672s) [noise=true , address=79.127.201.215:4357 , piece_id=75XWAG2VQWEYCWN2Q3WRIZSBI7LDOISZJPXQLQ2GIPNEN34VLSUQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.462584603 (6h0m17.835163958s) [noise=true , address=45.91.116.6:4007 , piece_id=A2AFVTZBXSGBOE7XIM3B5HRN4STITFBXO6MZRJ365URNAAOZ3PSQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.490392973 (6h0m17.862972328s) [noise=true , address=45.91.116.9:4264 , piece_id=GHM5BKZKMT2QOHHBOS4FSPOWEWSFQRGRVVR4GSNDPHVIFRZHQL2A, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.502040780 (6h0m17.874620135s) [noise=true , address=45.91.116.14:4219 , piece_id=PDW6JPRHCVMHHOWIHT6JKV4CTSC3JY6NDKDVFZSO47OYNPZ7PGGA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.600434096 (6h0m17.973013451s) [noise=true , address=79.127.219.51:4211 , piece_id=HO5PLWBVU5YAYNYMN352NA73SM4WBJV4WZ7XLSQTCQ76UAL2TOJA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.609008984 (6h0m17.981588339s) [noise=true , address=104.36.82.19:4423 , piece_id=PIJUHDUND36HQL67I33D7A3EULMJHS7N55MN4OFQJ4OWNQ42O47A, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.620825273 (6h0m17.993404628s) [noise=true , address=79.127.219.38:4023 , piece_id=PTPOZGLYV4MZTOFZNN3NCGYL7DFAST4PXKEUIVTMFL7HV4GZTCLQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.620838659 (6h0m17.993418014s) [seg_pos={0 0}, upload=4769]: Segment reached optimal threshold of 14 pieces. Jul 19 04:51:11.621067807 (6h0m17.993647163s) [noise=true , address=77.90.31.11:4564 , piece_id=L2U7JFJRDQOFA2AYVAGTI4ZW4VABCXKMXWBIUWQO2QNNUUBJ74BA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection; write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection Jul 19 04:51:11.621134234 (6h0m17.993713589s) [noise=true , address=77.90.31.10:4546 , piece_id=R6LCA5WZOEBDURL5S3MEK22YPVYXX6EXSS4J4W57543HELYXOKZA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK): protocol: read tcp 188.166.225.170:58476->77.90.31.10:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621174191 (6h0m17.993753546s) [noise=true , address=77.90.31.16:4768 , piece_id=YNLSONFM6COZEVF6ZNQR2EA6V4LSQFZJONIXR4AFTNNMFGPV57JA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UchewmtxBrtSZSpVAJFdZmcTsQVQxkjK9re6cgUGqDhyonQeF): protocol: read tcp 188.166.225.170:56326->77.90.31.16:4768: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621216842 (6h0m17.993796197s) [noise=true , address=77.90.31.18:4813 , piece_id=VDOT7BB4O7XYJHW6BFTAYQZZYLZU5PAUXVJGSXJ7FDI4TH425COA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128WthA1x9wp1dV1vkpGnoveZQpU9CSgDCZfw5GMQuVpCNSVAM5): protocol: read tcp 188.166.225.170:49704->77.90.31.18:4813: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621234917 (6h0m17.993814272s) [seg_pos={0 0}, upload=4769]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 19 04:51:11.621301444 (6h0m17.993880799s) [upload=4769]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 19 04:51:11.621305402 (6h0m17.993884758s) [upload=4769]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 19 04:51:11.923136970 (6h0m18.295716325s) [upload=4769]: Done uploading segments... Jul 19 04:51:11.923156627 (6h0m18.295735972s) [upload=4769]: Upload finished. err: <nil> Duration: 24.940215587s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [................................XX...X...X....X..X....XXXXX.] duration: 5.956742044s FAIL (2025-07-19 05:06:56 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ection; write tcp 188.166.225.170:36188->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36188->77.90.31.10:4517: use of closed network connection Jul 19 05:05:41.847565883 (6h14m48.220145238s) [seg_pos={0 0}, upload=4958]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 19 05:05:42.084038342 (6h14m48.456617697s) [upload=4958]: Done uploading segments... Jul 19 05:05:42.084064663 (6h14m48.456644018s) [upload=4958]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Jul 19 05:06:10.656276029 (6h15m17.028855385s) [upload=4971]: Starting upload Jul 19 05:06:10.656326216 (6h15m17.028905571s) [upload=4971]: Done starting upload Jul 19 05:06:10.656517231 (6h15m17.029096576s) [upload=4971]: Uploading segments... Jul 19 05:06:10.656546377 (6h15m17.029125732s) [upload=4971]: Got next segment. Inline: false Jul 19 05:06:10.656572427 (6h15m17.029151782s) [upload=4971]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 05:06:10.656576695 (6h15m17.02915604s) [upload=4971]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 05:06:10.894140643 (6h15m17.266719998s) [seg_pos={0 0}, upload=4971]: Begin upload segment... Jul 19 05:06:10.894202762 (6h15m17.266782118s) [seg_pos={0 0}, upload=4971]: Done begin upload segment. Jul 19 05:06:10.894225636 (6h15m17.266804991s) [noise=true , address=45.91.116.11:4119 , piece_id=WOSNNCJJCFOVHU5LPDJJJVRFZL4JI5GGXR6PDE5I6FMWTYOHXMNQ, seg_pos={0 0}, upload=4971]: Uploading piece... Jul 19 05:06:10.894239603 (6h15m17.266818968s) [noise=true , address=77.90.31.13:4627 , piece_id=W7OKPZLWR4ETITJJJQYEC5MATEVYKTOESKIW3T45AR7X7OW2PTIQ, seg_pos={0 0}, upload=4971]: Uploading piece... Jul 19 05:06:10.894613498 (6h15m17.267192853s) [noise=true , address=45.91.116.12:4225 , piece_id=FEAORHV5P6FOTKFQTL57LFNJBOEXKKW6YXBV4EGPY7ENYICF4HZQ, seg_pos={0 0}, upload=4971]: Uploading piece... Jul 19 05:06:10.895028152 (6h15m17.267607507s) [noise=true , address=77.90.31.10:4520 , piece_id=U7J4BEWUBZXQGQ4VVGVUE7ZL4YXU5CWQF4NTDQKQBW3R2KT3ICBQ, seg_pos={0 0}, upload=4971]: Uploading piece... Jul 19 05:06:10.895080372 (6h15m17.267659727s) [noise=true , address=77.90.31.10:4519 , piece_id=JXXKI2ULFDUJYSL33MAUEI73QCM5F3L6LUCKHJYQGV6P2LWENLJA, seg_pos={0 0}, upload=4971]: Uploading piece... Jul 19 05:06:10.895305061 (6h15m17.267884416s) [noise=true , address=45.91.116.12:4226 , piece_id=2VYSGVPZDHXFGRZJITNDPCPYU4C35VYVWWJU3SE3VZ5P7OEJE46Q, seg_pos={0 0}, upload=4971]: Uploading piece... Jul 19 05:06:10.895411916 (6h15m17.267991271s) [noise=true , address=45.91.116.8:4047 , piece_id=DTD2PZR72MQBK72KJLJSNHPH3I4WO3EOSTC2HMDSHZYZ66PAP6CA, seg_pos={0 0}, upload=4971]: Uploading piece... Jul 19 05:06:10.895576090 (6h15m17.268155445s) [noise=true , address=77.90.31.12:4590 , piece_id=CQVAKS3RXC6JCDRKA4R3AKKBRVB4KOVW4MCRGGTAE473WQ7PN4PQ, seg_pos={0 0}, upload=4971]: Uploading piece... Jul 19 05:06:10.895721348 (6h15m17.268300703s) [noise=true , address=45.91.116.4:4938 , piece_id=UJKB2LQPMR2ZCABG6R5KVVRW7HXZH2K7R4C3Z4TXH73FJZDHSRHA, seg_pos={0 0}, upload=4971]: Uploading piece... Jul 19 05:06:10.896095063 (6h15m17.268674418s) [noise=true , address=77.90.31.16:4734 , piece_id=4P24MNXH3SWO6AW6YZQSOAJV3MPGXO7AWV4USNZFQPVNYKLMJITQ, seg_pos={0 0}, upload=4971]: Uploading piece... Jul 19 05:06:10.896392553 (6h15m17.268971908s) [noise=true , address=45.91.116.9:4261 , piece_id=PCFZ7DBBS4645WSUFVRCMZREYNVWTTXP7LAFZ62M24VOUTE5NUTQ, seg_pos={0 0}, upload=4971]: Uploading piece... Jul 19 05:06:10.896704390 (6h15m17.269283745s) [noise=true , address=45.91.116.10:4083 , piece_id=OS4B3WPS6634RESFG5NMPK3YXBOT4FVTFFMPTYA2JOP73RCXIB2Q, seg_pos={0 0}, upload=4971]: Uploading piece... Jul 19 05:06:10.896735549 (6h15m17.269314904s) [noise=true , address=77.90.31.17:4771 , piece_id=GZEKXMTRPNFFMEUSED3JPONTD7WUJRQLGKEOK526PUNSBGBEOIFA, seg_pos={0 0}, upload=4971]: Uploading piece... Jul 19 05:06:10.897025003 (6h15m17.269604358s) [noise=true , address=45.91.116.13:4155 , piece_id=WPYVSGXQML5FDZFMQIH53BAMGGHUSQCT3QBB622AGC6LUBRDBLLA, seg_pos={0 0}, upload=4971]: Uploading piece... Jul 19 05:06:10.897329496 (6h15m17.269908851s) [noise=true , address=77.90.31.18:4807 , piece_id=VI2M6F2RX2ZDRB3VSYTRZVIP72MRANUQFJVEVJPHG6MOOEKB3JTQ, seg_pos={0 0}, upload=4971]: Uploading piece... Jul 19 05:06:10.897613429 (6h15m17.270192774s) [noise=true , address=45.91.116.5:4299 , piece_id=ZRFG3A3QIUKD2GDGMTJC5WOHO7WH3XEGI3OHMTN33UCMJH75BN3Q, seg_pos={0 0}, upload=4971]: Uploading piece... Jul 19 05:06:12.629052655 (6h15m19.001632s) [upload=4971]: Next returned nil segment Jul 19 05:06:12.629069227 (6h15m19.001648583s) [upload=4971]: Waiting for error group managing segments... Jul 19 05:06:12.906950458 (6h15m19.279529813s) [noise=true , address=45.91.116.9:4261 , piece_id=PCFZ7DBBS4645WSUFVRCMZREYNVWTTXP7LAFZ62M24VOUTE5NUTQ, seg_pos={0 0}, upload=4971]: Done uploading piece. err: <nil> Jul 19 05:06:12.920268145 (6h15m19.29284749s) [noise=true , address=45.91.116.8:4047 , piece_id=DTD2PZR72MQBK72KJLJSNHPH3I4WO3EOSTC2HMDSHZYZ66PAP6CA, seg_pos={0 0}, upload=4971]: Done uploading piece. err: <nil> Jul 19 05:06:12.925685361 (6h15m19.298264716s) [noise=true , address=45.91.116.11:4119 , piece_id=WOSNNCJJCFOVHU5LPDJJJVRFZL4JI5GGXR6PDE5I6FMWTYOHXMNQ, seg_pos={0 0}, upload=4971]: Done uploading piece. err: <nil> Jul 19 05:06:12.989285430 (6h15m19.361864785s) [noise=true , address=45.91.116.13:4155 , piece_id=WPYVSGXQML5FDZFMQIH53BAMGGHUSQCT3QBB622AGC6LUBRDBLLA, seg_pos={0 0}, upload=4971]: Done uploading piece. err: <nil> Jul 19 05:06:12.991576755 (6h15m19.36415611s) [noise=true , address=45.91.116.12:4226 , piece_id=2VYSGVPZDHXFGRZJITNDPCPYU4C35VYVWWJU3SE3VZ5P7OEJE46Q, seg_pos={0 0}, upload=4971]: Done uploading piece. err: <nil> Jul 19 05:06:12.994417591 (6h15m19.366996946s) [noise=true , address=45.91.116.4:4938 , piece_id=UJKB2LQPMR2ZCABG6R5KVVRW7HXZH2K7R4C3Z4TXH73FJZDHSRHA, seg_pos={0 0}, upload=4971]: Done uploading piece. err: <nil> Jul 19 05:06:13.016990598 (6h15m19.389569953s) [noise=true , address=45.91.116.5:4299 , piece_id=ZRFG3A3QIUKD2GDGMTJC5WOHO7WH3XEGI3OHMTN33UCMJH75BN3Q, seg_pos={0 0}, upload=4971]: Done uploading piece. err: <nil> Jul 19 05:06:13.061490276 (6h15m19.434069621s) [noise=true , address=45.91.116.10:4083 , piece_id=OS4B3WPS6634RESFG5NMPK3YXBOT4FVTFFMPTYA2JOP73RCXIB2Q, seg_pos={0 0}, upload=4971]: Done uploading piece. err: <nil> Jul 19 05:06:13.087738056 (6h15m19.460317411s) [noise=true , address=45.91.116.12:4225 , piece_id=FEAORHV5P6FOTKFQTL57LFNJBOEXKKW6YXBV4EGPY7ENYICF4HZQ, seg_pos={0 0}, upload=4971]: Done uploading piece. err: <nil> Jul 19 05:06:20.062962657 (6h15m26.435542002s) [noise=true , address=77.90.31.10:4520 , piece_id=U7J4BEWUBZXQGQ4VVGVUE7ZL4YXU5CWQF4NTDQKQBW3R2KT3ICBQ, seg_pos={0 0}, upload=4971]: Done uploading piece. err: <nil> Jul 19 05:06:25.282503273 (6h15m31.655082628s) [noise=true , address=77.90.31.18:4807 , piece_id=VI2M6F2RX2ZDRB3VSYTRZVIP72MRANUQFJVEVJPHG6MOOEKB3JTQ, seg_pos={0 0}, upload=4971]: Done uploading piece. err: <nil> Jul 19 05:06:33.431431980 (6h15m39.804011335s) [noise=true , address=77.90.31.13:4627 , piece_id=W7OKPZLWR4ETITJJJQYEC5MATEVYKTOESKIW3T45AR7X7OW2PTIQ, seg_pos={0 0}, upload=4971]: Done uploading piece. err: <nil> Jul 19 05:06:40.007921706 (6h15m46.380501061s) [noise=true , address=77.90.31.16:4734 , piece_id=4P24MNXH3SWO6AW6YZQSOAJV3MPGXO7AWV4USNZFQPVNYKLMJITQ, seg_pos={0 0}, upload=4971]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Jul 19 05:06:40.008083245 (6h15m46.380662591s) [noise=true , address=77.90.31.10:4519 , piece_id=JXXKI2ULFDUJYSL33MAUEI73QCM5F3L6LUCKHJYQGV6P2LWENLJA, seg_pos={0 0}, upload=4971]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:33330->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:33330->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33330->77.90.31.10:4519: use of closed network connection Jul 19 05:06:40.008103685 (6h15m46.38068305s) [noise=true , address=77.90.31.17:4771 , piece_id=GZEKXMTRPNFFMEUSED3JPONTD7WUJRQLGKEOK526PUNSBGBEOIFA, seg_pos={0 0}, upload=4971]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 188.166.225.170:60096->77.90.31.17:4771: use of closed network connection; write tcp 188.166.225.170:60096->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60096->77.90.31.17:4771: use of closed network connection Jul 19 05:06:40.008154202 (6h15m46.380733567s) [noise=true , address=77.90.31.12:4590 , piece_id=CQVAKS3RXC6JCDRKA4R3AKKBRVB4KOVW4MCRGGTAE473WQ7PN4PQ, seg_pos={0 0}, upload=4971]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:33586->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:33586->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33586->77.90.31.12:4590: use of closed network connection Jul 19 05:06:40.008196201 (6h15m46.380775557s) [seg_pos={0 0}, upload=4971]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 19 05:06:40.248804847 (6h15m46.621384202s) [upload=4971]: Done uploading segments... Jul 19 05:06:40.248830677 (6h15m46.621410032s) [upload=4971]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.956762282s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.................................XX...X...X....X..X....XXXXX] duration: 30.24274787s FAIL (2025-07-19 05:06:40 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ection; write tcp 188.166.225.170:36188->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36188->77.90.31.10:4517: use of closed network connection Jul 19 05:05:41.847565883 (6h14m48.220145238s) [seg_pos={0 0}, upload=4958]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 19 05:05:42.084038342 (6h14m48.456617697s) [upload=4958]: Done uploading segments... Jul 19 05:05:42.084064663 (6h14m48.456644018s) [upload=4958]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Jul 19 05:06:10.656276029 (6h15m17.028855385s) [upload=4971]: Starting upload Jul 19 05:06:10.656326216 (6h15m17.028905571s) [upload=4971]: Done starting upload Jul 19 05:06:10.656517231 (6h15m17.029096576s) [upload=4971]: Uploading segments... Jul 19 05:06:10.656546377 (6h15m17.029125732s) [upload=4971]: Got next segment. Inline: false Jul 19 05:06:10.656572427 (6h15m17.029151782s) [upload=4971]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 05:06:10.656576695 (6h15m17.02915604s) [upload=4971]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 05:06:10.894140643 (6h15m17.266719998s) [seg_pos={0 0}, upload=4971]: Begin upload segment... Jul 19 05:06:10.894202762 (6h15m17.266782118s) [seg_pos={0 0}, upload=4971]: Done begin upload segment. Jul 19 05:06:10.894225636 (6h15m17.266804991s) [noise=true , address=45.91.116.11:4119 , piece_id=WOSNNCJJCFOVHU5LPDJJJVRFZL4JI5GGXR6PDE5I6FMWTYOHXMNQ, seg_pos={0 0}, upload=4971]: Uploading piece... Jul 19 05:06:10.894239603 (6h15m17.266818968s) [noise=true , address=77.90.31.13:4627 , piece_id=W7OKPZLWR4ETITJJJQYEC5MATEVYKTOESKIW3T45AR7X7OW2PTIQ, seg_pos={0 0}, upload=4971]: Uploading piece... Jul 19 05:06:10.894613498 (6h15m17.267192853s) [noise=true , address=45.91.116.12:4225 , piece_id=FEAORHV5P6FOTKFQTL57LFNJBOEXKKW6YXBV4EGPY7ENYICF4HZQ, seg_pos={0 0}, upload=4971]: Uploading piece... Jul 19 05:06:10.895028152 (6h15m17.267607507s) [noise=true , address=77.90.31.10:4520 , piece_id=U7J4BEWUBZXQGQ4VVGVUE7ZL4YXU5CWQF4NTDQKQBW3R2KT3ICBQ, seg_pos={0 0}, upload=4971]: Uploading piece... Jul 19 05:06:10.895080372 (6h15m17.267659727s) [noise=true , address=77.90.31.10:4519 , piece_id=JXXKI2ULFDUJYSL33MAUEI73QCM5F3L6LUCKHJYQGV6P2LWENLJA, seg_pos={0 0}, upload=4971]: Uploading piece... Jul 19 05:06:10.895305061 (6h15m17.267884416s) [noise=true , address=45.91.116.12:4226 , piece_id=2VYSGVPZDHXFGRZJITNDPCPYU4C35VYVWWJU3SE3VZ5P7OEJE46Q, seg_pos={0 0}, upload=4971]: Uploading piece... Jul 19 05:06:10.895411916 (6h15m17.267991271s) [noise=true , address=45.91.116.8:4047 , piece_id=DTD2PZR72MQBK72KJLJSNHPH3I4WO3EOSTC2HMDSHZYZ66PAP6CA, seg_pos={0 0}, upload=4971]: Uploading piece... Jul 19 05:06:10.895576090 (6h15m17.268155445s) [noise=true , address=77.90.31.12:4590 , piece_id=CQVAKS3RXC6JCDRKA4R3AKKBRVB4KOVW4MCRGGTAE473WQ7PN4PQ, seg_pos={0 0}, upload=4971]: Uploading piece... Jul 19 05:06:10.895721348 (6h15m17.268300703s) [noise=true , address=45.91.116.4:4938 , piece_id=UJKB2LQPMR2ZCABG6R5KVVRW7HXZH2K7R4C3Z4TXH73FJZDHSRHA, seg_pos={0 0}, upload=4971]: Uploading piece... Jul 19 05:06:10.896095063 (6h15m17.268674418s) [noise=true , address=77.90.31.16:4734 , piece_id=4P24MNXH3SWO6AW6YZQSOAJV3MPGXO7AWV4USNZFQPVNYKLMJITQ, seg_pos={0 0}, upload=4971]: Uploading piece... Jul 19 05:06:10.896392553 (6h15m17.268971908s) [noise=true , address=45.91.116.9:4261 , piece_id=PCFZ7DBBS4645WSUFVRCMZREYNVWTTXP7LAFZ62M24VOUTE5NUTQ, seg_pos={0 0}, upload=4971]: Uploading piece... Jul 19 05:06:10.896704390 (6h15m17.269283745s) [noise=true , address=45.91.116.10:4083 , piece_id=OS4B3WPS6634RESFG5NMPK3YXBOT4FVTFFMPTYA2JOP73RCXIB2Q, seg_pos={0 0}, upload=4971]: Uploading piece... Jul 19 05:06:10.896735549 (6h15m17.269314904s) [noise=true , address=77.90.31.17:4771 , piece_id=GZEKXMTRPNFFMEUSED3JPONTD7WUJRQLGKEOK526PUNSBGBEOIFA, seg_pos={0 0}, upload=4971]: Uploading piece... Jul 19 05:06:10.897025003 (6h15m17.269604358s) [noise=true , address=45.91.116.13:4155 , piece_id=WPYVSGXQML5FDZFMQIH53BAMGGHUSQCT3QBB622AGC6LUBRDBLLA, seg_pos={0 0}, upload=4971]: Uploading piece... Jul 19 05:06:10.897329496 (6h15m17.269908851s) [noise=true , address=77.90.31.18:4807 , piece_id=VI2M6F2RX2ZDRB3VSYTRZVIP72MRANUQFJVEVJPHG6MOOEKB3JTQ, seg_pos={0 0}, upload=4971]: Uploading piece... Jul 19 05:06:10.897613429 (6h15m17.270192774s) [noise=true , address=45.91.116.5:4299 , piece_id=ZRFG3A3QIUKD2GDGMTJC5WOHO7WH3XEGI3OHMTN33UCMJH75BN3Q, seg_pos={0 0}, upload=4971]: Uploading piece... Jul 19 05:06:12.629052655 (6h15m19.001632s) [upload=4971]: Next returned nil segment Jul 19 05:06:12.629069227 (6h15m19.001648583s) [upload=4971]: Waiting for error group managing segments... Jul 19 05:06:12.906950458 (6h15m19.279529813s) [noise=true , address=45.91.116.9:4261 , piece_id=PCFZ7DBBS4645WSUFVRCMZREYNVWTTXP7LAFZ62M24VOUTE5NUTQ, seg_pos={0 0}, upload=4971]: Done uploading piece. err: <nil> Jul 19 05:06:12.920268145 (6h15m19.29284749s) [noise=true , address=45.91.116.8:4047 , piece_id=DTD2PZR72MQBK72KJLJSNHPH3I4WO3EOSTC2HMDSHZYZ66PAP6CA, seg_pos={0 0}, upload=4971]: Done uploading piece. err: <nil> Jul 19 05:06:12.925685361 (6h15m19.298264716s) [noise=true , address=45.91.116.11:4119 , piece_id=WOSNNCJJCFOVHU5LPDJJJVRFZL4JI5GGXR6PDE5I6FMWTYOHXMNQ, seg_pos={0 0}, upload=4971]: Done uploading piece. err: <nil> Jul 19 05:06:12.989285430 (6h15m19.361864785s) [noise=true , address=45.91.116.13:4155 , piece_id=WPYVSGXQML5FDZFMQIH53BAMGGHUSQCT3QBB622AGC6LUBRDBLLA, seg_pos={0 0}, upload=4971]: Done uploading piece. err: <nil> Jul 19 05:06:12.991576755 (6h15m19.36415611s) [noise=true , address=45.91.116.12:4226 , piece_id=2VYSGVPZDHXFGRZJITNDPCPYU4C35VYVWWJU3SE3VZ5P7OEJE46Q, seg_pos={0 0}, upload=4971]: Done uploading piece. err: <nil> Jul 19 05:06:12.994417591 (6h15m19.366996946s) [noise=true , address=45.91.116.4:4938 , piece_id=UJKB2LQPMR2ZCABG6R5KVVRW7HXZH2K7R4C3Z4TXH73FJZDHSRHA, seg_pos={0 0}, upload=4971]: Done uploading piece. err: <nil> Jul 19 05:06:13.016990598 (6h15m19.389569953s) [noise=true , address=45.91.116.5:4299 , piece_id=ZRFG3A3QIUKD2GDGMTJC5WOHO7WH3XEGI3OHMTN33UCMJH75BN3Q, seg_pos={0 0}, upload=4971]: Done uploading piece. err: <nil> Jul 19 05:06:13.061490276 (6h15m19.434069621s) [noise=true , address=45.91.116.10:4083 , piece_id=OS4B3WPS6634RESFG5NMPK3YXBOT4FVTFFMPTYA2JOP73RCXIB2Q, seg_pos={0 0}, upload=4971]: Done uploading piece. err: <nil> Jul 19 05:06:13.087738056 (6h15m19.460317411s) [noise=true , address=45.91.116.12:4225 , piece_id=FEAORHV5P6FOTKFQTL57LFNJBOEXKKW6YXBV4EGPY7ENYICF4HZQ, seg_pos={0 0}, upload=4971]: Done uploading piece. err: <nil> Jul 19 05:06:20.062962657 (6h15m26.435542002s) [noise=true , address=77.90.31.10:4520 , piece_id=U7J4BEWUBZXQGQ4VVGVUE7ZL4YXU5CWQF4NTDQKQBW3R2KT3ICBQ, seg_pos={0 0}, upload=4971]: Done uploading piece. err: <nil> Jul 19 05:06:25.282503273 (6h15m31.655082628s) [noise=true , address=77.90.31.18:4807 , piece_id=VI2M6F2RX2ZDRB3VSYTRZVIP72MRANUQFJVEVJPHG6MOOEKB3JTQ, seg_pos={0 0}, upload=4971]: Done uploading piece. err: <nil> Jul 19 05:06:33.431431980 (6h15m39.804011335s) [noise=true , address=77.90.31.13:4627 , piece_id=W7OKPZLWR4ETITJJJQYEC5MATEVYKTOESKIW3T45AR7X7OW2PTIQ, seg_pos={0 0}, upload=4971]: Done uploading piece. err: <nil> Jul 19 05:06:40.007921706 (6h15m46.380501061s) [noise=true , address=77.90.31.16:4734 , piece_id=4P24MNXH3SWO6AW6YZQSOAJV3MPGXO7AWV4USNZFQPVNYKLMJITQ, seg_pos={0 0}, upload=4971]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Jul 19 05:06:40.008083245 (6h15m46.380662591s) [noise=true , address=77.90.31.10:4519 , piece_id=JXXKI2ULFDUJYSL33MAUEI73QCM5F3L6LUCKHJYQGV6P2LWENLJA, seg_pos={0 0}, upload=4971]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:33330->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:33330->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33330->77.90.31.10:4519: use of closed network connection Jul 19 05:06:40.008103685 (6h15m46.38068305s) [noise=true , address=77.90.31.17:4771 , piece_id=GZEKXMTRPNFFMEUSED3JPONTD7WUJRQLGKEOK526PUNSBGBEOIFA, seg_pos={0 0}, upload=4971]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 188.166.225.170:60096->77.90.31.17:4771: use of closed network connection; write tcp 188.166.225.170:60096->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60096->77.90.31.17:4771: use of closed network connection Jul 19 05:06:40.008154202 (6h15m46.380733567s) [noise=true , address=77.90.31.12:4590 , piece_id=CQVAKS3RXC6JCDRKA4R3AKKBRVB4KOVW4MCRGGTAE473WQ7PN4PQ, seg_pos={0 0}, upload=4971]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:33586->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:33586->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33586->77.90.31.12:4590: use of closed network connection Jul 19 05:06:40.008196201 (6h15m46.380775557s) [seg_pos={0 0}, upload=4971]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 19 05:06:40.248804847 (6h15m46.621384202s) [upload=4971]: Done uploading segments... Jul 19 05:06:40.248830677 (6h15m46.621410032s) [upload=4971]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.242776065s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 5.819614288s FAIL (2025-07-19 05:05:47 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 4855s) [upload=4769]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 04:51:09.109709627 (6h0m15.482288982s) [upload=4769]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 04:51:09.556145753 (6h0m15.928725108s) [seg_pos={0 0}, upload=4769]: Begin upload segment... Jul 19 04:51:09.556194817 (6h0m15.928774172s) [seg_pos={0 0}, upload=4769]: Done begin upload segment. Jul 19 04:51:09.556223893 (6h0m15.928803248s) [noise=true , address=79.127.219.38:4023 , piece_id=PTPOZGLYV4MZTOFZNN3NCGYL7DFAST4PXKEUIVTMFL7HV4GZTCLQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556252197 (6h0m15.928831552s) [noise=true , address=45.91.95.4:4011 , piece_id=ZMBAS6KKHYVIRE6OBL2NFVCWMM4TKQ7J64BD3SYXIK54XCEDI2XA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556579383 (6h0m15.929158738s) [noise=true , address=45.91.116.6:4007 , piece_id=A2AFVTZBXSGBOE7XIM3B5HRN4STITFBXO6MZRJ365URNAAOZ3PSQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556854469 (6h0m15.929433824s) [noise=true , address=104.36.82.19:4423 , piece_id=PIJUHDUND36HQL67I33D7A3EULMJHS7N55MN4OFQJ4OWNQ42O47A, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557094107 (6h0m15.929673462s) [noise=true , address=45.91.116.9:4264 , piece_id=GHM5BKZKMT2QOHHBOS4FSPOWEWSFQRGRVVR4GSNDPHVIFRZHQL2A, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557336381 (6h0m15.929915736s) [noise=true , address=79.127.205.235:4660 , piece_id=UOUOGE2VP7PX7EMRZLPMCWFCYEZERM3DQMCQ6NG4J2K2DEIZR63Q, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557557885 (6h0m15.93013724s) [noise=true , address=79.127.205.246:4280 , piece_id=FLBOZI3UGTQXAQBLZ2F45Z3OEYL65F5PQBXRF2DQMZ7NGF7VDRPA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557762296 (6h0m15.930341651s) [noise=true , address=79.127.205.233:4622 , piece_id=6WWNYQTH5MGAJOEB7BXKZDJOMCAZXDW66J3OEANGDNJDNUMLZM7Q, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558014689 (6h0m15.930594044s) [noise=true , address=64.49.13.138:4482 , piece_id=MAW2M4US2OQNOS3WI67AA7NIBTJDE5T2JQTXKFPBLGVGXKVCFGTQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558215974 (6h0m15.930795319s) [noise=true , address=77.90.31.10:4546 , piece_id=R6LCA5WZOEBDURL5S3MEK22YPVYXX6EXSS4J4W57543HELYXOKZA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558409364 (6h0m15.930988719s) [noise=true , address=45.91.95.5:4019 , piece_id=YTSBCN2LPHEMRRAZIHJGVB7NBOTRXVEHD2TWUCTTLBNFFYEWWDKA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558623504 (6h0m15.931202859s) [noise=true , address=79.127.201.215:4357 , piece_id=75XWAG2VQWEYCWN2Q3WRIZSBI7LDOISZJPXQLQ2GIPNEN34VLSUQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558805162 (6h0m15.931384517s) [noise=true , address=45.91.116.14:4219 , piece_id=PDW6JPRHCVMHHOWIHT6JKV4CTSC3JY6NDKDVFZSO47OYNPZ7PGGA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558999373 (6h0m15.931578728s) [noise=true , address=77.90.31.11:4564 , piece_id=L2U7JFJRDQOFA2AYVAGTI4ZW4VABCXKMXWBIUWQO2QNNUUBJ74BA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559184768 (6h0m15.931764123s) [noise=true , address=77.90.31.16:4768 , piece_id=YNLSONFM6COZEVF6ZNQR2EA6V4LSQFZJONIXR4AFTNNMFGPV57JA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559368719 (6h0m15.931948074s) [noise=true , address=79.127.220.98:4432 , piece_id=2SUTFQ7FVDPAM2H4W3OLW6AVH3VJ33IJFE4L52SA3ET5LGBMOMPQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559549706 (6h0m15.932129061s) [noise=true , address=77.90.31.18:4813 , piece_id=VDOT7BB4O7XYJHW6BFTAYQZZYLZU5PAUXVJGSXJ7FDI4TH425COA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559727836 (6h0m15.932307191s) [noise=true , address=79.127.219.51:4211 , piece_id=HO5PLWBVU5YAYNYMN352NA73SM4WBJV4WZ7XLSQTCQ76UAL2TOJA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:10.663359087 (6h0m17.035938442s) [upload=4769]: Next returned nil segment Jul 19 04:51:10.663378664 (6h0m17.035958019s) [upload=4769]: Waiting for error group managing segments... Jul 19 04:51:10.962945479 (6h0m17.335524834s) [noise=true , address=79.127.205.233:4622 , piece_id=6WWNYQTH5MGAJOEB7BXKZDJOMCAZXDW66J3OEANGDNJDNUMLZM7Q, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:10.967689807 (6h0m17.340269152s) [noise=true , address=79.127.205.235:4660 , piece_id=UOUOGE2VP7PX7EMRZLPMCWFCYEZERM3DQMCQ6NG4J2K2DEIZR63Q, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:10.976968631 (6h0m17.349547986s) [noise=true , address=64.49.13.138:4482 , piece_id=MAW2M4US2OQNOS3WI67AA7NIBTJDE5T2JQTXKFPBLGVGXKVCFGTQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.048750700 (6h0m17.421330055s) [noise=true , address=79.127.205.246:4280 , piece_id=FLBOZI3UGTQXAQBLZ2F45Z3OEYL65F5PQBXRF2DQMZ7NGF7VDRPA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.221642403 (6h0m17.594221758s) [noise=true , address=79.127.220.98:4432 , piece_id=2SUTFQ7FVDPAM2H4W3OLW6AVH3VJ33IJFE4L52SA3ET5LGBMOMPQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.317070435 (6h0m17.68964979s) [noise=true , address=45.91.95.4:4011 , piece_id=ZMBAS6KKHYVIRE6OBL2NFVCWMM4TKQ7J64BD3SYXIK54XCEDI2XA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.324002660 (6h0m17.696582015s) [noise=true , address=45.91.95.5:4019 , piece_id=YTSBCN2LPHEMRRAZIHJGVB7NBOTRXVEHD2TWUCTTLBNFFYEWWDKA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.460976317 (6h0m17.833555672s) [noise=true , address=79.127.201.215:4357 , piece_id=75XWAG2VQWEYCWN2Q3WRIZSBI7LDOISZJPXQLQ2GIPNEN34VLSUQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.462584603 (6h0m17.835163958s) [noise=true , address=45.91.116.6:4007 , piece_id=A2AFVTZBXSGBOE7XIM3B5HRN4STITFBXO6MZRJ365URNAAOZ3PSQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.490392973 (6h0m17.862972328s) [noise=true , address=45.91.116.9:4264 , piece_id=GHM5BKZKMT2QOHHBOS4FSPOWEWSFQRGRVVR4GSNDPHVIFRZHQL2A, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.502040780 (6h0m17.874620135s) [noise=true , address=45.91.116.14:4219 , piece_id=PDW6JPRHCVMHHOWIHT6JKV4CTSC3JY6NDKDVFZSO47OYNPZ7PGGA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.600434096 (6h0m17.973013451s) [noise=true , address=79.127.219.51:4211 , piece_id=HO5PLWBVU5YAYNYMN352NA73SM4WBJV4WZ7XLSQTCQ76UAL2TOJA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.609008984 (6h0m17.981588339s) [noise=true , address=104.36.82.19:4423 , piece_id=PIJUHDUND36HQL67I33D7A3EULMJHS7N55MN4OFQJ4OWNQ42O47A, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.620825273 (6h0m17.993404628s) [noise=true , address=79.127.219.38:4023 , piece_id=PTPOZGLYV4MZTOFZNN3NCGYL7DFAST4PXKEUIVTMFL7HV4GZTCLQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.620838659 (6h0m17.993418014s) [seg_pos={0 0}, upload=4769]: Segment reached optimal threshold of 14 pieces. Jul 19 04:51:11.621067807 (6h0m17.993647163s) [noise=true , address=77.90.31.11:4564 , piece_id=L2U7JFJRDQOFA2AYVAGTI4ZW4VABCXKMXWBIUWQO2QNNUUBJ74BA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection; write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection Jul 19 04:51:11.621134234 (6h0m17.993713589s) [noise=true , address=77.90.31.10:4546 , piece_id=R6LCA5WZOEBDURL5S3MEK22YPVYXX6EXSS4J4W57543HELYXOKZA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK): protocol: read tcp 188.166.225.170:58476->77.90.31.10:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621174191 (6h0m17.993753546s) [noise=true , address=77.90.31.16:4768 , piece_id=YNLSONFM6COZEVF6ZNQR2EA6V4LSQFZJONIXR4AFTNNMFGPV57JA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UchewmtxBrtSZSpVAJFdZmcTsQVQxkjK9re6cgUGqDhyonQeF): protocol: read tcp 188.166.225.170:56326->77.90.31.16:4768: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621216842 (6h0m17.993796197s) [noise=true , address=77.90.31.18:4813 , piece_id=VDOT7BB4O7XYJHW6BFTAYQZZYLZU5PAUXVJGSXJ7FDI4TH425COA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128WthA1x9wp1dV1vkpGnoveZQpU9CSgDCZfw5GMQuVpCNSVAM5): protocol: read tcp 188.166.225.170:49704->77.90.31.18:4813: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621234917 (6h0m17.993814272s) [seg_pos={0 0}, upload=4769]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 19 04:51:11.621301444 (6h0m17.993880799s) [upload=4769]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 19 04:51:11.621305402 (6h0m17.993884758s) [upload=4769]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 19 04:51:11.923136970 (6h0m18.295716325s) [upload=4769]: Done uploading segments... Jul 19 04:51:11.923156627 (6h0m18.295735972s) [upload=4769]: Upload finished. err: <nil> Duration: 5.819637191s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [..................................XX...X...X....X..X....XXXX] duration: 30.23871209s FAIL (2025-07-19 05:05:42 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== work connection; write tcp 188.166.225.170:38324->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38324->77.90.31.16:4735: use of closed network connection Jul 19 05:04:54.419314875 (6h14m0.79189423s) [seg_pos={0 0}, upload=4947]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 19 05:04:54.664118007 (6h14m1.036697362s) [upload=4947]: Done uploading segments... Jul 19 05:04:54.664146190 (6h14m1.036725545s) [upload=4947]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Jul 19 05:05:12.495087221 (6h14m18.867666576s) [upload=4958]: Starting upload Jul 19 05:05:12.495145612 (6h14m18.867724967s) [upload=4958]: Done starting upload Jul 19 05:05:12.495231146 (6h14m18.867810501s) [upload=4958]: Uploading segments... Jul 19 05:05:12.495387355 (6h14m18.86796671s) [upload=4958]: Got next segment. Inline: false Jul 19 05:05:12.495419056 (6h14m18.867998401s) [upload=4958]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 05:05:12.495424125 (6h14m18.86800347s) [upload=4958]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 05:05:12.816820470 (6h14m19.189399855s) [seg_pos={0 0}, upload=4958]: Begin upload segment... Jul 19 05:05:12.816882909 (6h14m19.189462264s) [seg_pos={0 0}, upload=4958]: Done begin upload segment. Jul 19 05:05:12.816899441 (6h14m19.189478796s) [noise=true , address=77.90.31.10:4517 , piece_id=NQO2OFIRACOHPDU64QETHREZHIKKXG2GPDTJE65EZ2G7IXRFPVKA, seg_pos={0 0}, upload=4958]: Uploading piece... Jul 19 05:05:12.816916042 (6h14m19.189495397s) [noise=true , address=45.91.116.4:4938 , piece_id=TVZQT5XBSRUGE4IFCCWHQY7OTC6C4X5B4FWQREYRW3KTPJE5AHRA, seg_pos={0 0}, upload=4958]: Uploading piece... Jul 19 05:05:12.816980035 (6h14m19.1895594s) [noise=true , address=45.91.116.5:4298 , piece_id=X4WMJMAB5BJBUIHHLQBMTVFZASETVMEJK6JJY23S7J4YCVTGXL2Q, seg_pos={0 0}, upload=4958]: Uploading piece... Jul 19 05:05:12.817717386 (6h14m19.190296781s) [noise=true , address=77.90.31.17:4772 , piece_id=UE2C3F4C2J472KPRRAFL7X4X3I5RPPGFQMOXK7KDLGHFXTRXNURA, seg_pos={0 0}, upload=4958]: Uploading piece... Jul 19 05:05:12.817733768 (6h14m19.190313133s) [noise=true , address=45.91.116.10:4082 , piece_id=KFUYHCDUW5MXDXTQ2YECKHK3KJDIKUHVT5WDYJRWTVIHLAKIBSKA, seg_pos={0 0}, upload=4958]: Uploading piece... Jul 19 05:05:12.818428637 (6h14m19.191007992s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7FEYDKBY56OCM6OHQOD56ZFOOHIJTRDOCYRRJ3DHIR3ZX7TLZUQ, seg_pos={0 0}, upload=4958]: Uploading piece... Jul 19 05:05:12.819173994 (6h14m19.191753349s) [noise=true , address=77.90.31.19:4842 , piece_id=BM5SEBDLA56IZ2IVJH4RGH5CF7S5FUTMK53Z7YCU6FRCWJMRODMA, seg_pos={0 0}, upload=4958]: Uploading piece... Jul 19 05:05:12.819612493 (6h14m19.192191858s) [noise=true , address=45.91.116.14:4189 , piece_id=6Z7G5HY6Y53JTOHNCOEPF3OHXXBMYD3L46W3G3JCFTPTFF7SLWEQ, seg_pos={0 0}, upload=4958]: Uploading piece... Jul 19 05:05:12.820070189 (6h14m19.192649544s) [noise=true , address=45.91.116.11:4117 , piece_id=ZTM4KFP5UWB3VVHPR2PBF3NS335XKCDJ2Y2C4TCQZOGLLB54MTAQ, seg_pos={0 0}, upload=4958]: Uploading piece... Jul 19 05:05:12.820451068 (6h14m19.193030423s) [noise=true , address=77.90.31.13:4626 , piece_id=G7I2XLZMHI2XHXUE64TIOLFG2VFAIAD62YSY5KLMO57XNRIWMKIA, seg_pos={0 0}, upload=4958]: Uploading piece... Jul 19 05:05:12.820782132 (6h14m19.193361487s) [noise=true , address=45.91.116.9:4261 , piece_id=YWUFIKX2RHKZEEH2GDW5JMD2Y5FJAEETOIOKGGL3EH4KRAR2XLNA, seg_pos={0 0}, upload=4958]: Uploading piece... Jul 19 05:05:12.821090962 (6h14m19.193670317s) [noise=true , address=45.91.116.9:4263 , piece_id=RQ7AIZL22DZOGOEOTZ4S4V656SLPG3K5TKFNH7G4CBVF7DBHRL2Q, seg_pos={0 0}, upload=4958]: Uploading piece... Jul 19 05:05:12.821410845 (6h14m19.1939902s) [noise=true , address=45.91.116.10:4081 , piece_id=JGYXIT6LBGYL5EANR4UBY4XRDI43LQNYV6PE2XQUDZ422BLLSYKQ, seg_pos={0 0}, upload=4958]: Uploading piece... Jul 19 05:05:12.821690249 (6h14m19.194269604s) [noise=true , address=45.91.116.14:4191 , piece_id=KOC5G2RUF4ACV2ZB2BWHTT3IJ43ILY4ZJ7NIO3CGUI6V6NAZ6QGA, seg_pos={0 0}, upload=4958]: Uploading piece... Jul 19 05:05:12.821990504 (6h14m19.194569859s) [noise=true , address=77.90.31.19:4841 , piece_id=W76GRVUXSP267PDUM257GJMOCATWFKXC2YYTXQQT6HDDIMNYPWVQ, seg_pos={0 0}, upload=4958]: Uploading piece... Jul 19 05:05:12.823132950 (6h14m19.195712315s) [noise=true , address=77.90.31.10:4520 , piece_id=CVS347QAQH36X3UZTCFQZAGXHA4PMGJRWHGPXQXDZKJIBQLUWG3A, seg_pos={0 0}, upload=4958]: Uploading piece... Jul 19 05:05:14.557564334 (6h14m20.930143689s) [upload=4958]: Next returned nil segment Jul 19 05:05:14.557585223 (6h14m20.930164579s) [upload=4958]: Waiting for error group managing segments... Jul 19 05:05:14.833205730 (6h14m21.205785085s) [noise=true , address=45.91.116.10:4081 , piece_id=JGYXIT6LBGYL5EANR4UBY4XRDI43LQNYV6PE2XQUDZ422BLLSYKQ, seg_pos={0 0}, upload=4958]: Done uploading piece. err: <nil> Jul 19 05:05:14.836674397 (6h14m21.209253753s) [noise=true , address=45.91.116.5:4298 , piece_id=X4WMJMAB5BJBUIHHLQBMTVFZASETVMEJK6JJY23S7J4YCVTGXL2Q, seg_pos={0 0}, upload=4958]: Done uploading piece. err: <nil> Jul 19 05:05:14.863812671 (6h14m21.236392026s) [noise=true , address=45.91.116.4:4938 , piece_id=TVZQT5XBSRUGE4IFCCWHQY7OTC6C4X5B4FWQREYRW3KTPJE5AHRA, seg_pos={0 0}, upload=4958]: Done uploading piece. err: <nil> Jul 19 05:05:14.921764211 (6h14m21.294343566s) [noise=true , address=45.91.116.9:4261 , piece_id=YWUFIKX2RHKZEEH2GDW5JMD2Y5FJAEETOIOKGGL3EH4KRAR2XLNA, seg_pos={0 0}, upload=4958]: Done uploading piece. err: <nil> Jul 19 05:05:14.926647236 (6h14m21.299226591s) [noise=true , address=45.91.116.10:4082 , piece_id=KFUYHCDUW5MXDXTQ2YECKHK3KJDIKUHVT5WDYJRWTVIHLAKIBSKA, seg_pos={0 0}, upload=4958]: Done uploading piece. err: <nil> Jul 19 05:05:14.932926532 (6h14m21.305505887s) [noise=true , address=45.91.116.13:4155 , piece_id=Y7FEYDKBY56OCM6OHQOD56ZFOOHIJTRDOCYRRJ3DHIR3ZX7TLZUQ, seg_pos={0 0}, upload=4958]: Done uploading piece. err: <nil> Jul 19 05:05:14.951446273 (6h14m21.324025628s) [noise=true , address=45.91.116.14:4191 , piece_id=KOC5G2RUF4ACV2ZB2BWHTT3IJ43ILY4ZJ7NIO3CGUI6V6NAZ6QGA, seg_pos={0 0}, upload=4958]: Done uploading piece. err: <nil> Jul 19 05:05:15.009071949 (6h14m21.381651304s) [noise=true , address=45.91.116.14:4189 , piece_id=6Z7G5HY6Y53JTOHNCOEPF3OHXXBMYD3L46W3G3JCFTPTFF7SLWEQ, seg_pos={0 0}, upload=4958]: Done uploading piece. err: <nil> Jul 19 05:05:15.009885836 (6h14m21.382465191s) [noise=true , address=45.91.116.9:4263 , piece_id=RQ7AIZL22DZOGOEOTZ4S4V656SLPG3K5TKFNH7G4CBVF7DBHRL2Q, seg_pos={0 0}, upload=4958]: Done uploading piece. err: <nil> Jul 19 05:05:15.026495553 (6h14m21.399074908s) [noise=true , address=45.91.116.11:4117 , piece_id=ZTM4KFP5UWB3VVHPR2PBF3NS335XKCDJ2Y2C4TCQZOGLLB54MTAQ, seg_pos={0 0}, upload=4958]: Done uploading piece. err: <nil> Jul 19 05:05:27.517038725 (6h14m33.88961808s) [noise=true , address=77.90.31.19:4842 , piece_id=BM5SEBDLA56IZ2IVJH4RGH5CF7S5FUTMK53Z7YCU6FRCWJMRODMA, seg_pos={0 0}, upload=4958]: Done uploading piece. err: <nil> Jul 19 05:05:34.774342491 (6h14m41.146921846s) [noise=true , address=77.90.31.13:4626 , piece_id=G7I2XLZMHI2XHXUE64TIOLFG2VFAIAD62YSY5KLMO57XNRIWMKIA, seg_pos={0 0}, upload=4958]: Done uploading piece. err: <nil> Jul 19 05:05:41.847054935 (6h14m48.21963429s) [noise=true , address=77.90.31.19:4841 , piece_id=W76GRVUXSP267PDUM257GJMOCATWFKXC2YYTXQQT6HDDIMNYPWVQ, seg_pos={0 0}, upload=4958]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:54862->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:54862->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54862->77.90.31.19:4841: use of closed network connection Jul 19 05:05:41.847278974 (6h14m48.219858329s) [noise=true , address=77.90.31.17:4772 , piece_id=UE2C3F4C2J472KPRRAFL7X4X3I5RPPGFQMOXK7KDLGHFXTRXNURA, seg_pos={0 0}, upload=4958]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: expected piece hash; context deadline exceeded; EOF Jul 19 05:05:41.847473375 (6h14m48.22005274s) [noise=true , address=77.90.31.10:4520 , piece_id=CVS347QAQH36X3UZTCFQZAGXHA4PMGJRWHGPXQXDZKJIBQLUWG3A, seg_pos={0 0}, upload=4958]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 188.166.225.170:52278->77.90.31.10:4520: use of closed network connection; write tcp 188.166.225.170:52278->77.90.31.10:4520: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52278->77.90.31.10:4520: use of closed network connection Jul 19 05:05:41.847546345 (6h14m48.2201257s) [noise=true , address=77.90.31.10:4517 , piece_id=NQO2OFIRACOHPDU64QETHREZHIKKXG2GPDTJE65EZ2G7IXRFPVKA, seg_pos={0 0}, upload=4958]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:36188->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:36188->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36188->77.90.31.10:4517: use of closed network connection Jul 19 05:05:41.847565883 (6h14m48.220145238s) [seg_pos={0 0}, upload=4958]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 19 05:05:42.084038342 (6h14m48.456617697s) [upload=4958]: Done uploading segments... Jul 19 05:05:42.084064663 (6h14m48.456644018s) [upload=4958]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.238737107s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [...................................XX...X...X....X..X....XXX] duration: 30.246783778s FAIL (2025-07-19 05:04:54 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 01493142 (6h13m20.574072497s) [seg_pos={0 0}, upload=4939]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 19 05:04:14.442040800 (6h13m20.814620155s) [upload=4939]: Done uploading segments... Jul 19 05:04:14.442067992 (6h13m20.814647347s) [upload=4939]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Jul 19 05:04:25.062975524 (6h13m31.435554889s) [upload=4947]: Starting upload Jul 19 05:04:25.063050326 (6h13m31.435629681s) [upload=4947]: Done starting upload Jul 19 05:04:25.063155357 (6h13m31.435734712s) [upload=4947]: Uploading segments... Jul 19 05:04:25.063237875 (6h13m31.43581723s) [upload=4947]: Got next segment. Inline: false Jul 19 05:04:25.063271609 (6h13m31.435850964s) [upload=4947]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 05:04:25.063276660 (6h13m31.435856015s) [upload=4947]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 05:04:25.294241168 (6h13m31.666820523s) [seg_pos={0 0}, upload=4947]: Begin upload segment... Jul 19 05:04:25.294284011 (6h13m31.666863366s) [seg_pos={0 0}, upload=4947]: Done begin upload segment. Jul 19 05:04:25.294297586 (6h13m31.666876941s) [noise=true , address=77.90.31.11:4553 , piece_id=TB7IAF77R54HUVN77BVIF5J4Y6GTUBYYYJD2ATFAP2L3TN7CMEHQ, seg_pos={0 0}, upload=4947]: Uploading piece... Jul 19 05:04:25.294308607 (6h13m31.666887962s) [noise=true , address=45.91.116.7:4009 , piece_id=MHW6WZXKPFHZSRFX5CSZCBIYFKUDJEZ46JUNXELGMNUP4Q5YXIFQ, seg_pos={0 0}, upload=4947]: Uploading piece... Jul 19 05:04:25.294349495 (6h13m31.66692885s) [noise=true , address=45.91.116.14:4190 , piece_id=VXT7XAAIMZ33NIUM5GCDY7CTOFRH6H2SKSUMRHWBKTKBBVQ3NWJA, seg_pos={0 0}, upload=4947]: Uploading piece... Jul 19 05:04:25.294525852 (6h13m31.667105208s) [noise=true , address=77.90.31.16:4735 , piece_id=35IMYCGD4WKCD4HHMZKV5OEF4JUDQ72QWZ5I7B7FUDLJPSKLV3AQ, seg_pos={0 0}, upload=4947]: Uploading piece... Jul 19 05:04:25.294652816 (6h13m31.667232171s) [noise=true , address=45.91.116.8:4048 , piece_id=B7KQRCPQUXLWO5AEM2EV5XOQJBRL2KLTS6J3NZZJWBFIHWWQQOGA, seg_pos={0 0}, upload=4947]: Uploading piece... Jul 19 05:04:25.294796310 (6h13m31.667375665s) [noise=true , address=77.90.31.19:4842 , piece_id=XNG3OIQXHGZF2OYASIXC7YWD6XRFU7RSMGNLGIIFZTXEBRD6LHGQ, seg_pos={0 0}, upload=4947]: Uploading piece... Jul 19 05:04:25.294904327 (6h13m31.667483682s) [noise=true , address=104.36.82.9:4006 , piece_id=54FLFMKVSDJTLBFNVHPMLJOY3IAIKNOAQX3GQOPJXZTE3SZMRACA, seg_pos={0 0}, upload=4947]: Uploading piece... Jul 19 05:04:25.295086065 (6h13m31.66766541s) [noise=true , address=45.91.116.5:4299 , piece_id=6PDEJKI6ZRKU46NJ4HE4UQ6TLVHVGOLVUZROPKD2XGKJEILNY2CQ, seg_pos={0 0}, upload=4947]: Uploading piece... Jul 19 05:04:25.295349309 (6h13m31.667928664s) [noise=true , address=45.91.116.4:4939 , piece_id=A5KWQKRQSRGHKKHIKCIUNYSUFX4ONBB2CTDF2OUISTCIIY34NZ4A, seg_pos={0 0}, upload=4947]: Uploading piece... Jul 19 05:04:25.295686304 (6h13m31.668265659s) [noise=true , address=77.90.31.13:4627 , piece_id=SAVOTAM3SQIMOYVP25MMBSYMHF73E245IB75GZW27NOGHNIDZY3A, seg_pos={0 0}, upload=4947]: Uploading piece... Jul 19 05:04:25.295926694 (6h13m31.668506049s) [noise=true , address=45.91.116.13:4155 , piece_id=UYBNTGQBRDKWCV5DGKKUUSM7Q2E6YQAG4BARI6JLROWNT6GFQRCQ, seg_pos={0 0}, upload=4947]: Uploading piece... Jul 19 05:04:25.296202381 (6h13m31.668781736s) [noise=true , address=45.91.116.7:4011 , piece_id=4R5YUFDQTB7CKHHBNVX5ZTQQW2NF54GS6YVBX2WB2EAMSB5GO5PQ, seg_pos={0 0}, upload=4947]: Uploading piece... Jul 19 05:04:25.296398957 (6h13m31.668978312s) [noise=true , address=45.91.116.6:4974 , piece_id=KM7KSHTFV67VDGFDDYQXESN4GH3ZSPDDTEE5YOD437SQGAM26FCQ, seg_pos={0 0}, upload=4947]: Uploading piece... Jul 19 05:04:25.296599071 (6h13m31.669178426s) [noise=true , address=45.91.116.5:4297 , piece_id=BPPHDW35GVW5YTYYJHVIWZYAHDT43VBIIC3PH7LSK76QBD7W5IYQ, seg_pos={0 0}, upload=4947]: Uploading piece... Jul 19 05:04:25.296912350 (6h13m31.669491705s) [noise=true , address=77.90.31.11:4555 , piece_id=WMGKK3PJES55VZXDFTFCVJJSHX3ET3GCZ7UGAR4X6MALUON75DMQ, seg_pos={0 0}, upload=4947]: Uploading piece... Jul 19 05:04:25.297121510 (6h13m31.669700865s) [noise=true , address=45.91.116.14:4191 , piece_id=YGSRZZRTE7VREHQKQZSHT6D5ZO7OYQKKEYRJ56K7JOOTK6U6XYUA, seg_pos={0 0}, upload=4947]: Uploading piece... Jul 19 05:04:27.019208544 (6h13m33.391787899s) [upload=4947]: Next returned nil segment Jul 19 05:04:27.019228112 (6h13m33.391807457s) [upload=4947]: Waiting for error group managing segments... Jul 19 05:04:27.295342934 (6h13m33.667922289s) [noise=true , address=45.91.116.5:4299 , piece_id=6PDEJKI6ZRKU46NJ4HE4UQ6TLVHVGOLVUZROPKD2XGKJEILNY2CQ, seg_pos={0 0}, upload=4947]: Done uploading piece. err: <nil> Jul 19 05:04:27.313593741 (6h13m33.686173096s) [noise=true , address=45.91.116.14:4190 , piece_id=VXT7XAAIMZ33NIUM5GCDY7CTOFRH6H2SKSUMRHWBKTKBBVQ3NWJA, seg_pos={0 0}, upload=4947]: Done uploading piece. err: <nil> Jul 19 05:04:27.319297996 (6h13m33.691877341s) [noise=true , address=45.91.116.13:4155 , piece_id=UYBNTGQBRDKWCV5DGKKUUSM7Q2E6YQAG4BARI6JLROWNT6GFQRCQ, seg_pos={0 0}, upload=4947]: Done uploading piece. err: <nil> Jul 19 05:04:27.320901415 (6h13m33.69348077s) [noise=true , address=45.91.116.7:4011 , piece_id=4R5YUFDQTB7CKHHBNVX5ZTQQW2NF54GS6YVBX2WB2EAMSB5GO5PQ, seg_pos={0 0}, upload=4947]: Done uploading piece. err: <nil> Jul 19 05:04:27.399554451 (6h13m33.772133806s) [noise=true , address=45.91.116.4:4939 , piece_id=A5KWQKRQSRGHKKHIKCIUNYSUFX4ONBB2CTDF2OUISTCIIY34NZ4A, seg_pos={0 0}, upload=4947]: Done uploading piece. err: <nil> Jul 19 05:04:27.406268059 (6h13m33.778847414s) [noise=true , address=45.91.116.5:4297 , piece_id=BPPHDW35GVW5YTYYJHVIWZYAHDT43VBIIC3PH7LSK76QBD7W5IYQ, seg_pos={0 0}, upload=4947]: Done uploading piece. err: <nil> Jul 19 05:04:27.406522836 (6h13m33.779102191s) [noise=true , address=45.91.116.14:4191 , piece_id=YGSRZZRTE7VREHQKQZSHT6D5ZO7OYQKKEYRJ56K7JOOTK6U6XYUA, seg_pos={0 0}, upload=4947]: Done uploading piece. err: <nil> Jul 19 05:04:27.413032854 (6h13m33.785612209s) [noise=true , address=45.91.116.8:4048 , piece_id=B7KQRCPQUXLWO5AEM2EV5XOQJBRL2KLTS6J3NZZJWBFIHWWQQOGA, seg_pos={0 0}, upload=4947]: Done uploading piece. err: <nil> Jul 19 05:04:27.455474905 (6h13m33.82805426s) [noise=true , address=104.36.82.9:4006 , piece_id=54FLFMKVSDJTLBFNVHPMLJOY3IAIKNOAQX3GQOPJXZTE3SZMRACA, seg_pos={0 0}, upload=4947]: Done uploading piece. err: <nil> Jul 19 05:04:27.461903236 (6h13m33.834482591s) [noise=true , address=45.91.116.6:4974 , piece_id=KM7KSHTFV67VDGFDDYQXESN4GH3ZSPDDTEE5YOD437SQGAM26FCQ, seg_pos={0 0}, upload=4947]: Done uploading piece. err: <nil> Jul 19 05:04:27.482739179 (6h13m33.855318534s) [noise=true , address=45.91.116.7:4009 , piece_id=MHW6WZXKPFHZSRFX5CSZCBIYFKUDJEZ46JUNXELGMNUP4Q5YXIFQ, seg_pos={0 0}, upload=4947]: Done uploading piece. err: <nil> Jul 19 05:04:41.625353298 (6h13m47.997932693s) [noise=true , address=77.90.31.19:4842 , piece_id=XNG3OIQXHGZF2OYASIXC7YWD6XRFU7RSMGNLGIIFZTXEBRD6LHGQ, seg_pos={0 0}, upload=4947]: Done uploading piece. err: <nil> Jul 19 05:04:54.419205786 (6h14m0.791785141s) [noise=true , address=77.90.31.11:4553 , piece_id=TB7IAF77R54HUVN77BVIF5J4Y6GTUBYYYJD2ATFAP2L3TN7CMEHQ, seg_pos={0 0}, upload=4947]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:60420->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:60420->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60420->77.90.31.11:4553: use of closed network connection Jul 19 05:04:54.419245552 (6h14m0.791824897s) [noise=true , address=77.90.31.11:4555 , piece_id=WMGKK3PJES55VZXDFTFCVJJSHX3ET3GCZ7UGAR4X6MALUON75DMQ, seg_pos={0 0}, upload=4947]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: write tcp 188.166.225.170:44430->77.90.31.11:4555: use of closed network connection; write tcp 188.166.225.170:44430->77.90.31.11:4555: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44430->77.90.31.11:4555: use of closed network connection Jul 19 05:04:54.419257736 (6h14m0.791837091s) [noise=true , address=77.90.31.13:4627 , piece_id=SAVOTAM3SQIMOYVP25MMBSYMHF73E245IB75GZW27NOGHNIDZY3A, seg_pos={0 0}, upload=4947]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:39584->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:39584->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39584->77.90.31.13:4627: use of closed network connection Jul 19 05:04:54.419271111 (6h14m0.791850466s) [noise=true , address=77.90.31.16:4735 , piece_id=35IMYCGD4WKCD4HHMZKV5OEF4JUDQ72QWZ5I7B7FUDLJPSKLV3AQ, seg_pos={0 0}, upload=4947]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:38324->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:38324->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38324->77.90.31.16:4735: use of closed network connection Jul 19 05:04:54.419314875 (6h14m0.79189423s) [seg_pos={0 0}, upload=4947]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 19 05:04:54.664118007 (6h14m1.036697362s) [upload=4947]: Done uploading segments... Jul 19 05:04:54.664146190 (6h14m1.036725545s) [upload=4947]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.246807092s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 5.770678791s FAIL (2025-07-19 05:04:44 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 4855s) [upload=4769]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 04:51:09.109709627 (6h0m15.482288982s) [upload=4769]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 04:51:09.556145753 (6h0m15.928725108s) [seg_pos={0 0}, upload=4769]: Begin upload segment... Jul 19 04:51:09.556194817 (6h0m15.928774172s) [seg_pos={0 0}, upload=4769]: Done begin upload segment. Jul 19 04:51:09.556223893 (6h0m15.928803248s) [noise=true , address=79.127.219.38:4023 , piece_id=PTPOZGLYV4MZTOFZNN3NCGYL7DFAST4PXKEUIVTMFL7HV4GZTCLQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556252197 (6h0m15.928831552s) [noise=true , address=45.91.95.4:4011 , piece_id=ZMBAS6KKHYVIRE6OBL2NFVCWMM4TKQ7J64BD3SYXIK54XCEDI2XA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556579383 (6h0m15.929158738s) [noise=true , address=45.91.116.6:4007 , piece_id=A2AFVTZBXSGBOE7XIM3B5HRN4STITFBXO6MZRJ365URNAAOZ3PSQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556854469 (6h0m15.929433824s) [noise=true , address=104.36.82.19:4423 , piece_id=PIJUHDUND36HQL67I33D7A3EULMJHS7N55MN4OFQJ4OWNQ42O47A, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557094107 (6h0m15.929673462s) [noise=true , address=45.91.116.9:4264 , piece_id=GHM5BKZKMT2QOHHBOS4FSPOWEWSFQRGRVVR4GSNDPHVIFRZHQL2A, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557336381 (6h0m15.929915736s) [noise=true , address=79.127.205.235:4660 , piece_id=UOUOGE2VP7PX7EMRZLPMCWFCYEZERM3DQMCQ6NG4J2K2DEIZR63Q, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557557885 (6h0m15.93013724s) [noise=true , address=79.127.205.246:4280 , piece_id=FLBOZI3UGTQXAQBLZ2F45Z3OEYL65F5PQBXRF2DQMZ7NGF7VDRPA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557762296 (6h0m15.930341651s) [noise=true , address=79.127.205.233:4622 , piece_id=6WWNYQTH5MGAJOEB7BXKZDJOMCAZXDW66J3OEANGDNJDNUMLZM7Q, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558014689 (6h0m15.930594044s) [noise=true , address=64.49.13.138:4482 , piece_id=MAW2M4US2OQNOS3WI67AA7NIBTJDE5T2JQTXKFPBLGVGXKVCFGTQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558215974 (6h0m15.930795319s) [noise=true , address=77.90.31.10:4546 , piece_id=R6LCA5WZOEBDURL5S3MEK22YPVYXX6EXSS4J4W57543HELYXOKZA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558409364 (6h0m15.930988719s) [noise=true , address=45.91.95.5:4019 , piece_id=YTSBCN2LPHEMRRAZIHJGVB7NBOTRXVEHD2TWUCTTLBNFFYEWWDKA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558623504 (6h0m15.931202859s) [noise=true , address=79.127.201.215:4357 , piece_id=75XWAG2VQWEYCWN2Q3WRIZSBI7LDOISZJPXQLQ2GIPNEN34VLSUQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558805162 (6h0m15.931384517s) [noise=true , address=45.91.116.14:4219 , piece_id=PDW6JPRHCVMHHOWIHT6JKV4CTSC3JY6NDKDVFZSO47OYNPZ7PGGA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558999373 (6h0m15.931578728s) [noise=true , address=77.90.31.11:4564 , piece_id=L2U7JFJRDQOFA2AYVAGTI4ZW4VABCXKMXWBIUWQO2QNNUUBJ74BA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559184768 (6h0m15.931764123s) [noise=true , address=77.90.31.16:4768 , piece_id=YNLSONFM6COZEVF6ZNQR2EA6V4LSQFZJONIXR4AFTNNMFGPV57JA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559368719 (6h0m15.931948074s) [noise=true , address=79.127.220.98:4432 , piece_id=2SUTFQ7FVDPAM2H4W3OLW6AVH3VJ33IJFE4L52SA3ET5LGBMOMPQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559549706 (6h0m15.932129061s) [noise=true , address=77.90.31.18:4813 , piece_id=VDOT7BB4O7XYJHW6BFTAYQZZYLZU5PAUXVJGSXJ7FDI4TH425COA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559727836 (6h0m15.932307191s) [noise=true , address=79.127.219.51:4211 , piece_id=HO5PLWBVU5YAYNYMN352NA73SM4WBJV4WZ7XLSQTCQ76UAL2TOJA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:10.663359087 (6h0m17.035938442s) [upload=4769]: Next returned nil segment Jul 19 04:51:10.663378664 (6h0m17.035958019s) [upload=4769]: Waiting for error group managing segments... Jul 19 04:51:10.962945479 (6h0m17.335524834s) [noise=true , address=79.127.205.233:4622 , piece_id=6WWNYQTH5MGAJOEB7BXKZDJOMCAZXDW66J3OEANGDNJDNUMLZM7Q, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:10.967689807 (6h0m17.340269152s) [noise=true , address=79.127.205.235:4660 , piece_id=UOUOGE2VP7PX7EMRZLPMCWFCYEZERM3DQMCQ6NG4J2K2DEIZR63Q, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:10.976968631 (6h0m17.349547986s) [noise=true , address=64.49.13.138:4482 , piece_id=MAW2M4US2OQNOS3WI67AA7NIBTJDE5T2JQTXKFPBLGVGXKVCFGTQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.048750700 (6h0m17.421330055s) [noise=true , address=79.127.205.246:4280 , piece_id=FLBOZI3UGTQXAQBLZ2F45Z3OEYL65F5PQBXRF2DQMZ7NGF7VDRPA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.221642403 (6h0m17.594221758s) [noise=true , address=79.127.220.98:4432 , piece_id=2SUTFQ7FVDPAM2H4W3OLW6AVH3VJ33IJFE4L52SA3ET5LGBMOMPQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.317070435 (6h0m17.68964979s) [noise=true , address=45.91.95.4:4011 , piece_id=ZMBAS6KKHYVIRE6OBL2NFVCWMM4TKQ7J64BD3SYXIK54XCEDI2XA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.324002660 (6h0m17.696582015s) [noise=true , address=45.91.95.5:4019 , piece_id=YTSBCN2LPHEMRRAZIHJGVB7NBOTRXVEHD2TWUCTTLBNFFYEWWDKA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.460976317 (6h0m17.833555672s) [noise=true , address=79.127.201.215:4357 , piece_id=75XWAG2VQWEYCWN2Q3WRIZSBI7LDOISZJPXQLQ2GIPNEN34VLSUQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.462584603 (6h0m17.835163958s) [noise=true , address=45.91.116.6:4007 , piece_id=A2AFVTZBXSGBOE7XIM3B5HRN4STITFBXO6MZRJ365URNAAOZ3PSQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.490392973 (6h0m17.862972328s) [noise=true , address=45.91.116.9:4264 , piece_id=GHM5BKZKMT2QOHHBOS4FSPOWEWSFQRGRVVR4GSNDPHVIFRZHQL2A, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.502040780 (6h0m17.874620135s) [noise=true , address=45.91.116.14:4219 , piece_id=PDW6JPRHCVMHHOWIHT6JKV4CTSC3JY6NDKDVFZSO47OYNPZ7PGGA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.600434096 (6h0m17.973013451s) [noise=true , address=79.127.219.51:4211 , piece_id=HO5PLWBVU5YAYNYMN352NA73SM4WBJV4WZ7XLSQTCQ76UAL2TOJA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.609008984 (6h0m17.981588339s) [noise=true , address=104.36.82.19:4423 , piece_id=PIJUHDUND36HQL67I33D7A3EULMJHS7N55MN4OFQJ4OWNQ42O47A, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.620825273 (6h0m17.993404628s) [noise=true , address=79.127.219.38:4023 , piece_id=PTPOZGLYV4MZTOFZNN3NCGYL7DFAST4PXKEUIVTMFL7HV4GZTCLQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.620838659 (6h0m17.993418014s) [seg_pos={0 0}, upload=4769]: Segment reached optimal threshold of 14 pieces. Jul 19 04:51:11.621067807 (6h0m17.993647163s) [noise=true , address=77.90.31.11:4564 , piece_id=L2U7JFJRDQOFA2AYVAGTI4ZW4VABCXKMXWBIUWQO2QNNUUBJ74BA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection; write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection Jul 19 04:51:11.621134234 (6h0m17.993713589s) [noise=true , address=77.90.31.10:4546 , piece_id=R6LCA5WZOEBDURL5S3MEK22YPVYXX6EXSS4J4W57543HELYXOKZA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK): protocol: read tcp 188.166.225.170:58476->77.90.31.10:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621174191 (6h0m17.993753546s) [noise=true , address=77.90.31.16:4768 , piece_id=YNLSONFM6COZEVF6ZNQR2EA6V4LSQFZJONIXR4AFTNNMFGPV57JA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UchewmtxBrtSZSpVAJFdZmcTsQVQxkjK9re6cgUGqDhyonQeF): protocol: read tcp 188.166.225.170:56326->77.90.31.16:4768: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621216842 (6h0m17.993796197s) [noise=true , address=77.90.31.18:4813 , piece_id=VDOT7BB4O7XYJHW6BFTAYQZZYLZU5PAUXVJGSXJ7FDI4TH425COA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128WthA1x9wp1dV1vkpGnoveZQpU9CSgDCZfw5GMQuVpCNSVAM5): protocol: read tcp 188.166.225.170:49704->77.90.31.18:4813: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621234917 (6h0m17.993814272s) [seg_pos={0 0}, upload=4769]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 19 04:51:11.621301444 (6h0m17.993880799s) [upload=4769]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 19 04:51:11.621305402 (6h0m17.993884758s) [upload=4769]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 19 04:51:11.923136970 (6h0m18.295716325s) [upload=4769]: Done uploading segments... Jul 19 04:51:11.923156627 (6h0m18.295735972s) [upload=4769]: Upload finished. err: <nil> Duration: 5.770703919s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [....................................XX...X...X....X..X....XX] duration: 30.241646279s FAIL (2025-07-19 05:04:14 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 4925]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:51946->77.90.31.15:4699: use of closed network connection; write tcp 188.166.225.170:51946->77.90.31.15:4699: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51946->77.90.31.15:4699: use of closed network connection Jul 19 05:03:06.533247682 (6h12m12.905827037s) [seg_pos={0 0}, upload=4925]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 19 05:03:06.775784086 (6h12m13.148363451s) [upload=4925]: Done uploading segments... Jul 19 05:03:06.775815406 (6h12m13.148394791s) [upload=4925]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Jul 19 05:03:44.864120299 (6h12m51.236699654s) [upload=4939]: Starting upload Jul 19 05:03:44.864182970 (6h12m51.236762325s) [upload=4939]: Done starting upload Jul 19 05:03:44.864276959 (6h12m51.236856304s) [upload=4939]: Uploading segments... Jul 19 05:03:44.864413570 (6h12m51.236992915s) [upload=4939]: Got next segment. Inline: false Jul 19 05:03:44.864447625 (6h12m51.23702698s) [upload=4939]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 05:03:44.864452055 (6h12m51.23703141s) [upload=4939]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 05:03:45.115665474 (6h12m51.488244829s) [seg_pos={0 0}, upload=4939]: Begin upload segment... Jul 19 05:03:45.115718175 (6h12m51.48829753s) [seg_pos={0 0}, upload=4939]: Done begin upload segment. Jul 19 05:03:45.115741389 (6h12m51.488320744s) [noise=true , address=104.36.82.13:4186 , piece_id=4ZPHE5YJRIVLNUC3YF2JZQTM5EMIFAR5FR4F6X62KEWBCLCF66PQ, seg_pos={0 0}, upload=4939]: Uploading piece... Jul 19 05:03:45.115795513 (6h12m51.488374868s) [noise=true , address=77.90.31.13:4628 , piece_id=FCMOWACIWBGSSFVHV22Q7V2KIAY27EHE3CKIHX5KAYTZNLEHQEYA, seg_pos={0 0}, upload=4939]: Uploading piece... Jul 19 05:03:45.116158808 (6h12m51.488738163s) [noise=true , address=45.91.116.14:4191 , piece_id=TUJXONY7RDRBJQ32GHGYUVMHZYQNX3HGR76WVA2Q5VUQK5ELJ3FQ, seg_pos={0 0}, upload=4939]: Uploading piece... Jul 19 05:03:45.116333593 (6h12m51.488912938s) [noise=true , address=45.91.116.9:4262 , piece_id=SY6ZMA7RBXTYTTUDG7CUSUIBDQDH6KSUYQEVGOF2277W3VBHNXPQ, seg_pos={0 0}, upload=4939]: Uploading piece... Jul 19 05:03:45.116443223 (6h12m51.489022578s) [noise=true , address=77.90.31.16:4734 , piece_id=K3GMMB3LZEBGY7SK3Y4SYOCP53ZQDJE6A7QKHEAPHZJ5FPHS7SXA, seg_pos={0 0}, upload=4939]: Uploading piece... Jul 19 05:03:45.116737546 (6h12m51.489316891s) [noise=true , address=77.90.31.21:4878 , piece_id=33ICURJROGZ7AAUL2V4TF7DGUBZVFMIILOZK32JJLM7H7YSNPDDQ, seg_pos={0 0}, upload=4939]: Uploading piece... Jul 19 05:03:45.117597912 (6h12m51.490177267s) [noise=true , address=104.36.82.10:4078 , piece_id=ZVL3QTM2S6SAKLPT3BZJJUED7XEBBH72XBM6YCAWOP7EAF5VENWA, seg_pos={0 0}, upload=4939]: Uploading piece... Jul 19 05:03:45.118828717 (6h12m51.491408072s) [noise=true , address=77.90.31.12:4592 , piece_id=WNVERLNK3OAHNCRZPTJBWUZ4TGUSIZGE2MTXZN2VVGKGBLOYV5FQ, seg_pos={0 0}, upload=4939]: Uploading piece... Jul 19 05:03:45.119518137 (6h12m51.492097492s) [noise=true , address=45.91.116.14:4189 , piece_id=5VTVNCGJXELSPFW27IYSMX2AHBSOZF65K33VTL3CQCH3BD7XUKJA, seg_pos={0 0}, upload=4939]: Uploading piece... Jul 19 05:03:45.119617917 (6h12m51.492197272s) [noise=true , address=45.91.116.4:4940 , piece_id=TSPQDQ7BPAXCAWR3BRPWFICR5I2AH4NH74K2SZDYSS5TFCGMPHKQ, seg_pos={0 0}, upload=4939]: Uploading piece... Jul 19 05:03:45.119930426 (6h12m51.492509781s) [noise=true , address=104.36.82.3:4765 , piece_id=PMWXLX4WAM4PQOHOZNI7IBMD4AXZ6WPDHI6EYUWQA7X3RI7RDCKA, seg_pos={0 0}, upload=4939]: Uploading piece... Jul 19 05:03:45.120276017 (6h12m51.492855372s) [noise=true , address=104.36.82.6:4875 , piece_id=SZFDTUWZDOLEDJIF2HRIKE7O2CFC5OQAPKOQTTL45O4PTI5SNA3Q, seg_pos={0 0}, upload=4939]: Uploading piece... Jul 19 05:03:45.120553368 (6h12m51.493132723s) [noise=true , address=45.91.116.11:4117 , piece_id=R7A7PQWO3YJ4HSGJPCKYYH6OMTV3PFGSUM7QRTEMGTUQISKOYYZQ, seg_pos={0 0}, upload=4939]: Uploading piece... Jul 19 05:03:45.120849865 (6h12m51.493429221s) [noise=true , address=104.36.82.7:4910 , piece_id=UBFAZYTQKRTQK23WRUE2YQCHEDXPIZNEQ622SYKGMWW7YGWL465A, seg_pos={0 0}, upload=4939]: Uploading piece... Jul 19 05:03:45.121138487 (6h12m51.493717842s) [noise=true , address=45.91.116.5:4299 , piece_id=MTSIPD6QVY3UCK2L4ZDZ4UKMOIDMZFWQVCWFQNFLUV2LOWHP76NQ, seg_pos={0 0}, upload=4939]: Uploading piece... Jul 19 05:03:45.121423393 (6h12m51.494002748s) [noise=true , address=77.90.31.17:4770 , piece_id=6IX73G3QBSK7LLIRRV25Q2SXXZTQFSJQYVAKUTNMKBI6KSOSEB5A, seg_pos={0 0}, upload=4939]: Uploading piece... Jul 19 05:03:46.836034706 (6h12m53.208614051s) [upload=4939]: Next returned nil segment Jul 19 05:03:46.836053923 (6h12m53.208633278s) [upload=4939]: Waiting for error group managing segments... Jul 19 05:03:47.166909631 (6h12m53.539488986s) [noise=true , address=104.36.82.10:4078 , piece_id=ZVL3QTM2S6SAKLPT3BZJJUED7XEBBH72XBM6YCAWOP7EAF5VENWA, seg_pos={0 0}, upload=4939]: Done uploading piece. err: <nil> Jul 19 05:03:47.170976693 (6h12m53.543556048s) [noise=true , address=45.91.116.14:4191 , piece_id=TUJXONY7RDRBJQ32GHGYUVMHZYQNX3HGR76WVA2Q5VUQK5ELJ3FQ, seg_pos={0 0}, upload=4939]: Done uploading piece. err: <nil> Jul 19 05:03:47.175582207 (6h12m53.548161552s) [noise=true , address=45.91.116.11:4117 , piece_id=R7A7PQWO3YJ4HSGJPCKYYH6OMTV3PFGSUM7QRTEMGTUQISKOYYZQ, seg_pos={0 0}, upload=4939]: Done uploading piece. err: <nil> Jul 19 05:03:47.181418145 (6h12m53.5539975s) [noise=true , address=45.91.116.4:4940 , piece_id=TSPQDQ7BPAXCAWR3BRPWFICR5I2AH4NH74K2SZDYSS5TFCGMPHKQ, seg_pos={0 0}, upload=4939]: Done uploading piece. err: <nil> Jul 19 05:03:47.191436166 (6h12m53.564015521s) [noise=true , address=45.91.116.14:4189 , piece_id=5VTVNCGJXELSPFW27IYSMX2AHBSOZF65K33VTL3CQCH3BD7XUKJA, seg_pos={0 0}, upload=4939]: Done uploading piece. err: <nil> Jul 19 05:03:47.239695198 (6h12m53.612274553s) [noise=true , address=104.36.82.13:4186 , piece_id=4ZPHE5YJRIVLNUC3YF2JZQTM5EMIFAR5FR4F6X62KEWBCLCF66PQ, seg_pos={0 0}, upload=4939]: Done uploading piece. err: <nil> Jul 19 05:03:47.247479896 (6h12m53.620059301s) [noise=true , address=45.91.116.9:4262 , piece_id=SY6ZMA7RBXTYTTUDG7CUSUIBDQDH6KSUYQEVGOF2277W3VBHNXPQ, seg_pos={0 0}, upload=4939]: Done uploading piece. err: <nil> Jul 19 05:03:47.254180889 (6h12m53.626760244s) [noise=true , address=104.36.82.7:4910 , piece_id=UBFAZYTQKRTQK23WRUE2YQCHEDXPIZNEQ622SYKGMWW7YGWL465A, seg_pos={0 0}, upload=4939]: Done uploading piece. err: <nil> Jul 19 05:03:47.257135352 (6h12m53.629714697s) [noise=true , address=104.36.82.6:4875 , piece_id=SZFDTUWZDOLEDJIF2HRIKE7O2CFC5OQAPKOQTTL45O4PTI5SNA3Q, seg_pos={0 0}, upload=4939]: Done uploading piece. err: <nil> Jul 19 05:03:47.268904665 (6h12m53.64148402s) [noise=true , address=104.36.82.3:4765 , piece_id=PMWXLX4WAM4PQOHOZNI7IBMD4AXZ6WPDHI6EYUWQA7X3RI7RDCKA, seg_pos={0 0}, upload=4939]: Done uploading piece. err: <nil> Jul 19 05:03:47.288612540 (6h12m53.661191885s) [noise=true , address=45.91.116.5:4299 , piece_id=MTSIPD6QVY3UCK2L4ZDZ4UKMOIDMZFWQVCWFQNFLUV2LOWHP76NQ, seg_pos={0 0}, upload=4939]: Done uploading piece. err: <nil> Jul 19 05:04:13.421838111 (6h13m19.794417466s) [noise=true , address=77.90.31.12:4592 , piece_id=WNVERLNK3OAHNCRZPTJBWUZ4TGUSIZGE2MTXZN2VVGKGBLOYV5FQ, seg_pos={0 0}, upload=4939]: Done uploading piece. err: <nil> Jul 19 05:04:14.201414481 (6h13m20.573993836s) [noise=true , address=77.90.31.21:4878 , piece_id=33ICURJROGZ7AAUL2V4TF7DGUBZVFMIILOZK32JJLM7H7YSNPDDQ, seg_pos={0 0}, upload=4939]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 188.166.225.170:48628->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:48628->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48628->77.90.31.21:4878: use of closed network connection Jul 19 05:04:14.201449188 (6h13m20.574028543s) [noise=true , address=77.90.31.16:4734 , piece_id=K3GMMB3LZEBGY7SK3Y4SYOCP53ZQDJE6A7QKHEAPHZJ5FPHS7SXA, seg_pos={0 0}, upload=4939]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Jul 19 05:04:14.201455690 (6h13m20.574035045s) [noise=true , address=77.90.31.17:4770 , piece_id=6IX73G3QBSK7LLIRRV25Q2SXXZTQFSJQYVAKUTNMKBI6KSOSEB5A, seg_pos={0 0}, upload=4939]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: expected piece hash; context deadline exceeded; EOF Jul 19 05:04:14.201467092 (6h13m20.574046447s) [noise=true , address=77.90.31.13:4628 , piece_id=FCMOWACIWBGSSFVHV22Q7V2KIAY27EHE3CKIHX5KAYTZNLEHQEYA, seg_pos={0 0}, upload=4939]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:44876->77.90.31.13:4628: use of closed network connection; write tcp 188.166.225.170:44876->77.90.31.13:4628: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44876->77.90.31.13:4628: use of closed network connection Jul 19 05:04:14.201493142 (6h13m20.574072497s) [seg_pos={0 0}, upload=4939]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 19 05:04:14.442040800 (6h13m20.814620155s) [upload=4939]: Done uploading segments... Jul 19 05:04:14.442067992 (6h13m20.814647347s) [upload=4939]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.241669144s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 9.995585553s FAIL (2025-07-19 05:03:40 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 4855s) [upload=4769]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 04:51:09.109709627 (6h0m15.482288982s) [upload=4769]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 04:51:09.556145753 (6h0m15.928725108s) [seg_pos={0 0}, upload=4769]: Begin upload segment... Jul 19 04:51:09.556194817 (6h0m15.928774172s) [seg_pos={0 0}, upload=4769]: Done begin upload segment. Jul 19 04:51:09.556223893 (6h0m15.928803248s) [noise=true , address=79.127.219.38:4023 , piece_id=PTPOZGLYV4MZTOFZNN3NCGYL7DFAST4PXKEUIVTMFL7HV4GZTCLQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556252197 (6h0m15.928831552s) [noise=true , address=45.91.95.4:4011 , piece_id=ZMBAS6KKHYVIRE6OBL2NFVCWMM4TKQ7J64BD3SYXIK54XCEDI2XA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556579383 (6h0m15.929158738s) [noise=true , address=45.91.116.6:4007 , piece_id=A2AFVTZBXSGBOE7XIM3B5HRN4STITFBXO6MZRJ365URNAAOZ3PSQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556854469 (6h0m15.929433824s) [noise=true , address=104.36.82.19:4423 , piece_id=PIJUHDUND36HQL67I33D7A3EULMJHS7N55MN4OFQJ4OWNQ42O47A, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557094107 (6h0m15.929673462s) [noise=true , address=45.91.116.9:4264 , piece_id=GHM5BKZKMT2QOHHBOS4FSPOWEWSFQRGRVVR4GSNDPHVIFRZHQL2A, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557336381 (6h0m15.929915736s) [noise=true , address=79.127.205.235:4660 , piece_id=UOUOGE2VP7PX7EMRZLPMCWFCYEZERM3DQMCQ6NG4J2K2DEIZR63Q, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557557885 (6h0m15.93013724s) [noise=true , address=79.127.205.246:4280 , piece_id=FLBOZI3UGTQXAQBLZ2F45Z3OEYL65F5PQBXRF2DQMZ7NGF7VDRPA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557762296 (6h0m15.930341651s) [noise=true , address=79.127.205.233:4622 , piece_id=6WWNYQTH5MGAJOEB7BXKZDJOMCAZXDW66J3OEANGDNJDNUMLZM7Q, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558014689 (6h0m15.930594044s) [noise=true , address=64.49.13.138:4482 , piece_id=MAW2M4US2OQNOS3WI67AA7NIBTJDE5T2JQTXKFPBLGVGXKVCFGTQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558215974 (6h0m15.930795319s) [noise=true , address=77.90.31.10:4546 , piece_id=R6LCA5WZOEBDURL5S3MEK22YPVYXX6EXSS4J4W57543HELYXOKZA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558409364 (6h0m15.930988719s) [noise=true , address=45.91.95.5:4019 , piece_id=YTSBCN2LPHEMRRAZIHJGVB7NBOTRXVEHD2TWUCTTLBNFFYEWWDKA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558623504 (6h0m15.931202859s) [noise=true , address=79.127.201.215:4357 , piece_id=75XWAG2VQWEYCWN2Q3WRIZSBI7LDOISZJPXQLQ2GIPNEN34VLSUQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558805162 (6h0m15.931384517s) [noise=true , address=45.91.116.14:4219 , piece_id=PDW6JPRHCVMHHOWIHT6JKV4CTSC3JY6NDKDVFZSO47OYNPZ7PGGA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558999373 (6h0m15.931578728s) [noise=true , address=77.90.31.11:4564 , piece_id=L2U7JFJRDQOFA2AYVAGTI4ZW4VABCXKMXWBIUWQO2QNNUUBJ74BA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559184768 (6h0m15.931764123s) [noise=true , address=77.90.31.16:4768 , piece_id=YNLSONFM6COZEVF6ZNQR2EA6V4LSQFZJONIXR4AFTNNMFGPV57JA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559368719 (6h0m15.931948074s) [noise=true , address=79.127.220.98:4432 , piece_id=2SUTFQ7FVDPAM2H4W3OLW6AVH3VJ33IJFE4L52SA3ET5LGBMOMPQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559549706 (6h0m15.932129061s) [noise=true , address=77.90.31.18:4813 , piece_id=VDOT7BB4O7XYJHW6BFTAYQZZYLZU5PAUXVJGSXJ7FDI4TH425COA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559727836 (6h0m15.932307191s) [noise=true , address=79.127.219.51:4211 , piece_id=HO5PLWBVU5YAYNYMN352NA73SM4WBJV4WZ7XLSQTCQ76UAL2TOJA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:10.663359087 (6h0m17.035938442s) [upload=4769]: Next returned nil segment Jul 19 04:51:10.663378664 (6h0m17.035958019s) [upload=4769]: Waiting for error group managing segments... Jul 19 04:51:10.962945479 (6h0m17.335524834s) [noise=true , address=79.127.205.233:4622 , piece_id=6WWNYQTH5MGAJOEB7BXKZDJOMCAZXDW66J3OEANGDNJDNUMLZM7Q, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:10.967689807 (6h0m17.340269152s) [noise=true , address=79.127.205.235:4660 , piece_id=UOUOGE2VP7PX7EMRZLPMCWFCYEZERM3DQMCQ6NG4J2K2DEIZR63Q, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:10.976968631 (6h0m17.349547986s) [noise=true , address=64.49.13.138:4482 , piece_id=MAW2M4US2OQNOS3WI67AA7NIBTJDE5T2JQTXKFPBLGVGXKVCFGTQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.048750700 (6h0m17.421330055s) [noise=true , address=79.127.205.246:4280 , piece_id=FLBOZI3UGTQXAQBLZ2F45Z3OEYL65F5PQBXRF2DQMZ7NGF7VDRPA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.221642403 (6h0m17.594221758s) [noise=true , address=79.127.220.98:4432 , piece_id=2SUTFQ7FVDPAM2H4W3OLW6AVH3VJ33IJFE4L52SA3ET5LGBMOMPQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.317070435 (6h0m17.68964979s) [noise=true , address=45.91.95.4:4011 , piece_id=ZMBAS6KKHYVIRE6OBL2NFVCWMM4TKQ7J64BD3SYXIK54XCEDI2XA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.324002660 (6h0m17.696582015s) [noise=true , address=45.91.95.5:4019 , piece_id=YTSBCN2LPHEMRRAZIHJGVB7NBOTRXVEHD2TWUCTTLBNFFYEWWDKA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.460976317 (6h0m17.833555672s) [noise=true , address=79.127.201.215:4357 , piece_id=75XWAG2VQWEYCWN2Q3WRIZSBI7LDOISZJPXQLQ2GIPNEN34VLSUQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.462584603 (6h0m17.835163958s) [noise=true , address=45.91.116.6:4007 , piece_id=A2AFVTZBXSGBOE7XIM3B5HRN4STITFBXO6MZRJ365URNAAOZ3PSQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.490392973 (6h0m17.862972328s) [noise=true , address=45.91.116.9:4264 , piece_id=GHM5BKZKMT2QOHHBOS4FSPOWEWSFQRGRVVR4GSNDPHVIFRZHQL2A, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.502040780 (6h0m17.874620135s) [noise=true , address=45.91.116.14:4219 , piece_id=PDW6JPRHCVMHHOWIHT6JKV4CTSC3JY6NDKDVFZSO47OYNPZ7PGGA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.600434096 (6h0m17.973013451s) [noise=true , address=79.127.219.51:4211 , piece_id=HO5PLWBVU5YAYNYMN352NA73SM4WBJV4WZ7XLSQTCQ76UAL2TOJA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.609008984 (6h0m17.981588339s) [noise=true , address=104.36.82.19:4423 , piece_id=PIJUHDUND36HQL67I33D7A3EULMJHS7N55MN4OFQJ4OWNQ42O47A, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.620825273 (6h0m17.993404628s) [noise=true , address=79.127.219.38:4023 , piece_id=PTPOZGLYV4MZTOFZNN3NCGYL7DFAST4PXKEUIVTMFL7HV4GZTCLQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.620838659 (6h0m17.993418014s) [seg_pos={0 0}, upload=4769]: Segment reached optimal threshold of 14 pieces. Jul 19 04:51:11.621067807 (6h0m17.993647163s) [noise=true , address=77.90.31.11:4564 , piece_id=L2U7JFJRDQOFA2AYVAGTI4ZW4VABCXKMXWBIUWQO2QNNUUBJ74BA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection; write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection Jul 19 04:51:11.621134234 (6h0m17.993713589s) [noise=true , address=77.90.31.10:4546 , piece_id=R6LCA5WZOEBDURL5S3MEK22YPVYXX6EXSS4J4W57543HELYXOKZA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK): protocol: read tcp 188.166.225.170:58476->77.90.31.10:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621174191 (6h0m17.993753546s) [noise=true , address=77.90.31.16:4768 , piece_id=YNLSONFM6COZEVF6ZNQR2EA6V4LSQFZJONIXR4AFTNNMFGPV57JA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UchewmtxBrtSZSpVAJFdZmcTsQVQxkjK9re6cgUGqDhyonQeF): protocol: read tcp 188.166.225.170:56326->77.90.31.16:4768: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621216842 (6h0m17.993796197s) [noise=true , address=77.90.31.18:4813 , piece_id=VDOT7BB4O7XYJHW6BFTAYQZZYLZU5PAUXVJGSXJ7FDI4TH425COA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128WthA1x9wp1dV1vkpGnoveZQpU9CSgDCZfw5GMQuVpCNSVAM5): protocol: read tcp 188.166.225.170:49704->77.90.31.18:4813: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621234917 (6h0m17.993814272s) [seg_pos={0 0}, upload=4769]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 19 04:51:11.621301444 (6h0m17.993880799s) [upload=4769]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 19 04:51:11.621305402 (6h0m17.993884758s) [upload=4769]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 19 04:51:11.923136970 (6h0m18.295716325s) [upload=4769]: Done uploading segments... Jul 19 04:51:11.923156627 (6h0m18.295735972s) [upload=4769]: Upload finished. err: <nil> Duration: 9.995608447s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.....................................XX...X...X....X..X....X] duration: 30.245110122s FAIL (2025-07-19 05:03:06 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 4909]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 19 05:01:28.708760519 (6h10m35.081339874s) [upload=4909]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 19 05:01:29.082227724 (6h10m35.454807079s) [upload=4909]: Done uploading segments... Jul 19 05:01:29.082245619 (6h10m35.454824974s) [upload=4909]: Upload finished. err: <nil> Jul 19 05:02:37.175498145 (6h11m43.54807751s) [upload=4925]: Starting upload Jul 19 05:02:37.175628474 (6h11m43.548207829s) [upload=4925]: Done starting upload Jul 19 05:02:37.175717755 (6h11m43.54829713s) [upload=4925]: Uploading segments... Jul 19 05:02:37.176047947 (6h11m43.548627332s) [upload=4925]: Got next segment. Inline: false Jul 19 05:02:37.176090769 (6h11m43.548670154s) [upload=4925]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 05:02:37.176097472 (6h11m43.548676847s) [upload=4925]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 05:02:37.420416547 (6h11m43.792995902s) [seg_pos={0 0}, upload=4925]: Begin upload segment... Jul 19 05:02:37.420466282 (6h11m43.793045637s) [seg_pos={0 0}, upload=4925]: Done begin upload segment. Jul 19 05:02:37.420489857 (6h11m43.793069202s) [noise=true , address=77.90.31.21:4878 , piece_id=CER5K5SB2SHVA56LHYBTV4XGOETIL2TCYZFPKAS2JRNU466LEWGA, seg_pos={0 0}, upload=4925]: Uploading piece... Jul 19 05:02:37.420539963 (6h11m43.793119318s) [noise=true , address=77.90.31.15:4697 , piece_id=GTPV46CWTH2S7AR5AXS7NEOPKPLU7RPIRL3XCZRJD3WQNODZSPLA, seg_pos={0 0}, upload=4925]: Uploading piece... Jul 19 05:02:37.420887918 (6h11m43.793467273s) [noise=true , address=45.91.116.14:4190 , piece_id=S7GEO6VXAPJFU6M2LVY525SSDLUM2SNP73AI56A7ZGTXNHWS6RJA, seg_pos={0 0}, upload=4925]: Uploading piece... Jul 19 05:02:37.421075027 (6h11m43.793654382s) [noise=true , address=104.36.82.9:4007 , piece_id=6MNBAMZJ4VBJYVGPTN35J2JSKQBKXZX2JUMQHKNZA7OE36GUYEXA, seg_pos={0 0}, upload=4925]: Uploading piece... Jul 19 05:02:37.421166572 (6h11m43.793745927s) [noise=true , address=45.91.116.5:4298 , piece_id=TAA2NYNNGYVYNQ5BKJ6EBJ3UMI737LMV6SLAYGQ7FFCORK4CZS4Q, seg_pos={0 0}, upload=4925]: Uploading piece... Jul 19 05:02:37.421410068 (6h11m43.793989423s) [noise=true , address=77.90.31.15:4699 , piece_id=MR3AYIL46SZWXIRAV7JVFYEWGREP4ME4BZTV3LWTVR2TA4JBUJ3Q, seg_pos={0 0}, upload=4925]: Uploading piece... Jul 19 05:02:37.421493167 (6h11m43.794072522s) [noise=true , address=45.91.116.12:4226 , piece_id=E7XXPGDOGRSRDA3AUBZJKNEZGBXMJBFZGSM4JC4M2BTEUAKMCL5Q, seg_pos={0 0}, upload=4925]: Uploading piece... Jul 19 05:02:37.421855060 (6h11m43.794434405s) [noise=true , address=45.91.116.11:4118 , piece_id=DWY3A3QGAB7SCX473QLO55UNZSIXYZRE7NO5EPGEX4FA3JCTRQPQ, seg_pos={0 0}, upload=4925]: Uploading piece... Jul 19 05:02:37.422167978 (6h11m43.794747333s) [noise=true , address=45.91.116.9:4263 , piece_id=5RW6U4FPSZHUZITQKBOV3INKLXLOQX2SK2CCXKYMVAGJ7BMFHA3A, seg_pos={0 0}, upload=4925]: Uploading piece... Jul 19 05:02:37.422434749 (6h11m43.795014104s) [noise=true , address=77.90.31.16:4735 , piece_id=PQKOY6ZW7FGMFZOKR3Q54AZD2725ZHYVMCJG6R6DPVRFXLP6T45Q, seg_pos={0 0}, upload=4925]: Uploading piece... Jul 19 05:02:37.422642046 (6h11m43.795221401s) [noise=true , address=77.90.31.14:4662 , piece_id=F5X75EONQ5C7XJSXCF7TMILNIV4NDJKWYO52U3U2OYKBJV3MQY4A, seg_pos={0 0}, upload=4925]: Uploading piece... Jul 19 05:02:37.422903305 (6h11m43.79548266s) [noise=true , address=77.90.31.16:4734 , piece_id=E3N2BN6ICSXU7VQABIMXCGRVNPIPBZFQYT6YQGKDVO2VD2XIG3OA, seg_pos={0 0}, upload=4925]: Uploading piece... Jul 19 05:02:37.423135039 (6h11m43.795714394s) [noise=true , address=77.90.31.14:4663 , piece_id=IXQ3GNG5TP2PDVGOBAAYOCIA7SCOEXOXHSVDJJVZVXN66D2KPZMA, seg_pos={0 0}, upload=4925]: Uploading piece... Jul 19 05:02:37.423409434 (6h11m43.795988789s) [noise=true , address=45.91.116.12:4227 , piece_id=WWGQZ6F4LQB6RFU6FL2ZFG5LQNDOC6XDXBR6REXV2EFG3GZ3KGXA, seg_pos={0 0}, upload=4925]: Uploading piece... Jul 19 05:02:37.423812125 (6h11m43.79639148s) [noise=true , address=77.90.31.10:4519 , piece_id=FYTPDXUOHWNUTDET25WYASKHUTNFKA44AUR6ZCL7DU2QAUC2TSEA, seg_pos={0 0}, upload=4925]: Uploading piece... Jul 19 05:02:37.424165982 (6h11m43.796745337s) [noise=true , address=45.91.116.11:4117 , piece_id=YF4ASOSW6EX5A2N6QJCZ2JPTZH3CLKEWGRZYQSCIGOAKKVS56YRQ, seg_pos={0 0}, upload=4925]: Uploading piece... Jul 19 05:02:39.183548694 (6h11m45.556128049s) [upload=4925]: Next returned nil segment Jul 19 05:02:39.183567671 (6h11m45.556147026s) [upload=4925]: Waiting for error group managing segments... Jul 19 05:02:39.466183052 (6h11m45.838762407s) [noise=true , address=45.91.116.12:4226 , piece_id=E7XXPGDOGRSRDA3AUBZJKNEZGBXMJBFZGSM4JC4M2BTEUAKMCL5Q, seg_pos={0 0}, upload=4925]: Done uploading piece. err: <nil> Jul 19 05:02:39.469980459 (6h11m45.842559824s) [noise=true , address=45.91.116.14:4190 , piece_id=S7GEO6VXAPJFU6M2LVY525SSDLUM2SNP73AI56A7ZGTXNHWS6RJA, seg_pos={0 0}, upload=4925]: Done uploading piece. err: <nil> Jul 19 05:02:39.480368338 (6h11m45.852947693s) [noise=true , address=45.91.116.5:4298 , piece_id=TAA2NYNNGYVYNQ5BKJ6EBJ3UMI737LMV6SLAYGQ7FFCORK4CZS4Q, seg_pos={0 0}, upload=4925]: Done uploading piece. err: <nil> Jul 19 05:02:39.489731406 (6h11m45.862310761s) [noise=true , address=104.36.82.9:4007 , piece_id=6MNBAMZJ4VBJYVGPTN35J2JSKQBKXZX2JUMQHKNZA7OE36GUYEXA, seg_pos={0 0}, upload=4925]: Done uploading piece. err: <nil> Jul 19 05:02:39.553117096 (6h11m45.925696441s) [noise=true , address=45.91.116.11:4117 , piece_id=YF4ASOSW6EX5A2N6QJCZ2JPTZH3CLKEWGRZYQSCIGOAKKVS56YRQ, seg_pos={0 0}, upload=4925]: Done uploading piece. err: <nil> Jul 19 05:02:39.553961081 (6h11m45.926540436s) [noise=true , address=45.91.116.11:4118 , piece_id=DWY3A3QGAB7SCX473QLO55UNZSIXYZRE7NO5EPGEX4FA3JCTRQPQ, seg_pos={0 0}, upload=4925]: Done uploading piece. err: <nil> Jul 19 05:02:39.564133598 (6h11m45.936712953s) [noise=true , address=45.91.116.9:4263 , piece_id=5RW6U4FPSZHUZITQKBOV3INKLXLOQX2SK2CCXKYMVAGJ7BMFHA3A, seg_pos={0 0}, upload=4925]: Done uploading piece. err: <nil> Jul 19 05:02:39.564509658 (6h11m45.937089013s) [noise=true , address=45.91.116.12:4227 , piece_id=WWGQZ6F4LQB6RFU6FL2ZFG5LQNDOC6XDXBR6REXV2EFG3GZ3KGXA, seg_pos={0 0}, upload=4925]: Done uploading piece. err: <nil> Jul 19 05:02:49.175614184 (6h11m55.548193549s) [noise=true , address=77.90.31.14:4662 , piece_id=F5X75EONQ5C7XJSXCF7TMILNIV4NDJKWYO52U3U2OYKBJV3MQY4A, seg_pos={0 0}, upload=4925]: Done uploading piece. err: <nil> Jul 19 05:03:01.490571726 (6h12m7.863151081s) [noise=true , address=77.90.31.10:4519 , piece_id=FYTPDXUOHWNUTDET25WYASKHUTNFKA44AUR6ZCL7DU2QAUC2TSEA, seg_pos={0 0}, upload=4925]: Done uploading piece. err: <nil> Jul 19 05:03:03.150509176 (6h12m9.523088531s) [noise=true , address=77.90.31.15:4697 , piece_id=GTPV46CWTH2S7AR5AXS7NEOPKPLU7RPIRL3XCZRJD3WQNODZSPLA, seg_pos={0 0}, upload=4925]: Done uploading piece. err: <nil> Jul 19 05:03:04.580327203 (6h12m10.952906558s) [noise=true , address=77.90.31.21:4878 , piece_id=CER5K5SB2SHVA56LHYBTV4XGOETIL2TCYZFPKAS2JRNU466LEWGA, seg_pos={0 0}, upload=4925]: Done uploading piece. err: <nil> Jul 19 05:03:06.533087545 (6h12m12.905666901s) [noise=true , address=77.90.31.16:4735 , piece_id=PQKOY6ZW7FGMFZOKR3Q54AZD2725ZHYVMCJG6R6DPVRFXLP6T45Q, seg_pos={0 0}, upload=4925]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:46634->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:46634->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46634->77.90.31.16:4735: use of closed network connection Jul 19 05:03:06.533150576 (6h12m12.905729931s) [noise=true , address=77.90.31.16:4734 , piece_id=E3N2BN6ICSXU7VQABIMXCGRVNPIPBZFQYT6YQGKDVO2VD2XIG3OA, seg_pos={0 0}, upload=4925]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:33108->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:33108->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33108->77.90.31.16:4734: use of closed network connection Jul 19 05:03:06.533186534 (6h12m12.905765889s) [noise=true , address=77.90.31.14:4663 , piece_id=IXQ3GNG5TP2PDVGOBAAYOCIA7SCOEXOXHSVDJJVZVXN66D2KPZMA, seg_pos={0 0}, upload=4925]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: write tcp 188.166.225.170:53054->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:53054->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53054->77.90.31.14:4663: use of closed network connection Jul 19 05:03:06.533211522 (6h12m12.905790877s) [noise=true , address=77.90.31.15:4699 , piece_id=MR3AYIL46SZWXIRAV7JVFYEWGREP4ME4BZTV3LWTVR2TA4JBUJ3Q, seg_pos={0 0}, upload=4925]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:51946->77.90.31.15:4699: use of closed network connection; write tcp 188.166.225.170:51946->77.90.31.15:4699: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51946->77.90.31.15:4699: use of closed network connection Jul 19 05:03:06.533247682 (6h12m12.905827037s) [seg_pos={0 0}, upload=4925]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 19 05:03:06.775784086 (6h12m13.148363451s) [upload=4925]: Done uploading segments... Jul 19 05:03:06.775815406 (6h12m13.148394791s) [upload=4925]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.245134669s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 5.613068822s FAIL (2025-07-19 05:02:40 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 4855s) [upload=4769]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 04:51:09.109709627 (6h0m15.482288982s) [upload=4769]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 04:51:09.556145753 (6h0m15.928725108s) [seg_pos={0 0}, upload=4769]: Begin upload segment... Jul 19 04:51:09.556194817 (6h0m15.928774172s) [seg_pos={0 0}, upload=4769]: Done begin upload segment. Jul 19 04:51:09.556223893 (6h0m15.928803248s) [noise=true , address=79.127.219.38:4023 , piece_id=PTPOZGLYV4MZTOFZNN3NCGYL7DFAST4PXKEUIVTMFL7HV4GZTCLQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556252197 (6h0m15.928831552s) [noise=true , address=45.91.95.4:4011 , piece_id=ZMBAS6KKHYVIRE6OBL2NFVCWMM4TKQ7J64BD3SYXIK54XCEDI2XA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556579383 (6h0m15.929158738s) [noise=true , address=45.91.116.6:4007 , piece_id=A2AFVTZBXSGBOE7XIM3B5HRN4STITFBXO6MZRJ365URNAAOZ3PSQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556854469 (6h0m15.929433824s) [noise=true , address=104.36.82.19:4423 , piece_id=PIJUHDUND36HQL67I33D7A3EULMJHS7N55MN4OFQJ4OWNQ42O47A, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557094107 (6h0m15.929673462s) [noise=true , address=45.91.116.9:4264 , piece_id=GHM5BKZKMT2QOHHBOS4FSPOWEWSFQRGRVVR4GSNDPHVIFRZHQL2A, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557336381 (6h0m15.929915736s) [noise=true , address=79.127.205.235:4660 , piece_id=UOUOGE2VP7PX7EMRZLPMCWFCYEZERM3DQMCQ6NG4J2K2DEIZR63Q, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557557885 (6h0m15.93013724s) [noise=true , address=79.127.205.246:4280 , piece_id=FLBOZI3UGTQXAQBLZ2F45Z3OEYL65F5PQBXRF2DQMZ7NGF7VDRPA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557762296 (6h0m15.930341651s) [noise=true , address=79.127.205.233:4622 , piece_id=6WWNYQTH5MGAJOEB7BXKZDJOMCAZXDW66J3OEANGDNJDNUMLZM7Q, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558014689 (6h0m15.930594044s) [noise=true , address=64.49.13.138:4482 , piece_id=MAW2M4US2OQNOS3WI67AA7NIBTJDE5T2JQTXKFPBLGVGXKVCFGTQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558215974 (6h0m15.930795319s) [noise=true , address=77.90.31.10:4546 , piece_id=R6LCA5WZOEBDURL5S3MEK22YPVYXX6EXSS4J4W57543HELYXOKZA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558409364 (6h0m15.930988719s) [noise=true , address=45.91.95.5:4019 , piece_id=YTSBCN2LPHEMRRAZIHJGVB7NBOTRXVEHD2TWUCTTLBNFFYEWWDKA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558623504 (6h0m15.931202859s) [noise=true , address=79.127.201.215:4357 , piece_id=75XWAG2VQWEYCWN2Q3WRIZSBI7LDOISZJPXQLQ2GIPNEN34VLSUQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558805162 (6h0m15.931384517s) [noise=true , address=45.91.116.14:4219 , piece_id=PDW6JPRHCVMHHOWIHT6JKV4CTSC3JY6NDKDVFZSO47OYNPZ7PGGA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558999373 (6h0m15.931578728s) [noise=true , address=77.90.31.11:4564 , piece_id=L2U7JFJRDQOFA2AYVAGTI4ZW4VABCXKMXWBIUWQO2QNNUUBJ74BA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559184768 (6h0m15.931764123s) [noise=true , address=77.90.31.16:4768 , piece_id=YNLSONFM6COZEVF6ZNQR2EA6V4LSQFZJONIXR4AFTNNMFGPV57JA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559368719 (6h0m15.931948074s) [noise=true , address=79.127.220.98:4432 , piece_id=2SUTFQ7FVDPAM2H4W3OLW6AVH3VJ33IJFE4L52SA3ET5LGBMOMPQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559549706 (6h0m15.932129061s) [noise=true , address=77.90.31.18:4813 , piece_id=VDOT7BB4O7XYJHW6BFTAYQZZYLZU5PAUXVJGSXJ7FDI4TH425COA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559727836 (6h0m15.932307191s) [noise=true , address=79.127.219.51:4211 , piece_id=HO5PLWBVU5YAYNYMN352NA73SM4WBJV4WZ7XLSQTCQ76UAL2TOJA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:10.663359087 (6h0m17.035938442s) [upload=4769]: Next returned nil segment Jul 19 04:51:10.663378664 (6h0m17.035958019s) [upload=4769]: Waiting for error group managing segments... Jul 19 04:51:10.962945479 (6h0m17.335524834s) [noise=true , address=79.127.205.233:4622 , piece_id=6WWNYQTH5MGAJOEB7BXKZDJOMCAZXDW66J3OEANGDNJDNUMLZM7Q, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:10.967689807 (6h0m17.340269152s) [noise=true , address=79.127.205.235:4660 , piece_id=UOUOGE2VP7PX7EMRZLPMCWFCYEZERM3DQMCQ6NG4J2K2DEIZR63Q, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:10.976968631 (6h0m17.349547986s) [noise=true , address=64.49.13.138:4482 , piece_id=MAW2M4US2OQNOS3WI67AA7NIBTJDE5T2JQTXKFPBLGVGXKVCFGTQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.048750700 (6h0m17.421330055s) [noise=true , address=79.127.205.246:4280 , piece_id=FLBOZI3UGTQXAQBLZ2F45Z3OEYL65F5PQBXRF2DQMZ7NGF7VDRPA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.221642403 (6h0m17.594221758s) [noise=true , address=79.127.220.98:4432 , piece_id=2SUTFQ7FVDPAM2H4W3OLW6AVH3VJ33IJFE4L52SA3ET5LGBMOMPQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.317070435 (6h0m17.68964979s) [noise=true , address=45.91.95.4:4011 , piece_id=ZMBAS6KKHYVIRE6OBL2NFVCWMM4TKQ7J64BD3SYXIK54XCEDI2XA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.324002660 (6h0m17.696582015s) [noise=true , address=45.91.95.5:4019 , piece_id=YTSBCN2LPHEMRRAZIHJGVB7NBOTRXVEHD2TWUCTTLBNFFYEWWDKA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.460976317 (6h0m17.833555672s) [noise=true , address=79.127.201.215:4357 , piece_id=75XWAG2VQWEYCWN2Q3WRIZSBI7LDOISZJPXQLQ2GIPNEN34VLSUQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.462584603 (6h0m17.835163958s) [noise=true , address=45.91.116.6:4007 , piece_id=A2AFVTZBXSGBOE7XIM3B5HRN4STITFBXO6MZRJ365URNAAOZ3PSQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.490392973 (6h0m17.862972328s) [noise=true , address=45.91.116.9:4264 , piece_id=GHM5BKZKMT2QOHHBOS4FSPOWEWSFQRGRVVR4GSNDPHVIFRZHQL2A, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.502040780 (6h0m17.874620135s) [noise=true , address=45.91.116.14:4219 , piece_id=PDW6JPRHCVMHHOWIHT6JKV4CTSC3JY6NDKDVFZSO47OYNPZ7PGGA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.600434096 (6h0m17.973013451s) [noise=true , address=79.127.219.51:4211 , piece_id=HO5PLWBVU5YAYNYMN352NA73SM4WBJV4WZ7XLSQTCQ76UAL2TOJA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.609008984 (6h0m17.981588339s) [noise=true , address=104.36.82.19:4423 , piece_id=PIJUHDUND36HQL67I33D7A3EULMJHS7N55MN4OFQJ4OWNQ42O47A, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.620825273 (6h0m17.993404628s) [noise=true , address=79.127.219.38:4023 , piece_id=PTPOZGLYV4MZTOFZNN3NCGYL7DFAST4PXKEUIVTMFL7HV4GZTCLQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.620838659 (6h0m17.993418014s) [seg_pos={0 0}, upload=4769]: Segment reached optimal threshold of 14 pieces. Jul 19 04:51:11.621067807 (6h0m17.993647163s) [noise=true , address=77.90.31.11:4564 , piece_id=L2U7JFJRDQOFA2AYVAGTI4ZW4VABCXKMXWBIUWQO2QNNUUBJ74BA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection; write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection Jul 19 04:51:11.621134234 (6h0m17.993713589s) [noise=true , address=77.90.31.10:4546 , piece_id=R6LCA5WZOEBDURL5S3MEK22YPVYXX6EXSS4J4W57543HELYXOKZA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK): protocol: read tcp 188.166.225.170:58476->77.90.31.10:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621174191 (6h0m17.993753546s) [noise=true , address=77.90.31.16:4768 , piece_id=YNLSONFM6COZEVF6ZNQR2EA6V4LSQFZJONIXR4AFTNNMFGPV57JA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UchewmtxBrtSZSpVAJFdZmcTsQVQxkjK9re6cgUGqDhyonQeF): protocol: read tcp 188.166.225.170:56326->77.90.31.16:4768: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621216842 (6h0m17.993796197s) [noise=true , address=77.90.31.18:4813 , piece_id=VDOT7BB4O7XYJHW6BFTAYQZZYLZU5PAUXVJGSXJ7FDI4TH425COA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128WthA1x9wp1dV1vkpGnoveZQpU9CSgDCZfw5GMQuVpCNSVAM5): protocol: read tcp 188.166.225.170:49704->77.90.31.18:4813: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621234917 (6h0m17.993814272s) [seg_pos={0 0}, upload=4769]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 19 04:51:11.621301444 (6h0m17.993880799s) [upload=4769]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 19 04:51:11.621305402 (6h0m17.993884758s) [upload=4769]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 19 04:51:11.923136970 (6h0m18.295716325s) [upload=4769]: Done uploading segments... Jul 19 04:51:11.923156627 (6h0m18.295735972s) [upload=4769]: Upload finished. err: <nil> Duration: 5.613094321s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 6.01184914s FAIL (2025-07-19 05:01:59 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 4855s) [upload=4769]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 04:51:09.109709627 (6h0m15.482288982s) [upload=4769]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 04:51:09.556145753 (6h0m15.928725108s) [seg_pos={0 0}, upload=4769]: Begin upload segment... Jul 19 04:51:09.556194817 (6h0m15.928774172s) [seg_pos={0 0}, upload=4769]: Done begin upload segment. Jul 19 04:51:09.556223893 (6h0m15.928803248s) [noise=true , address=79.127.219.38:4023 , piece_id=PTPOZGLYV4MZTOFZNN3NCGYL7DFAST4PXKEUIVTMFL7HV4GZTCLQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556252197 (6h0m15.928831552s) [noise=true , address=45.91.95.4:4011 , piece_id=ZMBAS6KKHYVIRE6OBL2NFVCWMM4TKQ7J64BD3SYXIK54XCEDI2XA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556579383 (6h0m15.929158738s) [noise=true , address=45.91.116.6:4007 , piece_id=A2AFVTZBXSGBOE7XIM3B5HRN4STITFBXO6MZRJ365URNAAOZ3PSQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556854469 (6h0m15.929433824s) [noise=true , address=104.36.82.19:4423 , piece_id=PIJUHDUND36HQL67I33D7A3EULMJHS7N55MN4OFQJ4OWNQ42O47A, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557094107 (6h0m15.929673462s) [noise=true , address=45.91.116.9:4264 , piece_id=GHM5BKZKMT2QOHHBOS4FSPOWEWSFQRGRVVR4GSNDPHVIFRZHQL2A, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557336381 (6h0m15.929915736s) [noise=true , address=79.127.205.235:4660 , piece_id=UOUOGE2VP7PX7EMRZLPMCWFCYEZERM3DQMCQ6NG4J2K2DEIZR63Q, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557557885 (6h0m15.93013724s) [noise=true , address=79.127.205.246:4280 , piece_id=FLBOZI3UGTQXAQBLZ2F45Z3OEYL65F5PQBXRF2DQMZ7NGF7VDRPA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557762296 (6h0m15.930341651s) [noise=true , address=79.127.205.233:4622 , piece_id=6WWNYQTH5MGAJOEB7BXKZDJOMCAZXDW66J3OEANGDNJDNUMLZM7Q, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558014689 (6h0m15.930594044s) [noise=true , address=64.49.13.138:4482 , piece_id=MAW2M4US2OQNOS3WI67AA7NIBTJDE5T2JQTXKFPBLGVGXKVCFGTQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558215974 (6h0m15.930795319s) [noise=true , address=77.90.31.10:4546 , piece_id=R6LCA5WZOEBDURL5S3MEK22YPVYXX6EXSS4J4W57543HELYXOKZA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558409364 (6h0m15.930988719s) [noise=true , address=45.91.95.5:4019 , piece_id=YTSBCN2LPHEMRRAZIHJGVB7NBOTRXVEHD2TWUCTTLBNFFYEWWDKA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558623504 (6h0m15.931202859s) [noise=true , address=79.127.201.215:4357 , piece_id=75XWAG2VQWEYCWN2Q3WRIZSBI7LDOISZJPXQLQ2GIPNEN34VLSUQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558805162 (6h0m15.931384517s) [noise=true , address=45.91.116.14:4219 , piece_id=PDW6JPRHCVMHHOWIHT6JKV4CTSC3JY6NDKDVFZSO47OYNPZ7PGGA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558999373 (6h0m15.931578728s) [noise=true , address=77.90.31.11:4564 , piece_id=L2U7JFJRDQOFA2AYVAGTI4ZW4VABCXKMXWBIUWQO2QNNUUBJ74BA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559184768 (6h0m15.931764123s) [noise=true , address=77.90.31.16:4768 , piece_id=YNLSONFM6COZEVF6ZNQR2EA6V4LSQFZJONIXR4AFTNNMFGPV57JA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559368719 (6h0m15.931948074s) [noise=true , address=79.127.220.98:4432 , piece_id=2SUTFQ7FVDPAM2H4W3OLW6AVH3VJ33IJFE4L52SA3ET5LGBMOMPQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559549706 (6h0m15.932129061s) [noise=true , address=77.90.31.18:4813 , piece_id=VDOT7BB4O7XYJHW6BFTAYQZZYLZU5PAUXVJGSXJ7FDI4TH425COA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559727836 (6h0m15.932307191s) [noise=true , address=79.127.219.51:4211 , piece_id=HO5PLWBVU5YAYNYMN352NA73SM4WBJV4WZ7XLSQTCQ76UAL2TOJA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:10.663359087 (6h0m17.035938442s) [upload=4769]: Next returned nil segment Jul 19 04:51:10.663378664 (6h0m17.035958019s) [upload=4769]: Waiting for error group managing segments... Jul 19 04:51:10.962945479 (6h0m17.335524834s) [noise=true , address=79.127.205.233:4622 , piece_id=6WWNYQTH5MGAJOEB7BXKZDJOMCAZXDW66J3OEANGDNJDNUMLZM7Q, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:10.967689807 (6h0m17.340269152s) [noise=true , address=79.127.205.235:4660 , piece_id=UOUOGE2VP7PX7EMRZLPMCWFCYEZERM3DQMCQ6NG4J2K2DEIZR63Q, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:10.976968631 (6h0m17.349547986s) [noise=true , address=64.49.13.138:4482 , piece_id=MAW2M4US2OQNOS3WI67AA7NIBTJDE5T2JQTXKFPBLGVGXKVCFGTQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.048750700 (6h0m17.421330055s) [noise=true , address=79.127.205.246:4280 , piece_id=FLBOZI3UGTQXAQBLZ2F45Z3OEYL65F5PQBXRF2DQMZ7NGF7VDRPA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.221642403 (6h0m17.594221758s) [noise=true , address=79.127.220.98:4432 , piece_id=2SUTFQ7FVDPAM2H4W3OLW6AVH3VJ33IJFE4L52SA3ET5LGBMOMPQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.317070435 (6h0m17.68964979s) [noise=true , address=45.91.95.4:4011 , piece_id=ZMBAS6KKHYVIRE6OBL2NFVCWMM4TKQ7J64BD3SYXIK54XCEDI2XA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.324002660 (6h0m17.696582015s) [noise=true , address=45.91.95.5:4019 , piece_id=YTSBCN2LPHEMRRAZIHJGVB7NBOTRXVEHD2TWUCTTLBNFFYEWWDKA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.460976317 (6h0m17.833555672s) [noise=true , address=79.127.201.215:4357 , piece_id=75XWAG2VQWEYCWN2Q3WRIZSBI7LDOISZJPXQLQ2GIPNEN34VLSUQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.462584603 (6h0m17.835163958s) [noise=true , address=45.91.116.6:4007 , piece_id=A2AFVTZBXSGBOE7XIM3B5HRN4STITFBXO6MZRJ365URNAAOZ3PSQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.490392973 (6h0m17.862972328s) [noise=true , address=45.91.116.9:4264 , piece_id=GHM5BKZKMT2QOHHBOS4FSPOWEWSFQRGRVVR4GSNDPHVIFRZHQL2A, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.502040780 (6h0m17.874620135s) [noise=true , address=45.91.116.14:4219 , piece_id=PDW6JPRHCVMHHOWIHT6JKV4CTSC3JY6NDKDVFZSO47OYNPZ7PGGA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.600434096 (6h0m17.973013451s) [noise=true , address=79.127.219.51:4211 , piece_id=HO5PLWBVU5YAYNYMN352NA73SM4WBJV4WZ7XLSQTCQ76UAL2TOJA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.609008984 (6h0m17.981588339s) [noise=true , address=104.36.82.19:4423 , piece_id=PIJUHDUND36HQL67I33D7A3EULMJHS7N55MN4OFQJ4OWNQ42O47A, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.620825273 (6h0m17.993404628s) [noise=true , address=79.127.219.38:4023 , piece_id=PTPOZGLYV4MZTOFZNN3NCGYL7DFAST4PXKEUIVTMFL7HV4GZTCLQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.620838659 (6h0m17.993418014s) [seg_pos={0 0}, upload=4769]: Segment reached optimal threshold of 14 pieces. Jul 19 04:51:11.621067807 (6h0m17.993647163s) [noise=true , address=77.90.31.11:4564 , piece_id=L2U7JFJRDQOFA2AYVAGTI4ZW4VABCXKMXWBIUWQO2QNNUUBJ74BA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection; write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection Jul 19 04:51:11.621134234 (6h0m17.993713589s) [noise=true , address=77.90.31.10:4546 , piece_id=R6LCA5WZOEBDURL5S3MEK22YPVYXX6EXSS4J4W57543HELYXOKZA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK): protocol: read tcp 188.166.225.170:58476->77.90.31.10:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621174191 (6h0m17.993753546s) [noise=true , address=77.90.31.16:4768 , piece_id=YNLSONFM6COZEVF6ZNQR2EA6V4LSQFZJONIXR4AFTNNMFGPV57JA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UchewmtxBrtSZSpVAJFdZmcTsQVQxkjK9re6cgUGqDhyonQeF): protocol: read tcp 188.166.225.170:56326->77.90.31.16:4768: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621216842 (6h0m17.993796197s) [noise=true , address=77.90.31.18:4813 , piece_id=VDOT7BB4O7XYJHW6BFTAYQZZYLZU5PAUXVJGSXJ7FDI4TH425COA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128WthA1x9wp1dV1vkpGnoveZQpU9CSgDCZfw5GMQuVpCNSVAM5): protocol: read tcp 188.166.225.170:49704->77.90.31.18:4813: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621234917 (6h0m17.993814272s) [seg_pos={0 0}, upload=4769]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 19 04:51:11.621301444 (6h0m17.993880799s) [upload=4769]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 19 04:51:11.621305402 (6h0m17.993884758s) [upload=4769]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 19 04:51:11.923136970 (6h0m18.295716325s) [upload=4769]: Done uploading segments... Jul 19 04:51:11.923156627 (6h0m18.295735972s) [upload=4769]: Upload finished. err: <nil> Duration: 6.011880099s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 5.756132528s FAIL (2025-07-19 05:01:36 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 4855s) [upload=4769]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 04:51:09.109709627 (6h0m15.482288982s) [upload=4769]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 04:51:09.556145753 (6h0m15.928725108s) [seg_pos={0 0}, upload=4769]: Begin upload segment... Jul 19 04:51:09.556194817 (6h0m15.928774172s) [seg_pos={0 0}, upload=4769]: Done begin upload segment. Jul 19 04:51:09.556223893 (6h0m15.928803248s) [noise=true , address=79.127.219.38:4023 , piece_id=PTPOZGLYV4MZTOFZNN3NCGYL7DFAST4PXKEUIVTMFL7HV4GZTCLQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556252197 (6h0m15.928831552s) [noise=true , address=45.91.95.4:4011 , piece_id=ZMBAS6KKHYVIRE6OBL2NFVCWMM4TKQ7J64BD3SYXIK54XCEDI2XA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556579383 (6h0m15.929158738s) [noise=true , address=45.91.116.6:4007 , piece_id=A2AFVTZBXSGBOE7XIM3B5HRN4STITFBXO6MZRJ365URNAAOZ3PSQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556854469 (6h0m15.929433824s) [noise=true , address=104.36.82.19:4423 , piece_id=PIJUHDUND36HQL67I33D7A3EULMJHS7N55MN4OFQJ4OWNQ42O47A, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557094107 (6h0m15.929673462s) [noise=true , address=45.91.116.9:4264 , piece_id=GHM5BKZKMT2QOHHBOS4FSPOWEWSFQRGRVVR4GSNDPHVIFRZHQL2A, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557336381 (6h0m15.929915736s) [noise=true , address=79.127.205.235:4660 , piece_id=UOUOGE2VP7PX7EMRZLPMCWFCYEZERM3DQMCQ6NG4J2K2DEIZR63Q, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557557885 (6h0m15.93013724s) [noise=true , address=79.127.205.246:4280 , piece_id=FLBOZI3UGTQXAQBLZ2F45Z3OEYL65F5PQBXRF2DQMZ7NGF7VDRPA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557762296 (6h0m15.930341651s) [noise=true , address=79.127.205.233:4622 , piece_id=6WWNYQTH5MGAJOEB7BXKZDJOMCAZXDW66J3OEANGDNJDNUMLZM7Q, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558014689 (6h0m15.930594044s) [noise=true , address=64.49.13.138:4482 , piece_id=MAW2M4US2OQNOS3WI67AA7NIBTJDE5T2JQTXKFPBLGVGXKVCFGTQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558215974 (6h0m15.930795319s) [noise=true , address=77.90.31.10:4546 , piece_id=R6LCA5WZOEBDURL5S3MEK22YPVYXX6EXSS4J4W57543HELYXOKZA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558409364 (6h0m15.930988719s) [noise=true , address=45.91.95.5:4019 , piece_id=YTSBCN2LPHEMRRAZIHJGVB7NBOTRXVEHD2TWUCTTLBNFFYEWWDKA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558623504 (6h0m15.931202859s) [noise=true , address=79.127.201.215:4357 , piece_id=75XWAG2VQWEYCWN2Q3WRIZSBI7LDOISZJPXQLQ2GIPNEN34VLSUQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558805162 (6h0m15.931384517s) [noise=true , address=45.91.116.14:4219 , piece_id=PDW6JPRHCVMHHOWIHT6JKV4CTSC3JY6NDKDVFZSO47OYNPZ7PGGA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558999373 (6h0m15.931578728s) [noise=true , address=77.90.31.11:4564 , piece_id=L2U7JFJRDQOFA2AYVAGTI4ZW4VABCXKMXWBIUWQO2QNNUUBJ74BA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559184768 (6h0m15.931764123s) [noise=true , address=77.90.31.16:4768 , piece_id=YNLSONFM6COZEVF6ZNQR2EA6V4LSQFZJONIXR4AFTNNMFGPV57JA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559368719 (6h0m15.931948074s) [noise=true , address=79.127.220.98:4432 , piece_id=2SUTFQ7FVDPAM2H4W3OLW6AVH3VJ33IJFE4L52SA3ET5LGBMOMPQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559549706 (6h0m15.932129061s) [noise=true , address=77.90.31.18:4813 , piece_id=VDOT7BB4O7XYJHW6BFTAYQZZYLZU5PAUXVJGSXJ7FDI4TH425COA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559727836 (6h0m15.932307191s) [noise=true , address=79.127.219.51:4211 , piece_id=HO5PLWBVU5YAYNYMN352NA73SM4WBJV4WZ7XLSQTCQ76UAL2TOJA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:10.663359087 (6h0m17.035938442s) [upload=4769]: Next returned nil segment Jul 19 04:51:10.663378664 (6h0m17.035958019s) [upload=4769]: Waiting for error group managing segments... Jul 19 04:51:10.962945479 (6h0m17.335524834s) [noise=true , address=79.127.205.233:4622 , piece_id=6WWNYQTH5MGAJOEB7BXKZDJOMCAZXDW66J3OEANGDNJDNUMLZM7Q, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:10.967689807 (6h0m17.340269152s) [noise=true , address=79.127.205.235:4660 , piece_id=UOUOGE2VP7PX7EMRZLPMCWFCYEZERM3DQMCQ6NG4J2K2DEIZR63Q, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:10.976968631 (6h0m17.349547986s) [noise=true , address=64.49.13.138:4482 , piece_id=MAW2M4US2OQNOS3WI67AA7NIBTJDE5T2JQTXKFPBLGVGXKVCFGTQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.048750700 (6h0m17.421330055s) [noise=true , address=79.127.205.246:4280 , piece_id=FLBOZI3UGTQXAQBLZ2F45Z3OEYL65F5PQBXRF2DQMZ7NGF7VDRPA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.221642403 (6h0m17.594221758s) [noise=true , address=79.127.220.98:4432 , piece_id=2SUTFQ7FVDPAM2H4W3OLW6AVH3VJ33IJFE4L52SA3ET5LGBMOMPQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.317070435 (6h0m17.68964979s) [noise=true , address=45.91.95.4:4011 , piece_id=ZMBAS6KKHYVIRE6OBL2NFVCWMM4TKQ7J64BD3SYXIK54XCEDI2XA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.324002660 (6h0m17.696582015s) [noise=true , address=45.91.95.5:4019 , piece_id=YTSBCN2LPHEMRRAZIHJGVB7NBOTRXVEHD2TWUCTTLBNFFYEWWDKA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.460976317 (6h0m17.833555672s) [noise=true , address=79.127.201.215:4357 , piece_id=75XWAG2VQWEYCWN2Q3WRIZSBI7LDOISZJPXQLQ2GIPNEN34VLSUQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.462584603 (6h0m17.835163958s) [noise=true , address=45.91.116.6:4007 , piece_id=A2AFVTZBXSGBOE7XIM3B5HRN4STITFBXO6MZRJ365URNAAOZ3PSQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.490392973 (6h0m17.862972328s) [noise=true , address=45.91.116.9:4264 , piece_id=GHM5BKZKMT2QOHHBOS4FSPOWEWSFQRGRVVR4GSNDPHVIFRZHQL2A, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.502040780 (6h0m17.874620135s) [noise=true , address=45.91.116.14:4219 , piece_id=PDW6JPRHCVMHHOWIHT6JKV4CTSC3JY6NDKDVFZSO47OYNPZ7PGGA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.600434096 (6h0m17.973013451s) [noise=true , address=79.127.219.51:4211 , piece_id=HO5PLWBVU5YAYNYMN352NA73SM4WBJV4WZ7XLSQTCQ76UAL2TOJA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.609008984 (6h0m17.981588339s) [noise=true , address=104.36.82.19:4423 , piece_id=PIJUHDUND36HQL67I33D7A3EULMJHS7N55MN4OFQJ4OWNQ42O47A, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.620825273 (6h0m17.993404628s) [noise=true , address=79.127.219.38:4023 , piece_id=PTPOZGLYV4MZTOFZNN3NCGYL7DFAST4PXKEUIVTMFL7HV4GZTCLQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.620838659 (6h0m17.993418014s) [seg_pos={0 0}, upload=4769]: Segment reached optimal threshold of 14 pieces. Jul 19 04:51:11.621067807 (6h0m17.993647163s) [noise=true , address=77.90.31.11:4564 , piece_id=L2U7JFJRDQOFA2AYVAGTI4ZW4VABCXKMXWBIUWQO2QNNUUBJ74BA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection; write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection Jul 19 04:51:11.621134234 (6h0m17.993713589s) [noise=true , address=77.90.31.10:4546 , piece_id=R6LCA5WZOEBDURL5S3MEK22YPVYXX6EXSS4J4W57543HELYXOKZA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK): protocol: read tcp 188.166.225.170:58476->77.90.31.10:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621174191 (6h0m17.993753546s) [noise=true , address=77.90.31.16:4768 , piece_id=YNLSONFM6COZEVF6ZNQR2EA6V4LSQFZJONIXR4AFTNNMFGPV57JA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UchewmtxBrtSZSpVAJFdZmcTsQVQxkjK9re6cgUGqDhyonQeF): protocol: read tcp 188.166.225.170:56326->77.90.31.16:4768: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621216842 (6h0m17.993796197s) [noise=true , address=77.90.31.18:4813 , piece_id=VDOT7BB4O7XYJHW6BFTAYQZZYLZU5PAUXVJGSXJ7FDI4TH425COA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128WthA1x9wp1dV1vkpGnoveZQpU9CSgDCZfw5GMQuVpCNSVAM5): protocol: read tcp 188.166.225.170:49704->77.90.31.18:4813: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621234917 (6h0m17.993814272s) [seg_pos={0 0}, upload=4769]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 19 04:51:11.621301444 (6h0m17.993880799s) [upload=4769]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 19 04:51:11.621305402 (6h0m17.993884758s) [upload=4769]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 19 04:51:11.923136970 (6h0m18.295716325s) [upload=4769]: Done uploading segments... Jul 19 04:51:11.923156627 (6h0m18.295735972s) [upload=4769]: Upload finished. err: <nil> Duration: 5.756145573s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [......................................XX...X...X....X..X....] duration: 7.497508596s FAIL (2025-07-19 05:01:31 UTC) orbiter warning: ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== :1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: read tcp 188.166.225.170:50230->77.90.31.19:4841: use of closed network connection; context canceled Jul 19 04:56:54.832707046 (6h6m1.205286401s) [seg_pos={0 0}, upload=4846]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 19 04:56:54.832779073 (6h6m1.205358428s) [upload=4846]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 19 04:56:54.832783182 (6h6m1.205362527s) [upload=4846]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 19 04:56:55.133341022 (6h6m1.505920377s) [upload=4846]: Done uploading segments... Jul 19 04:56:55.133358805 (6h6m1.50593816s) [upload=4846]: Upload finished. err: <nil> Jul 19 04:57:50.449535830 (6h6m56.822115185s) [upload=4860]: Starting upload Jul 19 04:57:50.449598590 (6h6m56.822177935s) [upload=4860]: Done starting upload Jul 19 04:57:50.449668684 (6h6m56.822248039s) [upload=4860]: Uploading segments... Jul 19 04:57:50.449781009 (6h6m56.822360364s) [upload=4860]: Got next segment. Inline: false Jul 19 04:57:50.449828921 (6h6m56.822408276s) [upload=4860]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 04:57:50.449833700 (6h6m56.822413055s) [upload=4860]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 04:57:50.682857520 (6h6m57.055436875s) [seg_pos={0 0}, upload=4860]: Begin upload segment... Jul 19 04:57:50.682898699 (6h6m57.055478054s) [seg_pos={0 0}, upload=4860]: Done begin upload segment. Jul 19 04:57:50.682913377 (6h6m57.055492732s) [noise=true , address=45.91.116.6:4974 , piece_id=2IEHPCFHSH7LGYE4M6UER42T22OPGFVWEGGQ6CIZ2FZ2HP3YAZWA, seg_pos={0 0}, upload=4860]: Uploading piece... Jul 19 04:57:50.682920360 (6h6m57.055499705s) [noise=true , address=77.90.31.11:4553 , piece_id=ECHSCGV2FSRCWW5M4QKGZMCQ4AHFCIUFBWPRURFCPFJR5YUOC42A, seg_pos={0 0}, upload=4860]: Uploading piece... Jul 19 04:57:50.682990905 (6h6m57.05557026s) [noise=true , address=104.36.82.16:4294 , piece_id=4HRMC3IOCWF5I252RH6JTAANPYLX4QFM3UCKXOD67BTAUJ6EDOCQ, seg_pos={0 0}, upload=4860]: Uploading piece... Jul 19 04:57:50.683253236 (6h6m57.055832591s) [noise=true , address=104.36.82.5:4839 , piece_id=W3C5L2U6NK2DYJSOIFNC2PF2XFXEOHAAWABP7UA3KM7LIXAWSPSQ, seg_pos={0 0}, upload=4860]: Uploading piece... Jul 19 04:57:50.683500520 (6h6m57.056079875s) [noise=true , address=77.90.31.17:4771 , piece_id=GQQ5R52KDOOI5WOXKTJKZXTIECNU3QWDM4KLGOR7UMORDPMHJUJQ, seg_pos={0 0}, upload=4860]: Uploading piece... Jul 19 04:57:50.683736031 (6h6m57.056315376s) [noise=true , address=45.91.116.4:4938 , piece_id=REUW3UKOOZX56BI3V5ALKCID53EIRRDBMSA2BHCLMQSZGTYJGMDA, seg_pos={0 0}, upload=4860]: Uploading piece... Jul 19 04:57:50.684122380 (6h6m57.056701725s) [noise=true , address=45.91.116.7:4011 , piece_id=LI35TO3RUCTEYICC7HUI6P34Z2KJH3S43L2GRAE646FAVXNHRXWA, seg_pos={0 0}, upload=4860]: Uploading piece... Jul 19 04:57:50.684336981 (6h6m57.056916336s) [noise=true , address=104.36.82.10:4077 , piece_id=ZCLW6MI23GWBBSZVV44LHDG4BBLKB656IJCNCVC2PGGVFJK4KV6Q, seg_pos={0 0}, upload=4860]: Uploading piece... Jul 19 04:57:50.684555689 (6h6m57.057135044s) [noise=true , address=104.36.82.3:4767 , piece_id=MNQW4WEQRFJQXUKITTVSOQN4J3T7AEXPYN37Z6OMFLFOASRQKZYA, seg_pos={0 0}, upload=4860]: Uploading piece... Jul 19 04:57:50.684820506 (6h6m57.057399861s) [noise=true , address=45.91.116.5:4298 , piece_id=AUGUELVJFWXEPSOEWRCGZM6X4TI2JOUQNYP6WP4VL25ZC7FAPRTQ, seg_pos={0 0}, upload=4860]: Uploading piece... Jul 19 04:57:50.685013646 (6h6m57.057593001s) [noise=true , address=45.91.116.5:4299 , piece_id=RPRYWZHCNVP72ZBR4BGMCOI245RG7ZEIWKVCNX3HRZVODGJIM26A, seg_pos={0 0}, upload=4860]: Uploading piece... Jul 19 04:57:50.685194391 (6h6m57.057773746s) [noise=true , address=45.91.116.9:4263 , piece_id=CPEMTRRYLTUFXTAHXB7P2THNI7T5LZCFO26XPIGATVNH2UWQH6PA, seg_pos={0 0}, upload=4860]: Uploading piece... Jul 19 04:57:50.685376059 (6h6m57.057955414s) [noise=true , address=45.91.116.10:4081 , piece_id=IN6OJGRO2YDSGNIY3ZNTYTVYLEWKHFKOT24PADM7K7YQNXK5BTEA, seg_pos={0 0}, upload=4860]: Uploading piece... Jul 19 04:57:50.685559630 (6h6m57.058138985s) [noise=true , address=77.90.31.14:4661 , piece_id=X6UIHHUTOKC6R3BIJXZETTR4EBDJ4EHQYCYJVG3KLSAPVMZZMADA, seg_pos={0 0}, upload=4860]: Uploading piece... Jul 19 04:57:50.685741949 (6h6m57.058321304s) [noise=true , address=77.90.31.18:4806 , piece_id=MYZZEPG546S5I67HCIBSNMTK2UGT7XOBMTGYPTPZAIMNLA6TWAKA, seg_pos={0 0}, upload=4860]: Uploading piece... Jul 19 04:57:50.685953323 (6h6m57.058532678s) [noise=true , address=45.91.116.9:4261 , piece_id=6GLY2NCWLS5X6IHCVYB76Z4F2UJ47GXH5HJ6BUBUU7DVW5BT5I5Q, seg_pos={0 0}, upload=4860]: Uploading piece... Jul 19 04:57:52.382568857 (6h6m58.755148202s) [upload=4860]: Next returned nil segment Jul 19 04:57:52.382588915 (6h6m58.75516827s) [upload=4860]: Waiting for error group managing segments... Jul 19 04:57:52.712870771 (6h6m59.085450116s) [noise=true , address=45.91.116.5:4298 , piece_id=AUGUELVJFWXEPSOEWRCGZM6X4TI2JOUQNYP6WP4VL25ZC7FAPRTQ, seg_pos={0 0}, upload=4860]: Done uploading piece. err: <nil> Jul 19 04:57:52.713384304 (6h6m59.085963649s) [noise=true , address=45.91.116.9:4263 , piece_id=CPEMTRRYLTUFXTAHXB7P2THNI7T5LZCFO26XPIGATVNH2UWQH6PA, seg_pos={0 0}, upload=4860]: Done uploading piece. err: <nil> Jul 19 04:57:52.719689228 (6h6m59.092268583s) [noise=true , address=104.36.82.5:4839 , piece_id=W3C5L2U6NK2DYJSOIFNC2PF2XFXEOHAAWABP7UA3KM7LIXAWSPSQ, seg_pos={0 0}, upload=4860]: Done uploading piece. err: <nil> Jul 19 04:57:52.723321257 (6h6m59.095900612s) [noise=true , address=104.36.82.10:4077 , piece_id=ZCLW6MI23GWBBSZVV44LHDG4BBLKB656IJCNCVC2PGGVFJK4KV6Q, seg_pos={0 0}, upload=4860]: Done uploading piece. err: <nil> Jul 19 04:57:52.797624770 (6h6m59.170204115s) [noise=true , address=45.91.116.6:4974 , piece_id=2IEHPCFHSH7LGYE4M6UER42T22OPGFVWEGGQ6CIZ2FZ2HP3YAZWA, seg_pos={0 0}, upload=4860]: Done uploading piece. err: <nil> Jul 19 04:57:52.798066866 (6h6m59.170646221s) [noise=true , address=45.91.116.4:4938 , piece_id=REUW3UKOOZX56BI3V5ALKCID53EIRRDBMSA2BHCLMQSZGTYJGMDA, seg_pos={0 0}, upload=4860]: Done uploading piece. err: <nil> Jul 19 04:57:52.820454393 (6h6m59.193033738s) [noise=true , address=104.36.82.16:4294 , piece_id=4HRMC3IOCWF5I252RH6JTAANPYLX4QFM3UCKXOD67BTAUJ6EDOCQ, seg_pos={0 0}, upload=4860]: Done uploading piece. err: <nil> Jul 19 04:57:52.864818369 (6h6m59.237397724s) [noise=true , address=45.91.116.10:4081 , piece_id=IN6OJGRO2YDSGNIY3ZNTYTVYLEWKHFKOT24PADM7K7YQNXK5BTEA, seg_pos={0 0}, upload=4860]: Done uploading piece. err: <nil> Jul 19 04:57:52.873179036 (6h6m59.245758391s) [noise=true , address=45.91.116.9:4261 , piece_id=6GLY2NCWLS5X6IHCVYB76Z4F2UJ47GXH5HJ6BUBUU7DVW5BT5I5Q, seg_pos={0 0}, upload=4860]: Done uploading piece. err: <nil> Jul 19 04:57:52.881637702 (6h6m59.254217047s) [noise=true , address=45.91.116.5:4299 , piece_id=RPRYWZHCNVP72ZBR4BGMCOI245RG7ZEIWKVCNX3HRZVODGJIM26A, seg_pos={0 0}, upload=4860]: Done uploading piece. err: <nil> Jul 19 04:57:52.884454370 (6h6m59.257033725s) [noise=true , address=104.36.82.3:4767 , piece_id=MNQW4WEQRFJQXUKITTVSOQN4J3T7AEXPYN37Z6OMFLFOASRQKZYA, seg_pos={0 0}, upload=4860]: Done uploading piece. err: <nil> Jul 19 04:57:54.501916491 (6h7m0.874495846s) [noise=true , address=45.91.116.7:4011 , piece_id=LI35TO3RUCTEYICC7HUI6P34Z2KJH3S43L2GRAE646FAVXNHRXWA, seg_pos={0 0}, upload=4860]: Done uploading piece. err: <nil> Jul 19 04:58:19.391729500 (6h7m25.764308845s) [noise=true , address=77.90.31.11:4553 , piece_id=ECHSCGV2FSRCWW5M4QKGZMCQ4AHFCIUFBWPRURFCPFJR5YUOC42A, seg_pos={0 0}, upload=4860]: Done uploading piece. err: <nil> Jul 19 04:58:19.391756702 (6h7m25.764336047s) [seg_pos={0 0}, upload=4860]: Segment reached optimal threshold of 13 pieces. Jul 19 04:58:19.391845321 (6h7m25.764424676s) [noise=true , address=77.90.31.14:4661 , piece_id=X6UIHHUTOKC6R3BIJXZETTR4EBDJ4EHQYCYJVG3KLSAPVMZZMADA, seg_pos={0 0}, upload=4860]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Jul 19 04:58:19.392247070 (6h7m25.764826425s) [noise=true , address=77.90.31.18:4806 , piece_id=MYZZEPG546S5I67HCIBSNMTK2UGT7XOBMTGYPTPZAIMNLA6TWAKA, seg_pos={0 0}, upload=4860]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: write tcp 188.166.225.170:45822->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:45822->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45822->77.90.31.18:4806: use of closed network connection Jul 19 04:58:19.392314228 (6h7m25.764893583s) [noise=true , address=77.90.31.17:4771 , piece_id=GQQ5R52KDOOI5WOXKTJKZXTIECNU3QWDM4KLGOR7UMORDPMHJUJQ, seg_pos={0 0}, upload=4860]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2): protocol: write tcp 188.166.225.170:46240->77.90.31.17:4771: use of closed network connection; write tcp 188.166.225.170:46240->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46240->77.90.31.17:4771: use of closed network connection Jul 19 04:58:19.392340238 (6h7m25.764919593s) [seg_pos={0 0}, upload=4860]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 19 04:58:19.392427785 (6h7m25.76500714s) [upload=4860]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 19 04:58:19.392432124 (6h7m25.76501148s) [upload=4860]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 19 04:58:19.702762764 (6h7m26.075342109s) [upload=4860]: Done uploading segments... Jul 19 04:58:19.702781199 (6h7m26.075360554s) [upload=4860]: Upload finished. err: <nil> Duration: 7.497537722s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 6.002314197s FAIL (2025-07-19 05:01:20 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 4855s) [upload=4769]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 04:51:09.109709627 (6h0m15.482288982s) [upload=4769]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 04:51:09.556145753 (6h0m15.928725108s) [seg_pos={0 0}, upload=4769]: Begin upload segment... Jul 19 04:51:09.556194817 (6h0m15.928774172s) [seg_pos={0 0}, upload=4769]: Done begin upload segment. Jul 19 04:51:09.556223893 (6h0m15.928803248s) [noise=true , address=79.127.219.38:4023 , piece_id=PTPOZGLYV4MZTOFZNN3NCGYL7DFAST4PXKEUIVTMFL7HV4GZTCLQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556252197 (6h0m15.928831552s) [noise=true , address=45.91.95.4:4011 , piece_id=ZMBAS6KKHYVIRE6OBL2NFVCWMM4TKQ7J64BD3SYXIK54XCEDI2XA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556579383 (6h0m15.929158738s) [noise=true , address=45.91.116.6:4007 , piece_id=A2AFVTZBXSGBOE7XIM3B5HRN4STITFBXO6MZRJ365URNAAOZ3PSQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556854469 (6h0m15.929433824s) [noise=true , address=104.36.82.19:4423 , piece_id=PIJUHDUND36HQL67I33D7A3EULMJHS7N55MN4OFQJ4OWNQ42O47A, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557094107 (6h0m15.929673462s) [noise=true , address=45.91.116.9:4264 , piece_id=GHM5BKZKMT2QOHHBOS4FSPOWEWSFQRGRVVR4GSNDPHVIFRZHQL2A, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557336381 (6h0m15.929915736s) [noise=true , address=79.127.205.235:4660 , piece_id=UOUOGE2VP7PX7EMRZLPMCWFCYEZERM3DQMCQ6NG4J2K2DEIZR63Q, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557557885 (6h0m15.93013724s) [noise=true , address=79.127.205.246:4280 , piece_id=FLBOZI3UGTQXAQBLZ2F45Z3OEYL65F5PQBXRF2DQMZ7NGF7VDRPA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557762296 (6h0m15.930341651s) [noise=true , address=79.127.205.233:4622 , piece_id=6WWNYQTH5MGAJOEB7BXKZDJOMCAZXDW66J3OEANGDNJDNUMLZM7Q, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558014689 (6h0m15.930594044s) [noise=true , address=64.49.13.138:4482 , piece_id=MAW2M4US2OQNOS3WI67AA7NIBTJDE5T2JQTXKFPBLGVGXKVCFGTQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558215974 (6h0m15.930795319s) [noise=true , address=77.90.31.10:4546 , piece_id=R6LCA5WZOEBDURL5S3MEK22YPVYXX6EXSS4J4W57543HELYXOKZA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558409364 (6h0m15.930988719s) [noise=true , address=45.91.95.5:4019 , piece_id=YTSBCN2LPHEMRRAZIHJGVB7NBOTRXVEHD2TWUCTTLBNFFYEWWDKA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558623504 (6h0m15.931202859s) [noise=true , address=79.127.201.215:4357 , piece_id=75XWAG2VQWEYCWN2Q3WRIZSBI7LDOISZJPXQLQ2GIPNEN34VLSUQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558805162 (6h0m15.931384517s) [noise=true , address=45.91.116.14:4219 , piece_id=PDW6JPRHCVMHHOWIHT6JKV4CTSC3JY6NDKDVFZSO47OYNPZ7PGGA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558999373 (6h0m15.931578728s) [noise=true , address=77.90.31.11:4564 , piece_id=L2U7JFJRDQOFA2AYVAGTI4ZW4VABCXKMXWBIUWQO2QNNUUBJ74BA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559184768 (6h0m15.931764123s) [noise=true , address=77.90.31.16:4768 , piece_id=YNLSONFM6COZEVF6ZNQR2EA6V4LSQFZJONIXR4AFTNNMFGPV57JA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559368719 (6h0m15.931948074s) [noise=true , address=79.127.220.98:4432 , piece_id=2SUTFQ7FVDPAM2H4W3OLW6AVH3VJ33IJFE4L52SA3ET5LGBMOMPQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559549706 (6h0m15.932129061s) [noise=true , address=77.90.31.18:4813 , piece_id=VDOT7BB4O7XYJHW6BFTAYQZZYLZU5PAUXVJGSXJ7FDI4TH425COA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559727836 (6h0m15.932307191s) [noise=true , address=79.127.219.51:4211 , piece_id=HO5PLWBVU5YAYNYMN352NA73SM4WBJV4WZ7XLSQTCQ76UAL2TOJA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:10.663359087 (6h0m17.035938442s) [upload=4769]: Next returned nil segment Jul 19 04:51:10.663378664 (6h0m17.035958019s) [upload=4769]: Waiting for error group managing segments... Jul 19 04:51:10.962945479 (6h0m17.335524834s) [noise=true , address=79.127.205.233:4622 , piece_id=6WWNYQTH5MGAJOEB7BXKZDJOMCAZXDW66J3OEANGDNJDNUMLZM7Q, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:10.967689807 (6h0m17.340269152s) [noise=true , address=79.127.205.235:4660 , piece_id=UOUOGE2VP7PX7EMRZLPMCWFCYEZERM3DQMCQ6NG4J2K2DEIZR63Q, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:10.976968631 (6h0m17.349547986s) [noise=true , address=64.49.13.138:4482 , piece_id=MAW2M4US2OQNOS3WI67AA7NIBTJDE5T2JQTXKFPBLGVGXKVCFGTQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.048750700 (6h0m17.421330055s) [noise=true , address=79.127.205.246:4280 , piece_id=FLBOZI3UGTQXAQBLZ2F45Z3OEYL65F5PQBXRF2DQMZ7NGF7VDRPA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.221642403 (6h0m17.594221758s) [noise=true , address=79.127.220.98:4432 , piece_id=2SUTFQ7FVDPAM2H4W3OLW6AVH3VJ33IJFE4L52SA3ET5LGBMOMPQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.317070435 (6h0m17.68964979s) [noise=true , address=45.91.95.4:4011 , piece_id=ZMBAS6KKHYVIRE6OBL2NFVCWMM4TKQ7J64BD3SYXIK54XCEDI2XA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.324002660 (6h0m17.696582015s) [noise=true , address=45.91.95.5:4019 , piece_id=YTSBCN2LPHEMRRAZIHJGVB7NBOTRXVEHD2TWUCTTLBNFFYEWWDKA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.460976317 (6h0m17.833555672s) [noise=true , address=79.127.201.215:4357 , piece_id=75XWAG2VQWEYCWN2Q3WRIZSBI7LDOISZJPXQLQ2GIPNEN34VLSUQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.462584603 (6h0m17.835163958s) [noise=true , address=45.91.116.6:4007 , piece_id=A2AFVTZBXSGBOE7XIM3B5HRN4STITFBXO6MZRJ365URNAAOZ3PSQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.490392973 (6h0m17.862972328s) [noise=true , address=45.91.116.9:4264 , piece_id=GHM5BKZKMT2QOHHBOS4FSPOWEWSFQRGRVVR4GSNDPHVIFRZHQL2A, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.502040780 (6h0m17.874620135s) [noise=true , address=45.91.116.14:4219 , piece_id=PDW6JPRHCVMHHOWIHT6JKV4CTSC3JY6NDKDVFZSO47OYNPZ7PGGA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.600434096 (6h0m17.973013451s) [noise=true , address=79.127.219.51:4211 , piece_id=HO5PLWBVU5YAYNYMN352NA73SM4WBJV4WZ7XLSQTCQ76UAL2TOJA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.609008984 (6h0m17.981588339s) [noise=true , address=104.36.82.19:4423 , piece_id=PIJUHDUND36HQL67I33D7A3EULMJHS7N55MN4OFQJ4OWNQ42O47A, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.620825273 (6h0m17.993404628s) [noise=true , address=79.127.219.38:4023 , piece_id=PTPOZGLYV4MZTOFZNN3NCGYL7DFAST4PXKEUIVTMFL7HV4GZTCLQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.620838659 (6h0m17.993418014s) [seg_pos={0 0}, upload=4769]: Segment reached optimal threshold of 14 pieces. Jul 19 04:51:11.621067807 (6h0m17.993647163s) [noise=true , address=77.90.31.11:4564 , piece_id=L2U7JFJRDQOFA2AYVAGTI4ZW4VABCXKMXWBIUWQO2QNNUUBJ74BA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection; write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection Jul 19 04:51:11.621134234 (6h0m17.993713589s) [noise=true , address=77.90.31.10:4546 , piece_id=R6LCA5WZOEBDURL5S3MEK22YPVYXX6EXSS4J4W57543HELYXOKZA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK): protocol: read tcp 188.166.225.170:58476->77.90.31.10:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621174191 (6h0m17.993753546s) [noise=true , address=77.90.31.16:4768 , piece_id=YNLSONFM6COZEVF6ZNQR2EA6V4LSQFZJONIXR4AFTNNMFGPV57JA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UchewmtxBrtSZSpVAJFdZmcTsQVQxkjK9re6cgUGqDhyonQeF): protocol: read tcp 188.166.225.170:56326->77.90.31.16:4768: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621216842 (6h0m17.993796197s) [noise=true , address=77.90.31.18:4813 , piece_id=VDOT7BB4O7XYJHW6BFTAYQZZYLZU5PAUXVJGSXJ7FDI4TH425COA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128WthA1x9wp1dV1vkpGnoveZQpU9CSgDCZfw5GMQuVpCNSVAM5): protocol: read tcp 188.166.225.170:49704->77.90.31.18:4813: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621234917 (6h0m17.993814272s) [seg_pos={0 0}, upload=4769]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 19 04:51:11.621301444 (6h0m17.993880799s) [upload=4769]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 19 04:51:11.621305402 (6h0m17.993884758s) [upload=4769]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 19 04:51:11.923136970 (6h0m18.295716325s) [upload=4769]: Done uploading segments... Jul 19 04:51:11.923156627 (6h0m18.295735972s) [upload=4769]: Upload finished. err: <nil> Duration: 6.002342491s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.......................................XX...X...X....X..X...] duration: 9.541475166s FAIL (2025-07-19 05:01:06 UTC) orbiter warning: ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== :1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: read tcp 188.166.225.170:50230->77.90.31.19:4841: use of closed network connection; context canceled Jul 19 04:56:54.832707046 (6h6m1.205286401s) [seg_pos={0 0}, upload=4846]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 19 04:56:54.832779073 (6h6m1.205358428s) [upload=4846]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 19 04:56:54.832783182 (6h6m1.205362527s) [upload=4846]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 19 04:56:55.133341022 (6h6m1.505920377s) [upload=4846]: Done uploading segments... Jul 19 04:56:55.133358805 (6h6m1.50593816s) [upload=4846]: Upload finished. err: <nil> Jul 19 04:57:50.449535830 (6h6m56.822115185s) [upload=4860]: Starting upload Jul 19 04:57:50.449598590 (6h6m56.822177935s) [upload=4860]: Done starting upload Jul 19 04:57:50.449668684 (6h6m56.822248039s) [upload=4860]: Uploading segments... Jul 19 04:57:50.449781009 (6h6m56.822360364s) [upload=4860]: Got next segment. Inline: false Jul 19 04:57:50.449828921 (6h6m56.822408276s) [upload=4860]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 04:57:50.449833700 (6h6m56.822413055s) [upload=4860]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 04:57:50.682857520 (6h6m57.055436875s) [seg_pos={0 0}, upload=4860]: Begin upload segment... Jul 19 04:57:50.682898699 (6h6m57.055478054s) [seg_pos={0 0}, upload=4860]: Done begin upload segment. Jul 19 04:57:50.682913377 (6h6m57.055492732s) [noise=true , address=45.91.116.6:4974 , piece_id=2IEHPCFHSH7LGYE4M6UER42T22OPGFVWEGGQ6CIZ2FZ2HP3YAZWA, seg_pos={0 0}, upload=4860]: Uploading piece... Jul 19 04:57:50.682920360 (6h6m57.055499705s) [noise=true , address=77.90.31.11:4553 , piece_id=ECHSCGV2FSRCWW5M4QKGZMCQ4AHFCIUFBWPRURFCPFJR5YUOC42A, seg_pos={0 0}, upload=4860]: Uploading piece... Jul 19 04:57:50.682990905 (6h6m57.05557026s) [noise=true , address=104.36.82.16:4294 , piece_id=4HRMC3IOCWF5I252RH6JTAANPYLX4QFM3UCKXOD67BTAUJ6EDOCQ, seg_pos={0 0}, upload=4860]: Uploading piece... Jul 19 04:57:50.683253236 (6h6m57.055832591s) [noise=true , address=104.36.82.5:4839 , piece_id=W3C5L2U6NK2DYJSOIFNC2PF2XFXEOHAAWABP7UA3KM7LIXAWSPSQ, seg_pos={0 0}, upload=4860]: Uploading piece... Jul 19 04:57:50.683500520 (6h6m57.056079875s) [noise=true , address=77.90.31.17:4771 , piece_id=GQQ5R52KDOOI5WOXKTJKZXTIECNU3QWDM4KLGOR7UMORDPMHJUJQ, seg_pos={0 0}, upload=4860]: Uploading piece... Jul 19 04:57:50.683736031 (6h6m57.056315376s) [noise=true , address=45.91.116.4:4938 , piece_id=REUW3UKOOZX56BI3V5ALKCID53EIRRDBMSA2BHCLMQSZGTYJGMDA, seg_pos={0 0}, upload=4860]: Uploading piece... Jul 19 04:57:50.684122380 (6h6m57.056701725s) [noise=true , address=45.91.116.7:4011 , piece_id=LI35TO3RUCTEYICC7HUI6P34Z2KJH3S43L2GRAE646FAVXNHRXWA, seg_pos={0 0}, upload=4860]: Uploading piece... Jul 19 04:57:50.684336981 (6h6m57.056916336s) [noise=true , address=104.36.82.10:4077 , piece_id=ZCLW6MI23GWBBSZVV44LHDG4BBLKB656IJCNCVC2PGGVFJK4KV6Q, seg_pos={0 0}, upload=4860]: Uploading piece... Jul 19 04:57:50.684555689 (6h6m57.057135044s) [noise=true , address=104.36.82.3:4767 , piece_id=MNQW4WEQRFJQXUKITTVSOQN4J3T7AEXPYN37Z6OMFLFOASRQKZYA, seg_pos={0 0}, upload=4860]: Uploading piece... Jul 19 04:57:50.684820506 (6h6m57.057399861s) [noise=true , address=45.91.116.5:4298 , piece_id=AUGUELVJFWXEPSOEWRCGZM6X4TI2JOUQNYP6WP4VL25ZC7FAPRTQ, seg_pos={0 0}, upload=4860]: Uploading piece... Jul 19 04:57:50.685013646 (6h6m57.057593001s) [noise=true , address=45.91.116.5:4299 , piece_id=RPRYWZHCNVP72ZBR4BGMCOI245RG7ZEIWKVCNX3HRZVODGJIM26A, seg_pos={0 0}, upload=4860]: Uploading piece... Jul 19 04:57:50.685194391 (6h6m57.057773746s) [noise=true , address=45.91.116.9:4263 , piece_id=CPEMTRRYLTUFXTAHXB7P2THNI7T5LZCFO26XPIGATVNH2UWQH6PA, seg_pos={0 0}, upload=4860]: Uploading piece... Jul 19 04:57:50.685376059 (6h6m57.057955414s) [noise=true , address=45.91.116.10:4081 , piece_id=IN6OJGRO2YDSGNIY3ZNTYTVYLEWKHFKOT24PADM7K7YQNXK5BTEA, seg_pos={0 0}, upload=4860]: Uploading piece... Jul 19 04:57:50.685559630 (6h6m57.058138985s) [noise=true , address=77.90.31.14:4661 , piece_id=X6UIHHUTOKC6R3BIJXZETTR4EBDJ4EHQYCYJVG3KLSAPVMZZMADA, seg_pos={0 0}, upload=4860]: Uploading piece... Jul 19 04:57:50.685741949 (6h6m57.058321304s) [noise=true , address=77.90.31.18:4806 , piece_id=MYZZEPG546S5I67HCIBSNMTK2UGT7XOBMTGYPTPZAIMNLA6TWAKA, seg_pos={0 0}, upload=4860]: Uploading piece... Jul 19 04:57:50.685953323 (6h6m57.058532678s) [noise=true , address=45.91.116.9:4261 , piece_id=6GLY2NCWLS5X6IHCVYB76Z4F2UJ47GXH5HJ6BUBUU7DVW5BT5I5Q, seg_pos={0 0}, upload=4860]: Uploading piece... Jul 19 04:57:52.382568857 (6h6m58.755148202s) [upload=4860]: Next returned nil segment Jul 19 04:57:52.382588915 (6h6m58.75516827s) [upload=4860]: Waiting for error group managing segments... Jul 19 04:57:52.712870771 (6h6m59.085450116s) [noise=true , address=45.91.116.5:4298 , piece_id=AUGUELVJFWXEPSOEWRCGZM6X4TI2JOUQNYP6WP4VL25ZC7FAPRTQ, seg_pos={0 0}, upload=4860]: Done uploading piece. err: <nil> Jul 19 04:57:52.713384304 (6h6m59.085963649s) [noise=true , address=45.91.116.9:4263 , piece_id=CPEMTRRYLTUFXTAHXB7P2THNI7T5LZCFO26XPIGATVNH2UWQH6PA, seg_pos={0 0}, upload=4860]: Done uploading piece. err: <nil> Jul 19 04:57:52.719689228 (6h6m59.092268583s) [noise=true , address=104.36.82.5:4839 , piece_id=W3C5L2U6NK2DYJSOIFNC2PF2XFXEOHAAWABP7UA3KM7LIXAWSPSQ, seg_pos={0 0}, upload=4860]: Done uploading piece. err: <nil> Jul 19 04:57:52.723321257 (6h6m59.095900612s) [noise=true , address=104.36.82.10:4077 , piece_id=ZCLW6MI23GWBBSZVV44LHDG4BBLKB656IJCNCVC2PGGVFJK4KV6Q, seg_pos={0 0}, upload=4860]: Done uploading piece. err: <nil> Jul 19 04:57:52.797624770 (6h6m59.170204115s) [noise=true , address=45.91.116.6:4974 , piece_id=2IEHPCFHSH7LGYE4M6UER42T22OPGFVWEGGQ6CIZ2FZ2HP3YAZWA, seg_pos={0 0}, upload=4860]: Done uploading piece. err: <nil> Jul 19 04:57:52.798066866 (6h6m59.170646221s) [noise=true , address=45.91.116.4:4938 , piece_id=REUW3UKOOZX56BI3V5ALKCID53EIRRDBMSA2BHCLMQSZGTYJGMDA, seg_pos={0 0}, upload=4860]: Done uploading piece. err: <nil> Jul 19 04:57:52.820454393 (6h6m59.193033738s) [noise=true , address=104.36.82.16:4294 , piece_id=4HRMC3IOCWF5I252RH6JTAANPYLX4QFM3UCKXOD67BTAUJ6EDOCQ, seg_pos={0 0}, upload=4860]: Done uploading piece. err: <nil> Jul 19 04:57:52.864818369 (6h6m59.237397724s) [noise=true , address=45.91.116.10:4081 , piece_id=IN6OJGRO2YDSGNIY3ZNTYTVYLEWKHFKOT24PADM7K7YQNXK5BTEA, seg_pos={0 0}, upload=4860]: Done uploading piece. err: <nil> Jul 19 04:57:52.873179036 (6h6m59.245758391s) [noise=true , address=45.91.116.9:4261 , piece_id=6GLY2NCWLS5X6IHCVYB76Z4F2UJ47GXH5HJ6BUBUU7DVW5BT5I5Q, seg_pos={0 0}, upload=4860]: Done uploading piece. err: <nil> Jul 19 04:57:52.881637702 (6h6m59.254217047s) [noise=true , address=45.91.116.5:4299 , piece_id=RPRYWZHCNVP72ZBR4BGMCOI245RG7ZEIWKVCNX3HRZVODGJIM26A, seg_pos={0 0}, upload=4860]: Done uploading piece. err: <nil> Jul 19 04:57:52.884454370 (6h6m59.257033725s) [noise=true , address=104.36.82.3:4767 , piece_id=MNQW4WEQRFJQXUKITTVSOQN4J3T7AEXPYN37Z6OMFLFOASRQKZYA, seg_pos={0 0}, upload=4860]: Done uploading piece. err: <nil> Jul 19 04:57:54.501916491 (6h7m0.874495846s) [noise=true , address=45.91.116.7:4011 , piece_id=LI35TO3RUCTEYICC7HUI6P34Z2KJH3S43L2GRAE646FAVXNHRXWA, seg_pos={0 0}, upload=4860]: Done uploading piece. err: <nil> Jul 19 04:58:19.391729500 (6h7m25.764308845s) [noise=true , address=77.90.31.11:4553 , piece_id=ECHSCGV2FSRCWW5M4QKGZMCQ4AHFCIUFBWPRURFCPFJR5YUOC42A, seg_pos={0 0}, upload=4860]: Done uploading piece. err: <nil> Jul 19 04:58:19.391756702 (6h7m25.764336047s) [seg_pos={0 0}, upload=4860]: Segment reached optimal threshold of 13 pieces. Jul 19 04:58:19.391845321 (6h7m25.764424676s) [noise=true , address=77.90.31.14:4661 , piece_id=X6UIHHUTOKC6R3BIJXZETTR4EBDJ4EHQYCYJVG3KLSAPVMZZMADA, seg_pos={0 0}, upload=4860]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Jul 19 04:58:19.392247070 (6h7m25.764826425s) [noise=true , address=77.90.31.18:4806 , piece_id=MYZZEPG546S5I67HCIBSNMTK2UGT7XOBMTGYPTPZAIMNLA6TWAKA, seg_pos={0 0}, upload=4860]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: write tcp 188.166.225.170:45822->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:45822->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45822->77.90.31.18:4806: use of closed network connection Jul 19 04:58:19.392314228 (6h7m25.764893583s) [noise=true , address=77.90.31.17:4771 , piece_id=GQQ5R52KDOOI5WOXKTJKZXTIECNU3QWDM4KLGOR7UMORDPMHJUJQ, seg_pos={0 0}, upload=4860]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2): protocol: write tcp 188.166.225.170:46240->77.90.31.17:4771: use of closed network connection; write tcp 188.166.225.170:46240->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46240->77.90.31.17:4771: use of closed network connection Jul 19 04:58:19.392340238 (6h7m25.764919593s) [seg_pos={0 0}, upload=4860]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 19 04:58:19.392427785 (6h7m25.76500714s) [upload=4860]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 19 04:58:19.392432124 (6h7m25.76501148s) [upload=4860]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 19 04:58:19.702762764 (6h7m26.075342109s) [upload=4860]: Done uploading segments... Jul 19 04:58:19.702781199 (6h7m26.075360554s) [upload=4860]: Upload finished. err: <nil> Duration: 9.541497009s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 7.561934261s FAIL (2025-07-19 05:01:02 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 4855s) [upload=4769]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 04:51:09.109709627 (6h0m15.482288982s) [upload=4769]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 04:51:09.556145753 (6h0m15.928725108s) [seg_pos={0 0}, upload=4769]: Begin upload segment... Jul 19 04:51:09.556194817 (6h0m15.928774172s) [seg_pos={0 0}, upload=4769]: Done begin upload segment. Jul 19 04:51:09.556223893 (6h0m15.928803248s) [noise=true , address=79.127.219.38:4023 , piece_id=PTPOZGLYV4MZTOFZNN3NCGYL7DFAST4PXKEUIVTMFL7HV4GZTCLQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556252197 (6h0m15.928831552s) [noise=true , address=45.91.95.4:4011 , piece_id=ZMBAS6KKHYVIRE6OBL2NFVCWMM4TKQ7J64BD3SYXIK54XCEDI2XA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556579383 (6h0m15.929158738s) [noise=true , address=45.91.116.6:4007 , piece_id=A2AFVTZBXSGBOE7XIM3B5HRN4STITFBXO6MZRJ365URNAAOZ3PSQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556854469 (6h0m15.929433824s) [noise=true , address=104.36.82.19:4423 , piece_id=PIJUHDUND36HQL67I33D7A3EULMJHS7N55MN4OFQJ4OWNQ42O47A, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557094107 (6h0m15.929673462s) [noise=true , address=45.91.116.9:4264 , piece_id=GHM5BKZKMT2QOHHBOS4FSPOWEWSFQRGRVVR4GSNDPHVIFRZHQL2A, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557336381 (6h0m15.929915736s) [noise=true , address=79.127.205.235:4660 , piece_id=UOUOGE2VP7PX7EMRZLPMCWFCYEZERM3DQMCQ6NG4J2K2DEIZR63Q, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557557885 (6h0m15.93013724s) [noise=true , address=79.127.205.246:4280 , piece_id=FLBOZI3UGTQXAQBLZ2F45Z3OEYL65F5PQBXRF2DQMZ7NGF7VDRPA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557762296 (6h0m15.930341651s) [noise=true , address=79.127.205.233:4622 , piece_id=6WWNYQTH5MGAJOEB7BXKZDJOMCAZXDW66J3OEANGDNJDNUMLZM7Q, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558014689 (6h0m15.930594044s) [noise=true , address=64.49.13.138:4482 , piece_id=MAW2M4US2OQNOS3WI67AA7NIBTJDE5T2JQTXKFPBLGVGXKVCFGTQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558215974 (6h0m15.930795319s) [noise=true , address=77.90.31.10:4546 , piece_id=R6LCA5WZOEBDURL5S3MEK22YPVYXX6EXSS4J4W57543HELYXOKZA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558409364 (6h0m15.930988719s) [noise=true , address=45.91.95.5:4019 , piece_id=YTSBCN2LPHEMRRAZIHJGVB7NBOTRXVEHD2TWUCTTLBNFFYEWWDKA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558623504 (6h0m15.931202859s) [noise=true , address=79.127.201.215:4357 , piece_id=75XWAG2VQWEYCWN2Q3WRIZSBI7LDOISZJPXQLQ2GIPNEN34VLSUQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558805162 (6h0m15.931384517s) [noise=true , address=45.91.116.14:4219 , piece_id=PDW6JPRHCVMHHOWIHT6JKV4CTSC3JY6NDKDVFZSO47OYNPZ7PGGA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558999373 (6h0m15.931578728s) [noise=true , address=77.90.31.11:4564 , piece_id=L2U7JFJRDQOFA2AYVAGTI4ZW4VABCXKMXWBIUWQO2QNNUUBJ74BA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559184768 (6h0m15.931764123s) [noise=true , address=77.90.31.16:4768 , piece_id=YNLSONFM6COZEVF6ZNQR2EA6V4LSQFZJONIXR4AFTNNMFGPV57JA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559368719 (6h0m15.931948074s) [noise=true , address=79.127.220.98:4432 , piece_id=2SUTFQ7FVDPAM2H4W3OLW6AVH3VJ33IJFE4L52SA3ET5LGBMOMPQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559549706 (6h0m15.932129061s) [noise=true , address=77.90.31.18:4813 , piece_id=VDOT7BB4O7XYJHW6BFTAYQZZYLZU5PAUXVJGSXJ7FDI4TH425COA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559727836 (6h0m15.932307191s) [noise=true , address=79.127.219.51:4211 , piece_id=HO5PLWBVU5YAYNYMN352NA73SM4WBJV4WZ7XLSQTCQ76UAL2TOJA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:10.663359087 (6h0m17.035938442s) [upload=4769]: Next returned nil segment Jul 19 04:51:10.663378664 (6h0m17.035958019s) [upload=4769]: Waiting for error group managing segments... Jul 19 04:51:10.962945479 (6h0m17.335524834s) [noise=true , address=79.127.205.233:4622 , piece_id=6WWNYQTH5MGAJOEB7BXKZDJOMCAZXDW66J3OEANGDNJDNUMLZM7Q, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:10.967689807 (6h0m17.340269152s) [noise=true , address=79.127.205.235:4660 , piece_id=UOUOGE2VP7PX7EMRZLPMCWFCYEZERM3DQMCQ6NG4J2K2DEIZR63Q, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:10.976968631 (6h0m17.349547986s) [noise=true , address=64.49.13.138:4482 , piece_id=MAW2M4US2OQNOS3WI67AA7NIBTJDE5T2JQTXKFPBLGVGXKVCFGTQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.048750700 (6h0m17.421330055s) [noise=true , address=79.127.205.246:4280 , piece_id=FLBOZI3UGTQXAQBLZ2F45Z3OEYL65F5PQBXRF2DQMZ7NGF7VDRPA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.221642403 (6h0m17.594221758s) [noise=true , address=79.127.220.98:4432 , piece_id=2SUTFQ7FVDPAM2H4W3OLW6AVH3VJ33IJFE4L52SA3ET5LGBMOMPQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.317070435 (6h0m17.68964979s) [noise=true , address=45.91.95.4:4011 , piece_id=ZMBAS6KKHYVIRE6OBL2NFVCWMM4TKQ7J64BD3SYXIK54XCEDI2XA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.324002660 (6h0m17.696582015s) [noise=true , address=45.91.95.5:4019 , piece_id=YTSBCN2LPHEMRRAZIHJGVB7NBOTRXVEHD2TWUCTTLBNFFYEWWDKA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.460976317 (6h0m17.833555672s) [noise=true , address=79.127.201.215:4357 , piece_id=75XWAG2VQWEYCWN2Q3WRIZSBI7LDOISZJPXQLQ2GIPNEN34VLSUQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.462584603 (6h0m17.835163958s) [noise=true , address=45.91.116.6:4007 , piece_id=A2AFVTZBXSGBOE7XIM3B5HRN4STITFBXO6MZRJ365URNAAOZ3PSQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.490392973 (6h0m17.862972328s) [noise=true , address=45.91.116.9:4264 , piece_id=GHM5BKZKMT2QOHHBOS4FSPOWEWSFQRGRVVR4GSNDPHVIFRZHQL2A, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.502040780 (6h0m17.874620135s) [noise=true , address=45.91.116.14:4219 , piece_id=PDW6JPRHCVMHHOWIHT6JKV4CTSC3JY6NDKDVFZSO47OYNPZ7PGGA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.600434096 (6h0m17.973013451s) [noise=true , address=79.127.219.51:4211 , piece_id=HO5PLWBVU5YAYNYMN352NA73SM4WBJV4WZ7XLSQTCQ76UAL2TOJA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.609008984 (6h0m17.981588339s) [noise=true , address=104.36.82.19:4423 , piece_id=PIJUHDUND36HQL67I33D7A3EULMJHS7N55MN4OFQJ4OWNQ42O47A, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.620825273 (6h0m17.993404628s) [noise=true , address=79.127.219.38:4023 , piece_id=PTPOZGLYV4MZTOFZNN3NCGYL7DFAST4PXKEUIVTMFL7HV4GZTCLQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.620838659 (6h0m17.993418014s) [seg_pos={0 0}, upload=4769]: Segment reached optimal threshold of 14 pieces. Jul 19 04:51:11.621067807 (6h0m17.993647163s) [noise=true , address=77.90.31.11:4564 , piece_id=L2U7JFJRDQOFA2AYVAGTI4ZW4VABCXKMXWBIUWQO2QNNUUBJ74BA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection; write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection Jul 19 04:51:11.621134234 (6h0m17.993713589s) [noise=true , address=77.90.31.10:4546 , piece_id=R6LCA5WZOEBDURL5S3MEK22YPVYXX6EXSS4J4W57543HELYXOKZA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK): protocol: read tcp 188.166.225.170:58476->77.90.31.10:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621174191 (6h0m17.993753546s) [noise=true , address=77.90.31.16:4768 , piece_id=YNLSONFM6COZEVF6ZNQR2EA6V4LSQFZJONIXR4AFTNNMFGPV57JA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UchewmtxBrtSZSpVAJFdZmcTsQVQxkjK9re6cgUGqDhyonQeF): protocol: read tcp 188.166.225.170:56326->77.90.31.16:4768: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621216842 (6h0m17.993796197s) [noise=true , address=77.90.31.18:4813 , piece_id=VDOT7BB4O7XYJHW6BFTAYQZZYLZU5PAUXVJGSXJ7FDI4TH425COA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128WthA1x9wp1dV1vkpGnoveZQpU9CSgDCZfw5GMQuVpCNSVAM5): protocol: read tcp 188.166.225.170:49704->77.90.31.18:4813: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621234917 (6h0m17.993814272s) [seg_pos={0 0}, upload=4769]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 19 04:51:11.621301444 (6h0m17.993880799s) [upload=4769]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 19 04:51:11.621305402 (6h0m17.993884758s) [upload=4769]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 19 04:51:11.923136970 (6h0m18.295716325s) [upload=4769]: Done uploading segments... Jul 19 04:51:11.923156627 (6h0m18.295735972s) [upload=4769]: Upload finished. err: <nil> Duration: 7.561964378s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 6.050509202s FAIL (2025-07-19 05:00:35 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 4855s) [upload=4769]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 04:51:09.109709627 (6h0m15.482288982s) [upload=4769]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 04:51:09.556145753 (6h0m15.928725108s) [seg_pos={0 0}, upload=4769]: Begin upload segment... Jul 19 04:51:09.556194817 (6h0m15.928774172s) [seg_pos={0 0}, upload=4769]: Done begin upload segment. Jul 19 04:51:09.556223893 (6h0m15.928803248s) [noise=true , address=79.127.219.38:4023 , piece_id=PTPOZGLYV4MZTOFZNN3NCGYL7DFAST4PXKEUIVTMFL7HV4GZTCLQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556252197 (6h0m15.928831552s) [noise=true , address=45.91.95.4:4011 , piece_id=ZMBAS6KKHYVIRE6OBL2NFVCWMM4TKQ7J64BD3SYXIK54XCEDI2XA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556579383 (6h0m15.929158738s) [noise=true , address=45.91.116.6:4007 , piece_id=A2AFVTZBXSGBOE7XIM3B5HRN4STITFBXO6MZRJ365URNAAOZ3PSQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556854469 (6h0m15.929433824s) [noise=true , address=104.36.82.19:4423 , piece_id=PIJUHDUND36HQL67I33D7A3EULMJHS7N55MN4OFQJ4OWNQ42O47A, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557094107 (6h0m15.929673462s) [noise=true , address=45.91.116.9:4264 , piece_id=GHM5BKZKMT2QOHHBOS4FSPOWEWSFQRGRVVR4GSNDPHVIFRZHQL2A, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557336381 (6h0m15.929915736s) [noise=true , address=79.127.205.235:4660 , piece_id=UOUOGE2VP7PX7EMRZLPMCWFCYEZERM3DQMCQ6NG4J2K2DEIZR63Q, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557557885 (6h0m15.93013724s) [noise=true , address=79.127.205.246:4280 , piece_id=FLBOZI3UGTQXAQBLZ2F45Z3OEYL65F5PQBXRF2DQMZ7NGF7VDRPA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557762296 (6h0m15.930341651s) [noise=true , address=79.127.205.233:4622 , piece_id=6WWNYQTH5MGAJOEB7BXKZDJOMCAZXDW66J3OEANGDNJDNUMLZM7Q, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558014689 (6h0m15.930594044s) [noise=true , address=64.49.13.138:4482 , piece_id=MAW2M4US2OQNOS3WI67AA7NIBTJDE5T2JQTXKFPBLGVGXKVCFGTQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558215974 (6h0m15.930795319s) [noise=true , address=77.90.31.10:4546 , piece_id=R6LCA5WZOEBDURL5S3MEK22YPVYXX6EXSS4J4W57543HELYXOKZA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558409364 (6h0m15.930988719s) [noise=true , address=45.91.95.5:4019 , piece_id=YTSBCN2LPHEMRRAZIHJGVB7NBOTRXVEHD2TWUCTTLBNFFYEWWDKA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558623504 (6h0m15.931202859s) [noise=true , address=79.127.201.215:4357 , piece_id=75XWAG2VQWEYCWN2Q3WRIZSBI7LDOISZJPXQLQ2GIPNEN34VLSUQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558805162 (6h0m15.931384517s) [noise=true , address=45.91.116.14:4219 , piece_id=PDW6JPRHCVMHHOWIHT6JKV4CTSC3JY6NDKDVFZSO47OYNPZ7PGGA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558999373 (6h0m15.931578728s) [noise=true , address=77.90.31.11:4564 , piece_id=L2U7JFJRDQOFA2AYVAGTI4ZW4VABCXKMXWBIUWQO2QNNUUBJ74BA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559184768 (6h0m15.931764123s) [noise=true , address=77.90.31.16:4768 , piece_id=YNLSONFM6COZEVF6ZNQR2EA6V4LSQFZJONIXR4AFTNNMFGPV57JA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559368719 (6h0m15.931948074s) [noise=true , address=79.127.220.98:4432 , piece_id=2SUTFQ7FVDPAM2H4W3OLW6AVH3VJ33IJFE4L52SA3ET5LGBMOMPQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559549706 (6h0m15.932129061s) [noise=true , address=77.90.31.18:4813 , piece_id=VDOT7BB4O7XYJHW6BFTAYQZZYLZU5PAUXVJGSXJ7FDI4TH425COA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559727836 (6h0m15.932307191s) [noise=true , address=79.127.219.51:4211 , piece_id=HO5PLWBVU5YAYNYMN352NA73SM4WBJV4WZ7XLSQTCQ76UAL2TOJA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:10.663359087 (6h0m17.035938442s) [upload=4769]: Next returned nil segment Jul 19 04:51:10.663378664 (6h0m17.035958019s) [upload=4769]: Waiting for error group managing segments... Jul 19 04:51:10.962945479 (6h0m17.335524834s) [noise=true , address=79.127.205.233:4622 , piece_id=6WWNYQTH5MGAJOEB7BXKZDJOMCAZXDW66J3OEANGDNJDNUMLZM7Q, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:10.967689807 (6h0m17.340269152s) [noise=true , address=79.127.205.235:4660 , piece_id=UOUOGE2VP7PX7EMRZLPMCWFCYEZERM3DQMCQ6NG4J2K2DEIZR63Q, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:10.976968631 (6h0m17.349547986s) [noise=true , address=64.49.13.138:4482 , piece_id=MAW2M4US2OQNOS3WI67AA7NIBTJDE5T2JQTXKFPBLGVGXKVCFGTQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.048750700 (6h0m17.421330055s) [noise=true , address=79.127.205.246:4280 , piece_id=FLBOZI3UGTQXAQBLZ2F45Z3OEYL65F5PQBXRF2DQMZ7NGF7VDRPA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.221642403 (6h0m17.594221758s) [noise=true , address=79.127.220.98:4432 , piece_id=2SUTFQ7FVDPAM2H4W3OLW6AVH3VJ33IJFE4L52SA3ET5LGBMOMPQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.317070435 (6h0m17.68964979s) [noise=true , address=45.91.95.4:4011 , piece_id=ZMBAS6KKHYVIRE6OBL2NFVCWMM4TKQ7J64BD3SYXIK54XCEDI2XA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.324002660 (6h0m17.696582015s) [noise=true , address=45.91.95.5:4019 , piece_id=YTSBCN2LPHEMRRAZIHJGVB7NBOTRXVEHD2TWUCTTLBNFFYEWWDKA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.460976317 (6h0m17.833555672s) [noise=true , address=79.127.201.215:4357 , piece_id=75XWAG2VQWEYCWN2Q3WRIZSBI7LDOISZJPXQLQ2GIPNEN34VLSUQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.462584603 (6h0m17.835163958s) [noise=true , address=45.91.116.6:4007 , piece_id=A2AFVTZBXSGBOE7XIM3B5HRN4STITFBXO6MZRJ365URNAAOZ3PSQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.490392973 (6h0m17.862972328s) [noise=true , address=45.91.116.9:4264 , piece_id=GHM5BKZKMT2QOHHBOS4FSPOWEWSFQRGRVVR4GSNDPHVIFRZHQL2A, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.502040780 (6h0m17.874620135s) [noise=true , address=45.91.116.14:4219 , piece_id=PDW6JPRHCVMHHOWIHT6JKV4CTSC3JY6NDKDVFZSO47OYNPZ7PGGA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.600434096 (6h0m17.973013451s) [noise=true , address=79.127.219.51:4211 , piece_id=HO5PLWBVU5YAYNYMN352NA73SM4WBJV4WZ7XLSQTCQ76UAL2TOJA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.609008984 (6h0m17.981588339s) [noise=true , address=104.36.82.19:4423 , piece_id=PIJUHDUND36HQL67I33D7A3EULMJHS7N55MN4OFQJ4OWNQ42O47A, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.620825273 (6h0m17.993404628s) [noise=true , address=79.127.219.38:4023 , piece_id=PTPOZGLYV4MZTOFZNN3NCGYL7DFAST4PXKEUIVTMFL7HV4GZTCLQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.620838659 (6h0m17.993418014s) [seg_pos={0 0}, upload=4769]: Segment reached optimal threshold of 14 pieces. Jul 19 04:51:11.621067807 (6h0m17.993647163s) [noise=true , address=77.90.31.11:4564 , piece_id=L2U7JFJRDQOFA2AYVAGTI4ZW4VABCXKMXWBIUWQO2QNNUUBJ74BA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection; write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection Jul 19 04:51:11.621134234 (6h0m17.993713589s) [noise=true , address=77.90.31.10:4546 , piece_id=R6LCA5WZOEBDURL5S3MEK22YPVYXX6EXSS4J4W57543HELYXOKZA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK): protocol: read tcp 188.166.225.170:58476->77.90.31.10:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621174191 (6h0m17.993753546s) [noise=true , address=77.90.31.16:4768 , piece_id=YNLSONFM6COZEVF6ZNQR2EA6V4LSQFZJONIXR4AFTNNMFGPV57JA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UchewmtxBrtSZSpVAJFdZmcTsQVQxkjK9re6cgUGqDhyonQeF): protocol: read tcp 188.166.225.170:56326->77.90.31.16:4768: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621216842 (6h0m17.993796197s) [noise=true , address=77.90.31.18:4813 , piece_id=VDOT7BB4O7XYJHW6BFTAYQZZYLZU5PAUXVJGSXJ7FDI4TH425COA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128WthA1x9wp1dV1vkpGnoveZQpU9CSgDCZfw5GMQuVpCNSVAM5): protocol: read tcp 188.166.225.170:49704->77.90.31.18:4813: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621234917 (6h0m17.993814272s) [seg_pos={0 0}, upload=4769]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 19 04:51:11.621301444 (6h0m17.993880799s) [upload=4769]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 19 04:51:11.621305402 (6h0m17.993884758s) [upload=4769]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 19 04:51:11.923136970 (6h0m18.295716325s) [upload=4769]: Done uploading segments... Jul 19 04:51:11.923156627 (6h0m18.295735972s) [upload=4769]: Upload finished. err: <nil> Duration: 6.050537456s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [........................................XX...X...X....X..X..] duration: 5.948090593s FAIL (2025-07-19 04:59:50 UTC) orbiter warning: ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== :1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: read tcp 188.166.225.170:50230->77.90.31.19:4841: use of closed network connection; context canceled Jul 19 04:56:54.832707046 (6h6m1.205286401s) [seg_pos={0 0}, upload=4846]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 19 04:56:54.832779073 (6h6m1.205358428s) [upload=4846]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 19 04:56:54.832783182 (6h6m1.205362527s) [upload=4846]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 19 04:56:55.133341022 (6h6m1.505920377s) [upload=4846]: Done uploading segments... Jul 19 04:56:55.133358805 (6h6m1.50593816s) [upload=4846]: Upload finished. err: <nil> Jul 19 04:57:50.449535830 (6h6m56.822115185s) [upload=4860]: Starting upload Jul 19 04:57:50.449598590 (6h6m56.822177935s) [upload=4860]: Done starting upload Jul 19 04:57:50.449668684 (6h6m56.822248039s) [upload=4860]: Uploading segments... Jul 19 04:57:50.449781009 (6h6m56.822360364s) [upload=4860]: Got next segment. Inline: false Jul 19 04:57:50.449828921 (6h6m56.822408276s) [upload=4860]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 04:57:50.449833700 (6h6m56.822413055s) [upload=4860]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 04:57:50.682857520 (6h6m57.055436875s) [seg_pos={0 0}, upload=4860]: Begin upload segment... Jul 19 04:57:50.682898699 (6h6m57.055478054s) [seg_pos={0 0}, upload=4860]: Done begin upload segment. Jul 19 04:57:50.682913377 (6h6m57.055492732s) [noise=true , address=45.91.116.6:4974 , piece_id=2IEHPCFHSH7LGYE4M6UER42T22OPGFVWEGGQ6CIZ2FZ2HP3YAZWA, seg_pos={0 0}, upload=4860]: Uploading piece... Jul 19 04:57:50.682920360 (6h6m57.055499705s) [noise=true , address=77.90.31.11:4553 , piece_id=ECHSCGV2FSRCWW5M4QKGZMCQ4AHFCIUFBWPRURFCPFJR5YUOC42A, seg_pos={0 0}, upload=4860]: Uploading piece... Jul 19 04:57:50.682990905 (6h6m57.05557026s) [noise=true , address=104.36.82.16:4294 , piece_id=4HRMC3IOCWF5I252RH6JTAANPYLX4QFM3UCKXOD67BTAUJ6EDOCQ, seg_pos={0 0}, upload=4860]: Uploading piece... Jul 19 04:57:50.683253236 (6h6m57.055832591s) [noise=true , address=104.36.82.5:4839 , piece_id=W3C5L2U6NK2DYJSOIFNC2PF2XFXEOHAAWABP7UA3KM7LIXAWSPSQ, seg_pos={0 0}, upload=4860]: Uploading piece... Jul 19 04:57:50.683500520 (6h6m57.056079875s) [noise=true , address=77.90.31.17:4771 , piece_id=GQQ5R52KDOOI5WOXKTJKZXTIECNU3QWDM4KLGOR7UMORDPMHJUJQ, seg_pos={0 0}, upload=4860]: Uploading piece... Jul 19 04:57:50.683736031 (6h6m57.056315376s) [noise=true , address=45.91.116.4:4938 , piece_id=REUW3UKOOZX56BI3V5ALKCID53EIRRDBMSA2BHCLMQSZGTYJGMDA, seg_pos={0 0}, upload=4860]: Uploading piece... Jul 19 04:57:50.684122380 (6h6m57.056701725s) [noise=true , address=45.91.116.7:4011 , piece_id=LI35TO3RUCTEYICC7HUI6P34Z2KJH3S43L2GRAE646FAVXNHRXWA, seg_pos={0 0}, upload=4860]: Uploading piece... Jul 19 04:57:50.684336981 (6h6m57.056916336s) [noise=true , address=104.36.82.10:4077 , piece_id=ZCLW6MI23GWBBSZVV44LHDG4BBLKB656IJCNCVC2PGGVFJK4KV6Q, seg_pos={0 0}, upload=4860]: Uploading piece... Jul 19 04:57:50.684555689 (6h6m57.057135044s) [noise=true , address=104.36.82.3:4767 , piece_id=MNQW4WEQRFJQXUKITTVSOQN4J3T7AEXPYN37Z6OMFLFOASRQKZYA, seg_pos={0 0}, upload=4860]: Uploading piece... Jul 19 04:57:50.684820506 (6h6m57.057399861s) [noise=true , address=45.91.116.5:4298 , piece_id=AUGUELVJFWXEPSOEWRCGZM6X4TI2JOUQNYP6WP4VL25ZC7FAPRTQ, seg_pos={0 0}, upload=4860]: Uploading piece... Jul 19 04:57:50.685013646 (6h6m57.057593001s) [noise=true , address=45.91.116.5:4299 , piece_id=RPRYWZHCNVP72ZBR4BGMCOI245RG7ZEIWKVCNX3HRZVODGJIM26A, seg_pos={0 0}, upload=4860]: Uploading piece... Jul 19 04:57:50.685194391 (6h6m57.057773746s) [noise=true , address=45.91.116.9:4263 , piece_id=CPEMTRRYLTUFXTAHXB7P2THNI7T5LZCFO26XPIGATVNH2UWQH6PA, seg_pos={0 0}, upload=4860]: Uploading piece... Jul 19 04:57:50.685376059 (6h6m57.057955414s) [noise=true , address=45.91.116.10:4081 , piece_id=IN6OJGRO2YDSGNIY3ZNTYTVYLEWKHFKOT24PADM7K7YQNXK5BTEA, seg_pos={0 0}, upload=4860]: Uploading piece... Jul 19 04:57:50.685559630 (6h6m57.058138985s) [noise=true , address=77.90.31.14:4661 , piece_id=X6UIHHUTOKC6R3BIJXZETTR4EBDJ4EHQYCYJVG3KLSAPVMZZMADA, seg_pos={0 0}, upload=4860]: Uploading piece... Jul 19 04:57:50.685741949 (6h6m57.058321304s) [noise=true , address=77.90.31.18:4806 , piece_id=MYZZEPG546S5I67HCIBSNMTK2UGT7XOBMTGYPTPZAIMNLA6TWAKA, seg_pos={0 0}, upload=4860]: Uploading piece... Jul 19 04:57:50.685953323 (6h6m57.058532678s) [noise=true , address=45.91.116.9:4261 , piece_id=6GLY2NCWLS5X6IHCVYB76Z4F2UJ47GXH5HJ6BUBUU7DVW5BT5I5Q, seg_pos={0 0}, upload=4860]: Uploading piece... Jul 19 04:57:52.382568857 (6h6m58.755148202s) [upload=4860]: Next returned nil segment Jul 19 04:57:52.382588915 (6h6m58.75516827s) [upload=4860]: Waiting for error group managing segments... Jul 19 04:57:52.712870771 (6h6m59.085450116s) [noise=true , address=45.91.116.5:4298 , piece_id=AUGUELVJFWXEPSOEWRCGZM6X4TI2JOUQNYP6WP4VL25ZC7FAPRTQ, seg_pos={0 0}, upload=4860]: Done uploading piece. err: <nil> Jul 19 04:57:52.713384304 (6h6m59.085963649s) [noise=true , address=45.91.116.9:4263 , piece_id=CPEMTRRYLTUFXTAHXB7P2THNI7T5LZCFO26XPIGATVNH2UWQH6PA, seg_pos={0 0}, upload=4860]: Done uploading piece. err: <nil> Jul 19 04:57:52.719689228 (6h6m59.092268583s) [noise=true , address=104.36.82.5:4839 , piece_id=W3C5L2U6NK2DYJSOIFNC2PF2XFXEOHAAWABP7UA3KM7LIXAWSPSQ, seg_pos={0 0}, upload=4860]: Done uploading piece. err: <nil> Jul 19 04:57:52.723321257 (6h6m59.095900612s) [noise=true , address=104.36.82.10:4077 , piece_id=ZCLW6MI23GWBBSZVV44LHDG4BBLKB656IJCNCVC2PGGVFJK4KV6Q, seg_pos={0 0}, upload=4860]: Done uploading piece. err: <nil> Jul 19 04:57:52.797624770 (6h6m59.170204115s) [noise=true , address=45.91.116.6:4974 , piece_id=2IEHPCFHSH7LGYE4M6UER42T22OPGFVWEGGQ6CIZ2FZ2HP3YAZWA, seg_pos={0 0}, upload=4860]: Done uploading piece. err: <nil> Jul 19 04:57:52.798066866 (6h6m59.170646221s) [noise=true , address=45.91.116.4:4938 , piece_id=REUW3UKOOZX56BI3V5ALKCID53EIRRDBMSA2BHCLMQSZGTYJGMDA, seg_pos={0 0}, upload=4860]: Done uploading piece. err: <nil> Jul 19 04:57:52.820454393 (6h6m59.193033738s) [noise=true , address=104.36.82.16:4294 , piece_id=4HRMC3IOCWF5I252RH6JTAANPYLX4QFM3UCKXOD67BTAUJ6EDOCQ, seg_pos={0 0}, upload=4860]: Done uploading piece. err: <nil> Jul 19 04:57:52.864818369 (6h6m59.237397724s) [noise=true , address=45.91.116.10:4081 , piece_id=IN6OJGRO2YDSGNIY3ZNTYTVYLEWKHFKOT24PADM7K7YQNXK5BTEA, seg_pos={0 0}, upload=4860]: Done uploading piece. err: <nil> Jul 19 04:57:52.873179036 (6h6m59.245758391s) [noise=true , address=45.91.116.9:4261 , piece_id=6GLY2NCWLS5X6IHCVYB76Z4F2UJ47GXH5HJ6BUBUU7DVW5BT5I5Q, seg_pos={0 0}, upload=4860]: Done uploading piece. err: <nil> Jul 19 04:57:52.881637702 (6h6m59.254217047s) [noise=true , address=45.91.116.5:4299 , piece_id=RPRYWZHCNVP72ZBR4BGMCOI245RG7ZEIWKVCNX3HRZVODGJIM26A, seg_pos={0 0}, upload=4860]: Done uploading piece. err: <nil> Jul 19 04:57:52.884454370 (6h6m59.257033725s) [noise=true , address=104.36.82.3:4767 , piece_id=MNQW4WEQRFJQXUKITTVSOQN4J3T7AEXPYN37Z6OMFLFOASRQKZYA, seg_pos={0 0}, upload=4860]: Done uploading piece. err: <nil> Jul 19 04:57:54.501916491 (6h7m0.874495846s) [noise=true , address=45.91.116.7:4011 , piece_id=LI35TO3RUCTEYICC7HUI6P34Z2KJH3S43L2GRAE646FAVXNHRXWA, seg_pos={0 0}, upload=4860]: Done uploading piece. err: <nil> Jul 19 04:58:19.391729500 (6h7m25.764308845s) [noise=true , address=77.90.31.11:4553 , piece_id=ECHSCGV2FSRCWW5M4QKGZMCQ4AHFCIUFBWPRURFCPFJR5YUOC42A, seg_pos={0 0}, upload=4860]: Done uploading piece. err: <nil> Jul 19 04:58:19.391756702 (6h7m25.764336047s) [seg_pos={0 0}, upload=4860]: Segment reached optimal threshold of 13 pieces. Jul 19 04:58:19.391845321 (6h7m25.764424676s) [noise=true , address=77.90.31.14:4661 , piece_id=X6UIHHUTOKC6R3BIJXZETTR4EBDJ4EHQYCYJVG3KLSAPVMZZMADA, seg_pos={0 0}, upload=4860]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Jul 19 04:58:19.392247070 (6h7m25.764826425s) [noise=true , address=77.90.31.18:4806 , piece_id=MYZZEPG546S5I67HCIBSNMTK2UGT7XOBMTGYPTPZAIMNLA6TWAKA, seg_pos={0 0}, upload=4860]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: write tcp 188.166.225.170:45822->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:45822->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45822->77.90.31.18:4806: use of closed network connection Jul 19 04:58:19.392314228 (6h7m25.764893583s) [noise=true , address=77.90.31.17:4771 , piece_id=GQQ5R52KDOOI5WOXKTJKZXTIECNU3QWDM4KLGOR7UMORDPMHJUJQ, seg_pos={0 0}, upload=4860]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2): protocol: write tcp 188.166.225.170:46240->77.90.31.17:4771: use of closed network connection; write tcp 188.166.225.170:46240->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46240->77.90.31.17:4771: use of closed network connection Jul 19 04:58:19.392340238 (6h7m25.764919593s) [seg_pos={0 0}, upload=4860]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 19 04:58:19.392427785 (6h7m25.76500714s) [upload=4860]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 19 04:58:19.392432124 (6h7m25.76501148s) [upload=4860]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 19 04:58:19.702762764 (6h7m26.075342109s) [upload=4860]: Done uploading segments... Jul 19 04:58:19.702781199 (6h7m26.075360554s) [upload=4860]: Upload finished. err: <nil> Duration: 5.948114849s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 5.535063664s FAIL (2025-07-19 04:59:33 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 4855s) [upload=4769]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 04:51:09.109709627 (6h0m15.482288982s) [upload=4769]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 04:51:09.556145753 (6h0m15.928725108s) [seg_pos={0 0}, upload=4769]: Begin upload segment... Jul 19 04:51:09.556194817 (6h0m15.928774172s) [seg_pos={0 0}, upload=4769]: Done begin upload segment. Jul 19 04:51:09.556223893 (6h0m15.928803248s) [noise=true , address=79.127.219.38:4023 , piece_id=PTPOZGLYV4MZTOFZNN3NCGYL7DFAST4PXKEUIVTMFL7HV4GZTCLQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556252197 (6h0m15.928831552s) [noise=true , address=45.91.95.4:4011 , piece_id=ZMBAS6KKHYVIRE6OBL2NFVCWMM4TKQ7J64BD3SYXIK54XCEDI2XA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556579383 (6h0m15.929158738s) [noise=true , address=45.91.116.6:4007 , piece_id=A2AFVTZBXSGBOE7XIM3B5HRN4STITFBXO6MZRJ365URNAAOZ3PSQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556854469 (6h0m15.929433824s) [noise=true , address=104.36.82.19:4423 , piece_id=PIJUHDUND36HQL67I33D7A3EULMJHS7N55MN4OFQJ4OWNQ42O47A, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557094107 (6h0m15.929673462s) [noise=true , address=45.91.116.9:4264 , piece_id=GHM5BKZKMT2QOHHBOS4FSPOWEWSFQRGRVVR4GSNDPHVIFRZHQL2A, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557336381 (6h0m15.929915736s) [noise=true , address=79.127.205.235:4660 , piece_id=UOUOGE2VP7PX7EMRZLPMCWFCYEZERM3DQMCQ6NG4J2K2DEIZR63Q, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557557885 (6h0m15.93013724s) [noise=true , address=79.127.205.246:4280 , piece_id=FLBOZI3UGTQXAQBLZ2F45Z3OEYL65F5PQBXRF2DQMZ7NGF7VDRPA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557762296 (6h0m15.930341651s) [noise=true , address=79.127.205.233:4622 , piece_id=6WWNYQTH5MGAJOEB7BXKZDJOMCAZXDW66J3OEANGDNJDNUMLZM7Q, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558014689 (6h0m15.930594044s) [noise=true , address=64.49.13.138:4482 , piece_id=MAW2M4US2OQNOS3WI67AA7NIBTJDE5T2JQTXKFPBLGVGXKVCFGTQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558215974 (6h0m15.930795319s) [noise=true , address=77.90.31.10:4546 , piece_id=R6LCA5WZOEBDURL5S3MEK22YPVYXX6EXSS4J4W57543HELYXOKZA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558409364 (6h0m15.930988719s) [noise=true , address=45.91.95.5:4019 , piece_id=YTSBCN2LPHEMRRAZIHJGVB7NBOTRXVEHD2TWUCTTLBNFFYEWWDKA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558623504 (6h0m15.931202859s) [noise=true , address=79.127.201.215:4357 , piece_id=75XWAG2VQWEYCWN2Q3WRIZSBI7LDOISZJPXQLQ2GIPNEN34VLSUQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558805162 (6h0m15.931384517s) [noise=true , address=45.91.116.14:4219 , piece_id=PDW6JPRHCVMHHOWIHT6JKV4CTSC3JY6NDKDVFZSO47OYNPZ7PGGA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558999373 (6h0m15.931578728s) [noise=true , address=77.90.31.11:4564 , piece_id=L2U7JFJRDQOFA2AYVAGTI4ZW4VABCXKMXWBIUWQO2QNNUUBJ74BA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559184768 (6h0m15.931764123s) [noise=true , address=77.90.31.16:4768 , piece_id=YNLSONFM6COZEVF6ZNQR2EA6V4LSQFZJONIXR4AFTNNMFGPV57JA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559368719 (6h0m15.931948074s) [noise=true , address=79.127.220.98:4432 , piece_id=2SUTFQ7FVDPAM2H4W3OLW6AVH3VJ33IJFE4L52SA3ET5LGBMOMPQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559549706 (6h0m15.932129061s) [noise=true , address=77.90.31.18:4813 , piece_id=VDOT7BB4O7XYJHW6BFTAYQZZYLZU5PAUXVJGSXJ7FDI4TH425COA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559727836 (6h0m15.932307191s) [noise=true , address=79.127.219.51:4211 , piece_id=HO5PLWBVU5YAYNYMN352NA73SM4WBJV4WZ7XLSQTCQ76UAL2TOJA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:10.663359087 (6h0m17.035938442s) [upload=4769]: Next returned nil segment Jul 19 04:51:10.663378664 (6h0m17.035958019s) [upload=4769]: Waiting for error group managing segments... Jul 19 04:51:10.962945479 (6h0m17.335524834s) [noise=true , address=79.127.205.233:4622 , piece_id=6WWNYQTH5MGAJOEB7BXKZDJOMCAZXDW66J3OEANGDNJDNUMLZM7Q, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:10.967689807 (6h0m17.340269152s) [noise=true , address=79.127.205.235:4660 , piece_id=UOUOGE2VP7PX7EMRZLPMCWFCYEZERM3DQMCQ6NG4J2K2DEIZR63Q, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:10.976968631 (6h0m17.349547986s) [noise=true , address=64.49.13.138:4482 , piece_id=MAW2M4US2OQNOS3WI67AA7NIBTJDE5T2JQTXKFPBLGVGXKVCFGTQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.048750700 (6h0m17.421330055s) [noise=true , address=79.127.205.246:4280 , piece_id=FLBOZI3UGTQXAQBLZ2F45Z3OEYL65F5PQBXRF2DQMZ7NGF7VDRPA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.221642403 (6h0m17.594221758s) [noise=true , address=79.127.220.98:4432 , piece_id=2SUTFQ7FVDPAM2H4W3OLW6AVH3VJ33IJFE4L52SA3ET5LGBMOMPQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.317070435 (6h0m17.68964979s) [noise=true , address=45.91.95.4:4011 , piece_id=ZMBAS6KKHYVIRE6OBL2NFVCWMM4TKQ7J64BD3SYXIK54XCEDI2XA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.324002660 (6h0m17.696582015s) [noise=true , address=45.91.95.5:4019 , piece_id=YTSBCN2LPHEMRRAZIHJGVB7NBOTRXVEHD2TWUCTTLBNFFYEWWDKA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.460976317 (6h0m17.833555672s) [noise=true , address=79.127.201.215:4357 , piece_id=75XWAG2VQWEYCWN2Q3WRIZSBI7LDOISZJPXQLQ2GIPNEN34VLSUQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.462584603 (6h0m17.835163958s) [noise=true , address=45.91.116.6:4007 , piece_id=A2AFVTZBXSGBOE7XIM3B5HRN4STITFBXO6MZRJ365URNAAOZ3PSQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.490392973 (6h0m17.862972328s) [noise=true , address=45.91.116.9:4264 , piece_id=GHM5BKZKMT2QOHHBOS4FSPOWEWSFQRGRVVR4GSNDPHVIFRZHQL2A, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.502040780 (6h0m17.874620135s) [noise=true , address=45.91.116.14:4219 , piece_id=PDW6JPRHCVMHHOWIHT6JKV4CTSC3JY6NDKDVFZSO47OYNPZ7PGGA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.600434096 (6h0m17.973013451s) [noise=true , address=79.127.219.51:4211 , piece_id=HO5PLWBVU5YAYNYMN352NA73SM4WBJV4WZ7XLSQTCQ76UAL2TOJA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.609008984 (6h0m17.981588339s) [noise=true , address=104.36.82.19:4423 , piece_id=PIJUHDUND36HQL67I33D7A3EULMJHS7N55MN4OFQJ4OWNQ42O47A, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.620825273 (6h0m17.993404628s) [noise=true , address=79.127.219.38:4023 , piece_id=PTPOZGLYV4MZTOFZNN3NCGYL7DFAST4PXKEUIVTMFL7HV4GZTCLQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.620838659 (6h0m17.993418014s) [seg_pos={0 0}, upload=4769]: Segment reached optimal threshold of 14 pieces. Jul 19 04:51:11.621067807 (6h0m17.993647163s) [noise=true , address=77.90.31.11:4564 , piece_id=L2U7JFJRDQOFA2AYVAGTI4ZW4VABCXKMXWBIUWQO2QNNUUBJ74BA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection; write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection Jul 19 04:51:11.621134234 (6h0m17.993713589s) [noise=true , address=77.90.31.10:4546 , piece_id=R6LCA5WZOEBDURL5S3MEK22YPVYXX6EXSS4J4W57543HELYXOKZA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK): protocol: read tcp 188.166.225.170:58476->77.90.31.10:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621174191 (6h0m17.993753546s) [noise=true , address=77.90.31.16:4768 , piece_id=YNLSONFM6COZEVF6ZNQR2EA6V4LSQFZJONIXR4AFTNNMFGPV57JA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UchewmtxBrtSZSpVAJFdZmcTsQVQxkjK9re6cgUGqDhyonQeF): protocol: read tcp 188.166.225.170:56326->77.90.31.16:4768: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621216842 (6h0m17.993796197s) [noise=true , address=77.90.31.18:4813 , piece_id=VDOT7BB4O7XYJHW6BFTAYQZZYLZU5PAUXVJGSXJ7FDI4TH425COA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128WthA1x9wp1dV1vkpGnoveZQpU9CSgDCZfw5GMQuVpCNSVAM5): protocol: read tcp 188.166.225.170:49704->77.90.31.18:4813: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621234917 (6h0m17.993814272s) [seg_pos={0 0}, upload=4769]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 19 04:51:11.621301444 (6h0m17.993880799s) [upload=4769]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 19 04:51:11.621305402 (6h0m17.993884758s) [upload=4769]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 19 04:51:11.923136970 (6h0m18.295716325s) [upload=4769]: Done uploading segments... Jul 19 04:51:11.923156627 (6h0m18.295735972s) [upload=4769]: Upload finished. err: <nil> Duration: 5.535083521s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.........................................XX...X...X....X..X.] duration: 8.354567673s FAIL (2025-07-19 04:58:52 UTC) orbiter warning: ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== :1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: read tcp 188.166.225.170:50230->77.90.31.19:4841: use of closed network connection; context canceled Jul 19 04:56:54.832707046 (6h6m1.205286401s) [seg_pos={0 0}, upload=4846]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 19 04:56:54.832779073 (6h6m1.205358428s) [upload=4846]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 19 04:56:54.832783182 (6h6m1.205362527s) [upload=4846]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 19 04:56:55.133341022 (6h6m1.505920377s) [upload=4846]: Done uploading segments... Jul 19 04:56:55.133358805 (6h6m1.50593816s) [upload=4846]: Upload finished. err: <nil> Jul 19 04:57:50.449535830 (6h6m56.822115185s) [upload=4860]: Starting upload Jul 19 04:57:50.449598590 (6h6m56.822177935s) [upload=4860]: Done starting upload Jul 19 04:57:50.449668684 (6h6m56.822248039s) [upload=4860]: Uploading segments... Jul 19 04:57:50.449781009 (6h6m56.822360364s) [upload=4860]: Got next segment. Inline: false Jul 19 04:57:50.449828921 (6h6m56.822408276s) [upload=4860]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 04:57:50.449833700 (6h6m56.822413055s) [upload=4860]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 04:57:50.682857520 (6h6m57.055436875s) [seg_pos={0 0}, upload=4860]: Begin upload segment... Jul 19 04:57:50.682898699 (6h6m57.055478054s) [seg_pos={0 0}, upload=4860]: Done begin upload segment. Jul 19 04:57:50.682913377 (6h6m57.055492732s) [noise=true , address=45.91.116.6:4974 , piece_id=2IEHPCFHSH7LGYE4M6UER42T22OPGFVWEGGQ6CIZ2FZ2HP3YAZWA, seg_pos={0 0}, upload=4860]: Uploading piece... Jul 19 04:57:50.682920360 (6h6m57.055499705s) [noise=true , address=77.90.31.11:4553 , piece_id=ECHSCGV2FSRCWW5M4QKGZMCQ4AHFCIUFBWPRURFCPFJR5YUOC42A, seg_pos={0 0}, upload=4860]: Uploading piece... Jul 19 04:57:50.682990905 (6h6m57.05557026s) [noise=true , address=104.36.82.16:4294 , piece_id=4HRMC3IOCWF5I252RH6JTAANPYLX4QFM3UCKXOD67BTAUJ6EDOCQ, seg_pos={0 0}, upload=4860]: Uploading piece... Jul 19 04:57:50.683253236 (6h6m57.055832591s) [noise=true , address=104.36.82.5:4839 , piece_id=W3C5L2U6NK2DYJSOIFNC2PF2XFXEOHAAWABP7UA3KM7LIXAWSPSQ, seg_pos={0 0}, upload=4860]: Uploading piece... Jul 19 04:57:50.683500520 (6h6m57.056079875s) [noise=true , address=77.90.31.17:4771 , piece_id=GQQ5R52KDOOI5WOXKTJKZXTIECNU3QWDM4KLGOR7UMORDPMHJUJQ, seg_pos={0 0}, upload=4860]: Uploading piece... Jul 19 04:57:50.683736031 (6h6m57.056315376s) [noise=true , address=45.91.116.4:4938 , piece_id=REUW3UKOOZX56BI3V5ALKCID53EIRRDBMSA2BHCLMQSZGTYJGMDA, seg_pos={0 0}, upload=4860]: Uploading piece... Jul 19 04:57:50.684122380 (6h6m57.056701725s) [noise=true , address=45.91.116.7:4011 , piece_id=LI35TO3RUCTEYICC7HUI6P34Z2KJH3S43L2GRAE646FAVXNHRXWA, seg_pos={0 0}, upload=4860]: Uploading piece... Jul 19 04:57:50.684336981 (6h6m57.056916336s) [noise=true , address=104.36.82.10:4077 , piece_id=ZCLW6MI23GWBBSZVV44LHDG4BBLKB656IJCNCVC2PGGVFJK4KV6Q, seg_pos={0 0}, upload=4860]: Uploading piece... Jul 19 04:57:50.684555689 (6h6m57.057135044s) [noise=true , address=104.36.82.3:4767 , piece_id=MNQW4WEQRFJQXUKITTVSOQN4J3T7AEXPYN37Z6OMFLFOASRQKZYA, seg_pos={0 0}, upload=4860]: Uploading piece... Jul 19 04:57:50.684820506 (6h6m57.057399861s) [noise=true , address=45.91.116.5:4298 , piece_id=AUGUELVJFWXEPSOEWRCGZM6X4TI2JOUQNYP6WP4VL25ZC7FAPRTQ, seg_pos={0 0}, upload=4860]: Uploading piece... Jul 19 04:57:50.685013646 (6h6m57.057593001s) [noise=true , address=45.91.116.5:4299 , piece_id=RPRYWZHCNVP72ZBR4BGMCOI245RG7ZEIWKVCNX3HRZVODGJIM26A, seg_pos={0 0}, upload=4860]: Uploading piece... Jul 19 04:57:50.685194391 (6h6m57.057773746s) [noise=true , address=45.91.116.9:4263 , piece_id=CPEMTRRYLTUFXTAHXB7P2THNI7T5LZCFO26XPIGATVNH2UWQH6PA, seg_pos={0 0}, upload=4860]: Uploading piece... Jul 19 04:57:50.685376059 (6h6m57.057955414s) [noise=true , address=45.91.116.10:4081 , piece_id=IN6OJGRO2YDSGNIY3ZNTYTVYLEWKHFKOT24PADM7K7YQNXK5BTEA, seg_pos={0 0}, upload=4860]: Uploading piece... Jul 19 04:57:50.685559630 (6h6m57.058138985s) [noise=true , address=77.90.31.14:4661 , piece_id=X6UIHHUTOKC6R3BIJXZETTR4EBDJ4EHQYCYJVG3KLSAPVMZZMADA, seg_pos={0 0}, upload=4860]: Uploading piece... Jul 19 04:57:50.685741949 (6h6m57.058321304s) [noise=true , address=77.90.31.18:4806 , piece_id=MYZZEPG546S5I67HCIBSNMTK2UGT7XOBMTGYPTPZAIMNLA6TWAKA, seg_pos={0 0}, upload=4860]: Uploading piece... Jul 19 04:57:50.685953323 (6h6m57.058532678s) [noise=true , address=45.91.116.9:4261 , piece_id=6GLY2NCWLS5X6IHCVYB76Z4F2UJ47GXH5HJ6BUBUU7DVW5BT5I5Q, seg_pos={0 0}, upload=4860]: Uploading piece... Jul 19 04:57:52.382568857 (6h6m58.755148202s) [upload=4860]: Next returned nil segment Jul 19 04:57:52.382588915 (6h6m58.75516827s) [upload=4860]: Waiting for error group managing segments... Jul 19 04:57:52.712870771 (6h6m59.085450116s) [noise=true , address=45.91.116.5:4298 , piece_id=AUGUELVJFWXEPSOEWRCGZM6X4TI2JOUQNYP6WP4VL25ZC7FAPRTQ, seg_pos={0 0}, upload=4860]: Done uploading piece. err: <nil> Jul 19 04:57:52.713384304 (6h6m59.085963649s) [noise=true , address=45.91.116.9:4263 , piece_id=CPEMTRRYLTUFXTAHXB7P2THNI7T5LZCFO26XPIGATVNH2UWQH6PA, seg_pos={0 0}, upload=4860]: Done uploading piece. err: <nil> Jul 19 04:57:52.719689228 (6h6m59.092268583s) [noise=true , address=104.36.82.5:4839 , piece_id=W3C5L2U6NK2DYJSOIFNC2PF2XFXEOHAAWABP7UA3KM7LIXAWSPSQ, seg_pos={0 0}, upload=4860]: Done uploading piece. err: <nil> Jul 19 04:57:52.723321257 (6h6m59.095900612s) [noise=true , address=104.36.82.10:4077 , piece_id=ZCLW6MI23GWBBSZVV44LHDG4BBLKB656IJCNCVC2PGGVFJK4KV6Q, seg_pos={0 0}, upload=4860]: Done uploading piece. err: <nil> Jul 19 04:57:52.797624770 (6h6m59.170204115s) [noise=true , address=45.91.116.6:4974 , piece_id=2IEHPCFHSH7LGYE4M6UER42T22OPGFVWEGGQ6CIZ2FZ2HP3YAZWA, seg_pos={0 0}, upload=4860]: Done uploading piece. err: <nil> Jul 19 04:57:52.798066866 (6h6m59.170646221s) [noise=true , address=45.91.116.4:4938 , piece_id=REUW3UKOOZX56BI3V5ALKCID53EIRRDBMSA2BHCLMQSZGTYJGMDA, seg_pos={0 0}, upload=4860]: Done uploading piece. err: <nil> Jul 19 04:57:52.820454393 (6h6m59.193033738s) [noise=true , address=104.36.82.16:4294 , piece_id=4HRMC3IOCWF5I252RH6JTAANPYLX4QFM3UCKXOD67BTAUJ6EDOCQ, seg_pos={0 0}, upload=4860]: Done uploading piece. err: <nil> Jul 19 04:57:52.864818369 (6h6m59.237397724s) [noise=true , address=45.91.116.10:4081 , piece_id=IN6OJGRO2YDSGNIY3ZNTYTVYLEWKHFKOT24PADM7K7YQNXK5BTEA, seg_pos={0 0}, upload=4860]: Done uploading piece. err: <nil> Jul 19 04:57:52.873179036 (6h6m59.245758391s) [noise=true , address=45.91.116.9:4261 , piece_id=6GLY2NCWLS5X6IHCVYB76Z4F2UJ47GXH5HJ6BUBUU7DVW5BT5I5Q, seg_pos={0 0}, upload=4860]: Done uploading piece. err: <nil> Jul 19 04:57:52.881637702 (6h6m59.254217047s) [noise=true , address=45.91.116.5:4299 , piece_id=RPRYWZHCNVP72ZBR4BGMCOI245RG7ZEIWKVCNX3HRZVODGJIM26A, seg_pos={0 0}, upload=4860]: Done uploading piece. err: <nil> Jul 19 04:57:52.884454370 (6h6m59.257033725s) [noise=true , address=104.36.82.3:4767 , piece_id=MNQW4WEQRFJQXUKITTVSOQN4J3T7AEXPYN37Z6OMFLFOASRQKZYA, seg_pos={0 0}, upload=4860]: Done uploading piece. err: <nil> Jul 19 04:57:54.501916491 (6h7m0.874495846s) [noise=true , address=45.91.116.7:4011 , piece_id=LI35TO3RUCTEYICC7HUI6P34Z2KJH3S43L2GRAE646FAVXNHRXWA, seg_pos={0 0}, upload=4860]: Done uploading piece. err: <nil> Jul 19 04:58:19.391729500 (6h7m25.764308845s) [noise=true , address=77.90.31.11:4553 , piece_id=ECHSCGV2FSRCWW5M4QKGZMCQ4AHFCIUFBWPRURFCPFJR5YUOC42A, seg_pos={0 0}, upload=4860]: Done uploading piece. err: <nil> Jul 19 04:58:19.391756702 (6h7m25.764336047s) [seg_pos={0 0}, upload=4860]: Segment reached optimal threshold of 13 pieces. Jul 19 04:58:19.391845321 (6h7m25.764424676s) [noise=true , address=77.90.31.14:4661 , piece_id=X6UIHHUTOKC6R3BIJXZETTR4EBDJ4EHQYCYJVG3KLSAPVMZZMADA, seg_pos={0 0}, upload=4860]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Jul 19 04:58:19.392247070 (6h7m25.764826425s) [noise=true , address=77.90.31.18:4806 , piece_id=MYZZEPG546S5I67HCIBSNMTK2UGT7XOBMTGYPTPZAIMNLA6TWAKA, seg_pos={0 0}, upload=4860]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: write tcp 188.166.225.170:45822->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:45822->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45822->77.90.31.18:4806: use of closed network connection Jul 19 04:58:19.392314228 (6h7m25.764893583s) [noise=true , address=77.90.31.17:4771 , piece_id=GQQ5R52KDOOI5WOXKTJKZXTIECNU3QWDM4KLGOR7UMORDPMHJUJQ, seg_pos={0 0}, upload=4860]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2): protocol: write tcp 188.166.225.170:46240->77.90.31.17:4771: use of closed network connection; write tcp 188.166.225.170:46240->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46240->77.90.31.17:4771: use of closed network connection Jul 19 04:58:19.392340238 (6h7m25.764919593s) [seg_pos={0 0}, upload=4860]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 19 04:58:19.392427785 (6h7m25.76500714s) [upload=4860]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 19 04:58:19.392432124 (6h7m25.76501148s) [upload=4860]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 19 04:58:19.702762764 (6h7m26.075342109s) [upload=4860]: Done uploading segments... Jul 19 04:58:19.702781199 (6h7m26.075360554s) [upload=4860]: Upload finished. err: <nil> Duration: 8.354596189s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 21.496162777s FAIL (2025-07-19 04:58:37 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 4855s) [upload=4769]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 04:51:09.109709627 (6h0m15.482288982s) [upload=4769]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 04:51:09.556145753 (6h0m15.928725108s) [seg_pos={0 0}, upload=4769]: Begin upload segment... Jul 19 04:51:09.556194817 (6h0m15.928774172s) [seg_pos={0 0}, upload=4769]: Done begin upload segment. Jul 19 04:51:09.556223893 (6h0m15.928803248s) [noise=true , address=79.127.219.38:4023 , piece_id=PTPOZGLYV4MZTOFZNN3NCGYL7DFAST4PXKEUIVTMFL7HV4GZTCLQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556252197 (6h0m15.928831552s) [noise=true , address=45.91.95.4:4011 , piece_id=ZMBAS6KKHYVIRE6OBL2NFVCWMM4TKQ7J64BD3SYXIK54XCEDI2XA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556579383 (6h0m15.929158738s) [noise=true , address=45.91.116.6:4007 , piece_id=A2AFVTZBXSGBOE7XIM3B5HRN4STITFBXO6MZRJ365URNAAOZ3PSQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556854469 (6h0m15.929433824s) [noise=true , address=104.36.82.19:4423 , piece_id=PIJUHDUND36HQL67I33D7A3EULMJHS7N55MN4OFQJ4OWNQ42O47A, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557094107 (6h0m15.929673462s) [noise=true , address=45.91.116.9:4264 , piece_id=GHM5BKZKMT2QOHHBOS4FSPOWEWSFQRGRVVR4GSNDPHVIFRZHQL2A, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557336381 (6h0m15.929915736s) [noise=true , address=79.127.205.235:4660 , piece_id=UOUOGE2VP7PX7EMRZLPMCWFCYEZERM3DQMCQ6NG4J2K2DEIZR63Q, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557557885 (6h0m15.93013724s) [noise=true , address=79.127.205.246:4280 , piece_id=FLBOZI3UGTQXAQBLZ2F45Z3OEYL65F5PQBXRF2DQMZ7NGF7VDRPA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557762296 (6h0m15.930341651s) [noise=true , address=79.127.205.233:4622 , piece_id=6WWNYQTH5MGAJOEB7BXKZDJOMCAZXDW66J3OEANGDNJDNUMLZM7Q, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558014689 (6h0m15.930594044s) [noise=true , address=64.49.13.138:4482 , piece_id=MAW2M4US2OQNOS3WI67AA7NIBTJDE5T2JQTXKFPBLGVGXKVCFGTQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558215974 (6h0m15.930795319s) [noise=true , address=77.90.31.10:4546 , piece_id=R6LCA5WZOEBDURL5S3MEK22YPVYXX6EXSS4J4W57543HELYXOKZA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558409364 (6h0m15.930988719s) [noise=true , address=45.91.95.5:4019 , piece_id=YTSBCN2LPHEMRRAZIHJGVB7NBOTRXVEHD2TWUCTTLBNFFYEWWDKA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558623504 (6h0m15.931202859s) [noise=true , address=79.127.201.215:4357 , piece_id=75XWAG2VQWEYCWN2Q3WRIZSBI7LDOISZJPXQLQ2GIPNEN34VLSUQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558805162 (6h0m15.931384517s) [noise=true , address=45.91.116.14:4219 , piece_id=PDW6JPRHCVMHHOWIHT6JKV4CTSC3JY6NDKDVFZSO47OYNPZ7PGGA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558999373 (6h0m15.931578728s) [noise=true , address=77.90.31.11:4564 , piece_id=L2U7JFJRDQOFA2AYVAGTI4ZW4VABCXKMXWBIUWQO2QNNUUBJ74BA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559184768 (6h0m15.931764123s) [noise=true , address=77.90.31.16:4768 , piece_id=YNLSONFM6COZEVF6ZNQR2EA6V4LSQFZJONIXR4AFTNNMFGPV57JA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559368719 (6h0m15.931948074s) [noise=true , address=79.127.220.98:4432 , piece_id=2SUTFQ7FVDPAM2H4W3OLW6AVH3VJ33IJFE4L52SA3ET5LGBMOMPQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559549706 (6h0m15.932129061s) [noise=true , address=77.90.31.18:4813 , piece_id=VDOT7BB4O7XYJHW6BFTAYQZZYLZU5PAUXVJGSXJ7FDI4TH425COA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559727836 (6h0m15.932307191s) [noise=true , address=79.127.219.51:4211 , piece_id=HO5PLWBVU5YAYNYMN352NA73SM4WBJV4WZ7XLSQTCQ76UAL2TOJA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:10.663359087 (6h0m17.035938442s) [upload=4769]: Next returned nil segment Jul 19 04:51:10.663378664 (6h0m17.035958019s) [upload=4769]: Waiting for error group managing segments... Jul 19 04:51:10.962945479 (6h0m17.335524834s) [noise=true , address=79.127.205.233:4622 , piece_id=6WWNYQTH5MGAJOEB7BXKZDJOMCAZXDW66J3OEANGDNJDNUMLZM7Q, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:10.967689807 (6h0m17.340269152s) [noise=true , address=79.127.205.235:4660 , piece_id=UOUOGE2VP7PX7EMRZLPMCWFCYEZERM3DQMCQ6NG4J2K2DEIZR63Q, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:10.976968631 (6h0m17.349547986s) [noise=true , address=64.49.13.138:4482 , piece_id=MAW2M4US2OQNOS3WI67AA7NIBTJDE5T2JQTXKFPBLGVGXKVCFGTQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.048750700 (6h0m17.421330055s) [noise=true , address=79.127.205.246:4280 , piece_id=FLBOZI3UGTQXAQBLZ2F45Z3OEYL65F5PQBXRF2DQMZ7NGF7VDRPA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.221642403 (6h0m17.594221758s) [noise=true , address=79.127.220.98:4432 , piece_id=2SUTFQ7FVDPAM2H4W3OLW6AVH3VJ33IJFE4L52SA3ET5LGBMOMPQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.317070435 (6h0m17.68964979s) [noise=true , address=45.91.95.4:4011 , piece_id=ZMBAS6KKHYVIRE6OBL2NFVCWMM4TKQ7J64BD3SYXIK54XCEDI2XA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.324002660 (6h0m17.696582015s) [noise=true , address=45.91.95.5:4019 , piece_id=YTSBCN2LPHEMRRAZIHJGVB7NBOTRXVEHD2TWUCTTLBNFFYEWWDKA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.460976317 (6h0m17.833555672s) [noise=true , address=79.127.201.215:4357 , piece_id=75XWAG2VQWEYCWN2Q3WRIZSBI7LDOISZJPXQLQ2GIPNEN34VLSUQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.462584603 (6h0m17.835163958s) [noise=true , address=45.91.116.6:4007 , piece_id=A2AFVTZBXSGBOE7XIM3B5HRN4STITFBXO6MZRJ365URNAAOZ3PSQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.490392973 (6h0m17.862972328s) [noise=true , address=45.91.116.9:4264 , piece_id=GHM5BKZKMT2QOHHBOS4FSPOWEWSFQRGRVVR4GSNDPHVIFRZHQL2A, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.502040780 (6h0m17.874620135s) [noise=true , address=45.91.116.14:4219 , piece_id=PDW6JPRHCVMHHOWIHT6JKV4CTSC3JY6NDKDVFZSO47OYNPZ7PGGA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.600434096 (6h0m17.973013451s) [noise=true , address=79.127.219.51:4211 , piece_id=HO5PLWBVU5YAYNYMN352NA73SM4WBJV4WZ7XLSQTCQ76UAL2TOJA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.609008984 (6h0m17.981588339s) [noise=true , address=104.36.82.19:4423 , piece_id=PIJUHDUND36HQL67I33D7A3EULMJHS7N55MN4OFQJ4OWNQ42O47A, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.620825273 (6h0m17.993404628s) [noise=true , address=79.127.219.38:4023 , piece_id=PTPOZGLYV4MZTOFZNN3NCGYL7DFAST4PXKEUIVTMFL7HV4GZTCLQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.620838659 (6h0m17.993418014s) [seg_pos={0 0}, upload=4769]: Segment reached optimal threshold of 14 pieces. Jul 19 04:51:11.621067807 (6h0m17.993647163s) [noise=true , address=77.90.31.11:4564 , piece_id=L2U7JFJRDQOFA2AYVAGTI4ZW4VABCXKMXWBIUWQO2QNNUUBJ74BA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection; write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection Jul 19 04:51:11.621134234 (6h0m17.993713589s) [noise=true , address=77.90.31.10:4546 , piece_id=R6LCA5WZOEBDURL5S3MEK22YPVYXX6EXSS4J4W57543HELYXOKZA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK): protocol: read tcp 188.166.225.170:58476->77.90.31.10:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621174191 (6h0m17.993753546s) [noise=true , address=77.90.31.16:4768 , piece_id=YNLSONFM6COZEVF6ZNQR2EA6V4LSQFZJONIXR4AFTNNMFGPV57JA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UchewmtxBrtSZSpVAJFdZmcTsQVQxkjK9re6cgUGqDhyonQeF): protocol: read tcp 188.166.225.170:56326->77.90.31.16:4768: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621216842 (6h0m17.993796197s) [noise=true , address=77.90.31.18:4813 , piece_id=VDOT7BB4O7XYJHW6BFTAYQZZYLZU5PAUXVJGSXJ7FDI4TH425COA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128WthA1x9wp1dV1vkpGnoveZQpU9CSgDCZfw5GMQuVpCNSVAM5): protocol: read tcp 188.166.225.170:49704->77.90.31.18:4813: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621234917 (6h0m17.993814272s) [seg_pos={0 0}, upload=4769]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 19 04:51:11.621301444 (6h0m17.993880799s) [upload=4769]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 19 04:51:11.621305402 (6h0m17.993884758s) [upload=4769]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 19 04:51:11.923136970 (6h0m18.295716325s) [upload=4769]: Done uploading segments... Jul 19 04:51:11.923156627 (6h0m18.295735972s) [upload=4769]: Upload finished. err: <nil> Duration: 21.496187333s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..........................................XX...X...X....X..X] duration: 30.001547258s FAIL (2025-07-19 04:58:19 UTC) orbiter warning: ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== :1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: read tcp 188.166.225.170:50230->77.90.31.19:4841: use of closed network connection; context canceled Jul 19 04:56:54.832707046 (6h6m1.205286401s) [seg_pos={0 0}, upload=4846]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 19 04:56:54.832779073 (6h6m1.205358428s) [upload=4846]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 19 04:56:54.832783182 (6h6m1.205362527s) [upload=4846]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 19 04:56:55.133341022 (6h6m1.505920377s) [upload=4846]: Done uploading segments... Jul 19 04:56:55.133358805 (6h6m1.50593816s) [upload=4846]: Upload finished. err: <nil> Jul 19 04:57:50.449535830 (6h6m56.822115185s) [upload=4860]: Starting upload Jul 19 04:57:50.449598590 (6h6m56.822177935s) [upload=4860]: Done starting upload Jul 19 04:57:50.449668684 (6h6m56.822248039s) [upload=4860]: Uploading segments... Jul 19 04:57:50.449781009 (6h6m56.822360364s) [upload=4860]: Got next segment. Inline: false Jul 19 04:57:50.449828921 (6h6m56.822408276s) [upload=4860]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 04:57:50.449833700 (6h6m56.822413055s) [upload=4860]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 04:57:50.682857520 (6h6m57.055436875s) [seg_pos={0 0}, upload=4860]: Begin upload segment... Jul 19 04:57:50.682898699 (6h6m57.055478054s) [seg_pos={0 0}, upload=4860]: Done begin upload segment. Jul 19 04:57:50.682913377 (6h6m57.055492732s) [noise=true , address=45.91.116.6:4974 , piece_id=2IEHPCFHSH7LGYE4M6UER42T22OPGFVWEGGQ6CIZ2FZ2HP3YAZWA, seg_pos={0 0}, upload=4860]: Uploading piece... Jul 19 04:57:50.682920360 (6h6m57.055499705s) [noise=true , address=77.90.31.11:4553 , piece_id=ECHSCGV2FSRCWW5M4QKGZMCQ4AHFCIUFBWPRURFCPFJR5YUOC42A, seg_pos={0 0}, upload=4860]: Uploading piece... Jul 19 04:57:50.682990905 (6h6m57.05557026s) [noise=true , address=104.36.82.16:4294 , piece_id=4HRMC3IOCWF5I252RH6JTAANPYLX4QFM3UCKXOD67BTAUJ6EDOCQ, seg_pos={0 0}, upload=4860]: Uploading piece... Jul 19 04:57:50.683253236 (6h6m57.055832591s) [noise=true , address=104.36.82.5:4839 , piece_id=W3C5L2U6NK2DYJSOIFNC2PF2XFXEOHAAWABP7UA3KM7LIXAWSPSQ, seg_pos={0 0}, upload=4860]: Uploading piece... Jul 19 04:57:50.683500520 (6h6m57.056079875s) [noise=true , address=77.90.31.17:4771 , piece_id=GQQ5R52KDOOI5WOXKTJKZXTIECNU3QWDM4KLGOR7UMORDPMHJUJQ, seg_pos={0 0}, upload=4860]: Uploading piece... Jul 19 04:57:50.683736031 (6h6m57.056315376s) [noise=true , address=45.91.116.4:4938 , piece_id=REUW3UKOOZX56BI3V5ALKCID53EIRRDBMSA2BHCLMQSZGTYJGMDA, seg_pos={0 0}, upload=4860]: Uploading piece... Jul 19 04:57:50.684122380 (6h6m57.056701725s) [noise=true , address=45.91.116.7:4011 , piece_id=LI35TO3RUCTEYICC7HUI6P34Z2KJH3S43L2GRAE646FAVXNHRXWA, seg_pos={0 0}, upload=4860]: Uploading piece... Jul 19 04:57:50.684336981 (6h6m57.056916336s) [noise=true , address=104.36.82.10:4077 , piece_id=ZCLW6MI23GWBBSZVV44LHDG4BBLKB656IJCNCVC2PGGVFJK4KV6Q, seg_pos={0 0}, upload=4860]: Uploading piece... Jul 19 04:57:50.684555689 (6h6m57.057135044s) [noise=true , address=104.36.82.3:4767 , piece_id=MNQW4WEQRFJQXUKITTVSOQN4J3T7AEXPYN37Z6OMFLFOASRQKZYA, seg_pos={0 0}, upload=4860]: Uploading piece... Jul 19 04:57:50.684820506 (6h6m57.057399861s) [noise=true , address=45.91.116.5:4298 , piece_id=AUGUELVJFWXEPSOEWRCGZM6X4TI2JOUQNYP6WP4VL25ZC7FAPRTQ, seg_pos={0 0}, upload=4860]: Uploading piece... Jul 19 04:57:50.685013646 (6h6m57.057593001s) [noise=true , address=45.91.116.5:4299 , piece_id=RPRYWZHCNVP72ZBR4BGMCOI245RG7ZEIWKVCNX3HRZVODGJIM26A, seg_pos={0 0}, upload=4860]: Uploading piece... Jul 19 04:57:50.685194391 (6h6m57.057773746s) [noise=true , address=45.91.116.9:4263 , piece_id=CPEMTRRYLTUFXTAHXB7P2THNI7T5LZCFO26XPIGATVNH2UWQH6PA, seg_pos={0 0}, upload=4860]: Uploading piece... Jul 19 04:57:50.685376059 (6h6m57.057955414s) [noise=true , address=45.91.116.10:4081 , piece_id=IN6OJGRO2YDSGNIY3ZNTYTVYLEWKHFKOT24PADM7K7YQNXK5BTEA, seg_pos={0 0}, upload=4860]: Uploading piece... Jul 19 04:57:50.685559630 (6h6m57.058138985s) [noise=true , address=77.90.31.14:4661 , piece_id=X6UIHHUTOKC6R3BIJXZETTR4EBDJ4EHQYCYJVG3KLSAPVMZZMADA, seg_pos={0 0}, upload=4860]: Uploading piece... Jul 19 04:57:50.685741949 (6h6m57.058321304s) [noise=true , address=77.90.31.18:4806 , piece_id=MYZZEPG546S5I67HCIBSNMTK2UGT7XOBMTGYPTPZAIMNLA6TWAKA, seg_pos={0 0}, upload=4860]: Uploading piece... Jul 19 04:57:50.685953323 (6h6m57.058532678s) [noise=true , address=45.91.116.9:4261 , piece_id=6GLY2NCWLS5X6IHCVYB76Z4F2UJ47GXH5HJ6BUBUU7DVW5BT5I5Q, seg_pos={0 0}, upload=4860]: Uploading piece... Jul 19 04:57:52.382568857 (6h6m58.755148202s) [upload=4860]: Next returned nil segment Jul 19 04:57:52.382588915 (6h6m58.75516827s) [upload=4860]: Waiting for error group managing segments... Jul 19 04:57:52.712870771 (6h6m59.085450116s) [noise=true , address=45.91.116.5:4298 , piece_id=AUGUELVJFWXEPSOEWRCGZM6X4TI2JOUQNYP6WP4VL25ZC7FAPRTQ, seg_pos={0 0}, upload=4860]: Done uploading piece. err: <nil> Jul 19 04:57:52.713384304 (6h6m59.085963649s) [noise=true , address=45.91.116.9:4263 , piece_id=CPEMTRRYLTUFXTAHXB7P2THNI7T5LZCFO26XPIGATVNH2UWQH6PA, seg_pos={0 0}, upload=4860]: Done uploading piece. err: <nil> Jul 19 04:57:52.719689228 (6h6m59.092268583s) [noise=true , address=104.36.82.5:4839 , piece_id=W3C5L2U6NK2DYJSOIFNC2PF2XFXEOHAAWABP7UA3KM7LIXAWSPSQ, seg_pos={0 0}, upload=4860]: Done uploading piece. err: <nil> Jul 19 04:57:52.723321257 (6h6m59.095900612s) [noise=true , address=104.36.82.10:4077 , piece_id=ZCLW6MI23GWBBSZVV44LHDG4BBLKB656IJCNCVC2PGGVFJK4KV6Q, seg_pos={0 0}, upload=4860]: Done uploading piece. err: <nil> Jul 19 04:57:52.797624770 (6h6m59.170204115s) [noise=true , address=45.91.116.6:4974 , piece_id=2IEHPCFHSH7LGYE4M6UER42T22OPGFVWEGGQ6CIZ2FZ2HP3YAZWA, seg_pos={0 0}, upload=4860]: Done uploading piece. err: <nil> Jul 19 04:57:52.798066866 (6h6m59.170646221s) [noise=true , address=45.91.116.4:4938 , piece_id=REUW3UKOOZX56BI3V5ALKCID53EIRRDBMSA2BHCLMQSZGTYJGMDA, seg_pos={0 0}, upload=4860]: Done uploading piece. err: <nil> Jul 19 04:57:52.820454393 (6h6m59.193033738s) [noise=true , address=104.36.82.16:4294 , piece_id=4HRMC3IOCWF5I252RH6JTAANPYLX4QFM3UCKXOD67BTAUJ6EDOCQ, seg_pos={0 0}, upload=4860]: Done uploading piece. err: <nil> Jul 19 04:57:52.864818369 (6h6m59.237397724s) [noise=true , address=45.91.116.10:4081 , piece_id=IN6OJGRO2YDSGNIY3ZNTYTVYLEWKHFKOT24PADM7K7YQNXK5BTEA, seg_pos={0 0}, upload=4860]: Done uploading piece. err: <nil> Jul 19 04:57:52.873179036 (6h6m59.245758391s) [noise=true , address=45.91.116.9:4261 , piece_id=6GLY2NCWLS5X6IHCVYB76Z4F2UJ47GXH5HJ6BUBUU7DVW5BT5I5Q, seg_pos={0 0}, upload=4860]: Done uploading piece. err: <nil> Jul 19 04:57:52.881637702 (6h6m59.254217047s) [noise=true , address=45.91.116.5:4299 , piece_id=RPRYWZHCNVP72ZBR4BGMCOI245RG7ZEIWKVCNX3HRZVODGJIM26A, seg_pos={0 0}, upload=4860]: Done uploading piece. err: <nil> Jul 19 04:57:52.884454370 (6h6m59.257033725s) [noise=true , address=104.36.82.3:4767 , piece_id=MNQW4WEQRFJQXUKITTVSOQN4J3T7AEXPYN37Z6OMFLFOASRQKZYA, seg_pos={0 0}, upload=4860]: Done uploading piece. err: <nil> Jul 19 04:57:54.501916491 (6h7m0.874495846s) [noise=true , address=45.91.116.7:4011 , piece_id=LI35TO3RUCTEYICC7HUI6P34Z2KJH3S43L2GRAE646FAVXNHRXWA, seg_pos={0 0}, upload=4860]: Done uploading piece. err: <nil> Jul 19 04:58:19.391729500 (6h7m25.764308845s) [noise=true , address=77.90.31.11:4553 , piece_id=ECHSCGV2FSRCWW5M4QKGZMCQ4AHFCIUFBWPRURFCPFJR5YUOC42A, seg_pos={0 0}, upload=4860]: Done uploading piece. err: <nil> Jul 19 04:58:19.391756702 (6h7m25.764336047s) [seg_pos={0 0}, upload=4860]: Segment reached optimal threshold of 13 pieces. Jul 19 04:58:19.391845321 (6h7m25.764424676s) [noise=true , address=77.90.31.14:4661 , piece_id=X6UIHHUTOKC6R3BIJXZETTR4EBDJ4EHQYCYJVG3KLSAPVMZZMADA, seg_pos={0 0}, upload=4860]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Jul 19 04:58:19.392247070 (6h7m25.764826425s) [noise=true , address=77.90.31.18:4806 , piece_id=MYZZEPG546S5I67HCIBSNMTK2UGT7XOBMTGYPTPZAIMNLA6TWAKA, seg_pos={0 0}, upload=4860]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: write tcp 188.166.225.170:45822->77.90.31.18:4806: use of closed network connection; write tcp 188.166.225.170:45822->77.90.31.18:4806: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45822->77.90.31.18:4806: use of closed network connection Jul 19 04:58:19.392314228 (6h7m25.764893583s) [noise=true , address=77.90.31.17:4771 , piece_id=GQQ5R52KDOOI5WOXKTJKZXTIECNU3QWDM4KLGOR7UMORDPMHJUJQ, seg_pos={0 0}, upload=4860]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2): protocol: write tcp 188.166.225.170:46240->77.90.31.17:4771: use of closed network connection; write tcp 188.166.225.170:46240->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46240->77.90.31.17:4771: use of closed network connection Jul 19 04:58:19.392340238 (6h7m25.764919593s) [seg_pos={0 0}, upload=4860]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 19 04:58:19.392427785 (6h7m25.76500714s) [upload=4860]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 19 04:58:19.392432124 (6h7m25.76501148s) [upload=4860]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 19 04:58:19.702762764 (6h7m26.075342109s) [upload=4860]: Done uploading segments... Jul 19 04:58:19.702781199 (6h7m26.075360554s) [upload=4860]: Upload finished. err: <nil> Duration: 30.001568138s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 22.619526126s FAIL (2025-07-19 04:57:57 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 4855s) [upload=4769]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 04:51:09.109709627 (6h0m15.482288982s) [upload=4769]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 04:51:09.556145753 (6h0m15.928725108s) [seg_pos={0 0}, upload=4769]: Begin upload segment... Jul 19 04:51:09.556194817 (6h0m15.928774172s) [seg_pos={0 0}, upload=4769]: Done begin upload segment. Jul 19 04:51:09.556223893 (6h0m15.928803248s) [noise=true , address=79.127.219.38:4023 , piece_id=PTPOZGLYV4MZTOFZNN3NCGYL7DFAST4PXKEUIVTMFL7HV4GZTCLQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556252197 (6h0m15.928831552s) [noise=true , address=45.91.95.4:4011 , piece_id=ZMBAS6KKHYVIRE6OBL2NFVCWMM4TKQ7J64BD3SYXIK54XCEDI2XA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556579383 (6h0m15.929158738s) [noise=true , address=45.91.116.6:4007 , piece_id=A2AFVTZBXSGBOE7XIM3B5HRN4STITFBXO6MZRJ365URNAAOZ3PSQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556854469 (6h0m15.929433824s) [noise=true , address=104.36.82.19:4423 , piece_id=PIJUHDUND36HQL67I33D7A3EULMJHS7N55MN4OFQJ4OWNQ42O47A, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557094107 (6h0m15.929673462s) [noise=true , address=45.91.116.9:4264 , piece_id=GHM5BKZKMT2QOHHBOS4FSPOWEWSFQRGRVVR4GSNDPHVIFRZHQL2A, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557336381 (6h0m15.929915736s) [noise=true , address=79.127.205.235:4660 , piece_id=UOUOGE2VP7PX7EMRZLPMCWFCYEZERM3DQMCQ6NG4J2K2DEIZR63Q, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557557885 (6h0m15.93013724s) [noise=true , address=79.127.205.246:4280 , piece_id=FLBOZI3UGTQXAQBLZ2F45Z3OEYL65F5PQBXRF2DQMZ7NGF7VDRPA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557762296 (6h0m15.930341651s) [noise=true , address=79.127.205.233:4622 , piece_id=6WWNYQTH5MGAJOEB7BXKZDJOMCAZXDW66J3OEANGDNJDNUMLZM7Q, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558014689 (6h0m15.930594044s) [noise=true , address=64.49.13.138:4482 , piece_id=MAW2M4US2OQNOS3WI67AA7NIBTJDE5T2JQTXKFPBLGVGXKVCFGTQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558215974 (6h0m15.930795319s) [noise=true , address=77.90.31.10:4546 , piece_id=R6LCA5WZOEBDURL5S3MEK22YPVYXX6EXSS4J4W57543HELYXOKZA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558409364 (6h0m15.930988719s) [noise=true , address=45.91.95.5:4019 , piece_id=YTSBCN2LPHEMRRAZIHJGVB7NBOTRXVEHD2TWUCTTLBNFFYEWWDKA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558623504 (6h0m15.931202859s) [noise=true , address=79.127.201.215:4357 , piece_id=75XWAG2VQWEYCWN2Q3WRIZSBI7LDOISZJPXQLQ2GIPNEN34VLSUQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558805162 (6h0m15.931384517s) [noise=true , address=45.91.116.14:4219 , piece_id=PDW6JPRHCVMHHOWIHT6JKV4CTSC3JY6NDKDVFZSO47OYNPZ7PGGA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558999373 (6h0m15.931578728s) [noise=true , address=77.90.31.11:4564 , piece_id=L2U7JFJRDQOFA2AYVAGTI4ZW4VABCXKMXWBIUWQO2QNNUUBJ74BA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559184768 (6h0m15.931764123s) [noise=true , address=77.90.31.16:4768 , piece_id=YNLSONFM6COZEVF6ZNQR2EA6V4LSQFZJONIXR4AFTNNMFGPV57JA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559368719 (6h0m15.931948074s) [noise=true , address=79.127.220.98:4432 , piece_id=2SUTFQ7FVDPAM2H4W3OLW6AVH3VJ33IJFE4L52SA3ET5LGBMOMPQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559549706 (6h0m15.932129061s) [noise=true , address=77.90.31.18:4813 , piece_id=VDOT7BB4O7XYJHW6BFTAYQZZYLZU5PAUXVJGSXJ7FDI4TH425COA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559727836 (6h0m15.932307191s) [noise=true , address=79.127.219.51:4211 , piece_id=HO5PLWBVU5YAYNYMN352NA73SM4WBJV4WZ7XLSQTCQ76UAL2TOJA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:10.663359087 (6h0m17.035938442s) [upload=4769]: Next returned nil segment Jul 19 04:51:10.663378664 (6h0m17.035958019s) [upload=4769]: Waiting for error group managing segments... Jul 19 04:51:10.962945479 (6h0m17.335524834s) [noise=true , address=79.127.205.233:4622 , piece_id=6WWNYQTH5MGAJOEB7BXKZDJOMCAZXDW66J3OEANGDNJDNUMLZM7Q, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:10.967689807 (6h0m17.340269152s) [noise=true , address=79.127.205.235:4660 , piece_id=UOUOGE2VP7PX7EMRZLPMCWFCYEZERM3DQMCQ6NG4J2K2DEIZR63Q, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:10.976968631 (6h0m17.349547986s) [noise=true , address=64.49.13.138:4482 , piece_id=MAW2M4US2OQNOS3WI67AA7NIBTJDE5T2JQTXKFPBLGVGXKVCFGTQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.048750700 (6h0m17.421330055s) [noise=true , address=79.127.205.246:4280 , piece_id=FLBOZI3UGTQXAQBLZ2F45Z3OEYL65F5PQBXRF2DQMZ7NGF7VDRPA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.221642403 (6h0m17.594221758s) [noise=true , address=79.127.220.98:4432 , piece_id=2SUTFQ7FVDPAM2H4W3OLW6AVH3VJ33IJFE4L52SA3ET5LGBMOMPQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.317070435 (6h0m17.68964979s) [noise=true , address=45.91.95.4:4011 , piece_id=ZMBAS6KKHYVIRE6OBL2NFVCWMM4TKQ7J64BD3SYXIK54XCEDI2XA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.324002660 (6h0m17.696582015s) [noise=true , address=45.91.95.5:4019 , piece_id=YTSBCN2LPHEMRRAZIHJGVB7NBOTRXVEHD2TWUCTTLBNFFYEWWDKA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.460976317 (6h0m17.833555672s) [noise=true , address=79.127.201.215:4357 , piece_id=75XWAG2VQWEYCWN2Q3WRIZSBI7LDOISZJPXQLQ2GIPNEN34VLSUQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.462584603 (6h0m17.835163958s) [noise=true , address=45.91.116.6:4007 , piece_id=A2AFVTZBXSGBOE7XIM3B5HRN4STITFBXO6MZRJ365URNAAOZ3PSQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.490392973 (6h0m17.862972328s) [noise=true , address=45.91.116.9:4264 , piece_id=GHM5BKZKMT2QOHHBOS4FSPOWEWSFQRGRVVR4GSNDPHVIFRZHQL2A, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.502040780 (6h0m17.874620135s) [noise=true , address=45.91.116.14:4219 , piece_id=PDW6JPRHCVMHHOWIHT6JKV4CTSC3JY6NDKDVFZSO47OYNPZ7PGGA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.600434096 (6h0m17.973013451s) [noise=true , address=79.127.219.51:4211 , piece_id=HO5PLWBVU5YAYNYMN352NA73SM4WBJV4WZ7XLSQTCQ76UAL2TOJA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.609008984 (6h0m17.981588339s) [noise=true , address=104.36.82.19:4423 , piece_id=PIJUHDUND36HQL67I33D7A3EULMJHS7N55MN4OFQJ4OWNQ42O47A, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.620825273 (6h0m17.993404628s) [noise=true , address=79.127.219.38:4023 , piece_id=PTPOZGLYV4MZTOFZNN3NCGYL7DFAST4PXKEUIVTMFL7HV4GZTCLQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.620838659 (6h0m17.993418014s) [seg_pos={0 0}, upload=4769]: Segment reached optimal threshold of 14 pieces. Jul 19 04:51:11.621067807 (6h0m17.993647163s) [noise=true , address=77.90.31.11:4564 , piece_id=L2U7JFJRDQOFA2AYVAGTI4ZW4VABCXKMXWBIUWQO2QNNUUBJ74BA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection; write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection Jul 19 04:51:11.621134234 (6h0m17.993713589s) [noise=true , address=77.90.31.10:4546 , piece_id=R6LCA5WZOEBDURL5S3MEK22YPVYXX6EXSS4J4W57543HELYXOKZA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK): protocol: read tcp 188.166.225.170:58476->77.90.31.10:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621174191 (6h0m17.993753546s) [noise=true , address=77.90.31.16:4768 , piece_id=YNLSONFM6COZEVF6ZNQR2EA6V4LSQFZJONIXR4AFTNNMFGPV57JA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UchewmtxBrtSZSpVAJFdZmcTsQVQxkjK9re6cgUGqDhyonQeF): protocol: read tcp 188.166.225.170:56326->77.90.31.16:4768: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621216842 (6h0m17.993796197s) [noise=true , address=77.90.31.18:4813 , piece_id=VDOT7BB4O7XYJHW6BFTAYQZZYLZU5PAUXVJGSXJ7FDI4TH425COA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128WthA1x9wp1dV1vkpGnoveZQpU9CSgDCZfw5GMQuVpCNSVAM5): protocol: read tcp 188.166.225.170:49704->77.90.31.18:4813: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621234917 (6h0m17.993814272s) [seg_pos={0 0}, upload=4769]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 19 04:51:11.621301444 (6h0m17.993880799s) [upload=4769]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 19 04:51:11.621305402 (6h0m17.993884758s) [upload=4769]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 19 04:51:11.923136970 (6h0m18.295716325s) [upload=4769]: Done uploading segments... Jul 19 04:51:11.923156627 (6h0m18.295735972s) [upload=4769]: Upload finished. err: <nil> Duration: 22.619549211s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...........................................XX...X...X....X..] duration: 5.962077536s FAIL (2025-07-19 04:56:57 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== , piece_id=H3RS5SKJEAS6EJVUVL23PZJZK4ESBPMT4XFQLDWJ772HLO3ATNFA, seg_pos={0 0}, upload=4814]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2): protocol: read tcp 188.166.225.170:44382->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:54:49.483148896 (6h3m55.855728251s) [seg_pos={0 0}, upload=4814]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 19 04:54:49.483297240 (6h3m55.855876595s) [upload=4814]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 19 04:54:49.483302940 (6h3m55.855882296s) [upload=4814]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 19 04:54:49.768720852 (6h3m56.141300197s) [upload=4814]: Done uploading segments... Jul 19 04:54:49.768736953 (6h3m56.141316309s) [upload=4814]: Upload finished. err: <nil> Jul 19 04:55:09.399582574 (6h4m15.772161929s) [upload=4816]: Starting upload Jul 19 04:55:09.399653200 (6h4m15.772232555s) [upload=4816]: Done starting upload Jul 19 04:55:09.399839355 (6h4m15.77241871s) [upload=4816]: Uploading segments... Jul 19 04:55:09.399932795 (6h4m15.77251215s) [upload=4816]: Got next segment. Inline: false Jul 19 04:55:09.399963403 (6h4m15.772542758s) [upload=4816]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 04:55:09.399968362 (6h4m15.772547717s) [upload=4816]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 04:55:09.636242181 (6h4m16.008821566s) [seg_pos={0 0}, upload=4816]: Begin upload segment... Jul 19 04:55:09.636305171 (6h4m16.008884566s) [seg_pos={0 0}, upload=4816]: Done begin upload segment. Jul 19 04:55:09.636338345 (6h4m16.00891771s) [noise=true , address=77.90.31.17:4772 , piece_id=KNWEKMQKTZLH5SDXZVFYNSXTKEVRFWNHQZDX7AC2PT4UY75HXH6Q, seg_pos={0 0}, upload=4816]: Uploading piece... Jul 19 04:55:09.636351590 (6h4m16.008930966s) [noise=true , address=104.36.82.5:4839 , piece_id=JOLMF2TL5GPDPHULGWDSMMECQVZTLAP6OOYLWMUOXZ5SFOJNWEPA, seg_pos={0 0}, upload=4816]: Uploading piece... Jul 19 04:55:09.637061027 (6h4m16.009640402s) [noise=true , address=77.90.31.10:4519 , piece_id=5DHT7APO4F5FE5BUZ4NPFIYLHWTSTL4ONWNSPKEYUI3U4MDJZ74Q, seg_pos={0 0}, upload=4816]: Uploading piece... Jul 19 04:55:09.637175126 (6h4m16.009754511s) [noise=true , address=45.91.116.4:4938 , piece_id=6GLS7RJDQB4GWGG4OQ5LH5H7HPY3JICLHGMHGMCRK4ULAWV4MMGA, seg_pos={0 0}, upload=4816]: Uploading piece... Jul 19 04:55:09.637481442 (6h4m16.010060797s) [noise=true , address=77.90.31.10:4517 , piece_id=OVFR6D5HRV2GIVZFNMSARX5V4NPZ6VQOMVO7YPCKCRS7M2DSCVVQ, seg_pos={0 0}, upload=4816]: Uploading piece... Jul 19 04:55:09.637688919 (6h4m16.010268284s) [noise=true , address=77.90.31.19:4841 , piece_id=6A6R24Q5QRQ6IY7EOMXJXS3V2XSJ2QWLKGFCEMHSWWLEJEX5T4WA, seg_pos={0 0}, upload=4816]: Uploading piece... Jul 19 04:55:09.637961350 (6h4m16.010540715s) [noise=true , address=45.91.116.10:4083 , piece_id=VC4UZRAUWRUPIW7UNFE6MUFLUZ2OR45WFIIFIWKSXUQOGRSQO4QA, seg_pos={0 0}, upload=4816]: Uploading piece... Jul 19 04:55:09.638182553 (6h4m16.010761918s) [noise=true , address=45.91.116.6:4975 , piece_id=5N6OHHGIL7FCDDRROSLADQ7IVTHXJZ6JHTBYMOHZSVKUDJK5PK2Q, seg_pos={0 0}, upload=4816]: Uploading piece... Jul 19 04:55:09.638338452 (6h4m16.010917817s) [noise=true , address=45.91.116.7:4009 , piece_id=TPU2ZB7JE75FNRD6EJUUQLULA4BBK3R5V6M45BH6C4O335GRSNVQ, seg_pos={0 0}, upload=4816]: Uploading piece... Jul 19 04:55:09.638609661 (6h4m16.011189016s) [noise=true , address=77.90.31.16:4733 , piece_id=2CJ2Y7UZXFUAM76C2AE7C7ICNRH5YSWFLRLQ5OW75NJIUIQ6X34Q, seg_pos={0 0}, upload=4816]: Uploading piece... Jul 19 04:55:09.638672220 (6h4m16.011251575s) [noise=true , address=77.90.31.19:4842 , piece_id=AIZ2TITLX6ICQH2FUGZQGTMADMTBAV7KQH7AQGSDV2QN4NVIEFRA, seg_pos={0 0}, upload=4816]: Uploading piece... Jul 19 04:55:09.639034553 (6h4m16.011613908s) [noise=true , address=104.36.82.16:4294 , piece_id=J3POTH76JRKBRDUJ35OBQSZQSJSG7FDEVL63ZHDQAFED4OQKLGVQ, seg_pos={0 0}, upload=4816]: Uploading piece... Jul 19 04:55:09.639048791 (6h4m16.011628156s) [noise=true , address=77.90.31.21:4879 , piece_id=LSMV6DN3SRZ632E74MHGAZMYXHB5QSBQZE3GUCQJJCE65FG6VTEQ, seg_pos={0 0}, upload=4816]: Uploading piece... Jul 19 04:55:09.645656865 (6h4m16.01823622s) [noise=true , address=45.91.116.13:4155 , piece_id=XOWTOIPZTIFR4WS2ZD73RKYM3G7522VIMQEAIKAMXQJWWJ27SHAQ, seg_pos={0 0}, upload=4816]: Uploading piece... Jul 19 04:55:09.650516984 (6h4m16.023096339s) [noise=true , address=104.36.82.3:4767 , piece_id=KGQIBQLUR7U3WV2OBJRVRBUZYH5CKFK3QQZFY6HKU6HHL62ZLH6A, seg_pos={0 0}, upload=4816]: Uploading piece... Jul 19 04:55:09.653463011 (6h4m16.026042376s) [noise=true , address=45.91.116.4:4937 , piece_id=C3WMKEGWLKJR76HKWVH24FWEOBSIUM56KQIBFQ3Z653IJOCDJQMA, seg_pos={0 0}, upload=4816]: Uploading piece... Jul 19 04:55:11.264865890 (6h4m17.637445245s) [upload=4816]: Next returned nil segment Jul 19 04:55:11.264882751 (6h4m17.637462096s) [upload=4816]: Waiting for error group managing segments... Jul 19 04:55:11.651472245 (6h4m18.0240516s) [noise=true , address=45.91.116.7:4009 , piece_id=TPU2ZB7JE75FNRD6EJUUQLULA4BBK3R5V6M45BH6C4O335GRSNVQ, seg_pos={0 0}, upload=4816]: Done uploading piece. err: <nil> Jul 19 04:55:11.674433390 (6h4m18.047012745s) [noise=true , address=45.91.116.6:4975 , piece_id=5N6OHHGIL7FCDDRROSLADQ7IVTHXJZ6JHTBYMOHZSVKUDJK5PK2Q, seg_pos={0 0}, upload=4816]: Done uploading piece. err: <nil> Jul 19 04:55:11.696721285 (6h4m18.06930064s) [noise=true , address=104.36.82.5:4839 , piece_id=JOLMF2TL5GPDPHULGWDSMMECQVZTLAP6OOYLWMUOXZ5SFOJNWEPA, seg_pos={0 0}, upload=4816]: Done uploading piece. err: <nil> Jul 19 04:55:11.735650613 (6h4m18.108229968s) [noise=true , address=45.91.116.4:4938 , piece_id=6GLS7RJDQB4GWGG4OQ5LH5H7HPY3JICLHGMHGMCRK4ULAWV4MMGA, seg_pos={0 0}, upload=4816]: Done uploading piece. err: <nil> Jul 19 04:55:11.763301912 (6h4m18.135881267s) [noise=true , address=45.91.116.4:4937 , piece_id=C3WMKEGWLKJR76HKWVH24FWEOBSIUM56KQIBFQ3Z653IJOCDJQMA, seg_pos={0 0}, upload=4816]: Done uploading piece. err: <nil> Jul 19 04:55:11.782646817 (6h4m18.155226172s) [noise=true , address=104.36.82.16:4294 , piece_id=J3POTH76JRKBRDUJ35OBQSZQSJSG7FDEVL63ZHDQAFED4OQKLGVQ, seg_pos={0 0}, upload=4816]: Done uploading piece. err: <nil> Jul 19 04:55:11.796102432 (6h4m18.168681787s) [noise=true , address=104.36.82.3:4767 , piece_id=KGQIBQLUR7U3WV2OBJRVRBUZYH5CKFK3QQZFY6HKU6HHL62ZLH6A, seg_pos={0 0}, upload=4816]: Done uploading piece. err: <nil> Jul 19 04:55:11.834706729 (6h4m18.207286104s) [noise=true , address=45.91.116.13:4155 , piece_id=XOWTOIPZTIFR4WS2ZD73RKYM3G7522VIMQEAIKAMXQJWWJ27SHAQ, seg_pos={0 0}, upload=4816]: Done uploading piece. err: <nil> Jul 19 04:55:11.949325290 (6h4m18.321904635s) [noise=true , address=45.91.116.10:4083 , piece_id=VC4UZRAUWRUPIW7UNFE6MUFLUZ2OR45WFIIFIWKSXUQOGRSQO4QA, seg_pos={0 0}, upload=4816]: Done uploading piece. err: <nil> Jul 19 04:55:28.501817807 (6h4m34.874397162s) [noise=true , address=77.90.31.19:4842 , piece_id=AIZ2TITLX6ICQH2FUGZQGTMADMTBAV7KQH7AQGSDV2QN4NVIEFRA, seg_pos={0 0}, upload=4816]: Done uploading piece. err: <nil> Jul 19 04:55:28.779257974 (6h4m35.151837329s) [noise=true , address=77.90.31.17:4772 , piece_id=KNWEKMQKTZLH5SDXZVFYNSXTKEVRFWNHQZDX7AC2PT4UY75HXH6Q, seg_pos={0 0}, upload=4816]: Done uploading piece. err: <nil> Jul 19 04:55:34.666405539 (6h4m41.038984894s) [noise=true , address=77.90.31.21:4879 , piece_id=LSMV6DN3SRZ632E74MHGAZMYXHB5QSBQZE3GUCQJJCE65FG6VTEQ, seg_pos={0 0}, upload=4816]: Done uploading piece. err: <nil> Jul 19 04:55:37.245493876 (6h4m43.618073231s) [noise=true , address=77.90.31.19:4841 , piece_id=6A6R24Q5QRQ6IY7EOMXJXS3V2XSJ2QWLKGFCEMHSWWLEJEX5T4WA, seg_pos={0 0}, upload=4816]: Done uploading piece. err: <nil> Jul 19 04:55:37.245518853 (6h4m43.618098208s) [seg_pos={0 0}, upload=4816]: Segment reached optimal threshold of 13 pieces. Jul 19 04:55:37.245691143 (6h4m43.618270498s) [noise=true , address=77.90.31.16:4733 , piece_id=2CJ2Y7UZXFUAM76C2AE7C7ICNRH5YSWFLRLQ5OW75NJIUIQ6X34Q, seg_pos={0 0}, upload=4816]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf): protocol: write tcp 188.166.225.170:32862->77.90.31.16:4733: use of closed network connection; write tcp 188.166.225.170:32862->77.90.31.16:4733: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32862->77.90.31.16:4733: use of closed network connection Jul 19 04:55:37.245739325 (6h4m43.61831868s) [noise=true , address=77.90.31.10:4517 , piece_id=OVFR6D5HRV2GIVZFNMSARX5V4NPZ6VQOMVO7YPCKCRS7M2DSCVVQ, seg_pos={0 0}, upload=4816]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Jul 19 04:55:37.245767339 (6h4m43.618346694s) [noise=true , address=77.90.31.10:4519 , piece_id=5DHT7APO4F5FE5BUZ4NPFIYLHWTSTL4ONWNSPKEYUI3U4MDJZ74Q, seg_pos={0 0}, upload=4816]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Jul 19 04:55:37.245791565 (6h4m43.61837092s) [seg_pos={0 0}, upload=4816]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 19 04:55:37.245896426 (6h4m43.618475771s) [upload=4816]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 19 04:55:37.245900734 (6h4m43.618480089s) [upload=4816]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 19 04:55:37.634125928 (6h4m44.006705283s) [upload=4816]: Done uploading segments... Jul 19 04:55:37.634142831 (6h4m44.006722186s) [upload=4816]: Upload finished. err: <nil> Duration: 5.96210542s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 23.949403695s FAIL (2025-07-19 04:56:47 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 4855s) [upload=4769]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 04:51:09.109709627 (6h0m15.482288982s) [upload=4769]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 04:51:09.556145753 (6h0m15.928725108s) [seg_pos={0 0}, upload=4769]: Begin upload segment... Jul 19 04:51:09.556194817 (6h0m15.928774172s) [seg_pos={0 0}, upload=4769]: Done begin upload segment. Jul 19 04:51:09.556223893 (6h0m15.928803248s) [noise=true , address=79.127.219.38:4023 , piece_id=PTPOZGLYV4MZTOFZNN3NCGYL7DFAST4PXKEUIVTMFL7HV4GZTCLQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556252197 (6h0m15.928831552s) [noise=true , address=45.91.95.4:4011 , piece_id=ZMBAS6KKHYVIRE6OBL2NFVCWMM4TKQ7J64BD3SYXIK54XCEDI2XA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556579383 (6h0m15.929158738s) [noise=true , address=45.91.116.6:4007 , piece_id=A2AFVTZBXSGBOE7XIM3B5HRN4STITFBXO6MZRJ365URNAAOZ3PSQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556854469 (6h0m15.929433824s) [noise=true , address=104.36.82.19:4423 , piece_id=PIJUHDUND36HQL67I33D7A3EULMJHS7N55MN4OFQJ4OWNQ42O47A, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557094107 (6h0m15.929673462s) [noise=true , address=45.91.116.9:4264 , piece_id=GHM5BKZKMT2QOHHBOS4FSPOWEWSFQRGRVVR4GSNDPHVIFRZHQL2A, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557336381 (6h0m15.929915736s) [noise=true , address=79.127.205.235:4660 , piece_id=UOUOGE2VP7PX7EMRZLPMCWFCYEZERM3DQMCQ6NG4J2K2DEIZR63Q, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557557885 (6h0m15.93013724s) [noise=true , address=79.127.205.246:4280 , piece_id=FLBOZI3UGTQXAQBLZ2F45Z3OEYL65F5PQBXRF2DQMZ7NGF7VDRPA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557762296 (6h0m15.930341651s) [noise=true , address=79.127.205.233:4622 , piece_id=6WWNYQTH5MGAJOEB7BXKZDJOMCAZXDW66J3OEANGDNJDNUMLZM7Q, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558014689 (6h0m15.930594044s) [noise=true , address=64.49.13.138:4482 , piece_id=MAW2M4US2OQNOS3WI67AA7NIBTJDE5T2JQTXKFPBLGVGXKVCFGTQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558215974 (6h0m15.930795319s) [noise=true , address=77.90.31.10:4546 , piece_id=R6LCA5WZOEBDURL5S3MEK22YPVYXX6EXSS4J4W57543HELYXOKZA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558409364 (6h0m15.930988719s) [noise=true , address=45.91.95.5:4019 , piece_id=YTSBCN2LPHEMRRAZIHJGVB7NBOTRXVEHD2TWUCTTLBNFFYEWWDKA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558623504 (6h0m15.931202859s) [noise=true , address=79.127.201.215:4357 , piece_id=75XWAG2VQWEYCWN2Q3WRIZSBI7LDOISZJPXQLQ2GIPNEN34VLSUQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558805162 (6h0m15.931384517s) [noise=true , address=45.91.116.14:4219 , piece_id=PDW6JPRHCVMHHOWIHT6JKV4CTSC3JY6NDKDVFZSO47OYNPZ7PGGA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558999373 (6h0m15.931578728s) [noise=true , address=77.90.31.11:4564 , piece_id=L2U7JFJRDQOFA2AYVAGTI4ZW4VABCXKMXWBIUWQO2QNNUUBJ74BA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559184768 (6h0m15.931764123s) [noise=true , address=77.90.31.16:4768 , piece_id=YNLSONFM6COZEVF6ZNQR2EA6V4LSQFZJONIXR4AFTNNMFGPV57JA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559368719 (6h0m15.931948074s) [noise=true , address=79.127.220.98:4432 , piece_id=2SUTFQ7FVDPAM2H4W3OLW6AVH3VJ33IJFE4L52SA3ET5LGBMOMPQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559549706 (6h0m15.932129061s) [noise=true , address=77.90.31.18:4813 , piece_id=VDOT7BB4O7XYJHW6BFTAYQZZYLZU5PAUXVJGSXJ7FDI4TH425COA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559727836 (6h0m15.932307191s) [noise=true , address=79.127.219.51:4211 , piece_id=HO5PLWBVU5YAYNYMN352NA73SM4WBJV4WZ7XLSQTCQ76UAL2TOJA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:10.663359087 (6h0m17.035938442s) [upload=4769]: Next returned nil segment Jul 19 04:51:10.663378664 (6h0m17.035958019s) [upload=4769]: Waiting for error group managing segments... Jul 19 04:51:10.962945479 (6h0m17.335524834s) [noise=true , address=79.127.205.233:4622 , piece_id=6WWNYQTH5MGAJOEB7BXKZDJOMCAZXDW66J3OEANGDNJDNUMLZM7Q, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:10.967689807 (6h0m17.340269152s) [noise=true , address=79.127.205.235:4660 , piece_id=UOUOGE2VP7PX7EMRZLPMCWFCYEZERM3DQMCQ6NG4J2K2DEIZR63Q, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:10.976968631 (6h0m17.349547986s) [noise=true , address=64.49.13.138:4482 , piece_id=MAW2M4US2OQNOS3WI67AA7NIBTJDE5T2JQTXKFPBLGVGXKVCFGTQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.048750700 (6h0m17.421330055s) [noise=true , address=79.127.205.246:4280 , piece_id=FLBOZI3UGTQXAQBLZ2F45Z3OEYL65F5PQBXRF2DQMZ7NGF7VDRPA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.221642403 (6h0m17.594221758s) [noise=true , address=79.127.220.98:4432 , piece_id=2SUTFQ7FVDPAM2H4W3OLW6AVH3VJ33IJFE4L52SA3ET5LGBMOMPQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.317070435 (6h0m17.68964979s) [noise=true , address=45.91.95.4:4011 , piece_id=ZMBAS6KKHYVIRE6OBL2NFVCWMM4TKQ7J64BD3SYXIK54XCEDI2XA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.324002660 (6h0m17.696582015s) [noise=true , address=45.91.95.5:4019 , piece_id=YTSBCN2LPHEMRRAZIHJGVB7NBOTRXVEHD2TWUCTTLBNFFYEWWDKA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.460976317 (6h0m17.833555672s) [noise=true , address=79.127.201.215:4357 , piece_id=75XWAG2VQWEYCWN2Q3WRIZSBI7LDOISZJPXQLQ2GIPNEN34VLSUQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.462584603 (6h0m17.835163958s) [noise=true , address=45.91.116.6:4007 , piece_id=A2AFVTZBXSGBOE7XIM3B5HRN4STITFBXO6MZRJ365URNAAOZ3PSQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.490392973 (6h0m17.862972328s) [noise=true , address=45.91.116.9:4264 , piece_id=GHM5BKZKMT2QOHHBOS4FSPOWEWSFQRGRVVR4GSNDPHVIFRZHQL2A, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.502040780 (6h0m17.874620135s) [noise=true , address=45.91.116.14:4219 , piece_id=PDW6JPRHCVMHHOWIHT6JKV4CTSC3JY6NDKDVFZSO47OYNPZ7PGGA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.600434096 (6h0m17.973013451s) [noise=true , address=79.127.219.51:4211 , piece_id=HO5PLWBVU5YAYNYMN352NA73SM4WBJV4WZ7XLSQTCQ76UAL2TOJA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.609008984 (6h0m17.981588339s) [noise=true , address=104.36.82.19:4423 , piece_id=PIJUHDUND36HQL67I33D7A3EULMJHS7N55MN4OFQJ4OWNQ42O47A, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.620825273 (6h0m17.993404628s) [noise=true , address=79.127.219.38:4023 , piece_id=PTPOZGLYV4MZTOFZNN3NCGYL7DFAST4PXKEUIVTMFL7HV4GZTCLQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.620838659 (6h0m17.993418014s) [seg_pos={0 0}, upload=4769]: Segment reached optimal threshold of 14 pieces. Jul 19 04:51:11.621067807 (6h0m17.993647163s) [noise=true , address=77.90.31.11:4564 , piece_id=L2U7JFJRDQOFA2AYVAGTI4ZW4VABCXKMXWBIUWQO2QNNUUBJ74BA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection; write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection Jul 19 04:51:11.621134234 (6h0m17.993713589s) [noise=true , address=77.90.31.10:4546 , piece_id=R6LCA5WZOEBDURL5S3MEK22YPVYXX6EXSS4J4W57543HELYXOKZA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK): protocol: read tcp 188.166.225.170:58476->77.90.31.10:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621174191 (6h0m17.993753546s) [noise=true , address=77.90.31.16:4768 , piece_id=YNLSONFM6COZEVF6ZNQR2EA6V4LSQFZJONIXR4AFTNNMFGPV57JA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UchewmtxBrtSZSpVAJFdZmcTsQVQxkjK9re6cgUGqDhyonQeF): protocol: read tcp 188.166.225.170:56326->77.90.31.16:4768: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621216842 (6h0m17.993796197s) [noise=true , address=77.90.31.18:4813 , piece_id=VDOT7BB4O7XYJHW6BFTAYQZZYLZU5PAUXVJGSXJ7FDI4TH425COA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128WthA1x9wp1dV1vkpGnoveZQpU9CSgDCZfw5GMQuVpCNSVAM5): protocol: read tcp 188.166.225.170:49704->77.90.31.18:4813: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621234917 (6h0m17.993814272s) [seg_pos={0 0}, upload=4769]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 19 04:51:11.621301444 (6h0m17.993880799s) [upload=4769]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 19 04:51:11.621305402 (6h0m17.993884758s) [upload=4769]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 19 04:51:11.923136970 (6h0m18.295716325s) [upload=4769]: Done uploading segments... Jul 19 04:51:11.923156627 (6h0m18.295735972s) [upload=4769]: Upload finished. err: <nil> Duration: 23.949427812s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [............................................XX...X...X....X.] duration: 5.851490067s FAIL (2025-07-19 04:56:18 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== , piece_id=H3RS5SKJEAS6EJVUVL23PZJZK4ESBPMT4XFQLDWJ772HLO3ATNFA, seg_pos={0 0}, upload=4814]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2): protocol: read tcp 188.166.225.170:44382->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:54:49.483148896 (6h3m55.855728251s) [seg_pos={0 0}, upload=4814]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 19 04:54:49.483297240 (6h3m55.855876595s) [upload=4814]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 19 04:54:49.483302940 (6h3m55.855882296s) [upload=4814]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 19 04:54:49.768720852 (6h3m56.141300197s) [upload=4814]: Done uploading segments... Jul 19 04:54:49.768736953 (6h3m56.141316309s) [upload=4814]: Upload finished. err: <nil> Jul 19 04:55:09.399582574 (6h4m15.772161929s) [upload=4816]: Starting upload Jul 19 04:55:09.399653200 (6h4m15.772232555s) [upload=4816]: Done starting upload Jul 19 04:55:09.399839355 (6h4m15.77241871s) [upload=4816]: Uploading segments... Jul 19 04:55:09.399932795 (6h4m15.77251215s) [upload=4816]: Got next segment. Inline: false Jul 19 04:55:09.399963403 (6h4m15.772542758s) [upload=4816]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 04:55:09.399968362 (6h4m15.772547717s) [upload=4816]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 04:55:09.636242181 (6h4m16.008821566s) [seg_pos={0 0}, upload=4816]: Begin upload segment... Jul 19 04:55:09.636305171 (6h4m16.008884566s) [seg_pos={0 0}, upload=4816]: Done begin upload segment. Jul 19 04:55:09.636338345 (6h4m16.00891771s) [noise=true , address=77.90.31.17:4772 , piece_id=KNWEKMQKTZLH5SDXZVFYNSXTKEVRFWNHQZDX7AC2PT4UY75HXH6Q, seg_pos={0 0}, upload=4816]: Uploading piece... Jul 19 04:55:09.636351590 (6h4m16.008930966s) [noise=true , address=104.36.82.5:4839 , piece_id=JOLMF2TL5GPDPHULGWDSMMECQVZTLAP6OOYLWMUOXZ5SFOJNWEPA, seg_pos={0 0}, upload=4816]: Uploading piece... Jul 19 04:55:09.637061027 (6h4m16.009640402s) [noise=true , address=77.90.31.10:4519 , piece_id=5DHT7APO4F5FE5BUZ4NPFIYLHWTSTL4ONWNSPKEYUI3U4MDJZ74Q, seg_pos={0 0}, upload=4816]: Uploading piece... Jul 19 04:55:09.637175126 (6h4m16.009754511s) [noise=true , address=45.91.116.4:4938 , piece_id=6GLS7RJDQB4GWGG4OQ5LH5H7HPY3JICLHGMHGMCRK4ULAWV4MMGA, seg_pos={0 0}, upload=4816]: Uploading piece... Jul 19 04:55:09.637481442 (6h4m16.010060797s) [noise=true , address=77.90.31.10:4517 , piece_id=OVFR6D5HRV2GIVZFNMSARX5V4NPZ6VQOMVO7YPCKCRS7M2DSCVVQ, seg_pos={0 0}, upload=4816]: Uploading piece... Jul 19 04:55:09.637688919 (6h4m16.010268284s) [noise=true , address=77.90.31.19:4841 , piece_id=6A6R24Q5QRQ6IY7EOMXJXS3V2XSJ2QWLKGFCEMHSWWLEJEX5T4WA, seg_pos={0 0}, upload=4816]: Uploading piece... Jul 19 04:55:09.637961350 (6h4m16.010540715s) [noise=true , address=45.91.116.10:4083 , piece_id=VC4UZRAUWRUPIW7UNFE6MUFLUZ2OR45WFIIFIWKSXUQOGRSQO4QA, seg_pos={0 0}, upload=4816]: Uploading piece... Jul 19 04:55:09.638182553 (6h4m16.010761918s) [noise=true , address=45.91.116.6:4975 , piece_id=5N6OHHGIL7FCDDRROSLADQ7IVTHXJZ6JHTBYMOHZSVKUDJK5PK2Q, seg_pos={0 0}, upload=4816]: Uploading piece... Jul 19 04:55:09.638338452 (6h4m16.010917817s) [noise=true , address=45.91.116.7:4009 , piece_id=TPU2ZB7JE75FNRD6EJUUQLULA4BBK3R5V6M45BH6C4O335GRSNVQ, seg_pos={0 0}, upload=4816]: Uploading piece... Jul 19 04:55:09.638609661 (6h4m16.011189016s) [noise=true , address=77.90.31.16:4733 , piece_id=2CJ2Y7UZXFUAM76C2AE7C7ICNRH5YSWFLRLQ5OW75NJIUIQ6X34Q, seg_pos={0 0}, upload=4816]: Uploading piece... Jul 19 04:55:09.638672220 (6h4m16.011251575s) [noise=true , address=77.90.31.19:4842 , piece_id=AIZ2TITLX6ICQH2FUGZQGTMADMTBAV7KQH7AQGSDV2QN4NVIEFRA, seg_pos={0 0}, upload=4816]: Uploading piece... Jul 19 04:55:09.639034553 (6h4m16.011613908s) [noise=true , address=104.36.82.16:4294 , piece_id=J3POTH76JRKBRDUJ35OBQSZQSJSG7FDEVL63ZHDQAFED4OQKLGVQ, seg_pos={0 0}, upload=4816]: Uploading piece... Jul 19 04:55:09.639048791 (6h4m16.011628156s) [noise=true , address=77.90.31.21:4879 , piece_id=LSMV6DN3SRZ632E74MHGAZMYXHB5QSBQZE3GUCQJJCE65FG6VTEQ, seg_pos={0 0}, upload=4816]: Uploading piece... Jul 19 04:55:09.645656865 (6h4m16.01823622s) [noise=true , address=45.91.116.13:4155 , piece_id=XOWTOIPZTIFR4WS2ZD73RKYM3G7522VIMQEAIKAMXQJWWJ27SHAQ, seg_pos={0 0}, upload=4816]: Uploading piece... Jul 19 04:55:09.650516984 (6h4m16.023096339s) [noise=true , address=104.36.82.3:4767 , piece_id=KGQIBQLUR7U3WV2OBJRVRBUZYH5CKFK3QQZFY6HKU6HHL62ZLH6A, seg_pos={0 0}, upload=4816]: Uploading piece... Jul 19 04:55:09.653463011 (6h4m16.026042376s) [noise=true , address=45.91.116.4:4937 , piece_id=C3WMKEGWLKJR76HKWVH24FWEOBSIUM56KQIBFQ3Z653IJOCDJQMA, seg_pos={0 0}, upload=4816]: Uploading piece... Jul 19 04:55:11.264865890 (6h4m17.637445245s) [upload=4816]: Next returned nil segment Jul 19 04:55:11.264882751 (6h4m17.637462096s) [upload=4816]: Waiting for error group managing segments... Jul 19 04:55:11.651472245 (6h4m18.0240516s) [noise=true , address=45.91.116.7:4009 , piece_id=TPU2ZB7JE75FNRD6EJUUQLULA4BBK3R5V6M45BH6C4O335GRSNVQ, seg_pos={0 0}, upload=4816]: Done uploading piece. err: <nil> Jul 19 04:55:11.674433390 (6h4m18.047012745s) [noise=true , address=45.91.116.6:4975 , piece_id=5N6OHHGIL7FCDDRROSLADQ7IVTHXJZ6JHTBYMOHZSVKUDJK5PK2Q, seg_pos={0 0}, upload=4816]: Done uploading piece. err: <nil> Jul 19 04:55:11.696721285 (6h4m18.06930064s) [noise=true , address=104.36.82.5:4839 , piece_id=JOLMF2TL5GPDPHULGWDSMMECQVZTLAP6OOYLWMUOXZ5SFOJNWEPA, seg_pos={0 0}, upload=4816]: Done uploading piece. err: <nil> Jul 19 04:55:11.735650613 (6h4m18.108229968s) [noise=true , address=45.91.116.4:4938 , piece_id=6GLS7RJDQB4GWGG4OQ5LH5H7HPY3JICLHGMHGMCRK4ULAWV4MMGA, seg_pos={0 0}, upload=4816]: Done uploading piece. err: <nil> Jul 19 04:55:11.763301912 (6h4m18.135881267s) [noise=true , address=45.91.116.4:4937 , piece_id=C3WMKEGWLKJR76HKWVH24FWEOBSIUM56KQIBFQ3Z653IJOCDJQMA, seg_pos={0 0}, upload=4816]: Done uploading piece. err: <nil> Jul 19 04:55:11.782646817 (6h4m18.155226172s) [noise=true , address=104.36.82.16:4294 , piece_id=J3POTH76JRKBRDUJ35OBQSZQSJSG7FDEVL63ZHDQAFED4OQKLGVQ, seg_pos={0 0}, upload=4816]: Done uploading piece. err: <nil> Jul 19 04:55:11.796102432 (6h4m18.168681787s) [noise=true , address=104.36.82.3:4767 , piece_id=KGQIBQLUR7U3WV2OBJRVRBUZYH5CKFK3QQZFY6HKU6HHL62ZLH6A, seg_pos={0 0}, upload=4816]: Done uploading piece. err: <nil> Jul 19 04:55:11.834706729 (6h4m18.207286104s) [noise=true , address=45.91.116.13:4155 , piece_id=XOWTOIPZTIFR4WS2ZD73RKYM3G7522VIMQEAIKAMXQJWWJ27SHAQ, seg_pos={0 0}, upload=4816]: Done uploading piece. err: <nil> Jul 19 04:55:11.949325290 (6h4m18.321904635s) [noise=true , address=45.91.116.10:4083 , piece_id=VC4UZRAUWRUPIW7UNFE6MUFLUZ2OR45WFIIFIWKSXUQOGRSQO4QA, seg_pos={0 0}, upload=4816]: Done uploading piece. err: <nil> Jul 19 04:55:28.501817807 (6h4m34.874397162s) [noise=true , address=77.90.31.19:4842 , piece_id=AIZ2TITLX6ICQH2FUGZQGTMADMTBAV7KQH7AQGSDV2QN4NVIEFRA, seg_pos={0 0}, upload=4816]: Done uploading piece. err: <nil> Jul 19 04:55:28.779257974 (6h4m35.151837329s) [noise=true , address=77.90.31.17:4772 , piece_id=KNWEKMQKTZLH5SDXZVFYNSXTKEVRFWNHQZDX7AC2PT4UY75HXH6Q, seg_pos={0 0}, upload=4816]: Done uploading piece. err: <nil> Jul 19 04:55:34.666405539 (6h4m41.038984894s) [noise=true , address=77.90.31.21:4879 , piece_id=LSMV6DN3SRZ632E74MHGAZMYXHB5QSBQZE3GUCQJJCE65FG6VTEQ, seg_pos={0 0}, upload=4816]: Done uploading piece. err: <nil> Jul 19 04:55:37.245493876 (6h4m43.618073231s) [noise=true , address=77.90.31.19:4841 , piece_id=6A6R24Q5QRQ6IY7EOMXJXS3V2XSJ2QWLKGFCEMHSWWLEJEX5T4WA, seg_pos={0 0}, upload=4816]: Done uploading piece. err: <nil> Jul 19 04:55:37.245518853 (6h4m43.618098208s) [seg_pos={0 0}, upload=4816]: Segment reached optimal threshold of 13 pieces. Jul 19 04:55:37.245691143 (6h4m43.618270498s) [noise=true , address=77.90.31.16:4733 , piece_id=2CJ2Y7UZXFUAM76C2AE7C7ICNRH5YSWFLRLQ5OW75NJIUIQ6X34Q, seg_pos={0 0}, upload=4816]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf): protocol: write tcp 188.166.225.170:32862->77.90.31.16:4733: use of closed network connection; write tcp 188.166.225.170:32862->77.90.31.16:4733: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32862->77.90.31.16:4733: use of closed network connection Jul 19 04:55:37.245739325 (6h4m43.61831868s) [noise=true , address=77.90.31.10:4517 , piece_id=OVFR6D5HRV2GIVZFNMSARX5V4NPZ6VQOMVO7YPCKCRS7M2DSCVVQ, seg_pos={0 0}, upload=4816]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Jul 19 04:55:37.245767339 (6h4m43.618346694s) [noise=true , address=77.90.31.10:4519 , piece_id=5DHT7APO4F5FE5BUZ4NPFIYLHWTSTL4ONWNSPKEYUI3U4MDJZ74Q, seg_pos={0 0}, upload=4816]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Jul 19 04:55:37.245791565 (6h4m43.61837092s) [seg_pos={0 0}, upload=4816]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 19 04:55:37.245896426 (6h4m43.618475771s) [upload=4816]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 19 04:55:37.245900734 (6h4m43.618480089s) [upload=4816]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 19 04:55:37.634125928 (6h4m44.006705283s) [upload=4816]: Done uploading segments... Jul 19 04:55:37.634142831 (6h4m44.006722186s) [upload=4816]: Upload finished. err: <nil> Duration: 5.851514655s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.............................................XX...X...X....X] duration: 30.001965327s FAIL (2025-07-19 04:55:38 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== , piece_id=H3RS5SKJEAS6EJVUVL23PZJZK4ESBPMT4XFQLDWJ772HLO3ATNFA, seg_pos={0 0}, upload=4814]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2): protocol: read tcp 188.166.225.170:44382->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:54:49.483148896 (6h3m55.855728251s) [seg_pos={0 0}, upload=4814]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 19 04:54:49.483297240 (6h3m55.855876595s) [upload=4814]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 19 04:54:49.483302940 (6h3m55.855882296s) [upload=4814]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 19 04:54:49.768720852 (6h3m56.141300197s) [upload=4814]: Done uploading segments... Jul 19 04:54:49.768736953 (6h3m56.141316309s) [upload=4814]: Upload finished. err: <nil> Jul 19 04:55:09.399582574 (6h4m15.772161929s) [upload=4816]: Starting upload Jul 19 04:55:09.399653200 (6h4m15.772232555s) [upload=4816]: Done starting upload Jul 19 04:55:09.399839355 (6h4m15.77241871s) [upload=4816]: Uploading segments... Jul 19 04:55:09.399932795 (6h4m15.77251215s) [upload=4816]: Got next segment. Inline: false Jul 19 04:55:09.399963403 (6h4m15.772542758s) [upload=4816]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 04:55:09.399968362 (6h4m15.772547717s) [upload=4816]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 04:55:09.636242181 (6h4m16.008821566s) [seg_pos={0 0}, upload=4816]: Begin upload segment... Jul 19 04:55:09.636305171 (6h4m16.008884566s) [seg_pos={0 0}, upload=4816]: Done begin upload segment. Jul 19 04:55:09.636338345 (6h4m16.00891771s) [noise=true , address=77.90.31.17:4772 , piece_id=KNWEKMQKTZLH5SDXZVFYNSXTKEVRFWNHQZDX7AC2PT4UY75HXH6Q, seg_pos={0 0}, upload=4816]: Uploading piece... Jul 19 04:55:09.636351590 (6h4m16.008930966s) [noise=true , address=104.36.82.5:4839 , piece_id=JOLMF2TL5GPDPHULGWDSMMECQVZTLAP6OOYLWMUOXZ5SFOJNWEPA, seg_pos={0 0}, upload=4816]: Uploading piece... Jul 19 04:55:09.637061027 (6h4m16.009640402s) [noise=true , address=77.90.31.10:4519 , piece_id=5DHT7APO4F5FE5BUZ4NPFIYLHWTSTL4ONWNSPKEYUI3U4MDJZ74Q, seg_pos={0 0}, upload=4816]: Uploading piece... Jul 19 04:55:09.637175126 (6h4m16.009754511s) [noise=true , address=45.91.116.4:4938 , piece_id=6GLS7RJDQB4GWGG4OQ5LH5H7HPY3JICLHGMHGMCRK4ULAWV4MMGA, seg_pos={0 0}, upload=4816]: Uploading piece... Jul 19 04:55:09.637481442 (6h4m16.010060797s) [noise=true , address=77.90.31.10:4517 , piece_id=OVFR6D5HRV2GIVZFNMSARX5V4NPZ6VQOMVO7YPCKCRS7M2DSCVVQ, seg_pos={0 0}, upload=4816]: Uploading piece... Jul 19 04:55:09.637688919 (6h4m16.010268284s) [noise=true , address=77.90.31.19:4841 , piece_id=6A6R24Q5QRQ6IY7EOMXJXS3V2XSJ2QWLKGFCEMHSWWLEJEX5T4WA, seg_pos={0 0}, upload=4816]: Uploading piece... Jul 19 04:55:09.637961350 (6h4m16.010540715s) [noise=true , address=45.91.116.10:4083 , piece_id=VC4UZRAUWRUPIW7UNFE6MUFLUZ2OR45WFIIFIWKSXUQOGRSQO4QA, seg_pos={0 0}, upload=4816]: Uploading piece... Jul 19 04:55:09.638182553 (6h4m16.010761918s) [noise=true , address=45.91.116.6:4975 , piece_id=5N6OHHGIL7FCDDRROSLADQ7IVTHXJZ6JHTBYMOHZSVKUDJK5PK2Q, seg_pos={0 0}, upload=4816]: Uploading piece... Jul 19 04:55:09.638338452 (6h4m16.010917817s) [noise=true , address=45.91.116.7:4009 , piece_id=TPU2ZB7JE75FNRD6EJUUQLULA4BBK3R5V6M45BH6C4O335GRSNVQ, seg_pos={0 0}, upload=4816]: Uploading piece... Jul 19 04:55:09.638609661 (6h4m16.011189016s) [noise=true , address=77.90.31.16:4733 , piece_id=2CJ2Y7UZXFUAM76C2AE7C7ICNRH5YSWFLRLQ5OW75NJIUIQ6X34Q, seg_pos={0 0}, upload=4816]: Uploading piece... Jul 19 04:55:09.638672220 (6h4m16.011251575s) [noise=true , address=77.90.31.19:4842 , piece_id=AIZ2TITLX6ICQH2FUGZQGTMADMTBAV7KQH7AQGSDV2QN4NVIEFRA, seg_pos={0 0}, upload=4816]: Uploading piece... Jul 19 04:55:09.639034553 (6h4m16.011613908s) [noise=true , address=104.36.82.16:4294 , piece_id=J3POTH76JRKBRDUJ35OBQSZQSJSG7FDEVL63ZHDQAFED4OQKLGVQ, seg_pos={0 0}, upload=4816]: Uploading piece... Jul 19 04:55:09.639048791 (6h4m16.011628156s) [noise=true , address=77.90.31.21:4879 , piece_id=LSMV6DN3SRZ632E74MHGAZMYXHB5QSBQZE3GUCQJJCE65FG6VTEQ, seg_pos={0 0}, upload=4816]: Uploading piece... Jul 19 04:55:09.645656865 (6h4m16.01823622s) [noise=true , address=45.91.116.13:4155 , piece_id=XOWTOIPZTIFR4WS2ZD73RKYM3G7522VIMQEAIKAMXQJWWJ27SHAQ, seg_pos={0 0}, upload=4816]: Uploading piece... Jul 19 04:55:09.650516984 (6h4m16.023096339s) [noise=true , address=104.36.82.3:4767 , piece_id=KGQIBQLUR7U3WV2OBJRVRBUZYH5CKFK3QQZFY6HKU6HHL62ZLH6A, seg_pos={0 0}, upload=4816]: Uploading piece... Jul 19 04:55:09.653463011 (6h4m16.026042376s) [noise=true , address=45.91.116.4:4937 , piece_id=C3WMKEGWLKJR76HKWVH24FWEOBSIUM56KQIBFQ3Z653IJOCDJQMA, seg_pos={0 0}, upload=4816]: Uploading piece... Jul 19 04:55:11.264865890 (6h4m17.637445245s) [upload=4816]: Next returned nil segment Jul 19 04:55:11.264882751 (6h4m17.637462096s) [upload=4816]: Waiting for error group managing segments... Jul 19 04:55:11.651472245 (6h4m18.0240516s) [noise=true , address=45.91.116.7:4009 , piece_id=TPU2ZB7JE75FNRD6EJUUQLULA4BBK3R5V6M45BH6C4O335GRSNVQ, seg_pos={0 0}, upload=4816]: Done uploading piece. err: <nil> Jul 19 04:55:11.674433390 (6h4m18.047012745s) [noise=true , address=45.91.116.6:4975 , piece_id=5N6OHHGIL7FCDDRROSLADQ7IVTHXJZ6JHTBYMOHZSVKUDJK5PK2Q, seg_pos={0 0}, upload=4816]: Done uploading piece. err: <nil> Jul 19 04:55:11.696721285 (6h4m18.06930064s) [noise=true , address=104.36.82.5:4839 , piece_id=JOLMF2TL5GPDPHULGWDSMMECQVZTLAP6OOYLWMUOXZ5SFOJNWEPA, seg_pos={0 0}, upload=4816]: Done uploading piece. err: <nil> Jul 19 04:55:11.735650613 (6h4m18.108229968s) [noise=true , address=45.91.116.4:4938 , piece_id=6GLS7RJDQB4GWGG4OQ5LH5H7HPY3JICLHGMHGMCRK4ULAWV4MMGA, seg_pos={0 0}, upload=4816]: Done uploading piece. err: <nil> Jul 19 04:55:11.763301912 (6h4m18.135881267s) [noise=true , address=45.91.116.4:4937 , piece_id=C3WMKEGWLKJR76HKWVH24FWEOBSIUM56KQIBFQ3Z653IJOCDJQMA, seg_pos={0 0}, upload=4816]: Done uploading piece. err: <nil> Jul 19 04:55:11.782646817 (6h4m18.155226172s) [noise=true , address=104.36.82.16:4294 , piece_id=J3POTH76JRKBRDUJ35OBQSZQSJSG7FDEVL63ZHDQAFED4OQKLGVQ, seg_pos={0 0}, upload=4816]: Done uploading piece. err: <nil> Jul 19 04:55:11.796102432 (6h4m18.168681787s) [noise=true , address=104.36.82.3:4767 , piece_id=KGQIBQLUR7U3WV2OBJRVRBUZYH5CKFK3QQZFY6HKU6HHL62ZLH6A, seg_pos={0 0}, upload=4816]: Done uploading piece. err: <nil> Jul 19 04:55:11.834706729 (6h4m18.207286104s) [noise=true , address=45.91.116.13:4155 , piece_id=XOWTOIPZTIFR4WS2ZD73RKYM3G7522VIMQEAIKAMXQJWWJ27SHAQ, seg_pos={0 0}, upload=4816]: Done uploading piece. err: <nil> Jul 19 04:55:11.949325290 (6h4m18.321904635s) [noise=true , address=45.91.116.10:4083 , piece_id=VC4UZRAUWRUPIW7UNFE6MUFLUZ2OR45WFIIFIWKSXUQOGRSQO4QA, seg_pos={0 0}, upload=4816]: Done uploading piece. err: <nil> Jul 19 04:55:28.501817807 (6h4m34.874397162s) [noise=true , address=77.90.31.19:4842 , piece_id=AIZ2TITLX6ICQH2FUGZQGTMADMTBAV7KQH7AQGSDV2QN4NVIEFRA, seg_pos={0 0}, upload=4816]: Done uploading piece. err: <nil> Jul 19 04:55:28.779257974 (6h4m35.151837329s) [noise=true , address=77.90.31.17:4772 , piece_id=KNWEKMQKTZLH5SDXZVFYNSXTKEVRFWNHQZDX7AC2PT4UY75HXH6Q, seg_pos={0 0}, upload=4816]: Done uploading piece. err: <nil> Jul 19 04:55:34.666405539 (6h4m41.038984894s) [noise=true , address=77.90.31.21:4879 , piece_id=LSMV6DN3SRZ632E74MHGAZMYXHB5QSBQZE3GUCQJJCE65FG6VTEQ, seg_pos={0 0}, upload=4816]: Done uploading piece. err: <nil> Jul 19 04:55:37.245493876 (6h4m43.618073231s) [noise=true , address=77.90.31.19:4841 , piece_id=6A6R24Q5QRQ6IY7EOMXJXS3V2XSJ2QWLKGFCEMHSWWLEJEX5T4WA, seg_pos={0 0}, upload=4816]: Done uploading piece. err: <nil> Jul 19 04:55:37.245518853 (6h4m43.618098208s) [seg_pos={0 0}, upload=4816]: Segment reached optimal threshold of 13 pieces. Jul 19 04:55:37.245691143 (6h4m43.618270498s) [noise=true , address=77.90.31.16:4733 , piece_id=2CJ2Y7UZXFUAM76C2AE7C7ICNRH5YSWFLRLQ5OW75NJIUIQ6X34Q, seg_pos={0 0}, upload=4816]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf): protocol: write tcp 188.166.225.170:32862->77.90.31.16:4733: use of closed network connection; write tcp 188.166.225.170:32862->77.90.31.16:4733: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32862->77.90.31.16:4733: use of closed network connection Jul 19 04:55:37.245739325 (6h4m43.61831868s) [noise=true , address=77.90.31.10:4517 , piece_id=OVFR6D5HRV2GIVZFNMSARX5V4NPZ6VQOMVO7YPCKCRS7M2DSCVVQ, seg_pos={0 0}, upload=4816]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Jul 19 04:55:37.245767339 (6h4m43.618346694s) [noise=true , address=77.90.31.10:4519 , piece_id=5DHT7APO4F5FE5BUZ4NPFIYLHWTSTL4ONWNSPKEYUI3U4MDJZ74Q, seg_pos={0 0}, upload=4816]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Jul 19 04:55:37.245791565 (6h4m43.61837092s) [seg_pos={0 0}, upload=4816]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 19 04:55:37.245896426 (6h4m43.618475771s) [upload=4816]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 19 04:55:37.245900734 (6h4m43.618480089s) [upload=4816]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 19 04:55:37.634125928 (6h4m44.006705283s) [upload=4816]: Done uploading segments... Jul 19 04:55:37.634142831 (6h4m44.006722186s) [upload=4816]: Upload finished. err: <nil> Duration: 30.001990575s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 17.044718349s FAIL (2025-07-19 04:55:32 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 4855s) [upload=4769]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 04:51:09.109709627 (6h0m15.482288982s) [upload=4769]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 04:51:09.556145753 (6h0m15.928725108s) [seg_pos={0 0}, upload=4769]: Begin upload segment... Jul 19 04:51:09.556194817 (6h0m15.928774172s) [seg_pos={0 0}, upload=4769]: Done begin upload segment. Jul 19 04:51:09.556223893 (6h0m15.928803248s) [noise=true , address=79.127.219.38:4023 , piece_id=PTPOZGLYV4MZTOFZNN3NCGYL7DFAST4PXKEUIVTMFL7HV4GZTCLQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556252197 (6h0m15.928831552s) [noise=true , address=45.91.95.4:4011 , piece_id=ZMBAS6KKHYVIRE6OBL2NFVCWMM4TKQ7J64BD3SYXIK54XCEDI2XA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556579383 (6h0m15.929158738s) [noise=true , address=45.91.116.6:4007 , piece_id=A2AFVTZBXSGBOE7XIM3B5HRN4STITFBXO6MZRJ365URNAAOZ3PSQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556854469 (6h0m15.929433824s) [noise=true , address=104.36.82.19:4423 , piece_id=PIJUHDUND36HQL67I33D7A3EULMJHS7N55MN4OFQJ4OWNQ42O47A, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557094107 (6h0m15.929673462s) [noise=true , address=45.91.116.9:4264 , piece_id=GHM5BKZKMT2QOHHBOS4FSPOWEWSFQRGRVVR4GSNDPHVIFRZHQL2A, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557336381 (6h0m15.929915736s) [noise=true , address=79.127.205.235:4660 , piece_id=UOUOGE2VP7PX7EMRZLPMCWFCYEZERM3DQMCQ6NG4J2K2DEIZR63Q, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557557885 (6h0m15.93013724s) [noise=true , address=79.127.205.246:4280 , piece_id=FLBOZI3UGTQXAQBLZ2F45Z3OEYL65F5PQBXRF2DQMZ7NGF7VDRPA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557762296 (6h0m15.930341651s) [noise=true , address=79.127.205.233:4622 , piece_id=6WWNYQTH5MGAJOEB7BXKZDJOMCAZXDW66J3OEANGDNJDNUMLZM7Q, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558014689 (6h0m15.930594044s) [noise=true , address=64.49.13.138:4482 , piece_id=MAW2M4US2OQNOS3WI67AA7NIBTJDE5T2JQTXKFPBLGVGXKVCFGTQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558215974 (6h0m15.930795319s) [noise=true , address=77.90.31.10:4546 , piece_id=R6LCA5WZOEBDURL5S3MEK22YPVYXX6EXSS4J4W57543HELYXOKZA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558409364 (6h0m15.930988719s) [noise=true , address=45.91.95.5:4019 , piece_id=YTSBCN2LPHEMRRAZIHJGVB7NBOTRXVEHD2TWUCTTLBNFFYEWWDKA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558623504 (6h0m15.931202859s) [noise=true , address=79.127.201.215:4357 , piece_id=75XWAG2VQWEYCWN2Q3WRIZSBI7LDOISZJPXQLQ2GIPNEN34VLSUQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558805162 (6h0m15.931384517s) [noise=true , address=45.91.116.14:4219 , piece_id=PDW6JPRHCVMHHOWIHT6JKV4CTSC3JY6NDKDVFZSO47OYNPZ7PGGA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558999373 (6h0m15.931578728s) [noise=true , address=77.90.31.11:4564 , piece_id=L2U7JFJRDQOFA2AYVAGTI4ZW4VABCXKMXWBIUWQO2QNNUUBJ74BA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559184768 (6h0m15.931764123s) [noise=true , address=77.90.31.16:4768 , piece_id=YNLSONFM6COZEVF6ZNQR2EA6V4LSQFZJONIXR4AFTNNMFGPV57JA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559368719 (6h0m15.931948074s) [noise=true , address=79.127.220.98:4432 , piece_id=2SUTFQ7FVDPAM2H4W3OLW6AVH3VJ33IJFE4L52SA3ET5LGBMOMPQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559549706 (6h0m15.932129061s) [noise=true , address=77.90.31.18:4813 , piece_id=VDOT7BB4O7XYJHW6BFTAYQZZYLZU5PAUXVJGSXJ7FDI4TH425COA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559727836 (6h0m15.932307191s) [noise=true , address=79.127.219.51:4211 , piece_id=HO5PLWBVU5YAYNYMN352NA73SM4WBJV4WZ7XLSQTCQ76UAL2TOJA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:10.663359087 (6h0m17.035938442s) [upload=4769]: Next returned nil segment Jul 19 04:51:10.663378664 (6h0m17.035958019s) [upload=4769]: Waiting for error group managing segments... Jul 19 04:51:10.962945479 (6h0m17.335524834s) [noise=true , address=79.127.205.233:4622 , piece_id=6WWNYQTH5MGAJOEB7BXKZDJOMCAZXDW66J3OEANGDNJDNUMLZM7Q, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:10.967689807 (6h0m17.340269152s) [noise=true , address=79.127.205.235:4660 , piece_id=UOUOGE2VP7PX7EMRZLPMCWFCYEZERM3DQMCQ6NG4J2K2DEIZR63Q, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:10.976968631 (6h0m17.349547986s) [noise=true , address=64.49.13.138:4482 , piece_id=MAW2M4US2OQNOS3WI67AA7NIBTJDE5T2JQTXKFPBLGVGXKVCFGTQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.048750700 (6h0m17.421330055s) [noise=true , address=79.127.205.246:4280 , piece_id=FLBOZI3UGTQXAQBLZ2F45Z3OEYL65F5PQBXRF2DQMZ7NGF7VDRPA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.221642403 (6h0m17.594221758s) [noise=true , address=79.127.220.98:4432 , piece_id=2SUTFQ7FVDPAM2H4W3OLW6AVH3VJ33IJFE4L52SA3ET5LGBMOMPQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.317070435 (6h0m17.68964979s) [noise=true , address=45.91.95.4:4011 , piece_id=ZMBAS6KKHYVIRE6OBL2NFVCWMM4TKQ7J64BD3SYXIK54XCEDI2XA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.324002660 (6h0m17.696582015s) [noise=true , address=45.91.95.5:4019 , piece_id=YTSBCN2LPHEMRRAZIHJGVB7NBOTRXVEHD2TWUCTTLBNFFYEWWDKA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.460976317 (6h0m17.833555672s) [noise=true , address=79.127.201.215:4357 , piece_id=75XWAG2VQWEYCWN2Q3WRIZSBI7LDOISZJPXQLQ2GIPNEN34VLSUQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.462584603 (6h0m17.835163958s) [noise=true , address=45.91.116.6:4007 , piece_id=A2AFVTZBXSGBOE7XIM3B5HRN4STITFBXO6MZRJ365URNAAOZ3PSQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.490392973 (6h0m17.862972328s) [noise=true , address=45.91.116.9:4264 , piece_id=GHM5BKZKMT2QOHHBOS4FSPOWEWSFQRGRVVR4GSNDPHVIFRZHQL2A, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.502040780 (6h0m17.874620135s) [noise=true , address=45.91.116.14:4219 , piece_id=PDW6JPRHCVMHHOWIHT6JKV4CTSC3JY6NDKDVFZSO47OYNPZ7PGGA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.600434096 (6h0m17.973013451s) [noise=true , address=79.127.219.51:4211 , piece_id=HO5PLWBVU5YAYNYMN352NA73SM4WBJV4WZ7XLSQTCQ76UAL2TOJA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.609008984 (6h0m17.981588339s) [noise=true , address=104.36.82.19:4423 , piece_id=PIJUHDUND36HQL67I33D7A3EULMJHS7N55MN4OFQJ4OWNQ42O47A, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.620825273 (6h0m17.993404628s) [noise=true , address=79.127.219.38:4023 , piece_id=PTPOZGLYV4MZTOFZNN3NCGYL7DFAST4PXKEUIVTMFL7HV4GZTCLQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.620838659 (6h0m17.993418014s) [seg_pos={0 0}, upload=4769]: Segment reached optimal threshold of 14 pieces. Jul 19 04:51:11.621067807 (6h0m17.993647163s) [noise=true , address=77.90.31.11:4564 , piece_id=L2U7JFJRDQOFA2AYVAGTI4ZW4VABCXKMXWBIUWQO2QNNUUBJ74BA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection; write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection Jul 19 04:51:11.621134234 (6h0m17.993713589s) [noise=true , address=77.90.31.10:4546 , piece_id=R6LCA5WZOEBDURL5S3MEK22YPVYXX6EXSS4J4W57543HELYXOKZA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK): protocol: read tcp 188.166.225.170:58476->77.90.31.10:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621174191 (6h0m17.993753546s) [noise=true , address=77.90.31.16:4768 , piece_id=YNLSONFM6COZEVF6ZNQR2EA6V4LSQFZJONIXR4AFTNNMFGPV57JA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UchewmtxBrtSZSpVAJFdZmcTsQVQxkjK9re6cgUGqDhyonQeF): protocol: read tcp 188.166.225.170:56326->77.90.31.16:4768: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621216842 (6h0m17.993796197s) [noise=true , address=77.90.31.18:4813 , piece_id=VDOT7BB4O7XYJHW6BFTAYQZZYLZU5PAUXVJGSXJ7FDI4TH425COA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128WthA1x9wp1dV1vkpGnoveZQpU9CSgDCZfw5GMQuVpCNSVAM5): protocol: read tcp 188.166.225.170:49704->77.90.31.18:4813: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621234917 (6h0m17.993814272s) [seg_pos={0 0}, upload=4769]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 19 04:51:11.621301444 (6h0m17.993880799s) [upload=4769]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 19 04:51:11.621305402 (6h0m17.993884758s) [upload=4769]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 19 04:51:11.923136970 (6h0m18.295716325s) [upload=4769]: Done uploading segments... Jul 19 04:51:11.923156627 (6h0m18.295735972s) [upload=4769]: Upload finished. err: <nil> Duration: 17.044737074s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..............................................XX...X...X....] duration: 5.885405281s FAIL (2025-07-19 04:54:52 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 3) ======LOGS====== rue , address=45.91.116.11:4119 , piece_id=KTLVVJP2QCMK4KRWZ5SE2KD5GUCA55Z4LSEJ5K4DQ7XPVWKEVKAQ, seg_pos={0 0}, upload=4776]: Uploading piece... Jul 19 04:51:37.812867503 (6h0m44.185446858s) [noise=true , address=45.91.116.5:4299 , piece_id=WMPIHWASRJ4B74WDUWAOTMEKO3CFOY4BUVC43AX5XVM47PCG6UDQ, seg_pos={0 0}, upload=4776]: Uploading piece... Jul 19 04:51:37.813060833 (6h0m44.185640188s) [noise=true , address=77.90.31.17:4772 , piece_id=HJY5L2HO3OG3M5TDOAHW6MWPEGMJ4NJXEYGIORMV6D5Q4R5POHLQ, seg_pos={0 0}, upload=4776]: Uploading piece... Jul 19 04:51:37.813337573 (6h0m44.185916929s) [noise=true , address=45.91.116.6:4973 , piece_id=2SL6JAYP4LTT5RKTUM34T4T2QG5JX3YVTIH3LLBCRJD2MJRHUJZA, seg_pos={0 0}, upload=4776]: Uploading piece... Jul 19 04:51:37.813378882 (6h0m44.185958237s) [noise=true , address=77.90.31.11:4553 , piece_id=3KUTEWRXBHVGFCA6AXRRR3FTJYQEE7QOW6UFVUSUFMJ2L6527GXQ, seg_pos={0 0}, upload=4776]: Uploading piece... Jul 19 04:51:37.813633609 (6h0m44.186212964s) [noise=true , address=45.91.116.12:4225 , piece_id=GLJL3AAISDRWMWX5ULJCKAEGACS5YHWNMZNTEYMHCTCWR6TLEEEQ, seg_pos={0 0}, upload=4776]: Uploading piece... Jul 19 04:51:37.813830185 (6h0m44.18640954s) [noise=true , address=45.91.116.6:4974 , piece_id=VE6BT2CDEE36HVWQYUBLEITHUHI5IR223PE5XP5FI7YAYPO7MZKQ, seg_pos={0 0}, upload=4776]: Uploading piece... Jul 19 04:51:37.813920779 (6h0m44.186500124s) [noise=true , address=104.36.82.5:4839 , piece_id=HE7RLNECHYYDVXFIHKMWF32OUWM2TIV5OTFFX3WEZDAFBWYHU4TQ, seg_pos={0 0}, upload=4776]: Uploading piece... Jul 19 04:51:37.814123376 (6h0m44.186702731s) [noise=true , address=45.91.116.8:4045 , piece_id=Y2LNSNTKMH2QAMO4J6AQWQW7TWXC4QAW7H2VATFT2ENXYNCIGYMQ, seg_pos={0 0}, upload=4776]: Uploading piece... Jul 19 04:51:37.814174234 (6h0m44.186753589s) [noise=true , address=45.91.116.4:4938 , piece_id=TRHEJKBA3OWNONXLKBFFMGPCQFCWTB2XBE7WSQOITUPGMC7HG5IQ, seg_pos={0 0}, upload=4776]: Uploading piece... Jul 19 04:51:37.814414464 (6h0m44.186993819s) [noise=true , address=45.91.116.9:4262 , piece_id=KMPD4V4P4JAR4ZGADXMGQVR4Y7K67XC4DEFN5XN27XCVL3HQOHNQ, seg_pos={0 0}, upload=4776]: Uploading piece... Jul 19 04:51:37.814426847 (6h0m44.187006202s) [noise=true , address=104.36.82.7:4911 , piece_id=BXSTWCNP6O2R3OIBHMEPZW4DMGYIUP4KGJXVQWWQ3HBXWREE7KHA, seg_pos={0 0}, upload=4776]: Uploading piece... Jul 19 04:51:37.814640716 (6h0m44.187220071s) [noise=true , address=104.36.82.11:4113 , piece_id=J47Y3LBFN7N5RENSKXBBE64HEHJLZS4U273HJXCZBABCZLYRU4OQ, seg_pos={0 0}, upload=4776]: Uploading piece... Jul 19 04:51:37.814715981 (6h0m44.187295336s) [noise=true , address=45.91.116.10:4082 , piece_id=SCUONBPQGCDONRQFZCUJ256C7AKWUP43TTYYFBTCVS6SHHEIKWKA, seg_pos={0 0}, upload=4776]: Uploading piece... Jul 19 04:51:37.814875566 (6h0m44.187454911s) [noise=true , address=77.90.31.10:4520 , piece_id=HWHBZ3ZIRQNLW7IPML5O3CJOHHZWCZQWWYFRKKGPFEKA2PYMOONA, seg_pos={0 0}, upload=4776]: Uploading piece... Jul 19 04:51:39.567334189 (6h0m45.939913544s) [upload=4776]: Next returned nil segment Jul 19 04:51:39.567352354 (6h0m45.939931709s) [upload=4776]: Waiting for error group managing segments... Jul 19 04:51:39.882055541 (6h0m46.254634886s) [noise=true , address=104.36.82.7:4911 , piece_id=BXSTWCNP6O2R3OIBHMEPZW4DMGYIUP4KGJXVQWWQ3HBXWREE7KHA, seg_pos={0 0}, upload=4776]: Done uploading piece. err: <nil> Jul 19 04:51:39.950840033 (6h0m46.323419388s) [noise=true , address=104.36.82.11:4113 , piece_id=J47Y3LBFN7N5RENSKXBBE64HEHJLZS4U273HJXCZBABCZLYRU4OQ, seg_pos={0 0}, upload=4776]: Done uploading piece. err: <nil> Jul 19 04:51:40.030269575 (6h0m46.40284894s) [noise=true , address=104.36.82.5:4839 , piece_id=HE7RLNECHYYDVXFIHKMWF32OUWM2TIV5OTFFX3WEZDAFBWYHU4TQ, seg_pos={0 0}, upload=4776]: Done uploading piece. err: <nil> Jul 19 04:52:06.886171942 (6h1m13.258751347s) [noise=true , address=77.90.31.11:4553 , piece_id=3KUTEWRXBHVGFCA6AXRRR3FTJYQEE7QOW6UFVUSUFMJ2L6527GXQ, seg_pos={0 0}, upload=4776]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: rpc: dial tcp 77.90.31.11:4553: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:52:06.886226055 (6h1m13.25880541s) [noise=true , address=45.91.116.10:4082 , piece_id=SCUONBPQGCDONRQFZCUJ256C7AKWUP43TTYYFBTCVS6SHHEIKWKA, seg_pos={0 0}, upload=4776]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:52:06.886263507 (6h1m13.258842852s) [noise=true , address=77.90.31.10:4520 , piece_id=HWHBZ3ZIRQNLW7IPML5O3CJOHHZWCZQWWYFRKKGPFEKA2PYMOONA, seg_pos={0 0}, upload=4776]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: rpc: dial tcp 77.90.31.10:4520: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:52:06.886300778 (6h1m13.258880143s) [noise=true , address=45.91.116.12:4225 , piece_id=GLJL3AAISDRWMWX5ULJCKAEGACS5YHWNMZNTEYMHCTCWR6TLEEEQ, seg_pos={0 0}, upload=4776]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: rpc: dial tcp 45.91.116.12:4225: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:52:06.886328180 (6h1m13.258907585s) [noise=true , address=45.91.116.11:4119 , piece_id=KTLVVJP2QCMK4KRWZ5SE2KD5GUCA55Z4LSEJ5K4DQ7XPVWKEVKAQ, seg_pos={0 0}, upload=4776]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: rpc: dial tcp 45.91.116.11:4119: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:52:06.886392123 (6h1m13.258971478s) [noise=true , address=45.91.116.6:4974 , piece_id=VE6BT2CDEE36HVWQYUBLEITHUHI5IR223PE5XP5FI7YAYPO7MZKQ, seg_pos={0 0}, upload=4776]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: write tcp 188.166.225.170:48642->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:52:06.886404346 (6h1m13.258983701s) [noise=true , address=77.90.31.17:4772 , piece_id=HJY5L2HO3OG3M5TDOAHW6MWPEGMJ4NJXEYGIORMV6D5Q4R5POHLQ, seg_pos={0 0}, upload=4776]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:49268->77.90.31.17:4772: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:52:06.886435656 (6h1m13.259015011s) [noise=true , address=45.91.116.4:4938 , piece_id=TRHEJKBA3OWNONXLKBFFMGPCQFCWTB2XBE7WSQOITUPGMC7HG5IQ, seg_pos={0 0}, upload=4776]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: rpc: dial tcp 45.91.116.4:4938: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:52:06.886453570 (6h1m13.259032925s) [noise=true , address=45.91.116.8:4045 , piece_id=Y2LNSNTKMH2QAMO4J6AQWQW7TWXC4QAW7H2VATFT2ENXYNCIGYMQ, seg_pos={0 0}, upload=4776]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: rpc: dial tcp 45.91.116.8:4045: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:52:06.886473689 (6h1m13.259053044s) [noise=true , address=77.90.31.18:4807 , piece_id=NICW3OGWT627257ZXPRSC3ASCUEWDSX6FKRMPKUKGO2OUVPUFZJA, seg_pos={0 0}, upload=4776]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:52:06.886487124 (6h1m13.259066469s) [noise=true , address=45.91.116.6:4973 , piece_id=2SL6JAYP4LTT5RKTUM34T4T2QG5JX3YVTIH3LLBCRJD2MJRHUJZA, seg_pos={0 0}, upload=4776]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: rpc: dial tcp 45.91.116.6:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:52:06.886505169 (6h1m13.259084514s) [noise=true , address=45.91.116.5:4299 , piece_id=WMPIHWASRJ4B74WDUWAOTMEKO3CFOY4BUVC43AX5XVM47PCG6UDQ, seg_pos={0 0}, upload=4776]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: rpc: dial tcp 45.91.116.5:4299: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:52:06.886518144 (6h1m13.259097499s) [noise=true , address=45.91.116.9:4262 , piece_id=KMPD4V4P4JAR4ZGADXMGQVR4Y7K67XC4DEFN5XN27XCVL3HQOHNQ, seg_pos={0 0}, upload=4776]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: write tcp 188.166.225.170:51384->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:52:06.886540957 (6h1m13.259120322s) [seg_pos={0 0}, upload=4776]: Done waiting for segment. successful: 3 optimal: 13 errs: <nil> Jul 19 04:52:07.115640319 (6h1m13.488219674s) [upload=4776]: Done uploading segments... Jul 19 04:52:07.115664435 (6h1m13.48824379s) [upload=4776]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 3) Duration: 5.885427464s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 5.683016207s FAIL (2025-07-19 04:54:35 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 4855s) [upload=4769]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 04:51:09.109709627 (6h0m15.482288982s) [upload=4769]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 04:51:09.556145753 (6h0m15.928725108s) [seg_pos={0 0}, upload=4769]: Begin upload segment... Jul 19 04:51:09.556194817 (6h0m15.928774172s) [seg_pos={0 0}, upload=4769]: Done begin upload segment. Jul 19 04:51:09.556223893 (6h0m15.928803248s) [noise=true , address=79.127.219.38:4023 , piece_id=PTPOZGLYV4MZTOFZNN3NCGYL7DFAST4PXKEUIVTMFL7HV4GZTCLQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556252197 (6h0m15.928831552s) [noise=true , address=45.91.95.4:4011 , piece_id=ZMBAS6KKHYVIRE6OBL2NFVCWMM4TKQ7J64BD3SYXIK54XCEDI2XA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556579383 (6h0m15.929158738s) [noise=true , address=45.91.116.6:4007 , piece_id=A2AFVTZBXSGBOE7XIM3B5HRN4STITFBXO6MZRJ365URNAAOZ3PSQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556854469 (6h0m15.929433824s) [noise=true , address=104.36.82.19:4423 , piece_id=PIJUHDUND36HQL67I33D7A3EULMJHS7N55MN4OFQJ4OWNQ42O47A, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557094107 (6h0m15.929673462s) [noise=true , address=45.91.116.9:4264 , piece_id=GHM5BKZKMT2QOHHBOS4FSPOWEWSFQRGRVVR4GSNDPHVIFRZHQL2A, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557336381 (6h0m15.929915736s) [noise=true , address=79.127.205.235:4660 , piece_id=UOUOGE2VP7PX7EMRZLPMCWFCYEZERM3DQMCQ6NG4J2K2DEIZR63Q, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557557885 (6h0m15.93013724s) [noise=true , address=79.127.205.246:4280 , piece_id=FLBOZI3UGTQXAQBLZ2F45Z3OEYL65F5PQBXRF2DQMZ7NGF7VDRPA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557762296 (6h0m15.930341651s) [noise=true , address=79.127.205.233:4622 , piece_id=6WWNYQTH5MGAJOEB7BXKZDJOMCAZXDW66J3OEANGDNJDNUMLZM7Q, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558014689 (6h0m15.930594044s) [noise=true , address=64.49.13.138:4482 , piece_id=MAW2M4US2OQNOS3WI67AA7NIBTJDE5T2JQTXKFPBLGVGXKVCFGTQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558215974 (6h0m15.930795319s) [noise=true , address=77.90.31.10:4546 , piece_id=R6LCA5WZOEBDURL5S3MEK22YPVYXX6EXSS4J4W57543HELYXOKZA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558409364 (6h0m15.930988719s) [noise=true , address=45.91.95.5:4019 , piece_id=YTSBCN2LPHEMRRAZIHJGVB7NBOTRXVEHD2TWUCTTLBNFFYEWWDKA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558623504 (6h0m15.931202859s) [noise=true , address=79.127.201.215:4357 , piece_id=75XWAG2VQWEYCWN2Q3WRIZSBI7LDOISZJPXQLQ2GIPNEN34VLSUQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558805162 (6h0m15.931384517s) [noise=true , address=45.91.116.14:4219 , piece_id=PDW6JPRHCVMHHOWIHT6JKV4CTSC3JY6NDKDVFZSO47OYNPZ7PGGA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558999373 (6h0m15.931578728s) [noise=true , address=77.90.31.11:4564 , piece_id=L2U7JFJRDQOFA2AYVAGTI4ZW4VABCXKMXWBIUWQO2QNNUUBJ74BA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559184768 (6h0m15.931764123s) [noise=true , address=77.90.31.16:4768 , piece_id=YNLSONFM6COZEVF6ZNQR2EA6V4LSQFZJONIXR4AFTNNMFGPV57JA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559368719 (6h0m15.931948074s) [noise=true , address=79.127.220.98:4432 , piece_id=2SUTFQ7FVDPAM2H4W3OLW6AVH3VJ33IJFE4L52SA3ET5LGBMOMPQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559549706 (6h0m15.932129061s) [noise=true , address=77.90.31.18:4813 , piece_id=VDOT7BB4O7XYJHW6BFTAYQZZYLZU5PAUXVJGSXJ7FDI4TH425COA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559727836 (6h0m15.932307191s) [noise=true , address=79.127.219.51:4211 , piece_id=HO5PLWBVU5YAYNYMN352NA73SM4WBJV4WZ7XLSQTCQ76UAL2TOJA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:10.663359087 (6h0m17.035938442s) [upload=4769]: Next returned nil segment Jul 19 04:51:10.663378664 (6h0m17.035958019s) [upload=4769]: Waiting for error group managing segments... Jul 19 04:51:10.962945479 (6h0m17.335524834s) [noise=true , address=79.127.205.233:4622 , piece_id=6WWNYQTH5MGAJOEB7BXKZDJOMCAZXDW66J3OEANGDNJDNUMLZM7Q, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:10.967689807 (6h0m17.340269152s) [noise=true , address=79.127.205.235:4660 , piece_id=UOUOGE2VP7PX7EMRZLPMCWFCYEZERM3DQMCQ6NG4J2K2DEIZR63Q, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:10.976968631 (6h0m17.349547986s) [noise=true , address=64.49.13.138:4482 , piece_id=MAW2M4US2OQNOS3WI67AA7NIBTJDE5T2JQTXKFPBLGVGXKVCFGTQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.048750700 (6h0m17.421330055s) [noise=true , address=79.127.205.246:4280 , piece_id=FLBOZI3UGTQXAQBLZ2F45Z3OEYL65F5PQBXRF2DQMZ7NGF7VDRPA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.221642403 (6h0m17.594221758s) [noise=true , address=79.127.220.98:4432 , piece_id=2SUTFQ7FVDPAM2H4W3OLW6AVH3VJ33IJFE4L52SA3ET5LGBMOMPQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.317070435 (6h0m17.68964979s) [noise=true , address=45.91.95.4:4011 , piece_id=ZMBAS6KKHYVIRE6OBL2NFVCWMM4TKQ7J64BD3SYXIK54XCEDI2XA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.324002660 (6h0m17.696582015s) [noise=true , address=45.91.95.5:4019 , piece_id=YTSBCN2LPHEMRRAZIHJGVB7NBOTRXVEHD2TWUCTTLBNFFYEWWDKA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.460976317 (6h0m17.833555672s) [noise=true , address=79.127.201.215:4357 , piece_id=75XWAG2VQWEYCWN2Q3WRIZSBI7LDOISZJPXQLQ2GIPNEN34VLSUQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.462584603 (6h0m17.835163958s) [noise=true , address=45.91.116.6:4007 , piece_id=A2AFVTZBXSGBOE7XIM3B5HRN4STITFBXO6MZRJ365URNAAOZ3PSQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.490392973 (6h0m17.862972328s) [noise=true , address=45.91.116.9:4264 , piece_id=GHM5BKZKMT2QOHHBOS4FSPOWEWSFQRGRVVR4GSNDPHVIFRZHQL2A, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.502040780 (6h0m17.874620135s) [noise=true , address=45.91.116.14:4219 , piece_id=PDW6JPRHCVMHHOWIHT6JKV4CTSC3JY6NDKDVFZSO47OYNPZ7PGGA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.600434096 (6h0m17.973013451s) [noise=true , address=79.127.219.51:4211 , piece_id=HO5PLWBVU5YAYNYMN352NA73SM4WBJV4WZ7XLSQTCQ76UAL2TOJA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.609008984 (6h0m17.981588339s) [noise=true , address=104.36.82.19:4423 , piece_id=PIJUHDUND36HQL67I33D7A3EULMJHS7N55MN4OFQJ4OWNQ42O47A, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.620825273 (6h0m17.993404628s) [noise=true , address=79.127.219.38:4023 , piece_id=PTPOZGLYV4MZTOFZNN3NCGYL7DFAST4PXKEUIVTMFL7HV4GZTCLQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.620838659 (6h0m17.993418014s) [seg_pos={0 0}, upload=4769]: Segment reached optimal threshold of 14 pieces. Jul 19 04:51:11.621067807 (6h0m17.993647163s) [noise=true , address=77.90.31.11:4564 , piece_id=L2U7JFJRDQOFA2AYVAGTI4ZW4VABCXKMXWBIUWQO2QNNUUBJ74BA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection; write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection Jul 19 04:51:11.621134234 (6h0m17.993713589s) [noise=true , address=77.90.31.10:4546 , piece_id=R6LCA5WZOEBDURL5S3MEK22YPVYXX6EXSS4J4W57543HELYXOKZA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK): protocol: read tcp 188.166.225.170:58476->77.90.31.10:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621174191 (6h0m17.993753546s) [noise=true , address=77.90.31.16:4768 , piece_id=YNLSONFM6COZEVF6ZNQR2EA6V4LSQFZJONIXR4AFTNNMFGPV57JA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UchewmtxBrtSZSpVAJFdZmcTsQVQxkjK9re6cgUGqDhyonQeF): protocol: read tcp 188.166.225.170:56326->77.90.31.16:4768: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621216842 (6h0m17.993796197s) [noise=true , address=77.90.31.18:4813 , piece_id=VDOT7BB4O7XYJHW6BFTAYQZZYLZU5PAUXVJGSXJ7FDI4TH425COA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128WthA1x9wp1dV1vkpGnoveZQpU9CSgDCZfw5GMQuVpCNSVAM5): protocol: read tcp 188.166.225.170:49704->77.90.31.18:4813: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621234917 (6h0m17.993814272s) [seg_pos={0 0}, upload=4769]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 19 04:51:11.621301444 (6h0m17.993880799s) [upload=4769]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 19 04:51:11.621305402 (6h0m17.993884758s) [upload=4769]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 19 04:51:11.923136970 (6h0m18.295716325s) [upload=4769]: Done uploading segments... Jul 19 04:51:11.923156627 (6h0m18.295735972s) [upload=4769]: Upload finished. err: <nil> Duration: 5.683040203s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...............................................XX...X...X...] duration: 5.994727359s FAIL (2025-07-19 04:54:14 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 3) ======LOGS====== rue , address=45.91.116.11:4119 , piece_id=KTLVVJP2QCMK4KRWZ5SE2KD5GUCA55Z4LSEJ5K4DQ7XPVWKEVKAQ, seg_pos={0 0}, upload=4776]: Uploading piece... Jul 19 04:51:37.812867503 (6h0m44.185446858s) [noise=true , address=45.91.116.5:4299 , piece_id=WMPIHWASRJ4B74WDUWAOTMEKO3CFOY4BUVC43AX5XVM47PCG6UDQ, seg_pos={0 0}, upload=4776]: Uploading piece... Jul 19 04:51:37.813060833 (6h0m44.185640188s) [noise=true , address=77.90.31.17:4772 , piece_id=HJY5L2HO3OG3M5TDOAHW6MWPEGMJ4NJXEYGIORMV6D5Q4R5POHLQ, seg_pos={0 0}, upload=4776]: Uploading piece... Jul 19 04:51:37.813337573 (6h0m44.185916929s) [noise=true , address=45.91.116.6:4973 , piece_id=2SL6JAYP4LTT5RKTUM34T4T2QG5JX3YVTIH3LLBCRJD2MJRHUJZA, seg_pos={0 0}, upload=4776]: Uploading piece... Jul 19 04:51:37.813378882 (6h0m44.185958237s) [noise=true , address=77.90.31.11:4553 , piece_id=3KUTEWRXBHVGFCA6AXRRR3FTJYQEE7QOW6UFVUSUFMJ2L6527GXQ, seg_pos={0 0}, upload=4776]: Uploading piece... Jul 19 04:51:37.813633609 (6h0m44.186212964s) [noise=true , address=45.91.116.12:4225 , piece_id=GLJL3AAISDRWMWX5ULJCKAEGACS5YHWNMZNTEYMHCTCWR6TLEEEQ, seg_pos={0 0}, upload=4776]: Uploading piece... Jul 19 04:51:37.813830185 (6h0m44.18640954s) [noise=true , address=45.91.116.6:4974 , piece_id=VE6BT2CDEE36HVWQYUBLEITHUHI5IR223PE5XP5FI7YAYPO7MZKQ, seg_pos={0 0}, upload=4776]: Uploading piece... Jul 19 04:51:37.813920779 (6h0m44.186500124s) [noise=true , address=104.36.82.5:4839 , piece_id=HE7RLNECHYYDVXFIHKMWF32OUWM2TIV5OTFFX3WEZDAFBWYHU4TQ, seg_pos={0 0}, upload=4776]: Uploading piece... Jul 19 04:51:37.814123376 (6h0m44.186702731s) [noise=true , address=45.91.116.8:4045 , piece_id=Y2LNSNTKMH2QAMO4J6AQWQW7TWXC4QAW7H2VATFT2ENXYNCIGYMQ, seg_pos={0 0}, upload=4776]: Uploading piece... Jul 19 04:51:37.814174234 (6h0m44.186753589s) [noise=true , address=45.91.116.4:4938 , piece_id=TRHEJKBA3OWNONXLKBFFMGPCQFCWTB2XBE7WSQOITUPGMC7HG5IQ, seg_pos={0 0}, upload=4776]: Uploading piece... Jul 19 04:51:37.814414464 (6h0m44.186993819s) [noise=true , address=45.91.116.9:4262 , piece_id=KMPD4V4P4JAR4ZGADXMGQVR4Y7K67XC4DEFN5XN27XCVL3HQOHNQ, seg_pos={0 0}, upload=4776]: Uploading piece... Jul 19 04:51:37.814426847 (6h0m44.187006202s) [noise=true , address=104.36.82.7:4911 , piece_id=BXSTWCNP6O2R3OIBHMEPZW4DMGYIUP4KGJXVQWWQ3HBXWREE7KHA, seg_pos={0 0}, upload=4776]: Uploading piece... Jul 19 04:51:37.814640716 (6h0m44.187220071s) [noise=true , address=104.36.82.11:4113 , piece_id=J47Y3LBFN7N5RENSKXBBE64HEHJLZS4U273HJXCZBABCZLYRU4OQ, seg_pos={0 0}, upload=4776]: Uploading piece... Jul 19 04:51:37.814715981 (6h0m44.187295336s) [noise=true , address=45.91.116.10:4082 , piece_id=SCUONBPQGCDONRQFZCUJ256C7AKWUP43TTYYFBTCVS6SHHEIKWKA, seg_pos={0 0}, upload=4776]: Uploading piece... Jul 19 04:51:37.814875566 (6h0m44.187454911s) [noise=true , address=77.90.31.10:4520 , piece_id=HWHBZ3ZIRQNLW7IPML5O3CJOHHZWCZQWWYFRKKGPFEKA2PYMOONA, seg_pos={0 0}, upload=4776]: Uploading piece... Jul 19 04:51:39.567334189 (6h0m45.939913544s) [upload=4776]: Next returned nil segment Jul 19 04:51:39.567352354 (6h0m45.939931709s) [upload=4776]: Waiting for error group managing segments... Jul 19 04:51:39.882055541 (6h0m46.254634886s) [noise=true , address=104.36.82.7:4911 , piece_id=BXSTWCNP6O2R3OIBHMEPZW4DMGYIUP4KGJXVQWWQ3HBXWREE7KHA, seg_pos={0 0}, upload=4776]: Done uploading piece. err: <nil> Jul 19 04:51:39.950840033 (6h0m46.323419388s) [noise=true , address=104.36.82.11:4113 , piece_id=J47Y3LBFN7N5RENSKXBBE64HEHJLZS4U273HJXCZBABCZLYRU4OQ, seg_pos={0 0}, upload=4776]: Done uploading piece. err: <nil> Jul 19 04:51:40.030269575 (6h0m46.40284894s) [noise=true , address=104.36.82.5:4839 , piece_id=HE7RLNECHYYDVXFIHKMWF32OUWM2TIV5OTFFX3WEZDAFBWYHU4TQ, seg_pos={0 0}, upload=4776]: Done uploading piece. err: <nil> Jul 19 04:52:06.886171942 (6h1m13.258751347s) [noise=true , address=77.90.31.11:4553 , piece_id=3KUTEWRXBHVGFCA6AXRRR3FTJYQEE7QOW6UFVUSUFMJ2L6527GXQ, seg_pos={0 0}, upload=4776]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: rpc: dial tcp 77.90.31.11:4553: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:52:06.886226055 (6h1m13.25880541s) [noise=true , address=45.91.116.10:4082 , piece_id=SCUONBPQGCDONRQFZCUJ256C7AKWUP43TTYYFBTCVS6SHHEIKWKA, seg_pos={0 0}, upload=4776]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:52:06.886263507 (6h1m13.258842852s) [noise=true , address=77.90.31.10:4520 , piece_id=HWHBZ3ZIRQNLW7IPML5O3CJOHHZWCZQWWYFRKKGPFEKA2PYMOONA, seg_pos={0 0}, upload=4776]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: rpc: dial tcp 77.90.31.10:4520: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:52:06.886300778 (6h1m13.258880143s) [noise=true , address=45.91.116.12:4225 , piece_id=GLJL3AAISDRWMWX5ULJCKAEGACS5YHWNMZNTEYMHCTCWR6TLEEEQ, seg_pos={0 0}, upload=4776]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: rpc: dial tcp 45.91.116.12:4225: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:52:06.886328180 (6h1m13.258907585s) [noise=true , address=45.91.116.11:4119 , piece_id=KTLVVJP2QCMK4KRWZ5SE2KD5GUCA55Z4LSEJ5K4DQ7XPVWKEVKAQ, seg_pos={0 0}, upload=4776]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: rpc: dial tcp 45.91.116.11:4119: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:52:06.886392123 (6h1m13.258971478s) [noise=true , address=45.91.116.6:4974 , piece_id=VE6BT2CDEE36HVWQYUBLEITHUHI5IR223PE5XP5FI7YAYPO7MZKQ, seg_pos={0 0}, upload=4776]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: write tcp 188.166.225.170:48642->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:52:06.886404346 (6h1m13.258983701s) [noise=true , address=77.90.31.17:4772 , piece_id=HJY5L2HO3OG3M5TDOAHW6MWPEGMJ4NJXEYGIORMV6D5Q4R5POHLQ, seg_pos={0 0}, upload=4776]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:49268->77.90.31.17:4772: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:52:06.886435656 (6h1m13.259015011s) [noise=true , address=45.91.116.4:4938 , piece_id=TRHEJKBA3OWNONXLKBFFMGPCQFCWTB2XBE7WSQOITUPGMC7HG5IQ, seg_pos={0 0}, upload=4776]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: rpc: dial tcp 45.91.116.4:4938: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:52:06.886453570 (6h1m13.259032925s) [noise=true , address=45.91.116.8:4045 , piece_id=Y2LNSNTKMH2QAMO4J6AQWQW7TWXC4QAW7H2VATFT2ENXYNCIGYMQ, seg_pos={0 0}, upload=4776]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: rpc: dial tcp 45.91.116.8:4045: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:52:06.886473689 (6h1m13.259053044s) [noise=true , address=77.90.31.18:4807 , piece_id=NICW3OGWT627257ZXPRSC3ASCUEWDSX6FKRMPKUKGO2OUVPUFZJA, seg_pos={0 0}, upload=4776]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:52:06.886487124 (6h1m13.259066469s) [noise=true , address=45.91.116.6:4973 , piece_id=2SL6JAYP4LTT5RKTUM34T4T2QG5JX3YVTIH3LLBCRJD2MJRHUJZA, seg_pos={0 0}, upload=4776]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: rpc: dial tcp 45.91.116.6:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:52:06.886505169 (6h1m13.259084514s) [noise=true , address=45.91.116.5:4299 , piece_id=WMPIHWASRJ4B74WDUWAOTMEKO3CFOY4BUVC43AX5XVM47PCG6UDQ, seg_pos={0 0}, upload=4776]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: rpc: dial tcp 45.91.116.5:4299: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:52:06.886518144 (6h1m13.259097499s) [noise=true , address=45.91.116.9:4262 , piece_id=KMPD4V4P4JAR4ZGADXMGQVR4Y7K67XC4DEFN5XN27XCVL3HQOHNQ, seg_pos={0 0}, upload=4776]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: write tcp 188.166.225.170:51384->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:52:06.886540957 (6h1m13.259120322s) [seg_pos={0 0}, upload=4776]: Done waiting for segment. successful: 3 optimal: 13 errs: <nil> Jul 19 04:52:07.115640319 (6h1m13.488219674s) [upload=4776]: Done uploading segments... Jul 19 04:52:07.115664435 (6h1m13.48824379s) [upload=4776]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 3) Duration: 5.99474921s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 5.521845272s FAIL (2025-07-19 04:53:32 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 4855s) [upload=4769]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 19 04:51:09.109709627 (6h0m15.482288982s) [upload=4769]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 19 04:51:09.556145753 (6h0m15.928725108s) [seg_pos={0 0}, upload=4769]: Begin upload segment... Jul 19 04:51:09.556194817 (6h0m15.928774172s) [seg_pos={0 0}, upload=4769]: Done begin upload segment. Jul 19 04:51:09.556223893 (6h0m15.928803248s) [noise=true , address=79.127.219.38:4023 , piece_id=PTPOZGLYV4MZTOFZNN3NCGYL7DFAST4PXKEUIVTMFL7HV4GZTCLQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556252197 (6h0m15.928831552s) [noise=true , address=45.91.95.4:4011 , piece_id=ZMBAS6KKHYVIRE6OBL2NFVCWMM4TKQ7J64BD3SYXIK54XCEDI2XA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556579383 (6h0m15.929158738s) [noise=true , address=45.91.116.6:4007 , piece_id=A2AFVTZBXSGBOE7XIM3B5HRN4STITFBXO6MZRJ365URNAAOZ3PSQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.556854469 (6h0m15.929433824s) [noise=true , address=104.36.82.19:4423 , piece_id=PIJUHDUND36HQL67I33D7A3EULMJHS7N55MN4OFQJ4OWNQ42O47A, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557094107 (6h0m15.929673462s) [noise=true , address=45.91.116.9:4264 , piece_id=GHM5BKZKMT2QOHHBOS4FSPOWEWSFQRGRVVR4GSNDPHVIFRZHQL2A, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557336381 (6h0m15.929915736s) [noise=true , address=79.127.205.235:4660 , piece_id=UOUOGE2VP7PX7EMRZLPMCWFCYEZERM3DQMCQ6NG4J2K2DEIZR63Q, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557557885 (6h0m15.93013724s) [noise=true , address=79.127.205.246:4280 , piece_id=FLBOZI3UGTQXAQBLZ2F45Z3OEYL65F5PQBXRF2DQMZ7NGF7VDRPA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.557762296 (6h0m15.930341651s) [noise=true , address=79.127.205.233:4622 , piece_id=6WWNYQTH5MGAJOEB7BXKZDJOMCAZXDW66J3OEANGDNJDNUMLZM7Q, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558014689 (6h0m15.930594044s) [noise=true , address=64.49.13.138:4482 , piece_id=MAW2M4US2OQNOS3WI67AA7NIBTJDE5T2JQTXKFPBLGVGXKVCFGTQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558215974 (6h0m15.930795319s) [noise=true , address=77.90.31.10:4546 , piece_id=R6LCA5WZOEBDURL5S3MEK22YPVYXX6EXSS4J4W57543HELYXOKZA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558409364 (6h0m15.930988719s) [noise=true , address=45.91.95.5:4019 , piece_id=YTSBCN2LPHEMRRAZIHJGVB7NBOTRXVEHD2TWUCTTLBNFFYEWWDKA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558623504 (6h0m15.931202859s) [noise=true , address=79.127.201.215:4357 , piece_id=75XWAG2VQWEYCWN2Q3WRIZSBI7LDOISZJPXQLQ2GIPNEN34VLSUQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558805162 (6h0m15.931384517s) [noise=true , address=45.91.116.14:4219 , piece_id=PDW6JPRHCVMHHOWIHT6JKV4CTSC3JY6NDKDVFZSO47OYNPZ7PGGA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.558999373 (6h0m15.931578728s) [noise=true , address=77.90.31.11:4564 , piece_id=L2U7JFJRDQOFA2AYVAGTI4ZW4VABCXKMXWBIUWQO2QNNUUBJ74BA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559184768 (6h0m15.931764123s) [noise=true , address=77.90.31.16:4768 , piece_id=YNLSONFM6COZEVF6ZNQR2EA6V4LSQFZJONIXR4AFTNNMFGPV57JA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559368719 (6h0m15.931948074s) [noise=true , address=79.127.220.98:4432 , piece_id=2SUTFQ7FVDPAM2H4W3OLW6AVH3VJ33IJFE4L52SA3ET5LGBMOMPQ, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559549706 (6h0m15.932129061s) [noise=true , address=77.90.31.18:4813 , piece_id=VDOT7BB4O7XYJHW6BFTAYQZZYLZU5PAUXVJGSXJ7FDI4TH425COA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:09.559727836 (6h0m15.932307191s) [noise=true , address=79.127.219.51:4211 , piece_id=HO5PLWBVU5YAYNYMN352NA73SM4WBJV4WZ7XLSQTCQ76UAL2TOJA, seg_pos={0 0}, upload=4769]: Uploading piece... Jul 19 04:51:10.663359087 (6h0m17.035938442s) [upload=4769]: Next returned nil segment Jul 19 04:51:10.663378664 (6h0m17.035958019s) [upload=4769]: Waiting for error group managing segments... Jul 19 04:51:10.962945479 (6h0m17.335524834s) [noise=true , address=79.127.205.233:4622 , piece_id=6WWNYQTH5MGAJOEB7BXKZDJOMCAZXDW66J3OEANGDNJDNUMLZM7Q, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:10.967689807 (6h0m17.340269152s) [noise=true , address=79.127.205.235:4660 , piece_id=UOUOGE2VP7PX7EMRZLPMCWFCYEZERM3DQMCQ6NG4J2K2DEIZR63Q, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:10.976968631 (6h0m17.349547986s) [noise=true , address=64.49.13.138:4482 , piece_id=MAW2M4US2OQNOS3WI67AA7NIBTJDE5T2JQTXKFPBLGVGXKVCFGTQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.048750700 (6h0m17.421330055s) [noise=true , address=79.127.205.246:4280 , piece_id=FLBOZI3UGTQXAQBLZ2F45Z3OEYL65F5PQBXRF2DQMZ7NGF7VDRPA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.221642403 (6h0m17.594221758s) [noise=true , address=79.127.220.98:4432 , piece_id=2SUTFQ7FVDPAM2H4W3OLW6AVH3VJ33IJFE4L52SA3ET5LGBMOMPQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.317070435 (6h0m17.68964979s) [noise=true , address=45.91.95.4:4011 , piece_id=ZMBAS6KKHYVIRE6OBL2NFVCWMM4TKQ7J64BD3SYXIK54XCEDI2XA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.324002660 (6h0m17.696582015s) [noise=true , address=45.91.95.5:4019 , piece_id=YTSBCN2LPHEMRRAZIHJGVB7NBOTRXVEHD2TWUCTTLBNFFYEWWDKA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.460976317 (6h0m17.833555672s) [noise=true , address=79.127.201.215:4357 , piece_id=75XWAG2VQWEYCWN2Q3WRIZSBI7LDOISZJPXQLQ2GIPNEN34VLSUQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.462584603 (6h0m17.835163958s) [noise=true , address=45.91.116.6:4007 , piece_id=A2AFVTZBXSGBOE7XIM3B5HRN4STITFBXO6MZRJ365URNAAOZ3PSQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.490392973 (6h0m17.862972328s) [noise=true , address=45.91.116.9:4264 , piece_id=GHM5BKZKMT2QOHHBOS4FSPOWEWSFQRGRVVR4GSNDPHVIFRZHQL2A, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.502040780 (6h0m17.874620135s) [noise=true , address=45.91.116.14:4219 , piece_id=PDW6JPRHCVMHHOWIHT6JKV4CTSC3JY6NDKDVFZSO47OYNPZ7PGGA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.600434096 (6h0m17.973013451s) [noise=true , address=79.127.219.51:4211 , piece_id=HO5PLWBVU5YAYNYMN352NA73SM4WBJV4WZ7XLSQTCQ76UAL2TOJA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.609008984 (6h0m17.981588339s) [noise=true , address=104.36.82.19:4423 , piece_id=PIJUHDUND36HQL67I33D7A3EULMJHS7N55MN4OFQJ4OWNQ42O47A, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.620825273 (6h0m17.993404628s) [noise=true , address=79.127.219.38:4023 , piece_id=PTPOZGLYV4MZTOFZNN3NCGYL7DFAST4PXKEUIVTMFL7HV4GZTCLQ, seg_pos={0 0}, upload=4769]: Done uploading piece. err: <nil> Jul 19 04:51:11.620838659 (6h0m17.993418014s) [seg_pos={0 0}, upload=4769]: Segment reached optimal threshold of 14 pieces. Jul 19 04:51:11.621067807 (6h0m17.993647163s) [noise=true , address=77.90.31.11:4564 , piece_id=L2U7JFJRDQOFA2AYVAGTI4ZW4VABCXKMXWBIUWQO2QNNUUBJ74BA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection; write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45814->77.90.31.11:4564: use of closed network connection Jul 19 04:51:11.621134234 (6h0m17.993713589s) [noise=true , address=77.90.31.10:4546 , piece_id=R6LCA5WZOEBDURL5S3MEK22YPVYXX6EXSS4J4W57543HELYXOKZA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK): protocol: read tcp 188.166.225.170:58476->77.90.31.10:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621174191 (6h0m17.993753546s) [noise=true , address=77.90.31.16:4768 , piece_id=YNLSONFM6COZEVF6ZNQR2EA6V4LSQFZJONIXR4AFTNNMFGPV57JA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UchewmtxBrtSZSpVAJFdZmcTsQVQxkjK9re6cgUGqDhyonQeF): protocol: read tcp 188.166.225.170:56326->77.90.31.16:4768: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621216842 (6h0m17.993796197s) [noise=true , address=77.90.31.18:4813 , piece_id=VDOT7BB4O7XYJHW6BFTAYQZZYLZU5PAUXVJGSXJ7FDI4TH425COA, seg_pos={0 0}, upload=4769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128WthA1x9wp1dV1vkpGnoveZQpU9CSgDCZfw5GMQuVpCNSVAM5): protocol: read tcp 188.166.225.170:49704->77.90.31.18:4813: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:51:11.621234917 (6h0m17.993814272s) [seg_pos={0 0}, upload=4769]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 19 04:51:11.621301444 (6h0m17.993880799s) [upload=4769]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 19 04:51:11.621305402 (6h0m17.993884758s) [upload=4769]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 19 04:51:11.923136970 (6h0m18.295716325s) [upload=4769]: Done uploading segments... Jul 19 04:51:11.923156627 (6h0m18.295735972s) [upload=4769]: Upload finished. err: <nil> Duration: 5.521869628s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [................................................XX...X...X..] duration: 5.731515236s FAIL (2025-07-19 04:53:17 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 3) ======LOGS====== rue , address=45.91.116.11:4119 , piece_id=KTLVVJP2QCMK4KRWZ5SE2KD5GUCA55Z4LSEJ5K4DQ7XPVWKEVKAQ, seg_pos={0 0}, upload=4776]: Uploading piece... Jul 19 04:51:37.812867503 (6h0m44.185446858s) [noise=true , address=45.91.116.5:4299 , piece_id=WMPIHWASRJ4B74WDUWAOTMEKO3CFOY4BUVC43AX5XVM47PCG6UDQ, seg_pos={0 0}, upload=4776]: Uploading piece... Jul 19 04:51:37.813060833 (6h0m44.185640188s) [noise=true , address=77.90.31.17:4772 , piece_id=HJY5L2HO3OG3M5TDOAHW6MWPEGMJ4NJXEYGIORMV6D5Q4R5POHLQ, seg_pos={0 0}, upload=4776]: Uploading piece... Jul 19 04:51:37.813337573 (6h0m44.185916929s) [noise=true , address=45.91.116.6:4973 , piece_id=2SL6JAYP4LTT5RKTUM34T4T2QG5JX3YVTIH3LLBCRJD2MJRHUJZA, seg_pos={0 0}, upload=4776]: Uploading piece... Jul 19 04:51:37.813378882 (6h0m44.185958237s) [noise=true , address=77.90.31.11:4553 , piece_id=3KUTEWRXBHVGFCA6AXRRR3FTJYQEE7QOW6UFVUSUFMJ2L6527GXQ, seg_pos={0 0}, upload=4776]: Uploading piece... Jul 19 04:51:37.813633609 (6h0m44.186212964s) [noise=true , address=45.91.116.12:4225 , piece_id=GLJL3AAISDRWMWX5ULJCKAEGACS5YHWNMZNTEYMHCTCWR6TLEEEQ, seg_pos={0 0}, upload=4776]: Uploading piece... Jul 19 04:51:37.813830185 (6h0m44.18640954s) [noise=true , address=45.91.116.6:4974 , piece_id=VE6BT2CDEE36HVWQYUBLEITHUHI5IR223PE5XP5FI7YAYPO7MZKQ, seg_pos={0 0}, upload=4776]: Uploading piece... Jul 19 04:51:37.813920779 (6h0m44.186500124s) [noise=true , address=104.36.82.5:4839 , piece_id=HE7RLNECHYYDVXFIHKMWF32OUWM2TIV5OTFFX3WEZDAFBWYHU4TQ, seg_pos={0 0}, upload=4776]: Uploading piece... Jul 19 04:51:37.814123376 (6h0m44.186702731s) [noise=true , address=45.91.116.8:4045 , piece_id=Y2LNSNTKMH2QAMO4J6AQWQW7TWXC4QAW7H2VATFT2ENXYNCIGYMQ, seg_pos={0 0}, upload=4776]: Uploading piece... Jul 19 04:51:37.814174234 (6h0m44.186753589s) [noise=true , address=45.91.116.4:4938 , piece_id=TRHEJKBA3OWNONXLKBFFMGPCQFCWTB2XBE7WSQOITUPGMC7HG5IQ, seg_pos={0 0}, upload=4776]: Uploading piece... Jul 19 04:51:37.814414464 (6h0m44.186993819s) [noise=true , address=45.91.116.9:4262 , piece_id=KMPD4V4P4JAR4ZGADXMGQVR4Y7K67XC4DEFN5XN27XCVL3HQOHNQ, seg_pos={0 0}, upload=4776]: Uploading piece... Jul 19 04:51:37.814426847 (6h0m44.187006202s) [noise=true , address=104.36.82.7:4911 , piece_id=BXSTWCNP6O2R3OIBHMEPZW4DMGYIUP4KGJXVQWWQ3HBXWREE7KHA, seg_pos={0 0}, upload=4776]: Uploading piece... Jul 19 04:51:37.814640716 (6h0m44.187220071s) [noise=true , address=104.36.82.11:4113 , piece_id=J47Y3LBFN7N5RENSKXBBE64HEHJLZS4U273HJXCZBABCZLYRU4OQ, seg_pos={0 0}, upload=4776]: Uploading piece... Jul 19 04:51:37.814715981 (6h0m44.187295336s) [noise=true , address=45.91.116.10:4082 , piece_id=SCUONBPQGCDONRQFZCUJ256C7AKWUP43TTYYFBTCVS6SHHEIKWKA, seg_pos={0 0}, upload=4776]: Uploading piece... Jul 19 04:51:37.814875566 (6h0m44.187454911s) [noise=true , address=77.90.31.10:4520 , piece_id=HWHBZ3ZIRQNLW7IPML5O3CJOHHZWCZQWWYFRKKGPFEKA2PYMOONA, seg_pos={0 0}, upload=4776]: Uploading piece... Jul 19 04:51:39.567334189 (6h0m45.939913544s) [upload=4776]: Next returned nil segment Jul 19 04:51:39.567352354 (6h0m45.939931709s) [upload=4776]: Waiting for error group managing segments... Jul 19 04:51:39.882055541 (6h0m46.254634886s) [noise=true , address=104.36.82.7:4911 , piece_id=BXSTWCNP6O2R3OIBHMEPZW4DMGYIUP4KGJXVQWWQ3HBXWREE7KHA, seg_pos={0 0}, upload=4776]: Done uploading piece. err: <nil> Jul 19 04:51:39.950840033 (6h0m46.323419388s) [noise=true , address=104.36.82.11:4113 , piece_id=J47Y3LBFN7N5RENSKXBBE64HEHJLZS4U273HJXCZBABCZLYRU4OQ, seg_pos={0 0}, upload=4776]: Done uploading piece. err: <nil> Jul 19 04:51:40.030269575 (6h0m46.40284894s) [noise=true , address=104.36.82.5:4839 , piece_id=HE7RLNECHYYDVXFIHKMWF32OUWM2TIV5OTFFX3WEZDAFBWYHU4TQ, seg_pos={0 0}, upload=4776]: Done uploading piece. err: <nil> Jul 19 04:52:06.886171942 (6h1m13.258751347s) [noise=true , address=77.90.31.11:4553 , piece_id=3KUTEWRXBHVGFCA6AXRRR3FTJYQEE7QOW6UFVUSUFMJ2L6527GXQ, seg_pos={0 0}, upload=4776]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: rpc: dial tcp 77.90.31.11:4553: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:52:06.886226055 (6h1m13.25880541s) [noise=true , address=45.91.116.10:4082 , piece_id=SCUONBPQGCDONRQFZCUJ256C7AKWUP43TTYYFBTCVS6SHHEIKWKA, seg_pos={0 0}, upload=4776]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:52:06.886263507 (6h1m13.258842852s) [noise=true , address=77.90.31.10:4520 , piece_id=HWHBZ3ZIRQNLW7IPML5O3CJOHHZWCZQWWYFRKKGPFEKA2PYMOONA, seg_pos={0 0}, upload=4776]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: rpc: dial tcp 77.90.31.10:4520: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:52:06.886300778 (6h1m13.258880143s) [noise=true , address=45.91.116.12:4225 , piece_id=GLJL3AAISDRWMWX5ULJCKAEGACS5YHWNMZNTEYMHCTCWR6TLEEEQ, seg_pos={0 0}, upload=4776]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: rpc: dial tcp 45.91.116.12:4225: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:52:06.886328180 (6h1m13.258907585s) [noise=true , address=45.91.116.11:4119 , piece_id=KTLVVJP2QCMK4KRWZ5SE2KD5GUCA55Z4LSEJ5K4DQ7XPVWKEVKAQ, seg_pos={0 0}, upload=4776]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: rpc: dial tcp 45.91.116.11:4119: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:52:06.886392123 (6h1m13.258971478s) [noise=true , address=45.91.116.6:4974 , piece_id=VE6BT2CDEE36HVWQYUBLEITHUHI5IR223PE5XP5FI7YAYPO7MZKQ, seg_pos={0 0}, upload=4776]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: write tcp 188.166.225.170:48642->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:52:06.886404346 (6h1m13.258983701s) [noise=true , address=77.90.31.17:4772 , piece_id=HJY5L2HO3OG3M5TDOAHW6MWPEGMJ4NJXEYGIORMV6D5Q4R5POHLQ, seg_pos={0 0}, upload=4776]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:49268->77.90.31.17:4772: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:52:06.886435656 (6h1m13.259015011s) [noise=true , address=45.91.116.4:4938 , piece_id=TRHEJKBA3OWNONXLKBFFMGPCQFCWTB2XBE7WSQOITUPGMC7HG5IQ, seg_pos={0 0}, upload=4776]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: rpc: dial tcp 45.91.116.4:4938: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:52:06.886453570 (6h1m13.259032925s) [noise=true , address=45.91.116.8:4045 , piece_id=Y2LNSNTKMH2QAMO4J6AQWQW7TWXC4QAW7H2VATFT2ENXYNCIGYMQ, seg_pos={0 0}, upload=4776]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: rpc: dial tcp 45.91.116.8:4045: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:52:06.886473689 (6h1m13.259053044s) [noise=true , address=77.90.31.18:4807 , piece_id=NICW3OGWT627257ZXPRSC3ASCUEWDSX6FKRMPKUKGO2OUVPUFZJA, seg_pos={0 0}, upload=4776]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:52:06.886487124 (6h1m13.259066469s) [noise=true , address=45.91.116.6:4973 , piece_id=2SL6JAYP4LTT5RKTUM34T4T2QG5JX3YVTIH3LLBCRJD2MJRHUJZA, seg_pos={0 0}, upload=4776]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: rpc: dial tcp 45.91.116.6:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:52:06.886505169 (6h1m13.259084514s) [noise=true , address=45.91.116.5:4299 , piece_id=WMPIHWASRJ4B74WDUWAOTMEKO3CFOY4BUVC43AX5XVM47PCG6UDQ, seg_pos={0 0}, upload=4776]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: rpc: dial tcp 45.91.116.5:4299: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:52:06.886518144 (6h1m13.259097499s) [noise=true , address=45.91.116.9:4262 , piece_id=KMPD4V4P4JAR4ZGADXMGQVR4Y7K67XC4DEFN5XN27XCVL3HQOHNQ, seg_pos={0 0}, upload=4776]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: write tcp 188.166.225.170:51384->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 19 04:52:06.886540957 (6h1m13.259120322s) [seg_pos={0 0}, upload=4776]: Done waiting for segment. successful: 3 optimal: 13 errs: <nil> Jul 19 04:52:07.115640319 (6h1m13.488219674s) [upload=4776]: Done uploading segments... Jul 19 04:52:07.115664435 (6h1m13.48824379s) [upload=4776]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 3) Duration: 5.731544912s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)