Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....................X...............................X...XX.] duration: 7.941067533s FAIL (2025-09-23 14:01:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 4:00:52.632202144 (29m38.094294617s) [upload=438]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 14:00:52.632207815 (29m38.094300298s) [upload=438]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 14:00:53.022181978 (29m38.484274461s) [seg_pos={0 0}, upload=438]: Begin upload segment... Sep 23 14:00:53.022253353 (29m38.484345836s) [seg_pos={0 0}, upload=438]: Done begin upload segment. Sep 23 14:00:53.022311474 (29m38.484403957s) [noise=true , address=145.79.227.2:4115 , piece_id=DOQ4K26LCLRMI7JV2OSXUWU6YN2JGZFB6MD4ZER73LLIF6V7LS2Q, seg_pos={0 0}, upload=438]: Uploading piece... Sep 23 14:00:53.022325401 (29m38.484417884s) [noise=true , address=145.79.227.4:4185 , piece_id=722XLL2R2MJ5VM7UH724XG4ECKXUL7LRP2J6EV3XP56OL6TYEMWA, seg_pos={0 0}, upload=438]: Uploading piece... Sep 23 14:00:53.022997209 (29m38.485089683s) [noise=true , address=45.91.116.14:4191 , piece_id=2ECVPRNT6D55LZIYNE3IEDJEKL5VUWRLLZRU3RDDSYG6KQZ2VLJQ, seg_pos={0 0}, upload=438]: Uploading piece... Sep 23 14:00:53.023174999 (29m38.485267472s) [noise=true , address=145.79.227.14:4840 , piece_id=2P5VSHWRYY5FADA7AAW7MAIRHLNUD7EFAQ3VIMN2FPHWIZSZTBZA, seg_pos={0 0}, upload=438]: Uploading piece... Sep 23 14:00:53.023561134 (29m38.485653617s) [noise=true , address=77.90.31.13:4626 , piece_id=YF2QO3MZZJ4NEFZU2W5PV2E3QJMJHBMCYV24HYBFJXHGCJIIOEQQ, seg_pos={0 0}, upload=438]: Uploading piece... Sep 23 14:00:53.023933803 (29m38.486026286s) [noise=true , address=145.79.227.5:4223 , piece_id=NX7JL4Q5G4F2GTSSLQEWLT2FRGMR7FVKVK7GCHT7KMUUOC7NLXSQ, seg_pos={0 0}, upload=438]: Uploading piece... Sep 23 14:00:53.024393820 (29m38.486486293s) [noise=true , address=77.90.31.17:4772 , piece_id=FF6YRSO6BWWBGROBJXWYNVHXCPP3QJ2SEV5RNKO5FVQR2ESP5Y4Q, seg_pos={0 0}, upload=438]: Uploading piece... Sep 23 14:00:53.024718127 (29m38.4868106s) [noise=true , address=145.79.227.18:4006 , piece_id=SLTLJVDGCO3U2MKZPV56BC4UG5GK7SNKZOG3GTTR5RHCXMMIHVTA, seg_pos={0 0}, upload=438]: Uploading piece... Sep 23 14:00:53.026372096 (29m38.488464589s) [noise=true , address=45.91.116.14:4189 , piece_id=XLDG747S6MVFTVZC463E7JIID43W3OFBKD4HSBGFRUQEBOOZCYRA, seg_pos={0 0}, upload=438]: Uploading piece... Sep 23 14:00:53.027404983 (29m38.489497466s) [noise=true , address=145.79.227.6:4258 , piece_id=MX42637VOCBJ5X2VVSF7Z7SGTJWBMKS6UVUUH34E3THLYKHWP2RQ, seg_pos={0 0}, upload=438]: Uploading piece... Sep 23 14:00:53.027901498 (29m38.489993981s) [noise=true , address=145.79.227.11:4729 , piece_id=24IT2IEKAFAMZBYVHRS3SFVCW7E56ZMZKR5HIPX7HOFS2G7DCLMA, seg_pos={0 0}, upload=438]: Uploading piece... Sep 23 14:00:53.028517672 (29m38.490610155s) [noise=true , address=145.79.227.13:4802 , piece_id=YNPY4YP2HIDXDUGCWLW3IHLRFVPCLWKFC6IK5PY2QTJYDLEWBO5A, seg_pos={0 0}, upload=438]: Uploading piece... Sep 23 14:00:53.028824757 (29m38.49091724s) [noise=true , address=45.91.116.4:4939 , piece_id=FCFZOL3BBBFT5KVERWHWDBGMJ2QKZ2YQBDZMUH72DZ7FHZ4GTVEQ, seg_pos={0 0}, upload=438]: Uploading piece... Sep 23 14:00:53.029261588 (29m38.491354071s) [noise=true , address=45.91.116.8:4048 , piece_id=32OT24TVD66UTLB4LYH5LTZPR7GWLK7HYJBKKO66TD4U6PZTBAWQ, seg_pos={0 0}, upload=438]: Uploading piece... Sep 23 14:00:53.030074366 (29m38.492166849s) [noise=true , address=45.91.116.10:4082 , piece_id=MS23543NIW23BTMEQCLURGCW3JYDBVJYTNKQJGU7KGRS7AAB24AQ, seg_pos={0 0}, upload=438]: Uploading piece... Sep 23 14:00:53.031463381 (29m38.493555864s) [noise=true , address=77.90.31.19:4841 , piece_id=GBPGEEFH7NHSYNX2CLMTJDVXRVZEHMYCXVVOA4IVTHWCXNHRPVGQ, seg_pos={0 0}, upload=438]: Uploading piece... Sep 23 14:00:54.750616035 (29m40.212708518s) [upload=438]: Next returned nil segment Sep 23 14:00:54.750640462 (29m40.212732975s) [upload=438]: Waiting for error group managing segments... Sep 23 14:00:55.150366084 (29m40.612458567s) [noise=true , address=45.91.116.14:4189 , piece_id=XLDG747S6MVFTVZC463E7JIID43W3OFBKD4HSBGFRUQEBOOZCYRA, seg_pos={0 0}, upload=438]: Done uploading piece. err: <nil> Sep 23 14:00:55.164565562 (29m40.626658035s) [noise=true , address=77.90.31.17:4772 , piece_id=FF6YRSO6BWWBGROBJXWYNVHXCPP3QJ2SEV5RNKO5FVQR2ESP5Y4Q, seg_pos={0 0}, upload=438]: Done uploading piece. err: <nil> Sep 23 14:00:55.166583545 (29m40.628676028s) [noise=true , address=45.91.116.8:4048 , piece_id=32OT24TVD66UTLB4LYH5LTZPR7GWLK7HYJBKKO66TD4U6PZTBAWQ, seg_pos={0 0}, upload=438]: Done uploading piece. err: <nil> Sep 23 14:00:55.186422612 (29m40.648515095s) [noise=true , address=45.91.116.14:4191 , piece_id=2ECVPRNT6D55LZIYNE3IEDJEKL5VUWRLLZRU3RDDSYG6KQZ2VLJQ, seg_pos={0 0}, upload=438]: Done uploading piece. err: <nil> Sep 23 14:00:55.190561293 (29m40.652653776s) [noise=true , address=45.91.116.10:4082 , piece_id=MS23543NIW23BTMEQCLURGCW3JYDBVJYTNKQJGU7KGRS7AAB24AQ, seg_pos={0 0}, upload=438]: Done uploading piece. err: <nil> Sep 23 14:00:55.210007071 (29m40.672099544s) [noise=true , address=77.90.31.13:4626 , piece_id=YF2QO3MZZJ4NEFZU2W5PV2E3QJMJHBMCYV24HYBFJXHGCJIIOEQQ, seg_pos={0 0}, upload=438]: Done uploading piece. err: <nil> Sep 23 14:00:55.231461584 (29m40.693554067s) [noise=true , address=45.91.116.4:4939 , piece_id=FCFZOL3BBBFT5KVERWHWDBGMJ2QKZ2YQBDZMUH72DZ7FHZ4GTVEQ, seg_pos={0 0}, upload=438]: Done uploading piece. err: <nil> Sep 23 14:00:55.244042030 (29m40.706134513s) [noise=true , address=77.90.31.19:4841 , piece_id=GBPGEEFH7NHSYNX2CLMTJDVXRVZEHMYCXVVOA4IVTHWCXNHRPVGQ, seg_pos={0 0}, upload=438]: Done uploading piece. err: <nil> Sep 23 14:01:13.471619594 (29m58.933712078s) [noise=true , address=145.79.227.4:4185 , piece_id=722XLL2R2MJ5VM7UH724XG4ECKXUL7LRP2J6EV3XP56OL6TYEMWA, seg_pos={0 0}, upload=438]: Done uploading piece. err: <nil> Sep 23 14:01:19.185894931 (30m4.647987414s) [noise=true , address=145.79.227.18:4006 , piece_id=SLTLJVDGCO3U2MKZPV56BC4UG5GK7SNKZOG3GTTR5RHCXMMIHVTA, seg_pos={0 0}, upload=438]: Done uploading piece. err: <nil> Sep 23 14:01:21.537546227 (30m6.99963871s) [noise=true , address=145.79.227.11:4729 , piece_id=24IT2IEKAFAMZBYVHRS3SFVCW7E56ZMZKR5HIPX7HOFS2G7DCLMA, seg_pos={0 0}, upload=438]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: write tcp 188.166.225.170:60980->145.79.227.11:4729: use of closed network connection; write tcp 188.166.225.170:60980->145.79.227.11:4729: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60980->145.79.227.11:4729: use of closed network connection Sep 23 14:01:21.537649614 (30m6.999742097s) [noise=true , address=145.79.227.14:4840 , piece_id=2P5VSHWRYY5FADA7AAW7MAIRHLNUD7EFAQ3VIMN2FPHWIZSZTBZA, seg_pos={0 0}, upload=438]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: write tcp 188.166.225.170:53062->145.79.227.14:4840: use of closed network connection; write tcp 188.166.225.170:53062->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53062->145.79.227.14:4840: use of closed network connection Sep 23 14:01:21.537715650 (30m6.999808133s) [noise=true , address=145.79.227.13:4802 , piece_id=YNPY4YP2HIDXDUGCWLW3IHLRFVPCLWKFC6IK5PY2QTJYDLEWBO5A, seg_pos={0 0}, upload=438]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: write tcp 188.166.225.170:46828->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:46828->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46828->145.79.227.13:4802: use of closed network connection Sep 23 14:01:21.537968992 (30m7.000061475s) [noise=true , address=145.79.227.6:4258 , piece_id=MX42637VOCBJ5X2VVSF7Z7SGTJWBMKS6UVUUH34E3THLYKHWP2RQ, seg_pos={0 0}, upload=438]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 188.166.225.170:44340->145.79.227.6:4258: use of closed network connection; write tcp 188.166.225.170:44340->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44340->145.79.227.6:4258: use of closed network connection Sep 23 14:01:21.538031461 (30m7.000123934s) [noise=true , address=145.79.227.2:4115 , piece_id=DOQ4K26LCLRMI7JV2OSXUWU6YN2JGZFB6MD4ZER73LLIF6V7LS2Q, seg_pos={0 0}, upload=438]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: write tcp 188.166.225.170:35206->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:35206->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35206->145.79.227.2:4115: use of closed network connection Sep 23 14:01:21.538039636 (30m7.000132119s) [noise=true , address=145.79.227.5:4223 , piece_id=NX7JL4Q5G4F2GTSSLQEWLT2FRGMR7FVKVK7GCHT7KMUUOC7NLXSQ, seg_pos={0 0}, upload=438]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: write tcp 188.166.225.170:48638->145.79.227.5:4223: use of closed network connection; write tcp 188.166.225.170:48638->145.79.227.5:4223: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48638->145.79.227.5:4223: use of closed network connection Sep 23 14:01:21.538071347 (30m7.00016384s) [seg_pos={0 0}, upload=438]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Sep 23 14:01:21.922512835 (30m7.384605318s) [upload=438]: Done uploading segments... Sep 23 14:01:21.922553312 (30m7.384645795s) [upload=438]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 7.941088904s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [......................X...............................X...XX] duration: 30.387095903s FAIL (2025-09-23 14:01:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 4:00:52.632202144 (29m38.094294617s) [upload=438]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 14:00:52.632207815 (29m38.094300298s) [upload=438]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 14:00:53.022181978 (29m38.484274461s) [seg_pos={0 0}, upload=438]: Begin upload segment... Sep 23 14:00:53.022253353 (29m38.484345836s) [seg_pos={0 0}, upload=438]: Done begin upload segment. Sep 23 14:00:53.022311474 (29m38.484403957s) [noise=true , address=145.79.227.2:4115 , piece_id=DOQ4K26LCLRMI7JV2OSXUWU6YN2JGZFB6MD4ZER73LLIF6V7LS2Q, seg_pos={0 0}, upload=438]: Uploading piece... Sep 23 14:00:53.022325401 (29m38.484417884s) [noise=true , address=145.79.227.4:4185 , piece_id=722XLL2R2MJ5VM7UH724XG4ECKXUL7LRP2J6EV3XP56OL6TYEMWA, seg_pos={0 0}, upload=438]: Uploading piece... Sep 23 14:00:53.022997209 (29m38.485089683s) [noise=true , address=45.91.116.14:4191 , piece_id=2ECVPRNT6D55LZIYNE3IEDJEKL5VUWRLLZRU3RDDSYG6KQZ2VLJQ, seg_pos={0 0}, upload=438]: Uploading piece... Sep 23 14:00:53.023174999 (29m38.485267472s) [noise=true , address=145.79.227.14:4840 , piece_id=2P5VSHWRYY5FADA7AAW7MAIRHLNUD7EFAQ3VIMN2FPHWIZSZTBZA, seg_pos={0 0}, upload=438]: Uploading piece... Sep 23 14:00:53.023561134 (29m38.485653617s) [noise=true , address=77.90.31.13:4626 , piece_id=YF2QO3MZZJ4NEFZU2W5PV2E3QJMJHBMCYV24HYBFJXHGCJIIOEQQ, seg_pos={0 0}, upload=438]: Uploading piece... Sep 23 14:00:53.023933803 (29m38.486026286s) [noise=true , address=145.79.227.5:4223 , piece_id=NX7JL4Q5G4F2GTSSLQEWLT2FRGMR7FVKVK7GCHT7KMUUOC7NLXSQ, seg_pos={0 0}, upload=438]: Uploading piece... Sep 23 14:00:53.024393820 (29m38.486486293s) [noise=true , address=77.90.31.17:4772 , piece_id=FF6YRSO6BWWBGROBJXWYNVHXCPP3QJ2SEV5RNKO5FVQR2ESP5Y4Q, seg_pos={0 0}, upload=438]: Uploading piece... Sep 23 14:00:53.024718127 (29m38.4868106s) [noise=true , address=145.79.227.18:4006 , piece_id=SLTLJVDGCO3U2MKZPV56BC4UG5GK7SNKZOG3GTTR5RHCXMMIHVTA, seg_pos={0 0}, upload=438]: Uploading piece... Sep 23 14:00:53.026372096 (29m38.488464589s) [noise=true , address=45.91.116.14:4189 , piece_id=XLDG747S6MVFTVZC463E7JIID43W3OFBKD4HSBGFRUQEBOOZCYRA, seg_pos={0 0}, upload=438]: Uploading piece... Sep 23 14:00:53.027404983 (29m38.489497466s) [noise=true , address=145.79.227.6:4258 , piece_id=MX42637VOCBJ5X2VVSF7Z7SGTJWBMKS6UVUUH34E3THLYKHWP2RQ, seg_pos={0 0}, upload=438]: Uploading piece... Sep 23 14:00:53.027901498 (29m38.489993981s) [noise=true , address=145.79.227.11:4729 , piece_id=24IT2IEKAFAMZBYVHRS3SFVCW7E56ZMZKR5HIPX7HOFS2G7DCLMA, seg_pos={0 0}, upload=438]: Uploading piece... Sep 23 14:00:53.028517672 (29m38.490610155s) [noise=true , address=145.79.227.13:4802 , piece_id=YNPY4YP2HIDXDUGCWLW3IHLRFVPCLWKFC6IK5PY2QTJYDLEWBO5A, seg_pos={0 0}, upload=438]: Uploading piece... Sep 23 14:00:53.028824757 (29m38.49091724s) [noise=true , address=45.91.116.4:4939 , piece_id=FCFZOL3BBBFT5KVERWHWDBGMJ2QKZ2YQBDZMUH72DZ7FHZ4GTVEQ, seg_pos={0 0}, upload=438]: Uploading piece... Sep 23 14:00:53.029261588 (29m38.491354071s) [noise=true , address=45.91.116.8:4048 , piece_id=32OT24TVD66UTLB4LYH5LTZPR7GWLK7HYJBKKO66TD4U6PZTBAWQ, seg_pos={0 0}, upload=438]: Uploading piece... Sep 23 14:00:53.030074366 (29m38.492166849s) [noise=true , address=45.91.116.10:4082 , piece_id=MS23543NIW23BTMEQCLURGCW3JYDBVJYTNKQJGU7KGRS7AAB24AQ, seg_pos={0 0}, upload=438]: Uploading piece... Sep 23 14:00:53.031463381 (29m38.493555864s) [noise=true , address=77.90.31.19:4841 , piece_id=GBPGEEFH7NHSYNX2CLMTJDVXRVZEHMYCXVVOA4IVTHWCXNHRPVGQ, seg_pos={0 0}, upload=438]: Uploading piece... Sep 23 14:00:54.750616035 (29m40.212708518s) [upload=438]: Next returned nil segment Sep 23 14:00:54.750640462 (29m40.212732975s) [upload=438]: Waiting for error group managing segments... Sep 23 14:00:55.150366084 (29m40.612458567s) [noise=true , address=45.91.116.14:4189 , piece_id=XLDG747S6MVFTVZC463E7JIID43W3OFBKD4HSBGFRUQEBOOZCYRA, seg_pos={0 0}, upload=438]: Done uploading piece. err: <nil> Sep 23 14:00:55.164565562 (29m40.626658035s) [noise=true , address=77.90.31.17:4772 , piece_id=FF6YRSO6BWWBGROBJXWYNVHXCPP3QJ2SEV5RNKO5FVQR2ESP5Y4Q, seg_pos={0 0}, upload=438]: Done uploading piece. err: <nil> Sep 23 14:00:55.166583545 (29m40.628676028s) [noise=true , address=45.91.116.8:4048 , piece_id=32OT24TVD66UTLB4LYH5LTZPR7GWLK7HYJBKKO66TD4U6PZTBAWQ, seg_pos={0 0}, upload=438]: Done uploading piece. err: <nil> Sep 23 14:00:55.186422612 (29m40.648515095s) [noise=true , address=45.91.116.14:4191 , piece_id=2ECVPRNT6D55LZIYNE3IEDJEKL5VUWRLLZRU3RDDSYG6KQZ2VLJQ, seg_pos={0 0}, upload=438]: Done uploading piece. err: <nil> Sep 23 14:00:55.190561293 (29m40.652653776s) [noise=true , address=45.91.116.10:4082 , piece_id=MS23543NIW23BTMEQCLURGCW3JYDBVJYTNKQJGU7KGRS7AAB24AQ, seg_pos={0 0}, upload=438]: Done uploading piece. err: <nil> Sep 23 14:00:55.210007071 (29m40.672099544s) [noise=true , address=77.90.31.13:4626 , piece_id=YF2QO3MZZJ4NEFZU2W5PV2E3QJMJHBMCYV24HYBFJXHGCJIIOEQQ, seg_pos={0 0}, upload=438]: Done uploading piece. err: <nil> Sep 23 14:00:55.231461584 (29m40.693554067s) [noise=true , address=45.91.116.4:4939 , piece_id=FCFZOL3BBBFT5KVERWHWDBGMJ2QKZ2YQBDZMUH72DZ7FHZ4GTVEQ, seg_pos={0 0}, upload=438]: Done uploading piece. err: <nil> Sep 23 14:00:55.244042030 (29m40.706134513s) [noise=true , address=77.90.31.19:4841 , piece_id=GBPGEEFH7NHSYNX2CLMTJDVXRVZEHMYCXVVOA4IVTHWCXNHRPVGQ, seg_pos={0 0}, upload=438]: Done uploading piece. err: <nil> Sep 23 14:01:13.471619594 (29m58.933712078s) [noise=true , address=145.79.227.4:4185 , piece_id=722XLL2R2MJ5VM7UH724XG4ECKXUL7LRP2J6EV3XP56OL6TYEMWA, seg_pos={0 0}, upload=438]: Done uploading piece. err: <nil> Sep 23 14:01:19.185894931 (30m4.647987414s) [noise=true , address=145.79.227.18:4006 , piece_id=SLTLJVDGCO3U2MKZPV56BC4UG5GK7SNKZOG3GTTR5RHCXMMIHVTA, seg_pos={0 0}, upload=438]: Done uploading piece. err: <nil> Sep 23 14:01:21.537546227 (30m6.99963871s) [noise=true , address=145.79.227.11:4729 , piece_id=24IT2IEKAFAMZBYVHRS3SFVCW7E56ZMZKR5HIPX7HOFS2G7DCLMA, seg_pos={0 0}, upload=438]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: write tcp 188.166.225.170:60980->145.79.227.11:4729: use of closed network connection; write tcp 188.166.225.170:60980->145.79.227.11:4729: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60980->145.79.227.11:4729: use of closed network connection Sep 23 14:01:21.537649614 (30m6.999742097s) [noise=true , address=145.79.227.14:4840 , piece_id=2P5VSHWRYY5FADA7AAW7MAIRHLNUD7EFAQ3VIMN2FPHWIZSZTBZA, seg_pos={0 0}, upload=438]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: write tcp 188.166.225.170:53062->145.79.227.14:4840: use of closed network connection; write tcp 188.166.225.170:53062->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53062->145.79.227.14:4840: use of closed network connection Sep 23 14:01:21.537715650 (30m6.999808133s) [noise=true , address=145.79.227.13:4802 , piece_id=YNPY4YP2HIDXDUGCWLW3IHLRFVPCLWKFC6IK5PY2QTJYDLEWBO5A, seg_pos={0 0}, upload=438]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: write tcp 188.166.225.170:46828->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:46828->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46828->145.79.227.13:4802: use of closed network connection Sep 23 14:01:21.537968992 (30m7.000061475s) [noise=true , address=145.79.227.6:4258 , piece_id=MX42637VOCBJ5X2VVSF7Z7SGTJWBMKS6UVUUH34E3THLYKHWP2RQ, seg_pos={0 0}, upload=438]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 188.166.225.170:44340->145.79.227.6:4258: use of closed network connection; write tcp 188.166.225.170:44340->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44340->145.79.227.6:4258: use of closed network connection Sep 23 14:01:21.538031461 (30m7.000123934s) [noise=true , address=145.79.227.2:4115 , piece_id=DOQ4K26LCLRMI7JV2OSXUWU6YN2JGZFB6MD4ZER73LLIF6V7LS2Q, seg_pos={0 0}, upload=438]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: write tcp 188.166.225.170:35206->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:35206->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35206->145.79.227.2:4115: use of closed network connection Sep 23 14:01:21.538039636 (30m7.000132119s) [noise=true , address=145.79.227.5:4223 , piece_id=NX7JL4Q5G4F2GTSSLQEWLT2FRGMR7FVKVK7GCHT7KMUUOC7NLXSQ, seg_pos={0 0}, upload=438]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: write tcp 188.166.225.170:48638->145.79.227.5:4223: use of closed network connection; write tcp 188.166.225.170:48638->145.79.227.5:4223: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48638->145.79.227.5:4223: use of closed network connection Sep 23 14:01:21.538071347 (30m7.00016384s) [seg_pos={0 0}, upload=438]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Sep 23 14:01:21.922512835 (30m7.384605318s) [upload=438]: Done uploading segments... Sep 23 14:01:21.922553312 (30m7.384645795s) [upload=438]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.387114999s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......................X...............................X...X] duration: 30.258934318s FAIL (2025-09-23 14:00:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 23 14:00:21.787990471 (29m7.250082954s) [upload=432]: Starting upload Sep 23 14:00:21.788069913 (29m7.250162396s) [upload=432]: Done starting upload Sep 23 14:00:21.788225318 (29m7.250317801s) [upload=432]: Uploading segments... Sep 23 14:00:21.788439185 (29m7.250531668s) [upload=432]: Got next segment. Inline: false Sep 23 14:00:21.788506604 (29m7.250599087s) [upload=432]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 14:00:21.788515952 (29m7.250608435s) [upload=432]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 14:00:22.142022763 (29m7.604115246s) [seg_pos={0 0}, upload=432]: Begin upload segment... Sep 23 14:00:22.142085323 (29m7.604177806s) [seg_pos={0 0}, upload=432]: Done begin upload segment. Sep 23 14:00:22.142113126 (29m7.604205609s) [noise=true , address=145.79.227.7:4295 , piece_id=SPNWY7R2CC7KCVJ5MLUQSG7ANC546NKIOASMV4V6MRGL7SMQECDA, seg_pos={0 0}, upload=432]: Uploading piece... Sep 23 14:00:22.142179172 (29m7.604271655s) [noise=true , address=45.91.116.8:4048 , piece_id=TTZ65VNVXWHWQ3UXXHUI5CT6F4EDYFLYZVBLOS53EL6D6UPI3AOQ, seg_pos={0 0}, upload=432]: Uploading piece... Sep 23 14:00:22.142801427 (29m7.60489391s) [noise=true , address=145.79.227.7:4293 , piece_id=KQOY7GDO447E2BDBHK3LYNGXDKFJKKYJG4DGHYFSCJP2QXD7NHWQ, seg_pos={0 0}, upload=432]: Uploading piece... Sep 23 14:00:22.143200526 (29m7.605293009s) [noise=true , address=45.91.116.5:4299 , piece_id=BGGQKVHAA7ESLSHGDGXXZ6BYD4G2LHHMCOFUR7DECCVUWDY7I4EQ, seg_pos={0 0}, upload=432]: Uploading piece... Sep 23 14:00:22.143608523 (29m7.605701016s) [noise=true , address=45.91.116.7:4011 , piece_id=Y3L53KWHUGO5X74H4TOXVB4J7DZOFETATUXAME7LQNH5V7WK7BEA, seg_pos={0 0}, upload=432]: Uploading piece... Sep 23 14:00:22.144050224 (29m7.606142707s) [noise=true , address=145.79.227.14:4840 , piece_id=MVWRFI4RNMZ55HQUBCUY5BQ6LJG47HRQPZRTKZYEOQ5UFCGJN7BQ, seg_pos={0 0}, upload=432]: Uploading piece... Sep 23 14:00:22.144654755 (29m7.606747239s) [noise=true , address=145.79.227.15:4875 , piece_id=HSPSH4NM5A5KSYBOYJVS3L6YXEVHLFHL3FSBOM73T4RXCELV4CNQ, seg_pos={0 0}, upload=432]: Uploading piece... Sep 23 14:00:22.145241623 (29m7.607334106s) [noise=true , address=45.91.116.10:4083 , piece_id=OZUROAR5BBZM5UM4GT6U47O7ZDYXWIAFLPYC5TTSJIXJBAVK2UPA, seg_pos={0 0}, upload=432]: Uploading piece... Sep 23 14:00:22.146870604 (29m7.608963087s) [noise=true , address=45.91.116.4:4939 , piece_id=WNNYMZUUQXVANEZQBBXQ5B26RJOF7QMNZ6D5DFGEZX7XYBOTIBDA, seg_pos={0 0}, upload=432]: Uploading piece... Sep 23 14:00:22.147253183 (29m7.609345666s) [noise=true , address=145.79.227.5:4223 , piece_id=4ETC4F5EVJZ75VP7PEJ4UBRJDNRV45MW2L5BIDD2DUC2YYKTBMJA, seg_pos={0 0}, upload=432]: Uploading piece... Sep 23 14:00:22.148936599 (29m7.611029092s) [noise=true , address=45.91.116.5:4297 , piece_id=STGRTYYNERAF4BW4D2RSISPWH3YOJWXJKE6CQERJBNG2WSQ5P5IQ, seg_pos={0 0}, upload=432]: Uploading piece... Sep 23 14:00:22.150733279 (29m7.612825762s) [noise=true , address=145.79.227.2:4115 , piece_id=QR6MCHT3XD3FQXEEWDWZIGK65Y2VDDYBCSAHN7QAJYOZVSKHHFWQ, seg_pos={0 0}, upload=432]: Uploading piece... Sep 23 14:00:22.152827007 (29m7.61491949s) [noise=true , address=45.91.116.6:4975 , piece_id=XS2L7OGXUDPLPV66VNLRXJGQN5T5EPCRFTV6E6YWTMJ4PJZ3KLWA, seg_pos={0 0}, upload=432]: Uploading piece... Sep 23 14:00:22.155074536 (29m7.617167029s) [noise=true , address=145.79.227.13:4802 , piece_id=OWNNEEDGEFV7VL4KSBXF556DW2ZAJU6XT55Y5PRBSLWFIRDHZ2ZA, seg_pos={0 0}, upload=432]: Uploading piece... Sep 23 14:00:22.156432692 (29m7.618525185s) [noise=true , address=45.91.116.13:4154 , piece_id=ZSTJF2S3SQLWFF7PFHM7TC3TNJYUBQMOZOB3OH5LUQGWGHBPBZRQ, seg_pos={0 0}, upload=432]: Uploading piece... Sep 23 14:00:22.157716387 (29m7.61980887s) [noise=true , address=45.91.116.14:4190 , piece_id=RTXOQSVIYMAYJIO74GUNAND77SAZTC4XHPAJXRHPAMMNW4V7BUYQ, seg_pos={0 0}, upload=432]: Uploading piece... Sep 23 14:00:23.868030421 (29m9.330122904s) [upload=432]: Next returned nil segment Sep 23 14:00:23.868054357 (29m9.33014684s) [upload=432]: Waiting for error group managing segments... Sep 23 14:00:24.270533893 (29m9.732626376s) [noise=true , address=45.91.116.7:4011 , piece_id=Y3L53KWHUGO5X74H4TOXVB4J7DZOFETATUXAME7LQNH5V7WK7BEA, seg_pos={0 0}, upload=432]: Done uploading piece. err: <nil> Sep 23 14:00:24.341513050 (29m9.803605533s) [noise=true , address=45.91.116.5:4297 , piece_id=STGRTYYNERAF4BW4D2RSISPWH3YOJWXJKE6CQERJBNG2WSQ5P5IQ, seg_pos={0 0}, upload=432]: Done uploading piece. err: <nil> Sep 23 14:00:24.360637276 (29m9.822729759s) [noise=true , address=45.91.116.14:4190 , piece_id=RTXOQSVIYMAYJIO74GUNAND77SAZTC4XHPAJXRHPAMMNW4V7BUYQ, seg_pos={0 0}, upload=432]: Done uploading piece. err: <nil> Sep 23 14:00:24.368536738 (29m9.830629221s) [noise=true , address=45.91.116.6:4975 , piece_id=XS2L7OGXUDPLPV66VNLRXJGQN5T5EPCRFTV6E6YWTMJ4PJZ3KLWA, seg_pos={0 0}, upload=432]: Done uploading piece. err: <nil> Sep 23 14:00:24.382527179 (29m9.844619662s) [noise=true , address=45.91.116.8:4048 , piece_id=TTZ65VNVXWHWQ3UXXHUI5CT6F4EDYFLYZVBLOS53EL6D6UPI3AOQ, seg_pos={0 0}, upload=432]: Done uploading piece. err: <nil> Sep 23 14:00:24.384722018 (29m9.846814491s) [noise=true , address=45.91.116.5:4299 , piece_id=BGGQKVHAA7ESLSHGDGXXZ6BYD4G2LHHMCOFUR7DECCVUWDY7I4EQ, seg_pos={0 0}, upload=432]: Done uploading piece. err: <nil> Sep 23 14:00:24.392787636 (29m9.854880139s) [noise=true , address=45.91.116.4:4939 , piece_id=WNNYMZUUQXVANEZQBBXQ5B26RJOF7QMNZ6D5DFGEZX7XYBOTIBDA, seg_pos={0 0}, upload=432]: Done uploading piece. err: <nil> Sep 23 14:00:24.394214052 (29m9.856306535s) [noise=true , address=45.91.116.10:4083 , piece_id=OZUROAR5BBZM5UM4GT6U47O7ZDYXWIAFLPYC5TTSJIXJBAVK2UPA, seg_pos={0 0}, upload=432]: Done uploading piece. err: <nil> Sep 23 14:00:24.421079358 (29m9.883171842s) [noise=true , address=45.91.116.13:4154 , piece_id=ZSTJF2S3SQLWFF7PFHM7TC3TNJYUBQMOZOB3OH5LUQGWGHBPBZRQ, seg_pos={0 0}, upload=432]: Done uploading piece. err: <nil> Sep 23 14:00:29.603530875 (29m15.065623348s) [noise=true , address=145.79.227.7:4295 , piece_id=SPNWY7R2CC7KCVJ5MLUQSG7ANC546NKIOASMV4V6MRGL7SMQECDA, seg_pos={0 0}, upload=432]: Done uploading piece. err: <nil> Sep 23 14:00:35.490981400 (29m20.953073883s) [noise=true , address=145.79.227.2:4115 , piece_id=QR6MCHT3XD3FQXEEWDWZIGK65Y2VDDYBCSAHN7QAJYOZVSKHHFWQ, seg_pos={0 0}, upload=432]: Done uploading piece. err: <nil> Sep 23 14:00:36.255350691 (29m21.717443175s) [noise=true , address=145.79.227.7:4293 , piece_id=KQOY7GDO447E2BDBHK3LYNGXDKFJKKYJG4DGHYFSCJP2QXD7NHWQ, seg_pos={0 0}, upload=432]: Done uploading piece. err: <nil> Sep 23 14:00:51.045219095 (29m36.507311578s) [noise=true , address=145.79.227.13:4802 , piece_id=OWNNEEDGEFV7VL4KSBXF556DW2ZAJU6XT55Y5PRBSLWFIRDHZ2ZA, seg_pos={0 0}, upload=432]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: write tcp 188.166.225.170:35348->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:35348->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35348->145.79.227.13:4802: use of closed network connection Sep 23 14:00:51.045331770 (29m36.507424243s) [noise=true , address=145.79.227.14:4840 , piece_id=MVWRFI4RNMZ55HQUBCUY5BQ6LJG47HRQPZRTKZYEOQ5UFCGJN7BQ, seg_pos={0 0}, upload=432]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: write tcp 188.166.225.170:40086->145.79.227.14:4840: use of closed network connection; write tcp 188.166.225.170:40086->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40086->145.79.227.14:4840: use of closed network connection Sep 23 14:00:51.045392204 (29m36.507484687s) [noise=true , address=145.79.227.15:4875 , piece_id=HSPSH4NM5A5KSYBOYJVS3L6YXEVHLFHL3FSBOM73T4RXCELV4CNQ, seg_pos={0 0}, upload=432]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:57344->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:57344->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57344->145.79.227.15:4875: use of closed network connection Sep 23 14:00:51.045444204 (29m36.507536677s) [noise=true , address=145.79.227.5:4223 , piece_id=4ETC4F5EVJZ75VP7PEJ4UBRJDNRV45MW2L5BIDD2DUC2YYKTBMJA, seg_pos={0 0}, upload=432]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 14:00:51.045464853 (29m36.507557336s) [seg_pos={0 0}, upload=432]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 23 14:00:51.302793154 (29m36.764885627s) [upload=432]: Done uploading segments... Sep 23 14:00:51.302824483 (29m36.764916956s) [upload=432]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.258972791s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 16.882318239s FAIL (2025-09-23 14:00:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 88 (26m9.156611862s) [upload=388]: Starting upload Sep 23 13:57:23.694601205 (26m9.156693688s) [upload=388]: Done starting upload Sep 23 13:57:23.694812668 (26m9.156905151s) [upload=388]: Uploading segments... Sep 23 13:57:23.694953145 (26m9.157045658s) [upload=388]: Got next segment. Inline: false Sep 23 13:57:23.695037716 (26m9.157130199s) [upload=388]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 13:57:23.695045330 (26m9.157137803s) [upload=388]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 13:57:24.190620566 (26m9.652713049s) [seg_pos={0 0}, upload=388]: Begin upload segment... Sep 23 13:57:24.190689898 (26m9.652782381s) [seg_pos={0 0}, upload=388]: Done begin upload segment. Sep 23 13:57:24.190740063 (26m9.652832547s) [noise=true , address=145.79.227.19:4091 , piece_id=YENXMGIO55UZH4XS2JIDETS73NB5I267ZNHUYA2NBPBTNICBKCTA, seg_pos={0 0}, upload=388]: Uploading piece... Sep 23 13:57:24.190752077 (26m9.65284456s) [noise=true , address=145.79.227.29:4948 , piece_id=Z6YHGVGSFELOEKNFPGIT6H72L5FTMCEEFPPUWB5MZOC5FIGKW4CA, seg_pos={0 0}, upload=388]: Uploading piece... Sep 23 13:57:24.191368831 (26m9.653461334s) [noise=true , address=145.79.227.28:4759 , piece_id=7NBBUWOJ6E2VAP6L5J6VB6BGYRSQRF4Z3GH4ZWZ65GWTYARRE5AA, seg_pos={0 0}, upload=388]: Uploading piece... Sep 23 13:57:24.191515501 (26m9.653607984s) [noise=true , address=145.79.227.6:4272 , piece_id=LLJGTG3KRKBKOKERTN5J6EBQ67TJVHLHU5XEYNQWRJFGBEBYG46Q, seg_pos={0 0}, upload=388]: Uploading piece... Sep 23 13:57:24.191899572 (26m9.653992055s) [noise=true , address=77.90.31.11:4556 , piece_id=53GKZFMG4MQ72VHOL637G7UM7MNMYEFPPFGCP2UC2EAZGMOZHYDA, seg_pos={0 0}, upload=388]: Uploading piece... Sep 23 13:57:24.191983171 (26m9.654075644s) [noise=true , address=45.91.116.4:4966 , piece_id=QYNM4GKDHPYS7M4E4DYLB6QJS5SN22VA3YFUOB6YNKPMTJCUIEHQ, seg_pos={0 0}, upload=388]: Uploading piece... Sep 23 13:57:24.192324321 (26m9.654416804s) [noise=true , address=145.79.227.34:4916 , piece_id=KDJMSPO6JKDCXMVFYCLMJTCQW4I2TED6QDE6P3COCVLQ7SYVB33A, seg_pos={0 0}, upload=388]: Uploading piece... Sep 23 13:57:24.192458927 (26m9.65455141s) [noise=true , address=77.90.31.18:4824 , piece_id=M3DC45GZ576HMVLDUXX2O2RQLNTFEYOQOKLC2DSKNMHAPM5WHSMA, seg_pos={0 0}, upload=388]: Uploading piece... Sep 23 13:57:24.192696069 (26m9.654788552s) [noise=true , address=145.79.227.40:4057 , piece_id=HFURC5BTIQ5W7V2X75CLUGJSMR5FEIZFIZZG3SZJNSDVOL62CGQQ, seg_pos={0 0}, upload=388]: Uploading piece... Sep 23 13:57:24.192868898 (26m9.654961381s) [noise=true , address=145.79.227.12:4775 , piece_id=JPYKKBJFFI3SUOFD44VKC2GJ32AO4H6BEMIOHWPOS5JR5SEYHQWQ, seg_pos={0 0}, upload=388]: Uploading piece... Sep 23 13:57:24.193046977 (26m9.65513945s) [noise=true , address=145.79.227.49:4345 , piece_id=VXMMBEZQV5DGWJZEPSKFKOYLZIYSV3P75ITSCBEQIC2WW3W4IXGQ, seg_pos={0 0}, upload=388]: Uploading piece... Sep 23 13:57:24.193266214 (26m9.655358697s) [noise=true , address=145.79.227.23:4578 , piece_id=U4TAYSO4IG5U634KPCNTWELXIXX6RFSFLJC7AP3FXRKE5R3Z2IVQ, seg_pos={0 0}, upload=388]: Uploading piece... Sep 23 13:57:24.193645757 (26m9.65573824s) [noise=true , address=145.79.227.38:4989 , piece_id=NWWLGSULG5LWUPDTD647XNP4U63IIFNMLQNTDGL5KKKUJBER4PIQ, seg_pos={0 0}, upload=388]: Uploading piece... Sep 23 13:57:24.194120039 (26m9.656212513s) [noise=true , address=145.79.227.46:4635 , piece_id=VSETY7Q3Q3O6QDIQBSEKHCI7MZHGC7LM4LLNDJXFUYUVIGZVLXUA, seg_pos={0 0}, upload=388]: Uploading piece... Sep 23 13:57:24.194266850 (26m9.656359333s) [noise=true , address=145.79.227.2:4128 , piece_id=KG4A6OARKJFUDGHBIJSXOHGUQ2Y5ZTBEFXD576W5U5CTZQD36MPA, seg_pos={0 0}, upload=388]: Uploading piece... Sep 23 13:57:24.194628749 (26m9.656721222s) [noise=true , address=45.91.116.6:4987 , piece_id=KT2C25BRSWIBK2QAGRBRDJPPFPT7PYGY6K3ZLSCLNFTMR445QDIQ, seg_pos={0 0}, upload=388]: Uploading piece... Sep 23 13:57:24.195070971 (26m9.657163454s) [noise=true , address=145.79.227.4:4217 , piece_id=4WAOONYJVFJKZTCIVX7GWOGGK5XDCP5TGQRKZ3SCI42KFRO6KHCQ, seg_pos={0 0}, upload=388]: Uploading piece... Sep 23 13:57:24.195153838 (26m9.657246321s) [noise=true , address=145.79.227.11:4750 , piece_id=CJXKOVDLJVEKLEMEN3RSBO7576W32WMA6C3KEJDX3GS3XIUQZFZA, seg_pos={0 0}, upload=388]: Uploading piece... Sep 23 13:57:25.809063919 (26m11.271156392s) [upload=388]: Next returned nil segment Sep 23 13:57:25.809086972 (26m11.271179456s) [upload=388]: Waiting for error group managing segments... Sep 23 13:57:26.258715569 (26m11.720808052s) [noise=true , address=45.91.116.6:4987 , piece_id=KT2C25BRSWIBK2QAGRBRDJPPFPT7PYGY6K3ZLSCLNFTMR445QDIQ, seg_pos={0 0}, upload=388]: Done uploading piece. err: <nil> Sep 23 13:57:26.281240952 (26m11.743333435s) [noise=true , address=45.91.116.4:4966 , piece_id=QYNM4GKDHPYS7M4E4DYLB6QJS5SN22VA3YFUOB6YNKPMTJCUIEHQ, seg_pos={0 0}, upload=388]: Done uploading piece. err: <nil> Sep 23 13:57:26.289091760 (26m11.751184244s) [noise=true , address=77.90.31.11:4556 , piece_id=53GKZFMG4MQ72VHOL637G7UM7MNMYEFPPFGCP2UC2EAZGMOZHYDA, seg_pos={0 0}, upload=388]: Done uploading piece. err: <nil> Sep 23 13:57:26.311620449 (26m11.773712932s) [noise=true , address=77.90.31.18:4824 , piece_id=M3DC45GZ576HMVLDUXX2O2RQLNTFEYOQOKLC2DSKNMHAPM5WHSMA, seg_pos={0 0}, upload=388]: Done uploading piece. err: <nil> Sep 23 13:57:53.735443730 (26m39.197536203s) [noise=true , address=145.79.227.11:4750 , piece_id=CJXKOVDLJVEKLEMEN3RSBO7576W32WMA6C3KEJDX3GS3XIUQZFZA, seg_pos={0 0}, upload=388]: Done uploading piece. err: <nil> Sep 23 13:57:55.409099718 (26m40.871192202s) [noise=true , address=145.79.227.12:4775 , piece_id=JPYKKBJFFI3SUOFD44VKC2GJ32AO4H6BEMIOHWPOS5JR5SEYHQWQ, seg_pos={0 0}, upload=388]: Done uploading piece. err: <nil> Sep 23 13:58:05.387583288 (26m50.849675771s) [noise=true , address=145.79.227.2:4128 , piece_id=KG4A6OARKJFUDGHBIJSXOHGUQ2Y5ZTBEFXD576W5U5CTZQD36MPA, seg_pos={0 0}, upload=388]: Done uploading piece. err: <nil> Sep 23 13:58:11.832148818 (26m57.294241291s) [noise=true , address=145.79.227.23:4578 , piece_id=U4TAYSO4IG5U634KPCNTWELXIXX6RFSFLJC7AP3FXRKE5R3Z2IVQ, seg_pos={0 0}, upload=388]: Done uploading piece. err: <nil> Sep 23 13:58:13.997613861 (26m59.459706344s) [noise=true , address=145.79.227.28:4759 , piece_id=7NBBUWOJ6E2VAP6L5J6VB6BGYRSQRF4Z3GH4ZWZ65GWTYARRE5AA, seg_pos={0 0}, upload=388]: Done uploading piece. err: <nil> Sep 23 13:58:14.705556905 (27m0.167649388s) [noise=true , address=145.79.227.40:4057 , piece_id=HFURC5BTIQ5W7V2X75CLUGJSMR5FEIZFIZZG3SZJNSDVOL62CGQQ, seg_pos={0 0}, upload=388]: Done uploading piece. err: <nil> Sep 23 13:58:15.200396601 (27m0.662489084s) [noise=true , address=145.79.227.46:4635 , piece_id=VSETY7Q3Q3O6QDIQBSEKHCI7MZHGC7LM4LLNDJXFUYUVIGZVLXUA, seg_pos={0 0}, upload=388]: Done uploading piece. err: <nil> Sep 23 13:58:18.011996926 (27m3.474089409s) [noise=true , address=145.79.227.38:4989 , piece_id=NWWLGSULG5LWUPDTD647XNP4U63IIFNMLQNTDGL5KKKUJBER4PIQ, seg_pos={0 0}, upload=388]: Done uploading piece. err: <nil> Sep 23 13:58:20.722114840 (27m6.184207324s) [noise=true , address=145.79.227.19:4091 , piece_id=YENXMGIO55UZH4XS2JIDETS73NB5I267ZNHUYA2NBPBTNICBKCTA, seg_pos={0 0}, upload=388]: Done uploading piece. err: <nil> Sep 23 13:58:21.718721462 (27m7.180813935s) [noise=true , address=145.79.227.49:4345 , piece_id=VXMMBEZQV5DGWJZEPSKFKOYLZIYSV3P75ITSCBEQIC2WW3W4IXGQ, seg_pos={0 0}, upload=388]: Done uploading piece. err: ecclient: upload failed (node:1KSTyRJ62wcMHx8K7uwqp5e8L8yhxH7V3YtBHnYu6QJ3K3bXpQ, address:145.79.227.49:4345): protocol: write tcp 188.166.225.170:32840->145.79.227.49:4345: use of closed network connection; write tcp 188.166.225.170:32840->145.79.227.49:4345: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32840->145.79.227.49:4345: use of closed network connection Sep 23 13:58:21.718824248 (27m7.180916731s) [noise=true , address=145.79.227.4:4217 , piece_id=4WAOONYJVFJKZTCIVX7GWOGGK5XDCP5TGQRKZ3SCI42KFRO6KHCQ, seg_pos={0 0}, upload=388]: Done uploading piece. err: ecclient: upload failed (node:123KLtoTMCQ9DZUjJ8JJab4D5bc7UGTt7yjVV8wDgFTozdZ5gpi, address:145.79.227.4:4217): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 13:58:21.719046873 (27m7.181139356s) [noise=true , address=145.79.227.6:4272 , piece_id=LLJGTG3KRKBKOKERTN5J6EBQ67TJVHLHU5XEYNQWRJFGBEBYG46Q, seg_pos={0 0}, upload=388]: Done uploading piece. err: ecclient: upload failed (node:1UeKsX3pQfhvCukBvJf1sekK7N6sXkaWaJnZzETKoxyXWo2BFi, address:145.79.227.6:4272): protocol: write tcp 188.166.225.170:57934->145.79.227.6:4272: use of closed network connection; write tcp 188.166.225.170:57934->145.79.227.6:4272: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57934->145.79.227.6:4272: use of closed network connection Sep 23 13:58:21.719060188 (27m7.181152671s) [noise=true , address=145.79.227.29:4948 , piece_id=Z6YHGVGSFELOEKNFPGIT6H72L5FTMCEEFPPUWB5MZOC5FIGKW4CA, seg_pos={0 0}, upload=388]: Done uploading piece. err: ecclient: upload failed (node:12vRsfmHG1SH4gL4RzmiUY9uZyF7uy3UfEWipuH4xaGnQ6NuDZL, address:145.79.227.29:4948): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 13:58:21.719144297 (27m7.18123678s) [noise=true , address=145.79.227.34:4916 , piece_id=KDJMSPO6JKDCXMVFYCLMJTCQW4I2TED6QDE6P3COCVLQ7SYVB33A, seg_pos={0 0}, upload=388]: Done uploading piece. err: ecclient: upload failed (node:12sbCx4R1g3vGDnXQSARG3UD5pwzqUqkE1q37CUzn2skJftjLEa, address:145.79.227.34:4916): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 13:58:21.719182551 (27m7.181275034s) [seg_pos={0 0}, upload=388]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 23 13:58:21.954649324 (27m7.416741817s) [upload=388]: Done uploading segments... Sep 23 13:58:21.954684411 (27m7.416776894s) [upload=388]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 16.882357955s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 6.040152449s FAIL (2025-09-23 13:59:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 88 (26m9.156611862s) [upload=388]: Starting upload Sep 23 13:57:23.694601205 (26m9.156693688s) [upload=388]: Done starting upload Sep 23 13:57:23.694812668 (26m9.156905151s) [upload=388]: Uploading segments... Sep 23 13:57:23.694953145 (26m9.157045658s) [upload=388]: Got next segment. Inline: false Sep 23 13:57:23.695037716 (26m9.157130199s) [upload=388]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 13:57:23.695045330 (26m9.157137803s) [upload=388]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 13:57:24.190620566 (26m9.652713049s) [seg_pos={0 0}, upload=388]: Begin upload segment... Sep 23 13:57:24.190689898 (26m9.652782381s) [seg_pos={0 0}, upload=388]: Done begin upload segment. Sep 23 13:57:24.190740063 (26m9.652832547s) [noise=true , address=145.79.227.19:4091 , piece_id=YENXMGIO55UZH4XS2JIDETS73NB5I267ZNHUYA2NBPBTNICBKCTA, seg_pos={0 0}, upload=388]: Uploading piece... Sep 23 13:57:24.190752077 (26m9.65284456s) [noise=true , address=145.79.227.29:4948 , piece_id=Z6YHGVGSFELOEKNFPGIT6H72L5FTMCEEFPPUWB5MZOC5FIGKW4CA, seg_pos={0 0}, upload=388]: Uploading piece... Sep 23 13:57:24.191368831 (26m9.653461334s) [noise=true , address=145.79.227.28:4759 , piece_id=7NBBUWOJ6E2VAP6L5J6VB6BGYRSQRF4Z3GH4ZWZ65GWTYARRE5AA, seg_pos={0 0}, upload=388]: Uploading piece... Sep 23 13:57:24.191515501 (26m9.653607984s) [noise=true , address=145.79.227.6:4272 , piece_id=LLJGTG3KRKBKOKERTN5J6EBQ67TJVHLHU5XEYNQWRJFGBEBYG46Q, seg_pos={0 0}, upload=388]: Uploading piece... Sep 23 13:57:24.191899572 (26m9.653992055s) [noise=true , address=77.90.31.11:4556 , piece_id=53GKZFMG4MQ72VHOL637G7UM7MNMYEFPPFGCP2UC2EAZGMOZHYDA, seg_pos={0 0}, upload=388]: Uploading piece... Sep 23 13:57:24.191983171 (26m9.654075644s) [noise=true , address=45.91.116.4:4966 , piece_id=QYNM4GKDHPYS7M4E4DYLB6QJS5SN22VA3YFUOB6YNKPMTJCUIEHQ, seg_pos={0 0}, upload=388]: Uploading piece... Sep 23 13:57:24.192324321 (26m9.654416804s) [noise=true , address=145.79.227.34:4916 , piece_id=KDJMSPO6JKDCXMVFYCLMJTCQW4I2TED6QDE6P3COCVLQ7SYVB33A, seg_pos={0 0}, upload=388]: Uploading piece... Sep 23 13:57:24.192458927 (26m9.65455141s) [noise=true , address=77.90.31.18:4824 , piece_id=M3DC45GZ576HMVLDUXX2O2RQLNTFEYOQOKLC2DSKNMHAPM5WHSMA, seg_pos={0 0}, upload=388]: Uploading piece... Sep 23 13:57:24.192696069 (26m9.654788552s) [noise=true , address=145.79.227.40:4057 , piece_id=HFURC5BTIQ5W7V2X75CLUGJSMR5FEIZFIZZG3SZJNSDVOL62CGQQ, seg_pos={0 0}, upload=388]: Uploading piece... Sep 23 13:57:24.192868898 (26m9.654961381s) [noise=true , address=145.79.227.12:4775 , piece_id=JPYKKBJFFI3SUOFD44VKC2GJ32AO4H6BEMIOHWPOS5JR5SEYHQWQ, seg_pos={0 0}, upload=388]: Uploading piece... Sep 23 13:57:24.193046977 (26m9.65513945s) [noise=true , address=145.79.227.49:4345 , piece_id=VXMMBEZQV5DGWJZEPSKFKOYLZIYSV3P75ITSCBEQIC2WW3W4IXGQ, seg_pos={0 0}, upload=388]: Uploading piece... Sep 23 13:57:24.193266214 (26m9.655358697s) [noise=true , address=145.79.227.23:4578 , piece_id=U4TAYSO4IG5U634KPCNTWELXIXX6RFSFLJC7AP3FXRKE5R3Z2IVQ, seg_pos={0 0}, upload=388]: Uploading piece... Sep 23 13:57:24.193645757 (26m9.65573824s) [noise=true , address=145.79.227.38:4989 , piece_id=NWWLGSULG5LWUPDTD647XNP4U63IIFNMLQNTDGL5KKKUJBER4PIQ, seg_pos={0 0}, upload=388]: Uploading piece... Sep 23 13:57:24.194120039 (26m9.656212513s) [noise=true , address=145.79.227.46:4635 , piece_id=VSETY7Q3Q3O6QDIQBSEKHCI7MZHGC7LM4LLNDJXFUYUVIGZVLXUA, seg_pos={0 0}, upload=388]: Uploading piece... Sep 23 13:57:24.194266850 (26m9.656359333s) [noise=true , address=145.79.227.2:4128 , piece_id=KG4A6OARKJFUDGHBIJSXOHGUQ2Y5ZTBEFXD576W5U5CTZQD36MPA, seg_pos={0 0}, upload=388]: Uploading piece... Sep 23 13:57:24.194628749 (26m9.656721222s) [noise=true , address=45.91.116.6:4987 , piece_id=KT2C25BRSWIBK2QAGRBRDJPPFPT7PYGY6K3ZLSCLNFTMR445QDIQ, seg_pos={0 0}, upload=388]: Uploading piece... Sep 23 13:57:24.195070971 (26m9.657163454s) [noise=true , address=145.79.227.4:4217 , piece_id=4WAOONYJVFJKZTCIVX7GWOGGK5XDCP5TGQRKZ3SCI42KFRO6KHCQ, seg_pos={0 0}, upload=388]: Uploading piece... Sep 23 13:57:24.195153838 (26m9.657246321s) [noise=true , address=145.79.227.11:4750 , piece_id=CJXKOVDLJVEKLEMEN3RSBO7576W32WMA6C3KEJDX3GS3XIUQZFZA, seg_pos={0 0}, upload=388]: Uploading piece... Sep 23 13:57:25.809063919 (26m11.271156392s) [upload=388]: Next returned nil segment Sep 23 13:57:25.809086972 (26m11.271179456s) [upload=388]: Waiting for error group managing segments... Sep 23 13:57:26.258715569 (26m11.720808052s) [noise=true , address=45.91.116.6:4987 , piece_id=KT2C25BRSWIBK2QAGRBRDJPPFPT7PYGY6K3ZLSCLNFTMR445QDIQ, seg_pos={0 0}, upload=388]: Done uploading piece. err: <nil> Sep 23 13:57:26.281240952 (26m11.743333435s) [noise=true , address=45.91.116.4:4966 , piece_id=QYNM4GKDHPYS7M4E4DYLB6QJS5SN22VA3YFUOB6YNKPMTJCUIEHQ, seg_pos={0 0}, upload=388]: Done uploading piece. err: <nil> Sep 23 13:57:26.289091760 (26m11.751184244s) [noise=true , address=77.90.31.11:4556 , piece_id=53GKZFMG4MQ72VHOL637G7UM7MNMYEFPPFGCP2UC2EAZGMOZHYDA, seg_pos={0 0}, upload=388]: Done uploading piece. err: <nil> Sep 23 13:57:26.311620449 (26m11.773712932s) [noise=true , address=77.90.31.18:4824 , piece_id=M3DC45GZ576HMVLDUXX2O2RQLNTFEYOQOKLC2DSKNMHAPM5WHSMA, seg_pos={0 0}, upload=388]: Done uploading piece. err: <nil> Sep 23 13:57:53.735443730 (26m39.197536203s) [noise=true , address=145.79.227.11:4750 , piece_id=CJXKOVDLJVEKLEMEN3RSBO7576W32WMA6C3KEJDX3GS3XIUQZFZA, seg_pos={0 0}, upload=388]: Done uploading piece. err: <nil> Sep 23 13:57:55.409099718 (26m40.871192202s) [noise=true , address=145.79.227.12:4775 , piece_id=JPYKKBJFFI3SUOFD44VKC2GJ32AO4H6BEMIOHWPOS5JR5SEYHQWQ, seg_pos={0 0}, upload=388]: Done uploading piece. err: <nil> Sep 23 13:58:05.387583288 (26m50.849675771s) [noise=true , address=145.79.227.2:4128 , piece_id=KG4A6OARKJFUDGHBIJSXOHGUQ2Y5ZTBEFXD576W5U5CTZQD36MPA, seg_pos={0 0}, upload=388]: Done uploading piece. err: <nil> Sep 23 13:58:11.832148818 (26m57.294241291s) [noise=true , address=145.79.227.23:4578 , piece_id=U4TAYSO4IG5U634KPCNTWELXIXX6RFSFLJC7AP3FXRKE5R3Z2IVQ, seg_pos={0 0}, upload=388]: Done uploading piece. err: <nil> Sep 23 13:58:13.997613861 (26m59.459706344s) [noise=true , address=145.79.227.28:4759 , piece_id=7NBBUWOJ6E2VAP6L5J6VB6BGYRSQRF4Z3GH4ZWZ65GWTYARRE5AA, seg_pos={0 0}, upload=388]: Done uploading piece. err: <nil> Sep 23 13:58:14.705556905 (27m0.167649388s) [noise=true , address=145.79.227.40:4057 , piece_id=HFURC5BTIQ5W7V2X75CLUGJSMR5FEIZFIZZG3SZJNSDVOL62CGQQ, seg_pos={0 0}, upload=388]: Done uploading piece. err: <nil> Sep 23 13:58:15.200396601 (27m0.662489084s) [noise=true , address=145.79.227.46:4635 , piece_id=VSETY7Q3Q3O6QDIQBSEKHCI7MZHGC7LM4LLNDJXFUYUVIGZVLXUA, seg_pos={0 0}, upload=388]: Done uploading piece. err: <nil> Sep 23 13:58:18.011996926 (27m3.474089409s) [noise=true , address=145.79.227.38:4989 , piece_id=NWWLGSULG5LWUPDTD647XNP4U63IIFNMLQNTDGL5KKKUJBER4PIQ, seg_pos={0 0}, upload=388]: Done uploading piece. err: <nil> Sep 23 13:58:20.722114840 (27m6.184207324s) [noise=true , address=145.79.227.19:4091 , piece_id=YENXMGIO55UZH4XS2JIDETS73NB5I267ZNHUYA2NBPBTNICBKCTA, seg_pos={0 0}, upload=388]: Done uploading piece. err: <nil> Sep 23 13:58:21.718721462 (27m7.180813935s) [noise=true , address=145.79.227.49:4345 , piece_id=VXMMBEZQV5DGWJZEPSKFKOYLZIYSV3P75ITSCBEQIC2WW3W4IXGQ, seg_pos={0 0}, upload=388]: Done uploading piece. err: ecclient: upload failed (node:1KSTyRJ62wcMHx8K7uwqp5e8L8yhxH7V3YtBHnYu6QJ3K3bXpQ, address:145.79.227.49:4345): protocol: write tcp 188.166.225.170:32840->145.79.227.49:4345: use of closed network connection; write tcp 188.166.225.170:32840->145.79.227.49:4345: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32840->145.79.227.49:4345: use of closed network connection Sep 23 13:58:21.718824248 (27m7.180916731s) [noise=true , address=145.79.227.4:4217 , piece_id=4WAOONYJVFJKZTCIVX7GWOGGK5XDCP5TGQRKZ3SCI42KFRO6KHCQ, seg_pos={0 0}, upload=388]: Done uploading piece. err: ecclient: upload failed (node:123KLtoTMCQ9DZUjJ8JJab4D5bc7UGTt7yjVV8wDgFTozdZ5gpi, address:145.79.227.4:4217): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 13:58:21.719046873 (27m7.181139356s) [noise=true , address=145.79.227.6:4272 , piece_id=LLJGTG3KRKBKOKERTN5J6EBQ67TJVHLHU5XEYNQWRJFGBEBYG46Q, seg_pos={0 0}, upload=388]: Done uploading piece. err: ecclient: upload failed (node:1UeKsX3pQfhvCukBvJf1sekK7N6sXkaWaJnZzETKoxyXWo2BFi, address:145.79.227.6:4272): protocol: write tcp 188.166.225.170:57934->145.79.227.6:4272: use of closed network connection; write tcp 188.166.225.170:57934->145.79.227.6:4272: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57934->145.79.227.6:4272: use of closed network connection Sep 23 13:58:21.719060188 (27m7.181152671s) [noise=true , address=145.79.227.29:4948 , piece_id=Z6YHGVGSFELOEKNFPGIT6H72L5FTMCEEFPPUWB5MZOC5FIGKW4CA, seg_pos={0 0}, upload=388]: Done uploading piece. err: ecclient: upload failed (node:12vRsfmHG1SH4gL4RzmiUY9uZyF7uy3UfEWipuH4xaGnQ6NuDZL, address:145.79.227.29:4948): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 13:58:21.719144297 (27m7.18123678s) [noise=true , address=145.79.227.34:4916 , piece_id=KDJMSPO6JKDCXMVFYCLMJTCQW4I2TED6QDE6P3COCVLQ7SYVB33A, seg_pos={0 0}, upload=388]: Done uploading piece. err: ecclient: upload failed (node:12sbCx4R1g3vGDnXQSARG3UD5pwzqUqkE1q37CUzn2skJftjLEa, address:145.79.227.34:4916): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 13:58:21.719182551 (27m7.181275034s) [seg_pos={0 0}, upload=388]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 23 13:58:21.954649324 (27m7.416741817s) [upload=388]: Done uploading segments... Sep 23 13:58:21.954684411 (27m7.416776894s) [upload=388]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.040182815s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................X...............................X...] duration: 6.932028667s FAIL (2025-09-23 13:59:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 23 13:57:41.066095809 (26m26.528188282s) [upload=392]: Starting upload Sep 23 13:57:41.066165312 (26m26.528257795s) [upload=392]: Done starting upload Sep 23 13:57:41.066256055 (26m26.528348538s) [upload=392]: Uploading segments... Sep 23 13:57:41.066340375 (26m26.528432858s) [upload=392]: Got next segment. Inline: false Sep 23 13:57:41.066369190 (26m26.528461663s) [upload=392]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 13:57:41.066373588 (26m26.528466061s) [upload=392]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 13:57:41.334323455 (26m26.796415938s) [seg_pos={0 0}, upload=392]: Begin upload segment... Sep 23 13:57:41.334391965 (26m26.796484468s) [seg_pos={0 0}, upload=392]: Done begin upload segment. Sep 23 13:57:41.334419678 (26m26.796512151s) [noise=true , address=45.91.116.5:4299 , piece_id=6VPWCZE4LYS3LEJTDI6J3ZII5Y7VKT7WICQSOX6VGZEUKHNQMQIQ, seg_pos={0 0}, upload=392]: Uploading piece... Sep 23 13:57:41.334444845 (26m26.796537328s) [noise=true , address=45.91.116.6:4975 , piece_id=QH6BLGBHIMPCABQ6QJUFUGSA7WVSZHY6UBACBXBYZXB4FQUVYXXA, seg_pos={0 0}, upload=392]: Uploading piece... Sep 23 13:57:41.334540017 (26m26.7966325s) [noise=true , address=45.91.116.7:4010 , piece_id=BTHQKDPDVVGTQGWAISGEWBXJ7KOEHLQHADK3SK76VUV7QYCIDZPA, seg_pos={0 0}, upload=392]: Uploading piece... Sep 23 13:57:41.334903589 (26m26.796996072s) [noise=true , address=45.91.116.8:4048 , piece_id=L3VOQTF5RSICZ35GS64HB5JQMC45BKIE3KWFJ66WS3FWIJLFCFIA, seg_pos={0 0}, upload=392]: Uploading piece... Sep 23 13:57:41.335289525 (26m26.797382008s) [noise=true , address=145.79.227.15:4875 , piece_id=5POKXTCLKS2TK2MSI4ZCI57V2EZ23ZHLAYVGLUKQYLJCJWTCRARA, seg_pos={0 0}, upload=392]: Uploading piece... Sep 23 13:57:41.335676010 (26m26.797768503s) [noise=true , address=45.91.116.5:4297 , piece_id=YFHGAXC7WPOQVDZZ3X2ASRHGE2PR4P45SMLODZOPDCZT6CUTIDBQ, seg_pos={0 0}, upload=392]: Uploading piece... Sep 23 13:57:41.336061224 (26m26.798153697s) [noise=true , address=77.90.31.13:4626 , piece_id=FKHCFROBV34SLO2OPUPG2T5OQGX7R7R3WU7GM3BHVMSNYP5M3VEQ, seg_pos={0 0}, upload=392]: Uploading piece... Sep 23 13:57:41.336612884 (26m26.798705368s) [noise=true , address=45.91.116.10:4081 , piece_id=VKLZSZII7WSY24QSLARTNMFNUZAC5MNAQHZNJMXPCPBX6BFPYQNA, seg_pos={0 0}, upload=392]: Uploading piece... Sep 23 13:57:41.337061658 (26m26.799154141s) [noise=true , address=45.91.116.4:4938 , piece_id=HBNAWDXLER2TNVVEB3J25CCNIXOHZG2EUIF7TVDYIFYOZH7ONULQ, seg_pos={0 0}, upload=392]: Uploading piece... Sep 23 13:57:41.338297562 (26m26.800390035s) [noise=true , address=145.79.227.14:4837 , piece_id=VRFXSKCSCFC5LWOQUBT4KJ2ZGC7OG4NQKUOL5BJLR6YXD4RIN5CA, seg_pos={0 0}, upload=392]: Uploading piece... Sep 23 13:57:41.338439512 (26m26.800531995s) [noise=true , address=145.79.227.16:4912 , piece_id=QOR4BDUGHZ6TBE3CLPBXH42WI3MVKRWEIMQQ3OFQ2JBL5P77X3SA, seg_pos={0 0}, upload=392]: Uploading piece... Sep 23 13:57:41.339474333 (26m26.801566816s) [noise=true , address=77.90.31.16:4734 , piece_id=OJ2XMM4UF42YDDWXKJOHPGZHKKDUZVFOVOKELLWJNKSH2WF4GHNQ, seg_pos={0 0}, upload=392]: Uploading piece... Sep 23 13:57:41.339912316 (26m26.802004799s) [noise=true , address=145.79.227.13:4802 , piece_id=CXPEAOLANODVJWBQVBQCP2ROL36FMRNONI5UOGJNZVWUZC6EWBYA, seg_pos={0 0}, upload=392]: Uploading piece... Sep 23 13:57:41.340662545 (26m26.802755028s) [noise=true , address=145.79.227.4:4185 , piece_id=JRNF36ZUZP2USUIKUYXMO7JNK5FYEI4HERGROVUN523J4G6W4ACQ, seg_pos={0 0}, upload=392]: Uploading piece... Sep 23 13:57:41.341525990 (26m26.803618483s) [noise=true , address=77.90.31.14:4662 , piece_id=3FVBKW5U7NWOZCWA6PBXDCCUIF7YMPJETGEXXI75VHBD6XQITNDQ, seg_pos={0 0}, upload=392]: Uploading piece... Sep 23 13:57:41.346707496 (26m26.808799979s) [noise=true , address=45.91.116.13:4155 , piece_id=3YMVCTMI56N5USQ2XBO7QKBL74G6VU4O25BDNWNXIYNGZ3GOF7HQ, seg_pos={0 0}, upload=392]: Uploading piece... Sep 23 13:57:43.052151475 (26m28.514243968s) [upload=392]: Next returned nil segment Sep 23 13:57:43.052180601 (26m28.514273084s) [upload=392]: Waiting for error group managing segments... Sep 23 13:57:43.459300252 (26m28.921392735s) [noise=true , address=45.91.116.5:4299 , piece_id=6VPWCZE4LYS3LEJTDI6J3ZII5Y7VKT7WICQSOX6VGZEUKHNQMQIQ, seg_pos={0 0}, upload=392]: Done uploading piece. err: <nil> Sep 23 13:57:43.465051022 (26m28.927143495s) [noise=true , address=45.91.116.5:4297 , piece_id=YFHGAXC7WPOQVDZZ3X2ASRHGE2PR4P45SMLODZOPDCZT6CUTIDBQ, seg_pos={0 0}, upload=392]: Done uploading piece. err: <nil> Sep 23 13:57:43.482473177 (26m28.94456566s) [noise=true , address=45.91.116.8:4048 , piece_id=L3VOQTF5RSICZ35GS64HB5JQMC45BKIE3KWFJ66WS3FWIJLFCFIA, seg_pos={0 0}, upload=392]: Done uploading piece. err: <nil> Sep 23 13:57:43.488362281 (26m28.950454754s) [noise=true , address=77.90.31.14:4662 , piece_id=3FVBKW5U7NWOZCWA6PBXDCCUIF7YMPJETGEXXI75VHBD6XQITNDQ, seg_pos={0 0}, upload=392]: Done uploading piece. err: <nil> Sep 23 13:57:43.525020535 (26m28.987113009s) [noise=true , address=45.91.116.13:4155 , piece_id=3YMVCTMI56N5USQ2XBO7QKBL74G6VU4O25BDNWNXIYNGZ3GOF7HQ, seg_pos={0 0}, upload=392]: Done uploading piece. err: <nil> Sep 23 13:57:43.533065755 (26m28.995158238s) [noise=true , address=45.91.116.10:4081 , piece_id=VKLZSZII7WSY24QSLARTNMFNUZAC5MNAQHZNJMXPCPBX6BFPYQNA, seg_pos={0 0}, upload=392]: Done uploading piece. err: <nil> Sep 23 13:57:43.536199372 (26m28.998291845s) [noise=true , address=77.90.31.13:4626 , piece_id=FKHCFROBV34SLO2OPUPG2T5OQGX7R7R3WU7GM3BHVMSNYP5M3VEQ, seg_pos={0 0}, upload=392]: Done uploading piece. err: <nil> Sep 23 13:57:43.551487663 (26m29.013580146s) [noise=true , address=45.91.116.6:4975 , piece_id=QH6BLGBHIMPCABQ6QJUFUGSA7WVSZHY6UBACBXBYZXB4FQUVYXXA, seg_pos={0 0}, upload=392]: Done uploading piece. err: <nil> Sep 23 13:57:43.557996687 (26m29.020089171s) [noise=true , address=45.91.116.4:4938 , piece_id=HBNAWDXLER2TNVVEB3J25CCNIXOHZG2EUIF7TVDYIFYOZH7ONULQ, seg_pos={0 0}, upload=392]: Done uploading piece. err: <nil> Sep 23 13:57:43.633155514 (26m29.095247987s) [noise=true , address=45.91.116.7:4010 , piece_id=BTHQKDPDVVGTQGWAISGEWBXJ7KOEHLQHADK3SK76VUV7QYCIDZPA, seg_pos={0 0}, upload=392]: Done uploading piece. err: <nil> Sep 23 13:57:43.640850366 (26m29.102942839s) [noise=true , address=77.90.31.16:4734 , piece_id=OJ2XMM4UF42YDDWXKJOHPGZHKKDUZVFOVOKELLWJNKSH2WF4GHNQ, seg_pos={0 0}, upload=392]: Done uploading piece. err: <nil> Sep 23 13:58:01.032951537 (26m46.49504402s) [noise=true , address=145.79.227.4:4185 , piece_id=JRNF36ZUZP2USUIKUYXMO7JNK5FYEI4HERGROVUN523J4G6W4ACQ, seg_pos={0 0}, upload=392]: Done uploading piece. err: <nil> Sep 23 13:58:10.339466996 (26m55.801559469s) [noise=true , address=145.79.227.15:4875 , piece_id=5POKXTCLKS2TK2MSI4ZCI57V2EZ23ZHLAYVGLUKQYLJCJWTCRARA, seg_pos={0 0}, upload=392]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 13:58:10.339609988 (26m55.801702471s) [noise=true , address=145.79.227.13:4802 , piece_id=CXPEAOLANODVJWBQVBQCP2ROL36FMRNONI5UOGJNZVWUZC6EWBYA, seg_pos={0 0}, upload=392]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: write tcp 188.166.225.170:58416->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:58416->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58416->145.79.227.13:4802: use of closed network connection Sep 23 13:58:10.339706762 (26m55.801799235s) [noise=true , address=145.79.227.16:4912 , piece_id=QOR4BDUGHZ6TBE3CLPBXH42WI3MVKRWEIMQQ3OFQ2JBL5P77X3SA, seg_pos={0 0}, upload=392]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:35876->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:35876->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35876->145.79.227.16:4912: use of closed network connection Sep 23 13:58:10.339913025 (26m55.802005518s) [noise=true , address=145.79.227.14:4837 , piece_id=VRFXSKCSCFC5LWOQUBT4KJ2ZGC7OG4NQKUOL5BJLR6YXD4RIN5CA, seg_pos={0 0}, upload=392]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: write tcp 188.166.225.170:59930->145.79.227.14:4837: use of closed network connection; write tcp 188.166.225.170:59930->145.79.227.14:4837: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59930->145.79.227.14:4837: use of closed network connection Sep 23 13:58:10.339940838 (26m55.802033311s) [seg_pos={0 0}, upload=392]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 23 13:58:10.576077015 (26m56.038169508s) [upload=392]: Done uploading segments... Sep 23 13:58:10.576111120 (26m56.038203603s) [upload=392]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.932113558s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 6.021559883s FAIL (2025-09-23 13:58:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 88 (26m9.156611862s) [upload=388]: Starting upload Sep 23 13:57:23.694601205 (26m9.156693688s) [upload=388]: Done starting upload Sep 23 13:57:23.694812668 (26m9.156905151s) [upload=388]: Uploading segments... Sep 23 13:57:23.694953145 (26m9.157045658s) [upload=388]: Got next segment. Inline: false Sep 23 13:57:23.695037716 (26m9.157130199s) [upload=388]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 13:57:23.695045330 (26m9.157137803s) [upload=388]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 13:57:24.190620566 (26m9.652713049s) [seg_pos={0 0}, upload=388]: Begin upload segment... Sep 23 13:57:24.190689898 (26m9.652782381s) [seg_pos={0 0}, upload=388]: Done begin upload segment. Sep 23 13:57:24.190740063 (26m9.652832547s) [noise=true , address=145.79.227.19:4091 , piece_id=YENXMGIO55UZH4XS2JIDETS73NB5I267ZNHUYA2NBPBTNICBKCTA, seg_pos={0 0}, upload=388]: Uploading piece... Sep 23 13:57:24.190752077 (26m9.65284456s) [noise=true , address=145.79.227.29:4948 , piece_id=Z6YHGVGSFELOEKNFPGIT6H72L5FTMCEEFPPUWB5MZOC5FIGKW4CA, seg_pos={0 0}, upload=388]: Uploading piece... Sep 23 13:57:24.191368831 (26m9.653461334s) [noise=true , address=145.79.227.28:4759 , piece_id=7NBBUWOJ6E2VAP6L5J6VB6BGYRSQRF4Z3GH4ZWZ65GWTYARRE5AA, seg_pos={0 0}, upload=388]: Uploading piece... Sep 23 13:57:24.191515501 (26m9.653607984s) [noise=true , address=145.79.227.6:4272 , piece_id=LLJGTG3KRKBKOKERTN5J6EBQ67TJVHLHU5XEYNQWRJFGBEBYG46Q, seg_pos={0 0}, upload=388]: Uploading piece... Sep 23 13:57:24.191899572 (26m9.653992055s) [noise=true , address=77.90.31.11:4556 , piece_id=53GKZFMG4MQ72VHOL637G7UM7MNMYEFPPFGCP2UC2EAZGMOZHYDA, seg_pos={0 0}, upload=388]: Uploading piece... Sep 23 13:57:24.191983171 (26m9.654075644s) [noise=true , address=45.91.116.4:4966 , piece_id=QYNM4GKDHPYS7M4E4DYLB6QJS5SN22VA3YFUOB6YNKPMTJCUIEHQ, seg_pos={0 0}, upload=388]: Uploading piece... Sep 23 13:57:24.192324321 (26m9.654416804s) [noise=true , address=145.79.227.34:4916 , piece_id=KDJMSPO6JKDCXMVFYCLMJTCQW4I2TED6QDE6P3COCVLQ7SYVB33A, seg_pos={0 0}, upload=388]: Uploading piece... Sep 23 13:57:24.192458927 (26m9.65455141s) [noise=true , address=77.90.31.18:4824 , piece_id=M3DC45GZ576HMVLDUXX2O2RQLNTFEYOQOKLC2DSKNMHAPM5WHSMA, seg_pos={0 0}, upload=388]: Uploading piece... Sep 23 13:57:24.192696069 (26m9.654788552s) [noise=true , address=145.79.227.40:4057 , piece_id=HFURC5BTIQ5W7V2X75CLUGJSMR5FEIZFIZZG3SZJNSDVOL62CGQQ, seg_pos={0 0}, upload=388]: Uploading piece... Sep 23 13:57:24.192868898 (26m9.654961381s) [noise=true , address=145.79.227.12:4775 , piece_id=JPYKKBJFFI3SUOFD44VKC2GJ32AO4H6BEMIOHWPOS5JR5SEYHQWQ, seg_pos={0 0}, upload=388]: Uploading piece... Sep 23 13:57:24.193046977 (26m9.65513945s) [noise=true , address=145.79.227.49:4345 , piece_id=VXMMBEZQV5DGWJZEPSKFKOYLZIYSV3P75ITSCBEQIC2WW3W4IXGQ, seg_pos={0 0}, upload=388]: Uploading piece... Sep 23 13:57:24.193266214 (26m9.655358697s) [noise=true , address=145.79.227.23:4578 , piece_id=U4TAYSO4IG5U634KPCNTWELXIXX6RFSFLJC7AP3FXRKE5R3Z2IVQ, seg_pos={0 0}, upload=388]: Uploading piece... Sep 23 13:57:24.193645757 (26m9.65573824s) [noise=true , address=145.79.227.38:4989 , piece_id=NWWLGSULG5LWUPDTD647XNP4U63IIFNMLQNTDGL5KKKUJBER4PIQ, seg_pos={0 0}, upload=388]: Uploading piece... Sep 23 13:57:24.194120039 (26m9.656212513s) [noise=true , address=145.79.227.46:4635 , piece_id=VSETY7Q3Q3O6QDIQBSEKHCI7MZHGC7LM4LLNDJXFUYUVIGZVLXUA, seg_pos={0 0}, upload=388]: Uploading piece... Sep 23 13:57:24.194266850 (26m9.656359333s) [noise=true , address=145.79.227.2:4128 , piece_id=KG4A6OARKJFUDGHBIJSXOHGUQ2Y5ZTBEFXD576W5U5CTZQD36MPA, seg_pos={0 0}, upload=388]: Uploading piece... Sep 23 13:57:24.194628749 (26m9.656721222s) [noise=true , address=45.91.116.6:4987 , piece_id=KT2C25BRSWIBK2QAGRBRDJPPFPT7PYGY6K3ZLSCLNFTMR445QDIQ, seg_pos={0 0}, upload=388]: Uploading piece... Sep 23 13:57:24.195070971 (26m9.657163454s) [noise=true , address=145.79.227.4:4217 , piece_id=4WAOONYJVFJKZTCIVX7GWOGGK5XDCP5TGQRKZ3SCI42KFRO6KHCQ, seg_pos={0 0}, upload=388]: Uploading piece... Sep 23 13:57:24.195153838 (26m9.657246321s) [noise=true , address=145.79.227.11:4750 , piece_id=CJXKOVDLJVEKLEMEN3RSBO7576W32WMA6C3KEJDX3GS3XIUQZFZA, seg_pos={0 0}, upload=388]: Uploading piece... Sep 23 13:57:25.809063919 (26m11.271156392s) [upload=388]: Next returned nil segment Sep 23 13:57:25.809086972 (26m11.271179456s) [upload=388]: Waiting for error group managing segments... Sep 23 13:57:26.258715569 (26m11.720808052s) [noise=true , address=45.91.116.6:4987 , piece_id=KT2C25BRSWIBK2QAGRBRDJPPFPT7PYGY6K3ZLSCLNFTMR445QDIQ, seg_pos={0 0}, upload=388]: Done uploading piece. err: <nil> Sep 23 13:57:26.281240952 (26m11.743333435s) [noise=true , address=45.91.116.4:4966 , piece_id=QYNM4GKDHPYS7M4E4DYLB6QJS5SN22VA3YFUOB6YNKPMTJCUIEHQ, seg_pos={0 0}, upload=388]: Done uploading piece. err: <nil> Sep 23 13:57:26.289091760 (26m11.751184244s) [noise=true , address=77.90.31.11:4556 , piece_id=53GKZFMG4MQ72VHOL637G7UM7MNMYEFPPFGCP2UC2EAZGMOZHYDA, seg_pos={0 0}, upload=388]: Done uploading piece. err: <nil> Sep 23 13:57:26.311620449 (26m11.773712932s) [noise=true , address=77.90.31.18:4824 , piece_id=M3DC45GZ576HMVLDUXX2O2RQLNTFEYOQOKLC2DSKNMHAPM5WHSMA, seg_pos={0 0}, upload=388]: Done uploading piece. err: <nil> Sep 23 13:57:53.735443730 (26m39.197536203s) [noise=true , address=145.79.227.11:4750 , piece_id=CJXKOVDLJVEKLEMEN3RSBO7576W32WMA6C3KEJDX3GS3XIUQZFZA, seg_pos={0 0}, upload=388]: Done uploading piece. err: <nil> Sep 23 13:57:55.409099718 (26m40.871192202s) [noise=true , address=145.79.227.12:4775 , piece_id=JPYKKBJFFI3SUOFD44VKC2GJ32AO4H6BEMIOHWPOS5JR5SEYHQWQ, seg_pos={0 0}, upload=388]: Done uploading piece. err: <nil> Sep 23 13:58:05.387583288 (26m50.849675771s) [noise=true , address=145.79.227.2:4128 , piece_id=KG4A6OARKJFUDGHBIJSXOHGUQ2Y5ZTBEFXD576W5U5CTZQD36MPA, seg_pos={0 0}, upload=388]: Done uploading piece. err: <nil> Sep 23 13:58:11.832148818 (26m57.294241291s) [noise=true , address=145.79.227.23:4578 , piece_id=U4TAYSO4IG5U634KPCNTWELXIXX6RFSFLJC7AP3FXRKE5R3Z2IVQ, seg_pos={0 0}, upload=388]: Done uploading piece. err: <nil> Sep 23 13:58:13.997613861 (26m59.459706344s) [noise=true , address=145.79.227.28:4759 , piece_id=7NBBUWOJ6E2VAP6L5J6VB6BGYRSQRF4Z3GH4ZWZ65GWTYARRE5AA, seg_pos={0 0}, upload=388]: Done uploading piece. err: <nil> Sep 23 13:58:14.705556905 (27m0.167649388s) [noise=true , address=145.79.227.40:4057 , piece_id=HFURC5BTIQ5W7V2X75CLUGJSMR5FEIZFIZZG3SZJNSDVOL62CGQQ, seg_pos={0 0}, upload=388]: Done uploading piece. err: <nil> Sep 23 13:58:15.200396601 (27m0.662489084s) [noise=true , address=145.79.227.46:4635 , piece_id=VSETY7Q3Q3O6QDIQBSEKHCI7MZHGC7LM4LLNDJXFUYUVIGZVLXUA, seg_pos={0 0}, upload=388]: Done uploading piece. err: <nil> Sep 23 13:58:18.011996926 (27m3.474089409s) [noise=true , address=145.79.227.38:4989 , piece_id=NWWLGSULG5LWUPDTD647XNP4U63IIFNMLQNTDGL5KKKUJBER4PIQ, seg_pos={0 0}, upload=388]: Done uploading piece. err: <nil> Sep 23 13:58:20.722114840 (27m6.184207324s) [noise=true , address=145.79.227.19:4091 , piece_id=YENXMGIO55UZH4XS2JIDETS73NB5I267ZNHUYA2NBPBTNICBKCTA, seg_pos={0 0}, upload=388]: Done uploading piece. err: <nil> Sep 23 13:58:21.718721462 (27m7.180813935s) [noise=true , address=145.79.227.49:4345 , piece_id=VXMMBEZQV5DGWJZEPSKFKOYLZIYSV3P75ITSCBEQIC2WW3W4IXGQ, seg_pos={0 0}, upload=388]: Done uploading piece. err: ecclient: upload failed (node:1KSTyRJ62wcMHx8K7uwqp5e8L8yhxH7V3YtBHnYu6QJ3K3bXpQ, address:145.79.227.49:4345): protocol: write tcp 188.166.225.170:32840->145.79.227.49:4345: use of closed network connection; write tcp 188.166.225.170:32840->145.79.227.49:4345: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32840->145.79.227.49:4345: use of closed network connection Sep 23 13:58:21.718824248 (27m7.180916731s) [noise=true , address=145.79.227.4:4217 , piece_id=4WAOONYJVFJKZTCIVX7GWOGGK5XDCP5TGQRKZ3SCI42KFRO6KHCQ, seg_pos={0 0}, upload=388]: Done uploading piece. err: ecclient: upload failed (node:123KLtoTMCQ9DZUjJ8JJab4D5bc7UGTt7yjVV8wDgFTozdZ5gpi, address:145.79.227.4:4217): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 13:58:21.719046873 (27m7.181139356s) [noise=true , address=145.79.227.6:4272 , piece_id=LLJGTG3KRKBKOKERTN5J6EBQ67TJVHLHU5XEYNQWRJFGBEBYG46Q, seg_pos={0 0}, upload=388]: Done uploading piece. err: ecclient: upload failed (node:1UeKsX3pQfhvCukBvJf1sekK7N6sXkaWaJnZzETKoxyXWo2BFi, address:145.79.227.6:4272): protocol: write tcp 188.166.225.170:57934->145.79.227.6:4272: use of closed network connection; write tcp 188.166.225.170:57934->145.79.227.6:4272: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57934->145.79.227.6:4272: use of closed network connection Sep 23 13:58:21.719060188 (27m7.181152671s) [noise=true , address=145.79.227.29:4948 , piece_id=Z6YHGVGSFELOEKNFPGIT6H72L5FTMCEEFPPUWB5MZOC5FIGKW4CA, seg_pos={0 0}, upload=388]: Done uploading piece. err: ecclient: upload failed (node:12vRsfmHG1SH4gL4RzmiUY9uZyF7uy3UfEWipuH4xaGnQ6NuDZL, address:145.79.227.29:4948): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 13:58:21.719144297 (27m7.18123678s) [noise=true , address=145.79.227.34:4916 , piece_id=KDJMSPO6JKDCXMVFYCLMJTCQW4I2TED6QDE6P3COCVLQ7SYVB33A, seg_pos={0 0}, upload=388]: Done uploading piece. err: ecclient: upload failed (node:12sbCx4R1g3vGDnXQSARG3UD5pwzqUqkE1q37CUzn2skJftjLEa, address:145.79.227.34:4916): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 13:58:21.719182551 (27m7.181275034s) [seg_pos={0 0}, upload=388]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 23 13:58:21.954649324 (27m7.416741817s) [upload=388]: Done uploading segments... Sep 23 13:58:21.954684411 (27m7.416776894s) [upload=388]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.021589369s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................X...............................X..] duration: 6.191391223s FAIL (2025-09-23 13:58:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 23 13:57:41.066095809 (26m26.528188282s) [upload=392]: Starting upload Sep 23 13:57:41.066165312 (26m26.528257795s) [upload=392]: Done starting upload Sep 23 13:57:41.066256055 (26m26.528348538s) [upload=392]: Uploading segments... Sep 23 13:57:41.066340375 (26m26.528432858s) [upload=392]: Got next segment. Inline: false Sep 23 13:57:41.066369190 (26m26.528461663s) [upload=392]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 13:57:41.066373588 (26m26.528466061s) [upload=392]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 13:57:41.334323455 (26m26.796415938s) [seg_pos={0 0}, upload=392]: Begin upload segment... Sep 23 13:57:41.334391965 (26m26.796484468s) [seg_pos={0 0}, upload=392]: Done begin upload segment. Sep 23 13:57:41.334419678 (26m26.796512151s) [noise=true , address=45.91.116.5:4299 , piece_id=6VPWCZE4LYS3LEJTDI6J3ZII5Y7VKT7WICQSOX6VGZEUKHNQMQIQ, seg_pos={0 0}, upload=392]: Uploading piece... Sep 23 13:57:41.334444845 (26m26.796537328s) [noise=true , address=45.91.116.6:4975 , piece_id=QH6BLGBHIMPCABQ6QJUFUGSA7WVSZHY6UBACBXBYZXB4FQUVYXXA, seg_pos={0 0}, upload=392]: Uploading piece... Sep 23 13:57:41.334540017 (26m26.7966325s) [noise=true , address=45.91.116.7:4010 , piece_id=BTHQKDPDVVGTQGWAISGEWBXJ7KOEHLQHADK3SK76VUV7QYCIDZPA, seg_pos={0 0}, upload=392]: Uploading piece... Sep 23 13:57:41.334903589 (26m26.796996072s) [noise=true , address=45.91.116.8:4048 , piece_id=L3VOQTF5RSICZ35GS64HB5JQMC45BKIE3KWFJ66WS3FWIJLFCFIA, seg_pos={0 0}, upload=392]: Uploading piece... Sep 23 13:57:41.335289525 (26m26.797382008s) [noise=true , address=145.79.227.15:4875 , piece_id=5POKXTCLKS2TK2MSI4ZCI57V2EZ23ZHLAYVGLUKQYLJCJWTCRARA, seg_pos={0 0}, upload=392]: Uploading piece... Sep 23 13:57:41.335676010 (26m26.797768503s) [noise=true , address=45.91.116.5:4297 , piece_id=YFHGAXC7WPOQVDZZ3X2ASRHGE2PR4P45SMLODZOPDCZT6CUTIDBQ, seg_pos={0 0}, upload=392]: Uploading piece... Sep 23 13:57:41.336061224 (26m26.798153697s) [noise=true , address=77.90.31.13:4626 , piece_id=FKHCFROBV34SLO2OPUPG2T5OQGX7R7R3WU7GM3BHVMSNYP5M3VEQ, seg_pos={0 0}, upload=392]: Uploading piece... Sep 23 13:57:41.336612884 (26m26.798705368s) [noise=true , address=45.91.116.10:4081 , piece_id=VKLZSZII7WSY24QSLARTNMFNUZAC5MNAQHZNJMXPCPBX6BFPYQNA, seg_pos={0 0}, upload=392]: Uploading piece... Sep 23 13:57:41.337061658 (26m26.799154141s) [noise=true , address=45.91.116.4:4938 , piece_id=HBNAWDXLER2TNVVEB3J25CCNIXOHZG2EUIF7TVDYIFYOZH7ONULQ, seg_pos={0 0}, upload=392]: Uploading piece... Sep 23 13:57:41.338297562 (26m26.800390035s) [noise=true , address=145.79.227.14:4837 , piece_id=VRFXSKCSCFC5LWOQUBT4KJ2ZGC7OG4NQKUOL5BJLR6YXD4RIN5CA, seg_pos={0 0}, upload=392]: Uploading piece... Sep 23 13:57:41.338439512 (26m26.800531995s) [noise=true , address=145.79.227.16:4912 , piece_id=QOR4BDUGHZ6TBE3CLPBXH42WI3MVKRWEIMQQ3OFQ2JBL5P77X3SA, seg_pos={0 0}, upload=392]: Uploading piece... Sep 23 13:57:41.339474333 (26m26.801566816s) [noise=true , address=77.90.31.16:4734 , piece_id=OJ2XMM4UF42YDDWXKJOHPGZHKKDUZVFOVOKELLWJNKSH2WF4GHNQ, seg_pos={0 0}, upload=392]: Uploading piece... Sep 23 13:57:41.339912316 (26m26.802004799s) [noise=true , address=145.79.227.13:4802 , piece_id=CXPEAOLANODVJWBQVBQCP2ROL36FMRNONI5UOGJNZVWUZC6EWBYA, seg_pos={0 0}, upload=392]: Uploading piece... Sep 23 13:57:41.340662545 (26m26.802755028s) [noise=true , address=145.79.227.4:4185 , piece_id=JRNF36ZUZP2USUIKUYXMO7JNK5FYEI4HERGROVUN523J4G6W4ACQ, seg_pos={0 0}, upload=392]: Uploading piece... Sep 23 13:57:41.341525990 (26m26.803618483s) [noise=true , address=77.90.31.14:4662 , piece_id=3FVBKW5U7NWOZCWA6PBXDCCUIF7YMPJETGEXXI75VHBD6XQITNDQ, seg_pos={0 0}, upload=392]: Uploading piece... Sep 23 13:57:41.346707496 (26m26.808799979s) [noise=true , address=45.91.116.13:4155 , piece_id=3YMVCTMI56N5USQ2XBO7QKBL74G6VU4O25BDNWNXIYNGZ3GOF7HQ, seg_pos={0 0}, upload=392]: Uploading piece... Sep 23 13:57:43.052151475 (26m28.514243968s) [upload=392]: Next returned nil segment Sep 23 13:57:43.052180601 (26m28.514273084s) [upload=392]: Waiting for error group managing segments... Sep 23 13:57:43.459300252 (26m28.921392735s) [noise=true , address=45.91.116.5:4299 , piece_id=6VPWCZE4LYS3LEJTDI6J3ZII5Y7VKT7WICQSOX6VGZEUKHNQMQIQ, seg_pos={0 0}, upload=392]: Done uploading piece. err: <nil> Sep 23 13:57:43.465051022 (26m28.927143495s) [noise=true , address=45.91.116.5:4297 , piece_id=YFHGAXC7WPOQVDZZ3X2ASRHGE2PR4P45SMLODZOPDCZT6CUTIDBQ, seg_pos={0 0}, upload=392]: Done uploading piece. err: <nil> Sep 23 13:57:43.482473177 (26m28.94456566s) [noise=true , address=45.91.116.8:4048 , piece_id=L3VOQTF5RSICZ35GS64HB5JQMC45BKIE3KWFJ66WS3FWIJLFCFIA, seg_pos={0 0}, upload=392]: Done uploading piece. err: <nil> Sep 23 13:57:43.488362281 (26m28.950454754s) [noise=true , address=77.90.31.14:4662 , piece_id=3FVBKW5U7NWOZCWA6PBXDCCUIF7YMPJETGEXXI75VHBD6XQITNDQ, seg_pos={0 0}, upload=392]: Done uploading piece. err: <nil> Sep 23 13:57:43.525020535 (26m28.987113009s) [noise=true , address=45.91.116.13:4155 , piece_id=3YMVCTMI56N5USQ2XBO7QKBL74G6VU4O25BDNWNXIYNGZ3GOF7HQ, seg_pos={0 0}, upload=392]: Done uploading piece. err: <nil> Sep 23 13:57:43.533065755 (26m28.995158238s) [noise=true , address=45.91.116.10:4081 , piece_id=VKLZSZII7WSY24QSLARTNMFNUZAC5MNAQHZNJMXPCPBX6BFPYQNA, seg_pos={0 0}, upload=392]: Done uploading piece. err: <nil> Sep 23 13:57:43.536199372 (26m28.998291845s) [noise=true , address=77.90.31.13:4626 , piece_id=FKHCFROBV34SLO2OPUPG2T5OQGX7R7R3WU7GM3BHVMSNYP5M3VEQ, seg_pos={0 0}, upload=392]: Done uploading piece. err: <nil> Sep 23 13:57:43.551487663 (26m29.013580146s) [noise=true , address=45.91.116.6:4975 , piece_id=QH6BLGBHIMPCABQ6QJUFUGSA7WVSZHY6UBACBXBYZXB4FQUVYXXA, seg_pos={0 0}, upload=392]: Done uploading piece. err: <nil> Sep 23 13:57:43.557996687 (26m29.020089171s) [noise=true , address=45.91.116.4:4938 , piece_id=HBNAWDXLER2TNVVEB3J25CCNIXOHZG2EUIF7TVDYIFYOZH7ONULQ, seg_pos={0 0}, upload=392]: Done uploading piece. err: <nil> Sep 23 13:57:43.633155514 (26m29.095247987s) [noise=true , address=45.91.116.7:4010 , piece_id=BTHQKDPDVVGTQGWAISGEWBXJ7KOEHLQHADK3SK76VUV7QYCIDZPA, seg_pos={0 0}, upload=392]: Done uploading piece. err: <nil> Sep 23 13:57:43.640850366 (26m29.102942839s) [noise=true , address=77.90.31.16:4734 , piece_id=OJ2XMM4UF42YDDWXKJOHPGZHKKDUZVFOVOKELLWJNKSH2WF4GHNQ, seg_pos={0 0}, upload=392]: Done uploading piece. err: <nil> Sep 23 13:58:01.032951537 (26m46.49504402s) [noise=true , address=145.79.227.4:4185 , piece_id=JRNF36ZUZP2USUIKUYXMO7JNK5FYEI4HERGROVUN523J4G6W4ACQ, seg_pos={0 0}, upload=392]: Done uploading piece. err: <nil> Sep 23 13:58:10.339466996 (26m55.801559469s) [noise=true , address=145.79.227.15:4875 , piece_id=5POKXTCLKS2TK2MSI4ZCI57V2EZ23ZHLAYVGLUKQYLJCJWTCRARA, seg_pos={0 0}, upload=392]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 13:58:10.339609988 (26m55.801702471s) [noise=true , address=145.79.227.13:4802 , piece_id=CXPEAOLANODVJWBQVBQCP2ROL36FMRNONI5UOGJNZVWUZC6EWBYA, seg_pos={0 0}, upload=392]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: write tcp 188.166.225.170:58416->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:58416->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58416->145.79.227.13:4802: use of closed network connection Sep 23 13:58:10.339706762 (26m55.801799235s) [noise=true , address=145.79.227.16:4912 , piece_id=QOR4BDUGHZ6TBE3CLPBXH42WI3MVKRWEIMQQ3OFQ2JBL5P77X3SA, seg_pos={0 0}, upload=392]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:35876->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:35876->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35876->145.79.227.16:4912: use of closed network connection Sep 23 13:58:10.339913025 (26m55.802005518s) [noise=true , address=145.79.227.14:4837 , piece_id=VRFXSKCSCFC5LWOQUBT4KJ2ZGC7OG4NQKUOL5BJLR6YXD4RIN5CA, seg_pos={0 0}, upload=392]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: write tcp 188.166.225.170:59930->145.79.227.14:4837: use of closed network connection; write tcp 188.166.225.170:59930->145.79.227.14:4837: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59930->145.79.227.14:4837: use of closed network connection Sep 23 13:58:10.339940838 (26m55.802033311s) [seg_pos={0 0}, upload=392]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 23 13:58:10.576077015 (26m56.038169508s) [upload=392]: Done uploading segments... Sep 23 13:58:10.576111120 (26m56.038203603s) [upload=392]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.191419868s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 5.858189942s FAIL (2025-09-23 13:58:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 88 (26m9.156611862s) [upload=388]: Starting upload Sep 23 13:57:23.694601205 (26m9.156693688s) [upload=388]: Done starting upload Sep 23 13:57:23.694812668 (26m9.156905151s) [upload=388]: Uploading segments... Sep 23 13:57:23.694953145 (26m9.157045658s) [upload=388]: Got next segment. Inline: false Sep 23 13:57:23.695037716 (26m9.157130199s) [upload=388]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 13:57:23.695045330 (26m9.157137803s) [upload=388]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 13:57:24.190620566 (26m9.652713049s) [seg_pos={0 0}, upload=388]: Begin upload segment... Sep 23 13:57:24.190689898 (26m9.652782381s) [seg_pos={0 0}, upload=388]: Done begin upload segment. Sep 23 13:57:24.190740063 (26m9.652832547s) [noise=true , address=145.79.227.19:4091 , piece_id=YENXMGIO55UZH4XS2JIDETS73NB5I267ZNHUYA2NBPBTNICBKCTA, seg_pos={0 0}, upload=388]: Uploading piece... Sep 23 13:57:24.190752077 (26m9.65284456s) [noise=true , address=145.79.227.29:4948 , piece_id=Z6YHGVGSFELOEKNFPGIT6H72L5FTMCEEFPPUWB5MZOC5FIGKW4CA, seg_pos={0 0}, upload=388]: Uploading piece... Sep 23 13:57:24.191368831 (26m9.653461334s) [noise=true , address=145.79.227.28:4759 , piece_id=7NBBUWOJ6E2VAP6L5J6VB6BGYRSQRF4Z3GH4ZWZ65GWTYARRE5AA, seg_pos={0 0}, upload=388]: Uploading piece... Sep 23 13:57:24.191515501 (26m9.653607984s) [noise=true , address=145.79.227.6:4272 , piece_id=LLJGTG3KRKBKOKERTN5J6EBQ67TJVHLHU5XEYNQWRJFGBEBYG46Q, seg_pos={0 0}, upload=388]: Uploading piece... Sep 23 13:57:24.191899572 (26m9.653992055s) [noise=true , address=77.90.31.11:4556 , piece_id=53GKZFMG4MQ72VHOL637G7UM7MNMYEFPPFGCP2UC2EAZGMOZHYDA, seg_pos={0 0}, upload=388]: Uploading piece... Sep 23 13:57:24.191983171 (26m9.654075644s) [noise=true , address=45.91.116.4:4966 , piece_id=QYNM4GKDHPYS7M4E4DYLB6QJS5SN22VA3YFUOB6YNKPMTJCUIEHQ, seg_pos={0 0}, upload=388]: Uploading piece... Sep 23 13:57:24.192324321 (26m9.654416804s) [noise=true , address=145.79.227.34:4916 , piece_id=KDJMSPO6JKDCXMVFYCLMJTCQW4I2TED6QDE6P3COCVLQ7SYVB33A, seg_pos={0 0}, upload=388]: Uploading piece... Sep 23 13:57:24.192458927 (26m9.65455141s) [noise=true , address=77.90.31.18:4824 , piece_id=M3DC45GZ576HMVLDUXX2O2RQLNTFEYOQOKLC2DSKNMHAPM5WHSMA, seg_pos={0 0}, upload=388]: Uploading piece... Sep 23 13:57:24.192696069 (26m9.654788552s) [noise=true , address=145.79.227.40:4057 , piece_id=HFURC5BTIQ5W7V2X75CLUGJSMR5FEIZFIZZG3SZJNSDVOL62CGQQ, seg_pos={0 0}, upload=388]: Uploading piece... Sep 23 13:57:24.192868898 (26m9.654961381s) [noise=true , address=145.79.227.12:4775 , piece_id=JPYKKBJFFI3SUOFD44VKC2GJ32AO4H6BEMIOHWPOS5JR5SEYHQWQ, seg_pos={0 0}, upload=388]: Uploading piece... Sep 23 13:57:24.193046977 (26m9.65513945s) [noise=true , address=145.79.227.49:4345 , piece_id=VXMMBEZQV5DGWJZEPSKFKOYLZIYSV3P75ITSCBEQIC2WW3W4IXGQ, seg_pos={0 0}, upload=388]: Uploading piece... Sep 23 13:57:24.193266214 (26m9.655358697s) [noise=true , address=145.79.227.23:4578 , piece_id=U4TAYSO4IG5U634KPCNTWELXIXX6RFSFLJC7AP3FXRKE5R3Z2IVQ, seg_pos={0 0}, upload=388]: Uploading piece... Sep 23 13:57:24.193645757 (26m9.65573824s) [noise=true , address=145.79.227.38:4989 , piece_id=NWWLGSULG5LWUPDTD647XNP4U63IIFNMLQNTDGL5KKKUJBER4PIQ, seg_pos={0 0}, upload=388]: Uploading piece... Sep 23 13:57:24.194120039 (26m9.656212513s) [noise=true , address=145.79.227.46:4635 , piece_id=VSETY7Q3Q3O6QDIQBSEKHCI7MZHGC7LM4LLNDJXFUYUVIGZVLXUA, seg_pos={0 0}, upload=388]: Uploading piece... Sep 23 13:57:24.194266850 (26m9.656359333s) [noise=true , address=145.79.227.2:4128 , piece_id=KG4A6OARKJFUDGHBIJSXOHGUQ2Y5ZTBEFXD576W5U5CTZQD36MPA, seg_pos={0 0}, upload=388]: Uploading piece... Sep 23 13:57:24.194628749 (26m9.656721222s) [noise=true , address=45.91.116.6:4987 , piece_id=KT2C25BRSWIBK2QAGRBRDJPPFPT7PYGY6K3ZLSCLNFTMR445QDIQ, seg_pos={0 0}, upload=388]: Uploading piece... Sep 23 13:57:24.195070971 (26m9.657163454s) [noise=true , address=145.79.227.4:4217 , piece_id=4WAOONYJVFJKZTCIVX7GWOGGK5XDCP5TGQRKZ3SCI42KFRO6KHCQ, seg_pos={0 0}, upload=388]: Uploading piece... Sep 23 13:57:24.195153838 (26m9.657246321s) [noise=true , address=145.79.227.11:4750 , piece_id=CJXKOVDLJVEKLEMEN3RSBO7576W32WMA6C3KEJDX3GS3XIUQZFZA, seg_pos={0 0}, upload=388]: Uploading piece... Sep 23 13:57:25.809063919 (26m11.271156392s) [upload=388]: Next returned nil segment Sep 23 13:57:25.809086972 (26m11.271179456s) [upload=388]: Waiting for error group managing segments... Sep 23 13:57:26.258715569 (26m11.720808052s) [noise=true , address=45.91.116.6:4987 , piece_id=KT2C25BRSWIBK2QAGRBRDJPPFPT7PYGY6K3ZLSCLNFTMR445QDIQ, seg_pos={0 0}, upload=388]: Done uploading piece. err: <nil> Sep 23 13:57:26.281240952 (26m11.743333435s) [noise=true , address=45.91.116.4:4966 , piece_id=QYNM4GKDHPYS7M4E4DYLB6QJS5SN22VA3YFUOB6YNKPMTJCUIEHQ, seg_pos={0 0}, upload=388]: Done uploading piece. err: <nil> Sep 23 13:57:26.289091760 (26m11.751184244s) [noise=true , address=77.90.31.11:4556 , piece_id=53GKZFMG4MQ72VHOL637G7UM7MNMYEFPPFGCP2UC2EAZGMOZHYDA, seg_pos={0 0}, upload=388]: Done uploading piece. err: <nil> Sep 23 13:57:26.311620449 (26m11.773712932s) [noise=true , address=77.90.31.18:4824 , piece_id=M3DC45GZ576HMVLDUXX2O2RQLNTFEYOQOKLC2DSKNMHAPM5WHSMA, seg_pos={0 0}, upload=388]: Done uploading piece. err: <nil> Sep 23 13:57:53.735443730 (26m39.197536203s) [noise=true , address=145.79.227.11:4750 , piece_id=CJXKOVDLJVEKLEMEN3RSBO7576W32WMA6C3KEJDX3GS3XIUQZFZA, seg_pos={0 0}, upload=388]: Done uploading piece. err: <nil> Sep 23 13:57:55.409099718 (26m40.871192202s) [noise=true , address=145.79.227.12:4775 , piece_id=JPYKKBJFFI3SUOFD44VKC2GJ32AO4H6BEMIOHWPOS5JR5SEYHQWQ, seg_pos={0 0}, upload=388]: Done uploading piece. err: <nil> Sep 23 13:58:05.387583288 (26m50.849675771s) [noise=true , address=145.79.227.2:4128 , piece_id=KG4A6OARKJFUDGHBIJSXOHGUQ2Y5ZTBEFXD576W5U5CTZQD36MPA, seg_pos={0 0}, upload=388]: Done uploading piece. err: <nil> Sep 23 13:58:11.832148818 (26m57.294241291s) [noise=true , address=145.79.227.23:4578 , piece_id=U4TAYSO4IG5U634KPCNTWELXIXX6RFSFLJC7AP3FXRKE5R3Z2IVQ, seg_pos={0 0}, upload=388]: Done uploading piece. err: <nil> Sep 23 13:58:13.997613861 (26m59.459706344s) [noise=true , address=145.79.227.28:4759 , piece_id=7NBBUWOJ6E2VAP6L5J6VB6BGYRSQRF4Z3GH4ZWZ65GWTYARRE5AA, seg_pos={0 0}, upload=388]: Done uploading piece. err: <nil> Sep 23 13:58:14.705556905 (27m0.167649388s) [noise=true , address=145.79.227.40:4057 , piece_id=HFURC5BTIQ5W7V2X75CLUGJSMR5FEIZFIZZG3SZJNSDVOL62CGQQ, seg_pos={0 0}, upload=388]: Done uploading piece. err: <nil> Sep 23 13:58:15.200396601 (27m0.662489084s) [noise=true , address=145.79.227.46:4635 , piece_id=VSETY7Q3Q3O6QDIQBSEKHCI7MZHGC7LM4LLNDJXFUYUVIGZVLXUA, seg_pos={0 0}, upload=388]: Done uploading piece. err: <nil> Sep 23 13:58:18.011996926 (27m3.474089409s) [noise=true , address=145.79.227.38:4989 , piece_id=NWWLGSULG5LWUPDTD647XNP4U63IIFNMLQNTDGL5KKKUJBER4PIQ, seg_pos={0 0}, upload=388]: Done uploading piece. err: <nil> Sep 23 13:58:20.722114840 (27m6.184207324s) [noise=true , address=145.79.227.19:4091 , piece_id=YENXMGIO55UZH4XS2JIDETS73NB5I267ZNHUYA2NBPBTNICBKCTA, seg_pos={0 0}, upload=388]: Done uploading piece. err: <nil> Sep 23 13:58:21.718721462 (27m7.180813935s) [noise=true , address=145.79.227.49:4345 , piece_id=VXMMBEZQV5DGWJZEPSKFKOYLZIYSV3P75ITSCBEQIC2WW3W4IXGQ, seg_pos={0 0}, upload=388]: Done uploading piece. err: ecclient: upload failed (node:1KSTyRJ62wcMHx8K7uwqp5e8L8yhxH7V3YtBHnYu6QJ3K3bXpQ, address:145.79.227.49:4345): protocol: write tcp 188.166.225.170:32840->145.79.227.49:4345: use of closed network connection; write tcp 188.166.225.170:32840->145.79.227.49:4345: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32840->145.79.227.49:4345: use of closed network connection Sep 23 13:58:21.718824248 (27m7.180916731s) [noise=true , address=145.79.227.4:4217 , piece_id=4WAOONYJVFJKZTCIVX7GWOGGK5XDCP5TGQRKZ3SCI42KFRO6KHCQ, seg_pos={0 0}, upload=388]: Done uploading piece. err: ecclient: upload failed (node:123KLtoTMCQ9DZUjJ8JJab4D5bc7UGTt7yjVV8wDgFTozdZ5gpi, address:145.79.227.4:4217): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 13:58:21.719046873 (27m7.181139356s) [noise=true , address=145.79.227.6:4272 , piece_id=LLJGTG3KRKBKOKERTN5J6EBQ67TJVHLHU5XEYNQWRJFGBEBYG46Q, seg_pos={0 0}, upload=388]: Done uploading piece. err: ecclient: upload failed (node:1UeKsX3pQfhvCukBvJf1sekK7N6sXkaWaJnZzETKoxyXWo2BFi, address:145.79.227.6:4272): protocol: write tcp 188.166.225.170:57934->145.79.227.6:4272: use of closed network connection; write tcp 188.166.225.170:57934->145.79.227.6:4272: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57934->145.79.227.6:4272: use of closed network connection Sep 23 13:58:21.719060188 (27m7.181152671s) [noise=true , address=145.79.227.29:4948 , piece_id=Z6YHGVGSFELOEKNFPGIT6H72L5FTMCEEFPPUWB5MZOC5FIGKW4CA, seg_pos={0 0}, upload=388]: Done uploading piece. err: ecclient: upload failed (node:12vRsfmHG1SH4gL4RzmiUY9uZyF7uy3UfEWipuH4xaGnQ6NuDZL, address:145.79.227.29:4948): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 13:58:21.719144297 (27m7.18123678s) [noise=true , address=145.79.227.34:4916 , piece_id=KDJMSPO6JKDCXMVFYCLMJTCQW4I2TED6QDE6P3COCVLQ7SYVB33A, seg_pos={0 0}, upload=388]: Done uploading piece. err: ecclient: upload failed (node:12sbCx4R1g3vGDnXQSARG3UD5pwzqUqkE1q37CUzn2skJftjLEa, address:145.79.227.34:4916): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 13:58:21.719182551 (27m7.181275034s) [seg_pos={0 0}, upload=388]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 23 13:58:21.954649324 (27m7.416741817s) [upload=388]: Done uploading segments... Sep 23 13:58:21.954684411 (27m7.416776894s) [upload=388]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.858226131s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.237448663s FAIL (2025-09-23 13:58:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 88 (26m9.156611862s) [upload=388]: Starting upload Sep 23 13:57:23.694601205 (26m9.156693688s) [upload=388]: Done starting upload Sep 23 13:57:23.694812668 (26m9.156905151s) [upload=388]: Uploading segments... Sep 23 13:57:23.694953145 (26m9.157045658s) [upload=388]: Got next segment. Inline: false Sep 23 13:57:23.695037716 (26m9.157130199s) [upload=388]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 13:57:23.695045330 (26m9.157137803s) [upload=388]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 13:57:24.190620566 (26m9.652713049s) [seg_pos={0 0}, upload=388]: Begin upload segment... Sep 23 13:57:24.190689898 (26m9.652782381s) [seg_pos={0 0}, upload=388]: Done begin upload segment. Sep 23 13:57:24.190740063 (26m9.652832547s) [noise=true , address=145.79.227.19:4091 , piece_id=YENXMGIO55UZH4XS2JIDETS73NB5I267ZNHUYA2NBPBTNICBKCTA, seg_pos={0 0}, upload=388]: Uploading piece... Sep 23 13:57:24.190752077 (26m9.65284456s) [noise=true , address=145.79.227.29:4948 , piece_id=Z6YHGVGSFELOEKNFPGIT6H72L5FTMCEEFPPUWB5MZOC5FIGKW4CA, seg_pos={0 0}, upload=388]: Uploading piece... Sep 23 13:57:24.191368831 (26m9.653461334s) [noise=true , address=145.79.227.28:4759 , piece_id=7NBBUWOJ6E2VAP6L5J6VB6BGYRSQRF4Z3GH4ZWZ65GWTYARRE5AA, seg_pos={0 0}, upload=388]: Uploading piece... Sep 23 13:57:24.191515501 (26m9.653607984s) [noise=true , address=145.79.227.6:4272 , piece_id=LLJGTG3KRKBKOKERTN5J6EBQ67TJVHLHU5XEYNQWRJFGBEBYG46Q, seg_pos={0 0}, upload=388]: Uploading piece... Sep 23 13:57:24.191899572 (26m9.653992055s) [noise=true , address=77.90.31.11:4556 , piece_id=53GKZFMG4MQ72VHOL637G7UM7MNMYEFPPFGCP2UC2EAZGMOZHYDA, seg_pos={0 0}, upload=388]: Uploading piece... Sep 23 13:57:24.191983171 (26m9.654075644s) [noise=true , address=45.91.116.4:4966 , piece_id=QYNM4GKDHPYS7M4E4DYLB6QJS5SN22VA3YFUOB6YNKPMTJCUIEHQ, seg_pos={0 0}, upload=388]: Uploading piece... Sep 23 13:57:24.192324321 (26m9.654416804s) [noise=true , address=145.79.227.34:4916 , piece_id=KDJMSPO6JKDCXMVFYCLMJTCQW4I2TED6QDE6P3COCVLQ7SYVB33A, seg_pos={0 0}, upload=388]: Uploading piece... Sep 23 13:57:24.192458927 (26m9.65455141s) [noise=true , address=77.90.31.18:4824 , piece_id=M3DC45GZ576HMVLDUXX2O2RQLNTFEYOQOKLC2DSKNMHAPM5WHSMA, seg_pos={0 0}, upload=388]: Uploading piece... Sep 23 13:57:24.192696069 (26m9.654788552s) [noise=true , address=145.79.227.40:4057 , piece_id=HFURC5BTIQ5W7V2X75CLUGJSMR5FEIZFIZZG3SZJNSDVOL62CGQQ, seg_pos={0 0}, upload=388]: Uploading piece... Sep 23 13:57:24.192868898 (26m9.654961381s) [noise=true , address=145.79.227.12:4775 , piece_id=JPYKKBJFFI3SUOFD44VKC2GJ32AO4H6BEMIOHWPOS5JR5SEYHQWQ, seg_pos={0 0}, upload=388]: Uploading piece... Sep 23 13:57:24.193046977 (26m9.65513945s) [noise=true , address=145.79.227.49:4345 , piece_id=VXMMBEZQV5DGWJZEPSKFKOYLZIYSV3P75ITSCBEQIC2WW3W4IXGQ, seg_pos={0 0}, upload=388]: Uploading piece... Sep 23 13:57:24.193266214 (26m9.655358697s) [noise=true , address=145.79.227.23:4578 , piece_id=U4TAYSO4IG5U634KPCNTWELXIXX6RFSFLJC7AP3FXRKE5R3Z2IVQ, seg_pos={0 0}, upload=388]: Uploading piece... Sep 23 13:57:24.193645757 (26m9.65573824s) [noise=true , address=145.79.227.38:4989 , piece_id=NWWLGSULG5LWUPDTD647XNP4U63IIFNMLQNTDGL5KKKUJBER4PIQ, seg_pos={0 0}, upload=388]: Uploading piece... Sep 23 13:57:24.194120039 (26m9.656212513s) [noise=true , address=145.79.227.46:4635 , piece_id=VSETY7Q3Q3O6QDIQBSEKHCI7MZHGC7LM4LLNDJXFUYUVIGZVLXUA, seg_pos={0 0}, upload=388]: Uploading piece... Sep 23 13:57:24.194266850 (26m9.656359333s) [noise=true , address=145.79.227.2:4128 , piece_id=KG4A6OARKJFUDGHBIJSXOHGUQ2Y5ZTBEFXD576W5U5CTZQD36MPA, seg_pos={0 0}, upload=388]: Uploading piece... Sep 23 13:57:24.194628749 (26m9.656721222s) [noise=true , address=45.91.116.6:4987 , piece_id=KT2C25BRSWIBK2QAGRBRDJPPFPT7PYGY6K3ZLSCLNFTMR445QDIQ, seg_pos={0 0}, upload=388]: Uploading piece... Sep 23 13:57:24.195070971 (26m9.657163454s) [noise=true , address=145.79.227.4:4217 , piece_id=4WAOONYJVFJKZTCIVX7GWOGGK5XDCP5TGQRKZ3SCI42KFRO6KHCQ, seg_pos={0 0}, upload=388]: Uploading piece... Sep 23 13:57:24.195153838 (26m9.657246321s) [noise=true , address=145.79.227.11:4750 , piece_id=CJXKOVDLJVEKLEMEN3RSBO7576W32WMA6C3KEJDX3GS3XIUQZFZA, seg_pos={0 0}, upload=388]: Uploading piece... Sep 23 13:57:25.809063919 (26m11.271156392s) [upload=388]: Next returned nil segment Sep 23 13:57:25.809086972 (26m11.271179456s) [upload=388]: Waiting for error group managing segments... Sep 23 13:57:26.258715569 (26m11.720808052s) [noise=true , address=45.91.116.6:4987 , piece_id=KT2C25BRSWIBK2QAGRBRDJPPFPT7PYGY6K3ZLSCLNFTMR445QDIQ, seg_pos={0 0}, upload=388]: Done uploading piece. err: <nil> Sep 23 13:57:26.281240952 (26m11.743333435s) [noise=true , address=45.91.116.4:4966 , piece_id=QYNM4GKDHPYS7M4E4DYLB6QJS5SN22VA3YFUOB6YNKPMTJCUIEHQ, seg_pos={0 0}, upload=388]: Done uploading piece. err: <nil> Sep 23 13:57:26.289091760 (26m11.751184244s) [noise=true , address=77.90.31.11:4556 , piece_id=53GKZFMG4MQ72VHOL637G7UM7MNMYEFPPFGCP2UC2EAZGMOZHYDA, seg_pos={0 0}, upload=388]: Done uploading piece. err: <nil> Sep 23 13:57:26.311620449 (26m11.773712932s) [noise=true , address=77.90.31.18:4824 , piece_id=M3DC45GZ576HMVLDUXX2O2RQLNTFEYOQOKLC2DSKNMHAPM5WHSMA, seg_pos={0 0}, upload=388]: Done uploading piece. err: <nil> Sep 23 13:57:53.735443730 (26m39.197536203s) [noise=true , address=145.79.227.11:4750 , piece_id=CJXKOVDLJVEKLEMEN3RSBO7576W32WMA6C3KEJDX3GS3XIUQZFZA, seg_pos={0 0}, upload=388]: Done uploading piece. err: <nil> Sep 23 13:57:55.409099718 (26m40.871192202s) [noise=true , address=145.79.227.12:4775 , piece_id=JPYKKBJFFI3SUOFD44VKC2GJ32AO4H6BEMIOHWPOS5JR5SEYHQWQ, seg_pos={0 0}, upload=388]: Done uploading piece. err: <nil> Sep 23 13:58:05.387583288 (26m50.849675771s) [noise=true , address=145.79.227.2:4128 , piece_id=KG4A6OARKJFUDGHBIJSXOHGUQ2Y5ZTBEFXD576W5U5CTZQD36MPA, seg_pos={0 0}, upload=388]: Done uploading piece. err: <nil> Sep 23 13:58:11.832148818 (26m57.294241291s) [noise=true , address=145.79.227.23:4578 , piece_id=U4TAYSO4IG5U634KPCNTWELXIXX6RFSFLJC7AP3FXRKE5R3Z2IVQ, seg_pos={0 0}, upload=388]: Done uploading piece. err: <nil> Sep 23 13:58:13.997613861 (26m59.459706344s) [noise=true , address=145.79.227.28:4759 , piece_id=7NBBUWOJ6E2VAP6L5J6VB6BGYRSQRF4Z3GH4ZWZ65GWTYARRE5AA, seg_pos={0 0}, upload=388]: Done uploading piece. err: <nil> Sep 23 13:58:14.705556905 (27m0.167649388s) [noise=true , address=145.79.227.40:4057 , piece_id=HFURC5BTIQ5W7V2X75CLUGJSMR5FEIZFIZZG3SZJNSDVOL62CGQQ, seg_pos={0 0}, upload=388]: Done uploading piece. err: <nil> Sep 23 13:58:15.200396601 (27m0.662489084s) [noise=true , address=145.79.227.46:4635 , piece_id=VSETY7Q3Q3O6QDIQBSEKHCI7MZHGC7LM4LLNDJXFUYUVIGZVLXUA, seg_pos={0 0}, upload=388]: Done uploading piece. err: <nil> Sep 23 13:58:18.011996926 (27m3.474089409s) [noise=true , address=145.79.227.38:4989 , piece_id=NWWLGSULG5LWUPDTD647XNP4U63IIFNMLQNTDGL5KKKUJBER4PIQ, seg_pos={0 0}, upload=388]: Done uploading piece. err: <nil> Sep 23 13:58:20.722114840 (27m6.184207324s) [noise=true , address=145.79.227.19:4091 , piece_id=YENXMGIO55UZH4XS2JIDETS73NB5I267ZNHUYA2NBPBTNICBKCTA, seg_pos={0 0}, upload=388]: Done uploading piece. err: <nil> Sep 23 13:58:21.718721462 (27m7.180813935s) [noise=true , address=145.79.227.49:4345 , piece_id=VXMMBEZQV5DGWJZEPSKFKOYLZIYSV3P75ITSCBEQIC2WW3W4IXGQ, seg_pos={0 0}, upload=388]: Done uploading piece. err: ecclient: upload failed (node:1KSTyRJ62wcMHx8K7uwqp5e8L8yhxH7V3YtBHnYu6QJ3K3bXpQ, address:145.79.227.49:4345): protocol: write tcp 188.166.225.170:32840->145.79.227.49:4345: use of closed network connection; write tcp 188.166.225.170:32840->145.79.227.49:4345: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32840->145.79.227.49:4345: use of closed network connection Sep 23 13:58:21.718824248 (27m7.180916731s) [noise=true , address=145.79.227.4:4217 , piece_id=4WAOONYJVFJKZTCIVX7GWOGGK5XDCP5TGQRKZ3SCI42KFRO6KHCQ, seg_pos={0 0}, upload=388]: Done uploading piece. err: ecclient: upload failed (node:123KLtoTMCQ9DZUjJ8JJab4D5bc7UGTt7yjVV8wDgFTozdZ5gpi, address:145.79.227.4:4217): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 13:58:21.719046873 (27m7.181139356s) [noise=true , address=145.79.227.6:4272 , piece_id=LLJGTG3KRKBKOKERTN5J6EBQ67TJVHLHU5XEYNQWRJFGBEBYG46Q, seg_pos={0 0}, upload=388]: Done uploading piece. err: ecclient: upload failed (node:1UeKsX3pQfhvCukBvJf1sekK7N6sXkaWaJnZzETKoxyXWo2BFi, address:145.79.227.6:4272): protocol: write tcp 188.166.225.170:57934->145.79.227.6:4272: use of closed network connection; write tcp 188.166.225.170:57934->145.79.227.6:4272: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57934->145.79.227.6:4272: use of closed network connection Sep 23 13:58:21.719060188 (27m7.181152671s) [noise=true , address=145.79.227.29:4948 , piece_id=Z6YHGVGSFELOEKNFPGIT6H72L5FTMCEEFPPUWB5MZOC5FIGKW4CA, seg_pos={0 0}, upload=388]: Done uploading piece. err: ecclient: upload failed (node:12vRsfmHG1SH4gL4RzmiUY9uZyF7uy3UfEWipuH4xaGnQ6NuDZL, address:145.79.227.29:4948): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 13:58:21.719144297 (27m7.18123678s) [noise=true , address=145.79.227.34:4916 , piece_id=KDJMSPO6JKDCXMVFYCLMJTCQW4I2TED6QDE6P3COCVLQ7SYVB33A, seg_pos={0 0}, upload=388]: Done uploading piece. err: ecclient: upload failed (node:12sbCx4R1g3vGDnXQSARG3UD5pwzqUqkE1q37CUzn2skJftjLEa, address:145.79.227.34:4916): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 13:58:21.719182551 (27m7.181275034s) [seg_pos={0 0}, upload=388]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 23 13:58:21.954649324 (27m7.416741817s) [upload=388]: Done uploading segments... Sep 23 13:58:21.954684411 (27m7.416776894s) [upload=388]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.237476566s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................X...............................X.] duration: 9.431361352s FAIL (2025-09-23 13:58:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 23 13:57:41.066095809 (26m26.528188282s) [upload=392]: Starting upload Sep 23 13:57:41.066165312 (26m26.528257795s) [upload=392]: Done starting upload Sep 23 13:57:41.066256055 (26m26.528348538s) [upload=392]: Uploading segments... Sep 23 13:57:41.066340375 (26m26.528432858s) [upload=392]: Got next segment. Inline: false Sep 23 13:57:41.066369190 (26m26.528461663s) [upload=392]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 13:57:41.066373588 (26m26.528466061s) [upload=392]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 13:57:41.334323455 (26m26.796415938s) [seg_pos={0 0}, upload=392]: Begin upload segment... Sep 23 13:57:41.334391965 (26m26.796484468s) [seg_pos={0 0}, upload=392]: Done begin upload segment. Sep 23 13:57:41.334419678 (26m26.796512151s) [noise=true , address=45.91.116.5:4299 , piece_id=6VPWCZE4LYS3LEJTDI6J3ZII5Y7VKT7WICQSOX6VGZEUKHNQMQIQ, seg_pos={0 0}, upload=392]: Uploading piece... Sep 23 13:57:41.334444845 (26m26.796537328s) [noise=true , address=45.91.116.6:4975 , piece_id=QH6BLGBHIMPCABQ6QJUFUGSA7WVSZHY6UBACBXBYZXB4FQUVYXXA, seg_pos={0 0}, upload=392]: Uploading piece... Sep 23 13:57:41.334540017 (26m26.7966325s) [noise=true , address=45.91.116.7:4010 , piece_id=BTHQKDPDVVGTQGWAISGEWBXJ7KOEHLQHADK3SK76VUV7QYCIDZPA, seg_pos={0 0}, upload=392]: Uploading piece... Sep 23 13:57:41.334903589 (26m26.796996072s) [noise=true , address=45.91.116.8:4048 , piece_id=L3VOQTF5RSICZ35GS64HB5JQMC45BKIE3KWFJ66WS3FWIJLFCFIA, seg_pos={0 0}, upload=392]: Uploading piece... Sep 23 13:57:41.335289525 (26m26.797382008s) [noise=true , address=145.79.227.15:4875 , piece_id=5POKXTCLKS2TK2MSI4ZCI57V2EZ23ZHLAYVGLUKQYLJCJWTCRARA, seg_pos={0 0}, upload=392]: Uploading piece... Sep 23 13:57:41.335676010 (26m26.797768503s) [noise=true , address=45.91.116.5:4297 , piece_id=YFHGAXC7WPOQVDZZ3X2ASRHGE2PR4P45SMLODZOPDCZT6CUTIDBQ, seg_pos={0 0}, upload=392]: Uploading piece... Sep 23 13:57:41.336061224 (26m26.798153697s) [noise=true , address=77.90.31.13:4626 , piece_id=FKHCFROBV34SLO2OPUPG2T5OQGX7R7R3WU7GM3BHVMSNYP5M3VEQ, seg_pos={0 0}, upload=392]: Uploading piece... Sep 23 13:57:41.336612884 (26m26.798705368s) [noise=true , address=45.91.116.10:4081 , piece_id=VKLZSZII7WSY24QSLARTNMFNUZAC5MNAQHZNJMXPCPBX6BFPYQNA, seg_pos={0 0}, upload=392]: Uploading piece... Sep 23 13:57:41.337061658 (26m26.799154141s) [noise=true , address=45.91.116.4:4938 , piece_id=HBNAWDXLER2TNVVEB3J25CCNIXOHZG2EUIF7TVDYIFYOZH7ONULQ, seg_pos={0 0}, upload=392]: Uploading piece... Sep 23 13:57:41.338297562 (26m26.800390035s) [noise=true , address=145.79.227.14:4837 , piece_id=VRFXSKCSCFC5LWOQUBT4KJ2ZGC7OG4NQKUOL5BJLR6YXD4RIN5CA, seg_pos={0 0}, upload=392]: Uploading piece... Sep 23 13:57:41.338439512 (26m26.800531995s) [noise=true , address=145.79.227.16:4912 , piece_id=QOR4BDUGHZ6TBE3CLPBXH42WI3MVKRWEIMQQ3OFQ2JBL5P77X3SA, seg_pos={0 0}, upload=392]: Uploading piece... Sep 23 13:57:41.339474333 (26m26.801566816s) [noise=true , address=77.90.31.16:4734 , piece_id=OJ2XMM4UF42YDDWXKJOHPGZHKKDUZVFOVOKELLWJNKSH2WF4GHNQ, seg_pos={0 0}, upload=392]: Uploading piece... Sep 23 13:57:41.339912316 (26m26.802004799s) [noise=true , address=145.79.227.13:4802 , piece_id=CXPEAOLANODVJWBQVBQCP2ROL36FMRNONI5UOGJNZVWUZC6EWBYA, seg_pos={0 0}, upload=392]: Uploading piece... Sep 23 13:57:41.340662545 (26m26.802755028s) [noise=true , address=145.79.227.4:4185 , piece_id=JRNF36ZUZP2USUIKUYXMO7JNK5FYEI4HERGROVUN523J4G6W4ACQ, seg_pos={0 0}, upload=392]: Uploading piece... Sep 23 13:57:41.341525990 (26m26.803618483s) [noise=true , address=77.90.31.14:4662 , piece_id=3FVBKW5U7NWOZCWA6PBXDCCUIF7YMPJETGEXXI75VHBD6XQITNDQ, seg_pos={0 0}, upload=392]: Uploading piece... Sep 23 13:57:41.346707496 (26m26.808799979s) [noise=true , address=45.91.116.13:4155 , piece_id=3YMVCTMI56N5USQ2XBO7QKBL74G6VU4O25BDNWNXIYNGZ3GOF7HQ, seg_pos={0 0}, upload=392]: Uploading piece... Sep 23 13:57:43.052151475 (26m28.514243968s) [upload=392]: Next returned nil segment Sep 23 13:57:43.052180601 (26m28.514273084s) [upload=392]: Waiting for error group managing segments... Sep 23 13:57:43.459300252 (26m28.921392735s) [noise=true , address=45.91.116.5:4299 , piece_id=6VPWCZE4LYS3LEJTDI6J3ZII5Y7VKT7WICQSOX6VGZEUKHNQMQIQ, seg_pos={0 0}, upload=392]: Done uploading piece. err: <nil> Sep 23 13:57:43.465051022 (26m28.927143495s) [noise=true , address=45.91.116.5:4297 , piece_id=YFHGAXC7WPOQVDZZ3X2ASRHGE2PR4P45SMLODZOPDCZT6CUTIDBQ, seg_pos={0 0}, upload=392]: Done uploading piece. err: <nil> Sep 23 13:57:43.482473177 (26m28.94456566s) [noise=true , address=45.91.116.8:4048 , piece_id=L3VOQTF5RSICZ35GS64HB5JQMC45BKIE3KWFJ66WS3FWIJLFCFIA, seg_pos={0 0}, upload=392]: Done uploading piece. err: <nil> Sep 23 13:57:43.488362281 (26m28.950454754s) [noise=true , address=77.90.31.14:4662 , piece_id=3FVBKW5U7NWOZCWA6PBXDCCUIF7YMPJETGEXXI75VHBD6XQITNDQ, seg_pos={0 0}, upload=392]: Done uploading piece. err: <nil> Sep 23 13:57:43.525020535 (26m28.987113009s) [noise=true , address=45.91.116.13:4155 , piece_id=3YMVCTMI56N5USQ2XBO7QKBL74G6VU4O25BDNWNXIYNGZ3GOF7HQ, seg_pos={0 0}, upload=392]: Done uploading piece. err: <nil> Sep 23 13:57:43.533065755 (26m28.995158238s) [noise=true , address=45.91.116.10:4081 , piece_id=VKLZSZII7WSY24QSLARTNMFNUZAC5MNAQHZNJMXPCPBX6BFPYQNA, seg_pos={0 0}, upload=392]: Done uploading piece. err: <nil> Sep 23 13:57:43.536199372 (26m28.998291845s) [noise=true , address=77.90.31.13:4626 , piece_id=FKHCFROBV34SLO2OPUPG2T5OQGX7R7R3WU7GM3BHVMSNYP5M3VEQ, seg_pos={0 0}, upload=392]: Done uploading piece. err: <nil> Sep 23 13:57:43.551487663 (26m29.013580146s) [noise=true , address=45.91.116.6:4975 , piece_id=QH6BLGBHIMPCABQ6QJUFUGSA7WVSZHY6UBACBXBYZXB4FQUVYXXA, seg_pos={0 0}, upload=392]: Done uploading piece. err: <nil> Sep 23 13:57:43.557996687 (26m29.020089171s) [noise=true , address=45.91.116.4:4938 , piece_id=HBNAWDXLER2TNVVEB3J25CCNIXOHZG2EUIF7TVDYIFYOZH7ONULQ, seg_pos={0 0}, upload=392]: Done uploading piece. err: <nil> Sep 23 13:57:43.633155514 (26m29.095247987s) [noise=true , address=45.91.116.7:4010 , piece_id=BTHQKDPDVVGTQGWAISGEWBXJ7KOEHLQHADK3SK76VUV7QYCIDZPA, seg_pos={0 0}, upload=392]: Done uploading piece. err: <nil> Sep 23 13:57:43.640850366 (26m29.102942839s) [noise=true , address=77.90.31.16:4734 , piece_id=OJ2XMM4UF42YDDWXKJOHPGZHKKDUZVFOVOKELLWJNKSH2WF4GHNQ, seg_pos={0 0}, upload=392]: Done uploading piece. err: <nil> Sep 23 13:58:01.032951537 (26m46.49504402s) [noise=true , address=145.79.227.4:4185 , piece_id=JRNF36ZUZP2USUIKUYXMO7JNK5FYEI4HERGROVUN523J4G6W4ACQ, seg_pos={0 0}, upload=392]: Done uploading piece. err: <nil> Sep 23 13:58:10.339466996 (26m55.801559469s) [noise=true , address=145.79.227.15:4875 , piece_id=5POKXTCLKS2TK2MSI4ZCI57V2EZ23ZHLAYVGLUKQYLJCJWTCRARA, seg_pos={0 0}, upload=392]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 13:58:10.339609988 (26m55.801702471s) [noise=true , address=145.79.227.13:4802 , piece_id=CXPEAOLANODVJWBQVBQCP2ROL36FMRNONI5UOGJNZVWUZC6EWBYA, seg_pos={0 0}, upload=392]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: write tcp 188.166.225.170:58416->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:58416->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58416->145.79.227.13:4802: use of closed network connection Sep 23 13:58:10.339706762 (26m55.801799235s) [noise=true , address=145.79.227.16:4912 , piece_id=QOR4BDUGHZ6TBE3CLPBXH42WI3MVKRWEIMQQ3OFQ2JBL5P77X3SA, seg_pos={0 0}, upload=392]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:35876->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:35876->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35876->145.79.227.16:4912: use of closed network connection Sep 23 13:58:10.339913025 (26m55.802005518s) [noise=true , address=145.79.227.14:4837 , piece_id=VRFXSKCSCFC5LWOQUBT4KJ2ZGC7OG4NQKUOL5BJLR6YXD4RIN5CA, seg_pos={0 0}, upload=392]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: write tcp 188.166.225.170:59930->145.79.227.14:4837: use of closed network connection; write tcp 188.166.225.170:59930->145.79.227.14:4837: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59930->145.79.227.14:4837: use of closed network connection Sep 23 13:58:10.339940838 (26m55.802033311s) [seg_pos={0 0}, upload=392]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 23 13:58:10.576077015 (26m56.038169508s) [upload=392]: Done uploading segments... Sep 23 13:58:10.576111120 (26m56.038203603s) [upload=392]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 9.431379206s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................X...............................X] duration: 30.238296782s FAIL (2025-09-23 13:58:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 23 13:57:41.066095809 (26m26.528188282s) [upload=392]: Starting upload Sep 23 13:57:41.066165312 (26m26.528257795s) [upload=392]: Done starting upload Sep 23 13:57:41.066256055 (26m26.528348538s) [upload=392]: Uploading segments... Sep 23 13:57:41.066340375 (26m26.528432858s) [upload=392]: Got next segment. Inline: false Sep 23 13:57:41.066369190 (26m26.528461663s) [upload=392]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 13:57:41.066373588 (26m26.528466061s) [upload=392]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 13:57:41.334323455 (26m26.796415938s) [seg_pos={0 0}, upload=392]: Begin upload segment... Sep 23 13:57:41.334391965 (26m26.796484468s) [seg_pos={0 0}, upload=392]: Done begin upload segment. Sep 23 13:57:41.334419678 (26m26.796512151s) [noise=true , address=45.91.116.5:4299 , piece_id=6VPWCZE4LYS3LEJTDI6J3ZII5Y7VKT7WICQSOX6VGZEUKHNQMQIQ, seg_pos={0 0}, upload=392]: Uploading piece... Sep 23 13:57:41.334444845 (26m26.796537328s) [noise=true , address=45.91.116.6:4975 , piece_id=QH6BLGBHIMPCABQ6QJUFUGSA7WVSZHY6UBACBXBYZXB4FQUVYXXA, seg_pos={0 0}, upload=392]: Uploading piece... Sep 23 13:57:41.334540017 (26m26.7966325s) [noise=true , address=45.91.116.7:4010 , piece_id=BTHQKDPDVVGTQGWAISGEWBXJ7KOEHLQHADK3SK76VUV7QYCIDZPA, seg_pos={0 0}, upload=392]: Uploading piece... Sep 23 13:57:41.334903589 (26m26.796996072s) [noise=true , address=45.91.116.8:4048 , piece_id=L3VOQTF5RSICZ35GS64HB5JQMC45BKIE3KWFJ66WS3FWIJLFCFIA, seg_pos={0 0}, upload=392]: Uploading piece... Sep 23 13:57:41.335289525 (26m26.797382008s) [noise=true , address=145.79.227.15:4875 , piece_id=5POKXTCLKS2TK2MSI4ZCI57V2EZ23ZHLAYVGLUKQYLJCJWTCRARA, seg_pos={0 0}, upload=392]: Uploading piece... Sep 23 13:57:41.335676010 (26m26.797768503s) [noise=true , address=45.91.116.5:4297 , piece_id=YFHGAXC7WPOQVDZZ3X2ASRHGE2PR4P45SMLODZOPDCZT6CUTIDBQ, seg_pos={0 0}, upload=392]: Uploading piece... Sep 23 13:57:41.336061224 (26m26.798153697s) [noise=true , address=77.90.31.13:4626 , piece_id=FKHCFROBV34SLO2OPUPG2T5OQGX7R7R3WU7GM3BHVMSNYP5M3VEQ, seg_pos={0 0}, upload=392]: Uploading piece... Sep 23 13:57:41.336612884 (26m26.798705368s) [noise=true , address=45.91.116.10:4081 , piece_id=VKLZSZII7WSY24QSLARTNMFNUZAC5MNAQHZNJMXPCPBX6BFPYQNA, seg_pos={0 0}, upload=392]: Uploading piece... Sep 23 13:57:41.337061658 (26m26.799154141s) [noise=true , address=45.91.116.4:4938 , piece_id=HBNAWDXLER2TNVVEB3J25CCNIXOHZG2EUIF7TVDYIFYOZH7ONULQ, seg_pos={0 0}, upload=392]: Uploading piece... Sep 23 13:57:41.338297562 (26m26.800390035s) [noise=true , address=145.79.227.14:4837 , piece_id=VRFXSKCSCFC5LWOQUBT4KJ2ZGC7OG4NQKUOL5BJLR6YXD4RIN5CA, seg_pos={0 0}, upload=392]: Uploading piece... Sep 23 13:57:41.338439512 (26m26.800531995s) [noise=true , address=145.79.227.16:4912 , piece_id=QOR4BDUGHZ6TBE3CLPBXH42WI3MVKRWEIMQQ3OFQ2JBL5P77X3SA, seg_pos={0 0}, upload=392]: Uploading piece... Sep 23 13:57:41.339474333 (26m26.801566816s) [noise=true , address=77.90.31.16:4734 , piece_id=OJ2XMM4UF42YDDWXKJOHPGZHKKDUZVFOVOKELLWJNKSH2WF4GHNQ, seg_pos={0 0}, upload=392]: Uploading piece... Sep 23 13:57:41.339912316 (26m26.802004799s) [noise=true , address=145.79.227.13:4802 , piece_id=CXPEAOLANODVJWBQVBQCP2ROL36FMRNONI5UOGJNZVWUZC6EWBYA, seg_pos={0 0}, upload=392]: Uploading piece... Sep 23 13:57:41.340662545 (26m26.802755028s) [noise=true , address=145.79.227.4:4185 , piece_id=JRNF36ZUZP2USUIKUYXMO7JNK5FYEI4HERGROVUN523J4G6W4ACQ, seg_pos={0 0}, upload=392]: Uploading piece... Sep 23 13:57:41.341525990 (26m26.803618483s) [noise=true , address=77.90.31.14:4662 , piece_id=3FVBKW5U7NWOZCWA6PBXDCCUIF7YMPJETGEXXI75VHBD6XQITNDQ, seg_pos={0 0}, upload=392]: Uploading piece... Sep 23 13:57:41.346707496 (26m26.808799979s) [noise=true , address=45.91.116.13:4155 , piece_id=3YMVCTMI56N5USQ2XBO7QKBL74G6VU4O25BDNWNXIYNGZ3GOF7HQ, seg_pos={0 0}, upload=392]: Uploading piece... Sep 23 13:57:43.052151475 (26m28.514243968s) [upload=392]: Next returned nil segment Sep 23 13:57:43.052180601 (26m28.514273084s) [upload=392]: Waiting for error group managing segments... Sep 23 13:57:43.459300252 (26m28.921392735s) [noise=true , address=45.91.116.5:4299 , piece_id=6VPWCZE4LYS3LEJTDI6J3ZII5Y7VKT7WICQSOX6VGZEUKHNQMQIQ, seg_pos={0 0}, upload=392]: Done uploading piece. err: <nil> Sep 23 13:57:43.465051022 (26m28.927143495s) [noise=true , address=45.91.116.5:4297 , piece_id=YFHGAXC7WPOQVDZZ3X2ASRHGE2PR4P45SMLODZOPDCZT6CUTIDBQ, seg_pos={0 0}, upload=392]: Done uploading piece. err: <nil> Sep 23 13:57:43.482473177 (26m28.94456566s) [noise=true , address=45.91.116.8:4048 , piece_id=L3VOQTF5RSICZ35GS64HB5JQMC45BKIE3KWFJ66WS3FWIJLFCFIA, seg_pos={0 0}, upload=392]: Done uploading piece. err: <nil> Sep 23 13:57:43.488362281 (26m28.950454754s) [noise=true , address=77.90.31.14:4662 , piece_id=3FVBKW5U7NWOZCWA6PBXDCCUIF7YMPJETGEXXI75VHBD6XQITNDQ, seg_pos={0 0}, upload=392]: Done uploading piece. err: <nil> Sep 23 13:57:43.525020535 (26m28.987113009s) [noise=true , address=45.91.116.13:4155 , piece_id=3YMVCTMI56N5USQ2XBO7QKBL74G6VU4O25BDNWNXIYNGZ3GOF7HQ, seg_pos={0 0}, upload=392]: Done uploading piece. err: <nil> Sep 23 13:57:43.533065755 (26m28.995158238s) [noise=true , address=45.91.116.10:4081 , piece_id=VKLZSZII7WSY24QSLARTNMFNUZAC5MNAQHZNJMXPCPBX6BFPYQNA, seg_pos={0 0}, upload=392]: Done uploading piece. err: <nil> Sep 23 13:57:43.536199372 (26m28.998291845s) [noise=true , address=77.90.31.13:4626 , piece_id=FKHCFROBV34SLO2OPUPG2T5OQGX7R7R3WU7GM3BHVMSNYP5M3VEQ, seg_pos={0 0}, upload=392]: Done uploading piece. err: <nil> Sep 23 13:57:43.551487663 (26m29.013580146s) [noise=true , address=45.91.116.6:4975 , piece_id=QH6BLGBHIMPCABQ6QJUFUGSA7WVSZHY6UBACBXBYZXB4FQUVYXXA, seg_pos={0 0}, upload=392]: Done uploading piece. err: <nil> Sep 23 13:57:43.557996687 (26m29.020089171s) [noise=true , address=45.91.116.4:4938 , piece_id=HBNAWDXLER2TNVVEB3J25CCNIXOHZG2EUIF7TVDYIFYOZH7ONULQ, seg_pos={0 0}, upload=392]: Done uploading piece. err: <nil> Sep 23 13:57:43.633155514 (26m29.095247987s) [noise=true , address=45.91.116.7:4010 , piece_id=BTHQKDPDVVGTQGWAISGEWBXJ7KOEHLQHADK3SK76VUV7QYCIDZPA, seg_pos={0 0}, upload=392]: Done uploading piece. err: <nil> Sep 23 13:57:43.640850366 (26m29.102942839s) [noise=true , address=77.90.31.16:4734 , piece_id=OJ2XMM4UF42YDDWXKJOHPGZHKKDUZVFOVOKELLWJNKSH2WF4GHNQ, seg_pos={0 0}, upload=392]: Done uploading piece. err: <nil> Sep 23 13:58:01.032951537 (26m46.49504402s) [noise=true , address=145.79.227.4:4185 , piece_id=JRNF36ZUZP2USUIKUYXMO7JNK5FYEI4HERGROVUN523J4G6W4ACQ, seg_pos={0 0}, upload=392]: Done uploading piece. err: <nil> Sep 23 13:58:10.339466996 (26m55.801559469s) [noise=true , address=145.79.227.15:4875 , piece_id=5POKXTCLKS2TK2MSI4ZCI57V2EZ23ZHLAYVGLUKQYLJCJWTCRARA, seg_pos={0 0}, upload=392]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 13:58:10.339609988 (26m55.801702471s) [noise=true , address=145.79.227.13:4802 , piece_id=CXPEAOLANODVJWBQVBQCP2ROL36FMRNONI5UOGJNZVWUZC6EWBYA, seg_pos={0 0}, upload=392]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: write tcp 188.166.225.170:58416->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:58416->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58416->145.79.227.13:4802: use of closed network connection Sep 23 13:58:10.339706762 (26m55.801799235s) [noise=true , address=145.79.227.16:4912 , piece_id=QOR4BDUGHZ6TBE3CLPBXH42WI3MVKRWEIMQQ3OFQ2JBL5P77X3SA, seg_pos={0 0}, upload=392]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:35876->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:35876->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35876->145.79.227.16:4912: use of closed network connection Sep 23 13:58:10.339913025 (26m55.802005518s) [noise=true , address=145.79.227.14:4837 , piece_id=VRFXSKCSCFC5LWOQUBT4KJ2ZGC7OG4NQKUOL5BJLR6YXD4RIN5CA, seg_pos={0 0}, upload=392]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: write tcp 188.166.225.170:59930->145.79.227.14:4837: use of closed network connection; write tcp 188.166.225.170:59930->145.79.227.14:4837: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59930->145.79.227.14:4837: use of closed network connection Sep 23 13:58:10.339940838 (26m55.802033311s) [seg_pos={0 0}, upload=392]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 23 13:58:10.576077015 (26m56.038169508s) [upload=392]: Done uploading segments... Sep 23 13:58:10.576111120 (26m56.038203603s) [upload=392]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.2383272s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 8.145024768s FAIL (2025-09-23 13:57:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: context deadline exceeded ======LOGS====== Sep 23 13:37:08.880516737 (5m54.34260922s) [upload=85]: Starting upload Sep 23 13:37:08.880590076 (5m54.342682549s) [upload=85]: Done starting upload Sep 23 13:37:08.880678955 (5m54.342771438s) [upload=85]: Uploading segments... Sep 23 13:37:08.880816417 (5m54.3429089s) [upload=85]: Got next segment. Inline: false Sep 23 13:37:08.880868837 (5m54.34296132s) [upload=85]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 13:37:08.880873937 (5m54.34296642s) [upload=85]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 13:37:09.155479775 (5m54.617572258s) [seg_pos={0 0}, upload=85]: Begin upload segment... Sep 23 13:37:09.155561942 (5m54.617654425s) [seg_pos={0 0}, upload=85]: Done begin upload segment. Sep 23 13:37:09.155609312 (5m54.617701795s) [noise=true , address=145.79.227.5:4221 , piece_id=37XI3ODHTNYYIW3TYSGEZKLL4REO4C7J7W4F3M54QPBS27HU2N7Q, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.155653065 (5m54.617745548s) [noise=true , address=145.79.227.13:4801 , piece_id=KDNEMJNYSDPBUV67TZ6PFMKXECG2TLUFSFX742MK5XRK32C2NHXA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156257627 (5m54.61835011s) [noise=true , address=45.91.116.4:4938 , piece_id=YXX7C7LQDTPKFEVXO7E6HN6QNM562OWWOXPI232ZHCLSY3WE4BUA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156274629 (5m54.618367102s) [noise=true , address=145.79.227.14:4839 , piece_id=65NEICBQJXMM6N6KBD5KGI5SFTVAV7NJJKBAET6HDGKI5VD325OQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156622501 (5m54.618715004s) [noise=true , address=145.79.227.6:4259 , piece_id=LPZNOJHDSME5TCQX2HQ544ZANJCJKGWIUXFLIOMNNSZBLYPNJTVA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156777296 (5m54.618869779s) [noise=true , address=145.79.227.16:4912 , piece_id=YLC5DB6H62T67W5LPTPQIQMWGJAN5ODYJB5PXIVX6X7JNFVCTNTA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156984401 (5m54.619076884s) [noise=true , address=145.79.227.14:4838 , piece_id=MG6N5Q7OOXD4OI6CV2VCSGIOOI6VB4GYFATAZ5WZWXFCZN6EFFLQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.157122624 (5m54.619215107s) [noise=true , address=45.91.116.10:4083 , piece_id=VONNFDRLV44LE65ZK7ENEJPNRMA2IPUGGGO3QGYJZLZLWOMNLFRQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.157345388 (5m54.619437871s) [noise=true , address=145.79.227.15:4876 , piece_id=ZHNZB3WQ4QQWR2LO6AQ64ONLIZ2EKCOBYGXFNR5FBY4CZL6QLVMA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.158372935 (5m54.620465408s) [noise=true , address=45.91.116.14:4189 , piece_id=HBFO3FNIMMF5UIDBXTFQQG6WZ4SFGJI3W6CSJ5EBAGN3D7JI6MSA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.159036498 (5m54.621128982s) [noise=true , address=45.91.116.11:4117 , piece_id=QEF7PSZ4ZAV2XM546G6QVKSJVH4JGCSYRAU572ABBMRGKVP23N6A, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.159800133 (5m54.621892616s) [noise=true , address=145.79.227.12:4765 , piece_id=H4PQGKP3XUGR6ARQ3UW3ZVPNCPS5WA5NM6FYJG5TBU3FALNXQYNA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.161637361 (5m54.623729844s) [noise=true , address=145.79.227.17:4970 , piece_id=IW4WTHW5W4SFOYMCXENR3DGYCA2OQJ4OO5RSXPYX6SUIGWKA4LBA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.163221126 (5m54.62531361s) [noise=true , address=77.90.31.11:4553 , piece_id=7XXKVSGNIYW5ZN6G5AV3JWP56QHF2HI4KOXJZ2DNOSGDHVZXDQTQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.163665793 (5m54.625758277s) [noise=true , address=145.79.227.11:4731 , piece_id=XR2GN3LWECQJ4OTSM44QXNW4B6GE7P5C5EFZFKJXD2M4GCMJ54VQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.164366027 (5m54.62645851s) [noise=true , address=145.79.227.12:4767 , piece_id=7JPOEJPRMBWO5MCNTISEDMBSYFACGTPJJUCBDX6EVYUKR3E7MYSA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:10.933255114 (5m56.395347597s) [upload=85]: Next returned nil segment Sep 23 13:37:10.933279932 (5m56.395372415s) [upload=85]: Waiting for error group managing segments... Sep 23 13:37:11.355919055 (5m56.818011548s) [noise=true , address=45.91.116.10:4083 , piece_id=VONNFDRLV44LE65ZK7ENEJPNRMA2IPUGGGO3QGYJZLZLWOMNLFRQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.366419560 (5m56.828512043s) [noise=true , address=45.91.116.4:4938 , piece_id=YXX7C7LQDTPKFEVXO7E6HN6QNM562OWWOXPI232ZHCLSY3WE4BUA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.373438585 (5m56.835531068s) [noise=true , address=45.91.116.11:4117 , piece_id=QEF7PSZ4ZAV2XM546G6QVKSJVH4JGCSYRAU572ABBMRGKVP23N6A, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.388282841 (5m56.850375324s) [noise=true , address=45.91.116.14:4189 , piece_id=HBFO3FNIMMF5UIDBXTFQQG6WZ4SFGJI3W6CSJ5EBAGN3D7JI6MSA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.396367205 (5m56.858459688s) [noise=true , address=77.90.31.11:4553 , piece_id=7XXKVSGNIYW5ZN6G5AV3JWP56QHF2HI4KOXJZ2DNOSGDHVZXDQTQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.050842867 (5m57.51293534s) [noise=true , address=145.79.227.13:4801 , piece_id=KDNEMJNYSDPBUV67TZ6PFMKXECG2TLUFSFX742MK5XRK32C2NHXA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.054995253 (5m57.517087736s) [noise=true , address=145.79.227.15:4876 , piece_id=ZHNZB3WQ4QQWR2LO6AQ64ONLIZ2EKCOBYGXFNR5FBY4CZL6QLVMA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.072213210 (5m57.534305693s) [noise=true , address=145.79.227.17:4970 , piece_id=IW4WTHW5W4SFOYMCXENR3DGYCA2OQJ4OO5RSXPYX6SUIGWKA4LBA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.101168265 (5m57.563260748s) [noise=true , address=145.79.227.6:4259 , piece_id=LPZNOJHDSME5TCQX2HQ544ZANJCJKGWIUXFLIOMNNSZBLYPNJTVA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.106130595 (5m57.568223078s) [noise=true , address=145.79.227.12:4765 , piece_id=H4PQGKP3XUGR6ARQ3UW3ZVPNCPS5WA5NM6FYJG5TBU3FALNXQYNA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.118733793 (5m57.580826277s) [noise=true , address=145.79.227.14:4839 , piece_id=65NEICBQJXMM6N6KBD5KGI5SFTVAV7NJJKBAET6HDGKI5VD325OQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.508815721 (5m57.970908204s) [noise=true , address=145.79.227.11:4731 , piece_id=XR2GN3LWECQJ4OTSM44QXNW4B6GE7P5C5EFZFKJXD2M4GCMJ54VQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:13.008461678 (5m58.470554161s) [noise=true , address=145.79.227.16:4912 , piece_id=YLC5DB6H62T67W5LPTPQIQMWGJAN5ODYJB5PXIVX6X7JNFVCTNTA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:13.008503868 (5m58.470596351s) [seg_pos={0 0}, upload=85]: Segment reached optimal threshold of 13 pieces. Sep 23 13:37:13.008675254 (5m58.470767738s) [noise=true , address=145.79.227.12:4767 , piece_id=7JPOEJPRMBWO5MCNTISEDMBSYFACGTPJJUCBDX6EVYUKR3E7MYSA, seg_pos={0 0}, upload=85]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5): protocol: write tcp 188.166.225.170:41718->145.79.227.12:4767: use of closed network connection; write tcp 188.166.225.170:41718->145.79.227.12:4767: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41718->145.79.227.12:4767: use of closed network connection Sep 23 13:37:13.008689582 (5m58.470782065s) [noise=true , address=145.79.227.12:4767 , piece_id=7JPOEJPRMBWO5MCNTISEDMBSYFACGTPJJUCBDX6EVYUKR3E7MYSA, seg_pos={0 0}, upload=85]: Long tail cancellation due to optimal threshold Sep 23 13:37:13.008703729 (5m58.470796212s) [noise=true , address=145.79.227.5:4221 , piece_id=37XI3ODHTNYYIW3TYSGEZKLL4REO4C7J7W4F3M54QPBS27HU2N7Q, seg_pos={0 0}, upload=85]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR): protocol: expected piece hash; context canceled; EOF Sep 23 13:37:13.008716322 (5m58.470808806s) [noise=true , address=145.79.227.5:4221 , piece_id=37XI3ODHTNYYIW3TYSGEZKLL4REO4C7J7W4F3M54QPBS27HU2N7Q, seg_pos={0 0}, upload=85]: Long tail cancellation due to optimal threshold Sep 23 13:37:13.008800553 (5m58.470893037s) [noise=true , address=145.79.227.14:4838 , piece_id=MG6N5Q7OOXD4OI6CV2VCSGIOOI6VB4GYFATAZ5WZWXFCZN6EFFLQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: expected piece hash; context canceled; EOF Sep 23 13:37:13.008809150 (5m58.470901633s) [noise=true , address=145.79.227.14:4838 , piece_id=MG6N5Q7OOXD4OI6CV2VCSGIOOI6VB4GYFATAZ5WZWXFCZN6EFFLQ, seg_pos={0 0}, upload=85]: Long tail cancellation due to optimal threshold Sep 23 13:37:13.008829819 (5m58.470922302s) [seg_pos={0 0}, upload=85]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 23 13:37:13.008908238 (5m58.471000721s) [upload=85]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 23 13:37:13.008914450 (5m58.471006923s) [upload=85]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 23 13:37:13.352201772 (5m58.814294255s) [upload=85]: Done uploading segments... Sep 23 13:37:13.352227461 (5m58.814319944s) [upload=85]: Upload finished. err: <nil> Duration: 8.14505284s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 11.706426412s FAIL (2025-09-23 13:56:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: context deadline exceeded ======LOGS====== Sep 23 13:37:08.880516737 (5m54.34260922s) [upload=85]: Starting upload Sep 23 13:37:08.880590076 (5m54.342682549s) [upload=85]: Done starting upload Sep 23 13:37:08.880678955 (5m54.342771438s) [upload=85]: Uploading segments... Sep 23 13:37:08.880816417 (5m54.3429089s) [upload=85]: Got next segment. Inline: false Sep 23 13:37:08.880868837 (5m54.34296132s) [upload=85]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 13:37:08.880873937 (5m54.34296642s) [upload=85]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 13:37:09.155479775 (5m54.617572258s) [seg_pos={0 0}, upload=85]: Begin upload segment... Sep 23 13:37:09.155561942 (5m54.617654425s) [seg_pos={0 0}, upload=85]: Done begin upload segment. Sep 23 13:37:09.155609312 (5m54.617701795s) [noise=true , address=145.79.227.5:4221 , piece_id=37XI3ODHTNYYIW3TYSGEZKLL4REO4C7J7W4F3M54QPBS27HU2N7Q, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.155653065 (5m54.617745548s) [noise=true , address=145.79.227.13:4801 , piece_id=KDNEMJNYSDPBUV67TZ6PFMKXECG2TLUFSFX742MK5XRK32C2NHXA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156257627 (5m54.61835011s) [noise=true , address=45.91.116.4:4938 , piece_id=YXX7C7LQDTPKFEVXO7E6HN6QNM562OWWOXPI232ZHCLSY3WE4BUA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156274629 (5m54.618367102s) [noise=true , address=145.79.227.14:4839 , piece_id=65NEICBQJXMM6N6KBD5KGI5SFTVAV7NJJKBAET6HDGKI5VD325OQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156622501 (5m54.618715004s) [noise=true , address=145.79.227.6:4259 , piece_id=LPZNOJHDSME5TCQX2HQ544ZANJCJKGWIUXFLIOMNNSZBLYPNJTVA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156777296 (5m54.618869779s) [noise=true , address=145.79.227.16:4912 , piece_id=YLC5DB6H62T67W5LPTPQIQMWGJAN5ODYJB5PXIVX6X7JNFVCTNTA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156984401 (5m54.619076884s) [noise=true , address=145.79.227.14:4838 , piece_id=MG6N5Q7OOXD4OI6CV2VCSGIOOI6VB4GYFATAZ5WZWXFCZN6EFFLQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.157122624 (5m54.619215107s) [noise=true , address=45.91.116.10:4083 , piece_id=VONNFDRLV44LE65ZK7ENEJPNRMA2IPUGGGO3QGYJZLZLWOMNLFRQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.157345388 (5m54.619437871s) [noise=true , address=145.79.227.15:4876 , piece_id=ZHNZB3WQ4QQWR2LO6AQ64ONLIZ2EKCOBYGXFNR5FBY4CZL6QLVMA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.158372935 (5m54.620465408s) [noise=true , address=45.91.116.14:4189 , piece_id=HBFO3FNIMMF5UIDBXTFQQG6WZ4SFGJI3W6CSJ5EBAGN3D7JI6MSA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.159036498 (5m54.621128982s) [noise=true , address=45.91.116.11:4117 , piece_id=QEF7PSZ4ZAV2XM546G6QVKSJVH4JGCSYRAU572ABBMRGKVP23N6A, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.159800133 (5m54.621892616s) [noise=true , address=145.79.227.12:4765 , piece_id=H4PQGKP3XUGR6ARQ3UW3ZVPNCPS5WA5NM6FYJG5TBU3FALNXQYNA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.161637361 (5m54.623729844s) [noise=true , address=145.79.227.17:4970 , piece_id=IW4WTHW5W4SFOYMCXENR3DGYCA2OQJ4OO5RSXPYX6SUIGWKA4LBA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.163221126 (5m54.62531361s) [noise=true , address=77.90.31.11:4553 , piece_id=7XXKVSGNIYW5ZN6G5AV3JWP56QHF2HI4KOXJZ2DNOSGDHVZXDQTQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.163665793 (5m54.625758277s) [noise=true , address=145.79.227.11:4731 , piece_id=XR2GN3LWECQJ4OTSM44QXNW4B6GE7P5C5EFZFKJXD2M4GCMJ54VQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.164366027 (5m54.62645851s) [noise=true , address=145.79.227.12:4767 , piece_id=7JPOEJPRMBWO5MCNTISEDMBSYFACGTPJJUCBDX6EVYUKR3E7MYSA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:10.933255114 (5m56.395347597s) [upload=85]: Next returned nil segment Sep 23 13:37:10.933279932 (5m56.395372415s) [upload=85]: Waiting for error group managing segments... Sep 23 13:37:11.355919055 (5m56.818011548s) [noise=true , address=45.91.116.10:4083 , piece_id=VONNFDRLV44LE65ZK7ENEJPNRMA2IPUGGGO3QGYJZLZLWOMNLFRQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.366419560 (5m56.828512043s) [noise=true , address=45.91.116.4:4938 , piece_id=YXX7C7LQDTPKFEVXO7E6HN6QNM562OWWOXPI232ZHCLSY3WE4BUA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.373438585 (5m56.835531068s) [noise=true , address=45.91.116.11:4117 , piece_id=QEF7PSZ4ZAV2XM546G6QVKSJVH4JGCSYRAU572ABBMRGKVP23N6A, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.388282841 (5m56.850375324s) [noise=true , address=45.91.116.14:4189 , piece_id=HBFO3FNIMMF5UIDBXTFQQG6WZ4SFGJI3W6CSJ5EBAGN3D7JI6MSA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.396367205 (5m56.858459688s) [noise=true , address=77.90.31.11:4553 , piece_id=7XXKVSGNIYW5ZN6G5AV3JWP56QHF2HI4KOXJZ2DNOSGDHVZXDQTQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.050842867 (5m57.51293534s) [noise=true , address=145.79.227.13:4801 , piece_id=KDNEMJNYSDPBUV67TZ6PFMKXECG2TLUFSFX742MK5XRK32C2NHXA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.054995253 (5m57.517087736s) [noise=true , address=145.79.227.15:4876 , piece_id=ZHNZB3WQ4QQWR2LO6AQ64ONLIZ2EKCOBYGXFNR5FBY4CZL6QLVMA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.072213210 (5m57.534305693s) [noise=true , address=145.79.227.17:4970 , piece_id=IW4WTHW5W4SFOYMCXENR3DGYCA2OQJ4OO5RSXPYX6SUIGWKA4LBA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.101168265 (5m57.563260748s) [noise=true , address=145.79.227.6:4259 , piece_id=LPZNOJHDSME5TCQX2HQ544ZANJCJKGWIUXFLIOMNNSZBLYPNJTVA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.106130595 (5m57.568223078s) [noise=true , address=145.79.227.12:4765 , piece_id=H4PQGKP3XUGR6ARQ3UW3ZVPNCPS5WA5NM6FYJG5TBU3FALNXQYNA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.118733793 (5m57.580826277s) [noise=true , address=145.79.227.14:4839 , piece_id=65NEICBQJXMM6N6KBD5KGI5SFTVAV7NJJKBAET6HDGKI5VD325OQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.508815721 (5m57.970908204s) [noise=true , address=145.79.227.11:4731 , piece_id=XR2GN3LWECQJ4OTSM44QXNW4B6GE7P5C5EFZFKJXD2M4GCMJ54VQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:13.008461678 (5m58.470554161s) [noise=true , address=145.79.227.16:4912 , piece_id=YLC5DB6H62T67W5LPTPQIQMWGJAN5ODYJB5PXIVX6X7JNFVCTNTA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:13.008503868 (5m58.470596351s) [seg_pos={0 0}, upload=85]: Segment reached optimal threshold of 13 pieces. Sep 23 13:37:13.008675254 (5m58.470767738s) [noise=true , address=145.79.227.12:4767 , piece_id=7JPOEJPRMBWO5MCNTISEDMBSYFACGTPJJUCBDX6EVYUKR3E7MYSA, seg_pos={0 0}, upload=85]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5): protocol: write tcp 188.166.225.170:41718->145.79.227.12:4767: use of closed network connection; write tcp 188.166.225.170:41718->145.79.227.12:4767: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41718->145.79.227.12:4767: use of closed network connection Sep 23 13:37:13.008689582 (5m58.470782065s) [noise=true , address=145.79.227.12:4767 , piece_id=7JPOEJPRMBWO5MCNTISEDMBSYFACGTPJJUCBDX6EVYUKR3E7MYSA, seg_pos={0 0}, upload=85]: Long tail cancellation due to optimal threshold Sep 23 13:37:13.008703729 (5m58.470796212s) [noise=true , address=145.79.227.5:4221 , piece_id=37XI3ODHTNYYIW3TYSGEZKLL4REO4C7J7W4F3M54QPBS27HU2N7Q, seg_pos={0 0}, upload=85]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR): protocol: expected piece hash; context canceled; EOF Sep 23 13:37:13.008716322 (5m58.470808806s) [noise=true , address=145.79.227.5:4221 , piece_id=37XI3ODHTNYYIW3TYSGEZKLL4REO4C7J7W4F3M54QPBS27HU2N7Q, seg_pos={0 0}, upload=85]: Long tail cancellation due to optimal threshold Sep 23 13:37:13.008800553 (5m58.470893037s) [noise=true , address=145.79.227.14:4838 , piece_id=MG6N5Q7OOXD4OI6CV2VCSGIOOI6VB4GYFATAZ5WZWXFCZN6EFFLQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: expected piece hash; context canceled; EOF Sep 23 13:37:13.008809150 (5m58.470901633s) [noise=true , address=145.79.227.14:4838 , piece_id=MG6N5Q7OOXD4OI6CV2VCSGIOOI6VB4GYFATAZ5WZWXFCZN6EFFLQ, seg_pos={0 0}, upload=85]: Long tail cancellation due to optimal threshold Sep 23 13:37:13.008829819 (5m58.470922302s) [seg_pos={0 0}, upload=85]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 23 13:37:13.008908238 (5m58.471000721s) [upload=85]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 23 13:37:13.008914450 (5m58.471006923s) [upload=85]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 23 13:37:13.352201772 (5m58.814294255s) [upload=85]: Done uploading segments... Sep 23 13:37:13.352227461 (5m58.814319944s) [upload=85]: Upload finished. err: <nil> Duration: 11.706460456s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 12.658073394s FAIL (2025-09-23 13:55:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: context deadline exceeded ======LOGS====== Sep 23 13:37:08.880516737 (5m54.34260922s) [upload=85]: Starting upload Sep 23 13:37:08.880590076 (5m54.342682549s) [upload=85]: Done starting upload Sep 23 13:37:08.880678955 (5m54.342771438s) [upload=85]: Uploading segments... Sep 23 13:37:08.880816417 (5m54.3429089s) [upload=85]: Got next segment. Inline: false Sep 23 13:37:08.880868837 (5m54.34296132s) [upload=85]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 13:37:08.880873937 (5m54.34296642s) [upload=85]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 13:37:09.155479775 (5m54.617572258s) [seg_pos={0 0}, upload=85]: Begin upload segment... Sep 23 13:37:09.155561942 (5m54.617654425s) [seg_pos={0 0}, upload=85]: Done begin upload segment. Sep 23 13:37:09.155609312 (5m54.617701795s) [noise=true , address=145.79.227.5:4221 , piece_id=37XI3ODHTNYYIW3TYSGEZKLL4REO4C7J7W4F3M54QPBS27HU2N7Q, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.155653065 (5m54.617745548s) [noise=true , address=145.79.227.13:4801 , piece_id=KDNEMJNYSDPBUV67TZ6PFMKXECG2TLUFSFX742MK5XRK32C2NHXA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156257627 (5m54.61835011s) [noise=true , address=45.91.116.4:4938 , piece_id=YXX7C7LQDTPKFEVXO7E6HN6QNM562OWWOXPI232ZHCLSY3WE4BUA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156274629 (5m54.618367102s) [noise=true , address=145.79.227.14:4839 , piece_id=65NEICBQJXMM6N6KBD5KGI5SFTVAV7NJJKBAET6HDGKI5VD325OQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156622501 (5m54.618715004s) [noise=true , address=145.79.227.6:4259 , piece_id=LPZNOJHDSME5TCQX2HQ544ZANJCJKGWIUXFLIOMNNSZBLYPNJTVA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156777296 (5m54.618869779s) [noise=true , address=145.79.227.16:4912 , piece_id=YLC5DB6H62T67W5LPTPQIQMWGJAN5ODYJB5PXIVX6X7JNFVCTNTA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156984401 (5m54.619076884s) [noise=true , address=145.79.227.14:4838 , piece_id=MG6N5Q7OOXD4OI6CV2VCSGIOOI6VB4GYFATAZ5WZWXFCZN6EFFLQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.157122624 (5m54.619215107s) [noise=true , address=45.91.116.10:4083 , piece_id=VONNFDRLV44LE65ZK7ENEJPNRMA2IPUGGGO3QGYJZLZLWOMNLFRQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.157345388 (5m54.619437871s) [noise=true , address=145.79.227.15:4876 , piece_id=ZHNZB3WQ4QQWR2LO6AQ64ONLIZ2EKCOBYGXFNR5FBY4CZL6QLVMA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.158372935 (5m54.620465408s) [noise=true , address=45.91.116.14:4189 , piece_id=HBFO3FNIMMF5UIDBXTFQQG6WZ4SFGJI3W6CSJ5EBAGN3D7JI6MSA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.159036498 (5m54.621128982s) [noise=true , address=45.91.116.11:4117 , piece_id=QEF7PSZ4ZAV2XM546G6QVKSJVH4JGCSYRAU572ABBMRGKVP23N6A, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.159800133 (5m54.621892616s) [noise=true , address=145.79.227.12:4765 , piece_id=H4PQGKP3XUGR6ARQ3UW3ZVPNCPS5WA5NM6FYJG5TBU3FALNXQYNA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.161637361 (5m54.623729844s) [noise=true , address=145.79.227.17:4970 , piece_id=IW4WTHW5W4SFOYMCXENR3DGYCA2OQJ4OO5RSXPYX6SUIGWKA4LBA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.163221126 (5m54.62531361s) [noise=true , address=77.90.31.11:4553 , piece_id=7XXKVSGNIYW5ZN6G5AV3JWP56QHF2HI4KOXJZ2DNOSGDHVZXDQTQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.163665793 (5m54.625758277s) [noise=true , address=145.79.227.11:4731 , piece_id=XR2GN3LWECQJ4OTSM44QXNW4B6GE7P5C5EFZFKJXD2M4GCMJ54VQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.164366027 (5m54.62645851s) [noise=true , address=145.79.227.12:4767 , piece_id=7JPOEJPRMBWO5MCNTISEDMBSYFACGTPJJUCBDX6EVYUKR3E7MYSA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:10.933255114 (5m56.395347597s) [upload=85]: Next returned nil segment Sep 23 13:37:10.933279932 (5m56.395372415s) [upload=85]: Waiting for error group managing segments... Sep 23 13:37:11.355919055 (5m56.818011548s) [noise=true , address=45.91.116.10:4083 , piece_id=VONNFDRLV44LE65ZK7ENEJPNRMA2IPUGGGO3QGYJZLZLWOMNLFRQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.366419560 (5m56.828512043s) [noise=true , address=45.91.116.4:4938 , piece_id=YXX7C7LQDTPKFEVXO7E6HN6QNM562OWWOXPI232ZHCLSY3WE4BUA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.373438585 (5m56.835531068s) [noise=true , address=45.91.116.11:4117 , piece_id=QEF7PSZ4ZAV2XM546G6QVKSJVH4JGCSYRAU572ABBMRGKVP23N6A, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.388282841 (5m56.850375324s) [noise=true , address=45.91.116.14:4189 , piece_id=HBFO3FNIMMF5UIDBXTFQQG6WZ4SFGJI3W6CSJ5EBAGN3D7JI6MSA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.396367205 (5m56.858459688s) [noise=true , address=77.90.31.11:4553 , piece_id=7XXKVSGNIYW5ZN6G5AV3JWP56QHF2HI4KOXJZ2DNOSGDHVZXDQTQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.050842867 (5m57.51293534s) [noise=true , address=145.79.227.13:4801 , piece_id=KDNEMJNYSDPBUV67TZ6PFMKXECG2TLUFSFX742MK5XRK32C2NHXA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.054995253 (5m57.517087736s) [noise=true , address=145.79.227.15:4876 , piece_id=ZHNZB3WQ4QQWR2LO6AQ64ONLIZ2EKCOBYGXFNR5FBY4CZL6QLVMA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.072213210 (5m57.534305693s) [noise=true , address=145.79.227.17:4970 , piece_id=IW4WTHW5W4SFOYMCXENR3DGYCA2OQJ4OO5RSXPYX6SUIGWKA4LBA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.101168265 (5m57.563260748s) [noise=true , address=145.79.227.6:4259 , piece_id=LPZNOJHDSME5TCQX2HQ544ZANJCJKGWIUXFLIOMNNSZBLYPNJTVA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.106130595 (5m57.568223078s) [noise=true , address=145.79.227.12:4765 , piece_id=H4PQGKP3XUGR6ARQ3UW3ZVPNCPS5WA5NM6FYJG5TBU3FALNXQYNA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.118733793 (5m57.580826277s) [noise=true , address=145.79.227.14:4839 , piece_id=65NEICBQJXMM6N6KBD5KGI5SFTVAV7NJJKBAET6HDGKI5VD325OQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.508815721 (5m57.970908204s) [noise=true , address=145.79.227.11:4731 , piece_id=XR2GN3LWECQJ4OTSM44QXNW4B6GE7P5C5EFZFKJXD2M4GCMJ54VQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:13.008461678 (5m58.470554161s) [noise=true , address=145.79.227.16:4912 , piece_id=YLC5DB6H62T67W5LPTPQIQMWGJAN5ODYJB5PXIVX6X7JNFVCTNTA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:13.008503868 (5m58.470596351s) [seg_pos={0 0}, upload=85]: Segment reached optimal threshold of 13 pieces. Sep 23 13:37:13.008675254 (5m58.470767738s) [noise=true , address=145.79.227.12:4767 , piece_id=7JPOEJPRMBWO5MCNTISEDMBSYFACGTPJJUCBDX6EVYUKR3E7MYSA, seg_pos={0 0}, upload=85]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5): protocol: write tcp 188.166.225.170:41718->145.79.227.12:4767: use of closed network connection; write tcp 188.166.225.170:41718->145.79.227.12:4767: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41718->145.79.227.12:4767: use of closed network connection Sep 23 13:37:13.008689582 (5m58.470782065s) [noise=true , address=145.79.227.12:4767 , piece_id=7JPOEJPRMBWO5MCNTISEDMBSYFACGTPJJUCBDX6EVYUKR3E7MYSA, seg_pos={0 0}, upload=85]: Long tail cancellation due to optimal threshold Sep 23 13:37:13.008703729 (5m58.470796212s) [noise=true , address=145.79.227.5:4221 , piece_id=37XI3ODHTNYYIW3TYSGEZKLL4REO4C7J7W4F3M54QPBS27HU2N7Q, seg_pos={0 0}, upload=85]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR): protocol: expected piece hash; context canceled; EOF Sep 23 13:37:13.008716322 (5m58.470808806s) [noise=true , address=145.79.227.5:4221 , piece_id=37XI3ODHTNYYIW3TYSGEZKLL4REO4C7J7W4F3M54QPBS27HU2N7Q, seg_pos={0 0}, upload=85]: Long tail cancellation due to optimal threshold Sep 23 13:37:13.008800553 (5m58.470893037s) [noise=true , address=145.79.227.14:4838 , piece_id=MG6N5Q7OOXD4OI6CV2VCSGIOOI6VB4GYFATAZ5WZWXFCZN6EFFLQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: expected piece hash; context canceled; EOF Sep 23 13:37:13.008809150 (5m58.470901633s) [noise=true , address=145.79.227.14:4838 , piece_id=MG6N5Q7OOXD4OI6CV2VCSGIOOI6VB4GYFATAZ5WZWXFCZN6EFFLQ, seg_pos={0 0}, upload=85]: Long tail cancellation due to optimal threshold Sep 23 13:37:13.008829819 (5m58.470922302s) [seg_pos={0 0}, upload=85]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 23 13:37:13.008908238 (5m58.471000721s) [upload=85]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 23 13:37:13.008914450 (5m58.471006923s) [upload=85]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 23 13:37:13.352201772 (5m58.814294255s) [upload=85]: Done uploading segments... Sep 23 13:37:13.352227461 (5m58.814319944s) [upload=85]: Upload finished. err: <nil> Duration: 12.658107178s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 11.4207413s FAIL (2025-09-23 13:54:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: context deadline exceeded ======LOGS====== Sep 23 13:37:08.880516737 (5m54.34260922s) [upload=85]: Starting upload Sep 23 13:37:08.880590076 (5m54.342682549s) [upload=85]: Done starting upload Sep 23 13:37:08.880678955 (5m54.342771438s) [upload=85]: Uploading segments... Sep 23 13:37:08.880816417 (5m54.3429089s) [upload=85]: Got next segment. Inline: false Sep 23 13:37:08.880868837 (5m54.34296132s) [upload=85]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 13:37:08.880873937 (5m54.34296642s) [upload=85]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 13:37:09.155479775 (5m54.617572258s) [seg_pos={0 0}, upload=85]: Begin upload segment... Sep 23 13:37:09.155561942 (5m54.617654425s) [seg_pos={0 0}, upload=85]: Done begin upload segment. Sep 23 13:37:09.155609312 (5m54.617701795s) [noise=true , address=145.79.227.5:4221 , piece_id=37XI3ODHTNYYIW3TYSGEZKLL4REO4C7J7W4F3M54QPBS27HU2N7Q, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.155653065 (5m54.617745548s) [noise=true , address=145.79.227.13:4801 , piece_id=KDNEMJNYSDPBUV67TZ6PFMKXECG2TLUFSFX742MK5XRK32C2NHXA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156257627 (5m54.61835011s) [noise=true , address=45.91.116.4:4938 , piece_id=YXX7C7LQDTPKFEVXO7E6HN6QNM562OWWOXPI232ZHCLSY3WE4BUA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156274629 (5m54.618367102s) [noise=true , address=145.79.227.14:4839 , piece_id=65NEICBQJXMM6N6KBD5KGI5SFTVAV7NJJKBAET6HDGKI5VD325OQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156622501 (5m54.618715004s) [noise=true , address=145.79.227.6:4259 , piece_id=LPZNOJHDSME5TCQX2HQ544ZANJCJKGWIUXFLIOMNNSZBLYPNJTVA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156777296 (5m54.618869779s) [noise=true , address=145.79.227.16:4912 , piece_id=YLC5DB6H62T67W5LPTPQIQMWGJAN5ODYJB5PXIVX6X7JNFVCTNTA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156984401 (5m54.619076884s) [noise=true , address=145.79.227.14:4838 , piece_id=MG6N5Q7OOXD4OI6CV2VCSGIOOI6VB4GYFATAZ5WZWXFCZN6EFFLQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.157122624 (5m54.619215107s) [noise=true , address=45.91.116.10:4083 , piece_id=VONNFDRLV44LE65ZK7ENEJPNRMA2IPUGGGO3QGYJZLZLWOMNLFRQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.157345388 (5m54.619437871s) [noise=true , address=145.79.227.15:4876 , piece_id=ZHNZB3WQ4QQWR2LO6AQ64ONLIZ2EKCOBYGXFNR5FBY4CZL6QLVMA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.158372935 (5m54.620465408s) [noise=true , address=45.91.116.14:4189 , piece_id=HBFO3FNIMMF5UIDBXTFQQG6WZ4SFGJI3W6CSJ5EBAGN3D7JI6MSA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.159036498 (5m54.621128982s) [noise=true , address=45.91.116.11:4117 , piece_id=QEF7PSZ4ZAV2XM546G6QVKSJVH4JGCSYRAU572ABBMRGKVP23N6A, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.159800133 (5m54.621892616s) [noise=true , address=145.79.227.12:4765 , piece_id=H4PQGKP3XUGR6ARQ3UW3ZVPNCPS5WA5NM6FYJG5TBU3FALNXQYNA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.161637361 (5m54.623729844s) [noise=true , address=145.79.227.17:4970 , piece_id=IW4WTHW5W4SFOYMCXENR3DGYCA2OQJ4OO5RSXPYX6SUIGWKA4LBA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.163221126 (5m54.62531361s) [noise=true , address=77.90.31.11:4553 , piece_id=7XXKVSGNIYW5ZN6G5AV3JWP56QHF2HI4KOXJZ2DNOSGDHVZXDQTQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.163665793 (5m54.625758277s) [noise=true , address=145.79.227.11:4731 , piece_id=XR2GN3LWECQJ4OTSM44QXNW4B6GE7P5C5EFZFKJXD2M4GCMJ54VQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.164366027 (5m54.62645851s) [noise=true , address=145.79.227.12:4767 , piece_id=7JPOEJPRMBWO5MCNTISEDMBSYFACGTPJJUCBDX6EVYUKR3E7MYSA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:10.933255114 (5m56.395347597s) [upload=85]: Next returned nil segment Sep 23 13:37:10.933279932 (5m56.395372415s) [upload=85]: Waiting for error group managing segments... Sep 23 13:37:11.355919055 (5m56.818011548s) [noise=true , address=45.91.116.10:4083 , piece_id=VONNFDRLV44LE65ZK7ENEJPNRMA2IPUGGGO3QGYJZLZLWOMNLFRQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.366419560 (5m56.828512043s) [noise=true , address=45.91.116.4:4938 , piece_id=YXX7C7LQDTPKFEVXO7E6HN6QNM562OWWOXPI232ZHCLSY3WE4BUA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.373438585 (5m56.835531068s) [noise=true , address=45.91.116.11:4117 , piece_id=QEF7PSZ4ZAV2XM546G6QVKSJVH4JGCSYRAU572ABBMRGKVP23N6A, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.388282841 (5m56.850375324s) [noise=true , address=45.91.116.14:4189 , piece_id=HBFO3FNIMMF5UIDBXTFQQG6WZ4SFGJI3W6CSJ5EBAGN3D7JI6MSA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.396367205 (5m56.858459688s) [noise=true , address=77.90.31.11:4553 , piece_id=7XXKVSGNIYW5ZN6G5AV3JWP56QHF2HI4KOXJZ2DNOSGDHVZXDQTQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.050842867 (5m57.51293534s) [noise=true , address=145.79.227.13:4801 , piece_id=KDNEMJNYSDPBUV67TZ6PFMKXECG2TLUFSFX742MK5XRK32C2NHXA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.054995253 (5m57.517087736s) [noise=true , address=145.79.227.15:4876 , piece_id=ZHNZB3WQ4QQWR2LO6AQ64ONLIZ2EKCOBYGXFNR5FBY4CZL6QLVMA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.072213210 (5m57.534305693s) [noise=true , address=145.79.227.17:4970 , piece_id=IW4WTHW5W4SFOYMCXENR3DGYCA2OQJ4OO5RSXPYX6SUIGWKA4LBA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.101168265 (5m57.563260748s) [noise=true , address=145.79.227.6:4259 , piece_id=LPZNOJHDSME5TCQX2HQ544ZANJCJKGWIUXFLIOMNNSZBLYPNJTVA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.106130595 (5m57.568223078s) [noise=true , address=145.79.227.12:4765 , piece_id=H4PQGKP3XUGR6ARQ3UW3ZVPNCPS5WA5NM6FYJG5TBU3FALNXQYNA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.118733793 (5m57.580826277s) [noise=true , address=145.79.227.14:4839 , piece_id=65NEICBQJXMM6N6KBD5KGI5SFTVAV7NJJKBAET6HDGKI5VD325OQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.508815721 (5m57.970908204s) [noise=true , address=145.79.227.11:4731 , piece_id=XR2GN3LWECQJ4OTSM44QXNW4B6GE7P5C5EFZFKJXD2M4GCMJ54VQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:13.008461678 (5m58.470554161s) [noise=true , address=145.79.227.16:4912 , piece_id=YLC5DB6H62T67W5LPTPQIQMWGJAN5ODYJB5PXIVX6X7JNFVCTNTA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:13.008503868 (5m58.470596351s) [seg_pos={0 0}, upload=85]: Segment reached optimal threshold of 13 pieces. Sep 23 13:37:13.008675254 (5m58.470767738s) [noise=true , address=145.79.227.12:4767 , piece_id=7JPOEJPRMBWO5MCNTISEDMBSYFACGTPJJUCBDX6EVYUKR3E7MYSA, seg_pos={0 0}, upload=85]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5): protocol: write tcp 188.166.225.170:41718->145.79.227.12:4767: use of closed network connection; write tcp 188.166.225.170:41718->145.79.227.12:4767: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41718->145.79.227.12:4767: use of closed network connection Sep 23 13:37:13.008689582 (5m58.470782065s) [noise=true , address=145.79.227.12:4767 , piece_id=7JPOEJPRMBWO5MCNTISEDMBSYFACGTPJJUCBDX6EVYUKR3E7MYSA, seg_pos={0 0}, upload=85]: Long tail cancellation due to optimal threshold Sep 23 13:37:13.008703729 (5m58.470796212s) [noise=true , address=145.79.227.5:4221 , piece_id=37XI3ODHTNYYIW3TYSGEZKLL4REO4C7J7W4F3M54QPBS27HU2N7Q, seg_pos={0 0}, upload=85]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR): protocol: expected piece hash; context canceled; EOF Sep 23 13:37:13.008716322 (5m58.470808806s) [noise=true , address=145.79.227.5:4221 , piece_id=37XI3ODHTNYYIW3TYSGEZKLL4REO4C7J7W4F3M54QPBS27HU2N7Q, seg_pos={0 0}, upload=85]: Long tail cancellation due to optimal threshold Sep 23 13:37:13.008800553 (5m58.470893037s) [noise=true , address=145.79.227.14:4838 , piece_id=MG6N5Q7OOXD4OI6CV2VCSGIOOI6VB4GYFATAZ5WZWXFCZN6EFFLQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: expected piece hash; context canceled; EOF Sep 23 13:37:13.008809150 (5m58.470901633s) [noise=true , address=145.79.227.14:4838 , piece_id=MG6N5Q7OOXD4OI6CV2VCSGIOOI6VB4GYFATAZ5WZWXFCZN6EFFLQ, seg_pos={0 0}, upload=85]: Long tail cancellation due to optimal threshold Sep 23 13:37:13.008829819 (5m58.470922302s) [seg_pos={0 0}, upload=85]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 23 13:37:13.008908238 (5m58.471000721s) [upload=85]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 23 13:37:13.008914450 (5m58.471006923s) [upload=85]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 23 13:37:13.352201772 (5m58.814294255s) [upload=85]: Done uploading segments... Sep 23 13:37:13.352227461 (5m58.814319944s) [upload=85]: Upload finished. err: <nil> Duration: 11.420813377s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 6.771116299s FAIL (2025-09-23 13:53:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: context deadline exceeded ======LOGS====== Sep 23 13:37:08.880516737 (5m54.34260922s) [upload=85]: Starting upload Sep 23 13:37:08.880590076 (5m54.342682549s) [upload=85]: Done starting upload Sep 23 13:37:08.880678955 (5m54.342771438s) [upload=85]: Uploading segments... Sep 23 13:37:08.880816417 (5m54.3429089s) [upload=85]: Got next segment. Inline: false Sep 23 13:37:08.880868837 (5m54.34296132s) [upload=85]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 13:37:08.880873937 (5m54.34296642s) [upload=85]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 13:37:09.155479775 (5m54.617572258s) [seg_pos={0 0}, upload=85]: Begin upload segment... Sep 23 13:37:09.155561942 (5m54.617654425s) [seg_pos={0 0}, upload=85]: Done begin upload segment. Sep 23 13:37:09.155609312 (5m54.617701795s) [noise=true , address=145.79.227.5:4221 , piece_id=37XI3ODHTNYYIW3TYSGEZKLL4REO4C7J7W4F3M54QPBS27HU2N7Q, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.155653065 (5m54.617745548s) [noise=true , address=145.79.227.13:4801 , piece_id=KDNEMJNYSDPBUV67TZ6PFMKXECG2TLUFSFX742MK5XRK32C2NHXA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156257627 (5m54.61835011s) [noise=true , address=45.91.116.4:4938 , piece_id=YXX7C7LQDTPKFEVXO7E6HN6QNM562OWWOXPI232ZHCLSY3WE4BUA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156274629 (5m54.618367102s) [noise=true , address=145.79.227.14:4839 , piece_id=65NEICBQJXMM6N6KBD5KGI5SFTVAV7NJJKBAET6HDGKI5VD325OQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156622501 (5m54.618715004s) [noise=true , address=145.79.227.6:4259 , piece_id=LPZNOJHDSME5TCQX2HQ544ZANJCJKGWIUXFLIOMNNSZBLYPNJTVA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156777296 (5m54.618869779s) [noise=true , address=145.79.227.16:4912 , piece_id=YLC5DB6H62T67W5LPTPQIQMWGJAN5ODYJB5PXIVX6X7JNFVCTNTA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156984401 (5m54.619076884s) [noise=true , address=145.79.227.14:4838 , piece_id=MG6N5Q7OOXD4OI6CV2VCSGIOOI6VB4GYFATAZ5WZWXFCZN6EFFLQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.157122624 (5m54.619215107s) [noise=true , address=45.91.116.10:4083 , piece_id=VONNFDRLV44LE65ZK7ENEJPNRMA2IPUGGGO3QGYJZLZLWOMNLFRQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.157345388 (5m54.619437871s) [noise=true , address=145.79.227.15:4876 , piece_id=ZHNZB3WQ4QQWR2LO6AQ64ONLIZ2EKCOBYGXFNR5FBY4CZL6QLVMA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.158372935 (5m54.620465408s) [noise=true , address=45.91.116.14:4189 , piece_id=HBFO3FNIMMF5UIDBXTFQQG6WZ4SFGJI3W6CSJ5EBAGN3D7JI6MSA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.159036498 (5m54.621128982s) [noise=true , address=45.91.116.11:4117 , piece_id=QEF7PSZ4ZAV2XM546G6QVKSJVH4JGCSYRAU572ABBMRGKVP23N6A, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.159800133 (5m54.621892616s) [noise=true , address=145.79.227.12:4765 , piece_id=H4PQGKP3XUGR6ARQ3UW3ZVPNCPS5WA5NM6FYJG5TBU3FALNXQYNA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.161637361 (5m54.623729844s) [noise=true , address=145.79.227.17:4970 , piece_id=IW4WTHW5W4SFOYMCXENR3DGYCA2OQJ4OO5RSXPYX6SUIGWKA4LBA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.163221126 (5m54.62531361s) [noise=true , address=77.90.31.11:4553 , piece_id=7XXKVSGNIYW5ZN6G5AV3JWP56QHF2HI4KOXJZ2DNOSGDHVZXDQTQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.163665793 (5m54.625758277s) [noise=true , address=145.79.227.11:4731 , piece_id=XR2GN3LWECQJ4OTSM44QXNW4B6GE7P5C5EFZFKJXD2M4GCMJ54VQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.164366027 (5m54.62645851s) [noise=true , address=145.79.227.12:4767 , piece_id=7JPOEJPRMBWO5MCNTISEDMBSYFACGTPJJUCBDX6EVYUKR3E7MYSA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:10.933255114 (5m56.395347597s) [upload=85]: Next returned nil segment Sep 23 13:37:10.933279932 (5m56.395372415s) [upload=85]: Waiting for error group managing segments... Sep 23 13:37:11.355919055 (5m56.818011548s) [noise=true , address=45.91.116.10:4083 , piece_id=VONNFDRLV44LE65ZK7ENEJPNRMA2IPUGGGO3QGYJZLZLWOMNLFRQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.366419560 (5m56.828512043s) [noise=true , address=45.91.116.4:4938 , piece_id=YXX7C7LQDTPKFEVXO7E6HN6QNM562OWWOXPI232ZHCLSY3WE4BUA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.373438585 (5m56.835531068s) [noise=true , address=45.91.116.11:4117 , piece_id=QEF7PSZ4ZAV2XM546G6QVKSJVH4JGCSYRAU572ABBMRGKVP23N6A, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.388282841 (5m56.850375324s) [noise=true , address=45.91.116.14:4189 , piece_id=HBFO3FNIMMF5UIDBXTFQQG6WZ4SFGJI3W6CSJ5EBAGN3D7JI6MSA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.396367205 (5m56.858459688s) [noise=true , address=77.90.31.11:4553 , piece_id=7XXKVSGNIYW5ZN6G5AV3JWP56QHF2HI4KOXJZ2DNOSGDHVZXDQTQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.050842867 (5m57.51293534s) [noise=true , address=145.79.227.13:4801 , piece_id=KDNEMJNYSDPBUV67TZ6PFMKXECG2TLUFSFX742MK5XRK32C2NHXA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.054995253 (5m57.517087736s) [noise=true , address=145.79.227.15:4876 , piece_id=ZHNZB3WQ4QQWR2LO6AQ64ONLIZ2EKCOBYGXFNR5FBY4CZL6QLVMA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.072213210 (5m57.534305693s) [noise=true , address=145.79.227.17:4970 , piece_id=IW4WTHW5W4SFOYMCXENR3DGYCA2OQJ4OO5RSXPYX6SUIGWKA4LBA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.101168265 (5m57.563260748s) [noise=true , address=145.79.227.6:4259 , piece_id=LPZNOJHDSME5TCQX2HQ544ZANJCJKGWIUXFLIOMNNSZBLYPNJTVA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.106130595 (5m57.568223078s) [noise=true , address=145.79.227.12:4765 , piece_id=H4PQGKP3XUGR6ARQ3UW3ZVPNCPS5WA5NM6FYJG5TBU3FALNXQYNA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.118733793 (5m57.580826277s) [noise=true , address=145.79.227.14:4839 , piece_id=65NEICBQJXMM6N6KBD5KGI5SFTVAV7NJJKBAET6HDGKI5VD325OQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.508815721 (5m57.970908204s) [noise=true , address=145.79.227.11:4731 , piece_id=XR2GN3LWECQJ4OTSM44QXNW4B6GE7P5C5EFZFKJXD2M4GCMJ54VQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:13.008461678 (5m58.470554161s) [noise=true , address=145.79.227.16:4912 , piece_id=YLC5DB6H62T67W5LPTPQIQMWGJAN5ODYJB5PXIVX6X7JNFVCTNTA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:13.008503868 (5m58.470596351s) [seg_pos={0 0}, upload=85]: Segment reached optimal threshold of 13 pieces. Sep 23 13:37:13.008675254 (5m58.470767738s) [noise=true , address=145.79.227.12:4767 , piece_id=7JPOEJPRMBWO5MCNTISEDMBSYFACGTPJJUCBDX6EVYUKR3E7MYSA, seg_pos={0 0}, upload=85]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5): protocol: write tcp 188.166.225.170:41718->145.79.227.12:4767: use of closed network connection; write tcp 188.166.225.170:41718->145.79.227.12:4767: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41718->145.79.227.12:4767: use of closed network connection Sep 23 13:37:13.008689582 (5m58.470782065s) [noise=true , address=145.79.227.12:4767 , piece_id=7JPOEJPRMBWO5MCNTISEDMBSYFACGTPJJUCBDX6EVYUKR3E7MYSA, seg_pos={0 0}, upload=85]: Long tail cancellation due to optimal threshold Sep 23 13:37:13.008703729 (5m58.470796212s) [noise=true , address=145.79.227.5:4221 , piece_id=37XI3ODHTNYYIW3TYSGEZKLL4REO4C7J7W4F3M54QPBS27HU2N7Q, seg_pos={0 0}, upload=85]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR): protocol: expected piece hash; context canceled; EOF Sep 23 13:37:13.008716322 (5m58.470808806s) [noise=true , address=145.79.227.5:4221 , piece_id=37XI3ODHTNYYIW3TYSGEZKLL4REO4C7J7W4F3M54QPBS27HU2N7Q, seg_pos={0 0}, upload=85]: Long tail cancellation due to optimal threshold Sep 23 13:37:13.008800553 (5m58.470893037s) [noise=true , address=145.79.227.14:4838 , piece_id=MG6N5Q7OOXD4OI6CV2VCSGIOOI6VB4GYFATAZ5WZWXFCZN6EFFLQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: expected piece hash; context canceled; EOF Sep 23 13:37:13.008809150 (5m58.470901633s) [noise=true , address=145.79.227.14:4838 , piece_id=MG6N5Q7OOXD4OI6CV2VCSGIOOI6VB4GYFATAZ5WZWXFCZN6EFFLQ, seg_pos={0 0}, upload=85]: Long tail cancellation due to optimal threshold Sep 23 13:37:13.008829819 (5m58.470922302s) [seg_pos={0 0}, upload=85]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 23 13:37:13.008908238 (5m58.471000721s) [upload=85]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 23 13:37:13.008914450 (5m58.471006923s) [upload=85]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 23 13:37:13.352201772 (5m58.814294255s) [upload=85]: Done uploading segments... Sep 23 13:37:13.352227461 (5m58.814319944s) [upload=85]: Upload finished. err: <nil> Duration: 6.771146716s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 6.901042425s FAIL (2025-09-23 13:52:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: context deadline exceeded ======LOGS====== Sep 23 13:37:08.880516737 (5m54.34260922s) [upload=85]: Starting upload Sep 23 13:37:08.880590076 (5m54.342682549s) [upload=85]: Done starting upload Sep 23 13:37:08.880678955 (5m54.342771438s) [upload=85]: Uploading segments... Sep 23 13:37:08.880816417 (5m54.3429089s) [upload=85]: Got next segment. Inline: false Sep 23 13:37:08.880868837 (5m54.34296132s) [upload=85]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 13:37:08.880873937 (5m54.34296642s) [upload=85]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 13:37:09.155479775 (5m54.617572258s) [seg_pos={0 0}, upload=85]: Begin upload segment... Sep 23 13:37:09.155561942 (5m54.617654425s) [seg_pos={0 0}, upload=85]: Done begin upload segment. Sep 23 13:37:09.155609312 (5m54.617701795s) [noise=true , address=145.79.227.5:4221 , piece_id=37XI3ODHTNYYIW3TYSGEZKLL4REO4C7J7W4F3M54QPBS27HU2N7Q, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.155653065 (5m54.617745548s) [noise=true , address=145.79.227.13:4801 , piece_id=KDNEMJNYSDPBUV67TZ6PFMKXECG2TLUFSFX742MK5XRK32C2NHXA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156257627 (5m54.61835011s) [noise=true , address=45.91.116.4:4938 , piece_id=YXX7C7LQDTPKFEVXO7E6HN6QNM562OWWOXPI232ZHCLSY3WE4BUA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156274629 (5m54.618367102s) [noise=true , address=145.79.227.14:4839 , piece_id=65NEICBQJXMM6N6KBD5KGI5SFTVAV7NJJKBAET6HDGKI5VD325OQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156622501 (5m54.618715004s) [noise=true , address=145.79.227.6:4259 , piece_id=LPZNOJHDSME5TCQX2HQ544ZANJCJKGWIUXFLIOMNNSZBLYPNJTVA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156777296 (5m54.618869779s) [noise=true , address=145.79.227.16:4912 , piece_id=YLC5DB6H62T67W5LPTPQIQMWGJAN5ODYJB5PXIVX6X7JNFVCTNTA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156984401 (5m54.619076884s) [noise=true , address=145.79.227.14:4838 , piece_id=MG6N5Q7OOXD4OI6CV2VCSGIOOI6VB4GYFATAZ5WZWXFCZN6EFFLQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.157122624 (5m54.619215107s) [noise=true , address=45.91.116.10:4083 , piece_id=VONNFDRLV44LE65ZK7ENEJPNRMA2IPUGGGO3QGYJZLZLWOMNLFRQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.157345388 (5m54.619437871s) [noise=true , address=145.79.227.15:4876 , piece_id=ZHNZB3WQ4QQWR2LO6AQ64ONLIZ2EKCOBYGXFNR5FBY4CZL6QLVMA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.158372935 (5m54.620465408s) [noise=true , address=45.91.116.14:4189 , piece_id=HBFO3FNIMMF5UIDBXTFQQG6WZ4SFGJI3W6CSJ5EBAGN3D7JI6MSA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.159036498 (5m54.621128982s) [noise=true , address=45.91.116.11:4117 , piece_id=QEF7PSZ4ZAV2XM546G6QVKSJVH4JGCSYRAU572ABBMRGKVP23N6A, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.159800133 (5m54.621892616s) [noise=true , address=145.79.227.12:4765 , piece_id=H4PQGKP3XUGR6ARQ3UW3ZVPNCPS5WA5NM6FYJG5TBU3FALNXQYNA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.161637361 (5m54.623729844s) [noise=true , address=145.79.227.17:4970 , piece_id=IW4WTHW5W4SFOYMCXENR3DGYCA2OQJ4OO5RSXPYX6SUIGWKA4LBA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.163221126 (5m54.62531361s) [noise=true , address=77.90.31.11:4553 , piece_id=7XXKVSGNIYW5ZN6G5AV3JWP56QHF2HI4KOXJZ2DNOSGDHVZXDQTQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.163665793 (5m54.625758277s) [noise=true , address=145.79.227.11:4731 , piece_id=XR2GN3LWECQJ4OTSM44QXNW4B6GE7P5C5EFZFKJXD2M4GCMJ54VQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.164366027 (5m54.62645851s) [noise=true , address=145.79.227.12:4767 , piece_id=7JPOEJPRMBWO5MCNTISEDMBSYFACGTPJJUCBDX6EVYUKR3E7MYSA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:10.933255114 (5m56.395347597s) [upload=85]: Next returned nil segment Sep 23 13:37:10.933279932 (5m56.395372415s) [upload=85]: Waiting for error group managing segments... Sep 23 13:37:11.355919055 (5m56.818011548s) [noise=true , address=45.91.116.10:4083 , piece_id=VONNFDRLV44LE65ZK7ENEJPNRMA2IPUGGGO3QGYJZLZLWOMNLFRQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.366419560 (5m56.828512043s) [noise=true , address=45.91.116.4:4938 , piece_id=YXX7C7LQDTPKFEVXO7E6HN6QNM562OWWOXPI232ZHCLSY3WE4BUA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.373438585 (5m56.835531068s) [noise=true , address=45.91.116.11:4117 , piece_id=QEF7PSZ4ZAV2XM546G6QVKSJVH4JGCSYRAU572ABBMRGKVP23N6A, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.388282841 (5m56.850375324s) [noise=true , address=45.91.116.14:4189 , piece_id=HBFO3FNIMMF5UIDBXTFQQG6WZ4SFGJI3W6CSJ5EBAGN3D7JI6MSA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.396367205 (5m56.858459688s) [noise=true , address=77.90.31.11:4553 , piece_id=7XXKVSGNIYW5ZN6G5AV3JWP56QHF2HI4KOXJZ2DNOSGDHVZXDQTQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.050842867 (5m57.51293534s) [noise=true , address=145.79.227.13:4801 , piece_id=KDNEMJNYSDPBUV67TZ6PFMKXECG2TLUFSFX742MK5XRK32C2NHXA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.054995253 (5m57.517087736s) [noise=true , address=145.79.227.15:4876 , piece_id=ZHNZB3WQ4QQWR2LO6AQ64ONLIZ2EKCOBYGXFNR5FBY4CZL6QLVMA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.072213210 (5m57.534305693s) [noise=true , address=145.79.227.17:4970 , piece_id=IW4WTHW5W4SFOYMCXENR3DGYCA2OQJ4OO5RSXPYX6SUIGWKA4LBA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.101168265 (5m57.563260748s) [noise=true , address=145.79.227.6:4259 , piece_id=LPZNOJHDSME5TCQX2HQ544ZANJCJKGWIUXFLIOMNNSZBLYPNJTVA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.106130595 (5m57.568223078s) [noise=true , address=145.79.227.12:4765 , piece_id=H4PQGKP3XUGR6ARQ3UW3ZVPNCPS5WA5NM6FYJG5TBU3FALNXQYNA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.118733793 (5m57.580826277s) [noise=true , address=145.79.227.14:4839 , piece_id=65NEICBQJXMM6N6KBD5KGI5SFTVAV7NJJKBAET6HDGKI5VD325OQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.508815721 (5m57.970908204s) [noise=true , address=145.79.227.11:4731 , piece_id=XR2GN3LWECQJ4OTSM44QXNW4B6GE7P5C5EFZFKJXD2M4GCMJ54VQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:13.008461678 (5m58.470554161s) [noise=true , address=145.79.227.16:4912 , piece_id=YLC5DB6H62T67W5LPTPQIQMWGJAN5ODYJB5PXIVX6X7JNFVCTNTA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:13.008503868 (5m58.470596351s) [seg_pos={0 0}, upload=85]: Segment reached optimal threshold of 13 pieces. Sep 23 13:37:13.008675254 (5m58.470767738s) [noise=true , address=145.79.227.12:4767 , piece_id=7JPOEJPRMBWO5MCNTISEDMBSYFACGTPJJUCBDX6EVYUKR3E7MYSA, seg_pos={0 0}, upload=85]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5): protocol: write tcp 188.166.225.170:41718->145.79.227.12:4767: use of closed network connection; write tcp 188.166.225.170:41718->145.79.227.12:4767: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41718->145.79.227.12:4767: use of closed network connection Sep 23 13:37:13.008689582 (5m58.470782065s) [noise=true , address=145.79.227.12:4767 , piece_id=7JPOEJPRMBWO5MCNTISEDMBSYFACGTPJJUCBDX6EVYUKR3E7MYSA, seg_pos={0 0}, upload=85]: Long tail cancellation due to optimal threshold Sep 23 13:37:13.008703729 (5m58.470796212s) [noise=true , address=145.79.227.5:4221 , piece_id=37XI3ODHTNYYIW3TYSGEZKLL4REO4C7J7W4F3M54QPBS27HU2N7Q, seg_pos={0 0}, upload=85]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR): protocol: expected piece hash; context canceled; EOF Sep 23 13:37:13.008716322 (5m58.470808806s) [noise=true , address=145.79.227.5:4221 , piece_id=37XI3ODHTNYYIW3TYSGEZKLL4REO4C7J7W4F3M54QPBS27HU2N7Q, seg_pos={0 0}, upload=85]: Long tail cancellation due to optimal threshold Sep 23 13:37:13.008800553 (5m58.470893037s) [noise=true , address=145.79.227.14:4838 , piece_id=MG6N5Q7OOXD4OI6CV2VCSGIOOI6VB4GYFATAZ5WZWXFCZN6EFFLQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: expected piece hash; context canceled; EOF Sep 23 13:37:13.008809150 (5m58.470901633s) [noise=true , address=145.79.227.14:4838 , piece_id=MG6N5Q7OOXD4OI6CV2VCSGIOOI6VB4GYFATAZ5WZWXFCZN6EFFLQ, seg_pos={0 0}, upload=85]: Long tail cancellation due to optimal threshold Sep 23 13:37:13.008829819 (5m58.470922302s) [seg_pos={0 0}, upload=85]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 23 13:37:13.008908238 (5m58.471000721s) [upload=85]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 23 13:37:13.008914450 (5m58.471006923s) [upload=85]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 23 13:37:13.352201772 (5m58.814294255s) [upload=85]: Done uploading segments... Sep 23 13:37:13.352227461 (5m58.814319944s) [upload=85]: Upload finished. err: <nil> Duration: 6.901081188s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 12.956270421s FAIL (2025-09-23 13:52:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: context deadline exceeded ======LOGS====== Sep 23 13:37:08.880516737 (5m54.34260922s) [upload=85]: Starting upload Sep 23 13:37:08.880590076 (5m54.342682549s) [upload=85]: Done starting upload Sep 23 13:37:08.880678955 (5m54.342771438s) [upload=85]: Uploading segments... Sep 23 13:37:08.880816417 (5m54.3429089s) [upload=85]: Got next segment. Inline: false Sep 23 13:37:08.880868837 (5m54.34296132s) [upload=85]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 13:37:08.880873937 (5m54.34296642s) [upload=85]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 13:37:09.155479775 (5m54.617572258s) [seg_pos={0 0}, upload=85]: Begin upload segment... Sep 23 13:37:09.155561942 (5m54.617654425s) [seg_pos={0 0}, upload=85]: Done begin upload segment. Sep 23 13:37:09.155609312 (5m54.617701795s) [noise=true , address=145.79.227.5:4221 , piece_id=37XI3ODHTNYYIW3TYSGEZKLL4REO4C7J7W4F3M54QPBS27HU2N7Q, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.155653065 (5m54.617745548s) [noise=true , address=145.79.227.13:4801 , piece_id=KDNEMJNYSDPBUV67TZ6PFMKXECG2TLUFSFX742MK5XRK32C2NHXA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156257627 (5m54.61835011s) [noise=true , address=45.91.116.4:4938 , piece_id=YXX7C7LQDTPKFEVXO7E6HN6QNM562OWWOXPI232ZHCLSY3WE4BUA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156274629 (5m54.618367102s) [noise=true , address=145.79.227.14:4839 , piece_id=65NEICBQJXMM6N6KBD5KGI5SFTVAV7NJJKBAET6HDGKI5VD325OQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156622501 (5m54.618715004s) [noise=true , address=145.79.227.6:4259 , piece_id=LPZNOJHDSME5TCQX2HQ544ZANJCJKGWIUXFLIOMNNSZBLYPNJTVA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156777296 (5m54.618869779s) [noise=true , address=145.79.227.16:4912 , piece_id=YLC5DB6H62T67W5LPTPQIQMWGJAN5ODYJB5PXIVX6X7JNFVCTNTA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156984401 (5m54.619076884s) [noise=true , address=145.79.227.14:4838 , piece_id=MG6N5Q7OOXD4OI6CV2VCSGIOOI6VB4GYFATAZ5WZWXFCZN6EFFLQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.157122624 (5m54.619215107s) [noise=true , address=45.91.116.10:4083 , piece_id=VONNFDRLV44LE65ZK7ENEJPNRMA2IPUGGGO3QGYJZLZLWOMNLFRQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.157345388 (5m54.619437871s) [noise=true , address=145.79.227.15:4876 , piece_id=ZHNZB3WQ4QQWR2LO6AQ64ONLIZ2EKCOBYGXFNR5FBY4CZL6QLVMA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.158372935 (5m54.620465408s) [noise=true , address=45.91.116.14:4189 , piece_id=HBFO3FNIMMF5UIDBXTFQQG6WZ4SFGJI3W6CSJ5EBAGN3D7JI6MSA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.159036498 (5m54.621128982s) [noise=true , address=45.91.116.11:4117 , piece_id=QEF7PSZ4ZAV2XM546G6QVKSJVH4JGCSYRAU572ABBMRGKVP23N6A, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.159800133 (5m54.621892616s) [noise=true , address=145.79.227.12:4765 , piece_id=H4PQGKP3XUGR6ARQ3UW3ZVPNCPS5WA5NM6FYJG5TBU3FALNXQYNA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.161637361 (5m54.623729844s) [noise=true , address=145.79.227.17:4970 , piece_id=IW4WTHW5W4SFOYMCXENR3DGYCA2OQJ4OO5RSXPYX6SUIGWKA4LBA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.163221126 (5m54.62531361s) [noise=true , address=77.90.31.11:4553 , piece_id=7XXKVSGNIYW5ZN6G5AV3JWP56QHF2HI4KOXJZ2DNOSGDHVZXDQTQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.163665793 (5m54.625758277s) [noise=true , address=145.79.227.11:4731 , piece_id=XR2GN3LWECQJ4OTSM44QXNW4B6GE7P5C5EFZFKJXD2M4GCMJ54VQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.164366027 (5m54.62645851s) [noise=true , address=145.79.227.12:4767 , piece_id=7JPOEJPRMBWO5MCNTISEDMBSYFACGTPJJUCBDX6EVYUKR3E7MYSA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:10.933255114 (5m56.395347597s) [upload=85]: Next returned nil segment Sep 23 13:37:10.933279932 (5m56.395372415s) [upload=85]: Waiting for error group managing segments... Sep 23 13:37:11.355919055 (5m56.818011548s) [noise=true , address=45.91.116.10:4083 , piece_id=VONNFDRLV44LE65ZK7ENEJPNRMA2IPUGGGO3QGYJZLZLWOMNLFRQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.366419560 (5m56.828512043s) [noise=true , address=45.91.116.4:4938 , piece_id=YXX7C7LQDTPKFEVXO7E6HN6QNM562OWWOXPI232ZHCLSY3WE4BUA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.373438585 (5m56.835531068s) [noise=true , address=45.91.116.11:4117 , piece_id=QEF7PSZ4ZAV2XM546G6QVKSJVH4JGCSYRAU572ABBMRGKVP23N6A, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.388282841 (5m56.850375324s) [noise=true , address=45.91.116.14:4189 , piece_id=HBFO3FNIMMF5UIDBXTFQQG6WZ4SFGJI3W6CSJ5EBAGN3D7JI6MSA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.396367205 (5m56.858459688s) [noise=true , address=77.90.31.11:4553 , piece_id=7XXKVSGNIYW5ZN6G5AV3JWP56QHF2HI4KOXJZ2DNOSGDHVZXDQTQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.050842867 (5m57.51293534s) [noise=true , address=145.79.227.13:4801 , piece_id=KDNEMJNYSDPBUV67TZ6PFMKXECG2TLUFSFX742MK5XRK32C2NHXA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.054995253 (5m57.517087736s) [noise=true , address=145.79.227.15:4876 , piece_id=ZHNZB3WQ4QQWR2LO6AQ64ONLIZ2EKCOBYGXFNR5FBY4CZL6QLVMA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.072213210 (5m57.534305693s) [noise=true , address=145.79.227.17:4970 , piece_id=IW4WTHW5W4SFOYMCXENR3DGYCA2OQJ4OO5RSXPYX6SUIGWKA4LBA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.101168265 (5m57.563260748s) [noise=true , address=145.79.227.6:4259 , piece_id=LPZNOJHDSME5TCQX2HQ544ZANJCJKGWIUXFLIOMNNSZBLYPNJTVA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.106130595 (5m57.568223078s) [noise=true , address=145.79.227.12:4765 , piece_id=H4PQGKP3XUGR6ARQ3UW3ZVPNCPS5WA5NM6FYJG5TBU3FALNXQYNA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.118733793 (5m57.580826277s) [noise=true , address=145.79.227.14:4839 , piece_id=65NEICBQJXMM6N6KBD5KGI5SFTVAV7NJJKBAET6HDGKI5VD325OQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.508815721 (5m57.970908204s) [noise=true , address=145.79.227.11:4731 , piece_id=XR2GN3LWECQJ4OTSM44QXNW4B6GE7P5C5EFZFKJXD2M4GCMJ54VQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:13.008461678 (5m58.470554161s) [noise=true , address=145.79.227.16:4912 , piece_id=YLC5DB6H62T67W5LPTPQIQMWGJAN5ODYJB5PXIVX6X7JNFVCTNTA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:13.008503868 (5m58.470596351s) [seg_pos={0 0}, upload=85]: Segment reached optimal threshold of 13 pieces. Sep 23 13:37:13.008675254 (5m58.470767738s) [noise=true , address=145.79.227.12:4767 , piece_id=7JPOEJPRMBWO5MCNTISEDMBSYFACGTPJJUCBDX6EVYUKR3E7MYSA, seg_pos={0 0}, upload=85]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5): protocol: write tcp 188.166.225.170:41718->145.79.227.12:4767: use of closed network connection; write tcp 188.166.225.170:41718->145.79.227.12:4767: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41718->145.79.227.12:4767: use of closed network connection Sep 23 13:37:13.008689582 (5m58.470782065s) [noise=true , address=145.79.227.12:4767 , piece_id=7JPOEJPRMBWO5MCNTISEDMBSYFACGTPJJUCBDX6EVYUKR3E7MYSA, seg_pos={0 0}, upload=85]: Long tail cancellation due to optimal threshold Sep 23 13:37:13.008703729 (5m58.470796212s) [noise=true , address=145.79.227.5:4221 , piece_id=37XI3ODHTNYYIW3TYSGEZKLL4REO4C7J7W4F3M54QPBS27HU2N7Q, seg_pos={0 0}, upload=85]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR): protocol: expected piece hash; context canceled; EOF Sep 23 13:37:13.008716322 (5m58.470808806s) [noise=true , address=145.79.227.5:4221 , piece_id=37XI3ODHTNYYIW3TYSGEZKLL4REO4C7J7W4F3M54QPBS27HU2N7Q, seg_pos={0 0}, upload=85]: Long tail cancellation due to optimal threshold Sep 23 13:37:13.008800553 (5m58.470893037s) [noise=true , address=145.79.227.14:4838 , piece_id=MG6N5Q7OOXD4OI6CV2VCSGIOOI6VB4GYFATAZ5WZWXFCZN6EFFLQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: expected piece hash; context canceled; EOF Sep 23 13:37:13.008809150 (5m58.470901633s) [noise=true , address=145.79.227.14:4838 , piece_id=MG6N5Q7OOXD4OI6CV2VCSGIOOI6VB4GYFATAZ5WZWXFCZN6EFFLQ, seg_pos={0 0}, upload=85]: Long tail cancellation due to optimal threshold Sep 23 13:37:13.008829819 (5m58.470922302s) [seg_pos={0 0}, upload=85]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 23 13:37:13.008908238 (5m58.471000721s) [upload=85]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 23 13:37:13.008914450 (5m58.471006923s) [upload=85]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 23 13:37:13.352201772 (5m58.814294255s) [upload=85]: Done uploading segments... Sep 23 13:37:13.352227461 (5m58.814319944s) [upload=85]: Upload finished. err: <nil> Duration: 12.956300967s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 6.786823149s FAIL (2025-09-23 13:51:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: context deadline exceeded ======LOGS====== Sep 23 13:37:08.880516737 (5m54.34260922s) [upload=85]: Starting upload Sep 23 13:37:08.880590076 (5m54.342682549s) [upload=85]: Done starting upload Sep 23 13:37:08.880678955 (5m54.342771438s) [upload=85]: Uploading segments... Sep 23 13:37:08.880816417 (5m54.3429089s) [upload=85]: Got next segment. Inline: false Sep 23 13:37:08.880868837 (5m54.34296132s) [upload=85]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 13:37:08.880873937 (5m54.34296642s) [upload=85]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 13:37:09.155479775 (5m54.617572258s) [seg_pos={0 0}, upload=85]: Begin upload segment... Sep 23 13:37:09.155561942 (5m54.617654425s) [seg_pos={0 0}, upload=85]: Done begin upload segment. Sep 23 13:37:09.155609312 (5m54.617701795s) [noise=true , address=145.79.227.5:4221 , piece_id=37XI3ODHTNYYIW3TYSGEZKLL4REO4C7J7W4F3M54QPBS27HU2N7Q, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.155653065 (5m54.617745548s) [noise=true , address=145.79.227.13:4801 , piece_id=KDNEMJNYSDPBUV67TZ6PFMKXECG2TLUFSFX742MK5XRK32C2NHXA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156257627 (5m54.61835011s) [noise=true , address=45.91.116.4:4938 , piece_id=YXX7C7LQDTPKFEVXO7E6HN6QNM562OWWOXPI232ZHCLSY3WE4BUA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156274629 (5m54.618367102s) [noise=true , address=145.79.227.14:4839 , piece_id=65NEICBQJXMM6N6KBD5KGI5SFTVAV7NJJKBAET6HDGKI5VD325OQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156622501 (5m54.618715004s) [noise=true , address=145.79.227.6:4259 , piece_id=LPZNOJHDSME5TCQX2HQ544ZANJCJKGWIUXFLIOMNNSZBLYPNJTVA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156777296 (5m54.618869779s) [noise=true , address=145.79.227.16:4912 , piece_id=YLC5DB6H62T67W5LPTPQIQMWGJAN5ODYJB5PXIVX6X7JNFVCTNTA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156984401 (5m54.619076884s) [noise=true , address=145.79.227.14:4838 , piece_id=MG6N5Q7OOXD4OI6CV2VCSGIOOI6VB4GYFATAZ5WZWXFCZN6EFFLQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.157122624 (5m54.619215107s) [noise=true , address=45.91.116.10:4083 , piece_id=VONNFDRLV44LE65ZK7ENEJPNRMA2IPUGGGO3QGYJZLZLWOMNLFRQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.157345388 (5m54.619437871s) [noise=true , address=145.79.227.15:4876 , piece_id=ZHNZB3WQ4QQWR2LO6AQ64ONLIZ2EKCOBYGXFNR5FBY4CZL6QLVMA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.158372935 (5m54.620465408s) [noise=true , address=45.91.116.14:4189 , piece_id=HBFO3FNIMMF5UIDBXTFQQG6WZ4SFGJI3W6CSJ5EBAGN3D7JI6MSA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.159036498 (5m54.621128982s) [noise=true , address=45.91.116.11:4117 , piece_id=QEF7PSZ4ZAV2XM546G6QVKSJVH4JGCSYRAU572ABBMRGKVP23N6A, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.159800133 (5m54.621892616s) [noise=true , address=145.79.227.12:4765 , piece_id=H4PQGKP3XUGR6ARQ3UW3ZVPNCPS5WA5NM6FYJG5TBU3FALNXQYNA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.161637361 (5m54.623729844s) [noise=true , address=145.79.227.17:4970 , piece_id=IW4WTHW5W4SFOYMCXENR3DGYCA2OQJ4OO5RSXPYX6SUIGWKA4LBA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.163221126 (5m54.62531361s) [noise=true , address=77.90.31.11:4553 , piece_id=7XXKVSGNIYW5ZN6G5AV3JWP56QHF2HI4KOXJZ2DNOSGDHVZXDQTQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.163665793 (5m54.625758277s) [noise=true , address=145.79.227.11:4731 , piece_id=XR2GN3LWECQJ4OTSM44QXNW4B6GE7P5C5EFZFKJXD2M4GCMJ54VQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.164366027 (5m54.62645851s) [noise=true , address=145.79.227.12:4767 , piece_id=7JPOEJPRMBWO5MCNTISEDMBSYFACGTPJJUCBDX6EVYUKR3E7MYSA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:10.933255114 (5m56.395347597s) [upload=85]: Next returned nil segment Sep 23 13:37:10.933279932 (5m56.395372415s) [upload=85]: Waiting for error group managing segments... Sep 23 13:37:11.355919055 (5m56.818011548s) [noise=true , address=45.91.116.10:4083 , piece_id=VONNFDRLV44LE65ZK7ENEJPNRMA2IPUGGGO3QGYJZLZLWOMNLFRQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.366419560 (5m56.828512043s) [noise=true , address=45.91.116.4:4938 , piece_id=YXX7C7LQDTPKFEVXO7E6HN6QNM562OWWOXPI232ZHCLSY3WE4BUA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.373438585 (5m56.835531068s) [noise=true , address=45.91.116.11:4117 , piece_id=QEF7PSZ4ZAV2XM546G6QVKSJVH4JGCSYRAU572ABBMRGKVP23N6A, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.388282841 (5m56.850375324s) [noise=true , address=45.91.116.14:4189 , piece_id=HBFO3FNIMMF5UIDBXTFQQG6WZ4SFGJI3W6CSJ5EBAGN3D7JI6MSA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.396367205 (5m56.858459688s) [noise=true , address=77.90.31.11:4553 , piece_id=7XXKVSGNIYW5ZN6G5AV3JWP56QHF2HI4KOXJZ2DNOSGDHVZXDQTQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.050842867 (5m57.51293534s) [noise=true , address=145.79.227.13:4801 , piece_id=KDNEMJNYSDPBUV67TZ6PFMKXECG2TLUFSFX742MK5XRK32C2NHXA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.054995253 (5m57.517087736s) [noise=true , address=145.79.227.15:4876 , piece_id=ZHNZB3WQ4QQWR2LO6AQ64ONLIZ2EKCOBYGXFNR5FBY4CZL6QLVMA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.072213210 (5m57.534305693s) [noise=true , address=145.79.227.17:4970 , piece_id=IW4WTHW5W4SFOYMCXENR3DGYCA2OQJ4OO5RSXPYX6SUIGWKA4LBA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.101168265 (5m57.563260748s) [noise=true , address=145.79.227.6:4259 , piece_id=LPZNOJHDSME5TCQX2HQ544ZANJCJKGWIUXFLIOMNNSZBLYPNJTVA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.106130595 (5m57.568223078s) [noise=true , address=145.79.227.12:4765 , piece_id=H4PQGKP3XUGR6ARQ3UW3ZVPNCPS5WA5NM6FYJG5TBU3FALNXQYNA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.118733793 (5m57.580826277s) [noise=true , address=145.79.227.14:4839 , piece_id=65NEICBQJXMM6N6KBD5KGI5SFTVAV7NJJKBAET6HDGKI5VD325OQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.508815721 (5m57.970908204s) [noise=true , address=145.79.227.11:4731 , piece_id=XR2GN3LWECQJ4OTSM44QXNW4B6GE7P5C5EFZFKJXD2M4GCMJ54VQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:13.008461678 (5m58.470554161s) [noise=true , address=145.79.227.16:4912 , piece_id=YLC5DB6H62T67W5LPTPQIQMWGJAN5ODYJB5PXIVX6X7JNFVCTNTA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:13.008503868 (5m58.470596351s) [seg_pos={0 0}, upload=85]: Segment reached optimal threshold of 13 pieces. Sep 23 13:37:13.008675254 (5m58.470767738s) [noise=true , address=145.79.227.12:4767 , piece_id=7JPOEJPRMBWO5MCNTISEDMBSYFACGTPJJUCBDX6EVYUKR3E7MYSA, seg_pos={0 0}, upload=85]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5): protocol: write tcp 188.166.225.170:41718->145.79.227.12:4767: use of closed network connection; write tcp 188.166.225.170:41718->145.79.227.12:4767: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41718->145.79.227.12:4767: use of closed network connection Sep 23 13:37:13.008689582 (5m58.470782065s) [noise=true , address=145.79.227.12:4767 , piece_id=7JPOEJPRMBWO5MCNTISEDMBSYFACGTPJJUCBDX6EVYUKR3E7MYSA, seg_pos={0 0}, upload=85]: Long tail cancellation due to optimal threshold Sep 23 13:37:13.008703729 (5m58.470796212s) [noise=true , address=145.79.227.5:4221 , piece_id=37XI3ODHTNYYIW3TYSGEZKLL4REO4C7J7W4F3M54QPBS27HU2N7Q, seg_pos={0 0}, upload=85]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR): protocol: expected piece hash; context canceled; EOF Sep 23 13:37:13.008716322 (5m58.470808806s) [noise=true , address=145.79.227.5:4221 , piece_id=37XI3ODHTNYYIW3TYSGEZKLL4REO4C7J7W4F3M54QPBS27HU2N7Q, seg_pos={0 0}, upload=85]: Long tail cancellation due to optimal threshold Sep 23 13:37:13.008800553 (5m58.470893037s) [noise=true , address=145.79.227.14:4838 , piece_id=MG6N5Q7OOXD4OI6CV2VCSGIOOI6VB4GYFATAZ5WZWXFCZN6EFFLQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: expected piece hash; context canceled; EOF Sep 23 13:37:13.008809150 (5m58.470901633s) [noise=true , address=145.79.227.14:4838 , piece_id=MG6N5Q7OOXD4OI6CV2VCSGIOOI6VB4GYFATAZ5WZWXFCZN6EFFLQ, seg_pos={0 0}, upload=85]: Long tail cancellation due to optimal threshold Sep 23 13:37:13.008829819 (5m58.470922302s) [seg_pos={0 0}, upload=85]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 23 13:37:13.008908238 (5m58.471000721s) [upload=85]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 23 13:37:13.008914450 (5m58.471006923s) [upload=85]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 23 13:37:13.352201772 (5m58.814294255s) [upload=85]: Done uploading segments... Sep 23 13:37:13.352227461 (5m58.814319944s) [upload=85]: Upload finished. err: <nil> Duration: 6.786852254s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 7.459831562s FAIL (2025-09-23 13:51:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: context deadline exceeded ======LOGS====== Sep 23 13:37:08.880516737 (5m54.34260922s) [upload=85]: Starting upload Sep 23 13:37:08.880590076 (5m54.342682549s) [upload=85]: Done starting upload Sep 23 13:37:08.880678955 (5m54.342771438s) [upload=85]: Uploading segments... Sep 23 13:37:08.880816417 (5m54.3429089s) [upload=85]: Got next segment. Inline: false Sep 23 13:37:08.880868837 (5m54.34296132s) [upload=85]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 13:37:08.880873937 (5m54.34296642s) [upload=85]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 13:37:09.155479775 (5m54.617572258s) [seg_pos={0 0}, upload=85]: Begin upload segment... Sep 23 13:37:09.155561942 (5m54.617654425s) [seg_pos={0 0}, upload=85]: Done begin upload segment. Sep 23 13:37:09.155609312 (5m54.617701795s) [noise=true , address=145.79.227.5:4221 , piece_id=37XI3ODHTNYYIW3TYSGEZKLL4REO4C7J7W4F3M54QPBS27HU2N7Q, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.155653065 (5m54.617745548s) [noise=true , address=145.79.227.13:4801 , piece_id=KDNEMJNYSDPBUV67TZ6PFMKXECG2TLUFSFX742MK5XRK32C2NHXA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156257627 (5m54.61835011s) [noise=true , address=45.91.116.4:4938 , piece_id=YXX7C7LQDTPKFEVXO7E6HN6QNM562OWWOXPI232ZHCLSY3WE4BUA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156274629 (5m54.618367102s) [noise=true , address=145.79.227.14:4839 , piece_id=65NEICBQJXMM6N6KBD5KGI5SFTVAV7NJJKBAET6HDGKI5VD325OQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156622501 (5m54.618715004s) [noise=true , address=145.79.227.6:4259 , piece_id=LPZNOJHDSME5TCQX2HQ544ZANJCJKGWIUXFLIOMNNSZBLYPNJTVA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156777296 (5m54.618869779s) [noise=true , address=145.79.227.16:4912 , piece_id=YLC5DB6H62T67W5LPTPQIQMWGJAN5ODYJB5PXIVX6X7JNFVCTNTA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156984401 (5m54.619076884s) [noise=true , address=145.79.227.14:4838 , piece_id=MG6N5Q7OOXD4OI6CV2VCSGIOOI6VB4GYFATAZ5WZWXFCZN6EFFLQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.157122624 (5m54.619215107s) [noise=true , address=45.91.116.10:4083 , piece_id=VONNFDRLV44LE65ZK7ENEJPNRMA2IPUGGGO3QGYJZLZLWOMNLFRQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.157345388 (5m54.619437871s) [noise=true , address=145.79.227.15:4876 , piece_id=ZHNZB3WQ4QQWR2LO6AQ64ONLIZ2EKCOBYGXFNR5FBY4CZL6QLVMA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.158372935 (5m54.620465408s) [noise=true , address=45.91.116.14:4189 , piece_id=HBFO3FNIMMF5UIDBXTFQQG6WZ4SFGJI3W6CSJ5EBAGN3D7JI6MSA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.159036498 (5m54.621128982s) [noise=true , address=45.91.116.11:4117 , piece_id=QEF7PSZ4ZAV2XM546G6QVKSJVH4JGCSYRAU572ABBMRGKVP23N6A, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.159800133 (5m54.621892616s) [noise=true , address=145.79.227.12:4765 , piece_id=H4PQGKP3XUGR6ARQ3UW3ZVPNCPS5WA5NM6FYJG5TBU3FALNXQYNA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.161637361 (5m54.623729844s) [noise=true , address=145.79.227.17:4970 , piece_id=IW4WTHW5W4SFOYMCXENR3DGYCA2OQJ4OO5RSXPYX6SUIGWKA4LBA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.163221126 (5m54.62531361s) [noise=true , address=77.90.31.11:4553 , piece_id=7XXKVSGNIYW5ZN6G5AV3JWP56QHF2HI4KOXJZ2DNOSGDHVZXDQTQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.163665793 (5m54.625758277s) [noise=true , address=145.79.227.11:4731 , piece_id=XR2GN3LWECQJ4OTSM44QXNW4B6GE7P5C5EFZFKJXD2M4GCMJ54VQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.164366027 (5m54.62645851s) [noise=true , address=145.79.227.12:4767 , piece_id=7JPOEJPRMBWO5MCNTISEDMBSYFACGTPJJUCBDX6EVYUKR3E7MYSA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:10.933255114 (5m56.395347597s) [upload=85]: Next returned nil segment Sep 23 13:37:10.933279932 (5m56.395372415s) [upload=85]: Waiting for error group managing segments... Sep 23 13:37:11.355919055 (5m56.818011548s) [noise=true , address=45.91.116.10:4083 , piece_id=VONNFDRLV44LE65ZK7ENEJPNRMA2IPUGGGO3QGYJZLZLWOMNLFRQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.366419560 (5m56.828512043s) [noise=true , address=45.91.116.4:4938 , piece_id=YXX7C7LQDTPKFEVXO7E6HN6QNM562OWWOXPI232ZHCLSY3WE4BUA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.373438585 (5m56.835531068s) [noise=true , address=45.91.116.11:4117 , piece_id=QEF7PSZ4ZAV2XM546G6QVKSJVH4JGCSYRAU572ABBMRGKVP23N6A, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.388282841 (5m56.850375324s) [noise=true , address=45.91.116.14:4189 , piece_id=HBFO3FNIMMF5UIDBXTFQQG6WZ4SFGJI3W6CSJ5EBAGN3D7JI6MSA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.396367205 (5m56.858459688s) [noise=true , address=77.90.31.11:4553 , piece_id=7XXKVSGNIYW5ZN6G5AV3JWP56QHF2HI4KOXJZ2DNOSGDHVZXDQTQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.050842867 (5m57.51293534s) [noise=true , address=145.79.227.13:4801 , piece_id=KDNEMJNYSDPBUV67TZ6PFMKXECG2TLUFSFX742MK5XRK32C2NHXA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.054995253 (5m57.517087736s) [noise=true , address=145.79.227.15:4876 , piece_id=ZHNZB3WQ4QQWR2LO6AQ64ONLIZ2EKCOBYGXFNR5FBY4CZL6QLVMA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.072213210 (5m57.534305693s) [noise=true , address=145.79.227.17:4970 , piece_id=IW4WTHW5W4SFOYMCXENR3DGYCA2OQJ4OO5RSXPYX6SUIGWKA4LBA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.101168265 (5m57.563260748s) [noise=true , address=145.79.227.6:4259 , piece_id=LPZNOJHDSME5TCQX2HQ544ZANJCJKGWIUXFLIOMNNSZBLYPNJTVA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.106130595 (5m57.568223078s) [noise=true , address=145.79.227.12:4765 , piece_id=H4PQGKP3XUGR6ARQ3UW3ZVPNCPS5WA5NM6FYJG5TBU3FALNXQYNA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.118733793 (5m57.580826277s) [noise=true , address=145.79.227.14:4839 , piece_id=65NEICBQJXMM6N6KBD5KGI5SFTVAV7NJJKBAET6HDGKI5VD325OQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.508815721 (5m57.970908204s) [noise=true , address=145.79.227.11:4731 , piece_id=XR2GN3LWECQJ4OTSM44QXNW4B6GE7P5C5EFZFKJXD2M4GCMJ54VQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:13.008461678 (5m58.470554161s) [noise=true , address=145.79.227.16:4912 , piece_id=YLC5DB6H62T67W5LPTPQIQMWGJAN5ODYJB5PXIVX6X7JNFVCTNTA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:13.008503868 (5m58.470596351s) [seg_pos={0 0}, upload=85]: Segment reached optimal threshold of 13 pieces. Sep 23 13:37:13.008675254 (5m58.470767738s) [noise=true , address=145.79.227.12:4767 , piece_id=7JPOEJPRMBWO5MCNTISEDMBSYFACGTPJJUCBDX6EVYUKR3E7MYSA, seg_pos={0 0}, upload=85]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5): protocol: write tcp 188.166.225.170:41718->145.79.227.12:4767: use of closed network connection; write tcp 188.166.225.170:41718->145.79.227.12:4767: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41718->145.79.227.12:4767: use of closed network connection Sep 23 13:37:13.008689582 (5m58.470782065s) [noise=true , address=145.79.227.12:4767 , piece_id=7JPOEJPRMBWO5MCNTISEDMBSYFACGTPJJUCBDX6EVYUKR3E7MYSA, seg_pos={0 0}, upload=85]: Long tail cancellation due to optimal threshold Sep 23 13:37:13.008703729 (5m58.470796212s) [noise=true , address=145.79.227.5:4221 , piece_id=37XI3ODHTNYYIW3TYSGEZKLL4REO4C7J7W4F3M54QPBS27HU2N7Q, seg_pos={0 0}, upload=85]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR): protocol: expected piece hash; context canceled; EOF Sep 23 13:37:13.008716322 (5m58.470808806s) [noise=true , address=145.79.227.5:4221 , piece_id=37XI3ODHTNYYIW3TYSGEZKLL4REO4C7J7W4F3M54QPBS27HU2N7Q, seg_pos={0 0}, upload=85]: Long tail cancellation due to optimal threshold Sep 23 13:37:13.008800553 (5m58.470893037s) [noise=true , address=145.79.227.14:4838 , piece_id=MG6N5Q7OOXD4OI6CV2VCSGIOOI6VB4GYFATAZ5WZWXFCZN6EFFLQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: expected piece hash; context canceled; EOF Sep 23 13:37:13.008809150 (5m58.470901633s) [noise=true , address=145.79.227.14:4838 , piece_id=MG6N5Q7OOXD4OI6CV2VCSGIOOI6VB4GYFATAZ5WZWXFCZN6EFFLQ, seg_pos={0 0}, upload=85]: Long tail cancellation due to optimal threshold Sep 23 13:37:13.008829819 (5m58.470922302s) [seg_pos={0 0}, upload=85]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 23 13:37:13.008908238 (5m58.471000721s) [upload=85]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 23 13:37:13.008914450 (5m58.471006923s) [upload=85]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 23 13:37:13.352201772 (5m58.814294255s) [upload=85]: Done uploading segments... Sep 23 13:37:13.352227461 (5m58.814319944s) [upload=85]: Upload finished. err: <nil> Duration: 7.459871939s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 7.660795477s FAIL (2025-09-23 13:50:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: context deadline exceeded ======LOGS====== Sep 23 13:37:08.880516737 (5m54.34260922s) [upload=85]: Starting upload Sep 23 13:37:08.880590076 (5m54.342682549s) [upload=85]: Done starting upload Sep 23 13:37:08.880678955 (5m54.342771438s) [upload=85]: Uploading segments... Sep 23 13:37:08.880816417 (5m54.3429089s) [upload=85]: Got next segment. Inline: false Sep 23 13:37:08.880868837 (5m54.34296132s) [upload=85]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 13:37:08.880873937 (5m54.34296642s) [upload=85]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 13:37:09.155479775 (5m54.617572258s) [seg_pos={0 0}, upload=85]: Begin upload segment... Sep 23 13:37:09.155561942 (5m54.617654425s) [seg_pos={0 0}, upload=85]: Done begin upload segment. Sep 23 13:37:09.155609312 (5m54.617701795s) [noise=true , address=145.79.227.5:4221 , piece_id=37XI3ODHTNYYIW3TYSGEZKLL4REO4C7J7W4F3M54QPBS27HU2N7Q, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.155653065 (5m54.617745548s) [noise=true , address=145.79.227.13:4801 , piece_id=KDNEMJNYSDPBUV67TZ6PFMKXECG2TLUFSFX742MK5XRK32C2NHXA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156257627 (5m54.61835011s) [noise=true , address=45.91.116.4:4938 , piece_id=YXX7C7LQDTPKFEVXO7E6HN6QNM562OWWOXPI232ZHCLSY3WE4BUA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156274629 (5m54.618367102s) [noise=true , address=145.79.227.14:4839 , piece_id=65NEICBQJXMM6N6KBD5KGI5SFTVAV7NJJKBAET6HDGKI5VD325OQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156622501 (5m54.618715004s) [noise=true , address=145.79.227.6:4259 , piece_id=LPZNOJHDSME5TCQX2HQ544ZANJCJKGWIUXFLIOMNNSZBLYPNJTVA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156777296 (5m54.618869779s) [noise=true , address=145.79.227.16:4912 , piece_id=YLC5DB6H62T67W5LPTPQIQMWGJAN5ODYJB5PXIVX6X7JNFVCTNTA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156984401 (5m54.619076884s) [noise=true , address=145.79.227.14:4838 , piece_id=MG6N5Q7OOXD4OI6CV2VCSGIOOI6VB4GYFATAZ5WZWXFCZN6EFFLQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.157122624 (5m54.619215107s) [noise=true , address=45.91.116.10:4083 , piece_id=VONNFDRLV44LE65ZK7ENEJPNRMA2IPUGGGO3QGYJZLZLWOMNLFRQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.157345388 (5m54.619437871s) [noise=true , address=145.79.227.15:4876 , piece_id=ZHNZB3WQ4QQWR2LO6AQ64ONLIZ2EKCOBYGXFNR5FBY4CZL6QLVMA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.158372935 (5m54.620465408s) [noise=true , address=45.91.116.14:4189 , piece_id=HBFO3FNIMMF5UIDBXTFQQG6WZ4SFGJI3W6CSJ5EBAGN3D7JI6MSA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.159036498 (5m54.621128982s) [noise=true , address=45.91.116.11:4117 , piece_id=QEF7PSZ4ZAV2XM546G6QVKSJVH4JGCSYRAU572ABBMRGKVP23N6A, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.159800133 (5m54.621892616s) [noise=true , address=145.79.227.12:4765 , piece_id=H4PQGKP3XUGR6ARQ3UW3ZVPNCPS5WA5NM6FYJG5TBU3FALNXQYNA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.161637361 (5m54.623729844s) [noise=true , address=145.79.227.17:4970 , piece_id=IW4WTHW5W4SFOYMCXENR3DGYCA2OQJ4OO5RSXPYX6SUIGWKA4LBA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.163221126 (5m54.62531361s) [noise=true , address=77.90.31.11:4553 , piece_id=7XXKVSGNIYW5ZN6G5AV3JWP56QHF2HI4KOXJZ2DNOSGDHVZXDQTQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.163665793 (5m54.625758277s) [noise=true , address=145.79.227.11:4731 , piece_id=XR2GN3LWECQJ4OTSM44QXNW4B6GE7P5C5EFZFKJXD2M4GCMJ54VQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.164366027 (5m54.62645851s) [noise=true , address=145.79.227.12:4767 , piece_id=7JPOEJPRMBWO5MCNTISEDMBSYFACGTPJJUCBDX6EVYUKR3E7MYSA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:10.933255114 (5m56.395347597s) [upload=85]: Next returned nil segment Sep 23 13:37:10.933279932 (5m56.395372415s) [upload=85]: Waiting for error group managing segments... Sep 23 13:37:11.355919055 (5m56.818011548s) [noise=true , address=45.91.116.10:4083 , piece_id=VONNFDRLV44LE65ZK7ENEJPNRMA2IPUGGGO3QGYJZLZLWOMNLFRQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.366419560 (5m56.828512043s) [noise=true , address=45.91.116.4:4938 , piece_id=YXX7C7LQDTPKFEVXO7E6HN6QNM562OWWOXPI232ZHCLSY3WE4BUA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.373438585 (5m56.835531068s) [noise=true , address=45.91.116.11:4117 , piece_id=QEF7PSZ4ZAV2XM546G6QVKSJVH4JGCSYRAU572ABBMRGKVP23N6A, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.388282841 (5m56.850375324s) [noise=true , address=45.91.116.14:4189 , piece_id=HBFO3FNIMMF5UIDBXTFQQG6WZ4SFGJI3W6CSJ5EBAGN3D7JI6MSA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.396367205 (5m56.858459688s) [noise=true , address=77.90.31.11:4553 , piece_id=7XXKVSGNIYW5ZN6G5AV3JWP56QHF2HI4KOXJZ2DNOSGDHVZXDQTQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.050842867 (5m57.51293534s) [noise=true , address=145.79.227.13:4801 , piece_id=KDNEMJNYSDPBUV67TZ6PFMKXECG2TLUFSFX742MK5XRK32C2NHXA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.054995253 (5m57.517087736s) [noise=true , address=145.79.227.15:4876 , piece_id=ZHNZB3WQ4QQWR2LO6AQ64ONLIZ2EKCOBYGXFNR5FBY4CZL6QLVMA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.072213210 (5m57.534305693s) [noise=true , address=145.79.227.17:4970 , piece_id=IW4WTHW5W4SFOYMCXENR3DGYCA2OQJ4OO5RSXPYX6SUIGWKA4LBA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.101168265 (5m57.563260748s) [noise=true , address=145.79.227.6:4259 , piece_id=LPZNOJHDSME5TCQX2HQ544ZANJCJKGWIUXFLIOMNNSZBLYPNJTVA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.106130595 (5m57.568223078s) [noise=true , address=145.79.227.12:4765 , piece_id=H4PQGKP3XUGR6ARQ3UW3ZVPNCPS5WA5NM6FYJG5TBU3FALNXQYNA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.118733793 (5m57.580826277s) [noise=true , address=145.79.227.14:4839 , piece_id=65NEICBQJXMM6N6KBD5KGI5SFTVAV7NJJKBAET6HDGKI5VD325OQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.508815721 (5m57.970908204s) [noise=true , address=145.79.227.11:4731 , piece_id=XR2GN3LWECQJ4OTSM44QXNW4B6GE7P5C5EFZFKJXD2M4GCMJ54VQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:13.008461678 (5m58.470554161s) [noise=true , address=145.79.227.16:4912 , piece_id=YLC5DB6H62T67W5LPTPQIQMWGJAN5ODYJB5PXIVX6X7JNFVCTNTA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:13.008503868 (5m58.470596351s) [seg_pos={0 0}, upload=85]: Segment reached optimal threshold of 13 pieces. Sep 23 13:37:13.008675254 (5m58.470767738s) [noise=true , address=145.79.227.12:4767 , piece_id=7JPOEJPRMBWO5MCNTISEDMBSYFACGTPJJUCBDX6EVYUKR3E7MYSA, seg_pos={0 0}, upload=85]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5): protocol: write tcp 188.166.225.170:41718->145.79.227.12:4767: use of closed network connection; write tcp 188.166.225.170:41718->145.79.227.12:4767: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41718->145.79.227.12:4767: use of closed network connection Sep 23 13:37:13.008689582 (5m58.470782065s) [noise=true , address=145.79.227.12:4767 , piece_id=7JPOEJPRMBWO5MCNTISEDMBSYFACGTPJJUCBDX6EVYUKR3E7MYSA, seg_pos={0 0}, upload=85]: Long tail cancellation due to optimal threshold Sep 23 13:37:13.008703729 (5m58.470796212s) [noise=true , address=145.79.227.5:4221 , piece_id=37XI3ODHTNYYIW3TYSGEZKLL4REO4C7J7W4F3M54QPBS27HU2N7Q, seg_pos={0 0}, upload=85]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR): protocol: expected piece hash; context canceled; EOF Sep 23 13:37:13.008716322 (5m58.470808806s) [noise=true , address=145.79.227.5:4221 , piece_id=37XI3ODHTNYYIW3TYSGEZKLL4REO4C7J7W4F3M54QPBS27HU2N7Q, seg_pos={0 0}, upload=85]: Long tail cancellation due to optimal threshold Sep 23 13:37:13.008800553 (5m58.470893037s) [noise=true , address=145.79.227.14:4838 , piece_id=MG6N5Q7OOXD4OI6CV2VCSGIOOI6VB4GYFATAZ5WZWXFCZN6EFFLQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: expected piece hash; context canceled; EOF Sep 23 13:37:13.008809150 (5m58.470901633s) [noise=true , address=145.79.227.14:4838 , piece_id=MG6N5Q7OOXD4OI6CV2VCSGIOOI6VB4GYFATAZ5WZWXFCZN6EFFLQ, seg_pos={0 0}, upload=85]: Long tail cancellation due to optimal threshold Sep 23 13:37:13.008829819 (5m58.470922302s) [seg_pos={0 0}, upload=85]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 23 13:37:13.008908238 (5m58.471000721s) [upload=85]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 23 13:37:13.008914450 (5m58.471006923s) [upload=85]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 23 13:37:13.352201772 (5m58.814294255s) [upload=85]: Done uploading segments... Sep 23 13:37:13.352227461 (5m58.814319944s) [upload=85]: Upload finished. err: <nil> Duration: 7.660828169s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 7.803240149s FAIL (2025-09-23 13:50:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: context deadline exceeded ======LOGS====== Sep 23 13:37:08.880516737 (5m54.34260922s) [upload=85]: Starting upload Sep 23 13:37:08.880590076 (5m54.342682549s) [upload=85]: Done starting upload Sep 23 13:37:08.880678955 (5m54.342771438s) [upload=85]: Uploading segments... Sep 23 13:37:08.880816417 (5m54.3429089s) [upload=85]: Got next segment. Inline: false Sep 23 13:37:08.880868837 (5m54.34296132s) [upload=85]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 13:37:08.880873937 (5m54.34296642s) [upload=85]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 13:37:09.155479775 (5m54.617572258s) [seg_pos={0 0}, upload=85]: Begin upload segment... Sep 23 13:37:09.155561942 (5m54.617654425s) [seg_pos={0 0}, upload=85]: Done begin upload segment. Sep 23 13:37:09.155609312 (5m54.617701795s) [noise=true , address=145.79.227.5:4221 , piece_id=37XI3ODHTNYYIW3TYSGEZKLL4REO4C7J7W4F3M54QPBS27HU2N7Q, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.155653065 (5m54.617745548s) [noise=true , address=145.79.227.13:4801 , piece_id=KDNEMJNYSDPBUV67TZ6PFMKXECG2TLUFSFX742MK5XRK32C2NHXA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156257627 (5m54.61835011s) [noise=true , address=45.91.116.4:4938 , piece_id=YXX7C7LQDTPKFEVXO7E6HN6QNM562OWWOXPI232ZHCLSY3WE4BUA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156274629 (5m54.618367102s) [noise=true , address=145.79.227.14:4839 , piece_id=65NEICBQJXMM6N6KBD5KGI5SFTVAV7NJJKBAET6HDGKI5VD325OQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156622501 (5m54.618715004s) [noise=true , address=145.79.227.6:4259 , piece_id=LPZNOJHDSME5TCQX2HQ544ZANJCJKGWIUXFLIOMNNSZBLYPNJTVA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156777296 (5m54.618869779s) [noise=true , address=145.79.227.16:4912 , piece_id=YLC5DB6H62T67W5LPTPQIQMWGJAN5ODYJB5PXIVX6X7JNFVCTNTA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156984401 (5m54.619076884s) [noise=true , address=145.79.227.14:4838 , piece_id=MG6N5Q7OOXD4OI6CV2VCSGIOOI6VB4GYFATAZ5WZWXFCZN6EFFLQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.157122624 (5m54.619215107s) [noise=true , address=45.91.116.10:4083 , piece_id=VONNFDRLV44LE65ZK7ENEJPNRMA2IPUGGGO3QGYJZLZLWOMNLFRQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.157345388 (5m54.619437871s) [noise=true , address=145.79.227.15:4876 , piece_id=ZHNZB3WQ4QQWR2LO6AQ64ONLIZ2EKCOBYGXFNR5FBY4CZL6QLVMA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.158372935 (5m54.620465408s) [noise=true , address=45.91.116.14:4189 , piece_id=HBFO3FNIMMF5UIDBXTFQQG6WZ4SFGJI3W6CSJ5EBAGN3D7JI6MSA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.159036498 (5m54.621128982s) [noise=true , address=45.91.116.11:4117 , piece_id=QEF7PSZ4ZAV2XM546G6QVKSJVH4JGCSYRAU572ABBMRGKVP23N6A, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.159800133 (5m54.621892616s) [noise=true , address=145.79.227.12:4765 , piece_id=H4PQGKP3XUGR6ARQ3UW3ZVPNCPS5WA5NM6FYJG5TBU3FALNXQYNA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.161637361 (5m54.623729844s) [noise=true , address=145.79.227.17:4970 , piece_id=IW4WTHW5W4SFOYMCXENR3DGYCA2OQJ4OO5RSXPYX6SUIGWKA4LBA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.163221126 (5m54.62531361s) [noise=true , address=77.90.31.11:4553 , piece_id=7XXKVSGNIYW5ZN6G5AV3JWP56QHF2HI4KOXJZ2DNOSGDHVZXDQTQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.163665793 (5m54.625758277s) [noise=true , address=145.79.227.11:4731 , piece_id=XR2GN3LWECQJ4OTSM44QXNW4B6GE7P5C5EFZFKJXD2M4GCMJ54VQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.164366027 (5m54.62645851s) [noise=true , address=145.79.227.12:4767 , piece_id=7JPOEJPRMBWO5MCNTISEDMBSYFACGTPJJUCBDX6EVYUKR3E7MYSA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:10.933255114 (5m56.395347597s) [upload=85]: Next returned nil segment Sep 23 13:37:10.933279932 (5m56.395372415s) [upload=85]: Waiting for error group managing segments... Sep 23 13:37:11.355919055 (5m56.818011548s) [noise=true , address=45.91.116.10:4083 , piece_id=VONNFDRLV44LE65ZK7ENEJPNRMA2IPUGGGO3QGYJZLZLWOMNLFRQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.366419560 (5m56.828512043s) [noise=true , address=45.91.116.4:4938 , piece_id=YXX7C7LQDTPKFEVXO7E6HN6QNM562OWWOXPI232ZHCLSY3WE4BUA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.373438585 (5m56.835531068s) [noise=true , address=45.91.116.11:4117 , piece_id=QEF7PSZ4ZAV2XM546G6QVKSJVH4JGCSYRAU572ABBMRGKVP23N6A, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.388282841 (5m56.850375324s) [noise=true , address=45.91.116.14:4189 , piece_id=HBFO3FNIMMF5UIDBXTFQQG6WZ4SFGJI3W6CSJ5EBAGN3D7JI6MSA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.396367205 (5m56.858459688s) [noise=true , address=77.90.31.11:4553 , piece_id=7XXKVSGNIYW5ZN6G5AV3JWP56QHF2HI4KOXJZ2DNOSGDHVZXDQTQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.050842867 (5m57.51293534s) [noise=true , address=145.79.227.13:4801 , piece_id=KDNEMJNYSDPBUV67TZ6PFMKXECG2TLUFSFX742MK5XRK32C2NHXA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.054995253 (5m57.517087736s) [noise=true , address=145.79.227.15:4876 , piece_id=ZHNZB3WQ4QQWR2LO6AQ64ONLIZ2EKCOBYGXFNR5FBY4CZL6QLVMA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.072213210 (5m57.534305693s) [noise=true , address=145.79.227.17:4970 , piece_id=IW4WTHW5W4SFOYMCXENR3DGYCA2OQJ4OO5RSXPYX6SUIGWKA4LBA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.101168265 (5m57.563260748s) [noise=true , address=145.79.227.6:4259 , piece_id=LPZNOJHDSME5TCQX2HQ544ZANJCJKGWIUXFLIOMNNSZBLYPNJTVA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.106130595 (5m57.568223078s) [noise=true , address=145.79.227.12:4765 , piece_id=H4PQGKP3XUGR6ARQ3UW3ZVPNCPS5WA5NM6FYJG5TBU3FALNXQYNA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.118733793 (5m57.580826277s) [noise=true , address=145.79.227.14:4839 , piece_id=65NEICBQJXMM6N6KBD5KGI5SFTVAV7NJJKBAET6HDGKI5VD325OQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.508815721 (5m57.970908204s) [noise=true , address=145.79.227.11:4731 , piece_id=XR2GN3LWECQJ4OTSM44QXNW4B6GE7P5C5EFZFKJXD2M4GCMJ54VQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:13.008461678 (5m58.470554161s) [noise=true , address=145.79.227.16:4912 , piece_id=YLC5DB6H62T67W5LPTPQIQMWGJAN5ODYJB5PXIVX6X7JNFVCTNTA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:13.008503868 (5m58.470596351s) [seg_pos={0 0}, upload=85]: Segment reached optimal threshold of 13 pieces. Sep 23 13:37:13.008675254 (5m58.470767738s) [noise=true , address=145.79.227.12:4767 , piece_id=7JPOEJPRMBWO5MCNTISEDMBSYFACGTPJJUCBDX6EVYUKR3E7MYSA, seg_pos={0 0}, upload=85]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5): protocol: write tcp 188.166.225.170:41718->145.79.227.12:4767: use of closed network connection; write tcp 188.166.225.170:41718->145.79.227.12:4767: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41718->145.79.227.12:4767: use of closed network connection Sep 23 13:37:13.008689582 (5m58.470782065s) [noise=true , address=145.79.227.12:4767 , piece_id=7JPOEJPRMBWO5MCNTISEDMBSYFACGTPJJUCBDX6EVYUKR3E7MYSA, seg_pos={0 0}, upload=85]: Long tail cancellation due to optimal threshold Sep 23 13:37:13.008703729 (5m58.470796212s) [noise=true , address=145.79.227.5:4221 , piece_id=37XI3ODHTNYYIW3TYSGEZKLL4REO4C7J7W4F3M54QPBS27HU2N7Q, seg_pos={0 0}, upload=85]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR): protocol: expected piece hash; context canceled; EOF Sep 23 13:37:13.008716322 (5m58.470808806s) [noise=true , address=145.79.227.5:4221 , piece_id=37XI3ODHTNYYIW3TYSGEZKLL4REO4C7J7W4F3M54QPBS27HU2N7Q, seg_pos={0 0}, upload=85]: Long tail cancellation due to optimal threshold Sep 23 13:37:13.008800553 (5m58.470893037s) [noise=true , address=145.79.227.14:4838 , piece_id=MG6N5Q7OOXD4OI6CV2VCSGIOOI6VB4GYFATAZ5WZWXFCZN6EFFLQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: expected piece hash; context canceled; EOF Sep 23 13:37:13.008809150 (5m58.470901633s) [noise=true , address=145.79.227.14:4838 , piece_id=MG6N5Q7OOXD4OI6CV2VCSGIOOI6VB4GYFATAZ5WZWXFCZN6EFFLQ, seg_pos={0 0}, upload=85]: Long tail cancellation due to optimal threshold Sep 23 13:37:13.008829819 (5m58.470922302s) [seg_pos={0 0}, upload=85]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 23 13:37:13.008908238 (5m58.471000721s) [upload=85]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 23 13:37:13.008914450 (5m58.471006923s) [upload=85]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 23 13:37:13.352201772 (5m58.814294255s) [upload=85]: Done uploading segments... Sep 23 13:37:13.352227461 (5m58.814319944s) [upload=85]: Upload finished. err: <nil> Duration: 7.803270858s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 6.345219236s FAIL (2025-09-23 13:49:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: context deadline exceeded ======LOGS====== Sep 23 13:37:08.880516737 (5m54.34260922s) [upload=85]: Starting upload Sep 23 13:37:08.880590076 (5m54.342682549s) [upload=85]: Done starting upload Sep 23 13:37:08.880678955 (5m54.342771438s) [upload=85]: Uploading segments... Sep 23 13:37:08.880816417 (5m54.3429089s) [upload=85]: Got next segment. Inline: false Sep 23 13:37:08.880868837 (5m54.34296132s) [upload=85]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 13:37:08.880873937 (5m54.34296642s) [upload=85]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 13:37:09.155479775 (5m54.617572258s) [seg_pos={0 0}, upload=85]: Begin upload segment... Sep 23 13:37:09.155561942 (5m54.617654425s) [seg_pos={0 0}, upload=85]: Done begin upload segment. Sep 23 13:37:09.155609312 (5m54.617701795s) [noise=true , address=145.79.227.5:4221 , piece_id=37XI3ODHTNYYIW3TYSGEZKLL4REO4C7J7W4F3M54QPBS27HU2N7Q, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.155653065 (5m54.617745548s) [noise=true , address=145.79.227.13:4801 , piece_id=KDNEMJNYSDPBUV67TZ6PFMKXECG2TLUFSFX742MK5XRK32C2NHXA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156257627 (5m54.61835011s) [noise=true , address=45.91.116.4:4938 , piece_id=YXX7C7LQDTPKFEVXO7E6HN6QNM562OWWOXPI232ZHCLSY3WE4BUA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156274629 (5m54.618367102s) [noise=true , address=145.79.227.14:4839 , piece_id=65NEICBQJXMM6N6KBD5KGI5SFTVAV7NJJKBAET6HDGKI5VD325OQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156622501 (5m54.618715004s) [noise=true , address=145.79.227.6:4259 , piece_id=LPZNOJHDSME5TCQX2HQ544ZANJCJKGWIUXFLIOMNNSZBLYPNJTVA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156777296 (5m54.618869779s) [noise=true , address=145.79.227.16:4912 , piece_id=YLC5DB6H62T67W5LPTPQIQMWGJAN5ODYJB5PXIVX6X7JNFVCTNTA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156984401 (5m54.619076884s) [noise=true , address=145.79.227.14:4838 , piece_id=MG6N5Q7OOXD4OI6CV2VCSGIOOI6VB4GYFATAZ5WZWXFCZN6EFFLQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.157122624 (5m54.619215107s) [noise=true , address=45.91.116.10:4083 , piece_id=VONNFDRLV44LE65ZK7ENEJPNRMA2IPUGGGO3QGYJZLZLWOMNLFRQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.157345388 (5m54.619437871s) [noise=true , address=145.79.227.15:4876 , piece_id=ZHNZB3WQ4QQWR2LO6AQ64ONLIZ2EKCOBYGXFNR5FBY4CZL6QLVMA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.158372935 (5m54.620465408s) [noise=true , address=45.91.116.14:4189 , piece_id=HBFO3FNIMMF5UIDBXTFQQG6WZ4SFGJI3W6CSJ5EBAGN3D7JI6MSA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.159036498 (5m54.621128982s) [noise=true , address=45.91.116.11:4117 , piece_id=QEF7PSZ4ZAV2XM546G6QVKSJVH4JGCSYRAU572ABBMRGKVP23N6A, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.159800133 (5m54.621892616s) [noise=true , address=145.79.227.12:4765 , piece_id=H4PQGKP3XUGR6ARQ3UW3ZVPNCPS5WA5NM6FYJG5TBU3FALNXQYNA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.161637361 (5m54.623729844s) [noise=true , address=145.79.227.17:4970 , piece_id=IW4WTHW5W4SFOYMCXENR3DGYCA2OQJ4OO5RSXPYX6SUIGWKA4LBA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.163221126 (5m54.62531361s) [noise=true , address=77.90.31.11:4553 , piece_id=7XXKVSGNIYW5ZN6G5AV3JWP56QHF2HI4KOXJZ2DNOSGDHVZXDQTQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.163665793 (5m54.625758277s) [noise=true , address=145.79.227.11:4731 , piece_id=XR2GN3LWECQJ4OTSM44QXNW4B6GE7P5C5EFZFKJXD2M4GCMJ54VQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.164366027 (5m54.62645851s) [noise=true , address=145.79.227.12:4767 , piece_id=7JPOEJPRMBWO5MCNTISEDMBSYFACGTPJJUCBDX6EVYUKR3E7MYSA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:10.933255114 (5m56.395347597s) [upload=85]: Next returned nil segment Sep 23 13:37:10.933279932 (5m56.395372415s) [upload=85]: Waiting for error group managing segments... Sep 23 13:37:11.355919055 (5m56.818011548s) [noise=true , address=45.91.116.10:4083 , piece_id=VONNFDRLV44LE65ZK7ENEJPNRMA2IPUGGGO3QGYJZLZLWOMNLFRQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.366419560 (5m56.828512043s) [noise=true , address=45.91.116.4:4938 , piece_id=YXX7C7LQDTPKFEVXO7E6HN6QNM562OWWOXPI232ZHCLSY3WE4BUA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.373438585 (5m56.835531068s) [noise=true , address=45.91.116.11:4117 , piece_id=QEF7PSZ4ZAV2XM546G6QVKSJVH4JGCSYRAU572ABBMRGKVP23N6A, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.388282841 (5m56.850375324s) [noise=true , address=45.91.116.14:4189 , piece_id=HBFO3FNIMMF5UIDBXTFQQG6WZ4SFGJI3W6CSJ5EBAGN3D7JI6MSA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.396367205 (5m56.858459688s) [noise=true , address=77.90.31.11:4553 , piece_id=7XXKVSGNIYW5ZN6G5AV3JWP56QHF2HI4KOXJZ2DNOSGDHVZXDQTQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.050842867 (5m57.51293534s) [noise=true , address=145.79.227.13:4801 , piece_id=KDNEMJNYSDPBUV67TZ6PFMKXECG2TLUFSFX742MK5XRK32C2NHXA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.054995253 (5m57.517087736s) [noise=true , address=145.79.227.15:4876 , piece_id=ZHNZB3WQ4QQWR2LO6AQ64ONLIZ2EKCOBYGXFNR5FBY4CZL6QLVMA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.072213210 (5m57.534305693s) [noise=true , address=145.79.227.17:4970 , piece_id=IW4WTHW5W4SFOYMCXENR3DGYCA2OQJ4OO5RSXPYX6SUIGWKA4LBA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.101168265 (5m57.563260748s) [noise=true , address=145.79.227.6:4259 , piece_id=LPZNOJHDSME5TCQX2HQ544ZANJCJKGWIUXFLIOMNNSZBLYPNJTVA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.106130595 (5m57.568223078s) [noise=true , address=145.79.227.12:4765 , piece_id=H4PQGKP3XUGR6ARQ3UW3ZVPNCPS5WA5NM6FYJG5TBU3FALNXQYNA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.118733793 (5m57.580826277s) [noise=true , address=145.79.227.14:4839 , piece_id=65NEICBQJXMM6N6KBD5KGI5SFTVAV7NJJKBAET6HDGKI5VD325OQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.508815721 (5m57.970908204s) [noise=true , address=145.79.227.11:4731 , piece_id=XR2GN3LWECQJ4OTSM44QXNW4B6GE7P5C5EFZFKJXD2M4GCMJ54VQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:13.008461678 (5m58.470554161s) [noise=true , address=145.79.227.16:4912 , piece_id=YLC5DB6H62T67W5LPTPQIQMWGJAN5ODYJB5PXIVX6X7JNFVCTNTA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:13.008503868 (5m58.470596351s) [seg_pos={0 0}, upload=85]: Segment reached optimal threshold of 13 pieces. Sep 23 13:37:13.008675254 (5m58.470767738s) [noise=true , address=145.79.227.12:4767 , piece_id=7JPOEJPRMBWO5MCNTISEDMBSYFACGTPJJUCBDX6EVYUKR3E7MYSA, seg_pos={0 0}, upload=85]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5): protocol: write tcp 188.166.225.170:41718->145.79.227.12:4767: use of closed network connection; write tcp 188.166.225.170:41718->145.79.227.12:4767: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41718->145.79.227.12:4767: use of closed network connection Sep 23 13:37:13.008689582 (5m58.470782065s) [noise=true , address=145.79.227.12:4767 , piece_id=7JPOEJPRMBWO5MCNTISEDMBSYFACGTPJJUCBDX6EVYUKR3E7MYSA, seg_pos={0 0}, upload=85]: Long tail cancellation due to optimal threshold Sep 23 13:37:13.008703729 (5m58.470796212s) [noise=true , address=145.79.227.5:4221 , piece_id=37XI3ODHTNYYIW3TYSGEZKLL4REO4C7J7W4F3M54QPBS27HU2N7Q, seg_pos={0 0}, upload=85]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR): protocol: expected piece hash; context canceled; EOF Sep 23 13:37:13.008716322 (5m58.470808806s) [noise=true , address=145.79.227.5:4221 , piece_id=37XI3ODHTNYYIW3TYSGEZKLL4REO4C7J7W4F3M54QPBS27HU2N7Q, seg_pos={0 0}, upload=85]: Long tail cancellation due to optimal threshold Sep 23 13:37:13.008800553 (5m58.470893037s) [noise=true , address=145.79.227.14:4838 , piece_id=MG6N5Q7OOXD4OI6CV2VCSGIOOI6VB4GYFATAZ5WZWXFCZN6EFFLQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: expected piece hash; context canceled; EOF Sep 23 13:37:13.008809150 (5m58.470901633s) [noise=true , address=145.79.227.14:4838 , piece_id=MG6N5Q7OOXD4OI6CV2VCSGIOOI6VB4GYFATAZ5WZWXFCZN6EFFLQ, seg_pos={0 0}, upload=85]: Long tail cancellation due to optimal threshold Sep 23 13:37:13.008829819 (5m58.470922302s) [seg_pos={0 0}, upload=85]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 23 13:37:13.008908238 (5m58.471000721s) [upload=85]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 23 13:37:13.008914450 (5m58.471006923s) [upload=85]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 23 13:37:13.352201772 (5m58.814294255s) [upload=85]: Done uploading segments... Sep 23 13:37:13.352227461 (5m58.814319944s) [upload=85]: Upload finished. err: <nil> Duration: 6.345250536s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 6.436324065s FAIL (2025-09-23 13:48:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: context deadline exceeded ======LOGS====== Sep 23 13:37:08.880516737 (5m54.34260922s) [upload=85]: Starting upload Sep 23 13:37:08.880590076 (5m54.342682549s) [upload=85]: Done starting upload Sep 23 13:37:08.880678955 (5m54.342771438s) [upload=85]: Uploading segments... Sep 23 13:37:08.880816417 (5m54.3429089s) [upload=85]: Got next segment. Inline: false Sep 23 13:37:08.880868837 (5m54.34296132s) [upload=85]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 13:37:08.880873937 (5m54.34296642s) [upload=85]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 13:37:09.155479775 (5m54.617572258s) [seg_pos={0 0}, upload=85]: Begin upload segment... Sep 23 13:37:09.155561942 (5m54.617654425s) [seg_pos={0 0}, upload=85]: Done begin upload segment. Sep 23 13:37:09.155609312 (5m54.617701795s) [noise=true , address=145.79.227.5:4221 , piece_id=37XI3ODHTNYYIW3TYSGEZKLL4REO4C7J7W4F3M54QPBS27HU2N7Q, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.155653065 (5m54.617745548s) [noise=true , address=145.79.227.13:4801 , piece_id=KDNEMJNYSDPBUV67TZ6PFMKXECG2TLUFSFX742MK5XRK32C2NHXA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156257627 (5m54.61835011s) [noise=true , address=45.91.116.4:4938 , piece_id=YXX7C7LQDTPKFEVXO7E6HN6QNM562OWWOXPI232ZHCLSY3WE4BUA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156274629 (5m54.618367102s) [noise=true , address=145.79.227.14:4839 , piece_id=65NEICBQJXMM6N6KBD5KGI5SFTVAV7NJJKBAET6HDGKI5VD325OQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156622501 (5m54.618715004s) [noise=true , address=145.79.227.6:4259 , piece_id=LPZNOJHDSME5TCQX2HQ544ZANJCJKGWIUXFLIOMNNSZBLYPNJTVA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156777296 (5m54.618869779s) [noise=true , address=145.79.227.16:4912 , piece_id=YLC5DB6H62T67W5LPTPQIQMWGJAN5ODYJB5PXIVX6X7JNFVCTNTA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156984401 (5m54.619076884s) [noise=true , address=145.79.227.14:4838 , piece_id=MG6N5Q7OOXD4OI6CV2VCSGIOOI6VB4GYFATAZ5WZWXFCZN6EFFLQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.157122624 (5m54.619215107s) [noise=true , address=45.91.116.10:4083 , piece_id=VONNFDRLV44LE65ZK7ENEJPNRMA2IPUGGGO3QGYJZLZLWOMNLFRQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.157345388 (5m54.619437871s) [noise=true , address=145.79.227.15:4876 , piece_id=ZHNZB3WQ4QQWR2LO6AQ64ONLIZ2EKCOBYGXFNR5FBY4CZL6QLVMA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.158372935 (5m54.620465408s) [noise=true , address=45.91.116.14:4189 , piece_id=HBFO3FNIMMF5UIDBXTFQQG6WZ4SFGJI3W6CSJ5EBAGN3D7JI6MSA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.159036498 (5m54.621128982s) [noise=true , address=45.91.116.11:4117 , piece_id=QEF7PSZ4ZAV2XM546G6QVKSJVH4JGCSYRAU572ABBMRGKVP23N6A, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.159800133 (5m54.621892616s) [noise=true , address=145.79.227.12:4765 , piece_id=H4PQGKP3XUGR6ARQ3UW3ZVPNCPS5WA5NM6FYJG5TBU3FALNXQYNA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.161637361 (5m54.623729844s) [noise=true , address=145.79.227.17:4970 , piece_id=IW4WTHW5W4SFOYMCXENR3DGYCA2OQJ4OO5RSXPYX6SUIGWKA4LBA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.163221126 (5m54.62531361s) [noise=true , address=77.90.31.11:4553 , piece_id=7XXKVSGNIYW5ZN6G5AV3JWP56QHF2HI4KOXJZ2DNOSGDHVZXDQTQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.163665793 (5m54.625758277s) [noise=true , address=145.79.227.11:4731 , piece_id=XR2GN3LWECQJ4OTSM44QXNW4B6GE7P5C5EFZFKJXD2M4GCMJ54VQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.164366027 (5m54.62645851s) [noise=true , address=145.79.227.12:4767 , piece_id=7JPOEJPRMBWO5MCNTISEDMBSYFACGTPJJUCBDX6EVYUKR3E7MYSA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:10.933255114 (5m56.395347597s) [upload=85]: Next returned nil segment Sep 23 13:37:10.933279932 (5m56.395372415s) [upload=85]: Waiting for error group managing segments... Sep 23 13:37:11.355919055 (5m56.818011548s) [noise=true , address=45.91.116.10:4083 , piece_id=VONNFDRLV44LE65ZK7ENEJPNRMA2IPUGGGO3QGYJZLZLWOMNLFRQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.366419560 (5m56.828512043s) [noise=true , address=45.91.116.4:4938 , piece_id=YXX7C7LQDTPKFEVXO7E6HN6QNM562OWWOXPI232ZHCLSY3WE4BUA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.373438585 (5m56.835531068s) [noise=true , address=45.91.116.11:4117 , piece_id=QEF7PSZ4ZAV2XM546G6QVKSJVH4JGCSYRAU572ABBMRGKVP23N6A, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.388282841 (5m56.850375324s) [noise=true , address=45.91.116.14:4189 , piece_id=HBFO3FNIMMF5UIDBXTFQQG6WZ4SFGJI3W6CSJ5EBAGN3D7JI6MSA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.396367205 (5m56.858459688s) [noise=true , address=77.90.31.11:4553 , piece_id=7XXKVSGNIYW5ZN6G5AV3JWP56QHF2HI4KOXJZ2DNOSGDHVZXDQTQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.050842867 (5m57.51293534s) [noise=true , address=145.79.227.13:4801 , piece_id=KDNEMJNYSDPBUV67TZ6PFMKXECG2TLUFSFX742MK5XRK32C2NHXA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.054995253 (5m57.517087736s) [noise=true , address=145.79.227.15:4876 , piece_id=ZHNZB3WQ4QQWR2LO6AQ64ONLIZ2EKCOBYGXFNR5FBY4CZL6QLVMA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.072213210 (5m57.534305693s) [noise=true , address=145.79.227.17:4970 , piece_id=IW4WTHW5W4SFOYMCXENR3DGYCA2OQJ4OO5RSXPYX6SUIGWKA4LBA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.101168265 (5m57.563260748s) [noise=true , address=145.79.227.6:4259 , piece_id=LPZNOJHDSME5TCQX2HQ544ZANJCJKGWIUXFLIOMNNSZBLYPNJTVA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.106130595 (5m57.568223078s) [noise=true , address=145.79.227.12:4765 , piece_id=H4PQGKP3XUGR6ARQ3UW3ZVPNCPS5WA5NM6FYJG5TBU3FALNXQYNA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.118733793 (5m57.580826277s) [noise=true , address=145.79.227.14:4839 , piece_id=65NEICBQJXMM6N6KBD5KGI5SFTVAV7NJJKBAET6HDGKI5VD325OQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.508815721 (5m57.970908204s) [noise=true , address=145.79.227.11:4731 , piece_id=XR2GN3LWECQJ4OTSM44QXNW4B6GE7P5C5EFZFKJXD2M4GCMJ54VQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:13.008461678 (5m58.470554161s) [noise=true , address=145.79.227.16:4912 , piece_id=YLC5DB6H62T67W5LPTPQIQMWGJAN5ODYJB5PXIVX6X7JNFVCTNTA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:13.008503868 (5m58.470596351s) [seg_pos={0 0}, upload=85]: Segment reached optimal threshold of 13 pieces. Sep 23 13:37:13.008675254 (5m58.470767738s) [noise=true , address=145.79.227.12:4767 , piece_id=7JPOEJPRMBWO5MCNTISEDMBSYFACGTPJJUCBDX6EVYUKR3E7MYSA, seg_pos={0 0}, upload=85]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5): protocol: write tcp 188.166.225.170:41718->145.79.227.12:4767: use of closed network connection; write tcp 188.166.225.170:41718->145.79.227.12:4767: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41718->145.79.227.12:4767: use of closed network connection Sep 23 13:37:13.008689582 (5m58.470782065s) [noise=true , address=145.79.227.12:4767 , piece_id=7JPOEJPRMBWO5MCNTISEDMBSYFACGTPJJUCBDX6EVYUKR3E7MYSA, seg_pos={0 0}, upload=85]: Long tail cancellation due to optimal threshold Sep 23 13:37:13.008703729 (5m58.470796212s) [noise=true , address=145.79.227.5:4221 , piece_id=37XI3ODHTNYYIW3TYSGEZKLL4REO4C7J7W4F3M54QPBS27HU2N7Q, seg_pos={0 0}, upload=85]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR): protocol: expected piece hash; context canceled; EOF Sep 23 13:37:13.008716322 (5m58.470808806s) [noise=true , address=145.79.227.5:4221 , piece_id=37XI3ODHTNYYIW3TYSGEZKLL4REO4C7J7W4F3M54QPBS27HU2N7Q, seg_pos={0 0}, upload=85]: Long tail cancellation due to optimal threshold Sep 23 13:37:13.008800553 (5m58.470893037s) [noise=true , address=145.79.227.14:4838 , piece_id=MG6N5Q7OOXD4OI6CV2VCSGIOOI6VB4GYFATAZ5WZWXFCZN6EFFLQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: expected piece hash; context canceled; EOF Sep 23 13:37:13.008809150 (5m58.470901633s) [noise=true , address=145.79.227.14:4838 , piece_id=MG6N5Q7OOXD4OI6CV2VCSGIOOI6VB4GYFATAZ5WZWXFCZN6EFFLQ, seg_pos={0 0}, upload=85]: Long tail cancellation due to optimal threshold Sep 23 13:37:13.008829819 (5m58.470922302s) [seg_pos={0 0}, upload=85]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 23 13:37:13.008908238 (5m58.471000721s) [upload=85]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 23 13:37:13.008914450 (5m58.471006923s) [upload=85]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 23 13:37:13.352201772 (5m58.814294255s) [upload=85]: Done uploading segments... Sep 23 13:37:13.352227461 (5m58.814319944s) [upload=85]: Upload finished. err: <nil> Duration: 6.436349963s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 7.554846378s FAIL (2025-09-23 13:48:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: context deadline exceeded ======LOGS====== Sep 23 13:37:08.880516737 (5m54.34260922s) [upload=85]: Starting upload Sep 23 13:37:08.880590076 (5m54.342682549s) [upload=85]: Done starting upload Sep 23 13:37:08.880678955 (5m54.342771438s) [upload=85]: Uploading segments... Sep 23 13:37:08.880816417 (5m54.3429089s) [upload=85]: Got next segment. Inline: false Sep 23 13:37:08.880868837 (5m54.34296132s) [upload=85]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 13:37:08.880873937 (5m54.34296642s) [upload=85]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 13:37:09.155479775 (5m54.617572258s) [seg_pos={0 0}, upload=85]: Begin upload segment... Sep 23 13:37:09.155561942 (5m54.617654425s) [seg_pos={0 0}, upload=85]: Done begin upload segment. Sep 23 13:37:09.155609312 (5m54.617701795s) [noise=true , address=145.79.227.5:4221 , piece_id=37XI3ODHTNYYIW3TYSGEZKLL4REO4C7J7W4F3M54QPBS27HU2N7Q, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.155653065 (5m54.617745548s) [noise=true , address=145.79.227.13:4801 , piece_id=KDNEMJNYSDPBUV67TZ6PFMKXECG2TLUFSFX742MK5XRK32C2NHXA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156257627 (5m54.61835011s) [noise=true , address=45.91.116.4:4938 , piece_id=YXX7C7LQDTPKFEVXO7E6HN6QNM562OWWOXPI232ZHCLSY3WE4BUA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156274629 (5m54.618367102s) [noise=true , address=145.79.227.14:4839 , piece_id=65NEICBQJXMM6N6KBD5KGI5SFTVAV7NJJKBAET6HDGKI5VD325OQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156622501 (5m54.618715004s) [noise=true , address=145.79.227.6:4259 , piece_id=LPZNOJHDSME5TCQX2HQ544ZANJCJKGWIUXFLIOMNNSZBLYPNJTVA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156777296 (5m54.618869779s) [noise=true , address=145.79.227.16:4912 , piece_id=YLC5DB6H62T67W5LPTPQIQMWGJAN5ODYJB5PXIVX6X7JNFVCTNTA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156984401 (5m54.619076884s) [noise=true , address=145.79.227.14:4838 , piece_id=MG6N5Q7OOXD4OI6CV2VCSGIOOI6VB4GYFATAZ5WZWXFCZN6EFFLQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.157122624 (5m54.619215107s) [noise=true , address=45.91.116.10:4083 , piece_id=VONNFDRLV44LE65ZK7ENEJPNRMA2IPUGGGO3QGYJZLZLWOMNLFRQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.157345388 (5m54.619437871s) [noise=true , address=145.79.227.15:4876 , piece_id=ZHNZB3WQ4QQWR2LO6AQ64ONLIZ2EKCOBYGXFNR5FBY4CZL6QLVMA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.158372935 (5m54.620465408s) [noise=true , address=45.91.116.14:4189 , piece_id=HBFO3FNIMMF5UIDBXTFQQG6WZ4SFGJI3W6CSJ5EBAGN3D7JI6MSA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.159036498 (5m54.621128982s) [noise=true , address=45.91.116.11:4117 , piece_id=QEF7PSZ4ZAV2XM546G6QVKSJVH4JGCSYRAU572ABBMRGKVP23N6A, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.159800133 (5m54.621892616s) [noise=true , address=145.79.227.12:4765 , piece_id=H4PQGKP3XUGR6ARQ3UW3ZVPNCPS5WA5NM6FYJG5TBU3FALNXQYNA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.161637361 (5m54.623729844s) [noise=true , address=145.79.227.17:4970 , piece_id=IW4WTHW5W4SFOYMCXENR3DGYCA2OQJ4OO5RSXPYX6SUIGWKA4LBA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.163221126 (5m54.62531361s) [noise=true , address=77.90.31.11:4553 , piece_id=7XXKVSGNIYW5ZN6G5AV3JWP56QHF2HI4KOXJZ2DNOSGDHVZXDQTQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.163665793 (5m54.625758277s) [noise=true , address=145.79.227.11:4731 , piece_id=XR2GN3LWECQJ4OTSM44QXNW4B6GE7P5C5EFZFKJXD2M4GCMJ54VQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.164366027 (5m54.62645851s) [noise=true , address=145.79.227.12:4767 , piece_id=7JPOEJPRMBWO5MCNTISEDMBSYFACGTPJJUCBDX6EVYUKR3E7MYSA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:10.933255114 (5m56.395347597s) [upload=85]: Next returned nil segment Sep 23 13:37:10.933279932 (5m56.395372415s) [upload=85]: Waiting for error group managing segments... Sep 23 13:37:11.355919055 (5m56.818011548s) [noise=true , address=45.91.116.10:4083 , piece_id=VONNFDRLV44LE65ZK7ENEJPNRMA2IPUGGGO3QGYJZLZLWOMNLFRQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.366419560 (5m56.828512043s) [noise=true , address=45.91.116.4:4938 , piece_id=YXX7C7LQDTPKFEVXO7E6HN6QNM562OWWOXPI232ZHCLSY3WE4BUA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.373438585 (5m56.835531068s) [noise=true , address=45.91.116.11:4117 , piece_id=QEF7PSZ4ZAV2XM546G6QVKSJVH4JGCSYRAU572ABBMRGKVP23N6A, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.388282841 (5m56.850375324s) [noise=true , address=45.91.116.14:4189 , piece_id=HBFO3FNIMMF5UIDBXTFQQG6WZ4SFGJI3W6CSJ5EBAGN3D7JI6MSA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.396367205 (5m56.858459688s) [noise=true , address=77.90.31.11:4553 , piece_id=7XXKVSGNIYW5ZN6G5AV3JWP56QHF2HI4KOXJZ2DNOSGDHVZXDQTQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.050842867 (5m57.51293534s) [noise=true , address=145.79.227.13:4801 , piece_id=KDNEMJNYSDPBUV67TZ6PFMKXECG2TLUFSFX742MK5XRK32C2NHXA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.054995253 (5m57.517087736s) [noise=true , address=145.79.227.15:4876 , piece_id=ZHNZB3WQ4QQWR2LO6AQ64ONLIZ2EKCOBYGXFNR5FBY4CZL6QLVMA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.072213210 (5m57.534305693s) [noise=true , address=145.79.227.17:4970 , piece_id=IW4WTHW5W4SFOYMCXENR3DGYCA2OQJ4OO5RSXPYX6SUIGWKA4LBA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.101168265 (5m57.563260748s) [noise=true , address=145.79.227.6:4259 , piece_id=LPZNOJHDSME5TCQX2HQ544ZANJCJKGWIUXFLIOMNNSZBLYPNJTVA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.106130595 (5m57.568223078s) [noise=true , address=145.79.227.12:4765 , piece_id=H4PQGKP3XUGR6ARQ3UW3ZVPNCPS5WA5NM6FYJG5TBU3FALNXQYNA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.118733793 (5m57.580826277s) [noise=true , address=145.79.227.14:4839 , piece_id=65NEICBQJXMM6N6KBD5KGI5SFTVAV7NJJKBAET6HDGKI5VD325OQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.508815721 (5m57.970908204s) [noise=true , address=145.79.227.11:4731 , piece_id=XR2GN3LWECQJ4OTSM44QXNW4B6GE7P5C5EFZFKJXD2M4GCMJ54VQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:13.008461678 (5m58.470554161s) [noise=true , address=145.79.227.16:4912 , piece_id=YLC5DB6H62T67W5LPTPQIQMWGJAN5ODYJB5PXIVX6X7JNFVCTNTA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:13.008503868 (5m58.470596351s) [seg_pos={0 0}, upload=85]: Segment reached optimal threshold of 13 pieces. Sep 23 13:37:13.008675254 (5m58.470767738s) [noise=true , address=145.79.227.12:4767 , piece_id=7JPOEJPRMBWO5MCNTISEDMBSYFACGTPJJUCBDX6EVYUKR3E7MYSA, seg_pos={0 0}, upload=85]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5): protocol: write tcp 188.166.225.170:41718->145.79.227.12:4767: use of closed network connection; write tcp 188.166.225.170:41718->145.79.227.12:4767: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41718->145.79.227.12:4767: use of closed network connection Sep 23 13:37:13.008689582 (5m58.470782065s) [noise=true , address=145.79.227.12:4767 , piece_id=7JPOEJPRMBWO5MCNTISEDMBSYFACGTPJJUCBDX6EVYUKR3E7MYSA, seg_pos={0 0}, upload=85]: Long tail cancellation due to optimal threshold Sep 23 13:37:13.008703729 (5m58.470796212s) [noise=true , address=145.79.227.5:4221 , piece_id=37XI3ODHTNYYIW3TYSGEZKLL4REO4C7J7W4F3M54QPBS27HU2N7Q, seg_pos={0 0}, upload=85]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR): protocol: expected piece hash; context canceled; EOF Sep 23 13:37:13.008716322 (5m58.470808806s) [noise=true , address=145.79.227.5:4221 , piece_id=37XI3ODHTNYYIW3TYSGEZKLL4REO4C7J7W4F3M54QPBS27HU2N7Q, seg_pos={0 0}, upload=85]: Long tail cancellation due to optimal threshold Sep 23 13:37:13.008800553 (5m58.470893037s) [noise=true , address=145.79.227.14:4838 , piece_id=MG6N5Q7OOXD4OI6CV2VCSGIOOI6VB4GYFATAZ5WZWXFCZN6EFFLQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: expected piece hash; context canceled; EOF Sep 23 13:37:13.008809150 (5m58.470901633s) [noise=true , address=145.79.227.14:4838 , piece_id=MG6N5Q7OOXD4OI6CV2VCSGIOOI6VB4GYFATAZ5WZWXFCZN6EFFLQ, seg_pos={0 0}, upload=85]: Long tail cancellation due to optimal threshold Sep 23 13:37:13.008829819 (5m58.470922302s) [seg_pos={0 0}, upload=85]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 23 13:37:13.008908238 (5m58.471000721s) [upload=85]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 23 13:37:13.008914450 (5m58.471006923s) [upload=85]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 23 13:37:13.352201772 (5m58.814294255s) [upload=85]: Done uploading segments... Sep 23 13:37:13.352227461 (5m58.814319944s) [upload=85]: Upload finished. err: <nil> Duration: 7.554883439s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 6.804007153s FAIL (2025-09-23 13:48:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: context deadline exceeded ======LOGS====== Sep 23 13:37:08.880516737 (5m54.34260922s) [upload=85]: Starting upload Sep 23 13:37:08.880590076 (5m54.342682549s) [upload=85]: Done starting upload Sep 23 13:37:08.880678955 (5m54.342771438s) [upload=85]: Uploading segments... Sep 23 13:37:08.880816417 (5m54.3429089s) [upload=85]: Got next segment. Inline: false Sep 23 13:37:08.880868837 (5m54.34296132s) [upload=85]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 13:37:08.880873937 (5m54.34296642s) [upload=85]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 13:37:09.155479775 (5m54.617572258s) [seg_pos={0 0}, upload=85]: Begin upload segment... Sep 23 13:37:09.155561942 (5m54.617654425s) [seg_pos={0 0}, upload=85]: Done begin upload segment. Sep 23 13:37:09.155609312 (5m54.617701795s) [noise=true , address=145.79.227.5:4221 , piece_id=37XI3ODHTNYYIW3TYSGEZKLL4REO4C7J7W4F3M54QPBS27HU2N7Q, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.155653065 (5m54.617745548s) [noise=true , address=145.79.227.13:4801 , piece_id=KDNEMJNYSDPBUV67TZ6PFMKXECG2TLUFSFX742MK5XRK32C2NHXA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156257627 (5m54.61835011s) [noise=true , address=45.91.116.4:4938 , piece_id=YXX7C7LQDTPKFEVXO7E6HN6QNM562OWWOXPI232ZHCLSY3WE4BUA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156274629 (5m54.618367102s) [noise=true , address=145.79.227.14:4839 , piece_id=65NEICBQJXMM6N6KBD5KGI5SFTVAV7NJJKBAET6HDGKI5VD325OQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156622501 (5m54.618715004s) [noise=true , address=145.79.227.6:4259 , piece_id=LPZNOJHDSME5TCQX2HQ544ZANJCJKGWIUXFLIOMNNSZBLYPNJTVA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156777296 (5m54.618869779s) [noise=true , address=145.79.227.16:4912 , piece_id=YLC5DB6H62T67W5LPTPQIQMWGJAN5ODYJB5PXIVX6X7JNFVCTNTA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156984401 (5m54.619076884s) [noise=true , address=145.79.227.14:4838 , piece_id=MG6N5Q7OOXD4OI6CV2VCSGIOOI6VB4GYFATAZ5WZWXFCZN6EFFLQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.157122624 (5m54.619215107s) [noise=true , address=45.91.116.10:4083 , piece_id=VONNFDRLV44LE65ZK7ENEJPNRMA2IPUGGGO3QGYJZLZLWOMNLFRQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.157345388 (5m54.619437871s) [noise=true , address=145.79.227.15:4876 , piece_id=ZHNZB3WQ4QQWR2LO6AQ64ONLIZ2EKCOBYGXFNR5FBY4CZL6QLVMA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.158372935 (5m54.620465408s) [noise=true , address=45.91.116.14:4189 , piece_id=HBFO3FNIMMF5UIDBXTFQQG6WZ4SFGJI3W6CSJ5EBAGN3D7JI6MSA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.159036498 (5m54.621128982s) [noise=true , address=45.91.116.11:4117 , piece_id=QEF7PSZ4ZAV2XM546G6QVKSJVH4JGCSYRAU572ABBMRGKVP23N6A, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.159800133 (5m54.621892616s) [noise=true , address=145.79.227.12:4765 , piece_id=H4PQGKP3XUGR6ARQ3UW3ZVPNCPS5WA5NM6FYJG5TBU3FALNXQYNA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.161637361 (5m54.623729844s) [noise=true , address=145.79.227.17:4970 , piece_id=IW4WTHW5W4SFOYMCXENR3DGYCA2OQJ4OO5RSXPYX6SUIGWKA4LBA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.163221126 (5m54.62531361s) [noise=true , address=77.90.31.11:4553 , piece_id=7XXKVSGNIYW5ZN6G5AV3JWP56QHF2HI4KOXJZ2DNOSGDHVZXDQTQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.163665793 (5m54.625758277s) [noise=true , address=145.79.227.11:4731 , piece_id=XR2GN3LWECQJ4OTSM44QXNW4B6GE7P5C5EFZFKJXD2M4GCMJ54VQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.164366027 (5m54.62645851s) [noise=true , address=145.79.227.12:4767 , piece_id=7JPOEJPRMBWO5MCNTISEDMBSYFACGTPJJUCBDX6EVYUKR3E7MYSA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:10.933255114 (5m56.395347597s) [upload=85]: Next returned nil segment Sep 23 13:37:10.933279932 (5m56.395372415s) [upload=85]: Waiting for error group managing segments... Sep 23 13:37:11.355919055 (5m56.818011548s) [noise=true , address=45.91.116.10:4083 , piece_id=VONNFDRLV44LE65ZK7ENEJPNRMA2IPUGGGO3QGYJZLZLWOMNLFRQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.366419560 (5m56.828512043s) [noise=true , address=45.91.116.4:4938 , piece_id=YXX7C7LQDTPKFEVXO7E6HN6QNM562OWWOXPI232ZHCLSY3WE4BUA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.373438585 (5m56.835531068s) [noise=true , address=45.91.116.11:4117 , piece_id=QEF7PSZ4ZAV2XM546G6QVKSJVH4JGCSYRAU572ABBMRGKVP23N6A, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.388282841 (5m56.850375324s) [noise=true , address=45.91.116.14:4189 , piece_id=HBFO3FNIMMF5UIDBXTFQQG6WZ4SFGJI3W6CSJ5EBAGN3D7JI6MSA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.396367205 (5m56.858459688s) [noise=true , address=77.90.31.11:4553 , piece_id=7XXKVSGNIYW5ZN6G5AV3JWP56QHF2HI4KOXJZ2DNOSGDHVZXDQTQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.050842867 (5m57.51293534s) [noise=true , address=145.79.227.13:4801 , piece_id=KDNEMJNYSDPBUV67TZ6PFMKXECG2TLUFSFX742MK5XRK32C2NHXA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.054995253 (5m57.517087736s) [noise=true , address=145.79.227.15:4876 , piece_id=ZHNZB3WQ4QQWR2LO6AQ64ONLIZ2EKCOBYGXFNR5FBY4CZL6QLVMA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.072213210 (5m57.534305693s) [noise=true , address=145.79.227.17:4970 , piece_id=IW4WTHW5W4SFOYMCXENR3DGYCA2OQJ4OO5RSXPYX6SUIGWKA4LBA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.101168265 (5m57.563260748s) [noise=true , address=145.79.227.6:4259 , piece_id=LPZNOJHDSME5TCQX2HQ544ZANJCJKGWIUXFLIOMNNSZBLYPNJTVA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.106130595 (5m57.568223078s) [noise=true , address=145.79.227.12:4765 , piece_id=H4PQGKP3XUGR6ARQ3UW3ZVPNCPS5WA5NM6FYJG5TBU3FALNXQYNA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.118733793 (5m57.580826277s) [noise=true , address=145.79.227.14:4839 , piece_id=65NEICBQJXMM6N6KBD5KGI5SFTVAV7NJJKBAET6HDGKI5VD325OQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.508815721 (5m57.970908204s) [noise=true , address=145.79.227.11:4731 , piece_id=XR2GN3LWECQJ4OTSM44QXNW4B6GE7P5C5EFZFKJXD2M4GCMJ54VQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:13.008461678 (5m58.470554161s) [noise=true , address=145.79.227.16:4912 , piece_id=YLC5DB6H62T67W5LPTPQIQMWGJAN5ODYJB5PXIVX6X7JNFVCTNTA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:13.008503868 (5m58.470596351s) [seg_pos={0 0}, upload=85]: Segment reached optimal threshold of 13 pieces. Sep 23 13:37:13.008675254 (5m58.470767738s) [noise=true , address=145.79.227.12:4767 , piece_id=7JPOEJPRMBWO5MCNTISEDMBSYFACGTPJJUCBDX6EVYUKR3E7MYSA, seg_pos={0 0}, upload=85]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5): protocol: write tcp 188.166.225.170:41718->145.79.227.12:4767: use of closed network connection; write tcp 188.166.225.170:41718->145.79.227.12:4767: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41718->145.79.227.12:4767: use of closed network connection Sep 23 13:37:13.008689582 (5m58.470782065s) [noise=true , address=145.79.227.12:4767 , piece_id=7JPOEJPRMBWO5MCNTISEDMBSYFACGTPJJUCBDX6EVYUKR3E7MYSA, seg_pos={0 0}, upload=85]: Long tail cancellation due to optimal threshold Sep 23 13:37:13.008703729 (5m58.470796212s) [noise=true , address=145.79.227.5:4221 , piece_id=37XI3ODHTNYYIW3TYSGEZKLL4REO4C7J7W4F3M54QPBS27HU2N7Q, seg_pos={0 0}, upload=85]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR): protocol: expected piece hash; context canceled; EOF Sep 23 13:37:13.008716322 (5m58.470808806s) [noise=true , address=145.79.227.5:4221 , piece_id=37XI3ODHTNYYIW3TYSGEZKLL4REO4C7J7W4F3M54QPBS27HU2N7Q, seg_pos={0 0}, upload=85]: Long tail cancellation due to optimal threshold Sep 23 13:37:13.008800553 (5m58.470893037s) [noise=true , address=145.79.227.14:4838 , piece_id=MG6N5Q7OOXD4OI6CV2VCSGIOOI6VB4GYFATAZ5WZWXFCZN6EFFLQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: expected piece hash; context canceled; EOF Sep 23 13:37:13.008809150 (5m58.470901633s) [noise=true , address=145.79.227.14:4838 , piece_id=MG6N5Q7OOXD4OI6CV2VCSGIOOI6VB4GYFATAZ5WZWXFCZN6EFFLQ, seg_pos={0 0}, upload=85]: Long tail cancellation due to optimal threshold Sep 23 13:37:13.008829819 (5m58.470922302s) [seg_pos={0 0}, upload=85]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 23 13:37:13.008908238 (5m58.471000721s) [upload=85]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 23 13:37:13.008914450 (5m58.471006923s) [upload=85]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 23 13:37:13.352201772 (5m58.814294255s) [upload=85]: Done uploading segments... Sep 23 13:37:13.352227461 (5m58.814319944s) [upload=85]: Upload finished. err: <nil> Duration: 6.804047931s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 6.022031199s FAIL (2025-09-23 13:47:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: context deadline exceeded ======LOGS====== Sep 23 13:37:08.880516737 (5m54.34260922s) [upload=85]: Starting upload Sep 23 13:37:08.880590076 (5m54.342682549s) [upload=85]: Done starting upload Sep 23 13:37:08.880678955 (5m54.342771438s) [upload=85]: Uploading segments... Sep 23 13:37:08.880816417 (5m54.3429089s) [upload=85]: Got next segment. Inline: false Sep 23 13:37:08.880868837 (5m54.34296132s) [upload=85]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 13:37:08.880873937 (5m54.34296642s) [upload=85]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 13:37:09.155479775 (5m54.617572258s) [seg_pos={0 0}, upload=85]: Begin upload segment... Sep 23 13:37:09.155561942 (5m54.617654425s) [seg_pos={0 0}, upload=85]: Done begin upload segment. Sep 23 13:37:09.155609312 (5m54.617701795s) [noise=true , address=145.79.227.5:4221 , piece_id=37XI3ODHTNYYIW3TYSGEZKLL4REO4C7J7W4F3M54QPBS27HU2N7Q, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.155653065 (5m54.617745548s) [noise=true , address=145.79.227.13:4801 , piece_id=KDNEMJNYSDPBUV67TZ6PFMKXECG2TLUFSFX742MK5XRK32C2NHXA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156257627 (5m54.61835011s) [noise=true , address=45.91.116.4:4938 , piece_id=YXX7C7LQDTPKFEVXO7E6HN6QNM562OWWOXPI232ZHCLSY3WE4BUA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156274629 (5m54.618367102s) [noise=true , address=145.79.227.14:4839 , piece_id=65NEICBQJXMM6N6KBD5KGI5SFTVAV7NJJKBAET6HDGKI5VD325OQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156622501 (5m54.618715004s) [noise=true , address=145.79.227.6:4259 , piece_id=LPZNOJHDSME5TCQX2HQ544ZANJCJKGWIUXFLIOMNNSZBLYPNJTVA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156777296 (5m54.618869779s) [noise=true , address=145.79.227.16:4912 , piece_id=YLC5DB6H62T67W5LPTPQIQMWGJAN5ODYJB5PXIVX6X7JNFVCTNTA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156984401 (5m54.619076884s) [noise=true , address=145.79.227.14:4838 , piece_id=MG6N5Q7OOXD4OI6CV2VCSGIOOI6VB4GYFATAZ5WZWXFCZN6EFFLQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.157122624 (5m54.619215107s) [noise=true , address=45.91.116.10:4083 , piece_id=VONNFDRLV44LE65ZK7ENEJPNRMA2IPUGGGO3QGYJZLZLWOMNLFRQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.157345388 (5m54.619437871s) [noise=true , address=145.79.227.15:4876 , piece_id=ZHNZB3WQ4QQWR2LO6AQ64ONLIZ2EKCOBYGXFNR5FBY4CZL6QLVMA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.158372935 (5m54.620465408s) [noise=true , address=45.91.116.14:4189 , piece_id=HBFO3FNIMMF5UIDBXTFQQG6WZ4SFGJI3W6CSJ5EBAGN3D7JI6MSA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.159036498 (5m54.621128982s) [noise=true , address=45.91.116.11:4117 , piece_id=QEF7PSZ4ZAV2XM546G6QVKSJVH4JGCSYRAU572ABBMRGKVP23N6A, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.159800133 (5m54.621892616s) [noise=true , address=145.79.227.12:4765 , piece_id=H4PQGKP3XUGR6ARQ3UW3ZVPNCPS5WA5NM6FYJG5TBU3FALNXQYNA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.161637361 (5m54.623729844s) [noise=true , address=145.79.227.17:4970 , piece_id=IW4WTHW5W4SFOYMCXENR3DGYCA2OQJ4OO5RSXPYX6SUIGWKA4LBA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.163221126 (5m54.62531361s) [noise=true , address=77.90.31.11:4553 , piece_id=7XXKVSGNIYW5ZN6G5AV3JWP56QHF2HI4KOXJZ2DNOSGDHVZXDQTQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.163665793 (5m54.625758277s) [noise=true , address=145.79.227.11:4731 , piece_id=XR2GN3LWECQJ4OTSM44QXNW4B6GE7P5C5EFZFKJXD2M4GCMJ54VQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.164366027 (5m54.62645851s) [noise=true , address=145.79.227.12:4767 , piece_id=7JPOEJPRMBWO5MCNTISEDMBSYFACGTPJJUCBDX6EVYUKR3E7MYSA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:10.933255114 (5m56.395347597s) [upload=85]: Next returned nil segment Sep 23 13:37:10.933279932 (5m56.395372415s) [upload=85]: Waiting for error group managing segments... Sep 23 13:37:11.355919055 (5m56.818011548s) [noise=true , address=45.91.116.10:4083 , piece_id=VONNFDRLV44LE65ZK7ENEJPNRMA2IPUGGGO3QGYJZLZLWOMNLFRQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.366419560 (5m56.828512043s) [noise=true , address=45.91.116.4:4938 , piece_id=YXX7C7LQDTPKFEVXO7E6HN6QNM562OWWOXPI232ZHCLSY3WE4BUA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.373438585 (5m56.835531068s) [noise=true , address=45.91.116.11:4117 , piece_id=QEF7PSZ4ZAV2XM546G6QVKSJVH4JGCSYRAU572ABBMRGKVP23N6A, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.388282841 (5m56.850375324s) [noise=true , address=45.91.116.14:4189 , piece_id=HBFO3FNIMMF5UIDBXTFQQG6WZ4SFGJI3W6CSJ5EBAGN3D7JI6MSA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.396367205 (5m56.858459688s) [noise=true , address=77.90.31.11:4553 , piece_id=7XXKVSGNIYW5ZN6G5AV3JWP56QHF2HI4KOXJZ2DNOSGDHVZXDQTQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.050842867 (5m57.51293534s) [noise=true , address=145.79.227.13:4801 , piece_id=KDNEMJNYSDPBUV67TZ6PFMKXECG2TLUFSFX742MK5XRK32C2NHXA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.054995253 (5m57.517087736s) [noise=true , address=145.79.227.15:4876 , piece_id=ZHNZB3WQ4QQWR2LO6AQ64ONLIZ2EKCOBYGXFNR5FBY4CZL6QLVMA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.072213210 (5m57.534305693s) [noise=true , address=145.79.227.17:4970 , piece_id=IW4WTHW5W4SFOYMCXENR3DGYCA2OQJ4OO5RSXPYX6SUIGWKA4LBA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.101168265 (5m57.563260748s) [noise=true , address=145.79.227.6:4259 , piece_id=LPZNOJHDSME5TCQX2HQ544ZANJCJKGWIUXFLIOMNNSZBLYPNJTVA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.106130595 (5m57.568223078s) [noise=true , address=145.79.227.12:4765 , piece_id=H4PQGKP3XUGR6ARQ3UW3ZVPNCPS5WA5NM6FYJG5TBU3FALNXQYNA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.118733793 (5m57.580826277s) [noise=true , address=145.79.227.14:4839 , piece_id=65NEICBQJXMM6N6KBD5KGI5SFTVAV7NJJKBAET6HDGKI5VD325OQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.508815721 (5m57.970908204s) [noise=true , address=145.79.227.11:4731 , piece_id=XR2GN3LWECQJ4OTSM44QXNW4B6GE7P5C5EFZFKJXD2M4GCMJ54VQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:13.008461678 (5m58.470554161s) [noise=true , address=145.79.227.16:4912 , piece_id=YLC5DB6H62T67W5LPTPQIQMWGJAN5ODYJB5PXIVX6X7JNFVCTNTA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:13.008503868 (5m58.470596351s) [seg_pos={0 0}, upload=85]: Segment reached optimal threshold of 13 pieces. Sep 23 13:37:13.008675254 (5m58.470767738s) [noise=true , address=145.79.227.12:4767 , piece_id=7JPOEJPRMBWO5MCNTISEDMBSYFACGTPJJUCBDX6EVYUKR3E7MYSA, seg_pos={0 0}, upload=85]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5): protocol: write tcp 188.166.225.170:41718->145.79.227.12:4767: use of closed network connection; write tcp 188.166.225.170:41718->145.79.227.12:4767: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41718->145.79.227.12:4767: use of closed network connection Sep 23 13:37:13.008689582 (5m58.470782065s) [noise=true , address=145.79.227.12:4767 , piece_id=7JPOEJPRMBWO5MCNTISEDMBSYFACGTPJJUCBDX6EVYUKR3E7MYSA, seg_pos={0 0}, upload=85]: Long tail cancellation due to optimal threshold Sep 23 13:37:13.008703729 (5m58.470796212s) [noise=true , address=145.79.227.5:4221 , piece_id=37XI3ODHTNYYIW3TYSGEZKLL4REO4C7J7W4F3M54QPBS27HU2N7Q, seg_pos={0 0}, upload=85]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR): protocol: expected piece hash; context canceled; EOF Sep 23 13:37:13.008716322 (5m58.470808806s) [noise=true , address=145.79.227.5:4221 , piece_id=37XI3ODHTNYYIW3TYSGEZKLL4REO4C7J7W4F3M54QPBS27HU2N7Q, seg_pos={0 0}, upload=85]: Long tail cancellation due to optimal threshold Sep 23 13:37:13.008800553 (5m58.470893037s) [noise=true , address=145.79.227.14:4838 , piece_id=MG6N5Q7OOXD4OI6CV2VCSGIOOI6VB4GYFATAZ5WZWXFCZN6EFFLQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: expected piece hash; context canceled; EOF Sep 23 13:37:13.008809150 (5m58.470901633s) [noise=true , address=145.79.227.14:4838 , piece_id=MG6N5Q7OOXD4OI6CV2VCSGIOOI6VB4GYFATAZ5WZWXFCZN6EFFLQ, seg_pos={0 0}, upload=85]: Long tail cancellation due to optimal threshold Sep 23 13:37:13.008829819 (5m58.470922302s) [seg_pos={0 0}, upload=85]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 23 13:37:13.008908238 (5m58.471000721s) [upload=85]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 23 13:37:13.008914450 (5m58.471006923s) [upload=85]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 23 13:37:13.352201772 (5m58.814294255s) [upload=85]: Done uploading segments... Sep 23 13:37:13.352227461 (5m58.814319944s) [upload=85]: Upload finished. err: <nil> Duration: 6.02205876s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 8.339552393s FAIL (2025-09-23 13:47:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: context deadline exceeded ======LOGS====== Sep 23 13:37:08.880516737 (5m54.34260922s) [upload=85]: Starting upload Sep 23 13:37:08.880590076 (5m54.342682549s) [upload=85]: Done starting upload Sep 23 13:37:08.880678955 (5m54.342771438s) [upload=85]: Uploading segments... Sep 23 13:37:08.880816417 (5m54.3429089s) [upload=85]: Got next segment. Inline: false Sep 23 13:37:08.880868837 (5m54.34296132s) [upload=85]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 13:37:08.880873937 (5m54.34296642s) [upload=85]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 13:37:09.155479775 (5m54.617572258s) [seg_pos={0 0}, upload=85]: Begin upload segment... Sep 23 13:37:09.155561942 (5m54.617654425s) [seg_pos={0 0}, upload=85]: Done begin upload segment. Sep 23 13:37:09.155609312 (5m54.617701795s) [noise=true , address=145.79.227.5:4221 , piece_id=37XI3ODHTNYYIW3TYSGEZKLL4REO4C7J7W4F3M54QPBS27HU2N7Q, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.155653065 (5m54.617745548s) [noise=true , address=145.79.227.13:4801 , piece_id=KDNEMJNYSDPBUV67TZ6PFMKXECG2TLUFSFX742MK5XRK32C2NHXA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156257627 (5m54.61835011s) [noise=true , address=45.91.116.4:4938 , piece_id=YXX7C7LQDTPKFEVXO7E6HN6QNM562OWWOXPI232ZHCLSY3WE4BUA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156274629 (5m54.618367102s) [noise=true , address=145.79.227.14:4839 , piece_id=65NEICBQJXMM6N6KBD5KGI5SFTVAV7NJJKBAET6HDGKI5VD325OQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156622501 (5m54.618715004s) [noise=true , address=145.79.227.6:4259 , piece_id=LPZNOJHDSME5TCQX2HQ544ZANJCJKGWIUXFLIOMNNSZBLYPNJTVA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156777296 (5m54.618869779s) [noise=true , address=145.79.227.16:4912 , piece_id=YLC5DB6H62T67W5LPTPQIQMWGJAN5ODYJB5PXIVX6X7JNFVCTNTA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156984401 (5m54.619076884s) [noise=true , address=145.79.227.14:4838 , piece_id=MG6N5Q7OOXD4OI6CV2VCSGIOOI6VB4GYFATAZ5WZWXFCZN6EFFLQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.157122624 (5m54.619215107s) [noise=true , address=45.91.116.10:4083 , piece_id=VONNFDRLV44LE65ZK7ENEJPNRMA2IPUGGGO3QGYJZLZLWOMNLFRQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.157345388 (5m54.619437871s) [noise=true , address=145.79.227.15:4876 , piece_id=ZHNZB3WQ4QQWR2LO6AQ64ONLIZ2EKCOBYGXFNR5FBY4CZL6QLVMA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.158372935 (5m54.620465408s) [noise=true , address=45.91.116.14:4189 , piece_id=HBFO3FNIMMF5UIDBXTFQQG6WZ4SFGJI3W6CSJ5EBAGN3D7JI6MSA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.159036498 (5m54.621128982s) [noise=true , address=45.91.116.11:4117 , piece_id=QEF7PSZ4ZAV2XM546G6QVKSJVH4JGCSYRAU572ABBMRGKVP23N6A, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.159800133 (5m54.621892616s) [noise=true , address=145.79.227.12:4765 , piece_id=H4PQGKP3XUGR6ARQ3UW3ZVPNCPS5WA5NM6FYJG5TBU3FALNXQYNA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.161637361 (5m54.623729844s) [noise=true , address=145.79.227.17:4970 , piece_id=IW4WTHW5W4SFOYMCXENR3DGYCA2OQJ4OO5RSXPYX6SUIGWKA4LBA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.163221126 (5m54.62531361s) [noise=true , address=77.90.31.11:4553 , piece_id=7XXKVSGNIYW5ZN6G5AV3JWP56QHF2HI4KOXJZ2DNOSGDHVZXDQTQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.163665793 (5m54.625758277s) [noise=true , address=145.79.227.11:4731 , piece_id=XR2GN3LWECQJ4OTSM44QXNW4B6GE7P5C5EFZFKJXD2M4GCMJ54VQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.164366027 (5m54.62645851s) [noise=true , address=145.79.227.12:4767 , piece_id=7JPOEJPRMBWO5MCNTISEDMBSYFACGTPJJUCBDX6EVYUKR3E7MYSA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:10.933255114 (5m56.395347597s) [upload=85]: Next returned nil segment Sep 23 13:37:10.933279932 (5m56.395372415s) [upload=85]: Waiting for error group managing segments... Sep 23 13:37:11.355919055 (5m56.818011548s) [noise=true , address=45.91.116.10:4083 , piece_id=VONNFDRLV44LE65ZK7ENEJPNRMA2IPUGGGO3QGYJZLZLWOMNLFRQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.366419560 (5m56.828512043s) [noise=true , address=45.91.116.4:4938 , piece_id=YXX7C7LQDTPKFEVXO7E6HN6QNM562OWWOXPI232ZHCLSY3WE4BUA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.373438585 (5m56.835531068s) [noise=true , address=45.91.116.11:4117 , piece_id=QEF7PSZ4ZAV2XM546G6QVKSJVH4JGCSYRAU572ABBMRGKVP23N6A, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.388282841 (5m56.850375324s) [noise=true , address=45.91.116.14:4189 , piece_id=HBFO3FNIMMF5UIDBXTFQQG6WZ4SFGJI3W6CSJ5EBAGN3D7JI6MSA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.396367205 (5m56.858459688s) [noise=true , address=77.90.31.11:4553 , piece_id=7XXKVSGNIYW5ZN6G5AV3JWP56QHF2HI4KOXJZ2DNOSGDHVZXDQTQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.050842867 (5m57.51293534s) [noise=true , address=145.79.227.13:4801 , piece_id=KDNEMJNYSDPBUV67TZ6PFMKXECG2TLUFSFX742MK5XRK32C2NHXA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.054995253 (5m57.517087736s) [noise=true , address=145.79.227.15:4876 , piece_id=ZHNZB3WQ4QQWR2LO6AQ64ONLIZ2EKCOBYGXFNR5FBY4CZL6QLVMA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.072213210 (5m57.534305693s) [noise=true , address=145.79.227.17:4970 , piece_id=IW4WTHW5W4SFOYMCXENR3DGYCA2OQJ4OO5RSXPYX6SUIGWKA4LBA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.101168265 (5m57.563260748s) [noise=true , address=145.79.227.6:4259 , piece_id=LPZNOJHDSME5TCQX2HQ544ZANJCJKGWIUXFLIOMNNSZBLYPNJTVA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.106130595 (5m57.568223078s) [noise=true , address=145.79.227.12:4765 , piece_id=H4PQGKP3XUGR6ARQ3UW3ZVPNCPS5WA5NM6FYJG5TBU3FALNXQYNA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.118733793 (5m57.580826277s) [noise=true , address=145.79.227.14:4839 , piece_id=65NEICBQJXMM6N6KBD5KGI5SFTVAV7NJJKBAET6HDGKI5VD325OQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.508815721 (5m57.970908204s) [noise=true , address=145.79.227.11:4731 , piece_id=XR2GN3LWECQJ4OTSM44QXNW4B6GE7P5C5EFZFKJXD2M4GCMJ54VQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:13.008461678 (5m58.470554161s) [noise=true , address=145.79.227.16:4912 , piece_id=YLC5DB6H62T67W5LPTPQIQMWGJAN5ODYJB5PXIVX6X7JNFVCTNTA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:13.008503868 (5m58.470596351s) [seg_pos={0 0}, upload=85]: Segment reached optimal threshold of 13 pieces. Sep 23 13:37:13.008675254 (5m58.470767738s) [noise=true , address=145.79.227.12:4767 , piece_id=7JPOEJPRMBWO5MCNTISEDMBSYFACGTPJJUCBDX6EVYUKR3E7MYSA, seg_pos={0 0}, upload=85]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5): protocol: write tcp 188.166.225.170:41718->145.79.227.12:4767: use of closed network connection; write tcp 188.166.225.170:41718->145.79.227.12:4767: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41718->145.79.227.12:4767: use of closed network connection Sep 23 13:37:13.008689582 (5m58.470782065s) [noise=true , address=145.79.227.12:4767 , piece_id=7JPOEJPRMBWO5MCNTISEDMBSYFACGTPJJUCBDX6EVYUKR3E7MYSA, seg_pos={0 0}, upload=85]: Long tail cancellation due to optimal threshold Sep 23 13:37:13.008703729 (5m58.470796212s) [noise=true , address=145.79.227.5:4221 , piece_id=37XI3ODHTNYYIW3TYSGEZKLL4REO4C7J7W4F3M54QPBS27HU2N7Q, seg_pos={0 0}, upload=85]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR): protocol: expected piece hash; context canceled; EOF Sep 23 13:37:13.008716322 (5m58.470808806s) [noise=true , address=145.79.227.5:4221 , piece_id=37XI3ODHTNYYIW3TYSGEZKLL4REO4C7J7W4F3M54QPBS27HU2N7Q, seg_pos={0 0}, upload=85]: Long tail cancellation due to optimal threshold Sep 23 13:37:13.008800553 (5m58.470893037s) [noise=true , address=145.79.227.14:4838 , piece_id=MG6N5Q7OOXD4OI6CV2VCSGIOOI6VB4GYFATAZ5WZWXFCZN6EFFLQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: expected piece hash; context canceled; EOF Sep 23 13:37:13.008809150 (5m58.470901633s) [noise=true , address=145.79.227.14:4838 , piece_id=MG6N5Q7OOXD4OI6CV2VCSGIOOI6VB4GYFATAZ5WZWXFCZN6EFFLQ, seg_pos={0 0}, upload=85]: Long tail cancellation due to optimal threshold Sep 23 13:37:13.008829819 (5m58.470922302s) [seg_pos={0 0}, upload=85]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 23 13:37:13.008908238 (5m58.471000721s) [upload=85]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 23 13:37:13.008914450 (5m58.471006923s) [upload=85]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 23 13:37:13.352201772 (5m58.814294255s) [upload=85]: Done uploading segments... Sep 23 13:37:13.352227461 (5m58.814319944s) [upload=85]: Upload finished. err: <nil> Duration: 8.33958258s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 6.546190371s FAIL (2025-09-23 13:47:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: context deadline exceeded ======LOGS====== Sep 23 13:37:08.880516737 (5m54.34260922s) [upload=85]: Starting upload Sep 23 13:37:08.880590076 (5m54.342682549s) [upload=85]: Done starting upload Sep 23 13:37:08.880678955 (5m54.342771438s) [upload=85]: Uploading segments... Sep 23 13:37:08.880816417 (5m54.3429089s) [upload=85]: Got next segment. Inline: false Sep 23 13:37:08.880868837 (5m54.34296132s) [upload=85]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 13:37:08.880873937 (5m54.34296642s) [upload=85]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 13:37:09.155479775 (5m54.617572258s) [seg_pos={0 0}, upload=85]: Begin upload segment... Sep 23 13:37:09.155561942 (5m54.617654425s) [seg_pos={0 0}, upload=85]: Done begin upload segment. Sep 23 13:37:09.155609312 (5m54.617701795s) [noise=true , address=145.79.227.5:4221 , piece_id=37XI3ODHTNYYIW3TYSGEZKLL4REO4C7J7W4F3M54QPBS27HU2N7Q, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.155653065 (5m54.617745548s) [noise=true , address=145.79.227.13:4801 , piece_id=KDNEMJNYSDPBUV67TZ6PFMKXECG2TLUFSFX742MK5XRK32C2NHXA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156257627 (5m54.61835011s) [noise=true , address=45.91.116.4:4938 , piece_id=YXX7C7LQDTPKFEVXO7E6HN6QNM562OWWOXPI232ZHCLSY3WE4BUA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156274629 (5m54.618367102s) [noise=true , address=145.79.227.14:4839 , piece_id=65NEICBQJXMM6N6KBD5KGI5SFTVAV7NJJKBAET6HDGKI5VD325OQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156622501 (5m54.618715004s) [noise=true , address=145.79.227.6:4259 , piece_id=LPZNOJHDSME5TCQX2HQ544ZANJCJKGWIUXFLIOMNNSZBLYPNJTVA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156777296 (5m54.618869779s) [noise=true , address=145.79.227.16:4912 , piece_id=YLC5DB6H62T67W5LPTPQIQMWGJAN5ODYJB5PXIVX6X7JNFVCTNTA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156984401 (5m54.619076884s) [noise=true , address=145.79.227.14:4838 , piece_id=MG6N5Q7OOXD4OI6CV2VCSGIOOI6VB4GYFATAZ5WZWXFCZN6EFFLQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.157122624 (5m54.619215107s) [noise=true , address=45.91.116.10:4083 , piece_id=VONNFDRLV44LE65ZK7ENEJPNRMA2IPUGGGO3QGYJZLZLWOMNLFRQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.157345388 (5m54.619437871s) [noise=true , address=145.79.227.15:4876 , piece_id=ZHNZB3WQ4QQWR2LO6AQ64ONLIZ2EKCOBYGXFNR5FBY4CZL6QLVMA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.158372935 (5m54.620465408s) [noise=true , address=45.91.116.14:4189 , piece_id=HBFO3FNIMMF5UIDBXTFQQG6WZ4SFGJI3W6CSJ5EBAGN3D7JI6MSA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.159036498 (5m54.621128982s) [noise=true , address=45.91.116.11:4117 , piece_id=QEF7PSZ4ZAV2XM546G6QVKSJVH4JGCSYRAU572ABBMRGKVP23N6A, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.159800133 (5m54.621892616s) [noise=true , address=145.79.227.12:4765 , piece_id=H4PQGKP3XUGR6ARQ3UW3ZVPNCPS5WA5NM6FYJG5TBU3FALNXQYNA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.161637361 (5m54.623729844s) [noise=true , address=145.79.227.17:4970 , piece_id=IW4WTHW5W4SFOYMCXENR3DGYCA2OQJ4OO5RSXPYX6SUIGWKA4LBA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.163221126 (5m54.62531361s) [noise=true , address=77.90.31.11:4553 , piece_id=7XXKVSGNIYW5ZN6G5AV3JWP56QHF2HI4KOXJZ2DNOSGDHVZXDQTQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.163665793 (5m54.625758277s) [noise=true , address=145.79.227.11:4731 , piece_id=XR2GN3LWECQJ4OTSM44QXNW4B6GE7P5C5EFZFKJXD2M4GCMJ54VQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.164366027 (5m54.62645851s) [noise=true , address=145.79.227.12:4767 , piece_id=7JPOEJPRMBWO5MCNTISEDMBSYFACGTPJJUCBDX6EVYUKR3E7MYSA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:10.933255114 (5m56.395347597s) [upload=85]: Next returned nil segment Sep 23 13:37:10.933279932 (5m56.395372415s) [upload=85]: Waiting for error group managing segments... Sep 23 13:37:11.355919055 (5m56.818011548s) [noise=true , address=45.91.116.10:4083 , piece_id=VONNFDRLV44LE65ZK7ENEJPNRMA2IPUGGGO3QGYJZLZLWOMNLFRQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.366419560 (5m56.828512043s) [noise=true , address=45.91.116.4:4938 , piece_id=YXX7C7LQDTPKFEVXO7E6HN6QNM562OWWOXPI232ZHCLSY3WE4BUA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.373438585 (5m56.835531068s) [noise=true , address=45.91.116.11:4117 , piece_id=QEF7PSZ4ZAV2XM546G6QVKSJVH4JGCSYRAU572ABBMRGKVP23N6A, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.388282841 (5m56.850375324s) [noise=true , address=45.91.116.14:4189 , piece_id=HBFO3FNIMMF5UIDBXTFQQG6WZ4SFGJI3W6CSJ5EBAGN3D7JI6MSA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.396367205 (5m56.858459688s) [noise=true , address=77.90.31.11:4553 , piece_id=7XXKVSGNIYW5ZN6G5AV3JWP56QHF2HI4KOXJZ2DNOSGDHVZXDQTQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.050842867 (5m57.51293534s) [noise=true , address=145.79.227.13:4801 , piece_id=KDNEMJNYSDPBUV67TZ6PFMKXECG2TLUFSFX742MK5XRK32C2NHXA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.054995253 (5m57.517087736s) [noise=true , address=145.79.227.15:4876 , piece_id=ZHNZB3WQ4QQWR2LO6AQ64ONLIZ2EKCOBYGXFNR5FBY4CZL6QLVMA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.072213210 (5m57.534305693s) [noise=true , address=145.79.227.17:4970 , piece_id=IW4WTHW5W4SFOYMCXENR3DGYCA2OQJ4OO5RSXPYX6SUIGWKA4LBA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.101168265 (5m57.563260748s) [noise=true , address=145.79.227.6:4259 , piece_id=LPZNOJHDSME5TCQX2HQ544ZANJCJKGWIUXFLIOMNNSZBLYPNJTVA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.106130595 (5m57.568223078s) [noise=true , address=145.79.227.12:4765 , piece_id=H4PQGKP3XUGR6ARQ3UW3ZVPNCPS5WA5NM6FYJG5TBU3FALNXQYNA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.118733793 (5m57.580826277s) [noise=true , address=145.79.227.14:4839 , piece_id=65NEICBQJXMM6N6KBD5KGI5SFTVAV7NJJKBAET6HDGKI5VD325OQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.508815721 (5m57.970908204s) [noise=true , address=145.79.227.11:4731 , piece_id=XR2GN3LWECQJ4OTSM44QXNW4B6GE7P5C5EFZFKJXD2M4GCMJ54VQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:13.008461678 (5m58.470554161s) [noise=true , address=145.79.227.16:4912 , piece_id=YLC5DB6H62T67W5LPTPQIQMWGJAN5ODYJB5PXIVX6X7JNFVCTNTA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:13.008503868 (5m58.470596351s) [seg_pos={0 0}, upload=85]: Segment reached optimal threshold of 13 pieces. Sep 23 13:37:13.008675254 (5m58.470767738s) [noise=true , address=145.79.227.12:4767 , piece_id=7JPOEJPRMBWO5MCNTISEDMBSYFACGTPJJUCBDX6EVYUKR3E7MYSA, seg_pos={0 0}, upload=85]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5): protocol: write tcp 188.166.225.170:41718->145.79.227.12:4767: use of closed network connection; write tcp 188.166.225.170:41718->145.79.227.12:4767: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41718->145.79.227.12:4767: use of closed network connection Sep 23 13:37:13.008689582 (5m58.470782065s) [noise=true , address=145.79.227.12:4767 , piece_id=7JPOEJPRMBWO5MCNTISEDMBSYFACGTPJJUCBDX6EVYUKR3E7MYSA, seg_pos={0 0}, upload=85]: Long tail cancellation due to optimal threshold Sep 23 13:37:13.008703729 (5m58.470796212s) [noise=true , address=145.79.227.5:4221 , piece_id=37XI3ODHTNYYIW3TYSGEZKLL4REO4C7J7W4F3M54QPBS27HU2N7Q, seg_pos={0 0}, upload=85]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR): protocol: expected piece hash; context canceled; EOF Sep 23 13:37:13.008716322 (5m58.470808806s) [noise=true , address=145.79.227.5:4221 , piece_id=37XI3ODHTNYYIW3TYSGEZKLL4REO4C7J7W4F3M54QPBS27HU2N7Q, seg_pos={0 0}, upload=85]: Long tail cancellation due to optimal threshold Sep 23 13:37:13.008800553 (5m58.470893037s) [noise=true , address=145.79.227.14:4838 , piece_id=MG6N5Q7OOXD4OI6CV2VCSGIOOI6VB4GYFATAZ5WZWXFCZN6EFFLQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: expected piece hash; context canceled; EOF Sep 23 13:37:13.008809150 (5m58.470901633s) [noise=true , address=145.79.227.14:4838 , piece_id=MG6N5Q7OOXD4OI6CV2VCSGIOOI6VB4GYFATAZ5WZWXFCZN6EFFLQ, seg_pos={0 0}, upload=85]: Long tail cancellation due to optimal threshold Sep 23 13:37:13.008829819 (5m58.470922302s) [seg_pos={0 0}, upload=85]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 23 13:37:13.008908238 (5m58.471000721s) [upload=85]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 23 13:37:13.008914450 (5m58.471006923s) [upload=85]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 23 13:37:13.352201772 (5m58.814294255s) [upload=85]: Done uploading segments... Sep 23 13:37:13.352227461 (5m58.814319944s) [upload=85]: Upload finished. err: <nil> Duration: 6.546224006s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 6.528056121s FAIL (2025-09-23 13:45:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: context deadline exceeded ======LOGS====== Sep 23 13:37:08.880516737 (5m54.34260922s) [upload=85]: Starting upload Sep 23 13:37:08.880590076 (5m54.342682549s) [upload=85]: Done starting upload Sep 23 13:37:08.880678955 (5m54.342771438s) [upload=85]: Uploading segments... Sep 23 13:37:08.880816417 (5m54.3429089s) [upload=85]: Got next segment. Inline: false Sep 23 13:37:08.880868837 (5m54.34296132s) [upload=85]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 13:37:08.880873937 (5m54.34296642s) [upload=85]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 13:37:09.155479775 (5m54.617572258s) [seg_pos={0 0}, upload=85]: Begin upload segment... Sep 23 13:37:09.155561942 (5m54.617654425s) [seg_pos={0 0}, upload=85]: Done begin upload segment. Sep 23 13:37:09.155609312 (5m54.617701795s) [noise=true , address=145.79.227.5:4221 , piece_id=37XI3ODHTNYYIW3TYSGEZKLL4REO4C7J7W4F3M54QPBS27HU2N7Q, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.155653065 (5m54.617745548s) [noise=true , address=145.79.227.13:4801 , piece_id=KDNEMJNYSDPBUV67TZ6PFMKXECG2TLUFSFX742MK5XRK32C2NHXA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156257627 (5m54.61835011s) [noise=true , address=45.91.116.4:4938 , piece_id=YXX7C7LQDTPKFEVXO7E6HN6QNM562OWWOXPI232ZHCLSY3WE4BUA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156274629 (5m54.618367102s) [noise=true , address=145.79.227.14:4839 , piece_id=65NEICBQJXMM6N6KBD5KGI5SFTVAV7NJJKBAET6HDGKI5VD325OQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156622501 (5m54.618715004s) [noise=true , address=145.79.227.6:4259 , piece_id=LPZNOJHDSME5TCQX2HQ544ZANJCJKGWIUXFLIOMNNSZBLYPNJTVA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156777296 (5m54.618869779s) [noise=true , address=145.79.227.16:4912 , piece_id=YLC5DB6H62T67W5LPTPQIQMWGJAN5ODYJB5PXIVX6X7JNFVCTNTA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156984401 (5m54.619076884s) [noise=true , address=145.79.227.14:4838 , piece_id=MG6N5Q7OOXD4OI6CV2VCSGIOOI6VB4GYFATAZ5WZWXFCZN6EFFLQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.157122624 (5m54.619215107s) [noise=true , address=45.91.116.10:4083 , piece_id=VONNFDRLV44LE65ZK7ENEJPNRMA2IPUGGGO3QGYJZLZLWOMNLFRQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.157345388 (5m54.619437871s) [noise=true , address=145.79.227.15:4876 , piece_id=ZHNZB3WQ4QQWR2LO6AQ64ONLIZ2EKCOBYGXFNR5FBY4CZL6QLVMA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.158372935 (5m54.620465408s) [noise=true , address=45.91.116.14:4189 , piece_id=HBFO3FNIMMF5UIDBXTFQQG6WZ4SFGJI3W6CSJ5EBAGN3D7JI6MSA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.159036498 (5m54.621128982s) [noise=true , address=45.91.116.11:4117 , piece_id=QEF7PSZ4ZAV2XM546G6QVKSJVH4JGCSYRAU572ABBMRGKVP23N6A, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.159800133 (5m54.621892616s) [noise=true , address=145.79.227.12:4765 , piece_id=H4PQGKP3XUGR6ARQ3UW3ZVPNCPS5WA5NM6FYJG5TBU3FALNXQYNA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.161637361 (5m54.623729844s) [noise=true , address=145.79.227.17:4970 , piece_id=IW4WTHW5W4SFOYMCXENR3DGYCA2OQJ4OO5RSXPYX6SUIGWKA4LBA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.163221126 (5m54.62531361s) [noise=true , address=77.90.31.11:4553 , piece_id=7XXKVSGNIYW5ZN6G5AV3JWP56QHF2HI4KOXJZ2DNOSGDHVZXDQTQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.163665793 (5m54.625758277s) [noise=true , address=145.79.227.11:4731 , piece_id=XR2GN3LWECQJ4OTSM44QXNW4B6GE7P5C5EFZFKJXD2M4GCMJ54VQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.164366027 (5m54.62645851s) [noise=true , address=145.79.227.12:4767 , piece_id=7JPOEJPRMBWO5MCNTISEDMBSYFACGTPJJUCBDX6EVYUKR3E7MYSA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:10.933255114 (5m56.395347597s) [upload=85]: Next returned nil segment Sep 23 13:37:10.933279932 (5m56.395372415s) [upload=85]: Waiting for error group managing segments... Sep 23 13:37:11.355919055 (5m56.818011548s) [noise=true , address=45.91.116.10:4083 , piece_id=VONNFDRLV44LE65ZK7ENEJPNRMA2IPUGGGO3QGYJZLZLWOMNLFRQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.366419560 (5m56.828512043s) [noise=true , address=45.91.116.4:4938 , piece_id=YXX7C7LQDTPKFEVXO7E6HN6QNM562OWWOXPI232ZHCLSY3WE4BUA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.373438585 (5m56.835531068s) [noise=true , address=45.91.116.11:4117 , piece_id=QEF7PSZ4ZAV2XM546G6QVKSJVH4JGCSYRAU572ABBMRGKVP23N6A, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.388282841 (5m56.850375324s) [noise=true , address=45.91.116.14:4189 , piece_id=HBFO3FNIMMF5UIDBXTFQQG6WZ4SFGJI3W6CSJ5EBAGN3D7JI6MSA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.396367205 (5m56.858459688s) [noise=true , address=77.90.31.11:4553 , piece_id=7XXKVSGNIYW5ZN6G5AV3JWP56QHF2HI4KOXJZ2DNOSGDHVZXDQTQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.050842867 (5m57.51293534s) [noise=true , address=145.79.227.13:4801 , piece_id=KDNEMJNYSDPBUV67TZ6PFMKXECG2TLUFSFX742MK5XRK32C2NHXA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.054995253 (5m57.517087736s) [noise=true , address=145.79.227.15:4876 , piece_id=ZHNZB3WQ4QQWR2LO6AQ64ONLIZ2EKCOBYGXFNR5FBY4CZL6QLVMA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.072213210 (5m57.534305693s) [noise=true , address=145.79.227.17:4970 , piece_id=IW4WTHW5W4SFOYMCXENR3DGYCA2OQJ4OO5RSXPYX6SUIGWKA4LBA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.101168265 (5m57.563260748s) [noise=true , address=145.79.227.6:4259 , piece_id=LPZNOJHDSME5TCQX2HQ544ZANJCJKGWIUXFLIOMNNSZBLYPNJTVA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.106130595 (5m57.568223078s) [noise=true , address=145.79.227.12:4765 , piece_id=H4PQGKP3XUGR6ARQ3UW3ZVPNCPS5WA5NM6FYJG5TBU3FALNXQYNA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.118733793 (5m57.580826277s) [noise=true , address=145.79.227.14:4839 , piece_id=65NEICBQJXMM6N6KBD5KGI5SFTVAV7NJJKBAET6HDGKI5VD325OQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.508815721 (5m57.970908204s) [noise=true , address=145.79.227.11:4731 , piece_id=XR2GN3LWECQJ4OTSM44QXNW4B6GE7P5C5EFZFKJXD2M4GCMJ54VQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:13.008461678 (5m58.470554161s) [noise=true , address=145.79.227.16:4912 , piece_id=YLC5DB6H62T67W5LPTPQIQMWGJAN5ODYJB5PXIVX6X7JNFVCTNTA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:13.008503868 (5m58.470596351s) [seg_pos={0 0}, upload=85]: Segment reached optimal threshold of 13 pieces. Sep 23 13:37:13.008675254 (5m58.470767738s) [noise=true , address=145.79.227.12:4767 , piece_id=7JPOEJPRMBWO5MCNTISEDMBSYFACGTPJJUCBDX6EVYUKR3E7MYSA, seg_pos={0 0}, upload=85]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5): protocol: write tcp 188.166.225.170:41718->145.79.227.12:4767: use of closed network connection; write tcp 188.166.225.170:41718->145.79.227.12:4767: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41718->145.79.227.12:4767: use of closed network connection Sep 23 13:37:13.008689582 (5m58.470782065s) [noise=true , address=145.79.227.12:4767 , piece_id=7JPOEJPRMBWO5MCNTISEDMBSYFACGTPJJUCBDX6EVYUKR3E7MYSA, seg_pos={0 0}, upload=85]: Long tail cancellation due to optimal threshold Sep 23 13:37:13.008703729 (5m58.470796212s) [noise=true , address=145.79.227.5:4221 , piece_id=37XI3ODHTNYYIW3TYSGEZKLL4REO4C7J7W4F3M54QPBS27HU2N7Q, seg_pos={0 0}, upload=85]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR): protocol: expected piece hash; context canceled; EOF Sep 23 13:37:13.008716322 (5m58.470808806s) [noise=true , address=145.79.227.5:4221 , piece_id=37XI3ODHTNYYIW3TYSGEZKLL4REO4C7J7W4F3M54QPBS27HU2N7Q, seg_pos={0 0}, upload=85]: Long tail cancellation due to optimal threshold Sep 23 13:37:13.008800553 (5m58.470893037s) [noise=true , address=145.79.227.14:4838 , piece_id=MG6N5Q7OOXD4OI6CV2VCSGIOOI6VB4GYFATAZ5WZWXFCZN6EFFLQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: expected piece hash; context canceled; EOF Sep 23 13:37:13.008809150 (5m58.470901633s) [noise=true , address=145.79.227.14:4838 , piece_id=MG6N5Q7OOXD4OI6CV2VCSGIOOI6VB4GYFATAZ5WZWXFCZN6EFFLQ, seg_pos={0 0}, upload=85]: Long tail cancellation due to optimal threshold Sep 23 13:37:13.008829819 (5m58.470922302s) [seg_pos={0 0}, upload=85]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 23 13:37:13.008908238 (5m58.471000721s) [upload=85]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 23 13:37:13.008914450 (5m58.471006923s) [upload=85]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 23 13:37:13.352201772 (5m58.814294255s) [upload=85]: Done uploading segments... Sep 23 13:37:13.352227461 (5m58.814319944s) [upload=85]: Upload finished. err: <nil> Duration: 6.528094663s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 8.407882899s FAIL (2025-09-23 13:45:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: context deadline exceeded ======LOGS====== Sep 23 13:37:08.880516737 (5m54.34260922s) [upload=85]: Starting upload Sep 23 13:37:08.880590076 (5m54.342682549s) [upload=85]: Done starting upload Sep 23 13:37:08.880678955 (5m54.342771438s) [upload=85]: Uploading segments... Sep 23 13:37:08.880816417 (5m54.3429089s) [upload=85]: Got next segment. Inline: false Sep 23 13:37:08.880868837 (5m54.34296132s) [upload=85]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 13:37:08.880873937 (5m54.34296642s) [upload=85]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 13:37:09.155479775 (5m54.617572258s) [seg_pos={0 0}, upload=85]: Begin upload segment... Sep 23 13:37:09.155561942 (5m54.617654425s) [seg_pos={0 0}, upload=85]: Done begin upload segment. Sep 23 13:37:09.155609312 (5m54.617701795s) [noise=true , address=145.79.227.5:4221 , piece_id=37XI3ODHTNYYIW3TYSGEZKLL4REO4C7J7W4F3M54QPBS27HU2N7Q, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.155653065 (5m54.617745548s) [noise=true , address=145.79.227.13:4801 , piece_id=KDNEMJNYSDPBUV67TZ6PFMKXECG2TLUFSFX742MK5XRK32C2NHXA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156257627 (5m54.61835011s) [noise=true , address=45.91.116.4:4938 , piece_id=YXX7C7LQDTPKFEVXO7E6HN6QNM562OWWOXPI232ZHCLSY3WE4BUA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156274629 (5m54.618367102s) [noise=true , address=145.79.227.14:4839 , piece_id=65NEICBQJXMM6N6KBD5KGI5SFTVAV7NJJKBAET6HDGKI5VD325OQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156622501 (5m54.618715004s) [noise=true , address=145.79.227.6:4259 , piece_id=LPZNOJHDSME5TCQX2HQ544ZANJCJKGWIUXFLIOMNNSZBLYPNJTVA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156777296 (5m54.618869779s) [noise=true , address=145.79.227.16:4912 , piece_id=YLC5DB6H62T67W5LPTPQIQMWGJAN5ODYJB5PXIVX6X7JNFVCTNTA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156984401 (5m54.619076884s) [noise=true , address=145.79.227.14:4838 , piece_id=MG6N5Q7OOXD4OI6CV2VCSGIOOI6VB4GYFATAZ5WZWXFCZN6EFFLQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.157122624 (5m54.619215107s) [noise=true , address=45.91.116.10:4083 , piece_id=VONNFDRLV44LE65ZK7ENEJPNRMA2IPUGGGO3QGYJZLZLWOMNLFRQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.157345388 (5m54.619437871s) [noise=true , address=145.79.227.15:4876 , piece_id=ZHNZB3WQ4QQWR2LO6AQ64ONLIZ2EKCOBYGXFNR5FBY4CZL6QLVMA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.158372935 (5m54.620465408s) [noise=true , address=45.91.116.14:4189 , piece_id=HBFO3FNIMMF5UIDBXTFQQG6WZ4SFGJI3W6CSJ5EBAGN3D7JI6MSA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.159036498 (5m54.621128982s) [noise=true , address=45.91.116.11:4117 , piece_id=QEF7PSZ4ZAV2XM546G6QVKSJVH4JGCSYRAU572ABBMRGKVP23N6A, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.159800133 (5m54.621892616s) [noise=true , address=145.79.227.12:4765 , piece_id=H4PQGKP3XUGR6ARQ3UW3ZVPNCPS5WA5NM6FYJG5TBU3FALNXQYNA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.161637361 (5m54.623729844s) [noise=true , address=145.79.227.17:4970 , piece_id=IW4WTHW5W4SFOYMCXENR3DGYCA2OQJ4OO5RSXPYX6SUIGWKA4LBA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.163221126 (5m54.62531361s) [noise=true , address=77.90.31.11:4553 , piece_id=7XXKVSGNIYW5ZN6G5AV3JWP56QHF2HI4KOXJZ2DNOSGDHVZXDQTQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.163665793 (5m54.625758277s) [noise=true , address=145.79.227.11:4731 , piece_id=XR2GN3LWECQJ4OTSM44QXNW4B6GE7P5C5EFZFKJXD2M4GCMJ54VQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.164366027 (5m54.62645851s) [noise=true , address=145.79.227.12:4767 , piece_id=7JPOEJPRMBWO5MCNTISEDMBSYFACGTPJJUCBDX6EVYUKR3E7MYSA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:10.933255114 (5m56.395347597s) [upload=85]: Next returned nil segment Sep 23 13:37:10.933279932 (5m56.395372415s) [upload=85]: Waiting for error group managing segments... Sep 23 13:37:11.355919055 (5m56.818011548s) [noise=true , address=45.91.116.10:4083 , piece_id=VONNFDRLV44LE65ZK7ENEJPNRMA2IPUGGGO3QGYJZLZLWOMNLFRQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.366419560 (5m56.828512043s) [noise=true , address=45.91.116.4:4938 , piece_id=YXX7C7LQDTPKFEVXO7E6HN6QNM562OWWOXPI232ZHCLSY3WE4BUA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.373438585 (5m56.835531068s) [noise=true , address=45.91.116.11:4117 , piece_id=QEF7PSZ4ZAV2XM546G6QVKSJVH4JGCSYRAU572ABBMRGKVP23N6A, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.388282841 (5m56.850375324s) [noise=true , address=45.91.116.14:4189 , piece_id=HBFO3FNIMMF5UIDBXTFQQG6WZ4SFGJI3W6CSJ5EBAGN3D7JI6MSA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.396367205 (5m56.858459688s) [noise=true , address=77.90.31.11:4553 , piece_id=7XXKVSGNIYW5ZN6G5AV3JWP56QHF2HI4KOXJZ2DNOSGDHVZXDQTQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.050842867 (5m57.51293534s) [noise=true , address=145.79.227.13:4801 , piece_id=KDNEMJNYSDPBUV67TZ6PFMKXECG2TLUFSFX742MK5XRK32C2NHXA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.054995253 (5m57.517087736s) [noise=true , address=145.79.227.15:4876 , piece_id=ZHNZB3WQ4QQWR2LO6AQ64ONLIZ2EKCOBYGXFNR5FBY4CZL6QLVMA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.072213210 (5m57.534305693s) [noise=true , address=145.79.227.17:4970 , piece_id=IW4WTHW5W4SFOYMCXENR3DGYCA2OQJ4OO5RSXPYX6SUIGWKA4LBA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.101168265 (5m57.563260748s) [noise=true , address=145.79.227.6:4259 , piece_id=LPZNOJHDSME5TCQX2HQ544ZANJCJKGWIUXFLIOMNNSZBLYPNJTVA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.106130595 (5m57.568223078s) [noise=true , address=145.79.227.12:4765 , piece_id=H4PQGKP3XUGR6ARQ3UW3ZVPNCPS5WA5NM6FYJG5TBU3FALNXQYNA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.118733793 (5m57.580826277s) [noise=true , address=145.79.227.14:4839 , piece_id=65NEICBQJXMM6N6KBD5KGI5SFTVAV7NJJKBAET6HDGKI5VD325OQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.508815721 (5m57.970908204s) [noise=true , address=145.79.227.11:4731 , piece_id=XR2GN3LWECQJ4OTSM44QXNW4B6GE7P5C5EFZFKJXD2M4GCMJ54VQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:13.008461678 (5m58.470554161s) [noise=true , address=145.79.227.16:4912 , piece_id=YLC5DB6H62T67W5LPTPQIQMWGJAN5ODYJB5PXIVX6X7JNFVCTNTA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:13.008503868 (5m58.470596351s) [seg_pos={0 0}, upload=85]: Segment reached optimal threshold of 13 pieces. Sep 23 13:37:13.008675254 (5m58.470767738s) [noise=true , address=145.79.227.12:4767 , piece_id=7JPOEJPRMBWO5MCNTISEDMBSYFACGTPJJUCBDX6EVYUKR3E7MYSA, seg_pos={0 0}, upload=85]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5): protocol: write tcp 188.166.225.170:41718->145.79.227.12:4767: use of closed network connection; write tcp 188.166.225.170:41718->145.79.227.12:4767: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41718->145.79.227.12:4767: use of closed network connection Sep 23 13:37:13.008689582 (5m58.470782065s) [noise=true , address=145.79.227.12:4767 , piece_id=7JPOEJPRMBWO5MCNTISEDMBSYFACGTPJJUCBDX6EVYUKR3E7MYSA, seg_pos={0 0}, upload=85]: Long tail cancellation due to optimal threshold Sep 23 13:37:13.008703729 (5m58.470796212s) [noise=true , address=145.79.227.5:4221 , piece_id=37XI3ODHTNYYIW3TYSGEZKLL4REO4C7J7W4F3M54QPBS27HU2N7Q, seg_pos={0 0}, upload=85]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR): protocol: expected piece hash; context canceled; EOF Sep 23 13:37:13.008716322 (5m58.470808806s) [noise=true , address=145.79.227.5:4221 , piece_id=37XI3ODHTNYYIW3TYSGEZKLL4REO4C7J7W4F3M54QPBS27HU2N7Q, seg_pos={0 0}, upload=85]: Long tail cancellation due to optimal threshold Sep 23 13:37:13.008800553 (5m58.470893037s) [noise=true , address=145.79.227.14:4838 , piece_id=MG6N5Q7OOXD4OI6CV2VCSGIOOI6VB4GYFATAZ5WZWXFCZN6EFFLQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: expected piece hash; context canceled; EOF Sep 23 13:37:13.008809150 (5m58.470901633s) [noise=true , address=145.79.227.14:4838 , piece_id=MG6N5Q7OOXD4OI6CV2VCSGIOOI6VB4GYFATAZ5WZWXFCZN6EFFLQ, seg_pos={0 0}, upload=85]: Long tail cancellation due to optimal threshold Sep 23 13:37:13.008829819 (5m58.470922302s) [seg_pos={0 0}, upload=85]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 23 13:37:13.008908238 (5m58.471000721s) [upload=85]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 23 13:37:13.008914450 (5m58.471006923s) [upload=85]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 23 13:37:13.352201772 (5m58.814294255s) [upload=85]: Done uploading segments... Sep 23 13:37:13.352227461 (5m58.814319944s) [upload=85]: Upload finished. err: <nil> Duration: 8.40791463s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 6.401536735s FAIL (2025-09-23 13:45:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: context deadline exceeded ======LOGS====== Sep 23 13:37:08.880516737 (5m54.34260922s) [upload=85]: Starting upload Sep 23 13:37:08.880590076 (5m54.342682549s) [upload=85]: Done starting upload Sep 23 13:37:08.880678955 (5m54.342771438s) [upload=85]: Uploading segments... Sep 23 13:37:08.880816417 (5m54.3429089s) [upload=85]: Got next segment. Inline: false Sep 23 13:37:08.880868837 (5m54.34296132s) [upload=85]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 13:37:08.880873937 (5m54.34296642s) [upload=85]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 13:37:09.155479775 (5m54.617572258s) [seg_pos={0 0}, upload=85]: Begin upload segment... Sep 23 13:37:09.155561942 (5m54.617654425s) [seg_pos={0 0}, upload=85]: Done begin upload segment. Sep 23 13:37:09.155609312 (5m54.617701795s) [noise=true , address=145.79.227.5:4221 , piece_id=37XI3ODHTNYYIW3TYSGEZKLL4REO4C7J7W4F3M54QPBS27HU2N7Q, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.155653065 (5m54.617745548s) [noise=true , address=145.79.227.13:4801 , piece_id=KDNEMJNYSDPBUV67TZ6PFMKXECG2TLUFSFX742MK5XRK32C2NHXA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156257627 (5m54.61835011s) [noise=true , address=45.91.116.4:4938 , piece_id=YXX7C7LQDTPKFEVXO7E6HN6QNM562OWWOXPI232ZHCLSY3WE4BUA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156274629 (5m54.618367102s) [noise=true , address=145.79.227.14:4839 , piece_id=65NEICBQJXMM6N6KBD5KGI5SFTVAV7NJJKBAET6HDGKI5VD325OQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156622501 (5m54.618715004s) [noise=true , address=145.79.227.6:4259 , piece_id=LPZNOJHDSME5TCQX2HQ544ZANJCJKGWIUXFLIOMNNSZBLYPNJTVA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156777296 (5m54.618869779s) [noise=true , address=145.79.227.16:4912 , piece_id=YLC5DB6H62T67W5LPTPQIQMWGJAN5ODYJB5PXIVX6X7JNFVCTNTA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156984401 (5m54.619076884s) [noise=true , address=145.79.227.14:4838 , piece_id=MG6N5Q7OOXD4OI6CV2VCSGIOOI6VB4GYFATAZ5WZWXFCZN6EFFLQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.157122624 (5m54.619215107s) [noise=true , address=45.91.116.10:4083 , piece_id=VONNFDRLV44LE65ZK7ENEJPNRMA2IPUGGGO3QGYJZLZLWOMNLFRQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.157345388 (5m54.619437871s) [noise=true , address=145.79.227.15:4876 , piece_id=ZHNZB3WQ4QQWR2LO6AQ64ONLIZ2EKCOBYGXFNR5FBY4CZL6QLVMA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.158372935 (5m54.620465408s) [noise=true , address=45.91.116.14:4189 , piece_id=HBFO3FNIMMF5UIDBXTFQQG6WZ4SFGJI3W6CSJ5EBAGN3D7JI6MSA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.159036498 (5m54.621128982s) [noise=true , address=45.91.116.11:4117 , piece_id=QEF7PSZ4ZAV2XM546G6QVKSJVH4JGCSYRAU572ABBMRGKVP23N6A, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.159800133 (5m54.621892616s) [noise=true , address=145.79.227.12:4765 , piece_id=H4PQGKP3XUGR6ARQ3UW3ZVPNCPS5WA5NM6FYJG5TBU3FALNXQYNA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.161637361 (5m54.623729844s) [noise=true , address=145.79.227.17:4970 , piece_id=IW4WTHW5W4SFOYMCXENR3DGYCA2OQJ4OO5RSXPYX6SUIGWKA4LBA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.163221126 (5m54.62531361s) [noise=true , address=77.90.31.11:4553 , piece_id=7XXKVSGNIYW5ZN6G5AV3JWP56QHF2HI4KOXJZ2DNOSGDHVZXDQTQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.163665793 (5m54.625758277s) [noise=true , address=145.79.227.11:4731 , piece_id=XR2GN3LWECQJ4OTSM44QXNW4B6GE7P5C5EFZFKJXD2M4GCMJ54VQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.164366027 (5m54.62645851s) [noise=true , address=145.79.227.12:4767 , piece_id=7JPOEJPRMBWO5MCNTISEDMBSYFACGTPJJUCBDX6EVYUKR3E7MYSA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:10.933255114 (5m56.395347597s) [upload=85]: Next returned nil segment Sep 23 13:37:10.933279932 (5m56.395372415s) [upload=85]: Waiting for error group managing segments... Sep 23 13:37:11.355919055 (5m56.818011548s) [noise=true , address=45.91.116.10:4083 , piece_id=VONNFDRLV44LE65ZK7ENEJPNRMA2IPUGGGO3QGYJZLZLWOMNLFRQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.366419560 (5m56.828512043s) [noise=true , address=45.91.116.4:4938 , piece_id=YXX7C7LQDTPKFEVXO7E6HN6QNM562OWWOXPI232ZHCLSY3WE4BUA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.373438585 (5m56.835531068s) [noise=true , address=45.91.116.11:4117 , piece_id=QEF7PSZ4ZAV2XM546G6QVKSJVH4JGCSYRAU572ABBMRGKVP23N6A, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.388282841 (5m56.850375324s) [noise=true , address=45.91.116.14:4189 , piece_id=HBFO3FNIMMF5UIDBXTFQQG6WZ4SFGJI3W6CSJ5EBAGN3D7JI6MSA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.396367205 (5m56.858459688s) [noise=true , address=77.90.31.11:4553 , piece_id=7XXKVSGNIYW5ZN6G5AV3JWP56QHF2HI4KOXJZ2DNOSGDHVZXDQTQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.050842867 (5m57.51293534s) [noise=true , address=145.79.227.13:4801 , piece_id=KDNEMJNYSDPBUV67TZ6PFMKXECG2TLUFSFX742MK5XRK32C2NHXA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.054995253 (5m57.517087736s) [noise=true , address=145.79.227.15:4876 , piece_id=ZHNZB3WQ4QQWR2LO6AQ64ONLIZ2EKCOBYGXFNR5FBY4CZL6QLVMA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.072213210 (5m57.534305693s) [noise=true , address=145.79.227.17:4970 , piece_id=IW4WTHW5W4SFOYMCXENR3DGYCA2OQJ4OO5RSXPYX6SUIGWKA4LBA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.101168265 (5m57.563260748s) [noise=true , address=145.79.227.6:4259 , piece_id=LPZNOJHDSME5TCQX2HQ544ZANJCJKGWIUXFLIOMNNSZBLYPNJTVA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.106130595 (5m57.568223078s) [noise=true , address=145.79.227.12:4765 , piece_id=H4PQGKP3XUGR6ARQ3UW3ZVPNCPS5WA5NM6FYJG5TBU3FALNXQYNA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.118733793 (5m57.580826277s) [noise=true , address=145.79.227.14:4839 , piece_id=65NEICBQJXMM6N6KBD5KGI5SFTVAV7NJJKBAET6HDGKI5VD325OQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.508815721 (5m57.970908204s) [noise=true , address=145.79.227.11:4731 , piece_id=XR2GN3LWECQJ4OTSM44QXNW4B6GE7P5C5EFZFKJXD2M4GCMJ54VQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:13.008461678 (5m58.470554161s) [noise=true , address=145.79.227.16:4912 , piece_id=YLC5DB6H62T67W5LPTPQIQMWGJAN5ODYJB5PXIVX6X7JNFVCTNTA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:13.008503868 (5m58.470596351s) [seg_pos={0 0}, upload=85]: Segment reached optimal threshold of 13 pieces. Sep 23 13:37:13.008675254 (5m58.470767738s) [noise=true , address=145.79.227.12:4767 , piece_id=7JPOEJPRMBWO5MCNTISEDMBSYFACGTPJJUCBDX6EVYUKR3E7MYSA, seg_pos={0 0}, upload=85]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5): protocol: write tcp 188.166.225.170:41718->145.79.227.12:4767: use of closed network connection; write tcp 188.166.225.170:41718->145.79.227.12:4767: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41718->145.79.227.12:4767: use of closed network connection Sep 23 13:37:13.008689582 (5m58.470782065s) [noise=true , address=145.79.227.12:4767 , piece_id=7JPOEJPRMBWO5MCNTISEDMBSYFACGTPJJUCBDX6EVYUKR3E7MYSA, seg_pos={0 0}, upload=85]: Long tail cancellation due to optimal threshold Sep 23 13:37:13.008703729 (5m58.470796212s) [noise=true , address=145.79.227.5:4221 , piece_id=37XI3ODHTNYYIW3TYSGEZKLL4REO4C7J7W4F3M54QPBS27HU2N7Q, seg_pos={0 0}, upload=85]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR): protocol: expected piece hash; context canceled; EOF Sep 23 13:37:13.008716322 (5m58.470808806s) [noise=true , address=145.79.227.5:4221 , piece_id=37XI3ODHTNYYIW3TYSGEZKLL4REO4C7J7W4F3M54QPBS27HU2N7Q, seg_pos={0 0}, upload=85]: Long tail cancellation due to optimal threshold Sep 23 13:37:13.008800553 (5m58.470893037s) [noise=true , address=145.79.227.14:4838 , piece_id=MG6N5Q7OOXD4OI6CV2VCSGIOOI6VB4GYFATAZ5WZWXFCZN6EFFLQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: expected piece hash; context canceled; EOF Sep 23 13:37:13.008809150 (5m58.470901633s) [noise=true , address=145.79.227.14:4838 , piece_id=MG6N5Q7OOXD4OI6CV2VCSGIOOI6VB4GYFATAZ5WZWXFCZN6EFFLQ, seg_pos={0 0}, upload=85]: Long tail cancellation due to optimal threshold Sep 23 13:37:13.008829819 (5m58.470922302s) [seg_pos={0 0}, upload=85]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 23 13:37:13.008908238 (5m58.471000721s) [upload=85]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 23 13:37:13.008914450 (5m58.471006923s) [upload=85]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 23 13:37:13.352201772 (5m58.814294255s) [upload=85]: Done uploading segments... Sep 23 13:37:13.352227461 (5m58.814319944s) [upload=85]: Upload finished. err: <nil> Duration: 6.401590316s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 6.655664469s FAIL (2025-09-23 13:44:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: context deadline exceeded ======LOGS====== Sep 23 13:37:08.880516737 (5m54.34260922s) [upload=85]: Starting upload Sep 23 13:37:08.880590076 (5m54.342682549s) [upload=85]: Done starting upload Sep 23 13:37:08.880678955 (5m54.342771438s) [upload=85]: Uploading segments... Sep 23 13:37:08.880816417 (5m54.3429089s) [upload=85]: Got next segment. Inline: false Sep 23 13:37:08.880868837 (5m54.34296132s) [upload=85]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 13:37:08.880873937 (5m54.34296642s) [upload=85]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 13:37:09.155479775 (5m54.617572258s) [seg_pos={0 0}, upload=85]: Begin upload segment... Sep 23 13:37:09.155561942 (5m54.617654425s) [seg_pos={0 0}, upload=85]: Done begin upload segment. Sep 23 13:37:09.155609312 (5m54.617701795s) [noise=true , address=145.79.227.5:4221 , piece_id=37XI3ODHTNYYIW3TYSGEZKLL4REO4C7J7W4F3M54QPBS27HU2N7Q, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.155653065 (5m54.617745548s) [noise=true , address=145.79.227.13:4801 , piece_id=KDNEMJNYSDPBUV67TZ6PFMKXECG2TLUFSFX742MK5XRK32C2NHXA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156257627 (5m54.61835011s) [noise=true , address=45.91.116.4:4938 , piece_id=YXX7C7LQDTPKFEVXO7E6HN6QNM562OWWOXPI232ZHCLSY3WE4BUA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156274629 (5m54.618367102s) [noise=true , address=145.79.227.14:4839 , piece_id=65NEICBQJXMM6N6KBD5KGI5SFTVAV7NJJKBAET6HDGKI5VD325OQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156622501 (5m54.618715004s) [noise=true , address=145.79.227.6:4259 , piece_id=LPZNOJHDSME5TCQX2HQ544ZANJCJKGWIUXFLIOMNNSZBLYPNJTVA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156777296 (5m54.618869779s) [noise=true , address=145.79.227.16:4912 , piece_id=YLC5DB6H62T67W5LPTPQIQMWGJAN5ODYJB5PXIVX6X7JNFVCTNTA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156984401 (5m54.619076884s) [noise=true , address=145.79.227.14:4838 , piece_id=MG6N5Q7OOXD4OI6CV2VCSGIOOI6VB4GYFATAZ5WZWXFCZN6EFFLQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.157122624 (5m54.619215107s) [noise=true , address=45.91.116.10:4083 , piece_id=VONNFDRLV44LE65ZK7ENEJPNRMA2IPUGGGO3QGYJZLZLWOMNLFRQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.157345388 (5m54.619437871s) [noise=true , address=145.79.227.15:4876 , piece_id=ZHNZB3WQ4QQWR2LO6AQ64ONLIZ2EKCOBYGXFNR5FBY4CZL6QLVMA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.158372935 (5m54.620465408s) [noise=true , address=45.91.116.14:4189 , piece_id=HBFO3FNIMMF5UIDBXTFQQG6WZ4SFGJI3W6CSJ5EBAGN3D7JI6MSA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.159036498 (5m54.621128982s) [noise=true , address=45.91.116.11:4117 , piece_id=QEF7PSZ4ZAV2XM546G6QVKSJVH4JGCSYRAU572ABBMRGKVP23N6A, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.159800133 (5m54.621892616s) [noise=true , address=145.79.227.12:4765 , piece_id=H4PQGKP3XUGR6ARQ3UW3ZVPNCPS5WA5NM6FYJG5TBU3FALNXQYNA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.161637361 (5m54.623729844s) [noise=true , address=145.79.227.17:4970 , piece_id=IW4WTHW5W4SFOYMCXENR3DGYCA2OQJ4OO5RSXPYX6SUIGWKA4LBA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.163221126 (5m54.62531361s) [noise=true , address=77.90.31.11:4553 , piece_id=7XXKVSGNIYW5ZN6G5AV3JWP56QHF2HI4KOXJZ2DNOSGDHVZXDQTQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.163665793 (5m54.625758277s) [noise=true , address=145.79.227.11:4731 , piece_id=XR2GN3LWECQJ4OTSM44QXNW4B6GE7P5C5EFZFKJXD2M4GCMJ54VQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.164366027 (5m54.62645851s) [noise=true , address=145.79.227.12:4767 , piece_id=7JPOEJPRMBWO5MCNTISEDMBSYFACGTPJJUCBDX6EVYUKR3E7MYSA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:10.933255114 (5m56.395347597s) [upload=85]: Next returned nil segment Sep 23 13:37:10.933279932 (5m56.395372415s) [upload=85]: Waiting for error group managing segments... Sep 23 13:37:11.355919055 (5m56.818011548s) [noise=true , address=45.91.116.10:4083 , piece_id=VONNFDRLV44LE65ZK7ENEJPNRMA2IPUGGGO3QGYJZLZLWOMNLFRQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.366419560 (5m56.828512043s) [noise=true , address=45.91.116.4:4938 , piece_id=YXX7C7LQDTPKFEVXO7E6HN6QNM562OWWOXPI232ZHCLSY3WE4BUA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.373438585 (5m56.835531068s) [noise=true , address=45.91.116.11:4117 , piece_id=QEF7PSZ4ZAV2XM546G6QVKSJVH4JGCSYRAU572ABBMRGKVP23N6A, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.388282841 (5m56.850375324s) [noise=true , address=45.91.116.14:4189 , piece_id=HBFO3FNIMMF5UIDBXTFQQG6WZ4SFGJI3W6CSJ5EBAGN3D7JI6MSA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.396367205 (5m56.858459688s) [noise=true , address=77.90.31.11:4553 , piece_id=7XXKVSGNIYW5ZN6G5AV3JWP56QHF2HI4KOXJZ2DNOSGDHVZXDQTQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.050842867 (5m57.51293534s) [noise=true , address=145.79.227.13:4801 , piece_id=KDNEMJNYSDPBUV67TZ6PFMKXECG2TLUFSFX742MK5XRK32C2NHXA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.054995253 (5m57.517087736s) [noise=true , address=145.79.227.15:4876 , piece_id=ZHNZB3WQ4QQWR2LO6AQ64ONLIZ2EKCOBYGXFNR5FBY4CZL6QLVMA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.072213210 (5m57.534305693s) [noise=true , address=145.79.227.17:4970 , piece_id=IW4WTHW5W4SFOYMCXENR3DGYCA2OQJ4OO5RSXPYX6SUIGWKA4LBA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.101168265 (5m57.563260748s) [noise=true , address=145.79.227.6:4259 , piece_id=LPZNOJHDSME5TCQX2HQ544ZANJCJKGWIUXFLIOMNNSZBLYPNJTVA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.106130595 (5m57.568223078s) [noise=true , address=145.79.227.12:4765 , piece_id=H4PQGKP3XUGR6ARQ3UW3ZVPNCPS5WA5NM6FYJG5TBU3FALNXQYNA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.118733793 (5m57.580826277s) [noise=true , address=145.79.227.14:4839 , piece_id=65NEICBQJXMM6N6KBD5KGI5SFTVAV7NJJKBAET6HDGKI5VD325OQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.508815721 (5m57.970908204s) [noise=true , address=145.79.227.11:4731 , piece_id=XR2GN3LWECQJ4OTSM44QXNW4B6GE7P5C5EFZFKJXD2M4GCMJ54VQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:13.008461678 (5m58.470554161s) [noise=true , address=145.79.227.16:4912 , piece_id=YLC5DB6H62T67W5LPTPQIQMWGJAN5ODYJB5PXIVX6X7JNFVCTNTA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:13.008503868 (5m58.470596351s) [seg_pos={0 0}, upload=85]: Segment reached optimal threshold of 13 pieces. Sep 23 13:37:13.008675254 (5m58.470767738s) [noise=true , address=145.79.227.12:4767 , piece_id=7JPOEJPRMBWO5MCNTISEDMBSYFACGTPJJUCBDX6EVYUKR3E7MYSA, seg_pos={0 0}, upload=85]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5): protocol: write tcp 188.166.225.170:41718->145.79.227.12:4767: use of closed network connection; write tcp 188.166.225.170:41718->145.79.227.12:4767: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41718->145.79.227.12:4767: use of closed network connection Sep 23 13:37:13.008689582 (5m58.470782065s) [noise=true , address=145.79.227.12:4767 , piece_id=7JPOEJPRMBWO5MCNTISEDMBSYFACGTPJJUCBDX6EVYUKR3E7MYSA, seg_pos={0 0}, upload=85]: Long tail cancellation due to optimal threshold Sep 23 13:37:13.008703729 (5m58.470796212s) [noise=true , address=145.79.227.5:4221 , piece_id=37XI3ODHTNYYIW3TYSGEZKLL4REO4C7J7W4F3M54QPBS27HU2N7Q, seg_pos={0 0}, upload=85]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR): protocol: expected piece hash; context canceled; EOF Sep 23 13:37:13.008716322 (5m58.470808806s) [noise=true , address=145.79.227.5:4221 , piece_id=37XI3ODHTNYYIW3TYSGEZKLL4REO4C7J7W4F3M54QPBS27HU2N7Q, seg_pos={0 0}, upload=85]: Long tail cancellation due to optimal threshold Sep 23 13:37:13.008800553 (5m58.470893037s) [noise=true , address=145.79.227.14:4838 , piece_id=MG6N5Q7OOXD4OI6CV2VCSGIOOI6VB4GYFATAZ5WZWXFCZN6EFFLQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: expected piece hash; context canceled; EOF Sep 23 13:37:13.008809150 (5m58.470901633s) [noise=true , address=145.79.227.14:4838 , piece_id=MG6N5Q7OOXD4OI6CV2VCSGIOOI6VB4GYFATAZ5WZWXFCZN6EFFLQ, seg_pos={0 0}, upload=85]: Long tail cancellation due to optimal threshold Sep 23 13:37:13.008829819 (5m58.470922302s) [seg_pos={0 0}, upload=85]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 23 13:37:13.008908238 (5m58.471000721s) [upload=85]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 23 13:37:13.008914450 (5m58.471006923s) [upload=85]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 23 13:37:13.352201772 (5m58.814294255s) [upload=85]: Done uploading segments... Sep 23 13:37:13.352227461 (5m58.814319944s) [upload=85]: Upload finished. err: <nil> Duration: 6.655688215s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 7.371023506s FAIL (2025-09-23 13:44:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: context deadline exceeded ======LOGS====== Sep 23 13:37:08.880516737 (5m54.34260922s) [upload=85]: Starting upload Sep 23 13:37:08.880590076 (5m54.342682549s) [upload=85]: Done starting upload Sep 23 13:37:08.880678955 (5m54.342771438s) [upload=85]: Uploading segments... Sep 23 13:37:08.880816417 (5m54.3429089s) [upload=85]: Got next segment. Inline: false Sep 23 13:37:08.880868837 (5m54.34296132s) [upload=85]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 13:37:08.880873937 (5m54.34296642s) [upload=85]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 13:37:09.155479775 (5m54.617572258s) [seg_pos={0 0}, upload=85]: Begin upload segment... Sep 23 13:37:09.155561942 (5m54.617654425s) [seg_pos={0 0}, upload=85]: Done begin upload segment. Sep 23 13:37:09.155609312 (5m54.617701795s) [noise=true , address=145.79.227.5:4221 , piece_id=37XI3ODHTNYYIW3TYSGEZKLL4REO4C7J7W4F3M54QPBS27HU2N7Q, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.155653065 (5m54.617745548s) [noise=true , address=145.79.227.13:4801 , piece_id=KDNEMJNYSDPBUV67TZ6PFMKXECG2TLUFSFX742MK5XRK32C2NHXA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156257627 (5m54.61835011s) [noise=true , address=45.91.116.4:4938 , piece_id=YXX7C7LQDTPKFEVXO7E6HN6QNM562OWWOXPI232ZHCLSY3WE4BUA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156274629 (5m54.618367102s) [noise=true , address=145.79.227.14:4839 , piece_id=65NEICBQJXMM6N6KBD5KGI5SFTVAV7NJJKBAET6HDGKI5VD325OQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156622501 (5m54.618715004s) [noise=true , address=145.79.227.6:4259 , piece_id=LPZNOJHDSME5TCQX2HQ544ZANJCJKGWIUXFLIOMNNSZBLYPNJTVA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156777296 (5m54.618869779s) [noise=true , address=145.79.227.16:4912 , piece_id=YLC5DB6H62T67W5LPTPQIQMWGJAN5ODYJB5PXIVX6X7JNFVCTNTA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156984401 (5m54.619076884s) [noise=true , address=145.79.227.14:4838 , piece_id=MG6N5Q7OOXD4OI6CV2VCSGIOOI6VB4GYFATAZ5WZWXFCZN6EFFLQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.157122624 (5m54.619215107s) [noise=true , address=45.91.116.10:4083 , piece_id=VONNFDRLV44LE65ZK7ENEJPNRMA2IPUGGGO3QGYJZLZLWOMNLFRQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.157345388 (5m54.619437871s) [noise=true , address=145.79.227.15:4876 , piece_id=ZHNZB3WQ4QQWR2LO6AQ64ONLIZ2EKCOBYGXFNR5FBY4CZL6QLVMA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.158372935 (5m54.620465408s) [noise=true , address=45.91.116.14:4189 , piece_id=HBFO3FNIMMF5UIDBXTFQQG6WZ4SFGJI3W6CSJ5EBAGN3D7JI6MSA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.159036498 (5m54.621128982s) [noise=true , address=45.91.116.11:4117 , piece_id=QEF7PSZ4ZAV2XM546G6QVKSJVH4JGCSYRAU572ABBMRGKVP23N6A, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.159800133 (5m54.621892616s) [noise=true , address=145.79.227.12:4765 , piece_id=H4PQGKP3XUGR6ARQ3UW3ZVPNCPS5WA5NM6FYJG5TBU3FALNXQYNA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.161637361 (5m54.623729844s) [noise=true , address=145.79.227.17:4970 , piece_id=IW4WTHW5W4SFOYMCXENR3DGYCA2OQJ4OO5RSXPYX6SUIGWKA4LBA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.163221126 (5m54.62531361s) [noise=true , address=77.90.31.11:4553 , piece_id=7XXKVSGNIYW5ZN6G5AV3JWP56QHF2HI4KOXJZ2DNOSGDHVZXDQTQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.163665793 (5m54.625758277s) [noise=true , address=145.79.227.11:4731 , piece_id=XR2GN3LWECQJ4OTSM44QXNW4B6GE7P5C5EFZFKJXD2M4GCMJ54VQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.164366027 (5m54.62645851s) [noise=true , address=145.79.227.12:4767 , piece_id=7JPOEJPRMBWO5MCNTISEDMBSYFACGTPJJUCBDX6EVYUKR3E7MYSA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:10.933255114 (5m56.395347597s) [upload=85]: Next returned nil segment Sep 23 13:37:10.933279932 (5m56.395372415s) [upload=85]: Waiting for error group managing segments... Sep 23 13:37:11.355919055 (5m56.818011548s) [noise=true , address=45.91.116.10:4083 , piece_id=VONNFDRLV44LE65ZK7ENEJPNRMA2IPUGGGO3QGYJZLZLWOMNLFRQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.366419560 (5m56.828512043s) [noise=true , address=45.91.116.4:4938 , piece_id=YXX7C7LQDTPKFEVXO7E6HN6QNM562OWWOXPI232ZHCLSY3WE4BUA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.373438585 (5m56.835531068s) [noise=true , address=45.91.116.11:4117 , piece_id=QEF7PSZ4ZAV2XM546G6QVKSJVH4JGCSYRAU572ABBMRGKVP23N6A, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.388282841 (5m56.850375324s) [noise=true , address=45.91.116.14:4189 , piece_id=HBFO3FNIMMF5UIDBXTFQQG6WZ4SFGJI3W6CSJ5EBAGN3D7JI6MSA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.396367205 (5m56.858459688s) [noise=true , address=77.90.31.11:4553 , piece_id=7XXKVSGNIYW5ZN6G5AV3JWP56QHF2HI4KOXJZ2DNOSGDHVZXDQTQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.050842867 (5m57.51293534s) [noise=true , address=145.79.227.13:4801 , piece_id=KDNEMJNYSDPBUV67TZ6PFMKXECG2TLUFSFX742MK5XRK32C2NHXA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.054995253 (5m57.517087736s) [noise=true , address=145.79.227.15:4876 , piece_id=ZHNZB3WQ4QQWR2LO6AQ64ONLIZ2EKCOBYGXFNR5FBY4CZL6QLVMA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.072213210 (5m57.534305693s) [noise=true , address=145.79.227.17:4970 , piece_id=IW4WTHW5W4SFOYMCXENR3DGYCA2OQJ4OO5RSXPYX6SUIGWKA4LBA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.101168265 (5m57.563260748s) [noise=true , address=145.79.227.6:4259 , piece_id=LPZNOJHDSME5TCQX2HQ544ZANJCJKGWIUXFLIOMNNSZBLYPNJTVA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.106130595 (5m57.568223078s) [noise=true , address=145.79.227.12:4765 , piece_id=H4PQGKP3XUGR6ARQ3UW3ZVPNCPS5WA5NM6FYJG5TBU3FALNXQYNA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.118733793 (5m57.580826277s) [noise=true , address=145.79.227.14:4839 , piece_id=65NEICBQJXMM6N6KBD5KGI5SFTVAV7NJJKBAET6HDGKI5VD325OQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.508815721 (5m57.970908204s) [noise=true , address=145.79.227.11:4731 , piece_id=XR2GN3LWECQJ4OTSM44QXNW4B6GE7P5C5EFZFKJXD2M4GCMJ54VQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:13.008461678 (5m58.470554161s) [noise=true , address=145.79.227.16:4912 , piece_id=YLC5DB6H62T67W5LPTPQIQMWGJAN5ODYJB5PXIVX6X7JNFVCTNTA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:13.008503868 (5m58.470596351s) [seg_pos={0 0}, upload=85]: Segment reached optimal threshold of 13 pieces. Sep 23 13:37:13.008675254 (5m58.470767738s) [noise=true , address=145.79.227.12:4767 , piece_id=7JPOEJPRMBWO5MCNTISEDMBSYFACGTPJJUCBDX6EVYUKR3E7MYSA, seg_pos={0 0}, upload=85]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5): protocol: write tcp 188.166.225.170:41718->145.79.227.12:4767: use of closed network connection; write tcp 188.166.225.170:41718->145.79.227.12:4767: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41718->145.79.227.12:4767: use of closed network connection Sep 23 13:37:13.008689582 (5m58.470782065s) [noise=true , address=145.79.227.12:4767 , piece_id=7JPOEJPRMBWO5MCNTISEDMBSYFACGTPJJUCBDX6EVYUKR3E7MYSA, seg_pos={0 0}, upload=85]: Long tail cancellation due to optimal threshold Sep 23 13:37:13.008703729 (5m58.470796212s) [noise=true , address=145.79.227.5:4221 , piece_id=37XI3ODHTNYYIW3TYSGEZKLL4REO4C7J7W4F3M54QPBS27HU2N7Q, seg_pos={0 0}, upload=85]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR): protocol: expected piece hash; context canceled; EOF Sep 23 13:37:13.008716322 (5m58.470808806s) [noise=true , address=145.79.227.5:4221 , piece_id=37XI3ODHTNYYIW3TYSGEZKLL4REO4C7J7W4F3M54QPBS27HU2N7Q, seg_pos={0 0}, upload=85]: Long tail cancellation due to optimal threshold Sep 23 13:37:13.008800553 (5m58.470893037s) [noise=true , address=145.79.227.14:4838 , piece_id=MG6N5Q7OOXD4OI6CV2VCSGIOOI6VB4GYFATAZ5WZWXFCZN6EFFLQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: expected piece hash; context canceled; EOF Sep 23 13:37:13.008809150 (5m58.470901633s) [noise=true , address=145.79.227.14:4838 , piece_id=MG6N5Q7OOXD4OI6CV2VCSGIOOI6VB4GYFATAZ5WZWXFCZN6EFFLQ, seg_pos={0 0}, upload=85]: Long tail cancellation due to optimal threshold Sep 23 13:37:13.008829819 (5m58.470922302s) [seg_pos={0 0}, upload=85]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 23 13:37:13.008908238 (5m58.471000721s) [upload=85]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 23 13:37:13.008914450 (5m58.471006923s) [upload=85]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 23 13:37:13.352201772 (5m58.814294255s) [upload=85]: Done uploading segments... Sep 23 13:37:13.352227461 (5m58.814319944s) [upload=85]: Upload finished. err: <nil> Duration: 7.371053544s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 6.006328015s FAIL (2025-09-23 13:43:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: context deadline exceeded ======LOGS====== Sep 23 13:37:08.880516737 (5m54.34260922s) [upload=85]: Starting upload Sep 23 13:37:08.880590076 (5m54.342682549s) [upload=85]: Done starting upload Sep 23 13:37:08.880678955 (5m54.342771438s) [upload=85]: Uploading segments... Sep 23 13:37:08.880816417 (5m54.3429089s) [upload=85]: Got next segment. Inline: false Sep 23 13:37:08.880868837 (5m54.34296132s) [upload=85]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 13:37:08.880873937 (5m54.34296642s) [upload=85]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 13:37:09.155479775 (5m54.617572258s) [seg_pos={0 0}, upload=85]: Begin upload segment... Sep 23 13:37:09.155561942 (5m54.617654425s) [seg_pos={0 0}, upload=85]: Done begin upload segment. Sep 23 13:37:09.155609312 (5m54.617701795s) [noise=true , address=145.79.227.5:4221 , piece_id=37XI3ODHTNYYIW3TYSGEZKLL4REO4C7J7W4F3M54QPBS27HU2N7Q, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.155653065 (5m54.617745548s) [noise=true , address=145.79.227.13:4801 , piece_id=KDNEMJNYSDPBUV67TZ6PFMKXECG2TLUFSFX742MK5XRK32C2NHXA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156257627 (5m54.61835011s) [noise=true , address=45.91.116.4:4938 , piece_id=YXX7C7LQDTPKFEVXO7E6HN6QNM562OWWOXPI232ZHCLSY3WE4BUA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156274629 (5m54.618367102s) [noise=true , address=145.79.227.14:4839 , piece_id=65NEICBQJXMM6N6KBD5KGI5SFTVAV7NJJKBAET6HDGKI5VD325OQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156622501 (5m54.618715004s) [noise=true , address=145.79.227.6:4259 , piece_id=LPZNOJHDSME5TCQX2HQ544ZANJCJKGWIUXFLIOMNNSZBLYPNJTVA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156777296 (5m54.618869779s) [noise=true , address=145.79.227.16:4912 , piece_id=YLC5DB6H62T67W5LPTPQIQMWGJAN5ODYJB5PXIVX6X7JNFVCTNTA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156984401 (5m54.619076884s) [noise=true , address=145.79.227.14:4838 , piece_id=MG6N5Q7OOXD4OI6CV2VCSGIOOI6VB4GYFATAZ5WZWXFCZN6EFFLQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.157122624 (5m54.619215107s) [noise=true , address=45.91.116.10:4083 , piece_id=VONNFDRLV44LE65ZK7ENEJPNRMA2IPUGGGO3QGYJZLZLWOMNLFRQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.157345388 (5m54.619437871s) [noise=true , address=145.79.227.15:4876 , piece_id=ZHNZB3WQ4QQWR2LO6AQ64ONLIZ2EKCOBYGXFNR5FBY4CZL6QLVMA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.158372935 (5m54.620465408s) [noise=true , address=45.91.116.14:4189 , piece_id=HBFO3FNIMMF5UIDBXTFQQG6WZ4SFGJI3W6CSJ5EBAGN3D7JI6MSA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.159036498 (5m54.621128982s) [noise=true , address=45.91.116.11:4117 , piece_id=QEF7PSZ4ZAV2XM546G6QVKSJVH4JGCSYRAU572ABBMRGKVP23N6A, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.159800133 (5m54.621892616s) [noise=true , address=145.79.227.12:4765 , piece_id=H4PQGKP3XUGR6ARQ3UW3ZVPNCPS5WA5NM6FYJG5TBU3FALNXQYNA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.161637361 (5m54.623729844s) [noise=true , address=145.79.227.17:4970 , piece_id=IW4WTHW5W4SFOYMCXENR3DGYCA2OQJ4OO5RSXPYX6SUIGWKA4LBA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.163221126 (5m54.62531361s) [noise=true , address=77.90.31.11:4553 , piece_id=7XXKVSGNIYW5ZN6G5AV3JWP56QHF2HI4KOXJZ2DNOSGDHVZXDQTQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.163665793 (5m54.625758277s) [noise=true , address=145.79.227.11:4731 , piece_id=XR2GN3LWECQJ4OTSM44QXNW4B6GE7P5C5EFZFKJXD2M4GCMJ54VQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.164366027 (5m54.62645851s) [noise=true , address=145.79.227.12:4767 , piece_id=7JPOEJPRMBWO5MCNTISEDMBSYFACGTPJJUCBDX6EVYUKR3E7MYSA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:10.933255114 (5m56.395347597s) [upload=85]: Next returned nil segment Sep 23 13:37:10.933279932 (5m56.395372415s) [upload=85]: Waiting for error group managing segments... Sep 23 13:37:11.355919055 (5m56.818011548s) [noise=true , address=45.91.116.10:4083 , piece_id=VONNFDRLV44LE65ZK7ENEJPNRMA2IPUGGGO3QGYJZLZLWOMNLFRQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.366419560 (5m56.828512043s) [noise=true , address=45.91.116.4:4938 , piece_id=YXX7C7LQDTPKFEVXO7E6HN6QNM562OWWOXPI232ZHCLSY3WE4BUA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.373438585 (5m56.835531068s) [noise=true , address=45.91.116.11:4117 , piece_id=QEF7PSZ4ZAV2XM546G6QVKSJVH4JGCSYRAU572ABBMRGKVP23N6A, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.388282841 (5m56.850375324s) [noise=true , address=45.91.116.14:4189 , piece_id=HBFO3FNIMMF5UIDBXTFQQG6WZ4SFGJI3W6CSJ5EBAGN3D7JI6MSA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.396367205 (5m56.858459688s) [noise=true , address=77.90.31.11:4553 , piece_id=7XXKVSGNIYW5ZN6G5AV3JWP56QHF2HI4KOXJZ2DNOSGDHVZXDQTQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.050842867 (5m57.51293534s) [noise=true , address=145.79.227.13:4801 , piece_id=KDNEMJNYSDPBUV67TZ6PFMKXECG2TLUFSFX742MK5XRK32C2NHXA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.054995253 (5m57.517087736s) [noise=true , address=145.79.227.15:4876 , piece_id=ZHNZB3WQ4QQWR2LO6AQ64ONLIZ2EKCOBYGXFNR5FBY4CZL6QLVMA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.072213210 (5m57.534305693s) [noise=true , address=145.79.227.17:4970 , piece_id=IW4WTHW5W4SFOYMCXENR3DGYCA2OQJ4OO5RSXPYX6SUIGWKA4LBA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.101168265 (5m57.563260748s) [noise=true , address=145.79.227.6:4259 , piece_id=LPZNOJHDSME5TCQX2HQ544ZANJCJKGWIUXFLIOMNNSZBLYPNJTVA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.106130595 (5m57.568223078s) [noise=true , address=145.79.227.12:4765 , piece_id=H4PQGKP3XUGR6ARQ3UW3ZVPNCPS5WA5NM6FYJG5TBU3FALNXQYNA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.118733793 (5m57.580826277s) [noise=true , address=145.79.227.14:4839 , piece_id=65NEICBQJXMM6N6KBD5KGI5SFTVAV7NJJKBAET6HDGKI5VD325OQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.508815721 (5m57.970908204s) [noise=true , address=145.79.227.11:4731 , piece_id=XR2GN3LWECQJ4OTSM44QXNW4B6GE7P5C5EFZFKJXD2M4GCMJ54VQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:13.008461678 (5m58.470554161s) [noise=true , address=145.79.227.16:4912 , piece_id=YLC5DB6H62T67W5LPTPQIQMWGJAN5ODYJB5PXIVX6X7JNFVCTNTA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:13.008503868 (5m58.470596351s) [seg_pos={0 0}, upload=85]: Segment reached optimal threshold of 13 pieces. Sep 23 13:37:13.008675254 (5m58.470767738s) [noise=true , address=145.79.227.12:4767 , piece_id=7JPOEJPRMBWO5MCNTISEDMBSYFACGTPJJUCBDX6EVYUKR3E7MYSA, seg_pos={0 0}, upload=85]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5): protocol: write tcp 188.166.225.170:41718->145.79.227.12:4767: use of closed network connection; write tcp 188.166.225.170:41718->145.79.227.12:4767: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41718->145.79.227.12:4767: use of closed network connection Sep 23 13:37:13.008689582 (5m58.470782065s) [noise=true , address=145.79.227.12:4767 , piece_id=7JPOEJPRMBWO5MCNTISEDMBSYFACGTPJJUCBDX6EVYUKR3E7MYSA, seg_pos={0 0}, upload=85]: Long tail cancellation due to optimal threshold Sep 23 13:37:13.008703729 (5m58.470796212s) [noise=true , address=145.79.227.5:4221 , piece_id=37XI3ODHTNYYIW3TYSGEZKLL4REO4C7J7W4F3M54QPBS27HU2N7Q, seg_pos={0 0}, upload=85]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR): protocol: expected piece hash; context canceled; EOF Sep 23 13:37:13.008716322 (5m58.470808806s) [noise=true , address=145.79.227.5:4221 , piece_id=37XI3ODHTNYYIW3TYSGEZKLL4REO4C7J7W4F3M54QPBS27HU2N7Q, seg_pos={0 0}, upload=85]: Long tail cancellation due to optimal threshold Sep 23 13:37:13.008800553 (5m58.470893037s) [noise=true , address=145.79.227.14:4838 , piece_id=MG6N5Q7OOXD4OI6CV2VCSGIOOI6VB4GYFATAZ5WZWXFCZN6EFFLQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: expected piece hash; context canceled; EOF Sep 23 13:37:13.008809150 (5m58.470901633s) [noise=true , address=145.79.227.14:4838 , piece_id=MG6N5Q7OOXD4OI6CV2VCSGIOOI6VB4GYFATAZ5WZWXFCZN6EFFLQ, seg_pos={0 0}, upload=85]: Long tail cancellation due to optimal threshold Sep 23 13:37:13.008829819 (5m58.470922302s) [seg_pos={0 0}, upload=85]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 23 13:37:13.008908238 (5m58.471000721s) [upload=85]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 23 13:37:13.008914450 (5m58.471006923s) [upload=85]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 23 13:37:13.352201772 (5m58.814294255s) [upload=85]: Done uploading segments... Sep 23 13:37:13.352227461 (5m58.814319944s) [upload=85]: Upload finished. err: <nil> Duration: 6.006360717s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 6.734248552s FAIL (2025-09-23 13:42:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: context deadline exceeded ======LOGS====== Sep 23 13:37:08.880516737 (5m54.34260922s) [upload=85]: Starting upload Sep 23 13:37:08.880590076 (5m54.342682549s) [upload=85]: Done starting upload Sep 23 13:37:08.880678955 (5m54.342771438s) [upload=85]: Uploading segments... Sep 23 13:37:08.880816417 (5m54.3429089s) [upload=85]: Got next segment. Inline: false Sep 23 13:37:08.880868837 (5m54.34296132s) [upload=85]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 13:37:08.880873937 (5m54.34296642s) [upload=85]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 13:37:09.155479775 (5m54.617572258s) [seg_pos={0 0}, upload=85]: Begin upload segment... Sep 23 13:37:09.155561942 (5m54.617654425s) [seg_pos={0 0}, upload=85]: Done begin upload segment. Sep 23 13:37:09.155609312 (5m54.617701795s) [noise=true , address=145.79.227.5:4221 , piece_id=37XI3ODHTNYYIW3TYSGEZKLL4REO4C7J7W4F3M54QPBS27HU2N7Q, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.155653065 (5m54.617745548s) [noise=true , address=145.79.227.13:4801 , piece_id=KDNEMJNYSDPBUV67TZ6PFMKXECG2TLUFSFX742MK5XRK32C2NHXA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156257627 (5m54.61835011s) [noise=true , address=45.91.116.4:4938 , piece_id=YXX7C7LQDTPKFEVXO7E6HN6QNM562OWWOXPI232ZHCLSY3WE4BUA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156274629 (5m54.618367102s) [noise=true , address=145.79.227.14:4839 , piece_id=65NEICBQJXMM6N6KBD5KGI5SFTVAV7NJJKBAET6HDGKI5VD325OQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156622501 (5m54.618715004s) [noise=true , address=145.79.227.6:4259 , piece_id=LPZNOJHDSME5TCQX2HQ544ZANJCJKGWIUXFLIOMNNSZBLYPNJTVA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156777296 (5m54.618869779s) [noise=true , address=145.79.227.16:4912 , piece_id=YLC5DB6H62T67W5LPTPQIQMWGJAN5ODYJB5PXIVX6X7JNFVCTNTA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156984401 (5m54.619076884s) [noise=true , address=145.79.227.14:4838 , piece_id=MG6N5Q7OOXD4OI6CV2VCSGIOOI6VB4GYFATAZ5WZWXFCZN6EFFLQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.157122624 (5m54.619215107s) [noise=true , address=45.91.116.10:4083 , piece_id=VONNFDRLV44LE65ZK7ENEJPNRMA2IPUGGGO3QGYJZLZLWOMNLFRQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.157345388 (5m54.619437871s) [noise=true , address=145.79.227.15:4876 , piece_id=ZHNZB3WQ4QQWR2LO6AQ64ONLIZ2EKCOBYGXFNR5FBY4CZL6QLVMA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.158372935 (5m54.620465408s) [noise=true , address=45.91.116.14:4189 , piece_id=HBFO3FNIMMF5UIDBXTFQQG6WZ4SFGJI3W6CSJ5EBAGN3D7JI6MSA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.159036498 (5m54.621128982s) [noise=true , address=45.91.116.11:4117 , piece_id=QEF7PSZ4ZAV2XM546G6QVKSJVH4JGCSYRAU572ABBMRGKVP23N6A, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.159800133 (5m54.621892616s) [noise=true , address=145.79.227.12:4765 , piece_id=H4PQGKP3XUGR6ARQ3UW3ZVPNCPS5WA5NM6FYJG5TBU3FALNXQYNA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.161637361 (5m54.623729844s) [noise=true , address=145.79.227.17:4970 , piece_id=IW4WTHW5W4SFOYMCXENR3DGYCA2OQJ4OO5RSXPYX6SUIGWKA4LBA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.163221126 (5m54.62531361s) [noise=true , address=77.90.31.11:4553 , piece_id=7XXKVSGNIYW5ZN6G5AV3JWP56QHF2HI4KOXJZ2DNOSGDHVZXDQTQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.163665793 (5m54.625758277s) [noise=true , address=145.79.227.11:4731 , piece_id=XR2GN3LWECQJ4OTSM44QXNW4B6GE7P5C5EFZFKJXD2M4GCMJ54VQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.164366027 (5m54.62645851s) [noise=true , address=145.79.227.12:4767 , piece_id=7JPOEJPRMBWO5MCNTISEDMBSYFACGTPJJUCBDX6EVYUKR3E7MYSA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:10.933255114 (5m56.395347597s) [upload=85]: Next returned nil segment Sep 23 13:37:10.933279932 (5m56.395372415s) [upload=85]: Waiting for error group managing segments... Sep 23 13:37:11.355919055 (5m56.818011548s) [noise=true , address=45.91.116.10:4083 , piece_id=VONNFDRLV44LE65ZK7ENEJPNRMA2IPUGGGO3QGYJZLZLWOMNLFRQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.366419560 (5m56.828512043s) [noise=true , address=45.91.116.4:4938 , piece_id=YXX7C7LQDTPKFEVXO7E6HN6QNM562OWWOXPI232ZHCLSY3WE4BUA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.373438585 (5m56.835531068s) [noise=true , address=45.91.116.11:4117 , piece_id=QEF7PSZ4ZAV2XM546G6QVKSJVH4JGCSYRAU572ABBMRGKVP23N6A, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.388282841 (5m56.850375324s) [noise=true , address=45.91.116.14:4189 , piece_id=HBFO3FNIMMF5UIDBXTFQQG6WZ4SFGJI3W6CSJ5EBAGN3D7JI6MSA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.396367205 (5m56.858459688s) [noise=true , address=77.90.31.11:4553 , piece_id=7XXKVSGNIYW5ZN6G5AV3JWP56QHF2HI4KOXJZ2DNOSGDHVZXDQTQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.050842867 (5m57.51293534s) [noise=true , address=145.79.227.13:4801 , piece_id=KDNEMJNYSDPBUV67TZ6PFMKXECG2TLUFSFX742MK5XRK32C2NHXA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.054995253 (5m57.517087736s) [noise=true , address=145.79.227.15:4876 , piece_id=ZHNZB3WQ4QQWR2LO6AQ64ONLIZ2EKCOBYGXFNR5FBY4CZL6QLVMA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.072213210 (5m57.534305693s) [noise=true , address=145.79.227.17:4970 , piece_id=IW4WTHW5W4SFOYMCXENR3DGYCA2OQJ4OO5RSXPYX6SUIGWKA4LBA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.101168265 (5m57.563260748s) [noise=true , address=145.79.227.6:4259 , piece_id=LPZNOJHDSME5TCQX2HQ544ZANJCJKGWIUXFLIOMNNSZBLYPNJTVA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.106130595 (5m57.568223078s) [noise=true , address=145.79.227.12:4765 , piece_id=H4PQGKP3XUGR6ARQ3UW3ZVPNCPS5WA5NM6FYJG5TBU3FALNXQYNA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.118733793 (5m57.580826277s) [noise=true , address=145.79.227.14:4839 , piece_id=65NEICBQJXMM6N6KBD5KGI5SFTVAV7NJJKBAET6HDGKI5VD325OQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.508815721 (5m57.970908204s) [noise=true , address=145.79.227.11:4731 , piece_id=XR2GN3LWECQJ4OTSM44QXNW4B6GE7P5C5EFZFKJXD2M4GCMJ54VQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:13.008461678 (5m58.470554161s) [noise=true , address=145.79.227.16:4912 , piece_id=YLC5DB6H62T67W5LPTPQIQMWGJAN5ODYJB5PXIVX6X7JNFVCTNTA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:13.008503868 (5m58.470596351s) [seg_pos={0 0}, upload=85]: Segment reached optimal threshold of 13 pieces. Sep 23 13:37:13.008675254 (5m58.470767738s) [noise=true , address=145.79.227.12:4767 , piece_id=7JPOEJPRMBWO5MCNTISEDMBSYFACGTPJJUCBDX6EVYUKR3E7MYSA, seg_pos={0 0}, upload=85]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5): protocol: write tcp 188.166.225.170:41718->145.79.227.12:4767: use of closed network connection; write tcp 188.166.225.170:41718->145.79.227.12:4767: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41718->145.79.227.12:4767: use of closed network connection Sep 23 13:37:13.008689582 (5m58.470782065s) [noise=true , address=145.79.227.12:4767 , piece_id=7JPOEJPRMBWO5MCNTISEDMBSYFACGTPJJUCBDX6EVYUKR3E7MYSA, seg_pos={0 0}, upload=85]: Long tail cancellation due to optimal threshold Sep 23 13:37:13.008703729 (5m58.470796212s) [noise=true , address=145.79.227.5:4221 , piece_id=37XI3ODHTNYYIW3TYSGEZKLL4REO4C7J7W4F3M54QPBS27HU2N7Q, seg_pos={0 0}, upload=85]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR): protocol: expected piece hash; context canceled; EOF Sep 23 13:37:13.008716322 (5m58.470808806s) [noise=true , address=145.79.227.5:4221 , piece_id=37XI3ODHTNYYIW3TYSGEZKLL4REO4C7J7W4F3M54QPBS27HU2N7Q, seg_pos={0 0}, upload=85]: Long tail cancellation due to optimal threshold Sep 23 13:37:13.008800553 (5m58.470893037s) [noise=true , address=145.79.227.14:4838 , piece_id=MG6N5Q7OOXD4OI6CV2VCSGIOOI6VB4GYFATAZ5WZWXFCZN6EFFLQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: expected piece hash; context canceled; EOF Sep 23 13:37:13.008809150 (5m58.470901633s) [noise=true , address=145.79.227.14:4838 , piece_id=MG6N5Q7OOXD4OI6CV2VCSGIOOI6VB4GYFATAZ5WZWXFCZN6EFFLQ, seg_pos={0 0}, upload=85]: Long tail cancellation due to optimal threshold Sep 23 13:37:13.008829819 (5m58.470922302s) [seg_pos={0 0}, upload=85]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 23 13:37:13.008908238 (5m58.471000721s) [upload=85]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 23 13:37:13.008914450 (5m58.471006923s) [upload=85]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 23 13:37:13.352201772 (5m58.814294255s) [upload=85]: Done uploading segments... Sep 23 13:37:13.352227461 (5m58.814319944s) [upload=85]: Upload finished. err: <nil> Duration: 6.734277647s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 6.595192611s FAIL (2025-09-23 13:41:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: context deadline exceeded ======LOGS====== Sep 23 13:37:08.880516737 (5m54.34260922s) [upload=85]: Starting upload Sep 23 13:37:08.880590076 (5m54.342682549s) [upload=85]: Done starting upload Sep 23 13:37:08.880678955 (5m54.342771438s) [upload=85]: Uploading segments... Sep 23 13:37:08.880816417 (5m54.3429089s) [upload=85]: Got next segment. Inline: false Sep 23 13:37:08.880868837 (5m54.34296132s) [upload=85]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 13:37:08.880873937 (5m54.34296642s) [upload=85]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 13:37:09.155479775 (5m54.617572258s) [seg_pos={0 0}, upload=85]: Begin upload segment... Sep 23 13:37:09.155561942 (5m54.617654425s) [seg_pos={0 0}, upload=85]: Done begin upload segment. Sep 23 13:37:09.155609312 (5m54.617701795s) [noise=true , address=145.79.227.5:4221 , piece_id=37XI3ODHTNYYIW3TYSGEZKLL4REO4C7J7W4F3M54QPBS27HU2N7Q, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.155653065 (5m54.617745548s) [noise=true , address=145.79.227.13:4801 , piece_id=KDNEMJNYSDPBUV67TZ6PFMKXECG2TLUFSFX742MK5XRK32C2NHXA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156257627 (5m54.61835011s) [noise=true , address=45.91.116.4:4938 , piece_id=YXX7C7LQDTPKFEVXO7E6HN6QNM562OWWOXPI232ZHCLSY3WE4BUA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156274629 (5m54.618367102s) [noise=true , address=145.79.227.14:4839 , piece_id=65NEICBQJXMM6N6KBD5KGI5SFTVAV7NJJKBAET6HDGKI5VD325OQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156622501 (5m54.618715004s) [noise=true , address=145.79.227.6:4259 , piece_id=LPZNOJHDSME5TCQX2HQ544ZANJCJKGWIUXFLIOMNNSZBLYPNJTVA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156777296 (5m54.618869779s) [noise=true , address=145.79.227.16:4912 , piece_id=YLC5DB6H62T67W5LPTPQIQMWGJAN5ODYJB5PXIVX6X7JNFVCTNTA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156984401 (5m54.619076884s) [noise=true , address=145.79.227.14:4838 , piece_id=MG6N5Q7OOXD4OI6CV2VCSGIOOI6VB4GYFATAZ5WZWXFCZN6EFFLQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.157122624 (5m54.619215107s) [noise=true , address=45.91.116.10:4083 , piece_id=VONNFDRLV44LE65ZK7ENEJPNRMA2IPUGGGO3QGYJZLZLWOMNLFRQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.157345388 (5m54.619437871s) [noise=true , address=145.79.227.15:4876 , piece_id=ZHNZB3WQ4QQWR2LO6AQ64ONLIZ2EKCOBYGXFNR5FBY4CZL6QLVMA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.158372935 (5m54.620465408s) [noise=true , address=45.91.116.14:4189 , piece_id=HBFO3FNIMMF5UIDBXTFQQG6WZ4SFGJI3W6CSJ5EBAGN3D7JI6MSA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.159036498 (5m54.621128982s) [noise=true , address=45.91.116.11:4117 , piece_id=QEF7PSZ4ZAV2XM546G6QVKSJVH4JGCSYRAU572ABBMRGKVP23N6A, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.159800133 (5m54.621892616s) [noise=true , address=145.79.227.12:4765 , piece_id=H4PQGKP3XUGR6ARQ3UW3ZVPNCPS5WA5NM6FYJG5TBU3FALNXQYNA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.161637361 (5m54.623729844s) [noise=true , address=145.79.227.17:4970 , piece_id=IW4WTHW5W4SFOYMCXENR3DGYCA2OQJ4OO5RSXPYX6SUIGWKA4LBA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.163221126 (5m54.62531361s) [noise=true , address=77.90.31.11:4553 , piece_id=7XXKVSGNIYW5ZN6G5AV3JWP56QHF2HI4KOXJZ2DNOSGDHVZXDQTQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.163665793 (5m54.625758277s) [noise=true , address=145.79.227.11:4731 , piece_id=XR2GN3LWECQJ4OTSM44QXNW4B6GE7P5C5EFZFKJXD2M4GCMJ54VQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.164366027 (5m54.62645851s) [noise=true , address=145.79.227.12:4767 , piece_id=7JPOEJPRMBWO5MCNTISEDMBSYFACGTPJJUCBDX6EVYUKR3E7MYSA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:10.933255114 (5m56.395347597s) [upload=85]: Next returned nil segment Sep 23 13:37:10.933279932 (5m56.395372415s) [upload=85]: Waiting for error group managing segments... Sep 23 13:37:11.355919055 (5m56.818011548s) [noise=true , address=45.91.116.10:4083 , piece_id=VONNFDRLV44LE65ZK7ENEJPNRMA2IPUGGGO3QGYJZLZLWOMNLFRQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.366419560 (5m56.828512043s) [noise=true , address=45.91.116.4:4938 , piece_id=YXX7C7LQDTPKFEVXO7E6HN6QNM562OWWOXPI232ZHCLSY3WE4BUA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.373438585 (5m56.835531068s) [noise=true , address=45.91.116.11:4117 , piece_id=QEF7PSZ4ZAV2XM546G6QVKSJVH4JGCSYRAU572ABBMRGKVP23N6A, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.388282841 (5m56.850375324s) [noise=true , address=45.91.116.14:4189 , piece_id=HBFO3FNIMMF5UIDBXTFQQG6WZ4SFGJI3W6CSJ5EBAGN3D7JI6MSA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.396367205 (5m56.858459688s) [noise=true , address=77.90.31.11:4553 , piece_id=7XXKVSGNIYW5ZN6G5AV3JWP56QHF2HI4KOXJZ2DNOSGDHVZXDQTQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.050842867 (5m57.51293534s) [noise=true , address=145.79.227.13:4801 , piece_id=KDNEMJNYSDPBUV67TZ6PFMKXECG2TLUFSFX742MK5XRK32C2NHXA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.054995253 (5m57.517087736s) [noise=true , address=145.79.227.15:4876 , piece_id=ZHNZB3WQ4QQWR2LO6AQ64ONLIZ2EKCOBYGXFNR5FBY4CZL6QLVMA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.072213210 (5m57.534305693s) [noise=true , address=145.79.227.17:4970 , piece_id=IW4WTHW5W4SFOYMCXENR3DGYCA2OQJ4OO5RSXPYX6SUIGWKA4LBA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.101168265 (5m57.563260748s) [noise=true , address=145.79.227.6:4259 , piece_id=LPZNOJHDSME5TCQX2HQ544ZANJCJKGWIUXFLIOMNNSZBLYPNJTVA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.106130595 (5m57.568223078s) [noise=true , address=145.79.227.12:4765 , piece_id=H4PQGKP3XUGR6ARQ3UW3ZVPNCPS5WA5NM6FYJG5TBU3FALNXQYNA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.118733793 (5m57.580826277s) [noise=true , address=145.79.227.14:4839 , piece_id=65NEICBQJXMM6N6KBD5KGI5SFTVAV7NJJKBAET6HDGKI5VD325OQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.508815721 (5m57.970908204s) [noise=true , address=145.79.227.11:4731 , piece_id=XR2GN3LWECQJ4OTSM44QXNW4B6GE7P5C5EFZFKJXD2M4GCMJ54VQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:13.008461678 (5m58.470554161s) [noise=true , address=145.79.227.16:4912 , piece_id=YLC5DB6H62T67W5LPTPQIQMWGJAN5ODYJB5PXIVX6X7JNFVCTNTA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:13.008503868 (5m58.470596351s) [seg_pos={0 0}, upload=85]: Segment reached optimal threshold of 13 pieces. Sep 23 13:37:13.008675254 (5m58.470767738s) [noise=true , address=145.79.227.12:4767 , piece_id=7JPOEJPRMBWO5MCNTISEDMBSYFACGTPJJUCBDX6EVYUKR3E7MYSA, seg_pos={0 0}, upload=85]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5): protocol: write tcp 188.166.225.170:41718->145.79.227.12:4767: use of closed network connection; write tcp 188.166.225.170:41718->145.79.227.12:4767: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41718->145.79.227.12:4767: use of closed network connection Sep 23 13:37:13.008689582 (5m58.470782065s) [noise=true , address=145.79.227.12:4767 , piece_id=7JPOEJPRMBWO5MCNTISEDMBSYFACGTPJJUCBDX6EVYUKR3E7MYSA, seg_pos={0 0}, upload=85]: Long tail cancellation due to optimal threshold Sep 23 13:37:13.008703729 (5m58.470796212s) [noise=true , address=145.79.227.5:4221 , piece_id=37XI3ODHTNYYIW3TYSGEZKLL4REO4C7J7W4F3M54QPBS27HU2N7Q, seg_pos={0 0}, upload=85]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR): protocol: expected piece hash; context canceled; EOF Sep 23 13:37:13.008716322 (5m58.470808806s) [noise=true , address=145.79.227.5:4221 , piece_id=37XI3ODHTNYYIW3TYSGEZKLL4REO4C7J7W4F3M54QPBS27HU2N7Q, seg_pos={0 0}, upload=85]: Long tail cancellation due to optimal threshold Sep 23 13:37:13.008800553 (5m58.470893037s) [noise=true , address=145.79.227.14:4838 , piece_id=MG6N5Q7OOXD4OI6CV2VCSGIOOI6VB4GYFATAZ5WZWXFCZN6EFFLQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: expected piece hash; context canceled; EOF Sep 23 13:37:13.008809150 (5m58.470901633s) [noise=true , address=145.79.227.14:4838 , piece_id=MG6N5Q7OOXD4OI6CV2VCSGIOOI6VB4GYFATAZ5WZWXFCZN6EFFLQ, seg_pos={0 0}, upload=85]: Long tail cancellation due to optimal threshold Sep 23 13:37:13.008829819 (5m58.470922302s) [seg_pos={0 0}, upload=85]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 23 13:37:13.008908238 (5m58.471000721s) [upload=85]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 23 13:37:13.008914450 (5m58.471006923s) [upload=85]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 23 13:37:13.352201772 (5m58.814294255s) [upload=85]: Done uploading segments... Sep 23 13:37:13.352227461 (5m58.814319944s) [upload=85]: Upload finished. err: <nil> Duration: 6.595228791s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 7.217338325s FAIL (2025-09-23 13:41:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: context deadline exceeded ======LOGS====== Sep 23 13:37:08.880516737 (5m54.34260922s) [upload=85]: Starting upload Sep 23 13:37:08.880590076 (5m54.342682549s) [upload=85]: Done starting upload Sep 23 13:37:08.880678955 (5m54.342771438s) [upload=85]: Uploading segments... Sep 23 13:37:08.880816417 (5m54.3429089s) [upload=85]: Got next segment. Inline: false Sep 23 13:37:08.880868837 (5m54.34296132s) [upload=85]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 13:37:08.880873937 (5m54.34296642s) [upload=85]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 13:37:09.155479775 (5m54.617572258s) [seg_pos={0 0}, upload=85]: Begin upload segment... Sep 23 13:37:09.155561942 (5m54.617654425s) [seg_pos={0 0}, upload=85]: Done begin upload segment. Sep 23 13:37:09.155609312 (5m54.617701795s) [noise=true , address=145.79.227.5:4221 , piece_id=37XI3ODHTNYYIW3TYSGEZKLL4REO4C7J7W4F3M54QPBS27HU2N7Q, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.155653065 (5m54.617745548s) [noise=true , address=145.79.227.13:4801 , piece_id=KDNEMJNYSDPBUV67TZ6PFMKXECG2TLUFSFX742MK5XRK32C2NHXA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156257627 (5m54.61835011s) [noise=true , address=45.91.116.4:4938 , piece_id=YXX7C7LQDTPKFEVXO7E6HN6QNM562OWWOXPI232ZHCLSY3WE4BUA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156274629 (5m54.618367102s) [noise=true , address=145.79.227.14:4839 , piece_id=65NEICBQJXMM6N6KBD5KGI5SFTVAV7NJJKBAET6HDGKI5VD325OQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156622501 (5m54.618715004s) [noise=true , address=145.79.227.6:4259 , piece_id=LPZNOJHDSME5TCQX2HQ544ZANJCJKGWIUXFLIOMNNSZBLYPNJTVA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156777296 (5m54.618869779s) [noise=true , address=145.79.227.16:4912 , piece_id=YLC5DB6H62T67W5LPTPQIQMWGJAN5ODYJB5PXIVX6X7JNFVCTNTA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156984401 (5m54.619076884s) [noise=true , address=145.79.227.14:4838 , piece_id=MG6N5Q7OOXD4OI6CV2VCSGIOOI6VB4GYFATAZ5WZWXFCZN6EFFLQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.157122624 (5m54.619215107s) [noise=true , address=45.91.116.10:4083 , piece_id=VONNFDRLV44LE65ZK7ENEJPNRMA2IPUGGGO3QGYJZLZLWOMNLFRQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.157345388 (5m54.619437871s) [noise=true , address=145.79.227.15:4876 , piece_id=ZHNZB3WQ4QQWR2LO6AQ64ONLIZ2EKCOBYGXFNR5FBY4CZL6QLVMA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.158372935 (5m54.620465408s) [noise=true , address=45.91.116.14:4189 , piece_id=HBFO3FNIMMF5UIDBXTFQQG6WZ4SFGJI3W6CSJ5EBAGN3D7JI6MSA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.159036498 (5m54.621128982s) [noise=true , address=45.91.116.11:4117 , piece_id=QEF7PSZ4ZAV2XM546G6QVKSJVH4JGCSYRAU572ABBMRGKVP23N6A, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.159800133 (5m54.621892616s) [noise=true , address=145.79.227.12:4765 , piece_id=H4PQGKP3XUGR6ARQ3UW3ZVPNCPS5WA5NM6FYJG5TBU3FALNXQYNA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.161637361 (5m54.623729844s) [noise=true , address=145.79.227.17:4970 , piece_id=IW4WTHW5W4SFOYMCXENR3DGYCA2OQJ4OO5RSXPYX6SUIGWKA4LBA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.163221126 (5m54.62531361s) [noise=true , address=77.90.31.11:4553 , piece_id=7XXKVSGNIYW5ZN6G5AV3JWP56QHF2HI4KOXJZ2DNOSGDHVZXDQTQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.163665793 (5m54.625758277s) [noise=true , address=145.79.227.11:4731 , piece_id=XR2GN3LWECQJ4OTSM44QXNW4B6GE7P5C5EFZFKJXD2M4GCMJ54VQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.164366027 (5m54.62645851s) [noise=true , address=145.79.227.12:4767 , piece_id=7JPOEJPRMBWO5MCNTISEDMBSYFACGTPJJUCBDX6EVYUKR3E7MYSA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:10.933255114 (5m56.395347597s) [upload=85]: Next returned nil segment Sep 23 13:37:10.933279932 (5m56.395372415s) [upload=85]: Waiting for error group managing segments... Sep 23 13:37:11.355919055 (5m56.818011548s) [noise=true , address=45.91.116.10:4083 , piece_id=VONNFDRLV44LE65ZK7ENEJPNRMA2IPUGGGO3QGYJZLZLWOMNLFRQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.366419560 (5m56.828512043s) [noise=true , address=45.91.116.4:4938 , piece_id=YXX7C7LQDTPKFEVXO7E6HN6QNM562OWWOXPI232ZHCLSY3WE4BUA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.373438585 (5m56.835531068s) [noise=true , address=45.91.116.11:4117 , piece_id=QEF7PSZ4ZAV2XM546G6QVKSJVH4JGCSYRAU572ABBMRGKVP23N6A, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.388282841 (5m56.850375324s) [noise=true , address=45.91.116.14:4189 , piece_id=HBFO3FNIMMF5UIDBXTFQQG6WZ4SFGJI3W6CSJ5EBAGN3D7JI6MSA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.396367205 (5m56.858459688s) [noise=true , address=77.90.31.11:4553 , piece_id=7XXKVSGNIYW5ZN6G5AV3JWP56QHF2HI4KOXJZ2DNOSGDHVZXDQTQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.050842867 (5m57.51293534s) [noise=true , address=145.79.227.13:4801 , piece_id=KDNEMJNYSDPBUV67TZ6PFMKXECG2TLUFSFX742MK5XRK32C2NHXA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.054995253 (5m57.517087736s) [noise=true , address=145.79.227.15:4876 , piece_id=ZHNZB3WQ4QQWR2LO6AQ64ONLIZ2EKCOBYGXFNR5FBY4CZL6QLVMA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.072213210 (5m57.534305693s) [noise=true , address=145.79.227.17:4970 , piece_id=IW4WTHW5W4SFOYMCXENR3DGYCA2OQJ4OO5RSXPYX6SUIGWKA4LBA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.101168265 (5m57.563260748s) [noise=true , address=145.79.227.6:4259 , piece_id=LPZNOJHDSME5TCQX2HQ544ZANJCJKGWIUXFLIOMNNSZBLYPNJTVA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.106130595 (5m57.568223078s) [noise=true , address=145.79.227.12:4765 , piece_id=H4PQGKP3XUGR6ARQ3UW3ZVPNCPS5WA5NM6FYJG5TBU3FALNXQYNA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.118733793 (5m57.580826277s) [noise=true , address=145.79.227.14:4839 , piece_id=65NEICBQJXMM6N6KBD5KGI5SFTVAV7NJJKBAET6HDGKI5VD325OQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.508815721 (5m57.970908204s) [noise=true , address=145.79.227.11:4731 , piece_id=XR2GN3LWECQJ4OTSM44QXNW4B6GE7P5C5EFZFKJXD2M4GCMJ54VQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:13.008461678 (5m58.470554161s) [noise=true , address=145.79.227.16:4912 , piece_id=YLC5DB6H62T67W5LPTPQIQMWGJAN5ODYJB5PXIVX6X7JNFVCTNTA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:13.008503868 (5m58.470596351s) [seg_pos={0 0}, upload=85]: Segment reached optimal threshold of 13 pieces. Sep 23 13:37:13.008675254 (5m58.470767738s) [noise=true , address=145.79.227.12:4767 , piece_id=7JPOEJPRMBWO5MCNTISEDMBSYFACGTPJJUCBDX6EVYUKR3E7MYSA, seg_pos={0 0}, upload=85]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5): protocol: write tcp 188.166.225.170:41718->145.79.227.12:4767: use of closed network connection; write tcp 188.166.225.170:41718->145.79.227.12:4767: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41718->145.79.227.12:4767: use of closed network connection Sep 23 13:37:13.008689582 (5m58.470782065s) [noise=true , address=145.79.227.12:4767 , piece_id=7JPOEJPRMBWO5MCNTISEDMBSYFACGTPJJUCBDX6EVYUKR3E7MYSA, seg_pos={0 0}, upload=85]: Long tail cancellation due to optimal threshold Sep 23 13:37:13.008703729 (5m58.470796212s) [noise=true , address=145.79.227.5:4221 , piece_id=37XI3ODHTNYYIW3TYSGEZKLL4REO4C7J7W4F3M54QPBS27HU2N7Q, seg_pos={0 0}, upload=85]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR): protocol: expected piece hash; context canceled; EOF Sep 23 13:37:13.008716322 (5m58.470808806s) [noise=true , address=145.79.227.5:4221 , piece_id=37XI3ODHTNYYIW3TYSGEZKLL4REO4C7J7W4F3M54QPBS27HU2N7Q, seg_pos={0 0}, upload=85]: Long tail cancellation due to optimal threshold Sep 23 13:37:13.008800553 (5m58.470893037s) [noise=true , address=145.79.227.14:4838 , piece_id=MG6N5Q7OOXD4OI6CV2VCSGIOOI6VB4GYFATAZ5WZWXFCZN6EFFLQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: expected piece hash; context canceled; EOF Sep 23 13:37:13.008809150 (5m58.470901633s) [noise=true , address=145.79.227.14:4838 , piece_id=MG6N5Q7OOXD4OI6CV2VCSGIOOI6VB4GYFATAZ5WZWXFCZN6EFFLQ, seg_pos={0 0}, upload=85]: Long tail cancellation due to optimal threshold Sep 23 13:37:13.008829819 (5m58.470922302s) [seg_pos={0 0}, upload=85]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 23 13:37:13.008908238 (5m58.471000721s) [upload=85]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 23 13:37:13.008914450 (5m58.471006923s) [upload=85]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 23 13:37:13.352201772 (5m58.814294255s) [upload=85]: Done uploading segments... Sep 23 13:37:13.352227461 (5m58.814319944s) [upload=85]: Upload finished. err: <nil> Duration: 7.21737154s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 6.414513515s FAIL (2025-09-23 13:40:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: context deadline exceeded ======LOGS====== Sep 23 13:37:08.880516737 (5m54.34260922s) [upload=85]: Starting upload Sep 23 13:37:08.880590076 (5m54.342682549s) [upload=85]: Done starting upload Sep 23 13:37:08.880678955 (5m54.342771438s) [upload=85]: Uploading segments... Sep 23 13:37:08.880816417 (5m54.3429089s) [upload=85]: Got next segment. Inline: false Sep 23 13:37:08.880868837 (5m54.34296132s) [upload=85]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 13:37:08.880873937 (5m54.34296642s) [upload=85]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 13:37:09.155479775 (5m54.617572258s) [seg_pos={0 0}, upload=85]: Begin upload segment... Sep 23 13:37:09.155561942 (5m54.617654425s) [seg_pos={0 0}, upload=85]: Done begin upload segment. Sep 23 13:37:09.155609312 (5m54.617701795s) [noise=true , address=145.79.227.5:4221 , piece_id=37XI3ODHTNYYIW3TYSGEZKLL4REO4C7J7W4F3M54QPBS27HU2N7Q, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.155653065 (5m54.617745548s) [noise=true , address=145.79.227.13:4801 , piece_id=KDNEMJNYSDPBUV67TZ6PFMKXECG2TLUFSFX742MK5XRK32C2NHXA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156257627 (5m54.61835011s) [noise=true , address=45.91.116.4:4938 , piece_id=YXX7C7LQDTPKFEVXO7E6HN6QNM562OWWOXPI232ZHCLSY3WE4BUA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156274629 (5m54.618367102s) [noise=true , address=145.79.227.14:4839 , piece_id=65NEICBQJXMM6N6KBD5KGI5SFTVAV7NJJKBAET6HDGKI5VD325OQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156622501 (5m54.618715004s) [noise=true , address=145.79.227.6:4259 , piece_id=LPZNOJHDSME5TCQX2HQ544ZANJCJKGWIUXFLIOMNNSZBLYPNJTVA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156777296 (5m54.618869779s) [noise=true , address=145.79.227.16:4912 , piece_id=YLC5DB6H62T67W5LPTPQIQMWGJAN5ODYJB5PXIVX6X7JNFVCTNTA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156984401 (5m54.619076884s) [noise=true , address=145.79.227.14:4838 , piece_id=MG6N5Q7OOXD4OI6CV2VCSGIOOI6VB4GYFATAZ5WZWXFCZN6EFFLQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.157122624 (5m54.619215107s) [noise=true , address=45.91.116.10:4083 , piece_id=VONNFDRLV44LE65ZK7ENEJPNRMA2IPUGGGO3QGYJZLZLWOMNLFRQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.157345388 (5m54.619437871s) [noise=true , address=145.79.227.15:4876 , piece_id=ZHNZB3WQ4QQWR2LO6AQ64ONLIZ2EKCOBYGXFNR5FBY4CZL6QLVMA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.158372935 (5m54.620465408s) [noise=true , address=45.91.116.14:4189 , piece_id=HBFO3FNIMMF5UIDBXTFQQG6WZ4SFGJI3W6CSJ5EBAGN3D7JI6MSA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.159036498 (5m54.621128982s) [noise=true , address=45.91.116.11:4117 , piece_id=QEF7PSZ4ZAV2XM546G6QVKSJVH4JGCSYRAU572ABBMRGKVP23N6A, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.159800133 (5m54.621892616s) [noise=true , address=145.79.227.12:4765 , piece_id=H4PQGKP3XUGR6ARQ3UW3ZVPNCPS5WA5NM6FYJG5TBU3FALNXQYNA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.161637361 (5m54.623729844s) [noise=true , address=145.79.227.17:4970 , piece_id=IW4WTHW5W4SFOYMCXENR3DGYCA2OQJ4OO5RSXPYX6SUIGWKA4LBA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.163221126 (5m54.62531361s) [noise=true , address=77.90.31.11:4553 , piece_id=7XXKVSGNIYW5ZN6G5AV3JWP56QHF2HI4KOXJZ2DNOSGDHVZXDQTQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.163665793 (5m54.625758277s) [noise=true , address=145.79.227.11:4731 , piece_id=XR2GN3LWECQJ4OTSM44QXNW4B6GE7P5C5EFZFKJXD2M4GCMJ54VQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.164366027 (5m54.62645851s) [noise=true , address=145.79.227.12:4767 , piece_id=7JPOEJPRMBWO5MCNTISEDMBSYFACGTPJJUCBDX6EVYUKR3E7MYSA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:10.933255114 (5m56.395347597s) [upload=85]: Next returned nil segment Sep 23 13:37:10.933279932 (5m56.395372415s) [upload=85]: Waiting for error group managing segments... Sep 23 13:37:11.355919055 (5m56.818011548s) [noise=true , address=45.91.116.10:4083 , piece_id=VONNFDRLV44LE65ZK7ENEJPNRMA2IPUGGGO3QGYJZLZLWOMNLFRQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.366419560 (5m56.828512043s) [noise=true , address=45.91.116.4:4938 , piece_id=YXX7C7LQDTPKFEVXO7E6HN6QNM562OWWOXPI232ZHCLSY3WE4BUA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.373438585 (5m56.835531068s) [noise=true , address=45.91.116.11:4117 , piece_id=QEF7PSZ4ZAV2XM546G6QVKSJVH4JGCSYRAU572ABBMRGKVP23N6A, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.388282841 (5m56.850375324s) [noise=true , address=45.91.116.14:4189 , piece_id=HBFO3FNIMMF5UIDBXTFQQG6WZ4SFGJI3W6CSJ5EBAGN3D7JI6MSA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.396367205 (5m56.858459688s) [noise=true , address=77.90.31.11:4553 , piece_id=7XXKVSGNIYW5ZN6G5AV3JWP56QHF2HI4KOXJZ2DNOSGDHVZXDQTQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.050842867 (5m57.51293534s) [noise=true , address=145.79.227.13:4801 , piece_id=KDNEMJNYSDPBUV67TZ6PFMKXECG2TLUFSFX742MK5XRK32C2NHXA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.054995253 (5m57.517087736s) [noise=true , address=145.79.227.15:4876 , piece_id=ZHNZB3WQ4QQWR2LO6AQ64ONLIZ2EKCOBYGXFNR5FBY4CZL6QLVMA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.072213210 (5m57.534305693s) [noise=true , address=145.79.227.17:4970 , piece_id=IW4WTHW5W4SFOYMCXENR3DGYCA2OQJ4OO5RSXPYX6SUIGWKA4LBA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.101168265 (5m57.563260748s) [noise=true , address=145.79.227.6:4259 , piece_id=LPZNOJHDSME5TCQX2HQ544ZANJCJKGWIUXFLIOMNNSZBLYPNJTVA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.106130595 (5m57.568223078s) [noise=true , address=145.79.227.12:4765 , piece_id=H4PQGKP3XUGR6ARQ3UW3ZVPNCPS5WA5NM6FYJG5TBU3FALNXQYNA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.118733793 (5m57.580826277s) [noise=true , address=145.79.227.14:4839 , piece_id=65NEICBQJXMM6N6KBD5KGI5SFTVAV7NJJKBAET6HDGKI5VD325OQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.508815721 (5m57.970908204s) [noise=true , address=145.79.227.11:4731 , piece_id=XR2GN3LWECQJ4OTSM44QXNW4B6GE7P5C5EFZFKJXD2M4GCMJ54VQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:13.008461678 (5m58.470554161s) [noise=true , address=145.79.227.16:4912 , piece_id=YLC5DB6H62T67W5LPTPQIQMWGJAN5ODYJB5PXIVX6X7JNFVCTNTA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:13.008503868 (5m58.470596351s) [seg_pos={0 0}, upload=85]: Segment reached optimal threshold of 13 pieces. Sep 23 13:37:13.008675254 (5m58.470767738s) [noise=true , address=145.79.227.12:4767 , piece_id=7JPOEJPRMBWO5MCNTISEDMBSYFACGTPJJUCBDX6EVYUKR3E7MYSA, seg_pos={0 0}, upload=85]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5): protocol: write tcp 188.166.225.170:41718->145.79.227.12:4767: use of closed network connection; write tcp 188.166.225.170:41718->145.79.227.12:4767: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41718->145.79.227.12:4767: use of closed network connection Sep 23 13:37:13.008689582 (5m58.470782065s) [noise=true , address=145.79.227.12:4767 , piece_id=7JPOEJPRMBWO5MCNTISEDMBSYFACGTPJJUCBDX6EVYUKR3E7MYSA, seg_pos={0 0}, upload=85]: Long tail cancellation due to optimal threshold Sep 23 13:37:13.008703729 (5m58.470796212s) [noise=true , address=145.79.227.5:4221 , piece_id=37XI3ODHTNYYIW3TYSGEZKLL4REO4C7J7W4F3M54QPBS27HU2N7Q, seg_pos={0 0}, upload=85]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR): protocol: expected piece hash; context canceled; EOF Sep 23 13:37:13.008716322 (5m58.470808806s) [noise=true , address=145.79.227.5:4221 , piece_id=37XI3ODHTNYYIW3TYSGEZKLL4REO4C7J7W4F3M54QPBS27HU2N7Q, seg_pos={0 0}, upload=85]: Long tail cancellation due to optimal threshold Sep 23 13:37:13.008800553 (5m58.470893037s) [noise=true , address=145.79.227.14:4838 , piece_id=MG6N5Q7OOXD4OI6CV2VCSGIOOI6VB4GYFATAZ5WZWXFCZN6EFFLQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: expected piece hash; context canceled; EOF Sep 23 13:37:13.008809150 (5m58.470901633s) [noise=true , address=145.79.227.14:4838 , piece_id=MG6N5Q7OOXD4OI6CV2VCSGIOOI6VB4GYFATAZ5WZWXFCZN6EFFLQ, seg_pos={0 0}, upload=85]: Long tail cancellation due to optimal threshold Sep 23 13:37:13.008829819 (5m58.470922302s) [seg_pos={0 0}, upload=85]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 23 13:37:13.008908238 (5m58.471000721s) [upload=85]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 23 13:37:13.008914450 (5m58.471006923s) [upload=85]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 23 13:37:13.352201772 (5m58.814294255s) [upload=85]: Done uploading segments... Sep 23 13:37:13.352227461 (5m58.814319944s) [upload=85]: Upload finished. err: <nil> Duration: 6.41454805s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 6.448316384s FAIL (2025-09-23 13:39:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: context deadline exceeded ======LOGS====== Sep 23 13:37:08.880516737 (5m54.34260922s) [upload=85]: Starting upload Sep 23 13:37:08.880590076 (5m54.342682549s) [upload=85]: Done starting upload Sep 23 13:37:08.880678955 (5m54.342771438s) [upload=85]: Uploading segments... Sep 23 13:37:08.880816417 (5m54.3429089s) [upload=85]: Got next segment. Inline: false Sep 23 13:37:08.880868837 (5m54.34296132s) [upload=85]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 13:37:08.880873937 (5m54.34296642s) [upload=85]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 13:37:09.155479775 (5m54.617572258s) [seg_pos={0 0}, upload=85]: Begin upload segment... Sep 23 13:37:09.155561942 (5m54.617654425s) [seg_pos={0 0}, upload=85]: Done begin upload segment. Sep 23 13:37:09.155609312 (5m54.617701795s) [noise=true , address=145.79.227.5:4221 , piece_id=37XI3ODHTNYYIW3TYSGEZKLL4REO4C7J7W4F3M54QPBS27HU2N7Q, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.155653065 (5m54.617745548s) [noise=true , address=145.79.227.13:4801 , piece_id=KDNEMJNYSDPBUV67TZ6PFMKXECG2TLUFSFX742MK5XRK32C2NHXA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156257627 (5m54.61835011s) [noise=true , address=45.91.116.4:4938 , piece_id=YXX7C7LQDTPKFEVXO7E6HN6QNM562OWWOXPI232ZHCLSY3WE4BUA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156274629 (5m54.618367102s) [noise=true , address=145.79.227.14:4839 , piece_id=65NEICBQJXMM6N6KBD5KGI5SFTVAV7NJJKBAET6HDGKI5VD325OQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156622501 (5m54.618715004s) [noise=true , address=145.79.227.6:4259 , piece_id=LPZNOJHDSME5TCQX2HQ544ZANJCJKGWIUXFLIOMNNSZBLYPNJTVA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156777296 (5m54.618869779s) [noise=true , address=145.79.227.16:4912 , piece_id=YLC5DB6H62T67W5LPTPQIQMWGJAN5ODYJB5PXIVX6X7JNFVCTNTA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156984401 (5m54.619076884s) [noise=true , address=145.79.227.14:4838 , piece_id=MG6N5Q7OOXD4OI6CV2VCSGIOOI6VB4GYFATAZ5WZWXFCZN6EFFLQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.157122624 (5m54.619215107s) [noise=true , address=45.91.116.10:4083 , piece_id=VONNFDRLV44LE65ZK7ENEJPNRMA2IPUGGGO3QGYJZLZLWOMNLFRQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.157345388 (5m54.619437871s) [noise=true , address=145.79.227.15:4876 , piece_id=ZHNZB3WQ4QQWR2LO6AQ64ONLIZ2EKCOBYGXFNR5FBY4CZL6QLVMA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.158372935 (5m54.620465408s) [noise=true , address=45.91.116.14:4189 , piece_id=HBFO3FNIMMF5UIDBXTFQQG6WZ4SFGJI3W6CSJ5EBAGN3D7JI6MSA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.159036498 (5m54.621128982s) [noise=true , address=45.91.116.11:4117 , piece_id=QEF7PSZ4ZAV2XM546G6QVKSJVH4JGCSYRAU572ABBMRGKVP23N6A, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.159800133 (5m54.621892616s) [noise=true , address=145.79.227.12:4765 , piece_id=H4PQGKP3XUGR6ARQ3UW3ZVPNCPS5WA5NM6FYJG5TBU3FALNXQYNA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.161637361 (5m54.623729844s) [noise=true , address=145.79.227.17:4970 , piece_id=IW4WTHW5W4SFOYMCXENR3DGYCA2OQJ4OO5RSXPYX6SUIGWKA4LBA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.163221126 (5m54.62531361s) [noise=true , address=77.90.31.11:4553 , piece_id=7XXKVSGNIYW5ZN6G5AV3JWP56QHF2HI4KOXJZ2DNOSGDHVZXDQTQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.163665793 (5m54.625758277s) [noise=true , address=145.79.227.11:4731 , piece_id=XR2GN3LWECQJ4OTSM44QXNW4B6GE7P5C5EFZFKJXD2M4GCMJ54VQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.164366027 (5m54.62645851s) [noise=true , address=145.79.227.12:4767 , piece_id=7JPOEJPRMBWO5MCNTISEDMBSYFACGTPJJUCBDX6EVYUKR3E7MYSA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:10.933255114 (5m56.395347597s) [upload=85]: Next returned nil segment Sep 23 13:37:10.933279932 (5m56.395372415s) [upload=85]: Waiting for error group managing segments... Sep 23 13:37:11.355919055 (5m56.818011548s) [noise=true , address=45.91.116.10:4083 , piece_id=VONNFDRLV44LE65ZK7ENEJPNRMA2IPUGGGO3QGYJZLZLWOMNLFRQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.366419560 (5m56.828512043s) [noise=true , address=45.91.116.4:4938 , piece_id=YXX7C7LQDTPKFEVXO7E6HN6QNM562OWWOXPI232ZHCLSY3WE4BUA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.373438585 (5m56.835531068s) [noise=true , address=45.91.116.11:4117 , piece_id=QEF7PSZ4ZAV2XM546G6QVKSJVH4JGCSYRAU572ABBMRGKVP23N6A, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.388282841 (5m56.850375324s) [noise=true , address=45.91.116.14:4189 , piece_id=HBFO3FNIMMF5UIDBXTFQQG6WZ4SFGJI3W6CSJ5EBAGN3D7JI6MSA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.396367205 (5m56.858459688s) [noise=true , address=77.90.31.11:4553 , piece_id=7XXKVSGNIYW5ZN6G5AV3JWP56QHF2HI4KOXJZ2DNOSGDHVZXDQTQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.050842867 (5m57.51293534s) [noise=true , address=145.79.227.13:4801 , piece_id=KDNEMJNYSDPBUV67TZ6PFMKXECG2TLUFSFX742MK5XRK32C2NHXA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.054995253 (5m57.517087736s) [noise=true , address=145.79.227.15:4876 , piece_id=ZHNZB3WQ4QQWR2LO6AQ64ONLIZ2EKCOBYGXFNR5FBY4CZL6QLVMA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.072213210 (5m57.534305693s) [noise=true , address=145.79.227.17:4970 , piece_id=IW4WTHW5W4SFOYMCXENR3DGYCA2OQJ4OO5RSXPYX6SUIGWKA4LBA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.101168265 (5m57.563260748s) [noise=true , address=145.79.227.6:4259 , piece_id=LPZNOJHDSME5TCQX2HQ544ZANJCJKGWIUXFLIOMNNSZBLYPNJTVA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.106130595 (5m57.568223078s) [noise=true , address=145.79.227.12:4765 , piece_id=H4PQGKP3XUGR6ARQ3UW3ZVPNCPS5WA5NM6FYJG5TBU3FALNXQYNA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.118733793 (5m57.580826277s) [noise=true , address=145.79.227.14:4839 , piece_id=65NEICBQJXMM6N6KBD5KGI5SFTVAV7NJJKBAET6HDGKI5VD325OQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.508815721 (5m57.970908204s) [noise=true , address=145.79.227.11:4731 , piece_id=XR2GN3LWECQJ4OTSM44QXNW4B6GE7P5C5EFZFKJXD2M4GCMJ54VQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:13.008461678 (5m58.470554161s) [noise=true , address=145.79.227.16:4912 , piece_id=YLC5DB6H62T67W5LPTPQIQMWGJAN5ODYJB5PXIVX6X7JNFVCTNTA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:13.008503868 (5m58.470596351s) [seg_pos={0 0}, upload=85]: Segment reached optimal threshold of 13 pieces. Sep 23 13:37:13.008675254 (5m58.470767738s) [noise=true , address=145.79.227.12:4767 , piece_id=7JPOEJPRMBWO5MCNTISEDMBSYFACGTPJJUCBDX6EVYUKR3E7MYSA, seg_pos={0 0}, upload=85]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5): protocol: write tcp 188.166.225.170:41718->145.79.227.12:4767: use of closed network connection; write tcp 188.166.225.170:41718->145.79.227.12:4767: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41718->145.79.227.12:4767: use of closed network connection Sep 23 13:37:13.008689582 (5m58.470782065s) [noise=true , address=145.79.227.12:4767 , piece_id=7JPOEJPRMBWO5MCNTISEDMBSYFACGTPJJUCBDX6EVYUKR3E7MYSA, seg_pos={0 0}, upload=85]: Long tail cancellation due to optimal threshold Sep 23 13:37:13.008703729 (5m58.470796212s) [noise=true , address=145.79.227.5:4221 , piece_id=37XI3ODHTNYYIW3TYSGEZKLL4REO4C7J7W4F3M54QPBS27HU2N7Q, seg_pos={0 0}, upload=85]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR): protocol: expected piece hash; context canceled; EOF Sep 23 13:37:13.008716322 (5m58.470808806s) [noise=true , address=145.79.227.5:4221 , piece_id=37XI3ODHTNYYIW3TYSGEZKLL4REO4C7J7W4F3M54QPBS27HU2N7Q, seg_pos={0 0}, upload=85]: Long tail cancellation due to optimal threshold Sep 23 13:37:13.008800553 (5m58.470893037s) [noise=true , address=145.79.227.14:4838 , piece_id=MG6N5Q7OOXD4OI6CV2VCSGIOOI6VB4GYFATAZ5WZWXFCZN6EFFLQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: expected piece hash; context canceled; EOF Sep 23 13:37:13.008809150 (5m58.470901633s) [noise=true , address=145.79.227.14:4838 , piece_id=MG6N5Q7OOXD4OI6CV2VCSGIOOI6VB4GYFATAZ5WZWXFCZN6EFFLQ, seg_pos={0 0}, upload=85]: Long tail cancellation due to optimal threshold Sep 23 13:37:13.008829819 (5m58.470922302s) [seg_pos={0 0}, upload=85]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 23 13:37:13.008908238 (5m58.471000721s) [upload=85]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 23 13:37:13.008914450 (5m58.471006923s) [upload=85]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 23 13:37:13.352201772 (5m58.814294255s) [upload=85]: Done uploading segments... Sep 23 13:37:13.352227461 (5m58.814319944s) [upload=85]: Upload finished. err: <nil> Duration: 6.448350719s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 8.683090493s FAIL (2025-09-23 13:38:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: context deadline exceeded ======LOGS====== Sep 23 13:37:08.880516737 (5m54.34260922s) [upload=85]: Starting upload Sep 23 13:37:08.880590076 (5m54.342682549s) [upload=85]: Done starting upload Sep 23 13:37:08.880678955 (5m54.342771438s) [upload=85]: Uploading segments... Sep 23 13:37:08.880816417 (5m54.3429089s) [upload=85]: Got next segment. Inline: false Sep 23 13:37:08.880868837 (5m54.34296132s) [upload=85]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 13:37:08.880873937 (5m54.34296642s) [upload=85]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 13:37:09.155479775 (5m54.617572258s) [seg_pos={0 0}, upload=85]: Begin upload segment... Sep 23 13:37:09.155561942 (5m54.617654425s) [seg_pos={0 0}, upload=85]: Done begin upload segment. Sep 23 13:37:09.155609312 (5m54.617701795s) [noise=true , address=145.79.227.5:4221 , piece_id=37XI3ODHTNYYIW3TYSGEZKLL4REO4C7J7W4F3M54QPBS27HU2N7Q, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.155653065 (5m54.617745548s) [noise=true , address=145.79.227.13:4801 , piece_id=KDNEMJNYSDPBUV67TZ6PFMKXECG2TLUFSFX742MK5XRK32C2NHXA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156257627 (5m54.61835011s) [noise=true , address=45.91.116.4:4938 , piece_id=YXX7C7LQDTPKFEVXO7E6HN6QNM562OWWOXPI232ZHCLSY3WE4BUA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156274629 (5m54.618367102s) [noise=true , address=145.79.227.14:4839 , piece_id=65NEICBQJXMM6N6KBD5KGI5SFTVAV7NJJKBAET6HDGKI5VD325OQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156622501 (5m54.618715004s) [noise=true , address=145.79.227.6:4259 , piece_id=LPZNOJHDSME5TCQX2HQ544ZANJCJKGWIUXFLIOMNNSZBLYPNJTVA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156777296 (5m54.618869779s) [noise=true , address=145.79.227.16:4912 , piece_id=YLC5DB6H62T67W5LPTPQIQMWGJAN5ODYJB5PXIVX6X7JNFVCTNTA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156984401 (5m54.619076884s) [noise=true , address=145.79.227.14:4838 , piece_id=MG6N5Q7OOXD4OI6CV2VCSGIOOI6VB4GYFATAZ5WZWXFCZN6EFFLQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.157122624 (5m54.619215107s) [noise=true , address=45.91.116.10:4083 , piece_id=VONNFDRLV44LE65ZK7ENEJPNRMA2IPUGGGO3QGYJZLZLWOMNLFRQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.157345388 (5m54.619437871s) [noise=true , address=145.79.227.15:4876 , piece_id=ZHNZB3WQ4QQWR2LO6AQ64ONLIZ2EKCOBYGXFNR5FBY4CZL6QLVMA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.158372935 (5m54.620465408s) [noise=true , address=45.91.116.14:4189 , piece_id=HBFO3FNIMMF5UIDBXTFQQG6WZ4SFGJI3W6CSJ5EBAGN3D7JI6MSA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.159036498 (5m54.621128982s) [noise=true , address=45.91.116.11:4117 , piece_id=QEF7PSZ4ZAV2XM546G6QVKSJVH4JGCSYRAU572ABBMRGKVP23N6A, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.159800133 (5m54.621892616s) [noise=true , address=145.79.227.12:4765 , piece_id=H4PQGKP3XUGR6ARQ3UW3ZVPNCPS5WA5NM6FYJG5TBU3FALNXQYNA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.161637361 (5m54.623729844s) [noise=true , address=145.79.227.17:4970 , piece_id=IW4WTHW5W4SFOYMCXENR3DGYCA2OQJ4OO5RSXPYX6SUIGWKA4LBA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.163221126 (5m54.62531361s) [noise=true , address=77.90.31.11:4553 , piece_id=7XXKVSGNIYW5ZN6G5AV3JWP56QHF2HI4KOXJZ2DNOSGDHVZXDQTQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.163665793 (5m54.625758277s) [noise=true , address=145.79.227.11:4731 , piece_id=XR2GN3LWECQJ4OTSM44QXNW4B6GE7P5C5EFZFKJXD2M4GCMJ54VQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.164366027 (5m54.62645851s) [noise=true , address=145.79.227.12:4767 , piece_id=7JPOEJPRMBWO5MCNTISEDMBSYFACGTPJJUCBDX6EVYUKR3E7MYSA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:10.933255114 (5m56.395347597s) [upload=85]: Next returned nil segment Sep 23 13:37:10.933279932 (5m56.395372415s) [upload=85]: Waiting for error group managing segments... Sep 23 13:37:11.355919055 (5m56.818011548s) [noise=true , address=45.91.116.10:4083 , piece_id=VONNFDRLV44LE65ZK7ENEJPNRMA2IPUGGGO3QGYJZLZLWOMNLFRQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.366419560 (5m56.828512043s) [noise=true , address=45.91.116.4:4938 , piece_id=YXX7C7LQDTPKFEVXO7E6HN6QNM562OWWOXPI232ZHCLSY3WE4BUA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.373438585 (5m56.835531068s) [noise=true , address=45.91.116.11:4117 , piece_id=QEF7PSZ4ZAV2XM546G6QVKSJVH4JGCSYRAU572ABBMRGKVP23N6A, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.388282841 (5m56.850375324s) [noise=true , address=45.91.116.14:4189 , piece_id=HBFO3FNIMMF5UIDBXTFQQG6WZ4SFGJI3W6CSJ5EBAGN3D7JI6MSA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.396367205 (5m56.858459688s) [noise=true , address=77.90.31.11:4553 , piece_id=7XXKVSGNIYW5ZN6G5AV3JWP56QHF2HI4KOXJZ2DNOSGDHVZXDQTQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.050842867 (5m57.51293534s) [noise=true , address=145.79.227.13:4801 , piece_id=KDNEMJNYSDPBUV67TZ6PFMKXECG2TLUFSFX742MK5XRK32C2NHXA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.054995253 (5m57.517087736s) [noise=true , address=145.79.227.15:4876 , piece_id=ZHNZB3WQ4QQWR2LO6AQ64ONLIZ2EKCOBYGXFNR5FBY4CZL6QLVMA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.072213210 (5m57.534305693s) [noise=true , address=145.79.227.17:4970 , piece_id=IW4WTHW5W4SFOYMCXENR3DGYCA2OQJ4OO5RSXPYX6SUIGWKA4LBA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.101168265 (5m57.563260748s) [noise=true , address=145.79.227.6:4259 , piece_id=LPZNOJHDSME5TCQX2HQ544ZANJCJKGWIUXFLIOMNNSZBLYPNJTVA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.106130595 (5m57.568223078s) [noise=true , address=145.79.227.12:4765 , piece_id=H4PQGKP3XUGR6ARQ3UW3ZVPNCPS5WA5NM6FYJG5TBU3FALNXQYNA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.118733793 (5m57.580826277s) [noise=true , address=145.79.227.14:4839 , piece_id=65NEICBQJXMM6N6KBD5KGI5SFTVAV7NJJKBAET6HDGKI5VD325OQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.508815721 (5m57.970908204s) [noise=true , address=145.79.227.11:4731 , piece_id=XR2GN3LWECQJ4OTSM44QXNW4B6GE7P5C5EFZFKJXD2M4GCMJ54VQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:13.008461678 (5m58.470554161s) [noise=true , address=145.79.227.16:4912 , piece_id=YLC5DB6H62T67W5LPTPQIQMWGJAN5ODYJB5PXIVX6X7JNFVCTNTA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:13.008503868 (5m58.470596351s) [seg_pos={0 0}, upload=85]: Segment reached optimal threshold of 13 pieces. Sep 23 13:37:13.008675254 (5m58.470767738s) [noise=true , address=145.79.227.12:4767 , piece_id=7JPOEJPRMBWO5MCNTISEDMBSYFACGTPJJUCBDX6EVYUKR3E7MYSA, seg_pos={0 0}, upload=85]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5): protocol: write tcp 188.166.225.170:41718->145.79.227.12:4767: use of closed network connection; write tcp 188.166.225.170:41718->145.79.227.12:4767: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41718->145.79.227.12:4767: use of closed network connection Sep 23 13:37:13.008689582 (5m58.470782065s) [noise=true , address=145.79.227.12:4767 , piece_id=7JPOEJPRMBWO5MCNTISEDMBSYFACGTPJJUCBDX6EVYUKR3E7MYSA, seg_pos={0 0}, upload=85]: Long tail cancellation due to optimal threshold Sep 23 13:37:13.008703729 (5m58.470796212s) [noise=true , address=145.79.227.5:4221 , piece_id=37XI3ODHTNYYIW3TYSGEZKLL4REO4C7J7W4F3M54QPBS27HU2N7Q, seg_pos={0 0}, upload=85]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR): protocol: expected piece hash; context canceled; EOF Sep 23 13:37:13.008716322 (5m58.470808806s) [noise=true , address=145.79.227.5:4221 , piece_id=37XI3ODHTNYYIW3TYSGEZKLL4REO4C7J7W4F3M54QPBS27HU2N7Q, seg_pos={0 0}, upload=85]: Long tail cancellation due to optimal threshold Sep 23 13:37:13.008800553 (5m58.470893037s) [noise=true , address=145.79.227.14:4838 , piece_id=MG6N5Q7OOXD4OI6CV2VCSGIOOI6VB4GYFATAZ5WZWXFCZN6EFFLQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: expected piece hash; context canceled; EOF Sep 23 13:37:13.008809150 (5m58.470901633s) [noise=true , address=145.79.227.14:4838 , piece_id=MG6N5Q7OOXD4OI6CV2VCSGIOOI6VB4GYFATAZ5WZWXFCZN6EFFLQ, seg_pos={0 0}, upload=85]: Long tail cancellation due to optimal threshold Sep 23 13:37:13.008829819 (5m58.470922302s) [seg_pos={0 0}, upload=85]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 23 13:37:13.008908238 (5m58.471000721s) [upload=85]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 23 13:37:13.008914450 (5m58.471006923s) [upload=85]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 23 13:37:13.352201772 (5m58.814294255s) [upload=85]: Done uploading segments... Sep 23 13:37:13.352227461 (5m58.814319944s) [upload=85]: Upload finished. err: <nil> Duration: 8.683121253s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 8.893410709s FAIL (2025-09-23 13:38:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: context deadline exceeded ======LOGS====== Sep 23 13:37:08.880516737 (5m54.34260922s) [upload=85]: Starting upload Sep 23 13:37:08.880590076 (5m54.342682549s) [upload=85]: Done starting upload Sep 23 13:37:08.880678955 (5m54.342771438s) [upload=85]: Uploading segments... Sep 23 13:37:08.880816417 (5m54.3429089s) [upload=85]: Got next segment. Inline: false Sep 23 13:37:08.880868837 (5m54.34296132s) [upload=85]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 13:37:08.880873937 (5m54.34296642s) [upload=85]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 13:37:09.155479775 (5m54.617572258s) [seg_pos={0 0}, upload=85]: Begin upload segment... Sep 23 13:37:09.155561942 (5m54.617654425s) [seg_pos={0 0}, upload=85]: Done begin upload segment. Sep 23 13:37:09.155609312 (5m54.617701795s) [noise=true , address=145.79.227.5:4221 , piece_id=37XI3ODHTNYYIW3TYSGEZKLL4REO4C7J7W4F3M54QPBS27HU2N7Q, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.155653065 (5m54.617745548s) [noise=true , address=145.79.227.13:4801 , piece_id=KDNEMJNYSDPBUV67TZ6PFMKXECG2TLUFSFX742MK5XRK32C2NHXA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156257627 (5m54.61835011s) [noise=true , address=45.91.116.4:4938 , piece_id=YXX7C7LQDTPKFEVXO7E6HN6QNM562OWWOXPI232ZHCLSY3WE4BUA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156274629 (5m54.618367102s) [noise=true , address=145.79.227.14:4839 , piece_id=65NEICBQJXMM6N6KBD5KGI5SFTVAV7NJJKBAET6HDGKI5VD325OQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156622501 (5m54.618715004s) [noise=true , address=145.79.227.6:4259 , piece_id=LPZNOJHDSME5TCQX2HQ544ZANJCJKGWIUXFLIOMNNSZBLYPNJTVA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156777296 (5m54.618869779s) [noise=true , address=145.79.227.16:4912 , piece_id=YLC5DB6H62T67W5LPTPQIQMWGJAN5ODYJB5PXIVX6X7JNFVCTNTA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156984401 (5m54.619076884s) [noise=true , address=145.79.227.14:4838 , piece_id=MG6N5Q7OOXD4OI6CV2VCSGIOOI6VB4GYFATAZ5WZWXFCZN6EFFLQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.157122624 (5m54.619215107s) [noise=true , address=45.91.116.10:4083 , piece_id=VONNFDRLV44LE65ZK7ENEJPNRMA2IPUGGGO3QGYJZLZLWOMNLFRQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.157345388 (5m54.619437871s) [noise=true , address=145.79.227.15:4876 , piece_id=ZHNZB3WQ4QQWR2LO6AQ64ONLIZ2EKCOBYGXFNR5FBY4CZL6QLVMA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.158372935 (5m54.620465408s) [noise=true , address=45.91.116.14:4189 , piece_id=HBFO3FNIMMF5UIDBXTFQQG6WZ4SFGJI3W6CSJ5EBAGN3D7JI6MSA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.159036498 (5m54.621128982s) [noise=true , address=45.91.116.11:4117 , piece_id=QEF7PSZ4ZAV2XM546G6QVKSJVH4JGCSYRAU572ABBMRGKVP23N6A, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.159800133 (5m54.621892616s) [noise=true , address=145.79.227.12:4765 , piece_id=H4PQGKP3XUGR6ARQ3UW3ZVPNCPS5WA5NM6FYJG5TBU3FALNXQYNA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.161637361 (5m54.623729844s) [noise=true , address=145.79.227.17:4970 , piece_id=IW4WTHW5W4SFOYMCXENR3DGYCA2OQJ4OO5RSXPYX6SUIGWKA4LBA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.163221126 (5m54.62531361s) [noise=true , address=77.90.31.11:4553 , piece_id=7XXKVSGNIYW5ZN6G5AV3JWP56QHF2HI4KOXJZ2DNOSGDHVZXDQTQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.163665793 (5m54.625758277s) [noise=true , address=145.79.227.11:4731 , piece_id=XR2GN3LWECQJ4OTSM44QXNW4B6GE7P5C5EFZFKJXD2M4GCMJ54VQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.164366027 (5m54.62645851s) [noise=true , address=145.79.227.12:4767 , piece_id=7JPOEJPRMBWO5MCNTISEDMBSYFACGTPJJUCBDX6EVYUKR3E7MYSA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:10.933255114 (5m56.395347597s) [upload=85]: Next returned nil segment Sep 23 13:37:10.933279932 (5m56.395372415s) [upload=85]: Waiting for error group managing segments... Sep 23 13:37:11.355919055 (5m56.818011548s) [noise=true , address=45.91.116.10:4083 , piece_id=VONNFDRLV44LE65ZK7ENEJPNRMA2IPUGGGO3QGYJZLZLWOMNLFRQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.366419560 (5m56.828512043s) [noise=true , address=45.91.116.4:4938 , piece_id=YXX7C7LQDTPKFEVXO7E6HN6QNM562OWWOXPI232ZHCLSY3WE4BUA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.373438585 (5m56.835531068s) [noise=true , address=45.91.116.11:4117 , piece_id=QEF7PSZ4ZAV2XM546G6QVKSJVH4JGCSYRAU572ABBMRGKVP23N6A, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.388282841 (5m56.850375324s) [noise=true , address=45.91.116.14:4189 , piece_id=HBFO3FNIMMF5UIDBXTFQQG6WZ4SFGJI3W6CSJ5EBAGN3D7JI6MSA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.396367205 (5m56.858459688s) [noise=true , address=77.90.31.11:4553 , piece_id=7XXKVSGNIYW5ZN6G5AV3JWP56QHF2HI4KOXJZ2DNOSGDHVZXDQTQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.050842867 (5m57.51293534s) [noise=true , address=145.79.227.13:4801 , piece_id=KDNEMJNYSDPBUV67TZ6PFMKXECG2TLUFSFX742MK5XRK32C2NHXA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.054995253 (5m57.517087736s) [noise=true , address=145.79.227.15:4876 , piece_id=ZHNZB3WQ4QQWR2LO6AQ64ONLIZ2EKCOBYGXFNR5FBY4CZL6QLVMA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.072213210 (5m57.534305693s) [noise=true , address=145.79.227.17:4970 , piece_id=IW4WTHW5W4SFOYMCXENR3DGYCA2OQJ4OO5RSXPYX6SUIGWKA4LBA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.101168265 (5m57.563260748s) [noise=true , address=145.79.227.6:4259 , piece_id=LPZNOJHDSME5TCQX2HQ544ZANJCJKGWIUXFLIOMNNSZBLYPNJTVA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.106130595 (5m57.568223078s) [noise=true , address=145.79.227.12:4765 , piece_id=H4PQGKP3XUGR6ARQ3UW3ZVPNCPS5WA5NM6FYJG5TBU3FALNXQYNA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.118733793 (5m57.580826277s) [noise=true , address=145.79.227.14:4839 , piece_id=65NEICBQJXMM6N6KBD5KGI5SFTVAV7NJJKBAET6HDGKI5VD325OQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.508815721 (5m57.970908204s) [noise=true , address=145.79.227.11:4731 , piece_id=XR2GN3LWECQJ4OTSM44QXNW4B6GE7P5C5EFZFKJXD2M4GCMJ54VQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:13.008461678 (5m58.470554161s) [noise=true , address=145.79.227.16:4912 , piece_id=YLC5DB6H62T67W5LPTPQIQMWGJAN5ODYJB5PXIVX6X7JNFVCTNTA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:13.008503868 (5m58.470596351s) [seg_pos={0 0}, upload=85]: Segment reached optimal threshold of 13 pieces. Sep 23 13:37:13.008675254 (5m58.470767738s) [noise=true , address=145.79.227.12:4767 , piece_id=7JPOEJPRMBWO5MCNTISEDMBSYFACGTPJJUCBDX6EVYUKR3E7MYSA, seg_pos={0 0}, upload=85]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5): protocol: write tcp 188.166.225.170:41718->145.79.227.12:4767: use of closed network connection; write tcp 188.166.225.170:41718->145.79.227.12:4767: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41718->145.79.227.12:4767: use of closed network connection Sep 23 13:37:13.008689582 (5m58.470782065s) [noise=true , address=145.79.227.12:4767 , piece_id=7JPOEJPRMBWO5MCNTISEDMBSYFACGTPJJUCBDX6EVYUKR3E7MYSA, seg_pos={0 0}, upload=85]: Long tail cancellation due to optimal threshold Sep 23 13:37:13.008703729 (5m58.470796212s) [noise=true , address=145.79.227.5:4221 , piece_id=37XI3ODHTNYYIW3TYSGEZKLL4REO4C7J7W4F3M54QPBS27HU2N7Q, seg_pos={0 0}, upload=85]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR): protocol: expected piece hash; context canceled; EOF Sep 23 13:37:13.008716322 (5m58.470808806s) [noise=true , address=145.79.227.5:4221 , piece_id=37XI3ODHTNYYIW3TYSGEZKLL4REO4C7J7W4F3M54QPBS27HU2N7Q, seg_pos={0 0}, upload=85]: Long tail cancellation due to optimal threshold Sep 23 13:37:13.008800553 (5m58.470893037s) [noise=true , address=145.79.227.14:4838 , piece_id=MG6N5Q7OOXD4OI6CV2VCSGIOOI6VB4GYFATAZ5WZWXFCZN6EFFLQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: expected piece hash; context canceled; EOF Sep 23 13:37:13.008809150 (5m58.470901633s) [noise=true , address=145.79.227.14:4838 , piece_id=MG6N5Q7OOXD4OI6CV2VCSGIOOI6VB4GYFATAZ5WZWXFCZN6EFFLQ, seg_pos={0 0}, upload=85]: Long tail cancellation due to optimal threshold Sep 23 13:37:13.008829819 (5m58.470922302s) [seg_pos={0 0}, upload=85]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 23 13:37:13.008908238 (5m58.471000721s) [upload=85]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 23 13:37:13.008914450 (5m58.471006923s) [upload=85]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 23 13:37:13.352201772 (5m58.814294255s) [upload=85]: Done uploading segments... Sep 23 13:37:13.352227461 (5m58.814319944s) [upload=85]: Upload finished. err: <nil> Duration: 8.893442629s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.005647084s FAIL (2025-09-23 13:37:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: context deadline exceeded ======LOGS====== Sep 23 13:37:08.880516737 (5m54.34260922s) [upload=85]: Starting upload Sep 23 13:37:08.880590076 (5m54.342682549s) [upload=85]: Done starting upload Sep 23 13:37:08.880678955 (5m54.342771438s) [upload=85]: Uploading segments... Sep 23 13:37:08.880816417 (5m54.3429089s) [upload=85]: Got next segment. Inline: false Sep 23 13:37:08.880868837 (5m54.34296132s) [upload=85]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 13:37:08.880873937 (5m54.34296642s) [upload=85]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 13:37:09.155479775 (5m54.617572258s) [seg_pos={0 0}, upload=85]: Begin upload segment... Sep 23 13:37:09.155561942 (5m54.617654425s) [seg_pos={0 0}, upload=85]: Done begin upload segment. Sep 23 13:37:09.155609312 (5m54.617701795s) [noise=true , address=145.79.227.5:4221 , piece_id=37XI3ODHTNYYIW3TYSGEZKLL4REO4C7J7W4F3M54QPBS27HU2N7Q, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.155653065 (5m54.617745548s) [noise=true , address=145.79.227.13:4801 , piece_id=KDNEMJNYSDPBUV67TZ6PFMKXECG2TLUFSFX742MK5XRK32C2NHXA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156257627 (5m54.61835011s) [noise=true , address=45.91.116.4:4938 , piece_id=YXX7C7LQDTPKFEVXO7E6HN6QNM562OWWOXPI232ZHCLSY3WE4BUA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156274629 (5m54.618367102s) [noise=true , address=145.79.227.14:4839 , piece_id=65NEICBQJXMM6N6KBD5KGI5SFTVAV7NJJKBAET6HDGKI5VD325OQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156622501 (5m54.618715004s) [noise=true , address=145.79.227.6:4259 , piece_id=LPZNOJHDSME5TCQX2HQ544ZANJCJKGWIUXFLIOMNNSZBLYPNJTVA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156777296 (5m54.618869779s) [noise=true , address=145.79.227.16:4912 , piece_id=YLC5DB6H62T67W5LPTPQIQMWGJAN5ODYJB5PXIVX6X7JNFVCTNTA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.156984401 (5m54.619076884s) [noise=true , address=145.79.227.14:4838 , piece_id=MG6N5Q7OOXD4OI6CV2VCSGIOOI6VB4GYFATAZ5WZWXFCZN6EFFLQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.157122624 (5m54.619215107s) [noise=true , address=45.91.116.10:4083 , piece_id=VONNFDRLV44LE65ZK7ENEJPNRMA2IPUGGGO3QGYJZLZLWOMNLFRQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.157345388 (5m54.619437871s) [noise=true , address=145.79.227.15:4876 , piece_id=ZHNZB3WQ4QQWR2LO6AQ64ONLIZ2EKCOBYGXFNR5FBY4CZL6QLVMA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.158372935 (5m54.620465408s) [noise=true , address=45.91.116.14:4189 , piece_id=HBFO3FNIMMF5UIDBXTFQQG6WZ4SFGJI3W6CSJ5EBAGN3D7JI6MSA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.159036498 (5m54.621128982s) [noise=true , address=45.91.116.11:4117 , piece_id=QEF7PSZ4ZAV2XM546G6QVKSJVH4JGCSYRAU572ABBMRGKVP23N6A, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.159800133 (5m54.621892616s) [noise=true , address=145.79.227.12:4765 , piece_id=H4PQGKP3XUGR6ARQ3UW3ZVPNCPS5WA5NM6FYJG5TBU3FALNXQYNA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.161637361 (5m54.623729844s) [noise=true , address=145.79.227.17:4970 , piece_id=IW4WTHW5W4SFOYMCXENR3DGYCA2OQJ4OO5RSXPYX6SUIGWKA4LBA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.163221126 (5m54.62531361s) [noise=true , address=77.90.31.11:4553 , piece_id=7XXKVSGNIYW5ZN6G5AV3JWP56QHF2HI4KOXJZ2DNOSGDHVZXDQTQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.163665793 (5m54.625758277s) [noise=true , address=145.79.227.11:4731 , piece_id=XR2GN3LWECQJ4OTSM44QXNW4B6GE7P5C5EFZFKJXD2M4GCMJ54VQ, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:09.164366027 (5m54.62645851s) [noise=true , address=145.79.227.12:4767 , piece_id=7JPOEJPRMBWO5MCNTISEDMBSYFACGTPJJUCBDX6EVYUKR3E7MYSA, seg_pos={0 0}, upload=85]: Uploading piece... Sep 23 13:37:10.933255114 (5m56.395347597s) [upload=85]: Next returned nil segment Sep 23 13:37:10.933279932 (5m56.395372415s) [upload=85]: Waiting for error group managing segments... Sep 23 13:37:11.355919055 (5m56.818011548s) [noise=true , address=45.91.116.10:4083 , piece_id=VONNFDRLV44LE65ZK7ENEJPNRMA2IPUGGGO3QGYJZLZLWOMNLFRQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.366419560 (5m56.828512043s) [noise=true , address=45.91.116.4:4938 , piece_id=YXX7C7LQDTPKFEVXO7E6HN6QNM562OWWOXPI232ZHCLSY3WE4BUA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.373438585 (5m56.835531068s) [noise=true , address=45.91.116.11:4117 , piece_id=QEF7PSZ4ZAV2XM546G6QVKSJVH4JGCSYRAU572ABBMRGKVP23N6A, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.388282841 (5m56.850375324s) [noise=true , address=45.91.116.14:4189 , piece_id=HBFO3FNIMMF5UIDBXTFQQG6WZ4SFGJI3W6CSJ5EBAGN3D7JI6MSA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:11.396367205 (5m56.858459688s) [noise=true , address=77.90.31.11:4553 , piece_id=7XXKVSGNIYW5ZN6G5AV3JWP56QHF2HI4KOXJZ2DNOSGDHVZXDQTQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.050842867 (5m57.51293534s) [noise=true , address=145.79.227.13:4801 , piece_id=KDNEMJNYSDPBUV67TZ6PFMKXECG2TLUFSFX742MK5XRK32C2NHXA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.054995253 (5m57.517087736s) [noise=true , address=145.79.227.15:4876 , piece_id=ZHNZB3WQ4QQWR2LO6AQ64ONLIZ2EKCOBYGXFNR5FBY4CZL6QLVMA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.072213210 (5m57.534305693s) [noise=true , address=145.79.227.17:4970 , piece_id=IW4WTHW5W4SFOYMCXENR3DGYCA2OQJ4OO5RSXPYX6SUIGWKA4LBA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.101168265 (5m57.563260748s) [noise=true , address=145.79.227.6:4259 , piece_id=LPZNOJHDSME5TCQX2HQ544ZANJCJKGWIUXFLIOMNNSZBLYPNJTVA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.106130595 (5m57.568223078s) [noise=true , address=145.79.227.12:4765 , piece_id=H4PQGKP3XUGR6ARQ3UW3ZVPNCPS5WA5NM6FYJG5TBU3FALNXQYNA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.118733793 (5m57.580826277s) [noise=true , address=145.79.227.14:4839 , piece_id=65NEICBQJXMM6N6KBD5KGI5SFTVAV7NJJKBAET6HDGKI5VD325OQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:12.508815721 (5m57.970908204s) [noise=true , address=145.79.227.11:4731 , piece_id=XR2GN3LWECQJ4OTSM44QXNW4B6GE7P5C5EFZFKJXD2M4GCMJ54VQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:13.008461678 (5m58.470554161s) [noise=true , address=145.79.227.16:4912 , piece_id=YLC5DB6H62T67W5LPTPQIQMWGJAN5ODYJB5PXIVX6X7JNFVCTNTA, seg_pos={0 0}, upload=85]: Done uploading piece. err: <nil> Sep 23 13:37:13.008503868 (5m58.470596351s) [seg_pos={0 0}, upload=85]: Segment reached optimal threshold of 13 pieces. Sep 23 13:37:13.008675254 (5m58.470767738s) [noise=true , address=145.79.227.12:4767 , piece_id=7JPOEJPRMBWO5MCNTISEDMBSYFACGTPJJUCBDX6EVYUKR3E7MYSA, seg_pos={0 0}, upload=85]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5): protocol: write tcp 188.166.225.170:41718->145.79.227.12:4767: use of closed network connection; write tcp 188.166.225.170:41718->145.79.227.12:4767: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41718->145.79.227.12:4767: use of closed network connection Sep 23 13:37:13.008689582 (5m58.470782065s) [noise=true , address=145.79.227.12:4767 , piece_id=7JPOEJPRMBWO5MCNTISEDMBSYFACGTPJJUCBDX6EVYUKR3E7MYSA, seg_pos={0 0}, upload=85]: Long tail cancellation due to optimal threshold Sep 23 13:37:13.008703729 (5m58.470796212s) [noise=true , address=145.79.227.5:4221 , piece_id=37XI3ODHTNYYIW3TYSGEZKLL4REO4C7J7W4F3M54QPBS27HU2N7Q, seg_pos={0 0}, upload=85]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR): protocol: expected piece hash; context canceled; EOF Sep 23 13:37:13.008716322 (5m58.470808806s) [noise=true , address=145.79.227.5:4221 , piece_id=37XI3ODHTNYYIW3TYSGEZKLL4REO4C7J7W4F3M54QPBS27HU2N7Q, seg_pos={0 0}, upload=85]: Long tail cancellation due to optimal threshold Sep 23 13:37:13.008800553 (5m58.470893037s) [noise=true , address=145.79.227.14:4838 , piece_id=MG6N5Q7OOXD4OI6CV2VCSGIOOI6VB4GYFATAZ5WZWXFCZN6EFFLQ, seg_pos={0 0}, upload=85]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: expected piece hash; context canceled; EOF Sep 23 13:37:13.008809150 (5m58.470901633s) [noise=true , address=145.79.227.14:4838 , piece_id=MG6N5Q7OOXD4OI6CV2VCSGIOOI6VB4GYFATAZ5WZWXFCZN6EFFLQ, seg_pos={0 0}, upload=85]: Long tail cancellation due to optimal threshold Sep 23 13:37:13.008829819 (5m58.470922302s) [seg_pos={0 0}, upload=85]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 23 13:37:13.008908238 (5m58.471000721s) [upload=85]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 23 13:37:13.008914450 (5m58.471006923s) [upload=85]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 23 13:37:13.352201772 (5m58.814294255s) [upload=85]: Done uploading segments... Sep 23 13:37:13.352227461 (5m58.814319944s) [upload=85]: Upload finished. err: <nil> Duration: 30.005675017s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 6.329223159s FAIL (2025-09-23 13:30:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 23 12:45:46.780664005 (407h17m22.820237135s) [upload=358133]: Starting upload Sep 23 12:45:46.780766030 (407h17m22.82033915s) [upload=358133]: Done starting upload Sep 23 12:45:46.780906196 (407h17m22.820479326s) [upload=358133]: Uploading segments... Sep 23 12:45:46.781049119 (407h17m22.820622249s) [upload=358133]: Got next segment. Inline: false Sep 23 12:45:46.781085078 (407h17m22.820658208s) [upload=358133]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 12:45:46.781091199 (407h17m22.820664329s) [upload=358133]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 12:45:47.134779542 (407h17m23.174352682s) [seg_pos={0 0}, upload=358133]: Begin upload segment... Sep 23 12:45:47.134833926 (407h17m23.174407056s) [seg_pos={0 0}, upload=358133]: Done begin upload segment. Sep 23 12:45:47.134868011 (407h17m23.174441141s) [noise=true , address=145.79.227.18:4007 , piece_id=OMVCGFCUBO56VZNIPNPWIGVE2VDGIE74XCMBJCJGCQ45NGKTLVQQ, seg_pos={0 0}, upload=358133]: Uploading piece... Sep 23 12:45:47.135481949 (407h17m23.175055079s) [noise=true , address=45.91.116.11:4118 , piece_id=NFPQYRN6XQLLO5XABYVAUDGP7236SGODYEPFVCL5CTEAE5EGK5BA, seg_pos={0 0}, upload=358133]: Uploading piece... Sep 23 12:45:47.136025715 (407h17m23.175598845s) [noise=true , address=45.91.116.11:4119 , piece_id=FSMWXG3WWD4UA4QWOE66MUD2CAM7CJWAHOE476BPM5ZLX24WGG7A, seg_pos={0 0}, upload=358133]: Uploading piece... Sep 23 12:45:47.136069518 (407h17m23.175642648s) [noise=true , address=45.91.116.9:4262 , piece_id=ST3YZKNCSKBNBTYMYKQSXUD77ICDKBZQAYE4HFMWRAMV6HXZD22Q, seg_pos={0 0}, upload=358133]: Uploading piece... Sep 23 12:45:47.136416288 (407h17m23.175989418s) [noise=true , address=77.90.31.13:4626 , piece_id=N6VTJT2T74RVKVDQLXXX2P74FDW2ILEJWNI2HRK4QNIZ7AUYNUEA, seg_pos={0 0}, upload=358133]: Uploading piece... Sep 23 12:45:47.136624615 (407h17m23.176197735s) [noise=true , address=145.79.227.6:4259 , piece_id=PBQPI25TRRNVIRRV6P4GFXUUNHABPZMCUAYCMMMCLB6SFEBSICYA, seg_pos={0 0}, upload=358133]: Uploading piece... Sep 23 12:45:47.136803185 (407h17m23.176376315s) [noise=true , address=145.79.227.7:4295 , piece_id=JYVLPPEMVFUFXAMYNFEIMMWQJLBXL2RRHUVXY55LZULE2JM3UEFA, seg_pos={0 0}, upload=358133]: Uploading piece... Sep 23 12:45:47.136992296 (407h17m23.176565416s) [noise=true , address=145.79.227.13:4804 , piece_id=V5FFP33A4Z2SZUREHBB35I77AKDSHYL6WDEAK25CE76V3ZCGZ6YQ, seg_pos={0 0}, upload=358133]: Uploading piece... Sep 23 12:45:47.137186525 (407h17m23.176759655s) [noise=true , address=145.79.227.17:4969 , piece_id=MS2QT2XAW7CCC5YXUQ5CFMCH7LVK5C3CEOZN44HCZ5YWSIXMQO3A, seg_pos={0 0}, upload=358133]: Uploading piece... Sep 23 12:45:47.137369544 (407h17m23.176942674s) [noise=true , address=45.91.116.14:4191 , piece_id=JFV37O2MDPIZ4DEHSPJOCDVROQ2S2A4XEOMUM3CTHHK2QBEPJQYA, seg_pos={0 0}, upload=358133]: Uploading piece... Sep 23 12:45:47.137734108 (407h17m23.177307238s) [noise=true , address=77.90.31.16:4733 , piece_id=IMNP5DSWF4CPWT3P6UP2VJVEKT673YNL4SI5QUSX5MEKFFJN3DWA, seg_pos={0 0}, upload=358133]: Uploading piece... Sep 23 12:45:47.139890183 (407h17m23.179463324s) [noise=true , address=145.79.227.15:4876 , piece_id=WBXTTAYGVLBTML5I5IXG5AIBW4XDQ754P2HYWMWWN4MULNC562JQ, seg_pos={0 0}, upload=358133]: Uploading piece... Sep 23 12:45:47.140497660 (407h17m23.18007079s) [noise=true , address=145.79.227.2:4115 , piece_id=BYWPJNIHDC3ZQ2BW62YKMDDVJDNQ3NJ22R3IN367SLIX7FWDQ4JQ, seg_pos={0 0}, upload=358133]: Uploading piece... Sep 23 12:45:47.141101550 (407h17m23.18067468s) [noise=true , address=45.91.116.10:4083 , piece_id=3GXNPYQNMJ2QMCO2WLVFPJUCAOZGXGVHAGSY27YSVHBCGIL7ZWXA, seg_pos={0 0}, upload=358133]: Uploading piece... Sep 23 12:45:47.141613976 (407h17m23.181187126s) [noise=true , address=77.90.31.19:4843 , piece_id=EH7WKV5NTWIKNGYTLKIXZRIKHD7UAXJISMMNWWJNKLOTJGS2GINQ, seg_pos={0 0}, upload=358133]: Uploading piece... Sep 23 12:45:47.142070455 (407h17m23.181643585s) [noise=true , address=45.91.116.6:4974 , piece_id=5UZGQUDVTEOQU542GK7TLCFPWIN3ZB4XLHBPDC3XMDDR6UO7Z5GA, seg_pos={0 0}, upload=358133]: Uploading piece... Sep 23 12:45:48.922402091 (407h17m24.961975221s) [upload=358133]: Next returned nil segment Sep 23 12:45:48.922429062 (407h17m24.962002192s) [upload=358133]: Waiting for error group managing segments... Sep 23 12:45:49.311915921 (407h17m25.351489041s) [noise=true , address=45.91.116.9:4262 , piece_id=ST3YZKNCSKBNBTYMYKQSXUD77ICDKBZQAYE4HFMWRAMV6HXZD22Q, seg_pos={0 0}, upload=358133]: Done uploading piece. err: <nil> Sep 23 12:45:49.343112756 (407h17m25.382685886s) [noise=true , address=77.90.31.19:4843 , piece_id=EH7WKV5NTWIKNGYTLKIXZRIKHD7UAXJISMMNWWJNKLOTJGS2GINQ, seg_pos={0 0}, upload=358133]: Done uploading piece. err: <nil> Sep 23 12:45:49.345575395 (407h17m25.385148525s) [noise=true , address=45.91.116.14:4191 , piece_id=JFV37O2MDPIZ4DEHSPJOCDVROQ2S2A4XEOMUM3CTHHK2QBEPJQYA, seg_pos={0 0}, upload=358133]: Done uploading piece. err: <nil> Sep 23 12:45:49.346103410 (407h17m25.38567654s) [noise=true , address=45.91.116.10:4083 , piece_id=3GXNPYQNMJ2QMCO2WLVFPJUCAOZGXGVHAGSY27YSVHBCGIL7ZWXA, seg_pos={0 0}, upload=358133]: Done uploading piece. err: <nil> Sep 23 12:45:49.352547351 (407h17m25.392120501s) [noise=true , address=77.90.31.16:4733 , piece_id=IMNP5DSWF4CPWT3P6UP2VJVEKT673YNL4SI5QUSX5MEKFFJN3DWA, seg_pos={0 0}, upload=358133]: Done uploading piece. err: <nil> Sep 23 12:45:49.357361267 (407h17m25.396934387s) [noise=true , address=77.90.31.13:4626 , piece_id=N6VTJT2T74RVKVDQLXXX2P74FDW2ILEJWNI2HRK4QNIZ7AUYNUEA, seg_pos={0 0}, upload=358133]: Done uploading piece. err: <nil> Sep 23 12:45:49.358414112 (407h17m25.397987232s) [noise=true , address=45.91.116.11:4119 , piece_id=FSMWXG3WWD4UA4QWOE66MUD2CAM7CJWAHOE476BPM5ZLX24WGG7A, seg_pos={0 0}, upload=358133]: Done uploading piece. err: <nil> Sep 23 12:45:49.362534819 (407h17m25.402107949s) [noise=true , address=45.91.116.11:4118 , piece_id=NFPQYRN6XQLLO5XABYVAUDGP7236SGODYEPFVCL5CTEAE5EGK5BA, seg_pos={0 0}, upload=358133]: Done uploading piece. err: <nil> Sep 23 12:45:49.366463900 (407h17m25.40603703s) [noise=true , address=45.91.116.6:4974 , piece_id=5UZGQUDVTEOQU542GK7TLCFPWIN3ZB4XLHBPDC3XMDDR6UO7Z5GA, seg_pos={0 0}, upload=358133]: Done uploading piece. err: <nil> Sep 23 12:45:49.982739825 (407h17m26.022312955s) [noise=true , address=145.79.227.2:4115 , piece_id=BYWPJNIHDC3ZQ2BW62YKMDDVJDNQ3NJ22R3IN367SLIX7FWDQ4JQ, seg_pos={0 0}, upload=358133]: Done uploading piece. err: <nil> Sep 23 12:45:49.997202504 (407h17m26.036775634s) [noise=true , address=145.79.227.15:4876 , piece_id=WBXTTAYGVLBTML5I5IXG5AIBW4XDQ754P2HYWMWWN4MULNC562JQ, seg_pos={0 0}, upload=358133]: Done uploading piece. err: <nil> Sep 23 12:45:50.032210215 (407h17m26.071783335s) [noise=true , address=145.79.227.7:4295 , piece_id=JYVLPPEMVFUFXAMYNFEIMMWQJLBXL2RRHUVXY55LZULE2JM3UEFA, seg_pos={0 0}, upload=358133]: Done uploading piece. err: <nil> Sep 23 12:45:50.047004215 (407h17m26.086577345s) [noise=true , address=145.79.227.13:4804 , piece_id=V5FFP33A4Z2SZUREHBB35I77AKDSHYL6WDEAK25CE76V3ZCGZ6YQ, seg_pos={0 0}, upload=358133]: Done uploading piece. err: <nil> Sep 23 12:45:50.047042498 (407h17m26.086615628s) [seg_pos={0 0}, upload=358133]: Segment reached optimal threshold of 13 pieces. Sep 23 12:45:50.047208334 (407h17m26.086781464s) [noise=true , address=145.79.227.17:4969 , piece_id=MS2QT2XAW7CCC5YXUQ5CFMCH7LVK5C3CEOZN44HCZ5YWSIXMQO3A, seg_pos={0 0}, upload=358133]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: expected piece hash; context canceled; EOF Sep 23 12:45:50.047309657 (407h17m26.086882787s) [noise=true , address=145.79.227.18:4007 , piece_id=OMVCGFCUBO56VZNIPNPWIGVE2VDGIE74XCMBJCJGCQ45NGKTLVQQ, seg_pos={0 0}, upload=358133]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp): protocol: expected piece hash; context canceled; EOF Sep 23 12:45:50.047666186 (407h17m26.087239306s) [noise=true , address=145.79.227.6:4259 , piece_id=PBQPI25TRRNVIRRV6P4GFXUUNHABPZMCUAYCMMMCLB6SFEBSICYA, seg_pos={0 0}, upload=358133]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y): protocol: write tcp 188.166.225.170:47316->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:47316->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47316->145.79.227.6:4259: use of closed network connection Sep 23 12:45:50.047711582 (407h17m26.087284712s) [seg_pos={0 0}, upload=358133]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 23 12:45:50.047817103 (407h17m26.087390253s) [upload=358133]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 23 12:45:50.047823576 (407h17m26.087396706s) [upload=358133]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 23 12:45:50.337144492 (407h17m26.376717612s) [upload=358133]: Done uploading segments... Sep 23 12:45:50.337169410 (407h17m26.37674254s) [upload=358133]: Upload finished. err: <nil> Duration: 6.329267724s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 8.615619298s FAIL (2025-09-23 13:29:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 23 12:45:46.780664005 (407h17m22.820237135s) [upload=358133]: Starting upload Sep 23 12:45:46.780766030 (407h17m22.82033915s) [upload=358133]: Done starting upload Sep 23 12:45:46.780906196 (407h17m22.820479326s) [upload=358133]: Uploading segments... Sep 23 12:45:46.781049119 (407h17m22.820622249s) [upload=358133]: Got next segment. Inline: false Sep 23 12:45:46.781085078 (407h17m22.820658208s) [upload=358133]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 12:45:46.781091199 (407h17m22.820664329s) [upload=358133]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 12:45:47.134779542 (407h17m23.174352682s) [seg_pos={0 0}, upload=358133]: Begin upload segment... Sep 23 12:45:47.134833926 (407h17m23.174407056s) [seg_pos={0 0}, upload=358133]: Done begin upload segment. Sep 23 12:45:47.134868011 (407h17m23.174441141s) [noise=true , address=145.79.227.18:4007 , piece_id=OMVCGFCUBO56VZNIPNPWIGVE2VDGIE74XCMBJCJGCQ45NGKTLVQQ, seg_pos={0 0}, upload=358133]: Uploading piece... Sep 23 12:45:47.135481949 (407h17m23.175055079s) [noise=true , address=45.91.116.11:4118 , piece_id=NFPQYRN6XQLLO5XABYVAUDGP7236SGODYEPFVCL5CTEAE5EGK5BA, seg_pos={0 0}, upload=358133]: Uploading piece... Sep 23 12:45:47.136025715 (407h17m23.175598845s) [noise=true , address=45.91.116.11:4119 , piece_id=FSMWXG3WWD4UA4QWOE66MUD2CAM7CJWAHOE476BPM5ZLX24WGG7A, seg_pos={0 0}, upload=358133]: Uploading piece... Sep 23 12:45:47.136069518 (407h17m23.175642648s) [noise=true , address=45.91.116.9:4262 , piece_id=ST3YZKNCSKBNBTYMYKQSXUD77ICDKBZQAYE4HFMWRAMV6HXZD22Q, seg_pos={0 0}, upload=358133]: Uploading piece... Sep 23 12:45:47.136416288 (407h17m23.175989418s) [noise=true , address=77.90.31.13:4626 , piece_id=N6VTJT2T74RVKVDQLXXX2P74FDW2ILEJWNI2HRK4QNIZ7AUYNUEA, seg_pos={0 0}, upload=358133]: Uploading piece... Sep 23 12:45:47.136624615 (407h17m23.176197735s) [noise=true , address=145.79.227.6:4259 , piece_id=PBQPI25TRRNVIRRV6P4GFXUUNHABPZMCUAYCMMMCLB6SFEBSICYA, seg_pos={0 0}, upload=358133]: Uploading piece... Sep 23 12:45:47.136803185 (407h17m23.176376315s) [noise=true , address=145.79.227.7:4295 , piece_id=JYVLPPEMVFUFXAMYNFEIMMWQJLBXL2RRHUVXY55LZULE2JM3UEFA, seg_pos={0 0}, upload=358133]: Uploading piece... Sep 23 12:45:47.136992296 (407h17m23.176565416s) [noise=true , address=145.79.227.13:4804 , piece_id=V5FFP33A4Z2SZUREHBB35I77AKDSHYL6WDEAK25CE76V3ZCGZ6YQ, seg_pos={0 0}, upload=358133]: Uploading piece... Sep 23 12:45:47.137186525 (407h17m23.176759655s) [noise=true , address=145.79.227.17:4969 , piece_id=MS2QT2XAW7CCC5YXUQ5CFMCH7LVK5C3CEOZN44HCZ5YWSIXMQO3A, seg_pos={0 0}, upload=358133]: Uploading piece... Sep 23 12:45:47.137369544 (407h17m23.176942674s) [noise=true , address=45.91.116.14:4191 , piece_id=JFV37O2MDPIZ4DEHSPJOCDVROQ2S2A4XEOMUM3CTHHK2QBEPJQYA, seg_pos={0 0}, upload=358133]: Uploading piece... Sep 23 12:45:47.137734108 (407h17m23.177307238s) [noise=true , address=77.90.31.16:4733 , piece_id=IMNP5DSWF4CPWT3P6UP2VJVEKT673YNL4SI5QUSX5MEKFFJN3DWA, seg_pos={0 0}, upload=358133]: Uploading piece... Sep 23 12:45:47.139890183 (407h17m23.179463324s) [noise=true , address=145.79.227.15:4876 , piece_id=WBXTTAYGVLBTML5I5IXG5AIBW4XDQ754P2HYWMWWN4MULNC562JQ, seg_pos={0 0}, upload=358133]: Uploading piece... Sep 23 12:45:47.140497660 (407h17m23.18007079s) [noise=true , address=145.79.227.2:4115 , piece_id=BYWPJNIHDC3ZQ2BW62YKMDDVJDNQ3NJ22R3IN367SLIX7FWDQ4JQ, seg_pos={0 0}, upload=358133]: Uploading piece... Sep 23 12:45:47.141101550 (407h17m23.18067468s) [noise=true , address=45.91.116.10:4083 , piece_id=3GXNPYQNMJ2QMCO2WLVFPJUCAOZGXGVHAGSY27YSVHBCGIL7ZWXA, seg_pos={0 0}, upload=358133]: Uploading piece... Sep 23 12:45:47.141613976 (407h17m23.181187126s) [noise=true , address=77.90.31.19:4843 , piece_id=EH7WKV5NTWIKNGYTLKIXZRIKHD7UAXJISMMNWWJNKLOTJGS2GINQ, seg_pos={0 0}, upload=358133]: Uploading piece... Sep 23 12:45:47.142070455 (407h17m23.181643585s) [noise=true , address=45.91.116.6:4974 , piece_id=5UZGQUDVTEOQU542GK7TLCFPWIN3ZB4XLHBPDC3XMDDR6UO7Z5GA, seg_pos={0 0}, upload=358133]: Uploading piece... Sep 23 12:45:48.922402091 (407h17m24.961975221s) [upload=358133]: Next returned nil segment Sep 23 12:45:48.922429062 (407h17m24.962002192s) [upload=358133]: Waiting for error group managing segments... Sep 23 12:45:49.311915921 (407h17m25.351489041s) [noise=true , address=45.91.116.9:4262 , piece_id=ST3YZKNCSKBNBTYMYKQSXUD77ICDKBZQAYE4HFMWRAMV6HXZD22Q, seg_pos={0 0}, upload=358133]: Done uploading piece. err: <nil> Sep 23 12:45:49.343112756 (407h17m25.382685886s) [noise=true , address=77.90.31.19:4843 , piece_id=EH7WKV5NTWIKNGYTLKIXZRIKHD7UAXJISMMNWWJNKLOTJGS2GINQ, seg_pos={0 0}, upload=358133]: Done uploading piece. err: <nil> Sep 23 12:45:49.345575395 (407h17m25.385148525s) [noise=true , address=45.91.116.14:4191 , piece_id=JFV37O2MDPIZ4DEHSPJOCDVROQ2S2A4XEOMUM3CTHHK2QBEPJQYA, seg_pos={0 0}, upload=358133]: Done uploading piece. err: <nil> Sep 23 12:45:49.346103410 (407h17m25.38567654s) [noise=true , address=45.91.116.10:4083 , piece_id=3GXNPYQNMJ2QMCO2WLVFPJUCAOZGXGVHAGSY27YSVHBCGIL7ZWXA, seg_pos={0 0}, upload=358133]: Done uploading piece. err: <nil> Sep 23 12:45:49.352547351 (407h17m25.392120501s) [noise=true , address=77.90.31.16:4733 , piece_id=IMNP5DSWF4CPWT3P6UP2VJVEKT673YNL4SI5QUSX5MEKFFJN3DWA, seg_pos={0 0}, upload=358133]: Done uploading piece. err: <nil> Sep 23 12:45:49.357361267 (407h17m25.396934387s) [noise=true , address=77.90.31.13:4626 , piece_id=N6VTJT2T74RVKVDQLXXX2P74FDW2ILEJWNI2HRK4QNIZ7AUYNUEA, seg_pos={0 0}, upload=358133]: Done uploading piece. err: <nil> Sep 23 12:45:49.358414112 (407h17m25.397987232s) [noise=true , address=45.91.116.11:4119 , piece_id=FSMWXG3WWD4UA4QWOE66MUD2CAM7CJWAHOE476BPM5ZLX24WGG7A, seg_pos={0 0}, upload=358133]: Done uploading piece. err: <nil> Sep 23 12:45:49.362534819 (407h17m25.402107949s) [noise=true , address=45.91.116.11:4118 , piece_id=NFPQYRN6XQLLO5XABYVAUDGP7236SGODYEPFVCL5CTEAE5EGK5BA, seg_pos={0 0}, upload=358133]: Done uploading piece. err: <nil> Sep 23 12:45:49.366463900 (407h17m25.40603703s) [noise=true , address=45.91.116.6:4974 , piece_id=5UZGQUDVTEOQU542GK7TLCFPWIN3ZB4XLHBPDC3XMDDR6UO7Z5GA, seg_pos={0 0}, upload=358133]: Done uploading piece. err: <nil> Sep 23 12:45:49.982739825 (407h17m26.022312955s) [noise=true , address=145.79.227.2:4115 , piece_id=BYWPJNIHDC3ZQ2BW62YKMDDVJDNQ3NJ22R3IN367SLIX7FWDQ4JQ, seg_pos={0 0}, upload=358133]: Done uploading piece. err: <nil> Sep 23 12:45:49.997202504 (407h17m26.036775634s) [noise=true , address=145.79.227.15:4876 , piece_id=WBXTTAYGVLBTML5I5IXG5AIBW4XDQ754P2HYWMWWN4MULNC562JQ, seg_pos={0 0}, upload=358133]: Done uploading piece. err: <nil> Sep 23 12:45:50.032210215 (407h17m26.071783335s) [noise=true , address=145.79.227.7:4295 , piece_id=JYVLPPEMVFUFXAMYNFEIMMWQJLBXL2RRHUVXY55LZULE2JM3UEFA, seg_pos={0 0}, upload=358133]: Done uploading piece. err: <nil> Sep 23 12:45:50.047004215 (407h17m26.086577345s) [noise=true , address=145.79.227.13:4804 , piece_id=V5FFP33A4Z2SZUREHBB35I77AKDSHYL6WDEAK25CE76V3ZCGZ6YQ, seg_pos={0 0}, upload=358133]: Done uploading piece. err: <nil> Sep 23 12:45:50.047042498 (407h17m26.086615628s) [seg_pos={0 0}, upload=358133]: Segment reached optimal threshold of 13 pieces. Sep 23 12:45:50.047208334 (407h17m26.086781464s) [noise=true , address=145.79.227.17:4969 , piece_id=MS2QT2XAW7CCC5YXUQ5CFMCH7LVK5C3CEOZN44HCZ5YWSIXMQO3A, seg_pos={0 0}, upload=358133]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: expected piece hash; context canceled; EOF Sep 23 12:45:50.047309657 (407h17m26.086882787s) [noise=true , address=145.79.227.18:4007 , piece_id=OMVCGFCUBO56VZNIPNPWIGVE2VDGIE74XCMBJCJGCQ45NGKTLVQQ, seg_pos={0 0}, upload=358133]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp): protocol: expected piece hash; context canceled; EOF Sep 23 12:45:50.047666186 (407h17m26.087239306s) [noise=true , address=145.79.227.6:4259 , piece_id=PBQPI25TRRNVIRRV6P4GFXUUNHABPZMCUAYCMMMCLB6SFEBSICYA, seg_pos={0 0}, upload=358133]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y): protocol: write tcp 188.166.225.170:47316->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:47316->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47316->145.79.227.6:4259: use of closed network connection Sep 23 12:45:50.047711582 (407h17m26.087284712s) [seg_pos={0 0}, upload=358133]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 23 12:45:50.047817103 (407h17m26.087390253s) [upload=358133]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 23 12:45:50.047823576 (407h17m26.087396706s) [upload=358133]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 23 12:45:50.337144492 (407h17m26.376717612s) [upload=358133]: Done uploading segments... Sep 23 12:45:50.337169410 (407h17m26.37674254s) [upload=358133]: Upload finished. err: <nil> Duration: 8.615650367s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 6.976790776s FAIL (2025-09-23 13:28:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 23 12:45:46.780664005 (407h17m22.820237135s) [upload=358133]: Starting upload Sep 23 12:45:46.780766030 (407h17m22.82033915s) [upload=358133]: Done starting upload Sep 23 12:45:46.780906196 (407h17m22.820479326s) [upload=358133]: Uploading segments... Sep 23 12:45:46.781049119 (407h17m22.820622249s) [upload=358133]: Got next segment. Inline: false Sep 23 12:45:46.781085078 (407h17m22.820658208s) [upload=358133]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 12:45:46.781091199 (407h17m22.820664329s) [upload=358133]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 12:45:47.134779542 (407h17m23.174352682s) [seg_pos={0 0}, upload=358133]: Begin upload segment... Sep 23 12:45:47.134833926 (407h17m23.174407056s) [seg_pos={0 0}, upload=358133]: Done begin upload segment. Sep 23 12:45:47.134868011 (407h17m23.174441141s) [noise=true , address=145.79.227.18:4007 , piece_id=OMVCGFCUBO56VZNIPNPWIGVE2VDGIE74XCMBJCJGCQ45NGKTLVQQ, seg_pos={0 0}, upload=358133]: Uploading piece... Sep 23 12:45:47.135481949 (407h17m23.175055079s) [noise=true , address=45.91.116.11:4118 , piece_id=NFPQYRN6XQLLO5XABYVAUDGP7236SGODYEPFVCL5CTEAE5EGK5BA, seg_pos={0 0}, upload=358133]: Uploading piece... Sep 23 12:45:47.136025715 (407h17m23.175598845s) [noise=true , address=45.91.116.11:4119 , piece_id=FSMWXG3WWD4UA4QWOE66MUD2CAM7CJWAHOE476BPM5ZLX24WGG7A, seg_pos={0 0}, upload=358133]: Uploading piece... Sep 23 12:45:47.136069518 (407h17m23.175642648s) [noise=true , address=45.91.116.9:4262 , piece_id=ST3YZKNCSKBNBTYMYKQSXUD77ICDKBZQAYE4HFMWRAMV6HXZD22Q, seg_pos={0 0}, upload=358133]: Uploading piece... Sep 23 12:45:47.136416288 (407h17m23.175989418s) [noise=true , address=77.90.31.13:4626 , piece_id=N6VTJT2T74RVKVDQLXXX2P74FDW2ILEJWNI2HRK4QNIZ7AUYNUEA, seg_pos={0 0}, upload=358133]: Uploading piece... Sep 23 12:45:47.136624615 (407h17m23.176197735s) [noise=true , address=145.79.227.6:4259 , piece_id=PBQPI25TRRNVIRRV6P4GFXUUNHABPZMCUAYCMMMCLB6SFEBSICYA, seg_pos={0 0}, upload=358133]: Uploading piece... Sep 23 12:45:47.136803185 (407h17m23.176376315s) [noise=true , address=145.79.227.7:4295 , piece_id=JYVLPPEMVFUFXAMYNFEIMMWQJLBXL2RRHUVXY55LZULE2JM3UEFA, seg_pos={0 0}, upload=358133]: Uploading piece... Sep 23 12:45:47.136992296 (407h17m23.176565416s) [noise=true , address=145.79.227.13:4804 , piece_id=V5FFP33A4Z2SZUREHBB35I77AKDSHYL6WDEAK25CE76V3ZCGZ6YQ, seg_pos={0 0}, upload=358133]: Uploading piece... Sep 23 12:45:47.137186525 (407h17m23.176759655s) [noise=true , address=145.79.227.17:4969 , piece_id=MS2QT2XAW7CCC5YXUQ5CFMCH7LVK5C3CEOZN44HCZ5YWSIXMQO3A, seg_pos={0 0}, upload=358133]: Uploading piece... Sep 23 12:45:47.137369544 (407h17m23.176942674s) [noise=true , address=45.91.116.14:4191 , piece_id=JFV37O2MDPIZ4DEHSPJOCDVROQ2S2A4XEOMUM3CTHHK2QBEPJQYA, seg_pos={0 0}, upload=358133]: Uploading piece... Sep 23 12:45:47.137734108 (407h17m23.177307238s) [noise=true , address=77.90.31.16:4733 , piece_id=IMNP5DSWF4CPWT3P6UP2VJVEKT673YNL4SI5QUSX5MEKFFJN3DWA, seg_pos={0 0}, upload=358133]: Uploading piece... Sep 23 12:45:47.139890183 (407h17m23.179463324s) [noise=true , address=145.79.227.15:4876 , piece_id=WBXTTAYGVLBTML5I5IXG5AIBW4XDQ754P2HYWMWWN4MULNC562JQ, seg_pos={0 0}, upload=358133]: Uploading piece... Sep 23 12:45:47.140497660 (407h17m23.18007079s) [noise=true , address=145.79.227.2:4115 , piece_id=BYWPJNIHDC3ZQ2BW62YKMDDVJDNQ3NJ22R3IN367SLIX7FWDQ4JQ, seg_pos={0 0}, upload=358133]: Uploading piece... Sep 23 12:45:47.141101550 (407h17m23.18067468s) [noise=true , address=45.91.116.10:4083 , piece_id=3GXNPYQNMJ2QMCO2WLVFPJUCAOZGXGVHAGSY27YSVHBCGIL7ZWXA, seg_pos={0 0}, upload=358133]: Uploading piece... Sep 23 12:45:47.141613976 (407h17m23.181187126s) [noise=true , address=77.90.31.19:4843 , piece_id=EH7WKV5NTWIKNGYTLKIXZRIKHD7UAXJISMMNWWJNKLOTJGS2GINQ, seg_pos={0 0}, upload=358133]: Uploading piece... Sep 23 12:45:47.142070455 (407h17m23.181643585s) [noise=true , address=45.91.116.6:4974 , piece_id=5UZGQUDVTEOQU542GK7TLCFPWIN3ZB4XLHBPDC3XMDDR6UO7Z5GA, seg_pos={0 0}, upload=358133]: Uploading piece... Sep 23 12:45:48.922402091 (407h17m24.961975221s) [upload=358133]: Next returned nil segment Sep 23 12:45:48.922429062 (407h17m24.962002192s) [upload=358133]: Waiting for error group managing segments... Sep 23 12:45:49.311915921 (407h17m25.351489041s) [noise=true , address=45.91.116.9:4262 , piece_id=ST3YZKNCSKBNBTYMYKQSXUD77ICDKBZQAYE4HFMWRAMV6HXZD22Q, seg_pos={0 0}, upload=358133]: Done uploading piece. err: <nil> Sep 23 12:45:49.343112756 (407h17m25.382685886s) [noise=true , address=77.90.31.19:4843 , piece_id=EH7WKV5NTWIKNGYTLKIXZRIKHD7UAXJISMMNWWJNKLOTJGS2GINQ, seg_pos={0 0}, upload=358133]: Done uploading piece. err: <nil> Sep 23 12:45:49.345575395 (407h17m25.385148525s) [noise=true , address=45.91.116.14:4191 , piece_id=JFV37O2MDPIZ4DEHSPJOCDVROQ2S2A4XEOMUM3CTHHK2QBEPJQYA, seg_pos={0 0}, upload=358133]: Done uploading piece. err: <nil> Sep 23 12:45:49.346103410 (407h17m25.38567654s) [noise=true , address=45.91.116.10:4083 , piece_id=3GXNPYQNMJ2QMCO2WLVFPJUCAOZGXGVHAGSY27YSVHBCGIL7ZWXA, seg_pos={0 0}, upload=358133]: Done uploading piece. err: <nil> Sep 23 12:45:49.352547351 (407h17m25.392120501s) [noise=true , address=77.90.31.16:4733 , piece_id=IMNP5DSWF4CPWT3P6UP2VJVEKT673YNL4SI5QUSX5MEKFFJN3DWA, seg_pos={0 0}, upload=358133]: Done uploading piece. err: <nil> Sep 23 12:45:49.357361267 (407h17m25.396934387s) [noise=true , address=77.90.31.13:4626 , piece_id=N6VTJT2T74RVKVDQLXXX2P74FDW2ILEJWNI2HRK4QNIZ7AUYNUEA, seg_pos={0 0}, upload=358133]: Done uploading piece. err: <nil> Sep 23 12:45:49.358414112 (407h17m25.397987232s) [noise=true , address=45.91.116.11:4119 , piece_id=FSMWXG3WWD4UA4QWOE66MUD2CAM7CJWAHOE476BPM5ZLX24WGG7A, seg_pos={0 0}, upload=358133]: Done uploading piece. err: <nil> Sep 23 12:45:49.362534819 (407h17m25.402107949s) [noise=true , address=45.91.116.11:4118 , piece_id=NFPQYRN6XQLLO5XABYVAUDGP7236SGODYEPFVCL5CTEAE5EGK5BA, seg_pos={0 0}, upload=358133]: Done uploading piece. err: <nil> Sep 23 12:45:49.366463900 (407h17m25.40603703s) [noise=true , address=45.91.116.6:4974 , piece_id=5UZGQUDVTEOQU542GK7TLCFPWIN3ZB4XLHBPDC3XMDDR6UO7Z5GA, seg_pos={0 0}, upload=358133]: Done uploading piece. err: <nil> Sep 23 12:45:49.982739825 (407h17m26.022312955s) [noise=true , address=145.79.227.2:4115 , piece_id=BYWPJNIHDC3ZQ2BW62YKMDDVJDNQ3NJ22R3IN367SLIX7FWDQ4JQ, seg_pos={0 0}, upload=358133]: Done uploading piece. err: <nil> Sep 23 12:45:49.997202504 (407h17m26.036775634s) [noise=true , address=145.79.227.15:4876 , piece_id=WBXTTAYGVLBTML5I5IXG5AIBW4XDQ754P2HYWMWWN4MULNC562JQ, seg_pos={0 0}, upload=358133]: Done uploading piece. err: <nil> Sep 23 12:45:50.032210215 (407h17m26.071783335s) [noise=true , address=145.79.227.7:4295 , piece_id=JYVLPPEMVFUFXAMYNFEIMMWQJLBXL2RRHUVXY55LZULE2JM3UEFA, seg_pos={0 0}, upload=358133]: Done uploading piece. err: <nil> Sep 23 12:45:50.047004215 (407h17m26.086577345s) [noise=true , address=145.79.227.13:4804 , piece_id=V5FFP33A4Z2SZUREHBB35I77AKDSHYL6WDEAK25CE76V3ZCGZ6YQ, seg_pos={0 0}, upload=358133]: Done uploading piece. err: <nil> Sep 23 12:45:50.047042498 (407h17m26.086615628s) [seg_pos={0 0}, upload=358133]: Segment reached optimal threshold of 13 pieces. Sep 23 12:45:50.047208334 (407h17m26.086781464s) [noise=true , address=145.79.227.17:4969 , piece_id=MS2QT2XAW7CCC5YXUQ5CFMCH7LVK5C3CEOZN44HCZ5YWSIXMQO3A, seg_pos={0 0}, upload=358133]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: expected piece hash; context canceled; EOF Sep 23 12:45:50.047309657 (407h17m26.086882787s) [noise=true , address=145.79.227.18:4007 , piece_id=OMVCGFCUBO56VZNIPNPWIGVE2VDGIE74XCMBJCJGCQ45NGKTLVQQ, seg_pos={0 0}, upload=358133]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp): protocol: expected piece hash; context canceled; EOF Sep 23 12:45:50.047666186 (407h17m26.087239306s) [noise=true , address=145.79.227.6:4259 , piece_id=PBQPI25TRRNVIRRV6P4GFXUUNHABPZMCUAYCMMMCLB6SFEBSICYA, seg_pos={0 0}, upload=358133]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y): protocol: write tcp 188.166.225.170:47316->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:47316->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47316->145.79.227.6:4259: use of closed network connection Sep 23 12:45:50.047711582 (407h17m26.087284712s) [seg_pos={0 0}, upload=358133]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 23 12:45:50.047817103 (407h17m26.087390253s) [upload=358133]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 23 12:45:50.047823576 (407h17m26.087396706s) [upload=358133]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 23 12:45:50.337144492 (407h17m26.376717612s) [upload=358133]: Done uploading segments... Sep 23 12:45:50.337169410 (407h17m26.37674254s) [upload=358133]: Upload finished. err: <nil> Duration: 6.976817586s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 6.730058182s FAIL (2025-09-23 13:27:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 23 12:45:46.780664005 (407h17m22.820237135s) [upload=358133]: Starting upload Sep 23 12:45:46.780766030 (407h17m22.82033915s) [upload=358133]: Done starting upload Sep 23 12:45:46.780906196 (407h17m22.820479326s) [upload=358133]: Uploading segments... Sep 23 12:45:46.781049119 (407h17m22.820622249s) [upload=358133]: Got next segment. Inline: false Sep 23 12:45:46.781085078 (407h17m22.820658208s) [upload=358133]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 12:45:46.781091199 (407h17m22.820664329s) [upload=358133]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 12:45:47.134779542 (407h17m23.174352682s) [seg_pos={0 0}, upload=358133]: Begin upload segment... Sep 23 12:45:47.134833926 (407h17m23.174407056s) [seg_pos={0 0}, upload=358133]: Done begin upload segment. Sep 23 12:45:47.134868011 (407h17m23.174441141s) [noise=true , address=145.79.227.18:4007 , piece_id=OMVCGFCUBO56VZNIPNPWIGVE2VDGIE74XCMBJCJGCQ45NGKTLVQQ, seg_pos={0 0}, upload=358133]: Uploading piece... Sep 23 12:45:47.135481949 (407h17m23.175055079s) [noise=true , address=45.91.116.11:4118 , piece_id=NFPQYRN6XQLLO5XABYVAUDGP7236SGODYEPFVCL5CTEAE5EGK5BA, seg_pos={0 0}, upload=358133]: Uploading piece... Sep 23 12:45:47.136025715 (407h17m23.175598845s) [noise=true , address=45.91.116.11:4119 , piece_id=FSMWXG3WWD4UA4QWOE66MUD2CAM7CJWAHOE476BPM5ZLX24WGG7A, seg_pos={0 0}, upload=358133]: Uploading piece... Sep 23 12:45:47.136069518 (407h17m23.175642648s) [noise=true , address=45.91.116.9:4262 , piece_id=ST3YZKNCSKBNBTYMYKQSXUD77ICDKBZQAYE4HFMWRAMV6HXZD22Q, seg_pos={0 0}, upload=358133]: Uploading piece... Sep 23 12:45:47.136416288 (407h17m23.175989418s) [noise=true , address=77.90.31.13:4626 , piece_id=N6VTJT2T74RVKVDQLXXX2P74FDW2ILEJWNI2HRK4QNIZ7AUYNUEA, seg_pos={0 0}, upload=358133]: Uploading piece... Sep 23 12:45:47.136624615 (407h17m23.176197735s) [noise=true , address=145.79.227.6:4259 , piece_id=PBQPI25TRRNVIRRV6P4GFXUUNHABPZMCUAYCMMMCLB6SFEBSICYA, seg_pos={0 0}, upload=358133]: Uploading piece... Sep 23 12:45:47.136803185 (407h17m23.176376315s) [noise=true , address=145.79.227.7:4295 , piece_id=JYVLPPEMVFUFXAMYNFEIMMWQJLBXL2RRHUVXY55LZULE2JM3UEFA, seg_pos={0 0}, upload=358133]: Uploading piece... Sep 23 12:45:47.136992296 (407h17m23.176565416s) [noise=true , address=145.79.227.13:4804 , piece_id=V5FFP33A4Z2SZUREHBB35I77AKDSHYL6WDEAK25CE76V3ZCGZ6YQ, seg_pos={0 0}, upload=358133]: Uploading piece... Sep 23 12:45:47.137186525 (407h17m23.176759655s) [noise=true , address=145.79.227.17:4969 , piece_id=MS2QT2XAW7CCC5YXUQ5CFMCH7LVK5C3CEOZN44HCZ5YWSIXMQO3A, seg_pos={0 0}, upload=358133]: Uploading piece... Sep 23 12:45:47.137369544 (407h17m23.176942674s) [noise=true , address=45.91.116.14:4191 , piece_id=JFV37O2MDPIZ4DEHSPJOCDVROQ2S2A4XEOMUM3CTHHK2QBEPJQYA, seg_pos={0 0}, upload=358133]: Uploading piece... Sep 23 12:45:47.137734108 (407h17m23.177307238s) [noise=true , address=77.90.31.16:4733 , piece_id=IMNP5DSWF4CPWT3P6UP2VJVEKT673YNL4SI5QUSX5MEKFFJN3DWA, seg_pos={0 0}, upload=358133]: Uploading piece... Sep 23 12:45:47.139890183 (407h17m23.179463324s) [noise=true , address=145.79.227.15:4876 , piece_id=WBXTTAYGVLBTML5I5IXG5AIBW4XDQ754P2HYWMWWN4MULNC562JQ, seg_pos={0 0}, upload=358133]: Uploading piece... Sep 23 12:45:47.140497660 (407h17m23.18007079s) [noise=true , address=145.79.227.2:4115 , piece_id=BYWPJNIHDC3ZQ2BW62YKMDDVJDNQ3NJ22R3IN367SLIX7FWDQ4JQ, seg_pos={0 0}, upload=358133]: Uploading piece... Sep 23 12:45:47.141101550 (407h17m23.18067468s) [noise=true , address=45.91.116.10:4083 , piece_id=3GXNPYQNMJ2QMCO2WLVFPJUCAOZGXGVHAGSY27YSVHBCGIL7ZWXA, seg_pos={0 0}, upload=358133]: Uploading piece... Sep 23 12:45:47.141613976 (407h17m23.181187126s) [noise=true , address=77.90.31.19:4843 , piece_id=EH7WKV5NTWIKNGYTLKIXZRIKHD7UAXJISMMNWWJNKLOTJGS2GINQ, seg_pos={0 0}, upload=358133]: Uploading piece... Sep 23 12:45:47.142070455 (407h17m23.181643585s) [noise=true , address=45.91.116.6:4974 , piece_id=5UZGQUDVTEOQU542GK7TLCFPWIN3ZB4XLHBPDC3XMDDR6UO7Z5GA, seg_pos={0 0}, upload=358133]: Uploading piece... Sep 23 12:45:48.922402091 (407h17m24.961975221s) [upload=358133]: Next returned nil segment Sep 23 12:45:48.922429062 (407h17m24.962002192s) [upload=358133]: Waiting for error group managing segments... Sep 23 12:45:49.311915921 (407h17m25.351489041s) [noise=true , address=45.91.116.9:4262 , piece_id=ST3YZKNCSKBNBTYMYKQSXUD77ICDKBZQAYE4HFMWRAMV6HXZD22Q, seg_pos={0 0}, upload=358133]: Done uploading piece. err: <nil> Sep 23 12:45:49.343112756 (407h17m25.382685886s) [noise=true , address=77.90.31.19:4843 , piece_id=EH7WKV5NTWIKNGYTLKIXZRIKHD7UAXJISMMNWWJNKLOTJGS2GINQ, seg_pos={0 0}, upload=358133]: Done uploading piece. err: <nil> Sep 23 12:45:49.345575395 (407h17m25.385148525s) [noise=true , address=45.91.116.14:4191 , piece_id=JFV37O2MDPIZ4DEHSPJOCDVROQ2S2A4XEOMUM3CTHHK2QBEPJQYA, seg_pos={0 0}, upload=358133]: Done uploading piece. err: <nil> Sep 23 12:45:49.346103410 (407h17m25.38567654s) [noise=true , address=45.91.116.10:4083 , piece_id=3GXNPYQNMJ2QMCO2WLVFPJUCAOZGXGVHAGSY27YSVHBCGIL7ZWXA, seg_pos={0 0}, upload=358133]: Done uploading piece. err: <nil> Sep 23 12:45:49.352547351 (407h17m25.392120501s) [noise=true , address=77.90.31.16:4733 , piece_id=IMNP5DSWF4CPWT3P6UP2VJVEKT673YNL4SI5QUSX5MEKFFJN3DWA, seg_pos={0 0}, upload=358133]: Done uploading piece. err: <nil> Sep 23 12:45:49.357361267 (407h17m25.396934387s) [noise=true , address=77.90.31.13:4626 , piece_id=N6VTJT2T74RVKVDQLXXX2P74FDW2ILEJWNI2HRK4QNIZ7AUYNUEA, seg_pos={0 0}, upload=358133]: Done uploading piece. err: <nil> Sep 23 12:45:49.358414112 (407h17m25.397987232s) [noise=true , address=45.91.116.11:4119 , piece_id=FSMWXG3WWD4UA4QWOE66MUD2CAM7CJWAHOE476BPM5ZLX24WGG7A, seg_pos={0 0}, upload=358133]: Done uploading piece. err: <nil> Sep 23 12:45:49.362534819 (407h17m25.402107949s) [noise=true , address=45.91.116.11:4118 , piece_id=NFPQYRN6XQLLO5XABYVAUDGP7236SGODYEPFVCL5CTEAE5EGK5BA, seg_pos={0 0}, upload=358133]: Done uploading piece. err: <nil> Sep 23 12:45:49.366463900 (407h17m25.40603703s) [noise=true , address=45.91.116.6:4974 , piece_id=5UZGQUDVTEOQU542GK7TLCFPWIN3ZB4XLHBPDC3XMDDR6UO7Z5GA, seg_pos={0 0}, upload=358133]: Done uploading piece. err: <nil> Sep 23 12:45:49.982739825 (407h17m26.022312955s) [noise=true , address=145.79.227.2:4115 , piece_id=BYWPJNIHDC3ZQ2BW62YKMDDVJDNQ3NJ22R3IN367SLIX7FWDQ4JQ, seg_pos={0 0}, upload=358133]: Done uploading piece. err: <nil> Sep 23 12:45:49.997202504 (407h17m26.036775634s) [noise=true , address=145.79.227.15:4876 , piece_id=WBXTTAYGVLBTML5I5IXG5AIBW4XDQ754P2HYWMWWN4MULNC562JQ, seg_pos={0 0}, upload=358133]: Done uploading piece. err: <nil> Sep 23 12:45:50.032210215 (407h17m26.071783335s) [noise=true , address=145.79.227.7:4295 , piece_id=JYVLPPEMVFUFXAMYNFEIMMWQJLBXL2RRHUVXY55LZULE2JM3UEFA, seg_pos={0 0}, upload=358133]: Done uploading piece. err: <nil> Sep 23 12:45:50.047004215 (407h17m26.086577345s) [noise=true , address=145.79.227.13:4804 , piece_id=V5FFP33A4Z2SZUREHBB35I77AKDSHYL6WDEAK25CE76V3ZCGZ6YQ, seg_pos={0 0}, upload=358133]: Done uploading piece. err: <nil> Sep 23 12:45:50.047042498 (407h17m26.086615628s) [seg_pos={0 0}, upload=358133]: Segment reached optimal threshold of 13 pieces. Sep 23 12:45:50.047208334 (407h17m26.086781464s) [noise=true , address=145.79.227.17:4969 , piece_id=MS2QT2XAW7CCC5YXUQ5CFMCH7LVK5C3CEOZN44HCZ5YWSIXMQO3A, seg_pos={0 0}, upload=358133]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: expected piece hash; context canceled; EOF Sep 23 12:45:50.047309657 (407h17m26.086882787s) [noise=true , address=145.79.227.18:4007 , piece_id=OMVCGFCUBO56VZNIPNPWIGVE2VDGIE74XCMBJCJGCQ45NGKTLVQQ, seg_pos={0 0}, upload=358133]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp): protocol: expected piece hash; context canceled; EOF Sep 23 12:45:50.047666186 (407h17m26.087239306s) [noise=true , address=145.79.227.6:4259 , piece_id=PBQPI25TRRNVIRRV6P4GFXUUNHABPZMCUAYCMMMCLB6SFEBSICYA, seg_pos={0 0}, upload=358133]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y): protocol: write tcp 188.166.225.170:47316->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:47316->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47316->145.79.227.6:4259: use of closed network connection Sep 23 12:45:50.047711582 (407h17m26.087284712s) [seg_pos={0 0}, upload=358133]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 23 12:45:50.047817103 (407h17m26.087390253s) [upload=358133]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 23 12:45:50.047823576 (407h17m26.087396706s) [upload=358133]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 23 12:45:50.337144492 (407h17m26.376717612s) [upload=358133]: Done uploading segments... Sep 23 12:45:50.337169410 (407h17m26.37674254s) [upload=358133]: Upload finished. err: <nil> Duration: 6.730091254s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 7.233652235s FAIL (2025-09-23 13:27:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 23 12:45:46.780664005 (407h17m22.820237135s) [upload=358133]: Starting upload Sep 23 12:45:46.780766030 (407h17m22.82033915s) [upload=358133]: Done starting upload Sep 23 12:45:46.780906196 (407h17m22.820479326s) [upload=358133]: Uploading segments... Sep 23 12:45:46.781049119 (407h17m22.820622249s) [upload=358133]: Got next segment. Inline: false Sep 23 12:45:46.781085078 (407h17m22.820658208s) [upload=358133]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 12:45:46.781091199 (407h17m22.820664329s) [upload=358133]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 12:45:47.134779542 (407h17m23.174352682s) [seg_pos={0 0}, upload=358133]: Begin upload segment... Sep 23 12:45:47.134833926 (407h17m23.174407056s) [seg_pos={0 0}, upload=358133]: Done begin upload segment. Sep 23 12:45:47.134868011 (407h17m23.174441141s) [noise=true , address=145.79.227.18:4007 , piece_id=OMVCGFCUBO56VZNIPNPWIGVE2VDGIE74XCMBJCJGCQ45NGKTLVQQ, seg_pos={0 0}, upload=358133]: Uploading piece... Sep 23 12:45:47.135481949 (407h17m23.175055079s) [noise=true , address=45.91.116.11:4118 , piece_id=NFPQYRN6XQLLO5XABYVAUDGP7236SGODYEPFVCL5CTEAE5EGK5BA, seg_pos={0 0}, upload=358133]: Uploading piece... Sep 23 12:45:47.136025715 (407h17m23.175598845s) [noise=true , address=45.91.116.11:4119 , piece_id=FSMWXG3WWD4UA4QWOE66MUD2CAM7CJWAHOE476BPM5ZLX24WGG7A, seg_pos={0 0}, upload=358133]: Uploading piece... Sep 23 12:45:47.136069518 (407h17m23.175642648s) [noise=true , address=45.91.116.9:4262 , piece_id=ST3YZKNCSKBNBTYMYKQSXUD77ICDKBZQAYE4HFMWRAMV6HXZD22Q, seg_pos={0 0}, upload=358133]: Uploading piece... Sep 23 12:45:47.136416288 (407h17m23.175989418s) [noise=true , address=77.90.31.13:4626 , piece_id=N6VTJT2T74RVKVDQLXXX2P74FDW2ILEJWNI2HRK4QNIZ7AUYNUEA, seg_pos={0 0}, upload=358133]: Uploading piece... Sep 23 12:45:47.136624615 (407h17m23.176197735s) [noise=true , address=145.79.227.6:4259 , piece_id=PBQPI25TRRNVIRRV6P4GFXUUNHABPZMCUAYCMMMCLB6SFEBSICYA, seg_pos={0 0}, upload=358133]: Uploading piece... Sep 23 12:45:47.136803185 (407h17m23.176376315s) [noise=true , address=145.79.227.7:4295 , piece_id=JYVLPPEMVFUFXAMYNFEIMMWQJLBXL2RRHUVXY55LZULE2JM3UEFA, seg_pos={0 0}, upload=358133]: Uploading piece... Sep 23 12:45:47.136992296 (407h17m23.176565416s) [noise=true , address=145.79.227.13:4804 , piece_id=V5FFP33A4Z2SZUREHBB35I77AKDSHYL6WDEAK25CE76V3ZCGZ6YQ, seg_pos={0 0}, upload=358133]: Uploading piece... Sep 23 12:45:47.137186525 (407h17m23.176759655s) [noise=true , address=145.79.227.17:4969 , piece_id=MS2QT2XAW7CCC5YXUQ5CFMCH7LVK5C3CEOZN44HCZ5YWSIXMQO3A, seg_pos={0 0}, upload=358133]: Uploading piece... Sep 23 12:45:47.137369544 (407h17m23.176942674s) [noise=true , address=45.91.116.14:4191 , piece_id=JFV37O2MDPIZ4DEHSPJOCDVROQ2S2A4XEOMUM3CTHHK2QBEPJQYA, seg_pos={0 0}, upload=358133]: Uploading piece... Sep 23 12:45:47.137734108 (407h17m23.177307238s) [noise=true , address=77.90.31.16:4733 , piece_id=IMNP5DSWF4CPWT3P6UP2VJVEKT673YNL4SI5QUSX5MEKFFJN3DWA, seg_pos={0 0}, upload=358133]: Uploading piece... Sep 23 12:45:47.139890183 (407h17m23.179463324s) [noise=true , address=145.79.227.15:4876 , piece_id=WBXTTAYGVLBTML5I5IXG5AIBW4XDQ754P2HYWMWWN4MULNC562JQ, seg_pos={0 0}, upload=358133]: Uploading piece... Sep 23 12:45:47.140497660 (407h17m23.18007079s) [noise=true , address=145.79.227.2:4115 , piece_id=BYWPJNIHDC3ZQ2BW62YKMDDVJDNQ3NJ22R3IN367SLIX7FWDQ4JQ, seg_pos={0 0}, upload=358133]: Uploading piece... Sep 23 12:45:47.141101550 (407h17m23.18067468s) [noise=true , address=45.91.116.10:4083 , piece_id=3GXNPYQNMJ2QMCO2WLVFPJUCAOZGXGVHAGSY27YSVHBCGIL7ZWXA, seg_pos={0 0}, upload=358133]: Uploading piece... Sep 23 12:45:47.141613976 (407h17m23.181187126s) [noise=true , address=77.90.31.19:4843 , piece_id=EH7WKV5NTWIKNGYTLKIXZRIKHD7UAXJISMMNWWJNKLOTJGS2GINQ, seg_pos={0 0}, upload=358133]: Uploading piece... Sep 23 12:45:47.142070455 (407h17m23.181643585s) [noise=true , address=45.91.116.6:4974 , piece_id=5UZGQUDVTEOQU542GK7TLCFPWIN3ZB4XLHBPDC3XMDDR6UO7Z5GA, seg_pos={0 0}, upload=358133]: Uploading piece... Sep 23 12:45:48.922402091 (407h17m24.961975221s) [upload=358133]: Next returned nil segment Sep 23 12:45:48.922429062 (407h17m24.962002192s) [upload=358133]: Waiting for error group managing segments... Sep 23 12:45:49.311915921 (407h17m25.351489041s) [noise=true , address=45.91.116.9:4262 , piece_id=ST3YZKNCSKBNBTYMYKQSXUD77ICDKBZQAYE4HFMWRAMV6HXZD22Q, seg_pos={0 0}, upload=358133]: Done uploading piece. err: <nil> Sep 23 12:45:49.343112756 (407h17m25.382685886s) [noise=true , address=77.90.31.19:4843 , piece_id=EH7WKV5NTWIKNGYTLKIXZRIKHD7UAXJISMMNWWJNKLOTJGS2GINQ, seg_pos={0 0}, upload=358133]: Done uploading piece. err: <nil> Sep 23 12:45:49.345575395 (407h17m25.385148525s) [noise=true , address=45.91.116.14:4191 , piece_id=JFV37O2MDPIZ4DEHSPJOCDVROQ2S2A4XEOMUM3CTHHK2QBEPJQYA, seg_pos={0 0}, upload=358133]: Done uploading piece. err: <nil> Sep 23 12:45:49.346103410 (407h17m25.38567654s) [noise=true , address=45.91.116.10:4083 , piece_id=3GXNPYQNMJ2QMCO2WLVFPJUCAOZGXGVHAGSY27YSVHBCGIL7ZWXA, seg_pos={0 0}, upload=358133]: Done uploading piece. err: <nil> Sep 23 12:45:49.352547351 (407h17m25.392120501s) [noise=true , address=77.90.31.16:4733 , piece_id=IMNP5DSWF4CPWT3P6UP2VJVEKT673YNL4SI5QUSX5MEKFFJN3DWA, seg_pos={0 0}, upload=358133]: Done uploading piece. err: <nil> Sep 23 12:45:49.357361267 (407h17m25.396934387s) [noise=true , address=77.90.31.13:4626 , piece_id=N6VTJT2T74RVKVDQLXXX2P74FDW2ILEJWNI2HRK4QNIZ7AUYNUEA, seg_pos={0 0}, upload=358133]: Done uploading piece. err: <nil> Sep 23 12:45:49.358414112 (407h17m25.397987232s) [noise=true , address=45.91.116.11:4119 , piece_id=FSMWXG3WWD4UA4QWOE66MUD2CAM7CJWAHOE476BPM5ZLX24WGG7A, seg_pos={0 0}, upload=358133]: Done uploading piece. err: <nil> Sep 23 12:45:49.362534819 (407h17m25.402107949s) [noise=true , address=45.91.116.11:4118 , piece_id=NFPQYRN6XQLLO5XABYVAUDGP7236SGODYEPFVCL5CTEAE5EGK5BA, seg_pos={0 0}, upload=358133]: Done uploading piece. err: <nil> Sep 23 12:45:49.366463900 (407h17m25.40603703s) [noise=true , address=45.91.116.6:4974 , piece_id=5UZGQUDVTEOQU542GK7TLCFPWIN3ZB4XLHBPDC3XMDDR6UO7Z5GA, seg_pos={0 0}, upload=358133]: Done uploading piece. err: <nil> Sep 23 12:45:49.982739825 (407h17m26.022312955s) [noise=true , address=145.79.227.2:4115 , piece_id=BYWPJNIHDC3ZQ2BW62YKMDDVJDNQ3NJ22R3IN367SLIX7FWDQ4JQ, seg_pos={0 0}, upload=358133]: Done uploading piece. err: <nil> Sep 23 12:45:49.997202504 (407h17m26.036775634s) [noise=true , address=145.79.227.15:4876 , piece_id=WBXTTAYGVLBTML5I5IXG5AIBW4XDQ754P2HYWMWWN4MULNC562JQ, seg_pos={0 0}, upload=358133]: Done uploading piece. err: <nil> Sep 23 12:45:50.032210215 (407h17m26.071783335s) [noise=true , address=145.79.227.7:4295 , piece_id=JYVLPPEMVFUFXAMYNFEIMMWQJLBXL2RRHUVXY55LZULE2JM3UEFA, seg_pos={0 0}, upload=358133]: Done uploading piece. err: <nil> Sep 23 12:45:50.047004215 (407h17m26.086577345s) [noise=true , address=145.79.227.13:4804 , piece_id=V5FFP33A4Z2SZUREHBB35I77AKDSHYL6WDEAK25CE76V3ZCGZ6YQ, seg_pos={0 0}, upload=358133]: Done uploading piece. err: <nil> Sep 23 12:45:50.047042498 (407h17m26.086615628s) [seg_pos={0 0}, upload=358133]: Segment reached optimal threshold of 13 pieces. Sep 23 12:45:50.047208334 (407h17m26.086781464s) [noise=true , address=145.79.227.17:4969 , piece_id=MS2QT2XAW7CCC5YXUQ5CFMCH7LVK5C3CEOZN44HCZ5YWSIXMQO3A, seg_pos={0 0}, upload=358133]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: expected piece hash; context canceled; EOF Sep 23 12:45:50.047309657 (407h17m26.086882787s) [noise=true , address=145.79.227.18:4007 , piece_id=OMVCGFCUBO56VZNIPNPWIGVE2VDGIE74XCMBJCJGCQ45NGKTLVQQ, seg_pos={0 0}, upload=358133]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp): protocol: expected piece hash; context canceled; EOF Sep 23 12:45:50.047666186 (407h17m26.087239306s) [noise=true , address=145.79.227.6:4259 , piece_id=PBQPI25TRRNVIRRV6P4GFXUUNHABPZMCUAYCMMMCLB6SFEBSICYA, seg_pos={0 0}, upload=358133]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y): protocol: write tcp 188.166.225.170:47316->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:47316->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47316->145.79.227.6:4259: use of closed network connection Sep 23 12:45:50.047711582 (407h17m26.087284712s) [seg_pos={0 0}, upload=358133]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 23 12:45:50.047817103 (407h17m26.087390253s) [upload=358133]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 23 12:45:50.047823576 (407h17m26.087396706s) [upload=358133]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 23 12:45:50.337144492 (407h17m26.376717612s) [upload=358133]: Done uploading segments... Sep 23 12:45:50.337169410 (407h17m26.37674254s) [upload=358133]: Upload finished. err: <nil> Duration: 7.233800679s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 7.731227801s FAIL (2025-09-23 13:26:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 23 12:45:46.780664005 (407h17m22.820237135s) [upload=358133]: Starting upload Sep 23 12:45:46.780766030 (407h17m22.82033915s) [upload=358133]: Done starting upload Sep 23 12:45:46.780906196 (407h17m22.820479326s) [upload=358133]: Uploading segments... Sep 23 12:45:46.781049119 (407h17m22.820622249s) [upload=358133]: Got next segment. Inline: false Sep 23 12:45:46.781085078 (407h17m22.820658208s) [upload=358133]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 12:45:46.781091199 (407h17m22.820664329s) [upload=358133]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 12:45:47.134779542 (407h17m23.174352682s) [seg_pos={0 0}, upload=358133]: Begin upload segment... Sep 23 12:45:47.134833926 (407h17m23.174407056s) [seg_pos={0 0}, upload=358133]: Done begin upload segment. Sep 23 12:45:47.134868011 (407h17m23.174441141s) [noise=true , address=145.79.227.18:4007 , piece_id=OMVCGFCUBO56VZNIPNPWIGVE2VDGIE74XCMBJCJGCQ45NGKTLVQQ, seg_pos={0 0}, upload=358133]: Uploading piece... Sep 23 12:45:47.135481949 (407h17m23.175055079s) [noise=true , address=45.91.116.11:4118 , piece_id=NFPQYRN6XQLLO5XABYVAUDGP7236SGODYEPFVCL5CTEAE5EGK5BA, seg_pos={0 0}, upload=358133]: Uploading piece... Sep 23 12:45:47.136025715 (407h17m23.175598845s) [noise=true , address=45.91.116.11:4119 , piece_id=FSMWXG3WWD4UA4QWOE66MUD2CAM7CJWAHOE476BPM5ZLX24WGG7A, seg_pos={0 0}, upload=358133]: Uploading piece... Sep 23 12:45:47.136069518 (407h17m23.175642648s) [noise=true , address=45.91.116.9:4262 , piece_id=ST3YZKNCSKBNBTYMYKQSXUD77ICDKBZQAYE4HFMWRAMV6HXZD22Q, seg_pos={0 0}, upload=358133]: Uploading piece... Sep 23 12:45:47.136416288 (407h17m23.175989418s) [noise=true , address=77.90.31.13:4626 , piece_id=N6VTJT2T74RVKVDQLXXX2P74FDW2ILEJWNI2HRK4QNIZ7AUYNUEA, seg_pos={0 0}, upload=358133]: Uploading piece... Sep 23 12:45:47.136624615 (407h17m23.176197735s) [noise=true , address=145.79.227.6:4259 , piece_id=PBQPI25TRRNVIRRV6P4GFXUUNHABPZMCUAYCMMMCLB6SFEBSICYA, seg_pos={0 0}, upload=358133]: Uploading piece... Sep 23 12:45:47.136803185 (407h17m23.176376315s) [noise=true , address=145.79.227.7:4295 , piece_id=JYVLPPEMVFUFXAMYNFEIMMWQJLBXL2RRHUVXY55LZULE2JM3UEFA, seg_pos={0 0}, upload=358133]: Uploading piece... Sep 23 12:45:47.136992296 (407h17m23.176565416s) [noise=true , address=145.79.227.13:4804 , piece_id=V5FFP33A4Z2SZUREHBB35I77AKDSHYL6WDEAK25CE76V3ZCGZ6YQ, seg_pos={0 0}, upload=358133]: Uploading piece... Sep 23 12:45:47.137186525 (407h17m23.176759655s) [noise=true , address=145.79.227.17:4969 , piece_id=MS2QT2XAW7CCC5YXUQ5CFMCH7LVK5C3CEOZN44HCZ5YWSIXMQO3A, seg_pos={0 0}, upload=358133]: Uploading piece... Sep 23 12:45:47.137369544 (407h17m23.176942674s) [noise=true , address=45.91.116.14:4191 , piece_id=JFV37O2MDPIZ4DEHSPJOCDVROQ2S2A4XEOMUM3CTHHK2QBEPJQYA, seg_pos={0 0}, upload=358133]: Uploading piece... Sep 23 12:45:47.137734108 (407h17m23.177307238s) [noise=true , address=77.90.31.16:4733 , piece_id=IMNP5DSWF4CPWT3P6UP2VJVEKT673YNL4SI5QUSX5MEKFFJN3DWA, seg_pos={0 0}, upload=358133]: Uploading piece... Sep 23 12:45:47.139890183 (407h17m23.179463324s) [noise=true , address=145.79.227.15:4876 , piece_id=WBXTTAYGVLBTML5I5IXG5AIBW4XDQ754P2HYWMWWN4MULNC562JQ, seg_pos={0 0}, upload=358133]: Uploading piece... Sep 23 12:45:47.140497660 (407h17m23.18007079s) [noise=true , address=145.79.227.2:4115 , piece_id=BYWPJNIHDC3ZQ2BW62YKMDDVJDNQ3NJ22R3IN367SLIX7FWDQ4JQ, seg_pos={0 0}, upload=358133]: Uploading piece... Sep 23 12:45:47.141101550 (407h17m23.18067468s) [noise=true , address=45.91.116.10:4083 , piece_id=3GXNPYQNMJ2QMCO2WLVFPJUCAOZGXGVHAGSY27YSVHBCGIL7ZWXA, seg_pos={0 0}, upload=358133]: Uploading piece... Sep 23 12:45:47.141613976 (407h17m23.181187126s) [noise=true , address=77.90.31.19:4843 , piece_id=EH7WKV5NTWIKNGYTLKIXZRIKHD7UAXJISMMNWWJNKLOTJGS2GINQ, seg_pos={0 0}, upload=358133]: Uploading piece... Sep 23 12:45:47.142070455 (407h17m23.181643585s) [noise=true , address=45.91.116.6:4974 , piece_id=5UZGQUDVTEOQU542GK7TLCFPWIN3ZB4XLHBPDC3XMDDR6UO7Z5GA, seg_pos={0 0}, upload=358133]: Uploading piece... Sep 23 12:45:48.922402091 (407h17m24.961975221s) [upload=358133]: Next returned nil segment Sep 23 12:45:48.922429062 (407h17m24.962002192s) [upload=358133]: Waiting for error group managing segments... Sep 23 12:45:49.311915921 (407h17m25.351489041s) [noise=true , address=45.91.116.9:4262 , piece_id=ST3YZKNCSKBNBTYMYKQSXUD77ICDKBZQAYE4HFMWRAMV6HXZD22Q, seg_pos={0 0}, upload=358133]: Done uploading piece. err: <nil> Sep 23 12:45:49.343112756 (407h17m25.382685886s) [noise=true , address=77.90.31.19:4843 , piece_id=EH7WKV5NTWIKNGYTLKIXZRIKHD7UAXJISMMNWWJNKLOTJGS2GINQ, seg_pos={0 0}, upload=358133]: Done uploading piece. err: <nil> Sep 23 12:45:49.345575395 (407h17m25.385148525s) [noise=true , address=45.91.116.14:4191 , piece_id=JFV37O2MDPIZ4DEHSPJOCDVROQ2S2A4XEOMUM3CTHHK2QBEPJQYA, seg_pos={0 0}, upload=358133]: Done uploading piece. err: <nil> Sep 23 12:45:49.346103410 (407h17m25.38567654s) [noise=true , address=45.91.116.10:4083 , piece_id=3GXNPYQNMJ2QMCO2WLVFPJUCAOZGXGVHAGSY27YSVHBCGIL7ZWXA, seg_pos={0 0}, upload=358133]: Done uploading piece. err: <nil> Sep 23 12:45:49.352547351 (407h17m25.392120501s) [noise=true , address=77.90.31.16:4733 , piece_id=IMNP5DSWF4CPWT3P6UP2VJVEKT673YNL4SI5QUSX5MEKFFJN3DWA, seg_pos={0 0}, upload=358133]: Done uploading piece. err: <nil> Sep 23 12:45:49.357361267 (407h17m25.396934387s) [noise=true , address=77.90.31.13:4626 , piece_id=N6VTJT2T74RVKVDQLXXX2P74FDW2ILEJWNI2HRK4QNIZ7AUYNUEA, seg_pos={0 0}, upload=358133]: Done uploading piece. err: <nil> Sep 23 12:45:49.358414112 (407h17m25.397987232s) [noise=true , address=45.91.116.11:4119 , piece_id=FSMWXG3WWD4UA4QWOE66MUD2CAM7CJWAHOE476BPM5ZLX24WGG7A, seg_pos={0 0}, upload=358133]: Done uploading piece. err: <nil> Sep 23 12:45:49.362534819 (407h17m25.402107949s) [noise=true , address=45.91.116.11:4118 , piece_id=NFPQYRN6XQLLO5XABYVAUDGP7236SGODYEPFVCL5CTEAE5EGK5BA, seg_pos={0 0}, upload=358133]: Done uploading piece. err: <nil> Sep 23 12:45:49.366463900 (407h17m25.40603703s) [noise=true , address=45.91.116.6:4974 , piece_id=5UZGQUDVTEOQU542GK7TLCFPWIN3ZB4XLHBPDC3XMDDR6UO7Z5GA, seg_pos={0 0}, upload=358133]: Done uploading piece. err: <nil> Sep 23 12:45:49.982739825 (407h17m26.022312955s) [noise=true , address=145.79.227.2:4115 , piece_id=BYWPJNIHDC3ZQ2BW62YKMDDVJDNQ3NJ22R3IN367SLIX7FWDQ4JQ, seg_pos={0 0}, upload=358133]: Done uploading piece. err: <nil> Sep 23 12:45:49.997202504 (407h17m26.036775634s) [noise=true , address=145.79.227.15:4876 , piece_id=WBXTTAYGVLBTML5I5IXG5AIBW4XDQ754P2HYWMWWN4MULNC562JQ, seg_pos={0 0}, upload=358133]: Done uploading piece. err: <nil> Sep 23 12:45:50.032210215 (407h17m26.071783335s) [noise=true , address=145.79.227.7:4295 , piece_id=JYVLPPEMVFUFXAMYNFEIMMWQJLBXL2RRHUVXY55LZULE2JM3UEFA, seg_pos={0 0}, upload=358133]: Done uploading piece. err: <nil> Sep 23 12:45:50.047004215 (407h17m26.086577345s) [noise=true , address=145.79.227.13:4804 , piece_id=V5FFP33A4Z2SZUREHBB35I77AKDSHYL6WDEAK25CE76V3ZCGZ6YQ, seg_pos={0 0}, upload=358133]: Done uploading piece. err: <nil> Sep 23 12:45:50.047042498 (407h17m26.086615628s) [seg_pos={0 0}, upload=358133]: Segment reached optimal threshold of 13 pieces. Sep 23 12:45:50.047208334 (407h17m26.086781464s) [noise=true , address=145.79.227.17:4969 , piece_id=MS2QT2XAW7CCC5YXUQ5CFMCH7LVK5C3CEOZN44HCZ5YWSIXMQO3A, seg_pos={0 0}, upload=358133]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: expected piece hash; context canceled; EOF Sep 23 12:45:50.047309657 (407h17m26.086882787s) [noise=true , address=145.79.227.18:4007 , piece_id=OMVCGFCUBO56VZNIPNPWIGVE2VDGIE74XCMBJCJGCQ45NGKTLVQQ, seg_pos={0 0}, upload=358133]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp): protocol: expected piece hash; context canceled; EOF Sep 23 12:45:50.047666186 (407h17m26.087239306s) [noise=true , address=145.79.227.6:4259 , piece_id=PBQPI25TRRNVIRRV6P4GFXUUNHABPZMCUAYCMMMCLB6SFEBSICYA, seg_pos={0 0}, upload=358133]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y): protocol: write tcp 188.166.225.170:47316->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:47316->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47316->145.79.227.6:4259: use of closed network connection Sep 23 12:45:50.047711582 (407h17m26.087284712s) [seg_pos={0 0}, upload=358133]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 23 12:45:50.047817103 (407h17m26.087390253s) [upload=358133]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 23 12:45:50.047823576 (407h17m26.087396706s) [upload=358133]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 23 12:45:50.337144492 (407h17m26.376717612s) [upload=358133]: Done uploading segments... Sep 23 12:45:50.337169410 (407h17m26.37674254s) [upload=358133]: Upload finished. err: <nil> Duration: 7.73128532s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)