Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 17 out of 60 is failed (threshold=6) [.X.X..XX.X.....X...............X.X.X......X...X..X..XX.XXX..] duration: 5.001424159s trace: https://jaeger.dev.storj.io/trace/7ae4a4e06f2ab150 FAIL (2025-11-21 13:59:10 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001442462s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 5.863178326s trace: https://jaeger.dev.storj.io/trace/2c4b010d23e46d91 FAIL (2025-11-21 13:59:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 24) ======LOGS====== AMRUQ7TS2R6K7ILJ2YSG7UNGO4P5PFTOMHVF6FXOOIWW46Y2NA, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.291418442 (293h31m7.842888914s) [noise=true , address=89.149.201.147:4006 , piece_id=T2KROQWQR5LXKBJEJO6LICLBP5CUBETT3TOACRQCGYXKZ364TYMQ, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.291954664 (293h31m7.843425145s) [noise=true , address=185.167.204.206:4048 , piece_id=6DBNGIBFWN5PPV72YWVRSSQVYF2AZ7EHS55PJDLDLGZEGXRYYHBA, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.293302360 (293h31m7.844772851s) [noise=true , address=89.149.201.147:4021 , piece_id=K3ZPQXTSSUGIGIX33FCYOC6FRCTWDBRVRHUHHGEJLNWFCJFCIT4Q, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.294979363 (293h31m7.846449854s) [noise=true , address=81.171.14.70:4028 , piece_id=ESUHXDH256EM67DXYQ7O5THLHNAPPJ6I3ARZC5D3GSLJD7CNJV5A, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.295760250 (293h31m7.847230711s) [noise=true , address=81.171.14.70:4048 , piece_id=NUU26IUS2OGXAEVLUDEYTXJL4E6GZM3M7MP5BNBI6KTRWQ2RZT4Q, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.300934223 (293h31m7.852404674s) [noise=true , address=81.171.14.70:4031 , piece_id=LJW25GLWHJCV5CFUITJ5CPYDNWOFG3NBK4A5NUX2VVV4CYCDHWMQ, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.301942341 (293h31m7.853412814s) [noise=true , address=185.167.204.205:4011 , piece_id=O2KXUNUHCKAQZSL5BJRRRC6IIZTSXHM74OZ7REQI3RGSMFA3PN6Q, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.302417767 (293h31m7.853888218s) [noise=true , address=185.167.204.205:4020 , piece_id=LPUWKZY6RPB7TGO22BL4YRA7ETB2RHWUFOXPJX7TZE3OHSFXEH7A, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.304119598 (293h31m7.855590059s) [noise=true , address=89.149.201.147:4014 , piece_id=MM2F7NMVCD3AFX2UM7FRGSO5ZD2OTJPXJEUVKJ63IL5L74LEXZSQ, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:10.220344333 (293h31m8.771814794s) [upload=258480]: Next returned nil segment Nov 21 13:35:10.220363840 (293h31m8.771834301s) [upload=258480]: Waiting for error group managing segments... Nov 21 13:35:10.536640979 (293h31m9.08811144s) [noise=true , address=185.167.204.205:4027 , piece_id=7C2DDLUHJQCI62D2YTI76NDABUZRSPK3FIZ7N45OG5GX3SZWCHNA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.537115091 (293h31m9.088585552s) [noise=true , address=185.167.204.205:4020 , piece_id=LPUWKZY6RPB7TGO22BL4YRA7ETB2RHWUFOXPJX7TZE3OHSFXEH7A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.539139225 (293h31m9.090609686s) [noise=true , address=185.167.204.206:4056 , piece_id=7VD7D3HW5YIZUYRZ5LWEB4VDXN2R6OEU7CTCGLKZUVKEC44V743Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.539668604 (293h31m9.091139065s) [noise=true , address=185.167.204.206:4048 , piece_id=6DBNGIBFWN5PPV72YWVRSSQVYF2AZ7EHS55PJDLDLGZEGXRYYHBA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.541238333 (293h31m9.092708794s) [noise=true , address=185.167.204.205:4008 , piece_id=UE3HOAD4KZRUWKY63RDOZ5LQ7VYAQPNV653CNSWCRXD2WL5MR27A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.542375438 (293h31m9.093845919s) [noise=true , address=185.167.204.205:4036 , piece_id=F652JXWWTK5XUJBY5SSU6IKKJE72DGITQXBRYK3LMQEF63DMRIEQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.543153891 (293h31m9.094624342s) [noise=true , address=185.167.204.205:4002 , piece_id=YFXJFZTNYWU3AZXQYDU5VBWJCPRVBCYCANXPQ77UGR5ZBY5VV53Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.543618846 (293h31m9.095089307s) [noise=true , address=185.167.204.206:4047 , piece_id=OHARKEAXGZLVG5XHD6RYKKWS57JFGH4JR2TAFIQSPUXTRMJD4WZA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.543662328 (293h31m9.095132789s) [noise=true , address=185.167.204.206:4066 , piece_id=T5X6MZEPB6P3DZ2HVMXCC4XJX5JJFXKGASJRWJDCK5TBHDEA2OLQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.544259055 (293h31m9.095729516s) [noise=true , address=185.167.204.205:4011 , piece_id=O2KXUNUHCKAQZSL5BJRRRC6IIZTSXHM74OZ7REQI3RGSMFA3PN6Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.544805565 (293h31m9.096276016s) [noise=true , address=185.167.204.206:4060 , piece_id=VJPM3YAQI5N2VTEBJ5GZ75JCCBOQS5IBEFUHEWTOX3BAKSP7TWFQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.545043088 (293h31m9.096513559s) [noise=true , address=185.167.204.205:4007 , piece_id=R6HJTNG5VAAMG2Y6YXK4K4YBQFUSAJQ4774UCOSGJSRDM4RWLICA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.545385921 (293h31m9.096856382s) [noise=true , address=185.167.204.205:4034 , piece_id=H66BDBLG4T2BABKBNKMJLY6QPZT2VVSZC3GW6F5V2L2BBFOWE76Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.759454013 (293h31m10.310924484s) [noise=true , address=89.149.201.147:4006 , piece_id=T2KROQWQR5LXKBJEJO6LICLBP5CUBETT3TOACRQCGYXKZ364TYMQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.779325021 (293h31m10.330795482s) [noise=true , address=89.149.201.147:4015 , piece_id=VIFNKNEVMCI2CFBREZOLVGFJ2X7Y7ZX6FWDC6PO6CLVX2LFW4FYQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.797229894 (293h31m10.348700345s) [noise=true , address=89.149.201.147:4021 , piece_id=K3ZPQXTSSUGIGIX33FCYOC6FRCTWDBRVRHUHHGEJLNWFCJFCIT4Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.804823945 (293h31m10.356294406s) [noise=true , address=89.149.201.147:4012 , piece_id=3QAKWEG37QPAE4AORUSYURK47YIOTM5VHC563QWC43UGGPS7UV7Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.828388066 (293h31m10.379858537s) [noise=true , address=89.149.201.147:4014 , piece_id=MM2F7NMVCD3AFX2UM7FRGSO5ZD2OTJPXJEUVKJ63IL5L74LEXZSQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.848656089 (293h31m10.40012655s) [noise=true , address=89.149.201.147:4001 , piece_id=OGRBX7OEEUPFXWU3EOYD44FGBQ7FQ5CHHZ2RVRHFJGOQ74EK7K5A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:02.180750503 (293h32m0.732220964s) [noise=true , address=81.171.14.70:4028 , piece_id=ESUHXDH256EM67DXYQ7O5THLHNAPPJ6I3ARZC5D3GSLJD7CNJV5A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:02.321725695 (293h32m0.873196156s) [noise=true , address=81.171.14.70:4044 , piece_id=KMR7RCY3AFLFIAJWIQXYC6ZSX7QLL6OKSCQMYHDKKCRX4MZAC4TQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:03.575396105 (293h32m2.126866556s) [noise=true , address=81.171.14.70:4041 , piece_id=N36O2DAKHTDWU4BPERULQHIUWQQDM5TQUQD7Y36XDA7SPPQML2LQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:05.676531843 (293h32m4.228002294s) [noise=true , address=81.171.14.70:4047 , piece_id=Q4N3CMZGTJGYFS35Y75GJHLQLEYMRIX4SSGTJXEHCCXIAX5Y2T3Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:08.329832254 (293h32m6.881302715s) [noise=true , address=81.171.14.70:4043 , piece_id=TQCRSBAQ4AJPCVP2JZYD4HQF375R2K7P5AIMCZW5EJVFJYUGKYGA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:08.628985817 (293h32m7.180456278s) [noise=true , address=81.171.14.70:4042 , piece_id=E5AMRUQ7TS2R6K7ILJ2YSG7UNGO4P5PFTOMHVF6FXOOIWW46Y2NA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12vBozgWFrZfUEQAwsqBt6CDMW47rVjZaDfvKTsCdPrvX8swkhD, address:81.171.14.70:4042): protocol: read tcp 188.166.225.170:40010->81.171.14.70:4042: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 21 13:36:08.629102389 (293h32m7.18057285s) [noise=true , address=81.171.14.70:4048 , piece_id=NUU26IUS2OGXAEVLUDEYTXJL4E6GZM3M7MP5BNBI6KTRWQ2RZT4Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12bbN67ZebXjF4iSyaWu4TDsG1XugTm1YdNHqRdRCbZ2WWRunRG, address:81.171.14.70:4048): protocol: write tcp 188.166.225.170:55932->81.171.14.70:4048: use of closed network connection; write tcp 188.166.225.170:55932->81.171.14.70:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55932->81.171.14.70:4048: use of closed network connection Nov 21 13:36:08.629502702 (293h32m7.180973163s) [noise=true , address=81.171.14.70:4031 , piece_id=LJW25GLWHJCV5CFUITJ5CPYDNWOFG3NBK4A5NUX2VVV4CYCDHWMQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12drdYnnWsuMyFo6LkcewAGGWnorRCAhxPpNqK41LvGy2zkBvng, address:81.171.14.70:4031): protocol: write tcp 188.166.225.170:58232->81.171.14.70:4031: use of closed network connection; write tcp 188.166.225.170:58232->81.171.14.70:4031: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58232->81.171.14.70:4031: use of closed network connection Nov 21 13:36:08.629974720 (293h32m7.181445181s) [noise=true , address=81.171.14.70:4026 , piece_id=ATEKV2WY2WDNBLY7W4THWSO52AJ76Q3XVT7QDGT3EBMPWVRL5EKA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12RTn3BuxasVSPYmjqgyhqKn1t6fAtq9EBtUufXqcLq48My9g2b, address:81.171.14.70:4026): protocol: write tcp 188.166.225.170:41806->81.171.14.70:4026: use of closed network connection; context deadline exceeded Nov 21 13:36:08.630048300 (293h32m7.181518751s) [seg_pos={0 0}, upload=258480]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil> Nov 21 13:36:08.800451489 (293h32m7.35192194s) [upload=258480]: Done uploading segments... Nov 21 13:36:08.800522093 (293h32m7.351992554s) [upload=258480]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24) Duration: 5.863210277s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 17 out of 60 is failed (threshold=6) [..X.X..XX.X.....X...............X.X.X......X...X..X..XX.XXX.] duration: 3.792150558s trace: https://jaeger.dev.storj.io/trace/7ae4a4e06f2ab150 FAIL (2025-11-21 13:58:04 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.792187568s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 6.150042808s trace: https://jaeger.dev.storj.io/trace/2c4b010d23e46d91 FAIL (2025-11-21 13:57:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 24) ======LOGS====== AMRUQ7TS2R6K7ILJ2YSG7UNGO4P5PFTOMHVF6FXOOIWW46Y2NA, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.291418442 (293h31m7.842888914s) [noise=true , address=89.149.201.147:4006 , piece_id=T2KROQWQR5LXKBJEJO6LICLBP5CUBETT3TOACRQCGYXKZ364TYMQ, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.291954664 (293h31m7.843425145s) [noise=true , address=185.167.204.206:4048 , piece_id=6DBNGIBFWN5PPV72YWVRSSQVYF2AZ7EHS55PJDLDLGZEGXRYYHBA, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.293302360 (293h31m7.844772851s) [noise=true , address=89.149.201.147:4021 , piece_id=K3ZPQXTSSUGIGIX33FCYOC6FRCTWDBRVRHUHHGEJLNWFCJFCIT4Q, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.294979363 (293h31m7.846449854s) [noise=true , address=81.171.14.70:4028 , piece_id=ESUHXDH256EM67DXYQ7O5THLHNAPPJ6I3ARZC5D3GSLJD7CNJV5A, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.295760250 (293h31m7.847230711s) [noise=true , address=81.171.14.70:4048 , piece_id=NUU26IUS2OGXAEVLUDEYTXJL4E6GZM3M7MP5BNBI6KTRWQ2RZT4Q, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.300934223 (293h31m7.852404674s) [noise=true , address=81.171.14.70:4031 , piece_id=LJW25GLWHJCV5CFUITJ5CPYDNWOFG3NBK4A5NUX2VVV4CYCDHWMQ, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.301942341 (293h31m7.853412814s) [noise=true , address=185.167.204.205:4011 , piece_id=O2KXUNUHCKAQZSL5BJRRRC6IIZTSXHM74OZ7REQI3RGSMFA3PN6Q, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.302417767 (293h31m7.853888218s) [noise=true , address=185.167.204.205:4020 , piece_id=LPUWKZY6RPB7TGO22BL4YRA7ETB2RHWUFOXPJX7TZE3OHSFXEH7A, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.304119598 (293h31m7.855590059s) [noise=true , address=89.149.201.147:4014 , piece_id=MM2F7NMVCD3AFX2UM7FRGSO5ZD2OTJPXJEUVKJ63IL5L74LEXZSQ, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:10.220344333 (293h31m8.771814794s) [upload=258480]: Next returned nil segment Nov 21 13:35:10.220363840 (293h31m8.771834301s) [upload=258480]: Waiting for error group managing segments... Nov 21 13:35:10.536640979 (293h31m9.08811144s) [noise=true , address=185.167.204.205:4027 , piece_id=7C2DDLUHJQCI62D2YTI76NDABUZRSPK3FIZ7N45OG5GX3SZWCHNA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.537115091 (293h31m9.088585552s) [noise=true , address=185.167.204.205:4020 , piece_id=LPUWKZY6RPB7TGO22BL4YRA7ETB2RHWUFOXPJX7TZE3OHSFXEH7A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.539139225 (293h31m9.090609686s) [noise=true , address=185.167.204.206:4056 , piece_id=7VD7D3HW5YIZUYRZ5LWEB4VDXN2R6OEU7CTCGLKZUVKEC44V743Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.539668604 (293h31m9.091139065s) [noise=true , address=185.167.204.206:4048 , piece_id=6DBNGIBFWN5PPV72YWVRSSQVYF2AZ7EHS55PJDLDLGZEGXRYYHBA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.541238333 (293h31m9.092708794s) [noise=true , address=185.167.204.205:4008 , piece_id=UE3HOAD4KZRUWKY63RDOZ5LQ7VYAQPNV653CNSWCRXD2WL5MR27A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.542375438 (293h31m9.093845919s) [noise=true , address=185.167.204.205:4036 , piece_id=F652JXWWTK5XUJBY5SSU6IKKJE72DGITQXBRYK3LMQEF63DMRIEQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.543153891 (293h31m9.094624342s) [noise=true , address=185.167.204.205:4002 , piece_id=YFXJFZTNYWU3AZXQYDU5VBWJCPRVBCYCANXPQ77UGR5ZBY5VV53Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.543618846 (293h31m9.095089307s) [noise=true , address=185.167.204.206:4047 , piece_id=OHARKEAXGZLVG5XHD6RYKKWS57JFGH4JR2TAFIQSPUXTRMJD4WZA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.543662328 (293h31m9.095132789s) [noise=true , address=185.167.204.206:4066 , piece_id=T5X6MZEPB6P3DZ2HVMXCC4XJX5JJFXKGASJRWJDCK5TBHDEA2OLQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.544259055 (293h31m9.095729516s) [noise=true , address=185.167.204.205:4011 , piece_id=O2KXUNUHCKAQZSL5BJRRRC6IIZTSXHM74OZ7REQI3RGSMFA3PN6Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.544805565 (293h31m9.096276016s) [noise=true , address=185.167.204.206:4060 , piece_id=VJPM3YAQI5N2VTEBJ5GZ75JCCBOQS5IBEFUHEWTOX3BAKSP7TWFQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.545043088 (293h31m9.096513559s) [noise=true , address=185.167.204.205:4007 , piece_id=R6HJTNG5VAAMG2Y6YXK4K4YBQFUSAJQ4774UCOSGJSRDM4RWLICA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.545385921 (293h31m9.096856382s) [noise=true , address=185.167.204.205:4034 , piece_id=H66BDBLG4T2BABKBNKMJLY6QPZT2VVSZC3GW6F5V2L2BBFOWE76Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.759454013 (293h31m10.310924484s) [noise=true , address=89.149.201.147:4006 , piece_id=T2KROQWQR5LXKBJEJO6LICLBP5CUBETT3TOACRQCGYXKZ364TYMQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.779325021 (293h31m10.330795482s) [noise=true , address=89.149.201.147:4015 , piece_id=VIFNKNEVMCI2CFBREZOLVGFJ2X7Y7ZX6FWDC6PO6CLVX2LFW4FYQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.797229894 (293h31m10.348700345s) [noise=true , address=89.149.201.147:4021 , piece_id=K3ZPQXTSSUGIGIX33FCYOC6FRCTWDBRVRHUHHGEJLNWFCJFCIT4Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.804823945 (293h31m10.356294406s) [noise=true , address=89.149.201.147:4012 , piece_id=3QAKWEG37QPAE4AORUSYURK47YIOTM5VHC563QWC43UGGPS7UV7Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.828388066 (293h31m10.379858537s) [noise=true , address=89.149.201.147:4014 , piece_id=MM2F7NMVCD3AFX2UM7FRGSO5ZD2OTJPXJEUVKJ63IL5L74LEXZSQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.848656089 (293h31m10.40012655s) [noise=true , address=89.149.201.147:4001 , piece_id=OGRBX7OEEUPFXWU3EOYD44FGBQ7FQ5CHHZ2RVRHFJGOQ74EK7K5A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:02.180750503 (293h32m0.732220964s) [noise=true , address=81.171.14.70:4028 , piece_id=ESUHXDH256EM67DXYQ7O5THLHNAPPJ6I3ARZC5D3GSLJD7CNJV5A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:02.321725695 (293h32m0.873196156s) [noise=true , address=81.171.14.70:4044 , piece_id=KMR7RCY3AFLFIAJWIQXYC6ZSX7QLL6OKSCQMYHDKKCRX4MZAC4TQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:03.575396105 (293h32m2.126866556s) [noise=true , address=81.171.14.70:4041 , piece_id=N36O2DAKHTDWU4BPERULQHIUWQQDM5TQUQD7Y36XDA7SPPQML2LQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:05.676531843 (293h32m4.228002294s) [noise=true , address=81.171.14.70:4047 , piece_id=Q4N3CMZGTJGYFS35Y75GJHLQLEYMRIX4SSGTJXEHCCXIAX5Y2T3Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:08.329832254 (293h32m6.881302715s) [noise=true , address=81.171.14.70:4043 , piece_id=TQCRSBAQ4AJPCVP2JZYD4HQF375R2K7P5AIMCZW5EJVFJYUGKYGA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:08.628985817 (293h32m7.180456278s) [noise=true , address=81.171.14.70:4042 , piece_id=E5AMRUQ7TS2R6K7ILJ2YSG7UNGO4P5PFTOMHVF6FXOOIWW46Y2NA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12vBozgWFrZfUEQAwsqBt6CDMW47rVjZaDfvKTsCdPrvX8swkhD, address:81.171.14.70:4042): protocol: read tcp 188.166.225.170:40010->81.171.14.70:4042: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 21 13:36:08.629102389 (293h32m7.18057285s) [noise=true , address=81.171.14.70:4048 , piece_id=NUU26IUS2OGXAEVLUDEYTXJL4E6GZM3M7MP5BNBI6KTRWQ2RZT4Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12bbN67ZebXjF4iSyaWu4TDsG1XugTm1YdNHqRdRCbZ2WWRunRG, address:81.171.14.70:4048): protocol: write tcp 188.166.225.170:55932->81.171.14.70:4048: use of closed network connection; write tcp 188.166.225.170:55932->81.171.14.70:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55932->81.171.14.70:4048: use of closed network connection Nov 21 13:36:08.629502702 (293h32m7.180973163s) [noise=true , address=81.171.14.70:4031 , piece_id=LJW25GLWHJCV5CFUITJ5CPYDNWOFG3NBK4A5NUX2VVV4CYCDHWMQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12drdYnnWsuMyFo6LkcewAGGWnorRCAhxPpNqK41LvGy2zkBvng, address:81.171.14.70:4031): protocol: write tcp 188.166.225.170:58232->81.171.14.70:4031: use of closed network connection; write tcp 188.166.225.170:58232->81.171.14.70:4031: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58232->81.171.14.70:4031: use of closed network connection Nov 21 13:36:08.629974720 (293h32m7.181445181s) [noise=true , address=81.171.14.70:4026 , piece_id=ATEKV2WY2WDNBLY7W4THWSO52AJ76Q3XVT7QDGT3EBMPWVRL5EKA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12RTn3BuxasVSPYmjqgyhqKn1t6fAtq9EBtUufXqcLq48My9g2b, address:81.171.14.70:4026): protocol: write tcp 188.166.225.170:41806->81.171.14.70:4026: use of closed network connection; context deadline exceeded Nov 21 13:36:08.630048300 (293h32m7.181518751s) [seg_pos={0 0}, upload=258480]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil> Nov 21 13:36:08.800451489 (293h32m7.35192194s) [upload=258480]: Done uploading segments... Nov 21 13:36:08.800522093 (293h32m7.351992554s) [upload=258480]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24) Duration: 6.150060993s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 18 out of 60 is failed (threshold=6) [X..X.X..XX.X.....X...............X.X.X......X...X..X..XX.XXX] duration: 5.001474643s trace: https://jaeger.dev.storj.io/trace/7ae4a4e06f2ab150 FAIL (2025-11-21 13:57:36 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001499911s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 5.853423972s trace: https://jaeger.dev.storj.io/trace/2c4b010d23e46d91 FAIL (2025-11-21 13:56:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 24) ======LOGS====== AMRUQ7TS2R6K7ILJ2YSG7UNGO4P5PFTOMHVF6FXOOIWW46Y2NA, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.291418442 (293h31m7.842888914s) [noise=true , address=89.149.201.147:4006 , piece_id=T2KROQWQR5LXKBJEJO6LICLBP5CUBETT3TOACRQCGYXKZ364TYMQ, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.291954664 (293h31m7.843425145s) [noise=true , address=185.167.204.206:4048 , piece_id=6DBNGIBFWN5PPV72YWVRSSQVYF2AZ7EHS55PJDLDLGZEGXRYYHBA, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.293302360 (293h31m7.844772851s) [noise=true , address=89.149.201.147:4021 , piece_id=K3ZPQXTSSUGIGIX33FCYOC6FRCTWDBRVRHUHHGEJLNWFCJFCIT4Q, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.294979363 (293h31m7.846449854s) [noise=true , address=81.171.14.70:4028 , piece_id=ESUHXDH256EM67DXYQ7O5THLHNAPPJ6I3ARZC5D3GSLJD7CNJV5A, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.295760250 (293h31m7.847230711s) [noise=true , address=81.171.14.70:4048 , piece_id=NUU26IUS2OGXAEVLUDEYTXJL4E6GZM3M7MP5BNBI6KTRWQ2RZT4Q, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.300934223 (293h31m7.852404674s) [noise=true , address=81.171.14.70:4031 , piece_id=LJW25GLWHJCV5CFUITJ5CPYDNWOFG3NBK4A5NUX2VVV4CYCDHWMQ, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.301942341 (293h31m7.853412814s) [noise=true , address=185.167.204.205:4011 , piece_id=O2KXUNUHCKAQZSL5BJRRRC6IIZTSXHM74OZ7REQI3RGSMFA3PN6Q, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.302417767 (293h31m7.853888218s) [noise=true , address=185.167.204.205:4020 , piece_id=LPUWKZY6RPB7TGO22BL4YRA7ETB2RHWUFOXPJX7TZE3OHSFXEH7A, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.304119598 (293h31m7.855590059s) [noise=true , address=89.149.201.147:4014 , piece_id=MM2F7NMVCD3AFX2UM7FRGSO5ZD2OTJPXJEUVKJ63IL5L74LEXZSQ, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:10.220344333 (293h31m8.771814794s) [upload=258480]: Next returned nil segment Nov 21 13:35:10.220363840 (293h31m8.771834301s) [upload=258480]: Waiting for error group managing segments... Nov 21 13:35:10.536640979 (293h31m9.08811144s) [noise=true , address=185.167.204.205:4027 , piece_id=7C2DDLUHJQCI62D2YTI76NDABUZRSPK3FIZ7N45OG5GX3SZWCHNA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.537115091 (293h31m9.088585552s) [noise=true , address=185.167.204.205:4020 , piece_id=LPUWKZY6RPB7TGO22BL4YRA7ETB2RHWUFOXPJX7TZE3OHSFXEH7A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.539139225 (293h31m9.090609686s) [noise=true , address=185.167.204.206:4056 , piece_id=7VD7D3HW5YIZUYRZ5LWEB4VDXN2R6OEU7CTCGLKZUVKEC44V743Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.539668604 (293h31m9.091139065s) [noise=true , address=185.167.204.206:4048 , piece_id=6DBNGIBFWN5PPV72YWVRSSQVYF2AZ7EHS55PJDLDLGZEGXRYYHBA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.541238333 (293h31m9.092708794s) [noise=true , address=185.167.204.205:4008 , piece_id=UE3HOAD4KZRUWKY63RDOZ5LQ7VYAQPNV653CNSWCRXD2WL5MR27A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.542375438 (293h31m9.093845919s) [noise=true , address=185.167.204.205:4036 , piece_id=F652JXWWTK5XUJBY5SSU6IKKJE72DGITQXBRYK3LMQEF63DMRIEQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.543153891 (293h31m9.094624342s) [noise=true , address=185.167.204.205:4002 , piece_id=YFXJFZTNYWU3AZXQYDU5VBWJCPRVBCYCANXPQ77UGR5ZBY5VV53Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.543618846 (293h31m9.095089307s) [noise=true , address=185.167.204.206:4047 , piece_id=OHARKEAXGZLVG5XHD6RYKKWS57JFGH4JR2TAFIQSPUXTRMJD4WZA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.543662328 (293h31m9.095132789s) [noise=true , address=185.167.204.206:4066 , piece_id=T5X6MZEPB6P3DZ2HVMXCC4XJX5JJFXKGASJRWJDCK5TBHDEA2OLQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.544259055 (293h31m9.095729516s) [noise=true , address=185.167.204.205:4011 , piece_id=O2KXUNUHCKAQZSL5BJRRRC6IIZTSXHM74OZ7REQI3RGSMFA3PN6Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.544805565 (293h31m9.096276016s) [noise=true , address=185.167.204.206:4060 , piece_id=VJPM3YAQI5N2VTEBJ5GZ75JCCBOQS5IBEFUHEWTOX3BAKSP7TWFQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.545043088 (293h31m9.096513559s) [noise=true , address=185.167.204.205:4007 , piece_id=R6HJTNG5VAAMG2Y6YXK4K4YBQFUSAJQ4774UCOSGJSRDM4RWLICA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.545385921 (293h31m9.096856382s) [noise=true , address=185.167.204.205:4034 , piece_id=H66BDBLG4T2BABKBNKMJLY6QPZT2VVSZC3GW6F5V2L2BBFOWE76Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.759454013 (293h31m10.310924484s) [noise=true , address=89.149.201.147:4006 , piece_id=T2KROQWQR5LXKBJEJO6LICLBP5CUBETT3TOACRQCGYXKZ364TYMQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.779325021 (293h31m10.330795482s) [noise=true , address=89.149.201.147:4015 , piece_id=VIFNKNEVMCI2CFBREZOLVGFJ2X7Y7ZX6FWDC6PO6CLVX2LFW4FYQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.797229894 (293h31m10.348700345s) [noise=true , address=89.149.201.147:4021 , piece_id=K3ZPQXTSSUGIGIX33FCYOC6FRCTWDBRVRHUHHGEJLNWFCJFCIT4Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.804823945 (293h31m10.356294406s) [noise=true , address=89.149.201.147:4012 , piece_id=3QAKWEG37QPAE4AORUSYURK47YIOTM5VHC563QWC43UGGPS7UV7Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.828388066 (293h31m10.379858537s) [noise=true , address=89.149.201.147:4014 , piece_id=MM2F7NMVCD3AFX2UM7FRGSO5ZD2OTJPXJEUVKJ63IL5L74LEXZSQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.848656089 (293h31m10.40012655s) [noise=true , address=89.149.201.147:4001 , piece_id=OGRBX7OEEUPFXWU3EOYD44FGBQ7FQ5CHHZ2RVRHFJGOQ74EK7K5A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:02.180750503 (293h32m0.732220964s) [noise=true , address=81.171.14.70:4028 , piece_id=ESUHXDH256EM67DXYQ7O5THLHNAPPJ6I3ARZC5D3GSLJD7CNJV5A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:02.321725695 (293h32m0.873196156s) [noise=true , address=81.171.14.70:4044 , piece_id=KMR7RCY3AFLFIAJWIQXYC6ZSX7QLL6OKSCQMYHDKKCRX4MZAC4TQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:03.575396105 (293h32m2.126866556s) [noise=true , address=81.171.14.70:4041 , piece_id=N36O2DAKHTDWU4BPERULQHIUWQQDM5TQUQD7Y36XDA7SPPQML2LQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:05.676531843 (293h32m4.228002294s) [noise=true , address=81.171.14.70:4047 , piece_id=Q4N3CMZGTJGYFS35Y75GJHLQLEYMRIX4SSGTJXEHCCXIAX5Y2T3Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:08.329832254 (293h32m6.881302715s) [noise=true , address=81.171.14.70:4043 , piece_id=TQCRSBAQ4AJPCVP2JZYD4HQF375R2K7P5AIMCZW5EJVFJYUGKYGA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:08.628985817 (293h32m7.180456278s) [noise=true , address=81.171.14.70:4042 , piece_id=E5AMRUQ7TS2R6K7ILJ2YSG7UNGO4P5PFTOMHVF6FXOOIWW46Y2NA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12vBozgWFrZfUEQAwsqBt6CDMW47rVjZaDfvKTsCdPrvX8swkhD, address:81.171.14.70:4042): protocol: read tcp 188.166.225.170:40010->81.171.14.70:4042: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 21 13:36:08.629102389 (293h32m7.18057285s) [noise=true , address=81.171.14.70:4048 , piece_id=NUU26IUS2OGXAEVLUDEYTXJL4E6GZM3M7MP5BNBI6KTRWQ2RZT4Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12bbN67ZebXjF4iSyaWu4TDsG1XugTm1YdNHqRdRCbZ2WWRunRG, address:81.171.14.70:4048): protocol: write tcp 188.166.225.170:55932->81.171.14.70:4048: use of closed network connection; write tcp 188.166.225.170:55932->81.171.14.70:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55932->81.171.14.70:4048: use of closed network connection Nov 21 13:36:08.629502702 (293h32m7.180973163s) [noise=true , address=81.171.14.70:4031 , piece_id=LJW25GLWHJCV5CFUITJ5CPYDNWOFG3NBK4A5NUX2VVV4CYCDHWMQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12drdYnnWsuMyFo6LkcewAGGWnorRCAhxPpNqK41LvGy2zkBvng, address:81.171.14.70:4031): protocol: write tcp 188.166.225.170:58232->81.171.14.70:4031: use of closed network connection; write tcp 188.166.225.170:58232->81.171.14.70:4031: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58232->81.171.14.70:4031: use of closed network connection Nov 21 13:36:08.629974720 (293h32m7.181445181s) [noise=true , address=81.171.14.70:4026 , piece_id=ATEKV2WY2WDNBLY7W4THWSO52AJ76Q3XVT7QDGT3EBMPWVRL5EKA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12RTn3BuxasVSPYmjqgyhqKn1t6fAtq9EBtUufXqcLq48My9g2b, address:81.171.14.70:4026): protocol: write tcp 188.166.225.170:41806->81.171.14.70:4026: use of closed network connection; context deadline exceeded Nov 21 13:36:08.630048300 (293h32m7.181518751s) [seg_pos={0 0}, upload=258480]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil> Nov 21 13:36:08.800451489 (293h32m7.35192194s) [upload=258480]: Done uploading segments... Nov 21 13:36:08.800522093 (293h32m7.351992554s) [upload=258480]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24) Duration: 5.853450682s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 5.974136232s trace: https://jaeger.dev.storj.io/trace/2c4b010d23e46d91 FAIL (2025-11-21 13:56:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 24) ======LOGS====== AMRUQ7TS2R6K7ILJ2YSG7UNGO4P5PFTOMHVF6FXOOIWW46Y2NA, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.291418442 (293h31m7.842888914s) [noise=true , address=89.149.201.147:4006 , piece_id=T2KROQWQR5LXKBJEJO6LICLBP5CUBETT3TOACRQCGYXKZ364TYMQ, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.291954664 (293h31m7.843425145s) [noise=true , address=185.167.204.206:4048 , piece_id=6DBNGIBFWN5PPV72YWVRSSQVYF2AZ7EHS55PJDLDLGZEGXRYYHBA, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.293302360 (293h31m7.844772851s) [noise=true , address=89.149.201.147:4021 , piece_id=K3ZPQXTSSUGIGIX33FCYOC6FRCTWDBRVRHUHHGEJLNWFCJFCIT4Q, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.294979363 (293h31m7.846449854s) [noise=true , address=81.171.14.70:4028 , piece_id=ESUHXDH256EM67DXYQ7O5THLHNAPPJ6I3ARZC5D3GSLJD7CNJV5A, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.295760250 (293h31m7.847230711s) [noise=true , address=81.171.14.70:4048 , piece_id=NUU26IUS2OGXAEVLUDEYTXJL4E6GZM3M7MP5BNBI6KTRWQ2RZT4Q, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.300934223 (293h31m7.852404674s) [noise=true , address=81.171.14.70:4031 , piece_id=LJW25GLWHJCV5CFUITJ5CPYDNWOFG3NBK4A5NUX2VVV4CYCDHWMQ, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.301942341 (293h31m7.853412814s) [noise=true , address=185.167.204.205:4011 , piece_id=O2KXUNUHCKAQZSL5BJRRRC6IIZTSXHM74OZ7REQI3RGSMFA3PN6Q, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.302417767 (293h31m7.853888218s) [noise=true , address=185.167.204.205:4020 , piece_id=LPUWKZY6RPB7TGO22BL4YRA7ETB2RHWUFOXPJX7TZE3OHSFXEH7A, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.304119598 (293h31m7.855590059s) [noise=true , address=89.149.201.147:4014 , piece_id=MM2F7NMVCD3AFX2UM7FRGSO5ZD2OTJPXJEUVKJ63IL5L74LEXZSQ, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:10.220344333 (293h31m8.771814794s) [upload=258480]: Next returned nil segment Nov 21 13:35:10.220363840 (293h31m8.771834301s) [upload=258480]: Waiting for error group managing segments... Nov 21 13:35:10.536640979 (293h31m9.08811144s) [noise=true , address=185.167.204.205:4027 , piece_id=7C2DDLUHJQCI62D2YTI76NDABUZRSPK3FIZ7N45OG5GX3SZWCHNA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.537115091 (293h31m9.088585552s) [noise=true , address=185.167.204.205:4020 , piece_id=LPUWKZY6RPB7TGO22BL4YRA7ETB2RHWUFOXPJX7TZE3OHSFXEH7A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.539139225 (293h31m9.090609686s) [noise=true , address=185.167.204.206:4056 , piece_id=7VD7D3HW5YIZUYRZ5LWEB4VDXN2R6OEU7CTCGLKZUVKEC44V743Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.539668604 (293h31m9.091139065s) [noise=true , address=185.167.204.206:4048 , piece_id=6DBNGIBFWN5PPV72YWVRSSQVYF2AZ7EHS55PJDLDLGZEGXRYYHBA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.541238333 (293h31m9.092708794s) [noise=true , address=185.167.204.205:4008 , piece_id=UE3HOAD4KZRUWKY63RDOZ5LQ7VYAQPNV653CNSWCRXD2WL5MR27A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.542375438 (293h31m9.093845919s) [noise=true , address=185.167.204.205:4036 , piece_id=F652JXWWTK5XUJBY5SSU6IKKJE72DGITQXBRYK3LMQEF63DMRIEQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.543153891 (293h31m9.094624342s) [noise=true , address=185.167.204.205:4002 , piece_id=YFXJFZTNYWU3AZXQYDU5VBWJCPRVBCYCANXPQ77UGR5ZBY5VV53Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.543618846 (293h31m9.095089307s) [noise=true , address=185.167.204.206:4047 , piece_id=OHARKEAXGZLVG5XHD6RYKKWS57JFGH4JR2TAFIQSPUXTRMJD4WZA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.543662328 (293h31m9.095132789s) [noise=true , address=185.167.204.206:4066 , piece_id=T5X6MZEPB6P3DZ2HVMXCC4XJX5JJFXKGASJRWJDCK5TBHDEA2OLQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.544259055 (293h31m9.095729516s) [noise=true , address=185.167.204.205:4011 , piece_id=O2KXUNUHCKAQZSL5BJRRRC6IIZTSXHM74OZ7REQI3RGSMFA3PN6Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.544805565 (293h31m9.096276016s) [noise=true , address=185.167.204.206:4060 , piece_id=VJPM3YAQI5N2VTEBJ5GZ75JCCBOQS5IBEFUHEWTOX3BAKSP7TWFQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.545043088 (293h31m9.096513559s) [noise=true , address=185.167.204.205:4007 , piece_id=R6HJTNG5VAAMG2Y6YXK4K4YBQFUSAJQ4774UCOSGJSRDM4RWLICA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.545385921 (293h31m9.096856382s) [noise=true , address=185.167.204.205:4034 , piece_id=H66BDBLG4T2BABKBNKMJLY6QPZT2VVSZC3GW6F5V2L2BBFOWE76Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.759454013 (293h31m10.310924484s) [noise=true , address=89.149.201.147:4006 , piece_id=T2KROQWQR5LXKBJEJO6LICLBP5CUBETT3TOACRQCGYXKZ364TYMQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.779325021 (293h31m10.330795482s) [noise=true , address=89.149.201.147:4015 , piece_id=VIFNKNEVMCI2CFBREZOLVGFJ2X7Y7ZX6FWDC6PO6CLVX2LFW4FYQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.797229894 (293h31m10.348700345s) [noise=true , address=89.149.201.147:4021 , piece_id=K3ZPQXTSSUGIGIX33FCYOC6FRCTWDBRVRHUHHGEJLNWFCJFCIT4Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.804823945 (293h31m10.356294406s) [noise=true , address=89.149.201.147:4012 , piece_id=3QAKWEG37QPAE4AORUSYURK47YIOTM5VHC563QWC43UGGPS7UV7Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.828388066 (293h31m10.379858537s) [noise=true , address=89.149.201.147:4014 , piece_id=MM2F7NMVCD3AFX2UM7FRGSO5ZD2OTJPXJEUVKJ63IL5L74LEXZSQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.848656089 (293h31m10.40012655s) [noise=true , address=89.149.201.147:4001 , piece_id=OGRBX7OEEUPFXWU3EOYD44FGBQ7FQ5CHHZ2RVRHFJGOQ74EK7K5A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:02.180750503 (293h32m0.732220964s) [noise=true , address=81.171.14.70:4028 , piece_id=ESUHXDH256EM67DXYQ7O5THLHNAPPJ6I3ARZC5D3GSLJD7CNJV5A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:02.321725695 (293h32m0.873196156s) [noise=true , address=81.171.14.70:4044 , piece_id=KMR7RCY3AFLFIAJWIQXYC6ZSX7QLL6OKSCQMYHDKKCRX4MZAC4TQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:03.575396105 (293h32m2.126866556s) [noise=true , address=81.171.14.70:4041 , piece_id=N36O2DAKHTDWU4BPERULQHIUWQQDM5TQUQD7Y36XDA7SPPQML2LQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:05.676531843 (293h32m4.228002294s) [noise=true , address=81.171.14.70:4047 , piece_id=Q4N3CMZGTJGYFS35Y75GJHLQLEYMRIX4SSGTJXEHCCXIAX5Y2T3Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:08.329832254 (293h32m6.881302715s) [noise=true , address=81.171.14.70:4043 , piece_id=TQCRSBAQ4AJPCVP2JZYD4HQF375R2K7P5AIMCZW5EJVFJYUGKYGA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:08.628985817 (293h32m7.180456278s) [noise=true , address=81.171.14.70:4042 , piece_id=E5AMRUQ7TS2R6K7ILJ2YSG7UNGO4P5PFTOMHVF6FXOOIWW46Y2NA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12vBozgWFrZfUEQAwsqBt6CDMW47rVjZaDfvKTsCdPrvX8swkhD, address:81.171.14.70:4042): protocol: read tcp 188.166.225.170:40010->81.171.14.70:4042: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 21 13:36:08.629102389 (293h32m7.18057285s) [noise=true , address=81.171.14.70:4048 , piece_id=NUU26IUS2OGXAEVLUDEYTXJL4E6GZM3M7MP5BNBI6KTRWQ2RZT4Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12bbN67ZebXjF4iSyaWu4TDsG1XugTm1YdNHqRdRCbZ2WWRunRG, address:81.171.14.70:4048): protocol: write tcp 188.166.225.170:55932->81.171.14.70:4048: use of closed network connection; write tcp 188.166.225.170:55932->81.171.14.70:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55932->81.171.14.70:4048: use of closed network connection Nov 21 13:36:08.629502702 (293h32m7.180973163s) [noise=true , address=81.171.14.70:4031 , piece_id=LJW25GLWHJCV5CFUITJ5CPYDNWOFG3NBK4A5NUX2VVV4CYCDHWMQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12drdYnnWsuMyFo6LkcewAGGWnorRCAhxPpNqK41LvGy2zkBvng, address:81.171.14.70:4031): protocol: write tcp 188.166.225.170:58232->81.171.14.70:4031: use of closed network connection; write tcp 188.166.225.170:58232->81.171.14.70:4031: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58232->81.171.14.70:4031: use of closed network connection Nov 21 13:36:08.629974720 (293h32m7.181445181s) [noise=true , address=81.171.14.70:4026 , piece_id=ATEKV2WY2WDNBLY7W4THWSO52AJ76Q3XVT7QDGT3EBMPWVRL5EKA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12RTn3BuxasVSPYmjqgyhqKn1t6fAtq9EBtUufXqcLq48My9g2b, address:81.171.14.70:4026): protocol: write tcp 188.166.225.170:41806->81.171.14.70:4026: use of closed network connection; context deadline exceeded Nov 21 13:36:08.630048300 (293h32m7.181518751s) [seg_pos={0 0}, upload=258480]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil> Nov 21 13:36:08.800451489 (293h32m7.35192194s) [upload=258480]: Done uploading segments... Nov 21 13:36:08.800522093 (293h32m7.351992554s) [upload=258480]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24) Duration: 5.974164215s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 17 out of 60 is failed (threshold=6) [.X..X.X..XX.X.....X...............X.X.X......X...X..X..XX.XX] duration: 5.000895351s trace: https://jaeger.dev.storj.io/trace/7e3ed855c79dad80 FAIL (2025-11-21 13:56:23 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000922653s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 5.896192991s trace: https://jaeger.dev.storj.io/trace/2c4b010d23e46d91 FAIL (2025-11-21 13:55:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 24) ======LOGS====== AMRUQ7TS2R6K7ILJ2YSG7UNGO4P5PFTOMHVF6FXOOIWW46Y2NA, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.291418442 (293h31m7.842888914s) [noise=true , address=89.149.201.147:4006 , piece_id=T2KROQWQR5LXKBJEJO6LICLBP5CUBETT3TOACRQCGYXKZ364TYMQ, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.291954664 (293h31m7.843425145s) [noise=true , address=185.167.204.206:4048 , piece_id=6DBNGIBFWN5PPV72YWVRSSQVYF2AZ7EHS55PJDLDLGZEGXRYYHBA, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.293302360 (293h31m7.844772851s) [noise=true , address=89.149.201.147:4021 , piece_id=K3ZPQXTSSUGIGIX33FCYOC6FRCTWDBRVRHUHHGEJLNWFCJFCIT4Q, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.294979363 (293h31m7.846449854s) [noise=true , address=81.171.14.70:4028 , piece_id=ESUHXDH256EM67DXYQ7O5THLHNAPPJ6I3ARZC5D3GSLJD7CNJV5A, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.295760250 (293h31m7.847230711s) [noise=true , address=81.171.14.70:4048 , piece_id=NUU26IUS2OGXAEVLUDEYTXJL4E6GZM3M7MP5BNBI6KTRWQ2RZT4Q, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.300934223 (293h31m7.852404674s) [noise=true , address=81.171.14.70:4031 , piece_id=LJW25GLWHJCV5CFUITJ5CPYDNWOFG3NBK4A5NUX2VVV4CYCDHWMQ, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.301942341 (293h31m7.853412814s) [noise=true , address=185.167.204.205:4011 , piece_id=O2KXUNUHCKAQZSL5BJRRRC6IIZTSXHM74OZ7REQI3RGSMFA3PN6Q, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.302417767 (293h31m7.853888218s) [noise=true , address=185.167.204.205:4020 , piece_id=LPUWKZY6RPB7TGO22BL4YRA7ETB2RHWUFOXPJX7TZE3OHSFXEH7A, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.304119598 (293h31m7.855590059s) [noise=true , address=89.149.201.147:4014 , piece_id=MM2F7NMVCD3AFX2UM7FRGSO5ZD2OTJPXJEUVKJ63IL5L74LEXZSQ, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:10.220344333 (293h31m8.771814794s) [upload=258480]: Next returned nil segment Nov 21 13:35:10.220363840 (293h31m8.771834301s) [upload=258480]: Waiting for error group managing segments... Nov 21 13:35:10.536640979 (293h31m9.08811144s) [noise=true , address=185.167.204.205:4027 , piece_id=7C2DDLUHJQCI62D2YTI76NDABUZRSPK3FIZ7N45OG5GX3SZWCHNA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.537115091 (293h31m9.088585552s) [noise=true , address=185.167.204.205:4020 , piece_id=LPUWKZY6RPB7TGO22BL4YRA7ETB2RHWUFOXPJX7TZE3OHSFXEH7A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.539139225 (293h31m9.090609686s) [noise=true , address=185.167.204.206:4056 , piece_id=7VD7D3HW5YIZUYRZ5LWEB4VDXN2R6OEU7CTCGLKZUVKEC44V743Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.539668604 (293h31m9.091139065s) [noise=true , address=185.167.204.206:4048 , piece_id=6DBNGIBFWN5PPV72YWVRSSQVYF2AZ7EHS55PJDLDLGZEGXRYYHBA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.541238333 (293h31m9.092708794s) [noise=true , address=185.167.204.205:4008 , piece_id=UE3HOAD4KZRUWKY63RDOZ5LQ7VYAQPNV653CNSWCRXD2WL5MR27A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.542375438 (293h31m9.093845919s) [noise=true , address=185.167.204.205:4036 , piece_id=F652JXWWTK5XUJBY5SSU6IKKJE72DGITQXBRYK3LMQEF63DMRIEQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.543153891 (293h31m9.094624342s) [noise=true , address=185.167.204.205:4002 , piece_id=YFXJFZTNYWU3AZXQYDU5VBWJCPRVBCYCANXPQ77UGR5ZBY5VV53Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.543618846 (293h31m9.095089307s) [noise=true , address=185.167.204.206:4047 , piece_id=OHARKEAXGZLVG5XHD6RYKKWS57JFGH4JR2TAFIQSPUXTRMJD4WZA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.543662328 (293h31m9.095132789s) [noise=true , address=185.167.204.206:4066 , piece_id=T5X6MZEPB6P3DZ2HVMXCC4XJX5JJFXKGASJRWJDCK5TBHDEA2OLQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.544259055 (293h31m9.095729516s) [noise=true , address=185.167.204.205:4011 , piece_id=O2KXUNUHCKAQZSL5BJRRRC6IIZTSXHM74OZ7REQI3RGSMFA3PN6Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.544805565 (293h31m9.096276016s) [noise=true , address=185.167.204.206:4060 , piece_id=VJPM3YAQI5N2VTEBJ5GZ75JCCBOQS5IBEFUHEWTOX3BAKSP7TWFQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.545043088 (293h31m9.096513559s) [noise=true , address=185.167.204.205:4007 , piece_id=R6HJTNG5VAAMG2Y6YXK4K4YBQFUSAJQ4774UCOSGJSRDM4RWLICA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.545385921 (293h31m9.096856382s) [noise=true , address=185.167.204.205:4034 , piece_id=H66BDBLG4T2BABKBNKMJLY6QPZT2VVSZC3GW6F5V2L2BBFOWE76Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.759454013 (293h31m10.310924484s) [noise=true , address=89.149.201.147:4006 , piece_id=T2KROQWQR5LXKBJEJO6LICLBP5CUBETT3TOACRQCGYXKZ364TYMQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.779325021 (293h31m10.330795482s) [noise=true , address=89.149.201.147:4015 , piece_id=VIFNKNEVMCI2CFBREZOLVGFJ2X7Y7ZX6FWDC6PO6CLVX2LFW4FYQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.797229894 (293h31m10.348700345s) [noise=true , address=89.149.201.147:4021 , piece_id=K3ZPQXTSSUGIGIX33FCYOC6FRCTWDBRVRHUHHGEJLNWFCJFCIT4Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.804823945 (293h31m10.356294406s) [noise=true , address=89.149.201.147:4012 , piece_id=3QAKWEG37QPAE4AORUSYURK47YIOTM5VHC563QWC43UGGPS7UV7Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.828388066 (293h31m10.379858537s) [noise=true , address=89.149.201.147:4014 , piece_id=MM2F7NMVCD3AFX2UM7FRGSO5ZD2OTJPXJEUVKJ63IL5L74LEXZSQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.848656089 (293h31m10.40012655s) [noise=true , address=89.149.201.147:4001 , piece_id=OGRBX7OEEUPFXWU3EOYD44FGBQ7FQ5CHHZ2RVRHFJGOQ74EK7K5A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:02.180750503 (293h32m0.732220964s) [noise=true , address=81.171.14.70:4028 , piece_id=ESUHXDH256EM67DXYQ7O5THLHNAPPJ6I3ARZC5D3GSLJD7CNJV5A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:02.321725695 (293h32m0.873196156s) [noise=true , address=81.171.14.70:4044 , piece_id=KMR7RCY3AFLFIAJWIQXYC6ZSX7QLL6OKSCQMYHDKKCRX4MZAC4TQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:03.575396105 (293h32m2.126866556s) [noise=true , address=81.171.14.70:4041 , piece_id=N36O2DAKHTDWU4BPERULQHIUWQQDM5TQUQD7Y36XDA7SPPQML2LQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:05.676531843 (293h32m4.228002294s) [noise=true , address=81.171.14.70:4047 , piece_id=Q4N3CMZGTJGYFS35Y75GJHLQLEYMRIX4SSGTJXEHCCXIAX5Y2T3Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:08.329832254 (293h32m6.881302715s) [noise=true , address=81.171.14.70:4043 , piece_id=TQCRSBAQ4AJPCVP2JZYD4HQF375R2K7P5AIMCZW5EJVFJYUGKYGA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:08.628985817 (293h32m7.180456278s) [noise=true , address=81.171.14.70:4042 , piece_id=E5AMRUQ7TS2R6K7ILJ2YSG7UNGO4P5PFTOMHVF6FXOOIWW46Y2NA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12vBozgWFrZfUEQAwsqBt6CDMW47rVjZaDfvKTsCdPrvX8swkhD, address:81.171.14.70:4042): protocol: read tcp 188.166.225.170:40010->81.171.14.70:4042: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 21 13:36:08.629102389 (293h32m7.18057285s) [noise=true , address=81.171.14.70:4048 , piece_id=NUU26IUS2OGXAEVLUDEYTXJL4E6GZM3M7MP5BNBI6KTRWQ2RZT4Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12bbN67ZebXjF4iSyaWu4TDsG1XugTm1YdNHqRdRCbZ2WWRunRG, address:81.171.14.70:4048): protocol: write tcp 188.166.225.170:55932->81.171.14.70:4048: use of closed network connection; write tcp 188.166.225.170:55932->81.171.14.70:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55932->81.171.14.70:4048: use of closed network connection Nov 21 13:36:08.629502702 (293h32m7.180973163s) [noise=true , address=81.171.14.70:4031 , piece_id=LJW25GLWHJCV5CFUITJ5CPYDNWOFG3NBK4A5NUX2VVV4CYCDHWMQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12drdYnnWsuMyFo6LkcewAGGWnorRCAhxPpNqK41LvGy2zkBvng, address:81.171.14.70:4031): protocol: write tcp 188.166.225.170:58232->81.171.14.70:4031: use of closed network connection; write tcp 188.166.225.170:58232->81.171.14.70:4031: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58232->81.171.14.70:4031: use of closed network connection Nov 21 13:36:08.629974720 (293h32m7.181445181s) [noise=true , address=81.171.14.70:4026 , piece_id=ATEKV2WY2WDNBLY7W4THWSO52AJ76Q3XVT7QDGT3EBMPWVRL5EKA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12RTn3BuxasVSPYmjqgyhqKn1t6fAtq9EBtUufXqcLq48My9g2b, address:81.171.14.70:4026): protocol: write tcp 188.166.225.170:41806->81.171.14.70:4026: use of closed network connection; context deadline exceeded Nov 21 13:36:08.630048300 (293h32m7.181518751s) [seg_pos={0 0}, upload=258480]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil> Nov 21 13:36:08.800451489 (293h32m7.35192194s) [upload=258480]: Done uploading segments... Nov 21 13:36:08.800522093 (293h32m7.351992554s) [upload=258480]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24) Duration: 5.896225331s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 17 out of 60 is failed (threshold=6) [X.X..X.X..XX.X.....X...............X.X.X......X...X..X..XX.X] duration: 5.001038363s trace: https://jaeger.dev.storj.io/trace/51a9decb02bf13a0 FAIL (2025-11-21 13:55:28 UTC) orbiter warning: context deadline exceeded Duration: 5.001064673s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 5.841366144s trace: https://jaeger.dev.storj.io/trace/2c4b010d23e46d91 FAIL (2025-11-21 13:54:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 24) ======LOGS====== AMRUQ7TS2R6K7ILJ2YSG7UNGO4P5PFTOMHVF6FXOOIWW46Y2NA, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.291418442 (293h31m7.842888914s) [noise=true , address=89.149.201.147:4006 , piece_id=T2KROQWQR5LXKBJEJO6LICLBP5CUBETT3TOACRQCGYXKZ364TYMQ, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.291954664 (293h31m7.843425145s) [noise=true , address=185.167.204.206:4048 , piece_id=6DBNGIBFWN5PPV72YWVRSSQVYF2AZ7EHS55PJDLDLGZEGXRYYHBA, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.293302360 (293h31m7.844772851s) [noise=true , address=89.149.201.147:4021 , piece_id=K3ZPQXTSSUGIGIX33FCYOC6FRCTWDBRVRHUHHGEJLNWFCJFCIT4Q, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.294979363 (293h31m7.846449854s) [noise=true , address=81.171.14.70:4028 , piece_id=ESUHXDH256EM67DXYQ7O5THLHNAPPJ6I3ARZC5D3GSLJD7CNJV5A, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.295760250 (293h31m7.847230711s) [noise=true , address=81.171.14.70:4048 , piece_id=NUU26IUS2OGXAEVLUDEYTXJL4E6GZM3M7MP5BNBI6KTRWQ2RZT4Q, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.300934223 (293h31m7.852404674s) [noise=true , address=81.171.14.70:4031 , piece_id=LJW25GLWHJCV5CFUITJ5CPYDNWOFG3NBK4A5NUX2VVV4CYCDHWMQ, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.301942341 (293h31m7.853412814s) [noise=true , address=185.167.204.205:4011 , piece_id=O2KXUNUHCKAQZSL5BJRRRC6IIZTSXHM74OZ7REQI3RGSMFA3PN6Q, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.302417767 (293h31m7.853888218s) [noise=true , address=185.167.204.205:4020 , piece_id=LPUWKZY6RPB7TGO22BL4YRA7ETB2RHWUFOXPJX7TZE3OHSFXEH7A, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.304119598 (293h31m7.855590059s) [noise=true , address=89.149.201.147:4014 , piece_id=MM2F7NMVCD3AFX2UM7FRGSO5ZD2OTJPXJEUVKJ63IL5L74LEXZSQ, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:10.220344333 (293h31m8.771814794s) [upload=258480]: Next returned nil segment Nov 21 13:35:10.220363840 (293h31m8.771834301s) [upload=258480]: Waiting for error group managing segments... Nov 21 13:35:10.536640979 (293h31m9.08811144s) [noise=true , address=185.167.204.205:4027 , piece_id=7C2DDLUHJQCI62D2YTI76NDABUZRSPK3FIZ7N45OG5GX3SZWCHNA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.537115091 (293h31m9.088585552s) [noise=true , address=185.167.204.205:4020 , piece_id=LPUWKZY6RPB7TGO22BL4YRA7ETB2RHWUFOXPJX7TZE3OHSFXEH7A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.539139225 (293h31m9.090609686s) [noise=true , address=185.167.204.206:4056 , piece_id=7VD7D3HW5YIZUYRZ5LWEB4VDXN2R6OEU7CTCGLKZUVKEC44V743Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.539668604 (293h31m9.091139065s) [noise=true , address=185.167.204.206:4048 , piece_id=6DBNGIBFWN5PPV72YWVRSSQVYF2AZ7EHS55PJDLDLGZEGXRYYHBA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.541238333 (293h31m9.092708794s) [noise=true , address=185.167.204.205:4008 , piece_id=UE3HOAD4KZRUWKY63RDOZ5LQ7VYAQPNV653CNSWCRXD2WL5MR27A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.542375438 (293h31m9.093845919s) [noise=true , address=185.167.204.205:4036 , piece_id=F652JXWWTK5XUJBY5SSU6IKKJE72DGITQXBRYK3LMQEF63DMRIEQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.543153891 (293h31m9.094624342s) [noise=true , address=185.167.204.205:4002 , piece_id=YFXJFZTNYWU3AZXQYDU5VBWJCPRVBCYCANXPQ77UGR5ZBY5VV53Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.543618846 (293h31m9.095089307s) [noise=true , address=185.167.204.206:4047 , piece_id=OHARKEAXGZLVG5XHD6RYKKWS57JFGH4JR2TAFIQSPUXTRMJD4WZA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.543662328 (293h31m9.095132789s) [noise=true , address=185.167.204.206:4066 , piece_id=T5X6MZEPB6P3DZ2HVMXCC4XJX5JJFXKGASJRWJDCK5TBHDEA2OLQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.544259055 (293h31m9.095729516s) [noise=true , address=185.167.204.205:4011 , piece_id=O2KXUNUHCKAQZSL5BJRRRC6IIZTSXHM74OZ7REQI3RGSMFA3PN6Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.544805565 (293h31m9.096276016s) [noise=true , address=185.167.204.206:4060 , piece_id=VJPM3YAQI5N2VTEBJ5GZ75JCCBOQS5IBEFUHEWTOX3BAKSP7TWFQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.545043088 (293h31m9.096513559s) [noise=true , address=185.167.204.205:4007 , piece_id=R6HJTNG5VAAMG2Y6YXK4K4YBQFUSAJQ4774UCOSGJSRDM4RWLICA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.545385921 (293h31m9.096856382s) [noise=true , address=185.167.204.205:4034 , piece_id=H66BDBLG4T2BABKBNKMJLY6QPZT2VVSZC3GW6F5V2L2BBFOWE76Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.759454013 (293h31m10.310924484s) [noise=true , address=89.149.201.147:4006 , piece_id=T2KROQWQR5LXKBJEJO6LICLBP5CUBETT3TOACRQCGYXKZ364TYMQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.779325021 (293h31m10.330795482s) [noise=true , address=89.149.201.147:4015 , piece_id=VIFNKNEVMCI2CFBREZOLVGFJ2X7Y7ZX6FWDC6PO6CLVX2LFW4FYQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.797229894 (293h31m10.348700345s) [noise=true , address=89.149.201.147:4021 , piece_id=K3ZPQXTSSUGIGIX33FCYOC6FRCTWDBRVRHUHHGEJLNWFCJFCIT4Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.804823945 (293h31m10.356294406s) [noise=true , address=89.149.201.147:4012 , piece_id=3QAKWEG37QPAE4AORUSYURK47YIOTM5VHC563QWC43UGGPS7UV7Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.828388066 (293h31m10.379858537s) [noise=true , address=89.149.201.147:4014 , piece_id=MM2F7NMVCD3AFX2UM7FRGSO5ZD2OTJPXJEUVKJ63IL5L74LEXZSQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.848656089 (293h31m10.40012655s) [noise=true , address=89.149.201.147:4001 , piece_id=OGRBX7OEEUPFXWU3EOYD44FGBQ7FQ5CHHZ2RVRHFJGOQ74EK7K5A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:02.180750503 (293h32m0.732220964s) [noise=true , address=81.171.14.70:4028 , piece_id=ESUHXDH256EM67DXYQ7O5THLHNAPPJ6I3ARZC5D3GSLJD7CNJV5A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:02.321725695 (293h32m0.873196156s) [noise=true , address=81.171.14.70:4044 , piece_id=KMR7RCY3AFLFIAJWIQXYC6ZSX7QLL6OKSCQMYHDKKCRX4MZAC4TQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:03.575396105 (293h32m2.126866556s) [noise=true , address=81.171.14.70:4041 , piece_id=N36O2DAKHTDWU4BPERULQHIUWQQDM5TQUQD7Y36XDA7SPPQML2LQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:05.676531843 (293h32m4.228002294s) [noise=true , address=81.171.14.70:4047 , piece_id=Q4N3CMZGTJGYFS35Y75GJHLQLEYMRIX4SSGTJXEHCCXIAX5Y2T3Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:08.329832254 (293h32m6.881302715s) [noise=true , address=81.171.14.70:4043 , piece_id=TQCRSBAQ4AJPCVP2JZYD4HQF375R2K7P5AIMCZW5EJVFJYUGKYGA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:08.628985817 (293h32m7.180456278s) [noise=true , address=81.171.14.70:4042 , piece_id=E5AMRUQ7TS2R6K7ILJ2YSG7UNGO4P5PFTOMHVF6FXOOIWW46Y2NA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12vBozgWFrZfUEQAwsqBt6CDMW47rVjZaDfvKTsCdPrvX8swkhD, address:81.171.14.70:4042): protocol: read tcp 188.166.225.170:40010->81.171.14.70:4042: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 21 13:36:08.629102389 (293h32m7.18057285s) [noise=true , address=81.171.14.70:4048 , piece_id=NUU26IUS2OGXAEVLUDEYTXJL4E6GZM3M7MP5BNBI6KTRWQ2RZT4Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12bbN67ZebXjF4iSyaWu4TDsG1XugTm1YdNHqRdRCbZ2WWRunRG, address:81.171.14.70:4048): protocol: write tcp 188.166.225.170:55932->81.171.14.70:4048: use of closed network connection; write tcp 188.166.225.170:55932->81.171.14.70:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55932->81.171.14.70:4048: use of closed network connection Nov 21 13:36:08.629502702 (293h32m7.180973163s) [noise=true , address=81.171.14.70:4031 , piece_id=LJW25GLWHJCV5CFUITJ5CPYDNWOFG3NBK4A5NUX2VVV4CYCDHWMQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12drdYnnWsuMyFo6LkcewAGGWnorRCAhxPpNqK41LvGy2zkBvng, address:81.171.14.70:4031): protocol: write tcp 188.166.225.170:58232->81.171.14.70:4031: use of closed network connection; write tcp 188.166.225.170:58232->81.171.14.70:4031: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58232->81.171.14.70:4031: use of closed network connection Nov 21 13:36:08.629974720 (293h32m7.181445181s) [noise=true , address=81.171.14.70:4026 , piece_id=ATEKV2WY2WDNBLY7W4THWSO52AJ76Q3XVT7QDGT3EBMPWVRL5EKA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12RTn3BuxasVSPYmjqgyhqKn1t6fAtq9EBtUufXqcLq48My9g2b, address:81.171.14.70:4026): protocol: write tcp 188.166.225.170:41806->81.171.14.70:4026: use of closed network connection; context deadline exceeded Nov 21 13:36:08.630048300 (293h32m7.181518751s) [seg_pos={0 0}, upload=258480]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil> Nov 21 13:36:08.800451489 (293h32m7.35192194s) [upload=258480]: Done uploading segments... Nov 21 13:36:08.800522093 (293h32m7.351992554s) [upload=258480]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24) Duration: 5.841398254s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 16 out of 60 is failed (threshold=6) [.X.X..X.X..XX.X.....X...............X.X.X......X...X..X..XX.] duration: 3.73274819s trace: https://jaeger.dev.storj.io/trace/61e148d9a182a9f1 FAIL (2025-11-21 13:54:32 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.732788377s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 5.355193911s trace: https://jaeger.dev.storj.io/trace/2c4b010d23e46d91 FAIL (2025-11-21 13:53:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 24) ======LOGS====== AMRUQ7TS2R6K7ILJ2YSG7UNGO4P5PFTOMHVF6FXOOIWW46Y2NA, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.291418442 (293h31m7.842888914s) [noise=true , address=89.149.201.147:4006 , piece_id=T2KROQWQR5LXKBJEJO6LICLBP5CUBETT3TOACRQCGYXKZ364TYMQ, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.291954664 (293h31m7.843425145s) [noise=true , address=185.167.204.206:4048 , piece_id=6DBNGIBFWN5PPV72YWVRSSQVYF2AZ7EHS55PJDLDLGZEGXRYYHBA, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.293302360 (293h31m7.844772851s) [noise=true , address=89.149.201.147:4021 , piece_id=K3ZPQXTSSUGIGIX33FCYOC6FRCTWDBRVRHUHHGEJLNWFCJFCIT4Q, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.294979363 (293h31m7.846449854s) [noise=true , address=81.171.14.70:4028 , piece_id=ESUHXDH256EM67DXYQ7O5THLHNAPPJ6I3ARZC5D3GSLJD7CNJV5A, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.295760250 (293h31m7.847230711s) [noise=true , address=81.171.14.70:4048 , piece_id=NUU26IUS2OGXAEVLUDEYTXJL4E6GZM3M7MP5BNBI6KTRWQ2RZT4Q, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.300934223 (293h31m7.852404674s) [noise=true , address=81.171.14.70:4031 , piece_id=LJW25GLWHJCV5CFUITJ5CPYDNWOFG3NBK4A5NUX2VVV4CYCDHWMQ, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.301942341 (293h31m7.853412814s) [noise=true , address=185.167.204.205:4011 , piece_id=O2KXUNUHCKAQZSL5BJRRRC6IIZTSXHM74OZ7REQI3RGSMFA3PN6Q, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.302417767 (293h31m7.853888218s) [noise=true , address=185.167.204.205:4020 , piece_id=LPUWKZY6RPB7TGO22BL4YRA7ETB2RHWUFOXPJX7TZE3OHSFXEH7A, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.304119598 (293h31m7.855590059s) [noise=true , address=89.149.201.147:4014 , piece_id=MM2F7NMVCD3AFX2UM7FRGSO5ZD2OTJPXJEUVKJ63IL5L74LEXZSQ, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:10.220344333 (293h31m8.771814794s) [upload=258480]: Next returned nil segment Nov 21 13:35:10.220363840 (293h31m8.771834301s) [upload=258480]: Waiting for error group managing segments... Nov 21 13:35:10.536640979 (293h31m9.08811144s) [noise=true , address=185.167.204.205:4027 , piece_id=7C2DDLUHJQCI62D2YTI76NDABUZRSPK3FIZ7N45OG5GX3SZWCHNA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.537115091 (293h31m9.088585552s) [noise=true , address=185.167.204.205:4020 , piece_id=LPUWKZY6RPB7TGO22BL4YRA7ETB2RHWUFOXPJX7TZE3OHSFXEH7A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.539139225 (293h31m9.090609686s) [noise=true , address=185.167.204.206:4056 , piece_id=7VD7D3HW5YIZUYRZ5LWEB4VDXN2R6OEU7CTCGLKZUVKEC44V743Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.539668604 (293h31m9.091139065s) [noise=true , address=185.167.204.206:4048 , piece_id=6DBNGIBFWN5PPV72YWVRSSQVYF2AZ7EHS55PJDLDLGZEGXRYYHBA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.541238333 (293h31m9.092708794s) [noise=true , address=185.167.204.205:4008 , piece_id=UE3HOAD4KZRUWKY63RDOZ5LQ7VYAQPNV653CNSWCRXD2WL5MR27A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.542375438 (293h31m9.093845919s) [noise=true , address=185.167.204.205:4036 , piece_id=F652JXWWTK5XUJBY5SSU6IKKJE72DGITQXBRYK3LMQEF63DMRIEQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.543153891 (293h31m9.094624342s) [noise=true , address=185.167.204.205:4002 , piece_id=YFXJFZTNYWU3AZXQYDU5VBWJCPRVBCYCANXPQ77UGR5ZBY5VV53Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.543618846 (293h31m9.095089307s) [noise=true , address=185.167.204.206:4047 , piece_id=OHARKEAXGZLVG5XHD6RYKKWS57JFGH4JR2TAFIQSPUXTRMJD4WZA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.543662328 (293h31m9.095132789s) [noise=true , address=185.167.204.206:4066 , piece_id=T5X6MZEPB6P3DZ2HVMXCC4XJX5JJFXKGASJRWJDCK5TBHDEA2OLQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.544259055 (293h31m9.095729516s) [noise=true , address=185.167.204.205:4011 , piece_id=O2KXUNUHCKAQZSL5BJRRRC6IIZTSXHM74OZ7REQI3RGSMFA3PN6Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.544805565 (293h31m9.096276016s) [noise=true , address=185.167.204.206:4060 , piece_id=VJPM3YAQI5N2VTEBJ5GZ75JCCBOQS5IBEFUHEWTOX3BAKSP7TWFQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.545043088 (293h31m9.096513559s) [noise=true , address=185.167.204.205:4007 , piece_id=R6HJTNG5VAAMG2Y6YXK4K4YBQFUSAJQ4774UCOSGJSRDM4RWLICA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.545385921 (293h31m9.096856382s) [noise=true , address=185.167.204.205:4034 , piece_id=H66BDBLG4T2BABKBNKMJLY6QPZT2VVSZC3GW6F5V2L2BBFOWE76Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.759454013 (293h31m10.310924484s) [noise=true , address=89.149.201.147:4006 , piece_id=T2KROQWQR5LXKBJEJO6LICLBP5CUBETT3TOACRQCGYXKZ364TYMQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.779325021 (293h31m10.330795482s) [noise=true , address=89.149.201.147:4015 , piece_id=VIFNKNEVMCI2CFBREZOLVGFJ2X7Y7ZX6FWDC6PO6CLVX2LFW4FYQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.797229894 (293h31m10.348700345s) [noise=true , address=89.149.201.147:4021 , piece_id=K3ZPQXTSSUGIGIX33FCYOC6FRCTWDBRVRHUHHGEJLNWFCJFCIT4Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.804823945 (293h31m10.356294406s) [noise=true , address=89.149.201.147:4012 , piece_id=3QAKWEG37QPAE4AORUSYURK47YIOTM5VHC563QWC43UGGPS7UV7Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.828388066 (293h31m10.379858537s) [noise=true , address=89.149.201.147:4014 , piece_id=MM2F7NMVCD3AFX2UM7FRGSO5ZD2OTJPXJEUVKJ63IL5L74LEXZSQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.848656089 (293h31m10.40012655s) [noise=true , address=89.149.201.147:4001 , piece_id=OGRBX7OEEUPFXWU3EOYD44FGBQ7FQ5CHHZ2RVRHFJGOQ74EK7K5A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:02.180750503 (293h32m0.732220964s) [noise=true , address=81.171.14.70:4028 , piece_id=ESUHXDH256EM67DXYQ7O5THLHNAPPJ6I3ARZC5D3GSLJD7CNJV5A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:02.321725695 (293h32m0.873196156s) [noise=true , address=81.171.14.70:4044 , piece_id=KMR7RCY3AFLFIAJWIQXYC6ZSX7QLL6OKSCQMYHDKKCRX4MZAC4TQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:03.575396105 (293h32m2.126866556s) [noise=true , address=81.171.14.70:4041 , piece_id=N36O2DAKHTDWU4BPERULQHIUWQQDM5TQUQD7Y36XDA7SPPQML2LQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:05.676531843 (293h32m4.228002294s) [noise=true , address=81.171.14.70:4047 , piece_id=Q4N3CMZGTJGYFS35Y75GJHLQLEYMRIX4SSGTJXEHCCXIAX5Y2T3Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:08.329832254 (293h32m6.881302715s) [noise=true , address=81.171.14.70:4043 , piece_id=TQCRSBAQ4AJPCVP2JZYD4HQF375R2K7P5AIMCZW5EJVFJYUGKYGA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:08.628985817 (293h32m7.180456278s) [noise=true , address=81.171.14.70:4042 , piece_id=E5AMRUQ7TS2R6K7ILJ2YSG7UNGO4P5PFTOMHVF6FXOOIWW46Y2NA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12vBozgWFrZfUEQAwsqBt6CDMW47rVjZaDfvKTsCdPrvX8swkhD, address:81.171.14.70:4042): protocol: read tcp 188.166.225.170:40010->81.171.14.70:4042: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 21 13:36:08.629102389 (293h32m7.18057285s) [noise=true , address=81.171.14.70:4048 , piece_id=NUU26IUS2OGXAEVLUDEYTXJL4E6GZM3M7MP5BNBI6KTRWQ2RZT4Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12bbN67ZebXjF4iSyaWu4TDsG1XugTm1YdNHqRdRCbZ2WWRunRG, address:81.171.14.70:4048): protocol: write tcp 188.166.225.170:55932->81.171.14.70:4048: use of closed network connection; write tcp 188.166.225.170:55932->81.171.14.70:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55932->81.171.14.70:4048: use of closed network connection Nov 21 13:36:08.629502702 (293h32m7.180973163s) [noise=true , address=81.171.14.70:4031 , piece_id=LJW25GLWHJCV5CFUITJ5CPYDNWOFG3NBK4A5NUX2VVV4CYCDHWMQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12drdYnnWsuMyFo6LkcewAGGWnorRCAhxPpNqK41LvGy2zkBvng, address:81.171.14.70:4031): protocol: write tcp 188.166.225.170:58232->81.171.14.70:4031: use of closed network connection; write tcp 188.166.225.170:58232->81.171.14.70:4031: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58232->81.171.14.70:4031: use of closed network connection Nov 21 13:36:08.629974720 (293h32m7.181445181s) [noise=true , address=81.171.14.70:4026 , piece_id=ATEKV2WY2WDNBLY7W4THWSO52AJ76Q3XVT7QDGT3EBMPWVRL5EKA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12RTn3BuxasVSPYmjqgyhqKn1t6fAtq9EBtUufXqcLq48My9g2b, address:81.171.14.70:4026): protocol: write tcp 188.166.225.170:41806->81.171.14.70:4026: use of closed network connection; context deadline exceeded Nov 21 13:36:08.630048300 (293h32m7.181518751s) [seg_pos={0 0}, upload=258480]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil> Nov 21 13:36:08.800451489 (293h32m7.35192194s) [upload=258480]: Done uploading segments... Nov 21 13:36:08.800522093 (293h32m7.351992554s) [upload=258480]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24) Duration: 5.355228236s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 16 out of 60 is failed (threshold=6) [..X.X..X.X..XX.X.....X...............X.X.X......X...X..X..XX] duration: 5.00299636s trace: https://jaeger.dev.storj.io/trace/61e148d9a182a9f1 FAIL (2025-11-21 13:53:26 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.003021429s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 5.900662953s trace: https://jaeger.dev.storj.io/trace/2c4b010d23e46d91 FAIL (2025-11-21 13:53:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 24) ======LOGS====== AMRUQ7TS2R6K7ILJ2YSG7UNGO4P5PFTOMHVF6FXOOIWW46Y2NA, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.291418442 (293h31m7.842888914s) [noise=true , address=89.149.201.147:4006 , piece_id=T2KROQWQR5LXKBJEJO6LICLBP5CUBETT3TOACRQCGYXKZ364TYMQ, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.291954664 (293h31m7.843425145s) [noise=true , address=185.167.204.206:4048 , piece_id=6DBNGIBFWN5PPV72YWVRSSQVYF2AZ7EHS55PJDLDLGZEGXRYYHBA, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.293302360 (293h31m7.844772851s) [noise=true , address=89.149.201.147:4021 , piece_id=K3ZPQXTSSUGIGIX33FCYOC6FRCTWDBRVRHUHHGEJLNWFCJFCIT4Q, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.294979363 (293h31m7.846449854s) [noise=true , address=81.171.14.70:4028 , piece_id=ESUHXDH256EM67DXYQ7O5THLHNAPPJ6I3ARZC5D3GSLJD7CNJV5A, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.295760250 (293h31m7.847230711s) [noise=true , address=81.171.14.70:4048 , piece_id=NUU26IUS2OGXAEVLUDEYTXJL4E6GZM3M7MP5BNBI6KTRWQ2RZT4Q, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.300934223 (293h31m7.852404674s) [noise=true , address=81.171.14.70:4031 , piece_id=LJW25GLWHJCV5CFUITJ5CPYDNWOFG3NBK4A5NUX2VVV4CYCDHWMQ, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.301942341 (293h31m7.853412814s) [noise=true , address=185.167.204.205:4011 , piece_id=O2KXUNUHCKAQZSL5BJRRRC6IIZTSXHM74OZ7REQI3RGSMFA3PN6Q, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.302417767 (293h31m7.853888218s) [noise=true , address=185.167.204.205:4020 , piece_id=LPUWKZY6RPB7TGO22BL4YRA7ETB2RHWUFOXPJX7TZE3OHSFXEH7A, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.304119598 (293h31m7.855590059s) [noise=true , address=89.149.201.147:4014 , piece_id=MM2F7NMVCD3AFX2UM7FRGSO5ZD2OTJPXJEUVKJ63IL5L74LEXZSQ, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:10.220344333 (293h31m8.771814794s) [upload=258480]: Next returned nil segment Nov 21 13:35:10.220363840 (293h31m8.771834301s) [upload=258480]: Waiting for error group managing segments... Nov 21 13:35:10.536640979 (293h31m9.08811144s) [noise=true , address=185.167.204.205:4027 , piece_id=7C2DDLUHJQCI62D2YTI76NDABUZRSPK3FIZ7N45OG5GX3SZWCHNA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.537115091 (293h31m9.088585552s) [noise=true , address=185.167.204.205:4020 , piece_id=LPUWKZY6RPB7TGO22BL4YRA7ETB2RHWUFOXPJX7TZE3OHSFXEH7A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.539139225 (293h31m9.090609686s) [noise=true , address=185.167.204.206:4056 , piece_id=7VD7D3HW5YIZUYRZ5LWEB4VDXN2R6OEU7CTCGLKZUVKEC44V743Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.539668604 (293h31m9.091139065s) [noise=true , address=185.167.204.206:4048 , piece_id=6DBNGIBFWN5PPV72YWVRSSQVYF2AZ7EHS55PJDLDLGZEGXRYYHBA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.541238333 (293h31m9.092708794s) [noise=true , address=185.167.204.205:4008 , piece_id=UE3HOAD4KZRUWKY63RDOZ5LQ7VYAQPNV653CNSWCRXD2WL5MR27A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.542375438 (293h31m9.093845919s) [noise=true , address=185.167.204.205:4036 , piece_id=F652JXWWTK5XUJBY5SSU6IKKJE72DGITQXBRYK3LMQEF63DMRIEQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.543153891 (293h31m9.094624342s) [noise=true , address=185.167.204.205:4002 , piece_id=YFXJFZTNYWU3AZXQYDU5VBWJCPRVBCYCANXPQ77UGR5ZBY5VV53Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.543618846 (293h31m9.095089307s) [noise=true , address=185.167.204.206:4047 , piece_id=OHARKEAXGZLVG5XHD6RYKKWS57JFGH4JR2TAFIQSPUXTRMJD4WZA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.543662328 (293h31m9.095132789s) [noise=true , address=185.167.204.206:4066 , piece_id=T5X6MZEPB6P3DZ2HVMXCC4XJX5JJFXKGASJRWJDCK5TBHDEA2OLQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.544259055 (293h31m9.095729516s) [noise=true , address=185.167.204.205:4011 , piece_id=O2KXUNUHCKAQZSL5BJRRRC6IIZTSXHM74OZ7REQI3RGSMFA3PN6Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.544805565 (293h31m9.096276016s) [noise=true , address=185.167.204.206:4060 , piece_id=VJPM3YAQI5N2VTEBJ5GZ75JCCBOQS5IBEFUHEWTOX3BAKSP7TWFQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.545043088 (293h31m9.096513559s) [noise=true , address=185.167.204.205:4007 , piece_id=R6HJTNG5VAAMG2Y6YXK4K4YBQFUSAJQ4774UCOSGJSRDM4RWLICA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.545385921 (293h31m9.096856382s) [noise=true , address=185.167.204.205:4034 , piece_id=H66BDBLG4T2BABKBNKMJLY6QPZT2VVSZC3GW6F5V2L2BBFOWE76Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.759454013 (293h31m10.310924484s) [noise=true , address=89.149.201.147:4006 , piece_id=T2KROQWQR5LXKBJEJO6LICLBP5CUBETT3TOACRQCGYXKZ364TYMQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.779325021 (293h31m10.330795482s) [noise=true , address=89.149.201.147:4015 , piece_id=VIFNKNEVMCI2CFBREZOLVGFJ2X7Y7ZX6FWDC6PO6CLVX2LFW4FYQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.797229894 (293h31m10.348700345s) [noise=true , address=89.149.201.147:4021 , piece_id=K3ZPQXTSSUGIGIX33FCYOC6FRCTWDBRVRHUHHGEJLNWFCJFCIT4Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.804823945 (293h31m10.356294406s) [noise=true , address=89.149.201.147:4012 , piece_id=3QAKWEG37QPAE4AORUSYURK47YIOTM5VHC563QWC43UGGPS7UV7Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.828388066 (293h31m10.379858537s) [noise=true , address=89.149.201.147:4014 , piece_id=MM2F7NMVCD3AFX2UM7FRGSO5ZD2OTJPXJEUVKJ63IL5L74LEXZSQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.848656089 (293h31m10.40012655s) [noise=true , address=89.149.201.147:4001 , piece_id=OGRBX7OEEUPFXWU3EOYD44FGBQ7FQ5CHHZ2RVRHFJGOQ74EK7K5A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:02.180750503 (293h32m0.732220964s) [noise=true , address=81.171.14.70:4028 , piece_id=ESUHXDH256EM67DXYQ7O5THLHNAPPJ6I3ARZC5D3GSLJD7CNJV5A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:02.321725695 (293h32m0.873196156s) [noise=true , address=81.171.14.70:4044 , piece_id=KMR7RCY3AFLFIAJWIQXYC6ZSX7QLL6OKSCQMYHDKKCRX4MZAC4TQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:03.575396105 (293h32m2.126866556s) [noise=true , address=81.171.14.70:4041 , piece_id=N36O2DAKHTDWU4BPERULQHIUWQQDM5TQUQD7Y36XDA7SPPQML2LQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:05.676531843 (293h32m4.228002294s) [noise=true , address=81.171.14.70:4047 , piece_id=Q4N3CMZGTJGYFS35Y75GJHLQLEYMRIX4SSGTJXEHCCXIAX5Y2T3Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:08.329832254 (293h32m6.881302715s) [noise=true , address=81.171.14.70:4043 , piece_id=TQCRSBAQ4AJPCVP2JZYD4HQF375R2K7P5AIMCZW5EJVFJYUGKYGA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:08.628985817 (293h32m7.180456278s) [noise=true , address=81.171.14.70:4042 , piece_id=E5AMRUQ7TS2R6K7ILJ2YSG7UNGO4P5PFTOMHVF6FXOOIWW46Y2NA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12vBozgWFrZfUEQAwsqBt6CDMW47rVjZaDfvKTsCdPrvX8swkhD, address:81.171.14.70:4042): protocol: read tcp 188.166.225.170:40010->81.171.14.70:4042: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 21 13:36:08.629102389 (293h32m7.18057285s) [noise=true , address=81.171.14.70:4048 , piece_id=NUU26IUS2OGXAEVLUDEYTXJL4E6GZM3M7MP5BNBI6KTRWQ2RZT4Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12bbN67ZebXjF4iSyaWu4TDsG1XugTm1YdNHqRdRCbZ2WWRunRG, address:81.171.14.70:4048): protocol: write tcp 188.166.225.170:55932->81.171.14.70:4048: use of closed network connection; write tcp 188.166.225.170:55932->81.171.14.70:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55932->81.171.14.70:4048: use of closed network connection Nov 21 13:36:08.629502702 (293h32m7.180973163s) [noise=true , address=81.171.14.70:4031 , piece_id=LJW25GLWHJCV5CFUITJ5CPYDNWOFG3NBK4A5NUX2VVV4CYCDHWMQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12drdYnnWsuMyFo6LkcewAGGWnorRCAhxPpNqK41LvGy2zkBvng, address:81.171.14.70:4031): protocol: write tcp 188.166.225.170:58232->81.171.14.70:4031: use of closed network connection; write tcp 188.166.225.170:58232->81.171.14.70:4031: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58232->81.171.14.70:4031: use of closed network connection Nov 21 13:36:08.629974720 (293h32m7.181445181s) [noise=true , address=81.171.14.70:4026 , piece_id=ATEKV2WY2WDNBLY7W4THWSO52AJ76Q3XVT7QDGT3EBMPWVRL5EKA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12RTn3BuxasVSPYmjqgyhqKn1t6fAtq9EBtUufXqcLq48My9g2b, address:81.171.14.70:4026): protocol: write tcp 188.166.225.170:41806->81.171.14.70:4026: use of closed network connection; context deadline exceeded Nov 21 13:36:08.630048300 (293h32m7.181518751s) [seg_pos={0 0}, upload=258480]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil> Nov 21 13:36:08.800451489 (293h32m7.35192194s) [upload=258480]: Done uploading segments... Nov 21 13:36:08.800522093 (293h32m7.351992554s) [upload=258480]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24) Duration: 5.900702809s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 8.758253479s trace: https://jaeger.dev.storj.io/trace/2c4b010d23e46d91 FAIL (2025-11-21 13:52:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 24) ======LOGS====== AMRUQ7TS2R6K7ILJ2YSG7UNGO4P5PFTOMHVF6FXOOIWW46Y2NA, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.291418442 (293h31m7.842888914s) [noise=true , address=89.149.201.147:4006 , piece_id=T2KROQWQR5LXKBJEJO6LICLBP5CUBETT3TOACRQCGYXKZ364TYMQ, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.291954664 (293h31m7.843425145s) [noise=true , address=185.167.204.206:4048 , piece_id=6DBNGIBFWN5PPV72YWVRSSQVYF2AZ7EHS55PJDLDLGZEGXRYYHBA, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.293302360 (293h31m7.844772851s) [noise=true , address=89.149.201.147:4021 , piece_id=K3ZPQXTSSUGIGIX33FCYOC6FRCTWDBRVRHUHHGEJLNWFCJFCIT4Q, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.294979363 (293h31m7.846449854s) [noise=true , address=81.171.14.70:4028 , piece_id=ESUHXDH256EM67DXYQ7O5THLHNAPPJ6I3ARZC5D3GSLJD7CNJV5A, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.295760250 (293h31m7.847230711s) [noise=true , address=81.171.14.70:4048 , piece_id=NUU26IUS2OGXAEVLUDEYTXJL4E6GZM3M7MP5BNBI6KTRWQ2RZT4Q, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.300934223 (293h31m7.852404674s) [noise=true , address=81.171.14.70:4031 , piece_id=LJW25GLWHJCV5CFUITJ5CPYDNWOFG3NBK4A5NUX2VVV4CYCDHWMQ, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.301942341 (293h31m7.853412814s) [noise=true , address=185.167.204.205:4011 , piece_id=O2KXUNUHCKAQZSL5BJRRRC6IIZTSXHM74OZ7REQI3RGSMFA3PN6Q, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.302417767 (293h31m7.853888218s) [noise=true , address=185.167.204.205:4020 , piece_id=LPUWKZY6RPB7TGO22BL4YRA7ETB2RHWUFOXPJX7TZE3OHSFXEH7A, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.304119598 (293h31m7.855590059s) [noise=true , address=89.149.201.147:4014 , piece_id=MM2F7NMVCD3AFX2UM7FRGSO5ZD2OTJPXJEUVKJ63IL5L74LEXZSQ, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:10.220344333 (293h31m8.771814794s) [upload=258480]: Next returned nil segment Nov 21 13:35:10.220363840 (293h31m8.771834301s) [upload=258480]: Waiting for error group managing segments... Nov 21 13:35:10.536640979 (293h31m9.08811144s) [noise=true , address=185.167.204.205:4027 , piece_id=7C2DDLUHJQCI62D2YTI76NDABUZRSPK3FIZ7N45OG5GX3SZWCHNA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.537115091 (293h31m9.088585552s) [noise=true , address=185.167.204.205:4020 , piece_id=LPUWKZY6RPB7TGO22BL4YRA7ETB2RHWUFOXPJX7TZE3OHSFXEH7A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.539139225 (293h31m9.090609686s) [noise=true , address=185.167.204.206:4056 , piece_id=7VD7D3HW5YIZUYRZ5LWEB4VDXN2R6OEU7CTCGLKZUVKEC44V743Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.539668604 (293h31m9.091139065s) [noise=true , address=185.167.204.206:4048 , piece_id=6DBNGIBFWN5PPV72YWVRSSQVYF2AZ7EHS55PJDLDLGZEGXRYYHBA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.541238333 (293h31m9.092708794s) [noise=true , address=185.167.204.205:4008 , piece_id=UE3HOAD4KZRUWKY63RDOZ5LQ7VYAQPNV653CNSWCRXD2WL5MR27A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.542375438 (293h31m9.093845919s) [noise=true , address=185.167.204.205:4036 , piece_id=F652JXWWTK5XUJBY5SSU6IKKJE72DGITQXBRYK3LMQEF63DMRIEQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.543153891 (293h31m9.094624342s) [noise=true , address=185.167.204.205:4002 , piece_id=YFXJFZTNYWU3AZXQYDU5VBWJCPRVBCYCANXPQ77UGR5ZBY5VV53Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.543618846 (293h31m9.095089307s) [noise=true , address=185.167.204.206:4047 , piece_id=OHARKEAXGZLVG5XHD6RYKKWS57JFGH4JR2TAFIQSPUXTRMJD4WZA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.543662328 (293h31m9.095132789s) [noise=true , address=185.167.204.206:4066 , piece_id=T5X6MZEPB6P3DZ2HVMXCC4XJX5JJFXKGASJRWJDCK5TBHDEA2OLQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.544259055 (293h31m9.095729516s) [noise=true , address=185.167.204.205:4011 , piece_id=O2KXUNUHCKAQZSL5BJRRRC6IIZTSXHM74OZ7REQI3RGSMFA3PN6Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.544805565 (293h31m9.096276016s) [noise=true , address=185.167.204.206:4060 , piece_id=VJPM3YAQI5N2VTEBJ5GZ75JCCBOQS5IBEFUHEWTOX3BAKSP7TWFQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.545043088 (293h31m9.096513559s) [noise=true , address=185.167.204.205:4007 , piece_id=R6HJTNG5VAAMG2Y6YXK4K4YBQFUSAJQ4774UCOSGJSRDM4RWLICA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.545385921 (293h31m9.096856382s) [noise=true , address=185.167.204.205:4034 , piece_id=H66BDBLG4T2BABKBNKMJLY6QPZT2VVSZC3GW6F5V2L2BBFOWE76Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.759454013 (293h31m10.310924484s) [noise=true , address=89.149.201.147:4006 , piece_id=T2KROQWQR5LXKBJEJO6LICLBP5CUBETT3TOACRQCGYXKZ364TYMQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.779325021 (293h31m10.330795482s) [noise=true , address=89.149.201.147:4015 , piece_id=VIFNKNEVMCI2CFBREZOLVGFJ2X7Y7ZX6FWDC6PO6CLVX2LFW4FYQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.797229894 (293h31m10.348700345s) [noise=true , address=89.149.201.147:4021 , piece_id=K3ZPQXTSSUGIGIX33FCYOC6FRCTWDBRVRHUHHGEJLNWFCJFCIT4Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.804823945 (293h31m10.356294406s) [noise=true , address=89.149.201.147:4012 , piece_id=3QAKWEG37QPAE4AORUSYURK47YIOTM5VHC563QWC43UGGPS7UV7Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.828388066 (293h31m10.379858537s) [noise=true , address=89.149.201.147:4014 , piece_id=MM2F7NMVCD3AFX2UM7FRGSO5ZD2OTJPXJEUVKJ63IL5L74LEXZSQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.848656089 (293h31m10.40012655s) [noise=true , address=89.149.201.147:4001 , piece_id=OGRBX7OEEUPFXWU3EOYD44FGBQ7FQ5CHHZ2RVRHFJGOQ74EK7K5A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:02.180750503 (293h32m0.732220964s) [noise=true , address=81.171.14.70:4028 , piece_id=ESUHXDH256EM67DXYQ7O5THLHNAPPJ6I3ARZC5D3GSLJD7CNJV5A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:02.321725695 (293h32m0.873196156s) [noise=true , address=81.171.14.70:4044 , piece_id=KMR7RCY3AFLFIAJWIQXYC6ZSX7QLL6OKSCQMYHDKKCRX4MZAC4TQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:03.575396105 (293h32m2.126866556s) [noise=true , address=81.171.14.70:4041 , piece_id=N36O2DAKHTDWU4BPERULQHIUWQQDM5TQUQD7Y36XDA7SPPQML2LQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:05.676531843 (293h32m4.228002294s) [noise=true , address=81.171.14.70:4047 , piece_id=Q4N3CMZGTJGYFS35Y75GJHLQLEYMRIX4SSGTJXEHCCXIAX5Y2T3Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:08.329832254 (293h32m6.881302715s) [noise=true , address=81.171.14.70:4043 , piece_id=TQCRSBAQ4AJPCVP2JZYD4HQF375R2K7P5AIMCZW5EJVFJYUGKYGA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:08.628985817 (293h32m7.180456278s) [noise=true , address=81.171.14.70:4042 , piece_id=E5AMRUQ7TS2R6K7ILJ2YSG7UNGO4P5PFTOMHVF6FXOOIWW46Y2NA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12vBozgWFrZfUEQAwsqBt6CDMW47rVjZaDfvKTsCdPrvX8swkhD, address:81.171.14.70:4042): protocol: read tcp 188.166.225.170:40010->81.171.14.70:4042: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 21 13:36:08.629102389 (293h32m7.18057285s) [noise=true , address=81.171.14.70:4048 , piece_id=NUU26IUS2OGXAEVLUDEYTXJL4E6GZM3M7MP5BNBI6KTRWQ2RZT4Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12bbN67ZebXjF4iSyaWu4TDsG1XugTm1YdNHqRdRCbZ2WWRunRG, address:81.171.14.70:4048): protocol: write tcp 188.166.225.170:55932->81.171.14.70:4048: use of closed network connection; write tcp 188.166.225.170:55932->81.171.14.70:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55932->81.171.14.70:4048: use of closed network connection Nov 21 13:36:08.629502702 (293h32m7.180973163s) [noise=true , address=81.171.14.70:4031 , piece_id=LJW25GLWHJCV5CFUITJ5CPYDNWOFG3NBK4A5NUX2VVV4CYCDHWMQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12drdYnnWsuMyFo6LkcewAGGWnorRCAhxPpNqK41LvGy2zkBvng, address:81.171.14.70:4031): protocol: write tcp 188.166.225.170:58232->81.171.14.70:4031: use of closed network connection; write tcp 188.166.225.170:58232->81.171.14.70:4031: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58232->81.171.14.70:4031: use of closed network connection Nov 21 13:36:08.629974720 (293h32m7.181445181s) [noise=true , address=81.171.14.70:4026 , piece_id=ATEKV2WY2WDNBLY7W4THWSO52AJ76Q3XVT7QDGT3EBMPWVRL5EKA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12RTn3BuxasVSPYmjqgyhqKn1t6fAtq9EBtUufXqcLq48My9g2b, address:81.171.14.70:4026): protocol: write tcp 188.166.225.170:41806->81.171.14.70:4026: use of closed network connection; context deadline exceeded Nov 21 13:36:08.630048300 (293h32m7.181518751s) [seg_pos={0 0}, upload=258480]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil> Nov 21 13:36:08.800451489 (293h32m7.35192194s) [upload=258480]: Done uploading segments... Nov 21 13:36:08.800522093 (293h32m7.351992554s) [upload=258480]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24) Duration: 8.758271383s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 5.480002329s trace: https://jaeger.dev.storj.io/trace/2c4b010d23e46d91 FAIL (2025-11-21 13:52:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 24) ======LOGS====== AMRUQ7TS2R6K7ILJ2YSG7UNGO4P5PFTOMHVF6FXOOIWW46Y2NA, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.291418442 (293h31m7.842888914s) [noise=true , address=89.149.201.147:4006 , piece_id=T2KROQWQR5LXKBJEJO6LICLBP5CUBETT3TOACRQCGYXKZ364TYMQ, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.291954664 (293h31m7.843425145s) [noise=true , address=185.167.204.206:4048 , piece_id=6DBNGIBFWN5PPV72YWVRSSQVYF2AZ7EHS55PJDLDLGZEGXRYYHBA, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.293302360 (293h31m7.844772851s) [noise=true , address=89.149.201.147:4021 , piece_id=K3ZPQXTSSUGIGIX33FCYOC6FRCTWDBRVRHUHHGEJLNWFCJFCIT4Q, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.294979363 (293h31m7.846449854s) [noise=true , address=81.171.14.70:4028 , piece_id=ESUHXDH256EM67DXYQ7O5THLHNAPPJ6I3ARZC5D3GSLJD7CNJV5A, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.295760250 (293h31m7.847230711s) [noise=true , address=81.171.14.70:4048 , piece_id=NUU26IUS2OGXAEVLUDEYTXJL4E6GZM3M7MP5BNBI6KTRWQ2RZT4Q, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.300934223 (293h31m7.852404674s) [noise=true , address=81.171.14.70:4031 , piece_id=LJW25GLWHJCV5CFUITJ5CPYDNWOFG3NBK4A5NUX2VVV4CYCDHWMQ, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.301942341 (293h31m7.853412814s) [noise=true , address=185.167.204.205:4011 , piece_id=O2KXUNUHCKAQZSL5BJRRRC6IIZTSXHM74OZ7REQI3RGSMFA3PN6Q, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.302417767 (293h31m7.853888218s) [noise=true , address=185.167.204.205:4020 , piece_id=LPUWKZY6RPB7TGO22BL4YRA7ETB2RHWUFOXPJX7TZE3OHSFXEH7A, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.304119598 (293h31m7.855590059s) [noise=true , address=89.149.201.147:4014 , piece_id=MM2F7NMVCD3AFX2UM7FRGSO5ZD2OTJPXJEUVKJ63IL5L74LEXZSQ, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:10.220344333 (293h31m8.771814794s) [upload=258480]: Next returned nil segment Nov 21 13:35:10.220363840 (293h31m8.771834301s) [upload=258480]: Waiting for error group managing segments... Nov 21 13:35:10.536640979 (293h31m9.08811144s) [noise=true , address=185.167.204.205:4027 , piece_id=7C2DDLUHJQCI62D2YTI76NDABUZRSPK3FIZ7N45OG5GX3SZWCHNA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.537115091 (293h31m9.088585552s) [noise=true , address=185.167.204.205:4020 , piece_id=LPUWKZY6RPB7TGO22BL4YRA7ETB2RHWUFOXPJX7TZE3OHSFXEH7A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.539139225 (293h31m9.090609686s) [noise=true , address=185.167.204.206:4056 , piece_id=7VD7D3HW5YIZUYRZ5LWEB4VDXN2R6OEU7CTCGLKZUVKEC44V743Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.539668604 (293h31m9.091139065s) [noise=true , address=185.167.204.206:4048 , piece_id=6DBNGIBFWN5PPV72YWVRSSQVYF2AZ7EHS55PJDLDLGZEGXRYYHBA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.541238333 (293h31m9.092708794s) [noise=true , address=185.167.204.205:4008 , piece_id=UE3HOAD4KZRUWKY63RDOZ5LQ7VYAQPNV653CNSWCRXD2WL5MR27A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.542375438 (293h31m9.093845919s) [noise=true , address=185.167.204.205:4036 , piece_id=F652JXWWTK5XUJBY5SSU6IKKJE72DGITQXBRYK3LMQEF63DMRIEQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.543153891 (293h31m9.094624342s) [noise=true , address=185.167.204.205:4002 , piece_id=YFXJFZTNYWU3AZXQYDU5VBWJCPRVBCYCANXPQ77UGR5ZBY5VV53Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.543618846 (293h31m9.095089307s) [noise=true , address=185.167.204.206:4047 , piece_id=OHARKEAXGZLVG5XHD6RYKKWS57JFGH4JR2TAFIQSPUXTRMJD4WZA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.543662328 (293h31m9.095132789s) [noise=true , address=185.167.204.206:4066 , piece_id=T5X6MZEPB6P3DZ2HVMXCC4XJX5JJFXKGASJRWJDCK5TBHDEA2OLQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.544259055 (293h31m9.095729516s) [noise=true , address=185.167.204.205:4011 , piece_id=O2KXUNUHCKAQZSL5BJRRRC6IIZTSXHM74OZ7REQI3RGSMFA3PN6Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.544805565 (293h31m9.096276016s) [noise=true , address=185.167.204.206:4060 , piece_id=VJPM3YAQI5N2VTEBJ5GZ75JCCBOQS5IBEFUHEWTOX3BAKSP7TWFQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.545043088 (293h31m9.096513559s) [noise=true , address=185.167.204.205:4007 , piece_id=R6HJTNG5VAAMG2Y6YXK4K4YBQFUSAJQ4774UCOSGJSRDM4RWLICA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.545385921 (293h31m9.096856382s) [noise=true , address=185.167.204.205:4034 , piece_id=H66BDBLG4T2BABKBNKMJLY6QPZT2VVSZC3GW6F5V2L2BBFOWE76Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.759454013 (293h31m10.310924484s) [noise=true , address=89.149.201.147:4006 , piece_id=T2KROQWQR5LXKBJEJO6LICLBP5CUBETT3TOACRQCGYXKZ364TYMQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.779325021 (293h31m10.330795482s) [noise=true , address=89.149.201.147:4015 , piece_id=VIFNKNEVMCI2CFBREZOLVGFJ2X7Y7ZX6FWDC6PO6CLVX2LFW4FYQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.797229894 (293h31m10.348700345s) [noise=true , address=89.149.201.147:4021 , piece_id=K3ZPQXTSSUGIGIX33FCYOC6FRCTWDBRVRHUHHGEJLNWFCJFCIT4Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.804823945 (293h31m10.356294406s) [noise=true , address=89.149.201.147:4012 , piece_id=3QAKWEG37QPAE4AORUSYURK47YIOTM5VHC563QWC43UGGPS7UV7Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.828388066 (293h31m10.379858537s) [noise=true , address=89.149.201.147:4014 , piece_id=MM2F7NMVCD3AFX2UM7FRGSO5ZD2OTJPXJEUVKJ63IL5L74LEXZSQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.848656089 (293h31m10.40012655s) [noise=true , address=89.149.201.147:4001 , piece_id=OGRBX7OEEUPFXWU3EOYD44FGBQ7FQ5CHHZ2RVRHFJGOQ74EK7K5A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:02.180750503 (293h32m0.732220964s) [noise=true , address=81.171.14.70:4028 , piece_id=ESUHXDH256EM67DXYQ7O5THLHNAPPJ6I3ARZC5D3GSLJD7CNJV5A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:02.321725695 (293h32m0.873196156s) [noise=true , address=81.171.14.70:4044 , piece_id=KMR7RCY3AFLFIAJWIQXYC6ZSX7QLL6OKSCQMYHDKKCRX4MZAC4TQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:03.575396105 (293h32m2.126866556s) [noise=true , address=81.171.14.70:4041 , piece_id=N36O2DAKHTDWU4BPERULQHIUWQQDM5TQUQD7Y36XDA7SPPQML2LQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:05.676531843 (293h32m4.228002294s) [noise=true , address=81.171.14.70:4047 , piece_id=Q4N3CMZGTJGYFS35Y75GJHLQLEYMRIX4SSGTJXEHCCXIAX5Y2T3Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:08.329832254 (293h32m6.881302715s) [noise=true , address=81.171.14.70:4043 , piece_id=TQCRSBAQ4AJPCVP2JZYD4HQF375R2K7P5AIMCZW5EJVFJYUGKYGA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:08.628985817 (293h32m7.180456278s) [noise=true , address=81.171.14.70:4042 , piece_id=E5AMRUQ7TS2R6K7ILJ2YSG7UNGO4P5PFTOMHVF6FXOOIWW46Y2NA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12vBozgWFrZfUEQAwsqBt6CDMW47rVjZaDfvKTsCdPrvX8swkhD, address:81.171.14.70:4042): protocol: read tcp 188.166.225.170:40010->81.171.14.70:4042: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 21 13:36:08.629102389 (293h32m7.18057285s) [noise=true , address=81.171.14.70:4048 , piece_id=NUU26IUS2OGXAEVLUDEYTXJL4E6GZM3M7MP5BNBI6KTRWQ2RZT4Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12bbN67ZebXjF4iSyaWu4TDsG1XugTm1YdNHqRdRCbZ2WWRunRG, address:81.171.14.70:4048): protocol: write tcp 188.166.225.170:55932->81.171.14.70:4048: use of closed network connection; write tcp 188.166.225.170:55932->81.171.14.70:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55932->81.171.14.70:4048: use of closed network connection Nov 21 13:36:08.629502702 (293h32m7.180973163s) [noise=true , address=81.171.14.70:4031 , piece_id=LJW25GLWHJCV5CFUITJ5CPYDNWOFG3NBK4A5NUX2VVV4CYCDHWMQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12drdYnnWsuMyFo6LkcewAGGWnorRCAhxPpNqK41LvGy2zkBvng, address:81.171.14.70:4031): protocol: write tcp 188.166.225.170:58232->81.171.14.70:4031: use of closed network connection; write tcp 188.166.225.170:58232->81.171.14.70:4031: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58232->81.171.14.70:4031: use of closed network connection Nov 21 13:36:08.629974720 (293h32m7.181445181s) [noise=true , address=81.171.14.70:4026 , piece_id=ATEKV2WY2WDNBLY7W4THWSO52AJ76Q3XVT7QDGT3EBMPWVRL5EKA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12RTn3BuxasVSPYmjqgyhqKn1t6fAtq9EBtUufXqcLq48My9g2b, address:81.171.14.70:4026): protocol: write tcp 188.166.225.170:41806->81.171.14.70:4026: use of closed network connection; context deadline exceeded Nov 21 13:36:08.630048300 (293h32m7.181518751s) [seg_pos={0 0}, upload=258480]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil> Nov 21 13:36:08.800451489 (293h32m7.35192194s) [upload=258480]: Done uploading segments... Nov 21 13:36:08.800522093 (293h32m7.351992554s) [upload=258480]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24) Duration: 5.480030132s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 16 out of 60 is failed (threshold=6) [X..X.X..X.X..XX.X.....X...............X.X.X......X...X..X..X] duration: 5.001872883s trace: https://jaeger.dev.storj.io/trace/4d9ceec647c56dfe FAIL (2025-11-21 13:52:31 UTC) orbiter warning: context deadline exceeded Duration: 5.00192883s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 5.823470067s trace: https://jaeger.dev.storj.io/trace/2c4b010d23e46d91 FAIL (2025-11-21 13:52:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 24) ======LOGS====== AMRUQ7TS2R6K7ILJ2YSG7UNGO4P5PFTOMHVF6FXOOIWW46Y2NA, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.291418442 (293h31m7.842888914s) [noise=true , address=89.149.201.147:4006 , piece_id=T2KROQWQR5LXKBJEJO6LICLBP5CUBETT3TOACRQCGYXKZ364TYMQ, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.291954664 (293h31m7.843425145s) [noise=true , address=185.167.204.206:4048 , piece_id=6DBNGIBFWN5PPV72YWVRSSQVYF2AZ7EHS55PJDLDLGZEGXRYYHBA, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.293302360 (293h31m7.844772851s) [noise=true , address=89.149.201.147:4021 , piece_id=K3ZPQXTSSUGIGIX33FCYOC6FRCTWDBRVRHUHHGEJLNWFCJFCIT4Q, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.294979363 (293h31m7.846449854s) [noise=true , address=81.171.14.70:4028 , piece_id=ESUHXDH256EM67DXYQ7O5THLHNAPPJ6I3ARZC5D3GSLJD7CNJV5A, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.295760250 (293h31m7.847230711s) [noise=true , address=81.171.14.70:4048 , piece_id=NUU26IUS2OGXAEVLUDEYTXJL4E6GZM3M7MP5BNBI6KTRWQ2RZT4Q, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.300934223 (293h31m7.852404674s) [noise=true , address=81.171.14.70:4031 , piece_id=LJW25GLWHJCV5CFUITJ5CPYDNWOFG3NBK4A5NUX2VVV4CYCDHWMQ, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.301942341 (293h31m7.853412814s) [noise=true , address=185.167.204.205:4011 , piece_id=O2KXUNUHCKAQZSL5BJRRRC6IIZTSXHM74OZ7REQI3RGSMFA3PN6Q, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.302417767 (293h31m7.853888218s) [noise=true , address=185.167.204.205:4020 , piece_id=LPUWKZY6RPB7TGO22BL4YRA7ETB2RHWUFOXPJX7TZE3OHSFXEH7A, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.304119598 (293h31m7.855590059s) [noise=true , address=89.149.201.147:4014 , piece_id=MM2F7NMVCD3AFX2UM7FRGSO5ZD2OTJPXJEUVKJ63IL5L74LEXZSQ, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:10.220344333 (293h31m8.771814794s) [upload=258480]: Next returned nil segment Nov 21 13:35:10.220363840 (293h31m8.771834301s) [upload=258480]: Waiting for error group managing segments... Nov 21 13:35:10.536640979 (293h31m9.08811144s) [noise=true , address=185.167.204.205:4027 , piece_id=7C2DDLUHJQCI62D2YTI76NDABUZRSPK3FIZ7N45OG5GX3SZWCHNA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.537115091 (293h31m9.088585552s) [noise=true , address=185.167.204.205:4020 , piece_id=LPUWKZY6RPB7TGO22BL4YRA7ETB2RHWUFOXPJX7TZE3OHSFXEH7A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.539139225 (293h31m9.090609686s) [noise=true , address=185.167.204.206:4056 , piece_id=7VD7D3HW5YIZUYRZ5LWEB4VDXN2R6OEU7CTCGLKZUVKEC44V743Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.539668604 (293h31m9.091139065s) [noise=true , address=185.167.204.206:4048 , piece_id=6DBNGIBFWN5PPV72YWVRSSQVYF2AZ7EHS55PJDLDLGZEGXRYYHBA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.541238333 (293h31m9.092708794s) [noise=true , address=185.167.204.205:4008 , piece_id=UE3HOAD4KZRUWKY63RDOZ5LQ7VYAQPNV653CNSWCRXD2WL5MR27A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.542375438 (293h31m9.093845919s) [noise=true , address=185.167.204.205:4036 , piece_id=F652JXWWTK5XUJBY5SSU6IKKJE72DGITQXBRYK3LMQEF63DMRIEQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.543153891 (293h31m9.094624342s) [noise=true , address=185.167.204.205:4002 , piece_id=YFXJFZTNYWU3AZXQYDU5VBWJCPRVBCYCANXPQ77UGR5ZBY5VV53Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.543618846 (293h31m9.095089307s) [noise=true , address=185.167.204.206:4047 , piece_id=OHARKEAXGZLVG5XHD6RYKKWS57JFGH4JR2TAFIQSPUXTRMJD4WZA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.543662328 (293h31m9.095132789s) [noise=true , address=185.167.204.206:4066 , piece_id=T5X6MZEPB6P3DZ2HVMXCC4XJX5JJFXKGASJRWJDCK5TBHDEA2OLQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.544259055 (293h31m9.095729516s) [noise=true , address=185.167.204.205:4011 , piece_id=O2KXUNUHCKAQZSL5BJRRRC6IIZTSXHM74OZ7REQI3RGSMFA3PN6Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.544805565 (293h31m9.096276016s) [noise=true , address=185.167.204.206:4060 , piece_id=VJPM3YAQI5N2VTEBJ5GZ75JCCBOQS5IBEFUHEWTOX3BAKSP7TWFQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.545043088 (293h31m9.096513559s) [noise=true , address=185.167.204.205:4007 , piece_id=R6HJTNG5VAAMG2Y6YXK4K4YBQFUSAJQ4774UCOSGJSRDM4RWLICA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.545385921 (293h31m9.096856382s) [noise=true , address=185.167.204.205:4034 , piece_id=H66BDBLG4T2BABKBNKMJLY6QPZT2VVSZC3GW6F5V2L2BBFOWE76Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.759454013 (293h31m10.310924484s) [noise=true , address=89.149.201.147:4006 , piece_id=T2KROQWQR5LXKBJEJO6LICLBP5CUBETT3TOACRQCGYXKZ364TYMQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.779325021 (293h31m10.330795482s) [noise=true , address=89.149.201.147:4015 , piece_id=VIFNKNEVMCI2CFBREZOLVGFJ2X7Y7ZX6FWDC6PO6CLVX2LFW4FYQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.797229894 (293h31m10.348700345s) [noise=true , address=89.149.201.147:4021 , piece_id=K3ZPQXTSSUGIGIX33FCYOC6FRCTWDBRVRHUHHGEJLNWFCJFCIT4Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.804823945 (293h31m10.356294406s) [noise=true , address=89.149.201.147:4012 , piece_id=3QAKWEG37QPAE4AORUSYURK47YIOTM5VHC563QWC43UGGPS7UV7Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.828388066 (293h31m10.379858537s) [noise=true , address=89.149.201.147:4014 , piece_id=MM2F7NMVCD3AFX2UM7FRGSO5ZD2OTJPXJEUVKJ63IL5L74LEXZSQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.848656089 (293h31m10.40012655s) [noise=true , address=89.149.201.147:4001 , piece_id=OGRBX7OEEUPFXWU3EOYD44FGBQ7FQ5CHHZ2RVRHFJGOQ74EK7K5A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:02.180750503 (293h32m0.732220964s) [noise=true , address=81.171.14.70:4028 , piece_id=ESUHXDH256EM67DXYQ7O5THLHNAPPJ6I3ARZC5D3GSLJD7CNJV5A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:02.321725695 (293h32m0.873196156s) [noise=true , address=81.171.14.70:4044 , piece_id=KMR7RCY3AFLFIAJWIQXYC6ZSX7QLL6OKSCQMYHDKKCRX4MZAC4TQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:03.575396105 (293h32m2.126866556s) [noise=true , address=81.171.14.70:4041 , piece_id=N36O2DAKHTDWU4BPERULQHIUWQQDM5TQUQD7Y36XDA7SPPQML2LQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:05.676531843 (293h32m4.228002294s) [noise=true , address=81.171.14.70:4047 , piece_id=Q4N3CMZGTJGYFS35Y75GJHLQLEYMRIX4SSGTJXEHCCXIAX5Y2T3Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:08.329832254 (293h32m6.881302715s) [noise=true , address=81.171.14.70:4043 , piece_id=TQCRSBAQ4AJPCVP2JZYD4HQF375R2K7P5AIMCZW5EJVFJYUGKYGA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:08.628985817 (293h32m7.180456278s) [noise=true , address=81.171.14.70:4042 , piece_id=E5AMRUQ7TS2R6K7ILJ2YSG7UNGO4P5PFTOMHVF6FXOOIWW46Y2NA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12vBozgWFrZfUEQAwsqBt6CDMW47rVjZaDfvKTsCdPrvX8swkhD, address:81.171.14.70:4042): protocol: read tcp 188.166.225.170:40010->81.171.14.70:4042: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 21 13:36:08.629102389 (293h32m7.18057285s) [noise=true , address=81.171.14.70:4048 , piece_id=NUU26IUS2OGXAEVLUDEYTXJL4E6GZM3M7MP5BNBI6KTRWQ2RZT4Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12bbN67ZebXjF4iSyaWu4TDsG1XugTm1YdNHqRdRCbZ2WWRunRG, address:81.171.14.70:4048): protocol: write tcp 188.166.225.170:55932->81.171.14.70:4048: use of closed network connection; write tcp 188.166.225.170:55932->81.171.14.70:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55932->81.171.14.70:4048: use of closed network connection Nov 21 13:36:08.629502702 (293h32m7.180973163s) [noise=true , address=81.171.14.70:4031 , piece_id=LJW25GLWHJCV5CFUITJ5CPYDNWOFG3NBK4A5NUX2VVV4CYCDHWMQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12drdYnnWsuMyFo6LkcewAGGWnorRCAhxPpNqK41LvGy2zkBvng, address:81.171.14.70:4031): protocol: write tcp 188.166.225.170:58232->81.171.14.70:4031: use of closed network connection; write tcp 188.166.225.170:58232->81.171.14.70:4031: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58232->81.171.14.70:4031: use of closed network connection Nov 21 13:36:08.629974720 (293h32m7.181445181s) [noise=true , address=81.171.14.70:4026 , piece_id=ATEKV2WY2WDNBLY7W4THWSO52AJ76Q3XVT7QDGT3EBMPWVRL5EKA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12RTn3BuxasVSPYmjqgyhqKn1t6fAtq9EBtUufXqcLq48My9g2b, address:81.171.14.70:4026): protocol: write tcp 188.166.225.170:41806->81.171.14.70:4026: use of closed network connection; context deadline exceeded Nov 21 13:36:08.630048300 (293h32m7.181518751s) [seg_pos={0 0}, upload=258480]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil> Nov 21 13:36:08.800451489 (293h32m7.35192194s) [upload=258480]: Done uploading segments... Nov 21 13:36:08.800522093 (293h32m7.351992554s) [upload=258480]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24) Duration: 5.823551942s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 7.902189501s trace: https://jaeger.dev.storj.io/trace/2c4b010d23e46d91 FAIL (2025-11-21 13:51:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 24) ======LOGS====== AMRUQ7TS2R6K7ILJ2YSG7UNGO4P5PFTOMHVF6FXOOIWW46Y2NA, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.291418442 (293h31m7.842888914s) [noise=true , address=89.149.201.147:4006 , piece_id=T2KROQWQR5LXKBJEJO6LICLBP5CUBETT3TOACRQCGYXKZ364TYMQ, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.291954664 (293h31m7.843425145s) [noise=true , address=185.167.204.206:4048 , piece_id=6DBNGIBFWN5PPV72YWVRSSQVYF2AZ7EHS55PJDLDLGZEGXRYYHBA, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.293302360 (293h31m7.844772851s) [noise=true , address=89.149.201.147:4021 , piece_id=K3ZPQXTSSUGIGIX33FCYOC6FRCTWDBRVRHUHHGEJLNWFCJFCIT4Q, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.294979363 (293h31m7.846449854s) [noise=true , address=81.171.14.70:4028 , piece_id=ESUHXDH256EM67DXYQ7O5THLHNAPPJ6I3ARZC5D3GSLJD7CNJV5A, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.295760250 (293h31m7.847230711s) [noise=true , address=81.171.14.70:4048 , piece_id=NUU26IUS2OGXAEVLUDEYTXJL4E6GZM3M7MP5BNBI6KTRWQ2RZT4Q, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.300934223 (293h31m7.852404674s) [noise=true , address=81.171.14.70:4031 , piece_id=LJW25GLWHJCV5CFUITJ5CPYDNWOFG3NBK4A5NUX2VVV4CYCDHWMQ, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.301942341 (293h31m7.853412814s) [noise=true , address=185.167.204.205:4011 , piece_id=O2KXUNUHCKAQZSL5BJRRRC6IIZTSXHM74OZ7REQI3RGSMFA3PN6Q, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.302417767 (293h31m7.853888218s) [noise=true , address=185.167.204.205:4020 , piece_id=LPUWKZY6RPB7TGO22BL4YRA7ETB2RHWUFOXPJX7TZE3OHSFXEH7A, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.304119598 (293h31m7.855590059s) [noise=true , address=89.149.201.147:4014 , piece_id=MM2F7NMVCD3AFX2UM7FRGSO5ZD2OTJPXJEUVKJ63IL5L74LEXZSQ, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:10.220344333 (293h31m8.771814794s) [upload=258480]: Next returned nil segment Nov 21 13:35:10.220363840 (293h31m8.771834301s) [upload=258480]: Waiting for error group managing segments... Nov 21 13:35:10.536640979 (293h31m9.08811144s) [noise=true , address=185.167.204.205:4027 , piece_id=7C2DDLUHJQCI62D2YTI76NDABUZRSPK3FIZ7N45OG5GX3SZWCHNA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.537115091 (293h31m9.088585552s) [noise=true , address=185.167.204.205:4020 , piece_id=LPUWKZY6RPB7TGO22BL4YRA7ETB2RHWUFOXPJX7TZE3OHSFXEH7A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.539139225 (293h31m9.090609686s) [noise=true , address=185.167.204.206:4056 , piece_id=7VD7D3HW5YIZUYRZ5LWEB4VDXN2R6OEU7CTCGLKZUVKEC44V743Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.539668604 (293h31m9.091139065s) [noise=true , address=185.167.204.206:4048 , piece_id=6DBNGIBFWN5PPV72YWVRSSQVYF2AZ7EHS55PJDLDLGZEGXRYYHBA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.541238333 (293h31m9.092708794s) [noise=true , address=185.167.204.205:4008 , piece_id=UE3HOAD4KZRUWKY63RDOZ5LQ7VYAQPNV653CNSWCRXD2WL5MR27A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.542375438 (293h31m9.093845919s) [noise=true , address=185.167.204.205:4036 , piece_id=F652JXWWTK5XUJBY5SSU6IKKJE72DGITQXBRYK3LMQEF63DMRIEQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.543153891 (293h31m9.094624342s) [noise=true , address=185.167.204.205:4002 , piece_id=YFXJFZTNYWU3AZXQYDU5VBWJCPRVBCYCANXPQ77UGR5ZBY5VV53Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.543618846 (293h31m9.095089307s) [noise=true , address=185.167.204.206:4047 , piece_id=OHARKEAXGZLVG5XHD6RYKKWS57JFGH4JR2TAFIQSPUXTRMJD4WZA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.543662328 (293h31m9.095132789s) [noise=true , address=185.167.204.206:4066 , piece_id=T5X6MZEPB6P3DZ2HVMXCC4XJX5JJFXKGASJRWJDCK5TBHDEA2OLQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.544259055 (293h31m9.095729516s) [noise=true , address=185.167.204.205:4011 , piece_id=O2KXUNUHCKAQZSL5BJRRRC6IIZTSXHM74OZ7REQI3RGSMFA3PN6Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.544805565 (293h31m9.096276016s) [noise=true , address=185.167.204.206:4060 , piece_id=VJPM3YAQI5N2VTEBJ5GZ75JCCBOQS5IBEFUHEWTOX3BAKSP7TWFQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.545043088 (293h31m9.096513559s) [noise=true , address=185.167.204.205:4007 , piece_id=R6HJTNG5VAAMG2Y6YXK4K4YBQFUSAJQ4774UCOSGJSRDM4RWLICA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.545385921 (293h31m9.096856382s) [noise=true , address=185.167.204.205:4034 , piece_id=H66BDBLG4T2BABKBNKMJLY6QPZT2VVSZC3GW6F5V2L2BBFOWE76Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.759454013 (293h31m10.310924484s) [noise=true , address=89.149.201.147:4006 , piece_id=T2KROQWQR5LXKBJEJO6LICLBP5CUBETT3TOACRQCGYXKZ364TYMQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.779325021 (293h31m10.330795482s) [noise=true , address=89.149.201.147:4015 , piece_id=VIFNKNEVMCI2CFBREZOLVGFJ2X7Y7ZX6FWDC6PO6CLVX2LFW4FYQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.797229894 (293h31m10.348700345s) [noise=true , address=89.149.201.147:4021 , piece_id=K3ZPQXTSSUGIGIX33FCYOC6FRCTWDBRVRHUHHGEJLNWFCJFCIT4Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.804823945 (293h31m10.356294406s) [noise=true , address=89.149.201.147:4012 , piece_id=3QAKWEG37QPAE4AORUSYURK47YIOTM5VHC563QWC43UGGPS7UV7Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.828388066 (293h31m10.379858537s) [noise=true , address=89.149.201.147:4014 , piece_id=MM2F7NMVCD3AFX2UM7FRGSO5ZD2OTJPXJEUVKJ63IL5L74LEXZSQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.848656089 (293h31m10.40012655s) [noise=true , address=89.149.201.147:4001 , piece_id=OGRBX7OEEUPFXWU3EOYD44FGBQ7FQ5CHHZ2RVRHFJGOQ74EK7K5A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:02.180750503 (293h32m0.732220964s) [noise=true , address=81.171.14.70:4028 , piece_id=ESUHXDH256EM67DXYQ7O5THLHNAPPJ6I3ARZC5D3GSLJD7CNJV5A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:02.321725695 (293h32m0.873196156s) [noise=true , address=81.171.14.70:4044 , piece_id=KMR7RCY3AFLFIAJWIQXYC6ZSX7QLL6OKSCQMYHDKKCRX4MZAC4TQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:03.575396105 (293h32m2.126866556s) [noise=true , address=81.171.14.70:4041 , piece_id=N36O2DAKHTDWU4BPERULQHIUWQQDM5TQUQD7Y36XDA7SPPQML2LQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:05.676531843 (293h32m4.228002294s) [noise=true , address=81.171.14.70:4047 , piece_id=Q4N3CMZGTJGYFS35Y75GJHLQLEYMRIX4SSGTJXEHCCXIAX5Y2T3Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:08.329832254 (293h32m6.881302715s) [noise=true , address=81.171.14.70:4043 , piece_id=TQCRSBAQ4AJPCVP2JZYD4HQF375R2K7P5AIMCZW5EJVFJYUGKYGA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:08.628985817 (293h32m7.180456278s) [noise=true , address=81.171.14.70:4042 , piece_id=E5AMRUQ7TS2R6K7ILJ2YSG7UNGO4P5PFTOMHVF6FXOOIWW46Y2NA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12vBozgWFrZfUEQAwsqBt6CDMW47rVjZaDfvKTsCdPrvX8swkhD, address:81.171.14.70:4042): protocol: read tcp 188.166.225.170:40010->81.171.14.70:4042: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 21 13:36:08.629102389 (293h32m7.18057285s) [noise=true , address=81.171.14.70:4048 , piece_id=NUU26IUS2OGXAEVLUDEYTXJL4E6GZM3M7MP5BNBI6KTRWQ2RZT4Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12bbN67ZebXjF4iSyaWu4TDsG1XugTm1YdNHqRdRCbZ2WWRunRG, address:81.171.14.70:4048): protocol: write tcp 188.166.225.170:55932->81.171.14.70:4048: use of closed network connection; write tcp 188.166.225.170:55932->81.171.14.70:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55932->81.171.14.70:4048: use of closed network connection Nov 21 13:36:08.629502702 (293h32m7.180973163s) [noise=true , address=81.171.14.70:4031 , piece_id=LJW25GLWHJCV5CFUITJ5CPYDNWOFG3NBK4A5NUX2VVV4CYCDHWMQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12drdYnnWsuMyFo6LkcewAGGWnorRCAhxPpNqK41LvGy2zkBvng, address:81.171.14.70:4031): protocol: write tcp 188.166.225.170:58232->81.171.14.70:4031: use of closed network connection; write tcp 188.166.225.170:58232->81.171.14.70:4031: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58232->81.171.14.70:4031: use of closed network connection Nov 21 13:36:08.629974720 (293h32m7.181445181s) [noise=true , address=81.171.14.70:4026 , piece_id=ATEKV2WY2WDNBLY7W4THWSO52AJ76Q3XVT7QDGT3EBMPWVRL5EKA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12RTn3BuxasVSPYmjqgyhqKn1t6fAtq9EBtUufXqcLq48My9g2b, address:81.171.14.70:4026): protocol: write tcp 188.166.225.170:41806->81.171.14.70:4026: use of closed network connection; context deadline exceeded Nov 21 13:36:08.630048300 (293h32m7.181518751s) [seg_pos={0 0}, upload=258480]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil> Nov 21 13:36:08.800451489 (293h32m7.35192194s) [upload=258480]: Done uploading segments... Nov 21 13:36:08.800522093 (293h32m7.351992554s) [upload=258480]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24) Duration: 7.902210302s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 15 out of 60 is failed (threshold=6) [.X..X.X..X.X..XX.X.....X...............X.X.X......X...X..X..] duration: 4.801319273s trace: https://jaeger.dev.storj.io/trace/44be1beb24d3d90a FAIL (2025-11-21 13:51:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.801349398s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 15 out of 60 is failed (threshold=6) [..X..X.X..X.X..XX.X.....X...............X.X.X......X...X..X.] duration: 3.742770652s trace: https://jaeger.dev.storj.io/trace/44be1beb24d3d90a FAIL (2025-11-21 13:50:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.742799585s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 5.892991596s trace: https://jaeger.dev.storj.io/trace/2c4b010d23e46d91 FAIL (2025-11-21 13:50:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 24) ======LOGS====== AMRUQ7TS2R6K7ILJ2YSG7UNGO4P5PFTOMHVF6FXOOIWW46Y2NA, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.291418442 (293h31m7.842888914s) [noise=true , address=89.149.201.147:4006 , piece_id=T2KROQWQR5LXKBJEJO6LICLBP5CUBETT3TOACRQCGYXKZ364TYMQ, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.291954664 (293h31m7.843425145s) [noise=true , address=185.167.204.206:4048 , piece_id=6DBNGIBFWN5PPV72YWVRSSQVYF2AZ7EHS55PJDLDLGZEGXRYYHBA, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.293302360 (293h31m7.844772851s) [noise=true , address=89.149.201.147:4021 , piece_id=K3ZPQXTSSUGIGIX33FCYOC6FRCTWDBRVRHUHHGEJLNWFCJFCIT4Q, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.294979363 (293h31m7.846449854s) [noise=true , address=81.171.14.70:4028 , piece_id=ESUHXDH256EM67DXYQ7O5THLHNAPPJ6I3ARZC5D3GSLJD7CNJV5A, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.295760250 (293h31m7.847230711s) [noise=true , address=81.171.14.70:4048 , piece_id=NUU26IUS2OGXAEVLUDEYTXJL4E6GZM3M7MP5BNBI6KTRWQ2RZT4Q, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.300934223 (293h31m7.852404674s) [noise=true , address=81.171.14.70:4031 , piece_id=LJW25GLWHJCV5CFUITJ5CPYDNWOFG3NBK4A5NUX2VVV4CYCDHWMQ, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.301942341 (293h31m7.853412814s) [noise=true , address=185.167.204.205:4011 , piece_id=O2KXUNUHCKAQZSL5BJRRRC6IIZTSXHM74OZ7REQI3RGSMFA3PN6Q, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.302417767 (293h31m7.853888218s) [noise=true , address=185.167.204.205:4020 , piece_id=LPUWKZY6RPB7TGO22BL4YRA7ETB2RHWUFOXPJX7TZE3OHSFXEH7A, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.304119598 (293h31m7.855590059s) [noise=true , address=89.149.201.147:4014 , piece_id=MM2F7NMVCD3AFX2UM7FRGSO5ZD2OTJPXJEUVKJ63IL5L74LEXZSQ, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:10.220344333 (293h31m8.771814794s) [upload=258480]: Next returned nil segment Nov 21 13:35:10.220363840 (293h31m8.771834301s) [upload=258480]: Waiting for error group managing segments... Nov 21 13:35:10.536640979 (293h31m9.08811144s) [noise=true , address=185.167.204.205:4027 , piece_id=7C2DDLUHJQCI62D2YTI76NDABUZRSPK3FIZ7N45OG5GX3SZWCHNA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.537115091 (293h31m9.088585552s) [noise=true , address=185.167.204.205:4020 , piece_id=LPUWKZY6RPB7TGO22BL4YRA7ETB2RHWUFOXPJX7TZE3OHSFXEH7A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.539139225 (293h31m9.090609686s) [noise=true , address=185.167.204.206:4056 , piece_id=7VD7D3HW5YIZUYRZ5LWEB4VDXN2R6OEU7CTCGLKZUVKEC44V743Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.539668604 (293h31m9.091139065s) [noise=true , address=185.167.204.206:4048 , piece_id=6DBNGIBFWN5PPV72YWVRSSQVYF2AZ7EHS55PJDLDLGZEGXRYYHBA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.541238333 (293h31m9.092708794s) [noise=true , address=185.167.204.205:4008 , piece_id=UE3HOAD4KZRUWKY63RDOZ5LQ7VYAQPNV653CNSWCRXD2WL5MR27A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.542375438 (293h31m9.093845919s) [noise=true , address=185.167.204.205:4036 , piece_id=F652JXWWTK5XUJBY5SSU6IKKJE72DGITQXBRYK3LMQEF63DMRIEQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.543153891 (293h31m9.094624342s) [noise=true , address=185.167.204.205:4002 , piece_id=YFXJFZTNYWU3AZXQYDU5VBWJCPRVBCYCANXPQ77UGR5ZBY5VV53Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.543618846 (293h31m9.095089307s) [noise=true , address=185.167.204.206:4047 , piece_id=OHARKEAXGZLVG5XHD6RYKKWS57JFGH4JR2TAFIQSPUXTRMJD4WZA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.543662328 (293h31m9.095132789s) [noise=true , address=185.167.204.206:4066 , piece_id=T5X6MZEPB6P3DZ2HVMXCC4XJX5JJFXKGASJRWJDCK5TBHDEA2OLQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.544259055 (293h31m9.095729516s) [noise=true , address=185.167.204.205:4011 , piece_id=O2KXUNUHCKAQZSL5BJRRRC6IIZTSXHM74OZ7REQI3RGSMFA3PN6Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.544805565 (293h31m9.096276016s) [noise=true , address=185.167.204.206:4060 , piece_id=VJPM3YAQI5N2VTEBJ5GZ75JCCBOQS5IBEFUHEWTOX3BAKSP7TWFQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.545043088 (293h31m9.096513559s) [noise=true , address=185.167.204.205:4007 , piece_id=R6HJTNG5VAAMG2Y6YXK4K4YBQFUSAJQ4774UCOSGJSRDM4RWLICA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.545385921 (293h31m9.096856382s) [noise=true , address=185.167.204.205:4034 , piece_id=H66BDBLG4T2BABKBNKMJLY6QPZT2VVSZC3GW6F5V2L2BBFOWE76Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.759454013 (293h31m10.310924484s) [noise=true , address=89.149.201.147:4006 , piece_id=T2KROQWQR5LXKBJEJO6LICLBP5CUBETT3TOACRQCGYXKZ364TYMQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.779325021 (293h31m10.330795482s) [noise=true , address=89.149.201.147:4015 , piece_id=VIFNKNEVMCI2CFBREZOLVGFJ2X7Y7ZX6FWDC6PO6CLVX2LFW4FYQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.797229894 (293h31m10.348700345s) [noise=true , address=89.149.201.147:4021 , piece_id=K3ZPQXTSSUGIGIX33FCYOC6FRCTWDBRVRHUHHGEJLNWFCJFCIT4Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.804823945 (293h31m10.356294406s) [noise=true , address=89.149.201.147:4012 , piece_id=3QAKWEG37QPAE4AORUSYURK47YIOTM5VHC563QWC43UGGPS7UV7Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.828388066 (293h31m10.379858537s) [noise=true , address=89.149.201.147:4014 , piece_id=MM2F7NMVCD3AFX2UM7FRGSO5ZD2OTJPXJEUVKJ63IL5L74LEXZSQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.848656089 (293h31m10.40012655s) [noise=true , address=89.149.201.147:4001 , piece_id=OGRBX7OEEUPFXWU3EOYD44FGBQ7FQ5CHHZ2RVRHFJGOQ74EK7K5A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:02.180750503 (293h32m0.732220964s) [noise=true , address=81.171.14.70:4028 , piece_id=ESUHXDH256EM67DXYQ7O5THLHNAPPJ6I3ARZC5D3GSLJD7CNJV5A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:02.321725695 (293h32m0.873196156s) [noise=true , address=81.171.14.70:4044 , piece_id=KMR7RCY3AFLFIAJWIQXYC6ZSX7QLL6OKSCQMYHDKKCRX4MZAC4TQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:03.575396105 (293h32m2.126866556s) [noise=true , address=81.171.14.70:4041 , piece_id=N36O2DAKHTDWU4BPERULQHIUWQQDM5TQUQD7Y36XDA7SPPQML2LQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:05.676531843 (293h32m4.228002294s) [noise=true , address=81.171.14.70:4047 , piece_id=Q4N3CMZGTJGYFS35Y75GJHLQLEYMRIX4SSGTJXEHCCXIAX5Y2T3Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:08.329832254 (293h32m6.881302715s) [noise=true , address=81.171.14.70:4043 , piece_id=TQCRSBAQ4AJPCVP2JZYD4HQF375R2K7P5AIMCZW5EJVFJYUGKYGA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:08.628985817 (293h32m7.180456278s) [noise=true , address=81.171.14.70:4042 , piece_id=E5AMRUQ7TS2R6K7ILJ2YSG7UNGO4P5PFTOMHVF6FXOOIWW46Y2NA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12vBozgWFrZfUEQAwsqBt6CDMW47rVjZaDfvKTsCdPrvX8swkhD, address:81.171.14.70:4042): protocol: read tcp 188.166.225.170:40010->81.171.14.70:4042: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 21 13:36:08.629102389 (293h32m7.18057285s) [noise=true , address=81.171.14.70:4048 , piece_id=NUU26IUS2OGXAEVLUDEYTXJL4E6GZM3M7MP5BNBI6KTRWQ2RZT4Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12bbN67ZebXjF4iSyaWu4TDsG1XugTm1YdNHqRdRCbZ2WWRunRG, address:81.171.14.70:4048): protocol: write tcp 188.166.225.170:55932->81.171.14.70:4048: use of closed network connection; write tcp 188.166.225.170:55932->81.171.14.70:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55932->81.171.14.70:4048: use of closed network connection Nov 21 13:36:08.629502702 (293h32m7.180973163s) [noise=true , address=81.171.14.70:4031 , piece_id=LJW25GLWHJCV5CFUITJ5CPYDNWOFG3NBK4A5NUX2VVV4CYCDHWMQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12drdYnnWsuMyFo6LkcewAGGWnorRCAhxPpNqK41LvGy2zkBvng, address:81.171.14.70:4031): protocol: write tcp 188.166.225.170:58232->81.171.14.70:4031: use of closed network connection; write tcp 188.166.225.170:58232->81.171.14.70:4031: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58232->81.171.14.70:4031: use of closed network connection Nov 21 13:36:08.629974720 (293h32m7.181445181s) [noise=true , address=81.171.14.70:4026 , piece_id=ATEKV2WY2WDNBLY7W4THWSO52AJ76Q3XVT7QDGT3EBMPWVRL5EKA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12RTn3BuxasVSPYmjqgyhqKn1t6fAtq9EBtUufXqcLq48My9g2b, address:81.171.14.70:4026): protocol: write tcp 188.166.225.170:41806->81.171.14.70:4026: use of closed network connection; context deadline exceeded Nov 21 13:36:08.630048300 (293h32m7.181518751s) [seg_pos={0 0}, upload=258480]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil> Nov 21 13:36:08.800451489 (293h32m7.35192194s) [upload=258480]: Done uploading segments... Nov 21 13:36:08.800522093 (293h32m7.351992554s) [upload=258480]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24) Duration: 5.893020681s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 15 out of 60 is failed (threshold=6) [...X..X.X..X.X..XX.X.....X...............X.X.X......X...X..X] duration: 5.000891584s trace: https://jaeger.dev.storj.io/trace/44be1beb24d3d90a FAIL (2025-11-21 13:50:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000915879s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [....X..X.X..X.X..XX.X.....X...............X.X.X......X...X..] duration: 5.001528026s trace: https://jaeger.dev.storj.io/trace/1db1cf5e3de19d3e FAIL (2025-11-21 13:49:56 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001558593s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 5.348086117s trace: https://jaeger.dev.storj.io/trace/2c4b010d23e46d91 FAIL (2025-11-21 13:49:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 24) ======LOGS====== AMRUQ7TS2R6K7ILJ2YSG7UNGO4P5PFTOMHVF6FXOOIWW46Y2NA, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.291418442 (293h31m7.842888914s) [noise=true , address=89.149.201.147:4006 , piece_id=T2KROQWQR5LXKBJEJO6LICLBP5CUBETT3TOACRQCGYXKZ364TYMQ, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.291954664 (293h31m7.843425145s) [noise=true , address=185.167.204.206:4048 , piece_id=6DBNGIBFWN5PPV72YWVRSSQVYF2AZ7EHS55PJDLDLGZEGXRYYHBA, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.293302360 (293h31m7.844772851s) [noise=true , address=89.149.201.147:4021 , piece_id=K3ZPQXTSSUGIGIX33FCYOC6FRCTWDBRVRHUHHGEJLNWFCJFCIT4Q, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.294979363 (293h31m7.846449854s) [noise=true , address=81.171.14.70:4028 , piece_id=ESUHXDH256EM67DXYQ7O5THLHNAPPJ6I3ARZC5D3GSLJD7CNJV5A, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.295760250 (293h31m7.847230711s) [noise=true , address=81.171.14.70:4048 , piece_id=NUU26IUS2OGXAEVLUDEYTXJL4E6GZM3M7MP5BNBI6KTRWQ2RZT4Q, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.300934223 (293h31m7.852404674s) [noise=true , address=81.171.14.70:4031 , piece_id=LJW25GLWHJCV5CFUITJ5CPYDNWOFG3NBK4A5NUX2VVV4CYCDHWMQ, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.301942341 (293h31m7.853412814s) [noise=true , address=185.167.204.205:4011 , piece_id=O2KXUNUHCKAQZSL5BJRRRC6IIZTSXHM74OZ7REQI3RGSMFA3PN6Q, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.302417767 (293h31m7.853888218s) [noise=true , address=185.167.204.205:4020 , piece_id=LPUWKZY6RPB7TGO22BL4YRA7ETB2RHWUFOXPJX7TZE3OHSFXEH7A, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.304119598 (293h31m7.855590059s) [noise=true , address=89.149.201.147:4014 , piece_id=MM2F7NMVCD3AFX2UM7FRGSO5ZD2OTJPXJEUVKJ63IL5L74LEXZSQ, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:10.220344333 (293h31m8.771814794s) [upload=258480]: Next returned nil segment Nov 21 13:35:10.220363840 (293h31m8.771834301s) [upload=258480]: Waiting for error group managing segments... Nov 21 13:35:10.536640979 (293h31m9.08811144s) [noise=true , address=185.167.204.205:4027 , piece_id=7C2DDLUHJQCI62D2YTI76NDABUZRSPK3FIZ7N45OG5GX3SZWCHNA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.537115091 (293h31m9.088585552s) [noise=true , address=185.167.204.205:4020 , piece_id=LPUWKZY6RPB7TGO22BL4YRA7ETB2RHWUFOXPJX7TZE3OHSFXEH7A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.539139225 (293h31m9.090609686s) [noise=true , address=185.167.204.206:4056 , piece_id=7VD7D3HW5YIZUYRZ5LWEB4VDXN2R6OEU7CTCGLKZUVKEC44V743Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.539668604 (293h31m9.091139065s) [noise=true , address=185.167.204.206:4048 , piece_id=6DBNGIBFWN5PPV72YWVRSSQVYF2AZ7EHS55PJDLDLGZEGXRYYHBA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.541238333 (293h31m9.092708794s) [noise=true , address=185.167.204.205:4008 , piece_id=UE3HOAD4KZRUWKY63RDOZ5LQ7VYAQPNV653CNSWCRXD2WL5MR27A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.542375438 (293h31m9.093845919s) [noise=true , address=185.167.204.205:4036 , piece_id=F652JXWWTK5XUJBY5SSU6IKKJE72DGITQXBRYK3LMQEF63DMRIEQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.543153891 (293h31m9.094624342s) [noise=true , address=185.167.204.205:4002 , piece_id=YFXJFZTNYWU3AZXQYDU5VBWJCPRVBCYCANXPQ77UGR5ZBY5VV53Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.543618846 (293h31m9.095089307s) [noise=true , address=185.167.204.206:4047 , piece_id=OHARKEAXGZLVG5XHD6RYKKWS57JFGH4JR2TAFIQSPUXTRMJD4WZA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.543662328 (293h31m9.095132789s) [noise=true , address=185.167.204.206:4066 , piece_id=T5X6MZEPB6P3DZ2HVMXCC4XJX5JJFXKGASJRWJDCK5TBHDEA2OLQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.544259055 (293h31m9.095729516s) [noise=true , address=185.167.204.205:4011 , piece_id=O2KXUNUHCKAQZSL5BJRRRC6IIZTSXHM74OZ7REQI3RGSMFA3PN6Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.544805565 (293h31m9.096276016s) [noise=true , address=185.167.204.206:4060 , piece_id=VJPM3YAQI5N2VTEBJ5GZ75JCCBOQS5IBEFUHEWTOX3BAKSP7TWFQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.545043088 (293h31m9.096513559s) [noise=true , address=185.167.204.205:4007 , piece_id=R6HJTNG5VAAMG2Y6YXK4K4YBQFUSAJQ4774UCOSGJSRDM4RWLICA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.545385921 (293h31m9.096856382s) [noise=true , address=185.167.204.205:4034 , piece_id=H66BDBLG4T2BABKBNKMJLY6QPZT2VVSZC3GW6F5V2L2BBFOWE76Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.759454013 (293h31m10.310924484s) [noise=true , address=89.149.201.147:4006 , piece_id=T2KROQWQR5LXKBJEJO6LICLBP5CUBETT3TOACRQCGYXKZ364TYMQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.779325021 (293h31m10.330795482s) [noise=true , address=89.149.201.147:4015 , piece_id=VIFNKNEVMCI2CFBREZOLVGFJ2X7Y7ZX6FWDC6PO6CLVX2LFW4FYQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.797229894 (293h31m10.348700345s) [noise=true , address=89.149.201.147:4021 , piece_id=K3ZPQXTSSUGIGIX33FCYOC6FRCTWDBRVRHUHHGEJLNWFCJFCIT4Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.804823945 (293h31m10.356294406s) [noise=true , address=89.149.201.147:4012 , piece_id=3QAKWEG37QPAE4AORUSYURK47YIOTM5VHC563QWC43UGGPS7UV7Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.828388066 (293h31m10.379858537s) [noise=true , address=89.149.201.147:4014 , piece_id=MM2F7NMVCD3AFX2UM7FRGSO5ZD2OTJPXJEUVKJ63IL5L74LEXZSQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.848656089 (293h31m10.40012655s) [noise=true , address=89.149.201.147:4001 , piece_id=OGRBX7OEEUPFXWU3EOYD44FGBQ7FQ5CHHZ2RVRHFJGOQ74EK7K5A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:02.180750503 (293h32m0.732220964s) [noise=true , address=81.171.14.70:4028 , piece_id=ESUHXDH256EM67DXYQ7O5THLHNAPPJ6I3ARZC5D3GSLJD7CNJV5A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:02.321725695 (293h32m0.873196156s) [noise=true , address=81.171.14.70:4044 , piece_id=KMR7RCY3AFLFIAJWIQXYC6ZSX7QLL6OKSCQMYHDKKCRX4MZAC4TQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:03.575396105 (293h32m2.126866556s) [noise=true , address=81.171.14.70:4041 , piece_id=N36O2DAKHTDWU4BPERULQHIUWQQDM5TQUQD7Y36XDA7SPPQML2LQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:05.676531843 (293h32m4.228002294s) [noise=true , address=81.171.14.70:4047 , piece_id=Q4N3CMZGTJGYFS35Y75GJHLQLEYMRIX4SSGTJXEHCCXIAX5Y2T3Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:08.329832254 (293h32m6.881302715s) [noise=true , address=81.171.14.70:4043 , piece_id=TQCRSBAQ4AJPCVP2JZYD4HQF375R2K7P5AIMCZW5EJVFJYUGKYGA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:08.628985817 (293h32m7.180456278s) [noise=true , address=81.171.14.70:4042 , piece_id=E5AMRUQ7TS2R6K7ILJ2YSG7UNGO4P5PFTOMHVF6FXOOIWW46Y2NA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12vBozgWFrZfUEQAwsqBt6CDMW47rVjZaDfvKTsCdPrvX8swkhD, address:81.171.14.70:4042): protocol: read tcp 188.166.225.170:40010->81.171.14.70:4042: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 21 13:36:08.629102389 (293h32m7.18057285s) [noise=true , address=81.171.14.70:4048 , piece_id=NUU26IUS2OGXAEVLUDEYTXJL4E6GZM3M7MP5BNBI6KTRWQ2RZT4Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12bbN67ZebXjF4iSyaWu4TDsG1XugTm1YdNHqRdRCbZ2WWRunRG, address:81.171.14.70:4048): protocol: write tcp 188.166.225.170:55932->81.171.14.70:4048: use of closed network connection; write tcp 188.166.225.170:55932->81.171.14.70:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55932->81.171.14.70:4048: use of closed network connection Nov 21 13:36:08.629502702 (293h32m7.180973163s) [noise=true , address=81.171.14.70:4031 , piece_id=LJW25GLWHJCV5CFUITJ5CPYDNWOFG3NBK4A5NUX2VVV4CYCDHWMQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12drdYnnWsuMyFo6LkcewAGGWnorRCAhxPpNqK41LvGy2zkBvng, address:81.171.14.70:4031): protocol: write tcp 188.166.225.170:58232->81.171.14.70:4031: use of closed network connection; write tcp 188.166.225.170:58232->81.171.14.70:4031: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58232->81.171.14.70:4031: use of closed network connection Nov 21 13:36:08.629974720 (293h32m7.181445181s) [noise=true , address=81.171.14.70:4026 , piece_id=ATEKV2WY2WDNBLY7W4THWSO52AJ76Q3XVT7QDGT3EBMPWVRL5EKA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12RTn3BuxasVSPYmjqgyhqKn1t6fAtq9EBtUufXqcLq48My9g2b, address:81.171.14.70:4026): protocol: write tcp 188.166.225.170:41806->81.171.14.70:4026: use of closed network connection; context deadline exceeded Nov 21 13:36:08.630048300 (293h32m7.181518751s) [seg_pos={0 0}, upload=258480]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil> Nov 21 13:36:08.800451489 (293h32m7.35192194s) [upload=258480]: Done uploading segments... Nov 21 13:36:08.800522093 (293h32m7.351992554s) [upload=258480]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24) Duration: 5.348485258s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 5.943998803s trace: https://jaeger.dev.storj.io/trace/2c4b010d23e46d91 FAIL (2025-11-21 13:49:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 24) ======LOGS====== AMRUQ7TS2R6K7ILJ2YSG7UNGO4P5PFTOMHVF6FXOOIWW46Y2NA, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.291418442 (293h31m7.842888914s) [noise=true , address=89.149.201.147:4006 , piece_id=T2KROQWQR5LXKBJEJO6LICLBP5CUBETT3TOACRQCGYXKZ364TYMQ, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.291954664 (293h31m7.843425145s) [noise=true , address=185.167.204.206:4048 , piece_id=6DBNGIBFWN5PPV72YWVRSSQVYF2AZ7EHS55PJDLDLGZEGXRYYHBA, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.293302360 (293h31m7.844772851s) [noise=true , address=89.149.201.147:4021 , piece_id=K3ZPQXTSSUGIGIX33FCYOC6FRCTWDBRVRHUHHGEJLNWFCJFCIT4Q, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.294979363 (293h31m7.846449854s) [noise=true , address=81.171.14.70:4028 , piece_id=ESUHXDH256EM67DXYQ7O5THLHNAPPJ6I3ARZC5D3GSLJD7CNJV5A, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.295760250 (293h31m7.847230711s) [noise=true , address=81.171.14.70:4048 , piece_id=NUU26IUS2OGXAEVLUDEYTXJL4E6GZM3M7MP5BNBI6KTRWQ2RZT4Q, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.300934223 (293h31m7.852404674s) [noise=true , address=81.171.14.70:4031 , piece_id=LJW25GLWHJCV5CFUITJ5CPYDNWOFG3NBK4A5NUX2VVV4CYCDHWMQ, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.301942341 (293h31m7.853412814s) [noise=true , address=185.167.204.205:4011 , piece_id=O2KXUNUHCKAQZSL5BJRRRC6IIZTSXHM74OZ7REQI3RGSMFA3PN6Q, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.302417767 (293h31m7.853888218s) [noise=true , address=185.167.204.205:4020 , piece_id=LPUWKZY6RPB7TGO22BL4YRA7ETB2RHWUFOXPJX7TZE3OHSFXEH7A, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.304119598 (293h31m7.855590059s) [noise=true , address=89.149.201.147:4014 , piece_id=MM2F7NMVCD3AFX2UM7FRGSO5ZD2OTJPXJEUVKJ63IL5L74LEXZSQ, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:10.220344333 (293h31m8.771814794s) [upload=258480]: Next returned nil segment Nov 21 13:35:10.220363840 (293h31m8.771834301s) [upload=258480]: Waiting for error group managing segments... Nov 21 13:35:10.536640979 (293h31m9.08811144s) [noise=true , address=185.167.204.205:4027 , piece_id=7C2DDLUHJQCI62D2YTI76NDABUZRSPK3FIZ7N45OG5GX3SZWCHNA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.537115091 (293h31m9.088585552s) [noise=true , address=185.167.204.205:4020 , piece_id=LPUWKZY6RPB7TGO22BL4YRA7ETB2RHWUFOXPJX7TZE3OHSFXEH7A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.539139225 (293h31m9.090609686s) [noise=true , address=185.167.204.206:4056 , piece_id=7VD7D3HW5YIZUYRZ5LWEB4VDXN2R6OEU7CTCGLKZUVKEC44V743Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.539668604 (293h31m9.091139065s) [noise=true , address=185.167.204.206:4048 , piece_id=6DBNGIBFWN5PPV72YWVRSSQVYF2AZ7EHS55PJDLDLGZEGXRYYHBA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.541238333 (293h31m9.092708794s) [noise=true , address=185.167.204.205:4008 , piece_id=UE3HOAD4KZRUWKY63RDOZ5LQ7VYAQPNV653CNSWCRXD2WL5MR27A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.542375438 (293h31m9.093845919s) [noise=true , address=185.167.204.205:4036 , piece_id=F652JXWWTK5XUJBY5SSU6IKKJE72DGITQXBRYK3LMQEF63DMRIEQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.543153891 (293h31m9.094624342s) [noise=true , address=185.167.204.205:4002 , piece_id=YFXJFZTNYWU3AZXQYDU5VBWJCPRVBCYCANXPQ77UGR5ZBY5VV53Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.543618846 (293h31m9.095089307s) [noise=true , address=185.167.204.206:4047 , piece_id=OHARKEAXGZLVG5XHD6RYKKWS57JFGH4JR2TAFIQSPUXTRMJD4WZA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.543662328 (293h31m9.095132789s) [noise=true , address=185.167.204.206:4066 , piece_id=T5X6MZEPB6P3DZ2HVMXCC4XJX5JJFXKGASJRWJDCK5TBHDEA2OLQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.544259055 (293h31m9.095729516s) [noise=true , address=185.167.204.205:4011 , piece_id=O2KXUNUHCKAQZSL5BJRRRC6IIZTSXHM74OZ7REQI3RGSMFA3PN6Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.544805565 (293h31m9.096276016s) [noise=true , address=185.167.204.206:4060 , piece_id=VJPM3YAQI5N2VTEBJ5GZ75JCCBOQS5IBEFUHEWTOX3BAKSP7TWFQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.545043088 (293h31m9.096513559s) [noise=true , address=185.167.204.205:4007 , piece_id=R6HJTNG5VAAMG2Y6YXK4K4YBQFUSAJQ4774UCOSGJSRDM4RWLICA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.545385921 (293h31m9.096856382s) [noise=true , address=185.167.204.205:4034 , piece_id=H66BDBLG4T2BABKBNKMJLY6QPZT2VVSZC3GW6F5V2L2BBFOWE76Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.759454013 (293h31m10.310924484s) [noise=true , address=89.149.201.147:4006 , piece_id=T2KROQWQR5LXKBJEJO6LICLBP5CUBETT3TOACRQCGYXKZ364TYMQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.779325021 (293h31m10.330795482s) [noise=true , address=89.149.201.147:4015 , piece_id=VIFNKNEVMCI2CFBREZOLVGFJ2X7Y7ZX6FWDC6PO6CLVX2LFW4FYQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.797229894 (293h31m10.348700345s) [noise=true , address=89.149.201.147:4021 , piece_id=K3ZPQXTSSUGIGIX33FCYOC6FRCTWDBRVRHUHHGEJLNWFCJFCIT4Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.804823945 (293h31m10.356294406s) [noise=true , address=89.149.201.147:4012 , piece_id=3QAKWEG37QPAE4AORUSYURK47YIOTM5VHC563QWC43UGGPS7UV7Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.828388066 (293h31m10.379858537s) [noise=true , address=89.149.201.147:4014 , piece_id=MM2F7NMVCD3AFX2UM7FRGSO5ZD2OTJPXJEUVKJ63IL5L74LEXZSQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.848656089 (293h31m10.40012655s) [noise=true , address=89.149.201.147:4001 , piece_id=OGRBX7OEEUPFXWU3EOYD44FGBQ7FQ5CHHZ2RVRHFJGOQ74EK7K5A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:02.180750503 (293h32m0.732220964s) [noise=true , address=81.171.14.70:4028 , piece_id=ESUHXDH256EM67DXYQ7O5THLHNAPPJ6I3ARZC5D3GSLJD7CNJV5A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:02.321725695 (293h32m0.873196156s) [noise=true , address=81.171.14.70:4044 , piece_id=KMR7RCY3AFLFIAJWIQXYC6ZSX7QLL6OKSCQMYHDKKCRX4MZAC4TQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:03.575396105 (293h32m2.126866556s) [noise=true , address=81.171.14.70:4041 , piece_id=N36O2DAKHTDWU4BPERULQHIUWQQDM5TQUQD7Y36XDA7SPPQML2LQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:05.676531843 (293h32m4.228002294s) [noise=true , address=81.171.14.70:4047 , piece_id=Q4N3CMZGTJGYFS35Y75GJHLQLEYMRIX4SSGTJXEHCCXIAX5Y2T3Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:08.329832254 (293h32m6.881302715s) [noise=true , address=81.171.14.70:4043 , piece_id=TQCRSBAQ4AJPCVP2JZYD4HQF375R2K7P5AIMCZW5EJVFJYUGKYGA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:08.628985817 (293h32m7.180456278s) [noise=true , address=81.171.14.70:4042 , piece_id=E5AMRUQ7TS2R6K7ILJ2YSG7UNGO4P5PFTOMHVF6FXOOIWW46Y2NA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12vBozgWFrZfUEQAwsqBt6CDMW47rVjZaDfvKTsCdPrvX8swkhD, address:81.171.14.70:4042): protocol: read tcp 188.166.225.170:40010->81.171.14.70:4042: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 21 13:36:08.629102389 (293h32m7.18057285s) [noise=true , address=81.171.14.70:4048 , piece_id=NUU26IUS2OGXAEVLUDEYTXJL4E6GZM3M7MP5BNBI6KTRWQ2RZT4Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12bbN67ZebXjF4iSyaWu4TDsG1XugTm1YdNHqRdRCbZ2WWRunRG, address:81.171.14.70:4048): protocol: write tcp 188.166.225.170:55932->81.171.14.70:4048: use of closed network connection; write tcp 188.166.225.170:55932->81.171.14.70:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55932->81.171.14.70:4048: use of closed network connection Nov 21 13:36:08.629502702 (293h32m7.180973163s) [noise=true , address=81.171.14.70:4031 , piece_id=LJW25GLWHJCV5CFUITJ5CPYDNWOFG3NBK4A5NUX2VVV4CYCDHWMQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12drdYnnWsuMyFo6LkcewAGGWnorRCAhxPpNqK41LvGy2zkBvng, address:81.171.14.70:4031): protocol: write tcp 188.166.225.170:58232->81.171.14.70:4031: use of closed network connection; write tcp 188.166.225.170:58232->81.171.14.70:4031: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58232->81.171.14.70:4031: use of closed network connection Nov 21 13:36:08.629974720 (293h32m7.181445181s) [noise=true , address=81.171.14.70:4026 , piece_id=ATEKV2WY2WDNBLY7W4THWSO52AJ76Q3XVT7QDGT3EBMPWVRL5EKA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12RTn3BuxasVSPYmjqgyhqKn1t6fAtq9EBtUufXqcLq48My9g2b, address:81.171.14.70:4026): protocol: write tcp 188.166.225.170:41806->81.171.14.70:4026: use of closed network connection; context deadline exceeded Nov 21 13:36:08.630048300 (293h32m7.181518751s) [seg_pos={0 0}, upload=258480]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil> Nov 21 13:36:08.800451489 (293h32m7.35192194s) [upload=258480]: Done uploading segments... Nov 21 13:36:08.800522093 (293h32m7.351992554s) [upload=258480]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24) Duration: 5.944030333s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 7.025147218s trace: https://jaeger.dev.storj.io/trace/2c4b010d23e46d91 FAIL (2025-11-21 13:49:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 24) ======LOGS====== AMRUQ7TS2R6K7ILJ2YSG7UNGO4P5PFTOMHVF6FXOOIWW46Y2NA, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.291418442 (293h31m7.842888914s) [noise=true , address=89.149.201.147:4006 , piece_id=T2KROQWQR5LXKBJEJO6LICLBP5CUBETT3TOACRQCGYXKZ364TYMQ, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.291954664 (293h31m7.843425145s) [noise=true , address=185.167.204.206:4048 , piece_id=6DBNGIBFWN5PPV72YWVRSSQVYF2AZ7EHS55PJDLDLGZEGXRYYHBA, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.293302360 (293h31m7.844772851s) [noise=true , address=89.149.201.147:4021 , piece_id=K3ZPQXTSSUGIGIX33FCYOC6FRCTWDBRVRHUHHGEJLNWFCJFCIT4Q, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.294979363 (293h31m7.846449854s) [noise=true , address=81.171.14.70:4028 , piece_id=ESUHXDH256EM67DXYQ7O5THLHNAPPJ6I3ARZC5D3GSLJD7CNJV5A, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.295760250 (293h31m7.847230711s) [noise=true , address=81.171.14.70:4048 , piece_id=NUU26IUS2OGXAEVLUDEYTXJL4E6GZM3M7MP5BNBI6KTRWQ2RZT4Q, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.300934223 (293h31m7.852404674s) [noise=true , address=81.171.14.70:4031 , piece_id=LJW25GLWHJCV5CFUITJ5CPYDNWOFG3NBK4A5NUX2VVV4CYCDHWMQ, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.301942341 (293h31m7.853412814s) [noise=true , address=185.167.204.205:4011 , piece_id=O2KXUNUHCKAQZSL5BJRRRC6IIZTSXHM74OZ7REQI3RGSMFA3PN6Q, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.302417767 (293h31m7.853888218s) [noise=true , address=185.167.204.205:4020 , piece_id=LPUWKZY6RPB7TGO22BL4YRA7ETB2RHWUFOXPJX7TZE3OHSFXEH7A, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.304119598 (293h31m7.855590059s) [noise=true , address=89.149.201.147:4014 , piece_id=MM2F7NMVCD3AFX2UM7FRGSO5ZD2OTJPXJEUVKJ63IL5L74LEXZSQ, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:10.220344333 (293h31m8.771814794s) [upload=258480]: Next returned nil segment Nov 21 13:35:10.220363840 (293h31m8.771834301s) [upload=258480]: Waiting for error group managing segments... Nov 21 13:35:10.536640979 (293h31m9.08811144s) [noise=true , address=185.167.204.205:4027 , piece_id=7C2DDLUHJQCI62D2YTI76NDABUZRSPK3FIZ7N45OG5GX3SZWCHNA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.537115091 (293h31m9.088585552s) [noise=true , address=185.167.204.205:4020 , piece_id=LPUWKZY6RPB7TGO22BL4YRA7ETB2RHWUFOXPJX7TZE3OHSFXEH7A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.539139225 (293h31m9.090609686s) [noise=true , address=185.167.204.206:4056 , piece_id=7VD7D3HW5YIZUYRZ5LWEB4VDXN2R6OEU7CTCGLKZUVKEC44V743Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.539668604 (293h31m9.091139065s) [noise=true , address=185.167.204.206:4048 , piece_id=6DBNGIBFWN5PPV72YWVRSSQVYF2AZ7EHS55PJDLDLGZEGXRYYHBA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.541238333 (293h31m9.092708794s) [noise=true , address=185.167.204.205:4008 , piece_id=UE3HOAD4KZRUWKY63RDOZ5LQ7VYAQPNV653CNSWCRXD2WL5MR27A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.542375438 (293h31m9.093845919s) [noise=true , address=185.167.204.205:4036 , piece_id=F652JXWWTK5XUJBY5SSU6IKKJE72DGITQXBRYK3LMQEF63DMRIEQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.543153891 (293h31m9.094624342s) [noise=true , address=185.167.204.205:4002 , piece_id=YFXJFZTNYWU3AZXQYDU5VBWJCPRVBCYCANXPQ77UGR5ZBY5VV53Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.543618846 (293h31m9.095089307s) [noise=true , address=185.167.204.206:4047 , piece_id=OHARKEAXGZLVG5XHD6RYKKWS57JFGH4JR2TAFIQSPUXTRMJD4WZA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.543662328 (293h31m9.095132789s) [noise=true , address=185.167.204.206:4066 , piece_id=T5X6MZEPB6P3DZ2HVMXCC4XJX5JJFXKGASJRWJDCK5TBHDEA2OLQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.544259055 (293h31m9.095729516s) [noise=true , address=185.167.204.205:4011 , piece_id=O2KXUNUHCKAQZSL5BJRRRC6IIZTSXHM74OZ7REQI3RGSMFA3PN6Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.544805565 (293h31m9.096276016s) [noise=true , address=185.167.204.206:4060 , piece_id=VJPM3YAQI5N2VTEBJ5GZ75JCCBOQS5IBEFUHEWTOX3BAKSP7TWFQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.545043088 (293h31m9.096513559s) [noise=true , address=185.167.204.205:4007 , piece_id=R6HJTNG5VAAMG2Y6YXK4K4YBQFUSAJQ4774UCOSGJSRDM4RWLICA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.545385921 (293h31m9.096856382s) [noise=true , address=185.167.204.205:4034 , piece_id=H66BDBLG4T2BABKBNKMJLY6QPZT2VVSZC3GW6F5V2L2BBFOWE76Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.759454013 (293h31m10.310924484s) [noise=true , address=89.149.201.147:4006 , piece_id=T2KROQWQR5LXKBJEJO6LICLBP5CUBETT3TOACRQCGYXKZ364TYMQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.779325021 (293h31m10.330795482s) [noise=true , address=89.149.201.147:4015 , piece_id=VIFNKNEVMCI2CFBREZOLVGFJ2X7Y7ZX6FWDC6PO6CLVX2LFW4FYQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.797229894 (293h31m10.348700345s) [noise=true , address=89.149.201.147:4021 , piece_id=K3ZPQXTSSUGIGIX33FCYOC6FRCTWDBRVRHUHHGEJLNWFCJFCIT4Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.804823945 (293h31m10.356294406s) [noise=true , address=89.149.201.147:4012 , piece_id=3QAKWEG37QPAE4AORUSYURK47YIOTM5VHC563QWC43UGGPS7UV7Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.828388066 (293h31m10.379858537s) [noise=true , address=89.149.201.147:4014 , piece_id=MM2F7NMVCD3AFX2UM7FRGSO5ZD2OTJPXJEUVKJ63IL5L74LEXZSQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.848656089 (293h31m10.40012655s) [noise=true , address=89.149.201.147:4001 , piece_id=OGRBX7OEEUPFXWU3EOYD44FGBQ7FQ5CHHZ2RVRHFJGOQ74EK7K5A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:02.180750503 (293h32m0.732220964s) [noise=true , address=81.171.14.70:4028 , piece_id=ESUHXDH256EM67DXYQ7O5THLHNAPPJ6I3ARZC5D3GSLJD7CNJV5A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:02.321725695 (293h32m0.873196156s) [noise=true , address=81.171.14.70:4044 , piece_id=KMR7RCY3AFLFIAJWIQXYC6ZSX7QLL6OKSCQMYHDKKCRX4MZAC4TQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:03.575396105 (293h32m2.126866556s) [noise=true , address=81.171.14.70:4041 , piece_id=N36O2DAKHTDWU4BPERULQHIUWQQDM5TQUQD7Y36XDA7SPPQML2LQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:05.676531843 (293h32m4.228002294s) [noise=true , address=81.171.14.70:4047 , piece_id=Q4N3CMZGTJGYFS35Y75GJHLQLEYMRIX4SSGTJXEHCCXIAX5Y2T3Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:08.329832254 (293h32m6.881302715s) [noise=true , address=81.171.14.70:4043 , piece_id=TQCRSBAQ4AJPCVP2JZYD4HQF375R2K7P5AIMCZW5EJVFJYUGKYGA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:08.628985817 (293h32m7.180456278s) [noise=true , address=81.171.14.70:4042 , piece_id=E5AMRUQ7TS2R6K7ILJ2YSG7UNGO4P5PFTOMHVF6FXOOIWW46Y2NA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12vBozgWFrZfUEQAwsqBt6CDMW47rVjZaDfvKTsCdPrvX8swkhD, address:81.171.14.70:4042): protocol: read tcp 188.166.225.170:40010->81.171.14.70:4042: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 21 13:36:08.629102389 (293h32m7.18057285s) [noise=true , address=81.171.14.70:4048 , piece_id=NUU26IUS2OGXAEVLUDEYTXJL4E6GZM3M7MP5BNBI6KTRWQ2RZT4Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12bbN67ZebXjF4iSyaWu4TDsG1XugTm1YdNHqRdRCbZ2WWRunRG, address:81.171.14.70:4048): protocol: write tcp 188.166.225.170:55932->81.171.14.70:4048: use of closed network connection; write tcp 188.166.225.170:55932->81.171.14.70:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55932->81.171.14.70:4048: use of closed network connection Nov 21 13:36:08.629502702 (293h32m7.180973163s) [noise=true , address=81.171.14.70:4031 , piece_id=LJW25GLWHJCV5CFUITJ5CPYDNWOFG3NBK4A5NUX2VVV4CYCDHWMQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12drdYnnWsuMyFo6LkcewAGGWnorRCAhxPpNqK41LvGy2zkBvng, address:81.171.14.70:4031): protocol: write tcp 188.166.225.170:58232->81.171.14.70:4031: use of closed network connection; write tcp 188.166.225.170:58232->81.171.14.70:4031: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58232->81.171.14.70:4031: use of closed network connection Nov 21 13:36:08.629974720 (293h32m7.181445181s) [noise=true , address=81.171.14.70:4026 , piece_id=ATEKV2WY2WDNBLY7W4THWSO52AJ76Q3XVT7QDGT3EBMPWVRL5EKA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12RTn3BuxasVSPYmjqgyhqKn1t6fAtq9EBtUufXqcLq48My9g2b, address:81.171.14.70:4026): protocol: write tcp 188.166.225.170:41806->81.171.14.70:4026: use of closed network connection; context deadline exceeded Nov 21 13:36:08.630048300 (293h32m7.181518751s) [seg_pos={0 0}, upload=258480]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil> Nov 21 13:36:08.800451489 (293h32m7.35192194s) [upload=258480]: Done uploading segments... Nov 21 13:36:08.800522093 (293h32m7.351992554s) [upload=258480]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24) Duration: 7.025174631s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 15 out of 60 is failed (threshold=6) [X....X..X.X..X.X..XX.X.....X...............X.X.X......X...X.] duration: 5.000707975s trace: https://jaeger.dev.storj.io/trace/1db1cf5e3de19d3e FAIL (2025-11-21 13:48:52 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000733234s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 4.876118484s trace: https://jaeger.dev.storj.io/trace/2c4b010d23e46d91 FAIL (2025-11-21 13:48:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 24) ======LOGS====== AMRUQ7TS2R6K7ILJ2YSG7UNGO4P5PFTOMHVF6FXOOIWW46Y2NA, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.291418442 (293h31m7.842888914s) [noise=true , address=89.149.201.147:4006 , piece_id=T2KROQWQR5LXKBJEJO6LICLBP5CUBETT3TOACRQCGYXKZ364TYMQ, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.291954664 (293h31m7.843425145s) [noise=true , address=185.167.204.206:4048 , piece_id=6DBNGIBFWN5PPV72YWVRSSQVYF2AZ7EHS55PJDLDLGZEGXRYYHBA, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.293302360 (293h31m7.844772851s) [noise=true , address=89.149.201.147:4021 , piece_id=K3ZPQXTSSUGIGIX33FCYOC6FRCTWDBRVRHUHHGEJLNWFCJFCIT4Q, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.294979363 (293h31m7.846449854s) [noise=true , address=81.171.14.70:4028 , piece_id=ESUHXDH256EM67DXYQ7O5THLHNAPPJ6I3ARZC5D3GSLJD7CNJV5A, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.295760250 (293h31m7.847230711s) [noise=true , address=81.171.14.70:4048 , piece_id=NUU26IUS2OGXAEVLUDEYTXJL4E6GZM3M7MP5BNBI6KTRWQ2RZT4Q, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.300934223 (293h31m7.852404674s) [noise=true , address=81.171.14.70:4031 , piece_id=LJW25GLWHJCV5CFUITJ5CPYDNWOFG3NBK4A5NUX2VVV4CYCDHWMQ, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.301942341 (293h31m7.853412814s) [noise=true , address=185.167.204.205:4011 , piece_id=O2KXUNUHCKAQZSL5BJRRRC6IIZTSXHM74OZ7REQI3RGSMFA3PN6Q, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.302417767 (293h31m7.853888218s) [noise=true , address=185.167.204.205:4020 , piece_id=LPUWKZY6RPB7TGO22BL4YRA7ETB2RHWUFOXPJX7TZE3OHSFXEH7A, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.304119598 (293h31m7.855590059s) [noise=true , address=89.149.201.147:4014 , piece_id=MM2F7NMVCD3AFX2UM7FRGSO5ZD2OTJPXJEUVKJ63IL5L74LEXZSQ, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:10.220344333 (293h31m8.771814794s) [upload=258480]: Next returned nil segment Nov 21 13:35:10.220363840 (293h31m8.771834301s) [upload=258480]: Waiting for error group managing segments... Nov 21 13:35:10.536640979 (293h31m9.08811144s) [noise=true , address=185.167.204.205:4027 , piece_id=7C2DDLUHJQCI62D2YTI76NDABUZRSPK3FIZ7N45OG5GX3SZWCHNA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.537115091 (293h31m9.088585552s) [noise=true , address=185.167.204.205:4020 , piece_id=LPUWKZY6RPB7TGO22BL4YRA7ETB2RHWUFOXPJX7TZE3OHSFXEH7A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.539139225 (293h31m9.090609686s) [noise=true , address=185.167.204.206:4056 , piece_id=7VD7D3HW5YIZUYRZ5LWEB4VDXN2R6OEU7CTCGLKZUVKEC44V743Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.539668604 (293h31m9.091139065s) [noise=true , address=185.167.204.206:4048 , piece_id=6DBNGIBFWN5PPV72YWVRSSQVYF2AZ7EHS55PJDLDLGZEGXRYYHBA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.541238333 (293h31m9.092708794s) [noise=true , address=185.167.204.205:4008 , piece_id=UE3HOAD4KZRUWKY63RDOZ5LQ7VYAQPNV653CNSWCRXD2WL5MR27A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.542375438 (293h31m9.093845919s) [noise=true , address=185.167.204.205:4036 , piece_id=F652JXWWTK5XUJBY5SSU6IKKJE72DGITQXBRYK3LMQEF63DMRIEQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.543153891 (293h31m9.094624342s) [noise=true , address=185.167.204.205:4002 , piece_id=YFXJFZTNYWU3AZXQYDU5VBWJCPRVBCYCANXPQ77UGR5ZBY5VV53Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.543618846 (293h31m9.095089307s) [noise=true , address=185.167.204.206:4047 , piece_id=OHARKEAXGZLVG5XHD6RYKKWS57JFGH4JR2TAFIQSPUXTRMJD4WZA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.543662328 (293h31m9.095132789s) [noise=true , address=185.167.204.206:4066 , piece_id=T5X6MZEPB6P3DZ2HVMXCC4XJX5JJFXKGASJRWJDCK5TBHDEA2OLQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.544259055 (293h31m9.095729516s) [noise=true , address=185.167.204.205:4011 , piece_id=O2KXUNUHCKAQZSL5BJRRRC6IIZTSXHM74OZ7REQI3RGSMFA3PN6Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.544805565 (293h31m9.096276016s) [noise=true , address=185.167.204.206:4060 , piece_id=VJPM3YAQI5N2VTEBJ5GZ75JCCBOQS5IBEFUHEWTOX3BAKSP7TWFQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.545043088 (293h31m9.096513559s) [noise=true , address=185.167.204.205:4007 , piece_id=R6HJTNG5VAAMG2Y6YXK4K4YBQFUSAJQ4774UCOSGJSRDM4RWLICA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.545385921 (293h31m9.096856382s) [noise=true , address=185.167.204.205:4034 , piece_id=H66BDBLG4T2BABKBNKMJLY6QPZT2VVSZC3GW6F5V2L2BBFOWE76Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.759454013 (293h31m10.310924484s) [noise=true , address=89.149.201.147:4006 , piece_id=T2KROQWQR5LXKBJEJO6LICLBP5CUBETT3TOACRQCGYXKZ364TYMQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.779325021 (293h31m10.330795482s) [noise=true , address=89.149.201.147:4015 , piece_id=VIFNKNEVMCI2CFBREZOLVGFJ2X7Y7ZX6FWDC6PO6CLVX2LFW4FYQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.797229894 (293h31m10.348700345s) [noise=true , address=89.149.201.147:4021 , piece_id=K3ZPQXTSSUGIGIX33FCYOC6FRCTWDBRVRHUHHGEJLNWFCJFCIT4Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.804823945 (293h31m10.356294406s) [noise=true , address=89.149.201.147:4012 , piece_id=3QAKWEG37QPAE4AORUSYURK47YIOTM5VHC563QWC43UGGPS7UV7Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.828388066 (293h31m10.379858537s) [noise=true , address=89.149.201.147:4014 , piece_id=MM2F7NMVCD3AFX2UM7FRGSO5ZD2OTJPXJEUVKJ63IL5L74LEXZSQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.848656089 (293h31m10.40012655s) [noise=true , address=89.149.201.147:4001 , piece_id=OGRBX7OEEUPFXWU3EOYD44FGBQ7FQ5CHHZ2RVRHFJGOQ74EK7K5A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:02.180750503 (293h32m0.732220964s) [noise=true , address=81.171.14.70:4028 , piece_id=ESUHXDH256EM67DXYQ7O5THLHNAPPJ6I3ARZC5D3GSLJD7CNJV5A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:02.321725695 (293h32m0.873196156s) [noise=true , address=81.171.14.70:4044 , piece_id=KMR7RCY3AFLFIAJWIQXYC6ZSX7QLL6OKSCQMYHDKKCRX4MZAC4TQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:03.575396105 (293h32m2.126866556s) [noise=true , address=81.171.14.70:4041 , piece_id=N36O2DAKHTDWU4BPERULQHIUWQQDM5TQUQD7Y36XDA7SPPQML2LQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:05.676531843 (293h32m4.228002294s) [noise=true , address=81.171.14.70:4047 , piece_id=Q4N3CMZGTJGYFS35Y75GJHLQLEYMRIX4SSGTJXEHCCXIAX5Y2T3Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:08.329832254 (293h32m6.881302715s) [noise=true , address=81.171.14.70:4043 , piece_id=TQCRSBAQ4AJPCVP2JZYD4HQF375R2K7P5AIMCZW5EJVFJYUGKYGA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:08.628985817 (293h32m7.180456278s) [noise=true , address=81.171.14.70:4042 , piece_id=E5AMRUQ7TS2R6K7ILJ2YSG7UNGO4P5PFTOMHVF6FXOOIWW46Y2NA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12vBozgWFrZfUEQAwsqBt6CDMW47rVjZaDfvKTsCdPrvX8swkhD, address:81.171.14.70:4042): protocol: read tcp 188.166.225.170:40010->81.171.14.70:4042: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 21 13:36:08.629102389 (293h32m7.18057285s) [noise=true , address=81.171.14.70:4048 , piece_id=NUU26IUS2OGXAEVLUDEYTXJL4E6GZM3M7MP5BNBI6KTRWQ2RZT4Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12bbN67ZebXjF4iSyaWu4TDsG1XugTm1YdNHqRdRCbZ2WWRunRG, address:81.171.14.70:4048): protocol: write tcp 188.166.225.170:55932->81.171.14.70:4048: use of closed network connection; write tcp 188.166.225.170:55932->81.171.14.70:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55932->81.171.14.70:4048: use of closed network connection Nov 21 13:36:08.629502702 (293h32m7.180973163s) [noise=true , address=81.171.14.70:4031 , piece_id=LJW25GLWHJCV5CFUITJ5CPYDNWOFG3NBK4A5NUX2VVV4CYCDHWMQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12drdYnnWsuMyFo6LkcewAGGWnorRCAhxPpNqK41LvGy2zkBvng, address:81.171.14.70:4031): protocol: write tcp 188.166.225.170:58232->81.171.14.70:4031: use of closed network connection; write tcp 188.166.225.170:58232->81.171.14.70:4031: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58232->81.171.14.70:4031: use of closed network connection Nov 21 13:36:08.629974720 (293h32m7.181445181s) [noise=true , address=81.171.14.70:4026 , piece_id=ATEKV2WY2WDNBLY7W4THWSO52AJ76Q3XVT7QDGT3EBMPWVRL5EKA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12RTn3BuxasVSPYmjqgyhqKn1t6fAtq9EBtUufXqcLq48My9g2b, address:81.171.14.70:4026): protocol: write tcp 188.166.225.170:41806->81.171.14.70:4026: use of closed network connection; context deadline exceeded Nov 21 13:36:08.630048300 (293h32m7.181518751s) [seg_pos={0 0}, upload=258480]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil> Nov 21 13:36:08.800451489 (293h32m7.35192194s) [upload=258480]: Done uploading segments... Nov 21 13:36:08.800522093 (293h32m7.351992554s) [upload=258480]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24) Duration: 4.876143822s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 5.89653765s trace: https://jaeger.dev.storj.io/trace/2c4b010d23e46d91 FAIL (2025-11-21 13:47:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 24) ======LOGS====== AMRUQ7TS2R6K7ILJ2YSG7UNGO4P5PFTOMHVF6FXOOIWW46Y2NA, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.291418442 (293h31m7.842888914s) [noise=true , address=89.149.201.147:4006 , piece_id=T2KROQWQR5LXKBJEJO6LICLBP5CUBETT3TOACRQCGYXKZ364TYMQ, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.291954664 (293h31m7.843425145s) [noise=true , address=185.167.204.206:4048 , piece_id=6DBNGIBFWN5PPV72YWVRSSQVYF2AZ7EHS55PJDLDLGZEGXRYYHBA, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.293302360 (293h31m7.844772851s) [noise=true , address=89.149.201.147:4021 , piece_id=K3ZPQXTSSUGIGIX33FCYOC6FRCTWDBRVRHUHHGEJLNWFCJFCIT4Q, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.294979363 (293h31m7.846449854s) [noise=true , address=81.171.14.70:4028 , piece_id=ESUHXDH256EM67DXYQ7O5THLHNAPPJ6I3ARZC5D3GSLJD7CNJV5A, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.295760250 (293h31m7.847230711s) [noise=true , address=81.171.14.70:4048 , piece_id=NUU26IUS2OGXAEVLUDEYTXJL4E6GZM3M7MP5BNBI6KTRWQ2RZT4Q, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.300934223 (293h31m7.852404674s) [noise=true , address=81.171.14.70:4031 , piece_id=LJW25GLWHJCV5CFUITJ5CPYDNWOFG3NBK4A5NUX2VVV4CYCDHWMQ, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.301942341 (293h31m7.853412814s) [noise=true , address=185.167.204.205:4011 , piece_id=O2KXUNUHCKAQZSL5BJRRRC6IIZTSXHM74OZ7REQI3RGSMFA3PN6Q, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.302417767 (293h31m7.853888218s) [noise=true , address=185.167.204.205:4020 , piece_id=LPUWKZY6RPB7TGO22BL4YRA7ETB2RHWUFOXPJX7TZE3OHSFXEH7A, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.304119598 (293h31m7.855590059s) [noise=true , address=89.149.201.147:4014 , piece_id=MM2F7NMVCD3AFX2UM7FRGSO5ZD2OTJPXJEUVKJ63IL5L74LEXZSQ, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:10.220344333 (293h31m8.771814794s) [upload=258480]: Next returned nil segment Nov 21 13:35:10.220363840 (293h31m8.771834301s) [upload=258480]: Waiting for error group managing segments... Nov 21 13:35:10.536640979 (293h31m9.08811144s) [noise=true , address=185.167.204.205:4027 , piece_id=7C2DDLUHJQCI62D2YTI76NDABUZRSPK3FIZ7N45OG5GX3SZWCHNA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.537115091 (293h31m9.088585552s) [noise=true , address=185.167.204.205:4020 , piece_id=LPUWKZY6RPB7TGO22BL4YRA7ETB2RHWUFOXPJX7TZE3OHSFXEH7A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.539139225 (293h31m9.090609686s) [noise=true , address=185.167.204.206:4056 , piece_id=7VD7D3HW5YIZUYRZ5LWEB4VDXN2R6OEU7CTCGLKZUVKEC44V743Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.539668604 (293h31m9.091139065s) [noise=true , address=185.167.204.206:4048 , piece_id=6DBNGIBFWN5PPV72YWVRSSQVYF2AZ7EHS55PJDLDLGZEGXRYYHBA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.541238333 (293h31m9.092708794s) [noise=true , address=185.167.204.205:4008 , piece_id=UE3HOAD4KZRUWKY63RDOZ5LQ7VYAQPNV653CNSWCRXD2WL5MR27A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.542375438 (293h31m9.093845919s) [noise=true , address=185.167.204.205:4036 , piece_id=F652JXWWTK5XUJBY5SSU6IKKJE72DGITQXBRYK3LMQEF63DMRIEQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.543153891 (293h31m9.094624342s) [noise=true , address=185.167.204.205:4002 , piece_id=YFXJFZTNYWU3AZXQYDU5VBWJCPRVBCYCANXPQ77UGR5ZBY5VV53Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.543618846 (293h31m9.095089307s) [noise=true , address=185.167.204.206:4047 , piece_id=OHARKEAXGZLVG5XHD6RYKKWS57JFGH4JR2TAFIQSPUXTRMJD4WZA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.543662328 (293h31m9.095132789s) [noise=true , address=185.167.204.206:4066 , piece_id=T5X6MZEPB6P3DZ2HVMXCC4XJX5JJFXKGASJRWJDCK5TBHDEA2OLQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.544259055 (293h31m9.095729516s) [noise=true , address=185.167.204.205:4011 , piece_id=O2KXUNUHCKAQZSL5BJRRRC6IIZTSXHM74OZ7REQI3RGSMFA3PN6Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.544805565 (293h31m9.096276016s) [noise=true , address=185.167.204.206:4060 , piece_id=VJPM3YAQI5N2VTEBJ5GZ75JCCBOQS5IBEFUHEWTOX3BAKSP7TWFQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.545043088 (293h31m9.096513559s) [noise=true , address=185.167.204.205:4007 , piece_id=R6HJTNG5VAAMG2Y6YXK4K4YBQFUSAJQ4774UCOSGJSRDM4RWLICA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.545385921 (293h31m9.096856382s) [noise=true , address=185.167.204.205:4034 , piece_id=H66BDBLG4T2BABKBNKMJLY6QPZT2VVSZC3GW6F5V2L2BBFOWE76Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.759454013 (293h31m10.310924484s) [noise=true , address=89.149.201.147:4006 , piece_id=T2KROQWQR5LXKBJEJO6LICLBP5CUBETT3TOACRQCGYXKZ364TYMQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.779325021 (293h31m10.330795482s) [noise=true , address=89.149.201.147:4015 , piece_id=VIFNKNEVMCI2CFBREZOLVGFJ2X7Y7ZX6FWDC6PO6CLVX2LFW4FYQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.797229894 (293h31m10.348700345s) [noise=true , address=89.149.201.147:4021 , piece_id=K3ZPQXTSSUGIGIX33FCYOC6FRCTWDBRVRHUHHGEJLNWFCJFCIT4Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.804823945 (293h31m10.356294406s) [noise=true , address=89.149.201.147:4012 , piece_id=3QAKWEG37QPAE4AORUSYURK47YIOTM5VHC563QWC43UGGPS7UV7Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.828388066 (293h31m10.379858537s) [noise=true , address=89.149.201.147:4014 , piece_id=MM2F7NMVCD3AFX2UM7FRGSO5ZD2OTJPXJEUVKJ63IL5L74LEXZSQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.848656089 (293h31m10.40012655s) [noise=true , address=89.149.201.147:4001 , piece_id=OGRBX7OEEUPFXWU3EOYD44FGBQ7FQ5CHHZ2RVRHFJGOQ74EK7K5A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:02.180750503 (293h32m0.732220964s) [noise=true , address=81.171.14.70:4028 , piece_id=ESUHXDH256EM67DXYQ7O5THLHNAPPJ6I3ARZC5D3GSLJD7CNJV5A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:02.321725695 (293h32m0.873196156s) [noise=true , address=81.171.14.70:4044 , piece_id=KMR7RCY3AFLFIAJWIQXYC6ZSX7QLL6OKSCQMYHDKKCRX4MZAC4TQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:03.575396105 (293h32m2.126866556s) [noise=true , address=81.171.14.70:4041 , piece_id=N36O2DAKHTDWU4BPERULQHIUWQQDM5TQUQD7Y36XDA7SPPQML2LQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:05.676531843 (293h32m4.228002294s) [noise=true , address=81.171.14.70:4047 , piece_id=Q4N3CMZGTJGYFS35Y75GJHLQLEYMRIX4SSGTJXEHCCXIAX5Y2T3Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:08.329832254 (293h32m6.881302715s) [noise=true , address=81.171.14.70:4043 , piece_id=TQCRSBAQ4AJPCVP2JZYD4HQF375R2K7P5AIMCZW5EJVFJYUGKYGA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:08.628985817 (293h32m7.180456278s) [noise=true , address=81.171.14.70:4042 , piece_id=E5AMRUQ7TS2R6K7ILJ2YSG7UNGO4P5PFTOMHVF6FXOOIWW46Y2NA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12vBozgWFrZfUEQAwsqBt6CDMW47rVjZaDfvKTsCdPrvX8swkhD, address:81.171.14.70:4042): protocol: read tcp 188.166.225.170:40010->81.171.14.70:4042: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 21 13:36:08.629102389 (293h32m7.18057285s) [noise=true , address=81.171.14.70:4048 , piece_id=NUU26IUS2OGXAEVLUDEYTXJL4E6GZM3M7MP5BNBI6KTRWQ2RZT4Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12bbN67ZebXjF4iSyaWu4TDsG1XugTm1YdNHqRdRCbZ2WWRunRG, address:81.171.14.70:4048): protocol: write tcp 188.166.225.170:55932->81.171.14.70:4048: use of closed network connection; write tcp 188.166.225.170:55932->81.171.14.70:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55932->81.171.14.70:4048: use of closed network connection Nov 21 13:36:08.629502702 (293h32m7.180973163s) [noise=true , address=81.171.14.70:4031 , piece_id=LJW25GLWHJCV5CFUITJ5CPYDNWOFG3NBK4A5NUX2VVV4CYCDHWMQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12drdYnnWsuMyFo6LkcewAGGWnorRCAhxPpNqK41LvGy2zkBvng, address:81.171.14.70:4031): protocol: write tcp 188.166.225.170:58232->81.171.14.70:4031: use of closed network connection; write tcp 188.166.225.170:58232->81.171.14.70:4031: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58232->81.171.14.70:4031: use of closed network connection Nov 21 13:36:08.629974720 (293h32m7.181445181s) [noise=true , address=81.171.14.70:4026 , piece_id=ATEKV2WY2WDNBLY7W4THWSO52AJ76Q3XVT7QDGT3EBMPWVRL5EKA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12RTn3BuxasVSPYmjqgyhqKn1t6fAtq9EBtUufXqcLq48My9g2b, address:81.171.14.70:4026): protocol: write tcp 188.166.225.170:41806->81.171.14.70:4026: use of closed network connection; context deadline exceeded Nov 21 13:36:08.630048300 (293h32m7.181518751s) [seg_pos={0 0}, upload=258480]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil> Nov 21 13:36:08.800451489 (293h32m7.35192194s) [upload=258480]: Done uploading segments... Nov 21 13:36:08.800522093 (293h32m7.351992554s) [upload=258480]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24) Duration: 5.896572565s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 15 out of 60 is failed (threshold=6) [.X....X..X.X..X.X..XX.X.....X...............X.X.X......X...X] duration: 5.001607449s trace: https://jaeger.dev.storj.io/trace/1db1cf5e3de19d3e FAIL (2025-11-21 13:47:38 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001630311s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [..X....X..X.X..X.X..XX.X.....X...............X.X.X......X...] duration: 3.722310715s trace: https://jaeger.dev.storj.io/trace/38e3ebf1da07c4a1 FAIL (2025-11-21 13:47:11 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.722341814s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 5.564725228s trace: https://jaeger.dev.storj.io/trace/2c4b010d23e46d91 FAIL (2025-11-21 13:47:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 24) ======LOGS====== AMRUQ7TS2R6K7ILJ2YSG7UNGO4P5PFTOMHVF6FXOOIWW46Y2NA, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.291418442 (293h31m7.842888914s) [noise=true , address=89.149.201.147:4006 , piece_id=T2KROQWQR5LXKBJEJO6LICLBP5CUBETT3TOACRQCGYXKZ364TYMQ, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.291954664 (293h31m7.843425145s) [noise=true , address=185.167.204.206:4048 , piece_id=6DBNGIBFWN5PPV72YWVRSSQVYF2AZ7EHS55PJDLDLGZEGXRYYHBA, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.293302360 (293h31m7.844772851s) [noise=true , address=89.149.201.147:4021 , piece_id=K3ZPQXTSSUGIGIX33FCYOC6FRCTWDBRVRHUHHGEJLNWFCJFCIT4Q, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.294979363 (293h31m7.846449854s) [noise=true , address=81.171.14.70:4028 , piece_id=ESUHXDH256EM67DXYQ7O5THLHNAPPJ6I3ARZC5D3GSLJD7CNJV5A, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.295760250 (293h31m7.847230711s) [noise=true , address=81.171.14.70:4048 , piece_id=NUU26IUS2OGXAEVLUDEYTXJL4E6GZM3M7MP5BNBI6KTRWQ2RZT4Q, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.300934223 (293h31m7.852404674s) [noise=true , address=81.171.14.70:4031 , piece_id=LJW25GLWHJCV5CFUITJ5CPYDNWOFG3NBK4A5NUX2VVV4CYCDHWMQ, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.301942341 (293h31m7.853412814s) [noise=true , address=185.167.204.205:4011 , piece_id=O2KXUNUHCKAQZSL5BJRRRC6IIZTSXHM74OZ7REQI3RGSMFA3PN6Q, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.302417767 (293h31m7.853888218s) [noise=true , address=185.167.204.205:4020 , piece_id=LPUWKZY6RPB7TGO22BL4YRA7ETB2RHWUFOXPJX7TZE3OHSFXEH7A, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.304119598 (293h31m7.855590059s) [noise=true , address=89.149.201.147:4014 , piece_id=MM2F7NMVCD3AFX2UM7FRGSO5ZD2OTJPXJEUVKJ63IL5L74LEXZSQ, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:10.220344333 (293h31m8.771814794s) [upload=258480]: Next returned nil segment Nov 21 13:35:10.220363840 (293h31m8.771834301s) [upload=258480]: Waiting for error group managing segments... Nov 21 13:35:10.536640979 (293h31m9.08811144s) [noise=true , address=185.167.204.205:4027 , piece_id=7C2DDLUHJQCI62D2YTI76NDABUZRSPK3FIZ7N45OG5GX3SZWCHNA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.537115091 (293h31m9.088585552s) [noise=true , address=185.167.204.205:4020 , piece_id=LPUWKZY6RPB7TGO22BL4YRA7ETB2RHWUFOXPJX7TZE3OHSFXEH7A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.539139225 (293h31m9.090609686s) [noise=true , address=185.167.204.206:4056 , piece_id=7VD7D3HW5YIZUYRZ5LWEB4VDXN2R6OEU7CTCGLKZUVKEC44V743Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.539668604 (293h31m9.091139065s) [noise=true , address=185.167.204.206:4048 , piece_id=6DBNGIBFWN5PPV72YWVRSSQVYF2AZ7EHS55PJDLDLGZEGXRYYHBA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.541238333 (293h31m9.092708794s) [noise=true , address=185.167.204.205:4008 , piece_id=UE3HOAD4KZRUWKY63RDOZ5LQ7VYAQPNV653CNSWCRXD2WL5MR27A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.542375438 (293h31m9.093845919s) [noise=true , address=185.167.204.205:4036 , piece_id=F652JXWWTK5XUJBY5SSU6IKKJE72DGITQXBRYK3LMQEF63DMRIEQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.543153891 (293h31m9.094624342s) [noise=true , address=185.167.204.205:4002 , piece_id=YFXJFZTNYWU3AZXQYDU5VBWJCPRVBCYCANXPQ77UGR5ZBY5VV53Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.543618846 (293h31m9.095089307s) [noise=true , address=185.167.204.206:4047 , piece_id=OHARKEAXGZLVG5XHD6RYKKWS57JFGH4JR2TAFIQSPUXTRMJD4WZA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.543662328 (293h31m9.095132789s) [noise=true , address=185.167.204.206:4066 , piece_id=T5X6MZEPB6P3DZ2HVMXCC4XJX5JJFXKGASJRWJDCK5TBHDEA2OLQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.544259055 (293h31m9.095729516s) [noise=true , address=185.167.204.205:4011 , piece_id=O2KXUNUHCKAQZSL5BJRRRC6IIZTSXHM74OZ7REQI3RGSMFA3PN6Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.544805565 (293h31m9.096276016s) [noise=true , address=185.167.204.206:4060 , piece_id=VJPM3YAQI5N2VTEBJ5GZ75JCCBOQS5IBEFUHEWTOX3BAKSP7TWFQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.545043088 (293h31m9.096513559s) [noise=true , address=185.167.204.205:4007 , piece_id=R6HJTNG5VAAMG2Y6YXK4K4YBQFUSAJQ4774UCOSGJSRDM4RWLICA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.545385921 (293h31m9.096856382s) [noise=true , address=185.167.204.205:4034 , piece_id=H66BDBLG4T2BABKBNKMJLY6QPZT2VVSZC3GW6F5V2L2BBFOWE76Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.759454013 (293h31m10.310924484s) [noise=true , address=89.149.201.147:4006 , piece_id=T2KROQWQR5LXKBJEJO6LICLBP5CUBETT3TOACRQCGYXKZ364TYMQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.779325021 (293h31m10.330795482s) [noise=true , address=89.149.201.147:4015 , piece_id=VIFNKNEVMCI2CFBREZOLVGFJ2X7Y7ZX6FWDC6PO6CLVX2LFW4FYQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.797229894 (293h31m10.348700345s) [noise=true , address=89.149.201.147:4021 , piece_id=K3ZPQXTSSUGIGIX33FCYOC6FRCTWDBRVRHUHHGEJLNWFCJFCIT4Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.804823945 (293h31m10.356294406s) [noise=true , address=89.149.201.147:4012 , piece_id=3QAKWEG37QPAE4AORUSYURK47YIOTM5VHC563QWC43UGGPS7UV7Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.828388066 (293h31m10.379858537s) [noise=true , address=89.149.201.147:4014 , piece_id=MM2F7NMVCD3AFX2UM7FRGSO5ZD2OTJPXJEUVKJ63IL5L74LEXZSQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.848656089 (293h31m10.40012655s) [noise=true , address=89.149.201.147:4001 , piece_id=OGRBX7OEEUPFXWU3EOYD44FGBQ7FQ5CHHZ2RVRHFJGOQ74EK7K5A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:02.180750503 (293h32m0.732220964s) [noise=true , address=81.171.14.70:4028 , piece_id=ESUHXDH256EM67DXYQ7O5THLHNAPPJ6I3ARZC5D3GSLJD7CNJV5A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:02.321725695 (293h32m0.873196156s) [noise=true , address=81.171.14.70:4044 , piece_id=KMR7RCY3AFLFIAJWIQXYC6ZSX7QLL6OKSCQMYHDKKCRX4MZAC4TQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:03.575396105 (293h32m2.126866556s) [noise=true , address=81.171.14.70:4041 , piece_id=N36O2DAKHTDWU4BPERULQHIUWQQDM5TQUQD7Y36XDA7SPPQML2LQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:05.676531843 (293h32m4.228002294s) [noise=true , address=81.171.14.70:4047 , piece_id=Q4N3CMZGTJGYFS35Y75GJHLQLEYMRIX4SSGTJXEHCCXIAX5Y2T3Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:08.329832254 (293h32m6.881302715s) [noise=true , address=81.171.14.70:4043 , piece_id=TQCRSBAQ4AJPCVP2JZYD4HQF375R2K7P5AIMCZW5EJVFJYUGKYGA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:08.628985817 (293h32m7.180456278s) [noise=true , address=81.171.14.70:4042 , piece_id=E5AMRUQ7TS2R6K7ILJ2YSG7UNGO4P5PFTOMHVF6FXOOIWW46Y2NA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12vBozgWFrZfUEQAwsqBt6CDMW47rVjZaDfvKTsCdPrvX8swkhD, address:81.171.14.70:4042): protocol: read tcp 188.166.225.170:40010->81.171.14.70:4042: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 21 13:36:08.629102389 (293h32m7.18057285s) [noise=true , address=81.171.14.70:4048 , piece_id=NUU26IUS2OGXAEVLUDEYTXJL4E6GZM3M7MP5BNBI6KTRWQ2RZT4Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12bbN67ZebXjF4iSyaWu4TDsG1XugTm1YdNHqRdRCbZ2WWRunRG, address:81.171.14.70:4048): protocol: write tcp 188.166.225.170:55932->81.171.14.70:4048: use of closed network connection; write tcp 188.166.225.170:55932->81.171.14.70:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55932->81.171.14.70:4048: use of closed network connection Nov 21 13:36:08.629502702 (293h32m7.180973163s) [noise=true , address=81.171.14.70:4031 , piece_id=LJW25GLWHJCV5CFUITJ5CPYDNWOFG3NBK4A5NUX2VVV4CYCDHWMQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12drdYnnWsuMyFo6LkcewAGGWnorRCAhxPpNqK41LvGy2zkBvng, address:81.171.14.70:4031): protocol: write tcp 188.166.225.170:58232->81.171.14.70:4031: use of closed network connection; write tcp 188.166.225.170:58232->81.171.14.70:4031: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58232->81.171.14.70:4031: use of closed network connection Nov 21 13:36:08.629974720 (293h32m7.181445181s) [noise=true , address=81.171.14.70:4026 , piece_id=ATEKV2WY2WDNBLY7W4THWSO52AJ76Q3XVT7QDGT3EBMPWVRL5EKA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12RTn3BuxasVSPYmjqgyhqKn1t6fAtq9EBtUufXqcLq48My9g2b, address:81.171.14.70:4026): protocol: write tcp 188.166.225.170:41806->81.171.14.70:4026: use of closed network connection; context deadline exceeded Nov 21 13:36:08.630048300 (293h32m7.181518751s) [seg_pos={0 0}, upload=258480]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil> Nov 21 13:36:08.800451489 (293h32m7.35192194s) [upload=258480]: Done uploading segments... Nov 21 13:36:08.800522093 (293h32m7.351992554s) [upload=258480]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24) Duration: 5.564762038s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [...X....X..X.X..X.X..XX.X.....X...............X.X.X......X..] duration: 3.740458952s trace: https://jaeger.dev.storj.io/trace/38e3ebf1da07c4a1 FAIL (2025-11-21 13:46:53 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.740488579s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 5.752052442s trace: https://jaeger.dev.storj.io/trace/2c4b010d23e46d91 FAIL (2025-11-21 13:46:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 24) ======LOGS====== AMRUQ7TS2R6K7ILJ2YSG7UNGO4P5PFTOMHVF6FXOOIWW46Y2NA, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.291418442 (293h31m7.842888914s) [noise=true , address=89.149.201.147:4006 , piece_id=T2KROQWQR5LXKBJEJO6LICLBP5CUBETT3TOACRQCGYXKZ364TYMQ, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.291954664 (293h31m7.843425145s) [noise=true , address=185.167.204.206:4048 , piece_id=6DBNGIBFWN5PPV72YWVRSSQVYF2AZ7EHS55PJDLDLGZEGXRYYHBA, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.293302360 (293h31m7.844772851s) [noise=true , address=89.149.201.147:4021 , piece_id=K3ZPQXTSSUGIGIX33FCYOC6FRCTWDBRVRHUHHGEJLNWFCJFCIT4Q, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.294979363 (293h31m7.846449854s) [noise=true , address=81.171.14.70:4028 , piece_id=ESUHXDH256EM67DXYQ7O5THLHNAPPJ6I3ARZC5D3GSLJD7CNJV5A, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.295760250 (293h31m7.847230711s) [noise=true , address=81.171.14.70:4048 , piece_id=NUU26IUS2OGXAEVLUDEYTXJL4E6GZM3M7MP5BNBI6KTRWQ2RZT4Q, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.300934223 (293h31m7.852404674s) [noise=true , address=81.171.14.70:4031 , piece_id=LJW25GLWHJCV5CFUITJ5CPYDNWOFG3NBK4A5NUX2VVV4CYCDHWMQ, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.301942341 (293h31m7.853412814s) [noise=true , address=185.167.204.205:4011 , piece_id=O2KXUNUHCKAQZSL5BJRRRC6IIZTSXHM74OZ7REQI3RGSMFA3PN6Q, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.302417767 (293h31m7.853888218s) [noise=true , address=185.167.204.205:4020 , piece_id=LPUWKZY6RPB7TGO22BL4YRA7ETB2RHWUFOXPJX7TZE3OHSFXEH7A, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.304119598 (293h31m7.855590059s) [noise=true , address=89.149.201.147:4014 , piece_id=MM2F7NMVCD3AFX2UM7FRGSO5ZD2OTJPXJEUVKJ63IL5L74LEXZSQ, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:10.220344333 (293h31m8.771814794s) [upload=258480]: Next returned nil segment Nov 21 13:35:10.220363840 (293h31m8.771834301s) [upload=258480]: Waiting for error group managing segments... Nov 21 13:35:10.536640979 (293h31m9.08811144s) [noise=true , address=185.167.204.205:4027 , piece_id=7C2DDLUHJQCI62D2YTI76NDABUZRSPK3FIZ7N45OG5GX3SZWCHNA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.537115091 (293h31m9.088585552s) [noise=true , address=185.167.204.205:4020 , piece_id=LPUWKZY6RPB7TGO22BL4YRA7ETB2RHWUFOXPJX7TZE3OHSFXEH7A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.539139225 (293h31m9.090609686s) [noise=true , address=185.167.204.206:4056 , piece_id=7VD7D3HW5YIZUYRZ5LWEB4VDXN2R6OEU7CTCGLKZUVKEC44V743Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.539668604 (293h31m9.091139065s) [noise=true , address=185.167.204.206:4048 , piece_id=6DBNGIBFWN5PPV72YWVRSSQVYF2AZ7EHS55PJDLDLGZEGXRYYHBA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.541238333 (293h31m9.092708794s) [noise=true , address=185.167.204.205:4008 , piece_id=UE3HOAD4KZRUWKY63RDOZ5LQ7VYAQPNV653CNSWCRXD2WL5MR27A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.542375438 (293h31m9.093845919s) [noise=true , address=185.167.204.205:4036 , piece_id=F652JXWWTK5XUJBY5SSU6IKKJE72DGITQXBRYK3LMQEF63DMRIEQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.543153891 (293h31m9.094624342s) [noise=true , address=185.167.204.205:4002 , piece_id=YFXJFZTNYWU3AZXQYDU5VBWJCPRVBCYCANXPQ77UGR5ZBY5VV53Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.543618846 (293h31m9.095089307s) [noise=true , address=185.167.204.206:4047 , piece_id=OHARKEAXGZLVG5XHD6RYKKWS57JFGH4JR2TAFIQSPUXTRMJD4WZA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.543662328 (293h31m9.095132789s) [noise=true , address=185.167.204.206:4066 , piece_id=T5X6MZEPB6P3DZ2HVMXCC4XJX5JJFXKGASJRWJDCK5TBHDEA2OLQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.544259055 (293h31m9.095729516s) [noise=true , address=185.167.204.205:4011 , piece_id=O2KXUNUHCKAQZSL5BJRRRC6IIZTSXHM74OZ7REQI3RGSMFA3PN6Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.544805565 (293h31m9.096276016s) [noise=true , address=185.167.204.206:4060 , piece_id=VJPM3YAQI5N2VTEBJ5GZ75JCCBOQS5IBEFUHEWTOX3BAKSP7TWFQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.545043088 (293h31m9.096513559s) [noise=true , address=185.167.204.205:4007 , piece_id=R6HJTNG5VAAMG2Y6YXK4K4YBQFUSAJQ4774UCOSGJSRDM4RWLICA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.545385921 (293h31m9.096856382s) [noise=true , address=185.167.204.205:4034 , piece_id=H66BDBLG4T2BABKBNKMJLY6QPZT2VVSZC3GW6F5V2L2BBFOWE76Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.759454013 (293h31m10.310924484s) [noise=true , address=89.149.201.147:4006 , piece_id=T2KROQWQR5LXKBJEJO6LICLBP5CUBETT3TOACRQCGYXKZ364TYMQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.779325021 (293h31m10.330795482s) [noise=true , address=89.149.201.147:4015 , piece_id=VIFNKNEVMCI2CFBREZOLVGFJ2X7Y7ZX6FWDC6PO6CLVX2LFW4FYQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.797229894 (293h31m10.348700345s) [noise=true , address=89.149.201.147:4021 , piece_id=K3ZPQXTSSUGIGIX33FCYOC6FRCTWDBRVRHUHHGEJLNWFCJFCIT4Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.804823945 (293h31m10.356294406s) [noise=true , address=89.149.201.147:4012 , piece_id=3QAKWEG37QPAE4AORUSYURK47YIOTM5VHC563QWC43UGGPS7UV7Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.828388066 (293h31m10.379858537s) [noise=true , address=89.149.201.147:4014 , piece_id=MM2F7NMVCD3AFX2UM7FRGSO5ZD2OTJPXJEUVKJ63IL5L74LEXZSQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.848656089 (293h31m10.40012655s) [noise=true , address=89.149.201.147:4001 , piece_id=OGRBX7OEEUPFXWU3EOYD44FGBQ7FQ5CHHZ2RVRHFJGOQ74EK7K5A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:02.180750503 (293h32m0.732220964s) [noise=true , address=81.171.14.70:4028 , piece_id=ESUHXDH256EM67DXYQ7O5THLHNAPPJ6I3ARZC5D3GSLJD7CNJV5A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:02.321725695 (293h32m0.873196156s) [noise=true , address=81.171.14.70:4044 , piece_id=KMR7RCY3AFLFIAJWIQXYC6ZSX7QLL6OKSCQMYHDKKCRX4MZAC4TQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:03.575396105 (293h32m2.126866556s) [noise=true , address=81.171.14.70:4041 , piece_id=N36O2DAKHTDWU4BPERULQHIUWQQDM5TQUQD7Y36XDA7SPPQML2LQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:05.676531843 (293h32m4.228002294s) [noise=true , address=81.171.14.70:4047 , piece_id=Q4N3CMZGTJGYFS35Y75GJHLQLEYMRIX4SSGTJXEHCCXIAX5Y2T3Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:08.329832254 (293h32m6.881302715s) [noise=true , address=81.171.14.70:4043 , piece_id=TQCRSBAQ4AJPCVP2JZYD4HQF375R2K7P5AIMCZW5EJVFJYUGKYGA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:08.628985817 (293h32m7.180456278s) [noise=true , address=81.171.14.70:4042 , piece_id=E5AMRUQ7TS2R6K7ILJ2YSG7UNGO4P5PFTOMHVF6FXOOIWW46Y2NA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12vBozgWFrZfUEQAwsqBt6CDMW47rVjZaDfvKTsCdPrvX8swkhD, address:81.171.14.70:4042): protocol: read tcp 188.166.225.170:40010->81.171.14.70:4042: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 21 13:36:08.629102389 (293h32m7.18057285s) [noise=true , address=81.171.14.70:4048 , piece_id=NUU26IUS2OGXAEVLUDEYTXJL4E6GZM3M7MP5BNBI6KTRWQ2RZT4Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12bbN67ZebXjF4iSyaWu4TDsG1XugTm1YdNHqRdRCbZ2WWRunRG, address:81.171.14.70:4048): protocol: write tcp 188.166.225.170:55932->81.171.14.70:4048: use of closed network connection; write tcp 188.166.225.170:55932->81.171.14.70:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55932->81.171.14.70:4048: use of closed network connection Nov 21 13:36:08.629502702 (293h32m7.180973163s) [noise=true , address=81.171.14.70:4031 , piece_id=LJW25GLWHJCV5CFUITJ5CPYDNWOFG3NBK4A5NUX2VVV4CYCDHWMQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12drdYnnWsuMyFo6LkcewAGGWnorRCAhxPpNqK41LvGy2zkBvng, address:81.171.14.70:4031): protocol: write tcp 188.166.225.170:58232->81.171.14.70:4031: use of closed network connection; write tcp 188.166.225.170:58232->81.171.14.70:4031: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58232->81.171.14.70:4031: use of closed network connection Nov 21 13:36:08.629974720 (293h32m7.181445181s) [noise=true , address=81.171.14.70:4026 , piece_id=ATEKV2WY2WDNBLY7W4THWSO52AJ76Q3XVT7QDGT3EBMPWVRL5EKA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12RTn3BuxasVSPYmjqgyhqKn1t6fAtq9EBtUufXqcLq48My9g2b, address:81.171.14.70:4026): protocol: write tcp 188.166.225.170:41806->81.171.14.70:4026: use of closed network connection; context deadline exceeded Nov 21 13:36:08.630048300 (293h32m7.181518751s) [seg_pos={0 0}, upload=258480]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil> Nov 21 13:36:08.800451489 (293h32m7.35192194s) [upload=258480]: Done uploading segments... Nov 21 13:36:08.800522093 (293h32m7.351992554s) [upload=258480]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24) Duration: 5.752083251s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [....X....X..X.X..X.X..XX.X.....X...............X.X.X......X.] duration: 3.48066255s trace: https://jaeger.dev.storj.io/trace/38e3ebf1da07c4a1 FAIL (2025-11-21 13:46:21 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.480691615s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [.....X....X..X.X..X.X..XX.X.....X...............X.X.X......X] duration: 5.001608261s trace: https://jaeger.dev.storj.io/trace/38e3ebf1da07c4a1 FAIL (2025-11-21 13:46:04 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001634s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 5.113816043s trace: https://jaeger.dev.storj.io/trace/2c4b010d23e46d91 FAIL (2025-11-21 13:45:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 24) ======LOGS====== AMRUQ7TS2R6K7ILJ2YSG7UNGO4P5PFTOMHVF6FXOOIWW46Y2NA, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.291418442 (293h31m7.842888914s) [noise=true , address=89.149.201.147:4006 , piece_id=T2KROQWQR5LXKBJEJO6LICLBP5CUBETT3TOACRQCGYXKZ364TYMQ, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.291954664 (293h31m7.843425145s) [noise=true , address=185.167.204.206:4048 , piece_id=6DBNGIBFWN5PPV72YWVRSSQVYF2AZ7EHS55PJDLDLGZEGXRYYHBA, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.293302360 (293h31m7.844772851s) [noise=true , address=89.149.201.147:4021 , piece_id=K3ZPQXTSSUGIGIX33FCYOC6FRCTWDBRVRHUHHGEJLNWFCJFCIT4Q, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.294979363 (293h31m7.846449854s) [noise=true , address=81.171.14.70:4028 , piece_id=ESUHXDH256EM67DXYQ7O5THLHNAPPJ6I3ARZC5D3GSLJD7CNJV5A, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.295760250 (293h31m7.847230711s) [noise=true , address=81.171.14.70:4048 , piece_id=NUU26IUS2OGXAEVLUDEYTXJL4E6GZM3M7MP5BNBI6KTRWQ2RZT4Q, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.300934223 (293h31m7.852404674s) [noise=true , address=81.171.14.70:4031 , piece_id=LJW25GLWHJCV5CFUITJ5CPYDNWOFG3NBK4A5NUX2VVV4CYCDHWMQ, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.301942341 (293h31m7.853412814s) [noise=true , address=185.167.204.205:4011 , piece_id=O2KXUNUHCKAQZSL5BJRRRC6IIZTSXHM74OZ7REQI3RGSMFA3PN6Q, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.302417767 (293h31m7.853888218s) [noise=true , address=185.167.204.205:4020 , piece_id=LPUWKZY6RPB7TGO22BL4YRA7ETB2RHWUFOXPJX7TZE3OHSFXEH7A, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.304119598 (293h31m7.855590059s) [noise=true , address=89.149.201.147:4014 , piece_id=MM2F7NMVCD3AFX2UM7FRGSO5ZD2OTJPXJEUVKJ63IL5L74LEXZSQ, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:10.220344333 (293h31m8.771814794s) [upload=258480]: Next returned nil segment Nov 21 13:35:10.220363840 (293h31m8.771834301s) [upload=258480]: Waiting for error group managing segments... Nov 21 13:35:10.536640979 (293h31m9.08811144s) [noise=true , address=185.167.204.205:4027 , piece_id=7C2DDLUHJQCI62D2YTI76NDABUZRSPK3FIZ7N45OG5GX3SZWCHNA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.537115091 (293h31m9.088585552s) [noise=true , address=185.167.204.205:4020 , piece_id=LPUWKZY6RPB7TGO22BL4YRA7ETB2RHWUFOXPJX7TZE3OHSFXEH7A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.539139225 (293h31m9.090609686s) [noise=true , address=185.167.204.206:4056 , piece_id=7VD7D3HW5YIZUYRZ5LWEB4VDXN2R6OEU7CTCGLKZUVKEC44V743Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.539668604 (293h31m9.091139065s) [noise=true , address=185.167.204.206:4048 , piece_id=6DBNGIBFWN5PPV72YWVRSSQVYF2AZ7EHS55PJDLDLGZEGXRYYHBA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.541238333 (293h31m9.092708794s) [noise=true , address=185.167.204.205:4008 , piece_id=UE3HOAD4KZRUWKY63RDOZ5LQ7VYAQPNV653CNSWCRXD2WL5MR27A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.542375438 (293h31m9.093845919s) [noise=true , address=185.167.204.205:4036 , piece_id=F652JXWWTK5XUJBY5SSU6IKKJE72DGITQXBRYK3LMQEF63DMRIEQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.543153891 (293h31m9.094624342s) [noise=true , address=185.167.204.205:4002 , piece_id=YFXJFZTNYWU3AZXQYDU5VBWJCPRVBCYCANXPQ77UGR5ZBY5VV53Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.543618846 (293h31m9.095089307s) [noise=true , address=185.167.204.206:4047 , piece_id=OHARKEAXGZLVG5XHD6RYKKWS57JFGH4JR2TAFIQSPUXTRMJD4WZA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.543662328 (293h31m9.095132789s) [noise=true , address=185.167.204.206:4066 , piece_id=T5X6MZEPB6P3DZ2HVMXCC4XJX5JJFXKGASJRWJDCK5TBHDEA2OLQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.544259055 (293h31m9.095729516s) [noise=true , address=185.167.204.205:4011 , piece_id=O2KXUNUHCKAQZSL5BJRRRC6IIZTSXHM74OZ7REQI3RGSMFA3PN6Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.544805565 (293h31m9.096276016s) [noise=true , address=185.167.204.206:4060 , piece_id=VJPM3YAQI5N2VTEBJ5GZ75JCCBOQS5IBEFUHEWTOX3BAKSP7TWFQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.545043088 (293h31m9.096513559s) [noise=true , address=185.167.204.205:4007 , piece_id=R6HJTNG5VAAMG2Y6YXK4K4YBQFUSAJQ4774UCOSGJSRDM4RWLICA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.545385921 (293h31m9.096856382s) [noise=true , address=185.167.204.205:4034 , piece_id=H66BDBLG4T2BABKBNKMJLY6QPZT2VVSZC3GW6F5V2L2BBFOWE76Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.759454013 (293h31m10.310924484s) [noise=true , address=89.149.201.147:4006 , piece_id=T2KROQWQR5LXKBJEJO6LICLBP5CUBETT3TOACRQCGYXKZ364TYMQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.779325021 (293h31m10.330795482s) [noise=true , address=89.149.201.147:4015 , piece_id=VIFNKNEVMCI2CFBREZOLVGFJ2X7Y7ZX6FWDC6PO6CLVX2LFW4FYQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.797229894 (293h31m10.348700345s) [noise=true , address=89.149.201.147:4021 , piece_id=K3ZPQXTSSUGIGIX33FCYOC6FRCTWDBRVRHUHHGEJLNWFCJFCIT4Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.804823945 (293h31m10.356294406s) [noise=true , address=89.149.201.147:4012 , piece_id=3QAKWEG37QPAE4AORUSYURK47YIOTM5VHC563QWC43UGGPS7UV7Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.828388066 (293h31m10.379858537s) [noise=true , address=89.149.201.147:4014 , piece_id=MM2F7NMVCD3AFX2UM7FRGSO5ZD2OTJPXJEUVKJ63IL5L74LEXZSQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.848656089 (293h31m10.40012655s) [noise=true , address=89.149.201.147:4001 , piece_id=OGRBX7OEEUPFXWU3EOYD44FGBQ7FQ5CHHZ2RVRHFJGOQ74EK7K5A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:02.180750503 (293h32m0.732220964s) [noise=true , address=81.171.14.70:4028 , piece_id=ESUHXDH256EM67DXYQ7O5THLHNAPPJ6I3ARZC5D3GSLJD7CNJV5A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:02.321725695 (293h32m0.873196156s) [noise=true , address=81.171.14.70:4044 , piece_id=KMR7RCY3AFLFIAJWIQXYC6ZSX7QLL6OKSCQMYHDKKCRX4MZAC4TQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:03.575396105 (293h32m2.126866556s) [noise=true , address=81.171.14.70:4041 , piece_id=N36O2DAKHTDWU4BPERULQHIUWQQDM5TQUQD7Y36XDA7SPPQML2LQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:05.676531843 (293h32m4.228002294s) [noise=true , address=81.171.14.70:4047 , piece_id=Q4N3CMZGTJGYFS35Y75GJHLQLEYMRIX4SSGTJXEHCCXIAX5Y2T3Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:08.329832254 (293h32m6.881302715s) [noise=true , address=81.171.14.70:4043 , piece_id=TQCRSBAQ4AJPCVP2JZYD4HQF375R2K7P5AIMCZW5EJVFJYUGKYGA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:08.628985817 (293h32m7.180456278s) [noise=true , address=81.171.14.70:4042 , piece_id=E5AMRUQ7TS2R6K7ILJ2YSG7UNGO4P5PFTOMHVF6FXOOIWW46Y2NA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12vBozgWFrZfUEQAwsqBt6CDMW47rVjZaDfvKTsCdPrvX8swkhD, address:81.171.14.70:4042): protocol: read tcp 188.166.225.170:40010->81.171.14.70:4042: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 21 13:36:08.629102389 (293h32m7.18057285s) [noise=true , address=81.171.14.70:4048 , piece_id=NUU26IUS2OGXAEVLUDEYTXJL4E6GZM3M7MP5BNBI6KTRWQ2RZT4Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12bbN67ZebXjF4iSyaWu4TDsG1XugTm1YdNHqRdRCbZ2WWRunRG, address:81.171.14.70:4048): protocol: write tcp 188.166.225.170:55932->81.171.14.70:4048: use of closed network connection; write tcp 188.166.225.170:55932->81.171.14.70:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55932->81.171.14.70:4048: use of closed network connection Nov 21 13:36:08.629502702 (293h32m7.180973163s) [noise=true , address=81.171.14.70:4031 , piece_id=LJW25GLWHJCV5CFUITJ5CPYDNWOFG3NBK4A5NUX2VVV4CYCDHWMQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12drdYnnWsuMyFo6LkcewAGGWnorRCAhxPpNqK41LvGy2zkBvng, address:81.171.14.70:4031): protocol: write tcp 188.166.225.170:58232->81.171.14.70:4031: use of closed network connection; write tcp 188.166.225.170:58232->81.171.14.70:4031: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58232->81.171.14.70:4031: use of closed network connection Nov 21 13:36:08.629974720 (293h32m7.181445181s) [noise=true , address=81.171.14.70:4026 , piece_id=ATEKV2WY2WDNBLY7W4THWSO52AJ76Q3XVT7QDGT3EBMPWVRL5EKA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12RTn3BuxasVSPYmjqgyhqKn1t6fAtq9EBtUufXqcLq48My9g2b, address:81.171.14.70:4026): protocol: write tcp 188.166.225.170:41806->81.171.14.70:4026: use of closed network connection; context deadline exceeded Nov 21 13:36:08.630048300 (293h32m7.181518751s) [seg_pos={0 0}, upload=258480]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil> Nov 21 13:36:08.800451489 (293h32m7.35192194s) [upload=258480]: Done uploading segments... Nov 21 13:36:08.800522093 (293h32m7.351992554s) [upload=258480]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24) Duration: 5.113925903s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 5.301026247s trace: https://jaeger.dev.storj.io/trace/2c4b010d23e46d91 FAIL (2025-11-21 13:44:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 24) ======LOGS====== AMRUQ7TS2R6K7ILJ2YSG7UNGO4P5PFTOMHVF6FXOOIWW46Y2NA, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.291418442 (293h31m7.842888914s) [noise=true , address=89.149.201.147:4006 , piece_id=T2KROQWQR5LXKBJEJO6LICLBP5CUBETT3TOACRQCGYXKZ364TYMQ, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.291954664 (293h31m7.843425145s) [noise=true , address=185.167.204.206:4048 , piece_id=6DBNGIBFWN5PPV72YWVRSSQVYF2AZ7EHS55PJDLDLGZEGXRYYHBA, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.293302360 (293h31m7.844772851s) [noise=true , address=89.149.201.147:4021 , piece_id=K3ZPQXTSSUGIGIX33FCYOC6FRCTWDBRVRHUHHGEJLNWFCJFCIT4Q, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.294979363 (293h31m7.846449854s) [noise=true , address=81.171.14.70:4028 , piece_id=ESUHXDH256EM67DXYQ7O5THLHNAPPJ6I3ARZC5D3GSLJD7CNJV5A, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.295760250 (293h31m7.847230711s) [noise=true , address=81.171.14.70:4048 , piece_id=NUU26IUS2OGXAEVLUDEYTXJL4E6GZM3M7MP5BNBI6KTRWQ2RZT4Q, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.300934223 (293h31m7.852404674s) [noise=true , address=81.171.14.70:4031 , piece_id=LJW25GLWHJCV5CFUITJ5CPYDNWOFG3NBK4A5NUX2VVV4CYCDHWMQ, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.301942341 (293h31m7.853412814s) [noise=true , address=185.167.204.205:4011 , piece_id=O2KXUNUHCKAQZSL5BJRRRC6IIZTSXHM74OZ7REQI3RGSMFA3PN6Q, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.302417767 (293h31m7.853888218s) [noise=true , address=185.167.204.205:4020 , piece_id=LPUWKZY6RPB7TGO22BL4YRA7ETB2RHWUFOXPJX7TZE3OHSFXEH7A, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.304119598 (293h31m7.855590059s) [noise=true , address=89.149.201.147:4014 , piece_id=MM2F7NMVCD3AFX2UM7FRGSO5ZD2OTJPXJEUVKJ63IL5L74LEXZSQ, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:10.220344333 (293h31m8.771814794s) [upload=258480]: Next returned nil segment Nov 21 13:35:10.220363840 (293h31m8.771834301s) [upload=258480]: Waiting for error group managing segments... Nov 21 13:35:10.536640979 (293h31m9.08811144s) [noise=true , address=185.167.204.205:4027 , piece_id=7C2DDLUHJQCI62D2YTI76NDABUZRSPK3FIZ7N45OG5GX3SZWCHNA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.537115091 (293h31m9.088585552s) [noise=true , address=185.167.204.205:4020 , piece_id=LPUWKZY6RPB7TGO22BL4YRA7ETB2RHWUFOXPJX7TZE3OHSFXEH7A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.539139225 (293h31m9.090609686s) [noise=true , address=185.167.204.206:4056 , piece_id=7VD7D3HW5YIZUYRZ5LWEB4VDXN2R6OEU7CTCGLKZUVKEC44V743Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.539668604 (293h31m9.091139065s) [noise=true , address=185.167.204.206:4048 , piece_id=6DBNGIBFWN5PPV72YWVRSSQVYF2AZ7EHS55PJDLDLGZEGXRYYHBA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.541238333 (293h31m9.092708794s) [noise=true , address=185.167.204.205:4008 , piece_id=UE3HOAD4KZRUWKY63RDOZ5LQ7VYAQPNV653CNSWCRXD2WL5MR27A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.542375438 (293h31m9.093845919s) [noise=true , address=185.167.204.205:4036 , piece_id=F652JXWWTK5XUJBY5SSU6IKKJE72DGITQXBRYK3LMQEF63DMRIEQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.543153891 (293h31m9.094624342s) [noise=true , address=185.167.204.205:4002 , piece_id=YFXJFZTNYWU3AZXQYDU5VBWJCPRVBCYCANXPQ77UGR5ZBY5VV53Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.543618846 (293h31m9.095089307s) [noise=true , address=185.167.204.206:4047 , piece_id=OHARKEAXGZLVG5XHD6RYKKWS57JFGH4JR2TAFIQSPUXTRMJD4WZA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.543662328 (293h31m9.095132789s) [noise=true , address=185.167.204.206:4066 , piece_id=T5X6MZEPB6P3DZ2HVMXCC4XJX5JJFXKGASJRWJDCK5TBHDEA2OLQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.544259055 (293h31m9.095729516s) [noise=true , address=185.167.204.205:4011 , piece_id=O2KXUNUHCKAQZSL5BJRRRC6IIZTSXHM74OZ7REQI3RGSMFA3PN6Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.544805565 (293h31m9.096276016s) [noise=true , address=185.167.204.206:4060 , piece_id=VJPM3YAQI5N2VTEBJ5GZ75JCCBOQS5IBEFUHEWTOX3BAKSP7TWFQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.545043088 (293h31m9.096513559s) [noise=true , address=185.167.204.205:4007 , piece_id=R6HJTNG5VAAMG2Y6YXK4K4YBQFUSAJQ4774UCOSGJSRDM4RWLICA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.545385921 (293h31m9.096856382s) [noise=true , address=185.167.204.205:4034 , piece_id=H66BDBLG4T2BABKBNKMJLY6QPZT2VVSZC3GW6F5V2L2BBFOWE76Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.759454013 (293h31m10.310924484s) [noise=true , address=89.149.201.147:4006 , piece_id=T2KROQWQR5LXKBJEJO6LICLBP5CUBETT3TOACRQCGYXKZ364TYMQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.779325021 (293h31m10.330795482s) [noise=true , address=89.149.201.147:4015 , piece_id=VIFNKNEVMCI2CFBREZOLVGFJ2X7Y7ZX6FWDC6PO6CLVX2LFW4FYQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.797229894 (293h31m10.348700345s) [noise=true , address=89.149.201.147:4021 , piece_id=K3ZPQXTSSUGIGIX33FCYOC6FRCTWDBRVRHUHHGEJLNWFCJFCIT4Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.804823945 (293h31m10.356294406s) [noise=true , address=89.149.201.147:4012 , piece_id=3QAKWEG37QPAE4AORUSYURK47YIOTM5VHC563QWC43UGGPS7UV7Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.828388066 (293h31m10.379858537s) [noise=true , address=89.149.201.147:4014 , piece_id=MM2F7NMVCD3AFX2UM7FRGSO5ZD2OTJPXJEUVKJ63IL5L74LEXZSQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.848656089 (293h31m10.40012655s) [noise=true , address=89.149.201.147:4001 , piece_id=OGRBX7OEEUPFXWU3EOYD44FGBQ7FQ5CHHZ2RVRHFJGOQ74EK7K5A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:02.180750503 (293h32m0.732220964s) [noise=true , address=81.171.14.70:4028 , piece_id=ESUHXDH256EM67DXYQ7O5THLHNAPPJ6I3ARZC5D3GSLJD7CNJV5A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:02.321725695 (293h32m0.873196156s) [noise=true , address=81.171.14.70:4044 , piece_id=KMR7RCY3AFLFIAJWIQXYC6ZSX7QLL6OKSCQMYHDKKCRX4MZAC4TQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:03.575396105 (293h32m2.126866556s) [noise=true , address=81.171.14.70:4041 , piece_id=N36O2DAKHTDWU4BPERULQHIUWQQDM5TQUQD7Y36XDA7SPPQML2LQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:05.676531843 (293h32m4.228002294s) [noise=true , address=81.171.14.70:4047 , piece_id=Q4N3CMZGTJGYFS35Y75GJHLQLEYMRIX4SSGTJXEHCCXIAX5Y2T3Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:08.329832254 (293h32m6.881302715s) [noise=true , address=81.171.14.70:4043 , piece_id=TQCRSBAQ4AJPCVP2JZYD4HQF375R2K7P5AIMCZW5EJVFJYUGKYGA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:08.628985817 (293h32m7.180456278s) [noise=true , address=81.171.14.70:4042 , piece_id=E5AMRUQ7TS2R6K7ILJ2YSG7UNGO4P5PFTOMHVF6FXOOIWW46Y2NA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12vBozgWFrZfUEQAwsqBt6CDMW47rVjZaDfvKTsCdPrvX8swkhD, address:81.171.14.70:4042): protocol: read tcp 188.166.225.170:40010->81.171.14.70:4042: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 21 13:36:08.629102389 (293h32m7.18057285s) [noise=true , address=81.171.14.70:4048 , piece_id=NUU26IUS2OGXAEVLUDEYTXJL4E6GZM3M7MP5BNBI6KTRWQ2RZT4Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12bbN67ZebXjF4iSyaWu4TDsG1XugTm1YdNHqRdRCbZ2WWRunRG, address:81.171.14.70:4048): protocol: write tcp 188.166.225.170:55932->81.171.14.70:4048: use of closed network connection; write tcp 188.166.225.170:55932->81.171.14.70:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55932->81.171.14.70:4048: use of closed network connection Nov 21 13:36:08.629502702 (293h32m7.180973163s) [noise=true , address=81.171.14.70:4031 , piece_id=LJW25GLWHJCV5CFUITJ5CPYDNWOFG3NBK4A5NUX2VVV4CYCDHWMQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12drdYnnWsuMyFo6LkcewAGGWnorRCAhxPpNqK41LvGy2zkBvng, address:81.171.14.70:4031): protocol: write tcp 188.166.225.170:58232->81.171.14.70:4031: use of closed network connection; write tcp 188.166.225.170:58232->81.171.14.70:4031: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58232->81.171.14.70:4031: use of closed network connection Nov 21 13:36:08.629974720 (293h32m7.181445181s) [noise=true , address=81.171.14.70:4026 , piece_id=ATEKV2WY2WDNBLY7W4THWSO52AJ76Q3XVT7QDGT3EBMPWVRL5EKA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12RTn3BuxasVSPYmjqgyhqKn1t6fAtq9EBtUufXqcLq48My9g2b, address:81.171.14.70:4026): protocol: write tcp 188.166.225.170:41806->81.171.14.70:4026: use of closed network connection; context deadline exceeded Nov 21 13:36:08.630048300 (293h32m7.181518751s) [seg_pos={0 0}, upload=258480]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil> Nov 21 13:36:08.800451489 (293h32m7.35192194s) [upload=258480]: Done uploading segments... Nov 21 13:36:08.800522093 (293h32m7.351992554s) [upload=258480]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24) Duration: 5.301045443s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [......X....X..X.X..X.X..XX.X.....X...............X.X.X......] duration: 4.613230689s trace: https://jaeger.dev.storj.io/trace/37d1ae6a9d531aca FAIL (2025-11-21 13:44:49 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.613256427s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [.......X....X..X.X..X.X..XX.X.....X...............X.X.X.....] duration: 5.001100563s trace: https://jaeger.dev.storj.io/trace/37d1ae6a9d531aca FAIL (2025-11-21 13:44:29 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001127724s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [........X....X..X.X..X.X..XX.X.....X...............X.X.X....] duration: 4.970863477s trace: https://jaeger.dev.storj.io/trace/37d1ae6a9d531aca FAIL (2025-11-21 13:44:01 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.971168979s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 5.40586545s trace: https://jaeger.dev.storj.io/trace/2c4b010d23e46d91 FAIL (2025-11-21 13:43:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 24) ======LOGS====== AMRUQ7TS2R6K7ILJ2YSG7UNGO4P5PFTOMHVF6FXOOIWW46Y2NA, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.291418442 (293h31m7.842888914s) [noise=true , address=89.149.201.147:4006 , piece_id=T2KROQWQR5LXKBJEJO6LICLBP5CUBETT3TOACRQCGYXKZ364TYMQ, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.291954664 (293h31m7.843425145s) [noise=true , address=185.167.204.206:4048 , piece_id=6DBNGIBFWN5PPV72YWVRSSQVYF2AZ7EHS55PJDLDLGZEGXRYYHBA, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.293302360 (293h31m7.844772851s) [noise=true , address=89.149.201.147:4021 , piece_id=K3ZPQXTSSUGIGIX33FCYOC6FRCTWDBRVRHUHHGEJLNWFCJFCIT4Q, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.294979363 (293h31m7.846449854s) [noise=true , address=81.171.14.70:4028 , piece_id=ESUHXDH256EM67DXYQ7O5THLHNAPPJ6I3ARZC5D3GSLJD7CNJV5A, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.295760250 (293h31m7.847230711s) [noise=true , address=81.171.14.70:4048 , piece_id=NUU26IUS2OGXAEVLUDEYTXJL4E6GZM3M7MP5BNBI6KTRWQ2RZT4Q, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.300934223 (293h31m7.852404674s) [noise=true , address=81.171.14.70:4031 , piece_id=LJW25GLWHJCV5CFUITJ5CPYDNWOFG3NBK4A5NUX2VVV4CYCDHWMQ, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.301942341 (293h31m7.853412814s) [noise=true , address=185.167.204.205:4011 , piece_id=O2KXUNUHCKAQZSL5BJRRRC6IIZTSXHM74OZ7REQI3RGSMFA3PN6Q, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.302417767 (293h31m7.853888218s) [noise=true , address=185.167.204.205:4020 , piece_id=LPUWKZY6RPB7TGO22BL4YRA7ETB2RHWUFOXPJX7TZE3OHSFXEH7A, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.304119598 (293h31m7.855590059s) [noise=true , address=89.149.201.147:4014 , piece_id=MM2F7NMVCD3AFX2UM7FRGSO5ZD2OTJPXJEUVKJ63IL5L74LEXZSQ, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:10.220344333 (293h31m8.771814794s) [upload=258480]: Next returned nil segment Nov 21 13:35:10.220363840 (293h31m8.771834301s) [upload=258480]: Waiting for error group managing segments... Nov 21 13:35:10.536640979 (293h31m9.08811144s) [noise=true , address=185.167.204.205:4027 , piece_id=7C2DDLUHJQCI62D2YTI76NDABUZRSPK3FIZ7N45OG5GX3SZWCHNA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.537115091 (293h31m9.088585552s) [noise=true , address=185.167.204.205:4020 , piece_id=LPUWKZY6RPB7TGO22BL4YRA7ETB2RHWUFOXPJX7TZE3OHSFXEH7A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.539139225 (293h31m9.090609686s) [noise=true , address=185.167.204.206:4056 , piece_id=7VD7D3HW5YIZUYRZ5LWEB4VDXN2R6OEU7CTCGLKZUVKEC44V743Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.539668604 (293h31m9.091139065s) [noise=true , address=185.167.204.206:4048 , piece_id=6DBNGIBFWN5PPV72YWVRSSQVYF2AZ7EHS55PJDLDLGZEGXRYYHBA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.541238333 (293h31m9.092708794s) [noise=true , address=185.167.204.205:4008 , piece_id=UE3HOAD4KZRUWKY63RDOZ5LQ7VYAQPNV653CNSWCRXD2WL5MR27A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.542375438 (293h31m9.093845919s) [noise=true , address=185.167.204.205:4036 , piece_id=F652JXWWTK5XUJBY5SSU6IKKJE72DGITQXBRYK3LMQEF63DMRIEQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.543153891 (293h31m9.094624342s) [noise=true , address=185.167.204.205:4002 , piece_id=YFXJFZTNYWU3AZXQYDU5VBWJCPRVBCYCANXPQ77UGR5ZBY5VV53Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.543618846 (293h31m9.095089307s) [noise=true , address=185.167.204.206:4047 , piece_id=OHARKEAXGZLVG5XHD6RYKKWS57JFGH4JR2TAFIQSPUXTRMJD4WZA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.543662328 (293h31m9.095132789s) [noise=true , address=185.167.204.206:4066 , piece_id=T5X6MZEPB6P3DZ2HVMXCC4XJX5JJFXKGASJRWJDCK5TBHDEA2OLQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.544259055 (293h31m9.095729516s) [noise=true , address=185.167.204.205:4011 , piece_id=O2KXUNUHCKAQZSL5BJRRRC6IIZTSXHM74OZ7REQI3RGSMFA3PN6Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.544805565 (293h31m9.096276016s) [noise=true , address=185.167.204.206:4060 , piece_id=VJPM3YAQI5N2VTEBJ5GZ75JCCBOQS5IBEFUHEWTOX3BAKSP7TWFQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.545043088 (293h31m9.096513559s) [noise=true , address=185.167.204.205:4007 , piece_id=R6HJTNG5VAAMG2Y6YXK4K4YBQFUSAJQ4774UCOSGJSRDM4RWLICA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.545385921 (293h31m9.096856382s) [noise=true , address=185.167.204.205:4034 , piece_id=H66BDBLG4T2BABKBNKMJLY6QPZT2VVSZC3GW6F5V2L2BBFOWE76Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.759454013 (293h31m10.310924484s) [noise=true , address=89.149.201.147:4006 , piece_id=T2KROQWQR5LXKBJEJO6LICLBP5CUBETT3TOACRQCGYXKZ364TYMQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.779325021 (293h31m10.330795482s) [noise=true , address=89.149.201.147:4015 , piece_id=VIFNKNEVMCI2CFBREZOLVGFJ2X7Y7ZX6FWDC6PO6CLVX2LFW4FYQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.797229894 (293h31m10.348700345s) [noise=true , address=89.149.201.147:4021 , piece_id=K3ZPQXTSSUGIGIX33FCYOC6FRCTWDBRVRHUHHGEJLNWFCJFCIT4Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.804823945 (293h31m10.356294406s) [noise=true , address=89.149.201.147:4012 , piece_id=3QAKWEG37QPAE4AORUSYURK47YIOTM5VHC563QWC43UGGPS7UV7Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.828388066 (293h31m10.379858537s) [noise=true , address=89.149.201.147:4014 , piece_id=MM2F7NMVCD3AFX2UM7FRGSO5ZD2OTJPXJEUVKJ63IL5L74LEXZSQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.848656089 (293h31m10.40012655s) [noise=true , address=89.149.201.147:4001 , piece_id=OGRBX7OEEUPFXWU3EOYD44FGBQ7FQ5CHHZ2RVRHFJGOQ74EK7K5A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:02.180750503 (293h32m0.732220964s) [noise=true , address=81.171.14.70:4028 , piece_id=ESUHXDH256EM67DXYQ7O5THLHNAPPJ6I3ARZC5D3GSLJD7CNJV5A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:02.321725695 (293h32m0.873196156s) [noise=true , address=81.171.14.70:4044 , piece_id=KMR7RCY3AFLFIAJWIQXYC6ZSX7QLL6OKSCQMYHDKKCRX4MZAC4TQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:03.575396105 (293h32m2.126866556s) [noise=true , address=81.171.14.70:4041 , piece_id=N36O2DAKHTDWU4BPERULQHIUWQQDM5TQUQD7Y36XDA7SPPQML2LQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:05.676531843 (293h32m4.228002294s) [noise=true , address=81.171.14.70:4047 , piece_id=Q4N3CMZGTJGYFS35Y75GJHLQLEYMRIX4SSGTJXEHCCXIAX5Y2T3Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:08.329832254 (293h32m6.881302715s) [noise=true , address=81.171.14.70:4043 , piece_id=TQCRSBAQ4AJPCVP2JZYD4HQF375R2K7P5AIMCZW5EJVFJYUGKYGA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:08.628985817 (293h32m7.180456278s) [noise=true , address=81.171.14.70:4042 , piece_id=E5AMRUQ7TS2R6K7ILJ2YSG7UNGO4P5PFTOMHVF6FXOOIWW46Y2NA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12vBozgWFrZfUEQAwsqBt6CDMW47rVjZaDfvKTsCdPrvX8swkhD, address:81.171.14.70:4042): protocol: read tcp 188.166.225.170:40010->81.171.14.70:4042: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 21 13:36:08.629102389 (293h32m7.18057285s) [noise=true , address=81.171.14.70:4048 , piece_id=NUU26IUS2OGXAEVLUDEYTXJL4E6GZM3M7MP5BNBI6KTRWQ2RZT4Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12bbN67ZebXjF4iSyaWu4TDsG1XugTm1YdNHqRdRCbZ2WWRunRG, address:81.171.14.70:4048): protocol: write tcp 188.166.225.170:55932->81.171.14.70:4048: use of closed network connection; write tcp 188.166.225.170:55932->81.171.14.70:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55932->81.171.14.70:4048: use of closed network connection Nov 21 13:36:08.629502702 (293h32m7.180973163s) [noise=true , address=81.171.14.70:4031 , piece_id=LJW25GLWHJCV5CFUITJ5CPYDNWOFG3NBK4A5NUX2VVV4CYCDHWMQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12drdYnnWsuMyFo6LkcewAGGWnorRCAhxPpNqK41LvGy2zkBvng, address:81.171.14.70:4031): protocol: write tcp 188.166.225.170:58232->81.171.14.70:4031: use of closed network connection; write tcp 188.166.225.170:58232->81.171.14.70:4031: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58232->81.171.14.70:4031: use of closed network connection Nov 21 13:36:08.629974720 (293h32m7.181445181s) [noise=true , address=81.171.14.70:4026 , piece_id=ATEKV2WY2WDNBLY7W4THWSO52AJ76Q3XVT7QDGT3EBMPWVRL5EKA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12RTn3BuxasVSPYmjqgyhqKn1t6fAtq9EBtUufXqcLq48My9g2b, address:81.171.14.70:4026): protocol: write tcp 188.166.225.170:41806->81.171.14.70:4026: use of closed network connection; context deadline exceeded Nov 21 13:36:08.630048300 (293h32m7.181518751s) [seg_pos={0 0}, upload=258480]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil> Nov 21 13:36:08.800451489 (293h32m7.35192194s) [upload=258480]: Done uploading segments... Nov 21 13:36:08.800522093 (293h32m7.351992554s) [upload=258480]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24) Duration: 5.405893263s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [.........X....X..X.X..X.X..XX.X.....X...............X.X.X...] duration: 4.891243992s trace: https://jaeger.dev.storj.io/trace/37d1ae6a9d531aca FAIL (2025-11-21 13:43:32 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.891276475s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 4.918640614s trace: https://jaeger.dev.storj.io/trace/2c4b010d23e46d91 FAIL (2025-11-21 13:42:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 24) ======LOGS====== AMRUQ7TS2R6K7ILJ2YSG7UNGO4P5PFTOMHVF6FXOOIWW46Y2NA, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.291418442 (293h31m7.842888914s) [noise=true , address=89.149.201.147:4006 , piece_id=T2KROQWQR5LXKBJEJO6LICLBP5CUBETT3TOACRQCGYXKZ364TYMQ, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.291954664 (293h31m7.843425145s) [noise=true , address=185.167.204.206:4048 , piece_id=6DBNGIBFWN5PPV72YWVRSSQVYF2AZ7EHS55PJDLDLGZEGXRYYHBA, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.293302360 (293h31m7.844772851s) [noise=true , address=89.149.201.147:4021 , piece_id=K3ZPQXTSSUGIGIX33FCYOC6FRCTWDBRVRHUHHGEJLNWFCJFCIT4Q, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.294979363 (293h31m7.846449854s) [noise=true , address=81.171.14.70:4028 , piece_id=ESUHXDH256EM67DXYQ7O5THLHNAPPJ6I3ARZC5D3GSLJD7CNJV5A, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.295760250 (293h31m7.847230711s) [noise=true , address=81.171.14.70:4048 , piece_id=NUU26IUS2OGXAEVLUDEYTXJL4E6GZM3M7MP5BNBI6KTRWQ2RZT4Q, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.300934223 (293h31m7.852404674s) [noise=true , address=81.171.14.70:4031 , piece_id=LJW25GLWHJCV5CFUITJ5CPYDNWOFG3NBK4A5NUX2VVV4CYCDHWMQ, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.301942341 (293h31m7.853412814s) [noise=true , address=185.167.204.205:4011 , piece_id=O2KXUNUHCKAQZSL5BJRRRC6IIZTSXHM74OZ7REQI3RGSMFA3PN6Q, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.302417767 (293h31m7.853888218s) [noise=true , address=185.167.204.205:4020 , piece_id=LPUWKZY6RPB7TGO22BL4YRA7ETB2RHWUFOXPJX7TZE3OHSFXEH7A, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.304119598 (293h31m7.855590059s) [noise=true , address=89.149.201.147:4014 , piece_id=MM2F7NMVCD3AFX2UM7FRGSO5ZD2OTJPXJEUVKJ63IL5L74LEXZSQ, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:10.220344333 (293h31m8.771814794s) [upload=258480]: Next returned nil segment Nov 21 13:35:10.220363840 (293h31m8.771834301s) [upload=258480]: Waiting for error group managing segments... Nov 21 13:35:10.536640979 (293h31m9.08811144s) [noise=true , address=185.167.204.205:4027 , piece_id=7C2DDLUHJQCI62D2YTI76NDABUZRSPK3FIZ7N45OG5GX3SZWCHNA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.537115091 (293h31m9.088585552s) [noise=true , address=185.167.204.205:4020 , piece_id=LPUWKZY6RPB7TGO22BL4YRA7ETB2RHWUFOXPJX7TZE3OHSFXEH7A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.539139225 (293h31m9.090609686s) [noise=true , address=185.167.204.206:4056 , piece_id=7VD7D3HW5YIZUYRZ5LWEB4VDXN2R6OEU7CTCGLKZUVKEC44V743Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.539668604 (293h31m9.091139065s) [noise=true , address=185.167.204.206:4048 , piece_id=6DBNGIBFWN5PPV72YWVRSSQVYF2AZ7EHS55PJDLDLGZEGXRYYHBA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.541238333 (293h31m9.092708794s) [noise=true , address=185.167.204.205:4008 , piece_id=UE3HOAD4KZRUWKY63RDOZ5LQ7VYAQPNV653CNSWCRXD2WL5MR27A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.542375438 (293h31m9.093845919s) [noise=true , address=185.167.204.205:4036 , piece_id=F652JXWWTK5XUJBY5SSU6IKKJE72DGITQXBRYK3LMQEF63DMRIEQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.543153891 (293h31m9.094624342s) [noise=true , address=185.167.204.205:4002 , piece_id=YFXJFZTNYWU3AZXQYDU5VBWJCPRVBCYCANXPQ77UGR5ZBY5VV53Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.543618846 (293h31m9.095089307s) [noise=true , address=185.167.204.206:4047 , piece_id=OHARKEAXGZLVG5XHD6RYKKWS57JFGH4JR2TAFIQSPUXTRMJD4WZA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.543662328 (293h31m9.095132789s) [noise=true , address=185.167.204.206:4066 , piece_id=T5X6MZEPB6P3DZ2HVMXCC4XJX5JJFXKGASJRWJDCK5TBHDEA2OLQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.544259055 (293h31m9.095729516s) [noise=true , address=185.167.204.205:4011 , piece_id=O2KXUNUHCKAQZSL5BJRRRC6IIZTSXHM74OZ7REQI3RGSMFA3PN6Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.544805565 (293h31m9.096276016s) [noise=true , address=185.167.204.206:4060 , piece_id=VJPM3YAQI5N2VTEBJ5GZ75JCCBOQS5IBEFUHEWTOX3BAKSP7TWFQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.545043088 (293h31m9.096513559s) [noise=true , address=185.167.204.205:4007 , piece_id=R6HJTNG5VAAMG2Y6YXK4K4YBQFUSAJQ4774UCOSGJSRDM4RWLICA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.545385921 (293h31m9.096856382s) [noise=true , address=185.167.204.205:4034 , piece_id=H66BDBLG4T2BABKBNKMJLY6QPZT2VVSZC3GW6F5V2L2BBFOWE76Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.759454013 (293h31m10.310924484s) [noise=true , address=89.149.201.147:4006 , piece_id=T2KROQWQR5LXKBJEJO6LICLBP5CUBETT3TOACRQCGYXKZ364TYMQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.779325021 (293h31m10.330795482s) [noise=true , address=89.149.201.147:4015 , piece_id=VIFNKNEVMCI2CFBREZOLVGFJ2X7Y7ZX6FWDC6PO6CLVX2LFW4FYQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.797229894 (293h31m10.348700345s) [noise=true , address=89.149.201.147:4021 , piece_id=K3ZPQXTSSUGIGIX33FCYOC6FRCTWDBRVRHUHHGEJLNWFCJFCIT4Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.804823945 (293h31m10.356294406s) [noise=true , address=89.149.201.147:4012 , piece_id=3QAKWEG37QPAE4AORUSYURK47YIOTM5VHC563QWC43UGGPS7UV7Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.828388066 (293h31m10.379858537s) [noise=true , address=89.149.201.147:4014 , piece_id=MM2F7NMVCD3AFX2UM7FRGSO5ZD2OTJPXJEUVKJ63IL5L74LEXZSQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.848656089 (293h31m10.40012655s) [noise=true , address=89.149.201.147:4001 , piece_id=OGRBX7OEEUPFXWU3EOYD44FGBQ7FQ5CHHZ2RVRHFJGOQ74EK7K5A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:02.180750503 (293h32m0.732220964s) [noise=true , address=81.171.14.70:4028 , piece_id=ESUHXDH256EM67DXYQ7O5THLHNAPPJ6I3ARZC5D3GSLJD7CNJV5A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:02.321725695 (293h32m0.873196156s) [noise=true , address=81.171.14.70:4044 , piece_id=KMR7RCY3AFLFIAJWIQXYC6ZSX7QLL6OKSCQMYHDKKCRX4MZAC4TQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:03.575396105 (293h32m2.126866556s) [noise=true , address=81.171.14.70:4041 , piece_id=N36O2DAKHTDWU4BPERULQHIUWQQDM5TQUQD7Y36XDA7SPPQML2LQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:05.676531843 (293h32m4.228002294s) [noise=true , address=81.171.14.70:4047 , piece_id=Q4N3CMZGTJGYFS35Y75GJHLQLEYMRIX4SSGTJXEHCCXIAX5Y2T3Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:08.329832254 (293h32m6.881302715s) [noise=true , address=81.171.14.70:4043 , piece_id=TQCRSBAQ4AJPCVP2JZYD4HQF375R2K7P5AIMCZW5EJVFJYUGKYGA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:08.628985817 (293h32m7.180456278s) [noise=true , address=81.171.14.70:4042 , piece_id=E5AMRUQ7TS2R6K7ILJ2YSG7UNGO4P5PFTOMHVF6FXOOIWW46Y2NA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12vBozgWFrZfUEQAwsqBt6CDMW47rVjZaDfvKTsCdPrvX8swkhD, address:81.171.14.70:4042): protocol: read tcp 188.166.225.170:40010->81.171.14.70:4042: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 21 13:36:08.629102389 (293h32m7.18057285s) [noise=true , address=81.171.14.70:4048 , piece_id=NUU26IUS2OGXAEVLUDEYTXJL4E6GZM3M7MP5BNBI6KTRWQ2RZT4Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12bbN67ZebXjF4iSyaWu4TDsG1XugTm1YdNHqRdRCbZ2WWRunRG, address:81.171.14.70:4048): protocol: write tcp 188.166.225.170:55932->81.171.14.70:4048: use of closed network connection; write tcp 188.166.225.170:55932->81.171.14.70:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55932->81.171.14.70:4048: use of closed network connection Nov 21 13:36:08.629502702 (293h32m7.180973163s) [noise=true , address=81.171.14.70:4031 , piece_id=LJW25GLWHJCV5CFUITJ5CPYDNWOFG3NBK4A5NUX2VVV4CYCDHWMQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12drdYnnWsuMyFo6LkcewAGGWnorRCAhxPpNqK41LvGy2zkBvng, address:81.171.14.70:4031): protocol: write tcp 188.166.225.170:58232->81.171.14.70:4031: use of closed network connection; write tcp 188.166.225.170:58232->81.171.14.70:4031: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58232->81.171.14.70:4031: use of closed network connection Nov 21 13:36:08.629974720 (293h32m7.181445181s) [noise=true , address=81.171.14.70:4026 , piece_id=ATEKV2WY2WDNBLY7W4THWSO52AJ76Q3XVT7QDGT3EBMPWVRL5EKA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12RTn3BuxasVSPYmjqgyhqKn1t6fAtq9EBtUufXqcLq48My9g2b, address:81.171.14.70:4026): protocol: write tcp 188.166.225.170:41806->81.171.14.70:4026: use of closed network connection; context deadline exceeded Nov 21 13:36:08.630048300 (293h32m7.181518751s) [seg_pos={0 0}, upload=258480]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil> Nov 21 13:36:08.800451489 (293h32m7.35192194s) [upload=258480]: Done uploading segments... Nov 21 13:36:08.800522093 (293h32m7.351992554s) [upload=258480]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24) Duration: 4.918673987s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [..........X....X..X.X..X.X..XX.X.....X...............X.X.X..] duration: 4.926493626s trace: https://jaeger.dev.storj.io/trace/37d1ae6a9d531aca FAIL (2025-11-21 13:42:32 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.926523604s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 6.747799586s trace: https://jaeger.dev.storj.io/trace/2c4b010d23e46d91 FAIL (2025-11-21 13:41:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 24) ======LOGS====== AMRUQ7TS2R6K7ILJ2YSG7UNGO4P5PFTOMHVF6FXOOIWW46Y2NA, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.291418442 (293h31m7.842888914s) [noise=true , address=89.149.201.147:4006 , piece_id=T2KROQWQR5LXKBJEJO6LICLBP5CUBETT3TOACRQCGYXKZ364TYMQ, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.291954664 (293h31m7.843425145s) [noise=true , address=185.167.204.206:4048 , piece_id=6DBNGIBFWN5PPV72YWVRSSQVYF2AZ7EHS55PJDLDLGZEGXRYYHBA, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.293302360 (293h31m7.844772851s) [noise=true , address=89.149.201.147:4021 , piece_id=K3ZPQXTSSUGIGIX33FCYOC6FRCTWDBRVRHUHHGEJLNWFCJFCIT4Q, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.294979363 (293h31m7.846449854s) [noise=true , address=81.171.14.70:4028 , piece_id=ESUHXDH256EM67DXYQ7O5THLHNAPPJ6I3ARZC5D3GSLJD7CNJV5A, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.295760250 (293h31m7.847230711s) [noise=true , address=81.171.14.70:4048 , piece_id=NUU26IUS2OGXAEVLUDEYTXJL4E6GZM3M7MP5BNBI6KTRWQ2RZT4Q, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.300934223 (293h31m7.852404674s) [noise=true , address=81.171.14.70:4031 , piece_id=LJW25GLWHJCV5CFUITJ5CPYDNWOFG3NBK4A5NUX2VVV4CYCDHWMQ, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.301942341 (293h31m7.853412814s) [noise=true , address=185.167.204.205:4011 , piece_id=O2KXUNUHCKAQZSL5BJRRRC6IIZTSXHM74OZ7REQI3RGSMFA3PN6Q, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.302417767 (293h31m7.853888218s) [noise=true , address=185.167.204.205:4020 , piece_id=LPUWKZY6RPB7TGO22BL4YRA7ETB2RHWUFOXPJX7TZE3OHSFXEH7A, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.304119598 (293h31m7.855590059s) [noise=true , address=89.149.201.147:4014 , piece_id=MM2F7NMVCD3AFX2UM7FRGSO5ZD2OTJPXJEUVKJ63IL5L74LEXZSQ, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:10.220344333 (293h31m8.771814794s) [upload=258480]: Next returned nil segment Nov 21 13:35:10.220363840 (293h31m8.771834301s) [upload=258480]: Waiting for error group managing segments... Nov 21 13:35:10.536640979 (293h31m9.08811144s) [noise=true , address=185.167.204.205:4027 , piece_id=7C2DDLUHJQCI62D2YTI76NDABUZRSPK3FIZ7N45OG5GX3SZWCHNA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.537115091 (293h31m9.088585552s) [noise=true , address=185.167.204.205:4020 , piece_id=LPUWKZY6RPB7TGO22BL4YRA7ETB2RHWUFOXPJX7TZE3OHSFXEH7A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.539139225 (293h31m9.090609686s) [noise=true , address=185.167.204.206:4056 , piece_id=7VD7D3HW5YIZUYRZ5LWEB4VDXN2R6OEU7CTCGLKZUVKEC44V743Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.539668604 (293h31m9.091139065s) [noise=true , address=185.167.204.206:4048 , piece_id=6DBNGIBFWN5PPV72YWVRSSQVYF2AZ7EHS55PJDLDLGZEGXRYYHBA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.541238333 (293h31m9.092708794s) [noise=true , address=185.167.204.205:4008 , piece_id=UE3HOAD4KZRUWKY63RDOZ5LQ7VYAQPNV653CNSWCRXD2WL5MR27A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.542375438 (293h31m9.093845919s) [noise=true , address=185.167.204.205:4036 , piece_id=F652JXWWTK5XUJBY5SSU6IKKJE72DGITQXBRYK3LMQEF63DMRIEQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.543153891 (293h31m9.094624342s) [noise=true , address=185.167.204.205:4002 , piece_id=YFXJFZTNYWU3AZXQYDU5VBWJCPRVBCYCANXPQ77UGR5ZBY5VV53Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.543618846 (293h31m9.095089307s) [noise=true , address=185.167.204.206:4047 , piece_id=OHARKEAXGZLVG5XHD6RYKKWS57JFGH4JR2TAFIQSPUXTRMJD4WZA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.543662328 (293h31m9.095132789s) [noise=true , address=185.167.204.206:4066 , piece_id=T5X6MZEPB6P3DZ2HVMXCC4XJX5JJFXKGASJRWJDCK5TBHDEA2OLQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.544259055 (293h31m9.095729516s) [noise=true , address=185.167.204.205:4011 , piece_id=O2KXUNUHCKAQZSL5BJRRRC6IIZTSXHM74OZ7REQI3RGSMFA3PN6Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.544805565 (293h31m9.096276016s) [noise=true , address=185.167.204.206:4060 , piece_id=VJPM3YAQI5N2VTEBJ5GZ75JCCBOQS5IBEFUHEWTOX3BAKSP7TWFQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.545043088 (293h31m9.096513559s) [noise=true , address=185.167.204.205:4007 , piece_id=R6HJTNG5VAAMG2Y6YXK4K4YBQFUSAJQ4774UCOSGJSRDM4RWLICA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.545385921 (293h31m9.096856382s) [noise=true , address=185.167.204.205:4034 , piece_id=H66BDBLG4T2BABKBNKMJLY6QPZT2VVSZC3GW6F5V2L2BBFOWE76Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.759454013 (293h31m10.310924484s) [noise=true , address=89.149.201.147:4006 , piece_id=T2KROQWQR5LXKBJEJO6LICLBP5CUBETT3TOACRQCGYXKZ364TYMQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.779325021 (293h31m10.330795482s) [noise=true , address=89.149.201.147:4015 , piece_id=VIFNKNEVMCI2CFBREZOLVGFJ2X7Y7ZX6FWDC6PO6CLVX2LFW4FYQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.797229894 (293h31m10.348700345s) [noise=true , address=89.149.201.147:4021 , piece_id=K3ZPQXTSSUGIGIX33FCYOC6FRCTWDBRVRHUHHGEJLNWFCJFCIT4Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.804823945 (293h31m10.356294406s) [noise=true , address=89.149.201.147:4012 , piece_id=3QAKWEG37QPAE4AORUSYURK47YIOTM5VHC563QWC43UGGPS7UV7Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.828388066 (293h31m10.379858537s) [noise=true , address=89.149.201.147:4014 , piece_id=MM2F7NMVCD3AFX2UM7FRGSO5ZD2OTJPXJEUVKJ63IL5L74LEXZSQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.848656089 (293h31m10.40012655s) [noise=true , address=89.149.201.147:4001 , piece_id=OGRBX7OEEUPFXWU3EOYD44FGBQ7FQ5CHHZ2RVRHFJGOQ74EK7K5A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:02.180750503 (293h32m0.732220964s) [noise=true , address=81.171.14.70:4028 , piece_id=ESUHXDH256EM67DXYQ7O5THLHNAPPJ6I3ARZC5D3GSLJD7CNJV5A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:02.321725695 (293h32m0.873196156s) [noise=true , address=81.171.14.70:4044 , piece_id=KMR7RCY3AFLFIAJWIQXYC6ZSX7QLL6OKSCQMYHDKKCRX4MZAC4TQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:03.575396105 (293h32m2.126866556s) [noise=true , address=81.171.14.70:4041 , piece_id=N36O2DAKHTDWU4BPERULQHIUWQQDM5TQUQD7Y36XDA7SPPQML2LQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:05.676531843 (293h32m4.228002294s) [noise=true , address=81.171.14.70:4047 , piece_id=Q4N3CMZGTJGYFS35Y75GJHLQLEYMRIX4SSGTJXEHCCXIAX5Y2T3Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:08.329832254 (293h32m6.881302715s) [noise=true , address=81.171.14.70:4043 , piece_id=TQCRSBAQ4AJPCVP2JZYD4HQF375R2K7P5AIMCZW5EJVFJYUGKYGA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:08.628985817 (293h32m7.180456278s) [noise=true , address=81.171.14.70:4042 , piece_id=E5AMRUQ7TS2R6K7ILJ2YSG7UNGO4P5PFTOMHVF6FXOOIWW46Y2NA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12vBozgWFrZfUEQAwsqBt6CDMW47rVjZaDfvKTsCdPrvX8swkhD, address:81.171.14.70:4042): protocol: read tcp 188.166.225.170:40010->81.171.14.70:4042: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 21 13:36:08.629102389 (293h32m7.18057285s) [noise=true , address=81.171.14.70:4048 , piece_id=NUU26IUS2OGXAEVLUDEYTXJL4E6GZM3M7MP5BNBI6KTRWQ2RZT4Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12bbN67ZebXjF4iSyaWu4TDsG1XugTm1YdNHqRdRCbZ2WWRunRG, address:81.171.14.70:4048): protocol: write tcp 188.166.225.170:55932->81.171.14.70:4048: use of closed network connection; write tcp 188.166.225.170:55932->81.171.14.70:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55932->81.171.14.70:4048: use of closed network connection Nov 21 13:36:08.629502702 (293h32m7.180973163s) [noise=true , address=81.171.14.70:4031 , piece_id=LJW25GLWHJCV5CFUITJ5CPYDNWOFG3NBK4A5NUX2VVV4CYCDHWMQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12drdYnnWsuMyFo6LkcewAGGWnorRCAhxPpNqK41LvGy2zkBvng, address:81.171.14.70:4031): protocol: write tcp 188.166.225.170:58232->81.171.14.70:4031: use of closed network connection; write tcp 188.166.225.170:58232->81.171.14.70:4031: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58232->81.171.14.70:4031: use of closed network connection Nov 21 13:36:08.629974720 (293h32m7.181445181s) [noise=true , address=81.171.14.70:4026 , piece_id=ATEKV2WY2WDNBLY7W4THWSO52AJ76Q3XVT7QDGT3EBMPWVRL5EKA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12RTn3BuxasVSPYmjqgyhqKn1t6fAtq9EBtUufXqcLq48My9g2b, address:81.171.14.70:4026): protocol: write tcp 188.166.225.170:41806->81.171.14.70:4026: use of closed network connection; context deadline exceeded Nov 21 13:36:08.630048300 (293h32m7.181518751s) [seg_pos={0 0}, upload=258480]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil> Nov 21 13:36:08.800451489 (293h32m7.35192194s) [upload=258480]: Done uploading segments... Nov 21 13:36:08.800522093 (293h32m7.351992554s) [upload=258480]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24) Duration: 6.747829623s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [X..........X....X..X.X..X.X..XX.X.....X...............X.X.X.] duration: 5.001208447s trace: https://jaeger.dev.storj.io/trace/37d1ae6a9d531aca FAIL (2025-11-21 13:41:34 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001250859s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [.X..........X....X..X.X..X.X..XX.X.....X...............X.X.X] duration: 5.001935671s trace: https://jaeger.dev.storj.io/trace/37d1ae6a9d531aca FAIL (2025-11-21 13:40:44 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001959626s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)