Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [X..........X....X..X.X..X.X..XX.X.....X...............X.X.X.] duration: 5.001208447s trace: https://jaeger.dev.storj.io/trace/37d1ae6a9d531aca FAIL (2025-11-21 13:41:34 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001250859s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [.X..........X....X..X.X..X.X..XX.X.....X...............X.X.X] duration: 5.001935671s trace: https://jaeger.dev.storj.io/trace/37d1ae6a9d531aca FAIL (2025-11-21 13:40:44 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001959626s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 5.398521558s trace: https://jaeger.dev.storj.io/trace/2c4b010d23e46d91 FAIL (2025-11-21 13:40:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 24) ======LOGS====== AMRUQ7TS2R6K7ILJ2YSG7UNGO4P5PFTOMHVF6FXOOIWW46Y2NA, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.291418442 (293h31m7.842888914s) [noise=true , address=89.149.201.147:4006 , piece_id=T2KROQWQR5LXKBJEJO6LICLBP5CUBETT3TOACRQCGYXKZ364TYMQ, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.291954664 (293h31m7.843425145s) [noise=true , address=185.167.204.206:4048 , piece_id=6DBNGIBFWN5PPV72YWVRSSQVYF2AZ7EHS55PJDLDLGZEGXRYYHBA, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.293302360 (293h31m7.844772851s) [noise=true , address=89.149.201.147:4021 , piece_id=K3ZPQXTSSUGIGIX33FCYOC6FRCTWDBRVRHUHHGEJLNWFCJFCIT4Q, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.294979363 (293h31m7.846449854s) [noise=true , address=81.171.14.70:4028 , piece_id=ESUHXDH256EM67DXYQ7O5THLHNAPPJ6I3ARZC5D3GSLJD7CNJV5A, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.295760250 (293h31m7.847230711s) [noise=true , address=81.171.14.70:4048 , piece_id=NUU26IUS2OGXAEVLUDEYTXJL4E6GZM3M7MP5BNBI6KTRWQ2RZT4Q, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.300934223 (293h31m7.852404674s) [noise=true , address=81.171.14.70:4031 , piece_id=LJW25GLWHJCV5CFUITJ5CPYDNWOFG3NBK4A5NUX2VVV4CYCDHWMQ, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.301942341 (293h31m7.853412814s) [noise=true , address=185.167.204.205:4011 , piece_id=O2KXUNUHCKAQZSL5BJRRRC6IIZTSXHM74OZ7REQI3RGSMFA3PN6Q, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.302417767 (293h31m7.853888218s) [noise=true , address=185.167.204.205:4020 , piece_id=LPUWKZY6RPB7TGO22BL4YRA7ETB2RHWUFOXPJX7TZE3OHSFXEH7A, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.304119598 (293h31m7.855590059s) [noise=true , address=89.149.201.147:4014 , piece_id=MM2F7NMVCD3AFX2UM7FRGSO5ZD2OTJPXJEUVKJ63IL5L74LEXZSQ, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:10.220344333 (293h31m8.771814794s) [upload=258480]: Next returned nil segment Nov 21 13:35:10.220363840 (293h31m8.771834301s) [upload=258480]: Waiting for error group managing segments... Nov 21 13:35:10.536640979 (293h31m9.08811144s) [noise=true , address=185.167.204.205:4027 , piece_id=7C2DDLUHJQCI62D2YTI76NDABUZRSPK3FIZ7N45OG5GX3SZWCHNA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.537115091 (293h31m9.088585552s) [noise=true , address=185.167.204.205:4020 , piece_id=LPUWKZY6RPB7TGO22BL4YRA7ETB2RHWUFOXPJX7TZE3OHSFXEH7A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.539139225 (293h31m9.090609686s) [noise=true , address=185.167.204.206:4056 , piece_id=7VD7D3HW5YIZUYRZ5LWEB4VDXN2R6OEU7CTCGLKZUVKEC44V743Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.539668604 (293h31m9.091139065s) [noise=true , address=185.167.204.206:4048 , piece_id=6DBNGIBFWN5PPV72YWVRSSQVYF2AZ7EHS55PJDLDLGZEGXRYYHBA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.541238333 (293h31m9.092708794s) [noise=true , address=185.167.204.205:4008 , piece_id=UE3HOAD4KZRUWKY63RDOZ5LQ7VYAQPNV653CNSWCRXD2WL5MR27A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.542375438 (293h31m9.093845919s) [noise=true , address=185.167.204.205:4036 , piece_id=F652JXWWTK5XUJBY5SSU6IKKJE72DGITQXBRYK3LMQEF63DMRIEQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.543153891 (293h31m9.094624342s) [noise=true , address=185.167.204.205:4002 , piece_id=YFXJFZTNYWU3AZXQYDU5VBWJCPRVBCYCANXPQ77UGR5ZBY5VV53Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.543618846 (293h31m9.095089307s) [noise=true , address=185.167.204.206:4047 , piece_id=OHARKEAXGZLVG5XHD6RYKKWS57JFGH4JR2TAFIQSPUXTRMJD4WZA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.543662328 (293h31m9.095132789s) [noise=true , address=185.167.204.206:4066 , piece_id=T5X6MZEPB6P3DZ2HVMXCC4XJX5JJFXKGASJRWJDCK5TBHDEA2OLQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.544259055 (293h31m9.095729516s) [noise=true , address=185.167.204.205:4011 , piece_id=O2KXUNUHCKAQZSL5BJRRRC6IIZTSXHM74OZ7REQI3RGSMFA3PN6Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.544805565 (293h31m9.096276016s) [noise=true , address=185.167.204.206:4060 , piece_id=VJPM3YAQI5N2VTEBJ5GZ75JCCBOQS5IBEFUHEWTOX3BAKSP7TWFQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.545043088 (293h31m9.096513559s) [noise=true , address=185.167.204.205:4007 , piece_id=R6HJTNG5VAAMG2Y6YXK4K4YBQFUSAJQ4774UCOSGJSRDM4RWLICA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.545385921 (293h31m9.096856382s) [noise=true , address=185.167.204.205:4034 , piece_id=H66BDBLG4T2BABKBNKMJLY6QPZT2VVSZC3GW6F5V2L2BBFOWE76Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.759454013 (293h31m10.310924484s) [noise=true , address=89.149.201.147:4006 , piece_id=T2KROQWQR5LXKBJEJO6LICLBP5CUBETT3TOACRQCGYXKZ364TYMQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.779325021 (293h31m10.330795482s) [noise=true , address=89.149.201.147:4015 , piece_id=VIFNKNEVMCI2CFBREZOLVGFJ2X7Y7ZX6FWDC6PO6CLVX2LFW4FYQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.797229894 (293h31m10.348700345s) [noise=true , address=89.149.201.147:4021 , piece_id=K3ZPQXTSSUGIGIX33FCYOC6FRCTWDBRVRHUHHGEJLNWFCJFCIT4Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.804823945 (293h31m10.356294406s) [noise=true , address=89.149.201.147:4012 , piece_id=3QAKWEG37QPAE4AORUSYURK47YIOTM5VHC563QWC43UGGPS7UV7Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.828388066 (293h31m10.379858537s) [noise=true , address=89.149.201.147:4014 , piece_id=MM2F7NMVCD3AFX2UM7FRGSO5ZD2OTJPXJEUVKJ63IL5L74LEXZSQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.848656089 (293h31m10.40012655s) [noise=true , address=89.149.201.147:4001 , piece_id=OGRBX7OEEUPFXWU3EOYD44FGBQ7FQ5CHHZ2RVRHFJGOQ74EK7K5A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:02.180750503 (293h32m0.732220964s) [noise=true , address=81.171.14.70:4028 , piece_id=ESUHXDH256EM67DXYQ7O5THLHNAPPJ6I3ARZC5D3GSLJD7CNJV5A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:02.321725695 (293h32m0.873196156s) [noise=true , address=81.171.14.70:4044 , piece_id=KMR7RCY3AFLFIAJWIQXYC6ZSX7QLL6OKSCQMYHDKKCRX4MZAC4TQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:03.575396105 (293h32m2.126866556s) [noise=true , address=81.171.14.70:4041 , piece_id=N36O2DAKHTDWU4BPERULQHIUWQQDM5TQUQD7Y36XDA7SPPQML2LQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:05.676531843 (293h32m4.228002294s) [noise=true , address=81.171.14.70:4047 , piece_id=Q4N3CMZGTJGYFS35Y75GJHLQLEYMRIX4SSGTJXEHCCXIAX5Y2T3Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:08.329832254 (293h32m6.881302715s) [noise=true , address=81.171.14.70:4043 , piece_id=TQCRSBAQ4AJPCVP2JZYD4HQF375R2K7P5AIMCZW5EJVFJYUGKYGA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:08.628985817 (293h32m7.180456278s) [noise=true , address=81.171.14.70:4042 , piece_id=E5AMRUQ7TS2R6K7ILJ2YSG7UNGO4P5PFTOMHVF6FXOOIWW46Y2NA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12vBozgWFrZfUEQAwsqBt6CDMW47rVjZaDfvKTsCdPrvX8swkhD, address:81.171.14.70:4042): protocol: read tcp 188.166.225.170:40010->81.171.14.70:4042: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 21 13:36:08.629102389 (293h32m7.18057285s) [noise=true , address=81.171.14.70:4048 , piece_id=NUU26IUS2OGXAEVLUDEYTXJL4E6GZM3M7MP5BNBI6KTRWQ2RZT4Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12bbN67ZebXjF4iSyaWu4TDsG1XugTm1YdNHqRdRCbZ2WWRunRG, address:81.171.14.70:4048): protocol: write tcp 188.166.225.170:55932->81.171.14.70:4048: use of closed network connection; write tcp 188.166.225.170:55932->81.171.14.70:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55932->81.171.14.70:4048: use of closed network connection Nov 21 13:36:08.629502702 (293h32m7.180973163s) [noise=true , address=81.171.14.70:4031 , piece_id=LJW25GLWHJCV5CFUITJ5CPYDNWOFG3NBK4A5NUX2VVV4CYCDHWMQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12drdYnnWsuMyFo6LkcewAGGWnorRCAhxPpNqK41LvGy2zkBvng, address:81.171.14.70:4031): protocol: write tcp 188.166.225.170:58232->81.171.14.70:4031: use of closed network connection; write tcp 188.166.225.170:58232->81.171.14.70:4031: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58232->81.171.14.70:4031: use of closed network connection Nov 21 13:36:08.629974720 (293h32m7.181445181s) [noise=true , address=81.171.14.70:4026 , piece_id=ATEKV2WY2WDNBLY7W4THWSO52AJ76Q3XVT7QDGT3EBMPWVRL5EKA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12RTn3BuxasVSPYmjqgyhqKn1t6fAtq9EBtUufXqcLq48My9g2b, address:81.171.14.70:4026): protocol: write tcp 188.166.225.170:41806->81.171.14.70:4026: use of closed network connection; context deadline exceeded Nov 21 13:36:08.630048300 (293h32m7.181518751s) [seg_pos={0 0}, upload=258480]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil> Nov 21 13:36:08.800451489 (293h32m7.35192194s) [upload=258480]: Done uploading segments... Nov 21 13:36:08.800522093 (293h32m7.351992554s) [upload=258480]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24) Duration: 5.398572544s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [..X..........X....X..X.X..X.X..XX.X.....X...............X.X.] duration: 3.953442209s trace: https://jaeger.dev.storj.io/trace/38d742f527a1a7c3 FAIL (2025-11-21 13:40:06 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.953557157s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 6.333830191s trace: https://jaeger.dev.storj.io/trace/2c4b010d23e46d91 FAIL (2025-11-21 13:39:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 24) ======LOGS====== AMRUQ7TS2R6K7ILJ2YSG7UNGO4P5PFTOMHVF6FXOOIWW46Y2NA, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.291418442 (293h31m7.842888914s) [noise=true , address=89.149.201.147:4006 , piece_id=T2KROQWQR5LXKBJEJO6LICLBP5CUBETT3TOACRQCGYXKZ364TYMQ, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.291954664 (293h31m7.843425145s) [noise=true , address=185.167.204.206:4048 , piece_id=6DBNGIBFWN5PPV72YWVRSSQVYF2AZ7EHS55PJDLDLGZEGXRYYHBA, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.293302360 (293h31m7.844772851s) [noise=true , address=89.149.201.147:4021 , piece_id=K3ZPQXTSSUGIGIX33FCYOC6FRCTWDBRVRHUHHGEJLNWFCJFCIT4Q, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.294979363 (293h31m7.846449854s) [noise=true , address=81.171.14.70:4028 , piece_id=ESUHXDH256EM67DXYQ7O5THLHNAPPJ6I3ARZC5D3GSLJD7CNJV5A, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.295760250 (293h31m7.847230711s) [noise=true , address=81.171.14.70:4048 , piece_id=NUU26IUS2OGXAEVLUDEYTXJL4E6GZM3M7MP5BNBI6KTRWQ2RZT4Q, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.300934223 (293h31m7.852404674s) [noise=true , address=81.171.14.70:4031 , piece_id=LJW25GLWHJCV5CFUITJ5CPYDNWOFG3NBK4A5NUX2VVV4CYCDHWMQ, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.301942341 (293h31m7.853412814s) [noise=true , address=185.167.204.205:4011 , piece_id=O2KXUNUHCKAQZSL5BJRRRC6IIZTSXHM74OZ7REQI3RGSMFA3PN6Q, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.302417767 (293h31m7.853888218s) [noise=true , address=185.167.204.205:4020 , piece_id=LPUWKZY6RPB7TGO22BL4YRA7ETB2RHWUFOXPJX7TZE3OHSFXEH7A, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.304119598 (293h31m7.855590059s) [noise=true , address=89.149.201.147:4014 , piece_id=MM2F7NMVCD3AFX2UM7FRGSO5ZD2OTJPXJEUVKJ63IL5L74LEXZSQ, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:10.220344333 (293h31m8.771814794s) [upload=258480]: Next returned nil segment Nov 21 13:35:10.220363840 (293h31m8.771834301s) [upload=258480]: Waiting for error group managing segments... Nov 21 13:35:10.536640979 (293h31m9.08811144s) [noise=true , address=185.167.204.205:4027 , piece_id=7C2DDLUHJQCI62D2YTI76NDABUZRSPK3FIZ7N45OG5GX3SZWCHNA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.537115091 (293h31m9.088585552s) [noise=true , address=185.167.204.205:4020 , piece_id=LPUWKZY6RPB7TGO22BL4YRA7ETB2RHWUFOXPJX7TZE3OHSFXEH7A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.539139225 (293h31m9.090609686s) [noise=true , address=185.167.204.206:4056 , piece_id=7VD7D3HW5YIZUYRZ5LWEB4VDXN2R6OEU7CTCGLKZUVKEC44V743Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.539668604 (293h31m9.091139065s) [noise=true , address=185.167.204.206:4048 , piece_id=6DBNGIBFWN5PPV72YWVRSSQVYF2AZ7EHS55PJDLDLGZEGXRYYHBA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.541238333 (293h31m9.092708794s) [noise=true , address=185.167.204.205:4008 , piece_id=UE3HOAD4KZRUWKY63RDOZ5LQ7VYAQPNV653CNSWCRXD2WL5MR27A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.542375438 (293h31m9.093845919s) [noise=true , address=185.167.204.205:4036 , piece_id=F652JXWWTK5XUJBY5SSU6IKKJE72DGITQXBRYK3LMQEF63DMRIEQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.543153891 (293h31m9.094624342s) [noise=true , address=185.167.204.205:4002 , piece_id=YFXJFZTNYWU3AZXQYDU5VBWJCPRVBCYCANXPQ77UGR5ZBY5VV53Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.543618846 (293h31m9.095089307s) [noise=true , address=185.167.204.206:4047 , piece_id=OHARKEAXGZLVG5XHD6RYKKWS57JFGH4JR2TAFIQSPUXTRMJD4WZA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.543662328 (293h31m9.095132789s) [noise=true , address=185.167.204.206:4066 , piece_id=T5X6MZEPB6P3DZ2HVMXCC4XJX5JJFXKGASJRWJDCK5TBHDEA2OLQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.544259055 (293h31m9.095729516s) [noise=true , address=185.167.204.205:4011 , piece_id=O2KXUNUHCKAQZSL5BJRRRC6IIZTSXHM74OZ7REQI3RGSMFA3PN6Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.544805565 (293h31m9.096276016s) [noise=true , address=185.167.204.206:4060 , piece_id=VJPM3YAQI5N2VTEBJ5GZ75JCCBOQS5IBEFUHEWTOX3BAKSP7TWFQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.545043088 (293h31m9.096513559s) [noise=true , address=185.167.204.205:4007 , piece_id=R6HJTNG5VAAMG2Y6YXK4K4YBQFUSAJQ4774UCOSGJSRDM4RWLICA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.545385921 (293h31m9.096856382s) [noise=true , address=185.167.204.205:4034 , piece_id=H66BDBLG4T2BABKBNKMJLY6QPZT2VVSZC3GW6F5V2L2BBFOWE76Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.759454013 (293h31m10.310924484s) [noise=true , address=89.149.201.147:4006 , piece_id=T2KROQWQR5LXKBJEJO6LICLBP5CUBETT3TOACRQCGYXKZ364TYMQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.779325021 (293h31m10.330795482s) [noise=true , address=89.149.201.147:4015 , piece_id=VIFNKNEVMCI2CFBREZOLVGFJ2X7Y7ZX6FWDC6PO6CLVX2LFW4FYQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.797229894 (293h31m10.348700345s) [noise=true , address=89.149.201.147:4021 , piece_id=K3ZPQXTSSUGIGIX33FCYOC6FRCTWDBRVRHUHHGEJLNWFCJFCIT4Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.804823945 (293h31m10.356294406s) [noise=true , address=89.149.201.147:4012 , piece_id=3QAKWEG37QPAE4AORUSYURK47YIOTM5VHC563QWC43UGGPS7UV7Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.828388066 (293h31m10.379858537s) [noise=true , address=89.149.201.147:4014 , piece_id=MM2F7NMVCD3AFX2UM7FRGSO5ZD2OTJPXJEUVKJ63IL5L74LEXZSQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.848656089 (293h31m10.40012655s) [noise=true , address=89.149.201.147:4001 , piece_id=OGRBX7OEEUPFXWU3EOYD44FGBQ7FQ5CHHZ2RVRHFJGOQ74EK7K5A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:02.180750503 (293h32m0.732220964s) [noise=true , address=81.171.14.70:4028 , piece_id=ESUHXDH256EM67DXYQ7O5THLHNAPPJ6I3ARZC5D3GSLJD7CNJV5A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:02.321725695 (293h32m0.873196156s) [noise=true , address=81.171.14.70:4044 , piece_id=KMR7RCY3AFLFIAJWIQXYC6ZSX7QLL6OKSCQMYHDKKCRX4MZAC4TQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:03.575396105 (293h32m2.126866556s) [noise=true , address=81.171.14.70:4041 , piece_id=N36O2DAKHTDWU4BPERULQHIUWQQDM5TQUQD7Y36XDA7SPPQML2LQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:05.676531843 (293h32m4.228002294s) [noise=true , address=81.171.14.70:4047 , piece_id=Q4N3CMZGTJGYFS35Y75GJHLQLEYMRIX4SSGTJXEHCCXIAX5Y2T3Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:08.329832254 (293h32m6.881302715s) [noise=true , address=81.171.14.70:4043 , piece_id=TQCRSBAQ4AJPCVP2JZYD4HQF375R2K7P5AIMCZW5EJVFJYUGKYGA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:08.628985817 (293h32m7.180456278s) [noise=true , address=81.171.14.70:4042 , piece_id=E5AMRUQ7TS2R6K7ILJ2YSG7UNGO4P5PFTOMHVF6FXOOIWW46Y2NA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12vBozgWFrZfUEQAwsqBt6CDMW47rVjZaDfvKTsCdPrvX8swkhD, address:81.171.14.70:4042): protocol: read tcp 188.166.225.170:40010->81.171.14.70:4042: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 21 13:36:08.629102389 (293h32m7.18057285s) [noise=true , address=81.171.14.70:4048 , piece_id=NUU26IUS2OGXAEVLUDEYTXJL4E6GZM3M7MP5BNBI6KTRWQ2RZT4Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12bbN67ZebXjF4iSyaWu4TDsG1XugTm1YdNHqRdRCbZ2WWRunRG, address:81.171.14.70:4048): protocol: write tcp 188.166.225.170:55932->81.171.14.70:4048: use of closed network connection; write tcp 188.166.225.170:55932->81.171.14.70:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55932->81.171.14.70:4048: use of closed network connection Nov 21 13:36:08.629502702 (293h32m7.180973163s) [noise=true , address=81.171.14.70:4031 , piece_id=LJW25GLWHJCV5CFUITJ5CPYDNWOFG3NBK4A5NUX2VVV4CYCDHWMQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12drdYnnWsuMyFo6LkcewAGGWnorRCAhxPpNqK41LvGy2zkBvng, address:81.171.14.70:4031): protocol: write tcp 188.166.225.170:58232->81.171.14.70:4031: use of closed network connection; write tcp 188.166.225.170:58232->81.171.14.70:4031: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58232->81.171.14.70:4031: use of closed network connection Nov 21 13:36:08.629974720 (293h32m7.181445181s) [noise=true , address=81.171.14.70:4026 , piece_id=ATEKV2WY2WDNBLY7W4THWSO52AJ76Q3XVT7QDGT3EBMPWVRL5EKA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12RTn3BuxasVSPYmjqgyhqKn1t6fAtq9EBtUufXqcLq48My9g2b, address:81.171.14.70:4026): protocol: write tcp 188.166.225.170:41806->81.171.14.70:4026: use of closed network connection; context deadline exceeded Nov 21 13:36:08.630048300 (293h32m7.181518751s) [seg_pos={0 0}, upload=258480]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil> Nov 21 13:36:08.800451489 (293h32m7.35192194s) [upload=258480]: Done uploading segments... Nov 21 13:36:08.800522093 (293h32m7.351992554s) [upload=258480]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24) Duration: 6.333897711s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [...X..........X....X..X.X..X.X..XX.X.....X...............X.X] duration: 5.001953547s trace: https://jaeger.dev.storj.io/trace/38d742f527a1a7c3 FAIL (2025-11-21 13:39:17 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001976951s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 5.810078946s trace: https://jaeger.dev.storj.io/trace/2c4b010d23e46d91 FAIL (2025-11-21 13:38:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 24) ======LOGS====== AMRUQ7TS2R6K7ILJ2YSG7UNGO4P5PFTOMHVF6FXOOIWW46Y2NA, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.291418442 (293h31m7.842888914s) [noise=true , address=89.149.201.147:4006 , piece_id=T2KROQWQR5LXKBJEJO6LICLBP5CUBETT3TOACRQCGYXKZ364TYMQ, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.291954664 (293h31m7.843425145s) [noise=true , address=185.167.204.206:4048 , piece_id=6DBNGIBFWN5PPV72YWVRSSQVYF2AZ7EHS55PJDLDLGZEGXRYYHBA, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.293302360 (293h31m7.844772851s) [noise=true , address=89.149.201.147:4021 , piece_id=K3ZPQXTSSUGIGIX33FCYOC6FRCTWDBRVRHUHHGEJLNWFCJFCIT4Q, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.294979363 (293h31m7.846449854s) [noise=true , address=81.171.14.70:4028 , piece_id=ESUHXDH256EM67DXYQ7O5THLHNAPPJ6I3ARZC5D3GSLJD7CNJV5A, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.295760250 (293h31m7.847230711s) [noise=true , address=81.171.14.70:4048 , piece_id=NUU26IUS2OGXAEVLUDEYTXJL4E6GZM3M7MP5BNBI6KTRWQ2RZT4Q, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.300934223 (293h31m7.852404674s) [noise=true , address=81.171.14.70:4031 , piece_id=LJW25GLWHJCV5CFUITJ5CPYDNWOFG3NBK4A5NUX2VVV4CYCDHWMQ, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.301942341 (293h31m7.853412814s) [noise=true , address=185.167.204.205:4011 , piece_id=O2KXUNUHCKAQZSL5BJRRRC6IIZTSXHM74OZ7REQI3RGSMFA3PN6Q, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.302417767 (293h31m7.853888218s) [noise=true , address=185.167.204.205:4020 , piece_id=LPUWKZY6RPB7TGO22BL4YRA7ETB2RHWUFOXPJX7TZE3OHSFXEH7A, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.304119598 (293h31m7.855590059s) [noise=true , address=89.149.201.147:4014 , piece_id=MM2F7NMVCD3AFX2UM7FRGSO5ZD2OTJPXJEUVKJ63IL5L74LEXZSQ, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:10.220344333 (293h31m8.771814794s) [upload=258480]: Next returned nil segment Nov 21 13:35:10.220363840 (293h31m8.771834301s) [upload=258480]: Waiting for error group managing segments... Nov 21 13:35:10.536640979 (293h31m9.08811144s) [noise=true , address=185.167.204.205:4027 , piece_id=7C2DDLUHJQCI62D2YTI76NDABUZRSPK3FIZ7N45OG5GX3SZWCHNA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.537115091 (293h31m9.088585552s) [noise=true , address=185.167.204.205:4020 , piece_id=LPUWKZY6RPB7TGO22BL4YRA7ETB2RHWUFOXPJX7TZE3OHSFXEH7A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.539139225 (293h31m9.090609686s) [noise=true , address=185.167.204.206:4056 , piece_id=7VD7D3HW5YIZUYRZ5LWEB4VDXN2R6OEU7CTCGLKZUVKEC44V743Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.539668604 (293h31m9.091139065s) [noise=true , address=185.167.204.206:4048 , piece_id=6DBNGIBFWN5PPV72YWVRSSQVYF2AZ7EHS55PJDLDLGZEGXRYYHBA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.541238333 (293h31m9.092708794s) [noise=true , address=185.167.204.205:4008 , piece_id=UE3HOAD4KZRUWKY63RDOZ5LQ7VYAQPNV653CNSWCRXD2WL5MR27A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.542375438 (293h31m9.093845919s) [noise=true , address=185.167.204.205:4036 , piece_id=F652JXWWTK5XUJBY5SSU6IKKJE72DGITQXBRYK3LMQEF63DMRIEQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.543153891 (293h31m9.094624342s) [noise=true , address=185.167.204.205:4002 , piece_id=YFXJFZTNYWU3AZXQYDU5VBWJCPRVBCYCANXPQ77UGR5ZBY5VV53Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.543618846 (293h31m9.095089307s) [noise=true , address=185.167.204.206:4047 , piece_id=OHARKEAXGZLVG5XHD6RYKKWS57JFGH4JR2TAFIQSPUXTRMJD4WZA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.543662328 (293h31m9.095132789s) [noise=true , address=185.167.204.206:4066 , piece_id=T5X6MZEPB6P3DZ2HVMXCC4XJX5JJFXKGASJRWJDCK5TBHDEA2OLQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.544259055 (293h31m9.095729516s) [noise=true , address=185.167.204.205:4011 , piece_id=O2KXUNUHCKAQZSL5BJRRRC6IIZTSXHM74OZ7REQI3RGSMFA3PN6Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.544805565 (293h31m9.096276016s) [noise=true , address=185.167.204.206:4060 , piece_id=VJPM3YAQI5N2VTEBJ5GZ75JCCBOQS5IBEFUHEWTOX3BAKSP7TWFQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.545043088 (293h31m9.096513559s) [noise=true , address=185.167.204.205:4007 , piece_id=R6HJTNG5VAAMG2Y6YXK4K4YBQFUSAJQ4774UCOSGJSRDM4RWLICA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.545385921 (293h31m9.096856382s) [noise=true , address=185.167.204.205:4034 , piece_id=H66BDBLG4T2BABKBNKMJLY6QPZT2VVSZC3GW6F5V2L2BBFOWE76Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.759454013 (293h31m10.310924484s) [noise=true , address=89.149.201.147:4006 , piece_id=T2KROQWQR5LXKBJEJO6LICLBP5CUBETT3TOACRQCGYXKZ364TYMQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.779325021 (293h31m10.330795482s) [noise=true , address=89.149.201.147:4015 , piece_id=VIFNKNEVMCI2CFBREZOLVGFJ2X7Y7ZX6FWDC6PO6CLVX2LFW4FYQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.797229894 (293h31m10.348700345s) [noise=true , address=89.149.201.147:4021 , piece_id=K3ZPQXTSSUGIGIX33FCYOC6FRCTWDBRVRHUHHGEJLNWFCJFCIT4Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.804823945 (293h31m10.356294406s) [noise=true , address=89.149.201.147:4012 , piece_id=3QAKWEG37QPAE4AORUSYURK47YIOTM5VHC563QWC43UGGPS7UV7Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.828388066 (293h31m10.379858537s) [noise=true , address=89.149.201.147:4014 , piece_id=MM2F7NMVCD3AFX2UM7FRGSO5ZD2OTJPXJEUVKJ63IL5L74LEXZSQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.848656089 (293h31m10.40012655s) [noise=true , address=89.149.201.147:4001 , piece_id=OGRBX7OEEUPFXWU3EOYD44FGBQ7FQ5CHHZ2RVRHFJGOQ74EK7K5A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:02.180750503 (293h32m0.732220964s) [noise=true , address=81.171.14.70:4028 , piece_id=ESUHXDH256EM67DXYQ7O5THLHNAPPJ6I3ARZC5D3GSLJD7CNJV5A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:02.321725695 (293h32m0.873196156s) [noise=true , address=81.171.14.70:4044 , piece_id=KMR7RCY3AFLFIAJWIQXYC6ZSX7QLL6OKSCQMYHDKKCRX4MZAC4TQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:03.575396105 (293h32m2.126866556s) [noise=true , address=81.171.14.70:4041 , piece_id=N36O2DAKHTDWU4BPERULQHIUWQQDM5TQUQD7Y36XDA7SPPQML2LQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:05.676531843 (293h32m4.228002294s) [noise=true , address=81.171.14.70:4047 , piece_id=Q4N3CMZGTJGYFS35Y75GJHLQLEYMRIX4SSGTJXEHCCXIAX5Y2T3Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:08.329832254 (293h32m6.881302715s) [noise=true , address=81.171.14.70:4043 , piece_id=TQCRSBAQ4AJPCVP2JZYD4HQF375R2K7P5AIMCZW5EJVFJYUGKYGA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:08.628985817 (293h32m7.180456278s) [noise=true , address=81.171.14.70:4042 , piece_id=E5AMRUQ7TS2R6K7ILJ2YSG7UNGO4P5PFTOMHVF6FXOOIWW46Y2NA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12vBozgWFrZfUEQAwsqBt6CDMW47rVjZaDfvKTsCdPrvX8swkhD, address:81.171.14.70:4042): protocol: read tcp 188.166.225.170:40010->81.171.14.70:4042: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 21 13:36:08.629102389 (293h32m7.18057285s) [noise=true , address=81.171.14.70:4048 , piece_id=NUU26IUS2OGXAEVLUDEYTXJL4E6GZM3M7MP5BNBI6KTRWQ2RZT4Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12bbN67ZebXjF4iSyaWu4TDsG1XugTm1YdNHqRdRCbZ2WWRunRG, address:81.171.14.70:4048): protocol: write tcp 188.166.225.170:55932->81.171.14.70:4048: use of closed network connection; write tcp 188.166.225.170:55932->81.171.14.70:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55932->81.171.14.70:4048: use of closed network connection Nov 21 13:36:08.629502702 (293h32m7.180973163s) [noise=true , address=81.171.14.70:4031 , piece_id=LJW25GLWHJCV5CFUITJ5CPYDNWOFG3NBK4A5NUX2VVV4CYCDHWMQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12drdYnnWsuMyFo6LkcewAGGWnorRCAhxPpNqK41LvGy2zkBvng, address:81.171.14.70:4031): protocol: write tcp 188.166.225.170:58232->81.171.14.70:4031: use of closed network connection; write tcp 188.166.225.170:58232->81.171.14.70:4031: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58232->81.171.14.70:4031: use of closed network connection Nov 21 13:36:08.629974720 (293h32m7.181445181s) [noise=true , address=81.171.14.70:4026 , piece_id=ATEKV2WY2WDNBLY7W4THWSO52AJ76Q3XVT7QDGT3EBMPWVRL5EKA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12RTn3BuxasVSPYmjqgyhqKn1t6fAtq9EBtUufXqcLq48My9g2b, address:81.171.14.70:4026): protocol: write tcp 188.166.225.170:41806->81.171.14.70:4026: use of closed network connection; context deadline exceeded Nov 21 13:36:08.630048300 (293h32m7.181518751s) [seg_pos={0 0}, upload=258480]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil> Nov 21 13:36:08.800451489 (293h32m7.35192194s) [upload=258480]: Done uploading segments... Nov 21 13:36:08.800522093 (293h32m7.351992554s) [upload=258480]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24) Duration: 5.810188625s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [....X..........X....X..X.X..X.X..XX.X.....X...............X.] duration: 5.001881388s trace: https://jaeger.dev.storj.io/trace/b8b29f29803dd4c FAIL (2025-11-21 13:38:26 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001910374s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 5.830288069s trace: https://jaeger.dev.storj.io/trace/2c4b010d23e46d91 FAIL (2025-11-21 13:37:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 24) ======LOGS====== AMRUQ7TS2R6K7ILJ2YSG7UNGO4P5PFTOMHVF6FXOOIWW46Y2NA, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.291418442 (293h31m7.842888914s) [noise=true , address=89.149.201.147:4006 , piece_id=T2KROQWQR5LXKBJEJO6LICLBP5CUBETT3TOACRQCGYXKZ364TYMQ, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.291954664 (293h31m7.843425145s) [noise=true , address=185.167.204.206:4048 , piece_id=6DBNGIBFWN5PPV72YWVRSSQVYF2AZ7EHS55PJDLDLGZEGXRYYHBA, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.293302360 (293h31m7.844772851s) [noise=true , address=89.149.201.147:4021 , piece_id=K3ZPQXTSSUGIGIX33FCYOC6FRCTWDBRVRHUHHGEJLNWFCJFCIT4Q, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.294979363 (293h31m7.846449854s) [noise=true , address=81.171.14.70:4028 , piece_id=ESUHXDH256EM67DXYQ7O5THLHNAPPJ6I3ARZC5D3GSLJD7CNJV5A, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.295760250 (293h31m7.847230711s) [noise=true , address=81.171.14.70:4048 , piece_id=NUU26IUS2OGXAEVLUDEYTXJL4E6GZM3M7MP5BNBI6KTRWQ2RZT4Q, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.300934223 (293h31m7.852404674s) [noise=true , address=81.171.14.70:4031 , piece_id=LJW25GLWHJCV5CFUITJ5CPYDNWOFG3NBK4A5NUX2VVV4CYCDHWMQ, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.301942341 (293h31m7.853412814s) [noise=true , address=185.167.204.205:4011 , piece_id=O2KXUNUHCKAQZSL5BJRRRC6IIZTSXHM74OZ7REQI3RGSMFA3PN6Q, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.302417767 (293h31m7.853888218s) [noise=true , address=185.167.204.205:4020 , piece_id=LPUWKZY6RPB7TGO22BL4YRA7ETB2RHWUFOXPJX7TZE3OHSFXEH7A, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.304119598 (293h31m7.855590059s) [noise=true , address=89.149.201.147:4014 , piece_id=MM2F7NMVCD3AFX2UM7FRGSO5ZD2OTJPXJEUVKJ63IL5L74LEXZSQ, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:10.220344333 (293h31m8.771814794s) [upload=258480]: Next returned nil segment Nov 21 13:35:10.220363840 (293h31m8.771834301s) [upload=258480]: Waiting for error group managing segments... Nov 21 13:35:10.536640979 (293h31m9.08811144s) [noise=true , address=185.167.204.205:4027 , piece_id=7C2DDLUHJQCI62D2YTI76NDABUZRSPK3FIZ7N45OG5GX3SZWCHNA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.537115091 (293h31m9.088585552s) [noise=true , address=185.167.204.205:4020 , piece_id=LPUWKZY6RPB7TGO22BL4YRA7ETB2RHWUFOXPJX7TZE3OHSFXEH7A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.539139225 (293h31m9.090609686s) [noise=true , address=185.167.204.206:4056 , piece_id=7VD7D3HW5YIZUYRZ5LWEB4VDXN2R6OEU7CTCGLKZUVKEC44V743Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.539668604 (293h31m9.091139065s) [noise=true , address=185.167.204.206:4048 , piece_id=6DBNGIBFWN5PPV72YWVRSSQVYF2AZ7EHS55PJDLDLGZEGXRYYHBA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.541238333 (293h31m9.092708794s) [noise=true , address=185.167.204.205:4008 , piece_id=UE3HOAD4KZRUWKY63RDOZ5LQ7VYAQPNV653CNSWCRXD2WL5MR27A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.542375438 (293h31m9.093845919s) [noise=true , address=185.167.204.205:4036 , piece_id=F652JXWWTK5XUJBY5SSU6IKKJE72DGITQXBRYK3LMQEF63DMRIEQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.543153891 (293h31m9.094624342s) [noise=true , address=185.167.204.205:4002 , piece_id=YFXJFZTNYWU3AZXQYDU5VBWJCPRVBCYCANXPQ77UGR5ZBY5VV53Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.543618846 (293h31m9.095089307s) [noise=true , address=185.167.204.206:4047 , piece_id=OHARKEAXGZLVG5XHD6RYKKWS57JFGH4JR2TAFIQSPUXTRMJD4WZA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.543662328 (293h31m9.095132789s) [noise=true , address=185.167.204.206:4066 , piece_id=T5X6MZEPB6P3DZ2HVMXCC4XJX5JJFXKGASJRWJDCK5TBHDEA2OLQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.544259055 (293h31m9.095729516s) [noise=true , address=185.167.204.205:4011 , piece_id=O2KXUNUHCKAQZSL5BJRRRC6IIZTSXHM74OZ7REQI3RGSMFA3PN6Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.544805565 (293h31m9.096276016s) [noise=true , address=185.167.204.206:4060 , piece_id=VJPM3YAQI5N2VTEBJ5GZ75JCCBOQS5IBEFUHEWTOX3BAKSP7TWFQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.545043088 (293h31m9.096513559s) [noise=true , address=185.167.204.205:4007 , piece_id=R6HJTNG5VAAMG2Y6YXK4K4YBQFUSAJQ4774UCOSGJSRDM4RWLICA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.545385921 (293h31m9.096856382s) [noise=true , address=185.167.204.205:4034 , piece_id=H66BDBLG4T2BABKBNKMJLY6QPZT2VVSZC3GW6F5V2L2BBFOWE76Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.759454013 (293h31m10.310924484s) [noise=true , address=89.149.201.147:4006 , piece_id=T2KROQWQR5LXKBJEJO6LICLBP5CUBETT3TOACRQCGYXKZ364TYMQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.779325021 (293h31m10.330795482s) [noise=true , address=89.149.201.147:4015 , piece_id=VIFNKNEVMCI2CFBREZOLVGFJ2X7Y7ZX6FWDC6PO6CLVX2LFW4FYQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.797229894 (293h31m10.348700345s) [noise=true , address=89.149.201.147:4021 , piece_id=K3ZPQXTSSUGIGIX33FCYOC6FRCTWDBRVRHUHHGEJLNWFCJFCIT4Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.804823945 (293h31m10.356294406s) [noise=true , address=89.149.201.147:4012 , piece_id=3QAKWEG37QPAE4AORUSYURK47YIOTM5VHC563QWC43UGGPS7UV7Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.828388066 (293h31m10.379858537s) [noise=true , address=89.149.201.147:4014 , piece_id=MM2F7NMVCD3AFX2UM7FRGSO5ZD2OTJPXJEUVKJ63IL5L74LEXZSQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.848656089 (293h31m10.40012655s) [noise=true , address=89.149.201.147:4001 , piece_id=OGRBX7OEEUPFXWU3EOYD44FGBQ7FQ5CHHZ2RVRHFJGOQ74EK7K5A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:02.180750503 (293h32m0.732220964s) [noise=true , address=81.171.14.70:4028 , piece_id=ESUHXDH256EM67DXYQ7O5THLHNAPPJ6I3ARZC5D3GSLJD7CNJV5A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:02.321725695 (293h32m0.873196156s) [noise=true , address=81.171.14.70:4044 , piece_id=KMR7RCY3AFLFIAJWIQXYC6ZSX7QLL6OKSCQMYHDKKCRX4MZAC4TQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:03.575396105 (293h32m2.126866556s) [noise=true , address=81.171.14.70:4041 , piece_id=N36O2DAKHTDWU4BPERULQHIUWQQDM5TQUQD7Y36XDA7SPPQML2LQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:05.676531843 (293h32m4.228002294s) [noise=true , address=81.171.14.70:4047 , piece_id=Q4N3CMZGTJGYFS35Y75GJHLQLEYMRIX4SSGTJXEHCCXIAX5Y2T3Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:08.329832254 (293h32m6.881302715s) [noise=true , address=81.171.14.70:4043 , piece_id=TQCRSBAQ4AJPCVP2JZYD4HQF375R2K7P5AIMCZW5EJVFJYUGKYGA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:08.628985817 (293h32m7.180456278s) [noise=true , address=81.171.14.70:4042 , piece_id=E5AMRUQ7TS2R6K7ILJ2YSG7UNGO4P5PFTOMHVF6FXOOIWW46Y2NA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12vBozgWFrZfUEQAwsqBt6CDMW47rVjZaDfvKTsCdPrvX8swkhD, address:81.171.14.70:4042): protocol: read tcp 188.166.225.170:40010->81.171.14.70:4042: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 21 13:36:08.629102389 (293h32m7.18057285s) [noise=true , address=81.171.14.70:4048 , piece_id=NUU26IUS2OGXAEVLUDEYTXJL4E6GZM3M7MP5BNBI6KTRWQ2RZT4Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12bbN67ZebXjF4iSyaWu4TDsG1XugTm1YdNHqRdRCbZ2WWRunRG, address:81.171.14.70:4048): protocol: write tcp 188.166.225.170:55932->81.171.14.70:4048: use of closed network connection; write tcp 188.166.225.170:55932->81.171.14.70:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55932->81.171.14.70:4048: use of closed network connection Nov 21 13:36:08.629502702 (293h32m7.180973163s) [noise=true , address=81.171.14.70:4031 , piece_id=LJW25GLWHJCV5CFUITJ5CPYDNWOFG3NBK4A5NUX2VVV4CYCDHWMQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12drdYnnWsuMyFo6LkcewAGGWnorRCAhxPpNqK41LvGy2zkBvng, address:81.171.14.70:4031): protocol: write tcp 188.166.225.170:58232->81.171.14.70:4031: use of closed network connection; write tcp 188.166.225.170:58232->81.171.14.70:4031: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58232->81.171.14.70:4031: use of closed network connection Nov 21 13:36:08.629974720 (293h32m7.181445181s) [noise=true , address=81.171.14.70:4026 , piece_id=ATEKV2WY2WDNBLY7W4THWSO52AJ76Q3XVT7QDGT3EBMPWVRL5EKA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12RTn3BuxasVSPYmjqgyhqKn1t6fAtq9EBtUufXqcLq48My9g2b, address:81.171.14.70:4026): protocol: write tcp 188.166.225.170:41806->81.171.14.70:4026: use of closed network connection; context deadline exceeded Nov 21 13:36:08.630048300 (293h32m7.181518751s) [seg_pos={0 0}, upload=258480]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil> Nov 21 13:36:08.800451489 (293h32m7.35192194s) [upload=258480]: Done uploading segments... Nov 21 13:36:08.800522093 (293h32m7.351992554s) [upload=258480]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24) Duration: 5.830316231s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [.....X..........X....X..X.X..X.X..XX.X.....X...............X] duration: 5.001004s trace: https://jaeger.dev.storj.io/trace/b8b29f29803dd4c FAIL (2025-11-21 13:37:26 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001020531s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 6.015927298s trace: https://jaeger.dev.storj.io/trace/2c4b010d23e46d91 FAIL (2025-11-21 13:37:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 24) ======LOGS====== AMRUQ7TS2R6K7ILJ2YSG7UNGO4P5PFTOMHVF6FXOOIWW46Y2NA, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.291418442 (293h31m7.842888914s) [noise=true , address=89.149.201.147:4006 , piece_id=T2KROQWQR5LXKBJEJO6LICLBP5CUBETT3TOACRQCGYXKZ364TYMQ, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.291954664 (293h31m7.843425145s) [noise=true , address=185.167.204.206:4048 , piece_id=6DBNGIBFWN5PPV72YWVRSSQVYF2AZ7EHS55PJDLDLGZEGXRYYHBA, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.293302360 (293h31m7.844772851s) [noise=true , address=89.149.201.147:4021 , piece_id=K3ZPQXTSSUGIGIX33FCYOC6FRCTWDBRVRHUHHGEJLNWFCJFCIT4Q, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.294979363 (293h31m7.846449854s) [noise=true , address=81.171.14.70:4028 , piece_id=ESUHXDH256EM67DXYQ7O5THLHNAPPJ6I3ARZC5D3GSLJD7CNJV5A, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.295760250 (293h31m7.847230711s) [noise=true , address=81.171.14.70:4048 , piece_id=NUU26IUS2OGXAEVLUDEYTXJL4E6GZM3M7MP5BNBI6KTRWQ2RZT4Q, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.300934223 (293h31m7.852404674s) [noise=true , address=81.171.14.70:4031 , piece_id=LJW25GLWHJCV5CFUITJ5CPYDNWOFG3NBK4A5NUX2VVV4CYCDHWMQ, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.301942341 (293h31m7.853412814s) [noise=true , address=185.167.204.205:4011 , piece_id=O2KXUNUHCKAQZSL5BJRRRC6IIZTSXHM74OZ7REQI3RGSMFA3PN6Q, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.302417767 (293h31m7.853888218s) [noise=true , address=185.167.204.205:4020 , piece_id=LPUWKZY6RPB7TGO22BL4YRA7ETB2RHWUFOXPJX7TZE3OHSFXEH7A, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.304119598 (293h31m7.855590059s) [noise=true , address=89.149.201.147:4014 , piece_id=MM2F7NMVCD3AFX2UM7FRGSO5ZD2OTJPXJEUVKJ63IL5L74LEXZSQ, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:10.220344333 (293h31m8.771814794s) [upload=258480]: Next returned nil segment Nov 21 13:35:10.220363840 (293h31m8.771834301s) [upload=258480]: Waiting for error group managing segments... Nov 21 13:35:10.536640979 (293h31m9.08811144s) [noise=true , address=185.167.204.205:4027 , piece_id=7C2DDLUHJQCI62D2YTI76NDABUZRSPK3FIZ7N45OG5GX3SZWCHNA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.537115091 (293h31m9.088585552s) [noise=true , address=185.167.204.205:4020 , piece_id=LPUWKZY6RPB7TGO22BL4YRA7ETB2RHWUFOXPJX7TZE3OHSFXEH7A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.539139225 (293h31m9.090609686s) [noise=true , address=185.167.204.206:4056 , piece_id=7VD7D3HW5YIZUYRZ5LWEB4VDXN2R6OEU7CTCGLKZUVKEC44V743Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.539668604 (293h31m9.091139065s) [noise=true , address=185.167.204.206:4048 , piece_id=6DBNGIBFWN5PPV72YWVRSSQVYF2AZ7EHS55PJDLDLGZEGXRYYHBA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.541238333 (293h31m9.092708794s) [noise=true , address=185.167.204.205:4008 , piece_id=UE3HOAD4KZRUWKY63RDOZ5LQ7VYAQPNV653CNSWCRXD2WL5MR27A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.542375438 (293h31m9.093845919s) [noise=true , address=185.167.204.205:4036 , piece_id=F652JXWWTK5XUJBY5SSU6IKKJE72DGITQXBRYK3LMQEF63DMRIEQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.543153891 (293h31m9.094624342s) [noise=true , address=185.167.204.205:4002 , piece_id=YFXJFZTNYWU3AZXQYDU5VBWJCPRVBCYCANXPQ77UGR5ZBY5VV53Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.543618846 (293h31m9.095089307s) [noise=true , address=185.167.204.206:4047 , piece_id=OHARKEAXGZLVG5XHD6RYKKWS57JFGH4JR2TAFIQSPUXTRMJD4WZA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.543662328 (293h31m9.095132789s) [noise=true , address=185.167.204.206:4066 , piece_id=T5X6MZEPB6P3DZ2HVMXCC4XJX5JJFXKGASJRWJDCK5TBHDEA2OLQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.544259055 (293h31m9.095729516s) [noise=true , address=185.167.204.205:4011 , piece_id=O2KXUNUHCKAQZSL5BJRRRC6IIZTSXHM74OZ7REQI3RGSMFA3PN6Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.544805565 (293h31m9.096276016s) [noise=true , address=185.167.204.206:4060 , piece_id=VJPM3YAQI5N2VTEBJ5GZ75JCCBOQS5IBEFUHEWTOX3BAKSP7TWFQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.545043088 (293h31m9.096513559s) [noise=true , address=185.167.204.205:4007 , piece_id=R6HJTNG5VAAMG2Y6YXK4K4YBQFUSAJQ4774UCOSGJSRDM4RWLICA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.545385921 (293h31m9.096856382s) [noise=true , address=185.167.204.205:4034 , piece_id=H66BDBLG4T2BABKBNKMJLY6QPZT2VVSZC3GW6F5V2L2BBFOWE76Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.759454013 (293h31m10.310924484s) [noise=true , address=89.149.201.147:4006 , piece_id=T2KROQWQR5LXKBJEJO6LICLBP5CUBETT3TOACRQCGYXKZ364TYMQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.779325021 (293h31m10.330795482s) [noise=true , address=89.149.201.147:4015 , piece_id=VIFNKNEVMCI2CFBREZOLVGFJ2X7Y7ZX6FWDC6PO6CLVX2LFW4FYQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.797229894 (293h31m10.348700345s) [noise=true , address=89.149.201.147:4021 , piece_id=K3ZPQXTSSUGIGIX33FCYOC6FRCTWDBRVRHUHHGEJLNWFCJFCIT4Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.804823945 (293h31m10.356294406s) [noise=true , address=89.149.201.147:4012 , piece_id=3QAKWEG37QPAE4AORUSYURK47YIOTM5VHC563QWC43UGGPS7UV7Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.828388066 (293h31m10.379858537s) [noise=true , address=89.149.201.147:4014 , piece_id=MM2F7NMVCD3AFX2UM7FRGSO5ZD2OTJPXJEUVKJ63IL5L74LEXZSQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.848656089 (293h31m10.40012655s) [noise=true , address=89.149.201.147:4001 , piece_id=OGRBX7OEEUPFXWU3EOYD44FGBQ7FQ5CHHZ2RVRHFJGOQ74EK7K5A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:02.180750503 (293h32m0.732220964s) [noise=true , address=81.171.14.70:4028 , piece_id=ESUHXDH256EM67DXYQ7O5THLHNAPPJ6I3ARZC5D3GSLJD7CNJV5A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:02.321725695 (293h32m0.873196156s) [noise=true , address=81.171.14.70:4044 , piece_id=KMR7RCY3AFLFIAJWIQXYC6ZSX7QLL6OKSCQMYHDKKCRX4MZAC4TQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:03.575396105 (293h32m2.126866556s) [noise=true , address=81.171.14.70:4041 , piece_id=N36O2DAKHTDWU4BPERULQHIUWQQDM5TQUQD7Y36XDA7SPPQML2LQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:05.676531843 (293h32m4.228002294s) [noise=true , address=81.171.14.70:4047 , piece_id=Q4N3CMZGTJGYFS35Y75GJHLQLEYMRIX4SSGTJXEHCCXIAX5Y2T3Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:08.329832254 (293h32m6.881302715s) [noise=true , address=81.171.14.70:4043 , piece_id=TQCRSBAQ4AJPCVP2JZYD4HQF375R2K7P5AIMCZW5EJVFJYUGKYGA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:08.628985817 (293h32m7.180456278s) [noise=true , address=81.171.14.70:4042 , piece_id=E5AMRUQ7TS2R6K7ILJ2YSG7UNGO4P5PFTOMHVF6FXOOIWW46Y2NA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12vBozgWFrZfUEQAwsqBt6CDMW47rVjZaDfvKTsCdPrvX8swkhD, address:81.171.14.70:4042): protocol: read tcp 188.166.225.170:40010->81.171.14.70:4042: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 21 13:36:08.629102389 (293h32m7.18057285s) [noise=true , address=81.171.14.70:4048 , piece_id=NUU26IUS2OGXAEVLUDEYTXJL4E6GZM3M7MP5BNBI6KTRWQ2RZT4Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12bbN67ZebXjF4iSyaWu4TDsG1XugTm1YdNHqRdRCbZ2WWRunRG, address:81.171.14.70:4048): protocol: write tcp 188.166.225.170:55932->81.171.14.70:4048: use of closed network connection; write tcp 188.166.225.170:55932->81.171.14.70:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55932->81.171.14.70:4048: use of closed network connection Nov 21 13:36:08.629502702 (293h32m7.180973163s) [noise=true , address=81.171.14.70:4031 , piece_id=LJW25GLWHJCV5CFUITJ5CPYDNWOFG3NBK4A5NUX2VVV4CYCDHWMQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12drdYnnWsuMyFo6LkcewAGGWnorRCAhxPpNqK41LvGy2zkBvng, address:81.171.14.70:4031): protocol: write tcp 188.166.225.170:58232->81.171.14.70:4031: use of closed network connection; write tcp 188.166.225.170:58232->81.171.14.70:4031: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58232->81.171.14.70:4031: use of closed network connection Nov 21 13:36:08.629974720 (293h32m7.181445181s) [noise=true , address=81.171.14.70:4026 , piece_id=ATEKV2WY2WDNBLY7W4THWSO52AJ76Q3XVT7QDGT3EBMPWVRL5EKA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12RTn3BuxasVSPYmjqgyhqKn1t6fAtq9EBtUufXqcLq48My9g2b, address:81.171.14.70:4026): protocol: write tcp 188.166.225.170:41806->81.171.14.70:4026: use of closed network connection; context deadline exceeded Nov 21 13:36:08.630048300 (293h32m7.181518751s) [seg_pos={0 0}, upload=258480]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil> Nov 21 13:36:08.800451489 (293h32m7.35192194s) [upload=258480]: Done uploading segments... Nov 21 13:36:08.800522093 (293h32m7.351992554s) [upload=258480]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24) Duration: 6.01595938s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 6.245461753s trace: https://jaeger.dev.storj.io/trace/2c4b010d23e46d91 FAIL (2025-11-21 13:36:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 24) ======LOGS====== AMRUQ7TS2R6K7ILJ2YSG7UNGO4P5PFTOMHVF6FXOOIWW46Y2NA, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.291418442 (293h31m7.842888914s) [noise=true , address=89.149.201.147:4006 , piece_id=T2KROQWQR5LXKBJEJO6LICLBP5CUBETT3TOACRQCGYXKZ364TYMQ, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.291954664 (293h31m7.843425145s) [noise=true , address=185.167.204.206:4048 , piece_id=6DBNGIBFWN5PPV72YWVRSSQVYF2AZ7EHS55PJDLDLGZEGXRYYHBA, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.293302360 (293h31m7.844772851s) [noise=true , address=89.149.201.147:4021 , piece_id=K3ZPQXTSSUGIGIX33FCYOC6FRCTWDBRVRHUHHGEJLNWFCJFCIT4Q, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.294979363 (293h31m7.846449854s) [noise=true , address=81.171.14.70:4028 , piece_id=ESUHXDH256EM67DXYQ7O5THLHNAPPJ6I3ARZC5D3GSLJD7CNJV5A, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.295760250 (293h31m7.847230711s) [noise=true , address=81.171.14.70:4048 , piece_id=NUU26IUS2OGXAEVLUDEYTXJL4E6GZM3M7MP5BNBI6KTRWQ2RZT4Q, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.300934223 (293h31m7.852404674s) [noise=true , address=81.171.14.70:4031 , piece_id=LJW25GLWHJCV5CFUITJ5CPYDNWOFG3NBK4A5NUX2VVV4CYCDHWMQ, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.301942341 (293h31m7.853412814s) [noise=true , address=185.167.204.205:4011 , piece_id=O2KXUNUHCKAQZSL5BJRRRC6IIZTSXHM74OZ7REQI3RGSMFA3PN6Q, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.302417767 (293h31m7.853888218s) [noise=true , address=185.167.204.205:4020 , piece_id=LPUWKZY6RPB7TGO22BL4YRA7ETB2RHWUFOXPJX7TZE3OHSFXEH7A, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.304119598 (293h31m7.855590059s) [noise=true , address=89.149.201.147:4014 , piece_id=MM2F7NMVCD3AFX2UM7FRGSO5ZD2OTJPXJEUVKJ63IL5L74LEXZSQ, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:10.220344333 (293h31m8.771814794s) [upload=258480]: Next returned nil segment Nov 21 13:35:10.220363840 (293h31m8.771834301s) [upload=258480]: Waiting for error group managing segments... Nov 21 13:35:10.536640979 (293h31m9.08811144s) [noise=true , address=185.167.204.205:4027 , piece_id=7C2DDLUHJQCI62D2YTI76NDABUZRSPK3FIZ7N45OG5GX3SZWCHNA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.537115091 (293h31m9.088585552s) [noise=true , address=185.167.204.205:4020 , piece_id=LPUWKZY6RPB7TGO22BL4YRA7ETB2RHWUFOXPJX7TZE3OHSFXEH7A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.539139225 (293h31m9.090609686s) [noise=true , address=185.167.204.206:4056 , piece_id=7VD7D3HW5YIZUYRZ5LWEB4VDXN2R6OEU7CTCGLKZUVKEC44V743Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.539668604 (293h31m9.091139065s) [noise=true , address=185.167.204.206:4048 , piece_id=6DBNGIBFWN5PPV72YWVRSSQVYF2AZ7EHS55PJDLDLGZEGXRYYHBA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.541238333 (293h31m9.092708794s) [noise=true , address=185.167.204.205:4008 , piece_id=UE3HOAD4KZRUWKY63RDOZ5LQ7VYAQPNV653CNSWCRXD2WL5MR27A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.542375438 (293h31m9.093845919s) [noise=true , address=185.167.204.205:4036 , piece_id=F652JXWWTK5XUJBY5SSU6IKKJE72DGITQXBRYK3LMQEF63DMRIEQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.543153891 (293h31m9.094624342s) [noise=true , address=185.167.204.205:4002 , piece_id=YFXJFZTNYWU3AZXQYDU5VBWJCPRVBCYCANXPQ77UGR5ZBY5VV53Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.543618846 (293h31m9.095089307s) [noise=true , address=185.167.204.206:4047 , piece_id=OHARKEAXGZLVG5XHD6RYKKWS57JFGH4JR2TAFIQSPUXTRMJD4WZA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.543662328 (293h31m9.095132789s) [noise=true , address=185.167.204.206:4066 , piece_id=T5X6MZEPB6P3DZ2HVMXCC4XJX5JJFXKGASJRWJDCK5TBHDEA2OLQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.544259055 (293h31m9.095729516s) [noise=true , address=185.167.204.205:4011 , piece_id=O2KXUNUHCKAQZSL5BJRRRC6IIZTSXHM74OZ7REQI3RGSMFA3PN6Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.544805565 (293h31m9.096276016s) [noise=true , address=185.167.204.206:4060 , piece_id=VJPM3YAQI5N2VTEBJ5GZ75JCCBOQS5IBEFUHEWTOX3BAKSP7TWFQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.545043088 (293h31m9.096513559s) [noise=true , address=185.167.204.205:4007 , piece_id=R6HJTNG5VAAMG2Y6YXK4K4YBQFUSAJQ4774UCOSGJSRDM4RWLICA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.545385921 (293h31m9.096856382s) [noise=true , address=185.167.204.205:4034 , piece_id=H66BDBLG4T2BABKBNKMJLY6QPZT2VVSZC3GW6F5V2L2BBFOWE76Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.759454013 (293h31m10.310924484s) [noise=true , address=89.149.201.147:4006 , piece_id=T2KROQWQR5LXKBJEJO6LICLBP5CUBETT3TOACRQCGYXKZ364TYMQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.779325021 (293h31m10.330795482s) [noise=true , address=89.149.201.147:4015 , piece_id=VIFNKNEVMCI2CFBREZOLVGFJ2X7Y7ZX6FWDC6PO6CLVX2LFW4FYQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.797229894 (293h31m10.348700345s) [noise=true , address=89.149.201.147:4021 , piece_id=K3ZPQXTSSUGIGIX33FCYOC6FRCTWDBRVRHUHHGEJLNWFCJFCIT4Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.804823945 (293h31m10.356294406s) [noise=true , address=89.149.201.147:4012 , piece_id=3QAKWEG37QPAE4AORUSYURK47YIOTM5VHC563QWC43UGGPS7UV7Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.828388066 (293h31m10.379858537s) [noise=true , address=89.149.201.147:4014 , piece_id=MM2F7NMVCD3AFX2UM7FRGSO5ZD2OTJPXJEUVKJ63IL5L74LEXZSQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.848656089 (293h31m10.40012655s) [noise=true , address=89.149.201.147:4001 , piece_id=OGRBX7OEEUPFXWU3EOYD44FGBQ7FQ5CHHZ2RVRHFJGOQ74EK7K5A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:02.180750503 (293h32m0.732220964s) [noise=true , address=81.171.14.70:4028 , piece_id=ESUHXDH256EM67DXYQ7O5THLHNAPPJ6I3ARZC5D3GSLJD7CNJV5A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:02.321725695 (293h32m0.873196156s) [noise=true , address=81.171.14.70:4044 , piece_id=KMR7RCY3AFLFIAJWIQXYC6ZSX7QLL6OKSCQMYHDKKCRX4MZAC4TQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:03.575396105 (293h32m2.126866556s) [noise=true , address=81.171.14.70:4041 , piece_id=N36O2DAKHTDWU4BPERULQHIUWQQDM5TQUQD7Y36XDA7SPPQML2LQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:05.676531843 (293h32m4.228002294s) [noise=true , address=81.171.14.70:4047 , piece_id=Q4N3CMZGTJGYFS35Y75GJHLQLEYMRIX4SSGTJXEHCCXIAX5Y2T3Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:08.329832254 (293h32m6.881302715s) [noise=true , address=81.171.14.70:4043 , piece_id=TQCRSBAQ4AJPCVP2JZYD4HQF375R2K7P5AIMCZW5EJVFJYUGKYGA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:08.628985817 (293h32m7.180456278s) [noise=true , address=81.171.14.70:4042 , piece_id=E5AMRUQ7TS2R6K7ILJ2YSG7UNGO4P5PFTOMHVF6FXOOIWW46Y2NA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12vBozgWFrZfUEQAwsqBt6CDMW47rVjZaDfvKTsCdPrvX8swkhD, address:81.171.14.70:4042): protocol: read tcp 188.166.225.170:40010->81.171.14.70:4042: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 21 13:36:08.629102389 (293h32m7.18057285s) [noise=true , address=81.171.14.70:4048 , piece_id=NUU26IUS2OGXAEVLUDEYTXJL4E6GZM3M7MP5BNBI6KTRWQ2RZT4Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12bbN67ZebXjF4iSyaWu4TDsG1XugTm1YdNHqRdRCbZ2WWRunRG, address:81.171.14.70:4048): protocol: write tcp 188.166.225.170:55932->81.171.14.70:4048: use of closed network connection; write tcp 188.166.225.170:55932->81.171.14.70:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55932->81.171.14.70:4048: use of closed network connection Nov 21 13:36:08.629502702 (293h32m7.180973163s) [noise=true , address=81.171.14.70:4031 , piece_id=LJW25GLWHJCV5CFUITJ5CPYDNWOFG3NBK4A5NUX2VVV4CYCDHWMQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12drdYnnWsuMyFo6LkcewAGGWnorRCAhxPpNqK41LvGy2zkBvng, address:81.171.14.70:4031): protocol: write tcp 188.166.225.170:58232->81.171.14.70:4031: use of closed network connection; write tcp 188.166.225.170:58232->81.171.14.70:4031: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58232->81.171.14.70:4031: use of closed network connection Nov 21 13:36:08.629974720 (293h32m7.181445181s) [noise=true , address=81.171.14.70:4026 , piece_id=ATEKV2WY2WDNBLY7W4THWSO52AJ76Q3XVT7QDGT3EBMPWVRL5EKA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12RTn3BuxasVSPYmjqgyhqKn1t6fAtq9EBtUufXqcLq48My9g2b, address:81.171.14.70:4026): protocol: write tcp 188.166.225.170:41806->81.171.14.70:4026: use of closed network connection; context deadline exceeded Nov 21 13:36:08.630048300 (293h32m7.181518751s) [seg_pos={0 0}, upload=258480]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil> Nov 21 13:36:08.800451489 (293h32m7.35192194s) [upload=258480]: Done uploading segments... Nov 21 13:36:08.800522093 (293h32m7.351992554s) [upload=258480]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24) Duration: 6.245527678s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [......X..........X....X..X.X..X.X..XX.X.....X...............] duration: 3.83950462s trace: https://jaeger.dev.storj.io/trace/38e695fc8e847ee9 FAIL (2025-11-21 13:36:16 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.839535038s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 6.709314815s trace: https://jaeger.dev.storj.io/trace/2c4b010d23e46d91 FAIL (2025-11-21 13:36:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 24) ======LOGS====== AMRUQ7TS2R6K7ILJ2YSG7UNGO4P5PFTOMHVF6FXOOIWW46Y2NA, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.291418442 (293h31m7.842888914s) [noise=true , address=89.149.201.147:4006 , piece_id=T2KROQWQR5LXKBJEJO6LICLBP5CUBETT3TOACRQCGYXKZ364TYMQ, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.291954664 (293h31m7.843425145s) [noise=true , address=185.167.204.206:4048 , piece_id=6DBNGIBFWN5PPV72YWVRSSQVYF2AZ7EHS55PJDLDLGZEGXRYYHBA, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.293302360 (293h31m7.844772851s) [noise=true , address=89.149.201.147:4021 , piece_id=K3ZPQXTSSUGIGIX33FCYOC6FRCTWDBRVRHUHHGEJLNWFCJFCIT4Q, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.294979363 (293h31m7.846449854s) [noise=true , address=81.171.14.70:4028 , piece_id=ESUHXDH256EM67DXYQ7O5THLHNAPPJ6I3ARZC5D3GSLJD7CNJV5A, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.295760250 (293h31m7.847230711s) [noise=true , address=81.171.14.70:4048 , piece_id=NUU26IUS2OGXAEVLUDEYTXJL4E6GZM3M7MP5BNBI6KTRWQ2RZT4Q, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.300934223 (293h31m7.852404674s) [noise=true , address=81.171.14.70:4031 , piece_id=LJW25GLWHJCV5CFUITJ5CPYDNWOFG3NBK4A5NUX2VVV4CYCDHWMQ, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.301942341 (293h31m7.853412814s) [noise=true , address=185.167.204.205:4011 , piece_id=O2KXUNUHCKAQZSL5BJRRRC6IIZTSXHM74OZ7REQI3RGSMFA3PN6Q, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.302417767 (293h31m7.853888218s) [noise=true , address=185.167.204.205:4020 , piece_id=LPUWKZY6RPB7TGO22BL4YRA7ETB2RHWUFOXPJX7TZE3OHSFXEH7A, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.304119598 (293h31m7.855590059s) [noise=true , address=89.149.201.147:4014 , piece_id=MM2F7NMVCD3AFX2UM7FRGSO5ZD2OTJPXJEUVKJ63IL5L74LEXZSQ, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:10.220344333 (293h31m8.771814794s) [upload=258480]: Next returned nil segment Nov 21 13:35:10.220363840 (293h31m8.771834301s) [upload=258480]: Waiting for error group managing segments... Nov 21 13:35:10.536640979 (293h31m9.08811144s) [noise=true , address=185.167.204.205:4027 , piece_id=7C2DDLUHJQCI62D2YTI76NDABUZRSPK3FIZ7N45OG5GX3SZWCHNA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.537115091 (293h31m9.088585552s) [noise=true , address=185.167.204.205:4020 , piece_id=LPUWKZY6RPB7TGO22BL4YRA7ETB2RHWUFOXPJX7TZE3OHSFXEH7A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.539139225 (293h31m9.090609686s) [noise=true , address=185.167.204.206:4056 , piece_id=7VD7D3HW5YIZUYRZ5LWEB4VDXN2R6OEU7CTCGLKZUVKEC44V743Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.539668604 (293h31m9.091139065s) [noise=true , address=185.167.204.206:4048 , piece_id=6DBNGIBFWN5PPV72YWVRSSQVYF2AZ7EHS55PJDLDLGZEGXRYYHBA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.541238333 (293h31m9.092708794s) [noise=true , address=185.167.204.205:4008 , piece_id=UE3HOAD4KZRUWKY63RDOZ5LQ7VYAQPNV653CNSWCRXD2WL5MR27A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.542375438 (293h31m9.093845919s) [noise=true , address=185.167.204.205:4036 , piece_id=F652JXWWTK5XUJBY5SSU6IKKJE72DGITQXBRYK3LMQEF63DMRIEQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.543153891 (293h31m9.094624342s) [noise=true , address=185.167.204.205:4002 , piece_id=YFXJFZTNYWU3AZXQYDU5VBWJCPRVBCYCANXPQ77UGR5ZBY5VV53Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.543618846 (293h31m9.095089307s) [noise=true , address=185.167.204.206:4047 , piece_id=OHARKEAXGZLVG5XHD6RYKKWS57JFGH4JR2TAFIQSPUXTRMJD4WZA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.543662328 (293h31m9.095132789s) [noise=true , address=185.167.204.206:4066 , piece_id=T5X6MZEPB6P3DZ2HVMXCC4XJX5JJFXKGASJRWJDCK5TBHDEA2OLQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.544259055 (293h31m9.095729516s) [noise=true , address=185.167.204.205:4011 , piece_id=O2KXUNUHCKAQZSL5BJRRRC6IIZTSXHM74OZ7REQI3RGSMFA3PN6Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.544805565 (293h31m9.096276016s) [noise=true , address=185.167.204.206:4060 , piece_id=VJPM3YAQI5N2VTEBJ5GZ75JCCBOQS5IBEFUHEWTOX3BAKSP7TWFQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.545043088 (293h31m9.096513559s) [noise=true , address=185.167.204.205:4007 , piece_id=R6HJTNG5VAAMG2Y6YXK4K4YBQFUSAJQ4774UCOSGJSRDM4RWLICA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.545385921 (293h31m9.096856382s) [noise=true , address=185.167.204.205:4034 , piece_id=H66BDBLG4T2BABKBNKMJLY6QPZT2VVSZC3GW6F5V2L2BBFOWE76Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.759454013 (293h31m10.310924484s) [noise=true , address=89.149.201.147:4006 , piece_id=T2KROQWQR5LXKBJEJO6LICLBP5CUBETT3TOACRQCGYXKZ364TYMQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.779325021 (293h31m10.330795482s) [noise=true , address=89.149.201.147:4015 , piece_id=VIFNKNEVMCI2CFBREZOLVGFJ2X7Y7ZX6FWDC6PO6CLVX2LFW4FYQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.797229894 (293h31m10.348700345s) [noise=true , address=89.149.201.147:4021 , piece_id=K3ZPQXTSSUGIGIX33FCYOC6FRCTWDBRVRHUHHGEJLNWFCJFCIT4Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.804823945 (293h31m10.356294406s) [noise=true , address=89.149.201.147:4012 , piece_id=3QAKWEG37QPAE4AORUSYURK47YIOTM5VHC563QWC43UGGPS7UV7Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.828388066 (293h31m10.379858537s) [noise=true , address=89.149.201.147:4014 , piece_id=MM2F7NMVCD3AFX2UM7FRGSO5ZD2OTJPXJEUVKJ63IL5L74LEXZSQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.848656089 (293h31m10.40012655s) [noise=true , address=89.149.201.147:4001 , piece_id=OGRBX7OEEUPFXWU3EOYD44FGBQ7FQ5CHHZ2RVRHFJGOQ74EK7K5A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:02.180750503 (293h32m0.732220964s) [noise=true , address=81.171.14.70:4028 , piece_id=ESUHXDH256EM67DXYQ7O5THLHNAPPJ6I3ARZC5D3GSLJD7CNJV5A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:02.321725695 (293h32m0.873196156s) [noise=true , address=81.171.14.70:4044 , piece_id=KMR7RCY3AFLFIAJWIQXYC6ZSX7QLL6OKSCQMYHDKKCRX4MZAC4TQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:03.575396105 (293h32m2.126866556s) [noise=true , address=81.171.14.70:4041 , piece_id=N36O2DAKHTDWU4BPERULQHIUWQQDM5TQUQD7Y36XDA7SPPQML2LQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:05.676531843 (293h32m4.228002294s) [noise=true , address=81.171.14.70:4047 , piece_id=Q4N3CMZGTJGYFS35Y75GJHLQLEYMRIX4SSGTJXEHCCXIAX5Y2T3Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:08.329832254 (293h32m6.881302715s) [noise=true , address=81.171.14.70:4043 , piece_id=TQCRSBAQ4AJPCVP2JZYD4HQF375R2K7P5AIMCZW5EJVFJYUGKYGA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:08.628985817 (293h32m7.180456278s) [noise=true , address=81.171.14.70:4042 , piece_id=E5AMRUQ7TS2R6K7ILJ2YSG7UNGO4P5PFTOMHVF6FXOOIWW46Y2NA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12vBozgWFrZfUEQAwsqBt6CDMW47rVjZaDfvKTsCdPrvX8swkhD, address:81.171.14.70:4042): protocol: read tcp 188.166.225.170:40010->81.171.14.70:4042: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 21 13:36:08.629102389 (293h32m7.18057285s) [noise=true , address=81.171.14.70:4048 , piece_id=NUU26IUS2OGXAEVLUDEYTXJL4E6GZM3M7MP5BNBI6KTRWQ2RZT4Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12bbN67ZebXjF4iSyaWu4TDsG1XugTm1YdNHqRdRCbZ2WWRunRG, address:81.171.14.70:4048): protocol: write tcp 188.166.225.170:55932->81.171.14.70:4048: use of closed network connection; write tcp 188.166.225.170:55932->81.171.14.70:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55932->81.171.14.70:4048: use of closed network connection Nov 21 13:36:08.629502702 (293h32m7.180973163s) [noise=true , address=81.171.14.70:4031 , piece_id=LJW25GLWHJCV5CFUITJ5CPYDNWOFG3NBK4A5NUX2VVV4CYCDHWMQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12drdYnnWsuMyFo6LkcewAGGWnorRCAhxPpNqK41LvGy2zkBvng, address:81.171.14.70:4031): protocol: write tcp 188.166.225.170:58232->81.171.14.70:4031: use of closed network connection; write tcp 188.166.225.170:58232->81.171.14.70:4031: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58232->81.171.14.70:4031: use of closed network connection Nov 21 13:36:08.629974720 (293h32m7.181445181s) [noise=true , address=81.171.14.70:4026 , piece_id=ATEKV2WY2WDNBLY7W4THWSO52AJ76Q3XVT7QDGT3EBMPWVRL5EKA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12RTn3BuxasVSPYmjqgyhqKn1t6fAtq9EBtUufXqcLq48My9g2b, address:81.171.14.70:4026): protocol: write tcp 188.166.225.170:41806->81.171.14.70:4026: use of closed network connection; context deadline exceeded Nov 21 13:36:08.630048300 (293h32m7.181518751s) [seg_pos={0 0}, upload=258480]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil> Nov 21 13:36:08.800451489 (293h32m7.35192194s) [upload=258480]: Done uploading segments... Nov 21 13:36:08.800522093 (293h32m7.351992554s) [upload=258480]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24) Duration: 6.709329073s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.174734449s trace: https://jaeger.dev.storj.io/trace/2c4b010d23e46d91 FAIL (2025-11-21 13:36:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 24) ======LOGS====== AMRUQ7TS2R6K7ILJ2YSG7UNGO4P5PFTOMHVF6FXOOIWW46Y2NA, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.291418442 (293h31m7.842888914s) [noise=true , address=89.149.201.147:4006 , piece_id=T2KROQWQR5LXKBJEJO6LICLBP5CUBETT3TOACRQCGYXKZ364TYMQ, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.291954664 (293h31m7.843425145s) [noise=true , address=185.167.204.206:4048 , piece_id=6DBNGIBFWN5PPV72YWVRSSQVYF2AZ7EHS55PJDLDLGZEGXRYYHBA, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.293302360 (293h31m7.844772851s) [noise=true , address=89.149.201.147:4021 , piece_id=K3ZPQXTSSUGIGIX33FCYOC6FRCTWDBRVRHUHHGEJLNWFCJFCIT4Q, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.294979363 (293h31m7.846449854s) [noise=true , address=81.171.14.70:4028 , piece_id=ESUHXDH256EM67DXYQ7O5THLHNAPPJ6I3ARZC5D3GSLJD7CNJV5A, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.295760250 (293h31m7.847230711s) [noise=true , address=81.171.14.70:4048 , piece_id=NUU26IUS2OGXAEVLUDEYTXJL4E6GZM3M7MP5BNBI6KTRWQ2RZT4Q, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.300934223 (293h31m7.852404674s) [noise=true , address=81.171.14.70:4031 , piece_id=LJW25GLWHJCV5CFUITJ5CPYDNWOFG3NBK4A5NUX2VVV4CYCDHWMQ, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.301942341 (293h31m7.853412814s) [noise=true , address=185.167.204.205:4011 , piece_id=O2KXUNUHCKAQZSL5BJRRRC6IIZTSXHM74OZ7REQI3RGSMFA3PN6Q, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.302417767 (293h31m7.853888218s) [noise=true , address=185.167.204.205:4020 , piece_id=LPUWKZY6RPB7TGO22BL4YRA7ETB2RHWUFOXPJX7TZE3OHSFXEH7A, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:09.304119598 (293h31m7.855590059s) [noise=true , address=89.149.201.147:4014 , piece_id=MM2F7NMVCD3AFX2UM7FRGSO5ZD2OTJPXJEUVKJ63IL5L74LEXZSQ, seg_pos={0 0}, upload=258480]: Uploading piece... Nov 21 13:35:10.220344333 (293h31m8.771814794s) [upload=258480]: Next returned nil segment Nov 21 13:35:10.220363840 (293h31m8.771834301s) [upload=258480]: Waiting for error group managing segments... Nov 21 13:35:10.536640979 (293h31m9.08811144s) [noise=true , address=185.167.204.205:4027 , piece_id=7C2DDLUHJQCI62D2YTI76NDABUZRSPK3FIZ7N45OG5GX3SZWCHNA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.537115091 (293h31m9.088585552s) [noise=true , address=185.167.204.205:4020 , piece_id=LPUWKZY6RPB7TGO22BL4YRA7ETB2RHWUFOXPJX7TZE3OHSFXEH7A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.539139225 (293h31m9.090609686s) [noise=true , address=185.167.204.206:4056 , piece_id=7VD7D3HW5YIZUYRZ5LWEB4VDXN2R6OEU7CTCGLKZUVKEC44V743Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.539668604 (293h31m9.091139065s) [noise=true , address=185.167.204.206:4048 , piece_id=6DBNGIBFWN5PPV72YWVRSSQVYF2AZ7EHS55PJDLDLGZEGXRYYHBA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.541238333 (293h31m9.092708794s) [noise=true , address=185.167.204.205:4008 , piece_id=UE3HOAD4KZRUWKY63RDOZ5LQ7VYAQPNV653CNSWCRXD2WL5MR27A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.542375438 (293h31m9.093845919s) [noise=true , address=185.167.204.205:4036 , piece_id=F652JXWWTK5XUJBY5SSU6IKKJE72DGITQXBRYK3LMQEF63DMRIEQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.543153891 (293h31m9.094624342s) [noise=true , address=185.167.204.205:4002 , piece_id=YFXJFZTNYWU3AZXQYDU5VBWJCPRVBCYCANXPQ77UGR5ZBY5VV53Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.543618846 (293h31m9.095089307s) [noise=true , address=185.167.204.206:4047 , piece_id=OHARKEAXGZLVG5XHD6RYKKWS57JFGH4JR2TAFIQSPUXTRMJD4WZA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.543662328 (293h31m9.095132789s) [noise=true , address=185.167.204.206:4066 , piece_id=T5X6MZEPB6P3DZ2HVMXCC4XJX5JJFXKGASJRWJDCK5TBHDEA2OLQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.544259055 (293h31m9.095729516s) [noise=true , address=185.167.204.205:4011 , piece_id=O2KXUNUHCKAQZSL5BJRRRC6IIZTSXHM74OZ7REQI3RGSMFA3PN6Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.544805565 (293h31m9.096276016s) [noise=true , address=185.167.204.206:4060 , piece_id=VJPM3YAQI5N2VTEBJ5GZ75JCCBOQS5IBEFUHEWTOX3BAKSP7TWFQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.545043088 (293h31m9.096513559s) [noise=true , address=185.167.204.205:4007 , piece_id=R6HJTNG5VAAMG2Y6YXK4K4YBQFUSAJQ4774UCOSGJSRDM4RWLICA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:10.545385921 (293h31m9.096856382s) [noise=true , address=185.167.204.205:4034 , piece_id=H66BDBLG4T2BABKBNKMJLY6QPZT2VVSZC3GW6F5V2L2BBFOWE76Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.759454013 (293h31m10.310924484s) [noise=true , address=89.149.201.147:4006 , piece_id=T2KROQWQR5LXKBJEJO6LICLBP5CUBETT3TOACRQCGYXKZ364TYMQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.779325021 (293h31m10.330795482s) [noise=true , address=89.149.201.147:4015 , piece_id=VIFNKNEVMCI2CFBREZOLVGFJ2X7Y7ZX6FWDC6PO6CLVX2LFW4FYQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.797229894 (293h31m10.348700345s) [noise=true , address=89.149.201.147:4021 , piece_id=K3ZPQXTSSUGIGIX33FCYOC6FRCTWDBRVRHUHHGEJLNWFCJFCIT4Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.804823945 (293h31m10.356294406s) [noise=true , address=89.149.201.147:4012 , piece_id=3QAKWEG37QPAE4AORUSYURK47YIOTM5VHC563QWC43UGGPS7UV7Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.828388066 (293h31m10.379858537s) [noise=true , address=89.149.201.147:4014 , piece_id=MM2F7NMVCD3AFX2UM7FRGSO5ZD2OTJPXJEUVKJ63IL5L74LEXZSQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:35:11.848656089 (293h31m10.40012655s) [noise=true , address=89.149.201.147:4001 , piece_id=OGRBX7OEEUPFXWU3EOYD44FGBQ7FQ5CHHZ2RVRHFJGOQ74EK7K5A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:02.180750503 (293h32m0.732220964s) [noise=true , address=81.171.14.70:4028 , piece_id=ESUHXDH256EM67DXYQ7O5THLHNAPPJ6I3ARZC5D3GSLJD7CNJV5A, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:02.321725695 (293h32m0.873196156s) [noise=true , address=81.171.14.70:4044 , piece_id=KMR7RCY3AFLFIAJWIQXYC6ZSX7QLL6OKSCQMYHDKKCRX4MZAC4TQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:03.575396105 (293h32m2.126866556s) [noise=true , address=81.171.14.70:4041 , piece_id=N36O2DAKHTDWU4BPERULQHIUWQQDM5TQUQD7Y36XDA7SPPQML2LQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:05.676531843 (293h32m4.228002294s) [noise=true , address=81.171.14.70:4047 , piece_id=Q4N3CMZGTJGYFS35Y75GJHLQLEYMRIX4SSGTJXEHCCXIAX5Y2T3Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:08.329832254 (293h32m6.881302715s) [noise=true , address=81.171.14.70:4043 , piece_id=TQCRSBAQ4AJPCVP2JZYD4HQF375R2K7P5AIMCZW5EJVFJYUGKYGA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: <nil> Nov 21 13:36:08.628985817 (293h32m7.180456278s) [noise=true , address=81.171.14.70:4042 , piece_id=E5AMRUQ7TS2R6K7ILJ2YSG7UNGO4P5PFTOMHVF6FXOOIWW46Y2NA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12vBozgWFrZfUEQAwsqBt6CDMW47rVjZaDfvKTsCdPrvX8swkhD, address:81.171.14.70:4042): protocol: read tcp 188.166.225.170:40010->81.171.14.70:4042: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 21 13:36:08.629102389 (293h32m7.18057285s) [noise=true , address=81.171.14.70:4048 , piece_id=NUU26IUS2OGXAEVLUDEYTXJL4E6GZM3M7MP5BNBI6KTRWQ2RZT4Q, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12bbN67ZebXjF4iSyaWu4TDsG1XugTm1YdNHqRdRCbZ2WWRunRG, address:81.171.14.70:4048): protocol: write tcp 188.166.225.170:55932->81.171.14.70:4048: use of closed network connection; write tcp 188.166.225.170:55932->81.171.14.70:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55932->81.171.14.70:4048: use of closed network connection Nov 21 13:36:08.629502702 (293h32m7.180973163s) [noise=true , address=81.171.14.70:4031 , piece_id=LJW25GLWHJCV5CFUITJ5CPYDNWOFG3NBK4A5NUX2VVV4CYCDHWMQ, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12drdYnnWsuMyFo6LkcewAGGWnorRCAhxPpNqK41LvGy2zkBvng, address:81.171.14.70:4031): protocol: write tcp 188.166.225.170:58232->81.171.14.70:4031: use of closed network connection; write tcp 188.166.225.170:58232->81.171.14.70:4031: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58232->81.171.14.70:4031: use of closed network connection Nov 21 13:36:08.629974720 (293h32m7.181445181s) [noise=true , address=81.171.14.70:4026 , piece_id=ATEKV2WY2WDNBLY7W4THWSO52AJ76Q3XVT7QDGT3EBMPWVRL5EKA, seg_pos={0 0}, upload=258480]: Done uploading piece. err: ecclient: upload failed (node:12RTn3BuxasVSPYmjqgyhqKn1t6fAtq9EBtUufXqcLq48My9g2b, address:81.171.14.70:4026): protocol: write tcp 188.166.225.170:41806->81.171.14.70:4026: use of closed network connection; context deadline exceeded Nov 21 13:36:08.630048300 (293h32m7.181518751s) [seg_pos={0 0}, upload=258480]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil> Nov 21 13:36:08.800451489 (293h32m7.35192194s) [upload=258480]: Done uploading segments... Nov 21 13:36:08.800522093 (293h32m7.351992554s) [upload=258480]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24) Duration: 1m0.174760539s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [.......X..........X....X..X.X..X.X..XX.X.....X..............] duration: 5.000748978s trace: https://jaeger.dev.storj.io/trace/38e695fc8e847ee9 FAIL (2025-11-21 13:35:24 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000774316s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [........X..........X....X..X.X..X.X..XX.X.....X.............] duration: 5.001346185s trace: https://jaeger.dev.storj.io/trace/38e695fc8e847ee9 FAIL (2025-11-21 13:34:12 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001431769s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [.........X..........X....X..X.X..X.X..XX.X.....X............] duration: 3.493634022s trace: https://jaeger.dev.storj.io/trace/38e695fc8e847ee9 FAIL (2025-11-21 13:33:42 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.493664049s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [..........X..........X....X..X.X..X.X..XX.X.....X...........] duration: 4.469971862s trace: https://jaeger.dev.storj.io/trace/38e695fc8e847ee9 FAIL (2025-11-21 13:32:46 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.470099775s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [...........X..........X....X..X.X..X.X..XX.X.....X..........] duration: 5.00171119s trace: https://jaeger.dev.storj.io/trace/38e695fc8e847ee9 FAIL (2025-11-21 13:31:34 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001858039s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [X...........X..........X....X..X.X..X.X..XX.X.....X.........] duration: 4.980977468s trace: https://jaeger.dev.storj.io/trace/38e695fc8e847ee9 FAIL (2025-11-21 13:30:47 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.981011193s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [.X...........X..........X....X..X.X..X.X..XX.X.....X........] duration: 4.528271096s trace: https://jaeger.dev.storj.io/trace/38e695fc8e847ee9 FAIL (2025-11-21 13:30:25 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.528312724s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [..X...........X..........X....X..X.X..X.X..XX.X.....X.......] duration: 5.001287834s trace: https://jaeger.dev.storj.io/trace/38e695fc8e847ee9 FAIL (2025-11-21 13:29:24 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001321819s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [...X...........X..........X....X..X.X..X.X..XX.X.....X......] duration: 3.881552709s trace: https://jaeger.dev.storj.io/trace/38e695fc8e847ee9 FAIL (2025-11-21 13:28:53 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.881667196s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [X...X...........X..........X....X..X.X..X.X..XX.X.....X.....] duration: 5.001587865s trace: https://jaeger.dev.storj.io/trace/38e695fc8e847ee9 FAIL (2025-11-21 13:28:32 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001682444s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [.X...X...........X..........X....X..X.X..X.X..XX.X.....X....] duration: 5.000764546s trace: https://jaeger.dev.storj.io/trace/38e695fc8e847ee9 FAIL (2025-11-21 13:28:17 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000793661s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [X.X...X...........X..........X....X..X.X..X.X..XX.X.....X...] duration: 3.484251806s trace: https://jaeger.dev.storj.io/trace/38e695fc8e847ee9 FAIL (2025-11-21 13:27:14 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.484281893s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [.X.X...X...........X..........X....X..X.X..X.X..XX.X.....X..] duration: 3.860988871s trace: https://jaeger.dev.storj.io/trace/38e695fc8e847ee9 FAIL (2025-11-21 13:26:23 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.861016003s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [..X.X...X...........X..........X....X..X.X..X.X..XX.X.....X.] duration: 5.000977882s trace: https://jaeger.dev.storj.io/trace/38e695fc8e847ee9 FAIL (2025-11-21 13:25:34 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001000275s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [...X.X...X...........X..........X....X..X.X..X.X..XX.X.....X] duration: 5.001045833s trace: https://jaeger.dev.storj.io/trace/38e695fc8e847ee9 FAIL (2025-11-21 13:25:16 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00107075s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [....X.X...X...........X..........X....X..X.X..X.X..XX.X.....] duration: 3.49092805s trace: https://jaeger.dev.storj.io/trace/3fb986d5e77d728e FAIL (2025-11-21 13:24:45 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.49095498s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [.....X.X...X...........X..........X....X..X.X..X.X..XX.X....] duration: 5.001312959s trace: https://jaeger.dev.storj.io/trace/3fb986d5e77d728e FAIL (2025-11-21 13:23:37 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001408952s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [......X.X...X...........X..........X....X..X.X..X.X..XX.X...] duration: 5.000705504s trace: https://jaeger.dev.storj.io/trace/3fb986d5e77d728e FAIL (2025-11-21 13:22:33 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000736524s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [.......X.X...X...........X..........X....X..X.X..X.X..XX.X..] duration: 4.76947488s trace: https://jaeger.dev.storj.io/trace/3fb986d5e77d728e FAIL (2025-11-21 13:21:24 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.769508143s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [........X.X...X...........X..........X....X..X.X..X.X..XX.X.] duration: 4.615917399s trace: https://jaeger.dev.storj.io/trace/3fb986d5e77d728e FAIL (2025-11-21 13:20:16 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.615945723s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [.........X.X...X...........X..........X....X..X.X..X.X..XX.X] duration: 5.001177335s trace: https://jaeger.dev.storj.io/trace/3fb986d5e77d728e FAIL (2025-11-21 13:19:06 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001200258s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [..........X.X...X...........X..........X....X..X.X..X.X..XX.] duration: 5.007670599s trace: https://jaeger.dev.storj.io/trace/4bf5fe69d880490a FAIL (2025-11-21 13:18:22 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.007725884s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [...........X.X...X...........X..........X....X..X.X..X.X..XX] duration: 5.001525617s trace: https://jaeger.dev.storj.io/trace/4bf5fe69d880490a FAIL (2025-11-21 13:17:27 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001557787s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [............X.X...X...........X..........X....X..X.X..X.X..X] duration: 5.001621269s trace: https://jaeger.dev.storj.io/trace/762f6836d837e54d FAIL (2025-11-21 13:17:08 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001661205s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [.............X.X...X...........X..........X....X..X.X..X.X..] duration: 5.000479838s trace: https://jaeger.dev.storj.io/trace/2fed387ecb27b04c FAIL (2025-11-21 13:16:48 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00051251s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [..............X.X...X...........X..........X....X..X.X..X.X.] duration: 3.842301084s trace: https://jaeger.dev.storj.io/trace/2fed387ecb27b04c FAIL (2025-11-21 13:16:25 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.842339156s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [...............X.X...X...........X..........X....X..X.X..X.X] duration: 5.001490992s trace: https://jaeger.dev.storj.io/trace/2fed387ecb27b04c FAIL (2025-11-21 13:15:15 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001513394s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [................X.X...X...........X..........X....X..X.X..X.] duration: 5.001118932s trace: https://jaeger.dev.storj.io/trace/3019a413613fc85c FAIL (2025-11-21 13:14:19 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001225646s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [X................X.X...X...........X..........X....X..X.X..X] duration: 5.001640416s trace: https://jaeger.dev.storj.io/trace/3019a413613fc85c FAIL (2025-11-21 13:13:18 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001668138s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [.X................X.X...X...........X..........X....X..X.X..] duration: 4.766766623s trace: https://jaeger.dev.storj.io/trace/3f34c3d4ab78a2ed FAIL (2025-11-21 13:12:04 UTC) orbiter warning: context deadline exceeded Duration: 4.766799255s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [..X................X.X...X...........X..........X....X..X.X.] duration: 3.952761341s trace: https://jaeger.dev.storj.io/trace/3f34c3d4ab78a2ed FAIL (2025-11-21 13:11:10 UTC) orbiter warning: context deadline exceeded Duration: 3.952794103s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [...X................X.X...X...........X..........X....X..X.X] duration: 5.001121907s trace: https://jaeger.dev.storj.io/trace/3f34c3d4ab78a2ed FAIL (2025-11-21 13:10:37 UTC) orbiter warning: context deadline exceeded Duration: 5.001147919s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [....X................X.X...X...........X..........X....X..X.] duration: 3.718100364s trace: https://jaeger.dev.storj.io/trace/57f90c1c2c3a28c4 FAIL (2025-11-21 13:09:24 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.718131903s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [.....X................X.X...X...........X..........X....X..X] duration: 5.002067049s trace: https://jaeger.dev.storj.io/trace/57f90c1c2c3a28c4 FAIL (2025-11-21 13:09:07 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002096926s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [......X................X.X...X...........X..........X....X..] duration: 3.627888033s trace: https://jaeger.dev.storj.io/trace/1280406a4026c84a FAIL (2025-11-21 13:08:40 UTC) orbiter warning: context deadline exceeded Duration: 3.627920947s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)