Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
2 out of 60 is failed (threshold=3) [......XX....................................................] duration:
4.878952194s
FAIL (2025-09-25 22:40:12 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 12)
======LOGS======
9 , piece_id=4YT7BXXDQVTR2QL4YBWVMO3YJCZQ57PTTEQW7GCZELWC42HQUDQQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.227500377 (56h31m29.68959285s) [noise=true , address=81.171.14.70:4034 , piece_id=EXVA3SFFPD3GJDYIFHAOP7GMJQHGZECMX5R6MHLRDMMCU7LOA7KA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.253492519 (56h31m29.715584993s) [noise=true , address=81.171.14.70:4028 , piece_id=Y237IBPHRRI7NMOYJG65AZI3LCCZ6JUJPX5WWGIDX5RYHCNVIG3A, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.267890396 (56h31m29.729982869s) [noise=true , address=89.149.201.147:4010 , piece_id=ANMXRI22IXAZFHKB7ZHFGEERKT2KGUYQEFKEOJNJDC5TFB6B3STQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.280031865 (56h31m29.742124348s) [noise=true , address=89.149.201.147:4020 , piece_id=RML2JG4QZSS6H5GTP7BW3B4SFVZLLHHAOUS3LDCOT2BGYTC3HACA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.294783955 (56h31m29.756876428s) [noise=true , address=89.149.201.147:4017 , piece_id=P576E6UE5UADQZYKR4XN4I5X4MCGMXT3B7PBGVNWBZ6I67ZVCJHA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.462969447 (56h31m29.92506192s) [noise=true , address=81.171.14.70:4035 , piece_id=GWVM3MZRGRVJ5RWG227JJXTULF5MUJ4FHWCNHLW7J3GDYXRSXRXA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.668506081 (56h31m30.130598554s) [noise=true , address=81.171.14.70:4045 , piece_id=BSKEEYML4LQZ7PWB45HQNX2YFJTFIEKIILDUZIWCXMVZZGGKHGUQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.702001372 (56h31m30.164093845s) [noise=true , address=89.149.201.147:4014 , piece_id=3VH5QXDWYX2NR3LODHSATJCP5ABF7BI52GJAR3VLPW42CPJJXQNA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.728242188 (56h31m30.190334661s) [noise=true , address=89.149.201.147:4016 , piece_id=PT5TUJDTS2TDBNLH5722GJTZ5BIO4LZKEDKBGUKHEF37TPADW7MA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.797046563 (56h31m30.259139036s) [noise=true , address=89.149.201.147:4012 , piece_id=TY6OVHC4ITI2E32GE7LP3O2YQ46FKWF6A6IIBN4JFAB7WPYXXJ2A, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:03:42.295273049 (56h32m27.757365543s) [noise=true , address=185.167.204.206:4056 , piece_id=4XAHI4JY2MPISEOZ6VJ5GCJ5WZJKHRXIDK2EQGEN7VSYNRYJH24A, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1uSTuNA28oH2s5Rkv2jaKoC2nZWSgpMJXyxwFTCKbr44EvEXrC, address:185.167.204.206:4056): protocol: rpc: dial tcp 185.167.204.206:4056: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295296423 (56h32m27.757388906s) [noise=true , address=185.167.204.205:4008 , piece_id=NXU5IGU2NUZPVR5KIUKLZ7QTPYLGDIKZQNMPBJRPCVYEVNIV7HMA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12ZHTTaGQUTftxEULFqWbH6w1HnWA7PavHrXnvMCD947FxtmsrS, address:185.167.204.205:4008): protocol: rpc: dial tcp 185.167.204.205:4008: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295427543 (56h32m27.757520026s) [noise=true , address=185.167.204.206:4064 , piece_id=7GVKAUOISAORFKOXUNOJMEEFNNUVVZUQ2RGCD6BNVCDTH6ONPQIQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1ZB3emcQntFxpr7zBiE98eesnjoTnNKinw3FPtGU2TgznGakeb, address:185.167.204.206:4064): protocol: rpc: dial tcp 185.167.204.206:4064: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295495543 (56h32m27.757588026s) [noise=true , address=185.167.204.206:4059 , piece_id=G4B2XQYXOG2LWBAIYBM4BWLI7PEPE2XDM5A57YHYR6SOWW7ODSVA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:121gTedy8TPx44wmLqdQpZ1gDDsL8D9K1EvQJffR7oKY3bT9DXh, address:185.167.204.206:4059): protocol: rpc: dial tcp 185.167.204.206:4059: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295562701 (56h32m27.757655184s) [noise=true , address=185.167.204.205:4030 , piece_id=U6SOG6BNEIWDGYWDRZOCLTROZVZXPPG5ZI52HLRPKTDKFVQN3EJA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12w5R2Whxgg4AYNNvTkKmFq4pzahF1uZiVzwgVygc9iZDAXMdSk, address:185.167.204.205:4030): protocol: rpc: dial tcp 185.167.204.205:4030: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295700453 (56h32m27.757792926s) [noise=true , address=185.167.204.205:4010 , piece_id=DPMSKQDDOU2PUXHINTG7JJXLKXJE2PC6B3KNSKN5UIGAKBM3GZVQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12oTu1GaRsH7fHqxLkQ9c9ikjG4zRN3NChssa5gfpLwrcEehoyV, address:185.167.204.205:4010): protocol: rpc: dial tcp 185.167.204.205:4010: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295713257 (56h32m27.75780574s) [noise=true , address=185.167.204.206:4053 , piece_id=74G7LWVYE7GO4GPSG67BD6QVE7UBLGIAKLSG447PWWLNBM2FZNTA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12SrFttJpndpv7TKJJYsKthWnGWBYQGHpS9CwqTd4mqBP9NCCm8, address:185.167.204.206:4053): protocol: rpc: dial tcp 185.167.204.206:4053: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295852051 (56h32m27.757944534s) [noise=true , address=185.167.204.206:4065 , piece_id=VKBLX4ASIKVHMUUCTD7GFFPEHJLKREGCDGI7LCVRSASDHBAWI6AQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12oPo26KZGZr7o7M18KZiX5PMAAMwHK8FPT59y1hSLhkB6z3Lcj, address:185.167.204.206:4065): protocol: rpc: dial tcp 185.167.204.206:4065: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295908840 (56h32m27.758001313s) [noise=true , address=185.167.204.206:4057 , piece_id=WNX7U6DN7HTE3XLVJAJ4FTCYFR3B2MLMLF5I46GALE463DWQ22WQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1xy38NywWy5CSJ7CfupY78vbkHg6cQzgNhM2Z1xnpvw9dwvMoT, address:185.167.204.206:4057): protocol: rpc: dial tcp 185.167.204.206:4057: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296027916 (56h32m27.758120399s) [noise=true , address=185.167.204.205:4003 , piece_id=NQX2I3KTMYFLS2B2C5Y2PGZRXCD7QJCZLABY4PTYVEFGAVCYX2EQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1CjWvvJ4FdZc41sRgsQFWz4Y1ATZKgzsvDcUTmBSrCC15ekCmF, address:185.167.204.205:4003): protocol: rpc: dial tcp 185.167.204.205:4003: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296100945 (56h32m27.758193418s) [noise=true , address=185.167.204.205:4032 , piece_id=AN4YMHK6NKM5RIWAZHX2FPBGQ3VRFRYBKHBPIAL2HQWBK5XXNNOA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12558dx7zMNDfabV55vSsP9XwdxXgoPM8aHW53femVZzkCRxsKN, address:185.167.204.205:4032): protocol: rpc: dial tcp 185.167.204.205:4032: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296150660 (56h32m27.758243133s) [noise=true , address=185.167.204.205:4029 , piece_id=UVEMQY4WGBKGGT75ILTBPUCFZ4U252QGPDTBPT2BNMOR3V5XYCPA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12DyF12w2JXdkGLfLx9VE1E5f28VnMg7M2b3osfsDtGX9o8Tvsn, address:185.167.204.205:4029): protocol: write tcp 188.166.225.170:36604->185.167.204.205:4029: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296276239 (56h32m27.758368722s) [noise=true , address=185.167.204.206:4069 , piece_id=VKW6XTWPZPD3MDANS3FIYYD2UPDW47WBCGLNTOVYVTWA5IT23DBQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12Uan1TT5oAM3axhaviVFbvh83gThiZvwHMwbCD31ZZens6BQ4U, address:185.167.204.206:4069): protocol: rpc: dial tcp 185.167.204.206:4069: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296438448 (56h32m27.758530931s) [noise=true , address=185.167.204.206:4048 , piece_id=PRBNW47DU6K26GYUUUZSYTYQO3GNFC6DEV2OYQDP4R2RV5L5VNXA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1sY4wHBkSVUDWWg8T8Vm1hpRbbpb3ttqnqDVY8uWyG27gqBwT9, address:185.167.204.206:4048): protocol: write tcp 188.166.225.170:59140->185.167.204.206:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296628780 (56h32m27.758721253s) [noise=true , address=185.167.204.206:4055 , piece_id=KH2E6T37MHNJJ56CR3EIKMEUEWMAVP2XNMZ7K5QVKCFZJIVXRTUA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12CYRieezm1mjtGchWdPrB2mNaHVRvTS85umjB6cQyYKnvisXSB, address:185.167.204.206:4055): protocol: rpc: dial tcp 185.167.204.206:4055: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.297155593 (56h32m27.759248076s) [noise=true , address=185.167.204.205:4022 , piece_id=VU4625JC4A4ZVXFGJZTDMDZXYCNXJ67OQTEKWRLMMHGHPEHY5J7A, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj, address:185.167.204.205:4022): protocol: write tcp 188.166.225.170:57938->185.167.204.205:4022: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.297184969 (56h32m27.759277442s) [seg_pos={0 0}, upload=49746]: Done waiting for segment. successful: 12 optimal: 26 errs: <nil>
Sep 25 22:03:42.465979471 (56h32m27.928071944s) [upload=49746]: Done uploading segments...
Sep 25 22:03:42.466010200 (56h32m27.928102673s) [upload=49746]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 12)
Duration: 4.878987042s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.eu1.gateway)antiblip:
1 out of 60 is failed (threshold=3) [.......................................X....................] duration:
182.466077ms remote:
"tcp:207.211.208.129:443" response headers:
Accept-Ranges=bytes Content-Length=352 Content-Security-Policy=block-all-mixed-content Server=Storj X-Request-Id=JdrtmpHLrtw X-Xss-Protection=1; mode=block Content-Type=application/xml Vary=Origin X-Amz-Request-Id=1868A6DD9BAF33BB Date=Thu, 25 Sep 2025 22:25:25 GMT target:
https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318
FAIL (2025-09-25 22:39:54 UTC)
orbiter warning: unexpected status code "500 Internal Server Error" "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>InternalError</Code><Message>We encountered an internal error, please try again.: cause(uplink: metaclient: context canceled)</Message><Key>hello-world.txt</Key><BucketName>uptime-checks</BucketName><Resource>/hello-world.txt</Resource><RequestId>1868A6DD9BAF33BB</RequestId><HostId></HostId></Error>"
Duration: 182.491776ms
edge
gateway
vhost
bgp
eu1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....................................................X......] duration:
5.982778277s
FAIL (2025-09-25 22:39:28 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Sep 25 22:33:20.609515379 (57h2m6.071607862s) [upload=50198]: Starting upload
Sep 25 22:33:20.609634665 (57h2m6.071727138s) [upload=50198]: Done starting upload
Sep 25 22:33:20.609747009 (57h2m6.071839492s) [upload=50198]: Uploading segments...
Sep 25 22:33:20.609932132 (57h2m6.072024605s) [upload=50198]: Got next segment. Inline: false
Sep 25 22:33:20.609977649 (57h2m6.072070132s) [upload=50198]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 25 22:33:20.609983990 (57h2m6.072076473s) [upload=50198]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 25 22:33:20.876293406 (57h2m6.338385889s) [seg_pos={0 0}, upload=50198]: Begin upload segment...
Sep 25 22:33:20.876365564 (57h2m6.338458047s) [seg_pos={0 0}, upload=50198]: Done begin upload segment.
Sep 25 22:33:20.876398446 (57h2m6.338490919s) [noise=true , address=45.91.116.7:4010 , piece_id=QZEN3HHQVP5SQ3ABXX6AUG2T7V2JTJYLKXPEM2FCIWEIRSZ2IOUA, seg_pos={0 0}, upload=50198]: Uploading piece...
Sep 25 22:33:20.876412874 (57h2m6.338505347s) [noise=true , address=145.79.227.11:4729 , piece_id=WTDDZ72XM433FAERK56RR7BWIFEFFMKJ7PKKIQQUCLWZTTTTB5TA, seg_pos={0 0}, upload=50198]: Uploading piece...
Sep 25 22:33:20.876460024 (57h2m6.338552507s) [noise=true , address=145.79.227.2:4115 , piece_id=GDDMOKVGUSPCUNXCYSADTSDEWV3WX6YLR3HSMC72AI7345WMWI6Q, seg_pos={0 0}, upload=50198]: Uploading piece...
Sep 25 22:33:20.876711823 (57h2m6.338804296s) [noise=true , address=145.79.227.4:4185 , piece_id=UDAN2TY4RX3ATTJAYBL5G3ZXKLYIXFIBPYJWRIG2MPD3LJH62UGA, seg_pos={0 0}, upload=50198]: Uploading piece...
Sep 25 22:33:20.877150188 (57h2m6.339242661s) [noise=true , address=145.79.227.6:4257 , piece_id=NLAZ3I5A6RU3PWBBSMICAETTWZ7UZLO6X5VEXUZPZ2ZB4KAI6I5A, seg_pos={0 0}, upload=50198]: Uploading piece...
Sep 25 22:33:20.877621355 (57h2m6.339713838s) [noise=true , address=145.79.227.17:4972 , piece_id=OLLDZMGRQBTZAQURMSU4AVHOAFCD3VBWCH6Y5WQXENHW2WB3D6BQ, seg_pos={0 0}, upload=50198]: Uploading piece...
Sep 25 22:33:20.877942707 (57h2m6.340035181s) [noise=true , address=145.79.227.18:4007 , piece_id=OWW73ZTC3OLJTGC4IWXZ44VT4ERGUE4YGGULZCSJKYEHNXRNLLRA, seg_pos={0 0}, upload=50198]: Uploading piece...
Sep 25 22:33:20.878252356 (57h2m6.340344829s) [noise=true , address=45.91.116.4:4939 , piece_id=7UY722GYY34MQHO6FCVNCMXKV67RUX2TTNXGUXGGDEBE63HPI6DQ, seg_pos={0 0}, upload=50198]: Uploading piece...
Sep 25 22:33:20.878573619 (57h2m6.340666092s) [noise=true , address=145.79.227.19:4079 , piece_id=TSXXBPAF2KFTUP2MDC6NKGLP2PF666PMSXPGYYKDPO25FRF7YQ6Q, seg_pos={0 0}, upload=50198]: Uploading piece...
Sep 25 22:33:20.881006501 (57h2m6.343098975s) [noise=true , address=45.91.116.10:4082 , piece_id=OJUG2YVVDOHECUD3DLDJANOWGXKS76BD3KIWFF5Q2OTRELONDWUQ, seg_pos={0 0}, upload=50198]: Uploading piece...
Sep 25 22:33:20.882423098 (57h2m6.344515571s) [noise=true , address=145.79.227.13:4802 , piece_id=UNCMDLJ7BWSS2SOAXZRUYR7EQTKZWOYFHJSVB22DHNH32VADTPNA, seg_pos={0 0}, upload=50198]: Uploading piece...
Sep 25 22:33:20.882592721 (57h2m6.344685194s) [noise=true , address=145.79.227.6:4259 , piece_id=VFXMPOPGL2UF6D6NLM7PW3GKM5YEQHT3ZDMZGOR5VEX3NNLOAXAQ, seg_pos={0 0}, upload=50198]: Uploading piece...
Sep 25 22:33:20.885196980 (57h2m6.347289464s) [noise=true , address=145.79.227.3:4151 , piece_id=ZMJ7WBX4DWOWTTJTLJXFZKBR4MRE2RS2FQWRM2YZUC6NHVEAUZFA, seg_pos={0 0}, upload=50198]: Uploading piece...
Sep 25 22:33:20.888905401 (57h2m6.350997874s) [noise=true , address=145.79.227.15:4874 , piece_id=I5VJYG36ZG63ZRJUC2WKXISA5C7EJQSV3XZJ7FLN5MWMC3CDBNEA, seg_pos={0 0}, upload=50198]: Uploading piece...
Sep 25 22:33:20.891203167 (57h2m6.35329565s) [noise=true , address=77.90.31.15:4697 , piece_id=RRP4PCTAVSJYH377LRZ43I36D4LWG7EPHE23LDBUZLP2DUR3RNDQ, seg_pos={0 0}, upload=50198]: Uploading piece...
Sep 25 22:33:20.892678096 (57h2m6.354770569s) [noise=true , address=145.79.227.15:4873 , piece_id=C3JJ5HPO237IQRGLKQ5QZSKEJM2HNOH4YNBMULOE7CXLFO4UEW3Q, seg_pos={0 0}, upload=50198]: Uploading piece...
Sep 25 22:33:22.883631748 (57h2m8.345724231s) [upload=50198]: Next returned nil segment
Sep 25 22:33:22.883655924 (57h2m8.345748397s) [upload=50198]: Waiting for error group managing segments...
Sep 25 22:33:23.186669559 (57h2m8.648762032s) [noise=true , address=77.90.31.15:4697 , piece_id=RRP4PCTAVSJYH377LRZ43I36D4LWG7EPHE23LDBUZLP2DUR3RNDQ, seg_pos={0 0}, upload=50198]: Done uploading piece. err: <nil>
Sep 25 22:33:23.402505165 (57h2m8.864597638s) [noise=true , address=45.91.116.7:4010 , piece_id=QZEN3HHQVP5SQ3ABXX6AUG2T7V2JTJYLKXPEM2FCIWEIRSZ2IOUA, seg_pos={0 0}, upload=50198]: Done uploading piece. err: <nil>
Sep 25 22:33:23.460847597 (57h2m8.92294007s) [noise=true , address=45.91.116.4:4939 , piece_id=7UY722GYY34MQHO6FCVNCMXKV67RUX2TTNXGUXGGDEBE63HPI6DQ, seg_pos={0 0}, upload=50198]: Done uploading piece. err: <nil>
Sep 25 22:33:23.475828172 (57h2m8.937920645s) [noise=true , address=45.91.116.10:4082 , piece_id=OJUG2YVVDOHECUD3DLDJANOWGXKS76BD3KIWFF5Q2OTRELONDWUQ, seg_pos={0 0}, upload=50198]: Done uploading piece. err: <nil>
Sep 25 22:33:41.316700325 (57h2m26.778792808s) [noise=true , address=145.79.227.18:4007 , piece_id=OWW73ZTC3OLJTGC4IWXZ44VT4ERGUE4YGGULZCSJKYEHNXRNLLRA, seg_pos={0 0}, upload=50198]: Done uploading piece. err: <nil>
Sep 25 22:33:42.134617358 (57h2m27.596709841s) [noise=true , address=145.79.227.11:4729 , piece_id=WTDDZ72XM433FAERK56RR7BWIFEFFMKJ7PKKIQQUCLWZTTTTB5TA, seg_pos={0 0}, upload=50198]: Done uploading piece. err: <nil>
Sep 25 22:33:43.316084677 (57h2m28.77817716s) [noise=true , address=145.79.227.15:4874 , piece_id=I5VJYG36ZG63ZRJUC2WKXISA5C7EJQSV3XZJ7FLN5MWMC3CDBNEA, seg_pos={0 0}, upload=50198]: Done uploading piece. err: <nil>
Sep 25 22:33:43.336895665 (57h2m28.798988148s) [noise=true , address=145.79.227.15:4873 , piece_id=C3JJ5HPO237IQRGLKQ5QZSKEJM2HNOH4YNBMULOE7CXLFO4UEW3Q, seg_pos={0 0}, upload=50198]: Done uploading piece. err: <nil>
Sep 25 22:33:44.766574920 (57h2m30.228667393s) [noise=true , address=145.79.227.17:4972 , piece_id=OLLDZMGRQBTZAQURMSU4AVHOAFCD3VBWCH6Y5WQXENHW2WB3D6BQ, seg_pos={0 0}, upload=50198]: Done uploading piece. err: <nil>
Sep 25 22:33:47.245353345 (57h2m32.707445818s) [noise=true , address=145.79.227.13:4802 , piece_id=UNCMDLJ7BWSS2SOAXZRUYR7EQTKZWOYFHJSVB22DHNH32VADTPNA, seg_pos={0 0}, upload=50198]: Done uploading piece. err: <nil>
Sep 25 22:33:48.389979384 (57h2m33.852071867s) [noise=true , address=145.79.227.19:4079 , piece_id=TSXXBPAF2KFTUP2MDC6NKGLP2PF666PMSXPGYYKDPO25FRF7YQ6Q, seg_pos={0 0}, upload=50198]: Done uploading piece. err: <nil>
Sep 25 22:33:49.870866176 (57h2m35.332958649s) [noise=true , address=145.79.227.3:4151 , piece_id=ZMJ7WBX4DWOWTTJTLJXFZKBR4MRE2RS2FQWRM2YZUC6NHVEAUZFA, seg_pos={0 0}, upload=50198]: Done uploading piece. err: <nil>
Sep 25 22:33:49.927808223 (57h2m35.389900706s) [noise=true , address=145.79.227.2:4115 , piece_id=GDDMOKVGUSPCUNXCYSADTSDEWV3WX6YLR3HSMC72AI7345WMWI6Q, seg_pos={0 0}, upload=50198]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: expected piece hash; context deadline exceeded; EOF
Sep 25 22:33:49.927871833 (57h2m35.389964316s) [noise=true , address=145.79.227.6:4259 , piece_id=VFXMPOPGL2UF6D6NLM7PW3GKM5YEQHT3ZDMZGOR5VEX3NNLOAXAQ, seg_pos={0 0}, upload=50198]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: expected piece hash; context deadline exceeded; EOF
Sep 25 22:33:49.927907451 (57h2m35.389999924s) [noise=true , address=145.79.227.4:4185 , piece_id=UDAN2TY4RX3ATTJAYBL5G3ZXKLYIXFIBPYJWRIG2MPD3LJH62UGA, seg_pos={0 0}, upload=50198]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:145.79.227.4:4185): protocol: write tcp 188.166.225.170:46508->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:46508->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46508->145.79.227.4:4185: use of closed network connection
Sep 25 22:33:49.927937458 (57h2m35.390029931s) [noise=true , address=145.79.227.6:4257 , piece_id=NLAZ3I5A6RU3PWBBSMICAETTWZ7UZLO6X5VEXUZPZ2ZB4KAI6I5A, seg_pos={0 0}, upload=50198]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: expected piece hash; context deadline exceeded; EOF
Sep 25 22:33:49.927971413 (57h2m35.390063886s) [seg_pos={0 0}, upload=50198]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Sep 25 22:33:50.167900726 (57h2m35.629993199s) [upload=50198]: Done uploading segments...
Sep 25 22:33:50.167951733 (57h2m35.630044216s) [upload=50198]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 5.982806351s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
2 out of 60 is failed (threshold=3) [.......XX...................................................] duration:
4.809947709s
FAIL (2025-09-25 22:39:12 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 12)
======LOGS======
9 , piece_id=4YT7BXXDQVTR2QL4YBWVMO3YJCZQ57PTTEQW7GCZELWC42HQUDQQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.227500377 (56h31m29.68959285s) [noise=true , address=81.171.14.70:4034 , piece_id=EXVA3SFFPD3GJDYIFHAOP7GMJQHGZECMX5R6MHLRDMMCU7LOA7KA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.253492519 (56h31m29.715584993s) [noise=true , address=81.171.14.70:4028 , piece_id=Y237IBPHRRI7NMOYJG65AZI3LCCZ6JUJPX5WWGIDX5RYHCNVIG3A, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.267890396 (56h31m29.729982869s) [noise=true , address=89.149.201.147:4010 , piece_id=ANMXRI22IXAZFHKB7ZHFGEERKT2KGUYQEFKEOJNJDC5TFB6B3STQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.280031865 (56h31m29.742124348s) [noise=true , address=89.149.201.147:4020 , piece_id=RML2JG4QZSS6H5GTP7BW3B4SFVZLLHHAOUS3LDCOT2BGYTC3HACA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.294783955 (56h31m29.756876428s) [noise=true , address=89.149.201.147:4017 , piece_id=P576E6UE5UADQZYKR4XN4I5X4MCGMXT3B7PBGVNWBZ6I67ZVCJHA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.462969447 (56h31m29.92506192s) [noise=true , address=81.171.14.70:4035 , piece_id=GWVM3MZRGRVJ5RWG227JJXTULF5MUJ4FHWCNHLW7J3GDYXRSXRXA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.668506081 (56h31m30.130598554s) [noise=true , address=81.171.14.70:4045 , piece_id=BSKEEYML4LQZ7PWB45HQNX2YFJTFIEKIILDUZIWCXMVZZGGKHGUQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.702001372 (56h31m30.164093845s) [noise=true , address=89.149.201.147:4014 , piece_id=3VH5QXDWYX2NR3LODHSATJCP5ABF7BI52GJAR3VLPW42CPJJXQNA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.728242188 (56h31m30.190334661s) [noise=true , address=89.149.201.147:4016 , piece_id=PT5TUJDTS2TDBNLH5722GJTZ5BIO4LZKEDKBGUKHEF37TPADW7MA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.797046563 (56h31m30.259139036s) [noise=true , address=89.149.201.147:4012 , piece_id=TY6OVHC4ITI2E32GE7LP3O2YQ46FKWF6A6IIBN4JFAB7WPYXXJ2A, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:03:42.295273049 (56h32m27.757365543s) [noise=true , address=185.167.204.206:4056 , piece_id=4XAHI4JY2MPISEOZ6VJ5GCJ5WZJKHRXIDK2EQGEN7VSYNRYJH24A, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1uSTuNA28oH2s5Rkv2jaKoC2nZWSgpMJXyxwFTCKbr44EvEXrC, address:185.167.204.206:4056): protocol: rpc: dial tcp 185.167.204.206:4056: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295296423 (56h32m27.757388906s) [noise=true , address=185.167.204.205:4008 , piece_id=NXU5IGU2NUZPVR5KIUKLZ7QTPYLGDIKZQNMPBJRPCVYEVNIV7HMA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12ZHTTaGQUTftxEULFqWbH6w1HnWA7PavHrXnvMCD947FxtmsrS, address:185.167.204.205:4008): protocol: rpc: dial tcp 185.167.204.205:4008: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295427543 (56h32m27.757520026s) [noise=true , address=185.167.204.206:4064 , piece_id=7GVKAUOISAORFKOXUNOJMEEFNNUVVZUQ2RGCD6BNVCDTH6ONPQIQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1ZB3emcQntFxpr7zBiE98eesnjoTnNKinw3FPtGU2TgznGakeb, address:185.167.204.206:4064): protocol: rpc: dial tcp 185.167.204.206:4064: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295495543 (56h32m27.757588026s) [noise=true , address=185.167.204.206:4059 , piece_id=G4B2XQYXOG2LWBAIYBM4BWLI7PEPE2XDM5A57YHYR6SOWW7ODSVA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:121gTedy8TPx44wmLqdQpZ1gDDsL8D9K1EvQJffR7oKY3bT9DXh, address:185.167.204.206:4059): protocol: rpc: dial tcp 185.167.204.206:4059: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295562701 (56h32m27.757655184s) [noise=true , address=185.167.204.205:4030 , piece_id=U6SOG6BNEIWDGYWDRZOCLTROZVZXPPG5ZI52HLRPKTDKFVQN3EJA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12w5R2Whxgg4AYNNvTkKmFq4pzahF1uZiVzwgVygc9iZDAXMdSk, address:185.167.204.205:4030): protocol: rpc: dial tcp 185.167.204.205:4030: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295700453 (56h32m27.757792926s) [noise=true , address=185.167.204.205:4010 , piece_id=DPMSKQDDOU2PUXHINTG7JJXLKXJE2PC6B3KNSKN5UIGAKBM3GZVQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12oTu1GaRsH7fHqxLkQ9c9ikjG4zRN3NChssa5gfpLwrcEehoyV, address:185.167.204.205:4010): protocol: rpc: dial tcp 185.167.204.205:4010: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295713257 (56h32m27.75780574s) [noise=true , address=185.167.204.206:4053 , piece_id=74G7LWVYE7GO4GPSG67BD6QVE7UBLGIAKLSG447PWWLNBM2FZNTA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12SrFttJpndpv7TKJJYsKthWnGWBYQGHpS9CwqTd4mqBP9NCCm8, address:185.167.204.206:4053): protocol: rpc: dial tcp 185.167.204.206:4053: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295852051 (56h32m27.757944534s) [noise=true , address=185.167.204.206:4065 , piece_id=VKBLX4ASIKVHMUUCTD7GFFPEHJLKREGCDGI7LCVRSASDHBAWI6AQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12oPo26KZGZr7o7M18KZiX5PMAAMwHK8FPT59y1hSLhkB6z3Lcj, address:185.167.204.206:4065): protocol: rpc: dial tcp 185.167.204.206:4065: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295908840 (56h32m27.758001313s) [noise=true , address=185.167.204.206:4057 , piece_id=WNX7U6DN7HTE3XLVJAJ4FTCYFR3B2MLMLF5I46GALE463DWQ22WQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1xy38NywWy5CSJ7CfupY78vbkHg6cQzgNhM2Z1xnpvw9dwvMoT, address:185.167.204.206:4057): protocol: rpc: dial tcp 185.167.204.206:4057: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296027916 (56h32m27.758120399s) [noise=true , address=185.167.204.205:4003 , piece_id=NQX2I3KTMYFLS2B2C5Y2PGZRXCD7QJCZLABY4PTYVEFGAVCYX2EQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1CjWvvJ4FdZc41sRgsQFWz4Y1ATZKgzsvDcUTmBSrCC15ekCmF, address:185.167.204.205:4003): protocol: rpc: dial tcp 185.167.204.205:4003: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296100945 (56h32m27.758193418s) [noise=true , address=185.167.204.205:4032 , piece_id=AN4YMHK6NKM5RIWAZHX2FPBGQ3VRFRYBKHBPIAL2HQWBK5XXNNOA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12558dx7zMNDfabV55vSsP9XwdxXgoPM8aHW53femVZzkCRxsKN, address:185.167.204.205:4032): protocol: rpc: dial tcp 185.167.204.205:4032: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296150660 (56h32m27.758243133s) [noise=true , address=185.167.204.205:4029 , piece_id=UVEMQY4WGBKGGT75ILTBPUCFZ4U252QGPDTBPT2BNMOR3V5XYCPA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12DyF12w2JXdkGLfLx9VE1E5f28VnMg7M2b3osfsDtGX9o8Tvsn, address:185.167.204.205:4029): protocol: write tcp 188.166.225.170:36604->185.167.204.205:4029: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296276239 (56h32m27.758368722s) [noise=true , address=185.167.204.206:4069 , piece_id=VKW6XTWPZPD3MDANS3FIYYD2UPDW47WBCGLNTOVYVTWA5IT23DBQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12Uan1TT5oAM3axhaviVFbvh83gThiZvwHMwbCD31ZZens6BQ4U, address:185.167.204.206:4069): protocol: rpc: dial tcp 185.167.204.206:4069: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296438448 (56h32m27.758530931s) [noise=true , address=185.167.204.206:4048 , piece_id=PRBNW47DU6K26GYUUUZSYTYQO3GNFC6DEV2OYQDP4R2RV5L5VNXA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1sY4wHBkSVUDWWg8T8Vm1hpRbbpb3ttqnqDVY8uWyG27gqBwT9, address:185.167.204.206:4048): protocol: write tcp 188.166.225.170:59140->185.167.204.206:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296628780 (56h32m27.758721253s) [noise=true , address=185.167.204.206:4055 , piece_id=KH2E6T37MHNJJ56CR3EIKMEUEWMAVP2XNMZ7K5QVKCFZJIVXRTUA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12CYRieezm1mjtGchWdPrB2mNaHVRvTS85umjB6cQyYKnvisXSB, address:185.167.204.206:4055): protocol: rpc: dial tcp 185.167.204.206:4055: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.297155593 (56h32m27.759248076s) [noise=true , address=185.167.204.205:4022 , piece_id=VU4625JC4A4ZVXFGJZTDMDZXYCNXJ67OQTEKWRLMMHGHPEHY5J7A, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj, address:185.167.204.205:4022): protocol: write tcp 188.166.225.170:57938->185.167.204.205:4022: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.297184969 (56h32m27.759277442s) [seg_pos={0 0}, upload=49746]: Done waiting for segment. successful: 12 optimal: 26 errs: <nil>
Sep 25 22:03:42.465979471 (56h32m27.928071944s) [upload=49746]: Done uploading segments...
Sep 25 22:03:42.466010200 (56h32m27.928102673s) [upload=49746]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 12)
Duration: 4.809982785s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.eu1.gateway)antiblip:
1 out of 60 is failed (threshold=3) [........................................X...................] duration:
180.168399ms remote:
"tcp:207.211.208.129:443" response headers:
Accept-Ranges=bytes Content-Length=352 Content-Security-Policy=block-all-mixed-content Server=Storj X-Request-Id=JdrtmpHLrtw X-Xss-Protection=1; mode=block Content-Type=application/xml Vary=Origin X-Amz-Request-Id=1868A6DD9BAF33BB Date=Thu, 25 Sep 2025 22:25:25 GMT target:
https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318
FAIL (2025-09-25 22:39:09 UTC)
orbiter warning: unexpected status code "500 Internal Server Error" "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>InternalError</Code><Message>We encountered an internal error, please try again.: cause(uplink: metaclient: context canceled)</Message><Key>hello-world.txt</Key><BucketName>uptime-checks</BucketName><Resource>/hello-world.txt</Resource><RequestId>1868A6DD9BAF33BB</RequestId><HostId></HostId></Error>"
Duration: 180.201622ms
edge
gateway
vhost
bgp
eu1
|
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.eu1.gateway)antiblip:
1 out of 60 is failed (threshold=3) [.........................................X..................] duration:
178.596705ms remote:
"tcp:207.211.208.129:443" response headers:
Accept-Ranges=bytes Content-Length=352 Content-Security-Policy=block-all-mixed-content Server=Storj X-Request-Id=JdrtmpHLrtw X-Xss-Protection=1; mode=block Content-Type=application/xml Vary=Origin X-Amz-Request-Id=1868A6DD9BAF33BB Date=Thu, 25 Sep 2025 22:25:25 GMT target:
https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318
FAIL (2025-09-25 22:38:53 UTC)
orbiter warning: unexpected status code "500 Internal Server Error" "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>InternalError</Code><Message>We encountered an internal error, please try again.: cause(uplink: metaclient: context canceled)</Message><Key>hello-world.txt</Key><BucketName>uptime-checks</BucketName><Resource>/hello-world.txt</Resource><RequestId>1868A6DD9BAF33BB</RequestId><HostId></HostId></Error>"
Duration: 178.628465ms
edge
gateway
vhost
bgp
eu1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [......................................................X.....] duration:
6.098709815s
FAIL (2025-09-25 22:38:19 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Sep 25 22:33:20.609515379 (57h2m6.071607862s) [upload=50198]: Starting upload
Sep 25 22:33:20.609634665 (57h2m6.071727138s) [upload=50198]: Done starting upload
Sep 25 22:33:20.609747009 (57h2m6.071839492s) [upload=50198]: Uploading segments...
Sep 25 22:33:20.609932132 (57h2m6.072024605s) [upload=50198]: Got next segment. Inline: false
Sep 25 22:33:20.609977649 (57h2m6.072070132s) [upload=50198]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 25 22:33:20.609983990 (57h2m6.072076473s) [upload=50198]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 25 22:33:20.876293406 (57h2m6.338385889s) [seg_pos={0 0}, upload=50198]: Begin upload segment...
Sep 25 22:33:20.876365564 (57h2m6.338458047s) [seg_pos={0 0}, upload=50198]: Done begin upload segment.
Sep 25 22:33:20.876398446 (57h2m6.338490919s) [noise=true , address=45.91.116.7:4010 , piece_id=QZEN3HHQVP5SQ3ABXX6AUG2T7V2JTJYLKXPEM2FCIWEIRSZ2IOUA, seg_pos={0 0}, upload=50198]: Uploading piece...
Sep 25 22:33:20.876412874 (57h2m6.338505347s) [noise=true , address=145.79.227.11:4729 , piece_id=WTDDZ72XM433FAERK56RR7BWIFEFFMKJ7PKKIQQUCLWZTTTTB5TA, seg_pos={0 0}, upload=50198]: Uploading piece...
Sep 25 22:33:20.876460024 (57h2m6.338552507s) [noise=true , address=145.79.227.2:4115 , piece_id=GDDMOKVGUSPCUNXCYSADTSDEWV3WX6YLR3HSMC72AI7345WMWI6Q, seg_pos={0 0}, upload=50198]: Uploading piece...
Sep 25 22:33:20.876711823 (57h2m6.338804296s) [noise=true , address=145.79.227.4:4185 , piece_id=UDAN2TY4RX3ATTJAYBL5G3ZXKLYIXFIBPYJWRIG2MPD3LJH62UGA, seg_pos={0 0}, upload=50198]: Uploading piece...
Sep 25 22:33:20.877150188 (57h2m6.339242661s) [noise=true , address=145.79.227.6:4257 , piece_id=NLAZ3I5A6RU3PWBBSMICAETTWZ7UZLO6X5VEXUZPZ2ZB4KAI6I5A, seg_pos={0 0}, upload=50198]: Uploading piece...
Sep 25 22:33:20.877621355 (57h2m6.339713838s) [noise=true , address=145.79.227.17:4972 , piece_id=OLLDZMGRQBTZAQURMSU4AVHOAFCD3VBWCH6Y5WQXENHW2WB3D6BQ, seg_pos={0 0}, upload=50198]: Uploading piece...
Sep 25 22:33:20.877942707 (57h2m6.340035181s) [noise=true , address=145.79.227.18:4007 , piece_id=OWW73ZTC3OLJTGC4IWXZ44VT4ERGUE4YGGULZCSJKYEHNXRNLLRA, seg_pos={0 0}, upload=50198]: Uploading piece...
Sep 25 22:33:20.878252356 (57h2m6.340344829s) [noise=true , address=45.91.116.4:4939 , piece_id=7UY722GYY34MQHO6FCVNCMXKV67RUX2TTNXGUXGGDEBE63HPI6DQ, seg_pos={0 0}, upload=50198]: Uploading piece...
Sep 25 22:33:20.878573619 (57h2m6.340666092s) [noise=true , address=145.79.227.19:4079 , piece_id=TSXXBPAF2KFTUP2MDC6NKGLP2PF666PMSXPGYYKDPO25FRF7YQ6Q, seg_pos={0 0}, upload=50198]: Uploading piece...
Sep 25 22:33:20.881006501 (57h2m6.343098975s) [noise=true , address=45.91.116.10:4082 , piece_id=OJUG2YVVDOHECUD3DLDJANOWGXKS76BD3KIWFF5Q2OTRELONDWUQ, seg_pos={0 0}, upload=50198]: Uploading piece...
Sep 25 22:33:20.882423098 (57h2m6.344515571s) [noise=true , address=145.79.227.13:4802 , piece_id=UNCMDLJ7BWSS2SOAXZRUYR7EQTKZWOYFHJSVB22DHNH32VADTPNA, seg_pos={0 0}, upload=50198]: Uploading piece...
Sep 25 22:33:20.882592721 (57h2m6.344685194s) [noise=true , address=145.79.227.6:4259 , piece_id=VFXMPOPGL2UF6D6NLM7PW3GKM5YEQHT3ZDMZGOR5VEX3NNLOAXAQ, seg_pos={0 0}, upload=50198]: Uploading piece...
Sep 25 22:33:20.885196980 (57h2m6.347289464s) [noise=true , address=145.79.227.3:4151 , piece_id=ZMJ7WBX4DWOWTTJTLJXFZKBR4MRE2RS2FQWRM2YZUC6NHVEAUZFA, seg_pos={0 0}, upload=50198]: Uploading piece...
Sep 25 22:33:20.888905401 (57h2m6.350997874s) [noise=true , address=145.79.227.15:4874 , piece_id=I5VJYG36ZG63ZRJUC2WKXISA5C7EJQSV3XZJ7FLN5MWMC3CDBNEA, seg_pos={0 0}, upload=50198]: Uploading piece...
Sep 25 22:33:20.891203167 (57h2m6.35329565s) [noise=true , address=77.90.31.15:4697 , piece_id=RRP4PCTAVSJYH377LRZ43I36D4LWG7EPHE23LDBUZLP2DUR3RNDQ, seg_pos={0 0}, upload=50198]: Uploading piece...
Sep 25 22:33:20.892678096 (57h2m6.354770569s) [noise=true , address=145.79.227.15:4873 , piece_id=C3JJ5HPO237IQRGLKQ5QZSKEJM2HNOH4YNBMULOE7CXLFO4UEW3Q, seg_pos={0 0}, upload=50198]: Uploading piece...
Sep 25 22:33:22.883631748 (57h2m8.345724231s) [upload=50198]: Next returned nil segment
Sep 25 22:33:22.883655924 (57h2m8.345748397s) [upload=50198]: Waiting for error group managing segments...
Sep 25 22:33:23.186669559 (57h2m8.648762032s) [noise=true , address=77.90.31.15:4697 , piece_id=RRP4PCTAVSJYH377LRZ43I36D4LWG7EPHE23LDBUZLP2DUR3RNDQ, seg_pos={0 0}, upload=50198]: Done uploading piece. err: <nil>
Sep 25 22:33:23.402505165 (57h2m8.864597638s) [noise=true , address=45.91.116.7:4010 , piece_id=QZEN3HHQVP5SQ3ABXX6AUG2T7V2JTJYLKXPEM2FCIWEIRSZ2IOUA, seg_pos={0 0}, upload=50198]: Done uploading piece. err: <nil>
Sep 25 22:33:23.460847597 (57h2m8.92294007s) [noise=true , address=45.91.116.4:4939 , piece_id=7UY722GYY34MQHO6FCVNCMXKV67RUX2TTNXGUXGGDEBE63HPI6DQ, seg_pos={0 0}, upload=50198]: Done uploading piece. err: <nil>
Sep 25 22:33:23.475828172 (57h2m8.937920645s) [noise=true , address=45.91.116.10:4082 , piece_id=OJUG2YVVDOHECUD3DLDJANOWGXKS76BD3KIWFF5Q2OTRELONDWUQ, seg_pos={0 0}, upload=50198]: Done uploading piece. err: <nil>
Sep 25 22:33:41.316700325 (57h2m26.778792808s) [noise=true , address=145.79.227.18:4007 , piece_id=OWW73ZTC3OLJTGC4IWXZ44VT4ERGUE4YGGULZCSJKYEHNXRNLLRA, seg_pos={0 0}, upload=50198]: Done uploading piece. err: <nil>
Sep 25 22:33:42.134617358 (57h2m27.596709841s) [noise=true , address=145.79.227.11:4729 , piece_id=WTDDZ72XM433FAERK56RR7BWIFEFFMKJ7PKKIQQUCLWZTTTTB5TA, seg_pos={0 0}, upload=50198]: Done uploading piece. err: <nil>
Sep 25 22:33:43.316084677 (57h2m28.77817716s) [noise=true , address=145.79.227.15:4874 , piece_id=I5VJYG36ZG63ZRJUC2WKXISA5C7EJQSV3XZJ7FLN5MWMC3CDBNEA, seg_pos={0 0}, upload=50198]: Done uploading piece. err: <nil>
Sep 25 22:33:43.336895665 (57h2m28.798988148s) [noise=true , address=145.79.227.15:4873 , piece_id=C3JJ5HPO237IQRGLKQ5QZSKEJM2HNOH4YNBMULOE7CXLFO4UEW3Q, seg_pos={0 0}, upload=50198]: Done uploading piece. err: <nil>
Sep 25 22:33:44.766574920 (57h2m30.228667393s) [noise=true , address=145.79.227.17:4972 , piece_id=OLLDZMGRQBTZAQURMSU4AVHOAFCD3VBWCH6Y5WQXENHW2WB3D6BQ, seg_pos={0 0}, upload=50198]: Done uploading piece. err: <nil>
Sep 25 22:33:47.245353345 (57h2m32.707445818s) [noise=true , address=145.79.227.13:4802 , piece_id=UNCMDLJ7BWSS2SOAXZRUYR7EQTKZWOYFHJSVB22DHNH32VADTPNA, seg_pos={0 0}, upload=50198]: Done uploading piece. err: <nil>
Sep 25 22:33:48.389979384 (57h2m33.852071867s) [noise=true , address=145.79.227.19:4079 , piece_id=TSXXBPAF2KFTUP2MDC6NKGLP2PF666PMSXPGYYKDPO25FRF7YQ6Q, seg_pos={0 0}, upload=50198]: Done uploading piece. err: <nil>
Sep 25 22:33:49.870866176 (57h2m35.332958649s) [noise=true , address=145.79.227.3:4151 , piece_id=ZMJ7WBX4DWOWTTJTLJXFZKBR4MRE2RS2FQWRM2YZUC6NHVEAUZFA, seg_pos={0 0}, upload=50198]: Done uploading piece. err: <nil>
Sep 25 22:33:49.927808223 (57h2m35.389900706s) [noise=true , address=145.79.227.2:4115 , piece_id=GDDMOKVGUSPCUNXCYSADTSDEWV3WX6YLR3HSMC72AI7345WMWI6Q, seg_pos={0 0}, upload=50198]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: expected piece hash; context deadline exceeded; EOF
Sep 25 22:33:49.927871833 (57h2m35.389964316s) [noise=true , address=145.79.227.6:4259 , piece_id=VFXMPOPGL2UF6D6NLM7PW3GKM5YEQHT3ZDMZGOR5VEX3NNLOAXAQ, seg_pos={0 0}, upload=50198]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: expected piece hash; context deadline exceeded; EOF
Sep 25 22:33:49.927907451 (57h2m35.389999924s) [noise=true , address=145.79.227.4:4185 , piece_id=UDAN2TY4RX3ATTJAYBL5G3ZXKLYIXFIBPYJWRIG2MPD3LJH62UGA, seg_pos={0 0}, upload=50198]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:145.79.227.4:4185): protocol: write tcp 188.166.225.170:46508->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:46508->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46508->145.79.227.4:4185: use of closed network connection
Sep 25 22:33:49.927937458 (57h2m35.390029931s) [noise=true , address=145.79.227.6:4257 , piece_id=NLAZ3I5A6RU3PWBBSMICAETTWZ7UZLO6X5VEXUZPZ2ZB4KAI6I5A, seg_pos={0 0}, upload=50198]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: expected piece hash; context deadline exceeded; EOF
Sep 25 22:33:49.927971413 (57h2m35.390063886s) [seg_pos={0 0}, upload=50198]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Sep 25 22:33:50.167900726 (57h2m35.629993199s) [upload=50198]: Done uploading segments...
Sep 25 22:33:50.167951733 (57h2m35.630044216s) [upload=50198]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 6.098749611s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
2 out of 60 is failed (threshold=3) [........XX..................................................] duration:
4.919538056s
FAIL (2025-09-25 22:38:02 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 12)
======LOGS======
9 , piece_id=4YT7BXXDQVTR2QL4YBWVMO3YJCZQ57PTTEQW7GCZELWC42HQUDQQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.227500377 (56h31m29.68959285s) [noise=true , address=81.171.14.70:4034 , piece_id=EXVA3SFFPD3GJDYIFHAOP7GMJQHGZECMX5R6MHLRDMMCU7LOA7KA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.253492519 (56h31m29.715584993s) [noise=true , address=81.171.14.70:4028 , piece_id=Y237IBPHRRI7NMOYJG65AZI3LCCZ6JUJPX5WWGIDX5RYHCNVIG3A, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.267890396 (56h31m29.729982869s) [noise=true , address=89.149.201.147:4010 , piece_id=ANMXRI22IXAZFHKB7ZHFGEERKT2KGUYQEFKEOJNJDC5TFB6B3STQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.280031865 (56h31m29.742124348s) [noise=true , address=89.149.201.147:4020 , piece_id=RML2JG4QZSS6H5GTP7BW3B4SFVZLLHHAOUS3LDCOT2BGYTC3HACA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.294783955 (56h31m29.756876428s) [noise=true , address=89.149.201.147:4017 , piece_id=P576E6UE5UADQZYKR4XN4I5X4MCGMXT3B7PBGVNWBZ6I67ZVCJHA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.462969447 (56h31m29.92506192s) [noise=true , address=81.171.14.70:4035 , piece_id=GWVM3MZRGRVJ5RWG227JJXTULF5MUJ4FHWCNHLW7J3GDYXRSXRXA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.668506081 (56h31m30.130598554s) [noise=true , address=81.171.14.70:4045 , piece_id=BSKEEYML4LQZ7PWB45HQNX2YFJTFIEKIILDUZIWCXMVZZGGKHGUQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.702001372 (56h31m30.164093845s) [noise=true , address=89.149.201.147:4014 , piece_id=3VH5QXDWYX2NR3LODHSATJCP5ABF7BI52GJAR3VLPW42CPJJXQNA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.728242188 (56h31m30.190334661s) [noise=true , address=89.149.201.147:4016 , piece_id=PT5TUJDTS2TDBNLH5722GJTZ5BIO4LZKEDKBGUKHEF37TPADW7MA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.797046563 (56h31m30.259139036s) [noise=true , address=89.149.201.147:4012 , piece_id=TY6OVHC4ITI2E32GE7LP3O2YQ46FKWF6A6IIBN4JFAB7WPYXXJ2A, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:03:42.295273049 (56h32m27.757365543s) [noise=true , address=185.167.204.206:4056 , piece_id=4XAHI4JY2MPISEOZ6VJ5GCJ5WZJKHRXIDK2EQGEN7VSYNRYJH24A, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1uSTuNA28oH2s5Rkv2jaKoC2nZWSgpMJXyxwFTCKbr44EvEXrC, address:185.167.204.206:4056): protocol: rpc: dial tcp 185.167.204.206:4056: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295296423 (56h32m27.757388906s) [noise=true , address=185.167.204.205:4008 , piece_id=NXU5IGU2NUZPVR5KIUKLZ7QTPYLGDIKZQNMPBJRPCVYEVNIV7HMA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12ZHTTaGQUTftxEULFqWbH6w1HnWA7PavHrXnvMCD947FxtmsrS, address:185.167.204.205:4008): protocol: rpc: dial tcp 185.167.204.205:4008: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295427543 (56h32m27.757520026s) [noise=true , address=185.167.204.206:4064 , piece_id=7GVKAUOISAORFKOXUNOJMEEFNNUVVZUQ2RGCD6BNVCDTH6ONPQIQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1ZB3emcQntFxpr7zBiE98eesnjoTnNKinw3FPtGU2TgznGakeb, address:185.167.204.206:4064): protocol: rpc: dial tcp 185.167.204.206:4064: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295495543 (56h32m27.757588026s) [noise=true , address=185.167.204.206:4059 , piece_id=G4B2XQYXOG2LWBAIYBM4BWLI7PEPE2XDM5A57YHYR6SOWW7ODSVA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:121gTedy8TPx44wmLqdQpZ1gDDsL8D9K1EvQJffR7oKY3bT9DXh, address:185.167.204.206:4059): protocol: rpc: dial tcp 185.167.204.206:4059: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295562701 (56h32m27.757655184s) [noise=true , address=185.167.204.205:4030 , piece_id=U6SOG6BNEIWDGYWDRZOCLTROZVZXPPG5ZI52HLRPKTDKFVQN3EJA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12w5R2Whxgg4AYNNvTkKmFq4pzahF1uZiVzwgVygc9iZDAXMdSk, address:185.167.204.205:4030): protocol: rpc: dial tcp 185.167.204.205:4030: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295700453 (56h32m27.757792926s) [noise=true , address=185.167.204.205:4010 , piece_id=DPMSKQDDOU2PUXHINTG7JJXLKXJE2PC6B3KNSKN5UIGAKBM3GZVQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12oTu1GaRsH7fHqxLkQ9c9ikjG4zRN3NChssa5gfpLwrcEehoyV, address:185.167.204.205:4010): protocol: rpc: dial tcp 185.167.204.205:4010: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295713257 (56h32m27.75780574s) [noise=true , address=185.167.204.206:4053 , piece_id=74G7LWVYE7GO4GPSG67BD6QVE7UBLGIAKLSG447PWWLNBM2FZNTA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12SrFttJpndpv7TKJJYsKthWnGWBYQGHpS9CwqTd4mqBP9NCCm8, address:185.167.204.206:4053): protocol: rpc: dial tcp 185.167.204.206:4053: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295852051 (56h32m27.757944534s) [noise=true , address=185.167.204.206:4065 , piece_id=VKBLX4ASIKVHMUUCTD7GFFPEHJLKREGCDGI7LCVRSASDHBAWI6AQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12oPo26KZGZr7o7M18KZiX5PMAAMwHK8FPT59y1hSLhkB6z3Lcj, address:185.167.204.206:4065): protocol: rpc: dial tcp 185.167.204.206:4065: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295908840 (56h32m27.758001313s) [noise=true , address=185.167.204.206:4057 , piece_id=WNX7U6DN7HTE3XLVJAJ4FTCYFR3B2MLMLF5I46GALE463DWQ22WQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1xy38NywWy5CSJ7CfupY78vbkHg6cQzgNhM2Z1xnpvw9dwvMoT, address:185.167.204.206:4057): protocol: rpc: dial tcp 185.167.204.206:4057: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296027916 (56h32m27.758120399s) [noise=true , address=185.167.204.205:4003 , piece_id=NQX2I3KTMYFLS2B2C5Y2PGZRXCD7QJCZLABY4PTYVEFGAVCYX2EQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1CjWvvJ4FdZc41sRgsQFWz4Y1ATZKgzsvDcUTmBSrCC15ekCmF, address:185.167.204.205:4003): protocol: rpc: dial tcp 185.167.204.205:4003: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296100945 (56h32m27.758193418s) [noise=true , address=185.167.204.205:4032 , piece_id=AN4YMHK6NKM5RIWAZHX2FPBGQ3VRFRYBKHBPIAL2HQWBK5XXNNOA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12558dx7zMNDfabV55vSsP9XwdxXgoPM8aHW53femVZzkCRxsKN, address:185.167.204.205:4032): protocol: rpc: dial tcp 185.167.204.205:4032: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296150660 (56h32m27.758243133s) [noise=true , address=185.167.204.205:4029 , piece_id=UVEMQY4WGBKGGT75ILTBPUCFZ4U252QGPDTBPT2BNMOR3V5XYCPA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12DyF12w2JXdkGLfLx9VE1E5f28VnMg7M2b3osfsDtGX9o8Tvsn, address:185.167.204.205:4029): protocol: write tcp 188.166.225.170:36604->185.167.204.205:4029: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296276239 (56h32m27.758368722s) [noise=true , address=185.167.204.206:4069 , piece_id=VKW6XTWPZPD3MDANS3FIYYD2UPDW47WBCGLNTOVYVTWA5IT23DBQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12Uan1TT5oAM3axhaviVFbvh83gThiZvwHMwbCD31ZZens6BQ4U, address:185.167.204.206:4069): protocol: rpc: dial tcp 185.167.204.206:4069: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296438448 (56h32m27.758530931s) [noise=true , address=185.167.204.206:4048 , piece_id=PRBNW47DU6K26GYUUUZSYTYQO3GNFC6DEV2OYQDP4R2RV5L5VNXA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1sY4wHBkSVUDWWg8T8Vm1hpRbbpb3ttqnqDVY8uWyG27gqBwT9, address:185.167.204.206:4048): protocol: write tcp 188.166.225.170:59140->185.167.204.206:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296628780 (56h32m27.758721253s) [noise=true , address=185.167.204.206:4055 , piece_id=KH2E6T37MHNJJ56CR3EIKMEUEWMAVP2XNMZ7K5QVKCFZJIVXRTUA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12CYRieezm1mjtGchWdPrB2mNaHVRvTS85umjB6cQyYKnvisXSB, address:185.167.204.206:4055): protocol: rpc: dial tcp 185.167.204.206:4055: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.297155593 (56h32m27.759248076s) [noise=true , address=185.167.204.205:4022 , piece_id=VU4625JC4A4ZVXFGJZTDMDZXYCNXJ67OQTEKWRLMMHGHPEHY5J7A, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj, address:185.167.204.205:4022): protocol: write tcp 188.166.225.170:57938->185.167.204.205:4022: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.297184969 (56h32m27.759277442s) [seg_pos={0 0}, upload=49746]: Done waiting for segment. successful: 12 optimal: 26 errs: <nil>
Sep 25 22:03:42.465979471 (56h32m27.928071944s) [upload=49746]: Done uploading segments...
Sep 25 22:03:42.466010200 (56h32m27.928102673s) [upload=49746]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 12)
Duration: 4.919578012s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.eu1.gateway)antiblip:
1 out of 60 is failed (threshold=3) [..........................................X.................] duration:
175.264912ms remote:
"tcp:207.211.208.129:443" response headers:
Accept-Ranges=bytes Content-Length=352 Content-Security-Policy=block-all-mixed-content Server=Storj X-Request-Id=JdrtmpHLrtw X-Xss-Protection=1; mode=block Content-Type=application/xml Vary=Origin X-Amz-Request-Id=1868A6DD9BAF33BB Date=Thu, 25 Sep 2025 22:25:25 GMT target:
https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318
FAIL (2025-09-25 22:37:38 UTC)
orbiter warning: unexpected status code "500 Internal Server Error" "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>InternalError</Code><Message>We encountered an internal error, please try again.: cause(uplink: metaclient: context canceled)</Message><Key>hello-world.txt</Key><BucketName>uptime-checks</BucketName><Resource>/hello-world.txt</Resource><RequestId>1868A6DD9BAF33BB</RequestId><HostId></HostId></Error>"
Duration: 175.294477ms
edge
gateway
vhost
bgp
eu1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......................................................X....] duration:
6.034881033s
FAIL (2025-09-25 22:37:25 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Sep 25 22:33:20.609515379 (57h2m6.071607862s) [upload=50198]: Starting upload
Sep 25 22:33:20.609634665 (57h2m6.071727138s) [upload=50198]: Done starting upload
Sep 25 22:33:20.609747009 (57h2m6.071839492s) [upload=50198]: Uploading segments...
Sep 25 22:33:20.609932132 (57h2m6.072024605s) [upload=50198]: Got next segment. Inline: false
Sep 25 22:33:20.609977649 (57h2m6.072070132s) [upload=50198]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 25 22:33:20.609983990 (57h2m6.072076473s) [upload=50198]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 25 22:33:20.876293406 (57h2m6.338385889s) [seg_pos={0 0}, upload=50198]: Begin upload segment...
Sep 25 22:33:20.876365564 (57h2m6.338458047s) [seg_pos={0 0}, upload=50198]: Done begin upload segment.
Sep 25 22:33:20.876398446 (57h2m6.338490919s) [noise=true , address=45.91.116.7:4010 , piece_id=QZEN3HHQVP5SQ3ABXX6AUG2T7V2JTJYLKXPEM2FCIWEIRSZ2IOUA, seg_pos={0 0}, upload=50198]: Uploading piece...
Sep 25 22:33:20.876412874 (57h2m6.338505347s) [noise=true , address=145.79.227.11:4729 , piece_id=WTDDZ72XM433FAERK56RR7BWIFEFFMKJ7PKKIQQUCLWZTTTTB5TA, seg_pos={0 0}, upload=50198]: Uploading piece...
Sep 25 22:33:20.876460024 (57h2m6.338552507s) [noise=true , address=145.79.227.2:4115 , piece_id=GDDMOKVGUSPCUNXCYSADTSDEWV3WX6YLR3HSMC72AI7345WMWI6Q, seg_pos={0 0}, upload=50198]: Uploading piece...
Sep 25 22:33:20.876711823 (57h2m6.338804296s) [noise=true , address=145.79.227.4:4185 , piece_id=UDAN2TY4RX3ATTJAYBL5G3ZXKLYIXFIBPYJWRIG2MPD3LJH62UGA, seg_pos={0 0}, upload=50198]: Uploading piece...
Sep 25 22:33:20.877150188 (57h2m6.339242661s) [noise=true , address=145.79.227.6:4257 , piece_id=NLAZ3I5A6RU3PWBBSMICAETTWZ7UZLO6X5VEXUZPZ2ZB4KAI6I5A, seg_pos={0 0}, upload=50198]: Uploading piece...
Sep 25 22:33:20.877621355 (57h2m6.339713838s) [noise=true , address=145.79.227.17:4972 , piece_id=OLLDZMGRQBTZAQURMSU4AVHOAFCD3VBWCH6Y5WQXENHW2WB3D6BQ, seg_pos={0 0}, upload=50198]: Uploading piece...
Sep 25 22:33:20.877942707 (57h2m6.340035181s) [noise=true , address=145.79.227.18:4007 , piece_id=OWW73ZTC3OLJTGC4IWXZ44VT4ERGUE4YGGULZCSJKYEHNXRNLLRA, seg_pos={0 0}, upload=50198]: Uploading piece...
Sep 25 22:33:20.878252356 (57h2m6.340344829s) [noise=true , address=45.91.116.4:4939 , piece_id=7UY722GYY34MQHO6FCVNCMXKV67RUX2TTNXGUXGGDEBE63HPI6DQ, seg_pos={0 0}, upload=50198]: Uploading piece...
Sep 25 22:33:20.878573619 (57h2m6.340666092s) [noise=true , address=145.79.227.19:4079 , piece_id=TSXXBPAF2KFTUP2MDC6NKGLP2PF666PMSXPGYYKDPO25FRF7YQ6Q, seg_pos={0 0}, upload=50198]: Uploading piece...
Sep 25 22:33:20.881006501 (57h2m6.343098975s) [noise=true , address=45.91.116.10:4082 , piece_id=OJUG2YVVDOHECUD3DLDJANOWGXKS76BD3KIWFF5Q2OTRELONDWUQ, seg_pos={0 0}, upload=50198]: Uploading piece...
Sep 25 22:33:20.882423098 (57h2m6.344515571s) [noise=true , address=145.79.227.13:4802 , piece_id=UNCMDLJ7BWSS2SOAXZRUYR7EQTKZWOYFHJSVB22DHNH32VADTPNA, seg_pos={0 0}, upload=50198]: Uploading piece...
Sep 25 22:33:20.882592721 (57h2m6.344685194s) [noise=true , address=145.79.227.6:4259 , piece_id=VFXMPOPGL2UF6D6NLM7PW3GKM5YEQHT3ZDMZGOR5VEX3NNLOAXAQ, seg_pos={0 0}, upload=50198]: Uploading piece...
Sep 25 22:33:20.885196980 (57h2m6.347289464s) [noise=true , address=145.79.227.3:4151 , piece_id=ZMJ7WBX4DWOWTTJTLJXFZKBR4MRE2RS2FQWRM2YZUC6NHVEAUZFA, seg_pos={0 0}, upload=50198]: Uploading piece...
Sep 25 22:33:20.888905401 (57h2m6.350997874s) [noise=true , address=145.79.227.15:4874 , piece_id=I5VJYG36ZG63ZRJUC2WKXISA5C7EJQSV3XZJ7FLN5MWMC3CDBNEA, seg_pos={0 0}, upload=50198]: Uploading piece...
Sep 25 22:33:20.891203167 (57h2m6.35329565s) [noise=true , address=77.90.31.15:4697 , piece_id=RRP4PCTAVSJYH377LRZ43I36D4LWG7EPHE23LDBUZLP2DUR3RNDQ, seg_pos={0 0}, upload=50198]: Uploading piece...
Sep 25 22:33:20.892678096 (57h2m6.354770569s) [noise=true , address=145.79.227.15:4873 , piece_id=C3JJ5HPO237IQRGLKQ5QZSKEJM2HNOH4YNBMULOE7CXLFO4UEW3Q, seg_pos={0 0}, upload=50198]: Uploading piece...
Sep 25 22:33:22.883631748 (57h2m8.345724231s) [upload=50198]: Next returned nil segment
Sep 25 22:33:22.883655924 (57h2m8.345748397s) [upload=50198]: Waiting for error group managing segments...
Sep 25 22:33:23.186669559 (57h2m8.648762032s) [noise=true , address=77.90.31.15:4697 , piece_id=RRP4PCTAVSJYH377LRZ43I36D4LWG7EPHE23LDBUZLP2DUR3RNDQ, seg_pos={0 0}, upload=50198]: Done uploading piece. err: <nil>
Sep 25 22:33:23.402505165 (57h2m8.864597638s) [noise=true , address=45.91.116.7:4010 , piece_id=QZEN3HHQVP5SQ3ABXX6AUG2T7V2JTJYLKXPEM2FCIWEIRSZ2IOUA, seg_pos={0 0}, upload=50198]: Done uploading piece. err: <nil>
Sep 25 22:33:23.460847597 (57h2m8.92294007s) [noise=true , address=45.91.116.4:4939 , piece_id=7UY722GYY34MQHO6FCVNCMXKV67RUX2TTNXGUXGGDEBE63HPI6DQ, seg_pos={0 0}, upload=50198]: Done uploading piece. err: <nil>
Sep 25 22:33:23.475828172 (57h2m8.937920645s) [noise=true , address=45.91.116.10:4082 , piece_id=OJUG2YVVDOHECUD3DLDJANOWGXKS76BD3KIWFF5Q2OTRELONDWUQ, seg_pos={0 0}, upload=50198]: Done uploading piece. err: <nil>
Sep 25 22:33:41.316700325 (57h2m26.778792808s) [noise=true , address=145.79.227.18:4007 , piece_id=OWW73ZTC3OLJTGC4IWXZ44VT4ERGUE4YGGULZCSJKYEHNXRNLLRA, seg_pos={0 0}, upload=50198]: Done uploading piece. err: <nil>
Sep 25 22:33:42.134617358 (57h2m27.596709841s) [noise=true , address=145.79.227.11:4729 , piece_id=WTDDZ72XM433FAERK56RR7BWIFEFFMKJ7PKKIQQUCLWZTTTTB5TA, seg_pos={0 0}, upload=50198]: Done uploading piece. err: <nil>
Sep 25 22:33:43.316084677 (57h2m28.77817716s) [noise=true , address=145.79.227.15:4874 , piece_id=I5VJYG36ZG63ZRJUC2WKXISA5C7EJQSV3XZJ7FLN5MWMC3CDBNEA, seg_pos={0 0}, upload=50198]: Done uploading piece. err: <nil>
Sep 25 22:33:43.336895665 (57h2m28.798988148s) [noise=true , address=145.79.227.15:4873 , piece_id=C3JJ5HPO237IQRGLKQ5QZSKEJM2HNOH4YNBMULOE7CXLFO4UEW3Q, seg_pos={0 0}, upload=50198]: Done uploading piece. err: <nil>
Sep 25 22:33:44.766574920 (57h2m30.228667393s) [noise=true , address=145.79.227.17:4972 , piece_id=OLLDZMGRQBTZAQURMSU4AVHOAFCD3VBWCH6Y5WQXENHW2WB3D6BQ, seg_pos={0 0}, upload=50198]: Done uploading piece. err: <nil>
Sep 25 22:33:47.245353345 (57h2m32.707445818s) [noise=true , address=145.79.227.13:4802 , piece_id=UNCMDLJ7BWSS2SOAXZRUYR7EQTKZWOYFHJSVB22DHNH32VADTPNA, seg_pos={0 0}, upload=50198]: Done uploading piece. err: <nil>
Sep 25 22:33:48.389979384 (57h2m33.852071867s) [noise=true , address=145.79.227.19:4079 , piece_id=TSXXBPAF2KFTUP2MDC6NKGLP2PF666PMSXPGYYKDPO25FRF7YQ6Q, seg_pos={0 0}, upload=50198]: Done uploading piece. err: <nil>
Sep 25 22:33:49.870866176 (57h2m35.332958649s) [noise=true , address=145.79.227.3:4151 , piece_id=ZMJ7WBX4DWOWTTJTLJXFZKBR4MRE2RS2FQWRM2YZUC6NHVEAUZFA, seg_pos={0 0}, upload=50198]: Done uploading piece. err: <nil>
Sep 25 22:33:49.927808223 (57h2m35.389900706s) [noise=true , address=145.79.227.2:4115 , piece_id=GDDMOKVGUSPCUNXCYSADTSDEWV3WX6YLR3HSMC72AI7345WMWI6Q, seg_pos={0 0}, upload=50198]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: expected piece hash; context deadline exceeded; EOF
Sep 25 22:33:49.927871833 (57h2m35.389964316s) [noise=true , address=145.79.227.6:4259 , piece_id=VFXMPOPGL2UF6D6NLM7PW3GKM5YEQHT3ZDMZGOR5VEX3NNLOAXAQ, seg_pos={0 0}, upload=50198]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: expected piece hash; context deadline exceeded; EOF
Sep 25 22:33:49.927907451 (57h2m35.389999924s) [noise=true , address=145.79.227.4:4185 , piece_id=UDAN2TY4RX3ATTJAYBL5G3ZXKLYIXFIBPYJWRIG2MPD3LJH62UGA, seg_pos={0 0}, upload=50198]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:145.79.227.4:4185): protocol: write tcp 188.166.225.170:46508->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:46508->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46508->145.79.227.4:4185: use of closed network connection
Sep 25 22:33:49.927937458 (57h2m35.390029931s) [noise=true , address=145.79.227.6:4257 , piece_id=NLAZ3I5A6RU3PWBBSMICAETTWZ7UZLO6X5VEXUZPZ2ZB4KAI6I5A, seg_pos={0 0}, upload=50198]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: expected piece hash; context deadline exceeded; EOF
Sep 25 22:33:49.927971413 (57h2m35.390063886s) [seg_pos={0 0}, upload=50198]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Sep 25 22:33:50.167900726 (57h2m35.629993199s) [upload=50198]: Done uploading segments...
Sep 25 22:33:50.167951733 (57h2m35.630044216s) [upload=50198]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 6.034915949s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
2 out of 60 is failed (threshold=3) [.........XX.................................................] duration:
6.385568394s
FAIL (2025-09-25 22:37:11 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 12)
======LOGS======
9 , piece_id=4YT7BXXDQVTR2QL4YBWVMO3YJCZQ57PTTEQW7GCZELWC42HQUDQQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.227500377 (56h31m29.68959285s) [noise=true , address=81.171.14.70:4034 , piece_id=EXVA3SFFPD3GJDYIFHAOP7GMJQHGZECMX5R6MHLRDMMCU7LOA7KA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.253492519 (56h31m29.715584993s) [noise=true , address=81.171.14.70:4028 , piece_id=Y237IBPHRRI7NMOYJG65AZI3LCCZ6JUJPX5WWGIDX5RYHCNVIG3A, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.267890396 (56h31m29.729982869s) [noise=true , address=89.149.201.147:4010 , piece_id=ANMXRI22IXAZFHKB7ZHFGEERKT2KGUYQEFKEOJNJDC5TFB6B3STQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.280031865 (56h31m29.742124348s) [noise=true , address=89.149.201.147:4020 , piece_id=RML2JG4QZSS6H5GTP7BW3B4SFVZLLHHAOUS3LDCOT2BGYTC3HACA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.294783955 (56h31m29.756876428s) [noise=true , address=89.149.201.147:4017 , piece_id=P576E6UE5UADQZYKR4XN4I5X4MCGMXT3B7PBGVNWBZ6I67ZVCJHA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.462969447 (56h31m29.92506192s) [noise=true , address=81.171.14.70:4035 , piece_id=GWVM3MZRGRVJ5RWG227JJXTULF5MUJ4FHWCNHLW7J3GDYXRSXRXA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.668506081 (56h31m30.130598554s) [noise=true , address=81.171.14.70:4045 , piece_id=BSKEEYML4LQZ7PWB45HQNX2YFJTFIEKIILDUZIWCXMVZZGGKHGUQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.702001372 (56h31m30.164093845s) [noise=true , address=89.149.201.147:4014 , piece_id=3VH5QXDWYX2NR3LODHSATJCP5ABF7BI52GJAR3VLPW42CPJJXQNA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.728242188 (56h31m30.190334661s) [noise=true , address=89.149.201.147:4016 , piece_id=PT5TUJDTS2TDBNLH5722GJTZ5BIO4LZKEDKBGUKHEF37TPADW7MA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.797046563 (56h31m30.259139036s) [noise=true , address=89.149.201.147:4012 , piece_id=TY6OVHC4ITI2E32GE7LP3O2YQ46FKWF6A6IIBN4JFAB7WPYXXJ2A, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:03:42.295273049 (56h32m27.757365543s) [noise=true , address=185.167.204.206:4056 , piece_id=4XAHI4JY2MPISEOZ6VJ5GCJ5WZJKHRXIDK2EQGEN7VSYNRYJH24A, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1uSTuNA28oH2s5Rkv2jaKoC2nZWSgpMJXyxwFTCKbr44EvEXrC, address:185.167.204.206:4056): protocol: rpc: dial tcp 185.167.204.206:4056: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295296423 (56h32m27.757388906s) [noise=true , address=185.167.204.205:4008 , piece_id=NXU5IGU2NUZPVR5KIUKLZ7QTPYLGDIKZQNMPBJRPCVYEVNIV7HMA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12ZHTTaGQUTftxEULFqWbH6w1HnWA7PavHrXnvMCD947FxtmsrS, address:185.167.204.205:4008): protocol: rpc: dial tcp 185.167.204.205:4008: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295427543 (56h32m27.757520026s) [noise=true , address=185.167.204.206:4064 , piece_id=7GVKAUOISAORFKOXUNOJMEEFNNUVVZUQ2RGCD6BNVCDTH6ONPQIQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1ZB3emcQntFxpr7zBiE98eesnjoTnNKinw3FPtGU2TgznGakeb, address:185.167.204.206:4064): protocol: rpc: dial tcp 185.167.204.206:4064: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295495543 (56h32m27.757588026s) [noise=true , address=185.167.204.206:4059 , piece_id=G4B2XQYXOG2LWBAIYBM4BWLI7PEPE2XDM5A57YHYR6SOWW7ODSVA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:121gTedy8TPx44wmLqdQpZ1gDDsL8D9K1EvQJffR7oKY3bT9DXh, address:185.167.204.206:4059): protocol: rpc: dial tcp 185.167.204.206:4059: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295562701 (56h32m27.757655184s) [noise=true , address=185.167.204.205:4030 , piece_id=U6SOG6BNEIWDGYWDRZOCLTROZVZXPPG5ZI52HLRPKTDKFVQN3EJA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12w5R2Whxgg4AYNNvTkKmFq4pzahF1uZiVzwgVygc9iZDAXMdSk, address:185.167.204.205:4030): protocol: rpc: dial tcp 185.167.204.205:4030: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295700453 (56h32m27.757792926s) [noise=true , address=185.167.204.205:4010 , piece_id=DPMSKQDDOU2PUXHINTG7JJXLKXJE2PC6B3KNSKN5UIGAKBM3GZVQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12oTu1GaRsH7fHqxLkQ9c9ikjG4zRN3NChssa5gfpLwrcEehoyV, address:185.167.204.205:4010): protocol: rpc: dial tcp 185.167.204.205:4010: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295713257 (56h32m27.75780574s) [noise=true , address=185.167.204.206:4053 , piece_id=74G7LWVYE7GO4GPSG67BD6QVE7UBLGIAKLSG447PWWLNBM2FZNTA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12SrFttJpndpv7TKJJYsKthWnGWBYQGHpS9CwqTd4mqBP9NCCm8, address:185.167.204.206:4053): protocol: rpc: dial tcp 185.167.204.206:4053: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295852051 (56h32m27.757944534s) [noise=true , address=185.167.204.206:4065 , piece_id=VKBLX4ASIKVHMUUCTD7GFFPEHJLKREGCDGI7LCVRSASDHBAWI6AQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12oPo26KZGZr7o7M18KZiX5PMAAMwHK8FPT59y1hSLhkB6z3Lcj, address:185.167.204.206:4065): protocol: rpc: dial tcp 185.167.204.206:4065: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295908840 (56h32m27.758001313s) [noise=true , address=185.167.204.206:4057 , piece_id=WNX7U6DN7HTE3XLVJAJ4FTCYFR3B2MLMLF5I46GALE463DWQ22WQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1xy38NywWy5CSJ7CfupY78vbkHg6cQzgNhM2Z1xnpvw9dwvMoT, address:185.167.204.206:4057): protocol: rpc: dial tcp 185.167.204.206:4057: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296027916 (56h32m27.758120399s) [noise=true , address=185.167.204.205:4003 , piece_id=NQX2I3KTMYFLS2B2C5Y2PGZRXCD7QJCZLABY4PTYVEFGAVCYX2EQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1CjWvvJ4FdZc41sRgsQFWz4Y1ATZKgzsvDcUTmBSrCC15ekCmF, address:185.167.204.205:4003): protocol: rpc: dial tcp 185.167.204.205:4003: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296100945 (56h32m27.758193418s) [noise=true , address=185.167.204.205:4032 , piece_id=AN4YMHK6NKM5RIWAZHX2FPBGQ3VRFRYBKHBPIAL2HQWBK5XXNNOA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12558dx7zMNDfabV55vSsP9XwdxXgoPM8aHW53femVZzkCRxsKN, address:185.167.204.205:4032): protocol: rpc: dial tcp 185.167.204.205:4032: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296150660 (56h32m27.758243133s) [noise=true , address=185.167.204.205:4029 , piece_id=UVEMQY4WGBKGGT75ILTBPUCFZ4U252QGPDTBPT2BNMOR3V5XYCPA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12DyF12w2JXdkGLfLx9VE1E5f28VnMg7M2b3osfsDtGX9o8Tvsn, address:185.167.204.205:4029): protocol: write tcp 188.166.225.170:36604->185.167.204.205:4029: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296276239 (56h32m27.758368722s) [noise=true , address=185.167.204.206:4069 , piece_id=VKW6XTWPZPD3MDANS3FIYYD2UPDW47WBCGLNTOVYVTWA5IT23DBQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12Uan1TT5oAM3axhaviVFbvh83gThiZvwHMwbCD31ZZens6BQ4U, address:185.167.204.206:4069): protocol: rpc: dial tcp 185.167.204.206:4069: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296438448 (56h32m27.758530931s) [noise=true , address=185.167.204.206:4048 , piece_id=PRBNW47DU6K26GYUUUZSYTYQO3GNFC6DEV2OYQDP4R2RV5L5VNXA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1sY4wHBkSVUDWWg8T8Vm1hpRbbpb3ttqnqDVY8uWyG27gqBwT9, address:185.167.204.206:4048): protocol: write tcp 188.166.225.170:59140->185.167.204.206:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296628780 (56h32m27.758721253s) [noise=true , address=185.167.204.206:4055 , piece_id=KH2E6T37MHNJJ56CR3EIKMEUEWMAVP2XNMZ7K5QVKCFZJIVXRTUA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12CYRieezm1mjtGchWdPrB2mNaHVRvTS85umjB6cQyYKnvisXSB, address:185.167.204.206:4055): protocol: rpc: dial tcp 185.167.204.206:4055: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.297155593 (56h32m27.759248076s) [noise=true , address=185.167.204.205:4022 , piece_id=VU4625JC4A4ZVXFGJZTDMDZXYCNXJ67OQTEKWRLMMHGHPEHY5J7A, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj, address:185.167.204.205:4022): protocol: write tcp 188.166.225.170:57938->185.167.204.205:4022: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.297184969 (56h32m27.759277442s) [seg_pos={0 0}, upload=49746]: Done waiting for segment. successful: 12 optimal: 26 errs: <nil>
Sep 25 22:03:42.465979471 (56h32m27.928071944s) [upload=49746]: Done uploading segments...
Sep 25 22:03:42.466010200 (56h32m27.928102673s) [upload=49746]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 12)
Duration: 6.38560347s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.eu1.gateway)antiblip:
1 out of 60 is failed (threshold=3) [...........................................X................] duration:
267.71835ms remote:
"tcp:207.211.208.129:443" response headers:
Accept-Ranges=bytes Content-Length=352 Content-Security-Policy=block-all-mixed-content Server=Storj X-Request-Id=JdrtmpHLrtw X-Xss-Protection=1; mode=block Content-Type=application/xml Vary=Origin X-Amz-Request-Id=1868A6DD9BAF33BB Date=Thu, 25 Sep 2025 22:25:25 GMT target:
https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318
FAIL (2025-09-25 22:37:05 UTC)
orbiter warning: unexpected status code "500 Internal Server Error" "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>InternalError</Code><Message>We encountered an internal error, please try again.: cause(uplink: metaclient: context canceled)</Message><Key>hello-world.txt</Key><BucketName>uptime-checks</BucketName><Resource>/hello-world.txt</Resource><RequestId>1868A6DD9BAF33BB</RequestId><HostId></HostId></Error>"
Duration: 267.748197ms
edge
gateway
vhost
bgp
eu1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [........................................................X...] duration:
5.953381175s
FAIL (2025-09-25 22:36:24 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Sep 25 22:33:20.609515379 (57h2m6.071607862s) [upload=50198]: Starting upload
Sep 25 22:33:20.609634665 (57h2m6.071727138s) [upload=50198]: Done starting upload
Sep 25 22:33:20.609747009 (57h2m6.071839492s) [upload=50198]: Uploading segments...
Sep 25 22:33:20.609932132 (57h2m6.072024605s) [upload=50198]: Got next segment. Inline: false
Sep 25 22:33:20.609977649 (57h2m6.072070132s) [upload=50198]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 25 22:33:20.609983990 (57h2m6.072076473s) [upload=50198]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 25 22:33:20.876293406 (57h2m6.338385889s) [seg_pos={0 0}, upload=50198]: Begin upload segment...
Sep 25 22:33:20.876365564 (57h2m6.338458047s) [seg_pos={0 0}, upload=50198]: Done begin upload segment.
Sep 25 22:33:20.876398446 (57h2m6.338490919s) [noise=true , address=45.91.116.7:4010 , piece_id=QZEN3HHQVP5SQ3ABXX6AUG2T7V2JTJYLKXPEM2FCIWEIRSZ2IOUA, seg_pos={0 0}, upload=50198]: Uploading piece...
Sep 25 22:33:20.876412874 (57h2m6.338505347s) [noise=true , address=145.79.227.11:4729 , piece_id=WTDDZ72XM433FAERK56RR7BWIFEFFMKJ7PKKIQQUCLWZTTTTB5TA, seg_pos={0 0}, upload=50198]: Uploading piece...
Sep 25 22:33:20.876460024 (57h2m6.338552507s) [noise=true , address=145.79.227.2:4115 , piece_id=GDDMOKVGUSPCUNXCYSADTSDEWV3WX6YLR3HSMC72AI7345WMWI6Q, seg_pos={0 0}, upload=50198]: Uploading piece...
Sep 25 22:33:20.876711823 (57h2m6.338804296s) [noise=true , address=145.79.227.4:4185 , piece_id=UDAN2TY4RX3ATTJAYBL5G3ZXKLYIXFIBPYJWRIG2MPD3LJH62UGA, seg_pos={0 0}, upload=50198]: Uploading piece...
Sep 25 22:33:20.877150188 (57h2m6.339242661s) [noise=true , address=145.79.227.6:4257 , piece_id=NLAZ3I5A6RU3PWBBSMICAETTWZ7UZLO6X5VEXUZPZ2ZB4KAI6I5A, seg_pos={0 0}, upload=50198]: Uploading piece...
Sep 25 22:33:20.877621355 (57h2m6.339713838s) [noise=true , address=145.79.227.17:4972 , piece_id=OLLDZMGRQBTZAQURMSU4AVHOAFCD3VBWCH6Y5WQXENHW2WB3D6BQ, seg_pos={0 0}, upload=50198]: Uploading piece...
Sep 25 22:33:20.877942707 (57h2m6.340035181s) [noise=true , address=145.79.227.18:4007 , piece_id=OWW73ZTC3OLJTGC4IWXZ44VT4ERGUE4YGGULZCSJKYEHNXRNLLRA, seg_pos={0 0}, upload=50198]: Uploading piece...
Sep 25 22:33:20.878252356 (57h2m6.340344829s) [noise=true , address=45.91.116.4:4939 , piece_id=7UY722GYY34MQHO6FCVNCMXKV67RUX2TTNXGUXGGDEBE63HPI6DQ, seg_pos={0 0}, upload=50198]: Uploading piece...
Sep 25 22:33:20.878573619 (57h2m6.340666092s) [noise=true , address=145.79.227.19:4079 , piece_id=TSXXBPAF2KFTUP2MDC6NKGLP2PF666PMSXPGYYKDPO25FRF7YQ6Q, seg_pos={0 0}, upload=50198]: Uploading piece...
Sep 25 22:33:20.881006501 (57h2m6.343098975s) [noise=true , address=45.91.116.10:4082 , piece_id=OJUG2YVVDOHECUD3DLDJANOWGXKS76BD3KIWFF5Q2OTRELONDWUQ, seg_pos={0 0}, upload=50198]: Uploading piece...
Sep 25 22:33:20.882423098 (57h2m6.344515571s) [noise=true , address=145.79.227.13:4802 , piece_id=UNCMDLJ7BWSS2SOAXZRUYR7EQTKZWOYFHJSVB22DHNH32VADTPNA, seg_pos={0 0}, upload=50198]: Uploading piece...
Sep 25 22:33:20.882592721 (57h2m6.344685194s) [noise=true , address=145.79.227.6:4259 , piece_id=VFXMPOPGL2UF6D6NLM7PW3GKM5YEQHT3ZDMZGOR5VEX3NNLOAXAQ, seg_pos={0 0}, upload=50198]: Uploading piece...
Sep 25 22:33:20.885196980 (57h2m6.347289464s) [noise=true , address=145.79.227.3:4151 , piece_id=ZMJ7WBX4DWOWTTJTLJXFZKBR4MRE2RS2FQWRM2YZUC6NHVEAUZFA, seg_pos={0 0}, upload=50198]: Uploading piece...
Sep 25 22:33:20.888905401 (57h2m6.350997874s) [noise=true , address=145.79.227.15:4874 , piece_id=I5VJYG36ZG63ZRJUC2WKXISA5C7EJQSV3XZJ7FLN5MWMC3CDBNEA, seg_pos={0 0}, upload=50198]: Uploading piece...
Sep 25 22:33:20.891203167 (57h2m6.35329565s) [noise=true , address=77.90.31.15:4697 , piece_id=RRP4PCTAVSJYH377LRZ43I36D4LWG7EPHE23LDBUZLP2DUR3RNDQ, seg_pos={0 0}, upload=50198]: Uploading piece...
Sep 25 22:33:20.892678096 (57h2m6.354770569s) [noise=true , address=145.79.227.15:4873 , piece_id=C3JJ5HPO237IQRGLKQ5QZSKEJM2HNOH4YNBMULOE7CXLFO4UEW3Q, seg_pos={0 0}, upload=50198]: Uploading piece...
Sep 25 22:33:22.883631748 (57h2m8.345724231s) [upload=50198]: Next returned nil segment
Sep 25 22:33:22.883655924 (57h2m8.345748397s) [upload=50198]: Waiting for error group managing segments...
Sep 25 22:33:23.186669559 (57h2m8.648762032s) [noise=true , address=77.90.31.15:4697 , piece_id=RRP4PCTAVSJYH377LRZ43I36D4LWG7EPHE23LDBUZLP2DUR3RNDQ, seg_pos={0 0}, upload=50198]: Done uploading piece. err: <nil>
Sep 25 22:33:23.402505165 (57h2m8.864597638s) [noise=true , address=45.91.116.7:4010 , piece_id=QZEN3HHQVP5SQ3ABXX6AUG2T7V2JTJYLKXPEM2FCIWEIRSZ2IOUA, seg_pos={0 0}, upload=50198]: Done uploading piece. err: <nil>
Sep 25 22:33:23.460847597 (57h2m8.92294007s) [noise=true , address=45.91.116.4:4939 , piece_id=7UY722GYY34MQHO6FCVNCMXKV67RUX2TTNXGUXGGDEBE63HPI6DQ, seg_pos={0 0}, upload=50198]: Done uploading piece. err: <nil>
Sep 25 22:33:23.475828172 (57h2m8.937920645s) [noise=true , address=45.91.116.10:4082 , piece_id=OJUG2YVVDOHECUD3DLDJANOWGXKS76BD3KIWFF5Q2OTRELONDWUQ, seg_pos={0 0}, upload=50198]: Done uploading piece. err: <nil>
Sep 25 22:33:41.316700325 (57h2m26.778792808s) [noise=true , address=145.79.227.18:4007 , piece_id=OWW73ZTC3OLJTGC4IWXZ44VT4ERGUE4YGGULZCSJKYEHNXRNLLRA, seg_pos={0 0}, upload=50198]: Done uploading piece. err: <nil>
Sep 25 22:33:42.134617358 (57h2m27.596709841s) [noise=true , address=145.79.227.11:4729 , piece_id=WTDDZ72XM433FAERK56RR7BWIFEFFMKJ7PKKIQQUCLWZTTTTB5TA, seg_pos={0 0}, upload=50198]: Done uploading piece. err: <nil>
Sep 25 22:33:43.316084677 (57h2m28.77817716s) [noise=true , address=145.79.227.15:4874 , piece_id=I5VJYG36ZG63ZRJUC2WKXISA5C7EJQSV3XZJ7FLN5MWMC3CDBNEA, seg_pos={0 0}, upload=50198]: Done uploading piece. err: <nil>
Sep 25 22:33:43.336895665 (57h2m28.798988148s) [noise=true , address=145.79.227.15:4873 , piece_id=C3JJ5HPO237IQRGLKQ5QZSKEJM2HNOH4YNBMULOE7CXLFO4UEW3Q, seg_pos={0 0}, upload=50198]: Done uploading piece. err: <nil>
Sep 25 22:33:44.766574920 (57h2m30.228667393s) [noise=true , address=145.79.227.17:4972 , piece_id=OLLDZMGRQBTZAQURMSU4AVHOAFCD3VBWCH6Y5WQXENHW2WB3D6BQ, seg_pos={0 0}, upload=50198]: Done uploading piece. err: <nil>
Sep 25 22:33:47.245353345 (57h2m32.707445818s) [noise=true , address=145.79.227.13:4802 , piece_id=UNCMDLJ7BWSS2SOAXZRUYR7EQTKZWOYFHJSVB22DHNH32VADTPNA, seg_pos={0 0}, upload=50198]: Done uploading piece. err: <nil>
Sep 25 22:33:48.389979384 (57h2m33.852071867s) [noise=true , address=145.79.227.19:4079 , piece_id=TSXXBPAF2KFTUP2MDC6NKGLP2PF666PMSXPGYYKDPO25FRF7YQ6Q, seg_pos={0 0}, upload=50198]: Done uploading piece. err: <nil>
Sep 25 22:33:49.870866176 (57h2m35.332958649s) [noise=true , address=145.79.227.3:4151 , piece_id=ZMJ7WBX4DWOWTTJTLJXFZKBR4MRE2RS2FQWRM2YZUC6NHVEAUZFA, seg_pos={0 0}, upload=50198]: Done uploading piece. err: <nil>
Sep 25 22:33:49.927808223 (57h2m35.389900706s) [noise=true , address=145.79.227.2:4115 , piece_id=GDDMOKVGUSPCUNXCYSADTSDEWV3WX6YLR3HSMC72AI7345WMWI6Q, seg_pos={0 0}, upload=50198]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: expected piece hash; context deadline exceeded; EOF
Sep 25 22:33:49.927871833 (57h2m35.389964316s) [noise=true , address=145.79.227.6:4259 , piece_id=VFXMPOPGL2UF6D6NLM7PW3GKM5YEQHT3ZDMZGOR5VEX3NNLOAXAQ, seg_pos={0 0}, upload=50198]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: expected piece hash; context deadline exceeded; EOF
Sep 25 22:33:49.927907451 (57h2m35.389999924s) [noise=true , address=145.79.227.4:4185 , piece_id=UDAN2TY4RX3ATTJAYBL5G3ZXKLYIXFIBPYJWRIG2MPD3LJH62UGA, seg_pos={0 0}, upload=50198]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:145.79.227.4:4185): protocol: write tcp 188.166.225.170:46508->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:46508->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46508->145.79.227.4:4185: use of closed network connection
Sep 25 22:33:49.927937458 (57h2m35.390029931s) [noise=true , address=145.79.227.6:4257 , piece_id=NLAZ3I5A6RU3PWBBSMICAETTWZ7UZLO6X5VEXUZPZ2ZB4KAI6I5A, seg_pos={0 0}, upload=50198]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: expected piece hash; context deadline exceeded; EOF
Sep 25 22:33:49.927971413 (57h2m35.390063886s) [seg_pos={0 0}, upload=50198]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Sep 25 22:33:50.167900726 (57h2m35.629993199s) [upload=50198]: Done uploading segments...
Sep 25 22:33:50.167951733 (57h2m35.630044216s) [upload=50198]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 5.953411002s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
2 out of 60 is failed (threshold=3) [..........XX................................................] duration:
4.853524909s
FAIL (2025-09-25 22:36:16 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 12)
======LOGS======
9 , piece_id=4YT7BXXDQVTR2QL4YBWVMO3YJCZQ57PTTEQW7GCZELWC42HQUDQQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.227500377 (56h31m29.68959285s) [noise=true , address=81.171.14.70:4034 , piece_id=EXVA3SFFPD3GJDYIFHAOP7GMJQHGZECMX5R6MHLRDMMCU7LOA7KA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.253492519 (56h31m29.715584993s) [noise=true , address=81.171.14.70:4028 , piece_id=Y237IBPHRRI7NMOYJG65AZI3LCCZ6JUJPX5WWGIDX5RYHCNVIG3A, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.267890396 (56h31m29.729982869s) [noise=true , address=89.149.201.147:4010 , piece_id=ANMXRI22IXAZFHKB7ZHFGEERKT2KGUYQEFKEOJNJDC5TFB6B3STQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.280031865 (56h31m29.742124348s) [noise=true , address=89.149.201.147:4020 , piece_id=RML2JG4QZSS6H5GTP7BW3B4SFVZLLHHAOUS3LDCOT2BGYTC3HACA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.294783955 (56h31m29.756876428s) [noise=true , address=89.149.201.147:4017 , piece_id=P576E6UE5UADQZYKR4XN4I5X4MCGMXT3B7PBGVNWBZ6I67ZVCJHA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.462969447 (56h31m29.92506192s) [noise=true , address=81.171.14.70:4035 , piece_id=GWVM3MZRGRVJ5RWG227JJXTULF5MUJ4FHWCNHLW7J3GDYXRSXRXA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.668506081 (56h31m30.130598554s) [noise=true , address=81.171.14.70:4045 , piece_id=BSKEEYML4LQZ7PWB45HQNX2YFJTFIEKIILDUZIWCXMVZZGGKHGUQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.702001372 (56h31m30.164093845s) [noise=true , address=89.149.201.147:4014 , piece_id=3VH5QXDWYX2NR3LODHSATJCP5ABF7BI52GJAR3VLPW42CPJJXQNA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.728242188 (56h31m30.190334661s) [noise=true , address=89.149.201.147:4016 , piece_id=PT5TUJDTS2TDBNLH5722GJTZ5BIO4LZKEDKBGUKHEF37TPADW7MA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.797046563 (56h31m30.259139036s) [noise=true , address=89.149.201.147:4012 , piece_id=TY6OVHC4ITI2E32GE7LP3O2YQ46FKWF6A6IIBN4JFAB7WPYXXJ2A, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:03:42.295273049 (56h32m27.757365543s) [noise=true , address=185.167.204.206:4056 , piece_id=4XAHI4JY2MPISEOZ6VJ5GCJ5WZJKHRXIDK2EQGEN7VSYNRYJH24A, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1uSTuNA28oH2s5Rkv2jaKoC2nZWSgpMJXyxwFTCKbr44EvEXrC, address:185.167.204.206:4056): protocol: rpc: dial tcp 185.167.204.206:4056: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295296423 (56h32m27.757388906s) [noise=true , address=185.167.204.205:4008 , piece_id=NXU5IGU2NUZPVR5KIUKLZ7QTPYLGDIKZQNMPBJRPCVYEVNIV7HMA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12ZHTTaGQUTftxEULFqWbH6w1HnWA7PavHrXnvMCD947FxtmsrS, address:185.167.204.205:4008): protocol: rpc: dial tcp 185.167.204.205:4008: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295427543 (56h32m27.757520026s) [noise=true , address=185.167.204.206:4064 , piece_id=7GVKAUOISAORFKOXUNOJMEEFNNUVVZUQ2RGCD6BNVCDTH6ONPQIQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1ZB3emcQntFxpr7zBiE98eesnjoTnNKinw3FPtGU2TgznGakeb, address:185.167.204.206:4064): protocol: rpc: dial tcp 185.167.204.206:4064: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295495543 (56h32m27.757588026s) [noise=true , address=185.167.204.206:4059 , piece_id=G4B2XQYXOG2LWBAIYBM4BWLI7PEPE2XDM5A57YHYR6SOWW7ODSVA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:121gTedy8TPx44wmLqdQpZ1gDDsL8D9K1EvQJffR7oKY3bT9DXh, address:185.167.204.206:4059): protocol: rpc: dial tcp 185.167.204.206:4059: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295562701 (56h32m27.757655184s) [noise=true , address=185.167.204.205:4030 , piece_id=U6SOG6BNEIWDGYWDRZOCLTROZVZXPPG5ZI52HLRPKTDKFVQN3EJA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12w5R2Whxgg4AYNNvTkKmFq4pzahF1uZiVzwgVygc9iZDAXMdSk, address:185.167.204.205:4030): protocol: rpc: dial tcp 185.167.204.205:4030: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295700453 (56h32m27.757792926s) [noise=true , address=185.167.204.205:4010 , piece_id=DPMSKQDDOU2PUXHINTG7JJXLKXJE2PC6B3KNSKN5UIGAKBM3GZVQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12oTu1GaRsH7fHqxLkQ9c9ikjG4zRN3NChssa5gfpLwrcEehoyV, address:185.167.204.205:4010): protocol: rpc: dial tcp 185.167.204.205:4010: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295713257 (56h32m27.75780574s) [noise=true , address=185.167.204.206:4053 , piece_id=74G7LWVYE7GO4GPSG67BD6QVE7UBLGIAKLSG447PWWLNBM2FZNTA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12SrFttJpndpv7TKJJYsKthWnGWBYQGHpS9CwqTd4mqBP9NCCm8, address:185.167.204.206:4053): protocol: rpc: dial tcp 185.167.204.206:4053: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295852051 (56h32m27.757944534s) [noise=true , address=185.167.204.206:4065 , piece_id=VKBLX4ASIKVHMUUCTD7GFFPEHJLKREGCDGI7LCVRSASDHBAWI6AQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12oPo26KZGZr7o7M18KZiX5PMAAMwHK8FPT59y1hSLhkB6z3Lcj, address:185.167.204.206:4065): protocol: rpc: dial tcp 185.167.204.206:4065: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295908840 (56h32m27.758001313s) [noise=true , address=185.167.204.206:4057 , piece_id=WNX7U6DN7HTE3XLVJAJ4FTCYFR3B2MLMLF5I46GALE463DWQ22WQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1xy38NywWy5CSJ7CfupY78vbkHg6cQzgNhM2Z1xnpvw9dwvMoT, address:185.167.204.206:4057): protocol: rpc: dial tcp 185.167.204.206:4057: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296027916 (56h32m27.758120399s) [noise=true , address=185.167.204.205:4003 , piece_id=NQX2I3KTMYFLS2B2C5Y2PGZRXCD7QJCZLABY4PTYVEFGAVCYX2EQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1CjWvvJ4FdZc41sRgsQFWz4Y1ATZKgzsvDcUTmBSrCC15ekCmF, address:185.167.204.205:4003): protocol: rpc: dial tcp 185.167.204.205:4003: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296100945 (56h32m27.758193418s) [noise=true , address=185.167.204.205:4032 , piece_id=AN4YMHK6NKM5RIWAZHX2FPBGQ3VRFRYBKHBPIAL2HQWBK5XXNNOA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12558dx7zMNDfabV55vSsP9XwdxXgoPM8aHW53femVZzkCRxsKN, address:185.167.204.205:4032): protocol: rpc: dial tcp 185.167.204.205:4032: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296150660 (56h32m27.758243133s) [noise=true , address=185.167.204.205:4029 , piece_id=UVEMQY4WGBKGGT75ILTBPUCFZ4U252QGPDTBPT2BNMOR3V5XYCPA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12DyF12w2JXdkGLfLx9VE1E5f28VnMg7M2b3osfsDtGX9o8Tvsn, address:185.167.204.205:4029): protocol: write tcp 188.166.225.170:36604->185.167.204.205:4029: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296276239 (56h32m27.758368722s) [noise=true , address=185.167.204.206:4069 , piece_id=VKW6XTWPZPD3MDANS3FIYYD2UPDW47WBCGLNTOVYVTWA5IT23DBQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12Uan1TT5oAM3axhaviVFbvh83gThiZvwHMwbCD31ZZens6BQ4U, address:185.167.204.206:4069): protocol: rpc: dial tcp 185.167.204.206:4069: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296438448 (56h32m27.758530931s) [noise=true , address=185.167.204.206:4048 , piece_id=PRBNW47DU6K26GYUUUZSYTYQO3GNFC6DEV2OYQDP4R2RV5L5VNXA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1sY4wHBkSVUDWWg8T8Vm1hpRbbpb3ttqnqDVY8uWyG27gqBwT9, address:185.167.204.206:4048): protocol: write tcp 188.166.225.170:59140->185.167.204.206:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296628780 (56h32m27.758721253s) [noise=true , address=185.167.204.206:4055 , piece_id=KH2E6T37MHNJJ56CR3EIKMEUEWMAVP2XNMZ7K5QVKCFZJIVXRTUA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12CYRieezm1mjtGchWdPrB2mNaHVRvTS85umjB6cQyYKnvisXSB, address:185.167.204.206:4055): protocol: rpc: dial tcp 185.167.204.206:4055: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.297155593 (56h32m27.759248076s) [noise=true , address=185.167.204.205:4022 , piece_id=VU4625JC4A4ZVXFGJZTDMDZXYCNXJ67OQTEKWRLMMHGHPEHY5J7A, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj, address:185.167.204.205:4022): protocol: write tcp 188.166.225.170:57938->185.167.204.205:4022: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.297184969 (56h32m27.759277442s) [seg_pos={0 0}, upload=49746]: Done waiting for segment. successful: 12 optimal: 26 errs: <nil>
Sep 25 22:03:42.465979471 (56h32m27.928071944s) [upload=49746]: Done uploading segments...
Sep 25 22:03:42.466010200 (56h32m27.928102673s) [upload=49746]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 12)
Duration: 4.853556549s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.eu1.gateway)antiblip:
1 out of 60 is failed (threshold=3) [............................................X...............] duration:
176.041601ms remote:
"tcp:207.211.208.129:443" response headers:
Accept-Ranges=bytes Content-Length=352 Content-Security-Policy=block-all-mixed-content Server=Storj X-Request-Id=JdrtmpHLrtw X-Xss-Protection=1; mode=block Content-Type=application/xml Vary=Origin X-Amz-Request-Id=1868A6DD9BAF33BB Date=Thu, 25 Sep 2025 22:25:25 GMT target:
https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318
FAIL (2025-09-25 22:35:58 UTC)
orbiter warning: unexpected status code "500 Internal Server Error" "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>InternalError</Code><Message>We encountered an internal error, please try again.: cause(uplink: metaclient: context canceled)</Message><Key>hello-world.txt</Key><BucketName>uptime-checks</BucketName><Resource>/hello-world.txt</Resource><RequestId>1868A6DD9BAF33BB</RequestId><HostId></HostId></Error>"
Duration: 176.067119ms
edge
gateway
vhost
bgp
eu1
|
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.eu1.gateway)antiblip:
1 out of 60 is failed (threshold=3) [.............................................X..............] duration:
174.128298ms remote:
"tcp:207.211.208.129:443" response headers:
Accept-Ranges=bytes Content-Length=352 Content-Security-Policy=block-all-mixed-content Server=Storj X-Request-Id=JdrtmpHLrtw X-Xss-Protection=1; mode=block Content-Type=application/xml Vary=Origin X-Amz-Request-Id=1868A6DD9BAF33BB Date=Thu, 25 Sep 2025 22:25:25 GMT target:
https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318
FAIL (2025-09-25 22:35:41 UTC)
orbiter warning: unexpected status code "500 Internal Server Error" "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>InternalError</Code><Message>We encountered an internal error, please try again.: cause(uplink: metaclient: context canceled)</Message><Key>hello-world.txt</Key><BucketName>uptime-checks</BucketName><Resource>/hello-world.txt</Resource><RequestId>1868A6DD9BAF33BB</RequestId><HostId></HostId></Error>"
Duration: 174.156081ms
edge
gateway
vhost
bgp
eu1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
2 out of 60 is failed (threshold=3) [...........XX...............................................] duration:
4.816088231s
FAIL (2025-09-25 22:35:22 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 12)
======LOGS======
9 , piece_id=4YT7BXXDQVTR2QL4YBWVMO3YJCZQ57PTTEQW7GCZELWC42HQUDQQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.227500377 (56h31m29.68959285s) [noise=true , address=81.171.14.70:4034 , piece_id=EXVA3SFFPD3GJDYIFHAOP7GMJQHGZECMX5R6MHLRDMMCU7LOA7KA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.253492519 (56h31m29.715584993s) [noise=true , address=81.171.14.70:4028 , piece_id=Y237IBPHRRI7NMOYJG65AZI3LCCZ6JUJPX5WWGIDX5RYHCNVIG3A, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.267890396 (56h31m29.729982869s) [noise=true , address=89.149.201.147:4010 , piece_id=ANMXRI22IXAZFHKB7ZHFGEERKT2KGUYQEFKEOJNJDC5TFB6B3STQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.280031865 (56h31m29.742124348s) [noise=true , address=89.149.201.147:4020 , piece_id=RML2JG4QZSS6H5GTP7BW3B4SFVZLLHHAOUS3LDCOT2BGYTC3HACA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.294783955 (56h31m29.756876428s) [noise=true , address=89.149.201.147:4017 , piece_id=P576E6UE5UADQZYKR4XN4I5X4MCGMXT3B7PBGVNWBZ6I67ZVCJHA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.462969447 (56h31m29.92506192s) [noise=true , address=81.171.14.70:4035 , piece_id=GWVM3MZRGRVJ5RWG227JJXTULF5MUJ4FHWCNHLW7J3GDYXRSXRXA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.668506081 (56h31m30.130598554s) [noise=true , address=81.171.14.70:4045 , piece_id=BSKEEYML4LQZ7PWB45HQNX2YFJTFIEKIILDUZIWCXMVZZGGKHGUQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.702001372 (56h31m30.164093845s) [noise=true , address=89.149.201.147:4014 , piece_id=3VH5QXDWYX2NR3LODHSATJCP5ABF7BI52GJAR3VLPW42CPJJXQNA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.728242188 (56h31m30.190334661s) [noise=true , address=89.149.201.147:4016 , piece_id=PT5TUJDTS2TDBNLH5722GJTZ5BIO4LZKEDKBGUKHEF37TPADW7MA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.797046563 (56h31m30.259139036s) [noise=true , address=89.149.201.147:4012 , piece_id=TY6OVHC4ITI2E32GE7LP3O2YQ46FKWF6A6IIBN4JFAB7WPYXXJ2A, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:03:42.295273049 (56h32m27.757365543s) [noise=true , address=185.167.204.206:4056 , piece_id=4XAHI4JY2MPISEOZ6VJ5GCJ5WZJKHRXIDK2EQGEN7VSYNRYJH24A, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1uSTuNA28oH2s5Rkv2jaKoC2nZWSgpMJXyxwFTCKbr44EvEXrC, address:185.167.204.206:4056): protocol: rpc: dial tcp 185.167.204.206:4056: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295296423 (56h32m27.757388906s) [noise=true , address=185.167.204.205:4008 , piece_id=NXU5IGU2NUZPVR5KIUKLZ7QTPYLGDIKZQNMPBJRPCVYEVNIV7HMA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12ZHTTaGQUTftxEULFqWbH6w1HnWA7PavHrXnvMCD947FxtmsrS, address:185.167.204.205:4008): protocol: rpc: dial tcp 185.167.204.205:4008: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295427543 (56h32m27.757520026s) [noise=true , address=185.167.204.206:4064 , piece_id=7GVKAUOISAORFKOXUNOJMEEFNNUVVZUQ2RGCD6BNVCDTH6ONPQIQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1ZB3emcQntFxpr7zBiE98eesnjoTnNKinw3FPtGU2TgznGakeb, address:185.167.204.206:4064): protocol: rpc: dial tcp 185.167.204.206:4064: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295495543 (56h32m27.757588026s) [noise=true , address=185.167.204.206:4059 , piece_id=G4B2XQYXOG2LWBAIYBM4BWLI7PEPE2XDM5A57YHYR6SOWW7ODSVA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:121gTedy8TPx44wmLqdQpZ1gDDsL8D9K1EvQJffR7oKY3bT9DXh, address:185.167.204.206:4059): protocol: rpc: dial tcp 185.167.204.206:4059: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295562701 (56h32m27.757655184s) [noise=true , address=185.167.204.205:4030 , piece_id=U6SOG6BNEIWDGYWDRZOCLTROZVZXPPG5ZI52HLRPKTDKFVQN3EJA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12w5R2Whxgg4AYNNvTkKmFq4pzahF1uZiVzwgVygc9iZDAXMdSk, address:185.167.204.205:4030): protocol: rpc: dial tcp 185.167.204.205:4030: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295700453 (56h32m27.757792926s) [noise=true , address=185.167.204.205:4010 , piece_id=DPMSKQDDOU2PUXHINTG7JJXLKXJE2PC6B3KNSKN5UIGAKBM3GZVQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12oTu1GaRsH7fHqxLkQ9c9ikjG4zRN3NChssa5gfpLwrcEehoyV, address:185.167.204.205:4010): protocol: rpc: dial tcp 185.167.204.205:4010: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295713257 (56h32m27.75780574s) [noise=true , address=185.167.204.206:4053 , piece_id=74G7LWVYE7GO4GPSG67BD6QVE7UBLGIAKLSG447PWWLNBM2FZNTA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12SrFttJpndpv7TKJJYsKthWnGWBYQGHpS9CwqTd4mqBP9NCCm8, address:185.167.204.206:4053): protocol: rpc: dial tcp 185.167.204.206:4053: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295852051 (56h32m27.757944534s) [noise=true , address=185.167.204.206:4065 , piece_id=VKBLX4ASIKVHMUUCTD7GFFPEHJLKREGCDGI7LCVRSASDHBAWI6AQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12oPo26KZGZr7o7M18KZiX5PMAAMwHK8FPT59y1hSLhkB6z3Lcj, address:185.167.204.206:4065): protocol: rpc: dial tcp 185.167.204.206:4065: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295908840 (56h32m27.758001313s) [noise=true , address=185.167.204.206:4057 , piece_id=WNX7U6DN7HTE3XLVJAJ4FTCYFR3B2MLMLF5I46GALE463DWQ22WQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1xy38NywWy5CSJ7CfupY78vbkHg6cQzgNhM2Z1xnpvw9dwvMoT, address:185.167.204.206:4057): protocol: rpc: dial tcp 185.167.204.206:4057: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296027916 (56h32m27.758120399s) [noise=true , address=185.167.204.205:4003 , piece_id=NQX2I3KTMYFLS2B2C5Y2PGZRXCD7QJCZLABY4PTYVEFGAVCYX2EQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1CjWvvJ4FdZc41sRgsQFWz4Y1ATZKgzsvDcUTmBSrCC15ekCmF, address:185.167.204.205:4003): protocol: rpc: dial tcp 185.167.204.205:4003: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296100945 (56h32m27.758193418s) [noise=true , address=185.167.204.205:4032 , piece_id=AN4YMHK6NKM5RIWAZHX2FPBGQ3VRFRYBKHBPIAL2HQWBK5XXNNOA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12558dx7zMNDfabV55vSsP9XwdxXgoPM8aHW53femVZzkCRxsKN, address:185.167.204.205:4032): protocol: rpc: dial tcp 185.167.204.205:4032: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296150660 (56h32m27.758243133s) [noise=true , address=185.167.204.205:4029 , piece_id=UVEMQY4WGBKGGT75ILTBPUCFZ4U252QGPDTBPT2BNMOR3V5XYCPA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12DyF12w2JXdkGLfLx9VE1E5f28VnMg7M2b3osfsDtGX9o8Tvsn, address:185.167.204.205:4029): protocol: write tcp 188.166.225.170:36604->185.167.204.205:4029: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296276239 (56h32m27.758368722s) [noise=true , address=185.167.204.206:4069 , piece_id=VKW6XTWPZPD3MDANS3FIYYD2UPDW47WBCGLNTOVYVTWA5IT23DBQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12Uan1TT5oAM3axhaviVFbvh83gThiZvwHMwbCD31ZZens6BQ4U, address:185.167.204.206:4069): protocol: rpc: dial tcp 185.167.204.206:4069: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296438448 (56h32m27.758530931s) [noise=true , address=185.167.204.206:4048 , piece_id=PRBNW47DU6K26GYUUUZSYTYQO3GNFC6DEV2OYQDP4R2RV5L5VNXA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1sY4wHBkSVUDWWg8T8Vm1hpRbbpb3ttqnqDVY8uWyG27gqBwT9, address:185.167.204.206:4048): protocol: write tcp 188.166.225.170:59140->185.167.204.206:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296628780 (56h32m27.758721253s) [noise=true , address=185.167.204.206:4055 , piece_id=KH2E6T37MHNJJ56CR3EIKMEUEWMAVP2XNMZ7K5QVKCFZJIVXRTUA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12CYRieezm1mjtGchWdPrB2mNaHVRvTS85umjB6cQyYKnvisXSB, address:185.167.204.206:4055): protocol: rpc: dial tcp 185.167.204.206:4055: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.297155593 (56h32m27.759248076s) [noise=true , address=185.167.204.205:4022 , piece_id=VU4625JC4A4ZVXFGJZTDMDZXYCNXJ67OQTEKWRLMMHGHPEHY5J7A, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj, address:185.167.204.205:4022): protocol: write tcp 188.166.225.170:57938->185.167.204.205:4022: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.297184969 (56h32m27.759277442s) [seg_pos={0 0}, upload=49746]: Done waiting for segment. successful: 12 optimal: 26 errs: <nil>
Sep 25 22:03:42.465979471 (56h32m27.928071944s) [upload=49746]: Done uploading segments...
Sep 25 22:03:42.466010200 (56h32m27.928102673s) [upload=49746]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 12)
Duration: 4.816118478s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................................................X..] duration:
6.188305558s
FAIL (2025-09-25 22:35:17 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Sep 25 22:33:20.609515379 (57h2m6.071607862s) [upload=50198]: Starting upload
Sep 25 22:33:20.609634665 (57h2m6.071727138s) [upload=50198]: Done starting upload
Sep 25 22:33:20.609747009 (57h2m6.071839492s) [upload=50198]: Uploading segments...
Sep 25 22:33:20.609932132 (57h2m6.072024605s) [upload=50198]: Got next segment. Inline: false
Sep 25 22:33:20.609977649 (57h2m6.072070132s) [upload=50198]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 25 22:33:20.609983990 (57h2m6.072076473s) [upload=50198]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 25 22:33:20.876293406 (57h2m6.338385889s) [seg_pos={0 0}, upload=50198]: Begin upload segment...
Sep 25 22:33:20.876365564 (57h2m6.338458047s) [seg_pos={0 0}, upload=50198]: Done begin upload segment.
Sep 25 22:33:20.876398446 (57h2m6.338490919s) [noise=true , address=45.91.116.7:4010 , piece_id=QZEN3HHQVP5SQ3ABXX6AUG2T7V2JTJYLKXPEM2FCIWEIRSZ2IOUA, seg_pos={0 0}, upload=50198]: Uploading piece...
Sep 25 22:33:20.876412874 (57h2m6.338505347s) [noise=true , address=145.79.227.11:4729 , piece_id=WTDDZ72XM433FAERK56RR7BWIFEFFMKJ7PKKIQQUCLWZTTTTB5TA, seg_pos={0 0}, upload=50198]: Uploading piece...
Sep 25 22:33:20.876460024 (57h2m6.338552507s) [noise=true , address=145.79.227.2:4115 , piece_id=GDDMOKVGUSPCUNXCYSADTSDEWV3WX6YLR3HSMC72AI7345WMWI6Q, seg_pos={0 0}, upload=50198]: Uploading piece...
Sep 25 22:33:20.876711823 (57h2m6.338804296s) [noise=true , address=145.79.227.4:4185 , piece_id=UDAN2TY4RX3ATTJAYBL5G3ZXKLYIXFIBPYJWRIG2MPD3LJH62UGA, seg_pos={0 0}, upload=50198]: Uploading piece...
Sep 25 22:33:20.877150188 (57h2m6.339242661s) [noise=true , address=145.79.227.6:4257 , piece_id=NLAZ3I5A6RU3PWBBSMICAETTWZ7UZLO6X5VEXUZPZ2ZB4KAI6I5A, seg_pos={0 0}, upload=50198]: Uploading piece...
Sep 25 22:33:20.877621355 (57h2m6.339713838s) [noise=true , address=145.79.227.17:4972 , piece_id=OLLDZMGRQBTZAQURMSU4AVHOAFCD3VBWCH6Y5WQXENHW2WB3D6BQ, seg_pos={0 0}, upload=50198]: Uploading piece...
Sep 25 22:33:20.877942707 (57h2m6.340035181s) [noise=true , address=145.79.227.18:4007 , piece_id=OWW73ZTC3OLJTGC4IWXZ44VT4ERGUE4YGGULZCSJKYEHNXRNLLRA, seg_pos={0 0}, upload=50198]: Uploading piece...
Sep 25 22:33:20.878252356 (57h2m6.340344829s) [noise=true , address=45.91.116.4:4939 , piece_id=7UY722GYY34MQHO6FCVNCMXKV67RUX2TTNXGUXGGDEBE63HPI6DQ, seg_pos={0 0}, upload=50198]: Uploading piece...
Sep 25 22:33:20.878573619 (57h2m6.340666092s) [noise=true , address=145.79.227.19:4079 , piece_id=TSXXBPAF2KFTUP2MDC6NKGLP2PF666PMSXPGYYKDPO25FRF7YQ6Q, seg_pos={0 0}, upload=50198]: Uploading piece...
Sep 25 22:33:20.881006501 (57h2m6.343098975s) [noise=true , address=45.91.116.10:4082 , piece_id=OJUG2YVVDOHECUD3DLDJANOWGXKS76BD3KIWFF5Q2OTRELONDWUQ, seg_pos={0 0}, upload=50198]: Uploading piece...
Sep 25 22:33:20.882423098 (57h2m6.344515571s) [noise=true , address=145.79.227.13:4802 , piece_id=UNCMDLJ7BWSS2SOAXZRUYR7EQTKZWOYFHJSVB22DHNH32VADTPNA, seg_pos={0 0}, upload=50198]: Uploading piece...
Sep 25 22:33:20.882592721 (57h2m6.344685194s) [noise=true , address=145.79.227.6:4259 , piece_id=VFXMPOPGL2UF6D6NLM7PW3GKM5YEQHT3ZDMZGOR5VEX3NNLOAXAQ, seg_pos={0 0}, upload=50198]: Uploading piece...
Sep 25 22:33:20.885196980 (57h2m6.347289464s) [noise=true , address=145.79.227.3:4151 , piece_id=ZMJ7WBX4DWOWTTJTLJXFZKBR4MRE2RS2FQWRM2YZUC6NHVEAUZFA, seg_pos={0 0}, upload=50198]: Uploading piece...
Sep 25 22:33:20.888905401 (57h2m6.350997874s) [noise=true , address=145.79.227.15:4874 , piece_id=I5VJYG36ZG63ZRJUC2WKXISA5C7EJQSV3XZJ7FLN5MWMC3CDBNEA, seg_pos={0 0}, upload=50198]: Uploading piece...
Sep 25 22:33:20.891203167 (57h2m6.35329565s) [noise=true , address=77.90.31.15:4697 , piece_id=RRP4PCTAVSJYH377LRZ43I36D4LWG7EPHE23LDBUZLP2DUR3RNDQ, seg_pos={0 0}, upload=50198]: Uploading piece...
Sep 25 22:33:20.892678096 (57h2m6.354770569s) [noise=true , address=145.79.227.15:4873 , piece_id=C3JJ5HPO237IQRGLKQ5QZSKEJM2HNOH4YNBMULOE7CXLFO4UEW3Q, seg_pos={0 0}, upload=50198]: Uploading piece...
Sep 25 22:33:22.883631748 (57h2m8.345724231s) [upload=50198]: Next returned nil segment
Sep 25 22:33:22.883655924 (57h2m8.345748397s) [upload=50198]: Waiting for error group managing segments...
Sep 25 22:33:23.186669559 (57h2m8.648762032s) [noise=true , address=77.90.31.15:4697 , piece_id=RRP4PCTAVSJYH377LRZ43I36D4LWG7EPHE23LDBUZLP2DUR3RNDQ, seg_pos={0 0}, upload=50198]: Done uploading piece. err: <nil>
Sep 25 22:33:23.402505165 (57h2m8.864597638s) [noise=true , address=45.91.116.7:4010 , piece_id=QZEN3HHQVP5SQ3ABXX6AUG2T7V2JTJYLKXPEM2FCIWEIRSZ2IOUA, seg_pos={0 0}, upload=50198]: Done uploading piece. err: <nil>
Sep 25 22:33:23.460847597 (57h2m8.92294007s) [noise=true , address=45.91.116.4:4939 , piece_id=7UY722GYY34MQHO6FCVNCMXKV67RUX2TTNXGUXGGDEBE63HPI6DQ, seg_pos={0 0}, upload=50198]: Done uploading piece. err: <nil>
Sep 25 22:33:23.475828172 (57h2m8.937920645s) [noise=true , address=45.91.116.10:4082 , piece_id=OJUG2YVVDOHECUD3DLDJANOWGXKS76BD3KIWFF5Q2OTRELONDWUQ, seg_pos={0 0}, upload=50198]: Done uploading piece. err: <nil>
Sep 25 22:33:41.316700325 (57h2m26.778792808s) [noise=true , address=145.79.227.18:4007 , piece_id=OWW73ZTC3OLJTGC4IWXZ44VT4ERGUE4YGGULZCSJKYEHNXRNLLRA, seg_pos={0 0}, upload=50198]: Done uploading piece. err: <nil>
Sep 25 22:33:42.134617358 (57h2m27.596709841s) [noise=true , address=145.79.227.11:4729 , piece_id=WTDDZ72XM433FAERK56RR7BWIFEFFMKJ7PKKIQQUCLWZTTTTB5TA, seg_pos={0 0}, upload=50198]: Done uploading piece. err: <nil>
Sep 25 22:33:43.316084677 (57h2m28.77817716s) [noise=true , address=145.79.227.15:4874 , piece_id=I5VJYG36ZG63ZRJUC2WKXISA5C7EJQSV3XZJ7FLN5MWMC3CDBNEA, seg_pos={0 0}, upload=50198]: Done uploading piece. err: <nil>
Sep 25 22:33:43.336895665 (57h2m28.798988148s) [noise=true , address=145.79.227.15:4873 , piece_id=C3JJ5HPO237IQRGLKQ5QZSKEJM2HNOH4YNBMULOE7CXLFO4UEW3Q, seg_pos={0 0}, upload=50198]: Done uploading piece. err: <nil>
Sep 25 22:33:44.766574920 (57h2m30.228667393s) [noise=true , address=145.79.227.17:4972 , piece_id=OLLDZMGRQBTZAQURMSU4AVHOAFCD3VBWCH6Y5WQXENHW2WB3D6BQ, seg_pos={0 0}, upload=50198]: Done uploading piece. err: <nil>
Sep 25 22:33:47.245353345 (57h2m32.707445818s) [noise=true , address=145.79.227.13:4802 , piece_id=UNCMDLJ7BWSS2SOAXZRUYR7EQTKZWOYFHJSVB22DHNH32VADTPNA, seg_pos={0 0}, upload=50198]: Done uploading piece. err: <nil>
Sep 25 22:33:48.389979384 (57h2m33.852071867s) [noise=true , address=145.79.227.19:4079 , piece_id=TSXXBPAF2KFTUP2MDC6NKGLP2PF666PMSXPGYYKDPO25FRF7YQ6Q, seg_pos={0 0}, upload=50198]: Done uploading piece. err: <nil>
Sep 25 22:33:49.870866176 (57h2m35.332958649s) [noise=true , address=145.79.227.3:4151 , piece_id=ZMJ7WBX4DWOWTTJTLJXFZKBR4MRE2RS2FQWRM2YZUC6NHVEAUZFA, seg_pos={0 0}, upload=50198]: Done uploading piece. err: <nil>
Sep 25 22:33:49.927808223 (57h2m35.389900706s) [noise=true , address=145.79.227.2:4115 , piece_id=GDDMOKVGUSPCUNXCYSADTSDEWV3WX6YLR3HSMC72AI7345WMWI6Q, seg_pos={0 0}, upload=50198]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: expected piece hash; context deadline exceeded; EOF
Sep 25 22:33:49.927871833 (57h2m35.389964316s) [noise=true , address=145.79.227.6:4259 , piece_id=VFXMPOPGL2UF6D6NLM7PW3GKM5YEQHT3ZDMZGOR5VEX3NNLOAXAQ, seg_pos={0 0}, upload=50198]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: expected piece hash; context deadline exceeded; EOF
Sep 25 22:33:49.927907451 (57h2m35.389999924s) [noise=true , address=145.79.227.4:4185 , piece_id=UDAN2TY4RX3ATTJAYBL5G3ZXKLYIXFIBPYJWRIG2MPD3LJH62UGA, seg_pos={0 0}, upload=50198]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:145.79.227.4:4185): protocol: write tcp 188.166.225.170:46508->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:46508->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46508->145.79.227.4:4185: use of closed network connection
Sep 25 22:33:49.927937458 (57h2m35.390029931s) [noise=true , address=145.79.227.6:4257 , piece_id=NLAZ3I5A6RU3PWBBSMICAETTWZ7UZLO6X5VEXUZPZ2ZB4KAI6I5A, seg_pos={0 0}, upload=50198]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: expected piece hash; context deadline exceeded; EOF
Sep 25 22:33:49.927971413 (57h2m35.390063886s) [seg_pos={0 0}, upload=50198]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Sep 25 22:33:50.167900726 (57h2m35.629993199s) [upload=50198]: Done uploading segments...
Sep 25 22:33:50.167951733 (57h2m35.630044216s) [upload=50198]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 6.18833875s
core
uplink
us-select-1
|
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.eu1.gateway)antiblip:
1 out of 60 is failed (threshold=3) [..............................................X.............] duration:
173.890185ms remote:
"tcp:207.211.208.129:443" response headers:
Accept-Ranges=bytes Content-Length=352 Content-Security-Policy=block-all-mixed-content Server=Storj X-Request-Id=JdrtmpHLrtw X-Xss-Protection=1; mode=block Content-Type=application/xml Vary=Origin X-Amz-Request-Id=1868A6DD9BAF33BB Date=Thu, 25 Sep 2025 22:25:25 GMT target:
https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318
FAIL (2025-09-25 22:35:06 UTC)
orbiter warning: unexpected status code "500 Internal Server Error" "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>InternalError</Code><Message>We encountered an internal error, please try again.: cause(uplink: metaclient: context canceled)</Message><Key>hello-world.txt</Key><BucketName>uptime-checks</BucketName><Resource>/hello-world.txt</Resource><RequestId>1868A6DD9BAF33BB</RequestId><HostId></HostId></Error>"
Duration: 173.922266ms
edge
gateway
vhost
bgp
eu1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
2 out of 60 is failed (threshold=3) [............XX..............................................] duration:
4.840323458s
FAIL (2025-09-25 22:34:46 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 12)
======LOGS======
9 , piece_id=4YT7BXXDQVTR2QL4YBWVMO3YJCZQ57PTTEQW7GCZELWC42HQUDQQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.227500377 (56h31m29.68959285s) [noise=true , address=81.171.14.70:4034 , piece_id=EXVA3SFFPD3GJDYIFHAOP7GMJQHGZECMX5R6MHLRDMMCU7LOA7KA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.253492519 (56h31m29.715584993s) [noise=true , address=81.171.14.70:4028 , piece_id=Y237IBPHRRI7NMOYJG65AZI3LCCZ6JUJPX5WWGIDX5RYHCNVIG3A, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.267890396 (56h31m29.729982869s) [noise=true , address=89.149.201.147:4010 , piece_id=ANMXRI22IXAZFHKB7ZHFGEERKT2KGUYQEFKEOJNJDC5TFB6B3STQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.280031865 (56h31m29.742124348s) [noise=true , address=89.149.201.147:4020 , piece_id=RML2JG4QZSS6H5GTP7BW3B4SFVZLLHHAOUS3LDCOT2BGYTC3HACA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.294783955 (56h31m29.756876428s) [noise=true , address=89.149.201.147:4017 , piece_id=P576E6UE5UADQZYKR4XN4I5X4MCGMXT3B7PBGVNWBZ6I67ZVCJHA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.462969447 (56h31m29.92506192s) [noise=true , address=81.171.14.70:4035 , piece_id=GWVM3MZRGRVJ5RWG227JJXTULF5MUJ4FHWCNHLW7J3GDYXRSXRXA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.668506081 (56h31m30.130598554s) [noise=true , address=81.171.14.70:4045 , piece_id=BSKEEYML4LQZ7PWB45HQNX2YFJTFIEKIILDUZIWCXMVZZGGKHGUQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.702001372 (56h31m30.164093845s) [noise=true , address=89.149.201.147:4014 , piece_id=3VH5QXDWYX2NR3LODHSATJCP5ABF7BI52GJAR3VLPW42CPJJXQNA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.728242188 (56h31m30.190334661s) [noise=true , address=89.149.201.147:4016 , piece_id=PT5TUJDTS2TDBNLH5722GJTZ5BIO4LZKEDKBGUKHEF37TPADW7MA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.797046563 (56h31m30.259139036s) [noise=true , address=89.149.201.147:4012 , piece_id=TY6OVHC4ITI2E32GE7LP3O2YQ46FKWF6A6IIBN4JFAB7WPYXXJ2A, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:03:42.295273049 (56h32m27.757365543s) [noise=true , address=185.167.204.206:4056 , piece_id=4XAHI4JY2MPISEOZ6VJ5GCJ5WZJKHRXIDK2EQGEN7VSYNRYJH24A, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1uSTuNA28oH2s5Rkv2jaKoC2nZWSgpMJXyxwFTCKbr44EvEXrC, address:185.167.204.206:4056): protocol: rpc: dial tcp 185.167.204.206:4056: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295296423 (56h32m27.757388906s) [noise=true , address=185.167.204.205:4008 , piece_id=NXU5IGU2NUZPVR5KIUKLZ7QTPYLGDIKZQNMPBJRPCVYEVNIV7HMA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12ZHTTaGQUTftxEULFqWbH6w1HnWA7PavHrXnvMCD947FxtmsrS, address:185.167.204.205:4008): protocol: rpc: dial tcp 185.167.204.205:4008: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295427543 (56h32m27.757520026s) [noise=true , address=185.167.204.206:4064 , piece_id=7GVKAUOISAORFKOXUNOJMEEFNNUVVZUQ2RGCD6BNVCDTH6ONPQIQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1ZB3emcQntFxpr7zBiE98eesnjoTnNKinw3FPtGU2TgznGakeb, address:185.167.204.206:4064): protocol: rpc: dial tcp 185.167.204.206:4064: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295495543 (56h32m27.757588026s) [noise=true , address=185.167.204.206:4059 , piece_id=G4B2XQYXOG2LWBAIYBM4BWLI7PEPE2XDM5A57YHYR6SOWW7ODSVA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:121gTedy8TPx44wmLqdQpZ1gDDsL8D9K1EvQJffR7oKY3bT9DXh, address:185.167.204.206:4059): protocol: rpc: dial tcp 185.167.204.206:4059: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295562701 (56h32m27.757655184s) [noise=true , address=185.167.204.205:4030 , piece_id=U6SOG6BNEIWDGYWDRZOCLTROZVZXPPG5ZI52HLRPKTDKFVQN3EJA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12w5R2Whxgg4AYNNvTkKmFq4pzahF1uZiVzwgVygc9iZDAXMdSk, address:185.167.204.205:4030): protocol: rpc: dial tcp 185.167.204.205:4030: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295700453 (56h32m27.757792926s) [noise=true , address=185.167.204.205:4010 , piece_id=DPMSKQDDOU2PUXHINTG7JJXLKXJE2PC6B3KNSKN5UIGAKBM3GZVQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12oTu1GaRsH7fHqxLkQ9c9ikjG4zRN3NChssa5gfpLwrcEehoyV, address:185.167.204.205:4010): protocol: rpc: dial tcp 185.167.204.205:4010: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295713257 (56h32m27.75780574s) [noise=true , address=185.167.204.206:4053 , piece_id=74G7LWVYE7GO4GPSG67BD6QVE7UBLGIAKLSG447PWWLNBM2FZNTA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12SrFttJpndpv7TKJJYsKthWnGWBYQGHpS9CwqTd4mqBP9NCCm8, address:185.167.204.206:4053): protocol: rpc: dial tcp 185.167.204.206:4053: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295852051 (56h32m27.757944534s) [noise=true , address=185.167.204.206:4065 , piece_id=VKBLX4ASIKVHMUUCTD7GFFPEHJLKREGCDGI7LCVRSASDHBAWI6AQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12oPo26KZGZr7o7M18KZiX5PMAAMwHK8FPT59y1hSLhkB6z3Lcj, address:185.167.204.206:4065): protocol: rpc: dial tcp 185.167.204.206:4065: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295908840 (56h32m27.758001313s) [noise=true , address=185.167.204.206:4057 , piece_id=WNX7U6DN7HTE3XLVJAJ4FTCYFR3B2MLMLF5I46GALE463DWQ22WQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1xy38NywWy5CSJ7CfupY78vbkHg6cQzgNhM2Z1xnpvw9dwvMoT, address:185.167.204.206:4057): protocol: rpc: dial tcp 185.167.204.206:4057: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296027916 (56h32m27.758120399s) [noise=true , address=185.167.204.205:4003 , piece_id=NQX2I3KTMYFLS2B2C5Y2PGZRXCD7QJCZLABY4PTYVEFGAVCYX2EQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1CjWvvJ4FdZc41sRgsQFWz4Y1ATZKgzsvDcUTmBSrCC15ekCmF, address:185.167.204.205:4003): protocol: rpc: dial tcp 185.167.204.205:4003: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296100945 (56h32m27.758193418s) [noise=true , address=185.167.204.205:4032 , piece_id=AN4YMHK6NKM5RIWAZHX2FPBGQ3VRFRYBKHBPIAL2HQWBK5XXNNOA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12558dx7zMNDfabV55vSsP9XwdxXgoPM8aHW53femVZzkCRxsKN, address:185.167.204.205:4032): protocol: rpc: dial tcp 185.167.204.205:4032: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296150660 (56h32m27.758243133s) [noise=true , address=185.167.204.205:4029 , piece_id=UVEMQY4WGBKGGT75ILTBPUCFZ4U252QGPDTBPT2BNMOR3V5XYCPA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12DyF12w2JXdkGLfLx9VE1E5f28VnMg7M2b3osfsDtGX9o8Tvsn, address:185.167.204.205:4029): protocol: write tcp 188.166.225.170:36604->185.167.204.205:4029: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296276239 (56h32m27.758368722s) [noise=true , address=185.167.204.206:4069 , piece_id=VKW6XTWPZPD3MDANS3FIYYD2UPDW47WBCGLNTOVYVTWA5IT23DBQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12Uan1TT5oAM3axhaviVFbvh83gThiZvwHMwbCD31ZZens6BQ4U, address:185.167.204.206:4069): protocol: rpc: dial tcp 185.167.204.206:4069: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296438448 (56h32m27.758530931s) [noise=true , address=185.167.204.206:4048 , piece_id=PRBNW47DU6K26GYUUUZSYTYQO3GNFC6DEV2OYQDP4R2RV5L5VNXA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1sY4wHBkSVUDWWg8T8Vm1hpRbbpb3ttqnqDVY8uWyG27gqBwT9, address:185.167.204.206:4048): protocol: write tcp 188.166.225.170:59140->185.167.204.206:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296628780 (56h32m27.758721253s) [noise=true , address=185.167.204.206:4055 , piece_id=KH2E6T37MHNJJ56CR3EIKMEUEWMAVP2XNMZ7K5QVKCFZJIVXRTUA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12CYRieezm1mjtGchWdPrB2mNaHVRvTS85umjB6cQyYKnvisXSB, address:185.167.204.206:4055): protocol: rpc: dial tcp 185.167.204.206:4055: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.297155593 (56h32m27.759248076s) [noise=true , address=185.167.204.205:4022 , piece_id=VU4625JC4A4ZVXFGJZTDMDZXYCNXJ67OQTEKWRLMMHGHPEHY5J7A, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj, address:185.167.204.205:4022): protocol: write tcp 188.166.225.170:57938->185.167.204.205:4022: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.297184969 (56h32m27.759277442s) [seg_pos={0 0}, upload=49746]: Done waiting for segment. successful: 12 optimal: 26 errs: <nil>
Sep 25 22:03:42.465979471 (56h32m27.928071944s) [upload=49746]: Done uploading segments...
Sep 25 22:03:42.466010200 (56h32m27.928102673s) [upload=49746]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 12)
Duration: 4.84036174s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................................................X.] duration:
6.030367721s
FAIL (2025-09-25 22:34:39 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Sep 25 22:33:20.609515379 (57h2m6.071607862s) [upload=50198]: Starting upload
Sep 25 22:33:20.609634665 (57h2m6.071727138s) [upload=50198]: Done starting upload
Sep 25 22:33:20.609747009 (57h2m6.071839492s) [upload=50198]: Uploading segments...
Sep 25 22:33:20.609932132 (57h2m6.072024605s) [upload=50198]: Got next segment. Inline: false
Sep 25 22:33:20.609977649 (57h2m6.072070132s) [upload=50198]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 25 22:33:20.609983990 (57h2m6.072076473s) [upload=50198]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 25 22:33:20.876293406 (57h2m6.338385889s) [seg_pos={0 0}, upload=50198]: Begin upload segment...
Sep 25 22:33:20.876365564 (57h2m6.338458047s) [seg_pos={0 0}, upload=50198]: Done begin upload segment.
Sep 25 22:33:20.876398446 (57h2m6.338490919s) [noise=true , address=45.91.116.7:4010 , piece_id=QZEN3HHQVP5SQ3ABXX6AUG2T7V2JTJYLKXPEM2FCIWEIRSZ2IOUA, seg_pos={0 0}, upload=50198]: Uploading piece...
Sep 25 22:33:20.876412874 (57h2m6.338505347s) [noise=true , address=145.79.227.11:4729 , piece_id=WTDDZ72XM433FAERK56RR7BWIFEFFMKJ7PKKIQQUCLWZTTTTB5TA, seg_pos={0 0}, upload=50198]: Uploading piece...
Sep 25 22:33:20.876460024 (57h2m6.338552507s) [noise=true , address=145.79.227.2:4115 , piece_id=GDDMOKVGUSPCUNXCYSADTSDEWV3WX6YLR3HSMC72AI7345WMWI6Q, seg_pos={0 0}, upload=50198]: Uploading piece...
Sep 25 22:33:20.876711823 (57h2m6.338804296s) [noise=true , address=145.79.227.4:4185 , piece_id=UDAN2TY4RX3ATTJAYBL5G3ZXKLYIXFIBPYJWRIG2MPD3LJH62UGA, seg_pos={0 0}, upload=50198]: Uploading piece...
Sep 25 22:33:20.877150188 (57h2m6.339242661s) [noise=true , address=145.79.227.6:4257 , piece_id=NLAZ3I5A6RU3PWBBSMICAETTWZ7UZLO6X5VEXUZPZ2ZB4KAI6I5A, seg_pos={0 0}, upload=50198]: Uploading piece...
Sep 25 22:33:20.877621355 (57h2m6.339713838s) [noise=true , address=145.79.227.17:4972 , piece_id=OLLDZMGRQBTZAQURMSU4AVHOAFCD3VBWCH6Y5WQXENHW2WB3D6BQ, seg_pos={0 0}, upload=50198]: Uploading piece...
Sep 25 22:33:20.877942707 (57h2m6.340035181s) [noise=true , address=145.79.227.18:4007 , piece_id=OWW73ZTC3OLJTGC4IWXZ44VT4ERGUE4YGGULZCSJKYEHNXRNLLRA, seg_pos={0 0}, upload=50198]: Uploading piece...
Sep 25 22:33:20.878252356 (57h2m6.340344829s) [noise=true , address=45.91.116.4:4939 , piece_id=7UY722GYY34MQHO6FCVNCMXKV67RUX2TTNXGUXGGDEBE63HPI6DQ, seg_pos={0 0}, upload=50198]: Uploading piece...
Sep 25 22:33:20.878573619 (57h2m6.340666092s) [noise=true , address=145.79.227.19:4079 , piece_id=TSXXBPAF2KFTUP2MDC6NKGLP2PF666PMSXPGYYKDPO25FRF7YQ6Q, seg_pos={0 0}, upload=50198]: Uploading piece...
Sep 25 22:33:20.881006501 (57h2m6.343098975s) [noise=true , address=45.91.116.10:4082 , piece_id=OJUG2YVVDOHECUD3DLDJANOWGXKS76BD3KIWFF5Q2OTRELONDWUQ, seg_pos={0 0}, upload=50198]: Uploading piece...
Sep 25 22:33:20.882423098 (57h2m6.344515571s) [noise=true , address=145.79.227.13:4802 , piece_id=UNCMDLJ7BWSS2SOAXZRUYR7EQTKZWOYFHJSVB22DHNH32VADTPNA, seg_pos={0 0}, upload=50198]: Uploading piece...
Sep 25 22:33:20.882592721 (57h2m6.344685194s) [noise=true , address=145.79.227.6:4259 , piece_id=VFXMPOPGL2UF6D6NLM7PW3GKM5YEQHT3ZDMZGOR5VEX3NNLOAXAQ, seg_pos={0 0}, upload=50198]: Uploading piece...
Sep 25 22:33:20.885196980 (57h2m6.347289464s) [noise=true , address=145.79.227.3:4151 , piece_id=ZMJ7WBX4DWOWTTJTLJXFZKBR4MRE2RS2FQWRM2YZUC6NHVEAUZFA, seg_pos={0 0}, upload=50198]: Uploading piece...
Sep 25 22:33:20.888905401 (57h2m6.350997874s) [noise=true , address=145.79.227.15:4874 , piece_id=I5VJYG36ZG63ZRJUC2WKXISA5C7EJQSV3XZJ7FLN5MWMC3CDBNEA, seg_pos={0 0}, upload=50198]: Uploading piece...
Sep 25 22:33:20.891203167 (57h2m6.35329565s) [noise=true , address=77.90.31.15:4697 , piece_id=RRP4PCTAVSJYH377LRZ43I36D4LWG7EPHE23LDBUZLP2DUR3RNDQ, seg_pos={0 0}, upload=50198]: Uploading piece...
Sep 25 22:33:20.892678096 (57h2m6.354770569s) [noise=true , address=145.79.227.15:4873 , piece_id=C3JJ5HPO237IQRGLKQ5QZSKEJM2HNOH4YNBMULOE7CXLFO4UEW3Q, seg_pos={0 0}, upload=50198]: Uploading piece...
Sep 25 22:33:22.883631748 (57h2m8.345724231s) [upload=50198]: Next returned nil segment
Sep 25 22:33:22.883655924 (57h2m8.345748397s) [upload=50198]: Waiting for error group managing segments...
Sep 25 22:33:23.186669559 (57h2m8.648762032s) [noise=true , address=77.90.31.15:4697 , piece_id=RRP4PCTAVSJYH377LRZ43I36D4LWG7EPHE23LDBUZLP2DUR3RNDQ, seg_pos={0 0}, upload=50198]: Done uploading piece. err: <nil>
Sep 25 22:33:23.402505165 (57h2m8.864597638s) [noise=true , address=45.91.116.7:4010 , piece_id=QZEN3HHQVP5SQ3ABXX6AUG2T7V2JTJYLKXPEM2FCIWEIRSZ2IOUA, seg_pos={0 0}, upload=50198]: Done uploading piece. err: <nil>
Sep 25 22:33:23.460847597 (57h2m8.92294007s) [noise=true , address=45.91.116.4:4939 , piece_id=7UY722GYY34MQHO6FCVNCMXKV67RUX2TTNXGUXGGDEBE63HPI6DQ, seg_pos={0 0}, upload=50198]: Done uploading piece. err: <nil>
Sep 25 22:33:23.475828172 (57h2m8.937920645s) [noise=true , address=45.91.116.10:4082 , piece_id=OJUG2YVVDOHECUD3DLDJANOWGXKS76BD3KIWFF5Q2OTRELONDWUQ, seg_pos={0 0}, upload=50198]: Done uploading piece. err: <nil>
Sep 25 22:33:41.316700325 (57h2m26.778792808s) [noise=true , address=145.79.227.18:4007 , piece_id=OWW73ZTC3OLJTGC4IWXZ44VT4ERGUE4YGGULZCSJKYEHNXRNLLRA, seg_pos={0 0}, upload=50198]: Done uploading piece. err: <nil>
Sep 25 22:33:42.134617358 (57h2m27.596709841s) [noise=true , address=145.79.227.11:4729 , piece_id=WTDDZ72XM433FAERK56RR7BWIFEFFMKJ7PKKIQQUCLWZTTTTB5TA, seg_pos={0 0}, upload=50198]: Done uploading piece. err: <nil>
Sep 25 22:33:43.316084677 (57h2m28.77817716s) [noise=true , address=145.79.227.15:4874 , piece_id=I5VJYG36ZG63ZRJUC2WKXISA5C7EJQSV3XZJ7FLN5MWMC3CDBNEA, seg_pos={0 0}, upload=50198]: Done uploading piece. err: <nil>
Sep 25 22:33:43.336895665 (57h2m28.798988148s) [noise=true , address=145.79.227.15:4873 , piece_id=C3JJ5HPO237IQRGLKQ5QZSKEJM2HNOH4YNBMULOE7CXLFO4UEW3Q, seg_pos={0 0}, upload=50198]: Done uploading piece. err: <nil>
Sep 25 22:33:44.766574920 (57h2m30.228667393s) [noise=true , address=145.79.227.17:4972 , piece_id=OLLDZMGRQBTZAQURMSU4AVHOAFCD3VBWCH6Y5WQXENHW2WB3D6BQ, seg_pos={0 0}, upload=50198]: Done uploading piece. err: <nil>
Sep 25 22:33:47.245353345 (57h2m32.707445818s) [noise=true , address=145.79.227.13:4802 , piece_id=UNCMDLJ7BWSS2SOAXZRUYR7EQTKZWOYFHJSVB22DHNH32VADTPNA, seg_pos={0 0}, upload=50198]: Done uploading piece. err: <nil>
Sep 25 22:33:48.389979384 (57h2m33.852071867s) [noise=true , address=145.79.227.19:4079 , piece_id=TSXXBPAF2KFTUP2MDC6NKGLP2PF666PMSXPGYYKDPO25FRF7YQ6Q, seg_pos={0 0}, upload=50198]: Done uploading piece. err: <nil>
Sep 25 22:33:49.870866176 (57h2m35.332958649s) [noise=true , address=145.79.227.3:4151 , piece_id=ZMJ7WBX4DWOWTTJTLJXFZKBR4MRE2RS2FQWRM2YZUC6NHVEAUZFA, seg_pos={0 0}, upload=50198]: Done uploading piece. err: <nil>
Sep 25 22:33:49.927808223 (57h2m35.389900706s) [noise=true , address=145.79.227.2:4115 , piece_id=GDDMOKVGUSPCUNXCYSADTSDEWV3WX6YLR3HSMC72AI7345WMWI6Q, seg_pos={0 0}, upload=50198]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: expected piece hash; context deadline exceeded; EOF
Sep 25 22:33:49.927871833 (57h2m35.389964316s) [noise=true , address=145.79.227.6:4259 , piece_id=VFXMPOPGL2UF6D6NLM7PW3GKM5YEQHT3ZDMZGOR5VEX3NNLOAXAQ, seg_pos={0 0}, upload=50198]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: expected piece hash; context deadline exceeded; EOF
Sep 25 22:33:49.927907451 (57h2m35.389999924s) [noise=true , address=145.79.227.4:4185 , piece_id=UDAN2TY4RX3ATTJAYBL5G3ZXKLYIXFIBPYJWRIG2MPD3LJH62UGA, seg_pos={0 0}, upload=50198]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:145.79.227.4:4185): protocol: write tcp 188.166.225.170:46508->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:46508->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46508->145.79.227.4:4185: use of closed network connection
Sep 25 22:33:49.927937458 (57h2m35.390029931s) [noise=true , address=145.79.227.6:4257 , piece_id=NLAZ3I5A6RU3PWBBSMICAETTWZ7UZLO6X5VEXUZPZ2ZB4KAI6I5A, seg_pos={0 0}, upload=50198]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: expected piece hash; context deadline exceeded; EOF
Sep 25 22:33:49.927971413 (57h2m35.390063886s) [seg_pos={0 0}, upload=50198]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Sep 25 22:33:50.167900726 (57h2m35.629993199s) [upload=50198]: Done uploading segments...
Sep 25 22:33:50.167951733 (57h2m35.630044216s) [upload=50198]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 6.030395334s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
2 out of 60 is failed (threshold=3) [.............XX.............................................] duration:
4.820997971s
FAIL (2025-09-25 22:34:12 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 12)
======LOGS======
9 , piece_id=4YT7BXXDQVTR2QL4YBWVMO3YJCZQ57PTTEQW7GCZELWC42HQUDQQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.227500377 (56h31m29.68959285s) [noise=true , address=81.171.14.70:4034 , piece_id=EXVA3SFFPD3GJDYIFHAOP7GMJQHGZECMX5R6MHLRDMMCU7LOA7KA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.253492519 (56h31m29.715584993s) [noise=true , address=81.171.14.70:4028 , piece_id=Y237IBPHRRI7NMOYJG65AZI3LCCZ6JUJPX5WWGIDX5RYHCNVIG3A, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.267890396 (56h31m29.729982869s) [noise=true , address=89.149.201.147:4010 , piece_id=ANMXRI22IXAZFHKB7ZHFGEERKT2KGUYQEFKEOJNJDC5TFB6B3STQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.280031865 (56h31m29.742124348s) [noise=true , address=89.149.201.147:4020 , piece_id=RML2JG4QZSS6H5GTP7BW3B4SFVZLLHHAOUS3LDCOT2BGYTC3HACA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.294783955 (56h31m29.756876428s) [noise=true , address=89.149.201.147:4017 , piece_id=P576E6UE5UADQZYKR4XN4I5X4MCGMXT3B7PBGVNWBZ6I67ZVCJHA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.462969447 (56h31m29.92506192s) [noise=true , address=81.171.14.70:4035 , piece_id=GWVM3MZRGRVJ5RWG227JJXTULF5MUJ4FHWCNHLW7J3GDYXRSXRXA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.668506081 (56h31m30.130598554s) [noise=true , address=81.171.14.70:4045 , piece_id=BSKEEYML4LQZ7PWB45HQNX2YFJTFIEKIILDUZIWCXMVZZGGKHGUQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.702001372 (56h31m30.164093845s) [noise=true , address=89.149.201.147:4014 , piece_id=3VH5QXDWYX2NR3LODHSATJCP5ABF7BI52GJAR3VLPW42CPJJXQNA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.728242188 (56h31m30.190334661s) [noise=true , address=89.149.201.147:4016 , piece_id=PT5TUJDTS2TDBNLH5722GJTZ5BIO4LZKEDKBGUKHEF37TPADW7MA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.797046563 (56h31m30.259139036s) [noise=true , address=89.149.201.147:4012 , piece_id=TY6OVHC4ITI2E32GE7LP3O2YQ46FKWF6A6IIBN4JFAB7WPYXXJ2A, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:03:42.295273049 (56h32m27.757365543s) [noise=true , address=185.167.204.206:4056 , piece_id=4XAHI4JY2MPISEOZ6VJ5GCJ5WZJKHRXIDK2EQGEN7VSYNRYJH24A, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1uSTuNA28oH2s5Rkv2jaKoC2nZWSgpMJXyxwFTCKbr44EvEXrC, address:185.167.204.206:4056): protocol: rpc: dial tcp 185.167.204.206:4056: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295296423 (56h32m27.757388906s) [noise=true , address=185.167.204.205:4008 , piece_id=NXU5IGU2NUZPVR5KIUKLZ7QTPYLGDIKZQNMPBJRPCVYEVNIV7HMA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12ZHTTaGQUTftxEULFqWbH6w1HnWA7PavHrXnvMCD947FxtmsrS, address:185.167.204.205:4008): protocol: rpc: dial tcp 185.167.204.205:4008: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295427543 (56h32m27.757520026s) [noise=true , address=185.167.204.206:4064 , piece_id=7GVKAUOISAORFKOXUNOJMEEFNNUVVZUQ2RGCD6BNVCDTH6ONPQIQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1ZB3emcQntFxpr7zBiE98eesnjoTnNKinw3FPtGU2TgznGakeb, address:185.167.204.206:4064): protocol: rpc: dial tcp 185.167.204.206:4064: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295495543 (56h32m27.757588026s) [noise=true , address=185.167.204.206:4059 , piece_id=G4B2XQYXOG2LWBAIYBM4BWLI7PEPE2XDM5A57YHYR6SOWW7ODSVA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:121gTedy8TPx44wmLqdQpZ1gDDsL8D9K1EvQJffR7oKY3bT9DXh, address:185.167.204.206:4059): protocol: rpc: dial tcp 185.167.204.206:4059: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295562701 (56h32m27.757655184s) [noise=true , address=185.167.204.205:4030 , piece_id=U6SOG6BNEIWDGYWDRZOCLTROZVZXPPG5ZI52HLRPKTDKFVQN3EJA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12w5R2Whxgg4AYNNvTkKmFq4pzahF1uZiVzwgVygc9iZDAXMdSk, address:185.167.204.205:4030): protocol: rpc: dial tcp 185.167.204.205:4030: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295700453 (56h32m27.757792926s) [noise=true , address=185.167.204.205:4010 , piece_id=DPMSKQDDOU2PUXHINTG7JJXLKXJE2PC6B3KNSKN5UIGAKBM3GZVQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12oTu1GaRsH7fHqxLkQ9c9ikjG4zRN3NChssa5gfpLwrcEehoyV, address:185.167.204.205:4010): protocol: rpc: dial tcp 185.167.204.205:4010: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295713257 (56h32m27.75780574s) [noise=true , address=185.167.204.206:4053 , piece_id=74G7LWVYE7GO4GPSG67BD6QVE7UBLGIAKLSG447PWWLNBM2FZNTA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12SrFttJpndpv7TKJJYsKthWnGWBYQGHpS9CwqTd4mqBP9NCCm8, address:185.167.204.206:4053): protocol: rpc: dial tcp 185.167.204.206:4053: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295852051 (56h32m27.757944534s) [noise=true , address=185.167.204.206:4065 , piece_id=VKBLX4ASIKVHMUUCTD7GFFPEHJLKREGCDGI7LCVRSASDHBAWI6AQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12oPo26KZGZr7o7M18KZiX5PMAAMwHK8FPT59y1hSLhkB6z3Lcj, address:185.167.204.206:4065): protocol: rpc: dial tcp 185.167.204.206:4065: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295908840 (56h32m27.758001313s) [noise=true , address=185.167.204.206:4057 , piece_id=WNX7U6DN7HTE3XLVJAJ4FTCYFR3B2MLMLF5I46GALE463DWQ22WQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1xy38NywWy5CSJ7CfupY78vbkHg6cQzgNhM2Z1xnpvw9dwvMoT, address:185.167.204.206:4057): protocol: rpc: dial tcp 185.167.204.206:4057: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296027916 (56h32m27.758120399s) [noise=true , address=185.167.204.205:4003 , piece_id=NQX2I3KTMYFLS2B2C5Y2PGZRXCD7QJCZLABY4PTYVEFGAVCYX2EQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1CjWvvJ4FdZc41sRgsQFWz4Y1ATZKgzsvDcUTmBSrCC15ekCmF, address:185.167.204.205:4003): protocol: rpc: dial tcp 185.167.204.205:4003: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296100945 (56h32m27.758193418s) [noise=true , address=185.167.204.205:4032 , piece_id=AN4YMHK6NKM5RIWAZHX2FPBGQ3VRFRYBKHBPIAL2HQWBK5XXNNOA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12558dx7zMNDfabV55vSsP9XwdxXgoPM8aHW53femVZzkCRxsKN, address:185.167.204.205:4032): protocol: rpc: dial tcp 185.167.204.205:4032: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296150660 (56h32m27.758243133s) [noise=true , address=185.167.204.205:4029 , piece_id=UVEMQY4WGBKGGT75ILTBPUCFZ4U252QGPDTBPT2BNMOR3V5XYCPA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12DyF12w2JXdkGLfLx9VE1E5f28VnMg7M2b3osfsDtGX9o8Tvsn, address:185.167.204.205:4029): protocol: write tcp 188.166.225.170:36604->185.167.204.205:4029: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296276239 (56h32m27.758368722s) [noise=true , address=185.167.204.206:4069 , piece_id=VKW6XTWPZPD3MDANS3FIYYD2UPDW47WBCGLNTOVYVTWA5IT23DBQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12Uan1TT5oAM3axhaviVFbvh83gThiZvwHMwbCD31ZZens6BQ4U, address:185.167.204.206:4069): protocol: rpc: dial tcp 185.167.204.206:4069: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296438448 (56h32m27.758530931s) [noise=true , address=185.167.204.206:4048 , piece_id=PRBNW47DU6K26GYUUUZSYTYQO3GNFC6DEV2OYQDP4R2RV5L5VNXA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1sY4wHBkSVUDWWg8T8Vm1hpRbbpb3ttqnqDVY8uWyG27gqBwT9, address:185.167.204.206:4048): protocol: write tcp 188.166.225.170:59140->185.167.204.206:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296628780 (56h32m27.758721253s) [noise=true , address=185.167.204.206:4055 , piece_id=KH2E6T37MHNJJ56CR3EIKMEUEWMAVP2XNMZ7K5QVKCFZJIVXRTUA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12CYRieezm1mjtGchWdPrB2mNaHVRvTS85umjB6cQyYKnvisXSB, address:185.167.204.206:4055): protocol: rpc: dial tcp 185.167.204.206:4055: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.297155593 (56h32m27.759248076s) [noise=true , address=185.167.204.205:4022 , piece_id=VU4625JC4A4ZVXFGJZTDMDZXYCNXJ67OQTEKWRLMMHGHPEHY5J7A, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj, address:185.167.204.205:4022): protocol: write tcp 188.166.225.170:57938->185.167.204.205:4022: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.297184969 (56h32m27.759277442s) [seg_pos={0 0}, upload=49746]: Done waiting for segment. successful: 12 optimal: 26 errs: <nil>
Sep 25 22:03:42.465979471 (56h32m27.928071944s) [upload=49746]: Done uploading segments...
Sep 25 22:03:42.466010200 (56h32m27.928102673s) [upload=49746]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 12)
Duration: 4.821047917s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.eu1.gateway)antiblip:
1 out of 60 is failed (threshold=3) [...............................................X............] duration:
176.366909ms remote:
"tcp:207.211.208.129:443" response headers:
Accept-Ranges=bytes Content-Length=352 Content-Security-Policy=block-all-mixed-content Server=Storj X-Request-Id=JdrtmpHLrtw X-Xss-Protection=1; mode=block Content-Type=application/xml Vary=Origin X-Amz-Request-Id=1868A6DD9BAF33BB Date=Thu, 25 Sep 2025 22:25:25 GMT target:
https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318
FAIL (2025-09-25 22:33:54 UTC)
orbiter warning: unexpected status code "500 Internal Server Error" "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>InternalError</Code><Message>We encountered an internal error, please try again.: cause(uplink: metaclient: context canceled)</Message><Key>hello-world.txt</Key><BucketName>uptime-checks</BucketName><Resource>/hello-world.txt</Resource><RequestId>1868A6DD9BAF33BB</RequestId><HostId></HostId></Error>"
Duration: 176.405423ms
edge
gateway
vhost
bgp
eu1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
30.242795706s
FAIL (2025-09-25 22:33:50 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Sep 25 22:33:20.609515379 (57h2m6.071607862s) [upload=50198]: Starting upload
Sep 25 22:33:20.609634665 (57h2m6.071727138s) [upload=50198]: Done starting upload
Sep 25 22:33:20.609747009 (57h2m6.071839492s) [upload=50198]: Uploading segments...
Sep 25 22:33:20.609932132 (57h2m6.072024605s) [upload=50198]: Got next segment. Inline: false
Sep 25 22:33:20.609977649 (57h2m6.072070132s) [upload=50198]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 25 22:33:20.609983990 (57h2m6.072076473s) [upload=50198]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 25 22:33:20.876293406 (57h2m6.338385889s) [seg_pos={0 0}, upload=50198]: Begin upload segment...
Sep 25 22:33:20.876365564 (57h2m6.338458047s) [seg_pos={0 0}, upload=50198]: Done begin upload segment.
Sep 25 22:33:20.876398446 (57h2m6.338490919s) [noise=true , address=45.91.116.7:4010 , piece_id=QZEN3HHQVP5SQ3ABXX6AUG2T7V2JTJYLKXPEM2FCIWEIRSZ2IOUA, seg_pos={0 0}, upload=50198]: Uploading piece...
Sep 25 22:33:20.876412874 (57h2m6.338505347s) [noise=true , address=145.79.227.11:4729 , piece_id=WTDDZ72XM433FAERK56RR7BWIFEFFMKJ7PKKIQQUCLWZTTTTB5TA, seg_pos={0 0}, upload=50198]: Uploading piece...
Sep 25 22:33:20.876460024 (57h2m6.338552507s) [noise=true , address=145.79.227.2:4115 , piece_id=GDDMOKVGUSPCUNXCYSADTSDEWV3WX6YLR3HSMC72AI7345WMWI6Q, seg_pos={0 0}, upload=50198]: Uploading piece...
Sep 25 22:33:20.876711823 (57h2m6.338804296s) [noise=true , address=145.79.227.4:4185 , piece_id=UDAN2TY4RX3ATTJAYBL5G3ZXKLYIXFIBPYJWRIG2MPD3LJH62UGA, seg_pos={0 0}, upload=50198]: Uploading piece...
Sep 25 22:33:20.877150188 (57h2m6.339242661s) [noise=true , address=145.79.227.6:4257 , piece_id=NLAZ3I5A6RU3PWBBSMICAETTWZ7UZLO6X5VEXUZPZ2ZB4KAI6I5A, seg_pos={0 0}, upload=50198]: Uploading piece...
Sep 25 22:33:20.877621355 (57h2m6.339713838s) [noise=true , address=145.79.227.17:4972 , piece_id=OLLDZMGRQBTZAQURMSU4AVHOAFCD3VBWCH6Y5WQXENHW2WB3D6BQ, seg_pos={0 0}, upload=50198]: Uploading piece...
Sep 25 22:33:20.877942707 (57h2m6.340035181s) [noise=true , address=145.79.227.18:4007 , piece_id=OWW73ZTC3OLJTGC4IWXZ44VT4ERGUE4YGGULZCSJKYEHNXRNLLRA, seg_pos={0 0}, upload=50198]: Uploading piece...
Sep 25 22:33:20.878252356 (57h2m6.340344829s) [noise=true , address=45.91.116.4:4939 , piece_id=7UY722GYY34MQHO6FCVNCMXKV67RUX2TTNXGUXGGDEBE63HPI6DQ, seg_pos={0 0}, upload=50198]: Uploading piece...
Sep 25 22:33:20.878573619 (57h2m6.340666092s) [noise=true , address=145.79.227.19:4079 , piece_id=TSXXBPAF2KFTUP2MDC6NKGLP2PF666PMSXPGYYKDPO25FRF7YQ6Q, seg_pos={0 0}, upload=50198]: Uploading piece...
Sep 25 22:33:20.881006501 (57h2m6.343098975s) [noise=true , address=45.91.116.10:4082 , piece_id=OJUG2YVVDOHECUD3DLDJANOWGXKS76BD3KIWFF5Q2OTRELONDWUQ, seg_pos={0 0}, upload=50198]: Uploading piece...
Sep 25 22:33:20.882423098 (57h2m6.344515571s) [noise=true , address=145.79.227.13:4802 , piece_id=UNCMDLJ7BWSS2SOAXZRUYR7EQTKZWOYFHJSVB22DHNH32VADTPNA, seg_pos={0 0}, upload=50198]: Uploading piece...
Sep 25 22:33:20.882592721 (57h2m6.344685194s) [noise=true , address=145.79.227.6:4259 , piece_id=VFXMPOPGL2UF6D6NLM7PW3GKM5YEQHT3ZDMZGOR5VEX3NNLOAXAQ, seg_pos={0 0}, upload=50198]: Uploading piece...
Sep 25 22:33:20.885196980 (57h2m6.347289464s) [noise=true , address=145.79.227.3:4151 , piece_id=ZMJ7WBX4DWOWTTJTLJXFZKBR4MRE2RS2FQWRM2YZUC6NHVEAUZFA, seg_pos={0 0}, upload=50198]: Uploading piece...
Sep 25 22:33:20.888905401 (57h2m6.350997874s) [noise=true , address=145.79.227.15:4874 , piece_id=I5VJYG36ZG63ZRJUC2WKXISA5C7EJQSV3XZJ7FLN5MWMC3CDBNEA, seg_pos={0 0}, upload=50198]: Uploading piece...
Sep 25 22:33:20.891203167 (57h2m6.35329565s) [noise=true , address=77.90.31.15:4697 , piece_id=RRP4PCTAVSJYH377LRZ43I36D4LWG7EPHE23LDBUZLP2DUR3RNDQ, seg_pos={0 0}, upload=50198]: Uploading piece...
Sep 25 22:33:20.892678096 (57h2m6.354770569s) [noise=true , address=145.79.227.15:4873 , piece_id=C3JJ5HPO237IQRGLKQ5QZSKEJM2HNOH4YNBMULOE7CXLFO4UEW3Q, seg_pos={0 0}, upload=50198]: Uploading piece...
Sep 25 22:33:22.883631748 (57h2m8.345724231s) [upload=50198]: Next returned nil segment
Sep 25 22:33:22.883655924 (57h2m8.345748397s) [upload=50198]: Waiting for error group managing segments...
Sep 25 22:33:23.186669559 (57h2m8.648762032s) [noise=true , address=77.90.31.15:4697 , piece_id=RRP4PCTAVSJYH377LRZ43I36D4LWG7EPHE23LDBUZLP2DUR3RNDQ, seg_pos={0 0}, upload=50198]: Done uploading piece. err: <nil>
Sep 25 22:33:23.402505165 (57h2m8.864597638s) [noise=true , address=45.91.116.7:4010 , piece_id=QZEN3HHQVP5SQ3ABXX6AUG2T7V2JTJYLKXPEM2FCIWEIRSZ2IOUA, seg_pos={0 0}, upload=50198]: Done uploading piece. err: <nil>
Sep 25 22:33:23.460847597 (57h2m8.92294007s) [noise=true , address=45.91.116.4:4939 , piece_id=7UY722GYY34MQHO6FCVNCMXKV67RUX2TTNXGUXGGDEBE63HPI6DQ, seg_pos={0 0}, upload=50198]: Done uploading piece. err: <nil>
Sep 25 22:33:23.475828172 (57h2m8.937920645s) [noise=true , address=45.91.116.10:4082 , piece_id=OJUG2YVVDOHECUD3DLDJANOWGXKS76BD3KIWFF5Q2OTRELONDWUQ, seg_pos={0 0}, upload=50198]: Done uploading piece. err: <nil>
Sep 25 22:33:41.316700325 (57h2m26.778792808s) [noise=true , address=145.79.227.18:4007 , piece_id=OWW73ZTC3OLJTGC4IWXZ44VT4ERGUE4YGGULZCSJKYEHNXRNLLRA, seg_pos={0 0}, upload=50198]: Done uploading piece. err: <nil>
Sep 25 22:33:42.134617358 (57h2m27.596709841s) [noise=true , address=145.79.227.11:4729 , piece_id=WTDDZ72XM433FAERK56RR7BWIFEFFMKJ7PKKIQQUCLWZTTTTB5TA, seg_pos={0 0}, upload=50198]: Done uploading piece. err: <nil>
Sep 25 22:33:43.316084677 (57h2m28.77817716s) [noise=true , address=145.79.227.15:4874 , piece_id=I5VJYG36ZG63ZRJUC2WKXISA5C7EJQSV3XZJ7FLN5MWMC3CDBNEA, seg_pos={0 0}, upload=50198]: Done uploading piece. err: <nil>
Sep 25 22:33:43.336895665 (57h2m28.798988148s) [noise=true , address=145.79.227.15:4873 , piece_id=C3JJ5HPO237IQRGLKQ5QZSKEJM2HNOH4YNBMULOE7CXLFO4UEW3Q, seg_pos={0 0}, upload=50198]: Done uploading piece. err: <nil>
Sep 25 22:33:44.766574920 (57h2m30.228667393s) [noise=true , address=145.79.227.17:4972 , piece_id=OLLDZMGRQBTZAQURMSU4AVHOAFCD3VBWCH6Y5WQXENHW2WB3D6BQ, seg_pos={0 0}, upload=50198]: Done uploading piece. err: <nil>
Sep 25 22:33:47.245353345 (57h2m32.707445818s) [noise=true , address=145.79.227.13:4802 , piece_id=UNCMDLJ7BWSS2SOAXZRUYR7EQTKZWOYFHJSVB22DHNH32VADTPNA, seg_pos={0 0}, upload=50198]: Done uploading piece. err: <nil>
Sep 25 22:33:48.389979384 (57h2m33.852071867s) [noise=true , address=145.79.227.19:4079 , piece_id=TSXXBPAF2KFTUP2MDC6NKGLP2PF666PMSXPGYYKDPO25FRF7YQ6Q, seg_pos={0 0}, upload=50198]: Done uploading piece. err: <nil>
Sep 25 22:33:49.870866176 (57h2m35.332958649s) [noise=true , address=145.79.227.3:4151 , piece_id=ZMJ7WBX4DWOWTTJTLJXFZKBR4MRE2RS2FQWRM2YZUC6NHVEAUZFA, seg_pos={0 0}, upload=50198]: Done uploading piece. err: <nil>
Sep 25 22:33:49.927808223 (57h2m35.389900706s) [noise=true , address=145.79.227.2:4115 , piece_id=GDDMOKVGUSPCUNXCYSADTSDEWV3WX6YLR3HSMC72AI7345WMWI6Q, seg_pos={0 0}, upload=50198]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: expected piece hash; context deadline exceeded; EOF
Sep 25 22:33:49.927871833 (57h2m35.389964316s) [noise=true , address=145.79.227.6:4259 , piece_id=VFXMPOPGL2UF6D6NLM7PW3GKM5YEQHT3ZDMZGOR5VEX3NNLOAXAQ, seg_pos={0 0}, upload=50198]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: expected piece hash; context deadline exceeded; EOF
Sep 25 22:33:49.927907451 (57h2m35.389999924s) [noise=true , address=145.79.227.4:4185 , piece_id=UDAN2TY4RX3ATTJAYBL5G3ZXKLYIXFIBPYJWRIG2MPD3LJH62UGA, seg_pos={0 0}, upload=50198]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:145.79.227.4:4185): protocol: write tcp 188.166.225.170:46508->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:46508->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46508->145.79.227.4:4185: use of closed network connection
Sep 25 22:33:49.927937458 (57h2m35.390029931s) [noise=true , address=145.79.227.6:4257 , piece_id=NLAZ3I5A6RU3PWBBSMICAETTWZ7UZLO6X5VEXUZPZ2ZB4KAI6I5A, seg_pos={0 0}, upload=50198]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: expected piece hash; context deadline exceeded; EOF
Sep 25 22:33:49.927971413 (57h2m35.390063886s) [seg_pos={0 0}, upload=50198]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Sep 25 22:33:50.167900726 (57h2m35.629993199s) [upload=50198]: Done uploading segments...
Sep 25 22:33:50.167951733 (57h2m35.630044216s) [upload=50198]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 30.242827166s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
2 out of 60 is failed (threshold=3) [..............XX............................................] duration:
4.808123365s
FAIL (2025-09-25 22:33:45 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 12)
======LOGS======
9 , piece_id=4YT7BXXDQVTR2QL4YBWVMO3YJCZQ57PTTEQW7GCZELWC42HQUDQQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.227500377 (56h31m29.68959285s) [noise=true , address=81.171.14.70:4034 , piece_id=EXVA3SFFPD3GJDYIFHAOP7GMJQHGZECMX5R6MHLRDMMCU7LOA7KA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.253492519 (56h31m29.715584993s) [noise=true , address=81.171.14.70:4028 , piece_id=Y237IBPHRRI7NMOYJG65AZI3LCCZ6JUJPX5WWGIDX5RYHCNVIG3A, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.267890396 (56h31m29.729982869s) [noise=true , address=89.149.201.147:4010 , piece_id=ANMXRI22IXAZFHKB7ZHFGEERKT2KGUYQEFKEOJNJDC5TFB6B3STQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.280031865 (56h31m29.742124348s) [noise=true , address=89.149.201.147:4020 , piece_id=RML2JG4QZSS6H5GTP7BW3B4SFVZLLHHAOUS3LDCOT2BGYTC3HACA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.294783955 (56h31m29.756876428s) [noise=true , address=89.149.201.147:4017 , piece_id=P576E6UE5UADQZYKR4XN4I5X4MCGMXT3B7PBGVNWBZ6I67ZVCJHA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.462969447 (56h31m29.92506192s) [noise=true , address=81.171.14.70:4035 , piece_id=GWVM3MZRGRVJ5RWG227JJXTULF5MUJ4FHWCNHLW7J3GDYXRSXRXA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.668506081 (56h31m30.130598554s) [noise=true , address=81.171.14.70:4045 , piece_id=BSKEEYML4LQZ7PWB45HQNX2YFJTFIEKIILDUZIWCXMVZZGGKHGUQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.702001372 (56h31m30.164093845s) [noise=true , address=89.149.201.147:4014 , piece_id=3VH5QXDWYX2NR3LODHSATJCP5ABF7BI52GJAR3VLPW42CPJJXQNA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.728242188 (56h31m30.190334661s) [noise=true , address=89.149.201.147:4016 , piece_id=PT5TUJDTS2TDBNLH5722GJTZ5BIO4LZKEDKBGUKHEF37TPADW7MA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.797046563 (56h31m30.259139036s) [noise=true , address=89.149.201.147:4012 , piece_id=TY6OVHC4ITI2E32GE7LP3O2YQ46FKWF6A6IIBN4JFAB7WPYXXJ2A, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:03:42.295273049 (56h32m27.757365543s) [noise=true , address=185.167.204.206:4056 , piece_id=4XAHI4JY2MPISEOZ6VJ5GCJ5WZJKHRXIDK2EQGEN7VSYNRYJH24A, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1uSTuNA28oH2s5Rkv2jaKoC2nZWSgpMJXyxwFTCKbr44EvEXrC, address:185.167.204.206:4056): protocol: rpc: dial tcp 185.167.204.206:4056: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295296423 (56h32m27.757388906s) [noise=true , address=185.167.204.205:4008 , piece_id=NXU5IGU2NUZPVR5KIUKLZ7QTPYLGDIKZQNMPBJRPCVYEVNIV7HMA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12ZHTTaGQUTftxEULFqWbH6w1HnWA7PavHrXnvMCD947FxtmsrS, address:185.167.204.205:4008): protocol: rpc: dial tcp 185.167.204.205:4008: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295427543 (56h32m27.757520026s) [noise=true , address=185.167.204.206:4064 , piece_id=7GVKAUOISAORFKOXUNOJMEEFNNUVVZUQ2RGCD6BNVCDTH6ONPQIQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1ZB3emcQntFxpr7zBiE98eesnjoTnNKinw3FPtGU2TgznGakeb, address:185.167.204.206:4064): protocol: rpc: dial tcp 185.167.204.206:4064: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295495543 (56h32m27.757588026s) [noise=true , address=185.167.204.206:4059 , piece_id=G4B2XQYXOG2LWBAIYBM4BWLI7PEPE2XDM5A57YHYR6SOWW7ODSVA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:121gTedy8TPx44wmLqdQpZ1gDDsL8D9K1EvQJffR7oKY3bT9DXh, address:185.167.204.206:4059): protocol: rpc: dial tcp 185.167.204.206:4059: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295562701 (56h32m27.757655184s) [noise=true , address=185.167.204.205:4030 , piece_id=U6SOG6BNEIWDGYWDRZOCLTROZVZXPPG5ZI52HLRPKTDKFVQN3EJA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12w5R2Whxgg4AYNNvTkKmFq4pzahF1uZiVzwgVygc9iZDAXMdSk, address:185.167.204.205:4030): protocol: rpc: dial tcp 185.167.204.205:4030: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295700453 (56h32m27.757792926s) [noise=true , address=185.167.204.205:4010 , piece_id=DPMSKQDDOU2PUXHINTG7JJXLKXJE2PC6B3KNSKN5UIGAKBM3GZVQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12oTu1GaRsH7fHqxLkQ9c9ikjG4zRN3NChssa5gfpLwrcEehoyV, address:185.167.204.205:4010): protocol: rpc: dial tcp 185.167.204.205:4010: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295713257 (56h32m27.75780574s) [noise=true , address=185.167.204.206:4053 , piece_id=74G7LWVYE7GO4GPSG67BD6QVE7UBLGIAKLSG447PWWLNBM2FZNTA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12SrFttJpndpv7TKJJYsKthWnGWBYQGHpS9CwqTd4mqBP9NCCm8, address:185.167.204.206:4053): protocol: rpc: dial tcp 185.167.204.206:4053: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295852051 (56h32m27.757944534s) [noise=true , address=185.167.204.206:4065 , piece_id=VKBLX4ASIKVHMUUCTD7GFFPEHJLKREGCDGI7LCVRSASDHBAWI6AQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12oPo26KZGZr7o7M18KZiX5PMAAMwHK8FPT59y1hSLhkB6z3Lcj, address:185.167.204.206:4065): protocol: rpc: dial tcp 185.167.204.206:4065: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295908840 (56h32m27.758001313s) [noise=true , address=185.167.204.206:4057 , piece_id=WNX7U6DN7HTE3XLVJAJ4FTCYFR3B2MLMLF5I46GALE463DWQ22WQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1xy38NywWy5CSJ7CfupY78vbkHg6cQzgNhM2Z1xnpvw9dwvMoT, address:185.167.204.206:4057): protocol: rpc: dial tcp 185.167.204.206:4057: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296027916 (56h32m27.758120399s) [noise=true , address=185.167.204.205:4003 , piece_id=NQX2I3KTMYFLS2B2C5Y2PGZRXCD7QJCZLABY4PTYVEFGAVCYX2EQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1CjWvvJ4FdZc41sRgsQFWz4Y1ATZKgzsvDcUTmBSrCC15ekCmF, address:185.167.204.205:4003): protocol: rpc: dial tcp 185.167.204.205:4003: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296100945 (56h32m27.758193418s) [noise=true , address=185.167.204.205:4032 , piece_id=AN4YMHK6NKM5RIWAZHX2FPBGQ3VRFRYBKHBPIAL2HQWBK5XXNNOA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12558dx7zMNDfabV55vSsP9XwdxXgoPM8aHW53femVZzkCRxsKN, address:185.167.204.205:4032): protocol: rpc: dial tcp 185.167.204.205:4032: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296150660 (56h32m27.758243133s) [noise=true , address=185.167.204.205:4029 , piece_id=UVEMQY4WGBKGGT75ILTBPUCFZ4U252QGPDTBPT2BNMOR3V5XYCPA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12DyF12w2JXdkGLfLx9VE1E5f28VnMg7M2b3osfsDtGX9o8Tvsn, address:185.167.204.205:4029): protocol: write tcp 188.166.225.170:36604->185.167.204.205:4029: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296276239 (56h32m27.758368722s) [noise=true , address=185.167.204.206:4069 , piece_id=VKW6XTWPZPD3MDANS3FIYYD2UPDW47WBCGLNTOVYVTWA5IT23DBQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12Uan1TT5oAM3axhaviVFbvh83gThiZvwHMwbCD31ZZens6BQ4U, address:185.167.204.206:4069): protocol: rpc: dial tcp 185.167.204.206:4069: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296438448 (56h32m27.758530931s) [noise=true , address=185.167.204.206:4048 , piece_id=PRBNW47DU6K26GYUUUZSYTYQO3GNFC6DEV2OYQDP4R2RV5L5VNXA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1sY4wHBkSVUDWWg8T8Vm1hpRbbpb3ttqnqDVY8uWyG27gqBwT9, address:185.167.204.206:4048): protocol: write tcp 188.166.225.170:59140->185.167.204.206:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296628780 (56h32m27.758721253s) [noise=true , address=185.167.204.206:4055 , piece_id=KH2E6T37MHNJJ56CR3EIKMEUEWMAVP2XNMZ7K5QVKCFZJIVXRTUA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12CYRieezm1mjtGchWdPrB2mNaHVRvTS85umjB6cQyYKnvisXSB, address:185.167.204.206:4055): protocol: rpc: dial tcp 185.167.204.206:4055: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.297155593 (56h32m27.759248076s) [noise=true , address=185.167.204.205:4022 , piece_id=VU4625JC4A4ZVXFGJZTDMDZXYCNXJ67OQTEKWRLMMHGHPEHY5J7A, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj, address:185.167.204.205:4022): protocol: write tcp 188.166.225.170:57938->185.167.204.205:4022: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.297184969 (56h32m27.759277442s) [seg_pos={0 0}, upload=49746]: Done waiting for segment. successful: 12 optimal: 26 errs: <nil>
Sep 25 22:03:42.465979471 (56h32m27.928071944s) [upload=49746]: Done uploading segments...
Sep 25 22:03:42.466010200 (56h32m27.928102673s) [upload=49746]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 12)
Duration: 4.808161096s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.eu1.gateway)antiblip:
1 out of 60 is failed (threshold=3) [................................................X...........] duration:
175.518344ms remote:
"tcp:207.211.208.129:443" response headers:
Accept-Ranges=bytes Content-Length=352 Content-Security-Policy=block-all-mixed-content Server=Storj X-Request-Id=JdrtmpHLrtw X-Xss-Protection=1; mode=block Content-Type=application/xml Vary=Origin X-Amz-Request-Id=1868A6DD9BAF33BB Date=Thu, 25 Sep 2025 22:25:25 GMT target:
https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318
FAIL (2025-09-25 22:33:32 UTC)
orbiter warning: unexpected status code "500 Internal Server Error" "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>InternalError</Code><Message>We encountered an internal error, please try again.: cause(uplink: metaclient: context canceled)</Message><Key>hello-world.txt</Key><BucketName>uptime-checks</BucketName><Resource>/hello-world.txt</Resource><RequestId>1868A6DD9BAF33BB</RequestId><HostId></HostId></Error>"
Duration: 175.546528ms
edge
gateway
vhost
bgp
eu1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
2 out of 60 is failed (threshold=3) [...............XX...........................................] duration:
5.925336002s
FAIL (2025-09-25 22:33:11 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 12)
======LOGS======
9 , piece_id=4YT7BXXDQVTR2QL4YBWVMO3YJCZQ57PTTEQW7GCZELWC42HQUDQQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.227500377 (56h31m29.68959285s) [noise=true , address=81.171.14.70:4034 , piece_id=EXVA3SFFPD3GJDYIFHAOP7GMJQHGZECMX5R6MHLRDMMCU7LOA7KA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.253492519 (56h31m29.715584993s) [noise=true , address=81.171.14.70:4028 , piece_id=Y237IBPHRRI7NMOYJG65AZI3LCCZ6JUJPX5WWGIDX5RYHCNVIG3A, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.267890396 (56h31m29.729982869s) [noise=true , address=89.149.201.147:4010 , piece_id=ANMXRI22IXAZFHKB7ZHFGEERKT2KGUYQEFKEOJNJDC5TFB6B3STQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.280031865 (56h31m29.742124348s) [noise=true , address=89.149.201.147:4020 , piece_id=RML2JG4QZSS6H5GTP7BW3B4SFVZLLHHAOUS3LDCOT2BGYTC3HACA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.294783955 (56h31m29.756876428s) [noise=true , address=89.149.201.147:4017 , piece_id=P576E6UE5UADQZYKR4XN4I5X4MCGMXT3B7PBGVNWBZ6I67ZVCJHA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.462969447 (56h31m29.92506192s) [noise=true , address=81.171.14.70:4035 , piece_id=GWVM3MZRGRVJ5RWG227JJXTULF5MUJ4FHWCNHLW7J3GDYXRSXRXA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.668506081 (56h31m30.130598554s) [noise=true , address=81.171.14.70:4045 , piece_id=BSKEEYML4LQZ7PWB45HQNX2YFJTFIEKIILDUZIWCXMVZZGGKHGUQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.702001372 (56h31m30.164093845s) [noise=true , address=89.149.201.147:4014 , piece_id=3VH5QXDWYX2NR3LODHSATJCP5ABF7BI52GJAR3VLPW42CPJJXQNA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.728242188 (56h31m30.190334661s) [noise=true , address=89.149.201.147:4016 , piece_id=PT5TUJDTS2TDBNLH5722GJTZ5BIO4LZKEDKBGUKHEF37TPADW7MA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.797046563 (56h31m30.259139036s) [noise=true , address=89.149.201.147:4012 , piece_id=TY6OVHC4ITI2E32GE7LP3O2YQ46FKWF6A6IIBN4JFAB7WPYXXJ2A, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:03:42.295273049 (56h32m27.757365543s) [noise=true , address=185.167.204.206:4056 , piece_id=4XAHI4JY2MPISEOZ6VJ5GCJ5WZJKHRXIDK2EQGEN7VSYNRYJH24A, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1uSTuNA28oH2s5Rkv2jaKoC2nZWSgpMJXyxwFTCKbr44EvEXrC, address:185.167.204.206:4056): protocol: rpc: dial tcp 185.167.204.206:4056: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295296423 (56h32m27.757388906s) [noise=true , address=185.167.204.205:4008 , piece_id=NXU5IGU2NUZPVR5KIUKLZ7QTPYLGDIKZQNMPBJRPCVYEVNIV7HMA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12ZHTTaGQUTftxEULFqWbH6w1HnWA7PavHrXnvMCD947FxtmsrS, address:185.167.204.205:4008): protocol: rpc: dial tcp 185.167.204.205:4008: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295427543 (56h32m27.757520026s) [noise=true , address=185.167.204.206:4064 , piece_id=7GVKAUOISAORFKOXUNOJMEEFNNUVVZUQ2RGCD6BNVCDTH6ONPQIQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1ZB3emcQntFxpr7zBiE98eesnjoTnNKinw3FPtGU2TgznGakeb, address:185.167.204.206:4064): protocol: rpc: dial tcp 185.167.204.206:4064: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295495543 (56h32m27.757588026s) [noise=true , address=185.167.204.206:4059 , piece_id=G4B2XQYXOG2LWBAIYBM4BWLI7PEPE2XDM5A57YHYR6SOWW7ODSVA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:121gTedy8TPx44wmLqdQpZ1gDDsL8D9K1EvQJffR7oKY3bT9DXh, address:185.167.204.206:4059): protocol: rpc: dial tcp 185.167.204.206:4059: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295562701 (56h32m27.757655184s) [noise=true , address=185.167.204.205:4030 , piece_id=U6SOG6BNEIWDGYWDRZOCLTROZVZXPPG5ZI52HLRPKTDKFVQN3EJA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12w5R2Whxgg4AYNNvTkKmFq4pzahF1uZiVzwgVygc9iZDAXMdSk, address:185.167.204.205:4030): protocol: rpc: dial tcp 185.167.204.205:4030: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295700453 (56h32m27.757792926s) [noise=true , address=185.167.204.205:4010 , piece_id=DPMSKQDDOU2PUXHINTG7JJXLKXJE2PC6B3KNSKN5UIGAKBM3GZVQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12oTu1GaRsH7fHqxLkQ9c9ikjG4zRN3NChssa5gfpLwrcEehoyV, address:185.167.204.205:4010): protocol: rpc: dial tcp 185.167.204.205:4010: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295713257 (56h32m27.75780574s) [noise=true , address=185.167.204.206:4053 , piece_id=74G7LWVYE7GO4GPSG67BD6QVE7UBLGIAKLSG447PWWLNBM2FZNTA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12SrFttJpndpv7TKJJYsKthWnGWBYQGHpS9CwqTd4mqBP9NCCm8, address:185.167.204.206:4053): protocol: rpc: dial tcp 185.167.204.206:4053: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295852051 (56h32m27.757944534s) [noise=true , address=185.167.204.206:4065 , piece_id=VKBLX4ASIKVHMUUCTD7GFFPEHJLKREGCDGI7LCVRSASDHBAWI6AQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12oPo26KZGZr7o7M18KZiX5PMAAMwHK8FPT59y1hSLhkB6z3Lcj, address:185.167.204.206:4065): protocol: rpc: dial tcp 185.167.204.206:4065: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295908840 (56h32m27.758001313s) [noise=true , address=185.167.204.206:4057 , piece_id=WNX7U6DN7HTE3XLVJAJ4FTCYFR3B2MLMLF5I46GALE463DWQ22WQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1xy38NywWy5CSJ7CfupY78vbkHg6cQzgNhM2Z1xnpvw9dwvMoT, address:185.167.204.206:4057): protocol: rpc: dial tcp 185.167.204.206:4057: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296027916 (56h32m27.758120399s) [noise=true , address=185.167.204.205:4003 , piece_id=NQX2I3KTMYFLS2B2C5Y2PGZRXCD7QJCZLABY4PTYVEFGAVCYX2EQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1CjWvvJ4FdZc41sRgsQFWz4Y1ATZKgzsvDcUTmBSrCC15ekCmF, address:185.167.204.205:4003): protocol: rpc: dial tcp 185.167.204.205:4003: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296100945 (56h32m27.758193418s) [noise=true , address=185.167.204.205:4032 , piece_id=AN4YMHK6NKM5RIWAZHX2FPBGQ3VRFRYBKHBPIAL2HQWBK5XXNNOA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12558dx7zMNDfabV55vSsP9XwdxXgoPM8aHW53femVZzkCRxsKN, address:185.167.204.205:4032): protocol: rpc: dial tcp 185.167.204.205:4032: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296150660 (56h32m27.758243133s) [noise=true , address=185.167.204.205:4029 , piece_id=UVEMQY4WGBKGGT75ILTBPUCFZ4U252QGPDTBPT2BNMOR3V5XYCPA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12DyF12w2JXdkGLfLx9VE1E5f28VnMg7M2b3osfsDtGX9o8Tvsn, address:185.167.204.205:4029): protocol: write tcp 188.166.225.170:36604->185.167.204.205:4029: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296276239 (56h32m27.758368722s) [noise=true , address=185.167.204.206:4069 , piece_id=VKW6XTWPZPD3MDANS3FIYYD2UPDW47WBCGLNTOVYVTWA5IT23DBQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12Uan1TT5oAM3axhaviVFbvh83gThiZvwHMwbCD31ZZens6BQ4U, address:185.167.204.206:4069): protocol: rpc: dial tcp 185.167.204.206:4069: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296438448 (56h32m27.758530931s) [noise=true , address=185.167.204.206:4048 , piece_id=PRBNW47DU6K26GYUUUZSYTYQO3GNFC6DEV2OYQDP4R2RV5L5VNXA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1sY4wHBkSVUDWWg8T8Vm1hpRbbpb3ttqnqDVY8uWyG27gqBwT9, address:185.167.204.206:4048): protocol: write tcp 188.166.225.170:59140->185.167.204.206:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296628780 (56h32m27.758721253s) [noise=true , address=185.167.204.206:4055 , piece_id=KH2E6T37MHNJJ56CR3EIKMEUEWMAVP2XNMZ7K5QVKCFZJIVXRTUA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12CYRieezm1mjtGchWdPrB2mNaHVRvTS85umjB6cQyYKnvisXSB, address:185.167.204.206:4055): protocol: rpc: dial tcp 185.167.204.206:4055: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.297155593 (56h32m27.759248076s) [noise=true , address=185.167.204.205:4022 , piece_id=VU4625JC4A4ZVXFGJZTDMDZXYCNXJ67OQTEKWRLMMHGHPEHY5J7A, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj, address:185.167.204.205:4022): protocol: write tcp 188.166.225.170:57938->185.167.204.205:4022: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.297184969 (56h32m27.759277442s) [seg_pos={0 0}, upload=49746]: Done waiting for segment. successful: 12 optimal: 26 errs: <nil>
Sep 25 22:03:42.465979471 (56h32m27.928071944s) [upload=49746]: Done uploading segments...
Sep 25 22:03:42.466010200 (56h32m27.928102673s) [upload=49746]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 12)
Duration: 5.925359878s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.eu1.gateway)antiblip:
1 out of 60 is failed (threshold=3) [.................................................X..........] duration:
179.716209ms remote:
"tcp:207.211.208.129:443" response headers:
Accept-Ranges=bytes Content-Length=352 Content-Security-Policy=block-all-mixed-content Server=Storj X-Request-Id=JdrtmpHLrtw X-Xss-Protection=1; mode=block Content-Type=application/xml Vary=Origin X-Amz-Request-Id=1868A6DD9BAF33BB Date=Thu, 25 Sep 2025 22:25:25 GMT target:
https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318
FAIL (2025-09-25 22:32:27 UTC)
orbiter warning: unexpected status code "500 Internal Server Error" "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>InternalError</Code><Message>We encountered an internal error, please try again.: cause(uplink: metaclient: context canceled)</Message><Key>hello-world.txt</Key><BucketName>uptime-checks</BucketName><Resource>/hello-world.txt</Resource><RequestId>1868A6DD9BAF33BB</RequestId><HostId></HostId></Error>"
Duration: 179.74295ms
edge
gateway
vhost
bgp
eu1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
2 out of 60 is failed (threshold=3) [................XX..........................................] duration:
4.871360474s
FAIL (2025-09-25 22:31:58 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 12)
======LOGS======
9 , piece_id=4YT7BXXDQVTR2QL4YBWVMO3YJCZQ57PTTEQW7GCZELWC42HQUDQQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.227500377 (56h31m29.68959285s) [noise=true , address=81.171.14.70:4034 , piece_id=EXVA3SFFPD3GJDYIFHAOP7GMJQHGZECMX5R6MHLRDMMCU7LOA7KA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.253492519 (56h31m29.715584993s) [noise=true , address=81.171.14.70:4028 , piece_id=Y237IBPHRRI7NMOYJG65AZI3LCCZ6JUJPX5WWGIDX5RYHCNVIG3A, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.267890396 (56h31m29.729982869s) [noise=true , address=89.149.201.147:4010 , piece_id=ANMXRI22IXAZFHKB7ZHFGEERKT2KGUYQEFKEOJNJDC5TFB6B3STQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.280031865 (56h31m29.742124348s) [noise=true , address=89.149.201.147:4020 , piece_id=RML2JG4QZSS6H5GTP7BW3B4SFVZLLHHAOUS3LDCOT2BGYTC3HACA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.294783955 (56h31m29.756876428s) [noise=true , address=89.149.201.147:4017 , piece_id=P576E6UE5UADQZYKR4XN4I5X4MCGMXT3B7PBGVNWBZ6I67ZVCJHA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.462969447 (56h31m29.92506192s) [noise=true , address=81.171.14.70:4035 , piece_id=GWVM3MZRGRVJ5RWG227JJXTULF5MUJ4FHWCNHLW7J3GDYXRSXRXA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.668506081 (56h31m30.130598554s) [noise=true , address=81.171.14.70:4045 , piece_id=BSKEEYML4LQZ7PWB45HQNX2YFJTFIEKIILDUZIWCXMVZZGGKHGUQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.702001372 (56h31m30.164093845s) [noise=true , address=89.149.201.147:4014 , piece_id=3VH5QXDWYX2NR3LODHSATJCP5ABF7BI52GJAR3VLPW42CPJJXQNA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.728242188 (56h31m30.190334661s) [noise=true , address=89.149.201.147:4016 , piece_id=PT5TUJDTS2TDBNLH5722GJTZ5BIO4LZKEDKBGUKHEF37TPADW7MA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.797046563 (56h31m30.259139036s) [noise=true , address=89.149.201.147:4012 , piece_id=TY6OVHC4ITI2E32GE7LP3O2YQ46FKWF6A6IIBN4JFAB7WPYXXJ2A, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:03:42.295273049 (56h32m27.757365543s) [noise=true , address=185.167.204.206:4056 , piece_id=4XAHI4JY2MPISEOZ6VJ5GCJ5WZJKHRXIDK2EQGEN7VSYNRYJH24A, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1uSTuNA28oH2s5Rkv2jaKoC2nZWSgpMJXyxwFTCKbr44EvEXrC, address:185.167.204.206:4056): protocol: rpc: dial tcp 185.167.204.206:4056: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295296423 (56h32m27.757388906s) [noise=true , address=185.167.204.205:4008 , piece_id=NXU5IGU2NUZPVR5KIUKLZ7QTPYLGDIKZQNMPBJRPCVYEVNIV7HMA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12ZHTTaGQUTftxEULFqWbH6w1HnWA7PavHrXnvMCD947FxtmsrS, address:185.167.204.205:4008): protocol: rpc: dial tcp 185.167.204.205:4008: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295427543 (56h32m27.757520026s) [noise=true , address=185.167.204.206:4064 , piece_id=7GVKAUOISAORFKOXUNOJMEEFNNUVVZUQ2RGCD6BNVCDTH6ONPQIQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1ZB3emcQntFxpr7zBiE98eesnjoTnNKinw3FPtGU2TgznGakeb, address:185.167.204.206:4064): protocol: rpc: dial tcp 185.167.204.206:4064: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295495543 (56h32m27.757588026s) [noise=true , address=185.167.204.206:4059 , piece_id=G4B2XQYXOG2LWBAIYBM4BWLI7PEPE2XDM5A57YHYR6SOWW7ODSVA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:121gTedy8TPx44wmLqdQpZ1gDDsL8D9K1EvQJffR7oKY3bT9DXh, address:185.167.204.206:4059): protocol: rpc: dial tcp 185.167.204.206:4059: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295562701 (56h32m27.757655184s) [noise=true , address=185.167.204.205:4030 , piece_id=U6SOG6BNEIWDGYWDRZOCLTROZVZXPPG5ZI52HLRPKTDKFVQN3EJA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12w5R2Whxgg4AYNNvTkKmFq4pzahF1uZiVzwgVygc9iZDAXMdSk, address:185.167.204.205:4030): protocol: rpc: dial tcp 185.167.204.205:4030: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295700453 (56h32m27.757792926s) [noise=true , address=185.167.204.205:4010 , piece_id=DPMSKQDDOU2PUXHINTG7JJXLKXJE2PC6B3KNSKN5UIGAKBM3GZVQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12oTu1GaRsH7fHqxLkQ9c9ikjG4zRN3NChssa5gfpLwrcEehoyV, address:185.167.204.205:4010): protocol: rpc: dial tcp 185.167.204.205:4010: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295713257 (56h32m27.75780574s) [noise=true , address=185.167.204.206:4053 , piece_id=74G7LWVYE7GO4GPSG67BD6QVE7UBLGIAKLSG447PWWLNBM2FZNTA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12SrFttJpndpv7TKJJYsKthWnGWBYQGHpS9CwqTd4mqBP9NCCm8, address:185.167.204.206:4053): protocol: rpc: dial tcp 185.167.204.206:4053: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295852051 (56h32m27.757944534s) [noise=true , address=185.167.204.206:4065 , piece_id=VKBLX4ASIKVHMUUCTD7GFFPEHJLKREGCDGI7LCVRSASDHBAWI6AQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12oPo26KZGZr7o7M18KZiX5PMAAMwHK8FPT59y1hSLhkB6z3Lcj, address:185.167.204.206:4065): protocol: rpc: dial tcp 185.167.204.206:4065: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295908840 (56h32m27.758001313s) [noise=true , address=185.167.204.206:4057 , piece_id=WNX7U6DN7HTE3XLVJAJ4FTCYFR3B2MLMLF5I46GALE463DWQ22WQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1xy38NywWy5CSJ7CfupY78vbkHg6cQzgNhM2Z1xnpvw9dwvMoT, address:185.167.204.206:4057): protocol: rpc: dial tcp 185.167.204.206:4057: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296027916 (56h32m27.758120399s) [noise=true , address=185.167.204.205:4003 , piece_id=NQX2I3KTMYFLS2B2C5Y2PGZRXCD7QJCZLABY4PTYVEFGAVCYX2EQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1CjWvvJ4FdZc41sRgsQFWz4Y1ATZKgzsvDcUTmBSrCC15ekCmF, address:185.167.204.205:4003): protocol: rpc: dial tcp 185.167.204.205:4003: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296100945 (56h32m27.758193418s) [noise=true , address=185.167.204.205:4032 , piece_id=AN4YMHK6NKM5RIWAZHX2FPBGQ3VRFRYBKHBPIAL2HQWBK5XXNNOA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12558dx7zMNDfabV55vSsP9XwdxXgoPM8aHW53femVZzkCRxsKN, address:185.167.204.205:4032): protocol: rpc: dial tcp 185.167.204.205:4032: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296150660 (56h32m27.758243133s) [noise=true , address=185.167.204.205:4029 , piece_id=UVEMQY4WGBKGGT75ILTBPUCFZ4U252QGPDTBPT2BNMOR3V5XYCPA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12DyF12w2JXdkGLfLx9VE1E5f28VnMg7M2b3osfsDtGX9o8Tvsn, address:185.167.204.205:4029): protocol: write tcp 188.166.225.170:36604->185.167.204.205:4029: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296276239 (56h32m27.758368722s) [noise=true , address=185.167.204.206:4069 , piece_id=VKW6XTWPZPD3MDANS3FIYYD2UPDW47WBCGLNTOVYVTWA5IT23DBQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12Uan1TT5oAM3axhaviVFbvh83gThiZvwHMwbCD31ZZens6BQ4U, address:185.167.204.206:4069): protocol: rpc: dial tcp 185.167.204.206:4069: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296438448 (56h32m27.758530931s) [noise=true , address=185.167.204.206:4048 , piece_id=PRBNW47DU6K26GYUUUZSYTYQO3GNFC6DEV2OYQDP4R2RV5L5VNXA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1sY4wHBkSVUDWWg8T8Vm1hpRbbpb3ttqnqDVY8uWyG27gqBwT9, address:185.167.204.206:4048): protocol: write tcp 188.166.225.170:59140->185.167.204.206:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296628780 (56h32m27.758721253s) [noise=true , address=185.167.204.206:4055 , piece_id=KH2E6T37MHNJJ56CR3EIKMEUEWMAVP2XNMZ7K5QVKCFZJIVXRTUA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12CYRieezm1mjtGchWdPrB2mNaHVRvTS85umjB6cQyYKnvisXSB, address:185.167.204.206:4055): protocol: rpc: dial tcp 185.167.204.206:4055: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.297155593 (56h32m27.759248076s) [noise=true , address=185.167.204.205:4022 , piece_id=VU4625JC4A4ZVXFGJZTDMDZXYCNXJ67OQTEKWRLMMHGHPEHY5J7A, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj, address:185.167.204.205:4022): protocol: write tcp 188.166.225.170:57938->185.167.204.205:4022: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.297184969 (56h32m27.759277442s) [seg_pos={0 0}, upload=49746]: Done waiting for segment. successful: 12 optimal: 26 errs: <nil>
Sep 25 22:03:42.465979471 (56h32m27.928071944s) [upload=49746]: Done uploading segments...
Sep 25 22:03:42.466010200 (56h32m27.928102673s) [upload=49746]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 12)
Duration: 4.871387535s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.eu1.gateway)antiblip:
1 out of 60 is failed (threshold=3) [..................................................X.........] duration:
220.963189ms remote:
"tcp:207.211.208.129:443" response headers:
Accept-Ranges=bytes Content-Length=352 Content-Security-Policy=block-all-mixed-content Server=Storj X-Request-Id=JdrtmpHLrtw X-Xss-Protection=1; mode=block Content-Type=application/xml Vary=Origin X-Amz-Request-Id=1868A6DD9BAF33BB Date=Thu, 25 Sep 2025 22:25:25 GMT target:
https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318
FAIL (2025-09-25 22:31:50 UTC)
orbiter warning: unexpected status code "500 Internal Server Error" "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>InternalError</Code><Message>We encountered an internal error, please try again.: cause(uplink: metaclient: context canceled)</Message><Key>hello-world.txt</Key><BucketName>uptime-checks</BucketName><Resource>/hello-world.txt</Resource><RequestId>1868A6DD9BAF33BB</RequestId><HostId></HostId></Error>"
Duration: 221.016832ms
edge
gateway
vhost
bgp
eu1
|
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.eu1.gateway)antiblip:
1 out of 60 is failed (threshold=3) [...................................................X........] duration:
175.404809ms remote:
"tcp:207.211.208.129:443" response headers:
Accept-Ranges=bytes Content-Length=352 Content-Security-Policy=block-all-mixed-content Server=Storj X-Request-Id=JdrtmpHLrtw X-Xss-Protection=1; mode=block Content-Type=application/xml Vary=Origin X-Amz-Request-Id=1868A6DD9BAF33BB Date=Thu, 25 Sep 2025 22:25:25 GMT target:
https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318
FAIL (2025-09-25 22:31:13 UTC)
orbiter warning: unexpected status code "500 Internal Server Error" "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>InternalError</Code><Message>We encountered an internal error, please try again.: cause(uplink: metaclient: context canceled)</Message><Key>hello-world.txt</Key><BucketName>uptime-checks</BucketName><Resource>/hello-world.txt</Resource><RequestId>1868A6DD9BAF33BB</RequestId><HostId></HostId></Error>"
Duration: 175.43678ms
edge
gateway
vhost
bgp
eu1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
2 out of 60 is failed (threshold=3) [.................XX.........................................] duration:
4.823761185s
FAIL (2025-09-25 22:31:11 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 12)
======LOGS======
9 , piece_id=4YT7BXXDQVTR2QL4YBWVMO3YJCZQ57PTTEQW7GCZELWC42HQUDQQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.227500377 (56h31m29.68959285s) [noise=true , address=81.171.14.70:4034 , piece_id=EXVA3SFFPD3GJDYIFHAOP7GMJQHGZECMX5R6MHLRDMMCU7LOA7KA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.253492519 (56h31m29.715584993s) [noise=true , address=81.171.14.70:4028 , piece_id=Y237IBPHRRI7NMOYJG65AZI3LCCZ6JUJPX5WWGIDX5RYHCNVIG3A, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.267890396 (56h31m29.729982869s) [noise=true , address=89.149.201.147:4010 , piece_id=ANMXRI22IXAZFHKB7ZHFGEERKT2KGUYQEFKEOJNJDC5TFB6B3STQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.280031865 (56h31m29.742124348s) [noise=true , address=89.149.201.147:4020 , piece_id=RML2JG4QZSS6H5GTP7BW3B4SFVZLLHHAOUS3LDCOT2BGYTC3HACA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.294783955 (56h31m29.756876428s) [noise=true , address=89.149.201.147:4017 , piece_id=P576E6UE5UADQZYKR4XN4I5X4MCGMXT3B7PBGVNWBZ6I67ZVCJHA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.462969447 (56h31m29.92506192s) [noise=true , address=81.171.14.70:4035 , piece_id=GWVM3MZRGRVJ5RWG227JJXTULF5MUJ4FHWCNHLW7J3GDYXRSXRXA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.668506081 (56h31m30.130598554s) [noise=true , address=81.171.14.70:4045 , piece_id=BSKEEYML4LQZ7PWB45HQNX2YFJTFIEKIILDUZIWCXMVZZGGKHGUQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.702001372 (56h31m30.164093845s) [noise=true , address=89.149.201.147:4014 , piece_id=3VH5QXDWYX2NR3LODHSATJCP5ABF7BI52GJAR3VLPW42CPJJXQNA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.728242188 (56h31m30.190334661s) [noise=true , address=89.149.201.147:4016 , piece_id=PT5TUJDTS2TDBNLH5722GJTZ5BIO4LZKEDKBGUKHEF37TPADW7MA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.797046563 (56h31m30.259139036s) [noise=true , address=89.149.201.147:4012 , piece_id=TY6OVHC4ITI2E32GE7LP3O2YQ46FKWF6A6IIBN4JFAB7WPYXXJ2A, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:03:42.295273049 (56h32m27.757365543s) [noise=true , address=185.167.204.206:4056 , piece_id=4XAHI4JY2MPISEOZ6VJ5GCJ5WZJKHRXIDK2EQGEN7VSYNRYJH24A, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1uSTuNA28oH2s5Rkv2jaKoC2nZWSgpMJXyxwFTCKbr44EvEXrC, address:185.167.204.206:4056): protocol: rpc: dial tcp 185.167.204.206:4056: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295296423 (56h32m27.757388906s) [noise=true , address=185.167.204.205:4008 , piece_id=NXU5IGU2NUZPVR5KIUKLZ7QTPYLGDIKZQNMPBJRPCVYEVNIV7HMA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12ZHTTaGQUTftxEULFqWbH6w1HnWA7PavHrXnvMCD947FxtmsrS, address:185.167.204.205:4008): protocol: rpc: dial tcp 185.167.204.205:4008: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295427543 (56h32m27.757520026s) [noise=true , address=185.167.204.206:4064 , piece_id=7GVKAUOISAORFKOXUNOJMEEFNNUVVZUQ2RGCD6BNVCDTH6ONPQIQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1ZB3emcQntFxpr7zBiE98eesnjoTnNKinw3FPtGU2TgznGakeb, address:185.167.204.206:4064): protocol: rpc: dial tcp 185.167.204.206:4064: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295495543 (56h32m27.757588026s) [noise=true , address=185.167.204.206:4059 , piece_id=G4B2XQYXOG2LWBAIYBM4BWLI7PEPE2XDM5A57YHYR6SOWW7ODSVA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:121gTedy8TPx44wmLqdQpZ1gDDsL8D9K1EvQJffR7oKY3bT9DXh, address:185.167.204.206:4059): protocol: rpc: dial tcp 185.167.204.206:4059: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295562701 (56h32m27.757655184s) [noise=true , address=185.167.204.205:4030 , piece_id=U6SOG6BNEIWDGYWDRZOCLTROZVZXPPG5ZI52HLRPKTDKFVQN3EJA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12w5R2Whxgg4AYNNvTkKmFq4pzahF1uZiVzwgVygc9iZDAXMdSk, address:185.167.204.205:4030): protocol: rpc: dial tcp 185.167.204.205:4030: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295700453 (56h32m27.757792926s) [noise=true , address=185.167.204.205:4010 , piece_id=DPMSKQDDOU2PUXHINTG7JJXLKXJE2PC6B3KNSKN5UIGAKBM3GZVQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12oTu1GaRsH7fHqxLkQ9c9ikjG4zRN3NChssa5gfpLwrcEehoyV, address:185.167.204.205:4010): protocol: rpc: dial tcp 185.167.204.205:4010: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295713257 (56h32m27.75780574s) [noise=true , address=185.167.204.206:4053 , piece_id=74G7LWVYE7GO4GPSG67BD6QVE7UBLGIAKLSG447PWWLNBM2FZNTA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12SrFttJpndpv7TKJJYsKthWnGWBYQGHpS9CwqTd4mqBP9NCCm8, address:185.167.204.206:4053): protocol: rpc: dial tcp 185.167.204.206:4053: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295852051 (56h32m27.757944534s) [noise=true , address=185.167.204.206:4065 , piece_id=VKBLX4ASIKVHMUUCTD7GFFPEHJLKREGCDGI7LCVRSASDHBAWI6AQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12oPo26KZGZr7o7M18KZiX5PMAAMwHK8FPT59y1hSLhkB6z3Lcj, address:185.167.204.206:4065): protocol: rpc: dial tcp 185.167.204.206:4065: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295908840 (56h32m27.758001313s) [noise=true , address=185.167.204.206:4057 , piece_id=WNX7U6DN7HTE3XLVJAJ4FTCYFR3B2MLMLF5I46GALE463DWQ22WQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1xy38NywWy5CSJ7CfupY78vbkHg6cQzgNhM2Z1xnpvw9dwvMoT, address:185.167.204.206:4057): protocol: rpc: dial tcp 185.167.204.206:4057: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296027916 (56h32m27.758120399s) [noise=true , address=185.167.204.205:4003 , piece_id=NQX2I3KTMYFLS2B2C5Y2PGZRXCD7QJCZLABY4PTYVEFGAVCYX2EQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1CjWvvJ4FdZc41sRgsQFWz4Y1ATZKgzsvDcUTmBSrCC15ekCmF, address:185.167.204.205:4003): protocol: rpc: dial tcp 185.167.204.205:4003: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296100945 (56h32m27.758193418s) [noise=true , address=185.167.204.205:4032 , piece_id=AN4YMHK6NKM5RIWAZHX2FPBGQ3VRFRYBKHBPIAL2HQWBK5XXNNOA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12558dx7zMNDfabV55vSsP9XwdxXgoPM8aHW53femVZzkCRxsKN, address:185.167.204.205:4032): protocol: rpc: dial tcp 185.167.204.205:4032: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296150660 (56h32m27.758243133s) [noise=true , address=185.167.204.205:4029 , piece_id=UVEMQY4WGBKGGT75ILTBPUCFZ4U252QGPDTBPT2BNMOR3V5XYCPA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12DyF12w2JXdkGLfLx9VE1E5f28VnMg7M2b3osfsDtGX9o8Tvsn, address:185.167.204.205:4029): protocol: write tcp 188.166.225.170:36604->185.167.204.205:4029: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296276239 (56h32m27.758368722s) [noise=true , address=185.167.204.206:4069 , piece_id=VKW6XTWPZPD3MDANS3FIYYD2UPDW47WBCGLNTOVYVTWA5IT23DBQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12Uan1TT5oAM3axhaviVFbvh83gThiZvwHMwbCD31ZZens6BQ4U, address:185.167.204.206:4069): protocol: rpc: dial tcp 185.167.204.206:4069: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296438448 (56h32m27.758530931s) [noise=true , address=185.167.204.206:4048 , piece_id=PRBNW47DU6K26GYUUUZSYTYQO3GNFC6DEV2OYQDP4R2RV5L5VNXA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1sY4wHBkSVUDWWg8T8Vm1hpRbbpb3ttqnqDVY8uWyG27gqBwT9, address:185.167.204.206:4048): protocol: write tcp 188.166.225.170:59140->185.167.204.206:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296628780 (56h32m27.758721253s) [noise=true , address=185.167.204.206:4055 , piece_id=KH2E6T37MHNJJ56CR3EIKMEUEWMAVP2XNMZ7K5QVKCFZJIVXRTUA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12CYRieezm1mjtGchWdPrB2mNaHVRvTS85umjB6cQyYKnvisXSB, address:185.167.204.206:4055): protocol: rpc: dial tcp 185.167.204.206:4055: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.297155593 (56h32m27.759248076s) [noise=true , address=185.167.204.205:4022 , piece_id=VU4625JC4A4ZVXFGJZTDMDZXYCNXJ67OQTEKWRLMMHGHPEHY5J7A, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj, address:185.167.204.205:4022): protocol: write tcp 188.166.225.170:57938->185.167.204.205:4022: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.297184969 (56h32m27.759277442s) [seg_pos={0 0}, upload=49746]: Done waiting for segment. successful: 12 optimal: 26 errs: <nil>
Sep 25 22:03:42.465979471 (56h32m27.928071944s) [upload=49746]: Done uploading segments...
Sep 25 22:03:42.466010200 (56h32m27.928102673s) [upload=49746]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 12)
Duration: 4.823796393s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.eu1.gateway)antiblip:
1 out of 60 is failed (threshold=3) [....................................................X.......] duration:
177.885633ms remote:
"tcp:207.211.208.129:443" response headers:
Accept-Ranges=bytes Content-Length=352 Content-Security-Policy=block-all-mixed-content Server=Storj X-Request-Id=JdrtmpHLrtw X-Xss-Protection=1; mode=block Content-Type=application/xml Vary=Origin X-Amz-Request-Id=1868A6DD9BAF33BB Date=Thu, 25 Sep 2025 22:25:25 GMT target:
https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318
FAIL (2025-09-25 22:30:44 UTC)
orbiter warning: unexpected status code "500 Internal Server Error" "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>InternalError</Code><Message>We encountered an internal error, please try again.: cause(uplink: metaclient: context canceled)</Message><Key>hello-world.txt</Key><BucketName>uptime-checks</BucketName><Resource>/hello-world.txt</Resource><RequestId>1868A6DD9BAF33BB</RequestId><HostId></HostId></Error>"
Duration: 177.916681ms
edge
gateway
vhost
bgp
eu1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
2 out of 60 is failed (threshold=3) [..................XX........................................] duration:
4.851894725s
FAIL (2025-09-25 22:30:02 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 12)
======LOGS======
9 , piece_id=4YT7BXXDQVTR2QL4YBWVMO3YJCZQ57PTTEQW7GCZELWC42HQUDQQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.227500377 (56h31m29.68959285s) [noise=true , address=81.171.14.70:4034 , piece_id=EXVA3SFFPD3GJDYIFHAOP7GMJQHGZECMX5R6MHLRDMMCU7LOA7KA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.253492519 (56h31m29.715584993s) [noise=true , address=81.171.14.70:4028 , piece_id=Y237IBPHRRI7NMOYJG65AZI3LCCZ6JUJPX5WWGIDX5RYHCNVIG3A, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.267890396 (56h31m29.729982869s) [noise=true , address=89.149.201.147:4010 , piece_id=ANMXRI22IXAZFHKB7ZHFGEERKT2KGUYQEFKEOJNJDC5TFB6B3STQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.280031865 (56h31m29.742124348s) [noise=true , address=89.149.201.147:4020 , piece_id=RML2JG4QZSS6H5GTP7BW3B4SFVZLLHHAOUS3LDCOT2BGYTC3HACA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.294783955 (56h31m29.756876428s) [noise=true , address=89.149.201.147:4017 , piece_id=P576E6UE5UADQZYKR4XN4I5X4MCGMXT3B7PBGVNWBZ6I67ZVCJHA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.462969447 (56h31m29.92506192s) [noise=true , address=81.171.14.70:4035 , piece_id=GWVM3MZRGRVJ5RWG227JJXTULF5MUJ4FHWCNHLW7J3GDYXRSXRXA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.668506081 (56h31m30.130598554s) [noise=true , address=81.171.14.70:4045 , piece_id=BSKEEYML4LQZ7PWB45HQNX2YFJTFIEKIILDUZIWCXMVZZGGKHGUQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.702001372 (56h31m30.164093845s) [noise=true , address=89.149.201.147:4014 , piece_id=3VH5QXDWYX2NR3LODHSATJCP5ABF7BI52GJAR3VLPW42CPJJXQNA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.728242188 (56h31m30.190334661s) [noise=true , address=89.149.201.147:4016 , piece_id=PT5TUJDTS2TDBNLH5722GJTZ5BIO4LZKEDKBGUKHEF37TPADW7MA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.797046563 (56h31m30.259139036s) [noise=true , address=89.149.201.147:4012 , piece_id=TY6OVHC4ITI2E32GE7LP3O2YQ46FKWF6A6IIBN4JFAB7WPYXXJ2A, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:03:42.295273049 (56h32m27.757365543s) [noise=true , address=185.167.204.206:4056 , piece_id=4XAHI4JY2MPISEOZ6VJ5GCJ5WZJKHRXIDK2EQGEN7VSYNRYJH24A, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1uSTuNA28oH2s5Rkv2jaKoC2nZWSgpMJXyxwFTCKbr44EvEXrC, address:185.167.204.206:4056): protocol: rpc: dial tcp 185.167.204.206:4056: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295296423 (56h32m27.757388906s) [noise=true , address=185.167.204.205:4008 , piece_id=NXU5IGU2NUZPVR5KIUKLZ7QTPYLGDIKZQNMPBJRPCVYEVNIV7HMA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12ZHTTaGQUTftxEULFqWbH6w1HnWA7PavHrXnvMCD947FxtmsrS, address:185.167.204.205:4008): protocol: rpc: dial tcp 185.167.204.205:4008: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295427543 (56h32m27.757520026s) [noise=true , address=185.167.204.206:4064 , piece_id=7GVKAUOISAORFKOXUNOJMEEFNNUVVZUQ2RGCD6BNVCDTH6ONPQIQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1ZB3emcQntFxpr7zBiE98eesnjoTnNKinw3FPtGU2TgznGakeb, address:185.167.204.206:4064): protocol: rpc: dial tcp 185.167.204.206:4064: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295495543 (56h32m27.757588026s) [noise=true , address=185.167.204.206:4059 , piece_id=G4B2XQYXOG2LWBAIYBM4BWLI7PEPE2XDM5A57YHYR6SOWW7ODSVA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:121gTedy8TPx44wmLqdQpZ1gDDsL8D9K1EvQJffR7oKY3bT9DXh, address:185.167.204.206:4059): protocol: rpc: dial tcp 185.167.204.206:4059: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295562701 (56h32m27.757655184s) [noise=true , address=185.167.204.205:4030 , piece_id=U6SOG6BNEIWDGYWDRZOCLTROZVZXPPG5ZI52HLRPKTDKFVQN3EJA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12w5R2Whxgg4AYNNvTkKmFq4pzahF1uZiVzwgVygc9iZDAXMdSk, address:185.167.204.205:4030): protocol: rpc: dial tcp 185.167.204.205:4030: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295700453 (56h32m27.757792926s) [noise=true , address=185.167.204.205:4010 , piece_id=DPMSKQDDOU2PUXHINTG7JJXLKXJE2PC6B3KNSKN5UIGAKBM3GZVQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12oTu1GaRsH7fHqxLkQ9c9ikjG4zRN3NChssa5gfpLwrcEehoyV, address:185.167.204.205:4010): protocol: rpc: dial tcp 185.167.204.205:4010: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295713257 (56h32m27.75780574s) [noise=true , address=185.167.204.206:4053 , piece_id=74G7LWVYE7GO4GPSG67BD6QVE7UBLGIAKLSG447PWWLNBM2FZNTA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12SrFttJpndpv7TKJJYsKthWnGWBYQGHpS9CwqTd4mqBP9NCCm8, address:185.167.204.206:4053): protocol: rpc: dial tcp 185.167.204.206:4053: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295852051 (56h32m27.757944534s) [noise=true , address=185.167.204.206:4065 , piece_id=VKBLX4ASIKVHMUUCTD7GFFPEHJLKREGCDGI7LCVRSASDHBAWI6AQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12oPo26KZGZr7o7M18KZiX5PMAAMwHK8FPT59y1hSLhkB6z3Lcj, address:185.167.204.206:4065): protocol: rpc: dial tcp 185.167.204.206:4065: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295908840 (56h32m27.758001313s) [noise=true , address=185.167.204.206:4057 , piece_id=WNX7U6DN7HTE3XLVJAJ4FTCYFR3B2MLMLF5I46GALE463DWQ22WQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1xy38NywWy5CSJ7CfupY78vbkHg6cQzgNhM2Z1xnpvw9dwvMoT, address:185.167.204.206:4057): protocol: rpc: dial tcp 185.167.204.206:4057: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296027916 (56h32m27.758120399s) [noise=true , address=185.167.204.205:4003 , piece_id=NQX2I3KTMYFLS2B2C5Y2PGZRXCD7QJCZLABY4PTYVEFGAVCYX2EQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1CjWvvJ4FdZc41sRgsQFWz4Y1ATZKgzsvDcUTmBSrCC15ekCmF, address:185.167.204.205:4003): protocol: rpc: dial tcp 185.167.204.205:4003: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296100945 (56h32m27.758193418s) [noise=true , address=185.167.204.205:4032 , piece_id=AN4YMHK6NKM5RIWAZHX2FPBGQ3VRFRYBKHBPIAL2HQWBK5XXNNOA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12558dx7zMNDfabV55vSsP9XwdxXgoPM8aHW53femVZzkCRxsKN, address:185.167.204.205:4032): protocol: rpc: dial tcp 185.167.204.205:4032: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296150660 (56h32m27.758243133s) [noise=true , address=185.167.204.205:4029 , piece_id=UVEMQY4WGBKGGT75ILTBPUCFZ4U252QGPDTBPT2BNMOR3V5XYCPA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12DyF12w2JXdkGLfLx9VE1E5f28VnMg7M2b3osfsDtGX9o8Tvsn, address:185.167.204.205:4029): protocol: write tcp 188.166.225.170:36604->185.167.204.205:4029: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296276239 (56h32m27.758368722s) [noise=true , address=185.167.204.206:4069 , piece_id=VKW6XTWPZPD3MDANS3FIYYD2UPDW47WBCGLNTOVYVTWA5IT23DBQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12Uan1TT5oAM3axhaviVFbvh83gThiZvwHMwbCD31ZZens6BQ4U, address:185.167.204.206:4069): protocol: rpc: dial tcp 185.167.204.206:4069: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296438448 (56h32m27.758530931s) [noise=true , address=185.167.204.206:4048 , piece_id=PRBNW47DU6K26GYUUUZSYTYQO3GNFC6DEV2OYQDP4R2RV5L5VNXA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1sY4wHBkSVUDWWg8T8Vm1hpRbbpb3ttqnqDVY8uWyG27gqBwT9, address:185.167.204.206:4048): protocol: write tcp 188.166.225.170:59140->185.167.204.206:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296628780 (56h32m27.758721253s) [noise=true , address=185.167.204.206:4055 , piece_id=KH2E6T37MHNJJ56CR3EIKMEUEWMAVP2XNMZ7K5QVKCFZJIVXRTUA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12CYRieezm1mjtGchWdPrB2mNaHVRvTS85umjB6cQyYKnvisXSB, address:185.167.204.206:4055): protocol: rpc: dial tcp 185.167.204.206:4055: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.297155593 (56h32m27.759248076s) [noise=true , address=185.167.204.205:4022 , piece_id=VU4625JC4A4ZVXFGJZTDMDZXYCNXJ67OQTEKWRLMMHGHPEHY5J7A, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj, address:185.167.204.205:4022): protocol: write tcp 188.166.225.170:57938->185.167.204.205:4022: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.297184969 (56h32m27.759277442s) [seg_pos={0 0}, upload=49746]: Done waiting for segment. successful: 12 optimal: 26 errs: <nil>
Sep 25 22:03:42.465979471 (56h32m27.928071944s) [upload=49746]: Done uploading segments...
Sep 25 22:03:42.466010200 (56h32m27.928102673s) [upload=49746]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 12)
Duration: 4.851948878s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.eu1.gateway)antiblip:
1 out of 60 is failed (threshold=3) [.....................................................X......] duration:
177.179047ms remote:
"tcp:207.211.208.129:443" response headers:
Accept-Ranges=bytes Content-Length=352 Content-Security-Policy=block-all-mixed-content Server=Storj X-Request-Id=JdrtmpHLrtw X-Xss-Protection=1; mode=block Content-Type=application/xml Vary=Origin X-Amz-Request-Id=1868A6DD9BAF33BB Date=Thu, 25 Sep 2025 22:25:25 GMT target:
https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318
FAIL (2025-09-25 22:29:46 UTC)
orbiter warning: unexpected status code "500 Internal Server Error" "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>InternalError</Code><Message>We encountered an internal error, please try again.: cause(uplink: metaclient: context canceled)</Message><Key>hello-world.txt</Key><BucketName>uptime-checks</BucketName><Resource>/hello-world.txt</Resource><RequestId>1868A6DD9BAF33BB</RequestId><HostId></HostId></Error>"
Duration: 177.230276ms
edge
gateway
vhost
bgp
eu1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
2 out of 60 is failed (threshold=3) [...................XX.......................................] duration:
4.783357681s
FAIL (2025-09-25 22:29:21 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 12)
======LOGS======
9 , piece_id=4YT7BXXDQVTR2QL4YBWVMO3YJCZQ57PTTEQW7GCZELWC42HQUDQQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.227500377 (56h31m29.68959285s) [noise=true , address=81.171.14.70:4034 , piece_id=EXVA3SFFPD3GJDYIFHAOP7GMJQHGZECMX5R6MHLRDMMCU7LOA7KA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.253492519 (56h31m29.715584993s) [noise=true , address=81.171.14.70:4028 , piece_id=Y237IBPHRRI7NMOYJG65AZI3LCCZ6JUJPX5WWGIDX5RYHCNVIG3A, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.267890396 (56h31m29.729982869s) [noise=true , address=89.149.201.147:4010 , piece_id=ANMXRI22IXAZFHKB7ZHFGEERKT2KGUYQEFKEOJNJDC5TFB6B3STQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.280031865 (56h31m29.742124348s) [noise=true , address=89.149.201.147:4020 , piece_id=RML2JG4QZSS6H5GTP7BW3B4SFVZLLHHAOUS3LDCOT2BGYTC3HACA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.294783955 (56h31m29.756876428s) [noise=true , address=89.149.201.147:4017 , piece_id=P576E6UE5UADQZYKR4XN4I5X4MCGMXT3B7PBGVNWBZ6I67ZVCJHA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.462969447 (56h31m29.92506192s) [noise=true , address=81.171.14.70:4035 , piece_id=GWVM3MZRGRVJ5RWG227JJXTULF5MUJ4FHWCNHLW7J3GDYXRSXRXA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.668506081 (56h31m30.130598554s) [noise=true , address=81.171.14.70:4045 , piece_id=BSKEEYML4LQZ7PWB45HQNX2YFJTFIEKIILDUZIWCXMVZZGGKHGUQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.702001372 (56h31m30.164093845s) [noise=true , address=89.149.201.147:4014 , piece_id=3VH5QXDWYX2NR3LODHSATJCP5ABF7BI52GJAR3VLPW42CPJJXQNA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.728242188 (56h31m30.190334661s) [noise=true , address=89.149.201.147:4016 , piece_id=PT5TUJDTS2TDBNLH5722GJTZ5BIO4LZKEDKBGUKHEF37TPADW7MA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.797046563 (56h31m30.259139036s) [noise=true , address=89.149.201.147:4012 , piece_id=TY6OVHC4ITI2E32GE7LP3O2YQ46FKWF6A6IIBN4JFAB7WPYXXJ2A, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:03:42.295273049 (56h32m27.757365543s) [noise=true , address=185.167.204.206:4056 , piece_id=4XAHI4JY2MPISEOZ6VJ5GCJ5WZJKHRXIDK2EQGEN7VSYNRYJH24A, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1uSTuNA28oH2s5Rkv2jaKoC2nZWSgpMJXyxwFTCKbr44EvEXrC, address:185.167.204.206:4056): protocol: rpc: dial tcp 185.167.204.206:4056: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295296423 (56h32m27.757388906s) [noise=true , address=185.167.204.205:4008 , piece_id=NXU5IGU2NUZPVR5KIUKLZ7QTPYLGDIKZQNMPBJRPCVYEVNIV7HMA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12ZHTTaGQUTftxEULFqWbH6w1HnWA7PavHrXnvMCD947FxtmsrS, address:185.167.204.205:4008): protocol: rpc: dial tcp 185.167.204.205:4008: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295427543 (56h32m27.757520026s) [noise=true , address=185.167.204.206:4064 , piece_id=7GVKAUOISAORFKOXUNOJMEEFNNUVVZUQ2RGCD6BNVCDTH6ONPQIQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1ZB3emcQntFxpr7zBiE98eesnjoTnNKinw3FPtGU2TgznGakeb, address:185.167.204.206:4064): protocol: rpc: dial tcp 185.167.204.206:4064: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295495543 (56h32m27.757588026s) [noise=true , address=185.167.204.206:4059 , piece_id=G4B2XQYXOG2LWBAIYBM4BWLI7PEPE2XDM5A57YHYR6SOWW7ODSVA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:121gTedy8TPx44wmLqdQpZ1gDDsL8D9K1EvQJffR7oKY3bT9DXh, address:185.167.204.206:4059): protocol: rpc: dial tcp 185.167.204.206:4059: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295562701 (56h32m27.757655184s) [noise=true , address=185.167.204.205:4030 , piece_id=U6SOG6BNEIWDGYWDRZOCLTROZVZXPPG5ZI52HLRPKTDKFVQN3EJA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12w5R2Whxgg4AYNNvTkKmFq4pzahF1uZiVzwgVygc9iZDAXMdSk, address:185.167.204.205:4030): protocol: rpc: dial tcp 185.167.204.205:4030: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295700453 (56h32m27.757792926s) [noise=true , address=185.167.204.205:4010 , piece_id=DPMSKQDDOU2PUXHINTG7JJXLKXJE2PC6B3KNSKN5UIGAKBM3GZVQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12oTu1GaRsH7fHqxLkQ9c9ikjG4zRN3NChssa5gfpLwrcEehoyV, address:185.167.204.205:4010): protocol: rpc: dial tcp 185.167.204.205:4010: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295713257 (56h32m27.75780574s) [noise=true , address=185.167.204.206:4053 , piece_id=74G7LWVYE7GO4GPSG67BD6QVE7UBLGIAKLSG447PWWLNBM2FZNTA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12SrFttJpndpv7TKJJYsKthWnGWBYQGHpS9CwqTd4mqBP9NCCm8, address:185.167.204.206:4053): protocol: rpc: dial tcp 185.167.204.206:4053: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295852051 (56h32m27.757944534s) [noise=true , address=185.167.204.206:4065 , piece_id=VKBLX4ASIKVHMUUCTD7GFFPEHJLKREGCDGI7LCVRSASDHBAWI6AQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12oPo26KZGZr7o7M18KZiX5PMAAMwHK8FPT59y1hSLhkB6z3Lcj, address:185.167.204.206:4065): protocol: rpc: dial tcp 185.167.204.206:4065: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295908840 (56h32m27.758001313s) [noise=true , address=185.167.204.206:4057 , piece_id=WNX7U6DN7HTE3XLVJAJ4FTCYFR3B2MLMLF5I46GALE463DWQ22WQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1xy38NywWy5CSJ7CfupY78vbkHg6cQzgNhM2Z1xnpvw9dwvMoT, address:185.167.204.206:4057): protocol: rpc: dial tcp 185.167.204.206:4057: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296027916 (56h32m27.758120399s) [noise=true , address=185.167.204.205:4003 , piece_id=NQX2I3KTMYFLS2B2C5Y2PGZRXCD7QJCZLABY4PTYVEFGAVCYX2EQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1CjWvvJ4FdZc41sRgsQFWz4Y1ATZKgzsvDcUTmBSrCC15ekCmF, address:185.167.204.205:4003): protocol: rpc: dial tcp 185.167.204.205:4003: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296100945 (56h32m27.758193418s) [noise=true , address=185.167.204.205:4032 , piece_id=AN4YMHK6NKM5RIWAZHX2FPBGQ3VRFRYBKHBPIAL2HQWBK5XXNNOA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12558dx7zMNDfabV55vSsP9XwdxXgoPM8aHW53femVZzkCRxsKN, address:185.167.204.205:4032): protocol: rpc: dial tcp 185.167.204.205:4032: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296150660 (56h32m27.758243133s) [noise=true , address=185.167.204.205:4029 , piece_id=UVEMQY4WGBKGGT75ILTBPUCFZ4U252QGPDTBPT2BNMOR3V5XYCPA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12DyF12w2JXdkGLfLx9VE1E5f28VnMg7M2b3osfsDtGX9o8Tvsn, address:185.167.204.205:4029): protocol: write tcp 188.166.225.170:36604->185.167.204.205:4029: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296276239 (56h32m27.758368722s) [noise=true , address=185.167.204.206:4069 , piece_id=VKW6XTWPZPD3MDANS3FIYYD2UPDW47WBCGLNTOVYVTWA5IT23DBQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12Uan1TT5oAM3axhaviVFbvh83gThiZvwHMwbCD31ZZens6BQ4U, address:185.167.204.206:4069): protocol: rpc: dial tcp 185.167.204.206:4069: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296438448 (56h32m27.758530931s) [noise=true , address=185.167.204.206:4048 , piece_id=PRBNW47DU6K26GYUUUZSYTYQO3GNFC6DEV2OYQDP4R2RV5L5VNXA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1sY4wHBkSVUDWWg8T8Vm1hpRbbpb3ttqnqDVY8uWyG27gqBwT9, address:185.167.204.206:4048): protocol: write tcp 188.166.225.170:59140->185.167.204.206:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296628780 (56h32m27.758721253s) [noise=true , address=185.167.204.206:4055 , piece_id=KH2E6T37MHNJJ56CR3EIKMEUEWMAVP2XNMZ7K5QVKCFZJIVXRTUA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12CYRieezm1mjtGchWdPrB2mNaHVRvTS85umjB6cQyYKnvisXSB, address:185.167.204.206:4055): protocol: rpc: dial tcp 185.167.204.206:4055: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.297155593 (56h32m27.759248076s) [noise=true , address=185.167.204.205:4022 , piece_id=VU4625JC4A4ZVXFGJZTDMDZXYCNXJ67OQTEKWRLMMHGHPEHY5J7A, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj, address:185.167.204.205:4022): protocol: write tcp 188.166.225.170:57938->185.167.204.205:4022: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.297184969 (56h32m27.759277442s) [seg_pos={0 0}, upload=49746]: Done waiting for segment. successful: 12 optimal: 26 errs: <nil>
Sep 25 22:03:42.465979471 (56h32m27.928071944s) [upload=49746]: Done uploading segments...
Sep 25 22:03:42.466010200 (56h32m27.928102673s) [upload=49746]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 12)
Duration: 4.783386055s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.eu1.gateway)antiblip:
1 out of 60 is failed (threshold=3) [......................................................X.....] duration:
167.754111ms remote:
"tcp:207.211.208.129:443" response headers:
Accept-Ranges=bytes Content-Length=352 Content-Security-Policy=block-all-mixed-content Server=Storj X-Request-Id=JdrtmpHLrtw X-Xss-Protection=1; mode=block Content-Type=application/xml Vary=Origin X-Amz-Request-Id=1868A6DD9BAF33BB Date=Thu, 25 Sep 2025 22:25:25 GMT target:
https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318
FAIL (2025-09-25 22:28:32 UTC)
orbiter warning: unexpected status code "500 Internal Server Error" "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>InternalError</Code><Message>We encountered an internal error, please try again.: cause(uplink: metaclient: context canceled)</Message><Key>hello-world.txt</Key><BucketName>uptime-checks</BucketName><Resource>/hello-world.txt</Resource><RequestId>1868A6DD9BAF33BB</RequestId><HostId></HostId></Error>"
Duration: 167.786011ms
edge
gateway
vhost
bgp
eu1
|
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.eu1.gateway)antiblip:
1 out of 60 is failed (threshold=3) [.......................................................X....] duration:
180.339384ms remote:
"tcp:207.211.208.129:443" response headers:
Accept-Ranges=bytes Content-Length=352 Content-Security-Policy=block-all-mixed-content Server=Storj X-Request-Id=JdrtmpHLrtw X-Xss-Protection=1; mode=block Content-Type=application/xml Vary=Origin X-Amz-Request-Id=1868A6DD9BAF33BB Date=Thu, 25 Sep 2025 22:25:25 GMT target:
https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318
FAIL (2025-09-25 22:28:08 UTC)
orbiter warning: unexpected status code "500 Internal Server Error" "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>InternalError</Code><Message>We encountered an internal error, please try again.: cause(uplink: metaclient: context canceled)</Message><Key>hello-world.txt</Key><BucketName>uptime-checks</BucketName><Resource>/hello-world.txt</Resource><RequestId>1868A6DD9BAF33BB</RequestId><HostId></HostId></Error>"
Duration: 180.372259ms
edge
gateway
vhost
bgp
eu1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
2 out of 60 is failed (threshold=3) [....................XX......................................] duration:
4.971776041s
FAIL (2025-09-25 22:28:07 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 12)
======LOGS======
9 , piece_id=4YT7BXXDQVTR2QL4YBWVMO3YJCZQ57PTTEQW7GCZELWC42HQUDQQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.227500377 (56h31m29.68959285s) [noise=true , address=81.171.14.70:4034 , piece_id=EXVA3SFFPD3GJDYIFHAOP7GMJQHGZECMX5R6MHLRDMMCU7LOA7KA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.253492519 (56h31m29.715584993s) [noise=true , address=81.171.14.70:4028 , piece_id=Y237IBPHRRI7NMOYJG65AZI3LCCZ6JUJPX5WWGIDX5RYHCNVIG3A, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.267890396 (56h31m29.729982869s) [noise=true , address=89.149.201.147:4010 , piece_id=ANMXRI22IXAZFHKB7ZHFGEERKT2KGUYQEFKEOJNJDC5TFB6B3STQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.280031865 (56h31m29.742124348s) [noise=true , address=89.149.201.147:4020 , piece_id=RML2JG4QZSS6H5GTP7BW3B4SFVZLLHHAOUS3LDCOT2BGYTC3HACA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.294783955 (56h31m29.756876428s) [noise=true , address=89.149.201.147:4017 , piece_id=P576E6UE5UADQZYKR4XN4I5X4MCGMXT3B7PBGVNWBZ6I67ZVCJHA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.462969447 (56h31m29.92506192s) [noise=true , address=81.171.14.70:4035 , piece_id=GWVM3MZRGRVJ5RWG227JJXTULF5MUJ4FHWCNHLW7J3GDYXRSXRXA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.668506081 (56h31m30.130598554s) [noise=true , address=81.171.14.70:4045 , piece_id=BSKEEYML4LQZ7PWB45HQNX2YFJTFIEKIILDUZIWCXMVZZGGKHGUQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.702001372 (56h31m30.164093845s) [noise=true , address=89.149.201.147:4014 , piece_id=3VH5QXDWYX2NR3LODHSATJCP5ABF7BI52GJAR3VLPW42CPJJXQNA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.728242188 (56h31m30.190334661s) [noise=true , address=89.149.201.147:4016 , piece_id=PT5TUJDTS2TDBNLH5722GJTZ5BIO4LZKEDKBGUKHEF37TPADW7MA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.797046563 (56h31m30.259139036s) [noise=true , address=89.149.201.147:4012 , piece_id=TY6OVHC4ITI2E32GE7LP3O2YQ46FKWF6A6IIBN4JFAB7WPYXXJ2A, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:03:42.295273049 (56h32m27.757365543s) [noise=true , address=185.167.204.206:4056 , piece_id=4XAHI4JY2MPISEOZ6VJ5GCJ5WZJKHRXIDK2EQGEN7VSYNRYJH24A, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1uSTuNA28oH2s5Rkv2jaKoC2nZWSgpMJXyxwFTCKbr44EvEXrC, address:185.167.204.206:4056): protocol: rpc: dial tcp 185.167.204.206:4056: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295296423 (56h32m27.757388906s) [noise=true , address=185.167.204.205:4008 , piece_id=NXU5IGU2NUZPVR5KIUKLZ7QTPYLGDIKZQNMPBJRPCVYEVNIV7HMA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12ZHTTaGQUTftxEULFqWbH6w1HnWA7PavHrXnvMCD947FxtmsrS, address:185.167.204.205:4008): protocol: rpc: dial tcp 185.167.204.205:4008: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295427543 (56h32m27.757520026s) [noise=true , address=185.167.204.206:4064 , piece_id=7GVKAUOISAORFKOXUNOJMEEFNNUVVZUQ2RGCD6BNVCDTH6ONPQIQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1ZB3emcQntFxpr7zBiE98eesnjoTnNKinw3FPtGU2TgznGakeb, address:185.167.204.206:4064): protocol: rpc: dial tcp 185.167.204.206:4064: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295495543 (56h32m27.757588026s) [noise=true , address=185.167.204.206:4059 , piece_id=G4B2XQYXOG2LWBAIYBM4BWLI7PEPE2XDM5A57YHYR6SOWW7ODSVA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:121gTedy8TPx44wmLqdQpZ1gDDsL8D9K1EvQJffR7oKY3bT9DXh, address:185.167.204.206:4059): protocol: rpc: dial tcp 185.167.204.206:4059: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295562701 (56h32m27.757655184s) [noise=true , address=185.167.204.205:4030 , piece_id=U6SOG6BNEIWDGYWDRZOCLTROZVZXPPG5ZI52HLRPKTDKFVQN3EJA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12w5R2Whxgg4AYNNvTkKmFq4pzahF1uZiVzwgVygc9iZDAXMdSk, address:185.167.204.205:4030): protocol: rpc: dial tcp 185.167.204.205:4030: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295700453 (56h32m27.757792926s) [noise=true , address=185.167.204.205:4010 , piece_id=DPMSKQDDOU2PUXHINTG7JJXLKXJE2PC6B3KNSKN5UIGAKBM3GZVQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12oTu1GaRsH7fHqxLkQ9c9ikjG4zRN3NChssa5gfpLwrcEehoyV, address:185.167.204.205:4010): protocol: rpc: dial tcp 185.167.204.205:4010: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295713257 (56h32m27.75780574s) [noise=true , address=185.167.204.206:4053 , piece_id=74G7LWVYE7GO4GPSG67BD6QVE7UBLGIAKLSG447PWWLNBM2FZNTA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12SrFttJpndpv7TKJJYsKthWnGWBYQGHpS9CwqTd4mqBP9NCCm8, address:185.167.204.206:4053): protocol: rpc: dial tcp 185.167.204.206:4053: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295852051 (56h32m27.757944534s) [noise=true , address=185.167.204.206:4065 , piece_id=VKBLX4ASIKVHMUUCTD7GFFPEHJLKREGCDGI7LCVRSASDHBAWI6AQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12oPo26KZGZr7o7M18KZiX5PMAAMwHK8FPT59y1hSLhkB6z3Lcj, address:185.167.204.206:4065): protocol: rpc: dial tcp 185.167.204.206:4065: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295908840 (56h32m27.758001313s) [noise=true , address=185.167.204.206:4057 , piece_id=WNX7U6DN7HTE3XLVJAJ4FTCYFR3B2MLMLF5I46GALE463DWQ22WQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1xy38NywWy5CSJ7CfupY78vbkHg6cQzgNhM2Z1xnpvw9dwvMoT, address:185.167.204.206:4057): protocol: rpc: dial tcp 185.167.204.206:4057: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296027916 (56h32m27.758120399s) [noise=true , address=185.167.204.205:4003 , piece_id=NQX2I3KTMYFLS2B2C5Y2PGZRXCD7QJCZLABY4PTYVEFGAVCYX2EQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1CjWvvJ4FdZc41sRgsQFWz4Y1ATZKgzsvDcUTmBSrCC15ekCmF, address:185.167.204.205:4003): protocol: rpc: dial tcp 185.167.204.205:4003: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296100945 (56h32m27.758193418s) [noise=true , address=185.167.204.205:4032 , piece_id=AN4YMHK6NKM5RIWAZHX2FPBGQ3VRFRYBKHBPIAL2HQWBK5XXNNOA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12558dx7zMNDfabV55vSsP9XwdxXgoPM8aHW53femVZzkCRxsKN, address:185.167.204.205:4032): protocol: rpc: dial tcp 185.167.204.205:4032: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296150660 (56h32m27.758243133s) [noise=true , address=185.167.204.205:4029 , piece_id=UVEMQY4WGBKGGT75ILTBPUCFZ4U252QGPDTBPT2BNMOR3V5XYCPA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12DyF12w2JXdkGLfLx9VE1E5f28VnMg7M2b3osfsDtGX9o8Tvsn, address:185.167.204.205:4029): protocol: write tcp 188.166.225.170:36604->185.167.204.205:4029: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296276239 (56h32m27.758368722s) [noise=true , address=185.167.204.206:4069 , piece_id=VKW6XTWPZPD3MDANS3FIYYD2UPDW47WBCGLNTOVYVTWA5IT23DBQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12Uan1TT5oAM3axhaviVFbvh83gThiZvwHMwbCD31ZZens6BQ4U, address:185.167.204.206:4069): protocol: rpc: dial tcp 185.167.204.206:4069: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296438448 (56h32m27.758530931s) [noise=true , address=185.167.204.206:4048 , piece_id=PRBNW47DU6K26GYUUUZSYTYQO3GNFC6DEV2OYQDP4R2RV5L5VNXA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1sY4wHBkSVUDWWg8T8Vm1hpRbbpb3ttqnqDVY8uWyG27gqBwT9, address:185.167.204.206:4048): protocol: write tcp 188.166.225.170:59140->185.167.204.206:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296628780 (56h32m27.758721253s) [noise=true , address=185.167.204.206:4055 , piece_id=KH2E6T37MHNJJ56CR3EIKMEUEWMAVP2XNMZ7K5QVKCFZJIVXRTUA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12CYRieezm1mjtGchWdPrB2mNaHVRvTS85umjB6cQyYKnvisXSB, address:185.167.204.206:4055): protocol: rpc: dial tcp 185.167.204.206:4055: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.297155593 (56h32m27.759248076s) [noise=true , address=185.167.204.205:4022 , piece_id=VU4625JC4A4ZVXFGJZTDMDZXYCNXJ67OQTEKWRLMMHGHPEHY5J7A, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj, address:185.167.204.205:4022): protocol: write tcp 188.166.225.170:57938->185.167.204.205:4022: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.297184969 (56h32m27.759277442s) [seg_pos={0 0}, upload=49746]: Done waiting for segment. successful: 12 optimal: 26 errs: <nil>
Sep 25 22:03:42.465979471 (56h32m27.928071944s) [upload=49746]: Done uploading segments...
Sep 25 22:03:42.466010200 (56h32m27.928102673s) [upload=49746]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 12)
Duration: 4.971800538s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.eu1.gateway)antiblip:
1 out of 60 is failed (threshold=3) [........................................................X...] duration:
172.530877ms remote:
"tcp:207.211.208.129:443" response headers:
Accept-Ranges=bytes Content-Length=352 Content-Security-Policy=block-all-mixed-content Server=Storj X-Request-Id=JdrtmpHLrtw X-Xss-Protection=1; mode=block Content-Type=application/xml Vary=Origin X-Amz-Request-Id=1868A6DD9BAF33BB Date=Thu, 25 Sep 2025 22:25:25 GMT target:
https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318
FAIL (2025-09-25 22:27:50 UTC)
orbiter warning: unexpected status code "500 Internal Server Error" "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>InternalError</Code><Message>We encountered an internal error, please try again.: cause(uplink: metaclient: context canceled)</Message><Key>hello-world.txt</Key><BucketName>uptime-checks</BucketName><Resource>/hello-world.txt</Resource><RequestId>1868A6DD9BAF33BB</RequestId><HostId></HostId></Error>"
Duration: 172.61138ms
edge
gateway
vhost
bgp
eu1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
2 out of 60 is failed (threshold=3) [.....................XX.....................................] duration:
4.800973093s
FAIL (2025-09-25 22:27:39 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 12)
======LOGS======
9 , piece_id=4YT7BXXDQVTR2QL4YBWVMO3YJCZQ57PTTEQW7GCZELWC42HQUDQQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.227500377 (56h31m29.68959285s) [noise=true , address=81.171.14.70:4034 , piece_id=EXVA3SFFPD3GJDYIFHAOP7GMJQHGZECMX5R6MHLRDMMCU7LOA7KA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.253492519 (56h31m29.715584993s) [noise=true , address=81.171.14.70:4028 , piece_id=Y237IBPHRRI7NMOYJG65AZI3LCCZ6JUJPX5WWGIDX5RYHCNVIG3A, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.267890396 (56h31m29.729982869s) [noise=true , address=89.149.201.147:4010 , piece_id=ANMXRI22IXAZFHKB7ZHFGEERKT2KGUYQEFKEOJNJDC5TFB6B3STQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.280031865 (56h31m29.742124348s) [noise=true , address=89.149.201.147:4020 , piece_id=RML2JG4QZSS6H5GTP7BW3B4SFVZLLHHAOUS3LDCOT2BGYTC3HACA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.294783955 (56h31m29.756876428s) [noise=true , address=89.149.201.147:4017 , piece_id=P576E6UE5UADQZYKR4XN4I5X4MCGMXT3B7PBGVNWBZ6I67ZVCJHA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.462969447 (56h31m29.92506192s) [noise=true , address=81.171.14.70:4035 , piece_id=GWVM3MZRGRVJ5RWG227JJXTULF5MUJ4FHWCNHLW7J3GDYXRSXRXA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.668506081 (56h31m30.130598554s) [noise=true , address=81.171.14.70:4045 , piece_id=BSKEEYML4LQZ7PWB45HQNX2YFJTFIEKIILDUZIWCXMVZZGGKHGUQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.702001372 (56h31m30.164093845s) [noise=true , address=89.149.201.147:4014 , piece_id=3VH5QXDWYX2NR3LODHSATJCP5ABF7BI52GJAR3VLPW42CPJJXQNA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.728242188 (56h31m30.190334661s) [noise=true , address=89.149.201.147:4016 , piece_id=PT5TUJDTS2TDBNLH5722GJTZ5BIO4LZKEDKBGUKHEF37TPADW7MA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.797046563 (56h31m30.259139036s) [noise=true , address=89.149.201.147:4012 , piece_id=TY6OVHC4ITI2E32GE7LP3O2YQ46FKWF6A6IIBN4JFAB7WPYXXJ2A, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:03:42.295273049 (56h32m27.757365543s) [noise=true , address=185.167.204.206:4056 , piece_id=4XAHI4JY2MPISEOZ6VJ5GCJ5WZJKHRXIDK2EQGEN7VSYNRYJH24A, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1uSTuNA28oH2s5Rkv2jaKoC2nZWSgpMJXyxwFTCKbr44EvEXrC, address:185.167.204.206:4056): protocol: rpc: dial tcp 185.167.204.206:4056: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295296423 (56h32m27.757388906s) [noise=true , address=185.167.204.205:4008 , piece_id=NXU5IGU2NUZPVR5KIUKLZ7QTPYLGDIKZQNMPBJRPCVYEVNIV7HMA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12ZHTTaGQUTftxEULFqWbH6w1HnWA7PavHrXnvMCD947FxtmsrS, address:185.167.204.205:4008): protocol: rpc: dial tcp 185.167.204.205:4008: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295427543 (56h32m27.757520026s) [noise=true , address=185.167.204.206:4064 , piece_id=7GVKAUOISAORFKOXUNOJMEEFNNUVVZUQ2RGCD6BNVCDTH6ONPQIQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1ZB3emcQntFxpr7zBiE98eesnjoTnNKinw3FPtGU2TgznGakeb, address:185.167.204.206:4064): protocol: rpc: dial tcp 185.167.204.206:4064: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295495543 (56h32m27.757588026s) [noise=true , address=185.167.204.206:4059 , piece_id=G4B2XQYXOG2LWBAIYBM4BWLI7PEPE2XDM5A57YHYR6SOWW7ODSVA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:121gTedy8TPx44wmLqdQpZ1gDDsL8D9K1EvQJffR7oKY3bT9DXh, address:185.167.204.206:4059): protocol: rpc: dial tcp 185.167.204.206:4059: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295562701 (56h32m27.757655184s) [noise=true , address=185.167.204.205:4030 , piece_id=U6SOG6BNEIWDGYWDRZOCLTROZVZXPPG5ZI52HLRPKTDKFVQN3EJA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12w5R2Whxgg4AYNNvTkKmFq4pzahF1uZiVzwgVygc9iZDAXMdSk, address:185.167.204.205:4030): protocol: rpc: dial tcp 185.167.204.205:4030: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295700453 (56h32m27.757792926s) [noise=true , address=185.167.204.205:4010 , piece_id=DPMSKQDDOU2PUXHINTG7JJXLKXJE2PC6B3KNSKN5UIGAKBM3GZVQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12oTu1GaRsH7fHqxLkQ9c9ikjG4zRN3NChssa5gfpLwrcEehoyV, address:185.167.204.205:4010): protocol: rpc: dial tcp 185.167.204.205:4010: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295713257 (56h32m27.75780574s) [noise=true , address=185.167.204.206:4053 , piece_id=74G7LWVYE7GO4GPSG67BD6QVE7UBLGIAKLSG447PWWLNBM2FZNTA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12SrFttJpndpv7TKJJYsKthWnGWBYQGHpS9CwqTd4mqBP9NCCm8, address:185.167.204.206:4053): protocol: rpc: dial tcp 185.167.204.206:4053: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295852051 (56h32m27.757944534s) [noise=true , address=185.167.204.206:4065 , piece_id=VKBLX4ASIKVHMUUCTD7GFFPEHJLKREGCDGI7LCVRSASDHBAWI6AQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12oPo26KZGZr7o7M18KZiX5PMAAMwHK8FPT59y1hSLhkB6z3Lcj, address:185.167.204.206:4065): protocol: rpc: dial tcp 185.167.204.206:4065: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295908840 (56h32m27.758001313s) [noise=true , address=185.167.204.206:4057 , piece_id=WNX7U6DN7HTE3XLVJAJ4FTCYFR3B2MLMLF5I46GALE463DWQ22WQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1xy38NywWy5CSJ7CfupY78vbkHg6cQzgNhM2Z1xnpvw9dwvMoT, address:185.167.204.206:4057): protocol: rpc: dial tcp 185.167.204.206:4057: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296027916 (56h32m27.758120399s) [noise=true , address=185.167.204.205:4003 , piece_id=NQX2I3KTMYFLS2B2C5Y2PGZRXCD7QJCZLABY4PTYVEFGAVCYX2EQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1CjWvvJ4FdZc41sRgsQFWz4Y1ATZKgzsvDcUTmBSrCC15ekCmF, address:185.167.204.205:4003): protocol: rpc: dial tcp 185.167.204.205:4003: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296100945 (56h32m27.758193418s) [noise=true , address=185.167.204.205:4032 , piece_id=AN4YMHK6NKM5RIWAZHX2FPBGQ3VRFRYBKHBPIAL2HQWBK5XXNNOA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12558dx7zMNDfabV55vSsP9XwdxXgoPM8aHW53femVZzkCRxsKN, address:185.167.204.205:4032): protocol: rpc: dial tcp 185.167.204.205:4032: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296150660 (56h32m27.758243133s) [noise=true , address=185.167.204.205:4029 , piece_id=UVEMQY4WGBKGGT75ILTBPUCFZ4U252QGPDTBPT2BNMOR3V5XYCPA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12DyF12w2JXdkGLfLx9VE1E5f28VnMg7M2b3osfsDtGX9o8Tvsn, address:185.167.204.205:4029): protocol: write tcp 188.166.225.170:36604->185.167.204.205:4029: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296276239 (56h32m27.758368722s) [noise=true , address=185.167.204.206:4069 , piece_id=VKW6XTWPZPD3MDANS3FIYYD2UPDW47WBCGLNTOVYVTWA5IT23DBQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12Uan1TT5oAM3axhaviVFbvh83gThiZvwHMwbCD31ZZens6BQ4U, address:185.167.204.206:4069): protocol: rpc: dial tcp 185.167.204.206:4069: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296438448 (56h32m27.758530931s) [noise=true , address=185.167.204.206:4048 , piece_id=PRBNW47DU6K26GYUUUZSYTYQO3GNFC6DEV2OYQDP4R2RV5L5VNXA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1sY4wHBkSVUDWWg8T8Vm1hpRbbpb3ttqnqDVY8uWyG27gqBwT9, address:185.167.204.206:4048): protocol: write tcp 188.166.225.170:59140->185.167.204.206:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296628780 (56h32m27.758721253s) [noise=true , address=185.167.204.206:4055 , piece_id=KH2E6T37MHNJJ56CR3EIKMEUEWMAVP2XNMZ7K5QVKCFZJIVXRTUA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12CYRieezm1mjtGchWdPrB2mNaHVRvTS85umjB6cQyYKnvisXSB, address:185.167.204.206:4055): protocol: rpc: dial tcp 185.167.204.206:4055: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.297155593 (56h32m27.759248076s) [noise=true , address=185.167.204.205:4022 , piece_id=VU4625JC4A4ZVXFGJZTDMDZXYCNXJ67OQTEKWRLMMHGHPEHY5J7A, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj, address:185.167.204.205:4022): protocol: write tcp 188.166.225.170:57938->185.167.204.205:4022: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.297184969 (56h32m27.759277442s) [seg_pos={0 0}, upload=49746]: Done waiting for segment. successful: 12 optimal: 26 errs: <nil>
Sep 25 22:03:42.465979471 (56h32m27.928071944s) [upload=49746]: Done uploading segments...
Sep 25 22:03:42.466010200 (56h32m27.928102673s) [upload=49746]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 12)
Duration: 4.801003542s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
2 out of 60 is failed (threshold=3) [......................XX....................................] duration:
4.837689547s
FAIL (2025-09-25 22:26:59 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 12)
======LOGS======
9 , piece_id=4YT7BXXDQVTR2QL4YBWVMO3YJCZQ57PTTEQW7GCZELWC42HQUDQQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.227500377 (56h31m29.68959285s) [noise=true , address=81.171.14.70:4034 , piece_id=EXVA3SFFPD3GJDYIFHAOP7GMJQHGZECMX5R6MHLRDMMCU7LOA7KA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.253492519 (56h31m29.715584993s) [noise=true , address=81.171.14.70:4028 , piece_id=Y237IBPHRRI7NMOYJG65AZI3LCCZ6JUJPX5WWGIDX5RYHCNVIG3A, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.267890396 (56h31m29.729982869s) [noise=true , address=89.149.201.147:4010 , piece_id=ANMXRI22IXAZFHKB7ZHFGEERKT2KGUYQEFKEOJNJDC5TFB6B3STQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.280031865 (56h31m29.742124348s) [noise=true , address=89.149.201.147:4020 , piece_id=RML2JG4QZSS6H5GTP7BW3B4SFVZLLHHAOUS3LDCOT2BGYTC3HACA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.294783955 (56h31m29.756876428s) [noise=true , address=89.149.201.147:4017 , piece_id=P576E6UE5UADQZYKR4XN4I5X4MCGMXT3B7PBGVNWBZ6I67ZVCJHA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.462969447 (56h31m29.92506192s) [noise=true , address=81.171.14.70:4035 , piece_id=GWVM3MZRGRVJ5RWG227JJXTULF5MUJ4FHWCNHLW7J3GDYXRSXRXA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.668506081 (56h31m30.130598554s) [noise=true , address=81.171.14.70:4045 , piece_id=BSKEEYML4LQZ7PWB45HQNX2YFJTFIEKIILDUZIWCXMVZZGGKHGUQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.702001372 (56h31m30.164093845s) [noise=true , address=89.149.201.147:4014 , piece_id=3VH5QXDWYX2NR3LODHSATJCP5ABF7BI52GJAR3VLPW42CPJJXQNA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.728242188 (56h31m30.190334661s) [noise=true , address=89.149.201.147:4016 , piece_id=PT5TUJDTS2TDBNLH5722GJTZ5BIO4LZKEDKBGUKHEF37TPADW7MA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.797046563 (56h31m30.259139036s) [noise=true , address=89.149.201.147:4012 , piece_id=TY6OVHC4ITI2E32GE7LP3O2YQ46FKWF6A6IIBN4JFAB7WPYXXJ2A, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:03:42.295273049 (56h32m27.757365543s) [noise=true , address=185.167.204.206:4056 , piece_id=4XAHI4JY2MPISEOZ6VJ5GCJ5WZJKHRXIDK2EQGEN7VSYNRYJH24A, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1uSTuNA28oH2s5Rkv2jaKoC2nZWSgpMJXyxwFTCKbr44EvEXrC, address:185.167.204.206:4056): protocol: rpc: dial tcp 185.167.204.206:4056: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295296423 (56h32m27.757388906s) [noise=true , address=185.167.204.205:4008 , piece_id=NXU5IGU2NUZPVR5KIUKLZ7QTPYLGDIKZQNMPBJRPCVYEVNIV7HMA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12ZHTTaGQUTftxEULFqWbH6w1HnWA7PavHrXnvMCD947FxtmsrS, address:185.167.204.205:4008): protocol: rpc: dial tcp 185.167.204.205:4008: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295427543 (56h32m27.757520026s) [noise=true , address=185.167.204.206:4064 , piece_id=7GVKAUOISAORFKOXUNOJMEEFNNUVVZUQ2RGCD6BNVCDTH6ONPQIQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1ZB3emcQntFxpr7zBiE98eesnjoTnNKinw3FPtGU2TgznGakeb, address:185.167.204.206:4064): protocol: rpc: dial tcp 185.167.204.206:4064: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295495543 (56h32m27.757588026s) [noise=true , address=185.167.204.206:4059 , piece_id=G4B2XQYXOG2LWBAIYBM4BWLI7PEPE2XDM5A57YHYR6SOWW7ODSVA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:121gTedy8TPx44wmLqdQpZ1gDDsL8D9K1EvQJffR7oKY3bT9DXh, address:185.167.204.206:4059): protocol: rpc: dial tcp 185.167.204.206:4059: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295562701 (56h32m27.757655184s) [noise=true , address=185.167.204.205:4030 , piece_id=U6SOG6BNEIWDGYWDRZOCLTROZVZXPPG5ZI52HLRPKTDKFVQN3EJA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12w5R2Whxgg4AYNNvTkKmFq4pzahF1uZiVzwgVygc9iZDAXMdSk, address:185.167.204.205:4030): protocol: rpc: dial tcp 185.167.204.205:4030: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295700453 (56h32m27.757792926s) [noise=true , address=185.167.204.205:4010 , piece_id=DPMSKQDDOU2PUXHINTG7JJXLKXJE2PC6B3KNSKN5UIGAKBM3GZVQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12oTu1GaRsH7fHqxLkQ9c9ikjG4zRN3NChssa5gfpLwrcEehoyV, address:185.167.204.205:4010): protocol: rpc: dial tcp 185.167.204.205:4010: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295713257 (56h32m27.75780574s) [noise=true , address=185.167.204.206:4053 , piece_id=74G7LWVYE7GO4GPSG67BD6QVE7UBLGIAKLSG447PWWLNBM2FZNTA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12SrFttJpndpv7TKJJYsKthWnGWBYQGHpS9CwqTd4mqBP9NCCm8, address:185.167.204.206:4053): protocol: rpc: dial tcp 185.167.204.206:4053: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295852051 (56h32m27.757944534s) [noise=true , address=185.167.204.206:4065 , piece_id=VKBLX4ASIKVHMUUCTD7GFFPEHJLKREGCDGI7LCVRSASDHBAWI6AQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12oPo26KZGZr7o7M18KZiX5PMAAMwHK8FPT59y1hSLhkB6z3Lcj, address:185.167.204.206:4065): protocol: rpc: dial tcp 185.167.204.206:4065: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295908840 (56h32m27.758001313s) [noise=true , address=185.167.204.206:4057 , piece_id=WNX7U6DN7HTE3XLVJAJ4FTCYFR3B2MLMLF5I46GALE463DWQ22WQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1xy38NywWy5CSJ7CfupY78vbkHg6cQzgNhM2Z1xnpvw9dwvMoT, address:185.167.204.206:4057): protocol: rpc: dial tcp 185.167.204.206:4057: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296027916 (56h32m27.758120399s) [noise=true , address=185.167.204.205:4003 , piece_id=NQX2I3KTMYFLS2B2C5Y2PGZRXCD7QJCZLABY4PTYVEFGAVCYX2EQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1CjWvvJ4FdZc41sRgsQFWz4Y1ATZKgzsvDcUTmBSrCC15ekCmF, address:185.167.204.205:4003): protocol: rpc: dial tcp 185.167.204.205:4003: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296100945 (56h32m27.758193418s) [noise=true , address=185.167.204.205:4032 , piece_id=AN4YMHK6NKM5RIWAZHX2FPBGQ3VRFRYBKHBPIAL2HQWBK5XXNNOA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12558dx7zMNDfabV55vSsP9XwdxXgoPM8aHW53femVZzkCRxsKN, address:185.167.204.205:4032): protocol: rpc: dial tcp 185.167.204.205:4032: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296150660 (56h32m27.758243133s) [noise=true , address=185.167.204.205:4029 , piece_id=UVEMQY4WGBKGGT75ILTBPUCFZ4U252QGPDTBPT2BNMOR3V5XYCPA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12DyF12w2JXdkGLfLx9VE1E5f28VnMg7M2b3osfsDtGX9o8Tvsn, address:185.167.204.205:4029): protocol: write tcp 188.166.225.170:36604->185.167.204.205:4029: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296276239 (56h32m27.758368722s) [noise=true , address=185.167.204.206:4069 , piece_id=VKW6XTWPZPD3MDANS3FIYYD2UPDW47WBCGLNTOVYVTWA5IT23DBQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12Uan1TT5oAM3axhaviVFbvh83gThiZvwHMwbCD31ZZens6BQ4U, address:185.167.204.206:4069): protocol: rpc: dial tcp 185.167.204.206:4069: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296438448 (56h32m27.758530931s) [noise=true , address=185.167.204.206:4048 , piece_id=PRBNW47DU6K26GYUUUZSYTYQO3GNFC6DEV2OYQDP4R2RV5L5VNXA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1sY4wHBkSVUDWWg8T8Vm1hpRbbpb3ttqnqDVY8uWyG27gqBwT9, address:185.167.204.206:4048): protocol: write tcp 188.166.225.170:59140->185.167.204.206:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296628780 (56h32m27.758721253s) [noise=true , address=185.167.204.206:4055 , piece_id=KH2E6T37MHNJJ56CR3EIKMEUEWMAVP2XNMZ7K5QVKCFZJIVXRTUA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12CYRieezm1mjtGchWdPrB2mNaHVRvTS85umjB6cQyYKnvisXSB, address:185.167.204.206:4055): protocol: rpc: dial tcp 185.167.204.206:4055: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.297155593 (56h32m27.759248076s) [noise=true , address=185.167.204.205:4022 , piece_id=VU4625JC4A4ZVXFGJZTDMDZXYCNXJ67OQTEKWRLMMHGHPEHY5J7A, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj, address:185.167.204.205:4022): protocol: write tcp 188.166.225.170:57938->185.167.204.205:4022: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.297184969 (56h32m27.759277442s) [seg_pos={0 0}, upload=49746]: Done waiting for segment. successful: 12 optimal: 26 errs: <nil>
Sep 25 22:03:42.465979471 (56h32m27.928071944s) [upload=49746]: Done uploading segments...
Sep 25 22:03:42.466010200 (56h32m27.928102673s) [upload=49746]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 12)
Duration: 4.837721278s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.eu1.gateway)antiblip:
1 out of 60 is failed (threshold=3) [.........................................................X..] duration:
209.582719ms remote:
"tcp:207.211.208.129:443" response headers:
Accept-Ranges=bytes Content-Length=352 Content-Security-Policy=block-all-mixed-content Server=Storj X-Request-Id=JdrtmpHLrtw X-Xss-Protection=1; mode=block Content-Type=application/xml Vary=Origin X-Amz-Request-Id=1868A6DD9BAF33BB Date=Thu, 25 Sep 2025 22:25:25 GMT target:
https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318
FAIL (2025-09-25 22:26:45 UTC)
orbiter warning: unexpected status code "500 Internal Server Error" "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>InternalError</Code><Message>We encountered an internal error, please try again.: cause(uplink: metaclient: context canceled)</Message><Key>hello-world.txt</Key><BucketName>uptime-checks</BucketName><Resource>/hello-world.txt</Resource><RequestId>1868A6DD9BAF33BB</RequestId><HostId></HostId></Error>"
Duration: 209.613108ms
edge
gateway
vhost
bgp
eu1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
2 out of 60 is failed (threshold=3) [.......................XX...................................] duration:
4.843676628s
FAIL (2025-09-25 22:26:41 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 12)
======LOGS======
9 , piece_id=4YT7BXXDQVTR2QL4YBWVMO3YJCZQ57PTTEQW7GCZELWC42HQUDQQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.227500377 (56h31m29.68959285s) [noise=true , address=81.171.14.70:4034 , piece_id=EXVA3SFFPD3GJDYIFHAOP7GMJQHGZECMX5R6MHLRDMMCU7LOA7KA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.253492519 (56h31m29.715584993s) [noise=true , address=81.171.14.70:4028 , piece_id=Y237IBPHRRI7NMOYJG65AZI3LCCZ6JUJPX5WWGIDX5RYHCNVIG3A, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.267890396 (56h31m29.729982869s) [noise=true , address=89.149.201.147:4010 , piece_id=ANMXRI22IXAZFHKB7ZHFGEERKT2KGUYQEFKEOJNJDC5TFB6B3STQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.280031865 (56h31m29.742124348s) [noise=true , address=89.149.201.147:4020 , piece_id=RML2JG4QZSS6H5GTP7BW3B4SFVZLLHHAOUS3LDCOT2BGYTC3HACA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.294783955 (56h31m29.756876428s) [noise=true , address=89.149.201.147:4017 , piece_id=P576E6UE5UADQZYKR4XN4I5X4MCGMXT3B7PBGVNWBZ6I67ZVCJHA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.462969447 (56h31m29.92506192s) [noise=true , address=81.171.14.70:4035 , piece_id=GWVM3MZRGRVJ5RWG227JJXTULF5MUJ4FHWCNHLW7J3GDYXRSXRXA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.668506081 (56h31m30.130598554s) [noise=true , address=81.171.14.70:4045 , piece_id=BSKEEYML4LQZ7PWB45HQNX2YFJTFIEKIILDUZIWCXMVZZGGKHGUQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.702001372 (56h31m30.164093845s) [noise=true , address=89.149.201.147:4014 , piece_id=3VH5QXDWYX2NR3LODHSATJCP5ABF7BI52GJAR3VLPW42CPJJXQNA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.728242188 (56h31m30.190334661s) [noise=true , address=89.149.201.147:4016 , piece_id=PT5TUJDTS2TDBNLH5722GJTZ5BIO4LZKEDKBGUKHEF37TPADW7MA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.797046563 (56h31m30.259139036s) [noise=true , address=89.149.201.147:4012 , piece_id=TY6OVHC4ITI2E32GE7LP3O2YQ46FKWF6A6IIBN4JFAB7WPYXXJ2A, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:03:42.295273049 (56h32m27.757365543s) [noise=true , address=185.167.204.206:4056 , piece_id=4XAHI4JY2MPISEOZ6VJ5GCJ5WZJKHRXIDK2EQGEN7VSYNRYJH24A, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1uSTuNA28oH2s5Rkv2jaKoC2nZWSgpMJXyxwFTCKbr44EvEXrC, address:185.167.204.206:4056): protocol: rpc: dial tcp 185.167.204.206:4056: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295296423 (56h32m27.757388906s) [noise=true , address=185.167.204.205:4008 , piece_id=NXU5IGU2NUZPVR5KIUKLZ7QTPYLGDIKZQNMPBJRPCVYEVNIV7HMA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12ZHTTaGQUTftxEULFqWbH6w1HnWA7PavHrXnvMCD947FxtmsrS, address:185.167.204.205:4008): protocol: rpc: dial tcp 185.167.204.205:4008: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295427543 (56h32m27.757520026s) [noise=true , address=185.167.204.206:4064 , piece_id=7GVKAUOISAORFKOXUNOJMEEFNNUVVZUQ2RGCD6BNVCDTH6ONPQIQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1ZB3emcQntFxpr7zBiE98eesnjoTnNKinw3FPtGU2TgznGakeb, address:185.167.204.206:4064): protocol: rpc: dial tcp 185.167.204.206:4064: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295495543 (56h32m27.757588026s) [noise=true , address=185.167.204.206:4059 , piece_id=G4B2XQYXOG2LWBAIYBM4BWLI7PEPE2XDM5A57YHYR6SOWW7ODSVA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:121gTedy8TPx44wmLqdQpZ1gDDsL8D9K1EvQJffR7oKY3bT9DXh, address:185.167.204.206:4059): protocol: rpc: dial tcp 185.167.204.206:4059: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295562701 (56h32m27.757655184s) [noise=true , address=185.167.204.205:4030 , piece_id=U6SOG6BNEIWDGYWDRZOCLTROZVZXPPG5ZI52HLRPKTDKFVQN3EJA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12w5R2Whxgg4AYNNvTkKmFq4pzahF1uZiVzwgVygc9iZDAXMdSk, address:185.167.204.205:4030): protocol: rpc: dial tcp 185.167.204.205:4030: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295700453 (56h32m27.757792926s) [noise=true , address=185.167.204.205:4010 , piece_id=DPMSKQDDOU2PUXHINTG7JJXLKXJE2PC6B3KNSKN5UIGAKBM3GZVQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12oTu1GaRsH7fHqxLkQ9c9ikjG4zRN3NChssa5gfpLwrcEehoyV, address:185.167.204.205:4010): protocol: rpc: dial tcp 185.167.204.205:4010: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295713257 (56h32m27.75780574s) [noise=true , address=185.167.204.206:4053 , piece_id=74G7LWVYE7GO4GPSG67BD6QVE7UBLGIAKLSG447PWWLNBM2FZNTA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12SrFttJpndpv7TKJJYsKthWnGWBYQGHpS9CwqTd4mqBP9NCCm8, address:185.167.204.206:4053): protocol: rpc: dial tcp 185.167.204.206:4053: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295852051 (56h32m27.757944534s) [noise=true , address=185.167.204.206:4065 , piece_id=VKBLX4ASIKVHMUUCTD7GFFPEHJLKREGCDGI7LCVRSASDHBAWI6AQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12oPo26KZGZr7o7M18KZiX5PMAAMwHK8FPT59y1hSLhkB6z3Lcj, address:185.167.204.206:4065): protocol: rpc: dial tcp 185.167.204.206:4065: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295908840 (56h32m27.758001313s) [noise=true , address=185.167.204.206:4057 , piece_id=WNX7U6DN7HTE3XLVJAJ4FTCYFR3B2MLMLF5I46GALE463DWQ22WQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1xy38NywWy5CSJ7CfupY78vbkHg6cQzgNhM2Z1xnpvw9dwvMoT, address:185.167.204.206:4057): protocol: rpc: dial tcp 185.167.204.206:4057: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296027916 (56h32m27.758120399s) [noise=true , address=185.167.204.205:4003 , piece_id=NQX2I3KTMYFLS2B2C5Y2PGZRXCD7QJCZLABY4PTYVEFGAVCYX2EQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1CjWvvJ4FdZc41sRgsQFWz4Y1ATZKgzsvDcUTmBSrCC15ekCmF, address:185.167.204.205:4003): protocol: rpc: dial tcp 185.167.204.205:4003: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296100945 (56h32m27.758193418s) [noise=true , address=185.167.204.205:4032 , piece_id=AN4YMHK6NKM5RIWAZHX2FPBGQ3VRFRYBKHBPIAL2HQWBK5XXNNOA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12558dx7zMNDfabV55vSsP9XwdxXgoPM8aHW53femVZzkCRxsKN, address:185.167.204.205:4032): protocol: rpc: dial tcp 185.167.204.205:4032: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296150660 (56h32m27.758243133s) [noise=true , address=185.167.204.205:4029 , piece_id=UVEMQY4WGBKGGT75ILTBPUCFZ4U252QGPDTBPT2BNMOR3V5XYCPA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12DyF12w2JXdkGLfLx9VE1E5f28VnMg7M2b3osfsDtGX9o8Tvsn, address:185.167.204.205:4029): protocol: write tcp 188.166.225.170:36604->185.167.204.205:4029: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296276239 (56h32m27.758368722s) [noise=true , address=185.167.204.206:4069 , piece_id=VKW6XTWPZPD3MDANS3FIYYD2UPDW47WBCGLNTOVYVTWA5IT23DBQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12Uan1TT5oAM3axhaviVFbvh83gThiZvwHMwbCD31ZZens6BQ4U, address:185.167.204.206:4069): protocol: rpc: dial tcp 185.167.204.206:4069: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296438448 (56h32m27.758530931s) [noise=true , address=185.167.204.206:4048 , piece_id=PRBNW47DU6K26GYUUUZSYTYQO3GNFC6DEV2OYQDP4R2RV5L5VNXA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1sY4wHBkSVUDWWg8T8Vm1hpRbbpb3ttqnqDVY8uWyG27gqBwT9, address:185.167.204.206:4048): protocol: write tcp 188.166.225.170:59140->185.167.204.206:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296628780 (56h32m27.758721253s) [noise=true , address=185.167.204.206:4055 , piece_id=KH2E6T37MHNJJ56CR3EIKMEUEWMAVP2XNMZ7K5QVKCFZJIVXRTUA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12CYRieezm1mjtGchWdPrB2mNaHVRvTS85umjB6cQyYKnvisXSB, address:185.167.204.206:4055): protocol: rpc: dial tcp 185.167.204.206:4055: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.297155593 (56h32m27.759248076s) [noise=true , address=185.167.204.205:4022 , piece_id=VU4625JC4A4ZVXFGJZTDMDZXYCNXJ67OQTEKWRLMMHGHPEHY5J7A, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj, address:185.167.204.205:4022): protocol: write tcp 188.166.225.170:57938->185.167.204.205:4022: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.297184969 (56h32m27.759277442s) [seg_pos={0 0}, upload=49746]: Done waiting for segment. successful: 12 optimal: 26 errs: <nil>
Sep 25 22:03:42.465979471 (56h32m27.928071944s) [upload=49746]: Done uploading segments...
Sep 25 22:03:42.466010200 (56h32m27.928102673s) [upload=49746]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 12)
Duration: 4.843706755s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.eu1.gateway)antiblip:
1 out of 60 is failed (threshold=3) [..........................................................X.] duration:
179.642638ms remote:
"tcp:207.211.208.129:443" response headers:
Accept-Ranges=bytes Content-Length=352 Content-Security-Policy=block-all-mixed-content Server=Storj X-Request-Id=JdrtmpHLrtw X-Xss-Protection=1; mode=block Content-Type=application/xml Vary=Origin X-Amz-Request-Id=1868A6DD9BAF33BB Date=Thu, 25 Sep 2025 22:25:25 GMT target:
https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318
FAIL (2025-09-25 22:26:27 UTC)
orbiter warning: unexpected status code "500 Internal Server Error" "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>InternalError</Code><Message>We encountered an internal error, please try again.: cause(uplink: metaclient: context canceled)</Message><Key>hello-world.txt</Key><BucketName>uptime-checks</BucketName><Resource>/hello-world.txt</Resource><RequestId>1868A6DD9BAF33BB</RequestId><HostId></HostId></Error>"
Duration: 179.672526ms
edge
gateway
vhost
bgp
eu1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
2 out of 60 is failed (threshold=3) [........................XX..................................] duration:
4.841263246s
FAIL (2025-09-25 22:25:30 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 12)
======LOGS======
9 , piece_id=4YT7BXXDQVTR2QL4YBWVMO3YJCZQ57PTTEQW7GCZELWC42HQUDQQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.227500377 (56h31m29.68959285s) [noise=true , address=81.171.14.70:4034 , piece_id=EXVA3SFFPD3GJDYIFHAOP7GMJQHGZECMX5R6MHLRDMMCU7LOA7KA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.253492519 (56h31m29.715584993s) [noise=true , address=81.171.14.70:4028 , piece_id=Y237IBPHRRI7NMOYJG65AZI3LCCZ6JUJPX5WWGIDX5RYHCNVIG3A, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.267890396 (56h31m29.729982869s) [noise=true , address=89.149.201.147:4010 , piece_id=ANMXRI22IXAZFHKB7ZHFGEERKT2KGUYQEFKEOJNJDC5TFB6B3STQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.280031865 (56h31m29.742124348s) [noise=true , address=89.149.201.147:4020 , piece_id=RML2JG4QZSS6H5GTP7BW3B4SFVZLLHHAOUS3LDCOT2BGYTC3HACA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.294783955 (56h31m29.756876428s) [noise=true , address=89.149.201.147:4017 , piece_id=P576E6UE5UADQZYKR4XN4I5X4MCGMXT3B7PBGVNWBZ6I67ZVCJHA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.462969447 (56h31m29.92506192s) [noise=true , address=81.171.14.70:4035 , piece_id=GWVM3MZRGRVJ5RWG227JJXTULF5MUJ4FHWCNHLW7J3GDYXRSXRXA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.668506081 (56h31m30.130598554s) [noise=true , address=81.171.14.70:4045 , piece_id=BSKEEYML4LQZ7PWB45HQNX2YFJTFIEKIILDUZIWCXMVZZGGKHGUQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.702001372 (56h31m30.164093845s) [noise=true , address=89.149.201.147:4014 , piece_id=3VH5QXDWYX2NR3LODHSATJCP5ABF7BI52GJAR3VLPW42CPJJXQNA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.728242188 (56h31m30.190334661s) [noise=true , address=89.149.201.147:4016 , piece_id=PT5TUJDTS2TDBNLH5722GJTZ5BIO4LZKEDKBGUKHEF37TPADW7MA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.797046563 (56h31m30.259139036s) [noise=true , address=89.149.201.147:4012 , piece_id=TY6OVHC4ITI2E32GE7LP3O2YQ46FKWF6A6IIBN4JFAB7WPYXXJ2A, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:03:42.295273049 (56h32m27.757365543s) [noise=true , address=185.167.204.206:4056 , piece_id=4XAHI4JY2MPISEOZ6VJ5GCJ5WZJKHRXIDK2EQGEN7VSYNRYJH24A, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1uSTuNA28oH2s5Rkv2jaKoC2nZWSgpMJXyxwFTCKbr44EvEXrC, address:185.167.204.206:4056): protocol: rpc: dial tcp 185.167.204.206:4056: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295296423 (56h32m27.757388906s) [noise=true , address=185.167.204.205:4008 , piece_id=NXU5IGU2NUZPVR5KIUKLZ7QTPYLGDIKZQNMPBJRPCVYEVNIV7HMA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12ZHTTaGQUTftxEULFqWbH6w1HnWA7PavHrXnvMCD947FxtmsrS, address:185.167.204.205:4008): protocol: rpc: dial tcp 185.167.204.205:4008: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295427543 (56h32m27.757520026s) [noise=true , address=185.167.204.206:4064 , piece_id=7GVKAUOISAORFKOXUNOJMEEFNNUVVZUQ2RGCD6BNVCDTH6ONPQIQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1ZB3emcQntFxpr7zBiE98eesnjoTnNKinw3FPtGU2TgznGakeb, address:185.167.204.206:4064): protocol: rpc: dial tcp 185.167.204.206:4064: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295495543 (56h32m27.757588026s) [noise=true , address=185.167.204.206:4059 , piece_id=G4B2XQYXOG2LWBAIYBM4BWLI7PEPE2XDM5A57YHYR6SOWW7ODSVA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:121gTedy8TPx44wmLqdQpZ1gDDsL8D9K1EvQJffR7oKY3bT9DXh, address:185.167.204.206:4059): protocol: rpc: dial tcp 185.167.204.206:4059: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295562701 (56h32m27.757655184s) [noise=true , address=185.167.204.205:4030 , piece_id=U6SOG6BNEIWDGYWDRZOCLTROZVZXPPG5ZI52HLRPKTDKFVQN3EJA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12w5R2Whxgg4AYNNvTkKmFq4pzahF1uZiVzwgVygc9iZDAXMdSk, address:185.167.204.205:4030): protocol: rpc: dial tcp 185.167.204.205:4030: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295700453 (56h32m27.757792926s) [noise=true , address=185.167.204.205:4010 , piece_id=DPMSKQDDOU2PUXHINTG7JJXLKXJE2PC6B3KNSKN5UIGAKBM3GZVQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12oTu1GaRsH7fHqxLkQ9c9ikjG4zRN3NChssa5gfpLwrcEehoyV, address:185.167.204.205:4010): protocol: rpc: dial tcp 185.167.204.205:4010: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295713257 (56h32m27.75780574s) [noise=true , address=185.167.204.206:4053 , piece_id=74G7LWVYE7GO4GPSG67BD6QVE7UBLGIAKLSG447PWWLNBM2FZNTA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12SrFttJpndpv7TKJJYsKthWnGWBYQGHpS9CwqTd4mqBP9NCCm8, address:185.167.204.206:4053): protocol: rpc: dial tcp 185.167.204.206:4053: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295852051 (56h32m27.757944534s) [noise=true , address=185.167.204.206:4065 , piece_id=VKBLX4ASIKVHMUUCTD7GFFPEHJLKREGCDGI7LCVRSASDHBAWI6AQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12oPo26KZGZr7o7M18KZiX5PMAAMwHK8FPT59y1hSLhkB6z3Lcj, address:185.167.204.206:4065): protocol: rpc: dial tcp 185.167.204.206:4065: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295908840 (56h32m27.758001313s) [noise=true , address=185.167.204.206:4057 , piece_id=WNX7U6DN7HTE3XLVJAJ4FTCYFR3B2MLMLF5I46GALE463DWQ22WQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1xy38NywWy5CSJ7CfupY78vbkHg6cQzgNhM2Z1xnpvw9dwvMoT, address:185.167.204.206:4057): protocol: rpc: dial tcp 185.167.204.206:4057: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296027916 (56h32m27.758120399s) [noise=true , address=185.167.204.205:4003 , piece_id=NQX2I3KTMYFLS2B2C5Y2PGZRXCD7QJCZLABY4PTYVEFGAVCYX2EQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1CjWvvJ4FdZc41sRgsQFWz4Y1ATZKgzsvDcUTmBSrCC15ekCmF, address:185.167.204.205:4003): protocol: rpc: dial tcp 185.167.204.205:4003: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296100945 (56h32m27.758193418s) [noise=true , address=185.167.204.205:4032 , piece_id=AN4YMHK6NKM5RIWAZHX2FPBGQ3VRFRYBKHBPIAL2HQWBK5XXNNOA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12558dx7zMNDfabV55vSsP9XwdxXgoPM8aHW53femVZzkCRxsKN, address:185.167.204.205:4032): protocol: rpc: dial tcp 185.167.204.205:4032: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296150660 (56h32m27.758243133s) [noise=true , address=185.167.204.205:4029 , piece_id=UVEMQY4WGBKGGT75ILTBPUCFZ4U252QGPDTBPT2BNMOR3V5XYCPA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12DyF12w2JXdkGLfLx9VE1E5f28VnMg7M2b3osfsDtGX9o8Tvsn, address:185.167.204.205:4029): protocol: write tcp 188.166.225.170:36604->185.167.204.205:4029: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296276239 (56h32m27.758368722s) [noise=true , address=185.167.204.206:4069 , piece_id=VKW6XTWPZPD3MDANS3FIYYD2UPDW47WBCGLNTOVYVTWA5IT23DBQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12Uan1TT5oAM3axhaviVFbvh83gThiZvwHMwbCD31ZZens6BQ4U, address:185.167.204.206:4069): protocol: rpc: dial tcp 185.167.204.206:4069: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296438448 (56h32m27.758530931s) [noise=true , address=185.167.204.206:4048 , piece_id=PRBNW47DU6K26GYUUUZSYTYQO3GNFC6DEV2OYQDP4R2RV5L5VNXA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1sY4wHBkSVUDWWg8T8Vm1hpRbbpb3ttqnqDVY8uWyG27gqBwT9, address:185.167.204.206:4048): protocol: write tcp 188.166.225.170:59140->185.167.204.206:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296628780 (56h32m27.758721253s) [noise=true , address=185.167.204.206:4055 , piece_id=KH2E6T37MHNJJ56CR3EIKMEUEWMAVP2XNMZ7K5QVKCFZJIVXRTUA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12CYRieezm1mjtGchWdPrB2mNaHVRvTS85umjB6cQyYKnvisXSB, address:185.167.204.206:4055): protocol: rpc: dial tcp 185.167.204.206:4055: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.297155593 (56h32m27.759248076s) [noise=true , address=185.167.204.205:4022 , piece_id=VU4625JC4A4ZVXFGJZTDMDZXYCNXJ67OQTEKWRLMMHGHPEHY5J7A, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj, address:185.167.204.205:4022): protocol: write tcp 188.166.225.170:57938->185.167.204.205:4022: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.297184969 (56h32m27.759277442s) [seg_pos={0 0}, upload=49746]: Done waiting for segment. successful: 12 optimal: 26 errs: <nil>
Sep 25 22:03:42.465979471 (56h32m27.928071944s) [upload=49746]: Done uploading segments...
Sep 25 22:03:42.466010200 (56h32m27.928102673s) [upload=49746]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 12)
Duration: 4.841305608s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.eu1.gateway)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
217.26078ms remote:
"tcp:207.211.208.129:443" response headers:
Accept-Ranges=bytes Content-Length=352 Content-Security-Policy=block-all-mixed-content Server=Storj X-Request-Id=JdrtmpHLrtw X-Xss-Protection=1; mode=block Content-Type=application/xml Vary=Origin X-Amz-Request-Id=1868A6DD9BAF33BB Date=Thu, 25 Sep 2025 22:25:25 GMT target:
https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jwhnbh22a7huxgm5mpnu45kaahuq&Signature=gznnRNzM%2FpTrDIdAeC%2Bpz3LKg0A%3D&Expires=1771549318
FAIL (2025-09-25 22:25:25 UTC)
orbiter warning: unexpected status code "500 Internal Server Error" "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>InternalError</Code><Message>We encountered an internal error, please try again.: cause(uplink: metaclient: context canceled)</Message><Key>hello-world.txt</Key><BucketName>uptime-checks</BucketName><Resource>/hello-world.txt</Resource><RequestId>1868A6DD9BAF33BB</RequestId><HostId></HostId></Error>"
Duration: 217.293161ms
edge
gateway
vhost
bgp
eu1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
2 out of 60 is failed (threshold=3) [.........................XX.................................] duration:
4.925089291s
FAIL (2025-09-25 22:24:56 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 12)
======LOGS======
9 , piece_id=4YT7BXXDQVTR2QL4YBWVMO3YJCZQ57PTTEQW7GCZELWC42HQUDQQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.227500377 (56h31m29.68959285s) [noise=true , address=81.171.14.70:4034 , piece_id=EXVA3SFFPD3GJDYIFHAOP7GMJQHGZECMX5R6MHLRDMMCU7LOA7KA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.253492519 (56h31m29.715584993s) [noise=true , address=81.171.14.70:4028 , piece_id=Y237IBPHRRI7NMOYJG65AZI3LCCZ6JUJPX5WWGIDX5RYHCNVIG3A, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.267890396 (56h31m29.729982869s) [noise=true , address=89.149.201.147:4010 , piece_id=ANMXRI22IXAZFHKB7ZHFGEERKT2KGUYQEFKEOJNJDC5TFB6B3STQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.280031865 (56h31m29.742124348s) [noise=true , address=89.149.201.147:4020 , piece_id=RML2JG4QZSS6H5GTP7BW3B4SFVZLLHHAOUS3LDCOT2BGYTC3HACA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.294783955 (56h31m29.756876428s) [noise=true , address=89.149.201.147:4017 , piece_id=P576E6UE5UADQZYKR4XN4I5X4MCGMXT3B7PBGVNWBZ6I67ZVCJHA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.462969447 (56h31m29.92506192s) [noise=true , address=81.171.14.70:4035 , piece_id=GWVM3MZRGRVJ5RWG227JJXTULF5MUJ4FHWCNHLW7J3GDYXRSXRXA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.668506081 (56h31m30.130598554s) [noise=true , address=81.171.14.70:4045 , piece_id=BSKEEYML4LQZ7PWB45HQNX2YFJTFIEKIILDUZIWCXMVZZGGKHGUQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.702001372 (56h31m30.164093845s) [noise=true , address=89.149.201.147:4014 , piece_id=3VH5QXDWYX2NR3LODHSATJCP5ABF7BI52GJAR3VLPW42CPJJXQNA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.728242188 (56h31m30.190334661s) [noise=true , address=89.149.201.147:4016 , piece_id=PT5TUJDTS2TDBNLH5722GJTZ5BIO4LZKEDKBGUKHEF37TPADW7MA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.797046563 (56h31m30.259139036s) [noise=true , address=89.149.201.147:4012 , piece_id=TY6OVHC4ITI2E32GE7LP3O2YQ46FKWF6A6IIBN4JFAB7WPYXXJ2A, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:03:42.295273049 (56h32m27.757365543s) [noise=true , address=185.167.204.206:4056 , piece_id=4XAHI4JY2MPISEOZ6VJ5GCJ5WZJKHRXIDK2EQGEN7VSYNRYJH24A, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1uSTuNA28oH2s5Rkv2jaKoC2nZWSgpMJXyxwFTCKbr44EvEXrC, address:185.167.204.206:4056): protocol: rpc: dial tcp 185.167.204.206:4056: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295296423 (56h32m27.757388906s) [noise=true , address=185.167.204.205:4008 , piece_id=NXU5IGU2NUZPVR5KIUKLZ7QTPYLGDIKZQNMPBJRPCVYEVNIV7HMA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12ZHTTaGQUTftxEULFqWbH6w1HnWA7PavHrXnvMCD947FxtmsrS, address:185.167.204.205:4008): protocol: rpc: dial tcp 185.167.204.205:4008: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295427543 (56h32m27.757520026s) [noise=true , address=185.167.204.206:4064 , piece_id=7GVKAUOISAORFKOXUNOJMEEFNNUVVZUQ2RGCD6BNVCDTH6ONPQIQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1ZB3emcQntFxpr7zBiE98eesnjoTnNKinw3FPtGU2TgznGakeb, address:185.167.204.206:4064): protocol: rpc: dial tcp 185.167.204.206:4064: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295495543 (56h32m27.757588026s) [noise=true , address=185.167.204.206:4059 , piece_id=G4B2XQYXOG2LWBAIYBM4BWLI7PEPE2XDM5A57YHYR6SOWW7ODSVA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:121gTedy8TPx44wmLqdQpZ1gDDsL8D9K1EvQJffR7oKY3bT9DXh, address:185.167.204.206:4059): protocol: rpc: dial tcp 185.167.204.206:4059: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295562701 (56h32m27.757655184s) [noise=true , address=185.167.204.205:4030 , piece_id=U6SOG6BNEIWDGYWDRZOCLTROZVZXPPG5ZI52HLRPKTDKFVQN3EJA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12w5R2Whxgg4AYNNvTkKmFq4pzahF1uZiVzwgVygc9iZDAXMdSk, address:185.167.204.205:4030): protocol: rpc: dial tcp 185.167.204.205:4030: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295700453 (56h32m27.757792926s) [noise=true , address=185.167.204.205:4010 , piece_id=DPMSKQDDOU2PUXHINTG7JJXLKXJE2PC6B3KNSKN5UIGAKBM3GZVQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12oTu1GaRsH7fHqxLkQ9c9ikjG4zRN3NChssa5gfpLwrcEehoyV, address:185.167.204.205:4010): protocol: rpc: dial tcp 185.167.204.205:4010: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295713257 (56h32m27.75780574s) [noise=true , address=185.167.204.206:4053 , piece_id=74G7LWVYE7GO4GPSG67BD6QVE7UBLGIAKLSG447PWWLNBM2FZNTA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12SrFttJpndpv7TKJJYsKthWnGWBYQGHpS9CwqTd4mqBP9NCCm8, address:185.167.204.206:4053): protocol: rpc: dial tcp 185.167.204.206:4053: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295852051 (56h32m27.757944534s) [noise=true , address=185.167.204.206:4065 , piece_id=VKBLX4ASIKVHMUUCTD7GFFPEHJLKREGCDGI7LCVRSASDHBAWI6AQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12oPo26KZGZr7o7M18KZiX5PMAAMwHK8FPT59y1hSLhkB6z3Lcj, address:185.167.204.206:4065): protocol: rpc: dial tcp 185.167.204.206:4065: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295908840 (56h32m27.758001313s) [noise=true , address=185.167.204.206:4057 , piece_id=WNX7U6DN7HTE3XLVJAJ4FTCYFR3B2MLMLF5I46GALE463DWQ22WQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1xy38NywWy5CSJ7CfupY78vbkHg6cQzgNhM2Z1xnpvw9dwvMoT, address:185.167.204.206:4057): protocol: rpc: dial tcp 185.167.204.206:4057: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296027916 (56h32m27.758120399s) [noise=true , address=185.167.204.205:4003 , piece_id=NQX2I3KTMYFLS2B2C5Y2PGZRXCD7QJCZLABY4PTYVEFGAVCYX2EQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1CjWvvJ4FdZc41sRgsQFWz4Y1ATZKgzsvDcUTmBSrCC15ekCmF, address:185.167.204.205:4003): protocol: rpc: dial tcp 185.167.204.205:4003: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296100945 (56h32m27.758193418s) [noise=true , address=185.167.204.205:4032 , piece_id=AN4YMHK6NKM5RIWAZHX2FPBGQ3VRFRYBKHBPIAL2HQWBK5XXNNOA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12558dx7zMNDfabV55vSsP9XwdxXgoPM8aHW53femVZzkCRxsKN, address:185.167.204.205:4032): protocol: rpc: dial tcp 185.167.204.205:4032: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296150660 (56h32m27.758243133s) [noise=true , address=185.167.204.205:4029 , piece_id=UVEMQY4WGBKGGT75ILTBPUCFZ4U252QGPDTBPT2BNMOR3V5XYCPA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12DyF12w2JXdkGLfLx9VE1E5f28VnMg7M2b3osfsDtGX9o8Tvsn, address:185.167.204.205:4029): protocol: write tcp 188.166.225.170:36604->185.167.204.205:4029: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296276239 (56h32m27.758368722s) [noise=true , address=185.167.204.206:4069 , piece_id=VKW6XTWPZPD3MDANS3FIYYD2UPDW47WBCGLNTOVYVTWA5IT23DBQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12Uan1TT5oAM3axhaviVFbvh83gThiZvwHMwbCD31ZZens6BQ4U, address:185.167.204.206:4069): protocol: rpc: dial tcp 185.167.204.206:4069: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296438448 (56h32m27.758530931s) [noise=true , address=185.167.204.206:4048 , piece_id=PRBNW47DU6K26GYUUUZSYTYQO3GNFC6DEV2OYQDP4R2RV5L5VNXA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1sY4wHBkSVUDWWg8T8Vm1hpRbbpb3ttqnqDVY8uWyG27gqBwT9, address:185.167.204.206:4048): protocol: write tcp 188.166.225.170:59140->185.167.204.206:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296628780 (56h32m27.758721253s) [noise=true , address=185.167.204.206:4055 , piece_id=KH2E6T37MHNJJ56CR3EIKMEUEWMAVP2XNMZ7K5QVKCFZJIVXRTUA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12CYRieezm1mjtGchWdPrB2mNaHVRvTS85umjB6cQyYKnvisXSB, address:185.167.204.206:4055): protocol: rpc: dial tcp 185.167.204.206:4055: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.297155593 (56h32m27.759248076s) [noise=true , address=185.167.204.205:4022 , piece_id=VU4625JC4A4ZVXFGJZTDMDZXYCNXJ67OQTEKWRLMMHGHPEHY5J7A, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj, address:185.167.204.205:4022): protocol: write tcp 188.166.225.170:57938->185.167.204.205:4022: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.297184969 (56h32m27.759277442s) [seg_pos={0 0}, upload=49746]: Done waiting for segment. successful: 12 optimal: 26 errs: <nil>
Sep 25 22:03:42.465979471 (56h32m27.928071944s) [upload=49746]: Done uploading segments...
Sep 25 22:03:42.466010200 (56h32m27.928102673s) [upload=49746]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 12)
Duration: 4.925117926s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
2 out of 60 is failed (threshold=3) [..........................XX................................] duration:
4.86803576s
FAIL (2025-09-25 22:24:18 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 12)
======LOGS======
9 , piece_id=4YT7BXXDQVTR2QL4YBWVMO3YJCZQ57PTTEQW7GCZELWC42HQUDQQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.227500377 (56h31m29.68959285s) [noise=true , address=81.171.14.70:4034 , piece_id=EXVA3SFFPD3GJDYIFHAOP7GMJQHGZECMX5R6MHLRDMMCU7LOA7KA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.253492519 (56h31m29.715584993s) [noise=true , address=81.171.14.70:4028 , piece_id=Y237IBPHRRI7NMOYJG65AZI3LCCZ6JUJPX5WWGIDX5RYHCNVIG3A, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.267890396 (56h31m29.729982869s) [noise=true , address=89.149.201.147:4010 , piece_id=ANMXRI22IXAZFHKB7ZHFGEERKT2KGUYQEFKEOJNJDC5TFB6B3STQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.280031865 (56h31m29.742124348s) [noise=true , address=89.149.201.147:4020 , piece_id=RML2JG4QZSS6H5GTP7BW3B4SFVZLLHHAOUS3LDCOT2BGYTC3HACA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.294783955 (56h31m29.756876428s) [noise=true , address=89.149.201.147:4017 , piece_id=P576E6UE5UADQZYKR4XN4I5X4MCGMXT3B7PBGVNWBZ6I67ZVCJHA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.462969447 (56h31m29.92506192s) [noise=true , address=81.171.14.70:4035 , piece_id=GWVM3MZRGRVJ5RWG227JJXTULF5MUJ4FHWCNHLW7J3GDYXRSXRXA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.668506081 (56h31m30.130598554s) [noise=true , address=81.171.14.70:4045 , piece_id=BSKEEYML4LQZ7PWB45HQNX2YFJTFIEKIILDUZIWCXMVZZGGKHGUQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.702001372 (56h31m30.164093845s) [noise=true , address=89.149.201.147:4014 , piece_id=3VH5QXDWYX2NR3LODHSATJCP5ABF7BI52GJAR3VLPW42CPJJXQNA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.728242188 (56h31m30.190334661s) [noise=true , address=89.149.201.147:4016 , piece_id=PT5TUJDTS2TDBNLH5722GJTZ5BIO4LZKEDKBGUKHEF37TPADW7MA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.797046563 (56h31m30.259139036s) [noise=true , address=89.149.201.147:4012 , piece_id=TY6OVHC4ITI2E32GE7LP3O2YQ46FKWF6A6IIBN4JFAB7WPYXXJ2A, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:03:42.295273049 (56h32m27.757365543s) [noise=true , address=185.167.204.206:4056 , piece_id=4XAHI4JY2MPISEOZ6VJ5GCJ5WZJKHRXIDK2EQGEN7VSYNRYJH24A, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1uSTuNA28oH2s5Rkv2jaKoC2nZWSgpMJXyxwFTCKbr44EvEXrC, address:185.167.204.206:4056): protocol: rpc: dial tcp 185.167.204.206:4056: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295296423 (56h32m27.757388906s) [noise=true , address=185.167.204.205:4008 , piece_id=NXU5IGU2NUZPVR5KIUKLZ7QTPYLGDIKZQNMPBJRPCVYEVNIV7HMA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12ZHTTaGQUTftxEULFqWbH6w1HnWA7PavHrXnvMCD947FxtmsrS, address:185.167.204.205:4008): protocol: rpc: dial tcp 185.167.204.205:4008: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295427543 (56h32m27.757520026s) [noise=true , address=185.167.204.206:4064 , piece_id=7GVKAUOISAORFKOXUNOJMEEFNNUVVZUQ2RGCD6BNVCDTH6ONPQIQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1ZB3emcQntFxpr7zBiE98eesnjoTnNKinw3FPtGU2TgznGakeb, address:185.167.204.206:4064): protocol: rpc: dial tcp 185.167.204.206:4064: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295495543 (56h32m27.757588026s) [noise=true , address=185.167.204.206:4059 , piece_id=G4B2XQYXOG2LWBAIYBM4BWLI7PEPE2XDM5A57YHYR6SOWW7ODSVA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:121gTedy8TPx44wmLqdQpZ1gDDsL8D9K1EvQJffR7oKY3bT9DXh, address:185.167.204.206:4059): protocol: rpc: dial tcp 185.167.204.206:4059: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295562701 (56h32m27.757655184s) [noise=true , address=185.167.204.205:4030 , piece_id=U6SOG6BNEIWDGYWDRZOCLTROZVZXPPG5ZI52HLRPKTDKFVQN3EJA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12w5R2Whxgg4AYNNvTkKmFq4pzahF1uZiVzwgVygc9iZDAXMdSk, address:185.167.204.205:4030): protocol: rpc: dial tcp 185.167.204.205:4030: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295700453 (56h32m27.757792926s) [noise=true , address=185.167.204.205:4010 , piece_id=DPMSKQDDOU2PUXHINTG7JJXLKXJE2PC6B3KNSKN5UIGAKBM3GZVQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12oTu1GaRsH7fHqxLkQ9c9ikjG4zRN3NChssa5gfpLwrcEehoyV, address:185.167.204.205:4010): protocol: rpc: dial tcp 185.167.204.205:4010: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295713257 (56h32m27.75780574s) [noise=true , address=185.167.204.206:4053 , piece_id=74G7LWVYE7GO4GPSG67BD6QVE7UBLGIAKLSG447PWWLNBM2FZNTA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12SrFttJpndpv7TKJJYsKthWnGWBYQGHpS9CwqTd4mqBP9NCCm8, address:185.167.204.206:4053): protocol: rpc: dial tcp 185.167.204.206:4053: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295852051 (56h32m27.757944534s) [noise=true , address=185.167.204.206:4065 , piece_id=VKBLX4ASIKVHMUUCTD7GFFPEHJLKREGCDGI7LCVRSASDHBAWI6AQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12oPo26KZGZr7o7M18KZiX5PMAAMwHK8FPT59y1hSLhkB6z3Lcj, address:185.167.204.206:4065): protocol: rpc: dial tcp 185.167.204.206:4065: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295908840 (56h32m27.758001313s) [noise=true , address=185.167.204.206:4057 , piece_id=WNX7U6DN7HTE3XLVJAJ4FTCYFR3B2MLMLF5I46GALE463DWQ22WQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1xy38NywWy5CSJ7CfupY78vbkHg6cQzgNhM2Z1xnpvw9dwvMoT, address:185.167.204.206:4057): protocol: rpc: dial tcp 185.167.204.206:4057: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296027916 (56h32m27.758120399s) [noise=true , address=185.167.204.205:4003 , piece_id=NQX2I3KTMYFLS2B2C5Y2PGZRXCD7QJCZLABY4PTYVEFGAVCYX2EQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1CjWvvJ4FdZc41sRgsQFWz4Y1ATZKgzsvDcUTmBSrCC15ekCmF, address:185.167.204.205:4003): protocol: rpc: dial tcp 185.167.204.205:4003: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296100945 (56h32m27.758193418s) [noise=true , address=185.167.204.205:4032 , piece_id=AN4YMHK6NKM5RIWAZHX2FPBGQ3VRFRYBKHBPIAL2HQWBK5XXNNOA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12558dx7zMNDfabV55vSsP9XwdxXgoPM8aHW53femVZzkCRxsKN, address:185.167.204.205:4032): protocol: rpc: dial tcp 185.167.204.205:4032: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296150660 (56h32m27.758243133s) [noise=true , address=185.167.204.205:4029 , piece_id=UVEMQY4WGBKGGT75ILTBPUCFZ4U252QGPDTBPT2BNMOR3V5XYCPA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12DyF12w2JXdkGLfLx9VE1E5f28VnMg7M2b3osfsDtGX9o8Tvsn, address:185.167.204.205:4029): protocol: write tcp 188.166.225.170:36604->185.167.204.205:4029: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296276239 (56h32m27.758368722s) [noise=true , address=185.167.204.206:4069 , piece_id=VKW6XTWPZPD3MDANS3FIYYD2UPDW47WBCGLNTOVYVTWA5IT23DBQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12Uan1TT5oAM3axhaviVFbvh83gThiZvwHMwbCD31ZZens6BQ4U, address:185.167.204.206:4069): protocol: rpc: dial tcp 185.167.204.206:4069: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296438448 (56h32m27.758530931s) [noise=true , address=185.167.204.206:4048 , piece_id=PRBNW47DU6K26GYUUUZSYTYQO3GNFC6DEV2OYQDP4R2RV5L5VNXA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1sY4wHBkSVUDWWg8T8Vm1hpRbbpb3ttqnqDVY8uWyG27gqBwT9, address:185.167.204.206:4048): protocol: write tcp 188.166.225.170:59140->185.167.204.206:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296628780 (56h32m27.758721253s) [noise=true , address=185.167.204.206:4055 , piece_id=KH2E6T37MHNJJ56CR3EIKMEUEWMAVP2XNMZ7K5QVKCFZJIVXRTUA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12CYRieezm1mjtGchWdPrB2mNaHVRvTS85umjB6cQyYKnvisXSB, address:185.167.204.206:4055): protocol: rpc: dial tcp 185.167.204.206:4055: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.297155593 (56h32m27.759248076s) [noise=true , address=185.167.204.205:4022 , piece_id=VU4625JC4A4ZVXFGJZTDMDZXYCNXJ67OQTEKWRLMMHGHPEHY5J7A, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj, address:185.167.204.205:4022): protocol: write tcp 188.166.225.170:57938->185.167.204.205:4022: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.297184969 (56h32m27.759277442s) [seg_pos={0 0}, upload=49746]: Done waiting for segment. successful: 12 optimal: 26 errs: <nil>
Sep 25 22:03:42.465979471 (56h32m27.928071944s) [upload=49746]: Done uploading segments...
Sep 25 22:03:42.466010200 (56h32m27.928102673s) [upload=49746]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 12)
Duration: 4.868064636s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
2 out of 60 is failed (threshold=3) [...........................XX...............................] duration:
4.871277622s
FAIL (2025-09-25 22:23:22 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 12)
======LOGS======
9 , piece_id=4YT7BXXDQVTR2QL4YBWVMO3YJCZQ57PTTEQW7GCZELWC42HQUDQQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.227500377 (56h31m29.68959285s) [noise=true , address=81.171.14.70:4034 , piece_id=EXVA3SFFPD3GJDYIFHAOP7GMJQHGZECMX5R6MHLRDMMCU7LOA7KA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.253492519 (56h31m29.715584993s) [noise=true , address=81.171.14.70:4028 , piece_id=Y237IBPHRRI7NMOYJG65AZI3LCCZ6JUJPX5WWGIDX5RYHCNVIG3A, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.267890396 (56h31m29.729982869s) [noise=true , address=89.149.201.147:4010 , piece_id=ANMXRI22IXAZFHKB7ZHFGEERKT2KGUYQEFKEOJNJDC5TFB6B3STQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.280031865 (56h31m29.742124348s) [noise=true , address=89.149.201.147:4020 , piece_id=RML2JG4QZSS6H5GTP7BW3B4SFVZLLHHAOUS3LDCOT2BGYTC3HACA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.294783955 (56h31m29.756876428s) [noise=true , address=89.149.201.147:4017 , piece_id=P576E6UE5UADQZYKR4XN4I5X4MCGMXT3B7PBGVNWBZ6I67ZVCJHA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.462969447 (56h31m29.92506192s) [noise=true , address=81.171.14.70:4035 , piece_id=GWVM3MZRGRVJ5RWG227JJXTULF5MUJ4FHWCNHLW7J3GDYXRSXRXA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.668506081 (56h31m30.130598554s) [noise=true , address=81.171.14.70:4045 , piece_id=BSKEEYML4LQZ7PWB45HQNX2YFJTFIEKIILDUZIWCXMVZZGGKHGUQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.702001372 (56h31m30.164093845s) [noise=true , address=89.149.201.147:4014 , piece_id=3VH5QXDWYX2NR3LODHSATJCP5ABF7BI52GJAR3VLPW42CPJJXQNA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.728242188 (56h31m30.190334661s) [noise=true , address=89.149.201.147:4016 , piece_id=PT5TUJDTS2TDBNLH5722GJTZ5BIO4LZKEDKBGUKHEF37TPADW7MA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:02:44.797046563 (56h31m30.259139036s) [noise=true , address=89.149.201.147:4012 , piece_id=TY6OVHC4ITI2E32GE7LP3O2YQ46FKWF6A6IIBN4JFAB7WPYXXJ2A, seg_pos={0 0}, upload=49746]: Done uploading piece. err: <nil>
Sep 25 22:03:42.295273049 (56h32m27.757365543s) [noise=true , address=185.167.204.206:4056 , piece_id=4XAHI4JY2MPISEOZ6VJ5GCJ5WZJKHRXIDK2EQGEN7VSYNRYJH24A, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1uSTuNA28oH2s5Rkv2jaKoC2nZWSgpMJXyxwFTCKbr44EvEXrC, address:185.167.204.206:4056): protocol: rpc: dial tcp 185.167.204.206:4056: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295296423 (56h32m27.757388906s) [noise=true , address=185.167.204.205:4008 , piece_id=NXU5IGU2NUZPVR5KIUKLZ7QTPYLGDIKZQNMPBJRPCVYEVNIV7HMA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12ZHTTaGQUTftxEULFqWbH6w1HnWA7PavHrXnvMCD947FxtmsrS, address:185.167.204.205:4008): protocol: rpc: dial tcp 185.167.204.205:4008: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295427543 (56h32m27.757520026s) [noise=true , address=185.167.204.206:4064 , piece_id=7GVKAUOISAORFKOXUNOJMEEFNNUVVZUQ2RGCD6BNVCDTH6ONPQIQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1ZB3emcQntFxpr7zBiE98eesnjoTnNKinw3FPtGU2TgznGakeb, address:185.167.204.206:4064): protocol: rpc: dial tcp 185.167.204.206:4064: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295495543 (56h32m27.757588026s) [noise=true , address=185.167.204.206:4059 , piece_id=G4B2XQYXOG2LWBAIYBM4BWLI7PEPE2XDM5A57YHYR6SOWW7ODSVA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:121gTedy8TPx44wmLqdQpZ1gDDsL8D9K1EvQJffR7oKY3bT9DXh, address:185.167.204.206:4059): protocol: rpc: dial tcp 185.167.204.206:4059: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295562701 (56h32m27.757655184s) [noise=true , address=185.167.204.205:4030 , piece_id=U6SOG6BNEIWDGYWDRZOCLTROZVZXPPG5ZI52HLRPKTDKFVQN3EJA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12w5R2Whxgg4AYNNvTkKmFq4pzahF1uZiVzwgVygc9iZDAXMdSk, address:185.167.204.205:4030): protocol: rpc: dial tcp 185.167.204.205:4030: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295700453 (56h32m27.757792926s) [noise=true , address=185.167.204.205:4010 , piece_id=DPMSKQDDOU2PUXHINTG7JJXLKXJE2PC6B3KNSKN5UIGAKBM3GZVQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12oTu1GaRsH7fHqxLkQ9c9ikjG4zRN3NChssa5gfpLwrcEehoyV, address:185.167.204.205:4010): protocol: rpc: dial tcp 185.167.204.205:4010: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295713257 (56h32m27.75780574s) [noise=true , address=185.167.204.206:4053 , piece_id=74G7LWVYE7GO4GPSG67BD6QVE7UBLGIAKLSG447PWWLNBM2FZNTA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12SrFttJpndpv7TKJJYsKthWnGWBYQGHpS9CwqTd4mqBP9NCCm8, address:185.167.204.206:4053): protocol: rpc: dial tcp 185.167.204.206:4053: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295852051 (56h32m27.757944534s) [noise=true , address=185.167.204.206:4065 , piece_id=VKBLX4ASIKVHMUUCTD7GFFPEHJLKREGCDGI7LCVRSASDHBAWI6AQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12oPo26KZGZr7o7M18KZiX5PMAAMwHK8FPT59y1hSLhkB6z3Lcj, address:185.167.204.206:4065): protocol: rpc: dial tcp 185.167.204.206:4065: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.295908840 (56h32m27.758001313s) [noise=true , address=185.167.204.206:4057 , piece_id=WNX7U6DN7HTE3XLVJAJ4FTCYFR3B2MLMLF5I46GALE463DWQ22WQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1xy38NywWy5CSJ7CfupY78vbkHg6cQzgNhM2Z1xnpvw9dwvMoT, address:185.167.204.206:4057): protocol: rpc: dial tcp 185.167.204.206:4057: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296027916 (56h32m27.758120399s) [noise=true , address=185.167.204.205:4003 , piece_id=NQX2I3KTMYFLS2B2C5Y2PGZRXCD7QJCZLABY4PTYVEFGAVCYX2EQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1CjWvvJ4FdZc41sRgsQFWz4Y1ATZKgzsvDcUTmBSrCC15ekCmF, address:185.167.204.205:4003): protocol: rpc: dial tcp 185.167.204.205:4003: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296100945 (56h32m27.758193418s) [noise=true , address=185.167.204.205:4032 , piece_id=AN4YMHK6NKM5RIWAZHX2FPBGQ3VRFRYBKHBPIAL2HQWBK5XXNNOA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12558dx7zMNDfabV55vSsP9XwdxXgoPM8aHW53femVZzkCRxsKN, address:185.167.204.205:4032): protocol: rpc: dial tcp 185.167.204.205:4032: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296150660 (56h32m27.758243133s) [noise=true , address=185.167.204.205:4029 , piece_id=UVEMQY4WGBKGGT75ILTBPUCFZ4U252QGPDTBPT2BNMOR3V5XYCPA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12DyF12w2JXdkGLfLx9VE1E5f28VnMg7M2b3osfsDtGX9o8Tvsn, address:185.167.204.205:4029): protocol: write tcp 188.166.225.170:36604->185.167.204.205:4029: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296276239 (56h32m27.758368722s) [noise=true , address=185.167.204.206:4069 , piece_id=VKW6XTWPZPD3MDANS3FIYYD2UPDW47WBCGLNTOVYVTWA5IT23DBQ, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12Uan1TT5oAM3axhaviVFbvh83gThiZvwHMwbCD31ZZens6BQ4U, address:185.167.204.206:4069): protocol: rpc: dial tcp 185.167.204.206:4069: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296438448 (56h32m27.758530931s) [noise=true , address=185.167.204.206:4048 , piece_id=PRBNW47DU6K26GYUUUZSYTYQO3GNFC6DEV2OYQDP4R2RV5L5VNXA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1sY4wHBkSVUDWWg8T8Vm1hpRbbpb3ttqnqDVY8uWyG27gqBwT9, address:185.167.204.206:4048): protocol: write tcp 188.166.225.170:59140->185.167.204.206:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.296628780 (56h32m27.758721253s) [noise=true , address=185.167.204.206:4055 , piece_id=KH2E6T37MHNJJ56CR3EIKMEUEWMAVP2XNMZ7K5QVKCFZJIVXRTUA, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:12CYRieezm1mjtGchWdPrB2mNaHVRvTS85umjB6cQyYKnvisXSB, address:185.167.204.206:4055): protocol: rpc: dial tcp 185.167.204.206:4055: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.297155593 (56h32m27.759248076s) [noise=true , address=185.167.204.205:4022 , piece_id=VU4625JC4A4ZVXFGJZTDMDZXYCNXJ67OQTEKWRLMMHGHPEHY5J7A, seg_pos={0 0}, upload=49746]: Done uploading piece. err: ecclient: upload failed (node:1fT9qH1igPUNfrzpwa2VkBX7heh475sNqoN4Nsf9FYSwi1VmHj, address:185.167.204.205:4022): protocol: write tcp 188.166.225.170:57938->185.167.204.205:4022: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 25 22:03:42.297184969 (56h32m27.759277442s) [seg_pos={0 0}, upload=49746]: Done waiting for segment. successful: 12 optimal: 26 errs: <nil>
Sep 25 22:03:42.465979471 (56h32m27.928071944s) [upload=49746]: Done uploading segments...
Sep 25 22:03:42.466010200 (56h32m27.928102673s) [upload=49746]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 12)
Duration: 4.871311707s
core
uplink
eu-select-1
eu1-placement-15
|