Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...............................X............................] duration:
6.191532067s trace:
https://jaeger.dev.storj.io/trace/3e4560ab56ee4602
FAIL (2025-11-16 16:30:15 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.191562055s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [................................X...........................] duration:
5.950120233s trace:
https://jaeger.dev.storj.io/trace/3e4560ab56ee4602
FAIL (2025-11-16 16:29:47 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.950143428s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [....................X.......................................] duration:
5.761973883s trace:
https://jaeger.dev.storj.io/trace/7dbd01be7ae5d01e
FAIL (2025-11-16 16:29:44 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 24)
======LOGS======
OHA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.079132696 (176h0m30.630603157s) [noise=true , address=81.171.14.70:4040 , piece_id=YG6ZBK6QTKUS3FCFD2EWU7HTFLOEEVG57SIER44E67QDP5HYCHOA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.079319683 (176h0m30.630790144s) [noise=true , address=185.167.204.205:4015 , piece_id=6UIFX7GXE2SZ465VAP56YQILR3A3OHDRKIS345EE6GBDG77KJNZQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.079646465 (176h0m30.631116926s) [noise=true , address=185.167.204.206:4062 , piece_id=PAJX3ZCWM5YBZA7YDBFYXCET5CKWBOFJ47IUWK4CYPZSWK6HYQVQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.080002833 (176h0m30.631473294s) [noise=true , address=81.171.14.70:4031 , piece_id=54T5MOZZ4YJKQFUDOCCFF2SAE7XS2N77H2AOHCBECAWE5OBUXTZQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.080062066 (176h0m30.631532527s) [noise=true , address=185.167.204.206:4067 , piece_id=2BDMQZLCV4UWWP5HB6GF6NCL5UOGNU7VNYGJIIJCKQHRTX5OPGPQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.080438673 (176h0m30.631909134s) [noise=true , address=185.167.204.205:4034 , piece_id=CZFPF35OQ3U4IH46KILFPFMVY6OC36ZUGZ7EYYBOTB3I4TXMPERA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.080675103 (176h0m30.632145564s) [noise=true , address=185.167.204.205:4014 , piece_id=57C2VUYOEULOTXUQHG7A2XOWZLXYFIEYYRYW5ARX5PKTGWKIEOAQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.081315833 (176h0m30.632786294s) [noise=true , address=81.171.14.70:4039 , piece_id=NMZ5357FVIKD2F3GCKY6YNNV73BAID2YBLVNSSM6VD5PHGLBHEIA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.082053738 (176h0m30.633524189s) [noise=true , address=185.167.204.205:4001 , piece_id=TBPYFHMXWO53LJQQDUA3I2MD2WP6BHQXAH3KIF6XXMWK27HXLCHA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.082605429 (176h0m30.63407589s) [noise=true , address=81.171.14.70:4034 , piece_id=NXBDCCCEWSZ2TMSET6OXBNQV6Q5ESXIQJUFK4WJJ3UV7FJTTAUXQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:33.280203414 (176h0m31.831673875s) [upload=155002]: Next returned nil segment
Nov 16 16:04:33.280222810 (176h0m31.831693271s) [upload=155002]: Waiting for error group managing segments...
Nov 16 16:04:33.838289781 (176h0m32.389760242s) [noise=true , address=185.167.204.205:4013 , piece_id=KF32EXUYGKQEDBPRSVAMNZV3ZMBF6DLT3U32BKX5LXLRBXBVTOQA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.848846382 (176h0m32.400316833s) [noise=true , address=185.167.204.206:4039 , piece_id=6HTUHESIRTSKR5EDMWZ2K56OKE5Y22QQM5ARYTUXR7KMKWIOS7GQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.862791868 (176h0m32.414262329s) [noise=true , address=185.167.204.205:4021 , piece_id=MV3TV4EI3HN3J5NY5XYVKT46QJTJEC5PITXERSSZMS2B7W3DSDTA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.879947135 (176h0m32.431417596s) [noise=true , address=185.167.204.205:4001 , piece_id=TBPYFHMXWO53LJQQDUA3I2MD2WP6BHQXAH3KIF6XXMWK27HXLCHA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.886853657 (176h0m32.438324118s) [noise=true , address=185.167.204.206:4067 , piece_id=2BDMQZLCV4UWWP5HB6GF6NCL5UOGNU7VNYGJIIJCKQHRTX5OPGPQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.918801783 (176h0m32.470272324s) [noise=true , address=185.167.204.206:4051 , piece_id=AU2S6M3E2CQEMAS7LKZJIVWITZUHIHCMWQ2QWT7ILGBRTAUCUB7Q, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.929982083 (176h0m32.481452574s) [noise=true , address=185.167.204.205:4015 , piece_id=6UIFX7GXE2SZ465VAP56YQILR3A3OHDRKIS345EE6GBDG77KJNZQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.935247409 (176h0m32.48671786s) [noise=true , address=185.167.204.206:4038 , piece_id=FBNPAJOOXEAT3KLKQIYOWHGHRGAWJABV4LAA3UYX6SGJB3U2LD7A, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.969630813 (176h0m32.521101264s) [noise=true , address=185.167.204.206:4055 , piece_id=B2TTK2ZLIRM3YIKP5HQ3WOL26KBHSXWWX5ZXXCMWU34AH7CHOOCQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.999043600 (176h0m32.550514061s) [noise=true , address=185.167.204.205:4034 , piece_id=CZFPF35OQ3U4IH46KILFPFMVY6OC36ZUGZ7EYYBOTB3I4TXMPERA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.022951616 (176h0m32.574422077s) [noise=true , address=185.167.204.205:4014 , piece_id=57C2VUYOEULOTXUQHG7A2XOWZLXYFIEYYRYW5ARX5PKTGWKIEOAQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.064488381 (176h0m32.615958832s) [noise=true , address=185.167.204.206:4058 , piece_id=POK7HBLE5AT2STCSL2UJ5R67LNVK3PN53NIRIBX4M7LJYC4C2CHQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.122257987 (176h0m32.673728468s) [noise=true , address=185.167.204.205:4029 , piece_id=VF7BTI4K75HTRVJOHIKMDJN3UBHZ7KCQBT3H44XB5G7ZVFD4HXJQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.136282884 (176h0m32.687753345s) [noise=true , address=185.167.204.206:4041 , piece_id=RTQSCXR54KTKBYS6AHEKPO7ZPUHVQQNUNQJ6JI4JLTGKOLUZH3AQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.136821679 (176h0m32.68829215s) [noise=true , address=185.167.204.206:4062 , piece_id=PAJX3ZCWM5YBZA7YDBFYXCET5CKWBOFJ47IUWK4CYPZSWK6HYQVQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.156096042 (176h0m32.707566493s) [noise=true , address=185.167.204.206:4066 , piece_id=CPTBHDHCT6R2M3UA6X2ZWMPJUKAUVV4I2CIATZ2ZOIWH63Q3UT4Q, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.540981809 (176h0m33.0924523s) [noise=true , address=89.149.201.147:4012 , piece_id=NEIYQHVVWRUVJFF5UD3RZSRHUSNSKXBCNGTMIGOH5MDH2AOZISCA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:21.258617296 (176h1m19.810087757s) [noise=true , address=81.171.14.70:4034 , piece_id=NXBDCCCEWSZ2TMSET6OXBNQV6Q5ESXIQJUFK4WJJ3UV7FJTTAUXQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:22.223432573 (176h1m20.774903034s) [noise=true , address=81.171.14.70:4048 , piece_id=5WUEBVU7UFKHYZ23BPIQUMUUBH3ZXXQWBLE7MALJGECCDTTEK5EA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:22.238494073 (176h1m20.789964534s) [noise=true , address=81.171.14.70:4033 , piece_id=LUBQIVFAU7J72K5XDDS5S2ZXQUGFA5AINL4R5FV64KY24VS5KV5Q, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:22.350384206 (176h1m20.901854667s) [noise=true , address=81.171.14.70:4039 , piece_id=NMZ5357FVIKD2F3GCKY6YNNV73BAID2YBLVNSSM6VD5PHGLBHEIA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:23.704484814 (176h1m22.255955275s) [noise=true , address=81.171.14.70:4037 , piece_id=3IJ7J7RBMOORW4YPKIYJ3C6ZQRYBH7WQMLEKEQYG5UJ4TGKE2W2A, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:24.501497158 (176h1m23.052967619s) [noise=true , address=81.171.14.70:4031 , piece_id=54T5MOZZ4YJKQFUDOCCFF2SAE7XS2N77H2AOHCBECAWE5OBUXTZQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:29.810461698 (176h1m28.361932159s) [noise=true , address=81.171.14.70:4025 , piece_id=3HAT6FJSGV6VOHJBMO6L6L7677BOYVCDV3SR26LJZ6HZOZWJJ64A, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:31.413018058 (176h1m29.964488509s) [noise=true , address=81.171.14.70:4040 , piece_id=YG6ZBK6QTKUS3FCFD2EWU7HTFLOEEVG57SIER44E67QDP5HYCHOA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:1gannW154cmVP6N5BowytKfDorZDRwx2M5pi6akqjZFU38PGjV, address:81.171.14.70:4040): protocol: write tcp 188.166.225.170:59244->81.171.14.70:4040: use of closed network connection; write tcp 188.166.225.170:59244->81.171.14.70:4040: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59244->81.171.14.70:4040: use of closed network connection
Nov 16 16:05:31.413111366 (176h1m29.964581837s) [noise=true , address=81.171.14.70:4035 , piece_id=RGMXRH2Y5XBAFISK5LNAF547SAORD6XSWFUKYWOPJINXXODI5KVQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:12eLYJKne3yP1fZ1oPDLM7Xqs3GQ3ju4ZQedAiojXdVkcxNpzeA, address:81.171.14.70:4035): protocol: read tcp 188.166.225.170:55142->81.171.14.70:4035: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:05:31.413619965 (176h1m29.965090436s) [noise=true , address=81.171.14.70:4030 , piece_id=ZE5OAUTWLWNSMWSFGPEJTCXEN6L3FLWL34QBERTS3JZE7X2I2OHA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:12ZoRwa8wLw66Pjumy1ybp6wrZ79RrSU55uYZpwaY3xi95iSYde, address:81.171.14.70:4030): protocol: read tcp 188.166.225.170:40822->81.171.14.70:4030: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:05:31.414846280 (176h1m29.966316741s) [noise=true , address=81.171.14.70:4029 , piece_id=SIXL2XSCF7IO2BPOUKGH5VVXI6M7O4KJVZARYIRELR2GM4T4RABA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:18yEfU2X4jSrom1JDtQ22cUEvomcg91ohvay2Nsr1Vr8juo9to, address:81.171.14.70:4029): protocol: read tcp 188.166.225.170:56584->81.171.14.70:4029: use of closed network connection; conn closed: op
Nov 16 16:05:31.414946261 (176h1m29.966416722s) [seg_pos={0 0}, upload=155002]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil>
Nov 16 16:05:31.591745995 (176h1m30.143216456s) [upload=155002]: Done uploading segments...
Nov 16 16:05:31.591788596 (176h1m30.143259057s) [upload=155002]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24)
Duration: 5.761991377s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.................................X..........................] duration:
6.024680703s trace:
https://jaeger.dev.storj.io/trace/3e4560ab56ee4602
FAIL (2025-11-16 16:29:31 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.024707775s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..................................X.........................] duration:
5.898967674s trace:
https://jaeger.dev.storj.io/trace/70f81d2d9b012a3a
FAIL (2025-11-16 16:29:24 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 4)
======LOGS======
ddress=45.91.116.5:4297 , piece_id=OSMI3VGAKSCIZMDQ3Y2VZQBUOJKSXM6RWNWSAV57RDTL5TR3NIJA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.462247309 (176h7m35.013717781s) [noise=true , address=45.91.116.7:4011 , piece_id=ORJVRMFIOEXZ6OB6QFQDL764Q2FCWOJT63NBIRI6SOOS2FEIBCFQ, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.463420774 (176h7m35.014891235s) [noise=true , address=145.79.227.9:4367 , piece_id=LOFHPWCHNEZ4E4WANJM5BE2TIWVFH4LGEJJYK4ETB2LZWYCAVGZA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.464113112 (176h7m35.015583573s) [noise=true , address=145.79.227.22:4511 , piece_id=HES465EP3OPD5JBVGVYMGLUOHJKSSMJDXMJVWET5GHRPFZJFIFOA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.464764873 (176h7m35.016235334s) [noise=true , address=45.91.116.8:4048 , piece_id=ULVK52VWCW3KNEPWSDYGRHWHIJWEQU2XNFEPYJFXWLVJI6AGZQOA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.465452753 (176h7m35.016923214s) [noise=true , address=145.79.227.28:4725 , piece_id=CI2LSBGPW7RYH4RGWODGWLUTN353EODNV6IB2YQGHOZ3FPSEEK5A, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.466223270 (176h7m35.017693731s) [noise=true , address=145.79.227.16:4910 , piece_id=3U2SEVMPILIEBRHS3M4YT4Z4PSYAQAIC2SKGYMW4FNODACZLOQCQ, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.466955855 (176h7m35.018426306s) [noise=true , address=145.79.227.4:4186 , piece_id=5PZIK3V72CUQQ6M2HJZFVM5OVX4HPHY7ZHJ2BAK5S3ZQOGM4J5OA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.467675205 (176h7m35.019145666s) [noise=true , address=145.79.227.8:4331 , piece_id=4O27HU7AMR6G4Y43HJYN22UXVLETBDBE7JHL4JUZKRM54SK5E4JA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.468178092 (176h7m35.019648553s) [noise=true , address=145.79.227.8:4329 , piece_id=GO5IAWQYS3UARWYAQAEUIIS456DFWCFWRJQG2KOVVL5KPXWPK4AA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:39.111804678 (176h7m37.663275129s) [upload=155107]: Next returned nil segment
Nov 16 16:11:39.111825767 (176h7m37.663296218s) [upload=155107]: Waiting for error group managing segments...
Nov 16 16:11:39.896671313 (176h7m38.448141765s) [noise=true , address=45.91.116.12:4227 , piece_id=CQA2LHRSFOA7EIBIFVC75EWVHFZMWMY7ODMETEAJRRNYSQFY5TVQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil>
Nov 16 16:11:39.954993420 (176h7m38.506463881s) [noise=true , address=45.91.116.8:4048 , piece_id=ULVK52VWCW3KNEPWSDYGRHWHIJWEQU2XNFEPYJFXWLVJI6AGZQOA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil>
Nov 16 16:11:40.066014029 (176h7m38.61748449s) [noise=true , address=45.91.116.7:4011 , piece_id=ORJVRMFIOEXZ6OB6QFQDL764Q2FCWOJT63NBIRI6SOOS2FEIBCFQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil>
Nov 16 16:11:41.102688250 (176h7m39.654158711s) [noise=true , address=45.91.116.5:4297 , piece_id=OSMI3VGAKSCIZMDQ3Y2VZQBUOJKSXM6RWNWSAV57RDTL5TR3NIJA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil>
Nov 16 16:12:04.264842280 (176h8m2.816312741s) [noise=true , address=145.79.227.17:4972 , piece_id=NV6FSBQM6YUTKYFAJPQCURWCZBHJ3AKJOLLGCRYTJ3KQ54U4GOWQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection
Nov 16 16:12:04.265084832 (176h8m2.816555293s) [noise=true , address=145.79.227.16:4910 , piece_id=3U2SEVMPILIEBRHS3M4YT4Z4PSYAQAIC2SKGYMW4FNODACZLOQCQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection
Nov 16 16:12:04.270267942 (176h8m2.821738423s) [noise=true , address=145.79.227.8:4329 , piece_id=GO5IAWQYS3UARWYAQAEUIIS456DFWCFWRJQG2KOVVL5KPXWPK4AA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: expected piece hash; context deadline exceeded; EOF
Nov 16 16:12:04.270306515 (176h8m2.821776986s) [noise=true , address=145.79.227.9:4367 , piece_id=LOFHPWCHNEZ4E4WANJM5BE2TIWVFH4LGEJJYK4ETB2LZWYCAVGZA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection
Nov 16 16:12:04.270323538 (176h8m2.821794099s) [noise=true , address=145.79.227.28:4725 , piece_id=CI2LSBGPW7RYH4RGWODGWLUTN353EODNV6IB2YQGHOZ3FPSEEK5A, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection
Nov 16 16:12:04.270544138 (176h8m2.822014739s) [noise=true , address=145.79.227.14:4840 , piece_id=RSHGL65RI7HZGU3W3EROFL23BCK2EW3YN6CAD3NO5OVE5QM3QG3Q, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: read tcp 188.166.225.170:42148->145.79.227.14:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:12:04.270571570 (176h8m2.822042121s) [noise=true , address=145.79.227.31:4797 , piece_id=AWSPIPRKSBOKQFZRRB34RUPLYVJGKAOOJQ4U7RRATQJELM3OCFHA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection
Nov 16 16:12:04.270609783 (176h8m2.822080254s) [noise=true , address=145.79.227.22:4511 , piece_id=HES465EP3OPD5JBVGVYMGLUOHJKSSMJDXMJVWET5GHRPFZJFIFOA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: read tcp 188.166.225.170:51004->145.79.227.22:4511: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:12:04.270625552 (176h8m2.822096083s) [noise=true , address=145.79.227.8:4331 , piece_id=4O27HU7AMR6G4Y43HJYN22UXVLETBDBE7JHL4JUZKRM54SK5E4JA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: read tcp 188.166.225.170:44510->145.79.227.8:4331: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:12:04.270742155 (176h8m2.822212616s) [noise=true , address=145.79.227.4:4186 , piece_id=5PZIK3V72CUQQ6M2HJZFVM5OVX4HPHY7ZHJ2BAK5S3ZQOGM4J5OA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection
Nov 16 16:12:04.270762814 (176h8m2.822233275s) [noise=true , address=145.79.227.24:4581 , piece_id=XRCS52EJDB7HWD4RDZK3UNB6VKHOTWWID42RGBJZQHTGG4NDYIPA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: read tcp 188.166.225.170:44130->145.79.227.24:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:12:04.270781470 (176h8m2.822251921s) [noise=true , address=145.79.227.11:4732 , piece_id=FQGAVGZ5CID3LBET3CVK6FSCOP5OUYGS44XYWMHU2RLIT3MBN62A, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: read tcp 188.166.225.170:42420->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:12:04.270825082 (176h8m2.822295544s) [seg_pos={0 0}, upload=155107]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil>
Nov 16 16:12:05.262661603 (176h8m3.814132064s) [upload=155107]: Done uploading segments...
Nov 16 16:12:05.262728080 (176h8m3.814198541s) [upload=155107]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4)
Duration: 5.898985557s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [.....................X......................................] duration:
5.776436045s trace:
https://jaeger.dev.storj.io/trace/7dbd01be7ae5d01e
FAIL (2025-11-16 16:29:03 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 24)
======LOGS======
OHA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.079132696 (176h0m30.630603157s) [noise=true , address=81.171.14.70:4040 , piece_id=YG6ZBK6QTKUS3FCFD2EWU7HTFLOEEVG57SIER44E67QDP5HYCHOA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.079319683 (176h0m30.630790144s) [noise=true , address=185.167.204.205:4015 , piece_id=6UIFX7GXE2SZ465VAP56YQILR3A3OHDRKIS345EE6GBDG77KJNZQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.079646465 (176h0m30.631116926s) [noise=true , address=185.167.204.206:4062 , piece_id=PAJX3ZCWM5YBZA7YDBFYXCET5CKWBOFJ47IUWK4CYPZSWK6HYQVQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.080002833 (176h0m30.631473294s) [noise=true , address=81.171.14.70:4031 , piece_id=54T5MOZZ4YJKQFUDOCCFF2SAE7XS2N77H2AOHCBECAWE5OBUXTZQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.080062066 (176h0m30.631532527s) [noise=true , address=185.167.204.206:4067 , piece_id=2BDMQZLCV4UWWP5HB6GF6NCL5UOGNU7VNYGJIIJCKQHRTX5OPGPQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.080438673 (176h0m30.631909134s) [noise=true , address=185.167.204.205:4034 , piece_id=CZFPF35OQ3U4IH46KILFPFMVY6OC36ZUGZ7EYYBOTB3I4TXMPERA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.080675103 (176h0m30.632145564s) [noise=true , address=185.167.204.205:4014 , piece_id=57C2VUYOEULOTXUQHG7A2XOWZLXYFIEYYRYW5ARX5PKTGWKIEOAQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.081315833 (176h0m30.632786294s) [noise=true , address=81.171.14.70:4039 , piece_id=NMZ5357FVIKD2F3GCKY6YNNV73BAID2YBLVNSSM6VD5PHGLBHEIA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.082053738 (176h0m30.633524189s) [noise=true , address=185.167.204.205:4001 , piece_id=TBPYFHMXWO53LJQQDUA3I2MD2WP6BHQXAH3KIF6XXMWK27HXLCHA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.082605429 (176h0m30.63407589s) [noise=true , address=81.171.14.70:4034 , piece_id=NXBDCCCEWSZ2TMSET6OXBNQV6Q5ESXIQJUFK4WJJ3UV7FJTTAUXQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:33.280203414 (176h0m31.831673875s) [upload=155002]: Next returned nil segment
Nov 16 16:04:33.280222810 (176h0m31.831693271s) [upload=155002]: Waiting for error group managing segments...
Nov 16 16:04:33.838289781 (176h0m32.389760242s) [noise=true , address=185.167.204.205:4013 , piece_id=KF32EXUYGKQEDBPRSVAMNZV3ZMBF6DLT3U32BKX5LXLRBXBVTOQA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.848846382 (176h0m32.400316833s) [noise=true , address=185.167.204.206:4039 , piece_id=6HTUHESIRTSKR5EDMWZ2K56OKE5Y22QQM5ARYTUXR7KMKWIOS7GQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.862791868 (176h0m32.414262329s) [noise=true , address=185.167.204.205:4021 , piece_id=MV3TV4EI3HN3J5NY5XYVKT46QJTJEC5PITXERSSZMS2B7W3DSDTA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.879947135 (176h0m32.431417596s) [noise=true , address=185.167.204.205:4001 , piece_id=TBPYFHMXWO53LJQQDUA3I2MD2WP6BHQXAH3KIF6XXMWK27HXLCHA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.886853657 (176h0m32.438324118s) [noise=true , address=185.167.204.206:4067 , piece_id=2BDMQZLCV4UWWP5HB6GF6NCL5UOGNU7VNYGJIIJCKQHRTX5OPGPQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.918801783 (176h0m32.470272324s) [noise=true , address=185.167.204.206:4051 , piece_id=AU2S6M3E2CQEMAS7LKZJIVWITZUHIHCMWQ2QWT7ILGBRTAUCUB7Q, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.929982083 (176h0m32.481452574s) [noise=true , address=185.167.204.205:4015 , piece_id=6UIFX7GXE2SZ465VAP56YQILR3A3OHDRKIS345EE6GBDG77KJNZQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.935247409 (176h0m32.48671786s) [noise=true , address=185.167.204.206:4038 , piece_id=FBNPAJOOXEAT3KLKQIYOWHGHRGAWJABV4LAA3UYX6SGJB3U2LD7A, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.969630813 (176h0m32.521101264s) [noise=true , address=185.167.204.206:4055 , piece_id=B2TTK2ZLIRM3YIKP5HQ3WOL26KBHSXWWX5ZXXCMWU34AH7CHOOCQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.999043600 (176h0m32.550514061s) [noise=true , address=185.167.204.205:4034 , piece_id=CZFPF35OQ3U4IH46KILFPFMVY6OC36ZUGZ7EYYBOTB3I4TXMPERA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.022951616 (176h0m32.574422077s) [noise=true , address=185.167.204.205:4014 , piece_id=57C2VUYOEULOTXUQHG7A2XOWZLXYFIEYYRYW5ARX5PKTGWKIEOAQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.064488381 (176h0m32.615958832s) [noise=true , address=185.167.204.206:4058 , piece_id=POK7HBLE5AT2STCSL2UJ5R67LNVK3PN53NIRIBX4M7LJYC4C2CHQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.122257987 (176h0m32.673728468s) [noise=true , address=185.167.204.205:4029 , piece_id=VF7BTI4K75HTRVJOHIKMDJN3UBHZ7KCQBT3H44XB5G7ZVFD4HXJQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.136282884 (176h0m32.687753345s) [noise=true , address=185.167.204.206:4041 , piece_id=RTQSCXR54KTKBYS6AHEKPO7ZPUHVQQNUNQJ6JI4JLTGKOLUZH3AQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.136821679 (176h0m32.68829215s) [noise=true , address=185.167.204.206:4062 , piece_id=PAJX3ZCWM5YBZA7YDBFYXCET5CKWBOFJ47IUWK4CYPZSWK6HYQVQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.156096042 (176h0m32.707566493s) [noise=true , address=185.167.204.206:4066 , piece_id=CPTBHDHCT6R2M3UA6X2ZWMPJUKAUVV4I2CIATZ2ZOIWH63Q3UT4Q, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.540981809 (176h0m33.0924523s) [noise=true , address=89.149.201.147:4012 , piece_id=NEIYQHVVWRUVJFF5UD3RZSRHUSNSKXBCNGTMIGOH5MDH2AOZISCA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:21.258617296 (176h1m19.810087757s) [noise=true , address=81.171.14.70:4034 , piece_id=NXBDCCCEWSZ2TMSET6OXBNQV6Q5ESXIQJUFK4WJJ3UV7FJTTAUXQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:22.223432573 (176h1m20.774903034s) [noise=true , address=81.171.14.70:4048 , piece_id=5WUEBVU7UFKHYZ23BPIQUMUUBH3ZXXQWBLE7MALJGECCDTTEK5EA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:22.238494073 (176h1m20.789964534s) [noise=true , address=81.171.14.70:4033 , piece_id=LUBQIVFAU7J72K5XDDS5S2ZXQUGFA5AINL4R5FV64KY24VS5KV5Q, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:22.350384206 (176h1m20.901854667s) [noise=true , address=81.171.14.70:4039 , piece_id=NMZ5357FVIKD2F3GCKY6YNNV73BAID2YBLVNSSM6VD5PHGLBHEIA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:23.704484814 (176h1m22.255955275s) [noise=true , address=81.171.14.70:4037 , piece_id=3IJ7J7RBMOORW4YPKIYJ3C6ZQRYBH7WQMLEKEQYG5UJ4TGKE2W2A, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:24.501497158 (176h1m23.052967619s) [noise=true , address=81.171.14.70:4031 , piece_id=54T5MOZZ4YJKQFUDOCCFF2SAE7XS2N77H2AOHCBECAWE5OBUXTZQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:29.810461698 (176h1m28.361932159s) [noise=true , address=81.171.14.70:4025 , piece_id=3HAT6FJSGV6VOHJBMO6L6L7677BOYVCDV3SR26LJZ6HZOZWJJ64A, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:31.413018058 (176h1m29.964488509s) [noise=true , address=81.171.14.70:4040 , piece_id=YG6ZBK6QTKUS3FCFD2EWU7HTFLOEEVG57SIER44E67QDP5HYCHOA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:1gannW154cmVP6N5BowytKfDorZDRwx2M5pi6akqjZFU38PGjV, address:81.171.14.70:4040): protocol: write tcp 188.166.225.170:59244->81.171.14.70:4040: use of closed network connection; write tcp 188.166.225.170:59244->81.171.14.70:4040: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59244->81.171.14.70:4040: use of closed network connection
Nov 16 16:05:31.413111366 (176h1m29.964581837s) [noise=true , address=81.171.14.70:4035 , piece_id=RGMXRH2Y5XBAFISK5LNAF547SAORD6XSWFUKYWOPJINXXODI5KVQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:12eLYJKne3yP1fZ1oPDLM7Xqs3GQ3ju4ZQedAiojXdVkcxNpzeA, address:81.171.14.70:4035): protocol: read tcp 188.166.225.170:55142->81.171.14.70:4035: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:05:31.413619965 (176h1m29.965090436s) [noise=true , address=81.171.14.70:4030 , piece_id=ZE5OAUTWLWNSMWSFGPEJTCXEN6L3FLWL34QBERTS3JZE7X2I2OHA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:12ZoRwa8wLw66Pjumy1ybp6wrZ79RrSU55uYZpwaY3xi95iSYde, address:81.171.14.70:4030): protocol: read tcp 188.166.225.170:40822->81.171.14.70:4030: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:05:31.414846280 (176h1m29.966316741s) [noise=true , address=81.171.14.70:4029 , piece_id=SIXL2XSCF7IO2BPOUKGH5VVXI6M7O4KJVZARYIRELR2GM4T4RABA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:18yEfU2X4jSrom1JDtQ22cUEvomcg91ohvay2Nsr1Vr8juo9to, address:81.171.14.70:4029): protocol: read tcp 188.166.225.170:56584->81.171.14.70:4029: use of closed network connection; conn closed: op
Nov 16 16:05:31.414946261 (176h1m29.966416722s) [seg_pos={0 0}, upload=155002]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil>
Nov 16 16:05:31.591745995 (176h1m30.143216456s) [upload=155002]: Done uploading segments...
Nov 16 16:05:31.591788596 (176h1m30.143259057s) [upload=155002]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24)
Duration: 5.776466413s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...................................X........................] duration:
5.966253198s trace:
https://jaeger.dev.storj.io/trace/70f81d2d9b012a3a
FAIL (2025-11-16 16:29:00 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 4)
======LOGS======
ddress=45.91.116.5:4297 , piece_id=OSMI3VGAKSCIZMDQ3Y2VZQBUOJKSXM6RWNWSAV57RDTL5TR3NIJA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.462247309 (176h7m35.013717781s) [noise=true , address=45.91.116.7:4011 , piece_id=ORJVRMFIOEXZ6OB6QFQDL764Q2FCWOJT63NBIRI6SOOS2FEIBCFQ, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.463420774 (176h7m35.014891235s) [noise=true , address=145.79.227.9:4367 , piece_id=LOFHPWCHNEZ4E4WANJM5BE2TIWVFH4LGEJJYK4ETB2LZWYCAVGZA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.464113112 (176h7m35.015583573s) [noise=true , address=145.79.227.22:4511 , piece_id=HES465EP3OPD5JBVGVYMGLUOHJKSSMJDXMJVWET5GHRPFZJFIFOA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.464764873 (176h7m35.016235334s) [noise=true , address=45.91.116.8:4048 , piece_id=ULVK52VWCW3KNEPWSDYGRHWHIJWEQU2XNFEPYJFXWLVJI6AGZQOA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.465452753 (176h7m35.016923214s) [noise=true , address=145.79.227.28:4725 , piece_id=CI2LSBGPW7RYH4RGWODGWLUTN353EODNV6IB2YQGHOZ3FPSEEK5A, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.466223270 (176h7m35.017693731s) [noise=true , address=145.79.227.16:4910 , piece_id=3U2SEVMPILIEBRHS3M4YT4Z4PSYAQAIC2SKGYMW4FNODACZLOQCQ, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.466955855 (176h7m35.018426306s) [noise=true , address=145.79.227.4:4186 , piece_id=5PZIK3V72CUQQ6M2HJZFVM5OVX4HPHY7ZHJ2BAK5S3ZQOGM4J5OA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.467675205 (176h7m35.019145666s) [noise=true , address=145.79.227.8:4331 , piece_id=4O27HU7AMR6G4Y43HJYN22UXVLETBDBE7JHL4JUZKRM54SK5E4JA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.468178092 (176h7m35.019648553s) [noise=true , address=145.79.227.8:4329 , piece_id=GO5IAWQYS3UARWYAQAEUIIS456DFWCFWRJQG2KOVVL5KPXWPK4AA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:39.111804678 (176h7m37.663275129s) [upload=155107]: Next returned nil segment
Nov 16 16:11:39.111825767 (176h7m37.663296218s) [upload=155107]: Waiting for error group managing segments...
Nov 16 16:11:39.896671313 (176h7m38.448141765s) [noise=true , address=45.91.116.12:4227 , piece_id=CQA2LHRSFOA7EIBIFVC75EWVHFZMWMY7ODMETEAJRRNYSQFY5TVQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil>
Nov 16 16:11:39.954993420 (176h7m38.506463881s) [noise=true , address=45.91.116.8:4048 , piece_id=ULVK52VWCW3KNEPWSDYGRHWHIJWEQU2XNFEPYJFXWLVJI6AGZQOA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil>
Nov 16 16:11:40.066014029 (176h7m38.61748449s) [noise=true , address=45.91.116.7:4011 , piece_id=ORJVRMFIOEXZ6OB6QFQDL764Q2FCWOJT63NBIRI6SOOS2FEIBCFQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil>
Nov 16 16:11:41.102688250 (176h7m39.654158711s) [noise=true , address=45.91.116.5:4297 , piece_id=OSMI3VGAKSCIZMDQ3Y2VZQBUOJKSXM6RWNWSAV57RDTL5TR3NIJA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil>
Nov 16 16:12:04.264842280 (176h8m2.816312741s) [noise=true , address=145.79.227.17:4972 , piece_id=NV6FSBQM6YUTKYFAJPQCURWCZBHJ3AKJOLLGCRYTJ3KQ54U4GOWQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection
Nov 16 16:12:04.265084832 (176h8m2.816555293s) [noise=true , address=145.79.227.16:4910 , piece_id=3U2SEVMPILIEBRHS3M4YT4Z4PSYAQAIC2SKGYMW4FNODACZLOQCQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection
Nov 16 16:12:04.270267942 (176h8m2.821738423s) [noise=true , address=145.79.227.8:4329 , piece_id=GO5IAWQYS3UARWYAQAEUIIS456DFWCFWRJQG2KOVVL5KPXWPK4AA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: expected piece hash; context deadline exceeded; EOF
Nov 16 16:12:04.270306515 (176h8m2.821776986s) [noise=true , address=145.79.227.9:4367 , piece_id=LOFHPWCHNEZ4E4WANJM5BE2TIWVFH4LGEJJYK4ETB2LZWYCAVGZA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection
Nov 16 16:12:04.270323538 (176h8m2.821794099s) [noise=true , address=145.79.227.28:4725 , piece_id=CI2LSBGPW7RYH4RGWODGWLUTN353EODNV6IB2YQGHOZ3FPSEEK5A, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection
Nov 16 16:12:04.270544138 (176h8m2.822014739s) [noise=true , address=145.79.227.14:4840 , piece_id=RSHGL65RI7HZGU3W3EROFL23BCK2EW3YN6CAD3NO5OVE5QM3QG3Q, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: read tcp 188.166.225.170:42148->145.79.227.14:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:12:04.270571570 (176h8m2.822042121s) [noise=true , address=145.79.227.31:4797 , piece_id=AWSPIPRKSBOKQFZRRB34RUPLYVJGKAOOJQ4U7RRATQJELM3OCFHA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection
Nov 16 16:12:04.270609783 (176h8m2.822080254s) [noise=true , address=145.79.227.22:4511 , piece_id=HES465EP3OPD5JBVGVYMGLUOHJKSSMJDXMJVWET5GHRPFZJFIFOA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: read tcp 188.166.225.170:51004->145.79.227.22:4511: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:12:04.270625552 (176h8m2.822096083s) [noise=true , address=145.79.227.8:4331 , piece_id=4O27HU7AMR6G4Y43HJYN22UXVLETBDBE7JHL4JUZKRM54SK5E4JA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: read tcp 188.166.225.170:44510->145.79.227.8:4331: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:12:04.270742155 (176h8m2.822212616s) [noise=true , address=145.79.227.4:4186 , piece_id=5PZIK3V72CUQQ6M2HJZFVM5OVX4HPHY7ZHJ2BAK5S3ZQOGM4J5OA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection
Nov 16 16:12:04.270762814 (176h8m2.822233275s) [noise=true , address=145.79.227.24:4581 , piece_id=XRCS52EJDB7HWD4RDZK3UNB6VKHOTWWID42RGBJZQHTGG4NDYIPA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: read tcp 188.166.225.170:44130->145.79.227.24:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:12:04.270781470 (176h8m2.822251921s) [noise=true , address=145.79.227.11:4732 , piece_id=FQGAVGZ5CID3LBET3CVK6FSCOP5OUYGS44XYWMHU2RLIT3MBN62A, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: read tcp 188.166.225.170:42420->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:12:04.270825082 (176h8m2.822295544s) [seg_pos={0 0}, upload=155107]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil>
Nov 16 16:12:05.262661603 (176h8m3.814132064s) [upload=155107]: Done uploading segments...
Nov 16 16:12:05.262728080 (176h8m3.814198541s) [upload=155107]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4)
Duration: 5.966408716s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [......................X.....................................] duration:
5.823329758s trace:
https://jaeger.dev.storj.io/trace/7dbd01be7ae5d01e
FAIL (2025-11-16 16:28:45 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 24)
======LOGS======
OHA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.079132696 (176h0m30.630603157s) [noise=true , address=81.171.14.70:4040 , piece_id=YG6ZBK6QTKUS3FCFD2EWU7HTFLOEEVG57SIER44E67QDP5HYCHOA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.079319683 (176h0m30.630790144s) [noise=true , address=185.167.204.205:4015 , piece_id=6UIFX7GXE2SZ465VAP56YQILR3A3OHDRKIS345EE6GBDG77KJNZQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.079646465 (176h0m30.631116926s) [noise=true , address=185.167.204.206:4062 , piece_id=PAJX3ZCWM5YBZA7YDBFYXCET5CKWBOFJ47IUWK4CYPZSWK6HYQVQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.080002833 (176h0m30.631473294s) [noise=true , address=81.171.14.70:4031 , piece_id=54T5MOZZ4YJKQFUDOCCFF2SAE7XS2N77H2AOHCBECAWE5OBUXTZQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.080062066 (176h0m30.631532527s) [noise=true , address=185.167.204.206:4067 , piece_id=2BDMQZLCV4UWWP5HB6GF6NCL5UOGNU7VNYGJIIJCKQHRTX5OPGPQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.080438673 (176h0m30.631909134s) [noise=true , address=185.167.204.205:4034 , piece_id=CZFPF35OQ3U4IH46KILFPFMVY6OC36ZUGZ7EYYBOTB3I4TXMPERA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.080675103 (176h0m30.632145564s) [noise=true , address=185.167.204.205:4014 , piece_id=57C2VUYOEULOTXUQHG7A2XOWZLXYFIEYYRYW5ARX5PKTGWKIEOAQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.081315833 (176h0m30.632786294s) [noise=true , address=81.171.14.70:4039 , piece_id=NMZ5357FVIKD2F3GCKY6YNNV73BAID2YBLVNSSM6VD5PHGLBHEIA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.082053738 (176h0m30.633524189s) [noise=true , address=185.167.204.205:4001 , piece_id=TBPYFHMXWO53LJQQDUA3I2MD2WP6BHQXAH3KIF6XXMWK27HXLCHA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.082605429 (176h0m30.63407589s) [noise=true , address=81.171.14.70:4034 , piece_id=NXBDCCCEWSZ2TMSET6OXBNQV6Q5ESXIQJUFK4WJJ3UV7FJTTAUXQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:33.280203414 (176h0m31.831673875s) [upload=155002]: Next returned nil segment
Nov 16 16:04:33.280222810 (176h0m31.831693271s) [upload=155002]: Waiting for error group managing segments...
Nov 16 16:04:33.838289781 (176h0m32.389760242s) [noise=true , address=185.167.204.205:4013 , piece_id=KF32EXUYGKQEDBPRSVAMNZV3ZMBF6DLT3U32BKX5LXLRBXBVTOQA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.848846382 (176h0m32.400316833s) [noise=true , address=185.167.204.206:4039 , piece_id=6HTUHESIRTSKR5EDMWZ2K56OKE5Y22QQM5ARYTUXR7KMKWIOS7GQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.862791868 (176h0m32.414262329s) [noise=true , address=185.167.204.205:4021 , piece_id=MV3TV4EI3HN3J5NY5XYVKT46QJTJEC5PITXERSSZMS2B7W3DSDTA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.879947135 (176h0m32.431417596s) [noise=true , address=185.167.204.205:4001 , piece_id=TBPYFHMXWO53LJQQDUA3I2MD2WP6BHQXAH3KIF6XXMWK27HXLCHA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.886853657 (176h0m32.438324118s) [noise=true , address=185.167.204.206:4067 , piece_id=2BDMQZLCV4UWWP5HB6GF6NCL5UOGNU7VNYGJIIJCKQHRTX5OPGPQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.918801783 (176h0m32.470272324s) [noise=true , address=185.167.204.206:4051 , piece_id=AU2S6M3E2CQEMAS7LKZJIVWITZUHIHCMWQ2QWT7ILGBRTAUCUB7Q, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.929982083 (176h0m32.481452574s) [noise=true , address=185.167.204.205:4015 , piece_id=6UIFX7GXE2SZ465VAP56YQILR3A3OHDRKIS345EE6GBDG77KJNZQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.935247409 (176h0m32.48671786s) [noise=true , address=185.167.204.206:4038 , piece_id=FBNPAJOOXEAT3KLKQIYOWHGHRGAWJABV4LAA3UYX6SGJB3U2LD7A, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.969630813 (176h0m32.521101264s) [noise=true , address=185.167.204.206:4055 , piece_id=B2TTK2ZLIRM3YIKP5HQ3WOL26KBHSXWWX5ZXXCMWU34AH7CHOOCQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.999043600 (176h0m32.550514061s) [noise=true , address=185.167.204.205:4034 , piece_id=CZFPF35OQ3U4IH46KILFPFMVY6OC36ZUGZ7EYYBOTB3I4TXMPERA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.022951616 (176h0m32.574422077s) [noise=true , address=185.167.204.205:4014 , piece_id=57C2VUYOEULOTXUQHG7A2XOWZLXYFIEYYRYW5ARX5PKTGWKIEOAQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.064488381 (176h0m32.615958832s) [noise=true , address=185.167.204.206:4058 , piece_id=POK7HBLE5AT2STCSL2UJ5R67LNVK3PN53NIRIBX4M7LJYC4C2CHQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.122257987 (176h0m32.673728468s) [noise=true , address=185.167.204.205:4029 , piece_id=VF7BTI4K75HTRVJOHIKMDJN3UBHZ7KCQBT3H44XB5G7ZVFD4HXJQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.136282884 (176h0m32.687753345s) [noise=true , address=185.167.204.206:4041 , piece_id=RTQSCXR54KTKBYS6AHEKPO7ZPUHVQQNUNQJ6JI4JLTGKOLUZH3AQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.136821679 (176h0m32.68829215s) [noise=true , address=185.167.204.206:4062 , piece_id=PAJX3ZCWM5YBZA7YDBFYXCET5CKWBOFJ47IUWK4CYPZSWK6HYQVQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.156096042 (176h0m32.707566493s) [noise=true , address=185.167.204.206:4066 , piece_id=CPTBHDHCT6R2M3UA6X2ZWMPJUKAUVV4I2CIATZ2ZOIWH63Q3UT4Q, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.540981809 (176h0m33.0924523s) [noise=true , address=89.149.201.147:4012 , piece_id=NEIYQHVVWRUVJFF5UD3RZSRHUSNSKXBCNGTMIGOH5MDH2AOZISCA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:21.258617296 (176h1m19.810087757s) [noise=true , address=81.171.14.70:4034 , piece_id=NXBDCCCEWSZ2TMSET6OXBNQV6Q5ESXIQJUFK4WJJ3UV7FJTTAUXQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:22.223432573 (176h1m20.774903034s) [noise=true , address=81.171.14.70:4048 , piece_id=5WUEBVU7UFKHYZ23BPIQUMUUBH3ZXXQWBLE7MALJGECCDTTEK5EA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:22.238494073 (176h1m20.789964534s) [noise=true , address=81.171.14.70:4033 , piece_id=LUBQIVFAU7J72K5XDDS5S2ZXQUGFA5AINL4R5FV64KY24VS5KV5Q, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:22.350384206 (176h1m20.901854667s) [noise=true , address=81.171.14.70:4039 , piece_id=NMZ5357FVIKD2F3GCKY6YNNV73BAID2YBLVNSSM6VD5PHGLBHEIA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:23.704484814 (176h1m22.255955275s) [noise=true , address=81.171.14.70:4037 , piece_id=3IJ7J7RBMOORW4YPKIYJ3C6ZQRYBH7WQMLEKEQYG5UJ4TGKE2W2A, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:24.501497158 (176h1m23.052967619s) [noise=true , address=81.171.14.70:4031 , piece_id=54T5MOZZ4YJKQFUDOCCFF2SAE7XS2N77H2AOHCBECAWE5OBUXTZQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:29.810461698 (176h1m28.361932159s) [noise=true , address=81.171.14.70:4025 , piece_id=3HAT6FJSGV6VOHJBMO6L6L7677BOYVCDV3SR26LJZ6HZOZWJJ64A, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:31.413018058 (176h1m29.964488509s) [noise=true , address=81.171.14.70:4040 , piece_id=YG6ZBK6QTKUS3FCFD2EWU7HTFLOEEVG57SIER44E67QDP5HYCHOA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:1gannW154cmVP6N5BowytKfDorZDRwx2M5pi6akqjZFU38PGjV, address:81.171.14.70:4040): protocol: write tcp 188.166.225.170:59244->81.171.14.70:4040: use of closed network connection; write tcp 188.166.225.170:59244->81.171.14.70:4040: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59244->81.171.14.70:4040: use of closed network connection
Nov 16 16:05:31.413111366 (176h1m29.964581837s) [noise=true , address=81.171.14.70:4035 , piece_id=RGMXRH2Y5XBAFISK5LNAF547SAORD6XSWFUKYWOPJINXXODI5KVQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:12eLYJKne3yP1fZ1oPDLM7Xqs3GQ3ju4ZQedAiojXdVkcxNpzeA, address:81.171.14.70:4035): protocol: read tcp 188.166.225.170:55142->81.171.14.70:4035: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:05:31.413619965 (176h1m29.965090436s) [noise=true , address=81.171.14.70:4030 , piece_id=ZE5OAUTWLWNSMWSFGPEJTCXEN6L3FLWL34QBERTS3JZE7X2I2OHA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:12ZoRwa8wLw66Pjumy1ybp6wrZ79RrSU55uYZpwaY3xi95iSYde, address:81.171.14.70:4030): protocol: read tcp 188.166.225.170:40822->81.171.14.70:4030: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:05:31.414846280 (176h1m29.966316741s) [noise=true , address=81.171.14.70:4029 , piece_id=SIXL2XSCF7IO2BPOUKGH5VVXI6M7O4KJVZARYIRELR2GM4T4RABA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:18yEfU2X4jSrom1JDtQ22cUEvomcg91ohvay2Nsr1Vr8juo9to, address:81.171.14.70:4029): protocol: read tcp 188.166.225.170:56584->81.171.14.70:4029: use of closed network connection; conn closed: op
Nov 16 16:05:31.414946261 (176h1m29.966416722s) [seg_pos={0 0}, upload=155002]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil>
Nov 16 16:05:31.591745995 (176h1m30.143216456s) [upload=155002]: Done uploading segments...
Nov 16 16:05:31.591788596 (176h1m30.143259057s) [upload=155002]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24)
Duration: 5.82335617s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..................................X.........................] duration:
6.111303012s trace:
https://jaeger.dev.storj.io/trace/3e4560ab56ee4602
FAIL (2025-11-16 16:28:41 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.111332407s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [.......................X....................................] duration:
5.795219414s trace:
https://jaeger.dev.storj.io/trace/7dbd01be7ae5d01e
FAIL (2025-11-16 16:28:16 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 24)
======LOGS======
OHA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.079132696 (176h0m30.630603157s) [noise=true , address=81.171.14.70:4040 , piece_id=YG6ZBK6QTKUS3FCFD2EWU7HTFLOEEVG57SIER44E67QDP5HYCHOA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.079319683 (176h0m30.630790144s) [noise=true , address=185.167.204.205:4015 , piece_id=6UIFX7GXE2SZ465VAP56YQILR3A3OHDRKIS345EE6GBDG77KJNZQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.079646465 (176h0m30.631116926s) [noise=true , address=185.167.204.206:4062 , piece_id=PAJX3ZCWM5YBZA7YDBFYXCET5CKWBOFJ47IUWK4CYPZSWK6HYQVQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.080002833 (176h0m30.631473294s) [noise=true , address=81.171.14.70:4031 , piece_id=54T5MOZZ4YJKQFUDOCCFF2SAE7XS2N77H2AOHCBECAWE5OBUXTZQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.080062066 (176h0m30.631532527s) [noise=true , address=185.167.204.206:4067 , piece_id=2BDMQZLCV4UWWP5HB6GF6NCL5UOGNU7VNYGJIIJCKQHRTX5OPGPQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.080438673 (176h0m30.631909134s) [noise=true , address=185.167.204.205:4034 , piece_id=CZFPF35OQ3U4IH46KILFPFMVY6OC36ZUGZ7EYYBOTB3I4TXMPERA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.080675103 (176h0m30.632145564s) [noise=true , address=185.167.204.205:4014 , piece_id=57C2VUYOEULOTXUQHG7A2XOWZLXYFIEYYRYW5ARX5PKTGWKIEOAQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.081315833 (176h0m30.632786294s) [noise=true , address=81.171.14.70:4039 , piece_id=NMZ5357FVIKD2F3GCKY6YNNV73BAID2YBLVNSSM6VD5PHGLBHEIA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.082053738 (176h0m30.633524189s) [noise=true , address=185.167.204.205:4001 , piece_id=TBPYFHMXWO53LJQQDUA3I2MD2WP6BHQXAH3KIF6XXMWK27HXLCHA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.082605429 (176h0m30.63407589s) [noise=true , address=81.171.14.70:4034 , piece_id=NXBDCCCEWSZ2TMSET6OXBNQV6Q5ESXIQJUFK4WJJ3UV7FJTTAUXQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:33.280203414 (176h0m31.831673875s) [upload=155002]: Next returned nil segment
Nov 16 16:04:33.280222810 (176h0m31.831693271s) [upload=155002]: Waiting for error group managing segments...
Nov 16 16:04:33.838289781 (176h0m32.389760242s) [noise=true , address=185.167.204.205:4013 , piece_id=KF32EXUYGKQEDBPRSVAMNZV3ZMBF6DLT3U32BKX5LXLRBXBVTOQA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.848846382 (176h0m32.400316833s) [noise=true , address=185.167.204.206:4039 , piece_id=6HTUHESIRTSKR5EDMWZ2K56OKE5Y22QQM5ARYTUXR7KMKWIOS7GQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.862791868 (176h0m32.414262329s) [noise=true , address=185.167.204.205:4021 , piece_id=MV3TV4EI3HN3J5NY5XYVKT46QJTJEC5PITXERSSZMS2B7W3DSDTA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.879947135 (176h0m32.431417596s) [noise=true , address=185.167.204.205:4001 , piece_id=TBPYFHMXWO53LJQQDUA3I2MD2WP6BHQXAH3KIF6XXMWK27HXLCHA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.886853657 (176h0m32.438324118s) [noise=true , address=185.167.204.206:4067 , piece_id=2BDMQZLCV4UWWP5HB6GF6NCL5UOGNU7VNYGJIIJCKQHRTX5OPGPQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.918801783 (176h0m32.470272324s) [noise=true , address=185.167.204.206:4051 , piece_id=AU2S6M3E2CQEMAS7LKZJIVWITZUHIHCMWQ2QWT7ILGBRTAUCUB7Q, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.929982083 (176h0m32.481452574s) [noise=true , address=185.167.204.205:4015 , piece_id=6UIFX7GXE2SZ465VAP56YQILR3A3OHDRKIS345EE6GBDG77KJNZQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.935247409 (176h0m32.48671786s) [noise=true , address=185.167.204.206:4038 , piece_id=FBNPAJOOXEAT3KLKQIYOWHGHRGAWJABV4LAA3UYX6SGJB3U2LD7A, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.969630813 (176h0m32.521101264s) [noise=true , address=185.167.204.206:4055 , piece_id=B2TTK2ZLIRM3YIKP5HQ3WOL26KBHSXWWX5ZXXCMWU34AH7CHOOCQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.999043600 (176h0m32.550514061s) [noise=true , address=185.167.204.205:4034 , piece_id=CZFPF35OQ3U4IH46KILFPFMVY6OC36ZUGZ7EYYBOTB3I4TXMPERA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.022951616 (176h0m32.574422077s) [noise=true , address=185.167.204.205:4014 , piece_id=57C2VUYOEULOTXUQHG7A2XOWZLXYFIEYYRYW5ARX5PKTGWKIEOAQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.064488381 (176h0m32.615958832s) [noise=true , address=185.167.204.206:4058 , piece_id=POK7HBLE5AT2STCSL2UJ5R67LNVK3PN53NIRIBX4M7LJYC4C2CHQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.122257987 (176h0m32.673728468s) [noise=true , address=185.167.204.205:4029 , piece_id=VF7BTI4K75HTRVJOHIKMDJN3UBHZ7KCQBT3H44XB5G7ZVFD4HXJQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.136282884 (176h0m32.687753345s) [noise=true , address=185.167.204.206:4041 , piece_id=RTQSCXR54KTKBYS6AHEKPO7ZPUHVQQNUNQJ6JI4JLTGKOLUZH3AQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.136821679 (176h0m32.68829215s) [noise=true , address=185.167.204.206:4062 , piece_id=PAJX3ZCWM5YBZA7YDBFYXCET5CKWBOFJ47IUWK4CYPZSWK6HYQVQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.156096042 (176h0m32.707566493s) [noise=true , address=185.167.204.206:4066 , piece_id=CPTBHDHCT6R2M3UA6X2ZWMPJUKAUVV4I2CIATZ2ZOIWH63Q3UT4Q, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.540981809 (176h0m33.0924523s) [noise=true , address=89.149.201.147:4012 , piece_id=NEIYQHVVWRUVJFF5UD3RZSRHUSNSKXBCNGTMIGOH5MDH2AOZISCA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:21.258617296 (176h1m19.810087757s) [noise=true , address=81.171.14.70:4034 , piece_id=NXBDCCCEWSZ2TMSET6OXBNQV6Q5ESXIQJUFK4WJJ3UV7FJTTAUXQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:22.223432573 (176h1m20.774903034s) [noise=true , address=81.171.14.70:4048 , piece_id=5WUEBVU7UFKHYZ23BPIQUMUUBH3ZXXQWBLE7MALJGECCDTTEK5EA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:22.238494073 (176h1m20.789964534s) [noise=true , address=81.171.14.70:4033 , piece_id=LUBQIVFAU7J72K5XDDS5S2ZXQUGFA5AINL4R5FV64KY24VS5KV5Q, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:22.350384206 (176h1m20.901854667s) [noise=true , address=81.171.14.70:4039 , piece_id=NMZ5357FVIKD2F3GCKY6YNNV73BAID2YBLVNSSM6VD5PHGLBHEIA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:23.704484814 (176h1m22.255955275s) [noise=true , address=81.171.14.70:4037 , piece_id=3IJ7J7RBMOORW4YPKIYJ3C6ZQRYBH7WQMLEKEQYG5UJ4TGKE2W2A, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:24.501497158 (176h1m23.052967619s) [noise=true , address=81.171.14.70:4031 , piece_id=54T5MOZZ4YJKQFUDOCCFF2SAE7XS2N77H2AOHCBECAWE5OBUXTZQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:29.810461698 (176h1m28.361932159s) [noise=true , address=81.171.14.70:4025 , piece_id=3HAT6FJSGV6VOHJBMO6L6L7677BOYVCDV3SR26LJZ6HZOZWJJ64A, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:31.413018058 (176h1m29.964488509s) [noise=true , address=81.171.14.70:4040 , piece_id=YG6ZBK6QTKUS3FCFD2EWU7HTFLOEEVG57SIER44E67QDP5HYCHOA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:1gannW154cmVP6N5BowytKfDorZDRwx2M5pi6akqjZFU38PGjV, address:81.171.14.70:4040): protocol: write tcp 188.166.225.170:59244->81.171.14.70:4040: use of closed network connection; write tcp 188.166.225.170:59244->81.171.14.70:4040: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59244->81.171.14.70:4040: use of closed network connection
Nov 16 16:05:31.413111366 (176h1m29.964581837s) [noise=true , address=81.171.14.70:4035 , piece_id=RGMXRH2Y5XBAFISK5LNAF547SAORD6XSWFUKYWOPJINXXODI5KVQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:12eLYJKne3yP1fZ1oPDLM7Xqs3GQ3ju4ZQedAiojXdVkcxNpzeA, address:81.171.14.70:4035): protocol: read tcp 188.166.225.170:55142->81.171.14.70:4035: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:05:31.413619965 (176h1m29.965090436s) [noise=true , address=81.171.14.70:4030 , piece_id=ZE5OAUTWLWNSMWSFGPEJTCXEN6L3FLWL34QBERTS3JZE7X2I2OHA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:12ZoRwa8wLw66Pjumy1ybp6wrZ79RrSU55uYZpwaY3xi95iSYde, address:81.171.14.70:4030): protocol: read tcp 188.166.225.170:40822->81.171.14.70:4030: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:05:31.414846280 (176h1m29.966316741s) [noise=true , address=81.171.14.70:4029 , piece_id=SIXL2XSCF7IO2BPOUKGH5VVXI6M7O4KJVZARYIRELR2GM4T4RABA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:18yEfU2X4jSrom1JDtQ22cUEvomcg91ohvay2Nsr1Vr8juo9to, address:81.171.14.70:4029): protocol: read tcp 188.166.225.170:56584->81.171.14.70:4029: use of closed network connection; conn closed: op
Nov 16 16:05:31.414946261 (176h1m29.966416722s) [seg_pos={0 0}, upload=155002]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil>
Nov 16 16:05:31.591745995 (176h1m30.143216456s) [upload=155002]: Done uploading segments...
Nov 16 16:05:31.591788596 (176h1m30.143259057s) [upload=155002]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24)
Duration: 5.795241125s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...................................X........................] duration:
6.060107329s trace:
https://jaeger.dev.storj.io/trace/3e4560ab56ee4602
FAIL (2025-11-16 16:28:11 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.060134099s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [....................................X.......................] duration:
6.003167404s trace:
https://jaeger.dev.storj.io/trace/70f81d2d9b012a3a
FAIL (2025-11-16 16:27:51 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 4)
======LOGS======
ddress=45.91.116.5:4297 , piece_id=OSMI3VGAKSCIZMDQ3Y2VZQBUOJKSXM6RWNWSAV57RDTL5TR3NIJA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.462247309 (176h7m35.013717781s) [noise=true , address=45.91.116.7:4011 , piece_id=ORJVRMFIOEXZ6OB6QFQDL764Q2FCWOJT63NBIRI6SOOS2FEIBCFQ, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.463420774 (176h7m35.014891235s) [noise=true , address=145.79.227.9:4367 , piece_id=LOFHPWCHNEZ4E4WANJM5BE2TIWVFH4LGEJJYK4ETB2LZWYCAVGZA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.464113112 (176h7m35.015583573s) [noise=true , address=145.79.227.22:4511 , piece_id=HES465EP3OPD5JBVGVYMGLUOHJKSSMJDXMJVWET5GHRPFZJFIFOA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.464764873 (176h7m35.016235334s) [noise=true , address=45.91.116.8:4048 , piece_id=ULVK52VWCW3KNEPWSDYGRHWHIJWEQU2XNFEPYJFXWLVJI6AGZQOA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.465452753 (176h7m35.016923214s) [noise=true , address=145.79.227.28:4725 , piece_id=CI2LSBGPW7RYH4RGWODGWLUTN353EODNV6IB2YQGHOZ3FPSEEK5A, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.466223270 (176h7m35.017693731s) [noise=true , address=145.79.227.16:4910 , piece_id=3U2SEVMPILIEBRHS3M4YT4Z4PSYAQAIC2SKGYMW4FNODACZLOQCQ, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.466955855 (176h7m35.018426306s) [noise=true , address=145.79.227.4:4186 , piece_id=5PZIK3V72CUQQ6M2HJZFVM5OVX4HPHY7ZHJ2BAK5S3ZQOGM4J5OA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.467675205 (176h7m35.019145666s) [noise=true , address=145.79.227.8:4331 , piece_id=4O27HU7AMR6G4Y43HJYN22UXVLETBDBE7JHL4JUZKRM54SK5E4JA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.468178092 (176h7m35.019648553s) [noise=true , address=145.79.227.8:4329 , piece_id=GO5IAWQYS3UARWYAQAEUIIS456DFWCFWRJQG2KOVVL5KPXWPK4AA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:39.111804678 (176h7m37.663275129s) [upload=155107]: Next returned nil segment
Nov 16 16:11:39.111825767 (176h7m37.663296218s) [upload=155107]: Waiting for error group managing segments...
Nov 16 16:11:39.896671313 (176h7m38.448141765s) [noise=true , address=45.91.116.12:4227 , piece_id=CQA2LHRSFOA7EIBIFVC75EWVHFZMWMY7ODMETEAJRRNYSQFY5TVQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil>
Nov 16 16:11:39.954993420 (176h7m38.506463881s) [noise=true , address=45.91.116.8:4048 , piece_id=ULVK52VWCW3KNEPWSDYGRHWHIJWEQU2XNFEPYJFXWLVJI6AGZQOA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil>
Nov 16 16:11:40.066014029 (176h7m38.61748449s) [noise=true , address=45.91.116.7:4011 , piece_id=ORJVRMFIOEXZ6OB6QFQDL764Q2FCWOJT63NBIRI6SOOS2FEIBCFQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil>
Nov 16 16:11:41.102688250 (176h7m39.654158711s) [noise=true , address=45.91.116.5:4297 , piece_id=OSMI3VGAKSCIZMDQ3Y2VZQBUOJKSXM6RWNWSAV57RDTL5TR3NIJA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil>
Nov 16 16:12:04.264842280 (176h8m2.816312741s) [noise=true , address=145.79.227.17:4972 , piece_id=NV6FSBQM6YUTKYFAJPQCURWCZBHJ3AKJOLLGCRYTJ3KQ54U4GOWQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection
Nov 16 16:12:04.265084832 (176h8m2.816555293s) [noise=true , address=145.79.227.16:4910 , piece_id=3U2SEVMPILIEBRHS3M4YT4Z4PSYAQAIC2SKGYMW4FNODACZLOQCQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection
Nov 16 16:12:04.270267942 (176h8m2.821738423s) [noise=true , address=145.79.227.8:4329 , piece_id=GO5IAWQYS3UARWYAQAEUIIS456DFWCFWRJQG2KOVVL5KPXWPK4AA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: expected piece hash; context deadline exceeded; EOF
Nov 16 16:12:04.270306515 (176h8m2.821776986s) [noise=true , address=145.79.227.9:4367 , piece_id=LOFHPWCHNEZ4E4WANJM5BE2TIWVFH4LGEJJYK4ETB2LZWYCAVGZA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection
Nov 16 16:12:04.270323538 (176h8m2.821794099s) [noise=true , address=145.79.227.28:4725 , piece_id=CI2LSBGPW7RYH4RGWODGWLUTN353EODNV6IB2YQGHOZ3FPSEEK5A, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection
Nov 16 16:12:04.270544138 (176h8m2.822014739s) [noise=true , address=145.79.227.14:4840 , piece_id=RSHGL65RI7HZGU3W3EROFL23BCK2EW3YN6CAD3NO5OVE5QM3QG3Q, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: read tcp 188.166.225.170:42148->145.79.227.14:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:12:04.270571570 (176h8m2.822042121s) [noise=true , address=145.79.227.31:4797 , piece_id=AWSPIPRKSBOKQFZRRB34RUPLYVJGKAOOJQ4U7RRATQJELM3OCFHA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection
Nov 16 16:12:04.270609783 (176h8m2.822080254s) [noise=true , address=145.79.227.22:4511 , piece_id=HES465EP3OPD5JBVGVYMGLUOHJKSSMJDXMJVWET5GHRPFZJFIFOA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: read tcp 188.166.225.170:51004->145.79.227.22:4511: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:12:04.270625552 (176h8m2.822096083s) [noise=true , address=145.79.227.8:4331 , piece_id=4O27HU7AMR6G4Y43HJYN22UXVLETBDBE7JHL4JUZKRM54SK5E4JA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: read tcp 188.166.225.170:44510->145.79.227.8:4331: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:12:04.270742155 (176h8m2.822212616s) [noise=true , address=145.79.227.4:4186 , piece_id=5PZIK3V72CUQQ6M2HJZFVM5OVX4HPHY7ZHJ2BAK5S3ZQOGM4J5OA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection
Nov 16 16:12:04.270762814 (176h8m2.822233275s) [noise=true , address=145.79.227.24:4581 , piece_id=XRCS52EJDB7HWD4RDZK3UNB6VKHOTWWID42RGBJZQHTGG4NDYIPA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: read tcp 188.166.225.170:44130->145.79.227.24:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:12:04.270781470 (176h8m2.822251921s) [noise=true , address=145.79.227.11:4732 , piece_id=FQGAVGZ5CID3LBET3CVK6FSCOP5OUYGS44XYWMHU2RLIT3MBN62A, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: read tcp 188.166.225.170:42420->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:12:04.270825082 (176h8m2.822295544s) [seg_pos={0 0}, upload=155107]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil>
Nov 16 16:12:05.262661603 (176h8m3.814132064s) [upload=155107]: Done uploading segments...
Nov 16 16:12:05.262728080 (176h8m3.814198541s) [upload=155107]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4)
Duration: 6.003195756s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [........................X...................................] duration:
5.842474765s trace:
https://jaeger.dev.storj.io/trace/7dbd01be7ae5d01e
FAIL (2025-11-16 16:27:35 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 24)
======LOGS======
OHA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.079132696 (176h0m30.630603157s) [noise=true , address=81.171.14.70:4040 , piece_id=YG6ZBK6QTKUS3FCFD2EWU7HTFLOEEVG57SIER44E67QDP5HYCHOA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.079319683 (176h0m30.630790144s) [noise=true , address=185.167.204.205:4015 , piece_id=6UIFX7GXE2SZ465VAP56YQILR3A3OHDRKIS345EE6GBDG77KJNZQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.079646465 (176h0m30.631116926s) [noise=true , address=185.167.204.206:4062 , piece_id=PAJX3ZCWM5YBZA7YDBFYXCET5CKWBOFJ47IUWK4CYPZSWK6HYQVQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.080002833 (176h0m30.631473294s) [noise=true , address=81.171.14.70:4031 , piece_id=54T5MOZZ4YJKQFUDOCCFF2SAE7XS2N77H2AOHCBECAWE5OBUXTZQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.080062066 (176h0m30.631532527s) [noise=true , address=185.167.204.206:4067 , piece_id=2BDMQZLCV4UWWP5HB6GF6NCL5UOGNU7VNYGJIIJCKQHRTX5OPGPQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.080438673 (176h0m30.631909134s) [noise=true , address=185.167.204.205:4034 , piece_id=CZFPF35OQ3U4IH46KILFPFMVY6OC36ZUGZ7EYYBOTB3I4TXMPERA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.080675103 (176h0m30.632145564s) [noise=true , address=185.167.204.205:4014 , piece_id=57C2VUYOEULOTXUQHG7A2XOWZLXYFIEYYRYW5ARX5PKTGWKIEOAQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.081315833 (176h0m30.632786294s) [noise=true , address=81.171.14.70:4039 , piece_id=NMZ5357FVIKD2F3GCKY6YNNV73BAID2YBLVNSSM6VD5PHGLBHEIA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.082053738 (176h0m30.633524189s) [noise=true , address=185.167.204.205:4001 , piece_id=TBPYFHMXWO53LJQQDUA3I2MD2WP6BHQXAH3KIF6XXMWK27HXLCHA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.082605429 (176h0m30.63407589s) [noise=true , address=81.171.14.70:4034 , piece_id=NXBDCCCEWSZ2TMSET6OXBNQV6Q5ESXIQJUFK4WJJ3UV7FJTTAUXQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:33.280203414 (176h0m31.831673875s) [upload=155002]: Next returned nil segment
Nov 16 16:04:33.280222810 (176h0m31.831693271s) [upload=155002]: Waiting for error group managing segments...
Nov 16 16:04:33.838289781 (176h0m32.389760242s) [noise=true , address=185.167.204.205:4013 , piece_id=KF32EXUYGKQEDBPRSVAMNZV3ZMBF6DLT3U32BKX5LXLRBXBVTOQA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.848846382 (176h0m32.400316833s) [noise=true , address=185.167.204.206:4039 , piece_id=6HTUHESIRTSKR5EDMWZ2K56OKE5Y22QQM5ARYTUXR7KMKWIOS7GQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.862791868 (176h0m32.414262329s) [noise=true , address=185.167.204.205:4021 , piece_id=MV3TV4EI3HN3J5NY5XYVKT46QJTJEC5PITXERSSZMS2B7W3DSDTA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.879947135 (176h0m32.431417596s) [noise=true , address=185.167.204.205:4001 , piece_id=TBPYFHMXWO53LJQQDUA3I2MD2WP6BHQXAH3KIF6XXMWK27HXLCHA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.886853657 (176h0m32.438324118s) [noise=true , address=185.167.204.206:4067 , piece_id=2BDMQZLCV4UWWP5HB6GF6NCL5UOGNU7VNYGJIIJCKQHRTX5OPGPQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.918801783 (176h0m32.470272324s) [noise=true , address=185.167.204.206:4051 , piece_id=AU2S6M3E2CQEMAS7LKZJIVWITZUHIHCMWQ2QWT7ILGBRTAUCUB7Q, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.929982083 (176h0m32.481452574s) [noise=true , address=185.167.204.205:4015 , piece_id=6UIFX7GXE2SZ465VAP56YQILR3A3OHDRKIS345EE6GBDG77KJNZQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.935247409 (176h0m32.48671786s) [noise=true , address=185.167.204.206:4038 , piece_id=FBNPAJOOXEAT3KLKQIYOWHGHRGAWJABV4LAA3UYX6SGJB3U2LD7A, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.969630813 (176h0m32.521101264s) [noise=true , address=185.167.204.206:4055 , piece_id=B2TTK2ZLIRM3YIKP5HQ3WOL26KBHSXWWX5ZXXCMWU34AH7CHOOCQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.999043600 (176h0m32.550514061s) [noise=true , address=185.167.204.205:4034 , piece_id=CZFPF35OQ3U4IH46KILFPFMVY6OC36ZUGZ7EYYBOTB3I4TXMPERA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.022951616 (176h0m32.574422077s) [noise=true , address=185.167.204.205:4014 , piece_id=57C2VUYOEULOTXUQHG7A2XOWZLXYFIEYYRYW5ARX5PKTGWKIEOAQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.064488381 (176h0m32.615958832s) [noise=true , address=185.167.204.206:4058 , piece_id=POK7HBLE5AT2STCSL2UJ5R67LNVK3PN53NIRIBX4M7LJYC4C2CHQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.122257987 (176h0m32.673728468s) [noise=true , address=185.167.204.205:4029 , piece_id=VF7BTI4K75HTRVJOHIKMDJN3UBHZ7KCQBT3H44XB5G7ZVFD4HXJQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.136282884 (176h0m32.687753345s) [noise=true , address=185.167.204.206:4041 , piece_id=RTQSCXR54KTKBYS6AHEKPO7ZPUHVQQNUNQJ6JI4JLTGKOLUZH3AQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.136821679 (176h0m32.68829215s) [noise=true , address=185.167.204.206:4062 , piece_id=PAJX3ZCWM5YBZA7YDBFYXCET5CKWBOFJ47IUWK4CYPZSWK6HYQVQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.156096042 (176h0m32.707566493s) [noise=true , address=185.167.204.206:4066 , piece_id=CPTBHDHCT6R2M3UA6X2ZWMPJUKAUVV4I2CIATZ2ZOIWH63Q3UT4Q, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.540981809 (176h0m33.0924523s) [noise=true , address=89.149.201.147:4012 , piece_id=NEIYQHVVWRUVJFF5UD3RZSRHUSNSKXBCNGTMIGOH5MDH2AOZISCA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:21.258617296 (176h1m19.810087757s) [noise=true , address=81.171.14.70:4034 , piece_id=NXBDCCCEWSZ2TMSET6OXBNQV6Q5ESXIQJUFK4WJJ3UV7FJTTAUXQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:22.223432573 (176h1m20.774903034s) [noise=true , address=81.171.14.70:4048 , piece_id=5WUEBVU7UFKHYZ23BPIQUMUUBH3ZXXQWBLE7MALJGECCDTTEK5EA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:22.238494073 (176h1m20.789964534s) [noise=true , address=81.171.14.70:4033 , piece_id=LUBQIVFAU7J72K5XDDS5S2ZXQUGFA5AINL4R5FV64KY24VS5KV5Q, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:22.350384206 (176h1m20.901854667s) [noise=true , address=81.171.14.70:4039 , piece_id=NMZ5357FVIKD2F3GCKY6YNNV73BAID2YBLVNSSM6VD5PHGLBHEIA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:23.704484814 (176h1m22.255955275s) [noise=true , address=81.171.14.70:4037 , piece_id=3IJ7J7RBMOORW4YPKIYJ3C6ZQRYBH7WQMLEKEQYG5UJ4TGKE2W2A, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:24.501497158 (176h1m23.052967619s) [noise=true , address=81.171.14.70:4031 , piece_id=54T5MOZZ4YJKQFUDOCCFF2SAE7XS2N77H2AOHCBECAWE5OBUXTZQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:29.810461698 (176h1m28.361932159s) [noise=true , address=81.171.14.70:4025 , piece_id=3HAT6FJSGV6VOHJBMO6L6L7677BOYVCDV3SR26LJZ6HZOZWJJ64A, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:31.413018058 (176h1m29.964488509s) [noise=true , address=81.171.14.70:4040 , piece_id=YG6ZBK6QTKUS3FCFD2EWU7HTFLOEEVG57SIER44E67QDP5HYCHOA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:1gannW154cmVP6N5BowytKfDorZDRwx2M5pi6akqjZFU38PGjV, address:81.171.14.70:4040): protocol: write tcp 188.166.225.170:59244->81.171.14.70:4040: use of closed network connection; write tcp 188.166.225.170:59244->81.171.14.70:4040: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59244->81.171.14.70:4040: use of closed network connection
Nov 16 16:05:31.413111366 (176h1m29.964581837s) [noise=true , address=81.171.14.70:4035 , piece_id=RGMXRH2Y5XBAFISK5LNAF547SAORD6XSWFUKYWOPJINXXODI5KVQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:12eLYJKne3yP1fZ1oPDLM7Xqs3GQ3ju4ZQedAiojXdVkcxNpzeA, address:81.171.14.70:4035): protocol: read tcp 188.166.225.170:55142->81.171.14.70:4035: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:05:31.413619965 (176h1m29.965090436s) [noise=true , address=81.171.14.70:4030 , piece_id=ZE5OAUTWLWNSMWSFGPEJTCXEN6L3FLWL34QBERTS3JZE7X2I2OHA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:12ZoRwa8wLw66Pjumy1ybp6wrZ79RrSU55uYZpwaY3xi95iSYde, address:81.171.14.70:4030): protocol: read tcp 188.166.225.170:40822->81.171.14.70:4030: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:05:31.414846280 (176h1m29.966316741s) [noise=true , address=81.171.14.70:4029 , piece_id=SIXL2XSCF7IO2BPOUKGH5VVXI6M7O4KJVZARYIRELR2GM4T4RABA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:18yEfU2X4jSrom1JDtQ22cUEvomcg91ohvay2Nsr1Vr8juo9to, address:81.171.14.70:4029): protocol: read tcp 188.166.225.170:56584->81.171.14.70:4029: use of closed network connection; conn closed: op
Nov 16 16:05:31.414946261 (176h1m29.966416722s) [seg_pos={0 0}, upload=155002]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil>
Nov 16 16:05:31.591745995 (176h1m30.143216456s) [upload=155002]: Done uploading segments...
Nov 16 16:05:31.591788596 (176h1m30.143259057s) [upload=155002]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24)
Duration: 5.842499841s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [....................................X.......................] duration:
6.291377809s trace:
https://jaeger.dev.storj.io/trace/3e4560ab56ee4602
FAIL (2025-11-16 16:27:20 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.291403889s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [.........................X..................................] duration:
5.724859564s trace:
https://jaeger.dev.storj.io/trace/7dbd01be7ae5d01e
FAIL (2025-11-16 16:27:17 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 24)
======LOGS======
OHA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.079132696 (176h0m30.630603157s) [noise=true , address=81.171.14.70:4040 , piece_id=YG6ZBK6QTKUS3FCFD2EWU7HTFLOEEVG57SIER44E67QDP5HYCHOA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.079319683 (176h0m30.630790144s) [noise=true , address=185.167.204.205:4015 , piece_id=6UIFX7GXE2SZ465VAP56YQILR3A3OHDRKIS345EE6GBDG77KJNZQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.079646465 (176h0m30.631116926s) [noise=true , address=185.167.204.206:4062 , piece_id=PAJX3ZCWM5YBZA7YDBFYXCET5CKWBOFJ47IUWK4CYPZSWK6HYQVQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.080002833 (176h0m30.631473294s) [noise=true , address=81.171.14.70:4031 , piece_id=54T5MOZZ4YJKQFUDOCCFF2SAE7XS2N77H2AOHCBECAWE5OBUXTZQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.080062066 (176h0m30.631532527s) [noise=true , address=185.167.204.206:4067 , piece_id=2BDMQZLCV4UWWP5HB6GF6NCL5UOGNU7VNYGJIIJCKQHRTX5OPGPQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.080438673 (176h0m30.631909134s) [noise=true , address=185.167.204.205:4034 , piece_id=CZFPF35OQ3U4IH46KILFPFMVY6OC36ZUGZ7EYYBOTB3I4TXMPERA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.080675103 (176h0m30.632145564s) [noise=true , address=185.167.204.205:4014 , piece_id=57C2VUYOEULOTXUQHG7A2XOWZLXYFIEYYRYW5ARX5PKTGWKIEOAQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.081315833 (176h0m30.632786294s) [noise=true , address=81.171.14.70:4039 , piece_id=NMZ5357FVIKD2F3GCKY6YNNV73BAID2YBLVNSSM6VD5PHGLBHEIA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.082053738 (176h0m30.633524189s) [noise=true , address=185.167.204.205:4001 , piece_id=TBPYFHMXWO53LJQQDUA3I2MD2WP6BHQXAH3KIF6XXMWK27HXLCHA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.082605429 (176h0m30.63407589s) [noise=true , address=81.171.14.70:4034 , piece_id=NXBDCCCEWSZ2TMSET6OXBNQV6Q5ESXIQJUFK4WJJ3UV7FJTTAUXQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:33.280203414 (176h0m31.831673875s) [upload=155002]: Next returned nil segment
Nov 16 16:04:33.280222810 (176h0m31.831693271s) [upload=155002]: Waiting for error group managing segments...
Nov 16 16:04:33.838289781 (176h0m32.389760242s) [noise=true , address=185.167.204.205:4013 , piece_id=KF32EXUYGKQEDBPRSVAMNZV3ZMBF6DLT3U32BKX5LXLRBXBVTOQA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.848846382 (176h0m32.400316833s) [noise=true , address=185.167.204.206:4039 , piece_id=6HTUHESIRTSKR5EDMWZ2K56OKE5Y22QQM5ARYTUXR7KMKWIOS7GQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.862791868 (176h0m32.414262329s) [noise=true , address=185.167.204.205:4021 , piece_id=MV3TV4EI3HN3J5NY5XYVKT46QJTJEC5PITXERSSZMS2B7W3DSDTA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.879947135 (176h0m32.431417596s) [noise=true , address=185.167.204.205:4001 , piece_id=TBPYFHMXWO53LJQQDUA3I2MD2WP6BHQXAH3KIF6XXMWK27HXLCHA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.886853657 (176h0m32.438324118s) [noise=true , address=185.167.204.206:4067 , piece_id=2BDMQZLCV4UWWP5HB6GF6NCL5UOGNU7VNYGJIIJCKQHRTX5OPGPQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.918801783 (176h0m32.470272324s) [noise=true , address=185.167.204.206:4051 , piece_id=AU2S6M3E2CQEMAS7LKZJIVWITZUHIHCMWQ2QWT7ILGBRTAUCUB7Q, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.929982083 (176h0m32.481452574s) [noise=true , address=185.167.204.205:4015 , piece_id=6UIFX7GXE2SZ465VAP56YQILR3A3OHDRKIS345EE6GBDG77KJNZQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.935247409 (176h0m32.48671786s) [noise=true , address=185.167.204.206:4038 , piece_id=FBNPAJOOXEAT3KLKQIYOWHGHRGAWJABV4LAA3UYX6SGJB3U2LD7A, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.969630813 (176h0m32.521101264s) [noise=true , address=185.167.204.206:4055 , piece_id=B2TTK2ZLIRM3YIKP5HQ3WOL26KBHSXWWX5ZXXCMWU34AH7CHOOCQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.999043600 (176h0m32.550514061s) [noise=true , address=185.167.204.205:4034 , piece_id=CZFPF35OQ3U4IH46KILFPFMVY6OC36ZUGZ7EYYBOTB3I4TXMPERA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.022951616 (176h0m32.574422077s) [noise=true , address=185.167.204.205:4014 , piece_id=57C2VUYOEULOTXUQHG7A2XOWZLXYFIEYYRYW5ARX5PKTGWKIEOAQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.064488381 (176h0m32.615958832s) [noise=true , address=185.167.204.206:4058 , piece_id=POK7HBLE5AT2STCSL2UJ5R67LNVK3PN53NIRIBX4M7LJYC4C2CHQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.122257987 (176h0m32.673728468s) [noise=true , address=185.167.204.205:4029 , piece_id=VF7BTI4K75HTRVJOHIKMDJN3UBHZ7KCQBT3H44XB5G7ZVFD4HXJQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.136282884 (176h0m32.687753345s) [noise=true , address=185.167.204.206:4041 , piece_id=RTQSCXR54KTKBYS6AHEKPO7ZPUHVQQNUNQJ6JI4JLTGKOLUZH3AQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.136821679 (176h0m32.68829215s) [noise=true , address=185.167.204.206:4062 , piece_id=PAJX3ZCWM5YBZA7YDBFYXCET5CKWBOFJ47IUWK4CYPZSWK6HYQVQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.156096042 (176h0m32.707566493s) [noise=true , address=185.167.204.206:4066 , piece_id=CPTBHDHCT6R2M3UA6X2ZWMPJUKAUVV4I2CIATZ2ZOIWH63Q3UT4Q, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.540981809 (176h0m33.0924523s) [noise=true , address=89.149.201.147:4012 , piece_id=NEIYQHVVWRUVJFF5UD3RZSRHUSNSKXBCNGTMIGOH5MDH2AOZISCA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:21.258617296 (176h1m19.810087757s) [noise=true , address=81.171.14.70:4034 , piece_id=NXBDCCCEWSZ2TMSET6OXBNQV6Q5ESXIQJUFK4WJJ3UV7FJTTAUXQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:22.223432573 (176h1m20.774903034s) [noise=true , address=81.171.14.70:4048 , piece_id=5WUEBVU7UFKHYZ23BPIQUMUUBH3ZXXQWBLE7MALJGECCDTTEK5EA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:22.238494073 (176h1m20.789964534s) [noise=true , address=81.171.14.70:4033 , piece_id=LUBQIVFAU7J72K5XDDS5S2ZXQUGFA5AINL4R5FV64KY24VS5KV5Q, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:22.350384206 (176h1m20.901854667s) [noise=true , address=81.171.14.70:4039 , piece_id=NMZ5357FVIKD2F3GCKY6YNNV73BAID2YBLVNSSM6VD5PHGLBHEIA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:23.704484814 (176h1m22.255955275s) [noise=true , address=81.171.14.70:4037 , piece_id=3IJ7J7RBMOORW4YPKIYJ3C6ZQRYBH7WQMLEKEQYG5UJ4TGKE2W2A, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:24.501497158 (176h1m23.052967619s) [noise=true , address=81.171.14.70:4031 , piece_id=54T5MOZZ4YJKQFUDOCCFF2SAE7XS2N77H2AOHCBECAWE5OBUXTZQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:29.810461698 (176h1m28.361932159s) [noise=true , address=81.171.14.70:4025 , piece_id=3HAT6FJSGV6VOHJBMO6L6L7677BOYVCDV3SR26LJZ6HZOZWJJ64A, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:31.413018058 (176h1m29.964488509s) [noise=true , address=81.171.14.70:4040 , piece_id=YG6ZBK6QTKUS3FCFD2EWU7HTFLOEEVG57SIER44E67QDP5HYCHOA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:1gannW154cmVP6N5BowytKfDorZDRwx2M5pi6akqjZFU38PGjV, address:81.171.14.70:4040): protocol: write tcp 188.166.225.170:59244->81.171.14.70:4040: use of closed network connection; write tcp 188.166.225.170:59244->81.171.14.70:4040: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59244->81.171.14.70:4040: use of closed network connection
Nov 16 16:05:31.413111366 (176h1m29.964581837s) [noise=true , address=81.171.14.70:4035 , piece_id=RGMXRH2Y5XBAFISK5LNAF547SAORD6XSWFUKYWOPJINXXODI5KVQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:12eLYJKne3yP1fZ1oPDLM7Xqs3GQ3ju4ZQedAiojXdVkcxNpzeA, address:81.171.14.70:4035): protocol: read tcp 188.166.225.170:55142->81.171.14.70:4035: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:05:31.413619965 (176h1m29.965090436s) [noise=true , address=81.171.14.70:4030 , piece_id=ZE5OAUTWLWNSMWSFGPEJTCXEN6L3FLWL34QBERTS3JZE7X2I2OHA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:12ZoRwa8wLw66Pjumy1ybp6wrZ79RrSU55uYZpwaY3xi95iSYde, address:81.171.14.70:4030): protocol: read tcp 188.166.225.170:40822->81.171.14.70:4030: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:05:31.414846280 (176h1m29.966316741s) [noise=true , address=81.171.14.70:4029 , piece_id=SIXL2XSCF7IO2BPOUKGH5VVXI6M7O4KJVZARYIRELR2GM4T4RABA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:18yEfU2X4jSrom1JDtQ22cUEvomcg91ohvay2Nsr1Vr8juo9to, address:81.171.14.70:4029): protocol: read tcp 188.166.225.170:56584->81.171.14.70:4029: use of closed network connection; conn closed: op
Nov 16 16:05:31.414946261 (176h1m29.966416722s) [seg_pos={0 0}, upload=155002]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil>
Nov 16 16:05:31.591745995 (176h1m30.143216456s) [upload=155002]: Done uploading segments...
Nov 16 16:05:31.591788596 (176h1m30.143259057s) [upload=155002]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24)
Duration: 5.724884931s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....................................X......................] duration:
6.073159967s trace:
https://jaeger.dev.storj.io/trace/70f81d2d9b012a3a
FAIL (2025-11-16 16:27:09 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 4)
======LOGS======
ddress=45.91.116.5:4297 , piece_id=OSMI3VGAKSCIZMDQ3Y2VZQBUOJKSXM6RWNWSAV57RDTL5TR3NIJA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.462247309 (176h7m35.013717781s) [noise=true , address=45.91.116.7:4011 , piece_id=ORJVRMFIOEXZ6OB6QFQDL764Q2FCWOJT63NBIRI6SOOS2FEIBCFQ, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.463420774 (176h7m35.014891235s) [noise=true , address=145.79.227.9:4367 , piece_id=LOFHPWCHNEZ4E4WANJM5BE2TIWVFH4LGEJJYK4ETB2LZWYCAVGZA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.464113112 (176h7m35.015583573s) [noise=true , address=145.79.227.22:4511 , piece_id=HES465EP3OPD5JBVGVYMGLUOHJKSSMJDXMJVWET5GHRPFZJFIFOA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.464764873 (176h7m35.016235334s) [noise=true , address=45.91.116.8:4048 , piece_id=ULVK52VWCW3KNEPWSDYGRHWHIJWEQU2XNFEPYJFXWLVJI6AGZQOA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.465452753 (176h7m35.016923214s) [noise=true , address=145.79.227.28:4725 , piece_id=CI2LSBGPW7RYH4RGWODGWLUTN353EODNV6IB2YQGHOZ3FPSEEK5A, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.466223270 (176h7m35.017693731s) [noise=true , address=145.79.227.16:4910 , piece_id=3U2SEVMPILIEBRHS3M4YT4Z4PSYAQAIC2SKGYMW4FNODACZLOQCQ, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.466955855 (176h7m35.018426306s) [noise=true , address=145.79.227.4:4186 , piece_id=5PZIK3V72CUQQ6M2HJZFVM5OVX4HPHY7ZHJ2BAK5S3ZQOGM4J5OA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.467675205 (176h7m35.019145666s) [noise=true , address=145.79.227.8:4331 , piece_id=4O27HU7AMR6G4Y43HJYN22UXVLETBDBE7JHL4JUZKRM54SK5E4JA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.468178092 (176h7m35.019648553s) [noise=true , address=145.79.227.8:4329 , piece_id=GO5IAWQYS3UARWYAQAEUIIS456DFWCFWRJQG2KOVVL5KPXWPK4AA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:39.111804678 (176h7m37.663275129s) [upload=155107]: Next returned nil segment
Nov 16 16:11:39.111825767 (176h7m37.663296218s) [upload=155107]: Waiting for error group managing segments...
Nov 16 16:11:39.896671313 (176h7m38.448141765s) [noise=true , address=45.91.116.12:4227 , piece_id=CQA2LHRSFOA7EIBIFVC75EWVHFZMWMY7ODMETEAJRRNYSQFY5TVQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil>
Nov 16 16:11:39.954993420 (176h7m38.506463881s) [noise=true , address=45.91.116.8:4048 , piece_id=ULVK52VWCW3KNEPWSDYGRHWHIJWEQU2XNFEPYJFXWLVJI6AGZQOA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil>
Nov 16 16:11:40.066014029 (176h7m38.61748449s) [noise=true , address=45.91.116.7:4011 , piece_id=ORJVRMFIOEXZ6OB6QFQDL764Q2FCWOJT63NBIRI6SOOS2FEIBCFQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil>
Nov 16 16:11:41.102688250 (176h7m39.654158711s) [noise=true , address=45.91.116.5:4297 , piece_id=OSMI3VGAKSCIZMDQ3Y2VZQBUOJKSXM6RWNWSAV57RDTL5TR3NIJA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil>
Nov 16 16:12:04.264842280 (176h8m2.816312741s) [noise=true , address=145.79.227.17:4972 , piece_id=NV6FSBQM6YUTKYFAJPQCURWCZBHJ3AKJOLLGCRYTJ3KQ54U4GOWQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection
Nov 16 16:12:04.265084832 (176h8m2.816555293s) [noise=true , address=145.79.227.16:4910 , piece_id=3U2SEVMPILIEBRHS3M4YT4Z4PSYAQAIC2SKGYMW4FNODACZLOQCQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection
Nov 16 16:12:04.270267942 (176h8m2.821738423s) [noise=true , address=145.79.227.8:4329 , piece_id=GO5IAWQYS3UARWYAQAEUIIS456DFWCFWRJQG2KOVVL5KPXWPK4AA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: expected piece hash; context deadline exceeded; EOF
Nov 16 16:12:04.270306515 (176h8m2.821776986s) [noise=true , address=145.79.227.9:4367 , piece_id=LOFHPWCHNEZ4E4WANJM5BE2TIWVFH4LGEJJYK4ETB2LZWYCAVGZA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection
Nov 16 16:12:04.270323538 (176h8m2.821794099s) [noise=true , address=145.79.227.28:4725 , piece_id=CI2LSBGPW7RYH4RGWODGWLUTN353EODNV6IB2YQGHOZ3FPSEEK5A, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection
Nov 16 16:12:04.270544138 (176h8m2.822014739s) [noise=true , address=145.79.227.14:4840 , piece_id=RSHGL65RI7HZGU3W3EROFL23BCK2EW3YN6CAD3NO5OVE5QM3QG3Q, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: read tcp 188.166.225.170:42148->145.79.227.14:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:12:04.270571570 (176h8m2.822042121s) [noise=true , address=145.79.227.31:4797 , piece_id=AWSPIPRKSBOKQFZRRB34RUPLYVJGKAOOJQ4U7RRATQJELM3OCFHA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection
Nov 16 16:12:04.270609783 (176h8m2.822080254s) [noise=true , address=145.79.227.22:4511 , piece_id=HES465EP3OPD5JBVGVYMGLUOHJKSSMJDXMJVWET5GHRPFZJFIFOA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: read tcp 188.166.225.170:51004->145.79.227.22:4511: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:12:04.270625552 (176h8m2.822096083s) [noise=true , address=145.79.227.8:4331 , piece_id=4O27HU7AMR6G4Y43HJYN22UXVLETBDBE7JHL4JUZKRM54SK5E4JA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: read tcp 188.166.225.170:44510->145.79.227.8:4331: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:12:04.270742155 (176h8m2.822212616s) [noise=true , address=145.79.227.4:4186 , piece_id=5PZIK3V72CUQQ6M2HJZFVM5OVX4HPHY7ZHJ2BAK5S3ZQOGM4J5OA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection
Nov 16 16:12:04.270762814 (176h8m2.822233275s) [noise=true , address=145.79.227.24:4581 , piece_id=XRCS52EJDB7HWD4RDZK3UNB6VKHOTWWID42RGBJZQHTGG4NDYIPA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: read tcp 188.166.225.170:44130->145.79.227.24:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:12:04.270781470 (176h8m2.822251921s) [noise=true , address=145.79.227.11:4732 , piece_id=FQGAVGZ5CID3LBET3CVK6FSCOP5OUYGS44XYWMHU2RLIT3MBN62A, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: read tcp 188.166.225.170:42420->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:12:04.270825082 (176h8m2.822295544s) [seg_pos={0 0}, upload=155107]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil>
Nov 16 16:12:05.262661603 (176h8m3.814132064s) [upload=155107]: Done uploading segments...
Nov 16 16:12:05.262728080 (176h8m3.814198541s) [upload=155107]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4)
Duration: 6.073190335s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [..........................X.................................] duration:
5.754867864s trace:
https://jaeger.dev.storj.io/trace/7dbd01be7ae5d01e
FAIL (2025-11-16 16:26:54 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 24)
======LOGS======
OHA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.079132696 (176h0m30.630603157s) [noise=true , address=81.171.14.70:4040 , piece_id=YG6ZBK6QTKUS3FCFD2EWU7HTFLOEEVG57SIER44E67QDP5HYCHOA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.079319683 (176h0m30.630790144s) [noise=true , address=185.167.204.205:4015 , piece_id=6UIFX7GXE2SZ465VAP56YQILR3A3OHDRKIS345EE6GBDG77KJNZQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.079646465 (176h0m30.631116926s) [noise=true , address=185.167.204.206:4062 , piece_id=PAJX3ZCWM5YBZA7YDBFYXCET5CKWBOFJ47IUWK4CYPZSWK6HYQVQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.080002833 (176h0m30.631473294s) [noise=true , address=81.171.14.70:4031 , piece_id=54T5MOZZ4YJKQFUDOCCFF2SAE7XS2N77H2AOHCBECAWE5OBUXTZQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.080062066 (176h0m30.631532527s) [noise=true , address=185.167.204.206:4067 , piece_id=2BDMQZLCV4UWWP5HB6GF6NCL5UOGNU7VNYGJIIJCKQHRTX5OPGPQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.080438673 (176h0m30.631909134s) [noise=true , address=185.167.204.205:4034 , piece_id=CZFPF35OQ3U4IH46KILFPFMVY6OC36ZUGZ7EYYBOTB3I4TXMPERA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.080675103 (176h0m30.632145564s) [noise=true , address=185.167.204.205:4014 , piece_id=57C2VUYOEULOTXUQHG7A2XOWZLXYFIEYYRYW5ARX5PKTGWKIEOAQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.081315833 (176h0m30.632786294s) [noise=true , address=81.171.14.70:4039 , piece_id=NMZ5357FVIKD2F3GCKY6YNNV73BAID2YBLVNSSM6VD5PHGLBHEIA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.082053738 (176h0m30.633524189s) [noise=true , address=185.167.204.205:4001 , piece_id=TBPYFHMXWO53LJQQDUA3I2MD2WP6BHQXAH3KIF6XXMWK27HXLCHA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.082605429 (176h0m30.63407589s) [noise=true , address=81.171.14.70:4034 , piece_id=NXBDCCCEWSZ2TMSET6OXBNQV6Q5ESXIQJUFK4WJJ3UV7FJTTAUXQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:33.280203414 (176h0m31.831673875s) [upload=155002]: Next returned nil segment
Nov 16 16:04:33.280222810 (176h0m31.831693271s) [upload=155002]: Waiting for error group managing segments...
Nov 16 16:04:33.838289781 (176h0m32.389760242s) [noise=true , address=185.167.204.205:4013 , piece_id=KF32EXUYGKQEDBPRSVAMNZV3ZMBF6DLT3U32BKX5LXLRBXBVTOQA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.848846382 (176h0m32.400316833s) [noise=true , address=185.167.204.206:4039 , piece_id=6HTUHESIRTSKR5EDMWZ2K56OKE5Y22QQM5ARYTUXR7KMKWIOS7GQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.862791868 (176h0m32.414262329s) [noise=true , address=185.167.204.205:4021 , piece_id=MV3TV4EI3HN3J5NY5XYVKT46QJTJEC5PITXERSSZMS2B7W3DSDTA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.879947135 (176h0m32.431417596s) [noise=true , address=185.167.204.205:4001 , piece_id=TBPYFHMXWO53LJQQDUA3I2MD2WP6BHQXAH3KIF6XXMWK27HXLCHA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.886853657 (176h0m32.438324118s) [noise=true , address=185.167.204.206:4067 , piece_id=2BDMQZLCV4UWWP5HB6GF6NCL5UOGNU7VNYGJIIJCKQHRTX5OPGPQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.918801783 (176h0m32.470272324s) [noise=true , address=185.167.204.206:4051 , piece_id=AU2S6M3E2CQEMAS7LKZJIVWITZUHIHCMWQ2QWT7ILGBRTAUCUB7Q, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.929982083 (176h0m32.481452574s) [noise=true , address=185.167.204.205:4015 , piece_id=6UIFX7GXE2SZ465VAP56YQILR3A3OHDRKIS345EE6GBDG77KJNZQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.935247409 (176h0m32.48671786s) [noise=true , address=185.167.204.206:4038 , piece_id=FBNPAJOOXEAT3KLKQIYOWHGHRGAWJABV4LAA3UYX6SGJB3U2LD7A, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.969630813 (176h0m32.521101264s) [noise=true , address=185.167.204.206:4055 , piece_id=B2TTK2ZLIRM3YIKP5HQ3WOL26KBHSXWWX5ZXXCMWU34AH7CHOOCQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.999043600 (176h0m32.550514061s) [noise=true , address=185.167.204.205:4034 , piece_id=CZFPF35OQ3U4IH46KILFPFMVY6OC36ZUGZ7EYYBOTB3I4TXMPERA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.022951616 (176h0m32.574422077s) [noise=true , address=185.167.204.205:4014 , piece_id=57C2VUYOEULOTXUQHG7A2XOWZLXYFIEYYRYW5ARX5PKTGWKIEOAQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.064488381 (176h0m32.615958832s) [noise=true , address=185.167.204.206:4058 , piece_id=POK7HBLE5AT2STCSL2UJ5R67LNVK3PN53NIRIBX4M7LJYC4C2CHQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.122257987 (176h0m32.673728468s) [noise=true , address=185.167.204.205:4029 , piece_id=VF7BTI4K75HTRVJOHIKMDJN3UBHZ7KCQBT3H44XB5G7ZVFD4HXJQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.136282884 (176h0m32.687753345s) [noise=true , address=185.167.204.206:4041 , piece_id=RTQSCXR54KTKBYS6AHEKPO7ZPUHVQQNUNQJ6JI4JLTGKOLUZH3AQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.136821679 (176h0m32.68829215s) [noise=true , address=185.167.204.206:4062 , piece_id=PAJX3ZCWM5YBZA7YDBFYXCET5CKWBOFJ47IUWK4CYPZSWK6HYQVQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.156096042 (176h0m32.707566493s) [noise=true , address=185.167.204.206:4066 , piece_id=CPTBHDHCT6R2M3UA6X2ZWMPJUKAUVV4I2CIATZ2ZOIWH63Q3UT4Q, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.540981809 (176h0m33.0924523s) [noise=true , address=89.149.201.147:4012 , piece_id=NEIYQHVVWRUVJFF5UD3RZSRHUSNSKXBCNGTMIGOH5MDH2AOZISCA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:21.258617296 (176h1m19.810087757s) [noise=true , address=81.171.14.70:4034 , piece_id=NXBDCCCEWSZ2TMSET6OXBNQV6Q5ESXIQJUFK4WJJ3UV7FJTTAUXQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:22.223432573 (176h1m20.774903034s) [noise=true , address=81.171.14.70:4048 , piece_id=5WUEBVU7UFKHYZ23BPIQUMUUBH3ZXXQWBLE7MALJGECCDTTEK5EA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:22.238494073 (176h1m20.789964534s) [noise=true , address=81.171.14.70:4033 , piece_id=LUBQIVFAU7J72K5XDDS5S2ZXQUGFA5AINL4R5FV64KY24VS5KV5Q, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:22.350384206 (176h1m20.901854667s) [noise=true , address=81.171.14.70:4039 , piece_id=NMZ5357FVIKD2F3GCKY6YNNV73BAID2YBLVNSSM6VD5PHGLBHEIA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:23.704484814 (176h1m22.255955275s) [noise=true , address=81.171.14.70:4037 , piece_id=3IJ7J7RBMOORW4YPKIYJ3C6ZQRYBH7WQMLEKEQYG5UJ4TGKE2W2A, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:24.501497158 (176h1m23.052967619s) [noise=true , address=81.171.14.70:4031 , piece_id=54T5MOZZ4YJKQFUDOCCFF2SAE7XS2N77H2AOHCBECAWE5OBUXTZQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:29.810461698 (176h1m28.361932159s) [noise=true , address=81.171.14.70:4025 , piece_id=3HAT6FJSGV6VOHJBMO6L6L7677BOYVCDV3SR26LJZ6HZOZWJJ64A, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:31.413018058 (176h1m29.964488509s) [noise=true , address=81.171.14.70:4040 , piece_id=YG6ZBK6QTKUS3FCFD2EWU7HTFLOEEVG57SIER44E67QDP5HYCHOA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:1gannW154cmVP6N5BowytKfDorZDRwx2M5pi6akqjZFU38PGjV, address:81.171.14.70:4040): protocol: write tcp 188.166.225.170:59244->81.171.14.70:4040: use of closed network connection; write tcp 188.166.225.170:59244->81.171.14.70:4040: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59244->81.171.14.70:4040: use of closed network connection
Nov 16 16:05:31.413111366 (176h1m29.964581837s) [noise=true , address=81.171.14.70:4035 , piece_id=RGMXRH2Y5XBAFISK5LNAF547SAORD6XSWFUKYWOPJINXXODI5KVQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:12eLYJKne3yP1fZ1oPDLM7Xqs3GQ3ju4ZQedAiojXdVkcxNpzeA, address:81.171.14.70:4035): protocol: read tcp 188.166.225.170:55142->81.171.14.70:4035: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:05:31.413619965 (176h1m29.965090436s) [noise=true , address=81.171.14.70:4030 , piece_id=ZE5OAUTWLWNSMWSFGPEJTCXEN6L3FLWL34QBERTS3JZE7X2I2OHA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:12ZoRwa8wLw66Pjumy1ybp6wrZ79RrSU55uYZpwaY3xi95iSYde, address:81.171.14.70:4030): protocol: read tcp 188.166.225.170:40822->81.171.14.70:4030: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:05:31.414846280 (176h1m29.966316741s) [noise=true , address=81.171.14.70:4029 , piece_id=SIXL2XSCF7IO2BPOUKGH5VVXI6M7O4KJVZARYIRELR2GM4T4RABA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:18yEfU2X4jSrom1JDtQ22cUEvomcg91ohvay2Nsr1Vr8juo9to, address:81.171.14.70:4029): protocol: read tcp 188.166.225.170:56584->81.171.14.70:4029: use of closed network connection; conn closed: op
Nov 16 16:05:31.414946261 (176h1m29.966416722s) [seg_pos={0 0}, upload=155002]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil>
Nov 16 16:05:31.591745995 (176h1m30.143216456s) [upload=155002]: Done uploading segments...
Nov 16 16:05:31.591788596 (176h1m30.143259057s) [upload=155002]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24)
Duration: 5.754899805s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....................................X......................] duration:
6.049831952s trace:
https://jaeger.dev.storj.io/trace/3e4560ab56ee4602
FAIL (2025-11-16 16:26:47 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.049857991s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [......................................X.....................] duration:
5.930699276s trace:
https://jaeger.dev.storj.io/trace/70f81d2d9b012a3a
FAIL (2025-11-16 16:26:44 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 4)
======LOGS======
ddress=45.91.116.5:4297 , piece_id=OSMI3VGAKSCIZMDQ3Y2VZQBUOJKSXM6RWNWSAV57RDTL5TR3NIJA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.462247309 (176h7m35.013717781s) [noise=true , address=45.91.116.7:4011 , piece_id=ORJVRMFIOEXZ6OB6QFQDL764Q2FCWOJT63NBIRI6SOOS2FEIBCFQ, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.463420774 (176h7m35.014891235s) [noise=true , address=145.79.227.9:4367 , piece_id=LOFHPWCHNEZ4E4WANJM5BE2TIWVFH4LGEJJYK4ETB2LZWYCAVGZA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.464113112 (176h7m35.015583573s) [noise=true , address=145.79.227.22:4511 , piece_id=HES465EP3OPD5JBVGVYMGLUOHJKSSMJDXMJVWET5GHRPFZJFIFOA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.464764873 (176h7m35.016235334s) [noise=true , address=45.91.116.8:4048 , piece_id=ULVK52VWCW3KNEPWSDYGRHWHIJWEQU2XNFEPYJFXWLVJI6AGZQOA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.465452753 (176h7m35.016923214s) [noise=true , address=145.79.227.28:4725 , piece_id=CI2LSBGPW7RYH4RGWODGWLUTN353EODNV6IB2YQGHOZ3FPSEEK5A, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.466223270 (176h7m35.017693731s) [noise=true , address=145.79.227.16:4910 , piece_id=3U2SEVMPILIEBRHS3M4YT4Z4PSYAQAIC2SKGYMW4FNODACZLOQCQ, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.466955855 (176h7m35.018426306s) [noise=true , address=145.79.227.4:4186 , piece_id=5PZIK3V72CUQQ6M2HJZFVM5OVX4HPHY7ZHJ2BAK5S3ZQOGM4J5OA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.467675205 (176h7m35.019145666s) [noise=true , address=145.79.227.8:4331 , piece_id=4O27HU7AMR6G4Y43HJYN22UXVLETBDBE7JHL4JUZKRM54SK5E4JA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.468178092 (176h7m35.019648553s) [noise=true , address=145.79.227.8:4329 , piece_id=GO5IAWQYS3UARWYAQAEUIIS456DFWCFWRJQG2KOVVL5KPXWPK4AA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:39.111804678 (176h7m37.663275129s) [upload=155107]: Next returned nil segment
Nov 16 16:11:39.111825767 (176h7m37.663296218s) [upload=155107]: Waiting for error group managing segments...
Nov 16 16:11:39.896671313 (176h7m38.448141765s) [noise=true , address=45.91.116.12:4227 , piece_id=CQA2LHRSFOA7EIBIFVC75EWVHFZMWMY7ODMETEAJRRNYSQFY5TVQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil>
Nov 16 16:11:39.954993420 (176h7m38.506463881s) [noise=true , address=45.91.116.8:4048 , piece_id=ULVK52VWCW3KNEPWSDYGRHWHIJWEQU2XNFEPYJFXWLVJI6AGZQOA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil>
Nov 16 16:11:40.066014029 (176h7m38.61748449s) [noise=true , address=45.91.116.7:4011 , piece_id=ORJVRMFIOEXZ6OB6QFQDL764Q2FCWOJT63NBIRI6SOOS2FEIBCFQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil>
Nov 16 16:11:41.102688250 (176h7m39.654158711s) [noise=true , address=45.91.116.5:4297 , piece_id=OSMI3VGAKSCIZMDQ3Y2VZQBUOJKSXM6RWNWSAV57RDTL5TR3NIJA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil>
Nov 16 16:12:04.264842280 (176h8m2.816312741s) [noise=true , address=145.79.227.17:4972 , piece_id=NV6FSBQM6YUTKYFAJPQCURWCZBHJ3AKJOLLGCRYTJ3KQ54U4GOWQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection
Nov 16 16:12:04.265084832 (176h8m2.816555293s) [noise=true , address=145.79.227.16:4910 , piece_id=3U2SEVMPILIEBRHS3M4YT4Z4PSYAQAIC2SKGYMW4FNODACZLOQCQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection
Nov 16 16:12:04.270267942 (176h8m2.821738423s) [noise=true , address=145.79.227.8:4329 , piece_id=GO5IAWQYS3UARWYAQAEUIIS456DFWCFWRJQG2KOVVL5KPXWPK4AA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: expected piece hash; context deadline exceeded; EOF
Nov 16 16:12:04.270306515 (176h8m2.821776986s) [noise=true , address=145.79.227.9:4367 , piece_id=LOFHPWCHNEZ4E4WANJM5BE2TIWVFH4LGEJJYK4ETB2LZWYCAVGZA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection
Nov 16 16:12:04.270323538 (176h8m2.821794099s) [noise=true , address=145.79.227.28:4725 , piece_id=CI2LSBGPW7RYH4RGWODGWLUTN353EODNV6IB2YQGHOZ3FPSEEK5A, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection
Nov 16 16:12:04.270544138 (176h8m2.822014739s) [noise=true , address=145.79.227.14:4840 , piece_id=RSHGL65RI7HZGU3W3EROFL23BCK2EW3YN6CAD3NO5OVE5QM3QG3Q, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: read tcp 188.166.225.170:42148->145.79.227.14:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:12:04.270571570 (176h8m2.822042121s) [noise=true , address=145.79.227.31:4797 , piece_id=AWSPIPRKSBOKQFZRRB34RUPLYVJGKAOOJQ4U7RRATQJELM3OCFHA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection
Nov 16 16:12:04.270609783 (176h8m2.822080254s) [noise=true , address=145.79.227.22:4511 , piece_id=HES465EP3OPD5JBVGVYMGLUOHJKSSMJDXMJVWET5GHRPFZJFIFOA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: read tcp 188.166.225.170:51004->145.79.227.22:4511: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:12:04.270625552 (176h8m2.822096083s) [noise=true , address=145.79.227.8:4331 , piece_id=4O27HU7AMR6G4Y43HJYN22UXVLETBDBE7JHL4JUZKRM54SK5E4JA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: read tcp 188.166.225.170:44510->145.79.227.8:4331: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:12:04.270742155 (176h8m2.822212616s) [noise=true , address=145.79.227.4:4186 , piece_id=5PZIK3V72CUQQ6M2HJZFVM5OVX4HPHY7ZHJ2BAK5S3ZQOGM4J5OA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection
Nov 16 16:12:04.270762814 (176h8m2.822233275s) [noise=true , address=145.79.227.24:4581 , piece_id=XRCS52EJDB7HWD4RDZK3UNB6VKHOTWWID42RGBJZQHTGG4NDYIPA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: read tcp 188.166.225.170:44130->145.79.227.24:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:12:04.270781470 (176h8m2.822251921s) [noise=true , address=145.79.227.11:4732 , piece_id=FQGAVGZ5CID3LBET3CVK6FSCOP5OUYGS44XYWMHU2RLIT3MBN62A, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: read tcp 188.166.225.170:42420->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:12:04.270825082 (176h8m2.822295544s) [seg_pos={0 0}, upload=155107]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil>
Nov 16 16:12:05.262661603 (176h8m3.814132064s) [upload=155107]: Done uploading segments...
Nov 16 16:12:05.262728080 (176h8m3.814198541s) [upload=155107]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4)
Duration: 5.930726016s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [X...........................................................] duration:
3.472971734s trace:
https://jaeger.dev.storj.io/trace/1026d6559b04863a
FAIL (2025-11-16 16:26:31 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.472998454s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [.X..........................................................] duration:
3.531591257s trace:
https://jaeger.dev.storj.io/trace/1026d6559b04863a
FAIL (2025-11-16 16:26:00 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.531662052s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......................................X....................] duration:
5.851048926s trace:
https://jaeger.dev.storj.io/trace/70f81d2d9b012a3a
FAIL (2025-11-16 16:25:53 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 4)
======LOGS======
ddress=45.91.116.5:4297 , piece_id=OSMI3VGAKSCIZMDQ3Y2VZQBUOJKSXM6RWNWSAV57RDTL5TR3NIJA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.462247309 (176h7m35.013717781s) [noise=true , address=45.91.116.7:4011 , piece_id=ORJVRMFIOEXZ6OB6QFQDL764Q2FCWOJT63NBIRI6SOOS2FEIBCFQ, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.463420774 (176h7m35.014891235s) [noise=true , address=145.79.227.9:4367 , piece_id=LOFHPWCHNEZ4E4WANJM5BE2TIWVFH4LGEJJYK4ETB2LZWYCAVGZA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.464113112 (176h7m35.015583573s) [noise=true , address=145.79.227.22:4511 , piece_id=HES465EP3OPD5JBVGVYMGLUOHJKSSMJDXMJVWET5GHRPFZJFIFOA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.464764873 (176h7m35.016235334s) [noise=true , address=45.91.116.8:4048 , piece_id=ULVK52VWCW3KNEPWSDYGRHWHIJWEQU2XNFEPYJFXWLVJI6AGZQOA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.465452753 (176h7m35.016923214s) [noise=true , address=145.79.227.28:4725 , piece_id=CI2LSBGPW7RYH4RGWODGWLUTN353EODNV6IB2YQGHOZ3FPSEEK5A, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.466223270 (176h7m35.017693731s) [noise=true , address=145.79.227.16:4910 , piece_id=3U2SEVMPILIEBRHS3M4YT4Z4PSYAQAIC2SKGYMW4FNODACZLOQCQ, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.466955855 (176h7m35.018426306s) [noise=true , address=145.79.227.4:4186 , piece_id=5PZIK3V72CUQQ6M2HJZFVM5OVX4HPHY7ZHJ2BAK5S3ZQOGM4J5OA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.467675205 (176h7m35.019145666s) [noise=true , address=145.79.227.8:4331 , piece_id=4O27HU7AMR6G4Y43HJYN22UXVLETBDBE7JHL4JUZKRM54SK5E4JA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.468178092 (176h7m35.019648553s) [noise=true , address=145.79.227.8:4329 , piece_id=GO5IAWQYS3UARWYAQAEUIIS456DFWCFWRJQG2KOVVL5KPXWPK4AA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:39.111804678 (176h7m37.663275129s) [upload=155107]: Next returned nil segment
Nov 16 16:11:39.111825767 (176h7m37.663296218s) [upload=155107]: Waiting for error group managing segments...
Nov 16 16:11:39.896671313 (176h7m38.448141765s) [noise=true , address=45.91.116.12:4227 , piece_id=CQA2LHRSFOA7EIBIFVC75EWVHFZMWMY7ODMETEAJRRNYSQFY5TVQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil>
Nov 16 16:11:39.954993420 (176h7m38.506463881s) [noise=true , address=45.91.116.8:4048 , piece_id=ULVK52VWCW3KNEPWSDYGRHWHIJWEQU2XNFEPYJFXWLVJI6AGZQOA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil>
Nov 16 16:11:40.066014029 (176h7m38.61748449s) [noise=true , address=45.91.116.7:4011 , piece_id=ORJVRMFIOEXZ6OB6QFQDL764Q2FCWOJT63NBIRI6SOOS2FEIBCFQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil>
Nov 16 16:11:41.102688250 (176h7m39.654158711s) [noise=true , address=45.91.116.5:4297 , piece_id=OSMI3VGAKSCIZMDQ3Y2VZQBUOJKSXM6RWNWSAV57RDTL5TR3NIJA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil>
Nov 16 16:12:04.264842280 (176h8m2.816312741s) [noise=true , address=145.79.227.17:4972 , piece_id=NV6FSBQM6YUTKYFAJPQCURWCZBHJ3AKJOLLGCRYTJ3KQ54U4GOWQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection
Nov 16 16:12:04.265084832 (176h8m2.816555293s) [noise=true , address=145.79.227.16:4910 , piece_id=3U2SEVMPILIEBRHS3M4YT4Z4PSYAQAIC2SKGYMW4FNODACZLOQCQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection
Nov 16 16:12:04.270267942 (176h8m2.821738423s) [noise=true , address=145.79.227.8:4329 , piece_id=GO5IAWQYS3UARWYAQAEUIIS456DFWCFWRJQG2KOVVL5KPXWPK4AA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: expected piece hash; context deadline exceeded; EOF
Nov 16 16:12:04.270306515 (176h8m2.821776986s) [noise=true , address=145.79.227.9:4367 , piece_id=LOFHPWCHNEZ4E4WANJM5BE2TIWVFH4LGEJJYK4ETB2LZWYCAVGZA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection
Nov 16 16:12:04.270323538 (176h8m2.821794099s) [noise=true , address=145.79.227.28:4725 , piece_id=CI2LSBGPW7RYH4RGWODGWLUTN353EODNV6IB2YQGHOZ3FPSEEK5A, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection
Nov 16 16:12:04.270544138 (176h8m2.822014739s) [noise=true , address=145.79.227.14:4840 , piece_id=RSHGL65RI7HZGU3W3EROFL23BCK2EW3YN6CAD3NO5OVE5QM3QG3Q, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: read tcp 188.166.225.170:42148->145.79.227.14:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:12:04.270571570 (176h8m2.822042121s) [noise=true , address=145.79.227.31:4797 , piece_id=AWSPIPRKSBOKQFZRRB34RUPLYVJGKAOOJQ4U7RRATQJELM3OCFHA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection
Nov 16 16:12:04.270609783 (176h8m2.822080254s) [noise=true , address=145.79.227.22:4511 , piece_id=HES465EP3OPD5JBVGVYMGLUOHJKSSMJDXMJVWET5GHRPFZJFIFOA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: read tcp 188.166.225.170:51004->145.79.227.22:4511: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:12:04.270625552 (176h8m2.822096083s) [noise=true , address=145.79.227.8:4331 , piece_id=4O27HU7AMR6G4Y43HJYN22UXVLETBDBE7JHL4JUZKRM54SK5E4JA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: read tcp 188.166.225.170:44510->145.79.227.8:4331: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:12:04.270742155 (176h8m2.822212616s) [noise=true , address=145.79.227.4:4186 , piece_id=5PZIK3V72CUQQ6M2HJZFVM5OVX4HPHY7ZHJ2BAK5S3ZQOGM4J5OA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection
Nov 16 16:12:04.270762814 (176h8m2.822233275s) [noise=true , address=145.79.227.24:4581 , piece_id=XRCS52EJDB7HWD4RDZK3UNB6VKHOTWWID42RGBJZQHTGG4NDYIPA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: read tcp 188.166.225.170:44130->145.79.227.24:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:12:04.270781470 (176h8m2.822251921s) [noise=true , address=145.79.227.11:4732 , piece_id=FQGAVGZ5CID3LBET3CVK6FSCOP5OUYGS44XYWMHU2RLIT3MBN62A, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: read tcp 188.166.225.170:42420->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:12:04.270825082 (176h8m2.822295544s) [seg_pos={0 0}, upload=155107]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil>
Nov 16 16:12:05.262661603 (176h8m3.814132064s) [upload=155107]: Done uploading segments...
Nov 16 16:12:05.262728080 (176h8m3.814198541s) [upload=155107]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4)
Duration: 5.851075947s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [......................................X.....................] duration:
6.191300078s trace:
https://jaeger.dev.storj.io/trace/3e4560ab56ee4602
FAIL (2025-11-16 16:25:46 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.191328392s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [...........................X................................] duration:
5.221888032s trace:
https://jaeger.dev.storj.io/trace/7dbd01be7ae5d01e
FAIL (2025-11-16 16:25:45 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 24)
======LOGS======
OHA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.079132696 (176h0m30.630603157s) [noise=true , address=81.171.14.70:4040 , piece_id=YG6ZBK6QTKUS3FCFD2EWU7HTFLOEEVG57SIER44E67QDP5HYCHOA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.079319683 (176h0m30.630790144s) [noise=true , address=185.167.204.205:4015 , piece_id=6UIFX7GXE2SZ465VAP56YQILR3A3OHDRKIS345EE6GBDG77KJNZQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.079646465 (176h0m30.631116926s) [noise=true , address=185.167.204.206:4062 , piece_id=PAJX3ZCWM5YBZA7YDBFYXCET5CKWBOFJ47IUWK4CYPZSWK6HYQVQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.080002833 (176h0m30.631473294s) [noise=true , address=81.171.14.70:4031 , piece_id=54T5MOZZ4YJKQFUDOCCFF2SAE7XS2N77H2AOHCBECAWE5OBUXTZQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.080062066 (176h0m30.631532527s) [noise=true , address=185.167.204.206:4067 , piece_id=2BDMQZLCV4UWWP5HB6GF6NCL5UOGNU7VNYGJIIJCKQHRTX5OPGPQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.080438673 (176h0m30.631909134s) [noise=true , address=185.167.204.205:4034 , piece_id=CZFPF35OQ3U4IH46KILFPFMVY6OC36ZUGZ7EYYBOTB3I4TXMPERA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.080675103 (176h0m30.632145564s) [noise=true , address=185.167.204.205:4014 , piece_id=57C2VUYOEULOTXUQHG7A2XOWZLXYFIEYYRYW5ARX5PKTGWKIEOAQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.081315833 (176h0m30.632786294s) [noise=true , address=81.171.14.70:4039 , piece_id=NMZ5357FVIKD2F3GCKY6YNNV73BAID2YBLVNSSM6VD5PHGLBHEIA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.082053738 (176h0m30.633524189s) [noise=true , address=185.167.204.205:4001 , piece_id=TBPYFHMXWO53LJQQDUA3I2MD2WP6BHQXAH3KIF6XXMWK27HXLCHA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.082605429 (176h0m30.63407589s) [noise=true , address=81.171.14.70:4034 , piece_id=NXBDCCCEWSZ2TMSET6OXBNQV6Q5ESXIQJUFK4WJJ3UV7FJTTAUXQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:33.280203414 (176h0m31.831673875s) [upload=155002]: Next returned nil segment
Nov 16 16:04:33.280222810 (176h0m31.831693271s) [upload=155002]: Waiting for error group managing segments...
Nov 16 16:04:33.838289781 (176h0m32.389760242s) [noise=true , address=185.167.204.205:4013 , piece_id=KF32EXUYGKQEDBPRSVAMNZV3ZMBF6DLT3U32BKX5LXLRBXBVTOQA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.848846382 (176h0m32.400316833s) [noise=true , address=185.167.204.206:4039 , piece_id=6HTUHESIRTSKR5EDMWZ2K56OKE5Y22QQM5ARYTUXR7KMKWIOS7GQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.862791868 (176h0m32.414262329s) [noise=true , address=185.167.204.205:4021 , piece_id=MV3TV4EI3HN3J5NY5XYVKT46QJTJEC5PITXERSSZMS2B7W3DSDTA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.879947135 (176h0m32.431417596s) [noise=true , address=185.167.204.205:4001 , piece_id=TBPYFHMXWO53LJQQDUA3I2MD2WP6BHQXAH3KIF6XXMWK27HXLCHA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.886853657 (176h0m32.438324118s) [noise=true , address=185.167.204.206:4067 , piece_id=2BDMQZLCV4UWWP5HB6GF6NCL5UOGNU7VNYGJIIJCKQHRTX5OPGPQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.918801783 (176h0m32.470272324s) [noise=true , address=185.167.204.206:4051 , piece_id=AU2S6M3E2CQEMAS7LKZJIVWITZUHIHCMWQ2QWT7ILGBRTAUCUB7Q, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.929982083 (176h0m32.481452574s) [noise=true , address=185.167.204.205:4015 , piece_id=6UIFX7GXE2SZ465VAP56YQILR3A3OHDRKIS345EE6GBDG77KJNZQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.935247409 (176h0m32.48671786s) [noise=true , address=185.167.204.206:4038 , piece_id=FBNPAJOOXEAT3KLKQIYOWHGHRGAWJABV4LAA3UYX6SGJB3U2LD7A, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.969630813 (176h0m32.521101264s) [noise=true , address=185.167.204.206:4055 , piece_id=B2TTK2ZLIRM3YIKP5HQ3WOL26KBHSXWWX5ZXXCMWU34AH7CHOOCQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.999043600 (176h0m32.550514061s) [noise=true , address=185.167.204.205:4034 , piece_id=CZFPF35OQ3U4IH46KILFPFMVY6OC36ZUGZ7EYYBOTB3I4TXMPERA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.022951616 (176h0m32.574422077s) [noise=true , address=185.167.204.205:4014 , piece_id=57C2VUYOEULOTXUQHG7A2XOWZLXYFIEYYRYW5ARX5PKTGWKIEOAQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.064488381 (176h0m32.615958832s) [noise=true , address=185.167.204.206:4058 , piece_id=POK7HBLE5AT2STCSL2UJ5R67LNVK3PN53NIRIBX4M7LJYC4C2CHQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.122257987 (176h0m32.673728468s) [noise=true , address=185.167.204.205:4029 , piece_id=VF7BTI4K75HTRVJOHIKMDJN3UBHZ7KCQBT3H44XB5G7ZVFD4HXJQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.136282884 (176h0m32.687753345s) [noise=true , address=185.167.204.206:4041 , piece_id=RTQSCXR54KTKBYS6AHEKPO7ZPUHVQQNUNQJ6JI4JLTGKOLUZH3AQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.136821679 (176h0m32.68829215s) [noise=true , address=185.167.204.206:4062 , piece_id=PAJX3ZCWM5YBZA7YDBFYXCET5CKWBOFJ47IUWK4CYPZSWK6HYQVQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.156096042 (176h0m32.707566493s) [noise=true , address=185.167.204.206:4066 , piece_id=CPTBHDHCT6R2M3UA6X2ZWMPJUKAUVV4I2CIATZ2ZOIWH63Q3UT4Q, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.540981809 (176h0m33.0924523s) [noise=true , address=89.149.201.147:4012 , piece_id=NEIYQHVVWRUVJFF5UD3RZSRHUSNSKXBCNGTMIGOH5MDH2AOZISCA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:21.258617296 (176h1m19.810087757s) [noise=true , address=81.171.14.70:4034 , piece_id=NXBDCCCEWSZ2TMSET6OXBNQV6Q5ESXIQJUFK4WJJ3UV7FJTTAUXQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:22.223432573 (176h1m20.774903034s) [noise=true , address=81.171.14.70:4048 , piece_id=5WUEBVU7UFKHYZ23BPIQUMUUBH3ZXXQWBLE7MALJGECCDTTEK5EA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:22.238494073 (176h1m20.789964534s) [noise=true , address=81.171.14.70:4033 , piece_id=LUBQIVFAU7J72K5XDDS5S2ZXQUGFA5AINL4R5FV64KY24VS5KV5Q, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:22.350384206 (176h1m20.901854667s) [noise=true , address=81.171.14.70:4039 , piece_id=NMZ5357FVIKD2F3GCKY6YNNV73BAID2YBLVNSSM6VD5PHGLBHEIA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:23.704484814 (176h1m22.255955275s) [noise=true , address=81.171.14.70:4037 , piece_id=3IJ7J7RBMOORW4YPKIYJ3C6ZQRYBH7WQMLEKEQYG5UJ4TGKE2W2A, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:24.501497158 (176h1m23.052967619s) [noise=true , address=81.171.14.70:4031 , piece_id=54T5MOZZ4YJKQFUDOCCFF2SAE7XS2N77H2AOHCBECAWE5OBUXTZQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:29.810461698 (176h1m28.361932159s) [noise=true , address=81.171.14.70:4025 , piece_id=3HAT6FJSGV6VOHJBMO6L6L7677BOYVCDV3SR26LJZ6HZOZWJJ64A, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:31.413018058 (176h1m29.964488509s) [noise=true , address=81.171.14.70:4040 , piece_id=YG6ZBK6QTKUS3FCFD2EWU7HTFLOEEVG57SIER44E67QDP5HYCHOA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:1gannW154cmVP6N5BowytKfDorZDRwx2M5pi6akqjZFU38PGjV, address:81.171.14.70:4040): protocol: write tcp 188.166.225.170:59244->81.171.14.70:4040: use of closed network connection; write tcp 188.166.225.170:59244->81.171.14.70:4040: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59244->81.171.14.70:4040: use of closed network connection
Nov 16 16:05:31.413111366 (176h1m29.964581837s) [noise=true , address=81.171.14.70:4035 , piece_id=RGMXRH2Y5XBAFISK5LNAF547SAORD6XSWFUKYWOPJINXXODI5KVQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:12eLYJKne3yP1fZ1oPDLM7Xqs3GQ3ju4ZQedAiojXdVkcxNpzeA, address:81.171.14.70:4035): protocol: read tcp 188.166.225.170:55142->81.171.14.70:4035: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:05:31.413619965 (176h1m29.965090436s) [noise=true , address=81.171.14.70:4030 , piece_id=ZE5OAUTWLWNSMWSFGPEJTCXEN6L3FLWL34QBERTS3JZE7X2I2OHA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:12ZoRwa8wLw66Pjumy1ybp6wrZ79RrSU55uYZpwaY3xi95iSYde, address:81.171.14.70:4030): protocol: read tcp 188.166.225.170:40822->81.171.14.70:4030: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:05:31.414846280 (176h1m29.966316741s) [noise=true , address=81.171.14.70:4029 , piece_id=SIXL2XSCF7IO2BPOUKGH5VVXI6M7O4KJVZARYIRELR2GM4T4RABA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:18yEfU2X4jSrom1JDtQ22cUEvomcg91ohvay2Nsr1Vr8juo9to, address:81.171.14.70:4029): protocol: read tcp 188.166.225.170:56584->81.171.14.70:4029: use of closed network connection; conn closed: op
Nov 16 16:05:31.414946261 (176h1m29.966416722s) [seg_pos={0 0}, upload=155002]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil>
Nov 16 16:05:31.591745995 (176h1m30.143216456s) [upload=155002]: Done uploading segments...
Nov 16 16:05:31.591788596 (176h1m30.143259057s) [upload=155002]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24)
Duration: 5.221920593s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [..X.........................................................] duration:
3.462259606s trace:
https://jaeger.dev.storj.io/trace/1026d6559b04863a
FAIL (2025-11-16 16:25:42 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.462276908s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......................................X....................] duration:
6.070828631s trace:
https://jaeger.dev.storj.io/trace/3e4560ab56ee4602
FAIL (2025-11-16 16:25:23 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.070884147s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [........................................X...................] duration:
6.029206703s trace:
https://jaeger.dev.storj.io/trace/70f81d2d9b012a3a
FAIL (2025-11-16 16:25:18 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 4)
======LOGS======
ddress=45.91.116.5:4297 , piece_id=OSMI3VGAKSCIZMDQ3Y2VZQBUOJKSXM6RWNWSAV57RDTL5TR3NIJA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.462247309 (176h7m35.013717781s) [noise=true , address=45.91.116.7:4011 , piece_id=ORJVRMFIOEXZ6OB6QFQDL764Q2FCWOJT63NBIRI6SOOS2FEIBCFQ, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.463420774 (176h7m35.014891235s) [noise=true , address=145.79.227.9:4367 , piece_id=LOFHPWCHNEZ4E4WANJM5BE2TIWVFH4LGEJJYK4ETB2LZWYCAVGZA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.464113112 (176h7m35.015583573s) [noise=true , address=145.79.227.22:4511 , piece_id=HES465EP3OPD5JBVGVYMGLUOHJKSSMJDXMJVWET5GHRPFZJFIFOA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.464764873 (176h7m35.016235334s) [noise=true , address=45.91.116.8:4048 , piece_id=ULVK52VWCW3KNEPWSDYGRHWHIJWEQU2XNFEPYJFXWLVJI6AGZQOA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.465452753 (176h7m35.016923214s) [noise=true , address=145.79.227.28:4725 , piece_id=CI2LSBGPW7RYH4RGWODGWLUTN353EODNV6IB2YQGHOZ3FPSEEK5A, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.466223270 (176h7m35.017693731s) [noise=true , address=145.79.227.16:4910 , piece_id=3U2SEVMPILIEBRHS3M4YT4Z4PSYAQAIC2SKGYMW4FNODACZLOQCQ, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.466955855 (176h7m35.018426306s) [noise=true , address=145.79.227.4:4186 , piece_id=5PZIK3V72CUQQ6M2HJZFVM5OVX4HPHY7ZHJ2BAK5S3ZQOGM4J5OA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.467675205 (176h7m35.019145666s) [noise=true , address=145.79.227.8:4331 , piece_id=4O27HU7AMR6G4Y43HJYN22UXVLETBDBE7JHL4JUZKRM54SK5E4JA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.468178092 (176h7m35.019648553s) [noise=true , address=145.79.227.8:4329 , piece_id=GO5IAWQYS3UARWYAQAEUIIS456DFWCFWRJQG2KOVVL5KPXWPK4AA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:39.111804678 (176h7m37.663275129s) [upload=155107]: Next returned nil segment
Nov 16 16:11:39.111825767 (176h7m37.663296218s) [upload=155107]: Waiting for error group managing segments...
Nov 16 16:11:39.896671313 (176h7m38.448141765s) [noise=true , address=45.91.116.12:4227 , piece_id=CQA2LHRSFOA7EIBIFVC75EWVHFZMWMY7ODMETEAJRRNYSQFY5TVQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil>
Nov 16 16:11:39.954993420 (176h7m38.506463881s) [noise=true , address=45.91.116.8:4048 , piece_id=ULVK52VWCW3KNEPWSDYGRHWHIJWEQU2XNFEPYJFXWLVJI6AGZQOA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil>
Nov 16 16:11:40.066014029 (176h7m38.61748449s) [noise=true , address=45.91.116.7:4011 , piece_id=ORJVRMFIOEXZ6OB6QFQDL764Q2FCWOJT63NBIRI6SOOS2FEIBCFQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil>
Nov 16 16:11:41.102688250 (176h7m39.654158711s) [noise=true , address=45.91.116.5:4297 , piece_id=OSMI3VGAKSCIZMDQ3Y2VZQBUOJKSXM6RWNWSAV57RDTL5TR3NIJA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil>
Nov 16 16:12:04.264842280 (176h8m2.816312741s) [noise=true , address=145.79.227.17:4972 , piece_id=NV6FSBQM6YUTKYFAJPQCURWCZBHJ3AKJOLLGCRYTJ3KQ54U4GOWQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection
Nov 16 16:12:04.265084832 (176h8m2.816555293s) [noise=true , address=145.79.227.16:4910 , piece_id=3U2SEVMPILIEBRHS3M4YT4Z4PSYAQAIC2SKGYMW4FNODACZLOQCQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection
Nov 16 16:12:04.270267942 (176h8m2.821738423s) [noise=true , address=145.79.227.8:4329 , piece_id=GO5IAWQYS3UARWYAQAEUIIS456DFWCFWRJQG2KOVVL5KPXWPK4AA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: expected piece hash; context deadline exceeded; EOF
Nov 16 16:12:04.270306515 (176h8m2.821776986s) [noise=true , address=145.79.227.9:4367 , piece_id=LOFHPWCHNEZ4E4WANJM5BE2TIWVFH4LGEJJYK4ETB2LZWYCAVGZA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection
Nov 16 16:12:04.270323538 (176h8m2.821794099s) [noise=true , address=145.79.227.28:4725 , piece_id=CI2LSBGPW7RYH4RGWODGWLUTN353EODNV6IB2YQGHOZ3FPSEEK5A, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection
Nov 16 16:12:04.270544138 (176h8m2.822014739s) [noise=true , address=145.79.227.14:4840 , piece_id=RSHGL65RI7HZGU3W3EROFL23BCK2EW3YN6CAD3NO5OVE5QM3QG3Q, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: read tcp 188.166.225.170:42148->145.79.227.14:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:12:04.270571570 (176h8m2.822042121s) [noise=true , address=145.79.227.31:4797 , piece_id=AWSPIPRKSBOKQFZRRB34RUPLYVJGKAOOJQ4U7RRATQJELM3OCFHA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection
Nov 16 16:12:04.270609783 (176h8m2.822080254s) [noise=true , address=145.79.227.22:4511 , piece_id=HES465EP3OPD5JBVGVYMGLUOHJKSSMJDXMJVWET5GHRPFZJFIFOA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: read tcp 188.166.225.170:51004->145.79.227.22:4511: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:12:04.270625552 (176h8m2.822096083s) [noise=true , address=145.79.227.8:4331 , piece_id=4O27HU7AMR6G4Y43HJYN22UXVLETBDBE7JHL4JUZKRM54SK5E4JA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: read tcp 188.166.225.170:44510->145.79.227.8:4331: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:12:04.270742155 (176h8m2.822212616s) [noise=true , address=145.79.227.4:4186 , piece_id=5PZIK3V72CUQQ6M2HJZFVM5OVX4HPHY7ZHJ2BAK5S3ZQOGM4J5OA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection
Nov 16 16:12:04.270762814 (176h8m2.822233275s) [noise=true , address=145.79.227.24:4581 , piece_id=XRCS52EJDB7HWD4RDZK3UNB6VKHOTWWID42RGBJZQHTGG4NDYIPA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: read tcp 188.166.225.170:44130->145.79.227.24:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:12:04.270781470 (176h8m2.822251921s) [noise=true , address=145.79.227.11:4732 , piece_id=FQGAVGZ5CID3LBET3CVK6FSCOP5OUYGS44XYWMHU2RLIT3MBN62A, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: read tcp 188.166.225.170:42420->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:12:04.270825082 (176h8m2.822295544s) [seg_pos={0 0}, upload=155107]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil>
Nov 16 16:12:05.262661603 (176h8m3.814132064s) [upload=155107]: Done uploading segments...
Nov 16 16:12:05.262728080 (176h8m3.814198541s) [upload=155107]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4)
Duration: 6.029223225s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [............................X...............................] duration:
5.732878639s trace:
https://jaeger.dev.storj.io/trace/7dbd01be7ae5d01e
FAIL (2025-11-16 16:25:18 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 24)
======LOGS======
OHA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.079132696 (176h0m30.630603157s) [noise=true , address=81.171.14.70:4040 , piece_id=YG6ZBK6QTKUS3FCFD2EWU7HTFLOEEVG57SIER44E67QDP5HYCHOA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.079319683 (176h0m30.630790144s) [noise=true , address=185.167.204.205:4015 , piece_id=6UIFX7GXE2SZ465VAP56YQILR3A3OHDRKIS345EE6GBDG77KJNZQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.079646465 (176h0m30.631116926s) [noise=true , address=185.167.204.206:4062 , piece_id=PAJX3ZCWM5YBZA7YDBFYXCET5CKWBOFJ47IUWK4CYPZSWK6HYQVQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.080002833 (176h0m30.631473294s) [noise=true , address=81.171.14.70:4031 , piece_id=54T5MOZZ4YJKQFUDOCCFF2SAE7XS2N77H2AOHCBECAWE5OBUXTZQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.080062066 (176h0m30.631532527s) [noise=true , address=185.167.204.206:4067 , piece_id=2BDMQZLCV4UWWP5HB6GF6NCL5UOGNU7VNYGJIIJCKQHRTX5OPGPQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.080438673 (176h0m30.631909134s) [noise=true , address=185.167.204.205:4034 , piece_id=CZFPF35OQ3U4IH46KILFPFMVY6OC36ZUGZ7EYYBOTB3I4TXMPERA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.080675103 (176h0m30.632145564s) [noise=true , address=185.167.204.205:4014 , piece_id=57C2VUYOEULOTXUQHG7A2XOWZLXYFIEYYRYW5ARX5PKTGWKIEOAQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.081315833 (176h0m30.632786294s) [noise=true , address=81.171.14.70:4039 , piece_id=NMZ5357FVIKD2F3GCKY6YNNV73BAID2YBLVNSSM6VD5PHGLBHEIA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.082053738 (176h0m30.633524189s) [noise=true , address=185.167.204.205:4001 , piece_id=TBPYFHMXWO53LJQQDUA3I2MD2WP6BHQXAH3KIF6XXMWK27HXLCHA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.082605429 (176h0m30.63407589s) [noise=true , address=81.171.14.70:4034 , piece_id=NXBDCCCEWSZ2TMSET6OXBNQV6Q5ESXIQJUFK4WJJ3UV7FJTTAUXQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:33.280203414 (176h0m31.831673875s) [upload=155002]: Next returned nil segment
Nov 16 16:04:33.280222810 (176h0m31.831693271s) [upload=155002]: Waiting for error group managing segments...
Nov 16 16:04:33.838289781 (176h0m32.389760242s) [noise=true , address=185.167.204.205:4013 , piece_id=KF32EXUYGKQEDBPRSVAMNZV3ZMBF6DLT3U32BKX5LXLRBXBVTOQA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.848846382 (176h0m32.400316833s) [noise=true , address=185.167.204.206:4039 , piece_id=6HTUHESIRTSKR5EDMWZ2K56OKE5Y22QQM5ARYTUXR7KMKWIOS7GQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.862791868 (176h0m32.414262329s) [noise=true , address=185.167.204.205:4021 , piece_id=MV3TV4EI3HN3J5NY5XYVKT46QJTJEC5PITXERSSZMS2B7W3DSDTA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.879947135 (176h0m32.431417596s) [noise=true , address=185.167.204.205:4001 , piece_id=TBPYFHMXWO53LJQQDUA3I2MD2WP6BHQXAH3KIF6XXMWK27HXLCHA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.886853657 (176h0m32.438324118s) [noise=true , address=185.167.204.206:4067 , piece_id=2BDMQZLCV4UWWP5HB6GF6NCL5UOGNU7VNYGJIIJCKQHRTX5OPGPQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.918801783 (176h0m32.470272324s) [noise=true , address=185.167.204.206:4051 , piece_id=AU2S6M3E2CQEMAS7LKZJIVWITZUHIHCMWQ2QWT7ILGBRTAUCUB7Q, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.929982083 (176h0m32.481452574s) [noise=true , address=185.167.204.205:4015 , piece_id=6UIFX7GXE2SZ465VAP56YQILR3A3OHDRKIS345EE6GBDG77KJNZQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.935247409 (176h0m32.48671786s) [noise=true , address=185.167.204.206:4038 , piece_id=FBNPAJOOXEAT3KLKQIYOWHGHRGAWJABV4LAA3UYX6SGJB3U2LD7A, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.969630813 (176h0m32.521101264s) [noise=true , address=185.167.204.206:4055 , piece_id=B2TTK2ZLIRM3YIKP5HQ3WOL26KBHSXWWX5ZXXCMWU34AH7CHOOCQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.999043600 (176h0m32.550514061s) [noise=true , address=185.167.204.205:4034 , piece_id=CZFPF35OQ3U4IH46KILFPFMVY6OC36ZUGZ7EYYBOTB3I4TXMPERA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.022951616 (176h0m32.574422077s) [noise=true , address=185.167.204.205:4014 , piece_id=57C2VUYOEULOTXUQHG7A2XOWZLXYFIEYYRYW5ARX5PKTGWKIEOAQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.064488381 (176h0m32.615958832s) [noise=true , address=185.167.204.206:4058 , piece_id=POK7HBLE5AT2STCSL2UJ5R67LNVK3PN53NIRIBX4M7LJYC4C2CHQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.122257987 (176h0m32.673728468s) [noise=true , address=185.167.204.205:4029 , piece_id=VF7BTI4K75HTRVJOHIKMDJN3UBHZ7KCQBT3H44XB5G7ZVFD4HXJQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.136282884 (176h0m32.687753345s) [noise=true , address=185.167.204.206:4041 , piece_id=RTQSCXR54KTKBYS6AHEKPO7ZPUHVQQNUNQJ6JI4JLTGKOLUZH3AQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.136821679 (176h0m32.68829215s) [noise=true , address=185.167.204.206:4062 , piece_id=PAJX3ZCWM5YBZA7YDBFYXCET5CKWBOFJ47IUWK4CYPZSWK6HYQVQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.156096042 (176h0m32.707566493s) [noise=true , address=185.167.204.206:4066 , piece_id=CPTBHDHCT6R2M3UA6X2ZWMPJUKAUVV4I2CIATZ2ZOIWH63Q3UT4Q, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.540981809 (176h0m33.0924523s) [noise=true , address=89.149.201.147:4012 , piece_id=NEIYQHVVWRUVJFF5UD3RZSRHUSNSKXBCNGTMIGOH5MDH2AOZISCA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:21.258617296 (176h1m19.810087757s) [noise=true , address=81.171.14.70:4034 , piece_id=NXBDCCCEWSZ2TMSET6OXBNQV6Q5ESXIQJUFK4WJJ3UV7FJTTAUXQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:22.223432573 (176h1m20.774903034s) [noise=true , address=81.171.14.70:4048 , piece_id=5WUEBVU7UFKHYZ23BPIQUMUUBH3ZXXQWBLE7MALJGECCDTTEK5EA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:22.238494073 (176h1m20.789964534s) [noise=true , address=81.171.14.70:4033 , piece_id=LUBQIVFAU7J72K5XDDS5S2ZXQUGFA5AINL4R5FV64KY24VS5KV5Q, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:22.350384206 (176h1m20.901854667s) [noise=true , address=81.171.14.70:4039 , piece_id=NMZ5357FVIKD2F3GCKY6YNNV73BAID2YBLVNSSM6VD5PHGLBHEIA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:23.704484814 (176h1m22.255955275s) [noise=true , address=81.171.14.70:4037 , piece_id=3IJ7J7RBMOORW4YPKIYJ3C6ZQRYBH7WQMLEKEQYG5UJ4TGKE2W2A, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:24.501497158 (176h1m23.052967619s) [noise=true , address=81.171.14.70:4031 , piece_id=54T5MOZZ4YJKQFUDOCCFF2SAE7XS2N77H2AOHCBECAWE5OBUXTZQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:29.810461698 (176h1m28.361932159s) [noise=true , address=81.171.14.70:4025 , piece_id=3HAT6FJSGV6VOHJBMO6L6L7677BOYVCDV3SR26LJZ6HZOZWJJ64A, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:31.413018058 (176h1m29.964488509s) [noise=true , address=81.171.14.70:4040 , piece_id=YG6ZBK6QTKUS3FCFD2EWU7HTFLOEEVG57SIER44E67QDP5HYCHOA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:1gannW154cmVP6N5BowytKfDorZDRwx2M5pi6akqjZFU38PGjV, address:81.171.14.70:4040): protocol: write tcp 188.166.225.170:59244->81.171.14.70:4040: use of closed network connection; write tcp 188.166.225.170:59244->81.171.14.70:4040: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59244->81.171.14.70:4040: use of closed network connection
Nov 16 16:05:31.413111366 (176h1m29.964581837s) [noise=true , address=81.171.14.70:4035 , piece_id=RGMXRH2Y5XBAFISK5LNAF547SAORD6XSWFUKYWOPJINXXODI5KVQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:12eLYJKne3yP1fZ1oPDLM7Xqs3GQ3ju4ZQedAiojXdVkcxNpzeA, address:81.171.14.70:4035): protocol: read tcp 188.166.225.170:55142->81.171.14.70:4035: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:05:31.413619965 (176h1m29.965090436s) [noise=true , address=81.171.14.70:4030 , piece_id=ZE5OAUTWLWNSMWSFGPEJTCXEN6L3FLWL34QBERTS3JZE7X2I2OHA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:12ZoRwa8wLw66Pjumy1ybp6wrZ79RrSU55uYZpwaY3xi95iSYde, address:81.171.14.70:4030): protocol: read tcp 188.166.225.170:40822->81.171.14.70:4030: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:05:31.414846280 (176h1m29.966316741s) [noise=true , address=81.171.14.70:4029 , piece_id=SIXL2XSCF7IO2BPOUKGH5VVXI6M7O4KJVZARYIRELR2GM4T4RABA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:18yEfU2X4jSrom1JDtQ22cUEvomcg91ohvay2Nsr1Vr8juo9to, address:81.171.14.70:4029): protocol: read tcp 188.166.225.170:56584->81.171.14.70:4029: use of closed network connection; conn closed: op
Nov 16 16:05:31.414946261 (176h1m29.966416722s) [seg_pos={0 0}, upload=155002]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil>
Nov 16 16:05:31.591745995 (176h1m30.143216456s) [upload=155002]: Done uploading segments...
Nov 16 16:05:31.591788596 (176h1m30.143259057s) [upload=155002]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24)
Duration: 5.732897025s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................................X..................] duration:
6.025483203s trace:
https://jaeger.dev.storj.io/trace/70f81d2d9b012a3a
FAIL (2025-11-16 16:25:00 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 4)
======LOGS======
ddress=45.91.116.5:4297 , piece_id=OSMI3VGAKSCIZMDQ3Y2VZQBUOJKSXM6RWNWSAV57RDTL5TR3NIJA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.462247309 (176h7m35.013717781s) [noise=true , address=45.91.116.7:4011 , piece_id=ORJVRMFIOEXZ6OB6QFQDL764Q2FCWOJT63NBIRI6SOOS2FEIBCFQ, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.463420774 (176h7m35.014891235s) [noise=true , address=145.79.227.9:4367 , piece_id=LOFHPWCHNEZ4E4WANJM5BE2TIWVFH4LGEJJYK4ETB2LZWYCAVGZA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.464113112 (176h7m35.015583573s) [noise=true , address=145.79.227.22:4511 , piece_id=HES465EP3OPD5JBVGVYMGLUOHJKSSMJDXMJVWET5GHRPFZJFIFOA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.464764873 (176h7m35.016235334s) [noise=true , address=45.91.116.8:4048 , piece_id=ULVK52VWCW3KNEPWSDYGRHWHIJWEQU2XNFEPYJFXWLVJI6AGZQOA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.465452753 (176h7m35.016923214s) [noise=true , address=145.79.227.28:4725 , piece_id=CI2LSBGPW7RYH4RGWODGWLUTN353EODNV6IB2YQGHOZ3FPSEEK5A, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.466223270 (176h7m35.017693731s) [noise=true , address=145.79.227.16:4910 , piece_id=3U2SEVMPILIEBRHS3M4YT4Z4PSYAQAIC2SKGYMW4FNODACZLOQCQ, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.466955855 (176h7m35.018426306s) [noise=true , address=145.79.227.4:4186 , piece_id=5PZIK3V72CUQQ6M2HJZFVM5OVX4HPHY7ZHJ2BAK5S3ZQOGM4J5OA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.467675205 (176h7m35.019145666s) [noise=true , address=145.79.227.8:4331 , piece_id=4O27HU7AMR6G4Y43HJYN22UXVLETBDBE7JHL4JUZKRM54SK5E4JA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.468178092 (176h7m35.019648553s) [noise=true , address=145.79.227.8:4329 , piece_id=GO5IAWQYS3UARWYAQAEUIIS456DFWCFWRJQG2KOVVL5KPXWPK4AA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:39.111804678 (176h7m37.663275129s) [upload=155107]: Next returned nil segment
Nov 16 16:11:39.111825767 (176h7m37.663296218s) [upload=155107]: Waiting for error group managing segments...
Nov 16 16:11:39.896671313 (176h7m38.448141765s) [noise=true , address=45.91.116.12:4227 , piece_id=CQA2LHRSFOA7EIBIFVC75EWVHFZMWMY7ODMETEAJRRNYSQFY5TVQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil>
Nov 16 16:11:39.954993420 (176h7m38.506463881s) [noise=true , address=45.91.116.8:4048 , piece_id=ULVK52VWCW3KNEPWSDYGRHWHIJWEQU2XNFEPYJFXWLVJI6AGZQOA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil>
Nov 16 16:11:40.066014029 (176h7m38.61748449s) [noise=true , address=45.91.116.7:4011 , piece_id=ORJVRMFIOEXZ6OB6QFQDL764Q2FCWOJT63NBIRI6SOOS2FEIBCFQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil>
Nov 16 16:11:41.102688250 (176h7m39.654158711s) [noise=true , address=45.91.116.5:4297 , piece_id=OSMI3VGAKSCIZMDQ3Y2VZQBUOJKSXM6RWNWSAV57RDTL5TR3NIJA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil>
Nov 16 16:12:04.264842280 (176h8m2.816312741s) [noise=true , address=145.79.227.17:4972 , piece_id=NV6FSBQM6YUTKYFAJPQCURWCZBHJ3AKJOLLGCRYTJ3KQ54U4GOWQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection
Nov 16 16:12:04.265084832 (176h8m2.816555293s) [noise=true , address=145.79.227.16:4910 , piece_id=3U2SEVMPILIEBRHS3M4YT4Z4PSYAQAIC2SKGYMW4FNODACZLOQCQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection
Nov 16 16:12:04.270267942 (176h8m2.821738423s) [noise=true , address=145.79.227.8:4329 , piece_id=GO5IAWQYS3UARWYAQAEUIIS456DFWCFWRJQG2KOVVL5KPXWPK4AA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: expected piece hash; context deadline exceeded; EOF
Nov 16 16:12:04.270306515 (176h8m2.821776986s) [noise=true , address=145.79.227.9:4367 , piece_id=LOFHPWCHNEZ4E4WANJM5BE2TIWVFH4LGEJJYK4ETB2LZWYCAVGZA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection
Nov 16 16:12:04.270323538 (176h8m2.821794099s) [noise=true , address=145.79.227.28:4725 , piece_id=CI2LSBGPW7RYH4RGWODGWLUTN353EODNV6IB2YQGHOZ3FPSEEK5A, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection
Nov 16 16:12:04.270544138 (176h8m2.822014739s) [noise=true , address=145.79.227.14:4840 , piece_id=RSHGL65RI7HZGU3W3EROFL23BCK2EW3YN6CAD3NO5OVE5QM3QG3Q, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: read tcp 188.166.225.170:42148->145.79.227.14:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:12:04.270571570 (176h8m2.822042121s) [noise=true , address=145.79.227.31:4797 , piece_id=AWSPIPRKSBOKQFZRRB34RUPLYVJGKAOOJQ4U7RRATQJELM3OCFHA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection
Nov 16 16:12:04.270609783 (176h8m2.822080254s) [noise=true , address=145.79.227.22:4511 , piece_id=HES465EP3OPD5JBVGVYMGLUOHJKSSMJDXMJVWET5GHRPFZJFIFOA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: read tcp 188.166.225.170:51004->145.79.227.22:4511: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:12:04.270625552 (176h8m2.822096083s) [noise=true , address=145.79.227.8:4331 , piece_id=4O27HU7AMR6G4Y43HJYN22UXVLETBDBE7JHL4JUZKRM54SK5E4JA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: read tcp 188.166.225.170:44510->145.79.227.8:4331: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:12:04.270742155 (176h8m2.822212616s) [noise=true , address=145.79.227.4:4186 , piece_id=5PZIK3V72CUQQ6M2HJZFVM5OVX4HPHY7ZHJ2BAK5S3ZQOGM4J5OA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection
Nov 16 16:12:04.270762814 (176h8m2.822233275s) [noise=true , address=145.79.227.24:4581 , piece_id=XRCS52EJDB7HWD4RDZK3UNB6VKHOTWWID42RGBJZQHTGG4NDYIPA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: read tcp 188.166.225.170:44130->145.79.227.24:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:12:04.270781470 (176h8m2.822251921s) [noise=true , address=145.79.227.11:4732 , piece_id=FQGAVGZ5CID3LBET3CVK6FSCOP5OUYGS44XYWMHU2RLIT3MBN62A, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: read tcp 188.166.225.170:42420->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:12:04.270825082 (176h8m2.822295544s) [seg_pos={0 0}, upload=155107]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil>
Nov 16 16:12:05.262661603 (176h8m3.814132064s) [upload=155107]: Done uploading segments...
Nov 16 16:12:05.262728080 (176h8m3.814198541s) [upload=155107]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4)
Duration: 6.025514354s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [...X........................................................] duration:
3.604960397s trace:
https://jaeger.dev.storj.io/trace/1026d6559b04863a
FAIL (2025-11-16 16:24:53 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.605016202s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [.............................X..............................] duration:
5.717441216s trace:
https://jaeger.dev.storj.io/trace/7dbd01be7ae5d01e
FAIL (2025-11-16 16:24:51 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 24)
======LOGS======
OHA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.079132696 (176h0m30.630603157s) [noise=true , address=81.171.14.70:4040 , piece_id=YG6ZBK6QTKUS3FCFD2EWU7HTFLOEEVG57SIER44E67QDP5HYCHOA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.079319683 (176h0m30.630790144s) [noise=true , address=185.167.204.205:4015 , piece_id=6UIFX7GXE2SZ465VAP56YQILR3A3OHDRKIS345EE6GBDG77KJNZQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.079646465 (176h0m30.631116926s) [noise=true , address=185.167.204.206:4062 , piece_id=PAJX3ZCWM5YBZA7YDBFYXCET5CKWBOFJ47IUWK4CYPZSWK6HYQVQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.080002833 (176h0m30.631473294s) [noise=true , address=81.171.14.70:4031 , piece_id=54T5MOZZ4YJKQFUDOCCFF2SAE7XS2N77H2AOHCBECAWE5OBUXTZQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.080062066 (176h0m30.631532527s) [noise=true , address=185.167.204.206:4067 , piece_id=2BDMQZLCV4UWWP5HB6GF6NCL5UOGNU7VNYGJIIJCKQHRTX5OPGPQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.080438673 (176h0m30.631909134s) [noise=true , address=185.167.204.205:4034 , piece_id=CZFPF35OQ3U4IH46KILFPFMVY6OC36ZUGZ7EYYBOTB3I4TXMPERA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.080675103 (176h0m30.632145564s) [noise=true , address=185.167.204.205:4014 , piece_id=57C2VUYOEULOTXUQHG7A2XOWZLXYFIEYYRYW5ARX5PKTGWKIEOAQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.081315833 (176h0m30.632786294s) [noise=true , address=81.171.14.70:4039 , piece_id=NMZ5357FVIKD2F3GCKY6YNNV73BAID2YBLVNSSM6VD5PHGLBHEIA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.082053738 (176h0m30.633524189s) [noise=true , address=185.167.204.205:4001 , piece_id=TBPYFHMXWO53LJQQDUA3I2MD2WP6BHQXAH3KIF6XXMWK27HXLCHA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.082605429 (176h0m30.63407589s) [noise=true , address=81.171.14.70:4034 , piece_id=NXBDCCCEWSZ2TMSET6OXBNQV6Q5ESXIQJUFK4WJJ3UV7FJTTAUXQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:33.280203414 (176h0m31.831673875s) [upload=155002]: Next returned nil segment
Nov 16 16:04:33.280222810 (176h0m31.831693271s) [upload=155002]: Waiting for error group managing segments...
Nov 16 16:04:33.838289781 (176h0m32.389760242s) [noise=true , address=185.167.204.205:4013 , piece_id=KF32EXUYGKQEDBPRSVAMNZV3ZMBF6DLT3U32BKX5LXLRBXBVTOQA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.848846382 (176h0m32.400316833s) [noise=true , address=185.167.204.206:4039 , piece_id=6HTUHESIRTSKR5EDMWZ2K56OKE5Y22QQM5ARYTUXR7KMKWIOS7GQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.862791868 (176h0m32.414262329s) [noise=true , address=185.167.204.205:4021 , piece_id=MV3TV4EI3HN3J5NY5XYVKT46QJTJEC5PITXERSSZMS2B7W3DSDTA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.879947135 (176h0m32.431417596s) [noise=true , address=185.167.204.205:4001 , piece_id=TBPYFHMXWO53LJQQDUA3I2MD2WP6BHQXAH3KIF6XXMWK27HXLCHA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.886853657 (176h0m32.438324118s) [noise=true , address=185.167.204.206:4067 , piece_id=2BDMQZLCV4UWWP5HB6GF6NCL5UOGNU7VNYGJIIJCKQHRTX5OPGPQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.918801783 (176h0m32.470272324s) [noise=true , address=185.167.204.206:4051 , piece_id=AU2S6M3E2CQEMAS7LKZJIVWITZUHIHCMWQ2QWT7ILGBRTAUCUB7Q, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.929982083 (176h0m32.481452574s) [noise=true , address=185.167.204.205:4015 , piece_id=6UIFX7GXE2SZ465VAP56YQILR3A3OHDRKIS345EE6GBDG77KJNZQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.935247409 (176h0m32.48671786s) [noise=true , address=185.167.204.206:4038 , piece_id=FBNPAJOOXEAT3KLKQIYOWHGHRGAWJABV4LAA3UYX6SGJB3U2LD7A, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.969630813 (176h0m32.521101264s) [noise=true , address=185.167.204.206:4055 , piece_id=B2TTK2ZLIRM3YIKP5HQ3WOL26KBHSXWWX5ZXXCMWU34AH7CHOOCQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.999043600 (176h0m32.550514061s) [noise=true , address=185.167.204.205:4034 , piece_id=CZFPF35OQ3U4IH46KILFPFMVY6OC36ZUGZ7EYYBOTB3I4TXMPERA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.022951616 (176h0m32.574422077s) [noise=true , address=185.167.204.205:4014 , piece_id=57C2VUYOEULOTXUQHG7A2XOWZLXYFIEYYRYW5ARX5PKTGWKIEOAQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.064488381 (176h0m32.615958832s) [noise=true , address=185.167.204.206:4058 , piece_id=POK7HBLE5AT2STCSL2UJ5R67LNVK3PN53NIRIBX4M7LJYC4C2CHQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.122257987 (176h0m32.673728468s) [noise=true , address=185.167.204.205:4029 , piece_id=VF7BTI4K75HTRVJOHIKMDJN3UBHZ7KCQBT3H44XB5G7ZVFD4HXJQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.136282884 (176h0m32.687753345s) [noise=true , address=185.167.204.206:4041 , piece_id=RTQSCXR54KTKBYS6AHEKPO7ZPUHVQQNUNQJ6JI4JLTGKOLUZH3AQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.136821679 (176h0m32.68829215s) [noise=true , address=185.167.204.206:4062 , piece_id=PAJX3ZCWM5YBZA7YDBFYXCET5CKWBOFJ47IUWK4CYPZSWK6HYQVQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.156096042 (176h0m32.707566493s) [noise=true , address=185.167.204.206:4066 , piece_id=CPTBHDHCT6R2M3UA6X2ZWMPJUKAUVV4I2CIATZ2ZOIWH63Q3UT4Q, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.540981809 (176h0m33.0924523s) [noise=true , address=89.149.201.147:4012 , piece_id=NEIYQHVVWRUVJFF5UD3RZSRHUSNSKXBCNGTMIGOH5MDH2AOZISCA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:21.258617296 (176h1m19.810087757s) [noise=true , address=81.171.14.70:4034 , piece_id=NXBDCCCEWSZ2TMSET6OXBNQV6Q5ESXIQJUFK4WJJ3UV7FJTTAUXQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:22.223432573 (176h1m20.774903034s) [noise=true , address=81.171.14.70:4048 , piece_id=5WUEBVU7UFKHYZ23BPIQUMUUBH3ZXXQWBLE7MALJGECCDTTEK5EA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:22.238494073 (176h1m20.789964534s) [noise=true , address=81.171.14.70:4033 , piece_id=LUBQIVFAU7J72K5XDDS5S2ZXQUGFA5AINL4R5FV64KY24VS5KV5Q, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:22.350384206 (176h1m20.901854667s) [noise=true , address=81.171.14.70:4039 , piece_id=NMZ5357FVIKD2F3GCKY6YNNV73BAID2YBLVNSSM6VD5PHGLBHEIA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:23.704484814 (176h1m22.255955275s) [noise=true , address=81.171.14.70:4037 , piece_id=3IJ7J7RBMOORW4YPKIYJ3C6ZQRYBH7WQMLEKEQYG5UJ4TGKE2W2A, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:24.501497158 (176h1m23.052967619s) [noise=true , address=81.171.14.70:4031 , piece_id=54T5MOZZ4YJKQFUDOCCFF2SAE7XS2N77H2AOHCBECAWE5OBUXTZQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:29.810461698 (176h1m28.361932159s) [noise=true , address=81.171.14.70:4025 , piece_id=3HAT6FJSGV6VOHJBMO6L6L7677BOYVCDV3SR26LJZ6HZOZWJJ64A, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:31.413018058 (176h1m29.964488509s) [noise=true , address=81.171.14.70:4040 , piece_id=YG6ZBK6QTKUS3FCFD2EWU7HTFLOEEVG57SIER44E67QDP5HYCHOA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:1gannW154cmVP6N5BowytKfDorZDRwx2M5pi6akqjZFU38PGjV, address:81.171.14.70:4040): protocol: write tcp 188.166.225.170:59244->81.171.14.70:4040: use of closed network connection; write tcp 188.166.225.170:59244->81.171.14.70:4040: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59244->81.171.14.70:4040: use of closed network connection
Nov 16 16:05:31.413111366 (176h1m29.964581837s) [noise=true , address=81.171.14.70:4035 , piece_id=RGMXRH2Y5XBAFISK5LNAF547SAORD6XSWFUKYWOPJINXXODI5KVQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:12eLYJKne3yP1fZ1oPDLM7Xqs3GQ3ju4ZQedAiojXdVkcxNpzeA, address:81.171.14.70:4035): protocol: read tcp 188.166.225.170:55142->81.171.14.70:4035: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:05:31.413619965 (176h1m29.965090436s) [noise=true , address=81.171.14.70:4030 , piece_id=ZE5OAUTWLWNSMWSFGPEJTCXEN6L3FLWL34QBERTS3JZE7X2I2OHA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:12ZoRwa8wLw66Pjumy1ybp6wrZ79RrSU55uYZpwaY3xi95iSYde, address:81.171.14.70:4030): protocol: read tcp 188.166.225.170:40822->81.171.14.70:4030: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:05:31.414846280 (176h1m29.966316741s) [noise=true , address=81.171.14.70:4029 , piece_id=SIXL2XSCF7IO2BPOUKGH5VVXI6M7O4KJVZARYIRELR2GM4T4RABA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:18yEfU2X4jSrom1JDtQ22cUEvomcg91ohvay2Nsr1Vr8juo9to, address:81.171.14.70:4029): protocol: read tcp 188.166.225.170:56584->81.171.14.70:4029: use of closed network connection; conn closed: op
Nov 16 16:05:31.414946261 (176h1m29.966416722s) [seg_pos={0 0}, upload=155002]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil>
Nov 16 16:05:31.591745995 (176h1m30.143216456s) [upload=155002]: Done uploading segments...
Nov 16 16:05:31.591788596 (176h1m30.143259057s) [upload=155002]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24)
Duration: 5.717472817s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [........................................X...................] duration:
5.978981753s trace:
https://jaeger.dev.storj.io/trace/3e4560ab56ee4602
FAIL (2025-11-16 16:24:43 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.979031958s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [....X.......................................................] duration:
3.509772513s trace:
https://jaeger.dev.storj.io/trace/1026d6559b04863a
FAIL (2025-11-16 16:24:14 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.509809284s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................................X..................] duration:
6.711643228s trace:
https://jaeger.dev.storj.io/trace/3e4560ab56ee4602
FAIL (2025-11-16 16:24:05 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.71167067s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [..............................X.............................] duration:
5.718160472s trace:
https://jaeger.dev.storj.io/trace/7dbd01be7ae5d01e
FAIL (2025-11-16 16:24:00 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 24)
======LOGS======
OHA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.079132696 (176h0m30.630603157s) [noise=true , address=81.171.14.70:4040 , piece_id=YG6ZBK6QTKUS3FCFD2EWU7HTFLOEEVG57SIER44E67QDP5HYCHOA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.079319683 (176h0m30.630790144s) [noise=true , address=185.167.204.205:4015 , piece_id=6UIFX7GXE2SZ465VAP56YQILR3A3OHDRKIS345EE6GBDG77KJNZQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.079646465 (176h0m30.631116926s) [noise=true , address=185.167.204.206:4062 , piece_id=PAJX3ZCWM5YBZA7YDBFYXCET5CKWBOFJ47IUWK4CYPZSWK6HYQVQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.080002833 (176h0m30.631473294s) [noise=true , address=81.171.14.70:4031 , piece_id=54T5MOZZ4YJKQFUDOCCFF2SAE7XS2N77H2AOHCBECAWE5OBUXTZQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.080062066 (176h0m30.631532527s) [noise=true , address=185.167.204.206:4067 , piece_id=2BDMQZLCV4UWWP5HB6GF6NCL5UOGNU7VNYGJIIJCKQHRTX5OPGPQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.080438673 (176h0m30.631909134s) [noise=true , address=185.167.204.205:4034 , piece_id=CZFPF35OQ3U4IH46KILFPFMVY6OC36ZUGZ7EYYBOTB3I4TXMPERA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.080675103 (176h0m30.632145564s) [noise=true , address=185.167.204.205:4014 , piece_id=57C2VUYOEULOTXUQHG7A2XOWZLXYFIEYYRYW5ARX5PKTGWKIEOAQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.081315833 (176h0m30.632786294s) [noise=true , address=81.171.14.70:4039 , piece_id=NMZ5357FVIKD2F3GCKY6YNNV73BAID2YBLVNSSM6VD5PHGLBHEIA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.082053738 (176h0m30.633524189s) [noise=true , address=185.167.204.205:4001 , piece_id=TBPYFHMXWO53LJQQDUA3I2MD2WP6BHQXAH3KIF6XXMWK27HXLCHA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.082605429 (176h0m30.63407589s) [noise=true , address=81.171.14.70:4034 , piece_id=NXBDCCCEWSZ2TMSET6OXBNQV6Q5ESXIQJUFK4WJJ3UV7FJTTAUXQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:33.280203414 (176h0m31.831673875s) [upload=155002]: Next returned nil segment
Nov 16 16:04:33.280222810 (176h0m31.831693271s) [upload=155002]: Waiting for error group managing segments...
Nov 16 16:04:33.838289781 (176h0m32.389760242s) [noise=true , address=185.167.204.205:4013 , piece_id=KF32EXUYGKQEDBPRSVAMNZV3ZMBF6DLT3U32BKX5LXLRBXBVTOQA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.848846382 (176h0m32.400316833s) [noise=true , address=185.167.204.206:4039 , piece_id=6HTUHESIRTSKR5EDMWZ2K56OKE5Y22QQM5ARYTUXR7KMKWIOS7GQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.862791868 (176h0m32.414262329s) [noise=true , address=185.167.204.205:4021 , piece_id=MV3TV4EI3HN3J5NY5XYVKT46QJTJEC5PITXERSSZMS2B7W3DSDTA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.879947135 (176h0m32.431417596s) [noise=true , address=185.167.204.205:4001 , piece_id=TBPYFHMXWO53LJQQDUA3I2MD2WP6BHQXAH3KIF6XXMWK27HXLCHA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.886853657 (176h0m32.438324118s) [noise=true , address=185.167.204.206:4067 , piece_id=2BDMQZLCV4UWWP5HB6GF6NCL5UOGNU7VNYGJIIJCKQHRTX5OPGPQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.918801783 (176h0m32.470272324s) [noise=true , address=185.167.204.206:4051 , piece_id=AU2S6M3E2CQEMAS7LKZJIVWITZUHIHCMWQ2QWT7ILGBRTAUCUB7Q, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.929982083 (176h0m32.481452574s) [noise=true , address=185.167.204.205:4015 , piece_id=6UIFX7GXE2SZ465VAP56YQILR3A3OHDRKIS345EE6GBDG77KJNZQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.935247409 (176h0m32.48671786s) [noise=true , address=185.167.204.206:4038 , piece_id=FBNPAJOOXEAT3KLKQIYOWHGHRGAWJABV4LAA3UYX6SGJB3U2LD7A, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.969630813 (176h0m32.521101264s) [noise=true , address=185.167.204.206:4055 , piece_id=B2TTK2ZLIRM3YIKP5HQ3WOL26KBHSXWWX5ZXXCMWU34AH7CHOOCQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.999043600 (176h0m32.550514061s) [noise=true , address=185.167.204.205:4034 , piece_id=CZFPF35OQ3U4IH46KILFPFMVY6OC36ZUGZ7EYYBOTB3I4TXMPERA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.022951616 (176h0m32.574422077s) [noise=true , address=185.167.204.205:4014 , piece_id=57C2VUYOEULOTXUQHG7A2XOWZLXYFIEYYRYW5ARX5PKTGWKIEOAQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.064488381 (176h0m32.615958832s) [noise=true , address=185.167.204.206:4058 , piece_id=POK7HBLE5AT2STCSL2UJ5R67LNVK3PN53NIRIBX4M7LJYC4C2CHQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.122257987 (176h0m32.673728468s) [noise=true , address=185.167.204.205:4029 , piece_id=VF7BTI4K75HTRVJOHIKMDJN3UBHZ7KCQBT3H44XB5G7ZVFD4HXJQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.136282884 (176h0m32.687753345s) [noise=true , address=185.167.204.206:4041 , piece_id=RTQSCXR54KTKBYS6AHEKPO7ZPUHVQQNUNQJ6JI4JLTGKOLUZH3AQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.136821679 (176h0m32.68829215s) [noise=true , address=185.167.204.206:4062 , piece_id=PAJX3ZCWM5YBZA7YDBFYXCET5CKWBOFJ47IUWK4CYPZSWK6HYQVQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.156096042 (176h0m32.707566493s) [noise=true , address=185.167.204.206:4066 , piece_id=CPTBHDHCT6R2M3UA6X2ZWMPJUKAUVV4I2CIATZ2ZOIWH63Q3UT4Q, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.540981809 (176h0m33.0924523s) [noise=true , address=89.149.201.147:4012 , piece_id=NEIYQHVVWRUVJFF5UD3RZSRHUSNSKXBCNGTMIGOH5MDH2AOZISCA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:21.258617296 (176h1m19.810087757s) [noise=true , address=81.171.14.70:4034 , piece_id=NXBDCCCEWSZ2TMSET6OXBNQV6Q5ESXIQJUFK4WJJ3UV7FJTTAUXQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:22.223432573 (176h1m20.774903034s) [noise=true , address=81.171.14.70:4048 , piece_id=5WUEBVU7UFKHYZ23BPIQUMUUBH3ZXXQWBLE7MALJGECCDTTEK5EA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:22.238494073 (176h1m20.789964534s) [noise=true , address=81.171.14.70:4033 , piece_id=LUBQIVFAU7J72K5XDDS5S2ZXQUGFA5AINL4R5FV64KY24VS5KV5Q, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:22.350384206 (176h1m20.901854667s) [noise=true , address=81.171.14.70:4039 , piece_id=NMZ5357FVIKD2F3GCKY6YNNV73BAID2YBLVNSSM6VD5PHGLBHEIA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:23.704484814 (176h1m22.255955275s) [noise=true , address=81.171.14.70:4037 , piece_id=3IJ7J7RBMOORW4YPKIYJ3C6ZQRYBH7WQMLEKEQYG5UJ4TGKE2W2A, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:24.501497158 (176h1m23.052967619s) [noise=true , address=81.171.14.70:4031 , piece_id=54T5MOZZ4YJKQFUDOCCFF2SAE7XS2N77H2AOHCBECAWE5OBUXTZQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:29.810461698 (176h1m28.361932159s) [noise=true , address=81.171.14.70:4025 , piece_id=3HAT6FJSGV6VOHJBMO6L6L7677BOYVCDV3SR26LJZ6HZOZWJJ64A, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:31.413018058 (176h1m29.964488509s) [noise=true , address=81.171.14.70:4040 , piece_id=YG6ZBK6QTKUS3FCFD2EWU7HTFLOEEVG57SIER44E67QDP5HYCHOA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:1gannW154cmVP6N5BowytKfDorZDRwx2M5pi6akqjZFU38PGjV, address:81.171.14.70:4040): protocol: write tcp 188.166.225.170:59244->81.171.14.70:4040: use of closed network connection; write tcp 188.166.225.170:59244->81.171.14.70:4040: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59244->81.171.14.70:4040: use of closed network connection
Nov 16 16:05:31.413111366 (176h1m29.964581837s) [noise=true , address=81.171.14.70:4035 , piece_id=RGMXRH2Y5XBAFISK5LNAF547SAORD6XSWFUKYWOPJINXXODI5KVQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:12eLYJKne3yP1fZ1oPDLM7Xqs3GQ3ju4ZQedAiojXdVkcxNpzeA, address:81.171.14.70:4035): protocol: read tcp 188.166.225.170:55142->81.171.14.70:4035: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:05:31.413619965 (176h1m29.965090436s) [noise=true , address=81.171.14.70:4030 , piece_id=ZE5OAUTWLWNSMWSFGPEJTCXEN6L3FLWL34QBERTS3JZE7X2I2OHA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:12ZoRwa8wLw66Pjumy1ybp6wrZ79RrSU55uYZpwaY3xi95iSYde, address:81.171.14.70:4030): protocol: read tcp 188.166.225.170:40822->81.171.14.70:4030: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:05:31.414846280 (176h1m29.966316741s) [noise=true , address=81.171.14.70:4029 , piece_id=SIXL2XSCF7IO2BPOUKGH5VVXI6M7O4KJVZARYIRELR2GM4T4RABA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:18yEfU2X4jSrom1JDtQ22cUEvomcg91ohvay2Nsr1Vr8juo9to, address:81.171.14.70:4029): protocol: read tcp 188.166.225.170:56584->81.171.14.70:4029: use of closed network connection; conn closed: op
Nov 16 16:05:31.414946261 (176h1m29.966416722s) [seg_pos={0 0}, upload=155002]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil>
Nov 16 16:05:31.591745995 (176h1m30.143216456s) [upload=155002]: Done uploading segments...
Nov 16 16:05:31.591788596 (176h1m30.143259057s) [upload=155002]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24)
Duration: 5.718194397s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................................X.................] duration:
5.938716308s trace:
https://jaeger.dev.storj.io/trace/70f81d2d9b012a3a
FAIL (2025-11-16 16:23:57 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 4)
======LOGS======
ddress=45.91.116.5:4297 , piece_id=OSMI3VGAKSCIZMDQ3Y2VZQBUOJKSXM6RWNWSAV57RDTL5TR3NIJA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.462247309 (176h7m35.013717781s) [noise=true , address=45.91.116.7:4011 , piece_id=ORJVRMFIOEXZ6OB6QFQDL764Q2FCWOJT63NBIRI6SOOS2FEIBCFQ, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.463420774 (176h7m35.014891235s) [noise=true , address=145.79.227.9:4367 , piece_id=LOFHPWCHNEZ4E4WANJM5BE2TIWVFH4LGEJJYK4ETB2LZWYCAVGZA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.464113112 (176h7m35.015583573s) [noise=true , address=145.79.227.22:4511 , piece_id=HES465EP3OPD5JBVGVYMGLUOHJKSSMJDXMJVWET5GHRPFZJFIFOA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.464764873 (176h7m35.016235334s) [noise=true , address=45.91.116.8:4048 , piece_id=ULVK52VWCW3KNEPWSDYGRHWHIJWEQU2XNFEPYJFXWLVJI6AGZQOA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.465452753 (176h7m35.016923214s) [noise=true , address=145.79.227.28:4725 , piece_id=CI2LSBGPW7RYH4RGWODGWLUTN353EODNV6IB2YQGHOZ3FPSEEK5A, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.466223270 (176h7m35.017693731s) [noise=true , address=145.79.227.16:4910 , piece_id=3U2SEVMPILIEBRHS3M4YT4Z4PSYAQAIC2SKGYMW4FNODACZLOQCQ, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.466955855 (176h7m35.018426306s) [noise=true , address=145.79.227.4:4186 , piece_id=5PZIK3V72CUQQ6M2HJZFVM5OVX4HPHY7ZHJ2BAK5S3ZQOGM4J5OA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.467675205 (176h7m35.019145666s) [noise=true , address=145.79.227.8:4331 , piece_id=4O27HU7AMR6G4Y43HJYN22UXVLETBDBE7JHL4JUZKRM54SK5E4JA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.468178092 (176h7m35.019648553s) [noise=true , address=145.79.227.8:4329 , piece_id=GO5IAWQYS3UARWYAQAEUIIS456DFWCFWRJQG2KOVVL5KPXWPK4AA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:39.111804678 (176h7m37.663275129s) [upload=155107]: Next returned nil segment
Nov 16 16:11:39.111825767 (176h7m37.663296218s) [upload=155107]: Waiting for error group managing segments...
Nov 16 16:11:39.896671313 (176h7m38.448141765s) [noise=true , address=45.91.116.12:4227 , piece_id=CQA2LHRSFOA7EIBIFVC75EWVHFZMWMY7ODMETEAJRRNYSQFY5TVQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil>
Nov 16 16:11:39.954993420 (176h7m38.506463881s) [noise=true , address=45.91.116.8:4048 , piece_id=ULVK52VWCW3KNEPWSDYGRHWHIJWEQU2XNFEPYJFXWLVJI6AGZQOA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil>
Nov 16 16:11:40.066014029 (176h7m38.61748449s) [noise=true , address=45.91.116.7:4011 , piece_id=ORJVRMFIOEXZ6OB6QFQDL764Q2FCWOJT63NBIRI6SOOS2FEIBCFQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil>
Nov 16 16:11:41.102688250 (176h7m39.654158711s) [noise=true , address=45.91.116.5:4297 , piece_id=OSMI3VGAKSCIZMDQ3Y2VZQBUOJKSXM6RWNWSAV57RDTL5TR3NIJA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil>
Nov 16 16:12:04.264842280 (176h8m2.816312741s) [noise=true , address=145.79.227.17:4972 , piece_id=NV6FSBQM6YUTKYFAJPQCURWCZBHJ3AKJOLLGCRYTJ3KQ54U4GOWQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection
Nov 16 16:12:04.265084832 (176h8m2.816555293s) [noise=true , address=145.79.227.16:4910 , piece_id=3U2SEVMPILIEBRHS3M4YT4Z4PSYAQAIC2SKGYMW4FNODACZLOQCQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection
Nov 16 16:12:04.270267942 (176h8m2.821738423s) [noise=true , address=145.79.227.8:4329 , piece_id=GO5IAWQYS3UARWYAQAEUIIS456DFWCFWRJQG2KOVVL5KPXWPK4AA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: expected piece hash; context deadline exceeded; EOF
Nov 16 16:12:04.270306515 (176h8m2.821776986s) [noise=true , address=145.79.227.9:4367 , piece_id=LOFHPWCHNEZ4E4WANJM5BE2TIWVFH4LGEJJYK4ETB2LZWYCAVGZA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection
Nov 16 16:12:04.270323538 (176h8m2.821794099s) [noise=true , address=145.79.227.28:4725 , piece_id=CI2LSBGPW7RYH4RGWODGWLUTN353EODNV6IB2YQGHOZ3FPSEEK5A, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection
Nov 16 16:12:04.270544138 (176h8m2.822014739s) [noise=true , address=145.79.227.14:4840 , piece_id=RSHGL65RI7HZGU3W3EROFL23BCK2EW3YN6CAD3NO5OVE5QM3QG3Q, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: read tcp 188.166.225.170:42148->145.79.227.14:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:12:04.270571570 (176h8m2.822042121s) [noise=true , address=145.79.227.31:4797 , piece_id=AWSPIPRKSBOKQFZRRB34RUPLYVJGKAOOJQ4U7RRATQJELM3OCFHA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection
Nov 16 16:12:04.270609783 (176h8m2.822080254s) [noise=true , address=145.79.227.22:4511 , piece_id=HES465EP3OPD5JBVGVYMGLUOHJKSSMJDXMJVWET5GHRPFZJFIFOA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: read tcp 188.166.225.170:51004->145.79.227.22:4511: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:12:04.270625552 (176h8m2.822096083s) [noise=true , address=145.79.227.8:4331 , piece_id=4O27HU7AMR6G4Y43HJYN22UXVLETBDBE7JHL4JUZKRM54SK5E4JA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: read tcp 188.166.225.170:44510->145.79.227.8:4331: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:12:04.270742155 (176h8m2.822212616s) [noise=true , address=145.79.227.4:4186 , piece_id=5PZIK3V72CUQQ6M2HJZFVM5OVX4HPHY7ZHJ2BAK5S3ZQOGM4J5OA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection
Nov 16 16:12:04.270762814 (176h8m2.822233275s) [noise=true , address=145.79.227.24:4581 , piece_id=XRCS52EJDB7HWD4RDZK3UNB6VKHOTWWID42RGBJZQHTGG4NDYIPA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: read tcp 188.166.225.170:44130->145.79.227.24:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:12:04.270781470 (176h8m2.822251921s) [noise=true , address=145.79.227.11:4732 , piece_id=FQGAVGZ5CID3LBET3CVK6FSCOP5OUYGS44XYWMHU2RLIT3MBN62A, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: read tcp 188.166.225.170:42420->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:12:04.270825082 (176h8m2.822295544s) [seg_pos={0 0}, upload=155107]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil>
Nov 16 16:12:05.262661603 (176h8m3.814132064s) [upload=155107]: Done uploading segments...
Nov 16 16:12:05.262728080 (176h8m3.814198541s) [upload=155107]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4)
Duration: 5.938744042s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................................X.................] duration:
6.305564379s trace:
https://jaeger.dev.storj.io/trace/3e4560ab56ee4602
FAIL (2025-11-16 16:23:32 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.305594055s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................................X................] duration:
6.139346695s trace:
https://jaeger.dev.storj.io/trace/70f81d2d9b012a3a
FAIL (2025-11-16 16:23:07 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 4)
======LOGS======
ddress=45.91.116.5:4297 , piece_id=OSMI3VGAKSCIZMDQ3Y2VZQBUOJKSXM6RWNWSAV57RDTL5TR3NIJA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.462247309 (176h7m35.013717781s) [noise=true , address=45.91.116.7:4011 , piece_id=ORJVRMFIOEXZ6OB6QFQDL764Q2FCWOJT63NBIRI6SOOS2FEIBCFQ, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.463420774 (176h7m35.014891235s) [noise=true , address=145.79.227.9:4367 , piece_id=LOFHPWCHNEZ4E4WANJM5BE2TIWVFH4LGEJJYK4ETB2LZWYCAVGZA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.464113112 (176h7m35.015583573s) [noise=true , address=145.79.227.22:4511 , piece_id=HES465EP3OPD5JBVGVYMGLUOHJKSSMJDXMJVWET5GHRPFZJFIFOA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.464764873 (176h7m35.016235334s) [noise=true , address=45.91.116.8:4048 , piece_id=ULVK52VWCW3KNEPWSDYGRHWHIJWEQU2XNFEPYJFXWLVJI6AGZQOA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.465452753 (176h7m35.016923214s) [noise=true , address=145.79.227.28:4725 , piece_id=CI2LSBGPW7RYH4RGWODGWLUTN353EODNV6IB2YQGHOZ3FPSEEK5A, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.466223270 (176h7m35.017693731s) [noise=true , address=145.79.227.16:4910 , piece_id=3U2SEVMPILIEBRHS3M4YT4Z4PSYAQAIC2SKGYMW4FNODACZLOQCQ, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.466955855 (176h7m35.018426306s) [noise=true , address=145.79.227.4:4186 , piece_id=5PZIK3V72CUQQ6M2HJZFVM5OVX4HPHY7ZHJ2BAK5S3ZQOGM4J5OA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.467675205 (176h7m35.019145666s) [noise=true , address=145.79.227.8:4331 , piece_id=4O27HU7AMR6G4Y43HJYN22UXVLETBDBE7JHL4JUZKRM54SK5E4JA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.468178092 (176h7m35.019648553s) [noise=true , address=145.79.227.8:4329 , piece_id=GO5IAWQYS3UARWYAQAEUIIS456DFWCFWRJQG2KOVVL5KPXWPK4AA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:39.111804678 (176h7m37.663275129s) [upload=155107]: Next returned nil segment
Nov 16 16:11:39.111825767 (176h7m37.663296218s) [upload=155107]: Waiting for error group managing segments...
Nov 16 16:11:39.896671313 (176h7m38.448141765s) [noise=true , address=45.91.116.12:4227 , piece_id=CQA2LHRSFOA7EIBIFVC75EWVHFZMWMY7ODMETEAJRRNYSQFY5TVQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil>
Nov 16 16:11:39.954993420 (176h7m38.506463881s) [noise=true , address=45.91.116.8:4048 , piece_id=ULVK52VWCW3KNEPWSDYGRHWHIJWEQU2XNFEPYJFXWLVJI6AGZQOA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil>
Nov 16 16:11:40.066014029 (176h7m38.61748449s) [noise=true , address=45.91.116.7:4011 , piece_id=ORJVRMFIOEXZ6OB6QFQDL764Q2FCWOJT63NBIRI6SOOS2FEIBCFQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil>
Nov 16 16:11:41.102688250 (176h7m39.654158711s) [noise=true , address=45.91.116.5:4297 , piece_id=OSMI3VGAKSCIZMDQ3Y2VZQBUOJKSXM6RWNWSAV57RDTL5TR3NIJA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil>
Nov 16 16:12:04.264842280 (176h8m2.816312741s) [noise=true , address=145.79.227.17:4972 , piece_id=NV6FSBQM6YUTKYFAJPQCURWCZBHJ3AKJOLLGCRYTJ3KQ54U4GOWQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection
Nov 16 16:12:04.265084832 (176h8m2.816555293s) [noise=true , address=145.79.227.16:4910 , piece_id=3U2SEVMPILIEBRHS3M4YT4Z4PSYAQAIC2SKGYMW4FNODACZLOQCQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection
Nov 16 16:12:04.270267942 (176h8m2.821738423s) [noise=true , address=145.79.227.8:4329 , piece_id=GO5IAWQYS3UARWYAQAEUIIS456DFWCFWRJQG2KOVVL5KPXWPK4AA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: expected piece hash; context deadline exceeded; EOF
Nov 16 16:12:04.270306515 (176h8m2.821776986s) [noise=true , address=145.79.227.9:4367 , piece_id=LOFHPWCHNEZ4E4WANJM5BE2TIWVFH4LGEJJYK4ETB2LZWYCAVGZA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection
Nov 16 16:12:04.270323538 (176h8m2.821794099s) [noise=true , address=145.79.227.28:4725 , piece_id=CI2LSBGPW7RYH4RGWODGWLUTN353EODNV6IB2YQGHOZ3FPSEEK5A, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection
Nov 16 16:12:04.270544138 (176h8m2.822014739s) [noise=true , address=145.79.227.14:4840 , piece_id=RSHGL65RI7HZGU3W3EROFL23BCK2EW3YN6CAD3NO5OVE5QM3QG3Q, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: read tcp 188.166.225.170:42148->145.79.227.14:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:12:04.270571570 (176h8m2.822042121s) [noise=true , address=145.79.227.31:4797 , piece_id=AWSPIPRKSBOKQFZRRB34RUPLYVJGKAOOJQ4U7RRATQJELM3OCFHA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection
Nov 16 16:12:04.270609783 (176h8m2.822080254s) [noise=true , address=145.79.227.22:4511 , piece_id=HES465EP3OPD5JBVGVYMGLUOHJKSSMJDXMJVWET5GHRPFZJFIFOA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: read tcp 188.166.225.170:51004->145.79.227.22:4511: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:12:04.270625552 (176h8m2.822096083s) [noise=true , address=145.79.227.8:4331 , piece_id=4O27HU7AMR6G4Y43HJYN22UXVLETBDBE7JHL4JUZKRM54SK5E4JA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: read tcp 188.166.225.170:44510->145.79.227.8:4331: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:12:04.270742155 (176h8m2.822212616s) [noise=true , address=145.79.227.4:4186 , piece_id=5PZIK3V72CUQQ6M2HJZFVM5OVX4HPHY7ZHJ2BAK5S3ZQOGM4J5OA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection
Nov 16 16:12:04.270762814 (176h8m2.822233275s) [noise=true , address=145.79.227.24:4581 , piece_id=XRCS52EJDB7HWD4RDZK3UNB6VKHOTWWID42RGBJZQHTGG4NDYIPA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: read tcp 188.166.225.170:44130->145.79.227.24:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:12:04.270781470 (176h8m2.822251921s) [noise=true , address=145.79.227.11:4732 , piece_id=FQGAVGZ5CID3LBET3CVK6FSCOP5OUYGS44XYWMHU2RLIT3MBN62A, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: read tcp 188.166.225.170:42420->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:12:04.270825082 (176h8m2.822295544s) [seg_pos={0 0}, upload=155107]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil>
Nov 16 16:12:05.262661603 (176h8m3.814132064s) [upload=155107]: Done uploading segments...
Nov 16 16:12:05.262728080 (176h8m3.814198541s) [upload=155107]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4)
Duration: 6.13937528s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [.....X......................................................] duration:
3.501105663s trace:
https://jaeger.dev.storj.io/trace/1026d6559b04863a
FAIL (2025-11-16 16:23:06 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.501141391s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [......X.....................................................] duration:
3.4332384s trace:
https://jaeger.dev.storj.io/trace/1026d6559b04863a
FAIL (2025-11-16 16:22:51 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.433264489s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [...............................X............................] duration:
5.662997403s trace:
https://jaeger.dev.storj.io/trace/7dbd01be7ae5d01e
FAIL (2025-11-16 16:22:49 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 24)
======LOGS======
OHA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.079132696 (176h0m30.630603157s) [noise=true , address=81.171.14.70:4040 , piece_id=YG6ZBK6QTKUS3FCFD2EWU7HTFLOEEVG57SIER44E67QDP5HYCHOA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.079319683 (176h0m30.630790144s) [noise=true , address=185.167.204.205:4015 , piece_id=6UIFX7GXE2SZ465VAP56YQILR3A3OHDRKIS345EE6GBDG77KJNZQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.079646465 (176h0m30.631116926s) [noise=true , address=185.167.204.206:4062 , piece_id=PAJX3ZCWM5YBZA7YDBFYXCET5CKWBOFJ47IUWK4CYPZSWK6HYQVQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.080002833 (176h0m30.631473294s) [noise=true , address=81.171.14.70:4031 , piece_id=54T5MOZZ4YJKQFUDOCCFF2SAE7XS2N77H2AOHCBECAWE5OBUXTZQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.080062066 (176h0m30.631532527s) [noise=true , address=185.167.204.206:4067 , piece_id=2BDMQZLCV4UWWP5HB6GF6NCL5UOGNU7VNYGJIIJCKQHRTX5OPGPQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.080438673 (176h0m30.631909134s) [noise=true , address=185.167.204.205:4034 , piece_id=CZFPF35OQ3U4IH46KILFPFMVY6OC36ZUGZ7EYYBOTB3I4TXMPERA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.080675103 (176h0m30.632145564s) [noise=true , address=185.167.204.205:4014 , piece_id=57C2VUYOEULOTXUQHG7A2XOWZLXYFIEYYRYW5ARX5PKTGWKIEOAQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.081315833 (176h0m30.632786294s) [noise=true , address=81.171.14.70:4039 , piece_id=NMZ5357FVIKD2F3GCKY6YNNV73BAID2YBLVNSSM6VD5PHGLBHEIA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.082053738 (176h0m30.633524189s) [noise=true , address=185.167.204.205:4001 , piece_id=TBPYFHMXWO53LJQQDUA3I2MD2WP6BHQXAH3KIF6XXMWK27HXLCHA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.082605429 (176h0m30.63407589s) [noise=true , address=81.171.14.70:4034 , piece_id=NXBDCCCEWSZ2TMSET6OXBNQV6Q5ESXIQJUFK4WJJ3UV7FJTTAUXQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:33.280203414 (176h0m31.831673875s) [upload=155002]: Next returned nil segment
Nov 16 16:04:33.280222810 (176h0m31.831693271s) [upload=155002]: Waiting for error group managing segments...
Nov 16 16:04:33.838289781 (176h0m32.389760242s) [noise=true , address=185.167.204.205:4013 , piece_id=KF32EXUYGKQEDBPRSVAMNZV3ZMBF6DLT3U32BKX5LXLRBXBVTOQA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.848846382 (176h0m32.400316833s) [noise=true , address=185.167.204.206:4039 , piece_id=6HTUHESIRTSKR5EDMWZ2K56OKE5Y22QQM5ARYTUXR7KMKWIOS7GQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.862791868 (176h0m32.414262329s) [noise=true , address=185.167.204.205:4021 , piece_id=MV3TV4EI3HN3J5NY5XYVKT46QJTJEC5PITXERSSZMS2B7W3DSDTA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.879947135 (176h0m32.431417596s) [noise=true , address=185.167.204.205:4001 , piece_id=TBPYFHMXWO53LJQQDUA3I2MD2WP6BHQXAH3KIF6XXMWK27HXLCHA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.886853657 (176h0m32.438324118s) [noise=true , address=185.167.204.206:4067 , piece_id=2BDMQZLCV4UWWP5HB6GF6NCL5UOGNU7VNYGJIIJCKQHRTX5OPGPQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.918801783 (176h0m32.470272324s) [noise=true , address=185.167.204.206:4051 , piece_id=AU2S6M3E2CQEMAS7LKZJIVWITZUHIHCMWQ2QWT7ILGBRTAUCUB7Q, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.929982083 (176h0m32.481452574s) [noise=true , address=185.167.204.205:4015 , piece_id=6UIFX7GXE2SZ465VAP56YQILR3A3OHDRKIS345EE6GBDG77KJNZQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.935247409 (176h0m32.48671786s) [noise=true , address=185.167.204.206:4038 , piece_id=FBNPAJOOXEAT3KLKQIYOWHGHRGAWJABV4LAA3UYX6SGJB3U2LD7A, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.969630813 (176h0m32.521101264s) [noise=true , address=185.167.204.206:4055 , piece_id=B2TTK2ZLIRM3YIKP5HQ3WOL26KBHSXWWX5ZXXCMWU34AH7CHOOCQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.999043600 (176h0m32.550514061s) [noise=true , address=185.167.204.205:4034 , piece_id=CZFPF35OQ3U4IH46KILFPFMVY6OC36ZUGZ7EYYBOTB3I4TXMPERA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.022951616 (176h0m32.574422077s) [noise=true , address=185.167.204.205:4014 , piece_id=57C2VUYOEULOTXUQHG7A2XOWZLXYFIEYYRYW5ARX5PKTGWKIEOAQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.064488381 (176h0m32.615958832s) [noise=true , address=185.167.204.206:4058 , piece_id=POK7HBLE5AT2STCSL2UJ5R67LNVK3PN53NIRIBX4M7LJYC4C2CHQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.122257987 (176h0m32.673728468s) [noise=true , address=185.167.204.205:4029 , piece_id=VF7BTI4K75HTRVJOHIKMDJN3UBHZ7KCQBT3H44XB5G7ZVFD4HXJQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.136282884 (176h0m32.687753345s) [noise=true , address=185.167.204.206:4041 , piece_id=RTQSCXR54KTKBYS6AHEKPO7ZPUHVQQNUNQJ6JI4JLTGKOLUZH3AQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.136821679 (176h0m32.68829215s) [noise=true , address=185.167.204.206:4062 , piece_id=PAJX3ZCWM5YBZA7YDBFYXCET5CKWBOFJ47IUWK4CYPZSWK6HYQVQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.156096042 (176h0m32.707566493s) [noise=true , address=185.167.204.206:4066 , piece_id=CPTBHDHCT6R2M3UA6X2ZWMPJUKAUVV4I2CIATZ2ZOIWH63Q3UT4Q, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.540981809 (176h0m33.0924523s) [noise=true , address=89.149.201.147:4012 , piece_id=NEIYQHVVWRUVJFF5UD3RZSRHUSNSKXBCNGTMIGOH5MDH2AOZISCA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:21.258617296 (176h1m19.810087757s) [noise=true , address=81.171.14.70:4034 , piece_id=NXBDCCCEWSZ2TMSET6OXBNQV6Q5ESXIQJUFK4WJJ3UV7FJTTAUXQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:22.223432573 (176h1m20.774903034s) [noise=true , address=81.171.14.70:4048 , piece_id=5WUEBVU7UFKHYZ23BPIQUMUUBH3ZXXQWBLE7MALJGECCDTTEK5EA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:22.238494073 (176h1m20.789964534s) [noise=true , address=81.171.14.70:4033 , piece_id=LUBQIVFAU7J72K5XDDS5S2ZXQUGFA5AINL4R5FV64KY24VS5KV5Q, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:22.350384206 (176h1m20.901854667s) [noise=true , address=81.171.14.70:4039 , piece_id=NMZ5357FVIKD2F3GCKY6YNNV73BAID2YBLVNSSM6VD5PHGLBHEIA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:23.704484814 (176h1m22.255955275s) [noise=true , address=81.171.14.70:4037 , piece_id=3IJ7J7RBMOORW4YPKIYJ3C6ZQRYBH7WQMLEKEQYG5UJ4TGKE2W2A, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:24.501497158 (176h1m23.052967619s) [noise=true , address=81.171.14.70:4031 , piece_id=54T5MOZZ4YJKQFUDOCCFF2SAE7XS2N77H2AOHCBECAWE5OBUXTZQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:29.810461698 (176h1m28.361932159s) [noise=true , address=81.171.14.70:4025 , piece_id=3HAT6FJSGV6VOHJBMO6L6L7677BOYVCDV3SR26LJZ6HZOZWJJ64A, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:31.413018058 (176h1m29.964488509s) [noise=true , address=81.171.14.70:4040 , piece_id=YG6ZBK6QTKUS3FCFD2EWU7HTFLOEEVG57SIER44E67QDP5HYCHOA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:1gannW154cmVP6N5BowytKfDorZDRwx2M5pi6akqjZFU38PGjV, address:81.171.14.70:4040): protocol: write tcp 188.166.225.170:59244->81.171.14.70:4040: use of closed network connection; write tcp 188.166.225.170:59244->81.171.14.70:4040: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59244->81.171.14.70:4040: use of closed network connection
Nov 16 16:05:31.413111366 (176h1m29.964581837s) [noise=true , address=81.171.14.70:4035 , piece_id=RGMXRH2Y5XBAFISK5LNAF547SAORD6XSWFUKYWOPJINXXODI5KVQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:12eLYJKne3yP1fZ1oPDLM7Xqs3GQ3ju4ZQedAiojXdVkcxNpzeA, address:81.171.14.70:4035): protocol: read tcp 188.166.225.170:55142->81.171.14.70:4035: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:05:31.413619965 (176h1m29.965090436s) [noise=true , address=81.171.14.70:4030 , piece_id=ZE5OAUTWLWNSMWSFGPEJTCXEN6L3FLWL34QBERTS3JZE7X2I2OHA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:12ZoRwa8wLw66Pjumy1ybp6wrZ79RrSU55uYZpwaY3xi95iSYde, address:81.171.14.70:4030): protocol: read tcp 188.166.225.170:40822->81.171.14.70:4030: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:05:31.414846280 (176h1m29.966316741s) [noise=true , address=81.171.14.70:4029 , piece_id=SIXL2XSCF7IO2BPOUKGH5VVXI6M7O4KJVZARYIRELR2GM4T4RABA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:18yEfU2X4jSrom1JDtQ22cUEvomcg91ohvay2Nsr1Vr8juo9to, address:81.171.14.70:4029): protocol: read tcp 188.166.225.170:56584->81.171.14.70:4029: use of closed network connection; conn closed: op
Nov 16 16:05:31.414946261 (176h1m29.966416722s) [seg_pos={0 0}, upload=155002]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil>
Nov 16 16:05:31.591745995 (176h1m30.143216456s) [upload=155002]: Done uploading segments...
Nov 16 16:05:31.591788596 (176h1m30.143259057s) [upload=155002]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24)
Duration: 5.663024565s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................................X................] duration:
6.227034303s trace:
https://jaeger.dev.storj.io/trace/3e4560ab56ee4602
FAIL (2025-11-16 16:22:49 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.227068407s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [................................X...........................] duration:
5.725616013s trace:
https://jaeger.dev.storj.io/trace/7dbd01be7ae5d01e
FAIL (2025-11-16 16:22:29 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 24)
======LOGS======
OHA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.079132696 (176h0m30.630603157s) [noise=true , address=81.171.14.70:4040 , piece_id=YG6ZBK6QTKUS3FCFD2EWU7HTFLOEEVG57SIER44E67QDP5HYCHOA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.079319683 (176h0m30.630790144s) [noise=true , address=185.167.204.205:4015 , piece_id=6UIFX7GXE2SZ465VAP56YQILR3A3OHDRKIS345EE6GBDG77KJNZQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.079646465 (176h0m30.631116926s) [noise=true , address=185.167.204.206:4062 , piece_id=PAJX3ZCWM5YBZA7YDBFYXCET5CKWBOFJ47IUWK4CYPZSWK6HYQVQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.080002833 (176h0m30.631473294s) [noise=true , address=81.171.14.70:4031 , piece_id=54T5MOZZ4YJKQFUDOCCFF2SAE7XS2N77H2AOHCBECAWE5OBUXTZQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.080062066 (176h0m30.631532527s) [noise=true , address=185.167.204.206:4067 , piece_id=2BDMQZLCV4UWWP5HB6GF6NCL5UOGNU7VNYGJIIJCKQHRTX5OPGPQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.080438673 (176h0m30.631909134s) [noise=true , address=185.167.204.205:4034 , piece_id=CZFPF35OQ3U4IH46KILFPFMVY6OC36ZUGZ7EYYBOTB3I4TXMPERA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.080675103 (176h0m30.632145564s) [noise=true , address=185.167.204.205:4014 , piece_id=57C2VUYOEULOTXUQHG7A2XOWZLXYFIEYYRYW5ARX5PKTGWKIEOAQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.081315833 (176h0m30.632786294s) [noise=true , address=81.171.14.70:4039 , piece_id=NMZ5357FVIKD2F3GCKY6YNNV73BAID2YBLVNSSM6VD5PHGLBHEIA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.082053738 (176h0m30.633524189s) [noise=true , address=185.167.204.205:4001 , piece_id=TBPYFHMXWO53LJQQDUA3I2MD2WP6BHQXAH3KIF6XXMWK27HXLCHA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.082605429 (176h0m30.63407589s) [noise=true , address=81.171.14.70:4034 , piece_id=NXBDCCCEWSZ2TMSET6OXBNQV6Q5ESXIQJUFK4WJJ3UV7FJTTAUXQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:33.280203414 (176h0m31.831673875s) [upload=155002]: Next returned nil segment
Nov 16 16:04:33.280222810 (176h0m31.831693271s) [upload=155002]: Waiting for error group managing segments...
Nov 16 16:04:33.838289781 (176h0m32.389760242s) [noise=true , address=185.167.204.205:4013 , piece_id=KF32EXUYGKQEDBPRSVAMNZV3ZMBF6DLT3U32BKX5LXLRBXBVTOQA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.848846382 (176h0m32.400316833s) [noise=true , address=185.167.204.206:4039 , piece_id=6HTUHESIRTSKR5EDMWZ2K56OKE5Y22QQM5ARYTUXR7KMKWIOS7GQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.862791868 (176h0m32.414262329s) [noise=true , address=185.167.204.205:4021 , piece_id=MV3TV4EI3HN3J5NY5XYVKT46QJTJEC5PITXERSSZMS2B7W3DSDTA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.879947135 (176h0m32.431417596s) [noise=true , address=185.167.204.205:4001 , piece_id=TBPYFHMXWO53LJQQDUA3I2MD2WP6BHQXAH3KIF6XXMWK27HXLCHA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.886853657 (176h0m32.438324118s) [noise=true , address=185.167.204.206:4067 , piece_id=2BDMQZLCV4UWWP5HB6GF6NCL5UOGNU7VNYGJIIJCKQHRTX5OPGPQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.918801783 (176h0m32.470272324s) [noise=true , address=185.167.204.206:4051 , piece_id=AU2S6M3E2CQEMAS7LKZJIVWITZUHIHCMWQ2QWT7ILGBRTAUCUB7Q, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.929982083 (176h0m32.481452574s) [noise=true , address=185.167.204.205:4015 , piece_id=6UIFX7GXE2SZ465VAP56YQILR3A3OHDRKIS345EE6GBDG77KJNZQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.935247409 (176h0m32.48671786s) [noise=true , address=185.167.204.206:4038 , piece_id=FBNPAJOOXEAT3KLKQIYOWHGHRGAWJABV4LAA3UYX6SGJB3U2LD7A, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.969630813 (176h0m32.521101264s) [noise=true , address=185.167.204.206:4055 , piece_id=B2TTK2ZLIRM3YIKP5HQ3WOL26KBHSXWWX5ZXXCMWU34AH7CHOOCQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.999043600 (176h0m32.550514061s) [noise=true , address=185.167.204.205:4034 , piece_id=CZFPF35OQ3U4IH46KILFPFMVY6OC36ZUGZ7EYYBOTB3I4TXMPERA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.022951616 (176h0m32.574422077s) [noise=true , address=185.167.204.205:4014 , piece_id=57C2VUYOEULOTXUQHG7A2XOWZLXYFIEYYRYW5ARX5PKTGWKIEOAQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.064488381 (176h0m32.615958832s) [noise=true , address=185.167.204.206:4058 , piece_id=POK7HBLE5AT2STCSL2UJ5R67LNVK3PN53NIRIBX4M7LJYC4C2CHQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.122257987 (176h0m32.673728468s) [noise=true , address=185.167.204.205:4029 , piece_id=VF7BTI4K75HTRVJOHIKMDJN3UBHZ7KCQBT3H44XB5G7ZVFD4HXJQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.136282884 (176h0m32.687753345s) [noise=true , address=185.167.204.206:4041 , piece_id=RTQSCXR54KTKBYS6AHEKPO7ZPUHVQQNUNQJ6JI4JLTGKOLUZH3AQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.136821679 (176h0m32.68829215s) [noise=true , address=185.167.204.206:4062 , piece_id=PAJX3ZCWM5YBZA7YDBFYXCET5CKWBOFJ47IUWK4CYPZSWK6HYQVQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.156096042 (176h0m32.707566493s) [noise=true , address=185.167.204.206:4066 , piece_id=CPTBHDHCT6R2M3UA6X2ZWMPJUKAUVV4I2CIATZ2ZOIWH63Q3UT4Q, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.540981809 (176h0m33.0924523s) [noise=true , address=89.149.201.147:4012 , piece_id=NEIYQHVVWRUVJFF5UD3RZSRHUSNSKXBCNGTMIGOH5MDH2AOZISCA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:21.258617296 (176h1m19.810087757s) [noise=true , address=81.171.14.70:4034 , piece_id=NXBDCCCEWSZ2TMSET6OXBNQV6Q5ESXIQJUFK4WJJ3UV7FJTTAUXQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:22.223432573 (176h1m20.774903034s) [noise=true , address=81.171.14.70:4048 , piece_id=5WUEBVU7UFKHYZ23BPIQUMUUBH3ZXXQWBLE7MALJGECCDTTEK5EA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:22.238494073 (176h1m20.789964534s) [noise=true , address=81.171.14.70:4033 , piece_id=LUBQIVFAU7J72K5XDDS5S2ZXQUGFA5AINL4R5FV64KY24VS5KV5Q, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:22.350384206 (176h1m20.901854667s) [noise=true , address=81.171.14.70:4039 , piece_id=NMZ5357FVIKD2F3GCKY6YNNV73BAID2YBLVNSSM6VD5PHGLBHEIA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:23.704484814 (176h1m22.255955275s) [noise=true , address=81.171.14.70:4037 , piece_id=3IJ7J7RBMOORW4YPKIYJ3C6ZQRYBH7WQMLEKEQYG5UJ4TGKE2W2A, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:24.501497158 (176h1m23.052967619s) [noise=true , address=81.171.14.70:4031 , piece_id=54T5MOZZ4YJKQFUDOCCFF2SAE7XS2N77H2AOHCBECAWE5OBUXTZQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:29.810461698 (176h1m28.361932159s) [noise=true , address=81.171.14.70:4025 , piece_id=3HAT6FJSGV6VOHJBMO6L6L7677BOYVCDV3SR26LJZ6HZOZWJJ64A, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:31.413018058 (176h1m29.964488509s) [noise=true , address=81.171.14.70:4040 , piece_id=YG6ZBK6QTKUS3FCFD2EWU7HTFLOEEVG57SIER44E67QDP5HYCHOA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:1gannW154cmVP6N5BowytKfDorZDRwx2M5pi6akqjZFU38PGjV, address:81.171.14.70:4040): protocol: write tcp 188.166.225.170:59244->81.171.14.70:4040: use of closed network connection; write tcp 188.166.225.170:59244->81.171.14.70:4040: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59244->81.171.14.70:4040: use of closed network connection
Nov 16 16:05:31.413111366 (176h1m29.964581837s) [noise=true , address=81.171.14.70:4035 , piece_id=RGMXRH2Y5XBAFISK5LNAF547SAORD6XSWFUKYWOPJINXXODI5KVQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:12eLYJKne3yP1fZ1oPDLM7Xqs3GQ3ju4ZQedAiojXdVkcxNpzeA, address:81.171.14.70:4035): protocol: read tcp 188.166.225.170:55142->81.171.14.70:4035: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:05:31.413619965 (176h1m29.965090436s) [noise=true , address=81.171.14.70:4030 , piece_id=ZE5OAUTWLWNSMWSFGPEJTCXEN6L3FLWL34QBERTS3JZE7X2I2OHA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:12ZoRwa8wLw66Pjumy1ybp6wrZ79RrSU55uYZpwaY3xi95iSYde, address:81.171.14.70:4030): protocol: read tcp 188.166.225.170:40822->81.171.14.70:4030: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:05:31.414846280 (176h1m29.966316741s) [noise=true , address=81.171.14.70:4029 , piece_id=SIXL2XSCF7IO2BPOUKGH5VVXI6M7O4KJVZARYIRELR2GM4T4RABA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:18yEfU2X4jSrom1JDtQ22cUEvomcg91ohvay2Nsr1Vr8juo9to, address:81.171.14.70:4029): protocol: read tcp 188.166.225.170:56584->81.171.14.70:4029: use of closed network connection; conn closed: op
Nov 16 16:05:31.414946261 (176h1m29.966416722s) [seg_pos={0 0}, upload=155002]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil>
Nov 16 16:05:31.591745995 (176h1m30.143216456s) [upload=155002]: Done uploading segments...
Nov 16 16:05:31.591788596 (176h1m30.143259057s) [upload=155002]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24)
Duration: 5.725642293s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [............................................X...............] duration:
6.228065687s trace:
https://jaeger.dev.storj.io/trace/3e4560ab56ee4602
FAIL (2025-11-16 16:22:25 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.228096436s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [............................................X...............] duration:
6.050126666s trace:
https://jaeger.dev.storj.io/trace/70f81d2d9b012a3a
FAIL (2025-11-16 16:22:04 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 4)
======LOGS======
ddress=45.91.116.5:4297 , piece_id=OSMI3VGAKSCIZMDQ3Y2VZQBUOJKSXM6RWNWSAV57RDTL5TR3NIJA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.462247309 (176h7m35.013717781s) [noise=true , address=45.91.116.7:4011 , piece_id=ORJVRMFIOEXZ6OB6QFQDL764Q2FCWOJT63NBIRI6SOOS2FEIBCFQ, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.463420774 (176h7m35.014891235s) [noise=true , address=145.79.227.9:4367 , piece_id=LOFHPWCHNEZ4E4WANJM5BE2TIWVFH4LGEJJYK4ETB2LZWYCAVGZA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.464113112 (176h7m35.015583573s) [noise=true , address=145.79.227.22:4511 , piece_id=HES465EP3OPD5JBVGVYMGLUOHJKSSMJDXMJVWET5GHRPFZJFIFOA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.464764873 (176h7m35.016235334s) [noise=true , address=45.91.116.8:4048 , piece_id=ULVK52VWCW3KNEPWSDYGRHWHIJWEQU2XNFEPYJFXWLVJI6AGZQOA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.465452753 (176h7m35.016923214s) [noise=true , address=145.79.227.28:4725 , piece_id=CI2LSBGPW7RYH4RGWODGWLUTN353EODNV6IB2YQGHOZ3FPSEEK5A, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.466223270 (176h7m35.017693731s) [noise=true , address=145.79.227.16:4910 , piece_id=3U2SEVMPILIEBRHS3M4YT4Z4PSYAQAIC2SKGYMW4FNODACZLOQCQ, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.466955855 (176h7m35.018426306s) [noise=true , address=145.79.227.4:4186 , piece_id=5PZIK3V72CUQQ6M2HJZFVM5OVX4HPHY7ZHJ2BAK5S3ZQOGM4J5OA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.467675205 (176h7m35.019145666s) [noise=true , address=145.79.227.8:4331 , piece_id=4O27HU7AMR6G4Y43HJYN22UXVLETBDBE7JHL4JUZKRM54SK5E4JA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.468178092 (176h7m35.019648553s) [noise=true , address=145.79.227.8:4329 , piece_id=GO5IAWQYS3UARWYAQAEUIIS456DFWCFWRJQG2KOVVL5KPXWPK4AA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:39.111804678 (176h7m37.663275129s) [upload=155107]: Next returned nil segment
Nov 16 16:11:39.111825767 (176h7m37.663296218s) [upload=155107]: Waiting for error group managing segments...
Nov 16 16:11:39.896671313 (176h7m38.448141765s) [noise=true , address=45.91.116.12:4227 , piece_id=CQA2LHRSFOA7EIBIFVC75EWVHFZMWMY7ODMETEAJRRNYSQFY5TVQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil>
Nov 16 16:11:39.954993420 (176h7m38.506463881s) [noise=true , address=45.91.116.8:4048 , piece_id=ULVK52VWCW3KNEPWSDYGRHWHIJWEQU2XNFEPYJFXWLVJI6AGZQOA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil>
Nov 16 16:11:40.066014029 (176h7m38.61748449s) [noise=true , address=45.91.116.7:4011 , piece_id=ORJVRMFIOEXZ6OB6QFQDL764Q2FCWOJT63NBIRI6SOOS2FEIBCFQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil>
Nov 16 16:11:41.102688250 (176h7m39.654158711s) [noise=true , address=45.91.116.5:4297 , piece_id=OSMI3VGAKSCIZMDQ3Y2VZQBUOJKSXM6RWNWSAV57RDTL5TR3NIJA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil>
Nov 16 16:12:04.264842280 (176h8m2.816312741s) [noise=true , address=145.79.227.17:4972 , piece_id=NV6FSBQM6YUTKYFAJPQCURWCZBHJ3AKJOLLGCRYTJ3KQ54U4GOWQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection
Nov 16 16:12:04.265084832 (176h8m2.816555293s) [noise=true , address=145.79.227.16:4910 , piece_id=3U2SEVMPILIEBRHS3M4YT4Z4PSYAQAIC2SKGYMW4FNODACZLOQCQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection
Nov 16 16:12:04.270267942 (176h8m2.821738423s) [noise=true , address=145.79.227.8:4329 , piece_id=GO5IAWQYS3UARWYAQAEUIIS456DFWCFWRJQG2KOVVL5KPXWPK4AA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: expected piece hash; context deadline exceeded; EOF
Nov 16 16:12:04.270306515 (176h8m2.821776986s) [noise=true , address=145.79.227.9:4367 , piece_id=LOFHPWCHNEZ4E4WANJM5BE2TIWVFH4LGEJJYK4ETB2LZWYCAVGZA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection
Nov 16 16:12:04.270323538 (176h8m2.821794099s) [noise=true , address=145.79.227.28:4725 , piece_id=CI2LSBGPW7RYH4RGWODGWLUTN353EODNV6IB2YQGHOZ3FPSEEK5A, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection
Nov 16 16:12:04.270544138 (176h8m2.822014739s) [noise=true , address=145.79.227.14:4840 , piece_id=RSHGL65RI7HZGU3W3EROFL23BCK2EW3YN6CAD3NO5OVE5QM3QG3Q, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: read tcp 188.166.225.170:42148->145.79.227.14:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:12:04.270571570 (176h8m2.822042121s) [noise=true , address=145.79.227.31:4797 , piece_id=AWSPIPRKSBOKQFZRRB34RUPLYVJGKAOOJQ4U7RRATQJELM3OCFHA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection
Nov 16 16:12:04.270609783 (176h8m2.822080254s) [noise=true , address=145.79.227.22:4511 , piece_id=HES465EP3OPD5JBVGVYMGLUOHJKSSMJDXMJVWET5GHRPFZJFIFOA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: read tcp 188.166.225.170:51004->145.79.227.22:4511: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:12:04.270625552 (176h8m2.822096083s) [noise=true , address=145.79.227.8:4331 , piece_id=4O27HU7AMR6G4Y43HJYN22UXVLETBDBE7JHL4JUZKRM54SK5E4JA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: read tcp 188.166.225.170:44510->145.79.227.8:4331: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:12:04.270742155 (176h8m2.822212616s) [noise=true , address=145.79.227.4:4186 , piece_id=5PZIK3V72CUQQ6M2HJZFVM5OVX4HPHY7ZHJ2BAK5S3ZQOGM4J5OA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection
Nov 16 16:12:04.270762814 (176h8m2.822233275s) [noise=true , address=145.79.227.24:4581 , piece_id=XRCS52EJDB7HWD4RDZK3UNB6VKHOTWWID42RGBJZQHTGG4NDYIPA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: read tcp 188.166.225.170:44130->145.79.227.24:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:12:04.270781470 (176h8m2.822251921s) [noise=true , address=145.79.227.11:4732 , piece_id=FQGAVGZ5CID3LBET3CVK6FSCOP5OUYGS44XYWMHU2RLIT3MBN62A, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: read tcp 188.166.225.170:42420->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:12:04.270825082 (176h8m2.822295544s) [seg_pos={0 0}, upload=155107]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil>
Nov 16 16:12:05.262661603 (176h8m3.814132064s) [upload=155107]: Done uploading segments...
Nov 16 16:12:05.262728080 (176h8m3.814198541s) [upload=155107]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4)
Duration: 6.050159147s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.............................................X..............] duration:
6.359877849s trace:
https://jaeger.dev.storj.io/trace/3e4560ab56ee4602
FAIL (2025-11-16 16:22:04 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.359981398s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [.......X....................................................] duration:
3.543350415s trace:
https://jaeger.dev.storj.io/trace/1026d6559b04863a
FAIL (2025-11-16 16:21:38 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.543376685s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..............................................X.............] duration:
6.75497486s trace:
https://jaeger.dev.storj.io/trace/3e4560ab56ee4602
FAIL (2025-11-16 16:21:28 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.754997965s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [.................................X..........................] duration:
5.826239298s trace:
https://jaeger.dev.storj.io/trace/7dbd01be7ae5d01e
FAIL (2025-11-16 16:21:26 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 24)
======LOGS======
OHA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.079132696 (176h0m30.630603157s) [noise=true , address=81.171.14.70:4040 , piece_id=YG6ZBK6QTKUS3FCFD2EWU7HTFLOEEVG57SIER44E67QDP5HYCHOA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.079319683 (176h0m30.630790144s) [noise=true , address=185.167.204.205:4015 , piece_id=6UIFX7GXE2SZ465VAP56YQILR3A3OHDRKIS345EE6GBDG77KJNZQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.079646465 (176h0m30.631116926s) [noise=true , address=185.167.204.206:4062 , piece_id=PAJX3ZCWM5YBZA7YDBFYXCET5CKWBOFJ47IUWK4CYPZSWK6HYQVQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.080002833 (176h0m30.631473294s) [noise=true , address=81.171.14.70:4031 , piece_id=54T5MOZZ4YJKQFUDOCCFF2SAE7XS2N77H2AOHCBECAWE5OBUXTZQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.080062066 (176h0m30.631532527s) [noise=true , address=185.167.204.206:4067 , piece_id=2BDMQZLCV4UWWP5HB6GF6NCL5UOGNU7VNYGJIIJCKQHRTX5OPGPQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.080438673 (176h0m30.631909134s) [noise=true , address=185.167.204.205:4034 , piece_id=CZFPF35OQ3U4IH46KILFPFMVY6OC36ZUGZ7EYYBOTB3I4TXMPERA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.080675103 (176h0m30.632145564s) [noise=true , address=185.167.204.205:4014 , piece_id=57C2VUYOEULOTXUQHG7A2XOWZLXYFIEYYRYW5ARX5PKTGWKIEOAQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.081315833 (176h0m30.632786294s) [noise=true , address=81.171.14.70:4039 , piece_id=NMZ5357FVIKD2F3GCKY6YNNV73BAID2YBLVNSSM6VD5PHGLBHEIA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.082053738 (176h0m30.633524189s) [noise=true , address=185.167.204.205:4001 , piece_id=TBPYFHMXWO53LJQQDUA3I2MD2WP6BHQXAH3KIF6XXMWK27HXLCHA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.082605429 (176h0m30.63407589s) [noise=true , address=81.171.14.70:4034 , piece_id=NXBDCCCEWSZ2TMSET6OXBNQV6Q5ESXIQJUFK4WJJ3UV7FJTTAUXQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:33.280203414 (176h0m31.831673875s) [upload=155002]: Next returned nil segment
Nov 16 16:04:33.280222810 (176h0m31.831693271s) [upload=155002]: Waiting for error group managing segments...
Nov 16 16:04:33.838289781 (176h0m32.389760242s) [noise=true , address=185.167.204.205:4013 , piece_id=KF32EXUYGKQEDBPRSVAMNZV3ZMBF6DLT3U32BKX5LXLRBXBVTOQA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.848846382 (176h0m32.400316833s) [noise=true , address=185.167.204.206:4039 , piece_id=6HTUHESIRTSKR5EDMWZ2K56OKE5Y22QQM5ARYTUXR7KMKWIOS7GQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.862791868 (176h0m32.414262329s) [noise=true , address=185.167.204.205:4021 , piece_id=MV3TV4EI3HN3J5NY5XYVKT46QJTJEC5PITXERSSZMS2B7W3DSDTA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.879947135 (176h0m32.431417596s) [noise=true , address=185.167.204.205:4001 , piece_id=TBPYFHMXWO53LJQQDUA3I2MD2WP6BHQXAH3KIF6XXMWK27HXLCHA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.886853657 (176h0m32.438324118s) [noise=true , address=185.167.204.206:4067 , piece_id=2BDMQZLCV4UWWP5HB6GF6NCL5UOGNU7VNYGJIIJCKQHRTX5OPGPQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.918801783 (176h0m32.470272324s) [noise=true , address=185.167.204.206:4051 , piece_id=AU2S6M3E2CQEMAS7LKZJIVWITZUHIHCMWQ2QWT7ILGBRTAUCUB7Q, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.929982083 (176h0m32.481452574s) [noise=true , address=185.167.204.205:4015 , piece_id=6UIFX7GXE2SZ465VAP56YQILR3A3OHDRKIS345EE6GBDG77KJNZQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.935247409 (176h0m32.48671786s) [noise=true , address=185.167.204.206:4038 , piece_id=FBNPAJOOXEAT3KLKQIYOWHGHRGAWJABV4LAA3UYX6SGJB3U2LD7A, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.969630813 (176h0m32.521101264s) [noise=true , address=185.167.204.206:4055 , piece_id=B2TTK2ZLIRM3YIKP5HQ3WOL26KBHSXWWX5ZXXCMWU34AH7CHOOCQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.999043600 (176h0m32.550514061s) [noise=true , address=185.167.204.205:4034 , piece_id=CZFPF35OQ3U4IH46KILFPFMVY6OC36ZUGZ7EYYBOTB3I4TXMPERA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.022951616 (176h0m32.574422077s) [noise=true , address=185.167.204.205:4014 , piece_id=57C2VUYOEULOTXUQHG7A2XOWZLXYFIEYYRYW5ARX5PKTGWKIEOAQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.064488381 (176h0m32.615958832s) [noise=true , address=185.167.204.206:4058 , piece_id=POK7HBLE5AT2STCSL2UJ5R67LNVK3PN53NIRIBX4M7LJYC4C2CHQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.122257987 (176h0m32.673728468s) [noise=true , address=185.167.204.205:4029 , piece_id=VF7BTI4K75HTRVJOHIKMDJN3UBHZ7KCQBT3H44XB5G7ZVFD4HXJQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.136282884 (176h0m32.687753345s) [noise=true , address=185.167.204.206:4041 , piece_id=RTQSCXR54KTKBYS6AHEKPO7ZPUHVQQNUNQJ6JI4JLTGKOLUZH3AQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.136821679 (176h0m32.68829215s) [noise=true , address=185.167.204.206:4062 , piece_id=PAJX3ZCWM5YBZA7YDBFYXCET5CKWBOFJ47IUWK4CYPZSWK6HYQVQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.156096042 (176h0m32.707566493s) [noise=true , address=185.167.204.206:4066 , piece_id=CPTBHDHCT6R2M3UA6X2ZWMPJUKAUVV4I2CIATZ2ZOIWH63Q3UT4Q, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.540981809 (176h0m33.0924523s) [noise=true , address=89.149.201.147:4012 , piece_id=NEIYQHVVWRUVJFF5UD3RZSRHUSNSKXBCNGTMIGOH5MDH2AOZISCA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:21.258617296 (176h1m19.810087757s) [noise=true , address=81.171.14.70:4034 , piece_id=NXBDCCCEWSZ2TMSET6OXBNQV6Q5ESXIQJUFK4WJJ3UV7FJTTAUXQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:22.223432573 (176h1m20.774903034s) [noise=true , address=81.171.14.70:4048 , piece_id=5WUEBVU7UFKHYZ23BPIQUMUUBH3ZXXQWBLE7MALJGECCDTTEK5EA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:22.238494073 (176h1m20.789964534s) [noise=true , address=81.171.14.70:4033 , piece_id=LUBQIVFAU7J72K5XDDS5S2ZXQUGFA5AINL4R5FV64KY24VS5KV5Q, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:22.350384206 (176h1m20.901854667s) [noise=true , address=81.171.14.70:4039 , piece_id=NMZ5357FVIKD2F3GCKY6YNNV73BAID2YBLVNSSM6VD5PHGLBHEIA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:23.704484814 (176h1m22.255955275s) [noise=true , address=81.171.14.70:4037 , piece_id=3IJ7J7RBMOORW4YPKIYJ3C6ZQRYBH7WQMLEKEQYG5UJ4TGKE2W2A, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:24.501497158 (176h1m23.052967619s) [noise=true , address=81.171.14.70:4031 , piece_id=54T5MOZZ4YJKQFUDOCCFF2SAE7XS2N77H2AOHCBECAWE5OBUXTZQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:29.810461698 (176h1m28.361932159s) [noise=true , address=81.171.14.70:4025 , piece_id=3HAT6FJSGV6VOHJBMO6L6L7677BOYVCDV3SR26LJZ6HZOZWJJ64A, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:31.413018058 (176h1m29.964488509s) [noise=true , address=81.171.14.70:4040 , piece_id=YG6ZBK6QTKUS3FCFD2EWU7HTFLOEEVG57SIER44E67QDP5HYCHOA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:1gannW154cmVP6N5BowytKfDorZDRwx2M5pi6akqjZFU38PGjV, address:81.171.14.70:4040): protocol: write tcp 188.166.225.170:59244->81.171.14.70:4040: use of closed network connection; write tcp 188.166.225.170:59244->81.171.14.70:4040: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59244->81.171.14.70:4040: use of closed network connection
Nov 16 16:05:31.413111366 (176h1m29.964581837s) [noise=true , address=81.171.14.70:4035 , piece_id=RGMXRH2Y5XBAFISK5LNAF547SAORD6XSWFUKYWOPJINXXODI5KVQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:12eLYJKne3yP1fZ1oPDLM7Xqs3GQ3ju4ZQedAiojXdVkcxNpzeA, address:81.171.14.70:4035): protocol: read tcp 188.166.225.170:55142->81.171.14.70:4035: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:05:31.413619965 (176h1m29.965090436s) [noise=true , address=81.171.14.70:4030 , piece_id=ZE5OAUTWLWNSMWSFGPEJTCXEN6L3FLWL34QBERTS3JZE7X2I2OHA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:12ZoRwa8wLw66Pjumy1ybp6wrZ79RrSU55uYZpwaY3xi95iSYde, address:81.171.14.70:4030): protocol: read tcp 188.166.225.170:40822->81.171.14.70:4030: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:05:31.414846280 (176h1m29.966316741s) [noise=true , address=81.171.14.70:4029 , piece_id=SIXL2XSCF7IO2BPOUKGH5VVXI6M7O4KJVZARYIRELR2GM4T4RABA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:18yEfU2X4jSrom1JDtQ22cUEvomcg91ohvay2Nsr1Vr8juo9to, address:81.171.14.70:4029): protocol: read tcp 188.166.225.170:56584->81.171.14.70:4029: use of closed network connection; conn closed: op
Nov 16 16:05:31.414946261 (176h1m29.966416722s) [seg_pos={0 0}, upload=155002]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil>
Nov 16 16:05:31.591745995 (176h1m30.143216456s) [upload=155002]: Done uploading segments...
Nov 16 16:05:31.591788596 (176h1m30.143259057s) [upload=155002]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24)
Duration: 5.82626688s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [........X...................................................] duration:
3.567417733s trace:
https://jaeger.dev.storj.io/trace/1026d6559b04863a
FAIL (2025-11-16 16:21:08 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.567471915s
core
uplink
us-select-1
us1-placement-20
|