Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [............................................X...............] duration:
6.228065687s trace:
https://jaeger.dev.storj.io/trace/3e4560ab56ee4602
FAIL (2025-11-16 16:22:25 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.228096436s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [............................................X...............] duration:
6.050126666s trace:
https://jaeger.dev.storj.io/trace/70f81d2d9b012a3a
FAIL (2025-11-16 16:22:04 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 4)
======LOGS======
ddress=45.91.116.5:4297 , piece_id=OSMI3VGAKSCIZMDQ3Y2VZQBUOJKSXM6RWNWSAV57RDTL5TR3NIJA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.462247309 (176h7m35.013717781s) [noise=true , address=45.91.116.7:4011 , piece_id=ORJVRMFIOEXZ6OB6QFQDL764Q2FCWOJT63NBIRI6SOOS2FEIBCFQ, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.463420774 (176h7m35.014891235s) [noise=true , address=145.79.227.9:4367 , piece_id=LOFHPWCHNEZ4E4WANJM5BE2TIWVFH4LGEJJYK4ETB2LZWYCAVGZA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.464113112 (176h7m35.015583573s) [noise=true , address=145.79.227.22:4511 , piece_id=HES465EP3OPD5JBVGVYMGLUOHJKSSMJDXMJVWET5GHRPFZJFIFOA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.464764873 (176h7m35.016235334s) [noise=true , address=45.91.116.8:4048 , piece_id=ULVK52VWCW3KNEPWSDYGRHWHIJWEQU2XNFEPYJFXWLVJI6AGZQOA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.465452753 (176h7m35.016923214s) [noise=true , address=145.79.227.28:4725 , piece_id=CI2LSBGPW7RYH4RGWODGWLUTN353EODNV6IB2YQGHOZ3FPSEEK5A, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.466223270 (176h7m35.017693731s) [noise=true , address=145.79.227.16:4910 , piece_id=3U2SEVMPILIEBRHS3M4YT4Z4PSYAQAIC2SKGYMW4FNODACZLOQCQ, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.466955855 (176h7m35.018426306s) [noise=true , address=145.79.227.4:4186 , piece_id=5PZIK3V72CUQQ6M2HJZFVM5OVX4HPHY7ZHJ2BAK5S3ZQOGM4J5OA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.467675205 (176h7m35.019145666s) [noise=true , address=145.79.227.8:4331 , piece_id=4O27HU7AMR6G4Y43HJYN22UXVLETBDBE7JHL4JUZKRM54SK5E4JA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.468178092 (176h7m35.019648553s) [noise=true , address=145.79.227.8:4329 , piece_id=GO5IAWQYS3UARWYAQAEUIIS456DFWCFWRJQG2KOVVL5KPXWPK4AA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:39.111804678 (176h7m37.663275129s) [upload=155107]: Next returned nil segment
Nov 16 16:11:39.111825767 (176h7m37.663296218s) [upload=155107]: Waiting for error group managing segments...
Nov 16 16:11:39.896671313 (176h7m38.448141765s) [noise=true , address=45.91.116.12:4227 , piece_id=CQA2LHRSFOA7EIBIFVC75EWVHFZMWMY7ODMETEAJRRNYSQFY5TVQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil>
Nov 16 16:11:39.954993420 (176h7m38.506463881s) [noise=true , address=45.91.116.8:4048 , piece_id=ULVK52VWCW3KNEPWSDYGRHWHIJWEQU2XNFEPYJFXWLVJI6AGZQOA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil>
Nov 16 16:11:40.066014029 (176h7m38.61748449s) [noise=true , address=45.91.116.7:4011 , piece_id=ORJVRMFIOEXZ6OB6QFQDL764Q2FCWOJT63NBIRI6SOOS2FEIBCFQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil>
Nov 16 16:11:41.102688250 (176h7m39.654158711s) [noise=true , address=45.91.116.5:4297 , piece_id=OSMI3VGAKSCIZMDQ3Y2VZQBUOJKSXM6RWNWSAV57RDTL5TR3NIJA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil>
Nov 16 16:12:04.264842280 (176h8m2.816312741s) [noise=true , address=145.79.227.17:4972 , piece_id=NV6FSBQM6YUTKYFAJPQCURWCZBHJ3AKJOLLGCRYTJ3KQ54U4GOWQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection
Nov 16 16:12:04.265084832 (176h8m2.816555293s) [noise=true , address=145.79.227.16:4910 , piece_id=3U2SEVMPILIEBRHS3M4YT4Z4PSYAQAIC2SKGYMW4FNODACZLOQCQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection
Nov 16 16:12:04.270267942 (176h8m2.821738423s) [noise=true , address=145.79.227.8:4329 , piece_id=GO5IAWQYS3UARWYAQAEUIIS456DFWCFWRJQG2KOVVL5KPXWPK4AA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: expected piece hash; context deadline exceeded; EOF
Nov 16 16:12:04.270306515 (176h8m2.821776986s) [noise=true , address=145.79.227.9:4367 , piece_id=LOFHPWCHNEZ4E4WANJM5BE2TIWVFH4LGEJJYK4ETB2LZWYCAVGZA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection
Nov 16 16:12:04.270323538 (176h8m2.821794099s) [noise=true , address=145.79.227.28:4725 , piece_id=CI2LSBGPW7RYH4RGWODGWLUTN353EODNV6IB2YQGHOZ3FPSEEK5A, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection
Nov 16 16:12:04.270544138 (176h8m2.822014739s) [noise=true , address=145.79.227.14:4840 , piece_id=RSHGL65RI7HZGU3W3EROFL23BCK2EW3YN6CAD3NO5OVE5QM3QG3Q, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: read tcp 188.166.225.170:42148->145.79.227.14:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:12:04.270571570 (176h8m2.822042121s) [noise=true , address=145.79.227.31:4797 , piece_id=AWSPIPRKSBOKQFZRRB34RUPLYVJGKAOOJQ4U7RRATQJELM3OCFHA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection
Nov 16 16:12:04.270609783 (176h8m2.822080254s) [noise=true , address=145.79.227.22:4511 , piece_id=HES465EP3OPD5JBVGVYMGLUOHJKSSMJDXMJVWET5GHRPFZJFIFOA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: read tcp 188.166.225.170:51004->145.79.227.22:4511: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:12:04.270625552 (176h8m2.822096083s) [noise=true , address=145.79.227.8:4331 , piece_id=4O27HU7AMR6G4Y43HJYN22UXVLETBDBE7JHL4JUZKRM54SK5E4JA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: read tcp 188.166.225.170:44510->145.79.227.8:4331: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:12:04.270742155 (176h8m2.822212616s) [noise=true , address=145.79.227.4:4186 , piece_id=5PZIK3V72CUQQ6M2HJZFVM5OVX4HPHY7ZHJ2BAK5S3ZQOGM4J5OA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection
Nov 16 16:12:04.270762814 (176h8m2.822233275s) [noise=true , address=145.79.227.24:4581 , piece_id=XRCS52EJDB7HWD4RDZK3UNB6VKHOTWWID42RGBJZQHTGG4NDYIPA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: read tcp 188.166.225.170:44130->145.79.227.24:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:12:04.270781470 (176h8m2.822251921s) [noise=true , address=145.79.227.11:4732 , piece_id=FQGAVGZ5CID3LBET3CVK6FSCOP5OUYGS44XYWMHU2RLIT3MBN62A, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: read tcp 188.166.225.170:42420->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:12:04.270825082 (176h8m2.822295544s) [seg_pos={0 0}, upload=155107]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil>
Nov 16 16:12:05.262661603 (176h8m3.814132064s) [upload=155107]: Done uploading segments...
Nov 16 16:12:05.262728080 (176h8m3.814198541s) [upload=155107]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4)
Duration: 6.050159147s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.............................................X..............] duration:
6.359877849s trace:
https://jaeger.dev.storj.io/trace/3e4560ab56ee4602
FAIL (2025-11-16 16:22:04 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.359981398s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [.......X....................................................] duration:
3.543350415s trace:
https://jaeger.dev.storj.io/trace/1026d6559b04863a
FAIL (2025-11-16 16:21:38 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.543376685s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..............................................X.............] duration:
6.75497486s trace:
https://jaeger.dev.storj.io/trace/3e4560ab56ee4602
FAIL (2025-11-16 16:21:28 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.754997965s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [.................................X..........................] duration:
5.826239298s trace:
https://jaeger.dev.storj.io/trace/7dbd01be7ae5d01e
FAIL (2025-11-16 16:21:26 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 24)
======LOGS======
OHA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.079132696 (176h0m30.630603157s) [noise=true , address=81.171.14.70:4040 , piece_id=YG6ZBK6QTKUS3FCFD2EWU7HTFLOEEVG57SIER44E67QDP5HYCHOA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.079319683 (176h0m30.630790144s) [noise=true , address=185.167.204.205:4015 , piece_id=6UIFX7GXE2SZ465VAP56YQILR3A3OHDRKIS345EE6GBDG77KJNZQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.079646465 (176h0m30.631116926s) [noise=true , address=185.167.204.206:4062 , piece_id=PAJX3ZCWM5YBZA7YDBFYXCET5CKWBOFJ47IUWK4CYPZSWK6HYQVQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.080002833 (176h0m30.631473294s) [noise=true , address=81.171.14.70:4031 , piece_id=54T5MOZZ4YJKQFUDOCCFF2SAE7XS2N77H2AOHCBECAWE5OBUXTZQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.080062066 (176h0m30.631532527s) [noise=true , address=185.167.204.206:4067 , piece_id=2BDMQZLCV4UWWP5HB6GF6NCL5UOGNU7VNYGJIIJCKQHRTX5OPGPQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.080438673 (176h0m30.631909134s) [noise=true , address=185.167.204.205:4034 , piece_id=CZFPF35OQ3U4IH46KILFPFMVY6OC36ZUGZ7EYYBOTB3I4TXMPERA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.080675103 (176h0m30.632145564s) [noise=true , address=185.167.204.205:4014 , piece_id=57C2VUYOEULOTXUQHG7A2XOWZLXYFIEYYRYW5ARX5PKTGWKIEOAQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.081315833 (176h0m30.632786294s) [noise=true , address=81.171.14.70:4039 , piece_id=NMZ5357FVIKD2F3GCKY6YNNV73BAID2YBLVNSSM6VD5PHGLBHEIA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.082053738 (176h0m30.633524189s) [noise=true , address=185.167.204.205:4001 , piece_id=TBPYFHMXWO53LJQQDUA3I2MD2WP6BHQXAH3KIF6XXMWK27HXLCHA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.082605429 (176h0m30.63407589s) [noise=true , address=81.171.14.70:4034 , piece_id=NXBDCCCEWSZ2TMSET6OXBNQV6Q5ESXIQJUFK4WJJ3UV7FJTTAUXQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:33.280203414 (176h0m31.831673875s) [upload=155002]: Next returned nil segment
Nov 16 16:04:33.280222810 (176h0m31.831693271s) [upload=155002]: Waiting for error group managing segments...
Nov 16 16:04:33.838289781 (176h0m32.389760242s) [noise=true , address=185.167.204.205:4013 , piece_id=KF32EXUYGKQEDBPRSVAMNZV3ZMBF6DLT3U32BKX5LXLRBXBVTOQA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.848846382 (176h0m32.400316833s) [noise=true , address=185.167.204.206:4039 , piece_id=6HTUHESIRTSKR5EDMWZ2K56OKE5Y22QQM5ARYTUXR7KMKWIOS7GQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.862791868 (176h0m32.414262329s) [noise=true , address=185.167.204.205:4021 , piece_id=MV3TV4EI3HN3J5NY5XYVKT46QJTJEC5PITXERSSZMS2B7W3DSDTA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.879947135 (176h0m32.431417596s) [noise=true , address=185.167.204.205:4001 , piece_id=TBPYFHMXWO53LJQQDUA3I2MD2WP6BHQXAH3KIF6XXMWK27HXLCHA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.886853657 (176h0m32.438324118s) [noise=true , address=185.167.204.206:4067 , piece_id=2BDMQZLCV4UWWP5HB6GF6NCL5UOGNU7VNYGJIIJCKQHRTX5OPGPQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.918801783 (176h0m32.470272324s) [noise=true , address=185.167.204.206:4051 , piece_id=AU2S6M3E2CQEMAS7LKZJIVWITZUHIHCMWQ2QWT7ILGBRTAUCUB7Q, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.929982083 (176h0m32.481452574s) [noise=true , address=185.167.204.205:4015 , piece_id=6UIFX7GXE2SZ465VAP56YQILR3A3OHDRKIS345EE6GBDG77KJNZQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.935247409 (176h0m32.48671786s) [noise=true , address=185.167.204.206:4038 , piece_id=FBNPAJOOXEAT3KLKQIYOWHGHRGAWJABV4LAA3UYX6SGJB3U2LD7A, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.969630813 (176h0m32.521101264s) [noise=true , address=185.167.204.206:4055 , piece_id=B2TTK2ZLIRM3YIKP5HQ3WOL26KBHSXWWX5ZXXCMWU34AH7CHOOCQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.999043600 (176h0m32.550514061s) [noise=true , address=185.167.204.205:4034 , piece_id=CZFPF35OQ3U4IH46KILFPFMVY6OC36ZUGZ7EYYBOTB3I4TXMPERA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.022951616 (176h0m32.574422077s) [noise=true , address=185.167.204.205:4014 , piece_id=57C2VUYOEULOTXUQHG7A2XOWZLXYFIEYYRYW5ARX5PKTGWKIEOAQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.064488381 (176h0m32.615958832s) [noise=true , address=185.167.204.206:4058 , piece_id=POK7HBLE5AT2STCSL2UJ5R67LNVK3PN53NIRIBX4M7LJYC4C2CHQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.122257987 (176h0m32.673728468s) [noise=true , address=185.167.204.205:4029 , piece_id=VF7BTI4K75HTRVJOHIKMDJN3UBHZ7KCQBT3H44XB5G7ZVFD4HXJQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.136282884 (176h0m32.687753345s) [noise=true , address=185.167.204.206:4041 , piece_id=RTQSCXR54KTKBYS6AHEKPO7ZPUHVQQNUNQJ6JI4JLTGKOLUZH3AQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.136821679 (176h0m32.68829215s) [noise=true , address=185.167.204.206:4062 , piece_id=PAJX3ZCWM5YBZA7YDBFYXCET5CKWBOFJ47IUWK4CYPZSWK6HYQVQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.156096042 (176h0m32.707566493s) [noise=true , address=185.167.204.206:4066 , piece_id=CPTBHDHCT6R2M3UA6X2ZWMPJUKAUVV4I2CIATZ2ZOIWH63Q3UT4Q, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.540981809 (176h0m33.0924523s) [noise=true , address=89.149.201.147:4012 , piece_id=NEIYQHVVWRUVJFF5UD3RZSRHUSNSKXBCNGTMIGOH5MDH2AOZISCA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:21.258617296 (176h1m19.810087757s) [noise=true , address=81.171.14.70:4034 , piece_id=NXBDCCCEWSZ2TMSET6OXBNQV6Q5ESXIQJUFK4WJJ3UV7FJTTAUXQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:22.223432573 (176h1m20.774903034s) [noise=true , address=81.171.14.70:4048 , piece_id=5WUEBVU7UFKHYZ23BPIQUMUUBH3ZXXQWBLE7MALJGECCDTTEK5EA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:22.238494073 (176h1m20.789964534s) [noise=true , address=81.171.14.70:4033 , piece_id=LUBQIVFAU7J72K5XDDS5S2ZXQUGFA5AINL4R5FV64KY24VS5KV5Q, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:22.350384206 (176h1m20.901854667s) [noise=true , address=81.171.14.70:4039 , piece_id=NMZ5357FVIKD2F3GCKY6YNNV73BAID2YBLVNSSM6VD5PHGLBHEIA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:23.704484814 (176h1m22.255955275s) [noise=true , address=81.171.14.70:4037 , piece_id=3IJ7J7RBMOORW4YPKIYJ3C6ZQRYBH7WQMLEKEQYG5UJ4TGKE2W2A, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:24.501497158 (176h1m23.052967619s) [noise=true , address=81.171.14.70:4031 , piece_id=54T5MOZZ4YJKQFUDOCCFF2SAE7XS2N77H2AOHCBECAWE5OBUXTZQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:29.810461698 (176h1m28.361932159s) [noise=true , address=81.171.14.70:4025 , piece_id=3HAT6FJSGV6VOHJBMO6L6L7677BOYVCDV3SR26LJZ6HZOZWJJ64A, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:31.413018058 (176h1m29.964488509s) [noise=true , address=81.171.14.70:4040 , piece_id=YG6ZBK6QTKUS3FCFD2EWU7HTFLOEEVG57SIER44E67QDP5HYCHOA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:1gannW154cmVP6N5BowytKfDorZDRwx2M5pi6akqjZFU38PGjV, address:81.171.14.70:4040): protocol: write tcp 188.166.225.170:59244->81.171.14.70:4040: use of closed network connection; write tcp 188.166.225.170:59244->81.171.14.70:4040: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59244->81.171.14.70:4040: use of closed network connection
Nov 16 16:05:31.413111366 (176h1m29.964581837s) [noise=true , address=81.171.14.70:4035 , piece_id=RGMXRH2Y5XBAFISK5LNAF547SAORD6XSWFUKYWOPJINXXODI5KVQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:12eLYJKne3yP1fZ1oPDLM7Xqs3GQ3ju4ZQedAiojXdVkcxNpzeA, address:81.171.14.70:4035): protocol: read tcp 188.166.225.170:55142->81.171.14.70:4035: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:05:31.413619965 (176h1m29.965090436s) [noise=true , address=81.171.14.70:4030 , piece_id=ZE5OAUTWLWNSMWSFGPEJTCXEN6L3FLWL34QBERTS3JZE7X2I2OHA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:12ZoRwa8wLw66Pjumy1ybp6wrZ79RrSU55uYZpwaY3xi95iSYde, address:81.171.14.70:4030): protocol: read tcp 188.166.225.170:40822->81.171.14.70:4030: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:05:31.414846280 (176h1m29.966316741s) [noise=true , address=81.171.14.70:4029 , piece_id=SIXL2XSCF7IO2BPOUKGH5VVXI6M7O4KJVZARYIRELR2GM4T4RABA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:18yEfU2X4jSrom1JDtQ22cUEvomcg91ohvay2Nsr1Vr8juo9to, address:81.171.14.70:4029): protocol: read tcp 188.166.225.170:56584->81.171.14.70:4029: use of closed network connection; conn closed: op
Nov 16 16:05:31.414946261 (176h1m29.966416722s) [seg_pos={0 0}, upload=155002]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil>
Nov 16 16:05:31.591745995 (176h1m30.143216456s) [upload=155002]: Done uploading segments...
Nov 16 16:05:31.591788596 (176h1m30.143259057s) [upload=155002]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24)
Duration: 5.82626688s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [........X...................................................] duration:
3.567417733s trace:
https://jaeger.dev.storj.io/trace/1026d6559b04863a
FAIL (2025-11-16 16:21:08 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.567471915s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [..................................X.........................] duration:
5.438025813s trace:
https://jaeger.dev.storj.io/trace/7dbd01be7ae5d01e
FAIL (2025-11-16 16:20:59 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 24)
======LOGS======
OHA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.079132696 (176h0m30.630603157s) [noise=true , address=81.171.14.70:4040 , piece_id=YG6ZBK6QTKUS3FCFD2EWU7HTFLOEEVG57SIER44E67QDP5HYCHOA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.079319683 (176h0m30.630790144s) [noise=true , address=185.167.204.205:4015 , piece_id=6UIFX7GXE2SZ465VAP56YQILR3A3OHDRKIS345EE6GBDG77KJNZQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.079646465 (176h0m30.631116926s) [noise=true , address=185.167.204.206:4062 , piece_id=PAJX3ZCWM5YBZA7YDBFYXCET5CKWBOFJ47IUWK4CYPZSWK6HYQVQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.080002833 (176h0m30.631473294s) [noise=true , address=81.171.14.70:4031 , piece_id=54T5MOZZ4YJKQFUDOCCFF2SAE7XS2N77H2AOHCBECAWE5OBUXTZQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.080062066 (176h0m30.631532527s) [noise=true , address=185.167.204.206:4067 , piece_id=2BDMQZLCV4UWWP5HB6GF6NCL5UOGNU7VNYGJIIJCKQHRTX5OPGPQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.080438673 (176h0m30.631909134s) [noise=true , address=185.167.204.205:4034 , piece_id=CZFPF35OQ3U4IH46KILFPFMVY6OC36ZUGZ7EYYBOTB3I4TXMPERA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.080675103 (176h0m30.632145564s) [noise=true , address=185.167.204.205:4014 , piece_id=57C2VUYOEULOTXUQHG7A2XOWZLXYFIEYYRYW5ARX5PKTGWKIEOAQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.081315833 (176h0m30.632786294s) [noise=true , address=81.171.14.70:4039 , piece_id=NMZ5357FVIKD2F3GCKY6YNNV73BAID2YBLVNSSM6VD5PHGLBHEIA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.082053738 (176h0m30.633524189s) [noise=true , address=185.167.204.205:4001 , piece_id=TBPYFHMXWO53LJQQDUA3I2MD2WP6BHQXAH3KIF6XXMWK27HXLCHA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.082605429 (176h0m30.63407589s) [noise=true , address=81.171.14.70:4034 , piece_id=NXBDCCCEWSZ2TMSET6OXBNQV6Q5ESXIQJUFK4WJJ3UV7FJTTAUXQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:33.280203414 (176h0m31.831673875s) [upload=155002]: Next returned nil segment
Nov 16 16:04:33.280222810 (176h0m31.831693271s) [upload=155002]: Waiting for error group managing segments...
Nov 16 16:04:33.838289781 (176h0m32.389760242s) [noise=true , address=185.167.204.205:4013 , piece_id=KF32EXUYGKQEDBPRSVAMNZV3ZMBF6DLT3U32BKX5LXLRBXBVTOQA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.848846382 (176h0m32.400316833s) [noise=true , address=185.167.204.206:4039 , piece_id=6HTUHESIRTSKR5EDMWZ2K56OKE5Y22QQM5ARYTUXR7KMKWIOS7GQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.862791868 (176h0m32.414262329s) [noise=true , address=185.167.204.205:4021 , piece_id=MV3TV4EI3HN3J5NY5XYVKT46QJTJEC5PITXERSSZMS2B7W3DSDTA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.879947135 (176h0m32.431417596s) [noise=true , address=185.167.204.205:4001 , piece_id=TBPYFHMXWO53LJQQDUA3I2MD2WP6BHQXAH3KIF6XXMWK27HXLCHA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.886853657 (176h0m32.438324118s) [noise=true , address=185.167.204.206:4067 , piece_id=2BDMQZLCV4UWWP5HB6GF6NCL5UOGNU7VNYGJIIJCKQHRTX5OPGPQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.918801783 (176h0m32.470272324s) [noise=true , address=185.167.204.206:4051 , piece_id=AU2S6M3E2CQEMAS7LKZJIVWITZUHIHCMWQ2QWT7ILGBRTAUCUB7Q, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.929982083 (176h0m32.481452574s) [noise=true , address=185.167.204.205:4015 , piece_id=6UIFX7GXE2SZ465VAP56YQILR3A3OHDRKIS345EE6GBDG77KJNZQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.935247409 (176h0m32.48671786s) [noise=true , address=185.167.204.206:4038 , piece_id=FBNPAJOOXEAT3KLKQIYOWHGHRGAWJABV4LAA3UYX6SGJB3U2LD7A, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.969630813 (176h0m32.521101264s) [noise=true , address=185.167.204.206:4055 , piece_id=B2TTK2ZLIRM3YIKP5HQ3WOL26KBHSXWWX5ZXXCMWU34AH7CHOOCQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.999043600 (176h0m32.550514061s) [noise=true , address=185.167.204.205:4034 , piece_id=CZFPF35OQ3U4IH46KILFPFMVY6OC36ZUGZ7EYYBOTB3I4TXMPERA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.022951616 (176h0m32.574422077s) [noise=true , address=185.167.204.205:4014 , piece_id=57C2VUYOEULOTXUQHG7A2XOWZLXYFIEYYRYW5ARX5PKTGWKIEOAQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.064488381 (176h0m32.615958832s) [noise=true , address=185.167.204.206:4058 , piece_id=POK7HBLE5AT2STCSL2UJ5R67LNVK3PN53NIRIBX4M7LJYC4C2CHQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.122257987 (176h0m32.673728468s) [noise=true , address=185.167.204.205:4029 , piece_id=VF7BTI4K75HTRVJOHIKMDJN3UBHZ7KCQBT3H44XB5G7ZVFD4HXJQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.136282884 (176h0m32.687753345s) [noise=true , address=185.167.204.206:4041 , piece_id=RTQSCXR54KTKBYS6AHEKPO7ZPUHVQQNUNQJ6JI4JLTGKOLUZH3AQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.136821679 (176h0m32.68829215s) [noise=true , address=185.167.204.206:4062 , piece_id=PAJX3ZCWM5YBZA7YDBFYXCET5CKWBOFJ47IUWK4CYPZSWK6HYQVQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.156096042 (176h0m32.707566493s) [noise=true , address=185.167.204.206:4066 , piece_id=CPTBHDHCT6R2M3UA6X2ZWMPJUKAUVV4I2CIATZ2ZOIWH63Q3UT4Q, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.540981809 (176h0m33.0924523s) [noise=true , address=89.149.201.147:4012 , piece_id=NEIYQHVVWRUVJFF5UD3RZSRHUSNSKXBCNGTMIGOH5MDH2AOZISCA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:21.258617296 (176h1m19.810087757s) [noise=true , address=81.171.14.70:4034 , piece_id=NXBDCCCEWSZ2TMSET6OXBNQV6Q5ESXIQJUFK4WJJ3UV7FJTTAUXQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:22.223432573 (176h1m20.774903034s) [noise=true , address=81.171.14.70:4048 , piece_id=5WUEBVU7UFKHYZ23BPIQUMUUBH3ZXXQWBLE7MALJGECCDTTEK5EA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:22.238494073 (176h1m20.789964534s) [noise=true , address=81.171.14.70:4033 , piece_id=LUBQIVFAU7J72K5XDDS5S2ZXQUGFA5AINL4R5FV64KY24VS5KV5Q, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:22.350384206 (176h1m20.901854667s) [noise=true , address=81.171.14.70:4039 , piece_id=NMZ5357FVIKD2F3GCKY6YNNV73BAID2YBLVNSSM6VD5PHGLBHEIA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:23.704484814 (176h1m22.255955275s) [noise=true , address=81.171.14.70:4037 , piece_id=3IJ7J7RBMOORW4YPKIYJ3C6ZQRYBH7WQMLEKEQYG5UJ4TGKE2W2A, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:24.501497158 (176h1m23.052967619s) [noise=true , address=81.171.14.70:4031 , piece_id=54T5MOZZ4YJKQFUDOCCFF2SAE7XS2N77H2AOHCBECAWE5OBUXTZQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:29.810461698 (176h1m28.361932159s) [noise=true , address=81.171.14.70:4025 , piece_id=3HAT6FJSGV6VOHJBMO6L6L7677BOYVCDV3SR26LJZ6HZOZWJJ64A, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:31.413018058 (176h1m29.964488509s) [noise=true , address=81.171.14.70:4040 , piece_id=YG6ZBK6QTKUS3FCFD2EWU7HTFLOEEVG57SIER44E67QDP5HYCHOA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:1gannW154cmVP6N5BowytKfDorZDRwx2M5pi6akqjZFU38PGjV, address:81.171.14.70:4040): protocol: write tcp 188.166.225.170:59244->81.171.14.70:4040: use of closed network connection; write tcp 188.166.225.170:59244->81.171.14.70:4040: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59244->81.171.14.70:4040: use of closed network connection
Nov 16 16:05:31.413111366 (176h1m29.964581837s) [noise=true , address=81.171.14.70:4035 , piece_id=RGMXRH2Y5XBAFISK5LNAF547SAORD6XSWFUKYWOPJINXXODI5KVQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:12eLYJKne3yP1fZ1oPDLM7Xqs3GQ3ju4ZQedAiojXdVkcxNpzeA, address:81.171.14.70:4035): protocol: read tcp 188.166.225.170:55142->81.171.14.70:4035: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:05:31.413619965 (176h1m29.965090436s) [noise=true , address=81.171.14.70:4030 , piece_id=ZE5OAUTWLWNSMWSFGPEJTCXEN6L3FLWL34QBERTS3JZE7X2I2OHA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:12ZoRwa8wLw66Pjumy1ybp6wrZ79RrSU55uYZpwaY3xi95iSYde, address:81.171.14.70:4030): protocol: read tcp 188.166.225.170:40822->81.171.14.70:4030: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:05:31.414846280 (176h1m29.966316741s) [noise=true , address=81.171.14.70:4029 , piece_id=SIXL2XSCF7IO2BPOUKGH5VVXI6M7O4KJVZARYIRELR2GM4T4RABA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:18yEfU2X4jSrom1JDtQ22cUEvomcg91ohvay2Nsr1Vr8juo9to, address:81.171.14.70:4029): protocol: read tcp 188.166.225.170:56584->81.171.14.70:4029: use of closed network connection; conn closed: op
Nov 16 16:05:31.414946261 (176h1m29.966416722s) [seg_pos={0 0}, upload=155002]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil>
Nov 16 16:05:31.591745995 (176h1m30.143216456s) [upload=155002]: Done uploading segments...
Nov 16 16:05:31.591788596 (176h1m30.143259057s) [upload=155002]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24)
Duration: 5.438043847s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.............................................X..............] duration:
5.978964228s trace:
https://jaeger.dev.storj.io/trace/70f81d2d9b012a3a
FAIL (2025-11-16 16:20:54 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 4)
======LOGS======
ddress=45.91.116.5:4297 , piece_id=OSMI3VGAKSCIZMDQ3Y2VZQBUOJKSXM6RWNWSAV57RDTL5TR3NIJA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.462247309 (176h7m35.013717781s) [noise=true , address=45.91.116.7:4011 , piece_id=ORJVRMFIOEXZ6OB6QFQDL764Q2FCWOJT63NBIRI6SOOS2FEIBCFQ, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.463420774 (176h7m35.014891235s) [noise=true , address=145.79.227.9:4367 , piece_id=LOFHPWCHNEZ4E4WANJM5BE2TIWVFH4LGEJJYK4ETB2LZWYCAVGZA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.464113112 (176h7m35.015583573s) [noise=true , address=145.79.227.22:4511 , piece_id=HES465EP3OPD5JBVGVYMGLUOHJKSSMJDXMJVWET5GHRPFZJFIFOA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.464764873 (176h7m35.016235334s) [noise=true , address=45.91.116.8:4048 , piece_id=ULVK52VWCW3KNEPWSDYGRHWHIJWEQU2XNFEPYJFXWLVJI6AGZQOA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.465452753 (176h7m35.016923214s) [noise=true , address=145.79.227.28:4725 , piece_id=CI2LSBGPW7RYH4RGWODGWLUTN353EODNV6IB2YQGHOZ3FPSEEK5A, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.466223270 (176h7m35.017693731s) [noise=true , address=145.79.227.16:4910 , piece_id=3U2SEVMPILIEBRHS3M4YT4Z4PSYAQAIC2SKGYMW4FNODACZLOQCQ, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.466955855 (176h7m35.018426306s) [noise=true , address=145.79.227.4:4186 , piece_id=5PZIK3V72CUQQ6M2HJZFVM5OVX4HPHY7ZHJ2BAK5S3ZQOGM4J5OA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.467675205 (176h7m35.019145666s) [noise=true , address=145.79.227.8:4331 , piece_id=4O27HU7AMR6G4Y43HJYN22UXVLETBDBE7JHL4JUZKRM54SK5E4JA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.468178092 (176h7m35.019648553s) [noise=true , address=145.79.227.8:4329 , piece_id=GO5IAWQYS3UARWYAQAEUIIS456DFWCFWRJQG2KOVVL5KPXWPK4AA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:39.111804678 (176h7m37.663275129s) [upload=155107]: Next returned nil segment
Nov 16 16:11:39.111825767 (176h7m37.663296218s) [upload=155107]: Waiting for error group managing segments...
Nov 16 16:11:39.896671313 (176h7m38.448141765s) [noise=true , address=45.91.116.12:4227 , piece_id=CQA2LHRSFOA7EIBIFVC75EWVHFZMWMY7ODMETEAJRRNYSQFY5TVQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil>
Nov 16 16:11:39.954993420 (176h7m38.506463881s) [noise=true , address=45.91.116.8:4048 , piece_id=ULVK52VWCW3KNEPWSDYGRHWHIJWEQU2XNFEPYJFXWLVJI6AGZQOA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil>
Nov 16 16:11:40.066014029 (176h7m38.61748449s) [noise=true , address=45.91.116.7:4011 , piece_id=ORJVRMFIOEXZ6OB6QFQDL764Q2FCWOJT63NBIRI6SOOS2FEIBCFQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil>
Nov 16 16:11:41.102688250 (176h7m39.654158711s) [noise=true , address=45.91.116.5:4297 , piece_id=OSMI3VGAKSCIZMDQ3Y2VZQBUOJKSXM6RWNWSAV57RDTL5TR3NIJA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil>
Nov 16 16:12:04.264842280 (176h8m2.816312741s) [noise=true , address=145.79.227.17:4972 , piece_id=NV6FSBQM6YUTKYFAJPQCURWCZBHJ3AKJOLLGCRYTJ3KQ54U4GOWQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection
Nov 16 16:12:04.265084832 (176h8m2.816555293s) [noise=true , address=145.79.227.16:4910 , piece_id=3U2SEVMPILIEBRHS3M4YT4Z4PSYAQAIC2SKGYMW4FNODACZLOQCQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection
Nov 16 16:12:04.270267942 (176h8m2.821738423s) [noise=true , address=145.79.227.8:4329 , piece_id=GO5IAWQYS3UARWYAQAEUIIS456DFWCFWRJQG2KOVVL5KPXWPK4AA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: expected piece hash; context deadline exceeded; EOF
Nov 16 16:12:04.270306515 (176h8m2.821776986s) [noise=true , address=145.79.227.9:4367 , piece_id=LOFHPWCHNEZ4E4WANJM5BE2TIWVFH4LGEJJYK4ETB2LZWYCAVGZA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection
Nov 16 16:12:04.270323538 (176h8m2.821794099s) [noise=true , address=145.79.227.28:4725 , piece_id=CI2LSBGPW7RYH4RGWODGWLUTN353EODNV6IB2YQGHOZ3FPSEEK5A, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection
Nov 16 16:12:04.270544138 (176h8m2.822014739s) [noise=true , address=145.79.227.14:4840 , piece_id=RSHGL65RI7HZGU3W3EROFL23BCK2EW3YN6CAD3NO5OVE5QM3QG3Q, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: read tcp 188.166.225.170:42148->145.79.227.14:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:12:04.270571570 (176h8m2.822042121s) [noise=true , address=145.79.227.31:4797 , piece_id=AWSPIPRKSBOKQFZRRB34RUPLYVJGKAOOJQ4U7RRATQJELM3OCFHA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection
Nov 16 16:12:04.270609783 (176h8m2.822080254s) [noise=true , address=145.79.227.22:4511 , piece_id=HES465EP3OPD5JBVGVYMGLUOHJKSSMJDXMJVWET5GHRPFZJFIFOA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: read tcp 188.166.225.170:51004->145.79.227.22:4511: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:12:04.270625552 (176h8m2.822096083s) [noise=true , address=145.79.227.8:4331 , piece_id=4O27HU7AMR6G4Y43HJYN22UXVLETBDBE7JHL4JUZKRM54SK5E4JA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: read tcp 188.166.225.170:44510->145.79.227.8:4331: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:12:04.270742155 (176h8m2.822212616s) [noise=true , address=145.79.227.4:4186 , piece_id=5PZIK3V72CUQQ6M2HJZFVM5OVX4HPHY7ZHJ2BAK5S3ZQOGM4J5OA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection
Nov 16 16:12:04.270762814 (176h8m2.822233275s) [noise=true , address=145.79.227.24:4581 , piece_id=XRCS52EJDB7HWD4RDZK3UNB6VKHOTWWID42RGBJZQHTGG4NDYIPA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: read tcp 188.166.225.170:44130->145.79.227.24:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:12:04.270781470 (176h8m2.822251921s) [noise=true , address=145.79.227.11:4732 , piece_id=FQGAVGZ5CID3LBET3CVK6FSCOP5OUYGS44XYWMHU2RLIT3MBN62A, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: read tcp 188.166.225.170:42420->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:12:04.270825082 (176h8m2.822295544s) [seg_pos={0 0}, upload=155107]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil>
Nov 16 16:12:05.262661603 (176h8m3.814132064s) [upload=155107]: Done uploading segments...
Nov 16 16:12:05.262728080 (176h8m3.814198541s) [upload=155107]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4)
Duration: 5.978992674s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [.........X..................................................] duration:
3.584129417s trace:
https://jaeger.dev.storj.io/trace/1026d6559b04863a
FAIL (2025-11-16 16:20:52 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.58415755s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...............................................X............] duration:
6.156353345s trace:
https://jaeger.dev.storj.io/trace/3e4560ab56ee4602
FAIL (2025-11-16 16:20:49 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.156369888s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [...................................X........................] duration:
5.423677653s trace:
https://jaeger.dev.storj.io/trace/7dbd01be7ae5d01e
FAIL (2025-11-16 16:20:42 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 24)
======LOGS======
OHA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.079132696 (176h0m30.630603157s) [noise=true , address=81.171.14.70:4040 , piece_id=YG6ZBK6QTKUS3FCFD2EWU7HTFLOEEVG57SIER44E67QDP5HYCHOA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.079319683 (176h0m30.630790144s) [noise=true , address=185.167.204.205:4015 , piece_id=6UIFX7GXE2SZ465VAP56YQILR3A3OHDRKIS345EE6GBDG77KJNZQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.079646465 (176h0m30.631116926s) [noise=true , address=185.167.204.206:4062 , piece_id=PAJX3ZCWM5YBZA7YDBFYXCET5CKWBOFJ47IUWK4CYPZSWK6HYQVQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.080002833 (176h0m30.631473294s) [noise=true , address=81.171.14.70:4031 , piece_id=54T5MOZZ4YJKQFUDOCCFF2SAE7XS2N77H2AOHCBECAWE5OBUXTZQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.080062066 (176h0m30.631532527s) [noise=true , address=185.167.204.206:4067 , piece_id=2BDMQZLCV4UWWP5HB6GF6NCL5UOGNU7VNYGJIIJCKQHRTX5OPGPQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.080438673 (176h0m30.631909134s) [noise=true , address=185.167.204.205:4034 , piece_id=CZFPF35OQ3U4IH46KILFPFMVY6OC36ZUGZ7EYYBOTB3I4TXMPERA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.080675103 (176h0m30.632145564s) [noise=true , address=185.167.204.205:4014 , piece_id=57C2VUYOEULOTXUQHG7A2XOWZLXYFIEYYRYW5ARX5PKTGWKIEOAQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.081315833 (176h0m30.632786294s) [noise=true , address=81.171.14.70:4039 , piece_id=NMZ5357FVIKD2F3GCKY6YNNV73BAID2YBLVNSSM6VD5PHGLBHEIA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.082053738 (176h0m30.633524189s) [noise=true , address=185.167.204.205:4001 , piece_id=TBPYFHMXWO53LJQQDUA3I2MD2WP6BHQXAH3KIF6XXMWK27HXLCHA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.082605429 (176h0m30.63407589s) [noise=true , address=81.171.14.70:4034 , piece_id=NXBDCCCEWSZ2TMSET6OXBNQV6Q5ESXIQJUFK4WJJ3UV7FJTTAUXQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:33.280203414 (176h0m31.831673875s) [upload=155002]: Next returned nil segment
Nov 16 16:04:33.280222810 (176h0m31.831693271s) [upload=155002]: Waiting for error group managing segments...
Nov 16 16:04:33.838289781 (176h0m32.389760242s) [noise=true , address=185.167.204.205:4013 , piece_id=KF32EXUYGKQEDBPRSVAMNZV3ZMBF6DLT3U32BKX5LXLRBXBVTOQA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.848846382 (176h0m32.400316833s) [noise=true , address=185.167.204.206:4039 , piece_id=6HTUHESIRTSKR5EDMWZ2K56OKE5Y22QQM5ARYTUXR7KMKWIOS7GQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.862791868 (176h0m32.414262329s) [noise=true , address=185.167.204.205:4021 , piece_id=MV3TV4EI3HN3J5NY5XYVKT46QJTJEC5PITXERSSZMS2B7W3DSDTA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.879947135 (176h0m32.431417596s) [noise=true , address=185.167.204.205:4001 , piece_id=TBPYFHMXWO53LJQQDUA3I2MD2WP6BHQXAH3KIF6XXMWK27HXLCHA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.886853657 (176h0m32.438324118s) [noise=true , address=185.167.204.206:4067 , piece_id=2BDMQZLCV4UWWP5HB6GF6NCL5UOGNU7VNYGJIIJCKQHRTX5OPGPQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.918801783 (176h0m32.470272324s) [noise=true , address=185.167.204.206:4051 , piece_id=AU2S6M3E2CQEMAS7LKZJIVWITZUHIHCMWQ2QWT7ILGBRTAUCUB7Q, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.929982083 (176h0m32.481452574s) [noise=true , address=185.167.204.205:4015 , piece_id=6UIFX7GXE2SZ465VAP56YQILR3A3OHDRKIS345EE6GBDG77KJNZQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.935247409 (176h0m32.48671786s) [noise=true , address=185.167.204.206:4038 , piece_id=FBNPAJOOXEAT3KLKQIYOWHGHRGAWJABV4LAA3UYX6SGJB3U2LD7A, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.969630813 (176h0m32.521101264s) [noise=true , address=185.167.204.206:4055 , piece_id=B2TTK2ZLIRM3YIKP5HQ3WOL26KBHSXWWX5ZXXCMWU34AH7CHOOCQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.999043600 (176h0m32.550514061s) [noise=true , address=185.167.204.205:4034 , piece_id=CZFPF35OQ3U4IH46KILFPFMVY6OC36ZUGZ7EYYBOTB3I4TXMPERA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.022951616 (176h0m32.574422077s) [noise=true , address=185.167.204.205:4014 , piece_id=57C2VUYOEULOTXUQHG7A2XOWZLXYFIEYYRYW5ARX5PKTGWKIEOAQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.064488381 (176h0m32.615958832s) [noise=true , address=185.167.204.206:4058 , piece_id=POK7HBLE5AT2STCSL2UJ5R67LNVK3PN53NIRIBX4M7LJYC4C2CHQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.122257987 (176h0m32.673728468s) [noise=true , address=185.167.204.205:4029 , piece_id=VF7BTI4K75HTRVJOHIKMDJN3UBHZ7KCQBT3H44XB5G7ZVFD4HXJQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.136282884 (176h0m32.687753345s) [noise=true , address=185.167.204.206:4041 , piece_id=RTQSCXR54KTKBYS6AHEKPO7ZPUHVQQNUNQJ6JI4JLTGKOLUZH3AQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.136821679 (176h0m32.68829215s) [noise=true , address=185.167.204.206:4062 , piece_id=PAJX3ZCWM5YBZA7YDBFYXCET5CKWBOFJ47IUWK4CYPZSWK6HYQVQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.156096042 (176h0m32.707566493s) [noise=true , address=185.167.204.206:4066 , piece_id=CPTBHDHCT6R2M3UA6X2ZWMPJUKAUVV4I2CIATZ2ZOIWH63Q3UT4Q, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.540981809 (176h0m33.0924523s) [noise=true , address=89.149.201.147:4012 , piece_id=NEIYQHVVWRUVJFF5UD3RZSRHUSNSKXBCNGTMIGOH5MDH2AOZISCA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:21.258617296 (176h1m19.810087757s) [noise=true , address=81.171.14.70:4034 , piece_id=NXBDCCCEWSZ2TMSET6OXBNQV6Q5ESXIQJUFK4WJJ3UV7FJTTAUXQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:22.223432573 (176h1m20.774903034s) [noise=true , address=81.171.14.70:4048 , piece_id=5WUEBVU7UFKHYZ23BPIQUMUUBH3ZXXQWBLE7MALJGECCDTTEK5EA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:22.238494073 (176h1m20.789964534s) [noise=true , address=81.171.14.70:4033 , piece_id=LUBQIVFAU7J72K5XDDS5S2ZXQUGFA5AINL4R5FV64KY24VS5KV5Q, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:22.350384206 (176h1m20.901854667s) [noise=true , address=81.171.14.70:4039 , piece_id=NMZ5357FVIKD2F3GCKY6YNNV73BAID2YBLVNSSM6VD5PHGLBHEIA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:23.704484814 (176h1m22.255955275s) [noise=true , address=81.171.14.70:4037 , piece_id=3IJ7J7RBMOORW4YPKIYJ3C6ZQRYBH7WQMLEKEQYG5UJ4TGKE2W2A, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:24.501497158 (176h1m23.052967619s) [noise=true , address=81.171.14.70:4031 , piece_id=54T5MOZZ4YJKQFUDOCCFF2SAE7XS2N77H2AOHCBECAWE5OBUXTZQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:29.810461698 (176h1m28.361932159s) [noise=true , address=81.171.14.70:4025 , piece_id=3HAT6FJSGV6VOHJBMO6L6L7677BOYVCDV3SR26LJZ6HZOZWJJ64A, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:31.413018058 (176h1m29.964488509s) [noise=true , address=81.171.14.70:4040 , piece_id=YG6ZBK6QTKUS3FCFD2EWU7HTFLOEEVG57SIER44E67QDP5HYCHOA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:1gannW154cmVP6N5BowytKfDorZDRwx2M5pi6akqjZFU38PGjV, address:81.171.14.70:4040): protocol: write tcp 188.166.225.170:59244->81.171.14.70:4040: use of closed network connection; write tcp 188.166.225.170:59244->81.171.14.70:4040: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59244->81.171.14.70:4040: use of closed network connection
Nov 16 16:05:31.413111366 (176h1m29.964581837s) [noise=true , address=81.171.14.70:4035 , piece_id=RGMXRH2Y5XBAFISK5LNAF547SAORD6XSWFUKYWOPJINXXODI5KVQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:12eLYJKne3yP1fZ1oPDLM7Xqs3GQ3ju4ZQedAiojXdVkcxNpzeA, address:81.171.14.70:4035): protocol: read tcp 188.166.225.170:55142->81.171.14.70:4035: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:05:31.413619965 (176h1m29.965090436s) [noise=true , address=81.171.14.70:4030 , piece_id=ZE5OAUTWLWNSMWSFGPEJTCXEN6L3FLWL34QBERTS3JZE7X2I2OHA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:12ZoRwa8wLw66Pjumy1ybp6wrZ79RrSU55uYZpwaY3xi95iSYde, address:81.171.14.70:4030): protocol: read tcp 188.166.225.170:40822->81.171.14.70:4030: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:05:31.414846280 (176h1m29.966316741s) [noise=true , address=81.171.14.70:4029 , piece_id=SIXL2XSCF7IO2BPOUKGH5VVXI6M7O4KJVZARYIRELR2GM4T4RABA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:18yEfU2X4jSrom1JDtQ22cUEvomcg91ohvay2Nsr1Vr8juo9to, address:81.171.14.70:4029): protocol: read tcp 188.166.225.170:56584->81.171.14.70:4029: use of closed network connection; conn closed: op
Nov 16 16:05:31.414946261 (176h1m29.966416722s) [seg_pos={0 0}, upload=155002]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil>
Nov 16 16:05:31.591745995 (176h1m30.143216456s) [upload=155002]: Done uploading segments...
Nov 16 16:05:31.591788596 (176h1m30.143259057s) [upload=155002]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24)
Duration: 5.423706639s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [..........X.................................................] duration:
3.497770542s trace:
https://jaeger.dev.storj.io/trace/1026d6559b04863a
FAIL (2025-11-16 16:20:24 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.497798636s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..............................................X.............] duration:
6.022324246s trace:
https://jaeger.dev.storj.io/trace/70f81d2d9b012a3a
FAIL (2025-11-16 16:19:59 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 4)
======LOGS======
ddress=45.91.116.5:4297 , piece_id=OSMI3VGAKSCIZMDQ3Y2VZQBUOJKSXM6RWNWSAV57RDTL5TR3NIJA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.462247309 (176h7m35.013717781s) [noise=true , address=45.91.116.7:4011 , piece_id=ORJVRMFIOEXZ6OB6QFQDL764Q2FCWOJT63NBIRI6SOOS2FEIBCFQ, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.463420774 (176h7m35.014891235s) [noise=true , address=145.79.227.9:4367 , piece_id=LOFHPWCHNEZ4E4WANJM5BE2TIWVFH4LGEJJYK4ETB2LZWYCAVGZA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.464113112 (176h7m35.015583573s) [noise=true , address=145.79.227.22:4511 , piece_id=HES465EP3OPD5JBVGVYMGLUOHJKSSMJDXMJVWET5GHRPFZJFIFOA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.464764873 (176h7m35.016235334s) [noise=true , address=45.91.116.8:4048 , piece_id=ULVK52VWCW3KNEPWSDYGRHWHIJWEQU2XNFEPYJFXWLVJI6AGZQOA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.465452753 (176h7m35.016923214s) [noise=true , address=145.79.227.28:4725 , piece_id=CI2LSBGPW7RYH4RGWODGWLUTN353EODNV6IB2YQGHOZ3FPSEEK5A, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.466223270 (176h7m35.017693731s) [noise=true , address=145.79.227.16:4910 , piece_id=3U2SEVMPILIEBRHS3M4YT4Z4PSYAQAIC2SKGYMW4FNODACZLOQCQ, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.466955855 (176h7m35.018426306s) [noise=true , address=145.79.227.4:4186 , piece_id=5PZIK3V72CUQQ6M2HJZFVM5OVX4HPHY7ZHJ2BAK5S3ZQOGM4J5OA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.467675205 (176h7m35.019145666s) [noise=true , address=145.79.227.8:4331 , piece_id=4O27HU7AMR6G4Y43HJYN22UXVLETBDBE7JHL4JUZKRM54SK5E4JA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.468178092 (176h7m35.019648553s) [noise=true , address=145.79.227.8:4329 , piece_id=GO5IAWQYS3UARWYAQAEUIIS456DFWCFWRJQG2KOVVL5KPXWPK4AA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:39.111804678 (176h7m37.663275129s) [upload=155107]: Next returned nil segment
Nov 16 16:11:39.111825767 (176h7m37.663296218s) [upload=155107]: Waiting for error group managing segments...
Nov 16 16:11:39.896671313 (176h7m38.448141765s) [noise=true , address=45.91.116.12:4227 , piece_id=CQA2LHRSFOA7EIBIFVC75EWVHFZMWMY7ODMETEAJRRNYSQFY5TVQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil>
Nov 16 16:11:39.954993420 (176h7m38.506463881s) [noise=true , address=45.91.116.8:4048 , piece_id=ULVK52VWCW3KNEPWSDYGRHWHIJWEQU2XNFEPYJFXWLVJI6AGZQOA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil>
Nov 16 16:11:40.066014029 (176h7m38.61748449s) [noise=true , address=45.91.116.7:4011 , piece_id=ORJVRMFIOEXZ6OB6QFQDL764Q2FCWOJT63NBIRI6SOOS2FEIBCFQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil>
Nov 16 16:11:41.102688250 (176h7m39.654158711s) [noise=true , address=45.91.116.5:4297 , piece_id=OSMI3VGAKSCIZMDQ3Y2VZQBUOJKSXM6RWNWSAV57RDTL5TR3NIJA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil>
Nov 16 16:12:04.264842280 (176h8m2.816312741s) [noise=true , address=145.79.227.17:4972 , piece_id=NV6FSBQM6YUTKYFAJPQCURWCZBHJ3AKJOLLGCRYTJ3KQ54U4GOWQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection
Nov 16 16:12:04.265084832 (176h8m2.816555293s) [noise=true , address=145.79.227.16:4910 , piece_id=3U2SEVMPILIEBRHS3M4YT4Z4PSYAQAIC2SKGYMW4FNODACZLOQCQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection
Nov 16 16:12:04.270267942 (176h8m2.821738423s) [noise=true , address=145.79.227.8:4329 , piece_id=GO5IAWQYS3UARWYAQAEUIIS456DFWCFWRJQG2KOVVL5KPXWPK4AA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: expected piece hash; context deadline exceeded; EOF
Nov 16 16:12:04.270306515 (176h8m2.821776986s) [noise=true , address=145.79.227.9:4367 , piece_id=LOFHPWCHNEZ4E4WANJM5BE2TIWVFH4LGEJJYK4ETB2LZWYCAVGZA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection
Nov 16 16:12:04.270323538 (176h8m2.821794099s) [noise=true , address=145.79.227.28:4725 , piece_id=CI2LSBGPW7RYH4RGWODGWLUTN353EODNV6IB2YQGHOZ3FPSEEK5A, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection
Nov 16 16:12:04.270544138 (176h8m2.822014739s) [noise=true , address=145.79.227.14:4840 , piece_id=RSHGL65RI7HZGU3W3EROFL23BCK2EW3YN6CAD3NO5OVE5QM3QG3Q, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: read tcp 188.166.225.170:42148->145.79.227.14:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:12:04.270571570 (176h8m2.822042121s) [noise=true , address=145.79.227.31:4797 , piece_id=AWSPIPRKSBOKQFZRRB34RUPLYVJGKAOOJQ4U7RRATQJELM3OCFHA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection
Nov 16 16:12:04.270609783 (176h8m2.822080254s) [noise=true , address=145.79.227.22:4511 , piece_id=HES465EP3OPD5JBVGVYMGLUOHJKSSMJDXMJVWET5GHRPFZJFIFOA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: read tcp 188.166.225.170:51004->145.79.227.22:4511: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:12:04.270625552 (176h8m2.822096083s) [noise=true , address=145.79.227.8:4331 , piece_id=4O27HU7AMR6G4Y43HJYN22UXVLETBDBE7JHL4JUZKRM54SK5E4JA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: read tcp 188.166.225.170:44510->145.79.227.8:4331: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:12:04.270742155 (176h8m2.822212616s) [noise=true , address=145.79.227.4:4186 , piece_id=5PZIK3V72CUQQ6M2HJZFVM5OVX4HPHY7ZHJ2BAK5S3ZQOGM4J5OA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection
Nov 16 16:12:04.270762814 (176h8m2.822233275s) [noise=true , address=145.79.227.24:4581 , piece_id=XRCS52EJDB7HWD4RDZK3UNB6VKHOTWWID42RGBJZQHTGG4NDYIPA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: read tcp 188.166.225.170:44130->145.79.227.24:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:12:04.270781470 (176h8m2.822251921s) [noise=true , address=145.79.227.11:4732 , piece_id=FQGAVGZ5CID3LBET3CVK6FSCOP5OUYGS44XYWMHU2RLIT3MBN62A, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: read tcp 188.166.225.170:42420->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:12:04.270825082 (176h8m2.822295544s) [seg_pos={0 0}, upload=155107]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil>
Nov 16 16:12:05.262661603 (176h8m3.814132064s) [upload=155107]: Done uploading segments...
Nov 16 16:12:05.262728080 (176h8m3.814198541s) [upload=155107]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4)
Duration: 6.022350044s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [....................................X.......................] duration:
5.363266296s trace:
https://jaeger.dev.storj.io/trace/7dbd01be7ae5d01e
FAIL (2025-11-16 16:19:43 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 24)
======LOGS======
OHA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.079132696 (176h0m30.630603157s) [noise=true , address=81.171.14.70:4040 , piece_id=YG6ZBK6QTKUS3FCFD2EWU7HTFLOEEVG57SIER44E67QDP5HYCHOA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.079319683 (176h0m30.630790144s) [noise=true , address=185.167.204.205:4015 , piece_id=6UIFX7GXE2SZ465VAP56YQILR3A3OHDRKIS345EE6GBDG77KJNZQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.079646465 (176h0m30.631116926s) [noise=true , address=185.167.204.206:4062 , piece_id=PAJX3ZCWM5YBZA7YDBFYXCET5CKWBOFJ47IUWK4CYPZSWK6HYQVQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.080002833 (176h0m30.631473294s) [noise=true , address=81.171.14.70:4031 , piece_id=54T5MOZZ4YJKQFUDOCCFF2SAE7XS2N77H2AOHCBECAWE5OBUXTZQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.080062066 (176h0m30.631532527s) [noise=true , address=185.167.204.206:4067 , piece_id=2BDMQZLCV4UWWP5HB6GF6NCL5UOGNU7VNYGJIIJCKQHRTX5OPGPQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.080438673 (176h0m30.631909134s) [noise=true , address=185.167.204.205:4034 , piece_id=CZFPF35OQ3U4IH46KILFPFMVY6OC36ZUGZ7EYYBOTB3I4TXMPERA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.080675103 (176h0m30.632145564s) [noise=true , address=185.167.204.205:4014 , piece_id=57C2VUYOEULOTXUQHG7A2XOWZLXYFIEYYRYW5ARX5PKTGWKIEOAQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.081315833 (176h0m30.632786294s) [noise=true , address=81.171.14.70:4039 , piece_id=NMZ5357FVIKD2F3GCKY6YNNV73BAID2YBLVNSSM6VD5PHGLBHEIA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.082053738 (176h0m30.633524189s) [noise=true , address=185.167.204.205:4001 , piece_id=TBPYFHMXWO53LJQQDUA3I2MD2WP6BHQXAH3KIF6XXMWK27HXLCHA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.082605429 (176h0m30.63407589s) [noise=true , address=81.171.14.70:4034 , piece_id=NXBDCCCEWSZ2TMSET6OXBNQV6Q5ESXIQJUFK4WJJ3UV7FJTTAUXQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:33.280203414 (176h0m31.831673875s) [upload=155002]: Next returned nil segment
Nov 16 16:04:33.280222810 (176h0m31.831693271s) [upload=155002]: Waiting for error group managing segments...
Nov 16 16:04:33.838289781 (176h0m32.389760242s) [noise=true , address=185.167.204.205:4013 , piece_id=KF32EXUYGKQEDBPRSVAMNZV3ZMBF6DLT3U32BKX5LXLRBXBVTOQA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.848846382 (176h0m32.400316833s) [noise=true , address=185.167.204.206:4039 , piece_id=6HTUHESIRTSKR5EDMWZ2K56OKE5Y22QQM5ARYTUXR7KMKWIOS7GQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.862791868 (176h0m32.414262329s) [noise=true , address=185.167.204.205:4021 , piece_id=MV3TV4EI3HN3J5NY5XYVKT46QJTJEC5PITXERSSZMS2B7W3DSDTA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.879947135 (176h0m32.431417596s) [noise=true , address=185.167.204.205:4001 , piece_id=TBPYFHMXWO53LJQQDUA3I2MD2WP6BHQXAH3KIF6XXMWK27HXLCHA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.886853657 (176h0m32.438324118s) [noise=true , address=185.167.204.206:4067 , piece_id=2BDMQZLCV4UWWP5HB6GF6NCL5UOGNU7VNYGJIIJCKQHRTX5OPGPQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.918801783 (176h0m32.470272324s) [noise=true , address=185.167.204.206:4051 , piece_id=AU2S6M3E2CQEMAS7LKZJIVWITZUHIHCMWQ2QWT7ILGBRTAUCUB7Q, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.929982083 (176h0m32.481452574s) [noise=true , address=185.167.204.205:4015 , piece_id=6UIFX7GXE2SZ465VAP56YQILR3A3OHDRKIS345EE6GBDG77KJNZQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.935247409 (176h0m32.48671786s) [noise=true , address=185.167.204.206:4038 , piece_id=FBNPAJOOXEAT3KLKQIYOWHGHRGAWJABV4LAA3UYX6SGJB3U2LD7A, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.969630813 (176h0m32.521101264s) [noise=true , address=185.167.204.206:4055 , piece_id=B2TTK2ZLIRM3YIKP5HQ3WOL26KBHSXWWX5ZXXCMWU34AH7CHOOCQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.999043600 (176h0m32.550514061s) [noise=true , address=185.167.204.205:4034 , piece_id=CZFPF35OQ3U4IH46KILFPFMVY6OC36ZUGZ7EYYBOTB3I4TXMPERA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.022951616 (176h0m32.574422077s) [noise=true , address=185.167.204.205:4014 , piece_id=57C2VUYOEULOTXUQHG7A2XOWZLXYFIEYYRYW5ARX5PKTGWKIEOAQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.064488381 (176h0m32.615958832s) [noise=true , address=185.167.204.206:4058 , piece_id=POK7HBLE5AT2STCSL2UJ5R67LNVK3PN53NIRIBX4M7LJYC4C2CHQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.122257987 (176h0m32.673728468s) [noise=true , address=185.167.204.205:4029 , piece_id=VF7BTI4K75HTRVJOHIKMDJN3UBHZ7KCQBT3H44XB5G7ZVFD4HXJQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.136282884 (176h0m32.687753345s) [noise=true , address=185.167.204.206:4041 , piece_id=RTQSCXR54KTKBYS6AHEKPO7ZPUHVQQNUNQJ6JI4JLTGKOLUZH3AQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.136821679 (176h0m32.68829215s) [noise=true , address=185.167.204.206:4062 , piece_id=PAJX3ZCWM5YBZA7YDBFYXCET5CKWBOFJ47IUWK4CYPZSWK6HYQVQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.156096042 (176h0m32.707566493s) [noise=true , address=185.167.204.206:4066 , piece_id=CPTBHDHCT6R2M3UA6X2ZWMPJUKAUVV4I2CIATZ2ZOIWH63Q3UT4Q, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.540981809 (176h0m33.0924523s) [noise=true , address=89.149.201.147:4012 , piece_id=NEIYQHVVWRUVJFF5UD3RZSRHUSNSKXBCNGTMIGOH5MDH2AOZISCA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:21.258617296 (176h1m19.810087757s) [noise=true , address=81.171.14.70:4034 , piece_id=NXBDCCCEWSZ2TMSET6OXBNQV6Q5ESXIQJUFK4WJJ3UV7FJTTAUXQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:22.223432573 (176h1m20.774903034s) [noise=true , address=81.171.14.70:4048 , piece_id=5WUEBVU7UFKHYZ23BPIQUMUUBH3ZXXQWBLE7MALJGECCDTTEK5EA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:22.238494073 (176h1m20.789964534s) [noise=true , address=81.171.14.70:4033 , piece_id=LUBQIVFAU7J72K5XDDS5S2ZXQUGFA5AINL4R5FV64KY24VS5KV5Q, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:22.350384206 (176h1m20.901854667s) [noise=true , address=81.171.14.70:4039 , piece_id=NMZ5357FVIKD2F3GCKY6YNNV73BAID2YBLVNSSM6VD5PHGLBHEIA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:23.704484814 (176h1m22.255955275s) [noise=true , address=81.171.14.70:4037 , piece_id=3IJ7J7RBMOORW4YPKIYJ3C6ZQRYBH7WQMLEKEQYG5UJ4TGKE2W2A, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:24.501497158 (176h1m23.052967619s) [noise=true , address=81.171.14.70:4031 , piece_id=54T5MOZZ4YJKQFUDOCCFF2SAE7XS2N77H2AOHCBECAWE5OBUXTZQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:29.810461698 (176h1m28.361932159s) [noise=true , address=81.171.14.70:4025 , piece_id=3HAT6FJSGV6VOHJBMO6L6L7677BOYVCDV3SR26LJZ6HZOZWJJ64A, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:31.413018058 (176h1m29.964488509s) [noise=true , address=81.171.14.70:4040 , piece_id=YG6ZBK6QTKUS3FCFD2EWU7HTFLOEEVG57SIER44E67QDP5HYCHOA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:1gannW154cmVP6N5BowytKfDorZDRwx2M5pi6akqjZFU38PGjV, address:81.171.14.70:4040): protocol: write tcp 188.166.225.170:59244->81.171.14.70:4040: use of closed network connection; write tcp 188.166.225.170:59244->81.171.14.70:4040: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59244->81.171.14.70:4040: use of closed network connection
Nov 16 16:05:31.413111366 (176h1m29.964581837s) [noise=true , address=81.171.14.70:4035 , piece_id=RGMXRH2Y5XBAFISK5LNAF547SAORD6XSWFUKYWOPJINXXODI5KVQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:12eLYJKne3yP1fZ1oPDLM7Xqs3GQ3ju4ZQedAiojXdVkcxNpzeA, address:81.171.14.70:4035): protocol: read tcp 188.166.225.170:55142->81.171.14.70:4035: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:05:31.413619965 (176h1m29.965090436s) [noise=true , address=81.171.14.70:4030 , piece_id=ZE5OAUTWLWNSMWSFGPEJTCXEN6L3FLWL34QBERTS3JZE7X2I2OHA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:12ZoRwa8wLw66Pjumy1ybp6wrZ79RrSU55uYZpwaY3xi95iSYde, address:81.171.14.70:4030): protocol: read tcp 188.166.225.170:40822->81.171.14.70:4030: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:05:31.414846280 (176h1m29.966316741s) [noise=true , address=81.171.14.70:4029 , piece_id=SIXL2XSCF7IO2BPOUKGH5VVXI6M7O4KJVZARYIRELR2GM4T4RABA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:18yEfU2X4jSrom1JDtQ22cUEvomcg91ohvay2Nsr1Vr8juo9to, address:81.171.14.70:4029): protocol: read tcp 188.166.225.170:56584->81.171.14.70:4029: use of closed network connection; conn closed: op
Nov 16 16:05:31.414946261 (176h1m29.966416722s) [seg_pos={0 0}, upload=155002]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil>
Nov 16 16:05:31.591745995 (176h1m30.143216456s) [upload=155002]: Done uploading segments...
Nov 16 16:05:31.591788596 (176h1m30.143259057s) [upload=155002]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24)
Duration: 5.363296044s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [...........X................................................] duration:
3.597113571s trace:
https://jaeger.dev.storj.io/trace/1026d6559b04863a
FAIL (2025-11-16 16:19:43 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.597163416s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [................................................X...........] duration:
6.236704961s trace:
https://jaeger.dev.storj.io/trace/3e4560ab56ee4602
FAIL (2025-11-16 16:19:42 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.23673091s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...............................................X............] duration:
6.059306398s trace:
https://jaeger.dev.storj.io/trace/70f81d2d9b012a3a
FAIL (2025-11-16 16:19:05 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 4)
======LOGS======
ddress=45.91.116.5:4297 , piece_id=OSMI3VGAKSCIZMDQ3Y2VZQBUOJKSXM6RWNWSAV57RDTL5TR3NIJA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.462247309 (176h7m35.013717781s) [noise=true , address=45.91.116.7:4011 , piece_id=ORJVRMFIOEXZ6OB6QFQDL764Q2FCWOJT63NBIRI6SOOS2FEIBCFQ, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.463420774 (176h7m35.014891235s) [noise=true , address=145.79.227.9:4367 , piece_id=LOFHPWCHNEZ4E4WANJM5BE2TIWVFH4LGEJJYK4ETB2LZWYCAVGZA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.464113112 (176h7m35.015583573s) [noise=true , address=145.79.227.22:4511 , piece_id=HES465EP3OPD5JBVGVYMGLUOHJKSSMJDXMJVWET5GHRPFZJFIFOA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.464764873 (176h7m35.016235334s) [noise=true , address=45.91.116.8:4048 , piece_id=ULVK52VWCW3KNEPWSDYGRHWHIJWEQU2XNFEPYJFXWLVJI6AGZQOA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.465452753 (176h7m35.016923214s) [noise=true , address=145.79.227.28:4725 , piece_id=CI2LSBGPW7RYH4RGWODGWLUTN353EODNV6IB2YQGHOZ3FPSEEK5A, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.466223270 (176h7m35.017693731s) [noise=true , address=145.79.227.16:4910 , piece_id=3U2SEVMPILIEBRHS3M4YT4Z4PSYAQAIC2SKGYMW4FNODACZLOQCQ, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.466955855 (176h7m35.018426306s) [noise=true , address=145.79.227.4:4186 , piece_id=5PZIK3V72CUQQ6M2HJZFVM5OVX4HPHY7ZHJ2BAK5S3ZQOGM4J5OA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.467675205 (176h7m35.019145666s) [noise=true , address=145.79.227.8:4331 , piece_id=4O27HU7AMR6G4Y43HJYN22UXVLETBDBE7JHL4JUZKRM54SK5E4JA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.468178092 (176h7m35.019648553s) [noise=true , address=145.79.227.8:4329 , piece_id=GO5IAWQYS3UARWYAQAEUIIS456DFWCFWRJQG2KOVVL5KPXWPK4AA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:39.111804678 (176h7m37.663275129s) [upload=155107]: Next returned nil segment
Nov 16 16:11:39.111825767 (176h7m37.663296218s) [upload=155107]: Waiting for error group managing segments...
Nov 16 16:11:39.896671313 (176h7m38.448141765s) [noise=true , address=45.91.116.12:4227 , piece_id=CQA2LHRSFOA7EIBIFVC75EWVHFZMWMY7ODMETEAJRRNYSQFY5TVQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil>
Nov 16 16:11:39.954993420 (176h7m38.506463881s) [noise=true , address=45.91.116.8:4048 , piece_id=ULVK52VWCW3KNEPWSDYGRHWHIJWEQU2XNFEPYJFXWLVJI6AGZQOA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil>
Nov 16 16:11:40.066014029 (176h7m38.61748449s) [noise=true , address=45.91.116.7:4011 , piece_id=ORJVRMFIOEXZ6OB6QFQDL764Q2FCWOJT63NBIRI6SOOS2FEIBCFQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil>
Nov 16 16:11:41.102688250 (176h7m39.654158711s) [noise=true , address=45.91.116.5:4297 , piece_id=OSMI3VGAKSCIZMDQ3Y2VZQBUOJKSXM6RWNWSAV57RDTL5TR3NIJA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil>
Nov 16 16:12:04.264842280 (176h8m2.816312741s) [noise=true , address=145.79.227.17:4972 , piece_id=NV6FSBQM6YUTKYFAJPQCURWCZBHJ3AKJOLLGCRYTJ3KQ54U4GOWQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection
Nov 16 16:12:04.265084832 (176h8m2.816555293s) [noise=true , address=145.79.227.16:4910 , piece_id=3U2SEVMPILIEBRHS3M4YT4Z4PSYAQAIC2SKGYMW4FNODACZLOQCQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection
Nov 16 16:12:04.270267942 (176h8m2.821738423s) [noise=true , address=145.79.227.8:4329 , piece_id=GO5IAWQYS3UARWYAQAEUIIS456DFWCFWRJQG2KOVVL5KPXWPK4AA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: expected piece hash; context deadline exceeded; EOF
Nov 16 16:12:04.270306515 (176h8m2.821776986s) [noise=true , address=145.79.227.9:4367 , piece_id=LOFHPWCHNEZ4E4WANJM5BE2TIWVFH4LGEJJYK4ETB2LZWYCAVGZA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection
Nov 16 16:12:04.270323538 (176h8m2.821794099s) [noise=true , address=145.79.227.28:4725 , piece_id=CI2LSBGPW7RYH4RGWODGWLUTN353EODNV6IB2YQGHOZ3FPSEEK5A, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection
Nov 16 16:12:04.270544138 (176h8m2.822014739s) [noise=true , address=145.79.227.14:4840 , piece_id=RSHGL65RI7HZGU3W3EROFL23BCK2EW3YN6CAD3NO5OVE5QM3QG3Q, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: read tcp 188.166.225.170:42148->145.79.227.14:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:12:04.270571570 (176h8m2.822042121s) [noise=true , address=145.79.227.31:4797 , piece_id=AWSPIPRKSBOKQFZRRB34RUPLYVJGKAOOJQ4U7RRATQJELM3OCFHA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection
Nov 16 16:12:04.270609783 (176h8m2.822080254s) [noise=true , address=145.79.227.22:4511 , piece_id=HES465EP3OPD5JBVGVYMGLUOHJKSSMJDXMJVWET5GHRPFZJFIFOA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: read tcp 188.166.225.170:51004->145.79.227.22:4511: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:12:04.270625552 (176h8m2.822096083s) [noise=true , address=145.79.227.8:4331 , piece_id=4O27HU7AMR6G4Y43HJYN22UXVLETBDBE7JHL4JUZKRM54SK5E4JA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: read tcp 188.166.225.170:44510->145.79.227.8:4331: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:12:04.270742155 (176h8m2.822212616s) [noise=true , address=145.79.227.4:4186 , piece_id=5PZIK3V72CUQQ6M2HJZFVM5OVX4HPHY7ZHJ2BAK5S3ZQOGM4J5OA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection
Nov 16 16:12:04.270762814 (176h8m2.822233275s) [noise=true , address=145.79.227.24:4581 , piece_id=XRCS52EJDB7HWD4RDZK3UNB6VKHOTWWID42RGBJZQHTGG4NDYIPA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: read tcp 188.166.225.170:44130->145.79.227.24:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:12:04.270781470 (176h8m2.822251921s) [noise=true , address=145.79.227.11:4732 , piece_id=FQGAVGZ5CID3LBET3CVK6FSCOP5OUYGS44XYWMHU2RLIT3MBN62A, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: read tcp 188.166.225.170:42420->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:12:04.270825082 (176h8m2.822295544s) [seg_pos={0 0}, upload=155107]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil>
Nov 16 16:12:05.262661603 (176h8m3.814132064s) [upload=155107]: Done uploading segments...
Nov 16 16:12:05.262728080 (176h8m3.814198541s) [upload=155107]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4)
Duration: 6.059342387s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [.....................................X......................] duration:
5.431840546s trace:
https://jaeger.dev.storj.io/trace/7dbd01be7ae5d01e
FAIL (2025-11-16 16:18:57 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 24)
======LOGS======
OHA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.079132696 (176h0m30.630603157s) [noise=true , address=81.171.14.70:4040 , piece_id=YG6ZBK6QTKUS3FCFD2EWU7HTFLOEEVG57SIER44E67QDP5HYCHOA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.079319683 (176h0m30.630790144s) [noise=true , address=185.167.204.205:4015 , piece_id=6UIFX7GXE2SZ465VAP56YQILR3A3OHDRKIS345EE6GBDG77KJNZQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.079646465 (176h0m30.631116926s) [noise=true , address=185.167.204.206:4062 , piece_id=PAJX3ZCWM5YBZA7YDBFYXCET5CKWBOFJ47IUWK4CYPZSWK6HYQVQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.080002833 (176h0m30.631473294s) [noise=true , address=81.171.14.70:4031 , piece_id=54T5MOZZ4YJKQFUDOCCFF2SAE7XS2N77H2AOHCBECAWE5OBUXTZQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.080062066 (176h0m30.631532527s) [noise=true , address=185.167.204.206:4067 , piece_id=2BDMQZLCV4UWWP5HB6GF6NCL5UOGNU7VNYGJIIJCKQHRTX5OPGPQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.080438673 (176h0m30.631909134s) [noise=true , address=185.167.204.205:4034 , piece_id=CZFPF35OQ3U4IH46KILFPFMVY6OC36ZUGZ7EYYBOTB3I4TXMPERA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.080675103 (176h0m30.632145564s) [noise=true , address=185.167.204.205:4014 , piece_id=57C2VUYOEULOTXUQHG7A2XOWZLXYFIEYYRYW5ARX5PKTGWKIEOAQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.081315833 (176h0m30.632786294s) [noise=true , address=81.171.14.70:4039 , piece_id=NMZ5357FVIKD2F3GCKY6YNNV73BAID2YBLVNSSM6VD5PHGLBHEIA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.082053738 (176h0m30.633524189s) [noise=true , address=185.167.204.205:4001 , piece_id=TBPYFHMXWO53LJQQDUA3I2MD2WP6BHQXAH3KIF6XXMWK27HXLCHA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.082605429 (176h0m30.63407589s) [noise=true , address=81.171.14.70:4034 , piece_id=NXBDCCCEWSZ2TMSET6OXBNQV6Q5ESXIQJUFK4WJJ3UV7FJTTAUXQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:33.280203414 (176h0m31.831673875s) [upload=155002]: Next returned nil segment
Nov 16 16:04:33.280222810 (176h0m31.831693271s) [upload=155002]: Waiting for error group managing segments...
Nov 16 16:04:33.838289781 (176h0m32.389760242s) [noise=true , address=185.167.204.205:4013 , piece_id=KF32EXUYGKQEDBPRSVAMNZV3ZMBF6DLT3U32BKX5LXLRBXBVTOQA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.848846382 (176h0m32.400316833s) [noise=true , address=185.167.204.206:4039 , piece_id=6HTUHESIRTSKR5EDMWZ2K56OKE5Y22QQM5ARYTUXR7KMKWIOS7GQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.862791868 (176h0m32.414262329s) [noise=true , address=185.167.204.205:4021 , piece_id=MV3TV4EI3HN3J5NY5XYVKT46QJTJEC5PITXERSSZMS2B7W3DSDTA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.879947135 (176h0m32.431417596s) [noise=true , address=185.167.204.205:4001 , piece_id=TBPYFHMXWO53LJQQDUA3I2MD2WP6BHQXAH3KIF6XXMWK27HXLCHA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.886853657 (176h0m32.438324118s) [noise=true , address=185.167.204.206:4067 , piece_id=2BDMQZLCV4UWWP5HB6GF6NCL5UOGNU7VNYGJIIJCKQHRTX5OPGPQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.918801783 (176h0m32.470272324s) [noise=true , address=185.167.204.206:4051 , piece_id=AU2S6M3E2CQEMAS7LKZJIVWITZUHIHCMWQ2QWT7ILGBRTAUCUB7Q, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.929982083 (176h0m32.481452574s) [noise=true , address=185.167.204.205:4015 , piece_id=6UIFX7GXE2SZ465VAP56YQILR3A3OHDRKIS345EE6GBDG77KJNZQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.935247409 (176h0m32.48671786s) [noise=true , address=185.167.204.206:4038 , piece_id=FBNPAJOOXEAT3KLKQIYOWHGHRGAWJABV4LAA3UYX6SGJB3U2LD7A, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.969630813 (176h0m32.521101264s) [noise=true , address=185.167.204.206:4055 , piece_id=B2TTK2ZLIRM3YIKP5HQ3WOL26KBHSXWWX5ZXXCMWU34AH7CHOOCQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.999043600 (176h0m32.550514061s) [noise=true , address=185.167.204.205:4034 , piece_id=CZFPF35OQ3U4IH46KILFPFMVY6OC36ZUGZ7EYYBOTB3I4TXMPERA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.022951616 (176h0m32.574422077s) [noise=true , address=185.167.204.205:4014 , piece_id=57C2VUYOEULOTXUQHG7A2XOWZLXYFIEYYRYW5ARX5PKTGWKIEOAQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.064488381 (176h0m32.615958832s) [noise=true , address=185.167.204.206:4058 , piece_id=POK7HBLE5AT2STCSL2UJ5R67LNVK3PN53NIRIBX4M7LJYC4C2CHQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.122257987 (176h0m32.673728468s) [noise=true , address=185.167.204.205:4029 , piece_id=VF7BTI4K75HTRVJOHIKMDJN3UBHZ7KCQBT3H44XB5G7ZVFD4HXJQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.136282884 (176h0m32.687753345s) [noise=true , address=185.167.204.206:4041 , piece_id=RTQSCXR54KTKBYS6AHEKPO7ZPUHVQQNUNQJ6JI4JLTGKOLUZH3AQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.136821679 (176h0m32.68829215s) [noise=true , address=185.167.204.206:4062 , piece_id=PAJX3ZCWM5YBZA7YDBFYXCET5CKWBOFJ47IUWK4CYPZSWK6HYQVQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.156096042 (176h0m32.707566493s) [noise=true , address=185.167.204.206:4066 , piece_id=CPTBHDHCT6R2M3UA6X2ZWMPJUKAUVV4I2CIATZ2ZOIWH63Q3UT4Q, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.540981809 (176h0m33.0924523s) [noise=true , address=89.149.201.147:4012 , piece_id=NEIYQHVVWRUVJFF5UD3RZSRHUSNSKXBCNGTMIGOH5MDH2AOZISCA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:21.258617296 (176h1m19.810087757s) [noise=true , address=81.171.14.70:4034 , piece_id=NXBDCCCEWSZ2TMSET6OXBNQV6Q5ESXIQJUFK4WJJ3UV7FJTTAUXQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:22.223432573 (176h1m20.774903034s) [noise=true , address=81.171.14.70:4048 , piece_id=5WUEBVU7UFKHYZ23BPIQUMUUBH3ZXXQWBLE7MALJGECCDTTEK5EA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:22.238494073 (176h1m20.789964534s) [noise=true , address=81.171.14.70:4033 , piece_id=LUBQIVFAU7J72K5XDDS5S2ZXQUGFA5AINL4R5FV64KY24VS5KV5Q, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:22.350384206 (176h1m20.901854667s) [noise=true , address=81.171.14.70:4039 , piece_id=NMZ5357FVIKD2F3GCKY6YNNV73BAID2YBLVNSSM6VD5PHGLBHEIA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:23.704484814 (176h1m22.255955275s) [noise=true , address=81.171.14.70:4037 , piece_id=3IJ7J7RBMOORW4YPKIYJ3C6ZQRYBH7WQMLEKEQYG5UJ4TGKE2W2A, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:24.501497158 (176h1m23.052967619s) [noise=true , address=81.171.14.70:4031 , piece_id=54T5MOZZ4YJKQFUDOCCFF2SAE7XS2N77H2AOHCBECAWE5OBUXTZQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:29.810461698 (176h1m28.361932159s) [noise=true , address=81.171.14.70:4025 , piece_id=3HAT6FJSGV6VOHJBMO6L6L7677BOYVCDV3SR26LJZ6HZOZWJJ64A, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:31.413018058 (176h1m29.964488509s) [noise=true , address=81.171.14.70:4040 , piece_id=YG6ZBK6QTKUS3FCFD2EWU7HTFLOEEVG57SIER44E67QDP5HYCHOA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:1gannW154cmVP6N5BowytKfDorZDRwx2M5pi6akqjZFU38PGjV, address:81.171.14.70:4040): protocol: write tcp 188.166.225.170:59244->81.171.14.70:4040: use of closed network connection; write tcp 188.166.225.170:59244->81.171.14.70:4040: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59244->81.171.14.70:4040: use of closed network connection
Nov 16 16:05:31.413111366 (176h1m29.964581837s) [noise=true , address=81.171.14.70:4035 , piece_id=RGMXRH2Y5XBAFISK5LNAF547SAORD6XSWFUKYWOPJINXXODI5KVQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:12eLYJKne3yP1fZ1oPDLM7Xqs3GQ3ju4ZQedAiojXdVkcxNpzeA, address:81.171.14.70:4035): protocol: read tcp 188.166.225.170:55142->81.171.14.70:4035: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:05:31.413619965 (176h1m29.965090436s) [noise=true , address=81.171.14.70:4030 , piece_id=ZE5OAUTWLWNSMWSFGPEJTCXEN6L3FLWL34QBERTS3JZE7X2I2OHA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:12ZoRwa8wLw66Pjumy1ybp6wrZ79RrSU55uYZpwaY3xi95iSYde, address:81.171.14.70:4030): protocol: read tcp 188.166.225.170:40822->81.171.14.70:4030: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:05:31.414846280 (176h1m29.966316741s) [noise=true , address=81.171.14.70:4029 , piece_id=SIXL2XSCF7IO2BPOUKGH5VVXI6M7O4KJVZARYIRELR2GM4T4RABA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:18yEfU2X4jSrom1JDtQ22cUEvomcg91ohvay2Nsr1Vr8juo9to, address:81.171.14.70:4029): protocol: read tcp 188.166.225.170:56584->81.171.14.70:4029: use of closed network connection; conn closed: op
Nov 16 16:05:31.414946261 (176h1m29.966416722s) [seg_pos={0 0}, upload=155002]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil>
Nov 16 16:05:31.591745995 (176h1m30.143216456s) [upload=155002]: Done uploading segments...
Nov 16 16:05:31.591788596 (176h1m30.143259057s) [upload=155002]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24)
Duration: 5.431867306s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [............X...............................................] duration:
3.545276222s trace:
https://jaeger.dev.storj.io/trace/1026d6559b04863a
FAIL (2025-11-16 16:18:39 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.545302172s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.................................................X..........] duration:
6.375760327s trace:
https://jaeger.dev.storj.io/trace/3e4560ab56ee4602
FAIL (2025-11-16 16:18:35 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.375791185s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [................................................X...........] duration:
5.886524159s trace:
https://jaeger.dev.storj.io/trace/70f81d2d9b012a3a
FAIL (2025-11-16 16:18:26 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 4)
======LOGS======
ddress=45.91.116.5:4297 , piece_id=OSMI3VGAKSCIZMDQ3Y2VZQBUOJKSXM6RWNWSAV57RDTL5TR3NIJA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.462247309 (176h7m35.013717781s) [noise=true , address=45.91.116.7:4011 , piece_id=ORJVRMFIOEXZ6OB6QFQDL764Q2FCWOJT63NBIRI6SOOS2FEIBCFQ, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.463420774 (176h7m35.014891235s) [noise=true , address=145.79.227.9:4367 , piece_id=LOFHPWCHNEZ4E4WANJM5BE2TIWVFH4LGEJJYK4ETB2LZWYCAVGZA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.464113112 (176h7m35.015583573s) [noise=true , address=145.79.227.22:4511 , piece_id=HES465EP3OPD5JBVGVYMGLUOHJKSSMJDXMJVWET5GHRPFZJFIFOA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.464764873 (176h7m35.016235334s) [noise=true , address=45.91.116.8:4048 , piece_id=ULVK52VWCW3KNEPWSDYGRHWHIJWEQU2XNFEPYJFXWLVJI6AGZQOA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.465452753 (176h7m35.016923214s) [noise=true , address=145.79.227.28:4725 , piece_id=CI2LSBGPW7RYH4RGWODGWLUTN353EODNV6IB2YQGHOZ3FPSEEK5A, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.466223270 (176h7m35.017693731s) [noise=true , address=145.79.227.16:4910 , piece_id=3U2SEVMPILIEBRHS3M4YT4Z4PSYAQAIC2SKGYMW4FNODACZLOQCQ, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.466955855 (176h7m35.018426306s) [noise=true , address=145.79.227.4:4186 , piece_id=5PZIK3V72CUQQ6M2HJZFVM5OVX4HPHY7ZHJ2BAK5S3ZQOGM4J5OA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.467675205 (176h7m35.019145666s) [noise=true , address=145.79.227.8:4331 , piece_id=4O27HU7AMR6G4Y43HJYN22UXVLETBDBE7JHL4JUZKRM54SK5E4JA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.468178092 (176h7m35.019648553s) [noise=true , address=145.79.227.8:4329 , piece_id=GO5IAWQYS3UARWYAQAEUIIS456DFWCFWRJQG2KOVVL5KPXWPK4AA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:39.111804678 (176h7m37.663275129s) [upload=155107]: Next returned nil segment
Nov 16 16:11:39.111825767 (176h7m37.663296218s) [upload=155107]: Waiting for error group managing segments...
Nov 16 16:11:39.896671313 (176h7m38.448141765s) [noise=true , address=45.91.116.12:4227 , piece_id=CQA2LHRSFOA7EIBIFVC75EWVHFZMWMY7ODMETEAJRRNYSQFY5TVQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil>
Nov 16 16:11:39.954993420 (176h7m38.506463881s) [noise=true , address=45.91.116.8:4048 , piece_id=ULVK52VWCW3KNEPWSDYGRHWHIJWEQU2XNFEPYJFXWLVJI6AGZQOA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil>
Nov 16 16:11:40.066014029 (176h7m38.61748449s) [noise=true , address=45.91.116.7:4011 , piece_id=ORJVRMFIOEXZ6OB6QFQDL764Q2FCWOJT63NBIRI6SOOS2FEIBCFQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil>
Nov 16 16:11:41.102688250 (176h7m39.654158711s) [noise=true , address=45.91.116.5:4297 , piece_id=OSMI3VGAKSCIZMDQ3Y2VZQBUOJKSXM6RWNWSAV57RDTL5TR3NIJA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil>
Nov 16 16:12:04.264842280 (176h8m2.816312741s) [noise=true , address=145.79.227.17:4972 , piece_id=NV6FSBQM6YUTKYFAJPQCURWCZBHJ3AKJOLLGCRYTJ3KQ54U4GOWQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection
Nov 16 16:12:04.265084832 (176h8m2.816555293s) [noise=true , address=145.79.227.16:4910 , piece_id=3U2SEVMPILIEBRHS3M4YT4Z4PSYAQAIC2SKGYMW4FNODACZLOQCQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection
Nov 16 16:12:04.270267942 (176h8m2.821738423s) [noise=true , address=145.79.227.8:4329 , piece_id=GO5IAWQYS3UARWYAQAEUIIS456DFWCFWRJQG2KOVVL5KPXWPK4AA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: expected piece hash; context deadline exceeded; EOF
Nov 16 16:12:04.270306515 (176h8m2.821776986s) [noise=true , address=145.79.227.9:4367 , piece_id=LOFHPWCHNEZ4E4WANJM5BE2TIWVFH4LGEJJYK4ETB2LZWYCAVGZA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection
Nov 16 16:12:04.270323538 (176h8m2.821794099s) [noise=true , address=145.79.227.28:4725 , piece_id=CI2LSBGPW7RYH4RGWODGWLUTN353EODNV6IB2YQGHOZ3FPSEEK5A, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection
Nov 16 16:12:04.270544138 (176h8m2.822014739s) [noise=true , address=145.79.227.14:4840 , piece_id=RSHGL65RI7HZGU3W3EROFL23BCK2EW3YN6CAD3NO5OVE5QM3QG3Q, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: read tcp 188.166.225.170:42148->145.79.227.14:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:12:04.270571570 (176h8m2.822042121s) [noise=true , address=145.79.227.31:4797 , piece_id=AWSPIPRKSBOKQFZRRB34RUPLYVJGKAOOJQ4U7RRATQJELM3OCFHA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection
Nov 16 16:12:04.270609783 (176h8m2.822080254s) [noise=true , address=145.79.227.22:4511 , piece_id=HES465EP3OPD5JBVGVYMGLUOHJKSSMJDXMJVWET5GHRPFZJFIFOA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: read tcp 188.166.225.170:51004->145.79.227.22:4511: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:12:04.270625552 (176h8m2.822096083s) [noise=true , address=145.79.227.8:4331 , piece_id=4O27HU7AMR6G4Y43HJYN22UXVLETBDBE7JHL4JUZKRM54SK5E4JA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: read tcp 188.166.225.170:44510->145.79.227.8:4331: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:12:04.270742155 (176h8m2.822212616s) [noise=true , address=145.79.227.4:4186 , piece_id=5PZIK3V72CUQQ6M2HJZFVM5OVX4HPHY7ZHJ2BAK5S3ZQOGM4J5OA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection
Nov 16 16:12:04.270762814 (176h8m2.822233275s) [noise=true , address=145.79.227.24:4581 , piece_id=XRCS52EJDB7HWD4RDZK3UNB6VKHOTWWID42RGBJZQHTGG4NDYIPA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: read tcp 188.166.225.170:44130->145.79.227.24:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:12:04.270781470 (176h8m2.822251921s) [noise=true , address=145.79.227.11:4732 , piece_id=FQGAVGZ5CID3LBET3CVK6FSCOP5OUYGS44XYWMHU2RLIT3MBN62A, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: read tcp 188.166.225.170:42420->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:12:04.270825082 (176h8m2.822295544s) [seg_pos={0 0}, upload=155107]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil>
Nov 16 16:12:05.262661603 (176h8m3.814132064s) [upload=155107]: Done uploading segments...
Nov 16 16:12:05.262728080 (176h8m3.814198541s) [upload=155107]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4)
Duration: 5.886547414s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [......................................X.....................] duration:
5.677878447s trace:
https://jaeger.dev.storj.io/trace/7dbd01be7ae5d01e
FAIL (2025-11-16 16:18:21 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 24)
======LOGS======
OHA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.079132696 (176h0m30.630603157s) [noise=true , address=81.171.14.70:4040 , piece_id=YG6ZBK6QTKUS3FCFD2EWU7HTFLOEEVG57SIER44E67QDP5HYCHOA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.079319683 (176h0m30.630790144s) [noise=true , address=185.167.204.205:4015 , piece_id=6UIFX7GXE2SZ465VAP56YQILR3A3OHDRKIS345EE6GBDG77KJNZQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.079646465 (176h0m30.631116926s) [noise=true , address=185.167.204.206:4062 , piece_id=PAJX3ZCWM5YBZA7YDBFYXCET5CKWBOFJ47IUWK4CYPZSWK6HYQVQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.080002833 (176h0m30.631473294s) [noise=true , address=81.171.14.70:4031 , piece_id=54T5MOZZ4YJKQFUDOCCFF2SAE7XS2N77H2AOHCBECAWE5OBUXTZQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.080062066 (176h0m30.631532527s) [noise=true , address=185.167.204.206:4067 , piece_id=2BDMQZLCV4UWWP5HB6GF6NCL5UOGNU7VNYGJIIJCKQHRTX5OPGPQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.080438673 (176h0m30.631909134s) [noise=true , address=185.167.204.205:4034 , piece_id=CZFPF35OQ3U4IH46KILFPFMVY6OC36ZUGZ7EYYBOTB3I4TXMPERA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.080675103 (176h0m30.632145564s) [noise=true , address=185.167.204.205:4014 , piece_id=57C2VUYOEULOTXUQHG7A2XOWZLXYFIEYYRYW5ARX5PKTGWKIEOAQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.081315833 (176h0m30.632786294s) [noise=true , address=81.171.14.70:4039 , piece_id=NMZ5357FVIKD2F3GCKY6YNNV73BAID2YBLVNSSM6VD5PHGLBHEIA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.082053738 (176h0m30.633524189s) [noise=true , address=185.167.204.205:4001 , piece_id=TBPYFHMXWO53LJQQDUA3I2MD2WP6BHQXAH3KIF6XXMWK27HXLCHA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.082605429 (176h0m30.63407589s) [noise=true , address=81.171.14.70:4034 , piece_id=NXBDCCCEWSZ2TMSET6OXBNQV6Q5ESXIQJUFK4WJJ3UV7FJTTAUXQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:33.280203414 (176h0m31.831673875s) [upload=155002]: Next returned nil segment
Nov 16 16:04:33.280222810 (176h0m31.831693271s) [upload=155002]: Waiting for error group managing segments...
Nov 16 16:04:33.838289781 (176h0m32.389760242s) [noise=true , address=185.167.204.205:4013 , piece_id=KF32EXUYGKQEDBPRSVAMNZV3ZMBF6DLT3U32BKX5LXLRBXBVTOQA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.848846382 (176h0m32.400316833s) [noise=true , address=185.167.204.206:4039 , piece_id=6HTUHESIRTSKR5EDMWZ2K56OKE5Y22QQM5ARYTUXR7KMKWIOS7GQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.862791868 (176h0m32.414262329s) [noise=true , address=185.167.204.205:4021 , piece_id=MV3TV4EI3HN3J5NY5XYVKT46QJTJEC5PITXERSSZMS2B7W3DSDTA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.879947135 (176h0m32.431417596s) [noise=true , address=185.167.204.205:4001 , piece_id=TBPYFHMXWO53LJQQDUA3I2MD2WP6BHQXAH3KIF6XXMWK27HXLCHA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.886853657 (176h0m32.438324118s) [noise=true , address=185.167.204.206:4067 , piece_id=2BDMQZLCV4UWWP5HB6GF6NCL5UOGNU7VNYGJIIJCKQHRTX5OPGPQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.918801783 (176h0m32.470272324s) [noise=true , address=185.167.204.206:4051 , piece_id=AU2S6M3E2CQEMAS7LKZJIVWITZUHIHCMWQ2QWT7ILGBRTAUCUB7Q, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.929982083 (176h0m32.481452574s) [noise=true , address=185.167.204.205:4015 , piece_id=6UIFX7GXE2SZ465VAP56YQILR3A3OHDRKIS345EE6GBDG77KJNZQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.935247409 (176h0m32.48671786s) [noise=true , address=185.167.204.206:4038 , piece_id=FBNPAJOOXEAT3KLKQIYOWHGHRGAWJABV4LAA3UYX6SGJB3U2LD7A, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.969630813 (176h0m32.521101264s) [noise=true , address=185.167.204.206:4055 , piece_id=B2TTK2ZLIRM3YIKP5HQ3WOL26KBHSXWWX5ZXXCMWU34AH7CHOOCQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.999043600 (176h0m32.550514061s) [noise=true , address=185.167.204.205:4034 , piece_id=CZFPF35OQ3U4IH46KILFPFMVY6OC36ZUGZ7EYYBOTB3I4TXMPERA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.022951616 (176h0m32.574422077s) [noise=true , address=185.167.204.205:4014 , piece_id=57C2VUYOEULOTXUQHG7A2XOWZLXYFIEYYRYW5ARX5PKTGWKIEOAQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.064488381 (176h0m32.615958832s) [noise=true , address=185.167.204.206:4058 , piece_id=POK7HBLE5AT2STCSL2UJ5R67LNVK3PN53NIRIBX4M7LJYC4C2CHQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.122257987 (176h0m32.673728468s) [noise=true , address=185.167.204.205:4029 , piece_id=VF7BTI4K75HTRVJOHIKMDJN3UBHZ7KCQBT3H44XB5G7ZVFD4HXJQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.136282884 (176h0m32.687753345s) [noise=true , address=185.167.204.206:4041 , piece_id=RTQSCXR54KTKBYS6AHEKPO7ZPUHVQQNUNQJ6JI4JLTGKOLUZH3AQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.136821679 (176h0m32.68829215s) [noise=true , address=185.167.204.206:4062 , piece_id=PAJX3ZCWM5YBZA7YDBFYXCET5CKWBOFJ47IUWK4CYPZSWK6HYQVQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.156096042 (176h0m32.707566493s) [noise=true , address=185.167.204.206:4066 , piece_id=CPTBHDHCT6R2M3UA6X2ZWMPJUKAUVV4I2CIATZ2ZOIWH63Q3UT4Q, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.540981809 (176h0m33.0924523s) [noise=true , address=89.149.201.147:4012 , piece_id=NEIYQHVVWRUVJFF5UD3RZSRHUSNSKXBCNGTMIGOH5MDH2AOZISCA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:21.258617296 (176h1m19.810087757s) [noise=true , address=81.171.14.70:4034 , piece_id=NXBDCCCEWSZ2TMSET6OXBNQV6Q5ESXIQJUFK4WJJ3UV7FJTTAUXQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:22.223432573 (176h1m20.774903034s) [noise=true , address=81.171.14.70:4048 , piece_id=5WUEBVU7UFKHYZ23BPIQUMUUBH3ZXXQWBLE7MALJGECCDTTEK5EA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:22.238494073 (176h1m20.789964534s) [noise=true , address=81.171.14.70:4033 , piece_id=LUBQIVFAU7J72K5XDDS5S2ZXQUGFA5AINL4R5FV64KY24VS5KV5Q, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:22.350384206 (176h1m20.901854667s) [noise=true , address=81.171.14.70:4039 , piece_id=NMZ5357FVIKD2F3GCKY6YNNV73BAID2YBLVNSSM6VD5PHGLBHEIA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:23.704484814 (176h1m22.255955275s) [noise=true , address=81.171.14.70:4037 , piece_id=3IJ7J7RBMOORW4YPKIYJ3C6ZQRYBH7WQMLEKEQYG5UJ4TGKE2W2A, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:24.501497158 (176h1m23.052967619s) [noise=true , address=81.171.14.70:4031 , piece_id=54T5MOZZ4YJKQFUDOCCFF2SAE7XS2N77H2AOHCBECAWE5OBUXTZQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:29.810461698 (176h1m28.361932159s) [noise=true , address=81.171.14.70:4025 , piece_id=3HAT6FJSGV6VOHJBMO6L6L7677BOYVCDV3SR26LJZ6HZOZWJJ64A, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:31.413018058 (176h1m29.964488509s) [noise=true , address=81.171.14.70:4040 , piece_id=YG6ZBK6QTKUS3FCFD2EWU7HTFLOEEVG57SIER44E67QDP5HYCHOA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:1gannW154cmVP6N5BowytKfDorZDRwx2M5pi6akqjZFU38PGjV, address:81.171.14.70:4040): protocol: write tcp 188.166.225.170:59244->81.171.14.70:4040: use of closed network connection; write tcp 188.166.225.170:59244->81.171.14.70:4040: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59244->81.171.14.70:4040: use of closed network connection
Nov 16 16:05:31.413111366 (176h1m29.964581837s) [noise=true , address=81.171.14.70:4035 , piece_id=RGMXRH2Y5XBAFISK5LNAF547SAORD6XSWFUKYWOPJINXXODI5KVQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:12eLYJKne3yP1fZ1oPDLM7Xqs3GQ3ju4ZQedAiojXdVkcxNpzeA, address:81.171.14.70:4035): protocol: read tcp 188.166.225.170:55142->81.171.14.70:4035: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:05:31.413619965 (176h1m29.965090436s) [noise=true , address=81.171.14.70:4030 , piece_id=ZE5OAUTWLWNSMWSFGPEJTCXEN6L3FLWL34QBERTS3JZE7X2I2OHA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:12ZoRwa8wLw66Pjumy1ybp6wrZ79RrSU55uYZpwaY3xi95iSYde, address:81.171.14.70:4030): protocol: read tcp 188.166.225.170:40822->81.171.14.70:4030: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:05:31.414846280 (176h1m29.966316741s) [noise=true , address=81.171.14.70:4029 , piece_id=SIXL2XSCF7IO2BPOUKGH5VVXI6M7O4KJVZARYIRELR2GM4T4RABA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:18yEfU2X4jSrom1JDtQ22cUEvomcg91ohvay2Nsr1Vr8juo9to, address:81.171.14.70:4029): protocol: read tcp 188.166.225.170:56584->81.171.14.70:4029: use of closed network connection; conn closed: op
Nov 16 16:05:31.414946261 (176h1m29.966416722s) [seg_pos={0 0}, upload=155002]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil>
Nov 16 16:05:31.591745995 (176h1m30.143216456s) [upload=155002]: Done uploading segments...
Nov 16 16:05:31.591788596 (176h1m30.143259057s) [upload=155002]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24)
Duration: 5.677907413s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..................................................X.........] duration:
6.04661753s trace:
https://jaeger.dev.storj.io/trace/3e4560ab56ee4602
FAIL (2025-11-16 16:18:07 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.046649431s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.................................................X..........] duration:
5.965361992s trace:
https://jaeger.dev.storj.io/trace/70f81d2d9b012a3a
FAIL (2025-11-16 16:18:03 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 4)
======LOGS======
ddress=45.91.116.5:4297 , piece_id=OSMI3VGAKSCIZMDQ3Y2VZQBUOJKSXM6RWNWSAV57RDTL5TR3NIJA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.462247309 (176h7m35.013717781s) [noise=true , address=45.91.116.7:4011 , piece_id=ORJVRMFIOEXZ6OB6QFQDL764Q2FCWOJT63NBIRI6SOOS2FEIBCFQ, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.463420774 (176h7m35.014891235s) [noise=true , address=145.79.227.9:4367 , piece_id=LOFHPWCHNEZ4E4WANJM5BE2TIWVFH4LGEJJYK4ETB2LZWYCAVGZA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.464113112 (176h7m35.015583573s) [noise=true , address=145.79.227.22:4511 , piece_id=HES465EP3OPD5JBVGVYMGLUOHJKSSMJDXMJVWET5GHRPFZJFIFOA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.464764873 (176h7m35.016235334s) [noise=true , address=45.91.116.8:4048 , piece_id=ULVK52VWCW3KNEPWSDYGRHWHIJWEQU2XNFEPYJFXWLVJI6AGZQOA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.465452753 (176h7m35.016923214s) [noise=true , address=145.79.227.28:4725 , piece_id=CI2LSBGPW7RYH4RGWODGWLUTN353EODNV6IB2YQGHOZ3FPSEEK5A, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.466223270 (176h7m35.017693731s) [noise=true , address=145.79.227.16:4910 , piece_id=3U2SEVMPILIEBRHS3M4YT4Z4PSYAQAIC2SKGYMW4FNODACZLOQCQ, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.466955855 (176h7m35.018426306s) [noise=true , address=145.79.227.4:4186 , piece_id=5PZIK3V72CUQQ6M2HJZFVM5OVX4HPHY7ZHJ2BAK5S3ZQOGM4J5OA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.467675205 (176h7m35.019145666s) [noise=true , address=145.79.227.8:4331 , piece_id=4O27HU7AMR6G4Y43HJYN22UXVLETBDBE7JHL4JUZKRM54SK5E4JA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.468178092 (176h7m35.019648553s) [noise=true , address=145.79.227.8:4329 , piece_id=GO5IAWQYS3UARWYAQAEUIIS456DFWCFWRJQG2KOVVL5KPXWPK4AA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:39.111804678 (176h7m37.663275129s) [upload=155107]: Next returned nil segment
Nov 16 16:11:39.111825767 (176h7m37.663296218s) [upload=155107]: Waiting for error group managing segments...
Nov 16 16:11:39.896671313 (176h7m38.448141765s) [noise=true , address=45.91.116.12:4227 , piece_id=CQA2LHRSFOA7EIBIFVC75EWVHFZMWMY7ODMETEAJRRNYSQFY5TVQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil>
Nov 16 16:11:39.954993420 (176h7m38.506463881s) [noise=true , address=45.91.116.8:4048 , piece_id=ULVK52VWCW3KNEPWSDYGRHWHIJWEQU2XNFEPYJFXWLVJI6AGZQOA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil>
Nov 16 16:11:40.066014029 (176h7m38.61748449s) [noise=true , address=45.91.116.7:4011 , piece_id=ORJVRMFIOEXZ6OB6QFQDL764Q2FCWOJT63NBIRI6SOOS2FEIBCFQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil>
Nov 16 16:11:41.102688250 (176h7m39.654158711s) [noise=true , address=45.91.116.5:4297 , piece_id=OSMI3VGAKSCIZMDQ3Y2VZQBUOJKSXM6RWNWSAV57RDTL5TR3NIJA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil>
Nov 16 16:12:04.264842280 (176h8m2.816312741s) [noise=true , address=145.79.227.17:4972 , piece_id=NV6FSBQM6YUTKYFAJPQCURWCZBHJ3AKJOLLGCRYTJ3KQ54U4GOWQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection
Nov 16 16:12:04.265084832 (176h8m2.816555293s) [noise=true , address=145.79.227.16:4910 , piece_id=3U2SEVMPILIEBRHS3M4YT4Z4PSYAQAIC2SKGYMW4FNODACZLOQCQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection
Nov 16 16:12:04.270267942 (176h8m2.821738423s) [noise=true , address=145.79.227.8:4329 , piece_id=GO5IAWQYS3UARWYAQAEUIIS456DFWCFWRJQG2KOVVL5KPXWPK4AA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: expected piece hash; context deadline exceeded; EOF
Nov 16 16:12:04.270306515 (176h8m2.821776986s) [noise=true , address=145.79.227.9:4367 , piece_id=LOFHPWCHNEZ4E4WANJM5BE2TIWVFH4LGEJJYK4ETB2LZWYCAVGZA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection
Nov 16 16:12:04.270323538 (176h8m2.821794099s) [noise=true , address=145.79.227.28:4725 , piece_id=CI2LSBGPW7RYH4RGWODGWLUTN353EODNV6IB2YQGHOZ3FPSEEK5A, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection
Nov 16 16:12:04.270544138 (176h8m2.822014739s) [noise=true , address=145.79.227.14:4840 , piece_id=RSHGL65RI7HZGU3W3EROFL23BCK2EW3YN6CAD3NO5OVE5QM3QG3Q, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: read tcp 188.166.225.170:42148->145.79.227.14:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:12:04.270571570 (176h8m2.822042121s) [noise=true , address=145.79.227.31:4797 , piece_id=AWSPIPRKSBOKQFZRRB34RUPLYVJGKAOOJQ4U7RRATQJELM3OCFHA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection
Nov 16 16:12:04.270609783 (176h8m2.822080254s) [noise=true , address=145.79.227.22:4511 , piece_id=HES465EP3OPD5JBVGVYMGLUOHJKSSMJDXMJVWET5GHRPFZJFIFOA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: read tcp 188.166.225.170:51004->145.79.227.22:4511: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:12:04.270625552 (176h8m2.822096083s) [noise=true , address=145.79.227.8:4331 , piece_id=4O27HU7AMR6G4Y43HJYN22UXVLETBDBE7JHL4JUZKRM54SK5E4JA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: read tcp 188.166.225.170:44510->145.79.227.8:4331: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:12:04.270742155 (176h8m2.822212616s) [noise=true , address=145.79.227.4:4186 , piece_id=5PZIK3V72CUQQ6M2HJZFVM5OVX4HPHY7ZHJ2BAK5S3ZQOGM4J5OA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection
Nov 16 16:12:04.270762814 (176h8m2.822233275s) [noise=true , address=145.79.227.24:4581 , piece_id=XRCS52EJDB7HWD4RDZK3UNB6VKHOTWWID42RGBJZQHTGG4NDYIPA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: read tcp 188.166.225.170:44130->145.79.227.24:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:12:04.270781470 (176h8m2.822251921s) [noise=true , address=145.79.227.11:4732 , piece_id=FQGAVGZ5CID3LBET3CVK6FSCOP5OUYGS44XYWMHU2RLIT3MBN62A, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: read tcp 188.166.225.170:42420->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:12:04.270825082 (176h8m2.822295544s) [seg_pos={0 0}, upload=155107]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil>
Nov 16 16:12:05.262661603 (176h8m3.814132064s) [upload=155107]: Done uploading segments...
Nov 16 16:12:05.262728080 (176h8m3.814198541s) [upload=155107]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4)
Duration: 5.965393363s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [.......................................X....................] duration:
5.67861221s trace:
https://jaeger.dev.storj.io/trace/7dbd01be7ae5d01e
FAIL (2025-11-16 16:17:54 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 24)
======LOGS======
OHA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.079132696 (176h0m30.630603157s) [noise=true , address=81.171.14.70:4040 , piece_id=YG6ZBK6QTKUS3FCFD2EWU7HTFLOEEVG57SIER44E67QDP5HYCHOA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.079319683 (176h0m30.630790144s) [noise=true , address=185.167.204.205:4015 , piece_id=6UIFX7GXE2SZ465VAP56YQILR3A3OHDRKIS345EE6GBDG77KJNZQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.079646465 (176h0m30.631116926s) [noise=true , address=185.167.204.206:4062 , piece_id=PAJX3ZCWM5YBZA7YDBFYXCET5CKWBOFJ47IUWK4CYPZSWK6HYQVQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.080002833 (176h0m30.631473294s) [noise=true , address=81.171.14.70:4031 , piece_id=54T5MOZZ4YJKQFUDOCCFF2SAE7XS2N77H2AOHCBECAWE5OBUXTZQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.080062066 (176h0m30.631532527s) [noise=true , address=185.167.204.206:4067 , piece_id=2BDMQZLCV4UWWP5HB6GF6NCL5UOGNU7VNYGJIIJCKQHRTX5OPGPQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.080438673 (176h0m30.631909134s) [noise=true , address=185.167.204.205:4034 , piece_id=CZFPF35OQ3U4IH46KILFPFMVY6OC36ZUGZ7EYYBOTB3I4TXMPERA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.080675103 (176h0m30.632145564s) [noise=true , address=185.167.204.205:4014 , piece_id=57C2VUYOEULOTXUQHG7A2XOWZLXYFIEYYRYW5ARX5PKTGWKIEOAQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.081315833 (176h0m30.632786294s) [noise=true , address=81.171.14.70:4039 , piece_id=NMZ5357FVIKD2F3GCKY6YNNV73BAID2YBLVNSSM6VD5PHGLBHEIA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.082053738 (176h0m30.633524189s) [noise=true , address=185.167.204.205:4001 , piece_id=TBPYFHMXWO53LJQQDUA3I2MD2WP6BHQXAH3KIF6XXMWK27HXLCHA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.082605429 (176h0m30.63407589s) [noise=true , address=81.171.14.70:4034 , piece_id=NXBDCCCEWSZ2TMSET6OXBNQV6Q5ESXIQJUFK4WJJ3UV7FJTTAUXQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:33.280203414 (176h0m31.831673875s) [upload=155002]: Next returned nil segment
Nov 16 16:04:33.280222810 (176h0m31.831693271s) [upload=155002]: Waiting for error group managing segments...
Nov 16 16:04:33.838289781 (176h0m32.389760242s) [noise=true , address=185.167.204.205:4013 , piece_id=KF32EXUYGKQEDBPRSVAMNZV3ZMBF6DLT3U32BKX5LXLRBXBVTOQA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.848846382 (176h0m32.400316833s) [noise=true , address=185.167.204.206:4039 , piece_id=6HTUHESIRTSKR5EDMWZ2K56OKE5Y22QQM5ARYTUXR7KMKWIOS7GQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.862791868 (176h0m32.414262329s) [noise=true , address=185.167.204.205:4021 , piece_id=MV3TV4EI3HN3J5NY5XYVKT46QJTJEC5PITXERSSZMS2B7W3DSDTA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.879947135 (176h0m32.431417596s) [noise=true , address=185.167.204.205:4001 , piece_id=TBPYFHMXWO53LJQQDUA3I2MD2WP6BHQXAH3KIF6XXMWK27HXLCHA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.886853657 (176h0m32.438324118s) [noise=true , address=185.167.204.206:4067 , piece_id=2BDMQZLCV4UWWP5HB6GF6NCL5UOGNU7VNYGJIIJCKQHRTX5OPGPQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.918801783 (176h0m32.470272324s) [noise=true , address=185.167.204.206:4051 , piece_id=AU2S6M3E2CQEMAS7LKZJIVWITZUHIHCMWQ2QWT7ILGBRTAUCUB7Q, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.929982083 (176h0m32.481452574s) [noise=true , address=185.167.204.205:4015 , piece_id=6UIFX7GXE2SZ465VAP56YQILR3A3OHDRKIS345EE6GBDG77KJNZQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.935247409 (176h0m32.48671786s) [noise=true , address=185.167.204.206:4038 , piece_id=FBNPAJOOXEAT3KLKQIYOWHGHRGAWJABV4LAA3UYX6SGJB3U2LD7A, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.969630813 (176h0m32.521101264s) [noise=true , address=185.167.204.206:4055 , piece_id=B2TTK2ZLIRM3YIKP5HQ3WOL26KBHSXWWX5ZXXCMWU34AH7CHOOCQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.999043600 (176h0m32.550514061s) [noise=true , address=185.167.204.205:4034 , piece_id=CZFPF35OQ3U4IH46KILFPFMVY6OC36ZUGZ7EYYBOTB3I4TXMPERA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.022951616 (176h0m32.574422077s) [noise=true , address=185.167.204.205:4014 , piece_id=57C2VUYOEULOTXUQHG7A2XOWZLXYFIEYYRYW5ARX5PKTGWKIEOAQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.064488381 (176h0m32.615958832s) [noise=true , address=185.167.204.206:4058 , piece_id=POK7HBLE5AT2STCSL2UJ5R67LNVK3PN53NIRIBX4M7LJYC4C2CHQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.122257987 (176h0m32.673728468s) [noise=true , address=185.167.204.205:4029 , piece_id=VF7BTI4K75HTRVJOHIKMDJN3UBHZ7KCQBT3H44XB5G7ZVFD4HXJQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.136282884 (176h0m32.687753345s) [noise=true , address=185.167.204.206:4041 , piece_id=RTQSCXR54KTKBYS6AHEKPO7ZPUHVQQNUNQJ6JI4JLTGKOLUZH3AQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.136821679 (176h0m32.68829215s) [noise=true , address=185.167.204.206:4062 , piece_id=PAJX3ZCWM5YBZA7YDBFYXCET5CKWBOFJ47IUWK4CYPZSWK6HYQVQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.156096042 (176h0m32.707566493s) [noise=true , address=185.167.204.206:4066 , piece_id=CPTBHDHCT6R2M3UA6X2ZWMPJUKAUVV4I2CIATZ2ZOIWH63Q3UT4Q, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.540981809 (176h0m33.0924523s) [noise=true , address=89.149.201.147:4012 , piece_id=NEIYQHVVWRUVJFF5UD3RZSRHUSNSKXBCNGTMIGOH5MDH2AOZISCA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:21.258617296 (176h1m19.810087757s) [noise=true , address=81.171.14.70:4034 , piece_id=NXBDCCCEWSZ2TMSET6OXBNQV6Q5ESXIQJUFK4WJJ3UV7FJTTAUXQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:22.223432573 (176h1m20.774903034s) [noise=true , address=81.171.14.70:4048 , piece_id=5WUEBVU7UFKHYZ23BPIQUMUUBH3ZXXQWBLE7MALJGECCDTTEK5EA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:22.238494073 (176h1m20.789964534s) [noise=true , address=81.171.14.70:4033 , piece_id=LUBQIVFAU7J72K5XDDS5S2ZXQUGFA5AINL4R5FV64KY24VS5KV5Q, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:22.350384206 (176h1m20.901854667s) [noise=true , address=81.171.14.70:4039 , piece_id=NMZ5357FVIKD2F3GCKY6YNNV73BAID2YBLVNSSM6VD5PHGLBHEIA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:23.704484814 (176h1m22.255955275s) [noise=true , address=81.171.14.70:4037 , piece_id=3IJ7J7RBMOORW4YPKIYJ3C6ZQRYBH7WQMLEKEQYG5UJ4TGKE2W2A, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:24.501497158 (176h1m23.052967619s) [noise=true , address=81.171.14.70:4031 , piece_id=54T5MOZZ4YJKQFUDOCCFF2SAE7XS2N77H2AOHCBECAWE5OBUXTZQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:29.810461698 (176h1m28.361932159s) [noise=true , address=81.171.14.70:4025 , piece_id=3HAT6FJSGV6VOHJBMO6L6L7677BOYVCDV3SR26LJZ6HZOZWJJ64A, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:31.413018058 (176h1m29.964488509s) [noise=true , address=81.171.14.70:4040 , piece_id=YG6ZBK6QTKUS3FCFD2EWU7HTFLOEEVG57SIER44E67QDP5HYCHOA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:1gannW154cmVP6N5BowytKfDorZDRwx2M5pi6akqjZFU38PGjV, address:81.171.14.70:4040): protocol: write tcp 188.166.225.170:59244->81.171.14.70:4040: use of closed network connection; write tcp 188.166.225.170:59244->81.171.14.70:4040: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59244->81.171.14.70:4040: use of closed network connection
Nov 16 16:05:31.413111366 (176h1m29.964581837s) [noise=true , address=81.171.14.70:4035 , piece_id=RGMXRH2Y5XBAFISK5LNAF547SAORD6XSWFUKYWOPJINXXODI5KVQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:12eLYJKne3yP1fZ1oPDLM7Xqs3GQ3ju4ZQedAiojXdVkcxNpzeA, address:81.171.14.70:4035): protocol: read tcp 188.166.225.170:55142->81.171.14.70:4035: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:05:31.413619965 (176h1m29.965090436s) [noise=true , address=81.171.14.70:4030 , piece_id=ZE5OAUTWLWNSMWSFGPEJTCXEN6L3FLWL34QBERTS3JZE7X2I2OHA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:12ZoRwa8wLw66Pjumy1ybp6wrZ79RrSU55uYZpwaY3xi95iSYde, address:81.171.14.70:4030): protocol: read tcp 188.166.225.170:40822->81.171.14.70:4030: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:05:31.414846280 (176h1m29.966316741s) [noise=true , address=81.171.14.70:4029 , piece_id=SIXL2XSCF7IO2BPOUKGH5VVXI6M7O4KJVZARYIRELR2GM4T4RABA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:18yEfU2X4jSrom1JDtQ22cUEvomcg91ohvay2Nsr1Vr8juo9to, address:81.171.14.70:4029): protocol: read tcp 188.166.225.170:56584->81.171.14.70:4029: use of closed network connection; conn closed: op
Nov 16 16:05:31.414946261 (176h1m29.966416722s) [seg_pos={0 0}, upload=155002]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil>
Nov 16 16:05:31.591745995 (176h1m30.143216456s) [upload=155002]: Done uploading segments...
Nov 16 16:05:31.591788596 (176h1m30.143259057s) [upload=155002]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24)
Duration: 5.678632678s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [.............X..............................................] duration:
3.40802744s trace:
https://jaeger.dev.storj.io/trace/1026d6559b04863a
FAIL (2025-11-16 16:17:42 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.40804856s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [........................................X...................] duration:
6.556673001s trace:
https://jaeger.dev.storj.io/trace/7dbd01be7ae5d01e
FAIL (2025-11-16 16:17:36 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 24)
======LOGS======
OHA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.079132696 (176h0m30.630603157s) [noise=true , address=81.171.14.70:4040 , piece_id=YG6ZBK6QTKUS3FCFD2EWU7HTFLOEEVG57SIER44E67QDP5HYCHOA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.079319683 (176h0m30.630790144s) [noise=true , address=185.167.204.205:4015 , piece_id=6UIFX7GXE2SZ465VAP56YQILR3A3OHDRKIS345EE6GBDG77KJNZQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.079646465 (176h0m30.631116926s) [noise=true , address=185.167.204.206:4062 , piece_id=PAJX3ZCWM5YBZA7YDBFYXCET5CKWBOFJ47IUWK4CYPZSWK6HYQVQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.080002833 (176h0m30.631473294s) [noise=true , address=81.171.14.70:4031 , piece_id=54T5MOZZ4YJKQFUDOCCFF2SAE7XS2N77H2AOHCBECAWE5OBUXTZQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.080062066 (176h0m30.631532527s) [noise=true , address=185.167.204.206:4067 , piece_id=2BDMQZLCV4UWWP5HB6GF6NCL5UOGNU7VNYGJIIJCKQHRTX5OPGPQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.080438673 (176h0m30.631909134s) [noise=true , address=185.167.204.205:4034 , piece_id=CZFPF35OQ3U4IH46KILFPFMVY6OC36ZUGZ7EYYBOTB3I4TXMPERA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.080675103 (176h0m30.632145564s) [noise=true , address=185.167.204.205:4014 , piece_id=57C2VUYOEULOTXUQHG7A2XOWZLXYFIEYYRYW5ARX5PKTGWKIEOAQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.081315833 (176h0m30.632786294s) [noise=true , address=81.171.14.70:4039 , piece_id=NMZ5357FVIKD2F3GCKY6YNNV73BAID2YBLVNSSM6VD5PHGLBHEIA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.082053738 (176h0m30.633524189s) [noise=true , address=185.167.204.205:4001 , piece_id=TBPYFHMXWO53LJQQDUA3I2MD2WP6BHQXAH3KIF6XXMWK27HXLCHA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.082605429 (176h0m30.63407589s) [noise=true , address=81.171.14.70:4034 , piece_id=NXBDCCCEWSZ2TMSET6OXBNQV6Q5ESXIQJUFK4WJJ3UV7FJTTAUXQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:33.280203414 (176h0m31.831673875s) [upload=155002]: Next returned nil segment
Nov 16 16:04:33.280222810 (176h0m31.831693271s) [upload=155002]: Waiting for error group managing segments...
Nov 16 16:04:33.838289781 (176h0m32.389760242s) [noise=true , address=185.167.204.205:4013 , piece_id=KF32EXUYGKQEDBPRSVAMNZV3ZMBF6DLT3U32BKX5LXLRBXBVTOQA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.848846382 (176h0m32.400316833s) [noise=true , address=185.167.204.206:4039 , piece_id=6HTUHESIRTSKR5EDMWZ2K56OKE5Y22QQM5ARYTUXR7KMKWIOS7GQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.862791868 (176h0m32.414262329s) [noise=true , address=185.167.204.205:4021 , piece_id=MV3TV4EI3HN3J5NY5XYVKT46QJTJEC5PITXERSSZMS2B7W3DSDTA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.879947135 (176h0m32.431417596s) [noise=true , address=185.167.204.205:4001 , piece_id=TBPYFHMXWO53LJQQDUA3I2MD2WP6BHQXAH3KIF6XXMWK27HXLCHA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.886853657 (176h0m32.438324118s) [noise=true , address=185.167.204.206:4067 , piece_id=2BDMQZLCV4UWWP5HB6GF6NCL5UOGNU7VNYGJIIJCKQHRTX5OPGPQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.918801783 (176h0m32.470272324s) [noise=true , address=185.167.204.206:4051 , piece_id=AU2S6M3E2CQEMAS7LKZJIVWITZUHIHCMWQ2QWT7ILGBRTAUCUB7Q, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.929982083 (176h0m32.481452574s) [noise=true , address=185.167.204.205:4015 , piece_id=6UIFX7GXE2SZ465VAP56YQILR3A3OHDRKIS345EE6GBDG77KJNZQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.935247409 (176h0m32.48671786s) [noise=true , address=185.167.204.206:4038 , piece_id=FBNPAJOOXEAT3KLKQIYOWHGHRGAWJABV4LAA3UYX6SGJB3U2LD7A, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.969630813 (176h0m32.521101264s) [noise=true , address=185.167.204.206:4055 , piece_id=B2TTK2ZLIRM3YIKP5HQ3WOL26KBHSXWWX5ZXXCMWU34AH7CHOOCQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.999043600 (176h0m32.550514061s) [noise=true , address=185.167.204.205:4034 , piece_id=CZFPF35OQ3U4IH46KILFPFMVY6OC36ZUGZ7EYYBOTB3I4TXMPERA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.022951616 (176h0m32.574422077s) [noise=true , address=185.167.204.205:4014 , piece_id=57C2VUYOEULOTXUQHG7A2XOWZLXYFIEYYRYW5ARX5PKTGWKIEOAQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.064488381 (176h0m32.615958832s) [noise=true , address=185.167.204.206:4058 , piece_id=POK7HBLE5AT2STCSL2UJ5R67LNVK3PN53NIRIBX4M7LJYC4C2CHQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.122257987 (176h0m32.673728468s) [noise=true , address=185.167.204.205:4029 , piece_id=VF7BTI4K75HTRVJOHIKMDJN3UBHZ7KCQBT3H44XB5G7ZVFD4HXJQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.136282884 (176h0m32.687753345s) [noise=true , address=185.167.204.206:4041 , piece_id=RTQSCXR54KTKBYS6AHEKPO7ZPUHVQQNUNQJ6JI4JLTGKOLUZH3AQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.136821679 (176h0m32.68829215s) [noise=true , address=185.167.204.206:4062 , piece_id=PAJX3ZCWM5YBZA7YDBFYXCET5CKWBOFJ47IUWK4CYPZSWK6HYQVQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.156096042 (176h0m32.707566493s) [noise=true , address=185.167.204.206:4066 , piece_id=CPTBHDHCT6R2M3UA6X2ZWMPJUKAUVV4I2CIATZ2ZOIWH63Q3UT4Q, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.540981809 (176h0m33.0924523s) [noise=true , address=89.149.201.147:4012 , piece_id=NEIYQHVVWRUVJFF5UD3RZSRHUSNSKXBCNGTMIGOH5MDH2AOZISCA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:21.258617296 (176h1m19.810087757s) [noise=true , address=81.171.14.70:4034 , piece_id=NXBDCCCEWSZ2TMSET6OXBNQV6Q5ESXIQJUFK4WJJ3UV7FJTTAUXQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:22.223432573 (176h1m20.774903034s) [noise=true , address=81.171.14.70:4048 , piece_id=5WUEBVU7UFKHYZ23BPIQUMUUBH3ZXXQWBLE7MALJGECCDTTEK5EA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:22.238494073 (176h1m20.789964534s) [noise=true , address=81.171.14.70:4033 , piece_id=LUBQIVFAU7J72K5XDDS5S2ZXQUGFA5AINL4R5FV64KY24VS5KV5Q, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:22.350384206 (176h1m20.901854667s) [noise=true , address=81.171.14.70:4039 , piece_id=NMZ5357FVIKD2F3GCKY6YNNV73BAID2YBLVNSSM6VD5PHGLBHEIA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:23.704484814 (176h1m22.255955275s) [noise=true , address=81.171.14.70:4037 , piece_id=3IJ7J7RBMOORW4YPKIYJ3C6ZQRYBH7WQMLEKEQYG5UJ4TGKE2W2A, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:24.501497158 (176h1m23.052967619s) [noise=true , address=81.171.14.70:4031 , piece_id=54T5MOZZ4YJKQFUDOCCFF2SAE7XS2N77H2AOHCBECAWE5OBUXTZQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:29.810461698 (176h1m28.361932159s) [noise=true , address=81.171.14.70:4025 , piece_id=3HAT6FJSGV6VOHJBMO6L6L7677BOYVCDV3SR26LJZ6HZOZWJJ64A, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:31.413018058 (176h1m29.964488509s) [noise=true , address=81.171.14.70:4040 , piece_id=YG6ZBK6QTKUS3FCFD2EWU7HTFLOEEVG57SIER44E67QDP5HYCHOA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:1gannW154cmVP6N5BowytKfDorZDRwx2M5pi6akqjZFU38PGjV, address:81.171.14.70:4040): protocol: write tcp 188.166.225.170:59244->81.171.14.70:4040: use of closed network connection; write tcp 188.166.225.170:59244->81.171.14.70:4040: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59244->81.171.14.70:4040: use of closed network connection
Nov 16 16:05:31.413111366 (176h1m29.964581837s) [noise=true , address=81.171.14.70:4035 , piece_id=RGMXRH2Y5XBAFISK5LNAF547SAORD6XSWFUKYWOPJINXXODI5KVQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:12eLYJKne3yP1fZ1oPDLM7Xqs3GQ3ju4ZQedAiojXdVkcxNpzeA, address:81.171.14.70:4035): protocol: read tcp 188.166.225.170:55142->81.171.14.70:4035: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:05:31.413619965 (176h1m29.965090436s) [noise=true , address=81.171.14.70:4030 , piece_id=ZE5OAUTWLWNSMWSFGPEJTCXEN6L3FLWL34QBERTS3JZE7X2I2OHA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:12ZoRwa8wLw66Pjumy1ybp6wrZ79RrSU55uYZpwaY3xi95iSYde, address:81.171.14.70:4030): protocol: read tcp 188.166.225.170:40822->81.171.14.70:4030: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:05:31.414846280 (176h1m29.966316741s) [noise=true , address=81.171.14.70:4029 , piece_id=SIXL2XSCF7IO2BPOUKGH5VVXI6M7O4KJVZARYIRELR2GM4T4RABA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:18yEfU2X4jSrom1JDtQ22cUEvomcg91ohvay2Nsr1Vr8juo9to, address:81.171.14.70:4029): protocol: read tcp 188.166.225.170:56584->81.171.14.70:4029: use of closed network connection; conn closed: op
Nov 16 16:05:31.414946261 (176h1m29.966416722s) [seg_pos={0 0}, upload=155002]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil>
Nov 16 16:05:31.591745995 (176h1m30.143216456s) [upload=155002]: Done uploading segments...
Nov 16 16:05:31.591788596 (176h1m30.143259057s) [upload=155002]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24)
Duration: 6.556704882s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..................................................X.........] duration:
5.883542995s trace:
https://jaeger.dev.storj.io/trace/70f81d2d9b012a3a
FAIL (2025-11-16 16:17:23 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 4)
======LOGS======
ddress=45.91.116.5:4297 , piece_id=OSMI3VGAKSCIZMDQ3Y2VZQBUOJKSXM6RWNWSAV57RDTL5TR3NIJA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.462247309 (176h7m35.013717781s) [noise=true , address=45.91.116.7:4011 , piece_id=ORJVRMFIOEXZ6OB6QFQDL764Q2FCWOJT63NBIRI6SOOS2FEIBCFQ, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.463420774 (176h7m35.014891235s) [noise=true , address=145.79.227.9:4367 , piece_id=LOFHPWCHNEZ4E4WANJM5BE2TIWVFH4LGEJJYK4ETB2LZWYCAVGZA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.464113112 (176h7m35.015583573s) [noise=true , address=145.79.227.22:4511 , piece_id=HES465EP3OPD5JBVGVYMGLUOHJKSSMJDXMJVWET5GHRPFZJFIFOA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.464764873 (176h7m35.016235334s) [noise=true , address=45.91.116.8:4048 , piece_id=ULVK52VWCW3KNEPWSDYGRHWHIJWEQU2XNFEPYJFXWLVJI6AGZQOA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.465452753 (176h7m35.016923214s) [noise=true , address=145.79.227.28:4725 , piece_id=CI2LSBGPW7RYH4RGWODGWLUTN353EODNV6IB2YQGHOZ3FPSEEK5A, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.466223270 (176h7m35.017693731s) [noise=true , address=145.79.227.16:4910 , piece_id=3U2SEVMPILIEBRHS3M4YT4Z4PSYAQAIC2SKGYMW4FNODACZLOQCQ, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.466955855 (176h7m35.018426306s) [noise=true , address=145.79.227.4:4186 , piece_id=5PZIK3V72CUQQ6M2HJZFVM5OVX4HPHY7ZHJ2BAK5S3ZQOGM4J5OA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.467675205 (176h7m35.019145666s) [noise=true , address=145.79.227.8:4331 , piece_id=4O27HU7AMR6G4Y43HJYN22UXVLETBDBE7JHL4JUZKRM54SK5E4JA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.468178092 (176h7m35.019648553s) [noise=true , address=145.79.227.8:4329 , piece_id=GO5IAWQYS3UARWYAQAEUIIS456DFWCFWRJQG2KOVVL5KPXWPK4AA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:39.111804678 (176h7m37.663275129s) [upload=155107]: Next returned nil segment
Nov 16 16:11:39.111825767 (176h7m37.663296218s) [upload=155107]: Waiting for error group managing segments...
Nov 16 16:11:39.896671313 (176h7m38.448141765s) [noise=true , address=45.91.116.12:4227 , piece_id=CQA2LHRSFOA7EIBIFVC75EWVHFZMWMY7ODMETEAJRRNYSQFY5TVQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil>
Nov 16 16:11:39.954993420 (176h7m38.506463881s) [noise=true , address=45.91.116.8:4048 , piece_id=ULVK52VWCW3KNEPWSDYGRHWHIJWEQU2XNFEPYJFXWLVJI6AGZQOA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil>
Nov 16 16:11:40.066014029 (176h7m38.61748449s) [noise=true , address=45.91.116.7:4011 , piece_id=ORJVRMFIOEXZ6OB6QFQDL764Q2FCWOJT63NBIRI6SOOS2FEIBCFQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil>
Nov 16 16:11:41.102688250 (176h7m39.654158711s) [noise=true , address=45.91.116.5:4297 , piece_id=OSMI3VGAKSCIZMDQ3Y2VZQBUOJKSXM6RWNWSAV57RDTL5TR3NIJA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil>
Nov 16 16:12:04.264842280 (176h8m2.816312741s) [noise=true , address=145.79.227.17:4972 , piece_id=NV6FSBQM6YUTKYFAJPQCURWCZBHJ3AKJOLLGCRYTJ3KQ54U4GOWQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection
Nov 16 16:12:04.265084832 (176h8m2.816555293s) [noise=true , address=145.79.227.16:4910 , piece_id=3U2SEVMPILIEBRHS3M4YT4Z4PSYAQAIC2SKGYMW4FNODACZLOQCQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection
Nov 16 16:12:04.270267942 (176h8m2.821738423s) [noise=true , address=145.79.227.8:4329 , piece_id=GO5IAWQYS3UARWYAQAEUIIS456DFWCFWRJQG2KOVVL5KPXWPK4AA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: expected piece hash; context deadline exceeded; EOF
Nov 16 16:12:04.270306515 (176h8m2.821776986s) [noise=true , address=145.79.227.9:4367 , piece_id=LOFHPWCHNEZ4E4WANJM5BE2TIWVFH4LGEJJYK4ETB2LZWYCAVGZA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection
Nov 16 16:12:04.270323538 (176h8m2.821794099s) [noise=true , address=145.79.227.28:4725 , piece_id=CI2LSBGPW7RYH4RGWODGWLUTN353EODNV6IB2YQGHOZ3FPSEEK5A, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection
Nov 16 16:12:04.270544138 (176h8m2.822014739s) [noise=true , address=145.79.227.14:4840 , piece_id=RSHGL65RI7HZGU3W3EROFL23BCK2EW3YN6CAD3NO5OVE5QM3QG3Q, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: read tcp 188.166.225.170:42148->145.79.227.14:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:12:04.270571570 (176h8m2.822042121s) [noise=true , address=145.79.227.31:4797 , piece_id=AWSPIPRKSBOKQFZRRB34RUPLYVJGKAOOJQ4U7RRATQJELM3OCFHA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection
Nov 16 16:12:04.270609783 (176h8m2.822080254s) [noise=true , address=145.79.227.22:4511 , piece_id=HES465EP3OPD5JBVGVYMGLUOHJKSSMJDXMJVWET5GHRPFZJFIFOA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: read tcp 188.166.225.170:51004->145.79.227.22:4511: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:12:04.270625552 (176h8m2.822096083s) [noise=true , address=145.79.227.8:4331 , piece_id=4O27HU7AMR6G4Y43HJYN22UXVLETBDBE7JHL4JUZKRM54SK5E4JA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: read tcp 188.166.225.170:44510->145.79.227.8:4331: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:12:04.270742155 (176h8m2.822212616s) [noise=true , address=145.79.227.4:4186 , piece_id=5PZIK3V72CUQQ6M2HJZFVM5OVX4HPHY7ZHJ2BAK5S3ZQOGM4J5OA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection
Nov 16 16:12:04.270762814 (176h8m2.822233275s) [noise=true , address=145.79.227.24:4581 , piece_id=XRCS52EJDB7HWD4RDZK3UNB6VKHOTWWID42RGBJZQHTGG4NDYIPA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: read tcp 188.166.225.170:44130->145.79.227.24:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:12:04.270781470 (176h8m2.822251921s) [noise=true , address=145.79.227.11:4732 , piece_id=FQGAVGZ5CID3LBET3CVK6FSCOP5OUYGS44XYWMHU2RLIT3MBN62A, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: read tcp 188.166.225.170:42420->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:12:04.270825082 (176h8m2.822295544s) [seg_pos={0 0}, upload=155107]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil>
Nov 16 16:12:05.262661603 (176h8m3.814132064s) [upload=155107]: Done uploading segments...
Nov 16 16:12:05.262728080 (176h8m3.814198541s) [upload=155107]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4)
Duration: 5.883590526s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [..............X.............................................] duration:
3.564401379s trace:
https://jaeger.dev.storj.io/trace/1026d6559b04863a
FAIL (2025-11-16 16:17:23 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.564444812s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...................................................X........] duration:
6.3176301s trace:
https://jaeger.dev.storj.io/trace/3e4560ab56ee4602
FAIL (2025-11-16 16:17:17 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.317655358s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...................................................X........] duration:
5.824647688s trace:
https://jaeger.dev.storj.io/trace/70f81d2d9b012a3a
FAIL (2025-11-16 16:16:59 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 4)
======LOGS======
ddress=45.91.116.5:4297 , piece_id=OSMI3VGAKSCIZMDQ3Y2VZQBUOJKSXM6RWNWSAV57RDTL5TR3NIJA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.462247309 (176h7m35.013717781s) [noise=true , address=45.91.116.7:4011 , piece_id=ORJVRMFIOEXZ6OB6QFQDL764Q2FCWOJT63NBIRI6SOOS2FEIBCFQ, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.463420774 (176h7m35.014891235s) [noise=true , address=145.79.227.9:4367 , piece_id=LOFHPWCHNEZ4E4WANJM5BE2TIWVFH4LGEJJYK4ETB2LZWYCAVGZA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.464113112 (176h7m35.015583573s) [noise=true , address=145.79.227.22:4511 , piece_id=HES465EP3OPD5JBVGVYMGLUOHJKSSMJDXMJVWET5GHRPFZJFIFOA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.464764873 (176h7m35.016235334s) [noise=true , address=45.91.116.8:4048 , piece_id=ULVK52VWCW3KNEPWSDYGRHWHIJWEQU2XNFEPYJFXWLVJI6AGZQOA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.465452753 (176h7m35.016923214s) [noise=true , address=145.79.227.28:4725 , piece_id=CI2LSBGPW7RYH4RGWODGWLUTN353EODNV6IB2YQGHOZ3FPSEEK5A, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.466223270 (176h7m35.017693731s) [noise=true , address=145.79.227.16:4910 , piece_id=3U2SEVMPILIEBRHS3M4YT4Z4PSYAQAIC2SKGYMW4FNODACZLOQCQ, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.466955855 (176h7m35.018426306s) [noise=true , address=145.79.227.4:4186 , piece_id=5PZIK3V72CUQQ6M2HJZFVM5OVX4HPHY7ZHJ2BAK5S3ZQOGM4J5OA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.467675205 (176h7m35.019145666s) [noise=true , address=145.79.227.8:4331 , piece_id=4O27HU7AMR6G4Y43HJYN22UXVLETBDBE7JHL4JUZKRM54SK5E4JA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.468178092 (176h7m35.019648553s) [noise=true , address=145.79.227.8:4329 , piece_id=GO5IAWQYS3UARWYAQAEUIIS456DFWCFWRJQG2KOVVL5KPXWPK4AA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:39.111804678 (176h7m37.663275129s) [upload=155107]: Next returned nil segment
Nov 16 16:11:39.111825767 (176h7m37.663296218s) [upload=155107]: Waiting for error group managing segments...
Nov 16 16:11:39.896671313 (176h7m38.448141765s) [noise=true , address=45.91.116.12:4227 , piece_id=CQA2LHRSFOA7EIBIFVC75EWVHFZMWMY7ODMETEAJRRNYSQFY5TVQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil>
Nov 16 16:11:39.954993420 (176h7m38.506463881s) [noise=true , address=45.91.116.8:4048 , piece_id=ULVK52VWCW3KNEPWSDYGRHWHIJWEQU2XNFEPYJFXWLVJI6AGZQOA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil>
Nov 16 16:11:40.066014029 (176h7m38.61748449s) [noise=true , address=45.91.116.7:4011 , piece_id=ORJVRMFIOEXZ6OB6QFQDL764Q2FCWOJT63NBIRI6SOOS2FEIBCFQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil>
Nov 16 16:11:41.102688250 (176h7m39.654158711s) [noise=true , address=45.91.116.5:4297 , piece_id=OSMI3VGAKSCIZMDQ3Y2VZQBUOJKSXM6RWNWSAV57RDTL5TR3NIJA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil>
Nov 16 16:12:04.264842280 (176h8m2.816312741s) [noise=true , address=145.79.227.17:4972 , piece_id=NV6FSBQM6YUTKYFAJPQCURWCZBHJ3AKJOLLGCRYTJ3KQ54U4GOWQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection
Nov 16 16:12:04.265084832 (176h8m2.816555293s) [noise=true , address=145.79.227.16:4910 , piece_id=3U2SEVMPILIEBRHS3M4YT4Z4PSYAQAIC2SKGYMW4FNODACZLOQCQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection
Nov 16 16:12:04.270267942 (176h8m2.821738423s) [noise=true , address=145.79.227.8:4329 , piece_id=GO5IAWQYS3UARWYAQAEUIIS456DFWCFWRJQG2KOVVL5KPXWPK4AA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: expected piece hash; context deadline exceeded; EOF
Nov 16 16:12:04.270306515 (176h8m2.821776986s) [noise=true , address=145.79.227.9:4367 , piece_id=LOFHPWCHNEZ4E4WANJM5BE2TIWVFH4LGEJJYK4ETB2LZWYCAVGZA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection
Nov 16 16:12:04.270323538 (176h8m2.821794099s) [noise=true , address=145.79.227.28:4725 , piece_id=CI2LSBGPW7RYH4RGWODGWLUTN353EODNV6IB2YQGHOZ3FPSEEK5A, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection
Nov 16 16:12:04.270544138 (176h8m2.822014739s) [noise=true , address=145.79.227.14:4840 , piece_id=RSHGL65RI7HZGU3W3EROFL23BCK2EW3YN6CAD3NO5OVE5QM3QG3Q, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: read tcp 188.166.225.170:42148->145.79.227.14:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:12:04.270571570 (176h8m2.822042121s) [noise=true , address=145.79.227.31:4797 , piece_id=AWSPIPRKSBOKQFZRRB34RUPLYVJGKAOOJQ4U7RRATQJELM3OCFHA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection
Nov 16 16:12:04.270609783 (176h8m2.822080254s) [noise=true , address=145.79.227.22:4511 , piece_id=HES465EP3OPD5JBVGVYMGLUOHJKSSMJDXMJVWET5GHRPFZJFIFOA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: read tcp 188.166.225.170:51004->145.79.227.22:4511: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:12:04.270625552 (176h8m2.822096083s) [noise=true , address=145.79.227.8:4331 , piece_id=4O27HU7AMR6G4Y43HJYN22UXVLETBDBE7JHL4JUZKRM54SK5E4JA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: read tcp 188.166.225.170:44510->145.79.227.8:4331: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:12:04.270742155 (176h8m2.822212616s) [noise=true , address=145.79.227.4:4186 , piece_id=5PZIK3V72CUQQ6M2HJZFVM5OVX4HPHY7ZHJ2BAK5S3ZQOGM4J5OA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection
Nov 16 16:12:04.270762814 (176h8m2.822233275s) [noise=true , address=145.79.227.24:4581 , piece_id=XRCS52EJDB7HWD4RDZK3UNB6VKHOTWWID42RGBJZQHTGG4NDYIPA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: read tcp 188.166.225.170:44130->145.79.227.24:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:12:04.270781470 (176h8m2.822251921s) [noise=true , address=145.79.227.11:4732 , piece_id=FQGAVGZ5CID3LBET3CVK6FSCOP5OUYGS44XYWMHU2RLIT3MBN62A, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: read tcp 188.166.225.170:42420->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:12:04.270825082 (176h8m2.822295544s) [seg_pos={0 0}, upload=155107]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil>
Nov 16 16:12:05.262661603 (176h8m3.814132064s) [upload=155107]: Done uploading segments...
Nov 16 16:12:05.262728080 (176h8m3.814198541s) [upload=155107]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4)
Duration: 5.824673166s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [.........................................X..................] duration:
5.775760064s trace:
https://jaeger.dev.storj.io/trace/7dbd01be7ae5d01e
FAIL (2025-11-16 16:16:58 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 24)
======LOGS======
OHA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.079132696 (176h0m30.630603157s) [noise=true , address=81.171.14.70:4040 , piece_id=YG6ZBK6QTKUS3FCFD2EWU7HTFLOEEVG57SIER44E67QDP5HYCHOA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.079319683 (176h0m30.630790144s) [noise=true , address=185.167.204.205:4015 , piece_id=6UIFX7GXE2SZ465VAP56YQILR3A3OHDRKIS345EE6GBDG77KJNZQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.079646465 (176h0m30.631116926s) [noise=true , address=185.167.204.206:4062 , piece_id=PAJX3ZCWM5YBZA7YDBFYXCET5CKWBOFJ47IUWK4CYPZSWK6HYQVQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.080002833 (176h0m30.631473294s) [noise=true , address=81.171.14.70:4031 , piece_id=54T5MOZZ4YJKQFUDOCCFF2SAE7XS2N77H2AOHCBECAWE5OBUXTZQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.080062066 (176h0m30.631532527s) [noise=true , address=185.167.204.206:4067 , piece_id=2BDMQZLCV4UWWP5HB6GF6NCL5UOGNU7VNYGJIIJCKQHRTX5OPGPQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.080438673 (176h0m30.631909134s) [noise=true , address=185.167.204.205:4034 , piece_id=CZFPF35OQ3U4IH46KILFPFMVY6OC36ZUGZ7EYYBOTB3I4TXMPERA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.080675103 (176h0m30.632145564s) [noise=true , address=185.167.204.205:4014 , piece_id=57C2VUYOEULOTXUQHG7A2XOWZLXYFIEYYRYW5ARX5PKTGWKIEOAQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.081315833 (176h0m30.632786294s) [noise=true , address=81.171.14.70:4039 , piece_id=NMZ5357FVIKD2F3GCKY6YNNV73BAID2YBLVNSSM6VD5PHGLBHEIA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.082053738 (176h0m30.633524189s) [noise=true , address=185.167.204.205:4001 , piece_id=TBPYFHMXWO53LJQQDUA3I2MD2WP6BHQXAH3KIF6XXMWK27HXLCHA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.082605429 (176h0m30.63407589s) [noise=true , address=81.171.14.70:4034 , piece_id=NXBDCCCEWSZ2TMSET6OXBNQV6Q5ESXIQJUFK4WJJ3UV7FJTTAUXQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:33.280203414 (176h0m31.831673875s) [upload=155002]: Next returned nil segment
Nov 16 16:04:33.280222810 (176h0m31.831693271s) [upload=155002]: Waiting for error group managing segments...
Nov 16 16:04:33.838289781 (176h0m32.389760242s) [noise=true , address=185.167.204.205:4013 , piece_id=KF32EXUYGKQEDBPRSVAMNZV3ZMBF6DLT3U32BKX5LXLRBXBVTOQA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.848846382 (176h0m32.400316833s) [noise=true , address=185.167.204.206:4039 , piece_id=6HTUHESIRTSKR5EDMWZ2K56OKE5Y22QQM5ARYTUXR7KMKWIOS7GQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.862791868 (176h0m32.414262329s) [noise=true , address=185.167.204.205:4021 , piece_id=MV3TV4EI3HN3J5NY5XYVKT46QJTJEC5PITXERSSZMS2B7W3DSDTA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.879947135 (176h0m32.431417596s) [noise=true , address=185.167.204.205:4001 , piece_id=TBPYFHMXWO53LJQQDUA3I2MD2WP6BHQXAH3KIF6XXMWK27HXLCHA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.886853657 (176h0m32.438324118s) [noise=true , address=185.167.204.206:4067 , piece_id=2BDMQZLCV4UWWP5HB6GF6NCL5UOGNU7VNYGJIIJCKQHRTX5OPGPQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.918801783 (176h0m32.470272324s) [noise=true , address=185.167.204.206:4051 , piece_id=AU2S6M3E2CQEMAS7LKZJIVWITZUHIHCMWQ2QWT7ILGBRTAUCUB7Q, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.929982083 (176h0m32.481452574s) [noise=true , address=185.167.204.205:4015 , piece_id=6UIFX7GXE2SZ465VAP56YQILR3A3OHDRKIS345EE6GBDG77KJNZQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.935247409 (176h0m32.48671786s) [noise=true , address=185.167.204.206:4038 , piece_id=FBNPAJOOXEAT3KLKQIYOWHGHRGAWJABV4LAA3UYX6SGJB3U2LD7A, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.969630813 (176h0m32.521101264s) [noise=true , address=185.167.204.206:4055 , piece_id=B2TTK2ZLIRM3YIKP5HQ3WOL26KBHSXWWX5ZXXCMWU34AH7CHOOCQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.999043600 (176h0m32.550514061s) [noise=true , address=185.167.204.205:4034 , piece_id=CZFPF35OQ3U4IH46KILFPFMVY6OC36ZUGZ7EYYBOTB3I4TXMPERA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.022951616 (176h0m32.574422077s) [noise=true , address=185.167.204.205:4014 , piece_id=57C2VUYOEULOTXUQHG7A2XOWZLXYFIEYYRYW5ARX5PKTGWKIEOAQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.064488381 (176h0m32.615958832s) [noise=true , address=185.167.204.206:4058 , piece_id=POK7HBLE5AT2STCSL2UJ5R67LNVK3PN53NIRIBX4M7LJYC4C2CHQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.122257987 (176h0m32.673728468s) [noise=true , address=185.167.204.205:4029 , piece_id=VF7BTI4K75HTRVJOHIKMDJN3UBHZ7KCQBT3H44XB5G7ZVFD4HXJQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.136282884 (176h0m32.687753345s) [noise=true , address=185.167.204.206:4041 , piece_id=RTQSCXR54KTKBYS6AHEKPO7ZPUHVQQNUNQJ6JI4JLTGKOLUZH3AQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.136821679 (176h0m32.68829215s) [noise=true , address=185.167.204.206:4062 , piece_id=PAJX3ZCWM5YBZA7YDBFYXCET5CKWBOFJ47IUWK4CYPZSWK6HYQVQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.156096042 (176h0m32.707566493s) [noise=true , address=185.167.204.206:4066 , piece_id=CPTBHDHCT6R2M3UA6X2ZWMPJUKAUVV4I2CIATZ2ZOIWH63Q3UT4Q, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.540981809 (176h0m33.0924523s) [noise=true , address=89.149.201.147:4012 , piece_id=NEIYQHVVWRUVJFF5UD3RZSRHUSNSKXBCNGTMIGOH5MDH2AOZISCA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:21.258617296 (176h1m19.810087757s) [noise=true , address=81.171.14.70:4034 , piece_id=NXBDCCCEWSZ2TMSET6OXBNQV6Q5ESXIQJUFK4WJJ3UV7FJTTAUXQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:22.223432573 (176h1m20.774903034s) [noise=true , address=81.171.14.70:4048 , piece_id=5WUEBVU7UFKHYZ23BPIQUMUUBH3ZXXQWBLE7MALJGECCDTTEK5EA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:22.238494073 (176h1m20.789964534s) [noise=true , address=81.171.14.70:4033 , piece_id=LUBQIVFAU7J72K5XDDS5S2ZXQUGFA5AINL4R5FV64KY24VS5KV5Q, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:22.350384206 (176h1m20.901854667s) [noise=true , address=81.171.14.70:4039 , piece_id=NMZ5357FVIKD2F3GCKY6YNNV73BAID2YBLVNSSM6VD5PHGLBHEIA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:23.704484814 (176h1m22.255955275s) [noise=true , address=81.171.14.70:4037 , piece_id=3IJ7J7RBMOORW4YPKIYJ3C6ZQRYBH7WQMLEKEQYG5UJ4TGKE2W2A, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:24.501497158 (176h1m23.052967619s) [noise=true , address=81.171.14.70:4031 , piece_id=54T5MOZZ4YJKQFUDOCCFF2SAE7XS2N77H2AOHCBECAWE5OBUXTZQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:29.810461698 (176h1m28.361932159s) [noise=true , address=81.171.14.70:4025 , piece_id=3HAT6FJSGV6VOHJBMO6L6L7677BOYVCDV3SR26LJZ6HZOZWJJ64A, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:31.413018058 (176h1m29.964488509s) [noise=true , address=81.171.14.70:4040 , piece_id=YG6ZBK6QTKUS3FCFD2EWU7HTFLOEEVG57SIER44E67QDP5HYCHOA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:1gannW154cmVP6N5BowytKfDorZDRwx2M5pi6akqjZFU38PGjV, address:81.171.14.70:4040): protocol: write tcp 188.166.225.170:59244->81.171.14.70:4040: use of closed network connection; write tcp 188.166.225.170:59244->81.171.14.70:4040: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59244->81.171.14.70:4040: use of closed network connection
Nov 16 16:05:31.413111366 (176h1m29.964581837s) [noise=true , address=81.171.14.70:4035 , piece_id=RGMXRH2Y5XBAFISK5LNAF547SAORD6XSWFUKYWOPJINXXODI5KVQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:12eLYJKne3yP1fZ1oPDLM7Xqs3GQ3ju4ZQedAiojXdVkcxNpzeA, address:81.171.14.70:4035): protocol: read tcp 188.166.225.170:55142->81.171.14.70:4035: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:05:31.413619965 (176h1m29.965090436s) [noise=true , address=81.171.14.70:4030 , piece_id=ZE5OAUTWLWNSMWSFGPEJTCXEN6L3FLWL34QBERTS3JZE7X2I2OHA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:12ZoRwa8wLw66Pjumy1ybp6wrZ79RrSU55uYZpwaY3xi95iSYde, address:81.171.14.70:4030): protocol: read tcp 188.166.225.170:40822->81.171.14.70:4030: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:05:31.414846280 (176h1m29.966316741s) [noise=true , address=81.171.14.70:4029 , piece_id=SIXL2XSCF7IO2BPOUKGH5VVXI6M7O4KJVZARYIRELR2GM4T4RABA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:18yEfU2X4jSrom1JDtQ22cUEvomcg91ohvay2Nsr1Vr8juo9to, address:81.171.14.70:4029): protocol: read tcp 188.166.225.170:56584->81.171.14.70:4029: use of closed network connection; conn closed: op
Nov 16 16:05:31.414946261 (176h1m29.966416722s) [seg_pos={0 0}, upload=155002]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil>
Nov 16 16:05:31.591745995 (176h1m30.143216456s) [upload=155002]: Done uploading segments...
Nov 16 16:05:31.591788596 (176h1m30.143259057s) [upload=155002]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24)
Duration: 5.775793558s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [....................................................X.......] duration:
11.349012088s trace:
https://jaeger.dev.storj.io/trace/3e4560ab56ee4602
FAIL (2025-11-16 16:16:47 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 11.349041464s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [..........................................X.................] duration:
5.88281882s trace:
https://jaeger.dev.storj.io/trace/7dbd01be7ae5d01e
FAIL (2025-11-16 16:16:31 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 24)
======LOGS======
OHA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.079132696 (176h0m30.630603157s) [noise=true , address=81.171.14.70:4040 , piece_id=YG6ZBK6QTKUS3FCFD2EWU7HTFLOEEVG57SIER44E67QDP5HYCHOA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.079319683 (176h0m30.630790144s) [noise=true , address=185.167.204.205:4015 , piece_id=6UIFX7GXE2SZ465VAP56YQILR3A3OHDRKIS345EE6GBDG77KJNZQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.079646465 (176h0m30.631116926s) [noise=true , address=185.167.204.206:4062 , piece_id=PAJX3ZCWM5YBZA7YDBFYXCET5CKWBOFJ47IUWK4CYPZSWK6HYQVQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.080002833 (176h0m30.631473294s) [noise=true , address=81.171.14.70:4031 , piece_id=54T5MOZZ4YJKQFUDOCCFF2SAE7XS2N77H2AOHCBECAWE5OBUXTZQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.080062066 (176h0m30.631532527s) [noise=true , address=185.167.204.206:4067 , piece_id=2BDMQZLCV4UWWP5HB6GF6NCL5UOGNU7VNYGJIIJCKQHRTX5OPGPQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.080438673 (176h0m30.631909134s) [noise=true , address=185.167.204.205:4034 , piece_id=CZFPF35OQ3U4IH46KILFPFMVY6OC36ZUGZ7EYYBOTB3I4TXMPERA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.080675103 (176h0m30.632145564s) [noise=true , address=185.167.204.205:4014 , piece_id=57C2VUYOEULOTXUQHG7A2XOWZLXYFIEYYRYW5ARX5PKTGWKIEOAQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.081315833 (176h0m30.632786294s) [noise=true , address=81.171.14.70:4039 , piece_id=NMZ5357FVIKD2F3GCKY6YNNV73BAID2YBLVNSSM6VD5PHGLBHEIA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.082053738 (176h0m30.633524189s) [noise=true , address=185.167.204.205:4001 , piece_id=TBPYFHMXWO53LJQQDUA3I2MD2WP6BHQXAH3KIF6XXMWK27HXLCHA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.082605429 (176h0m30.63407589s) [noise=true , address=81.171.14.70:4034 , piece_id=NXBDCCCEWSZ2TMSET6OXBNQV6Q5ESXIQJUFK4WJJ3UV7FJTTAUXQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:33.280203414 (176h0m31.831673875s) [upload=155002]: Next returned nil segment
Nov 16 16:04:33.280222810 (176h0m31.831693271s) [upload=155002]: Waiting for error group managing segments...
Nov 16 16:04:33.838289781 (176h0m32.389760242s) [noise=true , address=185.167.204.205:4013 , piece_id=KF32EXUYGKQEDBPRSVAMNZV3ZMBF6DLT3U32BKX5LXLRBXBVTOQA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.848846382 (176h0m32.400316833s) [noise=true , address=185.167.204.206:4039 , piece_id=6HTUHESIRTSKR5EDMWZ2K56OKE5Y22QQM5ARYTUXR7KMKWIOS7GQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.862791868 (176h0m32.414262329s) [noise=true , address=185.167.204.205:4021 , piece_id=MV3TV4EI3HN3J5NY5XYVKT46QJTJEC5PITXERSSZMS2B7W3DSDTA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.879947135 (176h0m32.431417596s) [noise=true , address=185.167.204.205:4001 , piece_id=TBPYFHMXWO53LJQQDUA3I2MD2WP6BHQXAH3KIF6XXMWK27HXLCHA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.886853657 (176h0m32.438324118s) [noise=true , address=185.167.204.206:4067 , piece_id=2BDMQZLCV4UWWP5HB6GF6NCL5UOGNU7VNYGJIIJCKQHRTX5OPGPQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.918801783 (176h0m32.470272324s) [noise=true , address=185.167.204.206:4051 , piece_id=AU2S6M3E2CQEMAS7LKZJIVWITZUHIHCMWQ2QWT7ILGBRTAUCUB7Q, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.929982083 (176h0m32.481452574s) [noise=true , address=185.167.204.205:4015 , piece_id=6UIFX7GXE2SZ465VAP56YQILR3A3OHDRKIS345EE6GBDG77KJNZQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.935247409 (176h0m32.48671786s) [noise=true , address=185.167.204.206:4038 , piece_id=FBNPAJOOXEAT3KLKQIYOWHGHRGAWJABV4LAA3UYX6SGJB3U2LD7A, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.969630813 (176h0m32.521101264s) [noise=true , address=185.167.204.206:4055 , piece_id=B2TTK2ZLIRM3YIKP5HQ3WOL26KBHSXWWX5ZXXCMWU34AH7CHOOCQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.999043600 (176h0m32.550514061s) [noise=true , address=185.167.204.205:4034 , piece_id=CZFPF35OQ3U4IH46KILFPFMVY6OC36ZUGZ7EYYBOTB3I4TXMPERA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.022951616 (176h0m32.574422077s) [noise=true , address=185.167.204.205:4014 , piece_id=57C2VUYOEULOTXUQHG7A2XOWZLXYFIEYYRYW5ARX5PKTGWKIEOAQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.064488381 (176h0m32.615958832s) [noise=true , address=185.167.204.206:4058 , piece_id=POK7HBLE5AT2STCSL2UJ5R67LNVK3PN53NIRIBX4M7LJYC4C2CHQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.122257987 (176h0m32.673728468s) [noise=true , address=185.167.204.205:4029 , piece_id=VF7BTI4K75HTRVJOHIKMDJN3UBHZ7KCQBT3H44XB5G7ZVFD4HXJQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.136282884 (176h0m32.687753345s) [noise=true , address=185.167.204.206:4041 , piece_id=RTQSCXR54KTKBYS6AHEKPO7ZPUHVQQNUNQJ6JI4JLTGKOLUZH3AQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.136821679 (176h0m32.68829215s) [noise=true , address=185.167.204.206:4062 , piece_id=PAJX3ZCWM5YBZA7YDBFYXCET5CKWBOFJ47IUWK4CYPZSWK6HYQVQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.156096042 (176h0m32.707566493s) [noise=true , address=185.167.204.206:4066 , piece_id=CPTBHDHCT6R2M3UA6X2ZWMPJUKAUVV4I2CIATZ2ZOIWH63Q3UT4Q, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.540981809 (176h0m33.0924523s) [noise=true , address=89.149.201.147:4012 , piece_id=NEIYQHVVWRUVJFF5UD3RZSRHUSNSKXBCNGTMIGOH5MDH2AOZISCA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:21.258617296 (176h1m19.810087757s) [noise=true , address=81.171.14.70:4034 , piece_id=NXBDCCCEWSZ2TMSET6OXBNQV6Q5ESXIQJUFK4WJJ3UV7FJTTAUXQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:22.223432573 (176h1m20.774903034s) [noise=true , address=81.171.14.70:4048 , piece_id=5WUEBVU7UFKHYZ23BPIQUMUUBH3ZXXQWBLE7MALJGECCDTTEK5EA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:22.238494073 (176h1m20.789964534s) [noise=true , address=81.171.14.70:4033 , piece_id=LUBQIVFAU7J72K5XDDS5S2ZXQUGFA5AINL4R5FV64KY24VS5KV5Q, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:22.350384206 (176h1m20.901854667s) [noise=true , address=81.171.14.70:4039 , piece_id=NMZ5357FVIKD2F3GCKY6YNNV73BAID2YBLVNSSM6VD5PHGLBHEIA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:23.704484814 (176h1m22.255955275s) [noise=true , address=81.171.14.70:4037 , piece_id=3IJ7J7RBMOORW4YPKIYJ3C6ZQRYBH7WQMLEKEQYG5UJ4TGKE2W2A, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:24.501497158 (176h1m23.052967619s) [noise=true , address=81.171.14.70:4031 , piece_id=54T5MOZZ4YJKQFUDOCCFF2SAE7XS2N77H2AOHCBECAWE5OBUXTZQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:29.810461698 (176h1m28.361932159s) [noise=true , address=81.171.14.70:4025 , piece_id=3HAT6FJSGV6VOHJBMO6L6L7677BOYVCDV3SR26LJZ6HZOZWJJ64A, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:31.413018058 (176h1m29.964488509s) [noise=true , address=81.171.14.70:4040 , piece_id=YG6ZBK6QTKUS3FCFD2EWU7HTFLOEEVG57SIER44E67QDP5HYCHOA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:1gannW154cmVP6N5BowytKfDorZDRwx2M5pi6akqjZFU38PGjV, address:81.171.14.70:4040): protocol: write tcp 188.166.225.170:59244->81.171.14.70:4040: use of closed network connection; write tcp 188.166.225.170:59244->81.171.14.70:4040: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59244->81.171.14.70:4040: use of closed network connection
Nov 16 16:05:31.413111366 (176h1m29.964581837s) [noise=true , address=81.171.14.70:4035 , piece_id=RGMXRH2Y5XBAFISK5LNAF547SAORD6XSWFUKYWOPJINXXODI5KVQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:12eLYJKne3yP1fZ1oPDLM7Xqs3GQ3ju4ZQedAiojXdVkcxNpzeA, address:81.171.14.70:4035): protocol: read tcp 188.166.225.170:55142->81.171.14.70:4035: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:05:31.413619965 (176h1m29.965090436s) [noise=true , address=81.171.14.70:4030 , piece_id=ZE5OAUTWLWNSMWSFGPEJTCXEN6L3FLWL34QBERTS3JZE7X2I2OHA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:12ZoRwa8wLw66Pjumy1ybp6wrZ79RrSU55uYZpwaY3xi95iSYde, address:81.171.14.70:4030): protocol: read tcp 188.166.225.170:40822->81.171.14.70:4030: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:05:31.414846280 (176h1m29.966316741s) [noise=true , address=81.171.14.70:4029 , piece_id=SIXL2XSCF7IO2BPOUKGH5VVXI6M7O4KJVZARYIRELR2GM4T4RABA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:18yEfU2X4jSrom1JDtQ22cUEvomcg91ohvay2Nsr1Vr8juo9to, address:81.171.14.70:4029): protocol: read tcp 188.166.225.170:56584->81.171.14.70:4029: use of closed network connection; conn closed: op
Nov 16 16:05:31.414946261 (176h1m29.966416722s) [seg_pos={0 0}, upload=155002]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil>
Nov 16 16:05:31.591745995 (176h1m30.143216456s) [upload=155002]: Done uploading segments...
Nov 16 16:05:31.591788596 (176h1m30.143259057s) [upload=155002]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24)
Duration: 5.882846132s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [...............X............................................] duration:
4.359759975s trace:
https://jaeger.dev.storj.io/trace/1026d6559b04863a
FAIL (2025-11-16 16:16:21 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.359820962s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [....................................................X.......] duration:
6.037208159s trace:
https://jaeger.dev.storj.io/trace/70f81d2d9b012a3a
FAIL (2025-11-16 16:15:53 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 4)
======LOGS======
ddress=45.91.116.5:4297 , piece_id=OSMI3VGAKSCIZMDQ3Y2VZQBUOJKSXM6RWNWSAV57RDTL5TR3NIJA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.462247309 (176h7m35.013717781s) [noise=true , address=45.91.116.7:4011 , piece_id=ORJVRMFIOEXZ6OB6QFQDL764Q2FCWOJT63NBIRI6SOOS2FEIBCFQ, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.463420774 (176h7m35.014891235s) [noise=true , address=145.79.227.9:4367 , piece_id=LOFHPWCHNEZ4E4WANJM5BE2TIWVFH4LGEJJYK4ETB2LZWYCAVGZA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.464113112 (176h7m35.015583573s) [noise=true , address=145.79.227.22:4511 , piece_id=HES465EP3OPD5JBVGVYMGLUOHJKSSMJDXMJVWET5GHRPFZJFIFOA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.464764873 (176h7m35.016235334s) [noise=true , address=45.91.116.8:4048 , piece_id=ULVK52VWCW3KNEPWSDYGRHWHIJWEQU2XNFEPYJFXWLVJI6AGZQOA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.465452753 (176h7m35.016923214s) [noise=true , address=145.79.227.28:4725 , piece_id=CI2LSBGPW7RYH4RGWODGWLUTN353EODNV6IB2YQGHOZ3FPSEEK5A, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.466223270 (176h7m35.017693731s) [noise=true , address=145.79.227.16:4910 , piece_id=3U2SEVMPILIEBRHS3M4YT4Z4PSYAQAIC2SKGYMW4FNODACZLOQCQ, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.466955855 (176h7m35.018426306s) [noise=true , address=145.79.227.4:4186 , piece_id=5PZIK3V72CUQQ6M2HJZFVM5OVX4HPHY7ZHJ2BAK5S3ZQOGM4J5OA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.467675205 (176h7m35.019145666s) [noise=true , address=145.79.227.8:4331 , piece_id=4O27HU7AMR6G4Y43HJYN22UXVLETBDBE7JHL4JUZKRM54SK5E4JA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.468178092 (176h7m35.019648553s) [noise=true , address=145.79.227.8:4329 , piece_id=GO5IAWQYS3UARWYAQAEUIIS456DFWCFWRJQG2KOVVL5KPXWPK4AA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:39.111804678 (176h7m37.663275129s) [upload=155107]: Next returned nil segment
Nov 16 16:11:39.111825767 (176h7m37.663296218s) [upload=155107]: Waiting for error group managing segments...
Nov 16 16:11:39.896671313 (176h7m38.448141765s) [noise=true , address=45.91.116.12:4227 , piece_id=CQA2LHRSFOA7EIBIFVC75EWVHFZMWMY7ODMETEAJRRNYSQFY5TVQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil>
Nov 16 16:11:39.954993420 (176h7m38.506463881s) [noise=true , address=45.91.116.8:4048 , piece_id=ULVK52VWCW3KNEPWSDYGRHWHIJWEQU2XNFEPYJFXWLVJI6AGZQOA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil>
Nov 16 16:11:40.066014029 (176h7m38.61748449s) [noise=true , address=45.91.116.7:4011 , piece_id=ORJVRMFIOEXZ6OB6QFQDL764Q2FCWOJT63NBIRI6SOOS2FEIBCFQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil>
Nov 16 16:11:41.102688250 (176h7m39.654158711s) [noise=true , address=45.91.116.5:4297 , piece_id=OSMI3VGAKSCIZMDQ3Y2VZQBUOJKSXM6RWNWSAV57RDTL5TR3NIJA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil>
Nov 16 16:12:04.264842280 (176h8m2.816312741s) [noise=true , address=145.79.227.17:4972 , piece_id=NV6FSBQM6YUTKYFAJPQCURWCZBHJ3AKJOLLGCRYTJ3KQ54U4GOWQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection
Nov 16 16:12:04.265084832 (176h8m2.816555293s) [noise=true , address=145.79.227.16:4910 , piece_id=3U2SEVMPILIEBRHS3M4YT4Z4PSYAQAIC2SKGYMW4FNODACZLOQCQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection
Nov 16 16:12:04.270267942 (176h8m2.821738423s) [noise=true , address=145.79.227.8:4329 , piece_id=GO5IAWQYS3UARWYAQAEUIIS456DFWCFWRJQG2KOVVL5KPXWPK4AA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: expected piece hash; context deadline exceeded; EOF
Nov 16 16:12:04.270306515 (176h8m2.821776986s) [noise=true , address=145.79.227.9:4367 , piece_id=LOFHPWCHNEZ4E4WANJM5BE2TIWVFH4LGEJJYK4ETB2LZWYCAVGZA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection
Nov 16 16:12:04.270323538 (176h8m2.821794099s) [noise=true , address=145.79.227.28:4725 , piece_id=CI2LSBGPW7RYH4RGWODGWLUTN353EODNV6IB2YQGHOZ3FPSEEK5A, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection
Nov 16 16:12:04.270544138 (176h8m2.822014739s) [noise=true , address=145.79.227.14:4840 , piece_id=RSHGL65RI7HZGU3W3EROFL23BCK2EW3YN6CAD3NO5OVE5QM3QG3Q, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: read tcp 188.166.225.170:42148->145.79.227.14:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:12:04.270571570 (176h8m2.822042121s) [noise=true , address=145.79.227.31:4797 , piece_id=AWSPIPRKSBOKQFZRRB34RUPLYVJGKAOOJQ4U7RRATQJELM3OCFHA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection
Nov 16 16:12:04.270609783 (176h8m2.822080254s) [noise=true , address=145.79.227.22:4511 , piece_id=HES465EP3OPD5JBVGVYMGLUOHJKSSMJDXMJVWET5GHRPFZJFIFOA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: read tcp 188.166.225.170:51004->145.79.227.22:4511: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:12:04.270625552 (176h8m2.822096083s) [noise=true , address=145.79.227.8:4331 , piece_id=4O27HU7AMR6G4Y43HJYN22UXVLETBDBE7JHL4JUZKRM54SK5E4JA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: read tcp 188.166.225.170:44510->145.79.227.8:4331: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:12:04.270742155 (176h8m2.822212616s) [noise=true , address=145.79.227.4:4186 , piece_id=5PZIK3V72CUQQ6M2HJZFVM5OVX4HPHY7ZHJ2BAK5S3ZQOGM4J5OA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection
Nov 16 16:12:04.270762814 (176h8m2.822233275s) [noise=true , address=145.79.227.24:4581 , piece_id=XRCS52EJDB7HWD4RDZK3UNB6VKHOTWWID42RGBJZQHTGG4NDYIPA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: read tcp 188.166.225.170:44130->145.79.227.24:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:12:04.270781470 (176h8m2.822251921s) [noise=true , address=145.79.227.11:4732 , piece_id=FQGAVGZ5CID3LBET3CVK6FSCOP5OUYGS44XYWMHU2RLIT3MBN62A, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: read tcp 188.166.225.170:42420->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:12:04.270825082 (176h8m2.822295544s) [seg_pos={0 0}, upload=155107]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil>
Nov 16 16:12:05.262661603 (176h8m3.814132064s) [upload=155107]: Done uploading segments...
Nov 16 16:12:05.262728080 (176h8m3.814198541s) [upload=155107]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4)
Duration: 6.037237695s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [...........................................X................] duration:
5.655652469s trace:
https://jaeger.dev.storj.io/trace/7dbd01be7ae5d01e
FAIL (2025-11-16 16:15:39 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 24)
======LOGS======
OHA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.079132696 (176h0m30.630603157s) [noise=true , address=81.171.14.70:4040 , piece_id=YG6ZBK6QTKUS3FCFD2EWU7HTFLOEEVG57SIER44E67QDP5HYCHOA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.079319683 (176h0m30.630790144s) [noise=true , address=185.167.204.205:4015 , piece_id=6UIFX7GXE2SZ465VAP56YQILR3A3OHDRKIS345EE6GBDG77KJNZQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.079646465 (176h0m30.631116926s) [noise=true , address=185.167.204.206:4062 , piece_id=PAJX3ZCWM5YBZA7YDBFYXCET5CKWBOFJ47IUWK4CYPZSWK6HYQVQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.080002833 (176h0m30.631473294s) [noise=true , address=81.171.14.70:4031 , piece_id=54T5MOZZ4YJKQFUDOCCFF2SAE7XS2N77H2AOHCBECAWE5OBUXTZQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.080062066 (176h0m30.631532527s) [noise=true , address=185.167.204.206:4067 , piece_id=2BDMQZLCV4UWWP5HB6GF6NCL5UOGNU7VNYGJIIJCKQHRTX5OPGPQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.080438673 (176h0m30.631909134s) [noise=true , address=185.167.204.205:4034 , piece_id=CZFPF35OQ3U4IH46KILFPFMVY6OC36ZUGZ7EYYBOTB3I4TXMPERA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.080675103 (176h0m30.632145564s) [noise=true , address=185.167.204.205:4014 , piece_id=57C2VUYOEULOTXUQHG7A2XOWZLXYFIEYYRYW5ARX5PKTGWKIEOAQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.081315833 (176h0m30.632786294s) [noise=true , address=81.171.14.70:4039 , piece_id=NMZ5357FVIKD2F3GCKY6YNNV73BAID2YBLVNSSM6VD5PHGLBHEIA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.082053738 (176h0m30.633524189s) [noise=true , address=185.167.204.205:4001 , piece_id=TBPYFHMXWO53LJQQDUA3I2MD2WP6BHQXAH3KIF6XXMWK27HXLCHA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.082605429 (176h0m30.63407589s) [noise=true , address=81.171.14.70:4034 , piece_id=NXBDCCCEWSZ2TMSET6OXBNQV6Q5ESXIQJUFK4WJJ3UV7FJTTAUXQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:33.280203414 (176h0m31.831673875s) [upload=155002]: Next returned nil segment
Nov 16 16:04:33.280222810 (176h0m31.831693271s) [upload=155002]: Waiting for error group managing segments...
Nov 16 16:04:33.838289781 (176h0m32.389760242s) [noise=true , address=185.167.204.205:4013 , piece_id=KF32EXUYGKQEDBPRSVAMNZV3ZMBF6DLT3U32BKX5LXLRBXBVTOQA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.848846382 (176h0m32.400316833s) [noise=true , address=185.167.204.206:4039 , piece_id=6HTUHESIRTSKR5EDMWZ2K56OKE5Y22QQM5ARYTUXR7KMKWIOS7GQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.862791868 (176h0m32.414262329s) [noise=true , address=185.167.204.205:4021 , piece_id=MV3TV4EI3HN3J5NY5XYVKT46QJTJEC5PITXERSSZMS2B7W3DSDTA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.879947135 (176h0m32.431417596s) [noise=true , address=185.167.204.205:4001 , piece_id=TBPYFHMXWO53LJQQDUA3I2MD2WP6BHQXAH3KIF6XXMWK27HXLCHA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.886853657 (176h0m32.438324118s) [noise=true , address=185.167.204.206:4067 , piece_id=2BDMQZLCV4UWWP5HB6GF6NCL5UOGNU7VNYGJIIJCKQHRTX5OPGPQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.918801783 (176h0m32.470272324s) [noise=true , address=185.167.204.206:4051 , piece_id=AU2S6M3E2CQEMAS7LKZJIVWITZUHIHCMWQ2QWT7ILGBRTAUCUB7Q, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.929982083 (176h0m32.481452574s) [noise=true , address=185.167.204.205:4015 , piece_id=6UIFX7GXE2SZ465VAP56YQILR3A3OHDRKIS345EE6GBDG77KJNZQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.935247409 (176h0m32.48671786s) [noise=true , address=185.167.204.206:4038 , piece_id=FBNPAJOOXEAT3KLKQIYOWHGHRGAWJABV4LAA3UYX6SGJB3U2LD7A, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.969630813 (176h0m32.521101264s) [noise=true , address=185.167.204.206:4055 , piece_id=B2TTK2ZLIRM3YIKP5HQ3WOL26KBHSXWWX5ZXXCMWU34AH7CHOOCQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.999043600 (176h0m32.550514061s) [noise=true , address=185.167.204.205:4034 , piece_id=CZFPF35OQ3U4IH46KILFPFMVY6OC36ZUGZ7EYYBOTB3I4TXMPERA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.022951616 (176h0m32.574422077s) [noise=true , address=185.167.204.205:4014 , piece_id=57C2VUYOEULOTXUQHG7A2XOWZLXYFIEYYRYW5ARX5PKTGWKIEOAQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.064488381 (176h0m32.615958832s) [noise=true , address=185.167.204.206:4058 , piece_id=POK7HBLE5AT2STCSL2UJ5R67LNVK3PN53NIRIBX4M7LJYC4C2CHQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.122257987 (176h0m32.673728468s) [noise=true , address=185.167.204.205:4029 , piece_id=VF7BTI4K75HTRVJOHIKMDJN3UBHZ7KCQBT3H44XB5G7ZVFD4HXJQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.136282884 (176h0m32.687753345s) [noise=true , address=185.167.204.206:4041 , piece_id=RTQSCXR54KTKBYS6AHEKPO7ZPUHVQQNUNQJ6JI4JLTGKOLUZH3AQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.136821679 (176h0m32.68829215s) [noise=true , address=185.167.204.206:4062 , piece_id=PAJX3ZCWM5YBZA7YDBFYXCET5CKWBOFJ47IUWK4CYPZSWK6HYQVQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.156096042 (176h0m32.707566493s) [noise=true , address=185.167.204.206:4066 , piece_id=CPTBHDHCT6R2M3UA6X2ZWMPJUKAUVV4I2CIATZ2ZOIWH63Q3UT4Q, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.540981809 (176h0m33.0924523s) [noise=true , address=89.149.201.147:4012 , piece_id=NEIYQHVVWRUVJFF5UD3RZSRHUSNSKXBCNGTMIGOH5MDH2AOZISCA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:21.258617296 (176h1m19.810087757s) [noise=true , address=81.171.14.70:4034 , piece_id=NXBDCCCEWSZ2TMSET6OXBNQV6Q5ESXIQJUFK4WJJ3UV7FJTTAUXQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:22.223432573 (176h1m20.774903034s) [noise=true , address=81.171.14.70:4048 , piece_id=5WUEBVU7UFKHYZ23BPIQUMUUBH3ZXXQWBLE7MALJGECCDTTEK5EA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:22.238494073 (176h1m20.789964534s) [noise=true , address=81.171.14.70:4033 , piece_id=LUBQIVFAU7J72K5XDDS5S2ZXQUGFA5AINL4R5FV64KY24VS5KV5Q, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:22.350384206 (176h1m20.901854667s) [noise=true , address=81.171.14.70:4039 , piece_id=NMZ5357FVIKD2F3GCKY6YNNV73BAID2YBLVNSSM6VD5PHGLBHEIA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:23.704484814 (176h1m22.255955275s) [noise=true , address=81.171.14.70:4037 , piece_id=3IJ7J7RBMOORW4YPKIYJ3C6ZQRYBH7WQMLEKEQYG5UJ4TGKE2W2A, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:24.501497158 (176h1m23.052967619s) [noise=true , address=81.171.14.70:4031 , piece_id=54T5MOZZ4YJKQFUDOCCFF2SAE7XS2N77H2AOHCBECAWE5OBUXTZQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:29.810461698 (176h1m28.361932159s) [noise=true , address=81.171.14.70:4025 , piece_id=3HAT6FJSGV6VOHJBMO6L6L7677BOYVCDV3SR26LJZ6HZOZWJJ64A, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:31.413018058 (176h1m29.964488509s) [noise=true , address=81.171.14.70:4040 , piece_id=YG6ZBK6QTKUS3FCFD2EWU7HTFLOEEVG57SIER44E67QDP5HYCHOA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:1gannW154cmVP6N5BowytKfDorZDRwx2M5pi6akqjZFU38PGjV, address:81.171.14.70:4040): protocol: write tcp 188.166.225.170:59244->81.171.14.70:4040: use of closed network connection; write tcp 188.166.225.170:59244->81.171.14.70:4040: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59244->81.171.14.70:4040: use of closed network connection
Nov 16 16:05:31.413111366 (176h1m29.964581837s) [noise=true , address=81.171.14.70:4035 , piece_id=RGMXRH2Y5XBAFISK5LNAF547SAORD6XSWFUKYWOPJINXXODI5KVQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:12eLYJKne3yP1fZ1oPDLM7Xqs3GQ3ju4ZQedAiojXdVkcxNpzeA, address:81.171.14.70:4035): protocol: read tcp 188.166.225.170:55142->81.171.14.70:4035: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:05:31.413619965 (176h1m29.965090436s) [noise=true , address=81.171.14.70:4030 , piece_id=ZE5OAUTWLWNSMWSFGPEJTCXEN6L3FLWL34QBERTS3JZE7X2I2OHA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:12ZoRwa8wLw66Pjumy1ybp6wrZ79RrSU55uYZpwaY3xi95iSYde, address:81.171.14.70:4030): protocol: read tcp 188.166.225.170:40822->81.171.14.70:4030: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:05:31.414846280 (176h1m29.966316741s) [noise=true , address=81.171.14.70:4029 , piece_id=SIXL2XSCF7IO2BPOUKGH5VVXI6M7O4KJVZARYIRELR2GM4T4RABA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:18yEfU2X4jSrom1JDtQ22cUEvomcg91ohvay2Nsr1Vr8juo9to, address:81.171.14.70:4029): protocol: read tcp 188.166.225.170:56584->81.171.14.70:4029: use of closed network connection; conn closed: op
Nov 16 16:05:31.414946261 (176h1m29.966416722s) [seg_pos={0 0}, upload=155002]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil>
Nov 16 16:05:31.591745995 (176h1m30.143216456s) [upload=155002]: Done uploading segments...
Nov 16 16:05:31.591788596 (176h1m30.143259057s) [upload=155002]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24)
Duration: 5.655680191s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [................X...........................................] duration:
3.475851697s trace:
https://jaeger.dev.storj.io/trace/1026d6559b04863a
FAIL (2025-11-16 16:15:39 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.47587357s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....................................................X......] duration:
10.091298186s trace:
https://jaeger.dev.storj.io/trace/3e4560ab56ee4602
FAIL (2025-11-16 16:15:38 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 10.091328913s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [.................X..........................................] duration:
3.491903987s trace:
https://jaeger.dev.storj.io/trace/1026d6559b04863a
FAIL (2025-11-16 16:15:21 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.491953642s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [............................................X...............] duration:
5.877351392s trace:
https://jaeger.dev.storj.io/trace/7dbd01be7ae5d01e
FAIL (2025-11-16 16:15:10 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 24)
======LOGS======
OHA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.079132696 (176h0m30.630603157s) [noise=true , address=81.171.14.70:4040 , piece_id=YG6ZBK6QTKUS3FCFD2EWU7HTFLOEEVG57SIER44E67QDP5HYCHOA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.079319683 (176h0m30.630790144s) [noise=true , address=185.167.204.205:4015 , piece_id=6UIFX7GXE2SZ465VAP56YQILR3A3OHDRKIS345EE6GBDG77KJNZQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.079646465 (176h0m30.631116926s) [noise=true , address=185.167.204.206:4062 , piece_id=PAJX3ZCWM5YBZA7YDBFYXCET5CKWBOFJ47IUWK4CYPZSWK6HYQVQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.080002833 (176h0m30.631473294s) [noise=true , address=81.171.14.70:4031 , piece_id=54T5MOZZ4YJKQFUDOCCFF2SAE7XS2N77H2AOHCBECAWE5OBUXTZQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.080062066 (176h0m30.631532527s) [noise=true , address=185.167.204.206:4067 , piece_id=2BDMQZLCV4UWWP5HB6GF6NCL5UOGNU7VNYGJIIJCKQHRTX5OPGPQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.080438673 (176h0m30.631909134s) [noise=true , address=185.167.204.205:4034 , piece_id=CZFPF35OQ3U4IH46KILFPFMVY6OC36ZUGZ7EYYBOTB3I4TXMPERA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.080675103 (176h0m30.632145564s) [noise=true , address=185.167.204.205:4014 , piece_id=57C2VUYOEULOTXUQHG7A2XOWZLXYFIEYYRYW5ARX5PKTGWKIEOAQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.081315833 (176h0m30.632786294s) [noise=true , address=81.171.14.70:4039 , piece_id=NMZ5357FVIKD2F3GCKY6YNNV73BAID2YBLVNSSM6VD5PHGLBHEIA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.082053738 (176h0m30.633524189s) [noise=true , address=185.167.204.205:4001 , piece_id=TBPYFHMXWO53LJQQDUA3I2MD2WP6BHQXAH3KIF6XXMWK27HXLCHA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.082605429 (176h0m30.63407589s) [noise=true , address=81.171.14.70:4034 , piece_id=NXBDCCCEWSZ2TMSET6OXBNQV6Q5ESXIQJUFK4WJJ3UV7FJTTAUXQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:33.280203414 (176h0m31.831673875s) [upload=155002]: Next returned nil segment
Nov 16 16:04:33.280222810 (176h0m31.831693271s) [upload=155002]: Waiting for error group managing segments...
Nov 16 16:04:33.838289781 (176h0m32.389760242s) [noise=true , address=185.167.204.205:4013 , piece_id=KF32EXUYGKQEDBPRSVAMNZV3ZMBF6DLT3U32BKX5LXLRBXBVTOQA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.848846382 (176h0m32.400316833s) [noise=true , address=185.167.204.206:4039 , piece_id=6HTUHESIRTSKR5EDMWZ2K56OKE5Y22QQM5ARYTUXR7KMKWIOS7GQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.862791868 (176h0m32.414262329s) [noise=true , address=185.167.204.205:4021 , piece_id=MV3TV4EI3HN3J5NY5XYVKT46QJTJEC5PITXERSSZMS2B7W3DSDTA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.879947135 (176h0m32.431417596s) [noise=true , address=185.167.204.205:4001 , piece_id=TBPYFHMXWO53LJQQDUA3I2MD2WP6BHQXAH3KIF6XXMWK27HXLCHA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.886853657 (176h0m32.438324118s) [noise=true , address=185.167.204.206:4067 , piece_id=2BDMQZLCV4UWWP5HB6GF6NCL5UOGNU7VNYGJIIJCKQHRTX5OPGPQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.918801783 (176h0m32.470272324s) [noise=true , address=185.167.204.206:4051 , piece_id=AU2S6M3E2CQEMAS7LKZJIVWITZUHIHCMWQ2QWT7ILGBRTAUCUB7Q, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.929982083 (176h0m32.481452574s) [noise=true , address=185.167.204.205:4015 , piece_id=6UIFX7GXE2SZ465VAP56YQILR3A3OHDRKIS345EE6GBDG77KJNZQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.935247409 (176h0m32.48671786s) [noise=true , address=185.167.204.206:4038 , piece_id=FBNPAJOOXEAT3KLKQIYOWHGHRGAWJABV4LAA3UYX6SGJB3U2LD7A, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.969630813 (176h0m32.521101264s) [noise=true , address=185.167.204.206:4055 , piece_id=B2TTK2ZLIRM3YIKP5HQ3WOL26KBHSXWWX5ZXXCMWU34AH7CHOOCQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.999043600 (176h0m32.550514061s) [noise=true , address=185.167.204.205:4034 , piece_id=CZFPF35OQ3U4IH46KILFPFMVY6OC36ZUGZ7EYYBOTB3I4TXMPERA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.022951616 (176h0m32.574422077s) [noise=true , address=185.167.204.205:4014 , piece_id=57C2VUYOEULOTXUQHG7A2XOWZLXYFIEYYRYW5ARX5PKTGWKIEOAQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.064488381 (176h0m32.615958832s) [noise=true , address=185.167.204.206:4058 , piece_id=POK7HBLE5AT2STCSL2UJ5R67LNVK3PN53NIRIBX4M7LJYC4C2CHQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.122257987 (176h0m32.673728468s) [noise=true , address=185.167.204.205:4029 , piece_id=VF7BTI4K75HTRVJOHIKMDJN3UBHZ7KCQBT3H44XB5G7ZVFD4HXJQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.136282884 (176h0m32.687753345s) [noise=true , address=185.167.204.206:4041 , piece_id=RTQSCXR54KTKBYS6AHEKPO7ZPUHVQQNUNQJ6JI4JLTGKOLUZH3AQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.136821679 (176h0m32.68829215s) [noise=true , address=185.167.204.206:4062 , piece_id=PAJX3ZCWM5YBZA7YDBFYXCET5CKWBOFJ47IUWK4CYPZSWK6HYQVQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.156096042 (176h0m32.707566493s) [noise=true , address=185.167.204.206:4066 , piece_id=CPTBHDHCT6R2M3UA6X2ZWMPJUKAUVV4I2CIATZ2ZOIWH63Q3UT4Q, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.540981809 (176h0m33.0924523s) [noise=true , address=89.149.201.147:4012 , piece_id=NEIYQHVVWRUVJFF5UD3RZSRHUSNSKXBCNGTMIGOH5MDH2AOZISCA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:21.258617296 (176h1m19.810087757s) [noise=true , address=81.171.14.70:4034 , piece_id=NXBDCCCEWSZ2TMSET6OXBNQV6Q5ESXIQJUFK4WJJ3UV7FJTTAUXQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:22.223432573 (176h1m20.774903034s) [noise=true , address=81.171.14.70:4048 , piece_id=5WUEBVU7UFKHYZ23BPIQUMUUBH3ZXXQWBLE7MALJGECCDTTEK5EA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:22.238494073 (176h1m20.789964534s) [noise=true , address=81.171.14.70:4033 , piece_id=LUBQIVFAU7J72K5XDDS5S2ZXQUGFA5AINL4R5FV64KY24VS5KV5Q, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:22.350384206 (176h1m20.901854667s) [noise=true , address=81.171.14.70:4039 , piece_id=NMZ5357FVIKD2F3GCKY6YNNV73BAID2YBLVNSSM6VD5PHGLBHEIA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:23.704484814 (176h1m22.255955275s) [noise=true , address=81.171.14.70:4037 , piece_id=3IJ7J7RBMOORW4YPKIYJ3C6ZQRYBH7WQMLEKEQYG5UJ4TGKE2W2A, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:24.501497158 (176h1m23.052967619s) [noise=true , address=81.171.14.70:4031 , piece_id=54T5MOZZ4YJKQFUDOCCFF2SAE7XS2N77H2AOHCBECAWE5OBUXTZQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:29.810461698 (176h1m28.361932159s) [noise=true , address=81.171.14.70:4025 , piece_id=3HAT6FJSGV6VOHJBMO6L6L7677BOYVCDV3SR26LJZ6HZOZWJJ64A, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:31.413018058 (176h1m29.964488509s) [noise=true , address=81.171.14.70:4040 , piece_id=YG6ZBK6QTKUS3FCFD2EWU7HTFLOEEVG57SIER44E67QDP5HYCHOA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:1gannW154cmVP6N5BowytKfDorZDRwx2M5pi6akqjZFU38PGjV, address:81.171.14.70:4040): protocol: write tcp 188.166.225.170:59244->81.171.14.70:4040: use of closed network connection; write tcp 188.166.225.170:59244->81.171.14.70:4040: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59244->81.171.14.70:4040: use of closed network connection
Nov 16 16:05:31.413111366 (176h1m29.964581837s) [noise=true , address=81.171.14.70:4035 , piece_id=RGMXRH2Y5XBAFISK5LNAF547SAORD6XSWFUKYWOPJINXXODI5KVQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:12eLYJKne3yP1fZ1oPDLM7Xqs3GQ3ju4ZQedAiojXdVkcxNpzeA, address:81.171.14.70:4035): protocol: read tcp 188.166.225.170:55142->81.171.14.70:4035: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:05:31.413619965 (176h1m29.965090436s) [noise=true , address=81.171.14.70:4030 , piece_id=ZE5OAUTWLWNSMWSFGPEJTCXEN6L3FLWL34QBERTS3JZE7X2I2OHA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:12ZoRwa8wLw66Pjumy1ybp6wrZ79RrSU55uYZpwaY3xi95iSYde, address:81.171.14.70:4030): protocol: read tcp 188.166.225.170:40822->81.171.14.70:4030: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:05:31.414846280 (176h1m29.966316741s) [noise=true , address=81.171.14.70:4029 , piece_id=SIXL2XSCF7IO2BPOUKGH5VVXI6M7O4KJVZARYIRELR2GM4T4RABA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:18yEfU2X4jSrom1JDtQ22cUEvomcg91ohvay2Nsr1Vr8juo9to, address:81.171.14.70:4029): protocol: read tcp 188.166.225.170:56584->81.171.14.70:4029: use of closed network connection; conn closed: op
Nov 16 16:05:31.414946261 (176h1m29.966416722s) [seg_pos={0 0}, upload=155002]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil>
Nov 16 16:05:31.591745995 (176h1m30.143216456s) [upload=155002]: Done uploading segments...
Nov 16 16:05:31.591788596 (176h1m30.143259057s) [upload=155002]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24)
Duration: 5.877375388s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [......................................................X.....] duration:
6.140656376s trace:
https://jaeger.dev.storj.io/trace/3e4560ab56ee4602
FAIL (2025-11-16 16:15:01 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.140687044s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....................................................X......] duration:
6.154069606s trace:
https://jaeger.dev.storj.io/trace/70f81d2d9b012a3a
FAIL (2025-11-16 16:14:52 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 4)
======LOGS======
ddress=45.91.116.5:4297 , piece_id=OSMI3VGAKSCIZMDQ3Y2VZQBUOJKSXM6RWNWSAV57RDTL5TR3NIJA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.462247309 (176h7m35.013717781s) [noise=true , address=45.91.116.7:4011 , piece_id=ORJVRMFIOEXZ6OB6QFQDL764Q2FCWOJT63NBIRI6SOOS2FEIBCFQ, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.463420774 (176h7m35.014891235s) [noise=true , address=145.79.227.9:4367 , piece_id=LOFHPWCHNEZ4E4WANJM5BE2TIWVFH4LGEJJYK4ETB2LZWYCAVGZA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.464113112 (176h7m35.015583573s) [noise=true , address=145.79.227.22:4511 , piece_id=HES465EP3OPD5JBVGVYMGLUOHJKSSMJDXMJVWET5GHRPFZJFIFOA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.464764873 (176h7m35.016235334s) [noise=true , address=45.91.116.8:4048 , piece_id=ULVK52VWCW3KNEPWSDYGRHWHIJWEQU2XNFEPYJFXWLVJI6AGZQOA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.465452753 (176h7m35.016923214s) [noise=true , address=145.79.227.28:4725 , piece_id=CI2LSBGPW7RYH4RGWODGWLUTN353EODNV6IB2YQGHOZ3FPSEEK5A, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.466223270 (176h7m35.017693731s) [noise=true , address=145.79.227.16:4910 , piece_id=3U2SEVMPILIEBRHS3M4YT4Z4PSYAQAIC2SKGYMW4FNODACZLOQCQ, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.466955855 (176h7m35.018426306s) [noise=true , address=145.79.227.4:4186 , piece_id=5PZIK3V72CUQQ6M2HJZFVM5OVX4HPHY7ZHJ2BAK5S3ZQOGM4J5OA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.467675205 (176h7m35.019145666s) [noise=true , address=145.79.227.8:4331 , piece_id=4O27HU7AMR6G4Y43HJYN22UXVLETBDBE7JHL4JUZKRM54SK5E4JA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.468178092 (176h7m35.019648553s) [noise=true , address=145.79.227.8:4329 , piece_id=GO5IAWQYS3UARWYAQAEUIIS456DFWCFWRJQG2KOVVL5KPXWPK4AA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:39.111804678 (176h7m37.663275129s) [upload=155107]: Next returned nil segment
Nov 16 16:11:39.111825767 (176h7m37.663296218s) [upload=155107]: Waiting for error group managing segments...
Nov 16 16:11:39.896671313 (176h7m38.448141765s) [noise=true , address=45.91.116.12:4227 , piece_id=CQA2LHRSFOA7EIBIFVC75EWVHFZMWMY7ODMETEAJRRNYSQFY5TVQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil>
Nov 16 16:11:39.954993420 (176h7m38.506463881s) [noise=true , address=45.91.116.8:4048 , piece_id=ULVK52VWCW3KNEPWSDYGRHWHIJWEQU2XNFEPYJFXWLVJI6AGZQOA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil>
Nov 16 16:11:40.066014029 (176h7m38.61748449s) [noise=true , address=45.91.116.7:4011 , piece_id=ORJVRMFIOEXZ6OB6QFQDL764Q2FCWOJT63NBIRI6SOOS2FEIBCFQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil>
Nov 16 16:11:41.102688250 (176h7m39.654158711s) [noise=true , address=45.91.116.5:4297 , piece_id=OSMI3VGAKSCIZMDQ3Y2VZQBUOJKSXM6RWNWSAV57RDTL5TR3NIJA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil>
Nov 16 16:12:04.264842280 (176h8m2.816312741s) [noise=true , address=145.79.227.17:4972 , piece_id=NV6FSBQM6YUTKYFAJPQCURWCZBHJ3AKJOLLGCRYTJ3KQ54U4GOWQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection
Nov 16 16:12:04.265084832 (176h8m2.816555293s) [noise=true , address=145.79.227.16:4910 , piece_id=3U2SEVMPILIEBRHS3M4YT4Z4PSYAQAIC2SKGYMW4FNODACZLOQCQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection
Nov 16 16:12:04.270267942 (176h8m2.821738423s) [noise=true , address=145.79.227.8:4329 , piece_id=GO5IAWQYS3UARWYAQAEUIIS456DFWCFWRJQG2KOVVL5KPXWPK4AA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: expected piece hash; context deadline exceeded; EOF
Nov 16 16:12:04.270306515 (176h8m2.821776986s) [noise=true , address=145.79.227.9:4367 , piece_id=LOFHPWCHNEZ4E4WANJM5BE2TIWVFH4LGEJJYK4ETB2LZWYCAVGZA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection
Nov 16 16:12:04.270323538 (176h8m2.821794099s) [noise=true , address=145.79.227.28:4725 , piece_id=CI2LSBGPW7RYH4RGWODGWLUTN353EODNV6IB2YQGHOZ3FPSEEK5A, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection
Nov 16 16:12:04.270544138 (176h8m2.822014739s) [noise=true , address=145.79.227.14:4840 , piece_id=RSHGL65RI7HZGU3W3EROFL23BCK2EW3YN6CAD3NO5OVE5QM3QG3Q, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: read tcp 188.166.225.170:42148->145.79.227.14:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:12:04.270571570 (176h8m2.822042121s) [noise=true , address=145.79.227.31:4797 , piece_id=AWSPIPRKSBOKQFZRRB34RUPLYVJGKAOOJQ4U7RRATQJELM3OCFHA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection
Nov 16 16:12:04.270609783 (176h8m2.822080254s) [noise=true , address=145.79.227.22:4511 , piece_id=HES465EP3OPD5JBVGVYMGLUOHJKSSMJDXMJVWET5GHRPFZJFIFOA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: read tcp 188.166.225.170:51004->145.79.227.22:4511: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:12:04.270625552 (176h8m2.822096083s) [noise=true , address=145.79.227.8:4331 , piece_id=4O27HU7AMR6G4Y43HJYN22UXVLETBDBE7JHL4JUZKRM54SK5E4JA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: read tcp 188.166.225.170:44510->145.79.227.8:4331: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:12:04.270742155 (176h8m2.822212616s) [noise=true , address=145.79.227.4:4186 , piece_id=5PZIK3V72CUQQ6M2HJZFVM5OVX4HPHY7ZHJ2BAK5S3ZQOGM4J5OA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection
Nov 16 16:12:04.270762814 (176h8m2.822233275s) [noise=true , address=145.79.227.24:4581 , piece_id=XRCS52EJDB7HWD4RDZK3UNB6VKHOTWWID42RGBJZQHTGG4NDYIPA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: read tcp 188.166.225.170:44130->145.79.227.24:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:12:04.270781470 (176h8m2.822251921s) [noise=true , address=145.79.227.11:4732 , piece_id=FQGAVGZ5CID3LBET3CVK6FSCOP5OUYGS44XYWMHU2RLIT3MBN62A, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: read tcp 188.166.225.170:42420->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:12:04.270825082 (176h8m2.822295544s) [seg_pos={0 0}, upload=155107]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil>
Nov 16 16:12:05.262661603 (176h8m3.814132064s) [upload=155107]: Done uploading segments...
Nov 16 16:12:05.262728080 (176h8m3.814198541s) [upload=155107]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4)
Duration: 6.154104371s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [.............................................X..............] duration:
5.373715846s trace:
https://jaeger.dev.storj.io/trace/7dbd01be7ae5d01e
FAIL (2025-11-16 16:14:35 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 24)
======LOGS======
OHA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.079132696 (176h0m30.630603157s) [noise=true , address=81.171.14.70:4040 , piece_id=YG6ZBK6QTKUS3FCFD2EWU7HTFLOEEVG57SIER44E67QDP5HYCHOA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.079319683 (176h0m30.630790144s) [noise=true , address=185.167.204.205:4015 , piece_id=6UIFX7GXE2SZ465VAP56YQILR3A3OHDRKIS345EE6GBDG77KJNZQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.079646465 (176h0m30.631116926s) [noise=true , address=185.167.204.206:4062 , piece_id=PAJX3ZCWM5YBZA7YDBFYXCET5CKWBOFJ47IUWK4CYPZSWK6HYQVQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.080002833 (176h0m30.631473294s) [noise=true , address=81.171.14.70:4031 , piece_id=54T5MOZZ4YJKQFUDOCCFF2SAE7XS2N77H2AOHCBECAWE5OBUXTZQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.080062066 (176h0m30.631532527s) [noise=true , address=185.167.204.206:4067 , piece_id=2BDMQZLCV4UWWP5HB6GF6NCL5UOGNU7VNYGJIIJCKQHRTX5OPGPQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.080438673 (176h0m30.631909134s) [noise=true , address=185.167.204.205:4034 , piece_id=CZFPF35OQ3U4IH46KILFPFMVY6OC36ZUGZ7EYYBOTB3I4TXMPERA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.080675103 (176h0m30.632145564s) [noise=true , address=185.167.204.205:4014 , piece_id=57C2VUYOEULOTXUQHG7A2XOWZLXYFIEYYRYW5ARX5PKTGWKIEOAQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.081315833 (176h0m30.632786294s) [noise=true , address=81.171.14.70:4039 , piece_id=NMZ5357FVIKD2F3GCKY6YNNV73BAID2YBLVNSSM6VD5PHGLBHEIA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.082053738 (176h0m30.633524189s) [noise=true , address=185.167.204.205:4001 , piece_id=TBPYFHMXWO53LJQQDUA3I2MD2WP6BHQXAH3KIF6XXMWK27HXLCHA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.082605429 (176h0m30.63407589s) [noise=true , address=81.171.14.70:4034 , piece_id=NXBDCCCEWSZ2TMSET6OXBNQV6Q5ESXIQJUFK4WJJ3UV7FJTTAUXQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:33.280203414 (176h0m31.831673875s) [upload=155002]: Next returned nil segment
Nov 16 16:04:33.280222810 (176h0m31.831693271s) [upload=155002]: Waiting for error group managing segments...
Nov 16 16:04:33.838289781 (176h0m32.389760242s) [noise=true , address=185.167.204.205:4013 , piece_id=KF32EXUYGKQEDBPRSVAMNZV3ZMBF6DLT3U32BKX5LXLRBXBVTOQA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.848846382 (176h0m32.400316833s) [noise=true , address=185.167.204.206:4039 , piece_id=6HTUHESIRTSKR5EDMWZ2K56OKE5Y22QQM5ARYTUXR7KMKWIOS7GQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.862791868 (176h0m32.414262329s) [noise=true , address=185.167.204.205:4021 , piece_id=MV3TV4EI3HN3J5NY5XYVKT46QJTJEC5PITXERSSZMS2B7W3DSDTA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.879947135 (176h0m32.431417596s) [noise=true , address=185.167.204.205:4001 , piece_id=TBPYFHMXWO53LJQQDUA3I2MD2WP6BHQXAH3KIF6XXMWK27HXLCHA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.886853657 (176h0m32.438324118s) [noise=true , address=185.167.204.206:4067 , piece_id=2BDMQZLCV4UWWP5HB6GF6NCL5UOGNU7VNYGJIIJCKQHRTX5OPGPQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.918801783 (176h0m32.470272324s) [noise=true , address=185.167.204.206:4051 , piece_id=AU2S6M3E2CQEMAS7LKZJIVWITZUHIHCMWQ2QWT7ILGBRTAUCUB7Q, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.929982083 (176h0m32.481452574s) [noise=true , address=185.167.204.205:4015 , piece_id=6UIFX7GXE2SZ465VAP56YQILR3A3OHDRKIS345EE6GBDG77KJNZQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.935247409 (176h0m32.48671786s) [noise=true , address=185.167.204.206:4038 , piece_id=FBNPAJOOXEAT3KLKQIYOWHGHRGAWJABV4LAA3UYX6SGJB3U2LD7A, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.969630813 (176h0m32.521101264s) [noise=true , address=185.167.204.206:4055 , piece_id=B2TTK2ZLIRM3YIKP5HQ3WOL26KBHSXWWX5ZXXCMWU34AH7CHOOCQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.999043600 (176h0m32.550514061s) [noise=true , address=185.167.204.205:4034 , piece_id=CZFPF35OQ3U4IH46KILFPFMVY6OC36ZUGZ7EYYBOTB3I4TXMPERA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.022951616 (176h0m32.574422077s) [noise=true , address=185.167.204.205:4014 , piece_id=57C2VUYOEULOTXUQHG7A2XOWZLXYFIEYYRYW5ARX5PKTGWKIEOAQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.064488381 (176h0m32.615958832s) [noise=true , address=185.167.204.206:4058 , piece_id=POK7HBLE5AT2STCSL2UJ5R67LNVK3PN53NIRIBX4M7LJYC4C2CHQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.122257987 (176h0m32.673728468s) [noise=true , address=185.167.204.205:4029 , piece_id=VF7BTI4K75HTRVJOHIKMDJN3UBHZ7KCQBT3H44XB5G7ZVFD4HXJQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.136282884 (176h0m32.687753345s) [noise=true , address=185.167.204.206:4041 , piece_id=RTQSCXR54KTKBYS6AHEKPO7ZPUHVQQNUNQJ6JI4JLTGKOLUZH3AQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.136821679 (176h0m32.68829215s) [noise=true , address=185.167.204.206:4062 , piece_id=PAJX3ZCWM5YBZA7YDBFYXCET5CKWBOFJ47IUWK4CYPZSWK6HYQVQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.156096042 (176h0m32.707566493s) [noise=true , address=185.167.204.206:4066 , piece_id=CPTBHDHCT6R2M3UA6X2ZWMPJUKAUVV4I2CIATZ2ZOIWH63Q3UT4Q, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.540981809 (176h0m33.0924523s) [noise=true , address=89.149.201.147:4012 , piece_id=NEIYQHVVWRUVJFF5UD3RZSRHUSNSKXBCNGTMIGOH5MDH2AOZISCA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:21.258617296 (176h1m19.810087757s) [noise=true , address=81.171.14.70:4034 , piece_id=NXBDCCCEWSZ2TMSET6OXBNQV6Q5ESXIQJUFK4WJJ3UV7FJTTAUXQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:22.223432573 (176h1m20.774903034s) [noise=true , address=81.171.14.70:4048 , piece_id=5WUEBVU7UFKHYZ23BPIQUMUUBH3ZXXQWBLE7MALJGECCDTTEK5EA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:22.238494073 (176h1m20.789964534s) [noise=true , address=81.171.14.70:4033 , piece_id=LUBQIVFAU7J72K5XDDS5S2ZXQUGFA5AINL4R5FV64KY24VS5KV5Q, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:22.350384206 (176h1m20.901854667s) [noise=true , address=81.171.14.70:4039 , piece_id=NMZ5357FVIKD2F3GCKY6YNNV73BAID2YBLVNSSM6VD5PHGLBHEIA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:23.704484814 (176h1m22.255955275s) [noise=true , address=81.171.14.70:4037 , piece_id=3IJ7J7RBMOORW4YPKIYJ3C6ZQRYBH7WQMLEKEQYG5UJ4TGKE2W2A, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:24.501497158 (176h1m23.052967619s) [noise=true , address=81.171.14.70:4031 , piece_id=54T5MOZZ4YJKQFUDOCCFF2SAE7XS2N77H2AOHCBECAWE5OBUXTZQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:29.810461698 (176h1m28.361932159s) [noise=true , address=81.171.14.70:4025 , piece_id=3HAT6FJSGV6VOHJBMO6L6L7677BOYVCDV3SR26LJZ6HZOZWJJ64A, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:31.413018058 (176h1m29.964488509s) [noise=true , address=81.171.14.70:4040 , piece_id=YG6ZBK6QTKUS3FCFD2EWU7HTFLOEEVG57SIER44E67QDP5HYCHOA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:1gannW154cmVP6N5BowytKfDorZDRwx2M5pi6akqjZFU38PGjV, address:81.171.14.70:4040): protocol: write tcp 188.166.225.170:59244->81.171.14.70:4040: use of closed network connection; write tcp 188.166.225.170:59244->81.171.14.70:4040: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59244->81.171.14.70:4040: use of closed network connection
Nov 16 16:05:31.413111366 (176h1m29.964581837s) [noise=true , address=81.171.14.70:4035 , piece_id=RGMXRH2Y5XBAFISK5LNAF547SAORD6XSWFUKYWOPJINXXODI5KVQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:12eLYJKne3yP1fZ1oPDLM7Xqs3GQ3ju4ZQedAiojXdVkcxNpzeA, address:81.171.14.70:4035): protocol: read tcp 188.166.225.170:55142->81.171.14.70:4035: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:05:31.413619965 (176h1m29.965090436s) [noise=true , address=81.171.14.70:4030 , piece_id=ZE5OAUTWLWNSMWSFGPEJTCXEN6L3FLWL34QBERTS3JZE7X2I2OHA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:12ZoRwa8wLw66Pjumy1ybp6wrZ79RrSU55uYZpwaY3xi95iSYde, address:81.171.14.70:4030): protocol: read tcp 188.166.225.170:40822->81.171.14.70:4030: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:05:31.414846280 (176h1m29.966316741s) [noise=true , address=81.171.14.70:4029 , piece_id=SIXL2XSCF7IO2BPOUKGH5VVXI6M7O4KJVZARYIRELR2GM4T4RABA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:18yEfU2X4jSrom1JDtQ22cUEvomcg91ohvay2Nsr1Vr8juo9to, address:81.171.14.70:4029): protocol: read tcp 188.166.225.170:56584->81.171.14.70:4029: use of closed network connection; conn closed: op
Nov 16 16:05:31.414946261 (176h1m29.966416722s) [seg_pos={0 0}, upload=155002]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil>
Nov 16 16:05:31.591745995 (176h1m30.143216456s) [upload=155002]: Done uploading segments...
Nov 16 16:05:31.591788596 (176h1m30.143259057s) [upload=155002]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24)
Duration: 5.373750812s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [..................X.........................................] duration:
3.53763927s trace:
https://jaeger.dev.storj.io/trace/1026d6559b04863a
FAIL (2025-11-16 16:14:30 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.537663876s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [......................................................X.....] duration:
5.942053676s trace:
https://jaeger.dev.storj.io/trace/70f81d2d9b012a3a
FAIL (2025-11-16 16:14:27 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 4)
======LOGS======
ddress=45.91.116.5:4297 , piece_id=OSMI3VGAKSCIZMDQ3Y2VZQBUOJKSXM6RWNWSAV57RDTL5TR3NIJA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.462247309 (176h7m35.013717781s) [noise=true , address=45.91.116.7:4011 , piece_id=ORJVRMFIOEXZ6OB6QFQDL764Q2FCWOJT63NBIRI6SOOS2FEIBCFQ, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.463420774 (176h7m35.014891235s) [noise=true , address=145.79.227.9:4367 , piece_id=LOFHPWCHNEZ4E4WANJM5BE2TIWVFH4LGEJJYK4ETB2LZWYCAVGZA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.464113112 (176h7m35.015583573s) [noise=true , address=145.79.227.22:4511 , piece_id=HES465EP3OPD5JBVGVYMGLUOHJKSSMJDXMJVWET5GHRPFZJFIFOA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.464764873 (176h7m35.016235334s) [noise=true , address=45.91.116.8:4048 , piece_id=ULVK52VWCW3KNEPWSDYGRHWHIJWEQU2XNFEPYJFXWLVJI6AGZQOA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.465452753 (176h7m35.016923214s) [noise=true , address=145.79.227.28:4725 , piece_id=CI2LSBGPW7RYH4RGWODGWLUTN353EODNV6IB2YQGHOZ3FPSEEK5A, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.466223270 (176h7m35.017693731s) [noise=true , address=145.79.227.16:4910 , piece_id=3U2SEVMPILIEBRHS3M4YT4Z4PSYAQAIC2SKGYMW4FNODACZLOQCQ, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.466955855 (176h7m35.018426306s) [noise=true , address=145.79.227.4:4186 , piece_id=5PZIK3V72CUQQ6M2HJZFVM5OVX4HPHY7ZHJ2BAK5S3ZQOGM4J5OA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.467675205 (176h7m35.019145666s) [noise=true , address=145.79.227.8:4331 , piece_id=4O27HU7AMR6G4Y43HJYN22UXVLETBDBE7JHL4JUZKRM54SK5E4JA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.468178092 (176h7m35.019648553s) [noise=true , address=145.79.227.8:4329 , piece_id=GO5IAWQYS3UARWYAQAEUIIS456DFWCFWRJQG2KOVVL5KPXWPK4AA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:39.111804678 (176h7m37.663275129s) [upload=155107]: Next returned nil segment
Nov 16 16:11:39.111825767 (176h7m37.663296218s) [upload=155107]: Waiting for error group managing segments...
Nov 16 16:11:39.896671313 (176h7m38.448141765s) [noise=true , address=45.91.116.12:4227 , piece_id=CQA2LHRSFOA7EIBIFVC75EWVHFZMWMY7ODMETEAJRRNYSQFY5TVQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil>
Nov 16 16:11:39.954993420 (176h7m38.506463881s) [noise=true , address=45.91.116.8:4048 , piece_id=ULVK52VWCW3KNEPWSDYGRHWHIJWEQU2XNFEPYJFXWLVJI6AGZQOA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil>
Nov 16 16:11:40.066014029 (176h7m38.61748449s) [noise=true , address=45.91.116.7:4011 , piece_id=ORJVRMFIOEXZ6OB6QFQDL764Q2FCWOJT63NBIRI6SOOS2FEIBCFQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil>
Nov 16 16:11:41.102688250 (176h7m39.654158711s) [noise=true , address=45.91.116.5:4297 , piece_id=OSMI3VGAKSCIZMDQ3Y2VZQBUOJKSXM6RWNWSAV57RDTL5TR3NIJA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil>
Nov 16 16:12:04.264842280 (176h8m2.816312741s) [noise=true , address=145.79.227.17:4972 , piece_id=NV6FSBQM6YUTKYFAJPQCURWCZBHJ3AKJOLLGCRYTJ3KQ54U4GOWQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection
Nov 16 16:12:04.265084832 (176h8m2.816555293s) [noise=true , address=145.79.227.16:4910 , piece_id=3U2SEVMPILIEBRHS3M4YT4Z4PSYAQAIC2SKGYMW4FNODACZLOQCQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection
Nov 16 16:12:04.270267942 (176h8m2.821738423s) [noise=true , address=145.79.227.8:4329 , piece_id=GO5IAWQYS3UARWYAQAEUIIS456DFWCFWRJQG2KOVVL5KPXWPK4AA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: expected piece hash; context deadline exceeded; EOF
Nov 16 16:12:04.270306515 (176h8m2.821776986s) [noise=true , address=145.79.227.9:4367 , piece_id=LOFHPWCHNEZ4E4WANJM5BE2TIWVFH4LGEJJYK4ETB2LZWYCAVGZA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection
Nov 16 16:12:04.270323538 (176h8m2.821794099s) [noise=true , address=145.79.227.28:4725 , piece_id=CI2LSBGPW7RYH4RGWODGWLUTN353EODNV6IB2YQGHOZ3FPSEEK5A, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection
Nov 16 16:12:04.270544138 (176h8m2.822014739s) [noise=true , address=145.79.227.14:4840 , piece_id=RSHGL65RI7HZGU3W3EROFL23BCK2EW3YN6CAD3NO5OVE5QM3QG3Q, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: read tcp 188.166.225.170:42148->145.79.227.14:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:12:04.270571570 (176h8m2.822042121s) [noise=true , address=145.79.227.31:4797 , piece_id=AWSPIPRKSBOKQFZRRB34RUPLYVJGKAOOJQ4U7RRATQJELM3OCFHA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection
Nov 16 16:12:04.270609783 (176h8m2.822080254s) [noise=true , address=145.79.227.22:4511 , piece_id=HES465EP3OPD5JBVGVYMGLUOHJKSSMJDXMJVWET5GHRPFZJFIFOA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: read tcp 188.166.225.170:51004->145.79.227.22:4511: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:12:04.270625552 (176h8m2.822096083s) [noise=true , address=145.79.227.8:4331 , piece_id=4O27HU7AMR6G4Y43HJYN22UXVLETBDBE7JHL4JUZKRM54SK5E4JA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: read tcp 188.166.225.170:44510->145.79.227.8:4331: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:12:04.270742155 (176h8m2.822212616s) [noise=true , address=145.79.227.4:4186 , piece_id=5PZIK3V72CUQQ6M2HJZFVM5OVX4HPHY7ZHJ2BAK5S3ZQOGM4J5OA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection
Nov 16 16:12:04.270762814 (176h8m2.822233275s) [noise=true , address=145.79.227.24:4581 , piece_id=XRCS52EJDB7HWD4RDZK3UNB6VKHOTWWID42RGBJZQHTGG4NDYIPA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: read tcp 188.166.225.170:44130->145.79.227.24:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:12:04.270781470 (176h8m2.822251921s) [noise=true , address=145.79.227.11:4732 , piece_id=FQGAVGZ5CID3LBET3CVK6FSCOP5OUYGS44XYWMHU2RLIT3MBN62A, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: read tcp 188.166.225.170:42420->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:12:04.270825082 (176h8m2.822295544s) [seg_pos={0 0}, upload=155107]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil>
Nov 16 16:12:05.262661603 (176h8m3.814132064s) [upload=155107]: Done uploading segments...
Nov 16 16:12:05.262728080 (176h8m3.814198541s) [upload=155107]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4)
Duration: 5.942083472s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......................................................X....] duration:
5.999913903s trace:
https://jaeger.dev.storj.io/trace/70f81d2d9b012a3a
FAIL (2025-11-16 16:14:04 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 4)
======LOGS======
ddress=45.91.116.5:4297 , piece_id=OSMI3VGAKSCIZMDQ3Y2VZQBUOJKSXM6RWNWSAV57RDTL5TR3NIJA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.462247309 (176h7m35.013717781s) [noise=true , address=45.91.116.7:4011 , piece_id=ORJVRMFIOEXZ6OB6QFQDL764Q2FCWOJT63NBIRI6SOOS2FEIBCFQ, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.463420774 (176h7m35.014891235s) [noise=true , address=145.79.227.9:4367 , piece_id=LOFHPWCHNEZ4E4WANJM5BE2TIWVFH4LGEJJYK4ETB2LZWYCAVGZA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.464113112 (176h7m35.015583573s) [noise=true , address=145.79.227.22:4511 , piece_id=HES465EP3OPD5JBVGVYMGLUOHJKSSMJDXMJVWET5GHRPFZJFIFOA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.464764873 (176h7m35.016235334s) [noise=true , address=45.91.116.8:4048 , piece_id=ULVK52VWCW3KNEPWSDYGRHWHIJWEQU2XNFEPYJFXWLVJI6AGZQOA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.465452753 (176h7m35.016923214s) [noise=true , address=145.79.227.28:4725 , piece_id=CI2LSBGPW7RYH4RGWODGWLUTN353EODNV6IB2YQGHOZ3FPSEEK5A, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.466223270 (176h7m35.017693731s) [noise=true , address=145.79.227.16:4910 , piece_id=3U2SEVMPILIEBRHS3M4YT4Z4PSYAQAIC2SKGYMW4FNODACZLOQCQ, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.466955855 (176h7m35.018426306s) [noise=true , address=145.79.227.4:4186 , piece_id=5PZIK3V72CUQQ6M2HJZFVM5OVX4HPHY7ZHJ2BAK5S3ZQOGM4J5OA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.467675205 (176h7m35.019145666s) [noise=true , address=145.79.227.8:4331 , piece_id=4O27HU7AMR6G4Y43HJYN22UXVLETBDBE7JHL4JUZKRM54SK5E4JA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:36.468178092 (176h7m35.019648553s) [noise=true , address=145.79.227.8:4329 , piece_id=GO5IAWQYS3UARWYAQAEUIIS456DFWCFWRJQG2KOVVL5KPXWPK4AA, seg_pos={0 0}, upload=155107]: Uploading piece...
Nov 16 16:11:39.111804678 (176h7m37.663275129s) [upload=155107]: Next returned nil segment
Nov 16 16:11:39.111825767 (176h7m37.663296218s) [upload=155107]: Waiting for error group managing segments...
Nov 16 16:11:39.896671313 (176h7m38.448141765s) [noise=true , address=45.91.116.12:4227 , piece_id=CQA2LHRSFOA7EIBIFVC75EWVHFZMWMY7ODMETEAJRRNYSQFY5TVQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil>
Nov 16 16:11:39.954993420 (176h7m38.506463881s) [noise=true , address=45.91.116.8:4048 , piece_id=ULVK52VWCW3KNEPWSDYGRHWHIJWEQU2XNFEPYJFXWLVJI6AGZQOA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil>
Nov 16 16:11:40.066014029 (176h7m38.61748449s) [noise=true , address=45.91.116.7:4011 , piece_id=ORJVRMFIOEXZ6OB6QFQDL764Q2FCWOJT63NBIRI6SOOS2FEIBCFQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil>
Nov 16 16:11:41.102688250 (176h7m39.654158711s) [noise=true , address=45.91.116.5:4297 , piece_id=OSMI3VGAKSCIZMDQ3Y2VZQBUOJKSXM6RWNWSAV57RDTL5TR3NIJA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: <nil>
Nov 16 16:12:04.264842280 (176h8m2.816312741s) [noise=true , address=145.79.227.17:4972 , piece_id=NV6FSBQM6YUTKYFAJPQCURWCZBHJ3AKJOLLGCRYTJ3KQ54U4GOWQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52522->145.79.227.17:4972: use of closed network connection
Nov 16 16:12:04.265084832 (176h8m2.816555293s) [noise=true , address=145.79.227.16:4910 , piece_id=3U2SEVMPILIEBRHS3M4YT4Z4PSYAQAIC2SKGYMW4FNODACZLOQCQ, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57568->145.79.227.16:4910: use of closed network connection
Nov 16 16:12:04.270267942 (176h8m2.821738423s) [noise=true , address=145.79.227.8:4329 , piece_id=GO5IAWQYS3UARWYAQAEUIIS456DFWCFWRJQG2KOVVL5KPXWPK4AA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: expected piece hash; context deadline exceeded; EOF
Nov 16 16:12:04.270306515 (176h8m2.821776986s) [noise=true , address=145.79.227.9:4367 , piece_id=LOFHPWCHNEZ4E4WANJM5BE2TIWVFH4LGEJJYK4ETB2LZWYCAVGZA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection; write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56132->145.79.227.9:4367: use of closed network connection
Nov 16 16:12:04.270323538 (176h8m2.821794099s) [noise=true , address=145.79.227.28:4725 , piece_id=CI2LSBGPW7RYH4RGWODGWLUTN353EODNV6IB2YQGHOZ3FPSEEK5A, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49278->145.79.227.28:4725: use of closed network connection
Nov 16 16:12:04.270544138 (176h8m2.822014739s) [noise=true , address=145.79.227.14:4840 , piece_id=RSHGL65RI7HZGU3W3EROFL23BCK2EW3YN6CAD3NO5OVE5QM3QG3Q, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: read tcp 188.166.225.170:42148->145.79.227.14:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:12:04.270571570 (176h8m2.822042121s) [noise=true , address=145.79.227.31:4797 , piece_id=AWSPIPRKSBOKQFZRRB34RUPLYVJGKAOOJQ4U7RRATQJELM3OCFHA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41664->145.79.227.31:4797: use of closed network connection
Nov 16 16:12:04.270609783 (176h8m2.822080254s) [noise=true , address=145.79.227.22:4511 , piece_id=HES465EP3OPD5JBVGVYMGLUOHJKSSMJDXMJVWET5GHRPFZJFIFOA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: read tcp 188.166.225.170:51004->145.79.227.22:4511: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:12:04.270625552 (176h8m2.822096083s) [noise=true , address=145.79.227.8:4331 , piece_id=4O27HU7AMR6G4Y43HJYN22UXVLETBDBE7JHL4JUZKRM54SK5E4JA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: read tcp 188.166.225.170:44510->145.79.227.8:4331: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:12:04.270742155 (176h8m2.822212616s) [noise=true , address=145.79.227.4:4186 , piece_id=5PZIK3V72CUQQ6M2HJZFVM5OVX4HPHY7ZHJ2BAK5S3ZQOGM4J5OA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44974->145.79.227.4:4186: use of closed network connection
Nov 16 16:12:04.270762814 (176h8m2.822233275s) [noise=true , address=145.79.227.24:4581 , piece_id=XRCS52EJDB7HWD4RDZK3UNB6VKHOTWWID42RGBJZQHTGG4NDYIPA, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: read tcp 188.166.225.170:44130->145.79.227.24:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:12:04.270781470 (176h8m2.822251921s) [noise=true , address=145.79.227.11:4732 , piece_id=FQGAVGZ5CID3LBET3CVK6FSCOP5OUYGS44XYWMHU2RLIT3MBN62A, seg_pos={0 0}, upload=155107]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: read tcp 188.166.225.170:42420->145.79.227.11:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:12:04.270825082 (176h8m2.822295544s) [seg_pos={0 0}, upload=155107]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil>
Nov 16 16:12:05.262661603 (176h8m3.814132064s) [upload=155107]: Done uploading segments...
Nov 16 16:12:05.262728080 (176h8m3.814198541s) [upload=155107]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4)
Duration: 5.999966242s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......................................................X....] duration:
6.636959414s trace:
https://jaeger.dev.storj.io/trace/3e4560ab56ee4602
FAIL (2025-11-16 16:14:03 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.636994753s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [..............................................X.............] duration:
5.963181558s trace:
https://jaeger.dev.storj.io/trace/7dbd01be7ae5d01e
FAIL (2025-11-16 16:13:48 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 24)
======LOGS======
OHA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.079132696 (176h0m30.630603157s) [noise=true , address=81.171.14.70:4040 , piece_id=YG6ZBK6QTKUS3FCFD2EWU7HTFLOEEVG57SIER44E67QDP5HYCHOA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.079319683 (176h0m30.630790144s) [noise=true , address=185.167.204.205:4015 , piece_id=6UIFX7GXE2SZ465VAP56YQILR3A3OHDRKIS345EE6GBDG77KJNZQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.079646465 (176h0m30.631116926s) [noise=true , address=185.167.204.206:4062 , piece_id=PAJX3ZCWM5YBZA7YDBFYXCET5CKWBOFJ47IUWK4CYPZSWK6HYQVQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.080002833 (176h0m30.631473294s) [noise=true , address=81.171.14.70:4031 , piece_id=54T5MOZZ4YJKQFUDOCCFF2SAE7XS2N77H2AOHCBECAWE5OBUXTZQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.080062066 (176h0m30.631532527s) [noise=true , address=185.167.204.206:4067 , piece_id=2BDMQZLCV4UWWP5HB6GF6NCL5UOGNU7VNYGJIIJCKQHRTX5OPGPQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.080438673 (176h0m30.631909134s) [noise=true , address=185.167.204.205:4034 , piece_id=CZFPF35OQ3U4IH46KILFPFMVY6OC36ZUGZ7EYYBOTB3I4TXMPERA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.080675103 (176h0m30.632145564s) [noise=true , address=185.167.204.205:4014 , piece_id=57C2VUYOEULOTXUQHG7A2XOWZLXYFIEYYRYW5ARX5PKTGWKIEOAQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.081315833 (176h0m30.632786294s) [noise=true , address=81.171.14.70:4039 , piece_id=NMZ5357FVIKD2F3GCKY6YNNV73BAID2YBLVNSSM6VD5PHGLBHEIA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.082053738 (176h0m30.633524189s) [noise=true , address=185.167.204.205:4001 , piece_id=TBPYFHMXWO53LJQQDUA3I2MD2WP6BHQXAH3KIF6XXMWK27HXLCHA, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:32.082605429 (176h0m30.63407589s) [noise=true , address=81.171.14.70:4034 , piece_id=NXBDCCCEWSZ2TMSET6OXBNQV6Q5ESXIQJUFK4WJJ3UV7FJTTAUXQ, seg_pos={0 0}, upload=155002]: Uploading piece...
Nov 16 16:04:33.280203414 (176h0m31.831673875s) [upload=155002]: Next returned nil segment
Nov 16 16:04:33.280222810 (176h0m31.831693271s) [upload=155002]: Waiting for error group managing segments...
Nov 16 16:04:33.838289781 (176h0m32.389760242s) [noise=true , address=185.167.204.205:4013 , piece_id=KF32EXUYGKQEDBPRSVAMNZV3ZMBF6DLT3U32BKX5LXLRBXBVTOQA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.848846382 (176h0m32.400316833s) [noise=true , address=185.167.204.206:4039 , piece_id=6HTUHESIRTSKR5EDMWZ2K56OKE5Y22QQM5ARYTUXR7KMKWIOS7GQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.862791868 (176h0m32.414262329s) [noise=true , address=185.167.204.205:4021 , piece_id=MV3TV4EI3HN3J5NY5XYVKT46QJTJEC5PITXERSSZMS2B7W3DSDTA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.879947135 (176h0m32.431417596s) [noise=true , address=185.167.204.205:4001 , piece_id=TBPYFHMXWO53LJQQDUA3I2MD2WP6BHQXAH3KIF6XXMWK27HXLCHA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.886853657 (176h0m32.438324118s) [noise=true , address=185.167.204.206:4067 , piece_id=2BDMQZLCV4UWWP5HB6GF6NCL5UOGNU7VNYGJIIJCKQHRTX5OPGPQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.918801783 (176h0m32.470272324s) [noise=true , address=185.167.204.206:4051 , piece_id=AU2S6M3E2CQEMAS7LKZJIVWITZUHIHCMWQ2QWT7ILGBRTAUCUB7Q, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.929982083 (176h0m32.481452574s) [noise=true , address=185.167.204.205:4015 , piece_id=6UIFX7GXE2SZ465VAP56YQILR3A3OHDRKIS345EE6GBDG77KJNZQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.935247409 (176h0m32.48671786s) [noise=true , address=185.167.204.206:4038 , piece_id=FBNPAJOOXEAT3KLKQIYOWHGHRGAWJABV4LAA3UYX6SGJB3U2LD7A, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.969630813 (176h0m32.521101264s) [noise=true , address=185.167.204.206:4055 , piece_id=B2TTK2ZLIRM3YIKP5HQ3WOL26KBHSXWWX5ZXXCMWU34AH7CHOOCQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:33.999043600 (176h0m32.550514061s) [noise=true , address=185.167.204.205:4034 , piece_id=CZFPF35OQ3U4IH46KILFPFMVY6OC36ZUGZ7EYYBOTB3I4TXMPERA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.022951616 (176h0m32.574422077s) [noise=true , address=185.167.204.205:4014 , piece_id=57C2VUYOEULOTXUQHG7A2XOWZLXYFIEYYRYW5ARX5PKTGWKIEOAQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.064488381 (176h0m32.615958832s) [noise=true , address=185.167.204.206:4058 , piece_id=POK7HBLE5AT2STCSL2UJ5R67LNVK3PN53NIRIBX4M7LJYC4C2CHQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.122257987 (176h0m32.673728468s) [noise=true , address=185.167.204.205:4029 , piece_id=VF7BTI4K75HTRVJOHIKMDJN3UBHZ7KCQBT3H44XB5G7ZVFD4HXJQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.136282884 (176h0m32.687753345s) [noise=true , address=185.167.204.206:4041 , piece_id=RTQSCXR54KTKBYS6AHEKPO7ZPUHVQQNUNQJ6JI4JLTGKOLUZH3AQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.136821679 (176h0m32.68829215s) [noise=true , address=185.167.204.206:4062 , piece_id=PAJX3ZCWM5YBZA7YDBFYXCET5CKWBOFJ47IUWK4CYPZSWK6HYQVQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.156096042 (176h0m32.707566493s) [noise=true , address=185.167.204.206:4066 , piece_id=CPTBHDHCT6R2M3UA6X2ZWMPJUKAUVV4I2CIATZ2ZOIWH63Q3UT4Q, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:04:34.540981809 (176h0m33.0924523s) [noise=true , address=89.149.201.147:4012 , piece_id=NEIYQHVVWRUVJFF5UD3RZSRHUSNSKXBCNGTMIGOH5MDH2AOZISCA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:21.258617296 (176h1m19.810087757s) [noise=true , address=81.171.14.70:4034 , piece_id=NXBDCCCEWSZ2TMSET6OXBNQV6Q5ESXIQJUFK4WJJ3UV7FJTTAUXQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:22.223432573 (176h1m20.774903034s) [noise=true , address=81.171.14.70:4048 , piece_id=5WUEBVU7UFKHYZ23BPIQUMUUBH3ZXXQWBLE7MALJGECCDTTEK5EA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:22.238494073 (176h1m20.789964534s) [noise=true , address=81.171.14.70:4033 , piece_id=LUBQIVFAU7J72K5XDDS5S2ZXQUGFA5AINL4R5FV64KY24VS5KV5Q, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:22.350384206 (176h1m20.901854667s) [noise=true , address=81.171.14.70:4039 , piece_id=NMZ5357FVIKD2F3GCKY6YNNV73BAID2YBLVNSSM6VD5PHGLBHEIA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:23.704484814 (176h1m22.255955275s) [noise=true , address=81.171.14.70:4037 , piece_id=3IJ7J7RBMOORW4YPKIYJ3C6ZQRYBH7WQMLEKEQYG5UJ4TGKE2W2A, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:24.501497158 (176h1m23.052967619s) [noise=true , address=81.171.14.70:4031 , piece_id=54T5MOZZ4YJKQFUDOCCFF2SAE7XS2N77H2AOHCBECAWE5OBUXTZQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:29.810461698 (176h1m28.361932159s) [noise=true , address=81.171.14.70:4025 , piece_id=3HAT6FJSGV6VOHJBMO6L6L7677BOYVCDV3SR26LJZ6HZOZWJJ64A, seg_pos={0 0}, upload=155002]: Done uploading piece. err: <nil>
Nov 16 16:05:31.413018058 (176h1m29.964488509s) [noise=true , address=81.171.14.70:4040 , piece_id=YG6ZBK6QTKUS3FCFD2EWU7HTFLOEEVG57SIER44E67QDP5HYCHOA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:1gannW154cmVP6N5BowytKfDorZDRwx2M5pi6akqjZFU38PGjV, address:81.171.14.70:4040): protocol: write tcp 188.166.225.170:59244->81.171.14.70:4040: use of closed network connection; write tcp 188.166.225.170:59244->81.171.14.70:4040: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59244->81.171.14.70:4040: use of closed network connection
Nov 16 16:05:31.413111366 (176h1m29.964581837s) [noise=true , address=81.171.14.70:4035 , piece_id=RGMXRH2Y5XBAFISK5LNAF547SAORD6XSWFUKYWOPJINXXODI5KVQ, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:12eLYJKne3yP1fZ1oPDLM7Xqs3GQ3ju4ZQedAiojXdVkcxNpzeA, address:81.171.14.70:4035): protocol: read tcp 188.166.225.170:55142->81.171.14.70:4035: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:05:31.413619965 (176h1m29.965090436s) [noise=true , address=81.171.14.70:4030 , piece_id=ZE5OAUTWLWNSMWSFGPEJTCXEN6L3FLWL34QBERTS3JZE7X2I2OHA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:12ZoRwa8wLw66Pjumy1ybp6wrZ79RrSU55uYZpwaY3xi95iSYde, address:81.171.14.70:4030): protocol: read tcp 188.166.225.170:40822->81.171.14.70:4030: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 16 16:05:31.414846280 (176h1m29.966316741s) [noise=true , address=81.171.14.70:4029 , piece_id=SIXL2XSCF7IO2BPOUKGH5VVXI6M7O4KJVZARYIRELR2GM4T4RABA, seg_pos={0 0}, upload=155002]: Done uploading piece. err: ecclient: upload failed (node:18yEfU2X4jSrom1JDtQ22cUEvomcg91ohvay2Nsr1Vr8juo9to, address:81.171.14.70:4029): protocol: read tcp 188.166.225.170:56584->81.171.14.70:4029: use of closed network connection; conn closed: op
Nov 16 16:05:31.414946261 (176h1m29.966416722s) [seg_pos={0 0}, upload=155002]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil>
Nov 16 16:05:31.591745995 (176h1m30.143216456s) [upload=155002]: Done uploading segments...
Nov 16 16:05:31.591788596 (176h1m30.143259057s) [upload=155002]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24)
Duration: 5.963285566s
core
uplink
eu-select-1
eu1-placement-15
|