Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....X......................................................] duration:
5.638975181s trace:
https://jaeger.dev.storj.io/trace/6c800b4c3a84b58c
FAIL (2025-11-22 13:12:52 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 5)
======LOGS======
N5RGYNBMYFXXBJDBTUCN37EAT7WBRNTX2YQA, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.890374814 (316h29m10.441845275s) [noise=true , address=145.79.227.32:4858 , piece_id=K5L3LEUKZWHS36XJHNNY4TYRXCP4DG5UAKCZXPJJTSG26CKUI4HQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.890535981 (316h29m10.442006442s) [noise=true , address=77.90.31.21:4890 , piece_id=DUEFPJUTYMD564Z4III63LZM6L4IL2ZTGM3QDZ6CG7WV5APHBNRA, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.890774064 (316h29m10.442244525s) [noise=true , address=45.91.116.14:4213 , piece_id=HPFISTE2N3B4RBDPAZAGVZJ5WYLCPHTCYH4WMJ56IWEHR3ZBEMOQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.890934851 (316h29m10.442405312s) [noise=true , address=45.91.116.6:4000 , piece_id=T4CI3KKNMFPT3ZIODHE7BQ4DVHVH2QTSO6OB7QZPL5SNPXQJE6TA, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.891167002 (316h29m10.442637463s) [noise=true , address=77.90.31.17:4787 , piece_id=IYIEVH6CARRYCARKOLKWVBICQDNDRCTGQMWF3VJZ6YBC3GO5OWHQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.891369828 (316h29m10.442840289s) [noise=true , address=45.91.116.7:4039 , piece_id=HSRY63FXTEPCP2T46V77RVSC24GGRBAYLLU7DTKOLWGVOYHF6YVA, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.891556875 (316h29m10.443027336s) [noise=true , address=77.90.31.11:4572 , piece_id=G4XSIPSWOLSEOGGFA2NLOSDCGTLZPI3P5LDNNLB5RGMU5OQGWXGQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.891896812 (316h29m10.443367273s) [noise=true , address=77.90.31.19:4863 , piece_id=3PM3Q6X3ADKX2RKIIFDMK57LTJNJHNFK4UYPKFK3ZEQLYQ23EP3A, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.892019235 (316h29m10.443489696s) [noise=true , address=77.90.31.10:4524 , piece_id=Z2IGDHPDU6JAR5CAFQBGYHO3TQ66K4BZOAMXSC7X3RFWLGHOOMTQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.892316521 (316h29m10.443786982s) [noise=true , address=77.90.31.13:4644 , piece_id=UPDZGC3CMJXA3AOJX65VLASU37NLFGPIJLCAPN32TEQB5QHTU7EQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.892682318 (316h29m10.444152779s) [noise=true , address=45.91.116.9:4274 , piece_id=MXWRAXYLBBGLUVTA5GP3JVX2IRM2UI2IC35D7KX3XMFEBXLIDX5A, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:13.353758506 (316h29m11.905228957s) [upload=278731]: Next returned nil segment
Nov 22 12:33:13.353772043 (316h29m11.905242504s) [upload=278731]: Waiting for error group managing segments...
Nov 22 12:33:13.751703718 (316h29m12.303174169s) [noise=true , address=145.79.227.32:4858 , piece_id=K5L3LEUKZWHS36XJHNNY4TYRXCP4DG5UAKCZXPJJTSG26CKUI4HQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil>
Nov 22 12:33:13.753660293 (316h29m12.305130764s) [noise=true , address=145.79.227.26:4674 , piece_id=UP7VIVRDRL6KERJSX3VLWTKGW3ESGCQ7BLYCBXGBB5UNIDE6OHQQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil>
Nov 22 12:33:13.760046123 (316h29m12.311516574s) [noise=true , address=145.79.227.3:4167 , piece_id=4STIVXGU7SOXWAIPG2PXWBLNDD7EYQ56E2DBN2UZ2AZ3TZSXCTIQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil>
Nov 22 12:33:57.373975014 (316h29m55.925445475s) [noise=true , address=77.90.31.13:4644 , piece_id=UPDZGC3CMJXA3AOJX65VLASU37NLFGPIJLCAPN32TEQB5QHTU7EQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil>
Nov 22 12:34:06.122275078 (316h30m4.673745539s) [noise=true , address=45.91.116.13:4181 , piece_id=MQNJEBQFZMHNDKAH2K3JEA7YBK2UK5BGSNCJF3MASF3CKVA5VD6Q, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil>
Nov 22 12:34:10.949171331 (316h30m9.500641792s) [noise=true , address=77.90.31.21:4890 , piece_id=DUEFPJUTYMD564Z4III63LZM6L4IL2ZTGM3QDZ6CG7WV5APHBNRA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1iE3mDnEhGDsMBtmUXC4EwvSrpyHLoybzVXkC9eoFuYEMHeBrd, address:77.90.31.21:4890): protocol: read tcp 188.166.225.170:55900->77.90.31.21:4890: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.949753650 (316h30m9.501224111s) [noise=true , address=45.91.116.12:4237 , piece_id=OG6XREYEGOYVFFMXCGBERRAIEMI2QMUGSBJBYP7GJQZ7FOAI4BIA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1FMjHdNGV42a9QZyMmSPTJWQrA8CdSRvXqrimnt6UGVreCzmgF, address:45.91.116.12:4237): protocol: rpc: dial tcp 45.91.116.12:4237: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.950066235 (316h30m9.501536706s) [noise=true , address=77.90.31.19:4863 , piece_id=3PM3Q6X3ADKX2RKIIFDMK57LTJNJHNFK4UYPKFK3ZEQLYQ23EP3A, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:124x1cbCPJRK8YkgSP6UzHymoLu1E1znnfg5WBiD89kSDLi8JeL, address:77.90.31.19:4863): protocol: write tcp 188.166.225.170:45968->77.90.31.19:4863: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.950246558 (316h30m9.50171702s) [noise=true , address=77.90.31.10:4524 , piece_id=Z2IGDHPDU6JAR5CAFQBGYHO3TQ66K4BZOAMXSC7X3RFWLGHOOMTQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12uN7CznoK4AD8FqFkNUndawdGJ9rmjAe7oNnjJkV844ScY87qe, address:77.90.31.10:4524): protocol: write tcp 188.166.225.170:40266->77.90.31.10:4524: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.950593058 (316h30m9.502063529s) [noise=true , address=77.90.31.14:4690 , piece_id=RJ3KXY3HURT7Q4HZYTOLSAL7Y6VMDGGRIQP6XHE4YB6OAA3PJRAA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.950853535 (316h30m9.502324006s) [noise=true , address=45.91.116.14:4213 , piece_id=HPFISTE2N3B4RBDPAZAGVZJ5WYLCPHTCYH4WMJ56IWEHR3ZBEMOQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12tQRqowH4aDr9h3s1WLs1o7na1ueiafwgUeeHfAB6g7mcE6e1t, address:45.91.116.14:4213): protocol: rpc: dial tcp 45.91.116.14:4213: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.951186438 (316h30m9.502656889s) [noise=true , address=45.91.116.6:4000 , piece_id=T4CI3KKNMFPT3ZIODHE7BQ4DVHVH2QTSO6OB7QZPL5SNPXQJE6TA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12CuCqHJYbanbuaEMHvtcuQXhvcdahD8eyT2fKDhqtxdSZ6ES4f, address:45.91.116.6:4000): protocol: rpc: dial tcp 45.91.116.6:4000: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.951409864 (316h30m9.502880325s) [noise=true , address=45.91.116.7:4039 , piece_id=HSRY63FXTEPCP2T46V77RVSC24GGRBAYLLU7DTKOLWGVOYHF6YVA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:16oq1Yv7JfHrg4gdHfhwuMj4UsWeQQPayKV6Pj1NY2CgJhuPr2, address:45.91.116.7:4039): protocol: write tcp 188.166.225.170:36904->45.91.116.7:4039: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.951660191 (316h30m9.503130652s) [noise=true , address=77.90.31.11:4572 , piece_id=G4XSIPSWOLSEOGGFA2NLOSDCGTLZPI3P5LDNNLB5RGMU5OQGWXGQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1ZAfYZ4wgpZVvatj6wtHona3hTaPcLxAmXC1DT2uKQyo8nLQQ7, address:77.90.31.11:4572): protocol: rpc: dial tcp 77.90.31.11:4572: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.952019855 (316h30m9.503490316s) [noise=true , address=45.91.116.9:4274 , piece_id=MXWRAXYLBBGLUVTA5GP3JVX2IRM2UI2IC35D7KX3XMFEBXLIDX5A, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12g6jFa38dTz5UdgsQaEuM2MSH1H7cxaysWsVmbHkM2Dd2W2iDb, address:45.91.116.9:4274): protocol: rpc: dial tcp 45.91.116.9:4274: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.952290972 (316h30m9.503761443s) [noise=true , address=77.90.31.15:4720 , piece_id=BDQUF6C5TT3ASE3DIN6SSVJHOZAJK4ZEW33OIYPTGB4LDVRELSUA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12VbsxFqMB3vVphSAJqYpbBzua7Goq45E6ovJSaExkpLtfkcc74, address:77.90.31.15:4720): protocol: rpc: dial tcp 77.90.31.15:4720: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.952544654 (316h30m9.504015105s) [noise=true , address=45.91.116.10:4110 , piece_id=H6FYA6K4N4AQYU3MN5RGYNBMYFXXBJDBTUCN37EAT7WBRNTX2YQA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:123KwusATu1ZKtNNVHZZZBE67ZEnVb8puyfQidBi3fEWREqdf1h, address:45.91.116.10:4110): protocol: rpc: dial tcp 45.91.116.10:4110: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.953548155 (316h30m9.505018617s) [noise=true , address=77.90.31.17:4787 , piece_id=IYIEVH6CARRYCARKOLKWVBICQDNDRCTGQMWF3VJZ6YBC3GO5OWHQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1Lietonb5aVbaZQt6wQ2EoxwJkSEn9oL9Dwyrc9mTyomEKikNt, address:77.90.31.17:4787): protocol: read tcp 188.166.225.170:57908->77.90.31.17:4787: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.953579816 (316h30m9.505050277s) [seg_pos={0 0}, upload=278731]: Done waiting for segment. successful: 5 optimal: 14 errs: <nil>
Nov 22 12:34:11.195719123 (316h30m9.747189584s) [upload=278731]: Done uploading segments...
Nov 22 12:34:11.195932840 (316h30m9.747403301s) [upload=278731]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 5)
Duration: 5.639006319s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [X..X........................................................] duration:
12.36125498s trace:
https://jaeger.dev.storj.io/trace/92df32505639913
FAIL (2025-11-22 13:12:49 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 12.361270248s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.X..........................................................] duration:
5.885095847s trace:
https://jaeger.dev.storj.io/trace/5ae3d2bf80495000
FAIL (2025-11-22 13:12:38 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Nov 22 12:31:07.276308940 (316h27m5.827779401s) [upload=278701]: Starting upload
Nov 22 12:31:07.276483523 (316h27m5.827953994s) [upload=278701]: Done starting upload
Nov 22 12:31:07.276562243 (316h27m5.828032704s) [upload=278701]: Uploading segments...
Nov 22 12:31:07.276950593 (316h27m5.828421054s) [upload=278701]: Got next segment. Inline: false
Nov 22 12:31:07.277027639 (316h27m5.8284981s) [upload=278701]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:31:07.277034181 (316h27m5.828504643s) [upload=278701]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:31:07.520212027 (316h27m6.071682488s) [seg_pos={0 0}, upload=278701]: Begin upload segment...
Nov 22 12:31:07.520269256 (316h27m6.071739717s) [seg_pos={0 0}, upload=278701]: Done begin upload segment.
Nov 22 12:31:07.520308992 (316h27m6.071779453s) [noise=true , address=145.79.227.27:4689 , piece_id=76S65KEZ7I2E76RR2JXOZ3ONGXCWSJA5XB7XQVFEEKGGNC34Z4EQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.520426746 (316h27m6.071897207s) [noise=true , address=145.79.227.21:4474 , piece_id=PCRWYQVCAZCPOILAEDHJGDBFKX4GEJ6JXCF7FUS7OVFRUTDPJI4Q, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.521175282 (316h27m6.072645743s) [noise=true , address=145.79.227.17:4972 , piece_id=UZK7LCIOBGGDNJZUVDBOLBJHGOIA4SUT2O3PD67PYQD3J2DGBJEA, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.521527573 (316h27m6.072998044s) [noise=true , address=145.79.227.23:4546 , piece_id=73UPPN2SYWNWEVWHCF4BRGKNO5GCRGOV3S5JZPX4MW75EJZRQOWQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.521694671 (316h27m6.073165132s) [noise=true , address=45.91.116.10:4082 , piece_id=GBDYXAWONOHQGBGM3NCRQYDSPFOAIHRJHBB2W6ZGJLMZUJLV65XA, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.522219410 (316h27m6.073689871s) [noise=true , address=145.79.227.20:4437 , piece_id=CZ32MB4AEZZ6YKCNWKRYALVAK7MT3KNXDSZCQTPUV2Q53NX6Y3BQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.522261240 (316h27m6.073731691s) [noise=true , address=145.79.227.16:4909 , piece_id=ZTL4ER5YJWSDGO5F5KLRUL5WRTA6WKVOJOERIBTRIIVEWXUY5Z2Q, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.522719151 (316h27m6.074189612s) [noise=true , address=145.79.227.26:4654 , piece_id=EM4RRGMPPNVR7VTXP3BED43FJFIUDZYVT4S5OUK2TRXZHYNMQ64Q, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.523017269 (316h27m6.07448772s) [noise=true , address=77.90.31.13:4627 , piece_id=Q7U5BFXHYQBBJQCU4EQGAR75WKL5QMBJ3LO34UYGO7QHLMDJBKWQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.523216949 (316h27m6.07468741s) [noise=true , address=145.79.227.25:4618 , piece_id=CV6M3CTPQR2MPQZFU6W34MFQERT6MMN3RRF2N3W35FGI464OSUUA, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.523744804 (316h27m6.075215265s) [noise=true , address=145.79.227.6:4258 , piece_id=B5D4TCVJORSAWDAU4DQ5VAV4CCRF5WUI24DUVUORAEKLAWGI7ABQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.523828153 (316h27m6.075298614s) [noise=true , address=145.79.227.24:4583 , piece_id=MLW4X34EWXKPOCFVI4VIQSL6RMUOJ5HRYHQ4HY35JIFT4ZFJ6F3Q, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.524255396 (316h27m6.075725858s) [noise=true , address=45.91.116.7:4009 , piece_id=FXQGF3UQN5CPFBS2HWD7BAXR4QIOF42MMK3VJQ7AK4THYJWOFFLQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.524662051 (316h27m6.076132512s) [noise=true , address=145.79.227.15:4874 , piece_id=TYAXIF3ZJQQSFGFANYF56DG4O5WBBXK4M7OT67E7ARYNZ663HESA, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.524746221 (316h27m6.076216682s) [noise=true , address=145.79.227.28:4726 , piece_id=3LBJV2B4CVXEMHCOPRW7DR5QBKPS6DFMPTHQSVNR4XRTZZVIGBQA, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.525227157 (316h27m6.076697608s) [noise=true , address=77.90.31.21:4879 , piece_id=GHTH55OGGXBAZZ5FGPCDYVHDYEETM7DNVTPSTTFFFRZMKZZ3XRXQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:09.134610631 (316h27m7.686081093s) [upload=278701]: Next returned nil segment
Nov 22 12:31:09.134628024 (316h27m7.686098495s) [upload=278701]: Waiting for error group managing segments...
Nov 22 12:31:09.502125778 (316h27m8.053596239s) [noise=true , address=145.79.227.16:4909 , piece_id=ZTL4ER5YJWSDGO5F5KLRUL5WRTA6WKVOJOERIBTRIIVEWXUY5Z2Q, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.503083742 (316h27m8.054554193s) [noise=true , address=145.79.227.20:4437 , piece_id=CZ32MB4AEZZ6YKCNWKRYALVAK7MT3KNXDSZCQTPUV2Q53NX6Y3BQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.517732716 (316h27m8.069203177s) [noise=true , address=145.79.227.25:4618 , piece_id=CV6M3CTPQR2MPQZFU6W34MFQERT6MMN3RRF2N3W35FGI464OSUUA, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.518029582 (316h27m8.069500043s) [noise=true , address=145.79.227.28:4726 , piece_id=3LBJV2B4CVXEMHCOPRW7DR5QBKPS6DFMPTHQSVNR4XRTZZVIGBQA, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.519935051 (316h27m8.071405512s) [noise=true , address=145.79.227.26:4654 , piece_id=EM4RRGMPPNVR7VTXP3BED43FJFIUDZYVT4S5OUK2TRXZHYNMQ64Q, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.520607661 (316h27m8.072078122s) [noise=true , address=145.79.227.27:4689 , piece_id=76S65KEZ7I2E76RR2JXOZ3ONGXCWSJA5XB7XQVFEEKGGNC34Z4EQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.521187736 (316h27m8.072658197s) [noise=true , address=145.79.227.23:4546 , piece_id=73UPPN2SYWNWEVWHCF4BRGKNO5GCRGOV3S5JZPX4MW75EJZRQOWQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.532974760 (316h27m8.084445221s) [noise=true , address=145.79.227.24:4583 , piece_id=MLW4X34EWXKPOCFVI4VIQSL6RMUOJ5HRYHQ4HY35JIFT4ZFJ6F3Q, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.533688800 (316h27m8.085159271s) [noise=true , address=145.79.227.6:4258 , piece_id=B5D4TCVJORSAWDAU4DQ5VAV4CCRF5WUI24DUVUORAEKLAWGI7ABQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.534460109 (316h27m8.08593057s) [noise=true , address=145.79.227.15:4874 , piece_id=TYAXIF3ZJQQSFGFANYF56DG4O5WBBXK4M7OT67E7ARYNZ663HESA, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.542955375 (316h27m8.094425826s) [noise=true , address=145.79.227.17:4972 , piece_id=UZK7LCIOBGGDNJZUVDBOLBJHGOIA4SUT2O3PD67PYQD3J2DGBJEA, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.563098982 (316h27m8.114569443s) [noise=true , address=145.79.227.21:4474 , piece_id=PCRWYQVCAZCPOILAEDHJGDBFKX4GEJ6JXCF7FUS7OVFRUTDPJI4Q, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:36.585475346 (316h27m35.136945807s) [noise=true , address=45.91.116.10:4082 , piece_id=GBDYXAWONOHQGBGM3NCRQYDSPFOAIHRJHBB2W6ZGJLMZUJLV65XA, seg_pos={0 0}, upload=278701]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: read tcp 188.166.225.170:54326->45.91.116.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:36.585499923 (316h27m35.136970384s) [noise=true , address=77.90.31.21:4879 , piece_id=GHTH55OGGXBAZZ5FGPCDYVHDYEETM7DNVTPSTTFFFRZMKZZ3XRXQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: rpc: dial tcp 77.90.31.21:4879: i/o timeout; context deadline exceeded
Nov 22 12:31:36.585749648 (316h27m35.137220119s) [noise=true , address=77.90.31.13:4627 , piece_id=Q7U5BFXHYQBBJQCU4EQGAR75WKL5QMBJ3LO34UYGO7QHLMDJBKWQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:36.585848086 (316h27m35.137318547s) [noise=true , address=45.91.116.7:4009 , piece_id=FXQGF3UQN5CPFBS2HWD7BAXR4QIOF42MMK3VJQ7AK4THYJWOFFLQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:36.585900936 (316h27m35.137371397s) [seg_pos={0 0}, upload=278701]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Nov 22 12:31:36.828881025 (316h27m35.380351486s) [upload=278701]: Done uploading segments...
Nov 22 12:31:36.828915801 (316h27m35.380386262s) [upload=278701]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 5.885126074s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [X.....X.....................................................] duration:
5.676887151s trace:
https://jaeger.dev.storj.io/trace/6c800b4c3a84b58c
FAIL (2025-11-22 13:12:25 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 5)
======LOGS======
N5RGYNBMYFXXBJDBTUCN37EAT7WBRNTX2YQA, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.890374814 (316h29m10.441845275s) [noise=true , address=145.79.227.32:4858 , piece_id=K5L3LEUKZWHS36XJHNNY4TYRXCP4DG5UAKCZXPJJTSG26CKUI4HQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.890535981 (316h29m10.442006442s) [noise=true , address=77.90.31.21:4890 , piece_id=DUEFPJUTYMD564Z4III63LZM6L4IL2ZTGM3QDZ6CG7WV5APHBNRA, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.890774064 (316h29m10.442244525s) [noise=true , address=45.91.116.14:4213 , piece_id=HPFISTE2N3B4RBDPAZAGVZJ5WYLCPHTCYH4WMJ56IWEHR3ZBEMOQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.890934851 (316h29m10.442405312s) [noise=true , address=45.91.116.6:4000 , piece_id=T4CI3KKNMFPT3ZIODHE7BQ4DVHVH2QTSO6OB7QZPL5SNPXQJE6TA, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.891167002 (316h29m10.442637463s) [noise=true , address=77.90.31.17:4787 , piece_id=IYIEVH6CARRYCARKOLKWVBICQDNDRCTGQMWF3VJZ6YBC3GO5OWHQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.891369828 (316h29m10.442840289s) [noise=true , address=45.91.116.7:4039 , piece_id=HSRY63FXTEPCP2T46V77RVSC24GGRBAYLLU7DTKOLWGVOYHF6YVA, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.891556875 (316h29m10.443027336s) [noise=true , address=77.90.31.11:4572 , piece_id=G4XSIPSWOLSEOGGFA2NLOSDCGTLZPI3P5LDNNLB5RGMU5OQGWXGQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.891896812 (316h29m10.443367273s) [noise=true , address=77.90.31.19:4863 , piece_id=3PM3Q6X3ADKX2RKIIFDMK57LTJNJHNFK4UYPKFK3ZEQLYQ23EP3A, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.892019235 (316h29m10.443489696s) [noise=true , address=77.90.31.10:4524 , piece_id=Z2IGDHPDU6JAR5CAFQBGYHO3TQ66K4BZOAMXSC7X3RFWLGHOOMTQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.892316521 (316h29m10.443786982s) [noise=true , address=77.90.31.13:4644 , piece_id=UPDZGC3CMJXA3AOJX65VLASU37NLFGPIJLCAPN32TEQB5QHTU7EQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.892682318 (316h29m10.444152779s) [noise=true , address=45.91.116.9:4274 , piece_id=MXWRAXYLBBGLUVTA5GP3JVX2IRM2UI2IC35D7KX3XMFEBXLIDX5A, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:13.353758506 (316h29m11.905228957s) [upload=278731]: Next returned nil segment
Nov 22 12:33:13.353772043 (316h29m11.905242504s) [upload=278731]: Waiting for error group managing segments...
Nov 22 12:33:13.751703718 (316h29m12.303174169s) [noise=true , address=145.79.227.32:4858 , piece_id=K5L3LEUKZWHS36XJHNNY4TYRXCP4DG5UAKCZXPJJTSG26CKUI4HQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil>
Nov 22 12:33:13.753660293 (316h29m12.305130764s) [noise=true , address=145.79.227.26:4674 , piece_id=UP7VIVRDRL6KERJSX3VLWTKGW3ESGCQ7BLYCBXGBB5UNIDE6OHQQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil>
Nov 22 12:33:13.760046123 (316h29m12.311516574s) [noise=true , address=145.79.227.3:4167 , piece_id=4STIVXGU7SOXWAIPG2PXWBLNDD7EYQ56E2DBN2UZ2AZ3TZSXCTIQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil>
Nov 22 12:33:57.373975014 (316h29m55.925445475s) [noise=true , address=77.90.31.13:4644 , piece_id=UPDZGC3CMJXA3AOJX65VLASU37NLFGPIJLCAPN32TEQB5QHTU7EQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil>
Nov 22 12:34:06.122275078 (316h30m4.673745539s) [noise=true , address=45.91.116.13:4181 , piece_id=MQNJEBQFZMHNDKAH2K3JEA7YBK2UK5BGSNCJF3MASF3CKVA5VD6Q, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil>
Nov 22 12:34:10.949171331 (316h30m9.500641792s) [noise=true , address=77.90.31.21:4890 , piece_id=DUEFPJUTYMD564Z4III63LZM6L4IL2ZTGM3QDZ6CG7WV5APHBNRA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1iE3mDnEhGDsMBtmUXC4EwvSrpyHLoybzVXkC9eoFuYEMHeBrd, address:77.90.31.21:4890): protocol: read tcp 188.166.225.170:55900->77.90.31.21:4890: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.949753650 (316h30m9.501224111s) [noise=true , address=45.91.116.12:4237 , piece_id=OG6XREYEGOYVFFMXCGBERRAIEMI2QMUGSBJBYP7GJQZ7FOAI4BIA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1FMjHdNGV42a9QZyMmSPTJWQrA8CdSRvXqrimnt6UGVreCzmgF, address:45.91.116.12:4237): protocol: rpc: dial tcp 45.91.116.12:4237: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.950066235 (316h30m9.501536706s) [noise=true , address=77.90.31.19:4863 , piece_id=3PM3Q6X3ADKX2RKIIFDMK57LTJNJHNFK4UYPKFK3ZEQLYQ23EP3A, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:124x1cbCPJRK8YkgSP6UzHymoLu1E1znnfg5WBiD89kSDLi8JeL, address:77.90.31.19:4863): protocol: write tcp 188.166.225.170:45968->77.90.31.19:4863: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.950246558 (316h30m9.50171702s) [noise=true , address=77.90.31.10:4524 , piece_id=Z2IGDHPDU6JAR5CAFQBGYHO3TQ66K4BZOAMXSC7X3RFWLGHOOMTQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12uN7CznoK4AD8FqFkNUndawdGJ9rmjAe7oNnjJkV844ScY87qe, address:77.90.31.10:4524): protocol: write tcp 188.166.225.170:40266->77.90.31.10:4524: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.950593058 (316h30m9.502063529s) [noise=true , address=77.90.31.14:4690 , piece_id=RJ3KXY3HURT7Q4HZYTOLSAL7Y6VMDGGRIQP6XHE4YB6OAA3PJRAA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.950853535 (316h30m9.502324006s) [noise=true , address=45.91.116.14:4213 , piece_id=HPFISTE2N3B4RBDPAZAGVZJ5WYLCPHTCYH4WMJ56IWEHR3ZBEMOQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12tQRqowH4aDr9h3s1WLs1o7na1ueiafwgUeeHfAB6g7mcE6e1t, address:45.91.116.14:4213): protocol: rpc: dial tcp 45.91.116.14:4213: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.951186438 (316h30m9.502656889s) [noise=true , address=45.91.116.6:4000 , piece_id=T4CI3KKNMFPT3ZIODHE7BQ4DVHVH2QTSO6OB7QZPL5SNPXQJE6TA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12CuCqHJYbanbuaEMHvtcuQXhvcdahD8eyT2fKDhqtxdSZ6ES4f, address:45.91.116.6:4000): protocol: rpc: dial tcp 45.91.116.6:4000: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.951409864 (316h30m9.502880325s) [noise=true , address=45.91.116.7:4039 , piece_id=HSRY63FXTEPCP2T46V77RVSC24GGRBAYLLU7DTKOLWGVOYHF6YVA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:16oq1Yv7JfHrg4gdHfhwuMj4UsWeQQPayKV6Pj1NY2CgJhuPr2, address:45.91.116.7:4039): protocol: write tcp 188.166.225.170:36904->45.91.116.7:4039: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.951660191 (316h30m9.503130652s) [noise=true , address=77.90.31.11:4572 , piece_id=G4XSIPSWOLSEOGGFA2NLOSDCGTLZPI3P5LDNNLB5RGMU5OQGWXGQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1ZAfYZ4wgpZVvatj6wtHona3hTaPcLxAmXC1DT2uKQyo8nLQQ7, address:77.90.31.11:4572): protocol: rpc: dial tcp 77.90.31.11:4572: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.952019855 (316h30m9.503490316s) [noise=true , address=45.91.116.9:4274 , piece_id=MXWRAXYLBBGLUVTA5GP3JVX2IRM2UI2IC35D7KX3XMFEBXLIDX5A, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12g6jFa38dTz5UdgsQaEuM2MSH1H7cxaysWsVmbHkM2Dd2W2iDb, address:45.91.116.9:4274): protocol: rpc: dial tcp 45.91.116.9:4274: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.952290972 (316h30m9.503761443s) [noise=true , address=77.90.31.15:4720 , piece_id=BDQUF6C5TT3ASE3DIN6SSVJHOZAJK4ZEW33OIYPTGB4LDVRELSUA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12VbsxFqMB3vVphSAJqYpbBzua7Goq45E6ovJSaExkpLtfkcc74, address:77.90.31.15:4720): protocol: rpc: dial tcp 77.90.31.15:4720: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.952544654 (316h30m9.504015105s) [noise=true , address=45.91.116.10:4110 , piece_id=H6FYA6K4N4AQYU3MN5RGYNBMYFXXBJDBTUCN37EAT7WBRNTX2YQA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:123KwusATu1ZKtNNVHZZZBE67ZEnVb8puyfQidBi3fEWREqdf1h, address:45.91.116.10:4110): protocol: rpc: dial tcp 45.91.116.10:4110: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.953548155 (316h30m9.505018617s) [noise=true , address=77.90.31.17:4787 , piece_id=IYIEVH6CARRYCARKOLKWVBICQDNDRCTGQMWF3VJZ6YBC3GO5OWHQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1Lietonb5aVbaZQt6wQ2EoxwJkSEn9oL9Dwyrc9mTyomEKikNt, address:77.90.31.17:4787): protocol: read tcp 188.166.225.170:57908->77.90.31.17:4787: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.953579816 (316h30m9.505050277s) [seg_pos={0 0}, upload=278731]: Done waiting for segment. successful: 5 optimal: 14 errs: <nil>
Nov 22 12:34:11.195719123 (316h30m9.747189584s) [upload=278731]: Done uploading segments...
Nov 22 12:34:11.195932840 (316h30m9.747403301s) [upload=278731]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 5)
Duration: 5.676932397s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
5.900797075s trace:
https://jaeger.dev.storj.io/trace/570ccd0ce1345814
FAIL (2025-11-22 13:12:25 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.90083197s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.X..X.......................................................] duration:
21.376050474s trace:
https://jaeger.dev.storj.io/trace/92df32505639913
FAIL (2025-11-22 13:12:10 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 21.376086202s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..X.........................................................] duration:
5.943770383s trace:
https://jaeger.dev.storj.io/trace/5ae3d2bf80495000
FAIL (2025-11-22 13:12:09 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Nov 22 12:31:07.276308940 (316h27m5.827779401s) [upload=278701]: Starting upload
Nov 22 12:31:07.276483523 (316h27m5.827953994s) [upload=278701]: Done starting upload
Nov 22 12:31:07.276562243 (316h27m5.828032704s) [upload=278701]: Uploading segments...
Nov 22 12:31:07.276950593 (316h27m5.828421054s) [upload=278701]: Got next segment. Inline: false
Nov 22 12:31:07.277027639 (316h27m5.8284981s) [upload=278701]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:31:07.277034181 (316h27m5.828504643s) [upload=278701]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:31:07.520212027 (316h27m6.071682488s) [seg_pos={0 0}, upload=278701]: Begin upload segment...
Nov 22 12:31:07.520269256 (316h27m6.071739717s) [seg_pos={0 0}, upload=278701]: Done begin upload segment.
Nov 22 12:31:07.520308992 (316h27m6.071779453s) [noise=true , address=145.79.227.27:4689 , piece_id=76S65KEZ7I2E76RR2JXOZ3ONGXCWSJA5XB7XQVFEEKGGNC34Z4EQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.520426746 (316h27m6.071897207s) [noise=true , address=145.79.227.21:4474 , piece_id=PCRWYQVCAZCPOILAEDHJGDBFKX4GEJ6JXCF7FUS7OVFRUTDPJI4Q, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.521175282 (316h27m6.072645743s) [noise=true , address=145.79.227.17:4972 , piece_id=UZK7LCIOBGGDNJZUVDBOLBJHGOIA4SUT2O3PD67PYQD3J2DGBJEA, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.521527573 (316h27m6.072998044s) [noise=true , address=145.79.227.23:4546 , piece_id=73UPPN2SYWNWEVWHCF4BRGKNO5GCRGOV3S5JZPX4MW75EJZRQOWQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.521694671 (316h27m6.073165132s) [noise=true , address=45.91.116.10:4082 , piece_id=GBDYXAWONOHQGBGM3NCRQYDSPFOAIHRJHBB2W6ZGJLMZUJLV65XA, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.522219410 (316h27m6.073689871s) [noise=true , address=145.79.227.20:4437 , piece_id=CZ32MB4AEZZ6YKCNWKRYALVAK7MT3KNXDSZCQTPUV2Q53NX6Y3BQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.522261240 (316h27m6.073731691s) [noise=true , address=145.79.227.16:4909 , piece_id=ZTL4ER5YJWSDGO5F5KLRUL5WRTA6WKVOJOERIBTRIIVEWXUY5Z2Q, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.522719151 (316h27m6.074189612s) [noise=true , address=145.79.227.26:4654 , piece_id=EM4RRGMPPNVR7VTXP3BED43FJFIUDZYVT4S5OUK2TRXZHYNMQ64Q, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.523017269 (316h27m6.07448772s) [noise=true , address=77.90.31.13:4627 , piece_id=Q7U5BFXHYQBBJQCU4EQGAR75WKL5QMBJ3LO34UYGO7QHLMDJBKWQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.523216949 (316h27m6.07468741s) [noise=true , address=145.79.227.25:4618 , piece_id=CV6M3CTPQR2MPQZFU6W34MFQERT6MMN3RRF2N3W35FGI464OSUUA, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.523744804 (316h27m6.075215265s) [noise=true , address=145.79.227.6:4258 , piece_id=B5D4TCVJORSAWDAU4DQ5VAV4CCRF5WUI24DUVUORAEKLAWGI7ABQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.523828153 (316h27m6.075298614s) [noise=true , address=145.79.227.24:4583 , piece_id=MLW4X34EWXKPOCFVI4VIQSL6RMUOJ5HRYHQ4HY35JIFT4ZFJ6F3Q, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.524255396 (316h27m6.075725858s) [noise=true , address=45.91.116.7:4009 , piece_id=FXQGF3UQN5CPFBS2HWD7BAXR4QIOF42MMK3VJQ7AK4THYJWOFFLQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.524662051 (316h27m6.076132512s) [noise=true , address=145.79.227.15:4874 , piece_id=TYAXIF3ZJQQSFGFANYF56DG4O5WBBXK4M7OT67E7ARYNZ663HESA, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.524746221 (316h27m6.076216682s) [noise=true , address=145.79.227.28:4726 , piece_id=3LBJV2B4CVXEMHCOPRW7DR5QBKPS6DFMPTHQSVNR4XRTZZVIGBQA, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.525227157 (316h27m6.076697608s) [noise=true , address=77.90.31.21:4879 , piece_id=GHTH55OGGXBAZZ5FGPCDYVHDYEETM7DNVTPSTTFFFRZMKZZ3XRXQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:09.134610631 (316h27m7.686081093s) [upload=278701]: Next returned nil segment
Nov 22 12:31:09.134628024 (316h27m7.686098495s) [upload=278701]: Waiting for error group managing segments...
Nov 22 12:31:09.502125778 (316h27m8.053596239s) [noise=true , address=145.79.227.16:4909 , piece_id=ZTL4ER5YJWSDGO5F5KLRUL5WRTA6WKVOJOERIBTRIIVEWXUY5Z2Q, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.503083742 (316h27m8.054554193s) [noise=true , address=145.79.227.20:4437 , piece_id=CZ32MB4AEZZ6YKCNWKRYALVAK7MT3KNXDSZCQTPUV2Q53NX6Y3BQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.517732716 (316h27m8.069203177s) [noise=true , address=145.79.227.25:4618 , piece_id=CV6M3CTPQR2MPQZFU6W34MFQERT6MMN3RRF2N3W35FGI464OSUUA, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.518029582 (316h27m8.069500043s) [noise=true , address=145.79.227.28:4726 , piece_id=3LBJV2B4CVXEMHCOPRW7DR5QBKPS6DFMPTHQSVNR4XRTZZVIGBQA, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.519935051 (316h27m8.071405512s) [noise=true , address=145.79.227.26:4654 , piece_id=EM4RRGMPPNVR7VTXP3BED43FJFIUDZYVT4S5OUK2TRXZHYNMQ64Q, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.520607661 (316h27m8.072078122s) [noise=true , address=145.79.227.27:4689 , piece_id=76S65KEZ7I2E76RR2JXOZ3ONGXCWSJA5XB7XQVFEEKGGNC34Z4EQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.521187736 (316h27m8.072658197s) [noise=true , address=145.79.227.23:4546 , piece_id=73UPPN2SYWNWEVWHCF4BRGKNO5GCRGOV3S5JZPX4MW75EJZRQOWQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.532974760 (316h27m8.084445221s) [noise=true , address=145.79.227.24:4583 , piece_id=MLW4X34EWXKPOCFVI4VIQSL6RMUOJ5HRYHQ4HY35JIFT4ZFJ6F3Q, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.533688800 (316h27m8.085159271s) [noise=true , address=145.79.227.6:4258 , piece_id=B5D4TCVJORSAWDAU4DQ5VAV4CCRF5WUI24DUVUORAEKLAWGI7ABQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.534460109 (316h27m8.08593057s) [noise=true , address=145.79.227.15:4874 , piece_id=TYAXIF3ZJQQSFGFANYF56DG4O5WBBXK4M7OT67E7ARYNZ663HESA, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.542955375 (316h27m8.094425826s) [noise=true , address=145.79.227.17:4972 , piece_id=UZK7LCIOBGGDNJZUVDBOLBJHGOIA4SUT2O3PD67PYQD3J2DGBJEA, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.563098982 (316h27m8.114569443s) [noise=true , address=145.79.227.21:4474 , piece_id=PCRWYQVCAZCPOILAEDHJGDBFKX4GEJ6JXCF7FUS7OVFRUTDPJI4Q, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:36.585475346 (316h27m35.136945807s) [noise=true , address=45.91.116.10:4082 , piece_id=GBDYXAWONOHQGBGM3NCRQYDSPFOAIHRJHBB2W6ZGJLMZUJLV65XA, seg_pos={0 0}, upload=278701]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: read tcp 188.166.225.170:54326->45.91.116.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:36.585499923 (316h27m35.136970384s) [noise=true , address=77.90.31.21:4879 , piece_id=GHTH55OGGXBAZZ5FGPCDYVHDYEETM7DNVTPSTTFFFRZMKZZ3XRXQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: rpc: dial tcp 77.90.31.21:4879: i/o timeout; context deadline exceeded
Nov 22 12:31:36.585749648 (316h27m35.137220119s) [noise=true , address=77.90.31.13:4627 , piece_id=Q7U5BFXHYQBBJQCU4EQGAR75WKL5QMBJ3LO34UYGO7QHLMDJBKWQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:36.585848086 (316h27m35.137318547s) [noise=true , address=45.91.116.7:4009 , piece_id=FXQGF3UQN5CPFBS2HWD7BAXR4QIOF42MMK3VJQ7AK4THYJWOFFLQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:36.585900936 (316h27m35.137371397s) [seg_pos={0 0}, upload=278701]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Nov 22 12:31:36.828881025 (316h27m35.380351486s) [upload=278701]: Done uploading segments...
Nov 22 12:31:36.828915801 (316h27m35.380386262s) [upload=278701]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 5.943804609s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.X..........................................................] duration:
15.124095666s trace:
https://jaeger.dev.storj.io/trace/570ccd0ce1345814
FAIL (2025-11-22 13:12:09 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 15.12415107s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [.X.....X....................................................] duration:
5.625187496s trace:
https://jaeger.dev.storj.io/trace/6c800b4c3a84b58c
FAIL (2025-11-22 13:12:05 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 5)
======LOGS======
N5RGYNBMYFXXBJDBTUCN37EAT7WBRNTX2YQA, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.890374814 (316h29m10.441845275s) [noise=true , address=145.79.227.32:4858 , piece_id=K5L3LEUKZWHS36XJHNNY4TYRXCP4DG5UAKCZXPJJTSG26CKUI4HQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.890535981 (316h29m10.442006442s) [noise=true , address=77.90.31.21:4890 , piece_id=DUEFPJUTYMD564Z4III63LZM6L4IL2ZTGM3QDZ6CG7WV5APHBNRA, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.890774064 (316h29m10.442244525s) [noise=true , address=45.91.116.14:4213 , piece_id=HPFISTE2N3B4RBDPAZAGVZJ5WYLCPHTCYH4WMJ56IWEHR3ZBEMOQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.890934851 (316h29m10.442405312s) [noise=true , address=45.91.116.6:4000 , piece_id=T4CI3KKNMFPT3ZIODHE7BQ4DVHVH2QTSO6OB7QZPL5SNPXQJE6TA, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.891167002 (316h29m10.442637463s) [noise=true , address=77.90.31.17:4787 , piece_id=IYIEVH6CARRYCARKOLKWVBICQDNDRCTGQMWF3VJZ6YBC3GO5OWHQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.891369828 (316h29m10.442840289s) [noise=true , address=45.91.116.7:4039 , piece_id=HSRY63FXTEPCP2T46V77RVSC24GGRBAYLLU7DTKOLWGVOYHF6YVA, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.891556875 (316h29m10.443027336s) [noise=true , address=77.90.31.11:4572 , piece_id=G4XSIPSWOLSEOGGFA2NLOSDCGTLZPI3P5LDNNLB5RGMU5OQGWXGQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.891896812 (316h29m10.443367273s) [noise=true , address=77.90.31.19:4863 , piece_id=3PM3Q6X3ADKX2RKIIFDMK57LTJNJHNFK4UYPKFK3ZEQLYQ23EP3A, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.892019235 (316h29m10.443489696s) [noise=true , address=77.90.31.10:4524 , piece_id=Z2IGDHPDU6JAR5CAFQBGYHO3TQ66K4BZOAMXSC7X3RFWLGHOOMTQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.892316521 (316h29m10.443786982s) [noise=true , address=77.90.31.13:4644 , piece_id=UPDZGC3CMJXA3AOJX65VLASU37NLFGPIJLCAPN32TEQB5QHTU7EQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.892682318 (316h29m10.444152779s) [noise=true , address=45.91.116.9:4274 , piece_id=MXWRAXYLBBGLUVTA5GP3JVX2IRM2UI2IC35D7KX3XMFEBXLIDX5A, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:13.353758506 (316h29m11.905228957s) [upload=278731]: Next returned nil segment
Nov 22 12:33:13.353772043 (316h29m11.905242504s) [upload=278731]: Waiting for error group managing segments...
Nov 22 12:33:13.751703718 (316h29m12.303174169s) [noise=true , address=145.79.227.32:4858 , piece_id=K5L3LEUKZWHS36XJHNNY4TYRXCP4DG5UAKCZXPJJTSG26CKUI4HQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil>
Nov 22 12:33:13.753660293 (316h29m12.305130764s) [noise=true , address=145.79.227.26:4674 , piece_id=UP7VIVRDRL6KERJSX3VLWTKGW3ESGCQ7BLYCBXGBB5UNIDE6OHQQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil>
Nov 22 12:33:13.760046123 (316h29m12.311516574s) [noise=true , address=145.79.227.3:4167 , piece_id=4STIVXGU7SOXWAIPG2PXWBLNDD7EYQ56E2DBN2UZ2AZ3TZSXCTIQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil>
Nov 22 12:33:57.373975014 (316h29m55.925445475s) [noise=true , address=77.90.31.13:4644 , piece_id=UPDZGC3CMJXA3AOJX65VLASU37NLFGPIJLCAPN32TEQB5QHTU7EQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil>
Nov 22 12:34:06.122275078 (316h30m4.673745539s) [noise=true , address=45.91.116.13:4181 , piece_id=MQNJEBQFZMHNDKAH2K3JEA7YBK2UK5BGSNCJF3MASF3CKVA5VD6Q, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil>
Nov 22 12:34:10.949171331 (316h30m9.500641792s) [noise=true , address=77.90.31.21:4890 , piece_id=DUEFPJUTYMD564Z4III63LZM6L4IL2ZTGM3QDZ6CG7WV5APHBNRA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1iE3mDnEhGDsMBtmUXC4EwvSrpyHLoybzVXkC9eoFuYEMHeBrd, address:77.90.31.21:4890): protocol: read tcp 188.166.225.170:55900->77.90.31.21:4890: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.949753650 (316h30m9.501224111s) [noise=true , address=45.91.116.12:4237 , piece_id=OG6XREYEGOYVFFMXCGBERRAIEMI2QMUGSBJBYP7GJQZ7FOAI4BIA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1FMjHdNGV42a9QZyMmSPTJWQrA8CdSRvXqrimnt6UGVreCzmgF, address:45.91.116.12:4237): protocol: rpc: dial tcp 45.91.116.12:4237: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.950066235 (316h30m9.501536706s) [noise=true , address=77.90.31.19:4863 , piece_id=3PM3Q6X3ADKX2RKIIFDMK57LTJNJHNFK4UYPKFK3ZEQLYQ23EP3A, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:124x1cbCPJRK8YkgSP6UzHymoLu1E1znnfg5WBiD89kSDLi8JeL, address:77.90.31.19:4863): protocol: write tcp 188.166.225.170:45968->77.90.31.19:4863: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.950246558 (316h30m9.50171702s) [noise=true , address=77.90.31.10:4524 , piece_id=Z2IGDHPDU6JAR5CAFQBGYHO3TQ66K4BZOAMXSC7X3RFWLGHOOMTQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12uN7CznoK4AD8FqFkNUndawdGJ9rmjAe7oNnjJkV844ScY87qe, address:77.90.31.10:4524): protocol: write tcp 188.166.225.170:40266->77.90.31.10:4524: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.950593058 (316h30m9.502063529s) [noise=true , address=77.90.31.14:4690 , piece_id=RJ3KXY3HURT7Q4HZYTOLSAL7Y6VMDGGRIQP6XHE4YB6OAA3PJRAA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.950853535 (316h30m9.502324006s) [noise=true , address=45.91.116.14:4213 , piece_id=HPFISTE2N3B4RBDPAZAGVZJ5WYLCPHTCYH4WMJ56IWEHR3ZBEMOQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12tQRqowH4aDr9h3s1WLs1o7na1ueiafwgUeeHfAB6g7mcE6e1t, address:45.91.116.14:4213): protocol: rpc: dial tcp 45.91.116.14:4213: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.951186438 (316h30m9.502656889s) [noise=true , address=45.91.116.6:4000 , piece_id=T4CI3KKNMFPT3ZIODHE7BQ4DVHVH2QTSO6OB7QZPL5SNPXQJE6TA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12CuCqHJYbanbuaEMHvtcuQXhvcdahD8eyT2fKDhqtxdSZ6ES4f, address:45.91.116.6:4000): protocol: rpc: dial tcp 45.91.116.6:4000: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.951409864 (316h30m9.502880325s) [noise=true , address=45.91.116.7:4039 , piece_id=HSRY63FXTEPCP2T46V77RVSC24GGRBAYLLU7DTKOLWGVOYHF6YVA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:16oq1Yv7JfHrg4gdHfhwuMj4UsWeQQPayKV6Pj1NY2CgJhuPr2, address:45.91.116.7:4039): protocol: write tcp 188.166.225.170:36904->45.91.116.7:4039: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.951660191 (316h30m9.503130652s) [noise=true , address=77.90.31.11:4572 , piece_id=G4XSIPSWOLSEOGGFA2NLOSDCGTLZPI3P5LDNNLB5RGMU5OQGWXGQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1ZAfYZ4wgpZVvatj6wtHona3hTaPcLxAmXC1DT2uKQyo8nLQQ7, address:77.90.31.11:4572): protocol: rpc: dial tcp 77.90.31.11:4572: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.952019855 (316h30m9.503490316s) [noise=true , address=45.91.116.9:4274 , piece_id=MXWRAXYLBBGLUVTA5GP3JVX2IRM2UI2IC35D7KX3XMFEBXLIDX5A, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12g6jFa38dTz5UdgsQaEuM2MSH1H7cxaysWsVmbHkM2Dd2W2iDb, address:45.91.116.9:4274): protocol: rpc: dial tcp 45.91.116.9:4274: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.952290972 (316h30m9.503761443s) [noise=true , address=77.90.31.15:4720 , piece_id=BDQUF6C5TT3ASE3DIN6SSVJHOZAJK4ZEW33OIYPTGB4LDVRELSUA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12VbsxFqMB3vVphSAJqYpbBzua7Goq45E6ovJSaExkpLtfkcc74, address:77.90.31.15:4720): protocol: rpc: dial tcp 77.90.31.15:4720: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.952544654 (316h30m9.504015105s) [noise=true , address=45.91.116.10:4110 , piece_id=H6FYA6K4N4AQYU3MN5RGYNBMYFXXBJDBTUCN37EAT7WBRNTX2YQA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:123KwusATu1ZKtNNVHZZZBE67ZEnVb8puyfQidBi3fEWREqdf1h, address:45.91.116.10:4110): protocol: rpc: dial tcp 45.91.116.10:4110: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.953548155 (316h30m9.505018617s) [noise=true , address=77.90.31.17:4787 , piece_id=IYIEVH6CARRYCARKOLKWVBICQDNDRCTGQMWF3VJZ6YBC3GO5OWHQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1Lietonb5aVbaZQt6wQ2EoxwJkSEn9oL9Dwyrc9mTyomEKikNt, address:77.90.31.17:4787): protocol: read tcp 188.166.225.170:57908->77.90.31.17:4787: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.953579816 (316h30m9.505050277s) [seg_pos={0 0}, upload=278731]: Done waiting for segment. successful: 5 optimal: 14 errs: <nil>
Nov 22 12:34:11.195719123 (316h30m9.747189584s) [upload=278731]: Done uploading segments...
Nov 22 12:34:11.195932840 (316h30m9.747403301s) [upload=278731]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 5)
Duration: 5.62522055s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
5 out of 60 is failed (threshold=6) [.X..X.XX.............................................X......] duration:
3.52026903s trace:
https://jaeger.dev.storj.io/trace/38596674a32f39c1
FAIL (2025-11-22 13:11:56 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.52030041s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [..X.........................................................] duration:
6.739633333s trace:
https://jaeger.dev.storj.io/trace/570ccd0ce1345814
FAIL (2025-11-22 13:11:40 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.739661648s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [..X.....X...................................................] duration:
5.729867274s trace:
https://jaeger.dev.storj.io/trace/6c800b4c3a84b58c
FAIL (2025-11-22 13:11:30 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 5)
======LOGS======
N5RGYNBMYFXXBJDBTUCN37EAT7WBRNTX2YQA, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.890374814 (316h29m10.441845275s) [noise=true , address=145.79.227.32:4858 , piece_id=K5L3LEUKZWHS36XJHNNY4TYRXCP4DG5UAKCZXPJJTSG26CKUI4HQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.890535981 (316h29m10.442006442s) [noise=true , address=77.90.31.21:4890 , piece_id=DUEFPJUTYMD564Z4III63LZM6L4IL2ZTGM3QDZ6CG7WV5APHBNRA, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.890774064 (316h29m10.442244525s) [noise=true , address=45.91.116.14:4213 , piece_id=HPFISTE2N3B4RBDPAZAGVZJ5WYLCPHTCYH4WMJ56IWEHR3ZBEMOQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.890934851 (316h29m10.442405312s) [noise=true , address=45.91.116.6:4000 , piece_id=T4CI3KKNMFPT3ZIODHE7BQ4DVHVH2QTSO6OB7QZPL5SNPXQJE6TA, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.891167002 (316h29m10.442637463s) [noise=true , address=77.90.31.17:4787 , piece_id=IYIEVH6CARRYCARKOLKWVBICQDNDRCTGQMWF3VJZ6YBC3GO5OWHQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.891369828 (316h29m10.442840289s) [noise=true , address=45.91.116.7:4039 , piece_id=HSRY63FXTEPCP2T46V77RVSC24GGRBAYLLU7DTKOLWGVOYHF6YVA, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.891556875 (316h29m10.443027336s) [noise=true , address=77.90.31.11:4572 , piece_id=G4XSIPSWOLSEOGGFA2NLOSDCGTLZPI3P5LDNNLB5RGMU5OQGWXGQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.891896812 (316h29m10.443367273s) [noise=true , address=77.90.31.19:4863 , piece_id=3PM3Q6X3ADKX2RKIIFDMK57LTJNJHNFK4UYPKFK3ZEQLYQ23EP3A, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.892019235 (316h29m10.443489696s) [noise=true , address=77.90.31.10:4524 , piece_id=Z2IGDHPDU6JAR5CAFQBGYHO3TQ66K4BZOAMXSC7X3RFWLGHOOMTQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.892316521 (316h29m10.443786982s) [noise=true , address=77.90.31.13:4644 , piece_id=UPDZGC3CMJXA3AOJX65VLASU37NLFGPIJLCAPN32TEQB5QHTU7EQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.892682318 (316h29m10.444152779s) [noise=true , address=45.91.116.9:4274 , piece_id=MXWRAXYLBBGLUVTA5GP3JVX2IRM2UI2IC35D7KX3XMFEBXLIDX5A, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:13.353758506 (316h29m11.905228957s) [upload=278731]: Next returned nil segment
Nov 22 12:33:13.353772043 (316h29m11.905242504s) [upload=278731]: Waiting for error group managing segments...
Nov 22 12:33:13.751703718 (316h29m12.303174169s) [noise=true , address=145.79.227.32:4858 , piece_id=K5L3LEUKZWHS36XJHNNY4TYRXCP4DG5UAKCZXPJJTSG26CKUI4HQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil>
Nov 22 12:33:13.753660293 (316h29m12.305130764s) [noise=true , address=145.79.227.26:4674 , piece_id=UP7VIVRDRL6KERJSX3VLWTKGW3ESGCQ7BLYCBXGBB5UNIDE6OHQQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil>
Nov 22 12:33:13.760046123 (316h29m12.311516574s) [noise=true , address=145.79.227.3:4167 , piece_id=4STIVXGU7SOXWAIPG2PXWBLNDD7EYQ56E2DBN2UZ2AZ3TZSXCTIQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil>
Nov 22 12:33:57.373975014 (316h29m55.925445475s) [noise=true , address=77.90.31.13:4644 , piece_id=UPDZGC3CMJXA3AOJX65VLASU37NLFGPIJLCAPN32TEQB5QHTU7EQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil>
Nov 22 12:34:06.122275078 (316h30m4.673745539s) [noise=true , address=45.91.116.13:4181 , piece_id=MQNJEBQFZMHNDKAH2K3JEA7YBK2UK5BGSNCJF3MASF3CKVA5VD6Q, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil>
Nov 22 12:34:10.949171331 (316h30m9.500641792s) [noise=true , address=77.90.31.21:4890 , piece_id=DUEFPJUTYMD564Z4III63LZM6L4IL2ZTGM3QDZ6CG7WV5APHBNRA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1iE3mDnEhGDsMBtmUXC4EwvSrpyHLoybzVXkC9eoFuYEMHeBrd, address:77.90.31.21:4890): protocol: read tcp 188.166.225.170:55900->77.90.31.21:4890: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.949753650 (316h30m9.501224111s) [noise=true , address=45.91.116.12:4237 , piece_id=OG6XREYEGOYVFFMXCGBERRAIEMI2QMUGSBJBYP7GJQZ7FOAI4BIA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1FMjHdNGV42a9QZyMmSPTJWQrA8CdSRvXqrimnt6UGVreCzmgF, address:45.91.116.12:4237): protocol: rpc: dial tcp 45.91.116.12:4237: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.950066235 (316h30m9.501536706s) [noise=true , address=77.90.31.19:4863 , piece_id=3PM3Q6X3ADKX2RKIIFDMK57LTJNJHNFK4UYPKFK3ZEQLYQ23EP3A, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:124x1cbCPJRK8YkgSP6UzHymoLu1E1znnfg5WBiD89kSDLi8JeL, address:77.90.31.19:4863): protocol: write tcp 188.166.225.170:45968->77.90.31.19:4863: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.950246558 (316h30m9.50171702s) [noise=true , address=77.90.31.10:4524 , piece_id=Z2IGDHPDU6JAR5CAFQBGYHO3TQ66K4BZOAMXSC7X3RFWLGHOOMTQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12uN7CznoK4AD8FqFkNUndawdGJ9rmjAe7oNnjJkV844ScY87qe, address:77.90.31.10:4524): protocol: write tcp 188.166.225.170:40266->77.90.31.10:4524: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.950593058 (316h30m9.502063529s) [noise=true , address=77.90.31.14:4690 , piece_id=RJ3KXY3HURT7Q4HZYTOLSAL7Y6VMDGGRIQP6XHE4YB6OAA3PJRAA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.950853535 (316h30m9.502324006s) [noise=true , address=45.91.116.14:4213 , piece_id=HPFISTE2N3B4RBDPAZAGVZJ5WYLCPHTCYH4WMJ56IWEHR3ZBEMOQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12tQRqowH4aDr9h3s1WLs1o7na1ueiafwgUeeHfAB6g7mcE6e1t, address:45.91.116.14:4213): protocol: rpc: dial tcp 45.91.116.14:4213: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.951186438 (316h30m9.502656889s) [noise=true , address=45.91.116.6:4000 , piece_id=T4CI3KKNMFPT3ZIODHE7BQ4DVHVH2QTSO6OB7QZPL5SNPXQJE6TA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12CuCqHJYbanbuaEMHvtcuQXhvcdahD8eyT2fKDhqtxdSZ6ES4f, address:45.91.116.6:4000): protocol: rpc: dial tcp 45.91.116.6:4000: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.951409864 (316h30m9.502880325s) [noise=true , address=45.91.116.7:4039 , piece_id=HSRY63FXTEPCP2T46V77RVSC24GGRBAYLLU7DTKOLWGVOYHF6YVA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:16oq1Yv7JfHrg4gdHfhwuMj4UsWeQQPayKV6Pj1NY2CgJhuPr2, address:45.91.116.7:4039): protocol: write tcp 188.166.225.170:36904->45.91.116.7:4039: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.951660191 (316h30m9.503130652s) [noise=true , address=77.90.31.11:4572 , piece_id=G4XSIPSWOLSEOGGFA2NLOSDCGTLZPI3P5LDNNLB5RGMU5OQGWXGQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1ZAfYZ4wgpZVvatj6wtHona3hTaPcLxAmXC1DT2uKQyo8nLQQ7, address:77.90.31.11:4572): protocol: rpc: dial tcp 77.90.31.11:4572: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.952019855 (316h30m9.503490316s) [noise=true , address=45.91.116.9:4274 , piece_id=MXWRAXYLBBGLUVTA5GP3JVX2IRM2UI2IC35D7KX3XMFEBXLIDX5A, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12g6jFa38dTz5UdgsQaEuM2MSH1H7cxaysWsVmbHkM2Dd2W2iDb, address:45.91.116.9:4274): protocol: rpc: dial tcp 45.91.116.9:4274: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.952290972 (316h30m9.503761443s) [noise=true , address=77.90.31.15:4720 , piece_id=BDQUF6C5TT3ASE3DIN6SSVJHOZAJK4ZEW33OIYPTGB4LDVRELSUA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12VbsxFqMB3vVphSAJqYpbBzua7Goq45E6ovJSaExkpLtfkcc74, address:77.90.31.15:4720): protocol: rpc: dial tcp 77.90.31.15:4720: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.952544654 (316h30m9.504015105s) [noise=true , address=45.91.116.10:4110 , piece_id=H6FYA6K4N4AQYU3MN5RGYNBMYFXXBJDBTUCN37EAT7WBRNTX2YQA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:123KwusATu1ZKtNNVHZZZBE67ZEnVb8puyfQidBi3fEWREqdf1h, address:45.91.116.10:4110): protocol: rpc: dial tcp 45.91.116.10:4110: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.953548155 (316h30m9.505018617s) [noise=true , address=77.90.31.17:4787 , piece_id=IYIEVH6CARRYCARKOLKWVBICQDNDRCTGQMWF3VJZ6YBC3GO5OWHQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1Lietonb5aVbaZQt6wQ2EoxwJkSEn9oL9Dwyrc9mTyomEKikNt, address:77.90.31.17:4787): protocol: read tcp 188.166.225.170:57908->77.90.31.17:4787: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.953579816 (316h30m9.505050277s) [seg_pos={0 0}, upload=278731]: Done waiting for segment. successful: 5 optimal: 14 errs: <nil>
Nov 22 12:34:11.195719123 (316h30m9.747189584s) [upload=278731]: Done uploading segments...
Nov 22 12:34:11.195932840 (316h30m9.747403301s) [upload=278731]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 5)
Duration: 5.729904416s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
5 out of 60 is failed (threshold=6) [..X..X.XX.............................................X.....] duration:
3.470944927s trace:
https://jaeger.dev.storj.io/trace/38596674a32f39c1
FAIL (2025-11-22 13:11:28 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.470965325s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..X..X......................................................] duration:
14.702714393s trace:
https://jaeger.dev.storj.io/trace/92df32505639913
FAIL (2025-11-22 13:11:20 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 14.702743978s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [...X.....X..................................................] duration:
5.778457379s trace:
https://jaeger.dev.storj.io/trace/6c800b4c3a84b58c
FAIL (2025-11-22 13:11:07 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 5)
======LOGS======
N5RGYNBMYFXXBJDBTUCN37EAT7WBRNTX2YQA, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.890374814 (316h29m10.441845275s) [noise=true , address=145.79.227.32:4858 , piece_id=K5L3LEUKZWHS36XJHNNY4TYRXCP4DG5UAKCZXPJJTSG26CKUI4HQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.890535981 (316h29m10.442006442s) [noise=true , address=77.90.31.21:4890 , piece_id=DUEFPJUTYMD564Z4III63LZM6L4IL2ZTGM3QDZ6CG7WV5APHBNRA, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.890774064 (316h29m10.442244525s) [noise=true , address=45.91.116.14:4213 , piece_id=HPFISTE2N3B4RBDPAZAGVZJ5WYLCPHTCYH4WMJ56IWEHR3ZBEMOQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.890934851 (316h29m10.442405312s) [noise=true , address=45.91.116.6:4000 , piece_id=T4CI3KKNMFPT3ZIODHE7BQ4DVHVH2QTSO6OB7QZPL5SNPXQJE6TA, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.891167002 (316h29m10.442637463s) [noise=true , address=77.90.31.17:4787 , piece_id=IYIEVH6CARRYCARKOLKWVBICQDNDRCTGQMWF3VJZ6YBC3GO5OWHQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.891369828 (316h29m10.442840289s) [noise=true , address=45.91.116.7:4039 , piece_id=HSRY63FXTEPCP2T46V77RVSC24GGRBAYLLU7DTKOLWGVOYHF6YVA, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.891556875 (316h29m10.443027336s) [noise=true , address=77.90.31.11:4572 , piece_id=G4XSIPSWOLSEOGGFA2NLOSDCGTLZPI3P5LDNNLB5RGMU5OQGWXGQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.891896812 (316h29m10.443367273s) [noise=true , address=77.90.31.19:4863 , piece_id=3PM3Q6X3ADKX2RKIIFDMK57LTJNJHNFK4UYPKFK3ZEQLYQ23EP3A, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.892019235 (316h29m10.443489696s) [noise=true , address=77.90.31.10:4524 , piece_id=Z2IGDHPDU6JAR5CAFQBGYHO3TQ66K4BZOAMXSC7X3RFWLGHOOMTQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.892316521 (316h29m10.443786982s) [noise=true , address=77.90.31.13:4644 , piece_id=UPDZGC3CMJXA3AOJX65VLASU37NLFGPIJLCAPN32TEQB5QHTU7EQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.892682318 (316h29m10.444152779s) [noise=true , address=45.91.116.9:4274 , piece_id=MXWRAXYLBBGLUVTA5GP3JVX2IRM2UI2IC35D7KX3XMFEBXLIDX5A, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:13.353758506 (316h29m11.905228957s) [upload=278731]: Next returned nil segment
Nov 22 12:33:13.353772043 (316h29m11.905242504s) [upload=278731]: Waiting for error group managing segments...
Nov 22 12:33:13.751703718 (316h29m12.303174169s) [noise=true , address=145.79.227.32:4858 , piece_id=K5L3LEUKZWHS36XJHNNY4TYRXCP4DG5UAKCZXPJJTSG26CKUI4HQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil>
Nov 22 12:33:13.753660293 (316h29m12.305130764s) [noise=true , address=145.79.227.26:4674 , piece_id=UP7VIVRDRL6KERJSX3VLWTKGW3ESGCQ7BLYCBXGBB5UNIDE6OHQQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil>
Nov 22 12:33:13.760046123 (316h29m12.311516574s) [noise=true , address=145.79.227.3:4167 , piece_id=4STIVXGU7SOXWAIPG2PXWBLNDD7EYQ56E2DBN2UZ2AZ3TZSXCTIQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil>
Nov 22 12:33:57.373975014 (316h29m55.925445475s) [noise=true , address=77.90.31.13:4644 , piece_id=UPDZGC3CMJXA3AOJX65VLASU37NLFGPIJLCAPN32TEQB5QHTU7EQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil>
Nov 22 12:34:06.122275078 (316h30m4.673745539s) [noise=true , address=45.91.116.13:4181 , piece_id=MQNJEBQFZMHNDKAH2K3JEA7YBK2UK5BGSNCJF3MASF3CKVA5VD6Q, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil>
Nov 22 12:34:10.949171331 (316h30m9.500641792s) [noise=true , address=77.90.31.21:4890 , piece_id=DUEFPJUTYMD564Z4III63LZM6L4IL2ZTGM3QDZ6CG7WV5APHBNRA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1iE3mDnEhGDsMBtmUXC4EwvSrpyHLoybzVXkC9eoFuYEMHeBrd, address:77.90.31.21:4890): protocol: read tcp 188.166.225.170:55900->77.90.31.21:4890: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.949753650 (316h30m9.501224111s) [noise=true , address=45.91.116.12:4237 , piece_id=OG6XREYEGOYVFFMXCGBERRAIEMI2QMUGSBJBYP7GJQZ7FOAI4BIA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1FMjHdNGV42a9QZyMmSPTJWQrA8CdSRvXqrimnt6UGVreCzmgF, address:45.91.116.12:4237): protocol: rpc: dial tcp 45.91.116.12:4237: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.950066235 (316h30m9.501536706s) [noise=true , address=77.90.31.19:4863 , piece_id=3PM3Q6X3ADKX2RKIIFDMK57LTJNJHNFK4UYPKFK3ZEQLYQ23EP3A, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:124x1cbCPJRK8YkgSP6UzHymoLu1E1znnfg5WBiD89kSDLi8JeL, address:77.90.31.19:4863): protocol: write tcp 188.166.225.170:45968->77.90.31.19:4863: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.950246558 (316h30m9.50171702s) [noise=true , address=77.90.31.10:4524 , piece_id=Z2IGDHPDU6JAR5CAFQBGYHO3TQ66K4BZOAMXSC7X3RFWLGHOOMTQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12uN7CznoK4AD8FqFkNUndawdGJ9rmjAe7oNnjJkV844ScY87qe, address:77.90.31.10:4524): protocol: write tcp 188.166.225.170:40266->77.90.31.10:4524: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.950593058 (316h30m9.502063529s) [noise=true , address=77.90.31.14:4690 , piece_id=RJ3KXY3HURT7Q4HZYTOLSAL7Y6VMDGGRIQP6XHE4YB6OAA3PJRAA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.950853535 (316h30m9.502324006s) [noise=true , address=45.91.116.14:4213 , piece_id=HPFISTE2N3B4RBDPAZAGVZJ5WYLCPHTCYH4WMJ56IWEHR3ZBEMOQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12tQRqowH4aDr9h3s1WLs1o7na1ueiafwgUeeHfAB6g7mcE6e1t, address:45.91.116.14:4213): protocol: rpc: dial tcp 45.91.116.14:4213: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.951186438 (316h30m9.502656889s) [noise=true , address=45.91.116.6:4000 , piece_id=T4CI3KKNMFPT3ZIODHE7BQ4DVHVH2QTSO6OB7QZPL5SNPXQJE6TA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12CuCqHJYbanbuaEMHvtcuQXhvcdahD8eyT2fKDhqtxdSZ6ES4f, address:45.91.116.6:4000): protocol: rpc: dial tcp 45.91.116.6:4000: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.951409864 (316h30m9.502880325s) [noise=true , address=45.91.116.7:4039 , piece_id=HSRY63FXTEPCP2T46V77RVSC24GGRBAYLLU7DTKOLWGVOYHF6YVA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:16oq1Yv7JfHrg4gdHfhwuMj4UsWeQQPayKV6Pj1NY2CgJhuPr2, address:45.91.116.7:4039): protocol: write tcp 188.166.225.170:36904->45.91.116.7:4039: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.951660191 (316h30m9.503130652s) [noise=true , address=77.90.31.11:4572 , piece_id=G4XSIPSWOLSEOGGFA2NLOSDCGTLZPI3P5LDNNLB5RGMU5OQGWXGQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1ZAfYZ4wgpZVvatj6wtHona3hTaPcLxAmXC1DT2uKQyo8nLQQ7, address:77.90.31.11:4572): protocol: rpc: dial tcp 77.90.31.11:4572: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.952019855 (316h30m9.503490316s) [noise=true , address=45.91.116.9:4274 , piece_id=MXWRAXYLBBGLUVTA5GP3JVX2IRM2UI2IC35D7KX3XMFEBXLIDX5A, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12g6jFa38dTz5UdgsQaEuM2MSH1H7cxaysWsVmbHkM2Dd2W2iDb, address:45.91.116.9:4274): protocol: rpc: dial tcp 45.91.116.9:4274: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.952290972 (316h30m9.503761443s) [noise=true , address=77.90.31.15:4720 , piece_id=BDQUF6C5TT3ASE3DIN6SSVJHOZAJK4ZEW33OIYPTGB4LDVRELSUA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12VbsxFqMB3vVphSAJqYpbBzua7Goq45E6ovJSaExkpLtfkcc74, address:77.90.31.15:4720): protocol: rpc: dial tcp 77.90.31.15:4720: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.952544654 (316h30m9.504015105s) [noise=true , address=45.91.116.10:4110 , piece_id=H6FYA6K4N4AQYU3MN5RGYNBMYFXXBJDBTUCN37EAT7WBRNTX2YQA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:123KwusATu1ZKtNNVHZZZBE67ZEnVb8puyfQidBi3fEWREqdf1h, address:45.91.116.10:4110): protocol: rpc: dial tcp 45.91.116.10:4110: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.953548155 (316h30m9.505018617s) [noise=true , address=77.90.31.17:4787 , piece_id=IYIEVH6CARRYCARKOLKWVBICQDNDRCTGQMWF3VJZ6YBC3GO5OWHQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1Lietonb5aVbaZQt6wQ2EoxwJkSEn9oL9Dwyrc9mTyomEKikNt, address:77.90.31.17:4787): protocol: read tcp 188.166.225.170:57908->77.90.31.17:4787: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.953579816 (316h30m9.505050277s) [seg_pos={0 0}, upload=278731]: Done waiting for segment. successful: 5 optimal: 14 errs: <nil>
Nov 22 12:34:11.195719123 (316h30m9.747189584s) [upload=278731]: Done uploading segments...
Nov 22 12:34:11.195932840 (316h30m9.747403301s) [upload=278731]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 5)
Duration: 5.778511422s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
5 out of 60 is failed (threshold=6) [...X..X.XX.............................................X....] duration:
4.306865133s trace:
https://jaeger.dev.storj.io/trace/38596674a32f39c1
FAIL (2025-11-22 13:11:02 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.306894579s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [...X........................................................] duration:
5.618797287s trace:
https://jaeger.dev.storj.io/trace/570ccd0ce1345814
FAIL (2025-11-22 13:10:57 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.618821133s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...X........................................................] duration:
5.733671017s trace:
https://jaeger.dev.storj.io/trace/5ae3d2bf80495000
FAIL (2025-11-22 13:10:55 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Nov 22 12:31:07.276308940 (316h27m5.827779401s) [upload=278701]: Starting upload
Nov 22 12:31:07.276483523 (316h27m5.827953994s) [upload=278701]: Done starting upload
Nov 22 12:31:07.276562243 (316h27m5.828032704s) [upload=278701]: Uploading segments...
Nov 22 12:31:07.276950593 (316h27m5.828421054s) [upload=278701]: Got next segment. Inline: false
Nov 22 12:31:07.277027639 (316h27m5.8284981s) [upload=278701]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:31:07.277034181 (316h27m5.828504643s) [upload=278701]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:31:07.520212027 (316h27m6.071682488s) [seg_pos={0 0}, upload=278701]: Begin upload segment...
Nov 22 12:31:07.520269256 (316h27m6.071739717s) [seg_pos={0 0}, upload=278701]: Done begin upload segment.
Nov 22 12:31:07.520308992 (316h27m6.071779453s) [noise=true , address=145.79.227.27:4689 , piece_id=76S65KEZ7I2E76RR2JXOZ3ONGXCWSJA5XB7XQVFEEKGGNC34Z4EQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.520426746 (316h27m6.071897207s) [noise=true , address=145.79.227.21:4474 , piece_id=PCRWYQVCAZCPOILAEDHJGDBFKX4GEJ6JXCF7FUS7OVFRUTDPJI4Q, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.521175282 (316h27m6.072645743s) [noise=true , address=145.79.227.17:4972 , piece_id=UZK7LCIOBGGDNJZUVDBOLBJHGOIA4SUT2O3PD67PYQD3J2DGBJEA, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.521527573 (316h27m6.072998044s) [noise=true , address=145.79.227.23:4546 , piece_id=73UPPN2SYWNWEVWHCF4BRGKNO5GCRGOV3S5JZPX4MW75EJZRQOWQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.521694671 (316h27m6.073165132s) [noise=true , address=45.91.116.10:4082 , piece_id=GBDYXAWONOHQGBGM3NCRQYDSPFOAIHRJHBB2W6ZGJLMZUJLV65XA, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.522219410 (316h27m6.073689871s) [noise=true , address=145.79.227.20:4437 , piece_id=CZ32MB4AEZZ6YKCNWKRYALVAK7MT3KNXDSZCQTPUV2Q53NX6Y3BQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.522261240 (316h27m6.073731691s) [noise=true , address=145.79.227.16:4909 , piece_id=ZTL4ER5YJWSDGO5F5KLRUL5WRTA6WKVOJOERIBTRIIVEWXUY5Z2Q, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.522719151 (316h27m6.074189612s) [noise=true , address=145.79.227.26:4654 , piece_id=EM4RRGMPPNVR7VTXP3BED43FJFIUDZYVT4S5OUK2TRXZHYNMQ64Q, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.523017269 (316h27m6.07448772s) [noise=true , address=77.90.31.13:4627 , piece_id=Q7U5BFXHYQBBJQCU4EQGAR75WKL5QMBJ3LO34UYGO7QHLMDJBKWQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.523216949 (316h27m6.07468741s) [noise=true , address=145.79.227.25:4618 , piece_id=CV6M3CTPQR2MPQZFU6W34MFQERT6MMN3RRF2N3W35FGI464OSUUA, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.523744804 (316h27m6.075215265s) [noise=true , address=145.79.227.6:4258 , piece_id=B5D4TCVJORSAWDAU4DQ5VAV4CCRF5WUI24DUVUORAEKLAWGI7ABQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.523828153 (316h27m6.075298614s) [noise=true , address=145.79.227.24:4583 , piece_id=MLW4X34EWXKPOCFVI4VIQSL6RMUOJ5HRYHQ4HY35JIFT4ZFJ6F3Q, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.524255396 (316h27m6.075725858s) [noise=true , address=45.91.116.7:4009 , piece_id=FXQGF3UQN5CPFBS2HWD7BAXR4QIOF42MMK3VJQ7AK4THYJWOFFLQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.524662051 (316h27m6.076132512s) [noise=true , address=145.79.227.15:4874 , piece_id=TYAXIF3ZJQQSFGFANYF56DG4O5WBBXK4M7OT67E7ARYNZ663HESA, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.524746221 (316h27m6.076216682s) [noise=true , address=145.79.227.28:4726 , piece_id=3LBJV2B4CVXEMHCOPRW7DR5QBKPS6DFMPTHQSVNR4XRTZZVIGBQA, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.525227157 (316h27m6.076697608s) [noise=true , address=77.90.31.21:4879 , piece_id=GHTH55OGGXBAZZ5FGPCDYVHDYEETM7DNVTPSTTFFFRZMKZZ3XRXQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:09.134610631 (316h27m7.686081093s) [upload=278701]: Next returned nil segment
Nov 22 12:31:09.134628024 (316h27m7.686098495s) [upload=278701]: Waiting for error group managing segments...
Nov 22 12:31:09.502125778 (316h27m8.053596239s) [noise=true , address=145.79.227.16:4909 , piece_id=ZTL4ER5YJWSDGO5F5KLRUL5WRTA6WKVOJOERIBTRIIVEWXUY5Z2Q, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.503083742 (316h27m8.054554193s) [noise=true , address=145.79.227.20:4437 , piece_id=CZ32MB4AEZZ6YKCNWKRYALVAK7MT3KNXDSZCQTPUV2Q53NX6Y3BQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.517732716 (316h27m8.069203177s) [noise=true , address=145.79.227.25:4618 , piece_id=CV6M3CTPQR2MPQZFU6W34MFQERT6MMN3RRF2N3W35FGI464OSUUA, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.518029582 (316h27m8.069500043s) [noise=true , address=145.79.227.28:4726 , piece_id=3LBJV2B4CVXEMHCOPRW7DR5QBKPS6DFMPTHQSVNR4XRTZZVIGBQA, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.519935051 (316h27m8.071405512s) [noise=true , address=145.79.227.26:4654 , piece_id=EM4RRGMPPNVR7VTXP3BED43FJFIUDZYVT4S5OUK2TRXZHYNMQ64Q, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.520607661 (316h27m8.072078122s) [noise=true , address=145.79.227.27:4689 , piece_id=76S65KEZ7I2E76RR2JXOZ3ONGXCWSJA5XB7XQVFEEKGGNC34Z4EQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.521187736 (316h27m8.072658197s) [noise=true , address=145.79.227.23:4546 , piece_id=73UPPN2SYWNWEVWHCF4BRGKNO5GCRGOV3S5JZPX4MW75EJZRQOWQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.532974760 (316h27m8.084445221s) [noise=true , address=145.79.227.24:4583 , piece_id=MLW4X34EWXKPOCFVI4VIQSL6RMUOJ5HRYHQ4HY35JIFT4ZFJ6F3Q, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.533688800 (316h27m8.085159271s) [noise=true , address=145.79.227.6:4258 , piece_id=B5D4TCVJORSAWDAU4DQ5VAV4CCRF5WUI24DUVUORAEKLAWGI7ABQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.534460109 (316h27m8.08593057s) [noise=true , address=145.79.227.15:4874 , piece_id=TYAXIF3ZJQQSFGFANYF56DG4O5WBBXK4M7OT67E7ARYNZ663HESA, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.542955375 (316h27m8.094425826s) [noise=true , address=145.79.227.17:4972 , piece_id=UZK7LCIOBGGDNJZUVDBOLBJHGOIA4SUT2O3PD67PYQD3J2DGBJEA, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.563098982 (316h27m8.114569443s) [noise=true , address=145.79.227.21:4474 , piece_id=PCRWYQVCAZCPOILAEDHJGDBFKX4GEJ6JXCF7FUS7OVFRUTDPJI4Q, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:36.585475346 (316h27m35.136945807s) [noise=true , address=45.91.116.10:4082 , piece_id=GBDYXAWONOHQGBGM3NCRQYDSPFOAIHRJHBB2W6ZGJLMZUJLV65XA, seg_pos={0 0}, upload=278701]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: read tcp 188.166.225.170:54326->45.91.116.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:36.585499923 (316h27m35.136970384s) [noise=true , address=77.90.31.21:4879 , piece_id=GHTH55OGGXBAZZ5FGPCDYVHDYEETM7DNVTPSTTFFFRZMKZZ3XRXQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: rpc: dial tcp 77.90.31.21:4879: i/o timeout; context deadline exceeded
Nov 22 12:31:36.585749648 (316h27m35.137220119s) [noise=true , address=77.90.31.13:4627 , piece_id=Q7U5BFXHYQBBJQCU4EQGAR75WKL5QMBJ3LO34UYGO7QHLMDJBKWQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:36.585848086 (316h27m35.137318547s) [noise=true , address=45.91.116.7:4009 , piece_id=FXQGF3UQN5CPFBS2HWD7BAXR4QIOF42MMK3VJQ7AK4THYJWOFFLQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:36.585900936 (316h27m35.137371397s) [seg_pos={0 0}, upload=278701]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Nov 22 12:31:36.828881025 (316h27m35.380351486s) [upload=278701]: Done uploading segments...
Nov 22 12:31:36.828915801 (316h27m35.380386262s) [upload=278701]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 5.733713809s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [....X.....X.................................................] duration:
5.718244855s trace:
https://jaeger.dev.storj.io/trace/6c800b4c3a84b58c
FAIL (2025-11-22 13:10:44 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 5)
======LOGS======
N5RGYNBMYFXXBJDBTUCN37EAT7WBRNTX2YQA, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.890374814 (316h29m10.441845275s) [noise=true , address=145.79.227.32:4858 , piece_id=K5L3LEUKZWHS36XJHNNY4TYRXCP4DG5UAKCZXPJJTSG26CKUI4HQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.890535981 (316h29m10.442006442s) [noise=true , address=77.90.31.21:4890 , piece_id=DUEFPJUTYMD564Z4III63LZM6L4IL2ZTGM3QDZ6CG7WV5APHBNRA, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.890774064 (316h29m10.442244525s) [noise=true , address=45.91.116.14:4213 , piece_id=HPFISTE2N3B4RBDPAZAGVZJ5WYLCPHTCYH4WMJ56IWEHR3ZBEMOQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.890934851 (316h29m10.442405312s) [noise=true , address=45.91.116.6:4000 , piece_id=T4CI3KKNMFPT3ZIODHE7BQ4DVHVH2QTSO6OB7QZPL5SNPXQJE6TA, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.891167002 (316h29m10.442637463s) [noise=true , address=77.90.31.17:4787 , piece_id=IYIEVH6CARRYCARKOLKWVBICQDNDRCTGQMWF3VJZ6YBC3GO5OWHQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.891369828 (316h29m10.442840289s) [noise=true , address=45.91.116.7:4039 , piece_id=HSRY63FXTEPCP2T46V77RVSC24GGRBAYLLU7DTKOLWGVOYHF6YVA, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.891556875 (316h29m10.443027336s) [noise=true , address=77.90.31.11:4572 , piece_id=G4XSIPSWOLSEOGGFA2NLOSDCGTLZPI3P5LDNNLB5RGMU5OQGWXGQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.891896812 (316h29m10.443367273s) [noise=true , address=77.90.31.19:4863 , piece_id=3PM3Q6X3ADKX2RKIIFDMK57LTJNJHNFK4UYPKFK3ZEQLYQ23EP3A, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.892019235 (316h29m10.443489696s) [noise=true , address=77.90.31.10:4524 , piece_id=Z2IGDHPDU6JAR5CAFQBGYHO3TQ66K4BZOAMXSC7X3RFWLGHOOMTQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.892316521 (316h29m10.443786982s) [noise=true , address=77.90.31.13:4644 , piece_id=UPDZGC3CMJXA3AOJX65VLASU37NLFGPIJLCAPN32TEQB5QHTU7EQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.892682318 (316h29m10.444152779s) [noise=true , address=45.91.116.9:4274 , piece_id=MXWRAXYLBBGLUVTA5GP3JVX2IRM2UI2IC35D7KX3XMFEBXLIDX5A, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:13.353758506 (316h29m11.905228957s) [upload=278731]: Next returned nil segment
Nov 22 12:33:13.353772043 (316h29m11.905242504s) [upload=278731]: Waiting for error group managing segments...
Nov 22 12:33:13.751703718 (316h29m12.303174169s) [noise=true , address=145.79.227.32:4858 , piece_id=K5L3LEUKZWHS36XJHNNY4TYRXCP4DG5UAKCZXPJJTSG26CKUI4HQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil>
Nov 22 12:33:13.753660293 (316h29m12.305130764s) [noise=true , address=145.79.227.26:4674 , piece_id=UP7VIVRDRL6KERJSX3VLWTKGW3ESGCQ7BLYCBXGBB5UNIDE6OHQQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil>
Nov 22 12:33:13.760046123 (316h29m12.311516574s) [noise=true , address=145.79.227.3:4167 , piece_id=4STIVXGU7SOXWAIPG2PXWBLNDD7EYQ56E2DBN2UZ2AZ3TZSXCTIQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil>
Nov 22 12:33:57.373975014 (316h29m55.925445475s) [noise=true , address=77.90.31.13:4644 , piece_id=UPDZGC3CMJXA3AOJX65VLASU37NLFGPIJLCAPN32TEQB5QHTU7EQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil>
Nov 22 12:34:06.122275078 (316h30m4.673745539s) [noise=true , address=45.91.116.13:4181 , piece_id=MQNJEBQFZMHNDKAH2K3JEA7YBK2UK5BGSNCJF3MASF3CKVA5VD6Q, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil>
Nov 22 12:34:10.949171331 (316h30m9.500641792s) [noise=true , address=77.90.31.21:4890 , piece_id=DUEFPJUTYMD564Z4III63LZM6L4IL2ZTGM3QDZ6CG7WV5APHBNRA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1iE3mDnEhGDsMBtmUXC4EwvSrpyHLoybzVXkC9eoFuYEMHeBrd, address:77.90.31.21:4890): protocol: read tcp 188.166.225.170:55900->77.90.31.21:4890: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.949753650 (316h30m9.501224111s) [noise=true , address=45.91.116.12:4237 , piece_id=OG6XREYEGOYVFFMXCGBERRAIEMI2QMUGSBJBYP7GJQZ7FOAI4BIA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1FMjHdNGV42a9QZyMmSPTJWQrA8CdSRvXqrimnt6UGVreCzmgF, address:45.91.116.12:4237): protocol: rpc: dial tcp 45.91.116.12:4237: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.950066235 (316h30m9.501536706s) [noise=true , address=77.90.31.19:4863 , piece_id=3PM3Q6X3ADKX2RKIIFDMK57LTJNJHNFK4UYPKFK3ZEQLYQ23EP3A, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:124x1cbCPJRK8YkgSP6UzHymoLu1E1znnfg5WBiD89kSDLi8JeL, address:77.90.31.19:4863): protocol: write tcp 188.166.225.170:45968->77.90.31.19:4863: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.950246558 (316h30m9.50171702s) [noise=true , address=77.90.31.10:4524 , piece_id=Z2IGDHPDU6JAR5CAFQBGYHO3TQ66K4BZOAMXSC7X3RFWLGHOOMTQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12uN7CznoK4AD8FqFkNUndawdGJ9rmjAe7oNnjJkV844ScY87qe, address:77.90.31.10:4524): protocol: write tcp 188.166.225.170:40266->77.90.31.10:4524: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.950593058 (316h30m9.502063529s) [noise=true , address=77.90.31.14:4690 , piece_id=RJ3KXY3HURT7Q4HZYTOLSAL7Y6VMDGGRIQP6XHE4YB6OAA3PJRAA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.950853535 (316h30m9.502324006s) [noise=true , address=45.91.116.14:4213 , piece_id=HPFISTE2N3B4RBDPAZAGVZJ5WYLCPHTCYH4WMJ56IWEHR3ZBEMOQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12tQRqowH4aDr9h3s1WLs1o7na1ueiafwgUeeHfAB6g7mcE6e1t, address:45.91.116.14:4213): protocol: rpc: dial tcp 45.91.116.14:4213: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.951186438 (316h30m9.502656889s) [noise=true , address=45.91.116.6:4000 , piece_id=T4CI3KKNMFPT3ZIODHE7BQ4DVHVH2QTSO6OB7QZPL5SNPXQJE6TA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12CuCqHJYbanbuaEMHvtcuQXhvcdahD8eyT2fKDhqtxdSZ6ES4f, address:45.91.116.6:4000): protocol: rpc: dial tcp 45.91.116.6:4000: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.951409864 (316h30m9.502880325s) [noise=true , address=45.91.116.7:4039 , piece_id=HSRY63FXTEPCP2T46V77RVSC24GGRBAYLLU7DTKOLWGVOYHF6YVA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:16oq1Yv7JfHrg4gdHfhwuMj4UsWeQQPayKV6Pj1NY2CgJhuPr2, address:45.91.116.7:4039): protocol: write tcp 188.166.225.170:36904->45.91.116.7:4039: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.951660191 (316h30m9.503130652s) [noise=true , address=77.90.31.11:4572 , piece_id=G4XSIPSWOLSEOGGFA2NLOSDCGTLZPI3P5LDNNLB5RGMU5OQGWXGQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1ZAfYZ4wgpZVvatj6wtHona3hTaPcLxAmXC1DT2uKQyo8nLQQ7, address:77.90.31.11:4572): protocol: rpc: dial tcp 77.90.31.11:4572: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.952019855 (316h30m9.503490316s) [noise=true , address=45.91.116.9:4274 , piece_id=MXWRAXYLBBGLUVTA5GP3JVX2IRM2UI2IC35D7KX3XMFEBXLIDX5A, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12g6jFa38dTz5UdgsQaEuM2MSH1H7cxaysWsVmbHkM2Dd2W2iDb, address:45.91.116.9:4274): protocol: rpc: dial tcp 45.91.116.9:4274: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.952290972 (316h30m9.503761443s) [noise=true , address=77.90.31.15:4720 , piece_id=BDQUF6C5TT3ASE3DIN6SSVJHOZAJK4ZEW33OIYPTGB4LDVRELSUA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12VbsxFqMB3vVphSAJqYpbBzua7Goq45E6ovJSaExkpLtfkcc74, address:77.90.31.15:4720): protocol: rpc: dial tcp 77.90.31.15:4720: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.952544654 (316h30m9.504015105s) [noise=true , address=45.91.116.10:4110 , piece_id=H6FYA6K4N4AQYU3MN5RGYNBMYFXXBJDBTUCN37EAT7WBRNTX2YQA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:123KwusATu1ZKtNNVHZZZBE67ZEnVb8puyfQidBi3fEWREqdf1h, address:45.91.116.10:4110): protocol: rpc: dial tcp 45.91.116.10:4110: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.953548155 (316h30m9.505018617s) [noise=true , address=77.90.31.17:4787 , piece_id=IYIEVH6CARRYCARKOLKWVBICQDNDRCTGQMWF3VJZ6YBC3GO5OWHQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1Lietonb5aVbaZQt6wQ2EoxwJkSEn9oL9Dwyrc9mTyomEKikNt, address:77.90.31.17:4787): protocol: read tcp 188.166.225.170:57908->77.90.31.17:4787: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.953579816 (316h30m9.505050277s) [seg_pos={0 0}, upload=278731]: Done waiting for segment. successful: 5 optimal: 14 errs: <nil>
Nov 22 12:34:11.195719123 (316h30m9.747189584s) [upload=278731]: Done uploading segments...
Nov 22 12:34:11.195932840 (316h30m9.747403301s) [upload=278731]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 5)
Duration: 5.718271434s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [X..X..X.....................................................] duration:
15.191760682s trace:
https://jaeger.dev.storj.io/trace/92df32505639913
FAIL (2025-11-22 13:10:44 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 15.191788434s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
5 out of 60 is failed (threshold=6) [....X..X.XX.............................................X...] duration:
3.426622923s trace:
https://jaeger.dev.storj.io/trace/38596674a32f39c1
FAIL (2025-11-22 13:10:35 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.426654052s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [X...X.......................................................] duration:
5.885861184s trace:
https://jaeger.dev.storj.io/trace/570ccd0ce1345814
FAIL (2025-11-22 13:10:24 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.885891883s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.X..X..X....................................................] duration:
17.605840163s trace:
https://jaeger.dev.storj.io/trace/92df32505639913
FAIL (2025-11-22 13:10:01 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 17.605872064s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
5 out of 60 is failed (threshold=6) [.....X..X.XX.............................................X..] duration:
3.445742688s trace:
https://jaeger.dev.storj.io/trace/38596674a32f39c1
FAIL (2025-11-22 13:10:01 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.445771885s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [....X.......................................................] duration:
5.764807777s trace:
https://jaeger.dev.storj.io/trace/5ae3d2bf80495000
FAIL (2025-11-22 13:09:43 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Nov 22 12:31:07.276308940 (316h27m5.827779401s) [upload=278701]: Starting upload
Nov 22 12:31:07.276483523 (316h27m5.827953994s) [upload=278701]: Done starting upload
Nov 22 12:31:07.276562243 (316h27m5.828032704s) [upload=278701]: Uploading segments...
Nov 22 12:31:07.276950593 (316h27m5.828421054s) [upload=278701]: Got next segment. Inline: false
Nov 22 12:31:07.277027639 (316h27m5.8284981s) [upload=278701]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:31:07.277034181 (316h27m5.828504643s) [upload=278701]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:31:07.520212027 (316h27m6.071682488s) [seg_pos={0 0}, upload=278701]: Begin upload segment...
Nov 22 12:31:07.520269256 (316h27m6.071739717s) [seg_pos={0 0}, upload=278701]: Done begin upload segment.
Nov 22 12:31:07.520308992 (316h27m6.071779453s) [noise=true , address=145.79.227.27:4689 , piece_id=76S65KEZ7I2E76RR2JXOZ3ONGXCWSJA5XB7XQVFEEKGGNC34Z4EQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.520426746 (316h27m6.071897207s) [noise=true , address=145.79.227.21:4474 , piece_id=PCRWYQVCAZCPOILAEDHJGDBFKX4GEJ6JXCF7FUS7OVFRUTDPJI4Q, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.521175282 (316h27m6.072645743s) [noise=true , address=145.79.227.17:4972 , piece_id=UZK7LCIOBGGDNJZUVDBOLBJHGOIA4SUT2O3PD67PYQD3J2DGBJEA, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.521527573 (316h27m6.072998044s) [noise=true , address=145.79.227.23:4546 , piece_id=73UPPN2SYWNWEVWHCF4BRGKNO5GCRGOV3S5JZPX4MW75EJZRQOWQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.521694671 (316h27m6.073165132s) [noise=true , address=45.91.116.10:4082 , piece_id=GBDYXAWONOHQGBGM3NCRQYDSPFOAIHRJHBB2W6ZGJLMZUJLV65XA, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.522219410 (316h27m6.073689871s) [noise=true , address=145.79.227.20:4437 , piece_id=CZ32MB4AEZZ6YKCNWKRYALVAK7MT3KNXDSZCQTPUV2Q53NX6Y3BQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.522261240 (316h27m6.073731691s) [noise=true , address=145.79.227.16:4909 , piece_id=ZTL4ER5YJWSDGO5F5KLRUL5WRTA6WKVOJOERIBTRIIVEWXUY5Z2Q, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.522719151 (316h27m6.074189612s) [noise=true , address=145.79.227.26:4654 , piece_id=EM4RRGMPPNVR7VTXP3BED43FJFIUDZYVT4S5OUK2TRXZHYNMQ64Q, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.523017269 (316h27m6.07448772s) [noise=true , address=77.90.31.13:4627 , piece_id=Q7U5BFXHYQBBJQCU4EQGAR75WKL5QMBJ3LO34UYGO7QHLMDJBKWQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.523216949 (316h27m6.07468741s) [noise=true , address=145.79.227.25:4618 , piece_id=CV6M3CTPQR2MPQZFU6W34MFQERT6MMN3RRF2N3W35FGI464OSUUA, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.523744804 (316h27m6.075215265s) [noise=true , address=145.79.227.6:4258 , piece_id=B5D4TCVJORSAWDAU4DQ5VAV4CCRF5WUI24DUVUORAEKLAWGI7ABQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.523828153 (316h27m6.075298614s) [noise=true , address=145.79.227.24:4583 , piece_id=MLW4X34EWXKPOCFVI4VIQSL6RMUOJ5HRYHQ4HY35JIFT4ZFJ6F3Q, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.524255396 (316h27m6.075725858s) [noise=true , address=45.91.116.7:4009 , piece_id=FXQGF3UQN5CPFBS2HWD7BAXR4QIOF42MMK3VJQ7AK4THYJWOFFLQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.524662051 (316h27m6.076132512s) [noise=true , address=145.79.227.15:4874 , piece_id=TYAXIF3ZJQQSFGFANYF56DG4O5WBBXK4M7OT67E7ARYNZ663HESA, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.524746221 (316h27m6.076216682s) [noise=true , address=145.79.227.28:4726 , piece_id=3LBJV2B4CVXEMHCOPRW7DR5QBKPS6DFMPTHQSVNR4XRTZZVIGBQA, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.525227157 (316h27m6.076697608s) [noise=true , address=77.90.31.21:4879 , piece_id=GHTH55OGGXBAZZ5FGPCDYVHDYEETM7DNVTPSTTFFFRZMKZZ3XRXQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:09.134610631 (316h27m7.686081093s) [upload=278701]: Next returned nil segment
Nov 22 12:31:09.134628024 (316h27m7.686098495s) [upload=278701]: Waiting for error group managing segments...
Nov 22 12:31:09.502125778 (316h27m8.053596239s) [noise=true , address=145.79.227.16:4909 , piece_id=ZTL4ER5YJWSDGO5F5KLRUL5WRTA6WKVOJOERIBTRIIVEWXUY5Z2Q, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.503083742 (316h27m8.054554193s) [noise=true , address=145.79.227.20:4437 , piece_id=CZ32MB4AEZZ6YKCNWKRYALVAK7MT3KNXDSZCQTPUV2Q53NX6Y3BQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.517732716 (316h27m8.069203177s) [noise=true , address=145.79.227.25:4618 , piece_id=CV6M3CTPQR2MPQZFU6W34MFQERT6MMN3RRF2N3W35FGI464OSUUA, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.518029582 (316h27m8.069500043s) [noise=true , address=145.79.227.28:4726 , piece_id=3LBJV2B4CVXEMHCOPRW7DR5QBKPS6DFMPTHQSVNR4XRTZZVIGBQA, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.519935051 (316h27m8.071405512s) [noise=true , address=145.79.227.26:4654 , piece_id=EM4RRGMPPNVR7VTXP3BED43FJFIUDZYVT4S5OUK2TRXZHYNMQ64Q, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.520607661 (316h27m8.072078122s) [noise=true , address=145.79.227.27:4689 , piece_id=76S65KEZ7I2E76RR2JXOZ3ONGXCWSJA5XB7XQVFEEKGGNC34Z4EQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.521187736 (316h27m8.072658197s) [noise=true , address=145.79.227.23:4546 , piece_id=73UPPN2SYWNWEVWHCF4BRGKNO5GCRGOV3S5JZPX4MW75EJZRQOWQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.532974760 (316h27m8.084445221s) [noise=true , address=145.79.227.24:4583 , piece_id=MLW4X34EWXKPOCFVI4VIQSL6RMUOJ5HRYHQ4HY35JIFT4ZFJ6F3Q, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.533688800 (316h27m8.085159271s) [noise=true , address=145.79.227.6:4258 , piece_id=B5D4TCVJORSAWDAU4DQ5VAV4CCRF5WUI24DUVUORAEKLAWGI7ABQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.534460109 (316h27m8.08593057s) [noise=true , address=145.79.227.15:4874 , piece_id=TYAXIF3ZJQQSFGFANYF56DG4O5WBBXK4M7OT67E7ARYNZ663HESA, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.542955375 (316h27m8.094425826s) [noise=true , address=145.79.227.17:4972 , piece_id=UZK7LCIOBGGDNJZUVDBOLBJHGOIA4SUT2O3PD67PYQD3J2DGBJEA, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.563098982 (316h27m8.114569443s) [noise=true , address=145.79.227.21:4474 , piece_id=PCRWYQVCAZCPOILAEDHJGDBFKX4GEJ6JXCF7FUS7OVFRUTDPJI4Q, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:36.585475346 (316h27m35.136945807s) [noise=true , address=45.91.116.10:4082 , piece_id=GBDYXAWONOHQGBGM3NCRQYDSPFOAIHRJHBB2W6ZGJLMZUJLV65XA, seg_pos={0 0}, upload=278701]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: read tcp 188.166.225.170:54326->45.91.116.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:36.585499923 (316h27m35.136970384s) [noise=true , address=77.90.31.21:4879 , piece_id=GHTH55OGGXBAZZ5FGPCDYVHDYEETM7DNVTPSTTFFFRZMKZZ3XRXQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: rpc: dial tcp 77.90.31.21:4879: i/o timeout; context deadline exceeded
Nov 22 12:31:36.585749648 (316h27m35.137220119s) [noise=true , address=77.90.31.13:4627 , piece_id=Q7U5BFXHYQBBJQCU4EQGAR75WKL5QMBJ3LO34UYGO7QHLMDJBKWQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:36.585848086 (316h27m35.137318547s) [noise=true , address=45.91.116.7:4009 , piece_id=FXQGF3UQN5CPFBS2HWD7BAXR4QIOF42MMK3VJQ7AK4THYJWOFFLQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:36.585900936 (316h27m35.137371397s) [seg_pos={0 0}, upload=278701]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Nov 22 12:31:36.828881025 (316h27m35.380351486s) [upload=278701]: Done uploading segments...
Nov 22 12:31:36.828915801 (316h27m35.380386262s) [upload=278701]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 5.764861188s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [.X...X......................................................] duration:
5.707639249s trace:
https://jaeger.dev.storj.io/trace/570ccd0ce1345814
FAIL (2025-11-22 13:09:43 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.70774461s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [.....X.....X................................................] duration:
5.672831198s trace:
https://jaeger.dev.storj.io/trace/6c800b4c3a84b58c
FAIL (2025-11-22 13:09:37 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 5)
======LOGS======
N5RGYNBMYFXXBJDBTUCN37EAT7WBRNTX2YQA, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.890374814 (316h29m10.441845275s) [noise=true , address=145.79.227.32:4858 , piece_id=K5L3LEUKZWHS36XJHNNY4TYRXCP4DG5UAKCZXPJJTSG26CKUI4HQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.890535981 (316h29m10.442006442s) [noise=true , address=77.90.31.21:4890 , piece_id=DUEFPJUTYMD564Z4III63LZM6L4IL2ZTGM3QDZ6CG7WV5APHBNRA, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.890774064 (316h29m10.442244525s) [noise=true , address=45.91.116.14:4213 , piece_id=HPFISTE2N3B4RBDPAZAGVZJ5WYLCPHTCYH4WMJ56IWEHR3ZBEMOQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.890934851 (316h29m10.442405312s) [noise=true , address=45.91.116.6:4000 , piece_id=T4CI3KKNMFPT3ZIODHE7BQ4DVHVH2QTSO6OB7QZPL5SNPXQJE6TA, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.891167002 (316h29m10.442637463s) [noise=true , address=77.90.31.17:4787 , piece_id=IYIEVH6CARRYCARKOLKWVBICQDNDRCTGQMWF3VJZ6YBC3GO5OWHQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.891369828 (316h29m10.442840289s) [noise=true , address=45.91.116.7:4039 , piece_id=HSRY63FXTEPCP2T46V77RVSC24GGRBAYLLU7DTKOLWGVOYHF6YVA, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.891556875 (316h29m10.443027336s) [noise=true , address=77.90.31.11:4572 , piece_id=G4XSIPSWOLSEOGGFA2NLOSDCGTLZPI3P5LDNNLB5RGMU5OQGWXGQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.891896812 (316h29m10.443367273s) [noise=true , address=77.90.31.19:4863 , piece_id=3PM3Q6X3ADKX2RKIIFDMK57LTJNJHNFK4UYPKFK3ZEQLYQ23EP3A, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.892019235 (316h29m10.443489696s) [noise=true , address=77.90.31.10:4524 , piece_id=Z2IGDHPDU6JAR5CAFQBGYHO3TQ66K4BZOAMXSC7X3RFWLGHOOMTQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.892316521 (316h29m10.443786982s) [noise=true , address=77.90.31.13:4644 , piece_id=UPDZGC3CMJXA3AOJX65VLASU37NLFGPIJLCAPN32TEQB5QHTU7EQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.892682318 (316h29m10.444152779s) [noise=true , address=45.91.116.9:4274 , piece_id=MXWRAXYLBBGLUVTA5GP3JVX2IRM2UI2IC35D7KX3XMFEBXLIDX5A, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:13.353758506 (316h29m11.905228957s) [upload=278731]: Next returned nil segment
Nov 22 12:33:13.353772043 (316h29m11.905242504s) [upload=278731]: Waiting for error group managing segments...
Nov 22 12:33:13.751703718 (316h29m12.303174169s) [noise=true , address=145.79.227.32:4858 , piece_id=K5L3LEUKZWHS36XJHNNY4TYRXCP4DG5UAKCZXPJJTSG26CKUI4HQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil>
Nov 22 12:33:13.753660293 (316h29m12.305130764s) [noise=true , address=145.79.227.26:4674 , piece_id=UP7VIVRDRL6KERJSX3VLWTKGW3ESGCQ7BLYCBXGBB5UNIDE6OHQQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil>
Nov 22 12:33:13.760046123 (316h29m12.311516574s) [noise=true , address=145.79.227.3:4167 , piece_id=4STIVXGU7SOXWAIPG2PXWBLNDD7EYQ56E2DBN2UZ2AZ3TZSXCTIQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil>
Nov 22 12:33:57.373975014 (316h29m55.925445475s) [noise=true , address=77.90.31.13:4644 , piece_id=UPDZGC3CMJXA3AOJX65VLASU37NLFGPIJLCAPN32TEQB5QHTU7EQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil>
Nov 22 12:34:06.122275078 (316h30m4.673745539s) [noise=true , address=45.91.116.13:4181 , piece_id=MQNJEBQFZMHNDKAH2K3JEA7YBK2UK5BGSNCJF3MASF3CKVA5VD6Q, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil>
Nov 22 12:34:10.949171331 (316h30m9.500641792s) [noise=true , address=77.90.31.21:4890 , piece_id=DUEFPJUTYMD564Z4III63LZM6L4IL2ZTGM3QDZ6CG7WV5APHBNRA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1iE3mDnEhGDsMBtmUXC4EwvSrpyHLoybzVXkC9eoFuYEMHeBrd, address:77.90.31.21:4890): protocol: read tcp 188.166.225.170:55900->77.90.31.21:4890: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.949753650 (316h30m9.501224111s) [noise=true , address=45.91.116.12:4237 , piece_id=OG6XREYEGOYVFFMXCGBERRAIEMI2QMUGSBJBYP7GJQZ7FOAI4BIA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1FMjHdNGV42a9QZyMmSPTJWQrA8CdSRvXqrimnt6UGVreCzmgF, address:45.91.116.12:4237): protocol: rpc: dial tcp 45.91.116.12:4237: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.950066235 (316h30m9.501536706s) [noise=true , address=77.90.31.19:4863 , piece_id=3PM3Q6X3ADKX2RKIIFDMK57LTJNJHNFK4UYPKFK3ZEQLYQ23EP3A, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:124x1cbCPJRK8YkgSP6UzHymoLu1E1znnfg5WBiD89kSDLi8JeL, address:77.90.31.19:4863): protocol: write tcp 188.166.225.170:45968->77.90.31.19:4863: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.950246558 (316h30m9.50171702s) [noise=true , address=77.90.31.10:4524 , piece_id=Z2IGDHPDU6JAR5CAFQBGYHO3TQ66K4BZOAMXSC7X3RFWLGHOOMTQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12uN7CznoK4AD8FqFkNUndawdGJ9rmjAe7oNnjJkV844ScY87qe, address:77.90.31.10:4524): protocol: write tcp 188.166.225.170:40266->77.90.31.10:4524: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.950593058 (316h30m9.502063529s) [noise=true , address=77.90.31.14:4690 , piece_id=RJ3KXY3HURT7Q4HZYTOLSAL7Y6VMDGGRIQP6XHE4YB6OAA3PJRAA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.950853535 (316h30m9.502324006s) [noise=true , address=45.91.116.14:4213 , piece_id=HPFISTE2N3B4RBDPAZAGVZJ5WYLCPHTCYH4WMJ56IWEHR3ZBEMOQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12tQRqowH4aDr9h3s1WLs1o7na1ueiafwgUeeHfAB6g7mcE6e1t, address:45.91.116.14:4213): protocol: rpc: dial tcp 45.91.116.14:4213: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.951186438 (316h30m9.502656889s) [noise=true , address=45.91.116.6:4000 , piece_id=T4CI3KKNMFPT3ZIODHE7BQ4DVHVH2QTSO6OB7QZPL5SNPXQJE6TA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12CuCqHJYbanbuaEMHvtcuQXhvcdahD8eyT2fKDhqtxdSZ6ES4f, address:45.91.116.6:4000): protocol: rpc: dial tcp 45.91.116.6:4000: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.951409864 (316h30m9.502880325s) [noise=true , address=45.91.116.7:4039 , piece_id=HSRY63FXTEPCP2T46V77RVSC24GGRBAYLLU7DTKOLWGVOYHF6YVA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:16oq1Yv7JfHrg4gdHfhwuMj4UsWeQQPayKV6Pj1NY2CgJhuPr2, address:45.91.116.7:4039): protocol: write tcp 188.166.225.170:36904->45.91.116.7:4039: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.951660191 (316h30m9.503130652s) [noise=true , address=77.90.31.11:4572 , piece_id=G4XSIPSWOLSEOGGFA2NLOSDCGTLZPI3P5LDNNLB5RGMU5OQGWXGQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1ZAfYZ4wgpZVvatj6wtHona3hTaPcLxAmXC1DT2uKQyo8nLQQ7, address:77.90.31.11:4572): protocol: rpc: dial tcp 77.90.31.11:4572: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.952019855 (316h30m9.503490316s) [noise=true , address=45.91.116.9:4274 , piece_id=MXWRAXYLBBGLUVTA5GP3JVX2IRM2UI2IC35D7KX3XMFEBXLIDX5A, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12g6jFa38dTz5UdgsQaEuM2MSH1H7cxaysWsVmbHkM2Dd2W2iDb, address:45.91.116.9:4274): protocol: rpc: dial tcp 45.91.116.9:4274: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.952290972 (316h30m9.503761443s) [noise=true , address=77.90.31.15:4720 , piece_id=BDQUF6C5TT3ASE3DIN6SSVJHOZAJK4ZEW33OIYPTGB4LDVRELSUA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12VbsxFqMB3vVphSAJqYpbBzua7Goq45E6ovJSaExkpLtfkcc74, address:77.90.31.15:4720): protocol: rpc: dial tcp 77.90.31.15:4720: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.952544654 (316h30m9.504015105s) [noise=true , address=45.91.116.10:4110 , piece_id=H6FYA6K4N4AQYU3MN5RGYNBMYFXXBJDBTUCN37EAT7WBRNTX2YQA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:123KwusATu1ZKtNNVHZZZBE67ZEnVb8puyfQidBi3fEWREqdf1h, address:45.91.116.10:4110): protocol: rpc: dial tcp 45.91.116.10:4110: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.953548155 (316h30m9.505018617s) [noise=true , address=77.90.31.17:4787 , piece_id=IYIEVH6CARRYCARKOLKWVBICQDNDRCTGQMWF3VJZ6YBC3GO5OWHQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1Lietonb5aVbaZQt6wQ2EoxwJkSEn9oL9Dwyrc9mTyomEKikNt, address:77.90.31.17:4787): protocol: read tcp 188.166.225.170:57908->77.90.31.17:4787: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.953579816 (316h30m9.505050277s) [seg_pos={0 0}, upload=278731]: Done waiting for segment. successful: 5 optimal: 14 errs: <nil>
Nov 22 12:34:11.195719123 (316h30m9.747189584s) [upload=278731]: Done uploading segments...
Nov 22 12:34:11.195932840 (316h30m9.747403301s) [upload=278731]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 5)
Duration: 5.672855926s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
5 out of 60 is failed (threshold=6) [......X..X.XX.............................................X.] duration:
4.311027028s trace:
https://jaeger.dev.storj.io/trace/38596674a32f39c1
FAIL (2025-11-22 13:09:28 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.311053729s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....X......................................................] duration:
5.664952846s trace:
https://jaeger.dev.storj.io/trace/5ae3d2bf80495000
FAIL (2025-11-22 13:08:58 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Nov 22 12:31:07.276308940 (316h27m5.827779401s) [upload=278701]: Starting upload
Nov 22 12:31:07.276483523 (316h27m5.827953994s) [upload=278701]: Done starting upload
Nov 22 12:31:07.276562243 (316h27m5.828032704s) [upload=278701]: Uploading segments...
Nov 22 12:31:07.276950593 (316h27m5.828421054s) [upload=278701]: Got next segment. Inline: false
Nov 22 12:31:07.277027639 (316h27m5.8284981s) [upload=278701]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:31:07.277034181 (316h27m5.828504643s) [upload=278701]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:31:07.520212027 (316h27m6.071682488s) [seg_pos={0 0}, upload=278701]: Begin upload segment...
Nov 22 12:31:07.520269256 (316h27m6.071739717s) [seg_pos={0 0}, upload=278701]: Done begin upload segment.
Nov 22 12:31:07.520308992 (316h27m6.071779453s) [noise=true , address=145.79.227.27:4689 , piece_id=76S65KEZ7I2E76RR2JXOZ3ONGXCWSJA5XB7XQVFEEKGGNC34Z4EQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.520426746 (316h27m6.071897207s) [noise=true , address=145.79.227.21:4474 , piece_id=PCRWYQVCAZCPOILAEDHJGDBFKX4GEJ6JXCF7FUS7OVFRUTDPJI4Q, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.521175282 (316h27m6.072645743s) [noise=true , address=145.79.227.17:4972 , piece_id=UZK7LCIOBGGDNJZUVDBOLBJHGOIA4SUT2O3PD67PYQD3J2DGBJEA, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.521527573 (316h27m6.072998044s) [noise=true , address=145.79.227.23:4546 , piece_id=73UPPN2SYWNWEVWHCF4BRGKNO5GCRGOV3S5JZPX4MW75EJZRQOWQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.521694671 (316h27m6.073165132s) [noise=true , address=45.91.116.10:4082 , piece_id=GBDYXAWONOHQGBGM3NCRQYDSPFOAIHRJHBB2W6ZGJLMZUJLV65XA, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.522219410 (316h27m6.073689871s) [noise=true , address=145.79.227.20:4437 , piece_id=CZ32MB4AEZZ6YKCNWKRYALVAK7MT3KNXDSZCQTPUV2Q53NX6Y3BQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.522261240 (316h27m6.073731691s) [noise=true , address=145.79.227.16:4909 , piece_id=ZTL4ER5YJWSDGO5F5KLRUL5WRTA6WKVOJOERIBTRIIVEWXUY5Z2Q, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.522719151 (316h27m6.074189612s) [noise=true , address=145.79.227.26:4654 , piece_id=EM4RRGMPPNVR7VTXP3BED43FJFIUDZYVT4S5OUK2TRXZHYNMQ64Q, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.523017269 (316h27m6.07448772s) [noise=true , address=77.90.31.13:4627 , piece_id=Q7U5BFXHYQBBJQCU4EQGAR75WKL5QMBJ3LO34UYGO7QHLMDJBKWQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.523216949 (316h27m6.07468741s) [noise=true , address=145.79.227.25:4618 , piece_id=CV6M3CTPQR2MPQZFU6W34MFQERT6MMN3RRF2N3W35FGI464OSUUA, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.523744804 (316h27m6.075215265s) [noise=true , address=145.79.227.6:4258 , piece_id=B5D4TCVJORSAWDAU4DQ5VAV4CCRF5WUI24DUVUORAEKLAWGI7ABQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.523828153 (316h27m6.075298614s) [noise=true , address=145.79.227.24:4583 , piece_id=MLW4X34EWXKPOCFVI4VIQSL6RMUOJ5HRYHQ4HY35JIFT4ZFJ6F3Q, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.524255396 (316h27m6.075725858s) [noise=true , address=45.91.116.7:4009 , piece_id=FXQGF3UQN5CPFBS2HWD7BAXR4QIOF42MMK3VJQ7AK4THYJWOFFLQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.524662051 (316h27m6.076132512s) [noise=true , address=145.79.227.15:4874 , piece_id=TYAXIF3ZJQQSFGFANYF56DG4O5WBBXK4M7OT67E7ARYNZ663HESA, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.524746221 (316h27m6.076216682s) [noise=true , address=145.79.227.28:4726 , piece_id=3LBJV2B4CVXEMHCOPRW7DR5QBKPS6DFMPTHQSVNR4XRTZZVIGBQA, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.525227157 (316h27m6.076697608s) [noise=true , address=77.90.31.21:4879 , piece_id=GHTH55OGGXBAZZ5FGPCDYVHDYEETM7DNVTPSTTFFFRZMKZZ3XRXQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:09.134610631 (316h27m7.686081093s) [upload=278701]: Next returned nil segment
Nov 22 12:31:09.134628024 (316h27m7.686098495s) [upload=278701]: Waiting for error group managing segments...
Nov 22 12:31:09.502125778 (316h27m8.053596239s) [noise=true , address=145.79.227.16:4909 , piece_id=ZTL4ER5YJWSDGO5F5KLRUL5WRTA6WKVOJOERIBTRIIVEWXUY5Z2Q, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.503083742 (316h27m8.054554193s) [noise=true , address=145.79.227.20:4437 , piece_id=CZ32MB4AEZZ6YKCNWKRYALVAK7MT3KNXDSZCQTPUV2Q53NX6Y3BQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.517732716 (316h27m8.069203177s) [noise=true , address=145.79.227.25:4618 , piece_id=CV6M3CTPQR2MPQZFU6W34MFQERT6MMN3RRF2N3W35FGI464OSUUA, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.518029582 (316h27m8.069500043s) [noise=true , address=145.79.227.28:4726 , piece_id=3LBJV2B4CVXEMHCOPRW7DR5QBKPS6DFMPTHQSVNR4XRTZZVIGBQA, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.519935051 (316h27m8.071405512s) [noise=true , address=145.79.227.26:4654 , piece_id=EM4RRGMPPNVR7VTXP3BED43FJFIUDZYVT4S5OUK2TRXZHYNMQ64Q, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.520607661 (316h27m8.072078122s) [noise=true , address=145.79.227.27:4689 , piece_id=76S65KEZ7I2E76RR2JXOZ3ONGXCWSJA5XB7XQVFEEKGGNC34Z4EQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.521187736 (316h27m8.072658197s) [noise=true , address=145.79.227.23:4546 , piece_id=73UPPN2SYWNWEVWHCF4BRGKNO5GCRGOV3S5JZPX4MW75EJZRQOWQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.532974760 (316h27m8.084445221s) [noise=true , address=145.79.227.24:4583 , piece_id=MLW4X34EWXKPOCFVI4VIQSL6RMUOJ5HRYHQ4HY35JIFT4ZFJ6F3Q, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.533688800 (316h27m8.085159271s) [noise=true , address=145.79.227.6:4258 , piece_id=B5D4TCVJORSAWDAU4DQ5VAV4CCRF5WUI24DUVUORAEKLAWGI7ABQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.534460109 (316h27m8.08593057s) [noise=true , address=145.79.227.15:4874 , piece_id=TYAXIF3ZJQQSFGFANYF56DG4O5WBBXK4M7OT67E7ARYNZ663HESA, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.542955375 (316h27m8.094425826s) [noise=true , address=145.79.227.17:4972 , piece_id=UZK7LCIOBGGDNJZUVDBOLBJHGOIA4SUT2O3PD67PYQD3J2DGBJEA, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.563098982 (316h27m8.114569443s) [noise=true , address=145.79.227.21:4474 , piece_id=PCRWYQVCAZCPOILAEDHJGDBFKX4GEJ6JXCF7FUS7OVFRUTDPJI4Q, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:36.585475346 (316h27m35.136945807s) [noise=true , address=45.91.116.10:4082 , piece_id=GBDYXAWONOHQGBGM3NCRQYDSPFOAIHRJHBB2W6ZGJLMZUJLV65XA, seg_pos={0 0}, upload=278701]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: read tcp 188.166.225.170:54326->45.91.116.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:36.585499923 (316h27m35.136970384s) [noise=true , address=77.90.31.21:4879 , piece_id=GHTH55OGGXBAZZ5FGPCDYVHDYEETM7DNVTPSTTFFFRZMKZZ3XRXQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: rpc: dial tcp 77.90.31.21:4879: i/o timeout; context deadline exceeded
Nov 22 12:31:36.585749648 (316h27m35.137220119s) [noise=true , address=77.90.31.13:4627 , piece_id=Q7U5BFXHYQBBJQCU4EQGAR75WKL5QMBJ3LO34UYGO7QHLMDJBKWQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:36.585848086 (316h27m35.137318547s) [noise=true , address=45.91.116.7:4009 , piece_id=FXQGF3UQN5CPFBS2HWD7BAXR4QIOF42MMK3VJQ7AK4THYJWOFFLQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:36.585900936 (316h27m35.137371397s) [seg_pos={0 0}, upload=278701]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Nov 22 12:31:36.828881025 (316h27m35.380351486s) [upload=278701]: Done uploading segments...
Nov 22 12:31:36.828915801 (316h27m35.380386262s) [upload=278701]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 5.665018843s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [..X...X.....................................................] duration:
5.684684008s trace:
https://jaeger.dev.storj.io/trace/570ccd0ce1345814
FAIL (2025-11-22 13:08:56 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.684709907s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
5 out of 60 is failed (threshold=6) [.......X..X.XX.............................................X] duration:
5.002476938s trace:
https://jaeger.dev.storj.io/trace/38596674a32f39c1
FAIL (2025-11-22 13:08:51 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.002502929s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [..X..X..X...................................................] duration:
12.768118038s trace:
https://jaeger.dev.storj.io/trace/92df32505639913
FAIL (2025-11-22 13:08:47 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 12.768145088s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [......X.....................................................] duration:
6.053224092s trace:
https://jaeger.dev.storj.io/trace/5ae3d2bf80495000
FAIL (2025-11-22 13:08:41 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Nov 22 12:31:07.276308940 (316h27m5.827779401s) [upload=278701]: Starting upload
Nov 22 12:31:07.276483523 (316h27m5.827953994s) [upload=278701]: Done starting upload
Nov 22 12:31:07.276562243 (316h27m5.828032704s) [upload=278701]: Uploading segments...
Nov 22 12:31:07.276950593 (316h27m5.828421054s) [upload=278701]: Got next segment. Inline: false
Nov 22 12:31:07.277027639 (316h27m5.8284981s) [upload=278701]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:31:07.277034181 (316h27m5.828504643s) [upload=278701]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:31:07.520212027 (316h27m6.071682488s) [seg_pos={0 0}, upload=278701]: Begin upload segment...
Nov 22 12:31:07.520269256 (316h27m6.071739717s) [seg_pos={0 0}, upload=278701]: Done begin upload segment.
Nov 22 12:31:07.520308992 (316h27m6.071779453s) [noise=true , address=145.79.227.27:4689 , piece_id=76S65KEZ7I2E76RR2JXOZ3ONGXCWSJA5XB7XQVFEEKGGNC34Z4EQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.520426746 (316h27m6.071897207s) [noise=true , address=145.79.227.21:4474 , piece_id=PCRWYQVCAZCPOILAEDHJGDBFKX4GEJ6JXCF7FUS7OVFRUTDPJI4Q, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.521175282 (316h27m6.072645743s) [noise=true , address=145.79.227.17:4972 , piece_id=UZK7LCIOBGGDNJZUVDBOLBJHGOIA4SUT2O3PD67PYQD3J2DGBJEA, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.521527573 (316h27m6.072998044s) [noise=true , address=145.79.227.23:4546 , piece_id=73UPPN2SYWNWEVWHCF4BRGKNO5GCRGOV3S5JZPX4MW75EJZRQOWQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.521694671 (316h27m6.073165132s) [noise=true , address=45.91.116.10:4082 , piece_id=GBDYXAWONOHQGBGM3NCRQYDSPFOAIHRJHBB2W6ZGJLMZUJLV65XA, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.522219410 (316h27m6.073689871s) [noise=true , address=145.79.227.20:4437 , piece_id=CZ32MB4AEZZ6YKCNWKRYALVAK7MT3KNXDSZCQTPUV2Q53NX6Y3BQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.522261240 (316h27m6.073731691s) [noise=true , address=145.79.227.16:4909 , piece_id=ZTL4ER5YJWSDGO5F5KLRUL5WRTA6WKVOJOERIBTRIIVEWXUY5Z2Q, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.522719151 (316h27m6.074189612s) [noise=true , address=145.79.227.26:4654 , piece_id=EM4RRGMPPNVR7VTXP3BED43FJFIUDZYVT4S5OUK2TRXZHYNMQ64Q, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.523017269 (316h27m6.07448772s) [noise=true , address=77.90.31.13:4627 , piece_id=Q7U5BFXHYQBBJQCU4EQGAR75WKL5QMBJ3LO34UYGO7QHLMDJBKWQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.523216949 (316h27m6.07468741s) [noise=true , address=145.79.227.25:4618 , piece_id=CV6M3CTPQR2MPQZFU6W34MFQERT6MMN3RRF2N3W35FGI464OSUUA, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.523744804 (316h27m6.075215265s) [noise=true , address=145.79.227.6:4258 , piece_id=B5D4TCVJORSAWDAU4DQ5VAV4CCRF5WUI24DUVUORAEKLAWGI7ABQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.523828153 (316h27m6.075298614s) [noise=true , address=145.79.227.24:4583 , piece_id=MLW4X34EWXKPOCFVI4VIQSL6RMUOJ5HRYHQ4HY35JIFT4ZFJ6F3Q, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.524255396 (316h27m6.075725858s) [noise=true , address=45.91.116.7:4009 , piece_id=FXQGF3UQN5CPFBS2HWD7BAXR4QIOF42MMK3VJQ7AK4THYJWOFFLQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.524662051 (316h27m6.076132512s) [noise=true , address=145.79.227.15:4874 , piece_id=TYAXIF3ZJQQSFGFANYF56DG4O5WBBXK4M7OT67E7ARYNZ663HESA, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.524746221 (316h27m6.076216682s) [noise=true , address=145.79.227.28:4726 , piece_id=3LBJV2B4CVXEMHCOPRW7DR5QBKPS6DFMPTHQSVNR4XRTZZVIGBQA, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.525227157 (316h27m6.076697608s) [noise=true , address=77.90.31.21:4879 , piece_id=GHTH55OGGXBAZZ5FGPCDYVHDYEETM7DNVTPSTTFFFRZMKZZ3XRXQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:09.134610631 (316h27m7.686081093s) [upload=278701]: Next returned nil segment
Nov 22 12:31:09.134628024 (316h27m7.686098495s) [upload=278701]: Waiting for error group managing segments...
Nov 22 12:31:09.502125778 (316h27m8.053596239s) [noise=true , address=145.79.227.16:4909 , piece_id=ZTL4ER5YJWSDGO5F5KLRUL5WRTA6WKVOJOERIBTRIIVEWXUY5Z2Q, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.503083742 (316h27m8.054554193s) [noise=true , address=145.79.227.20:4437 , piece_id=CZ32MB4AEZZ6YKCNWKRYALVAK7MT3KNXDSZCQTPUV2Q53NX6Y3BQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.517732716 (316h27m8.069203177s) [noise=true , address=145.79.227.25:4618 , piece_id=CV6M3CTPQR2MPQZFU6W34MFQERT6MMN3RRF2N3W35FGI464OSUUA, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.518029582 (316h27m8.069500043s) [noise=true , address=145.79.227.28:4726 , piece_id=3LBJV2B4CVXEMHCOPRW7DR5QBKPS6DFMPTHQSVNR4XRTZZVIGBQA, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.519935051 (316h27m8.071405512s) [noise=true , address=145.79.227.26:4654 , piece_id=EM4RRGMPPNVR7VTXP3BED43FJFIUDZYVT4S5OUK2TRXZHYNMQ64Q, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.520607661 (316h27m8.072078122s) [noise=true , address=145.79.227.27:4689 , piece_id=76S65KEZ7I2E76RR2JXOZ3ONGXCWSJA5XB7XQVFEEKGGNC34Z4EQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.521187736 (316h27m8.072658197s) [noise=true , address=145.79.227.23:4546 , piece_id=73UPPN2SYWNWEVWHCF4BRGKNO5GCRGOV3S5JZPX4MW75EJZRQOWQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.532974760 (316h27m8.084445221s) [noise=true , address=145.79.227.24:4583 , piece_id=MLW4X34EWXKPOCFVI4VIQSL6RMUOJ5HRYHQ4HY35JIFT4ZFJ6F3Q, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.533688800 (316h27m8.085159271s) [noise=true , address=145.79.227.6:4258 , piece_id=B5D4TCVJORSAWDAU4DQ5VAV4CCRF5WUI24DUVUORAEKLAWGI7ABQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.534460109 (316h27m8.08593057s) [noise=true , address=145.79.227.15:4874 , piece_id=TYAXIF3ZJQQSFGFANYF56DG4O5WBBXK4M7OT67E7ARYNZ663HESA, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.542955375 (316h27m8.094425826s) [noise=true , address=145.79.227.17:4972 , piece_id=UZK7LCIOBGGDNJZUVDBOLBJHGOIA4SUT2O3PD67PYQD3J2DGBJEA, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.563098982 (316h27m8.114569443s) [noise=true , address=145.79.227.21:4474 , piece_id=PCRWYQVCAZCPOILAEDHJGDBFKX4GEJ6JXCF7FUS7OVFRUTDPJI4Q, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:36.585475346 (316h27m35.136945807s) [noise=true , address=45.91.116.10:4082 , piece_id=GBDYXAWONOHQGBGM3NCRQYDSPFOAIHRJHBB2W6ZGJLMZUJLV65XA, seg_pos={0 0}, upload=278701]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: read tcp 188.166.225.170:54326->45.91.116.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:36.585499923 (316h27m35.136970384s) [noise=true , address=77.90.31.21:4879 , piece_id=GHTH55OGGXBAZZ5FGPCDYVHDYEETM7DNVTPSTTFFFRZMKZZ3XRXQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: rpc: dial tcp 77.90.31.21:4879: i/o timeout; context deadline exceeded
Nov 22 12:31:36.585749648 (316h27m35.137220119s) [noise=true , address=77.90.31.13:4627 , piece_id=Q7U5BFXHYQBBJQCU4EQGAR75WKL5QMBJ3LO34UYGO7QHLMDJBKWQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:36.585848086 (316h27m35.137318547s) [noise=true , address=45.91.116.7:4009 , piece_id=FXQGF3UQN5CPFBS2HWD7BAXR4QIOF42MMK3VJQ7AK4THYJWOFFLQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:36.585900936 (316h27m35.137371397s) [seg_pos={0 0}, upload=278701]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Nov 22 12:31:36.828881025 (316h27m35.380351486s) [upload=278701]: Done uploading segments...
Nov 22 12:31:36.828915801 (316h27m35.380386262s) [upload=278701]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 6.053251184s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [......X.....X...............................................] duration:
5.777145553s trace:
https://jaeger.dev.storj.io/trace/6c800b4c3a84b58c
FAIL (2025-11-22 13:08:36 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 5)
======LOGS======
N5RGYNBMYFXXBJDBTUCN37EAT7WBRNTX2YQA, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.890374814 (316h29m10.441845275s) [noise=true , address=145.79.227.32:4858 , piece_id=K5L3LEUKZWHS36XJHNNY4TYRXCP4DG5UAKCZXPJJTSG26CKUI4HQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.890535981 (316h29m10.442006442s) [noise=true , address=77.90.31.21:4890 , piece_id=DUEFPJUTYMD564Z4III63LZM6L4IL2ZTGM3QDZ6CG7WV5APHBNRA, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.890774064 (316h29m10.442244525s) [noise=true , address=45.91.116.14:4213 , piece_id=HPFISTE2N3B4RBDPAZAGVZJ5WYLCPHTCYH4WMJ56IWEHR3ZBEMOQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.890934851 (316h29m10.442405312s) [noise=true , address=45.91.116.6:4000 , piece_id=T4CI3KKNMFPT3ZIODHE7BQ4DVHVH2QTSO6OB7QZPL5SNPXQJE6TA, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.891167002 (316h29m10.442637463s) [noise=true , address=77.90.31.17:4787 , piece_id=IYIEVH6CARRYCARKOLKWVBICQDNDRCTGQMWF3VJZ6YBC3GO5OWHQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.891369828 (316h29m10.442840289s) [noise=true , address=45.91.116.7:4039 , piece_id=HSRY63FXTEPCP2T46V77RVSC24GGRBAYLLU7DTKOLWGVOYHF6YVA, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.891556875 (316h29m10.443027336s) [noise=true , address=77.90.31.11:4572 , piece_id=G4XSIPSWOLSEOGGFA2NLOSDCGTLZPI3P5LDNNLB5RGMU5OQGWXGQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.891896812 (316h29m10.443367273s) [noise=true , address=77.90.31.19:4863 , piece_id=3PM3Q6X3ADKX2RKIIFDMK57LTJNJHNFK4UYPKFK3ZEQLYQ23EP3A, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.892019235 (316h29m10.443489696s) [noise=true , address=77.90.31.10:4524 , piece_id=Z2IGDHPDU6JAR5CAFQBGYHO3TQ66K4BZOAMXSC7X3RFWLGHOOMTQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.892316521 (316h29m10.443786982s) [noise=true , address=77.90.31.13:4644 , piece_id=UPDZGC3CMJXA3AOJX65VLASU37NLFGPIJLCAPN32TEQB5QHTU7EQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.892682318 (316h29m10.444152779s) [noise=true , address=45.91.116.9:4274 , piece_id=MXWRAXYLBBGLUVTA5GP3JVX2IRM2UI2IC35D7KX3XMFEBXLIDX5A, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:13.353758506 (316h29m11.905228957s) [upload=278731]: Next returned nil segment
Nov 22 12:33:13.353772043 (316h29m11.905242504s) [upload=278731]: Waiting for error group managing segments...
Nov 22 12:33:13.751703718 (316h29m12.303174169s) [noise=true , address=145.79.227.32:4858 , piece_id=K5L3LEUKZWHS36XJHNNY4TYRXCP4DG5UAKCZXPJJTSG26CKUI4HQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil>
Nov 22 12:33:13.753660293 (316h29m12.305130764s) [noise=true , address=145.79.227.26:4674 , piece_id=UP7VIVRDRL6KERJSX3VLWTKGW3ESGCQ7BLYCBXGBB5UNIDE6OHQQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil>
Nov 22 12:33:13.760046123 (316h29m12.311516574s) [noise=true , address=145.79.227.3:4167 , piece_id=4STIVXGU7SOXWAIPG2PXWBLNDD7EYQ56E2DBN2UZ2AZ3TZSXCTIQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil>
Nov 22 12:33:57.373975014 (316h29m55.925445475s) [noise=true , address=77.90.31.13:4644 , piece_id=UPDZGC3CMJXA3AOJX65VLASU37NLFGPIJLCAPN32TEQB5QHTU7EQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil>
Nov 22 12:34:06.122275078 (316h30m4.673745539s) [noise=true , address=45.91.116.13:4181 , piece_id=MQNJEBQFZMHNDKAH2K3JEA7YBK2UK5BGSNCJF3MASF3CKVA5VD6Q, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil>
Nov 22 12:34:10.949171331 (316h30m9.500641792s) [noise=true , address=77.90.31.21:4890 , piece_id=DUEFPJUTYMD564Z4III63LZM6L4IL2ZTGM3QDZ6CG7WV5APHBNRA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1iE3mDnEhGDsMBtmUXC4EwvSrpyHLoybzVXkC9eoFuYEMHeBrd, address:77.90.31.21:4890): protocol: read tcp 188.166.225.170:55900->77.90.31.21:4890: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.949753650 (316h30m9.501224111s) [noise=true , address=45.91.116.12:4237 , piece_id=OG6XREYEGOYVFFMXCGBERRAIEMI2QMUGSBJBYP7GJQZ7FOAI4BIA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1FMjHdNGV42a9QZyMmSPTJWQrA8CdSRvXqrimnt6UGVreCzmgF, address:45.91.116.12:4237): protocol: rpc: dial tcp 45.91.116.12:4237: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.950066235 (316h30m9.501536706s) [noise=true , address=77.90.31.19:4863 , piece_id=3PM3Q6X3ADKX2RKIIFDMK57LTJNJHNFK4UYPKFK3ZEQLYQ23EP3A, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:124x1cbCPJRK8YkgSP6UzHymoLu1E1znnfg5WBiD89kSDLi8JeL, address:77.90.31.19:4863): protocol: write tcp 188.166.225.170:45968->77.90.31.19:4863: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.950246558 (316h30m9.50171702s) [noise=true , address=77.90.31.10:4524 , piece_id=Z2IGDHPDU6JAR5CAFQBGYHO3TQ66K4BZOAMXSC7X3RFWLGHOOMTQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12uN7CznoK4AD8FqFkNUndawdGJ9rmjAe7oNnjJkV844ScY87qe, address:77.90.31.10:4524): protocol: write tcp 188.166.225.170:40266->77.90.31.10:4524: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.950593058 (316h30m9.502063529s) [noise=true , address=77.90.31.14:4690 , piece_id=RJ3KXY3HURT7Q4HZYTOLSAL7Y6VMDGGRIQP6XHE4YB6OAA3PJRAA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.950853535 (316h30m9.502324006s) [noise=true , address=45.91.116.14:4213 , piece_id=HPFISTE2N3B4RBDPAZAGVZJ5WYLCPHTCYH4WMJ56IWEHR3ZBEMOQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12tQRqowH4aDr9h3s1WLs1o7na1ueiafwgUeeHfAB6g7mcE6e1t, address:45.91.116.14:4213): protocol: rpc: dial tcp 45.91.116.14:4213: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.951186438 (316h30m9.502656889s) [noise=true , address=45.91.116.6:4000 , piece_id=T4CI3KKNMFPT3ZIODHE7BQ4DVHVH2QTSO6OB7QZPL5SNPXQJE6TA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12CuCqHJYbanbuaEMHvtcuQXhvcdahD8eyT2fKDhqtxdSZ6ES4f, address:45.91.116.6:4000): protocol: rpc: dial tcp 45.91.116.6:4000: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.951409864 (316h30m9.502880325s) [noise=true , address=45.91.116.7:4039 , piece_id=HSRY63FXTEPCP2T46V77RVSC24GGRBAYLLU7DTKOLWGVOYHF6YVA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:16oq1Yv7JfHrg4gdHfhwuMj4UsWeQQPayKV6Pj1NY2CgJhuPr2, address:45.91.116.7:4039): protocol: write tcp 188.166.225.170:36904->45.91.116.7:4039: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.951660191 (316h30m9.503130652s) [noise=true , address=77.90.31.11:4572 , piece_id=G4XSIPSWOLSEOGGFA2NLOSDCGTLZPI3P5LDNNLB5RGMU5OQGWXGQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1ZAfYZ4wgpZVvatj6wtHona3hTaPcLxAmXC1DT2uKQyo8nLQQ7, address:77.90.31.11:4572): protocol: rpc: dial tcp 77.90.31.11:4572: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.952019855 (316h30m9.503490316s) [noise=true , address=45.91.116.9:4274 , piece_id=MXWRAXYLBBGLUVTA5GP3JVX2IRM2UI2IC35D7KX3XMFEBXLIDX5A, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12g6jFa38dTz5UdgsQaEuM2MSH1H7cxaysWsVmbHkM2Dd2W2iDb, address:45.91.116.9:4274): protocol: rpc: dial tcp 45.91.116.9:4274: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.952290972 (316h30m9.503761443s) [noise=true , address=77.90.31.15:4720 , piece_id=BDQUF6C5TT3ASE3DIN6SSVJHOZAJK4ZEW33OIYPTGB4LDVRELSUA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12VbsxFqMB3vVphSAJqYpbBzua7Goq45E6ovJSaExkpLtfkcc74, address:77.90.31.15:4720): protocol: rpc: dial tcp 77.90.31.15:4720: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.952544654 (316h30m9.504015105s) [noise=true , address=45.91.116.10:4110 , piece_id=H6FYA6K4N4AQYU3MN5RGYNBMYFXXBJDBTUCN37EAT7WBRNTX2YQA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:123KwusATu1ZKtNNVHZZZBE67ZEnVb8puyfQidBi3fEWREqdf1h, address:45.91.116.10:4110): protocol: rpc: dial tcp 45.91.116.10:4110: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.953548155 (316h30m9.505018617s) [noise=true , address=77.90.31.17:4787 , piece_id=IYIEVH6CARRYCARKOLKWVBICQDNDRCTGQMWF3VJZ6YBC3GO5OWHQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1Lietonb5aVbaZQt6wQ2EoxwJkSEn9oL9Dwyrc9mTyomEKikNt, address:77.90.31.17:4787): protocol: read tcp 188.166.225.170:57908->77.90.31.17:4787: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.953579816 (316h30m9.505050277s) [seg_pos={0 0}, upload=278731]: Done waiting for segment. successful: 5 optimal: 14 errs: <nil>
Nov 22 12:34:11.195719123 (316h30m9.747189584s) [upload=278731]: Done uploading segments...
Nov 22 12:34:11.195932840 (316h30m9.747403301s) [upload=278731]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 5)
Duration: 5.777181823s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [...X..X..X..................................................] duration:
14.468142597s trace:
https://jaeger.dev.storj.io/trace/92df32505639913
FAIL (2025-11-22 13:08:28 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 14.468283735s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [........X..X.XX.............................................] duration:
3.470187845s trace:
https://jaeger.dev.storj.io/trace/71781384eb27b231
FAIL (2025-11-22 13:08:24 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.470210428s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [.......X.....X..............................................] duration:
5.750573612s trace:
https://jaeger.dev.storj.io/trace/6c800b4c3a84b58c
FAIL (2025-11-22 13:08:13 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 5)
======LOGS======
N5RGYNBMYFXXBJDBTUCN37EAT7WBRNTX2YQA, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.890374814 (316h29m10.441845275s) [noise=true , address=145.79.227.32:4858 , piece_id=K5L3LEUKZWHS36XJHNNY4TYRXCP4DG5UAKCZXPJJTSG26CKUI4HQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.890535981 (316h29m10.442006442s) [noise=true , address=77.90.31.21:4890 , piece_id=DUEFPJUTYMD564Z4III63LZM6L4IL2ZTGM3QDZ6CG7WV5APHBNRA, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.890774064 (316h29m10.442244525s) [noise=true , address=45.91.116.14:4213 , piece_id=HPFISTE2N3B4RBDPAZAGVZJ5WYLCPHTCYH4WMJ56IWEHR3ZBEMOQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.890934851 (316h29m10.442405312s) [noise=true , address=45.91.116.6:4000 , piece_id=T4CI3KKNMFPT3ZIODHE7BQ4DVHVH2QTSO6OB7QZPL5SNPXQJE6TA, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.891167002 (316h29m10.442637463s) [noise=true , address=77.90.31.17:4787 , piece_id=IYIEVH6CARRYCARKOLKWVBICQDNDRCTGQMWF3VJZ6YBC3GO5OWHQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.891369828 (316h29m10.442840289s) [noise=true , address=45.91.116.7:4039 , piece_id=HSRY63FXTEPCP2T46V77RVSC24GGRBAYLLU7DTKOLWGVOYHF6YVA, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.891556875 (316h29m10.443027336s) [noise=true , address=77.90.31.11:4572 , piece_id=G4XSIPSWOLSEOGGFA2NLOSDCGTLZPI3P5LDNNLB5RGMU5OQGWXGQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.891896812 (316h29m10.443367273s) [noise=true , address=77.90.31.19:4863 , piece_id=3PM3Q6X3ADKX2RKIIFDMK57LTJNJHNFK4UYPKFK3ZEQLYQ23EP3A, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.892019235 (316h29m10.443489696s) [noise=true , address=77.90.31.10:4524 , piece_id=Z2IGDHPDU6JAR5CAFQBGYHO3TQ66K4BZOAMXSC7X3RFWLGHOOMTQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.892316521 (316h29m10.443786982s) [noise=true , address=77.90.31.13:4644 , piece_id=UPDZGC3CMJXA3AOJX65VLASU37NLFGPIJLCAPN32TEQB5QHTU7EQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.892682318 (316h29m10.444152779s) [noise=true , address=45.91.116.9:4274 , piece_id=MXWRAXYLBBGLUVTA5GP3JVX2IRM2UI2IC35D7KX3XMFEBXLIDX5A, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:13.353758506 (316h29m11.905228957s) [upload=278731]: Next returned nil segment
Nov 22 12:33:13.353772043 (316h29m11.905242504s) [upload=278731]: Waiting for error group managing segments...
Nov 22 12:33:13.751703718 (316h29m12.303174169s) [noise=true , address=145.79.227.32:4858 , piece_id=K5L3LEUKZWHS36XJHNNY4TYRXCP4DG5UAKCZXPJJTSG26CKUI4HQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil>
Nov 22 12:33:13.753660293 (316h29m12.305130764s) [noise=true , address=145.79.227.26:4674 , piece_id=UP7VIVRDRL6KERJSX3VLWTKGW3ESGCQ7BLYCBXGBB5UNIDE6OHQQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil>
Nov 22 12:33:13.760046123 (316h29m12.311516574s) [noise=true , address=145.79.227.3:4167 , piece_id=4STIVXGU7SOXWAIPG2PXWBLNDD7EYQ56E2DBN2UZ2AZ3TZSXCTIQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil>
Nov 22 12:33:57.373975014 (316h29m55.925445475s) [noise=true , address=77.90.31.13:4644 , piece_id=UPDZGC3CMJXA3AOJX65VLASU37NLFGPIJLCAPN32TEQB5QHTU7EQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil>
Nov 22 12:34:06.122275078 (316h30m4.673745539s) [noise=true , address=45.91.116.13:4181 , piece_id=MQNJEBQFZMHNDKAH2K3JEA7YBK2UK5BGSNCJF3MASF3CKVA5VD6Q, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil>
Nov 22 12:34:10.949171331 (316h30m9.500641792s) [noise=true , address=77.90.31.21:4890 , piece_id=DUEFPJUTYMD564Z4III63LZM6L4IL2ZTGM3QDZ6CG7WV5APHBNRA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1iE3mDnEhGDsMBtmUXC4EwvSrpyHLoybzVXkC9eoFuYEMHeBrd, address:77.90.31.21:4890): protocol: read tcp 188.166.225.170:55900->77.90.31.21:4890: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.949753650 (316h30m9.501224111s) [noise=true , address=45.91.116.12:4237 , piece_id=OG6XREYEGOYVFFMXCGBERRAIEMI2QMUGSBJBYP7GJQZ7FOAI4BIA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1FMjHdNGV42a9QZyMmSPTJWQrA8CdSRvXqrimnt6UGVreCzmgF, address:45.91.116.12:4237): protocol: rpc: dial tcp 45.91.116.12:4237: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.950066235 (316h30m9.501536706s) [noise=true , address=77.90.31.19:4863 , piece_id=3PM3Q6X3ADKX2RKIIFDMK57LTJNJHNFK4UYPKFK3ZEQLYQ23EP3A, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:124x1cbCPJRK8YkgSP6UzHymoLu1E1znnfg5WBiD89kSDLi8JeL, address:77.90.31.19:4863): protocol: write tcp 188.166.225.170:45968->77.90.31.19:4863: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.950246558 (316h30m9.50171702s) [noise=true , address=77.90.31.10:4524 , piece_id=Z2IGDHPDU6JAR5CAFQBGYHO3TQ66K4BZOAMXSC7X3RFWLGHOOMTQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12uN7CznoK4AD8FqFkNUndawdGJ9rmjAe7oNnjJkV844ScY87qe, address:77.90.31.10:4524): protocol: write tcp 188.166.225.170:40266->77.90.31.10:4524: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.950593058 (316h30m9.502063529s) [noise=true , address=77.90.31.14:4690 , piece_id=RJ3KXY3HURT7Q4HZYTOLSAL7Y6VMDGGRIQP6XHE4YB6OAA3PJRAA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.950853535 (316h30m9.502324006s) [noise=true , address=45.91.116.14:4213 , piece_id=HPFISTE2N3B4RBDPAZAGVZJ5WYLCPHTCYH4WMJ56IWEHR3ZBEMOQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12tQRqowH4aDr9h3s1WLs1o7na1ueiafwgUeeHfAB6g7mcE6e1t, address:45.91.116.14:4213): protocol: rpc: dial tcp 45.91.116.14:4213: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.951186438 (316h30m9.502656889s) [noise=true , address=45.91.116.6:4000 , piece_id=T4CI3KKNMFPT3ZIODHE7BQ4DVHVH2QTSO6OB7QZPL5SNPXQJE6TA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12CuCqHJYbanbuaEMHvtcuQXhvcdahD8eyT2fKDhqtxdSZ6ES4f, address:45.91.116.6:4000): protocol: rpc: dial tcp 45.91.116.6:4000: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.951409864 (316h30m9.502880325s) [noise=true , address=45.91.116.7:4039 , piece_id=HSRY63FXTEPCP2T46V77RVSC24GGRBAYLLU7DTKOLWGVOYHF6YVA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:16oq1Yv7JfHrg4gdHfhwuMj4UsWeQQPayKV6Pj1NY2CgJhuPr2, address:45.91.116.7:4039): protocol: write tcp 188.166.225.170:36904->45.91.116.7:4039: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.951660191 (316h30m9.503130652s) [noise=true , address=77.90.31.11:4572 , piece_id=G4XSIPSWOLSEOGGFA2NLOSDCGTLZPI3P5LDNNLB5RGMU5OQGWXGQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1ZAfYZ4wgpZVvatj6wtHona3hTaPcLxAmXC1DT2uKQyo8nLQQ7, address:77.90.31.11:4572): protocol: rpc: dial tcp 77.90.31.11:4572: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.952019855 (316h30m9.503490316s) [noise=true , address=45.91.116.9:4274 , piece_id=MXWRAXYLBBGLUVTA5GP3JVX2IRM2UI2IC35D7KX3XMFEBXLIDX5A, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12g6jFa38dTz5UdgsQaEuM2MSH1H7cxaysWsVmbHkM2Dd2W2iDb, address:45.91.116.9:4274): protocol: rpc: dial tcp 45.91.116.9:4274: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.952290972 (316h30m9.503761443s) [noise=true , address=77.90.31.15:4720 , piece_id=BDQUF6C5TT3ASE3DIN6SSVJHOZAJK4ZEW33OIYPTGB4LDVRELSUA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12VbsxFqMB3vVphSAJqYpbBzua7Goq45E6ovJSaExkpLtfkcc74, address:77.90.31.15:4720): protocol: rpc: dial tcp 77.90.31.15:4720: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.952544654 (316h30m9.504015105s) [noise=true , address=45.91.116.10:4110 , piece_id=H6FYA6K4N4AQYU3MN5RGYNBMYFXXBJDBTUCN37EAT7WBRNTX2YQA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:123KwusATu1ZKtNNVHZZZBE67ZEnVb8puyfQidBi3fEWREqdf1h, address:45.91.116.10:4110): protocol: rpc: dial tcp 45.91.116.10:4110: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.953548155 (316h30m9.505018617s) [noise=true , address=77.90.31.17:4787 , piece_id=IYIEVH6CARRYCARKOLKWVBICQDNDRCTGQMWF3VJZ6YBC3GO5OWHQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1Lietonb5aVbaZQt6wQ2EoxwJkSEn9oL9Dwyrc9mTyomEKikNt, address:77.90.31.17:4787): protocol: read tcp 188.166.225.170:57908->77.90.31.17:4787: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.953579816 (316h30m9.505050277s) [seg_pos={0 0}, upload=278731]: Done waiting for segment. successful: 5 optimal: 14 errs: <nil>
Nov 22 12:34:11.195719123 (316h30m9.747189584s) [upload=278731]: Done uploading segments...
Nov 22 12:34:11.195932840 (316h30m9.747403301s) [upload=278731]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 5)
Duration: 5.750601525s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [....X..X..X.................................................] duration:
11.686826467s trace:
https://jaeger.dev.storj.io/trace/92df32505639913
FAIL (2025-11-22 13:08:03 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 11.686859028s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......X....................................................] duration:
5.782588023s trace:
https://jaeger.dev.storj.io/trace/5ae3d2bf80495000
FAIL (2025-11-22 13:08:01 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Nov 22 12:31:07.276308940 (316h27m5.827779401s) [upload=278701]: Starting upload
Nov 22 12:31:07.276483523 (316h27m5.827953994s) [upload=278701]: Done starting upload
Nov 22 12:31:07.276562243 (316h27m5.828032704s) [upload=278701]: Uploading segments...
Nov 22 12:31:07.276950593 (316h27m5.828421054s) [upload=278701]: Got next segment. Inline: false
Nov 22 12:31:07.277027639 (316h27m5.8284981s) [upload=278701]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:31:07.277034181 (316h27m5.828504643s) [upload=278701]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:31:07.520212027 (316h27m6.071682488s) [seg_pos={0 0}, upload=278701]: Begin upload segment...
Nov 22 12:31:07.520269256 (316h27m6.071739717s) [seg_pos={0 0}, upload=278701]: Done begin upload segment.
Nov 22 12:31:07.520308992 (316h27m6.071779453s) [noise=true , address=145.79.227.27:4689 , piece_id=76S65KEZ7I2E76RR2JXOZ3ONGXCWSJA5XB7XQVFEEKGGNC34Z4EQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.520426746 (316h27m6.071897207s) [noise=true , address=145.79.227.21:4474 , piece_id=PCRWYQVCAZCPOILAEDHJGDBFKX4GEJ6JXCF7FUS7OVFRUTDPJI4Q, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.521175282 (316h27m6.072645743s) [noise=true , address=145.79.227.17:4972 , piece_id=UZK7LCIOBGGDNJZUVDBOLBJHGOIA4SUT2O3PD67PYQD3J2DGBJEA, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.521527573 (316h27m6.072998044s) [noise=true , address=145.79.227.23:4546 , piece_id=73UPPN2SYWNWEVWHCF4BRGKNO5GCRGOV3S5JZPX4MW75EJZRQOWQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.521694671 (316h27m6.073165132s) [noise=true , address=45.91.116.10:4082 , piece_id=GBDYXAWONOHQGBGM3NCRQYDSPFOAIHRJHBB2W6ZGJLMZUJLV65XA, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.522219410 (316h27m6.073689871s) [noise=true , address=145.79.227.20:4437 , piece_id=CZ32MB4AEZZ6YKCNWKRYALVAK7MT3KNXDSZCQTPUV2Q53NX6Y3BQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.522261240 (316h27m6.073731691s) [noise=true , address=145.79.227.16:4909 , piece_id=ZTL4ER5YJWSDGO5F5KLRUL5WRTA6WKVOJOERIBTRIIVEWXUY5Z2Q, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.522719151 (316h27m6.074189612s) [noise=true , address=145.79.227.26:4654 , piece_id=EM4RRGMPPNVR7VTXP3BED43FJFIUDZYVT4S5OUK2TRXZHYNMQ64Q, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.523017269 (316h27m6.07448772s) [noise=true , address=77.90.31.13:4627 , piece_id=Q7U5BFXHYQBBJQCU4EQGAR75WKL5QMBJ3LO34UYGO7QHLMDJBKWQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.523216949 (316h27m6.07468741s) [noise=true , address=145.79.227.25:4618 , piece_id=CV6M3CTPQR2MPQZFU6W34MFQERT6MMN3RRF2N3W35FGI464OSUUA, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.523744804 (316h27m6.075215265s) [noise=true , address=145.79.227.6:4258 , piece_id=B5D4TCVJORSAWDAU4DQ5VAV4CCRF5WUI24DUVUORAEKLAWGI7ABQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.523828153 (316h27m6.075298614s) [noise=true , address=145.79.227.24:4583 , piece_id=MLW4X34EWXKPOCFVI4VIQSL6RMUOJ5HRYHQ4HY35JIFT4ZFJ6F3Q, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.524255396 (316h27m6.075725858s) [noise=true , address=45.91.116.7:4009 , piece_id=FXQGF3UQN5CPFBS2HWD7BAXR4QIOF42MMK3VJQ7AK4THYJWOFFLQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.524662051 (316h27m6.076132512s) [noise=true , address=145.79.227.15:4874 , piece_id=TYAXIF3ZJQQSFGFANYF56DG4O5WBBXK4M7OT67E7ARYNZ663HESA, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.524746221 (316h27m6.076216682s) [noise=true , address=145.79.227.28:4726 , piece_id=3LBJV2B4CVXEMHCOPRW7DR5QBKPS6DFMPTHQSVNR4XRTZZVIGBQA, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.525227157 (316h27m6.076697608s) [noise=true , address=77.90.31.21:4879 , piece_id=GHTH55OGGXBAZZ5FGPCDYVHDYEETM7DNVTPSTTFFFRZMKZZ3XRXQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:09.134610631 (316h27m7.686081093s) [upload=278701]: Next returned nil segment
Nov 22 12:31:09.134628024 (316h27m7.686098495s) [upload=278701]: Waiting for error group managing segments...
Nov 22 12:31:09.502125778 (316h27m8.053596239s) [noise=true , address=145.79.227.16:4909 , piece_id=ZTL4ER5YJWSDGO5F5KLRUL5WRTA6WKVOJOERIBTRIIVEWXUY5Z2Q, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.503083742 (316h27m8.054554193s) [noise=true , address=145.79.227.20:4437 , piece_id=CZ32MB4AEZZ6YKCNWKRYALVAK7MT3KNXDSZCQTPUV2Q53NX6Y3BQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.517732716 (316h27m8.069203177s) [noise=true , address=145.79.227.25:4618 , piece_id=CV6M3CTPQR2MPQZFU6W34MFQERT6MMN3RRF2N3W35FGI464OSUUA, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.518029582 (316h27m8.069500043s) [noise=true , address=145.79.227.28:4726 , piece_id=3LBJV2B4CVXEMHCOPRW7DR5QBKPS6DFMPTHQSVNR4XRTZZVIGBQA, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.519935051 (316h27m8.071405512s) [noise=true , address=145.79.227.26:4654 , piece_id=EM4RRGMPPNVR7VTXP3BED43FJFIUDZYVT4S5OUK2TRXZHYNMQ64Q, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.520607661 (316h27m8.072078122s) [noise=true , address=145.79.227.27:4689 , piece_id=76S65KEZ7I2E76RR2JXOZ3ONGXCWSJA5XB7XQVFEEKGGNC34Z4EQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.521187736 (316h27m8.072658197s) [noise=true , address=145.79.227.23:4546 , piece_id=73UPPN2SYWNWEVWHCF4BRGKNO5GCRGOV3S5JZPX4MW75EJZRQOWQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.532974760 (316h27m8.084445221s) [noise=true , address=145.79.227.24:4583 , piece_id=MLW4X34EWXKPOCFVI4VIQSL6RMUOJ5HRYHQ4HY35JIFT4ZFJ6F3Q, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.533688800 (316h27m8.085159271s) [noise=true , address=145.79.227.6:4258 , piece_id=B5D4TCVJORSAWDAU4DQ5VAV4CCRF5WUI24DUVUORAEKLAWGI7ABQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.534460109 (316h27m8.08593057s) [noise=true , address=145.79.227.15:4874 , piece_id=TYAXIF3ZJQQSFGFANYF56DG4O5WBBXK4M7OT67E7ARYNZ663HESA, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.542955375 (316h27m8.094425826s) [noise=true , address=145.79.227.17:4972 , piece_id=UZK7LCIOBGGDNJZUVDBOLBJHGOIA4SUT2O3PD67PYQD3J2DGBJEA, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.563098982 (316h27m8.114569443s) [noise=true , address=145.79.227.21:4474 , piece_id=PCRWYQVCAZCPOILAEDHJGDBFKX4GEJ6JXCF7FUS7OVFRUTDPJI4Q, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:36.585475346 (316h27m35.136945807s) [noise=true , address=45.91.116.10:4082 , piece_id=GBDYXAWONOHQGBGM3NCRQYDSPFOAIHRJHBB2W6ZGJLMZUJLV65XA, seg_pos={0 0}, upload=278701]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: read tcp 188.166.225.170:54326->45.91.116.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:36.585499923 (316h27m35.136970384s) [noise=true , address=77.90.31.21:4879 , piece_id=GHTH55OGGXBAZZ5FGPCDYVHDYEETM7DNVTPSTTFFFRZMKZZ3XRXQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: rpc: dial tcp 77.90.31.21:4879: i/o timeout; context deadline exceeded
Nov 22 12:31:36.585749648 (316h27m35.137220119s) [noise=true , address=77.90.31.13:4627 , piece_id=Q7U5BFXHYQBBJQCU4EQGAR75WKL5QMBJ3LO34UYGO7QHLMDJBKWQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:36.585848086 (316h27m35.137318547s) [noise=true , address=45.91.116.7:4009 , piece_id=FXQGF3UQN5CPFBS2HWD7BAXR4QIOF42MMK3VJQ7AK4THYJWOFFLQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:36.585900936 (316h27m35.137371397s) [seg_pos={0 0}, upload=278701]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Nov 22 12:31:36.828881025 (316h27m35.380351486s) [upload=278701]: Done uploading segments...
Nov 22 12:31:36.828915801 (316h27m35.380386262s) [upload=278701]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 5.782614565s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [...X...X....................................................] duration:
5.941584799s trace:
https://jaeger.dev.storj.io/trace/570ccd0ce1345814
FAIL (2025-11-22 13:07:55 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.941600591s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [.........X..X.XX............................................] duration:
3.503502171s trace:
https://jaeger.dev.storj.io/trace/71781384eb27b231
FAIL (2025-11-22 13:07:51 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.503581443s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [........X.....X.............................................] duration:
5.714912467s trace:
https://jaeger.dev.storj.io/trace/6c800b4c3a84b58c
FAIL (2025-11-22 13:07:44 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 5)
======LOGS======
N5RGYNBMYFXXBJDBTUCN37EAT7WBRNTX2YQA, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.890374814 (316h29m10.441845275s) [noise=true , address=145.79.227.32:4858 , piece_id=K5L3LEUKZWHS36XJHNNY4TYRXCP4DG5UAKCZXPJJTSG26CKUI4HQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.890535981 (316h29m10.442006442s) [noise=true , address=77.90.31.21:4890 , piece_id=DUEFPJUTYMD564Z4III63LZM6L4IL2ZTGM3QDZ6CG7WV5APHBNRA, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.890774064 (316h29m10.442244525s) [noise=true , address=45.91.116.14:4213 , piece_id=HPFISTE2N3B4RBDPAZAGVZJ5WYLCPHTCYH4WMJ56IWEHR3ZBEMOQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.890934851 (316h29m10.442405312s) [noise=true , address=45.91.116.6:4000 , piece_id=T4CI3KKNMFPT3ZIODHE7BQ4DVHVH2QTSO6OB7QZPL5SNPXQJE6TA, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.891167002 (316h29m10.442637463s) [noise=true , address=77.90.31.17:4787 , piece_id=IYIEVH6CARRYCARKOLKWVBICQDNDRCTGQMWF3VJZ6YBC3GO5OWHQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.891369828 (316h29m10.442840289s) [noise=true , address=45.91.116.7:4039 , piece_id=HSRY63FXTEPCP2T46V77RVSC24GGRBAYLLU7DTKOLWGVOYHF6YVA, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.891556875 (316h29m10.443027336s) [noise=true , address=77.90.31.11:4572 , piece_id=G4XSIPSWOLSEOGGFA2NLOSDCGTLZPI3P5LDNNLB5RGMU5OQGWXGQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.891896812 (316h29m10.443367273s) [noise=true , address=77.90.31.19:4863 , piece_id=3PM3Q6X3ADKX2RKIIFDMK57LTJNJHNFK4UYPKFK3ZEQLYQ23EP3A, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.892019235 (316h29m10.443489696s) [noise=true , address=77.90.31.10:4524 , piece_id=Z2IGDHPDU6JAR5CAFQBGYHO3TQ66K4BZOAMXSC7X3RFWLGHOOMTQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.892316521 (316h29m10.443786982s) [noise=true , address=77.90.31.13:4644 , piece_id=UPDZGC3CMJXA3AOJX65VLASU37NLFGPIJLCAPN32TEQB5QHTU7EQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.892682318 (316h29m10.444152779s) [noise=true , address=45.91.116.9:4274 , piece_id=MXWRAXYLBBGLUVTA5GP3JVX2IRM2UI2IC35D7KX3XMFEBXLIDX5A, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:13.353758506 (316h29m11.905228957s) [upload=278731]: Next returned nil segment
Nov 22 12:33:13.353772043 (316h29m11.905242504s) [upload=278731]: Waiting for error group managing segments...
Nov 22 12:33:13.751703718 (316h29m12.303174169s) [noise=true , address=145.79.227.32:4858 , piece_id=K5L3LEUKZWHS36XJHNNY4TYRXCP4DG5UAKCZXPJJTSG26CKUI4HQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil>
Nov 22 12:33:13.753660293 (316h29m12.305130764s) [noise=true , address=145.79.227.26:4674 , piece_id=UP7VIVRDRL6KERJSX3VLWTKGW3ESGCQ7BLYCBXGBB5UNIDE6OHQQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil>
Nov 22 12:33:13.760046123 (316h29m12.311516574s) [noise=true , address=145.79.227.3:4167 , piece_id=4STIVXGU7SOXWAIPG2PXWBLNDD7EYQ56E2DBN2UZ2AZ3TZSXCTIQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil>
Nov 22 12:33:57.373975014 (316h29m55.925445475s) [noise=true , address=77.90.31.13:4644 , piece_id=UPDZGC3CMJXA3AOJX65VLASU37NLFGPIJLCAPN32TEQB5QHTU7EQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil>
Nov 22 12:34:06.122275078 (316h30m4.673745539s) [noise=true , address=45.91.116.13:4181 , piece_id=MQNJEBQFZMHNDKAH2K3JEA7YBK2UK5BGSNCJF3MASF3CKVA5VD6Q, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil>
Nov 22 12:34:10.949171331 (316h30m9.500641792s) [noise=true , address=77.90.31.21:4890 , piece_id=DUEFPJUTYMD564Z4III63LZM6L4IL2ZTGM3QDZ6CG7WV5APHBNRA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1iE3mDnEhGDsMBtmUXC4EwvSrpyHLoybzVXkC9eoFuYEMHeBrd, address:77.90.31.21:4890): protocol: read tcp 188.166.225.170:55900->77.90.31.21:4890: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.949753650 (316h30m9.501224111s) [noise=true , address=45.91.116.12:4237 , piece_id=OG6XREYEGOYVFFMXCGBERRAIEMI2QMUGSBJBYP7GJQZ7FOAI4BIA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1FMjHdNGV42a9QZyMmSPTJWQrA8CdSRvXqrimnt6UGVreCzmgF, address:45.91.116.12:4237): protocol: rpc: dial tcp 45.91.116.12:4237: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.950066235 (316h30m9.501536706s) [noise=true , address=77.90.31.19:4863 , piece_id=3PM3Q6X3ADKX2RKIIFDMK57LTJNJHNFK4UYPKFK3ZEQLYQ23EP3A, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:124x1cbCPJRK8YkgSP6UzHymoLu1E1znnfg5WBiD89kSDLi8JeL, address:77.90.31.19:4863): protocol: write tcp 188.166.225.170:45968->77.90.31.19:4863: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.950246558 (316h30m9.50171702s) [noise=true , address=77.90.31.10:4524 , piece_id=Z2IGDHPDU6JAR5CAFQBGYHO3TQ66K4BZOAMXSC7X3RFWLGHOOMTQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12uN7CznoK4AD8FqFkNUndawdGJ9rmjAe7oNnjJkV844ScY87qe, address:77.90.31.10:4524): protocol: write tcp 188.166.225.170:40266->77.90.31.10:4524: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.950593058 (316h30m9.502063529s) [noise=true , address=77.90.31.14:4690 , piece_id=RJ3KXY3HURT7Q4HZYTOLSAL7Y6VMDGGRIQP6XHE4YB6OAA3PJRAA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.950853535 (316h30m9.502324006s) [noise=true , address=45.91.116.14:4213 , piece_id=HPFISTE2N3B4RBDPAZAGVZJ5WYLCPHTCYH4WMJ56IWEHR3ZBEMOQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12tQRqowH4aDr9h3s1WLs1o7na1ueiafwgUeeHfAB6g7mcE6e1t, address:45.91.116.14:4213): protocol: rpc: dial tcp 45.91.116.14:4213: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.951186438 (316h30m9.502656889s) [noise=true , address=45.91.116.6:4000 , piece_id=T4CI3KKNMFPT3ZIODHE7BQ4DVHVH2QTSO6OB7QZPL5SNPXQJE6TA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12CuCqHJYbanbuaEMHvtcuQXhvcdahD8eyT2fKDhqtxdSZ6ES4f, address:45.91.116.6:4000): protocol: rpc: dial tcp 45.91.116.6:4000: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.951409864 (316h30m9.502880325s) [noise=true , address=45.91.116.7:4039 , piece_id=HSRY63FXTEPCP2T46V77RVSC24GGRBAYLLU7DTKOLWGVOYHF6YVA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:16oq1Yv7JfHrg4gdHfhwuMj4UsWeQQPayKV6Pj1NY2CgJhuPr2, address:45.91.116.7:4039): protocol: write tcp 188.166.225.170:36904->45.91.116.7:4039: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.951660191 (316h30m9.503130652s) [noise=true , address=77.90.31.11:4572 , piece_id=G4XSIPSWOLSEOGGFA2NLOSDCGTLZPI3P5LDNNLB5RGMU5OQGWXGQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1ZAfYZ4wgpZVvatj6wtHona3hTaPcLxAmXC1DT2uKQyo8nLQQ7, address:77.90.31.11:4572): protocol: rpc: dial tcp 77.90.31.11:4572: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.952019855 (316h30m9.503490316s) [noise=true , address=45.91.116.9:4274 , piece_id=MXWRAXYLBBGLUVTA5GP3JVX2IRM2UI2IC35D7KX3XMFEBXLIDX5A, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12g6jFa38dTz5UdgsQaEuM2MSH1H7cxaysWsVmbHkM2Dd2W2iDb, address:45.91.116.9:4274): protocol: rpc: dial tcp 45.91.116.9:4274: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.952290972 (316h30m9.503761443s) [noise=true , address=77.90.31.15:4720 , piece_id=BDQUF6C5TT3ASE3DIN6SSVJHOZAJK4ZEW33OIYPTGB4LDVRELSUA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12VbsxFqMB3vVphSAJqYpbBzua7Goq45E6ovJSaExkpLtfkcc74, address:77.90.31.15:4720): protocol: rpc: dial tcp 77.90.31.15:4720: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.952544654 (316h30m9.504015105s) [noise=true , address=45.91.116.10:4110 , piece_id=H6FYA6K4N4AQYU3MN5RGYNBMYFXXBJDBTUCN37EAT7WBRNTX2YQA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:123KwusATu1ZKtNNVHZZZBE67ZEnVb8puyfQidBi3fEWREqdf1h, address:45.91.116.10:4110): protocol: rpc: dial tcp 45.91.116.10:4110: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.953548155 (316h30m9.505018617s) [noise=true , address=77.90.31.17:4787 , piece_id=IYIEVH6CARRYCARKOLKWVBICQDNDRCTGQMWF3VJZ6YBC3GO5OWHQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1Lietonb5aVbaZQt6wQ2EoxwJkSEn9oL9Dwyrc9mTyomEKikNt, address:77.90.31.17:4787): protocol: read tcp 188.166.225.170:57908->77.90.31.17:4787: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.953579816 (316h30m9.505050277s) [seg_pos={0 0}, upload=278731]: Done waiting for segment. successful: 5 optimal: 14 errs: <nil>
Nov 22 12:34:11.195719123 (316h30m9.747189584s) [upload=278731]: Done uploading segments...
Nov 22 12:34:11.195932840 (316h30m9.747403301s) [upload=278731]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 5)
Duration: 5.71495042s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [.........X.....X............................................] duration:
5.599490074s trace:
https://jaeger.dev.storj.io/trace/6c800b4c3a84b58c
FAIL (2025-11-22 13:07:27 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 5)
======LOGS======
N5RGYNBMYFXXBJDBTUCN37EAT7WBRNTX2YQA, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.890374814 (316h29m10.441845275s) [noise=true , address=145.79.227.32:4858 , piece_id=K5L3LEUKZWHS36XJHNNY4TYRXCP4DG5UAKCZXPJJTSG26CKUI4HQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.890535981 (316h29m10.442006442s) [noise=true , address=77.90.31.21:4890 , piece_id=DUEFPJUTYMD564Z4III63LZM6L4IL2ZTGM3QDZ6CG7WV5APHBNRA, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.890774064 (316h29m10.442244525s) [noise=true , address=45.91.116.14:4213 , piece_id=HPFISTE2N3B4RBDPAZAGVZJ5WYLCPHTCYH4WMJ56IWEHR3ZBEMOQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.890934851 (316h29m10.442405312s) [noise=true , address=45.91.116.6:4000 , piece_id=T4CI3KKNMFPT3ZIODHE7BQ4DVHVH2QTSO6OB7QZPL5SNPXQJE6TA, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.891167002 (316h29m10.442637463s) [noise=true , address=77.90.31.17:4787 , piece_id=IYIEVH6CARRYCARKOLKWVBICQDNDRCTGQMWF3VJZ6YBC3GO5OWHQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.891369828 (316h29m10.442840289s) [noise=true , address=45.91.116.7:4039 , piece_id=HSRY63FXTEPCP2T46V77RVSC24GGRBAYLLU7DTKOLWGVOYHF6YVA, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.891556875 (316h29m10.443027336s) [noise=true , address=77.90.31.11:4572 , piece_id=G4XSIPSWOLSEOGGFA2NLOSDCGTLZPI3P5LDNNLB5RGMU5OQGWXGQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.891896812 (316h29m10.443367273s) [noise=true , address=77.90.31.19:4863 , piece_id=3PM3Q6X3ADKX2RKIIFDMK57LTJNJHNFK4UYPKFK3ZEQLYQ23EP3A, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.892019235 (316h29m10.443489696s) [noise=true , address=77.90.31.10:4524 , piece_id=Z2IGDHPDU6JAR5CAFQBGYHO3TQ66K4BZOAMXSC7X3RFWLGHOOMTQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.892316521 (316h29m10.443786982s) [noise=true , address=77.90.31.13:4644 , piece_id=UPDZGC3CMJXA3AOJX65VLASU37NLFGPIJLCAPN32TEQB5QHTU7EQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.892682318 (316h29m10.444152779s) [noise=true , address=45.91.116.9:4274 , piece_id=MXWRAXYLBBGLUVTA5GP3JVX2IRM2UI2IC35D7KX3XMFEBXLIDX5A, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:13.353758506 (316h29m11.905228957s) [upload=278731]: Next returned nil segment
Nov 22 12:33:13.353772043 (316h29m11.905242504s) [upload=278731]: Waiting for error group managing segments...
Nov 22 12:33:13.751703718 (316h29m12.303174169s) [noise=true , address=145.79.227.32:4858 , piece_id=K5L3LEUKZWHS36XJHNNY4TYRXCP4DG5UAKCZXPJJTSG26CKUI4HQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil>
Nov 22 12:33:13.753660293 (316h29m12.305130764s) [noise=true , address=145.79.227.26:4674 , piece_id=UP7VIVRDRL6KERJSX3VLWTKGW3ESGCQ7BLYCBXGBB5UNIDE6OHQQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil>
Nov 22 12:33:13.760046123 (316h29m12.311516574s) [noise=true , address=145.79.227.3:4167 , piece_id=4STIVXGU7SOXWAIPG2PXWBLNDD7EYQ56E2DBN2UZ2AZ3TZSXCTIQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil>
Nov 22 12:33:57.373975014 (316h29m55.925445475s) [noise=true , address=77.90.31.13:4644 , piece_id=UPDZGC3CMJXA3AOJX65VLASU37NLFGPIJLCAPN32TEQB5QHTU7EQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil>
Nov 22 12:34:06.122275078 (316h30m4.673745539s) [noise=true , address=45.91.116.13:4181 , piece_id=MQNJEBQFZMHNDKAH2K3JEA7YBK2UK5BGSNCJF3MASF3CKVA5VD6Q, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil>
Nov 22 12:34:10.949171331 (316h30m9.500641792s) [noise=true , address=77.90.31.21:4890 , piece_id=DUEFPJUTYMD564Z4III63LZM6L4IL2ZTGM3QDZ6CG7WV5APHBNRA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1iE3mDnEhGDsMBtmUXC4EwvSrpyHLoybzVXkC9eoFuYEMHeBrd, address:77.90.31.21:4890): protocol: read tcp 188.166.225.170:55900->77.90.31.21:4890: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.949753650 (316h30m9.501224111s) [noise=true , address=45.91.116.12:4237 , piece_id=OG6XREYEGOYVFFMXCGBERRAIEMI2QMUGSBJBYP7GJQZ7FOAI4BIA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1FMjHdNGV42a9QZyMmSPTJWQrA8CdSRvXqrimnt6UGVreCzmgF, address:45.91.116.12:4237): protocol: rpc: dial tcp 45.91.116.12:4237: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.950066235 (316h30m9.501536706s) [noise=true , address=77.90.31.19:4863 , piece_id=3PM3Q6X3ADKX2RKIIFDMK57LTJNJHNFK4UYPKFK3ZEQLYQ23EP3A, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:124x1cbCPJRK8YkgSP6UzHymoLu1E1znnfg5WBiD89kSDLi8JeL, address:77.90.31.19:4863): protocol: write tcp 188.166.225.170:45968->77.90.31.19:4863: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.950246558 (316h30m9.50171702s) [noise=true , address=77.90.31.10:4524 , piece_id=Z2IGDHPDU6JAR5CAFQBGYHO3TQ66K4BZOAMXSC7X3RFWLGHOOMTQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12uN7CznoK4AD8FqFkNUndawdGJ9rmjAe7oNnjJkV844ScY87qe, address:77.90.31.10:4524): protocol: write tcp 188.166.225.170:40266->77.90.31.10:4524: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.950593058 (316h30m9.502063529s) [noise=true , address=77.90.31.14:4690 , piece_id=RJ3KXY3HURT7Q4HZYTOLSAL7Y6VMDGGRIQP6XHE4YB6OAA3PJRAA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.950853535 (316h30m9.502324006s) [noise=true , address=45.91.116.14:4213 , piece_id=HPFISTE2N3B4RBDPAZAGVZJ5WYLCPHTCYH4WMJ56IWEHR3ZBEMOQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12tQRqowH4aDr9h3s1WLs1o7na1ueiafwgUeeHfAB6g7mcE6e1t, address:45.91.116.14:4213): protocol: rpc: dial tcp 45.91.116.14:4213: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.951186438 (316h30m9.502656889s) [noise=true , address=45.91.116.6:4000 , piece_id=T4CI3KKNMFPT3ZIODHE7BQ4DVHVH2QTSO6OB7QZPL5SNPXQJE6TA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12CuCqHJYbanbuaEMHvtcuQXhvcdahD8eyT2fKDhqtxdSZ6ES4f, address:45.91.116.6:4000): protocol: rpc: dial tcp 45.91.116.6:4000: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.951409864 (316h30m9.502880325s) [noise=true , address=45.91.116.7:4039 , piece_id=HSRY63FXTEPCP2T46V77RVSC24GGRBAYLLU7DTKOLWGVOYHF6YVA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:16oq1Yv7JfHrg4gdHfhwuMj4UsWeQQPayKV6Pj1NY2CgJhuPr2, address:45.91.116.7:4039): protocol: write tcp 188.166.225.170:36904->45.91.116.7:4039: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.951660191 (316h30m9.503130652s) [noise=true , address=77.90.31.11:4572 , piece_id=G4XSIPSWOLSEOGGFA2NLOSDCGTLZPI3P5LDNNLB5RGMU5OQGWXGQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1ZAfYZ4wgpZVvatj6wtHona3hTaPcLxAmXC1DT2uKQyo8nLQQ7, address:77.90.31.11:4572): protocol: rpc: dial tcp 77.90.31.11:4572: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.952019855 (316h30m9.503490316s) [noise=true , address=45.91.116.9:4274 , piece_id=MXWRAXYLBBGLUVTA5GP3JVX2IRM2UI2IC35D7KX3XMFEBXLIDX5A, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12g6jFa38dTz5UdgsQaEuM2MSH1H7cxaysWsVmbHkM2Dd2W2iDb, address:45.91.116.9:4274): protocol: rpc: dial tcp 45.91.116.9:4274: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.952290972 (316h30m9.503761443s) [noise=true , address=77.90.31.15:4720 , piece_id=BDQUF6C5TT3ASE3DIN6SSVJHOZAJK4ZEW33OIYPTGB4LDVRELSUA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12VbsxFqMB3vVphSAJqYpbBzua7Goq45E6ovJSaExkpLtfkcc74, address:77.90.31.15:4720): protocol: rpc: dial tcp 77.90.31.15:4720: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.952544654 (316h30m9.504015105s) [noise=true , address=45.91.116.10:4110 , piece_id=H6FYA6K4N4AQYU3MN5RGYNBMYFXXBJDBTUCN37EAT7WBRNTX2YQA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:123KwusATu1ZKtNNVHZZZBE67ZEnVb8puyfQidBi3fEWREqdf1h, address:45.91.116.10:4110): protocol: rpc: dial tcp 45.91.116.10:4110: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.953548155 (316h30m9.505018617s) [noise=true , address=77.90.31.17:4787 , piece_id=IYIEVH6CARRYCARKOLKWVBICQDNDRCTGQMWF3VJZ6YBC3GO5OWHQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1Lietonb5aVbaZQt6wQ2EoxwJkSEn9oL9Dwyrc9mTyomEKikNt, address:77.90.31.17:4787): protocol: read tcp 188.166.225.170:57908->77.90.31.17:4787: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.953579816 (316h30m9.505050277s) [seg_pos={0 0}, upload=278731]: Done waiting for segment. successful: 5 optimal: 14 errs: <nil>
Nov 22 12:34:11.195719123 (316h30m9.747189584s) [upload=278731]: Done uploading segments...
Nov 22 12:34:11.195932840 (316h30m9.747403301s) [upload=278731]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 5)
Duration: 5.599508138s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [....X...X...................................................] duration:
5.766006779s trace:
https://jaeger.dev.storj.io/trace/570ccd0ce1345814
FAIL (2025-11-22 13:07:19 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.766037117s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [..........X.....X...........................................] duration:
5.624967726s trace:
https://jaeger.dev.storj.io/trace/6c800b4c3a84b58c
FAIL (2025-11-22 13:07:09 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 5)
======LOGS======
N5RGYNBMYFXXBJDBTUCN37EAT7WBRNTX2YQA, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.890374814 (316h29m10.441845275s) [noise=true , address=145.79.227.32:4858 , piece_id=K5L3LEUKZWHS36XJHNNY4TYRXCP4DG5UAKCZXPJJTSG26CKUI4HQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.890535981 (316h29m10.442006442s) [noise=true , address=77.90.31.21:4890 , piece_id=DUEFPJUTYMD564Z4III63LZM6L4IL2ZTGM3QDZ6CG7WV5APHBNRA, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.890774064 (316h29m10.442244525s) [noise=true , address=45.91.116.14:4213 , piece_id=HPFISTE2N3B4RBDPAZAGVZJ5WYLCPHTCYH4WMJ56IWEHR3ZBEMOQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.890934851 (316h29m10.442405312s) [noise=true , address=45.91.116.6:4000 , piece_id=T4CI3KKNMFPT3ZIODHE7BQ4DVHVH2QTSO6OB7QZPL5SNPXQJE6TA, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.891167002 (316h29m10.442637463s) [noise=true , address=77.90.31.17:4787 , piece_id=IYIEVH6CARRYCARKOLKWVBICQDNDRCTGQMWF3VJZ6YBC3GO5OWHQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.891369828 (316h29m10.442840289s) [noise=true , address=45.91.116.7:4039 , piece_id=HSRY63FXTEPCP2T46V77RVSC24GGRBAYLLU7DTKOLWGVOYHF6YVA, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.891556875 (316h29m10.443027336s) [noise=true , address=77.90.31.11:4572 , piece_id=G4XSIPSWOLSEOGGFA2NLOSDCGTLZPI3P5LDNNLB5RGMU5OQGWXGQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.891896812 (316h29m10.443367273s) [noise=true , address=77.90.31.19:4863 , piece_id=3PM3Q6X3ADKX2RKIIFDMK57LTJNJHNFK4UYPKFK3ZEQLYQ23EP3A, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.892019235 (316h29m10.443489696s) [noise=true , address=77.90.31.10:4524 , piece_id=Z2IGDHPDU6JAR5CAFQBGYHO3TQ66K4BZOAMXSC7X3RFWLGHOOMTQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.892316521 (316h29m10.443786982s) [noise=true , address=77.90.31.13:4644 , piece_id=UPDZGC3CMJXA3AOJX65VLASU37NLFGPIJLCAPN32TEQB5QHTU7EQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.892682318 (316h29m10.444152779s) [noise=true , address=45.91.116.9:4274 , piece_id=MXWRAXYLBBGLUVTA5GP3JVX2IRM2UI2IC35D7KX3XMFEBXLIDX5A, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:13.353758506 (316h29m11.905228957s) [upload=278731]: Next returned nil segment
Nov 22 12:33:13.353772043 (316h29m11.905242504s) [upload=278731]: Waiting for error group managing segments...
Nov 22 12:33:13.751703718 (316h29m12.303174169s) [noise=true , address=145.79.227.32:4858 , piece_id=K5L3LEUKZWHS36XJHNNY4TYRXCP4DG5UAKCZXPJJTSG26CKUI4HQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil>
Nov 22 12:33:13.753660293 (316h29m12.305130764s) [noise=true , address=145.79.227.26:4674 , piece_id=UP7VIVRDRL6KERJSX3VLWTKGW3ESGCQ7BLYCBXGBB5UNIDE6OHQQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil>
Nov 22 12:33:13.760046123 (316h29m12.311516574s) [noise=true , address=145.79.227.3:4167 , piece_id=4STIVXGU7SOXWAIPG2PXWBLNDD7EYQ56E2DBN2UZ2AZ3TZSXCTIQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil>
Nov 22 12:33:57.373975014 (316h29m55.925445475s) [noise=true , address=77.90.31.13:4644 , piece_id=UPDZGC3CMJXA3AOJX65VLASU37NLFGPIJLCAPN32TEQB5QHTU7EQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil>
Nov 22 12:34:06.122275078 (316h30m4.673745539s) [noise=true , address=45.91.116.13:4181 , piece_id=MQNJEBQFZMHNDKAH2K3JEA7YBK2UK5BGSNCJF3MASF3CKVA5VD6Q, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil>
Nov 22 12:34:10.949171331 (316h30m9.500641792s) [noise=true , address=77.90.31.21:4890 , piece_id=DUEFPJUTYMD564Z4III63LZM6L4IL2ZTGM3QDZ6CG7WV5APHBNRA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1iE3mDnEhGDsMBtmUXC4EwvSrpyHLoybzVXkC9eoFuYEMHeBrd, address:77.90.31.21:4890): protocol: read tcp 188.166.225.170:55900->77.90.31.21:4890: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.949753650 (316h30m9.501224111s) [noise=true , address=45.91.116.12:4237 , piece_id=OG6XREYEGOYVFFMXCGBERRAIEMI2QMUGSBJBYP7GJQZ7FOAI4BIA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1FMjHdNGV42a9QZyMmSPTJWQrA8CdSRvXqrimnt6UGVreCzmgF, address:45.91.116.12:4237): protocol: rpc: dial tcp 45.91.116.12:4237: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.950066235 (316h30m9.501536706s) [noise=true , address=77.90.31.19:4863 , piece_id=3PM3Q6X3ADKX2RKIIFDMK57LTJNJHNFK4UYPKFK3ZEQLYQ23EP3A, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:124x1cbCPJRK8YkgSP6UzHymoLu1E1znnfg5WBiD89kSDLi8JeL, address:77.90.31.19:4863): protocol: write tcp 188.166.225.170:45968->77.90.31.19:4863: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.950246558 (316h30m9.50171702s) [noise=true , address=77.90.31.10:4524 , piece_id=Z2IGDHPDU6JAR5CAFQBGYHO3TQ66K4BZOAMXSC7X3RFWLGHOOMTQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12uN7CznoK4AD8FqFkNUndawdGJ9rmjAe7oNnjJkV844ScY87qe, address:77.90.31.10:4524): protocol: write tcp 188.166.225.170:40266->77.90.31.10:4524: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.950593058 (316h30m9.502063529s) [noise=true , address=77.90.31.14:4690 , piece_id=RJ3KXY3HURT7Q4HZYTOLSAL7Y6VMDGGRIQP6XHE4YB6OAA3PJRAA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.950853535 (316h30m9.502324006s) [noise=true , address=45.91.116.14:4213 , piece_id=HPFISTE2N3B4RBDPAZAGVZJ5WYLCPHTCYH4WMJ56IWEHR3ZBEMOQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12tQRqowH4aDr9h3s1WLs1o7na1ueiafwgUeeHfAB6g7mcE6e1t, address:45.91.116.14:4213): protocol: rpc: dial tcp 45.91.116.14:4213: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.951186438 (316h30m9.502656889s) [noise=true , address=45.91.116.6:4000 , piece_id=T4CI3KKNMFPT3ZIODHE7BQ4DVHVH2QTSO6OB7QZPL5SNPXQJE6TA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12CuCqHJYbanbuaEMHvtcuQXhvcdahD8eyT2fKDhqtxdSZ6ES4f, address:45.91.116.6:4000): protocol: rpc: dial tcp 45.91.116.6:4000: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.951409864 (316h30m9.502880325s) [noise=true , address=45.91.116.7:4039 , piece_id=HSRY63FXTEPCP2T46V77RVSC24GGRBAYLLU7DTKOLWGVOYHF6YVA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:16oq1Yv7JfHrg4gdHfhwuMj4UsWeQQPayKV6Pj1NY2CgJhuPr2, address:45.91.116.7:4039): protocol: write tcp 188.166.225.170:36904->45.91.116.7:4039: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.951660191 (316h30m9.503130652s) [noise=true , address=77.90.31.11:4572 , piece_id=G4XSIPSWOLSEOGGFA2NLOSDCGTLZPI3P5LDNNLB5RGMU5OQGWXGQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1ZAfYZ4wgpZVvatj6wtHona3hTaPcLxAmXC1DT2uKQyo8nLQQ7, address:77.90.31.11:4572): protocol: rpc: dial tcp 77.90.31.11:4572: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.952019855 (316h30m9.503490316s) [noise=true , address=45.91.116.9:4274 , piece_id=MXWRAXYLBBGLUVTA5GP3JVX2IRM2UI2IC35D7KX3XMFEBXLIDX5A, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12g6jFa38dTz5UdgsQaEuM2MSH1H7cxaysWsVmbHkM2Dd2W2iDb, address:45.91.116.9:4274): protocol: rpc: dial tcp 45.91.116.9:4274: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.952290972 (316h30m9.503761443s) [noise=true , address=77.90.31.15:4720 , piece_id=BDQUF6C5TT3ASE3DIN6SSVJHOZAJK4ZEW33OIYPTGB4LDVRELSUA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12VbsxFqMB3vVphSAJqYpbBzua7Goq45E6ovJSaExkpLtfkcc74, address:77.90.31.15:4720): protocol: rpc: dial tcp 77.90.31.15:4720: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.952544654 (316h30m9.504015105s) [noise=true , address=45.91.116.10:4110 , piece_id=H6FYA6K4N4AQYU3MN5RGYNBMYFXXBJDBTUCN37EAT7WBRNTX2YQA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:123KwusATu1ZKtNNVHZZZBE67ZEnVb8puyfQidBi3fEWREqdf1h, address:45.91.116.10:4110): protocol: rpc: dial tcp 45.91.116.10:4110: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.953548155 (316h30m9.505018617s) [noise=true , address=77.90.31.17:4787 , piece_id=IYIEVH6CARRYCARKOLKWVBICQDNDRCTGQMWF3VJZ6YBC3GO5OWHQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1Lietonb5aVbaZQt6wQ2EoxwJkSEn9oL9Dwyrc9mTyomEKikNt, address:77.90.31.17:4787): protocol: read tcp 188.166.225.170:57908->77.90.31.17:4787: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.953579816 (316h30m9.505050277s) [seg_pos={0 0}, upload=278731]: Done waiting for segment. successful: 5 optimal: 14 errs: <nil>
Nov 22 12:34:11.195719123 (316h30m9.747189584s) [upload=278731]: Done uploading segments...
Nov 22 12:34:11.195932840 (316h30m9.747403301s) [upload=278731]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 5)
Duration: 5.625004567s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [..........X..X.XX...........................................] duration:
3.542147529s trace:
https://jaeger.dev.storj.io/trace/71781384eb27b231
FAIL (2025-11-22 13:07:07 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.542179389s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.....X..X..X................................................] duration:
6.195371191s trace:
https://jaeger.dev.storj.io/trace/92df32505639913
FAIL (2025-11-22 13:07:00 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.195389026s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [........X...................................................] duration:
5.689489618s trace:
https://jaeger.dev.storj.io/trace/5ae3d2bf80495000
FAIL (2025-11-22 13:06:56 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Nov 22 12:31:07.276308940 (316h27m5.827779401s) [upload=278701]: Starting upload
Nov 22 12:31:07.276483523 (316h27m5.827953994s) [upload=278701]: Done starting upload
Nov 22 12:31:07.276562243 (316h27m5.828032704s) [upload=278701]: Uploading segments...
Nov 22 12:31:07.276950593 (316h27m5.828421054s) [upload=278701]: Got next segment. Inline: false
Nov 22 12:31:07.277027639 (316h27m5.8284981s) [upload=278701]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:31:07.277034181 (316h27m5.828504643s) [upload=278701]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:31:07.520212027 (316h27m6.071682488s) [seg_pos={0 0}, upload=278701]: Begin upload segment...
Nov 22 12:31:07.520269256 (316h27m6.071739717s) [seg_pos={0 0}, upload=278701]: Done begin upload segment.
Nov 22 12:31:07.520308992 (316h27m6.071779453s) [noise=true , address=145.79.227.27:4689 , piece_id=76S65KEZ7I2E76RR2JXOZ3ONGXCWSJA5XB7XQVFEEKGGNC34Z4EQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.520426746 (316h27m6.071897207s) [noise=true , address=145.79.227.21:4474 , piece_id=PCRWYQVCAZCPOILAEDHJGDBFKX4GEJ6JXCF7FUS7OVFRUTDPJI4Q, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.521175282 (316h27m6.072645743s) [noise=true , address=145.79.227.17:4972 , piece_id=UZK7LCIOBGGDNJZUVDBOLBJHGOIA4SUT2O3PD67PYQD3J2DGBJEA, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.521527573 (316h27m6.072998044s) [noise=true , address=145.79.227.23:4546 , piece_id=73UPPN2SYWNWEVWHCF4BRGKNO5GCRGOV3S5JZPX4MW75EJZRQOWQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.521694671 (316h27m6.073165132s) [noise=true , address=45.91.116.10:4082 , piece_id=GBDYXAWONOHQGBGM3NCRQYDSPFOAIHRJHBB2W6ZGJLMZUJLV65XA, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.522219410 (316h27m6.073689871s) [noise=true , address=145.79.227.20:4437 , piece_id=CZ32MB4AEZZ6YKCNWKRYALVAK7MT3KNXDSZCQTPUV2Q53NX6Y3BQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.522261240 (316h27m6.073731691s) [noise=true , address=145.79.227.16:4909 , piece_id=ZTL4ER5YJWSDGO5F5KLRUL5WRTA6WKVOJOERIBTRIIVEWXUY5Z2Q, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.522719151 (316h27m6.074189612s) [noise=true , address=145.79.227.26:4654 , piece_id=EM4RRGMPPNVR7VTXP3BED43FJFIUDZYVT4S5OUK2TRXZHYNMQ64Q, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.523017269 (316h27m6.07448772s) [noise=true , address=77.90.31.13:4627 , piece_id=Q7U5BFXHYQBBJQCU4EQGAR75WKL5QMBJ3LO34UYGO7QHLMDJBKWQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.523216949 (316h27m6.07468741s) [noise=true , address=145.79.227.25:4618 , piece_id=CV6M3CTPQR2MPQZFU6W34MFQERT6MMN3RRF2N3W35FGI464OSUUA, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.523744804 (316h27m6.075215265s) [noise=true , address=145.79.227.6:4258 , piece_id=B5D4TCVJORSAWDAU4DQ5VAV4CCRF5WUI24DUVUORAEKLAWGI7ABQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.523828153 (316h27m6.075298614s) [noise=true , address=145.79.227.24:4583 , piece_id=MLW4X34EWXKPOCFVI4VIQSL6RMUOJ5HRYHQ4HY35JIFT4ZFJ6F3Q, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.524255396 (316h27m6.075725858s) [noise=true , address=45.91.116.7:4009 , piece_id=FXQGF3UQN5CPFBS2HWD7BAXR4QIOF42MMK3VJQ7AK4THYJWOFFLQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.524662051 (316h27m6.076132512s) [noise=true , address=145.79.227.15:4874 , piece_id=TYAXIF3ZJQQSFGFANYF56DG4O5WBBXK4M7OT67E7ARYNZ663HESA, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.524746221 (316h27m6.076216682s) [noise=true , address=145.79.227.28:4726 , piece_id=3LBJV2B4CVXEMHCOPRW7DR5QBKPS6DFMPTHQSVNR4XRTZZVIGBQA, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.525227157 (316h27m6.076697608s) [noise=true , address=77.90.31.21:4879 , piece_id=GHTH55OGGXBAZZ5FGPCDYVHDYEETM7DNVTPSTTFFFRZMKZZ3XRXQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:09.134610631 (316h27m7.686081093s) [upload=278701]: Next returned nil segment
Nov 22 12:31:09.134628024 (316h27m7.686098495s) [upload=278701]: Waiting for error group managing segments...
Nov 22 12:31:09.502125778 (316h27m8.053596239s) [noise=true , address=145.79.227.16:4909 , piece_id=ZTL4ER5YJWSDGO5F5KLRUL5WRTA6WKVOJOERIBTRIIVEWXUY5Z2Q, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.503083742 (316h27m8.054554193s) [noise=true , address=145.79.227.20:4437 , piece_id=CZ32MB4AEZZ6YKCNWKRYALVAK7MT3KNXDSZCQTPUV2Q53NX6Y3BQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.517732716 (316h27m8.069203177s) [noise=true , address=145.79.227.25:4618 , piece_id=CV6M3CTPQR2MPQZFU6W34MFQERT6MMN3RRF2N3W35FGI464OSUUA, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.518029582 (316h27m8.069500043s) [noise=true , address=145.79.227.28:4726 , piece_id=3LBJV2B4CVXEMHCOPRW7DR5QBKPS6DFMPTHQSVNR4XRTZZVIGBQA, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.519935051 (316h27m8.071405512s) [noise=true , address=145.79.227.26:4654 , piece_id=EM4RRGMPPNVR7VTXP3BED43FJFIUDZYVT4S5OUK2TRXZHYNMQ64Q, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.520607661 (316h27m8.072078122s) [noise=true , address=145.79.227.27:4689 , piece_id=76S65KEZ7I2E76RR2JXOZ3ONGXCWSJA5XB7XQVFEEKGGNC34Z4EQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.521187736 (316h27m8.072658197s) [noise=true , address=145.79.227.23:4546 , piece_id=73UPPN2SYWNWEVWHCF4BRGKNO5GCRGOV3S5JZPX4MW75EJZRQOWQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.532974760 (316h27m8.084445221s) [noise=true , address=145.79.227.24:4583 , piece_id=MLW4X34EWXKPOCFVI4VIQSL6RMUOJ5HRYHQ4HY35JIFT4ZFJ6F3Q, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.533688800 (316h27m8.085159271s) [noise=true , address=145.79.227.6:4258 , piece_id=B5D4TCVJORSAWDAU4DQ5VAV4CCRF5WUI24DUVUORAEKLAWGI7ABQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.534460109 (316h27m8.08593057s) [noise=true , address=145.79.227.15:4874 , piece_id=TYAXIF3ZJQQSFGFANYF56DG4O5WBBXK4M7OT67E7ARYNZ663HESA, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.542955375 (316h27m8.094425826s) [noise=true , address=145.79.227.17:4972 , piece_id=UZK7LCIOBGGDNJZUVDBOLBJHGOIA4SUT2O3PD67PYQD3J2DGBJEA, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.563098982 (316h27m8.114569443s) [noise=true , address=145.79.227.21:4474 , piece_id=PCRWYQVCAZCPOILAEDHJGDBFKX4GEJ6JXCF7FUS7OVFRUTDPJI4Q, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:36.585475346 (316h27m35.136945807s) [noise=true , address=45.91.116.10:4082 , piece_id=GBDYXAWONOHQGBGM3NCRQYDSPFOAIHRJHBB2W6ZGJLMZUJLV65XA, seg_pos={0 0}, upload=278701]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: read tcp 188.166.225.170:54326->45.91.116.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:36.585499923 (316h27m35.136970384s) [noise=true , address=77.90.31.21:4879 , piece_id=GHTH55OGGXBAZZ5FGPCDYVHDYEETM7DNVTPSTTFFFRZMKZZ3XRXQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: rpc: dial tcp 77.90.31.21:4879: i/o timeout; context deadline exceeded
Nov 22 12:31:36.585749648 (316h27m35.137220119s) [noise=true , address=77.90.31.13:4627 , piece_id=Q7U5BFXHYQBBJQCU4EQGAR75WKL5QMBJ3LO34UYGO7QHLMDJBKWQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:36.585848086 (316h27m35.137318547s) [noise=true , address=45.91.116.7:4009 , piece_id=FXQGF3UQN5CPFBS2HWD7BAXR4QIOF42MMK3VJQ7AK4THYJWOFFLQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:36.585900936 (316h27m35.137371397s) [seg_pos={0 0}, upload=278701]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Nov 22 12:31:36.828881025 (316h27m35.380351486s) [upload=278701]: Done uploading segments...
Nov 22 12:31:36.828915801 (316h27m35.380386262s) [upload=278701]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 5.689521798s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [.....X...X..................................................] duration:
5.612665864s trace:
https://jaeger.dev.storj.io/trace/570ccd0ce1345814
FAIL (2025-11-22 13:06:38 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.612695561s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [...........X.....X..........................................] duration:
5.550049347s trace:
https://jaeger.dev.storj.io/trace/6c800b4c3a84b58c
FAIL (2025-11-22 13:06:31 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 5)
======LOGS======
N5RGYNBMYFXXBJDBTUCN37EAT7WBRNTX2YQA, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.890374814 (316h29m10.441845275s) [noise=true , address=145.79.227.32:4858 , piece_id=K5L3LEUKZWHS36XJHNNY4TYRXCP4DG5UAKCZXPJJTSG26CKUI4HQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.890535981 (316h29m10.442006442s) [noise=true , address=77.90.31.21:4890 , piece_id=DUEFPJUTYMD564Z4III63LZM6L4IL2ZTGM3QDZ6CG7WV5APHBNRA, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.890774064 (316h29m10.442244525s) [noise=true , address=45.91.116.14:4213 , piece_id=HPFISTE2N3B4RBDPAZAGVZJ5WYLCPHTCYH4WMJ56IWEHR3ZBEMOQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.890934851 (316h29m10.442405312s) [noise=true , address=45.91.116.6:4000 , piece_id=T4CI3KKNMFPT3ZIODHE7BQ4DVHVH2QTSO6OB7QZPL5SNPXQJE6TA, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.891167002 (316h29m10.442637463s) [noise=true , address=77.90.31.17:4787 , piece_id=IYIEVH6CARRYCARKOLKWVBICQDNDRCTGQMWF3VJZ6YBC3GO5OWHQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.891369828 (316h29m10.442840289s) [noise=true , address=45.91.116.7:4039 , piece_id=HSRY63FXTEPCP2T46V77RVSC24GGRBAYLLU7DTKOLWGVOYHF6YVA, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.891556875 (316h29m10.443027336s) [noise=true , address=77.90.31.11:4572 , piece_id=G4XSIPSWOLSEOGGFA2NLOSDCGTLZPI3P5LDNNLB5RGMU5OQGWXGQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.891896812 (316h29m10.443367273s) [noise=true , address=77.90.31.19:4863 , piece_id=3PM3Q6X3ADKX2RKIIFDMK57LTJNJHNFK4UYPKFK3ZEQLYQ23EP3A, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.892019235 (316h29m10.443489696s) [noise=true , address=77.90.31.10:4524 , piece_id=Z2IGDHPDU6JAR5CAFQBGYHO3TQ66K4BZOAMXSC7X3RFWLGHOOMTQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.892316521 (316h29m10.443786982s) [noise=true , address=77.90.31.13:4644 , piece_id=UPDZGC3CMJXA3AOJX65VLASU37NLFGPIJLCAPN32TEQB5QHTU7EQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.892682318 (316h29m10.444152779s) [noise=true , address=45.91.116.9:4274 , piece_id=MXWRAXYLBBGLUVTA5GP3JVX2IRM2UI2IC35D7KX3XMFEBXLIDX5A, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:13.353758506 (316h29m11.905228957s) [upload=278731]: Next returned nil segment
Nov 22 12:33:13.353772043 (316h29m11.905242504s) [upload=278731]: Waiting for error group managing segments...
Nov 22 12:33:13.751703718 (316h29m12.303174169s) [noise=true , address=145.79.227.32:4858 , piece_id=K5L3LEUKZWHS36XJHNNY4TYRXCP4DG5UAKCZXPJJTSG26CKUI4HQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil>
Nov 22 12:33:13.753660293 (316h29m12.305130764s) [noise=true , address=145.79.227.26:4674 , piece_id=UP7VIVRDRL6KERJSX3VLWTKGW3ESGCQ7BLYCBXGBB5UNIDE6OHQQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil>
Nov 22 12:33:13.760046123 (316h29m12.311516574s) [noise=true , address=145.79.227.3:4167 , piece_id=4STIVXGU7SOXWAIPG2PXWBLNDD7EYQ56E2DBN2UZ2AZ3TZSXCTIQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil>
Nov 22 12:33:57.373975014 (316h29m55.925445475s) [noise=true , address=77.90.31.13:4644 , piece_id=UPDZGC3CMJXA3AOJX65VLASU37NLFGPIJLCAPN32TEQB5QHTU7EQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil>
Nov 22 12:34:06.122275078 (316h30m4.673745539s) [noise=true , address=45.91.116.13:4181 , piece_id=MQNJEBQFZMHNDKAH2K3JEA7YBK2UK5BGSNCJF3MASF3CKVA5VD6Q, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil>
Nov 22 12:34:10.949171331 (316h30m9.500641792s) [noise=true , address=77.90.31.21:4890 , piece_id=DUEFPJUTYMD564Z4III63LZM6L4IL2ZTGM3QDZ6CG7WV5APHBNRA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1iE3mDnEhGDsMBtmUXC4EwvSrpyHLoybzVXkC9eoFuYEMHeBrd, address:77.90.31.21:4890): protocol: read tcp 188.166.225.170:55900->77.90.31.21:4890: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.949753650 (316h30m9.501224111s) [noise=true , address=45.91.116.12:4237 , piece_id=OG6XREYEGOYVFFMXCGBERRAIEMI2QMUGSBJBYP7GJQZ7FOAI4BIA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1FMjHdNGV42a9QZyMmSPTJWQrA8CdSRvXqrimnt6UGVreCzmgF, address:45.91.116.12:4237): protocol: rpc: dial tcp 45.91.116.12:4237: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.950066235 (316h30m9.501536706s) [noise=true , address=77.90.31.19:4863 , piece_id=3PM3Q6X3ADKX2RKIIFDMK57LTJNJHNFK4UYPKFK3ZEQLYQ23EP3A, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:124x1cbCPJRK8YkgSP6UzHymoLu1E1znnfg5WBiD89kSDLi8JeL, address:77.90.31.19:4863): protocol: write tcp 188.166.225.170:45968->77.90.31.19:4863: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.950246558 (316h30m9.50171702s) [noise=true , address=77.90.31.10:4524 , piece_id=Z2IGDHPDU6JAR5CAFQBGYHO3TQ66K4BZOAMXSC7X3RFWLGHOOMTQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12uN7CznoK4AD8FqFkNUndawdGJ9rmjAe7oNnjJkV844ScY87qe, address:77.90.31.10:4524): protocol: write tcp 188.166.225.170:40266->77.90.31.10:4524: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.950593058 (316h30m9.502063529s) [noise=true , address=77.90.31.14:4690 , piece_id=RJ3KXY3HURT7Q4HZYTOLSAL7Y6VMDGGRIQP6XHE4YB6OAA3PJRAA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.950853535 (316h30m9.502324006s) [noise=true , address=45.91.116.14:4213 , piece_id=HPFISTE2N3B4RBDPAZAGVZJ5WYLCPHTCYH4WMJ56IWEHR3ZBEMOQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12tQRqowH4aDr9h3s1WLs1o7na1ueiafwgUeeHfAB6g7mcE6e1t, address:45.91.116.14:4213): protocol: rpc: dial tcp 45.91.116.14:4213: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.951186438 (316h30m9.502656889s) [noise=true , address=45.91.116.6:4000 , piece_id=T4CI3KKNMFPT3ZIODHE7BQ4DVHVH2QTSO6OB7QZPL5SNPXQJE6TA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12CuCqHJYbanbuaEMHvtcuQXhvcdahD8eyT2fKDhqtxdSZ6ES4f, address:45.91.116.6:4000): protocol: rpc: dial tcp 45.91.116.6:4000: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.951409864 (316h30m9.502880325s) [noise=true , address=45.91.116.7:4039 , piece_id=HSRY63FXTEPCP2T46V77RVSC24GGRBAYLLU7DTKOLWGVOYHF6YVA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:16oq1Yv7JfHrg4gdHfhwuMj4UsWeQQPayKV6Pj1NY2CgJhuPr2, address:45.91.116.7:4039): protocol: write tcp 188.166.225.170:36904->45.91.116.7:4039: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.951660191 (316h30m9.503130652s) [noise=true , address=77.90.31.11:4572 , piece_id=G4XSIPSWOLSEOGGFA2NLOSDCGTLZPI3P5LDNNLB5RGMU5OQGWXGQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1ZAfYZ4wgpZVvatj6wtHona3hTaPcLxAmXC1DT2uKQyo8nLQQ7, address:77.90.31.11:4572): protocol: rpc: dial tcp 77.90.31.11:4572: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.952019855 (316h30m9.503490316s) [noise=true , address=45.91.116.9:4274 , piece_id=MXWRAXYLBBGLUVTA5GP3JVX2IRM2UI2IC35D7KX3XMFEBXLIDX5A, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12g6jFa38dTz5UdgsQaEuM2MSH1H7cxaysWsVmbHkM2Dd2W2iDb, address:45.91.116.9:4274): protocol: rpc: dial tcp 45.91.116.9:4274: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.952290972 (316h30m9.503761443s) [noise=true , address=77.90.31.15:4720 , piece_id=BDQUF6C5TT3ASE3DIN6SSVJHOZAJK4ZEW33OIYPTGB4LDVRELSUA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12VbsxFqMB3vVphSAJqYpbBzua7Goq45E6ovJSaExkpLtfkcc74, address:77.90.31.15:4720): protocol: rpc: dial tcp 77.90.31.15:4720: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.952544654 (316h30m9.504015105s) [noise=true , address=45.91.116.10:4110 , piece_id=H6FYA6K4N4AQYU3MN5RGYNBMYFXXBJDBTUCN37EAT7WBRNTX2YQA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:123KwusATu1ZKtNNVHZZZBE67ZEnVb8puyfQidBi3fEWREqdf1h, address:45.91.116.10:4110): protocol: rpc: dial tcp 45.91.116.10:4110: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.953548155 (316h30m9.505018617s) [noise=true , address=77.90.31.17:4787 , piece_id=IYIEVH6CARRYCARKOLKWVBICQDNDRCTGQMWF3VJZ6YBC3GO5OWHQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1Lietonb5aVbaZQt6wQ2EoxwJkSEn9oL9Dwyrc9mTyomEKikNt, address:77.90.31.17:4787): protocol: read tcp 188.166.225.170:57908->77.90.31.17:4787: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.953579816 (316h30m9.505050277s) [seg_pos={0 0}, upload=278731]: Done waiting for segment. successful: 5 optimal: 14 errs: <nil>
Nov 22 12:34:11.195719123 (316h30m9.747189584s) [upload=278731]: Done uploading segments...
Nov 22 12:34:11.195932840 (316h30m9.747403301s) [upload=278731]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 5)
Duration: 5.550081559s
core
uplink
us-select-1
us1-placement-20
|