Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [.......................................X...X................] duration:
5.945197841s trace:
https://jaeger.dev.storj.io/trace/570ccd0ce1345814
FAIL (2025-11-22 12:42:45 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.945214703s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.........................................X..X..X............] duration:
10.863012096s trace:
https://jaeger.dev.storj.io/trace/92df32505639913
FAIL (2025-11-22 12:42:41 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 10.863041674s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [........................................X...X...............] duration:
5.867337981s trace:
https://jaeger.dev.storj.io/trace/570ccd0ce1345814
FAIL (2025-11-22 12:42:22 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.867687378s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.............................................X..............] duration:
5.616591606s trace:
https://jaeger.dev.storj.io/trace/5ae3d2bf80495000
FAIL (2025-11-22 12:42:11 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Nov 22 12:31:07.276308940 (316h27m5.827779401s) [upload=278701]: Starting upload
Nov 22 12:31:07.276483523 (316h27m5.827953994s) [upload=278701]: Done starting upload
Nov 22 12:31:07.276562243 (316h27m5.828032704s) [upload=278701]: Uploading segments...
Nov 22 12:31:07.276950593 (316h27m5.828421054s) [upload=278701]: Got next segment. Inline: false
Nov 22 12:31:07.277027639 (316h27m5.8284981s) [upload=278701]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:31:07.277034181 (316h27m5.828504643s) [upload=278701]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:31:07.520212027 (316h27m6.071682488s) [seg_pos={0 0}, upload=278701]: Begin upload segment...
Nov 22 12:31:07.520269256 (316h27m6.071739717s) [seg_pos={0 0}, upload=278701]: Done begin upload segment.
Nov 22 12:31:07.520308992 (316h27m6.071779453s) [noise=true , address=145.79.227.27:4689 , piece_id=76S65KEZ7I2E76RR2JXOZ3ONGXCWSJA5XB7XQVFEEKGGNC34Z4EQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.520426746 (316h27m6.071897207s) [noise=true , address=145.79.227.21:4474 , piece_id=PCRWYQVCAZCPOILAEDHJGDBFKX4GEJ6JXCF7FUS7OVFRUTDPJI4Q, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.521175282 (316h27m6.072645743s) [noise=true , address=145.79.227.17:4972 , piece_id=UZK7LCIOBGGDNJZUVDBOLBJHGOIA4SUT2O3PD67PYQD3J2DGBJEA, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.521527573 (316h27m6.072998044s) [noise=true , address=145.79.227.23:4546 , piece_id=73UPPN2SYWNWEVWHCF4BRGKNO5GCRGOV3S5JZPX4MW75EJZRQOWQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.521694671 (316h27m6.073165132s) [noise=true , address=45.91.116.10:4082 , piece_id=GBDYXAWONOHQGBGM3NCRQYDSPFOAIHRJHBB2W6ZGJLMZUJLV65XA, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.522219410 (316h27m6.073689871s) [noise=true , address=145.79.227.20:4437 , piece_id=CZ32MB4AEZZ6YKCNWKRYALVAK7MT3KNXDSZCQTPUV2Q53NX6Y3BQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.522261240 (316h27m6.073731691s) [noise=true , address=145.79.227.16:4909 , piece_id=ZTL4ER5YJWSDGO5F5KLRUL5WRTA6WKVOJOERIBTRIIVEWXUY5Z2Q, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.522719151 (316h27m6.074189612s) [noise=true , address=145.79.227.26:4654 , piece_id=EM4RRGMPPNVR7VTXP3BED43FJFIUDZYVT4S5OUK2TRXZHYNMQ64Q, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.523017269 (316h27m6.07448772s) [noise=true , address=77.90.31.13:4627 , piece_id=Q7U5BFXHYQBBJQCU4EQGAR75WKL5QMBJ3LO34UYGO7QHLMDJBKWQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.523216949 (316h27m6.07468741s) [noise=true , address=145.79.227.25:4618 , piece_id=CV6M3CTPQR2MPQZFU6W34MFQERT6MMN3RRF2N3W35FGI464OSUUA, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.523744804 (316h27m6.075215265s) [noise=true , address=145.79.227.6:4258 , piece_id=B5D4TCVJORSAWDAU4DQ5VAV4CCRF5WUI24DUVUORAEKLAWGI7ABQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.523828153 (316h27m6.075298614s) [noise=true , address=145.79.227.24:4583 , piece_id=MLW4X34EWXKPOCFVI4VIQSL6RMUOJ5HRYHQ4HY35JIFT4ZFJ6F3Q, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.524255396 (316h27m6.075725858s) [noise=true , address=45.91.116.7:4009 , piece_id=FXQGF3UQN5CPFBS2HWD7BAXR4QIOF42MMK3VJQ7AK4THYJWOFFLQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.524662051 (316h27m6.076132512s) [noise=true , address=145.79.227.15:4874 , piece_id=TYAXIF3ZJQQSFGFANYF56DG4O5WBBXK4M7OT67E7ARYNZ663HESA, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.524746221 (316h27m6.076216682s) [noise=true , address=145.79.227.28:4726 , piece_id=3LBJV2B4CVXEMHCOPRW7DR5QBKPS6DFMPTHQSVNR4XRTZZVIGBQA, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.525227157 (316h27m6.076697608s) [noise=true , address=77.90.31.21:4879 , piece_id=GHTH55OGGXBAZZ5FGPCDYVHDYEETM7DNVTPSTTFFFRZMKZZ3XRXQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:09.134610631 (316h27m7.686081093s) [upload=278701]: Next returned nil segment
Nov 22 12:31:09.134628024 (316h27m7.686098495s) [upload=278701]: Waiting for error group managing segments...
Nov 22 12:31:09.502125778 (316h27m8.053596239s) [noise=true , address=145.79.227.16:4909 , piece_id=ZTL4ER5YJWSDGO5F5KLRUL5WRTA6WKVOJOERIBTRIIVEWXUY5Z2Q, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.503083742 (316h27m8.054554193s) [noise=true , address=145.79.227.20:4437 , piece_id=CZ32MB4AEZZ6YKCNWKRYALVAK7MT3KNXDSZCQTPUV2Q53NX6Y3BQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.517732716 (316h27m8.069203177s) [noise=true , address=145.79.227.25:4618 , piece_id=CV6M3CTPQR2MPQZFU6W34MFQERT6MMN3RRF2N3W35FGI464OSUUA, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.518029582 (316h27m8.069500043s) [noise=true , address=145.79.227.28:4726 , piece_id=3LBJV2B4CVXEMHCOPRW7DR5QBKPS6DFMPTHQSVNR4XRTZZVIGBQA, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.519935051 (316h27m8.071405512s) [noise=true , address=145.79.227.26:4654 , piece_id=EM4RRGMPPNVR7VTXP3BED43FJFIUDZYVT4S5OUK2TRXZHYNMQ64Q, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.520607661 (316h27m8.072078122s) [noise=true , address=145.79.227.27:4689 , piece_id=76S65KEZ7I2E76RR2JXOZ3ONGXCWSJA5XB7XQVFEEKGGNC34Z4EQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.521187736 (316h27m8.072658197s) [noise=true , address=145.79.227.23:4546 , piece_id=73UPPN2SYWNWEVWHCF4BRGKNO5GCRGOV3S5JZPX4MW75EJZRQOWQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.532974760 (316h27m8.084445221s) [noise=true , address=145.79.227.24:4583 , piece_id=MLW4X34EWXKPOCFVI4VIQSL6RMUOJ5HRYHQ4HY35JIFT4ZFJ6F3Q, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.533688800 (316h27m8.085159271s) [noise=true , address=145.79.227.6:4258 , piece_id=B5D4TCVJORSAWDAU4DQ5VAV4CCRF5WUI24DUVUORAEKLAWGI7ABQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.534460109 (316h27m8.08593057s) [noise=true , address=145.79.227.15:4874 , piece_id=TYAXIF3ZJQQSFGFANYF56DG4O5WBBXK4M7OT67E7ARYNZ663HESA, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.542955375 (316h27m8.094425826s) [noise=true , address=145.79.227.17:4972 , piece_id=UZK7LCIOBGGDNJZUVDBOLBJHGOIA4SUT2O3PD67PYQD3J2DGBJEA, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.563098982 (316h27m8.114569443s) [noise=true , address=145.79.227.21:4474 , piece_id=PCRWYQVCAZCPOILAEDHJGDBFKX4GEJ6JXCF7FUS7OVFRUTDPJI4Q, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:36.585475346 (316h27m35.136945807s) [noise=true , address=45.91.116.10:4082 , piece_id=GBDYXAWONOHQGBGM3NCRQYDSPFOAIHRJHBB2W6ZGJLMZUJLV65XA, seg_pos={0 0}, upload=278701]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: read tcp 188.166.225.170:54326->45.91.116.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:36.585499923 (316h27m35.136970384s) [noise=true , address=77.90.31.21:4879 , piece_id=GHTH55OGGXBAZZ5FGPCDYVHDYEETM7DNVTPSTTFFFRZMKZZ3XRXQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: rpc: dial tcp 77.90.31.21:4879: i/o timeout; context deadline exceeded
Nov 22 12:31:36.585749648 (316h27m35.137220119s) [noise=true , address=77.90.31.13:4627 , piece_id=Q7U5BFXHYQBBJQCU4EQGAR75WKL5QMBJ3LO34UYGO7QHLMDJBKWQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:36.585848086 (316h27m35.137318547s) [noise=true , address=45.91.116.7:4009 , piece_id=FXQGF3UQN5CPFBS2HWD7BAXR4QIOF42MMK3VJQ7AK4THYJWOFFLQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:36.585900936 (316h27m35.137371397s) [seg_pos={0 0}, upload=278701]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Nov 22 12:31:36.828881025 (316h27m35.380351486s) [upload=278701]: Done uploading segments...
Nov 22 12:31:36.828915801 (316h27m35.380386262s) [upload=278701]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 5.616644216s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [..........................................X..X..X...........] duration:
6.341228306s trace:
https://jaeger.dev.storj.io/trace/92df32505639913
FAIL (2025-11-22 12:42:07 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.341271647s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [............................................X..X.XX.........] duration:
3.489566872s trace:
https://jaeger.dev.storj.io/trace/71781384eb27b231
FAIL (2025-11-22 12:42:04 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.48959717s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [.........................................X...X..............] duration:
5.745276905s trace:
https://jaeger.dev.storj.io/trace/570ccd0ce1345814
FAIL (2025-11-22 12:42:03 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.745317703s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [...........................................X.....X..........] duration:
5.802835058s trace:
https://jaeger.dev.storj.io/trace/6c800b4c3a84b58c
FAIL (2025-11-22 12:41:50 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 5)
======LOGS======
N5RGYNBMYFXXBJDBTUCN37EAT7WBRNTX2YQA, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.890374814 (316h29m10.441845275s) [noise=true , address=145.79.227.32:4858 , piece_id=K5L3LEUKZWHS36XJHNNY4TYRXCP4DG5UAKCZXPJJTSG26CKUI4HQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.890535981 (316h29m10.442006442s) [noise=true , address=77.90.31.21:4890 , piece_id=DUEFPJUTYMD564Z4III63LZM6L4IL2ZTGM3QDZ6CG7WV5APHBNRA, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.890774064 (316h29m10.442244525s) [noise=true , address=45.91.116.14:4213 , piece_id=HPFISTE2N3B4RBDPAZAGVZJ5WYLCPHTCYH4WMJ56IWEHR3ZBEMOQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.890934851 (316h29m10.442405312s) [noise=true , address=45.91.116.6:4000 , piece_id=T4CI3KKNMFPT3ZIODHE7BQ4DVHVH2QTSO6OB7QZPL5SNPXQJE6TA, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.891167002 (316h29m10.442637463s) [noise=true , address=77.90.31.17:4787 , piece_id=IYIEVH6CARRYCARKOLKWVBICQDNDRCTGQMWF3VJZ6YBC3GO5OWHQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.891369828 (316h29m10.442840289s) [noise=true , address=45.91.116.7:4039 , piece_id=HSRY63FXTEPCP2T46V77RVSC24GGRBAYLLU7DTKOLWGVOYHF6YVA, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.891556875 (316h29m10.443027336s) [noise=true , address=77.90.31.11:4572 , piece_id=G4XSIPSWOLSEOGGFA2NLOSDCGTLZPI3P5LDNNLB5RGMU5OQGWXGQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.891896812 (316h29m10.443367273s) [noise=true , address=77.90.31.19:4863 , piece_id=3PM3Q6X3ADKX2RKIIFDMK57LTJNJHNFK4UYPKFK3ZEQLYQ23EP3A, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.892019235 (316h29m10.443489696s) [noise=true , address=77.90.31.10:4524 , piece_id=Z2IGDHPDU6JAR5CAFQBGYHO3TQ66K4BZOAMXSC7X3RFWLGHOOMTQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.892316521 (316h29m10.443786982s) [noise=true , address=77.90.31.13:4644 , piece_id=UPDZGC3CMJXA3AOJX65VLASU37NLFGPIJLCAPN32TEQB5QHTU7EQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.892682318 (316h29m10.444152779s) [noise=true , address=45.91.116.9:4274 , piece_id=MXWRAXYLBBGLUVTA5GP3JVX2IRM2UI2IC35D7KX3XMFEBXLIDX5A, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:13.353758506 (316h29m11.905228957s) [upload=278731]: Next returned nil segment
Nov 22 12:33:13.353772043 (316h29m11.905242504s) [upload=278731]: Waiting for error group managing segments...
Nov 22 12:33:13.751703718 (316h29m12.303174169s) [noise=true , address=145.79.227.32:4858 , piece_id=K5L3LEUKZWHS36XJHNNY4TYRXCP4DG5UAKCZXPJJTSG26CKUI4HQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil>
Nov 22 12:33:13.753660293 (316h29m12.305130764s) [noise=true , address=145.79.227.26:4674 , piece_id=UP7VIVRDRL6KERJSX3VLWTKGW3ESGCQ7BLYCBXGBB5UNIDE6OHQQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil>
Nov 22 12:33:13.760046123 (316h29m12.311516574s) [noise=true , address=145.79.227.3:4167 , piece_id=4STIVXGU7SOXWAIPG2PXWBLNDD7EYQ56E2DBN2UZ2AZ3TZSXCTIQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil>
Nov 22 12:33:57.373975014 (316h29m55.925445475s) [noise=true , address=77.90.31.13:4644 , piece_id=UPDZGC3CMJXA3AOJX65VLASU37NLFGPIJLCAPN32TEQB5QHTU7EQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil>
Nov 22 12:34:06.122275078 (316h30m4.673745539s) [noise=true , address=45.91.116.13:4181 , piece_id=MQNJEBQFZMHNDKAH2K3JEA7YBK2UK5BGSNCJF3MASF3CKVA5VD6Q, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil>
Nov 22 12:34:10.949171331 (316h30m9.500641792s) [noise=true , address=77.90.31.21:4890 , piece_id=DUEFPJUTYMD564Z4III63LZM6L4IL2ZTGM3QDZ6CG7WV5APHBNRA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1iE3mDnEhGDsMBtmUXC4EwvSrpyHLoybzVXkC9eoFuYEMHeBrd, address:77.90.31.21:4890): protocol: read tcp 188.166.225.170:55900->77.90.31.21:4890: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.949753650 (316h30m9.501224111s) [noise=true , address=45.91.116.12:4237 , piece_id=OG6XREYEGOYVFFMXCGBERRAIEMI2QMUGSBJBYP7GJQZ7FOAI4BIA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1FMjHdNGV42a9QZyMmSPTJWQrA8CdSRvXqrimnt6UGVreCzmgF, address:45.91.116.12:4237): protocol: rpc: dial tcp 45.91.116.12:4237: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.950066235 (316h30m9.501536706s) [noise=true , address=77.90.31.19:4863 , piece_id=3PM3Q6X3ADKX2RKIIFDMK57LTJNJHNFK4UYPKFK3ZEQLYQ23EP3A, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:124x1cbCPJRK8YkgSP6UzHymoLu1E1znnfg5WBiD89kSDLi8JeL, address:77.90.31.19:4863): protocol: write tcp 188.166.225.170:45968->77.90.31.19:4863: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.950246558 (316h30m9.50171702s) [noise=true , address=77.90.31.10:4524 , piece_id=Z2IGDHPDU6JAR5CAFQBGYHO3TQ66K4BZOAMXSC7X3RFWLGHOOMTQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12uN7CznoK4AD8FqFkNUndawdGJ9rmjAe7oNnjJkV844ScY87qe, address:77.90.31.10:4524): protocol: write tcp 188.166.225.170:40266->77.90.31.10:4524: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.950593058 (316h30m9.502063529s) [noise=true , address=77.90.31.14:4690 , piece_id=RJ3KXY3HURT7Q4HZYTOLSAL7Y6VMDGGRIQP6XHE4YB6OAA3PJRAA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.950853535 (316h30m9.502324006s) [noise=true , address=45.91.116.14:4213 , piece_id=HPFISTE2N3B4RBDPAZAGVZJ5WYLCPHTCYH4WMJ56IWEHR3ZBEMOQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12tQRqowH4aDr9h3s1WLs1o7na1ueiafwgUeeHfAB6g7mcE6e1t, address:45.91.116.14:4213): protocol: rpc: dial tcp 45.91.116.14:4213: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.951186438 (316h30m9.502656889s) [noise=true , address=45.91.116.6:4000 , piece_id=T4CI3KKNMFPT3ZIODHE7BQ4DVHVH2QTSO6OB7QZPL5SNPXQJE6TA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12CuCqHJYbanbuaEMHvtcuQXhvcdahD8eyT2fKDhqtxdSZ6ES4f, address:45.91.116.6:4000): protocol: rpc: dial tcp 45.91.116.6:4000: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.951409864 (316h30m9.502880325s) [noise=true , address=45.91.116.7:4039 , piece_id=HSRY63FXTEPCP2T46V77RVSC24GGRBAYLLU7DTKOLWGVOYHF6YVA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:16oq1Yv7JfHrg4gdHfhwuMj4UsWeQQPayKV6Pj1NY2CgJhuPr2, address:45.91.116.7:4039): protocol: write tcp 188.166.225.170:36904->45.91.116.7:4039: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.951660191 (316h30m9.503130652s) [noise=true , address=77.90.31.11:4572 , piece_id=G4XSIPSWOLSEOGGFA2NLOSDCGTLZPI3P5LDNNLB5RGMU5OQGWXGQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1ZAfYZ4wgpZVvatj6wtHona3hTaPcLxAmXC1DT2uKQyo8nLQQ7, address:77.90.31.11:4572): protocol: rpc: dial tcp 77.90.31.11:4572: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.952019855 (316h30m9.503490316s) [noise=true , address=45.91.116.9:4274 , piece_id=MXWRAXYLBBGLUVTA5GP3JVX2IRM2UI2IC35D7KX3XMFEBXLIDX5A, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12g6jFa38dTz5UdgsQaEuM2MSH1H7cxaysWsVmbHkM2Dd2W2iDb, address:45.91.116.9:4274): protocol: rpc: dial tcp 45.91.116.9:4274: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.952290972 (316h30m9.503761443s) [noise=true , address=77.90.31.15:4720 , piece_id=BDQUF6C5TT3ASE3DIN6SSVJHOZAJK4ZEW33OIYPTGB4LDVRELSUA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12VbsxFqMB3vVphSAJqYpbBzua7Goq45E6ovJSaExkpLtfkcc74, address:77.90.31.15:4720): protocol: rpc: dial tcp 77.90.31.15:4720: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.952544654 (316h30m9.504015105s) [noise=true , address=45.91.116.10:4110 , piece_id=H6FYA6K4N4AQYU3MN5RGYNBMYFXXBJDBTUCN37EAT7WBRNTX2YQA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:123KwusATu1ZKtNNVHZZZBE67ZEnVb8puyfQidBi3fEWREqdf1h, address:45.91.116.10:4110): protocol: rpc: dial tcp 45.91.116.10:4110: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.953548155 (316h30m9.505018617s) [noise=true , address=77.90.31.17:4787 , piece_id=IYIEVH6CARRYCARKOLKWVBICQDNDRCTGQMWF3VJZ6YBC3GO5OWHQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1Lietonb5aVbaZQt6wQ2EoxwJkSEn9oL9Dwyrc9mTyomEKikNt, address:77.90.31.17:4787): protocol: read tcp 188.166.225.170:57908->77.90.31.17:4787: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.953579816 (316h30m9.505050277s) [seg_pos={0 0}, upload=278731]: Done waiting for segment. successful: 5 optimal: 14 errs: <nil>
Nov 22 12:34:11.195719123 (316h30m9.747189584s) [upload=278731]: Done uploading segments...
Nov 22 12:34:11.195932840 (316h30m9.747403301s) [upload=278731]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 5)
Duration: 5.802861599s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [..........................................X...X.............] duration:
5.764561197s trace:
https://jaeger.dev.storj.io/trace/570ccd0ce1345814
FAIL (2025-11-22 12:41:45 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.764618305s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..............................................X.............] duration:
5.701195018s trace:
https://jaeger.dev.storj.io/trace/5ae3d2bf80495000
FAIL (2025-11-22 12:41:33 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Nov 22 12:31:07.276308940 (316h27m5.827779401s) [upload=278701]: Starting upload
Nov 22 12:31:07.276483523 (316h27m5.827953994s) [upload=278701]: Done starting upload
Nov 22 12:31:07.276562243 (316h27m5.828032704s) [upload=278701]: Uploading segments...
Nov 22 12:31:07.276950593 (316h27m5.828421054s) [upload=278701]: Got next segment. Inline: false
Nov 22 12:31:07.277027639 (316h27m5.8284981s) [upload=278701]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:31:07.277034181 (316h27m5.828504643s) [upload=278701]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:31:07.520212027 (316h27m6.071682488s) [seg_pos={0 0}, upload=278701]: Begin upload segment...
Nov 22 12:31:07.520269256 (316h27m6.071739717s) [seg_pos={0 0}, upload=278701]: Done begin upload segment.
Nov 22 12:31:07.520308992 (316h27m6.071779453s) [noise=true , address=145.79.227.27:4689 , piece_id=76S65KEZ7I2E76RR2JXOZ3ONGXCWSJA5XB7XQVFEEKGGNC34Z4EQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.520426746 (316h27m6.071897207s) [noise=true , address=145.79.227.21:4474 , piece_id=PCRWYQVCAZCPOILAEDHJGDBFKX4GEJ6JXCF7FUS7OVFRUTDPJI4Q, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.521175282 (316h27m6.072645743s) [noise=true , address=145.79.227.17:4972 , piece_id=UZK7LCIOBGGDNJZUVDBOLBJHGOIA4SUT2O3PD67PYQD3J2DGBJEA, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.521527573 (316h27m6.072998044s) [noise=true , address=145.79.227.23:4546 , piece_id=73UPPN2SYWNWEVWHCF4BRGKNO5GCRGOV3S5JZPX4MW75EJZRQOWQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.521694671 (316h27m6.073165132s) [noise=true , address=45.91.116.10:4082 , piece_id=GBDYXAWONOHQGBGM3NCRQYDSPFOAIHRJHBB2W6ZGJLMZUJLV65XA, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.522219410 (316h27m6.073689871s) [noise=true , address=145.79.227.20:4437 , piece_id=CZ32MB4AEZZ6YKCNWKRYALVAK7MT3KNXDSZCQTPUV2Q53NX6Y3BQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.522261240 (316h27m6.073731691s) [noise=true , address=145.79.227.16:4909 , piece_id=ZTL4ER5YJWSDGO5F5KLRUL5WRTA6WKVOJOERIBTRIIVEWXUY5Z2Q, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.522719151 (316h27m6.074189612s) [noise=true , address=145.79.227.26:4654 , piece_id=EM4RRGMPPNVR7VTXP3BED43FJFIUDZYVT4S5OUK2TRXZHYNMQ64Q, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.523017269 (316h27m6.07448772s) [noise=true , address=77.90.31.13:4627 , piece_id=Q7U5BFXHYQBBJQCU4EQGAR75WKL5QMBJ3LO34UYGO7QHLMDJBKWQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.523216949 (316h27m6.07468741s) [noise=true , address=145.79.227.25:4618 , piece_id=CV6M3CTPQR2MPQZFU6W34MFQERT6MMN3RRF2N3W35FGI464OSUUA, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.523744804 (316h27m6.075215265s) [noise=true , address=145.79.227.6:4258 , piece_id=B5D4TCVJORSAWDAU4DQ5VAV4CCRF5WUI24DUVUORAEKLAWGI7ABQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.523828153 (316h27m6.075298614s) [noise=true , address=145.79.227.24:4583 , piece_id=MLW4X34EWXKPOCFVI4VIQSL6RMUOJ5HRYHQ4HY35JIFT4ZFJ6F3Q, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.524255396 (316h27m6.075725858s) [noise=true , address=45.91.116.7:4009 , piece_id=FXQGF3UQN5CPFBS2HWD7BAXR4QIOF42MMK3VJQ7AK4THYJWOFFLQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.524662051 (316h27m6.076132512s) [noise=true , address=145.79.227.15:4874 , piece_id=TYAXIF3ZJQQSFGFANYF56DG4O5WBBXK4M7OT67E7ARYNZ663HESA, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.524746221 (316h27m6.076216682s) [noise=true , address=145.79.227.28:4726 , piece_id=3LBJV2B4CVXEMHCOPRW7DR5QBKPS6DFMPTHQSVNR4XRTZZVIGBQA, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.525227157 (316h27m6.076697608s) [noise=true , address=77.90.31.21:4879 , piece_id=GHTH55OGGXBAZZ5FGPCDYVHDYEETM7DNVTPSTTFFFRZMKZZ3XRXQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:09.134610631 (316h27m7.686081093s) [upload=278701]: Next returned nil segment
Nov 22 12:31:09.134628024 (316h27m7.686098495s) [upload=278701]: Waiting for error group managing segments...
Nov 22 12:31:09.502125778 (316h27m8.053596239s) [noise=true , address=145.79.227.16:4909 , piece_id=ZTL4ER5YJWSDGO5F5KLRUL5WRTA6WKVOJOERIBTRIIVEWXUY5Z2Q, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.503083742 (316h27m8.054554193s) [noise=true , address=145.79.227.20:4437 , piece_id=CZ32MB4AEZZ6YKCNWKRYALVAK7MT3KNXDSZCQTPUV2Q53NX6Y3BQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.517732716 (316h27m8.069203177s) [noise=true , address=145.79.227.25:4618 , piece_id=CV6M3CTPQR2MPQZFU6W34MFQERT6MMN3RRF2N3W35FGI464OSUUA, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.518029582 (316h27m8.069500043s) [noise=true , address=145.79.227.28:4726 , piece_id=3LBJV2B4CVXEMHCOPRW7DR5QBKPS6DFMPTHQSVNR4XRTZZVIGBQA, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.519935051 (316h27m8.071405512s) [noise=true , address=145.79.227.26:4654 , piece_id=EM4RRGMPPNVR7VTXP3BED43FJFIUDZYVT4S5OUK2TRXZHYNMQ64Q, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.520607661 (316h27m8.072078122s) [noise=true , address=145.79.227.27:4689 , piece_id=76S65KEZ7I2E76RR2JXOZ3ONGXCWSJA5XB7XQVFEEKGGNC34Z4EQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.521187736 (316h27m8.072658197s) [noise=true , address=145.79.227.23:4546 , piece_id=73UPPN2SYWNWEVWHCF4BRGKNO5GCRGOV3S5JZPX4MW75EJZRQOWQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.532974760 (316h27m8.084445221s) [noise=true , address=145.79.227.24:4583 , piece_id=MLW4X34EWXKPOCFVI4VIQSL6RMUOJ5HRYHQ4HY35JIFT4ZFJ6F3Q, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.533688800 (316h27m8.085159271s) [noise=true , address=145.79.227.6:4258 , piece_id=B5D4TCVJORSAWDAU4DQ5VAV4CCRF5WUI24DUVUORAEKLAWGI7ABQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.534460109 (316h27m8.08593057s) [noise=true , address=145.79.227.15:4874 , piece_id=TYAXIF3ZJQQSFGFANYF56DG4O5WBBXK4M7OT67E7ARYNZ663HESA, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.542955375 (316h27m8.094425826s) [noise=true , address=145.79.227.17:4972 , piece_id=UZK7LCIOBGGDNJZUVDBOLBJHGOIA4SUT2O3PD67PYQD3J2DGBJEA, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.563098982 (316h27m8.114569443s) [noise=true , address=145.79.227.21:4474 , piece_id=PCRWYQVCAZCPOILAEDHJGDBFKX4GEJ6JXCF7FUS7OVFRUTDPJI4Q, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:36.585475346 (316h27m35.136945807s) [noise=true , address=45.91.116.10:4082 , piece_id=GBDYXAWONOHQGBGM3NCRQYDSPFOAIHRJHBB2W6ZGJLMZUJLV65XA, seg_pos={0 0}, upload=278701]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: read tcp 188.166.225.170:54326->45.91.116.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:36.585499923 (316h27m35.136970384s) [noise=true , address=77.90.31.21:4879 , piece_id=GHTH55OGGXBAZZ5FGPCDYVHDYEETM7DNVTPSTTFFFRZMKZZ3XRXQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: rpc: dial tcp 77.90.31.21:4879: i/o timeout; context deadline exceeded
Nov 22 12:31:36.585749648 (316h27m35.137220119s) [noise=true , address=77.90.31.13:4627 , piece_id=Q7U5BFXHYQBBJQCU4EQGAR75WKL5QMBJ3LO34UYGO7QHLMDJBKWQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:36.585848086 (316h27m35.137318547s) [noise=true , address=45.91.116.7:4009 , piece_id=FXQGF3UQN5CPFBS2HWD7BAXR4QIOF42MMK3VJQ7AK4THYJWOFFLQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:36.585900936 (316h27m35.137371397s) [seg_pos={0 0}, upload=278701]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Nov 22 12:31:36.828881025 (316h27m35.380351486s) [upload=278701]: Done uploading segments...
Nov 22 12:31:36.828915801 (316h27m35.380386262s) [upload=278701]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 5.701226848s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [.............................................X..X.XX........] duration:
3.355150951s trace:
https://jaeger.dev.storj.io/trace/71781384eb27b231
FAIL (2025-11-22 12:41:20 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.355182211s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [...........................................X..X..X..........] duration:
6.204195923s trace:
https://jaeger.dev.storj.io/trace/92df32505639913
FAIL (2025-11-22 12:41:16 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.204223384s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [...........................................X...X............] duration:
5.634633632s trace:
https://jaeger.dev.storj.io/trace/570ccd0ce1345814
FAIL (2025-11-22 12:41:12 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.634677968s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [............................................X.....X.........] duration:
5.670184727s trace:
https://jaeger.dev.storj.io/trace/6c800b4c3a84b58c
FAIL (2025-11-22 12:40:59 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 5)
======LOGS======
N5RGYNBMYFXXBJDBTUCN37EAT7WBRNTX2YQA, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.890374814 (316h29m10.441845275s) [noise=true , address=145.79.227.32:4858 , piece_id=K5L3LEUKZWHS36XJHNNY4TYRXCP4DG5UAKCZXPJJTSG26CKUI4HQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.890535981 (316h29m10.442006442s) [noise=true , address=77.90.31.21:4890 , piece_id=DUEFPJUTYMD564Z4III63LZM6L4IL2ZTGM3QDZ6CG7WV5APHBNRA, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.890774064 (316h29m10.442244525s) [noise=true , address=45.91.116.14:4213 , piece_id=HPFISTE2N3B4RBDPAZAGVZJ5WYLCPHTCYH4WMJ56IWEHR3ZBEMOQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.890934851 (316h29m10.442405312s) [noise=true , address=45.91.116.6:4000 , piece_id=T4CI3KKNMFPT3ZIODHE7BQ4DVHVH2QTSO6OB7QZPL5SNPXQJE6TA, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.891167002 (316h29m10.442637463s) [noise=true , address=77.90.31.17:4787 , piece_id=IYIEVH6CARRYCARKOLKWVBICQDNDRCTGQMWF3VJZ6YBC3GO5OWHQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.891369828 (316h29m10.442840289s) [noise=true , address=45.91.116.7:4039 , piece_id=HSRY63FXTEPCP2T46V77RVSC24GGRBAYLLU7DTKOLWGVOYHF6YVA, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.891556875 (316h29m10.443027336s) [noise=true , address=77.90.31.11:4572 , piece_id=G4XSIPSWOLSEOGGFA2NLOSDCGTLZPI3P5LDNNLB5RGMU5OQGWXGQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.891896812 (316h29m10.443367273s) [noise=true , address=77.90.31.19:4863 , piece_id=3PM3Q6X3ADKX2RKIIFDMK57LTJNJHNFK4UYPKFK3ZEQLYQ23EP3A, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.892019235 (316h29m10.443489696s) [noise=true , address=77.90.31.10:4524 , piece_id=Z2IGDHPDU6JAR5CAFQBGYHO3TQ66K4BZOAMXSC7X3RFWLGHOOMTQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.892316521 (316h29m10.443786982s) [noise=true , address=77.90.31.13:4644 , piece_id=UPDZGC3CMJXA3AOJX65VLASU37NLFGPIJLCAPN32TEQB5QHTU7EQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.892682318 (316h29m10.444152779s) [noise=true , address=45.91.116.9:4274 , piece_id=MXWRAXYLBBGLUVTA5GP3JVX2IRM2UI2IC35D7KX3XMFEBXLIDX5A, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:13.353758506 (316h29m11.905228957s) [upload=278731]: Next returned nil segment
Nov 22 12:33:13.353772043 (316h29m11.905242504s) [upload=278731]: Waiting for error group managing segments...
Nov 22 12:33:13.751703718 (316h29m12.303174169s) [noise=true , address=145.79.227.32:4858 , piece_id=K5L3LEUKZWHS36XJHNNY4TYRXCP4DG5UAKCZXPJJTSG26CKUI4HQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil>
Nov 22 12:33:13.753660293 (316h29m12.305130764s) [noise=true , address=145.79.227.26:4674 , piece_id=UP7VIVRDRL6KERJSX3VLWTKGW3ESGCQ7BLYCBXGBB5UNIDE6OHQQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil>
Nov 22 12:33:13.760046123 (316h29m12.311516574s) [noise=true , address=145.79.227.3:4167 , piece_id=4STIVXGU7SOXWAIPG2PXWBLNDD7EYQ56E2DBN2UZ2AZ3TZSXCTIQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil>
Nov 22 12:33:57.373975014 (316h29m55.925445475s) [noise=true , address=77.90.31.13:4644 , piece_id=UPDZGC3CMJXA3AOJX65VLASU37NLFGPIJLCAPN32TEQB5QHTU7EQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil>
Nov 22 12:34:06.122275078 (316h30m4.673745539s) [noise=true , address=45.91.116.13:4181 , piece_id=MQNJEBQFZMHNDKAH2K3JEA7YBK2UK5BGSNCJF3MASF3CKVA5VD6Q, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil>
Nov 22 12:34:10.949171331 (316h30m9.500641792s) [noise=true , address=77.90.31.21:4890 , piece_id=DUEFPJUTYMD564Z4III63LZM6L4IL2ZTGM3QDZ6CG7WV5APHBNRA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1iE3mDnEhGDsMBtmUXC4EwvSrpyHLoybzVXkC9eoFuYEMHeBrd, address:77.90.31.21:4890): protocol: read tcp 188.166.225.170:55900->77.90.31.21:4890: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.949753650 (316h30m9.501224111s) [noise=true , address=45.91.116.12:4237 , piece_id=OG6XREYEGOYVFFMXCGBERRAIEMI2QMUGSBJBYP7GJQZ7FOAI4BIA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1FMjHdNGV42a9QZyMmSPTJWQrA8CdSRvXqrimnt6UGVreCzmgF, address:45.91.116.12:4237): protocol: rpc: dial tcp 45.91.116.12:4237: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.950066235 (316h30m9.501536706s) [noise=true , address=77.90.31.19:4863 , piece_id=3PM3Q6X3ADKX2RKIIFDMK57LTJNJHNFK4UYPKFK3ZEQLYQ23EP3A, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:124x1cbCPJRK8YkgSP6UzHymoLu1E1znnfg5WBiD89kSDLi8JeL, address:77.90.31.19:4863): protocol: write tcp 188.166.225.170:45968->77.90.31.19:4863: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.950246558 (316h30m9.50171702s) [noise=true , address=77.90.31.10:4524 , piece_id=Z2IGDHPDU6JAR5CAFQBGYHO3TQ66K4BZOAMXSC7X3RFWLGHOOMTQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12uN7CznoK4AD8FqFkNUndawdGJ9rmjAe7oNnjJkV844ScY87qe, address:77.90.31.10:4524): protocol: write tcp 188.166.225.170:40266->77.90.31.10:4524: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.950593058 (316h30m9.502063529s) [noise=true , address=77.90.31.14:4690 , piece_id=RJ3KXY3HURT7Q4HZYTOLSAL7Y6VMDGGRIQP6XHE4YB6OAA3PJRAA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.950853535 (316h30m9.502324006s) [noise=true , address=45.91.116.14:4213 , piece_id=HPFISTE2N3B4RBDPAZAGVZJ5WYLCPHTCYH4WMJ56IWEHR3ZBEMOQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12tQRqowH4aDr9h3s1WLs1o7na1ueiafwgUeeHfAB6g7mcE6e1t, address:45.91.116.14:4213): protocol: rpc: dial tcp 45.91.116.14:4213: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.951186438 (316h30m9.502656889s) [noise=true , address=45.91.116.6:4000 , piece_id=T4CI3KKNMFPT3ZIODHE7BQ4DVHVH2QTSO6OB7QZPL5SNPXQJE6TA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12CuCqHJYbanbuaEMHvtcuQXhvcdahD8eyT2fKDhqtxdSZ6ES4f, address:45.91.116.6:4000): protocol: rpc: dial tcp 45.91.116.6:4000: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.951409864 (316h30m9.502880325s) [noise=true , address=45.91.116.7:4039 , piece_id=HSRY63FXTEPCP2T46V77RVSC24GGRBAYLLU7DTKOLWGVOYHF6YVA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:16oq1Yv7JfHrg4gdHfhwuMj4UsWeQQPayKV6Pj1NY2CgJhuPr2, address:45.91.116.7:4039): protocol: write tcp 188.166.225.170:36904->45.91.116.7:4039: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.951660191 (316h30m9.503130652s) [noise=true , address=77.90.31.11:4572 , piece_id=G4XSIPSWOLSEOGGFA2NLOSDCGTLZPI3P5LDNNLB5RGMU5OQGWXGQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1ZAfYZ4wgpZVvatj6wtHona3hTaPcLxAmXC1DT2uKQyo8nLQQ7, address:77.90.31.11:4572): protocol: rpc: dial tcp 77.90.31.11:4572: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.952019855 (316h30m9.503490316s) [noise=true , address=45.91.116.9:4274 , piece_id=MXWRAXYLBBGLUVTA5GP3JVX2IRM2UI2IC35D7KX3XMFEBXLIDX5A, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12g6jFa38dTz5UdgsQaEuM2MSH1H7cxaysWsVmbHkM2Dd2W2iDb, address:45.91.116.9:4274): protocol: rpc: dial tcp 45.91.116.9:4274: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.952290972 (316h30m9.503761443s) [noise=true , address=77.90.31.15:4720 , piece_id=BDQUF6C5TT3ASE3DIN6SSVJHOZAJK4ZEW33OIYPTGB4LDVRELSUA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12VbsxFqMB3vVphSAJqYpbBzua7Goq45E6ovJSaExkpLtfkcc74, address:77.90.31.15:4720): protocol: rpc: dial tcp 77.90.31.15:4720: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.952544654 (316h30m9.504015105s) [noise=true , address=45.91.116.10:4110 , piece_id=H6FYA6K4N4AQYU3MN5RGYNBMYFXXBJDBTUCN37EAT7WBRNTX2YQA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:123KwusATu1ZKtNNVHZZZBE67ZEnVb8puyfQidBi3fEWREqdf1h, address:45.91.116.10:4110): protocol: rpc: dial tcp 45.91.116.10:4110: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.953548155 (316h30m9.505018617s) [noise=true , address=77.90.31.17:4787 , piece_id=IYIEVH6CARRYCARKOLKWVBICQDNDRCTGQMWF3VJZ6YBC3GO5OWHQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1Lietonb5aVbaZQt6wQ2EoxwJkSEn9oL9Dwyrc9mTyomEKikNt, address:77.90.31.17:4787): protocol: read tcp 188.166.225.170:57908->77.90.31.17:4787: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.953579816 (316h30m9.505050277s) [seg_pos={0 0}, upload=278731]: Done waiting for segment. successful: 5 optimal: 14 errs: <nil>
Nov 22 12:34:11.195719123 (316h30m9.747189584s) [upload=278731]: Done uploading segments...
Nov 22 12:34:11.195932840 (316h30m9.747403301s) [upload=278731]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 5)
Duration: 5.670236805s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...............................................X............] duration:
5.865414329s trace:
https://jaeger.dev.storj.io/trace/5ae3d2bf80495000
FAIL (2025-11-22 12:40:33 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Nov 22 12:31:07.276308940 (316h27m5.827779401s) [upload=278701]: Starting upload
Nov 22 12:31:07.276483523 (316h27m5.827953994s) [upload=278701]: Done starting upload
Nov 22 12:31:07.276562243 (316h27m5.828032704s) [upload=278701]: Uploading segments...
Nov 22 12:31:07.276950593 (316h27m5.828421054s) [upload=278701]: Got next segment. Inline: false
Nov 22 12:31:07.277027639 (316h27m5.8284981s) [upload=278701]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:31:07.277034181 (316h27m5.828504643s) [upload=278701]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:31:07.520212027 (316h27m6.071682488s) [seg_pos={0 0}, upload=278701]: Begin upload segment...
Nov 22 12:31:07.520269256 (316h27m6.071739717s) [seg_pos={0 0}, upload=278701]: Done begin upload segment.
Nov 22 12:31:07.520308992 (316h27m6.071779453s) [noise=true , address=145.79.227.27:4689 , piece_id=76S65KEZ7I2E76RR2JXOZ3ONGXCWSJA5XB7XQVFEEKGGNC34Z4EQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.520426746 (316h27m6.071897207s) [noise=true , address=145.79.227.21:4474 , piece_id=PCRWYQVCAZCPOILAEDHJGDBFKX4GEJ6JXCF7FUS7OVFRUTDPJI4Q, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.521175282 (316h27m6.072645743s) [noise=true , address=145.79.227.17:4972 , piece_id=UZK7LCIOBGGDNJZUVDBOLBJHGOIA4SUT2O3PD67PYQD3J2DGBJEA, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.521527573 (316h27m6.072998044s) [noise=true , address=145.79.227.23:4546 , piece_id=73UPPN2SYWNWEVWHCF4BRGKNO5GCRGOV3S5JZPX4MW75EJZRQOWQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.521694671 (316h27m6.073165132s) [noise=true , address=45.91.116.10:4082 , piece_id=GBDYXAWONOHQGBGM3NCRQYDSPFOAIHRJHBB2W6ZGJLMZUJLV65XA, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.522219410 (316h27m6.073689871s) [noise=true , address=145.79.227.20:4437 , piece_id=CZ32MB4AEZZ6YKCNWKRYALVAK7MT3KNXDSZCQTPUV2Q53NX6Y3BQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.522261240 (316h27m6.073731691s) [noise=true , address=145.79.227.16:4909 , piece_id=ZTL4ER5YJWSDGO5F5KLRUL5WRTA6WKVOJOERIBTRIIVEWXUY5Z2Q, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.522719151 (316h27m6.074189612s) [noise=true , address=145.79.227.26:4654 , piece_id=EM4RRGMPPNVR7VTXP3BED43FJFIUDZYVT4S5OUK2TRXZHYNMQ64Q, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.523017269 (316h27m6.07448772s) [noise=true , address=77.90.31.13:4627 , piece_id=Q7U5BFXHYQBBJQCU4EQGAR75WKL5QMBJ3LO34UYGO7QHLMDJBKWQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.523216949 (316h27m6.07468741s) [noise=true , address=145.79.227.25:4618 , piece_id=CV6M3CTPQR2MPQZFU6W34MFQERT6MMN3RRF2N3W35FGI464OSUUA, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.523744804 (316h27m6.075215265s) [noise=true , address=145.79.227.6:4258 , piece_id=B5D4TCVJORSAWDAU4DQ5VAV4CCRF5WUI24DUVUORAEKLAWGI7ABQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.523828153 (316h27m6.075298614s) [noise=true , address=145.79.227.24:4583 , piece_id=MLW4X34EWXKPOCFVI4VIQSL6RMUOJ5HRYHQ4HY35JIFT4ZFJ6F3Q, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.524255396 (316h27m6.075725858s) [noise=true , address=45.91.116.7:4009 , piece_id=FXQGF3UQN5CPFBS2HWD7BAXR4QIOF42MMK3VJQ7AK4THYJWOFFLQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.524662051 (316h27m6.076132512s) [noise=true , address=145.79.227.15:4874 , piece_id=TYAXIF3ZJQQSFGFANYF56DG4O5WBBXK4M7OT67E7ARYNZ663HESA, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.524746221 (316h27m6.076216682s) [noise=true , address=145.79.227.28:4726 , piece_id=3LBJV2B4CVXEMHCOPRW7DR5QBKPS6DFMPTHQSVNR4XRTZZVIGBQA, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.525227157 (316h27m6.076697608s) [noise=true , address=77.90.31.21:4879 , piece_id=GHTH55OGGXBAZZ5FGPCDYVHDYEETM7DNVTPSTTFFFRZMKZZ3XRXQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:09.134610631 (316h27m7.686081093s) [upload=278701]: Next returned nil segment
Nov 22 12:31:09.134628024 (316h27m7.686098495s) [upload=278701]: Waiting for error group managing segments...
Nov 22 12:31:09.502125778 (316h27m8.053596239s) [noise=true , address=145.79.227.16:4909 , piece_id=ZTL4ER5YJWSDGO5F5KLRUL5WRTA6WKVOJOERIBTRIIVEWXUY5Z2Q, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.503083742 (316h27m8.054554193s) [noise=true , address=145.79.227.20:4437 , piece_id=CZ32MB4AEZZ6YKCNWKRYALVAK7MT3KNXDSZCQTPUV2Q53NX6Y3BQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.517732716 (316h27m8.069203177s) [noise=true , address=145.79.227.25:4618 , piece_id=CV6M3CTPQR2MPQZFU6W34MFQERT6MMN3RRF2N3W35FGI464OSUUA, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.518029582 (316h27m8.069500043s) [noise=true , address=145.79.227.28:4726 , piece_id=3LBJV2B4CVXEMHCOPRW7DR5QBKPS6DFMPTHQSVNR4XRTZZVIGBQA, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.519935051 (316h27m8.071405512s) [noise=true , address=145.79.227.26:4654 , piece_id=EM4RRGMPPNVR7VTXP3BED43FJFIUDZYVT4S5OUK2TRXZHYNMQ64Q, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.520607661 (316h27m8.072078122s) [noise=true , address=145.79.227.27:4689 , piece_id=76S65KEZ7I2E76RR2JXOZ3ONGXCWSJA5XB7XQVFEEKGGNC34Z4EQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.521187736 (316h27m8.072658197s) [noise=true , address=145.79.227.23:4546 , piece_id=73UPPN2SYWNWEVWHCF4BRGKNO5GCRGOV3S5JZPX4MW75EJZRQOWQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.532974760 (316h27m8.084445221s) [noise=true , address=145.79.227.24:4583 , piece_id=MLW4X34EWXKPOCFVI4VIQSL6RMUOJ5HRYHQ4HY35JIFT4ZFJ6F3Q, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.533688800 (316h27m8.085159271s) [noise=true , address=145.79.227.6:4258 , piece_id=B5D4TCVJORSAWDAU4DQ5VAV4CCRF5WUI24DUVUORAEKLAWGI7ABQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.534460109 (316h27m8.08593057s) [noise=true , address=145.79.227.15:4874 , piece_id=TYAXIF3ZJQQSFGFANYF56DG4O5WBBXK4M7OT67E7ARYNZ663HESA, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.542955375 (316h27m8.094425826s) [noise=true , address=145.79.227.17:4972 , piece_id=UZK7LCIOBGGDNJZUVDBOLBJHGOIA4SUT2O3PD67PYQD3J2DGBJEA, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.563098982 (316h27m8.114569443s) [noise=true , address=145.79.227.21:4474 , piece_id=PCRWYQVCAZCPOILAEDHJGDBFKX4GEJ6JXCF7FUS7OVFRUTDPJI4Q, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:36.585475346 (316h27m35.136945807s) [noise=true , address=45.91.116.10:4082 , piece_id=GBDYXAWONOHQGBGM3NCRQYDSPFOAIHRJHBB2W6ZGJLMZUJLV65XA, seg_pos={0 0}, upload=278701]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: read tcp 188.166.225.170:54326->45.91.116.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:36.585499923 (316h27m35.136970384s) [noise=true , address=77.90.31.21:4879 , piece_id=GHTH55OGGXBAZZ5FGPCDYVHDYEETM7DNVTPSTTFFFRZMKZZ3XRXQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: rpc: dial tcp 77.90.31.21:4879: i/o timeout; context deadline exceeded
Nov 22 12:31:36.585749648 (316h27m35.137220119s) [noise=true , address=77.90.31.13:4627 , piece_id=Q7U5BFXHYQBBJQCU4EQGAR75WKL5QMBJ3LO34UYGO7QHLMDJBKWQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:36.585848086 (316h27m35.137318547s) [noise=true , address=45.91.116.7:4009 , piece_id=FXQGF3UQN5CPFBS2HWD7BAXR4QIOF42MMK3VJQ7AK4THYJWOFFLQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:36.585900936 (316h27m35.137371397s) [seg_pos={0 0}, upload=278701]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Nov 22 12:31:36.828881025 (316h27m35.380351486s) [upload=278701]: Done uploading segments...
Nov 22 12:31:36.828915801 (316h27m35.380386262s) [upload=278701]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 5.865442673s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [..............................................X..X.XX.......] duration:
3.347507615s trace:
https://jaeger.dev.storj.io/trace/71781384eb27b231
FAIL (2025-11-22 12:40:21 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.347539896s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [............................................X...X...........] duration:
5.847224823s trace:
https://jaeger.dev.storj.io/trace/570ccd0ce1345814
FAIL (2025-11-22 12:40:15 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.847282903s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [............................................X..X..X.........] duration:
6.022488907s trace:
https://jaeger.dev.storj.io/trace/92df32505639913
FAIL (2025-11-22 12:40:12 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.022537009s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [................................................X...........] duration:
5.896211512s trace:
https://jaeger.dev.storj.io/trace/5ae3d2bf80495000
FAIL (2025-11-22 12:40:03 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Nov 22 12:31:07.276308940 (316h27m5.827779401s) [upload=278701]: Starting upload
Nov 22 12:31:07.276483523 (316h27m5.827953994s) [upload=278701]: Done starting upload
Nov 22 12:31:07.276562243 (316h27m5.828032704s) [upload=278701]: Uploading segments...
Nov 22 12:31:07.276950593 (316h27m5.828421054s) [upload=278701]: Got next segment. Inline: false
Nov 22 12:31:07.277027639 (316h27m5.8284981s) [upload=278701]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:31:07.277034181 (316h27m5.828504643s) [upload=278701]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:31:07.520212027 (316h27m6.071682488s) [seg_pos={0 0}, upload=278701]: Begin upload segment...
Nov 22 12:31:07.520269256 (316h27m6.071739717s) [seg_pos={0 0}, upload=278701]: Done begin upload segment.
Nov 22 12:31:07.520308992 (316h27m6.071779453s) [noise=true , address=145.79.227.27:4689 , piece_id=76S65KEZ7I2E76RR2JXOZ3ONGXCWSJA5XB7XQVFEEKGGNC34Z4EQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.520426746 (316h27m6.071897207s) [noise=true , address=145.79.227.21:4474 , piece_id=PCRWYQVCAZCPOILAEDHJGDBFKX4GEJ6JXCF7FUS7OVFRUTDPJI4Q, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.521175282 (316h27m6.072645743s) [noise=true , address=145.79.227.17:4972 , piece_id=UZK7LCIOBGGDNJZUVDBOLBJHGOIA4SUT2O3PD67PYQD3J2DGBJEA, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.521527573 (316h27m6.072998044s) [noise=true , address=145.79.227.23:4546 , piece_id=73UPPN2SYWNWEVWHCF4BRGKNO5GCRGOV3S5JZPX4MW75EJZRQOWQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.521694671 (316h27m6.073165132s) [noise=true , address=45.91.116.10:4082 , piece_id=GBDYXAWONOHQGBGM3NCRQYDSPFOAIHRJHBB2W6ZGJLMZUJLV65XA, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.522219410 (316h27m6.073689871s) [noise=true , address=145.79.227.20:4437 , piece_id=CZ32MB4AEZZ6YKCNWKRYALVAK7MT3KNXDSZCQTPUV2Q53NX6Y3BQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.522261240 (316h27m6.073731691s) [noise=true , address=145.79.227.16:4909 , piece_id=ZTL4ER5YJWSDGO5F5KLRUL5WRTA6WKVOJOERIBTRIIVEWXUY5Z2Q, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.522719151 (316h27m6.074189612s) [noise=true , address=145.79.227.26:4654 , piece_id=EM4RRGMPPNVR7VTXP3BED43FJFIUDZYVT4S5OUK2TRXZHYNMQ64Q, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.523017269 (316h27m6.07448772s) [noise=true , address=77.90.31.13:4627 , piece_id=Q7U5BFXHYQBBJQCU4EQGAR75WKL5QMBJ3LO34UYGO7QHLMDJBKWQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.523216949 (316h27m6.07468741s) [noise=true , address=145.79.227.25:4618 , piece_id=CV6M3CTPQR2MPQZFU6W34MFQERT6MMN3RRF2N3W35FGI464OSUUA, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.523744804 (316h27m6.075215265s) [noise=true , address=145.79.227.6:4258 , piece_id=B5D4TCVJORSAWDAU4DQ5VAV4CCRF5WUI24DUVUORAEKLAWGI7ABQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.523828153 (316h27m6.075298614s) [noise=true , address=145.79.227.24:4583 , piece_id=MLW4X34EWXKPOCFVI4VIQSL6RMUOJ5HRYHQ4HY35JIFT4ZFJ6F3Q, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.524255396 (316h27m6.075725858s) [noise=true , address=45.91.116.7:4009 , piece_id=FXQGF3UQN5CPFBS2HWD7BAXR4QIOF42MMK3VJQ7AK4THYJWOFFLQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.524662051 (316h27m6.076132512s) [noise=true , address=145.79.227.15:4874 , piece_id=TYAXIF3ZJQQSFGFANYF56DG4O5WBBXK4M7OT67E7ARYNZ663HESA, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.524746221 (316h27m6.076216682s) [noise=true , address=145.79.227.28:4726 , piece_id=3LBJV2B4CVXEMHCOPRW7DR5QBKPS6DFMPTHQSVNR4XRTZZVIGBQA, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.525227157 (316h27m6.076697608s) [noise=true , address=77.90.31.21:4879 , piece_id=GHTH55OGGXBAZZ5FGPCDYVHDYEETM7DNVTPSTTFFFRZMKZZ3XRXQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:09.134610631 (316h27m7.686081093s) [upload=278701]: Next returned nil segment
Nov 22 12:31:09.134628024 (316h27m7.686098495s) [upload=278701]: Waiting for error group managing segments...
Nov 22 12:31:09.502125778 (316h27m8.053596239s) [noise=true , address=145.79.227.16:4909 , piece_id=ZTL4ER5YJWSDGO5F5KLRUL5WRTA6WKVOJOERIBTRIIVEWXUY5Z2Q, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.503083742 (316h27m8.054554193s) [noise=true , address=145.79.227.20:4437 , piece_id=CZ32MB4AEZZ6YKCNWKRYALVAK7MT3KNXDSZCQTPUV2Q53NX6Y3BQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.517732716 (316h27m8.069203177s) [noise=true , address=145.79.227.25:4618 , piece_id=CV6M3CTPQR2MPQZFU6W34MFQERT6MMN3RRF2N3W35FGI464OSUUA, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.518029582 (316h27m8.069500043s) [noise=true , address=145.79.227.28:4726 , piece_id=3LBJV2B4CVXEMHCOPRW7DR5QBKPS6DFMPTHQSVNR4XRTZZVIGBQA, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.519935051 (316h27m8.071405512s) [noise=true , address=145.79.227.26:4654 , piece_id=EM4RRGMPPNVR7VTXP3BED43FJFIUDZYVT4S5OUK2TRXZHYNMQ64Q, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.520607661 (316h27m8.072078122s) [noise=true , address=145.79.227.27:4689 , piece_id=76S65KEZ7I2E76RR2JXOZ3ONGXCWSJA5XB7XQVFEEKGGNC34Z4EQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.521187736 (316h27m8.072658197s) [noise=true , address=145.79.227.23:4546 , piece_id=73UPPN2SYWNWEVWHCF4BRGKNO5GCRGOV3S5JZPX4MW75EJZRQOWQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.532974760 (316h27m8.084445221s) [noise=true , address=145.79.227.24:4583 , piece_id=MLW4X34EWXKPOCFVI4VIQSL6RMUOJ5HRYHQ4HY35JIFT4ZFJ6F3Q, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.533688800 (316h27m8.085159271s) [noise=true , address=145.79.227.6:4258 , piece_id=B5D4TCVJORSAWDAU4DQ5VAV4CCRF5WUI24DUVUORAEKLAWGI7ABQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.534460109 (316h27m8.08593057s) [noise=true , address=145.79.227.15:4874 , piece_id=TYAXIF3ZJQQSFGFANYF56DG4O5WBBXK4M7OT67E7ARYNZ663HESA, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.542955375 (316h27m8.094425826s) [noise=true , address=145.79.227.17:4972 , piece_id=UZK7LCIOBGGDNJZUVDBOLBJHGOIA4SUT2O3PD67PYQD3J2DGBJEA, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.563098982 (316h27m8.114569443s) [noise=true , address=145.79.227.21:4474 , piece_id=PCRWYQVCAZCPOILAEDHJGDBFKX4GEJ6JXCF7FUS7OVFRUTDPJI4Q, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:36.585475346 (316h27m35.136945807s) [noise=true , address=45.91.116.10:4082 , piece_id=GBDYXAWONOHQGBGM3NCRQYDSPFOAIHRJHBB2W6ZGJLMZUJLV65XA, seg_pos={0 0}, upload=278701]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: read tcp 188.166.225.170:54326->45.91.116.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:36.585499923 (316h27m35.136970384s) [noise=true , address=77.90.31.21:4879 , piece_id=GHTH55OGGXBAZZ5FGPCDYVHDYEETM7DNVTPSTTFFFRZMKZZ3XRXQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: rpc: dial tcp 77.90.31.21:4879: i/o timeout; context deadline exceeded
Nov 22 12:31:36.585749648 (316h27m35.137220119s) [noise=true , address=77.90.31.13:4627 , piece_id=Q7U5BFXHYQBBJQCU4EQGAR75WKL5QMBJ3LO34UYGO7QHLMDJBKWQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:36.585848086 (316h27m35.137318547s) [noise=true , address=45.91.116.7:4009 , piece_id=FXQGF3UQN5CPFBS2HWD7BAXR4QIOF42MMK3VJQ7AK4THYJWOFFLQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:36.585900936 (316h27m35.137371397s) [seg_pos={0 0}, upload=278701]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Nov 22 12:31:36.828881025 (316h27m35.380351486s) [upload=278701]: Done uploading segments...
Nov 22 12:31:36.828915801 (316h27m35.380386262s) [upload=278701]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 5.89624179s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [.............................................X.....X........] duration:
5.748919292s trace:
https://jaeger.dev.storj.io/trace/6c800b4c3a84b58c
FAIL (2025-11-22 12:39:53 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 5)
======LOGS======
N5RGYNBMYFXXBJDBTUCN37EAT7WBRNTX2YQA, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.890374814 (316h29m10.441845275s) [noise=true , address=145.79.227.32:4858 , piece_id=K5L3LEUKZWHS36XJHNNY4TYRXCP4DG5UAKCZXPJJTSG26CKUI4HQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.890535981 (316h29m10.442006442s) [noise=true , address=77.90.31.21:4890 , piece_id=DUEFPJUTYMD564Z4III63LZM6L4IL2ZTGM3QDZ6CG7WV5APHBNRA, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.890774064 (316h29m10.442244525s) [noise=true , address=45.91.116.14:4213 , piece_id=HPFISTE2N3B4RBDPAZAGVZJ5WYLCPHTCYH4WMJ56IWEHR3ZBEMOQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.890934851 (316h29m10.442405312s) [noise=true , address=45.91.116.6:4000 , piece_id=T4CI3KKNMFPT3ZIODHE7BQ4DVHVH2QTSO6OB7QZPL5SNPXQJE6TA, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.891167002 (316h29m10.442637463s) [noise=true , address=77.90.31.17:4787 , piece_id=IYIEVH6CARRYCARKOLKWVBICQDNDRCTGQMWF3VJZ6YBC3GO5OWHQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.891369828 (316h29m10.442840289s) [noise=true , address=45.91.116.7:4039 , piece_id=HSRY63FXTEPCP2T46V77RVSC24GGRBAYLLU7DTKOLWGVOYHF6YVA, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.891556875 (316h29m10.443027336s) [noise=true , address=77.90.31.11:4572 , piece_id=G4XSIPSWOLSEOGGFA2NLOSDCGTLZPI3P5LDNNLB5RGMU5OQGWXGQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.891896812 (316h29m10.443367273s) [noise=true , address=77.90.31.19:4863 , piece_id=3PM3Q6X3ADKX2RKIIFDMK57LTJNJHNFK4UYPKFK3ZEQLYQ23EP3A, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.892019235 (316h29m10.443489696s) [noise=true , address=77.90.31.10:4524 , piece_id=Z2IGDHPDU6JAR5CAFQBGYHO3TQ66K4BZOAMXSC7X3RFWLGHOOMTQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.892316521 (316h29m10.443786982s) [noise=true , address=77.90.31.13:4644 , piece_id=UPDZGC3CMJXA3AOJX65VLASU37NLFGPIJLCAPN32TEQB5QHTU7EQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.892682318 (316h29m10.444152779s) [noise=true , address=45.91.116.9:4274 , piece_id=MXWRAXYLBBGLUVTA5GP3JVX2IRM2UI2IC35D7KX3XMFEBXLIDX5A, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:13.353758506 (316h29m11.905228957s) [upload=278731]: Next returned nil segment
Nov 22 12:33:13.353772043 (316h29m11.905242504s) [upload=278731]: Waiting for error group managing segments...
Nov 22 12:33:13.751703718 (316h29m12.303174169s) [noise=true , address=145.79.227.32:4858 , piece_id=K5L3LEUKZWHS36XJHNNY4TYRXCP4DG5UAKCZXPJJTSG26CKUI4HQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil>
Nov 22 12:33:13.753660293 (316h29m12.305130764s) [noise=true , address=145.79.227.26:4674 , piece_id=UP7VIVRDRL6KERJSX3VLWTKGW3ESGCQ7BLYCBXGBB5UNIDE6OHQQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil>
Nov 22 12:33:13.760046123 (316h29m12.311516574s) [noise=true , address=145.79.227.3:4167 , piece_id=4STIVXGU7SOXWAIPG2PXWBLNDD7EYQ56E2DBN2UZ2AZ3TZSXCTIQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil>
Nov 22 12:33:57.373975014 (316h29m55.925445475s) [noise=true , address=77.90.31.13:4644 , piece_id=UPDZGC3CMJXA3AOJX65VLASU37NLFGPIJLCAPN32TEQB5QHTU7EQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil>
Nov 22 12:34:06.122275078 (316h30m4.673745539s) [noise=true , address=45.91.116.13:4181 , piece_id=MQNJEBQFZMHNDKAH2K3JEA7YBK2UK5BGSNCJF3MASF3CKVA5VD6Q, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil>
Nov 22 12:34:10.949171331 (316h30m9.500641792s) [noise=true , address=77.90.31.21:4890 , piece_id=DUEFPJUTYMD564Z4III63LZM6L4IL2ZTGM3QDZ6CG7WV5APHBNRA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1iE3mDnEhGDsMBtmUXC4EwvSrpyHLoybzVXkC9eoFuYEMHeBrd, address:77.90.31.21:4890): protocol: read tcp 188.166.225.170:55900->77.90.31.21:4890: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.949753650 (316h30m9.501224111s) [noise=true , address=45.91.116.12:4237 , piece_id=OG6XREYEGOYVFFMXCGBERRAIEMI2QMUGSBJBYP7GJQZ7FOAI4BIA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1FMjHdNGV42a9QZyMmSPTJWQrA8CdSRvXqrimnt6UGVreCzmgF, address:45.91.116.12:4237): protocol: rpc: dial tcp 45.91.116.12:4237: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.950066235 (316h30m9.501536706s) [noise=true , address=77.90.31.19:4863 , piece_id=3PM3Q6X3ADKX2RKIIFDMK57LTJNJHNFK4UYPKFK3ZEQLYQ23EP3A, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:124x1cbCPJRK8YkgSP6UzHymoLu1E1znnfg5WBiD89kSDLi8JeL, address:77.90.31.19:4863): protocol: write tcp 188.166.225.170:45968->77.90.31.19:4863: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.950246558 (316h30m9.50171702s) [noise=true , address=77.90.31.10:4524 , piece_id=Z2IGDHPDU6JAR5CAFQBGYHO3TQ66K4BZOAMXSC7X3RFWLGHOOMTQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12uN7CznoK4AD8FqFkNUndawdGJ9rmjAe7oNnjJkV844ScY87qe, address:77.90.31.10:4524): protocol: write tcp 188.166.225.170:40266->77.90.31.10:4524: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.950593058 (316h30m9.502063529s) [noise=true , address=77.90.31.14:4690 , piece_id=RJ3KXY3HURT7Q4HZYTOLSAL7Y6VMDGGRIQP6XHE4YB6OAA3PJRAA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.950853535 (316h30m9.502324006s) [noise=true , address=45.91.116.14:4213 , piece_id=HPFISTE2N3B4RBDPAZAGVZJ5WYLCPHTCYH4WMJ56IWEHR3ZBEMOQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12tQRqowH4aDr9h3s1WLs1o7na1ueiafwgUeeHfAB6g7mcE6e1t, address:45.91.116.14:4213): protocol: rpc: dial tcp 45.91.116.14:4213: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.951186438 (316h30m9.502656889s) [noise=true , address=45.91.116.6:4000 , piece_id=T4CI3KKNMFPT3ZIODHE7BQ4DVHVH2QTSO6OB7QZPL5SNPXQJE6TA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12CuCqHJYbanbuaEMHvtcuQXhvcdahD8eyT2fKDhqtxdSZ6ES4f, address:45.91.116.6:4000): protocol: rpc: dial tcp 45.91.116.6:4000: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.951409864 (316h30m9.502880325s) [noise=true , address=45.91.116.7:4039 , piece_id=HSRY63FXTEPCP2T46V77RVSC24GGRBAYLLU7DTKOLWGVOYHF6YVA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:16oq1Yv7JfHrg4gdHfhwuMj4UsWeQQPayKV6Pj1NY2CgJhuPr2, address:45.91.116.7:4039): protocol: write tcp 188.166.225.170:36904->45.91.116.7:4039: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.951660191 (316h30m9.503130652s) [noise=true , address=77.90.31.11:4572 , piece_id=G4XSIPSWOLSEOGGFA2NLOSDCGTLZPI3P5LDNNLB5RGMU5OQGWXGQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1ZAfYZ4wgpZVvatj6wtHona3hTaPcLxAmXC1DT2uKQyo8nLQQ7, address:77.90.31.11:4572): protocol: rpc: dial tcp 77.90.31.11:4572: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.952019855 (316h30m9.503490316s) [noise=true , address=45.91.116.9:4274 , piece_id=MXWRAXYLBBGLUVTA5GP3JVX2IRM2UI2IC35D7KX3XMFEBXLIDX5A, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12g6jFa38dTz5UdgsQaEuM2MSH1H7cxaysWsVmbHkM2Dd2W2iDb, address:45.91.116.9:4274): protocol: rpc: dial tcp 45.91.116.9:4274: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.952290972 (316h30m9.503761443s) [noise=true , address=77.90.31.15:4720 , piece_id=BDQUF6C5TT3ASE3DIN6SSVJHOZAJK4ZEW33OIYPTGB4LDVRELSUA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12VbsxFqMB3vVphSAJqYpbBzua7Goq45E6ovJSaExkpLtfkcc74, address:77.90.31.15:4720): protocol: rpc: dial tcp 77.90.31.15:4720: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.952544654 (316h30m9.504015105s) [noise=true , address=45.91.116.10:4110 , piece_id=H6FYA6K4N4AQYU3MN5RGYNBMYFXXBJDBTUCN37EAT7WBRNTX2YQA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:123KwusATu1ZKtNNVHZZZBE67ZEnVb8puyfQidBi3fEWREqdf1h, address:45.91.116.10:4110): protocol: rpc: dial tcp 45.91.116.10:4110: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.953548155 (316h30m9.505018617s) [noise=true , address=77.90.31.17:4787 , piece_id=IYIEVH6CARRYCARKOLKWVBICQDNDRCTGQMWF3VJZ6YBC3GO5OWHQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1Lietonb5aVbaZQt6wQ2EoxwJkSEn9oL9Dwyrc9mTyomEKikNt, address:77.90.31.17:4787): protocol: read tcp 188.166.225.170:57908->77.90.31.17:4787: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.953579816 (316h30m9.505050277s) [seg_pos={0 0}, upload=278731]: Done waiting for segment. successful: 5 optimal: 14 errs: <nil>
Nov 22 12:34:11.195719123 (316h30m9.747189584s) [upload=278731]: Done uploading segments...
Nov 22 12:34:11.195932840 (316h30m9.747403301s) [upload=278731]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 5)
Duration: 5.748947746s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [.............................................X...X..........] duration:
5.835902224s trace:
https://jaeger.dev.storj.io/trace/570ccd0ce1345814
FAIL (2025-11-22 12:39:19 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.835928844s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [...............................................X..X.XX......] duration:
3.434852545s trace:
https://jaeger.dev.storj.io/trace/71781384eb27b231
FAIL (2025-11-22 12:39:15 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.434884115s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [..............................................X.....X.......] duration:
5.914161704s trace:
https://jaeger.dev.storj.io/trace/6c800b4c3a84b58c
FAIL (2025-11-22 12:39:03 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 5)
======LOGS======
N5RGYNBMYFXXBJDBTUCN37EAT7WBRNTX2YQA, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.890374814 (316h29m10.441845275s) [noise=true , address=145.79.227.32:4858 , piece_id=K5L3LEUKZWHS36XJHNNY4TYRXCP4DG5UAKCZXPJJTSG26CKUI4HQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.890535981 (316h29m10.442006442s) [noise=true , address=77.90.31.21:4890 , piece_id=DUEFPJUTYMD564Z4III63LZM6L4IL2ZTGM3QDZ6CG7WV5APHBNRA, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.890774064 (316h29m10.442244525s) [noise=true , address=45.91.116.14:4213 , piece_id=HPFISTE2N3B4RBDPAZAGVZJ5WYLCPHTCYH4WMJ56IWEHR3ZBEMOQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.890934851 (316h29m10.442405312s) [noise=true , address=45.91.116.6:4000 , piece_id=T4CI3KKNMFPT3ZIODHE7BQ4DVHVH2QTSO6OB7QZPL5SNPXQJE6TA, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.891167002 (316h29m10.442637463s) [noise=true , address=77.90.31.17:4787 , piece_id=IYIEVH6CARRYCARKOLKWVBICQDNDRCTGQMWF3VJZ6YBC3GO5OWHQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.891369828 (316h29m10.442840289s) [noise=true , address=45.91.116.7:4039 , piece_id=HSRY63FXTEPCP2T46V77RVSC24GGRBAYLLU7DTKOLWGVOYHF6YVA, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.891556875 (316h29m10.443027336s) [noise=true , address=77.90.31.11:4572 , piece_id=G4XSIPSWOLSEOGGFA2NLOSDCGTLZPI3P5LDNNLB5RGMU5OQGWXGQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.891896812 (316h29m10.443367273s) [noise=true , address=77.90.31.19:4863 , piece_id=3PM3Q6X3ADKX2RKIIFDMK57LTJNJHNFK4UYPKFK3ZEQLYQ23EP3A, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.892019235 (316h29m10.443489696s) [noise=true , address=77.90.31.10:4524 , piece_id=Z2IGDHPDU6JAR5CAFQBGYHO3TQ66K4BZOAMXSC7X3RFWLGHOOMTQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.892316521 (316h29m10.443786982s) [noise=true , address=77.90.31.13:4644 , piece_id=UPDZGC3CMJXA3AOJX65VLASU37NLFGPIJLCAPN32TEQB5QHTU7EQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.892682318 (316h29m10.444152779s) [noise=true , address=45.91.116.9:4274 , piece_id=MXWRAXYLBBGLUVTA5GP3JVX2IRM2UI2IC35D7KX3XMFEBXLIDX5A, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:13.353758506 (316h29m11.905228957s) [upload=278731]: Next returned nil segment
Nov 22 12:33:13.353772043 (316h29m11.905242504s) [upload=278731]: Waiting for error group managing segments...
Nov 22 12:33:13.751703718 (316h29m12.303174169s) [noise=true , address=145.79.227.32:4858 , piece_id=K5L3LEUKZWHS36XJHNNY4TYRXCP4DG5UAKCZXPJJTSG26CKUI4HQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil>
Nov 22 12:33:13.753660293 (316h29m12.305130764s) [noise=true , address=145.79.227.26:4674 , piece_id=UP7VIVRDRL6KERJSX3VLWTKGW3ESGCQ7BLYCBXGBB5UNIDE6OHQQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil>
Nov 22 12:33:13.760046123 (316h29m12.311516574s) [noise=true , address=145.79.227.3:4167 , piece_id=4STIVXGU7SOXWAIPG2PXWBLNDD7EYQ56E2DBN2UZ2AZ3TZSXCTIQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil>
Nov 22 12:33:57.373975014 (316h29m55.925445475s) [noise=true , address=77.90.31.13:4644 , piece_id=UPDZGC3CMJXA3AOJX65VLASU37NLFGPIJLCAPN32TEQB5QHTU7EQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil>
Nov 22 12:34:06.122275078 (316h30m4.673745539s) [noise=true , address=45.91.116.13:4181 , piece_id=MQNJEBQFZMHNDKAH2K3JEA7YBK2UK5BGSNCJF3MASF3CKVA5VD6Q, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil>
Nov 22 12:34:10.949171331 (316h30m9.500641792s) [noise=true , address=77.90.31.21:4890 , piece_id=DUEFPJUTYMD564Z4III63LZM6L4IL2ZTGM3QDZ6CG7WV5APHBNRA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1iE3mDnEhGDsMBtmUXC4EwvSrpyHLoybzVXkC9eoFuYEMHeBrd, address:77.90.31.21:4890): protocol: read tcp 188.166.225.170:55900->77.90.31.21:4890: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.949753650 (316h30m9.501224111s) [noise=true , address=45.91.116.12:4237 , piece_id=OG6XREYEGOYVFFMXCGBERRAIEMI2QMUGSBJBYP7GJQZ7FOAI4BIA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1FMjHdNGV42a9QZyMmSPTJWQrA8CdSRvXqrimnt6UGVreCzmgF, address:45.91.116.12:4237): protocol: rpc: dial tcp 45.91.116.12:4237: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.950066235 (316h30m9.501536706s) [noise=true , address=77.90.31.19:4863 , piece_id=3PM3Q6X3ADKX2RKIIFDMK57LTJNJHNFK4UYPKFK3ZEQLYQ23EP3A, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:124x1cbCPJRK8YkgSP6UzHymoLu1E1znnfg5WBiD89kSDLi8JeL, address:77.90.31.19:4863): protocol: write tcp 188.166.225.170:45968->77.90.31.19:4863: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.950246558 (316h30m9.50171702s) [noise=true , address=77.90.31.10:4524 , piece_id=Z2IGDHPDU6JAR5CAFQBGYHO3TQ66K4BZOAMXSC7X3RFWLGHOOMTQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12uN7CznoK4AD8FqFkNUndawdGJ9rmjAe7oNnjJkV844ScY87qe, address:77.90.31.10:4524): protocol: write tcp 188.166.225.170:40266->77.90.31.10:4524: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.950593058 (316h30m9.502063529s) [noise=true , address=77.90.31.14:4690 , piece_id=RJ3KXY3HURT7Q4HZYTOLSAL7Y6VMDGGRIQP6XHE4YB6OAA3PJRAA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.950853535 (316h30m9.502324006s) [noise=true , address=45.91.116.14:4213 , piece_id=HPFISTE2N3B4RBDPAZAGVZJ5WYLCPHTCYH4WMJ56IWEHR3ZBEMOQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12tQRqowH4aDr9h3s1WLs1o7na1ueiafwgUeeHfAB6g7mcE6e1t, address:45.91.116.14:4213): protocol: rpc: dial tcp 45.91.116.14:4213: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.951186438 (316h30m9.502656889s) [noise=true , address=45.91.116.6:4000 , piece_id=T4CI3KKNMFPT3ZIODHE7BQ4DVHVH2QTSO6OB7QZPL5SNPXQJE6TA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12CuCqHJYbanbuaEMHvtcuQXhvcdahD8eyT2fKDhqtxdSZ6ES4f, address:45.91.116.6:4000): protocol: rpc: dial tcp 45.91.116.6:4000: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.951409864 (316h30m9.502880325s) [noise=true , address=45.91.116.7:4039 , piece_id=HSRY63FXTEPCP2T46V77RVSC24GGRBAYLLU7DTKOLWGVOYHF6YVA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:16oq1Yv7JfHrg4gdHfhwuMj4UsWeQQPayKV6Pj1NY2CgJhuPr2, address:45.91.116.7:4039): protocol: write tcp 188.166.225.170:36904->45.91.116.7:4039: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.951660191 (316h30m9.503130652s) [noise=true , address=77.90.31.11:4572 , piece_id=G4XSIPSWOLSEOGGFA2NLOSDCGTLZPI3P5LDNNLB5RGMU5OQGWXGQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1ZAfYZ4wgpZVvatj6wtHona3hTaPcLxAmXC1DT2uKQyo8nLQQ7, address:77.90.31.11:4572): protocol: rpc: dial tcp 77.90.31.11:4572: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.952019855 (316h30m9.503490316s) [noise=true , address=45.91.116.9:4274 , piece_id=MXWRAXYLBBGLUVTA5GP3JVX2IRM2UI2IC35D7KX3XMFEBXLIDX5A, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12g6jFa38dTz5UdgsQaEuM2MSH1H7cxaysWsVmbHkM2Dd2W2iDb, address:45.91.116.9:4274): protocol: rpc: dial tcp 45.91.116.9:4274: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.952290972 (316h30m9.503761443s) [noise=true , address=77.90.31.15:4720 , piece_id=BDQUF6C5TT3ASE3DIN6SSVJHOZAJK4ZEW33OIYPTGB4LDVRELSUA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12VbsxFqMB3vVphSAJqYpbBzua7Goq45E6ovJSaExkpLtfkcc74, address:77.90.31.15:4720): protocol: rpc: dial tcp 77.90.31.15:4720: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.952544654 (316h30m9.504015105s) [noise=true , address=45.91.116.10:4110 , piece_id=H6FYA6K4N4AQYU3MN5RGYNBMYFXXBJDBTUCN37EAT7WBRNTX2YQA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:123KwusATu1ZKtNNVHZZZBE67ZEnVb8puyfQidBi3fEWREqdf1h, address:45.91.116.10:4110): protocol: rpc: dial tcp 45.91.116.10:4110: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.953548155 (316h30m9.505018617s) [noise=true , address=77.90.31.17:4787 , piece_id=IYIEVH6CARRYCARKOLKWVBICQDNDRCTGQMWF3VJZ6YBC3GO5OWHQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1Lietonb5aVbaZQt6wQ2EoxwJkSEn9oL9Dwyrc9mTyomEKikNt, address:77.90.31.17:4787): protocol: read tcp 188.166.225.170:57908->77.90.31.17:4787: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.953579816 (316h30m9.505050277s) [seg_pos={0 0}, upload=278731]: Done waiting for segment. successful: 5 optimal: 14 errs: <nil>
Nov 22 12:34:11.195719123 (316h30m9.747189584s) [upload=278731]: Done uploading segments...
Nov 22 12:34:11.195932840 (316h30m9.747403301s) [upload=278731]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 5)
Duration: 5.914194396s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.............................................X..X..X........] duration:
6.120527913s trace:
https://jaeger.dev.storj.io/trace/92df32505639913
FAIL (2025-11-22 12:39:03 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.120554593s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.................................................X..........] duration:
5.770225193s trace:
https://jaeger.dev.storj.io/trace/5ae3d2bf80495000
FAIL (2025-11-22 12:38:50 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Nov 22 12:31:07.276308940 (316h27m5.827779401s) [upload=278701]: Starting upload
Nov 22 12:31:07.276483523 (316h27m5.827953994s) [upload=278701]: Done starting upload
Nov 22 12:31:07.276562243 (316h27m5.828032704s) [upload=278701]: Uploading segments...
Nov 22 12:31:07.276950593 (316h27m5.828421054s) [upload=278701]: Got next segment. Inline: false
Nov 22 12:31:07.277027639 (316h27m5.8284981s) [upload=278701]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:31:07.277034181 (316h27m5.828504643s) [upload=278701]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:31:07.520212027 (316h27m6.071682488s) [seg_pos={0 0}, upload=278701]: Begin upload segment...
Nov 22 12:31:07.520269256 (316h27m6.071739717s) [seg_pos={0 0}, upload=278701]: Done begin upload segment.
Nov 22 12:31:07.520308992 (316h27m6.071779453s) [noise=true , address=145.79.227.27:4689 , piece_id=76S65KEZ7I2E76RR2JXOZ3ONGXCWSJA5XB7XQVFEEKGGNC34Z4EQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.520426746 (316h27m6.071897207s) [noise=true , address=145.79.227.21:4474 , piece_id=PCRWYQVCAZCPOILAEDHJGDBFKX4GEJ6JXCF7FUS7OVFRUTDPJI4Q, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.521175282 (316h27m6.072645743s) [noise=true , address=145.79.227.17:4972 , piece_id=UZK7LCIOBGGDNJZUVDBOLBJHGOIA4SUT2O3PD67PYQD3J2DGBJEA, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.521527573 (316h27m6.072998044s) [noise=true , address=145.79.227.23:4546 , piece_id=73UPPN2SYWNWEVWHCF4BRGKNO5GCRGOV3S5JZPX4MW75EJZRQOWQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.521694671 (316h27m6.073165132s) [noise=true , address=45.91.116.10:4082 , piece_id=GBDYXAWONOHQGBGM3NCRQYDSPFOAIHRJHBB2W6ZGJLMZUJLV65XA, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.522219410 (316h27m6.073689871s) [noise=true , address=145.79.227.20:4437 , piece_id=CZ32MB4AEZZ6YKCNWKRYALVAK7MT3KNXDSZCQTPUV2Q53NX6Y3BQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.522261240 (316h27m6.073731691s) [noise=true , address=145.79.227.16:4909 , piece_id=ZTL4ER5YJWSDGO5F5KLRUL5WRTA6WKVOJOERIBTRIIVEWXUY5Z2Q, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.522719151 (316h27m6.074189612s) [noise=true , address=145.79.227.26:4654 , piece_id=EM4RRGMPPNVR7VTXP3BED43FJFIUDZYVT4S5OUK2TRXZHYNMQ64Q, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.523017269 (316h27m6.07448772s) [noise=true , address=77.90.31.13:4627 , piece_id=Q7U5BFXHYQBBJQCU4EQGAR75WKL5QMBJ3LO34UYGO7QHLMDJBKWQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.523216949 (316h27m6.07468741s) [noise=true , address=145.79.227.25:4618 , piece_id=CV6M3CTPQR2MPQZFU6W34MFQERT6MMN3RRF2N3W35FGI464OSUUA, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.523744804 (316h27m6.075215265s) [noise=true , address=145.79.227.6:4258 , piece_id=B5D4TCVJORSAWDAU4DQ5VAV4CCRF5WUI24DUVUORAEKLAWGI7ABQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.523828153 (316h27m6.075298614s) [noise=true , address=145.79.227.24:4583 , piece_id=MLW4X34EWXKPOCFVI4VIQSL6RMUOJ5HRYHQ4HY35JIFT4ZFJ6F3Q, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.524255396 (316h27m6.075725858s) [noise=true , address=45.91.116.7:4009 , piece_id=FXQGF3UQN5CPFBS2HWD7BAXR4QIOF42MMK3VJQ7AK4THYJWOFFLQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.524662051 (316h27m6.076132512s) [noise=true , address=145.79.227.15:4874 , piece_id=TYAXIF3ZJQQSFGFANYF56DG4O5WBBXK4M7OT67E7ARYNZ663HESA, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.524746221 (316h27m6.076216682s) [noise=true , address=145.79.227.28:4726 , piece_id=3LBJV2B4CVXEMHCOPRW7DR5QBKPS6DFMPTHQSVNR4XRTZZVIGBQA, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.525227157 (316h27m6.076697608s) [noise=true , address=77.90.31.21:4879 , piece_id=GHTH55OGGXBAZZ5FGPCDYVHDYEETM7DNVTPSTTFFFRZMKZZ3XRXQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:09.134610631 (316h27m7.686081093s) [upload=278701]: Next returned nil segment
Nov 22 12:31:09.134628024 (316h27m7.686098495s) [upload=278701]: Waiting for error group managing segments...
Nov 22 12:31:09.502125778 (316h27m8.053596239s) [noise=true , address=145.79.227.16:4909 , piece_id=ZTL4ER5YJWSDGO5F5KLRUL5WRTA6WKVOJOERIBTRIIVEWXUY5Z2Q, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.503083742 (316h27m8.054554193s) [noise=true , address=145.79.227.20:4437 , piece_id=CZ32MB4AEZZ6YKCNWKRYALVAK7MT3KNXDSZCQTPUV2Q53NX6Y3BQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.517732716 (316h27m8.069203177s) [noise=true , address=145.79.227.25:4618 , piece_id=CV6M3CTPQR2MPQZFU6W34MFQERT6MMN3RRF2N3W35FGI464OSUUA, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.518029582 (316h27m8.069500043s) [noise=true , address=145.79.227.28:4726 , piece_id=3LBJV2B4CVXEMHCOPRW7DR5QBKPS6DFMPTHQSVNR4XRTZZVIGBQA, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.519935051 (316h27m8.071405512s) [noise=true , address=145.79.227.26:4654 , piece_id=EM4RRGMPPNVR7VTXP3BED43FJFIUDZYVT4S5OUK2TRXZHYNMQ64Q, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.520607661 (316h27m8.072078122s) [noise=true , address=145.79.227.27:4689 , piece_id=76S65KEZ7I2E76RR2JXOZ3ONGXCWSJA5XB7XQVFEEKGGNC34Z4EQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.521187736 (316h27m8.072658197s) [noise=true , address=145.79.227.23:4546 , piece_id=73UPPN2SYWNWEVWHCF4BRGKNO5GCRGOV3S5JZPX4MW75EJZRQOWQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.532974760 (316h27m8.084445221s) [noise=true , address=145.79.227.24:4583 , piece_id=MLW4X34EWXKPOCFVI4VIQSL6RMUOJ5HRYHQ4HY35JIFT4ZFJ6F3Q, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.533688800 (316h27m8.085159271s) [noise=true , address=145.79.227.6:4258 , piece_id=B5D4TCVJORSAWDAU4DQ5VAV4CCRF5WUI24DUVUORAEKLAWGI7ABQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.534460109 (316h27m8.08593057s) [noise=true , address=145.79.227.15:4874 , piece_id=TYAXIF3ZJQQSFGFANYF56DG4O5WBBXK4M7OT67E7ARYNZ663HESA, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.542955375 (316h27m8.094425826s) [noise=true , address=145.79.227.17:4972 , piece_id=UZK7LCIOBGGDNJZUVDBOLBJHGOIA4SUT2O3PD67PYQD3J2DGBJEA, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.563098982 (316h27m8.114569443s) [noise=true , address=145.79.227.21:4474 , piece_id=PCRWYQVCAZCPOILAEDHJGDBFKX4GEJ6JXCF7FUS7OVFRUTDPJI4Q, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:36.585475346 (316h27m35.136945807s) [noise=true , address=45.91.116.10:4082 , piece_id=GBDYXAWONOHQGBGM3NCRQYDSPFOAIHRJHBB2W6ZGJLMZUJLV65XA, seg_pos={0 0}, upload=278701]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: read tcp 188.166.225.170:54326->45.91.116.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:36.585499923 (316h27m35.136970384s) [noise=true , address=77.90.31.21:4879 , piece_id=GHTH55OGGXBAZZ5FGPCDYVHDYEETM7DNVTPSTTFFFRZMKZZ3XRXQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: rpc: dial tcp 77.90.31.21:4879: i/o timeout; context deadline exceeded
Nov 22 12:31:36.585749648 (316h27m35.137220119s) [noise=true , address=77.90.31.13:4627 , piece_id=Q7U5BFXHYQBBJQCU4EQGAR75WKL5QMBJ3LO34UYGO7QHLMDJBKWQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:36.585848086 (316h27m35.137318547s) [noise=true , address=45.91.116.7:4009 , piece_id=FXQGF3UQN5CPFBS2HWD7BAXR4QIOF42MMK3VJQ7AK4THYJWOFFLQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:36.585900936 (316h27m35.137371397s) [seg_pos={0 0}, upload=278701]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Nov 22 12:31:36.828881025 (316h27m35.380351486s) [upload=278701]: Done uploading segments...
Nov 22 12:31:36.828915801 (316h27m35.380386262s) [upload=278701]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 5.770256232s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [..............................................X..X..X.......] duration:
6.055198146s trace:
https://jaeger.dev.storj.io/trace/92df32505639913
FAIL (2025-11-22 12:38:39 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.055228143s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [..............................................X...X.........] duration:
5.860070936s trace:
https://jaeger.dev.storj.io/trace/570ccd0ce1345814
FAIL (2025-11-22 12:38:38 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.860103688s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [...............................................X.....X......] duration:
5.684912941s trace:
https://jaeger.dev.storj.io/trace/6c800b4c3a84b58c
FAIL (2025-11-22 12:38:35 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 5)
======LOGS======
N5RGYNBMYFXXBJDBTUCN37EAT7WBRNTX2YQA, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.890374814 (316h29m10.441845275s) [noise=true , address=145.79.227.32:4858 , piece_id=K5L3LEUKZWHS36XJHNNY4TYRXCP4DG5UAKCZXPJJTSG26CKUI4HQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.890535981 (316h29m10.442006442s) [noise=true , address=77.90.31.21:4890 , piece_id=DUEFPJUTYMD564Z4III63LZM6L4IL2ZTGM3QDZ6CG7WV5APHBNRA, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.890774064 (316h29m10.442244525s) [noise=true , address=45.91.116.14:4213 , piece_id=HPFISTE2N3B4RBDPAZAGVZJ5WYLCPHTCYH4WMJ56IWEHR3ZBEMOQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.890934851 (316h29m10.442405312s) [noise=true , address=45.91.116.6:4000 , piece_id=T4CI3KKNMFPT3ZIODHE7BQ4DVHVH2QTSO6OB7QZPL5SNPXQJE6TA, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.891167002 (316h29m10.442637463s) [noise=true , address=77.90.31.17:4787 , piece_id=IYIEVH6CARRYCARKOLKWVBICQDNDRCTGQMWF3VJZ6YBC3GO5OWHQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.891369828 (316h29m10.442840289s) [noise=true , address=45.91.116.7:4039 , piece_id=HSRY63FXTEPCP2T46V77RVSC24GGRBAYLLU7DTKOLWGVOYHF6YVA, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.891556875 (316h29m10.443027336s) [noise=true , address=77.90.31.11:4572 , piece_id=G4XSIPSWOLSEOGGFA2NLOSDCGTLZPI3P5LDNNLB5RGMU5OQGWXGQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.891896812 (316h29m10.443367273s) [noise=true , address=77.90.31.19:4863 , piece_id=3PM3Q6X3ADKX2RKIIFDMK57LTJNJHNFK4UYPKFK3ZEQLYQ23EP3A, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.892019235 (316h29m10.443489696s) [noise=true , address=77.90.31.10:4524 , piece_id=Z2IGDHPDU6JAR5CAFQBGYHO3TQ66K4BZOAMXSC7X3RFWLGHOOMTQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.892316521 (316h29m10.443786982s) [noise=true , address=77.90.31.13:4644 , piece_id=UPDZGC3CMJXA3AOJX65VLASU37NLFGPIJLCAPN32TEQB5QHTU7EQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.892682318 (316h29m10.444152779s) [noise=true , address=45.91.116.9:4274 , piece_id=MXWRAXYLBBGLUVTA5GP3JVX2IRM2UI2IC35D7KX3XMFEBXLIDX5A, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:13.353758506 (316h29m11.905228957s) [upload=278731]: Next returned nil segment
Nov 22 12:33:13.353772043 (316h29m11.905242504s) [upload=278731]: Waiting for error group managing segments...
Nov 22 12:33:13.751703718 (316h29m12.303174169s) [noise=true , address=145.79.227.32:4858 , piece_id=K5L3LEUKZWHS36XJHNNY4TYRXCP4DG5UAKCZXPJJTSG26CKUI4HQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil>
Nov 22 12:33:13.753660293 (316h29m12.305130764s) [noise=true , address=145.79.227.26:4674 , piece_id=UP7VIVRDRL6KERJSX3VLWTKGW3ESGCQ7BLYCBXGBB5UNIDE6OHQQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil>
Nov 22 12:33:13.760046123 (316h29m12.311516574s) [noise=true , address=145.79.227.3:4167 , piece_id=4STIVXGU7SOXWAIPG2PXWBLNDD7EYQ56E2DBN2UZ2AZ3TZSXCTIQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil>
Nov 22 12:33:57.373975014 (316h29m55.925445475s) [noise=true , address=77.90.31.13:4644 , piece_id=UPDZGC3CMJXA3AOJX65VLASU37NLFGPIJLCAPN32TEQB5QHTU7EQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil>
Nov 22 12:34:06.122275078 (316h30m4.673745539s) [noise=true , address=45.91.116.13:4181 , piece_id=MQNJEBQFZMHNDKAH2K3JEA7YBK2UK5BGSNCJF3MASF3CKVA5VD6Q, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil>
Nov 22 12:34:10.949171331 (316h30m9.500641792s) [noise=true , address=77.90.31.21:4890 , piece_id=DUEFPJUTYMD564Z4III63LZM6L4IL2ZTGM3QDZ6CG7WV5APHBNRA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1iE3mDnEhGDsMBtmUXC4EwvSrpyHLoybzVXkC9eoFuYEMHeBrd, address:77.90.31.21:4890): protocol: read tcp 188.166.225.170:55900->77.90.31.21:4890: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.949753650 (316h30m9.501224111s) [noise=true , address=45.91.116.12:4237 , piece_id=OG6XREYEGOYVFFMXCGBERRAIEMI2QMUGSBJBYP7GJQZ7FOAI4BIA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1FMjHdNGV42a9QZyMmSPTJWQrA8CdSRvXqrimnt6UGVreCzmgF, address:45.91.116.12:4237): protocol: rpc: dial tcp 45.91.116.12:4237: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.950066235 (316h30m9.501536706s) [noise=true , address=77.90.31.19:4863 , piece_id=3PM3Q6X3ADKX2RKIIFDMK57LTJNJHNFK4UYPKFK3ZEQLYQ23EP3A, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:124x1cbCPJRK8YkgSP6UzHymoLu1E1znnfg5WBiD89kSDLi8JeL, address:77.90.31.19:4863): protocol: write tcp 188.166.225.170:45968->77.90.31.19:4863: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.950246558 (316h30m9.50171702s) [noise=true , address=77.90.31.10:4524 , piece_id=Z2IGDHPDU6JAR5CAFQBGYHO3TQ66K4BZOAMXSC7X3RFWLGHOOMTQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12uN7CznoK4AD8FqFkNUndawdGJ9rmjAe7oNnjJkV844ScY87qe, address:77.90.31.10:4524): protocol: write tcp 188.166.225.170:40266->77.90.31.10:4524: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.950593058 (316h30m9.502063529s) [noise=true , address=77.90.31.14:4690 , piece_id=RJ3KXY3HURT7Q4HZYTOLSAL7Y6VMDGGRIQP6XHE4YB6OAA3PJRAA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.950853535 (316h30m9.502324006s) [noise=true , address=45.91.116.14:4213 , piece_id=HPFISTE2N3B4RBDPAZAGVZJ5WYLCPHTCYH4WMJ56IWEHR3ZBEMOQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12tQRqowH4aDr9h3s1WLs1o7na1ueiafwgUeeHfAB6g7mcE6e1t, address:45.91.116.14:4213): protocol: rpc: dial tcp 45.91.116.14:4213: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.951186438 (316h30m9.502656889s) [noise=true , address=45.91.116.6:4000 , piece_id=T4CI3KKNMFPT3ZIODHE7BQ4DVHVH2QTSO6OB7QZPL5SNPXQJE6TA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12CuCqHJYbanbuaEMHvtcuQXhvcdahD8eyT2fKDhqtxdSZ6ES4f, address:45.91.116.6:4000): protocol: rpc: dial tcp 45.91.116.6:4000: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.951409864 (316h30m9.502880325s) [noise=true , address=45.91.116.7:4039 , piece_id=HSRY63FXTEPCP2T46V77RVSC24GGRBAYLLU7DTKOLWGVOYHF6YVA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:16oq1Yv7JfHrg4gdHfhwuMj4UsWeQQPayKV6Pj1NY2CgJhuPr2, address:45.91.116.7:4039): protocol: write tcp 188.166.225.170:36904->45.91.116.7:4039: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.951660191 (316h30m9.503130652s) [noise=true , address=77.90.31.11:4572 , piece_id=G4XSIPSWOLSEOGGFA2NLOSDCGTLZPI3P5LDNNLB5RGMU5OQGWXGQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1ZAfYZ4wgpZVvatj6wtHona3hTaPcLxAmXC1DT2uKQyo8nLQQ7, address:77.90.31.11:4572): protocol: rpc: dial tcp 77.90.31.11:4572: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.952019855 (316h30m9.503490316s) [noise=true , address=45.91.116.9:4274 , piece_id=MXWRAXYLBBGLUVTA5GP3JVX2IRM2UI2IC35D7KX3XMFEBXLIDX5A, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12g6jFa38dTz5UdgsQaEuM2MSH1H7cxaysWsVmbHkM2Dd2W2iDb, address:45.91.116.9:4274): protocol: rpc: dial tcp 45.91.116.9:4274: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.952290972 (316h30m9.503761443s) [noise=true , address=77.90.31.15:4720 , piece_id=BDQUF6C5TT3ASE3DIN6SSVJHOZAJK4ZEW33OIYPTGB4LDVRELSUA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12VbsxFqMB3vVphSAJqYpbBzua7Goq45E6ovJSaExkpLtfkcc74, address:77.90.31.15:4720): protocol: rpc: dial tcp 77.90.31.15:4720: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.952544654 (316h30m9.504015105s) [noise=true , address=45.91.116.10:4110 , piece_id=H6FYA6K4N4AQYU3MN5RGYNBMYFXXBJDBTUCN37EAT7WBRNTX2YQA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:123KwusATu1ZKtNNVHZZZBE67ZEnVb8puyfQidBi3fEWREqdf1h, address:45.91.116.10:4110): protocol: rpc: dial tcp 45.91.116.10:4110: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.953548155 (316h30m9.505018617s) [noise=true , address=77.90.31.17:4787 , piece_id=IYIEVH6CARRYCARKOLKWVBICQDNDRCTGQMWF3VJZ6YBC3GO5OWHQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1Lietonb5aVbaZQt6wQ2EoxwJkSEn9oL9Dwyrc9mTyomEKikNt, address:77.90.31.17:4787): protocol: read tcp 188.166.225.170:57908->77.90.31.17:4787: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.953579816 (316h30m9.505050277s) [seg_pos={0 0}, upload=278731]: Done waiting for segment. successful: 5 optimal: 14 errs: <nil>
Nov 22 12:34:11.195719123 (316h30m9.747189584s) [upload=278731]: Done uploading segments...
Nov 22 12:34:11.195932840 (316h30m9.747403301s) [upload=278731]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 5)
Duration: 5.684972153s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
5 out of 60 is failed (threshold=6) [X...............................................X..X.XX.....] duration:
4.188454955s trace:
https://jaeger.dev.storj.io/trace/71781384eb27b231
FAIL (2025-11-22 12:38:18 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.188632663s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [...............................................X...X........] duration:
5.898086122s trace:
https://jaeger.dev.storj.io/trace/570ccd0ce1345814
FAIL (2025-11-22 12:38:12 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.898119937s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..................................................X.........] duration:
5.701413492s trace:
https://jaeger.dev.storj.io/trace/5ae3d2bf80495000
FAIL (2025-11-22 12:37:56 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Nov 22 12:31:07.276308940 (316h27m5.827779401s) [upload=278701]: Starting upload
Nov 22 12:31:07.276483523 (316h27m5.827953994s) [upload=278701]: Done starting upload
Nov 22 12:31:07.276562243 (316h27m5.828032704s) [upload=278701]: Uploading segments...
Nov 22 12:31:07.276950593 (316h27m5.828421054s) [upload=278701]: Got next segment. Inline: false
Nov 22 12:31:07.277027639 (316h27m5.8284981s) [upload=278701]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:31:07.277034181 (316h27m5.828504643s) [upload=278701]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:31:07.520212027 (316h27m6.071682488s) [seg_pos={0 0}, upload=278701]: Begin upload segment...
Nov 22 12:31:07.520269256 (316h27m6.071739717s) [seg_pos={0 0}, upload=278701]: Done begin upload segment.
Nov 22 12:31:07.520308992 (316h27m6.071779453s) [noise=true , address=145.79.227.27:4689 , piece_id=76S65KEZ7I2E76RR2JXOZ3ONGXCWSJA5XB7XQVFEEKGGNC34Z4EQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.520426746 (316h27m6.071897207s) [noise=true , address=145.79.227.21:4474 , piece_id=PCRWYQVCAZCPOILAEDHJGDBFKX4GEJ6JXCF7FUS7OVFRUTDPJI4Q, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.521175282 (316h27m6.072645743s) [noise=true , address=145.79.227.17:4972 , piece_id=UZK7LCIOBGGDNJZUVDBOLBJHGOIA4SUT2O3PD67PYQD3J2DGBJEA, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.521527573 (316h27m6.072998044s) [noise=true , address=145.79.227.23:4546 , piece_id=73UPPN2SYWNWEVWHCF4BRGKNO5GCRGOV3S5JZPX4MW75EJZRQOWQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.521694671 (316h27m6.073165132s) [noise=true , address=45.91.116.10:4082 , piece_id=GBDYXAWONOHQGBGM3NCRQYDSPFOAIHRJHBB2W6ZGJLMZUJLV65XA, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.522219410 (316h27m6.073689871s) [noise=true , address=145.79.227.20:4437 , piece_id=CZ32MB4AEZZ6YKCNWKRYALVAK7MT3KNXDSZCQTPUV2Q53NX6Y3BQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.522261240 (316h27m6.073731691s) [noise=true , address=145.79.227.16:4909 , piece_id=ZTL4ER5YJWSDGO5F5KLRUL5WRTA6WKVOJOERIBTRIIVEWXUY5Z2Q, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.522719151 (316h27m6.074189612s) [noise=true , address=145.79.227.26:4654 , piece_id=EM4RRGMPPNVR7VTXP3BED43FJFIUDZYVT4S5OUK2TRXZHYNMQ64Q, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.523017269 (316h27m6.07448772s) [noise=true , address=77.90.31.13:4627 , piece_id=Q7U5BFXHYQBBJQCU4EQGAR75WKL5QMBJ3LO34UYGO7QHLMDJBKWQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.523216949 (316h27m6.07468741s) [noise=true , address=145.79.227.25:4618 , piece_id=CV6M3CTPQR2MPQZFU6W34MFQERT6MMN3RRF2N3W35FGI464OSUUA, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.523744804 (316h27m6.075215265s) [noise=true , address=145.79.227.6:4258 , piece_id=B5D4TCVJORSAWDAU4DQ5VAV4CCRF5WUI24DUVUORAEKLAWGI7ABQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.523828153 (316h27m6.075298614s) [noise=true , address=145.79.227.24:4583 , piece_id=MLW4X34EWXKPOCFVI4VIQSL6RMUOJ5HRYHQ4HY35JIFT4ZFJ6F3Q, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.524255396 (316h27m6.075725858s) [noise=true , address=45.91.116.7:4009 , piece_id=FXQGF3UQN5CPFBS2HWD7BAXR4QIOF42MMK3VJQ7AK4THYJWOFFLQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.524662051 (316h27m6.076132512s) [noise=true , address=145.79.227.15:4874 , piece_id=TYAXIF3ZJQQSFGFANYF56DG4O5WBBXK4M7OT67E7ARYNZ663HESA, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.524746221 (316h27m6.076216682s) [noise=true , address=145.79.227.28:4726 , piece_id=3LBJV2B4CVXEMHCOPRW7DR5QBKPS6DFMPTHQSVNR4XRTZZVIGBQA, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.525227157 (316h27m6.076697608s) [noise=true , address=77.90.31.21:4879 , piece_id=GHTH55OGGXBAZZ5FGPCDYVHDYEETM7DNVTPSTTFFFRZMKZZ3XRXQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:09.134610631 (316h27m7.686081093s) [upload=278701]: Next returned nil segment
Nov 22 12:31:09.134628024 (316h27m7.686098495s) [upload=278701]: Waiting for error group managing segments...
Nov 22 12:31:09.502125778 (316h27m8.053596239s) [noise=true , address=145.79.227.16:4909 , piece_id=ZTL4ER5YJWSDGO5F5KLRUL5WRTA6WKVOJOERIBTRIIVEWXUY5Z2Q, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.503083742 (316h27m8.054554193s) [noise=true , address=145.79.227.20:4437 , piece_id=CZ32MB4AEZZ6YKCNWKRYALVAK7MT3KNXDSZCQTPUV2Q53NX6Y3BQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.517732716 (316h27m8.069203177s) [noise=true , address=145.79.227.25:4618 , piece_id=CV6M3CTPQR2MPQZFU6W34MFQERT6MMN3RRF2N3W35FGI464OSUUA, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.518029582 (316h27m8.069500043s) [noise=true , address=145.79.227.28:4726 , piece_id=3LBJV2B4CVXEMHCOPRW7DR5QBKPS6DFMPTHQSVNR4XRTZZVIGBQA, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.519935051 (316h27m8.071405512s) [noise=true , address=145.79.227.26:4654 , piece_id=EM4RRGMPPNVR7VTXP3BED43FJFIUDZYVT4S5OUK2TRXZHYNMQ64Q, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.520607661 (316h27m8.072078122s) [noise=true , address=145.79.227.27:4689 , piece_id=76S65KEZ7I2E76RR2JXOZ3ONGXCWSJA5XB7XQVFEEKGGNC34Z4EQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.521187736 (316h27m8.072658197s) [noise=true , address=145.79.227.23:4546 , piece_id=73UPPN2SYWNWEVWHCF4BRGKNO5GCRGOV3S5JZPX4MW75EJZRQOWQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.532974760 (316h27m8.084445221s) [noise=true , address=145.79.227.24:4583 , piece_id=MLW4X34EWXKPOCFVI4VIQSL6RMUOJ5HRYHQ4HY35JIFT4ZFJ6F3Q, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.533688800 (316h27m8.085159271s) [noise=true , address=145.79.227.6:4258 , piece_id=B5D4TCVJORSAWDAU4DQ5VAV4CCRF5WUI24DUVUORAEKLAWGI7ABQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.534460109 (316h27m8.08593057s) [noise=true , address=145.79.227.15:4874 , piece_id=TYAXIF3ZJQQSFGFANYF56DG4O5WBBXK4M7OT67E7ARYNZ663HESA, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.542955375 (316h27m8.094425826s) [noise=true , address=145.79.227.17:4972 , piece_id=UZK7LCIOBGGDNJZUVDBOLBJHGOIA4SUT2O3PD67PYQD3J2DGBJEA, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.563098982 (316h27m8.114569443s) [noise=true , address=145.79.227.21:4474 , piece_id=PCRWYQVCAZCPOILAEDHJGDBFKX4GEJ6JXCF7FUS7OVFRUTDPJI4Q, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:36.585475346 (316h27m35.136945807s) [noise=true , address=45.91.116.10:4082 , piece_id=GBDYXAWONOHQGBGM3NCRQYDSPFOAIHRJHBB2W6ZGJLMZUJLV65XA, seg_pos={0 0}, upload=278701]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: read tcp 188.166.225.170:54326->45.91.116.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:36.585499923 (316h27m35.136970384s) [noise=true , address=77.90.31.21:4879 , piece_id=GHTH55OGGXBAZZ5FGPCDYVHDYEETM7DNVTPSTTFFFRZMKZZ3XRXQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: rpc: dial tcp 77.90.31.21:4879: i/o timeout; context deadline exceeded
Nov 22 12:31:36.585749648 (316h27m35.137220119s) [noise=true , address=77.90.31.13:4627 , piece_id=Q7U5BFXHYQBBJQCU4EQGAR75WKL5QMBJ3LO34UYGO7QHLMDJBKWQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:36.585848086 (316h27m35.137318547s) [noise=true , address=45.91.116.7:4009 , piece_id=FXQGF3UQN5CPFBS2HWD7BAXR4QIOF42MMK3VJQ7AK4THYJWOFFLQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:36.585900936 (316h27m35.137371397s) [seg_pos={0 0}, upload=278701]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Nov 22 12:31:36.828881025 (316h27m35.380351486s) [upload=278701]: Done uploading segments...
Nov 22 12:31:36.828915801 (316h27m35.380386262s) [upload=278701]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 5.701445442s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [................................................X.....X.....] duration:
5.764279519s trace:
https://jaeger.dev.storj.io/trace/6c800b4c3a84b58c
FAIL (2025-11-22 12:37:55 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 5)
======LOGS======
N5RGYNBMYFXXBJDBTUCN37EAT7WBRNTX2YQA, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.890374814 (316h29m10.441845275s) [noise=true , address=145.79.227.32:4858 , piece_id=K5L3LEUKZWHS36XJHNNY4TYRXCP4DG5UAKCZXPJJTSG26CKUI4HQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.890535981 (316h29m10.442006442s) [noise=true , address=77.90.31.21:4890 , piece_id=DUEFPJUTYMD564Z4III63LZM6L4IL2ZTGM3QDZ6CG7WV5APHBNRA, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.890774064 (316h29m10.442244525s) [noise=true , address=45.91.116.14:4213 , piece_id=HPFISTE2N3B4RBDPAZAGVZJ5WYLCPHTCYH4WMJ56IWEHR3ZBEMOQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.890934851 (316h29m10.442405312s) [noise=true , address=45.91.116.6:4000 , piece_id=T4CI3KKNMFPT3ZIODHE7BQ4DVHVH2QTSO6OB7QZPL5SNPXQJE6TA, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.891167002 (316h29m10.442637463s) [noise=true , address=77.90.31.17:4787 , piece_id=IYIEVH6CARRYCARKOLKWVBICQDNDRCTGQMWF3VJZ6YBC3GO5OWHQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.891369828 (316h29m10.442840289s) [noise=true , address=45.91.116.7:4039 , piece_id=HSRY63FXTEPCP2T46V77RVSC24GGRBAYLLU7DTKOLWGVOYHF6YVA, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.891556875 (316h29m10.443027336s) [noise=true , address=77.90.31.11:4572 , piece_id=G4XSIPSWOLSEOGGFA2NLOSDCGTLZPI3P5LDNNLB5RGMU5OQGWXGQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.891896812 (316h29m10.443367273s) [noise=true , address=77.90.31.19:4863 , piece_id=3PM3Q6X3ADKX2RKIIFDMK57LTJNJHNFK4UYPKFK3ZEQLYQ23EP3A, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.892019235 (316h29m10.443489696s) [noise=true , address=77.90.31.10:4524 , piece_id=Z2IGDHPDU6JAR5CAFQBGYHO3TQ66K4BZOAMXSC7X3RFWLGHOOMTQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.892316521 (316h29m10.443786982s) [noise=true , address=77.90.31.13:4644 , piece_id=UPDZGC3CMJXA3AOJX65VLASU37NLFGPIJLCAPN32TEQB5QHTU7EQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.892682318 (316h29m10.444152779s) [noise=true , address=45.91.116.9:4274 , piece_id=MXWRAXYLBBGLUVTA5GP3JVX2IRM2UI2IC35D7KX3XMFEBXLIDX5A, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:13.353758506 (316h29m11.905228957s) [upload=278731]: Next returned nil segment
Nov 22 12:33:13.353772043 (316h29m11.905242504s) [upload=278731]: Waiting for error group managing segments...
Nov 22 12:33:13.751703718 (316h29m12.303174169s) [noise=true , address=145.79.227.32:4858 , piece_id=K5L3LEUKZWHS36XJHNNY4TYRXCP4DG5UAKCZXPJJTSG26CKUI4HQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil>
Nov 22 12:33:13.753660293 (316h29m12.305130764s) [noise=true , address=145.79.227.26:4674 , piece_id=UP7VIVRDRL6KERJSX3VLWTKGW3ESGCQ7BLYCBXGBB5UNIDE6OHQQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil>
Nov 22 12:33:13.760046123 (316h29m12.311516574s) [noise=true , address=145.79.227.3:4167 , piece_id=4STIVXGU7SOXWAIPG2PXWBLNDD7EYQ56E2DBN2UZ2AZ3TZSXCTIQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil>
Nov 22 12:33:57.373975014 (316h29m55.925445475s) [noise=true , address=77.90.31.13:4644 , piece_id=UPDZGC3CMJXA3AOJX65VLASU37NLFGPIJLCAPN32TEQB5QHTU7EQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil>
Nov 22 12:34:06.122275078 (316h30m4.673745539s) [noise=true , address=45.91.116.13:4181 , piece_id=MQNJEBQFZMHNDKAH2K3JEA7YBK2UK5BGSNCJF3MASF3CKVA5VD6Q, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil>
Nov 22 12:34:10.949171331 (316h30m9.500641792s) [noise=true , address=77.90.31.21:4890 , piece_id=DUEFPJUTYMD564Z4III63LZM6L4IL2ZTGM3QDZ6CG7WV5APHBNRA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1iE3mDnEhGDsMBtmUXC4EwvSrpyHLoybzVXkC9eoFuYEMHeBrd, address:77.90.31.21:4890): protocol: read tcp 188.166.225.170:55900->77.90.31.21:4890: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.949753650 (316h30m9.501224111s) [noise=true , address=45.91.116.12:4237 , piece_id=OG6XREYEGOYVFFMXCGBERRAIEMI2QMUGSBJBYP7GJQZ7FOAI4BIA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1FMjHdNGV42a9QZyMmSPTJWQrA8CdSRvXqrimnt6UGVreCzmgF, address:45.91.116.12:4237): protocol: rpc: dial tcp 45.91.116.12:4237: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.950066235 (316h30m9.501536706s) [noise=true , address=77.90.31.19:4863 , piece_id=3PM3Q6X3ADKX2RKIIFDMK57LTJNJHNFK4UYPKFK3ZEQLYQ23EP3A, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:124x1cbCPJRK8YkgSP6UzHymoLu1E1znnfg5WBiD89kSDLi8JeL, address:77.90.31.19:4863): protocol: write tcp 188.166.225.170:45968->77.90.31.19:4863: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.950246558 (316h30m9.50171702s) [noise=true , address=77.90.31.10:4524 , piece_id=Z2IGDHPDU6JAR5CAFQBGYHO3TQ66K4BZOAMXSC7X3RFWLGHOOMTQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12uN7CznoK4AD8FqFkNUndawdGJ9rmjAe7oNnjJkV844ScY87qe, address:77.90.31.10:4524): protocol: write tcp 188.166.225.170:40266->77.90.31.10:4524: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.950593058 (316h30m9.502063529s) [noise=true , address=77.90.31.14:4690 , piece_id=RJ3KXY3HURT7Q4HZYTOLSAL7Y6VMDGGRIQP6XHE4YB6OAA3PJRAA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.950853535 (316h30m9.502324006s) [noise=true , address=45.91.116.14:4213 , piece_id=HPFISTE2N3B4RBDPAZAGVZJ5WYLCPHTCYH4WMJ56IWEHR3ZBEMOQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12tQRqowH4aDr9h3s1WLs1o7na1ueiafwgUeeHfAB6g7mcE6e1t, address:45.91.116.14:4213): protocol: rpc: dial tcp 45.91.116.14:4213: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.951186438 (316h30m9.502656889s) [noise=true , address=45.91.116.6:4000 , piece_id=T4CI3KKNMFPT3ZIODHE7BQ4DVHVH2QTSO6OB7QZPL5SNPXQJE6TA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12CuCqHJYbanbuaEMHvtcuQXhvcdahD8eyT2fKDhqtxdSZ6ES4f, address:45.91.116.6:4000): protocol: rpc: dial tcp 45.91.116.6:4000: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.951409864 (316h30m9.502880325s) [noise=true , address=45.91.116.7:4039 , piece_id=HSRY63FXTEPCP2T46V77RVSC24GGRBAYLLU7DTKOLWGVOYHF6YVA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:16oq1Yv7JfHrg4gdHfhwuMj4UsWeQQPayKV6Pj1NY2CgJhuPr2, address:45.91.116.7:4039): protocol: write tcp 188.166.225.170:36904->45.91.116.7:4039: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.951660191 (316h30m9.503130652s) [noise=true , address=77.90.31.11:4572 , piece_id=G4XSIPSWOLSEOGGFA2NLOSDCGTLZPI3P5LDNNLB5RGMU5OQGWXGQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1ZAfYZ4wgpZVvatj6wtHona3hTaPcLxAmXC1DT2uKQyo8nLQQ7, address:77.90.31.11:4572): protocol: rpc: dial tcp 77.90.31.11:4572: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.952019855 (316h30m9.503490316s) [noise=true , address=45.91.116.9:4274 , piece_id=MXWRAXYLBBGLUVTA5GP3JVX2IRM2UI2IC35D7KX3XMFEBXLIDX5A, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12g6jFa38dTz5UdgsQaEuM2MSH1H7cxaysWsVmbHkM2Dd2W2iDb, address:45.91.116.9:4274): protocol: rpc: dial tcp 45.91.116.9:4274: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.952290972 (316h30m9.503761443s) [noise=true , address=77.90.31.15:4720 , piece_id=BDQUF6C5TT3ASE3DIN6SSVJHOZAJK4ZEW33OIYPTGB4LDVRELSUA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12VbsxFqMB3vVphSAJqYpbBzua7Goq45E6ovJSaExkpLtfkcc74, address:77.90.31.15:4720): protocol: rpc: dial tcp 77.90.31.15:4720: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.952544654 (316h30m9.504015105s) [noise=true , address=45.91.116.10:4110 , piece_id=H6FYA6K4N4AQYU3MN5RGYNBMYFXXBJDBTUCN37EAT7WBRNTX2YQA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:123KwusATu1ZKtNNVHZZZBE67ZEnVb8puyfQidBi3fEWREqdf1h, address:45.91.116.10:4110): protocol: rpc: dial tcp 45.91.116.10:4110: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.953548155 (316h30m9.505018617s) [noise=true , address=77.90.31.17:4787 , piece_id=IYIEVH6CARRYCARKOLKWVBICQDNDRCTGQMWF3VJZ6YBC3GO5OWHQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1Lietonb5aVbaZQt6wQ2EoxwJkSEn9oL9Dwyrc9mTyomEKikNt, address:77.90.31.17:4787): protocol: read tcp 188.166.225.170:57908->77.90.31.17:4787: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.953579816 (316h30m9.505050277s) [seg_pos={0 0}, upload=278731]: Done waiting for segment. successful: 5 optimal: 14 errs: <nil>
Nov 22 12:34:11.195719123 (316h30m9.747189584s) [upload=278731]: Done uploading segments...
Nov 22 12:34:11.195932840 (316h30m9.747403301s) [upload=278731]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 5)
Duration: 5.76430089s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [...............................................X..X..X......] duration:
5.724216704s trace:
https://jaeger.dev.storj.io/trace/92df32505639913
FAIL (2025-11-22 12:37:47 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.724247725s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...................................................X........] duration:
5.652752915s trace:
https://jaeger.dev.storj.io/trace/5ae3d2bf80495000
FAIL (2025-11-22 12:37:31 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Nov 22 12:31:07.276308940 (316h27m5.827779401s) [upload=278701]: Starting upload
Nov 22 12:31:07.276483523 (316h27m5.827953994s) [upload=278701]: Done starting upload
Nov 22 12:31:07.276562243 (316h27m5.828032704s) [upload=278701]: Uploading segments...
Nov 22 12:31:07.276950593 (316h27m5.828421054s) [upload=278701]: Got next segment. Inline: false
Nov 22 12:31:07.277027639 (316h27m5.8284981s) [upload=278701]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:31:07.277034181 (316h27m5.828504643s) [upload=278701]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:31:07.520212027 (316h27m6.071682488s) [seg_pos={0 0}, upload=278701]: Begin upload segment...
Nov 22 12:31:07.520269256 (316h27m6.071739717s) [seg_pos={0 0}, upload=278701]: Done begin upload segment.
Nov 22 12:31:07.520308992 (316h27m6.071779453s) [noise=true , address=145.79.227.27:4689 , piece_id=76S65KEZ7I2E76RR2JXOZ3ONGXCWSJA5XB7XQVFEEKGGNC34Z4EQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.520426746 (316h27m6.071897207s) [noise=true , address=145.79.227.21:4474 , piece_id=PCRWYQVCAZCPOILAEDHJGDBFKX4GEJ6JXCF7FUS7OVFRUTDPJI4Q, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.521175282 (316h27m6.072645743s) [noise=true , address=145.79.227.17:4972 , piece_id=UZK7LCIOBGGDNJZUVDBOLBJHGOIA4SUT2O3PD67PYQD3J2DGBJEA, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.521527573 (316h27m6.072998044s) [noise=true , address=145.79.227.23:4546 , piece_id=73UPPN2SYWNWEVWHCF4BRGKNO5GCRGOV3S5JZPX4MW75EJZRQOWQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.521694671 (316h27m6.073165132s) [noise=true , address=45.91.116.10:4082 , piece_id=GBDYXAWONOHQGBGM3NCRQYDSPFOAIHRJHBB2W6ZGJLMZUJLV65XA, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.522219410 (316h27m6.073689871s) [noise=true , address=145.79.227.20:4437 , piece_id=CZ32MB4AEZZ6YKCNWKRYALVAK7MT3KNXDSZCQTPUV2Q53NX6Y3BQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.522261240 (316h27m6.073731691s) [noise=true , address=145.79.227.16:4909 , piece_id=ZTL4ER5YJWSDGO5F5KLRUL5WRTA6WKVOJOERIBTRIIVEWXUY5Z2Q, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.522719151 (316h27m6.074189612s) [noise=true , address=145.79.227.26:4654 , piece_id=EM4RRGMPPNVR7VTXP3BED43FJFIUDZYVT4S5OUK2TRXZHYNMQ64Q, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.523017269 (316h27m6.07448772s) [noise=true , address=77.90.31.13:4627 , piece_id=Q7U5BFXHYQBBJQCU4EQGAR75WKL5QMBJ3LO34UYGO7QHLMDJBKWQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.523216949 (316h27m6.07468741s) [noise=true , address=145.79.227.25:4618 , piece_id=CV6M3CTPQR2MPQZFU6W34MFQERT6MMN3RRF2N3W35FGI464OSUUA, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.523744804 (316h27m6.075215265s) [noise=true , address=145.79.227.6:4258 , piece_id=B5D4TCVJORSAWDAU4DQ5VAV4CCRF5WUI24DUVUORAEKLAWGI7ABQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.523828153 (316h27m6.075298614s) [noise=true , address=145.79.227.24:4583 , piece_id=MLW4X34EWXKPOCFVI4VIQSL6RMUOJ5HRYHQ4HY35JIFT4ZFJ6F3Q, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.524255396 (316h27m6.075725858s) [noise=true , address=45.91.116.7:4009 , piece_id=FXQGF3UQN5CPFBS2HWD7BAXR4QIOF42MMK3VJQ7AK4THYJWOFFLQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.524662051 (316h27m6.076132512s) [noise=true , address=145.79.227.15:4874 , piece_id=TYAXIF3ZJQQSFGFANYF56DG4O5WBBXK4M7OT67E7ARYNZ663HESA, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.524746221 (316h27m6.076216682s) [noise=true , address=145.79.227.28:4726 , piece_id=3LBJV2B4CVXEMHCOPRW7DR5QBKPS6DFMPTHQSVNR4XRTZZVIGBQA, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.525227157 (316h27m6.076697608s) [noise=true , address=77.90.31.21:4879 , piece_id=GHTH55OGGXBAZZ5FGPCDYVHDYEETM7DNVTPSTTFFFRZMKZZ3XRXQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:09.134610631 (316h27m7.686081093s) [upload=278701]: Next returned nil segment
Nov 22 12:31:09.134628024 (316h27m7.686098495s) [upload=278701]: Waiting for error group managing segments...
Nov 22 12:31:09.502125778 (316h27m8.053596239s) [noise=true , address=145.79.227.16:4909 , piece_id=ZTL4ER5YJWSDGO5F5KLRUL5WRTA6WKVOJOERIBTRIIVEWXUY5Z2Q, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.503083742 (316h27m8.054554193s) [noise=true , address=145.79.227.20:4437 , piece_id=CZ32MB4AEZZ6YKCNWKRYALVAK7MT3KNXDSZCQTPUV2Q53NX6Y3BQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.517732716 (316h27m8.069203177s) [noise=true , address=145.79.227.25:4618 , piece_id=CV6M3CTPQR2MPQZFU6W34MFQERT6MMN3RRF2N3W35FGI464OSUUA, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.518029582 (316h27m8.069500043s) [noise=true , address=145.79.227.28:4726 , piece_id=3LBJV2B4CVXEMHCOPRW7DR5QBKPS6DFMPTHQSVNR4XRTZZVIGBQA, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.519935051 (316h27m8.071405512s) [noise=true , address=145.79.227.26:4654 , piece_id=EM4RRGMPPNVR7VTXP3BED43FJFIUDZYVT4S5OUK2TRXZHYNMQ64Q, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.520607661 (316h27m8.072078122s) [noise=true , address=145.79.227.27:4689 , piece_id=76S65KEZ7I2E76RR2JXOZ3ONGXCWSJA5XB7XQVFEEKGGNC34Z4EQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.521187736 (316h27m8.072658197s) [noise=true , address=145.79.227.23:4546 , piece_id=73UPPN2SYWNWEVWHCF4BRGKNO5GCRGOV3S5JZPX4MW75EJZRQOWQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.532974760 (316h27m8.084445221s) [noise=true , address=145.79.227.24:4583 , piece_id=MLW4X34EWXKPOCFVI4VIQSL6RMUOJ5HRYHQ4HY35JIFT4ZFJ6F3Q, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.533688800 (316h27m8.085159271s) [noise=true , address=145.79.227.6:4258 , piece_id=B5D4TCVJORSAWDAU4DQ5VAV4CCRF5WUI24DUVUORAEKLAWGI7ABQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.534460109 (316h27m8.08593057s) [noise=true , address=145.79.227.15:4874 , piece_id=TYAXIF3ZJQQSFGFANYF56DG4O5WBBXK4M7OT67E7ARYNZ663HESA, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.542955375 (316h27m8.094425826s) [noise=true , address=145.79.227.17:4972 , piece_id=UZK7LCIOBGGDNJZUVDBOLBJHGOIA4SUT2O3PD67PYQD3J2DGBJEA, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.563098982 (316h27m8.114569443s) [noise=true , address=145.79.227.21:4474 , piece_id=PCRWYQVCAZCPOILAEDHJGDBFKX4GEJ6JXCF7FUS7OVFRUTDPJI4Q, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:36.585475346 (316h27m35.136945807s) [noise=true , address=45.91.116.10:4082 , piece_id=GBDYXAWONOHQGBGM3NCRQYDSPFOAIHRJHBB2W6ZGJLMZUJLV65XA, seg_pos={0 0}, upload=278701]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: read tcp 188.166.225.170:54326->45.91.116.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:36.585499923 (316h27m35.136970384s) [noise=true , address=77.90.31.21:4879 , piece_id=GHTH55OGGXBAZZ5FGPCDYVHDYEETM7DNVTPSTTFFFRZMKZZ3XRXQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: rpc: dial tcp 77.90.31.21:4879: i/o timeout; context deadline exceeded
Nov 22 12:31:36.585749648 (316h27m35.137220119s) [noise=true , address=77.90.31.13:4627 , piece_id=Q7U5BFXHYQBBJQCU4EQGAR75WKL5QMBJ3LO34UYGO7QHLMDJBKWQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:36.585848086 (316h27m35.137318547s) [noise=true , address=45.91.116.7:4009 , piece_id=FXQGF3UQN5CPFBS2HWD7BAXR4QIOF42MMK3VJQ7AK4THYJWOFFLQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:36.585900936 (316h27m35.137371397s) [seg_pos={0 0}, upload=278701]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Nov 22 12:31:36.828881025 (316h27m35.380351486s) [upload=278701]: Done uploading segments...
Nov 22 12:31:36.828915801 (316h27m35.380386262s) [upload=278701]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 5.652786639s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
5 out of 60 is failed (threshold=6) [.X...............................................X..X.XX....] duration:
3.339019882s trace:
https://jaeger.dev.storj.io/trace/71781384eb27b231
FAIL (2025-11-22 12:37:30 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.339052695s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [................................................X..X..X.....] duration:
5.973806257s trace:
https://jaeger.dev.storj.io/trace/92df32505639913
FAIL (2025-11-22 12:37:14 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.973837176s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [................................................X...X.......] duration:
5.951906586s trace:
https://jaeger.dev.storj.io/trace/570ccd0ce1345814
FAIL (2025-11-22 12:37:11 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.951939028s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [.................................................X.....X....] duration:
5.722488483s trace:
https://jaeger.dev.storj.io/trace/6c800b4c3a84b58c
FAIL (2025-11-22 12:36:49 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 5)
======LOGS======
N5RGYNBMYFXXBJDBTUCN37EAT7WBRNTX2YQA, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.890374814 (316h29m10.441845275s) [noise=true , address=145.79.227.32:4858 , piece_id=K5L3LEUKZWHS36XJHNNY4TYRXCP4DG5UAKCZXPJJTSG26CKUI4HQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.890535981 (316h29m10.442006442s) [noise=true , address=77.90.31.21:4890 , piece_id=DUEFPJUTYMD564Z4III63LZM6L4IL2ZTGM3QDZ6CG7WV5APHBNRA, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.890774064 (316h29m10.442244525s) [noise=true , address=45.91.116.14:4213 , piece_id=HPFISTE2N3B4RBDPAZAGVZJ5WYLCPHTCYH4WMJ56IWEHR3ZBEMOQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.890934851 (316h29m10.442405312s) [noise=true , address=45.91.116.6:4000 , piece_id=T4CI3KKNMFPT3ZIODHE7BQ4DVHVH2QTSO6OB7QZPL5SNPXQJE6TA, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.891167002 (316h29m10.442637463s) [noise=true , address=77.90.31.17:4787 , piece_id=IYIEVH6CARRYCARKOLKWVBICQDNDRCTGQMWF3VJZ6YBC3GO5OWHQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.891369828 (316h29m10.442840289s) [noise=true , address=45.91.116.7:4039 , piece_id=HSRY63FXTEPCP2T46V77RVSC24GGRBAYLLU7DTKOLWGVOYHF6YVA, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.891556875 (316h29m10.443027336s) [noise=true , address=77.90.31.11:4572 , piece_id=G4XSIPSWOLSEOGGFA2NLOSDCGTLZPI3P5LDNNLB5RGMU5OQGWXGQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.891896812 (316h29m10.443367273s) [noise=true , address=77.90.31.19:4863 , piece_id=3PM3Q6X3ADKX2RKIIFDMK57LTJNJHNFK4UYPKFK3ZEQLYQ23EP3A, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.892019235 (316h29m10.443489696s) [noise=true , address=77.90.31.10:4524 , piece_id=Z2IGDHPDU6JAR5CAFQBGYHO3TQ66K4BZOAMXSC7X3RFWLGHOOMTQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.892316521 (316h29m10.443786982s) [noise=true , address=77.90.31.13:4644 , piece_id=UPDZGC3CMJXA3AOJX65VLASU37NLFGPIJLCAPN32TEQB5QHTU7EQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.892682318 (316h29m10.444152779s) [noise=true , address=45.91.116.9:4274 , piece_id=MXWRAXYLBBGLUVTA5GP3JVX2IRM2UI2IC35D7KX3XMFEBXLIDX5A, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:13.353758506 (316h29m11.905228957s) [upload=278731]: Next returned nil segment
Nov 22 12:33:13.353772043 (316h29m11.905242504s) [upload=278731]: Waiting for error group managing segments...
Nov 22 12:33:13.751703718 (316h29m12.303174169s) [noise=true , address=145.79.227.32:4858 , piece_id=K5L3LEUKZWHS36XJHNNY4TYRXCP4DG5UAKCZXPJJTSG26CKUI4HQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil>
Nov 22 12:33:13.753660293 (316h29m12.305130764s) [noise=true , address=145.79.227.26:4674 , piece_id=UP7VIVRDRL6KERJSX3VLWTKGW3ESGCQ7BLYCBXGBB5UNIDE6OHQQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil>
Nov 22 12:33:13.760046123 (316h29m12.311516574s) [noise=true , address=145.79.227.3:4167 , piece_id=4STIVXGU7SOXWAIPG2PXWBLNDD7EYQ56E2DBN2UZ2AZ3TZSXCTIQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil>
Nov 22 12:33:57.373975014 (316h29m55.925445475s) [noise=true , address=77.90.31.13:4644 , piece_id=UPDZGC3CMJXA3AOJX65VLASU37NLFGPIJLCAPN32TEQB5QHTU7EQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil>
Nov 22 12:34:06.122275078 (316h30m4.673745539s) [noise=true , address=45.91.116.13:4181 , piece_id=MQNJEBQFZMHNDKAH2K3JEA7YBK2UK5BGSNCJF3MASF3CKVA5VD6Q, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil>
Nov 22 12:34:10.949171331 (316h30m9.500641792s) [noise=true , address=77.90.31.21:4890 , piece_id=DUEFPJUTYMD564Z4III63LZM6L4IL2ZTGM3QDZ6CG7WV5APHBNRA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1iE3mDnEhGDsMBtmUXC4EwvSrpyHLoybzVXkC9eoFuYEMHeBrd, address:77.90.31.21:4890): protocol: read tcp 188.166.225.170:55900->77.90.31.21:4890: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.949753650 (316h30m9.501224111s) [noise=true , address=45.91.116.12:4237 , piece_id=OG6XREYEGOYVFFMXCGBERRAIEMI2QMUGSBJBYP7GJQZ7FOAI4BIA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1FMjHdNGV42a9QZyMmSPTJWQrA8CdSRvXqrimnt6UGVreCzmgF, address:45.91.116.12:4237): protocol: rpc: dial tcp 45.91.116.12:4237: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.950066235 (316h30m9.501536706s) [noise=true , address=77.90.31.19:4863 , piece_id=3PM3Q6X3ADKX2RKIIFDMK57LTJNJHNFK4UYPKFK3ZEQLYQ23EP3A, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:124x1cbCPJRK8YkgSP6UzHymoLu1E1znnfg5WBiD89kSDLi8JeL, address:77.90.31.19:4863): protocol: write tcp 188.166.225.170:45968->77.90.31.19:4863: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.950246558 (316h30m9.50171702s) [noise=true , address=77.90.31.10:4524 , piece_id=Z2IGDHPDU6JAR5CAFQBGYHO3TQ66K4BZOAMXSC7X3RFWLGHOOMTQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12uN7CznoK4AD8FqFkNUndawdGJ9rmjAe7oNnjJkV844ScY87qe, address:77.90.31.10:4524): protocol: write tcp 188.166.225.170:40266->77.90.31.10:4524: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.950593058 (316h30m9.502063529s) [noise=true , address=77.90.31.14:4690 , piece_id=RJ3KXY3HURT7Q4HZYTOLSAL7Y6VMDGGRIQP6XHE4YB6OAA3PJRAA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.950853535 (316h30m9.502324006s) [noise=true , address=45.91.116.14:4213 , piece_id=HPFISTE2N3B4RBDPAZAGVZJ5WYLCPHTCYH4WMJ56IWEHR3ZBEMOQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12tQRqowH4aDr9h3s1WLs1o7na1ueiafwgUeeHfAB6g7mcE6e1t, address:45.91.116.14:4213): protocol: rpc: dial tcp 45.91.116.14:4213: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.951186438 (316h30m9.502656889s) [noise=true , address=45.91.116.6:4000 , piece_id=T4CI3KKNMFPT3ZIODHE7BQ4DVHVH2QTSO6OB7QZPL5SNPXQJE6TA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12CuCqHJYbanbuaEMHvtcuQXhvcdahD8eyT2fKDhqtxdSZ6ES4f, address:45.91.116.6:4000): protocol: rpc: dial tcp 45.91.116.6:4000: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.951409864 (316h30m9.502880325s) [noise=true , address=45.91.116.7:4039 , piece_id=HSRY63FXTEPCP2T46V77RVSC24GGRBAYLLU7DTKOLWGVOYHF6YVA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:16oq1Yv7JfHrg4gdHfhwuMj4UsWeQQPayKV6Pj1NY2CgJhuPr2, address:45.91.116.7:4039): protocol: write tcp 188.166.225.170:36904->45.91.116.7:4039: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.951660191 (316h30m9.503130652s) [noise=true , address=77.90.31.11:4572 , piece_id=G4XSIPSWOLSEOGGFA2NLOSDCGTLZPI3P5LDNNLB5RGMU5OQGWXGQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1ZAfYZ4wgpZVvatj6wtHona3hTaPcLxAmXC1DT2uKQyo8nLQQ7, address:77.90.31.11:4572): protocol: rpc: dial tcp 77.90.31.11:4572: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.952019855 (316h30m9.503490316s) [noise=true , address=45.91.116.9:4274 , piece_id=MXWRAXYLBBGLUVTA5GP3JVX2IRM2UI2IC35D7KX3XMFEBXLIDX5A, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12g6jFa38dTz5UdgsQaEuM2MSH1H7cxaysWsVmbHkM2Dd2W2iDb, address:45.91.116.9:4274): protocol: rpc: dial tcp 45.91.116.9:4274: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.952290972 (316h30m9.503761443s) [noise=true , address=77.90.31.15:4720 , piece_id=BDQUF6C5TT3ASE3DIN6SSVJHOZAJK4ZEW33OIYPTGB4LDVRELSUA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12VbsxFqMB3vVphSAJqYpbBzua7Goq45E6ovJSaExkpLtfkcc74, address:77.90.31.15:4720): protocol: rpc: dial tcp 77.90.31.15:4720: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.952544654 (316h30m9.504015105s) [noise=true , address=45.91.116.10:4110 , piece_id=H6FYA6K4N4AQYU3MN5RGYNBMYFXXBJDBTUCN37EAT7WBRNTX2YQA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:123KwusATu1ZKtNNVHZZZBE67ZEnVb8puyfQidBi3fEWREqdf1h, address:45.91.116.10:4110): protocol: rpc: dial tcp 45.91.116.10:4110: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.953548155 (316h30m9.505018617s) [noise=true , address=77.90.31.17:4787 , piece_id=IYIEVH6CARRYCARKOLKWVBICQDNDRCTGQMWF3VJZ6YBC3GO5OWHQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1Lietonb5aVbaZQt6wQ2EoxwJkSEn9oL9Dwyrc9mTyomEKikNt, address:77.90.31.17:4787): protocol: read tcp 188.166.225.170:57908->77.90.31.17:4787: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.953579816 (316h30m9.505050277s) [seg_pos={0 0}, upload=278731]: Done waiting for segment. successful: 5 optimal: 14 errs: <nil>
Nov 22 12:34:11.195719123 (316h30m9.747189584s) [upload=278731]: Done uploading segments...
Nov 22 12:34:11.195932840 (316h30m9.747403301s) [upload=278731]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 5)
Duration: 5.722588915s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [....................................................X.......] duration:
5.862724698s trace:
https://jaeger.dev.storj.io/trace/5ae3d2bf80495000
FAIL (2025-11-22 12:36:31 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Nov 22 12:31:07.276308940 (316h27m5.827779401s) [upload=278701]: Starting upload
Nov 22 12:31:07.276483523 (316h27m5.827953994s) [upload=278701]: Done starting upload
Nov 22 12:31:07.276562243 (316h27m5.828032704s) [upload=278701]: Uploading segments...
Nov 22 12:31:07.276950593 (316h27m5.828421054s) [upload=278701]: Got next segment. Inline: false
Nov 22 12:31:07.277027639 (316h27m5.8284981s) [upload=278701]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:31:07.277034181 (316h27m5.828504643s) [upload=278701]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:31:07.520212027 (316h27m6.071682488s) [seg_pos={0 0}, upload=278701]: Begin upload segment...
Nov 22 12:31:07.520269256 (316h27m6.071739717s) [seg_pos={0 0}, upload=278701]: Done begin upload segment.
Nov 22 12:31:07.520308992 (316h27m6.071779453s) [noise=true , address=145.79.227.27:4689 , piece_id=76S65KEZ7I2E76RR2JXOZ3ONGXCWSJA5XB7XQVFEEKGGNC34Z4EQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.520426746 (316h27m6.071897207s) [noise=true , address=145.79.227.21:4474 , piece_id=PCRWYQVCAZCPOILAEDHJGDBFKX4GEJ6JXCF7FUS7OVFRUTDPJI4Q, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.521175282 (316h27m6.072645743s) [noise=true , address=145.79.227.17:4972 , piece_id=UZK7LCIOBGGDNJZUVDBOLBJHGOIA4SUT2O3PD67PYQD3J2DGBJEA, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.521527573 (316h27m6.072998044s) [noise=true , address=145.79.227.23:4546 , piece_id=73UPPN2SYWNWEVWHCF4BRGKNO5GCRGOV3S5JZPX4MW75EJZRQOWQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.521694671 (316h27m6.073165132s) [noise=true , address=45.91.116.10:4082 , piece_id=GBDYXAWONOHQGBGM3NCRQYDSPFOAIHRJHBB2W6ZGJLMZUJLV65XA, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.522219410 (316h27m6.073689871s) [noise=true , address=145.79.227.20:4437 , piece_id=CZ32MB4AEZZ6YKCNWKRYALVAK7MT3KNXDSZCQTPUV2Q53NX6Y3BQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.522261240 (316h27m6.073731691s) [noise=true , address=145.79.227.16:4909 , piece_id=ZTL4ER5YJWSDGO5F5KLRUL5WRTA6WKVOJOERIBTRIIVEWXUY5Z2Q, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.522719151 (316h27m6.074189612s) [noise=true , address=145.79.227.26:4654 , piece_id=EM4RRGMPPNVR7VTXP3BED43FJFIUDZYVT4S5OUK2TRXZHYNMQ64Q, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.523017269 (316h27m6.07448772s) [noise=true , address=77.90.31.13:4627 , piece_id=Q7U5BFXHYQBBJQCU4EQGAR75WKL5QMBJ3LO34UYGO7QHLMDJBKWQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.523216949 (316h27m6.07468741s) [noise=true , address=145.79.227.25:4618 , piece_id=CV6M3CTPQR2MPQZFU6W34MFQERT6MMN3RRF2N3W35FGI464OSUUA, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.523744804 (316h27m6.075215265s) [noise=true , address=145.79.227.6:4258 , piece_id=B5D4TCVJORSAWDAU4DQ5VAV4CCRF5WUI24DUVUORAEKLAWGI7ABQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.523828153 (316h27m6.075298614s) [noise=true , address=145.79.227.24:4583 , piece_id=MLW4X34EWXKPOCFVI4VIQSL6RMUOJ5HRYHQ4HY35JIFT4ZFJ6F3Q, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.524255396 (316h27m6.075725858s) [noise=true , address=45.91.116.7:4009 , piece_id=FXQGF3UQN5CPFBS2HWD7BAXR4QIOF42MMK3VJQ7AK4THYJWOFFLQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.524662051 (316h27m6.076132512s) [noise=true , address=145.79.227.15:4874 , piece_id=TYAXIF3ZJQQSFGFANYF56DG4O5WBBXK4M7OT67E7ARYNZ663HESA, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.524746221 (316h27m6.076216682s) [noise=true , address=145.79.227.28:4726 , piece_id=3LBJV2B4CVXEMHCOPRW7DR5QBKPS6DFMPTHQSVNR4XRTZZVIGBQA, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.525227157 (316h27m6.076697608s) [noise=true , address=77.90.31.21:4879 , piece_id=GHTH55OGGXBAZZ5FGPCDYVHDYEETM7DNVTPSTTFFFRZMKZZ3XRXQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:09.134610631 (316h27m7.686081093s) [upload=278701]: Next returned nil segment
Nov 22 12:31:09.134628024 (316h27m7.686098495s) [upload=278701]: Waiting for error group managing segments...
Nov 22 12:31:09.502125778 (316h27m8.053596239s) [noise=true , address=145.79.227.16:4909 , piece_id=ZTL4ER5YJWSDGO5F5KLRUL5WRTA6WKVOJOERIBTRIIVEWXUY5Z2Q, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.503083742 (316h27m8.054554193s) [noise=true , address=145.79.227.20:4437 , piece_id=CZ32MB4AEZZ6YKCNWKRYALVAK7MT3KNXDSZCQTPUV2Q53NX6Y3BQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.517732716 (316h27m8.069203177s) [noise=true , address=145.79.227.25:4618 , piece_id=CV6M3CTPQR2MPQZFU6W34MFQERT6MMN3RRF2N3W35FGI464OSUUA, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.518029582 (316h27m8.069500043s) [noise=true , address=145.79.227.28:4726 , piece_id=3LBJV2B4CVXEMHCOPRW7DR5QBKPS6DFMPTHQSVNR4XRTZZVIGBQA, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.519935051 (316h27m8.071405512s) [noise=true , address=145.79.227.26:4654 , piece_id=EM4RRGMPPNVR7VTXP3BED43FJFIUDZYVT4S5OUK2TRXZHYNMQ64Q, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.520607661 (316h27m8.072078122s) [noise=true , address=145.79.227.27:4689 , piece_id=76S65KEZ7I2E76RR2JXOZ3ONGXCWSJA5XB7XQVFEEKGGNC34Z4EQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.521187736 (316h27m8.072658197s) [noise=true , address=145.79.227.23:4546 , piece_id=73UPPN2SYWNWEVWHCF4BRGKNO5GCRGOV3S5JZPX4MW75EJZRQOWQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.532974760 (316h27m8.084445221s) [noise=true , address=145.79.227.24:4583 , piece_id=MLW4X34EWXKPOCFVI4VIQSL6RMUOJ5HRYHQ4HY35JIFT4ZFJ6F3Q, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.533688800 (316h27m8.085159271s) [noise=true , address=145.79.227.6:4258 , piece_id=B5D4TCVJORSAWDAU4DQ5VAV4CCRF5WUI24DUVUORAEKLAWGI7ABQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.534460109 (316h27m8.08593057s) [noise=true , address=145.79.227.15:4874 , piece_id=TYAXIF3ZJQQSFGFANYF56DG4O5WBBXK4M7OT67E7ARYNZ663HESA, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.542955375 (316h27m8.094425826s) [noise=true , address=145.79.227.17:4972 , piece_id=UZK7LCIOBGGDNJZUVDBOLBJHGOIA4SUT2O3PD67PYQD3J2DGBJEA, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.563098982 (316h27m8.114569443s) [noise=true , address=145.79.227.21:4474 , piece_id=PCRWYQVCAZCPOILAEDHJGDBFKX4GEJ6JXCF7FUS7OVFRUTDPJI4Q, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:36.585475346 (316h27m35.136945807s) [noise=true , address=45.91.116.10:4082 , piece_id=GBDYXAWONOHQGBGM3NCRQYDSPFOAIHRJHBB2W6ZGJLMZUJLV65XA, seg_pos={0 0}, upload=278701]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: read tcp 188.166.225.170:54326->45.91.116.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:36.585499923 (316h27m35.136970384s) [noise=true , address=77.90.31.21:4879 , piece_id=GHTH55OGGXBAZZ5FGPCDYVHDYEETM7DNVTPSTTFFFRZMKZZ3XRXQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: rpc: dial tcp 77.90.31.21:4879: i/o timeout; context deadline exceeded
Nov 22 12:31:36.585749648 (316h27m35.137220119s) [noise=true , address=77.90.31.13:4627 , piece_id=Q7U5BFXHYQBBJQCU4EQGAR75WKL5QMBJ3LO34UYGO7QHLMDJBKWQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:36.585848086 (316h27m35.137318547s) [noise=true , address=45.91.116.7:4009 , piece_id=FXQGF3UQN5CPFBS2HWD7BAXR4QIOF42MMK3VJQ7AK4THYJWOFFLQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:36.585900936 (316h27m35.137371397s) [seg_pos={0 0}, upload=278701]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Nov 22 12:31:36.828881025 (316h27m35.380351486s) [upload=278701]: Done uploading segments...
Nov 22 12:31:36.828915801 (316h27m35.380386262s) [upload=278701]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 5.862753032s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [.................................................X...X......] duration:
5.639340985s trace:
https://jaeger.dev.storj.io/trace/570ccd0ce1345814
FAIL (2025-11-22 12:36:31 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.639360673s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
5 out of 60 is failed (threshold=6) [..X...............................................X..X.XX...] duration:
3.434375268s trace:
https://jaeger.dev.storj.io/trace/71781384eb27b231
FAIL (2025-11-22 12:36:19 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.434424582s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [..................................................X.....X...] duration:
5.827492309s trace:
https://jaeger.dev.storj.io/trace/6c800b4c3a84b58c
FAIL (2025-11-22 12:36:18 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 5)
======LOGS======
N5RGYNBMYFXXBJDBTUCN37EAT7WBRNTX2YQA, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.890374814 (316h29m10.441845275s) [noise=true , address=145.79.227.32:4858 , piece_id=K5L3LEUKZWHS36XJHNNY4TYRXCP4DG5UAKCZXPJJTSG26CKUI4HQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.890535981 (316h29m10.442006442s) [noise=true , address=77.90.31.21:4890 , piece_id=DUEFPJUTYMD564Z4III63LZM6L4IL2ZTGM3QDZ6CG7WV5APHBNRA, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.890774064 (316h29m10.442244525s) [noise=true , address=45.91.116.14:4213 , piece_id=HPFISTE2N3B4RBDPAZAGVZJ5WYLCPHTCYH4WMJ56IWEHR3ZBEMOQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.890934851 (316h29m10.442405312s) [noise=true , address=45.91.116.6:4000 , piece_id=T4CI3KKNMFPT3ZIODHE7BQ4DVHVH2QTSO6OB7QZPL5SNPXQJE6TA, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.891167002 (316h29m10.442637463s) [noise=true , address=77.90.31.17:4787 , piece_id=IYIEVH6CARRYCARKOLKWVBICQDNDRCTGQMWF3VJZ6YBC3GO5OWHQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.891369828 (316h29m10.442840289s) [noise=true , address=45.91.116.7:4039 , piece_id=HSRY63FXTEPCP2T46V77RVSC24GGRBAYLLU7DTKOLWGVOYHF6YVA, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.891556875 (316h29m10.443027336s) [noise=true , address=77.90.31.11:4572 , piece_id=G4XSIPSWOLSEOGGFA2NLOSDCGTLZPI3P5LDNNLB5RGMU5OQGWXGQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.891896812 (316h29m10.443367273s) [noise=true , address=77.90.31.19:4863 , piece_id=3PM3Q6X3ADKX2RKIIFDMK57LTJNJHNFK4UYPKFK3ZEQLYQ23EP3A, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.892019235 (316h29m10.443489696s) [noise=true , address=77.90.31.10:4524 , piece_id=Z2IGDHPDU6JAR5CAFQBGYHO3TQ66K4BZOAMXSC7X3RFWLGHOOMTQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.892316521 (316h29m10.443786982s) [noise=true , address=77.90.31.13:4644 , piece_id=UPDZGC3CMJXA3AOJX65VLASU37NLFGPIJLCAPN32TEQB5QHTU7EQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.892682318 (316h29m10.444152779s) [noise=true , address=45.91.116.9:4274 , piece_id=MXWRAXYLBBGLUVTA5GP3JVX2IRM2UI2IC35D7KX3XMFEBXLIDX5A, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:13.353758506 (316h29m11.905228957s) [upload=278731]: Next returned nil segment
Nov 22 12:33:13.353772043 (316h29m11.905242504s) [upload=278731]: Waiting for error group managing segments...
Nov 22 12:33:13.751703718 (316h29m12.303174169s) [noise=true , address=145.79.227.32:4858 , piece_id=K5L3LEUKZWHS36XJHNNY4TYRXCP4DG5UAKCZXPJJTSG26CKUI4HQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil>
Nov 22 12:33:13.753660293 (316h29m12.305130764s) [noise=true , address=145.79.227.26:4674 , piece_id=UP7VIVRDRL6KERJSX3VLWTKGW3ESGCQ7BLYCBXGBB5UNIDE6OHQQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil>
Nov 22 12:33:13.760046123 (316h29m12.311516574s) [noise=true , address=145.79.227.3:4167 , piece_id=4STIVXGU7SOXWAIPG2PXWBLNDD7EYQ56E2DBN2UZ2AZ3TZSXCTIQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil>
Nov 22 12:33:57.373975014 (316h29m55.925445475s) [noise=true , address=77.90.31.13:4644 , piece_id=UPDZGC3CMJXA3AOJX65VLASU37NLFGPIJLCAPN32TEQB5QHTU7EQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil>
Nov 22 12:34:06.122275078 (316h30m4.673745539s) [noise=true , address=45.91.116.13:4181 , piece_id=MQNJEBQFZMHNDKAH2K3JEA7YBK2UK5BGSNCJF3MASF3CKVA5VD6Q, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil>
Nov 22 12:34:10.949171331 (316h30m9.500641792s) [noise=true , address=77.90.31.21:4890 , piece_id=DUEFPJUTYMD564Z4III63LZM6L4IL2ZTGM3QDZ6CG7WV5APHBNRA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1iE3mDnEhGDsMBtmUXC4EwvSrpyHLoybzVXkC9eoFuYEMHeBrd, address:77.90.31.21:4890): protocol: read tcp 188.166.225.170:55900->77.90.31.21:4890: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.949753650 (316h30m9.501224111s) [noise=true , address=45.91.116.12:4237 , piece_id=OG6XREYEGOYVFFMXCGBERRAIEMI2QMUGSBJBYP7GJQZ7FOAI4BIA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1FMjHdNGV42a9QZyMmSPTJWQrA8CdSRvXqrimnt6UGVreCzmgF, address:45.91.116.12:4237): protocol: rpc: dial tcp 45.91.116.12:4237: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.950066235 (316h30m9.501536706s) [noise=true , address=77.90.31.19:4863 , piece_id=3PM3Q6X3ADKX2RKIIFDMK57LTJNJHNFK4UYPKFK3ZEQLYQ23EP3A, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:124x1cbCPJRK8YkgSP6UzHymoLu1E1znnfg5WBiD89kSDLi8JeL, address:77.90.31.19:4863): protocol: write tcp 188.166.225.170:45968->77.90.31.19:4863: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.950246558 (316h30m9.50171702s) [noise=true , address=77.90.31.10:4524 , piece_id=Z2IGDHPDU6JAR5CAFQBGYHO3TQ66K4BZOAMXSC7X3RFWLGHOOMTQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12uN7CznoK4AD8FqFkNUndawdGJ9rmjAe7oNnjJkV844ScY87qe, address:77.90.31.10:4524): protocol: write tcp 188.166.225.170:40266->77.90.31.10:4524: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.950593058 (316h30m9.502063529s) [noise=true , address=77.90.31.14:4690 , piece_id=RJ3KXY3HURT7Q4HZYTOLSAL7Y6VMDGGRIQP6XHE4YB6OAA3PJRAA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.950853535 (316h30m9.502324006s) [noise=true , address=45.91.116.14:4213 , piece_id=HPFISTE2N3B4RBDPAZAGVZJ5WYLCPHTCYH4WMJ56IWEHR3ZBEMOQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12tQRqowH4aDr9h3s1WLs1o7na1ueiafwgUeeHfAB6g7mcE6e1t, address:45.91.116.14:4213): protocol: rpc: dial tcp 45.91.116.14:4213: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.951186438 (316h30m9.502656889s) [noise=true , address=45.91.116.6:4000 , piece_id=T4CI3KKNMFPT3ZIODHE7BQ4DVHVH2QTSO6OB7QZPL5SNPXQJE6TA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12CuCqHJYbanbuaEMHvtcuQXhvcdahD8eyT2fKDhqtxdSZ6ES4f, address:45.91.116.6:4000): protocol: rpc: dial tcp 45.91.116.6:4000: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.951409864 (316h30m9.502880325s) [noise=true , address=45.91.116.7:4039 , piece_id=HSRY63FXTEPCP2T46V77RVSC24GGRBAYLLU7DTKOLWGVOYHF6YVA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:16oq1Yv7JfHrg4gdHfhwuMj4UsWeQQPayKV6Pj1NY2CgJhuPr2, address:45.91.116.7:4039): protocol: write tcp 188.166.225.170:36904->45.91.116.7:4039: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.951660191 (316h30m9.503130652s) [noise=true , address=77.90.31.11:4572 , piece_id=G4XSIPSWOLSEOGGFA2NLOSDCGTLZPI3P5LDNNLB5RGMU5OQGWXGQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1ZAfYZ4wgpZVvatj6wtHona3hTaPcLxAmXC1DT2uKQyo8nLQQ7, address:77.90.31.11:4572): protocol: rpc: dial tcp 77.90.31.11:4572: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.952019855 (316h30m9.503490316s) [noise=true , address=45.91.116.9:4274 , piece_id=MXWRAXYLBBGLUVTA5GP3JVX2IRM2UI2IC35D7KX3XMFEBXLIDX5A, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12g6jFa38dTz5UdgsQaEuM2MSH1H7cxaysWsVmbHkM2Dd2W2iDb, address:45.91.116.9:4274): protocol: rpc: dial tcp 45.91.116.9:4274: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.952290972 (316h30m9.503761443s) [noise=true , address=77.90.31.15:4720 , piece_id=BDQUF6C5TT3ASE3DIN6SSVJHOZAJK4ZEW33OIYPTGB4LDVRELSUA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12VbsxFqMB3vVphSAJqYpbBzua7Goq45E6ovJSaExkpLtfkcc74, address:77.90.31.15:4720): protocol: rpc: dial tcp 77.90.31.15:4720: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.952544654 (316h30m9.504015105s) [noise=true , address=45.91.116.10:4110 , piece_id=H6FYA6K4N4AQYU3MN5RGYNBMYFXXBJDBTUCN37EAT7WBRNTX2YQA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:123KwusATu1ZKtNNVHZZZBE67ZEnVb8puyfQidBi3fEWREqdf1h, address:45.91.116.10:4110): protocol: rpc: dial tcp 45.91.116.10:4110: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.953548155 (316h30m9.505018617s) [noise=true , address=77.90.31.17:4787 , piece_id=IYIEVH6CARRYCARKOLKWVBICQDNDRCTGQMWF3VJZ6YBC3GO5OWHQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1Lietonb5aVbaZQt6wQ2EoxwJkSEn9oL9Dwyrc9mTyomEKikNt, address:77.90.31.17:4787): protocol: read tcp 188.166.225.170:57908->77.90.31.17:4787: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.953579816 (316h30m9.505050277s) [seg_pos={0 0}, upload=278731]: Done waiting for segment. successful: 5 optimal: 14 errs: <nil>
Nov 22 12:34:11.195719123 (316h30m9.747189584s) [upload=278731]: Done uploading segments...
Nov 22 12:34:11.195932840 (316h30m9.747403301s) [upload=278731]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 5)
Duration: 5.827522316s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.................................................X..X..X....] duration:
6.587913262s trace:
https://jaeger.dev.storj.io/trace/92df32505639913
FAIL (2025-11-22 12:36:17 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.587940092s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [..................................................X...X.....] duration:
5.880693824s trace:
https://jaeger.dev.storj.io/trace/570ccd0ce1345814
FAIL (2025-11-22 12:35:43 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.880709403s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
5 out of 60 is failed (threshold=6) [...X...............................................X..X.XX..] duration:
3.525751573s trace:
https://jaeger.dev.storj.io/trace/71781384eb27b231
FAIL (2025-11-22 12:35:39 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.525780839s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [..................................................X..X..X...] duration:
5.892496951s trace:
https://jaeger.dev.storj.io/trace/92df32505639913
FAIL (2025-11-22 12:35:34 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.892525846s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....................................................X......] duration:
5.752645035s trace:
https://jaeger.dev.storj.io/trace/5ae3d2bf80495000
FAIL (2025-11-22 12:35:25 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Nov 22 12:31:07.276308940 (316h27m5.827779401s) [upload=278701]: Starting upload
Nov 22 12:31:07.276483523 (316h27m5.827953994s) [upload=278701]: Done starting upload
Nov 22 12:31:07.276562243 (316h27m5.828032704s) [upload=278701]: Uploading segments...
Nov 22 12:31:07.276950593 (316h27m5.828421054s) [upload=278701]: Got next segment. Inline: false
Nov 22 12:31:07.277027639 (316h27m5.8284981s) [upload=278701]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:31:07.277034181 (316h27m5.828504643s) [upload=278701]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:31:07.520212027 (316h27m6.071682488s) [seg_pos={0 0}, upload=278701]: Begin upload segment...
Nov 22 12:31:07.520269256 (316h27m6.071739717s) [seg_pos={0 0}, upload=278701]: Done begin upload segment.
Nov 22 12:31:07.520308992 (316h27m6.071779453s) [noise=true , address=145.79.227.27:4689 , piece_id=76S65KEZ7I2E76RR2JXOZ3ONGXCWSJA5XB7XQVFEEKGGNC34Z4EQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.520426746 (316h27m6.071897207s) [noise=true , address=145.79.227.21:4474 , piece_id=PCRWYQVCAZCPOILAEDHJGDBFKX4GEJ6JXCF7FUS7OVFRUTDPJI4Q, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.521175282 (316h27m6.072645743s) [noise=true , address=145.79.227.17:4972 , piece_id=UZK7LCIOBGGDNJZUVDBOLBJHGOIA4SUT2O3PD67PYQD3J2DGBJEA, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.521527573 (316h27m6.072998044s) [noise=true , address=145.79.227.23:4546 , piece_id=73UPPN2SYWNWEVWHCF4BRGKNO5GCRGOV3S5JZPX4MW75EJZRQOWQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.521694671 (316h27m6.073165132s) [noise=true , address=45.91.116.10:4082 , piece_id=GBDYXAWONOHQGBGM3NCRQYDSPFOAIHRJHBB2W6ZGJLMZUJLV65XA, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.522219410 (316h27m6.073689871s) [noise=true , address=145.79.227.20:4437 , piece_id=CZ32MB4AEZZ6YKCNWKRYALVAK7MT3KNXDSZCQTPUV2Q53NX6Y3BQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.522261240 (316h27m6.073731691s) [noise=true , address=145.79.227.16:4909 , piece_id=ZTL4ER5YJWSDGO5F5KLRUL5WRTA6WKVOJOERIBTRIIVEWXUY5Z2Q, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.522719151 (316h27m6.074189612s) [noise=true , address=145.79.227.26:4654 , piece_id=EM4RRGMPPNVR7VTXP3BED43FJFIUDZYVT4S5OUK2TRXZHYNMQ64Q, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.523017269 (316h27m6.07448772s) [noise=true , address=77.90.31.13:4627 , piece_id=Q7U5BFXHYQBBJQCU4EQGAR75WKL5QMBJ3LO34UYGO7QHLMDJBKWQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.523216949 (316h27m6.07468741s) [noise=true , address=145.79.227.25:4618 , piece_id=CV6M3CTPQR2MPQZFU6W34MFQERT6MMN3RRF2N3W35FGI464OSUUA, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.523744804 (316h27m6.075215265s) [noise=true , address=145.79.227.6:4258 , piece_id=B5D4TCVJORSAWDAU4DQ5VAV4CCRF5WUI24DUVUORAEKLAWGI7ABQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.523828153 (316h27m6.075298614s) [noise=true , address=145.79.227.24:4583 , piece_id=MLW4X34EWXKPOCFVI4VIQSL6RMUOJ5HRYHQ4HY35JIFT4ZFJ6F3Q, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.524255396 (316h27m6.075725858s) [noise=true , address=45.91.116.7:4009 , piece_id=FXQGF3UQN5CPFBS2HWD7BAXR4QIOF42MMK3VJQ7AK4THYJWOFFLQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.524662051 (316h27m6.076132512s) [noise=true , address=145.79.227.15:4874 , piece_id=TYAXIF3ZJQQSFGFANYF56DG4O5WBBXK4M7OT67E7ARYNZ663HESA, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.524746221 (316h27m6.076216682s) [noise=true , address=145.79.227.28:4726 , piece_id=3LBJV2B4CVXEMHCOPRW7DR5QBKPS6DFMPTHQSVNR4XRTZZVIGBQA, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.525227157 (316h27m6.076697608s) [noise=true , address=77.90.31.21:4879 , piece_id=GHTH55OGGXBAZZ5FGPCDYVHDYEETM7DNVTPSTTFFFRZMKZZ3XRXQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:09.134610631 (316h27m7.686081093s) [upload=278701]: Next returned nil segment
Nov 22 12:31:09.134628024 (316h27m7.686098495s) [upload=278701]: Waiting for error group managing segments...
Nov 22 12:31:09.502125778 (316h27m8.053596239s) [noise=true , address=145.79.227.16:4909 , piece_id=ZTL4ER5YJWSDGO5F5KLRUL5WRTA6WKVOJOERIBTRIIVEWXUY5Z2Q, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.503083742 (316h27m8.054554193s) [noise=true , address=145.79.227.20:4437 , piece_id=CZ32MB4AEZZ6YKCNWKRYALVAK7MT3KNXDSZCQTPUV2Q53NX6Y3BQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.517732716 (316h27m8.069203177s) [noise=true , address=145.79.227.25:4618 , piece_id=CV6M3CTPQR2MPQZFU6W34MFQERT6MMN3RRF2N3W35FGI464OSUUA, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.518029582 (316h27m8.069500043s) [noise=true , address=145.79.227.28:4726 , piece_id=3LBJV2B4CVXEMHCOPRW7DR5QBKPS6DFMPTHQSVNR4XRTZZVIGBQA, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.519935051 (316h27m8.071405512s) [noise=true , address=145.79.227.26:4654 , piece_id=EM4RRGMPPNVR7VTXP3BED43FJFIUDZYVT4S5OUK2TRXZHYNMQ64Q, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.520607661 (316h27m8.072078122s) [noise=true , address=145.79.227.27:4689 , piece_id=76S65KEZ7I2E76RR2JXOZ3ONGXCWSJA5XB7XQVFEEKGGNC34Z4EQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.521187736 (316h27m8.072658197s) [noise=true , address=145.79.227.23:4546 , piece_id=73UPPN2SYWNWEVWHCF4BRGKNO5GCRGOV3S5JZPX4MW75EJZRQOWQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.532974760 (316h27m8.084445221s) [noise=true , address=145.79.227.24:4583 , piece_id=MLW4X34EWXKPOCFVI4VIQSL6RMUOJ5HRYHQ4HY35JIFT4ZFJ6F3Q, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.533688800 (316h27m8.085159271s) [noise=true , address=145.79.227.6:4258 , piece_id=B5D4TCVJORSAWDAU4DQ5VAV4CCRF5WUI24DUVUORAEKLAWGI7ABQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.534460109 (316h27m8.08593057s) [noise=true , address=145.79.227.15:4874 , piece_id=TYAXIF3ZJQQSFGFANYF56DG4O5WBBXK4M7OT67E7ARYNZ663HESA, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.542955375 (316h27m8.094425826s) [noise=true , address=145.79.227.17:4972 , piece_id=UZK7LCIOBGGDNJZUVDBOLBJHGOIA4SUT2O3PD67PYQD3J2DGBJEA, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.563098982 (316h27m8.114569443s) [noise=true , address=145.79.227.21:4474 , piece_id=PCRWYQVCAZCPOILAEDHJGDBFKX4GEJ6JXCF7FUS7OVFRUTDPJI4Q, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:36.585475346 (316h27m35.136945807s) [noise=true , address=45.91.116.10:4082 , piece_id=GBDYXAWONOHQGBGM3NCRQYDSPFOAIHRJHBB2W6ZGJLMZUJLV65XA, seg_pos={0 0}, upload=278701]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: read tcp 188.166.225.170:54326->45.91.116.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:36.585499923 (316h27m35.136970384s) [noise=true , address=77.90.31.21:4879 , piece_id=GHTH55OGGXBAZZ5FGPCDYVHDYEETM7DNVTPSTTFFFRZMKZZ3XRXQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: rpc: dial tcp 77.90.31.21:4879: i/o timeout; context deadline exceeded
Nov 22 12:31:36.585749648 (316h27m35.137220119s) [noise=true , address=77.90.31.13:4627 , piece_id=Q7U5BFXHYQBBJQCU4EQGAR75WKL5QMBJ3LO34UYGO7QHLMDJBKWQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:36.585848086 (316h27m35.137318547s) [noise=true , address=45.91.116.7:4009 , piece_id=FXQGF3UQN5CPFBS2HWD7BAXR4QIOF42MMK3VJQ7AK4THYJWOFFLQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:36.585900936 (316h27m35.137371397s) [seg_pos={0 0}, upload=278701]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Nov 22 12:31:36.828881025 (316h27m35.380351486s) [upload=278701]: Done uploading segments...
Nov 22 12:31:36.828915801 (316h27m35.380386262s) [upload=278701]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 5.752676255s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [...................................................X.....X..] duration:
5.60363869s trace:
https://jaeger.dev.storj.io/trace/6c800b4c3a84b58c
FAIL (2025-11-22 12:35:14 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 5)
======LOGS======
N5RGYNBMYFXXBJDBTUCN37EAT7WBRNTX2YQA, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.890374814 (316h29m10.441845275s) [noise=true , address=145.79.227.32:4858 , piece_id=K5L3LEUKZWHS36XJHNNY4TYRXCP4DG5UAKCZXPJJTSG26CKUI4HQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.890535981 (316h29m10.442006442s) [noise=true , address=77.90.31.21:4890 , piece_id=DUEFPJUTYMD564Z4III63LZM6L4IL2ZTGM3QDZ6CG7WV5APHBNRA, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.890774064 (316h29m10.442244525s) [noise=true , address=45.91.116.14:4213 , piece_id=HPFISTE2N3B4RBDPAZAGVZJ5WYLCPHTCYH4WMJ56IWEHR3ZBEMOQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.890934851 (316h29m10.442405312s) [noise=true , address=45.91.116.6:4000 , piece_id=T4CI3KKNMFPT3ZIODHE7BQ4DVHVH2QTSO6OB7QZPL5SNPXQJE6TA, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.891167002 (316h29m10.442637463s) [noise=true , address=77.90.31.17:4787 , piece_id=IYIEVH6CARRYCARKOLKWVBICQDNDRCTGQMWF3VJZ6YBC3GO5OWHQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.891369828 (316h29m10.442840289s) [noise=true , address=45.91.116.7:4039 , piece_id=HSRY63FXTEPCP2T46V77RVSC24GGRBAYLLU7DTKOLWGVOYHF6YVA, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.891556875 (316h29m10.443027336s) [noise=true , address=77.90.31.11:4572 , piece_id=G4XSIPSWOLSEOGGFA2NLOSDCGTLZPI3P5LDNNLB5RGMU5OQGWXGQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.891896812 (316h29m10.443367273s) [noise=true , address=77.90.31.19:4863 , piece_id=3PM3Q6X3ADKX2RKIIFDMK57LTJNJHNFK4UYPKFK3ZEQLYQ23EP3A, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.892019235 (316h29m10.443489696s) [noise=true , address=77.90.31.10:4524 , piece_id=Z2IGDHPDU6JAR5CAFQBGYHO3TQ66K4BZOAMXSC7X3RFWLGHOOMTQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.892316521 (316h29m10.443786982s) [noise=true , address=77.90.31.13:4644 , piece_id=UPDZGC3CMJXA3AOJX65VLASU37NLFGPIJLCAPN32TEQB5QHTU7EQ, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:11.892682318 (316h29m10.444152779s) [noise=true , address=45.91.116.9:4274 , piece_id=MXWRAXYLBBGLUVTA5GP3JVX2IRM2UI2IC35D7KX3XMFEBXLIDX5A, seg_pos={0 0}, upload=278731]: Uploading piece...
Nov 22 12:33:13.353758506 (316h29m11.905228957s) [upload=278731]: Next returned nil segment
Nov 22 12:33:13.353772043 (316h29m11.905242504s) [upload=278731]: Waiting for error group managing segments...
Nov 22 12:33:13.751703718 (316h29m12.303174169s) [noise=true , address=145.79.227.32:4858 , piece_id=K5L3LEUKZWHS36XJHNNY4TYRXCP4DG5UAKCZXPJJTSG26CKUI4HQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil>
Nov 22 12:33:13.753660293 (316h29m12.305130764s) [noise=true , address=145.79.227.26:4674 , piece_id=UP7VIVRDRL6KERJSX3VLWTKGW3ESGCQ7BLYCBXGBB5UNIDE6OHQQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil>
Nov 22 12:33:13.760046123 (316h29m12.311516574s) [noise=true , address=145.79.227.3:4167 , piece_id=4STIVXGU7SOXWAIPG2PXWBLNDD7EYQ56E2DBN2UZ2AZ3TZSXCTIQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil>
Nov 22 12:33:57.373975014 (316h29m55.925445475s) [noise=true , address=77.90.31.13:4644 , piece_id=UPDZGC3CMJXA3AOJX65VLASU37NLFGPIJLCAPN32TEQB5QHTU7EQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil>
Nov 22 12:34:06.122275078 (316h30m4.673745539s) [noise=true , address=45.91.116.13:4181 , piece_id=MQNJEBQFZMHNDKAH2K3JEA7YBK2UK5BGSNCJF3MASF3CKVA5VD6Q, seg_pos={0 0}, upload=278731]: Done uploading piece. err: <nil>
Nov 22 12:34:10.949171331 (316h30m9.500641792s) [noise=true , address=77.90.31.21:4890 , piece_id=DUEFPJUTYMD564Z4III63LZM6L4IL2ZTGM3QDZ6CG7WV5APHBNRA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1iE3mDnEhGDsMBtmUXC4EwvSrpyHLoybzVXkC9eoFuYEMHeBrd, address:77.90.31.21:4890): protocol: read tcp 188.166.225.170:55900->77.90.31.21:4890: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.949753650 (316h30m9.501224111s) [noise=true , address=45.91.116.12:4237 , piece_id=OG6XREYEGOYVFFMXCGBERRAIEMI2QMUGSBJBYP7GJQZ7FOAI4BIA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1FMjHdNGV42a9QZyMmSPTJWQrA8CdSRvXqrimnt6UGVreCzmgF, address:45.91.116.12:4237): protocol: rpc: dial tcp 45.91.116.12:4237: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.950066235 (316h30m9.501536706s) [noise=true , address=77.90.31.19:4863 , piece_id=3PM3Q6X3ADKX2RKIIFDMK57LTJNJHNFK4UYPKFK3ZEQLYQ23EP3A, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:124x1cbCPJRK8YkgSP6UzHymoLu1E1znnfg5WBiD89kSDLi8JeL, address:77.90.31.19:4863): protocol: write tcp 188.166.225.170:45968->77.90.31.19:4863: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.950246558 (316h30m9.50171702s) [noise=true , address=77.90.31.10:4524 , piece_id=Z2IGDHPDU6JAR5CAFQBGYHO3TQ66K4BZOAMXSC7X3RFWLGHOOMTQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12uN7CznoK4AD8FqFkNUndawdGJ9rmjAe7oNnjJkV844ScY87qe, address:77.90.31.10:4524): protocol: write tcp 188.166.225.170:40266->77.90.31.10:4524: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.950593058 (316h30m9.502063529s) [noise=true , address=77.90.31.14:4690 , piece_id=RJ3KXY3HURT7Q4HZYTOLSAL7Y6VMDGGRIQP6XHE4YB6OAA3PJRAA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv, address:77.90.31.14:4690): protocol: rpc: dial tcp 77.90.31.14:4690: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.950853535 (316h30m9.502324006s) [noise=true , address=45.91.116.14:4213 , piece_id=HPFISTE2N3B4RBDPAZAGVZJ5WYLCPHTCYH4WMJ56IWEHR3ZBEMOQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12tQRqowH4aDr9h3s1WLs1o7na1ueiafwgUeeHfAB6g7mcE6e1t, address:45.91.116.14:4213): protocol: rpc: dial tcp 45.91.116.14:4213: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.951186438 (316h30m9.502656889s) [noise=true , address=45.91.116.6:4000 , piece_id=T4CI3KKNMFPT3ZIODHE7BQ4DVHVH2QTSO6OB7QZPL5SNPXQJE6TA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12CuCqHJYbanbuaEMHvtcuQXhvcdahD8eyT2fKDhqtxdSZ6ES4f, address:45.91.116.6:4000): protocol: rpc: dial tcp 45.91.116.6:4000: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.951409864 (316h30m9.502880325s) [noise=true , address=45.91.116.7:4039 , piece_id=HSRY63FXTEPCP2T46V77RVSC24GGRBAYLLU7DTKOLWGVOYHF6YVA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:16oq1Yv7JfHrg4gdHfhwuMj4UsWeQQPayKV6Pj1NY2CgJhuPr2, address:45.91.116.7:4039): protocol: write tcp 188.166.225.170:36904->45.91.116.7:4039: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.951660191 (316h30m9.503130652s) [noise=true , address=77.90.31.11:4572 , piece_id=G4XSIPSWOLSEOGGFA2NLOSDCGTLZPI3P5LDNNLB5RGMU5OQGWXGQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1ZAfYZ4wgpZVvatj6wtHona3hTaPcLxAmXC1DT2uKQyo8nLQQ7, address:77.90.31.11:4572): protocol: rpc: dial tcp 77.90.31.11:4572: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.952019855 (316h30m9.503490316s) [noise=true , address=45.91.116.9:4274 , piece_id=MXWRAXYLBBGLUVTA5GP3JVX2IRM2UI2IC35D7KX3XMFEBXLIDX5A, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12g6jFa38dTz5UdgsQaEuM2MSH1H7cxaysWsVmbHkM2Dd2W2iDb, address:45.91.116.9:4274): protocol: rpc: dial tcp 45.91.116.9:4274: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.952290972 (316h30m9.503761443s) [noise=true , address=77.90.31.15:4720 , piece_id=BDQUF6C5TT3ASE3DIN6SSVJHOZAJK4ZEW33OIYPTGB4LDVRELSUA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:12VbsxFqMB3vVphSAJqYpbBzua7Goq45E6ovJSaExkpLtfkcc74, address:77.90.31.15:4720): protocol: rpc: dial tcp 77.90.31.15:4720: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.952544654 (316h30m9.504015105s) [noise=true , address=45.91.116.10:4110 , piece_id=H6FYA6K4N4AQYU3MN5RGYNBMYFXXBJDBTUCN37EAT7WBRNTX2YQA, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:123KwusATu1ZKtNNVHZZZBE67ZEnVb8puyfQidBi3fEWREqdf1h, address:45.91.116.10:4110): protocol: rpc: dial tcp 45.91.116.10:4110: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.953548155 (316h30m9.505018617s) [noise=true , address=77.90.31.17:4787 , piece_id=IYIEVH6CARRYCARKOLKWVBICQDNDRCTGQMWF3VJZ6YBC3GO5OWHQ, seg_pos={0 0}, upload=278731]: Done uploading piece. err: ecclient: upload failed (node:1Lietonb5aVbaZQt6wQ2EoxwJkSEn9oL9Dwyrc9mTyomEKikNt, address:77.90.31.17:4787): protocol: read tcp 188.166.225.170:57908->77.90.31.17:4787: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:10.953579816 (316h30m9.505050277s) [seg_pos={0 0}, upload=278731]: Done waiting for segment. successful: 5 optimal: 14 errs: <nil>
Nov 22 12:34:11.195719123 (316h30m9.747189584s) [upload=278731]: Done uploading segments...
Nov 22 12:34:11.195932840 (316h30m9.747403301s) [upload=278731]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 5)
Duration: 5.603669831s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
5 out of 60 is failed (threshold=6) [....X...............................................X..X.XX.] duration:
3.583591782s trace:
https://jaeger.dev.storj.io/trace/71781384eb27b231
FAIL (2025-11-22 12:34:50 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.58362207s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [...................................................X...X....] duration:
5.791103107s trace:
https://jaeger.dev.storj.io/trace/570ccd0ce1345814
FAIL (2025-11-22 12:34:33 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.791594735s
core
uplink
us-select-1
us1-placement-20
|