Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.....................................................X..X..X] duration:
30.000833255s trace:
https://jaeger.dev.storj.io/trace/92df32505639913
FAIL (2025-11-22 12:33:48 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.000861769s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
5 out of 60 is failed (threshold=6) [.....X...............................................X..X.XX] duration:
5.001634104s trace:
https://jaeger.dev.storj.io/trace/71781384eb27b231
FAIL (2025-11-22 12:33:41 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001660103s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [X......................................................X....] duration:
5.677647651s trace:
https://jaeger.dev.storj.io/trace/5ae3d2bf80495000
FAIL (2025-11-22 12:33:29 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Nov 22 12:31:07.276308940 (316h27m5.827779401s) [upload=278701]: Starting upload
Nov 22 12:31:07.276483523 (316h27m5.827953994s) [upload=278701]: Done starting upload
Nov 22 12:31:07.276562243 (316h27m5.828032704s) [upload=278701]: Uploading segments...
Nov 22 12:31:07.276950593 (316h27m5.828421054s) [upload=278701]: Got next segment. Inline: false
Nov 22 12:31:07.277027639 (316h27m5.8284981s) [upload=278701]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:31:07.277034181 (316h27m5.828504643s) [upload=278701]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:31:07.520212027 (316h27m6.071682488s) [seg_pos={0 0}, upload=278701]: Begin upload segment...
Nov 22 12:31:07.520269256 (316h27m6.071739717s) [seg_pos={0 0}, upload=278701]: Done begin upload segment.
Nov 22 12:31:07.520308992 (316h27m6.071779453s) [noise=true , address=145.79.227.27:4689 , piece_id=76S65KEZ7I2E76RR2JXOZ3ONGXCWSJA5XB7XQVFEEKGGNC34Z4EQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.520426746 (316h27m6.071897207s) [noise=true , address=145.79.227.21:4474 , piece_id=PCRWYQVCAZCPOILAEDHJGDBFKX4GEJ6JXCF7FUS7OVFRUTDPJI4Q, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.521175282 (316h27m6.072645743s) [noise=true , address=145.79.227.17:4972 , piece_id=UZK7LCIOBGGDNJZUVDBOLBJHGOIA4SUT2O3PD67PYQD3J2DGBJEA, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.521527573 (316h27m6.072998044s) [noise=true , address=145.79.227.23:4546 , piece_id=73UPPN2SYWNWEVWHCF4BRGKNO5GCRGOV3S5JZPX4MW75EJZRQOWQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.521694671 (316h27m6.073165132s) [noise=true , address=45.91.116.10:4082 , piece_id=GBDYXAWONOHQGBGM3NCRQYDSPFOAIHRJHBB2W6ZGJLMZUJLV65XA, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.522219410 (316h27m6.073689871s) [noise=true , address=145.79.227.20:4437 , piece_id=CZ32MB4AEZZ6YKCNWKRYALVAK7MT3KNXDSZCQTPUV2Q53NX6Y3BQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.522261240 (316h27m6.073731691s) [noise=true , address=145.79.227.16:4909 , piece_id=ZTL4ER5YJWSDGO5F5KLRUL5WRTA6WKVOJOERIBTRIIVEWXUY5Z2Q, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.522719151 (316h27m6.074189612s) [noise=true , address=145.79.227.26:4654 , piece_id=EM4RRGMPPNVR7VTXP3BED43FJFIUDZYVT4S5OUK2TRXZHYNMQ64Q, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.523017269 (316h27m6.07448772s) [noise=true , address=77.90.31.13:4627 , piece_id=Q7U5BFXHYQBBJQCU4EQGAR75WKL5QMBJ3LO34UYGO7QHLMDJBKWQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.523216949 (316h27m6.07468741s) [noise=true , address=145.79.227.25:4618 , piece_id=CV6M3CTPQR2MPQZFU6W34MFQERT6MMN3RRF2N3W35FGI464OSUUA, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.523744804 (316h27m6.075215265s) [noise=true , address=145.79.227.6:4258 , piece_id=B5D4TCVJORSAWDAU4DQ5VAV4CCRF5WUI24DUVUORAEKLAWGI7ABQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.523828153 (316h27m6.075298614s) [noise=true , address=145.79.227.24:4583 , piece_id=MLW4X34EWXKPOCFVI4VIQSL6RMUOJ5HRYHQ4HY35JIFT4ZFJ6F3Q, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.524255396 (316h27m6.075725858s) [noise=true , address=45.91.116.7:4009 , piece_id=FXQGF3UQN5CPFBS2HWD7BAXR4QIOF42MMK3VJQ7AK4THYJWOFFLQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.524662051 (316h27m6.076132512s) [noise=true , address=145.79.227.15:4874 , piece_id=TYAXIF3ZJQQSFGFANYF56DG4O5WBBXK4M7OT67E7ARYNZ663HESA, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.524746221 (316h27m6.076216682s) [noise=true , address=145.79.227.28:4726 , piece_id=3LBJV2B4CVXEMHCOPRW7DR5QBKPS6DFMPTHQSVNR4XRTZZVIGBQA, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.525227157 (316h27m6.076697608s) [noise=true , address=77.90.31.21:4879 , piece_id=GHTH55OGGXBAZZ5FGPCDYVHDYEETM7DNVTPSTTFFFRZMKZZ3XRXQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:09.134610631 (316h27m7.686081093s) [upload=278701]: Next returned nil segment
Nov 22 12:31:09.134628024 (316h27m7.686098495s) [upload=278701]: Waiting for error group managing segments...
Nov 22 12:31:09.502125778 (316h27m8.053596239s) [noise=true , address=145.79.227.16:4909 , piece_id=ZTL4ER5YJWSDGO5F5KLRUL5WRTA6WKVOJOERIBTRIIVEWXUY5Z2Q, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.503083742 (316h27m8.054554193s) [noise=true , address=145.79.227.20:4437 , piece_id=CZ32MB4AEZZ6YKCNWKRYALVAK7MT3KNXDSZCQTPUV2Q53NX6Y3BQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.517732716 (316h27m8.069203177s) [noise=true , address=145.79.227.25:4618 , piece_id=CV6M3CTPQR2MPQZFU6W34MFQERT6MMN3RRF2N3W35FGI464OSUUA, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.518029582 (316h27m8.069500043s) [noise=true , address=145.79.227.28:4726 , piece_id=3LBJV2B4CVXEMHCOPRW7DR5QBKPS6DFMPTHQSVNR4XRTZZVIGBQA, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.519935051 (316h27m8.071405512s) [noise=true , address=145.79.227.26:4654 , piece_id=EM4RRGMPPNVR7VTXP3BED43FJFIUDZYVT4S5OUK2TRXZHYNMQ64Q, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.520607661 (316h27m8.072078122s) [noise=true , address=145.79.227.27:4689 , piece_id=76S65KEZ7I2E76RR2JXOZ3ONGXCWSJA5XB7XQVFEEKGGNC34Z4EQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.521187736 (316h27m8.072658197s) [noise=true , address=145.79.227.23:4546 , piece_id=73UPPN2SYWNWEVWHCF4BRGKNO5GCRGOV3S5JZPX4MW75EJZRQOWQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.532974760 (316h27m8.084445221s) [noise=true , address=145.79.227.24:4583 , piece_id=MLW4X34EWXKPOCFVI4VIQSL6RMUOJ5HRYHQ4HY35JIFT4ZFJ6F3Q, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.533688800 (316h27m8.085159271s) [noise=true , address=145.79.227.6:4258 , piece_id=B5D4TCVJORSAWDAU4DQ5VAV4CCRF5WUI24DUVUORAEKLAWGI7ABQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.534460109 (316h27m8.08593057s) [noise=true , address=145.79.227.15:4874 , piece_id=TYAXIF3ZJQQSFGFANYF56DG4O5WBBXK4M7OT67E7ARYNZ663HESA, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.542955375 (316h27m8.094425826s) [noise=true , address=145.79.227.17:4972 , piece_id=UZK7LCIOBGGDNJZUVDBOLBJHGOIA4SUT2O3PD67PYQD3J2DGBJEA, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.563098982 (316h27m8.114569443s) [noise=true , address=145.79.227.21:4474 , piece_id=PCRWYQVCAZCPOILAEDHJGDBFKX4GEJ6JXCF7FUS7OVFRUTDPJI4Q, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:36.585475346 (316h27m35.136945807s) [noise=true , address=45.91.116.10:4082 , piece_id=GBDYXAWONOHQGBGM3NCRQYDSPFOAIHRJHBB2W6ZGJLMZUJLV65XA, seg_pos={0 0}, upload=278701]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: read tcp 188.166.225.170:54326->45.91.116.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:36.585499923 (316h27m35.136970384s) [noise=true , address=77.90.31.21:4879 , piece_id=GHTH55OGGXBAZZ5FGPCDYVHDYEETM7DNVTPSTTFFFRZMKZZ3XRXQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: rpc: dial tcp 77.90.31.21:4879: i/o timeout; context deadline exceeded
Nov 22 12:31:36.585749648 (316h27m35.137220119s) [noise=true , address=77.90.31.13:4627 , piece_id=Q7U5BFXHYQBBJQCU4EQGAR75WKL5QMBJ3LO34UYGO7QHLMDJBKWQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:36.585848086 (316h27m35.137318547s) [noise=true , address=45.91.116.7:4009 , piece_id=FXQGF3UQN5CPFBS2HWD7BAXR4QIOF42MMK3VJQ7AK4THYJWOFFLQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:36.585900936 (316h27m35.137371397s) [seg_pos={0 0}, upload=278701]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Nov 22 12:31:36.828881025 (316h27m35.380351486s) [upload=278701]: Done uploading segments...
Nov 22 12:31:36.828915801 (316h27m35.380386262s) [upload=278701]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 5.677678851s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [......X...............................................X..X.X] duration:
5.001444305s trace:
https://jaeger.dev.storj.io/trace/7c814404745ee005
FAIL (2025-11-22 12:33:20 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001473281s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.X......................................................X...] duration:
5.607803065s trace:
https://jaeger.dev.storj.io/trace/5ae3d2bf80495000
FAIL (2025-11-22 12:33:11 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Nov 22 12:31:07.276308940 (316h27m5.827779401s) [upload=278701]: Starting upload
Nov 22 12:31:07.276483523 (316h27m5.827953994s) [upload=278701]: Done starting upload
Nov 22 12:31:07.276562243 (316h27m5.828032704s) [upload=278701]: Uploading segments...
Nov 22 12:31:07.276950593 (316h27m5.828421054s) [upload=278701]: Got next segment. Inline: false
Nov 22 12:31:07.277027639 (316h27m5.8284981s) [upload=278701]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:31:07.277034181 (316h27m5.828504643s) [upload=278701]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:31:07.520212027 (316h27m6.071682488s) [seg_pos={0 0}, upload=278701]: Begin upload segment...
Nov 22 12:31:07.520269256 (316h27m6.071739717s) [seg_pos={0 0}, upload=278701]: Done begin upload segment.
Nov 22 12:31:07.520308992 (316h27m6.071779453s) [noise=true , address=145.79.227.27:4689 , piece_id=76S65KEZ7I2E76RR2JXOZ3ONGXCWSJA5XB7XQVFEEKGGNC34Z4EQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.520426746 (316h27m6.071897207s) [noise=true , address=145.79.227.21:4474 , piece_id=PCRWYQVCAZCPOILAEDHJGDBFKX4GEJ6JXCF7FUS7OVFRUTDPJI4Q, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.521175282 (316h27m6.072645743s) [noise=true , address=145.79.227.17:4972 , piece_id=UZK7LCIOBGGDNJZUVDBOLBJHGOIA4SUT2O3PD67PYQD3J2DGBJEA, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.521527573 (316h27m6.072998044s) [noise=true , address=145.79.227.23:4546 , piece_id=73UPPN2SYWNWEVWHCF4BRGKNO5GCRGOV3S5JZPX4MW75EJZRQOWQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.521694671 (316h27m6.073165132s) [noise=true , address=45.91.116.10:4082 , piece_id=GBDYXAWONOHQGBGM3NCRQYDSPFOAIHRJHBB2W6ZGJLMZUJLV65XA, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.522219410 (316h27m6.073689871s) [noise=true , address=145.79.227.20:4437 , piece_id=CZ32MB4AEZZ6YKCNWKRYALVAK7MT3KNXDSZCQTPUV2Q53NX6Y3BQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.522261240 (316h27m6.073731691s) [noise=true , address=145.79.227.16:4909 , piece_id=ZTL4ER5YJWSDGO5F5KLRUL5WRTA6WKVOJOERIBTRIIVEWXUY5Z2Q, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.522719151 (316h27m6.074189612s) [noise=true , address=145.79.227.26:4654 , piece_id=EM4RRGMPPNVR7VTXP3BED43FJFIUDZYVT4S5OUK2TRXZHYNMQ64Q, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.523017269 (316h27m6.07448772s) [noise=true , address=77.90.31.13:4627 , piece_id=Q7U5BFXHYQBBJQCU4EQGAR75WKL5QMBJ3LO34UYGO7QHLMDJBKWQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.523216949 (316h27m6.07468741s) [noise=true , address=145.79.227.25:4618 , piece_id=CV6M3CTPQR2MPQZFU6W34MFQERT6MMN3RRF2N3W35FGI464OSUUA, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.523744804 (316h27m6.075215265s) [noise=true , address=145.79.227.6:4258 , piece_id=B5D4TCVJORSAWDAU4DQ5VAV4CCRF5WUI24DUVUORAEKLAWGI7ABQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.523828153 (316h27m6.075298614s) [noise=true , address=145.79.227.24:4583 , piece_id=MLW4X34EWXKPOCFVI4VIQSL6RMUOJ5HRYHQ4HY35JIFT4ZFJ6F3Q, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.524255396 (316h27m6.075725858s) [noise=true , address=45.91.116.7:4009 , piece_id=FXQGF3UQN5CPFBS2HWD7BAXR4QIOF42MMK3VJQ7AK4THYJWOFFLQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.524662051 (316h27m6.076132512s) [noise=true , address=145.79.227.15:4874 , piece_id=TYAXIF3ZJQQSFGFANYF56DG4O5WBBXK4M7OT67E7ARYNZ663HESA, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.524746221 (316h27m6.076216682s) [noise=true , address=145.79.227.28:4726 , piece_id=3LBJV2B4CVXEMHCOPRW7DR5QBKPS6DFMPTHQSVNR4XRTZZVIGBQA, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.525227157 (316h27m6.076697608s) [noise=true , address=77.90.31.21:4879 , piece_id=GHTH55OGGXBAZZ5FGPCDYVHDYEETM7DNVTPSTTFFFRZMKZZ3XRXQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:09.134610631 (316h27m7.686081093s) [upload=278701]: Next returned nil segment
Nov 22 12:31:09.134628024 (316h27m7.686098495s) [upload=278701]: Waiting for error group managing segments...
Nov 22 12:31:09.502125778 (316h27m8.053596239s) [noise=true , address=145.79.227.16:4909 , piece_id=ZTL4ER5YJWSDGO5F5KLRUL5WRTA6WKVOJOERIBTRIIVEWXUY5Z2Q, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.503083742 (316h27m8.054554193s) [noise=true , address=145.79.227.20:4437 , piece_id=CZ32MB4AEZZ6YKCNWKRYALVAK7MT3KNXDSZCQTPUV2Q53NX6Y3BQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.517732716 (316h27m8.069203177s) [noise=true , address=145.79.227.25:4618 , piece_id=CV6M3CTPQR2MPQZFU6W34MFQERT6MMN3RRF2N3W35FGI464OSUUA, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.518029582 (316h27m8.069500043s) [noise=true , address=145.79.227.28:4726 , piece_id=3LBJV2B4CVXEMHCOPRW7DR5QBKPS6DFMPTHQSVNR4XRTZZVIGBQA, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.519935051 (316h27m8.071405512s) [noise=true , address=145.79.227.26:4654 , piece_id=EM4RRGMPPNVR7VTXP3BED43FJFIUDZYVT4S5OUK2TRXZHYNMQ64Q, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.520607661 (316h27m8.072078122s) [noise=true , address=145.79.227.27:4689 , piece_id=76S65KEZ7I2E76RR2JXOZ3ONGXCWSJA5XB7XQVFEEKGGNC34Z4EQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.521187736 (316h27m8.072658197s) [noise=true , address=145.79.227.23:4546 , piece_id=73UPPN2SYWNWEVWHCF4BRGKNO5GCRGOV3S5JZPX4MW75EJZRQOWQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.532974760 (316h27m8.084445221s) [noise=true , address=145.79.227.24:4583 , piece_id=MLW4X34EWXKPOCFVI4VIQSL6RMUOJ5HRYHQ4HY35JIFT4ZFJ6F3Q, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.533688800 (316h27m8.085159271s) [noise=true , address=145.79.227.6:4258 , piece_id=B5D4TCVJORSAWDAU4DQ5VAV4CCRF5WUI24DUVUORAEKLAWGI7ABQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.534460109 (316h27m8.08593057s) [noise=true , address=145.79.227.15:4874 , piece_id=TYAXIF3ZJQQSFGFANYF56DG4O5WBBXK4M7OT67E7ARYNZ663HESA, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.542955375 (316h27m8.094425826s) [noise=true , address=145.79.227.17:4972 , piece_id=UZK7LCIOBGGDNJZUVDBOLBJHGOIA4SUT2O3PD67PYQD3J2DGBJEA, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.563098982 (316h27m8.114569443s) [noise=true , address=145.79.227.21:4474 , piece_id=PCRWYQVCAZCPOILAEDHJGDBFKX4GEJ6JXCF7FUS7OVFRUTDPJI4Q, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:36.585475346 (316h27m35.136945807s) [noise=true , address=45.91.116.10:4082 , piece_id=GBDYXAWONOHQGBGM3NCRQYDSPFOAIHRJHBB2W6ZGJLMZUJLV65XA, seg_pos={0 0}, upload=278701]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: read tcp 188.166.225.170:54326->45.91.116.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:36.585499923 (316h27m35.136970384s) [noise=true , address=77.90.31.21:4879 , piece_id=GHTH55OGGXBAZZ5FGPCDYVHDYEETM7DNVTPSTTFFFRZMKZZ3XRXQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: rpc: dial tcp 77.90.31.21:4879: i/o timeout; context deadline exceeded
Nov 22 12:31:36.585749648 (316h27m35.137220119s) [noise=true , address=77.90.31.13:4627 , piece_id=Q7U5BFXHYQBBJQCU4EQGAR75WKL5QMBJ3LO34UYGO7QHLMDJBKWQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:36.585848086 (316h27m35.137318547s) [noise=true , address=45.91.116.7:4009 , piece_id=FXQGF3UQN5CPFBS2HWD7BAXR4QIOF42MMK3VJQ7AK4THYJWOFFLQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:36.585900936 (316h27m35.137371397s) [seg_pos={0 0}, upload=278701]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Nov 22 12:31:36.828881025 (316h27m35.380351486s) [upload=278701]: Done uploading segments...
Nov 22 12:31:36.828915801 (316h27m35.380386262s) [upload=278701]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 5.607827181s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..X......................................................X..] duration:
5.598376606s trace:
https://jaeger.dev.storj.io/trace/5ae3d2bf80495000
FAIL (2025-11-22 12:32:51 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Nov 22 12:31:07.276308940 (316h27m5.827779401s) [upload=278701]: Starting upload
Nov 22 12:31:07.276483523 (316h27m5.827953994s) [upload=278701]: Done starting upload
Nov 22 12:31:07.276562243 (316h27m5.828032704s) [upload=278701]: Uploading segments...
Nov 22 12:31:07.276950593 (316h27m5.828421054s) [upload=278701]: Got next segment. Inline: false
Nov 22 12:31:07.277027639 (316h27m5.8284981s) [upload=278701]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:31:07.277034181 (316h27m5.828504643s) [upload=278701]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:31:07.520212027 (316h27m6.071682488s) [seg_pos={0 0}, upload=278701]: Begin upload segment...
Nov 22 12:31:07.520269256 (316h27m6.071739717s) [seg_pos={0 0}, upload=278701]: Done begin upload segment.
Nov 22 12:31:07.520308992 (316h27m6.071779453s) [noise=true , address=145.79.227.27:4689 , piece_id=76S65KEZ7I2E76RR2JXOZ3ONGXCWSJA5XB7XQVFEEKGGNC34Z4EQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.520426746 (316h27m6.071897207s) [noise=true , address=145.79.227.21:4474 , piece_id=PCRWYQVCAZCPOILAEDHJGDBFKX4GEJ6JXCF7FUS7OVFRUTDPJI4Q, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.521175282 (316h27m6.072645743s) [noise=true , address=145.79.227.17:4972 , piece_id=UZK7LCIOBGGDNJZUVDBOLBJHGOIA4SUT2O3PD67PYQD3J2DGBJEA, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.521527573 (316h27m6.072998044s) [noise=true , address=145.79.227.23:4546 , piece_id=73UPPN2SYWNWEVWHCF4BRGKNO5GCRGOV3S5JZPX4MW75EJZRQOWQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.521694671 (316h27m6.073165132s) [noise=true , address=45.91.116.10:4082 , piece_id=GBDYXAWONOHQGBGM3NCRQYDSPFOAIHRJHBB2W6ZGJLMZUJLV65XA, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.522219410 (316h27m6.073689871s) [noise=true , address=145.79.227.20:4437 , piece_id=CZ32MB4AEZZ6YKCNWKRYALVAK7MT3KNXDSZCQTPUV2Q53NX6Y3BQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.522261240 (316h27m6.073731691s) [noise=true , address=145.79.227.16:4909 , piece_id=ZTL4ER5YJWSDGO5F5KLRUL5WRTA6WKVOJOERIBTRIIVEWXUY5Z2Q, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.522719151 (316h27m6.074189612s) [noise=true , address=145.79.227.26:4654 , piece_id=EM4RRGMPPNVR7VTXP3BED43FJFIUDZYVT4S5OUK2TRXZHYNMQ64Q, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.523017269 (316h27m6.07448772s) [noise=true , address=77.90.31.13:4627 , piece_id=Q7U5BFXHYQBBJQCU4EQGAR75WKL5QMBJ3LO34UYGO7QHLMDJBKWQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.523216949 (316h27m6.07468741s) [noise=true , address=145.79.227.25:4618 , piece_id=CV6M3CTPQR2MPQZFU6W34MFQERT6MMN3RRF2N3W35FGI464OSUUA, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.523744804 (316h27m6.075215265s) [noise=true , address=145.79.227.6:4258 , piece_id=B5D4TCVJORSAWDAU4DQ5VAV4CCRF5WUI24DUVUORAEKLAWGI7ABQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.523828153 (316h27m6.075298614s) [noise=true , address=145.79.227.24:4583 , piece_id=MLW4X34EWXKPOCFVI4VIQSL6RMUOJ5HRYHQ4HY35JIFT4ZFJ6F3Q, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.524255396 (316h27m6.075725858s) [noise=true , address=45.91.116.7:4009 , piece_id=FXQGF3UQN5CPFBS2HWD7BAXR4QIOF42MMK3VJQ7AK4THYJWOFFLQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.524662051 (316h27m6.076132512s) [noise=true , address=145.79.227.15:4874 , piece_id=TYAXIF3ZJQQSFGFANYF56DG4O5WBBXK4M7OT67E7ARYNZ663HESA, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.524746221 (316h27m6.076216682s) [noise=true , address=145.79.227.28:4726 , piece_id=3LBJV2B4CVXEMHCOPRW7DR5QBKPS6DFMPTHQSVNR4XRTZZVIGBQA, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.525227157 (316h27m6.076697608s) [noise=true , address=77.90.31.21:4879 , piece_id=GHTH55OGGXBAZZ5FGPCDYVHDYEETM7DNVTPSTTFFFRZMKZZ3XRXQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:09.134610631 (316h27m7.686081093s) [upload=278701]: Next returned nil segment
Nov 22 12:31:09.134628024 (316h27m7.686098495s) [upload=278701]: Waiting for error group managing segments...
Nov 22 12:31:09.502125778 (316h27m8.053596239s) [noise=true , address=145.79.227.16:4909 , piece_id=ZTL4ER5YJWSDGO5F5KLRUL5WRTA6WKVOJOERIBTRIIVEWXUY5Z2Q, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.503083742 (316h27m8.054554193s) [noise=true , address=145.79.227.20:4437 , piece_id=CZ32MB4AEZZ6YKCNWKRYALVAK7MT3KNXDSZCQTPUV2Q53NX6Y3BQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.517732716 (316h27m8.069203177s) [noise=true , address=145.79.227.25:4618 , piece_id=CV6M3CTPQR2MPQZFU6W34MFQERT6MMN3RRF2N3W35FGI464OSUUA, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.518029582 (316h27m8.069500043s) [noise=true , address=145.79.227.28:4726 , piece_id=3LBJV2B4CVXEMHCOPRW7DR5QBKPS6DFMPTHQSVNR4XRTZZVIGBQA, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.519935051 (316h27m8.071405512s) [noise=true , address=145.79.227.26:4654 , piece_id=EM4RRGMPPNVR7VTXP3BED43FJFIUDZYVT4S5OUK2TRXZHYNMQ64Q, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.520607661 (316h27m8.072078122s) [noise=true , address=145.79.227.27:4689 , piece_id=76S65KEZ7I2E76RR2JXOZ3ONGXCWSJA5XB7XQVFEEKGGNC34Z4EQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.521187736 (316h27m8.072658197s) [noise=true , address=145.79.227.23:4546 , piece_id=73UPPN2SYWNWEVWHCF4BRGKNO5GCRGOV3S5JZPX4MW75EJZRQOWQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.532974760 (316h27m8.084445221s) [noise=true , address=145.79.227.24:4583 , piece_id=MLW4X34EWXKPOCFVI4VIQSL6RMUOJ5HRYHQ4HY35JIFT4ZFJ6F3Q, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.533688800 (316h27m8.085159271s) [noise=true , address=145.79.227.6:4258 , piece_id=B5D4TCVJORSAWDAU4DQ5VAV4CCRF5WUI24DUVUORAEKLAWGI7ABQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.534460109 (316h27m8.08593057s) [noise=true , address=145.79.227.15:4874 , piece_id=TYAXIF3ZJQQSFGFANYF56DG4O5WBBXK4M7OT67E7ARYNZ663HESA, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.542955375 (316h27m8.094425826s) [noise=true , address=145.79.227.17:4972 , piece_id=UZK7LCIOBGGDNJZUVDBOLBJHGOIA4SUT2O3PD67PYQD3J2DGBJEA, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.563098982 (316h27m8.114569443s) [noise=true , address=145.79.227.21:4474 , piece_id=PCRWYQVCAZCPOILAEDHJGDBFKX4GEJ6JXCF7FUS7OVFRUTDPJI4Q, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:36.585475346 (316h27m35.136945807s) [noise=true , address=45.91.116.10:4082 , piece_id=GBDYXAWONOHQGBGM3NCRQYDSPFOAIHRJHBB2W6ZGJLMZUJLV65XA, seg_pos={0 0}, upload=278701]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: read tcp 188.166.225.170:54326->45.91.116.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:36.585499923 (316h27m35.136970384s) [noise=true , address=77.90.31.21:4879 , piece_id=GHTH55OGGXBAZZ5FGPCDYVHDYEETM7DNVTPSTTFFFRZMKZZ3XRXQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: rpc: dial tcp 77.90.31.21:4879: i/o timeout; context deadline exceeded
Nov 22 12:31:36.585749648 (316h27m35.137220119s) [noise=true , address=77.90.31.13:4627 , piece_id=Q7U5BFXHYQBBJQCU4EQGAR75WKL5QMBJ3LO34UYGO7QHLMDJBKWQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:36.585848086 (316h27m35.137318547s) [noise=true , address=45.91.116.7:4009 , piece_id=FXQGF3UQN5CPFBS2HWD7BAXR4QIOF42MMK3VJQ7AK4THYJWOFFLQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:36.585900936 (316h27m35.137371397s) [seg_pos={0 0}, upload=278701]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Nov 22 12:31:36.828881025 (316h27m35.380351486s) [upload=278701]: Done uploading segments...
Nov 22 12:31:36.828915801 (316h27m35.380386262s) [upload=278701]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 5.598403807s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [.....................................................X...X..] duration:
6.030989932s trace:
https://jaeger.dev.storj.io/trace/570ccd0ce1345814
FAIL (2025-11-22 12:32:40 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.031012445s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [......................................................X.....] duration:
5.678420023s trace:
https://jaeger.dev.storj.io/trace/78d837dffbcbc1eb
FAIL (2025-11-22 12:32:20 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 6)
======LOGS======
Nov 22 12:27:49.568493454 (316h23m48.119963905s) [noise=true , address=77.90.31.19:4876 , piece_id=RDATPSCRLI5BNZVPSBOSAPUK5GFYD2MWCIPRAL6NMPA4Z6GDNZ5A, seg_pos={0 0}, upload=278652]: Uploading piece...
Nov 22 12:27:49.570659288 (316h23m48.122129749s) [noise=true , address=45.91.116.5:4306 , piece_id=F437446OQSONT5YJK35AQ26QMDMC3HP2CWUPEP46WKZGWNRL2V5A, seg_pos={0 0}, upload=278652]: Uploading piece...
Nov 22 12:27:49.572948898 (316h23m48.124419359s) [noise=true , address=64.49.13.140:4506 , piece_id=B5MGJGEBUK6H2JD6LVVKYO34GYGTKQ2RRLTT53QMJHDJGLAH2PPA, seg_pos={0 0}, upload=278652]: Uploading piece...
Nov 22 12:27:49.573111158 (316h23m48.124581609s) [noise=true , address=45.91.116.4:4948 , piece_id=WNZONHIDT5WLFQW6ZYMZOOQ2DT4ATWKBMRGKVLI5SKY53MAMCEYQ, seg_pos={0 0}, upload=278652]: Uploading piece...
Nov 22 12:27:49.574907569 (316h23m48.12637803s) [noise=true , address=45.91.116.10:4087 , piece_id=W4QTE4CCAQGK6KODK2DFBY6YEGEGYOHZS7WBJMV7IN3PCHOMOG6A, seg_pos={0 0}, upload=278652]: Uploading piece...
Nov 22 12:27:49.576925240 (316h23m48.128395701s) [noise=true , address=77.90.31.10:4539 , piece_id=I5AHSBYZXFUVBLW4L4R4CFVJOFFPLQB7F5AK2GBYDXISM6N72PDQ, seg_pos={0 0}, upload=278652]: Uploading piece...
Nov 22 12:27:49.579508780 (316h23m48.130979241s) [noise=true , address=145.79.227.33:4894 , piece_id=QD6HDQDS37Q6HOIU522KZZ44LXF2AKRVKL5PIQFG437KTXYMKC6Q, seg_pos={0 0}, upload=278652]: Uploading piece...
Nov 22 12:27:49.580586642 (316h23m48.132057104s) [noise=true , address=77.90.31.13:4630 , piece_id=FNU2RL7WP5G4RR3HMDEEKYKNJBLTH6SKZ3FFPZK4B7U4A2RRWWKQ, seg_pos={0 0}, upload=278652]: Uploading piece...
Nov 22 12:27:49.585268507 (316h23m48.136738968s) [noise=true , address=77.90.31.12:4608 , piece_id=X2N6DK2U6TUZXCP356BDGEBOWY44UEOCG73CWEY57GY7OA7ZTNXA, seg_pos={0 0}, upload=278652]: Uploading piece...
Nov 22 12:27:49.588099719 (316h23m48.13957018s) [noise=true , address=45.91.116.6:4986 , piece_id=RHKQEB2Y7KCD4CIROR6DSYPTVG2SUNDKE7C5QW7DHHM24YJHMNOQ, seg_pos={0 0}, upload=278652]: Uploading piece...
Nov 22 12:27:49.589734080 (316h23m48.141204531s) [noise=true , address=145.79.227.12:4790 , piece_id=ZCJGR7J7ZDQWNMWSLFPHSGRGRQYLNMDSO52Z2TFQCQCBXIBVPZYA, seg_pos={0 0}, upload=278652]: Uploading piece...
Nov 22 12:27:49.592364720 (316h23m48.143835201s) [noise=true , address=77.90.31.17:4799 , piece_id=K2U42DVPQMOBOXQ4J2FZDWEQURTV42XSVR23AT32YCKEXAPD24RA, seg_pos={0 0}, upload=278652]: Uploading piece...
Nov 22 12:27:49.600851380 (316h23m48.152321841s) [noise=true , address=145.79.227.7:4314 , piece_id=D2CQYNQ7U5IP4QRSTASQIO5JBXYPWSHOP3HN4SVVOTN6LIQNSN3A, seg_pos={0 0}, upload=278652]: Uploading piece...
Nov 22 12:27:50.972898701 (316h23m49.524369162s) [upload=278652]: Next returned nil segment
Nov 22 12:27:50.972914963 (316h23m49.524385414s) [upload=278652]: Waiting for error group managing segments...
Nov 22 12:27:51.338580227 (316h23m49.890050688s) [noise=true , address=64.49.13.140:4506 , piece_id=B5MGJGEBUK6H2JD6LVVKYO34GYGTKQ2RRLTT53QMJHDJGLAH2PPA, seg_pos={0 0}, upload=278652]: Done uploading piece. err: <nil>
Nov 22 12:27:51.425396702 (316h23m49.976867163s) [noise=true , address=145.79.227.5:4256 , piece_id=UOFRVFML2YA53NWWQZMB3GA5PW4NFULCJJHMALH7CHHNGSG3N7VA, seg_pos={0 0}, upload=278652]: Done uploading piece. err: <nil>
Nov 22 12:27:51.444596362 (316h23m49.996066823s) [noise=true , address=145.79.227.49:4354 , piece_id=357B2N2MEKIVI7ERX5GY33FDAZOO2A3ENZV74UVPEBYSYDNI5GIQ, seg_pos={0 0}, upload=278652]: Done uploading piece. err: <nil>
Nov 22 12:27:51.448630013 (316h23m50.000100474s) [noise=true , address=145.79.227.33:4894 , piece_id=QD6HDQDS37Q6HOIU522KZZ44LXF2AKRVKL5PIQFG437KTXYMKC6Q, seg_pos={0 0}, upload=278652]: Done uploading piece. err: <nil>
Nov 22 12:27:51.466112773 (316h23m50.017583234s) [noise=true , address=145.79.227.12:4790 , piece_id=ZCJGR7J7ZDQWNMWSLFPHSGRGRQYLNMDSO52Z2TFQCQCBXIBVPZYA, seg_pos={0 0}, upload=278652]: Done uploading piece. err: <nil>
Nov 22 12:27:51.471437682 (316h23m50.022908133s) [noise=true , address=145.79.227.7:4314 , piece_id=D2CQYNQ7U5IP4QRSTASQIO5JBXYPWSHOP3HN4SVVOTN6LIQNSN3A, seg_pos={0 0}, upload=278652]: Done uploading piece. err: <nil>
Nov 22 12:28:48.617770024 (316h24m47.169240485s) [noise=true , address=77.90.31.17:4799 , piece_id=K2U42DVPQMOBOXQ4J2FZDWEQURTV42XSVR23AT32YCKEXAPD24RA, seg_pos={0 0}, upload=278652]: Done uploading piece. err: ecclient: upload failed (node:12UhAksgXZuFGAVheaYzP7D4kwVyAjUAq3BtrvdSxszjNCFCTWT, address:77.90.31.17:4799): protocol: rpc: dial tcp 77.90.31.17:4799: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:48.617979923 (316h24m47.169450384s) [noise=true , address=45.91.116.5:4306 , piece_id=F437446OQSONT5YJK35AQ26QMDMC3HP2CWUPEP46WKZGWNRL2V5A, seg_pos={0 0}, upload=278652]: Done uploading piece. err: ecclient: upload failed (node:1QuFvRW9wGKdKRXPK8UdR8uqTm53CnHaF9CsUz2bUh2Ptw9r1F, address:45.91.116.5:4306): protocol: rpc: dial tcp 45.91.116.5:4306: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:48.618033846 (316h24m47.169504308s) [noise=true , address=77.90.31.21:4885 , piece_id=5OL3NVLKPFB23XQFUDQLBX6EJHPF5VICQYSYQKCSDY5SWHILW4QA, seg_pos={0 0}, upload=278652]: Done uploading piece. err: ecclient: upload failed (node:12TfMGrvCMPvQ9ZC1pVoDqjFnUHXtzwwnZB6pCV1em4pdmbK2MP, address:77.90.31.21:4885): protocol: rpc: dial tcp 77.90.31.21:4885: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:48.618228647 (316h24m47.169699098s) [noise=true , address=45.91.116.4:4948 , piece_id=WNZONHIDT5WLFQW6ZYMZOOQ2DT4ATWKBMRGKVLI5SKY53MAMCEYQ, seg_pos={0 0}, upload=278652]: Done uploading piece. err: ecclient: upload failed (node:184g6qHdezLW3d2CKNW8GRgdzeVLQuMEhd8B8c1zEnkKNkeSTu, address:45.91.116.4:4948): protocol: rpc: dial tcp 45.91.116.4:4948: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:48.618249357 (316h24m47.169719848s) [noise=true , address=45.91.116.9:4279 , piece_id=LSPEKODU2YPLRUI6MY37VVRL6QNUJ2IXTZX6BBJHEVDMLTZGNQVQ, seg_pos={0 0}, upload=278652]: Done uploading piece. err: ecclient: upload failed (node:124aq2BHBfu1qSRXCx5XVZJU5U7gzadYHSRLYT8fcwcrtu9z5jG, address:45.91.116.9:4279): protocol: rpc: dial tcp 45.91.116.9:4279: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:48.618444829 (316h24m47.16991529s) [noise=true , address=77.90.31.10:4539 , piece_id=I5AHSBYZXFUVBLW4L4R4CFVJOFFPLQB7F5AK2GBYDXISM6N72PDQ, seg_pos={0 0}, upload=278652]: Done uploading piece. err: ecclient: upload failed (node:1DYHaVii53Q4FhtYNBMC64MP1uKHffSTMdfETK8kS94e4ZPfka, address:77.90.31.10:4539): protocol: rpc: dial tcp 77.90.31.10:4539: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:48.618487821 (316h24m47.169958272s) [noise=true , address=45.91.116.14:4211 , piece_id=5HS3X7L6ACA6KJNWBBEK36IIBOFNANCMF4QL367VJA5OHCJW4SOQ, seg_pos={0 0}, upload=278652]: Done uploading piece. err: ecclient: upload failed (node:1TkBbeKqsAZHyZdNpDgCb2uexqgV59Atgtv7EM1rd4KJEZ4BqF, address:45.91.116.14:4211): protocol: rpc: dial tcp 45.91.116.14:4211: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:48.618612950 (316h24m47.170083451s) [noise=true , address=45.91.116.6:4986 , piece_id=RHKQEB2Y7KCD4CIROR6DSYPTVG2SUNDKE7C5QW7DHHM24YJHMNOQ, seg_pos={0 0}, upload=278652]: Done uploading piece. err: ecclient: upload failed (node:12t1cuEr8MRLBbJ4GNSdYBxdhNQ6xDbTr8PHoEuVhMqh1ksgWon, address:45.91.116.6:4986): protocol: write tcp 188.166.225.170:41196->45.91.116.6:4986: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:48.618698352 (316h24m47.170168813s) [noise=true , address=77.90.31.13:4630 , piece_id=FNU2RL7WP5G4RR3HMDEEKYKNJBLTH6SKZ3FFPZK4B7U4A2RRWWKQ, seg_pos={0 0}, upload=278652]: Done uploading piece. err: ecclient: upload failed (node:1gyQJLnzWmRESyd8m6G7aZWsQwVfqRQNARxRmd7GwwfYwhuzVn, address:77.90.31.13:4630): protocol: rpc: dial tcp 77.90.31.13:4630: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:48.618875479 (316h24m47.17034594s) [noise=true , address=77.90.31.12:4608 , piece_id=X2N6DK2U6TUZXCP356BDGEBOWY44UEOCG73CWEY57GY7OA7ZTNXA, seg_pos={0 0}, upload=278652]: Done uploading piece. err: ecclient: upload failed (node:12AgtYnGt6M45rGyTZRr4qCFp1BEkgNPBxxsWqBKEn1Cx7AbYLu, address:77.90.31.12:4608): protocol: rpc: dial tcp 77.90.31.12:4608: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:48.619011127 (316h24m47.170481588s) [noise=true , address=77.90.31.19:4876 , piece_id=RDATPSCRLI5BNZVPSBOSAPUK5GFYD2MWCIPRAL6NMPA4Z6GDNZ5A, seg_pos={0 0}, upload=278652]: Done uploading piece. err: ecclient: upload failed (node:12H3UNBGeAufYSdVvEgYkv2QWeFtjKMvHvYGeqGeBvjEHqDDjBB, address:77.90.31.19:4876): protocol: rpc: dial tcp 77.90.31.19:4876: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:48.619102862 (316h24m47.170573323s) [noise=true , address=45.91.116.10:4087 , piece_id=W4QTE4CCAQGK6KODK2DFBY6YEGEGYOHZS7WBJMV7IN3PCHOMOG6A, seg_pos={0 0}, upload=278652]: Done uploading piece. err: ecclient: upload failed (node:125T5WzC4RDcp21oJ8zBMMBq5uw2X32Wf5FszZiWUdsSR95Pd2H, address:45.91.116.10:4087): protocol: write tcp 188.166.225.170:50656->45.91.116.10:4087: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:48.619135034 (316h24m47.170605495s) [seg_pos={0 0}, upload=278652]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil>
Nov 22 12:28:48.858048038 (316h24m47.409518489s) [upload=278652]: Done uploading segments...
Nov 22 12:28:48.858131627 (316h24m47.409602088s) [upload=278652]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6)
Duration: 5.678442586s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [......................................................X..X..] duration:
8.082876313s trace:
https://jaeger.dev.storj.io/trace/3e7bcddde35fe2d
FAIL (2025-11-22 12:32:18 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 8.082907852s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=6) [.......X...............................................X..X.] duration:
3.344867136s trace:
https://jaeger.dev.storj.io/trace/47ae36a3e1dc74d6
FAIL (2025-11-22 12:32:17 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.344896642s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [......................................................X...X.] duration:
10.797283219s trace:
https://jaeger.dev.storj.io/trace/570ccd0ce1345814
FAIL (2025-11-22 12:32:07 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 10.797308285s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [...X......................................................X.] duration:
6.007429098s trace:
https://jaeger.dev.storj.io/trace/5ae3d2bf80495000
FAIL (2025-11-22 12:32:05 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Nov 22 12:31:07.276308940 (316h27m5.827779401s) [upload=278701]: Starting upload
Nov 22 12:31:07.276483523 (316h27m5.827953994s) [upload=278701]: Done starting upload
Nov 22 12:31:07.276562243 (316h27m5.828032704s) [upload=278701]: Uploading segments...
Nov 22 12:31:07.276950593 (316h27m5.828421054s) [upload=278701]: Got next segment. Inline: false
Nov 22 12:31:07.277027639 (316h27m5.8284981s) [upload=278701]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:31:07.277034181 (316h27m5.828504643s) [upload=278701]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:31:07.520212027 (316h27m6.071682488s) [seg_pos={0 0}, upload=278701]: Begin upload segment...
Nov 22 12:31:07.520269256 (316h27m6.071739717s) [seg_pos={0 0}, upload=278701]: Done begin upload segment.
Nov 22 12:31:07.520308992 (316h27m6.071779453s) [noise=true , address=145.79.227.27:4689 , piece_id=76S65KEZ7I2E76RR2JXOZ3ONGXCWSJA5XB7XQVFEEKGGNC34Z4EQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.520426746 (316h27m6.071897207s) [noise=true , address=145.79.227.21:4474 , piece_id=PCRWYQVCAZCPOILAEDHJGDBFKX4GEJ6JXCF7FUS7OVFRUTDPJI4Q, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.521175282 (316h27m6.072645743s) [noise=true , address=145.79.227.17:4972 , piece_id=UZK7LCIOBGGDNJZUVDBOLBJHGOIA4SUT2O3PD67PYQD3J2DGBJEA, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.521527573 (316h27m6.072998044s) [noise=true , address=145.79.227.23:4546 , piece_id=73UPPN2SYWNWEVWHCF4BRGKNO5GCRGOV3S5JZPX4MW75EJZRQOWQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.521694671 (316h27m6.073165132s) [noise=true , address=45.91.116.10:4082 , piece_id=GBDYXAWONOHQGBGM3NCRQYDSPFOAIHRJHBB2W6ZGJLMZUJLV65XA, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.522219410 (316h27m6.073689871s) [noise=true , address=145.79.227.20:4437 , piece_id=CZ32MB4AEZZ6YKCNWKRYALVAK7MT3KNXDSZCQTPUV2Q53NX6Y3BQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.522261240 (316h27m6.073731691s) [noise=true , address=145.79.227.16:4909 , piece_id=ZTL4ER5YJWSDGO5F5KLRUL5WRTA6WKVOJOERIBTRIIVEWXUY5Z2Q, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.522719151 (316h27m6.074189612s) [noise=true , address=145.79.227.26:4654 , piece_id=EM4RRGMPPNVR7VTXP3BED43FJFIUDZYVT4S5OUK2TRXZHYNMQ64Q, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.523017269 (316h27m6.07448772s) [noise=true , address=77.90.31.13:4627 , piece_id=Q7U5BFXHYQBBJQCU4EQGAR75WKL5QMBJ3LO34UYGO7QHLMDJBKWQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.523216949 (316h27m6.07468741s) [noise=true , address=145.79.227.25:4618 , piece_id=CV6M3CTPQR2MPQZFU6W34MFQERT6MMN3RRF2N3W35FGI464OSUUA, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.523744804 (316h27m6.075215265s) [noise=true , address=145.79.227.6:4258 , piece_id=B5D4TCVJORSAWDAU4DQ5VAV4CCRF5WUI24DUVUORAEKLAWGI7ABQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.523828153 (316h27m6.075298614s) [noise=true , address=145.79.227.24:4583 , piece_id=MLW4X34EWXKPOCFVI4VIQSL6RMUOJ5HRYHQ4HY35JIFT4ZFJ6F3Q, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.524255396 (316h27m6.075725858s) [noise=true , address=45.91.116.7:4009 , piece_id=FXQGF3UQN5CPFBS2HWD7BAXR4QIOF42MMK3VJQ7AK4THYJWOFFLQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.524662051 (316h27m6.076132512s) [noise=true , address=145.79.227.15:4874 , piece_id=TYAXIF3ZJQQSFGFANYF56DG4O5WBBXK4M7OT67E7ARYNZ663HESA, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.524746221 (316h27m6.076216682s) [noise=true , address=145.79.227.28:4726 , piece_id=3LBJV2B4CVXEMHCOPRW7DR5QBKPS6DFMPTHQSVNR4XRTZZVIGBQA, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.525227157 (316h27m6.076697608s) [noise=true , address=77.90.31.21:4879 , piece_id=GHTH55OGGXBAZZ5FGPCDYVHDYEETM7DNVTPSTTFFFRZMKZZ3XRXQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:09.134610631 (316h27m7.686081093s) [upload=278701]: Next returned nil segment
Nov 22 12:31:09.134628024 (316h27m7.686098495s) [upload=278701]: Waiting for error group managing segments...
Nov 22 12:31:09.502125778 (316h27m8.053596239s) [noise=true , address=145.79.227.16:4909 , piece_id=ZTL4ER5YJWSDGO5F5KLRUL5WRTA6WKVOJOERIBTRIIVEWXUY5Z2Q, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.503083742 (316h27m8.054554193s) [noise=true , address=145.79.227.20:4437 , piece_id=CZ32MB4AEZZ6YKCNWKRYALVAK7MT3KNXDSZCQTPUV2Q53NX6Y3BQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.517732716 (316h27m8.069203177s) [noise=true , address=145.79.227.25:4618 , piece_id=CV6M3CTPQR2MPQZFU6W34MFQERT6MMN3RRF2N3W35FGI464OSUUA, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.518029582 (316h27m8.069500043s) [noise=true , address=145.79.227.28:4726 , piece_id=3LBJV2B4CVXEMHCOPRW7DR5QBKPS6DFMPTHQSVNR4XRTZZVIGBQA, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.519935051 (316h27m8.071405512s) [noise=true , address=145.79.227.26:4654 , piece_id=EM4RRGMPPNVR7VTXP3BED43FJFIUDZYVT4S5OUK2TRXZHYNMQ64Q, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.520607661 (316h27m8.072078122s) [noise=true , address=145.79.227.27:4689 , piece_id=76S65KEZ7I2E76RR2JXOZ3ONGXCWSJA5XB7XQVFEEKGGNC34Z4EQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.521187736 (316h27m8.072658197s) [noise=true , address=145.79.227.23:4546 , piece_id=73UPPN2SYWNWEVWHCF4BRGKNO5GCRGOV3S5JZPX4MW75EJZRQOWQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.532974760 (316h27m8.084445221s) [noise=true , address=145.79.227.24:4583 , piece_id=MLW4X34EWXKPOCFVI4VIQSL6RMUOJ5HRYHQ4HY35JIFT4ZFJ6F3Q, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.533688800 (316h27m8.085159271s) [noise=true , address=145.79.227.6:4258 , piece_id=B5D4TCVJORSAWDAU4DQ5VAV4CCRF5WUI24DUVUORAEKLAWGI7ABQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.534460109 (316h27m8.08593057s) [noise=true , address=145.79.227.15:4874 , piece_id=TYAXIF3ZJQQSFGFANYF56DG4O5WBBXK4M7OT67E7ARYNZ663HESA, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.542955375 (316h27m8.094425826s) [noise=true , address=145.79.227.17:4972 , piece_id=UZK7LCIOBGGDNJZUVDBOLBJHGOIA4SUT2O3PD67PYQD3J2DGBJEA, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.563098982 (316h27m8.114569443s) [noise=true , address=145.79.227.21:4474 , piece_id=PCRWYQVCAZCPOILAEDHJGDBFKX4GEJ6JXCF7FUS7OVFRUTDPJI4Q, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:36.585475346 (316h27m35.136945807s) [noise=true , address=45.91.116.10:4082 , piece_id=GBDYXAWONOHQGBGM3NCRQYDSPFOAIHRJHBB2W6ZGJLMZUJLV65XA, seg_pos={0 0}, upload=278701]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: read tcp 188.166.225.170:54326->45.91.116.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:36.585499923 (316h27m35.136970384s) [noise=true , address=77.90.31.21:4879 , piece_id=GHTH55OGGXBAZZ5FGPCDYVHDYEETM7DNVTPSTTFFFRZMKZZ3XRXQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: rpc: dial tcp 77.90.31.21:4879: i/o timeout; context deadline exceeded
Nov 22 12:31:36.585749648 (316h27m35.137220119s) [noise=true , address=77.90.31.13:4627 , piece_id=Q7U5BFXHYQBBJQCU4EQGAR75WKL5QMBJ3LO34UYGO7QHLMDJBKWQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:36.585848086 (316h27m35.137318547s) [noise=true , address=45.91.116.7:4009 , piece_id=FXQGF3UQN5CPFBS2HWD7BAXR4QIOF42MMK3VJQ7AK4THYJWOFFLQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:36.585900936 (316h27m35.137371397s) [seg_pos={0 0}, upload=278701]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Nov 22 12:31:36.828881025 (316h27m35.380351486s) [upload=278701]: Done uploading segments...
Nov 22 12:31:36.828915801 (316h27m35.380386262s) [upload=278701]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 6.007457512s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......................................................X....] duration:
37.913538538s trace:
https://jaeger.dev.storj.io/trace/78d837dffbcbc1eb
FAIL (2025-11-22 12:31:48 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 6)
======LOGS======
Nov 22 12:27:49.568493454 (316h23m48.119963905s) [noise=true , address=77.90.31.19:4876 , piece_id=RDATPSCRLI5BNZVPSBOSAPUK5GFYD2MWCIPRAL6NMPA4Z6GDNZ5A, seg_pos={0 0}, upload=278652]: Uploading piece...
Nov 22 12:27:49.570659288 (316h23m48.122129749s) [noise=true , address=45.91.116.5:4306 , piece_id=F437446OQSONT5YJK35AQ26QMDMC3HP2CWUPEP46WKZGWNRL2V5A, seg_pos={0 0}, upload=278652]: Uploading piece...
Nov 22 12:27:49.572948898 (316h23m48.124419359s) [noise=true , address=64.49.13.140:4506 , piece_id=B5MGJGEBUK6H2JD6LVVKYO34GYGTKQ2RRLTT53QMJHDJGLAH2PPA, seg_pos={0 0}, upload=278652]: Uploading piece...
Nov 22 12:27:49.573111158 (316h23m48.124581609s) [noise=true , address=45.91.116.4:4948 , piece_id=WNZONHIDT5WLFQW6ZYMZOOQ2DT4ATWKBMRGKVLI5SKY53MAMCEYQ, seg_pos={0 0}, upload=278652]: Uploading piece...
Nov 22 12:27:49.574907569 (316h23m48.12637803s) [noise=true , address=45.91.116.10:4087 , piece_id=W4QTE4CCAQGK6KODK2DFBY6YEGEGYOHZS7WBJMV7IN3PCHOMOG6A, seg_pos={0 0}, upload=278652]: Uploading piece...
Nov 22 12:27:49.576925240 (316h23m48.128395701s) [noise=true , address=77.90.31.10:4539 , piece_id=I5AHSBYZXFUVBLW4L4R4CFVJOFFPLQB7F5AK2GBYDXISM6N72PDQ, seg_pos={0 0}, upload=278652]: Uploading piece...
Nov 22 12:27:49.579508780 (316h23m48.130979241s) [noise=true , address=145.79.227.33:4894 , piece_id=QD6HDQDS37Q6HOIU522KZZ44LXF2AKRVKL5PIQFG437KTXYMKC6Q, seg_pos={0 0}, upload=278652]: Uploading piece...
Nov 22 12:27:49.580586642 (316h23m48.132057104s) [noise=true , address=77.90.31.13:4630 , piece_id=FNU2RL7WP5G4RR3HMDEEKYKNJBLTH6SKZ3FFPZK4B7U4A2RRWWKQ, seg_pos={0 0}, upload=278652]: Uploading piece...
Nov 22 12:27:49.585268507 (316h23m48.136738968s) [noise=true , address=77.90.31.12:4608 , piece_id=X2N6DK2U6TUZXCP356BDGEBOWY44UEOCG73CWEY57GY7OA7ZTNXA, seg_pos={0 0}, upload=278652]: Uploading piece...
Nov 22 12:27:49.588099719 (316h23m48.13957018s) [noise=true , address=45.91.116.6:4986 , piece_id=RHKQEB2Y7KCD4CIROR6DSYPTVG2SUNDKE7C5QW7DHHM24YJHMNOQ, seg_pos={0 0}, upload=278652]: Uploading piece...
Nov 22 12:27:49.589734080 (316h23m48.141204531s) [noise=true , address=145.79.227.12:4790 , piece_id=ZCJGR7J7ZDQWNMWSLFPHSGRGRQYLNMDSO52Z2TFQCQCBXIBVPZYA, seg_pos={0 0}, upload=278652]: Uploading piece...
Nov 22 12:27:49.592364720 (316h23m48.143835201s) [noise=true , address=77.90.31.17:4799 , piece_id=K2U42DVPQMOBOXQ4J2FZDWEQURTV42XSVR23AT32YCKEXAPD24RA, seg_pos={0 0}, upload=278652]: Uploading piece...
Nov 22 12:27:49.600851380 (316h23m48.152321841s) [noise=true , address=145.79.227.7:4314 , piece_id=D2CQYNQ7U5IP4QRSTASQIO5JBXYPWSHOP3HN4SVVOTN6LIQNSN3A, seg_pos={0 0}, upload=278652]: Uploading piece...
Nov 22 12:27:50.972898701 (316h23m49.524369162s) [upload=278652]: Next returned nil segment
Nov 22 12:27:50.972914963 (316h23m49.524385414s) [upload=278652]: Waiting for error group managing segments...
Nov 22 12:27:51.338580227 (316h23m49.890050688s) [noise=true , address=64.49.13.140:4506 , piece_id=B5MGJGEBUK6H2JD6LVVKYO34GYGTKQ2RRLTT53QMJHDJGLAH2PPA, seg_pos={0 0}, upload=278652]: Done uploading piece. err: <nil>
Nov 22 12:27:51.425396702 (316h23m49.976867163s) [noise=true , address=145.79.227.5:4256 , piece_id=UOFRVFML2YA53NWWQZMB3GA5PW4NFULCJJHMALH7CHHNGSG3N7VA, seg_pos={0 0}, upload=278652]: Done uploading piece. err: <nil>
Nov 22 12:27:51.444596362 (316h23m49.996066823s) [noise=true , address=145.79.227.49:4354 , piece_id=357B2N2MEKIVI7ERX5GY33FDAZOO2A3ENZV74UVPEBYSYDNI5GIQ, seg_pos={0 0}, upload=278652]: Done uploading piece. err: <nil>
Nov 22 12:27:51.448630013 (316h23m50.000100474s) [noise=true , address=145.79.227.33:4894 , piece_id=QD6HDQDS37Q6HOIU522KZZ44LXF2AKRVKL5PIQFG437KTXYMKC6Q, seg_pos={0 0}, upload=278652]: Done uploading piece. err: <nil>
Nov 22 12:27:51.466112773 (316h23m50.017583234s) [noise=true , address=145.79.227.12:4790 , piece_id=ZCJGR7J7ZDQWNMWSLFPHSGRGRQYLNMDSO52Z2TFQCQCBXIBVPZYA, seg_pos={0 0}, upload=278652]: Done uploading piece. err: <nil>
Nov 22 12:27:51.471437682 (316h23m50.022908133s) [noise=true , address=145.79.227.7:4314 , piece_id=D2CQYNQ7U5IP4QRSTASQIO5JBXYPWSHOP3HN4SVVOTN6LIQNSN3A, seg_pos={0 0}, upload=278652]: Done uploading piece. err: <nil>
Nov 22 12:28:48.617770024 (316h24m47.169240485s) [noise=true , address=77.90.31.17:4799 , piece_id=K2U42DVPQMOBOXQ4J2FZDWEQURTV42XSVR23AT32YCKEXAPD24RA, seg_pos={0 0}, upload=278652]: Done uploading piece. err: ecclient: upload failed (node:12UhAksgXZuFGAVheaYzP7D4kwVyAjUAq3BtrvdSxszjNCFCTWT, address:77.90.31.17:4799): protocol: rpc: dial tcp 77.90.31.17:4799: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:48.617979923 (316h24m47.169450384s) [noise=true , address=45.91.116.5:4306 , piece_id=F437446OQSONT5YJK35AQ26QMDMC3HP2CWUPEP46WKZGWNRL2V5A, seg_pos={0 0}, upload=278652]: Done uploading piece. err: ecclient: upload failed (node:1QuFvRW9wGKdKRXPK8UdR8uqTm53CnHaF9CsUz2bUh2Ptw9r1F, address:45.91.116.5:4306): protocol: rpc: dial tcp 45.91.116.5:4306: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:48.618033846 (316h24m47.169504308s) [noise=true , address=77.90.31.21:4885 , piece_id=5OL3NVLKPFB23XQFUDQLBX6EJHPF5VICQYSYQKCSDY5SWHILW4QA, seg_pos={0 0}, upload=278652]: Done uploading piece. err: ecclient: upload failed (node:12TfMGrvCMPvQ9ZC1pVoDqjFnUHXtzwwnZB6pCV1em4pdmbK2MP, address:77.90.31.21:4885): protocol: rpc: dial tcp 77.90.31.21:4885: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:48.618228647 (316h24m47.169699098s) [noise=true , address=45.91.116.4:4948 , piece_id=WNZONHIDT5WLFQW6ZYMZOOQ2DT4ATWKBMRGKVLI5SKY53MAMCEYQ, seg_pos={0 0}, upload=278652]: Done uploading piece. err: ecclient: upload failed (node:184g6qHdezLW3d2CKNW8GRgdzeVLQuMEhd8B8c1zEnkKNkeSTu, address:45.91.116.4:4948): protocol: rpc: dial tcp 45.91.116.4:4948: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:48.618249357 (316h24m47.169719848s) [noise=true , address=45.91.116.9:4279 , piece_id=LSPEKODU2YPLRUI6MY37VVRL6QNUJ2IXTZX6BBJHEVDMLTZGNQVQ, seg_pos={0 0}, upload=278652]: Done uploading piece. err: ecclient: upload failed (node:124aq2BHBfu1qSRXCx5XVZJU5U7gzadYHSRLYT8fcwcrtu9z5jG, address:45.91.116.9:4279): protocol: rpc: dial tcp 45.91.116.9:4279: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:48.618444829 (316h24m47.16991529s) [noise=true , address=77.90.31.10:4539 , piece_id=I5AHSBYZXFUVBLW4L4R4CFVJOFFPLQB7F5AK2GBYDXISM6N72PDQ, seg_pos={0 0}, upload=278652]: Done uploading piece. err: ecclient: upload failed (node:1DYHaVii53Q4FhtYNBMC64MP1uKHffSTMdfETK8kS94e4ZPfka, address:77.90.31.10:4539): protocol: rpc: dial tcp 77.90.31.10:4539: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:48.618487821 (316h24m47.169958272s) [noise=true , address=45.91.116.14:4211 , piece_id=5HS3X7L6ACA6KJNWBBEK36IIBOFNANCMF4QL367VJA5OHCJW4SOQ, seg_pos={0 0}, upload=278652]: Done uploading piece. err: ecclient: upload failed (node:1TkBbeKqsAZHyZdNpDgCb2uexqgV59Atgtv7EM1rd4KJEZ4BqF, address:45.91.116.14:4211): protocol: rpc: dial tcp 45.91.116.14:4211: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:48.618612950 (316h24m47.170083451s) [noise=true , address=45.91.116.6:4986 , piece_id=RHKQEB2Y7KCD4CIROR6DSYPTVG2SUNDKE7C5QW7DHHM24YJHMNOQ, seg_pos={0 0}, upload=278652]: Done uploading piece. err: ecclient: upload failed (node:12t1cuEr8MRLBbJ4GNSdYBxdhNQ6xDbTr8PHoEuVhMqh1ksgWon, address:45.91.116.6:4986): protocol: write tcp 188.166.225.170:41196->45.91.116.6:4986: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:48.618698352 (316h24m47.170168813s) [noise=true , address=77.90.31.13:4630 , piece_id=FNU2RL7WP5G4RR3HMDEEKYKNJBLTH6SKZ3FFPZK4B7U4A2RRWWKQ, seg_pos={0 0}, upload=278652]: Done uploading piece. err: ecclient: upload failed (node:1gyQJLnzWmRESyd8m6G7aZWsQwVfqRQNARxRmd7GwwfYwhuzVn, address:77.90.31.13:4630): protocol: rpc: dial tcp 77.90.31.13:4630: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:48.618875479 (316h24m47.17034594s) [noise=true , address=77.90.31.12:4608 , piece_id=X2N6DK2U6TUZXCP356BDGEBOWY44UEOCG73CWEY57GY7OA7ZTNXA, seg_pos={0 0}, upload=278652]: Done uploading piece. err: ecclient: upload failed (node:12AgtYnGt6M45rGyTZRr4qCFp1BEkgNPBxxsWqBKEn1Cx7AbYLu, address:77.90.31.12:4608): protocol: rpc: dial tcp 77.90.31.12:4608: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:48.619011127 (316h24m47.170481588s) [noise=true , address=77.90.31.19:4876 , piece_id=RDATPSCRLI5BNZVPSBOSAPUK5GFYD2MWCIPRAL6NMPA4Z6GDNZ5A, seg_pos={0 0}, upload=278652]: Done uploading piece. err: ecclient: upload failed (node:12H3UNBGeAufYSdVvEgYkv2QWeFtjKMvHvYGeqGeBvjEHqDDjBB, address:77.90.31.19:4876): protocol: rpc: dial tcp 77.90.31.19:4876: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:48.619102862 (316h24m47.170573323s) [noise=true , address=45.91.116.10:4087 , piece_id=W4QTE4CCAQGK6KODK2DFBY6YEGEGYOHZS7WBJMV7IN3PCHOMOG6A, seg_pos={0 0}, upload=278652]: Done uploading piece. err: ecclient: upload failed (node:125T5WzC4RDcp21oJ8zBMMBq5uw2X32Wf5FszZiWUdsSR95Pd2H, address:45.91.116.10:4087): protocol: write tcp 188.166.225.170:50656->45.91.116.10:4087: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:48.619135034 (316h24m47.170605495s) [seg_pos={0 0}, upload=278652]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil>
Nov 22 12:28:48.858048038 (316h24m47.409518489s) [upload=278652]: Done uploading segments...
Nov 22 12:28:48.858131627 (316h24m47.409602088s) [upload=278652]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6)
Duration: 37.913559116s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.......................................................X..X.] duration:
10.71149276s trace:
https://jaeger.dev.storj.io/trace/3e7bcddde35fe2d
FAIL (2025-11-22 12:31:43 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 10.711524791s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [.......................................................X...X] duration:
1m0.000898998s trace:
https://jaeger.dev.storj.io/trace/570ccd0ce1345814
FAIL (2025-11-22 12:31:42 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.000948082s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [....X......................................................X] duration:
30.24531242s trace:
https://jaeger.dev.storj.io/trace/5ae3d2bf80495000
FAIL (2025-11-22 12:31:36 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Nov 22 12:31:07.276308940 (316h27m5.827779401s) [upload=278701]: Starting upload
Nov 22 12:31:07.276483523 (316h27m5.827953994s) [upload=278701]: Done starting upload
Nov 22 12:31:07.276562243 (316h27m5.828032704s) [upload=278701]: Uploading segments...
Nov 22 12:31:07.276950593 (316h27m5.828421054s) [upload=278701]: Got next segment. Inline: false
Nov 22 12:31:07.277027639 (316h27m5.8284981s) [upload=278701]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:31:07.277034181 (316h27m5.828504643s) [upload=278701]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:31:07.520212027 (316h27m6.071682488s) [seg_pos={0 0}, upload=278701]: Begin upload segment...
Nov 22 12:31:07.520269256 (316h27m6.071739717s) [seg_pos={0 0}, upload=278701]: Done begin upload segment.
Nov 22 12:31:07.520308992 (316h27m6.071779453s) [noise=true , address=145.79.227.27:4689 , piece_id=76S65KEZ7I2E76RR2JXOZ3ONGXCWSJA5XB7XQVFEEKGGNC34Z4EQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.520426746 (316h27m6.071897207s) [noise=true , address=145.79.227.21:4474 , piece_id=PCRWYQVCAZCPOILAEDHJGDBFKX4GEJ6JXCF7FUS7OVFRUTDPJI4Q, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.521175282 (316h27m6.072645743s) [noise=true , address=145.79.227.17:4972 , piece_id=UZK7LCIOBGGDNJZUVDBOLBJHGOIA4SUT2O3PD67PYQD3J2DGBJEA, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.521527573 (316h27m6.072998044s) [noise=true , address=145.79.227.23:4546 , piece_id=73UPPN2SYWNWEVWHCF4BRGKNO5GCRGOV3S5JZPX4MW75EJZRQOWQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.521694671 (316h27m6.073165132s) [noise=true , address=45.91.116.10:4082 , piece_id=GBDYXAWONOHQGBGM3NCRQYDSPFOAIHRJHBB2W6ZGJLMZUJLV65XA, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.522219410 (316h27m6.073689871s) [noise=true , address=145.79.227.20:4437 , piece_id=CZ32MB4AEZZ6YKCNWKRYALVAK7MT3KNXDSZCQTPUV2Q53NX6Y3BQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.522261240 (316h27m6.073731691s) [noise=true , address=145.79.227.16:4909 , piece_id=ZTL4ER5YJWSDGO5F5KLRUL5WRTA6WKVOJOERIBTRIIVEWXUY5Z2Q, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.522719151 (316h27m6.074189612s) [noise=true , address=145.79.227.26:4654 , piece_id=EM4RRGMPPNVR7VTXP3BED43FJFIUDZYVT4S5OUK2TRXZHYNMQ64Q, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.523017269 (316h27m6.07448772s) [noise=true , address=77.90.31.13:4627 , piece_id=Q7U5BFXHYQBBJQCU4EQGAR75WKL5QMBJ3LO34UYGO7QHLMDJBKWQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.523216949 (316h27m6.07468741s) [noise=true , address=145.79.227.25:4618 , piece_id=CV6M3CTPQR2MPQZFU6W34MFQERT6MMN3RRF2N3W35FGI464OSUUA, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.523744804 (316h27m6.075215265s) [noise=true , address=145.79.227.6:4258 , piece_id=B5D4TCVJORSAWDAU4DQ5VAV4CCRF5WUI24DUVUORAEKLAWGI7ABQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.523828153 (316h27m6.075298614s) [noise=true , address=145.79.227.24:4583 , piece_id=MLW4X34EWXKPOCFVI4VIQSL6RMUOJ5HRYHQ4HY35JIFT4ZFJ6F3Q, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.524255396 (316h27m6.075725858s) [noise=true , address=45.91.116.7:4009 , piece_id=FXQGF3UQN5CPFBS2HWD7BAXR4QIOF42MMK3VJQ7AK4THYJWOFFLQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.524662051 (316h27m6.076132512s) [noise=true , address=145.79.227.15:4874 , piece_id=TYAXIF3ZJQQSFGFANYF56DG4O5WBBXK4M7OT67E7ARYNZ663HESA, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.524746221 (316h27m6.076216682s) [noise=true , address=145.79.227.28:4726 , piece_id=3LBJV2B4CVXEMHCOPRW7DR5QBKPS6DFMPTHQSVNR4XRTZZVIGBQA, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:07.525227157 (316h27m6.076697608s) [noise=true , address=77.90.31.21:4879 , piece_id=GHTH55OGGXBAZZ5FGPCDYVHDYEETM7DNVTPSTTFFFRZMKZZ3XRXQ, seg_pos={0 0}, upload=278701]: Uploading piece...
Nov 22 12:31:09.134610631 (316h27m7.686081093s) [upload=278701]: Next returned nil segment
Nov 22 12:31:09.134628024 (316h27m7.686098495s) [upload=278701]: Waiting for error group managing segments...
Nov 22 12:31:09.502125778 (316h27m8.053596239s) [noise=true , address=145.79.227.16:4909 , piece_id=ZTL4ER5YJWSDGO5F5KLRUL5WRTA6WKVOJOERIBTRIIVEWXUY5Z2Q, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.503083742 (316h27m8.054554193s) [noise=true , address=145.79.227.20:4437 , piece_id=CZ32MB4AEZZ6YKCNWKRYALVAK7MT3KNXDSZCQTPUV2Q53NX6Y3BQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.517732716 (316h27m8.069203177s) [noise=true , address=145.79.227.25:4618 , piece_id=CV6M3CTPQR2MPQZFU6W34MFQERT6MMN3RRF2N3W35FGI464OSUUA, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.518029582 (316h27m8.069500043s) [noise=true , address=145.79.227.28:4726 , piece_id=3LBJV2B4CVXEMHCOPRW7DR5QBKPS6DFMPTHQSVNR4XRTZZVIGBQA, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.519935051 (316h27m8.071405512s) [noise=true , address=145.79.227.26:4654 , piece_id=EM4RRGMPPNVR7VTXP3BED43FJFIUDZYVT4S5OUK2TRXZHYNMQ64Q, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.520607661 (316h27m8.072078122s) [noise=true , address=145.79.227.27:4689 , piece_id=76S65KEZ7I2E76RR2JXOZ3ONGXCWSJA5XB7XQVFEEKGGNC34Z4EQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.521187736 (316h27m8.072658197s) [noise=true , address=145.79.227.23:4546 , piece_id=73UPPN2SYWNWEVWHCF4BRGKNO5GCRGOV3S5JZPX4MW75EJZRQOWQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.532974760 (316h27m8.084445221s) [noise=true , address=145.79.227.24:4583 , piece_id=MLW4X34EWXKPOCFVI4VIQSL6RMUOJ5HRYHQ4HY35JIFT4ZFJ6F3Q, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.533688800 (316h27m8.085159271s) [noise=true , address=145.79.227.6:4258 , piece_id=B5D4TCVJORSAWDAU4DQ5VAV4CCRF5WUI24DUVUORAEKLAWGI7ABQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.534460109 (316h27m8.08593057s) [noise=true , address=145.79.227.15:4874 , piece_id=TYAXIF3ZJQQSFGFANYF56DG4O5WBBXK4M7OT67E7ARYNZ663HESA, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.542955375 (316h27m8.094425826s) [noise=true , address=145.79.227.17:4972 , piece_id=UZK7LCIOBGGDNJZUVDBOLBJHGOIA4SUT2O3PD67PYQD3J2DGBJEA, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:09.563098982 (316h27m8.114569443s) [noise=true , address=145.79.227.21:4474 , piece_id=PCRWYQVCAZCPOILAEDHJGDBFKX4GEJ6JXCF7FUS7OVFRUTDPJI4Q, seg_pos={0 0}, upload=278701]: Done uploading piece. err: <nil>
Nov 22 12:31:36.585475346 (316h27m35.136945807s) [noise=true , address=45.91.116.10:4082 , piece_id=GBDYXAWONOHQGBGM3NCRQYDSPFOAIHRJHBB2W6ZGJLMZUJLV65XA, seg_pos={0 0}, upload=278701]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: read tcp 188.166.225.170:54326->45.91.116.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:36.585499923 (316h27m35.136970384s) [noise=true , address=77.90.31.21:4879 , piece_id=GHTH55OGGXBAZZ5FGPCDYVHDYEETM7DNVTPSTTFFFRZMKZZ3XRXQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: rpc: dial tcp 77.90.31.21:4879: i/o timeout; context deadline exceeded
Nov 22 12:31:36.585749648 (316h27m35.137220119s) [noise=true , address=77.90.31.13:4627 , piece_id=Q7U5BFXHYQBBJQCU4EQGAR75WKL5QMBJ3LO34UYGO7QHLMDJBKWQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:36.585848086 (316h27m35.137318547s) [noise=true , address=45.91.116.7:4009 , piece_id=FXQGF3UQN5CPFBS2HWD7BAXR4QIOF42MMK3VJQ7AK4THYJWOFFLQ, seg_pos={0 0}, upload=278701]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:36.585900936 (316h27m35.137371397s) [seg_pos={0 0}, upload=278701]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Nov 22 12:31:36.828881025 (316h27m35.380351486s) [upload=278701]: Done uploading segments...
Nov 22 12:31:36.828915801 (316h27m35.380386262s) [upload=278701]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 30.245338571s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=6) [........X...............................................X..X] duration:
5.001417603s trace:
https://jaeger.dev.storj.io/trace/47ae36a3e1dc74d6
FAIL (2025-11-22 12:31:21 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.00144697s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [........................................................X..X] duration:
30.001370573s trace:
https://jaeger.dev.storj.io/trace/3e7bcddde35fe2d
FAIL (2025-11-22 12:31:18 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.001429996s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [........................................................X...] duration:
5.733282792s trace:
https://jaeger.dev.storj.io/trace/78d837dffbcbc1eb
FAIL (2025-11-22 12:30:28 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 6)
======LOGS======
Nov 22 12:27:49.568493454 (316h23m48.119963905s) [noise=true , address=77.90.31.19:4876 , piece_id=RDATPSCRLI5BNZVPSBOSAPUK5GFYD2MWCIPRAL6NMPA4Z6GDNZ5A, seg_pos={0 0}, upload=278652]: Uploading piece...
Nov 22 12:27:49.570659288 (316h23m48.122129749s) [noise=true , address=45.91.116.5:4306 , piece_id=F437446OQSONT5YJK35AQ26QMDMC3HP2CWUPEP46WKZGWNRL2V5A, seg_pos={0 0}, upload=278652]: Uploading piece...
Nov 22 12:27:49.572948898 (316h23m48.124419359s) [noise=true , address=64.49.13.140:4506 , piece_id=B5MGJGEBUK6H2JD6LVVKYO34GYGTKQ2RRLTT53QMJHDJGLAH2PPA, seg_pos={0 0}, upload=278652]: Uploading piece...
Nov 22 12:27:49.573111158 (316h23m48.124581609s) [noise=true , address=45.91.116.4:4948 , piece_id=WNZONHIDT5WLFQW6ZYMZOOQ2DT4ATWKBMRGKVLI5SKY53MAMCEYQ, seg_pos={0 0}, upload=278652]: Uploading piece...
Nov 22 12:27:49.574907569 (316h23m48.12637803s) [noise=true , address=45.91.116.10:4087 , piece_id=W4QTE4CCAQGK6KODK2DFBY6YEGEGYOHZS7WBJMV7IN3PCHOMOG6A, seg_pos={0 0}, upload=278652]: Uploading piece...
Nov 22 12:27:49.576925240 (316h23m48.128395701s) [noise=true , address=77.90.31.10:4539 , piece_id=I5AHSBYZXFUVBLW4L4R4CFVJOFFPLQB7F5AK2GBYDXISM6N72PDQ, seg_pos={0 0}, upload=278652]: Uploading piece...
Nov 22 12:27:49.579508780 (316h23m48.130979241s) [noise=true , address=145.79.227.33:4894 , piece_id=QD6HDQDS37Q6HOIU522KZZ44LXF2AKRVKL5PIQFG437KTXYMKC6Q, seg_pos={0 0}, upload=278652]: Uploading piece...
Nov 22 12:27:49.580586642 (316h23m48.132057104s) [noise=true , address=77.90.31.13:4630 , piece_id=FNU2RL7WP5G4RR3HMDEEKYKNJBLTH6SKZ3FFPZK4B7U4A2RRWWKQ, seg_pos={0 0}, upload=278652]: Uploading piece...
Nov 22 12:27:49.585268507 (316h23m48.136738968s) [noise=true , address=77.90.31.12:4608 , piece_id=X2N6DK2U6TUZXCP356BDGEBOWY44UEOCG73CWEY57GY7OA7ZTNXA, seg_pos={0 0}, upload=278652]: Uploading piece...
Nov 22 12:27:49.588099719 (316h23m48.13957018s) [noise=true , address=45.91.116.6:4986 , piece_id=RHKQEB2Y7KCD4CIROR6DSYPTVG2SUNDKE7C5QW7DHHM24YJHMNOQ, seg_pos={0 0}, upload=278652]: Uploading piece...
Nov 22 12:27:49.589734080 (316h23m48.141204531s) [noise=true , address=145.79.227.12:4790 , piece_id=ZCJGR7J7ZDQWNMWSLFPHSGRGRQYLNMDSO52Z2TFQCQCBXIBVPZYA, seg_pos={0 0}, upload=278652]: Uploading piece...
Nov 22 12:27:49.592364720 (316h23m48.143835201s) [noise=true , address=77.90.31.17:4799 , piece_id=K2U42DVPQMOBOXQ4J2FZDWEQURTV42XSVR23AT32YCKEXAPD24RA, seg_pos={0 0}, upload=278652]: Uploading piece...
Nov 22 12:27:49.600851380 (316h23m48.152321841s) [noise=true , address=145.79.227.7:4314 , piece_id=D2CQYNQ7U5IP4QRSTASQIO5JBXYPWSHOP3HN4SVVOTN6LIQNSN3A, seg_pos={0 0}, upload=278652]: Uploading piece...
Nov 22 12:27:50.972898701 (316h23m49.524369162s) [upload=278652]: Next returned nil segment
Nov 22 12:27:50.972914963 (316h23m49.524385414s) [upload=278652]: Waiting for error group managing segments...
Nov 22 12:27:51.338580227 (316h23m49.890050688s) [noise=true , address=64.49.13.140:4506 , piece_id=B5MGJGEBUK6H2JD6LVVKYO34GYGTKQ2RRLTT53QMJHDJGLAH2PPA, seg_pos={0 0}, upload=278652]: Done uploading piece. err: <nil>
Nov 22 12:27:51.425396702 (316h23m49.976867163s) [noise=true , address=145.79.227.5:4256 , piece_id=UOFRVFML2YA53NWWQZMB3GA5PW4NFULCJJHMALH7CHHNGSG3N7VA, seg_pos={0 0}, upload=278652]: Done uploading piece. err: <nil>
Nov 22 12:27:51.444596362 (316h23m49.996066823s) [noise=true , address=145.79.227.49:4354 , piece_id=357B2N2MEKIVI7ERX5GY33FDAZOO2A3ENZV74UVPEBYSYDNI5GIQ, seg_pos={0 0}, upload=278652]: Done uploading piece. err: <nil>
Nov 22 12:27:51.448630013 (316h23m50.000100474s) [noise=true , address=145.79.227.33:4894 , piece_id=QD6HDQDS37Q6HOIU522KZZ44LXF2AKRVKL5PIQFG437KTXYMKC6Q, seg_pos={0 0}, upload=278652]: Done uploading piece. err: <nil>
Nov 22 12:27:51.466112773 (316h23m50.017583234s) [noise=true , address=145.79.227.12:4790 , piece_id=ZCJGR7J7ZDQWNMWSLFPHSGRGRQYLNMDSO52Z2TFQCQCBXIBVPZYA, seg_pos={0 0}, upload=278652]: Done uploading piece. err: <nil>
Nov 22 12:27:51.471437682 (316h23m50.022908133s) [noise=true , address=145.79.227.7:4314 , piece_id=D2CQYNQ7U5IP4QRSTASQIO5JBXYPWSHOP3HN4SVVOTN6LIQNSN3A, seg_pos={0 0}, upload=278652]: Done uploading piece. err: <nil>
Nov 22 12:28:48.617770024 (316h24m47.169240485s) [noise=true , address=77.90.31.17:4799 , piece_id=K2U42DVPQMOBOXQ4J2FZDWEQURTV42XSVR23AT32YCKEXAPD24RA, seg_pos={0 0}, upload=278652]: Done uploading piece. err: ecclient: upload failed (node:12UhAksgXZuFGAVheaYzP7D4kwVyAjUAq3BtrvdSxszjNCFCTWT, address:77.90.31.17:4799): protocol: rpc: dial tcp 77.90.31.17:4799: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:48.617979923 (316h24m47.169450384s) [noise=true , address=45.91.116.5:4306 , piece_id=F437446OQSONT5YJK35AQ26QMDMC3HP2CWUPEP46WKZGWNRL2V5A, seg_pos={0 0}, upload=278652]: Done uploading piece. err: ecclient: upload failed (node:1QuFvRW9wGKdKRXPK8UdR8uqTm53CnHaF9CsUz2bUh2Ptw9r1F, address:45.91.116.5:4306): protocol: rpc: dial tcp 45.91.116.5:4306: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:48.618033846 (316h24m47.169504308s) [noise=true , address=77.90.31.21:4885 , piece_id=5OL3NVLKPFB23XQFUDQLBX6EJHPF5VICQYSYQKCSDY5SWHILW4QA, seg_pos={0 0}, upload=278652]: Done uploading piece. err: ecclient: upload failed (node:12TfMGrvCMPvQ9ZC1pVoDqjFnUHXtzwwnZB6pCV1em4pdmbK2MP, address:77.90.31.21:4885): protocol: rpc: dial tcp 77.90.31.21:4885: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:48.618228647 (316h24m47.169699098s) [noise=true , address=45.91.116.4:4948 , piece_id=WNZONHIDT5WLFQW6ZYMZOOQ2DT4ATWKBMRGKVLI5SKY53MAMCEYQ, seg_pos={0 0}, upload=278652]: Done uploading piece. err: ecclient: upload failed (node:184g6qHdezLW3d2CKNW8GRgdzeVLQuMEhd8B8c1zEnkKNkeSTu, address:45.91.116.4:4948): protocol: rpc: dial tcp 45.91.116.4:4948: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:48.618249357 (316h24m47.169719848s) [noise=true , address=45.91.116.9:4279 , piece_id=LSPEKODU2YPLRUI6MY37VVRL6QNUJ2IXTZX6BBJHEVDMLTZGNQVQ, seg_pos={0 0}, upload=278652]: Done uploading piece. err: ecclient: upload failed (node:124aq2BHBfu1qSRXCx5XVZJU5U7gzadYHSRLYT8fcwcrtu9z5jG, address:45.91.116.9:4279): protocol: rpc: dial tcp 45.91.116.9:4279: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:48.618444829 (316h24m47.16991529s) [noise=true , address=77.90.31.10:4539 , piece_id=I5AHSBYZXFUVBLW4L4R4CFVJOFFPLQB7F5AK2GBYDXISM6N72PDQ, seg_pos={0 0}, upload=278652]: Done uploading piece. err: ecclient: upload failed (node:1DYHaVii53Q4FhtYNBMC64MP1uKHffSTMdfETK8kS94e4ZPfka, address:77.90.31.10:4539): protocol: rpc: dial tcp 77.90.31.10:4539: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:48.618487821 (316h24m47.169958272s) [noise=true , address=45.91.116.14:4211 , piece_id=5HS3X7L6ACA6KJNWBBEK36IIBOFNANCMF4QL367VJA5OHCJW4SOQ, seg_pos={0 0}, upload=278652]: Done uploading piece. err: ecclient: upload failed (node:1TkBbeKqsAZHyZdNpDgCb2uexqgV59Atgtv7EM1rd4KJEZ4BqF, address:45.91.116.14:4211): protocol: rpc: dial tcp 45.91.116.14:4211: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:48.618612950 (316h24m47.170083451s) [noise=true , address=45.91.116.6:4986 , piece_id=RHKQEB2Y7KCD4CIROR6DSYPTVG2SUNDKE7C5QW7DHHM24YJHMNOQ, seg_pos={0 0}, upload=278652]: Done uploading piece. err: ecclient: upload failed (node:12t1cuEr8MRLBbJ4GNSdYBxdhNQ6xDbTr8PHoEuVhMqh1ksgWon, address:45.91.116.6:4986): protocol: write tcp 188.166.225.170:41196->45.91.116.6:4986: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:48.618698352 (316h24m47.170168813s) [noise=true , address=77.90.31.13:4630 , piece_id=FNU2RL7WP5G4RR3HMDEEKYKNJBLTH6SKZ3FFPZK4B7U4A2RRWWKQ, seg_pos={0 0}, upload=278652]: Done uploading piece. err: ecclient: upload failed (node:1gyQJLnzWmRESyd8m6G7aZWsQwVfqRQNARxRmd7GwwfYwhuzVn, address:77.90.31.13:4630): protocol: rpc: dial tcp 77.90.31.13:4630: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:48.618875479 (316h24m47.17034594s) [noise=true , address=77.90.31.12:4608 , piece_id=X2N6DK2U6TUZXCP356BDGEBOWY44UEOCG73CWEY57GY7OA7ZTNXA, seg_pos={0 0}, upload=278652]: Done uploading piece. err: ecclient: upload failed (node:12AgtYnGt6M45rGyTZRr4qCFp1BEkgNPBxxsWqBKEn1Cx7AbYLu, address:77.90.31.12:4608): protocol: rpc: dial tcp 77.90.31.12:4608: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:48.619011127 (316h24m47.170481588s) [noise=true , address=77.90.31.19:4876 , piece_id=RDATPSCRLI5BNZVPSBOSAPUK5GFYD2MWCIPRAL6NMPA4Z6GDNZ5A, seg_pos={0 0}, upload=278652]: Done uploading piece. err: ecclient: upload failed (node:12H3UNBGeAufYSdVvEgYkv2QWeFtjKMvHvYGeqGeBvjEHqDDjBB, address:77.90.31.19:4876): protocol: rpc: dial tcp 77.90.31.19:4876: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:48.619102862 (316h24m47.170573323s) [noise=true , address=45.91.116.10:4087 , piece_id=W4QTE4CCAQGK6KODK2DFBY6YEGEGYOHZS7WBJMV7IN3PCHOMOG6A, seg_pos={0 0}, upload=278652]: Done uploading piece. err: ecclient: upload failed (node:125T5WzC4RDcp21oJ8zBMMBq5uw2X32Wf5FszZiWUdsSR95Pd2H, address:45.91.116.10:4087): protocol: write tcp 188.166.225.170:50656->45.91.116.10:4087: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:48.619135034 (316h24m47.170605495s) [seg_pos={0 0}, upload=278652]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil>
Nov 22 12:28:48.858048038 (316h24m47.409518489s) [upload=278652]: Done uploading segments...
Nov 22 12:28:48.858131627 (316h24m47.409602088s) [upload=278652]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6)
Duration: 5.733302489s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [........................................................X...] duration:
5.695121946s trace:
https://jaeger.dev.storj.io/trace/666b0e258bc1797c
FAIL (2025-11-22 12:30:26 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.695151072s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [.........X...............................................X..] duration:
3.490957441s trace:
https://jaeger.dev.storj.io/trace/69bcc2ce5cc82585
FAIL (2025-11-22 12:30:16 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.490988429s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................................................X..] duration:
10.411742268s trace:
https://jaeger.dev.storj.io/trace/2b03c00e8c9bd6a8
FAIL (2025-11-22 12:30:07 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 10.411777453s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....X......................................................] duration:
5.713317344s trace:
https://jaeger.dev.storj.io/trace/76f1ecbff69b5d4e
FAIL (2025-11-22 12:30:00 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, Piece ID: I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ) context deadline exceeded; piecestore: (Node ID: 12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, Piece ID: YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A) context deadline exceeded; piecestore: (Node ID: 12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, Piece ID: P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ) context deadline exceeded; piecestore: (Node ID: 1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, Piece ID: ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA) context deadline exceeded; piecestore: (Node ID: 1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, Piece ID: G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ) context deadline exceeded; piecestore: (Node ID: 1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, Piece ID: NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA) context deadline exceeded
======LOGS======
nts...
Nov 22 11:53:53.345386809 (315h49m51.89685727s) [upload=278145]: Got next segment. Inline: false
Nov 22 11:53:53.345428137 (315h49m51.896898598s) [upload=278145]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 11:53:53.345433948 (315h49m51.896904419s) [upload=278145]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 11:53:54.603541081 (315h49m53.155011552s) [seg_pos={0 0}, upload=278145]: Begin upload segment...
Nov 22 11:53:54.603595445 (315h49m53.155065916s) [seg_pos={0 0}, upload=278145]: Done begin upload segment.
Nov 22 11:53:54.603668043 (315h49m53.155138494s) [noise=true , address=145.79.227.14:4837 , piece_id=TCYBUWS4YPUVLJ6QAPX2QIRT7O2HSBVEKVPPKHCTK336NNNPO7PQ, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.603684554 (315h49m53.155155005s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.604505828 (315h49m53.155976289s) [noise=true , address=45.91.116.11:4119 , piece_id=FGNP4Z5IL65VGJ7NMOJK5JW4Y644RFZG6MHWUOCLFKBJ26KF5W5A, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.604973038 (315h49m53.156443499s) [noise=true , address=145.79.227.28:4726 , piece_id=YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.605425900 (315h49m53.156896361s) [noise=true , address=145.79.227.4:4187 , piece_id=P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.606323339 (315h49m53.15779381s) [noise=true , address=77.90.31.15:4698 , piece_id=3Z7MYVQ62TX4BKJASUPTJTWDZH57PEVBSPT6OV2SMMZMTG2KCYZQ, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.606878326 (315h49m53.158348777s) [noise=true , address=145.79.227.31:4799 , piece_id=RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.607588267 (315h49m53.159058728s) [noise=true , address=145.79.227.25:4619 , piece_id=ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.610211763 (315h49m53.161682215s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.610336751 (315h49m53.161807212s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.612196994 (315h49m53.163667455s) [noise=true , address=145.79.227.9:4366 , piece_id=G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.612485593 (315h49m53.163956064s) [noise=true , address=145.79.227.21:4474 , piece_id=HYSRVRV3ALPITM7ZJ2GHPIGLTZ54YH4KYUZ36PTWAU6V2XBLURKQ, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.614299567 (315h49m53.165770028s) [noise=true , address=145.79.227.20:4438 , piece_id=NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.614454202 (315h49m53.165924673s) [noise=true , address=145.79.227.17:4969 , piece_id=I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.616397682 (315h49m53.167868143s) [noise=true , address=77.90.31.10:4517 , piece_id=KLXWC5RY274YKMLT2U5SBPCNQKIRXDOTPQ7MD76HXA3F2VCYRIEA, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.621720077 (315h49m53.173190538s) [noise=true , address=145.79.227.12:4767 , piece_id=FSJE5Q3Z7KHDGYUMIWRXJKTDOT5QFHZU7WQ4SHC3VQGGWJRNVEOQ, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:56.445932556 (315h49m54.997403007s) [upload=278145]: Next returned nil segment
Nov 22 11:53:56.445949048 (315h49m54.997419509s) [upload=278145]: Waiting for error group managing segments...
Nov 22 11:53:56.734588580 (315h49m55.286059041s) [noise=true , address=77.90.31.10:4517 , piece_id=KLXWC5RY274YKMLT2U5SBPCNQKIRXDOTPQ7MD76HXA3F2VCYRIEA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:56.796169791 (315h49m55.347640252s) [noise=true , address=145.79.227.20:4438 , piece_id=NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:56.800552848 (315h49m55.352023309s) [noise=true , address=45.91.116.11:4119 , piece_id=FGNP4Z5IL65VGJ7NMOJK5JW4Y644RFZG6MHWUOCLFKBJ26KF5W5A, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:56.802134119 (315h49m55.35360458s) [noise=true , address=145.79.227.4:4187 , piece_id=P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:56.811866140 (315h49m55.363336591s) [noise=true , address=145.79.227.17:4969 , piece_id=I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:56.820962632 (315h49m55.372433093s) [noise=true , address=77.90.31.15:4698 , piece_id=3Z7MYVQ62TX4BKJASUPTJTWDZH57PEVBSPT6OV2SMMZMTG2KCYZQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:56.853755886 (315h49m55.405226347s) [noise=true , address=145.79.227.21:4474 , piece_id=HYSRVRV3ALPITM7ZJ2GHPIGLTZ54YH4KYUZ36PTWAU6V2XBLURKQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:56.857215724 (315h49m55.408686185s) [noise=true , address=145.79.227.14:4837 , piece_id=TCYBUWS4YPUVLJ6QAPX2QIRT7O2HSBVEKVPPKHCTK336NNNPO7PQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:57.067648917 (315h49m55.619119378s) [noise=true , address=145.79.227.25:4619 , piece_id=ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:57.093474313 (315h49m55.644944774s) [noise=true , address=145.79.227.9:4366 , piece_id=G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:57.104509907 (315h49m55.655980378s) [noise=true , address=145.79.227.12:4767 , piece_id=FSJE5Q3Z7KHDGYUMIWRXJKTDOT5QFHZU7WQ4SHC3VQGGWJRNVEOQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:57.505088550 (315h49m56.056559011s) [noise=true , address=145.79.227.31:4799 , piece_id=RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:57.609615419 (315h49m56.16108589s) [noise=true , address=145.79.227.28:4726 , piece_id=YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:57.609676946 (315h49m56.161147437s) [seg_pos={0 0}, upload=278145]: Segment reached optimal threshold of 13 pieces.
Nov 22 11:53:57.609972077 (315h49m56.161442539s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW): protocol: write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection
Nov 22 11:53:57.609982889 (315h49m56.16145336s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold
Nov 22 11:53:57.610025941 (315h49m56.161496432s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk): protocol: expected piece hash; context canceled; EOF
Nov 22 11:53:57.610037101 (315h49m56.161507553s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold
Nov 22 11:53:57.610213978 (315h49m56.161684439s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19): protocol: write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection
Nov 22 11:53:57.610224649 (315h49m56.16169511s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold
Nov 22 11:53:57.610261058 (315h49m56.16173152s) [seg_pos={0 0}, upload=278145]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Nov 22 11:53:57.610421113 (315h49m56.161891584s) [upload=278145]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Nov 22 11:53:57.610430852 (315h49m56.161901313s) [upload=278145]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Nov 22 11:54:00.212174190 (315h49m58.763644651s) [upload=278145]: Done uploading segments...
Nov 22 11:54:00.212200729 (315h49m58.763671191s) [upload=278145]: Upload finished. err: <nil>
Duration: 5.713351208s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................................................X..] duration:
5.627803801s trace:
https://jaeger.dev.storj.io/trace/78d837dffbcbc1eb
FAIL (2025-11-22 12:29:49 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 6)
======LOGS======
Nov 22 12:27:49.568493454 (316h23m48.119963905s) [noise=true , address=77.90.31.19:4876 , piece_id=RDATPSCRLI5BNZVPSBOSAPUK5GFYD2MWCIPRAL6NMPA4Z6GDNZ5A, seg_pos={0 0}, upload=278652]: Uploading piece...
Nov 22 12:27:49.570659288 (316h23m48.122129749s) [noise=true , address=45.91.116.5:4306 , piece_id=F437446OQSONT5YJK35AQ26QMDMC3HP2CWUPEP46WKZGWNRL2V5A, seg_pos={0 0}, upload=278652]: Uploading piece...
Nov 22 12:27:49.572948898 (316h23m48.124419359s) [noise=true , address=64.49.13.140:4506 , piece_id=B5MGJGEBUK6H2JD6LVVKYO34GYGTKQ2RRLTT53QMJHDJGLAH2PPA, seg_pos={0 0}, upload=278652]: Uploading piece...
Nov 22 12:27:49.573111158 (316h23m48.124581609s) [noise=true , address=45.91.116.4:4948 , piece_id=WNZONHIDT5WLFQW6ZYMZOOQ2DT4ATWKBMRGKVLI5SKY53MAMCEYQ, seg_pos={0 0}, upload=278652]: Uploading piece...
Nov 22 12:27:49.574907569 (316h23m48.12637803s) [noise=true , address=45.91.116.10:4087 , piece_id=W4QTE4CCAQGK6KODK2DFBY6YEGEGYOHZS7WBJMV7IN3PCHOMOG6A, seg_pos={0 0}, upload=278652]: Uploading piece...
Nov 22 12:27:49.576925240 (316h23m48.128395701s) [noise=true , address=77.90.31.10:4539 , piece_id=I5AHSBYZXFUVBLW4L4R4CFVJOFFPLQB7F5AK2GBYDXISM6N72PDQ, seg_pos={0 0}, upload=278652]: Uploading piece...
Nov 22 12:27:49.579508780 (316h23m48.130979241s) [noise=true , address=145.79.227.33:4894 , piece_id=QD6HDQDS37Q6HOIU522KZZ44LXF2AKRVKL5PIQFG437KTXYMKC6Q, seg_pos={0 0}, upload=278652]: Uploading piece...
Nov 22 12:27:49.580586642 (316h23m48.132057104s) [noise=true , address=77.90.31.13:4630 , piece_id=FNU2RL7WP5G4RR3HMDEEKYKNJBLTH6SKZ3FFPZK4B7U4A2RRWWKQ, seg_pos={0 0}, upload=278652]: Uploading piece...
Nov 22 12:27:49.585268507 (316h23m48.136738968s) [noise=true , address=77.90.31.12:4608 , piece_id=X2N6DK2U6TUZXCP356BDGEBOWY44UEOCG73CWEY57GY7OA7ZTNXA, seg_pos={0 0}, upload=278652]: Uploading piece...
Nov 22 12:27:49.588099719 (316h23m48.13957018s) [noise=true , address=45.91.116.6:4986 , piece_id=RHKQEB2Y7KCD4CIROR6DSYPTVG2SUNDKE7C5QW7DHHM24YJHMNOQ, seg_pos={0 0}, upload=278652]: Uploading piece...
Nov 22 12:27:49.589734080 (316h23m48.141204531s) [noise=true , address=145.79.227.12:4790 , piece_id=ZCJGR7J7ZDQWNMWSLFPHSGRGRQYLNMDSO52Z2TFQCQCBXIBVPZYA, seg_pos={0 0}, upload=278652]: Uploading piece...
Nov 22 12:27:49.592364720 (316h23m48.143835201s) [noise=true , address=77.90.31.17:4799 , piece_id=K2U42DVPQMOBOXQ4J2FZDWEQURTV42XSVR23AT32YCKEXAPD24RA, seg_pos={0 0}, upload=278652]: Uploading piece...
Nov 22 12:27:49.600851380 (316h23m48.152321841s) [noise=true , address=145.79.227.7:4314 , piece_id=D2CQYNQ7U5IP4QRSTASQIO5JBXYPWSHOP3HN4SVVOTN6LIQNSN3A, seg_pos={0 0}, upload=278652]: Uploading piece...
Nov 22 12:27:50.972898701 (316h23m49.524369162s) [upload=278652]: Next returned nil segment
Nov 22 12:27:50.972914963 (316h23m49.524385414s) [upload=278652]: Waiting for error group managing segments...
Nov 22 12:27:51.338580227 (316h23m49.890050688s) [noise=true , address=64.49.13.140:4506 , piece_id=B5MGJGEBUK6H2JD6LVVKYO34GYGTKQ2RRLTT53QMJHDJGLAH2PPA, seg_pos={0 0}, upload=278652]: Done uploading piece. err: <nil>
Nov 22 12:27:51.425396702 (316h23m49.976867163s) [noise=true , address=145.79.227.5:4256 , piece_id=UOFRVFML2YA53NWWQZMB3GA5PW4NFULCJJHMALH7CHHNGSG3N7VA, seg_pos={0 0}, upload=278652]: Done uploading piece. err: <nil>
Nov 22 12:27:51.444596362 (316h23m49.996066823s) [noise=true , address=145.79.227.49:4354 , piece_id=357B2N2MEKIVI7ERX5GY33FDAZOO2A3ENZV74UVPEBYSYDNI5GIQ, seg_pos={0 0}, upload=278652]: Done uploading piece. err: <nil>
Nov 22 12:27:51.448630013 (316h23m50.000100474s) [noise=true , address=145.79.227.33:4894 , piece_id=QD6HDQDS37Q6HOIU522KZZ44LXF2AKRVKL5PIQFG437KTXYMKC6Q, seg_pos={0 0}, upload=278652]: Done uploading piece. err: <nil>
Nov 22 12:27:51.466112773 (316h23m50.017583234s) [noise=true , address=145.79.227.12:4790 , piece_id=ZCJGR7J7ZDQWNMWSLFPHSGRGRQYLNMDSO52Z2TFQCQCBXIBVPZYA, seg_pos={0 0}, upload=278652]: Done uploading piece. err: <nil>
Nov 22 12:27:51.471437682 (316h23m50.022908133s) [noise=true , address=145.79.227.7:4314 , piece_id=D2CQYNQ7U5IP4QRSTASQIO5JBXYPWSHOP3HN4SVVOTN6LIQNSN3A, seg_pos={0 0}, upload=278652]: Done uploading piece. err: <nil>
Nov 22 12:28:48.617770024 (316h24m47.169240485s) [noise=true , address=77.90.31.17:4799 , piece_id=K2U42DVPQMOBOXQ4J2FZDWEQURTV42XSVR23AT32YCKEXAPD24RA, seg_pos={0 0}, upload=278652]: Done uploading piece. err: ecclient: upload failed (node:12UhAksgXZuFGAVheaYzP7D4kwVyAjUAq3BtrvdSxszjNCFCTWT, address:77.90.31.17:4799): protocol: rpc: dial tcp 77.90.31.17:4799: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:48.617979923 (316h24m47.169450384s) [noise=true , address=45.91.116.5:4306 , piece_id=F437446OQSONT5YJK35AQ26QMDMC3HP2CWUPEP46WKZGWNRL2V5A, seg_pos={0 0}, upload=278652]: Done uploading piece. err: ecclient: upload failed (node:1QuFvRW9wGKdKRXPK8UdR8uqTm53CnHaF9CsUz2bUh2Ptw9r1F, address:45.91.116.5:4306): protocol: rpc: dial tcp 45.91.116.5:4306: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:48.618033846 (316h24m47.169504308s) [noise=true , address=77.90.31.21:4885 , piece_id=5OL3NVLKPFB23XQFUDQLBX6EJHPF5VICQYSYQKCSDY5SWHILW4QA, seg_pos={0 0}, upload=278652]: Done uploading piece. err: ecclient: upload failed (node:12TfMGrvCMPvQ9ZC1pVoDqjFnUHXtzwwnZB6pCV1em4pdmbK2MP, address:77.90.31.21:4885): protocol: rpc: dial tcp 77.90.31.21:4885: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:48.618228647 (316h24m47.169699098s) [noise=true , address=45.91.116.4:4948 , piece_id=WNZONHIDT5WLFQW6ZYMZOOQ2DT4ATWKBMRGKVLI5SKY53MAMCEYQ, seg_pos={0 0}, upload=278652]: Done uploading piece. err: ecclient: upload failed (node:184g6qHdezLW3d2CKNW8GRgdzeVLQuMEhd8B8c1zEnkKNkeSTu, address:45.91.116.4:4948): protocol: rpc: dial tcp 45.91.116.4:4948: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:48.618249357 (316h24m47.169719848s) [noise=true , address=45.91.116.9:4279 , piece_id=LSPEKODU2YPLRUI6MY37VVRL6QNUJ2IXTZX6BBJHEVDMLTZGNQVQ, seg_pos={0 0}, upload=278652]: Done uploading piece. err: ecclient: upload failed (node:124aq2BHBfu1qSRXCx5XVZJU5U7gzadYHSRLYT8fcwcrtu9z5jG, address:45.91.116.9:4279): protocol: rpc: dial tcp 45.91.116.9:4279: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:48.618444829 (316h24m47.16991529s) [noise=true , address=77.90.31.10:4539 , piece_id=I5AHSBYZXFUVBLW4L4R4CFVJOFFPLQB7F5AK2GBYDXISM6N72PDQ, seg_pos={0 0}, upload=278652]: Done uploading piece. err: ecclient: upload failed (node:1DYHaVii53Q4FhtYNBMC64MP1uKHffSTMdfETK8kS94e4ZPfka, address:77.90.31.10:4539): protocol: rpc: dial tcp 77.90.31.10:4539: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:48.618487821 (316h24m47.169958272s) [noise=true , address=45.91.116.14:4211 , piece_id=5HS3X7L6ACA6KJNWBBEK36IIBOFNANCMF4QL367VJA5OHCJW4SOQ, seg_pos={0 0}, upload=278652]: Done uploading piece. err: ecclient: upload failed (node:1TkBbeKqsAZHyZdNpDgCb2uexqgV59Atgtv7EM1rd4KJEZ4BqF, address:45.91.116.14:4211): protocol: rpc: dial tcp 45.91.116.14:4211: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:48.618612950 (316h24m47.170083451s) [noise=true , address=45.91.116.6:4986 , piece_id=RHKQEB2Y7KCD4CIROR6DSYPTVG2SUNDKE7C5QW7DHHM24YJHMNOQ, seg_pos={0 0}, upload=278652]: Done uploading piece. err: ecclient: upload failed (node:12t1cuEr8MRLBbJ4GNSdYBxdhNQ6xDbTr8PHoEuVhMqh1ksgWon, address:45.91.116.6:4986): protocol: write tcp 188.166.225.170:41196->45.91.116.6:4986: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:48.618698352 (316h24m47.170168813s) [noise=true , address=77.90.31.13:4630 , piece_id=FNU2RL7WP5G4RR3HMDEEKYKNJBLTH6SKZ3FFPZK4B7U4A2RRWWKQ, seg_pos={0 0}, upload=278652]: Done uploading piece. err: ecclient: upload failed (node:1gyQJLnzWmRESyd8m6G7aZWsQwVfqRQNARxRmd7GwwfYwhuzVn, address:77.90.31.13:4630): protocol: rpc: dial tcp 77.90.31.13:4630: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:48.618875479 (316h24m47.17034594s) [noise=true , address=77.90.31.12:4608 , piece_id=X2N6DK2U6TUZXCP356BDGEBOWY44UEOCG73CWEY57GY7OA7ZTNXA, seg_pos={0 0}, upload=278652]: Done uploading piece. err: ecclient: upload failed (node:12AgtYnGt6M45rGyTZRr4qCFp1BEkgNPBxxsWqBKEn1Cx7AbYLu, address:77.90.31.12:4608): protocol: rpc: dial tcp 77.90.31.12:4608: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:48.619011127 (316h24m47.170481588s) [noise=true , address=77.90.31.19:4876 , piece_id=RDATPSCRLI5BNZVPSBOSAPUK5GFYD2MWCIPRAL6NMPA4Z6GDNZ5A, seg_pos={0 0}, upload=278652]: Done uploading piece. err: ecclient: upload failed (node:12H3UNBGeAufYSdVvEgYkv2QWeFtjKMvHvYGeqGeBvjEHqDDjBB, address:77.90.31.19:4876): protocol: rpc: dial tcp 77.90.31.19:4876: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:48.619102862 (316h24m47.170573323s) [noise=true , address=45.91.116.10:4087 , piece_id=W4QTE4CCAQGK6KODK2DFBY6YEGEGYOHZS7WBJMV7IN3PCHOMOG6A, seg_pos={0 0}, upload=278652]: Done uploading piece. err: ecclient: upload failed (node:125T5WzC4RDcp21oJ8zBMMBq5uw2X32Wf5FszZiWUdsSR95Pd2H, address:45.91.116.10:4087): protocol: write tcp 188.166.225.170:50656->45.91.116.10:4087: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:48.619135034 (316h24m47.170605495s) [seg_pos={0 0}, upload=278652]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil>
Nov 22 12:28:48.858048038 (316h24m47.409518489s) [upload=278652]: Done uploading segments...
Nov 22 12:28:48.858131627 (316h24m47.409602088s) [upload=278652]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6)
Duration: 5.627935321s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.........................................................X..] duration:
5.687138878s trace:
https://jaeger.dev.storj.io/trace/666b0e258bc1797c
FAIL (2025-11-22 12:29:49 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.687172381s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [......X.....................................................] duration:
5.792328058s trace:
https://jaeger.dev.storj.io/trace/76f1ecbff69b5d4e
FAIL (2025-11-22 12:29:38 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, Piece ID: I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ) context deadline exceeded; piecestore: (Node ID: 12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, Piece ID: YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A) context deadline exceeded; piecestore: (Node ID: 12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, Piece ID: P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ) context deadline exceeded; piecestore: (Node ID: 1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, Piece ID: ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA) context deadline exceeded; piecestore: (Node ID: 1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, Piece ID: G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ) context deadline exceeded; piecestore: (Node ID: 1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, Piece ID: NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA) context deadline exceeded
======LOGS======
nts...
Nov 22 11:53:53.345386809 (315h49m51.89685727s) [upload=278145]: Got next segment. Inline: false
Nov 22 11:53:53.345428137 (315h49m51.896898598s) [upload=278145]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 11:53:53.345433948 (315h49m51.896904419s) [upload=278145]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 11:53:54.603541081 (315h49m53.155011552s) [seg_pos={0 0}, upload=278145]: Begin upload segment...
Nov 22 11:53:54.603595445 (315h49m53.155065916s) [seg_pos={0 0}, upload=278145]: Done begin upload segment.
Nov 22 11:53:54.603668043 (315h49m53.155138494s) [noise=true , address=145.79.227.14:4837 , piece_id=TCYBUWS4YPUVLJ6QAPX2QIRT7O2HSBVEKVPPKHCTK336NNNPO7PQ, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.603684554 (315h49m53.155155005s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.604505828 (315h49m53.155976289s) [noise=true , address=45.91.116.11:4119 , piece_id=FGNP4Z5IL65VGJ7NMOJK5JW4Y644RFZG6MHWUOCLFKBJ26KF5W5A, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.604973038 (315h49m53.156443499s) [noise=true , address=145.79.227.28:4726 , piece_id=YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.605425900 (315h49m53.156896361s) [noise=true , address=145.79.227.4:4187 , piece_id=P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.606323339 (315h49m53.15779381s) [noise=true , address=77.90.31.15:4698 , piece_id=3Z7MYVQ62TX4BKJASUPTJTWDZH57PEVBSPT6OV2SMMZMTG2KCYZQ, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.606878326 (315h49m53.158348777s) [noise=true , address=145.79.227.31:4799 , piece_id=RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.607588267 (315h49m53.159058728s) [noise=true , address=145.79.227.25:4619 , piece_id=ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.610211763 (315h49m53.161682215s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.610336751 (315h49m53.161807212s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.612196994 (315h49m53.163667455s) [noise=true , address=145.79.227.9:4366 , piece_id=G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.612485593 (315h49m53.163956064s) [noise=true , address=145.79.227.21:4474 , piece_id=HYSRVRV3ALPITM7ZJ2GHPIGLTZ54YH4KYUZ36PTWAU6V2XBLURKQ, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.614299567 (315h49m53.165770028s) [noise=true , address=145.79.227.20:4438 , piece_id=NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.614454202 (315h49m53.165924673s) [noise=true , address=145.79.227.17:4969 , piece_id=I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.616397682 (315h49m53.167868143s) [noise=true , address=77.90.31.10:4517 , piece_id=KLXWC5RY274YKMLT2U5SBPCNQKIRXDOTPQ7MD76HXA3F2VCYRIEA, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.621720077 (315h49m53.173190538s) [noise=true , address=145.79.227.12:4767 , piece_id=FSJE5Q3Z7KHDGYUMIWRXJKTDOT5QFHZU7WQ4SHC3VQGGWJRNVEOQ, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:56.445932556 (315h49m54.997403007s) [upload=278145]: Next returned nil segment
Nov 22 11:53:56.445949048 (315h49m54.997419509s) [upload=278145]: Waiting for error group managing segments...
Nov 22 11:53:56.734588580 (315h49m55.286059041s) [noise=true , address=77.90.31.10:4517 , piece_id=KLXWC5RY274YKMLT2U5SBPCNQKIRXDOTPQ7MD76HXA3F2VCYRIEA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:56.796169791 (315h49m55.347640252s) [noise=true , address=145.79.227.20:4438 , piece_id=NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:56.800552848 (315h49m55.352023309s) [noise=true , address=45.91.116.11:4119 , piece_id=FGNP4Z5IL65VGJ7NMOJK5JW4Y644RFZG6MHWUOCLFKBJ26KF5W5A, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:56.802134119 (315h49m55.35360458s) [noise=true , address=145.79.227.4:4187 , piece_id=P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:56.811866140 (315h49m55.363336591s) [noise=true , address=145.79.227.17:4969 , piece_id=I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:56.820962632 (315h49m55.372433093s) [noise=true , address=77.90.31.15:4698 , piece_id=3Z7MYVQ62TX4BKJASUPTJTWDZH57PEVBSPT6OV2SMMZMTG2KCYZQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:56.853755886 (315h49m55.405226347s) [noise=true , address=145.79.227.21:4474 , piece_id=HYSRVRV3ALPITM7ZJ2GHPIGLTZ54YH4KYUZ36PTWAU6V2XBLURKQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:56.857215724 (315h49m55.408686185s) [noise=true , address=145.79.227.14:4837 , piece_id=TCYBUWS4YPUVLJ6QAPX2QIRT7O2HSBVEKVPPKHCTK336NNNPO7PQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:57.067648917 (315h49m55.619119378s) [noise=true , address=145.79.227.25:4619 , piece_id=ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:57.093474313 (315h49m55.644944774s) [noise=true , address=145.79.227.9:4366 , piece_id=G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:57.104509907 (315h49m55.655980378s) [noise=true , address=145.79.227.12:4767 , piece_id=FSJE5Q3Z7KHDGYUMIWRXJKTDOT5QFHZU7WQ4SHC3VQGGWJRNVEOQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:57.505088550 (315h49m56.056559011s) [noise=true , address=145.79.227.31:4799 , piece_id=RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:57.609615419 (315h49m56.16108589s) [noise=true , address=145.79.227.28:4726 , piece_id=YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:57.609676946 (315h49m56.161147437s) [seg_pos={0 0}, upload=278145]: Segment reached optimal threshold of 13 pieces.
Nov 22 11:53:57.609972077 (315h49m56.161442539s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW): protocol: write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection
Nov 22 11:53:57.609982889 (315h49m56.16145336s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold
Nov 22 11:53:57.610025941 (315h49m56.161496432s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk): protocol: expected piece hash; context canceled; EOF
Nov 22 11:53:57.610037101 (315h49m56.161507553s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold
Nov 22 11:53:57.610213978 (315h49m56.161684439s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19): protocol: write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection
Nov 22 11:53:57.610224649 (315h49m56.16169511s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold
Nov 22 11:53:57.610261058 (315h49m56.16173152s) [seg_pos={0 0}, upload=278145]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Nov 22 11:53:57.610421113 (315h49m56.161891584s) [upload=278145]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Nov 22 11:53:57.610430852 (315h49m56.161901313s) [upload=278145]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Nov 22 11:54:00.212174190 (315h49m58.763644651s) [upload=278145]: Done uploading segments...
Nov 22 11:54:00.212200729 (315h49m58.763671191s) [upload=278145]: Upload finished. err: <nil>
Duration: 5.792469807s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [..........X...............................................X.] duration:
4.162242587s trace:
https://jaeger.dev.storj.io/trace/69bcc2ce5cc82585
FAIL (2025-11-22 12:29:07 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.162275411s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......X....................................................] duration:
22.209733373s trace:
https://jaeger.dev.storj.io/trace/76f1ecbff69b5d4e
FAIL (2025-11-22 12:28:57 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, Piece ID: I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ) context deadline exceeded; piecestore: (Node ID: 12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, Piece ID: YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A) context deadline exceeded; piecestore: (Node ID: 12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, Piece ID: P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ) context deadline exceeded; piecestore: (Node ID: 1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, Piece ID: ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA) context deadline exceeded; piecestore: (Node ID: 1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, Piece ID: G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ) context deadline exceeded; piecestore: (Node ID: 1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, Piece ID: NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA) context deadline exceeded
======LOGS======
nts...
Nov 22 11:53:53.345386809 (315h49m51.89685727s) [upload=278145]: Got next segment. Inline: false
Nov 22 11:53:53.345428137 (315h49m51.896898598s) [upload=278145]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 11:53:53.345433948 (315h49m51.896904419s) [upload=278145]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 11:53:54.603541081 (315h49m53.155011552s) [seg_pos={0 0}, upload=278145]: Begin upload segment...
Nov 22 11:53:54.603595445 (315h49m53.155065916s) [seg_pos={0 0}, upload=278145]: Done begin upload segment.
Nov 22 11:53:54.603668043 (315h49m53.155138494s) [noise=true , address=145.79.227.14:4837 , piece_id=TCYBUWS4YPUVLJ6QAPX2QIRT7O2HSBVEKVPPKHCTK336NNNPO7PQ, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.603684554 (315h49m53.155155005s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.604505828 (315h49m53.155976289s) [noise=true , address=45.91.116.11:4119 , piece_id=FGNP4Z5IL65VGJ7NMOJK5JW4Y644RFZG6MHWUOCLFKBJ26KF5W5A, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.604973038 (315h49m53.156443499s) [noise=true , address=145.79.227.28:4726 , piece_id=YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.605425900 (315h49m53.156896361s) [noise=true , address=145.79.227.4:4187 , piece_id=P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.606323339 (315h49m53.15779381s) [noise=true , address=77.90.31.15:4698 , piece_id=3Z7MYVQ62TX4BKJASUPTJTWDZH57PEVBSPT6OV2SMMZMTG2KCYZQ, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.606878326 (315h49m53.158348777s) [noise=true , address=145.79.227.31:4799 , piece_id=RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.607588267 (315h49m53.159058728s) [noise=true , address=145.79.227.25:4619 , piece_id=ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.610211763 (315h49m53.161682215s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.610336751 (315h49m53.161807212s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.612196994 (315h49m53.163667455s) [noise=true , address=145.79.227.9:4366 , piece_id=G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.612485593 (315h49m53.163956064s) [noise=true , address=145.79.227.21:4474 , piece_id=HYSRVRV3ALPITM7ZJ2GHPIGLTZ54YH4KYUZ36PTWAU6V2XBLURKQ, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.614299567 (315h49m53.165770028s) [noise=true , address=145.79.227.20:4438 , piece_id=NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.614454202 (315h49m53.165924673s) [noise=true , address=145.79.227.17:4969 , piece_id=I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.616397682 (315h49m53.167868143s) [noise=true , address=77.90.31.10:4517 , piece_id=KLXWC5RY274YKMLT2U5SBPCNQKIRXDOTPQ7MD76HXA3F2VCYRIEA, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.621720077 (315h49m53.173190538s) [noise=true , address=145.79.227.12:4767 , piece_id=FSJE5Q3Z7KHDGYUMIWRXJKTDOT5QFHZU7WQ4SHC3VQGGWJRNVEOQ, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:56.445932556 (315h49m54.997403007s) [upload=278145]: Next returned nil segment
Nov 22 11:53:56.445949048 (315h49m54.997419509s) [upload=278145]: Waiting for error group managing segments...
Nov 22 11:53:56.734588580 (315h49m55.286059041s) [noise=true , address=77.90.31.10:4517 , piece_id=KLXWC5RY274YKMLT2U5SBPCNQKIRXDOTPQ7MD76HXA3F2VCYRIEA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:56.796169791 (315h49m55.347640252s) [noise=true , address=145.79.227.20:4438 , piece_id=NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:56.800552848 (315h49m55.352023309s) [noise=true , address=45.91.116.11:4119 , piece_id=FGNP4Z5IL65VGJ7NMOJK5JW4Y644RFZG6MHWUOCLFKBJ26KF5W5A, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:56.802134119 (315h49m55.35360458s) [noise=true , address=145.79.227.4:4187 , piece_id=P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:56.811866140 (315h49m55.363336591s) [noise=true , address=145.79.227.17:4969 , piece_id=I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:56.820962632 (315h49m55.372433093s) [noise=true , address=77.90.31.15:4698 , piece_id=3Z7MYVQ62TX4BKJASUPTJTWDZH57PEVBSPT6OV2SMMZMTG2KCYZQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:56.853755886 (315h49m55.405226347s) [noise=true , address=145.79.227.21:4474 , piece_id=HYSRVRV3ALPITM7ZJ2GHPIGLTZ54YH4KYUZ36PTWAU6V2XBLURKQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:56.857215724 (315h49m55.408686185s) [noise=true , address=145.79.227.14:4837 , piece_id=TCYBUWS4YPUVLJ6QAPX2QIRT7O2HSBVEKVPPKHCTK336NNNPO7PQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:57.067648917 (315h49m55.619119378s) [noise=true , address=145.79.227.25:4619 , piece_id=ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:57.093474313 (315h49m55.644944774s) [noise=true , address=145.79.227.9:4366 , piece_id=G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:57.104509907 (315h49m55.655980378s) [noise=true , address=145.79.227.12:4767 , piece_id=FSJE5Q3Z7KHDGYUMIWRXJKTDOT5QFHZU7WQ4SHC3VQGGWJRNVEOQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:57.505088550 (315h49m56.056559011s) [noise=true , address=145.79.227.31:4799 , piece_id=RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:57.609615419 (315h49m56.16108589s) [noise=true , address=145.79.227.28:4726 , piece_id=YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:57.609676946 (315h49m56.161147437s) [seg_pos={0 0}, upload=278145]: Segment reached optimal threshold of 13 pieces.
Nov 22 11:53:57.609972077 (315h49m56.161442539s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW): protocol: write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection
Nov 22 11:53:57.609982889 (315h49m56.16145336s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold
Nov 22 11:53:57.610025941 (315h49m56.161496432s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk): protocol: expected piece hash; context canceled; EOF
Nov 22 11:53:57.610037101 (315h49m56.161507553s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold
Nov 22 11:53:57.610213978 (315h49m56.161684439s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19): protocol: write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection
Nov 22 11:53:57.610224649 (315h49m56.16169511s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold
Nov 22 11:53:57.610261058 (315h49m56.16173152s) [seg_pos={0 0}, upload=278145]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Nov 22 11:53:57.610421113 (315h49m56.161891584s) [upload=278145]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Nov 22 11:53:57.610430852 (315h49m56.161901313s) [upload=278145]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Nov 22 11:54:00.212174190 (315h49m58.763644651s) [upload=278145]: Done uploading segments...
Nov 22 11:54:00.212200729 (315h49m58.763671191s) [upload=278145]: Upload finished. err: <nil>
Duration: 22.209761728s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................................................X.] duration:
5.951946639s trace:
https://jaeger.dev.storj.io/trace/78d837dffbcbc1eb
FAIL (2025-11-22 12:28:55 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 6)
======LOGS======
Nov 22 12:27:49.568493454 (316h23m48.119963905s) [noise=true , address=77.90.31.19:4876 , piece_id=RDATPSCRLI5BNZVPSBOSAPUK5GFYD2MWCIPRAL6NMPA4Z6GDNZ5A, seg_pos={0 0}, upload=278652]: Uploading piece...
Nov 22 12:27:49.570659288 (316h23m48.122129749s) [noise=true , address=45.91.116.5:4306 , piece_id=F437446OQSONT5YJK35AQ26QMDMC3HP2CWUPEP46WKZGWNRL2V5A, seg_pos={0 0}, upload=278652]: Uploading piece...
Nov 22 12:27:49.572948898 (316h23m48.124419359s) [noise=true , address=64.49.13.140:4506 , piece_id=B5MGJGEBUK6H2JD6LVVKYO34GYGTKQ2RRLTT53QMJHDJGLAH2PPA, seg_pos={0 0}, upload=278652]: Uploading piece...
Nov 22 12:27:49.573111158 (316h23m48.124581609s) [noise=true , address=45.91.116.4:4948 , piece_id=WNZONHIDT5WLFQW6ZYMZOOQ2DT4ATWKBMRGKVLI5SKY53MAMCEYQ, seg_pos={0 0}, upload=278652]: Uploading piece...
Nov 22 12:27:49.574907569 (316h23m48.12637803s) [noise=true , address=45.91.116.10:4087 , piece_id=W4QTE4CCAQGK6KODK2DFBY6YEGEGYOHZS7WBJMV7IN3PCHOMOG6A, seg_pos={0 0}, upload=278652]: Uploading piece...
Nov 22 12:27:49.576925240 (316h23m48.128395701s) [noise=true , address=77.90.31.10:4539 , piece_id=I5AHSBYZXFUVBLW4L4R4CFVJOFFPLQB7F5AK2GBYDXISM6N72PDQ, seg_pos={0 0}, upload=278652]: Uploading piece...
Nov 22 12:27:49.579508780 (316h23m48.130979241s) [noise=true , address=145.79.227.33:4894 , piece_id=QD6HDQDS37Q6HOIU522KZZ44LXF2AKRVKL5PIQFG437KTXYMKC6Q, seg_pos={0 0}, upload=278652]: Uploading piece...
Nov 22 12:27:49.580586642 (316h23m48.132057104s) [noise=true , address=77.90.31.13:4630 , piece_id=FNU2RL7WP5G4RR3HMDEEKYKNJBLTH6SKZ3FFPZK4B7U4A2RRWWKQ, seg_pos={0 0}, upload=278652]: Uploading piece...
Nov 22 12:27:49.585268507 (316h23m48.136738968s) [noise=true , address=77.90.31.12:4608 , piece_id=X2N6DK2U6TUZXCP356BDGEBOWY44UEOCG73CWEY57GY7OA7ZTNXA, seg_pos={0 0}, upload=278652]: Uploading piece...
Nov 22 12:27:49.588099719 (316h23m48.13957018s) [noise=true , address=45.91.116.6:4986 , piece_id=RHKQEB2Y7KCD4CIROR6DSYPTVG2SUNDKE7C5QW7DHHM24YJHMNOQ, seg_pos={0 0}, upload=278652]: Uploading piece...
Nov 22 12:27:49.589734080 (316h23m48.141204531s) [noise=true , address=145.79.227.12:4790 , piece_id=ZCJGR7J7ZDQWNMWSLFPHSGRGRQYLNMDSO52Z2TFQCQCBXIBVPZYA, seg_pos={0 0}, upload=278652]: Uploading piece...
Nov 22 12:27:49.592364720 (316h23m48.143835201s) [noise=true , address=77.90.31.17:4799 , piece_id=K2U42DVPQMOBOXQ4J2FZDWEQURTV42XSVR23AT32YCKEXAPD24RA, seg_pos={0 0}, upload=278652]: Uploading piece...
Nov 22 12:27:49.600851380 (316h23m48.152321841s) [noise=true , address=145.79.227.7:4314 , piece_id=D2CQYNQ7U5IP4QRSTASQIO5JBXYPWSHOP3HN4SVVOTN6LIQNSN3A, seg_pos={0 0}, upload=278652]: Uploading piece...
Nov 22 12:27:50.972898701 (316h23m49.524369162s) [upload=278652]: Next returned nil segment
Nov 22 12:27:50.972914963 (316h23m49.524385414s) [upload=278652]: Waiting for error group managing segments...
Nov 22 12:27:51.338580227 (316h23m49.890050688s) [noise=true , address=64.49.13.140:4506 , piece_id=B5MGJGEBUK6H2JD6LVVKYO34GYGTKQ2RRLTT53QMJHDJGLAH2PPA, seg_pos={0 0}, upload=278652]: Done uploading piece. err: <nil>
Nov 22 12:27:51.425396702 (316h23m49.976867163s) [noise=true , address=145.79.227.5:4256 , piece_id=UOFRVFML2YA53NWWQZMB3GA5PW4NFULCJJHMALH7CHHNGSG3N7VA, seg_pos={0 0}, upload=278652]: Done uploading piece. err: <nil>
Nov 22 12:27:51.444596362 (316h23m49.996066823s) [noise=true , address=145.79.227.49:4354 , piece_id=357B2N2MEKIVI7ERX5GY33FDAZOO2A3ENZV74UVPEBYSYDNI5GIQ, seg_pos={0 0}, upload=278652]: Done uploading piece. err: <nil>
Nov 22 12:27:51.448630013 (316h23m50.000100474s) [noise=true , address=145.79.227.33:4894 , piece_id=QD6HDQDS37Q6HOIU522KZZ44LXF2AKRVKL5PIQFG437KTXYMKC6Q, seg_pos={0 0}, upload=278652]: Done uploading piece. err: <nil>
Nov 22 12:27:51.466112773 (316h23m50.017583234s) [noise=true , address=145.79.227.12:4790 , piece_id=ZCJGR7J7ZDQWNMWSLFPHSGRGRQYLNMDSO52Z2TFQCQCBXIBVPZYA, seg_pos={0 0}, upload=278652]: Done uploading piece. err: <nil>
Nov 22 12:27:51.471437682 (316h23m50.022908133s) [noise=true , address=145.79.227.7:4314 , piece_id=D2CQYNQ7U5IP4QRSTASQIO5JBXYPWSHOP3HN4SVVOTN6LIQNSN3A, seg_pos={0 0}, upload=278652]: Done uploading piece. err: <nil>
Nov 22 12:28:48.617770024 (316h24m47.169240485s) [noise=true , address=77.90.31.17:4799 , piece_id=K2U42DVPQMOBOXQ4J2FZDWEQURTV42XSVR23AT32YCKEXAPD24RA, seg_pos={0 0}, upload=278652]: Done uploading piece. err: ecclient: upload failed (node:12UhAksgXZuFGAVheaYzP7D4kwVyAjUAq3BtrvdSxszjNCFCTWT, address:77.90.31.17:4799): protocol: rpc: dial tcp 77.90.31.17:4799: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:48.617979923 (316h24m47.169450384s) [noise=true , address=45.91.116.5:4306 , piece_id=F437446OQSONT5YJK35AQ26QMDMC3HP2CWUPEP46WKZGWNRL2V5A, seg_pos={0 0}, upload=278652]: Done uploading piece. err: ecclient: upload failed (node:1QuFvRW9wGKdKRXPK8UdR8uqTm53CnHaF9CsUz2bUh2Ptw9r1F, address:45.91.116.5:4306): protocol: rpc: dial tcp 45.91.116.5:4306: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:48.618033846 (316h24m47.169504308s) [noise=true , address=77.90.31.21:4885 , piece_id=5OL3NVLKPFB23XQFUDQLBX6EJHPF5VICQYSYQKCSDY5SWHILW4QA, seg_pos={0 0}, upload=278652]: Done uploading piece. err: ecclient: upload failed (node:12TfMGrvCMPvQ9ZC1pVoDqjFnUHXtzwwnZB6pCV1em4pdmbK2MP, address:77.90.31.21:4885): protocol: rpc: dial tcp 77.90.31.21:4885: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:48.618228647 (316h24m47.169699098s) [noise=true , address=45.91.116.4:4948 , piece_id=WNZONHIDT5WLFQW6ZYMZOOQ2DT4ATWKBMRGKVLI5SKY53MAMCEYQ, seg_pos={0 0}, upload=278652]: Done uploading piece. err: ecclient: upload failed (node:184g6qHdezLW3d2CKNW8GRgdzeVLQuMEhd8B8c1zEnkKNkeSTu, address:45.91.116.4:4948): protocol: rpc: dial tcp 45.91.116.4:4948: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:48.618249357 (316h24m47.169719848s) [noise=true , address=45.91.116.9:4279 , piece_id=LSPEKODU2YPLRUI6MY37VVRL6QNUJ2IXTZX6BBJHEVDMLTZGNQVQ, seg_pos={0 0}, upload=278652]: Done uploading piece. err: ecclient: upload failed (node:124aq2BHBfu1qSRXCx5XVZJU5U7gzadYHSRLYT8fcwcrtu9z5jG, address:45.91.116.9:4279): protocol: rpc: dial tcp 45.91.116.9:4279: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:48.618444829 (316h24m47.16991529s) [noise=true , address=77.90.31.10:4539 , piece_id=I5AHSBYZXFUVBLW4L4R4CFVJOFFPLQB7F5AK2GBYDXISM6N72PDQ, seg_pos={0 0}, upload=278652]: Done uploading piece. err: ecclient: upload failed (node:1DYHaVii53Q4FhtYNBMC64MP1uKHffSTMdfETK8kS94e4ZPfka, address:77.90.31.10:4539): protocol: rpc: dial tcp 77.90.31.10:4539: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:48.618487821 (316h24m47.169958272s) [noise=true , address=45.91.116.14:4211 , piece_id=5HS3X7L6ACA6KJNWBBEK36IIBOFNANCMF4QL367VJA5OHCJW4SOQ, seg_pos={0 0}, upload=278652]: Done uploading piece. err: ecclient: upload failed (node:1TkBbeKqsAZHyZdNpDgCb2uexqgV59Atgtv7EM1rd4KJEZ4BqF, address:45.91.116.14:4211): protocol: rpc: dial tcp 45.91.116.14:4211: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:48.618612950 (316h24m47.170083451s) [noise=true , address=45.91.116.6:4986 , piece_id=RHKQEB2Y7KCD4CIROR6DSYPTVG2SUNDKE7C5QW7DHHM24YJHMNOQ, seg_pos={0 0}, upload=278652]: Done uploading piece. err: ecclient: upload failed (node:12t1cuEr8MRLBbJ4GNSdYBxdhNQ6xDbTr8PHoEuVhMqh1ksgWon, address:45.91.116.6:4986): protocol: write tcp 188.166.225.170:41196->45.91.116.6:4986: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:48.618698352 (316h24m47.170168813s) [noise=true , address=77.90.31.13:4630 , piece_id=FNU2RL7WP5G4RR3HMDEEKYKNJBLTH6SKZ3FFPZK4B7U4A2RRWWKQ, seg_pos={0 0}, upload=278652]: Done uploading piece. err: ecclient: upload failed (node:1gyQJLnzWmRESyd8m6G7aZWsQwVfqRQNARxRmd7GwwfYwhuzVn, address:77.90.31.13:4630): protocol: rpc: dial tcp 77.90.31.13:4630: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:48.618875479 (316h24m47.17034594s) [noise=true , address=77.90.31.12:4608 , piece_id=X2N6DK2U6TUZXCP356BDGEBOWY44UEOCG73CWEY57GY7OA7ZTNXA, seg_pos={0 0}, upload=278652]: Done uploading piece. err: ecclient: upload failed (node:12AgtYnGt6M45rGyTZRr4qCFp1BEkgNPBxxsWqBKEn1Cx7AbYLu, address:77.90.31.12:4608): protocol: rpc: dial tcp 77.90.31.12:4608: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:48.619011127 (316h24m47.170481588s) [noise=true , address=77.90.31.19:4876 , piece_id=RDATPSCRLI5BNZVPSBOSAPUK5GFYD2MWCIPRAL6NMPA4Z6GDNZ5A, seg_pos={0 0}, upload=278652]: Done uploading piece. err: ecclient: upload failed (node:12H3UNBGeAufYSdVvEgYkv2QWeFtjKMvHvYGeqGeBvjEHqDDjBB, address:77.90.31.19:4876): protocol: rpc: dial tcp 77.90.31.19:4876: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:48.619102862 (316h24m47.170573323s) [noise=true , address=45.91.116.10:4087 , piece_id=W4QTE4CCAQGK6KODK2DFBY6YEGEGYOHZS7WBJMV7IN3PCHOMOG6A, seg_pos={0 0}, upload=278652]: Done uploading piece. err: ecclient: upload failed (node:125T5WzC4RDcp21oJ8zBMMBq5uw2X32Wf5FszZiWUdsSR95Pd2H, address:45.91.116.10:4087): protocol: write tcp 188.166.225.170:50656->45.91.116.10:4087: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:48.619135034 (316h24m47.170605495s) [seg_pos={0 0}, upload=278652]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil>
Nov 22 12:28:48.858048038 (316h24m47.409518489s) [upload=278652]: Done uploading segments...
Nov 22 12:28:48.858131627 (316h24m47.409602088s) [upload=278652]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6)
Duration: 5.951970294s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................................................X.] duration:
7.697813062s trace:
https://jaeger.dev.storj.io/trace/2b03c00e8c9bd6a8
FAIL (2025-11-22 12:28:50 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 7.697830565s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [..........................................................X.] duration:
8.052062539s trace:
https://jaeger.dev.storj.io/trace/666b0e258bc1797c
FAIL (2025-11-22 12:28:50 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 8.052084821s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
1m0.24189208s trace:
https://jaeger.dev.storj.io/trace/78d837dffbcbc1eb
FAIL (2025-11-22 12:28:48 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 6)
======LOGS======
Nov 22 12:27:49.568493454 (316h23m48.119963905s) [noise=true , address=77.90.31.19:4876 , piece_id=RDATPSCRLI5BNZVPSBOSAPUK5GFYD2MWCIPRAL6NMPA4Z6GDNZ5A, seg_pos={0 0}, upload=278652]: Uploading piece...
Nov 22 12:27:49.570659288 (316h23m48.122129749s) [noise=true , address=45.91.116.5:4306 , piece_id=F437446OQSONT5YJK35AQ26QMDMC3HP2CWUPEP46WKZGWNRL2V5A, seg_pos={0 0}, upload=278652]: Uploading piece...
Nov 22 12:27:49.572948898 (316h23m48.124419359s) [noise=true , address=64.49.13.140:4506 , piece_id=B5MGJGEBUK6H2JD6LVVKYO34GYGTKQ2RRLTT53QMJHDJGLAH2PPA, seg_pos={0 0}, upload=278652]: Uploading piece...
Nov 22 12:27:49.573111158 (316h23m48.124581609s) [noise=true , address=45.91.116.4:4948 , piece_id=WNZONHIDT5WLFQW6ZYMZOOQ2DT4ATWKBMRGKVLI5SKY53MAMCEYQ, seg_pos={0 0}, upload=278652]: Uploading piece...
Nov 22 12:27:49.574907569 (316h23m48.12637803s) [noise=true , address=45.91.116.10:4087 , piece_id=W4QTE4CCAQGK6KODK2DFBY6YEGEGYOHZS7WBJMV7IN3PCHOMOG6A, seg_pos={0 0}, upload=278652]: Uploading piece...
Nov 22 12:27:49.576925240 (316h23m48.128395701s) [noise=true , address=77.90.31.10:4539 , piece_id=I5AHSBYZXFUVBLW4L4R4CFVJOFFPLQB7F5AK2GBYDXISM6N72PDQ, seg_pos={0 0}, upload=278652]: Uploading piece...
Nov 22 12:27:49.579508780 (316h23m48.130979241s) [noise=true , address=145.79.227.33:4894 , piece_id=QD6HDQDS37Q6HOIU522KZZ44LXF2AKRVKL5PIQFG437KTXYMKC6Q, seg_pos={0 0}, upload=278652]: Uploading piece...
Nov 22 12:27:49.580586642 (316h23m48.132057104s) [noise=true , address=77.90.31.13:4630 , piece_id=FNU2RL7WP5G4RR3HMDEEKYKNJBLTH6SKZ3FFPZK4B7U4A2RRWWKQ, seg_pos={0 0}, upload=278652]: Uploading piece...
Nov 22 12:27:49.585268507 (316h23m48.136738968s) [noise=true , address=77.90.31.12:4608 , piece_id=X2N6DK2U6TUZXCP356BDGEBOWY44UEOCG73CWEY57GY7OA7ZTNXA, seg_pos={0 0}, upload=278652]: Uploading piece...
Nov 22 12:27:49.588099719 (316h23m48.13957018s) [noise=true , address=45.91.116.6:4986 , piece_id=RHKQEB2Y7KCD4CIROR6DSYPTVG2SUNDKE7C5QW7DHHM24YJHMNOQ, seg_pos={0 0}, upload=278652]: Uploading piece...
Nov 22 12:27:49.589734080 (316h23m48.141204531s) [noise=true , address=145.79.227.12:4790 , piece_id=ZCJGR7J7ZDQWNMWSLFPHSGRGRQYLNMDSO52Z2TFQCQCBXIBVPZYA, seg_pos={0 0}, upload=278652]: Uploading piece...
Nov 22 12:27:49.592364720 (316h23m48.143835201s) [noise=true , address=77.90.31.17:4799 , piece_id=K2U42DVPQMOBOXQ4J2FZDWEQURTV42XSVR23AT32YCKEXAPD24RA, seg_pos={0 0}, upload=278652]: Uploading piece...
Nov 22 12:27:49.600851380 (316h23m48.152321841s) [noise=true , address=145.79.227.7:4314 , piece_id=D2CQYNQ7U5IP4QRSTASQIO5JBXYPWSHOP3HN4SVVOTN6LIQNSN3A, seg_pos={0 0}, upload=278652]: Uploading piece...
Nov 22 12:27:50.972898701 (316h23m49.524369162s) [upload=278652]: Next returned nil segment
Nov 22 12:27:50.972914963 (316h23m49.524385414s) [upload=278652]: Waiting for error group managing segments...
Nov 22 12:27:51.338580227 (316h23m49.890050688s) [noise=true , address=64.49.13.140:4506 , piece_id=B5MGJGEBUK6H2JD6LVVKYO34GYGTKQ2RRLTT53QMJHDJGLAH2PPA, seg_pos={0 0}, upload=278652]: Done uploading piece. err: <nil>
Nov 22 12:27:51.425396702 (316h23m49.976867163s) [noise=true , address=145.79.227.5:4256 , piece_id=UOFRVFML2YA53NWWQZMB3GA5PW4NFULCJJHMALH7CHHNGSG3N7VA, seg_pos={0 0}, upload=278652]: Done uploading piece. err: <nil>
Nov 22 12:27:51.444596362 (316h23m49.996066823s) [noise=true , address=145.79.227.49:4354 , piece_id=357B2N2MEKIVI7ERX5GY33FDAZOO2A3ENZV74UVPEBYSYDNI5GIQ, seg_pos={0 0}, upload=278652]: Done uploading piece. err: <nil>
Nov 22 12:27:51.448630013 (316h23m50.000100474s) [noise=true , address=145.79.227.33:4894 , piece_id=QD6HDQDS37Q6HOIU522KZZ44LXF2AKRVKL5PIQFG437KTXYMKC6Q, seg_pos={0 0}, upload=278652]: Done uploading piece. err: <nil>
Nov 22 12:27:51.466112773 (316h23m50.017583234s) [noise=true , address=145.79.227.12:4790 , piece_id=ZCJGR7J7ZDQWNMWSLFPHSGRGRQYLNMDSO52Z2TFQCQCBXIBVPZYA, seg_pos={0 0}, upload=278652]: Done uploading piece. err: <nil>
Nov 22 12:27:51.471437682 (316h23m50.022908133s) [noise=true , address=145.79.227.7:4314 , piece_id=D2CQYNQ7U5IP4QRSTASQIO5JBXYPWSHOP3HN4SVVOTN6LIQNSN3A, seg_pos={0 0}, upload=278652]: Done uploading piece. err: <nil>
Nov 22 12:28:48.617770024 (316h24m47.169240485s) [noise=true , address=77.90.31.17:4799 , piece_id=K2U42DVPQMOBOXQ4J2FZDWEQURTV42XSVR23AT32YCKEXAPD24RA, seg_pos={0 0}, upload=278652]: Done uploading piece. err: ecclient: upload failed (node:12UhAksgXZuFGAVheaYzP7D4kwVyAjUAq3BtrvdSxszjNCFCTWT, address:77.90.31.17:4799): protocol: rpc: dial tcp 77.90.31.17:4799: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:48.617979923 (316h24m47.169450384s) [noise=true , address=45.91.116.5:4306 , piece_id=F437446OQSONT5YJK35AQ26QMDMC3HP2CWUPEP46WKZGWNRL2V5A, seg_pos={0 0}, upload=278652]: Done uploading piece. err: ecclient: upload failed (node:1QuFvRW9wGKdKRXPK8UdR8uqTm53CnHaF9CsUz2bUh2Ptw9r1F, address:45.91.116.5:4306): protocol: rpc: dial tcp 45.91.116.5:4306: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:48.618033846 (316h24m47.169504308s) [noise=true , address=77.90.31.21:4885 , piece_id=5OL3NVLKPFB23XQFUDQLBX6EJHPF5VICQYSYQKCSDY5SWHILW4QA, seg_pos={0 0}, upload=278652]: Done uploading piece. err: ecclient: upload failed (node:12TfMGrvCMPvQ9ZC1pVoDqjFnUHXtzwwnZB6pCV1em4pdmbK2MP, address:77.90.31.21:4885): protocol: rpc: dial tcp 77.90.31.21:4885: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:48.618228647 (316h24m47.169699098s) [noise=true , address=45.91.116.4:4948 , piece_id=WNZONHIDT5WLFQW6ZYMZOOQ2DT4ATWKBMRGKVLI5SKY53MAMCEYQ, seg_pos={0 0}, upload=278652]: Done uploading piece. err: ecclient: upload failed (node:184g6qHdezLW3d2CKNW8GRgdzeVLQuMEhd8B8c1zEnkKNkeSTu, address:45.91.116.4:4948): protocol: rpc: dial tcp 45.91.116.4:4948: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:48.618249357 (316h24m47.169719848s) [noise=true , address=45.91.116.9:4279 , piece_id=LSPEKODU2YPLRUI6MY37VVRL6QNUJ2IXTZX6BBJHEVDMLTZGNQVQ, seg_pos={0 0}, upload=278652]: Done uploading piece. err: ecclient: upload failed (node:124aq2BHBfu1qSRXCx5XVZJU5U7gzadYHSRLYT8fcwcrtu9z5jG, address:45.91.116.9:4279): protocol: rpc: dial tcp 45.91.116.9:4279: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:48.618444829 (316h24m47.16991529s) [noise=true , address=77.90.31.10:4539 , piece_id=I5AHSBYZXFUVBLW4L4R4CFVJOFFPLQB7F5AK2GBYDXISM6N72PDQ, seg_pos={0 0}, upload=278652]: Done uploading piece. err: ecclient: upload failed (node:1DYHaVii53Q4FhtYNBMC64MP1uKHffSTMdfETK8kS94e4ZPfka, address:77.90.31.10:4539): protocol: rpc: dial tcp 77.90.31.10:4539: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:48.618487821 (316h24m47.169958272s) [noise=true , address=45.91.116.14:4211 , piece_id=5HS3X7L6ACA6KJNWBBEK36IIBOFNANCMF4QL367VJA5OHCJW4SOQ, seg_pos={0 0}, upload=278652]: Done uploading piece. err: ecclient: upload failed (node:1TkBbeKqsAZHyZdNpDgCb2uexqgV59Atgtv7EM1rd4KJEZ4BqF, address:45.91.116.14:4211): protocol: rpc: dial tcp 45.91.116.14:4211: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:48.618612950 (316h24m47.170083451s) [noise=true , address=45.91.116.6:4986 , piece_id=RHKQEB2Y7KCD4CIROR6DSYPTVG2SUNDKE7C5QW7DHHM24YJHMNOQ, seg_pos={0 0}, upload=278652]: Done uploading piece. err: ecclient: upload failed (node:12t1cuEr8MRLBbJ4GNSdYBxdhNQ6xDbTr8PHoEuVhMqh1ksgWon, address:45.91.116.6:4986): protocol: write tcp 188.166.225.170:41196->45.91.116.6:4986: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:48.618698352 (316h24m47.170168813s) [noise=true , address=77.90.31.13:4630 , piece_id=FNU2RL7WP5G4RR3HMDEEKYKNJBLTH6SKZ3FFPZK4B7U4A2RRWWKQ, seg_pos={0 0}, upload=278652]: Done uploading piece. err: ecclient: upload failed (node:1gyQJLnzWmRESyd8m6G7aZWsQwVfqRQNARxRmd7GwwfYwhuzVn, address:77.90.31.13:4630): protocol: rpc: dial tcp 77.90.31.13:4630: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:48.618875479 (316h24m47.17034594s) [noise=true , address=77.90.31.12:4608 , piece_id=X2N6DK2U6TUZXCP356BDGEBOWY44UEOCG73CWEY57GY7OA7ZTNXA, seg_pos={0 0}, upload=278652]: Done uploading piece. err: ecclient: upload failed (node:12AgtYnGt6M45rGyTZRr4qCFp1BEkgNPBxxsWqBKEn1Cx7AbYLu, address:77.90.31.12:4608): protocol: rpc: dial tcp 77.90.31.12:4608: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:48.619011127 (316h24m47.170481588s) [noise=true , address=77.90.31.19:4876 , piece_id=RDATPSCRLI5BNZVPSBOSAPUK5GFYD2MWCIPRAL6NMPA4Z6GDNZ5A, seg_pos={0 0}, upload=278652]: Done uploading piece. err: ecclient: upload failed (node:12H3UNBGeAufYSdVvEgYkv2QWeFtjKMvHvYGeqGeBvjEHqDDjBB, address:77.90.31.19:4876): protocol: rpc: dial tcp 77.90.31.19:4876: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:48.619102862 (316h24m47.170573323s) [noise=true , address=45.91.116.10:4087 , piece_id=W4QTE4CCAQGK6KODK2DFBY6YEGEGYOHZS7WBJMV7IN3PCHOMOG6A, seg_pos={0 0}, upload=278652]: Done uploading piece. err: ecclient: upload failed (node:125T5WzC4RDcp21oJ8zBMMBq5uw2X32Wf5FszZiWUdsSR95Pd2H, address:45.91.116.10:4087): protocol: write tcp 188.166.225.170:50656->45.91.116.10:4087: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:48.619135034 (316h24m47.170605495s) [seg_pos={0 0}, upload=278652]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil>
Nov 22 12:28:48.858048038 (316h24m47.409518489s) [upload=278652]: Done uploading segments...
Nov 22 12:28:48.858131627 (316h24m47.409602088s) [upload=278652]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6)
Duration: 1m0.241905786s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
30.001069757s trace:
https://jaeger.dev.storj.io/trace/2b03c00e8c9bd6a8
FAIL (2025-11-22 12:28:42 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.001096567s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
1m0.001521449s trace:
https://jaeger.dev.storj.io/trace/666b0e258bc1797c
FAIL (2025-11-22 12:28:41 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.001573819s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [...........X...............................................X] duration:
5.001156266s trace:
https://jaeger.dev.storj.io/trace/69bcc2ce5cc82585
FAIL (2025-11-22 12:28:22 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001188329s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [........X...................................................] duration:
5.857497921s trace:
https://jaeger.dev.storj.io/trace/76f1ecbff69b5d4e
FAIL (2025-11-22 12:27:28 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, Piece ID: I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ) context deadline exceeded; piecestore: (Node ID: 12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, Piece ID: YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A) context deadline exceeded; piecestore: (Node ID: 12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, Piece ID: P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ) context deadline exceeded; piecestore: (Node ID: 1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, Piece ID: ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA) context deadline exceeded; piecestore: (Node ID: 1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, Piece ID: G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ) context deadline exceeded; piecestore: (Node ID: 1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, Piece ID: NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA) context deadline exceeded
======LOGS======
nts...
Nov 22 11:53:53.345386809 (315h49m51.89685727s) [upload=278145]: Got next segment. Inline: false
Nov 22 11:53:53.345428137 (315h49m51.896898598s) [upload=278145]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 11:53:53.345433948 (315h49m51.896904419s) [upload=278145]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 11:53:54.603541081 (315h49m53.155011552s) [seg_pos={0 0}, upload=278145]: Begin upload segment...
Nov 22 11:53:54.603595445 (315h49m53.155065916s) [seg_pos={0 0}, upload=278145]: Done begin upload segment.
Nov 22 11:53:54.603668043 (315h49m53.155138494s) [noise=true , address=145.79.227.14:4837 , piece_id=TCYBUWS4YPUVLJ6QAPX2QIRT7O2HSBVEKVPPKHCTK336NNNPO7PQ, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.603684554 (315h49m53.155155005s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.604505828 (315h49m53.155976289s) [noise=true , address=45.91.116.11:4119 , piece_id=FGNP4Z5IL65VGJ7NMOJK5JW4Y644RFZG6MHWUOCLFKBJ26KF5W5A, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.604973038 (315h49m53.156443499s) [noise=true , address=145.79.227.28:4726 , piece_id=YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.605425900 (315h49m53.156896361s) [noise=true , address=145.79.227.4:4187 , piece_id=P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.606323339 (315h49m53.15779381s) [noise=true , address=77.90.31.15:4698 , piece_id=3Z7MYVQ62TX4BKJASUPTJTWDZH57PEVBSPT6OV2SMMZMTG2KCYZQ, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.606878326 (315h49m53.158348777s) [noise=true , address=145.79.227.31:4799 , piece_id=RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.607588267 (315h49m53.159058728s) [noise=true , address=145.79.227.25:4619 , piece_id=ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.610211763 (315h49m53.161682215s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.610336751 (315h49m53.161807212s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.612196994 (315h49m53.163667455s) [noise=true , address=145.79.227.9:4366 , piece_id=G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.612485593 (315h49m53.163956064s) [noise=true , address=145.79.227.21:4474 , piece_id=HYSRVRV3ALPITM7ZJ2GHPIGLTZ54YH4KYUZ36PTWAU6V2XBLURKQ, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.614299567 (315h49m53.165770028s) [noise=true , address=145.79.227.20:4438 , piece_id=NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.614454202 (315h49m53.165924673s) [noise=true , address=145.79.227.17:4969 , piece_id=I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.616397682 (315h49m53.167868143s) [noise=true , address=77.90.31.10:4517 , piece_id=KLXWC5RY274YKMLT2U5SBPCNQKIRXDOTPQ7MD76HXA3F2VCYRIEA, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.621720077 (315h49m53.173190538s) [noise=true , address=145.79.227.12:4767 , piece_id=FSJE5Q3Z7KHDGYUMIWRXJKTDOT5QFHZU7WQ4SHC3VQGGWJRNVEOQ, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:56.445932556 (315h49m54.997403007s) [upload=278145]: Next returned nil segment
Nov 22 11:53:56.445949048 (315h49m54.997419509s) [upload=278145]: Waiting for error group managing segments...
Nov 22 11:53:56.734588580 (315h49m55.286059041s) [noise=true , address=77.90.31.10:4517 , piece_id=KLXWC5RY274YKMLT2U5SBPCNQKIRXDOTPQ7MD76HXA3F2VCYRIEA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:56.796169791 (315h49m55.347640252s) [noise=true , address=145.79.227.20:4438 , piece_id=NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:56.800552848 (315h49m55.352023309s) [noise=true , address=45.91.116.11:4119 , piece_id=FGNP4Z5IL65VGJ7NMOJK5JW4Y644RFZG6MHWUOCLFKBJ26KF5W5A, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:56.802134119 (315h49m55.35360458s) [noise=true , address=145.79.227.4:4187 , piece_id=P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:56.811866140 (315h49m55.363336591s) [noise=true , address=145.79.227.17:4969 , piece_id=I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:56.820962632 (315h49m55.372433093s) [noise=true , address=77.90.31.15:4698 , piece_id=3Z7MYVQ62TX4BKJASUPTJTWDZH57PEVBSPT6OV2SMMZMTG2KCYZQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:56.853755886 (315h49m55.405226347s) [noise=true , address=145.79.227.21:4474 , piece_id=HYSRVRV3ALPITM7ZJ2GHPIGLTZ54YH4KYUZ36PTWAU6V2XBLURKQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:56.857215724 (315h49m55.408686185s) [noise=true , address=145.79.227.14:4837 , piece_id=TCYBUWS4YPUVLJ6QAPX2QIRT7O2HSBVEKVPPKHCTK336NNNPO7PQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:57.067648917 (315h49m55.619119378s) [noise=true , address=145.79.227.25:4619 , piece_id=ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:57.093474313 (315h49m55.644944774s) [noise=true , address=145.79.227.9:4366 , piece_id=G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:57.104509907 (315h49m55.655980378s) [noise=true , address=145.79.227.12:4767 , piece_id=FSJE5Q3Z7KHDGYUMIWRXJKTDOT5QFHZU7WQ4SHC3VQGGWJRNVEOQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:57.505088550 (315h49m56.056559011s) [noise=true , address=145.79.227.31:4799 , piece_id=RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:57.609615419 (315h49m56.16108589s) [noise=true , address=145.79.227.28:4726 , piece_id=YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:57.609676946 (315h49m56.161147437s) [seg_pos={0 0}, upload=278145]: Segment reached optimal threshold of 13 pieces.
Nov 22 11:53:57.609972077 (315h49m56.161442539s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW): protocol: write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection
Nov 22 11:53:57.609982889 (315h49m56.16145336s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold
Nov 22 11:53:57.610025941 (315h49m56.161496432s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk): protocol: expected piece hash; context canceled; EOF
Nov 22 11:53:57.610037101 (315h49m56.161507553s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold
Nov 22 11:53:57.610213978 (315h49m56.161684439s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19): protocol: write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection
Nov 22 11:53:57.610224649 (315h49m56.16169511s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold
Nov 22 11:53:57.610261058 (315h49m56.16173152s) [seg_pos={0 0}, upload=278145]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Nov 22 11:53:57.610421113 (315h49m56.161891584s) [upload=278145]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Nov 22 11:53:57.610430852 (315h49m56.161901313s) [upload=278145]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Nov 22 11:54:00.212174190 (315h49m58.763644651s) [upload=278145]: Done uploading segments...
Nov 22 11:54:00.212200729 (315h49m58.763671191s) [upload=278145]: Upload finished. err: <nil>
Duration: 5.857525733s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [............X...............................................] duration:
4.526657001s trace:
https://jaeger.dev.storj.io/trace/469b06ffc0b434d4
FAIL (2025-11-22 12:27:10 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.526685354s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........X..................................................] duration:
5.757537308s trace:
https://jaeger.dev.storj.io/trace/76f1ecbff69b5d4e
FAIL (2025-11-22 12:27:02 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, Piece ID: I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ) context deadline exceeded; piecestore: (Node ID: 12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, Piece ID: YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A) context deadline exceeded; piecestore: (Node ID: 12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, Piece ID: P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ) context deadline exceeded; piecestore: (Node ID: 1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, Piece ID: ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA) context deadline exceeded; piecestore: (Node ID: 1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, Piece ID: G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ) context deadline exceeded; piecestore: (Node ID: 1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, Piece ID: NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA) context deadline exceeded
======LOGS======
nts...
Nov 22 11:53:53.345386809 (315h49m51.89685727s) [upload=278145]: Got next segment. Inline: false
Nov 22 11:53:53.345428137 (315h49m51.896898598s) [upload=278145]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 11:53:53.345433948 (315h49m51.896904419s) [upload=278145]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 11:53:54.603541081 (315h49m53.155011552s) [seg_pos={0 0}, upload=278145]: Begin upload segment...
Nov 22 11:53:54.603595445 (315h49m53.155065916s) [seg_pos={0 0}, upload=278145]: Done begin upload segment.
Nov 22 11:53:54.603668043 (315h49m53.155138494s) [noise=true , address=145.79.227.14:4837 , piece_id=TCYBUWS4YPUVLJ6QAPX2QIRT7O2HSBVEKVPPKHCTK336NNNPO7PQ, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.603684554 (315h49m53.155155005s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.604505828 (315h49m53.155976289s) [noise=true , address=45.91.116.11:4119 , piece_id=FGNP4Z5IL65VGJ7NMOJK5JW4Y644RFZG6MHWUOCLFKBJ26KF5W5A, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.604973038 (315h49m53.156443499s) [noise=true , address=145.79.227.28:4726 , piece_id=YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.605425900 (315h49m53.156896361s) [noise=true , address=145.79.227.4:4187 , piece_id=P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.606323339 (315h49m53.15779381s) [noise=true , address=77.90.31.15:4698 , piece_id=3Z7MYVQ62TX4BKJASUPTJTWDZH57PEVBSPT6OV2SMMZMTG2KCYZQ, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.606878326 (315h49m53.158348777s) [noise=true , address=145.79.227.31:4799 , piece_id=RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.607588267 (315h49m53.159058728s) [noise=true , address=145.79.227.25:4619 , piece_id=ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.610211763 (315h49m53.161682215s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.610336751 (315h49m53.161807212s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.612196994 (315h49m53.163667455s) [noise=true , address=145.79.227.9:4366 , piece_id=G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.612485593 (315h49m53.163956064s) [noise=true , address=145.79.227.21:4474 , piece_id=HYSRVRV3ALPITM7ZJ2GHPIGLTZ54YH4KYUZ36PTWAU6V2XBLURKQ, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.614299567 (315h49m53.165770028s) [noise=true , address=145.79.227.20:4438 , piece_id=NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.614454202 (315h49m53.165924673s) [noise=true , address=145.79.227.17:4969 , piece_id=I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.616397682 (315h49m53.167868143s) [noise=true , address=77.90.31.10:4517 , piece_id=KLXWC5RY274YKMLT2U5SBPCNQKIRXDOTPQ7MD76HXA3F2VCYRIEA, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.621720077 (315h49m53.173190538s) [noise=true , address=145.79.227.12:4767 , piece_id=FSJE5Q3Z7KHDGYUMIWRXJKTDOT5QFHZU7WQ4SHC3VQGGWJRNVEOQ, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:56.445932556 (315h49m54.997403007s) [upload=278145]: Next returned nil segment
Nov 22 11:53:56.445949048 (315h49m54.997419509s) [upload=278145]: Waiting for error group managing segments...
Nov 22 11:53:56.734588580 (315h49m55.286059041s) [noise=true , address=77.90.31.10:4517 , piece_id=KLXWC5RY274YKMLT2U5SBPCNQKIRXDOTPQ7MD76HXA3F2VCYRIEA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:56.796169791 (315h49m55.347640252s) [noise=true , address=145.79.227.20:4438 , piece_id=NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:56.800552848 (315h49m55.352023309s) [noise=true , address=45.91.116.11:4119 , piece_id=FGNP4Z5IL65VGJ7NMOJK5JW4Y644RFZG6MHWUOCLFKBJ26KF5W5A, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:56.802134119 (315h49m55.35360458s) [noise=true , address=145.79.227.4:4187 , piece_id=P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:56.811866140 (315h49m55.363336591s) [noise=true , address=145.79.227.17:4969 , piece_id=I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:56.820962632 (315h49m55.372433093s) [noise=true , address=77.90.31.15:4698 , piece_id=3Z7MYVQ62TX4BKJASUPTJTWDZH57PEVBSPT6OV2SMMZMTG2KCYZQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:56.853755886 (315h49m55.405226347s) [noise=true , address=145.79.227.21:4474 , piece_id=HYSRVRV3ALPITM7ZJ2GHPIGLTZ54YH4KYUZ36PTWAU6V2XBLURKQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:56.857215724 (315h49m55.408686185s) [noise=true , address=145.79.227.14:4837 , piece_id=TCYBUWS4YPUVLJ6QAPX2QIRT7O2HSBVEKVPPKHCTK336NNNPO7PQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:57.067648917 (315h49m55.619119378s) [noise=true , address=145.79.227.25:4619 , piece_id=ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:57.093474313 (315h49m55.644944774s) [noise=true , address=145.79.227.9:4366 , piece_id=G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:57.104509907 (315h49m55.655980378s) [noise=true , address=145.79.227.12:4767 , piece_id=FSJE5Q3Z7KHDGYUMIWRXJKTDOT5QFHZU7WQ4SHC3VQGGWJRNVEOQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:57.505088550 (315h49m56.056559011s) [noise=true , address=145.79.227.31:4799 , piece_id=RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:57.609615419 (315h49m56.16108589s) [noise=true , address=145.79.227.28:4726 , piece_id=YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:57.609676946 (315h49m56.161147437s) [seg_pos={0 0}, upload=278145]: Segment reached optimal threshold of 13 pieces.
Nov 22 11:53:57.609972077 (315h49m56.161442539s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW): protocol: write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection
Nov 22 11:53:57.609982889 (315h49m56.16145336s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold
Nov 22 11:53:57.610025941 (315h49m56.161496432s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk): protocol: expected piece hash; context canceled; EOF
Nov 22 11:53:57.610037101 (315h49m56.161507553s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold
Nov 22 11:53:57.610213978 (315h49m56.161684439s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19): protocol: write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection
Nov 22 11:53:57.610224649 (315h49m56.16169511s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold
Nov 22 11:53:57.610261058 (315h49m56.16173152s) [seg_pos={0 0}, upload=278145]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Nov 22 11:53:57.610421113 (315h49m56.161891584s) [upload=278145]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Nov 22 11:53:57.610430852 (315h49m56.161901313s) [upload=278145]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Nov 22 11:54:00.212174190 (315h49m58.763644651s) [upload=278145]: Done uploading segments...
Nov 22 11:54:00.212200729 (315h49m58.763671191s) [upload=278145]: Upload finished. err: <nil>
Duration: 5.757566745s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........X.................................................] duration:
5.828794977s trace:
https://jaeger.dev.storj.io/trace/76f1ecbff69b5d4e
FAIL (2025-11-22 12:26:43 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, Piece ID: I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ) context deadline exceeded; piecestore: (Node ID: 12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, Piece ID: YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A) context deadline exceeded; piecestore: (Node ID: 12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, Piece ID: P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ) context deadline exceeded; piecestore: (Node ID: 1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, Piece ID: ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA) context deadline exceeded; piecestore: (Node ID: 1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, Piece ID: G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ) context deadline exceeded; piecestore: (Node ID: 1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, Piece ID: NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA) context deadline exceeded
======LOGS======
nts...
Nov 22 11:53:53.345386809 (315h49m51.89685727s) [upload=278145]: Got next segment. Inline: false
Nov 22 11:53:53.345428137 (315h49m51.896898598s) [upload=278145]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 11:53:53.345433948 (315h49m51.896904419s) [upload=278145]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 11:53:54.603541081 (315h49m53.155011552s) [seg_pos={0 0}, upload=278145]: Begin upload segment...
Nov 22 11:53:54.603595445 (315h49m53.155065916s) [seg_pos={0 0}, upload=278145]: Done begin upload segment.
Nov 22 11:53:54.603668043 (315h49m53.155138494s) [noise=true , address=145.79.227.14:4837 , piece_id=TCYBUWS4YPUVLJ6QAPX2QIRT7O2HSBVEKVPPKHCTK336NNNPO7PQ, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.603684554 (315h49m53.155155005s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.604505828 (315h49m53.155976289s) [noise=true , address=45.91.116.11:4119 , piece_id=FGNP4Z5IL65VGJ7NMOJK5JW4Y644RFZG6MHWUOCLFKBJ26KF5W5A, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.604973038 (315h49m53.156443499s) [noise=true , address=145.79.227.28:4726 , piece_id=YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.605425900 (315h49m53.156896361s) [noise=true , address=145.79.227.4:4187 , piece_id=P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.606323339 (315h49m53.15779381s) [noise=true , address=77.90.31.15:4698 , piece_id=3Z7MYVQ62TX4BKJASUPTJTWDZH57PEVBSPT6OV2SMMZMTG2KCYZQ, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.606878326 (315h49m53.158348777s) [noise=true , address=145.79.227.31:4799 , piece_id=RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.607588267 (315h49m53.159058728s) [noise=true , address=145.79.227.25:4619 , piece_id=ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.610211763 (315h49m53.161682215s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.610336751 (315h49m53.161807212s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.612196994 (315h49m53.163667455s) [noise=true , address=145.79.227.9:4366 , piece_id=G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.612485593 (315h49m53.163956064s) [noise=true , address=145.79.227.21:4474 , piece_id=HYSRVRV3ALPITM7ZJ2GHPIGLTZ54YH4KYUZ36PTWAU6V2XBLURKQ, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.614299567 (315h49m53.165770028s) [noise=true , address=145.79.227.20:4438 , piece_id=NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.614454202 (315h49m53.165924673s) [noise=true , address=145.79.227.17:4969 , piece_id=I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.616397682 (315h49m53.167868143s) [noise=true , address=77.90.31.10:4517 , piece_id=KLXWC5RY274YKMLT2U5SBPCNQKIRXDOTPQ7MD76HXA3F2VCYRIEA, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.621720077 (315h49m53.173190538s) [noise=true , address=145.79.227.12:4767 , piece_id=FSJE5Q3Z7KHDGYUMIWRXJKTDOT5QFHZU7WQ4SHC3VQGGWJRNVEOQ, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:56.445932556 (315h49m54.997403007s) [upload=278145]: Next returned nil segment
Nov 22 11:53:56.445949048 (315h49m54.997419509s) [upload=278145]: Waiting for error group managing segments...
Nov 22 11:53:56.734588580 (315h49m55.286059041s) [noise=true , address=77.90.31.10:4517 , piece_id=KLXWC5RY274YKMLT2U5SBPCNQKIRXDOTPQ7MD76HXA3F2VCYRIEA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:56.796169791 (315h49m55.347640252s) [noise=true , address=145.79.227.20:4438 , piece_id=NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:56.800552848 (315h49m55.352023309s) [noise=true , address=45.91.116.11:4119 , piece_id=FGNP4Z5IL65VGJ7NMOJK5JW4Y644RFZG6MHWUOCLFKBJ26KF5W5A, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:56.802134119 (315h49m55.35360458s) [noise=true , address=145.79.227.4:4187 , piece_id=P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:56.811866140 (315h49m55.363336591s) [noise=true , address=145.79.227.17:4969 , piece_id=I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:56.820962632 (315h49m55.372433093s) [noise=true , address=77.90.31.15:4698 , piece_id=3Z7MYVQ62TX4BKJASUPTJTWDZH57PEVBSPT6OV2SMMZMTG2KCYZQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:56.853755886 (315h49m55.405226347s) [noise=true , address=145.79.227.21:4474 , piece_id=HYSRVRV3ALPITM7ZJ2GHPIGLTZ54YH4KYUZ36PTWAU6V2XBLURKQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:56.857215724 (315h49m55.408686185s) [noise=true , address=145.79.227.14:4837 , piece_id=TCYBUWS4YPUVLJ6QAPX2QIRT7O2HSBVEKVPPKHCTK336NNNPO7PQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:57.067648917 (315h49m55.619119378s) [noise=true , address=145.79.227.25:4619 , piece_id=ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:57.093474313 (315h49m55.644944774s) [noise=true , address=145.79.227.9:4366 , piece_id=G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:57.104509907 (315h49m55.655980378s) [noise=true , address=145.79.227.12:4767 , piece_id=FSJE5Q3Z7KHDGYUMIWRXJKTDOT5QFHZU7WQ4SHC3VQGGWJRNVEOQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:57.505088550 (315h49m56.056559011s) [noise=true , address=145.79.227.31:4799 , piece_id=RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:57.609615419 (315h49m56.16108589s) [noise=true , address=145.79.227.28:4726 , piece_id=YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:57.609676946 (315h49m56.161147437s) [seg_pos={0 0}, upload=278145]: Segment reached optimal threshold of 13 pieces.
Nov 22 11:53:57.609972077 (315h49m56.161442539s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW): protocol: write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection
Nov 22 11:53:57.609982889 (315h49m56.16145336s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold
Nov 22 11:53:57.610025941 (315h49m56.161496432s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk): protocol: expected piece hash; context canceled; EOF
Nov 22 11:53:57.610037101 (315h49m56.161507553s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold
Nov 22 11:53:57.610213978 (315h49m56.161684439s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19): protocol: write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection
Nov 22 11:53:57.610224649 (315h49m56.16169511s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold
Nov 22 11:53:57.610261058 (315h49m56.16173152s) [seg_pos={0 0}, upload=278145]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Nov 22 11:53:57.610421113 (315h49m56.161891584s) [upload=278145]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Nov 22 11:53:57.610430852 (315h49m56.161901313s) [upload=278145]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Nov 22 11:54:00.212174190 (315h49m58.763644651s) [upload=278145]: Done uploading segments...
Nov 22 11:54:00.212200729 (315h49m58.763671191s) [upload=278145]: Upload finished. err: <nil>
Duration: 5.828823651s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [.............X..............................................] duration:
3.52491347s trace:
https://jaeger.dev.storj.io/trace/469b06ffc0b434d4
FAIL (2025-11-22 12:26:03 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.525008911s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........X................................................] duration:
5.83753046s trace:
https://jaeger.dev.storj.io/trace/76f1ecbff69b5d4e
FAIL (2025-11-22 12:25:50 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, Piece ID: I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ) context deadline exceeded; piecestore: (Node ID: 12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, Piece ID: YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A) context deadline exceeded; piecestore: (Node ID: 12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, Piece ID: P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ) context deadline exceeded; piecestore: (Node ID: 1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, Piece ID: ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA) context deadline exceeded; piecestore: (Node ID: 1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, Piece ID: G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ) context deadline exceeded; piecestore: (Node ID: 1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, Piece ID: NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA) context deadline exceeded
======LOGS======
nts...
Nov 22 11:53:53.345386809 (315h49m51.89685727s) [upload=278145]: Got next segment. Inline: false
Nov 22 11:53:53.345428137 (315h49m51.896898598s) [upload=278145]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 11:53:53.345433948 (315h49m51.896904419s) [upload=278145]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 11:53:54.603541081 (315h49m53.155011552s) [seg_pos={0 0}, upload=278145]: Begin upload segment...
Nov 22 11:53:54.603595445 (315h49m53.155065916s) [seg_pos={0 0}, upload=278145]: Done begin upload segment.
Nov 22 11:53:54.603668043 (315h49m53.155138494s) [noise=true , address=145.79.227.14:4837 , piece_id=TCYBUWS4YPUVLJ6QAPX2QIRT7O2HSBVEKVPPKHCTK336NNNPO7PQ, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.603684554 (315h49m53.155155005s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.604505828 (315h49m53.155976289s) [noise=true , address=45.91.116.11:4119 , piece_id=FGNP4Z5IL65VGJ7NMOJK5JW4Y644RFZG6MHWUOCLFKBJ26KF5W5A, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.604973038 (315h49m53.156443499s) [noise=true , address=145.79.227.28:4726 , piece_id=YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.605425900 (315h49m53.156896361s) [noise=true , address=145.79.227.4:4187 , piece_id=P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.606323339 (315h49m53.15779381s) [noise=true , address=77.90.31.15:4698 , piece_id=3Z7MYVQ62TX4BKJASUPTJTWDZH57PEVBSPT6OV2SMMZMTG2KCYZQ, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.606878326 (315h49m53.158348777s) [noise=true , address=145.79.227.31:4799 , piece_id=RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.607588267 (315h49m53.159058728s) [noise=true , address=145.79.227.25:4619 , piece_id=ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.610211763 (315h49m53.161682215s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.610336751 (315h49m53.161807212s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.612196994 (315h49m53.163667455s) [noise=true , address=145.79.227.9:4366 , piece_id=G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.612485593 (315h49m53.163956064s) [noise=true , address=145.79.227.21:4474 , piece_id=HYSRVRV3ALPITM7ZJ2GHPIGLTZ54YH4KYUZ36PTWAU6V2XBLURKQ, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.614299567 (315h49m53.165770028s) [noise=true , address=145.79.227.20:4438 , piece_id=NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.614454202 (315h49m53.165924673s) [noise=true , address=145.79.227.17:4969 , piece_id=I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.616397682 (315h49m53.167868143s) [noise=true , address=77.90.31.10:4517 , piece_id=KLXWC5RY274YKMLT2U5SBPCNQKIRXDOTPQ7MD76HXA3F2VCYRIEA, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.621720077 (315h49m53.173190538s) [noise=true , address=145.79.227.12:4767 , piece_id=FSJE5Q3Z7KHDGYUMIWRXJKTDOT5QFHZU7WQ4SHC3VQGGWJRNVEOQ, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:56.445932556 (315h49m54.997403007s) [upload=278145]: Next returned nil segment
Nov 22 11:53:56.445949048 (315h49m54.997419509s) [upload=278145]: Waiting for error group managing segments...
Nov 22 11:53:56.734588580 (315h49m55.286059041s) [noise=true , address=77.90.31.10:4517 , piece_id=KLXWC5RY274YKMLT2U5SBPCNQKIRXDOTPQ7MD76HXA3F2VCYRIEA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:56.796169791 (315h49m55.347640252s) [noise=true , address=145.79.227.20:4438 , piece_id=NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:56.800552848 (315h49m55.352023309s) [noise=true , address=45.91.116.11:4119 , piece_id=FGNP4Z5IL65VGJ7NMOJK5JW4Y644RFZG6MHWUOCLFKBJ26KF5W5A, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:56.802134119 (315h49m55.35360458s) [noise=true , address=145.79.227.4:4187 , piece_id=P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:56.811866140 (315h49m55.363336591s) [noise=true , address=145.79.227.17:4969 , piece_id=I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:56.820962632 (315h49m55.372433093s) [noise=true , address=77.90.31.15:4698 , piece_id=3Z7MYVQ62TX4BKJASUPTJTWDZH57PEVBSPT6OV2SMMZMTG2KCYZQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:56.853755886 (315h49m55.405226347s) [noise=true , address=145.79.227.21:4474 , piece_id=HYSRVRV3ALPITM7ZJ2GHPIGLTZ54YH4KYUZ36PTWAU6V2XBLURKQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:56.857215724 (315h49m55.408686185s) [noise=true , address=145.79.227.14:4837 , piece_id=TCYBUWS4YPUVLJ6QAPX2QIRT7O2HSBVEKVPPKHCTK336NNNPO7PQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:57.067648917 (315h49m55.619119378s) [noise=true , address=145.79.227.25:4619 , piece_id=ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:57.093474313 (315h49m55.644944774s) [noise=true , address=145.79.227.9:4366 , piece_id=G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:57.104509907 (315h49m55.655980378s) [noise=true , address=145.79.227.12:4767 , piece_id=FSJE5Q3Z7KHDGYUMIWRXJKTDOT5QFHZU7WQ4SHC3VQGGWJRNVEOQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:57.505088550 (315h49m56.056559011s) [noise=true , address=145.79.227.31:4799 , piece_id=RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:57.609615419 (315h49m56.16108589s) [noise=true , address=145.79.227.28:4726 , piece_id=YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:57.609676946 (315h49m56.161147437s) [seg_pos={0 0}, upload=278145]: Segment reached optimal threshold of 13 pieces.
Nov 22 11:53:57.609972077 (315h49m56.161442539s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW): protocol: write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection
Nov 22 11:53:57.609982889 (315h49m56.16145336s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold
Nov 22 11:53:57.610025941 (315h49m56.161496432s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk): protocol: expected piece hash; context canceled; EOF
Nov 22 11:53:57.610037101 (315h49m56.161507553s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold
Nov 22 11:53:57.610213978 (315h49m56.161684439s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19): protocol: write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection
Nov 22 11:53:57.610224649 (315h49m56.16169511s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold
Nov 22 11:53:57.610261058 (315h49m56.16173152s) [seg_pos={0 0}, upload=278145]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Nov 22 11:53:57.610421113 (315h49m56.161891584s) [upload=278145]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Nov 22 11:53:57.610430852 (315h49m56.161901313s) [upload=278145]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Nov 22 11:54:00.212174190 (315h49m58.763644651s) [upload=278145]: Done uploading segments...
Nov 22 11:54:00.212200729 (315h49m58.763671191s) [upload=278145]: Upload finished. err: <nil>
Duration: 5.837591445s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [..............X.............................................] duration:
3.457932825s trace:
https://jaeger.dev.storj.io/trace/469b06ffc0b434d4
FAIL (2025-11-22 12:25:34 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.457961248s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [...............X............................................] duration:
3.638230836s trace:
https://jaeger.dev.storj.io/trace/469b06ffc0b434d4
FAIL (2025-11-22 12:25:10 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.638280289s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [............X...............................................] duration:
5.633935858s trace:
https://jaeger.dev.storj.io/trace/76f1ecbff69b5d4e
FAIL (2025-11-22 12:24:52 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, Piece ID: I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ) context deadline exceeded; piecestore: (Node ID: 12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, Piece ID: YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A) context deadline exceeded; piecestore: (Node ID: 12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, Piece ID: P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ) context deadline exceeded; piecestore: (Node ID: 1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, Piece ID: ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA) context deadline exceeded; piecestore: (Node ID: 1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, Piece ID: G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ) context deadline exceeded; piecestore: (Node ID: 1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, Piece ID: NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA) context deadline exceeded
======LOGS======
nts...
Nov 22 11:53:53.345386809 (315h49m51.89685727s) [upload=278145]: Got next segment. Inline: false
Nov 22 11:53:53.345428137 (315h49m51.896898598s) [upload=278145]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 11:53:53.345433948 (315h49m51.896904419s) [upload=278145]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 11:53:54.603541081 (315h49m53.155011552s) [seg_pos={0 0}, upload=278145]: Begin upload segment...
Nov 22 11:53:54.603595445 (315h49m53.155065916s) [seg_pos={0 0}, upload=278145]: Done begin upload segment.
Nov 22 11:53:54.603668043 (315h49m53.155138494s) [noise=true , address=145.79.227.14:4837 , piece_id=TCYBUWS4YPUVLJ6QAPX2QIRT7O2HSBVEKVPPKHCTK336NNNPO7PQ, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.603684554 (315h49m53.155155005s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.604505828 (315h49m53.155976289s) [noise=true , address=45.91.116.11:4119 , piece_id=FGNP4Z5IL65VGJ7NMOJK5JW4Y644RFZG6MHWUOCLFKBJ26KF5W5A, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.604973038 (315h49m53.156443499s) [noise=true , address=145.79.227.28:4726 , piece_id=YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.605425900 (315h49m53.156896361s) [noise=true , address=145.79.227.4:4187 , piece_id=P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.606323339 (315h49m53.15779381s) [noise=true , address=77.90.31.15:4698 , piece_id=3Z7MYVQ62TX4BKJASUPTJTWDZH57PEVBSPT6OV2SMMZMTG2KCYZQ, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.606878326 (315h49m53.158348777s) [noise=true , address=145.79.227.31:4799 , piece_id=RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.607588267 (315h49m53.159058728s) [noise=true , address=145.79.227.25:4619 , piece_id=ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.610211763 (315h49m53.161682215s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.610336751 (315h49m53.161807212s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.612196994 (315h49m53.163667455s) [noise=true , address=145.79.227.9:4366 , piece_id=G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.612485593 (315h49m53.163956064s) [noise=true , address=145.79.227.21:4474 , piece_id=HYSRVRV3ALPITM7ZJ2GHPIGLTZ54YH4KYUZ36PTWAU6V2XBLURKQ, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.614299567 (315h49m53.165770028s) [noise=true , address=145.79.227.20:4438 , piece_id=NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.614454202 (315h49m53.165924673s) [noise=true , address=145.79.227.17:4969 , piece_id=I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.616397682 (315h49m53.167868143s) [noise=true , address=77.90.31.10:4517 , piece_id=KLXWC5RY274YKMLT2U5SBPCNQKIRXDOTPQ7MD76HXA3F2VCYRIEA, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.621720077 (315h49m53.173190538s) [noise=true , address=145.79.227.12:4767 , piece_id=FSJE5Q3Z7KHDGYUMIWRXJKTDOT5QFHZU7WQ4SHC3VQGGWJRNVEOQ, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:56.445932556 (315h49m54.997403007s) [upload=278145]: Next returned nil segment
Nov 22 11:53:56.445949048 (315h49m54.997419509s) [upload=278145]: Waiting for error group managing segments...
Nov 22 11:53:56.734588580 (315h49m55.286059041s) [noise=true , address=77.90.31.10:4517 , piece_id=KLXWC5RY274YKMLT2U5SBPCNQKIRXDOTPQ7MD76HXA3F2VCYRIEA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:56.796169791 (315h49m55.347640252s) [noise=true , address=145.79.227.20:4438 , piece_id=NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:56.800552848 (315h49m55.352023309s) [noise=true , address=45.91.116.11:4119 , piece_id=FGNP4Z5IL65VGJ7NMOJK5JW4Y644RFZG6MHWUOCLFKBJ26KF5W5A, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:56.802134119 (315h49m55.35360458s) [noise=true , address=145.79.227.4:4187 , piece_id=P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:56.811866140 (315h49m55.363336591s) [noise=true , address=145.79.227.17:4969 , piece_id=I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:56.820962632 (315h49m55.372433093s) [noise=true , address=77.90.31.15:4698 , piece_id=3Z7MYVQ62TX4BKJASUPTJTWDZH57PEVBSPT6OV2SMMZMTG2KCYZQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:56.853755886 (315h49m55.405226347s) [noise=true , address=145.79.227.21:4474 , piece_id=HYSRVRV3ALPITM7ZJ2GHPIGLTZ54YH4KYUZ36PTWAU6V2XBLURKQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:56.857215724 (315h49m55.408686185s) [noise=true , address=145.79.227.14:4837 , piece_id=TCYBUWS4YPUVLJ6QAPX2QIRT7O2HSBVEKVPPKHCTK336NNNPO7PQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:57.067648917 (315h49m55.619119378s) [noise=true , address=145.79.227.25:4619 , piece_id=ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:57.093474313 (315h49m55.644944774s) [noise=true , address=145.79.227.9:4366 , piece_id=G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:57.104509907 (315h49m55.655980378s) [noise=true , address=145.79.227.12:4767 , piece_id=FSJE5Q3Z7KHDGYUMIWRXJKTDOT5QFHZU7WQ4SHC3VQGGWJRNVEOQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:57.505088550 (315h49m56.056559011s) [noise=true , address=145.79.227.31:4799 , piece_id=RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:57.609615419 (315h49m56.16108589s) [noise=true , address=145.79.227.28:4726 , piece_id=YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:57.609676946 (315h49m56.161147437s) [seg_pos={0 0}, upload=278145]: Segment reached optimal threshold of 13 pieces.
Nov 22 11:53:57.609972077 (315h49m56.161442539s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW): protocol: write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection
Nov 22 11:53:57.609982889 (315h49m56.16145336s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold
Nov 22 11:53:57.610025941 (315h49m56.161496432s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk): protocol: expected piece hash; context canceled; EOF
Nov 22 11:53:57.610037101 (315h49m56.161507553s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold
Nov 22 11:53:57.610213978 (315h49m56.161684439s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19): protocol: write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection
Nov 22 11:53:57.610224649 (315h49m56.16169511s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold
Nov 22 11:53:57.610261058 (315h49m56.16173152s) [seg_pos={0 0}, upload=278145]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Nov 22 11:53:57.610421113 (315h49m56.161891584s) [upload=278145]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Nov 22 11:53:57.610430852 (315h49m56.161901313s) [upload=278145]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Nov 22 11:54:00.212174190 (315h49m58.763644651s) [upload=278145]: Done uploading segments...
Nov 22 11:54:00.212200729 (315h49m58.763671191s) [upload=278145]: Upload finished. err: <nil>
Duration: 5.634038014s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.............X..............................................] duration:
5.685049672s trace:
https://jaeger.dev.storj.io/trace/76f1ecbff69b5d4e
FAIL (2025-11-22 12:24:34 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, Piece ID: I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ) context deadline exceeded; piecestore: (Node ID: 12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, Piece ID: YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A) context deadline exceeded; piecestore: (Node ID: 12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, Piece ID: P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ) context deadline exceeded; piecestore: (Node ID: 1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, Piece ID: ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA) context deadline exceeded; piecestore: (Node ID: 1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, Piece ID: G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ) context deadline exceeded; piecestore: (Node ID: 1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, Piece ID: NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA) context deadline exceeded
======LOGS======
nts...
Nov 22 11:53:53.345386809 (315h49m51.89685727s) [upload=278145]: Got next segment. Inline: false
Nov 22 11:53:53.345428137 (315h49m51.896898598s) [upload=278145]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 11:53:53.345433948 (315h49m51.896904419s) [upload=278145]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 11:53:54.603541081 (315h49m53.155011552s) [seg_pos={0 0}, upload=278145]: Begin upload segment...
Nov 22 11:53:54.603595445 (315h49m53.155065916s) [seg_pos={0 0}, upload=278145]: Done begin upload segment.
Nov 22 11:53:54.603668043 (315h49m53.155138494s) [noise=true , address=145.79.227.14:4837 , piece_id=TCYBUWS4YPUVLJ6QAPX2QIRT7O2HSBVEKVPPKHCTK336NNNPO7PQ, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.603684554 (315h49m53.155155005s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.604505828 (315h49m53.155976289s) [noise=true , address=45.91.116.11:4119 , piece_id=FGNP4Z5IL65VGJ7NMOJK5JW4Y644RFZG6MHWUOCLFKBJ26KF5W5A, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.604973038 (315h49m53.156443499s) [noise=true , address=145.79.227.28:4726 , piece_id=YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.605425900 (315h49m53.156896361s) [noise=true , address=145.79.227.4:4187 , piece_id=P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.606323339 (315h49m53.15779381s) [noise=true , address=77.90.31.15:4698 , piece_id=3Z7MYVQ62TX4BKJASUPTJTWDZH57PEVBSPT6OV2SMMZMTG2KCYZQ, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.606878326 (315h49m53.158348777s) [noise=true , address=145.79.227.31:4799 , piece_id=RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.607588267 (315h49m53.159058728s) [noise=true , address=145.79.227.25:4619 , piece_id=ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.610211763 (315h49m53.161682215s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.610336751 (315h49m53.161807212s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.612196994 (315h49m53.163667455s) [noise=true , address=145.79.227.9:4366 , piece_id=G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.612485593 (315h49m53.163956064s) [noise=true , address=145.79.227.21:4474 , piece_id=HYSRVRV3ALPITM7ZJ2GHPIGLTZ54YH4KYUZ36PTWAU6V2XBLURKQ, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.614299567 (315h49m53.165770028s) [noise=true , address=145.79.227.20:4438 , piece_id=NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.614454202 (315h49m53.165924673s) [noise=true , address=145.79.227.17:4969 , piece_id=I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.616397682 (315h49m53.167868143s) [noise=true , address=77.90.31.10:4517 , piece_id=KLXWC5RY274YKMLT2U5SBPCNQKIRXDOTPQ7MD76HXA3F2VCYRIEA, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.621720077 (315h49m53.173190538s) [noise=true , address=145.79.227.12:4767 , piece_id=FSJE5Q3Z7KHDGYUMIWRXJKTDOT5QFHZU7WQ4SHC3VQGGWJRNVEOQ, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:56.445932556 (315h49m54.997403007s) [upload=278145]: Next returned nil segment
Nov 22 11:53:56.445949048 (315h49m54.997419509s) [upload=278145]: Waiting for error group managing segments...
Nov 22 11:53:56.734588580 (315h49m55.286059041s) [noise=true , address=77.90.31.10:4517 , piece_id=KLXWC5RY274YKMLT2U5SBPCNQKIRXDOTPQ7MD76HXA3F2VCYRIEA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:56.796169791 (315h49m55.347640252s) [noise=true , address=145.79.227.20:4438 , piece_id=NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:56.800552848 (315h49m55.352023309s) [noise=true , address=45.91.116.11:4119 , piece_id=FGNP4Z5IL65VGJ7NMOJK5JW4Y644RFZG6MHWUOCLFKBJ26KF5W5A, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:56.802134119 (315h49m55.35360458s) [noise=true , address=145.79.227.4:4187 , piece_id=P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:56.811866140 (315h49m55.363336591s) [noise=true , address=145.79.227.17:4969 , piece_id=I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:56.820962632 (315h49m55.372433093s) [noise=true , address=77.90.31.15:4698 , piece_id=3Z7MYVQ62TX4BKJASUPTJTWDZH57PEVBSPT6OV2SMMZMTG2KCYZQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:56.853755886 (315h49m55.405226347s) [noise=true , address=145.79.227.21:4474 , piece_id=HYSRVRV3ALPITM7ZJ2GHPIGLTZ54YH4KYUZ36PTWAU6V2XBLURKQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:56.857215724 (315h49m55.408686185s) [noise=true , address=145.79.227.14:4837 , piece_id=TCYBUWS4YPUVLJ6QAPX2QIRT7O2HSBVEKVPPKHCTK336NNNPO7PQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:57.067648917 (315h49m55.619119378s) [noise=true , address=145.79.227.25:4619 , piece_id=ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:57.093474313 (315h49m55.644944774s) [noise=true , address=145.79.227.9:4366 , piece_id=G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:57.104509907 (315h49m55.655980378s) [noise=true , address=145.79.227.12:4767 , piece_id=FSJE5Q3Z7KHDGYUMIWRXJKTDOT5QFHZU7WQ4SHC3VQGGWJRNVEOQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:57.505088550 (315h49m56.056559011s) [noise=true , address=145.79.227.31:4799 , piece_id=RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:57.609615419 (315h49m56.16108589s) [noise=true , address=145.79.227.28:4726 , piece_id=YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:57.609676946 (315h49m56.161147437s) [seg_pos={0 0}, upload=278145]: Segment reached optimal threshold of 13 pieces.
Nov 22 11:53:57.609972077 (315h49m56.161442539s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW): protocol: write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection
Nov 22 11:53:57.609982889 (315h49m56.16145336s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold
Nov 22 11:53:57.610025941 (315h49m56.161496432s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk): protocol: expected piece hash; context canceled; EOF
Nov 22 11:53:57.610037101 (315h49m56.161507553s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold
Nov 22 11:53:57.610213978 (315h49m56.161684439s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19): protocol: write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection
Nov 22 11:53:57.610224649 (315h49m56.16169511s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold
Nov 22 11:53:57.610261058 (315h49m56.16173152s) [seg_pos={0 0}, upload=278145]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Nov 22 11:53:57.610421113 (315h49m56.161891584s) [upload=278145]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Nov 22 11:53:57.610430852 (315h49m56.161901313s) [upload=278145]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Nov 22 11:54:00.212174190 (315h49m58.763644651s) [upload=278145]: Done uploading segments...
Nov 22 11:54:00.212200729 (315h49m58.763671191s) [upload=278145]: Upload finished. err: <nil>
Duration: 5.685078837s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [................X...........................................] duration:
3.400965468s trace:
https://jaeger.dev.storj.io/trace/469b06ffc0b434d4
FAIL (2025-11-22 12:24:32 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.40099794s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [.................X..........................................] duration:
3.459195679s trace:
https://jaeger.dev.storj.io/trace/469b06ffc0b434d4
FAIL (2025-11-22 12:23:34 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.45922192s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..............X.............................................] duration:
5.684049643s trace:
https://jaeger.dev.storj.io/trace/76f1ecbff69b5d4e
FAIL (2025-11-22 12:23:25 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, Piece ID: I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ) context deadline exceeded; piecestore: (Node ID: 12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, Piece ID: YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A) context deadline exceeded; piecestore: (Node ID: 12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, Piece ID: P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ) context deadline exceeded; piecestore: (Node ID: 1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, Piece ID: ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA) context deadline exceeded; piecestore: (Node ID: 1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, Piece ID: G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ) context deadline exceeded; piecestore: (Node ID: 1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, Piece ID: NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA) context deadline exceeded
======LOGS======
nts...
Nov 22 11:53:53.345386809 (315h49m51.89685727s) [upload=278145]: Got next segment. Inline: false
Nov 22 11:53:53.345428137 (315h49m51.896898598s) [upload=278145]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 11:53:53.345433948 (315h49m51.896904419s) [upload=278145]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 11:53:54.603541081 (315h49m53.155011552s) [seg_pos={0 0}, upload=278145]: Begin upload segment...
Nov 22 11:53:54.603595445 (315h49m53.155065916s) [seg_pos={0 0}, upload=278145]: Done begin upload segment.
Nov 22 11:53:54.603668043 (315h49m53.155138494s) [noise=true , address=145.79.227.14:4837 , piece_id=TCYBUWS4YPUVLJ6QAPX2QIRT7O2HSBVEKVPPKHCTK336NNNPO7PQ, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.603684554 (315h49m53.155155005s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.604505828 (315h49m53.155976289s) [noise=true , address=45.91.116.11:4119 , piece_id=FGNP4Z5IL65VGJ7NMOJK5JW4Y644RFZG6MHWUOCLFKBJ26KF5W5A, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.604973038 (315h49m53.156443499s) [noise=true , address=145.79.227.28:4726 , piece_id=YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.605425900 (315h49m53.156896361s) [noise=true , address=145.79.227.4:4187 , piece_id=P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.606323339 (315h49m53.15779381s) [noise=true , address=77.90.31.15:4698 , piece_id=3Z7MYVQ62TX4BKJASUPTJTWDZH57PEVBSPT6OV2SMMZMTG2KCYZQ, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.606878326 (315h49m53.158348777s) [noise=true , address=145.79.227.31:4799 , piece_id=RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.607588267 (315h49m53.159058728s) [noise=true , address=145.79.227.25:4619 , piece_id=ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.610211763 (315h49m53.161682215s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.610336751 (315h49m53.161807212s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.612196994 (315h49m53.163667455s) [noise=true , address=145.79.227.9:4366 , piece_id=G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.612485593 (315h49m53.163956064s) [noise=true , address=145.79.227.21:4474 , piece_id=HYSRVRV3ALPITM7ZJ2GHPIGLTZ54YH4KYUZ36PTWAU6V2XBLURKQ, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.614299567 (315h49m53.165770028s) [noise=true , address=145.79.227.20:4438 , piece_id=NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.614454202 (315h49m53.165924673s) [noise=true , address=145.79.227.17:4969 , piece_id=I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.616397682 (315h49m53.167868143s) [noise=true , address=77.90.31.10:4517 , piece_id=KLXWC5RY274YKMLT2U5SBPCNQKIRXDOTPQ7MD76HXA3F2VCYRIEA, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.621720077 (315h49m53.173190538s) [noise=true , address=145.79.227.12:4767 , piece_id=FSJE5Q3Z7KHDGYUMIWRXJKTDOT5QFHZU7WQ4SHC3VQGGWJRNVEOQ, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:56.445932556 (315h49m54.997403007s) [upload=278145]: Next returned nil segment
Nov 22 11:53:56.445949048 (315h49m54.997419509s) [upload=278145]: Waiting for error group managing segments...
Nov 22 11:53:56.734588580 (315h49m55.286059041s) [noise=true , address=77.90.31.10:4517 , piece_id=KLXWC5RY274YKMLT2U5SBPCNQKIRXDOTPQ7MD76HXA3F2VCYRIEA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:56.796169791 (315h49m55.347640252s) [noise=true , address=145.79.227.20:4438 , piece_id=NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:56.800552848 (315h49m55.352023309s) [noise=true , address=45.91.116.11:4119 , piece_id=FGNP4Z5IL65VGJ7NMOJK5JW4Y644RFZG6MHWUOCLFKBJ26KF5W5A, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:56.802134119 (315h49m55.35360458s) [noise=true , address=145.79.227.4:4187 , piece_id=P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:56.811866140 (315h49m55.363336591s) [noise=true , address=145.79.227.17:4969 , piece_id=I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:56.820962632 (315h49m55.372433093s) [noise=true , address=77.90.31.15:4698 , piece_id=3Z7MYVQ62TX4BKJASUPTJTWDZH57PEVBSPT6OV2SMMZMTG2KCYZQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:56.853755886 (315h49m55.405226347s) [noise=true , address=145.79.227.21:4474 , piece_id=HYSRVRV3ALPITM7ZJ2GHPIGLTZ54YH4KYUZ36PTWAU6V2XBLURKQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:56.857215724 (315h49m55.408686185s) [noise=true , address=145.79.227.14:4837 , piece_id=TCYBUWS4YPUVLJ6QAPX2QIRT7O2HSBVEKVPPKHCTK336NNNPO7PQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:57.067648917 (315h49m55.619119378s) [noise=true , address=145.79.227.25:4619 , piece_id=ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:57.093474313 (315h49m55.644944774s) [noise=true , address=145.79.227.9:4366 , piece_id=G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:57.104509907 (315h49m55.655980378s) [noise=true , address=145.79.227.12:4767 , piece_id=FSJE5Q3Z7KHDGYUMIWRXJKTDOT5QFHZU7WQ4SHC3VQGGWJRNVEOQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:57.505088550 (315h49m56.056559011s) [noise=true , address=145.79.227.31:4799 , piece_id=RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:57.609615419 (315h49m56.16108589s) [noise=true , address=145.79.227.28:4726 , piece_id=YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:57.609676946 (315h49m56.161147437s) [seg_pos={0 0}, upload=278145]: Segment reached optimal threshold of 13 pieces.
Nov 22 11:53:57.609972077 (315h49m56.161442539s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW): protocol: write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection
Nov 22 11:53:57.609982889 (315h49m56.16145336s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold
Nov 22 11:53:57.610025941 (315h49m56.161496432s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk): protocol: expected piece hash; context canceled; EOF
Nov 22 11:53:57.610037101 (315h49m56.161507553s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold
Nov 22 11:53:57.610213978 (315h49m56.161684439s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19): protocol: write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection
Nov 22 11:53:57.610224649 (315h49m56.16169511s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold
Nov 22 11:53:57.610261058 (315h49m56.16173152s) [seg_pos={0 0}, upload=278145]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Nov 22 11:53:57.610421113 (315h49m56.161891584s) [upload=278145]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Nov 22 11:53:57.610430852 (315h49m56.161901313s) [upload=278145]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Nov 22 11:54:00.212174190 (315h49m58.763644651s) [upload=278145]: Done uploading segments...
Nov 22 11:54:00.212200729 (315h49m58.763671191s) [upload=278145]: Upload finished. err: <nil>
Duration: 5.684245867s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...............X............................................] duration:
5.688813574s trace:
https://jaeger.dev.storj.io/trace/76f1ecbff69b5d4e
FAIL (2025-11-22 12:23:08 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, Piece ID: I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ) context deadline exceeded; piecestore: (Node ID: 12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, Piece ID: YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A) context deadline exceeded; piecestore: (Node ID: 12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, Piece ID: P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ) context deadline exceeded; piecestore: (Node ID: 1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, Piece ID: ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA) context deadline exceeded; piecestore: (Node ID: 1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, Piece ID: G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ) context deadline exceeded; piecestore: (Node ID: 1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, Piece ID: NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA) context deadline exceeded
======LOGS======
nts...
Nov 22 11:53:53.345386809 (315h49m51.89685727s) [upload=278145]: Got next segment. Inline: false
Nov 22 11:53:53.345428137 (315h49m51.896898598s) [upload=278145]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 11:53:53.345433948 (315h49m51.896904419s) [upload=278145]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 11:53:54.603541081 (315h49m53.155011552s) [seg_pos={0 0}, upload=278145]: Begin upload segment...
Nov 22 11:53:54.603595445 (315h49m53.155065916s) [seg_pos={0 0}, upload=278145]: Done begin upload segment.
Nov 22 11:53:54.603668043 (315h49m53.155138494s) [noise=true , address=145.79.227.14:4837 , piece_id=TCYBUWS4YPUVLJ6QAPX2QIRT7O2HSBVEKVPPKHCTK336NNNPO7PQ, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.603684554 (315h49m53.155155005s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.604505828 (315h49m53.155976289s) [noise=true , address=45.91.116.11:4119 , piece_id=FGNP4Z5IL65VGJ7NMOJK5JW4Y644RFZG6MHWUOCLFKBJ26KF5W5A, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.604973038 (315h49m53.156443499s) [noise=true , address=145.79.227.28:4726 , piece_id=YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.605425900 (315h49m53.156896361s) [noise=true , address=145.79.227.4:4187 , piece_id=P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.606323339 (315h49m53.15779381s) [noise=true , address=77.90.31.15:4698 , piece_id=3Z7MYVQ62TX4BKJASUPTJTWDZH57PEVBSPT6OV2SMMZMTG2KCYZQ, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.606878326 (315h49m53.158348777s) [noise=true , address=145.79.227.31:4799 , piece_id=RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.607588267 (315h49m53.159058728s) [noise=true , address=145.79.227.25:4619 , piece_id=ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.610211763 (315h49m53.161682215s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.610336751 (315h49m53.161807212s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.612196994 (315h49m53.163667455s) [noise=true , address=145.79.227.9:4366 , piece_id=G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.612485593 (315h49m53.163956064s) [noise=true , address=145.79.227.21:4474 , piece_id=HYSRVRV3ALPITM7ZJ2GHPIGLTZ54YH4KYUZ36PTWAU6V2XBLURKQ, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.614299567 (315h49m53.165770028s) [noise=true , address=145.79.227.20:4438 , piece_id=NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.614454202 (315h49m53.165924673s) [noise=true , address=145.79.227.17:4969 , piece_id=I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.616397682 (315h49m53.167868143s) [noise=true , address=77.90.31.10:4517 , piece_id=KLXWC5RY274YKMLT2U5SBPCNQKIRXDOTPQ7MD76HXA3F2VCYRIEA, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.621720077 (315h49m53.173190538s) [noise=true , address=145.79.227.12:4767 , piece_id=FSJE5Q3Z7KHDGYUMIWRXJKTDOT5QFHZU7WQ4SHC3VQGGWJRNVEOQ, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:56.445932556 (315h49m54.997403007s) [upload=278145]: Next returned nil segment
Nov 22 11:53:56.445949048 (315h49m54.997419509s) [upload=278145]: Waiting for error group managing segments...
Nov 22 11:53:56.734588580 (315h49m55.286059041s) [noise=true , address=77.90.31.10:4517 , piece_id=KLXWC5RY274YKMLT2U5SBPCNQKIRXDOTPQ7MD76HXA3F2VCYRIEA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:56.796169791 (315h49m55.347640252s) [noise=true , address=145.79.227.20:4438 , piece_id=NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:56.800552848 (315h49m55.352023309s) [noise=true , address=45.91.116.11:4119 , piece_id=FGNP4Z5IL65VGJ7NMOJK5JW4Y644RFZG6MHWUOCLFKBJ26KF5W5A, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:56.802134119 (315h49m55.35360458s) [noise=true , address=145.79.227.4:4187 , piece_id=P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:56.811866140 (315h49m55.363336591s) [noise=true , address=145.79.227.17:4969 , piece_id=I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:56.820962632 (315h49m55.372433093s) [noise=true , address=77.90.31.15:4698 , piece_id=3Z7MYVQ62TX4BKJASUPTJTWDZH57PEVBSPT6OV2SMMZMTG2KCYZQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:56.853755886 (315h49m55.405226347s) [noise=true , address=145.79.227.21:4474 , piece_id=HYSRVRV3ALPITM7ZJ2GHPIGLTZ54YH4KYUZ36PTWAU6V2XBLURKQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:56.857215724 (315h49m55.408686185s) [noise=true , address=145.79.227.14:4837 , piece_id=TCYBUWS4YPUVLJ6QAPX2QIRT7O2HSBVEKVPPKHCTK336NNNPO7PQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:57.067648917 (315h49m55.619119378s) [noise=true , address=145.79.227.25:4619 , piece_id=ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:57.093474313 (315h49m55.644944774s) [noise=true , address=145.79.227.9:4366 , piece_id=G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:57.104509907 (315h49m55.655980378s) [noise=true , address=145.79.227.12:4767 , piece_id=FSJE5Q3Z7KHDGYUMIWRXJKTDOT5QFHZU7WQ4SHC3VQGGWJRNVEOQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:57.505088550 (315h49m56.056559011s) [noise=true , address=145.79.227.31:4799 , piece_id=RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:57.609615419 (315h49m56.16108589s) [noise=true , address=145.79.227.28:4726 , piece_id=YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:57.609676946 (315h49m56.161147437s) [seg_pos={0 0}, upload=278145]: Segment reached optimal threshold of 13 pieces.
Nov 22 11:53:57.609972077 (315h49m56.161442539s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW): protocol: write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection
Nov 22 11:53:57.609982889 (315h49m56.16145336s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold
Nov 22 11:53:57.610025941 (315h49m56.161496432s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk): protocol: expected piece hash; context canceled; EOF
Nov 22 11:53:57.610037101 (315h49m56.161507553s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold
Nov 22 11:53:57.610213978 (315h49m56.161684439s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19): protocol: write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection
Nov 22 11:53:57.610224649 (315h49m56.16169511s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold
Nov 22 11:53:57.610261058 (315h49m56.16173152s) [seg_pos={0 0}, upload=278145]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Nov 22 11:53:57.610421113 (315h49m56.161891584s) [upload=278145]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Nov 22 11:53:57.610430852 (315h49m56.161901313s) [upload=278145]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Nov 22 11:54:00.212174190 (315h49m58.763644651s) [upload=278145]: Done uploading segments...
Nov 22 11:54:00.212200729 (315h49m58.763671191s) [upload=278145]: Upload finished. err: <nil>
Duration: 5.688839414s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [................X...........................................] duration:
5.859087259s trace:
https://jaeger.dev.storj.io/trace/76f1ecbff69b5d4e
FAIL (2025-11-22 12:22:41 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, Piece ID: I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ) context deadline exceeded; piecestore: (Node ID: 12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, Piece ID: YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A) context deadline exceeded; piecestore: (Node ID: 12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, Piece ID: P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ) context deadline exceeded; piecestore: (Node ID: 1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, Piece ID: ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA) context deadline exceeded; piecestore: (Node ID: 1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, Piece ID: G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ) context deadline exceeded; piecestore: (Node ID: 1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, Piece ID: NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA) context deadline exceeded
======LOGS======
nts...
Nov 22 11:53:53.345386809 (315h49m51.89685727s) [upload=278145]: Got next segment. Inline: false
Nov 22 11:53:53.345428137 (315h49m51.896898598s) [upload=278145]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 11:53:53.345433948 (315h49m51.896904419s) [upload=278145]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 11:53:54.603541081 (315h49m53.155011552s) [seg_pos={0 0}, upload=278145]: Begin upload segment...
Nov 22 11:53:54.603595445 (315h49m53.155065916s) [seg_pos={0 0}, upload=278145]: Done begin upload segment.
Nov 22 11:53:54.603668043 (315h49m53.155138494s) [noise=true , address=145.79.227.14:4837 , piece_id=TCYBUWS4YPUVLJ6QAPX2QIRT7O2HSBVEKVPPKHCTK336NNNPO7PQ, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.603684554 (315h49m53.155155005s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.604505828 (315h49m53.155976289s) [noise=true , address=45.91.116.11:4119 , piece_id=FGNP4Z5IL65VGJ7NMOJK5JW4Y644RFZG6MHWUOCLFKBJ26KF5W5A, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.604973038 (315h49m53.156443499s) [noise=true , address=145.79.227.28:4726 , piece_id=YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.605425900 (315h49m53.156896361s) [noise=true , address=145.79.227.4:4187 , piece_id=P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.606323339 (315h49m53.15779381s) [noise=true , address=77.90.31.15:4698 , piece_id=3Z7MYVQ62TX4BKJASUPTJTWDZH57PEVBSPT6OV2SMMZMTG2KCYZQ, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.606878326 (315h49m53.158348777s) [noise=true , address=145.79.227.31:4799 , piece_id=RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.607588267 (315h49m53.159058728s) [noise=true , address=145.79.227.25:4619 , piece_id=ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.610211763 (315h49m53.161682215s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.610336751 (315h49m53.161807212s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.612196994 (315h49m53.163667455s) [noise=true , address=145.79.227.9:4366 , piece_id=G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.612485593 (315h49m53.163956064s) [noise=true , address=145.79.227.21:4474 , piece_id=HYSRVRV3ALPITM7ZJ2GHPIGLTZ54YH4KYUZ36PTWAU6V2XBLURKQ, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.614299567 (315h49m53.165770028s) [noise=true , address=145.79.227.20:4438 , piece_id=NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.614454202 (315h49m53.165924673s) [noise=true , address=145.79.227.17:4969 , piece_id=I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.616397682 (315h49m53.167868143s) [noise=true , address=77.90.31.10:4517 , piece_id=KLXWC5RY274YKMLT2U5SBPCNQKIRXDOTPQ7MD76HXA3F2VCYRIEA, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:54.621720077 (315h49m53.173190538s) [noise=true , address=145.79.227.12:4767 , piece_id=FSJE5Q3Z7KHDGYUMIWRXJKTDOT5QFHZU7WQ4SHC3VQGGWJRNVEOQ, seg_pos={0 0}, upload=278145]: Uploading piece...
Nov 22 11:53:56.445932556 (315h49m54.997403007s) [upload=278145]: Next returned nil segment
Nov 22 11:53:56.445949048 (315h49m54.997419509s) [upload=278145]: Waiting for error group managing segments...
Nov 22 11:53:56.734588580 (315h49m55.286059041s) [noise=true , address=77.90.31.10:4517 , piece_id=KLXWC5RY274YKMLT2U5SBPCNQKIRXDOTPQ7MD76HXA3F2VCYRIEA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:56.796169791 (315h49m55.347640252s) [noise=true , address=145.79.227.20:4438 , piece_id=NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:56.800552848 (315h49m55.352023309s) [noise=true , address=45.91.116.11:4119 , piece_id=FGNP4Z5IL65VGJ7NMOJK5JW4Y644RFZG6MHWUOCLFKBJ26KF5W5A, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:56.802134119 (315h49m55.35360458s) [noise=true , address=145.79.227.4:4187 , piece_id=P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:56.811866140 (315h49m55.363336591s) [noise=true , address=145.79.227.17:4969 , piece_id=I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:56.820962632 (315h49m55.372433093s) [noise=true , address=77.90.31.15:4698 , piece_id=3Z7MYVQ62TX4BKJASUPTJTWDZH57PEVBSPT6OV2SMMZMTG2KCYZQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:56.853755886 (315h49m55.405226347s) [noise=true , address=145.79.227.21:4474 , piece_id=HYSRVRV3ALPITM7ZJ2GHPIGLTZ54YH4KYUZ36PTWAU6V2XBLURKQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:56.857215724 (315h49m55.408686185s) [noise=true , address=145.79.227.14:4837 , piece_id=TCYBUWS4YPUVLJ6QAPX2QIRT7O2HSBVEKVPPKHCTK336NNNPO7PQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:57.067648917 (315h49m55.619119378s) [noise=true , address=145.79.227.25:4619 , piece_id=ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:57.093474313 (315h49m55.644944774s) [noise=true , address=145.79.227.9:4366 , piece_id=G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:57.104509907 (315h49m55.655980378s) [noise=true , address=145.79.227.12:4767 , piece_id=FSJE5Q3Z7KHDGYUMIWRXJKTDOT5QFHZU7WQ4SHC3VQGGWJRNVEOQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:57.505088550 (315h49m56.056559011s) [noise=true , address=145.79.227.31:4799 , piece_id=RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:57.609615419 (315h49m56.16108589s) [noise=true , address=145.79.227.28:4726 , piece_id=YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil>
Nov 22 11:53:57.609676946 (315h49m56.161147437s) [seg_pos={0 0}, upload=278145]: Segment reached optimal threshold of 13 pieces.
Nov 22 11:53:57.609972077 (315h49m56.161442539s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW): protocol: write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection
Nov 22 11:53:57.609982889 (315h49m56.16145336s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold
Nov 22 11:53:57.610025941 (315h49m56.161496432s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk): protocol: expected piece hash; context canceled; EOF
Nov 22 11:53:57.610037101 (315h49m56.161507553s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold
Nov 22 11:53:57.610213978 (315h49m56.161684439s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19): protocol: write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection
Nov 22 11:53:57.610224649 (315h49m56.16169511s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold
Nov 22 11:53:57.610261058 (315h49m56.16173152s) [seg_pos={0 0}, upload=278145]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Nov 22 11:53:57.610421113 (315h49m56.161891584s) [upload=278145]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Nov 22 11:53:57.610430852 (315h49m56.161901313s) [upload=278145]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Nov 22 11:54:00.212174190 (315h49m58.763644651s) [upload=278145]: Done uploading segments...
Nov 22 11:54:00.212200729 (315h49m58.763671191s) [upload=278145]: Upload finished. err: <nil>
Duration: 5.859116464s
core
uplink
us-select-1
|