Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [.................................X........X........X........] duration:
25.383286062s trace:
https://jaeger.dev.storj.io/trace/621f2bd1cae766d0
FAIL (2026-01-07 07:24:41 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
927s) [noise=true , address=77.90.31.15:4727 , piece_id=UFRMVFFY2UKQLPR7P4TTN2UFNFZQ6BDPUP7INPYRJIK267R6WPTQ, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.304384464 (783h48m48.175467499s) [noise=true , address=145.79.227.18:4017 , piece_id=XEDPCH5DMCZIQUDMILRNT4NMMVSH67CDSFA2LXLZX3SJNC7K64JQ, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.304648907 (783h48m48.175731942s) [noise=true , address=145.79.227.43:4180 , piece_id=3ZAN75QUFNRD4K5QYAOZAYP2O4KK4SVZPXPTDOA7IYVIGLLEPYSQ, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.304909002 (783h48m48.175992037s) [noise=true , address=145.79.227.35:4484 , piece_id=6ZQ2M55FDBKKZAGXNN7GPGZCXPKJWRGPKDL5XQWBWREGXKRSCLBQ, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.305186050 (783h48m48.176269085s) [noise=true , address=64.49.13.169:4713 , piece_id=4X6L55C3YY3Y35RKP2LX7BVPJME7L5JWELQ2XK2E752NT7E6KCWA, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.305462475 (783h48m48.17654552s) [noise=true , address=145.79.227.41:4109 , piece_id=F62DIMJUKTLEX3TO4D6FT2PGGKOCDKYY7US6QZGW7PBGNXMLQRFQ, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.306063991 (783h48m48.177147037s) [noise=true , address=145.79.227.2:4127 , piece_id=J4JYZ2L5ULSHBLFH3N2FUKXBCDWEPVPNNNEBUW4LA3QAJYEGCBLA, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.306457621 (783h48m48.177540666s) [noise=true , address=145.79.227.49:4370 , piece_id=G3ATR7KHT5DDCJNAEZDDI55XTPPZMFDYKQLAR3AD45P3GCESOHGA, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.306836312 (783h48m48.177919357s) [noise=true , address=145.79.227.33:4888 , piece_id=NDWJJUO4V4UBYYWJZJM4FQGVUDR24WTCHABZJMUGZ4FMH3U6ZK7Q, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.307182321 (783h48m48.178265366s) [noise=true , address=145.79.227.30:4777 , piece_id=BBHGIQO2AEMZWMUOGMRJC4CKRWJOTEY3WQP4I4XRGMNRIOUOH5IQ, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.307621457 (783h48m48.178704503s) [noise=true , address=145.79.227.34:4931 , piece_id=QFHCIN7ZZBYM3CFF6UVA73EFMWRKJBOWX6KBFZC2SHZ5KN3WU34Q, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.307948189 (783h48m48.179031234s) [noise=true , address=77.90.31.21:4909 , piece_id=7KH356VERAYPY7GHMEOT34TADWH4U74L2JUPMQ6Y7WF52L5I6M6A, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.308347098 (783h48m48.179430133s) [noise=true , address=77.90.31.18:4830 , piece_id=TUICFKVMJ3S2KHW7MKFNJUR6Z5UN5QFZOBXJEYC43O3BE3QQDWHA, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.308510059 (783h48m48.179593104s) [noise=true , address=77.90.31.12:4619 , piece_id=DVOJATZI5UILEARIOOZ2K3WGDFFQTFONCK2HKP2ZXKLBAEQZLROQ, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.309050468 (783h48m48.180133514s) [noise=true , address=145.79.227.44:4198 , piece_id=4KNZZKJ5V3SOLZKPJYY6FSCG4SIAF67RDUWRQSXPAMDY5IO32YOA, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.309517858 (783h48m48.180600903s) [noise=true , address=64.49.13.155:4920 , piece_id=WMM6GIC2AJJHN2BOT7PLSKWVG7ONHG3CL3OF3PJQ5DH5ODFO24ZA, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:49.600689336 (783h48m49.471772381s) [upload=685644]: Next returned nil segment
Jan 7 07:17:49.600703433 (783h48m49.471786478s) [upload=685644]: Waiting for error group managing segments...
Jan 7 07:17:50.063713792 (783h48m49.934796847s) [noise=true , address=64.49.13.169:4713 , piece_id=4X6L55C3YY3Y35RKP2LX7BVPJME7L5JWELQ2XK2E752NT7E6KCWA, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.116339850 (783h48m49.987422895s) [noise=true , address=64.49.13.155:4920 , piece_id=WMM6GIC2AJJHN2BOT7PLSKWVG7ONHG3CL3OF3PJQ5DH5ODFO24ZA, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.154469825 (783h48m50.02555287s) [noise=true , address=145.79.227.32:4847 , piece_id=73A46W74UZTHTZBW6G35LSJZIA3ZRVFRRCC3KGJYGWZZH6NLICGA, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.158394037 (783h48m50.029477072s) [noise=true , address=145.79.227.44:4198 , piece_id=4KNZZKJ5V3SOLZKPJYY6FSCG4SIAF67RDUWRQSXPAMDY5IO32YOA, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.164547664 (783h48m50.035630709s) [noise=true , address=145.79.227.18:4017 , piece_id=XEDPCH5DMCZIQUDMILRNT4NMMVSH67CDSFA2LXLZX3SJNC7K64JQ, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.173751741 (783h48m50.044834786s) [noise=true , address=145.79.227.41:4109 , piece_id=F62DIMJUKTLEX3TO4D6FT2PGGKOCDKYY7US6QZGW7PBGNXMLQRFQ, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.181867575 (783h48m50.05295061s) [noise=true , address=145.79.227.34:4931 , piece_id=QFHCIN7ZZBYM3CFF6UVA73EFMWRKJBOWX6KBFZC2SHZ5KN3WU34Q, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.190204078 (783h48m50.061287113s) [noise=true , address=145.79.227.2:4127 , piece_id=J4JYZ2L5ULSHBLFH3N2FUKXBCDWEPVPNNNEBUW4LA3QAJYEGCBLA, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.215263345 (783h48m50.08634638s) [noise=true , address=145.79.227.33:4888 , piece_id=NDWJJUO4V4UBYYWJZJM4FQGVUDR24WTCHABZJMUGZ4FMH3U6ZK7Q, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.244106597 (783h48m50.115189632s) [noise=true , address=145.79.227.30:4777 , piece_id=BBHGIQO2AEMZWMUOGMRJC4CKRWJOTEY3WQP4I4XRGMNRIOUOH5IQ, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.468914424 (783h48m50.339997469s) [noise=true , address=77.90.31.18:4830 , piece_id=TUICFKVMJ3S2KHW7MKFNJUR6Z5UN5QFZOBXJEYC43O3BE3QQDWHA, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.533387670 (783h48m50.404470705s) [noise=true , address=77.90.31.12:4619 , piece_id=DVOJATZI5UILEARIOOZ2K3WGDFFQTFONCK2HKP2ZXKLBAEQZLROQ, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.560320373 (783h48m50.431403408s) [noise=true , address=77.90.31.11:4579 , piece_id=5LUJ6FFYUVXAD2A6DNE5IOFT22VZL4ZJ4S2NRR2MCA6GX5H3P4EQ, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.604949694 (783h48m50.476032739s) [noise=true , address=77.90.31.21:4909 , piece_id=7KH356VERAYPY7GHMEOT34TADWH4U74L2JUPMQ6Y7WF52L5I6M6A, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.604977047 (783h48m50.476060082s) [seg_pos={0 0}, upload=685644]: Segment reached optimal threshold of 14 pieces.
Jan 7 07:17:50.605285644 (783h48m50.476368689s) [noise=true , address=77.90.31.15:4727 , piece_id=UFRMVFFY2UKQLPR7P4TTN2UFNFZQ6BDPUP7INPYRJIK267R6WPTQ, seg_pos={0 0}, upload=685644]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xoAbv6zJuG9LjyCdsCrTWQaCDyLrWyv8RCrMTiqNqazoEwD1M): protocol: write tcp 188.166.225.170:47856->77.90.31.15:4727: use of closed network connection; write tcp 188.166.225.170:47856->77.90.31.15:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47856->77.90.31.15:4727: use of closed network connection
Jan 7 07:17:50.605295753 (783h48m50.476378808s) [noise=true , address=77.90.31.15:4727 , piece_id=UFRMVFFY2UKQLPR7P4TTN2UFNFZQ6BDPUP7INPYRJIK267R6WPTQ, seg_pos={0 0}, upload=685644]: Long tail cancellation due to optimal threshold
Jan 7 07:17:50.605401254 (783h48m50.476484299s) [noise=true , address=145.79.227.49:4370 , piece_id=G3ATR7KHT5DDCJNAEZDDI55XTPPZMFDYKQLAR3AD45P3GCESOHGA, seg_pos={0 0}, upload=685644]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LBgixGGgNeYs63ZSpkYJmrEGr3W9Ma1s2Sj8cCxU6mUxm1aC): protocol: expected piece hash; context canceled; EOF
Jan 7 07:17:50.605437523 (783h48m50.476520568s) [noise=true , address=145.79.227.49:4370 , piece_id=G3ATR7KHT5DDCJNAEZDDI55XTPPZMFDYKQLAR3AD45P3GCESOHGA, seg_pos={0 0}, upload=685644]: Long tail cancellation due to optimal threshold
Jan 7 07:17:50.605839750 (783h48m50.476922825s) [noise=true , address=145.79.227.43:4180 , piece_id=3ZAN75QUFNRD4K5QYAOZAYP2O4KK4SVZPXPTDOA7IYVIGLLEPYSQ, seg_pos={0 0}, upload=685644]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rhkcipPBZNFScBjBhWdigzSMwENSmzhCucnC2iuyc9udpweE3): protocol: expected piece hash; context canceled; EOF
Jan 7 07:17:50.605882010 (783h48m50.476965045s) [noise=true , address=145.79.227.43:4180 , piece_id=3ZAN75QUFNRD4K5QYAOZAYP2O4KK4SVZPXPTDOA7IYVIGLLEPYSQ, seg_pos={0 0}, upload=685644]: Long tail cancellation due to optimal threshold
Jan 7 07:17:50.605904593 (783h48m50.476987628s) [noise=true , address=145.79.227.35:4484 , piece_id=6ZQ2M55FDBKKZAGXNN7GPGZCXPKJWRGPKDL5XQWBWREGXKRSCLBQ, seg_pos={0 0}, upload=685644]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1a5S5Ar7jNbK9MESghbWfmEgBPNSMN2tEnZeeG1kKzeX2dnSX3): protocol: expected piece hash; context canceled; EOF
Jan 7 07:17:50.605924150 (783h48m50.477007205s) [noise=true , address=145.79.227.35:4484 , piece_id=6ZQ2M55FDBKKZAGXNN7GPGZCXPKJWRGPKDL5XQWBWREGXKRSCLBQ, seg_pos={0 0}, upload=685644]: Long tail cancellation due to optimal threshold
Jan 7 07:17:50.605947465 (783h48m50.47703051s) [seg_pos={0 0}, upload=685644]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jan 7 07:17:50.606035051 (783h48m50.477118086s) [upload=685644]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 7 07:17:50.606040612 (783h48m50.477123647s) [upload=685644]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 7 07:17:50.908947210 (783h48m50.780030255s) [upload=685644]: Done uploading segments...
Jan 7 07:17:50.908965275 (783h48m50.78004831s) [upload=685644]: Upload finished. err: <nil>
Duration: 25.383317001s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [...X...............X...................X.............X......] duration:
25.75087597s trace:
https://jaeger.dev.storj.io/trace/4389ca011670ab35
FAIL (2026-01-07 07:24:37 UTC)
orbiter warning: context deadline exceeded
Duration: 25.750904135s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
50 out of 60 is failed (threshold=6) [XXXXXXXXXXX.XXXX.XXXX..XXXXX.XXXXX..XXXXX.XXXX.XXXXXXXXX.XXX] duration:
5.001092286s trace:
https://jaeger.dev.storj.io/trace/301591b471dc3ee0
FAIL (2026-01-07 07:24:32 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.00110532s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
13 out of 60 is failed (threshold=3) [.X..X....X.X.....X....X.X.....X.......X....XX.........X....X] duration:
30.001121505s trace:
https://jaeger.dev.storj.io/trace/2959ec9bac4b8157
FAIL (2026-01-07 07:24:28 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.001132446s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [....X...............X...................X.............X.....] duration:
16.569752779s trace:
https://jaeger.dev.storj.io/trace/4389ca011670ab35
FAIL (2026-01-07 07:24:07 UTC)
orbiter warning: context deadline exceeded
Duration: 16.569782927s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [..................................X........X........X.......] duration:
15.404921368s trace:
https://jaeger.dev.storj.io/trace/621f2bd1cae766d0
FAIL (2026-01-07 07:24:00 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
927s) [noise=true , address=77.90.31.15:4727 , piece_id=UFRMVFFY2UKQLPR7P4TTN2UFNFZQ6BDPUP7INPYRJIK267R6WPTQ, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.304384464 (783h48m48.175467499s) [noise=true , address=145.79.227.18:4017 , piece_id=XEDPCH5DMCZIQUDMILRNT4NMMVSH67CDSFA2LXLZX3SJNC7K64JQ, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.304648907 (783h48m48.175731942s) [noise=true , address=145.79.227.43:4180 , piece_id=3ZAN75QUFNRD4K5QYAOZAYP2O4KK4SVZPXPTDOA7IYVIGLLEPYSQ, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.304909002 (783h48m48.175992037s) [noise=true , address=145.79.227.35:4484 , piece_id=6ZQ2M55FDBKKZAGXNN7GPGZCXPKJWRGPKDL5XQWBWREGXKRSCLBQ, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.305186050 (783h48m48.176269085s) [noise=true , address=64.49.13.169:4713 , piece_id=4X6L55C3YY3Y35RKP2LX7BVPJME7L5JWELQ2XK2E752NT7E6KCWA, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.305462475 (783h48m48.17654552s) [noise=true , address=145.79.227.41:4109 , piece_id=F62DIMJUKTLEX3TO4D6FT2PGGKOCDKYY7US6QZGW7PBGNXMLQRFQ, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.306063991 (783h48m48.177147037s) [noise=true , address=145.79.227.2:4127 , piece_id=J4JYZ2L5ULSHBLFH3N2FUKXBCDWEPVPNNNEBUW4LA3QAJYEGCBLA, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.306457621 (783h48m48.177540666s) [noise=true , address=145.79.227.49:4370 , piece_id=G3ATR7KHT5DDCJNAEZDDI55XTPPZMFDYKQLAR3AD45P3GCESOHGA, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.306836312 (783h48m48.177919357s) [noise=true , address=145.79.227.33:4888 , piece_id=NDWJJUO4V4UBYYWJZJM4FQGVUDR24WTCHABZJMUGZ4FMH3U6ZK7Q, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.307182321 (783h48m48.178265366s) [noise=true , address=145.79.227.30:4777 , piece_id=BBHGIQO2AEMZWMUOGMRJC4CKRWJOTEY3WQP4I4XRGMNRIOUOH5IQ, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.307621457 (783h48m48.178704503s) [noise=true , address=145.79.227.34:4931 , piece_id=QFHCIN7ZZBYM3CFF6UVA73EFMWRKJBOWX6KBFZC2SHZ5KN3WU34Q, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.307948189 (783h48m48.179031234s) [noise=true , address=77.90.31.21:4909 , piece_id=7KH356VERAYPY7GHMEOT34TADWH4U74L2JUPMQ6Y7WF52L5I6M6A, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.308347098 (783h48m48.179430133s) [noise=true , address=77.90.31.18:4830 , piece_id=TUICFKVMJ3S2KHW7MKFNJUR6Z5UN5QFZOBXJEYC43O3BE3QQDWHA, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.308510059 (783h48m48.179593104s) [noise=true , address=77.90.31.12:4619 , piece_id=DVOJATZI5UILEARIOOZ2K3WGDFFQTFONCK2HKP2ZXKLBAEQZLROQ, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.309050468 (783h48m48.180133514s) [noise=true , address=145.79.227.44:4198 , piece_id=4KNZZKJ5V3SOLZKPJYY6FSCG4SIAF67RDUWRQSXPAMDY5IO32YOA, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.309517858 (783h48m48.180600903s) [noise=true , address=64.49.13.155:4920 , piece_id=WMM6GIC2AJJHN2BOT7PLSKWVG7ONHG3CL3OF3PJQ5DH5ODFO24ZA, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:49.600689336 (783h48m49.471772381s) [upload=685644]: Next returned nil segment
Jan 7 07:17:49.600703433 (783h48m49.471786478s) [upload=685644]: Waiting for error group managing segments...
Jan 7 07:17:50.063713792 (783h48m49.934796847s) [noise=true , address=64.49.13.169:4713 , piece_id=4X6L55C3YY3Y35RKP2LX7BVPJME7L5JWELQ2XK2E752NT7E6KCWA, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.116339850 (783h48m49.987422895s) [noise=true , address=64.49.13.155:4920 , piece_id=WMM6GIC2AJJHN2BOT7PLSKWVG7ONHG3CL3OF3PJQ5DH5ODFO24ZA, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.154469825 (783h48m50.02555287s) [noise=true , address=145.79.227.32:4847 , piece_id=73A46W74UZTHTZBW6G35LSJZIA3ZRVFRRCC3KGJYGWZZH6NLICGA, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.158394037 (783h48m50.029477072s) [noise=true , address=145.79.227.44:4198 , piece_id=4KNZZKJ5V3SOLZKPJYY6FSCG4SIAF67RDUWRQSXPAMDY5IO32YOA, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.164547664 (783h48m50.035630709s) [noise=true , address=145.79.227.18:4017 , piece_id=XEDPCH5DMCZIQUDMILRNT4NMMVSH67CDSFA2LXLZX3SJNC7K64JQ, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.173751741 (783h48m50.044834786s) [noise=true , address=145.79.227.41:4109 , piece_id=F62DIMJUKTLEX3TO4D6FT2PGGKOCDKYY7US6QZGW7PBGNXMLQRFQ, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.181867575 (783h48m50.05295061s) [noise=true , address=145.79.227.34:4931 , piece_id=QFHCIN7ZZBYM3CFF6UVA73EFMWRKJBOWX6KBFZC2SHZ5KN3WU34Q, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.190204078 (783h48m50.061287113s) [noise=true , address=145.79.227.2:4127 , piece_id=J4JYZ2L5ULSHBLFH3N2FUKXBCDWEPVPNNNEBUW4LA3QAJYEGCBLA, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.215263345 (783h48m50.08634638s) [noise=true , address=145.79.227.33:4888 , piece_id=NDWJJUO4V4UBYYWJZJM4FQGVUDR24WTCHABZJMUGZ4FMH3U6ZK7Q, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.244106597 (783h48m50.115189632s) [noise=true , address=145.79.227.30:4777 , piece_id=BBHGIQO2AEMZWMUOGMRJC4CKRWJOTEY3WQP4I4XRGMNRIOUOH5IQ, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.468914424 (783h48m50.339997469s) [noise=true , address=77.90.31.18:4830 , piece_id=TUICFKVMJ3S2KHW7MKFNJUR6Z5UN5QFZOBXJEYC43O3BE3QQDWHA, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.533387670 (783h48m50.404470705s) [noise=true , address=77.90.31.12:4619 , piece_id=DVOJATZI5UILEARIOOZ2K3WGDFFQTFONCK2HKP2ZXKLBAEQZLROQ, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.560320373 (783h48m50.431403408s) [noise=true , address=77.90.31.11:4579 , piece_id=5LUJ6FFYUVXAD2A6DNE5IOFT22VZL4ZJ4S2NRR2MCA6GX5H3P4EQ, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.604949694 (783h48m50.476032739s) [noise=true , address=77.90.31.21:4909 , piece_id=7KH356VERAYPY7GHMEOT34TADWH4U74L2JUPMQ6Y7WF52L5I6M6A, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.604977047 (783h48m50.476060082s) [seg_pos={0 0}, upload=685644]: Segment reached optimal threshold of 14 pieces.
Jan 7 07:17:50.605285644 (783h48m50.476368689s) [noise=true , address=77.90.31.15:4727 , piece_id=UFRMVFFY2UKQLPR7P4TTN2UFNFZQ6BDPUP7INPYRJIK267R6WPTQ, seg_pos={0 0}, upload=685644]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xoAbv6zJuG9LjyCdsCrTWQaCDyLrWyv8RCrMTiqNqazoEwD1M): protocol: write tcp 188.166.225.170:47856->77.90.31.15:4727: use of closed network connection; write tcp 188.166.225.170:47856->77.90.31.15:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47856->77.90.31.15:4727: use of closed network connection
Jan 7 07:17:50.605295753 (783h48m50.476378808s) [noise=true , address=77.90.31.15:4727 , piece_id=UFRMVFFY2UKQLPR7P4TTN2UFNFZQ6BDPUP7INPYRJIK267R6WPTQ, seg_pos={0 0}, upload=685644]: Long tail cancellation due to optimal threshold
Jan 7 07:17:50.605401254 (783h48m50.476484299s) [noise=true , address=145.79.227.49:4370 , piece_id=G3ATR7KHT5DDCJNAEZDDI55XTPPZMFDYKQLAR3AD45P3GCESOHGA, seg_pos={0 0}, upload=685644]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LBgixGGgNeYs63ZSpkYJmrEGr3W9Ma1s2Sj8cCxU6mUxm1aC): protocol: expected piece hash; context canceled; EOF
Jan 7 07:17:50.605437523 (783h48m50.476520568s) [noise=true , address=145.79.227.49:4370 , piece_id=G3ATR7KHT5DDCJNAEZDDI55XTPPZMFDYKQLAR3AD45P3GCESOHGA, seg_pos={0 0}, upload=685644]: Long tail cancellation due to optimal threshold
Jan 7 07:17:50.605839750 (783h48m50.476922825s) [noise=true , address=145.79.227.43:4180 , piece_id=3ZAN75QUFNRD4K5QYAOZAYP2O4KK4SVZPXPTDOA7IYVIGLLEPYSQ, seg_pos={0 0}, upload=685644]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rhkcipPBZNFScBjBhWdigzSMwENSmzhCucnC2iuyc9udpweE3): protocol: expected piece hash; context canceled; EOF
Jan 7 07:17:50.605882010 (783h48m50.476965045s) [noise=true , address=145.79.227.43:4180 , piece_id=3ZAN75QUFNRD4K5QYAOZAYP2O4KK4SVZPXPTDOA7IYVIGLLEPYSQ, seg_pos={0 0}, upload=685644]: Long tail cancellation due to optimal threshold
Jan 7 07:17:50.605904593 (783h48m50.476987628s) [noise=true , address=145.79.227.35:4484 , piece_id=6ZQ2M55FDBKKZAGXNN7GPGZCXPKJWRGPKDL5XQWBWREGXKRSCLBQ, seg_pos={0 0}, upload=685644]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1a5S5Ar7jNbK9MESghbWfmEgBPNSMN2tEnZeeG1kKzeX2dnSX3): protocol: expected piece hash; context canceled; EOF
Jan 7 07:17:50.605924150 (783h48m50.477007205s) [noise=true , address=145.79.227.35:4484 , piece_id=6ZQ2M55FDBKKZAGXNN7GPGZCXPKJWRGPKDL5XQWBWREGXKRSCLBQ, seg_pos={0 0}, upload=685644]: Long tail cancellation due to optimal threshold
Jan 7 07:17:50.605947465 (783h48m50.47703051s) [seg_pos={0 0}, upload=685644]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jan 7 07:17:50.606035051 (783h48m50.477118086s) [upload=685644]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 7 07:17:50.606040612 (783h48m50.477123647s) [upload=685644]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 7 07:17:50.908947210 (783h48m50.780030255s) [upload=685644]: Done uploading segments...
Jan 7 07:17:50.908965275 (783h48m50.78004831s) [upload=685644]: Upload finished. err: <nil>
Duration: 15.404948089s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
13 out of 60 is failed (threshold=3) [X.X..X....X.X.....X....X.X.....X.......X....XX.........X....] duration:
25.372038126s trace:
https://jaeger.dev.storj.io/trace/2cf2060b793525bb
FAIL (2026-01-07 07:23:58 UTC)
orbiter warning: context deadline exceeded
Duration: 25.372068032s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
12 out of 60 is failed (threshold=3) [..X.........X..X.X..X.......XX.X....X....XX....X............] duration:
5.953336769s trace:
https://jaeger.dev.storj.io/trace/566429788081faee
FAIL (2026-01-07 07:23:40 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
) [upload=685582]: Uploading segments...
Jan 7 07:13:32.990108051 (783h44m32.861191096s) [upload=685582]: Got next segment. Inline: false
Jan 7 07:13:32.990479758 (783h44m32.861562803s) [upload=685582]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 7 07:13:32.990486191 (783h44m32.861569226s) [upload=685582]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 7 07:13:33.230730672 (783h44m33.101813717s) [seg_pos={0 0}, upload=685582]: Begin upload segment...
Jan 7 07:13:33.230783873 (783h44m33.101866908s) [seg_pos={0 0}, upload=685582]: Done begin upload segment.
Jan 7 07:13:33.230814070 (783h44m33.101897115s) [noise=true , address=45.91.116.13:4153 , piece_id=27PHIESDJJJKDKJITJODFYQLSF42BX3BVTMOXFJAUGK73D3Z47UA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.230992941 (783h44m33.102075976s) [noise=true , address=45.91.116.5:4300 , piece_id=5FMCKFKIRBUKZGZTTBKCMURBHZTBKQPHEZMHJWJHLM2PLEDFNXSA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.231512250 (783h44m33.102595285s) [noise=true , address=145.79.227.22:4511 , piece_id=VHGXFDPNKXDZI2ARWI7227A5CF3P572VJSQZS75BMMHPAVCPPO7Q, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.231639272 (783h44m33.102722307s) [noise=true , address=145.79.227.4:4185 , piece_id=SMLAUFNVX6M24OXQ2PK2TIP5MXF3KZYRGHBFXYN7OXD5L76PCY3Q, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.231935536 (783h44m33.103018571s) [noise=true , address=145.79.227.31:4798 , piece_id=TKL7NRLI7DYTXU4HEOBJHXQR5CGHU7IG53L2GLENDCORGVVN2KOQ, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.232203125 (783h44m33.10328616s) [noise=true , address=145.79.227.38:4978 , piece_id=QV3BBDP5GVUMTBI2HIE7FWUIUOQVKIOO2D6OJNW2VDCXDFKMISUA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.232580194 (783h44m33.103663229s) [noise=true , address=145.79.227.52:4445 , piece_id=D6QKUQIBGUGV3V2ZELIM6U5UMTHT2765CHQBIKQ2N3BJF3UXH3XA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.232942814 (783h44m33.104025849s) [noise=true , address=45.91.116.8:4047 , piece_id=D6ZXOD5IBAYIVGUWNC6OGIL3OTB7JF6OAFGNGIR2RH7NDWRAEORA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.233250680 (783h44m33.104333725s) [noise=true , address=145.79.227.40:4049 , piece_id=RV43PFYU6UWU5QFHCO7T6F3E3VBNEYQRMA4UCPXT54PP3T3ZD62Q, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.233527197 (783h44m33.104610242s) [noise=true , address=77.90.31.16:4736 , piece_id=6I7NM23KI67OH2RKW4NSGYGJYWZXMH5HVLHLOQFVNCSFW4WZJ5NA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.234521901 (783h44m33.105604936s) [noise=true , address=45.91.116.12:4226 , piece_id=V7NKTZUGNA2HMXA3FSYMIXMSDLQVEUTJB74IW32GYF4XTU45UY6Q, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.235153884 (783h44m33.106236919s) [noise=true , address=145.79.227.45:4231 , piece_id=ESQLZIZB2AVESVJRHVGX4T5AJDEKVGJEJ3JSGA5J25BOQ4UUF7KA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.235615984 (783h44m33.106699019s) [noise=true , address=145.79.227.11:4729 , piece_id=5EQJ3NL4R45GFF35F2WOYWCD77UGR6XUV6B4TIZASZBCBCX6APWA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.236105967 (783h44m33.107189002s) [noise=true , address=145.79.227.5:4221 , piece_id=SHQ2ARJ6XTS7NHVSSNAXW2VR7BPS26MSX76INSRBH6KXPM4SQKEA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.236621799 (783h44m33.107704844s) [noise=true , address=145.79.227.7:4293 , piece_id=WPVYNVVDT4G5DG2HIOLUDVJNIU5O3WFII2H3GIHEZDBST4F2DKGA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.239024364 (783h44m33.110107409s) [noise=true , address=77.90.31.18:4805 , piece_id=YIHVN23OWQTVKXHZIAY4K2C65LU67NA7C6GCC63PXZHCWAZGL7DQ, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:34.837564819 (783h44m34.708647854s) [upload=685582]: Next returned nil segment
Jan 7 07:13:34.837594546 (783h44m34.708677581s) [upload=685582]: Waiting for error group managing segments...
Jan 7 07:13:35.221987466 (783h44m35.093070501s) [noise=true , address=145.79.227.38:4978 , piece_id=QV3BBDP5GVUMTBI2HIE7FWUIUOQVKIOO2D6OJNW2VDCXDFKMISUA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.224032561 (783h44m35.095115596s) [noise=true , address=145.79.227.11:4729 , piece_id=5EQJ3NL4R45GFF35F2WOYWCD77UGR6XUV6B4TIZASZBCBCX6APWA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.224625309 (783h44m35.095708355s) [noise=true , address=145.79.227.52:4445 , piece_id=D6QKUQIBGUGV3V2ZELIM6U5UMTHT2765CHQBIKQ2N3BJF3UXH3XA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.225249578 (783h44m35.096332613s) [noise=true , address=145.79.227.22:4511 , piece_id=VHGXFDPNKXDZI2ARWI7227A5CF3P572VJSQZS75BMMHPAVCPPO7Q, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.233155391 (783h44m35.104238446s) [noise=true , address=145.79.227.45:4231 , piece_id=ESQLZIZB2AVESVJRHVGX4T5AJDEKVGJEJ3JSGA5J25BOQ4UUF7KA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.244418267 (783h44m35.115501302s) [noise=true , address=145.79.227.5:4221 , piece_id=SHQ2ARJ6XTS7NHVSSNAXW2VR7BPS26MSX76INSRBH6KXPM4SQKEA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.253004497 (783h44m35.124087542s) [noise=true , address=145.79.227.31:4798 , piece_id=TKL7NRLI7DYTXU4HEOBJHXQR5CGHU7IG53L2GLENDCORGVVN2KOQ, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.260994110 (783h44m35.132077145s) [noise=true , address=145.79.227.7:4293 , piece_id=WPVYNVVDT4G5DG2HIOLUDVJNIU5O3WFII2H3GIHEZDBST4F2DKGA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.298671554 (783h44m35.169754589s) [noise=true , address=145.79.227.4:4185 , piece_id=SMLAUFNVX6M24OXQ2PK2TIP5MXF3KZYRGHBFXYN7OXD5L76PCY3Q, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.306366076 (783h44m35.177449111s) [noise=true , address=145.79.227.40:4049 , piece_id=RV43PFYU6UWU5QFHCO7T6F3E3VBNEYQRMA4UCPXT54PP3T3ZD62Q, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.519691228 (783h44m35.390774263s) [noise=true , address=77.90.31.16:4736 , piece_id=6I7NM23KI67OH2RKW4NSGYGJYWZXMH5HVLHLOQFVNCSFW4WZJ5NA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.749540043 (783h44m35.620623078s) [noise=true , address=45.91.116.5:4300 , piece_id=5FMCKFKIRBUKZGZTTBKCMURBHZTBKQPHEZMHJWJHLM2PLEDFNXSA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.823582121 (783h44m35.694665156s) [noise=true , address=45.91.116.8:4047 , piece_id=D6ZXOD5IBAYIVGUWNC6OGIL3OTB7JF6OAFGNGIR2RH7NDWRAEORA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.823600715 (783h44m35.69468375s) [seg_pos={0 0}, upload=685582]: Segment reached optimal threshold of 13 pieces.
Jan 7 07:13:35.824004615 (783h44m35.69508765s) [noise=true , address=45.91.116.12:4226 , piece_id=V7NKTZUGNA2HMXA3FSYMIXMSDLQVEUTJB74IW32GYF4XTU45UY6Q, seg_pos={0 0}, upload=685582]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: write tcp 188.166.225.170:54196->45.91.116.12:4226: use of closed network connection; write tcp 188.166.225.170:54196->45.91.116.12:4226: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54196->45.91.116.12:4226: use of closed network connection
Jan 7 07:13:35.824012139 (783h44m35.695095174s) [noise=true , address=45.91.116.12:4226 , piece_id=V7NKTZUGNA2HMXA3FSYMIXMSDLQVEUTJB74IW32GYF4XTU45UY6Q, seg_pos={0 0}, upload=685582]: Long tail cancellation due to optimal threshold
Jan 7 07:13:35.824197492 (783h44m35.695280527s) [noise=true , address=45.91.116.13:4153 , piece_id=27PHIESDJJJKDKJITJODFYQLSF42BX3BVTMOXFJAUGK73D3Z47UA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: read tcp 188.166.225.170:33886->45.91.116.13:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 7 07:13:35.824205007 (783h44m35.695288042s) [noise=true , address=45.91.116.13:4153 , piece_id=27PHIESDJJJKDKJITJODFYQLSF42BX3BVTMOXFJAUGK73D3Z47UA, seg_pos={0 0}, upload=685582]: Long tail cancellation due to optimal threshold
Jan 7 07:13:35.824576704 (783h44m35.695659739s) [noise=true , address=77.90.31.18:4805 , piece_id=YIHVN23OWQTVKXHZIAY4K2C65LU67NA7C6GCC63PXZHCWAZGL7DQ, seg_pos={0 0}, upload=685582]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: read tcp 188.166.225.170:38598->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 7 07:13:35.824596592 (783h44m35.695679637s) [noise=true , address=77.90.31.18:4805 , piece_id=YIHVN23OWQTVKXHZIAY4K2C65LU67NA7C6GCC63PXZHCWAZGL7DQ, seg_pos={0 0}, upload=685582]: Long tail cancellation due to optimal threshold
Jan 7 07:13:35.824616139 (783h44m35.695699184s) [seg_pos={0 0}, upload=685582]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 7 07:13:35.824682215 (783h44m35.69576526s) [upload=685582]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 7 07:13:35.824686843 (783h44m35.695769878s) [upload=685582]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 7 07:13:36.105700746 (783h44m35.976783791s) [upload=685582]: Done uploading segments...
Jan 7 07:13:36.105723649 (783h44m35.976806684s) [upload=685582]: Upload finished. err: <nil>
Duration: 5.953364442s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [...................................X........X........X......] duration:
6.070157203s trace:
https://jaeger.dev.storj.io/trace/621f2bd1cae766d0
FAIL (2026-01-07 07:23:33 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
927s) [noise=true , address=77.90.31.15:4727 , piece_id=UFRMVFFY2UKQLPR7P4TTN2UFNFZQ6BDPUP7INPYRJIK267R6WPTQ, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.304384464 (783h48m48.175467499s) [noise=true , address=145.79.227.18:4017 , piece_id=XEDPCH5DMCZIQUDMILRNT4NMMVSH67CDSFA2LXLZX3SJNC7K64JQ, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.304648907 (783h48m48.175731942s) [noise=true , address=145.79.227.43:4180 , piece_id=3ZAN75QUFNRD4K5QYAOZAYP2O4KK4SVZPXPTDOA7IYVIGLLEPYSQ, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.304909002 (783h48m48.175992037s) [noise=true , address=145.79.227.35:4484 , piece_id=6ZQ2M55FDBKKZAGXNN7GPGZCXPKJWRGPKDL5XQWBWREGXKRSCLBQ, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.305186050 (783h48m48.176269085s) [noise=true , address=64.49.13.169:4713 , piece_id=4X6L55C3YY3Y35RKP2LX7BVPJME7L5JWELQ2XK2E752NT7E6KCWA, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.305462475 (783h48m48.17654552s) [noise=true , address=145.79.227.41:4109 , piece_id=F62DIMJUKTLEX3TO4D6FT2PGGKOCDKYY7US6QZGW7PBGNXMLQRFQ, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.306063991 (783h48m48.177147037s) [noise=true , address=145.79.227.2:4127 , piece_id=J4JYZ2L5ULSHBLFH3N2FUKXBCDWEPVPNNNEBUW4LA3QAJYEGCBLA, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.306457621 (783h48m48.177540666s) [noise=true , address=145.79.227.49:4370 , piece_id=G3ATR7KHT5DDCJNAEZDDI55XTPPZMFDYKQLAR3AD45P3GCESOHGA, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.306836312 (783h48m48.177919357s) [noise=true , address=145.79.227.33:4888 , piece_id=NDWJJUO4V4UBYYWJZJM4FQGVUDR24WTCHABZJMUGZ4FMH3U6ZK7Q, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.307182321 (783h48m48.178265366s) [noise=true , address=145.79.227.30:4777 , piece_id=BBHGIQO2AEMZWMUOGMRJC4CKRWJOTEY3WQP4I4XRGMNRIOUOH5IQ, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.307621457 (783h48m48.178704503s) [noise=true , address=145.79.227.34:4931 , piece_id=QFHCIN7ZZBYM3CFF6UVA73EFMWRKJBOWX6KBFZC2SHZ5KN3WU34Q, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.307948189 (783h48m48.179031234s) [noise=true , address=77.90.31.21:4909 , piece_id=7KH356VERAYPY7GHMEOT34TADWH4U74L2JUPMQ6Y7WF52L5I6M6A, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.308347098 (783h48m48.179430133s) [noise=true , address=77.90.31.18:4830 , piece_id=TUICFKVMJ3S2KHW7MKFNJUR6Z5UN5QFZOBXJEYC43O3BE3QQDWHA, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.308510059 (783h48m48.179593104s) [noise=true , address=77.90.31.12:4619 , piece_id=DVOJATZI5UILEARIOOZ2K3WGDFFQTFONCK2HKP2ZXKLBAEQZLROQ, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.309050468 (783h48m48.180133514s) [noise=true , address=145.79.227.44:4198 , piece_id=4KNZZKJ5V3SOLZKPJYY6FSCG4SIAF67RDUWRQSXPAMDY5IO32YOA, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.309517858 (783h48m48.180600903s) [noise=true , address=64.49.13.155:4920 , piece_id=WMM6GIC2AJJHN2BOT7PLSKWVG7ONHG3CL3OF3PJQ5DH5ODFO24ZA, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:49.600689336 (783h48m49.471772381s) [upload=685644]: Next returned nil segment
Jan 7 07:17:49.600703433 (783h48m49.471786478s) [upload=685644]: Waiting for error group managing segments...
Jan 7 07:17:50.063713792 (783h48m49.934796847s) [noise=true , address=64.49.13.169:4713 , piece_id=4X6L55C3YY3Y35RKP2LX7BVPJME7L5JWELQ2XK2E752NT7E6KCWA, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.116339850 (783h48m49.987422895s) [noise=true , address=64.49.13.155:4920 , piece_id=WMM6GIC2AJJHN2BOT7PLSKWVG7ONHG3CL3OF3PJQ5DH5ODFO24ZA, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.154469825 (783h48m50.02555287s) [noise=true , address=145.79.227.32:4847 , piece_id=73A46W74UZTHTZBW6G35LSJZIA3ZRVFRRCC3KGJYGWZZH6NLICGA, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.158394037 (783h48m50.029477072s) [noise=true , address=145.79.227.44:4198 , piece_id=4KNZZKJ5V3SOLZKPJYY6FSCG4SIAF67RDUWRQSXPAMDY5IO32YOA, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.164547664 (783h48m50.035630709s) [noise=true , address=145.79.227.18:4017 , piece_id=XEDPCH5DMCZIQUDMILRNT4NMMVSH67CDSFA2LXLZX3SJNC7K64JQ, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.173751741 (783h48m50.044834786s) [noise=true , address=145.79.227.41:4109 , piece_id=F62DIMJUKTLEX3TO4D6FT2PGGKOCDKYY7US6QZGW7PBGNXMLQRFQ, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.181867575 (783h48m50.05295061s) [noise=true , address=145.79.227.34:4931 , piece_id=QFHCIN7ZZBYM3CFF6UVA73EFMWRKJBOWX6KBFZC2SHZ5KN3WU34Q, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.190204078 (783h48m50.061287113s) [noise=true , address=145.79.227.2:4127 , piece_id=J4JYZ2L5ULSHBLFH3N2FUKXBCDWEPVPNNNEBUW4LA3QAJYEGCBLA, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.215263345 (783h48m50.08634638s) [noise=true , address=145.79.227.33:4888 , piece_id=NDWJJUO4V4UBYYWJZJM4FQGVUDR24WTCHABZJMUGZ4FMH3U6ZK7Q, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.244106597 (783h48m50.115189632s) [noise=true , address=145.79.227.30:4777 , piece_id=BBHGIQO2AEMZWMUOGMRJC4CKRWJOTEY3WQP4I4XRGMNRIOUOH5IQ, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.468914424 (783h48m50.339997469s) [noise=true , address=77.90.31.18:4830 , piece_id=TUICFKVMJ3S2KHW7MKFNJUR6Z5UN5QFZOBXJEYC43O3BE3QQDWHA, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.533387670 (783h48m50.404470705s) [noise=true , address=77.90.31.12:4619 , piece_id=DVOJATZI5UILEARIOOZ2K3WGDFFQTFONCK2HKP2ZXKLBAEQZLROQ, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.560320373 (783h48m50.431403408s) [noise=true , address=77.90.31.11:4579 , piece_id=5LUJ6FFYUVXAD2A6DNE5IOFT22VZL4ZJ4S2NRR2MCA6GX5H3P4EQ, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.604949694 (783h48m50.476032739s) [noise=true , address=77.90.31.21:4909 , piece_id=7KH356VERAYPY7GHMEOT34TADWH4U74L2JUPMQ6Y7WF52L5I6M6A, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.604977047 (783h48m50.476060082s) [seg_pos={0 0}, upload=685644]: Segment reached optimal threshold of 14 pieces.
Jan 7 07:17:50.605285644 (783h48m50.476368689s) [noise=true , address=77.90.31.15:4727 , piece_id=UFRMVFFY2UKQLPR7P4TTN2UFNFZQ6BDPUP7INPYRJIK267R6WPTQ, seg_pos={0 0}, upload=685644]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xoAbv6zJuG9LjyCdsCrTWQaCDyLrWyv8RCrMTiqNqazoEwD1M): protocol: write tcp 188.166.225.170:47856->77.90.31.15:4727: use of closed network connection; write tcp 188.166.225.170:47856->77.90.31.15:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47856->77.90.31.15:4727: use of closed network connection
Jan 7 07:17:50.605295753 (783h48m50.476378808s) [noise=true , address=77.90.31.15:4727 , piece_id=UFRMVFFY2UKQLPR7P4TTN2UFNFZQ6BDPUP7INPYRJIK267R6WPTQ, seg_pos={0 0}, upload=685644]: Long tail cancellation due to optimal threshold
Jan 7 07:17:50.605401254 (783h48m50.476484299s) [noise=true , address=145.79.227.49:4370 , piece_id=G3ATR7KHT5DDCJNAEZDDI55XTPPZMFDYKQLAR3AD45P3GCESOHGA, seg_pos={0 0}, upload=685644]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LBgixGGgNeYs63ZSpkYJmrEGr3W9Ma1s2Sj8cCxU6mUxm1aC): protocol: expected piece hash; context canceled; EOF
Jan 7 07:17:50.605437523 (783h48m50.476520568s) [noise=true , address=145.79.227.49:4370 , piece_id=G3ATR7KHT5DDCJNAEZDDI55XTPPZMFDYKQLAR3AD45P3GCESOHGA, seg_pos={0 0}, upload=685644]: Long tail cancellation due to optimal threshold
Jan 7 07:17:50.605839750 (783h48m50.476922825s) [noise=true , address=145.79.227.43:4180 , piece_id=3ZAN75QUFNRD4K5QYAOZAYP2O4KK4SVZPXPTDOA7IYVIGLLEPYSQ, seg_pos={0 0}, upload=685644]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rhkcipPBZNFScBjBhWdigzSMwENSmzhCucnC2iuyc9udpweE3): protocol: expected piece hash; context canceled; EOF
Jan 7 07:17:50.605882010 (783h48m50.476965045s) [noise=true , address=145.79.227.43:4180 , piece_id=3ZAN75QUFNRD4K5QYAOZAYP2O4KK4SVZPXPTDOA7IYVIGLLEPYSQ, seg_pos={0 0}, upload=685644]: Long tail cancellation due to optimal threshold
Jan 7 07:17:50.605904593 (783h48m50.476987628s) [noise=true , address=145.79.227.35:4484 , piece_id=6ZQ2M55FDBKKZAGXNN7GPGZCXPKJWRGPKDL5XQWBWREGXKRSCLBQ, seg_pos={0 0}, upload=685644]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1a5S5Ar7jNbK9MESghbWfmEgBPNSMN2tEnZeeG1kKzeX2dnSX3): protocol: expected piece hash; context canceled; EOF
Jan 7 07:17:50.605924150 (783h48m50.477007205s) [noise=true , address=145.79.227.35:4484 , piece_id=6ZQ2M55FDBKKZAGXNN7GPGZCXPKJWRGPKDL5XQWBWREGXKRSCLBQ, seg_pos={0 0}, upload=685644]: Long tail cancellation due to optimal threshold
Jan 7 07:17:50.605947465 (783h48m50.47703051s) [seg_pos={0 0}, upload=685644]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jan 7 07:17:50.606035051 (783h48m50.477118086s) [upload=685644]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 7 07:17:50.606040612 (783h48m50.477123647s) [upload=685644]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 7 07:17:50.908947210 (783h48m50.780030255s) [upload=685644]: Done uploading segments...
Jan 7 07:17:50.908965275 (783h48m50.78004831s) [upload=685644]: Upload finished. err: <nil>
Duration: 6.070174716s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [....................................X........X........X.....] duration:
56.00750511s trace:
https://jaeger.dev.storj.io/trace/621f2bd1cae766d0
FAIL (2026-01-07 07:23:26 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
927s) [noise=true , address=77.90.31.15:4727 , piece_id=UFRMVFFY2UKQLPR7P4TTN2UFNFZQ6BDPUP7INPYRJIK267R6WPTQ, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.304384464 (783h48m48.175467499s) [noise=true , address=145.79.227.18:4017 , piece_id=XEDPCH5DMCZIQUDMILRNT4NMMVSH67CDSFA2LXLZX3SJNC7K64JQ, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.304648907 (783h48m48.175731942s) [noise=true , address=145.79.227.43:4180 , piece_id=3ZAN75QUFNRD4K5QYAOZAYP2O4KK4SVZPXPTDOA7IYVIGLLEPYSQ, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.304909002 (783h48m48.175992037s) [noise=true , address=145.79.227.35:4484 , piece_id=6ZQ2M55FDBKKZAGXNN7GPGZCXPKJWRGPKDL5XQWBWREGXKRSCLBQ, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.305186050 (783h48m48.176269085s) [noise=true , address=64.49.13.169:4713 , piece_id=4X6L55C3YY3Y35RKP2LX7BVPJME7L5JWELQ2XK2E752NT7E6KCWA, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.305462475 (783h48m48.17654552s) [noise=true , address=145.79.227.41:4109 , piece_id=F62DIMJUKTLEX3TO4D6FT2PGGKOCDKYY7US6QZGW7PBGNXMLQRFQ, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.306063991 (783h48m48.177147037s) [noise=true , address=145.79.227.2:4127 , piece_id=J4JYZ2L5ULSHBLFH3N2FUKXBCDWEPVPNNNEBUW4LA3QAJYEGCBLA, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.306457621 (783h48m48.177540666s) [noise=true , address=145.79.227.49:4370 , piece_id=G3ATR7KHT5DDCJNAEZDDI55XTPPZMFDYKQLAR3AD45P3GCESOHGA, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.306836312 (783h48m48.177919357s) [noise=true , address=145.79.227.33:4888 , piece_id=NDWJJUO4V4UBYYWJZJM4FQGVUDR24WTCHABZJMUGZ4FMH3U6ZK7Q, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.307182321 (783h48m48.178265366s) [noise=true , address=145.79.227.30:4777 , piece_id=BBHGIQO2AEMZWMUOGMRJC4CKRWJOTEY3WQP4I4XRGMNRIOUOH5IQ, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.307621457 (783h48m48.178704503s) [noise=true , address=145.79.227.34:4931 , piece_id=QFHCIN7ZZBYM3CFF6UVA73EFMWRKJBOWX6KBFZC2SHZ5KN3WU34Q, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.307948189 (783h48m48.179031234s) [noise=true , address=77.90.31.21:4909 , piece_id=7KH356VERAYPY7GHMEOT34TADWH4U74L2JUPMQ6Y7WF52L5I6M6A, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.308347098 (783h48m48.179430133s) [noise=true , address=77.90.31.18:4830 , piece_id=TUICFKVMJ3S2KHW7MKFNJUR6Z5UN5QFZOBXJEYC43O3BE3QQDWHA, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.308510059 (783h48m48.179593104s) [noise=true , address=77.90.31.12:4619 , piece_id=DVOJATZI5UILEARIOOZ2K3WGDFFQTFONCK2HKP2ZXKLBAEQZLROQ, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.309050468 (783h48m48.180133514s) [noise=true , address=145.79.227.44:4198 , piece_id=4KNZZKJ5V3SOLZKPJYY6FSCG4SIAF67RDUWRQSXPAMDY5IO32YOA, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.309517858 (783h48m48.180600903s) [noise=true , address=64.49.13.155:4920 , piece_id=WMM6GIC2AJJHN2BOT7PLSKWVG7ONHG3CL3OF3PJQ5DH5ODFO24ZA, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:49.600689336 (783h48m49.471772381s) [upload=685644]: Next returned nil segment
Jan 7 07:17:49.600703433 (783h48m49.471786478s) [upload=685644]: Waiting for error group managing segments...
Jan 7 07:17:50.063713792 (783h48m49.934796847s) [noise=true , address=64.49.13.169:4713 , piece_id=4X6L55C3YY3Y35RKP2LX7BVPJME7L5JWELQ2XK2E752NT7E6KCWA, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.116339850 (783h48m49.987422895s) [noise=true , address=64.49.13.155:4920 , piece_id=WMM6GIC2AJJHN2BOT7PLSKWVG7ONHG3CL3OF3PJQ5DH5ODFO24ZA, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.154469825 (783h48m50.02555287s) [noise=true , address=145.79.227.32:4847 , piece_id=73A46W74UZTHTZBW6G35LSJZIA3ZRVFRRCC3KGJYGWZZH6NLICGA, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.158394037 (783h48m50.029477072s) [noise=true , address=145.79.227.44:4198 , piece_id=4KNZZKJ5V3SOLZKPJYY6FSCG4SIAF67RDUWRQSXPAMDY5IO32YOA, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.164547664 (783h48m50.035630709s) [noise=true , address=145.79.227.18:4017 , piece_id=XEDPCH5DMCZIQUDMILRNT4NMMVSH67CDSFA2LXLZX3SJNC7K64JQ, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.173751741 (783h48m50.044834786s) [noise=true , address=145.79.227.41:4109 , piece_id=F62DIMJUKTLEX3TO4D6FT2PGGKOCDKYY7US6QZGW7PBGNXMLQRFQ, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.181867575 (783h48m50.05295061s) [noise=true , address=145.79.227.34:4931 , piece_id=QFHCIN7ZZBYM3CFF6UVA73EFMWRKJBOWX6KBFZC2SHZ5KN3WU34Q, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.190204078 (783h48m50.061287113s) [noise=true , address=145.79.227.2:4127 , piece_id=J4JYZ2L5ULSHBLFH3N2FUKXBCDWEPVPNNNEBUW4LA3QAJYEGCBLA, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.215263345 (783h48m50.08634638s) [noise=true , address=145.79.227.33:4888 , piece_id=NDWJJUO4V4UBYYWJZJM4FQGVUDR24WTCHABZJMUGZ4FMH3U6ZK7Q, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.244106597 (783h48m50.115189632s) [noise=true , address=145.79.227.30:4777 , piece_id=BBHGIQO2AEMZWMUOGMRJC4CKRWJOTEY3WQP4I4XRGMNRIOUOH5IQ, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.468914424 (783h48m50.339997469s) [noise=true , address=77.90.31.18:4830 , piece_id=TUICFKVMJ3S2KHW7MKFNJUR6Z5UN5QFZOBXJEYC43O3BE3QQDWHA, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.533387670 (783h48m50.404470705s) [noise=true , address=77.90.31.12:4619 , piece_id=DVOJATZI5UILEARIOOZ2K3WGDFFQTFONCK2HKP2ZXKLBAEQZLROQ, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.560320373 (783h48m50.431403408s) [noise=true , address=77.90.31.11:4579 , piece_id=5LUJ6FFYUVXAD2A6DNE5IOFT22VZL4ZJ4S2NRR2MCA6GX5H3P4EQ, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.604949694 (783h48m50.476032739s) [noise=true , address=77.90.31.21:4909 , piece_id=7KH356VERAYPY7GHMEOT34TADWH4U74L2JUPMQ6Y7WF52L5I6M6A, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.604977047 (783h48m50.476060082s) [seg_pos={0 0}, upload=685644]: Segment reached optimal threshold of 14 pieces.
Jan 7 07:17:50.605285644 (783h48m50.476368689s) [noise=true , address=77.90.31.15:4727 , piece_id=UFRMVFFY2UKQLPR7P4TTN2UFNFZQ6BDPUP7INPYRJIK267R6WPTQ, seg_pos={0 0}, upload=685644]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xoAbv6zJuG9LjyCdsCrTWQaCDyLrWyv8RCrMTiqNqazoEwD1M): protocol: write tcp 188.166.225.170:47856->77.90.31.15:4727: use of closed network connection; write tcp 188.166.225.170:47856->77.90.31.15:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47856->77.90.31.15:4727: use of closed network connection
Jan 7 07:17:50.605295753 (783h48m50.476378808s) [noise=true , address=77.90.31.15:4727 , piece_id=UFRMVFFY2UKQLPR7P4TTN2UFNFZQ6BDPUP7INPYRJIK267R6WPTQ, seg_pos={0 0}, upload=685644]: Long tail cancellation due to optimal threshold
Jan 7 07:17:50.605401254 (783h48m50.476484299s) [noise=true , address=145.79.227.49:4370 , piece_id=G3ATR7KHT5DDCJNAEZDDI55XTPPZMFDYKQLAR3AD45P3GCESOHGA, seg_pos={0 0}, upload=685644]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LBgixGGgNeYs63ZSpkYJmrEGr3W9Ma1s2Sj8cCxU6mUxm1aC): protocol: expected piece hash; context canceled; EOF
Jan 7 07:17:50.605437523 (783h48m50.476520568s) [noise=true , address=145.79.227.49:4370 , piece_id=G3ATR7KHT5DDCJNAEZDDI55XTPPZMFDYKQLAR3AD45P3GCESOHGA, seg_pos={0 0}, upload=685644]: Long tail cancellation due to optimal threshold
Jan 7 07:17:50.605839750 (783h48m50.476922825s) [noise=true , address=145.79.227.43:4180 , piece_id=3ZAN75QUFNRD4K5QYAOZAYP2O4KK4SVZPXPTDOA7IYVIGLLEPYSQ, seg_pos={0 0}, upload=685644]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rhkcipPBZNFScBjBhWdigzSMwENSmzhCucnC2iuyc9udpweE3): protocol: expected piece hash; context canceled; EOF
Jan 7 07:17:50.605882010 (783h48m50.476965045s) [noise=true , address=145.79.227.43:4180 , piece_id=3ZAN75QUFNRD4K5QYAOZAYP2O4KK4SVZPXPTDOA7IYVIGLLEPYSQ, seg_pos={0 0}, upload=685644]: Long tail cancellation due to optimal threshold
Jan 7 07:17:50.605904593 (783h48m50.476987628s) [noise=true , address=145.79.227.35:4484 , piece_id=6ZQ2M55FDBKKZAGXNN7GPGZCXPKJWRGPKDL5XQWBWREGXKRSCLBQ, seg_pos={0 0}, upload=685644]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1a5S5Ar7jNbK9MESghbWfmEgBPNSMN2tEnZeeG1kKzeX2dnSX3): protocol: expected piece hash; context canceled; EOF
Jan 7 07:17:50.605924150 (783h48m50.477007205s) [noise=true , address=145.79.227.35:4484 , piece_id=6ZQ2M55FDBKKZAGXNN7GPGZCXPKJWRGPKDL5XQWBWREGXKRSCLBQ, seg_pos={0 0}, upload=685644]: Long tail cancellation due to optimal threshold
Jan 7 07:17:50.605947465 (783h48m50.47703051s) [seg_pos={0 0}, upload=685644]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jan 7 07:17:50.606035051 (783h48m50.477118086s) [upload=685644]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 7 07:17:50.606040612 (783h48m50.477123647s) [upload=685644]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 7 07:17:50.908947210 (783h48m50.780030255s) [upload=685644]: Done uploading segments...
Jan 7 07:17:50.908965275 (783h48m50.78004831s) [upload=685644]: Upload finished. err: <nil>
Duration: 56.007532673s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
49 out of 60 is failed (threshold=6) [.XXXXXXXXXXX.XXXX.XXXX..XXXXX.XXXXX..XXXXX.XXXX.XXXXXXXXX.XX] duration:
5.001472674s trace:
https://jaeger.dev.storj.io/trace/70e0bebe60058b61
FAIL (2026-01-07 07:23:24 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001499595s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [.....X...............X...................X.............X....] duration:
17.325058097s trace:
https://jaeger.dev.storj.io/trace/4389ca011670ab35
FAIL (2026-01-07 07:23:23 UTC)
orbiter warning: context deadline exceeded
Duration: 17.325084899s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
13 out of 60 is failed (threshold=3) [.X.X..X....X.X.....X....X.X.....X.......X....XX.........X...] duration:
20.216431941s trace:
https://jaeger.dev.storj.io/trace/2cf2060b793525bb
FAIL (2026-01-07 07:23:03 UTC)
orbiter warning: context deadline exceeded
Duration: 20.216465166s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
14 out of 60 is failed (threshold=3) [X.X.X..X....X.X.....X....X.X.....X.......X....XX.........X..] duration:
24.092394649s trace:
https://jaeger.dev.storj.io/trace/2cf2060b793525bb
FAIL (2026-01-07 07:22:36 UTC)
orbiter warning: context deadline exceeded
Duration: 24.092410178s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
12 out of 60 is failed (threshold=3) [...X.........X..X.X..X.......XX.X....X....XX....X...........] duration:
6.36193331s trace:
https://jaeger.dev.storj.io/trace/566429788081faee
FAIL (2026-01-07 07:22:35 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
) [upload=685582]: Uploading segments...
Jan 7 07:13:32.990108051 (783h44m32.861191096s) [upload=685582]: Got next segment. Inline: false
Jan 7 07:13:32.990479758 (783h44m32.861562803s) [upload=685582]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 7 07:13:32.990486191 (783h44m32.861569226s) [upload=685582]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 7 07:13:33.230730672 (783h44m33.101813717s) [seg_pos={0 0}, upload=685582]: Begin upload segment...
Jan 7 07:13:33.230783873 (783h44m33.101866908s) [seg_pos={0 0}, upload=685582]: Done begin upload segment.
Jan 7 07:13:33.230814070 (783h44m33.101897115s) [noise=true , address=45.91.116.13:4153 , piece_id=27PHIESDJJJKDKJITJODFYQLSF42BX3BVTMOXFJAUGK73D3Z47UA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.230992941 (783h44m33.102075976s) [noise=true , address=45.91.116.5:4300 , piece_id=5FMCKFKIRBUKZGZTTBKCMURBHZTBKQPHEZMHJWJHLM2PLEDFNXSA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.231512250 (783h44m33.102595285s) [noise=true , address=145.79.227.22:4511 , piece_id=VHGXFDPNKXDZI2ARWI7227A5CF3P572VJSQZS75BMMHPAVCPPO7Q, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.231639272 (783h44m33.102722307s) [noise=true , address=145.79.227.4:4185 , piece_id=SMLAUFNVX6M24OXQ2PK2TIP5MXF3KZYRGHBFXYN7OXD5L76PCY3Q, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.231935536 (783h44m33.103018571s) [noise=true , address=145.79.227.31:4798 , piece_id=TKL7NRLI7DYTXU4HEOBJHXQR5CGHU7IG53L2GLENDCORGVVN2KOQ, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.232203125 (783h44m33.10328616s) [noise=true , address=145.79.227.38:4978 , piece_id=QV3BBDP5GVUMTBI2HIE7FWUIUOQVKIOO2D6OJNW2VDCXDFKMISUA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.232580194 (783h44m33.103663229s) [noise=true , address=145.79.227.52:4445 , piece_id=D6QKUQIBGUGV3V2ZELIM6U5UMTHT2765CHQBIKQ2N3BJF3UXH3XA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.232942814 (783h44m33.104025849s) [noise=true , address=45.91.116.8:4047 , piece_id=D6ZXOD5IBAYIVGUWNC6OGIL3OTB7JF6OAFGNGIR2RH7NDWRAEORA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.233250680 (783h44m33.104333725s) [noise=true , address=145.79.227.40:4049 , piece_id=RV43PFYU6UWU5QFHCO7T6F3E3VBNEYQRMA4UCPXT54PP3T3ZD62Q, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.233527197 (783h44m33.104610242s) [noise=true , address=77.90.31.16:4736 , piece_id=6I7NM23KI67OH2RKW4NSGYGJYWZXMH5HVLHLOQFVNCSFW4WZJ5NA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.234521901 (783h44m33.105604936s) [noise=true , address=45.91.116.12:4226 , piece_id=V7NKTZUGNA2HMXA3FSYMIXMSDLQVEUTJB74IW32GYF4XTU45UY6Q, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.235153884 (783h44m33.106236919s) [noise=true , address=145.79.227.45:4231 , piece_id=ESQLZIZB2AVESVJRHVGX4T5AJDEKVGJEJ3JSGA5J25BOQ4UUF7KA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.235615984 (783h44m33.106699019s) [noise=true , address=145.79.227.11:4729 , piece_id=5EQJ3NL4R45GFF35F2WOYWCD77UGR6XUV6B4TIZASZBCBCX6APWA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.236105967 (783h44m33.107189002s) [noise=true , address=145.79.227.5:4221 , piece_id=SHQ2ARJ6XTS7NHVSSNAXW2VR7BPS26MSX76INSRBH6KXPM4SQKEA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.236621799 (783h44m33.107704844s) [noise=true , address=145.79.227.7:4293 , piece_id=WPVYNVVDT4G5DG2HIOLUDVJNIU5O3WFII2H3GIHEZDBST4F2DKGA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.239024364 (783h44m33.110107409s) [noise=true , address=77.90.31.18:4805 , piece_id=YIHVN23OWQTVKXHZIAY4K2C65LU67NA7C6GCC63PXZHCWAZGL7DQ, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:34.837564819 (783h44m34.708647854s) [upload=685582]: Next returned nil segment
Jan 7 07:13:34.837594546 (783h44m34.708677581s) [upload=685582]: Waiting for error group managing segments...
Jan 7 07:13:35.221987466 (783h44m35.093070501s) [noise=true , address=145.79.227.38:4978 , piece_id=QV3BBDP5GVUMTBI2HIE7FWUIUOQVKIOO2D6OJNW2VDCXDFKMISUA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.224032561 (783h44m35.095115596s) [noise=true , address=145.79.227.11:4729 , piece_id=5EQJ3NL4R45GFF35F2WOYWCD77UGR6XUV6B4TIZASZBCBCX6APWA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.224625309 (783h44m35.095708355s) [noise=true , address=145.79.227.52:4445 , piece_id=D6QKUQIBGUGV3V2ZELIM6U5UMTHT2765CHQBIKQ2N3BJF3UXH3XA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.225249578 (783h44m35.096332613s) [noise=true , address=145.79.227.22:4511 , piece_id=VHGXFDPNKXDZI2ARWI7227A5CF3P572VJSQZS75BMMHPAVCPPO7Q, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.233155391 (783h44m35.104238446s) [noise=true , address=145.79.227.45:4231 , piece_id=ESQLZIZB2AVESVJRHVGX4T5AJDEKVGJEJ3JSGA5J25BOQ4UUF7KA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.244418267 (783h44m35.115501302s) [noise=true , address=145.79.227.5:4221 , piece_id=SHQ2ARJ6XTS7NHVSSNAXW2VR7BPS26MSX76INSRBH6KXPM4SQKEA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.253004497 (783h44m35.124087542s) [noise=true , address=145.79.227.31:4798 , piece_id=TKL7NRLI7DYTXU4HEOBJHXQR5CGHU7IG53L2GLENDCORGVVN2KOQ, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.260994110 (783h44m35.132077145s) [noise=true , address=145.79.227.7:4293 , piece_id=WPVYNVVDT4G5DG2HIOLUDVJNIU5O3WFII2H3GIHEZDBST4F2DKGA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.298671554 (783h44m35.169754589s) [noise=true , address=145.79.227.4:4185 , piece_id=SMLAUFNVX6M24OXQ2PK2TIP5MXF3KZYRGHBFXYN7OXD5L76PCY3Q, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.306366076 (783h44m35.177449111s) [noise=true , address=145.79.227.40:4049 , piece_id=RV43PFYU6UWU5QFHCO7T6F3E3VBNEYQRMA4UCPXT54PP3T3ZD62Q, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.519691228 (783h44m35.390774263s) [noise=true , address=77.90.31.16:4736 , piece_id=6I7NM23KI67OH2RKW4NSGYGJYWZXMH5HVLHLOQFVNCSFW4WZJ5NA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.749540043 (783h44m35.620623078s) [noise=true , address=45.91.116.5:4300 , piece_id=5FMCKFKIRBUKZGZTTBKCMURBHZTBKQPHEZMHJWJHLM2PLEDFNXSA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.823582121 (783h44m35.694665156s) [noise=true , address=45.91.116.8:4047 , piece_id=D6ZXOD5IBAYIVGUWNC6OGIL3OTB7JF6OAFGNGIR2RH7NDWRAEORA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.823600715 (783h44m35.69468375s) [seg_pos={0 0}, upload=685582]: Segment reached optimal threshold of 13 pieces.
Jan 7 07:13:35.824004615 (783h44m35.69508765s) [noise=true , address=45.91.116.12:4226 , piece_id=V7NKTZUGNA2HMXA3FSYMIXMSDLQVEUTJB74IW32GYF4XTU45UY6Q, seg_pos={0 0}, upload=685582]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: write tcp 188.166.225.170:54196->45.91.116.12:4226: use of closed network connection; write tcp 188.166.225.170:54196->45.91.116.12:4226: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54196->45.91.116.12:4226: use of closed network connection
Jan 7 07:13:35.824012139 (783h44m35.695095174s) [noise=true , address=45.91.116.12:4226 , piece_id=V7NKTZUGNA2HMXA3FSYMIXMSDLQVEUTJB74IW32GYF4XTU45UY6Q, seg_pos={0 0}, upload=685582]: Long tail cancellation due to optimal threshold
Jan 7 07:13:35.824197492 (783h44m35.695280527s) [noise=true , address=45.91.116.13:4153 , piece_id=27PHIESDJJJKDKJITJODFYQLSF42BX3BVTMOXFJAUGK73D3Z47UA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: read tcp 188.166.225.170:33886->45.91.116.13:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 7 07:13:35.824205007 (783h44m35.695288042s) [noise=true , address=45.91.116.13:4153 , piece_id=27PHIESDJJJKDKJITJODFYQLSF42BX3BVTMOXFJAUGK73D3Z47UA, seg_pos={0 0}, upload=685582]: Long tail cancellation due to optimal threshold
Jan 7 07:13:35.824576704 (783h44m35.695659739s) [noise=true , address=77.90.31.18:4805 , piece_id=YIHVN23OWQTVKXHZIAY4K2C65LU67NA7C6GCC63PXZHCWAZGL7DQ, seg_pos={0 0}, upload=685582]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: read tcp 188.166.225.170:38598->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 7 07:13:35.824596592 (783h44m35.695679637s) [noise=true , address=77.90.31.18:4805 , piece_id=YIHVN23OWQTVKXHZIAY4K2C65LU67NA7C6GCC63PXZHCWAZGL7DQ, seg_pos={0 0}, upload=685582]: Long tail cancellation due to optimal threshold
Jan 7 07:13:35.824616139 (783h44m35.695699184s) [seg_pos={0 0}, upload=685582]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 7 07:13:35.824682215 (783h44m35.69576526s) [upload=685582]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 7 07:13:35.824686843 (783h44m35.695769878s) [upload=685582]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 7 07:13:36.105700746 (783h44m35.976783791s) [upload=685582]: Done uploading segments...
Jan 7 07:13:36.105723649 (783h44m35.976806684s) [upload=685582]: Upload finished. err: <nil>
Duration: 6.361955442s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
49 out of 60 is failed (threshold=6) [X.XXXXXXXXXXX.XXXX.XXXX..XXXXX.XXXXX..XXXXX.XXXX.XXXXXXXXX.X] duration:
5.001381537s trace:
https://jaeger.dev.storj.io/trace/2b362b192d1ef83b
FAIL (2026-01-07 07:22:29 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001411244s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [......X...............X...................X.............X...] duration:
29.660375878s trace:
https://jaeger.dev.storj.io/trace/4389ca011670ab35
FAIL (2026-01-07 07:22:23 UTC)
orbiter warning: context deadline exceeded
Duration: 29.660403379s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
12 out of 60 is failed (threshold=3) [....X.........X..X.X..X.......XX.X....X....XX....X..........] duration:
5.681764971s trace:
https://jaeger.dev.storj.io/trace/566429788081faee
FAIL (2026-01-07 07:22:15 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
) [upload=685582]: Uploading segments...
Jan 7 07:13:32.990108051 (783h44m32.861191096s) [upload=685582]: Got next segment. Inline: false
Jan 7 07:13:32.990479758 (783h44m32.861562803s) [upload=685582]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 7 07:13:32.990486191 (783h44m32.861569226s) [upload=685582]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 7 07:13:33.230730672 (783h44m33.101813717s) [seg_pos={0 0}, upload=685582]: Begin upload segment...
Jan 7 07:13:33.230783873 (783h44m33.101866908s) [seg_pos={0 0}, upload=685582]: Done begin upload segment.
Jan 7 07:13:33.230814070 (783h44m33.101897115s) [noise=true , address=45.91.116.13:4153 , piece_id=27PHIESDJJJKDKJITJODFYQLSF42BX3BVTMOXFJAUGK73D3Z47UA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.230992941 (783h44m33.102075976s) [noise=true , address=45.91.116.5:4300 , piece_id=5FMCKFKIRBUKZGZTTBKCMURBHZTBKQPHEZMHJWJHLM2PLEDFNXSA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.231512250 (783h44m33.102595285s) [noise=true , address=145.79.227.22:4511 , piece_id=VHGXFDPNKXDZI2ARWI7227A5CF3P572VJSQZS75BMMHPAVCPPO7Q, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.231639272 (783h44m33.102722307s) [noise=true , address=145.79.227.4:4185 , piece_id=SMLAUFNVX6M24OXQ2PK2TIP5MXF3KZYRGHBFXYN7OXD5L76PCY3Q, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.231935536 (783h44m33.103018571s) [noise=true , address=145.79.227.31:4798 , piece_id=TKL7NRLI7DYTXU4HEOBJHXQR5CGHU7IG53L2GLENDCORGVVN2KOQ, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.232203125 (783h44m33.10328616s) [noise=true , address=145.79.227.38:4978 , piece_id=QV3BBDP5GVUMTBI2HIE7FWUIUOQVKIOO2D6OJNW2VDCXDFKMISUA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.232580194 (783h44m33.103663229s) [noise=true , address=145.79.227.52:4445 , piece_id=D6QKUQIBGUGV3V2ZELIM6U5UMTHT2765CHQBIKQ2N3BJF3UXH3XA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.232942814 (783h44m33.104025849s) [noise=true , address=45.91.116.8:4047 , piece_id=D6ZXOD5IBAYIVGUWNC6OGIL3OTB7JF6OAFGNGIR2RH7NDWRAEORA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.233250680 (783h44m33.104333725s) [noise=true , address=145.79.227.40:4049 , piece_id=RV43PFYU6UWU5QFHCO7T6F3E3VBNEYQRMA4UCPXT54PP3T3ZD62Q, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.233527197 (783h44m33.104610242s) [noise=true , address=77.90.31.16:4736 , piece_id=6I7NM23KI67OH2RKW4NSGYGJYWZXMH5HVLHLOQFVNCSFW4WZJ5NA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.234521901 (783h44m33.105604936s) [noise=true , address=45.91.116.12:4226 , piece_id=V7NKTZUGNA2HMXA3FSYMIXMSDLQVEUTJB74IW32GYF4XTU45UY6Q, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.235153884 (783h44m33.106236919s) [noise=true , address=145.79.227.45:4231 , piece_id=ESQLZIZB2AVESVJRHVGX4T5AJDEKVGJEJ3JSGA5J25BOQ4UUF7KA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.235615984 (783h44m33.106699019s) [noise=true , address=145.79.227.11:4729 , piece_id=5EQJ3NL4R45GFF35F2WOYWCD77UGR6XUV6B4TIZASZBCBCX6APWA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.236105967 (783h44m33.107189002s) [noise=true , address=145.79.227.5:4221 , piece_id=SHQ2ARJ6XTS7NHVSSNAXW2VR7BPS26MSX76INSRBH6KXPM4SQKEA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.236621799 (783h44m33.107704844s) [noise=true , address=145.79.227.7:4293 , piece_id=WPVYNVVDT4G5DG2HIOLUDVJNIU5O3WFII2H3GIHEZDBST4F2DKGA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.239024364 (783h44m33.110107409s) [noise=true , address=77.90.31.18:4805 , piece_id=YIHVN23OWQTVKXHZIAY4K2C65LU67NA7C6GCC63PXZHCWAZGL7DQ, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:34.837564819 (783h44m34.708647854s) [upload=685582]: Next returned nil segment
Jan 7 07:13:34.837594546 (783h44m34.708677581s) [upload=685582]: Waiting for error group managing segments...
Jan 7 07:13:35.221987466 (783h44m35.093070501s) [noise=true , address=145.79.227.38:4978 , piece_id=QV3BBDP5GVUMTBI2HIE7FWUIUOQVKIOO2D6OJNW2VDCXDFKMISUA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.224032561 (783h44m35.095115596s) [noise=true , address=145.79.227.11:4729 , piece_id=5EQJ3NL4R45GFF35F2WOYWCD77UGR6XUV6B4TIZASZBCBCX6APWA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.224625309 (783h44m35.095708355s) [noise=true , address=145.79.227.52:4445 , piece_id=D6QKUQIBGUGV3V2ZELIM6U5UMTHT2765CHQBIKQ2N3BJF3UXH3XA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.225249578 (783h44m35.096332613s) [noise=true , address=145.79.227.22:4511 , piece_id=VHGXFDPNKXDZI2ARWI7227A5CF3P572VJSQZS75BMMHPAVCPPO7Q, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.233155391 (783h44m35.104238446s) [noise=true , address=145.79.227.45:4231 , piece_id=ESQLZIZB2AVESVJRHVGX4T5AJDEKVGJEJ3JSGA5J25BOQ4UUF7KA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.244418267 (783h44m35.115501302s) [noise=true , address=145.79.227.5:4221 , piece_id=SHQ2ARJ6XTS7NHVSSNAXW2VR7BPS26MSX76INSRBH6KXPM4SQKEA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.253004497 (783h44m35.124087542s) [noise=true , address=145.79.227.31:4798 , piece_id=TKL7NRLI7DYTXU4HEOBJHXQR5CGHU7IG53L2GLENDCORGVVN2KOQ, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.260994110 (783h44m35.132077145s) [noise=true , address=145.79.227.7:4293 , piece_id=WPVYNVVDT4G5DG2HIOLUDVJNIU5O3WFII2H3GIHEZDBST4F2DKGA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.298671554 (783h44m35.169754589s) [noise=true , address=145.79.227.4:4185 , piece_id=SMLAUFNVX6M24OXQ2PK2TIP5MXF3KZYRGHBFXYN7OXD5L76PCY3Q, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.306366076 (783h44m35.177449111s) [noise=true , address=145.79.227.40:4049 , piece_id=RV43PFYU6UWU5QFHCO7T6F3E3VBNEYQRMA4UCPXT54PP3T3ZD62Q, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.519691228 (783h44m35.390774263s) [noise=true , address=77.90.31.16:4736 , piece_id=6I7NM23KI67OH2RKW4NSGYGJYWZXMH5HVLHLOQFVNCSFW4WZJ5NA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.749540043 (783h44m35.620623078s) [noise=true , address=45.91.116.5:4300 , piece_id=5FMCKFKIRBUKZGZTTBKCMURBHZTBKQPHEZMHJWJHLM2PLEDFNXSA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.823582121 (783h44m35.694665156s) [noise=true , address=45.91.116.8:4047 , piece_id=D6ZXOD5IBAYIVGUWNC6OGIL3OTB7JF6OAFGNGIR2RH7NDWRAEORA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.823600715 (783h44m35.69468375s) [seg_pos={0 0}, upload=685582]: Segment reached optimal threshold of 13 pieces.
Jan 7 07:13:35.824004615 (783h44m35.69508765s) [noise=true , address=45.91.116.12:4226 , piece_id=V7NKTZUGNA2HMXA3FSYMIXMSDLQVEUTJB74IW32GYF4XTU45UY6Q, seg_pos={0 0}, upload=685582]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: write tcp 188.166.225.170:54196->45.91.116.12:4226: use of closed network connection; write tcp 188.166.225.170:54196->45.91.116.12:4226: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54196->45.91.116.12:4226: use of closed network connection
Jan 7 07:13:35.824012139 (783h44m35.695095174s) [noise=true , address=45.91.116.12:4226 , piece_id=V7NKTZUGNA2HMXA3FSYMIXMSDLQVEUTJB74IW32GYF4XTU45UY6Q, seg_pos={0 0}, upload=685582]: Long tail cancellation due to optimal threshold
Jan 7 07:13:35.824197492 (783h44m35.695280527s) [noise=true , address=45.91.116.13:4153 , piece_id=27PHIESDJJJKDKJITJODFYQLSF42BX3BVTMOXFJAUGK73D3Z47UA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: read tcp 188.166.225.170:33886->45.91.116.13:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 7 07:13:35.824205007 (783h44m35.695288042s) [noise=true , address=45.91.116.13:4153 , piece_id=27PHIESDJJJKDKJITJODFYQLSF42BX3BVTMOXFJAUGK73D3Z47UA, seg_pos={0 0}, upload=685582]: Long tail cancellation due to optimal threshold
Jan 7 07:13:35.824576704 (783h44m35.695659739s) [noise=true , address=77.90.31.18:4805 , piece_id=YIHVN23OWQTVKXHZIAY4K2C65LU67NA7C6GCC63PXZHCWAZGL7DQ, seg_pos={0 0}, upload=685582]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: read tcp 188.166.225.170:38598->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 7 07:13:35.824596592 (783h44m35.695679637s) [noise=true , address=77.90.31.18:4805 , piece_id=YIHVN23OWQTVKXHZIAY4K2C65LU67NA7C6GCC63PXZHCWAZGL7DQ, seg_pos={0 0}, upload=685582]: Long tail cancellation due to optimal threshold
Jan 7 07:13:35.824616139 (783h44m35.695699184s) [seg_pos={0 0}, upload=685582]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 7 07:13:35.824682215 (783h44m35.69576526s) [upload=685582]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 7 07:13:35.824686843 (783h44m35.695769878s) [upload=685582]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 7 07:13:36.105700746 (783h44m35.976783791s) [upload=685582]: Done uploading segments...
Jan 7 07:13:36.105723649 (783h44m35.976806684s) [upload=685582]: Upload finished. err: <nil>
Duration: 5.681795207s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
15 out of 60 is failed (threshold=3) [XX.X.X..X....X.X.....X....X.X.....X.......X....XX.........X.] duration:
22.430963022s trace:
https://jaeger.dev.storj.io/trace/2cf2060b793525bb
FAIL (2026-01-07 07:21:39 UTC)
orbiter warning: context deadline exceeded
Duration: 22.430989112s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
49 out of 60 is failed (threshold=6) [XX.XXXXXXXXXXX.XXXX.XXXX..XXXXX.XXXXX..XXXXX.XXXX.XXXXXXXXX.] duration:
3.844246368s trace:
https://jaeger.dev.storj.io/trace/203897c5cf45d824
FAIL (2026-01-07 07:21:32 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.84427352s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [.....................................X........X........X....] duration:
5.873069588s trace:
https://jaeger.dev.storj.io/trace/621f2bd1cae766d0
FAIL (2026-01-07 07:21:28 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
927s) [noise=true , address=77.90.31.15:4727 , piece_id=UFRMVFFY2UKQLPR7P4TTN2UFNFZQ6BDPUP7INPYRJIK267R6WPTQ, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.304384464 (783h48m48.175467499s) [noise=true , address=145.79.227.18:4017 , piece_id=XEDPCH5DMCZIQUDMILRNT4NMMVSH67CDSFA2LXLZX3SJNC7K64JQ, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.304648907 (783h48m48.175731942s) [noise=true , address=145.79.227.43:4180 , piece_id=3ZAN75QUFNRD4K5QYAOZAYP2O4KK4SVZPXPTDOA7IYVIGLLEPYSQ, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.304909002 (783h48m48.175992037s) [noise=true , address=145.79.227.35:4484 , piece_id=6ZQ2M55FDBKKZAGXNN7GPGZCXPKJWRGPKDL5XQWBWREGXKRSCLBQ, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.305186050 (783h48m48.176269085s) [noise=true , address=64.49.13.169:4713 , piece_id=4X6L55C3YY3Y35RKP2LX7BVPJME7L5JWELQ2XK2E752NT7E6KCWA, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.305462475 (783h48m48.17654552s) [noise=true , address=145.79.227.41:4109 , piece_id=F62DIMJUKTLEX3TO4D6FT2PGGKOCDKYY7US6QZGW7PBGNXMLQRFQ, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.306063991 (783h48m48.177147037s) [noise=true , address=145.79.227.2:4127 , piece_id=J4JYZ2L5ULSHBLFH3N2FUKXBCDWEPVPNNNEBUW4LA3QAJYEGCBLA, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.306457621 (783h48m48.177540666s) [noise=true , address=145.79.227.49:4370 , piece_id=G3ATR7KHT5DDCJNAEZDDI55XTPPZMFDYKQLAR3AD45P3GCESOHGA, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.306836312 (783h48m48.177919357s) [noise=true , address=145.79.227.33:4888 , piece_id=NDWJJUO4V4UBYYWJZJM4FQGVUDR24WTCHABZJMUGZ4FMH3U6ZK7Q, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.307182321 (783h48m48.178265366s) [noise=true , address=145.79.227.30:4777 , piece_id=BBHGIQO2AEMZWMUOGMRJC4CKRWJOTEY3WQP4I4XRGMNRIOUOH5IQ, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.307621457 (783h48m48.178704503s) [noise=true , address=145.79.227.34:4931 , piece_id=QFHCIN7ZZBYM3CFF6UVA73EFMWRKJBOWX6KBFZC2SHZ5KN3WU34Q, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.307948189 (783h48m48.179031234s) [noise=true , address=77.90.31.21:4909 , piece_id=7KH356VERAYPY7GHMEOT34TADWH4U74L2JUPMQ6Y7WF52L5I6M6A, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.308347098 (783h48m48.179430133s) [noise=true , address=77.90.31.18:4830 , piece_id=TUICFKVMJ3S2KHW7MKFNJUR6Z5UN5QFZOBXJEYC43O3BE3QQDWHA, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.308510059 (783h48m48.179593104s) [noise=true , address=77.90.31.12:4619 , piece_id=DVOJATZI5UILEARIOOZ2K3WGDFFQTFONCK2HKP2ZXKLBAEQZLROQ, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.309050468 (783h48m48.180133514s) [noise=true , address=145.79.227.44:4198 , piece_id=4KNZZKJ5V3SOLZKPJYY6FSCG4SIAF67RDUWRQSXPAMDY5IO32YOA, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.309517858 (783h48m48.180600903s) [noise=true , address=64.49.13.155:4920 , piece_id=WMM6GIC2AJJHN2BOT7PLSKWVG7ONHG3CL3OF3PJQ5DH5ODFO24ZA, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:49.600689336 (783h48m49.471772381s) [upload=685644]: Next returned nil segment
Jan 7 07:17:49.600703433 (783h48m49.471786478s) [upload=685644]: Waiting for error group managing segments...
Jan 7 07:17:50.063713792 (783h48m49.934796847s) [noise=true , address=64.49.13.169:4713 , piece_id=4X6L55C3YY3Y35RKP2LX7BVPJME7L5JWELQ2XK2E752NT7E6KCWA, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.116339850 (783h48m49.987422895s) [noise=true , address=64.49.13.155:4920 , piece_id=WMM6GIC2AJJHN2BOT7PLSKWVG7ONHG3CL3OF3PJQ5DH5ODFO24ZA, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.154469825 (783h48m50.02555287s) [noise=true , address=145.79.227.32:4847 , piece_id=73A46W74UZTHTZBW6G35LSJZIA3ZRVFRRCC3KGJYGWZZH6NLICGA, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.158394037 (783h48m50.029477072s) [noise=true , address=145.79.227.44:4198 , piece_id=4KNZZKJ5V3SOLZKPJYY6FSCG4SIAF67RDUWRQSXPAMDY5IO32YOA, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.164547664 (783h48m50.035630709s) [noise=true , address=145.79.227.18:4017 , piece_id=XEDPCH5DMCZIQUDMILRNT4NMMVSH67CDSFA2LXLZX3SJNC7K64JQ, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.173751741 (783h48m50.044834786s) [noise=true , address=145.79.227.41:4109 , piece_id=F62DIMJUKTLEX3TO4D6FT2PGGKOCDKYY7US6QZGW7PBGNXMLQRFQ, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.181867575 (783h48m50.05295061s) [noise=true , address=145.79.227.34:4931 , piece_id=QFHCIN7ZZBYM3CFF6UVA73EFMWRKJBOWX6KBFZC2SHZ5KN3WU34Q, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.190204078 (783h48m50.061287113s) [noise=true , address=145.79.227.2:4127 , piece_id=J4JYZ2L5ULSHBLFH3N2FUKXBCDWEPVPNNNEBUW4LA3QAJYEGCBLA, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.215263345 (783h48m50.08634638s) [noise=true , address=145.79.227.33:4888 , piece_id=NDWJJUO4V4UBYYWJZJM4FQGVUDR24WTCHABZJMUGZ4FMH3U6ZK7Q, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.244106597 (783h48m50.115189632s) [noise=true , address=145.79.227.30:4777 , piece_id=BBHGIQO2AEMZWMUOGMRJC4CKRWJOTEY3WQP4I4XRGMNRIOUOH5IQ, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.468914424 (783h48m50.339997469s) [noise=true , address=77.90.31.18:4830 , piece_id=TUICFKVMJ3S2KHW7MKFNJUR6Z5UN5QFZOBXJEYC43O3BE3QQDWHA, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.533387670 (783h48m50.404470705s) [noise=true , address=77.90.31.12:4619 , piece_id=DVOJATZI5UILEARIOOZ2K3WGDFFQTFONCK2HKP2ZXKLBAEQZLROQ, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.560320373 (783h48m50.431403408s) [noise=true , address=77.90.31.11:4579 , piece_id=5LUJ6FFYUVXAD2A6DNE5IOFT22VZL4ZJ4S2NRR2MCA6GX5H3P4EQ, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.604949694 (783h48m50.476032739s) [noise=true , address=77.90.31.21:4909 , piece_id=7KH356VERAYPY7GHMEOT34TADWH4U74L2JUPMQ6Y7WF52L5I6M6A, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.604977047 (783h48m50.476060082s) [seg_pos={0 0}, upload=685644]: Segment reached optimal threshold of 14 pieces.
Jan 7 07:17:50.605285644 (783h48m50.476368689s) [noise=true , address=77.90.31.15:4727 , piece_id=UFRMVFFY2UKQLPR7P4TTN2UFNFZQ6BDPUP7INPYRJIK267R6WPTQ, seg_pos={0 0}, upload=685644]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xoAbv6zJuG9LjyCdsCrTWQaCDyLrWyv8RCrMTiqNqazoEwD1M): protocol: write tcp 188.166.225.170:47856->77.90.31.15:4727: use of closed network connection; write tcp 188.166.225.170:47856->77.90.31.15:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47856->77.90.31.15:4727: use of closed network connection
Jan 7 07:17:50.605295753 (783h48m50.476378808s) [noise=true , address=77.90.31.15:4727 , piece_id=UFRMVFFY2UKQLPR7P4TTN2UFNFZQ6BDPUP7INPYRJIK267R6WPTQ, seg_pos={0 0}, upload=685644]: Long tail cancellation due to optimal threshold
Jan 7 07:17:50.605401254 (783h48m50.476484299s) [noise=true , address=145.79.227.49:4370 , piece_id=G3ATR7KHT5DDCJNAEZDDI55XTPPZMFDYKQLAR3AD45P3GCESOHGA, seg_pos={0 0}, upload=685644]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LBgixGGgNeYs63ZSpkYJmrEGr3W9Ma1s2Sj8cCxU6mUxm1aC): protocol: expected piece hash; context canceled; EOF
Jan 7 07:17:50.605437523 (783h48m50.476520568s) [noise=true , address=145.79.227.49:4370 , piece_id=G3ATR7KHT5DDCJNAEZDDI55XTPPZMFDYKQLAR3AD45P3GCESOHGA, seg_pos={0 0}, upload=685644]: Long tail cancellation due to optimal threshold
Jan 7 07:17:50.605839750 (783h48m50.476922825s) [noise=true , address=145.79.227.43:4180 , piece_id=3ZAN75QUFNRD4K5QYAOZAYP2O4KK4SVZPXPTDOA7IYVIGLLEPYSQ, seg_pos={0 0}, upload=685644]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rhkcipPBZNFScBjBhWdigzSMwENSmzhCucnC2iuyc9udpweE3): protocol: expected piece hash; context canceled; EOF
Jan 7 07:17:50.605882010 (783h48m50.476965045s) [noise=true , address=145.79.227.43:4180 , piece_id=3ZAN75QUFNRD4K5QYAOZAYP2O4KK4SVZPXPTDOA7IYVIGLLEPYSQ, seg_pos={0 0}, upload=685644]: Long tail cancellation due to optimal threshold
Jan 7 07:17:50.605904593 (783h48m50.476987628s) [noise=true , address=145.79.227.35:4484 , piece_id=6ZQ2M55FDBKKZAGXNN7GPGZCXPKJWRGPKDL5XQWBWREGXKRSCLBQ, seg_pos={0 0}, upload=685644]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1a5S5Ar7jNbK9MESghbWfmEgBPNSMN2tEnZeeG1kKzeX2dnSX3): protocol: expected piece hash; context canceled; EOF
Jan 7 07:17:50.605924150 (783h48m50.477007205s) [noise=true , address=145.79.227.35:4484 , piece_id=6ZQ2M55FDBKKZAGXNN7GPGZCXPKJWRGPKDL5XQWBWREGXKRSCLBQ, seg_pos={0 0}, upload=685644]: Long tail cancellation due to optimal threshold
Jan 7 07:17:50.605947465 (783h48m50.47703051s) [seg_pos={0 0}, upload=685644]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jan 7 07:17:50.606035051 (783h48m50.477118086s) [upload=685644]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 7 07:17:50.606040612 (783h48m50.477123647s) [upload=685644]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 7 07:17:50.908947210 (783h48m50.780030255s) [upload=685644]: Done uploading segments...
Jan 7 07:17:50.908965275 (783h48m50.78004831s) [upload=685644]: Upload finished. err: <nil>
Duration: 5.873096439s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [.......X...............X...................X.............X..] duration:
21.353755396s trace:
https://jaeger.dev.storj.io/trace/4389ca011670ab35
FAIL (2026-01-07 07:21:18 UTC)
orbiter warning: context deadline exceeded
Duration: 21.35378306s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
13 out of 60 is failed (threshold=3) [X....X.........X..X.X..X.......XX.X....X....XX....X.........] duration:
6.176515305s trace:
https://jaeger.dev.storj.io/trace/566429788081faee
FAIL (2026-01-07 07:21:17 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
) [upload=685582]: Uploading segments...
Jan 7 07:13:32.990108051 (783h44m32.861191096s) [upload=685582]: Got next segment. Inline: false
Jan 7 07:13:32.990479758 (783h44m32.861562803s) [upload=685582]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 7 07:13:32.990486191 (783h44m32.861569226s) [upload=685582]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 7 07:13:33.230730672 (783h44m33.101813717s) [seg_pos={0 0}, upload=685582]: Begin upload segment...
Jan 7 07:13:33.230783873 (783h44m33.101866908s) [seg_pos={0 0}, upload=685582]: Done begin upload segment.
Jan 7 07:13:33.230814070 (783h44m33.101897115s) [noise=true , address=45.91.116.13:4153 , piece_id=27PHIESDJJJKDKJITJODFYQLSF42BX3BVTMOXFJAUGK73D3Z47UA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.230992941 (783h44m33.102075976s) [noise=true , address=45.91.116.5:4300 , piece_id=5FMCKFKIRBUKZGZTTBKCMURBHZTBKQPHEZMHJWJHLM2PLEDFNXSA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.231512250 (783h44m33.102595285s) [noise=true , address=145.79.227.22:4511 , piece_id=VHGXFDPNKXDZI2ARWI7227A5CF3P572VJSQZS75BMMHPAVCPPO7Q, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.231639272 (783h44m33.102722307s) [noise=true , address=145.79.227.4:4185 , piece_id=SMLAUFNVX6M24OXQ2PK2TIP5MXF3KZYRGHBFXYN7OXD5L76PCY3Q, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.231935536 (783h44m33.103018571s) [noise=true , address=145.79.227.31:4798 , piece_id=TKL7NRLI7DYTXU4HEOBJHXQR5CGHU7IG53L2GLENDCORGVVN2KOQ, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.232203125 (783h44m33.10328616s) [noise=true , address=145.79.227.38:4978 , piece_id=QV3BBDP5GVUMTBI2HIE7FWUIUOQVKIOO2D6OJNW2VDCXDFKMISUA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.232580194 (783h44m33.103663229s) [noise=true , address=145.79.227.52:4445 , piece_id=D6QKUQIBGUGV3V2ZELIM6U5UMTHT2765CHQBIKQ2N3BJF3UXH3XA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.232942814 (783h44m33.104025849s) [noise=true , address=45.91.116.8:4047 , piece_id=D6ZXOD5IBAYIVGUWNC6OGIL3OTB7JF6OAFGNGIR2RH7NDWRAEORA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.233250680 (783h44m33.104333725s) [noise=true , address=145.79.227.40:4049 , piece_id=RV43PFYU6UWU5QFHCO7T6F3E3VBNEYQRMA4UCPXT54PP3T3ZD62Q, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.233527197 (783h44m33.104610242s) [noise=true , address=77.90.31.16:4736 , piece_id=6I7NM23KI67OH2RKW4NSGYGJYWZXMH5HVLHLOQFVNCSFW4WZJ5NA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.234521901 (783h44m33.105604936s) [noise=true , address=45.91.116.12:4226 , piece_id=V7NKTZUGNA2HMXA3FSYMIXMSDLQVEUTJB74IW32GYF4XTU45UY6Q, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.235153884 (783h44m33.106236919s) [noise=true , address=145.79.227.45:4231 , piece_id=ESQLZIZB2AVESVJRHVGX4T5AJDEKVGJEJ3JSGA5J25BOQ4UUF7KA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.235615984 (783h44m33.106699019s) [noise=true , address=145.79.227.11:4729 , piece_id=5EQJ3NL4R45GFF35F2WOYWCD77UGR6XUV6B4TIZASZBCBCX6APWA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.236105967 (783h44m33.107189002s) [noise=true , address=145.79.227.5:4221 , piece_id=SHQ2ARJ6XTS7NHVSSNAXW2VR7BPS26MSX76INSRBH6KXPM4SQKEA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.236621799 (783h44m33.107704844s) [noise=true , address=145.79.227.7:4293 , piece_id=WPVYNVVDT4G5DG2HIOLUDVJNIU5O3WFII2H3GIHEZDBST4F2DKGA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.239024364 (783h44m33.110107409s) [noise=true , address=77.90.31.18:4805 , piece_id=YIHVN23OWQTVKXHZIAY4K2C65LU67NA7C6GCC63PXZHCWAZGL7DQ, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:34.837564819 (783h44m34.708647854s) [upload=685582]: Next returned nil segment
Jan 7 07:13:34.837594546 (783h44m34.708677581s) [upload=685582]: Waiting for error group managing segments...
Jan 7 07:13:35.221987466 (783h44m35.093070501s) [noise=true , address=145.79.227.38:4978 , piece_id=QV3BBDP5GVUMTBI2HIE7FWUIUOQVKIOO2D6OJNW2VDCXDFKMISUA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.224032561 (783h44m35.095115596s) [noise=true , address=145.79.227.11:4729 , piece_id=5EQJ3NL4R45GFF35F2WOYWCD77UGR6XUV6B4TIZASZBCBCX6APWA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.224625309 (783h44m35.095708355s) [noise=true , address=145.79.227.52:4445 , piece_id=D6QKUQIBGUGV3V2ZELIM6U5UMTHT2765CHQBIKQ2N3BJF3UXH3XA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.225249578 (783h44m35.096332613s) [noise=true , address=145.79.227.22:4511 , piece_id=VHGXFDPNKXDZI2ARWI7227A5CF3P572VJSQZS75BMMHPAVCPPO7Q, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.233155391 (783h44m35.104238446s) [noise=true , address=145.79.227.45:4231 , piece_id=ESQLZIZB2AVESVJRHVGX4T5AJDEKVGJEJ3JSGA5J25BOQ4UUF7KA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.244418267 (783h44m35.115501302s) [noise=true , address=145.79.227.5:4221 , piece_id=SHQ2ARJ6XTS7NHVSSNAXW2VR7BPS26MSX76INSRBH6KXPM4SQKEA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.253004497 (783h44m35.124087542s) [noise=true , address=145.79.227.31:4798 , piece_id=TKL7NRLI7DYTXU4HEOBJHXQR5CGHU7IG53L2GLENDCORGVVN2KOQ, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.260994110 (783h44m35.132077145s) [noise=true , address=145.79.227.7:4293 , piece_id=WPVYNVVDT4G5DG2HIOLUDVJNIU5O3WFII2H3GIHEZDBST4F2DKGA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.298671554 (783h44m35.169754589s) [noise=true , address=145.79.227.4:4185 , piece_id=SMLAUFNVX6M24OXQ2PK2TIP5MXF3KZYRGHBFXYN7OXD5L76PCY3Q, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.306366076 (783h44m35.177449111s) [noise=true , address=145.79.227.40:4049 , piece_id=RV43PFYU6UWU5QFHCO7T6F3E3VBNEYQRMA4UCPXT54PP3T3ZD62Q, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.519691228 (783h44m35.390774263s) [noise=true , address=77.90.31.16:4736 , piece_id=6I7NM23KI67OH2RKW4NSGYGJYWZXMH5HVLHLOQFVNCSFW4WZJ5NA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.749540043 (783h44m35.620623078s) [noise=true , address=45.91.116.5:4300 , piece_id=5FMCKFKIRBUKZGZTTBKCMURBHZTBKQPHEZMHJWJHLM2PLEDFNXSA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.823582121 (783h44m35.694665156s) [noise=true , address=45.91.116.8:4047 , piece_id=D6ZXOD5IBAYIVGUWNC6OGIL3OTB7JF6OAFGNGIR2RH7NDWRAEORA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.823600715 (783h44m35.69468375s) [seg_pos={0 0}, upload=685582]: Segment reached optimal threshold of 13 pieces.
Jan 7 07:13:35.824004615 (783h44m35.69508765s) [noise=true , address=45.91.116.12:4226 , piece_id=V7NKTZUGNA2HMXA3FSYMIXMSDLQVEUTJB74IW32GYF4XTU45UY6Q, seg_pos={0 0}, upload=685582]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: write tcp 188.166.225.170:54196->45.91.116.12:4226: use of closed network connection; write tcp 188.166.225.170:54196->45.91.116.12:4226: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54196->45.91.116.12:4226: use of closed network connection
Jan 7 07:13:35.824012139 (783h44m35.695095174s) [noise=true , address=45.91.116.12:4226 , piece_id=V7NKTZUGNA2HMXA3FSYMIXMSDLQVEUTJB74IW32GYF4XTU45UY6Q, seg_pos={0 0}, upload=685582]: Long tail cancellation due to optimal threshold
Jan 7 07:13:35.824197492 (783h44m35.695280527s) [noise=true , address=45.91.116.13:4153 , piece_id=27PHIESDJJJKDKJITJODFYQLSF42BX3BVTMOXFJAUGK73D3Z47UA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: read tcp 188.166.225.170:33886->45.91.116.13:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 7 07:13:35.824205007 (783h44m35.695288042s) [noise=true , address=45.91.116.13:4153 , piece_id=27PHIESDJJJKDKJITJODFYQLSF42BX3BVTMOXFJAUGK73D3Z47UA, seg_pos={0 0}, upload=685582]: Long tail cancellation due to optimal threshold
Jan 7 07:13:35.824576704 (783h44m35.695659739s) [noise=true , address=77.90.31.18:4805 , piece_id=YIHVN23OWQTVKXHZIAY4K2C65LU67NA7C6GCC63PXZHCWAZGL7DQ, seg_pos={0 0}, upload=685582]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: read tcp 188.166.225.170:38598->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 7 07:13:35.824596592 (783h44m35.695679637s) [noise=true , address=77.90.31.18:4805 , piece_id=YIHVN23OWQTVKXHZIAY4K2C65LU67NA7C6GCC63PXZHCWAZGL7DQ, seg_pos={0 0}, upload=685582]: Long tail cancellation due to optimal threshold
Jan 7 07:13:35.824616139 (783h44m35.695699184s) [seg_pos={0 0}, upload=685582]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 7 07:13:35.824682215 (783h44m35.69576526s) [upload=685582]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 7 07:13:35.824686843 (783h44m35.695769878s) [upload=685582]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 7 07:13:36.105700746 (783h44m35.976783791s) [upload=685582]: Done uploading segments...
Jan 7 07:13:36.105723649 (783h44m35.976806684s) [upload=685582]: Upload finished. err: <nil>
Duration: 6.176541704s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [......................................X........X........X...] duration:
35.881424736s trace:
https://jaeger.dev.storj.io/trace/621f2bd1cae766d0
FAIL (2026-01-07 07:21:17 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
927s) [noise=true , address=77.90.31.15:4727 , piece_id=UFRMVFFY2UKQLPR7P4TTN2UFNFZQ6BDPUP7INPYRJIK267R6WPTQ, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.304384464 (783h48m48.175467499s) [noise=true , address=145.79.227.18:4017 , piece_id=XEDPCH5DMCZIQUDMILRNT4NMMVSH67CDSFA2LXLZX3SJNC7K64JQ, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.304648907 (783h48m48.175731942s) [noise=true , address=145.79.227.43:4180 , piece_id=3ZAN75QUFNRD4K5QYAOZAYP2O4KK4SVZPXPTDOA7IYVIGLLEPYSQ, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.304909002 (783h48m48.175992037s) [noise=true , address=145.79.227.35:4484 , piece_id=6ZQ2M55FDBKKZAGXNN7GPGZCXPKJWRGPKDL5XQWBWREGXKRSCLBQ, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.305186050 (783h48m48.176269085s) [noise=true , address=64.49.13.169:4713 , piece_id=4X6L55C3YY3Y35RKP2LX7BVPJME7L5JWELQ2XK2E752NT7E6KCWA, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.305462475 (783h48m48.17654552s) [noise=true , address=145.79.227.41:4109 , piece_id=F62DIMJUKTLEX3TO4D6FT2PGGKOCDKYY7US6QZGW7PBGNXMLQRFQ, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.306063991 (783h48m48.177147037s) [noise=true , address=145.79.227.2:4127 , piece_id=J4JYZ2L5ULSHBLFH3N2FUKXBCDWEPVPNNNEBUW4LA3QAJYEGCBLA, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.306457621 (783h48m48.177540666s) [noise=true , address=145.79.227.49:4370 , piece_id=G3ATR7KHT5DDCJNAEZDDI55XTPPZMFDYKQLAR3AD45P3GCESOHGA, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.306836312 (783h48m48.177919357s) [noise=true , address=145.79.227.33:4888 , piece_id=NDWJJUO4V4UBYYWJZJM4FQGVUDR24WTCHABZJMUGZ4FMH3U6ZK7Q, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.307182321 (783h48m48.178265366s) [noise=true , address=145.79.227.30:4777 , piece_id=BBHGIQO2AEMZWMUOGMRJC4CKRWJOTEY3WQP4I4XRGMNRIOUOH5IQ, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.307621457 (783h48m48.178704503s) [noise=true , address=145.79.227.34:4931 , piece_id=QFHCIN7ZZBYM3CFF6UVA73EFMWRKJBOWX6KBFZC2SHZ5KN3WU34Q, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.307948189 (783h48m48.179031234s) [noise=true , address=77.90.31.21:4909 , piece_id=7KH356VERAYPY7GHMEOT34TADWH4U74L2JUPMQ6Y7WF52L5I6M6A, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.308347098 (783h48m48.179430133s) [noise=true , address=77.90.31.18:4830 , piece_id=TUICFKVMJ3S2KHW7MKFNJUR6Z5UN5QFZOBXJEYC43O3BE3QQDWHA, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.308510059 (783h48m48.179593104s) [noise=true , address=77.90.31.12:4619 , piece_id=DVOJATZI5UILEARIOOZ2K3WGDFFQTFONCK2HKP2ZXKLBAEQZLROQ, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.309050468 (783h48m48.180133514s) [noise=true , address=145.79.227.44:4198 , piece_id=4KNZZKJ5V3SOLZKPJYY6FSCG4SIAF67RDUWRQSXPAMDY5IO32YOA, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.309517858 (783h48m48.180600903s) [noise=true , address=64.49.13.155:4920 , piece_id=WMM6GIC2AJJHN2BOT7PLSKWVG7ONHG3CL3OF3PJQ5DH5ODFO24ZA, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:49.600689336 (783h48m49.471772381s) [upload=685644]: Next returned nil segment
Jan 7 07:17:49.600703433 (783h48m49.471786478s) [upload=685644]: Waiting for error group managing segments...
Jan 7 07:17:50.063713792 (783h48m49.934796847s) [noise=true , address=64.49.13.169:4713 , piece_id=4X6L55C3YY3Y35RKP2LX7BVPJME7L5JWELQ2XK2E752NT7E6KCWA, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.116339850 (783h48m49.987422895s) [noise=true , address=64.49.13.155:4920 , piece_id=WMM6GIC2AJJHN2BOT7PLSKWVG7ONHG3CL3OF3PJQ5DH5ODFO24ZA, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.154469825 (783h48m50.02555287s) [noise=true , address=145.79.227.32:4847 , piece_id=73A46W74UZTHTZBW6G35LSJZIA3ZRVFRRCC3KGJYGWZZH6NLICGA, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.158394037 (783h48m50.029477072s) [noise=true , address=145.79.227.44:4198 , piece_id=4KNZZKJ5V3SOLZKPJYY6FSCG4SIAF67RDUWRQSXPAMDY5IO32YOA, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.164547664 (783h48m50.035630709s) [noise=true , address=145.79.227.18:4017 , piece_id=XEDPCH5DMCZIQUDMILRNT4NMMVSH67CDSFA2LXLZX3SJNC7K64JQ, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.173751741 (783h48m50.044834786s) [noise=true , address=145.79.227.41:4109 , piece_id=F62DIMJUKTLEX3TO4D6FT2PGGKOCDKYY7US6QZGW7PBGNXMLQRFQ, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.181867575 (783h48m50.05295061s) [noise=true , address=145.79.227.34:4931 , piece_id=QFHCIN7ZZBYM3CFF6UVA73EFMWRKJBOWX6KBFZC2SHZ5KN3WU34Q, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.190204078 (783h48m50.061287113s) [noise=true , address=145.79.227.2:4127 , piece_id=J4JYZ2L5ULSHBLFH3N2FUKXBCDWEPVPNNNEBUW4LA3QAJYEGCBLA, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.215263345 (783h48m50.08634638s) [noise=true , address=145.79.227.33:4888 , piece_id=NDWJJUO4V4UBYYWJZJM4FQGVUDR24WTCHABZJMUGZ4FMH3U6ZK7Q, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.244106597 (783h48m50.115189632s) [noise=true , address=145.79.227.30:4777 , piece_id=BBHGIQO2AEMZWMUOGMRJC4CKRWJOTEY3WQP4I4XRGMNRIOUOH5IQ, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.468914424 (783h48m50.339997469s) [noise=true , address=77.90.31.18:4830 , piece_id=TUICFKVMJ3S2KHW7MKFNJUR6Z5UN5QFZOBXJEYC43O3BE3QQDWHA, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.533387670 (783h48m50.404470705s) [noise=true , address=77.90.31.12:4619 , piece_id=DVOJATZI5UILEARIOOZ2K3WGDFFQTFONCK2HKP2ZXKLBAEQZLROQ, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.560320373 (783h48m50.431403408s) [noise=true , address=77.90.31.11:4579 , piece_id=5LUJ6FFYUVXAD2A6DNE5IOFT22VZL4ZJ4S2NRR2MCA6GX5H3P4EQ, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.604949694 (783h48m50.476032739s) [noise=true , address=77.90.31.21:4909 , piece_id=7KH356VERAYPY7GHMEOT34TADWH4U74L2JUPMQ6Y7WF52L5I6M6A, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.604977047 (783h48m50.476060082s) [seg_pos={0 0}, upload=685644]: Segment reached optimal threshold of 14 pieces.
Jan 7 07:17:50.605285644 (783h48m50.476368689s) [noise=true , address=77.90.31.15:4727 , piece_id=UFRMVFFY2UKQLPR7P4TTN2UFNFZQ6BDPUP7INPYRJIK267R6WPTQ, seg_pos={0 0}, upload=685644]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xoAbv6zJuG9LjyCdsCrTWQaCDyLrWyv8RCrMTiqNqazoEwD1M): protocol: write tcp 188.166.225.170:47856->77.90.31.15:4727: use of closed network connection; write tcp 188.166.225.170:47856->77.90.31.15:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47856->77.90.31.15:4727: use of closed network connection
Jan 7 07:17:50.605295753 (783h48m50.476378808s) [noise=true , address=77.90.31.15:4727 , piece_id=UFRMVFFY2UKQLPR7P4TTN2UFNFZQ6BDPUP7INPYRJIK267R6WPTQ, seg_pos={0 0}, upload=685644]: Long tail cancellation due to optimal threshold
Jan 7 07:17:50.605401254 (783h48m50.476484299s) [noise=true , address=145.79.227.49:4370 , piece_id=G3ATR7KHT5DDCJNAEZDDI55XTPPZMFDYKQLAR3AD45P3GCESOHGA, seg_pos={0 0}, upload=685644]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LBgixGGgNeYs63ZSpkYJmrEGr3W9Ma1s2Sj8cCxU6mUxm1aC): protocol: expected piece hash; context canceled; EOF
Jan 7 07:17:50.605437523 (783h48m50.476520568s) [noise=true , address=145.79.227.49:4370 , piece_id=G3ATR7KHT5DDCJNAEZDDI55XTPPZMFDYKQLAR3AD45P3GCESOHGA, seg_pos={0 0}, upload=685644]: Long tail cancellation due to optimal threshold
Jan 7 07:17:50.605839750 (783h48m50.476922825s) [noise=true , address=145.79.227.43:4180 , piece_id=3ZAN75QUFNRD4K5QYAOZAYP2O4KK4SVZPXPTDOA7IYVIGLLEPYSQ, seg_pos={0 0}, upload=685644]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rhkcipPBZNFScBjBhWdigzSMwENSmzhCucnC2iuyc9udpweE3): protocol: expected piece hash; context canceled; EOF
Jan 7 07:17:50.605882010 (783h48m50.476965045s) [noise=true , address=145.79.227.43:4180 , piece_id=3ZAN75QUFNRD4K5QYAOZAYP2O4KK4SVZPXPTDOA7IYVIGLLEPYSQ, seg_pos={0 0}, upload=685644]: Long tail cancellation due to optimal threshold
Jan 7 07:17:50.605904593 (783h48m50.476987628s) [noise=true , address=145.79.227.35:4484 , piece_id=6ZQ2M55FDBKKZAGXNN7GPGZCXPKJWRGPKDL5XQWBWREGXKRSCLBQ, seg_pos={0 0}, upload=685644]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1a5S5Ar7jNbK9MESghbWfmEgBPNSMN2tEnZeeG1kKzeX2dnSX3): protocol: expected piece hash; context canceled; EOF
Jan 7 07:17:50.605924150 (783h48m50.477007205s) [noise=true , address=145.79.227.35:4484 , piece_id=6ZQ2M55FDBKKZAGXNN7GPGZCXPKJWRGPKDL5XQWBWREGXKRSCLBQ, seg_pos={0 0}, upload=685644]: Long tail cancellation due to optimal threshold
Jan 7 07:17:50.605947465 (783h48m50.47703051s) [seg_pos={0 0}, upload=685644]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jan 7 07:17:50.606035051 (783h48m50.477118086s) [upload=685644]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 7 07:17:50.606040612 (783h48m50.477123647s) [upload=685644]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 7 07:17:50.908947210 (783h48m50.780030255s) [upload=685644]: Done uploading segments...
Jan 7 07:17:50.908965275 (783h48m50.78004831s) [upload=685644]: Upload finished. err: <nil>
Duration: 35.881449043s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
16 out of 60 is failed (threshold=3) [XXX.X.X..X....X.X.....X....X.X.....X.......X....XX.........X] duration:
30.002954406s trace:
https://jaeger.dev.storj.io/trace/2cf2060b793525bb
FAIL (2026-01-07 07:21:13 UTC)
orbiter warning: context deadline exceeded
Duration: 30.002977049s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
15 out of 60 is failed (threshold=3) [.XXX.X.X..X....X.X.....X....X.X.....X.......X....XX.........] duration:
22.081171597s trace:
https://jaeger.dev.storj.io/trace/3649792573630e65
FAIL (2026-01-07 07:20:34 UTC)
orbiter warning: context deadline exceeded
Duration: 22.081232173s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
13 out of 60 is failed (threshold=3) [.X....X.........X..X.X..X.......XX.X....X....XX....X........] duration:
5.615160562s trace:
https://jaeger.dev.storj.io/trace/566429788081faee
FAIL (2026-01-07 07:20:30 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
) [upload=685582]: Uploading segments...
Jan 7 07:13:32.990108051 (783h44m32.861191096s) [upload=685582]: Got next segment. Inline: false
Jan 7 07:13:32.990479758 (783h44m32.861562803s) [upload=685582]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 7 07:13:32.990486191 (783h44m32.861569226s) [upload=685582]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 7 07:13:33.230730672 (783h44m33.101813717s) [seg_pos={0 0}, upload=685582]: Begin upload segment...
Jan 7 07:13:33.230783873 (783h44m33.101866908s) [seg_pos={0 0}, upload=685582]: Done begin upload segment.
Jan 7 07:13:33.230814070 (783h44m33.101897115s) [noise=true , address=45.91.116.13:4153 , piece_id=27PHIESDJJJKDKJITJODFYQLSF42BX3BVTMOXFJAUGK73D3Z47UA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.230992941 (783h44m33.102075976s) [noise=true , address=45.91.116.5:4300 , piece_id=5FMCKFKIRBUKZGZTTBKCMURBHZTBKQPHEZMHJWJHLM2PLEDFNXSA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.231512250 (783h44m33.102595285s) [noise=true , address=145.79.227.22:4511 , piece_id=VHGXFDPNKXDZI2ARWI7227A5CF3P572VJSQZS75BMMHPAVCPPO7Q, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.231639272 (783h44m33.102722307s) [noise=true , address=145.79.227.4:4185 , piece_id=SMLAUFNVX6M24OXQ2PK2TIP5MXF3KZYRGHBFXYN7OXD5L76PCY3Q, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.231935536 (783h44m33.103018571s) [noise=true , address=145.79.227.31:4798 , piece_id=TKL7NRLI7DYTXU4HEOBJHXQR5CGHU7IG53L2GLENDCORGVVN2KOQ, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.232203125 (783h44m33.10328616s) [noise=true , address=145.79.227.38:4978 , piece_id=QV3BBDP5GVUMTBI2HIE7FWUIUOQVKIOO2D6OJNW2VDCXDFKMISUA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.232580194 (783h44m33.103663229s) [noise=true , address=145.79.227.52:4445 , piece_id=D6QKUQIBGUGV3V2ZELIM6U5UMTHT2765CHQBIKQ2N3BJF3UXH3XA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.232942814 (783h44m33.104025849s) [noise=true , address=45.91.116.8:4047 , piece_id=D6ZXOD5IBAYIVGUWNC6OGIL3OTB7JF6OAFGNGIR2RH7NDWRAEORA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.233250680 (783h44m33.104333725s) [noise=true , address=145.79.227.40:4049 , piece_id=RV43PFYU6UWU5QFHCO7T6F3E3VBNEYQRMA4UCPXT54PP3T3ZD62Q, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.233527197 (783h44m33.104610242s) [noise=true , address=77.90.31.16:4736 , piece_id=6I7NM23KI67OH2RKW4NSGYGJYWZXMH5HVLHLOQFVNCSFW4WZJ5NA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.234521901 (783h44m33.105604936s) [noise=true , address=45.91.116.12:4226 , piece_id=V7NKTZUGNA2HMXA3FSYMIXMSDLQVEUTJB74IW32GYF4XTU45UY6Q, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.235153884 (783h44m33.106236919s) [noise=true , address=145.79.227.45:4231 , piece_id=ESQLZIZB2AVESVJRHVGX4T5AJDEKVGJEJ3JSGA5J25BOQ4UUF7KA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.235615984 (783h44m33.106699019s) [noise=true , address=145.79.227.11:4729 , piece_id=5EQJ3NL4R45GFF35F2WOYWCD77UGR6XUV6B4TIZASZBCBCX6APWA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.236105967 (783h44m33.107189002s) [noise=true , address=145.79.227.5:4221 , piece_id=SHQ2ARJ6XTS7NHVSSNAXW2VR7BPS26MSX76INSRBH6KXPM4SQKEA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.236621799 (783h44m33.107704844s) [noise=true , address=145.79.227.7:4293 , piece_id=WPVYNVVDT4G5DG2HIOLUDVJNIU5O3WFII2H3GIHEZDBST4F2DKGA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.239024364 (783h44m33.110107409s) [noise=true , address=77.90.31.18:4805 , piece_id=YIHVN23OWQTVKXHZIAY4K2C65LU67NA7C6GCC63PXZHCWAZGL7DQ, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:34.837564819 (783h44m34.708647854s) [upload=685582]: Next returned nil segment
Jan 7 07:13:34.837594546 (783h44m34.708677581s) [upload=685582]: Waiting for error group managing segments...
Jan 7 07:13:35.221987466 (783h44m35.093070501s) [noise=true , address=145.79.227.38:4978 , piece_id=QV3BBDP5GVUMTBI2HIE7FWUIUOQVKIOO2D6OJNW2VDCXDFKMISUA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.224032561 (783h44m35.095115596s) [noise=true , address=145.79.227.11:4729 , piece_id=5EQJ3NL4R45GFF35F2WOYWCD77UGR6XUV6B4TIZASZBCBCX6APWA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.224625309 (783h44m35.095708355s) [noise=true , address=145.79.227.52:4445 , piece_id=D6QKUQIBGUGV3V2ZELIM6U5UMTHT2765CHQBIKQ2N3BJF3UXH3XA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.225249578 (783h44m35.096332613s) [noise=true , address=145.79.227.22:4511 , piece_id=VHGXFDPNKXDZI2ARWI7227A5CF3P572VJSQZS75BMMHPAVCPPO7Q, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.233155391 (783h44m35.104238446s) [noise=true , address=145.79.227.45:4231 , piece_id=ESQLZIZB2AVESVJRHVGX4T5AJDEKVGJEJ3JSGA5J25BOQ4UUF7KA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.244418267 (783h44m35.115501302s) [noise=true , address=145.79.227.5:4221 , piece_id=SHQ2ARJ6XTS7NHVSSNAXW2VR7BPS26MSX76INSRBH6KXPM4SQKEA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.253004497 (783h44m35.124087542s) [noise=true , address=145.79.227.31:4798 , piece_id=TKL7NRLI7DYTXU4HEOBJHXQR5CGHU7IG53L2GLENDCORGVVN2KOQ, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.260994110 (783h44m35.132077145s) [noise=true , address=145.79.227.7:4293 , piece_id=WPVYNVVDT4G5DG2HIOLUDVJNIU5O3WFII2H3GIHEZDBST4F2DKGA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.298671554 (783h44m35.169754589s) [noise=true , address=145.79.227.4:4185 , piece_id=SMLAUFNVX6M24OXQ2PK2TIP5MXF3KZYRGHBFXYN7OXD5L76PCY3Q, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.306366076 (783h44m35.177449111s) [noise=true , address=145.79.227.40:4049 , piece_id=RV43PFYU6UWU5QFHCO7T6F3E3VBNEYQRMA4UCPXT54PP3T3ZD62Q, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.519691228 (783h44m35.390774263s) [noise=true , address=77.90.31.16:4736 , piece_id=6I7NM23KI67OH2RKW4NSGYGJYWZXMH5HVLHLOQFVNCSFW4WZJ5NA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.749540043 (783h44m35.620623078s) [noise=true , address=45.91.116.5:4300 , piece_id=5FMCKFKIRBUKZGZTTBKCMURBHZTBKQPHEZMHJWJHLM2PLEDFNXSA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.823582121 (783h44m35.694665156s) [noise=true , address=45.91.116.8:4047 , piece_id=D6ZXOD5IBAYIVGUWNC6OGIL3OTB7JF6OAFGNGIR2RH7NDWRAEORA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.823600715 (783h44m35.69468375s) [seg_pos={0 0}, upload=685582]: Segment reached optimal threshold of 13 pieces.
Jan 7 07:13:35.824004615 (783h44m35.69508765s) [noise=true , address=45.91.116.12:4226 , piece_id=V7NKTZUGNA2HMXA3FSYMIXMSDLQVEUTJB74IW32GYF4XTU45UY6Q, seg_pos={0 0}, upload=685582]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: write tcp 188.166.225.170:54196->45.91.116.12:4226: use of closed network connection; write tcp 188.166.225.170:54196->45.91.116.12:4226: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54196->45.91.116.12:4226: use of closed network connection
Jan 7 07:13:35.824012139 (783h44m35.695095174s) [noise=true , address=45.91.116.12:4226 , piece_id=V7NKTZUGNA2HMXA3FSYMIXMSDLQVEUTJB74IW32GYF4XTU45UY6Q, seg_pos={0 0}, upload=685582]: Long tail cancellation due to optimal threshold
Jan 7 07:13:35.824197492 (783h44m35.695280527s) [noise=true , address=45.91.116.13:4153 , piece_id=27PHIESDJJJKDKJITJODFYQLSF42BX3BVTMOXFJAUGK73D3Z47UA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: read tcp 188.166.225.170:33886->45.91.116.13:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 7 07:13:35.824205007 (783h44m35.695288042s) [noise=true , address=45.91.116.13:4153 , piece_id=27PHIESDJJJKDKJITJODFYQLSF42BX3BVTMOXFJAUGK73D3Z47UA, seg_pos={0 0}, upload=685582]: Long tail cancellation due to optimal threshold
Jan 7 07:13:35.824576704 (783h44m35.695659739s) [noise=true , address=77.90.31.18:4805 , piece_id=YIHVN23OWQTVKXHZIAY4K2C65LU67NA7C6GCC63PXZHCWAZGL7DQ, seg_pos={0 0}, upload=685582]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: read tcp 188.166.225.170:38598->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 7 07:13:35.824596592 (783h44m35.695679637s) [noise=true , address=77.90.31.18:4805 , piece_id=YIHVN23OWQTVKXHZIAY4K2C65LU67NA7C6GCC63PXZHCWAZGL7DQ, seg_pos={0 0}, upload=685582]: Long tail cancellation due to optimal threshold
Jan 7 07:13:35.824616139 (783h44m35.695699184s) [seg_pos={0 0}, upload=685582]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 7 07:13:35.824682215 (783h44m35.69576526s) [upload=685582]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 7 07:13:35.824686843 (783h44m35.695769878s) [upload=685582]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 7 07:13:36.105700746 (783h44m35.976783791s) [upload=685582]: Done uploading segments...
Jan 7 07:13:36.105723649 (783h44m35.976806684s) [upload=685582]: Upload finished. err: <nil>
Duration: 5.615188745s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
50 out of 60 is failed (threshold=6) [XXX.XXXXXXXXXXX.XXXX.XXXX..XXXXX.XXXXX..XXXXX.XXXX.XXXXXXXXX] duration:
5.001477112s trace:
https://jaeger.dev.storj.io/trace/203897c5cf45d824
FAIL (2026-01-07 07:20:30 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001501207s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [.......................................X........X........X..] duration:
42.292144547s trace:
https://jaeger.dev.storj.io/trace/621f2bd1cae766d0
FAIL (2026-01-07 07:20:12 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
927s) [noise=true , address=77.90.31.15:4727 , piece_id=UFRMVFFY2UKQLPR7P4TTN2UFNFZQ6BDPUP7INPYRJIK267R6WPTQ, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.304384464 (783h48m48.175467499s) [noise=true , address=145.79.227.18:4017 , piece_id=XEDPCH5DMCZIQUDMILRNT4NMMVSH67CDSFA2LXLZX3SJNC7K64JQ, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.304648907 (783h48m48.175731942s) [noise=true , address=145.79.227.43:4180 , piece_id=3ZAN75QUFNRD4K5QYAOZAYP2O4KK4SVZPXPTDOA7IYVIGLLEPYSQ, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.304909002 (783h48m48.175992037s) [noise=true , address=145.79.227.35:4484 , piece_id=6ZQ2M55FDBKKZAGXNN7GPGZCXPKJWRGPKDL5XQWBWREGXKRSCLBQ, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.305186050 (783h48m48.176269085s) [noise=true , address=64.49.13.169:4713 , piece_id=4X6L55C3YY3Y35RKP2LX7BVPJME7L5JWELQ2XK2E752NT7E6KCWA, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.305462475 (783h48m48.17654552s) [noise=true , address=145.79.227.41:4109 , piece_id=F62DIMJUKTLEX3TO4D6FT2PGGKOCDKYY7US6QZGW7PBGNXMLQRFQ, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.306063991 (783h48m48.177147037s) [noise=true , address=145.79.227.2:4127 , piece_id=J4JYZ2L5ULSHBLFH3N2FUKXBCDWEPVPNNNEBUW4LA3QAJYEGCBLA, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.306457621 (783h48m48.177540666s) [noise=true , address=145.79.227.49:4370 , piece_id=G3ATR7KHT5DDCJNAEZDDI55XTPPZMFDYKQLAR3AD45P3GCESOHGA, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.306836312 (783h48m48.177919357s) [noise=true , address=145.79.227.33:4888 , piece_id=NDWJJUO4V4UBYYWJZJM4FQGVUDR24WTCHABZJMUGZ4FMH3U6ZK7Q, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.307182321 (783h48m48.178265366s) [noise=true , address=145.79.227.30:4777 , piece_id=BBHGIQO2AEMZWMUOGMRJC4CKRWJOTEY3WQP4I4XRGMNRIOUOH5IQ, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.307621457 (783h48m48.178704503s) [noise=true , address=145.79.227.34:4931 , piece_id=QFHCIN7ZZBYM3CFF6UVA73EFMWRKJBOWX6KBFZC2SHZ5KN3WU34Q, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.307948189 (783h48m48.179031234s) [noise=true , address=77.90.31.21:4909 , piece_id=7KH356VERAYPY7GHMEOT34TADWH4U74L2JUPMQ6Y7WF52L5I6M6A, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.308347098 (783h48m48.179430133s) [noise=true , address=77.90.31.18:4830 , piece_id=TUICFKVMJ3S2KHW7MKFNJUR6Z5UN5QFZOBXJEYC43O3BE3QQDWHA, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.308510059 (783h48m48.179593104s) [noise=true , address=77.90.31.12:4619 , piece_id=DVOJATZI5UILEARIOOZ2K3WGDFFQTFONCK2HKP2ZXKLBAEQZLROQ, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.309050468 (783h48m48.180133514s) [noise=true , address=145.79.227.44:4198 , piece_id=4KNZZKJ5V3SOLZKPJYY6FSCG4SIAF67RDUWRQSXPAMDY5IO32YOA, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.309517858 (783h48m48.180600903s) [noise=true , address=64.49.13.155:4920 , piece_id=WMM6GIC2AJJHN2BOT7PLSKWVG7ONHG3CL3OF3PJQ5DH5ODFO24ZA, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:49.600689336 (783h48m49.471772381s) [upload=685644]: Next returned nil segment
Jan 7 07:17:49.600703433 (783h48m49.471786478s) [upload=685644]: Waiting for error group managing segments...
Jan 7 07:17:50.063713792 (783h48m49.934796847s) [noise=true , address=64.49.13.169:4713 , piece_id=4X6L55C3YY3Y35RKP2LX7BVPJME7L5JWELQ2XK2E752NT7E6KCWA, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.116339850 (783h48m49.987422895s) [noise=true , address=64.49.13.155:4920 , piece_id=WMM6GIC2AJJHN2BOT7PLSKWVG7ONHG3CL3OF3PJQ5DH5ODFO24ZA, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.154469825 (783h48m50.02555287s) [noise=true , address=145.79.227.32:4847 , piece_id=73A46W74UZTHTZBW6G35LSJZIA3ZRVFRRCC3KGJYGWZZH6NLICGA, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.158394037 (783h48m50.029477072s) [noise=true , address=145.79.227.44:4198 , piece_id=4KNZZKJ5V3SOLZKPJYY6FSCG4SIAF67RDUWRQSXPAMDY5IO32YOA, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.164547664 (783h48m50.035630709s) [noise=true , address=145.79.227.18:4017 , piece_id=XEDPCH5DMCZIQUDMILRNT4NMMVSH67CDSFA2LXLZX3SJNC7K64JQ, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.173751741 (783h48m50.044834786s) [noise=true , address=145.79.227.41:4109 , piece_id=F62DIMJUKTLEX3TO4D6FT2PGGKOCDKYY7US6QZGW7PBGNXMLQRFQ, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.181867575 (783h48m50.05295061s) [noise=true , address=145.79.227.34:4931 , piece_id=QFHCIN7ZZBYM3CFF6UVA73EFMWRKJBOWX6KBFZC2SHZ5KN3WU34Q, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.190204078 (783h48m50.061287113s) [noise=true , address=145.79.227.2:4127 , piece_id=J4JYZ2L5ULSHBLFH3N2FUKXBCDWEPVPNNNEBUW4LA3QAJYEGCBLA, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.215263345 (783h48m50.08634638s) [noise=true , address=145.79.227.33:4888 , piece_id=NDWJJUO4V4UBYYWJZJM4FQGVUDR24WTCHABZJMUGZ4FMH3U6ZK7Q, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.244106597 (783h48m50.115189632s) [noise=true , address=145.79.227.30:4777 , piece_id=BBHGIQO2AEMZWMUOGMRJC4CKRWJOTEY3WQP4I4XRGMNRIOUOH5IQ, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.468914424 (783h48m50.339997469s) [noise=true , address=77.90.31.18:4830 , piece_id=TUICFKVMJ3S2KHW7MKFNJUR6Z5UN5QFZOBXJEYC43O3BE3QQDWHA, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.533387670 (783h48m50.404470705s) [noise=true , address=77.90.31.12:4619 , piece_id=DVOJATZI5UILEARIOOZ2K3WGDFFQTFONCK2HKP2ZXKLBAEQZLROQ, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.560320373 (783h48m50.431403408s) [noise=true , address=77.90.31.11:4579 , piece_id=5LUJ6FFYUVXAD2A6DNE5IOFT22VZL4ZJ4S2NRR2MCA6GX5H3P4EQ, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.604949694 (783h48m50.476032739s) [noise=true , address=77.90.31.21:4909 , piece_id=7KH356VERAYPY7GHMEOT34TADWH4U74L2JUPMQ6Y7WF52L5I6M6A, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.604977047 (783h48m50.476060082s) [seg_pos={0 0}, upload=685644]: Segment reached optimal threshold of 14 pieces.
Jan 7 07:17:50.605285644 (783h48m50.476368689s) [noise=true , address=77.90.31.15:4727 , piece_id=UFRMVFFY2UKQLPR7P4TTN2UFNFZQ6BDPUP7INPYRJIK267R6WPTQ, seg_pos={0 0}, upload=685644]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xoAbv6zJuG9LjyCdsCrTWQaCDyLrWyv8RCrMTiqNqazoEwD1M): protocol: write tcp 188.166.225.170:47856->77.90.31.15:4727: use of closed network connection; write tcp 188.166.225.170:47856->77.90.31.15:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47856->77.90.31.15:4727: use of closed network connection
Jan 7 07:17:50.605295753 (783h48m50.476378808s) [noise=true , address=77.90.31.15:4727 , piece_id=UFRMVFFY2UKQLPR7P4TTN2UFNFZQ6BDPUP7INPYRJIK267R6WPTQ, seg_pos={0 0}, upload=685644]: Long tail cancellation due to optimal threshold
Jan 7 07:17:50.605401254 (783h48m50.476484299s) [noise=true , address=145.79.227.49:4370 , piece_id=G3ATR7KHT5DDCJNAEZDDI55XTPPZMFDYKQLAR3AD45P3GCESOHGA, seg_pos={0 0}, upload=685644]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LBgixGGgNeYs63ZSpkYJmrEGr3W9Ma1s2Sj8cCxU6mUxm1aC): protocol: expected piece hash; context canceled; EOF
Jan 7 07:17:50.605437523 (783h48m50.476520568s) [noise=true , address=145.79.227.49:4370 , piece_id=G3ATR7KHT5DDCJNAEZDDI55XTPPZMFDYKQLAR3AD45P3GCESOHGA, seg_pos={0 0}, upload=685644]: Long tail cancellation due to optimal threshold
Jan 7 07:17:50.605839750 (783h48m50.476922825s) [noise=true , address=145.79.227.43:4180 , piece_id=3ZAN75QUFNRD4K5QYAOZAYP2O4KK4SVZPXPTDOA7IYVIGLLEPYSQ, seg_pos={0 0}, upload=685644]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rhkcipPBZNFScBjBhWdigzSMwENSmzhCucnC2iuyc9udpweE3): protocol: expected piece hash; context canceled; EOF
Jan 7 07:17:50.605882010 (783h48m50.476965045s) [noise=true , address=145.79.227.43:4180 , piece_id=3ZAN75QUFNRD4K5QYAOZAYP2O4KK4SVZPXPTDOA7IYVIGLLEPYSQ, seg_pos={0 0}, upload=685644]: Long tail cancellation due to optimal threshold
Jan 7 07:17:50.605904593 (783h48m50.476987628s) [noise=true , address=145.79.227.35:4484 , piece_id=6ZQ2M55FDBKKZAGXNN7GPGZCXPKJWRGPKDL5XQWBWREGXKRSCLBQ, seg_pos={0 0}, upload=685644]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1a5S5Ar7jNbK9MESghbWfmEgBPNSMN2tEnZeeG1kKzeX2dnSX3): protocol: expected piece hash; context canceled; EOF
Jan 7 07:17:50.605924150 (783h48m50.477007205s) [noise=true , address=145.79.227.35:4484 , piece_id=6ZQ2M55FDBKKZAGXNN7GPGZCXPKJWRGPKDL5XQWBWREGXKRSCLBQ, seg_pos={0 0}, upload=685644]: Long tail cancellation due to optimal threshold
Jan 7 07:17:50.605947465 (783h48m50.47703051s) [seg_pos={0 0}, upload=685644]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jan 7 07:17:50.606035051 (783h48m50.477118086s) [upload=685644]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 7 07:17:50.606040612 (783h48m50.477123647s) [upload=685644]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 7 07:17:50.908947210 (783h48m50.780030255s) [upload=685644]: Done uploading segments...
Jan 7 07:17:50.908965275 (783h48m50.78004831s) [upload=685644]: Upload finished. err: <nil>
Duration: 42.292175035s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [........X...............X...................X.............X.] duration:
21.025664211s trace:
https://jaeger.dev.storj.io/trace/4389ca011670ab35
FAIL (2026-01-07 07:20:09 UTC)
orbiter warning: context deadline exceeded
Duration: 21.025681353s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
13 out of 60 is failed (threshold=3) [..X....X.........X..X.X..X.......XX.X....X....XX....X.......] duration:
6.403419528s trace:
https://jaeger.dev.storj.io/trace/566429788081faee
FAIL (2026-01-07 07:20:08 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
) [upload=685582]: Uploading segments...
Jan 7 07:13:32.990108051 (783h44m32.861191096s) [upload=685582]: Got next segment. Inline: false
Jan 7 07:13:32.990479758 (783h44m32.861562803s) [upload=685582]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 7 07:13:32.990486191 (783h44m32.861569226s) [upload=685582]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 7 07:13:33.230730672 (783h44m33.101813717s) [seg_pos={0 0}, upload=685582]: Begin upload segment...
Jan 7 07:13:33.230783873 (783h44m33.101866908s) [seg_pos={0 0}, upload=685582]: Done begin upload segment.
Jan 7 07:13:33.230814070 (783h44m33.101897115s) [noise=true , address=45.91.116.13:4153 , piece_id=27PHIESDJJJKDKJITJODFYQLSF42BX3BVTMOXFJAUGK73D3Z47UA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.230992941 (783h44m33.102075976s) [noise=true , address=45.91.116.5:4300 , piece_id=5FMCKFKIRBUKZGZTTBKCMURBHZTBKQPHEZMHJWJHLM2PLEDFNXSA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.231512250 (783h44m33.102595285s) [noise=true , address=145.79.227.22:4511 , piece_id=VHGXFDPNKXDZI2ARWI7227A5CF3P572VJSQZS75BMMHPAVCPPO7Q, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.231639272 (783h44m33.102722307s) [noise=true , address=145.79.227.4:4185 , piece_id=SMLAUFNVX6M24OXQ2PK2TIP5MXF3KZYRGHBFXYN7OXD5L76PCY3Q, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.231935536 (783h44m33.103018571s) [noise=true , address=145.79.227.31:4798 , piece_id=TKL7NRLI7DYTXU4HEOBJHXQR5CGHU7IG53L2GLENDCORGVVN2KOQ, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.232203125 (783h44m33.10328616s) [noise=true , address=145.79.227.38:4978 , piece_id=QV3BBDP5GVUMTBI2HIE7FWUIUOQVKIOO2D6OJNW2VDCXDFKMISUA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.232580194 (783h44m33.103663229s) [noise=true , address=145.79.227.52:4445 , piece_id=D6QKUQIBGUGV3V2ZELIM6U5UMTHT2765CHQBIKQ2N3BJF3UXH3XA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.232942814 (783h44m33.104025849s) [noise=true , address=45.91.116.8:4047 , piece_id=D6ZXOD5IBAYIVGUWNC6OGIL3OTB7JF6OAFGNGIR2RH7NDWRAEORA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.233250680 (783h44m33.104333725s) [noise=true , address=145.79.227.40:4049 , piece_id=RV43PFYU6UWU5QFHCO7T6F3E3VBNEYQRMA4UCPXT54PP3T3ZD62Q, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.233527197 (783h44m33.104610242s) [noise=true , address=77.90.31.16:4736 , piece_id=6I7NM23KI67OH2RKW4NSGYGJYWZXMH5HVLHLOQFVNCSFW4WZJ5NA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.234521901 (783h44m33.105604936s) [noise=true , address=45.91.116.12:4226 , piece_id=V7NKTZUGNA2HMXA3FSYMIXMSDLQVEUTJB74IW32GYF4XTU45UY6Q, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.235153884 (783h44m33.106236919s) [noise=true , address=145.79.227.45:4231 , piece_id=ESQLZIZB2AVESVJRHVGX4T5AJDEKVGJEJ3JSGA5J25BOQ4UUF7KA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.235615984 (783h44m33.106699019s) [noise=true , address=145.79.227.11:4729 , piece_id=5EQJ3NL4R45GFF35F2WOYWCD77UGR6XUV6B4TIZASZBCBCX6APWA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.236105967 (783h44m33.107189002s) [noise=true , address=145.79.227.5:4221 , piece_id=SHQ2ARJ6XTS7NHVSSNAXW2VR7BPS26MSX76INSRBH6KXPM4SQKEA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.236621799 (783h44m33.107704844s) [noise=true , address=145.79.227.7:4293 , piece_id=WPVYNVVDT4G5DG2HIOLUDVJNIU5O3WFII2H3GIHEZDBST4F2DKGA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.239024364 (783h44m33.110107409s) [noise=true , address=77.90.31.18:4805 , piece_id=YIHVN23OWQTVKXHZIAY4K2C65LU67NA7C6GCC63PXZHCWAZGL7DQ, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:34.837564819 (783h44m34.708647854s) [upload=685582]: Next returned nil segment
Jan 7 07:13:34.837594546 (783h44m34.708677581s) [upload=685582]: Waiting for error group managing segments...
Jan 7 07:13:35.221987466 (783h44m35.093070501s) [noise=true , address=145.79.227.38:4978 , piece_id=QV3BBDP5GVUMTBI2HIE7FWUIUOQVKIOO2D6OJNW2VDCXDFKMISUA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.224032561 (783h44m35.095115596s) [noise=true , address=145.79.227.11:4729 , piece_id=5EQJ3NL4R45GFF35F2WOYWCD77UGR6XUV6B4TIZASZBCBCX6APWA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.224625309 (783h44m35.095708355s) [noise=true , address=145.79.227.52:4445 , piece_id=D6QKUQIBGUGV3V2ZELIM6U5UMTHT2765CHQBIKQ2N3BJF3UXH3XA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.225249578 (783h44m35.096332613s) [noise=true , address=145.79.227.22:4511 , piece_id=VHGXFDPNKXDZI2ARWI7227A5CF3P572VJSQZS75BMMHPAVCPPO7Q, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.233155391 (783h44m35.104238446s) [noise=true , address=145.79.227.45:4231 , piece_id=ESQLZIZB2AVESVJRHVGX4T5AJDEKVGJEJ3JSGA5J25BOQ4UUF7KA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.244418267 (783h44m35.115501302s) [noise=true , address=145.79.227.5:4221 , piece_id=SHQ2ARJ6XTS7NHVSSNAXW2VR7BPS26MSX76INSRBH6KXPM4SQKEA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.253004497 (783h44m35.124087542s) [noise=true , address=145.79.227.31:4798 , piece_id=TKL7NRLI7DYTXU4HEOBJHXQR5CGHU7IG53L2GLENDCORGVVN2KOQ, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.260994110 (783h44m35.132077145s) [noise=true , address=145.79.227.7:4293 , piece_id=WPVYNVVDT4G5DG2HIOLUDVJNIU5O3WFII2H3GIHEZDBST4F2DKGA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.298671554 (783h44m35.169754589s) [noise=true , address=145.79.227.4:4185 , piece_id=SMLAUFNVX6M24OXQ2PK2TIP5MXF3KZYRGHBFXYN7OXD5L76PCY3Q, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.306366076 (783h44m35.177449111s) [noise=true , address=145.79.227.40:4049 , piece_id=RV43PFYU6UWU5QFHCO7T6F3E3VBNEYQRMA4UCPXT54PP3T3ZD62Q, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.519691228 (783h44m35.390774263s) [noise=true , address=77.90.31.16:4736 , piece_id=6I7NM23KI67OH2RKW4NSGYGJYWZXMH5HVLHLOQFVNCSFW4WZJ5NA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.749540043 (783h44m35.620623078s) [noise=true , address=45.91.116.5:4300 , piece_id=5FMCKFKIRBUKZGZTTBKCMURBHZTBKQPHEZMHJWJHLM2PLEDFNXSA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.823582121 (783h44m35.694665156s) [noise=true , address=45.91.116.8:4047 , piece_id=D6ZXOD5IBAYIVGUWNC6OGIL3OTB7JF6OAFGNGIR2RH7NDWRAEORA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.823600715 (783h44m35.69468375s) [seg_pos={0 0}, upload=685582]: Segment reached optimal threshold of 13 pieces.
Jan 7 07:13:35.824004615 (783h44m35.69508765s) [noise=true , address=45.91.116.12:4226 , piece_id=V7NKTZUGNA2HMXA3FSYMIXMSDLQVEUTJB74IW32GYF4XTU45UY6Q, seg_pos={0 0}, upload=685582]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: write tcp 188.166.225.170:54196->45.91.116.12:4226: use of closed network connection; write tcp 188.166.225.170:54196->45.91.116.12:4226: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54196->45.91.116.12:4226: use of closed network connection
Jan 7 07:13:35.824012139 (783h44m35.695095174s) [noise=true , address=45.91.116.12:4226 , piece_id=V7NKTZUGNA2HMXA3FSYMIXMSDLQVEUTJB74IW32GYF4XTU45UY6Q, seg_pos={0 0}, upload=685582]: Long tail cancellation due to optimal threshold
Jan 7 07:13:35.824197492 (783h44m35.695280527s) [noise=true , address=45.91.116.13:4153 , piece_id=27PHIESDJJJKDKJITJODFYQLSF42BX3BVTMOXFJAUGK73D3Z47UA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: read tcp 188.166.225.170:33886->45.91.116.13:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 7 07:13:35.824205007 (783h44m35.695288042s) [noise=true , address=45.91.116.13:4153 , piece_id=27PHIESDJJJKDKJITJODFYQLSF42BX3BVTMOXFJAUGK73D3Z47UA, seg_pos={0 0}, upload=685582]: Long tail cancellation due to optimal threshold
Jan 7 07:13:35.824576704 (783h44m35.695659739s) [noise=true , address=77.90.31.18:4805 , piece_id=YIHVN23OWQTVKXHZIAY4K2C65LU67NA7C6GCC63PXZHCWAZGL7DQ, seg_pos={0 0}, upload=685582]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: read tcp 188.166.225.170:38598->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 7 07:13:35.824596592 (783h44m35.695679637s) [noise=true , address=77.90.31.18:4805 , piece_id=YIHVN23OWQTVKXHZIAY4K2C65LU67NA7C6GCC63PXZHCWAZGL7DQ, seg_pos={0 0}, upload=685582]: Long tail cancellation due to optimal threshold
Jan 7 07:13:35.824616139 (783h44m35.695699184s) [seg_pos={0 0}, upload=685582]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 7 07:13:35.824682215 (783h44m35.69576526s) [upload=685582]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 7 07:13:35.824686843 (783h44m35.695769878s) [upload=685582]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 7 07:13:36.105700746 (783h44m35.976783791s) [upload=685582]: Done uploading segments...
Jan 7 07:13:36.105723649 (783h44m35.976806684s) [upload=685582]: Upload finished. err: <nil>
Duration: 6.403453944s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
16 out of 60 is failed (threshold=3) [X.XXX.X.X..X....X.X.....X....X.X.....X.......X....XX........] duration:
21.168468973s trace:
https://jaeger.dev.storj.io/trace/3649792573630e65
FAIL (2026-01-07 07:20:01 UTC)
orbiter warning: context deadline exceeded
Duration: 21.168528135s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [.........X...............X...................X.............X] duration:
1m0.002070533s trace:
https://jaeger.dev.storj.io/trace/4389ca011670ab35
FAIL (2026-01-07 07:19:48 UTC)
orbiter warning: context deadline exceeded
Duration: 1m0.002093035s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
16 out of 60 is failed (threshold=3) [.X.XXX.X.X..X....X.X.....X....X.X.....X.......X....XX.......] duration:
22.007938718s trace:
https://jaeger.dev.storj.io/trace/3649792573630e65
FAIL (2026-01-07 07:19:39 UTC)
orbiter warning: context deadline exceeded
Duration: 22.00796562s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [........................................X........X........X.] duration:
42.143119558s trace:
https://jaeger.dev.storj.io/trace/621f2bd1cae766d0
FAIL (2026-01-07 07:19:29 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
927s) [noise=true , address=77.90.31.15:4727 , piece_id=UFRMVFFY2UKQLPR7P4TTN2UFNFZQ6BDPUP7INPYRJIK267R6WPTQ, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.304384464 (783h48m48.175467499s) [noise=true , address=145.79.227.18:4017 , piece_id=XEDPCH5DMCZIQUDMILRNT4NMMVSH67CDSFA2LXLZX3SJNC7K64JQ, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.304648907 (783h48m48.175731942s) [noise=true , address=145.79.227.43:4180 , piece_id=3ZAN75QUFNRD4K5QYAOZAYP2O4KK4SVZPXPTDOA7IYVIGLLEPYSQ, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.304909002 (783h48m48.175992037s) [noise=true , address=145.79.227.35:4484 , piece_id=6ZQ2M55FDBKKZAGXNN7GPGZCXPKJWRGPKDL5XQWBWREGXKRSCLBQ, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.305186050 (783h48m48.176269085s) [noise=true , address=64.49.13.169:4713 , piece_id=4X6L55C3YY3Y35RKP2LX7BVPJME7L5JWELQ2XK2E752NT7E6KCWA, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.305462475 (783h48m48.17654552s) [noise=true , address=145.79.227.41:4109 , piece_id=F62DIMJUKTLEX3TO4D6FT2PGGKOCDKYY7US6QZGW7PBGNXMLQRFQ, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.306063991 (783h48m48.177147037s) [noise=true , address=145.79.227.2:4127 , piece_id=J4JYZ2L5ULSHBLFH3N2FUKXBCDWEPVPNNNEBUW4LA3QAJYEGCBLA, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.306457621 (783h48m48.177540666s) [noise=true , address=145.79.227.49:4370 , piece_id=G3ATR7KHT5DDCJNAEZDDI55XTPPZMFDYKQLAR3AD45P3GCESOHGA, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.306836312 (783h48m48.177919357s) [noise=true , address=145.79.227.33:4888 , piece_id=NDWJJUO4V4UBYYWJZJM4FQGVUDR24WTCHABZJMUGZ4FMH3U6ZK7Q, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.307182321 (783h48m48.178265366s) [noise=true , address=145.79.227.30:4777 , piece_id=BBHGIQO2AEMZWMUOGMRJC4CKRWJOTEY3WQP4I4XRGMNRIOUOH5IQ, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.307621457 (783h48m48.178704503s) [noise=true , address=145.79.227.34:4931 , piece_id=QFHCIN7ZZBYM3CFF6UVA73EFMWRKJBOWX6KBFZC2SHZ5KN3WU34Q, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.307948189 (783h48m48.179031234s) [noise=true , address=77.90.31.21:4909 , piece_id=7KH356VERAYPY7GHMEOT34TADWH4U74L2JUPMQ6Y7WF52L5I6M6A, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.308347098 (783h48m48.179430133s) [noise=true , address=77.90.31.18:4830 , piece_id=TUICFKVMJ3S2KHW7MKFNJUR6Z5UN5QFZOBXJEYC43O3BE3QQDWHA, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.308510059 (783h48m48.179593104s) [noise=true , address=77.90.31.12:4619 , piece_id=DVOJATZI5UILEARIOOZ2K3WGDFFQTFONCK2HKP2ZXKLBAEQZLROQ, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.309050468 (783h48m48.180133514s) [noise=true , address=145.79.227.44:4198 , piece_id=4KNZZKJ5V3SOLZKPJYY6FSCG4SIAF67RDUWRQSXPAMDY5IO32YOA, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.309517858 (783h48m48.180600903s) [noise=true , address=64.49.13.155:4920 , piece_id=WMM6GIC2AJJHN2BOT7PLSKWVG7ONHG3CL3OF3PJQ5DH5ODFO24ZA, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:49.600689336 (783h48m49.471772381s) [upload=685644]: Next returned nil segment
Jan 7 07:17:49.600703433 (783h48m49.471786478s) [upload=685644]: Waiting for error group managing segments...
Jan 7 07:17:50.063713792 (783h48m49.934796847s) [noise=true , address=64.49.13.169:4713 , piece_id=4X6L55C3YY3Y35RKP2LX7BVPJME7L5JWELQ2XK2E752NT7E6KCWA, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.116339850 (783h48m49.987422895s) [noise=true , address=64.49.13.155:4920 , piece_id=WMM6GIC2AJJHN2BOT7PLSKWVG7ONHG3CL3OF3PJQ5DH5ODFO24ZA, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.154469825 (783h48m50.02555287s) [noise=true , address=145.79.227.32:4847 , piece_id=73A46W74UZTHTZBW6G35LSJZIA3ZRVFRRCC3KGJYGWZZH6NLICGA, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.158394037 (783h48m50.029477072s) [noise=true , address=145.79.227.44:4198 , piece_id=4KNZZKJ5V3SOLZKPJYY6FSCG4SIAF67RDUWRQSXPAMDY5IO32YOA, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.164547664 (783h48m50.035630709s) [noise=true , address=145.79.227.18:4017 , piece_id=XEDPCH5DMCZIQUDMILRNT4NMMVSH67CDSFA2LXLZX3SJNC7K64JQ, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.173751741 (783h48m50.044834786s) [noise=true , address=145.79.227.41:4109 , piece_id=F62DIMJUKTLEX3TO4D6FT2PGGKOCDKYY7US6QZGW7PBGNXMLQRFQ, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.181867575 (783h48m50.05295061s) [noise=true , address=145.79.227.34:4931 , piece_id=QFHCIN7ZZBYM3CFF6UVA73EFMWRKJBOWX6KBFZC2SHZ5KN3WU34Q, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.190204078 (783h48m50.061287113s) [noise=true , address=145.79.227.2:4127 , piece_id=J4JYZ2L5ULSHBLFH3N2FUKXBCDWEPVPNNNEBUW4LA3QAJYEGCBLA, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.215263345 (783h48m50.08634638s) [noise=true , address=145.79.227.33:4888 , piece_id=NDWJJUO4V4UBYYWJZJM4FQGVUDR24WTCHABZJMUGZ4FMH3U6ZK7Q, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.244106597 (783h48m50.115189632s) [noise=true , address=145.79.227.30:4777 , piece_id=BBHGIQO2AEMZWMUOGMRJC4CKRWJOTEY3WQP4I4XRGMNRIOUOH5IQ, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.468914424 (783h48m50.339997469s) [noise=true , address=77.90.31.18:4830 , piece_id=TUICFKVMJ3S2KHW7MKFNJUR6Z5UN5QFZOBXJEYC43O3BE3QQDWHA, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.533387670 (783h48m50.404470705s) [noise=true , address=77.90.31.12:4619 , piece_id=DVOJATZI5UILEARIOOZ2K3WGDFFQTFONCK2HKP2ZXKLBAEQZLROQ, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.560320373 (783h48m50.431403408s) [noise=true , address=77.90.31.11:4579 , piece_id=5LUJ6FFYUVXAD2A6DNE5IOFT22VZL4ZJ4S2NRR2MCA6GX5H3P4EQ, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.604949694 (783h48m50.476032739s) [noise=true , address=77.90.31.21:4909 , piece_id=7KH356VERAYPY7GHMEOT34TADWH4U74L2JUPMQ6Y7WF52L5I6M6A, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.604977047 (783h48m50.476060082s) [seg_pos={0 0}, upload=685644]: Segment reached optimal threshold of 14 pieces.
Jan 7 07:17:50.605285644 (783h48m50.476368689s) [noise=true , address=77.90.31.15:4727 , piece_id=UFRMVFFY2UKQLPR7P4TTN2UFNFZQ6BDPUP7INPYRJIK267R6WPTQ, seg_pos={0 0}, upload=685644]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xoAbv6zJuG9LjyCdsCrTWQaCDyLrWyv8RCrMTiqNqazoEwD1M): protocol: write tcp 188.166.225.170:47856->77.90.31.15:4727: use of closed network connection; write tcp 188.166.225.170:47856->77.90.31.15:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47856->77.90.31.15:4727: use of closed network connection
Jan 7 07:17:50.605295753 (783h48m50.476378808s) [noise=true , address=77.90.31.15:4727 , piece_id=UFRMVFFY2UKQLPR7P4TTN2UFNFZQ6BDPUP7INPYRJIK267R6WPTQ, seg_pos={0 0}, upload=685644]: Long tail cancellation due to optimal threshold
Jan 7 07:17:50.605401254 (783h48m50.476484299s) [noise=true , address=145.79.227.49:4370 , piece_id=G3ATR7KHT5DDCJNAEZDDI55XTPPZMFDYKQLAR3AD45P3GCESOHGA, seg_pos={0 0}, upload=685644]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LBgixGGgNeYs63ZSpkYJmrEGr3W9Ma1s2Sj8cCxU6mUxm1aC): protocol: expected piece hash; context canceled; EOF
Jan 7 07:17:50.605437523 (783h48m50.476520568s) [noise=true , address=145.79.227.49:4370 , piece_id=G3ATR7KHT5DDCJNAEZDDI55XTPPZMFDYKQLAR3AD45P3GCESOHGA, seg_pos={0 0}, upload=685644]: Long tail cancellation due to optimal threshold
Jan 7 07:17:50.605839750 (783h48m50.476922825s) [noise=true , address=145.79.227.43:4180 , piece_id=3ZAN75QUFNRD4K5QYAOZAYP2O4KK4SVZPXPTDOA7IYVIGLLEPYSQ, seg_pos={0 0}, upload=685644]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rhkcipPBZNFScBjBhWdigzSMwENSmzhCucnC2iuyc9udpweE3): protocol: expected piece hash; context canceled; EOF
Jan 7 07:17:50.605882010 (783h48m50.476965045s) [noise=true , address=145.79.227.43:4180 , piece_id=3ZAN75QUFNRD4K5QYAOZAYP2O4KK4SVZPXPTDOA7IYVIGLLEPYSQ, seg_pos={0 0}, upload=685644]: Long tail cancellation due to optimal threshold
Jan 7 07:17:50.605904593 (783h48m50.476987628s) [noise=true , address=145.79.227.35:4484 , piece_id=6ZQ2M55FDBKKZAGXNN7GPGZCXPKJWRGPKDL5XQWBWREGXKRSCLBQ, seg_pos={0 0}, upload=685644]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1a5S5Ar7jNbK9MESghbWfmEgBPNSMN2tEnZeeG1kKzeX2dnSX3): protocol: expected piece hash; context canceled; EOF
Jan 7 07:17:50.605924150 (783h48m50.477007205s) [noise=true , address=145.79.227.35:4484 , piece_id=6ZQ2M55FDBKKZAGXNN7GPGZCXPKJWRGPKDL5XQWBWREGXKRSCLBQ, seg_pos={0 0}, upload=685644]: Long tail cancellation due to optimal threshold
Jan 7 07:17:50.605947465 (783h48m50.47703051s) [seg_pos={0 0}, upload=685644]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jan 7 07:17:50.606035051 (783h48m50.477118086s) [upload=685644]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 7 07:17:50.606040612 (783h48m50.477123647s) [upload=685644]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 7 07:17:50.908947210 (783h48m50.780030255s) [upload=685644]: Done uploading segments...
Jan 7 07:17:50.908965275 (783h48m50.78004831s) [upload=685644]: Upload finished. err: <nil>
Duration: 42.143134086s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
13 out of 60 is failed (threshold=3) [...X....X.........X..X.X..X.......XX.X....X....XX....X......] duration:
6.157822879s trace:
https://jaeger.dev.storj.io/trace/566429788081faee
FAIL (2026-01-07 07:19:28 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
) [upload=685582]: Uploading segments...
Jan 7 07:13:32.990108051 (783h44m32.861191096s) [upload=685582]: Got next segment. Inline: false
Jan 7 07:13:32.990479758 (783h44m32.861562803s) [upload=685582]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 7 07:13:32.990486191 (783h44m32.861569226s) [upload=685582]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 7 07:13:33.230730672 (783h44m33.101813717s) [seg_pos={0 0}, upload=685582]: Begin upload segment...
Jan 7 07:13:33.230783873 (783h44m33.101866908s) [seg_pos={0 0}, upload=685582]: Done begin upload segment.
Jan 7 07:13:33.230814070 (783h44m33.101897115s) [noise=true , address=45.91.116.13:4153 , piece_id=27PHIESDJJJKDKJITJODFYQLSF42BX3BVTMOXFJAUGK73D3Z47UA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.230992941 (783h44m33.102075976s) [noise=true , address=45.91.116.5:4300 , piece_id=5FMCKFKIRBUKZGZTTBKCMURBHZTBKQPHEZMHJWJHLM2PLEDFNXSA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.231512250 (783h44m33.102595285s) [noise=true , address=145.79.227.22:4511 , piece_id=VHGXFDPNKXDZI2ARWI7227A5CF3P572VJSQZS75BMMHPAVCPPO7Q, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.231639272 (783h44m33.102722307s) [noise=true , address=145.79.227.4:4185 , piece_id=SMLAUFNVX6M24OXQ2PK2TIP5MXF3KZYRGHBFXYN7OXD5L76PCY3Q, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.231935536 (783h44m33.103018571s) [noise=true , address=145.79.227.31:4798 , piece_id=TKL7NRLI7DYTXU4HEOBJHXQR5CGHU7IG53L2GLENDCORGVVN2KOQ, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.232203125 (783h44m33.10328616s) [noise=true , address=145.79.227.38:4978 , piece_id=QV3BBDP5GVUMTBI2HIE7FWUIUOQVKIOO2D6OJNW2VDCXDFKMISUA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.232580194 (783h44m33.103663229s) [noise=true , address=145.79.227.52:4445 , piece_id=D6QKUQIBGUGV3V2ZELIM6U5UMTHT2765CHQBIKQ2N3BJF3UXH3XA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.232942814 (783h44m33.104025849s) [noise=true , address=45.91.116.8:4047 , piece_id=D6ZXOD5IBAYIVGUWNC6OGIL3OTB7JF6OAFGNGIR2RH7NDWRAEORA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.233250680 (783h44m33.104333725s) [noise=true , address=145.79.227.40:4049 , piece_id=RV43PFYU6UWU5QFHCO7T6F3E3VBNEYQRMA4UCPXT54PP3T3ZD62Q, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.233527197 (783h44m33.104610242s) [noise=true , address=77.90.31.16:4736 , piece_id=6I7NM23KI67OH2RKW4NSGYGJYWZXMH5HVLHLOQFVNCSFW4WZJ5NA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.234521901 (783h44m33.105604936s) [noise=true , address=45.91.116.12:4226 , piece_id=V7NKTZUGNA2HMXA3FSYMIXMSDLQVEUTJB74IW32GYF4XTU45UY6Q, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.235153884 (783h44m33.106236919s) [noise=true , address=145.79.227.45:4231 , piece_id=ESQLZIZB2AVESVJRHVGX4T5AJDEKVGJEJ3JSGA5J25BOQ4UUF7KA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.235615984 (783h44m33.106699019s) [noise=true , address=145.79.227.11:4729 , piece_id=5EQJ3NL4R45GFF35F2WOYWCD77UGR6XUV6B4TIZASZBCBCX6APWA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.236105967 (783h44m33.107189002s) [noise=true , address=145.79.227.5:4221 , piece_id=SHQ2ARJ6XTS7NHVSSNAXW2VR7BPS26MSX76INSRBH6KXPM4SQKEA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.236621799 (783h44m33.107704844s) [noise=true , address=145.79.227.7:4293 , piece_id=WPVYNVVDT4G5DG2HIOLUDVJNIU5O3WFII2H3GIHEZDBST4F2DKGA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.239024364 (783h44m33.110107409s) [noise=true , address=77.90.31.18:4805 , piece_id=YIHVN23OWQTVKXHZIAY4K2C65LU67NA7C6GCC63PXZHCWAZGL7DQ, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:34.837564819 (783h44m34.708647854s) [upload=685582]: Next returned nil segment
Jan 7 07:13:34.837594546 (783h44m34.708677581s) [upload=685582]: Waiting for error group managing segments...
Jan 7 07:13:35.221987466 (783h44m35.093070501s) [noise=true , address=145.79.227.38:4978 , piece_id=QV3BBDP5GVUMTBI2HIE7FWUIUOQVKIOO2D6OJNW2VDCXDFKMISUA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.224032561 (783h44m35.095115596s) [noise=true , address=145.79.227.11:4729 , piece_id=5EQJ3NL4R45GFF35F2WOYWCD77UGR6XUV6B4TIZASZBCBCX6APWA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.224625309 (783h44m35.095708355s) [noise=true , address=145.79.227.52:4445 , piece_id=D6QKUQIBGUGV3V2ZELIM6U5UMTHT2765CHQBIKQ2N3BJF3UXH3XA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.225249578 (783h44m35.096332613s) [noise=true , address=145.79.227.22:4511 , piece_id=VHGXFDPNKXDZI2ARWI7227A5CF3P572VJSQZS75BMMHPAVCPPO7Q, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.233155391 (783h44m35.104238446s) [noise=true , address=145.79.227.45:4231 , piece_id=ESQLZIZB2AVESVJRHVGX4T5AJDEKVGJEJ3JSGA5J25BOQ4UUF7KA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.244418267 (783h44m35.115501302s) [noise=true , address=145.79.227.5:4221 , piece_id=SHQ2ARJ6XTS7NHVSSNAXW2VR7BPS26MSX76INSRBH6KXPM4SQKEA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.253004497 (783h44m35.124087542s) [noise=true , address=145.79.227.31:4798 , piece_id=TKL7NRLI7DYTXU4HEOBJHXQR5CGHU7IG53L2GLENDCORGVVN2KOQ, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.260994110 (783h44m35.132077145s) [noise=true , address=145.79.227.7:4293 , piece_id=WPVYNVVDT4G5DG2HIOLUDVJNIU5O3WFII2H3GIHEZDBST4F2DKGA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.298671554 (783h44m35.169754589s) [noise=true , address=145.79.227.4:4185 , piece_id=SMLAUFNVX6M24OXQ2PK2TIP5MXF3KZYRGHBFXYN7OXD5L76PCY3Q, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.306366076 (783h44m35.177449111s) [noise=true , address=145.79.227.40:4049 , piece_id=RV43PFYU6UWU5QFHCO7T6F3E3VBNEYQRMA4UCPXT54PP3T3ZD62Q, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.519691228 (783h44m35.390774263s) [noise=true , address=77.90.31.16:4736 , piece_id=6I7NM23KI67OH2RKW4NSGYGJYWZXMH5HVLHLOQFVNCSFW4WZJ5NA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.749540043 (783h44m35.620623078s) [noise=true , address=45.91.116.5:4300 , piece_id=5FMCKFKIRBUKZGZTTBKCMURBHZTBKQPHEZMHJWJHLM2PLEDFNXSA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.823582121 (783h44m35.694665156s) [noise=true , address=45.91.116.8:4047 , piece_id=D6ZXOD5IBAYIVGUWNC6OGIL3OTB7JF6OAFGNGIR2RH7NDWRAEORA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.823600715 (783h44m35.69468375s) [seg_pos={0 0}, upload=685582]: Segment reached optimal threshold of 13 pieces.
Jan 7 07:13:35.824004615 (783h44m35.69508765s) [noise=true , address=45.91.116.12:4226 , piece_id=V7NKTZUGNA2HMXA3FSYMIXMSDLQVEUTJB74IW32GYF4XTU45UY6Q, seg_pos={0 0}, upload=685582]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: write tcp 188.166.225.170:54196->45.91.116.12:4226: use of closed network connection; write tcp 188.166.225.170:54196->45.91.116.12:4226: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54196->45.91.116.12:4226: use of closed network connection
Jan 7 07:13:35.824012139 (783h44m35.695095174s) [noise=true , address=45.91.116.12:4226 , piece_id=V7NKTZUGNA2HMXA3FSYMIXMSDLQVEUTJB74IW32GYF4XTU45UY6Q, seg_pos={0 0}, upload=685582]: Long tail cancellation due to optimal threshold
Jan 7 07:13:35.824197492 (783h44m35.695280527s) [noise=true , address=45.91.116.13:4153 , piece_id=27PHIESDJJJKDKJITJODFYQLSF42BX3BVTMOXFJAUGK73D3Z47UA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: read tcp 188.166.225.170:33886->45.91.116.13:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 7 07:13:35.824205007 (783h44m35.695288042s) [noise=true , address=45.91.116.13:4153 , piece_id=27PHIESDJJJKDKJITJODFYQLSF42BX3BVTMOXFJAUGK73D3Z47UA, seg_pos={0 0}, upload=685582]: Long tail cancellation due to optimal threshold
Jan 7 07:13:35.824576704 (783h44m35.695659739s) [noise=true , address=77.90.31.18:4805 , piece_id=YIHVN23OWQTVKXHZIAY4K2C65LU67NA7C6GCC63PXZHCWAZGL7DQ, seg_pos={0 0}, upload=685582]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: read tcp 188.166.225.170:38598->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 7 07:13:35.824596592 (783h44m35.695679637s) [noise=true , address=77.90.31.18:4805 , piece_id=YIHVN23OWQTVKXHZIAY4K2C65LU67NA7C6GCC63PXZHCWAZGL7DQ, seg_pos={0 0}, upload=685582]: Long tail cancellation due to optimal threshold
Jan 7 07:13:35.824616139 (783h44m35.695699184s) [seg_pos={0 0}, upload=685582]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 7 07:13:35.824682215 (783h44m35.69576526s) [upload=685582]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 7 07:13:35.824686843 (783h44m35.695769878s) [upload=685582]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 7 07:13:36.105700746 (783h44m35.976783791s) [upload=685582]: Done uploading segments...
Jan 7 07:13:36.105723649 (783h44m35.976806684s) [upload=685582]: Upload finished. err: <nil>
Duration: 6.157868446s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
50 out of 60 is failed (threshold=6) [XXXX.XXXXXXXXXXX.XXXX.XXXX..XXXXX.XXXXX..XXXXX.XXXX.XXXXXXXX] duration:
5.000760666s trace:
https://jaeger.dev.storj.io/trace/192ef5609804ea5f
FAIL (2026-01-07 07:19:19 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.000782969s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
16 out of 60 is failed (threshold=3) [..X.XXX.X.X..X....X.X.....X....X.X.....X.......X....XX......] duration:
26.025100961s trace:
https://jaeger.dev.storj.io/trace/3649792573630e65
FAIL (2026-01-07 07:19:07 UTC)
orbiter warning: context deadline exceeded
Duration: 26.025130096s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [X........................................X........X........X] duration:
1m0.002884585s trace:
https://jaeger.dev.storj.io/trace/621f2bd1cae766d0
FAIL (2026-01-07 07:18:47 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
927s) [noise=true , address=77.90.31.15:4727 , piece_id=UFRMVFFY2UKQLPR7P4TTN2UFNFZQ6BDPUP7INPYRJIK267R6WPTQ, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.304384464 (783h48m48.175467499s) [noise=true , address=145.79.227.18:4017 , piece_id=XEDPCH5DMCZIQUDMILRNT4NMMVSH67CDSFA2LXLZX3SJNC7K64JQ, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.304648907 (783h48m48.175731942s) [noise=true , address=145.79.227.43:4180 , piece_id=3ZAN75QUFNRD4K5QYAOZAYP2O4KK4SVZPXPTDOA7IYVIGLLEPYSQ, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.304909002 (783h48m48.175992037s) [noise=true , address=145.79.227.35:4484 , piece_id=6ZQ2M55FDBKKZAGXNN7GPGZCXPKJWRGPKDL5XQWBWREGXKRSCLBQ, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.305186050 (783h48m48.176269085s) [noise=true , address=64.49.13.169:4713 , piece_id=4X6L55C3YY3Y35RKP2LX7BVPJME7L5JWELQ2XK2E752NT7E6KCWA, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.305462475 (783h48m48.17654552s) [noise=true , address=145.79.227.41:4109 , piece_id=F62DIMJUKTLEX3TO4D6FT2PGGKOCDKYY7US6QZGW7PBGNXMLQRFQ, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.306063991 (783h48m48.177147037s) [noise=true , address=145.79.227.2:4127 , piece_id=J4JYZ2L5ULSHBLFH3N2FUKXBCDWEPVPNNNEBUW4LA3QAJYEGCBLA, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.306457621 (783h48m48.177540666s) [noise=true , address=145.79.227.49:4370 , piece_id=G3ATR7KHT5DDCJNAEZDDI55XTPPZMFDYKQLAR3AD45P3GCESOHGA, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.306836312 (783h48m48.177919357s) [noise=true , address=145.79.227.33:4888 , piece_id=NDWJJUO4V4UBYYWJZJM4FQGVUDR24WTCHABZJMUGZ4FMH3U6ZK7Q, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.307182321 (783h48m48.178265366s) [noise=true , address=145.79.227.30:4777 , piece_id=BBHGIQO2AEMZWMUOGMRJC4CKRWJOTEY3WQP4I4XRGMNRIOUOH5IQ, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.307621457 (783h48m48.178704503s) [noise=true , address=145.79.227.34:4931 , piece_id=QFHCIN7ZZBYM3CFF6UVA73EFMWRKJBOWX6KBFZC2SHZ5KN3WU34Q, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.307948189 (783h48m48.179031234s) [noise=true , address=77.90.31.21:4909 , piece_id=7KH356VERAYPY7GHMEOT34TADWH4U74L2JUPMQ6Y7WF52L5I6M6A, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.308347098 (783h48m48.179430133s) [noise=true , address=77.90.31.18:4830 , piece_id=TUICFKVMJ3S2KHW7MKFNJUR6Z5UN5QFZOBXJEYC43O3BE3QQDWHA, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.308510059 (783h48m48.179593104s) [noise=true , address=77.90.31.12:4619 , piece_id=DVOJATZI5UILEARIOOZ2K3WGDFFQTFONCK2HKP2ZXKLBAEQZLROQ, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.309050468 (783h48m48.180133514s) [noise=true , address=145.79.227.44:4198 , piece_id=4KNZZKJ5V3SOLZKPJYY6FSCG4SIAF67RDUWRQSXPAMDY5IO32YOA, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:48.309517858 (783h48m48.180600903s) [noise=true , address=64.49.13.155:4920 , piece_id=WMM6GIC2AJJHN2BOT7PLSKWVG7ONHG3CL3OF3PJQ5DH5ODFO24ZA, seg_pos={0 0}, upload=685644]: Uploading piece...
Jan 7 07:17:49.600689336 (783h48m49.471772381s) [upload=685644]: Next returned nil segment
Jan 7 07:17:49.600703433 (783h48m49.471786478s) [upload=685644]: Waiting for error group managing segments...
Jan 7 07:17:50.063713792 (783h48m49.934796847s) [noise=true , address=64.49.13.169:4713 , piece_id=4X6L55C3YY3Y35RKP2LX7BVPJME7L5JWELQ2XK2E752NT7E6KCWA, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.116339850 (783h48m49.987422895s) [noise=true , address=64.49.13.155:4920 , piece_id=WMM6GIC2AJJHN2BOT7PLSKWVG7ONHG3CL3OF3PJQ5DH5ODFO24ZA, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.154469825 (783h48m50.02555287s) [noise=true , address=145.79.227.32:4847 , piece_id=73A46W74UZTHTZBW6G35LSJZIA3ZRVFRRCC3KGJYGWZZH6NLICGA, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.158394037 (783h48m50.029477072s) [noise=true , address=145.79.227.44:4198 , piece_id=4KNZZKJ5V3SOLZKPJYY6FSCG4SIAF67RDUWRQSXPAMDY5IO32YOA, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.164547664 (783h48m50.035630709s) [noise=true , address=145.79.227.18:4017 , piece_id=XEDPCH5DMCZIQUDMILRNT4NMMVSH67CDSFA2LXLZX3SJNC7K64JQ, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.173751741 (783h48m50.044834786s) [noise=true , address=145.79.227.41:4109 , piece_id=F62DIMJUKTLEX3TO4D6FT2PGGKOCDKYY7US6QZGW7PBGNXMLQRFQ, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.181867575 (783h48m50.05295061s) [noise=true , address=145.79.227.34:4931 , piece_id=QFHCIN7ZZBYM3CFF6UVA73EFMWRKJBOWX6KBFZC2SHZ5KN3WU34Q, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.190204078 (783h48m50.061287113s) [noise=true , address=145.79.227.2:4127 , piece_id=J4JYZ2L5ULSHBLFH3N2FUKXBCDWEPVPNNNEBUW4LA3QAJYEGCBLA, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.215263345 (783h48m50.08634638s) [noise=true , address=145.79.227.33:4888 , piece_id=NDWJJUO4V4UBYYWJZJM4FQGVUDR24WTCHABZJMUGZ4FMH3U6ZK7Q, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.244106597 (783h48m50.115189632s) [noise=true , address=145.79.227.30:4777 , piece_id=BBHGIQO2AEMZWMUOGMRJC4CKRWJOTEY3WQP4I4XRGMNRIOUOH5IQ, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.468914424 (783h48m50.339997469s) [noise=true , address=77.90.31.18:4830 , piece_id=TUICFKVMJ3S2KHW7MKFNJUR6Z5UN5QFZOBXJEYC43O3BE3QQDWHA, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.533387670 (783h48m50.404470705s) [noise=true , address=77.90.31.12:4619 , piece_id=DVOJATZI5UILEARIOOZ2K3WGDFFQTFONCK2HKP2ZXKLBAEQZLROQ, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.560320373 (783h48m50.431403408s) [noise=true , address=77.90.31.11:4579 , piece_id=5LUJ6FFYUVXAD2A6DNE5IOFT22VZL4ZJ4S2NRR2MCA6GX5H3P4EQ, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.604949694 (783h48m50.476032739s) [noise=true , address=77.90.31.21:4909 , piece_id=7KH356VERAYPY7GHMEOT34TADWH4U74L2JUPMQ6Y7WF52L5I6M6A, seg_pos={0 0}, upload=685644]: Done uploading piece. err: <nil>
Jan 7 07:17:50.604977047 (783h48m50.476060082s) [seg_pos={0 0}, upload=685644]: Segment reached optimal threshold of 14 pieces.
Jan 7 07:17:50.605285644 (783h48m50.476368689s) [noise=true , address=77.90.31.15:4727 , piece_id=UFRMVFFY2UKQLPR7P4TTN2UFNFZQ6BDPUP7INPYRJIK267R6WPTQ, seg_pos={0 0}, upload=685644]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xoAbv6zJuG9LjyCdsCrTWQaCDyLrWyv8RCrMTiqNqazoEwD1M): protocol: write tcp 188.166.225.170:47856->77.90.31.15:4727: use of closed network connection; write tcp 188.166.225.170:47856->77.90.31.15:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47856->77.90.31.15:4727: use of closed network connection
Jan 7 07:17:50.605295753 (783h48m50.476378808s) [noise=true , address=77.90.31.15:4727 , piece_id=UFRMVFFY2UKQLPR7P4TTN2UFNFZQ6BDPUP7INPYRJIK267R6WPTQ, seg_pos={0 0}, upload=685644]: Long tail cancellation due to optimal threshold
Jan 7 07:17:50.605401254 (783h48m50.476484299s) [noise=true , address=145.79.227.49:4370 , piece_id=G3ATR7KHT5DDCJNAEZDDI55XTPPZMFDYKQLAR3AD45P3GCESOHGA, seg_pos={0 0}, upload=685644]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LBgixGGgNeYs63ZSpkYJmrEGr3W9Ma1s2Sj8cCxU6mUxm1aC): protocol: expected piece hash; context canceled; EOF
Jan 7 07:17:50.605437523 (783h48m50.476520568s) [noise=true , address=145.79.227.49:4370 , piece_id=G3ATR7KHT5DDCJNAEZDDI55XTPPZMFDYKQLAR3AD45P3GCESOHGA, seg_pos={0 0}, upload=685644]: Long tail cancellation due to optimal threshold
Jan 7 07:17:50.605839750 (783h48m50.476922825s) [noise=true , address=145.79.227.43:4180 , piece_id=3ZAN75QUFNRD4K5QYAOZAYP2O4KK4SVZPXPTDOA7IYVIGLLEPYSQ, seg_pos={0 0}, upload=685644]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rhkcipPBZNFScBjBhWdigzSMwENSmzhCucnC2iuyc9udpweE3): protocol: expected piece hash; context canceled; EOF
Jan 7 07:17:50.605882010 (783h48m50.476965045s) [noise=true , address=145.79.227.43:4180 , piece_id=3ZAN75QUFNRD4K5QYAOZAYP2O4KK4SVZPXPTDOA7IYVIGLLEPYSQ, seg_pos={0 0}, upload=685644]: Long tail cancellation due to optimal threshold
Jan 7 07:17:50.605904593 (783h48m50.476987628s) [noise=true , address=145.79.227.35:4484 , piece_id=6ZQ2M55FDBKKZAGXNN7GPGZCXPKJWRGPKDL5XQWBWREGXKRSCLBQ, seg_pos={0 0}, upload=685644]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1a5S5Ar7jNbK9MESghbWfmEgBPNSMN2tEnZeeG1kKzeX2dnSX3): protocol: expected piece hash; context canceled; EOF
Jan 7 07:17:50.605924150 (783h48m50.477007205s) [noise=true , address=145.79.227.35:4484 , piece_id=6ZQ2M55FDBKKZAGXNN7GPGZCXPKJWRGPKDL5XQWBWREGXKRSCLBQ, seg_pos={0 0}, upload=685644]: Long tail cancellation due to optimal threshold
Jan 7 07:17:50.605947465 (783h48m50.47703051s) [seg_pos={0 0}, upload=685644]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jan 7 07:17:50.606035051 (783h48m50.477118086s) [upload=685644]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 7 07:17:50.606040612 (783h48m50.477123647s) [upload=685644]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 7 07:17:50.908947210 (783h48m50.780030255s) [upload=685644]: Done uploading segments...
Jan 7 07:17:50.908965275 (783h48m50.78004831s) [upload=685644]: Upload finished. err: <nil>
Duration: 1m0.002903411s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [..........X...............X...................X.............] duration:
18.513905916s trace:
https://jaeger.dev.storj.io/trace/36161e151ba11c34
FAIL (2026-01-07 07:18:34 UTC)
orbiter warning: context deadline exceeded
Duration: 18.513932628s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
14 out of 60 is failed (threshold=3) [X...X....X.........X..X.X..X.......XX.X....X....XX....X.....] duration:
5.714012149s trace:
https://jaeger.dev.storj.io/trace/566429788081faee
FAIL (2026-01-07 07:18:22 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
) [upload=685582]: Uploading segments...
Jan 7 07:13:32.990108051 (783h44m32.861191096s) [upload=685582]: Got next segment. Inline: false
Jan 7 07:13:32.990479758 (783h44m32.861562803s) [upload=685582]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 7 07:13:32.990486191 (783h44m32.861569226s) [upload=685582]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 7 07:13:33.230730672 (783h44m33.101813717s) [seg_pos={0 0}, upload=685582]: Begin upload segment...
Jan 7 07:13:33.230783873 (783h44m33.101866908s) [seg_pos={0 0}, upload=685582]: Done begin upload segment.
Jan 7 07:13:33.230814070 (783h44m33.101897115s) [noise=true , address=45.91.116.13:4153 , piece_id=27PHIESDJJJKDKJITJODFYQLSF42BX3BVTMOXFJAUGK73D3Z47UA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.230992941 (783h44m33.102075976s) [noise=true , address=45.91.116.5:4300 , piece_id=5FMCKFKIRBUKZGZTTBKCMURBHZTBKQPHEZMHJWJHLM2PLEDFNXSA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.231512250 (783h44m33.102595285s) [noise=true , address=145.79.227.22:4511 , piece_id=VHGXFDPNKXDZI2ARWI7227A5CF3P572VJSQZS75BMMHPAVCPPO7Q, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.231639272 (783h44m33.102722307s) [noise=true , address=145.79.227.4:4185 , piece_id=SMLAUFNVX6M24OXQ2PK2TIP5MXF3KZYRGHBFXYN7OXD5L76PCY3Q, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.231935536 (783h44m33.103018571s) [noise=true , address=145.79.227.31:4798 , piece_id=TKL7NRLI7DYTXU4HEOBJHXQR5CGHU7IG53L2GLENDCORGVVN2KOQ, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.232203125 (783h44m33.10328616s) [noise=true , address=145.79.227.38:4978 , piece_id=QV3BBDP5GVUMTBI2HIE7FWUIUOQVKIOO2D6OJNW2VDCXDFKMISUA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.232580194 (783h44m33.103663229s) [noise=true , address=145.79.227.52:4445 , piece_id=D6QKUQIBGUGV3V2ZELIM6U5UMTHT2765CHQBIKQ2N3BJF3UXH3XA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.232942814 (783h44m33.104025849s) [noise=true , address=45.91.116.8:4047 , piece_id=D6ZXOD5IBAYIVGUWNC6OGIL3OTB7JF6OAFGNGIR2RH7NDWRAEORA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.233250680 (783h44m33.104333725s) [noise=true , address=145.79.227.40:4049 , piece_id=RV43PFYU6UWU5QFHCO7T6F3E3VBNEYQRMA4UCPXT54PP3T3ZD62Q, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.233527197 (783h44m33.104610242s) [noise=true , address=77.90.31.16:4736 , piece_id=6I7NM23KI67OH2RKW4NSGYGJYWZXMH5HVLHLOQFVNCSFW4WZJ5NA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.234521901 (783h44m33.105604936s) [noise=true , address=45.91.116.12:4226 , piece_id=V7NKTZUGNA2HMXA3FSYMIXMSDLQVEUTJB74IW32GYF4XTU45UY6Q, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.235153884 (783h44m33.106236919s) [noise=true , address=145.79.227.45:4231 , piece_id=ESQLZIZB2AVESVJRHVGX4T5AJDEKVGJEJ3JSGA5J25BOQ4UUF7KA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.235615984 (783h44m33.106699019s) [noise=true , address=145.79.227.11:4729 , piece_id=5EQJ3NL4R45GFF35F2WOYWCD77UGR6XUV6B4TIZASZBCBCX6APWA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.236105967 (783h44m33.107189002s) [noise=true , address=145.79.227.5:4221 , piece_id=SHQ2ARJ6XTS7NHVSSNAXW2VR7BPS26MSX76INSRBH6KXPM4SQKEA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.236621799 (783h44m33.107704844s) [noise=true , address=145.79.227.7:4293 , piece_id=WPVYNVVDT4G5DG2HIOLUDVJNIU5O3WFII2H3GIHEZDBST4F2DKGA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.239024364 (783h44m33.110107409s) [noise=true , address=77.90.31.18:4805 , piece_id=YIHVN23OWQTVKXHZIAY4K2C65LU67NA7C6GCC63PXZHCWAZGL7DQ, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:34.837564819 (783h44m34.708647854s) [upload=685582]: Next returned nil segment
Jan 7 07:13:34.837594546 (783h44m34.708677581s) [upload=685582]: Waiting for error group managing segments...
Jan 7 07:13:35.221987466 (783h44m35.093070501s) [noise=true , address=145.79.227.38:4978 , piece_id=QV3BBDP5GVUMTBI2HIE7FWUIUOQVKIOO2D6OJNW2VDCXDFKMISUA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.224032561 (783h44m35.095115596s) [noise=true , address=145.79.227.11:4729 , piece_id=5EQJ3NL4R45GFF35F2WOYWCD77UGR6XUV6B4TIZASZBCBCX6APWA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.224625309 (783h44m35.095708355s) [noise=true , address=145.79.227.52:4445 , piece_id=D6QKUQIBGUGV3V2ZELIM6U5UMTHT2765CHQBIKQ2N3BJF3UXH3XA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.225249578 (783h44m35.096332613s) [noise=true , address=145.79.227.22:4511 , piece_id=VHGXFDPNKXDZI2ARWI7227A5CF3P572VJSQZS75BMMHPAVCPPO7Q, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.233155391 (783h44m35.104238446s) [noise=true , address=145.79.227.45:4231 , piece_id=ESQLZIZB2AVESVJRHVGX4T5AJDEKVGJEJ3JSGA5J25BOQ4UUF7KA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.244418267 (783h44m35.115501302s) [noise=true , address=145.79.227.5:4221 , piece_id=SHQ2ARJ6XTS7NHVSSNAXW2VR7BPS26MSX76INSRBH6KXPM4SQKEA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.253004497 (783h44m35.124087542s) [noise=true , address=145.79.227.31:4798 , piece_id=TKL7NRLI7DYTXU4HEOBJHXQR5CGHU7IG53L2GLENDCORGVVN2KOQ, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.260994110 (783h44m35.132077145s) [noise=true , address=145.79.227.7:4293 , piece_id=WPVYNVVDT4G5DG2HIOLUDVJNIU5O3WFII2H3GIHEZDBST4F2DKGA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.298671554 (783h44m35.169754589s) [noise=true , address=145.79.227.4:4185 , piece_id=SMLAUFNVX6M24OXQ2PK2TIP5MXF3KZYRGHBFXYN7OXD5L76PCY3Q, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.306366076 (783h44m35.177449111s) [noise=true , address=145.79.227.40:4049 , piece_id=RV43PFYU6UWU5QFHCO7T6F3E3VBNEYQRMA4UCPXT54PP3T3ZD62Q, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.519691228 (783h44m35.390774263s) [noise=true , address=77.90.31.16:4736 , piece_id=6I7NM23KI67OH2RKW4NSGYGJYWZXMH5HVLHLOQFVNCSFW4WZJ5NA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.749540043 (783h44m35.620623078s) [noise=true , address=45.91.116.5:4300 , piece_id=5FMCKFKIRBUKZGZTTBKCMURBHZTBKQPHEZMHJWJHLM2PLEDFNXSA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.823582121 (783h44m35.694665156s) [noise=true , address=45.91.116.8:4047 , piece_id=D6ZXOD5IBAYIVGUWNC6OGIL3OTB7JF6OAFGNGIR2RH7NDWRAEORA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.823600715 (783h44m35.69468375s) [seg_pos={0 0}, upload=685582]: Segment reached optimal threshold of 13 pieces.
Jan 7 07:13:35.824004615 (783h44m35.69508765s) [noise=true , address=45.91.116.12:4226 , piece_id=V7NKTZUGNA2HMXA3FSYMIXMSDLQVEUTJB74IW32GYF4XTU45UY6Q, seg_pos={0 0}, upload=685582]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: write tcp 188.166.225.170:54196->45.91.116.12:4226: use of closed network connection; write tcp 188.166.225.170:54196->45.91.116.12:4226: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54196->45.91.116.12:4226: use of closed network connection
Jan 7 07:13:35.824012139 (783h44m35.695095174s) [noise=true , address=45.91.116.12:4226 , piece_id=V7NKTZUGNA2HMXA3FSYMIXMSDLQVEUTJB74IW32GYF4XTU45UY6Q, seg_pos={0 0}, upload=685582]: Long tail cancellation due to optimal threshold
Jan 7 07:13:35.824197492 (783h44m35.695280527s) [noise=true , address=45.91.116.13:4153 , piece_id=27PHIESDJJJKDKJITJODFYQLSF42BX3BVTMOXFJAUGK73D3Z47UA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: read tcp 188.166.225.170:33886->45.91.116.13:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 7 07:13:35.824205007 (783h44m35.695288042s) [noise=true , address=45.91.116.13:4153 , piece_id=27PHIESDJJJKDKJITJODFYQLSF42BX3BVTMOXFJAUGK73D3Z47UA, seg_pos={0 0}, upload=685582]: Long tail cancellation due to optimal threshold
Jan 7 07:13:35.824576704 (783h44m35.695659739s) [noise=true , address=77.90.31.18:4805 , piece_id=YIHVN23OWQTVKXHZIAY4K2C65LU67NA7C6GCC63PXZHCWAZGL7DQ, seg_pos={0 0}, upload=685582]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: read tcp 188.166.225.170:38598->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 7 07:13:35.824596592 (783h44m35.695679637s) [noise=true , address=77.90.31.18:4805 , piece_id=YIHVN23OWQTVKXHZIAY4K2C65LU67NA7C6GCC63PXZHCWAZGL7DQ, seg_pos={0 0}, upload=685582]: Long tail cancellation due to optimal threshold
Jan 7 07:13:35.824616139 (783h44m35.695699184s) [seg_pos={0 0}, upload=685582]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 7 07:13:35.824682215 (783h44m35.69576526s) [upload=685582]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 7 07:13:35.824686843 (783h44m35.695769878s) [upload=685582]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 7 07:13:36.105700746 (783h44m35.976783791s) [upload=685582]: Done uploading segments...
Jan 7 07:13:36.105723649 (783h44m35.976806684s) [upload=685582]: Upload finished. err: <nil>
Duration: 5.714037848s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
50 out of 60 is failed (threshold=6) [XXXXX.XXXXXXXXXXX.XXXX.XXXX..XXXXX.XXXXX..XXXXX.XXXX.XXXXXXX] duration:
5.000839387s trace:
https://jaeger.dev.storj.io/trace/3e4230c33abad7f9
FAIL (2026-01-07 07:18:06 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.000863473s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
16 out of 60 is failed (threshold=3) [...X.XXX.X.X..X....X.X.....X....X.X.....X.......X....XX.....] duration:
20.950756171s trace:
https://jaeger.dev.storj.io/trace/3649792573630e65
FAIL (2026-01-07 07:17:53 UTC)
orbiter warning: context deadline exceeded
Duration: 20.950782382s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
16 out of 60 is failed (threshold=3) [....X.XXX.X.X..X....X.X.....X....X.X.....X.......X....XX....] duration:
22.882184907s trace:
https://jaeger.dev.storj.io/trace/3649792573630e65
FAIL (2026-01-07 07:17:27 UTC)
orbiter warning: context deadline exceeded
Duration: 22.882197641s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
14 out of 60 is failed (threshold=3) [.X...X....X.........X..X.X..X.......XX.X....X....XX....X....] duration:
5.693147646s trace:
https://jaeger.dev.storj.io/trace/566429788081faee
FAIL (2026-01-07 07:17:26 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
) [upload=685582]: Uploading segments...
Jan 7 07:13:32.990108051 (783h44m32.861191096s) [upload=685582]: Got next segment. Inline: false
Jan 7 07:13:32.990479758 (783h44m32.861562803s) [upload=685582]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 7 07:13:32.990486191 (783h44m32.861569226s) [upload=685582]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 7 07:13:33.230730672 (783h44m33.101813717s) [seg_pos={0 0}, upload=685582]: Begin upload segment...
Jan 7 07:13:33.230783873 (783h44m33.101866908s) [seg_pos={0 0}, upload=685582]: Done begin upload segment.
Jan 7 07:13:33.230814070 (783h44m33.101897115s) [noise=true , address=45.91.116.13:4153 , piece_id=27PHIESDJJJKDKJITJODFYQLSF42BX3BVTMOXFJAUGK73D3Z47UA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.230992941 (783h44m33.102075976s) [noise=true , address=45.91.116.5:4300 , piece_id=5FMCKFKIRBUKZGZTTBKCMURBHZTBKQPHEZMHJWJHLM2PLEDFNXSA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.231512250 (783h44m33.102595285s) [noise=true , address=145.79.227.22:4511 , piece_id=VHGXFDPNKXDZI2ARWI7227A5CF3P572VJSQZS75BMMHPAVCPPO7Q, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.231639272 (783h44m33.102722307s) [noise=true , address=145.79.227.4:4185 , piece_id=SMLAUFNVX6M24OXQ2PK2TIP5MXF3KZYRGHBFXYN7OXD5L76PCY3Q, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.231935536 (783h44m33.103018571s) [noise=true , address=145.79.227.31:4798 , piece_id=TKL7NRLI7DYTXU4HEOBJHXQR5CGHU7IG53L2GLENDCORGVVN2KOQ, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.232203125 (783h44m33.10328616s) [noise=true , address=145.79.227.38:4978 , piece_id=QV3BBDP5GVUMTBI2HIE7FWUIUOQVKIOO2D6OJNW2VDCXDFKMISUA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.232580194 (783h44m33.103663229s) [noise=true , address=145.79.227.52:4445 , piece_id=D6QKUQIBGUGV3V2ZELIM6U5UMTHT2765CHQBIKQ2N3BJF3UXH3XA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.232942814 (783h44m33.104025849s) [noise=true , address=45.91.116.8:4047 , piece_id=D6ZXOD5IBAYIVGUWNC6OGIL3OTB7JF6OAFGNGIR2RH7NDWRAEORA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.233250680 (783h44m33.104333725s) [noise=true , address=145.79.227.40:4049 , piece_id=RV43PFYU6UWU5QFHCO7T6F3E3VBNEYQRMA4UCPXT54PP3T3ZD62Q, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.233527197 (783h44m33.104610242s) [noise=true , address=77.90.31.16:4736 , piece_id=6I7NM23KI67OH2RKW4NSGYGJYWZXMH5HVLHLOQFVNCSFW4WZJ5NA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.234521901 (783h44m33.105604936s) [noise=true , address=45.91.116.12:4226 , piece_id=V7NKTZUGNA2HMXA3FSYMIXMSDLQVEUTJB74IW32GYF4XTU45UY6Q, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.235153884 (783h44m33.106236919s) [noise=true , address=145.79.227.45:4231 , piece_id=ESQLZIZB2AVESVJRHVGX4T5AJDEKVGJEJ3JSGA5J25BOQ4UUF7KA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.235615984 (783h44m33.106699019s) [noise=true , address=145.79.227.11:4729 , piece_id=5EQJ3NL4R45GFF35F2WOYWCD77UGR6XUV6B4TIZASZBCBCX6APWA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.236105967 (783h44m33.107189002s) [noise=true , address=145.79.227.5:4221 , piece_id=SHQ2ARJ6XTS7NHVSSNAXW2VR7BPS26MSX76INSRBH6KXPM4SQKEA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.236621799 (783h44m33.107704844s) [noise=true , address=145.79.227.7:4293 , piece_id=WPVYNVVDT4G5DG2HIOLUDVJNIU5O3WFII2H3GIHEZDBST4F2DKGA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.239024364 (783h44m33.110107409s) [noise=true , address=77.90.31.18:4805 , piece_id=YIHVN23OWQTVKXHZIAY4K2C65LU67NA7C6GCC63PXZHCWAZGL7DQ, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:34.837564819 (783h44m34.708647854s) [upload=685582]: Next returned nil segment
Jan 7 07:13:34.837594546 (783h44m34.708677581s) [upload=685582]: Waiting for error group managing segments...
Jan 7 07:13:35.221987466 (783h44m35.093070501s) [noise=true , address=145.79.227.38:4978 , piece_id=QV3BBDP5GVUMTBI2HIE7FWUIUOQVKIOO2D6OJNW2VDCXDFKMISUA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.224032561 (783h44m35.095115596s) [noise=true , address=145.79.227.11:4729 , piece_id=5EQJ3NL4R45GFF35F2WOYWCD77UGR6XUV6B4TIZASZBCBCX6APWA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.224625309 (783h44m35.095708355s) [noise=true , address=145.79.227.52:4445 , piece_id=D6QKUQIBGUGV3V2ZELIM6U5UMTHT2765CHQBIKQ2N3BJF3UXH3XA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.225249578 (783h44m35.096332613s) [noise=true , address=145.79.227.22:4511 , piece_id=VHGXFDPNKXDZI2ARWI7227A5CF3P572VJSQZS75BMMHPAVCPPO7Q, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.233155391 (783h44m35.104238446s) [noise=true , address=145.79.227.45:4231 , piece_id=ESQLZIZB2AVESVJRHVGX4T5AJDEKVGJEJ3JSGA5J25BOQ4UUF7KA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.244418267 (783h44m35.115501302s) [noise=true , address=145.79.227.5:4221 , piece_id=SHQ2ARJ6XTS7NHVSSNAXW2VR7BPS26MSX76INSRBH6KXPM4SQKEA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.253004497 (783h44m35.124087542s) [noise=true , address=145.79.227.31:4798 , piece_id=TKL7NRLI7DYTXU4HEOBJHXQR5CGHU7IG53L2GLENDCORGVVN2KOQ, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.260994110 (783h44m35.132077145s) [noise=true , address=145.79.227.7:4293 , piece_id=WPVYNVVDT4G5DG2HIOLUDVJNIU5O3WFII2H3GIHEZDBST4F2DKGA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.298671554 (783h44m35.169754589s) [noise=true , address=145.79.227.4:4185 , piece_id=SMLAUFNVX6M24OXQ2PK2TIP5MXF3KZYRGHBFXYN7OXD5L76PCY3Q, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.306366076 (783h44m35.177449111s) [noise=true , address=145.79.227.40:4049 , piece_id=RV43PFYU6UWU5QFHCO7T6F3E3VBNEYQRMA4UCPXT54PP3T3ZD62Q, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.519691228 (783h44m35.390774263s) [noise=true , address=77.90.31.16:4736 , piece_id=6I7NM23KI67OH2RKW4NSGYGJYWZXMH5HVLHLOQFVNCSFW4WZJ5NA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.749540043 (783h44m35.620623078s) [noise=true , address=45.91.116.5:4300 , piece_id=5FMCKFKIRBUKZGZTTBKCMURBHZTBKQPHEZMHJWJHLM2PLEDFNXSA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.823582121 (783h44m35.694665156s) [noise=true , address=45.91.116.8:4047 , piece_id=D6ZXOD5IBAYIVGUWNC6OGIL3OTB7JF6OAFGNGIR2RH7NDWRAEORA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.823600715 (783h44m35.69468375s) [seg_pos={0 0}, upload=685582]: Segment reached optimal threshold of 13 pieces.
Jan 7 07:13:35.824004615 (783h44m35.69508765s) [noise=true , address=45.91.116.12:4226 , piece_id=V7NKTZUGNA2HMXA3FSYMIXMSDLQVEUTJB74IW32GYF4XTU45UY6Q, seg_pos={0 0}, upload=685582]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: write tcp 188.166.225.170:54196->45.91.116.12:4226: use of closed network connection; write tcp 188.166.225.170:54196->45.91.116.12:4226: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54196->45.91.116.12:4226: use of closed network connection
Jan 7 07:13:35.824012139 (783h44m35.695095174s) [noise=true , address=45.91.116.12:4226 , piece_id=V7NKTZUGNA2HMXA3FSYMIXMSDLQVEUTJB74IW32GYF4XTU45UY6Q, seg_pos={0 0}, upload=685582]: Long tail cancellation due to optimal threshold
Jan 7 07:13:35.824197492 (783h44m35.695280527s) [noise=true , address=45.91.116.13:4153 , piece_id=27PHIESDJJJKDKJITJODFYQLSF42BX3BVTMOXFJAUGK73D3Z47UA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: read tcp 188.166.225.170:33886->45.91.116.13:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 7 07:13:35.824205007 (783h44m35.695288042s) [noise=true , address=45.91.116.13:4153 , piece_id=27PHIESDJJJKDKJITJODFYQLSF42BX3BVTMOXFJAUGK73D3Z47UA, seg_pos={0 0}, upload=685582]: Long tail cancellation due to optimal threshold
Jan 7 07:13:35.824576704 (783h44m35.695659739s) [noise=true , address=77.90.31.18:4805 , piece_id=YIHVN23OWQTVKXHZIAY4K2C65LU67NA7C6GCC63PXZHCWAZGL7DQ, seg_pos={0 0}, upload=685582]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: read tcp 188.166.225.170:38598->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 7 07:13:35.824596592 (783h44m35.695679637s) [noise=true , address=77.90.31.18:4805 , piece_id=YIHVN23OWQTVKXHZIAY4K2C65LU67NA7C6GCC63PXZHCWAZGL7DQ, seg_pos={0 0}, upload=685582]: Long tail cancellation due to optimal threshold
Jan 7 07:13:35.824616139 (783h44m35.695699184s) [seg_pos={0 0}, upload=685582]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 7 07:13:35.824682215 (783h44m35.69576526s) [upload=685582]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 7 07:13:35.824686843 (783h44m35.695769878s) [upload=685582]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 7 07:13:36.105700746 (783h44m35.976783791s) [upload=685582]: Done uploading segments...
Jan 7 07:13:36.105723649 (783h44m35.976806684s) [upload=685582]: Upload finished. err: <nil>
Duration: 5.693166952s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
50 out of 60 is failed (threshold=6) [XXXXXX.XXXXXXXXXXX.XXXX.XXXX..XXXXX.XXXXX..XXXXX.XXXX.XXXXXX] duration:
5.001035047s trace:
https://jaeger.dev.storj.io/trace/49535e51b9c185b1
FAIL (2026-01-07 07:17:16 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001056446s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [...........X...............X...................X............] duration:
11.536545197s trace:
https://jaeger.dev.storj.io/trace/36161e151ba11c34
FAIL (2026-01-07 07:17:15 UTC)
orbiter warning: context deadline exceeded
Duration: 11.537343818s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [.X........................................X........X........] duration:
5.936947937s trace:
https://jaeger.dev.storj.io/trace/1ea01aa868761fdc
FAIL (2026-01-07 07:17:07 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
133194519 (783h41m50.004277565s) [noise=true , address=145.79.227.32:4851 , piece_id=5W74ZJOGSKONFPVTJJPHDWNFHLTWRU4KEVX25X4PQCKTRJZR3CZA, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.133698059 (783h41m50.004781104s) [noise=true , address=145.79.227.33:4897 , piece_id=NLJJNXY2DGHMZWGJRSKGTYDVYZLYZJSVYVIZ35IUCMSI34RD5TBA, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.134246032 (783h41m50.005329067s) [noise=true , address=145.79.227.44:4227 , piece_id=TCERBF6EX3JHAV63JYLUYA54OHYY33YEM25EQ6CIRS4746KNCULA, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.134617359 (783h41m50.005700404s) [noise=true , address=145.79.227.30:4795 , piece_id=G7EGUQIODKUJDE6FEWO5B5C3RQYQ37SAHQYC6BI7X4JS6A7VMOXQ, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.135018602 (783h41m50.006101637s) [noise=true , address=77.90.31.16:4741 , piece_id=G4TPC522KLMJBYHYW2SYTUMJFVP4K54R63ZIHOCPQINQO5WE7Q2Q, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.135492385 (783h41m50.00657542s) [noise=true , address=145.79.227.41:4092 , piece_id=3WQEAVSBFU5KF6WVNGACEVQ6M42JEMWW6OHCXPCXVNZCOG4XSMGQ, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.135895753 (783h41m50.006978799s) [noise=true , address=145.79.227.34:4926 , piece_id=AUBI2PYIJK5YTKFZHL5WF7BQOKNKEPSTZQBERRGEOFZN7RE7T2SA, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.136238677 (783h41m50.007321722s) [noise=true , address=193.149.164.143:4699 , piece_id=DFHC3CPJBT2U7MW5HSNGIGTPMD3QZD4ON62UJ2HNNVWRXOJ4A6PQ, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.136546201 (783h41m50.007629236s) [noise=true , address=145.79.227.36:4543 , piece_id=MUK6THUJD4QHK4VLL3R6DSZP47VV5RJAUPKG6RTPR7F6GTIC6GIA, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.136565809 (783h41m50.007648844s) [noise=true , address=145.79.227.29:4946 , piece_id=FR3BLE7ANKJYAQG5GZJ4RL4543FQ55K7VHSJMQGCLC7UDRNKCASA, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.137150974 (783h41m50.008234019s) [noise=true , address=145.79.227.35:4487 , piece_id=E5X6GYKRPSYHASZXZB62XA3LXB6EJPTMENDRW6BGEDVDUD5BW4TA, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.137656586 (783h41m50.008739621s) [noise=true , address=77.90.31.11:4584 , piece_id=LIDDXUURY5ARSGQ6ETUFION36NJFWMIGJHOSOIE4ZUXJPTJNCGSA, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.138273391 (783h41m50.009356436s) [noise=true , address=64.49.13.169:4713 , piece_id=6JKGRVL7KQOZVIIJM3OFQUHWUEF42KMW7Y6I5EPVKVRTKEQVBFNQ, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.138867362 (783h41m50.009950407s) [noise=true , address=64.49.13.155:4916 , piece_id=FPLKLM4RLSO54V7XX3YZF7CGXWSGKNBAKEYQ5FYFYHVE7CTS6PIA, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.139440193 (783h41m50.010523238s) [noise=true , address=145.79.227.49:4372 , piece_id=HQ535NWA7OLGPNMYWPMHDJL3ICULP4GEXMYTKLDTC2SPMQYCRZOQ, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.139988797 (783h41m50.011071832s) [noise=true , address=45.91.116.10:4110 , piece_id=VODTQS7BAXPWLTDDTTW75FXYRDBGM2PSJQTIGCRCRNAMMTE55LJA, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.140847663 (783h41m50.011930698s) [noise=true , address=45.91.116.14:4200 , piece_id=ON5HNVYRNJTJCNV4JU6NI2ITS2VKZNNMYOBZUZQ7OLXPYAUQL2JQ, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:51.568606354 (783h41m51.439689389s) [upload=685546]: Next returned nil segment
Jan 7 07:10:51.568621312 (783h41m51.439704347s) [upload=685546]: Waiting for error group managing segments...
Jan 7 07:10:51.990916323 (783h41m51.861999368s) [noise=true , address=64.49.13.155:4916 , piece_id=FPLKLM4RLSO54V7XX3YZF7CGXWSGKNBAKEYQ5FYFYHVE7CTS6PIA, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:51.994944574 (783h41m51.866027619s) [noise=true , address=145.79.227.44:4227 , piece_id=TCERBF6EX3JHAV63JYLUYA54OHYY33YEM25EQ6CIRS4746KNCULA, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:52.000551040 (783h41m51.871634105s) [noise=true , address=145.79.227.41:4092 , piece_id=3WQEAVSBFU5KF6WVNGACEVQ6M42JEMWW6OHCXPCXVNZCOG4XSMGQ, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:52.005752454 (783h41m51.876835509s) [noise=true , address=145.79.227.30:4795 , piece_id=G7EGUQIODKUJDE6FEWO5B5C3RQYQ37SAHQYC6BI7X4JS6A7VMOXQ, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:52.006180148 (783h41m51.877263203s) [noise=true , address=145.79.227.32:4851 , piece_id=5W74ZJOGSKONFPVTJJPHDWNFHLTWRU4KEVX25X4PQCKTRJZR3CZA, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:52.062554132 (783h41m51.933637177s) [noise=true , address=64.49.13.169:4713 , piece_id=6JKGRVL7KQOZVIIJM3OFQUHWUEF42KMW7Y6I5EPVKVRTKEQVBFNQ, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:52.070238635 (783h41m51.94132167s) [noise=true , address=145.79.227.34:4926 , piece_id=AUBI2PYIJK5YTKFZHL5WF7BQOKNKEPSTZQBERRGEOFZN7RE7T2SA, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:52.088972997 (783h41m51.960056032s) [noise=true , address=145.79.227.33:4897 , piece_id=NLJJNXY2DGHMZWGJRSKGTYDVYZLYZJSVYVIZ35IUCMSI34RD5TBA, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:52.281595960 (783h41m52.152678995s) [noise=true , address=77.90.31.12:4603 , piece_id=HPBI4VUKMT2KFKRW2JTX32HB2YGDGLB2V4BFRVVQ2725Y2IC2LOA, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:52.356241606 (783h41m52.227324641s) [noise=true , address=77.90.31.11:4584 , piece_id=LIDDXUURY5ARSGQ6ETUFION36NJFWMIGJHOSOIE4ZUXJPTJNCGSA, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:52.679275149 (783h41m52.550358204s) [noise=true , address=45.91.116.14:4200 , piece_id=ON5HNVYRNJTJCNV4JU6NI2ITS2VKZNNMYOBZUZQ7OLXPYAUQL2JQ, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:52.848093704 (783h41m52.719176749s) [noise=true , address=193.149.164.143:4699 , piece_id=DFHC3CPJBT2U7MW5HSNGIGTPMD3QZD4ON62UJ2HNNVWRXOJ4A6PQ, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:53.022732271 (783h41m52.893815316s) [noise=true , address=145.79.227.49:4372 , piece_id=HQ535NWA7OLGPNMYWPMHDJL3ICULP4GEXMYTKLDTC2SPMQYCRZOQ, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:53.048522630 (783h41m52.919605675s) [noise=true , address=145.79.227.36:4543 , piece_id=MUK6THUJD4QHK4VLL3R6DSZP47VV5RJAUPKG6RTPR7F6GTIC6GIA, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:53.048547036 (783h41m52.919630082s) [seg_pos={0 0}, upload=685546]: Segment reached optimal threshold of 14 pieces.
Jan 7 07:10:53.048761935 (783h41m52.919844971s) [noise=true , address=77.90.31.16:4741 , piece_id=G4TPC522KLMJBYHYW2SYTUMJFVP4K54R63ZIHOCPQINQO5WE7Q2Q, seg_pos={0 0}, upload=685546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR): protocol: expected piece hash; context canceled; EOF
Jan 7 07:10:53.048771714 (783h41m52.919854749s) [noise=true , address=77.90.31.16:4741 , piece_id=G4TPC522KLMJBYHYW2SYTUMJFVP4K54R63ZIHOCPQINQO5WE7Q2Q, seg_pos={0 0}, upload=685546]: Long tail cancellation due to optimal threshold
Jan 7 07:10:53.049042570 (783h41m52.920125605s) [noise=true , address=145.79.227.29:4946 , piece_id=FR3BLE7ANKJYAQG5GZJ4RL4543FQ55K7VHSJMQGCLC7UDRNKCASA, seg_pos={0 0}, upload=685546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12v9Q7aigjWPNXyshHmQt9YKrDu5af1vd1TxkDeHeKixSrWgrLX): protocol: expected piece hash; context canceled; EOF
Jan 7 07:10:53.049049483 (783h41m52.920132518s) [noise=true , address=145.79.227.29:4946 , piece_id=FR3BLE7ANKJYAQG5GZJ4RL4543FQ55K7VHSJMQGCLC7UDRNKCASA, seg_pos={0 0}, upload=685546]: Long tail cancellation due to optimal threshold
Jan 7 07:10:53.049146277 (783h41m52.920229322s) [noise=true , address=145.79.227.35:4487 , piece_id=E5X6GYKRPSYHASZXZB62XA3LXB6EJPTMENDRW6BGEDVDUD5BW4TA, seg_pos={0 0}, upload=685546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1AdZLDV7UBbV2DimCHzmoNxycLigvLMx6Fy9U9QB35WTtiV1KX): protocol: expected piece hash; context canceled; EOF
Jan 7 07:10:53.049151166 (783h41m52.920234212s) [noise=true , address=145.79.227.35:4487 , piece_id=E5X6GYKRPSYHASZXZB62XA3LXB6EJPTMENDRW6BGEDVDUD5BW4TA, seg_pos={0 0}, upload=685546]: Long tail cancellation due to optimal threshold
Jan 7 07:10:53.049299398 (783h41m52.920382434s) [noise=true , address=45.91.116.10:4110 , piece_id=VODTQS7BAXPWLTDDTTW75FXYRDBGM2PSJQTIGCRCRNAMMTE55LJA, seg_pos={0 0}, upload=685546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123KwusATu1ZKtNNVHZZZBE67ZEnVb8puyfQidBi3fEWREqdf1h): protocol: expected piece hash; context canceled; EOF
Jan 7 07:10:53.049308466 (783h41m52.920391512s) [noise=true , address=45.91.116.10:4110 , piece_id=VODTQS7BAXPWLTDDTTW75FXYRDBGM2PSJQTIGCRCRNAMMTE55LJA, seg_pos={0 0}, upload=685546]: Long tail cancellation due to optimal threshold
Jan 7 07:10:53.049324507 (783h41m52.920407542s) [seg_pos={0 0}, upload=685546]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jan 7 07:10:53.049388278 (783h41m52.920471313s) [upload=685546]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 7 07:10:53.049392767 (783h41m52.920475812s) [upload=685546]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 7 07:10:53.340338951 (783h41m53.211421996s) [upload=685546]: Done uploading segments...
Jan 7 07:10:53.340359300 (783h41m53.211442335s) [upload=685546]: Upload finished. err: <nil>
Duration: 5.936978697s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
16 out of 60 is failed (threshold=3) [.....X.XXX.X.X..X....X.X.....X....X.X.....X.......X....XX...] duration:
25.17345367s trace:
https://jaeger.dev.storj.io/trace/3649792573630e65
FAIL (2026-01-07 07:17:04 UTC)
orbiter warning: context deadline exceeded
Duration: 25.173477265s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [............X...............X...................X...........] duration:
34.582410729s trace:
https://jaeger.dev.storj.io/trace/36161e151ba11c34
FAIL (2026-01-07 07:17:03 UTC)
orbiter warning: context deadline exceeded
Duration: 34.582433342s
core
uplink
us-select-1
us1-placement-20
|