Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..........................X.X...............................] duration:
10.466698834s trace:
https://jaeger.dev.storj.io/trace/72d0191c7bb67437
FAIL (2025-12-28 12:19:09 UTC)
orbiter warning: context deadline exceeded
Duration: 10.466725524s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
33 out of 60 is failed (threshold=6) [.X..X.XXX..X....XX.XX.XXXXXXXXXX..XXXX...XX.XXX.XX.......X.X] duration:
5.001867937s trace:
https://jaeger.dev.storj.io/trace/5de400df53722e61
FAIL (2025-12-28 12:19:04 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001969169s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
38.041698426s trace:
https://jaeger.dev.storj.io/trace/307d4901d0d5949
FAIL (2025-12-28 12:18:46 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
ise=true , address=77.90.31.16:4743 , piece_id=NRCRNEOEKXPSKREUZKGGLQHLWG5JTKWCQCRUUHKUHB4HOISVHCFQ, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.355028753 (547h59m23.226111788s) [noise=true , address=145.79.227.30:4796 , piece_id=MUD7JHR4ETJYGG7SSOOWQ5JTKSQYV26X4AWZ6XHEBHLJYE3774CA, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.355039954 (547h59m23.226122989s) [noise=true , address=45.91.116.11:4122 , piece_id=MROLZP4Y2PKVJSUUXCSDQQGVGF7AJADXWEJFFPH6SU2DWTZR443A, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.355282126 (547h59m23.226365171s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.355390983 (547h59m23.226474018s) [noise=true , address=193.149.164.143:4700 , piece_id=3FBVJX372QBFVBDXCF5VTCIT2JUYHNXFCZPJ2FN7E34HA4S4WETQ, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.355527633 (547h59m23.226610668s) [noise=true , address=77.90.31.15:4727 , piece_id=KZ74K3R4G2BD73RTJFOKFGQ57SV456NQC4W3NTPT52KBQAU3SWNA, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.355648985 (547h59m23.22673202s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.355877279 (547h59m23.226960324s) [noise=true , address=145.79.227.12:4788 , piece_id=KFB2WTRQOSZ75WMHRNCRTPS2FSMIBR27TQ55V62J2PGWLFECB4RA, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.356182711 (547h59m23.227265756s) [noise=true , address=145.79.227.49:4351 , piece_id=FYLYJ6GRQIMVTNGLLY62Q35M5F4IEQW2EPKVMSDOVJIJJPJJXF3Q, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.356536194 (547h59m23.227619229s) [noise=true , address=145.79.227.19:4092 , piece_id=7UPEHPU5NSBVOPSDNKUAQQZXBHHBS36PKZOR5FBDKLWP4VA4ZSDA, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.356869479 (547h59m23.227952514s) [noise=true , address=145.79.227.33:4882 , piece_id=VJ6XYH77LJSKKBVCCVHHTGFV3WTSCYD4XUKUMSYOWZEGZ2QY4H3A, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.357157427 (547h59m23.228240462s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.357426930 (547h59m23.228509975s) [noise=true , address=145.79.227.40:4067 , piece_id=YOIOW6QPNOMK5ZD5EEWT2FLVGVZZMHNWANUQF4UE3R4FHSOFWSWQ, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.357751889 (547h59m23.228834924s) [noise=true , address=77.90.31.10:4536 , piece_id=HKI6JPRV4D2CBIVEGHRXSCDC2OROZLX5SW3VC7W35L4DWUEITV2Q, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.358953407 (547h59m23.230036442s) [noise=true , address=77.90.31.17:4790 , piece_id=AS6RYIPXJKC6CFSPVX7MLENRTQHI7RZFLTSPHFMIYG5OS7BKZHRA, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.359286491 (547h59m23.230369536s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:24.695915344 (547h59m24.566998389s) [upload=480503]: Next returned nil segment
Dec 28 11:28:24.695941574 (547h59m24.567024619s) [upload=480503]: Waiting for error group managing segments...
Dec 28 11:28:25.041244477 (547h59m24.912327512s) [noise=true , address=193.149.164.143:4700 , piece_id=3FBVJX372QBFVBDXCF5VTCIT2JUYHNXFCZPJ2FN7E34HA4S4WETQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:25.237036778 (547h59m25.108119823s) [noise=true , address=145.79.227.49:4351 , piece_id=FYLYJ6GRQIMVTNGLLY62Q35M5F4IEQW2EPKVMSDOVJIJJPJJXF3Q, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:25.249693558 (547h59m25.120776593s) [noise=true , address=145.79.227.12:4788 , piece_id=KFB2WTRQOSZ75WMHRNCRTPS2FSMIBR27TQ55V62J2PGWLFECB4RA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:25.256819657 (547h59m25.127902692s) [noise=true , address=145.79.227.19:4092 , piece_id=7UPEHPU5NSBVOPSDNKUAQQZXBHHBS36PKZOR5FBDKLWP4VA4ZSDA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:25.511089018 (547h59m25.382172053s) [noise=true , address=45.91.116.13:4170 , piece_id=3KJJFTI5M7MBDSZNTBWSMSJUU7B325XVVLXEH3JKX5N7N4WNMXAA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:25.523026599 (547h59m25.394109634s) [noise=true , address=77.90.31.17:4790 , piece_id=AS6RYIPXJKC6CFSPVX7MLENRTQHI7RZFLTSPHFMIYG5OS7BKZHRA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:25.524217066 (547h59m25.395300101s) [noise=true , address=45.91.116.11:4122 , piece_id=MROLZP4Y2PKVJSUUXCSDQQGVGF7AJADXWEJFFPH6SU2DWTZR443A, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:25.665135729 (547h59m25.536218774s) [noise=true , address=77.90.31.15:4727 , piece_id=KZ74K3R4G2BD73RTJFOKFGQ57SV456NQC4W3NTPT52KBQAU3SWNA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:25.917975247 (547h59m25.789058292s) [noise=true , address=77.90.31.16:4743 , piece_id=NRCRNEOEKXPSKREUZKGGLQHLWG5JTKWCQCRUUHKUHB4HOISVHCFQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:26.147871236 (547h59m26.018954271s) [noise=true , address=145.79.227.30:4796 , piece_id=MUD7JHR4ETJYGG7SSOOWQ5JTKSQYV26X4AWZ6XHEBHLJYE3774CA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:26.173243839 (547h59m26.044326874s) [noise=true , address=145.79.227.44:4209 , piece_id=PKQUKW3FOPJL7BRNFHNLAJ4DSKGMGA73ROILSWEFCT7MR7H5GNNQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:26.289483660 (547h59m26.160566695s) [noise=true , address=77.90.31.10:4536 , piece_id=HKI6JPRV4D2CBIVEGHRXSCDC2OROZLX5SW3VC7W35L4DWUEITV2Q, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:26.448835962 (547h59m26.319919018s) [noise=true , address=145.79.227.40:4067 , piece_id=YOIOW6QPNOMK5ZD5EEWT2FLVGVZZMHNWANUQF4UE3R4FHSOFWSWQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:26.571228809 (547h59m26.442311844s) [noise=true , address=145.79.227.33:4882 , piece_id=VJ6XYH77LJSKKBVCCVHHTGFV3WTSCYD4XUKUMSYOWZEGZ2QY4H3A, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:26.571248396 (547h59m26.442331441s) [seg_pos={0 0}, upload=480503]: Segment reached optimal threshold of 14 pieces.
Dec 28 11:28:26.571429892 (547h59m26.442512927s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NEV2ndHw4fohVUNsfrzwojd8S11TmMST6Sz59GVnXS8vDGhWk): protocol: expected piece hash; context canceled; EOF
Dec 28 11:28:26.571436033 (547h59m26.442519078s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold
Dec 28 11:28:26.571564158 (547h59m26.442647203s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection
Dec 28 11:28:26.571580588 (547h59m26.442663623s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold
Dec 28 11:28:26.571712249 (547h59m26.442795284s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Nw78dd9WPnmCwv5bFQR3VpV39776Pye2QVGSNMTs3zq2BtaGJ): protocol: expected piece hash; context canceled; EOF
Dec 28 11:28:26.571720666 (547h59m26.442803701s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold
Dec 28 11:28:26.571767825 (547h59m26.442850871s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv): protocol: expected piece hash; context canceled; EOF
Dec 28 11:28:26.571774688 (547h59m26.442857723s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold
Dec 28 11:28:26.571792493 (547h59m26.442875538s) [seg_pos={0 0}, upload=480503]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 11:28:26.571882343 (547h59m26.442965378s) [upload=480503]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 11:28:26.571887683 (547h59m26.442970728s) [upload=480503]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 11:28:26.858811294 (547h59m26.729894329s) [upload=480503]: Done uploading segments...
Dec 28 11:28:26.858832874 (547h59m26.72991592s) [upload=480503]: Upload finished. err: <nil>
Duration: 38.041739976s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
33 out of 60 is failed (threshold=6) [X.X..X.XXX..X....XX.XX.XXXXXXXXXX..XXXX...XX.XXX.XX.......X.] duration:
3.42321296s trace:
https://jaeger.dev.storj.io/trace/71077765df5041c9
FAIL (2025-12-28 12:18:33 UTC)
orbiter warning: context deadline exceeded
Duration: 3.423235181s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [...........................X.X..............................] duration:
24.126761362s trace:
https://jaeger.dev.storj.io/trace/72d0191c7bb67437
FAIL (2025-12-28 12:18:27 UTC)
orbiter warning: context deadline exceeded
Duration: 24.126788533s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.X..........................................................] duration:
27.996237093s trace:
https://jaeger.dev.storj.io/trace/307d4901d0d5949
FAIL (2025-12-28 12:18:02 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
ise=true , address=77.90.31.16:4743 , piece_id=NRCRNEOEKXPSKREUZKGGLQHLWG5JTKWCQCRUUHKUHB4HOISVHCFQ, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.355028753 (547h59m23.226111788s) [noise=true , address=145.79.227.30:4796 , piece_id=MUD7JHR4ETJYGG7SSOOWQ5JTKSQYV26X4AWZ6XHEBHLJYE3774CA, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.355039954 (547h59m23.226122989s) [noise=true , address=45.91.116.11:4122 , piece_id=MROLZP4Y2PKVJSUUXCSDQQGVGF7AJADXWEJFFPH6SU2DWTZR443A, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.355282126 (547h59m23.226365171s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.355390983 (547h59m23.226474018s) [noise=true , address=193.149.164.143:4700 , piece_id=3FBVJX372QBFVBDXCF5VTCIT2JUYHNXFCZPJ2FN7E34HA4S4WETQ, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.355527633 (547h59m23.226610668s) [noise=true , address=77.90.31.15:4727 , piece_id=KZ74K3R4G2BD73RTJFOKFGQ57SV456NQC4W3NTPT52KBQAU3SWNA, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.355648985 (547h59m23.22673202s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.355877279 (547h59m23.226960324s) [noise=true , address=145.79.227.12:4788 , piece_id=KFB2WTRQOSZ75WMHRNCRTPS2FSMIBR27TQ55V62J2PGWLFECB4RA, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.356182711 (547h59m23.227265756s) [noise=true , address=145.79.227.49:4351 , piece_id=FYLYJ6GRQIMVTNGLLY62Q35M5F4IEQW2EPKVMSDOVJIJJPJJXF3Q, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.356536194 (547h59m23.227619229s) [noise=true , address=145.79.227.19:4092 , piece_id=7UPEHPU5NSBVOPSDNKUAQQZXBHHBS36PKZOR5FBDKLWP4VA4ZSDA, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.356869479 (547h59m23.227952514s) [noise=true , address=145.79.227.33:4882 , piece_id=VJ6XYH77LJSKKBVCCVHHTGFV3WTSCYD4XUKUMSYOWZEGZ2QY4H3A, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.357157427 (547h59m23.228240462s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.357426930 (547h59m23.228509975s) [noise=true , address=145.79.227.40:4067 , piece_id=YOIOW6QPNOMK5ZD5EEWT2FLVGVZZMHNWANUQF4UE3R4FHSOFWSWQ, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.357751889 (547h59m23.228834924s) [noise=true , address=77.90.31.10:4536 , piece_id=HKI6JPRV4D2CBIVEGHRXSCDC2OROZLX5SW3VC7W35L4DWUEITV2Q, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.358953407 (547h59m23.230036442s) [noise=true , address=77.90.31.17:4790 , piece_id=AS6RYIPXJKC6CFSPVX7MLENRTQHI7RZFLTSPHFMIYG5OS7BKZHRA, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.359286491 (547h59m23.230369536s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:24.695915344 (547h59m24.566998389s) [upload=480503]: Next returned nil segment
Dec 28 11:28:24.695941574 (547h59m24.567024619s) [upload=480503]: Waiting for error group managing segments...
Dec 28 11:28:25.041244477 (547h59m24.912327512s) [noise=true , address=193.149.164.143:4700 , piece_id=3FBVJX372QBFVBDXCF5VTCIT2JUYHNXFCZPJ2FN7E34HA4S4WETQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:25.237036778 (547h59m25.108119823s) [noise=true , address=145.79.227.49:4351 , piece_id=FYLYJ6GRQIMVTNGLLY62Q35M5F4IEQW2EPKVMSDOVJIJJPJJXF3Q, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:25.249693558 (547h59m25.120776593s) [noise=true , address=145.79.227.12:4788 , piece_id=KFB2WTRQOSZ75WMHRNCRTPS2FSMIBR27TQ55V62J2PGWLFECB4RA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:25.256819657 (547h59m25.127902692s) [noise=true , address=145.79.227.19:4092 , piece_id=7UPEHPU5NSBVOPSDNKUAQQZXBHHBS36PKZOR5FBDKLWP4VA4ZSDA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:25.511089018 (547h59m25.382172053s) [noise=true , address=45.91.116.13:4170 , piece_id=3KJJFTI5M7MBDSZNTBWSMSJUU7B325XVVLXEH3JKX5N7N4WNMXAA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:25.523026599 (547h59m25.394109634s) [noise=true , address=77.90.31.17:4790 , piece_id=AS6RYIPXJKC6CFSPVX7MLENRTQHI7RZFLTSPHFMIYG5OS7BKZHRA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:25.524217066 (547h59m25.395300101s) [noise=true , address=45.91.116.11:4122 , piece_id=MROLZP4Y2PKVJSUUXCSDQQGVGF7AJADXWEJFFPH6SU2DWTZR443A, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:25.665135729 (547h59m25.536218774s) [noise=true , address=77.90.31.15:4727 , piece_id=KZ74K3R4G2BD73RTJFOKFGQ57SV456NQC4W3NTPT52KBQAU3SWNA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:25.917975247 (547h59m25.789058292s) [noise=true , address=77.90.31.16:4743 , piece_id=NRCRNEOEKXPSKREUZKGGLQHLWG5JTKWCQCRUUHKUHB4HOISVHCFQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:26.147871236 (547h59m26.018954271s) [noise=true , address=145.79.227.30:4796 , piece_id=MUD7JHR4ETJYGG7SSOOWQ5JTKSQYV26X4AWZ6XHEBHLJYE3774CA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:26.173243839 (547h59m26.044326874s) [noise=true , address=145.79.227.44:4209 , piece_id=PKQUKW3FOPJL7BRNFHNLAJ4DSKGMGA73ROILSWEFCT7MR7H5GNNQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:26.289483660 (547h59m26.160566695s) [noise=true , address=77.90.31.10:4536 , piece_id=HKI6JPRV4D2CBIVEGHRXSCDC2OROZLX5SW3VC7W35L4DWUEITV2Q, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:26.448835962 (547h59m26.319919018s) [noise=true , address=145.79.227.40:4067 , piece_id=YOIOW6QPNOMK5ZD5EEWT2FLVGVZZMHNWANUQF4UE3R4FHSOFWSWQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:26.571228809 (547h59m26.442311844s) [noise=true , address=145.79.227.33:4882 , piece_id=VJ6XYH77LJSKKBVCCVHHTGFV3WTSCYD4XUKUMSYOWZEGZ2QY4H3A, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:26.571248396 (547h59m26.442331441s) [seg_pos={0 0}, upload=480503]: Segment reached optimal threshold of 14 pieces.
Dec 28 11:28:26.571429892 (547h59m26.442512927s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NEV2ndHw4fohVUNsfrzwojd8S11TmMST6Sz59GVnXS8vDGhWk): protocol: expected piece hash; context canceled; EOF
Dec 28 11:28:26.571436033 (547h59m26.442519078s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold
Dec 28 11:28:26.571564158 (547h59m26.442647203s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection
Dec 28 11:28:26.571580588 (547h59m26.442663623s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold
Dec 28 11:28:26.571712249 (547h59m26.442795284s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Nw78dd9WPnmCwv5bFQR3VpV39776Pye2QVGSNMTs3zq2BtaGJ): protocol: expected piece hash; context canceled; EOF
Dec 28 11:28:26.571720666 (547h59m26.442803701s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold
Dec 28 11:28:26.571767825 (547h59m26.442850871s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv): protocol: expected piece hash; context canceled; EOF
Dec 28 11:28:26.571774688 (547h59m26.442857723s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold
Dec 28 11:28:26.571792493 (547h59m26.442875538s) [seg_pos={0 0}, upload=480503]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 11:28:26.571882343 (547h59m26.442965378s) [upload=480503]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 11:28:26.571887683 (547h59m26.442970728s) [upload=480503]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 11:28:26.858811294 (547h59m26.729894329s) [upload=480503]: Done uploading segments...
Dec 28 11:28:26.858832874 (547h59m26.72991592s) [upload=480503]: Upload finished. err: <nil>
Duration: 27.996268463s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
17 out of 60 is failed (threshold=3) [..X.....XX...XXX.......X..XX....X...X.........X..XX.X....XX.] duration:
5.988991866s trace:
https://jaeger.dev.storj.io/trace/5ccd801a95dd58c3
FAIL (2025-12-28 12:17:51 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 28 12:16:58.949402526 (548h47m58.820485571s) [upload=481207]: Starting upload
Dec 28 12:16:58.949505572 (548h47m58.820588627s) [upload=481207]: Done starting upload
Dec 28 12:16:58.949634147 (548h47m58.820717212s) [upload=481207]: Uploading segments...
Dec 28 12:16:58.949766119 (548h47m58.820849194s) [upload=481207]: Got next segment. Inline: false
Dec 28 12:16:58.949840250 (548h47m58.820923295s) [upload=481207]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 12:16:58.949846752 (548h47m58.820929787s) [upload=481207]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 12:16:59.193314795 (548h47m59.06439784s) [seg_pos={0 0}, upload=481207]: Begin upload segment...
Dec 28 12:16:59.193385349 (548h47m59.064468384s) [seg_pos={0 0}, upload=481207]: Done begin upload segment.
Dec 28 12:16:59.193435215 (548h47m59.06451826s) [noise=true , address=145.79.227.15:4873 , piece_id=FGFV5RVVSUDIZTN3LWB7LDZPUOHVHJ2LAPNPS7A53OEEPKMKAQKA, seg_pos={0 0}, upload=481207]: Uploading piece...
Dec 28 12:16:59.193479990 (548h47m59.064563025s) [noise=true , address=145.79.227.19:4078 , piece_id=35PYXFUKDBFE4H5DNFRTT5W5QBN42X4475PYSXHGRBAX3ZGQTVIA, seg_pos={0 0}, upload=481207]: Uploading piece...
Dec 28 12:16:59.194230008 (548h47m59.065313054s) [noise=true , address=77.90.31.18:4805 , piece_id=YQIKJWK7NSAXBPU2PFB7P7T5W5VOELIQD4WL54U2WIVTJAUH5LRA, seg_pos={0 0}, upload=481207]: Uploading piece...
Dec 28 12:16:59.194746150 (548h47m59.065829186s) [noise=true , address=145.79.227.42:4122 , piece_id=ZE2TOEIYYEHSXNTZKPJZCVUH5MOET755BUQBSZAONHF3CTP7TMYQ, seg_pos={0 0}, upload=481207]: Uploading piece...
Dec 28 12:16:59.195181019 (548h47m59.066264064s) [noise=true , address=45.91.116.6:4974 , piece_id=L4XFEFZEFL2Z27HYTF2HXAT7ARA7VVN7MXYHVGRU7OXASDG3TQNA, seg_pos={0 0}, upload=481207]: Uploading piece...
Dec 28 12:16:59.195441034 (548h47m59.066524069s) [noise=true , address=145.79.227.32:4836 , piece_id=FVVLJKKLQPIQ77MJVQJ426B6W6ZO427AOE3O5CJ7KKQ4Z7XJYLNQ, seg_pos={0 0}, upload=481207]: Uploading piece...
Dec 28 12:16:59.195721087 (548h47m59.066804122s) [noise=true , address=45.91.116.5:4299 , piece_id=4AG4YBDFICPEPVSB26E2RM6WSM3WQTTINXX3AW7RGYGSAVEILUNQ, seg_pos={0 0}, upload=481207]: Uploading piece...
Dec 28 12:16:59.196027971 (548h47m59.067111007s) [noise=true , address=45.91.116.4:4937 , piece_id=4R7EDIQPVFLJYQ5QFPYO74XA4LOLUV6Z2OOODVUMDE3YABAPVDWA, seg_pos={0 0}, upload=481207]: Uploading piece...
Dec 28 12:16:59.196305039 (548h47m59.067388074s) [noise=true , address=145.79.227.52:4445 , piece_id=FYGTLZJ72P5RL64IH5AMW2LOGOCZ4TVU6JW3E3H4ZV3EKAZ57MQQ, seg_pos={0 0}, upload=481207]: Uploading piece...
Dec 28 12:16:59.198058108 (548h47m59.069141153s) [noise=true , address=145.79.227.17:4972 , piece_id=AQLBNDQB3AP7LP2RKR6KEMCC7TCVBK45W57XE2OHSBITEXJS3GDA, seg_pos={0 0}, upload=481207]: Uploading piece...
Dec 28 12:16:59.199088079 (548h47m59.070171124s) [noise=true , address=145.79.227.50:4374 , piece_id=XX6B6UQSHAGO3GNMW6STMZ4GNYED6HCLAGKVBXKH3RWMJHIOKVHQ, seg_pos={0 0}, upload=481207]: Uploading piece...
Dec 28 12:16:59.199510012 (548h47m59.070593047s) [noise=true , address=77.90.31.12:4589 , piece_id=ESA6P5GKD5HFNACP3Q2NKXTKZIUG3GTRKIL2UZG2JJBKWPUAXWSA, seg_pos={0 0}, upload=481207]: Uploading piece...
Dec 28 12:16:59.199873254 (548h47m59.070956289s) [noise=true , address=145.79.227.25:4619 , piece_id=ECSVE6EKPVJYRFNT3I2IDYP4TES4S476L62P4X2ALURXIRUWNLNQ, seg_pos={0 0}, upload=481207]: Uploading piece...
Dec 28 12:16:59.200778758 (548h47m59.071861793s) [noise=true , address=145.79.227.6:4257 , piece_id=7LMSVXXZGIPY3PVMYOXDRWPZCL6IXMLC6XGDT6ALQ2KIWFEO6VLQ, seg_pos={0 0}, upload=481207]: Uploading piece...
Dec 28 12:16:59.201239776 (548h47m59.072322811s) [noise=true , address=45.91.116.11:4120 , piece_id=2KKJZU5JBSAONPPYDB4UL4EU54KEFOUS3UK7DRHZVYBUVZRCH7CQ, seg_pos={0 0}, upload=481207]: Uploading piece...
Dec 28 12:16:59.204367492 (548h47m59.075450527s) [noise=true , address=145.79.227.41:4086 , piece_id=O3UVVCSNITAH5SWWPLOMBC6XXBRKBCTOQM7ZS5U2SS3ECSA4UUKQ, seg_pos={0 0}, upload=481207]: Uploading piece...
Dec 28 12:17:00.922548035 (548h48m0.79363108s) [upload=481207]: Next returned nil segment
Dec 28 12:17:00.922565488 (548h48m0.793648533s) [upload=481207]: Waiting for error group managing segments...
Dec 28 12:17:01.206986753 (548h48m1.078069798s) [noise=true , address=145.79.227.52:4445 , piece_id=FYGTLZJ72P5RL64IH5AMW2LOGOCZ4TVU6JW3E3H4ZV3EKAZ57MQQ, seg_pos={0 0}, upload=481207]: Done uploading piece. err: <nil>
Dec 28 12:17:01.207409748 (548h48m1.078492793s) [noise=true , address=145.79.227.25:4619 , piece_id=ECSVE6EKPVJYRFNT3I2IDYP4TES4S476L62P4X2ALURXIRUWNLNQ, seg_pos={0 0}, upload=481207]: Done uploading piece. err: <nil>
Dec 28 12:17:01.207606522 (548h48m1.078689567s) [noise=true , address=145.79.227.15:4873 , piece_id=FGFV5RVVSUDIZTN3LWB7LDZPUOHVHJ2LAPNPS7A53OEEPKMKAQKA, seg_pos={0 0}, upload=481207]: Done uploading piece. err: <nil>
Dec 28 12:17:01.208408159 (548h48m1.079491204s) [noise=true , address=145.79.227.6:4257 , piece_id=7LMSVXXZGIPY3PVMYOXDRWPZCL6IXMLC6XGDT6ALQ2KIWFEO6VLQ, seg_pos={0 0}, upload=481207]: Done uploading piece. err: <nil>
Dec 28 12:17:01.210393729 (548h48m1.081476765s) [noise=true , address=145.79.227.17:4972 , piece_id=AQLBNDQB3AP7LP2RKR6KEMCC7TCVBK45W57XE2OHSBITEXJS3GDA, seg_pos={0 0}, upload=481207]: Done uploading piece. err: <nil>
Dec 28 12:17:01.216864441 (548h48m1.087947486s) [noise=true , address=145.79.227.42:4122 , piece_id=ZE2TOEIYYEHSXNTZKPJZCVUH5MOET755BUQBSZAONHF3CTP7TMYQ, seg_pos={0 0}, upload=481207]: Done uploading piece. err: <nil>
Dec 28 12:17:01.221106909 (548h48m1.092189955s) [noise=true , address=145.79.227.32:4836 , piece_id=FVVLJKKLQPIQ77MJVQJ426B6W6ZO427AOE3O5CJ7KKQ4Z7XJYLNQ, seg_pos={0 0}, upload=481207]: Done uploading piece. err: <nil>
Dec 28 12:17:01.234487518 (548h48m1.105570553s) [noise=true , address=145.79.227.41:4086 , piece_id=O3UVVCSNITAH5SWWPLOMBC6XXBRKBCTOQM7ZS5U2SS3ECSA4UUKQ, seg_pos={0 0}, upload=481207]: Done uploading piece. err: <nil>
Dec 28 12:17:01.245545976 (548h48m1.116629021s) [noise=true , address=145.79.227.19:4078 , piece_id=35PYXFUKDBFE4H5DNFRTT5W5QBN42X4475PYSXHGRBAX3ZGQTVIA, seg_pos={0 0}, upload=481207]: Done uploading piece. err: <nil>
Dec 28 12:17:01.259620896 (548h48m1.130703941s) [noise=true , address=145.79.227.50:4374 , piece_id=XX6B6UQSHAGO3GNMW6STMZ4GNYED6HCLAGKVBXKH3RWMJHIOKVHQ, seg_pos={0 0}, upload=481207]: Done uploading piece. err: <nil>
Dec 28 12:17:01.408085777 (548h48m1.279168822s) [noise=true , address=77.90.31.12:4589 , piece_id=ESA6P5GKD5HFNACP3Q2NKXTKZIUG3GTRKIL2UZG2JJBKWPUAXWSA, seg_pos={0 0}, upload=481207]: Done uploading piece. err: <nil>
Dec 28 12:17:01.426563172 (548h48m1.297646217s) [noise=true , address=45.91.116.5:4299 , piece_id=4AG4YBDFICPEPVSB26E2RM6WSM3WQTTINXX3AW7RGYGSAVEILUNQ, seg_pos={0 0}, upload=481207]: Done uploading piece. err: <nil>
Dec 28 12:17:01.474267428 (548h48m1.345350523s) [noise=true , address=45.91.116.11:4120 , piece_id=2KKJZU5JBSAONPPYDB4UL4EU54KEFOUS3UK7DRHZVYBUVZRCH7CQ, seg_pos={0 0}, upload=481207]: Done uploading piece. err: <nil>
Dec 28 12:17:01.474305952 (548h48m1.345388997s) [seg_pos={0 0}, upload=481207]: Segment reached optimal threshold of 13 pieces.
Dec 28 12:17:01.474758483 (548h48m1.345841528s) [noise=true , address=45.91.116.6:4974 , piece_id=L4XFEFZEFL2Z27HYTF2HXAT7ARA7VVN7MXYHVGRU7OXASDG3TQNA, seg_pos={0 0}, upload=481207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:41814->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:41814->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41814->45.91.116.6:4974: use of closed network connection
Dec 28 12:17:01.474808137 (548h48m1.345891212s) [noise=true , address=45.91.116.6:4974 , piece_id=L4XFEFZEFL2Z27HYTF2HXAT7ARA7VVN7MXYHVGRU7OXASDG3TQNA, seg_pos={0 0}, upload=481207]: Long tail cancellation due to optimal threshold
Dec 28 12:17:01.475121584 (548h48m1.346204629s) [noise=true , address=77.90.31.18:4805 , piece_id=YQIKJWK7NSAXBPU2PFB7P7T5W5VOELIQD4WL54U2WIVTJAUH5LRA, seg_pos={0 0}, upload=481207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF
Dec 28 12:17:01.475133327 (548h48m1.346216372s) [noise=true , address=77.90.31.18:4805 , piece_id=YQIKJWK7NSAXBPU2PFB7P7T5W5VOELIQD4WL54U2WIVTJAUH5LRA, seg_pos={0 0}, upload=481207]: Long tail cancellation due to optimal threshold
Dec 28 12:17:01.475357955 (548h48m1.34644102s) [noise=true , address=45.91.116.4:4937 , piece_id=4R7EDIQPVFLJYQ5QFPYO74XA4LOLUV6Z2OOODVUMDE3YABAPVDWA, seg_pos={0 0}, upload=481207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF
Dec 28 12:17:01.475402790 (548h48m1.346485835s) [noise=true , address=45.91.116.4:4937 , piece_id=4R7EDIQPVFLJYQ5QFPYO74XA4LOLUV6Z2OOODVUMDE3YABAPVDWA, seg_pos={0 0}, upload=481207]: Long tail cancellation due to optimal threshold
Dec 28 12:17:01.475448107 (548h48m1.346531152s) [seg_pos={0 0}, upload=481207]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 12:17:01.475964088 (548h48m1.347047133s) [upload=481207]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 12:17:01.475999305 (548h48m1.34708234s) [upload=481207]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 12:17:01.759113340 (548h48m1.630196395s) [upload=481207]: Done uploading segments...
Dec 28 12:17:01.759138238 (548h48m1.630221283s) [upload=481207]: Upload finished. err: <nil>
Duration: 5.989027814s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
33 out of 60 is failed (threshold=6) [.X.X..X.XXX..X....XX.XX.XXXXXXXXXX..XXXX...XX.XXX.XX.......X] duration:
5.000812473s trace:
https://jaeger.dev.storj.io/trace/71077765df5041c9
FAIL (2025-12-28 12:17:47 UTC)
orbiter warning: context deadline exceeded
Duration: 5.000835767s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
17 out of 60 is failed (threshold=3) [...X.....XX...XXX.......X..XX....X...X.........X..XX.X....XX] duration:
30.002978952s trace:
https://jaeger.dev.storj.io/trace/5ccd801a95dd58c3
FAIL (2025-12-28 12:17:28 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 28 12:16:58.949402526 (548h47m58.820485571s) [upload=481207]: Starting upload
Dec 28 12:16:58.949505572 (548h47m58.820588627s) [upload=481207]: Done starting upload
Dec 28 12:16:58.949634147 (548h47m58.820717212s) [upload=481207]: Uploading segments...
Dec 28 12:16:58.949766119 (548h47m58.820849194s) [upload=481207]: Got next segment. Inline: false
Dec 28 12:16:58.949840250 (548h47m58.820923295s) [upload=481207]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 12:16:58.949846752 (548h47m58.820929787s) [upload=481207]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 12:16:59.193314795 (548h47m59.06439784s) [seg_pos={0 0}, upload=481207]: Begin upload segment...
Dec 28 12:16:59.193385349 (548h47m59.064468384s) [seg_pos={0 0}, upload=481207]: Done begin upload segment.
Dec 28 12:16:59.193435215 (548h47m59.06451826s) [noise=true , address=145.79.227.15:4873 , piece_id=FGFV5RVVSUDIZTN3LWB7LDZPUOHVHJ2LAPNPS7A53OEEPKMKAQKA, seg_pos={0 0}, upload=481207]: Uploading piece...
Dec 28 12:16:59.193479990 (548h47m59.064563025s) [noise=true , address=145.79.227.19:4078 , piece_id=35PYXFUKDBFE4H5DNFRTT5W5QBN42X4475PYSXHGRBAX3ZGQTVIA, seg_pos={0 0}, upload=481207]: Uploading piece...
Dec 28 12:16:59.194230008 (548h47m59.065313054s) [noise=true , address=77.90.31.18:4805 , piece_id=YQIKJWK7NSAXBPU2PFB7P7T5W5VOELIQD4WL54U2WIVTJAUH5LRA, seg_pos={0 0}, upload=481207]: Uploading piece...
Dec 28 12:16:59.194746150 (548h47m59.065829186s) [noise=true , address=145.79.227.42:4122 , piece_id=ZE2TOEIYYEHSXNTZKPJZCVUH5MOET755BUQBSZAONHF3CTP7TMYQ, seg_pos={0 0}, upload=481207]: Uploading piece...
Dec 28 12:16:59.195181019 (548h47m59.066264064s) [noise=true , address=45.91.116.6:4974 , piece_id=L4XFEFZEFL2Z27HYTF2HXAT7ARA7VVN7MXYHVGRU7OXASDG3TQNA, seg_pos={0 0}, upload=481207]: Uploading piece...
Dec 28 12:16:59.195441034 (548h47m59.066524069s) [noise=true , address=145.79.227.32:4836 , piece_id=FVVLJKKLQPIQ77MJVQJ426B6W6ZO427AOE3O5CJ7KKQ4Z7XJYLNQ, seg_pos={0 0}, upload=481207]: Uploading piece...
Dec 28 12:16:59.195721087 (548h47m59.066804122s) [noise=true , address=45.91.116.5:4299 , piece_id=4AG4YBDFICPEPVSB26E2RM6WSM3WQTTINXX3AW7RGYGSAVEILUNQ, seg_pos={0 0}, upload=481207]: Uploading piece...
Dec 28 12:16:59.196027971 (548h47m59.067111007s) [noise=true , address=45.91.116.4:4937 , piece_id=4R7EDIQPVFLJYQ5QFPYO74XA4LOLUV6Z2OOODVUMDE3YABAPVDWA, seg_pos={0 0}, upload=481207]: Uploading piece...
Dec 28 12:16:59.196305039 (548h47m59.067388074s) [noise=true , address=145.79.227.52:4445 , piece_id=FYGTLZJ72P5RL64IH5AMW2LOGOCZ4TVU6JW3E3H4ZV3EKAZ57MQQ, seg_pos={0 0}, upload=481207]: Uploading piece...
Dec 28 12:16:59.198058108 (548h47m59.069141153s) [noise=true , address=145.79.227.17:4972 , piece_id=AQLBNDQB3AP7LP2RKR6KEMCC7TCVBK45W57XE2OHSBITEXJS3GDA, seg_pos={0 0}, upload=481207]: Uploading piece...
Dec 28 12:16:59.199088079 (548h47m59.070171124s) [noise=true , address=145.79.227.50:4374 , piece_id=XX6B6UQSHAGO3GNMW6STMZ4GNYED6HCLAGKVBXKH3RWMJHIOKVHQ, seg_pos={0 0}, upload=481207]: Uploading piece...
Dec 28 12:16:59.199510012 (548h47m59.070593047s) [noise=true , address=77.90.31.12:4589 , piece_id=ESA6P5GKD5HFNACP3Q2NKXTKZIUG3GTRKIL2UZG2JJBKWPUAXWSA, seg_pos={0 0}, upload=481207]: Uploading piece...
Dec 28 12:16:59.199873254 (548h47m59.070956289s) [noise=true , address=145.79.227.25:4619 , piece_id=ECSVE6EKPVJYRFNT3I2IDYP4TES4S476L62P4X2ALURXIRUWNLNQ, seg_pos={0 0}, upload=481207]: Uploading piece...
Dec 28 12:16:59.200778758 (548h47m59.071861793s) [noise=true , address=145.79.227.6:4257 , piece_id=7LMSVXXZGIPY3PVMYOXDRWPZCL6IXMLC6XGDT6ALQ2KIWFEO6VLQ, seg_pos={0 0}, upload=481207]: Uploading piece...
Dec 28 12:16:59.201239776 (548h47m59.072322811s) [noise=true , address=45.91.116.11:4120 , piece_id=2KKJZU5JBSAONPPYDB4UL4EU54KEFOUS3UK7DRHZVYBUVZRCH7CQ, seg_pos={0 0}, upload=481207]: Uploading piece...
Dec 28 12:16:59.204367492 (548h47m59.075450527s) [noise=true , address=145.79.227.41:4086 , piece_id=O3UVVCSNITAH5SWWPLOMBC6XXBRKBCTOQM7ZS5U2SS3ECSA4UUKQ, seg_pos={0 0}, upload=481207]: Uploading piece...
Dec 28 12:17:00.922548035 (548h48m0.79363108s) [upload=481207]: Next returned nil segment
Dec 28 12:17:00.922565488 (548h48m0.793648533s) [upload=481207]: Waiting for error group managing segments...
Dec 28 12:17:01.206986753 (548h48m1.078069798s) [noise=true , address=145.79.227.52:4445 , piece_id=FYGTLZJ72P5RL64IH5AMW2LOGOCZ4TVU6JW3E3H4ZV3EKAZ57MQQ, seg_pos={0 0}, upload=481207]: Done uploading piece. err: <nil>
Dec 28 12:17:01.207409748 (548h48m1.078492793s) [noise=true , address=145.79.227.25:4619 , piece_id=ECSVE6EKPVJYRFNT3I2IDYP4TES4S476L62P4X2ALURXIRUWNLNQ, seg_pos={0 0}, upload=481207]: Done uploading piece. err: <nil>
Dec 28 12:17:01.207606522 (548h48m1.078689567s) [noise=true , address=145.79.227.15:4873 , piece_id=FGFV5RVVSUDIZTN3LWB7LDZPUOHVHJ2LAPNPS7A53OEEPKMKAQKA, seg_pos={0 0}, upload=481207]: Done uploading piece. err: <nil>
Dec 28 12:17:01.208408159 (548h48m1.079491204s) [noise=true , address=145.79.227.6:4257 , piece_id=7LMSVXXZGIPY3PVMYOXDRWPZCL6IXMLC6XGDT6ALQ2KIWFEO6VLQ, seg_pos={0 0}, upload=481207]: Done uploading piece. err: <nil>
Dec 28 12:17:01.210393729 (548h48m1.081476765s) [noise=true , address=145.79.227.17:4972 , piece_id=AQLBNDQB3AP7LP2RKR6KEMCC7TCVBK45W57XE2OHSBITEXJS3GDA, seg_pos={0 0}, upload=481207]: Done uploading piece. err: <nil>
Dec 28 12:17:01.216864441 (548h48m1.087947486s) [noise=true , address=145.79.227.42:4122 , piece_id=ZE2TOEIYYEHSXNTZKPJZCVUH5MOET755BUQBSZAONHF3CTP7TMYQ, seg_pos={0 0}, upload=481207]: Done uploading piece. err: <nil>
Dec 28 12:17:01.221106909 (548h48m1.092189955s) [noise=true , address=145.79.227.32:4836 , piece_id=FVVLJKKLQPIQ77MJVQJ426B6W6ZO427AOE3O5CJ7KKQ4Z7XJYLNQ, seg_pos={0 0}, upload=481207]: Done uploading piece. err: <nil>
Dec 28 12:17:01.234487518 (548h48m1.105570553s) [noise=true , address=145.79.227.41:4086 , piece_id=O3UVVCSNITAH5SWWPLOMBC6XXBRKBCTOQM7ZS5U2SS3ECSA4UUKQ, seg_pos={0 0}, upload=481207]: Done uploading piece. err: <nil>
Dec 28 12:17:01.245545976 (548h48m1.116629021s) [noise=true , address=145.79.227.19:4078 , piece_id=35PYXFUKDBFE4H5DNFRTT5W5QBN42X4475PYSXHGRBAX3ZGQTVIA, seg_pos={0 0}, upload=481207]: Done uploading piece. err: <nil>
Dec 28 12:17:01.259620896 (548h48m1.130703941s) [noise=true , address=145.79.227.50:4374 , piece_id=XX6B6UQSHAGO3GNMW6STMZ4GNYED6HCLAGKVBXKH3RWMJHIOKVHQ, seg_pos={0 0}, upload=481207]: Done uploading piece. err: <nil>
Dec 28 12:17:01.408085777 (548h48m1.279168822s) [noise=true , address=77.90.31.12:4589 , piece_id=ESA6P5GKD5HFNACP3Q2NKXTKZIUG3GTRKIL2UZG2JJBKWPUAXWSA, seg_pos={0 0}, upload=481207]: Done uploading piece. err: <nil>
Dec 28 12:17:01.426563172 (548h48m1.297646217s) [noise=true , address=45.91.116.5:4299 , piece_id=4AG4YBDFICPEPVSB26E2RM6WSM3WQTTINXX3AW7RGYGSAVEILUNQ, seg_pos={0 0}, upload=481207]: Done uploading piece. err: <nil>
Dec 28 12:17:01.474267428 (548h48m1.345350523s) [noise=true , address=45.91.116.11:4120 , piece_id=2KKJZU5JBSAONPPYDB4UL4EU54KEFOUS3UK7DRHZVYBUVZRCH7CQ, seg_pos={0 0}, upload=481207]: Done uploading piece. err: <nil>
Dec 28 12:17:01.474305952 (548h48m1.345388997s) [seg_pos={0 0}, upload=481207]: Segment reached optimal threshold of 13 pieces.
Dec 28 12:17:01.474758483 (548h48m1.345841528s) [noise=true , address=45.91.116.6:4974 , piece_id=L4XFEFZEFL2Z27HYTF2HXAT7ARA7VVN7MXYHVGRU7OXASDG3TQNA, seg_pos={0 0}, upload=481207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:41814->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:41814->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41814->45.91.116.6:4974: use of closed network connection
Dec 28 12:17:01.474808137 (548h48m1.345891212s) [noise=true , address=45.91.116.6:4974 , piece_id=L4XFEFZEFL2Z27HYTF2HXAT7ARA7VVN7MXYHVGRU7OXASDG3TQNA, seg_pos={0 0}, upload=481207]: Long tail cancellation due to optimal threshold
Dec 28 12:17:01.475121584 (548h48m1.346204629s) [noise=true , address=77.90.31.18:4805 , piece_id=YQIKJWK7NSAXBPU2PFB7P7T5W5VOELIQD4WL54U2WIVTJAUH5LRA, seg_pos={0 0}, upload=481207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF
Dec 28 12:17:01.475133327 (548h48m1.346216372s) [noise=true , address=77.90.31.18:4805 , piece_id=YQIKJWK7NSAXBPU2PFB7P7T5W5VOELIQD4WL54U2WIVTJAUH5LRA, seg_pos={0 0}, upload=481207]: Long tail cancellation due to optimal threshold
Dec 28 12:17:01.475357955 (548h48m1.34644102s) [noise=true , address=45.91.116.4:4937 , piece_id=4R7EDIQPVFLJYQ5QFPYO74XA4LOLUV6Z2OOODVUMDE3YABAPVDWA, seg_pos={0 0}, upload=481207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF
Dec 28 12:17:01.475402790 (548h48m1.346485835s) [noise=true , address=45.91.116.4:4937 , piece_id=4R7EDIQPVFLJYQ5QFPYO74XA4LOLUV6Z2OOODVUMDE3YABAPVDWA, seg_pos={0 0}, upload=481207]: Long tail cancellation due to optimal threshold
Dec 28 12:17:01.475448107 (548h48m1.346531152s) [seg_pos={0 0}, upload=481207]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 12:17:01.475964088 (548h48m1.347047133s) [upload=481207]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 12:17:01.475999305 (548h48m1.34708234s) [upload=481207]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 12:17:01.759113340 (548h48m1.630196395s) [upload=481207]: Done uploading segments...
Dec 28 12:17:01.759138238 (548h48m1.630221283s) [upload=481207]: Upload finished. err: <nil>
Duration: 30.003001846s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [............................X.X.............................] duration:
10.090761137s trace:
https://jaeger.dev.storj.io/trace/72d0191c7bb67437
FAIL (2025-12-28 12:17:12 UTC)
orbiter warning: context deadline exceeded
Duration: 10.090806023s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [..X.........................................................] duration:
21.251045696s trace:
https://jaeger.dev.storj.io/trace/307d4901d0d5949
FAIL (2025-12-28 12:16:51 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
ise=true , address=77.90.31.16:4743 , piece_id=NRCRNEOEKXPSKREUZKGGLQHLWG5JTKWCQCRUUHKUHB4HOISVHCFQ, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.355028753 (547h59m23.226111788s) [noise=true , address=145.79.227.30:4796 , piece_id=MUD7JHR4ETJYGG7SSOOWQ5JTKSQYV26X4AWZ6XHEBHLJYE3774CA, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.355039954 (547h59m23.226122989s) [noise=true , address=45.91.116.11:4122 , piece_id=MROLZP4Y2PKVJSUUXCSDQQGVGF7AJADXWEJFFPH6SU2DWTZR443A, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.355282126 (547h59m23.226365171s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.355390983 (547h59m23.226474018s) [noise=true , address=193.149.164.143:4700 , piece_id=3FBVJX372QBFVBDXCF5VTCIT2JUYHNXFCZPJ2FN7E34HA4S4WETQ, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.355527633 (547h59m23.226610668s) [noise=true , address=77.90.31.15:4727 , piece_id=KZ74K3R4G2BD73RTJFOKFGQ57SV456NQC4W3NTPT52KBQAU3SWNA, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.355648985 (547h59m23.22673202s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.355877279 (547h59m23.226960324s) [noise=true , address=145.79.227.12:4788 , piece_id=KFB2WTRQOSZ75WMHRNCRTPS2FSMIBR27TQ55V62J2PGWLFECB4RA, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.356182711 (547h59m23.227265756s) [noise=true , address=145.79.227.49:4351 , piece_id=FYLYJ6GRQIMVTNGLLY62Q35M5F4IEQW2EPKVMSDOVJIJJPJJXF3Q, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.356536194 (547h59m23.227619229s) [noise=true , address=145.79.227.19:4092 , piece_id=7UPEHPU5NSBVOPSDNKUAQQZXBHHBS36PKZOR5FBDKLWP4VA4ZSDA, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.356869479 (547h59m23.227952514s) [noise=true , address=145.79.227.33:4882 , piece_id=VJ6XYH77LJSKKBVCCVHHTGFV3WTSCYD4XUKUMSYOWZEGZ2QY4H3A, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.357157427 (547h59m23.228240462s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.357426930 (547h59m23.228509975s) [noise=true , address=145.79.227.40:4067 , piece_id=YOIOW6QPNOMK5ZD5EEWT2FLVGVZZMHNWANUQF4UE3R4FHSOFWSWQ, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.357751889 (547h59m23.228834924s) [noise=true , address=77.90.31.10:4536 , piece_id=HKI6JPRV4D2CBIVEGHRXSCDC2OROZLX5SW3VC7W35L4DWUEITV2Q, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.358953407 (547h59m23.230036442s) [noise=true , address=77.90.31.17:4790 , piece_id=AS6RYIPXJKC6CFSPVX7MLENRTQHI7RZFLTSPHFMIYG5OS7BKZHRA, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.359286491 (547h59m23.230369536s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:24.695915344 (547h59m24.566998389s) [upload=480503]: Next returned nil segment
Dec 28 11:28:24.695941574 (547h59m24.567024619s) [upload=480503]: Waiting for error group managing segments...
Dec 28 11:28:25.041244477 (547h59m24.912327512s) [noise=true , address=193.149.164.143:4700 , piece_id=3FBVJX372QBFVBDXCF5VTCIT2JUYHNXFCZPJ2FN7E34HA4S4WETQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:25.237036778 (547h59m25.108119823s) [noise=true , address=145.79.227.49:4351 , piece_id=FYLYJ6GRQIMVTNGLLY62Q35M5F4IEQW2EPKVMSDOVJIJJPJJXF3Q, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:25.249693558 (547h59m25.120776593s) [noise=true , address=145.79.227.12:4788 , piece_id=KFB2WTRQOSZ75WMHRNCRTPS2FSMIBR27TQ55V62J2PGWLFECB4RA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:25.256819657 (547h59m25.127902692s) [noise=true , address=145.79.227.19:4092 , piece_id=7UPEHPU5NSBVOPSDNKUAQQZXBHHBS36PKZOR5FBDKLWP4VA4ZSDA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:25.511089018 (547h59m25.382172053s) [noise=true , address=45.91.116.13:4170 , piece_id=3KJJFTI5M7MBDSZNTBWSMSJUU7B325XVVLXEH3JKX5N7N4WNMXAA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:25.523026599 (547h59m25.394109634s) [noise=true , address=77.90.31.17:4790 , piece_id=AS6RYIPXJKC6CFSPVX7MLENRTQHI7RZFLTSPHFMIYG5OS7BKZHRA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:25.524217066 (547h59m25.395300101s) [noise=true , address=45.91.116.11:4122 , piece_id=MROLZP4Y2PKVJSUUXCSDQQGVGF7AJADXWEJFFPH6SU2DWTZR443A, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:25.665135729 (547h59m25.536218774s) [noise=true , address=77.90.31.15:4727 , piece_id=KZ74K3R4G2BD73RTJFOKFGQ57SV456NQC4W3NTPT52KBQAU3SWNA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:25.917975247 (547h59m25.789058292s) [noise=true , address=77.90.31.16:4743 , piece_id=NRCRNEOEKXPSKREUZKGGLQHLWG5JTKWCQCRUUHKUHB4HOISVHCFQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:26.147871236 (547h59m26.018954271s) [noise=true , address=145.79.227.30:4796 , piece_id=MUD7JHR4ETJYGG7SSOOWQ5JTKSQYV26X4AWZ6XHEBHLJYE3774CA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:26.173243839 (547h59m26.044326874s) [noise=true , address=145.79.227.44:4209 , piece_id=PKQUKW3FOPJL7BRNFHNLAJ4DSKGMGA73ROILSWEFCT7MR7H5GNNQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:26.289483660 (547h59m26.160566695s) [noise=true , address=77.90.31.10:4536 , piece_id=HKI6JPRV4D2CBIVEGHRXSCDC2OROZLX5SW3VC7W35L4DWUEITV2Q, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:26.448835962 (547h59m26.319919018s) [noise=true , address=145.79.227.40:4067 , piece_id=YOIOW6QPNOMK5ZD5EEWT2FLVGVZZMHNWANUQF4UE3R4FHSOFWSWQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:26.571228809 (547h59m26.442311844s) [noise=true , address=145.79.227.33:4882 , piece_id=VJ6XYH77LJSKKBVCCVHHTGFV3WTSCYD4XUKUMSYOWZEGZ2QY4H3A, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:26.571248396 (547h59m26.442331441s) [seg_pos={0 0}, upload=480503]: Segment reached optimal threshold of 14 pieces.
Dec 28 11:28:26.571429892 (547h59m26.442512927s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NEV2ndHw4fohVUNsfrzwojd8S11TmMST6Sz59GVnXS8vDGhWk): protocol: expected piece hash; context canceled; EOF
Dec 28 11:28:26.571436033 (547h59m26.442519078s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold
Dec 28 11:28:26.571564158 (547h59m26.442647203s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection
Dec 28 11:28:26.571580588 (547h59m26.442663623s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold
Dec 28 11:28:26.571712249 (547h59m26.442795284s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Nw78dd9WPnmCwv5bFQR3VpV39776Pye2QVGSNMTs3zq2BtaGJ): protocol: expected piece hash; context canceled; EOF
Dec 28 11:28:26.571720666 (547h59m26.442803701s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold
Dec 28 11:28:26.571767825 (547h59m26.442850871s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv): protocol: expected piece hash; context canceled; EOF
Dec 28 11:28:26.571774688 (547h59m26.442857723s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold
Dec 28 11:28:26.571792493 (547h59m26.442875538s) [seg_pos={0 0}, upload=480503]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 11:28:26.571882343 (547h59m26.442965378s) [upload=480503]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 11:28:26.571887683 (547h59m26.442970728s) [upload=480503]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 11:28:26.858811294 (547h59m26.729894329s) [upload=480503]: Done uploading segments...
Dec 28 11:28:26.858832874 (547h59m26.72991592s) [upload=480503]: Upload finished. err: <nil>
Duration: 21.251081234s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
33 out of 60 is failed (threshold=6) [X.X.X..X.XXX..X....XX.XX.XXXXXXXXXX..XXXX...XX.XXX.XX.......] duration:
3.561878891s trace:
https://jaeger.dev.storj.io/trace/4a7cc66cea50a588
FAIL (2025-12-28 12:16:43 UTC)
orbiter warning: context deadline exceeded
Duration: 3.561903869s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.............................X.X............................] duration:
9.235808429s trace:
https://jaeger.dev.storj.io/trace/72d0191c7bb67437
FAIL (2025-12-28 12:16:29 UTC)
orbiter warning: context deadline exceeded
Duration: 9.235837414s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
16 out of 60 is failed (threshold=3) [....X.....XX...XXX.......X..XX....X...X.........X..XX.X....X] duration:
30.002802371s trace:
https://jaeger.dev.storj.io/trace/46c966c523b002db
FAIL (2025-12-28 12:16:28 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 28 12:15:59.304009407 (548h46m59.175092442s) [upload=481193]: Starting upload
Dec 28 12:15:59.304096843 (548h46m59.175179878s) [upload=481193]: Done starting upload
Dec 28 12:15:59.304173199 (548h46m59.175256244s) [upload=481193]: Uploading segments...
Dec 28 12:15:59.304352881 (548h46m59.175435926s) [upload=481193]: Got next segment. Inline: false
Dec 28 12:15:59.304391154 (548h46m59.175474199s) [upload=481193]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 12:15:59.304396865 (548h46m59.17547991s) [upload=481193]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 12:15:59.546467667 (548h46m59.417550722s) [seg_pos={0 0}, upload=481193]: Begin upload segment...
Dec 28 12:15:59.546525427 (548h46m59.417608462s) [seg_pos={0 0}, upload=481193]: Done begin upload segment.
Dec 28 12:15:59.546561736 (548h46m59.417644781s) [noise=true , address=145.79.227.45:4229 , piece_id=7TMGCATQXNZVGHYYUH6MRROPGY2SPFELVLD6RGU3PL2BDBUNC4NA, seg_pos={0 0}, upload=481193]: Uploading piece...
Dec 28 12:15:59.546587506 (548h46m59.417670551s) [noise=true , address=145.79.227.13:4802 , piece_id=QCF3UDTVSERVEPKXBVWJWS5ZYCUP4SFI2UE2Z6C3CER3RNCTAYZA, seg_pos={0 0}, upload=481193]: Uploading piece...
Dec 28 12:15:59.547014568 (548h46m59.418097604s) [noise=true , address=45.91.116.8:4048 , piece_id=X7AWY4CGHTJ22LWIGC2VVSIKAMHSI36CBDZKFLYEIMLRN7Y27XOA, seg_pos={0 0}, upload=481193]: Uploading piece...
Dec 28 12:15:59.547158272 (548h46m59.418241307s) [noise=true , address=145.79.227.38:4977 , piece_id=CFHW6MOLXQGPS7XXTFHQ57T62A7JOVD4KENYLLPAWKLYRMLFREZA, seg_pos={0 0}, upload=481193]: Uploading piece...
Dec 28 12:15:59.547388180 (548h46m59.418471225s) [noise=true , address=45.91.116.11:4119 , piece_id=TVRK4NJS2P4V5UQRLOR47EYO37KKDELYUR45BWWM6ECUEONS4DAA, seg_pos={0 0}, upload=481193]: Uploading piece...
Dec 28 12:15:59.547440499 (548h46m59.418523534s) [noise=true , address=145.79.227.11:4731 , piece_id=FEHQG7VZPNITB5CSU63B2IIWC5FVSPX6R4UV2HY5BA2M3M7S2OBA, seg_pos={0 0}, upload=481193]: Uploading piece...
Dec 28 12:15:59.547727295 (548h46m59.41881033s) [noise=true , address=77.90.31.21:4877 , piece_id=PHC5I4BJ4ZKDRYS6MK6RKMMFJ6VEZH7GAU65T57W27BVCAUB7Z3Q, seg_pos={0 0}, upload=481193]: Uploading piece...
Dec 28 12:15:59.547787811 (548h46m59.418870846s) [noise=true , address=45.91.116.6:4976 , piece_id=YZK2T2AXFJZEN4EWZSRNHRM3GR67F4GSZHNVU2LYD2CNQUBE2N7Q, seg_pos={0 0}, upload=481193]: Uploading piece...
Dec 28 12:15:59.548030693 (548h46m59.419113738s) [noise=true , address=145.79.227.14:4837 , piece_id=STEDS2WQ32GXHV6HB2CCSH2KLWZVVEJJ73C3CBOMQOPMCEPXWJWA, seg_pos={0 0}, upload=481193]: Uploading piece...
Dec 28 12:15:59.548090197 (548h46m59.419173232s) [noise=true , address=145.79.227.18:4007 , piece_id=6VYD3HKQRMICQXRB3JB3M3C5PUBFKISP5K22WSTUWAT3UTKC6AWQ, seg_pos={0 0}, upload=481193]: Uploading piece...
Dec 28 12:15:59.548363608 (548h46m59.419446643s) [noise=true , address=145.79.227.40:4051 , piece_id=RKBPOJOPSHGH4XMGIWZUIM4V5JJLXYH3J2KLJY6NUKSS65FKTY7A, seg_pos={0 0}, upload=481193]: Uploading piece...
Dec 28 12:15:59.548387242 (548h46m59.419470277s) [noise=true , address=145.79.227.28:4726 , piece_id=65ZPIGL5R5BSOWROVH6K7WUEPYH4547ZPZPN5DD6MQHX52QSBZMQ, seg_pos={0 0}, upload=481193]: Uploading piece...
Dec 28 12:15:59.548669419 (548h46m59.419752454s) [noise=true , address=145.79.227.42:4123 , piece_id=WQWHREHVK6TT7IIVQMO5SX64OPQJFHVTYODQ5EXKPHOQSCX6CGXQ, seg_pos={0 0}, upload=481193]: Uploading piece...
Dec 28 12:15:59.548681793 (548h46m59.419764828s) [noise=true , address=145.79.227.4:4186 , piece_id=XPSGXFALW3YR5ZXF63J4QF653FWIIK3OVGOQYHMT3OSOBU6HR5HQ, seg_pos={0 0}, upload=481193]: Uploading piece...
Dec 28 12:15:59.548922472 (548h46m59.420005517s) [noise=true , address=77.90.31.16:4735 , piece_id=3JCFLPIJYUKONRKRAZLEUQXCXNZDQRGMEEEIGELTFHYT6KVHDGJA, seg_pos={0 0}, upload=481193]: Uploading piece...
Dec 28 12:15:59.548978037 (548h46m59.420061072s) [noise=true , address=145.79.227.52:4447 , piece_id=K2GZM5Z3AMRWOFXTNON7YTISAWNON5K4PBJXPLT5YKF7LI6R2W4A, seg_pos={0 0}, upload=481193]: Uploading piece...
Dec 28 12:16:01.272399099 (548h47m1.143482134s) [upload=481193]: Next returned nil segment
Dec 28 12:16:01.272417243 (548h47m1.143500278s) [upload=481193]: Waiting for error group managing segments...
Dec 28 12:16:01.555803016 (548h47m1.426886051s) [noise=true , address=145.79.227.28:4726 , piece_id=65ZPIGL5R5BSOWROVH6K7WUEPYH4547ZPZPN5DD6MQHX52QSBZMQ, seg_pos={0 0}, upload=481193]: Done uploading piece. err: <nil>
Dec 28 12:16:01.557698544 (548h47m1.42878159s) [noise=true , address=145.79.227.38:4977 , piece_id=CFHW6MOLXQGPS7XXTFHQ57T62A7JOVD4KENYLLPAWKLYRMLFREZA, seg_pos={0 0}, upload=481193]: Done uploading piece. err: <nil>
Dec 28 12:16:01.561172480 (548h47m1.432255525s) [noise=true , address=145.79.227.14:4837 , piece_id=STEDS2WQ32GXHV6HB2CCSH2KLWZVVEJJ73C3CBOMQOPMCEPXWJWA, seg_pos={0 0}, upload=481193]: Done uploading piece. err: <nil>
Dec 28 12:16:01.568554246 (548h47m1.439637291s) [noise=true , address=145.79.227.52:4447 , piece_id=K2GZM5Z3AMRWOFXTNON7YTISAWNON5K4PBJXPLT5YKF7LI6R2W4A, seg_pos={0 0}, upload=481193]: Done uploading piece. err: <nil>
Dec 28 12:16:01.569692844 (548h47m1.440775889s) [noise=true , address=145.79.227.42:4123 , piece_id=WQWHREHVK6TT7IIVQMO5SX64OPQJFHVTYODQ5EXKPHOQSCX6CGXQ, seg_pos={0 0}, upload=481193]: Done uploading piece. err: <nil>
Dec 28 12:16:01.570756930 (548h47m1.441839965s) [noise=true , address=145.79.227.11:4731 , piece_id=FEHQG7VZPNITB5CSU63B2IIWC5FVSPX6R4UV2HY5BA2M3M7S2OBA, seg_pos={0 0}, upload=481193]: Done uploading piece. err: <nil>
Dec 28 12:16:01.576611017 (548h47m1.447694062s) [noise=true , address=145.79.227.18:4007 , piece_id=6VYD3HKQRMICQXRB3JB3M3C5PUBFKISP5K22WSTUWAT3UTKC6AWQ, seg_pos={0 0}, upload=481193]: Done uploading piece. err: <nil>
Dec 28 12:16:01.586783807 (548h47m1.457866852s) [noise=true , address=145.79.227.45:4229 , piece_id=7TMGCATQXNZVGHYYUH6MRROPGY2SPFELVLD6RGU3PL2BDBUNC4NA, seg_pos={0 0}, upload=481193]: Done uploading piece. err: <nil>
Dec 28 12:16:01.593061811 (548h47m1.464144846s) [noise=true , address=145.79.227.13:4802 , piece_id=QCF3UDTVSERVEPKXBVWJWS5ZYCUP4SFI2UE2Z6C3CER3RNCTAYZA, seg_pos={0 0}, upload=481193]: Done uploading piece. err: <nil>
Dec 28 12:16:01.606540718 (548h47m1.477623763s) [noise=true , address=145.79.227.4:4186 , piece_id=XPSGXFALW3YR5ZXF63J4QF653FWIIK3OVGOQYHMT3OSOBU6HR5HQ, seg_pos={0 0}, upload=481193]: Done uploading piece. err: <nil>
Dec 28 12:16:01.804638047 (548h47m1.675721092s) [noise=true , address=45.91.116.6:4976 , piece_id=YZK2T2AXFJZEN4EWZSRNHRM3GR67F4GSZHNVU2LYD2CNQUBE2N7Q, seg_pos={0 0}, upload=481193]: Done uploading piece. err: <nil>
Dec 28 12:16:01.841219154 (548h47m1.712302199s) [noise=true , address=45.91.116.11:4119 , piece_id=TVRK4NJS2P4V5UQRLOR47EYO37KKDELYUR45BWWM6ECUEONS4DAA, seg_pos={0 0}, upload=481193]: Done uploading piece. err: <nil>
Dec 28 12:16:01.862923422 (548h47m1.734006457s) [noise=true , address=77.90.31.16:4735 , piece_id=3JCFLPIJYUKONRKRAZLEUQXCXNZDQRGMEEEIGELTFHYT6KVHDGJA, seg_pos={0 0}, upload=481193]: Done uploading piece. err: <nil>
Dec 28 12:16:01.862949211 (548h47m1.734032246s) [seg_pos={0 0}, upload=481193]: Segment reached optimal threshold of 13 pieces.
Dec 28 12:16:01.863186162 (548h47m1.734269207s) [noise=true , address=145.79.227.40:4051 , piece_id=RKBPOJOPSHGH4XMGIWZUIM4V5JJLXYH3J2KLJY6NUKSS65FKTY7A, seg_pos={0 0}, upload=481193]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF
Dec 28 12:16:01.863194087 (548h47m1.734277132s) [noise=true , address=145.79.227.40:4051 , piece_id=RKBPOJOPSHGH4XMGIWZUIM4V5JJLXYH3J2KLJY6NUKSS65FKTY7A, seg_pos={0 0}, upload=481193]: Long tail cancellation due to optimal threshold
Dec 28 12:16:01.863210018 (548h47m1.734293063s) [noise=true , address=45.91.116.8:4048 , piece_id=X7AWY4CGHTJ22LWIGC2VVSIKAMHSI36CBDZKFLYEIMLRN7Y27XOA, seg_pos={0 0}, upload=481193]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF
Dec 28 12:16:01.863218704 (548h47m1.734301749s) [noise=true , address=45.91.116.8:4048 , piece_id=X7AWY4CGHTJ22LWIGC2VVSIKAMHSI36CBDZKFLYEIMLRN7Y27XOA, seg_pos={0 0}, upload=481193]: Long tail cancellation due to optimal threshold
Dec 28 12:16:01.863450105 (548h47m1.73453315s) [noise=true , address=77.90.31.21:4877 , piece_id=PHC5I4BJ4ZKDRYS6MK6RKMMFJ6VEZH7GAU65T57W27BVCAUB7Z3Q, seg_pos={0 0}, upload=481193]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: context canceled; context canceled
Dec 28 12:16:01.863457048 (548h47m1.734540093s) [noise=true , address=77.90.31.21:4877 , piece_id=PHC5I4BJ4ZKDRYS6MK6RKMMFJ6VEZH7GAU65T57W27BVCAUB7Z3Q, seg_pos={0 0}, upload=481193]: Long tail cancellation due to optimal threshold
Dec 28 12:16:01.863473319 (548h47m1.734556364s) [seg_pos={0 0}, upload=481193]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 12:16:01.863558752 (548h47m1.734641787s) [upload=481193]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 12:16:01.863564603 (548h47m1.734647648s) [upload=481193]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 12:16:02.155336213 (548h47m2.026419258s) [upload=481193]: Done uploading segments...
Dec 28 12:16:02.155360950 (548h47m2.026443995s) [upload=481193]: Upload finished. err: <nil>
Duration: 30.002825524s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..............................X.X...........................] duration:
27.99257076s trace:
https://jaeger.dev.storj.io/trace/72d0191c7bb67437
FAIL (2025-12-28 12:16:11 UTC)
orbiter warning: context deadline exceeded
Duration: 27.992596039s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
33 out of 60 is failed (threshold=6) [.X.X.X..X.XXX..X....XX.XX.XXXXXXXXXX..XXXX...XX.XXX.XX......] duration:
5.001046208s trace:
https://jaeger.dev.storj.io/trace/4a7cc66cea50a588
FAIL (2025-12-28 12:16:06 UTC)
orbiter warning: context deadline exceeded
Duration: 5.00107316s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [X..X........................................................] duration:
31.220849646s trace:
https://jaeger.dev.storj.io/trace/307d4901d0d5949
FAIL (2025-12-28 12:15:54 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
ise=true , address=77.90.31.16:4743 , piece_id=NRCRNEOEKXPSKREUZKGGLQHLWG5JTKWCQCRUUHKUHB4HOISVHCFQ, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.355028753 (547h59m23.226111788s) [noise=true , address=145.79.227.30:4796 , piece_id=MUD7JHR4ETJYGG7SSOOWQ5JTKSQYV26X4AWZ6XHEBHLJYE3774CA, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.355039954 (547h59m23.226122989s) [noise=true , address=45.91.116.11:4122 , piece_id=MROLZP4Y2PKVJSUUXCSDQQGVGF7AJADXWEJFFPH6SU2DWTZR443A, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.355282126 (547h59m23.226365171s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.355390983 (547h59m23.226474018s) [noise=true , address=193.149.164.143:4700 , piece_id=3FBVJX372QBFVBDXCF5VTCIT2JUYHNXFCZPJ2FN7E34HA4S4WETQ, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.355527633 (547h59m23.226610668s) [noise=true , address=77.90.31.15:4727 , piece_id=KZ74K3R4G2BD73RTJFOKFGQ57SV456NQC4W3NTPT52KBQAU3SWNA, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.355648985 (547h59m23.22673202s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.355877279 (547h59m23.226960324s) [noise=true , address=145.79.227.12:4788 , piece_id=KFB2WTRQOSZ75WMHRNCRTPS2FSMIBR27TQ55V62J2PGWLFECB4RA, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.356182711 (547h59m23.227265756s) [noise=true , address=145.79.227.49:4351 , piece_id=FYLYJ6GRQIMVTNGLLY62Q35M5F4IEQW2EPKVMSDOVJIJJPJJXF3Q, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.356536194 (547h59m23.227619229s) [noise=true , address=145.79.227.19:4092 , piece_id=7UPEHPU5NSBVOPSDNKUAQQZXBHHBS36PKZOR5FBDKLWP4VA4ZSDA, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.356869479 (547h59m23.227952514s) [noise=true , address=145.79.227.33:4882 , piece_id=VJ6XYH77LJSKKBVCCVHHTGFV3WTSCYD4XUKUMSYOWZEGZ2QY4H3A, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.357157427 (547h59m23.228240462s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.357426930 (547h59m23.228509975s) [noise=true , address=145.79.227.40:4067 , piece_id=YOIOW6QPNOMK5ZD5EEWT2FLVGVZZMHNWANUQF4UE3R4FHSOFWSWQ, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.357751889 (547h59m23.228834924s) [noise=true , address=77.90.31.10:4536 , piece_id=HKI6JPRV4D2CBIVEGHRXSCDC2OROZLX5SW3VC7W35L4DWUEITV2Q, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.358953407 (547h59m23.230036442s) [noise=true , address=77.90.31.17:4790 , piece_id=AS6RYIPXJKC6CFSPVX7MLENRTQHI7RZFLTSPHFMIYG5OS7BKZHRA, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.359286491 (547h59m23.230369536s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:24.695915344 (547h59m24.566998389s) [upload=480503]: Next returned nil segment
Dec 28 11:28:24.695941574 (547h59m24.567024619s) [upload=480503]: Waiting for error group managing segments...
Dec 28 11:28:25.041244477 (547h59m24.912327512s) [noise=true , address=193.149.164.143:4700 , piece_id=3FBVJX372QBFVBDXCF5VTCIT2JUYHNXFCZPJ2FN7E34HA4S4WETQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:25.237036778 (547h59m25.108119823s) [noise=true , address=145.79.227.49:4351 , piece_id=FYLYJ6GRQIMVTNGLLY62Q35M5F4IEQW2EPKVMSDOVJIJJPJJXF3Q, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:25.249693558 (547h59m25.120776593s) [noise=true , address=145.79.227.12:4788 , piece_id=KFB2WTRQOSZ75WMHRNCRTPS2FSMIBR27TQ55V62J2PGWLFECB4RA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:25.256819657 (547h59m25.127902692s) [noise=true , address=145.79.227.19:4092 , piece_id=7UPEHPU5NSBVOPSDNKUAQQZXBHHBS36PKZOR5FBDKLWP4VA4ZSDA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:25.511089018 (547h59m25.382172053s) [noise=true , address=45.91.116.13:4170 , piece_id=3KJJFTI5M7MBDSZNTBWSMSJUU7B325XVVLXEH3JKX5N7N4WNMXAA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:25.523026599 (547h59m25.394109634s) [noise=true , address=77.90.31.17:4790 , piece_id=AS6RYIPXJKC6CFSPVX7MLENRTQHI7RZFLTSPHFMIYG5OS7BKZHRA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:25.524217066 (547h59m25.395300101s) [noise=true , address=45.91.116.11:4122 , piece_id=MROLZP4Y2PKVJSUUXCSDQQGVGF7AJADXWEJFFPH6SU2DWTZR443A, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:25.665135729 (547h59m25.536218774s) [noise=true , address=77.90.31.15:4727 , piece_id=KZ74K3R4G2BD73RTJFOKFGQ57SV456NQC4W3NTPT52KBQAU3SWNA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:25.917975247 (547h59m25.789058292s) [noise=true , address=77.90.31.16:4743 , piece_id=NRCRNEOEKXPSKREUZKGGLQHLWG5JTKWCQCRUUHKUHB4HOISVHCFQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:26.147871236 (547h59m26.018954271s) [noise=true , address=145.79.227.30:4796 , piece_id=MUD7JHR4ETJYGG7SSOOWQ5JTKSQYV26X4AWZ6XHEBHLJYE3774CA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:26.173243839 (547h59m26.044326874s) [noise=true , address=145.79.227.44:4209 , piece_id=PKQUKW3FOPJL7BRNFHNLAJ4DSKGMGA73ROILSWEFCT7MR7H5GNNQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:26.289483660 (547h59m26.160566695s) [noise=true , address=77.90.31.10:4536 , piece_id=HKI6JPRV4D2CBIVEGHRXSCDC2OROZLX5SW3VC7W35L4DWUEITV2Q, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:26.448835962 (547h59m26.319919018s) [noise=true , address=145.79.227.40:4067 , piece_id=YOIOW6QPNOMK5ZD5EEWT2FLVGVZZMHNWANUQF4UE3R4FHSOFWSWQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:26.571228809 (547h59m26.442311844s) [noise=true , address=145.79.227.33:4882 , piece_id=VJ6XYH77LJSKKBVCCVHHTGFV3WTSCYD4XUKUMSYOWZEGZ2QY4H3A, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:26.571248396 (547h59m26.442331441s) [seg_pos={0 0}, upload=480503]: Segment reached optimal threshold of 14 pieces.
Dec 28 11:28:26.571429892 (547h59m26.442512927s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NEV2ndHw4fohVUNsfrzwojd8S11TmMST6Sz59GVnXS8vDGhWk): protocol: expected piece hash; context canceled; EOF
Dec 28 11:28:26.571436033 (547h59m26.442519078s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold
Dec 28 11:28:26.571564158 (547h59m26.442647203s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection
Dec 28 11:28:26.571580588 (547h59m26.442663623s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold
Dec 28 11:28:26.571712249 (547h59m26.442795284s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Nw78dd9WPnmCwv5bFQR3VpV39776Pye2QVGSNMTs3zq2BtaGJ): protocol: expected piece hash; context canceled; EOF
Dec 28 11:28:26.571720666 (547h59m26.442803701s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold
Dec 28 11:28:26.571767825 (547h59m26.442850871s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv): protocol: expected piece hash; context canceled; EOF
Dec 28 11:28:26.571774688 (547h59m26.442857723s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold
Dec 28 11:28:26.571792493 (547h59m26.442875538s) [seg_pos={0 0}, upload=480503]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 11:28:26.571882343 (547h59m26.442965378s) [upload=480503]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 11:28:26.571887683 (547h59m26.442970728s) [upload=480503]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 11:28:26.858811294 (547h59m26.729894329s) [upload=480503]: Done uploading segments...
Dec 28 11:28:26.858832874 (547h59m26.72991592s) [upload=480503]: Upload finished. err: <nil>
Duration: 31.220911764s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
16 out of 60 is failed (threshold=3) [X....X.....XX...XXX.......X..XX....X...X.........X..XX.X....] duration:
5.777909568s trace:
https://jaeger.dev.storj.io/trace/76cfc3bec30f9bc9
FAIL (2025-12-28 12:15:23 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: DVDBH2I22B4XMYTEFZIZC5ENHCCNY2UYBKGPNXWOR5WU7ZSUEKYQ) context deadline exceeded; piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: N67F6BUDWMY7HQS2U33BXW4VZJ34MO73FY6WQWT5K2HBMPQCYH6Q) context deadline exceeded
======LOGS======
Dec 28 12:10:55.660430003 (548h41m55.531513048s) [upload=481121]: Starting upload
Dec 28 12:10:55.660499014 (548h41m55.531582049s) [upload=481121]: Done starting upload
Dec 28 12:10:55.660735474 (548h41m55.53181852s) [upload=481121]: Uploading segments...
Dec 28 12:10:55.660870462 (548h41m55.531953507s) [upload=481121]: Got next segment. Inline: false
Dec 28 12:10:55.660917210 (548h41m55.532000255s) [upload=481121]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 12:10:55.660924685 (548h41m55.53200773s) [upload=481121]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 12:10:55.909444035 (548h41m55.78052708s) [seg_pos={0 0}, upload=481121]: Begin upload segment...
Dec 28 12:10:55.909501735 (548h41m55.78058478s) [seg_pos={0 0}, upload=481121]: Done begin upload segment.
Dec 28 12:10:55.909539026 (548h41m55.780622071s) [noise=true , address=145.79.227.14:4837 , piece_id=UPBI5KCD7HTHZR6H2QMPS5LUGBCI7RXGBGNBJHG5I7RABK3XJ3FA, seg_pos={0 0}, upload=481121]: Uploading piece...
Dec 28 12:10:55.909568382 (548h41m55.780651427s) [noise=true , address=145.79.227.15:4873 , piece_id=3B6SG27RUZGJRYCT433Y5KCUEDZMYJY4LAE4IA7RTVPNQBMRWWFQ, seg_pos={0 0}, upload=481121]: Uploading piece...
Dec 28 12:10:55.910395015 (548h41m55.78147805s) [noise=true , address=145.79.227.42:4122 , piece_id=33XBPAO6JIVHHRTTHY6HFD5YXVTKROB5HRRDUV4THYKTPU2FZU6A, seg_pos={0 0}, upload=481121]: Uploading piece...
Dec 28 12:10:55.910684147 (548h41m55.781767182s) [noise=true , address=145.79.227.4:4186 , piece_id=PUB3YGQNDHW4AVBUIAPICCTOLRDLGDFA6YQTUYN6PQKX3TRZGUMA, seg_pos={0 0}, upload=481121]: Uploading piece...
Dec 28 12:10:55.910938651 (548h41m55.782021686s) [noise=true , address=145.79.227.45:4231 , piece_id=3D2JZTRHCEER64WWU3FBIJNCYFZOLDLQQYDD5DLOG563X4WML6KA, seg_pos={0 0}, upload=481121]: Uploading piece...
Dec 28 12:10:55.911205319 (548h41m55.782288354s) [noise=true , address=77.90.31.11:4554 , piece_id=N67F6BUDWMY7HQS2U33BXW4VZJ34MO73FY6WQWT5K2HBMPQCYH6Q, seg_pos={0 0}, upload=481121]: Uploading piece...
Dec 28 12:10:55.911457249 (548h41m55.782540284s) [noise=true , address=145.79.227.48:4301 , piece_id=3C2LRAO72K65765CNEPC6IQATH7XKRQ5C2HS4BG6VQYGM5EICS4Q, seg_pos={0 0}, upload=481121]: Uploading piece...
Dec 28 12:10:55.911748282 (548h41m55.782831317s) [noise=true , address=145.79.227.47:4265 , piece_id=AS47QV7TCZVOKCWGQH3E6O5PNEZ4YNC4VBGL7O4QRKXA6ZIGQWMA, seg_pos={0 0}, upload=481121]: Uploading piece...
Dec 28 12:10:55.912092448 (548h41m55.783175493s) [noise=true , address=77.90.31.13:4626 , piece_id=HFJFVBZJD6HYEAX2L6EF7UZVH3UEQ4GQBCDV2ZEYTU7RATF5UXGQ, seg_pos={0 0}, upload=481121]: Uploading piece...
Dec 28 12:10:55.912445791 (548h41m55.783528836s) [noise=true , address=45.91.116.14:4189 , piece_id=KH6OZOJORQK3QJFBAP3NJSBQTJE2MDENC74ZB25YBP45BEPQ7ICA, seg_pos={0 0}, upload=481121]: Uploading piece...
Dec 28 12:10:55.913104785 (548h41m55.784187831s) [noise=true , address=145.79.227.16:4910 , piece_id=GPVKLL4ZYNO6KLZ35R7X5HAMGCSVIIE3H5SS2UEOEJ7CNKDRFQHQ, seg_pos={0 0}, upload=481121]: Uploading piece...
Dec 28 12:10:55.913554612 (548h41m55.784637657s) [noise=true , address=77.90.31.16:4735 , piece_id=COCPUUARNUQ47ND6EIHR332XZJP7JCSVUNFV63QBIQJIZEIMQC2A, seg_pos={0 0}, upload=481121]: Uploading piece...
Dec 28 12:10:55.914012474 (548h41m55.785095519s) [noise=true , address=145.79.227.13:4803 , piece_id=53MLFZJ6QX2FQ6SG733SDERFTIGX6GNORWL5SAXDLZ2QM4TBKINA, seg_pos={0 0}, upload=481121]: Uploading piece...
Dec 28 12:10:55.914463302 (548h41m55.785546478s) [noise=true , address=145.79.227.11:4730 , piece_id=LIFLAF57ZA52IARNDGUVIYHQ6DFJC7VI3JNPY6T37A5YDKOUSRQQ, seg_pos={0 0}, upload=481121]: Uploading piece...
Dec 28 12:10:55.914861211 (548h41m55.785944246s) [noise=true , address=45.91.116.6:4974 , piece_id=DVDBH2I22B4XMYTEFZIZC5ENHCCNY2UYBKGPNXWOR5WU7ZSUEKYQ, seg_pos={0 0}, upload=481121]: Uploading piece...
Dec 28 12:10:55.915264708 (548h41m55.786347813s) [noise=true , address=145.79.227.39:4014 , piece_id=NFTS52EVDWICVF7OZ6DEXPTOAALSQPZLLST2YWEGYQCNIMUI4CPA, seg_pos={0 0}, upload=481121]: Uploading piece...
Dec 28 12:10:57.629256297 (548h41m57.500339342s) [upload=481121]: Next returned nil segment
Dec 28 12:10:57.629273770 (548h41m57.500356805s) [upload=481121]: Waiting for error group managing segments...
Dec 28 12:10:57.917456196 (548h41m57.788539231s) [noise=true , address=145.79.227.39:4014 , piece_id=NFTS52EVDWICVF7OZ6DEXPTOAALSQPZLLST2YWEGYQCNIMUI4CPA, seg_pos={0 0}, upload=481121]: Done uploading piece. err: <nil>
Dec 28 12:10:57.917907185 (548h41m57.78899022s) [noise=true , address=145.79.227.4:4186 , piece_id=PUB3YGQNDHW4AVBUIAPICCTOLRDLGDFA6YQTUYN6PQKX3TRZGUMA, seg_pos={0 0}, upload=481121]: Done uploading piece. err: <nil>
Dec 28 12:10:57.922473289 (548h41m57.793556334s) [noise=true , address=145.79.227.13:4803 , piece_id=53MLFZJ6QX2FQ6SG733SDERFTIGX6GNORWL5SAXDLZ2QM4TBKINA, seg_pos={0 0}, upload=481121]: Done uploading piece. err: <nil>
Dec 28 12:10:57.925552562 (548h41m57.796635607s) [noise=true , address=145.79.227.16:4910 , piece_id=GPVKLL4ZYNO6KLZ35R7X5HAMGCSVIIE3H5SS2UEOEJ7CNKDRFQHQ, seg_pos={0 0}, upload=481121]: Done uploading piece. err: <nil>
Dec 28 12:10:57.932490544 (548h41m57.803573579s) [noise=true , address=145.79.227.14:4837 , piece_id=UPBI5KCD7HTHZR6H2QMPS5LUGBCI7RXGBGNBJHG5I7RABK3XJ3FA, seg_pos={0 0}, upload=481121]: Done uploading piece. err: <nil>
Dec 28 12:10:57.932829238 (548h41m57.803912273s) [noise=true , address=145.79.227.42:4122 , piece_id=33XBPAO6JIVHHRTTHY6HFD5YXVTKROB5HRRDUV4THYKTPU2FZU6A, seg_pos={0 0}, upload=481121]: Done uploading piece. err: <nil>
Dec 28 12:10:57.937710152 (548h41m57.808793197s) [noise=true , address=145.79.227.11:4730 , piece_id=LIFLAF57ZA52IARNDGUVIYHQ6DFJC7VI3JNPY6T37A5YDKOUSRQQ, seg_pos={0 0}, upload=481121]: Done uploading piece. err: <nil>
Dec 28 12:10:57.940392400 (548h41m57.811475445s) [noise=true , address=145.79.227.45:4231 , piece_id=3D2JZTRHCEER64WWU3FBIJNCYFZOLDLQQYDD5DLOG563X4WML6KA, seg_pos={0 0}, upload=481121]: Done uploading piece. err: <nil>
Dec 28 12:10:57.945343066 (548h41m57.816426111s) [noise=true , address=145.79.227.48:4301 , piece_id=3C2LRAO72K65765CNEPC6IQATH7XKRQ5C2HS4BG6VQYGM5EICS4Q, seg_pos={0 0}, upload=481121]: Done uploading piece. err: <nil>
Dec 28 12:10:57.945399194 (548h41m57.816482229s) [noise=true , address=145.79.227.15:4873 , piece_id=3B6SG27RUZGJRYCT433Y5KCUEDZMYJY4LAE4IA7RTVPNQBMRWWFQ, seg_pos={0 0}, upload=481121]: Done uploading piece. err: <nil>
Dec 28 12:10:57.951082956 (548h41m57.822166001s) [noise=true , address=145.79.227.47:4265 , piece_id=AS47QV7TCZVOKCWGQH3E6O5PNEZ4YNC4VBGL7O4QRKXA6ZIGQWMA, seg_pos={0 0}, upload=481121]: Done uploading piece. err: <nil>
Dec 28 12:10:58.132138455 (548h41m58.00322151s) [noise=true , address=45.91.116.6:4974 , piece_id=DVDBH2I22B4XMYTEFZIZC5ENHCCNY2UYBKGPNXWOR5WU7ZSUEKYQ, seg_pos={0 0}, upload=481121]: Done uploading piece. err: <nil>
Dec 28 12:10:58.238648407 (548h41m58.109731442s) [noise=true , address=77.90.31.11:4554 , piece_id=N67F6BUDWMY7HQS2U33BXW4VZJ34MO73FY6WQWT5K2HBMPQCYH6Q, seg_pos={0 0}, upload=481121]: Done uploading piece. err: <nil>
Dec 28 12:10:58.238678594 (548h41m58.109761639s) [seg_pos={0 0}, upload=481121]: Segment reached optimal threshold of 13 pieces.
Dec 28 12:10:58.238881581 (548h41m58.109964626s) [noise=true , address=77.90.31.16:4735 , piece_id=COCPUUARNUQ47ND6EIHR332XZJP7JCSVUNFV63QBIQJIZEIMQC2A, seg_pos={0 0}, upload=481121]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF
Dec 28 12:10:58.238889496 (548h41m58.109972531s) [noise=true , address=77.90.31.16:4735 , piece_id=COCPUUARNUQ47ND6EIHR332XZJP7JCSVUNFV63QBIQJIZEIMQC2A, seg_pos={0 0}, upload=481121]: Long tail cancellation due to optimal threshold
Dec 28 12:10:58.238995468 (548h41m58.110078503s) [noise=true , address=45.91.116.14:4189 , piece_id=KH6OZOJORQK3QJFBAP3NJSBQTJE2MDENC74ZB25YBP45BEPQ7ICA, seg_pos={0 0}, upload=481121]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: write tcp 188.166.225.170:56090->45.91.116.14:4189: use of closed network connection; write tcp 188.166.225.170:56090->45.91.116.14:4189: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56090->45.91.116.14:4189: use of closed network connection
Dec 28 12:10:58.239016368 (548h41m58.110099403s) [noise=true , address=45.91.116.14:4189 , piece_id=KH6OZOJORQK3QJFBAP3NJSBQTJE2MDENC74ZB25YBP45BEPQ7ICA, seg_pos={0 0}, upload=481121]: Long tail cancellation due to optimal threshold
Dec 28 12:10:58.239125055 (548h41m58.11020809s) [noise=true , address=77.90.31.13:4626 , piece_id=HFJFVBZJD6HYEAX2L6EF7UZVH3UEQ4GQBCDV2ZEYTU7RATF5UXGQ, seg_pos={0 0}, upload=481121]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF
Dec 28 12:10:58.239131747 (548h41m58.110214792s) [noise=true , address=77.90.31.13:4626 , piece_id=HFJFVBZJD6HYEAX2L6EF7UZVH3UEQ4GQBCDV2ZEYTU7RATF5UXGQ, seg_pos={0 0}, upload=481121]: Long tail cancellation due to optimal threshold
Dec 28 12:10:58.239152227 (548h41m58.110235262s) [seg_pos={0 0}, upload=481121]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 12:10:58.239217340 (548h41m58.110300375s) [upload=481121]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 12:10:58.239223222 (548h41m58.110306267s) [upload=481121]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 12:10:58.538804292 (548h41m58.409887327s) [upload=481121]: Done uploading segments...
Dec 28 12:10:58.538831865 (548h41m58.4099149s) [upload=481121]: Upload finished. err: <nil>
Duration: 5.777938412s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [...............................X.X..........................] duration:
14.725339586s trace:
https://jaeger.dev.storj.io/trace/72d0191c7bb67437
FAIL (2025-12-28 12:15:11 UTC)
orbiter warning: context deadline exceeded
Duration: 14.72536742s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
33 out of 60 is failed (threshold=6) [..X.X.X..X.XXX..X....XX.XX.XXXXXXXXXX..XXXX...XX.XXX.XX.....] duration:
5.001222106s trace:
https://jaeger.dev.storj.io/trace/4a7cc66cea50a588
FAIL (2025-12-28 12:15:09 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001269907s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [.X..X.......................................................] duration:
34.892238467s trace:
https://jaeger.dev.storj.io/trace/307d4901d0d5949
FAIL (2025-12-28 12:15:01 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
ise=true , address=77.90.31.16:4743 , piece_id=NRCRNEOEKXPSKREUZKGGLQHLWG5JTKWCQCRUUHKUHB4HOISVHCFQ, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.355028753 (547h59m23.226111788s) [noise=true , address=145.79.227.30:4796 , piece_id=MUD7JHR4ETJYGG7SSOOWQ5JTKSQYV26X4AWZ6XHEBHLJYE3774CA, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.355039954 (547h59m23.226122989s) [noise=true , address=45.91.116.11:4122 , piece_id=MROLZP4Y2PKVJSUUXCSDQQGVGF7AJADXWEJFFPH6SU2DWTZR443A, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.355282126 (547h59m23.226365171s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.355390983 (547h59m23.226474018s) [noise=true , address=193.149.164.143:4700 , piece_id=3FBVJX372QBFVBDXCF5VTCIT2JUYHNXFCZPJ2FN7E34HA4S4WETQ, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.355527633 (547h59m23.226610668s) [noise=true , address=77.90.31.15:4727 , piece_id=KZ74K3R4G2BD73RTJFOKFGQ57SV456NQC4W3NTPT52KBQAU3SWNA, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.355648985 (547h59m23.22673202s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.355877279 (547h59m23.226960324s) [noise=true , address=145.79.227.12:4788 , piece_id=KFB2WTRQOSZ75WMHRNCRTPS2FSMIBR27TQ55V62J2PGWLFECB4RA, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.356182711 (547h59m23.227265756s) [noise=true , address=145.79.227.49:4351 , piece_id=FYLYJ6GRQIMVTNGLLY62Q35M5F4IEQW2EPKVMSDOVJIJJPJJXF3Q, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.356536194 (547h59m23.227619229s) [noise=true , address=145.79.227.19:4092 , piece_id=7UPEHPU5NSBVOPSDNKUAQQZXBHHBS36PKZOR5FBDKLWP4VA4ZSDA, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.356869479 (547h59m23.227952514s) [noise=true , address=145.79.227.33:4882 , piece_id=VJ6XYH77LJSKKBVCCVHHTGFV3WTSCYD4XUKUMSYOWZEGZ2QY4H3A, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.357157427 (547h59m23.228240462s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.357426930 (547h59m23.228509975s) [noise=true , address=145.79.227.40:4067 , piece_id=YOIOW6QPNOMK5ZD5EEWT2FLVGVZZMHNWANUQF4UE3R4FHSOFWSWQ, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.357751889 (547h59m23.228834924s) [noise=true , address=77.90.31.10:4536 , piece_id=HKI6JPRV4D2CBIVEGHRXSCDC2OROZLX5SW3VC7W35L4DWUEITV2Q, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.358953407 (547h59m23.230036442s) [noise=true , address=77.90.31.17:4790 , piece_id=AS6RYIPXJKC6CFSPVX7MLENRTQHI7RZFLTSPHFMIYG5OS7BKZHRA, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.359286491 (547h59m23.230369536s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:24.695915344 (547h59m24.566998389s) [upload=480503]: Next returned nil segment
Dec 28 11:28:24.695941574 (547h59m24.567024619s) [upload=480503]: Waiting for error group managing segments...
Dec 28 11:28:25.041244477 (547h59m24.912327512s) [noise=true , address=193.149.164.143:4700 , piece_id=3FBVJX372QBFVBDXCF5VTCIT2JUYHNXFCZPJ2FN7E34HA4S4WETQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:25.237036778 (547h59m25.108119823s) [noise=true , address=145.79.227.49:4351 , piece_id=FYLYJ6GRQIMVTNGLLY62Q35M5F4IEQW2EPKVMSDOVJIJJPJJXF3Q, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:25.249693558 (547h59m25.120776593s) [noise=true , address=145.79.227.12:4788 , piece_id=KFB2WTRQOSZ75WMHRNCRTPS2FSMIBR27TQ55V62J2PGWLFECB4RA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:25.256819657 (547h59m25.127902692s) [noise=true , address=145.79.227.19:4092 , piece_id=7UPEHPU5NSBVOPSDNKUAQQZXBHHBS36PKZOR5FBDKLWP4VA4ZSDA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:25.511089018 (547h59m25.382172053s) [noise=true , address=45.91.116.13:4170 , piece_id=3KJJFTI5M7MBDSZNTBWSMSJUU7B325XVVLXEH3JKX5N7N4WNMXAA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:25.523026599 (547h59m25.394109634s) [noise=true , address=77.90.31.17:4790 , piece_id=AS6RYIPXJKC6CFSPVX7MLENRTQHI7RZFLTSPHFMIYG5OS7BKZHRA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:25.524217066 (547h59m25.395300101s) [noise=true , address=45.91.116.11:4122 , piece_id=MROLZP4Y2PKVJSUUXCSDQQGVGF7AJADXWEJFFPH6SU2DWTZR443A, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:25.665135729 (547h59m25.536218774s) [noise=true , address=77.90.31.15:4727 , piece_id=KZ74K3R4G2BD73RTJFOKFGQ57SV456NQC4W3NTPT52KBQAU3SWNA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:25.917975247 (547h59m25.789058292s) [noise=true , address=77.90.31.16:4743 , piece_id=NRCRNEOEKXPSKREUZKGGLQHLWG5JTKWCQCRUUHKUHB4HOISVHCFQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:26.147871236 (547h59m26.018954271s) [noise=true , address=145.79.227.30:4796 , piece_id=MUD7JHR4ETJYGG7SSOOWQ5JTKSQYV26X4AWZ6XHEBHLJYE3774CA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:26.173243839 (547h59m26.044326874s) [noise=true , address=145.79.227.44:4209 , piece_id=PKQUKW3FOPJL7BRNFHNLAJ4DSKGMGA73ROILSWEFCT7MR7H5GNNQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:26.289483660 (547h59m26.160566695s) [noise=true , address=77.90.31.10:4536 , piece_id=HKI6JPRV4D2CBIVEGHRXSCDC2OROZLX5SW3VC7W35L4DWUEITV2Q, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:26.448835962 (547h59m26.319919018s) [noise=true , address=145.79.227.40:4067 , piece_id=YOIOW6QPNOMK5ZD5EEWT2FLVGVZZMHNWANUQF4UE3R4FHSOFWSWQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:26.571228809 (547h59m26.442311844s) [noise=true , address=145.79.227.33:4882 , piece_id=VJ6XYH77LJSKKBVCCVHHTGFV3WTSCYD4XUKUMSYOWZEGZ2QY4H3A, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:26.571248396 (547h59m26.442331441s) [seg_pos={0 0}, upload=480503]: Segment reached optimal threshold of 14 pieces.
Dec 28 11:28:26.571429892 (547h59m26.442512927s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NEV2ndHw4fohVUNsfrzwojd8S11TmMST6Sz59GVnXS8vDGhWk): protocol: expected piece hash; context canceled; EOF
Dec 28 11:28:26.571436033 (547h59m26.442519078s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold
Dec 28 11:28:26.571564158 (547h59m26.442647203s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection
Dec 28 11:28:26.571580588 (547h59m26.442663623s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold
Dec 28 11:28:26.571712249 (547h59m26.442795284s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Nw78dd9WPnmCwv5bFQR3VpV39776Pye2QVGSNMTs3zq2BtaGJ): protocol: expected piece hash; context canceled; EOF
Dec 28 11:28:26.571720666 (547h59m26.442803701s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold
Dec 28 11:28:26.571767825 (547h59m26.442850871s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv): protocol: expected piece hash; context canceled; EOF
Dec 28 11:28:26.571774688 (547h59m26.442857723s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold
Dec 28 11:28:26.571792493 (547h59m26.442875538s) [seg_pos={0 0}, upload=480503]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 11:28:26.571882343 (547h59m26.442965378s) [upload=480503]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 11:28:26.571887683 (547h59m26.442970728s) [upload=480503]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 11:28:26.858811294 (547h59m26.729894329s) [upload=480503]: Done uploading segments...
Dec 28 11:28:26.858832874 (547h59m26.72991592s) [upload=480503]: Upload finished. err: <nil>
Duration: 34.892257384s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [................................X.X.........................] duration:
16.058502718s trace:
https://jaeger.dev.storj.io/trace/72d0191c7bb67437
FAIL (2025-12-28 12:14:45 UTC)
orbiter warning: context deadline exceeded
Duration: 16.058530951s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
33 out of 60 is failed (threshold=6) [...X.X.X..X.XXX..X....XX.XX.XXXXXXXXXX..XXXX...XX.XXX.XX....] duration:
4.876299178s trace:
https://jaeger.dev.storj.io/trace/4a7cc66cea50a588
FAIL (2025-12-28 12:14:45 UTC)
orbiter warning: context deadline exceeded
Duration: 4.876332342s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
17 out of 60 is failed (threshold=3) [XX....X.....XX...XXX.......X..XX....X...X.........X..XX.X...] duration:
5.697089849s trace:
https://jaeger.dev.storj.io/trace/76cfc3bec30f9bc9
FAIL (2025-12-28 12:14:25 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: DVDBH2I22B4XMYTEFZIZC5ENHCCNY2UYBKGPNXWOR5WU7ZSUEKYQ) context deadline exceeded; piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: N67F6BUDWMY7HQS2U33BXW4VZJ34MO73FY6WQWT5K2HBMPQCYH6Q) context deadline exceeded
======LOGS======
Dec 28 12:10:55.660430003 (548h41m55.531513048s) [upload=481121]: Starting upload
Dec 28 12:10:55.660499014 (548h41m55.531582049s) [upload=481121]: Done starting upload
Dec 28 12:10:55.660735474 (548h41m55.53181852s) [upload=481121]: Uploading segments...
Dec 28 12:10:55.660870462 (548h41m55.531953507s) [upload=481121]: Got next segment. Inline: false
Dec 28 12:10:55.660917210 (548h41m55.532000255s) [upload=481121]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 12:10:55.660924685 (548h41m55.53200773s) [upload=481121]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 12:10:55.909444035 (548h41m55.78052708s) [seg_pos={0 0}, upload=481121]: Begin upload segment...
Dec 28 12:10:55.909501735 (548h41m55.78058478s) [seg_pos={0 0}, upload=481121]: Done begin upload segment.
Dec 28 12:10:55.909539026 (548h41m55.780622071s) [noise=true , address=145.79.227.14:4837 , piece_id=UPBI5KCD7HTHZR6H2QMPS5LUGBCI7RXGBGNBJHG5I7RABK3XJ3FA, seg_pos={0 0}, upload=481121]: Uploading piece...
Dec 28 12:10:55.909568382 (548h41m55.780651427s) [noise=true , address=145.79.227.15:4873 , piece_id=3B6SG27RUZGJRYCT433Y5KCUEDZMYJY4LAE4IA7RTVPNQBMRWWFQ, seg_pos={0 0}, upload=481121]: Uploading piece...
Dec 28 12:10:55.910395015 (548h41m55.78147805s) [noise=true , address=145.79.227.42:4122 , piece_id=33XBPAO6JIVHHRTTHY6HFD5YXVTKROB5HRRDUV4THYKTPU2FZU6A, seg_pos={0 0}, upload=481121]: Uploading piece...
Dec 28 12:10:55.910684147 (548h41m55.781767182s) [noise=true , address=145.79.227.4:4186 , piece_id=PUB3YGQNDHW4AVBUIAPICCTOLRDLGDFA6YQTUYN6PQKX3TRZGUMA, seg_pos={0 0}, upload=481121]: Uploading piece...
Dec 28 12:10:55.910938651 (548h41m55.782021686s) [noise=true , address=145.79.227.45:4231 , piece_id=3D2JZTRHCEER64WWU3FBIJNCYFZOLDLQQYDD5DLOG563X4WML6KA, seg_pos={0 0}, upload=481121]: Uploading piece...
Dec 28 12:10:55.911205319 (548h41m55.782288354s) [noise=true , address=77.90.31.11:4554 , piece_id=N67F6BUDWMY7HQS2U33BXW4VZJ34MO73FY6WQWT5K2HBMPQCYH6Q, seg_pos={0 0}, upload=481121]: Uploading piece...
Dec 28 12:10:55.911457249 (548h41m55.782540284s) [noise=true , address=145.79.227.48:4301 , piece_id=3C2LRAO72K65765CNEPC6IQATH7XKRQ5C2HS4BG6VQYGM5EICS4Q, seg_pos={0 0}, upload=481121]: Uploading piece...
Dec 28 12:10:55.911748282 (548h41m55.782831317s) [noise=true , address=145.79.227.47:4265 , piece_id=AS47QV7TCZVOKCWGQH3E6O5PNEZ4YNC4VBGL7O4QRKXA6ZIGQWMA, seg_pos={0 0}, upload=481121]: Uploading piece...
Dec 28 12:10:55.912092448 (548h41m55.783175493s) [noise=true , address=77.90.31.13:4626 , piece_id=HFJFVBZJD6HYEAX2L6EF7UZVH3UEQ4GQBCDV2ZEYTU7RATF5UXGQ, seg_pos={0 0}, upload=481121]: Uploading piece...
Dec 28 12:10:55.912445791 (548h41m55.783528836s) [noise=true , address=45.91.116.14:4189 , piece_id=KH6OZOJORQK3QJFBAP3NJSBQTJE2MDENC74ZB25YBP45BEPQ7ICA, seg_pos={0 0}, upload=481121]: Uploading piece...
Dec 28 12:10:55.913104785 (548h41m55.784187831s) [noise=true , address=145.79.227.16:4910 , piece_id=GPVKLL4ZYNO6KLZ35R7X5HAMGCSVIIE3H5SS2UEOEJ7CNKDRFQHQ, seg_pos={0 0}, upload=481121]: Uploading piece...
Dec 28 12:10:55.913554612 (548h41m55.784637657s) [noise=true , address=77.90.31.16:4735 , piece_id=COCPUUARNUQ47ND6EIHR332XZJP7JCSVUNFV63QBIQJIZEIMQC2A, seg_pos={0 0}, upload=481121]: Uploading piece...
Dec 28 12:10:55.914012474 (548h41m55.785095519s) [noise=true , address=145.79.227.13:4803 , piece_id=53MLFZJ6QX2FQ6SG733SDERFTIGX6GNORWL5SAXDLZ2QM4TBKINA, seg_pos={0 0}, upload=481121]: Uploading piece...
Dec 28 12:10:55.914463302 (548h41m55.785546478s) [noise=true , address=145.79.227.11:4730 , piece_id=LIFLAF57ZA52IARNDGUVIYHQ6DFJC7VI3JNPY6T37A5YDKOUSRQQ, seg_pos={0 0}, upload=481121]: Uploading piece...
Dec 28 12:10:55.914861211 (548h41m55.785944246s) [noise=true , address=45.91.116.6:4974 , piece_id=DVDBH2I22B4XMYTEFZIZC5ENHCCNY2UYBKGPNXWOR5WU7ZSUEKYQ, seg_pos={0 0}, upload=481121]: Uploading piece...
Dec 28 12:10:55.915264708 (548h41m55.786347813s) [noise=true , address=145.79.227.39:4014 , piece_id=NFTS52EVDWICVF7OZ6DEXPTOAALSQPZLLST2YWEGYQCNIMUI4CPA, seg_pos={0 0}, upload=481121]: Uploading piece...
Dec 28 12:10:57.629256297 (548h41m57.500339342s) [upload=481121]: Next returned nil segment
Dec 28 12:10:57.629273770 (548h41m57.500356805s) [upload=481121]: Waiting for error group managing segments...
Dec 28 12:10:57.917456196 (548h41m57.788539231s) [noise=true , address=145.79.227.39:4014 , piece_id=NFTS52EVDWICVF7OZ6DEXPTOAALSQPZLLST2YWEGYQCNIMUI4CPA, seg_pos={0 0}, upload=481121]: Done uploading piece. err: <nil>
Dec 28 12:10:57.917907185 (548h41m57.78899022s) [noise=true , address=145.79.227.4:4186 , piece_id=PUB3YGQNDHW4AVBUIAPICCTOLRDLGDFA6YQTUYN6PQKX3TRZGUMA, seg_pos={0 0}, upload=481121]: Done uploading piece. err: <nil>
Dec 28 12:10:57.922473289 (548h41m57.793556334s) [noise=true , address=145.79.227.13:4803 , piece_id=53MLFZJ6QX2FQ6SG733SDERFTIGX6GNORWL5SAXDLZ2QM4TBKINA, seg_pos={0 0}, upload=481121]: Done uploading piece. err: <nil>
Dec 28 12:10:57.925552562 (548h41m57.796635607s) [noise=true , address=145.79.227.16:4910 , piece_id=GPVKLL4ZYNO6KLZ35R7X5HAMGCSVIIE3H5SS2UEOEJ7CNKDRFQHQ, seg_pos={0 0}, upload=481121]: Done uploading piece. err: <nil>
Dec 28 12:10:57.932490544 (548h41m57.803573579s) [noise=true , address=145.79.227.14:4837 , piece_id=UPBI5KCD7HTHZR6H2QMPS5LUGBCI7RXGBGNBJHG5I7RABK3XJ3FA, seg_pos={0 0}, upload=481121]: Done uploading piece. err: <nil>
Dec 28 12:10:57.932829238 (548h41m57.803912273s) [noise=true , address=145.79.227.42:4122 , piece_id=33XBPAO6JIVHHRTTHY6HFD5YXVTKROB5HRRDUV4THYKTPU2FZU6A, seg_pos={0 0}, upload=481121]: Done uploading piece. err: <nil>
Dec 28 12:10:57.937710152 (548h41m57.808793197s) [noise=true , address=145.79.227.11:4730 , piece_id=LIFLAF57ZA52IARNDGUVIYHQ6DFJC7VI3JNPY6T37A5YDKOUSRQQ, seg_pos={0 0}, upload=481121]: Done uploading piece. err: <nil>
Dec 28 12:10:57.940392400 (548h41m57.811475445s) [noise=true , address=145.79.227.45:4231 , piece_id=3D2JZTRHCEER64WWU3FBIJNCYFZOLDLQQYDD5DLOG563X4WML6KA, seg_pos={0 0}, upload=481121]: Done uploading piece. err: <nil>
Dec 28 12:10:57.945343066 (548h41m57.816426111s) [noise=true , address=145.79.227.48:4301 , piece_id=3C2LRAO72K65765CNEPC6IQATH7XKRQ5C2HS4BG6VQYGM5EICS4Q, seg_pos={0 0}, upload=481121]: Done uploading piece. err: <nil>
Dec 28 12:10:57.945399194 (548h41m57.816482229s) [noise=true , address=145.79.227.15:4873 , piece_id=3B6SG27RUZGJRYCT433Y5KCUEDZMYJY4LAE4IA7RTVPNQBMRWWFQ, seg_pos={0 0}, upload=481121]: Done uploading piece. err: <nil>
Dec 28 12:10:57.951082956 (548h41m57.822166001s) [noise=true , address=145.79.227.47:4265 , piece_id=AS47QV7TCZVOKCWGQH3E6O5PNEZ4YNC4VBGL7O4QRKXA6ZIGQWMA, seg_pos={0 0}, upload=481121]: Done uploading piece. err: <nil>
Dec 28 12:10:58.132138455 (548h41m58.00322151s) [noise=true , address=45.91.116.6:4974 , piece_id=DVDBH2I22B4XMYTEFZIZC5ENHCCNY2UYBKGPNXWOR5WU7ZSUEKYQ, seg_pos={0 0}, upload=481121]: Done uploading piece. err: <nil>
Dec 28 12:10:58.238648407 (548h41m58.109731442s) [noise=true , address=77.90.31.11:4554 , piece_id=N67F6BUDWMY7HQS2U33BXW4VZJ34MO73FY6WQWT5K2HBMPQCYH6Q, seg_pos={0 0}, upload=481121]: Done uploading piece. err: <nil>
Dec 28 12:10:58.238678594 (548h41m58.109761639s) [seg_pos={0 0}, upload=481121]: Segment reached optimal threshold of 13 pieces.
Dec 28 12:10:58.238881581 (548h41m58.109964626s) [noise=true , address=77.90.31.16:4735 , piece_id=COCPUUARNUQ47ND6EIHR332XZJP7JCSVUNFV63QBIQJIZEIMQC2A, seg_pos={0 0}, upload=481121]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF
Dec 28 12:10:58.238889496 (548h41m58.109972531s) [noise=true , address=77.90.31.16:4735 , piece_id=COCPUUARNUQ47ND6EIHR332XZJP7JCSVUNFV63QBIQJIZEIMQC2A, seg_pos={0 0}, upload=481121]: Long tail cancellation due to optimal threshold
Dec 28 12:10:58.238995468 (548h41m58.110078503s) [noise=true , address=45.91.116.14:4189 , piece_id=KH6OZOJORQK3QJFBAP3NJSBQTJE2MDENC74ZB25YBP45BEPQ7ICA, seg_pos={0 0}, upload=481121]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: write tcp 188.166.225.170:56090->45.91.116.14:4189: use of closed network connection; write tcp 188.166.225.170:56090->45.91.116.14:4189: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56090->45.91.116.14:4189: use of closed network connection
Dec 28 12:10:58.239016368 (548h41m58.110099403s) [noise=true , address=45.91.116.14:4189 , piece_id=KH6OZOJORQK3QJFBAP3NJSBQTJE2MDENC74ZB25YBP45BEPQ7ICA, seg_pos={0 0}, upload=481121]: Long tail cancellation due to optimal threshold
Dec 28 12:10:58.239125055 (548h41m58.11020809s) [noise=true , address=77.90.31.13:4626 , piece_id=HFJFVBZJD6HYEAX2L6EF7UZVH3UEQ4GQBCDV2ZEYTU7RATF5UXGQ, seg_pos={0 0}, upload=481121]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF
Dec 28 12:10:58.239131747 (548h41m58.110214792s) [noise=true , address=77.90.31.13:4626 , piece_id=HFJFVBZJD6HYEAX2L6EF7UZVH3UEQ4GQBCDV2ZEYTU7RATF5UXGQ, seg_pos={0 0}, upload=481121]: Long tail cancellation due to optimal threshold
Dec 28 12:10:58.239152227 (548h41m58.110235262s) [seg_pos={0 0}, upload=481121]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 12:10:58.239217340 (548h41m58.110300375s) [upload=481121]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 12:10:58.239223222 (548h41m58.110306267s) [upload=481121]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 12:10:58.538804292 (548h41m58.409887327s) [upload=481121]: Done uploading segments...
Dec 28 12:10:58.538831865 (548h41m58.4099149s) [upload=481121]: Upload finished. err: <nil>
Duration: 5.697138432s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [..X..X......................................................] duration:
22.977187077s trace:
https://jaeger.dev.storj.io/trace/307d4901d0d5949
FAIL (2025-12-28 12:14:05 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
ise=true , address=77.90.31.16:4743 , piece_id=NRCRNEOEKXPSKREUZKGGLQHLWG5JTKWCQCRUUHKUHB4HOISVHCFQ, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.355028753 (547h59m23.226111788s) [noise=true , address=145.79.227.30:4796 , piece_id=MUD7JHR4ETJYGG7SSOOWQ5JTKSQYV26X4AWZ6XHEBHLJYE3774CA, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.355039954 (547h59m23.226122989s) [noise=true , address=45.91.116.11:4122 , piece_id=MROLZP4Y2PKVJSUUXCSDQQGVGF7AJADXWEJFFPH6SU2DWTZR443A, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.355282126 (547h59m23.226365171s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.355390983 (547h59m23.226474018s) [noise=true , address=193.149.164.143:4700 , piece_id=3FBVJX372QBFVBDXCF5VTCIT2JUYHNXFCZPJ2FN7E34HA4S4WETQ, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.355527633 (547h59m23.226610668s) [noise=true , address=77.90.31.15:4727 , piece_id=KZ74K3R4G2BD73RTJFOKFGQ57SV456NQC4W3NTPT52KBQAU3SWNA, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.355648985 (547h59m23.22673202s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.355877279 (547h59m23.226960324s) [noise=true , address=145.79.227.12:4788 , piece_id=KFB2WTRQOSZ75WMHRNCRTPS2FSMIBR27TQ55V62J2PGWLFECB4RA, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.356182711 (547h59m23.227265756s) [noise=true , address=145.79.227.49:4351 , piece_id=FYLYJ6GRQIMVTNGLLY62Q35M5F4IEQW2EPKVMSDOVJIJJPJJXF3Q, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.356536194 (547h59m23.227619229s) [noise=true , address=145.79.227.19:4092 , piece_id=7UPEHPU5NSBVOPSDNKUAQQZXBHHBS36PKZOR5FBDKLWP4VA4ZSDA, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.356869479 (547h59m23.227952514s) [noise=true , address=145.79.227.33:4882 , piece_id=VJ6XYH77LJSKKBVCCVHHTGFV3WTSCYD4XUKUMSYOWZEGZ2QY4H3A, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.357157427 (547h59m23.228240462s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.357426930 (547h59m23.228509975s) [noise=true , address=145.79.227.40:4067 , piece_id=YOIOW6QPNOMK5ZD5EEWT2FLVGVZZMHNWANUQF4UE3R4FHSOFWSWQ, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.357751889 (547h59m23.228834924s) [noise=true , address=77.90.31.10:4536 , piece_id=HKI6JPRV4D2CBIVEGHRXSCDC2OROZLX5SW3VC7W35L4DWUEITV2Q, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.358953407 (547h59m23.230036442s) [noise=true , address=77.90.31.17:4790 , piece_id=AS6RYIPXJKC6CFSPVX7MLENRTQHI7RZFLTSPHFMIYG5OS7BKZHRA, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.359286491 (547h59m23.230369536s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:24.695915344 (547h59m24.566998389s) [upload=480503]: Next returned nil segment
Dec 28 11:28:24.695941574 (547h59m24.567024619s) [upload=480503]: Waiting for error group managing segments...
Dec 28 11:28:25.041244477 (547h59m24.912327512s) [noise=true , address=193.149.164.143:4700 , piece_id=3FBVJX372QBFVBDXCF5VTCIT2JUYHNXFCZPJ2FN7E34HA4S4WETQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:25.237036778 (547h59m25.108119823s) [noise=true , address=145.79.227.49:4351 , piece_id=FYLYJ6GRQIMVTNGLLY62Q35M5F4IEQW2EPKVMSDOVJIJJPJJXF3Q, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:25.249693558 (547h59m25.120776593s) [noise=true , address=145.79.227.12:4788 , piece_id=KFB2WTRQOSZ75WMHRNCRTPS2FSMIBR27TQ55V62J2PGWLFECB4RA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:25.256819657 (547h59m25.127902692s) [noise=true , address=145.79.227.19:4092 , piece_id=7UPEHPU5NSBVOPSDNKUAQQZXBHHBS36PKZOR5FBDKLWP4VA4ZSDA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:25.511089018 (547h59m25.382172053s) [noise=true , address=45.91.116.13:4170 , piece_id=3KJJFTI5M7MBDSZNTBWSMSJUU7B325XVVLXEH3JKX5N7N4WNMXAA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:25.523026599 (547h59m25.394109634s) [noise=true , address=77.90.31.17:4790 , piece_id=AS6RYIPXJKC6CFSPVX7MLENRTQHI7RZFLTSPHFMIYG5OS7BKZHRA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:25.524217066 (547h59m25.395300101s) [noise=true , address=45.91.116.11:4122 , piece_id=MROLZP4Y2PKVJSUUXCSDQQGVGF7AJADXWEJFFPH6SU2DWTZR443A, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:25.665135729 (547h59m25.536218774s) [noise=true , address=77.90.31.15:4727 , piece_id=KZ74K3R4G2BD73RTJFOKFGQ57SV456NQC4W3NTPT52KBQAU3SWNA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:25.917975247 (547h59m25.789058292s) [noise=true , address=77.90.31.16:4743 , piece_id=NRCRNEOEKXPSKREUZKGGLQHLWG5JTKWCQCRUUHKUHB4HOISVHCFQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:26.147871236 (547h59m26.018954271s) [noise=true , address=145.79.227.30:4796 , piece_id=MUD7JHR4ETJYGG7SSOOWQ5JTKSQYV26X4AWZ6XHEBHLJYE3774CA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:26.173243839 (547h59m26.044326874s) [noise=true , address=145.79.227.44:4209 , piece_id=PKQUKW3FOPJL7BRNFHNLAJ4DSKGMGA73ROILSWEFCT7MR7H5GNNQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:26.289483660 (547h59m26.160566695s) [noise=true , address=77.90.31.10:4536 , piece_id=HKI6JPRV4D2CBIVEGHRXSCDC2OROZLX5SW3VC7W35L4DWUEITV2Q, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:26.448835962 (547h59m26.319919018s) [noise=true , address=145.79.227.40:4067 , piece_id=YOIOW6QPNOMK5ZD5EEWT2FLVGVZZMHNWANUQF4UE3R4FHSOFWSWQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:26.571228809 (547h59m26.442311844s) [noise=true , address=145.79.227.33:4882 , piece_id=VJ6XYH77LJSKKBVCCVHHTGFV3WTSCYD4XUKUMSYOWZEGZ2QY4H3A, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:26.571248396 (547h59m26.442331441s) [seg_pos={0 0}, upload=480503]: Segment reached optimal threshold of 14 pieces.
Dec 28 11:28:26.571429892 (547h59m26.442512927s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NEV2ndHw4fohVUNsfrzwojd8S11TmMST6Sz59GVnXS8vDGhWk): protocol: expected piece hash; context canceled; EOF
Dec 28 11:28:26.571436033 (547h59m26.442519078s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold
Dec 28 11:28:26.571564158 (547h59m26.442647203s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection
Dec 28 11:28:26.571580588 (547h59m26.442663623s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold
Dec 28 11:28:26.571712249 (547h59m26.442795284s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Nw78dd9WPnmCwv5bFQR3VpV39776Pye2QVGSNMTs3zq2BtaGJ): protocol: expected piece hash; context canceled; EOF
Dec 28 11:28:26.571720666 (547h59m26.442803701s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold
Dec 28 11:28:26.571767825 (547h59m26.442850871s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv): protocol: expected piece hash; context canceled; EOF
Dec 28 11:28:26.571774688 (547h59m26.442857723s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold
Dec 28 11:28:26.571792493 (547h59m26.442875538s) [seg_pos={0 0}, upload=480503]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 11:28:26.571882343 (547h59m26.442965378s) [upload=480503]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 11:28:26.571887683 (547h59m26.442970728s) [upload=480503]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 11:28:26.858811294 (547h59m26.729894329s) [upload=480503]: Done uploading segments...
Dec 28 11:28:26.858832874 (547h59m26.72991592s) [upload=480503]: Upload finished. err: <nil>
Duration: 22.977219458s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
33 out of 60 is failed (threshold=6) [....X.X.X..X.XXX..X....XX.XX.XXXXXXXXXX..XXXX...XX.XXX.XX...] duration:
3.412035365s trace:
https://jaeger.dev.storj.io/trace/4a7cc66cea50a588
FAIL (2025-12-28 12:13:50 UTC)
orbiter warning: context deadline exceeded
Duration: 3.412063638s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.................................X.X........................] duration:
7.983626931s trace:
https://jaeger.dev.storj.io/trace/72d0191c7bb67437
FAIL (2025-12-28 12:13:39 UTC)
orbiter warning: context deadline exceeded
Duration: 7.983651035s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
33 out of 60 is failed (threshold=6) [.....X.X.X..X.XXX..X....XX.XX.XXXXXXXXXX..XXXX...XX.XXX.XX..] duration:
4.707908025s trace:
https://jaeger.dev.storj.io/trace/4a7cc66cea50a588
FAIL (2025-12-28 12:13:33 UTC)
orbiter warning: context deadline exceeded
Duration: 4.70798394s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
17 out of 60 is failed (threshold=3) [.XX....X.....XX...XXX.......X..XX....X...X.........X..XX.X..] duration:
6.023823009s trace:
https://jaeger.dev.storj.io/trace/76cfc3bec30f9bc9
FAIL (2025-12-28 12:13:20 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: DVDBH2I22B4XMYTEFZIZC5ENHCCNY2UYBKGPNXWOR5WU7ZSUEKYQ) context deadline exceeded; piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: N67F6BUDWMY7HQS2U33BXW4VZJ34MO73FY6WQWT5K2HBMPQCYH6Q) context deadline exceeded
======LOGS======
Dec 28 12:10:55.660430003 (548h41m55.531513048s) [upload=481121]: Starting upload
Dec 28 12:10:55.660499014 (548h41m55.531582049s) [upload=481121]: Done starting upload
Dec 28 12:10:55.660735474 (548h41m55.53181852s) [upload=481121]: Uploading segments...
Dec 28 12:10:55.660870462 (548h41m55.531953507s) [upload=481121]: Got next segment. Inline: false
Dec 28 12:10:55.660917210 (548h41m55.532000255s) [upload=481121]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 12:10:55.660924685 (548h41m55.53200773s) [upload=481121]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 12:10:55.909444035 (548h41m55.78052708s) [seg_pos={0 0}, upload=481121]: Begin upload segment...
Dec 28 12:10:55.909501735 (548h41m55.78058478s) [seg_pos={0 0}, upload=481121]: Done begin upload segment.
Dec 28 12:10:55.909539026 (548h41m55.780622071s) [noise=true , address=145.79.227.14:4837 , piece_id=UPBI5KCD7HTHZR6H2QMPS5LUGBCI7RXGBGNBJHG5I7RABK3XJ3FA, seg_pos={0 0}, upload=481121]: Uploading piece...
Dec 28 12:10:55.909568382 (548h41m55.780651427s) [noise=true , address=145.79.227.15:4873 , piece_id=3B6SG27RUZGJRYCT433Y5KCUEDZMYJY4LAE4IA7RTVPNQBMRWWFQ, seg_pos={0 0}, upload=481121]: Uploading piece...
Dec 28 12:10:55.910395015 (548h41m55.78147805s) [noise=true , address=145.79.227.42:4122 , piece_id=33XBPAO6JIVHHRTTHY6HFD5YXVTKROB5HRRDUV4THYKTPU2FZU6A, seg_pos={0 0}, upload=481121]: Uploading piece...
Dec 28 12:10:55.910684147 (548h41m55.781767182s) [noise=true , address=145.79.227.4:4186 , piece_id=PUB3YGQNDHW4AVBUIAPICCTOLRDLGDFA6YQTUYN6PQKX3TRZGUMA, seg_pos={0 0}, upload=481121]: Uploading piece...
Dec 28 12:10:55.910938651 (548h41m55.782021686s) [noise=true , address=145.79.227.45:4231 , piece_id=3D2JZTRHCEER64WWU3FBIJNCYFZOLDLQQYDD5DLOG563X4WML6KA, seg_pos={0 0}, upload=481121]: Uploading piece...
Dec 28 12:10:55.911205319 (548h41m55.782288354s) [noise=true , address=77.90.31.11:4554 , piece_id=N67F6BUDWMY7HQS2U33BXW4VZJ34MO73FY6WQWT5K2HBMPQCYH6Q, seg_pos={0 0}, upload=481121]: Uploading piece...
Dec 28 12:10:55.911457249 (548h41m55.782540284s) [noise=true , address=145.79.227.48:4301 , piece_id=3C2LRAO72K65765CNEPC6IQATH7XKRQ5C2HS4BG6VQYGM5EICS4Q, seg_pos={0 0}, upload=481121]: Uploading piece...
Dec 28 12:10:55.911748282 (548h41m55.782831317s) [noise=true , address=145.79.227.47:4265 , piece_id=AS47QV7TCZVOKCWGQH3E6O5PNEZ4YNC4VBGL7O4QRKXA6ZIGQWMA, seg_pos={0 0}, upload=481121]: Uploading piece...
Dec 28 12:10:55.912092448 (548h41m55.783175493s) [noise=true , address=77.90.31.13:4626 , piece_id=HFJFVBZJD6HYEAX2L6EF7UZVH3UEQ4GQBCDV2ZEYTU7RATF5UXGQ, seg_pos={0 0}, upload=481121]: Uploading piece...
Dec 28 12:10:55.912445791 (548h41m55.783528836s) [noise=true , address=45.91.116.14:4189 , piece_id=KH6OZOJORQK3QJFBAP3NJSBQTJE2MDENC74ZB25YBP45BEPQ7ICA, seg_pos={0 0}, upload=481121]: Uploading piece...
Dec 28 12:10:55.913104785 (548h41m55.784187831s) [noise=true , address=145.79.227.16:4910 , piece_id=GPVKLL4ZYNO6KLZ35R7X5HAMGCSVIIE3H5SS2UEOEJ7CNKDRFQHQ, seg_pos={0 0}, upload=481121]: Uploading piece...
Dec 28 12:10:55.913554612 (548h41m55.784637657s) [noise=true , address=77.90.31.16:4735 , piece_id=COCPUUARNUQ47ND6EIHR332XZJP7JCSVUNFV63QBIQJIZEIMQC2A, seg_pos={0 0}, upload=481121]: Uploading piece...
Dec 28 12:10:55.914012474 (548h41m55.785095519s) [noise=true , address=145.79.227.13:4803 , piece_id=53MLFZJ6QX2FQ6SG733SDERFTIGX6GNORWL5SAXDLZ2QM4TBKINA, seg_pos={0 0}, upload=481121]: Uploading piece...
Dec 28 12:10:55.914463302 (548h41m55.785546478s) [noise=true , address=145.79.227.11:4730 , piece_id=LIFLAF57ZA52IARNDGUVIYHQ6DFJC7VI3JNPY6T37A5YDKOUSRQQ, seg_pos={0 0}, upload=481121]: Uploading piece...
Dec 28 12:10:55.914861211 (548h41m55.785944246s) [noise=true , address=45.91.116.6:4974 , piece_id=DVDBH2I22B4XMYTEFZIZC5ENHCCNY2UYBKGPNXWOR5WU7ZSUEKYQ, seg_pos={0 0}, upload=481121]: Uploading piece...
Dec 28 12:10:55.915264708 (548h41m55.786347813s) [noise=true , address=145.79.227.39:4014 , piece_id=NFTS52EVDWICVF7OZ6DEXPTOAALSQPZLLST2YWEGYQCNIMUI4CPA, seg_pos={0 0}, upload=481121]: Uploading piece...
Dec 28 12:10:57.629256297 (548h41m57.500339342s) [upload=481121]: Next returned nil segment
Dec 28 12:10:57.629273770 (548h41m57.500356805s) [upload=481121]: Waiting for error group managing segments...
Dec 28 12:10:57.917456196 (548h41m57.788539231s) [noise=true , address=145.79.227.39:4014 , piece_id=NFTS52EVDWICVF7OZ6DEXPTOAALSQPZLLST2YWEGYQCNIMUI4CPA, seg_pos={0 0}, upload=481121]: Done uploading piece. err: <nil>
Dec 28 12:10:57.917907185 (548h41m57.78899022s) [noise=true , address=145.79.227.4:4186 , piece_id=PUB3YGQNDHW4AVBUIAPICCTOLRDLGDFA6YQTUYN6PQKX3TRZGUMA, seg_pos={0 0}, upload=481121]: Done uploading piece. err: <nil>
Dec 28 12:10:57.922473289 (548h41m57.793556334s) [noise=true , address=145.79.227.13:4803 , piece_id=53MLFZJ6QX2FQ6SG733SDERFTIGX6GNORWL5SAXDLZ2QM4TBKINA, seg_pos={0 0}, upload=481121]: Done uploading piece. err: <nil>
Dec 28 12:10:57.925552562 (548h41m57.796635607s) [noise=true , address=145.79.227.16:4910 , piece_id=GPVKLL4ZYNO6KLZ35R7X5HAMGCSVIIE3H5SS2UEOEJ7CNKDRFQHQ, seg_pos={0 0}, upload=481121]: Done uploading piece. err: <nil>
Dec 28 12:10:57.932490544 (548h41m57.803573579s) [noise=true , address=145.79.227.14:4837 , piece_id=UPBI5KCD7HTHZR6H2QMPS5LUGBCI7RXGBGNBJHG5I7RABK3XJ3FA, seg_pos={0 0}, upload=481121]: Done uploading piece. err: <nil>
Dec 28 12:10:57.932829238 (548h41m57.803912273s) [noise=true , address=145.79.227.42:4122 , piece_id=33XBPAO6JIVHHRTTHY6HFD5YXVTKROB5HRRDUV4THYKTPU2FZU6A, seg_pos={0 0}, upload=481121]: Done uploading piece. err: <nil>
Dec 28 12:10:57.937710152 (548h41m57.808793197s) [noise=true , address=145.79.227.11:4730 , piece_id=LIFLAF57ZA52IARNDGUVIYHQ6DFJC7VI3JNPY6T37A5YDKOUSRQQ, seg_pos={0 0}, upload=481121]: Done uploading piece. err: <nil>
Dec 28 12:10:57.940392400 (548h41m57.811475445s) [noise=true , address=145.79.227.45:4231 , piece_id=3D2JZTRHCEER64WWU3FBIJNCYFZOLDLQQYDD5DLOG563X4WML6KA, seg_pos={0 0}, upload=481121]: Done uploading piece. err: <nil>
Dec 28 12:10:57.945343066 (548h41m57.816426111s) [noise=true , address=145.79.227.48:4301 , piece_id=3C2LRAO72K65765CNEPC6IQATH7XKRQ5C2HS4BG6VQYGM5EICS4Q, seg_pos={0 0}, upload=481121]: Done uploading piece. err: <nil>
Dec 28 12:10:57.945399194 (548h41m57.816482229s) [noise=true , address=145.79.227.15:4873 , piece_id=3B6SG27RUZGJRYCT433Y5KCUEDZMYJY4LAE4IA7RTVPNQBMRWWFQ, seg_pos={0 0}, upload=481121]: Done uploading piece. err: <nil>
Dec 28 12:10:57.951082956 (548h41m57.822166001s) [noise=true , address=145.79.227.47:4265 , piece_id=AS47QV7TCZVOKCWGQH3E6O5PNEZ4YNC4VBGL7O4QRKXA6ZIGQWMA, seg_pos={0 0}, upload=481121]: Done uploading piece. err: <nil>
Dec 28 12:10:58.132138455 (548h41m58.00322151s) [noise=true , address=45.91.116.6:4974 , piece_id=DVDBH2I22B4XMYTEFZIZC5ENHCCNY2UYBKGPNXWOR5WU7ZSUEKYQ, seg_pos={0 0}, upload=481121]: Done uploading piece. err: <nil>
Dec 28 12:10:58.238648407 (548h41m58.109731442s) [noise=true , address=77.90.31.11:4554 , piece_id=N67F6BUDWMY7HQS2U33BXW4VZJ34MO73FY6WQWT5K2HBMPQCYH6Q, seg_pos={0 0}, upload=481121]: Done uploading piece. err: <nil>
Dec 28 12:10:58.238678594 (548h41m58.109761639s) [seg_pos={0 0}, upload=481121]: Segment reached optimal threshold of 13 pieces.
Dec 28 12:10:58.238881581 (548h41m58.109964626s) [noise=true , address=77.90.31.16:4735 , piece_id=COCPUUARNUQ47ND6EIHR332XZJP7JCSVUNFV63QBIQJIZEIMQC2A, seg_pos={0 0}, upload=481121]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF
Dec 28 12:10:58.238889496 (548h41m58.109972531s) [noise=true , address=77.90.31.16:4735 , piece_id=COCPUUARNUQ47ND6EIHR332XZJP7JCSVUNFV63QBIQJIZEIMQC2A, seg_pos={0 0}, upload=481121]: Long tail cancellation due to optimal threshold
Dec 28 12:10:58.238995468 (548h41m58.110078503s) [noise=true , address=45.91.116.14:4189 , piece_id=KH6OZOJORQK3QJFBAP3NJSBQTJE2MDENC74ZB25YBP45BEPQ7ICA, seg_pos={0 0}, upload=481121]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: write tcp 188.166.225.170:56090->45.91.116.14:4189: use of closed network connection; write tcp 188.166.225.170:56090->45.91.116.14:4189: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56090->45.91.116.14:4189: use of closed network connection
Dec 28 12:10:58.239016368 (548h41m58.110099403s) [noise=true , address=45.91.116.14:4189 , piece_id=KH6OZOJORQK3QJFBAP3NJSBQTJE2MDENC74ZB25YBP45BEPQ7ICA, seg_pos={0 0}, upload=481121]: Long tail cancellation due to optimal threshold
Dec 28 12:10:58.239125055 (548h41m58.11020809s) [noise=true , address=77.90.31.13:4626 , piece_id=HFJFVBZJD6HYEAX2L6EF7UZVH3UEQ4GQBCDV2ZEYTU7RATF5UXGQ, seg_pos={0 0}, upload=481121]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF
Dec 28 12:10:58.239131747 (548h41m58.110214792s) [noise=true , address=77.90.31.13:4626 , piece_id=HFJFVBZJD6HYEAX2L6EF7UZVH3UEQ4GQBCDV2ZEYTU7RATF5UXGQ, seg_pos={0 0}, upload=481121]: Long tail cancellation due to optimal threshold
Dec 28 12:10:58.239152227 (548h41m58.110235262s) [seg_pos={0 0}, upload=481121]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 12:10:58.239217340 (548h41m58.110300375s) [upload=481121]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 12:10:58.239223222 (548h41m58.110306267s) [upload=481121]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 12:10:58.538804292 (548h41m58.409887327s) [upload=481121]: Done uploading segments...
Dec 28 12:10:58.538831865 (548h41m58.4099149s) [upload=481121]: Upload finished. err: <nil>
Duration: 6.023842484s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..................................X.X.......................] duration:
7.016161692s trace:
https://jaeger.dev.storj.io/trace/72d0191c7bb67437
FAIL (2025-12-28 12:13:19 UTC)
orbiter warning: context deadline exceeded
Duration: 7.016190457s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [...X..X.....................................................] duration:
15.404707838s trace:
https://jaeger.dev.storj.io/trace/307d4901d0d5949
FAIL (2025-12-28 12:12:53 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
ise=true , address=77.90.31.16:4743 , piece_id=NRCRNEOEKXPSKREUZKGGLQHLWG5JTKWCQCRUUHKUHB4HOISVHCFQ, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.355028753 (547h59m23.226111788s) [noise=true , address=145.79.227.30:4796 , piece_id=MUD7JHR4ETJYGG7SSOOWQ5JTKSQYV26X4AWZ6XHEBHLJYE3774CA, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.355039954 (547h59m23.226122989s) [noise=true , address=45.91.116.11:4122 , piece_id=MROLZP4Y2PKVJSUUXCSDQQGVGF7AJADXWEJFFPH6SU2DWTZR443A, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.355282126 (547h59m23.226365171s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.355390983 (547h59m23.226474018s) [noise=true , address=193.149.164.143:4700 , piece_id=3FBVJX372QBFVBDXCF5VTCIT2JUYHNXFCZPJ2FN7E34HA4S4WETQ, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.355527633 (547h59m23.226610668s) [noise=true , address=77.90.31.15:4727 , piece_id=KZ74K3R4G2BD73RTJFOKFGQ57SV456NQC4W3NTPT52KBQAU3SWNA, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.355648985 (547h59m23.22673202s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.355877279 (547h59m23.226960324s) [noise=true , address=145.79.227.12:4788 , piece_id=KFB2WTRQOSZ75WMHRNCRTPS2FSMIBR27TQ55V62J2PGWLFECB4RA, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.356182711 (547h59m23.227265756s) [noise=true , address=145.79.227.49:4351 , piece_id=FYLYJ6GRQIMVTNGLLY62Q35M5F4IEQW2EPKVMSDOVJIJJPJJXF3Q, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.356536194 (547h59m23.227619229s) [noise=true , address=145.79.227.19:4092 , piece_id=7UPEHPU5NSBVOPSDNKUAQQZXBHHBS36PKZOR5FBDKLWP4VA4ZSDA, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.356869479 (547h59m23.227952514s) [noise=true , address=145.79.227.33:4882 , piece_id=VJ6XYH77LJSKKBVCCVHHTGFV3WTSCYD4XUKUMSYOWZEGZ2QY4H3A, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.357157427 (547h59m23.228240462s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.357426930 (547h59m23.228509975s) [noise=true , address=145.79.227.40:4067 , piece_id=YOIOW6QPNOMK5ZD5EEWT2FLVGVZZMHNWANUQF4UE3R4FHSOFWSWQ, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.357751889 (547h59m23.228834924s) [noise=true , address=77.90.31.10:4536 , piece_id=HKI6JPRV4D2CBIVEGHRXSCDC2OROZLX5SW3VC7W35L4DWUEITV2Q, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.358953407 (547h59m23.230036442s) [noise=true , address=77.90.31.17:4790 , piece_id=AS6RYIPXJKC6CFSPVX7MLENRTQHI7RZFLTSPHFMIYG5OS7BKZHRA, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.359286491 (547h59m23.230369536s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:24.695915344 (547h59m24.566998389s) [upload=480503]: Next returned nil segment
Dec 28 11:28:24.695941574 (547h59m24.567024619s) [upload=480503]: Waiting for error group managing segments...
Dec 28 11:28:25.041244477 (547h59m24.912327512s) [noise=true , address=193.149.164.143:4700 , piece_id=3FBVJX372QBFVBDXCF5VTCIT2JUYHNXFCZPJ2FN7E34HA4S4WETQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:25.237036778 (547h59m25.108119823s) [noise=true , address=145.79.227.49:4351 , piece_id=FYLYJ6GRQIMVTNGLLY62Q35M5F4IEQW2EPKVMSDOVJIJJPJJXF3Q, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:25.249693558 (547h59m25.120776593s) [noise=true , address=145.79.227.12:4788 , piece_id=KFB2WTRQOSZ75WMHRNCRTPS2FSMIBR27TQ55V62J2PGWLFECB4RA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:25.256819657 (547h59m25.127902692s) [noise=true , address=145.79.227.19:4092 , piece_id=7UPEHPU5NSBVOPSDNKUAQQZXBHHBS36PKZOR5FBDKLWP4VA4ZSDA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:25.511089018 (547h59m25.382172053s) [noise=true , address=45.91.116.13:4170 , piece_id=3KJJFTI5M7MBDSZNTBWSMSJUU7B325XVVLXEH3JKX5N7N4WNMXAA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:25.523026599 (547h59m25.394109634s) [noise=true , address=77.90.31.17:4790 , piece_id=AS6RYIPXJKC6CFSPVX7MLENRTQHI7RZFLTSPHFMIYG5OS7BKZHRA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:25.524217066 (547h59m25.395300101s) [noise=true , address=45.91.116.11:4122 , piece_id=MROLZP4Y2PKVJSUUXCSDQQGVGF7AJADXWEJFFPH6SU2DWTZR443A, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:25.665135729 (547h59m25.536218774s) [noise=true , address=77.90.31.15:4727 , piece_id=KZ74K3R4G2BD73RTJFOKFGQ57SV456NQC4W3NTPT52KBQAU3SWNA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:25.917975247 (547h59m25.789058292s) [noise=true , address=77.90.31.16:4743 , piece_id=NRCRNEOEKXPSKREUZKGGLQHLWG5JTKWCQCRUUHKUHB4HOISVHCFQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:26.147871236 (547h59m26.018954271s) [noise=true , address=145.79.227.30:4796 , piece_id=MUD7JHR4ETJYGG7SSOOWQ5JTKSQYV26X4AWZ6XHEBHLJYE3774CA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:26.173243839 (547h59m26.044326874s) [noise=true , address=145.79.227.44:4209 , piece_id=PKQUKW3FOPJL7BRNFHNLAJ4DSKGMGA73ROILSWEFCT7MR7H5GNNQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:26.289483660 (547h59m26.160566695s) [noise=true , address=77.90.31.10:4536 , piece_id=HKI6JPRV4D2CBIVEGHRXSCDC2OROZLX5SW3VC7W35L4DWUEITV2Q, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:26.448835962 (547h59m26.319919018s) [noise=true , address=145.79.227.40:4067 , piece_id=YOIOW6QPNOMK5ZD5EEWT2FLVGVZZMHNWANUQF4UE3R4FHSOFWSWQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:26.571228809 (547h59m26.442311844s) [noise=true , address=145.79.227.33:4882 , piece_id=VJ6XYH77LJSKKBVCCVHHTGFV3WTSCYD4XUKUMSYOWZEGZ2QY4H3A, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:26.571248396 (547h59m26.442331441s) [seg_pos={0 0}, upload=480503]: Segment reached optimal threshold of 14 pieces.
Dec 28 11:28:26.571429892 (547h59m26.442512927s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NEV2ndHw4fohVUNsfrzwojd8S11TmMST6Sz59GVnXS8vDGhWk): protocol: expected piece hash; context canceled; EOF
Dec 28 11:28:26.571436033 (547h59m26.442519078s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold
Dec 28 11:28:26.571564158 (547h59m26.442647203s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection
Dec 28 11:28:26.571580588 (547h59m26.442663623s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold
Dec 28 11:28:26.571712249 (547h59m26.442795284s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Nw78dd9WPnmCwv5bFQR3VpV39776Pye2QVGSNMTs3zq2BtaGJ): protocol: expected piece hash; context canceled; EOF
Dec 28 11:28:26.571720666 (547h59m26.442803701s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold
Dec 28 11:28:26.571767825 (547h59m26.442850871s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv): protocol: expected piece hash; context canceled; EOF
Dec 28 11:28:26.571774688 (547h59m26.442857723s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold
Dec 28 11:28:26.571792493 (547h59m26.442875538s) [seg_pos={0 0}, upload=480503]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 11:28:26.571882343 (547h59m26.442965378s) [upload=480503]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 11:28:26.571887683 (547h59m26.442970728s) [upload=480503]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 11:28:26.858811294 (547h59m26.729894329s) [upload=480503]: Done uploading segments...
Dec 28 11:28:26.858832874 (547h59m26.72991592s) [upload=480503]: Upload finished. err: <nil>
Duration: 15.404732625s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
34 out of 60 is failed (threshold=6) [X.....X.X.X..X.XXX..X....XX.XX.XXXXXXXXXX..XXXX...XX.XXX.XX.] duration:
3.459640093s trace:
https://jaeger.dev.storj.io/trace/4a7cc66cea50a588
FAIL (2025-12-28 12:12:44 UTC)
orbiter warning: context deadline exceeded
Duration: 3.459667154s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [...................................X.X......................] duration:
20.278618439s trace:
https://jaeger.dev.storj.io/trace/72d0191c7bb67437
FAIL (2025-12-28 12:12:33 UTC)
orbiter warning: context deadline exceeded
Duration: 20.278651592s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [....X..X....................................................] duration:
35.700878779s trace:
https://jaeger.dev.storj.io/trace/307d4901d0d5949
FAIL (2025-12-28 12:12:32 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
ise=true , address=77.90.31.16:4743 , piece_id=NRCRNEOEKXPSKREUZKGGLQHLWG5JTKWCQCRUUHKUHB4HOISVHCFQ, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.355028753 (547h59m23.226111788s) [noise=true , address=145.79.227.30:4796 , piece_id=MUD7JHR4ETJYGG7SSOOWQ5JTKSQYV26X4AWZ6XHEBHLJYE3774CA, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.355039954 (547h59m23.226122989s) [noise=true , address=45.91.116.11:4122 , piece_id=MROLZP4Y2PKVJSUUXCSDQQGVGF7AJADXWEJFFPH6SU2DWTZR443A, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.355282126 (547h59m23.226365171s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.355390983 (547h59m23.226474018s) [noise=true , address=193.149.164.143:4700 , piece_id=3FBVJX372QBFVBDXCF5VTCIT2JUYHNXFCZPJ2FN7E34HA4S4WETQ, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.355527633 (547h59m23.226610668s) [noise=true , address=77.90.31.15:4727 , piece_id=KZ74K3R4G2BD73RTJFOKFGQ57SV456NQC4W3NTPT52KBQAU3SWNA, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.355648985 (547h59m23.22673202s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.355877279 (547h59m23.226960324s) [noise=true , address=145.79.227.12:4788 , piece_id=KFB2WTRQOSZ75WMHRNCRTPS2FSMIBR27TQ55V62J2PGWLFECB4RA, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.356182711 (547h59m23.227265756s) [noise=true , address=145.79.227.49:4351 , piece_id=FYLYJ6GRQIMVTNGLLY62Q35M5F4IEQW2EPKVMSDOVJIJJPJJXF3Q, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.356536194 (547h59m23.227619229s) [noise=true , address=145.79.227.19:4092 , piece_id=7UPEHPU5NSBVOPSDNKUAQQZXBHHBS36PKZOR5FBDKLWP4VA4ZSDA, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.356869479 (547h59m23.227952514s) [noise=true , address=145.79.227.33:4882 , piece_id=VJ6XYH77LJSKKBVCCVHHTGFV3WTSCYD4XUKUMSYOWZEGZ2QY4H3A, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.357157427 (547h59m23.228240462s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.357426930 (547h59m23.228509975s) [noise=true , address=145.79.227.40:4067 , piece_id=YOIOW6QPNOMK5ZD5EEWT2FLVGVZZMHNWANUQF4UE3R4FHSOFWSWQ, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.357751889 (547h59m23.228834924s) [noise=true , address=77.90.31.10:4536 , piece_id=HKI6JPRV4D2CBIVEGHRXSCDC2OROZLX5SW3VC7W35L4DWUEITV2Q, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.358953407 (547h59m23.230036442s) [noise=true , address=77.90.31.17:4790 , piece_id=AS6RYIPXJKC6CFSPVX7MLENRTQHI7RZFLTSPHFMIYG5OS7BKZHRA, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.359286491 (547h59m23.230369536s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:24.695915344 (547h59m24.566998389s) [upload=480503]: Next returned nil segment
Dec 28 11:28:24.695941574 (547h59m24.567024619s) [upload=480503]: Waiting for error group managing segments...
Dec 28 11:28:25.041244477 (547h59m24.912327512s) [noise=true , address=193.149.164.143:4700 , piece_id=3FBVJX372QBFVBDXCF5VTCIT2JUYHNXFCZPJ2FN7E34HA4S4WETQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:25.237036778 (547h59m25.108119823s) [noise=true , address=145.79.227.49:4351 , piece_id=FYLYJ6GRQIMVTNGLLY62Q35M5F4IEQW2EPKVMSDOVJIJJPJJXF3Q, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:25.249693558 (547h59m25.120776593s) [noise=true , address=145.79.227.12:4788 , piece_id=KFB2WTRQOSZ75WMHRNCRTPS2FSMIBR27TQ55V62J2PGWLFECB4RA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:25.256819657 (547h59m25.127902692s) [noise=true , address=145.79.227.19:4092 , piece_id=7UPEHPU5NSBVOPSDNKUAQQZXBHHBS36PKZOR5FBDKLWP4VA4ZSDA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:25.511089018 (547h59m25.382172053s) [noise=true , address=45.91.116.13:4170 , piece_id=3KJJFTI5M7MBDSZNTBWSMSJUU7B325XVVLXEH3JKX5N7N4WNMXAA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:25.523026599 (547h59m25.394109634s) [noise=true , address=77.90.31.17:4790 , piece_id=AS6RYIPXJKC6CFSPVX7MLENRTQHI7RZFLTSPHFMIYG5OS7BKZHRA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:25.524217066 (547h59m25.395300101s) [noise=true , address=45.91.116.11:4122 , piece_id=MROLZP4Y2PKVJSUUXCSDQQGVGF7AJADXWEJFFPH6SU2DWTZR443A, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:25.665135729 (547h59m25.536218774s) [noise=true , address=77.90.31.15:4727 , piece_id=KZ74K3R4G2BD73RTJFOKFGQ57SV456NQC4W3NTPT52KBQAU3SWNA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:25.917975247 (547h59m25.789058292s) [noise=true , address=77.90.31.16:4743 , piece_id=NRCRNEOEKXPSKREUZKGGLQHLWG5JTKWCQCRUUHKUHB4HOISVHCFQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:26.147871236 (547h59m26.018954271s) [noise=true , address=145.79.227.30:4796 , piece_id=MUD7JHR4ETJYGG7SSOOWQ5JTKSQYV26X4AWZ6XHEBHLJYE3774CA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:26.173243839 (547h59m26.044326874s) [noise=true , address=145.79.227.44:4209 , piece_id=PKQUKW3FOPJL7BRNFHNLAJ4DSKGMGA73ROILSWEFCT7MR7H5GNNQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:26.289483660 (547h59m26.160566695s) [noise=true , address=77.90.31.10:4536 , piece_id=HKI6JPRV4D2CBIVEGHRXSCDC2OROZLX5SW3VC7W35L4DWUEITV2Q, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:26.448835962 (547h59m26.319919018s) [noise=true , address=145.79.227.40:4067 , piece_id=YOIOW6QPNOMK5ZD5EEWT2FLVGVZZMHNWANUQF4UE3R4FHSOFWSWQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:26.571228809 (547h59m26.442311844s) [noise=true , address=145.79.227.33:4882 , piece_id=VJ6XYH77LJSKKBVCCVHHTGFV3WTSCYD4XUKUMSYOWZEGZ2QY4H3A, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:26.571248396 (547h59m26.442331441s) [seg_pos={0 0}, upload=480503]: Segment reached optimal threshold of 14 pieces.
Dec 28 11:28:26.571429892 (547h59m26.442512927s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NEV2ndHw4fohVUNsfrzwojd8S11TmMST6Sz59GVnXS8vDGhWk): protocol: expected piece hash; context canceled; EOF
Dec 28 11:28:26.571436033 (547h59m26.442519078s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold
Dec 28 11:28:26.571564158 (547h59m26.442647203s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection
Dec 28 11:28:26.571580588 (547h59m26.442663623s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold
Dec 28 11:28:26.571712249 (547h59m26.442795284s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Nw78dd9WPnmCwv5bFQR3VpV39776Pye2QVGSNMTs3zq2BtaGJ): protocol: expected piece hash; context canceled; EOF
Dec 28 11:28:26.571720666 (547h59m26.442803701s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold
Dec 28 11:28:26.571767825 (547h59m26.442850871s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv): protocol: expected piece hash; context canceled; EOF
Dec 28 11:28:26.571774688 (547h59m26.442857723s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold
Dec 28 11:28:26.571792493 (547h59m26.442875538s) [seg_pos={0 0}, upload=480503]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 11:28:26.571882343 (547h59m26.442965378s) [upload=480503]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 11:28:26.571887683 (547h59m26.442970728s) [upload=480503]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 11:28:26.858811294 (547h59m26.729894329s) [upload=480503]: Done uploading segments...
Dec 28 11:28:26.858832874 (547h59m26.72991592s) [upload=480503]: Upload finished. err: <nil>
Duration: 35.700892615s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
17 out of 60 is failed (threshold=3) [..XX....X.....XX...XXX.......X..XX....X...X.........X..XX.X.] duration:
5.720864077s trace:
https://jaeger.dev.storj.io/trace/76cfc3bec30f9bc9
FAIL (2025-12-28 12:12:05 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: DVDBH2I22B4XMYTEFZIZC5ENHCCNY2UYBKGPNXWOR5WU7ZSUEKYQ) context deadline exceeded; piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: N67F6BUDWMY7HQS2U33BXW4VZJ34MO73FY6WQWT5K2HBMPQCYH6Q) context deadline exceeded
======LOGS======
Dec 28 12:10:55.660430003 (548h41m55.531513048s) [upload=481121]: Starting upload
Dec 28 12:10:55.660499014 (548h41m55.531582049s) [upload=481121]: Done starting upload
Dec 28 12:10:55.660735474 (548h41m55.53181852s) [upload=481121]: Uploading segments...
Dec 28 12:10:55.660870462 (548h41m55.531953507s) [upload=481121]: Got next segment. Inline: false
Dec 28 12:10:55.660917210 (548h41m55.532000255s) [upload=481121]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 12:10:55.660924685 (548h41m55.53200773s) [upload=481121]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 12:10:55.909444035 (548h41m55.78052708s) [seg_pos={0 0}, upload=481121]: Begin upload segment...
Dec 28 12:10:55.909501735 (548h41m55.78058478s) [seg_pos={0 0}, upload=481121]: Done begin upload segment.
Dec 28 12:10:55.909539026 (548h41m55.780622071s) [noise=true , address=145.79.227.14:4837 , piece_id=UPBI5KCD7HTHZR6H2QMPS5LUGBCI7RXGBGNBJHG5I7RABK3XJ3FA, seg_pos={0 0}, upload=481121]: Uploading piece...
Dec 28 12:10:55.909568382 (548h41m55.780651427s) [noise=true , address=145.79.227.15:4873 , piece_id=3B6SG27RUZGJRYCT433Y5KCUEDZMYJY4LAE4IA7RTVPNQBMRWWFQ, seg_pos={0 0}, upload=481121]: Uploading piece...
Dec 28 12:10:55.910395015 (548h41m55.78147805s) [noise=true , address=145.79.227.42:4122 , piece_id=33XBPAO6JIVHHRTTHY6HFD5YXVTKROB5HRRDUV4THYKTPU2FZU6A, seg_pos={0 0}, upload=481121]: Uploading piece...
Dec 28 12:10:55.910684147 (548h41m55.781767182s) [noise=true , address=145.79.227.4:4186 , piece_id=PUB3YGQNDHW4AVBUIAPICCTOLRDLGDFA6YQTUYN6PQKX3TRZGUMA, seg_pos={0 0}, upload=481121]: Uploading piece...
Dec 28 12:10:55.910938651 (548h41m55.782021686s) [noise=true , address=145.79.227.45:4231 , piece_id=3D2JZTRHCEER64WWU3FBIJNCYFZOLDLQQYDD5DLOG563X4WML6KA, seg_pos={0 0}, upload=481121]: Uploading piece...
Dec 28 12:10:55.911205319 (548h41m55.782288354s) [noise=true , address=77.90.31.11:4554 , piece_id=N67F6BUDWMY7HQS2U33BXW4VZJ34MO73FY6WQWT5K2HBMPQCYH6Q, seg_pos={0 0}, upload=481121]: Uploading piece...
Dec 28 12:10:55.911457249 (548h41m55.782540284s) [noise=true , address=145.79.227.48:4301 , piece_id=3C2LRAO72K65765CNEPC6IQATH7XKRQ5C2HS4BG6VQYGM5EICS4Q, seg_pos={0 0}, upload=481121]: Uploading piece...
Dec 28 12:10:55.911748282 (548h41m55.782831317s) [noise=true , address=145.79.227.47:4265 , piece_id=AS47QV7TCZVOKCWGQH3E6O5PNEZ4YNC4VBGL7O4QRKXA6ZIGQWMA, seg_pos={0 0}, upload=481121]: Uploading piece...
Dec 28 12:10:55.912092448 (548h41m55.783175493s) [noise=true , address=77.90.31.13:4626 , piece_id=HFJFVBZJD6HYEAX2L6EF7UZVH3UEQ4GQBCDV2ZEYTU7RATF5UXGQ, seg_pos={0 0}, upload=481121]: Uploading piece...
Dec 28 12:10:55.912445791 (548h41m55.783528836s) [noise=true , address=45.91.116.14:4189 , piece_id=KH6OZOJORQK3QJFBAP3NJSBQTJE2MDENC74ZB25YBP45BEPQ7ICA, seg_pos={0 0}, upload=481121]: Uploading piece...
Dec 28 12:10:55.913104785 (548h41m55.784187831s) [noise=true , address=145.79.227.16:4910 , piece_id=GPVKLL4ZYNO6KLZ35R7X5HAMGCSVIIE3H5SS2UEOEJ7CNKDRFQHQ, seg_pos={0 0}, upload=481121]: Uploading piece...
Dec 28 12:10:55.913554612 (548h41m55.784637657s) [noise=true , address=77.90.31.16:4735 , piece_id=COCPUUARNUQ47ND6EIHR332XZJP7JCSVUNFV63QBIQJIZEIMQC2A, seg_pos={0 0}, upload=481121]: Uploading piece...
Dec 28 12:10:55.914012474 (548h41m55.785095519s) [noise=true , address=145.79.227.13:4803 , piece_id=53MLFZJ6QX2FQ6SG733SDERFTIGX6GNORWL5SAXDLZ2QM4TBKINA, seg_pos={0 0}, upload=481121]: Uploading piece...
Dec 28 12:10:55.914463302 (548h41m55.785546478s) [noise=true , address=145.79.227.11:4730 , piece_id=LIFLAF57ZA52IARNDGUVIYHQ6DFJC7VI3JNPY6T37A5YDKOUSRQQ, seg_pos={0 0}, upload=481121]: Uploading piece...
Dec 28 12:10:55.914861211 (548h41m55.785944246s) [noise=true , address=45.91.116.6:4974 , piece_id=DVDBH2I22B4XMYTEFZIZC5ENHCCNY2UYBKGPNXWOR5WU7ZSUEKYQ, seg_pos={0 0}, upload=481121]: Uploading piece...
Dec 28 12:10:55.915264708 (548h41m55.786347813s) [noise=true , address=145.79.227.39:4014 , piece_id=NFTS52EVDWICVF7OZ6DEXPTOAALSQPZLLST2YWEGYQCNIMUI4CPA, seg_pos={0 0}, upload=481121]: Uploading piece...
Dec 28 12:10:57.629256297 (548h41m57.500339342s) [upload=481121]: Next returned nil segment
Dec 28 12:10:57.629273770 (548h41m57.500356805s) [upload=481121]: Waiting for error group managing segments...
Dec 28 12:10:57.917456196 (548h41m57.788539231s) [noise=true , address=145.79.227.39:4014 , piece_id=NFTS52EVDWICVF7OZ6DEXPTOAALSQPZLLST2YWEGYQCNIMUI4CPA, seg_pos={0 0}, upload=481121]: Done uploading piece. err: <nil>
Dec 28 12:10:57.917907185 (548h41m57.78899022s) [noise=true , address=145.79.227.4:4186 , piece_id=PUB3YGQNDHW4AVBUIAPICCTOLRDLGDFA6YQTUYN6PQKX3TRZGUMA, seg_pos={0 0}, upload=481121]: Done uploading piece. err: <nil>
Dec 28 12:10:57.922473289 (548h41m57.793556334s) [noise=true , address=145.79.227.13:4803 , piece_id=53MLFZJ6QX2FQ6SG733SDERFTIGX6GNORWL5SAXDLZ2QM4TBKINA, seg_pos={0 0}, upload=481121]: Done uploading piece. err: <nil>
Dec 28 12:10:57.925552562 (548h41m57.796635607s) [noise=true , address=145.79.227.16:4910 , piece_id=GPVKLL4ZYNO6KLZ35R7X5HAMGCSVIIE3H5SS2UEOEJ7CNKDRFQHQ, seg_pos={0 0}, upload=481121]: Done uploading piece. err: <nil>
Dec 28 12:10:57.932490544 (548h41m57.803573579s) [noise=true , address=145.79.227.14:4837 , piece_id=UPBI5KCD7HTHZR6H2QMPS5LUGBCI7RXGBGNBJHG5I7RABK3XJ3FA, seg_pos={0 0}, upload=481121]: Done uploading piece. err: <nil>
Dec 28 12:10:57.932829238 (548h41m57.803912273s) [noise=true , address=145.79.227.42:4122 , piece_id=33XBPAO6JIVHHRTTHY6HFD5YXVTKROB5HRRDUV4THYKTPU2FZU6A, seg_pos={0 0}, upload=481121]: Done uploading piece. err: <nil>
Dec 28 12:10:57.937710152 (548h41m57.808793197s) [noise=true , address=145.79.227.11:4730 , piece_id=LIFLAF57ZA52IARNDGUVIYHQ6DFJC7VI3JNPY6T37A5YDKOUSRQQ, seg_pos={0 0}, upload=481121]: Done uploading piece. err: <nil>
Dec 28 12:10:57.940392400 (548h41m57.811475445s) [noise=true , address=145.79.227.45:4231 , piece_id=3D2JZTRHCEER64WWU3FBIJNCYFZOLDLQQYDD5DLOG563X4WML6KA, seg_pos={0 0}, upload=481121]: Done uploading piece. err: <nil>
Dec 28 12:10:57.945343066 (548h41m57.816426111s) [noise=true , address=145.79.227.48:4301 , piece_id=3C2LRAO72K65765CNEPC6IQATH7XKRQ5C2HS4BG6VQYGM5EICS4Q, seg_pos={0 0}, upload=481121]: Done uploading piece. err: <nil>
Dec 28 12:10:57.945399194 (548h41m57.816482229s) [noise=true , address=145.79.227.15:4873 , piece_id=3B6SG27RUZGJRYCT433Y5KCUEDZMYJY4LAE4IA7RTVPNQBMRWWFQ, seg_pos={0 0}, upload=481121]: Done uploading piece. err: <nil>
Dec 28 12:10:57.951082956 (548h41m57.822166001s) [noise=true , address=145.79.227.47:4265 , piece_id=AS47QV7TCZVOKCWGQH3E6O5PNEZ4YNC4VBGL7O4QRKXA6ZIGQWMA, seg_pos={0 0}, upload=481121]: Done uploading piece. err: <nil>
Dec 28 12:10:58.132138455 (548h41m58.00322151s) [noise=true , address=45.91.116.6:4974 , piece_id=DVDBH2I22B4XMYTEFZIZC5ENHCCNY2UYBKGPNXWOR5WU7ZSUEKYQ, seg_pos={0 0}, upload=481121]: Done uploading piece. err: <nil>
Dec 28 12:10:58.238648407 (548h41m58.109731442s) [noise=true , address=77.90.31.11:4554 , piece_id=N67F6BUDWMY7HQS2U33BXW4VZJ34MO73FY6WQWT5K2HBMPQCYH6Q, seg_pos={0 0}, upload=481121]: Done uploading piece. err: <nil>
Dec 28 12:10:58.238678594 (548h41m58.109761639s) [seg_pos={0 0}, upload=481121]: Segment reached optimal threshold of 13 pieces.
Dec 28 12:10:58.238881581 (548h41m58.109964626s) [noise=true , address=77.90.31.16:4735 , piece_id=COCPUUARNUQ47ND6EIHR332XZJP7JCSVUNFV63QBIQJIZEIMQC2A, seg_pos={0 0}, upload=481121]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF
Dec 28 12:10:58.238889496 (548h41m58.109972531s) [noise=true , address=77.90.31.16:4735 , piece_id=COCPUUARNUQ47ND6EIHR332XZJP7JCSVUNFV63QBIQJIZEIMQC2A, seg_pos={0 0}, upload=481121]: Long tail cancellation due to optimal threshold
Dec 28 12:10:58.238995468 (548h41m58.110078503s) [noise=true , address=45.91.116.14:4189 , piece_id=KH6OZOJORQK3QJFBAP3NJSBQTJE2MDENC74ZB25YBP45BEPQ7ICA, seg_pos={0 0}, upload=481121]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: write tcp 188.166.225.170:56090->45.91.116.14:4189: use of closed network connection; write tcp 188.166.225.170:56090->45.91.116.14:4189: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56090->45.91.116.14:4189: use of closed network connection
Dec 28 12:10:58.239016368 (548h41m58.110099403s) [noise=true , address=45.91.116.14:4189 , piece_id=KH6OZOJORQK3QJFBAP3NJSBQTJE2MDENC74ZB25YBP45BEPQ7ICA, seg_pos={0 0}, upload=481121]: Long tail cancellation due to optimal threshold
Dec 28 12:10:58.239125055 (548h41m58.11020809s) [noise=true , address=77.90.31.13:4626 , piece_id=HFJFVBZJD6HYEAX2L6EF7UZVH3UEQ4GQBCDV2ZEYTU7RATF5UXGQ, seg_pos={0 0}, upload=481121]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF
Dec 28 12:10:58.239131747 (548h41m58.110214792s) [noise=true , address=77.90.31.13:4626 , piece_id=HFJFVBZJD6HYEAX2L6EF7UZVH3UEQ4GQBCDV2ZEYTU7RATF5UXGQ, seg_pos={0 0}, upload=481121]: Long tail cancellation due to optimal threshold
Dec 28 12:10:58.239152227 (548h41m58.110235262s) [seg_pos={0 0}, upload=481121]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 12:10:58.239217340 (548h41m58.110300375s) [upload=481121]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 12:10:58.239223222 (548h41m58.110306267s) [upload=481121]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 12:10:58.538804292 (548h41m58.409887327s) [upload=481121]: Done uploading segments...
Dec 28 12:10:58.538831865 (548h41m58.4099149s) [upload=481121]: Upload finished. err: <nil>
Duration: 5.720915695s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [.....X..X...................................................] duration:
37.782461544s trace:
https://jaeger.dev.storj.io/trace/307d4901d0d5949
FAIL (2025-12-28 12:11:56 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
ise=true , address=77.90.31.16:4743 , piece_id=NRCRNEOEKXPSKREUZKGGLQHLWG5JTKWCQCRUUHKUHB4HOISVHCFQ, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.355028753 (547h59m23.226111788s) [noise=true , address=145.79.227.30:4796 , piece_id=MUD7JHR4ETJYGG7SSOOWQ5JTKSQYV26X4AWZ6XHEBHLJYE3774CA, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.355039954 (547h59m23.226122989s) [noise=true , address=45.91.116.11:4122 , piece_id=MROLZP4Y2PKVJSUUXCSDQQGVGF7AJADXWEJFFPH6SU2DWTZR443A, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.355282126 (547h59m23.226365171s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.355390983 (547h59m23.226474018s) [noise=true , address=193.149.164.143:4700 , piece_id=3FBVJX372QBFVBDXCF5VTCIT2JUYHNXFCZPJ2FN7E34HA4S4WETQ, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.355527633 (547h59m23.226610668s) [noise=true , address=77.90.31.15:4727 , piece_id=KZ74K3R4G2BD73RTJFOKFGQ57SV456NQC4W3NTPT52KBQAU3SWNA, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.355648985 (547h59m23.22673202s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.355877279 (547h59m23.226960324s) [noise=true , address=145.79.227.12:4788 , piece_id=KFB2WTRQOSZ75WMHRNCRTPS2FSMIBR27TQ55V62J2PGWLFECB4RA, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.356182711 (547h59m23.227265756s) [noise=true , address=145.79.227.49:4351 , piece_id=FYLYJ6GRQIMVTNGLLY62Q35M5F4IEQW2EPKVMSDOVJIJJPJJXF3Q, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.356536194 (547h59m23.227619229s) [noise=true , address=145.79.227.19:4092 , piece_id=7UPEHPU5NSBVOPSDNKUAQQZXBHHBS36PKZOR5FBDKLWP4VA4ZSDA, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.356869479 (547h59m23.227952514s) [noise=true , address=145.79.227.33:4882 , piece_id=VJ6XYH77LJSKKBVCCVHHTGFV3WTSCYD4XUKUMSYOWZEGZ2QY4H3A, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.357157427 (547h59m23.228240462s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.357426930 (547h59m23.228509975s) [noise=true , address=145.79.227.40:4067 , piece_id=YOIOW6QPNOMK5ZD5EEWT2FLVGVZZMHNWANUQF4UE3R4FHSOFWSWQ, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.357751889 (547h59m23.228834924s) [noise=true , address=77.90.31.10:4536 , piece_id=HKI6JPRV4D2CBIVEGHRXSCDC2OROZLX5SW3VC7W35L4DWUEITV2Q, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.358953407 (547h59m23.230036442s) [noise=true , address=77.90.31.17:4790 , piece_id=AS6RYIPXJKC6CFSPVX7MLENRTQHI7RZFLTSPHFMIYG5OS7BKZHRA, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.359286491 (547h59m23.230369536s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:24.695915344 (547h59m24.566998389s) [upload=480503]: Next returned nil segment
Dec 28 11:28:24.695941574 (547h59m24.567024619s) [upload=480503]: Waiting for error group managing segments...
Dec 28 11:28:25.041244477 (547h59m24.912327512s) [noise=true , address=193.149.164.143:4700 , piece_id=3FBVJX372QBFVBDXCF5VTCIT2JUYHNXFCZPJ2FN7E34HA4S4WETQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:25.237036778 (547h59m25.108119823s) [noise=true , address=145.79.227.49:4351 , piece_id=FYLYJ6GRQIMVTNGLLY62Q35M5F4IEQW2EPKVMSDOVJIJJPJJXF3Q, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:25.249693558 (547h59m25.120776593s) [noise=true , address=145.79.227.12:4788 , piece_id=KFB2WTRQOSZ75WMHRNCRTPS2FSMIBR27TQ55V62J2PGWLFECB4RA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:25.256819657 (547h59m25.127902692s) [noise=true , address=145.79.227.19:4092 , piece_id=7UPEHPU5NSBVOPSDNKUAQQZXBHHBS36PKZOR5FBDKLWP4VA4ZSDA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:25.511089018 (547h59m25.382172053s) [noise=true , address=45.91.116.13:4170 , piece_id=3KJJFTI5M7MBDSZNTBWSMSJUU7B325XVVLXEH3JKX5N7N4WNMXAA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:25.523026599 (547h59m25.394109634s) [noise=true , address=77.90.31.17:4790 , piece_id=AS6RYIPXJKC6CFSPVX7MLENRTQHI7RZFLTSPHFMIYG5OS7BKZHRA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:25.524217066 (547h59m25.395300101s) [noise=true , address=45.91.116.11:4122 , piece_id=MROLZP4Y2PKVJSUUXCSDQQGVGF7AJADXWEJFFPH6SU2DWTZR443A, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:25.665135729 (547h59m25.536218774s) [noise=true , address=77.90.31.15:4727 , piece_id=KZ74K3R4G2BD73RTJFOKFGQ57SV456NQC4W3NTPT52KBQAU3SWNA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:25.917975247 (547h59m25.789058292s) [noise=true , address=77.90.31.16:4743 , piece_id=NRCRNEOEKXPSKREUZKGGLQHLWG5JTKWCQCRUUHKUHB4HOISVHCFQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:26.147871236 (547h59m26.018954271s) [noise=true , address=145.79.227.30:4796 , piece_id=MUD7JHR4ETJYGG7SSOOWQ5JTKSQYV26X4AWZ6XHEBHLJYE3774CA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:26.173243839 (547h59m26.044326874s) [noise=true , address=145.79.227.44:4209 , piece_id=PKQUKW3FOPJL7BRNFHNLAJ4DSKGMGA73ROILSWEFCT7MR7H5GNNQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:26.289483660 (547h59m26.160566695s) [noise=true , address=77.90.31.10:4536 , piece_id=HKI6JPRV4D2CBIVEGHRXSCDC2OROZLX5SW3VC7W35L4DWUEITV2Q, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:26.448835962 (547h59m26.319919018s) [noise=true , address=145.79.227.40:4067 , piece_id=YOIOW6QPNOMK5ZD5EEWT2FLVGVZZMHNWANUQF4UE3R4FHSOFWSWQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:26.571228809 (547h59m26.442311844s) [noise=true , address=145.79.227.33:4882 , piece_id=VJ6XYH77LJSKKBVCCVHHTGFV3WTSCYD4XUKUMSYOWZEGZ2QY4H3A, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:26.571248396 (547h59m26.442331441s) [seg_pos={0 0}, upload=480503]: Segment reached optimal threshold of 14 pieces.
Dec 28 11:28:26.571429892 (547h59m26.442512927s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NEV2ndHw4fohVUNsfrzwojd8S11TmMST6Sz59GVnXS8vDGhWk): protocol: expected piece hash; context canceled; EOF
Dec 28 11:28:26.571436033 (547h59m26.442519078s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold
Dec 28 11:28:26.571564158 (547h59m26.442647203s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection
Dec 28 11:28:26.571580588 (547h59m26.442663623s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold
Dec 28 11:28:26.571712249 (547h59m26.442795284s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Nw78dd9WPnmCwv5bFQR3VpV39776Pye2QVGSNMTs3zq2BtaGJ): protocol: expected piece hash; context canceled; EOF
Dec 28 11:28:26.571720666 (547h59m26.442803701s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold
Dec 28 11:28:26.571767825 (547h59m26.442850871s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv): protocol: expected piece hash; context canceled; EOF
Dec 28 11:28:26.571774688 (547h59m26.442857723s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold
Dec 28 11:28:26.571792493 (547h59m26.442875538s) [seg_pos={0 0}, upload=480503]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 11:28:26.571882343 (547h59m26.442965378s) [upload=480503]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 11:28:26.571887683 (547h59m26.442970728s) [upload=480503]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 11:28:26.858811294 (547h59m26.729894329s) [upload=480503]: Done uploading segments...
Dec 28 11:28:26.858832874 (547h59m26.72991592s) [upload=480503]: Upload finished. err: <nil>
Duration: 37.782492322s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [....................................X.X.....................] duration:
7.206696111s trace:
https://jaeger.dev.storj.io/trace/72d0191c7bb67437
FAIL (2025-12-28 12:11:56 UTC)
orbiter warning: context deadline exceeded
Duration: 7.206723223s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
35 out of 60 is failed (threshold=6) [XX.....X.X.X..X.XXX..X....XX.XX.XXXXXXXXXX..XXXX...XX.XXX.XX] duration:
5.001138224s trace:
https://jaeger.dev.storj.io/trace/4a7cc66cea50a588
FAIL (2025-12-28 12:11:42 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001160476s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.....................................X.X....................] duration:
19.807943634s trace:
https://jaeger.dev.storj.io/trace/72d0191c7bb67437
FAIL (2025-12-28 12:11:40 UTC)
orbiter warning: context deadline exceeded
Duration: 19.807971568s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
17 out of 60 is failed (threshold=3) [...XX....X.....XX...XXX.......X..XX....X...X.........X..XX.X] duration:
30.002162821s trace:
https://jaeger.dev.storj.io/trace/76cfc3bec30f9bc9
FAIL (2025-12-28 12:11:24 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: DVDBH2I22B4XMYTEFZIZC5ENHCCNY2UYBKGPNXWOR5WU7ZSUEKYQ) context deadline exceeded; piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: N67F6BUDWMY7HQS2U33BXW4VZJ34MO73FY6WQWT5K2HBMPQCYH6Q) context deadline exceeded
======LOGS======
Dec 28 12:10:55.660430003 (548h41m55.531513048s) [upload=481121]: Starting upload
Dec 28 12:10:55.660499014 (548h41m55.531582049s) [upload=481121]: Done starting upload
Dec 28 12:10:55.660735474 (548h41m55.53181852s) [upload=481121]: Uploading segments...
Dec 28 12:10:55.660870462 (548h41m55.531953507s) [upload=481121]: Got next segment. Inline: false
Dec 28 12:10:55.660917210 (548h41m55.532000255s) [upload=481121]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 12:10:55.660924685 (548h41m55.53200773s) [upload=481121]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 12:10:55.909444035 (548h41m55.78052708s) [seg_pos={0 0}, upload=481121]: Begin upload segment...
Dec 28 12:10:55.909501735 (548h41m55.78058478s) [seg_pos={0 0}, upload=481121]: Done begin upload segment.
Dec 28 12:10:55.909539026 (548h41m55.780622071s) [noise=true , address=145.79.227.14:4837 , piece_id=UPBI5KCD7HTHZR6H2QMPS5LUGBCI7RXGBGNBJHG5I7RABK3XJ3FA, seg_pos={0 0}, upload=481121]: Uploading piece...
Dec 28 12:10:55.909568382 (548h41m55.780651427s) [noise=true , address=145.79.227.15:4873 , piece_id=3B6SG27RUZGJRYCT433Y5KCUEDZMYJY4LAE4IA7RTVPNQBMRWWFQ, seg_pos={0 0}, upload=481121]: Uploading piece...
Dec 28 12:10:55.910395015 (548h41m55.78147805s) [noise=true , address=145.79.227.42:4122 , piece_id=33XBPAO6JIVHHRTTHY6HFD5YXVTKROB5HRRDUV4THYKTPU2FZU6A, seg_pos={0 0}, upload=481121]: Uploading piece...
Dec 28 12:10:55.910684147 (548h41m55.781767182s) [noise=true , address=145.79.227.4:4186 , piece_id=PUB3YGQNDHW4AVBUIAPICCTOLRDLGDFA6YQTUYN6PQKX3TRZGUMA, seg_pos={0 0}, upload=481121]: Uploading piece...
Dec 28 12:10:55.910938651 (548h41m55.782021686s) [noise=true , address=145.79.227.45:4231 , piece_id=3D2JZTRHCEER64WWU3FBIJNCYFZOLDLQQYDD5DLOG563X4WML6KA, seg_pos={0 0}, upload=481121]: Uploading piece...
Dec 28 12:10:55.911205319 (548h41m55.782288354s) [noise=true , address=77.90.31.11:4554 , piece_id=N67F6BUDWMY7HQS2U33BXW4VZJ34MO73FY6WQWT5K2HBMPQCYH6Q, seg_pos={0 0}, upload=481121]: Uploading piece...
Dec 28 12:10:55.911457249 (548h41m55.782540284s) [noise=true , address=145.79.227.48:4301 , piece_id=3C2LRAO72K65765CNEPC6IQATH7XKRQ5C2HS4BG6VQYGM5EICS4Q, seg_pos={0 0}, upload=481121]: Uploading piece...
Dec 28 12:10:55.911748282 (548h41m55.782831317s) [noise=true , address=145.79.227.47:4265 , piece_id=AS47QV7TCZVOKCWGQH3E6O5PNEZ4YNC4VBGL7O4QRKXA6ZIGQWMA, seg_pos={0 0}, upload=481121]: Uploading piece...
Dec 28 12:10:55.912092448 (548h41m55.783175493s) [noise=true , address=77.90.31.13:4626 , piece_id=HFJFVBZJD6HYEAX2L6EF7UZVH3UEQ4GQBCDV2ZEYTU7RATF5UXGQ, seg_pos={0 0}, upload=481121]: Uploading piece...
Dec 28 12:10:55.912445791 (548h41m55.783528836s) [noise=true , address=45.91.116.14:4189 , piece_id=KH6OZOJORQK3QJFBAP3NJSBQTJE2MDENC74ZB25YBP45BEPQ7ICA, seg_pos={0 0}, upload=481121]: Uploading piece...
Dec 28 12:10:55.913104785 (548h41m55.784187831s) [noise=true , address=145.79.227.16:4910 , piece_id=GPVKLL4ZYNO6KLZ35R7X5HAMGCSVIIE3H5SS2UEOEJ7CNKDRFQHQ, seg_pos={0 0}, upload=481121]: Uploading piece...
Dec 28 12:10:55.913554612 (548h41m55.784637657s) [noise=true , address=77.90.31.16:4735 , piece_id=COCPUUARNUQ47ND6EIHR332XZJP7JCSVUNFV63QBIQJIZEIMQC2A, seg_pos={0 0}, upload=481121]: Uploading piece...
Dec 28 12:10:55.914012474 (548h41m55.785095519s) [noise=true , address=145.79.227.13:4803 , piece_id=53MLFZJ6QX2FQ6SG733SDERFTIGX6GNORWL5SAXDLZ2QM4TBKINA, seg_pos={0 0}, upload=481121]: Uploading piece...
Dec 28 12:10:55.914463302 (548h41m55.785546478s) [noise=true , address=145.79.227.11:4730 , piece_id=LIFLAF57ZA52IARNDGUVIYHQ6DFJC7VI3JNPY6T37A5YDKOUSRQQ, seg_pos={0 0}, upload=481121]: Uploading piece...
Dec 28 12:10:55.914861211 (548h41m55.785944246s) [noise=true , address=45.91.116.6:4974 , piece_id=DVDBH2I22B4XMYTEFZIZC5ENHCCNY2UYBKGPNXWOR5WU7ZSUEKYQ, seg_pos={0 0}, upload=481121]: Uploading piece...
Dec 28 12:10:55.915264708 (548h41m55.786347813s) [noise=true , address=145.79.227.39:4014 , piece_id=NFTS52EVDWICVF7OZ6DEXPTOAALSQPZLLST2YWEGYQCNIMUI4CPA, seg_pos={0 0}, upload=481121]: Uploading piece...
Dec 28 12:10:57.629256297 (548h41m57.500339342s) [upload=481121]: Next returned nil segment
Dec 28 12:10:57.629273770 (548h41m57.500356805s) [upload=481121]: Waiting for error group managing segments...
Dec 28 12:10:57.917456196 (548h41m57.788539231s) [noise=true , address=145.79.227.39:4014 , piece_id=NFTS52EVDWICVF7OZ6DEXPTOAALSQPZLLST2YWEGYQCNIMUI4CPA, seg_pos={0 0}, upload=481121]: Done uploading piece. err: <nil>
Dec 28 12:10:57.917907185 (548h41m57.78899022s) [noise=true , address=145.79.227.4:4186 , piece_id=PUB3YGQNDHW4AVBUIAPICCTOLRDLGDFA6YQTUYN6PQKX3TRZGUMA, seg_pos={0 0}, upload=481121]: Done uploading piece. err: <nil>
Dec 28 12:10:57.922473289 (548h41m57.793556334s) [noise=true , address=145.79.227.13:4803 , piece_id=53MLFZJ6QX2FQ6SG733SDERFTIGX6GNORWL5SAXDLZ2QM4TBKINA, seg_pos={0 0}, upload=481121]: Done uploading piece. err: <nil>
Dec 28 12:10:57.925552562 (548h41m57.796635607s) [noise=true , address=145.79.227.16:4910 , piece_id=GPVKLL4ZYNO6KLZ35R7X5HAMGCSVIIE3H5SS2UEOEJ7CNKDRFQHQ, seg_pos={0 0}, upload=481121]: Done uploading piece. err: <nil>
Dec 28 12:10:57.932490544 (548h41m57.803573579s) [noise=true , address=145.79.227.14:4837 , piece_id=UPBI5KCD7HTHZR6H2QMPS5LUGBCI7RXGBGNBJHG5I7RABK3XJ3FA, seg_pos={0 0}, upload=481121]: Done uploading piece. err: <nil>
Dec 28 12:10:57.932829238 (548h41m57.803912273s) [noise=true , address=145.79.227.42:4122 , piece_id=33XBPAO6JIVHHRTTHY6HFD5YXVTKROB5HRRDUV4THYKTPU2FZU6A, seg_pos={0 0}, upload=481121]: Done uploading piece. err: <nil>
Dec 28 12:10:57.937710152 (548h41m57.808793197s) [noise=true , address=145.79.227.11:4730 , piece_id=LIFLAF57ZA52IARNDGUVIYHQ6DFJC7VI3JNPY6T37A5YDKOUSRQQ, seg_pos={0 0}, upload=481121]: Done uploading piece. err: <nil>
Dec 28 12:10:57.940392400 (548h41m57.811475445s) [noise=true , address=145.79.227.45:4231 , piece_id=3D2JZTRHCEER64WWU3FBIJNCYFZOLDLQQYDD5DLOG563X4WML6KA, seg_pos={0 0}, upload=481121]: Done uploading piece. err: <nil>
Dec 28 12:10:57.945343066 (548h41m57.816426111s) [noise=true , address=145.79.227.48:4301 , piece_id=3C2LRAO72K65765CNEPC6IQATH7XKRQ5C2HS4BG6VQYGM5EICS4Q, seg_pos={0 0}, upload=481121]: Done uploading piece. err: <nil>
Dec 28 12:10:57.945399194 (548h41m57.816482229s) [noise=true , address=145.79.227.15:4873 , piece_id=3B6SG27RUZGJRYCT433Y5KCUEDZMYJY4LAE4IA7RTVPNQBMRWWFQ, seg_pos={0 0}, upload=481121]: Done uploading piece. err: <nil>
Dec 28 12:10:57.951082956 (548h41m57.822166001s) [noise=true , address=145.79.227.47:4265 , piece_id=AS47QV7TCZVOKCWGQH3E6O5PNEZ4YNC4VBGL7O4QRKXA6ZIGQWMA, seg_pos={0 0}, upload=481121]: Done uploading piece. err: <nil>
Dec 28 12:10:58.132138455 (548h41m58.00322151s) [noise=true , address=45.91.116.6:4974 , piece_id=DVDBH2I22B4XMYTEFZIZC5ENHCCNY2UYBKGPNXWOR5WU7ZSUEKYQ, seg_pos={0 0}, upload=481121]: Done uploading piece. err: <nil>
Dec 28 12:10:58.238648407 (548h41m58.109731442s) [noise=true , address=77.90.31.11:4554 , piece_id=N67F6BUDWMY7HQS2U33BXW4VZJ34MO73FY6WQWT5K2HBMPQCYH6Q, seg_pos={0 0}, upload=481121]: Done uploading piece. err: <nil>
Dec 28 12:10:58.238678594 (548h41m58.109761639s) [seg_pos={0 0}, upload=481121]: Segment reached optimal threshold of 13 pieces.
Dec 28 12:10:58.238881581 (548h41m58.109964626s) [noise=true , address=77.90.31.16:4735 , piece_id=COCPUUARNUQ47ND6EIHR332XZJP7JCSVUNFV63QBIQJIZEIMQC2A, seg_pos={0 0}, upload=481121]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF
Dec 28 12:10:58.238889496 (548h41m58.109972531s) [noise=true , address=77.90.31.16:4735 , piece_id=COCPUUARNUQ47ND6EIHR332XZJP7JCSVUNFV63QBIQJIZEIMQC2A, seg_pos={0 0}, upload=481121]: Long tail cancellation due to optimal threshold
Dec 28 12:10:58.238995468 (548h41m58.110078503s) [noise=true , address=45.91.116.14:4189 , piece_id=KH6OZOJORQK3QJFBAP3NJSBQTJE2MDENC74ZB25YBP45BEPQ7ICA, seg_pos={0 0}, upload=481121]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: write tcp 188.166.225.170:56090->45.91.116.14:4189: use of closed network connection; write tcp 188.166.225.170:56090->45.91.116.14:4189: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56090->45.91.116.14:4189: use of closed network connection
Dec 28 12:10:58.239016368 (548h41m58.110099403s) [noise=true , address=45.91.116.14:4189 , piece_id=KH6OZOJORQK3QJFBAP3NJSBQTJE2MDENC74ZB25YBP45BEPQ7ICA, seg_pos={0 0}, upload=481121]: Long tail cancellation due to optimal threshold
Dec 28 12:10:58.239125055 (548h41m58.11020809s) [noise=true , address=77.90.31.13:4626 , piece_id=HFJFVBZJD6HYEAX2L6EF7UZVH3UEQ4GQBCDV2ZEYTU7RATF5UXGQ, seg_pos={0 0}, upload=481121]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF
Dec 28 12:10:58.239131747 (548h41m58.110214792s) [noise=true , address=77.90.31.13:4626 , piece_id=HFJFVBZJD6HYEAX2L6EF7UZVH3UEQ4GQBCDV2ZEYTU7RATF5UXGQ, seg_pos={0 0}, upload=481121]: Long tail cancellation due to optimal threshold
Dec 28 12:10:58.239152227 (548h41m58.110235262s) [seg_pos={0 0}, upload=481121]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 12:10:58.239217340 (548h41m58.110300375s) [upload=481121]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 12:10:58.239223222 (548h41m58.110306267s) [upload=481121]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 12:10:58.538804292 (548h41m58.409887327s) [upload=481121]: Done uploading segments...
Dec 28 12:10:58.538831865 (548h41m58.4099149s) [upload=481121]: Upload finished. err: <nil>
Duration: 30.002186607s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
34 out of 60 is failed (threshold=6) [.XX.....X.X.X..X.XXX..X....XX.XX.XXXXXXXXXX..XXXX...XX.XXX.X] duration:
5.000795999s trace:
https://jaeger.dev.storj.io/trace/65423a661ede3fb1
FAIL (2025-12-28 12:11:14 UTC)
orbiter warning: context deadline exceeded
Duration: 5.000822539s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [......X..X..................................................] duration:
24.410180128s trace:
https://jaeger.dev.storj.io/trace/307d4901d0d5949
FAIL (2025-12-28 12:11:04 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
ise=true , address=77.90.31.16:4743 , piece_id=NRCRNEOEKXPSKREUZKGGLQHLWG5JTKWCQCRUUHKUHB4HOISVHCFQ, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.355028753 (547h59m23.226111788s) [noise=true , address=145.79.227.30:4796 , piece_id=MUD7JHR4ETJYGG7SSOOWQ5JTKSQYV26X4AWZ6XHEBHLJYE3774CA, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.355039954 (547h59m23.226122989s) [noise=true , address=45.91.116.11:4122 , piece_id=MROLZP4Y2PKVJSUUXCSDQQGVGF7AJADXWEJFFPH6SU2DWTZR443A, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.355282126 (547h59m23.226365171s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.355390983 (547h59m23.226474018s) [noise=true , address=193.149.164.143:4700 , piece_id=3FBVJX372QBFVBDXCF5VTCIT2JUYHNXFCZPJ2FN7E34HA4S4WETQ, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.355527633 (547h59m23.226610668s) [noise=true , address=77.90.31.15:4727 , piece_id=KZ74K3R4G2BD73RTJFOKFGQ57SV456NQC4W3NTPT52KBQAU3SWNA, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.355648985 (547h59m23.22673202s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.355877279 (547h59m23.226960324s) [noise=true , address=145.79.227.12:4788 , piece_id=KFB2WTRQOSZ75WMHRNCRTPS2FSMIBR27TQ55V62J2PGWLFECB4RA, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.356182711 (547h59m23.227265756s) [noise=true , address=145.79.227.49:4351 , piece_id=FYLYJ6GRQIMVTNGLLY62Q35M5F4IEQW2EPKVMSDOVJIJJPJJXF3Q, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.356536194 (547h59m23.227619229s) [noise=true , address=145.79.227.19:4092 , piece_id=7UPEHPU5NSBVOPSDNKUAQQZXBHHBS36PKZOR5FBDKLWP4VA4ZSDA, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.356869479 (547h59m23.227952514s) [noise=true , address=145.79.227.33:4882 , piece_id=VJ6XYH77LJSKKBVCCVHHTGFV3WTSCYD4XUKUMSYOWZEGZ2QY4H3A, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.357157427 (547h59m23.228240462s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.357426930 (547h59m23.228509975s) [noise=true , address=145.79.227.40:4067 , piece_id=YOIOW6QPNOMK5ZD5EEWT2FLVGVZZMHNWANUQF4UE3R4FHSOFWSWQ, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.357751889 (547h59m23.228834924s) [noise=true , address=77.90.31.10:4536 , piece_id=HKI6JPRV4D2CBIVEGHRXSCDC2OROZLX5SW3VC7W35L4DWUEITV2Q, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.358953407 (547h59m23.230036442s) [noise=true , address=77.90.31.17:4790 , piece_id=AS6RYIPXJKC6CFSPVX7MLENRTQHI7RZFLTSPHFMIYG5OS7BKZHRA, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.359286491 (547h59m23.230369536s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:24.695915344 (547h59m24.566998389s) [upload=480503]: Next returned nil segment
Dec 28 11:28:24.695941574 (547h59m24.567024619s) [upload=480503]: Waiting for error group managing segments...
Dec 28 11:28:25.041244477 (547h59m24.912327512s) [noise=true , address=193.149.164.143:4700 , piece_id=3FBVJX372QBFVBDXCF5VTCIT2JUYHNXFCZPJ2FN7E34HA4S4WETQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:25.237036778 (547h59m25.108119823s) [noise=true , address=145.79.227.49:4351 , piece_id=FYLYJ6GRQIMVTNGLLY62Q35M5F4IEQW2EPKVMSDOVJIJJPJJXF3Q, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:25.249693558 (547h59m25.120776593s) [noise=true , address=145.79.227.12:4788 , piece_id=KFB2WTRQOSZ75WMHRNCRTPS2FSMIBR27TQ55V62J2PGWLFECB4RA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:25.256819657 (547h59m25.127902692s) [noise=true , address=145.79.227.19:4092 , piece_id=7UPEHPU5NSBVOPSDNKUAQQZXBHHBS36PKZOR5FBDKLWP4VA4ZSDA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:25.511089018 (547h59m25.382172053s) [noise=true , address=45.91.116.13:4170 , piece_id=3KJJFTI5M7MBDSZNTBWSMSJUU7B325XVVLXEH3JKX5N7N4WNMXAA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:25.523026599 (547h59m25.394109634s) [noise=true , address=77.90.31.17:4790 , piece_id=AS6RYIPXJKC6CFSPVX7MLENRTQHI7RZFLTSPHFMIYG5OS7BKZHRA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:25.524217066 (547h59m25.395300101s) [noise=true , address=45.91.116.11:4122 , piece_id=MROLZP4Y2PKVJSUUXCSDQQGVGF7AJADXWEJFFPH6SU2DWTZR443A, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:25.665135729 (547h59m25.536218774s) [noise=true , address=77.90.31.15:4727 , piece_id=KZ74K3R4G2BD73RTJFOKFGQ57SV456NQC4W3NTPT52KBQAU3SWNA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:25.917975247 (547h59m25.789058292s) [noise=true , address=77.90.31.16:4743 , piece_id=NRCRNEOEKXPSKREUZKGGLQHLWG5JTKWCQCRUUHKUHB4HOISVHCFQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:26.147871236 (547h59m26.018954271s) [noise=true , address=145.79.227.30:4796 , piece_id=MUD7JHR4ETJYGG7SSOOWQ5JTKSQYV26X4AWZ6XHEBHLJYE3774CA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:26.173243839 (547h59m26.044326874s) [noise=true , address=145.79.227.44:4209 , piece_id=PKQUKW3FOPJL7BRNFHNLAJ4DSKGMGA73ROILSWEFCT7MR7H5GNNQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:26.289483660 (547h59m26.160566695s) [noise=true , address=77.90.31.10:4536 , piece_id=HKI6JPRV4D2CBIVEGHRXSCDC2OROZLX5SW3VC7W35L4DWUEITV2Q, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:26.448835962 (547h59m26.319919018s) [noise=true , address=145.79.227.40:4067 , piece_id=YOIOW6QPNOMK5ZD5EEWT2FLVGVZZMHNWANUQF4UE3R4FHSOFWSWQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:26.571228809 (547h59m26.442311844s) [noise=true , address=145.79.227.33:4882 , piece_id=VJ6XYH77LJSKKBVCCVHHTGFV3WTSCYD4XUKUMSYOWZEGZ2QY4H3A, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:26.571248396 (547h59m26.442331441s) [seg_pos={0 0}, upload=480503]: Segment reached optimal threshold of 14 pieces.
Dec 28 11:28:26.571429892 (547h59m26.442512927s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NEV2ndHw4fohVUNsfrzwojd8S11TmMST6Sz59GVnXS8vDGhWk): protocol: expected piece hash; context canceled; EOF
Dec 28 11:28:26.571436033 (547h59m26.442519078s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold
Dec 28 11:28:26.571564158 (547h59m26.442647203s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection
Dec 28 11:28:26.571580588 (547h59m26.442663623s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold
Dec 28 11:28:26.571712249 (547h59m26.442795284s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Nw78dd9WPnmCwv5bFQR3VpV39776Pye2QVGSNMTs3zq2BtaGJ): protocol: expected piece hash; context canceled; EOF
Dec 28 11:28:26.571720666 (547h59m26.442803701s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold
Dec 28 11:28:26.571767825 (547h59m26.442850871s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv): protocol: expected piece hash; context canceled; EOF
Dec 28 11:28:26.571774688 (547h59m26.442857723s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold
Dec 28 11:28:26.571792493 (547h59m26.442875538s) [seg_pos={0 0}, upload=480503]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 11:28:26.571882343 (547h59m26.442965378s) [upload=480503]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 11:28:26.571887683 (547h59m26.442970728s) [upload=480503]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 11:28:26.858811294 (547h59m26.729894329s) [upload=480503]: Done uploading segments...
Dec 28 11:28:26.858832874 (547h59m26.72991592s) [upload=480503]: Upload finished. err: <nil>
Duration: 24.410210144s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [......................................X.X...................] duration:
19.661682957s trace:
https://jaeger.dev.storj.io/trace/72d0191c7bb67437
FAIL (2025-12-28 12:10:49 UTC)
orbiter warning: context deadline exceeded
Duration: 19.661701943s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
16 out of 60 is failed (threshold=3) [....XX....X.....XX...XXX.......X..XX....X...X.........X..XX.] duration:
5.751019085s trace:
https://jaeger.dev.storj.io/trace/7a41828436e9e84d
FAIL (2025-12-28 12:10:34 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: YCYG5UQLXEGDJ2MKFLPRFPHHH2MULXBUSOSQDJLHTKGNEGTA3DHA) context deadline exceeded; piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: BJCMK2ZGBJZZIKSSQ3WBRPC5JFZID63LIXUDQWOTREXVZTFESHXQ) context deadline exceeded
======LOGS======
Dec 28 12:09:49.270068774 (548h40m49.14115182s) [upload=481105]: Starting upload
Dec 28 12:09:49.270226215 (548h40m49.14130927s) [upload=481105]: Done starting upload
Dec 28 12:09:49.270328359 (548h40m49.141411414s) [upload=481105]: Uploading segments...
Dec 28 12:09:49.270617760 (548h40m49.141700825s) [upload=481105]: Got next segment. Inline: false
Dec 28 12:09:49.270663156 (548h40m49.141746191s) [upload=481105]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 12:09:49.270668356 (548h40m49.141751401s) [upload=481105]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 12:09:49.520118369 (548h40m49.391201404s) [seg_pos={0 0}, upload=481105]: Begin upload segment...
Dec 28 12:09:49.520177161 (548h40m49.391260196s) [seg_pos={0 0}, upload=481105]: Done begin upload segment.
Dec 28 12:09:49.520241173 (548h40m49.391324208s) [noise=true , address=145.79.227.10:4401 , piece_id=I5JFACSAT6S42NPJABYG6Z6SDMA5EKCKAYULGOKVVPMCEUJ5KPDQ, seg_pos={0 0}, upload=481105]: Uploading piece...
Dec 28 12:09:49.520257073 (548h40m49.391340108s) [noise=true , address=145.79.227.39:4014 , piece_id=XR3FOUTHEJFJSSWXVJPZYHQL5AREBGZIK2JDK7WQMQVKW3PJIHGQ, seg_pos={0 0}, upload=481105]: Uploading piece...
Dec 28 12:09:49.521158971 (548h40m49.392242016s) [noise=true , address=145.79.227.38:4978 , piece_id=SA5JDC67HD2BTS5CLE7QKKPRJM3XTOBTF3LMU3HM7ZGLEG6LX6OQ, seg_pos={0 0}, upload=481105]: Uploading piece...
Dec 28 12:09:49.521858913 (548h40m49.392941958s) [noise=true , address=145.79.227.6:4259 , piece_id=3F34F6D6GDDMVUK4RFJU4YVQOAWVD2SK7NVRTYJWRVPJI4QK4AJQ, seg_pos={0 0}, upload=481105]: Uploading piece...
Dec 28 12:09:49.522054536 (548h40m49.393137571s) [noise=true , address=145.79.227.13:4804 , piece_id=ORFCBQO6QUV3CZUMZM2HGYR4TKKA7AERELID3R32UCUVPAIZUZLQ, seg_pos={0 0}, upload=481105]: Uploading piece...
Dec 28 12:09:49.522557123 (548h40m49.393640168s) [noise=true , address=145.79.227.41:4085 , piece_id=ZATZ7UELFMFHW2ROX4QKQ6KYONPGSN54BNMXPU367WYTTM4FNUUA, seg_pos={0 0}, upload=481105]: Uploading piece...
Dec 28 12:09:49.522582752 (548h40m49.393665787s) [noise=true , address=77.90.31.15:4700 , piece_id=YCYG5UQLXEGDJ2MKFLPRFPHHH2MULXBUSOSQDJLHTKGNEGTA3DHA, seg_pos={0 0}, upload=481105]: Uploading piece...
Dec 28 12:09:49.522637476 (548h40m49.393720511s) [noise=true , address=45.91.116.12:4226 , piece_id=VR4EOECAMZZ2LZFM35ABL7PHX2PGEOHHZYPAHUOUJER5VAQM6U7Q, seg_pos={0 0}, upload=481105]: Uploading piece...
Dec 28 12:09:49.522665911 (548h40m49.393748946s) [noise=true , address=45.91.116.5:4298 , piece_id=BJCMK2ZGBJZZIKSSQ3WBRPC5JFZID63LIXUDQWOTREXVZTFESHXQ, seg_pos={0 0}, upload=481105]: Uploading piece...
Dec 28 12:09:49.523225767 (548h40m49.394308802s) [noise=true , address=145.79.227.33:4870 , piece_id=I2GJF5Q4NHCTY6TZKA3N6N7UNW7H2R6VEHO5IYFCH4ED5BRUZDOQ, seg_pos={0 0}, upload=481105]: Uploading piece...
Dec 28 12:09:49.523275331 (548h40m49.394358366s) [noise=true , address=45.91.116.8:4048 , piece_id=HQP7DVTFF6I5DGMULU52NPJVS4G4GCMO2N5DNMCNRPWES5ECRGYA, seg_pos={0 0}, upload=481105]: Uploading piece...
Dec 28 12:09:49.523951017 (548h40m49.395034052s) [noise=true , address=145.79.227.46:4601 , piece_id=KPM3DJQ5TGGHOUR5PSQFXMBO5Z3GYO55D546MXBTGZHAXYFWH7VQ, seg_pos={0 0}, upload=481105]: Uploading piece...
Dec 28 12:09:49.524139847 (548h40m49.395222892s) [noise=true , address=145.79.227.19:4077 , piece_id=42TZ5LD55OU53VTGX7JDXR37GDDUMU6CC5M3XIEMODUFU7JX2C2A, seg_pos={0 0}, upload=481105]: Uploading piece...
Dec 28 12:09:49.524453635 (548h40m49.39553668s) [noise=true , address=145.79.227.34:4905 , piece_id=KUEC2YKSISPNX5TSIRDVIYONSSNQOS6JUOMWBOGE4Y4JS3ZXFFVQ, seg_pos={0 0}, upload=481105]: Uploading piece...
Dec 28 12:09:49.524775938 (548h40m49.395858983s) [noise=true , address=145.79.227.17:4972 , piece_id=PTUQLJCFQEHSVZWPYRKZ75GDZEL3ULYEQG5VWNY5S3CQHZQBS6MQ, seg_pos={0 0}, upload=481105]: Uploading piece...
Dec 28 12:09:49.525974010 (548h40m49.397057055s) [noise=true , address=145.79.227.2:4115 , piece_id=W2DWG57BIZTNTZ6FFL7PJVB5LFIT6JGEIRKDQPF5TCHABCP2NXLQ, seg_pos={0 0}, upload=481105]: Uploading piece...
Dec 28 12:09:51.245739912 (548h40m51.116822957s) [upload=481105]: Next returned nil segment
Dec 28 12:09:51.245758386 (548h40m51.116841422s) [upload=481105]: Waiting for error group managing segments...
Dec 28 12:09:51.523508388 (548h40m51.394591433s) [noise=true , address=145.79.227.39:4014 , piece_id=XR3FOUTHEJFJSSWXVJPZYHQL5AREBGZIK2JDK7WQMQVKW3PJIHGQ, seg_pos={0 0}, upload=481105]: Done uploading piece. err: <nil>
Dec 28 12:09:51.534635245 (548h40m51.40571828s) [noise=true , address=145.79.227.13:4804 , piece_id=ORFCBQO6QUV3CZUMZM2HGYR4TKKA7AERELID3R32UCUVPAIZUZLQ, seg_pos={0 0}, upload=481105]: Done uploading piece. err: <nil>
Dec 28 12:09:51.535168901 (548h40m51.406251936s) [noise=true , address=145.79.227.46:4601 , piece_id=KPM3DJQ5TGGHOUR5PSQFXMBO5Z3GYO55D546MXBTGZHAXYFWH7VQ, seg_pos={0 0}, upload=481105]: Done uploading piece. err: <nil>
Dec 28 12:09:51.550688613 (548h40m51.421771648s) [noise=true , address=145.79.227.10:4401 , piece_id=I5JFACSAT6S42NPJABYG6Z6SDMA5EKCKAYULGOKVVPMCEUJ5KPDQ, seg_pos={0 0}, upload=481105]: Done uploading piece. err: <nil>
Dec 28 12:09:51.554403737 (548h40m51.425486782s) [noise=true , address=145.79.227.6:4259 , piece_id=3F34F6D6GDDMVUK4RFJU4YVQOAWVD2SK7NVRTYJWRVPJI4QK4AJQ, seg_pos={0 0}, upload=481105]: Done uploading piece. err: <nil>
Dec 28 12:09:51.554900222 (548h40m51.425983267s) [noise=true , address=145.79.227.38:4978 , piece_id=SA5JDC67HD2BTS5CLE7QKKPRJM3XTOBTF3LMU3HM7ZGLEG6LX6OQ, seg_pos={0 0}, upload=481105]: Done uploading piece. err: <nil>
Dec 28 12:09:51.566910011 (548h40m51.437993046s) [noise=true , address=145.79.227.2:4115 , piece_id=W2DWG57BIZTNTZ6FFL7PJVB5LFIT6JGEIRKDQPF5TCHABCP2NXLQ, seg_pos={0 0}, upload=481105]: Done uploading piece. err: <nil>
Dec 28 12:09:51.570621960 (548h40m51.441705005s) [noise=true , address=145.79.227.17:4972 , piece_id=PTUQLJCFQEHSVZWPYRKZ75GDZEL3ULYEQG5VWNY5S3CQHZQBS6MQ, seg_pos={0 0}, upload=481105]: Done uploading piece. err: <nil>
Dec 28 12:09:51.575250041 (548h40m51.446333086s) [noise=true , address=145.79.227.19:4077 , piece_id=42TZ5LD55OU53VTGX7JDXR37GDDUMU6CC5M3XIEMODUFU7JX2C2A, seg_pos={0 0}, upload=481105]: Done uploading piece. err: <nil>
Dec 28 12:09:51.575912653 (548h40m51.446995698s) [noise=true , address=145.79.227.41:4085 , piece_id=ZATZ7UELFMFHW2ROX4QKQ6KYONPGSN54BNMXPU367WYTTM4FNUUA, seg_pos={0 0}, upload=481105]: Done uploading piece. err: <nil>
Dec 28 12:09:51.731049258 (548h40m51.602132303s) [noise=true , address=77.90.31.15:4700 , piece_id=YCYG5UQLXEGDJ2MKFLPRFPHHH2MULXBUSOSQDJLHTKGNEGTA3DHA, seg_pos={0 0}, upload=481105]: Done uploading piece. err: <nil>
Dec 28 12:09:51.762085327 (548h40m51.633168362s) [noise=true , address=45.91.116.5:4298 , piece_id=BJCMK2ZGBJZZIKSSQ3WBRPC5JFZID63LIXUDQWOTREXVZTFESHXQ, seg_pos={0 0}, upload=481105]: Done uploading piece. err: <nil>
Dec 28 12:09:51.916106017 (548h40m51.787189052s) [noise=true , address=45.91.116.12:4226 , piece_id=VR4EOECAMZZ2LZFM35ABL7PHX2PGEOHHZYPAHUOUJER5VAQM6U7Q, seg_pos={0 0}, upload=481105]: Done uploading piece. err: <nil>
Dec 28 12:09:51.916126285 (548h40m51.78720932s) [seg_pos={0 0}, upload=481105]: Segment reached optimal threshold of 13 pieces.
Dec 28 12:09:51.916415896 (548h40m51.787498931s) [noise=true , address=145.79.227.34:4905 , piece_id=KUEC2YKSISPNX5TSIRDVIYONSSNQOS6JUOMWBOGE4Y4JS3ZXFFVQ, seg_pos={0 0}, upload=481105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF
Dec 28 12:09:51.916422991 (548h40m51.787506026s) [noise=true , address=145.79.227.34:4905 , piece_id=KUEC2YKSISPNX5TSIRDVIYONSSNQOS6JUOMWBOGE4Y4JS3ZXFFVQ, seg_pos={0 0}, upload=481105]: Long tail cancellation due to optimal threshold
Dec 28 12:09:51.916522380 (548h40m51.787605415s) [noise=true , address=145.79.227.33:4870 , piece_id=I2GJF5Q4NHCTY6TZKA3N6N7UNW7H2R6VEHO5IYFCH4ED5BRUZDOQ, seg_pos={0 0}, upload=481105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF
Dec 28 12:09:51.916527249 (548h40m51.787610294s) [noise=true , address=145.79.227.33:4870 , piece_id=I2GJF5Q4NHCTY6TZKA3N6N7UNW7H2R6VEHO5IYFCH4ED5BRUZDOQ, seg_pos={0 0}, upload=481105]: Long tail cancellation due to optimal threshold
Dec 28 12:09:51.916831788 (548h40m51.787914823s) [noise=true , address=45.91.116.8:4048 , piece_id=HQP7DVTFF6I5DGMULU52NPJVS4G4GCMO2N5DNMCNRPWES5ECRGYA, seg_pos={0 0}, upload=481105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: write tcp 188.166.225.170:42952->45.91.116.8:4048: use of closed network connection; context canceled
Dec 28 12:09:51.916838782 (548h40m51.787921817s) [noise=true , address=45.91.116.8:4048 , piece_id=HQP7DVTFF6I5DGMULU52NPJVS4G4GCMO2N5DNMCNRPWES5ECRGYA, seg_pos={0 0}, upload=481105]: Long tail cancellation due to optimal threshold
Dec 28 12:09:51.916854181 (548h40m51.787937216s) [seg_pos={0 0}, upload=481105]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 12:09:51.916921119 (548h40m51.788004154s) [upload=481105]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 12:09:51.916925487 (548h40m51.788008522s) [upload=481105]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 12:09:52.213624470 (548h40m52.084707535s) [upload=481105]: Done uploading segments...
Dec 28 12:09:52.213647574 (548h40m52.084730609s) [upload=481105]: Upload finished. err: <nil>
Duration: 5.751039412s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.......................................X.X..................] duration:
16.579008132s trace:
https://jaeger.dev.storj.io/trace/72d0191c7bb67437
FAIL (2025-12-28 12:10:27 UTC)
orbiter warning: context deadline exceeded
Duration: 16.579037747s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
34 out of 60 is failed (threshold=6) [X.XX.....X.X.X..X.XXX..X....XX.XX.XXXXXXXXXX..XXXX...XX.XXX.] duration:
3.659155492s trace:
https://jaeger.dev.storj.io/trace/4a86c9fd753a5d31
FAIL (2025-12-28 12:10:22 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.659181141s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [.......X..X.................................................] duration:
40.042841673s trace:
https://jaeger.dev.storj.io/trace/307d4901d0d5949
FAIL (2025-12-28 12:10:21 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
ise=true , address=77.90.31.16:4743 , piece_id=NRCRNEOEKXPSKREUZKGGLQHLWG5JTKWCQCRUUHKUHB4HOISVHCFQ, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.355028753 (547h59m23.226111788s) [noise=true , address=145.79.227.30:4796 , piece_id=MUD7JHR4ETJYGG7SSOOWQ5JTKSQYV26X4AWZ6XHEBHLJYE3774CA, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.355039954 (547h59m23.226122989s) [noise=true , address=45.91.116.11:4122 , piece_id=MROLZP4Y2PKVJSUUXCSDQQGVGF7AJADXWEJFFPH6SU2DWTZR443A, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.355282126 (547h59m23.226365171s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.355390983 (547h59m23.226474018s) [noise=true , address=193.149.164.143:4700 , piece_id=3FBVJX372QBFVBDXCF5VTCIT2JUYHNXFCZPJ2FN7E34HA4S4WETQ, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.355527633 (547h59m23.226610668s) [noise=true , address=77.90.31.15:4727 , piece_id=KZ74K3R4G2BD73RTJFOKFGQ57SV456NQC4W3NTPT52KBQAU3SWNA, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.355648985 (547h59m23.22673202s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.355877279 (547h59m23.226960324s) [noise=true , address=145.79.227.12:4788 , piece_id=KFB2WTRQOSZ75WMHRNCRTPS2FSMIBR27TQ55V62J2PGWLFECB4RA, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.356182711 (547h59m23.227265756s) [noise=true , address=145.79.227.49:4351 , piece_id=FYLYJ6GRQIMVTNGLLY62Q35M5F4IEQW2EPKVMSDOVJIJJPJJXF3Q, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.356536194 (547h59m23.227619229s) [noise=true , address=145.79.227.19:4092 , piece_id=7UPEHPU5NSBVOPSDNKUAQQZXBHHBS36PKZOR5FBDKLWP4VA4ZSDA, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.356869479 (547h59m23.227952514s) [noise=true , address=145.79.227.33:4882 , piece_id=VJ6XYH77LJSKKBVCCVHHTGFV3WTSCYD4XUKUMSYOWZEGZ2QY4H3A, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.357157427 (547h59m23.228240462s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.357426930 (547h59m23.228509975s) [noise=true , address=145.79.227.40:4067 , piece_id=YOIOW6QPNOMK5ZD5EEWT2FLVGVZZMHNWANUQF4UE3R4FHSOFWSWQ, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.357751889 (547h59m23.228834924s) [noise=true , address=77.90.31.10:4536 , piece_id=HKI6JPRV4D2CBIVEGHRXSCDC2OROZLX5SW3VC7W35L4DWUEITV2Q, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.358953407 (547h59m23.230036442s) [noise=true , address=77.90.31.17:4790 , piece_id=AS6RYIPXJKC6CFSPVX7MLENRTQHI7RZFLTSPHFMIYG5OS7BKZHRA, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:23.359286491 (547h59m23.230369536s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Uploading piece...
Dec 28 11:28:24.695915344 (547h59m24.566998389s) [upload=480503]: Next returned nil segment
Dec 28 11:28:24.695941574 (547h59m24.567024619s) [upload=480503]: Waiting for error group managing segments...
Dec 28 11:28:25.041244477 (547h59m24.912327512s) [noise=true , address=193.149.164.143:4700 , piece_id=3FBVJX372QBFVBDXCF5VTCIT2JUYHNXFCZPJ2FN7E34HA4S4WETQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:25.237036778 (547h59m25.108119823s) [noise=true , address=145.79.227.49:4351 , piece_id=FYLYJ6GRQIMVTNGLLY62Q35M5F4IEQW2EPKVMSDOVJIJJPJJXF3Q, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:25.249693558 (547h59m25.120776593s) [noise=true , address=145.79.227.12:4788 , piece_id=KFB2WTRQOSZ75WMHRNCRTPS2FSMIBR27TQ55V62J2PGWLFECB4RA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:25.256819657 (547h59m25.127902692s) [noise=true , address=145.79.227.19:4092 , piece_id=7UPEHPU5NSBVOPSDNKUAQQZXBHHBS36PKZOR5FBDKLWP4VA4ZSDA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:25.511089018 (547h59m25.382172053s) [noise=true , address=45.91.116.13:4170 , piece_id=3KJJFTI5M7MBDSZNTBWSMSJUU7B325XVVLXEH3JKX5N7N4WNMXAA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:25.523026599 (547h59m25.394109634s) [noise=true , address=77.90.31.17:4790 , piece_id=AS6RYIPXJKC6CFSPVX7MLENRTQHI7RZFLTSPHFMIYG5OS7BKZHRA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:25.524217066 (547h59m25.395300101s) [noise=true , address=45.91.116.11:4122 , piece_id=MROLZP4Y2PKVJSUUXCSDQQGVGF7AJADXWEJFFPH6SU2DWTZR443A, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:25.665135729 (547h59m25.536218774s) [noise=true , address=77.90.31.15:4727 , piece_id=KZ74K3R4G2BD73RTJFOKFGQ57SV456NQC4W3NTPT52KBQAU3SWNA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:25.917975247 (547h59m25.789058292s) [noise=true , address=77.90.31.16:4743 , piece_id=NRCRNEOEKXPSKREUZKGGLQHLWG5JTKWCQCRUUHKUHB4HOISVHCFQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:26.147871236 (547h59m26.018954271s) [noise=true , address=145.79.227.30:4796 , piece_id=MUD7JHR4ETJYGG7SSOOWQ5JTKSQYV26X4AWZ6XHEBHLJYE3774CA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:26.173243839 (547h59m26.044326874s) [noise=true , address=145.79.227.44:4209 , piece_id=PKQUKW3FOPJL7BRNFHNLAJ4DSKGMGA73ROILSWEFCT7MR7H5GNNQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:26.289483660 (547h59m26.160566695s) [noise=true , address=77.90.31.10:4536 , piece_id=HKI6JPRV4D2CBIVEGHRXSCDC2OROZLX5SW3VC7W35L4DWUEITV2Q, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:26.448835962 (547h59m26.319919018s) [noise=true , address=145.79.227.40:4067 , piece_id=YOIOW6QPNOMK5ZD5EEWT2FLVGVZZMHNWANUQF4UE3R4FHSOFWSWQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:26.571228809 (547h59m26.442311844s) [noise=true , address=145.79.227.33:4882 , piece_id=VJ6XYH77LJSKKBVCCVHHTGFV3WTSCYD4XUKUMSYOWZEGZ2QY4H3A, seg_pos={0 0}, upload=480503]: Done uploading piece. err: <nil>
Dec 28 11:28:26.571248396 (547h59m26.442331441s) [seg_pos={0 0}, upload=480503]: Segment reached optimal threshold of 14 pieces.
Dec 28 11:28:26.571429892 (547h59m26.442512927s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NEV2ndHw4fohVUNsfrzwojd8S11TmMST6Sz59GVnXS8vDGhWk): protocol: expected piece hash; context canceled; EOF
Dec 28 11:28:26.571436033 (547h59m26.442519078s) [noise=true , address=77.90.31.19:4851 , piece_id=WAK6SXBKKSOCOUSWNLJZPDKYYVUC3ORDVPFIT2WPOCRRQLRNZNGQ, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold
Dec 28 11:28:26.571564158 (547h59m26.442647203s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58570->64.49.13.169:4705: use of closed network connection
Dec 28 11:28:26.571580588 (547h59m26.442663623s) [noise=true , address=64.49.13.169:4705 , piece_id=ZI2LEMHCJ2EV55RMCVWH4B5BA5Q7IHUSR6GFT53FFDYIUTNYHZIA, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold
Dec 28 11:28:26.571712249 (547h59m26.442795284s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Nw78dd9WPnmCwv5bFQR3VpV39776Pye2QVGSNMTs3zq2BtaGJ): protocol: expected piece hash; context canceled; EOF
Dec 28 11:28:26.571720666 (547h59m26.442803701s) [noise=true , address=145.79.227.34:4918 , piece_id=JJAR353C6QA5ZZKQOHQMVMDZOYOCKRRVYKBMCBUE522ZQF37YYFQ, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold
Dec 28 11:28:26.571767825 (547h59m26.442850871s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WPKeCcN1uEhGjZHAJgCoJMaHxAWouGCdaU1vZwAS7kX22rxcv): protocol: expected piece hash; context canceled; EOF
Dec 28 11:28:26.571774688 (547h59m26.442857723s) [noise=true , address=77.90.31.14:4690 , piece_id=RLNECUL6ZBXMGJ2P4L636RBJKVUTWOOMIQ4IIMOG5ROW5B2AY7SA, seg_pos={0 0}, upload=480503]: Long tail cancellation due to optimal threshold
Dec 28 11:28:26.571792493 (547h59m26.442875538s) [seg_pos={0 0}, upload=480503]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 11:28:26.571882343 (547h59m26.442965378s) [upload=480503]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 11:28:26.571887683 (547h59m26.442970728s) [upload=480503]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 11:28:26.858811294 (547h59m26.729894329s) [upload=480503]: Done uploading segments...
Dec 28 11:28:26.858832874 (547h59m26.72991592s) [upload=480503]: Upload finished. err: <nil>
Duration: 40.042884414s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
16 out of 60 is failed (threshold=3) [.....XX....X.....XX...XXX.......X..XX....X...X.........X..XX] duration:
30.005252847s trace:
https://jaeger.dev.storj.io/trace/7a41828436e9e84d
FAIL (2025-12-28 12:10:18 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: YCYG5UQLXEGDJ2MKFLPRFPHHH2MULXBUSOSQDJLHTKGNEGTA3DHA) context deadline exceeded; piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: BJCMK2ZGBJZZIKSSQ3WBRPC5JFZID63LIXUDQWOTREXVZTFESHXQ) context deadline exceeded
======LOGS======
Dec 28 12:09:49.270068774 (548h40m49.14115182s) [upload=481105]: Starting upload
Dec 28 12:09:49.270226215 (548h40m49.14130927s) [upload=481105]: Done starting upload
Dec 28 12:09:49.270328359 (548h40m49.141411414s) [upload=481105]: Uploading segments...
Dec 28 12:09:49.270617760 (548h40m49.141700825s) [upload=481105]: Got next segment. Inline: false
Dec 28 12:09:49.270663156 (548h40m49.141746191s) [upload=481105]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 12:09:49.270668356 (548h40m49.141751401s) [upload=481105]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 12:09:49.520118369 (548h40m49.391201404s) [seg_pos={0 0}, upload=481105]: Begin upload segment...
Dec 28 12:09:49.520177161 (548h40m49.391260196s) [seg_pos={0 0}, upload=481105]: Done begin upload segment.
Dec 28 12:09:49.520241173 (548h40m49.391324208s) [noise=true , address=145.79.227.10:4401 , piece_id=I5JFACSAT6S42NPJABYG6Z6SDMA5EKCKAYULGOKVVPMCEUJ5KPDQ, seg_pos={0 0}, upload=481105]: Uploading piece...
Dec 28 12:09:49.520257073 (548h40m49.391340108s) [noise=true , address=145.79.227.39:4014 , piece_id=XR3FOUTHEJFJSSWXVJPZYHQL5AREBGZIK2JDK7WQMQVKW3PJIHGQ, seg_pos={0 0}, upload=481105]: Uploading piece...
Dec 28 12:09:49.521158971 (548h40m49.392242016s) [noise=true , address=145.79.227.38:4978 , piece_id=SA5JDC67HD2BTS5CLE7QKKPRJM3XTOBTF3LMU3HM7ZGLEG6LX6OQ, seg_pos={0 0}, upload=481105]: Uploading piece...
Dec 28 12:09:49.521858913 (548h40m49.392941958s) [noise=true , address=145.79.227.6:4259 , piece_id=3F34F6D6GDDMVUK4RFJU4YVQOAWVD2SK7NVRTYJWRVPJI4QK4AJQ, seg_pos={0 0}, upload=481105]: Uploading piece...
Dec 28 12:09:49.522054536 (548h40m49.393137571s) [noise=true , address=145.79.227.13:4804 , piece_id=ORFCBQO6QUV3CZUMZM2HGYR4TKKA7AERELID3R32UCUVPAIZUZLQ, seg_pos={0 0}, upload=481105]: Uploading piece...
Dec 28 12:09:49.522557123 (548h40m49.393640168s) [noise=true , address=145.79.227.41:4085 , piece_id=ZATZ7UELFMFHW2ROX4QKQ6KYONPGSN54BNMXPU367WYTTM4FNUUA, seg_pos={0 0}, upload=481105]: Uploading piece...
Dec 28 12:09:49.522582752 (548h40m49.393665787s) [noise=true , address=77.90.31.15:4700 , piece_id=YCYG5UQLXEGDJ2MKFLPRFPHHH2MULXBUSOSQDJLHTKGNEGTA3DHA, seg_pos={0 0}, upload=481105]: Uploading piece...
Dec 28 12:09:49.522637476 (548h40m49.393720511s) [noise=true , address=45.91.116.12:4226 , piece_id=VR4EOECAMZZ2LZFM35ABL7PHX2PGEOHHZYPAHUOUJER5VAQM6U7Q, seg_pos={0 0}, upload=481105]: Uploading piece...
Dec 28 12:09:49.522665911 (548h40m49.393748946s) [noise=true , address=45.91.116.5:4298 , piece_id=BJCMK2ZGBJZZIKSSQ3WBRPC5JFZID63LIXUDQWOTREXVZTFESHXQ, seg_pos={0 0}, upload=481105]: Uploading piece...
Dec 28 12:09:49.523225767 (548h40m49.394308802s) [noise=true , address=145.79.227.33:4870 , piece_id=I2GJF5Q4NHCTY6TZKA3N6N7UNW7H2R6VEHO5IYFCH4ED5BRUZDOQ, seg_pos={0 0}, upload=481105]: Uploading piece...
Dec 28 12:09:49.523275331 (548h40m49.394358366s) [noise=true , address=45.91.116.8:4048 , piece_id=HQP7DVTFF6I5DGMULU52NPJVS4G4GCMO2N5DNMCNRPWES5ECRGYA, seg_pos={0 0}, upload=481105]: Uploading piece...
Dec 28 12:09:49.523951017 (548h40m49.395034052s) [noise=true , address=145.79.227.46:4601 , piece_id=KPM3DJQ5TGGHOUR5PSQFXMBO5Z3GYO55D546MXBTGZHAXYFWH7VQ, seg_pos={0 0}, upload=481105]: Uploading piece...
Dec 28 12:09:49.524139847 (548h40m49.395222892s) [noise=true , address=145.79.227.19:4077 , piece_id=42TZ5LD55OU53VTGX7JDXR37GDDUMU6CC5M3XIEMODUFU7JX2C2A, seg_pos={0 0}, upload=481105]: Uploading piece...
Dec 28 12:09:49.524453635 (548h40m49.39553668s) [noise=true , address=145.79.227.34:4905 , piece_id=KUEC2YKSISPNX5TSIRDVIYONSSNQOS6JUOMWBOGE4Y4JS3ZXFFVQ, seg_pos={0 0}, upload=481105]: Uploading piece...
Dec 28 12:09:49.524775938 (548h40m49.395858983s) [noise=true , address=145.79.227.17:4972 , piece_id=PTUQLJCFQEHSVZWPYRKZ75GDZEL3ULYEQG5VWNY5S3CQHZQBS6MQ, seg_pos={0 0}, upload=481105]: Uploading piece...
Dec 28 12:09:49.525974010 (548h40m49.397057055s) [noise=true , address=145.79.227.2:4115 , piece_id=W2DWG57BIZTNTZ6FFL7PJVB5LFIT6JGEIRKDQPF5TCHABCP2NXLQ, seg_pos={0 0}, upload=481105]: Uploading piece...
Dec 28 12:09:51.245739912 (548h40m51.116822957s) [upload=481105]: Next returned nil segment
Dec 28 12:09:51.245758386 (548h40m51.116841422s) [upload=481105]: Waiting for error group managing segments...
Dec 28 12:09:51.523508388 (548h40m51.394591433s) [noise=true , address=145.79.227.39:4014 , piece_id=XR3FOUTHEJFJSSWXVJPZYHQL5AREBGZIK2JDK7WQMQVKW3PJIHGQ, seg_pos={0 0}, upload=481105]: Done uploading piece. err: <nil>
Dec 28 12:09:51.534635245 (548h40m51.40571828s) [noise=true , address=145.79.227.13:4804 , piece_id=ORFCBQO6QUV3CZUMZM2HGYR4TKKA7AERELID3R32UCUVPAIZUZLQ, seg_pos={0 0}, upload=481105]: Done uploading piece. err: <nil>
Dec 28 12:09:51.535168901 (548h40m51.406251936s) [noise=true , address=145.79.227.46:4601 , piece_id=KPM3DJQ5TGGHOUR5PSQFXMBO5Z3GYO55D546MXBTGZHAXYFWH7VQ, seg_pos={0 0}, upload=481105]: Done uploading piece. err: <nil>
Dec 28 12:09:51.550688613 (548h40m51.421771648s) [noise=true , address=145.79.227.10:4401 , piece_id=I5JFACSAT6S42NPJABYG6Z6SDMA5EKCKAYULGOKVVPMCEUJ5KPDQ, seg_pos={0 0}, upload=481105]: Done uploading piece. err: <nil>
Dec 28 12:09:51.554403737 (548h40m51.425486782s) [noise=true , address=145.79.227.6:4259 , piece_id=3F34F6D6GDDMVUK4RFJU4YVQOAWVD2SK7NVRTYJWRVPJI4QK4AJQ, seg_pos={0 0}, upload=481105]: Done uploading piece. err: <nil>
Dec 28 12:09:51.554900222 (548h40m51.425983267s) [noise=true , address=145.79.227.38:4978 , piece_id=SA5JDC67HD2BTS5CLE7QKKPRJM3XTOBTF3LMU3HM7ZGLEG6LX6OQ, seg_pos={0 0}, upload=481105]: Done uploading piece. err: <nil>
Dec 28 12:09:51.566910011 (548h40m51.437993046s) [noise=true , address=145.79.227.2:4115 , piece_id=W2DWG57BIZTNTZ6FFL7PJVB5LFIT6JGEIRKDQPF5TCHABCP2NXLQ, seg_pos={0 0}, upload=481105]: Done uploading piece. err: <nil>
Dec 28 12:09:51.570621960 (548h40m51.441705005s) [noise=true , address=145.79.227.17:4972 , piece_id=PTUQLJCFQEHSVZWPYRKZ75GDZEL3ULYEQG5VWNY5S3CQHZQBS6MQ, seg_pos={0 0}, upload=481105]: Done uploading piece. err: <nil>
Dec 28 12:09:51.575250041 (548h40m51.446333086s) [noise=true , address=145.79.227.19:4077 , piece_id=42TZ5LD55OU53VTGX7JDXR37GDDUMU6CC5M3XIEMODUFU7JX2C2A, seg_pos={0 0}, upload=481105]: Done uploading piece. err: <nil>
Dec 28 12:09:51.575912653 (548h40m51.446995698s) [noise=true , address=145.79.227.41:4085 , piece_id=ZATZ7UELFMFHW2ROX4QKQ6KYONPGSN54BNMXPU367WYTTM4FNUUA, seg_pos={0 0}, upload=481105]: Done uploading piece. err: <nil>
Dec 28 12:09:51.731049258 (548h40m51.602132303s) [noise=true , address=77.90.31.15:4700 , piece_id=YCYG5UQLXEGDJ2MKFLPRFPHHH2MULXBUSOSQDJLHTKGNEGTA3DHA, seg_pos={0 0}, upload=481105]: Done uploading piece. err: <nil>
Dec 28 12:09:51.762085327 (548h40m51.633168362s) [noise=true , address=45.91.116.5:4298 , piece_id=BJCMK2ZGBJZZIKSSQ3WBRPC5JFZID63LIXUDQWOTREXVZTFESHXQ, seg_pos={0 0}, upload=481105]: Done uploading piece. err: <nil>
Dec 28 12:09:51.916106017 (548h40m51.787189052s) [noise=true , address=45.91.116.12:4226 , piece_id=VR4EOECAMZZ2LZFM35ABL7PHX2PGEOHHZYPAHUOUJER5VAQM6U7Q, seg_pos={0 0}, upload=481105]: Done uploading piece. err: <nil>
Dec 28 12:09:51.916126285 (548h40m51.78720932s) [seg_pos={0 0}, upload=481105]: Segment reached optimal threshold of 13 pieces.
Dec 28 12:09:51.916415896 (548h40m51.787498931s) [noise=true , address=145.79.227.34:4905 , piece_id=KUEC2YKSISPNX5TSIRDVIYONSSNQOS6JUOMWBOGE4Y4JS3ZXFFVQ, seg_pos={0 0}, upload=481105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF
Dec 28 12:09:51.916422991 (548h40m51.787506026s) [noise=true , address=145.79.227.34:4905 , piece_id=KUEC2YKSISPNX5TSIRDVIYONSSNQOS6JUOMWBOGE4Y4JS3ZXFFVQ, seg_pos={0 0}, upload=481105]: Long tail cancellation due to optimal threshold
Dec 28 12:09:51.916522380 (548h40m51.787605415s) [noise=true , address=145.79.227.33:4870 , piece_id=I2GJF5Q4NHCTY6TZKA3N6N7UNW7H2R6VEHO5IYFCH4ED5BRUZDOQ, seg_pos={0 0}, upload=481105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF
Dec 28 12:09:51.916527249 (548h40m51.787610294s) [noise=true , address=145.79.227.33:4870 , piece_id=I2GJF5Q4NHCTY6TZKA3N6N7UNW7H2R6VEHO5IYFCH4ED5BRUZDOQ, seg_pos={0 0}, upload=481105]: Long tail cancellation due to optimal threshold
Dec 28 12:09:51.916831788 (548h40m51.787914823s) [noise=true , address=45.91.116.8:4048 , piece_id=HQP7DVTFF6I5DGMULU52NPJVS4G4GCMO2N5DNMCNRPWES5ECRGYA, seg_pos={0 0}, upload=481105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: write tcp 188.166.225.170:42952->45.91.116.8:4048: use of closed network connection; context canceled
Dec 28 12:09:51.916838782 (548h40m51.787921817s) [noise=true , address=45.91.116.8:4048 , piece_id=HQP7DVTFF6I5DGMULU52NPJVS4G4GCMO2N5DNMCNRPWES5ECRGYA, seg_pos={0 0}, upload=481105]: Long tail cancellation due to optimal threshold
Dec 28 12:09:51.916854181 (548h40m51.787937216s) [seg_pos={0 0}, upload=481105]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 12:09:51.916921119 (548h40m51.788004154s) [upload=481105]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 12:09:51.916925487 (548h40m51.788008522s) [upload=481105]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 12:09:52.213624470 (548h40m52.084707535s) [upload=481105]: Done uploading segments...
Dec 28 12:09:52.213647574 (548h40m52.084730609s) [upload=481105]: Upload finished. err: <nil>
Duration: 30.005279327s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [........................................X.X.................] duration:
16.455111404s trace:
https://jaeger.dev.storj.io/trace/72d0191c7bb67437
FAIL (2025-12-28 12:09:51 UTC)
orbiter warning: context deadline exceeded
Duration: 16.455144846s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
34 out of 60 is failed (threshold=6) [.X.XX.....X.X.X..X.XXX..X....XX.XX.XXXXXXXXXX..XXXX...XX.XXX] duration:
5.000439041s trace:
https://jaeger.dev.storj.io/trace/4a86c9fd753a5d31
FAIL (2025-12-28 12:09:45 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.00046488s
core
uplink
us-select-1
us1-placement-20
|