Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
44 out of 60 is failed (threshold=6) [XXXX.XX..XXX.X.XX.XXX.XX.XXXXXX..XXXXXX...XXXX..XXXXXXXXXX.X] duration:
5.001053143s trace:
https://jaeger.dev.storj.io/trace/801cc079dfd52e2
FAIL (2026-01-05 09:22:26 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001096406s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [....X.............................................X.........] duration:
18.359437364s trace:
https://jaeger.dev.storj.io/trace/35e6dfe601545a69
FAIL (2026-01-05 09:22:24 UTC)
orbiter warning: context deadline exceeded
Duration: 18.359466982s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
27 out of 60 is failed (threshold=3) [XXXXXX.XXX...X....XX.X.X....XX....X..XX.XX......X.X.X..XXX..] duration:
15.542859461s trace:
https://jaeger.dev.storj.io/trace/6bc33b9da6112cd
FAIL (2026-01-05 09:22:19 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
7A, seg_pos={0 0}, upload=645638]: Uploading piece...
Jan 5 09:20:39.819337437 (737h51m39.690420482s) [noise=true , address=145.79.227.43:4191 , piece_id=BSM6NWJPFJZGSENPIRPHFJKOS2S6EEEFSNCAT3JVZ2DBQDM4I5JQ, seg_pos={0 0}, upload=645638]: Uploading piece...
Jan 5 09:20:39.819876133 (737h51m39.690959168s) [noise=true , address=145.79.227.32:4860 , piece_id=SAHLEMXIG63JDHFTXKBFDGUGFIKVTTGHA5APZ7VYW7RW62LTE5UQ, seg_pos={0 0}, upload=645638]: Uploading piece...
Jan 5 09:20:39.820218706 (737h51m39.691301751s) [noise=true , address=77.90.31.17:4786 , piece_id=LAULWIVBRGXC6S6VX2U5LTLPXCMAE7KU2HCOXVJ4LFGQJLOLKNQA, seg_pos={0 0}, upload=645638]: Uploading piece...
Jan 5 09:20:39.820761499 (737h51m39.691844544s) [noise=true , address=145.79.227.10:4435 , piece_id=LAVCQ6YZU6JLYGA4CB6FLVYDK2JXJFFL3FE6N7JZZQDA5KEKAYRQ, seg_pos={0 0}, upload=645638]: Uploading piece...
Jan 5 09:20:39.821121004 (737h51m39.692204039s) [noise=true , address=64.49.13.155:4918 , piece_id=WAMEGNFI6LPBIRJ5Z2HM5UED6Y7DPUC4FC56GFUM5MU3WGDNBVSA, seg_pos={0 0}, upload=645638]: Uploading piece...
Jan 5 09:20:39.821498272 (737h51m39.692581307s) [noise=true , address=145.79.227.33:4904 , piece_id=7FUOLIDMDFBETDXDYWDWMFR77GGZM7BR4Q36J6EX4AO4WJWJXRTQ, seg_pos={0 0}, upload=645638]: Uploading piece...
Jan 5 09:20:39.821944962 (737h51m39.693027997s) [noise=true , address=145.79.227.34:4932 , piece_id=CIBME66KBWA2REKAINEZQUGDZILM6OD36ODGAYPTYEKZ7METFFWA, seg_pos={0 0}, upload=645638]: Uploading piece...
Jan 5 09:20:39.822307192 (737h51m39.693390237s) [noise=true , address=45.91.116.9:4281 , piece_id=4SWY2HIJQQPDBY4PFKR7YFY3TTZ2IPSGN6YAKOYB3WSVKVDJODYA, seg_pos={0 0}, upload=645638]: Uploading piece...
Jan 5 09:20:39.822810150 (737h51m39.693893185s) [noise=true , address=145.79.227.40:4083 , piece_id=2ROGBVVX2LZACQRKL2CD4TOHOHYBJS4ZPPE4WXFAGJAUDLZPBXLQ, seg_pos={0 0}, upload=645638]: Uploading piece...
Jan 5 09:20:39.824181141 (737h51m39.695264176s) [noise=true , address=145.79.227.49:4344 , piece_id=OMH76SOXDDSLK4GNBVSGKRH6JIGZM7AVEHMX3JUZ5ILUTYECXSWQ, seg_pos={0 0}, upload=645638]: Uploading piece...
Jan 5 09:20:39.825441410 (737h51m39.696524446s) [noise=true , address=77.90.31.13:4631 , piece_id=KONA5PHEO3WERYMHTSRPHWGSGVS5ARNQEJIYEMRZGVQFRJ5GMALQ, seg_pos={0 0}, upload=645638]: Uploading piece...
Jan 5 09:20:39.826851755 (737h51m39.697934791s) [noise=true , address=77.90.31.11:4563 , piece_id=FSH2HF4HWZE6ADTPMJ3KULEVALE46TROVOVANXT7CLPJBNUX64EA, seg_pos={0 0}, upload=645638]: Uploading piece...
Jan 5 09:20:39.828547344 (737h51m39.69963038s) [noise=true , address=77.90.31.21:4886 , piece_id=FBJM277VXOWHLBC77QZRRFU5VSVOOEETLPRDVMMQ4FMXZUQBYH6A, seg_pos={0 0}, upload=645638]: Uploading piece...
Jan 5 09:20:39.830804523 (737h51m39.701887568s) [noise=true , address=145.79.227.44:4211 , piece_id=JAPLQHRW4D66MI3GIQUGTKXDGKHBT77QTGMSYDQJTAJIX4JJUZFQ, seg_pos={0 0}, upload=645638]: Uploading piece...
Jan 5 09:20:39.832429627 (737h51m39.703512673s) [noise=true , address=193.149.164.143:4703 , piece_id=ONPBUN2W67WQLBGU4KRH3YZCAJGK2XGGINDF77RPITG6DUYALTRA, seg_pos={0 0}, upload=645638]: Uploading piece...
Jan 5 09:20:41.249821392 (737h51m41.120904427s) [upload=645638]: Next returned nil segment
Jan 5 09:20:41.249841230 (737h51m41.120924265s) [upload=645638]: Waiting for error group managing segments...
Jan 5 09:20:41.690593524 (737h51m41.561676569s) [noise=true , address=145.79.227.49:4344 , piece_id=OMH76SOXDDSLK4GNBVSGKRH6JIGZM7AVEHMX3JUZ5ILUTYECXSWQ, seg_pos={0 0}, upload=645638]: Done uploading piece. err: <nil>
Jan 5 09:20:41.702709104 (737h51m41.57379215s) [noise=true , address=145.79.227.10:4435 , piece_id=LAVCQ6YZU6JLYGA4CB6FLVYDK2JXJFFL3FE6N7JZZQDA5KEKAYRQ, seg_pos={0 0}, upload=645638]: Done uploading piece. err: <nil>
Jan 5 09:20:41.705922394 (737h51m41.577005439s) [noise=true , address=145.79.227.32:4860 , piece_id=SAHLEMXIG63JDHFTXKBFDGUGFIKVTTGHA5APZ7VYW7RW62LTE5UQ, seg_pos={0 0}, upload=645638]: Done uploading piece. err: <nil>
Jan 5 09:20:42.008187135 (737h51m41.87927017s) [noise=true , address=77.90.31.13:4631 , piece_id=KONA5PHEO3WERYMHTSRPHWGSGVS5ARNQEJIYEMRZGVQFRJ5GMALQ, seg_pos={0 0}, upload=645638]: Done uploading piece. err: <nil>
Jan 5 09:20:42.571971402 (737h51m42.443054437s) [noise=true , address=145.79.227.33:4904 , piece_id=7FUOLIDMDFBETDXDYWDWMFR77GGZM7BR4Q36J6EX4AO4WJWJXRTQ, seg_pos={0 0}, upload=645638]: Done uploading piece. err: <nil>
Jan 5 09:20:42.590678753 (737h51m42.461761799s) [noise=true , address=145.79.227.34:4932 , piece_id=CIBME66KBWA2REKAINEZQUGDZILM6OD36ODGAYPTYEKZ7METFFWA, seg_pos={0 0}, upload=645638]: Done uploading piece. err: <nil>
Jan 5 09:20:42.626844268 (737h51m42.497927303s) [noise=true , address=145.79.227.43:4191 , piece_id=BSM6NWJPFJZGSENPIRPHFJKOS2S6EEEFSNCAT3JVZ2DBQDM4I5JQ, seg_pos={0 0}, upload=645638]: Done uploading piece. err: <nil>
Jan 5 09:20:42.636866652 (737h51m42.507949687s) [noise=true , address=145.79.227.35:4502 , piece_id=DG7ALRAPMPQMTXXES5IUBHKQWLWFXSWSJ37BFJUFEQJAGPKZ25IA, seg_pos={0 0}, upload=645638]: Done uploading piece. err: <nil>
Jan 5 09:20:42.693104690 (737h51m42.564187735s) [noise=true , address=145.79.227.40:4083 , piece_id=2ROGBVVX2LZACQRKL2CD4TOHOHYBJS4ZPPE4WXFAGJAUDLZPBXLQ, seg_pos={0 0}, upload=645638]: Done uploading piece. err: <nil>
Jan 5 09:20:42.960096876 (737h51m42.831179911s) [noise=true , address=193.149.164.143:4703 , piece_id=ONPBUN2W67WQLBGU4KRH3YZCAJGK2XGGINDF77RPITG6DUYALTRA, seg_pos={0 0}, upload=645638]: Done uploading piece. err: <nil>
Jan 5 09:20:43.059872435 (737h51m42.93095547s) [noise=true , address=145.79.227.30:4784 , piece_id=HUQUN57CKDGPGD7234XNDOQJH4YGNUJRS27QJFH4IJ4Z6XRNONBA, seg_pos={0 0}, upload=645638]: Done uploading piece. err: <nil>
Jan 5 09:20:43.092756041 (737h51m42.963839076s) [noise=true , address=145.79.227.44:4211 , piece_id=JAPLQHRW4D66MI3GIQUGTKXDGKHBT77QTGMSYDQJTAJIX4JJUZFQ, seg_pos={0 0}, upload=645638]: Done uploading piece. err: <nil>
Jan 5 09:20:43.274893281 (737h51m43.145976316s) [noise=true , address=77.90.31.21:4886 , piece_id=FBJM277VXOWHLBC77QZRRFU5VSVOOEETLPRDVMMQ4FMXZUQBYH6A, seg_pos={0 0}, upload=645638]: Done uploading piece. err: <nil>
Jan 5 09:20:43.837042204 (737h51m43.708125239s) [noise=true , address=45.91.116.9:4281 , piece_id=4SWY2HIJQQPDBY4PFKR7YFY3TTZ2IPSGN6YAKOYB3WSVKVDJODYA, seg_pos={0 0}, upload=645638]: Done uploading piece. err: <nil>
Jan 5 09:20:43.837063124 (737h51m43.708146159s) [seg_pos={0 0}, upload=645638]: Segment reached optimal threshold of 14 pieces.
Jan 5 09:20:43.837256932 (737h51m43.708339997s) [noise=true , address=77.90.31.11:4563 , piece_id=FSH2HF4HWZE6ADTPMJ3KULEVALE46TROVOVANXT7CLPJBNUX64EA, seg_pos={0 0}, upload=645638]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bCw4Dx7aFbxT7TeUKezRzHD4yQWYXXNJVgFfF3FGhnkVVk9Jb): protocol: expected piece hash; context canceled; EOF
Jan 5 09:20:43.837265549 (737h51m43.708348584s) [noise=true , address=77.90.31.11:4563 , piece_id=FSH2HF4HWZE6ADTPMJ3KULEVALE46TROVOVANXT7CLPJBNUX64EA, seg_pos={0 0}, upload=645638]: Long tail cancellation due to optimal threshold
Jan 5 09:20:43.837484166 (737h51m43.708567211s) [noise=true , address=64.49.13.169:4715 , piece_id=KKFCUQH62XP7TNYBQFEN72GZPRIMBX677IT5RHNRGEVMP3OG2F7A, seg_pos={0 0}, upload=645638]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:57434->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:57434->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57434->64.49.13.169:4715: use of closed network connection
Jan 5 09:20:43.837505115 (737h51m43.70858815s) [noise=true , address=64.49.13.169:4715 , piece_id=KKFCUQH62XP7TNYBQFEN72GZPRIMBX677IT5RHNRGEVMP3OG2F7A, seg_pos={0 0}, upload=645638]: Long tail cancellation due to optimal threshold
Jan 5 09:20:43.837582934 (737h51m43.708665979s) [noise=true , address=64.49.13.155:4918 , piece_id=WAMEGNFI6LPBIRJ5Z2HM5UED6Y7DPUC4FC56GFUM5MU3WGDNBVSA, seg_pos={0 0}, upload=645638]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YjPvRvhJkNJtmt5CJ7nUst9NicEGWzpe7Pn1GWes5uc9H1gTK): protocol: read tcp 188.166.225.170:43292->64.49.13.155:4918: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 5 09:20:43.837589787 (737h51m43.708672832s) [noise=true , address=64.49.13.155:4918 , piece_id=WAMEGNFI6LPBIRJ5Z2HM5UED6Y7DPUC4FC56GFUM5MU3WGDNBVSA, seg_pos={0 0}, upload=645638]: Long tail cancellation due to optimal threshold
Jan 5 09:20:43.837902242 (737h51m43.708985277s) [noise=true , address=77.90.31.17:4786 , piece_id=LAULWIVBRGXC6S6VX2U5LTLPXCMAE7KU2HCOXVJ4LFGQJLOLKNQA, seg_pos={0 0}, upload=645638]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128QrekLqpXLAJs7yqNziNLmt2i13SuMx1ZhMPdUpDosw71dgQh): protocol: expected piece hash; context canceled; EOF
Jan 5 09:20:43.837909375 (737h51m43.70899242s) [noise=true , address=77.90.31.17:4786 , piece_id=LAULWIVBRGXC6S6VX2U5LTLPXCMAE7KU2HCOXVJ4LFGQJLOLKNQA, seg_pos={0 0}, upload=645638]: Long tail cancellation due to optimal threshold
Jan 5 09:20:43.837927719 (737h51m43.709010754s) [seg_pos={0 0}, upload=645638]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jan 5 09:20:43.837996601 (737h51m43.709079636s) [upload=645638]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 5 09:20:43.838001461 (737h51m43.709084496s) [upload=645638]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 5 09:20:44.119746458 (737h51m43.990829493s) [upload=645638]: Done uploading segments...
Jan 5 09:20:44.119777267 (737h51m43.990860312s) [upload=645638]: Upload finished. err: <nil>
Duration: 15.542905418s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
13 out of 60 is failed (threshold=3) [X.X...X....................X..X..X.X.X.X..X...X...X..X......] duration:
7.639015603s trace:
https://jaeger.dev.storj.io/trace/6e7e86a0c49d3474
FAIL (2026-01-05 09:22:01 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Jan 5 09:17:54.838592341 (737h48m54.709675376s) [upload=645597]: Starting upload
Jan 5 09:17:54.838646896 (737h48m54.709729931s) [upload=645597]: Done starting upload
Jan 5 09:17:54.838710166 (737h48m54.709793201s) [upload=645597]: Uploading segments...
Jan 5 09:17:54.838864910 (737h48m54.709947955s) [upload=645597]: Got next segment. Inline: false
Jan 5 09:17:54.838898374 (737h48m54.709981409s) [upload=645597]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 5 09:17:54.838902923 (737h48m54.709985958s) [upload=645597]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 5 09:17:55.086410939 (737h48m54.957493974s) [seg_pos={0 0}, upload=645597]: Begin upload segment...
Jan 5 09:17:55.086467737 (737h48m54.957550772s) [seg_pos={0 0}, upload=645597]: Done begin upload segment.
Jan 5 09:17:55.086524434 (737h48m54.957607469s) [noise=true , address=145.79.227.2:4115 , piece_id=ATGM7WYGHXFWDN4BXJDTX34WZQSAG23TALV7XX3UBIA3CRNR6FTQ, seg_pos={0 0}, upload=645597]: Uploading piece...
Jan 5 09:17:55.086542950 (737h48m54.957625995s) [noise=true , address=145.79.227.52:4446 , piece_id=NSYEHINMINQWH4FYOFJ6CRGEITDV3BUN65J5AYSNWV4GKU2VBVXA, seg_pos={0 0}, upload=645597]: Uploading piece...
Jan 5 09:17:55.087164153 (737h48m54.958247188s) [noise=true , address=145.79.227.20:4437 , piece_id=ZG7MWW7CB2MBGAVRMGO4TUHYAIJKOSSI5GIQ3ZVD7WK7TKP4FTFQ, seg_pos={0 0}, upload=645597]: Uploading piece...
Jan 5 09:17:55.087585125 (737h48m54.95866817s) [noise=true , address=45.91.116.6:4975 , piece_id=BQWSC3Q3YHF72A57QEVHMFKKJBZI5CLJ46A7TP3EEO5RRCH3MJFA, seg_pos={0 0}, upload=645597]: Uploading piece...
Jan 5 09:17:55.087645098 (737h48m54.958728133s) [noise=true , address=45.91.116.14:4191 , piece_id=GOZG4DIDVDB6DJX6BRH74ZVCD5BQLYUTHMMFICGG5DXRTQBQOCEQ, seg_pos={0 0}, upload=645597]: Uploading piece...
Jan 5 09:17:55.087978223 (737h48m54.959061258s) [noise=true , address=145.79.227.38:4979 , piece_id=6CT2VJHMICL5XITOIOINR4KZKACBFN6JLBFQHCXIXWE7Q6ZMMICQ, seg_pos={0 0}, upload=645597]: Uploading piece...
Jan 5 09:17:55.088015594 (737h48m54.959098639s) [noise=true , address=145.79.227.6:4259 , piece_id=2N3EYFVAHSKP4XZBAQDZPZL3TPNKWDFEFOTHITQRTQTHULWZZLJA, seg_pos={0 0}, upload=645597]: Uploading piece...
Jan 5 09:17:55.088279256 (737h48m54.959362292s) [noise=true , address=77.90.31.19:4842 , piece_id=7Z2Y7NXZELYGUD4G5PVBLKRJZL3CLOF22ZREPSUGPT4JYJ4Y22PQ, seg_pos={0 0}, upload=645597]: Uploading piece...
Jan 5 09:17:55.088399405 (737h48m54.95948245s) [noise=true , address=77.90.31.10:4518 , piece_id=FSVPHUKXSS57ME2FIEDDDOLWN3YHQN5NE7BLUYM3S7BY5BICULMA, seg_pos={0 0}, upload=645597]: Uploading piece...
Jan 5 09:17:55.088619494 (737h48m54.959702529s) [noise=true , address=145.79.227.45:4230 , piece_id=YZHU7J2SMWXIME3FIPWG5QK3LFDTNHSBLB4WSSTRBJCWPBXW2NVA, seg_pos={0 0}, upload=645597]: Uploading piece...
Jan 5 09:17:55.088966394 (737h48m54.960049439s) [noise=true , address=145.79.227.24:4581 , piece_id=4G3SJ2KRQ3OISKABPCJ65C5G256IH6UAUCFZ4MEOADX2PHZUTAIA, seg_pos={0 0}, upload=645597]: Uploading piece...
Jan 5 09:17:55.089316721 (737h48m54.960399756s) [noise=true , address=145.79.227.18:4006 , piece_id=CPDP4Q5XGXRCB474UCZ6HEN77572EJDQF37ABK6BH5I7IRXYQVPQ, seg_pos={0 0}, upload=645597]: Uploading piece...
Jan 5 09:17:55.089776337 (737h48m54.960859372s) [noise=true , address=77.90.31.21:4880 , piece_id=KXUC3TEKHHO4VDM5TDJDECSV354232VGUC7SLNVT5GNJ3V3EFGCA, seg_pos={0 0}, upload=645597]: Uploading piece...
Jan 5 09:17:55.089850378 (737h48m54.960933413s) [noise=true , address=145.79.227.28:4727 , piece_id=4XXMSMSYFQAFMMKM24WTX4GVLTWIQAM3W62K646VN4MWVQCMWE6Q, seg_pos={0 0}, upload=645597]: Uploading piece...
Jan 5 09:17:55.090162883 (737h48m54.961245918s) [noise=true , address=45.91.116.13:4156 , piece_id=GM7AZGY2XU2YWVODUD373QFFFYLOLIFEZVY7UJBKKABKSEDFYH4Q, seg_pos={0 0}, upload=645597]: Uploading piece...
Jan 5 09:17:55.091675703 (737h48m54.962758738s) [noise=true , address=145.79.227.14:4838 , piece_id=5F2HPBJJJJ2WJHFFDVD55RMPGXKW7LY7U7QIEY4KZRLWTY3HDYYQ, seg_pos={0 0}, upload=645597]: Uploading piece...
Jan 5 09:17:56.794463178 (737h48m56.665546213s) [upload=645597]: Next returned nil segment
Jan 5 09:17:56.794479830 (737h48m56.665562865s) [upload=645597]: Waiting for error group managing segments...
Jan 5 09:17:57.076908991 (737h48m56.947992036s) [noise=true , address=145.79.227.24:4581 , piece_id=4G3SJ2KRQ3OISKABPCJ65C5G256IH6UAUCFZ4MEOADX2PHZUTAIA, seg_pos={0 0}, upload=645597]: Done uploading piece. err: <nil>
Jan 5 09:17:57.077566553 (737h48m56.948649598s) [noise=true , address=145.79.227.14:4838 , piece_id=5F2HPBJJJJ2WJHFFDVD55RMPGXKW7LY7U7QIEY4KZRLWTY3HDYYQ, seg_pos={0 0}, upload=645597]: Done uploading piece. err: <nil>
Jan 5 09:17:57.093208608 (737h48m56.964291653s) [noise=true , address=145.79.227.38:4979 , piece_id=6CT2VJHMICL5XITOIOINR4KZKACBFN6JLBFQHCXIXWE7Q6ZMMICQ, seg_pos={0 0}, upload=645597]: Done uploading piece. err: <nil>
Jan 5 09:17:57.097347419 (737h48m56.968430464s) [noise=true , address=145.79.227.28:4727 , piece_id=4XXMSMSYFQAFMMKM24WTX4GVLTWIQAM3W62K646VN4MWVQCMWE6Q, seg_pos={0 0}, upload=645597]: Done uploading piece. err: <nil>
Jan 5 09:17:57.097948074 (737h48m56.969031119s) [noise=true , address=145.79.227.2:4115 , piece_id=ATGM7WYGHXFWDN4BXJDTX34WZQSAG23TALV7XX3UBIA3CRNR6FTQ, seg_pos={0 0}, upload=645597]: Done uploading piece. err: <nil>
Jan 5 09:17:57.101168726 (737h48m56.972251771s) [noise=true , address=145.79.227.18:4006 , piece_id=CPDP4Q5XGXRCB474UCZ6HEN77572EJDQF37ABK6BH5I7IRXYQVPQ, seg_pos={0 0}, upload=645597]: Done uploading piece. err: <nil>
Jan 5 09:17:57.106539102 (737h48m56.977622137s) [noise=true , address=145.79.227.52:4446 , piece_id=NSYEHINMINQWH4FYOFJ6CRGEITDV3BUN65J5AYSNWV4GKU2VBVXA, seg_pos={0 0}, upload=645597]: Done uploading piece. err: <nil>
Jan 5 09:17:57.124938367 (737h48m56.996021413s) [noise=true , address=145.79.227.20:4437 , piece_id=ZG7MWW7CB2MBGAVRMGO4TUHYAIJKOSSI5GIQ3ZVD7WK7TKP4FTFQ, seg_pos={0 0}, upload=645597]: Done uploading piece. err: <nil>
Jan 5 09:17:57.142071331 (737h48m57.013154376s) [noise=true , address=145.79.227.6:4259 , piece_id=2N3EYFVAHSKP4XZBAQDZPZL3TPNKWDFEFOTHITQRTQTHULWZZLJA, seg_pos={0 0}, upload=645597]: Done uploading piece. err: <nil>
Jan 5 09:17:57.175080546 (737h48m57.046163581s) [noise=true , address=145.79.227.45:4230 , piece_id=YZHU7J2SMWXIME3FIPWG5QK3LFDTNHSBLB4WSSTRBJCWPBXW2NVA, seg_pos={0 0}, upload=645597]: Done uploading piece. err: <nil>
Jan 5 09:17:58.461065682 (737h48m58.332148717s) [noise=true , address=45.91.116.14:4191 , piece_id=GOZG4DIDVDB6DJX6BRH74ZVCD5BQLYUTHMMFICGG5DXRTQBQOCEQ, seg_pos={0 0}, upload=645597]: Done uploading piece. err: <nil>
Jan 5 09:17:59.174407502 (737h48m59.045490547s) [noise=true , address=45.91.116.6:4975 , piece_id=BQWSC3Q3YHF72A57QEVHMFKKJBZI5CLJ46A7TP3EEO5RRCH3MJFA, seg_pos={0 0}, upload=645597]: Done uploading piece. err: <nil>
Jan 5 09:18:03.798756456 (737h49m3.669839501s) [noise=true , address=77.90.31.19:4842 , piece_id=7Z2Y7NXZELYGUD4G5PVBLKRJZL3CLOF22ZREPSUGPT4JYJ4Y22PQ, seg_pos={0 0}, upload=645597]: Done uploading piece. err: <nil>
Jan 5 09:18:03.798796051 (737h49m3.669879086s) [seg_pos={0 0}, upload=645597]: Segment reached optimal threshold of 13 pieces.
Jan 5 09:18:03.799105350 (737h49m3.670188396s) [noise=true , address=77.90.31.21:4880 , piece_id=KXUC3TEKHHO4VDM5TDJDECSV354232VGUC7SLNVT5GNJ3V3EFGCA, seg_pos={0 0}, upload=645597]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF
Jan 5 09:18:03.799115099 (737h49m3.670198144s) [noise=true , address=77.90.31.21:4880 , piece_id=KXUC3TEKHHO4VDM5TDJDECSV354232VGUC7SLNVT5GNJ3V3EFGCA, seg_pos={0 0}, upload=645597]: Long tail cancellation due to optimal threshold
Jan 5 09:18:03.799379172 (737h49m3.670462217s) [noise=true , address=77.90.31.10:4518 , piece_id=FSVPHUKXSS57ME2FIEDDDOLWN3YHQN5NE7BLUYM3S7BY5BICULMA, seg_pos={0 0}, upload=645597]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: read tcp 188.166.225.170:41460->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 5 09:18:03.799415481 (737h49m3.670498516s) [noise=true , address=77.90.31.10:4518 , piece_id=FSVPHUKXSS57ME2FIEDDDOLWN3YHQN5NE7BLUYM3S7BY5BICULMA, seg_pos={0 0}, upload=645597]: Long tail cancellation due to optimal threshold
Jan 5 09:18:03.799559655 (737h49m3.670642701s) [noise=true , address=45.91.116.13:4156 , piece_id=GM7AZGY2XU2YWVODUD373QFFFYLOLIFEZVY7UJBKKABKSEDFYH4Q, seg_pos={0 0}, upload=645597]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: read tcp 188.166.225.170:34200->45.91.116.13:4156: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 5 09:18:03.799570056 (737h49m3.670653091s) [noise=true , address=45.91.116.13:4156 , piece_id=GM7AZGY2XU2YWVODUD373QFFFYLOLIFEZVY7UJBKKABKSEDFYH4Q, seg_pos={0 0}, upload=645597]: Long tail cancellation due to optimal threshold
Jan 5 09:18:03.799597147 (737h49m3.670680192s) [seg_pos={0 0}, upload=645597]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 5 09:18:03.799692008 (737h49m3.670775053s) [upload=645597]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 5 09:18:03.799698189 (737h49m3.670781234s) [upload=645597]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 5 09:18:04.095750160 (737h49m3.966833205s) [upload=645597]: Done uploading segments...
Jan 5 09:18:04.095772302 (737h49m3.966855347s) [upload=645597]: Upload finished. err: <nil>
Duration: 7.639041562s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
44 out of 60 is failed (threshold=6) [XXXXX.XX..XXX.X.XX.XXX.XX.XXXXXX..XXXXXX...XXXX..XXXXXXXXXX.] duration:
3.399294612s trace:
https://jaeger.dev.storj.io/trace/6a8ed939efbf89dc
FAIL (2026-01-05 09:22:00 UTC)
orbiter warning: context deadline exceeded
Duration: 3.399334549s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [...........X..X...X.........................................] duration:
21.106042931s trace:
https://jaeger.dev.storj.io/trace/7e43d4cb58eef6b9
FAIL (2026-01-05 09:21:52 UTC)
orbiter warning: context deadline exceeded
Duration: 21.10606903s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
27 out of 60 is failed (threshold=3) [.XXXXXX.XXX...X....XX.X.X....XX....X..XX.XX......X.X.X..XXX.] duration:
15.094447058s trace:
https://jaeger.dev.storj.io/trace/6bc33b9da6112cd
FAIL (2026-01-05 09:21:38 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
7A, seg_pos={0 0}, upload=645638]: Uploading piece...
Jan 5 09:20:39.819337437 (737h51m39.690420482s) [noise=true , address=145.79.227.43:4191 , piece_id=BSM6NWJPFJZGSENPIRPHFJKOS2S6EEEFSNCAT3JVZ2DBQDM4I5JQ, seg_pos={0 0}, upload=645638]: Uploading piece...
Jan 5 09:20:39.819876133 (737h51m39.690959168s) [noise=true , address=145.79.227.32:4860 , piece_id=SAHLEMXIG63JDHFTXKBFDGUGFIKVTTGHA5APZ7VYW7RW62LTE5UQ, seg_pos={0 0}, upload=645638]: Uploading piece...
Jan 5 09:20:39.820218706 (737h51m39.691301751s) [noise=true , address=77.90.31.17:4786 , piece_id=LAULWIVBRGXC6S6VX2U5LTLPXCMAE7KU2HCOXVJ4LFGQJLOLKNQA, seg_pos={0 0}, upload=645638]: Uploading piece...
Jan 5 09:20:39.820761499 (737h51m39.691844544s) [noise=true , address=145.79.227.10:4435 , piece_id=LAVCQ6YZU6JLYGA4CB6FLVYDK2JXJFFL3FE6N7JZZQDA5KEKAYRQ, seg_pos={0 0}, upload=645638]: Uploading piece...
Jan 5 09:20:39.821121004 (737h51m39.692204039s) [noise=true , address=64.49.13.155:4918 , piece_id=WAMEGNFI6LPBIRJ5Z2HM5UED6Y7DPUC4FC56GFUM5MU3WGDNBVSA, seg_pos={0 0}, upload=645638]: Uploading piece...
Jan 5 09:20:39.821498272 (737h51m39.692581307s) [noise=true , address=145.79.227.33:4904 , piece_id=7FUOLIDMDFBETDXDYWDWMFR77GGZM7BR4Q36J6EX4AO4WJWJXRTQ, seg_pos={0 0}, upload=645638]: Uploading piece...
Jan 5 09:20:39.821944962 (737h51m39.693027997s) [noise=true , address=145.79.227.34:4932 , piece_id=CIBME66KBWA2REKAINEZQUGDZILM6OD36ODGAYPTYEKZ7METFFWA, seg_pos={0 0}, upload=645638]: Uploading piece...
Jan 5 09:20:39.822307192 (737h51m39.693390237s) [noise=true , address=45.91.116.9:4281 , piece_id=4SWY2HIJQQPDBY4PFKR7YFY3TTZ2IPSGN6YAKOYB3WSVKVDJODYA, seg_pos={0 0}, upload=645638]: Uploading piece...
Jan 5 09:20:39.822810150 (737h51m39.693893185s) [noise=true , address=145.79.227.40:4083 , piece_id=2ROGBVVX2LZACQRKL2CD4TOHOHYBJS4ZPPE4WXFAGJAUDLZPBXLQ, seg_pos={0 0}, upload=645638]: Uploading piece...
Jan 5 09:20:39.824181141 (737h51m39.695264176s) [noise=true , address=145.79.227.49:4344 , piece_id=OMH76SOXDDSLK4GNBVSGKRH6JIGZM7AVEHMX3JUZ5ILUTYECXSWQ, seg_pos={0 0}, upload=645638]: Uploading piece...
Jan 5 09:20:39.825441410 (737h51m39.696524446s) [noise=true , address=77.90.31.13:4631 , piece_id=KONA5PHEO3WERYMHTSRPHWGSGVS5ARNQEJIYEMRZGVQFRJ5GMALQ, seg_pos={0 0}, upload=645638]: Uploading piece...
Jan 5 09:20:39.826851755 (737h51m39.697934791s) [noise=true , address=77.90.31.11:4563 , piece_id=FSH2HF4HWZE6ADTPMJ3KULEVALE46TROVOVANXT7CLPJBNUX64EA, seg_pos={0 0}, upload=645638]: Uploading piece...
Jan 5 09:20:39.828547344 (737h51m39.69963038s) [noise=true , address=77.90.31.21:4886 , piece_id=FBJM277VXOWHLBC77QZRRFU5VSVOOEETLPRDVMMQ4FMXZUQBYH6A, seg_pos={0 0}, upload=645638]: Uploading piece...
Jan 5 09:20:39.830804523 (737h51m39.701887568s) [noise=true , address=145.79.227.44:4211 , piece_id=JAPLQHRW4D66MI3GIQUGTKXDGKHBT77QTGMSYDQJTAJIX4JJUZFQ, seg_pos={0 0}, upload=645638]: Uploading piece...
Jan 5 09:20:39.832429627 (737h51m39.703512673s) [noise=true , address=193.149.164.143:4703 , piece_id=ONPBUN2W67WQLBGU4KRH3YZCAJGK2XGGINDF77RPITG6DUYALTRA, seg_pos={0 0}, upload=645638]: Uploading piece...
Jan 5 09:20:41.249821392 (737h51m41.120904427s) [upload=645638]: Next returned nil segment
Jan 5 09:20:41.249841230 (737h51m41.120924265s) [upload=645638]: Waiting for error group managing segments...
Jan 5 09:20:41.690593524 (737h51m41.561676569s) [noise=true , address=145.79.227.49:4344 , piece_id=OMH76SOXDDSLK4GNBVSGKRH6JIGZM7AVEHMX3JUZ5ILUTYECXSWQ, seg_pos={0 0}, upload=645638]: Done uploading piece. err: <nil>
Jan 5 09:20:41.702709104 (737h51m41.57379215s) [noise=true , address=145.79.227.10:4435 , piece_id=LAVCQ6YZU6JLYGA4CB6FLVYDK2JXJFFL3FE6N7JZZQDA5KEKAYRQ, seg_pos={0 0}, upload=645638]: Done uploading piece. err: <nil>
Jan 5 09:20:41.705922394 (737h51m41.577005439s) [noise=true , address=145.79.227.32:4860 , piece_id=SAHLEMXIG63JDHFTXKBFDGUGFIKVTTGHA5APZ7VYW7RW62LTE5UQ, seg_pos={0 0}, upload=645638]: Done uploading piece. err: <nil>
Jan 5 09:20:42.008187135 (737h51m41.87927017s) [noise=true , address=77.90.31.13:4631 , piece_id=KONA5PHEO3WERYMHTSRPHWGSGVS5ARNQEJIYEMRZGVQFRJ5GMALQ, seg_pos={0 0}, upload=645638]: Done uploading piece. err: <nil>
Jan 5 09:20:42.571971402 (737h51m42.443054437s) [noise=true , address=145.79.227.33:4904 , piece_id=7FUOLIDMDFBETDXDYWDWMFR77GGZM7BR4Q36J6EX4AO4WJWJXRTQ, seg_pos={0 0}, upload=645638]: Done uploading piece. err: <nil>
Jan 5 09:20:42.590678753 (737h51m42.461761799s) [noise=true , address=145.79.227.34:4932 , piece_id=CIBME66KBWA2REKAINEZQUGDZILM6OD36ODGAYPTYEKZ7METFFWA, seg_pos={0 0}, upload=645638]: Done uploading piece. err: <nil>
Jan 5 09:20:42.626844268 (737h51m42.497927303s) [noise=true , address=145.79.227.43:4191 , piece_id=BSM6NWJPFJZGSENPIRPHFJKOS2S6EEEFSNCAT3JVZ2DBQDM4I5JQ, seg_pos={0 0}, upload=645638]: Done uploading piece. err: <nil>
Jan 5 09:20:42.636866652 (737h51m42.507949687s) [noise=true , address=145.79.227.35:4502 , piece_id=DG7ALRAPMPQMTXXES5IUBHKQWLWFXSWSJ37BFJUFEQJAGPKZ25IA, seg_pos={0 0}, upload=645638]: Done uploading piece. err: <nil>
Jan 5 09:20:42.693104690 (737h51m42.564187735s) [noise=true , address=145.79.227.40:4083 , piece_id=2ROGBVVX2LZACQRKL2CD4TOHOHYBJS4ZPPE4WXFAGJAUDLZPBXLQ, seg_pos={0 0}, upload=645638]: Done uploading piece. err: <nil>
Jan 5 09:20:42.960096876 (737h51m42.831179911s) [noise=true , address=193.149.164.143:4703 , piece_id=ONPBUN2W67WQLBGU4KRH3YZCAJGK2XGGINDF77RPITG6DUYALTRA, seg_pos={0 0}, upload=645638]: Done uploading piece. err: <nil>
Jan 5 09:20:43.059872435 (737h51m42.93095547s) [noise=true , address=145.79.227.30:4784 , piece_id=HUQUN57CKDGPGD7234XNDOQJH4YGNUJRS27QJFH4IJ4Z6XRNONBA, seg_pos={0 0}, upload=645638]: Done uploading piece. err: <nil>
Jan 5 09:20:43.092756041 (737h51m42.963839076s) [noise=true , address=145.79.227.44:4211 , piece_id=JAPLQHRW4D66MI3GIQUGTKXDGKHBT77QTGMSYDQJTAJIX4JJUZFQ, seg_pos={0 0}, upload=645638]: Done uploading piece. err: <nil>
Jan 5 09:20:43.274893281 (737h51m43.145976316s) [noise=true , address=77.90.31.21:4886 , piece_id=FBJM277VXOWHLBC77QZRRFU5VSVOOEETLPRDVMMQ4FMXZUQBYH6A, seg_pos={0 0}, upload=645638]: Done uploading piece. err: <nil>
Jan 5 09:20:43.837042204 (737h51m43.708125239s) [noise=true , address=45.91.116.9:4281 , piece_id=4SWY2HIJQQPDBY4PFKR7YFY3TTZ2IPSGN6YAKOYB3WSVKVDJODYA, seg_pos={0 0}, upload=645638]: Done uploading piece. err: <nil>
Jan 5 09:20:43.837063124 (737h51m43.708146159s) [seg_pos={0 0}, upload=645638]: Segment reached optimal threshold of 14 pieces.
Jan 5 09:20:43.837256932 (737h51m43.708339997s) [noise=true , address=77.90.31.11:4563 , piece_id=FSH2HF4HWZE6ADTPMJ3KULEVALE46TROVOVANXT7CLPJBNUX64EA, seg_pos={0 0}, upload=645638]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bCw4Dx7aFbxT7TeUKezRzHD4yQWYXXNJVgFfF3FGhnkVVk9Jb): protocol: expected piece hash; context canceled; EOF
Jan 5 09:20:43.837265549 (737h51m43.708348584s) [noise=true , address=77.90.31.11:4563 , piece_id=FSH2HF4HWZE6ADTPMJ3KULEVALE46TROVOVANXT7CLPJBNUX64EA, seg_pos={0 0}, upload=645638]: Long tail cancellation due to optimal threshold
Jan 5 09:20:43.837484166 (737h51m43.708567211s) [noise=true , address=64.49.13.169:4715 , piece_id=KKFCUQH62XP7TNYBQFEN72GZPRIMBX677IT5RHNRGEVMP3OG2F7A, seg_pos={0 0}, upload=645638]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:57434->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:57434->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57434->64.49.13.169:4715: use of closed network connection
Jan 5 09:20:43.837505115 (737h51m43.70858815s) [noise=true , address=64.49.13.169:4715 , piece_id=KKFCUQH62XP7TNYBQFEN72GZPRIMBX677IT5RHNRGEVMP3OG2F7A, seg_pos={0 0}, upload=645638]: Long tail cancellation due to optimal threshold
Jan 5 09:20:43.837582934 (737h51m43.708665979s) [noise=true , address=64.49.13.155:4918 , piece_id=WAMEGNFI6LPBIRJ5Z2HM5UED6Y7DPUC4FC56GFUM5MU3WGDNBVSA, seg_pos={0 0}, upload=645638]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YjPvRvhJkNJtmt5CJ7nUst9NicEGWzpe7Pn1GWes5uc9H1gTK): protocol: read tcp 188.166.225.170:43292->64.49.13.155:4918: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 5 09:20:43.837589787 (737h51m43.708672832s) [noise=true , address=64.49.13.155:4918 , piece_id=WAMEGNFI6LPBIRJ5Z2HM5UED6Y7DPUC4FC56GFUM5MU3WGDNBVSA, seg_pos={0 0}, upload=645638]: Long tail cancellation due to optimal threshold
Jan 5 09:20:43.837902242 (737h51m43.708985277s) [noise=true , address=77.90.31.17:4786 , piece_id=LAULWIVBRGXC6S6VX2U5LTLPXCMAE7KU2HCOXVJ4LFGQJLOLKNQA, seg_pos={0 0}, upload=645638]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128QrekLqpXLAJs7yqNziNLmt2i13SuMx1ZhMPdUpDosw71dgQh): protocol: expected piece hash; context canceled; EOF
Jan 5 09:20:43.837909375 (737h51m43.70899242s) [noise=true , address=77.90.31.17:4786 , piece_id=LAULWIVBRGXC6S6VX2U5LTLPXCMAE7KU2HCOXVJ4LFGQJLOLKNQA, seg_pos={0 0}, upload=645638]: Long tail cancellation due to optimal threshold
Jan 5 09:20:43.837927719 (737h51m43.709010754s) [seg_pos={0 0}, upload=645638]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jan 5 09:20:43.837996601 (737h51m43.709079636s) [upload=645638]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 5 09:20:43.838001461 (737h51m43.709084496s) [upload=645638]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 5 09:20:44.119746458 (737h51m43.990829493s) [upload=645638]: Done uploading segments...
Jan 5 09:20:44.119777267 (737h51m43.990860312s) [upload=645638]: Upload finished. err: <nil>
Duration: 15.094478958s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
13 out of 60 is failed (threshold=3) [.X.X...X....................X..X..X.X.X.X..X...X...X..X.....] duration:
6.146686104s trace:
https://jaeger.dev.storj.io/trace/6e7e86a0c49d3474
FAIL (2026-01-05 09:21:23 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Jan 5 09:17:54.838592341 (737h48m54.709675376s) [upload=645597]: Starting upload
Jan 5 09:17:54.838646896 (737h48m54.709729931s) [upload=645597]: Done starting upload
Jan 5 09:17:54.838710166 (737h48m54.709793201s) [upload=645597]: Uploading segments...
Jan 5 09:17:54.838864910 (737h48m54.709947955s) [upload=645597]: Got next segment. Inline: false
Jan 5 09:17:54.838898374 (737h48m54.709981409s) [upload=645597]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 5 09:17:54.838902923 (737h48m54.709985958s) [upload=645597]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 5 09:17:55.086410939 (737h48m54.957493974s) [seg_pos={0 0}, upload=645597]: Begin upload segment...
Jan 5 09:17:55.086467737 (737h48m54.957550772s) [seg_pos={0 0}, upload=645597]: Done begin upload segment.
Jan 5 09:17:55.086524434 (737h48m54.957607469s) [noise=true , address=145.79.227.2:4115 , piece_id=ATGM7WYGHXFWDN4BXJDTX34WZQSAG23TALV7XX3UBIA3CRNR6FTQ, seg_pos={0 0}, upload=645597]: Uploading piece...
Jan 5 09:17:55.086542950 (737h48m54.957625995s) [noise=true , address=145.79.227.52:4446 , piece_id=NSYEHINMINQWH4FYOFJ6CRGEITDV3BUN65J5AYSNWV4GKU2VBVXA, seg_pos={0 0}, upload=645597]: Uploading piece...
Jan 5 09:17:55.087164153 (737h48m54.958247188s) [noise=true , address=145.79.227.20:4437 , piece_id=ZG7MWW7CB2MBGAVRMGO4TUHYAIJKOSSI5GIQ3ZVD7WK7TKP4FTFQ, seg_pos={0 0}, upload=645597]: Uploading piece...
Jan 5 09:17:55.087585125 (737h48m54.95866817s) [noise=true , address=45.91.116.6:4975 , piece_id=BQWSC3Q3YHF72A57QEVHMFKKJBZI5CLJ46A7TP3EEO5RRCH3MJFA, seg_pos={0 0}, upload=645597]: Uploading piece...
Jan 5 09:17:55.087645098 (737h48m54.958728133s) [noise=true , address=45.91.116.14:4191 , piece_id=GOZG4DIDVDB6DJX6BRH74ZVCD5BQLYUTHMMFICGG5DXRTQBQOCEQ, seg_pos={0 0}, upload=645597]: Uploading piece...
Jan 5 09:17:55.087978223 (737h48m54.959061258s) [noise=true , address=145.79.227.38:4979 , piece_id=6CT2VJHMICL5XITOIOINR4KZKACBFN6JLBFQHCXIXWE7Q6ZMMICQ, seg_pos={0 0}, upload=645597]: Uploading piece...
Jan 5 09:17:55.088015594 (737h48m54.959098639s) [noise=true , address=145.79.227.6:4259 , piece_id=2N3EYFVAHSKP4XZBAQDZPZL3TPNKWDFEFOTHITQRTQTHULWZZLJA, seg_pos={0 0}, upload=645597]: Uploading piece...
Jan 5 09:17:55.088279256 (737h48m54.959362292s) [noise=true , address=77.90.31.19:4842 , piece_id=7Z2Y7NXZELYGUD4G5PVBLKRJZL3CLOF22ZREPSUGPT4JYJ4Y22PQ, seg_pos={0 0}, upload=645597]: Uploading piece...
Jan 5 09:17:55.088399405 (737h48m54.95948245s) [noise=true , address=77.90.31.10:4518 , piece_id=FSVPHUKXSS57ME2FIEDDDOLWN3YHQN5NE7BLUYM3S7BY5BICULMA, seg_pos={0 0}, upload=645597]: Uploading piece...
Jan 5 09:17:55.088619494 (737h48m54.959702529s) [noise=true , address=145.79.227.45:4230 , piece_id=YZHU7J2SMWXIME3FIPWG5QK3LFDTNHSBLB4WSSTRBJCWPBXW2NVA, seg_pos={0 0}, upload=645597]: Uploading piece...
Jan 5 09:17:55.088966394 (737h48m54.960049439s) [noise=true , address=145.79.227.24:4581 , piece_id=4G3SJ2KRQ3OISKABPCJ65C5G256IH6UAUCFZ4MEOADX2PHZUTAIA, seg_pos={0 0}, upload=645597]: Uploading piece...
Jan 5 09:17:55.089316721 (737h48m54.960399756s) [noise=true , address=145.79.227.18:4006 , piece_id=CPDP4Q5XGXRCB474UCZ6HEN77572EJDQF37ABK6BH5I7IRXYQVPQ, seg_pos={0 0}, upload=645597]: Uploading piece...
Jan 5 09:17:55.089776337 (737h48m54.960859372s) [noise=true , address=77.90.31.21:4880 , piece_id=KXUC3TEKHHO4VDM5TDJDECSV354232VGUC7SLNVT5GNJ3V3EFGCA, seg_pos={0 0}, upload=645597]: Uploading piece...
Jan 5 09:17:55.089850378 (737h48m54.960933413s) [noise=true , address=145.79.227.28:4727 , piece_id=4XXMSMSYFQAFMMKM24WTX4GVLTWIQAM3W62K646VN4MWVQCMWE6Q, seg_pos={0 0}, upload=645597]: Uploading piece...
Jan 5 09:17:55.090162883 (737h48m54.961245918s) [noise=true , address=45.91.116.13:4156 , piece_id=GM7AZGY2XU2YWVODUD373QFFFYLOLIFEZVY7UJBKKABKSEDFYH4Q, seg_pos={0 0}, upload=645597]: Uploading piece...
Jan 5 09:17:55.091675703 (737h48m54.962758738s) [noise=true , address=145.79.227.14:4838 , piece_id=5F2HPBJJJJ2WJHFFDVD55RMPGXKW7LY7U7QIEY4KZRLWTY3HDYYQ, seg_pos={0 0}, upload=645597]: Uploading piece...
Jan 5 09:17:56.794463178 (737h48m56.665546213s) [upload=645597]: Next returned nil segment
Jan 5 09:17:56.794479830 (737h48m56.665562865s) [upload=645597]: Waiting for error group managing segments...
Jan 5 09:17:57.076908991 (737h48m56.947992036s) [noise=true , address=145.79.227.24:4581 , piece_id=4G3SJ2KRQ3OISKABPCJ65C5G256IH6UAUCFZ4MEOADX2PHZUTAIA, seg_pos={0 0}, upload=645597]: Done uploading piece. err: <nil>
Jan 5 09:17:57.077566553 (737h48m56.948649598s) [noise=true , address=145.79.227.14:4838 , piece_id=5F2HPBJJJJ2WJHFFDVD55RMPGXKW7LY7U7QIEY4KZRLWTY3HDYYQ, seg_pos={0 0}, upload=645597]: Done uploading piece. err: <nil>
Jan 5 09:17:57.093208608 (737h48m56.964291653s) [noise=true , address=145.79.227.38:4979 , piece_id=6CT2VJHMICL5XITOIOINR4KZKACBFN6JLBFQHCXIXWE7Q6ZMMICQ, seg_pos={0 0}, upload=645597]: Done uploading piece. err: <nil>
Jan 5 09:17:57.097347419 (737h48m56.968430464s) [noise=true , address=145.79.227.28:4727 , piece_id=4XXMSMSYFQAFMMKM24WTX4GVLTWIQAM3W62K646VN4MWVQCMWE6Q, seg_pos={0 0}, upload=645597]: Done uploading piece. err: <nil>
Jan 5 09:17:57.097948074 (737h48m56.969031119s) [noise=true , address=145.79.227.2:4115 , piece_id=ATGM7WYGHXFWDN4BXJDTX34WZQSAG23TALV7XX3UBIA3CRNR6FTQ, seg_pos={0 0}, upload=645597]: Done uploading piece. err: <nil>
Jan 5 09:17:57.101168726 (737h48m56.972251771s) [noise=true , address=145.79.227.18:4006 , piece_id=CPDP4Q5XGXRCB474UCZ6HEN77572EJDQF37ABK6BH5I7IRXYQVPQ, seg_pos={0 0}, upload=645597]: Done uploading piece. err: <nil>
Jan 5 09:17:57.106539102 (737h48m56.977622137s) [noise=true , address=145.79.227.52:4446 , piece_id=NSYEHINMINQWH4FYOFJ6CRGEITDV3BUN65J5AYSNWV4GKU2VBVXA, seg_pos={0 0}, upload=645597]: Done uploading piece. err: <nil>
Jan 5 09:17:57.124938367 (737h48m56.996021413s) [noise=true , address=145.79.227.20:4437 , piece_id=ZG7MWW7CB2MBGAVRMGO4TUHYAIJKOSSI5GIQ3ZVD7WK7TKP4FTFQ, seg_pos={0 0}, upload=645597]: Done uploading piece. err: <nil>
Jan 5 09:17:57.142071331 (737h48m57.013154376s) [noise=true , address=145.79.227.6:4259 , piece_id=2N3EYFVAHSKP4XZBAQDZPZL3TPNKWDFEFOTHITQRTQTHULWZZLJA, seg_pos={0 0}, upload=645597]: Done uploading piece. err: <nil>
Jan 5 09:17:57.175080546 (737h48m57.046163581s) [noise=true , address=145.79.227.45:4230 , piece_id=YZHU7J2SMWXIME3FIPWG5QK3LFDTNHSBLB4WSSTRBJCWPBXW2NVA, seg_pos={0 0}, upload=645597]: Done uploading piece. err: <nil>
Jan 5 09:17:58.461065682 (737h48m58.332148717s) [noise=true , address=45.91.116.14:4191 , piece_id=GOZG4DIDVDB6DJX6BRH74ZVCD5BQLYUTHMMFICGG5DXRTQBQOCEQ, seg_pos={0 0}, upload=645597]: Done uploading piece. err: <nil>
Jan 5 09:17:59.174407502 (737h48m59.045490547s) [noise=true , address=45.91.116.6:4975 , piece_id=BQWSC3Q3YHF72A57QEVHMFKKJBZI5CLJ46A7TP3EEO5RRCH3MJFA, seg_pos={0 0}, upload=645597]: Done uploading piece. err: <nil>
Jan 5 09:18:03.798756456 (737h49m3.669839501s) [noise=true , address=77.90.31.19:4842 , piece_id=7Z2Y7NXZELYGUD4G5PVBLKRJZL3CLOF22ZREPSUGPT4JYJ4Y22PQ, seg_pos={0 0}, upload=645597]: Done uploading piece. err: <nil>
Jan 5 09:18:03.798796051 (737h49m3.669879086s) [seg_pos={0 0}, upload=645597]: Segment reached optimal threshold of 13 pieces.
Jan 5 09:18:03.799105350 (737h49m3.670188396s) [noise=true , address=77.90.31.21:4880 , piece_id=KXUC3TEKHHO4VDM5TDJDECSV354232VGUC7SLNVT5GNJ3V3EFGCA, seg_pos={0 0}, upload=645597]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF
Jan 5 09:18:03.799115099 (737h49m3.670198144s) [noise=true , address=77.90.31.21:4880 , piece_id=KXUC3TEKHHO4VDM5TDJDECSV354232VGUC7SLNVT5GNJ3V3EFGCA, seg_pos={0 0}, upload=645597]: Long tail cancellation due to optimal threshold
Jan 5 09:18:03.799379172 (737h49m3.670462217s) [noise=true , address=77.90.31.10:4518 , piece_id=FSVPHUKXSS57ME2FIEDDDOLWN3YHQN5NE7BLUYM3S7BY5BICULMA, seg_pos={0 0}, upload=645597]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: read tcp 188.166.225.170:41460->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 5 09:18:03.799415481 (737h49m3.670498516s) [noise=true , address=77.90.31.10:4518 , piece_id=FSVPHUKXSS57ME2FIEDDDOLWN3YHQN5NE7BLUYM3S7BY5BICULMA, seg_pos={0 0}, upload=645597]: Long tail cancellation due to optimal threshold
Jan 5 09:18:03.799559655 (737h49m3.670642701s) [noise=true , address=45.91.116.13:4156 , piece_id=GM7AZGY2XU2YWVODUD373QFFFYLOLIFEZVY7UJBKKABKSEDFYH4Q, seg_pos={0 0}, upload=645597]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: read tcp 188.166.225.170:34200->45.91.116.13:4156: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 5 09:18:03.799570056 (737h49m3.670653091s) [noise=true , address=45.91.116.13:4156 , piece_id=GM7AZGY2XU2YWVODUD373QFFFYLOLIFEZVY7UJBKKABKSEDFYH4Q, seg_pos={0 0}, upload=645597]: Long tail cancellation due to optimal threshold
Jan 5 09:18:03.799597147 (737h49m3.670680192s) [seg_pos={0 0}, upload=645597]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 5 09:18:03.799692008 (737h49m3.670775053s) [upload=645597]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 5 09:18:03.799698189 (737h49m3.670781234s) [upload=645597]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 5 09:18:04.095750160 (737h49m3.966833205s) [upload=645597]: Done uploading segments...
Jan 5 09:18:04.095772302 (737h49m3.966855347s) [upload=645597]: Upload finished. err: <nil>
Duration: 6.146713667s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
28 out of 60 is failed (threshold=3) [X.XXXXXX.XXX...X....XX.X.X....XX....X..XX.XX......X.X.X..XXX] duration:
43.241286467s trace:
https://jaeger.dev.storj.io/trace/6bc33b9da6112cd
FAIL (2026-01-05 09:21:22 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
7A, seg_pos={0 0}, upload=645638]: Uploading piece...
Jan 5 09:20:39.819337437 (737h51m39.690420482s) [noise=true , address=145.79.227.43:4191 , piece_id=BSM6NWJPFJZGSENPIRPHFJKOS2S6EEEFSNCAT3JVZ2DBQDM4I5JQ, seg_pos={0 0}, upload=645638]: Uploading piece...
Jan 5 09:20:39.819876133 (737h51m39.690959168s) [noise=true , address=145.79.227.32:4860 , piece_id=SAHLEMXIG63JDHFTXKBFDGUGFIKVTTGHA5APZ7VYW7RW62LTE5UQ, seg_pos={0 0}, upload=645638]: Uploading piece...
Jan 5 09:20:39.820218706 (737h51m39.691301751s) [noise=true , address=77.90.31.17:4786 , piece_id=LAULWIVBRGXC6S6VX2U5LTLPXCMAE7KU2HCOXVJ4LFGQJLOLKNQA, seg_pos={0 0}, upload=645638]: Uploading piece...
Jan 5 09:20:39.820761499 (737h51m39.691844544s) [noise=true , address=145.79.227.10:4435 , piece_id=LAVCQ6YZU6JLYGA4CB6FLVYDK2JXJFFL3FE6N7JZZQDA5KEKAYRQ, seg_pos={0 0}, upload=645638]: Uploading piece...
Jan 5 09:20:39.821121004 (737h51m39.692204039s) [noise=true , address=64.49.13.155:4918 , piece_id=WAMEGNFI6LPBIRJ5Z2HM5UED6Y7DPUC4FC56GFUM5MU3WGDNBVSA, seg_pos={0 0}, upload=645638]: Uploading piece...
Jan 5 09:20:39.821498272 (737h51m39.692581307s) [noise=true , address=145.79.227.33:4904 , piece_id=7FUOLIDMDFBETDXDYWDWMFR77GGZM7BR4Q36J6EX4AO4WJWJXRTQ, seg_pos={0 0}, upload=645638]: Uploading piece...
Jan 5 09:20:39.821944962 (737h51m39.693027997s) [noise=true , address=145.79.227.34:4932 , piece_id=CIBME66KBWA2REKAINEZQUGDZILM6OD36ODGAYPTYEKZ7METFFWA, seg_pos={0 0}, upload=645638]: Uploading piece...
Jan 5 09:20:39.822307192 (737h51m39.693390237s) [noise=true , address=45.91.116.9:4281 , piece_id=4SWY2HIJQQPDBY4PFKR7YFY3TTZ2IPSGN6YAKOYB3WSVKVDJODYA, seg_pos={0 0}, upload=645638]: Uploading piece...
Jan 5 09:20:39.822810150 (737h51m39.693893185s) [noise=true , address=145.79.227.40:4083 , piece_id=2ROGBVVX2LZACQRKL2CD4TOHOHYBJS4ZPPE4WXFAGJAUDLZPBXLQ, seg_pos={0 0}, upload=645638]: Uploading piece...
Jan 5 09:20:39.824181141 (737h51m39.695264176s) [noise=true , address=145.79.227.49:4344 , piece_id=OMH76SOXDDSLK4GNBVSGKRH6JIGZM7AVEHMX3JUZ5ILUTYECXSWQ, seg_pos={0 0}, upload=645638]: Uploading piece...
Jan 5 09:20:39.825441410 (737h51m39.696524446s) [noise=true , address=77.90.31.13:4631 , piece_id=KONA5PHEO3WERYMHTSRPHWGSGVS5ARNQEJIYEMRZGVQFRJ5GMALQ, seg_pos={0 0}, upload=645638]: Uploading piece...
Jan 5 09:20:39.826851755 (737h51m39.697934791s) [noise=true , address=77.90.31.11:4563 , piece_id=FSH2HF4HWZE6ADTPMJ3KULEVALE46TROVOVANXT7CLPJBNUX64EA, seg_pos={0 0}, upload=645638]: Uploading piece...
Jan 5 09:20:39.828547344 (737h51m39.69963038s) [noise=true , address=77.90.31.21:4886 , piece_id=FBJM277VXOWHLBC77QZRRFU5VSVOOEETLPRDVMMQ4FMXZUQBYH6A, seg_pos={0 0}, upload=645638]: Uploading piece...
Jan 5 09:20:39.830804523 (737h51m39.701887568s) [noise=true , address=145.79.227.44:4211 , piece_id=JAPLQHRW4D66MI3GIQUGTKXDGKHBT77QTGMSYDQJTAJIX4JJUZFQ, seg_pos={0 0}, upload=645638]: Uploading piece...
Jan 5 09:20:39.832429627 (737h51m39.703512673s) [noise=true , address=193.149.164.143:4703 , piece_id=ONPBUN2W67WQLBGU4KRH3YZCAJGK2XGGINDF77RPITG6DUYALTRA, seg_pos={0 0}, upload=645638]: Uploading piece...
Jan 5 09:20:41.249821392 (737h51m41.120904427s) [upload=645638]: Next returned nil segment
Jan 5 09:20:41.249841230 (737h51m41.120924265s) [upload=645638]: Waiting for error group managing segments...
Jan 5 09:20:41.690593524 (737h51m41.561676569s) [noise=true , address=145.79.227.49:4344 , piece_id=OMH76SOXDDSLK4GNBVSGKRH6JIGZM7AVEHMX3JUZ5ILUTYECXSWQ, seg_pos={0 0}, upload=645638]: Done uploading piece. err: <nil>
Jan 5 09:20:41.702709104 (737h51m41.57379215s) [noise=true , address=145.79.227.10:4435 , piece_id=LAVCQ6YZU6JLYGA4CB6FLVYDK2JXJFFL3FE6N7JZZQDA5KEKAYRQ, seg_pos={0 0}, upload=645638]: Done uploading piece. err: <nil>
Jan 5 09:20:41.705922394 (737h51m41.577005439s) [noise=true , address=145.79.227.32:4860 , piece_id=SAHLEMXIG63JDHFTXKBFDGUGFIKVTTGHA5APZ7VYW7RW62LTE5UQ, seg_pos={0 0}, upload=645638]: Done uploading piece. err: <nil>
Jan 5 09:20:42.008187135 (737h51m41.87927017s) [noise=true , address=77.90.31.13:4631 , piece_id=KONA5PHEO3WERYMHTSRPHWGSGVS5ARNQEJIYEMRZGVQFRJ5GMALQ, seg_pos={0 0}, upload=645638]: Done uploading piece. err: <nil>
Jan 5 09:20:42.571971402 (737h51m42.443054437s) [noise=true , address=145.79.227.33:4904 , piece_id=7FUOLIDMDFBETDXDYWDWMFR77GGZM7BR4Q36J6EX4AO4WJWJXRTQ, seg_pos={0 0}, upload=645638]: Done uploading piece. err: <nil>
Jan 5 09:20:42.590678753 (737h51m42.461761799s) [noise=true , address=145.79.227.34:4932 , piece_id=CIBME66KBWA2REKAINEZQUGDZILM6OD36ODGAYPTYEKZ7METFFWA, seg_pos={0 0}, upload=645638]: Done uploading piece. err: <nil>
Jan 5 09:20:42.626844268 (737h51m42.497927303s) [noise=true , address=145.79.227.43:4191 , piece_id=BSM6NWJPFJZGSENPIRPHFJKOS2S6EEEFSNCAT3JVZ2DBQDM4I5JQ, seg_pos={0 0}, upload=645638]: Done uploading piece. err: <nil>
Jan 5 09:20:42.636866652 (737h51m42.507949687s) [noise=true , address=145.79.227.35:4502 , piece_id=DG7ALRAPMPQMTXXES5IUBHKQWLWFXSWSJ37BFJUFEQJAGPKZ25IA, seg_pos={0 0}, upload=645638]: Done uploading piece. err: <nil>
Jan 5 09:20:42.693104690 (737h51m42.564187735s) [noise=true , address=145.79.227.40:4083 , piece_id=2ROGBVVX2LZACQRKL2CD4TOHOHYBJS4ZPPE4WXFAGJAUDLZPBXLQ, seg_pos={0 0}, upload=645638]: Done uploading piece. err: <nil>
Jan 5 09:20:42.960096876 (737h51m42.831179911s) [noise=true , address=193.149.164.143:4703 , piece_id=ONPBUN2W67WQLBGU4KRH3YZCAJGK2XGGINDF77RPITG6DUYALTRA, seg_pos={0 0}, upload=645638]: Done uploading piece. err: <nil>
Jan 5 09:20:43.059872435 (737h51m42.93095547s) [noise=true , address=145.79.227.30:4784 , piece_id=HUQUN57CKDGPGD7234XNDOQJH4YGNUJRS27QJFH4IJ4Z6XRNONBA, seg_pos={0 0}, upload=645638]: Done uploading piece. err: <nil>
Jan 5 09:20:43.092756041 (737h51m42.963839076s) [noise=true , address=145.79.227.44:4211 , piece_id=JAPLQHRW4D66MI3GIQUGTKXDGKHBT77QTGMSYDQJTAJIX4JJUZFQ, seg_pos={0 0}, upload=645638]: Done uploading piece. err: <nil>
Jan 5 09:20:43.274893281 (737h51m43.145976316s) [noise=true , address=77.90.31.21:4886 , piece_id=FBJM277VXOWHLBC77QZRRFU5VSVOOEETLPRDVMMQ4FMXZUQBYH6A, seg_pos={0 0}, upload=645638]: Done uploading piece. err: <nil>
Jan 5 09:20:43.837042204 (737h51m43.708125239s) [noise=true , address=45.91.116.9:4281 , piece_id=4SWY2HIJQQPDBY4PFKR7YFY3TTZ2IPSGN6YAKOYB3WSVKVDJODYA, seg_pos={0 0}, upload=645638]: Done uploading piece. err: <nil>
Jan 5 09:20:43.837063124 (737h51m43.708146159s) [seg_pos={0 0}, upload=645638]: Segment reached optimal threshold of 14 pieces.
Jan 5 09:20:43.837256932 (737h51m43.708339997s) [noise=true , address=77.90.31.11:4563 , piece_id=FSH2HF4HWZE6ADTPMJ3KULEVALE46TROVOVANXT7CLPJBNUX64EA, seg_pos={0 0}, upload=645638]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bCw4Dx7aFbxT7TeUKezRzHD4yQWYXXNJVgFfF3FGhnkVVk9Jb): protocol: expected piece hash; context canceled; EOF
Jan 5 09:20:43.837265549 (737h51m43.708348584s) [noise=true , address=77.90.31.11:4563 , piece_id=FSH2HF4HWZE6ADTPMJ3KULEVALE46TROVOVANXT7CLPJBNUX64EA, seg_pos={0 0}, upload=645638]: Long tail cancellation due to optimal threshold
Jan 5 09:20:43.837484166 (737h51m43.708567211s) [noise=true , address=64.49.13.169:4715 , piece_id=KKFCUQH62XP7TNYBQFEN72GZPRIMBX677IT5RHNRGEVMP3OG2F7A, seg_pos={0 0}, upload=645638]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:57434->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:57434->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57434->64.49.13.169:4715: use of closed network connection
Jan 5 09:20:43.837505115 (737h51m43.70858815s) [noise=true , address=64.49.13.169:4715 , piece_id=KKFCUQH62XP7TNYBQFEN72GZPRIMBX677IT5RHNRGEVMP3OG2F7A, seg_pos={0 0}, upload=645638]: Long tail cancellation due to optimal threshold
Jan 5 09:20:43.837582934 (737h51m43.708665979s) [noise=true , address=64.49.13.155:4918 , piece_id=WAMEGNFI6LPBIRJ5Z2HM5UED6Y7DPUC4FC56GFUM5MU3WGDNBVSA, seg_pos={0 0}, upload=645638]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YjPvRvhJkNJtmt5CJ7nUst9NicEGWzpe7Pn1GWes5uc9H1gTK): protocol: read tcp 188.166.225.170:43292->64.49.13.155:4918: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 5 09:20:43.837589787 (737h51m43.708672832s) [noise=true , address=64.49.13.155:4918 , piece_id=WAMEGNFI6LPBIRJ5Z2HM5UED6Y7DPUC4FC56GFUM5MU3WGDNBVSA, seg_pos={0 0}, upload=645638]: Long tail cancellation due to optimal threshold
Jan 5 09:20:43.837902242 (737h51m43.708985277s) [noise=true , address=77.90.31.17:4786 , piece_id=LAULWIVBRGXC6S6VX2U5LTLPXCMAE7KU2HCOXVJ4LFGQJLOLKNQA, seg_pos={0 0}, upload=645638]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128QrekLqpXLAJs7yqNziNLmt2i13SuMx1ZhMPdUpDosw71dgQh): protocol: expected piece hash; context canceled; EOF
Jan 5 09:20:43.837909375 (737h51m43.70899242s) [noise=true , address=77.90.31.17:4786 , piece_id=LAULWIVBRGXC6S6VX2U5LTLPXCMAE7KU2HCOXVJ4LFGQJLOLKNQA, seg_pos={0 0}, upload=645638]: Long tail cancellation due to optimal threshold
Jan 5 09:20:43.837927719 (737h51m43.709010754s) [seg_pos={0 0}, upload=645638]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jan 5 09:20:43.837996601 (737h51m43.709079636s) [upload=645638]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 5 09:20:43.838001461 (737h51m43.709084496s) [upload=645638]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 5 09:20:44.119746458 (737h51m43.990829493s) [upload=645638]: Done uploading segments...
Jan 5 09:20:44.119777267 (737h51m43.990860312s) [upload=645638]: Upload finished. err: <nil>
Duration: 43.241335761s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [.....X.............................................X........] duration:
17.752293606s trace:
https://jaeger.dev.storj.io/trace/35e6dfe601545a69
FAIL (2026-01-05 09:21:11 UTC)
orbiter warning: context deadline exceeded
Duration: 17.752321137s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
45 out of 60 is failed (threshold=6) [XXXXXX.XX..XXX.X.XX.XXX.XX.XXXXXX..XXXXXX...XXXX..XXXXXXXXXX] duration:
5.000842651s trace:
https://jaeger.dev.storj.io/trace/6a8ed939efbf89dc
FAIL (2026-01-05 09:21:06 UTC)
orbiter warning: context deadline exceeded
Duration: 5.00086828s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [............X..X...X........................................] duration:
20.482683498s trace:
https://jaeger.dev.storj.io/trace/7e43d4cb58eef6b9
FAIL (2026-01-05 09:21:03 UTC)
orbiter warning: context deadline exceeded
Duration: 20.48271558s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [......X.............................................X.......] duration:
21.035432891s trace:
https://jaeger.dev.storj.io/trace/35e6dfe601545a69
FAIL (2026-01-05 09:20:38 UTC)
orbiter warning: context deadline exceeded
Duration: 21.035467557s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
13 out of 60 is failed (threshold=3) [..X.X...X....................X..X..X.X.X.X..X...X...X..X....] duration:
6.174302591s trace:
https://jaeger.dev.storj.io/trace/6e7e86a0c49d3474
FAIL (2026-01-05 09:20:33 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Jan 5 09:17:54.838592341 (737h48m54.709675376s) [upload=645597]: Starting upload
Jan 5 09:17:54.838646896 (737h48m54.709729931s) [upload=645597]: Done starting upload
Jan 5 09:17:54.838710166 (737h48m54.709793201s) [upload=645597]: Uploading segments...
Jan 5 09:17:54.838864910 (737h48m54.709947955s) [upload=645597]: Got next segment. Inline: false
Jan 5 09:17:54.838898374 (737h48m54.709981409s) [upload=645597]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 5 09:17:54.838902923 (737h48m54.709985958s) [upload=645597]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 5 09:17:55.086410939 (737h48m54.957493974s) [seg_pos={0 0}, upload=645597]: Begin upload segment...
Jan 5 09:17:55.086467737 (737h48m54.957550772s) [seg_pos={0 0}, upload=645597]: Done begin upload segment.
Jan 5 09:17:55.086524434 (737h48m54.957607469s) [noise=true , address=145.79.227.2:4115 , piece_id=ATGM7WYGHXFWDN4BXJDTX34WZQSAG23TALV7XX3UBIA3CRNR6FTQ, seg_pos={0 0}, upload=645597]: Uploading piece...
Jan 5 09:17:55.086542950 (737h48m54.957625995s) [noise=true , address=145.79.227.52:4446 , piece_id=NSYEHINMINQWH4FYOFJ6CRGEITDV3BUN65J5AYSNWV4GKU2VBVXA, seg_pos={0 0}, upload=645597]: Uploading piece...
Jan 5 09:17:55.087164153 (737h48m54.958247188s) [noise=true , address=145.79.227.20:4437 , piece_id=ZG7MWW7CB2MBGAVRMGO4TUHYAIJKOSSI5GIQ3ZVD7WK7TKP4FTFQ, seg_pos={0 0}, upload=645597]: Uploading piece...
Jan 5 09:17:55.087585125 (737h48m54.95866817s) [noise=true , address=45.91.116.6:4975 , piece_id=BQWSC3Q3YHF72A57QEVHMFKKJBZI5CLJ46A7TP3EEO5RRCH3MJFA, seg_pos={0 0}, upload=645597]: Uploading piece...
Jan 5 09:17:55.087645098 (737h48m54.958728133s) [noise=true , address=45.91.116.14:4191 , piece_id=GOZG4DIDVDB6DJX6BRH74ZVCD5BQLYUTHMMFICGG5DXRTQBQOCEQ, seg_pos={0 0}, upload=645597]: Uploading piece...
Jan 5 09:17:55.087978223 (737h48m54.959061258s) [noise=true , address=145.79.227.38:4979 , piece_id=6CT2VJHMICL5XITOIOINR4KZKACBFN6JLBFQHCXIXWE7Q6ZMMICQ, seg_pos={0 0}, upload=645597]: Uploading piece...
Jan 5 09:17:55.088015594 (737h48m54.959098639s) [noise=true , address=145.79.227.6:4259 , piece_id=2N3EYFVAHSKP4XZBAQDZPZL3TPNKWDFEFOTHITQRTQTHULWZZLJA, seg_pos={0 0}, upload=645597]: Uploading piece...
Jan 5 09:17:55.088279256 (737h48m54.959362292s) [noise=true , address=77.90.31.19:4842 , piece_id=7Z2Y7NXZELYGUD4G5PVBLKRJZL3CLOF22ZREPSUGPT4JYJ4Y22PQ, seg_pos={0 0}, upload=645597]: Uploading piece...
Jan 5 09:17:55.088399405 (737h48m54.95948245s) [noise=true , address=77.90.31.10:4518 , piece_id=FSVPHUKXSS57ME2FIEDDDOLWN3YHQN5NE7BLUYM3S7BY5BICULMA, seg_pos={0 0}, upload=645597]: Uploading piece...
Jan 5 09:17:55.088619494 (737h48m54.959702529s) [noise=true , address=145.79.227.45:4230 , piece_id=YZHU7J2SMWXIME3FIPWG5QK3LFDTNHSBLB4WSSTRBJCWPBXW2NVA, seg_pos={0 0}, upload=645597]: Uploading piece...
Jan 5 09:17:55.088966394 (737h48m54.960049439s) [noise=true , address=145.79.227.24:4581 , piece_id=4G3SJ2KRQ3OISKABPCJ65C5G256IH6UAUCFZ4MEOADX2PHZUTAIA, seg_pos={0 0}, upload=645597]: Uploading piece...
Jan 5 09:17:55.089316721 (737h48m54.960399756s) [noise=true , address=145.79.227.18:4006 , piece_id=CPDP4Q5XGXRCB474UCZ6HEN77572EJDQF37ABK6BH5I7IRXYQVPQ, seg_pos={0 0}, upload=645597]: Uploading piece...
Jan 5 09:17:55.089776337 (737h48m54.960859372s) [noise=true , address=77.90.31.21:4880 , piece_id=KXUC3TEKHHO4VDM5TDJDECSV354232VGUC7SLNVT5GNJ3V3EFGCA, seg_pos={0 0}, upload=645597]: Uploading piece...
Jan 5 09:17:55.089850378 (737h48m54.960933413s) [noise=true , address=145.79.227.28:4727 , piece_id=4XXMSMSYFQAFMMKM24WTX4GVLTWIQAM3W62K646VN4MWVQCMWE6Q, seg_pos={0 0}, upload=645597]: Uploading piece...
Jan 5 09:17:55.090162883 (737h48m54.961245918s) [noise=true , address=45.91.116.13:4156 , piece_id=GM7AZGY2XU2YWVODUD373QFFFYLOLIFEZVY7UJBKKABKSEDFYH4Q, seg_pos={0 0}, upload=645597]: Uploading piece...
Jan 5 09:17:55.091675703 (737h48m54.962758738s) [noise=true , address=145.79.227.14:4838 , piece_id=5F2HPBJJJJ2WJHFFDVD55RMPGXKW7LY7U7QIEY4KZRLWTY3HDYYQ, seg_pos={0 0}, upload=645597]: Uploading piece...
Jan 5 09:17:56.794463178 (737h48m56.665546213s) [upload=645597]: Next returned nil segment
Jan 5 09:17:56.794479830 (737h48m56.665562865s) [upload=645597]: Waiting for error group managing segments...
Jan 5 09:17:57.076908991 (737h48m56.947992036s) [noise=true , address=145.79.227.24:4581 , piece_id=4G3SJ2KRQ3OISKABPCJ65C5G256IH6UAUCFZ4MEOADX2PHZUTAIA, seg_pos={0 0}, upload=645597]: Done uploading piece. err: <nil>
Jan 5 09:17:57.077566553 (737h48m56.948649598s) [noise=true , address=145.79.227.14:4838 , piece_id=5F2HPBJJJJ2WJHFFDVD55RMPGXKW7LY7U7QIEY4KZRLWTY3HDYYQ, seg_pos={0 0}, upload=645597]: Done uploading piece. err: <nil>
Jan 5 09:17:57.093208608 (737h48m56.964291653s) [noise=true , address=145.79.227.38:4979 , piece_id=6CT2VJHMICL5XITOIOINR4KZKACBFN6JLBFQHCXIXWE7Q6ZMMICQ, seg_pos={0 0}, upload=645597]: Done uploading piece. err: <nil>
Jan 5 09:17:57.097347419 (737h48m56.968430464s) [noise=true , address=145.79.227.28:4727 , piece_id=4XXMSMSYFQAFMMKM24WTX4GVLTWIQAM3W62K646VN4MWVQCMWE6Q, seg_pos={0 0}, upload=645597]: Done uploading piece. err: <nil>
Jan 5 09:17:57.097948074 (737h48m56.969031119s) [noise=true , address=145.79.227.2:4115 , piece_id=ATGM7WYGHXFWDN4BXJDTX34WZQSAG23TALV7XX3UBIA3CRNR6FTQ, seg_pos={0 0}, upload=645597]: Done uploading piece. err: <nil>
Jan 5 09:17:57.101168726 (737h48m56.972251771s) [noise=true , address=145.79.227.18:4006 , piece_id=CPDP4Q5XGXRCB474UCZ6HEN77572EJDQF37ABK6BH5I7IRXYQVPQ, seg_pos={0 0}, upload=645597]: Done uploading piece. err: <nil>
Jan 5 09:17:57.106539102 (737h48m56.977622137s) [noise=true , address=145.79.227.52:4446 , piece_id=NSYEHINMINQWH4FYOFJ6CRGEITDV3BUN65J5AYSNWV4GKU2VBVXA, seg_pos={0 0}, upload=645597]: Done uploading piece. err: <nil>
Jan 5 09:17:57.124938367 (737h48m56.996021413s) [noise=true , address=145.79.227.20:4437 , piece_id=ZG7MWW7CB2MBGAVRMGO4TUHYAIJKOSSI5GIQ3ZVD7WK7TKP4FTFQ, seg_pos={0 0}, upload=645597]: Done uploading piece. err: <nil>
Jan 5 09:17:57.142071331 (737h48m57.013154376s) [noise=true , address=145.79.227.6:4259 , piece_id=2N3EYFVAHSKP4XZBAQDZPZL3TPNKWDFEFOTHITQRTQTHULWZZLJA, seg_pos={0 0}, upload=645597]: Done uploading piece. err: <nil>
Jan 5 09:17:57.175080546 (737h48m57.046163581s) [noise=true , address=145.79.227.45:4230 , piece_id=YZHU7J2SMWXIME3FIPWG5QK3LFDTNHSBLB4WSSTRBJCWPBXW2NVA, seg_pos={0 0}, upload=645597]: Done uploading piece. err: <nil>
Jan 5 09:17:58.461065682 (737h48m58.332148717s) [noise=true , address=45.91.116.14:4191 , piece_id=GOZG4DIDVDB6DJX6BRH74ZVCD5BQLYUTHMMFICGG5DXRTQBQOCEQ, seg_pos={0 0}, upload=645597]: Done uploading piece. err: <nil>
Jan 5 09:17:59.174407502 (737h48m59.045490547s) [noise=true , address=45.91.116.6:4975 , piece_id=BQWSC3Q3YHF72A57QEVHMFKKJBZI5CLJ46A7TP3EEO5RRCH3MJFA, seg_pos={0 0}, upload=645597]: Done uploading piece. err: <nil>
Jan 5 09:18:03.798756456 (737h49m3.669839501s) [noise=true , address=77.90.31.19:4842 , piece_id=7Z2Y7NXZELYGUD4G5PVBLKRJZL3CLOF22ZREPSUGPT4JYJ4Y22PQ, seg_pos={0 0}, upload=645597]: Done uploading piece. err: <nil>
Jan 5 09:18:03.798796051 (737h49m3.669879086s) [seg_pos={0 0}, upload=645597]: Segment reached optimal threshold of 13 pieces.
Jan 5 09:18:03.799105350 (737h49m3.670188396s) [noise=true , address=77.90.31.21:4880 , piece_id=KXUC3TEKHHO4VDM5TDJDECSV354232VGUC7SLNVT5GNJ3V3EFGCA, seg_pos={0 0}, upload=645597]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF
Jan 5 09:18:03.799115099 (737h49m3.670198144s) [noise=true , address=77.90.31.21:4880 , piece_id=KXUC3TEKHHO4VDM5TDJDECSV354232VGUC7SLNVT5GNJ3V3EFGCA, seg_pos={0 0}, upload=645597]: Long tail cancellation due to optimal threshold
Jan 5 09:18:03.799379172 (737h49m3.670462217s) [noise=true , address=77.90.31.10:4518 , piece_id=FSVPHUKXSS57ME2FIEDDDOLWN3YHQN5NE7BLUYM3S7BY5BICULMA, seg_pos={0 0}, upload=645597]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: read tcp 188.166.225.170:41460->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 5 09:18:03.799415481 (737h49m3.670498516s) [noise=true , address=77.90.31.10:4518 , piece_id=FSVPHUKXSS57ME2FIEDDDOLWN3YHQN5NE7BLUYM3S7BY5BICULMA, seg_pos={0 0}, upload=645597]: Long tail cancellation due to optimal threshold
Jan 5 09:18:03.799559655 (737h49m3.670642701s) [noise=true , address=45.91.116.13:4156 , piece_id=GM7AZGY2XU2YWVODUD373QFFFYLOLIFEZVY7UJBKKABKSEDFYH4Q, seg_pos={0 0}, upload=645597]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: read tcp 188.166.225.170:34200->45.91.116.13:4156: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 5 09:18:03.799570056 (737h49m3.670653091s) [noise=true , address=45.91.116.13:4156 , piece_id=GM7AZGY2XU2YWVODUD373QFFFYLOLIFEZVY7UJBKKABKSEDFYH4Q, seg_pos={0 0}, upload=645597]: Long tail cancellation due to optimal threshold
Jan 5 09:18:03.799597147 (737h49m3.670680192s) [seg_pos={0 0}, upload=645597]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 5 09:18:03.799692008 (737h49m3.670775053s) [upload=645597]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 5 09:18:03.799698189 (737h49m3.670781234s) [upload=645597]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 5 09:18:04.095750160 (737h49m3.966833205s) [upload=645597]: Done uploading segments...
Jan 5 09:18:04.095772302 (737h49m3.966855347s) [upload=645597]: Upload finished. err: <nil>
Duration: 6.174331337s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
28 out of 60 is failed (threshold=3) [XX.XXXXXX.XXX...X....XX.X.X....XX....X..XX.XX......X.X.X..XX] duration:
32.407952425s trace:
https://jaeger.dev.storj.io/trace/799c7d7f57339def
FAIL (2026-01-05 09:20:19 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
8:4016 , piece_id=EW4JDJC2J3VQMLMYWJAJL4KB44JKHSQSFQVR772EI5XUXKSBRMSA, seg_pos={0 0}, upload=645625]: Uploading piece...
Jan 5 09:19:47.730660297 (737h50m47.601743332s) [noise=true , address=77.90.31.21:4893 , piece_id=O25AZ7ZY2C2772T6PZSRD7GFBEJKBT3Q4SQDQH4AUA7ZDKDIZJKA, seg_pos={0 0}, upload=645625]: Uploading piece...
Jan 5 09:19:47.731003811 (737h50m47.602086856s) [noise=true , address=145.79.227.44:4200 , piece_id=2NHBZTZ6IQLBZXRCG7IQ6WNQ2WWIHII2G7XH523ILT2V2TDTJUPA, seg_pos={0 0}, upload=645625]: Uploading piece...
Jan 5 09:19:47.731370048 (737h50m47.602453083s) [noise=true , address=64.49.13.155:4915 , piece_id=HS5PYFM2P7TC7FEXA2AIGC77NQURHKFGIJVXQTIODA5DOSGOMI7Q, seg_pos={0 0}, upload=645625]: Uploading piece...
Jan 5 09:19:47.731740604 (737h50m47.602823649s) [noise=true , address=145.79.227.40:4052 , piece_id=J2PARIPQ3HRGI2XHT2FCMJGKW2V7JTAZRKZUQG5G47Y2VJIHBGYQ, seg_pos={0 0}, upload=645625]: Uploading piece...
Jan 5 09:19:47.732135916 (737h50m47.603218951s) [noise=true , address=77.90.31.14:4680 , piece_id=BBUTSFUHKJ3YMGGZRLCVFDV7KTGCNVAXQJ3MLE5ROJVWF7AOMAEQ, seg_pos={0 0}, upload=645625]: Uploading piece...
Jan 5 09:19:47.733754990 (737h50m47.604838025s) [noise=true , address=145.79.227.23:4576 , piece_id=WX44YPF46IFLMBVXLX4ALOLWSMKARXBYFZ44FOJJKP64TGT7BZMQ, seg_pos={0 0}, upload=645625]: Uploading piece...
Jan 5 09:19:47.735760778 (737h50m47.606843813s) [noise=true , address=145.79.227.52:4452 , piece_id=M5BNCVIW46VXRC2ICBRVQYGBH2VM2EOW6WN6JUASLICHDIAJRWSQ, seg_pos={0 0}, upload=645625]: Uploading piece...
Jan 5 09:19:47.739658771 (737h50m47.610741806s) [noise=true , address=145.79.227.33:4878 , piece_id=WFFI34HNFK2DOZHV7RF4K7XGSD2TMSOH4AX2EOK24YRJAPHCU26Q, seg_pos={0 0}, upload=645625]: Uploading piece...
Jan 5 09:19:47.742601575 (737h50m47.61368461s) [noise=true , address=64.49.13.169:4705 , piece_id=CIH4K3PLWRRX5YSATKDOFOMPBLSHSJ4A7RYGMQMLQQDMM47ZQ2YQ, seg_pos={0 0}, upload=645625]: Uploading piece...
Jan 5 09:19:47.743952748 (737h50m47.615035783s) [noise=true , address=145.79.227.35:4494 , piece_id=TJLDWC2O622E2JLTG7K3OXIN3G6QUTLD4YZVKOQM7VO4RBK7XW5Q, seg_pos={0 0}, upload=645625]: Uploading piece...
Jan 5 09:19:47.746488557 (737h50m47.617571592s) [noise=true , address=193.149.164.143:4702 , piece_id=FQR7XUR5GGGLESR2UO6DU3CQL7V54ZVB5RX44VTAWKCUV3R6D6VQ, seg_pos={0 0}, upload=645625]: Uploading piece...
Jan 5 09:19:47.749534466 (737h50m47.620617511s) [noise=true , address=145.79.227.29:4957 , piece_id=CKS5E6RBORRZW5IBO5Y36SCHUGLF4JUMIZZGJVR4SB3JSNFVAAHA, seg_pos={0 0}, upload=645625]: Uploading piece...
Jan 5 09:19:47.749891436 (737h50m47.620974471s) [noise=true , address=145.79.227.34:4933 , piece_id=YHPP2SDSKCTHAPA6N4CZFEV5WTWCE4OHDU35C7YCNEDMN2ZRKAZQ, seg_pos={0 0}, upload=645625]: Uploading piece...
Jan 5 09:19:47.751422291 (737h50m47.622505326s) [noise=true , address=45.91.116.11:4139 , piece_id=UZ54PHSVP54G4VS7R5BKWJ4OQTGV5T6HLHOIKD3PADQX5ZAIAFLQ, seg_pos={0 0}, upload=645625]: Uploading piece...
Jan 5 09:19:49.197591955 (737h50m49.06867499s) [upload=645625]: Next returned nil segment
Jan 5 09:19:49.197695632 (737h50m49.068778667s) [upload=645625]: Waiting for error group managing segments...
Jan 5 09:19:49.589273370 (737h50m49.460356415s) [noise=true , address=64.49.13.155:4915 , piece_id=HS5PYFM2P7TC7FEXA2AIGC77NQURHKFGIJVXQTIODA5DOSGOMI7Q, seg_pos={0 0}, upload=645625]: Done uploading piece. err: <nil>
Jan 5 09:19:49.592405905 (737h50m49.46348895s) [noise=true , address=145.79.227.40:4052 , piece_id=J2PARIPQ3HRGI2XHT2FCMJGKW2V7JTAZRKZUQG5G47Y2VJIHBGYQ, seg_pos={0 0}, upload=645625]: Done uploading piece. err: <nil>
Jan 5 09:19:49.608617143 (737h50m49.479700178s) [noise=true , address=145.79.227.52:4452 , piece_id=M5BNCVIW46VXRC2ICBRVQYGBH2VM2EOW6WN6JUASLICHDIAJRWSQ, seg_pos={0 0}, upload=645625]: Done uploading piece. err: <nil>
Jan 5 09:19:49.620529957 (737h50m49.491613002s) [noise=true , address=145.79.227.23:4576 , piece_id=WX44YPF46IFLMBVXLX4ALOLWSMKARXBYFZ44FOJJKP64TGT7BZMQ, seg_pos={0 0}, upload=645625]: Done uploading piece. err: <nil>
Jan 5 09:19:49.664048303 (737h50m49.535131338s) [noise=true , address=145.79.227.18:4016 , piece_id=EW4JDJC2J3VQMLMYWJAJL4KB44JKHSQSFQVR772EI5XUXKSBRMSA, seg_pos={0 0}, upload=645625]: Done uploading piece. err: <nil>
Jan 5 09:19:49.690155696 (737h50m49.561238741s) [noise=true , address=145.79.227.33:4878 , piece_id=WFFI34HNFK2DOZHV7RF4K7XGSD2TMSOH4AX2EOK24YRJAPHCU26Q, seg_pos={0 0}, upload=645625]: Done uploading piece. err: <nil>
Jan 5 09:19:49.885723469 (737h50m49.756806514s) [noise=true , address=77.90.31.21:4893 , piece_id=O25AZ7ZY2C2772T6PZSRD7GFBEJKBT3Q4SQDQH4AUA7ZDKDIZJKA, seg_pos={0 0}, upload=645625]: Done uploading piece. err: <nil>
Jan 5 09:19:50.062698609 (737h50m49.933781644s) [noise=true , address=45.91.116.5:4324 , piece_id=HLQ2X74RCDY5TGRXSS6V3LEZN6HUVVP6PNBHIAGTX226SOI6QXDQ, seg_pos={0 0}, upload=645625]: Done uploading piece. err: <nil>
Jan 5 09:19:50.528107694 (737h50m50.399190729s) [noise=true , address=145.79.227.30:4773 , piece_id=XUBU5NUNU2XHDQBPNSKIHV657VGEHYQKMTRRXSMSL5KG6OBGVWKA, seg_pos={0 0}, upload=645625]: Done uploading piece. err: <nil>
Jan 5 09:19:50.528893611 (737h50m50.399976646s) [noise=true , address=145.79.227.44:4200 , piece_id=2NHBZTZ6IQLBZXRCG7IQ6WNQ2WWIHII2G7XH523ILT2V2TDTJUPA, seg_pos={0 0}, upload=645625]: Done uploading piece. err: <nil>
Jan 5 09:19:50.568636217 (737h50m50.439719262s) [noise=true , address=145.79.227.35:4494 , piece_id=TJLDWC2O622E2JLTG7K3OXIN3G6QUTLD4YZVKOQM7VO4RBK7XW5Q, seg_pos={0 0}, upload=645625]: Done uploading piece. err: <nil>
Jan 5 09:19:50.654573095 (737h50m50.52565613s) [noise=true , address=145.79.227.49:4372 , piece_id=4Q46YWOMB5ERZZVHB5G6VYJOH7DZ3QAHVHRNBYFWZKCVLYBGBLJQ, seg_pos={0 0}, upload=645625]: Done uploading piece. err: <nil>
Jan 5 09:19:50.666652055 (737h50m50.5377351s) [noise=true , address=145.79.227.34:4933 , piece_id=YHPP2SDSKCTHAPA6N4CZFEV5WTWCE4OHDU35C7YCNEDMN2ZRKAZQ, seg_pos={0 0}, upload=645625]: Done uploading piece. err: <nil>
Jan 5 09:19:50.897865803 (737h50m50.768948838s) [noise=true , address=45.91.116.11:4139 , piece_id=UZ54PHSVP54G4VS7R5BKWJ4OQTGV5T6HLHOIKD3PADQX5ZAIAFLQ, seg_pos={0 0}, upload=645625]: Done uploading piece. err: <nil>
Jan 5 09:19:50.897885431 (737h50m50.768968476s) [seg_pos={0 0}, upload=645625]: Segment reached optimal threshold of 14 pieces.
Jan 5 09:19:50.898082996 (737h50m50.769166042s) [noise=true , address=77.90.31.14:4680 , piece_id=BBUTSFUHKJ3YMGGZRLCVFDV7KTGCNVAXQJ3MLE5ROJVWF7AOMAEQ, seg_pos={0 0}, upload=645625]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LqXx4vZczneGXWUEHnoMRYBZoaKRFQyhcU7LfKUoeSkP3JTd1): protocol: write tcp 188.166.225.170:50326->77.90.31.14:4680: use of closed network connection; write tcp 188.166.225.170:50326->77.90.31.14:4680: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50326->77.90.31.14:4680: use of closed network connection
Jan 5 09:19:50.898089480 (737h50m50.769172515s) [noise=true , address=77.90.31.14:4680 , piece_id=BBUTSFUHKJ3YMGGZRLCVFDV7KTGCNVAXQJ3MLE5ROJVWF7AOMAEQ, seg_pos={0 0}, upload=645625]: Long tail cancellation due to optimal threshold
Jan 5 09:19:50.898307785 (737h50m50.76939082s) [noise=true , address=193.149.164.143:4702 , piece_id=FQR7XUR5GGGLESR2UO6DU3CQL7V54ZVB5RX44VTAWKCUV3R6D6VQ, seg_pos={0 0}, upload=645625]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cTKvv93eK129pGYE3ZXt81vByFHzwJjesrkYWFAjXvSK3dTC1): protocol: expected piece hash; context canceled; EOF
Jan 5 09:19:50.898313876 (737h50m50.769396922s) [noise=true , address=193.149.164.143:4702 , piece_id=FQR7XUR5GGGLESR2UO6DU3CQL7V54ZVB5RX44VTAWKCUV3R6D6VQ, seg_pos={0 0}, upload=645625]: Long tail cancellation due to optimal threshold
Jan 5 09:19:50.898440348 (737h50m50.769523383s) [noise=true , address=64.49.13.169:4705 , piece_id=CIH4K3PLWRRX5YSATKDOFOMPBLSHSJ4A7RYGMQMLQQDMM47ZQ2YQ, seg_pos={0 0}, upload=645625]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:54026->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:54026->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54026->64.49.13.169:4705: use of closed network connection
Jan 5 09:19:50.898447481 (737h50m50.769530516s) [noise=true , address=64.49.13.169:4705 , piece_id=CIH4K3PLWRRX5YSATKDOFOMPBLSHSJ4A7RYGMQMLQQDMM47ZQ2YQ, seg_pos={0 0}, upload=645625]: Long tail cancellation due to optimal threshold
Jan 5 09:19:50.898573131 (737h50m50.769656176s) [noise=true , address=145.79.227.29:4957 , piece_id=CKS5E6RBORRZW5IBO5Y36SCHUGLF4JUMIZZGJVR4SB3JSNFVAAHA, seg_pos={0 0}, upload=645625]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129iWqwdULbj7gs2Rd5xjEQn1Mgp6EDVZyFpSaoAZraRwCveyQn): protocol: expected piece hash; context canceled; EOF
Jan 5 09:19:50.898595022 (737h50m50.769678077s) [noise=true , address=145.79.227.29:4957 , piece_id=CKS5E6RBORRZW5IBO5Y36SCHUGLF4JUMIZZGJVR4SB3JSNFVAAHA, seg_pos={0 0}, upload=645625]: Long tail cancellation due to optimal threshold
Jan 5 09:19:50.898632834 (737h50m50.769715879s) [seg_pos={0 0}, upload=645625]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jan 5 09:19:50.898790544 (737h50m50.769873589s) [upload=645625]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 5 09:19:50.898797768 (737h50m50.769880813s) [upload=645625]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 5 09:19:51.191441710 (737h50m51.062524745s) [upload=645625]: Done uploading segments...
Jan 5 09:19:51.191467559 (737h50m51.062550594s) [upload=645625]: Upload finished. err: <nil>
Duration: 32.407993553s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
45 out of 60 is failed (threshold=6) [XXXXXXX.XX..XXX.X.XX.XXX.XX.XXXXXX..XXXXXX...XXXX..XXXXXXXXX] duration:
5.000939767s trace:
https://jaeger.dev.storj.io/trace/3c2ac4927556a6e8
FAIL (2026-01-05 09:20:09 UTC)
orbiter warning: context deadline exceeded
Duration: 5.000966818s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
13 out of 60 is failed (threshold=3) [...X.X...X....................X..X..X.X.X.X..X...X...X..X...] duration:
5.700354837s trace:
https://jaeger.dev.storj.io/trace/6e7e86a0c49d3474
FAIL (2026-01-05 09:20:02 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Jan 5 09:17:54.838592341 (737h48m54.709675376s) [upload=645597]: Starting upload
Jan 5 09:17:54.838646896 (737h48m54.709729931s) [upload=645597]: Done starting upload
Jan 5 09:17:54.838710166 (737h48m54.709793201s) [upload=645597]: Uploading segments...
Jan 5 09:17:54.838864910 (737h48m54.709947955s) [upload=645597]: Got next segment. Inline: false
Jan 5 09:17:54.838898374 (737h48m54.709981409s) [upload=645597]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 5 09:17:54.838902923 (737h48m54.709985958s) [upload=645597]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 5 09:17:55.086410939 (737h48m54.957493974s) [seg_pos={0 0}, upload=645597]: Begin upload segment...
Jan 5 09:17:55.086467737 (737h48m54.957550772s) [seg_pos={0 0}, upload=645597]: Done begin upload segment.
Jan 5 09:17:55.086524434 (737h48m54.957607469s) [noise=true , address=145.79.227.2:4115 , piece_id=ATGM7WYGHXFWDN4BXJDTX34WZQSAG23TALV7XX3UBIA3CRNR6FTQ, seg_pos={0 0}, upload=645597]: Uploading piece...
Jan 5 09:17:55.086542950 (737h48m54.957625995s) [noise=true , address=145.79.227.52:4446 , piece_id=NSYEHINMINQWH4FYOFJ6CRGEITDV3BUN65J5AYSNWV4GKU2VBVXA, seg_pos={0 0}, upload=645597]: Uploading piece...
Jan 5 09:17:55.087164153 (737h48m54.958247188s) [noise=true , address=145.79.227.20:4437 , piece_id=ZG7MWW7CB2MBGAVRMGO4TUHYAIJKOSSI5GIQ3ZVD7WK7TKP4FTFQ, seg_pos={0 0}, upload=645597]: Uploading piece...
Jan 5 09:17:55.087585125 (737h48m54.95866817s) [noise=true , address=45.91.116.6:4975 , piece_id=BQWSC3Q3YHF72A57QEVHMFKKJBZI5CLJ46A7TP3EEO5RRCH3MJFA, seg_pos={0 0}, upload=645597]: Uploading piece...
Jan 5 09:17:55.087645098 (737h48m54.958728133s) [noise=true , address=45.91.116.14:4191 , piece_id=GOZG4DIDVDB6DJX6BRH74ZVCD5BQLYUTHMMFICGG5DXRTQBQOCEQ, seg_pos={0 0}, upload=645597]: Uploading piece...
Jan 5 09:17:55.087978223 (737h48m54.959061258s) [noise=true , address=145.79.227.38:4979 , piece_id=6CT2VJHMICL5XITOIOINR4KZKACBFN6JLBFQHCXIXWE7Q6ZMMICQ, seg_pos={0 0}, upload=645597]: Uploading piece...
Jan 5 09:17:55.088015594 (737h48m54.959098639s) [noise=true , address=145.79.227.6:4259 , piece_id=2N3EYFVAHSKP4XZBAQDZPZL3TPNKWDFEFOTHITQRTQTHULWZZLJA, seg_pos={0 0}, upload=645597]: Uploading piece...
Jan 5 09:17:55.088279256 (737h48m54.959362292s) [noise=true , address=77.90.31.19:4842 , piece_id=7Z2Y7NXZELYGUD4G5PVBLKRJZL3CLOF22ZREPSUGPT4JYJ4Y22PQ, seg_pos={0 0}, upload=645597]: Uploading piece...
Jan 5 09:17:55.088399405 (737h48m54.95948245s) [noise=true , address=77.90.31.10:4518 , piece_id=FSVPHUKXSS57ME2FIEDDDOLWN3YHQN5NE7BLUYM3S7BY5BICULMA, seg_pos={0 0}, upload=645597]: Uploading piece...
Jan 5 09:17:55.088619494 (737h48m54.959702529s) [noise=true , address=145.79.227.45:4230 , piece_id=YZHU7J2SMWXIME3FIPWG5QK3LFDTNHSBLB4WSSTRBJCWPBXW2NVA, seg_pos={0 0}, upload=645597]: Uploading piece...
Jan 5 09:17:55.088966394 (737h48m54.960049439s) [noise=true , address=145.79.227.24:4581 , piece_id=4G3SJ2KRQ3OISKABPCJ65C5G256IH6UAUCFZ4MEOADX2PHZUTAIA, seg_pos={0 0}, upload=645597]: Uploading piece...
Jan 5 09:17:55.089316721 (737h48m54.960399756s) [noise=true , address=145.79.227.18:4006 , piece_id=CPDP4Q5XGXRCB474UCZ6HEN77572EJDQF37ABK6BH5I7IRXYQVPQ, seg_pos={0 0}, upload=645597]: Uploading piece...
Jan 5 09:17:55.089776337 (737h48m54.960859372s) [noise=true , address=77.90.31.21:4880 , piece_id=KXUC3TEKHHO4VDM5TDJDECSV354232VGUC7SLNVT5GNJ3V3EFGCA, seg_pos={0 0}, upload=645597]: Uploading piece...
Jan 5 09:17:55.089850378 (737h48m54.960933413s) [noise=true , address=145.79.227.28:4727 , piece_id=4XXMSMSYFQAFMMKM24WTX4GVLTWIQAM3W62K646VN4MWVQCMWE6Q, seg_pos={0 0}, upload=645597]: Uploading piece...
Jan 5 09:17:55.090162883 (737h48m54.961245918s) [noise=true , address=45.91.116.13:4156 , piece_id=GM7AZGY2XU2YWVODUD373QFFFYLOLIFEZVY7UJBKKABKSEDFYH4Q, seg_pos={0 0}, upload=645597]: Uploading piece...
Jan 5 09:17:55.091675703 (737h48m54.962758738s) [noise=true , address=145.79.227.14:4838 , piece_id=5F2HPBJJJJ2WJHFFDVD55RMPGXKW7LY7U7QIEY4KZRLWTY3HDYYQ, seg_pos={0 0}, upload=645597]: Uploading piece...
Jan 5 09:17:56.794463178 (737h48m56.665546213s) [upload=645597]: Next returned nil segment
Jan 5 09:17:56.794479830 (737h48m56.665562865s) [upload=645597]: Waiting for error group managing segments...
Jan 5 09:17:57.076908991 (737h48m56.947992036s) [noise=true , address=145.79.227.24:4581 , piece_id=4G3SJ2KRQ3OISKABPCJ65C5G256IH6UAUCFZ4MEOADX2PHZUTAIA, seg_pos={0 0}, upload=645597]: Done uploading piece. err: <nil>
Jan 5 09:17:57.077566553 (737h48m56.948649598s) [noise=true , address=145.79.227.14:4838 , piece_id=5F2HPBJJJJ2WJHFFDVD55RMPGXKW7LY7U7QIEY4KZRLWTY3HDYYQ, seg_pos={0 0}, upload=645597]: Done uploading piece. err: <nil>
Jan 5 09:17:57.093208608 (737h48m56.964291653s) [noise=true , address=145.79.227.38:4979 , piece_id=6CT2VJHMICL5XITOIOINR4KZKACBFN6JLBFQHCXIXWE7Q6ZMMICQ, seg_pos={0 0}, upload=645597]: Done uploading piece. err: <nil>
Jan 5 09:17:57.097347419 (737h48m56.968430464s) [noise=true , address=145.79.227.28:4727 , piece_id=4XXMSMSYFQAFMMKM24WTX4GVLTWIQAM3W62K646VN4MWVQCMWE6Q, seg_pos={0 0}, upload=645597]: Done uploading piece. err: <nil>
Jan 5 09:17:57.097948074 (737h48m56.969031119s) [noise=true , address=145.79.227.2:4115 , piece_id=ATGM7WYGHXFWDN4BXJDTX34WZQSAG23TALV7XX3UBIA3CRNR6FTQ, seg_pos={0 0}, upload=645597]: Done uploading piece. err: <nil>
Jan 5 09:17:57.101168726 (737h48m56.972251771s) [noise=true , address=145.79.227.18:4006 , piece_id=CPDP4Q5XGXRCB474UCZ6HEN77572EJDQF37ABK6BH5I7IRXYQVPQ, seg_pos={0 0}, upload=645597]: Done uploading piece. err: <nil>
Jan 5 09:17:57.106539102 (737h48m56.977622137s) [noise=true , address=145.79.227.52:4446 , piece_id=NSYEHINMINQWH4FYOFJ6CRGEITDV3BUN65J5AYSNWV4GKU2VBVXA, seg_pos={0 0}, upload=645597]: Done uploading piece. err: <nil>
Jan 5 09:17:57.124938367 (737h48m56.996021413s) [noise=true , address=145.79.227.20:4437 , piece_id=ZG7MWW7CB2MBGAVRMGO4TUHYAIJKOSSI5GIQ3ZVD7WK7TKP4FTFQ, seg_pos={0 0}, upload=645597]: Done uploading piece. err: <nil>
Jan 5 09:17:57.142071331 (737h48m57.013154376s) [noise=true , address=145.79.227.6:4259 , piece_id=2N3EYFVAHSKP4XZBAQDZPZL3TPNKWDFEFOTHITQRTQTHULWZZLJA, seg_pos={0 0}, upload=645597]: Done uploading piece. err: <nil>
Jan 5 09:17:57.175080546 (737h48m57.046163581s) [noise=true , address=145.79.227.45:4230 , piece_id=YZHU7J2SMWXIME3FIPWG5QK3LFDTNHSBLB4WSSTRBJCWPBXW2NVA, seg_pos={0 0}, upload=645597]: Done uploading piece. err: <nil>
Jan 5 09:17:58.461065682 (737h48m58.332148717s) [noise=true , address=45.91.116.14:4191 , piece_id=GOZG4DIDVDB6DJX6BRH74ZVCD5BQLYUTHMMFICGG5DXRTQBQOCEQ, seg_pos={0 0}, upload=645597]: Done uploading piece. err: <nil>
Jan 5 09:17:59.174407502 (737h48m59.045490547s) [noise=true , address=45.91.116.6:4975 , piece_id=BQWSC3Q3YHF72A57QEVHMFKKJBZI5CLJ46A7TP3EEO5RRCH3MJFA, seg_pos={0 0}, upload=645597]: Done uploading piece. err: <nil>
Jan 5 09:18:03.798756456 (737h49m3.669839501s) [noise=true , address=77.90.31.19:4842 , piece_id=7Z2Y7NXZELYGUD4G5PVBLKRJZL3CLOF22ZREPSUGPT4JYJ4Y22PQ, seg_pos={0 0}, upload=645597]: Done uploading piece. err: <nil>
Jan 5 09:18:03.798796051 (737h49m3.669879086s) [seg_pos={0 0}, upload=645597]: Segment reached optimal threshold of 13 pieces.
Jan 5 09:18:03.799105350 (737h49m3.670188396s) [noise=true , address=77.90.31.21:4880 , piece_id=KXUC3TEKHHO4VDM5TDJDECSV354232VGUC7SLNVT5GNJ3V3EFGCA, seg_pos={0 0}, upload=645597]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF
Jan 5 09:18:03.799115099 (737h49m3.670198144s) [noise=true , address=77.90.31.21:4880 , piece_id=KXUC3TEKHHO4VDM5TDJDECSV354232VGUC7SLNVT5GNJ3V3EFGCA, seg_pos={0 0}, upload=645597]: Long tail cancellation due to optimal threshold
Jan 5 09:18:03.799379172 (737h49m3.670462217s) [noise=true , address=77.90.31.10:4518 , piece_id=FSVPHUKXSS57ME2FIEDDDOLWN3YHQN5NE7BLUYM3S7BY5BICULMA, seg_pos={0 0}, upload=645597]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: read tcp 188.166.225.170:41460->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 5 09:18:03.799415481 (737h49m3.670498516s) [noise=true , address=77.90.31.10:4518 , piece_id=FSVPHUKXSS57ME2FIEDDDOLWN3YHQN5NE7BLUYM3S7BY5BICULMA, seg_pos={0 0}, upload=645597]: Long tail cancellation due to optimal threshold
Jan 5 09:18:03.799559655 (737h49m3.670642701s) [noise=true , address=45.91.116.13:4156 , piece_id=GM7AZGY2XU2YWVODUD373QFFFYLOLIFEZVY7UJBKKABKSEDFYH4Q, seg_pos={0 0}, upload=645597]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: read tcp 188.166.225.170:34200->45.91.116.13:4156: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 5 09:18:03.799570056 (737h49m3.670653091s) [noise=true , address=45.91.116.13:4156 , piece_id=GM7AZGY2XU2YWVODUD373QFFFYLOLIFEZVY7UJBKKABKSEDFYH4Q, seg_pos={0 0}, upload=645597]: Long tail cancellation due to optimal threshold
Jan 5 09:18:03.799597147 (737h49m3.670680192s) [seg_pos={0 0}, upload=645597]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 5 09:18:03.799692008 (737h49m3.670775053s) [upload=645597]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 5 09:18:03.799698189 (737h49m3.670781234s) [upload=645597]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 5 09:18:04.095750160 (737h49m3.966833205s) [upload=645597]: Done uploading segments...
Jan 5 09:18:04.095772302 (737h49m3.966855347s) [upload=645597]: Upload finished. err: <nil>
Duration: 5.700391696s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [X............X..X...X.......................................] duration:
19.912932197s trace:
https://jaeger.dev.storj.io/trace/7e43d4cb58eef6b9
FAIL (2026-01-05 09:19:48 UTC)
orbiter warning: context deadline exceeded
Duration: 19.912958959s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
27 out of 60 is failed (threshold=3) [.XX.XXXXXX.XXX...X....XX.X.X....XX....X..XX.XX......X.X.X..X] duration:
1m0.003583674s trace:
https://jaeger.dev.storj.io/trace/3fd5318b857c33db
FAIL (2026-01-05 09:19:46 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1RXPfJXrWzrCxzxRUk34LWLVuuwHHud41qVjn58qAwz7QiScSn, Piece ID: QZVDAA4HPU2C6SKRGGQB2CLSQDVPDOBPUIRNFIZUHSFAXV4GOCSQ) context deadline exceeded; piecestore: (Node ID: 1jwTDXPShXfNdfMx3UXi22gQphBpFre1MhRsHyjq6xRz4wwTr9, Piece ID: ZIVM26ZK7N3OY5XJB75QMXIBRG4Y6363ZHI5SNYGPTCJECPTUNRQ) context deadline exceeded; piecestore: (Node ID: 1vQJmv9n2yWXZtCCaABrxkB3NN6vDwLdiBkRibDN3t6YNmcmGP, Piece ID: RUXUL5DV36RIJMXDRJ7WKUEW2OV3E3BGYZDPJUSYCPC2EIJFHE7A) context deadline exceeded
======LOGS======
.502006021 (737h49m47.373089066s) [noise=true , address=145.79.227.6:4288 , piece_id=TCDAPURTJLJXCXGOQ7LHOB2HKLD64U26LE4RRMDPGZLXL4C4YMQA, seg_pos={0 0}, upload=645609]: Uploading piece...
Jan 5 09:18:47.502294029 (737h49m47.373377064s) [noise=true , address=145.79.227.49:4353 , piece_id=Y3ZVY7FA2Z7JQ6XGOWNFIMWAXJ26POBBQ3PQ3QRZJENTOVLR7DJA, seg_pos={0 0}, upload=645609]: Uploading piece...
Jan 5 09:18:47.502539767 (737h49m47.373622812s) [noise=true , address=145.79.227.11:4748 , piece_id=5DQHX2J4LHFASTZ4FYW55XQO7VCKRBXH7K3XBMTRM7KW4WBO62MQ, seg_pos={0 0}, upload=645609]: Uploading piece...
Jan 5 09:18:47.502785054 (737h49m47.373868089s) [noise=true , address=45.91.116.9:4281 , piece_id=LA6VU4KYZ5AYCIOCWC67SF65AGAAN2NV6TOPQSJJHVPHYKIIOZWQ, seg_pos={0 0}, upload=645609]: Uploading piece...
Jan 5 09:18:47.503021324 (737h49m47.374104369s) [noise=true , address=145.79.227.30:4782 , piece_id=7FXJ7VS73CNUONPHSIOOKOMTFSLTDCJAP56GUPFTIBBYWUYWOK5A, seg_pos={0 0}, upload=645609]: Uploading piece...
Jan 5 09:18:47.503255941 (737h49m47.374338976s) [noise=true , address=145.79.227.44:4200 , piece_id=G4MWFV53PBS7SGPCHLQ35NOYRTT6MS6L7ISTC6PONIKTWZFVPLIA, seg_pos={0 0}, upload=645609]: Uploading piece...
Jan 5 09:18:47.503490448 (737h49m47.374573493s) [noise=true , address=77.90.31.18:4828 , piece_id=ZIVM26ZK7N3OY5XJB75QMXIBRG4Y6363ZHI5SNYGPTCJECPTUNRQ, seg_pos={0 0}, upload=645609]: Uploading piece...
Jan 5 09:18:47.503729293 (737h49m47.374812328s) [noise=true , address=64.49.13.138:4492 , piece_id=TDF7OSOXDTCJWANOSDKDDZSZAY6UQOA4UKVUFOFZYHVT66OUHP4A, seg_pos={0 0}, upload=645609]: Uploading piece...
Jan 5 09:18:47.503970611 (737h49m47.375053646s) [noise=true , address=145.79.227.34:4925 , piece_id=ZYAAUFNQTG7AJ7THT5VKV4SSFMKXAHVSKCBVO276U3C6EYHIXGPA, seg_pos={0 0}, upload=645609]: Uploading piece...
Jan 5 09:18:47.504214466 (737h49m47.375297501s) [noise=true , address=145.79.227.36:4526 , piece_id=F2NTSXKTTZITSM6KGT6IZZMZ3MKT4D6L5K3UTFSO74JWISWX3D5Q, seg_pos={0 0}, upload=645609]: Uploading piece...
Jan 5 09:18:47.504451557 (737h49m47.375534602s) [noise=true , address=45.91.116.6:4001 , piece_id=RUXUL5DV36RIJMXDRJ7WKUEW2OV3E3BGYZDPJUSYCPC2EIJFHE7A, seg_pos={0 0}, upload=645609]: Uploading piece...
Jan 5 09:18:47.506569530 (737h49m47.377652565s) [noise=true , address=45.91.116.12:4230 , piece_id=UN4V5QHXRCDKBPQ7MGALH4IVS6UHRWOF25Q73FCBHSMUIULA225Q, seg_pos={0 0}, upload=645609]: Uploading piece...
Jan 5 09:18:47.510022275 (737h49m47.38110532s) [noise=true , address=145.79.227.33:4904 , piece_id=WDSJ7U4INZFFDVBLMP67T7MVMNTTI3NGYEPSVYECRH5VGXET3TLQ, seg_pos={0 0}, upload=645609]: Uploading piece...
Jan 5 09:18:47.512017334 (737h49m47.383100379s) [noise=true , address=193.149.164.143:4703 , piece_id=TYA4CVXXQRBEROPSTN7K6GGUDE26AUQJTZPTR2OTMQFNT222KAVA, seg_pos={0 0}, upload=645609]: Uploading piece...
Jan 5 09:18:47.513845425 (737h49m47.38492846s) [noise=true , address=145.79.227.40:4082 , piece_id=5JQ4YYL5RCKCEG2EUSTG7JUDIS35ENMQ6AFURXKZZEJFKWX2OY5Q, seg_pos={0 0}, upload=645609]: Uploading piece...
Jan 5 09:18:47.515562294 (737h49m47.386645329s) [noise=true , address=145.79.227.43:4174 , piece_id=PJPNBBYFLKRSABGKGDKYXI2EGD4PP36KKHVM4GRUA2IY66HMHN4Q, seg_pos={0 0}, upload=645609]: Uploading piece...
Jan 5 09:18:48.913405475 (737h49m48.78448851s) [upload=645609]: Next returned nil segment
Jan 5 09:18:48.913421305 (737h49m48.78450437s) [upload=645609]: Waiting for error group managing segments...
Jan 5 09:18:49.289055482 (737h49m49.160138517s) [noise=true , address=64.49.13.138:4492 , piece_id=TDF7OSOXDTCJWANOSDKDDZSZAY6UQOA4UKVUFOFZYHVT66OUHP4A, seg_pos={0 0}, upload=645609]: Done uploading piece. err: <nil>
Jan 5 09:18:49.353260940 (737h49m49.224343975s) [noise=true , address=145.79.227.11:4748 , piece_id=5DQHX2J4LHFASTZ4FYW55XQO7VCKRBXH7K3XBMTRM7KW4WBO62MQ, seg_pos={0 0}, upload=645609]: Done uploading piece. err: <nil>
Jan 5 09:18:49.370805938 (737h49m49.241888983s) [noise=true , address=145.79.227.36:4526 , piece_id=F2NTSXKTTZITSM6KGT6IZZMZ3MKT4D6L5K3UTFSO74JWISWX3D5Q, seg_pos={0 0}, upload=645609]: Done uploading piece. err: <nil>
Jan 5 09:18:49.376622393 (737h49m49.247705438s) [noise=true , address=145.79.227.43:4174 , piece_id=PJPNBBYFLKRSABGKGDKYXI2EGD4PP36KKHVM4GRUA2IY66HMHN4Q, seg_pos={0 0}, upload=645609]: Done uploading piece. err: <nil>
Jan 5 09:18:49.376682869 (737h49m49.247765914s) [noise=true , address=145.79.227.33:4904 , piece_id=WDSJ7U4INZFFDVBLMP67T7MVMNTTI3NGYEPSVYECRH5VGXET3TLQ, seg_pos={0 0}, upload=645609]: Done uploading piece. err: <nil>
Jan 5 09:18:49.382027456 (737h49m49.253110491s) [noise=true , address=145.79.227.6:4288 , piece_id=TCDAPURTJLJXCXGOQ7LHOB2HKLD64U26LE4RRMDPGZLXL4C4YMQA, seg_pos={0 0}, upload=645609]: Done uploading piece. err: <nil>
Jan 5 09:18:49.445255323 (737h49m49.316338358s) [noise=true , address=145.79.227.18:4023 , piece_id=N7YFCGBBS5NGEIYWGYK3TCYYEHEOU5LGXKQXRDECPTHIUCF5DN4Q, seg_pos={0 0}, upload=645609]: Done uploading piece. err: <nil>
Jan 5 09:18:49.605072892 (737h49m49.476155927s) [noise=true , address=145.79.227.40:4082 , piece_id=5JQ4YYL5RCKCEG2EUSTG7JUDIS35ENMQ6AFURXKZZEJFKWX2OY5Q, seg_pos={0 0}, upload=645609]: Done uploading piece. err: <nil>
Jan 5 09:18:49.846651315 (737h49m49.71773435s) [noise=true , address=45.91.116.6:4001 , piece_id=RUXUL5DV36RIJMXDRJ7WKUEW2OV3E3BGYZDPJUSYCPC2EIJFHE7A, seg_pos={0 0}, upload=645609]: Done uploading piece. err: <nil>
Jan 5 09:18:49.904591111 (737h49m49.775674146s) [noise=true , address=45.91.116.4:4959 , piece_id=QZVDAA4HPU2C6SKRGGQB2CLSQDVPDOBPUIRNFIZUHSFAXV4GOCSQ, seg_pos={0 0}, upload=645609]: Done uploading piece. err: <nil>
Jan 5 09:18:50.091587984 (737h49m49.962671039s) [noise=true , address=77.90.31.18:4828 , piece_id=ZIVM26ZK7N3OY5XJB75QMXIBRG4Y6363ZHI5SNYGPTCJECPTUNRQ, seg_pos={0 0}, upload=645609]: Done uploading piece. err: <nil>
Jan 5 09:18:50.228211276 (737h49m50.099294311s) [noise=true , address=193.149.164.143:4703 , piece_id=TYA4CVXXQRBEROPSTN7K6GGUDE26AUQJTZPTR2OTMQFNT222KAVA, seg_pos={0 0}, upload=645609]: Done uploading piece. err: <nil>
Jan 5 09:18:50.241427261 (737h49m50.112510306s) [noise=true , address=145.79.227.44:4200 , piece_id=G4MWFV53PBS7SGPCHLQ35NOYRTT6MS6L7ISTC6PONIKTWZFVPLIA, seg_pos={0 0}, upload=645609]: Done uploading piece. err: <nil>
Jan 5 09:18:50.247949341 (737h49m50.119032386s) [noise=true , address=145.79.227.30:4782 , piece_id=7FXJ7VS73CNUONPHSIOOKOMTFSLTDCJAP56GUPFTIBBYWUYWOK5A, seg_pos={0 0}, upload=645609]: Done uploading piece. err: <nil>
Jan 5 09:18:50.247977123 (737h49m50.119060168s) [seg_pos={0 0}, upload=645609]: Segment reached optimal threshold of 14 pieces.
Jan 5 09:18:50.248307673 (737h49m50.119390718s) [noise=true , address=145.79.227.49:4353 , piece_id=Y3ZVY7FA2Z7JQ6XGOWNFIMWAXJ26POBBQ3PQ3QRZJENTOVLR7DJA, seg_pos={0 0}, upload=645609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BmzVLKbkqPMB7bJHGzFbgRRLu3TLg6TdTBCvYybsvz2yLpBbQ): protocol: expected piece hash; context canceled; EOF
Jan 5 09:18:50.248316580 (737h49m50.119399635s) [noise=true , address=145.79.227.49:4353 , piece_id=Y3ZVY7FA2Z7JQ6XGOWNFIMWAXJ26POBBQ3PQ3QRZJENTOVLR7DJA, seg_pos={0 0}, upload=645609]: Long tail cancellation due to optimal threshold
Jan 5 09:18:50.248527261 (737h49m50.119610306s) [noise=true , address=145.79.227.34:4925 , piece_id=ZYAAUFNQTG7AJ7THT5VKV4SSFMKXAHVSKCBVO276U3C6EYHIXGPA, seg_pos={0 0}, upload=645609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14ZdgenD4m2bMMmaHVyK3YivrzdSnrp72JvMJsk1uecW6KKFjn): protocol: expected piece hash; context canceled; EOF
Jan 5 09:18:50.248553771 (737h49m50.119636806s) [noise=true , address=145.79.227.34:4925 , piece_id=ZYAAUFNQTG7AJ7THT5VKV4SSFMKXAHVSKCBVO276U3C6EYHIXGPA, seg_pos={0 0}, upload=645609]: Long tail cancellation due to optimal threshold
Jan 5 09:18:50.248699559 (737h49m50.119782594s) [noise=true , address=45.91.116.9:4281 , piece_id=LA6VU4KYZ5AYCIOCWC67SF65AGAAN2NV6TOPQSJJHVPHYKIIOZWQ, seg_pos={0 0}, upload=645609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gxf7DpJZthyR6f1PExX56tJavjhPw9Gxb4h8mtmeYKW3t3tS1): protocol: read tcp 188.166.225.170:44648->45.91.116.9:4281: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 5 09:18:50.248708686 (737h49m50.119791731s) [noise=true , address=45.91.116.9:4281 , piece_id=LA6VU4KYZ5AYCIOCWC67SF65AGAAN2NV6TOPQSJJHVPHYKIIOZWQ, seg_pos={0 0}, upload=645609]: Long tail cancellation due to optimal threshold
Jan 5 09:18:50.248998018 (737h49m50.120081053s) [noise=true , address=45.91.116.12:4230 , piece_id=UN4V5QHXRCDKBPQ7MGALH4IVS6UHRWOF25Q73FCBHSMUIULA225Q, seg_pos={0 0}, upload=645609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127RqMv3h2jS5V4nVqPmi3BDwT9gdJBC93EvyA8VXf5w77yiAQP): protocol: read tcp 188.166.225.170:49196->45.91.116.12:4230: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled
Jan 5 09:18:50.249007656 (737h49m50.120090691s) [noise=true , address=45.91.116.12:4230 , piece_id=UN4V5QHXRCDKBPQ7MGALH4IVS6UHRWOF25Q73FCBHSMUIULA225Q, seg_pos={0 0}, upload=645609]: Long tail cancellation due to optimal threshold
Jan 5 09:18:50.249041190 (737h49m50.120124225s) [seg_pos={0 0}, upload=645609]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jan 5 09:18:50.249148333 (737h49m50.120231369s) [upload=645609]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 5 09:18:50.249154034 (737h49m50.120237079s) [upload=645609]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 5 09:18:50.546593738 (737h49m50.417676783s) [upload=645609]: Done uploading segments...
Jan 5 09:18:50.546616362 (737h49m50.417699397s) [upload=645609]: Upload finished. err: <nil>
Duration: 1m0.003644661s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
13 out of 60 is failed (threshold=3) [....X.X...X....................X..X..X.X.X.X..X...X...X..X..] duration:
6.188787572s trace:
https://jaeger.dev.storj.io/trace/6e7e86a0c49d3474
FAIL (2026-01-05 09:19:33 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Jan 5 09:17:54.838592341 (737h48m54.709675376s) [upload=645597]: Starting upload
Jan 5 09:17:54.838646896 (737h48m54.709729931s) [upload=645597]: Done starting upload
Jan 5 09:17:54.838710166 (737h48m54.709793201s) [upload=645597]: Uploading segments...
Jan 5 09:17:54.838864910 (737h48m54.709947955s) [upload=645597]: Got next segment. Inline: false
Jan 5 09:17:54.838898374 (737h48m54.709981409s) [upload=645597]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 5 09:17:54.838902923 (737h48m54.709985958s) [upload=645597]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 5 09:17:55.086410939 (737h48m54.957493974s) [seg_pos={0 0}, upload=645597]: Begin upload segment...
Jan 5 09:17:55.086467737 (737h48m54.957550772s) [seg_pos={0 0}, upload=645597]: Done begin upload segment.
Jan 5 09:17:55.086524434 (737h48m54.957607469s) [noise=true , address=145.79.227.2:4115 , piece_id=ATGM7WYGHXFWDN4BXJDTX34WZQSAG23TALV7XX3UBIA3CRNR6FTQ, seg_pos={0 0}, upload=645597]: Uploading piece...
Jan 5 09:17:55.086542950 (737h48m54.957625995s) [noise=true , address=145.79.227.52:4446 , piece_id=NSYEHINMINQWH4FYOFJ6CRGEITDV3BUN65J5AYSNWV4GKU2VBVXA, seg_pos={0 0}, upload=645597]: Uploading piece...
Jan 5 09:17:55.087164153 (737h48m54.958247188s) [noise=true , address=145.79.227.20:4437 , piece_id=ZG7MWW7CB2MBGAVRMGO4TUHYAIJKOSSI5GIQ3ZVD7WK7TKP4FTFQ, seg_pos={0 0}, upload=645597]: Uploading piece...
Jan 5 09:17:55.087585125 (737h48m54.95866817s) [noise=true , address=45.91.116.6:4975 , piece_id=BQWSC3Q3YHF72A57QEVHMFKKJBZI5CLJ46A7TP3EEO5RRCH3MJFA, seg_pos={0 0}, upload=645597]: Uploading piece...
Jan 5 09:17:55.087645098 (737h48m54.958728133s) [noise=true , address=45.91.116.14:4191 , piece_id=GOZG4DIDVDB6DJX6BRH74ZVCD5BQLYUTHMMFICGG5DXRTQBQOCEQ, seg_pos={0 0}, upload=645597]: Uploading piece...
Jan 5 09:17:55.087978223 (737h48m54.959061258s) [noise=true , address=145.79.227.38:4979 , piece_id=6CT2VJHMICL5XITOIOINR4KZKACBFN6JLBFQHCXIXWE7Q6ZMMICQ, seg_pos={0 0}, upload=645597]: Uploading piece...
Jan 5 09:17:55.088015594 (737h48m54.959098639s) [noise=true , address=145.79.227.6:4259 , piece_id=2N3EYFVAHSKP4XZBAQDZPZL3TPNKWDFEFOTHITQRTQTHULWZZLJA, seg_pos={0 0}, upload=645597]: Uploading piece...
Jan 5 09:17:55.088279256 (737h48m54.959362292s) [noise=true , address=77.90.31.19:4842 , piece_id=7Z2Y7NXZELYGUD4G5PVBLKRJZL3CLOF22ZREPSUGPT4JYJ4Y22PQ, seg_pos={0 0}, upload=645597]: Uploading piece...
Jan 5 09:17:55.088399405 (737h48m54.95948245s) [noise=true , address=77.90.31.10:4518 , piece_id=FSVPHUKXSS57ME2FIEDDDOLWN3YHQN5NE7BLUYM3S7BY5BICULMA, seg_pos={0 0}, upload=645597]: Uploading piece...
Jan 5 09:17:55.088619494 (737h48m54.959702529s) [noise=true , address=145.79.227.45:4230 , piece_id=YZHU7J2SMWXIME3FIPWG5QK3LFDTNHSBLB4WSSTRBJCWPBXW2NVA, seg_pos={0 0}, upload=645597]: Uploading piece...
Jan 5 09:17:55.088966394 (737h48m54.960049439s) [noise=true , address=145.79.227.24:4581 , piece_id=4G3SJ2KRQ3OISKABPCJ65C5G256IH6UAUCFZ4MEOADX2PHZUTAIA, seg_pos={0 0}, upload=645597]: Uploading piece...
Jan 5 09:17:55.089316721 (737h48m54.960399756s) [noise=true , address=145.79.227.18:4006 , piece_id=CPDP4Q5XGXRCB474UCZ6HEN77572EJDQF37ABK6BH5I7IRXYQVPQ, seg_pos={0 0}, upload=645597]: Uploading piece...
Jan 5 09:17:55.089776337 (737h48m54.960859372s) [noise=true , address=77.90.31.21:4880 , piece_id=KXUC3TEKHHO4VDM5TDJDECSV354232VGUC7SLNVT5GNJ3V3EFGCA, seg_pos={0 0}, upload=645597]: Uploading piece...
Jan 5 09:17:55.089850378 (737h48m54.960933413s) [noise=true , address=145.79.227.28:4727 , piece_id=4XXMSMSYFQAFMMKM24WTX4GVLTWIQAM3W62K646VN4MWVQCMWE6Q, seg_pos={0 0}, upload=645597]: Uploading piece...
Jan 5 09:17:55.090162883 (737h48m54.961245918s) [noise=true , address=45.91.116.13:4156 , piece_id=GM7AZGY2XU2YWVODUD373QFFFYLOLIFEZVY7UJBKKABKSEDFYH4Q, seg_pos={0 0}, upload=645597]: Uploading piece...
Jan 5 09:17:55.091675703 (737h48m54.962758738s) [noise=true , address=145.79.227.14:4838 , piece_id=5F2HPBJJJJ2WJHFFDVD55RMPGXKW7LY7U7QIEY4KZRLWTY3HDYYQ, seg_pos={0 0}, upload=645597]: Uploading piece...
Jan 5 09:17:56.794463178 (737h48m56.665546213s) [upload=645597]: Next returned nil segment
Jan 5 09:17:56.794479830 (737h48m56.665562865s) [upload=645597]: Waiting for error group managing segments...
Jan 5 09:17:57.076908991 (737h48m56.947992036s) [noise=true , address=145.79.227.24:4581 , piece_id=4G3SJ2KRQ3OISKABPCJ65C5G256IH6UAUCFZ4MEOADX2PHZUTAIA, seg_pos={0 0}, upload=645597]: Done uploading piece. err: <nil>
Jan 5 09:17:57.077566553 (737h48m56.948649598s) [noise=true , address=145.79.227.14:4838 , piece_id=5F2HPBJJJJ2WJHFFDVD55RMPGXKW7LY7U7QIEY4KZRLWTY3HDYYQ, seg_pos={0 0}, upload=645597]: Done uploading piece. err: <nil>
Jan 5 09:17:57.093208608 (737h48m56.964291653s) [noise=true , address=145.79.227.38:4979 , piece_id=6CT2VJHMICL5XITOIOINR4KZKACBFN6JLBFQHCXIXWE7Q6ZMMICQ, seg_pos={0 0}, upload=645597]: Done uploading piece. err: <nil>
Jan 5 09:17:57.097347419 (737h48m56.968430464s) [noise=true , address=145.79.227.28:4727 , piece_id=4XXMSMSYFQAFMMKM24WTX4GVLTWIQAM3W62K646VN4MWVQCMWE6Q, seg_pos={0 0}, upload=645597]: Done uploading piece. err: <nil>
Jan 5 09:17:57.097948074 (737h48m56.969031119s) [noise=true , address=145.79.227.2:4115 , piece_id=ATGM7WYGHXFWDN4BXJDTX34WZQSAG23TALV7XX3UBIA3CRNR6FTQ, seg_pos={0 0}, upload=645597]: Done uploading piece. err: <nil>
Jan 5 09:17:57.101168726 (737h48m56.972251771s) [noise=true , address=145.79.227.18:4006 , piece_id=CPDP4Q5XGXRCB474UCZ6HEN77572EJDQF37ABK6BH5I7IRXYQVPQ, seg_pos={0 0}, upload=645597]: Done uploading piece. err: <nil>
Jan 5 09:17:57.106539102 (737h48m56.977622137s) [noise=true , address=145.79.227.52:4446 , piece_id=NSYEHINMINQWH4FYOFJ6CRGEITDV3BUN65J5AYSNWV4GKU2VBVXA, seg_pos={0 0}, upload=645597]: Done uploading piece. err: <nil>
Jan 5 09:17:57.124938367 (737h48m56.996021413s) [noise=true , address=145.79.227.20:4437 , piece_id=ZG7MWW7CB2MBGAVRMGO4TUHYAIJKOSSI5GIQ3ZVD7WK7TKP4FTFQ, seg_pos={0 0}, upload=645597]: Done uploading piece. err: <nil>
Jan 5 09:17:57.142071331 (737h48m57.013154376s) [noise=true , address=145.79.227.6:4259 , piece_id=2N3EYFVAHSKP4XZBAQDZPZL3TPNKWDFEFOTHITQRTQTHULWZZLJA, seg_pos={0 0}, upload=645597]: Done uploading piece. err: <nil>
Jan 5 09:17:57.175080546 (737h48m57.046163581s) [noise=true , address=145.79.227.45:4230 , piece_id=YZHU7J2SMWXIME3FIPWG5QK3LFDTNHSBLB4WSSTRBJCWPBXW2NVA, seg_pos={0 0}, upload=645597]: Done uploading piece. err: <nil>
Jan 5 09:17:58.461065682 (737h48m58.332148717s) [noise=true , address=45.91.116.14:4191 , piece_id=GOZG4DIDVDB6DJX6BRH74ZVCD5BQLYUTHMMFICGG5DXRTQBQOCEQ, seg_pos={0 0}, upload=645597]: Done uploading piece. err: <nil>
Jan 5 09:17:59.174407502 (737h48m59.045490547s) [noise=true , address=45.91.116.6:4975 , piece_id=BQWSC3Q3YHF72A57QEVHMFKKJBZI5CLJ46A7TP3EEO5RRCH3MJFA, seg_pos={0 0}, upload=645597]: Done uploading piece. err: <nil>
Jan 5 09:18:03.798756456 (737h49m3.669839501s) [noise=true , address=77.90.31.19:4842 , piece_id=7Z2Y7NXZELYGUD4G5PVBLKRJZL3CLOF22ZREPSUGPT4JYJ4Y22PQ, seg_pos={0 0}, upload=645597]: Done uploading piece. err: <nil>
Jan 5 09:18:03.798796051 (737h49m3.669879086s) [seg_pos={0 0}, upload=645597]: Segment reached optimal threshold of 13 pieces.
Jan 5 09:18:03.799105350 (737h49m3.670188396s) [noise=true , address=77.90.31.21:4880 , piece_id=KXUC3TEKHHO4VDM5TDJDECSV354232VGUC7SLNVT5GNJ3V3EFGCA, seg_pos={0 0}, upload=645597]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF
Jan 5 09:18:03.799115099 (737h49m3.670198144s) [noise=true , address=77.90.31.21:4880 , piece_id=KXUC3TEKHHO4VDM5TDJDECSV354232VGUC7SLNVT5GNJ3V3EFGCA, seg_pos={0 0}, upload=645597]: Long tail cancellation due to optimal threshold
Jan 5 09:18:03.799379172 (737h49m3.670462217s) [noise=true , address=77.90.31.10:4518 , piece_id=FSVPHUKXSS57ME2FIEDDDOLWN3YHQN5NE7BLUYM3S7BY5BICULMA, seg_pos={0 0}, upload=645597]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: read tcp 188.166.225.170:41460->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 5 09:18:03.799415481 (737h49m3.670498516s) [noise=true , address=77.90.31.10:4518 , piece_id=FSVPHUKXSS57ME2FIEDDDOLWN3YHQN5NE7BLUYM3S7BY5BICULMA, seg_pos={0 0}, upload=645597]: Long tail cancellation due to optimal threshold
Jan 5 09:18:03.799559655 (737h49m3.670642701s) [noise=true , address=45.91.116.13:4156 , piece_id=GM7AZGY2XU2YWVODUD373QFFFYLOLIFEZVY7UJBKKABKSEDFYH4Q, seg_pos={0 0}, upload=645597]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: read tcp 188.166.225.170:34200->45.91.116.13:4156: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 5 09:18:03.799570056 (737h49m3.670653091s) [noise=true , address=45.91.116.13:4156 , piece_id=GM7AZGY2XU2YWVODUD373QFFFYLOLIFEZVY7UJBKKABKSEDFYH4Q, seg_pos={0 0}, upload=645597]: Long tail cancellation due to optimal threshold
Jan 5 09:18:03.799597147 (737h49m3.670680192s) [seg_pos={0 0}, upload=645597]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 5 09:18:03.799692008 (737h49m3.670775053s) [upload=645597]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 5 09:18:03.799698189 (737h49m3.670781234s) [upload=645597]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 5 09:18:04.095750160 (737h49m3.966833205s) [upload=645597]: Done uploading segments...
Jan 5 09:18:04.095772302 (737h49m3.966855347s) [upload=645597]: Upload finished. err: <nil>
Duration: 6.188818921s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
45 out of 60 is failed (threshold=6) [XXXXXXXX.XX..XXX.X.XX.XXX.XX.XXXXXX..XXXXXX...XXXX..XXXXXXXX] duration:
5.001438055s trace:
https://jaeger.dev.storj.io/trace/250a335921426ce6
FAIL (2026-01-05 09:19:33 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001458513s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [.......X.............................................X......] duration:
16.688596268s trace:
https://jaeger.dev.storj.io/trace/35e6dfe601545a69
FAIL (2026-01-05 09:19:23 UTC)
orbiter warning: context deadline exceeded
Duration: 16.688630253s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [........X.............................................X.....] duration:
15.414626351s trace:
https://jaeger.dev.storj.io/trace/35e6dfe601545a69
FAIL (2026-01-05 09:19:00 UTC)
orbiter warning: context deadline exceeded
Duration: 15.414646698s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [XX............X..X...X......................................] duration:
21.590199864s trace:
https://jaeger.dev.storj.io/trace/7e43d4cb58eef6b9
FAIL (2026-01-05 09:18:48 UTC)
orbiter warning: context deadline exceeded
Duration: 21.590225342s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
44 out of 60 is failed (threshold=6) [.XXXXXXXX.XX..XXX.X.XX.XXX.XX.XXXXXX..XXXXXX...XXXX..XXXXXXX] duration:
5.001245423s trace:
https://jaeger.dev.storj.io/trace/6ce502d327a1ccf
FAIL (2026-01-05 09:18:45 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001268569s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
13 out of 60 is failed (threshold=3) [.....X.X...X....................X..X..X.X.X.X..X...X...X..X.] duration:
6.926097563s trace:
https://jaeger.dev.storj.io/trace/6e7e86a0c49d3474
FAIL (2026-01-05 09:18:38 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Jan 5 09:17:54.838592341 (737h48m54.709675376s) [upload=645597]: Starting upload
Jan 5 09:17:54.838646896 (737h48m54.709729931s) [upload=645597]: Done starting upload
Jan 5 09:17:54.838710166 (737h48m54.709793201s) [upload=645597]: Uploading segments...
Jan 5 09:17:54.838864910 (737h48m54.709947955s) [upload=645597]: Got next segment. Inline: false
Jan 5 09:17:54.838898374 (737h48m54.709981409s) [upload=645597]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 5 09:17:54.838902923 (737h48m54.709985958s) [upload=645597]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 5 09:17:55.086410939 (737h48m54.957493974s) [seg_pos={0 0}, upload=645597]: Begin upload segment...
Jan 5 09:17:55.086467737 (737h48m54.957550772s) [seg_pos={0 0}, upload=645597]: Done begin upload segment.
Jan 5 09:17:55.086524434 (737h48m54.957607469s) [noise=true , address=145.79.227.2:4115 , piece_id=ATGM7WYGHXFWDN4BXJDTX34WZQSAG23TALV7XX3UBIA3CRNR6FTQ, seg_pos={0 0}, upload=645597]: Uploading piece...
Jan 5 09:17:55.086542950 (737h48m54.957625995s) [noise=true , address=145.79.227.52:4446 , piece_id=NSYEHINMINQWH4FYOFJ6CRGEITDV3BUN65J5AYSNWV4GKU2VBVXA, seg_pos={0 0}, upload=645597]: Uploading piece...
Jan 5 09:17:55.087164153 (737h48m54.958247188s) [noise=true , address=145.79.227.20:4437 , piece_id=ZG7MWW7CB2MBGAVRMGO4TUHYAIJKOSSI5GIQ3ZVD7WK7TKP4FTFQ, seg_pos={0 0}, upload=645597]: Uploading piece...
Jan 5 09:17:55.087585125 (737h48m54.95866817s) [noise=true , address=45.91.116.6:4975 , piece_id=BQWSC3Q3YHF72A57QEVHMFKKJBZI5CLJ46A7TP3EEO5RRCH3MJFA, seg_pos={0 0}, upload=645597]: Uploading piece...
Jan 5 09:17:55.087645098 (737h48m54.958728133s) [noise=true , address=45.91.116.14:4191 , piece_id=GOZG4DIDVDB6DJX6BRH74ZVCD5BQLYUTHMMFICGG5DXRTQBQOCEQ, seg_pos={0 0}, upload=645597]: Uploading piece...
Jan 5 09:17:55.087978223 (737h48m54.959061258s) [noise=true , address=145.79.227.38:4979 , piece_id=6CT2VJHMICL5XITOIOINR4KZKACBFN6JLBFQHCXIXWE7Q6ZMMICQ, seg_pos={0 0}, upload=645597]: Uploading piece...
Jan 5 09:17:55.088015594 (737h48m54.959098639s) [noise=true , address=145.79.227.6:4259 , piece_id=2N3EYFVAHSKP4XZBAQDZPZL3TPNKWDFEFOTHITQRTQTHULWZZLJA, seg_pos={0 0}, upload=645597]: Uploading piece...
Jan 5 09:17:55.088279256 (737h48m54.959362292s) [noise=true , address=77.90.31.19:4842 , piece_id=7Z2Y7NXZELYGUD4G5PVBLKRJZL3CLOF22ZREPSUGPT4JYJ4Y22PQ, seg_pos={0 0}, upload=645597]: Uploading piece...
Jan 5 09:17:55.088399405 (737h48m54.95948245s) [noise=true , address=77.90.31.10:4518 , piece_id=FSVPHUKXSS57ME2FIEDDDOLWN3YHQN5NE7BLUYM3S7BY5BICULMA, seg_pos={0 0}, upload=645597]: Uploading piece...
Jan 5 09:17:55.088619494 (737h48m54.959702529s) [noise=true , address=145.79.227.45:4230 , piece_id=YZHU7J2SMWXIME3FIPWG5QK3LFDTNHSBLB4WSSTRBJCWPBXW2NVA, seg_pos={0 0}, upload=645597]: Uploading piece...
Jan 5 09:17:55.088966394 (737h48m54.960049439s) [noise=true , address=145.79.227.24:4581 , piece_id=4G3SJ2KRQ3OISKABPCJ65C5G256IH6UAUCFZ4MEOADX2PHZUTAIA, seg_pos={0 0}, upload=645597]: Uploading piece...
Jan 5 09:17:55.089316721 (737h48m54.960399756s) [noise=true , address=145.79.227.18:4006 , piece_id=CPDP4Q5XGXRCB474UCZ6HEN77572EJDQF37ABK6BH5I7IRXYQVPQ, seg_pos={0 0}, upload=645597]: Uploading piece...
Jan 5 09:17:55.089776337 (737h48m54.960859372s) [noise=true , address=77.90.31.21:4880 , piece_id=KXUC3TEKHHO4VDM5TDJDECSV354232VGUC7SLNVT5GNJ3V3EFGCA, seg_pos={0 0}, upload=645597]: Uploading piece...
Jan 5 09:17:55.089850378 (737h48m54.960933413s) [noise=true , address=145.79.227.28:4727 , piece_id=4XXMSMSYFQAFMMKM24WTX4GVLTWIQAM3W62K646VN4MWVQCMWE6Q, seg_pos={0 0}, upload=645597]: Uploading piece...
Jan 5 09:17:55.090162883 (737h48m54.961245918s) [noise=true , address=45.91.116.13:4156 , piece_id=GM7AZGY2XU2YWVODUD373QFFFYLOLIFEZVY7UJBKKABKSEDFYH4Q, seg_pos={0 0}, upload=645597]: Uploading piece...
Jan 5 09:17:55.091675703 (737h48m54.962758738s) [noise=true , address=145.79.227.14:4838 , piece_id=5F2HPBJJJJ2WJHFFDVD55RMPGXKW7LY7U7QIEY4KZRLWTY3HDYYQ, seg_pos={0 0}, upload=645597]: Uploading piece...
Jan 5 09:17:56.794463178 (737h48m56.665546213s) [upload=645597]: Next returned nil segment
Jan 5 09:17:56.794479830 (737h48m56.665562865s) [upload=645597]: Waiting for error group managing segments...
Jan 5 09:17:57.076908991 (737h48m56.947992036s) [noise=true , address=145.79.227.24:4581 , piece_id=4G3SJ2KRQ3OISKABPCJ65C5G256IH6UAUCFZ4MEOADX2PHZUTAIA, seg_pos={0 0}, upload=645597]: Done uploading piece. err: <nil>
Jan 5 09:17:57.077566553 (737h48m56.948649598s) [noise=true , address=145.79.227.14:4838 , piece_id=5F2HPBJJJJ2WJHFFDVD55RMPGXKW7LY7U7QIEY4KZRLWTY3HDYYQ, seg_pos={0 0}, upload=645597]: Done uploading piece. err: <nil>
Jan 5 09:17:57.093208608 (737h48m56.964291653s) [noise=true , address=145.79.227.38:4979 , piece_id=6CT2VJHMICL5XITOIOINR4KZKACBFN6JLBFQHCXIXWE7Q6ZMMICQ, seg_pos={0 0}, upload=645597]: Done uploading piece. err: <nil>
Jan 5 09:17:57.097347419 (737h48m56.968430464s) [noise=true , address=145.79.227.28:4727 , piece_id=4XXMSMSYFQAFMMKM24WTX4GVLTWIQAM3W62K646VN4MWVQCMWE6Q, seg_pos={0 0}, upload=645597]: Done uploading piece. err: <nil>
Jan 5 09:17:57.097948074 (737h48m56.969031119s) [noise=true , address=145.79.227.2:4115 , piece_id=ATGM7WYGHXFWDN4BXJDTX34WZQSAG23TALV7XX3UBIA3CRNR6FTQ, seg_pos={0 0}, upload=645597]: Done uploading piece. err: <nil>
Jan 5 09:17:57.101168726 (737h48m56.972251771s) [noise=true , address=145.79.227.18:4006 , piece_id=CPDP4Q5XGXRCB474UCZ6HEN77572EJDQF37ABK6BH5I7IRXYQVPQ, seg_pos={0 0}, upload=645597]: Done uploading piece. err: <nil>
Jan 5 09:17:57.106539102 (737h48m56.977622137s) [noise=true , address=145.79.227.52:4446 , piece_id=NSYEHINMINQWH4FYOFJ6CRGEITDV3BUN65J5AYSNWV4GKU2VBVXA, seg_pos={0 0}, upload=645597]: Done uploading piece. err: <nil>
Jan 5 09:17:57.124938367 (737h48m56.996021413s) [noise=true , address=145.79.227.20:4437 , piece_id=ZG7MWW7CB2MBGAVRMGO4TUHYAIJKOSSI5GIQ3ZVD7WK7TKP4FTFQ, seg_pos={0 0}, upload=645597]: Done uploading piece. err: <nil>
Jan 5 09:17:57.142071331 (737h48m57.013154376s) [noise=true , address=145.79.227.6:4259 , piece_id=2N3EYFVAHSKP4XZBAQDZPZL3TPNKWDFEFOTHITQRTQTHULWZZLJA, seg_pos={0 0}, upload=645597]: Done uploading piece. err: <nil>
Jan 5 09:17:57.175080546 (737h48m57.046163581s) [noise=true , address=145.79.227.45:4230 , piece_id=YZHU7J2SMWXIME3FIPWG5QK3LFDTNHSBLB4WSSTRBJCWPBXW2NVA, seg_pos={0 0}, upload=645597]: Done uploading piece. err: <nil>
Jan 5 09:17:58.461065682 (737h48m58.332148717s) [noise=true , address=45.91.116.14:4191 , piece_id=GOZG4DIDVDB6DJX6BRH74ZVCD5BQLYUTHMMFICGG5DXRTQBQOCEQ, seg_pos={0 0}, upload=645597]: Done uploading piece. err: <nil>
Jan 5 09:17:59.174407502 (737h48m59.045490547s) [noise=true , address=45.91.116.6:4975 , piece_id=BQWSC3Q3YHF72A57QEVHMFKKJBZI5CLJ46A7TP3EEO5RRCH3MJFA, seg_pos={0 0}, upload=645597]: Done uploading piece. err: <nil>
Jan 5 09:18:03.798756456 (737h49m3.669839501s) [noise=true , address=77.90.31.19:4842 , piece_id=7Z2Y7NXZELYGUD4G5PVBLKRJZL3CLOF22ZREPSUGPT4JYJ4Y22PQ, seg_pos={0 0}, upload=645597]: Done uploading piece. err: <nil>
Jan 5 09:18:03.798796051 (737h49m3.669879086s) [seg_pos={0 0}, upload=645597]: Segment reached optimal threshold of 13 pieces.
Jan 5 09:18:03.799105350 (737h49m3.670188396s) [noise=true , address=77.90.31.21:4880 , piece_id=KXUC3TEKHHO4VDM5TDJDECSV354232VGUC7SLNVT5GNJ3V3EFGCA, seg_pos={0 0}, upload=645597]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF
Jan 5 09:18:03.799115099 (737h49m3.670198144s) [noise=true , address=77.90.31.21:4880 , piece_id=KXUC3TEKHHO4VDM5TDJDECSV354232VGUC7SLNVT5GNJ3V3EFGCA, seg_pos={0 0}, upload=645597]: Long tail cancellation due to optimal threshold
Jan 5 09:18:03.799379172 (737h49m3.670462217s) [noise=true , address=77.90.31.10:4518 , piece_id=FSVPHUKXSS57ME2FIEDDDOLWN3YHQN5NE7BLUYM3S7BY5BICULMA, seg_pos={0 0}, upload=645597]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: read tcp 188.166.225.170:41460->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 5 09:18:03.799415481 (737h49m3.670498516s) [noise=true , address=77.90.31.10:4518 , piece_id=FSVPHUKXSS57ME2FIEDDDOLWN3YHQN5NE7BLUYM3S7BY5BICULMA, seg_pos={0 0}, upload=645597]: Long tail cancellation due to optimal threshold
Jan 5 09:18:03.799559655 (737h49m3.670642701s) [noise=true , address=45.91.116.13:4156 , piece_id=GM7AZGY2XU2YWVODUD373QFFFYLOLIFEZVY7UJBKKABKSEDFYH4Q, seg_pos={0 0}, upload=645597]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: read tcp 188.166.225.170:34200->45.91.116.13:4156: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 5 09:18:03.799570056 (737h49m3.670653091s) [noise=true , address=45.91.116.13:4156 , piece_id=GM7AZGY2XU2YWVODUD373QFFFYLOLIFEZVY7UJBKKABKSEDFYH4Q, seg_pos={0 0}, upload=645597]: Long tail cancellation due to optimal threshold
Jan 5 09:18:03.799597147 (737h49m3.670680192s) [seg_pos={0 0}, upload=645597]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 5 09:18:03.799692008 (737h49m3.670775053s) [upload=645597]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 5 09:18:03.799698189 (737h49m3.670781234s) [upload=645597]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 5 09:18:04.095750160 (737h49m3.966833205s) [upload=645597]: Done uploading segments...
Jan 5 09:18:04.095772302 (737h49m3.966855347s) [upload=645597]: Upload finished. err: <nil>
Duration: 6.926122921s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [.........X.............................................X....] duration:
7.159755232s trace:
https://jaeger.dev.storj.io/trace/35e6dfe601545a69
FAIL (2026-01-05 09:18:30 UTC)
orbiter warning: context deadline exceeded
Duration: 7.15978542s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
14 out of 60 is failed (threshold=3) [X.....X.X...X....................X..X..X.X.X.X..X...X...X..X] duration:
30.003180518s trace:
https://jaeger.dev.storj.io/trace/6e7e86a0c49d3474
FAIL (2026-01-05 09:18:24 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Jan 5 09:17:54.838592341 (737h48m54.709675376s) [upload=645597]: Starting upload
Jan 5 09:17:54.838646896 (737h48m54.709729931s) [upload=645597]: Done starting upload
Jan 5 09:17:54.838710166 (737h48m54.709793201s) [upload=645597]: Uploading segments...
Jan 5 09:17:54.838864910 (737h48m54.709947955s) [upload=645597]: Got next segment. Inline: false
Jan 5 09:17:54.838898374 (737h48m54.709981409s) [upload=645597]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 5 09:17:54.838902923 (737h48m54.709985958s) [upload=645597]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 5 09:17:55.086410939 (737h48m54.957493974s) [seg_pos={0 0}, upload=645597]: Begin upload segment...
Jan 5 09:17:55.086467737 (737h48m54.957550772s) [seg_pos={0 0}, upload=645597]: Done begin upload segment.
Jan 5 09:17:55.086524434 (737h48m54.957607469s) [noise=true , address=145.79.227.2:4115 , piece_id=ATGM7WYGHXFWDN4BXJDTX34WZQSAG23TALV7XX3UBIA3CRNR6FTQ, seg_pos={0 0}, upload=645597]: Uploading piece...
Jan 5 09:17:55.086542950 (737h48m54.957625995s) [noise=true , address=145.79.227.52:4446 , piece_id=NSYEHINMINQWH4FYOFJ6CRGEITDV3BUN65J5AYSNWV4GKU2VBVXA, seg_pos={0 0}, upload=645597]: Uploading piece...
Jan 5 09:17:55.087164153 (737h48m54.958247188s) [noise=true , address=145.79.227.20:4437 , piece_id=ZG7MWW7CB2MBGAVRMGO4TUHYAIJKOSSI5GIQ3ZVD7WK7TKP4FTFQ, seg_pos={0 0}, upload=645597]: Uploading piece...
Jan 5 09:17:55.087585125 (737h48m54.95866817s) [noise=true , address=45.91.116.6:4975 , piece_id=BQWSC3Q3YHF72A57QEVHMFKKJBZI5CLJ46A7TP3EEO5RRCH3MJFA, seg_pos={0 0}, upload=645597]: Uploading piece...
Jan 5 09:17:55.087645098 (737h48m54.958728133s) [noise=true , address=45.91.116.14:4191 , piece_id=GOZG4DIDVDB6DJX6BRH74ZVCD5BQLYUTHMMFICGG5DXRTQBQOCEQ, seg_pos={0 0}, upload=645597]: Uploading piece...
Jan 5 09:17:55.087978223 (737h48m54.959061258s) [noise=true , address=145.79.227.38:4979 , piece_id=6CT2VJHMICL5XITOIOINR4KZKACBFN6JLBFQHCXIXWE7Q6ZMMICQ, seg_pos={0 0}, upload=645597]: Uploading piece...
Jan 5 09:17:55.088015594 (737h48m54.959098639s) [noise=true , address=145.79.227.6:4259 , piece_id=2N3EYFVAHSKP4XZBAQDZPZL3TPNKWDFEFOTHITQRTQTHULWZZLJA, seg_pos={0 0}, upload=645597]: Uploading piece...
Jan 5 09:17:55.088279256 (737h48m54.959362292s) [noise=true , address=77.90.31.19:4842 , piece_id=7Z2Y7NXZELYGUD4G5PVBLKRJZL3CLOF22ZREPSUGPT4JYJ4Y22PQ, seg_pos={0 0}, upload=645597]: Uploading piece...
Jan 5 09:17:55.088399405 (737h48m54.95948245s) [noise=true , address=77.90.31.10:4518 , piece_id=FSVPHUKXSS57ME2FIEDDDOLWN3YHQN5NE7BLUYM3S7BY5BICULMA, seg_pos={0 0}, upload=645597]: Uploading piece...
Jan 5 09:17:55.088619494 (737h48m54.959702529s) [noise=true , address=145.79.227.45:4230 , piece_id=YZHU7J2SMWXIME3FIPWG5QK3LFDTNHSBLB4WSSTRBJCWPBXW2NVA, seg_pos={0 0}, upload=645597]: Uploading piece...
Jan 5 09:17:55.088966394 (737h48m54.960049439s) [noise=true , address=145.79.227.24:4581 , piece_id=4G3SJ2KRQ3OISKABPCJ65C5G256IH6UAUCFZ4MEOADX2PHZUTAIA, seg_pos={0 0}, upload=645597]: Uploading piece...
Jan 5 09:17:55.089316721 (737h48m54.960399756s) [noise=true , address=145.79.227.18:4006 , piece_id=CPDP4Q5XGXRCB474UCZ6HEN77572EJDQF37ABK6BH5I7IRXYQVPQ, seg_pos={0 0}, upload=645597]: Uploading piece...
Jan 5 09:17:55.089776337 (737h48m54.960859372s) [noise=true , address=77.90.31.21:4880 , piece_id=KXUC3TEKHHO4VDM5TDJDECSV354232VGUC7SLNVT5GNJ3V3EFGCA, seg_pos={0 0}, upload=645597]: Uploading piece...
Jan 5 09:17:55.089850378 (737h48m54.960933413s) [noise=true , address=145.79.227.28:4727 , piece_id=4XXMSMSYFQAFMMKM24WTX4GVLTWIQAM3W62K646VN4MWVQCMWE6Q, seg_pos={0 0}, upload=645597]: Uploading piece...
Jan 5 09:17:55.090162883 (737h48m54.961245918s) [noise=true , address=45.91.116.13:4156 , piece_id=GM7AZGY2XU2YWVODUD373QFFFYLOLIFEZVY7UJBKKABKSEDFYH4Q, seg_pos={0 0}, upload=645597]: Uploading piece...
Jan 5 09:17:55.091675703 (737h48m54.962758738s) [noise=true , address=145.79.227.14:4838 , piece_id=5F2HPBJJJJ2WJHFFDVD55RMPGXKW7LY7U7QIEY4KZRLWTY3HDYYQ, seg_pos={0 0}, upload=645597]: Uploading piece...
Jan 5 09:17:56.794463178 (737h48m56.665546213s) [upload=645597]: Next returned nil segment
Jan 5 09:17:56.794479830 (737h48m56.665562865s) [upload=645597]: Waiting for error group managing segments...
Jan 5 09:17:57.076908991 (737h48m56.947992036s) [noise=true , address=145.79.227.24:4581 , piece_id=4G3SJ2KRQ3OISKABPCJ65C5G256IH6UAUCFZ4MEOADX2PHZUTAIA, seg_pos={0 0}, upload=645597]: Done uploading piece. err: <nil>
Jan 5 09:17:57.077566553 (737h48m56.948649598s) [noise=true , address=145.79.227.14:4838 , piece_id=5F2HPBJJJJ2WJHFFDVD55RMPGXKW7LY7U7QIEY4KZRLWTY3HDYYQ, seg_pos={0 0}, upload=645597]: Done uploading piece. err: <nil>
Jan 5 09:17:57.093208608 (737h48m56.964291653s) [noise=true , address=145.79.227.38:4979 , piece_id=6CT2VJHMICL5XITOIOINR4KZKACBFN6JLBFQHCXIXWE7Q6ZMMICQ, seg_pos={0 0}, upload=645597]: Done uploading piece. err: <nil>
Jan 5 09:17:57.097347419 (737h48m56.968430464s) [noise=true , address=145.79.227.28:4727 , piece_id=4XXMSMSYFQAFMMKM24WTX4GVLTWIQAM3W62K646VN4MWVQCMWE6Q, seg_pos={0 0}, upload=645597]: Done uploading piece. err: <nil>
Jan 5 09:17:57.097948074 (737h48m56.969031119s) [noise=true , address=145.79.227.2:4115 , piece_id=ATGM7WYGHXFWDN4BXJDTX34WZQSAG23TALV7XX3UBIA3CRNR6FTQ, seg_pos={0 0}, upload=645597]: Done uploading piece. err: <nil>
Jan 5 09:17:57.101168726 (737h48m56.972251771s) [noise=true , address=145.79.227.18:4006 , piece_id=CPDP4Q5XGXRCB474UCZ6HEN77572EJDQF37ABK6BH5I7IRXYQVPQ, seg_pos={0 0}, upload=645597]: Done uploading piece. err: <nil>
Jan 5 09:17:57.106539102 (737h48m56.977622137s) [noise=true , address=145.79.227.52:4446 , piece_id=NSYEHINMINQWH4FYOFJ6CRGEITDV3BUN65J5AYSNWV4GKU2VBVXA, seg_pos={0 0}, upload=645597]: Done uploading piece. err: <nil>
Jan 5 09:17:57.124938367 (737h48m56.996021413s) [noise=true , address=145.79.227.20:4437 , piece_id=ZG7MWW7CB2MBGAVRMGO4TUHYAIJKOSSI5GIQ3ZVD7WK7TKP4FTFQ, seg_pos={0 0}, upload=645597]: Done uploading piece. err: <nil>
Jan 5 09:17:57.142071331 (737h48m57.013154376s) [noise=true , address=145.79.227.6:4259 , piece_id=2N3EYFVAHSKP4XZBAQDZPZL3TPNKWDFEFOTHITQRTQTHULWZZLJA, seg_pos={0 0}, upload=645597]: Done uploading piece. err: <nil>
Jan 5 09:17:57.175080546 (737h48m57.046163581s) [noise=true , address=145.79.227.45:4230 , piece_id=YZHU7J2SMWXIME3FIPWG5QK3LFDTNHSBLB4WSSTRBJCWPBXW2NVA, seg_pos={0 0}, upload=645597]: Done uploading piece. err: <nil>
Jan 5 09:17:58.461065682 (737h48m58.332148717s) [noise=true , address=45.91.116.14:4191 , piece_id=GOZG4DIDVDB6DJX6BRH74ZVCD5BQLYUTHMMFICGG5DXRTQBQOCEQ, seg_pos={0 0}, upload=645597]: Done uploading piece. err: <nil>
Jan 5 09:17:59.174407502 (737h48m59.045490547s) [noise=true , address=45.91.116.6:4975 , piece_id=BQWSC3Q3YHF72A57QEVHMFKKJBZI5CLJ46A7TP3EEO5RRCH3MJFA, seg_pos={0 0}, upload=645597]: Done uploading piece. err: <nil>
Jan 5 09:18:03.798756456 (737h49m3.669839501s) [noise=true , address=77.90.31.19:4842 , piece_id=7Z2Y7NXZELYGUD4G5PVBLKRJZL3CLOF22ZREPSUGPT4JYJ4Y22PQ, seg_pos={0 0}, upload=645597]: Done uploading piece. err: <nil>
Jan 5 09:18:03.798796051 (737h49m3.669879086s) [seg_pos={0 0}, upload=645597]: Segment reached optimal threshold of 13 pieces.
Jan 5 09:18:03.799105350 (737h49m3.670188396s) [noise=true , address=77.90.31.21:4880 , piece_id=KXUC3TEKHHO4VDM5TDJDECSV354232VGUC7SLNVT5GNJ3V3EFGCA, seg_pos={0 0}, upload=645597]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF
Jan 5 09:18:03.799115099 (737h49m3.670198144s) [noise=true , address=77.90.31.21:4880 , piece_id=KXUC3TEKHHO4VDM5TDJDECSV354232VGUC7SLNVT5GNJ3V3EFGCA, seg_pos={0 0}, upload=645597]: Long tail cancellation due to optimal threshold
Jan 5 09:18:03.799379172 (737h49m3.670462217s) [noise=true , address=77.90.31.10:4518 , piece_id=FSVPHUKXSS57ME2FIEDDDOLWN3YHQN5NE7BLUYM3S7BY5BICULMA, seg_pos={0 0}, upload=645597]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: read tcp 188.166.225.170:41460->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 5 09:18:03.799415481 (737h49m3.670498516s) [noise=true , address=77.90.31.10:4518 , piece_id=FSVPHUKXSS57ME2FIEDDDOLWN3YHQN5NE7BLUYM3S7BY5BICULMA, seg_pos={0 0}, upload=645597]: Long tail cancellation due to optimal threshold
Jan 5 09:18:03.799559655 (737h49m3.670642701s) [noise=true , address=45.91.116.13:4156 , piece_id=GM7AZGY2XU2YWVODUD373QFFFYLOLIFEZVY7UJBKKABKSEDFYH4Q, seg_pos={0 0}, upload=645597]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: read tcp 188.166.225.170:34200->45.91.116.13:4156: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 5 09:18:03.799570056 (737h49m3.670653091s) [noise=true , address=45.91.116.13:4156 , piece_id=GM7AZGY2XU2YWVODUD373QFFFYLOLIFEZVY7UJBKKABKSEDFYH4Q, seg_pos={0 0}, upload=645597]: Long tail cancellation due to optimal threshold
Jan 5 09:18:03.799597147 (737h49m3.670680192s) [seg_pos={0 0}, upload=645597]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 5 09:18:03.799692008 (737h49m3.670775053s) [upload=645597]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 5 09:18:03.799698189 (737h49m3.670781234s) [upload=645597]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 5 09:18:04.095750160 (737h49m3.966833205s) [upload=645597]: Done uploading segments...
Jan 5 09:18:04.095772302 (737h49m3.966855347s) [upload=645597]: Upload finished. err: <nil>
Duration: 30.003205926s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
44 out of 60 is failed (threshold=6) [X.XXXXXXXX.XX..XXX.X.XX.XXX.XX.XXXXXX..XXXXXX...XXXX..XXXXXX] duration:
5.000833404s trace:
https://jaeger.dev.storj.io/trace/17987360c5cd1ea0
FAIL (2026-01-05 09:18:01 UTC)
orbiter warning: context deadline exceeded
Duration: 5.000860036s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [.XX............X..X...X.....................................] duration:
19.970499481s trace:
https://jaeger.dev.storj.io/trace/7e43d4cb58eef6b9
FAIL (2026-01-05 09:17:52 UTC)
orbiter warning: context deadline exceeded
Duration: 19.970525491s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
27 out of 60 is failed (threshold=3) [X.XX.XXXXXX.XXX...X....XX.X.X....XX....X..XX.XX......X.X.X..] duration:
15.413480413s trace:
https://jaeger.dev.storj.io/trace/1d2dd2d52810438a
FAIL (2026-01-05 09:17:49 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
noise=true , address=145.79.227.33:4899 , piece_id=KM6CEND447VWRRLLK57KSQ4MIFGQIK5FHX47PMHBRVHVSVMFCZ6A, seg_pos={0 0}, upload=645558]: Uploading piece...
Jan 5 09:15:29.200269970 (737h46m29.071353015s) [noise=true , address=77.90.31.18:4817 , piece_id=OTSJYC35STDIU7JBTRELHB5UTGNHNSGQIHFJ65FOMRJB4OFV6QWA, seg_pos={0 0}, upload=645558]: Uploading piece...
Jan 5 09:15:29.200601691 (737h46m29.071684726s) [noise=true , address=77.90.31.11:4560 , piece_id=3ICPCD764NRO4G3DEE2NEQOCOO7RESONGSM5GBAGVFCFFDOPILZA, seg_pos={0 0}, upload=645558]: Uploading piece...
Jan 5 09:15:29.200987557 (737h46m29.072070622s) [noise=true , address=45.91.116.8:4066 , piece_id=JQ2XRTMAWWVVTK4ZVF4OTGSWPD4YMNDQFCPRAOLIQ57KRTXSLHXQ, seg_pos={0 0}, upload=645558]: Uploading piece...
Jan 5 09:15:29.201343163 (737h46m29.072426209s) [noise=true , address=145.79.227.49:4364 , piece_id=SPEZRPRC7BOQ2L535RPH5YZ3WESHYSKVXWIPOXYDKCY7QWOSKP6A, seg_pos={0 0}, upload=645558]: Uploading piece...
Jan 5 09:15:29.201736713 (737h46m29.072819758s) [noise=true , address=145.79.227.20:4457 , piece_id=N3Z7W2HEAJYOYWW6XZNKKNTAQ6UWT6XFLM7ZEGMDVRFINJZ53ICA, seg_pos={0 0}, upload=645558]: Uploading piece...
Jan 5 09:15:29.202243488 (737h46m29.073326523s) [noise=true , address=64.49.13.169:4713 , piece_id=CEEOAYFOSQJOYMTKOBI2X2YGJV2D2VI2EZAWWQQVE7Z27GVUSIBA, seg_pos={0 0}, upload=645558]: Uploading piece...
Jan 5 09:15:29.202665501 (737h46m29.073748546s) [noise=true , address=77.90.31.17:4802 , piece_id=YKMHTVPOA6MF5IH7DTEMPAJBZWBNUGCDHT5E7BYLQWVDUSBLEM3A, seg_pos={0 0}, upload=645558]: Uploading piece...
Jan 5 09:15:29.204332386 (737h46m29.075415421s) [noise=true , address=145.79.227.40:4052 , piece_id=ORA5FT22S3PUHDZ4Q26YEAYW7VM2TNRZEPMD3H3NIKDDIQSQINUQ, seg_pos={0 0}, upload=645558]: Uploading piece...
Jan 5 09:15:29.206264855 (737h46m29.0773479s) [noise=true , address=145.79.227.30:4775 , piece_id=CAF6FWLCUNFVJ5FZW7LYXH244VHHTSCWWUP664MRGJFSXPPRKNWQ, seg_pos={0 0}, upload=645558]: Uploading piece...
Jan 5 09:15:29.207973889 (737h46m29.079056934s) [noise=true , address=145.79.227.26:4664 , piece_id=U5YGCVMR6SHNKZG6TFQTUBO4JBMHBE2X2JBIDVYZ2DYEWRODVB6A, seg_pos={0 0}, upload=645558]: Uploading piece...
Jan 5 09:15:29.209273434 (737h46m29.080356469s) [noise=true , address=145.79.227.43:4165 , piece_id=X2IW45ERSNU6CRZNDKR3R6ZCR42WFCMMIH46I6RCIGBSIA472VKQ, seg_pos={0 0}, upload=645558]: Uploading piece...
Jan 5 09:15:29.210820229 (737h46m29.081903274s) [noise=true , address=145.79.227.6:4291 , piece_id=W6WEK5EZ7VKJOW4ETADS3X5JAC3ADANF3XHIVAE7OVS54TFNWRSQ, seg_pos={0 0}, upload=645558]: Uploading piece...
Jan 5 09:15:29.212536928 (737h46m29.083619963s) [noise=true , address=64.49.13.155:4923 , piece_id=E6X3TYE7TAZJ7Z7JYPEXRPUZJV2PGDZU674ONMXZNRUSYBVOH3KA, seg_pos={0 0}, upload=645558]: Uploading piece...
Jan 5 09:15:29.213571047 (737h46m29.084654082s) [noise=true , address=45.91.116.13:4184 , piece_id=JTDKQZIKXO35GFKCDIMA2LIYDWDSOVY76HTQTT53POWURPZLMCRQ, seg_pos={0 0}, upload=645558]: Uploading piece...
Jan 5 09:15:29.214913814 (737h46m29.08599686s) [noise=true , address=145.79.227.44:4201 , piece_id=R5IPYMC4IG2KHK6G7T6T2SC7TDFHGSJW7XTQ4QEHCOOHJL64A6RA, seg_pos={0 0}, upload=645558]: Uploading piece...
Jan 5 09:15:30.651877289 (737h46m30.522960324s) [upload=645558]: Next returned nil segment
Jan 5 09:15:30.651890403 (737h46m30.522973448s) [upload=645558]: Waiting for error group managing segments...
Jan 5 09:15:31.074605361 (737h46m30.945688406s) [noise=true , address=145.79.227.20:4457 , piece_id=N3Z7W2HEAJYOYWW6XZNKKNTAQ6UWT6XFLM7ZEGMDVRFINJZ53ICA, seg_pos={0 0}, upload=645558]: Done uploading piece. err: <nil>
Jan 5 09:15:31.076109075 (737h46m30.947192121s) [noise=true , address=145.79.227.26:4664 , piece_id=U5YGCVMR6SHNKZG6TFQTUBO4JBMHBE2X2JBIDVYZ2DYEWRODVB6A, seg_pos={0 0}, upload=645558]: Done uploading piece. err: <nil>
Jan 5 09:15:31.076942272 (737h46m30.948025307s) [noise=true , address=145.79.227.34:4936 , piece_id=LJWK6NMSIFHRCWDY5YLLD7UT2BCL45LGQMO7LHIRF72LZGM7KWQQ, seg_pos={0 0}, upload=645558]: Done uploading piece. err: <nil>
Jan 5 09:15:31.143943183 (737h46m31.015026228s) [noise=true , address=145.79.227.6:4291 , piece_id=W6WEK5EZ7VKJOW4ETADS3X5JAC3ADANF3XHIVAE7OVS54TFNWRSQ, seg_pos={0 0}, upload=645558]: Done uploading piece. err: <nil>
Jan 5 09:15:31.163210440 (737h46m31.034293485s) [noise=true , address=145.79.227.49:4364 , piece_id=SPEZRPRC7BOQ2L535RPH5YZ3WESHYSKVXWIPOXYDKCY7QWOSKP6A, seg_pos={0 0}, upload=645558]: Done uploading piece. err: <nil>
Jan 5 09:15:31.588499871 (737h46m31.459582916s) [noise=true , address=45.91.116.13:4184 , piece_id=JTDKQZIKXO35GFKCDIMA2LIYDWDSOVY76HTQTT53POWURPZLMCRQ, seg_pos={0 0}, upload=645558]: Done uploading piece. err: <nil>
Jan 5 09:15:31.742054013 (737h46m31.613137058s) [noise=true , address=77.90.31.17:4802 , piece_id=YKMHTVPOA6MF5IH7DTEMPAJBZWBNUGCDHT5E7BYLQWVDUSBLEM3A, seg_pos={0 0}, upload=645558]: Done uploading piece. err: <nil>
Jan 5 09:15:31.939681488 (737h46m31.810764533s) [noise=true , address=145.79.227.30:4775 , piece_id=CAF6FWLCUNFVJ5FZW7LYXH244VHHTSCWWUP664MRGJFSXPPRKNWQ, seg_pos={0 0}, upload=645558]: Done uploading piece. err: <nil>
Jan 5 09:15:31.973476740 (737h46m31.844559775s) [noise=true , address=145.79.227.44:4201 , piece_id=R5IPYMC4IG2KHK6G7T6T2SC7TDFHGSJW7XTQ4QEHCOOHJL64A6RA, seg_pos={0 0}, upload=645558]: Done uploading piece. err: <nil>
Jan 5 09:15:32.038871842 (737h46m31.909954887s) [noise=true , address=145.79.227.43:4165 , piece_id=X2IW45ERSNU6CRZNDKR3R6ZCR42WFCMMIH46I6RCIGBSIA472VKQ, seg_pos={0 0}, upload=645558]: Done uploading piece. err: <nil>
Jan 5 09:15:32.090674573 (737h46m31.961757618s) [noise=true , address=77.90.31.18:4817 , piece_id=OTSJYC35STDIU7JBTRELHB5UTGNHNSGQIHFJ65FOMRJB4OFV6QWA, seg_pos={0 0}, upload=645558]: Done uploading piece. err: <nil>
Jan 5 09:15:32.220634288 (737h46m32.091717333s) [noise=true , address=77.90.31.11:4560 , piece_id=3ICPCD764NRO4G3DEE2NEQOCOO7RESONGSM5GBAGVFCFFDOPILZA, seg_pos={0 0}, upload=645558]: Done uploading piece. err: <nil>
Jan 5 09:15:32.375653890 (737h46m32.246736925s) [noise=true , address=45.91.116.8:4066 , piece_id=JQ2XRTMAWWVVTK4ZVF4OTGSWPD4YMNDQFCPRAOLIQ57KRTXSLHXQ, seg_pos={0 0}, upload=645558]: Done uploading piece. err: <nil>
Jan 5 09:15:32.388804811 (737h46m32.259887846s) [noise=true , address=145.79.227.40:4052 , piece_id=ORA5FT22S3PUHDZ4Q26YEAYW7VM2TNRZEPMD3H3NIKDDIQSQINUQ, seg_pos={0 0}, upload=645558]: Done uploading piece. err: <nil>
Jan 5 09:15:32.388825311 (737h46m32.259908356s) [seg_pos={0 0}, upload=645558]: Segment reached optimal threshold of 14 pieces.
Jan 5 09:15:32.389001336 (737h46m32.260084371s) [noise=true , address=64.49.13.169:4713 , piece_id=CEEOAYFOSQJOYMTKOBI2X2YGJV2D2VI2EZAWWQQVE7Z27GVUSIBA, seg_pos={0 0}, upload=645558]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17z3AiUfbxBX2eMDnUnqo1bbgojuQxJro7PsEaPjYzQjHQ3D6K): protocol: write tcp 188.166.225.170:38796->64.49.13.169:4713: use of closed network connection; write tcp 188.166.225.170:38796->64.49.13.169:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38796->64.49.13.169:4713: use of closed network connection
Jan 5 09:15:32.389007137 (737h46m32.260090182s) [noise=true , address=64.49.13.169:4713 , piece_id=CEEOAYFOSQJOYMTKOBI2X2YGJV2D2VI2EZAWWQQVE7Z27GVUSIBA, seg_pos={0 0}, upload=645558]: Long tail cancellation due to optimal threshold
Jan 5 09:15:32.389141393 (737h46m32.260224438s) [noise=true , address=45.91.116.4:4954 , piece_id=VOSR3HGKKGKIQOSQAO7OSILTNZIAHYJMUIFB7GAGQTJP3U6BSJPQ, seg_pos={0 0}, upload=645558]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tT1yhzA7AUuwesNf62NHyVxgSvbrmFzchYSmYdqihRjuAgvYa): protocol: expected piece hash; context canceled; EOF
Jan 5 09:15:32.389147284 (737h46m32.260230319s) [noise=true , address=45.91.116.4:4954 , piece_id=VOSR3HGKKGKIQOSQAO7OSILTNZIAHYJMUIFB7GAGQTJP3U6BSJPQ, seg_pos={0 0}, upload=645558]: Long tail cancellation due to optimal threshold
Jan 5 09:15:32.389207699 (737h46m32.260290734s) [noise=true , address=64.49.13.155:4923 , piece_id=E6X3TYE7TAZJ7Z7JYPEXRPUZJV2PGDZU674ONMXZNRUSYBVOH3KA, seg_pos={0 0}, upload=645558]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CJVCqcpsnRfG6zJwpd4Bkm4dKX7DV5AFEesUcRs38VUPuxS3z): protocol: expected piece hash; context canceled; EOF
Jan 5 09:15:32.389216355 (737h46m32.26029939s) [noise=true , address=64.49.13.155:4923 , piece_id=E6X3TYE7TAZJ7Z7JYPEXRPUZJV2PGDZU674ONMXZNRUSYBVOH3KA, seg_pos={0 0}, upload=645558]: Long tail cancellation due to optimal threshold
Jan 5 09:15:32.389247344 (737h46m32.260330379s) [noise=true , address=145.79.227.33:4899 , piece_id=KM6CEND447VWRRLLK57KSQ4MIFGQIK5FHX47PMHBRVHVSVMFCZ6A, seg_pos={0 0}, upload=645558]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b4pZVQRnjcQNn45VTgdsz5fpjYPMAzSXGR5KJ5BP9XHMkRnRw): protocol: expected piece hash; context canceled; EOF
Jan 5 09:15:32.389252634 (737h46m32.260335669s) [noise=true , address=145.79.227.33:4899 , piece_id=KM6CEND447VWRRLLK57KSQ4MIFGQIK5FHX47PMHBRVHVSVMFCZ6A, seg_pos={0 0}, upload=645558]: Long tail cancellation due to optimal threshold
Jan 5 09:15:32.389325463 (737h46m32.260408498s) [seg_pos={0 0}, upload=645558]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jan 5 09:15:32.389386449 (737h46m32.260469494s) [upload=645558]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 5 09:15:32.389390857 (737h46m32.260473892s) [upload=645558]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 5 09:15:32.693933677 (737h46m32.565016712s) [upload=645558]: Done uploading segments...
Jan 5 09:15:32.693952283 (737h46m32.565035318s) [upload=645558]: Upload finished. err: <nil>
Duration: 15.413503046s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [..........X.............................................X...] duration:
12.903421212s trace:
https://jaeger.dev.storj.io/trace/35e6dfe601545a69
FAIL (2026-01-05 09:17:41 UTC)
orbiter warning: context deadline exceeded
Duration: 12.903446359s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
13 out of 60 is failed (threshold=3) [.X.....X.X...X....................X..X..X.X.X.X..X...X...X..] duration:
5.811694075s trace:
https://jaeger.dev.storj.io/trace/1b4f05ae09b9a718
FAIL (2026-01-05 09:17:40 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Jan 5 09:16:40.595641966 (737h47m40.466725011s) [upload=645576]: Starting upload
Jan 5 09:16:40.595728321 (737h47m40.466811366s) [upload=645576]: Done starting upload
Jan 5 09:16:40.595774108 (737h47m40.466857163s) [upload=645576]: Uploading segments...
Jan 5 09:16:40.596067808 (737h47m40.467150843s) [upload=645576]: Got next segment. Inline: false
Jan 5 09:16:40.596103877 (737h47m40.467186922s) [upload=645576]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 5 09:16:40.596108565 (737h47m40.467191601s) [upload=645576]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 5 09:16:40.846965278 (737h47m40.718048323s) [seg_pos={0 0}, upload=645576]: Begin upload segment...
Jan 5 09:16:40.847049107 (737h47m40.718132152s) [seg_pos={0 0}, upload=645576]: Done begin upload segment.
Jan 5 09:16:40.847085095 (737h47m40.71816814s) [noise=true , address=145.79.227.11:4729 , piece_id=XZ33UE2YPTNVUYB6G7NG3ITFO46KYJRMLPHHEV7OS6GZJS47C27A, seg_pos={0 0}, upload=645576]: Uploading piece...
Jan 5 09:16:40.847959360 (737h47m40.719042405s) [noise=true , address=45.91.116.6:4974 , piece_id=4TZIH3RYDVNATCAV5GSVS527SEIKDLOIZQ7SF4Q4K7DCGTI6FIXQ, seg_pos={0 0}, upload=645576]: Uploading piece...
Jan 5 09:16:40.848024484 (737h47m40.719107519s) [noise=true , address=145.79.227.42:4121 , piece_id=ABSMYZAZCG3ZLGE2N7NRHGGSMPD7A6IQHPAS43ODCJBJGDERN3EA, seg_pos={0 0}, upload=645576]: Uploading piece...
Jan 5 09:16:40.848108143 (737h47m40.719191188s) [noise=true , address=145.79.227.36:4518 , piece_id=3DAHKIJEWOBIU5TOQIQQPGBN5XS56TERBZCRW2IXCWSSSTGMUJXQ, seg_pos={0 0}, upload=645576]: Uploading piece...
Jan 5 09:16:40.848615800 (737h47m40.719698845s) [noise=true , address=145.79.227.39:4015 , piece_id=76WUTGCPQE4D3AMKS7OJ6KFUTTCTGGFKO62OIDDJUPKNP3VHNDQQ, seg_pos={0 0}, upload=645576]: Uploading piece...
Jan 5 09:16:40.848791194 (737h47m40.719874229s) [noise=true , address=145.79.227.18:4007 , piece_id=I5UGDNLW2DIMVZVUT4OSAI272ZGLJOAYUCOAPTOUO74F5ABLDD5Q, seg_pos={0 0}, upload=645576]: Uploading piece...
Jan 5 09:16:40.849091476 (737h47m40.720174511s) [noise=true , address=145.79.227.50:4374 , piece_id=C7ST7BE7KVI55PKMTHFUSASZUY56FISZDYJZLXMBWWESGJ66Q4YQ, seg_pos={0 0}, upload=645576]: Uploading piece...
Jan 5 09:16:40.849189002 (737h47m40.720272037s) [noise=true , address=145.79.227.5:4221 , piece_id=LCSNTVLD5CCWD42IA3FG3KR4U4Y7G7UY6MSURPDYVF6WVOIQ3RVQ, seg_pos={0 0}, upload=645576]: Uploading piece...
Jan 5 09:16:40.849547575 (737h47m40.72063062s) [noise=true , address=45.91.116.11:4120 , piece_id=RB63LVVOOHN7T6ZSSHLB62WEONWRY2VXU5DWRQEUGE7YPH7ZCXDA, seg_pos={0 0}, upload=645576]: Uploading piece...
Jan 5 09:16:40.849978385 (737h47m40.72106143s) [noise=true , address=77.90.31.19:4842 , piece_id=CEZIHJX2YXU4EASHV67R6H7JP477TDCZHJ7EICDRGWPYEDA47LBA, seg_pos={0 0}, upload=645576]: Uploading piece...
Jan 5 09:16:40.850113472 (737h47m40.721196507s) [noise=true , address=145.79.227.4:4187 , piece_id=PUKU4SKHT3HVBFQUMZ7N2LT67Z6KHWD6V627WYDVAZF4YCBX55MQ, seg_pos={0 0}, upload=645576]: Uploading piece...
Jan 5 09:16:40.850604878 (737h47m40.721687923s) [noise=true , address=45.91.116.12:4227 , piece_id=BSUEGGASFYIOXG5ZBGZAB5TYJDNG7SGKEHB435OP3XXJK5WLKINA, seg_pos={0 0}, upload=645576]: Uploading piece...
Jan 5 09:16:40.850936439 (737h47m40.722019484s) [noise=true , address=145.79.227.6:4257 , piece_id=J7EGP2E3WBB4RBZEHZKZMOGYZJCET2ZE4SOQCJUOO5NLSHSLLJOA, seg_pos={0 0}, upload=645576]: Uploading piece...
Jan 5 09:16:40.851097315 (737h47m40.72218036s) [noise=true , address=45.91.116.13:4155 , piece_id=5TVSIH44JX7LXZHDOBBOYXVOGWWF3EWJM2JMZW4ACECKQTILHYQQ, seg_pos={0 0}, upload=645576]: Uploading piece...
Jan 5 09:16:40.851583541 (737h47m40.722666596s) [noise=true , address=77.90.31.13:4626 , piece_id=Q3LKMUFAPOCLHELCFMSXDKJI2JIDR33DTVZJZOUTMBG3SLRUWLMQ, seg_pos={0 0}, upload=645576]: Uploading piece...
Jan 5 09:16:40.852207930 (737h47m40.723290975s) [noise=true , address=77.90.31.17:4770 , piece_id=5EL6CHYFHCGKDL3EWHWMWQCE5QNLB3B3PGIICHCMDV3ZHW2VKIRQ, seg_pos={0 0}, upload=645576]: Uploading piece...
Jan 5 09:16:42.549913379 (737h47m42.420996414s) [upload=645576]: Next returned nil segment
Jan 5 09:16:42.549931083 (737h47m42.421014128s) [upload=645576]: Waiting for error group managing segments...
Jan 5 09:16:42.825075993 (737h47m42.696159028s) [noise=true , address=145.79.227.11:4729 , piece_id=XZ33UE2YPTNVUYB6G7NG3ITFO46KYJRMLPHHEV7OS6GZJS47C27A, seg_pos={0 0}, upload=645576]: Done uploading piece. err: <nil>
Jan 5 09:16:42.837593318 (737h47m42.708676353s) [noise=true , address=145.79.227.50:4374 , piece_id=C7ST7BE7KVI55PKMTHFUSASZUY56FISZDYJZLXMBWWESGJ66Q4YQ, seg_pos={0 0}, upload=645576]: Done uploading piece. err: <nil>
Jan 5 09:16:42.838358815 (737h47m42.709441851s) [noise=true , address=145.79.227.6:4257 , piece_id=J7EGP2E3WBB4RBZEHZKZMOGYZJCET2ZE4SOQCJUOO5NLSHSLLJOA, seg_pos={0 0}, upload=645576]: Done uploading piece. err: <nil>
Jan 5 09:16:42.842806805 (737h47m42.713889851s) [noise=true , address=145.79.227.42:4121 , piece_id=ABSMYZAZCG3ZLGE2N7NRHGGSMPD7A6IQHPAS43ODCJBJGDERN3EA, seg_pos={0 0}, upload=645576]: Done uploading piece. err: <nil>
Jan 5 09:16:42.847750569 (737h47m42.718833624s) [noise=true , address=145.79.227.18:4007 , piece_id=I5UGDNLW2DIMVZVUT4OSAI272ZGLJOAYUCOAPTOUO74F5ABLDD5Q, seg_pos={0 0}, upload=645576]: Done uploading piece. err: <nil>
Jan 5 09:16:42.854966409 (737h47m42.726049454s) [noise=true , address=145.79.227.39:4015 , piece_id=76WUTGCPQE4D3AMKS7OJ6KFUTTCTGGFKO62OIDDJUPKNP3VHNDQQ, seg_pos={0 0}, upload=645576]: Done uploading piece. err: <nil>
Jan 5 09:16:42.855447646 (737h47m42.726530691s) [noise=true , address=145.79.227.4:4187 , piece_id=PUKU4SKHT3HVBFQUMZ7N2LT67Z6KHWD6V627WYDVAZF4YCBX55MQ, seg_pos={0 0}, upload=645576]: Done uploading piece. err: <nil>
Jan 5 09:16:42.862649790 (737h47m42.733732825s) [noise=true , address=145.79.227.5:4221 , piece_id=LCSNTVLD5CCWD42IA3FG3KR4U4Y7G7UY6MSURPDYVF6WVOIQ3RVQ, seg_pos={0 0}, upload=645576]: Done uploading piece. err: <nil>
Jan 5 09:16:42.869453876 (737h47m42.740536921s) [noise=true , address=145.79.227.36:4518 , piece_id=3DAHKIJEWOBIU5TOQIQQPGBN5XS56TERBZCRW2IXCWSSSTGMUJXQ, seg_pos={0 0}, upload=645576]: Done uploading piece. err: <nil>
Jan 5 09:16:43.277955009 (737h47m43.149038044s) [noise=true , address=77.90.31.17:4770 , piece_id=5EL6CHYFHCGKDL3EWHWMWQCE5QNLB3B3PGIICHCMDV3ZHW2VKIRQ, seg_pos={0 0}, upload=645576]: Done uploading piece. err: <nil>
Jan 5 09:16:43.607405858 (737h47m43.478488893s) [noise=true , address=77.90.31.19:4842 , piece_id=CEZIHJX2YXU4EASHV67R6H7JP477TDCZHJ7EICDRGWPYEDA47LBA, seg_pos={0 0}, upload=645576]: Done uploading piece. err: <nil>
Jan 5 09:16:43.825842666 (737h47m43.696925701s) [noise=true , address=77.90.31.13:4626 , piece_id=Q3LKMUFAPOCLHELCFMSXDKJI2JIDR33DTVZJZOUTMBG3SLRUWLMQ, seg_pos={0 0}, upload=645576]: Done uploading piece. err: <nil>
Jan 5 09:16:44.199244073 (737h47m44.070327118s) [noise=true , address=45.91.116.13:4155 , piece_id=5TVSIH44JX7LXZHDOBBOYXVOGWWF3EWJM2JMZW4ACECKQTILHYQQ, seg_pos={0 0}, upload=645576]: Done uploading piece. err: <nil>
Jan 5 09:16:44.199268049 (737h47m44.070351094s) [seg_pos={0 0}, upload=645576]: Segment reached optimal threshold of 13 pieces.
Jan 5 09:16:44.199477518 (737h47m44.070560563s) [noise=true , address=45.91.116.6:4974 , piece_id=4TZIH3RYDVNATCAV5GSVS527SEIKDLOIZQ7SF4Q4K7DCGTI6FIXQ, seg_pos={0 0}, upload=645576]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF
Jan 5 09:16:44.199484531 (737h47m44.070567566s) [noise=true , address=45.91.116.6:4974 , piece_id=4TZIH3RYDVNATCAV5GSVS527SEIKDLOIZQ7SF4Q4K7DCGTI6FIXQ, seg_pos={0 0}, upload=645576]: Long tail cancellation due to optimal threshold
Jan 5 09:16:44.199589711 (737h47m44.070672766s) [noise=true , address=45.91.116.11:4120 , piece_id=RB63LVVOOHN7T6ZSSHLB62WEONWRY2VXU5DWRQEUGE7YPH7ZCXDA, seg_pos={0 0}, upload=645576]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF
Jan 5 09:16:44.199605111 (737h47m44.070688156s) [noise=true , address=45.91.116.11:4120 , piece_id=RB63LVVOOHN7T6ZSSHLB62WEONWRY2VXU5DWRQEUGE7YPH7ZCXDA, seg_pos={0 0}, upload=645576]: Long tail cancellation due to optimal threshold
Jan 5 09:16:44.199788771 (737h47m44.070871816s) [noise=true , address=45.91.116.12:4227 , piece_id=BSUEGGASFYIOXG5ZBGZAB5TYJDNG7SGKEHB435OP3XXJK5WLKINA, seg_pos={0 0}, upload=645576]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: read tcp 188.166.225.170:45038->45.91.116.12:4227: use of closed network connection; context canceled
Jan 5 09:16:44.199796185 (737h47m44.07087922s) [noise=true , address=45.91.116.12:4227 , piece_id=BSUEGGASFYIOXG5ZBGZAB5TYJDNG7SGKEHB435OP3XXJK5WLKINA, seg_pos={0 0}, upload=645576]: Long tail cancellation due to optimal threshold
Jan 5 09:16:44.199813127 (737h47m44.070896162s) [seg_pos={0 0}, upload=645576]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 5 09:16:44.199920281 (737h47m44.071003316s) [upload=645576]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 5 09:16:44.199926894 (737h47m44.071009939s) [upload=645576]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 5 09:16:44.505038446 (737h47m44.376121491s) [upload=645576]: Done uploading segments...
Jan 5 09:16:44.505056279 (737h47m44.376139314s) [upload=645576]: Upload finished. err: <nil>
Duration: 5.811799045s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
43 out of 60 is failed (threshold=6) [.X.XXXXXXXX.XX..XXX.X.XX.XXX.XX.XXXXXX..XXXXXX...XXXX..XXXXX] duration:
5.001416934s trace:
https://jaeger.dev.storj.io/trace/20168e86db8ed863
FAIL (2026-01-05 09:17:23 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001441362s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [...........X.............................................X..] duration:
7.813038801s trace:
https://jaeger.dev.storj.io/trace/35e6dfe601545a69
FAIL (2026-01-05 09:17:19 UTC)
orbiter warning: context deadline exceeded
Duration: 7.813068647s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
14 out of 60 is failed (threshold=3) [X.X.....X.X...X....................X..X..X.X.X.X..X...X...X.] duration:
6.825547328s trace:
https://jaeger.dev.storj.io/trace/1b4f05ae09b9a718
FAIL (2026-01-05 09:17:16 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Jan 5 09:16:40.595641966 (737h47m40.466725011s) [upload=645576]: Starting upload
Jan 5 09:16:40.595728321 (737h47m40.466811366s) [upload=645576]: Done starting upload
Jan 5 09:16:40.595774108 (737h47m40.466857163s) [upload=645576]: Uploading segments...
Jan 5 09:16:40.596067808 (737h47m40.467150843s) [upload=645576]: Got next segment. Inline: false
Jan 5 09:16:40.596103877 (737h47m40.467186922s) [upload=645576]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 5 09:16:40.596108565 (737h47m40.467191601s) [upload=645576]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 5 09:16:40.846965278 (737h47m40.718048323s) [seg_pos={0 0}, upload=645576]: Begin upload segment...
Jan 5 09:16:40.847049107 (737h47m40.718132152s) [seg_pos={0 0}, upload=645576]: Done begin upload segment.
Jan 5 09:16:40.847085095 (737h47m40.71816814s) [noise=true , address=145.79.227.11:4729 , piece_id=XZ33UE2YPTNVUYB6G7NG3ITFO46KYJRMLPHHEV7OS6GZJS47C27A, seg_pos={0 0}, upload=645576]: Uploading piece...
Jan 5 09:16:40.847959360 (737h47m40.719042405s) [noise=true , address=45.91.116.6:4974 , piece_id=4TZIH3RYDVNATCAV5GSVS527SEIKDLOIZQ7SF4Q4K7DCGTI6FIXQ, seg_pos={0 0}, upload=645576]: Uploading piece...
Jan 5 09:16:40.848024484 (737h47m40.719107519s) [noise=true , address=145.79.227.42:4121 , piece_id=ABSMYZAZCG3ZLGE2N7NRHGGSMPD7A6IQHPAS43ODCJBJGDERN3EA, seg_pos={0 0}, upload=645576]: Uploading piece...
Jan 5 09:16:40.848108143 (737h47m40.719191188s) [noise=true , address=145.79.227.36:4518 , piece_id=3DAHKIJEWOBIU5TOQIQQPGBN5XS56TERBZCRW2IXCWSSSTGMUJXQ, seg_pos={0 0}, upload=645576]: Uploading piece...
Jan 5 09:16:40.848615800 (737h47m40.719698845s) [noise=true , address=145.79.227.39:4015 , piece_id=76WUTGCPQE4D3AMKS7OJ6KFUTTCTGGFKO62OIDDJUPKNP3VHNDQQ, seg_pos={0 0}, upload=645576]: Uploading piece...
Jan 5 09:16:40.848791194 (737h47m40.719874229s) [noise=true , address=145.79.227.18:4007 , piece_id=I5UGDNLW2DIMVZVUT4OSAI272ZGLJOAYUCOAPTOUO74F5ABLDD5Q, seg_pos={0 0}, upload=645576]: Uploading piece...
Jan 5 09:16:40.849091476 (737h47m40.720174511s) [noise=true , address=145.79.227.50:4374 , piece_id=C7ST7BE7KVI55PKMTHFUSASZUY56FISZDYJZLXMBWWESGJ66Q4YQ, seg_pos={0 0}, upload=645576]: Uploading piece...
Jan 5 09:16:40.849189002 (737h47m40.720272037s) [noise=true , address=145.79.227.5:4221 , piece_id=LCSNTVLD5CCWD42IA3FG3KR4U4Y7G7UY6MSURPDYVF6WVOIQ3RVQ, seg_pos={0 0}, upload=645576]: Uploading piece...
Jan 5 09:16:40.849547575 (737h47m40.72063062s) [noise=true , address=45.91.116.11:4120 , piece_id=RB63LVVOOHN7T6ZSSHLB62WEONWRY2VXU5DWRQEUGE7YPH7ZCXDA, seg_pos={0 0}, upload=645576]: Uploading piece...
Jan 5 09:16:40.849978385 (737h47m40.72106143s) [noise=true , address=77.90.31.19:4842 , piece_id=CEZIHJX2YXU4EASHV67R6H7JP477TDCZHJ7EICDRGWPYEDA47LBA, seg_pos={0 0}, upload=645576]: Uploading piece...
Jan 5 09:16:40.850113472 (737h47m40.721196507s) [noise=true , address=145.79.227.4:4187 , piece_id=PUKU4SKHT3HVBFQUMZ7N2LT67Z6KHWD6V627WYDVAZF4YCBX55MQ, seg_pos={0 0}, upload=645576]: Uploading piece...
Jan 5 09:16:40.850604878 (737h47m40.721687923s) [noise=true , address=45.91.116.12:4227 , piece_id=BSUEGGASFYIOXG5ZBGZAB5TYJDNG7SGKEHB435OP3XXJK5WLKINA, seg_pos={0 0}, upload=645576]: Uploading piece...
Jan 5 09:16:40.850936439 (737h47m40.722019484s) [noise=true , address=145.79.227.6:4257 , piece_id=J7EGP2E3WBB4RBZEHZKZMOGYZJCET2ZE4SOQCJUOO5NLSHSLLJOA, seg_pos={0 0}, upload=645576]: Uploading piece...
Jan 5 09:16:40.851097315 (737h47m40.72218036s) [noise=true , address=45.91.116.13:4155 , piece_id=5TVSIH44JX7LXZHDOBBOYXVOGWWF3EWJM2JMZW4ACECKQTILHYQQ, seg_pos={0 0}, upload=645576]: Uploading piece...
Jan 5 09:16:40.851583541 (737h47m40.722666596s) [noise=true , address=77.90.31.13:4626 , piece_id=Q3LKMUFAPOCLHELCFMSXDKJI2JIDR33DTVZJZOUTMBG3SLRUWLMQ, seg_pos={0 0}, upload=645576]: Uploading piece...
Jan 5 09:16:40.852207930 (737h47m40.723290975s) [noise=true , address=77.90.31.17:4770 , piece_id=5EL6CHYFHCGKDL3EWHWMWQCE5QNLB3B3PGIICHCMDV3ZHW2VKIRQ, seg_pos={0 0}, upload=645576]: Uploading piece...
Jan 5 09:16:42.549913379 (737h47m42.420996414s) [upload=645576]: Next returned nil segment
Jan 5 09:16:42.549931083 (737h47m42.421014128s) [upload=645576]: Waiting for error group managing segments...
Jan 5 09:16:42.825075993 (737h47m42.696159028s) [noise=true , address=145.79.227.11:4729 , piece_id=XZ33UE2YPTNVUYB6G7NG3ITFO46KYJRMLPHHEV7OS6GZJS47C27A, seg_pos={0 0}, upload=645576]: Done uploading piece. err: <nil>
Jan 5 09:16:42.837593318 (737h47m42.708676353s) [noise=true , address=145.79.227.50:4374 , piece_id=C7ST7BE7KVI55PKMTHFUSASZUY56FISZDYJZLXMBWWESGJ66Q4YQ, seg_pos={0 0}, upload=645576]: Done uploading piece. err: <nil>
Jan 5 09:16:42.838358815 (737h47m42.709441851s) [noise=true , address=145.79.227.6:4257 , piece_id=J7EGP2E3WBB4RBZEHZKZMOGYZJCET2ZE4SOQCJUOO5NLSHSLLJOA, seg_pos={0 0}, upload=645576]: Done uploading piece. err: <nil>
Jan 5 09:16:42.842806805 (737h47m42.713889851s) [noise=true , address=145.79.227.42:4121 , piece_id=ABSMYZAZCG3ZLGE2N7NRHGGSMPD7A6IQHPAS43ODCJBJGDERN3EA, seg_pos={0 0}, upload=645576]: Done uploading piece. err: <nil>
Jan 5 09:16:42.847750569 (737h47m42.718833624s) [noise=true , address=145.79.227.18:4007 , piece_id=I5UGDNLW2DIMVZVUT4OSAI272ZGLJOAYUCOAPTOUO74F5ABLDD5Q, seg_pos={0 0}, upload=645576]: Done uploading piece. err: <nil>
Jan 5 09:16:42.854966409 (737h47m42.726049454s) [noise=true , address=145.79.227.39:4015 , piece_id=76WUTGCPQE4D3AMKS7OJ6KFUTTCTGGFKO62OIDDJUPKNP3VHNDQQ, seg_pos={0 0}, upload=645576]: Done uploading piece. err: <nil>
Jan 5 09:16:42.855447646 (737h47m42.726530691s) [noise=true , address=145.79.227.4:4187 , piece_id=PUKU4SKHT3HVBFQUMZ7N2LT67Z6KHWD6V627WYDVAZF4YCBX55MQ, seg_pos={0 0}, upload=645576]: Done uploading piece. err: <nil>
Jan 5 09:16:42.862649790 (737h47m42.733732825s) [noise=true , address=145.79.227.5:4221 , piece_id=LCSNTVLD5CCWD42IA3FG3KR4U4Y7G7UY6MSURPDYVF6WVOIQ3RVQ, seg_pos={0 0}, upload=645576]: Done uploading piece. err: <nil>
Jan 5 09:16:42.869453876 (737h47m42.740536921s) [noise=true , address=145.79.227.36:4518 , piece_id=3DAHKIJEWOBIU5TOQIQQPGBN5XS56TERBZCRW2IXCWSSSTGMUJXQ, seg_pos={0 0}, upload=645576]: Done uploading piece. err: <nil>
Jan 5 09:16:43.277955009 (737h47m43.149038044s) [noise=true , address=77.90.31.17:4770 , piece_id=5EL6CHYFHCGKDL3EWHWMWQCE5QNLB3B3PGIICHCMDV3ZHW2VKIRQ, seg_pos={0 0}, upload=645576]: Done uploading piece. err: <nil>
Jan 5 09:16:43.607405858 (737h47m43.478488893s) [noise=true , address=77.90.31.19:4842 , piece_id=CEZIHJX2YXU4EASHV67R6H7JP477TDCZHJ7EICDRGWPYEDA47LBA, seg_pos={0 0}, upload=645576]: Done uploading piece. err: <nil>
Jan 5 09:16:43.825842666 (737h47m43.696925701s) [noise=true , address=77.90.31.13:4626 , piece_id=Q3LKMUFAPOCLHELCFMSXDKJI2JIDR33DTVZJZOUTMBG3SLRUWLMQ, seg_pos={0 0}, upload=645576]: Done uploading piece. err: <nil>
Jan 5 09:16:44.199244073 (737h47m44.070327118s) [noise=true , address=45.91.116.13:4155 , piece_id=5TVSIH44JX7LXZHDOBBOYXVOGWWF3EWJM2JMZW4ACECKQTILHYQQ, seg_pos={0 0}, upload=645576]: Done uploading piece. err: <nil>
Jan 5 09:16:44.199268049 (737h47m44.070351094s) [seg_pos={0 0}, upload=645576]: Segment reached optimal threshold of 13 pieces.
Jan 5 09:16:44.199477518 (737h47m44.070560563s) [noise=true , address=45.91.116.6:4974 , piece_id=4TZIH3RYDVNATCAV5GSVS527SEIKDLOIZQ7SF4Q4K7DCGTI6FIXQ, seg_pos={0 0}, upload=645576]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF
Jan 5 09:16:44.199484531 (737h47m44.070567566s) [noise=true , address=45.91.116.6:4974 , piece_id=4TZIH3RYDVNATCAV5GSVS527SEIKDLOIZQ7SF4Q4K7DCGTI6FIXQ, seg_pos={0 0}, upload=645576]: Long tail cancellation due to optimal threshold
Jan 5 09:16:44.199589711 (737h47m44.070672766s) [noise=true , address=45.91.116.11:4120 , piece_id=RB63LVVOOHN7T6ZSSHLB62WEONWRY2VXU5DWRQEUGE7YPH7ZCXDA, seg_pos={0 0}, upload=645576]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF
Jan 5 09:16:44.199605111 (737h47m44.070688156s) [noise=true , address=45.91.116.11:4120 , piece_id=RB63LVVOOHN7T6ZSSHLB62WEONWRY2VXU5DWRQEUGE7YPH7ZCXDA, seg_pos={0 0}, upload=645576]: Long tail cancellation due to optimal threshold
Jan 5 09:16:44.199788771 (737h47m44.070871816s) [noise=true , address=45.91.116.12:4227 , piece_id=BSUEGGASFYIOXG5ZBGZAB5TYJDNG7SGKEHB435OP3XXJK5WLKINA, seg_pos={0 0}, upload=645576]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: read tcp 188.166.225.170:45038->45.91.116.12:4227: use of closed network connection; context canceled
Jan 5 09:16:44.199796185 (737h47m44.07087922s) [noise=true , address=45.91.116.12:4227 , piece_id=BSUEGGASFYIOXG5ZBGZAB5TYJDNG7SGKEHB435OP3XXJK5WLKINA, seg_pos={0 0}, upload=645576]: Long tail cancellation due to optimal threshold
Jan 5 09:16:44.199813127 (737h47m44.070896162s) [seg_pos={0 0}, upload=645576]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 5 09:16:44.199920281 (737h47m44.071003316s) [upload=645576]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 5 09:16:44.199926894 (737h47m44.071009939s) [upload=645576]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 5 09:16:44.505038446 (737h47m44.376121491s) [upload=645576]: Done uploading segments...
Jan 5 09:16:44.505056279 (737h47m44.376139314s) [upload=645576]: Upload finished. err: <nil>
Duration: 6.825572916s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
14 out of 60 is failed (threshold=3) [.X.X.....X.X...X....................X..X..X.X.X.X..X...X...X] duration:
30.002472419s trace:
https://jaeger.dev.storj.io/trace/1b4f05ae09b9a718
FAIL (2026-01-05 09:17:09 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Jan 5 09:16:40.595641966 (737h47m40.466725011s) [upload=645576]: Starting upload
Jan 5 09:16:40.595728321 (737h47m40.466811366s) [upload=645576]: Done starting upload
Jan 5 09:16:40.595774108 (737h47m40.466857163s) [upload=645576]: Uploading segments...
Jan 5 09:16:40.596067808 (737h47m40.467150843s) [upload=645576]: Got next segment. Inline: false
Jan 5 09:16:40.596103877 (737h47m40.467186922s) [upload=645576]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 5 09:16:40.596108565 (737h47m40.467191601s) [upload=645576]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 5 09:16:40.846965278 (737h47m40.718048323s) [seg_pos={0 0}, upload=645576]: Begin upload segment...
Jan 5 09:16:40.847049107 (737h47m40.718132152s) [seg_pos={0 0}, upload=645576]: Done begin upload segment.
Jan 5 09:16:40.847085095 (737h47m40.71816814s) [noise=true , address=145.79.227.11:4729 , piece_id=XZ33UE2YPTNVUYB6G7NG3ITFO46KYJRMLPHHEV7OS6GZJS47C27A, seg_pos={0 0}, upload=645576]: Uploading piece...
Jan 5 09:16:40.847959360 (737h47m40.719042405s) [noise=true , address=45.91.116.6:4974 , piece_id=4TZIH3RYDVNATCAV5GSVS527SEIKDLOIZQ7SF4Q4K7DCGTI6FIXQ, seg_pos={0 0}, upload=645576]: Uploading piece...
Jan 5 09:16:40.848024484 (737h47m40.719107519s) [noise=true , address=145.79.227.42:4121 , piece_id=ABSMYZAZCG3ZLGE2N7NRHGGSMPD7A6IQHPAS43ODCJBJGDERN3EA, seg_pos={0 0}, upload=645576]: Uploading piece...
Jan 5 09:16:40.848108143 (737h47m40.719191188s) [noise=true , address=145.79.227.36:4518 , piece_id=3DAHKIJEWOBIU5TOQIQQPGBN5XS56TERBZCRW2IXCWSSSTGMUJXQ, seg_pos={0 0}, upload=645576]: Uploading piece...
Jan 5 09:16:40.848615800 (737h47m40.719698845s) [noise=true , address=145.79.227.39:4015 , piece_id=76WUTGCPQE4D3AMKS7OJ6KFUTTCTGGFKO62OIDDJUPKNP3VHNDQQ, seg_pos={0 0}, upload=645576]: Uploading piece...
Jan 5 09:16:40.848791194 (737h47m40.719874229s) [noise=true , address=145.79.227.18:4007 , piece_id=I5UGDNLW2DIMVZVUT4OSAI272ZGLJOAYUCOAPTOUO74F5ABLDD5Q, seg_pos={0 0}, upload=645576]: Uploading piece...
Jan 5 09:16:40.849091476 (737h47m40.720174511s) [noise=true , address=145.79.227.50:4374 , piece_id=C7ST7BE7KVI55PKMTHFUSASZUY56FISZDYJZLXMBWWESGJ66Q4YQ, seg_pos={0 0}, upload=645576]: Uploading piece...
Jan 5 09:16:40.849189002 (737h47m40.720272037s) [noise=true , address=145.79.227.5:4221 , piece_id=LCSNTVLD5CCWD42IA3FG3KR4U4Y7G7UY6MSURPDYVF6WVOIQ3RVQ, seg_pos={0 0}, upload=645576]: Uploading piece...
Jan 5 09:16:40.849547575 (737h47m40.72063062s) [noise=true , address=45.91.116.11:4120 , piece_id=RB63LVVOOHN7T6ZSSHLB62WEONWRY2VXU5DWRQEUGE7YPH7ZCXDA, seg_pos={0 0}, upload=645576]: Uploading piece...
Jan 5 09:16:40.849978385 (737h47m40.72106143s) [noise=true , address=77.90.31.19:4842 , piece_id=CEZIHJX2YXU4EASHV67R6H7JP477TDCZHJ7EICDRGWPYEDA47LBA, seg_pos={0 0}, upload=645576]: Uploading piece...
Jan 5 09:16:40.850113472 (737h47m40.721196507s) [noise=true , address=145.79.227.4:4187 , piece_id=PUKU4SKHT3HVBFQUMZ7N2LT67Z6KHWD6V627WYDVAZF4YCBX55MQ, seg_pos={0 0}, upload=645576]: Uploading piece...
Jan 5 09:16:40.850604878 (737h47m40.721687923s) [noise=true , address=45.91.116.12:4227 , piece_id=BSUEGGASFYIOXG5ZBGZAB5TYJDNG7SGKEHB435OP3XXJK5WLKINA, seg_pos={0 0}, upload=645576]: Uploading piece...
Jan 5 09:16:40.850936439 (737h47m40.722019484s) [noise=true , address=145.79.227.6:4257 , piece_id=J7EGP2E3WBB4RBZEHZKZMOGYZJCET2ZE4SOQCJUOO5NLSHSLLJOA, seg_pos={0 0}, upload=645576]: Uploading piece...
Jan 5 09:16:40.851097315 (737h47m40.72218036s) [noise=true , address=45.91.116.13:4155 , piece_id=5TVSIH44JX7LXZHDOBBOYXVOGWWF3EWJM2JMZW4ACECKQTILHYQQ, seg_pos={0 0}, upload=645576]: Uploading piece...
Jan 5 09:16:40.851583541 (737h47m40.722666596s) [noise=true , address=77.90.31.13:4626 , piece_id=Q3LKMUFAPOCLHELCFMSXDKJI2JIDR33DTVZJZOUTMBG3SLRUWLMQ, seg_pos={0 0}, upload=645576]: Uploading piece...
Jan 5 09:16:40.852207930 (737h47m40.723290975s) [noise=true , address=77.90.31.17:4770 , piece_id=5EL6CHYFHCGKDL3EWHWMWQCE5QNLB3B3PGIICHCMDV3ZHW2VKIRQ, seg_pos={0 0}, upload=645576]: Uploading piece...
Jan 5 09:16:42.549913379 (737h47m42.420996414s) [upload=645576]: Next returned nil segment
Jan 5 09:16:42.549931083 (737h47m42.421014128s) [upload=645576]: Waiting for error group managing segments...
Jan 5 09:16:42.825075993 (737h47m42.696159028s) [noise=true , address=145.79.227.11:4729 , piece_id=XZ33UE2YPTNVUYB6G7NG3ITFO46KYJRMLPHHEV7OS6GZJS47C27A, seg_pos={0 0}, upload=645576]: Done uploading piece. err: <nil>
Jan 5 09:16:42.837593318 (737h47m42.708676353s) [noise=true , address=145.79.227.50:4374 , piece_id=C7ST7BE7KVI55PKMTHFUSASZUY56FISZDYJZLXMBWWESGJ66Q4YQ, seg_pos={0 0}, upload=645576]: Done uploading piece. err: <nil>
Jan 5 09:16:42.838358815 (737h47m42.709441851s) [noise=true , address=145.79.227.6:4257 , piece_id=J7EGP2E3WBB4RBZEHZKZMOGYZJCET2ZE4SOQCJUOO5NLSHSLLJOA, seg_pos={0 0}, upload=645576]: Done uploading piece. err: <nil>
Jan 5 09:16:42.842806805 (737h47m42.713889851s) [noise=true , address=145.79.227.42:4121 , piece_id=ABSMYZAZCG3ZLGE2N7NRHGGSMPD7A6IQHPAS43ODCJBJGDERN3EA, seg_pos={0 0}, upload=645576]: Done uploading piece. err: <nil>
Jan 5 09:16:42.847750569 (737h47m42.718833624s) [noise=true , address=145.79.227.18:4007 , piece_id=I5UGDNLW2DIMVZVUT4OSAI272ZGLJOAYUCOAPTOUO74F5ABLDD5Q, seg_pos={0 0}, upload=645576]: Done uploading piece. err: <nil>
Jan 5 09:16:42.854966409 (737h47m42.726049454s) [noise=true , address=145.79.227.39:4015 , piece_id=76WUTGCPQE4D3AMKS7OJ6KFUTTCTGGFKO62OIDDJUPKNP3VHNDQQ, seg_pos={0 0}, upload=645576]: Done uploading piece. err: <nil>
Jan 5 09:16:42.855447646 (737h47m42.726530691s) [noise=true , address=145.79.227.4:4187 , piece_id=PUKU4SKHT3HVBFQUMZ7N2LT67Z6KHWD6V627WYDVAZF4YCBX55MQ, seg_pos={0 0}, upload=645576]: Done uploading piece. err: <nil>
Jan 5 09:16:42.862649790 (737h47m42.733732825s) [noise=true , address=145.79.227.5:4221 , piece_id=LCSNTVLD5CCWD42IA3FG3KR4U4Y7G7UY6MSURPDYVF6WVOIQ3RVQ, seg_pos={0 0}, upload=645576]: Done uploading piece. err: <nil>
Jan 5 09:16:42.869453876 (737h47m42.740536921s) [noise=true , address=145.79.227.36:4518 , piece_id=3DAHKIJEWOBIU5TOQIQQPGBN5XS56TERBZCRW2IXCWSSSTGMUJXQ, seg_pos={0 0}, upload=645576]: Done uploading piece. err: <nil>
Jan 5 09:16:43.277955009 (737h47m43.149038044s) [noise=true , address=77.90.31.17:4770 , piece_id=5EL6CHYFHCGKDL3EWHWMWQCE5QNLB3B3PGIICHCMDV3ZHW2VKIRQ, seg_pos={0 0}, upload=645576]: Done uploading piece. err: <nil>
Jan 5 09:16:43.607405858 (737h47m43.478488893s) [noise=true , address=77.90.31.19:4842 , piece_id=CEZIHJX2YXU4EASHV67R6H7JP477TDCZHJ7EICDRGWPYEDA47LBA, seg_pos={0 0}, upload=645576]: Done uploading piece. err: <nil>
Jan 5 09:16:43.825842666 (737h47m43.696925701s) [noise=true , address=77.90.31.13:4626 , piece_id=Q3LKMUFAPOCLHELCFMSXDKJI2JIDR33DTVZJZOUTMBG3SLRUWLMQ, seg_pos={0 0}, upload=645576]: Done uploading piece. err: <nil>
Jan 5 09:16:44.199244073 (737h47m44.070327118s) [noise=true , address=45.91.116.13:4155 , piece_id=5TVSIH44JX7LXZHDOBBOYXVOGWWF3EWJM2JMZW4ACECKQTILHYQQ, seg_pos={0 0}, upload=645576]: Done uploading piece. err: <nil>
Jan 5 09:16:44.199268049 (737h47m44.070351094s) [seg_pos={0 0}, upload=645576]: Segment reached optimal threshold of 13 pieces.
Jan 5 09:16:44.199477518 (737h47m44.070560563s) [noise=true , address=45.91.116.6:4974 , piece_id=4TZIH3RYDVNATCAV5GSVS527SEIKDLOIZQ7SF4Q4K7DCGTI6FIXQ, seg_pos={0 0}, upload=645576]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF
Jan 5 09:16:44.199484531 (737h47m44.070567566s) [noise=true , address=45.91.116.6:4974 , piece_id=4TZIH3RYDVNATCAV5GSVS527SEIKDLOIZQ7SF4Q4K7DCGTI6FIXQ, seg_pos={0 0}, upload=645576]: Long tail cancellation due to optimal threshold
Jan 5 09:16:44.199589711 (737h47m44.070672766s) [noise=true , address=45.91.116.11:4120 , piece_id=RB63LVVOOHN7T6ZSSHLB62WEONWRY2VXU5DWRQEUGE7YPH7ZCXDA, seg_pos={0 0}, upload=645576]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF
Jan 5 09:16:44.199605111 (737h47m44.070688156s) [noise=true , address=45.91.116.11:4120 , piece_id=RB63LVVOOHN7T6ZSSHLB62WEONWRY2VXU5DWRQEUGE7YPH7ZCXDA, seg_pos={0 0}, upload=645576]: Long tail cancellation due to optimal threshold
Jan 5 09:16:44.199788771 (737h47m44.070871816s) [noise=true , address=45.91.116.12:4227 , piece_id=BSUEGGASFYIOXG5ZBGZAB5TYJDNG7SGKEHB435OP3XXJK5WLKINA, seg_pos={0 0}, upload=645576]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: read tcp 188.166.225.170:45038->45.91.116.12:4227: use of closed network connection; context canceled
Jan 5 09:16:44.199796185 (737h47m44.07087922s) [noise=true , address=45.91.116.12:4227 , piece_id=BSUEGGASFYIOXG5ZBGZAB5TYJDNG7SGKEHB435OP3XXJK5WLKINA, seg_pos={0 0}, upload=645576]: Long tail cancellation due to optimal threshold
Jan 5 09:16:44.199813127 (737h47m44.070896162s) [seg_pos={0 0}, upload=645576]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 5 09:16:44.199920281 (737h47m44.071003316s) [upload=645576]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 5 09:16:44.199926894 (737h47m44.071009939s) [upload=645576]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 5 09:16:44.505038446 (737h47m44.376121491s) [upload=645576]: Done uploading segments...
Jan 5 09:16:44.505056279 (737h47m44.376139314s) [upload=645576]: Upload finished. err: <nil>
Duration: 30.002490463s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
42 out of 60 is failed (threshold=6) [..X.XXXXXXXX.XX..XXX.X.XX.XXX.XX.XXXXXX..XXXXXX...XXXX..XXXX] duration:
5.001255595s trace:
https://jaeger.dev.storj.io/trace/a55caf1705f3ad0
FAIL (2026-01-05 09:16:54 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001268641s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
27 out of 60 is failed (threshold=3) [.X.XX.XXXXXX.XXX...X....XX.X.X....XX....X..XX.XX......X.X.X.] duration:
15.204141523s trace:
https://jaeger.dev.storj.io/trace/1d2dd2d52810438a
FAIL (2026-01-05 09:16:53 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
noise=true , address=145.79.227.33:4899 , piece_id=KM6CEND447VWRRLLK57KSQ4MIFGQIK5FHX47PMHBRVHVSVMFCZ6A, seg_pos={0 0}, upload=645558]: Uploading piece...
Jan 5 09:15:29.200269970 (737h46m29.071353015s) [noise=true , address=77.90.31.18:4817 , piece_id=OTSJYC35STDIU7JBTRELHB5UTGNHNSGQIHFJ65FOMRJB4OFV6QWA, seg_pos={0 0}, upload=645558]: Uploading piece...
Jan 5 09:15:29.200601691 (737h46m29.071684726s) [noise=true , address=77.90.31.11:4560 , piece_id=3ICPCD764NRO4G3DEE2NEQOCOO7RESONGSM5GBAGVFCFFDOPILZA, seg_pos={0 0}, upload=645558]: Uploading piece...
Jan 5 09:15:29.200987557 (737h46m29.072070622s) [noise=true , address=45.91.116.8:4066 , piece_id=JQ2XRTMAWWVVTK4ZVF4OTGSWPD4YMNDQFCPRAOLIQ57KRTXSLHXQ, seg_pos={0 0}, upload=645558]: Uploading piece...
Jan 5 09:15:29.201343163 (737h46m29.072426209s) [noise=true , address=145.79.227.49:4364 , piece_id=SPEZRPRC7BOQ2L535RPH5YZ3WESHYSKVXWIPOXYDKCY7QWOSKP6A, seg_pos={0 0}, upload=645558]: Uploading piece...
Jan 5 09:15:29.201736713 (737h46m29.072819758s) [noise=true , address=145.79.227.20:4457 , piece_id=N3Z7W2HEAJYOYWW6XZNKKNTAQ6UWT6XFLM7ZEGMDVRFINJZ53ICA, seg_pos={0 0}, upload=645558]: Uploading piece...
Jan 5 09:15:29.202243488 (737h46m29.073326523s) [noise=true , address=64.49.13.169:4713 , piece_id=CEEOAYFOSQJOYMTKOBI2X2YGJV2D2VI2EZAWWQQVE7Z27GVUSIBA, seg_pos={0 0}, upload=645558]: Uploading piece...
Jan 5 09:15:29.202665501 (737h46m29.073748546s) [noise=true , address=77.90.31.17:4802 , piece_id=YKMHTVPOA6MF5IH7DTEMPAJBZWBNUGCDHT5E7BYLQWVDUSBLEM3A, seg_pos={0 0}, upload=645558]: Uploading piece...
Jan 5 09:15:29.204332386 (737h46m29.075415421s) [noise=true , address=145.79.227.40:4052 , piece_id=ORA5FT22S3PUHDZ4Q26YEAYW7VM2TNRZEPMD3H3NIKDDIQSQINUQ, seg_pos={0 0}, upload=645558]: Uploading piece...
Jan 5 09:15:29.206264855 (737h46m29.0773479s) [noise=true , address=145.79.227.30:4775 , piece_id=CAF6FWLCUNFVJ5FZW7LYXH244VHHTSCWWUP664MRGJFSXPPRKNWQ, seg_pos={0 0}, upload=645558]: Uploading piece...
Jan 5 09:15:29.207973889 (737h46m29.079056934s) [noise=true , address=145.79.227.26:4664 , piece_id=U5YGCVMR6SHNKZG6TFQTUBO4JBMHBE2X2JBIDVYZ2DYEWRODVB6A, seg_pos={0 0}, upload=645558]: Uploading piece...
Jan 5 09:15:29.209273434 (737h46m29.080356469s) [noise=true , address=145.79.227.43:4165 , piece_id=X2IW45ERSNU6CRZNDKR3R6ZCR42WFCMMIH46I6RCIGBSIA472VKQ, seg_pos={0 0}, upload=645558]: Uploading piece...
Jan 5 09:15:29.210820229 (737h46m29.081903274s) [noise=true , address=145.79.227.6:4291 , piece_id=W6WEK5EZ7VKJOW4ETADS3X5JAC3ADANF3XHIVAE7OVS54TFNWRSQ, seg_pos={0 0}, upload=645558]: Uploading piece...
Jan 5 09:15:29.212536928 (737h46m29.083619963s) [noise=true , address=64.49.13.155:4923 , piece_id=E6X3TYE7TAZJ7Z7JYPEXRPUZJV2PGDZU674ONMXZNRUSYBVOH3KA, seg_pos={0 0}, upload=645558]: Uploading piece...
Jan 5 09:15:29.213571047 (737h46m29.084654082s) [noise=true , address=45.91.116.13:4184 , piece_id=JTDKQZIKXO35GFKCDIMA2LIYDWDSOVY76HTQTT53POWURPZLMCRQ, seg_pos={0 0}, upload=645558]: Uploading piece...
Jan 5 09:15:29.214913814 (737h46m29.08599686s) [noise=true , address=145.79.227.44:4201 , piece_id=R5IPYMC4IG2KHK6G7T6T2SC7TDFHGSJW7XTQ4QEHCOOHJL64A6RA, seg_pos={0 0}, upload=645558]: Uploading piece...
Jan 5 09:15:30.651877289 (737h46m30.522960324s) [upload=645558]: Next returned nil segment
Jan 5 09:15:30.651890403 (737h46m30.522973448s) [upload=645558]: Waiting for error group managing segments...
Jan 5 09:15:31.074605361 (737h46m30.945688406s) [noise=true , address=145.79.227.20:4457 , piece_id=N3Z7W2HEAJYOYWW6XZNKKNTAQ6UWT6XFLM7ZEGMDVRFINJZ53ICA, seg_pos={0 0}, upload=645558]: Done uploading piece. err: <nil>
Jan 5 09:15:31.076109075 (737h46m30.947192121s) [noise=true , address=145.79.227.26:4664 , piece_id=U5YGCVMR6SHNKZG6TFQTUBO4JBMHBE2X2JBIDVYZ2DYEWRODVB6A, seg_pos={0 0}, upload=645558]: Done uploading piece. err: <nil>
Jan 5 09:15:31.076942272 (737h46m30.948025307s) [noise=true , address=145.79.227.34:4936 , piece_id=LJWK6NMSIFHRCWDY5YLLD7UT2BCL45LGQMO7LHIRF72LZGM7KWQQ, seg_pos={0 0}, upload=645558]: Done uploading piece. err: <nil>
Jan 5 09:15:31.143943183 (737h46m31.015026228s) [noise=true , address=145.79.227.6:4291 , piece_id=W6WEK5EZ7VKJOW4ETADS3X5JAC3ADANF3XHIVAE7OVS54TFNWRSQ, seg_pos={0 0}, upload=645558]: Done uploading piece. err: <nil>
Jan 5 09:15:31.163210440 (737h46m31.034293485s) [noise=true , address=145.79.227.49:4364 , piece_id=SPEZRPRC7BOQ2L535RPH5YZ3WESHYSKVXWIPOXYDKCY7QWOSKP6A, seg_pos={0 0}, upload=645558]: Done uploading piece. err: <nil>
Jan 5 09:15:31.588499871 (737h46m31.459582916s) [noise=true , address=45.91.116.13:4184 , piece_id=JTDKQZIKXO35GFKCDIMA2LIYDWDSOVY76HTQTT53POWURPZLMCRQ, seg_pos={0 0}, upload=645558]: Done uploading piece. err: <nil>
Jan 5 09:15:31.742054013 (737h46m31.613137058s) [noise=true , address=77.90.31.17:4802 , piece_id=YKMHTVPOA6MF5IH7DTEMPAJBZWBNUGCDHT5E7BYLQWVDUSBLEM3A, seg_pos={0 0}, upload=645558]: Done uploading piece. err: <nil>
Jan 5 09:15:31.939681488 (737h46m31.810764533s) [noise=true , address=145.79.227.30:4775 , piece_id=CAF6FWLCUNFVJ5FZW7LYXH244VHHTSCWWUP664MRGJFSXPPRKNWQ, seg_pos={0 0}, upload=645558]: Done uploading piece. err: <nil>
Jan 5 09:15:31.973476740 (737h46m31.844559775s) [noise=true , address=145.79.227.44:4201 , piece_id=R5IPYMC4IG2KHK6G7T6T2SC7TDFHGSJW7XTQ4QEHCOOHJL64A6RA, seg_pos={0 0}, upload=645558]: Done uploading piece. err: <nil>
Jan 5 09:15:32.038871842 (737h46m31.909954887s) [noise=true , address=145.79.227.43:4165 , piece_id=X2IW45ERSNU6CRZNDKR3R6ZCR42WFCMMIH46I6RCIGBSIA472VKQ, seg_pos={0 0}, upload=645558]: Done uploading piece. err: <nil>
Jan 5 09:15:32.090674573 (737h46m31.961757618s) [noise=true , address=77.90.31.18:4817 , piece_id=OTSJYC35STDIU7JBTRELHB5UTGNHNSGQIHFJ65FOMRJB4OFV6QWA, seg_pos={0 0}, upload=645558]: Done uploading piece. err: <nil>
Jan 5 09:15:32.220634288 (737h46m32.091717333s) [noise=true , address=77.90.31.11:4560 , piece_id=3ICPCD764NRO4G3DEE2NEQOCOO7RESONGSM5GBAGVFCFFDOPILZA, seg_pos={0 0}, upload=645558]: Done uploading piece. err: <nil>
Jan 5 09:15:32.375653890 (737h46m32.246736925s) [noise=true , address=45.91.116.8:4066 , piece_id=JQ2XRTMAWWVVTK4ZVF4OTGSWPD4YMNDQFCPRAOLIQ57KRTXSLHXQ, seg_pos={0 0}, upload=645558]: Done uploading piece. err: <nil>
Jan 5 09:15:32.388804811 (737h46m32.259887846s) [noise=true , address=145.79.227.40:4052 , piece_id=ORA5FT22S3PUHDZ4Q26YEAYW7VM2TNRZEPMD3H3NIKDDIQSQINUQ, seg_pos={0 0}, upload=645558]: Done uploading piece. err: <nil>
Jan 5 09:15:32.388825311 (737h46m32.259908356s) [seg_pos={0 0}, upload=645558]: Segment reached optimal threshold of 14 pieces.
Jan 5 09:15:32.389001336 (737h46m32.260084371s) [noise=true , address=64.49.13.169:4713 , piece_id=CEEOAYFOSQJOYMTKOBI2X2YGJV2D2VI2EZAWWQQVE7Z27GVUSIBA, seg_pos={0 0}, upload=645558]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17z3AiUfbxBX2eMDnUnqo1bbgojuQxJro7PsEaPjYzQjHQ3D6K): protocol: write tcp 188.166.225.170:38796->64.49.13.169:4713: use of closed network connection; write tcp 188.166.225.170:38796->64.49.13.169:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38796->64.49.13.169:4713: use of closed network connection
Jan 5 09:15:32.389007137 (737h46m32.260090182s) [noise=true , address=64.49.13.169:4713 , piece_id=CEEOAYFOSQJOYMTKOBI2X2YGJV2D2VI2EZAWWQQVE7Z27GVUSIBA, seg_pos={0 0}, upload=645558]: Long tail cancellation due to optimal threshold
Jan 5 09:15:32.389141393 (737h46m32.260224438s) [noise=true , address=45.91.116.4:4954 , piece_id=VOSR3HGKKGKIQOSQAO7OSILTNZIAHYJMUIFB7GAGQTJP3U6BSJPQ, seg_pos={0 0}, upload=645558]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tT1yhzA7AUuwesNf62NHyVxgSvbrmFzchYSmYdqihRjuAgvYa): protocol: expected piece hash; context canceled; EOF
Jan 5 09:15:32.389147284 (737h46m32.260230319s) [noise=true , address=45.91.116.4:4954 , piece_id=VOSR3HGKKGKIQOSQAO7OSILTNZIAHYJMUIFB7GAGQTJP3U6BSJPQ, seg_pos={0 0}, upload=645558]: Long tail cancellation due to optimal threshold
Jan 5 09:15:32.389207699 (737h46m32.260290734s) [noise=true , address=64.49.13.155:4923 , piece_id=E6X3TYE7TAZJ7Z7JYPEXRPUZJV2PGDZU674ONMXZNRUSYBVOH3KA, seg_pos={0 0}, upload=645558]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CJVCqcpsnRfG6zJwpd4Bkm4dKX7DV5AFEesUcRs38VUPuxS3z): protocol: expected piece hash; context canceled; EOF
Jan 5 09:15:32.389216355 (737h46m32.26029939s) [noise=true , address=64.49.13.155:4923 , piece_id=E6X3TYE7TAZJ7Z7JYPEXRPUZJV2PGDZU674ONMXZNRUSYBVOH3KA, seg_pos={0 0}, upload=645558]: Long tail cancellation due to optimal threshold
Jan 5 09:15:32.389247344 (737h46m32.260330379s) [noise=true , address=145.79.227.33:4899 , piece_id=KM6CEND447VWRRLLK57KSQ4MIFGQIK5FHX47PMHBRVHVSVMFCZ6A, seg_pos={0 0}, upload=645558]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b4pZVQRnjcQNn45VTgdsz5fpjYPMAzSXGR5KJ5BP9XHMkRnRw): protocol: expected piece hash; context canceled; EOF
Jan 5 09:15:32.389252634 (737h46m32.260335669s) [noise=true , address=145.79.227.33:4899 , piece_id=KM6CEND447VWRRLLK57KSQ4MIFGQIK5FHX47PMHBRVHVSVMFCZ6A, seg_pos={0 0}, upload=645558]: Long tail cancellation due to optimal threshold
Jan 5 09:15:32.389325463 (737h46m32.260408498s) [seg_pos={0 0}, upload=645558]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jan 5 09:15:32.389386449 (737h46m32.260469494s) [upload=645558]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 5 09:15:32.389390857 (737h46m32.260473892s) [upload=645558]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 5 09:15:32.693933677 (737h46m32.565016712s) [upload=645558]: Done uploading segments...
Jan 5 09:15:32.693952283 (737h46m32.565035318s) [upload=645558]: Upload finished. err: <nil>
Duration: 15.204174016s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [..XX............X..X...X....................................] duration:
25.071104467s trace:
https://jaeger.dev.storj.io/trace/7e43d4cb58eef6b9
FAIL (2026-01-05 09:16:51 UTC)
orbiter warning: context deadline exceeded
Duration: 25.071125397s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [............X.............................................X.] duration:
14.213779262s trace:
https://jaeger.dev.storj.io/trace/35e6dfe601545a69
FAIL (2026-01-05 09:16:51 UTC)
orbiter warning: context deadline exceeded
Duration: 14.213805223s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
13 out of 60 is failed (threshold=3) [..X.X.....X.X...X....................X..X..X.X.X.X..X...X...] duration:
25.79536907s trace:
https://jaeger.dev.storj.io/trace/47e7436da730bafe
FAIL (2026-01-05 09:16:39 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: CSORWG66JCKA2B2ZUVUL2LWPBYEGAUWEMKARA4NRRM2GOZTQCSQQ) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: CXHE5GS2X4HNLNIIESI4YZH3NYPURSMASXS6H7CJRH32O2KSGSKQ) context deadline exceeded
======LOGS======
Jan 5 09:14:09.583977733 (737h45m9.455060778s) [upload=645537]: Starting upload
Jan 5 09:14:09.584034522 (737h45m9.455117567s) [upload=645537]: Done starting upload
Jan 5 09:14:09.584158388 (737h45m9.455241423s) [upload=645537]: Uploading segments...
Jan 5 09:14:09.584393254 (737h45m9.455476299s) [upload=645537]: Got next segment. Inline: false
Jan 5 09:14:09.584457306 (737h45m9.455540341s) [upload=645537]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 5 09:14:09.584467936 (737h45m9.455550971s) [upload=645537]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 5 09:14:09.828359528 (737h45m9.699442583s) [seg_pos={0 0}, upload=645537]: Begin upload segment...
Jan 5 09:14:09.828404934 (737h45m9.699487969s) [seg_pos={0 0}, upload=645537]: Done begin upload segment.
Jan 5 09:14:09.828430353 (737h45m9.699513388s) [noise=true , address=45.91.116.13:4155 , piece_id=VH43XLDBAT3K4KSMFKZERPVAR7CN3OTBXCEAPOGWOUFACEF5WMFA, seg_pos={0 0}, upload=645537]: Uploading piece...
Jan 5 09:14:09.828453486 (737h45m9.699536521s) [noise=true , address=145.79.227.38:4977 , piece_id=37V3B7XFCA56HRSZFFSKDYXOYN7GQ4UHRCVNHJIIJMY3IHVZVNUA, seg_pos={0 0}, upload=645537]: Uploading piece...
Jan 5 09:14:09.828845433 (737h45m9.699928468s) [noise=true , address=45.91.116.10:4081 , piece_id=QIFSZ6EURMEYLMSDMM5S2HOXHOWIRAXXMV5KIORC5BA5M6QP62XA, seg_pos={0 0}, upload=645537]: Uploading piece...
Jan 5 09:14:09.829185149 (737h45m9.700268194s) [noise=true , address=77.90.31.19:4842 , piece_id=CSORWG66JCKA2B2ZUVUL2LWPBYEGAUWEMKARA4NRRM2GOZTQCSQQ, seg_pos={0 0}, upload=645537]: Uploading piece...
Jan 5 09:14:09.829542641 (737h45m9.700625686s) [noise=true , address=45.91.116.6:4975 , piece_id=CXHE5GS2X4HNLNIIESI4YZH3NYPURSMASXS6H7CJRH32O2KSGSKQ, seg_pos={0 0}, upload=645537]: Uploading piece...
Jan 5 09:14:09.830011844 (737h45m9.701094879s) [noise=true , address=145.79.227.42:4122 , piece_id=KPAAJQLAY7G5T33NECXUKOUDKDDKWAOCCN3VDLGV6KRDWL7N77JA, seg_pos={0 0}, upload=645537]: Uploading piece...
Jan 5 09:14:09.830340450 (737h45m9.701423485s) [noise=true , address=145.79.227.35:4483 , piece_id=QU57OUARYAB7DXVHAEP57LPKVENYBK6FXULSQWHMU33R7ATBCQ6Q, seg_pos={0 0}, upload=645537]: Uploading piece...
Jan 5 09:14:09.830348675 (737h45m9.70143172s) [noise=true , address=145.79.227.13:4802 , piece_id=TIOBQBWIL4ECWPBNZIH2ZGUFKKPKRNS2UBHN2LORUI24RGWJZX6Q, seg_pos={0 0}, upload=645537]: Uploading piece...
Jan 5 09:14:09.830679314 (737h45m9.70176236s) [noise=true , address=145.79.227.52:4446 , piece_id=XTF4DJPTH4FWMULGOWBJADKNMFPUQDW5TBYILAB4YO5OHJQBLAMQ, seg_pos={0 0}, upload=645537]: Uploading piece...
Jan 5 09:14:09.830689735 (737h45m9.70177277s) [noise=true , address=145.79.227.22:4510 , piece_id=63L32REKAWJ466RU3FT5ME3RC5HACFTWC2VQVBRQMYD2GN4UY3OQ, seg_pos={0 0}, upload=645537]: Uploading piece...
Jan 5 09:14:09.830974738 (737h45m9.702057773s) [noise=true , address=145.79.227.16:4912 , piece_id=KZJSPJJMYM44WJVYW423WJJWFOXGSODGBLT3RD3NGRDLRFM4LBXQ, seg_pos={0 0}, upload=645537]: Uploading piece...
Jan 5 09:14:09.831255151 (737h45m9.702338186s) [noise=true , address=145.79.227.24:4581 , piece_id=KKGYLJLXJMSDHFDLAB4XPAY22NBIW4JVM2YZAKTBXO2RZ7RBHVSA, seg_pos={0 0}, upload=645537]: Uploading piece...
Jan 5 09:14:09.831607352 (737h45m9.702690387s) [noise=true , address=145.79.227.6:4258 , piece_id=PWLHHPPAVEGW3P5C335VL6CJ4K5LU2UIIFZMLMNHLMIP52NZBYAA, seg_pos={0 0}, upload=645537]: Uploading piece...
Jan 5 09:14:09.831983268 (737h45m9.703066303s) [noise=true , address=145.79.227.49:4337 , piece_id=ZUHCA2OBCBBFQKTWPZDHQSM5A3KMVJDMSDZCVQMS7C6LACZBHKKA, seg_pos={0 0}, upload=645537]: Uploading piece...
Jan 5 09:14:09.832681518 (737h45m9.703764553s) [noise=true , address=145.79.227.11:4732 , piece_id=422CH2PAE7PAIZNHDOHTSKRIE4O5TYKUX2THOCYOVR4GGQL5LHVA, seg_pos={0 0}, upload=645537]: Uploading piece...
Jan 5 09:14:09.834277647 (737h45m9.705360692s) [noise=true , address=77.90.31.15:4697 , piece_id=KVG2MEKUDL3DPN7VFC4ITG4HOROPGPZA5EL5IAADAU5DANWLD7TQ, seg_pos={0 0}, upload=645537]: Uploading piece...
Jan 5 09:14:11.528352880 (737h45m11.399435925s) [upload=645537]: Next returned nil segment
Jan 5 09:14:11.528375925 (737h45m11.39945896s) [upload=645537]: Waiting for error group managing segments...
Jan 5 09:14:11.807752182 (737h45m11.678835217s) [noise=true , address=145.79.227.11:4732 , piece_id=422CH2PAE7PAIZNHDOHTSKRIE4O5TYKUX2THOCYOVR4GGQL5LHVA, seg_pos={0 0}, upload=645537]: Done uploading piece. err: <nil>
Jan 5 09:14:11.810685898 (737h45m11.681768933s) [noise=true , address=145.79.227.42:4122 , piece_id=KPAAJQLAY7G5T33NECXUKOUDKDDKWAOCCN3VDLGV6KRDWL7N77JA, seg_pos={0 0}, upload=645537]: Done uploading piece. err: <nil>
Jan 5 09:14:11.811621249 (737h45m11.682704304s) [noise=true , address=145.79.227.52:4446 , piece_id=XTF4DJPTH4FWMULGOWBJADKNMFPUQDW5TBYILAB4YO5OHJQBLAMQ, seg_pos={0 0}, upload=645537]: Done uploading piece. err: <nil>
Jan 5 09:14:11.816776816 (737h45m11.687859861s) [noise=true , address=145.79.227.24:4581 , piece_id=KKGYLJLXJMSDHFDLAB4XPAY22NBIW4JVM2YZAKTBXO2RZ7RBHVSA, seg_pos={0 0}, upload=645537]: Done uploading piece. err: <nil>
Jan 5 09:14:11.824300482 (737h45m11.695383517s) [noise=true , address=145.79.227.16:4912 , piece_id=KZJSPJJMYM44WJVYW423WJJWFOXGSODGBLT3RD3NGRDLRFM4LBXQ, seg_pos={0 0}, upload=645537]: Done uploading piece. err: <nil>
Jan 5 09:14:11.828392294 (737h45m11.699475329s) [noise=true , address=145.79.227.22:4510 , piece_id=63L32REKAWJ466RU3FT5ME3RC5HACFTWC2VQVBRQMYD2GN4UY3OQ, seg_pos={0 0}, upload=645537]: Done uploading piece. err: <nil>
Jan 5 09:14:11.830672355 (737h45m11.70175539s) [noise=true , address=145.79.227.38:4977 , piece_id=37V3B7XFCA56HRSZFFSKDYXOYN7GQ4UHRCVNHJIIJMY3IHVZVNUA, seg_pos={0 0}, upload=645537]: Done uploading piece. err: <nil>
Jan 5 09:14:12.047202525 (737h45m11.91828557s) [noise=true , address=145.79.227.13:4802 , piece_id=TIOBQBWIL4ECWPBNZIH2ZGUFKKPKRNS2UBHN2LORUI24RGWJZX6Q, seg_pos={0 0}, upload=645537]: Done uploading piece. err: <nil>
Jan 5 09:14:12.242929059 (737h45m12.114012094s) [noise=true , address=145.79.227.6:4258 , piece_id=PWLHHPPAVEGW3P5C335VL6CJ4K5LU2UIIFZMLMNHLMIP52NZBYAA, seg_pos={0 0}, upload=645537]: Done uploading piece. err: <nil>
Jan 5 09:14:13.125455593 (737h45m12.996538638s) [noise=true , address=45.91.116.6:4975 , piece_id=CXHE5GS2X4HNLNIIESI4YZH3NYPURSMASXS6H7CJRH32O2KSGSKQ, seg_pos={0 0}, upload=645537]: Done uploading piece. err: <nil>
Jan 5 09:14:13.222042781 (737h45m13.093125826s) [noise=true , address=77.90.31.19:4842 , piece_id=CSORWG66JCKA2B2ZUVUL2LWPBYEGAUWEMKARA4NRRM2GOZTQCSQQ, seg_pos={0 0}, upload=645537]: Done uploading piece. err: <nil>
Jan 5 09:14:13.646169338 (737h45m13.517252393s) [noise=true , address=145.79.227.49:4337 , piece_id=ZUHCA2OBCBBFQKTWPZDHQSM5A3KMVJDMSDZCVQMS7C6LACZBHKKA, seg_pos={0 0}, upload=645537]: Done uploading piece. err: <nil>
Jan 5 09:14:13.680855606 (737h45m13.551938651s) [noise=true , address=145.79.227.35:4483 , piece_id=QU57OUARYAB7DXVHAEP57LPKVENYBK6FXULSQWHMU33R7ATBCQ6Q, seg_pos={0 0}, upload=645537]: Done uploading piece. err: <nil>
Jan 5 09:14:13.680882036 (737h45m13.551965081s) [seg_pos={0 0}, upload=645537]: Segment reached optimal threshold of 13 pieces.
Jan 5 09:14:13.681166217 (737h45m13.552249262s) [noise=true , address=45.91.116.10:4081 , piece_id=QIFSZ6EURMEYLMSDMM5S2HOXHOWIRAXXMV5KIORC5BA5M6QP62XA, seg_pos={0 0}, upload=645537]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF
Jan 5 09:14:13.681194141 (737h45m13.552277186s) [noise=true , address=45.91.116.10:4081 , piece_id=QIFSZ6EURMEYLMSDMM5S2HOXHOWIRAXXMV5KIORC5BA5M6QP62XA, seg_pos={0 0}, upload=645537]: Long tail cancellation due to optimal threshold
Jan 5 09:14:13.681312316 (737h45m13.552395351s) [noise=true , address=45.91.116.13:4155 , piece_id=VH43XLDBAT3K4KSMFKZERPVAR7CN3OTBXCEAPOGWOUFACEF5WMFA, seg_pos={0 0}, upload=645537]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF
Jan 5 09:14:13.681328416 (737h45m13.552411452s) [noise=true , address=45.91.116.13:4155 , piece_id=VH43XLDBAT3K4KSMFKZERPVAR7CN3OTBXCEAPOGWOUFACEF5WMFA, seg_pos={0 0}, upload=645537]: Long tail cancellation due to optimal threshold
Jan 5 09:14:13.681684615 (737h45m13.55276766s) [noise=true , address=77.90.31.15:4697 , piece_id=KVG2MEKUDL3DPN7VFC4ITG4HOROPGPZA5EL5IAADAU5DANWLD7TQ, seg_pos={0 0}, upload=645537]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: context canceled; context canceled
Jan 5 09:14:13.681696959 (737h45m13.552780004s) [noise=true , address=77.90.31.15:4697 , piece_id=KVG2MEKUDL3DPN7VFC4ITG4HOROPGPZA5EL5IAADAU5DANWLD7TQ, seg_pos={0 0}, upload=645537]: Long tail cancellation due to optimal threshold
Jan 5 09:14:13.681722557 (737h45m13.552805602s) [seg_pos={0 0}, upload=645537]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 5 09:14:13.681882151 (737h45m13.552965196s) [upload=645537]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 5 09:14:13.681890777 (737h45m13.552973832s) [upload=645537]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 5 09:14:13.965823732 (737h45m13.836906767s) [upload=645537]: Done uploading segments...
Jan 5 09:14:13.965844372 (737h45m13.836927407s) [upload=645537]: Upload finished. err: <nil>
Duration: 25.795397494s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [.............X.............................................X] duration:
1m0.001288285s trace:
https://jaeger.dev.storj.io/trace/35e6dfe601545a69
FAIL (2026-01-05 09:16:34 UTC)
orbiter warning: context deadline exceeded
Duration: 1m0.001313623s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
42 out of 60 is failed (threshold=6) [X..X.XXXXXXXX.XX..XXX.X.XX.XXX.XX.XXXXXX..XXXXXX...XXXX..XXX] duration:
5.001356567s trace:
https://jaeger.dev.storj.io/trace/2ee36bb5d98859ae
FAIL (2026-01-05 09:16:29 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001379392s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
27 out of 60 is failed (threshold=3) [..X.XX.XXXXXX.XXX...X....XX.X.X....XX....X..XX.XX......X.X.X] duration:
1m0.003313282s trace:
https://jaeger.dev.storj.io/trace/1d2dd2d52810438a
FAIL (2026-01-05 09:16:28 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
noise=true , address=145.79.227.33:4899 , piece_id=KM6CEND447VWRRLLK57KSQ4MIFGQIK5FHX47PMHBRVHVSVMFCZ6A, seg_pos={0 0}, upload=645558]: Uploading piece...
Jan 5 09:15:29.200269970 (737h46m29.071353015s) [noise=true , address=77.90.31.18:4817 , piece_id=OTSJYC35STDIU7JBTRELHB5UTGNHNSGQIHFJ65FOMRJB4OFV6QWA, seg_pos={0 0}, upload=645558]: Uploading piece...
Jan 5 09:15:29.200601691 (737h46m29.071684726s) [noise=true , address=77.90.31.11:4560 , piece_id=3ICPCD764NRO4G3DEE2NEQOCOO7RESONGSM5GBAGVFCFFDOPILZA, seg_pos={0 0}, upload=645558]: Uploading piece...
Jan 5 09:15:29.200987557 (737h46m29.072070622s) [noise=true , address=45.91.116.8:4066 , piece_id=JQ2XRTMAWWVVTK4ZVF4OTGSWPD4YMNDQFCPRAOLIQ57KRTXSLHXQ, seg_pos={0 0}, upload=645558]: Uploading piece...
Jan 5 09:15:29.201343163 (737h46m29.072426209s) [noise=true , address=145.79.227.49:4364 , piece_id=SPEZRPRC7BOQ2L535RPH5YZ3WESHYSKVXWIPOXYDKCY7QWOSKP6A, seg_pos={0 0}, upload=645558]: Uploading piece...
Jan 5 09:15:29.201736713 (737h46m29.072819758s) [noise=true , address=145.79.227.20:4457 , piece_id=N3Z7W2HEAJYOYWW6XZNKKNTAQ6UWT6XFLM7ZEGMDVRFINJZ53ICA, seg_pos={0 0}, upload=645558]: Uploading piece...
Jan 5 09:15:29.202243488 (737h46m29.073326523s) [noise=true , address=64.49.13.169:4713 , piece_id=CEEOAYFOSQJOYMTKOBI2X2YGJV2D2VI2EZAWWQQVE7Z27GVUSIBA, seg_pos={0 0}, upload=645558]: Uploading piece...
Jan 5 09:15:29.202665501 (737h46m29.073748546s) [noise=true , address=77.90.31.17:4802 , piece_id=YKMHTVPOA6MF5IH7DTEMPAJBZWBNUGCDHT5E7BYLQWVDUSBLEM3A, seg_pos={0 0}, upload=645558]: Uploading piece...
Jan 5 09:15:29.204332386 (737h46m29.075415421s) [noise=true , address=145.79.227.40:4052 , piece_id=ORA5FT22S3PUHDZ4Q26YEAYW7VM2TNRZEPMD3H3NIKDDIQSQINUQ, seg_pos={0 0}, upload=645558]: Uploading piece...
Jan 5 09:15:29.206264855 (737h46m29.0773479s) [noise=true , address=145.79.227.30:4775 , piece_id=CAF6FWLCUNFVJ5FZW7LYXH244VHHTSCWWUP664MRGJFSXPPRKNWQ, seg_pos={0 0}, upload=645558]: Uploading piece...
Jan 5 09:15:29.207973889 (737h46m29.079056934s) [noise=true , address=145.79.227.26:4664 , piece_id=U5YGCVMR6SHNKZG6TFQTUBO4JBMHBE2X2JBIDVYZ2DYEWRODVB6A, seg_pos={0 0}, upload=645558]: Uploading piece...
Jan 5 09:15:29.209273434 (737h46m29.080356469s) [noise=true , address=145.79.227.43:4165 , piece_id=X2IW45ERSNU6CRZNDKR3R6ZCR42WFCMMIH46I6RCIGBSIA472VKQ, seg_pos={0 0}, upload=645558]: Uploading piece...
Jan 5 09:15:29.210820229 (737h46m29.081903274s) [noise=true , address=145.79.227.6:4291 , piece_id=W6WEK5EZ7VKJOW4ETADS3X5JAC3ADANF3XHIVAE7OVS54TFNWRSQ, seg_pos={0 0}, upload=645558]: Uploading piece...
Jan 5 09:15:29.212536928 (737h46m29.083619963s) [noise=true , address=64.49.13.155:4923 , piece_id=E6X3TYE7TAZJ7Z7JYPEXRPUZJV2PGDZU674ONMXZNRUSYBVOH3KA, seg_pos={0 0}, upload=645558]: Uploading piece...
Jan 5 09:15:29.213571047 (737h46m29.084654082s) [noise=true , address=45.91.116.13:4184 , piece_id=JTDKQZIKXO35GFKCDIMA2LIYDWDSOVY76HTQTT53POWURPZLMCRQ, seg_pos={0 0}, upload=645558]: Uploading piece...
Jan 5 09:15:29.214913814 (737h46m29.08599686s) [noise=true , address=145.79.227.44:4201 , piece_id=R5IPYMC4IG2KHK6G7T6T2SC7TDFHGSJW7XTQ4QEHCOOHJL64A6RA, seg_pos={0 0}, upload=645558]: Uploading piece...
Jan 5 09:15:30.651877289 (737h46m30.522960324s) [upload=645558]: Next returned nil segment
Jan 5 09:15:30.651890403 (737h46m30.522973448s) [upload=645558]: Waiting for error group managing segments...
Jan 5 09:15:31.074605361 (737h46m30.945688406s) [noise=true , address=145.79.227.20:4457 , piece_id=N3Z7W2HEAJYOYWW6XZNKKNTAQ6UWT6XFLM7ZEGMDVRFINJZ53ICA, seg_pos={0 0}, upload=645558]: Done uploading piece. err: <nil>
Jan 5 09:15:31.076109075 (737h46m30.947192121s) [noise=true , address=145.79.227.26:4664 , piece_id=U5YGCVMR6SHNKZG6TFQTUBO4JBMHBE2X2JBIDVYZ2DYEWRODVB6A, seg_pos={0 0}, upload=645558]: Done uploading piece. err: <nil>
Jan 5 09:15:31.076942272 (737h46m30.948025307s) [noise=true , address=145.79.227.34:4936 , piece_id=LJWK6NMSIFHRCWDY5YLLD7UT2BCL45LGQMO7LHIRF72LZGM7KWQQ, seg_pos={0 0}, upload=645558]: Done uploading piece. err: <nil>
Jan 5 09:15:31.143943183 (737h46m31.015026228s) [noise=true , address=145.79.227.6:4291 , piece_id=W6WEK5EZ7VKJOW4ETADS3X5JAC3ADANF3XHIVAE7OVS54TFNWRSQ, seg_pos={0 0}, upload=645558]: Done uploading piece. err: <nil>
Jan 5 09:15:31.163210440 (737h46m31.034293485s) [noise=true , address=145.79.227.49:4364 , piece_id=SPEZRPRC7BOQ2L535RPH5YZ3WESHYSKVXWIPOXYDKCY7QWOSKP6A, seg_pos={0 0}, upload=645558]: Done uploading piece. err: <nil>
Jan 5 09:15:31.588499871 (737h46m31.459582916s) [noise=true , address=45.91.116.13:4184 , piece_id=JTDKQZIKXO35GFKCDIMA2LIYDWDSOVY76HTQTT53POWURPZLMCRQ, seg_pos={0 0}, upload=645558]: Done uploading piece. err: <nil>
Jan 5 09:15:31.742054013 (737h46m31.613137058s) [noise=true , address=77.90.31.17:4802 , piece_id=YKMHTVPOA6MF5IH7DTEMPAJBZWBNUGCDHT5E7BYLQWVDUSBLEM3A, seg_pos={0 0}, upload=645558]: Done uploading piece. err: <nil>
Jan 5 09:15:31.939681488 (737h46m31.810764533s) [noise=true , address=145.79.227.30:4775 , piece_id=CAF6FWLCUNFVJ5FZW7LYXH244VHHTSCWWUP664MRGJFSXPPRKNWQ, seg_pos={0 0}, upload=645558]: Done uploading piece. err: <nil>
Jan 5 09:15:31.973476740 (737h46m31.844559775s) [noise=true , address=145.79.227.44:4201 , piece_id=R5IPYMC4IG2KHK6G7T6T2SC7TDFHGSJW7XTQ4QEHCOOHJL64A6RA, seg_pos={0 0}, upload=645558]: Done uploading piece. err: <nil>
Jan 5 09:15:32.038871842 (737h46m31.909954887s) [noise=true , address=145.79.227.43:4165 , piece_id=X2IW45ERSNU6CRZNDKR3R6ZCR42WFCMMIH46I6RCIGBSIA472VKQ, seg_pos={0 0}, upload=645558]: Done uploading piece. err: <nil>
Jan 5 09:15:32.090674573 (737h46m31.961757618s) [noise=true , address=77.90.31.18:4817 , piece_id=OTSJYC35STDIU7JBTRELHB5UTGNHNSGQIHFJ65FOMRJB4OFV6QWA, seg_pos={0 0}, upload=645558]: Done uploading piece. err: <nil>
Jan 5 09:15:32.220634288 (737h46m32.091717333s) [noise=true , address=77.90.31.11:4560 , piece_id=3ICPCD764NRO4G3DEE2NEQOCOO7RESONGSM5GBAGVFCFFDOPILZA, seg_pos={0 0}, upload=645558]: Done uploading piece. err: <nil>
Jan 5 09:15:32.375653890 (737h46m32.246736925s) [noise=true , address=45.91.116.8:4066 , piece_id=JQ2XRTMAWWVVTK4ZVF4OTGSWPD4YMNDQFCPRAOLIQ57KRTXSLHXQ, seg_pos={0 0}, upload=645558]: Done uploading piece. err: <nil>
Jan 5 09:15:32.388804811 (737h46m32.259887846s) [noise=true , address=145.79.227.40:4052 , piece_id=ORA5FT22S3PUHDZ4Q26YEAYW7VM2TNRZEPMD3H3NIKDDIQSQINUQ, seg_pos={0 0}, upload=645558]: Done uploading piece. err: <nil>
Jan 5 09:15:32.388825311 (737h46m32.259908356s) [seg_pos={0 0}, upload=645558]: Segment reached optimal threshold of 14 pieces.
Jan 5 09:15:32.389001336 (737h46m32.260084371s) [noise=true , address=64.49.13.169:4713 , piece_id=CEEOAYFOSQJOYMTKOBI2X2YGJV2D2VI2EZAWWQQVE7Z27GVUSIBA, seg_pos={0 0}, upload=645558]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17z3AiUfbxBX2eMDnUnqo1bbgojuQxJro7PsEaPjYzQjHQ3D6K): protocol: write tcp 188.166.225.170:38796->64.49.13.169:4713: use of closed network connection; write tcp 188.166.225.170:38796->64.49.13.169:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38796->64.49.13.169:4713: use of closed network connection
Jan 5 09:15:32.389007137 (737h46m32.260090182s) [noise=true , address=64.49.13.169:4713 , piece_id=CEEOAYFOSQJOYMTKOBI2X2YGJV2D2VI2EZAWWQQVE7Z27GVUSIBA, seg_pos={0 0}, upload=645558]: Long tail cancellation due to optimal threshold
Jan 5 09:15:32.389141393 (737h46m32.260224438s) [noise=true , address=45.91.116.4:4954 , piece_id=VOSR3HGKKGKIQOSQAO7OSILTNZIAHYJMUIFB7GAGQTJP3U6BSJPQ, seg_pos={0 0}, upload=645558]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tT1yhzA7AUuwesNf62NHyVxgSvbrmFzchYSmYdqihRjuAgvYa): protocol: expected piece hash; context canceled; EOF
Jan 5 09:15:32.389147284 (737h46m32.260230319s) [noise=true , address=45.91.116.4:4954 , piece_id=VOSR3HGKKGKIQOSQAO7OSILTNZIAHYJMUIFB7GAGQTJP3U6BSJPQ, seg_pos={0 0}, upload=645558]: Long tail cancellation due to optimal threshold
Jan 5 09:15:32.389207699 (737h46m32.260290734s) [noise=true , address=64.49.13.155:4923 , piece_id=E6X3TYE7TAZJ7Z7JYPEXRPUZJV2PGDZU674ONMXZNRUSYBVOH3KA, seg_pos={0 0}, upload=645558]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CJVCqcpsnRfG6zJwpd4Bkm4dKX7DV5AFEesUcRs38VUPuxS3z): protocol: expected piece hash; context canceled; EOF
Jan 5 09:15:32.389216355 (737h46m32.26029939s) [noise=true , address=64.49.13.155:4923 , piece_id=E6X3TYE7TAZJ7Z7JYPEXRPUZJV2PGDZU674ONMXZNRUSYBVOH3KA, seg_pos={0 0}, upload=645558]: Long tail cancellation due to optimal threshold
Jan 5 09:15:32.389247344 (737h46m32.260330379s) [noise=true , address=145.79.227.33:4899 , piece_id=KM6CEND447VWRRLLK57KSQ4MIFGQIK5FHX47PMHBRVHVSVMFCZ6A, seg_pos={0 0}, upload=645558]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b4pZVQRnjcQNn45VTgdsz5fpjYPMAzSXGR5KJ5BP9XHMkRnRw): protocol: expected piece hash; context canceled; EOF
Jan 5 09:15:32.389252634 (737h46m32.260335669s) [noise=true , address=145.79.227.33:4899 , piece_id=KM6CEND447VWRRLLK57KSQ4MIFGQIK5FHX47PMHBRVHVSVMFCZ6A, seg_pos={0 0}, upload=645558]: Long tail cancellation due to optimal threshold
Jan 5 09:15:32.389325463 (737h46m32.260408498s) [seg_pos={0 0}, upload=645558]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jan 5 09:15:32.389386449 (737h46m32.260469494s) [upload=645558]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 5 09:15:32.389390857 (737h46m32.260473892s) [upload=645558]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 5 09:15:32.693933677 (737h46m32.565016712s) [upload=645558]: Done uploading segments...
Jan 5 09:15:32.693952283 (737h46m32.565035318s) [upload=645558]: Upload finished. err: <nil>
Duration: 1m0.003335926s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [...XX............X..X...X...................................] duration:
20.416249937s trace:
https://jaeger.dev.storj.io/trace/7e43d4cb58eef6b9
FAIL (2026-01-05 09:16:16 UTC)
orbiter warning: context deadline exceeded
Duration: 20.41634551s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
13 out of 60 is failed (threshold=3) [...X.X.....X.X...X....................X..X..X.X.X.X..X...X..] duration:
6.21500783s trace:
https://jaeger.dev.storj.io/trace/47e7436da730bafe
FAIL (2026-01-05 09:15:50 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: CSORWG66JCKA2B2ZUVUL2LWPBYEGAUWEMKARA4NRRM2GOZTQCSQQ) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: CXHE5GS2X4HNLNIIESI4YZH3NYPURSMASXS6H7CJRH32O2KSGSKQ) context deadline exceeded
======LOGS======
Jan 5 09:14:09.583977733 (737h45m9.455060778s) [upload=645537]: Starting upload
Jan 5 09:14:09.584034522 (737h45m9.455117567s) [upload=645537]: Done starting upload
Jan 5 09:14:09.584158388 (737h45m9.455241423s) [upload=645537]: Uploading segments...
Jan 5 09:14:09.584393254 (737h45m9.455476299s) [upload=645537]: Got next segment. Inline: false
Jan 5 09:14:09.584457306 (737h45m9.455540341s) [upload=645537]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 5 09:14:09.584467936 (737h45m9.455550971s) [upload=645537]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 5 09:14:09.828359528 (737h45m9.699442583s) [seg_pos={0 0}, upload=645537]: Begin upload segment...
Jan 5 09:14:09.828404934 (737h45m9.699487969s) [seg_pos={0 0}, upload=645537]: Done begin upload segment.
Jan 5 09:14:09.828430353 (737h45m9.699513388s) [noise=true , address=45.91.116.13:4155 , piece_id=VH43XLDBAT3K4KSMFKZERPVAR7CN3OTBXCEAPOGWOUFACEF5WMFA, seg_pos={0 0}, upload=645537]: Uploading piece...
Jan 5 09:14:09.828453486 (737h45m9.699536521s) [noise=true , address=145.79.227.38:4977 , piece_id=37V3B7XFCA56HRSZFFSKDYXOYN7GQ4UHRCVNHJIIJMY3IHVZVNUA, seg_pos={0 0}, upload=645537]: Uploading piece...
Jan 5 09:14:09.828845433 (737h45m9.699928468s) [noise=true , address=45.91.116.10:4081 , piece_id=QIFSZ6EURMEYLMSDMM5S2HOXHOWIRAXXMV5KIORC5BA5M6QP62XA, seg_pos={0 0}, upload=645537]: Uploading piece...
Jan 5 09:14:09.829185149 (737h45m9.700268194s) [noise=true , address=77.90.31.19:4842 , piece_id=CSORWG66JCKA2B2ZUVUL2LWPBYEGAUWEMKARA4NRRM2GOZTQCSQQ, seg_pos={0 0}, upload=645537]: Uploading piece...
Jan 5 09:14:09.829542641 (737h45m9.700625686s) [noise=true , address=45.91.116.6:4975 , piece_id=CXHE5GS2X4HNLNIIESI4YZH3NYPURSMASXS6H7CJRH32O2KSGSKQ, seg_pos={0 0}, upload=645537]: Uploading piece...
Jan 5 09:14:09.830011844 (737h45m9.701094879s) [noise=true , address=145.79.227.42:4122 , piece_id=KPAAJQLAY7G5T33NECXUKOUDKDDKWAOCCN3VDLGV6KRDWL7N77JA, seg_pos={0 0}, upload=645537]: Uploading piece...
Jan 5 09:14:09.830340450 (737h45m9.701423485s) [noise=true , address=145.79.227.35:4483 , piece_id=QU57OUARYAB7DXVHAEP57LPKVENYBK6FXULSQWHMU33R7ATBCQ6Q, seg_pos={0 0}, upload=645537]: Uploading piece...
Jan 5 09:14:09.830348675 (737h45m9.70143172s) [noise=true , address=145.79.227.13:4802 , piece_id=TIOBQBWIL4ECWPBNZIH2ZGUFKKPKRNS2UBHN2LORUI24RGWJZX6Q, seg_pos={0 0}, upload=645537]: Uploading piece...
Jan 5 09:14:09.830679314 (737h45m9.70176236s) [noise=true , address=145.79.227.52:4446 , piece_id=XTF4DJPTH4FWMULGOWBJADKNMFPUQDW5TBYILAB4YO5OHJQBLAMQ, seg_pos={0 0}, upload=645537]: Uploading piece...
Jan 5 09:14:09.830689735 (737h45m9.70177277s) [noise=true , address=145.79.227.22:4510 , piece_id=63L32REKAWJ466RU3FT5ME3RC5HACFTWC2VQVBRQMYD2GN4UY3OQ, seg_pos={0 0}, upload=645537]: Uploading piece...
Jan 5 09:14:09.830974738 (737h45m9.702057773s) [noise=true , address=145.79.227.16:4912 , piece_id=KZJSPJJMYM44WJVYW423WJJWFOXGSODGBLT3RD3NGRDLRFM4LBXQ, seg_pos={0 0}, upload=645537]: Uploading piece...
Jan 5 09:14:09.831255151 (737h45m9.702338186s) [noise=true , address=145.79.227.24:4581 , piece_id=KKGYLJLXJMSDHFDLAB4XPAY22NBIW4JVM2YZAKTBXO2RZ7RBHVSA, seg_pos={0 0}, upload=645537]: Uploading piece...
Jan 5 09:14:09.831607352 (737h45m9.702690387s) [noise=true , address=145.79.227.6:4258 , piece_id=PWLHHPPAVEGW3P5C335VL6CJ4K5LU2UIIFZMLMNHLMIP52NZBYAA, seg_pos={0 0}, upload=645537]: Uploading piece...
Jan 5 09:14:09.831983268 (737h45m9.703066303s) [noise=true , address=145.79.227.49:4337 , piece_id=ZUHCA2OBCBBFQKTWPZDHQSM5A3KMVJDMSDZCVQMS7C6LACZBHKKA, seg_pos={0 0}, upload=645537]: Uploading piece...
Jan 5 09:14:09.832681518 (737h45m9.703764553s) [noise=true , address=145.79.227.11:4732 , piece_id=422CH2PAE7PAIZNHDOHTSKRIE4O5TYKUX2THOCYOVR4GGQL5LHVA, seg_pos={0 0}, upload=645537]: Uploading piece...
Jan 5 09:14:09.834277647 (737h45m9.705360692s) [noise=true , address=77.90.31.15:4697 , piece_id=KVG2MEKUDL3DPN7VFC4ITG4HOROPGPZA5EL5IAADAU5DANWLD7TQ, seg_pos={0 0}, upload=645537]: Uploading piece...
Jan 5 09:14:11.528352880 (737h45m11.399435925s) [upload=645537]: Next returned nil segment
Jan 5 09:14:11.528375925 (737h45m11.39945896s) [upload=645537]: Waiting for error group managing segments...
Jan 5 09:14:11.807752182 (737h45m11.678835217s) [noise=true , address=145.79.227.11:4732 , piece_id=422CH2PAE7PAIZNHDOHTSKRIE4O5TYKUX2THOCYOVR4GGQL5LHVA, seg_pos={0 0}, upload=645537]: Done uploading piece. err: <nil>
Jan 5 09:14:11.810685898 (737h45m11.681768933s) [noise=true , address=145.79.227.42:4122 , piece_id=KPAAJQLAY7G5T33NECXUKOUDKDDKWAOCCN3VDLGV6KRDWL7N77JA, seg_pos={0 0}, upload=645537]: Done uploading piece. err: <nil>
Jan 5 09:14:11.811621249 (737h45m11.682704304s) [noise=true , address=145.79.227.52:4446 , piece_id=XTF4DJPTH4FWMULGOWBJADKNMFPUQDW5TBYILAB4YO5OHJQBLAMQ, seg_pos={0 0}, upload=645537]: Done uploading piece. err: <nil>
Jan 5 09:14:11.816776816 (737h45m11.687859861s) [noise=true , address=145.79.227.24:4581 , piece_id=KKGYLJLXJMSDHFDLAB4XPAY22NBIW4JVM2YZAKTBXO2RZ7RBHVSA, seg_pos={0 0}, upload=645537]: Done uploading piece. err: <nil>
Jan 5 09:14:11.824300482 (737h45m11.695383517s) [noise=true , address=145.79.227.16:4912 , piece_id=KZJSPJJMYM44WJVYW423WJJWFOXGSODGBLT3RD3NGRDLRFM4LBXQ, seg_pos={0 0}, upload=645537]: Done uploading piece. err: <nil>
Jan 5 09:14:11.828392294 (737h45m11.699475329s) [noise=true , address=145.79.227.22:4510 , piece_id=63L32REKAWJ466RU3FT5ME3RC5HACFTWC2VQVBRQMYD2GN4UY3OQ, seg_pos={0 0}, upload=645537]: Done uploading piece. err: <nil>
Jan 5 09:14:11.830672355 (737h45m11.70175539s) [noise=true , address=145.79.227.38:4977 , piece_id=37V3B7XFCA56HRSZFFSKDYXOYN7GQ4UHRCVNHJIIJMY3IHVZVNUA, seg_pos={0 0}, upload=645537]: Done uploading piece. err: <nil>
Jan 5 09:14:12.047202525 (737h45m11.91828557s) [noise=true , address=145.79.227.13:4802 , piece_id=TIOBQBWIL4ECWPBNZIH2ZGUFKKPKRNS2UBHN2LORUI24RGWJZX6Q, seg_pos={0 0}, upload=645537]: Done uploading piece. err: <nil>
Jan 5 09:14:12.242929059 (737h45m12.114012094s) [noise=true , address=145.79.227.6:4258 , piece_id=PWLHHPPAVEGW3P5C335VL6CJ4K5LU2UIIFZMLMNHLMIP52NZBYAA, seg_pos={0 0}, upload=645537]: Done uploading piece. err: <nil>
Jan 5 09:14:13.125455593 (737h45m12.996538638s) [noise=true , address=45.91.116.6:4975 , piece_id=CXHE5GS2X4HNLNIIESI4YZH3NYPURSMASXS6H7CJRH32O2KSGSKQ, seg_pos={0 0}, upload=645537]: Done uploading piece. err: <nil>
Jan 5 09:14:13.222042781 (737h45m13.093125826s) [noise=true , address=77.90.31.19:4842 , piece_id=CSORWG66JCKA2B2ZUVUL2LWPBYEGAUWEMKARA4NRRM2GOZTQCSQQ, seg_pos={0 0}, upload=645537]: Done uploading piece. err: <nil>
Jan 5 09:14:13.646169338 (737h45m13.517252393s) [noise=true , address=145.79.227.49:4337 , piece_id=ZUHCA2OBCBBFQKTWPZDHQSM5A3KMVJDMSDZCVQMS7C6LACZBHKKA, seg_pos={0 0}, upload=645537]: Done uploading piece. err: <nil>
Jan 5 09:14:13.680855606 (737h45m13.551938651s) [noise=true , address=145.79.227.35:4483 , piece_id=QU57OUARYAB7DXVHAEP57LPKVENYBK6FXULSQWHMU33R7ATBCQ6Q, seg_pos={0 0}, upload=645537]: Done uploading piece. err: <nil>
Jan 5 09:14:13.680882036 (737h45m13.551965081s) [seg_pos={0 0}, upload=645537]: Segment reached optimal threshold of 13 pieces.
Jan 5 09:14:13.681166217 (737h45m13.552249262s) [noise=true , address=45.91.116.10:4081 , piece_id=QIFSZ6EURMEYLMSDMM5S2HOXHOWIRAXXMV5KIORC5BA5M6QP62XA, seg_pos={0 0}, upload=645537]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF
Jan 5 09:14:13.681194141 (737h45m13.552277186s) [noise=true , address=45.91.116.10:4081 , piece_id=QIFSZ6EURMEYLMSDMM5S2HOXHOWIRAXXMV5KIORC5BA5M6QP62XA, seg_pos={0 0}, upload=645537]: Long tail cancellation due to optimal threshold
Jan 5 09:14:13.681312316 (737h45m13.552395351s) [noise=true , address=45.91.116.13:4155 , piece_id=VH43XLDBAT3K4KSMFKZERPVAR7CN3OTBXCEAPOGWOUFACEF5WMFA, seg_pos={0 0}, upload=645537]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF
Jan 5 09:14:13.681328416 (737h45m13.552411452s) [noise=true , address=45.91.116.13:4155 , piece_id=VH43XLDBAT3K4KSMFKZERPVAR7CN3OTBXCEAPOGWOUFACEF5WMFA, seg_pos={0 0}, upload=645537]: Long tail cancellation due to optimal threshold
Jan 5 09:14:13.681684615 (737h45m13.55276766s) [noise=true , address=77.90.31.15:4697 , piece_id=KVG2MEKUDL3DPN7VFC4ITG4HOROPGPZA5EL5IAADAU5DANWLD7TQ, seg_pos={0 0}, upload=645537]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: context canceled; context canceled
Jan 5 09:14:13.681696959 (737h45m13.552780004s) [noise=true , address=77.90.31.15:4697 , piece_id=KVG2MEKUDL3DPN7VFC4ITG4HOROPGPZA5EL5IAADAU5DANWLD7TQ, seg_pos={0 0}, upload=645537]: Long tail cancellation due to optimal threshold
Jan 5 09:14:13.681722557 (737h45m13.552805602s) [seg_pos={0 0}, upload=645537]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 5 09:14:13.681882151 (737h45m13.552965196s) [upload=645537]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 5 09:14:13.681890777 (737h45m13.552973832s) [upload=645537]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 5 09:14:13.965823732 (737h45m13.836906767s) [upload=645537]: Done uploading segments...
Jan 5 09:14:13.965844372 (737h45m13.836927407s) [upload=645537]: Upload finished. err: <nil>
Duration: 6.215091649s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
41 out of 60 is failed (threshold=6) [.X..X.XXXXXXXX.XX..XXX.X.XX.XXX.XX.XXXXXX..XXXXXX...XXXX..XX] duration:
5.001224397s trace:
https://jaeger.dev.storj.io/trace/765e1884597b6832
FAIL (2026-01-05 09:15:46 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001258312s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [....XX............X..X...X..................................] duration:
24.587774606s trace:
https://jaeger.dev.storj.io/trace/7e43d4cb58eef6b9
FAIL (2026-01-05 09:15:40 UTC)
orbiter warning: context deadline exceeded
Duration: 24.587805605s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
26 out of 60 is failed (threshold=3) [...X.XX.XXXXXX.XXX...X....XX.X.X....XX....X..XX.XX......X.X.] duration:
16.11410192s trace:
https://jaeger.dev.storj.io/trace/72e39ce038a90917
FAIL (2026-01-05 09:15:18 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
IQ, seg_pos={0 0}, upload=645540]: Uploading piece...
Jan 5 09:14:28.481519582 (737h45m28.352602617s) [noise=true , address=64.49.13.155:4914 , piece_id=3EM5DTAFBVWKNDIXORJ2TGIA3XQUVD4AQMHTAVMTVMIB7VAGNBVQ, seg_pos={0 0}, upload=645540]: Uploading piece...
Jan 5 09:14:28.481931877 (737h45m28.353014912s) [noise=true , address=64.49.13.169:4711 , piece_id=FMDUZXRNTFKEPSK6CKT4VTP3QOONLMYT727EKNQRWSDBBV6MGJZA, seg_pos={0 0}, upload=645540]: Uploading piece...
Jan 5 09:14:28.482244453 (737h45m28.353327488s) [noise=true , address=145.79.227.29:4960 , piece_id=MCITYCG4KURSEKILPRLALCELTWYRVRBR7ZKATC2KAEF55G2XQWCA, seg_pos={0 0}, upload=645540]: Uploading piece...
Jan 5 09:14:28.482549292 (737h45m28.353632327s) [noise=true , address=193.149.164.143:4704 , piece_id=LCIL7FLIUKCOJ5APU4FQ2YNP3OMPWSDJUJNR4GAAIOJRU3ZEIOIA, seg_pos={0 0}, upload=645540]: Uploading piece...
Jan 5 09:14:28.484011206 (737h45m28.355094241s) [noise=true , address=145.79.227.32:4864 , piece_id=XVF4YPZAZE5LIASSN2UAFGJ5WSSQS7HZRCRAZFVMTXD3HZ3IJKCA, seg_pos={0 0}, upload=645540]: Uploading piece...
Jan 5 09:14:28.484360561 (737h45m28.355443606s) [noise=true , address=145.79.227.34:4910 , piece_id=MH32XA2B5MZGDAKQOI6YKE4FVTPBW6RYEKN3BG6ENPP3DWLFQ5MA, seg_pos={0 0}, upload=645540]: Uploading piece...
Jan 5 09:14:28.484660773 (737h45m28.355743808s) [noise=true , address=145.79.227.40:4057 , piece_id=ODUFWMT76YTQXUMIYGP3QWBCYDWSJ5FPBHAPFLC5GRDQBHSEINPQ, seg_pos={0 0}, upload=645540]: Uploading piece...
Jan 5 09:14:28.484993167 (737h45m28.356076212s) [noise=true , address=145.79.227.33:4872 , piece_id=IREXLNQBCYJ63U7MOL2JPQQJLBGLPU4UZYTE2GB26QHFDQV5YXSQ, seg_pos={0 0}, upload=645540]: Uploading piece...
Jan 5 09:14:28.485354134 (737h45m28.356437169s) [noise=true , address=145.79.227.30:4764 , piece_id=MNE6OULVQR7G27FBUXXNBTFSO42BYFTSHAAOVZMN4XST5Q3RQ6PQ, seg_pos={0 0}, upload=645540]: Uploading piece...
Jan 5 09:14:28.486630264 (737h45m28.357713299s) [noise=true , address=145.79.227.49:4363 , piece_id=2MHKYQFZ76RFH2OSA3PPGGJSTNOHVAQIRXPSSSDHQ6WKIIX7WS6Q, seg_pos={0 0}, upload=645540]: Uploading piece...
Jan 5 09:14:28.487945468 (737h45m28.359028503s) [noise=true , address=45.91.116.6:4001 , piece_id=RMVGTXAMF46OQ4SKOWK4TVUK5IW3AOFW6F2PDVK3Q2YPXUFNEQOA, seg_pos={0 0}, upload=645540]: Uploading piece...
Jan 5 09:14:28.489301700 (737h45m28.360384735s) [noise=true , address=145.79.227.13:4815 , piece_id=NAXAASY7SVFKRC3ZQZR72DE6STEMW6F2WAHMZ2EXLRHUGC4BMZPQ, seg_pos={0 0}, upload=645540]: Uploading piece...
Jan 5 09:14:28.490649908 (737h45m28.361732943s) [noise=true , address=77.90.31.14:4667 , piece_id=25PMWPFX3M4XOOJCXOBR2X62KT5C7ZMAFV3CYF5XLU5YBTBU4ZLA, seg_pos={0 0}, upload=645540]: Uploading piece...
Jan 5 09:14:28.492018193 (737h45m28.363101238s) [noise=true , address=145.79.227.41:4093 , piece_id=OPX4E236YBQ45HBFBVPJMSO54ZO3RNI276A6VBTRTKPLC5BEB7WQ, seg_pos={0 0}, upload=645540]: Uploading piece...
Jan 5 09:14:28.493479696 (737h45m28.364562741s) [noise=true , address=145.79.227.9:4369 , piece_id=FI2C4ADG3GP6HXH4IX5QYH234CHHTBTGWX7LS4NQYG4UUSDRVJIQ, seg_pos={0 0}, upload=645540]: Uploading piece...
Jan 5 09:14:29.938698368 (737h45m29.809781413s) [upload=645540]: Next returned nil segment
Jan 5 09:14:29.938715170 (737h45m29.809798215s) [upload=645540]: Waiting for error group managing segments...
Jan 5 09:14:30.332009807 (737h45m30.203092852s) [noise=true , address=145.79.227.9:4369 , piece_id=FI2C4ADG3GP6HXH4IX5QYH234CHHTBTGWX7LS4NQYG4UUSDRVJIQ, seg_pos={0 0}, upload=645540]: Done uploading piece. err: <nil>
Jan 5 09:14:30.376702489 (737h45m30.247785524s) [noise=true , address=145.79.227.13:4815 , piece_id=NAXAASY7SVFKRC3ZQZR72DE6STEMW6F2WAHMZ2EXLRHUGC4BMZPQ, seg_pos={0 0}, upload=645540]: Done uploading piece. err: <nil>
Jan 5 09:14:30.378623848 (737h45m30.249706883s) [noise=true , address=145.79.227.41:4093 , piece_id=OPX4E236YBQ45HBFBVPJMSO54ZO3RNI276A6VBTRTKPLC5BEB7WQ, seg_pos={0 0}, upload=645540]: Done uploading piece. err: <nil>
Jan 5 09:14:30.417434269 (737h45m30.288517314s) [noise=true , address=145.79.227.29:4960 , piece_id=MCITYCG4KURSEKILPRLALCELTWYRVRBR7ZKATC2KAEF55G2XQWCA, seg_pos={0 0}, upload=645540]: Done uploading piece. err: <nil>
Jan 5 09:14:30.418611010 (737h45m30.289694045s) [noise=true , address=145.79.227.36:4536 , piece_id=W2VBPUCRFKE24RW4MFC4LW5GWLZ3VLCPNOX7FB4V2LEVKBHIHQUQ, seg_pos={0 0}, upload=645540]: Done uploading piece. err: <nil>
Jan 5 09:14:30.489767223 (737h45m30.360850258s) [noise=true , address=145.79.227.49:4363 , piece_id=2MHKYQFZ76RFH2OSA3PPGGJSTNOHVAQIRXPSSSDHQ6WKIIX7WS6Q, seg_pos={0 0}, upload=645540]: Done uploading piece. err: <nil>
Jan 5 09:14:30.665342908 (737h45m30.536425943s) [noise=true , address=77.90.31.18:4810 , piece_id=MFVITPU246NG25OVHP7M5TLGCVLUVKHZBH46MOVJOCEWJPM4EDIQ, seg_pos={0 0}, upload=645540]: Done uploading piece. err: <nil>
Jan 5 09:14:30.795718836 (737h45m30.666801871s) [noise=true , address=77.90.31.14:4667 , piece_id=25PMWPFX3M4XOOJCXOBR2X62KT5C7ZMAFV3CYF5XLU5YBTBU4ZLA, seg_pos={0 0}, upload=645540]: Done uploading piece. err: <nil>
Jan 5 09:14:30.882779834 (737h45m30.753862869s) [noise=true , address=45.91.116.6:4001 , piece_id=RMVGTXAMF46OQ4SKOWK4TVUK5IW3AOFW6F2PDVK3Q2YPXUFNEQOA, seg_pos={0 0}, upload=645540]: Done uploading piece. err: <nil>
Jan 5 09:14:31.236010964 (737h45m31.107093999s) [noise=true , address=193.149.164.143:4704 , piece_id=LCIL7FLIUKCOJ5APU4FQ2YNP3OMPWSDJUJNR4GAAIOJRU3ZEIOIA, seg_pos={0 0}, upload=645540]: Done uploading piece. err: <nil>
Jan 5 09:14:31.239540344 (737h45m31.110623389s) [noise=true , address=145.79.227.30:4764 , piece_id=MNE6OULVQR7G27FBUXXNBTFSO42BYFTSHAAOVZMN4XST5Q3RQ6PQ, seg_pos={0 0}, upload=645540]: Done uploading piece. err: <nil>
Jan 5 09:14:31.278936831 (737h45m31.150019876s) [noise=true , address=64.49.13.155:4914 , piece_id=3EM5DTAFBVWKNDIXORJ2TGIA3XQUVD4AQMHTAVMTVMIB7VAGNBVQ, seg_pos={0 0}, upload=645540]: Done uploading piece. err: <nil>
Jan 5 09:14:31.293259244 (737h45m31.164342319s) [noise=true , address=145.79.227.40:4057 , piece_id=ODUFWMT76YTQXUMIYGP3QWBCYDWSJ5FPBHAPFLC5GRDQBHSEINPQ, seg_pos={0 0}, upload=645540]: Done uploading piece. err: <nil>
Jan 5 09:14:31.304613935 (737h45m31.17569698s) [noise=true , address=145.79.227.33:4872 , piece_id=IREXLNQBCYJ63U7MOL2JPQQJLBGLPU4UZYTE2GB26QHFDQV5YXSQ, seg_pos={0 0}, upload=645540]: Done uploading piece. err: <nil>
Jan 5 09:14:31.304640466 (737h45m31.175723511s) [seg_pos={0 0}, upload=645540]: Segment reached optimal threshold of 14 pieces.
Jan 5 09:14:31.304890572 (737h45m31.175973617s) [noise=true , address=145.79.227.34:4910 , piece_id=MH32XA2B5MZGDAKQOI6YKE4FVTPBW6RYEKN3BG6ENPP3DWLFQ5MA, seg_pos={0 0}, upload=645540]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRYzNggkq7QWFTP9fB94dTrMMX8AYvKMt2zjmJBzT1VwqyPVZ): protocol: expected piece hash; context canceled; EOF
Jan 5 09:14:31.304899969 (737h45m31.175983014s) [noise=true , address=145.79.227.34:4910 , piece_id=MH32XA2B5MZGDAKQOI6YKE4FVTPBW6RYEKN3BG6ENPP3DWLFQ5MA, seg_pos={0 0}, upload=645540]: Long tail cancellation due to optimal threshold
Jan 5 09:14:31.305056848 (737h45m31.176139894s) [noise=true , address=64.49.13.169:4711 , piece_id=FMDUZXRNTFKEPSK6CKT4VTP3QOONLMYT727EKNQRWSDBBV6MGJZA, seg_pos={0 0}, upload=645540]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:45526->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:45526->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45526->64.49.13.169:4711: use of closed network connection
Jan 5 09:14:31.305065565 (737h45m31.17614861s) [noise=true , address=64.49.13.169:4711 , piece_id=FMDUZXRNTFKEPSK6CKT4VTP3QOONLMYT727EKNQRWSDBBV6MGJZA, seg_pos={0 0}, upload=645540]: Long tail cancellation due to optimal threshold
Jan 5 09:14:31.305087907 (737h45m31.176170952s) [noise=true , address=45.91.116.4:4954 , piece_id=5P7GC2WX6EU5ZMCQ6YLLE5US3INKBERSF6NIUCNVVBLX6DVAQX6Q, seg_pos={0 0}, upload=645540]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tT1yhzA7AUuwesNf62NHyVxgSvbrmFzchYSmYdqihRjuAgvYa): protocol: read tcp 188.166.225.170:47680->45.91.116.4:4954: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 5 09:14:31.305096514 (737h45m31.176179549s) [noise=true , address=45.91.116.4:4954 , piece_id=5P7GC2WX6EU5ZMCQ6YLLE5US3INKBERSF6NIUCNVVBLX6DVAQX6Q, seg_pos={0 0}, upload=645540]: Long tail cancellation due to optimal threshold
Jan 5 09:14:31.305227412 (737h45m31.176310457s) [noise=true , address=145.79.227.32:4864 , piece_id=XVF4YPZAZE5LIASSN2UAFGJ5WSSQS7HZRCRAZFVMTXD3HZ3IJKCA, seg_pos={0 0}, upload=645540]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1z6uPcGsR78PiEpT46NBJEdyH33Re1YU5CtrMMFVeFf1KD3egC): protocol: expected piece hash; context canceled; EOF
Jan 5 09:14:31.305236140 (737h45m31.176319175s) [noise=true , address=145.79.227.32:4864 , piece_id=XVF4YPZAZE5LIASSN2UAFGJ5WSSQS7HZRCRAZFVMTXD3HZ3IJKCA, seg_pos={0 0}, upload=645540]: Long tail cancellation due to optimal threshold
Jan 5 09:14:31.305255366 (737h45m31.176338401s) [seg_pos={0 0}, upload=645540]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jan 5 09:14:31.305338775 (737h45m31.17642182s) [upload=645540]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 5 09:14:31.305345979 (737h45m31.176429034s) [upload=645540]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 5 09:14:31.592583505 (737h45m31.46366654s) [upload=645540]: Done uploading segments...
Jan 5 09:14:31.592604706 (737h45m31.463687751s) [upload=645540]: Upload finished. err: <nil>
Duration: 16.11423894s
core
uplink
us-select-1
us1-placement-20
|