Orbiter History

Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure

Orbiter Check History

Showing 50 of 356948 reports (Page 223 of 7139)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [........................X..XX..X..................X.......X.] duration: 55.673216948s trace: https://jaeger.dev.storj.io/trace/697b20119e0593f8 FAIL (2025-12-30 16:03:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 01:57.034542821 (600h32m56.905625866s) [noise=true , address=77.90.31.13:4658 , piece_id=ASWEV2PJC5SIAPGOUC26KDJDG3XLRFW2E6CMV6LL66YIVB7Y5QWQ, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.034697947 (600h32m56.905780992s) [noise=true , address=145.79.227.25:4622 , piece_id=5BSWHUMVRADKEJX3SLEP7BRMUWQKLZQEJE2IDFUXLUQWMBU7PCTA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.035092007 (600h32m56.906175052s) [noise=true , address=77.90.31.15:4729 , piece_id=MF52XXD46K72RFKEULEWW7AOBBVWIIQT2CS7V7TG27L7C5WARCYQ, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.035493051 (600h32m56.906576086s) [noise=true , address=45.91.116.4:4949 , piece_id=K2LRBT6MVGHP3O4MGMTARHMO5FVGFKTMVWYQOR2J7GX4GZNUM6BA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.035882262 (600h32m56.906965307s) [noise=true , address=145.79.227.43:4178 , piece_id=OW36VWZ7KNAZ6JU5VYRP5IHCJZCLFNJZBNXHT2LLNDYKSSJW5FCA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.037716825 (600h32m56.90879987s) [noise=true , address=77.90.31.12:4619 , piece_id=RS3SIHFI6I4RMOTAFYN7NUIIE4WW7E7D3B7RSMCYO4BNN577U3RA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.038358467 (600h32m56.909441502s) [noise=true , address=77.90.31.19:4845 , piece_id=JSPTJOPQ55NKTUAL2HTSEQ4WRCGJNFFGPE7LMUNDMTKEFYZDM2UQ, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.040066269 (600h32m56.911149304s) [noise=true , address=77.90.31.18:4838 , piece_id=URNHRXNV45MVCL6ZWSFY3GQLYJT6U3MPZEIA7M74WF6ZXHQ7H5FA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.042000342 (600h32m56.913083377s) [noise=true , address=5.231.73.4:4028 , piece_id=Y4DYOOAI6GNLUXYDBAYANPWNC3T6QN6U5U64CUO5SXROKWZKVFKQ, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.044053191 (600h32m56.915136226s) [noise=true , address=77.90.31.15:4714 , piece_id=NGX4EVOIPZMCA37RULTAGRVXNI533GKFQJ2ERLFGXDSDZ5H6QHSA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.045847507 (600h32m56.916930552s) [noise=true , address=145.79.227.30:4767 , piece_id=PYUIYB7K2DV5YRAW2GLXCNR4AJ3QQRLWHTYWVH36OPGTVQHTZJUA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.047490265 (600h32m56.91857331s) [noise=true , address=145.79.227.29:4952 , piece_id=R54XH66RF55YEOR62STLKB5YXP6S4KE6NY74VW5J74ZX3YOSAOAA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.049047280 (600h32m56.920130325s) [noise=true , address=145.79.227.44:4197 , piece_id=DLWVGXMXAEU4Y7Q5S2HZQUV3VUHE763FHT26P4W3G4RUN2JLZ6EA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.050106617 (600h32m56.921189652s) [noise=true , address=145.79.227.33:4890 , piece_id=OTPNHUHTHEMYMPKYZNIU3QQ55GXICCYWD7NH4NDA7EHBFUGVBJSQ, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.051582247 (600h32m56.922665292s) [noise=true , address=77.90.31.19:4870 , piece_id=WTCTOZH6665YQ6OW2232VISBWYRUBHL74HML5GJHHJ4PSE2KEYKA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.052071889 (600h32m56.923154935s) [noise=true , address=145.79.227.19:4089 , piece_id=ISNJCL22W46DM5KUOAOOPM2T2VDGV3JM3PQF2MDF73RCF4GTM4SA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.052894146 (600h32m56.923977191s) [noise=true , address=145.79.227.40:4052 , piece_id=EV363HTTZXTV2YEMHZX466LLTK4SBS7RNBTZOWTCHTG2E4OKTPGQ, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:58.511258703 (600h32m58.382341748s) [upload=526348]: Next returned nil segment Dec 30 16:01:58.511280185 (600h32m58.38236322s) [upload=526348]: Waiting for error group managing segments... Dec 30 16:01:58.918592018 (600h32m58.789675063s) [noise=true , address=145.79.227.25:4622 , piece_id=5BSWHUMVRADKEJX3SLEP7BRMUWQKLZQEJE2IDFUXLUQWMBU7PCTA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:58.935469496 (600h32m58.806552531s) [noise=true , address=145.79.227.19:4089 , piece_id=ISNJCL22W46DM5KUOAOOPM2T2VDGV3JM3PQF2MDF73RCF4GTM4SA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.094374364 (600h32m58.965457399s) [noise=true , address=77.90.31.15:4729 , piece_id=MF52XXD46K72RFKEULEWW7AOBBVWIIQT2CS7V7TG27L7C5WARCYQ, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.174591008 (600h32m59.045674043s) [noise=true , address=77.90.31.19:4845 , piece_id=JSPTJOPQ55NKTUAL2HTSEQ4WRCGJNFFGPE7LMUNDMTKEFYZDM2UQ, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.180206382 (600h32m59.051289427s) [noise=true , address=5.231.73.4:4028 , piece_id=Y4DYOOAI6GNLUXYDBAYANPWNC3T6QN6U5U64CUO5SXROKWZKVFKQ, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.247271233 (600h32m59.118354268s) [noise=true , address=45.91.116.7:4014 , piece_id=RVQUWHPEV4XDF7YE4QEX6UKHZEYKE5PHAQ7NH73JB5QCV7PN3KYA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.260497178 (600h32m59.131580213s) [noise=true , address=45.91.116.4:4949 , piece_id=K2LRBT6MVGHP3O4MGMTARHMO5FVGFKTMVWYQOR2J7GX4GZNUM6BA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.401943773 (600h32m59.273026808s) [noise=true , address=77.90.31.13:4658 , piece_id=ASWEV2PJC5SIAPGOUC26KDJDG3XLRFW2E6CMV6LL66YIVB7Y5QWQ, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.508863164 (600h32m59.379946199s) [noise=true , address=77.90.31.19:4870 , piece_id=WTCTOZH6665YQ6OW2232VISBWYRUBHL74HML5GJHHJ4PSE2KEYKA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.743613925 (600h32m59.61469696s) [noise=true , address=77.90.31.15:4714 , piece_id=NGX4EVOIPZMCA37RULTAGRVXNI533GKFQJ2ERLFGXDSDZ5H6QHSA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.812479335 (600h32m59.68356237s) [noise=true , address=77.90.31.18:4838 , piece_id=URNHRXNV45MVCL6ZWSFY3GQLYJT6U3MPZEIA7M74WF6ZXHQ7H5FA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.823406823 (600h32m59.694489868s) [noise=true , address=145.79.227.43:4178 , piece_id=OW36VWZ7KNAZ6JU5VYRP5IHCJZCLFNJZBNXHT2LLNDYKSSJW5FCA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.843550510 (600h32m59.714633545s) [noise=true , address=145.79.227.29:4952 , piece_id=R54XH66RF55YEOR62STLKB5YXP6S4KE6NY74VW5J74ZX3YOSAOAA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.905172958 (600h32m59.776255993s) [noise=true , address=145.79.227.44:4197 , piece_id=DLWVGXMXAEU4Y7Q5S2HZQUV3VUHE763FHT26P4W3G4RUN2JLZ6EA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.905198386 (600h32m59.776281431s) [seg_pos={0 0}, upload=526348]: Segment reached optimal threshold of 14 pieces. Dec 30 16:01:59.905434836 (600h32m59.776517871s) [noise=true , address=77.90.31.12:4619 , piece_id=RS3SIHFI6I4RMOTAFYN7NUIIE4WW7E7D3B7RSMCYO4BNN577U3RA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX): protocol: expected piece hash; context canceled; EOF Dec 30 16:01:59.905443382 (600h32m59.776526437s) [noise=true , address=77.90.31.12:4619 , piece_id=RS3SIHFI6I4RMOTAFYN7NUIIE4WW7E7D3B7RSMCYO4BNN577U3RA, seg_pos={0 0}, upload=526348]: Long tail cancellation due to optimal threshold Dec 30 16:01:59.905573240 (600h32m59.776656285s) [noise=true , address=145.79.227.33:4890 , piece_id=OTPNHUHTHEMYMPKYZNIU3QQ55GXICCYWD7NH4NDA7EHBFUGVBJSQ, seg_pos={0 0}, upload=526348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XYhMcqH63UiHYE4ofxDxjXz2HMvN1RfQjgxxxHcUst7P3aN7M): protocol: expected piece hash; context canceled; EOF Dec 30 16:01:59.905610260 (600h32m59.776693295s) [noise=true , address=145.79.227.33:4890 , piece_id=OTPNHUHTHEMYMPKYZNIU3QQ55GXICCYWD7NH4NDA7EHBFUGVBJSQ, seg_pos={0 0}, upload=526348]: Long tail cancellation due to optimal threshold Dec 30 16:01:59.905756799 (600h32m59.776839844s) [noise=true , address=145.79.227.40:4052 , piece_id=EV363HTTZXTV2YEMHZX466LLTK4SBS7RNBTZOWTCHTG2E4OKTPGQ, seg_pos={0 0}, upload=526348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PViJBYAanwXRWhMq3WHR6k9Stxy8hjuY5FV9J7ErxCy4N1q3V): protocol: expected piece hash; context canceled; EOF Dec 30 16:01:59.905766168 (600h32m59.776849203s) [noise=true , address=145.79.227.40:4052 , piece_id=EV363HTTZXTV2YEMHZX466LLTK4SBS7RNBTZOWTCHTG2E4OKTPGQ, seg_pos={0 0}, upload=526348]: Long tail cancellation due to optimal threshold Dec 30 16:01:59.905893880 (600h32m59.776976915s) [noise=true , address=145.79.227.30:4767 , piece_id=PYUIYB7K2DV5YRAW2GLXCNR4AJ3QQRLWHTYWVH36OPGTVQHTZJUA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sfaYCUhNCAyQAPu1jac5dRWR35Qqc4ZcKg75mp7nVQak295ug): protocol: expected piece hash; context canceled; EOF Dec 30 16:01:59.905901936 (600h32m59.776984971s) [noise=true , address=145.79.227.30:4767 , piece_id=PYUIYB7K2DV5YRAW2GLXCNR4AJ3QQRLWHTYWVH36OPGTVQHTZJUA, seg_pos={0 0}, upload=526348]: Long tail cancellation due to optimal threshold Dec 30 16:01:59.905945239 (600h32m59.777028284s) [seg_pos={0 0}, upload=526348]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 16:01:59.906043476 (600h32m59.777126511s) [upload=526348]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 16:01:59.906048395 (600h32m59.77713143s) [upload=526348]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 16:02:00.203472835 (600h33m0.074555871s) [upload=526348]: Done uploading segments... Dec 30 16:02:00.203498004 (600h33m0.074581049s) [upload=526348]: Upload finished. err: <nil> Duration: 55.673242968s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.........XX.......XX.XX..X....X.......X......X.X..........X.] duration: 5.81552128s trace: https://jaeger.dev.storj.io/trace/c8dacfa2493181d FAIL (2025-12-30 16:03:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: TD7M7C6DBEPTOJ6LUBIMR5YQJMTBP74KYQIZTZTJTLSKVGMNVULA) context deadline exceeded; piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: YE2VAC3CM5KDPG5K6VGSIXEMS5MESXRCX7G4FM3EJKCSHPJOS22A) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: 2Y6IC654KNMHEZDZQEB5NH6SW7O6KSJIRAUBOTWXNQDOPTDKV73Q) context deadline exceeded ======LOGS====== 0 16:02:54.043586012 (600h33m53.914669057s) [upload=526359]: Starting upload Dec 30 16:02:54.043669701 (600h33m53.914752736s) [upload=526359]: Done starting upload Dec 30 16:02:54.043783207 (600h33m53.914866242s) [upload=526359]: Uploading segments... Dec 30 16:02:54.043888678 (600h33m53.914971723s) [upload=526359]: Got next segment. Inline: false Dec 30 16:02:54.043935177 (600h33m53.915018222s) [upload=526359]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 16:02:54.043941499 (600h33m53.915024544s) [upload=526359]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 16:02:54.288720686 (600h33m54.159803731s) [seg_pos={0 0}, upload=526359]: Begin upload segment... Dec 30 16:02:54.288790850 (600h33m54.159873885s) [seg_pos={0 0}, upload=526359]: Done begin upload segment. Dec 30 16:02:54.288850282 (600h33m54.159933327s) [noise=true , address=145.79.227.20:4438 , piece_id=VYKE7J7TBUXCE74LUIAZ5Z3IHJO6HU7EIQQAO7Z3SBZ7BZ5275QQ, seg_pos={0 0}, upload=526359]: Uploading piece... Dec 30 16:02:54.288855693 (600h33m54.159938738s) [noise=true , address=45.91.116.13:4156 , piece_id=KEI4HNEZAGSZKGZXLI5RMKPR3WUZGZ3UTHDJW2GDJULSELMJKXUA, seg_pos={0 0}, upload=526359]: Uploading piece... Dec 30 16:02:54.289380913 (600h33m54.160463959s) [noise=true , address=145.79.227.34:4905 , piece_id=VI3ET3QBCZGNWXDNWBU26QYDE62KQMHPU437OWBQTL3ATIIHSVPA, seg_pos={0 0}, upload=526359]: Uploading piece... Dec 30 16:02:54.289796444 (600h33m54.160879489s) [noise=true , address=145.79.227.29:4943 , piece_id=6AN7DLYBG35QV32DKHEOXTJ73JWHHOVF5CBPCYRHFIW3SKPQKWSA, seg_pos={0 0}, upload=526359]: Uploading piece... Dec 30 16:02:54.290423799 (600h33m54.161506854s) [noise=true , address=145.79.227.11:4729 , piece_id=KBIVWPOXWRKXCOD6Z35X7GGADEDCRRDO2PNMN5ZH2RJBHOK2NQ7Q, seg_pos={0 0}, upload=526359]: Uploading piece... Dec 30 16:02:54.290841253 (600h33m54.161924319s) [noise=true , address=45.91.116.7:4009 , piece_id=YE2VAC3CM5KDPG5K6VGSIXEMS5MESXRCX7G4FM3EJKCSHPJOS22A, seg_pos={0 0}, upload=526359]: Uploading piece... Dec 30 16:02:54.291317160 (600h33m54.162400205s) [noise=true , address=145.79.227.36:4517 , piece_id=C7NFGCBBZJRUCG5D6BNJKCRNPCFHES6YEG5S3YQ3AUKAMK5SRLAA, seg_pos={0 0}, upload=526359]: Uploading piece... Dec 30 16:02:54.291336247 (600h33m54.162419302s) [noise=true , address=77.90.31.15:4700 , piece_id=2Y6IC654KNMHEZDZQEB5NH6SW7O6KSJIRAUBOTWXNQDOPTDKV73Q, seg_pos={0 0}, upload=526359]: Uploading piece... Dec 30 16:02:54.291759332 (600h33m54.162842387s) [noise=true , address=77.90.31.17:4773 , piece_id=4GYS5FJFICA3VKBF6CYY26V636EXNOK52MENECMEZXFIKELRQKRA, seg_pos={0 0}, upload=526359]: Uploading piece... Dec 30 16:02:54.291781905 (600h33m54.16286494s) [noise=true , address=145.79.227.21:4474 , piece_id=RJ2N57AWUFATMQYBPO3RCX6QHW6IO3YIILQA4CKM4YHEQGRACLCQ, seg_pos={0 0}, upload=526359]: Uploading piece... Dec 30 16:02:54.292119598 (600h33m54.163202643s) [noise=true , address=77.90.31.10:4518 , piece_id=MEW3A5VWKABCX5H5VZ6H4XGVMKU5R6A4L34YXHB7FNFZE2IFMGZQ, seg_pos={0 0}, upload=526359]: Uploading piece... Dec 30 16:02:54.292308778 (600h33m54.163391814s) [noise=true , address=77.90.31.16:4736 , piece_id=TD7M7C6DBEPTOJ6LUBIMR5YQJMTBP74KYQIZTZTJTLSKVGMNVULA, seg_pos={0 0}, upload=526359]: Uploading piece... Dec 30 16:02:54.292456880 (600h33m54.163539925s) [noise=true , address=145.79.227.50:4373 , piece_id=REUHJESPRRY5WWEAF3MKKHDI7HDISFUK3ODWWTZYQ2T6FE3EKRAQ, seg_pos={0 0}, upload=526359]: Uploading piece... Dec 30 16:02:54.292772441 (600h33m54.163855486s) [noise=true , address=145.79.227.16:4911 , piece_id=TBQTXCTULL5QQPCUOHP4Z5QUOV7WGFWBNTIBBBCKC2LHPLHME5NQ, seg_pos={0 0}, upload=526359]: Uploading piece... Dec 30 16:02:54.292801046 (600h33m54.163884081s) [noise=true , address=145.79.227.13:4803 , piece_id=TSUN6UX3ZAYSMGDMSLYUDBX4KIMKX3FZZW7IA5FRXZVT7BJDDSEQ, seg_pos={0 0}, upload=526359]: Uploading piece... Dec 30 16:02:54.295223859 (600h33m54.166306894s) [noise=true , address=45.91.116.10:4082 , piece_id=2O4EWILMHWMT6ASGS25W3TAJBNHRVVIC6O4SX5OIGPBHNBIUWJDQ, seg_pos={0 0}, upload=526359]: Uploading piece... Dec 30 16:02:55.895795882 (600h33m55.766878917s) [upload=526359]: Next returned nil segment Dec 30 16:02:55.895815018 (600h33m55.766898063s) [upload=526359]: Waiting for error group managing segments... Dec 30 16:02:56.276129444 (600h33m56.147212509s) [noise=true , address=145.79.227.13:4803 , piece_id=TSUN6UX3ZAYSMGDMSLYUDBX4KIMKX3FZZW7IA5FRXZVT7BJDDSEQ, seg_pos={0 0}, upload=526359]: Done uploading piece. err: <nil> Dec 30 16:02:56.277669617 (600h33m56.148752652s) [noise=true , address=145.79.227.21:4474 , piece_id=RJ2N57AWUFATMQYBPO3RCX6QHW6IO3YIILQA4CKM4YHEQGRACLCQ, seg_pos={0 0}, upload=526359]: Done uploading piece. err: <nil> Dec 30 16:02:56.280043227 (600h33m56.151126272s) [noise=true , address=145.79.227.11:4729 , piece_id=KBIVWPOXWRKXCOD6Z35X7GGADEDCRRDO2PNMN5ZH2RJBHOK2NQ7Q, seg_pos={0 0}, upload=526359]: Done uploading piece. err: <nil> Dec 30 16:02:56.285185849 (600h33m56.156268894s) [noise=true , address=145.79.227.34:4905 , piece_id=VI3ET3QBCZGNWXDNWBU26QYDE62KQMHPU437OWBQTL3ATIIHSVPA, seg_pos={0 0}, upload=526359]: Done uploading piece. err: <nil> Dec 30 16:02:56.291145798 (600h33m56.162228853s) [noise=true , address=145.79.227.20:4438 , piece_id=VYKE7J7TBUXCE74LUIAZ5Z3IHJO6HU7EIQQAO7Z3SBZ7BZ5275QQ, seg_pos={0 0}, upload=526359]: Done uploading piece. err: <nil> Dec 30 16:02:56.320073330 (600h33m56.191156375s) [noise=true , address=145.79.227.36:4517 , piece_id=C7NFGCBBZJRUCG5D6BNJKCRNPCFHES6YEG5S3YQ3AUKAMK5SRLAA, seg_pos={0 0}, upload=526359]: Done uploading piece. err: <nil> Dec 30 16:02:56.356196825 (600h33m56.22727987s) [noise=true , address=145.79.227.16:4911 , piece_id=TBQTXCTULL5QQPCUOHP4Z5QUOV7WGFWBNTIBBBCKC2LHPLHME5NQ, seg_pos={0 0}, upload=526359]: Done uploading piece. err: <nil> Dec 30 16:02:56.366077118 (600h33m56.237160153s) [noise=true , address=145.79.227.50:4373 , piece_id=REUHJESPRRY5WWEAF3MKKHDI7HDISFUK3ODWWTZYQ2T6FE3EKRAQ, seg_pos={0 0}, upload=526359]: Done uploading piece. err: <nil> Dec 30 16:02:56.472393460 (600h33m56.343476495s) [noise=true , address=77.90.31.15:4700 , piece_id=2Y6IC654KNMHEZDZQEB5NH6SW7O6KSJIRAUBOTWXNQDOPTDKV73Q, seg_pos={0 0}, upload=526359]: Done uploading piece. err: <nil> Dec 30 16:02:56.550020283 (600h33m56.421103348s) [noise=true , address=77.90.31.10:4518 , piece_id=MEW3A5VWKABCX5H5VZ6H4XGVMKU5R6A4L34YXHB7FNFZE2IFMGZQ, seg_pos={0 0}, upload=526359]: Done uploading piece. err: <nil> Dec 30 16:02:56.597141219 (600h33m56.468224254s) [noise=true , address=45.91.116.7:4009 , piece_id=YE2VAC3CM5KDPG5K6VGSIXEMS5MESXRCX7G4FM3EJKCSHPJOS22A, seg_pos={0 0}, upload=526359]: Done uploading piece. err: <nil> Dec 30 16:02:56.597187597 (600h33m56.468270632s) [noise=true , address=77.90.31.17:4773 , piece_id=4GYS5FJFICA3VKBF6CYY26V636EXNOK52MENECMEZXFIKELRQKRA, seg_pos={0 0}, upload=526359]: Done uploading piece. err: <nil> Dec 30 16:02:57.244666124 (600h33m57.115749159s) [noise=true , address=77.90.31.16:4736 , piece_id=TD7M7C6DBEPTOJ6LUBIMR5YQJMTBP74KYQIZTZTJTLSKVGMNVULA, seg_pos={0 0}, upload=526359]: Done uploading piece. err: <nil> Dec 30 16:02:57.244690050 (600h33m57.115773085s) [seg_pos={0 0}, upload=526359]: Segment reached optimal threshold of 13 pieces. Dec 30 16:02:57.244892646 (600h33m57.115975681s) [noise=true , address=45.91.116.10:4082 , piece_id=2O4EWILMHWMT6ASGS25W3TAJBNHRVVIC6O4SX5OIGPBHNBIUWJDQ, seg_pos={0 0}, upload=526359]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: write tcp 188.166.225.170:42340->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:42340->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42340->45.91.116.10:4082: use of closed network connection Dec 30 16:02:57.244899669 (600h33m57.115982704s) [noise=true , address=45.91.116.10:4082 , piece_id=2O4EWILMHWMT6ASGS25W3TAJBNHRVVIC6O4SX5OIGPBHNBIUWJDQ, seg_pos={0 0}, upload=526359]: Long tail cancellation due to optimal threshold Dec 30 16:02:57.245125298 (600h33m57.116208333s) [noise=true , address=45.91.116.13:4156 , piece_id=KEI4HNEZAGSZKGZXLI5RMKPR3WUZGZ3UTHDJW2GDJULSELMJKXUA, seg_pos={0 0}, upload=526359]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 30 16:02:57.245154985 (600h33m57.11623802s) [noise=true , address=45.91.116.13:4156 , piece_id=KEI4HNEZAGSZKGZXLI5RMKPR3WUZGZ3UTHDJW2GDJULSELMJKXUA, seg_pos={0 0}, upload=526359]: Long tail cancellation due to optimal threshold Dec 30 16:02:57.245168410 (600h33m57.116251455s) [noise=true , address=145.79.227.29:4943 , piece_id=6AN7DLYBG35QV32DKHEOXTJ73JWHHOVF5CBPCYRHFIW3SKPQKWSA, seg_pos={0 0}, upload=526359]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz): protocol: expected piece hash; context canceled; EOF Dec 30 16:02:57.245175274 (600h33m57.116258309s) [noise=true , address=145.79.227.29:4943 , piece_id=6AN7DLYBG35QV32DKHEOXTJ73JWHHOVF5CBPCYRHFIW3SKPQKWSA, seg_pos={0 0}, upload=526359]: Long tail cancellation due to optimal threshold Dec 30 16:02:57.245191665 (600h33m57.1162747s) [seg_pos={0 0}, upload=526359]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 16:02:57.245257059 (600h33m57.116340094s) [upload=526359]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 16:02:57.245261027 (600h33m57.116344062s) [upload=526359]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 16:02:57.542655802 (600h33m57.413738837s) [upload=526359]: Done uploading segments... Dec 30 16:02:57.542681380 (600h33m57.413764416s) [upload=526359]: Upload finished. err: <nil> Duration: 5.815549926s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 12.137247501s trace: https://jaeger.dev.storj.io/trace/547648a2cfce50e7 FAIL (2025-12-30 16:03:38 UTC) orbiter warning: context deadline exceeded Duration: 12.137306614s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 43 out of 60 is failed (threshold=6) [.X.X.XXXXXX.XXXX...XX.X.XXXXXXXXX.XXXXXXX...XXXXX.XX..XXXXX.] duration: 3.464244404s trace: https://jaeger.dev.storj.io/trace/299d53628bbbc11a FAIL (2025-12-30 16:03:33 UTC) orbiter warning: context deadline exceeded Duration: 3.464269953s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [..........XX.......XX.XX..X....X.......X......X.X..........X] duration: 30.002385169s trace: https://jaeger.dev.storj.io/trace/c8dacfa2493181d FAIL (2025-12-30 16:03:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: TD7M7C6DBEPTOJ6LUBIMR5YQJMTBP74KYQIZTZTJTLSKVGMNVULA) context deadline exceeded; piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: YE2VAC3CM5KDPG5K6VGSIXEMS5MESXRCX7G4FM3EJKCSHPJOS22A) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: 2Y6IC654KNMHEZDZQEB5NH6SW7O6KSJIRAUBOTWXNQDOPTDKV73Q) context deadline exceeded ======LOGS====== 0 16:02:54.043586012 (600h33m53.914669057s) [upload=526359]: Starting upload Dec 30 16:02:54.043669701 (600h33m53.914752736s) [upload=526359]: Done starting upload Dec 30 16:02:54.043783207 (600h33m53.914866242s) [upload=526359]: Uploading segments... Dec 30 16:02:54.043888678 (600h33m53.914971723s) [upload=526359]: Got next segment. Inline: false Dec 30 16:02:54.043935177 (600h33m53.915018222s) [upload=526359]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 16:02:54.043941499 (600h33m53.915024544s) [upload=526359]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 16:02:54.288720686 (600h33m54.159803731s) [seg_pos={0 0}, upload=526359]: Begin upload segment... Dec 30 16:02:54.288790850 (600h33m54.159873885s) [seg_pos={0 0}, upload=526359]: Done begin upload segment. Dec 30 16:02:54.288850282 (600h33m54.159933327s) [noise=true , address=145.79.227.20:4438 , piece_id=VYKE7J7TBUXCE74LUIAZ5Z3IHJO6HU7EIQQAO7Z3SBZ7BZ5275QQ, seg_pos={0 0}, upload=526359]: Uploading piece... Dec 30 16:02:54.288855693 (600h33m54.159938738s) [noise=true , address=45.91.116.13:4156 , piece_id=KEI4HNEZAGSZKGZXLI5RMKPR3WUZGZ3UTHDJW2GDJULSELMJKXUA, seg_pos={0 0}, upload=526359]: Uploading piece... Dec 30 16:02:54.289380913 (600h33m54.160463959s) [noise=true , address=145.79.227.34:4905 , piece_id=VI3ET3QBCZGNWXDNWBU26QYDE62KQMHPU437OWBQTL3ATIIHSVPA, seg_pos={0 0}, upload=526359]: Uploading piece... Dec 30 16:02:54.289796444 (600h33m54.160879489s) [noise=true , address=145.79.227.29:4943 , piece_id=6AN7DLYBG35QV32DKHEOXTJ73JWHHOVF5CBPCYRHFIW3SKPQKWSA, seg_pos={0 0}, upload=526359]: Uploading piece... Dec 30 16:02:54.290423799 (600h33m54.161506854s) [noise=true , address=145.79.227.11:4729 , piece_id=KBIVWPOXWRKXCOD6Z35X7GGADEDCRRDO2PNMN5ZH2RJBHOK2NQ7Q, seg_pos={0 0}, upload=526359]: Uploading piece... Dec 30 16:02:54.290841253 (600h33m54.161924319s) [noise=true , address=45.91.116.7:4009 , piece_id=YE2VAC3CM5KDPG5K6VGSIXEMS5MESXRCX7G4FM3EJKCSHPJOS22A, seg_pos={0 0}, upload=526359]: Uploading piece... Dec 30 16:02:54.291317160 (600h33m54.162400205s) [noise=true , address=145.79.227.36:4517 , piece_id=C7NFGCBBZJRUCG5D6BNJKCRNPCFHES6YEG5S3YQ3AUKAMK5SRLAA, seg_pos={0 0}, upload=526359]: Uploading piece... Dec 30 16:02:54.291336247 (600h33m54.162419302s) [noise=true , address=77.90.31.15:4700 , piece_id=2Y6IC654KNMHEZDZQEB5NH6SW7O6KSJIRAUBOTWXNQDOPTDKV73Q, seg_pos={0 0}, upload=526359]: Uploading piece... Dec 30 16:02:54.291759332 (600h33m54.162842387s) [noise=true , address=77.90.31.17:4773 , piece_id=4GYS5FJFICA3VKBF6CYY26V636EXNOK52MENECMEZXFIKELRQKRA, seg_pos={0 0}, upload=526359]: Uploading piece... Dec 30 16:02:54.291781905 (600h33m54.16286494s) [noise=true , address=145.79.227.21:4474 , piece_id=RJ2N57AWUFATMQYBPO3RCX6QHW6IO3YIILQA4CKM4YHEQGRACLCQ, seg_pos={0 0}, upload=526359]: Uploading piece... Dec 30 16:02:54.292119598 (600h33m54.163202643s) [noise=true , address=77.90.31.10:4518 , piece_id=MEW3A5VWKABCX5H5VZ6H4XGVMKU5R6A4L34YXHB7FNFZE2IFMGZQ, seg_pos={0 0}, upload=526359]: Uploading piece... Dec 30 16:02:54.292308778 (600h33m54.163391814s) [noise=true , address=77.90.31.16:4736 , piece_id=TD7M7C6DBEPTOJ6LUBIMR5YQJMTBP74KYQIZTZTJTLSKVGMNVULA, seg_pos={0 0}, upload=526359]: Uploading piece... Dec 30 16:02:54.292456880 (600h33m54.163539925s) [noise=true , address=145.79.227.50:4373 , piece_id=REUHJESPRRY5WWEAF3MKKHDI7HDISFUK3ODWWTZYQ2T6FE3EKRAQ, seg_pos={0 0}, upload=526359]: Uploading piece... Dec 30 16:02:54.292772441 (600h33m54.163855486s) [noise=true , address=145.79.227.16:4911 , piece_id=TBQTXCTULL5QQPCUOHP4Z5QUOV7WGFWBNTIBBBCKC2LHPLHME5NQ, seg_pos={0 0}, upload=526359]: Uploading piece... Dec 30 16:02:54.292801046 (600h33m54.163884081s) [noise=true , address=145.79.227.13:4803 , piece_id=TSUN6UX3ZAYSMGDMSLYUDBX4KIMKX3FZZW7IA5FRXZVT7BJDDSEQ, seg_pos={0 0}, upload=526359]: Uploading piece... Dec 30 16:02:54.295223859 (600h33m54.166306894s) [noise=true , address=45.91.116.10:4082 , piece_id=2O4EWILMHWMT6ASGS25W3TAJBNHRVVIC6O4SX5OIGPBHNBIUWJDQ, seg_pos={0 0}, upload=526359]: Uploading piece... Dec 30 16:02:55.895795882 (600h33m55.766878917s) [upload=526359]: Next returned nil segment Dec 30 16:02:55.895815018 (600h33m55.766898063s) [upload=526359]: Waiting for error group managing segments... Dec 30 16:02:56.276129444 (600h33m56.147212509s) [noise=true , address=145.79.227.13:4803 , piece_id=TSUN6UX3ZAYSMGDMSLYUDBX4KIMKX3FZZW7IA5FRXZVT7BJDDSEQ, seg_pos={0 0}, upload=526359]: Done uploading piece. err: <nil> Dec 30 16:02:56.277669617 (600h33m56.148752652s) [noise=true , address=145.79.227.21:4474 , piece_id=RJ2N57AWUFATMQYBPO3RCX6QHW6IO3YIILQA4CKM4YHEQGRACLCQ, seg_pos={0 0}, upload=526359]: Done uploading piece. err: <nil> Dec 30 16:02:56.280043227 (600h33m56.151126272s) [noise=true , address=145.79.227.11:4729 , piece_id=KBIVWPOXWRKXCOD6Z35X7GGADEDCRRDO2PNMN5ZH2RJBHOK2NQ7Q, seg_pos={0 0}, upload=526359]: Done uploading piece. err: <nil> Dec 30 16:02:56.285185849 (600h33m56.156268894s) [noise=true , address=145.79.227.34:4905 , piece_id=VI3ET3QBCZGNWXDNWBU26QYDE62KQMHPU437OWBQTL3ATIIHSVPA, seg_pos={0 0}, upload=526359]: Done uploading piece. err: <nil> Dec 30 16:02:56.291145798 (600h33m56.162228853s) [noise=true , address=145.79.227.20:4438 , piece_id=VYKE7J7TBUXCE74LUIAZ5Z3IHJO6HU7EIQQAO7Z3SBZ7BZ5275QQ, seg_pos={0 0}, upload=526359]: Done uploading piece. err: <nil> Dec 30 16:02:56.320073330 (600h33m56.191156375s) [noise=true , address=145.79.227.36:4517 , piece_id=C7NFGCBBZJRUCG5D6BNJKCRNPCFHES6YEG5S3YQ3AUKAMK5SRLAA, seg_pos={0 0}, upload=526359]: Done uploading piece. err: <nil> Dec 30 16:02:56.356196825 (600h33m56.22727987s) [noise=true , address=145.79.227.16:4911 , piece_id=TBQTXCTULL5QQPCUOHP4Z5QUOV7WGFWBNTIBBBCKC2LHPLHME5NQ, seg_pos={0 0}, upload=526359]: Done uploading piece. err: <nil> Dec 30 16:02:56.366077118 (600h33m56.237160153s) [noise=true , address=145.79.227.50:4373 , piece_id=REUHJESPRRY5WWEAF3MKKHDI7HDISFUK3ODWWTZYQ2T6FE3EKRAQ, seg_pos={0 0}, upload=526359]: Done uploading piece. err: <nil> Dec 30 16:02:56.472393460 (600h33m56.343476495s) [noise=true , address=77.90.31.15:4700 , piece_id=2Y6IC654KNMHEZDZQEB5NH6SW7O6KSJIRAUBOTWXNQDOPTDKV73Q, seg_pos={0 0}, upload=526359]: Done uploading piece. err: <nil> Dec 30 16:02:56.550020283 (600h33m56.421103348s) [noise=true , address=77.90.31.10:4518 , piece_id=MEW3A5VWKABCX5H5VZ6H4XGVMKU5R6A4L34YXHB7FNFZE2IFMGZQ, seg_pos={0 0}, upload=526359]: Done uploading piece. err: <nil> Dec 30 16:02:56.597141219 (600h33m56.468224254s) [noise=true , address=45.91.116.7:4009 , piece_id=YE2VAC3CM5KDPG5K6VGSIXEMS5MESXRCX7G4FM3EJKCSHPJOS22A, seg_pos={0 0}, upload=526359]: Done uploading piece. err: <nil> Dec 30 16:02:56.597187597 (600h33m56.468270632s) [noise=true , address=77.90.31.17:4773 , piece_id=4GYS5FJFICA3VKBF6CYY26V636EXNOK52MENECMEZXFIKELRQKRA, seg_pos={0 0}, upload=526359]: Done uploading piece. err: <nil> Dec 30 16:02:57.244666124 (600h33m57.115749159s) [noise=true , address=77.90.31.16:4736 , piece_id=TD7M7C6DBEPTOJ6LUBIMR5YQJMTBP74KYQIZTZTJTLSKVGMNVULA, seg_pos={0 0}, upload=526359]: Done uploading piece. err: <nil> Dec 30 16:02:57.244690050 (600h33m57.115773085s) [seg_pos={0 0}, upload=526359]: Segment reached optimal threshold of 13 pieces. Dec 30 16:02:57.244892646 (600h33m57.115975681s) [noise=true , address=45.91.116.10:4082 , piece_id=2O4EWILMHWMT6ASGS25W3TAJBNHRVVIC6O4SX5OIGPBHNBIUWJDQ, seg_pos={0 0}, upload=526359]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: write tcp 188.166.225.170:42340->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:42340->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42340->45.91.116.10:4082: use of closed network connection Dec 30 16:02:57.244899669 (600h33m57.115982704s) [noise=true , address=45.91.116.10:4082 , piece_id=2O4EWILMHWMT6ASGS25W3TAJBNHRVVIC6O4SX5OIGPBHNBIUWJDQ, seg_pos={0 0}, upload=526359]: Long tail cancellation due to optimal threshold Dec 30 16:02:57.245125298 (600h33m57.116208333s) [noise=true , address=45.91.116.13:4156 , piece_id=KEI4HNEZAGSZKGZXLI5RMKPR3WUZGZ3UTHDJW2GDJULSELMJKXUA, seg_pos={0 0}, upload=526359]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 30 16:02:57.245154985 (600h33m57.11623802s) [noise=true , address=45.91.116.13:4156 , piece_id=KEI4HNEZAGSZKGZXLI5RMKPR3WUZGZ3UTHDJW2GDJULSELMJKXUA, seg_pos={0 0}, upload=526359]: Long tail cancellation due to optimal threshold Dec 30 16:02:57.245168410 (600h33m57.116251455s) [noise=true , address=145.79.227.29:4943 , piece_id=6AN7DLYBG35QV32DKHEOXTJ73JWHHOVF5CBPCYRHFIW3SKPQKWSA, seg_pos={0 0}, upload=526359]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz): protocol: expected piece hash; context canceled; EOF Dec 30 16:02:57.245175274 (600h33m57.116258309s) [noise=true , address=145.79.227.29:4943 , piece_id=6AN7DLYBG35QV32DKHEOXTJ73JWHHOVF5CBPCYRHFIW3SKPQKWSA, seg_pos={0 0}, upload=526359]: Long tail cancellation due to optimal threshold Dec 30 16:02:57.245191665 (600h33m57.1162747s) [seg_pos={0 0}, upload=526359]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 16:02:57.245257059 (600h33m57.116340094s) [upload=526359]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 16:02:57.245261027 (600h33m57.116344062s) [upload=526359]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 16:02:57.542655802 (600h33m57.413738837s) [upload=526359]: Done uploading segments... Dec 30 16:02:57.542681380 (600h33m57.413764416s) [upload=526359]: Upload finished. err: <nil> Duration: 30.002410507s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 6.51295153s trace: https://jaeger.dev.storj.io/trace/547648a2cfce50e7 FAIL (2025-12-30 16:03:12 UTC) orbiter warning: context deadline exceeded Duration: 6.512982641s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [.........................X..XX..X..................X.......X] duration: 1m0.003478679s trace: https://jaeger.dev.storj.io/trace/697b20119e0593f8 FAIL (2025-12-30 16:02:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 01:57.034542821 (600h32m56.905625866s) [noise=true , address=77.90.31.13:4658 , piece_id=ASWEV2PJC5SIAPGOUC26KDJDG3XLRFW2E6CMV6LL66YIVB7Y5QWQ, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.034697947 (600h32m56.905780992s) [noise=true , address=145.79.227.25:4622 , piece_id=5BSWHUMVRADKEJX3SLEP7BRMUWQKLZQEJE2IDFUXLUQWMBU7PCTA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.035092007 (600h32m56.906175052s) [noise=true , address=77.90.31.15:4729 , piece_id=MF52XXD46K72RFKEULEWW7AOBBVWIIQT2CS7V7TG27L7C5WARCYQ, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.035493051 (600h32m56.906576086s) [noise=true , address=45.91.116.4:4949 , piece_id=K2LRBT6MVGHP3O4MGMTARHMO5FVGFKTMVWYQOR2J7GX4GZNUM6BA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.035882262 (600h32m56.906965307s) [noise=true , address=145.79.227.43:4178 , piece_id=OW36VWZ7KNAZ6JU5VYRP5IHCJZCLFNJZBNXHT2LLNDYKSSJW5FCA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.037716825 (600h32m56.90879987s) [noise=true , address=77.90.31.12:4619 , piece_id=RS3SIHFI6I4RMOTAFYN7NUIIE4WW7E7D3B7RSMCYO4BNN577U3RA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.038358467 (600h32m56.909441502s) [noise=true , address=77.90.31.19:4845 , piece_id=JSPTJOPQ55NKTUAL2HTSEQ4WRCGJNFFGPE7LMUNDMTKEFYZDM2UQ, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.040066269 (600h32m56.911149304s) [noise=true , address=77.90.31.18:4838 , piece_id=URNHRXNV45MVCL6ZWSFY3GQLYJT6U3MPZEIA7M74WF6ZXHQ7H5FA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.042000342 (600h32m56.913083377s) [noise=true , address=5.231.73.4:4028 , piece_id=Y4DYOOAI6GNLUXYDBAYANPWNC3T6QN6U5U64CUO5SXROKWZKVFKQ, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.044053191 (600h32m56.915136226s) [noise=true , address=77.90.31.15:4714 , piece_id=NGX4EVOIPZMCA37RULTAGRVXNI533GKFQJ2ERLFGXDSDZ5H6QHSA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.045847507 (600h32m56.916930552s) [noise=true , address=145.79.227.30:4767 , piece_id=PYUIYB7K2DV5YRAW2GLXCNR4AJ3QQRLWHTYWVH36OPGTVQHTZJUA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.047490265 (600h32m56.91857331s) [noise=true , address=145.79.227.29:4952 , piece_id=R54XH66RF55YEOR62STLKB5YXP6S4KE6NY74VW5J74ZX3YOSAOAA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.049047280 (600h32m56.920130325s) [noise=true , address=145.79.227.44:4197 , piece_id=DLWVGXMXAEU4Y7Q5S2HZQUV3VUHE763FHT26P4W3G4RUN2JLZ6EA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.050106617 (600h32m56.921189652s) [noise=true , address=145.79.227.33:4890 , piece_id=OTPNHUHTHEMYMPKYZNIU3QQ55GXICCYWD7NH4NDA7EHBFUGVBJSQ, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.051582247 (600h32m56.922665292s) [noise=true , address=77.90.31.19:4870 , piece_id=WTCTOZH6665YQ6OW2232VISBWYRUBHL74HML5GJHHJ4PSE2KEYKA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.052071889 (600h32m56.923154935s) [noise=true , address=145.79.227.19:4089 , piece_id=ISNJCL22W46DM5KUOAOOPM2T2VDGV3JM3PQF2MDF73RCF4GTM4SA, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:57.052894146 (600h32m56.923977191s) [noise=true , address=145.79.227.40:4052 , piece_id=EV363HTTZXTV2YEMHZX466LLTK4SBS7RNBTZOWTCHTG2E4OKTPGQ, seg_pos={0 0}, upload=526348]: Uploading piece... Dec 30 16:01:58.511258703 (600h32m58.382341748s) [upload=526348]: Next returned nil segment Dec 30 16:01:58.511280185 (600h32m58.38236322s) [upload=526348]: Waiting for error group managing segments... Dec 30 16:01:58.918592018 (600h32m58.789675063s) [noise=true , address=145.79.227.25:4622 , piece_id=5BSWHUMVRADKEJX3SLEP7BRMUWQKLZQEJE2IDFUXLUQWMBU7PCTA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:58.935469496 (600h32m58.806552531s) [noise=true , address=145.79.227.19:4089 , piece_id=ISNJCL22W46DM5KUOAOOPM2T2VDGV3JM3PQF2MDF73RCF4GTM4SA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.094374364 (600h32m58.965457399s) [noise=true , address=77.90.31.15:4729 , piece_id=MF52XXD46K72RFKEULEWW7AOBBVWIIQT2CS7V7TG27L7C5WARCYQ, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.174591008 (600h32m59.045674043s) [noise=true , address=77.90.31.19:4845 , piece_id=JSPTJOPQ55NKTUAL2HTSEQ4WRCGJNFFGPE7LMUNDMTKEFYZDM2UQ, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.180206382 (600h32m59.051289427s) [noise=true , address=5.231.73.4:4028 , piece_id=Y4DYOOAI6GNLUXYDBAYANPWNC3T6QN6U5U64CUO5SXROKWZKVFKQ, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.247271233 (600h32m59.118354268s) [noise=true , address=45.91.116.7:4014 , piece_id=RVQUWHPEV4XDF7YE4QEX6UKHZEYKE5PHAQ7NH73JB5QCV7PN3KYA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.260497178 (600h32m59.131580213s) [noise=true , address=45.91.116.4:4949 , piece_id=K2LRBT6MVGHP3O4MGMTARHMO5FVGFKTMVWYQOR2J7GX4GZNUM6BA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.401943773 (600h32m59.273026808s) [noise=true , address=77.90.31.13:4658 , piece_id=ASWEV2PJC5SIAPGOUC26KDJDG3XLRFW2E6CMV6LL66YIVB7Y5QWQ, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.508863164 (600h32m59.379946199s) [noise=true , address=77.90.31.19:4870 , piece_id=WTCTOZH6665YQ6OW2232VISBWYRUBHL74HML5GJHHJ4PSE2KEYKA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.743613925 (600h32m59.61469696s) [noise=true , address=77.90.31.15:4714 , piece_id=NGX4EVOIPZMCA37RULTAGRVXNI533GKFQJ2ERLFGXDSDZ5H6QHSA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.812479335 (600h32m59.68356237s) [noise=true , address=77.90.31.18:4838 , piece_id=URNHRXNV45MVCL6ZWSFY3GQLYJT6U3MPZEIA7M74WF6ZXHQ7H5FA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.823406823 (600h32m59.694489868s) [noise=true , address=145.79.227.43:4178 , piece_id=OW36VWZ7KNAZ6JU5VYRP5IHCJZCLFNJZBNXHT2LLNDYKSSJW5FCA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.843550510 (600h32m59.714633545s) [noise=true , address=145.79.227.29:4952 , piece_id=R54XH66RF55YEOR62STLKB5YXP6S4KE6NY74VW5J74ZX3YOSAOAA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.905172958 (600h32m59.776255993s) [noise=true , address=145.79.227.44:4197 , piece_id=DLWVGXMXAEU4Y7Q5S2HZQUV3VUHE763FHT26P4W3G4RUN2JLZ6EA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: <nil> Dec 30 16:01:59.905198386 (600h32m59.776281431s) [seg_pos={0 0}, upload=526348]: Segment reached optimal threshold of 14 pieces. Dec 30 16:01:59.905434836 (600h32m59.776517871s) [noise=true , address=77.90.31.12:4619 , piece_id=RS3SIHFI6I4RMOTAFYN7NUIIE4WW7E7D3B7RSMCYO4BNN577U3RA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX): protocol: expected piece hash; context canceled; EOF Dec 30 16:01:59.905443382 (600h32m59.776526437s) [noise=true , address=77.90.31.12:4619 , piece_id=RS3SIHFI6I4RMOTAFYN7NUIIE4WW7E7D3B7RSMCYO4BNN577U3RA, seg_pos={0 0}, upload=526348]: Long tail cancellation due to optimal threshold Dec 30 16:01:59.905573240 (600h32m59.776656285s) [noise=true , address=145.79.227.33:4890 , piece_id=OTPNHUHTHEMYMPKYZNIU3QQ55GXICCYWD7NH4NDA7EHBFUGVBJSQ, seg_pos={0 0}, upload=526348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XYhMcqH63UiHYE4ofxDxjXz2HMvN1RfQjgxxxHcUst7P3aN7M): protocol: expected piece hash; context canceled; EOF Dec 30 16:01:59.905610260 (600h32m59.776693295s) [noise=true , address=145.79.227.33:4890 , piece_id=OTPNHUHTHEMYMPKYZNIU3QQ55GXICCYWD7NH4NDA7EHBFUGVBJSQ, seg_pos={0 0}, upload=526348]: Long tail cancellation due to optimal threshold Dec 30 16:01:59.905756799 (600h32m59.776839844s) [noise=true , address=145.79.227.40:4052 , piece_id=EV363HTTZXTV2YEMHZX466LLTK4SBS7RNBTZOWTCHTG2E4OKTPGQ, seg_pos={0 0}, upload=526348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PViJBYAanwXRWhMq3WHR6k9Stxy8hjuY5FV9J7ErxCy4N1q3V): protocol: expected piece hash; context canceled; EOF Dec 30 16:01:59.905766168 (600h32m59.776849203s) [noise=true , address=145.79.227.40:4052 , piece_id=EV363HTTZXTV2YEMHZX466LLTK4SBS7RNBTZOWTCHTG2E4OKTPGQ, seg_pos={0 0}, upload=526348]: Long tail cancellation due to optimal threshold Dec 30 16:01:59.905893880 (600h32m59.776976915s) [noise=true , address=145.79.227.30:4767 , piece_id=PYUIYB7K2DV5YRAW2GLXCNR4AJ3QQRLWHTYWVH36OPGTVQHTZJUA, seg_pos={0 0}, upload=526348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sfaYCUhNCAyQAPu1jac5dRWR35Qqc4ZcKg75mp7nVQak295ug): protocol: expected piece hash; context canceled; EOF Dec 30 16:01:59.905901936 (600h32m59.776984971s) [noise=true , address=145.79.227.30:4767 , piece_id=PYUIYB7K2DV5YRAW2GLXCNR4AJ3QQRLWHTYWVH36OPGTVQHTZJUA, seg_pos={0 0}, upload=526348]: Long tail cancellation due to optimal threshold Dec 30 16:01:59.905945239 (600h32m59.777028284s) [seg_pos={0 0}, upload=526348]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 16:01:59.906043476 (600h32m59.777126511s) [upload=526348]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 16:01:59.906048395 (600h32m59.77713143s) [upload=526348]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 16:02:00.203472835 (600h33m0.074555871s) [upload=526348]: Done uploading segments... Dec 30 16:02:00.203498004 (600h33m0.074581049s) [upload=526348]: Upload finished. err: <nil> Duration: 1m0.003492897s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 44 out of 60 is failed (threshold=6) [X.X.X.XXXXXX.XXXX...XX.X.XXXXXXXXX.XXXXXXX...XXXXX.XX..XXXXX] duration: 5.001265149s trace: https://jaeger.dev.storj.io/trace/299d53628bbbc11a FAIL (2025-12-30 16:02:46 UTC) orbiter warning: context deadline exceeded Duration: 5.001277382s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 43 out of 60 is failed (threshold=6) [.X.X.X.XXXXXX.XXXX...XX.X.XXXXXXXXX.XXXXXXX...XXXXX.XX..XXXX] duration: 5.00157163s trace: https://jaeger.dev.storj.io/trace/4f6a5e3fa90c3302 FAIL (2025-12-30 16:02:07 UTC) orbiter warning: context deadline exceeded Duration: 5.001596738s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 6.419563129s trace: https://jaeger.dev.storj.io/trace/547648a2cfce50e7 FAIL (2025-12-30 16:02:05 UTC) orbiter warning: context deadline exceeded Duration: 6.4196476s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [..........................X..XX..X..................X.......] duration: 20.305798219s trace: https://jaeger.dev.storj.io/trace/2ead4e08bdf51a6a FAIL (2025-12-30 16:01:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 43:4696 , piece_id=3WM5R3KUHXN4WVDOKBAOFX7RAI6ZUTV7UKHUOYAKHUYPM4M24N2A, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.239224572 (600h25m38.110307617s) [noise=true , address=45.91.116.12:4241 , piece_id=FYZGK7TFHSQ5BDLEMJIOGXWVWSZ45NRVSYQ4DT6DHBANOCBOZZWA, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.239811830 (600h25m38.110894865s) [noise=true , address=145.79.227.43:4189 , piece_id=DYHLPLLBNG6B3FXERO33ZDDSHFUHQIIYGQZJFE2UMJXMLUBYC3KQ, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.241164706 (600h25m38.112247741s) [noise=true , address=77.90.31.12:4604 , piece_id=OKY6LAK735LFQ4MDC5K6LJPFQITQSGNOZUGNU6US25RELWLK33BQ, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.244508833 (600h25m38.115591878s) [noise=true , address=145.79.227.49:4350 , piece_id=MA66IA2WTN55SFQDQ4LF7H5LNVUK3IMIHHRUH4TXWHWZPTQDBDCA, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.244943361 (600h25m38.116026396s) [noise=true , address=77.90.31.17:4800 , piece_id=K3HNTJHOSIIUBSKISKWG5YRGLKNKHDOWHITXEDDLKPATPIC4IKTQ, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.250127493 (600h25m38.121210528s) [noise=true , address=145.79.227.35:4516 , piece_id=RDI4UFYB6YC22A5QOGDTLRGCG63TSV6XJFKGLKGSJBC5CH3KCQMQ, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.250776008 (600h25m38.121859053s) [noise=true , address=145.79.227.39:4033 , piece_id=BLTCPLYP5MIQO5Y7WWPT5QIDV6QM3YWAFMLOEWFOVWFLZOO67QSA, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.253191467 (600h25m38.124274512s) [noise=true , address=145.79.227.44:4196 , piece_id=ZUDLORSKFXXIGHZ3YNIE3T7JF7OG55NZKPTSFOUVHAISGF7V7J5Q, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.254554973 (600h25m38.125638018s) [noise=true , address=45.91.116.5:4314 , piece_id=XWWPTNVPYYZ6VL26MACQEJTB2FYN6523YJGZK4I5B4OFRJNUMY3A, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.255815333 (600h25m38.126898378s) [noise=true , address=145.79.227.40:4077 , piece_id=AWXM3BGQVCAARS5F5UISWWIFAVWLWV2KML4FAP6BQ6QNXJTG77NA, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.259122731 (600h25m38.130205776s) [noise=true , address=64.49.13.169:4713 , piece_id=KI4XABHC36ADSOANW3HVM3TCWZ4ZXXXNP7MWJVESDKD5WO26DJHQ, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.265517899 (600h25m38.136600944s) [noise=true , address=77.90.31.19:4854 , piece_id=4W27FHSBO3L2TSAZL2JFCTUPHYV7RI5XDQ4SJWOCRCD5V4K4IK2Q, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.266738002 (600h25m38.137821047s) [noise=true , address=145.79.227.33:4877 , piece_id=GPUOFM2E3KZGDCEZQYPKADLJFDML5DS7OTWFW7AFB5DYAHBVBQ6Q, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.267328907 (600h25m38.138411952s) [noise=true , address=77.90.31.21:4911 , piece_id=B25VZGBKAGIB4ZN6OICFQFZTRQWZIIYDBFUWVNAW4QW26AVYSRPA, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:39.692439689 (600h25m39.563522734s) [upload=526249]: Next returned nil segment Dec 30 15:54:39.692451583 (600h25m39.563534618s) [upload=526249]: Waiting for error group managing segments... Dec 30 15:54:40.135912060 (600h25m40.006995095s) [noise=true , address=145.79.227.39:4033 , piece_id=BLTCPLYP5MIQO5Y7WWPT5QIDV6QM3YWAFMLOEWFOVWFLZOO67QSA, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:40.285764442 (600h25m40.156847487s) [noise=true , address=5.231.73.3:4032 , piece_id=7MBVEHMVMT6XTZJZHHWA65RMFNIB7EH4HU63WTLBPPB6CILGMYAA, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:40.400071651 (600h25m40.271154696s) [noise=true , address=77.90.31.12:4604 , piece_id=OKY6LAK735LFQ4MDC5K6LJPFQITQSGNOZUGNU6US25RELWLK33BQ, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:40.423162570 (600h25m40.294245605s) [noise=true , address=45.91.116.12:4241 , piece_id=FYZGK7TFHSQ5BDLEMJIOGXWVWSZ45NRVSYQ4DT6DHBANOCBOZZWA, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:40.428962032 (600h25m40.300045067s) [noise=true , address=77.90.31.17:4800 , piece_id=K3HNTJHOSIIUBSKISKWG5YRGLKNKHDOWHITXEDDLKPATPIC4IKTQ, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:40.543712124 (600h25m40.414795169s) [noise=true , address=45.91.116.5:4314 , piece_id=XWWPTNVPYYZ6VL26MACQEJTB2FYN6523YJGZK4I5B4OFRJNUMY3A, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:40.993222051 (600h25m40.864305086s) [noise=true , address=145.79.227.44:4196 , piece_id=ZUDLORSKFXXIGHZ3YNIE3T7JF7OG55NZKPTSFOUVHAISGF7V7J5Q, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:41.006811979 (600h25m40.877895024s) [noise=true , address=193.149.164.143:4696 , piece_id=3WM5R3KUHXN4WVDOKBAOFX7RAI6ZUTV7UKHUOYAKHUYPM4M24N2A, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:41.024553631 (600h25m40.895636666s) [noise=true , address=145.79.227.40:4077 , piece_id=AWXM3BGQVCAARS5F5UISWWIFAVWLWV2KML4FAP6BQ6QNXJTG77NA, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:41.054002647 (600h25m40.925085692s) [noise=true , address=145.79.227.49:4350 , piece_id=MA66IA2WTN55SFQDQ4LF7H5LNVUK3IMIHHRUH4TXWHWZPTQDBDCA, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:41.074754852 (600h25m40.945837897s) [noise=true , address=77.90.31.21:4911 , piece_id=B25VZGBKAGIB4ZN6OICFQFZTRQWZIIYDBFUWVNAW4QW26AVYSRPA, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:41.076478905 (600h25m40.94756195s) [noise=true , address=145.79.227.35:4516 , piece_id=RDI4UFYB6YC22A5QOGDTLRGCG63TSV6XJFKGLKGSJBC5CH3KCQMQ, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:41.091294145 (600h25m40.96237718s) [noise=true , address=145.79.227.30:4789 , piece_id=5JZFEA3P6EICC2MGUYOQWY4OWOVHXWZLBJBBSVUAMPLV2EHZOZLA, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:41.138722376 (600h25m41.009805421s) [noise=true , address=145.79.227.49:4359 , piece_id=MCTYZULSW2Z2A6HXPREG3GOEYF5QLBPKEB7DDGT32FNTWHC646CQ, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:41.138745481 (600h25m41.009828516s) [seg_pos={0 0}, upload=526249]: Segment reached optimal threshold of 14 pieces. Dec 30 15:54:41.138965780 (600h25m41.010048815s) [noise=true , address=145.79.227.33:4877 , piece_id=GPUOFM2E3KZGDCEZQYPKADLJFDML5DS7OTWFW7AFB5DYAHBVBQ6Q, seg_pos={0 0}, upload=526249]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17TuauVrjKmehRpXqmK7bemU9V33GDZu9RGBs6fanXvJkV1gdH): protocol: expected piece hash; context canceled; EOF Dec 30 15:54:41.138974496 (600h25m41.010057531s) [noise=true , address=145.79.227.33:4877 , piece_id=GPUOFM2E3KZGDCEZQYPKADLJFDML5DS7OTWFW7AFB5DYAHBVBQ6Q, seg_pos={0 0}, upload=526249]: Long tail cancellation due to optimal threshold Dec 30 15:54:41.139084886 (600h25m41.010167921s) [noise=true , address=77.90.31.19:4854 , piece_id=4W27FHSBO3L2TSAZL2JFCTUPHYV7RI5XDQ4SJWOCRCD5V4K4IK2Q, seg_pos={0 0}, upload=526249]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16HwqYhJnhseWEhzzv4QreZ953rB5cKxqFX8swEqaqiRXCRsAE): protocol: write tcp 188.166.225.170:48144->77.90.31.19:4854: use of closed network connection; write tcp 188.166.225.170:48144->77.90.31.19:4854: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48144->77.90.31.19:4854: use of closed network connection Dec 30 15:54:41.139093533 (600h25m41.010176568s) [noise=true , address=77.90.31.19:4854 , piece_id=4W27FHSBO3L2TSAZL2JFCTUPHYV7RI5XDQ4SJWOCRCD5V4K4IK2Q, seg_pos={0 0}, upload=526249]: Long tail cancellation due to optimal threshold Dec 30 15:54:41.139129352 (600h25m41.010212397s) [noise=true , address=145.79.227.43:4189 , piece_id=DYHLPLLBNG6B3FXERO33ZDDSHFUHQIIYGQZJFE2UMJXMLUBYC3KQ, seg_pos={0 0}, upload=526249]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13mB9AoQJubhEJAAVqzd4o4eq8AfNvLkdqHTDAPN3UyXXwE4B7): protocol: expected piece hash; context canceled; EOF Dec 30 15:54:41.139138429 (600h25m41.010221484s) [noise=true , address=145.79.227.43:4189 , piece_id=DYHLPLLBNG6B3FXERO33ZDDSHFUHQIIYGQZJFE2UMJXMLUBYC3KQ, seg_pos={0 0}, upload=526249]: Long tail cancellation due to optimal threshold Dec 30 15:54:41.139238890 (600h25m41.010321925s) [noise=true , address=64.49.13.169:4713 , piece_id=KI4XABHC36ADSOANW3HVM3TCWZ4ZXXXNP7MWJVESDKD5WO26DJHQ, seg_pos={0 0}, upload=526249]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17z3AiUfbxBX2eMDnUnqo1bbgojuQxJro7PsEaPjYzQjHQ3D6K): protocol: write tcp 188.166.225.170:35558->64.49.13.169:4713: use of closed network connection; write tcp 188.166.225.170:35558->64.49.13.169:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35558->64.49.13.169:4713: use of closed network connection Dec 30 15:54:41.139244420 (600h25m41.010327455s) [noise=true , address=64.49.13.169:4713 , piece_id=KI4XABHC36ADSOANW3HVM3TCWZ4ZXXXNP7MWJVESDKD5WO26DJHQ, seg_pos={0 0}, upload=526249]: Long tail cancellation due to optimal threshold Dec 30 15:54:41.139323681 (600h25m41.010406716s) [seg_pos={0 0}, upload=526249]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 15:54:41.139392121 (600h25m41.010475166s) [upload=526249]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 15:54:41.139396971 (600h25m41.010480006s) [upload=526249]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 15:54:41.438011559 (600h25m41.309094604s) [upload=526249]: Done uploading segments... Dec 30 15:54:41.438036206 (600h25m41.309119251s) [upload=526249]: Upload finished. err: <nil> Duration: 20.305825881s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [X..........XX.......XX.XX..X....X.......X......X.X..........] duration: 6.105876632s trace: https://jaeger.dev.storj.io/trace/54fff38414d25b0f FAIL (2025-12-30 16:01:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: AQGHSCVDMOODBX225O5SSOLNLJQYPAY2X7EJ7G5W6RP42ZJ4DGCQ) context deadline exceeded; piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: ZTXP5ESUQMWFEWTQUX2SMER7W2NJG5IISRWFAM3IKAKQFKJOGWSQ) context deadline exceeded ======LOGS====== Dec 30 15:53:33.440030169 (600h24m33.311113204s) [upload=526231]: Starting upload Dec 30 15:53:33.440113437 (600h24m33.311196492s) [upload=526231]: Done starting upload Dec 30 15:53:33.440168081 (600h24m33.311251116s) [upload=526231]: Uploading segments... Dec 30 15:53:33.440357231 (600h24m33.311440276s) [upload=526231]: Got next segment. Inline: false Dec 30 15:53:33.440411114 (600h24m33.311494159s) [upload=526231]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 15:53:33.440416885 (600h24m33.31149992s) [upload=526231]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 15:53:33.684984430 (600h24m33.556067505s) [seg_pos={0 0}, upload=526231]: Begin upload segment... Dec 30 15:53:33.685033994 (600h24m33.556117029s) [seg_pos={0 0}, upload=526231]: Done begin upload segment. Dec 30 15:53:33.685081235 (600h24m33.55616427s) [noise=true , address=145.79.227.36:4519 , piece_id=5JAZZV45ZTNDSTJZIX2EI7RK7MCBXBQUV2PLSF2HBCRFCIJEA6OQ, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.685200161 (600h24m33.556283196s) [noise=true , address=145.79.227.23:4545 , piece_id=NHTKPJFEFGAJOEXUNJJ7TU6N27JJP3O5TFSMWF5I2SMGEYXWEAKQ, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.685709841 (600h24m33.556792887s) [noise=true , address=77.90.31.14:4664 , piece_id=YLXNEMZBXUYTFQ4P6ZFHPA6BIUH4TZ2OBXEQAJWYSCUX333L777A, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.686084574 (600h24m33.557167619s) [noise=true , address=45.91.116.12:4227 , piece_id=5GQIOQC4VDIF3Z6J4TUPZG3XUC3AACZ53CPYB4JIBQD7F5BK2IFA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.686094794 (600h24m33.55717783s) [noise=true , address=77.90.31.10:4517 , piece_id=5Y32J2NPA664L4ZGRDDYYX66FPWND4DNZ5OZXAADJ3LOXK3IA5BA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.686429431 (600h24m33.557512476s) [noise=true , address=45.91.116.13:4153 , piece_id=AQGHSCVDMOODBX225O5SSOLNLJQYPAY2X7EJ7G5W6RP42ZJ4DGCQ, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.686489927 (600h24m33.557572962s) [noise=true , address=45.91.116.8:4047 , piece_id=TLKGZQ2VDRAGBZCULZ35ZR6GEJXQEE2HZ35Q52XBEO5C4FTRFICA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.686910547 (600h24m33.557993582s) [noise=true , address=145.79.227.37:4567 , piece_id=WSHVGPU7L73OET2M6WT7VPHGVRQV6ZCJRPAW7AEXBUQDFWQBNAUA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.686946536 (600h24m33.558029581s) [noise=true , address=145.79.227.14:4838 , piece_id=YB4VPIAQTBJ6G5XFXDVJ3OL6YFAWMEBHHS63SEDX5U4HYKXT3KJQ, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.687266755 (600h24m33.55834979s) [noise=true , address=145.79.227.6:4257 , piece_id=53MG7UDQR3MJYNMQXY3IOPPKOQEOI3KFUTGHQUEHOKCNG5NTAWXA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.687625419 (600h24m33.558708464s) [noise=true , address=45.91.116.11:4120 , piece_id=ZTXP5ESUQMWFEWTQUX2SMER7W2NJG5IISRWFAM3IKAKQFKJOGWSQ, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.688010913 (600h24m33.559093948s) [noise=true , address=145.79.227.19:4077 , piece_id=JEX52J3QTGUWMY3FO6BD5IJV5UPUUCKVV4MYZAOAU3BWMKFSN3KQ, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.688360248 (600h24m33.559443283s) [noise=true , address=145.79.227.16:4910 , piece_id=UHDFUXEQCMN73AUB7WEJBG5HY2G4E25MQGSBNK6AKHMQMDZCJYRA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.688673685 (600h24m33.55975672s) [noise=true , address=145.79.227.48:4303 , piece_id=IO7FUTELAP2L3VVZEQN6N6QG42WMAOS7H5JEPFAST5YEKMKHYLXA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.689057035 (600h24m33.56014007s) [noise=true , address=145.79.227.2:4114 , piece_id=3MJV2XXBY776ZUIO2KU4MEP37SP6P2NYFO255DCI6IFUZWUAGOWQ, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.690890025 (600h24m33.56197308s) [noise=true , address=145.79.227.3:4150 , piece_id=BUBUCPH3LTJO2CR3IVIXEOJOVS422KQT6L2Z22OBZP6DFPCK5KAA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:35.401137462 (600h24m35.272220507s) [upload=526231]: Next returned nil segment Dec 30 15:53:35.401154925 (600h24m35.27223797s) [upload=526231]: Waiting for error group managing segments... Dec 30 15:53:35.673442202 (600h24m35.544525237s) [noise=true , address=145.79.227.23:4545 , piece_id=NHTKPJFEFGAJOEXUNJJ7TU6N27JJP3O5TFSMWF5I2SMGEYXWEAKQ, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.674880822 (600h24m35.545963857s) [noise=true , address=145.79.227.2:4114 , piece_id=3MJV2XXBY776ZUIO2KU4MEP37SP6P2NYFO255DCI6IFUZWUAGOWQ, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.688801769 (600h24m35.559884814s) [noise=true , address=145.79.227.36:4519 , piece_id=5JAZZV45ZTNDSTJZIX2EI7RK7MCBXBQUV2PLSF2HBCRFCIJEA6OQ, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.689704678 (600h24m35.560787713s) [noise=true , address=145.79.227.6:4257 , piece_id=53MG7UDQR3MJYNMQXY3IOPPKOQEOI3KFUTGHQUEHOKCNG5NTAWXA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.690785346 (600h24m35.561868381s) [noise=true , address=145.79.227.14:4838 , piece_id=YB4VPIAQTBJ6G5XFXDVJ3OL6YFAWMEBHHS63SEDX5U4HYKXT3KJQ, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.692882529 (600h24m35.563965564s) [noise=true , address=145.79.227.37:4567 , piece_id=WSHVGPU7L73OET2M6WT7VPHGVRQV6ZCJRPAW7AEXBUQDFWQBNAUA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.696575461 (600h24m35.567658506s) [noise=true , address=145.79.227.16:4910 , piece_id=UHDFUXEQCMN73AUB7WEJBG5HY2G4E25MQGSBNK6AKHMQMDZCJYRA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.697864125 (600h24m35.56894716s) [noise=true , address=145.79.227.19:4077 , piece_id=JEX52J3QTGUWMY3FO6BD5IJV5UPUUCKVV4MYZAOAU3BWMKFSN3KQ, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.758530774 (600h24m35.629613819s) [noise=true , address=145.79.227.48:4303 , piece_id=IO7FUTELAP2L3VVZEQN6N6QG42WMAOS7H5JEPFAST5YEKMKHYLXA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.796785737 (600h24m35.667868772s) [noise=true , address=145.79.227.3:4150 , piece_id=BUBUCPH3LTJO2CR3IVIXEOJOVS422KQT6L2Z22OBZP6DFPCK5KAA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:36.029561278 (600h24m35.900644313s) [noise=true , address=77.90.31.10:4517 , piece_id=5Y32J2NPA664L4ZGRDDYYX66FPWND4DNZ5OZXAADJ3LOXK3IA5BA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:36.070210490 (600h24m35.941293536s) [noise=true , address=45.91.116.13:4153 , piece_id=AQGHSCVDMOODBX225O5SSOLNLJQYPAY2X7EJ7G5W6RP42ZJ4DGCQ, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:36.082591286 (600h24m35.953674331s) [noise=true , address=45.91.116.11:4120 , piece_id=ZTXP5ESUQMWFEWTQUX2SMER7W2NJG5IISRWFAM3IKAKQFKJOGWSQ, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:36.082610784 (600h24m35.953693819s) [seg_pos={0 0}, upload=526231]: Segment reached optimal threshold of 13 pieces. Dec 30 15:53:36.082858675 (600h24m35.953941711s) [noise=true , address=77.90.31.14:4664 , piece_id=YLXNEMZBXUYTFQ4P6ZFHPA6BIUH4TZ2OBXEQAJWYSCUX333L777A, seg_pos={0 0}, upload=526231]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: expected piece hash; context canceled; EOF Dec 30 15:53:36.082868053 (600h24m35.953951088s) [noise=true , address=77.90.31.14:4664 , piece_id=YLXNEMZBXUYTFQ4P6ZFHPA6BIUH4TZ2OBXEQAJWYSCUX333L777A, seg_pos={0 0}, upload=526231]: Long tail cancellation due to optimal threshold Dec 30 15:53:36.082953826 (600h24m35.954036871s) [noise=true , address=45.91.116.12:4227 , piece_id=5GQIOQC4VDIF3Z6J4TUPZG3XUC3AACZ53CPYB4JIBQD7F5BK2IFA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Dec 30 15:53:36.082969977 (600h24m35.954053012s) [noise=true , address=45.91.116.12:4227 , piece_id=5GQIOQC4VDIF3Z6J4TUPZG3XUC3AACZ53CPYB4JIBQD7F5BK2IFA, seg_pos={0 0}, upload=526231]: Long tail cancellation due to optimal threshold Dec 30 15:53:36.083266191 (600h24m35.954349226s) [noise=true , address=45.91.116.8:4047 , piece_id=TLKGZQ2VDRAGBZCULZ35ZR6GEJXQEE2HZ35Q52XBEO5C4FTRFICA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: read tcp 188.166.225.170:42816->45.91.116.8:4047: use of closed network connection; context canceled Dec 30 15:53:36.083274006 (600h24m35.954357041s) [noise=true , address=45.91.116.8:4047 , piece_id=TLKGZQ2VDRAGBZCULZ35ZR6GEJXQEE2HZ35Q52XBEO5C4FTRFICA, seg_pos={0 0}, upload=526231]: Long tail cancellation due to optimal threshold Dec 30 15:53:36.083296048 (600h24m35.954379083s) [seg_pos={0 0}, upload=526231]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 15:53:36.083385679 (600h24m35.954468714s) [upload=526231]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 15:53:36.083392341 (600h24m35.954475376s) [upload=526231]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 15:53:36.379743600 (600h24m36.250826635s) [upload=526231]: Done uploading segments... Dec 30 15:53:36.379758638 (600h24m36.250841673s) [upload=526231]: Upload finished. err: <nil> Duration: 6.105903644s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 43 out of 60 is failed (threshold=6) [X.X.X.X.XXXXXX.XXXX...XX.X.XXXXXXXXX.XXXXXXX...XXXXX.XX..XXX] duration: 5.001540571s trace: https://jaeger.dev.storj.io/trace/4a41aac538ff258c FAIL (2025-12-30 16:01:46 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001555209s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 13.125719807s trace: https://jaeger.dev.storj.io/trace/547648a2cfce50e7 FAIL (2025-12-30 16:01:12 UTC) orbiter warning: context deadline exceeded Duration: 13.125778498s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [...........................X..XX..X..................X......] duration: 39.013921697s trace: https://jaeger.dev.storj.io/trace/2ead4e08bdf51a6a FAIL (2025-12-30 16:01:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 43:4696 , piece_id=3WM5R3KUHXN4WVDOKBAOFX7RAI6ZUTV7UKHUOYAKHUYPM4M24N2A, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.239224572 (600h25m38.110307617s) [noise=true , address=45.91.116.12:4241 , piece_id=FYZGK7TFHSQ5BDLEMJIOGXWVWSZ45NRVSYQ4DT6DHBANOCBOZZWA, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.239811830 (600h25m38.110894865s) [noise=true , address=145.79.227.43:4189 , piece_id=DYHLPLLBNG6B3FXERO33ZDDSHFUHQIIYGQZJFE2UMJXMLUBYC3KQ, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.241164706 (600h25m38.112247741s) [noise=true , address=77.90.31.12:4604 , piece_id=OKY6LAK735LFQ4MDC5K6LJPFQITQSGNOZUGNU6US25RELWLK33BQ, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.244508833 (600h25m38.115591878s) [noise=true , address=145.79.227.49:4350 , piece_id=MA66IA2WTN55SFQDQ4LF7H5LNVUK3IMIHHRUH4TXWHWZPTQDBDCA, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.244943361 (600h25m38.116026396s) [noise=true , address=77.90.31.17:4800 , piece_id=K3HNTJHOSIIUBSKISKWG5YRGLKNKHDOWHITXEDDLKPATPIC4IKTQ, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.250127493 (600h25m38.121210528s) [noise=true , address=145.79.227.35:4516 , piece_id=RDI4UFYB6YC22A5QOGDTLRGCG63TSV6XJFKGLKGSJBC5CH3KCQMQ, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.250776008 (600h25m38.121859053s) [noise=true , address=145.79.227.39:4033 , piece_id=BLTCPLYP5MIQO5Y7WWPT5QIDV6QM3YWAFMLOEWFOVWFLZOO67QSA, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.253191467 (600h25m38.124274512s) [noise=true , address=145.79.227.44:4196 , piece_id=ZUDLORSKFXXIGHZ3YNIE3T7JF7OG55NZKPTSFOUVHAISGF7V7J5Q, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.254554973 (600h25m38.125638018s) [noise=true , address=45.91.116.5:4314 , piece_id=XWWPTNVPYYZ6VL26MACQEJTB2FYN6523YJGZK4I5B4OFRJNUMY3A, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.255815333 (600h25m38.126898378s) [noise=true , address=145.79.227.40:4077 , piece_id=AWXM3BGQVCAARS5F5UISWWIFAVWLWV2KML4FAP6BQ6QNXJTG77NA, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.259122731 (600h25m38.130205776s) [noise=true , address=64.49.13.169:4713 , piece_id=KI4XABHC36ADSOANW3HVM3TCWZ4ZXXXNP7MWJVESDKD5WO26DJHQ, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.265517899 (600h25m38.136600944s) [noise=true , address=77.90.31.19:4854 , piece_id=4W27FHSBO3L2TSAZL2JFCTUPHYV7RI5XDQ4SJWOCRCD5V4K4IK2Q, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.266738002 (600h25m38.137821047s) [noise=true , address=145.79.227.33:4877 , piece_id=GPUOFM2E3KZGDCEZQYPKADLJFDML5DS7OTWFW7AFB5DYAHBVBQ6Q, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.267328907 (600h25m38.138411952s) [noise=true , address=77.90.31.21:4911 , piece_id=B25VZGBKAGIB4ZN6OICFQFZTRQWZIIYDBFUWVNAW4QW26AVYSRPA, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:39.692439689 (600h25m39.563522734s) [upload=526249]: Next returned nil segment Dec 30 15:54:39.692451583 (600h25m39.563534618s) [upload=526249]: Waiting for error group managing segments... Dec 30 15:54:40.135912060 (600h25m40.006995095s) [noise=true , address=145.79.227.39:4033 , piece_id=BLTCPLYP5MIQO5Y7WWPT5QIDV6QM3YWAFMLOEWFOVWFLZOO67QSA, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:40.285764442 (600h25m40.156847487s) [noise=true , address=5.231.73.3:4032 , piece_id=7MBVEHMVMT6XTZJZHHWA65RMFNIB7EH4HU63WTLBPPB6CILGMYAA, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:40.400071651 (600h25m40.271154696s) [noise=true , address=77.90.31.12:4604 , piece_id=OKY6LAK735LFQ4MDC5K6LJPFQITQSGNOZUGNU6US25RELWLK33BQ, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:40.423162570 (600h25m40.294245605s) [noise=true , address=45.91.116.12:4241 , piece_id=FYZGK7TFHSQ5BDLEMJIOGXWVWSZ45NRVSYQ4DT6DHBANOCBOZZWA, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:40.428962032 (600h25m40.300045067s) [noise=true , address=77.90.31.17:4800 , piece_id=K3HNTJHOSIIUBSKISKWG5YRGLKNKHDOWHITXEDDLKPATPIC4IKTQ, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:40.543712124 (600h25m40.414795169s) [noise=true , address=45.91.116.5:4314 , piece_id=XWWPTNVPYYZ6VL26MACQEJTB2FYN6523YJGZK4I5B4OFRJNUMY3A, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:40.993222051 (600h25m40.864305086s) [noise=true , address=145.79.227.44:4196 , piece_id=ZUDLORSKFXXIGHZ3YNIE3T7JF7OG55NZKPTSFOUVHAISGF7V7J5Q, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:41.006811979 (600h25m40.877895024s) [noise=true , address=193.149.164.143:4696 , piece_id=3WM5R3KUHXN4WVDOKBAOFX7RAI6ZUTV7UKHUOYAKHUYPM4M24N2A, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:41.024553631 (600h25m40.895636666s) [noise=true , address=145.79.227.40:4077 , piece_id=AWXM3BGQVCAARS5F5UISWWIFAVWLWV2KML4FAP6BQ6QNXJTG77NA, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:41.054002647 (600h25m40.925085692s) [noise=true , address=145.79.227.49:4350 , piece_id=MA66IA2WTN55SFQDQ4LF7H5LNVUK3IMIHHRUH4TXWHWZPTQDBDCA, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:41.074754852 (600h25m40.945837897s) [noise=true , address=77.90.31.21:4911 , piece_id=B25VZGBKAGIB4ZN6OICFQFZTRQWZIIYDBFUWVNAW4QW26AVYSRPA, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:41.076478905 (600h25m40.94756195s) [noise=true , address=145.79.227.35:4516 , piece_id=RDI4UFYB6YC22A5QOGDTLRGCG63TSV6XJFKGLKGSJBC5CH3KCQMQ, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:41.091294145 (600h25m40.96237718s) [noise=true , address=145.79.227.30:4789 , piece_id=5JZFEA3P6EICC2MGUYOQWY4OWOVHXWZLBJBBSVUAMPLV2EHZOZLA, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:41.138722376 (600h25m41.009805421s) [noise=true , address=145.79.227.49:4359 , piece_id=MCTYZULSW2Z2A6HXPREG3GOEYF5QLBPKEB7DDGT32FNTWHC646CQ, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:41.138745481 (600h25m41.009828516s) [seg_pos={0 0}, upload=526249]: Segment reached optimal threshold of 14 pieces. Dec 30 15:54:41.138965780 (600h25m41.010048815s) [noise=true , address=145.79.227.33:4877 , piece_id=GPUOFM2E3KZGDCEZQYPKADLJFDML5DS7OTWFW7AFB5DYAHBVBQ6Q, seg_pos={0 0}, upload=526249]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17TuauVrjKmehRpXqmK7bemU9V33GDZu9RGBs6fanXvJkV1gdH): protocol: expected piece hash; context canceled; EOF Dec 30 15:54:41.138974496 (600h25m41.010057531s) [noise=true , address=145.79.227.33:4877 , piece_id=GPUOFM2E3KZGDCEZQYPKADLJFDML5DS7OTWFW7AFB5DYAHBVBQ6Q, seg_pos={0 0}, upload=526249]: Long tail cancellation due to optimal threshold Dec 30 15:54:41.139084886 (600h25m41.010167921s) [noise=true , address=77.90.31.19:4854 , piece_id=4W27FHSBO3L2TSAZL2JFCTUPHYV7RI5XDQ4SJWOCRCD5V4K4IK2Q, seg_pos={0 0}, upload=526249]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16HwqYhJnhseWEhzzv4QreZ953rB5cKxqFX8swEqaqiRXCRsAE): protocol: write tcp 188.166.225.170:48144->77.90.31.19:4854: use of closed network connection; write tcp 188.166.225.170:48144->77.90.31.19:4854: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48144->77.90.31.19:4854: use of closed network connection Dec 30 15:54:41.139093533 (600h25m41.010176568s) [noise=true , address=77.90.31.19:4854 , piece_id=4W27FHSBO3L2TSAZL2JFCTUPHYV7RI5XDQ4SJWOCRCD5V4K4IK2Q, seg_pos={0 0}, upload=526249]: Long tail cancellation due to optimal threshold Dec 30 15:54:41.139129352 (600h25m41.010212397s) [noise=true , address=145.79.227.43:4189 , piece_id=DYHLPLLBNG6B3FXERO33ZDDSHFUHQIIYGQZJFE2UMJXMLUBYC3KQ, seg_pos={0 0}, upload=526249]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13mB9AoQJubhEJAAVqzd4o4eq8AfNvLkdqHTDAPN3UyXXwE4B7): protocol: expected piece hash; context canceled; EOF Dec 30 15:54:41.139138429 (600h25m41.010221484s) [noise=true , address=145.79.227.43:4189 , piece_id=DYHLPLLBNG6B3FXERO33ZDDSHFUHQIIYGQZJFE2UMJXMLUBYC3KQ, seg_pos={0 0}, upload=526249]: Long tail cancellation due to optimal threshold Dec 30 15:54:41.139238890 (600h25m41.010321925s) [noise=true , address=64.49.13.169:4713 , piece_id=KI4XABHC36ADSOANW3HVM3TCWZ4ZXXXNP7MWJVESDKD5WO26DJHQ, seg_pos={0 0}, upload=526249]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17z3AiUfbxBX2eMDnUnqo1bbgojuQxJro7PsEaPjYzQjHQ3D6K): protocol: write tcp 188.166.225.170:35558->64.49.13.169:4713: use of closed network connection; write tcp 188.166.225.170:35558->64.49.13.169:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35558->64.49.13.169:4713: use of closed network connection Dec 30 15:54:41.139244420 (600h25m41.010327455s) [noise=true , address=64.49.13.169:4713 , piece_id=KI4XABHC36ADSOANW3HVM3TCWZ4ZXXXNP7MWJVESDKD5WO26DJHQ, seg_pos={0 0}, upload=526249]: Long tail cancellation due to optimal threshold Dec 30 15:54:41.139323681 (600h25m41.010406716s) [seg_pos={0 0}, upload=526249]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 15:54:41.139392121 (600h25m41.010475166s) [upload=526249]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 15:54:41.139396971 (600h25m41.010480006s) [upload=526249]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 15:54:41.438011559 (600h25m41.309094604s) [upload=526249]: Done uploading segments... Dec 30 15:54:41.438036206 (600h25m41.309119251s) [upload=526249]: Upload finished. err: <nil> Duration: 39.013968266s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 43 out of 60 is failed (threshold=6) [XX.X.X.X.XXXXXX.XXXX...XX.X.XXXXXXXXX.XXXXXXX...XXXXX.XX..XX] duration: 5.001019222s trace: https://jaeger.dev.storj.io/trace/20165a93b540b0de FAIL (2025-12-30 16:00:49 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001036925s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.X..........XX.......XX.XX..X....X.......X......X.X.........] duration: 6.158000618s trace: https://jaeger.dev.storj.io/trace/54fff38414d25b0f FAIL (2025-12-30 16:00:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: AQGHSCVDMOODBX225O5SSOLNLJQYPAY2X7EJ7G5W6RP42ZJ4DGCQ) context deadline exceeded; piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: ZTXP5ESUQMWFEWTQUX2SMER7W2NJG5IISRWFAM3IKAKQFKJOGWSQ) context deadline exceeded ======LOGS====== Dec 30 15:53:33.440030169 (600h24m33.311113204s) [upload=526231]: Starting upload Dec 30 15:53:33.440113437 (600h24m33.311196492s) [upload=526231]: Done starting upload Dec 30 15:53:33.440168081 (600h24m33.311251116s) [upload=526231]: Uploading segments... Dec 30 15:53:33.440357231 (600h24m33.311440276s) [upload=526231]: Got next segment. Inline: false Dec 30 15:53:33.440411114 (600h24m33.311494159s) [upload=526231]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 15:53:33.440416885 (600h24m33.31149992s) [upload=526231]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 15:53:33.684984430 (600h24m33.556067505s) [seg_pos={0 0}, upload=526231]: Begin upload segment... Dec 30 15:53:33.685033994 (600h24m33.556117029s) [seg_pos={0 0}, upload=526231]: Done begin upload segment. Dec 30 15:53:33.685081235 (600h24m33.55616427s) [noise=true , address=145.79.227.36:4519 , piece_id=5JAZZV45ZTNDSTJZIX2EI7RK7MCBXBQUV2PLSF2HBCRFCIJEA6OQ, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.685200161 (600h24m33.556283196s) [noise=true , address=145.79.227.23:4545 , piece_id=NHTKPJFEFGAJOEXUNJJ7TU6N27JJP3O5TFSMWF5I2SMGEYXWEAKQ, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.685709841 (600h24m33.556792887s) [noise=true , address=77.90.31.14:4664 , piece_id=YLXNEMZBXUYTFQ4P6ZFHPA6BIUH4TZ2OBXEQAJWYSCUX333L777A, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.686084574 (600h24m33.557167619s) [noise=true , address=45.91.116.12:4227 , piece_id=5GQIOQC4VDIF3Z6J4TUPZG3XUC3AACZ53CPYB4JIBQD7F5BK2IFA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.686094794 (600h24m33.55717783s) [noise=true , address=77.90.31.10:4517 , piece_id=5Y32J2NPA664L4ZGRDDYYX66FPWND4DNZ5OZXAADJ3LOXK3IA5BA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.686429431 (600h24m33.557512476s) [noise=true , address=45.91.116.13:4153 , piece_id=AQGHSCVDMOODBX225O5SSOLNLJQYPAY2X7EJ7G5W6RP42ZJ4DGCQ, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.686489927 (600h24m33.557572962s) [noise=true , address=45.91.116.8:4047 , piece_id=TLKGZQ2VDRAGBZCULZ35ZR6GEJXQEE2HZ35Q52XBEO5C4FTRFICA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.686910547 (600h24m33.557993582s) [noise=true , address=145.79.227.37:4567 , piece_id=WSHVGPU7L73OET2M6WT7VPHGVRQV6ZCJRPAW7AEXBUQDFWQBNAUA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.686946536 (600h24m33.558029581s) [noise=true , address=145.79.227.14:4838 , piece_id=YB4VPIAQTBJ6G5XFXDVJ3OL6YFAWMEBHHS63SEDX5U4HYKXT3KJQ, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.687266755 (600h24m33.55834979s) [noise=true , address=145.79.227.6:4257 , piece_id=53MG7UDQR3MJYNMQXY3IOPPKOQEOI3KFUTGHQUEHOKCNG5NTAWXA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.687625419 (600h24m33.558708464s) [noise=true , address=45.91.116.11:4120 , piece_id=ZTXP5ESUQMWFEWTQUX2SMER7W2NJG5IISRWFAM3IKAKQFKJOGWSQ, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.688010913 (600h24m33.559093948s) [noise=true , address=145.79.227.19:4077 , piece_id=JEX52J3QTGUWMY3FO6BD5IJV5UPUUCKVV4MYZAOAU3BWMKFSN3KQ, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.688360248 (600h24m33.559443283s) [noise=true , address=145.79.227.16:4910 , piece_id=UHDFUXEQCMN73AUB7WEJBG5HY2G4E25MQGSBNK6AKHMQMDZCJYRA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.688673685 (600h24m33.55975672s) [noise=true , address=145.79.227.48:4303 , piece_id=IO7FUTELAP2L3VVZEQN6N6QG42WMAOS7H5JEPFAST5YEKMKHYLXA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.689057035 (600h24m33.56014007s) [noise=true , address=145.79.227.2:4114 , piece_id=3MJV2XXBY776ZUIO2KU4MEP37SP6P2NYFO255DCI6IFUZWUAGOWQ, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.690890025 (600h24m33.56197308s) [noise=true , address=145.79.227.3:4150 , piece_id=BUBUCPH3LTJO2CR3IVIXEOJOVS422KQT6L2Z22OBZP6DFPCK5KAA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:35.401137462 (600h24m35.272220507s) [upload=526231]: Next returned nil segment Dec 30 15:53:35.401154925 (600h24m35.27223797s) [upload=526231]: Waiting for error group managing segments... Dec 30 15:53:35.673442202 (600h24m35.544525237s) [noise=true , address=145.79.227.23:4545 , piece_id=NHTKPJFEFGAJOEXUNJJ7TU6N27JJP3O5TFSMWF5I2SMGEYXWEAKQ, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.674880822 (600h24m35.545963857s) [noise=true , address=145.79.227.2:4114 , piece_id=3MJV2XXBY776ZUIO2KU4MEP37SP6P2NYFO255DCI6IFUZWUAGOWQ, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.688801769 (600h24m35.559884814s) [noise=true , address=145.79.227.36:4519 , piece_id=5JAZZV45ZTNDSTJZIX2EI7RK7MCBXBQUV2PLSF2HBCRFCIJEA6OQ, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.689704678 (600h24m35.560787713s) [noise=true , address=145.79.227.6:4257 , piece_id=53MG7UDQR3MJYNMQXY3IOPPKOQEOI3KFUTGHQUEHOKCNG5NTAWXA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.690785346 (600h24m35.561868381s) [noise=true , address=145.79.227.14:4838 , piece_id=YB4VPIAQTBJ6G5XFXDVJ3OL6YFAWMEBHHS63SEDX5U4HYKXT3KJQ, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.692882529 (600h24m35.563965564s) [noise=true , address=145.79.227.37:4567 , piece_id=WSHVGPU7L73OET2M6WT7VPHGVRQV6ZCJRPAW7AEXBUQDFWQBNAUA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.696575461 (600h24m35.567658506s) [noise=true , address=145.79.227.16:4910 , piece_id=UHDFUXEQCMN73AUB7WEJBG5HY2G4E25MQGSBNK6AKHMQMDZCJYRA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.697864125 (600h24m35.56894716s) [noise=true , address=145.79.227.19:4077 , piece_id=JEX52J3QTGUWMY3FO6BD5IJV5UPUUCKVV4MYZAOAU3BWMKFSN3KQ, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.758530774 (600h24m35.629613819s) [noise=true , address=145.79.227.48:4303 , piece_id=IO7FUTELAP2L3VVZEQN6N6QG42WMAOS7H5JEPFAST5YEKMKHYLXA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.796785737 (600h24m35.667868772s) [noise=true , address=145.79.227.3:4150 , piece_id=BUBUCPH3LTJO2CR3IVIXEOJOVS422KQT6L2Z22OBZP6DFPCK5KAA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:36.029561278 (600h24m35.900644313s) [noise=true , address=77.90.31.10:4517 , piece_id=5Y32J2NPA664L4ZGRDDYYX66FPWND4DNZ5OZXAADJ3LOXK3IA5BA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:36.070210490 (600h24m35.941293536s) [noise=true , address=45.91.116.13:4153 , piece_id=AQGHSCVDMOODBX225O5SSOLNLJQYPAY2X7EJ7G5W6RP42ZJ4DGCQ, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:36.082591286 (600h24m35.953674331s) [noise=true , address=45.91.116.11:4120 , piece_id=ZTXP5ESUQMWFEWTQUX2SMER7W2NJG5IISRWFAM3IKAKQFKJOGWSQ, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:36.082610784 (600h24m35.953693819s) [seg_pos={0 0}, upload=526231]: Segment reached optimal threshold of 13 pieces. Dec 30 15:53:36.082858675 (600h24m35.953941711s) [noise=true , address=77.90.31.14:4664 , piece_id=YLXNEMZBXUYTFQ4P6ZFHPA6BIUH4TZ2OBXEQAJWYSCUX333L777A, seg_pos={0 0}, upload=526231]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: expected piece hash; context canceled; EOF Dec 30 15:53:36.082868053 (600h24m35.953951088s) [noise=true , address=77.90.31.14:4664 , piece_id=YLXNEMZBXUYTFQ4P6ZFHPA6BIUH4TZ2OBXEQAJWYSCUX333L777A, seg_pos={0 0}, upload=526231]: Long tail cancellation due to optimal threshold Dec 30 15:53:36.082953826 (600h24m35.954036871s) [noise=true , address=45.91.116.12:4227 , piece_id=5GQIOQC4VDIF3Z6J4TUPZG3XUC3AACZ53CPYB4JIBQD7F5BK2IFA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Dec 30 15:53:36.082969977 (600h24m35.954053012s) [noise=true , address=45.91.116.12:4227 , piece_id=5GQIOQC4VDIF3Z6J4TUPZG3XUC3AACZ53CPYB4JIBQD7F5BK2IFA, seg_pos={0 0}, upload=526231]: Long tail cancellation due to optimal threshold Dec 30 15:53:36.083266191 (600h24m35.954349226s) [noise=true , address=45.91.116.8:4047 , piece_id=TLKGZQ2VDRAGBZCULZ35ZR6GEJXQEE2HZ35Q52XBEO5C4FTRFICA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: read tcp 188.166.225.170:42816->45.91.116.8:4047: use of closed network connection; context canceled Dec 30 15:53:36.083274006 (600h24m35.954357041s) [noise=true , address=45.91.116.8:4047 , piece_id=TLKGZQ2VDRAGBZCULZ35ZR6GEJXQEE2HZ35Q52XBEO5C4FTRFICA, seg_pos={0 0}, upload=526231]: Long tail cancellation due to optimal threshold Dec 30 15:53:36.083296048 (600h24m35.954379083s) [seg_pos={0 0}, upload=526231]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 15:53:36.083385679 (600h24m35.954468714s) [upload=526231]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 15:53:36.083392341 (600h24m35.954475376s) [upload=526231]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 15:53:36.379743600 (600h24m36.250826635s) [upload=526231]: Done uploading segments... Dec 30 15:53:36.379758638 (600h24m36.250841673s) [upload=526231]: Upload finished. err: <nil> Duration: 6.158035363s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [X.X..........XX.......XX.XX..X....X.......X......X.X........] duration: 5.85659312s trace: https://jaeger.dev.storj.io/trace/54fff38414d25b0f FAIL (2025-12-30 16:00:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: AQGHSCVDMOODBX225O5SSOLNLJQYPAY2X7EJ7G5W6RP42ZJ4DGCQ) context deadline exceeded; piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: ZTXP5ESUQMWFEWTQUX2SMER7W2NJG5IISRWFAM3IKAKQFKJOGWSQ) context deadline exceeded ======LOGS====== Dec 30 15:53:33.440030169 (600h24m33.311113204s) [upload=526231]: Starting upload Dec 30 15:53:33.440113437 (600h24m33.311196492s) [upload=526231]: Done starting upload Dec 30 15:53:33.440168081 (600h24m33.311251116s) [upload=526231]: Uploading segments... Dec 30 15:53:33.440357231 (600h24m33.311440276s) [upload=526231]: Got next segment. Inline: false Dec 30 15:53:33.440411114 (600h24m33.311494159s) [upload=526231]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 15:53:33.440416885 (600h24m33.31149992s) [upload=526231]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 15:53:33.684984430 (600h24m33.556067505s) [seg_pos={0 0}, upload=526231]: Begin upload segment... Dec 30 15:53:33.685033994 (600h24m33.556117029s) [seg_pos={0 0}, upload=526231]: Done begin upload segment. Dec 30 15:53:33.685081235 (600h24m33.55616427s) [noise=true , address=145.79.227.36:4519 , piece_id=5JAZZV45ZTNDSTJZIX2EI7RK7MCBXBQUV2PLSF2HBCRFCIJEA6OQ, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.685200161 (600h24m33.556283196s) [noise=true , address=145.79.227.23:4545 , piece_id=NHTKPJFEFGAJOEXUNJJ7TU6N27JJP3O5TFSMWF5I2SMGEYXWEAKQ, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.685709841 (600h24m33.556792887s) [noise=true , address=77.90.31.14:4664 , piece_id=YLXNEMZBXUYTFQ4P6ZFHPA6BIUH4TZ2OBXEQAJWYSCUX333L777A, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.686084574 (600h24m33.557167619s) [noise=true , address=45.91.116.12:4227 , piece_id=5GQIOQC4VDIF3Z6J4TUPZG3XUC3AACZ53CPYB4JIBQD7F5BK2IFA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.686094794 (600h24m33.55717783s) [noise=true , address=77.90.31.10:4517 , piece_id=5Y32J2NPA664L4ZGRDDYYX66FPWND4DNZ5OZXAADJ3LOXK3IA5BA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.686429431 (600h24m33.557512476s) [noise=true , address=45.91.116.13:4153 , piece_id=AQGHSCVDMOODBX225O5SSOLNLJQYPAY2X7EJ7G5W6RP42ZJ4DGCQ, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.686489927 (600h24m33.557572962s) [noise=true , address=45.91.116.8:4047 , piece_id=TLKGZQ2VDRAGBZCULZ35ZR6GEJXQEE2HZ35Q52XBEO5C4FTRFICA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.686910547 (600h24m33.557993582s) [noise=true , address=145.79.227.37:4567 , piece_id=WSHVGPU7L73OET2M6WT7VPHGVRQV6ZCJRPAW7AEXBUQDFWQBNAUA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.686946536 (600h24m33.558029581s) [noise=true , address=145.79.227.14:4838 , piece_id=YB4VPIAQTBJ6G5XFXDVJ3OL6YFAWMEBHHS63SEDX5U4HYKXT3KJQ, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.687266755 (600h24m33.55834979s) [noise=true , address=145.79.227.6:4257 , piece_id=53MG7UDQR3MJYNMQXY3IOPPKOQEOI3KFUTGHQUEHOKCNG5NTAWXA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.687625419 (600h24m33.558708464s) [noise=true , address=45.91.116.11:4120 , piece_id=ZTXP5ESUQMWFEWTQUX2SMER7W2NJG5IISRWFAM3IKAKQFKJOGWSQ, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.688010913 (600h24m33.559093948s) [noise=true , address=145.79.227.19:4077 , piece_id=JEX52J3QTGUWMY3FO6BD5IJV5UPUUCKVV4MYZAOAU3BWMKFSN3KQ, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.688360248 (600h24m33.559443283s) [noise=true , address=145.79.227.16:4910 , piece_id=UHDFUXEQCMN73AUB7WEJBG5HY2G4E25MQGSBNK6AKHMQMDZCJYRA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.688673685 (600h24m33.55975672s) [noise=true , address=145.79.227.48:4303 , piece_id=IO7FUTELAP2L3VVZEQN6N6QG42WMAOS7H5JEPFAST5YEKMKHYLXA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.689057035 (600h24m33.56014007s) [noise=true , address=145.79.227.2:4114 , piece_id=3MJV2XXBY776ZUIO2KU4MEP37SP6P2NYFO255DCI6IFUZWUAGOWQ, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.690890025 (600h24m33.56197308s) [noise=true , address=145.79.227.3:4150 , piece_id=BUBUCPH3LTJO2CR3IVIXEOJOVS422KQT6L2Z22OBZP6DFPCK5KAA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:35.401137462 (600h24m35.272220507s) [upload=526231]: Next returned nil segment Dec 30 15:53:35.401154925 (600h24m35.27223797s) [upload=526231]: Waiting for error group managing segments... Dec 30 15:53:35.673442202 (600h24m35.544525237s) [noise=true , address=145.79.227.23:4545 , piece_id=NHTKPJFEFGAJOEXUNJJ7TU6N27JJP3O5TFSMWF5I2SMGEYXWEAKQ, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.674880822 (600h24m35.545963857s) [noise=true , address=145.79.227.2:4114 , piece_id=3MJV2XXBY776ZUIO2KU4MEP37SP6P2NYFO255DCI6IFUZWUAGOWQ, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.688801769 (600h24m35.559884814s) [noise=true , address=145.79.227.36:4519 , piece_id=5JAZZV45ZTNDSTJZIX2EI7RK7MCBXBQUV2PLSF2HBCRFCIJEA6OQ, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.689704678 (600h24m35.560787713s) [noise=true , address=145.79.227.6:4257 , piece_id=53MG7UDQR3MJYNMQXY3IOPPKOQEOI3KFUTGHQUEHOKCNG5NTAWXA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.690785346 (600h24m35.561868381s) [noise=true , address=145.79.227.14:4838 , piece_id=YB4VPIAQTBJ6G5XFXDVJ3OL6YFAWMEBHHS63SEDX5U4HYKXT3KJQ, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.692882529 (600h24m35.563965564s) [noise=true , address=145.79.227.37:4567 , piece_id=WSHVGPU7L73OET2M6WT7VPHGVRQV6ZCJRPAW7AEXBUQDFWQBNAUA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.696575461 (600h24m35.567658506s) [noise=true , address=145.79.227.16:4910 , piece_id=UHDFUXEQCMN73AUB7WEJBG5HY2G4E25MQGSBNK6AKHMQMDZCJYRA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.697864125 (600h24m35.56894716s) [noise=true , address=145.79.227.19:4077 , piece_id=JEX52J3QTGUWMY3FO6BD5IJV5UPUUCKVV4MYZAOAU3BWMKFSN3KQ, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.758530774 (600h24m35.629613819s) [noise=true , address=145.79.227.48:4303 , piece_id=IO7FUTELAP2L3VVZEQN6N6QG42WMAOS7H5JEPFAST5YEKMKHYLXA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.796785737 (600h24m35.667868772s) [noise=true , address=145.79.227.3:4150 , piece_id=BUBUCPH3LTJO2CR3IVIXEOJOVS422KQT6L2Z22OBZP6DFPCK5KAA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:36.029561278 (600h24m35.900644313s) [noise=true , address=77.90.31.10:4517 , piece_id=5Y32J2NPA664L4ZGRDDYYX66FPWND4DNZ5OZXAADJ3LOXK3IA5BA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:36.070210490 (600h24m35.941293536s) [noise=true , address=45.91.116.13:4153 , piece_id=AQGHSCVDMOODBX225O5SSOLNLJQYPAY2X7EJ7G5W6RP42ZJ4DGCQ, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:36.082591286 (600h24m35.953674331s) [noise=true , address=45.91.116.11:4120 , piece_id=ZTXP5ESUQMWFEWTQUX2SMER7W2NJG5IISRWFAM3IKAKQFKJOGWSQ, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:36.082610784 (600h24m35.953693819s) [seg_pos={0 0}, upload=526231]: Segment reached optimal threshold of 13 pieces. Dec 30 15:53:36.082858675 (600h24m35.953941711s) [noise=true , address=77.90.31.14:4664 , piece_id=YLXNEMZBXUYTFQ4P6ZFHPA6BIUH4TZ2OBXEQAJWYSCUX333L777A, seg_pos={0 0}, upload=526231]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: expected piece hash; context canceled; EOF Dec 30 15:53:36.082868053 (600h24m35.953951088s) [noise=true , address=77.90.31.14:4664 , piece_id=YLXNEMZBXUYTFQ4P6ZFHPA6BIUH4TZ2OBXEQAJWYSCUX333L777A, seg_pos={0 0}, upload=526231]: Long tail cancellation due to optimal threshold Dec 30 15:53:36.082953826 (600h24m35.954036871s) [noise=true , address=45.91.116.12:4227 , piece_id=5GQIOQC4VDIF3Z6J4TUPZG3XUC3AACZ53CPYB4JIBQD7F5BK2IFA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Dec 30 15:53:36.082969977 (600h24m35.954053012s) [noise=true , address=45.91.116.12:4227 , piece_id=5GQIOQC4VDIF3Z6J4TUPZG3XUC3AACZ53CPYB4JIBQD7F5BK2IFA, seg_pos={0 0}, upload=526231]: Long tail cancellation due to optimal threshold Dec 30 15:53:36.083266191 (600h24m35.954349226s) [noise=true , address=45.91.116.8:4047 , piece_id=TLKGZQ2VDRAGBZCULZ35ZR6GEJXQEE2HZ35Q52XBEO5C4FTRFICA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: read tcp 188.166.225.170:42816->45.91.116.8:4047: use of closed network connection; context canceled Dec 30 15:53:36.083274006 (600h24m35.954357041s) [noise=true , address=45.91.116.8:4047 , piece_id=TLKGZQ2VDRAGBZCULZ35ZR6GEJXQEE2HZ35Q52XBEO5C4FTRFICA, seg_pos={0 0}, upload=526231]: Long tail cancellation due to optimal threshold Dec 30 15:53:36.083296048 (600h24m35.954379083s) [seg_pos={0 0}, upload=526231]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 15:53:36.083385679 (600h24m35.954468714s) [upload=526231]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 15:53:36.083392341 (600h24m35.954475376s) [upload=526231]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 15:53:36.379743600 (600h24m36.250826635s) [upload=526231]: Done uploading segments... Dec 30 15:53:36.379758638 (600h24m36.250841673s) [upload=526231]: Upload finished. err: <nil> Duration: 5.856628076s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [............................X..XX..X..................X.....] duration: 47.31526429s trace: https://jaeger.dev.storj.io/trace/2ead4e08bdf51a6a FAIL (2025-12-30 16:00:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 43:4696 , piece_id=3WM5R3KUHXN4WVDOKBAOFX7RAI6ZUTV7UKHUOYAKHUYPM4M24N2A, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.239224572 (600h25m38.110307617s) [noise=true , address=45.91.116.12:4241 , piece_id=FYZGK7TFHSQ5BDLEMJIOGXWVWSZ45NRVSYQ4DT6DHBANOCBOZZWA, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.239811830 (600h25m38.110894865s) [noise=true , address=145.79.227.43:4189 , piece_id=DYHLPLLBNG6B3FXERO33ZDDSHFUHQIIYGQZJFE2UMJXMLUBYC3KQ, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.241164706 (600h25m38.112247741s) [noise=true , address=77.90.31.12:4604 , piece_id=OKY6LAK735LFQ4MDC5K6LJPFQITQSGNOZUGNU6US25RELWLK33BQ, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.244508833 (600h25m38.115591878s) [noise=true , address=145.79.227.49:4350 , piece_id=MA66IA2WTN55SFQDQ4LF7H5LNVUK3IMIHHRUH4TXWHWZPTQDBDCA, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.244943361 (600h25m38.116026396s) [noise=true , address=77.90.31.17:4800 , piece_id=K3HNTJHOSIIUBSKISKWG5YRGLKNKHDOWHITXEDDLKPATPIC4IKTQ, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.250127493 (600h25m38.121210528s) [noise=true , address=145.79.227.35:4516 , piece_id=RDI4UFYB6YC22A5QOGDTLRGCG63TSV6XJFKGLKGSJBC5CH3KCQMQ, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.250776008 (600h25m38.121859053s) [noise=true , address=145.79.227.39:4033 , piece_id=BLTCPLYP5MIQO5Y7WWPT5QIDV6QM3YWAFMLOEWFOVWFLZOO67QSA, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.253191467 (600h25m38.124274512s) [noise=true , address=145.79.227.44:4196 , piece_id=ZUDLORSKFXXIGHZ3YNIE3T7JF7OG55NZKPTSFOUVHAISGF7V7J5Q, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.254554973 (600h25m38.125638018s) [noise=true , address=45.91.116.5:4314 , piece_id=XWWPTNVPYYZ6VL26MACQEJTB2FYN6523YJGZK4I5B4OFRJNUMY3A, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.255815333 (600h25m38.126898378s) [noise=true , address=145.79.227.40:4077 , piece_id=AWXM3BGQVCAARS5F5UISWWIFAVWLWV2KML4FAP6BQ6QNXJTG77NA, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.259122731 (600h25m38.130205776s) [noise=true , address=64.49.13.169:4713 , piece_id=KI4XABHC36ADSOANW3HVM3TCWZ4ZXXXNP7MWJVESDKD5WO26DJHQ, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.265517899 (600h25m38.136600944s) [noise=true , address=77.90.31.19:4854 , piece_id=4W27FHSBO3L2TSAZL2JFCTUPHYV7RI5XDQ4SJWOCRCD5V4K4IK2Q, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.266738002 (600h25m38.137821047s) [noise=true , address=145.79.227.33:4877 , piece_id=GPUOFM2E3KZGDCEZQYPKADLJFDML5DS7OTWFW7AFB5DYAHBVBQ6Q, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.267328907 (600h25m38.138411952s) [noise=true , address=77.90.31.21:4911 , piece_id=B25VZGBKAGIB4ZN6OICFQFZTRQWZIIYDBFUWVNAW4QW26AVYSRPA, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:39.692439689 (600h25m39.563522734s) [upload=526249]: Next returned nil segment Dec 30 15:54:39.692451583 (600h25m39.563534618s) [upload=526249]: Waiting for error group managing segments... Dec 30 15:54:40.135912060 (600h25m40.006995095s) [noise=true , address=145.79.227.39:4033 , piece_id=BLTCPLYP5MIQO5Y7WWPT5QIDV6QM3YWAFMLOEWFOVWFLZOO67QSA, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:40.285764442 (600h25m40.156847487s) [noise=true , address=5.231.73.3:4032 , piece_id=7MBVEHMVMT6XTZJZHHWA65RMFNIB7EH4HU63WTLBPPB6CILGMYAA, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:40.400071651 (600h25m40.271154696s) [noise=true , address=77.90.31.12:4604 , piece_id=OKY6LAK735LFQ4MDC5K6LJPFQITQSGNOZUGNU6US25RELWLK33BQ, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:40.423162570 (600h25m40.294245605s) [noise=true , address=45.91.116.12:4241 , piece_id=FYZGK7TFHSQ5BDLEMJIOGXWVWSZ45NRVSYQ4DT6DHBANOCBOZZWA, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:40.428962032 (600h25m40.300045067s) [noise=true , address=77.90.31.17:4800 , piece_id=K3HNTJHOSIIUBSKISKWG5YRGLKNKHDOWHITXEDDLKPATPIC4IKTQ, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:40.543712124 (600h25m40.414795169s) [noise=true , address=45.91.116.5:4314 , piece_id=XWWPTNVPYYZ6VL26MACQEJTB2FYN6523YJGZK4I5B4OFRJNUMY3A, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:40.993222051 (600h25m40.864305086s) [noise=true , address=145.79.227.44:4196 , piece_id=ZUDLORSKFXXIGHZ3YNIE3T7JF7OG55NZKPTSFOUVHAISGF7V7J5Q, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:41.006811979 (600h25m40.877895024s) [noise=true , address=193.149.164.143:4696 , piece_id=3WM5R3KUHXN4WVDOKBAOFX7RAI6ZUTV7UKHUOYAKHUYPM4M24N2A, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:41.024553631 (600h25m40.895636666s) [noise=true , address=145.79.227.40:4077 , piece_id=AWXM3BGQVCAARS5F5UISWWIFAVWLWV2KML4FAP6BQ6QNXJTG77NA, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:41.054002647 (600h25m40.925085692s) [noise=true , address=145.79.227.49:4350 , piece_id=MA66IA2WTN55SFQDQ4LF7H5LNVUK3IMIHHRUH4TXWHWZPTQDBDCA, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:41.074754852 (600h25m40.945837897s) [noise=true , address=77.90.31.21:4911 , piece_id=B25VZGBKAGIB4ZN6OICFQFZTRQWZIIYDBFUWVNAW4QW26AVYSRPA, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:41.076478905 (600h25m40.94756195s) [noise=true , address=145.79.227.35:4516 , piece_id=RDI4UFYB6YC22A5QOGDTLRGCG63TSV6XJFKGLKGSJBC5CH3KCQMQ, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:41.091294145 (600h25m40.96237718s) [noise=true , address=145.79.227.30:4789 , piece_id=5JZFEA3P6EICC2MGUYOQWY4OWOVHXWZLBJBBSVUAMPLV2EHZOZLA, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:41.138722376 (600h25m41.009805421s) [noise=true , address=145.79.227.49:4359 , piece_id=MCTYZULSW2Z2A6HXPREG3GOEYF5QLBPKEB7DDGT32FNTWHC646CQ, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:41.138745481 (600h25m41.009828516s) [seg_pos={0 0}, upload=526249]: Segment reached optimal threshold of 14 pieces. Dec 30 15:54:41.138965780 (600h25m41.010048815s) [noise=true , address=145.79.227.33:4877 , piece_id=GPUOFM2E3KZGDCEZQYPKADLJFDML5DS7OTWFW7AFB5DYAHBVBQ6Q, seg_pos={0 0}, upload=526249]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17TuauVrjKmehRpXqmK7bemU9V33GDZu9RGBs6fanXvJkV1gdH): protocol: expected piece hash; context canceled; EOF Dec 30 15:54:41.138974496 (600h25m41.010057531s) [noise=true , address=145.79.227.33:4877 , piece_id=GPUOFM2E3KZGDCEZQYPKADLJFDML5DS7OTWFW7AFB5DYAHBVBQ6Q, seg_pos={0 0}, upload=526249]: Long tail cancellation due to optimal threshold Dec 30 15:54:41.139084886 (600h25m41.010167921s) [noise=true , address=77.90.31.19:4854 , piece_id=4W27FHSBO3L2TSAZL2JFCTUPHYV7RI5XDQ4SJWOCRCD5V4K4IK2Q, seg_pos={0 0}, upload=526249]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16HwqYhJnhseWEhzzv4QreZ953rB5cKxqFX8swEqaqiRXCRsAE): protocol: write tcp 188.166.225.170:48144->77.90.31.19:4854: use of closed network connection; write tcp 188.166.225.170:48144->77.90.31.19:4854: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48144->77.90.31.19:4854: use of closed network connection Dec 30 15:54:41.139093533 (600h25m41.010176568s) [noise=true , address=77.90.31.19:4854 , piece_id=4W27FHSBO3L2TSAZL2JFCTUPHYV7RI5XDQ4SJWOCRCD5V4K4IK2Q, seg_pos={0 0}, upload=526249]: Long tail cancellation due to optimal threshold Dec 30 15:54:41.139129352 (600h25m41.010212397s) [noise=true , address=145.79.227.43:4189 , piece_id=DYHLPLLBNG6B3FXERO33ZDDSHFUHQIIYGQZJFE2UMJXMLUBYC3KQ, seg_pos={0 0}, upload=526249]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13mB9AoQJubhEJAAVqzd4o4eq8AfNvLkdqHTDAPN3UyXXwE4B7): protocol: expected piece hash; context canceled; EOF Dec 30 15:54:41.139138429 (600h25m41.010221484s) [noise=true , address=145.79.227.43:4189 , piece_id=DYHLPLLBNG6B3FXERO33ZDDSHFUHQIIYGQZJFE2UMJXMLUBYC3KQ, seg_pos={0 0}, upload=526249]: Long tail cancellation due to optimal threshold Dec 30 15:54:41.139238890 (600h25m41.010321925s) [noise=true , address=64.49.13.169:4713 , piece_id=KI4XABHC36ADSOANW3HVM3TCWZ4ZXXXNP7MWJVESDKD5WO26DJHQ, seg_pos={0 0}, upload=526249]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17z3AiUfbxBX2eMDnUnqo1bbgojuQxJro7PsEaPjYzQjHQ3D6K): protocol: write tcp 188.166.225.170:35558->64.49.13.169:4713: use of closed network connection; write tcp 188.166.225.170:35558->64.49.13.169:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35558->64.49.13.169:4713: use of closed network connection Dec 30 15:54:41.139244420 (600h25m41.010327455s) [noise=true , address=64.49.13.169:4713 , piece_id=KI4XABHC36ADSOANW3HVM3TCWZ4ZXXXNP7MWJVESDKD5WO26DJHQ, seg_pos={0 0}, upload=526249]: Long tail cancellation due to optimal threshold Dec 30 15:54:41.139323681 (600h25m41.010406716s) [seg_pos={0 0}, upload=526249]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 15:54:41.139392121 (600h25m41.010475166s) [upload=526249]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 15:54:41.139396971 (600h25m41.010480006s) [upload=526249]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 15:54:41.438011559 (600h25m41.309094604s) [upload=526249]: Done uploading segments... Dec 30 15:54:41.438036206 (600h25m41.309119251s) [upload=526249]: Upload finished. err: <nil> Duration: 47.315279088s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 12.420408722s trace: https://jaeger.dev.storj.io/trace/547648a2cfce50e7 FAIL (2025-12-30 16:00:08 UTC) orbiter warning: context deadline exceeded Duration: 12.420435663s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 11.873144028s trace: https://jaeger.dev.storj.io/trace/547648a2cfce50e7 FAIL (2025-12-30 15:59:52 UTC) orbiter warning: context deadline exceeded Duration: 11.873166801s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 42 out of 60 is failed (threshold=6) [.XX.X.X.X.XXXXXX.XXXX...XX.X.XXXXXXXXX.XXXXXXX...XXXXX.XX..X] duration: 5.000677096s trace: https://jaeger.dev.storj.io/trace/3cedbfae74c3fb1c FAIL (2025-12-30 15:59:37 UTC) orbiter warning: context deadline exceeded Duration: 5.000697293s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [X............................X..XX..X..................X....] duration: 44.644710086s trace: https://jaeger.dev.storj.io/trace/2ead4e08bdf51a6a FAIL (2025-12-30 15:59:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 43:4696 , piece_id=3WM5R3KUHXN4WVDOKBAOFX7RAI6ZUTV7UKHUOYAKHUYPM4M24N2A, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.239224572 (600h25m38.110307617s) [noise=true , address=45.91.116.12:4241 , piece_id=FYZGK7TFHSQ5BDLEMJIOGXWVWSZ45NRVSYQ4DT6DHBANOCBOZZWA, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.239811830 (600h25m38.110894865s) [noise=true , address=145.79.227.43:4189 , piece_id=DYHLPLLBNG6B3FXERO33ZDDSHFUHQIIYGQZJFE2UMJXMLUBYC3KQ, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.241164706 (600h25m38.112247741s) [noise=true , address=77.90.31.12:4604 , piece_id=OKY6LAK735LFQ4MDC5K6LJPFQITQSGNOZUGNU6US25RELWLK33BQ, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.244508833 (600h25m38.115591878s) [noise=true , address=145.79.227.49:4350 , piece_id=MA66IA2WTN55SFQDQ4LF7H5LNVUK3IMIHHRUH4TXWHWZPTQDBDCA, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.244943361 (600h25m38.116026396s) [noise=true , address=77.90.31.17:4800 , piece_id=K3HNTJHOSIIUBSKISKWG5YRGLKNKHDOWHITXEDDLKPATPIC4IKTQ, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.250127493 (600h25m38.121210528s) [noise=true , address=145.79.227.35:4516 , piece_id=RDI4UFYB6YC22A5QOGDTLRGCG63TSV6XJFKGLKGSJBC5CH3KCQMQ, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.250776008 (600h25m38.121859053s) [noise=true , address=145.79.227.39:4033 , piece_id=BLTCPLYP5MIQO5Y7WWPT5QIDV6QM3YWAFMLOEWFOVWFLZOO67QSA, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.253191467 (600h25m38.124274512s) [noise=true , address=145.79.227.44:4196 , piece_id=ZUDLORSKFXXIGHZ3YNIE3T7JF7OG55NZKPTSFOUVHAISGF7V7J5Q, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.254554973 (600h25m38.125638018s) [noise=true , address=45.91.116.5:4314 , piece_id=XWWPTNVPYYZ6VL26MACQEJTB2FYN6523YJGZK4I5B4OFRJNUMY3A, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.255815333 (600h25m38.126898378s) [noise=true , address=145.79.227.40:4077 , piece_id=AWXM3BGQVCAARS5F5UISWWIFAVWLWV2KML4FAP6BQ6QNXJTG77NA, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.259122731 (600h25m38.130205776s) [noise=true , address=64.49.13.169:4713 , piece_id=KI4XABHC36ADSOANW3HVM3TCWZ4ZXXXNP7MWJVESDKD5WO26DJHQ, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.265517899 (600h25m38.136600944s) [noise=true , address=77.90.31.19:4854 , piece_id=4W27FHSBO3L2TSAZL2JFCTUPHYV7RI5XDQ4SJWOCRCD5V4K4IK2Q, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.266738002 (600h25m38.137821047s) [noise=true , address=145.79.227.33:4877 , piece_id=GPUOFM2E3KZGDCEZQYPKADLJFDML5DS7OTWFW7AFB5DYAHBVBQ6Q, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.267328907 (600h25m38.138411952s) [noise=true , address=77.90.31.21:4911 , piece_id=B25VZGBKAGIB4ZN6OICFQFZTRQWZIIYDBFUWVNAW4QW26AVYSRPA, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:39.692439689 (600h25m39.563522734s) [upload=526249]: Next returned nil segment Dec 30 15:54:39.692451583 (600h25m39.563534618s) [upload=526249]: Waiting for error group managing segments... Dec 30 15:54:40.135912060 (600h25m40.006995095s) [noise=true , address=145.79.227.39:4033 , piece_id=BLTCPLYP5MIQO5Y7WWPT5QIDV6QM3YWAFMLOEWFOVWFLZOO67QSA, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:40.285764442 (600h25m40.156847487s) [noise=true , address=5.231.73.3:4032 , piece_id=7MBVEHMVMT6XTZJZHHWA65RMFNIB7EH4HU63WTLBPPB6CILGMYAA, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:40.400071651 (600h25m40.271154696s) [noise=true , address=77.90.31.12:4604 , piece_id=OKY6LAK735LFQ4MDC5K6LJPFQITQSGNOZUGNU6US25RELWLK33BQ, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:40.423162570 (600h25m40.294245605s) [noise=true , address=45.91.116.12:4241 , piece_id=FYZGK7TFHSQ5BDLEMJIOGXWVWSZ45NRVSYQ4DT6DHBANOCBOZZWA, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:40.428962032 (600h25m40.300045067s) [noise=true , address=77.90.31.17:4800 , piece_id=K3HNTJHOSIIUBSKISKWG5YRGLKNKHDOWHITXEDDLKPATPIC4IKTQ, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:40.543712124 (600h25m40.414795169s) [noise=true , address=45.91.116.5:4314 , piece_id=XWWPTNVPYYZ6VL26MACQEJTB2FYN6523YJGZK4I5B4OFRJNUMY3A, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:40.993222051 (600h25m40.864305086s) [noise=true , address=145.79.227.44:4196 , piece_id=ZUDLORSKFXXIGHZ3YNIE3T7JF7OG55NZKPTSFOUVHAISGF7V7J5Q, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:41.006811979 (600h25m40.877895024s) [noise=true , address=193.149.164.143:4696 , piece_id=3WM5R3KUHXN4WVDOKBAOFX7RAI6ZUTV7UKHUOYAKHUYPM4M24N2A, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:41.024553631 (600h25m40.895636666s) [noise=true , address=145.79.227.40:4077 , piece_id=AWXM3BGQVCAARS5F5UISWWIFAVWLWV2KML4FAP6BQ6QNXJTG77NA, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:41.054002647 (600h25m40.925085692s) [noise=true , address=145.79.227.49:4350 , piece_id=MA66IA2WTN55SFQDQ4LF7H5LNVUK3IMIHHRUH4TXWHWZPTQDBDCA, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:41.074754852 (600h25m40.945837897s) [noise=true , address=77.90.31.21:4911 , piece_id=B25VZGBKAGIB4ZN6OICFQFZTRQWZIIYDBFUWVNAW4QW26AVYSRPA, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:41.076478905 (600h25m40.94756195s) [noise=true , address=145.79.227.35:4516 , piece_id=RDI4UFYB6YC22A5QOGDTLRGCG63TSV6XJFKGLKGSJBC5CH3KCQMQ, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:41.091294145 (600h25m40.96237718s) [noise=true , address=145.79.227.30:4789 , piece_id=5JZFEA3P6EICC2MGUYOQWY4OWOVHXWZLBJBBSVUAMPLV2EHZOZLA, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:41.138722376 (600h25m41.009805421s) [noise=true , address=145.79.227.49:4359 , piece_id=MCTYZULSW2Z2A6HXPREG3GOEYF5QLBPKEB7DDGT32FNTWHC646CQ, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:41.138745481 (600h25m41.009828516s) [seg_pos={0 0}, upload=526249]: Segment reached optimal threshold of 14 pieces. Dec 30 15:54:41.138965780 (600h25m41.010048815s) [noise=true , address=145.79.227.33:4877 , piece_id=GPUOFM2E3KZGDCEZQYPKADLJFDML5DS7OTWFW7AFB5DYAHBVBQ6Q, seg_pos={0 0}, upload=526249]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17TuauVrjKmehRpXqmK7bemU9V33GDZu9RGBs6fanXvJkV1gdH): protocol: expected piece hash; context canceled; EOF Dec 30 15:54:41.138974496 (600h25m41.010057531s) [noise=true , address=145.79.227.33:4877 , piece_id=GPUOFM2E3KZGDCEZQYPKADLJFDML5DS7OTWFW7AFB5DYAHBVBQ6Q, seg_pos={0 0}, upload=526249]: Long tail cancellation due to optimal threshold Dec 30 15:54:41.139084886 (600h25m41.010167921s) [noise=true , address=77.90.31.19:4854 , piece_id=4W27FHSBO3L2TSAZL2JFCTUPHYV7RI5XDQ4SJWOCRCD5V4K4IK2Q, seg_pos={0 0}, upload=526249]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16HwqYhJnhseWEhzzv4QreZ953rB5cKxqFX8swEqaqiRXCRsAE): protocol: write tcp 188.166.225.170:48144->77.90.31.19:4854: use of closed network connection; write tcp 188.166.225.170:48144->77.90.31.19:4854: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48144->77.90.31.19:4854: use of closed network connection Dec 30 15:54:41.139093533 (600h25m41.010176568s) [noise=true , address=77.90.31.19:4854 , piece_id=4W27FHSBO3L2TSAZL2JFCTUPHYV7RI5XDQ4SJWOCRCD5V4K4IK2Q, seg_pos={0 0}, upload=526249]: Long tail cancellation due to optimal threshold Dec 30 15:54:41.139129352 (600h25m41.010212397s) [noise=true , address=145.79.227.43:4189 , piece_id=DYHLPLLBNG6B3FXERO33ZDDSHFUHQIIYGQZJFE2UMJXMLUBYC3KQ, seg_pos={0 0}, upload=526249]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13mB9AoQJubhEJAAVqzd4o4eq8AfNvLkdqHTDAPN3UyXXwE4B7): protocol: expected piece hash; context canceled; EOF Dec 30 15:54:41.139138429 (600h25m41.010221484s) [noise=true , address=145.79.227.43:4189 , piece_id=DYHLPLLBNG6B3FXERO33ZDDSHFUHQIIYGQZJFE2UMJXMLUBYC3KQ, seg_pos={0 0}, upload=526249]: Long tail cancellation due to optimal threshold Dec 30 15:54:41.139238890 (600h25m41.010321925s) [noise=true , address=64.49.13.169:4713 , piece_id=KI4XABHC36ADSOANW3HVM3TCWZ4ZXXXNP7MWJVESDKD5WO26DJHQ, seg_pos={0 0}, upload=526249]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17z3AiUfbxBX2eMDnUnqo1bbgojuQxJro7PsEaPjYzQjHQ3D6K): protocol: write tcp 188.166.225.170:35558->64.49.13.169:4713: use of closed network connection; write tcp 188.166.225.170:35558->64.49.13.169:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35558->64.49.13.169:4713: use of closed network connection Dec 30 15:54:41.139244420 (600h25m41.010327455s) [noise=true , address=64.49.13.169:4713 , piece_id=KI4XABHC36ADSOANW3HVM3TCWZ4ZXXXNP7MWJVESDKD5WO26DJHQ, seg_pos={0 0}, upload=526249]: Long tail cancellation due to optimal threshold Dec 30 15:54:41.139323681 (600h25m41.010406716s) [seg_pos={0 0}, upload=526249]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 15:54:41.139392121 (600h25m41.010475166s) [upload=526249]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 15:54:41.139396971 (600h25m41.010480006s) [upload=526249]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 15:54:41.438011559 (600h25m41.309094604s) [upload=526249]: Done uploading segments... Dec 30 15:54:41.438036206 (600h25m41.309119251s) [upload=526249]: Upload finished. err: <nil> Duration: 44.644738271s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.X.X..........XX.......XX.XX..X....X.......X......X.X.......] duration: 5.706569643s trace: https://jaeger.dev.storj.io/trace/54fff38414d25b0f FAIL (2025-12-30 15:59:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: AQGHSCVDMOODBX225O5SSOLNLJQYPAY2X7EJ7G5W6RP42ZJ4DGCQ) context deadline exceeded; piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: ZTXP5ESUQMWFEWTQUX2SMER7W2NJG5IISRWFAM3IKAKQFKJOGWSQ) context deadline exceeded ======LOGS====== Dec 30 15:53:33.440030169 (600h24m33.311113204s) [upload=526231]: Starting upload Dec 30 15:53:33.440113437 (600h24m33.311196492s) [upload=526231]: Done starting upload Dec 30 15:53:33.440168081 (600h24m33.311251116s) [upload=526231]: Uploading segments... Dec 30 15:53:33.440357231 (600h24m33.311440276s) [upload=526231]: Got next segment. Inline: false Dec 30 15:53:33.440411114 (600h24m33.311494159s) [upload=526231]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 15:53:33.440416885 (600h24m33.31149992s) [upload=526231]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 15:53:33.684984430 (600h24m33.556067505s) [seg_pos={0 0}, upload=526231]: Begin upload segment... Dec 30 15:53:33.685033994 (600h24m33.556117029s) [seg_pos={0 0}, upload=526231]: Done begin upload segment. Dec 30 15:53:33.685081235 (600h24m33.55616427s) [noise=true , address=145.79.227.36:4519 , piece_id=5JAZZV45ZTNDSTJZIX2EI7RK7MCBXBQUV2PLSF2HBCRFCIJEA6OQ, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.685200161 (600h24m33.556283196s) [noise=true , address=145.79.227.23:4545 , piece_id=NHTKPJFEFGAJOEXUNJJ7TU6N27JJP3O5TFSMWF5I2SMGEYXWEAKQ, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.685709841 (600h24m33.556792887s) [noise=true , address=77.90.31.14:4664 , piece_id=YLXNEMZBXUYTFQ4P6ZFHPA6BIUH4TZ2OBXEQAJWYSCUX333L777A, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.686084574 (600h24m33.557167619s) [noise=true , address=45.91.116.12:4227 , piece_id=5GQIOQC4VDIF3Z6J4TUPZG3XUC3AACZ53CPYB4JIBQD7F5BK2IFA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.686094794 (600h24m33.55717783s) [noise=true , address=77.90.31.10:4517 , piece_id=5Y32J2NPA664L4ZGRDDYYX66FPWND4DNZ5OZXAADJ3LOXK3IA5BA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.686429431 (600h24m33.557512476s) [noise=true , address=45.91.116.13:4153 , piece_id=AQGHSCVDMOODBX225O5SSOLNLJQYPAY2X7EJ7G5W6RP42ZJ4DGCQ, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.686489927 (600h24m33.557572962s) [noise=true , address=45.91.116.8:4047 , piece_id=TLKGZQ2VDRAGBZCULZ35ZR6GEJXQEE2HZ35Q52XBEO5C4FTRFICA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.686910547 (600h24m33.557993582s) [noise=true , address=145.79.227.37:4567 , piece_id=WSHVGPU7L73OET2M6WT7VPHGVRQV6ZCJRPAW7AEXBUQDFWQBNAUA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.686946536 (600h24m33.558029581s) [noise=true , address=145.79.227.14:4838 , piece_id=YB4VPIAQTBJ6G5XFXDVJ3OL6YFAWMEBHHS63SEDX5U4HYKXT3KJQ, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.687266755 (600h24m33.55834979s) [noise=true , address=145.79.227.6:4257 , piece_id=53MG7UDQR3MJYNMQXY3IOPPKOQEOI3KFUTGHQUEHOKCNG5NTAWXA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.687625419 (600h24m33.558708464s) [noise=true , address=45.91.116.11:4120 , piece_id=ZTXP5ESUQMWFEWTQUX2SMER7W2NJG5IISRWFAM3IKAKQFKJOGWSQ, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.688010913 (600h24m33.559093948s) [noise=true , address=145.79.227.19:4077 , piece_id=JEX52J3QTGUWMY3FO6BD5IJV5UPUUCKVV4MYZAOAU3BWMKFSN3KQ, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.688360248 (600h24m33.559443283s) [noise=true , address=145.79.227.16:4910 , piece_id=UHDFUXEQCMN73AUB7WEJBG5HY2G4E25MQGSBNK6AKHMQMDZCJYRA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.688673685 (600h24m33.55975672s) [noise=true , address=145.79.227.48:4303 , piece_id=IO7FUTELAP2L3VVZEQN6N6QG42WMAOS7H5JEPFAST5YEKMKHYLXA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.689057035 (600h24m33.56014007s) [noise=true , address=145.79.227.2:4114 , piece_id=3MJV2XXBY776ZUIO2KU4MEP37SP6P2NYFO255DCI6IFUZWUAGOWQ, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.690890025 (600h24m33.56197308s) [noise=true , address=145.79.227.3:4150 , piece_id=BUBUCPH3LTJO2CR3IVIXEOJOVS422KQT6L2Z22OBZP6DFPCK5KAA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:35.401137462 (600h24m35.272220507s) [upload=526231]: Next returned nil segment Dec 30 15:53:35.401154925 (600h24m35.27223797s) [upload=526231]: Waiting for error group managing segments... Dec 30 15:53:35.673442202 (600h24m35.544525237s) [noise=true , address=145.79.227.23:4545 , piece_id=NHTKPJFEFGAJOEXUNJJ7TU6N27JJP3O5TFSMWF5I2SMGEYXWEAKQ, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.674880822 (600h24m35.545963857s) [noise=true , address=145.79.227.2:4114 , piece_id=3MJV2XXBY776ZUIO2KU4MEP37SP6P2NYFO255DCI6IFUZWUAGOWQ, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.688801769 (600h24m35.559884814s) [noise=true , address=145.79.227.36:4519 , piece_id=5JAZZV45ZTNDSTJZIX2EI7RK7MCBXBQUV2PLSF2HBCRFCIJEA6OQ, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.689704678 (600h24m35.560787713s) [noise=true , address=145.79.227.6:4257 , piece_id=53MG7UDQR3MJYNMQXY3IOPPKOQEOI3KFUTGHQUEHOKCNG5NTAWXA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.690785346 (600h24m35.561868381s) [noise=true , address=145.79.227.14:4838 , piece_id=YB4VPIAQTBJ6G5XFXDVJ3OL6YFAWMEBHHS63SEDX5U4HYKXT3KJQ, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.692882529 (600h24m35.563965564s) [noise=true , address=145.79.227.37:4567 , piece_id=WSHVGPU7L73OET2M6WT7VPHGVRQV6ZCJRPAW7AEXBUQDFWQBNAUA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.696575461 (600h24m35.567658506s) [noise=true , address=145.79.227.16:4910 , piece_id=UHDFUXEQCMN73AUB7WEJBG5HY2G4E25MQGSBNK6AKHMQMDZCJYRA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.697864125 (600h24m35.56894716s) [noise=true , address=145.79.227.19:4077 , piece_id=JEX52J3QTGUWMY3FO6BD5IJV5UPUUCKVV4MYZAOAU3BWMKFSN3KQ, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.758530774 (600h24m35.629613819s) [noise=true , address=145.79.227.48:4303 , piece_id=IO7FUTELAP2L3VVZEQN6N6QG42WMAOS7H5JEPFAST5YEKMKHYLXA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.796785737 (600h24m35.667868772s) [noise=true , address=145.79.227.3:4150 , piece_id=BUBUCPH3LTJO2CR3IVIXEOJOVS422KQT6L2Z22OBZP6DFPCK5KAA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:36.029561278 (600h24m35.900644313s) [noise=true , address=77.90.31.10:4517 , piece_id=5Y32J2NPA664L4ZGRDDYYX66FPWND4DNZ5OZXAADJ3LOXK3IA5BA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:36.070210490 (600h24m35.941293536s) [noise=true , address=45.91.116.13:4153 , piece_id=AQGHSCVDMOODBX225O5SSOLNLJQYPAY2X7EJ7G5W6RP42ZJ4DGCQ, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:36.082591286 (600h24m35.953674331s) [noise=true , address=45.91.116.11:4120 , piece_id=ZTXP5ESUQMWFEWTQUX2SMER7W2NJG5IISRWFAM3IKAKQFKJOGWSQ, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:36.082610784 (600h24m35.953693819s) [seg_pos={0 0}, upload=526231]: Segment reached optimal threshold of 13 pieces. Dec 30 15:53:36.082858675 (600h24m35.953941711s) [noise=true , address=77.90.31.14:4664 , piece_id=YLXNEMZBXUYTFQ4P6ZFHPA6BIUH4TZ2OBXEQAJWYSCUX333L777A, seg_pos={0 0}, upload=526231]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: expected piece hash; context canceled; EOF Dec 30 15:53:36.082868053 (600h24m35.953951088s) [noise=true , address=77.90.31.14:4664 , piece_id=YLXNEMZBXUYTFQ4P6ZFHPA6BIUH4TZ2OBXEQAJWYSCUX333L777A, seg_pos={0 0}, upload=526231]: Long tail cancellation due to optimal threshold Dec 30 15:53:36.082953826 (600h24m35.954036871s) [noise=true , address=45.91.116.12:4227 , piece_id=5GQIOQC4VDIF3Z6J4TUPZG3XUC3AACZ53CPYB4JIBQD7F5BK2IFA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Dec 30 15:53:36.082969977 (600h24m35.954053012s) [noise=true , address=45.91.116.12:4227 , piece_id=5GQIOQC4VDIF3Z6J4TUPZG3XUC3AACZ53CPYB4JIBQD7F5BK2IFA, seg_pos={0 0}, upload=526231]: Long tail cancellation due to optimal threshold Dec 30 15:53:36.083266191 (600h24m35.954349226s) [noise=true , address=45.91.116.8:4047 , piece_id=TLKGZQ2VDRAGBZCULZ35ZR6GEJXQEE2HZ35Q52XBEO5C4FTRFICA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: read tcp 188.166.225.170:42816->45.91.116.8:4047: use of closed network connection; context canceled Dec 30 15:53:36.083274006 (600h24m35.954357041s) [noise=true , address=45.91.116.8:4047 , piece_id=TLKGZQ2VDRAGBZCULZ35ZR6GEJXQEE2HZ35Q52XBEO5C4FTRFICA, seg_pos={0 0}, upload=526231]: Long tail cancellation due to optimal threshold Dec 30 15:53:36.083296048 (600h24m35.954379083s) [seg_pos={0 0}, upload=526231]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 15:53:36.083385679 (600h24m35.954468714s) [upload=526231]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 15:53:36.083392341 (600h24m35.954475376s) [upload=526231]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 15:53:36.379743600 (600h24m36.250826635s) [upload=526231]: Done uploading segments... Dec 30 15:53:36.379758638 (600h24m36.250841673s) [upload=526231]: Upload finished. err: <nil> Duration: 5.706598037s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 42 out of 60 is failed (threshold=6) [X.XX.X.X.X.XXXXXX.XXXX...XX.X.XXXXXXXXX.XXXXXXX...XXXXX.XX..] duration: 3.571440548s trace: https://jaeger.dev.storj.io/trace/3532402db8c825 FAIL (2025-12-30 15:59:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.57156205s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 11.864370442s trace: https://jaeger.dev.storj.io/trace/547648a2cfce50e7 FAIL (2025-12-30 15:58:52 UTC) orbiter warning: context deadline exceeded Duration: 11.864393126s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [.X............................X..XX..X..................X...] duration: 35.096817257s trace: https://jaeger.dev.storj.io/trace/2ead4e08bdf51a6a FAIL (2025-12-30 15:58:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 43:4696 , piece_id=3WM5R3KUHXN4WVDOKBAOFX7RAI6ZUTV7UKHUOYAKHUYPM4M24N2A, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.239224572 (600h25m38.110307617s) [noise=true , address=45.91.116.12:4241 , piece_id=FYZGK7TFHSQ5BDLEMJIOGXWVWSZ45NRVSYQ4DT6DHBANOCBOZZWA, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.239811830 (600h25m38.110894865s) [noise=true , address=145.79.227.43:4189 , piece_id=DYHLPLLBNG6B3FXERO33ZDDSHFUHQIIYGQZJFE2UMJXMLUBYC3KQ, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.241164706 (600h25m38.112247741s) [noise=true , address=77.90.31.12:4604 , piece_id=OKY6LAK735LFQ4MDC5K6LJPFQITQSGNOZUGNU6US25RELWLK33BQ, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.244508833 (600h25m38.115591878s) [noise=true , address=145.79.227.49:4350 , piece_id=MA66IA2WTN55SFQDQ4LF7H5LNVUK3IMIHHRUH4TXWHWZPTQDBDCA, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.244943361 (600h25m38.116026396s) [noise=true , address=77.90.31.17:4800 , piece_id=K3HNTJHOSIIUBSKISKWG5YRGLKNKHDOWHITXEDDLKPATPIC4IKTQ, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.250127493 (600h25m38.121210528s) [noise=true , address=145.79.227.35:4516 , piece_id=RDI4UFYB6YC22A5QOGDTLRGCG63TSV6XJFKGLKGSJBC5CH3KCQMQ, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.250776008 (600h25m38.121859053s) [noise=true , address=145.79.227.39:4033 , piece_id=BLTCPLYP5MIQO5Y7WWPT5QIDV6QM3YWAFMLOEWFOVWFLZOO67QSA, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.253191467 (600h25m38.124274512s) [noise=true , address=145.79.227.44:4196 , piece_id=ZUDLORSKFXXIGHZ3YNIE3T7JF7OG55NZKPTSFOUVHAISGF7V7J5Q, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.254554973 (600h25m38.125638018s) [noise=true , address=45.91.116.5:4314 , piece_id=XWWPTNVPYYZ6VL26MACQEJTB2FYN6523YJGZK4I5B4OFRJNUMY3A, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.255815333 (600h25m38.126898378s) [noise=true , address=145.79.227.40:4077 , piece_id=AWXM3BGQVCAARS5F5UISWWIFAVWLWV2KML4FAP6BQ6QNXJTG77NA, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.259122731 (600h25m38.130205776s) [noise=true , address=64.49.13.169:4713 , piece_id=KI4XABHC36ADSOANW3HVM3TCWZ4ZXXXNP7MWJVESDKD5WO26DJHQ, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.265517899 (600h25m38.136600944s) [noise=true , address=77.90.31.19:4854 , piece_id=4W27FHSBO3L2TSAZL2JFCTUPHYV7RI5XDQ4SJWOCRCD5V4K4IK2Q, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.266738002 (600h25m38.137821047s) [noise=true , address=145.79.227.33:4877 , piece_id=GPUOFM2E3KZGDCEZQYPKADLJFDML5DS7OTWFW7AFB5DYAHBVBQ6Q, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.267328907 (600h25m38.138411952s) [noise=true , address=77.90.31.21:4911 , piece_id=B25VZGBKAGIB4ZN6OICFQFZTRQWZIIYDBFUWVNAW4QW26AVYSRPA, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:39.692439689 (600h25m39.563522734s) [upload=526249]: Next returned nil segment Dec 30 15:54:39.692451583 (600h25m39.563534618s) [upload=526249]: Waiting for error group managing segments... Dec 30 15:54:40.135912060 (600h25m40.006995095s) [noise=true , address=145.79.227.39:4033 , piece_id=BLTCPLYP5MIQO5Y7WWPT5QIDV6QM3YWAFMLOEWFOVWFLZOO67QSA, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:40.285764442 (600h25m40.156847487s) [noise=true , address=5.231.73.3:4032 , piece_id=7MBVEHMVMT6XTZJZHHWA65RMFNIB7EH4HU63WTLBPPB6CILGMYAA, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:40.400071651 (600h25m40.271154696s) [noise=true , address=77.90.31.12:4604 , piece_id=OKY6LAK735LFQ4MDC5K6LJPFQITQSGNOZUGNU6US25RELWLK33BQ, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:40.423162570 (600h25m40.294245605s) [noise=true , address=45.91.116.12:4241 , piece_id=FYZGK7TFHSQ5BDLEMJIOGXWVWSZ45NRVSYQ4DT6DHBANOCBOZZWA, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:40.428962032 (600h25m40.300045067s) [noise=true , address=77.90.31.17:4800 , piece_id=K3HNTJHOSIIUBSKISKWG5YRGLKNKHDOWHITXEDDLKPATPIC4IKTQ, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:40.543712124 (600h25m40.414795169s) [noise=true , address=45.91.116.5:4314 , piece_id=XWWPTNVPYYZ6VL26MACQEJTB2FYN6523YJGZK4I5B4OFRJNUMY3A, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:40.993222051 (600h25m40.864305086s) [noise=true , address=145.79.227.44:4196 , piece_id=ZUDLORSKFXXIGHZ3YNIE3T7JF7OG55NZKPTSFOUVHAISGF7V7J5Q, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:41.006811979 (600h25m40.877895024s) [noise=true , address=193.149.164.143:4696 , piece_id=3WM5R3KUHXN4WVDOKBAOFX7RAI6ZUTV7UKHUOYAKHUYPM4M24N2A, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:41.024553631 (600h25m40.895636666s) [noise=true , address=145.79.227.40:4077 , piece_id=AWXM3BGQVCAARS5F5UISWWIFAVWLWV2KML4FAP6BQ6QNXJTG77NA, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:41.054002647 (600h25m40.925085692s) [noise=true , address=145.79.227.49:4350 , piece_id=MA66IA2WTN55SFQDQ4LF7H5LNVUK3IMIHHRUH4TXWHWZPTQDBDCA, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:41.074754852 (600h25m40.945837897s) [noise=true , address=77.90.31.21:4911 , piece_id=B25VZGBKAGIB4ZN6OICFQFZTRQWZIIYDBFUWVNAW4QW26AVYSRPA, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:41.076478905 (600h25m40.94756195s) [noise=true , address=145.79.227.35:4516 , piece_id=RDI4UFYB6YC22A5QOGDTLRGCG63TSV6XJFKGLKGSJBC5CH3KCQMQ, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:41.091294145 (600h25m40.96237718s) [noise=true , address=145.79.227.30:4789 , piece_id=5JZFEA3P6EICC2MGUYOQWY4OWOVHXWZLBJBBSVUAMPLV2EHZOZLA, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:41.138722376 (600h25m41.009805421s) [noise=true , address=145.79.227.49:4359 , piece_id=MCTYZULSW2Z2A6HXPREG3GOEYF5QLBPKEB7DDGT32FNTWHC646CQ, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:41.138745481 (600h25m41.009828516s) [seg_pos={0 0}, upload=526249]: Segment reached optimal threshold of 14 pieces. Dec 30 15:54:41.138965780 (600h25m41.010048815s) [noise=true , address=145.79.227.33:4877 , piece_id=GPUOFM2E3KZGDCEZQYPKADLJFDML5DS7OTWFW7AFB5DYAHBVBQ6Q, seg_pos={0 0}, upload=526249]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17TuauVrjKmehRpXqmK7bemU9V33GDZu9RGBs6fanXvJkV1gdH): protocol: expected piece hash; context canceled; EOF Dec 30 15:54:41.138974496 (600h25m41.010057531s) [noise=true , address=145.79.227.33:4877 , piece_id=GPUOFM2E3KZGDCEZQYPKADLJFDML5DS7OTWFW7AFB5DYAHBVBQ6Q, seg_pos={0 0}, upload=526249]: Long tail cancellation due to optimal threshold Dec 30 15:54:41.139084886 (600h25m41.010167921s) [noise=true , address=77.90.31.19:4854 , piece_id=4W27FHSBO3L2TSAZL2JFCTUPHYV7RI5XDQ4SJWOCRCD5V4K4IK2Q, seg_pos={0 0}, upload=526249]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16HwqYhJnhseWEhzzv4QreZ953rB5cKxqFX8swEqaqiRXCRsAE): protocol: write tcp 188.166.225.170:48144->77.90.31.19:4854: use of closed network connection; write tcp 188.166.225.170:48144->77.90.31.19:4854: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48144->77.90.31.19:4854: use of closed network connection Dec 30 15:54:41.139093533 (600h25m41.010176568s) [noise=true , address=77.90.31.19:4854 , piece_id=4W27FHSBO3L2TSAZL2JFCTUPHYV7RI5XDQ4SJWOCRCD5V4K4IK2Q, seg_pos={0 0}, upload=526249]: Long tail cancellation due to optimal threshold Dec 30 15:54:41.139129352 (600h25m41.010212397s) [noise=true , address=145.79.227.43:4189 , piece_id=DYHLPLLBNG6B3FXERO33ZDDSHFUHQIIYGQZJFE2UMJXMLUBYC3KQ, seg_pos={0 0}, upload=526249]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13mB9AoQJubhEJAAVqzd4o4eq8AfNvLkdqHTDAPN3UyXXwE4B7): protocol: expected piece hash; context canceled; EOF Dec 30 15:54:41.139138429 (600h25m41.010221484s) [noise=true , address=145.79.227.43:4189 , piece_id=DYHLPLLBNG6B3FXERO33ZDDSHFUHQIIYGQZJFE2UMJXMLUBYC3KQ, seg_pos={0 0}, upload=526249]: Long tail cancellation due to optimal threshold Dec 30 15:54:41.139238890 (600h25m41.010321925s) [noise=true , address=64.49.13.169:4713 , piece_id=KI4XABHC36ADSOANW3HVM3TCWZ4ZXXXNP7MWJVESDKD5WO26DJHQ, seg_pos={0 0}, upload=526249]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17z3AiUfbxBX2eMDnUnqo1bbgojuQxJro7PsEaPjYzQjHQ3D6K): protocol: write tcp 188.166.225.170:35558->64.49.13.169:4713: use of closed network connection; write tcp 188.166.225.170:35558->64.49.13.169:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35558->64.49.13.169:4713: use of closed network connection Dec 30 15:54:41.139244420 (600h25m41.010327455s) [noise=true , address=64.49.13.169:4713 , piece_id=KI4XABHC36ADSOANW3HVM3TCWZ4ZXXXNP7MWJVESDKD5WO26DJHQ, seg_pos={0 0}, upload=526249]: Long tail cancellation due to optimal threshold Dec 30 15:54:41.139323681 (600h25m41.010406716s) [seg_pos={0 0}, upload=526249]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 15:54:41.139392121 (600h25m41.010475166s) [upload=526249]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 15:54:41.139396971 (600h25m41.010480006s) [upload=526249]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 15:54:41.438011559 (600h25m41.309094604s) [upload=526249]: Done uploading segments... Dec 30 15:54:41.438036206 (600h25m41.309119251s) [upload=526249]: Upload finished. err: <nil> Duration: 35.096843327s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 43 out of 60 is failed (threshold=6) [XX.XX.X.X.X.XXXXXX.XXXX...XX.X.XXXXXXXXX.XXXXXXX...XXXXX.XX.] duration: 3.621621356s trace: https://jaeger.dev.storj.io/trace/3532402db8c825 FAIL (2025-12-30 15:58:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.621637648s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 11.242636727s trace: https://jaeger.dev.storj.io/trace/547648a2cfce50e7 FAIL (2025-12-30 15:58:28 UTC) orbiter warning: context deadline exceeded Duration: 11.242664179s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [..X.X..........XX.......XX.XX..X....X.......X......X.X......] duration: 6.069129515s trace: https://jaeger.dev.storj.io/trace/54fff38414d25b0f FAIL (2025-12-30 15:58:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: AQGHSCVDMOODBX225O5SSOLNLJQYPAY2X7EJ7G5W6RP42ZJ4DGCQ) context deadline exceeded; piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: ZTXP5ESUQMWFEWTQUX2SMER7W2NJG5IISRWFAM3IKAKQFKJOGWSQ) context deadline exceeded ======LOGS====== Dec 30 15:53:33.440030169 (600h24m33.311113204s) [upload=526231]: Starting upload Dec 30 15:53:33.440113437 (600h24m33.311196492s) [upload=526231]: Done starting upload Dec 30 15:53:33.440168081 (600h24m33.311251116s) [upload=526231]: Uploading segments... Dec 30 15:53:33.440357231 (600h24m33.311440276s) [upload=526231]: Got next segment. Inline: false Dec 30 15:53:33.440411114 (600h24m33.311494159s) [upload=526231]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 15:53:33.440416885 (600h24m33.31149992s) [upload=526231]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 15:53:33.684984430 (600h24m33.556067505s) [seg_pos={0 0}, upload=526231]: Begin upload segment... Dec 30 15:53:33.685033994 (600h24m33.556117029s) [seg_pos={0 0}, upload=526231]: Done begin upload segment. Dec 30 15:53:33.685081235 (600h24m33.55616427s) [noise=true , address=145.79.227.36:4519 , piece_id=5JAZZV45ZTNDSTJZIX2EI7RK7MCBXBQUV2PLSF2HBCRFCIJEA6OQ, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.685200161 (600h24m33.556283196s) [noise=true , address=145.79.227.23:4545 , piece_id=NHTKPJFEFGAJOEXUNJJ7TU6N27JJP3O5TFSMWF5I2SMGEYXWEAKQ, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.685709841 (600h24m33.556792887s) [noise=true , address=77.90.31.14:4664 , piece_id=YLXNEMZBXUYTFQ4P6ZFHPA6BIUH4TZ2OBXEQAJWYSCUX333L777A, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.686084574 (600h24m33.557167619s) [noise=true , address=45.91.116.12:4227 , piece_id=5GQIOQC4VDIF3Z6J4TUPZG3XUC3AACZ53CPYB4JIBQD7F5BK2IFA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.686094794 (600h24m33.55717783s) [noise=true , address=77.90.31.10:4517 , piece_id=5Y32J2NPA664L4ZGRDDYYX66FPWND4DNZ5OZXAADJ3LOXK3IA5BA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.686429431 (600h24m33.557512476s) [noise=true , address=45.91.116.13:4153 , piece_id=AQGHSCVDMOODBX225O5SSOLNLJQYPAY2X7EJ7G5W6RP42ZJ4DGCQ, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.686489927 (600h24m33.557572962s) [noise=true , address=45.91.116.8:4047 , piece_id=TLKGZQ2VDRAGBZCULZ35ZR6GEJXQEE2HZ35Q52XBEO5C4FTRFICA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.686910547 (600h24m33.557993582s) [noise=true , address=145.79.227.37:4567 , piece_id=WSHVGPU7L73OET2M6WT7VPHGVRQV6ZCJRPAW7AEXBUQDFWQBNAUA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.686946536 (600h24m33.558029581s) [noise=true , address=145.79.227.14:4838 , piece_id=YB4VPIAQTBJ6G5XFXDVJ3OL6YFAWMEBHHS63SEDX5U4HYKXT3KJQ, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.687266755 (600h24m33.55834979s) [noise=true , address=145.79.227.6:4257 , piece_id=53MG7UDQR3MJYNMQXY3IOPPKOQEOI3KFUTGHQUEHOKCNG5NTAWXA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.687625419 (600h24m33.558708464s) [noise=true , address=45.91.116.11:4120 , piece_id=ZTXP5ESUQMWFEWTQUX2SMER7W2NJG5IISRWFAM3IKAKQFKJOGWSQ, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.688010913 (600h24m33.559093948s) [noise=true , address=145.79.227.19:4077 , piece_id=JEX52J3QTGUWMY3FO6BD5IJV5UPUUCKVV4MYZAOAU3BWMKFSN3KQ, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.688360248 (600h24m33.559443283s) [noise=true , address=145.79.227.16:4910 , piece_id=UHDFUXEQCMN73AUB7WEJBG5HY2G4E25MQGSBNK6AKHMQMDZCJYRA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.688673685 (600h24m33.55975672s) [noise=true , address=145.79.227.48:4303 , piece_id=IO7FUTELAP2L3VVZEQN6N6QG42WMAOS7H5JEPFAST5YEKMKHYLXA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.689057035 (600h24m33.56014007s) [noise=true , address=145.79.227.2:4114 , piece_id=3MJV2XXBY776ZUIO2KU4MEP37SP6P2NYFO255DCI6IFUZWUAGOWQ, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.690890025 (600h24m33.56197308s) [noise=true , address=145.79.227.3:4150 , piece_id=BUBUCPH3LTJO2CR3IVIXEOJOVS422KQT6L2Z22OBZP6DFPCK5KAA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:35.401137462 (600h24m35.272220507s) [upload=526231]: Next returned nil segment Dec 30 15:53:35.401154925 (600h24m35.27223797s) [upload=526231]: Waiting for error group managing segments... Dec 30 15:53:35.673442202 (600h24m35.544525237s) [noise=true , address=145.79.227.23:4545 , piece_id=NHTKPJFEFGAJOEXUNJJ7TU6N27JJP3O5TFSMWF5I2SMGEYXWEAKQ, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.674880822 (600h24m35.545963857s) [noise=true , address=145.79.227.2:4114 , piece_id=3MJV2XXBY776ZUIO2KU4MEP37SP6P2NYFO255DCI6IFUZWUAGOWQ, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.688801769 (600h24m35.559884814s) [noise=true , address=145.79.227.36:4519 , piece_id=5JAZZV45ZTNDSTJZIX2EI7RK7MCBXBQUV2PLSF2HBCRFCIJEA6OQ, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.689704678 (600h24m35.560787713s) [noise=true , address=145.79.227.6:4257 , piece_id=53MG7UDQR3MJYNMQXY3IOPPKOQEOI3KFUTGHQUEHOKCNG5NTAWXA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.690785346 (600h24m35.561868381s) [noise=true , address=145.79.227.14:4838 , piece_id=YB4VPIAQTBJ6G5XFXDVJ3OL6YFAWMEBHHS63SEDX5U4HYKXT3KJQ, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.692882529 (600h24m35.563965564s) [noise=true , address=145.79.227.37:4567 , piece_id=WSHVGPU7L73OET2M6WT7VPHGVRQV6ZCJRPAW7AEXBUQDFWQBNAUA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.696575461 (600h24m35.567658506s) [noise=true , address=145.79.227.16:4910 , piece_id=UHDFUXEQCMN73AUB7WEJBG5HY2G4E25MQGSBNK6AKHMQMDZCJYRA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.697864125 (600h24m35.56894716s) [noise=true , address=145.79.227.19:4077 , piece_id=JEX52J3QTGUWMY3FO6BD5IJV5UPUUCKVV4MYZAOAU3BWMKFSN3KQ, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.758530774 (600h24m35.629613819s) [noise=true , address=145.79.227.48:4303 , piece_id=IO7FUTELAP2L3VVZEQN6N6QG42WMAOS7H5JEPFAST5YEKMKHYLXA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.796785737 (600h24m35.667868772s) [noise=true , address=145.79.227.3:4150 , piece_id=BUBUCPH3LTJO2CR3IVIXEOJOVS422KQT6L2Z22OBZP6DFPCK5KAA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:36.029561278 (600h24m35.900644313s) [noise=true , address=77.90.31.10:4517 , piece_id=5Y32J2NPA664L4ZGRDDYYX66FPWND4DNZ5OZXAADJ3LOXK3IA5BA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:36.070210490 (600h24m35.941293536s) [noise=true , address=45.91.116.13:4153 , piece_id=AQGHSCVDMOODBX225O5SSOLNLJQYPAY2X7EJ7G5W6RP42ZJ4DGCQ, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:36.082591286 (600h24m35.953674331s) [noise=true , address=45.91.116.11:4120 , piece_id=ZTXP5ESUQMWFEWTQUX2SMER7W2NJG5IISRWFAM3IKAKQFKJOGWSQ, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:36.082610784 (600h24m35.953693819s) [seg_pos={0 0}, upload=526231]: Segment reached optimal threshold of 13 pieces. Dec 30 15:53:36.082858675 (600h24m35.953941711s) [noise=true , address=77.90.31.14:4664 , piece_id=YLXNEMZBXUYTFQ4P6ZFHPA6BIUH4TZ2OBXEQAJWYSCUX333L777A, seg_pos={0 0}, upload=526231]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: expected piece hash; context canceled; EOF Dec 30 15:53:36.082868053 (600h24m35.953951088s) [noise=true , address=77.90.31.14:4664 , piece_id=YLXNEMZBXUYTFQ4P6ZFHPA6BIUH4TZ2OBXEQAJWYSCUX333L777A, seg_pos={0 0}, upload=526231]: Long tail cancellation due to optimal threshold Dec 30 15:53:36.082953826 (600h24m35.954036871s) [noise=true , address=45.91.116.12:4227 , piece_id=5GQIOQC4VDIF3Z6J4TUPZG3XUC3AACZ53CPYB4JIBQD7F5BK2IFA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Dec 30 15:53:36.082969977 (600h24m35.954053012s) [noise=true , address=45.91.116.12:4227 , piece_id=5GQIOQC4VDIF3Z6J4TUPZG3XUC3AACZ53CPYB4JIBQD7F5BK2IFA, seg_pos={0 0}, upload=526231]: Long tail cancellation due to optimal threshold Dec 30 15:53:36.083266191 (600h24m35.954349226s) [noise=true , address=45.91.116.8:4047 , piece_id=TLKGZQ2VDRAGBZCULZ35ZR6GEJXQEE2HZ35Q52XBEO5C4FTRFICA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: read tcp 188.166.225.170:42816->45.91.116.8:4047: use of closed network connection; context canceled Dec 30 15:53:36.083274006 (600h24m35.954357041s) [noise=true , address=45.91.116.8:4047 , piece_id=TLKGZQ2VDRAGBZCULZ35ZR6GEJXQEE2HZ35Q52XBEO5C4FTRFICA, seg_pos={0 0}, upload=526231]: Long tail cancellation due to optimal threshold Dec 30 15:53:36.083296048 (600h24m35.954379083s) [seg_pos={0 0}, upload=526231]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 15:53:36.083385679 (600h24m35.954468714s) [upload=526231]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 15:53:36.083392341 (600h24m35.954475376s) [upload=526231]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 15:53:36.379743600 (600h24m36.250826635s) [upload=526231]: Done uploading segments... Dec 30 15:53:36.379758638 (600h24m36.250841673s) [upload=526231]: Upload finished. err: <nil> Duration: 6.069156667s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 44 out of 60 is failed (threshold=6) [XXX.XX.X.X.X.XXXXXX.XXXX...XX.X.XXXXXXXXX.XXXXXXX...XXXXX.XX] duration: 5.001920112s trace: https://jaeger.dev.storj.io/trace/3532402db8c825 FAIL (2025-12-30 15:58:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001943016s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [...X.X..........XX.......XX.XX..X....X.......X......X.X.....] duration: 27.711652435s trace: https://jaeger.dev.storj.io/trace/54fff38414d25b0f FAIL (2025-12-30 15:57:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: AQGHSCVDMOODBX225O5SSOLNLJQYPAY2X7EJ7G5W6RP42ZJ4DGCQ) context deadline exceeded; piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: ZTXP5ESUQMWFEWTQUX2SMER7W2NJG5IISRWFAM3IKAKQFKJOGWSQ) context deadline exceeded ======LOGS====== Dec 30 15:53:33.440030169 (600h24m33.311113204s) [upload=526231]: Starting upload Dec 30 15:53:33.440113437 (600h24m33.311196492s) [upload=526231]: Done starting upload Dec 30 15:53:33.440168081 (600h24m33.311251116s) [upload=526231]: Uploading segments... Dec 30 15:53:33.440357231 (600h24m33.311440276s) [upload=526231]: Got next segment. Inline: false Dec 30 15:53:33.440411114 (600h24m33.311494159s) [upload=526231]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 15:53:33.440416885 (600h24m33.31149992s) [upload=526231]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 15:53:33.684984430 (600h24m33.556067505s) [seg_pos={0 0}, upload=526231]: Begin upload segment... Dec 30 15:53:33.685033994 (600h24m33.556117029s) [seg_pos={0 0}, upload=526231]: Done begin upload segment. Dec 30 15:53:33.685081235 (600h24m33.55616427s) [noise=true , address=145.79.227.36:4519 , piece_id=5JAZZV45ZTNDSTJZIX2EI7RK7MCBXBQUV2PLSF2HBCRFCIJEA6OQ, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.685200161 (600h24m33.556283196s) [noise=true , address=145.79.227.23:4545 , piece_id=NHTKPJFEFGAJOEXUNJJ7TU6N27JJP3O5TFSMWF5I2SMGEYXWEAKQ, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.685709841 (600h24m33.556792887s) [noise=true , address=77.90.31.14:4664 , piece_id=YLXNEMZBXUYTFQ4P6ZFHPA6BIUH4TZ2OBXEQAJWYSCUX333L777A, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.686084574 (600h24m33.557167619s) [noise=true , address=45.91.116.12:4227 , piece_id=5GQIOQC4VDIF3Z6J4TUPZG3XUC3AACZ53CPYB4JIBQD7F5BK2IFA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.686094794 (600h24m33.55717783s) [noise=true , address=77.90.31.10:4517 , piece_id=5Y32J2NPA664L4ZGRDDYYX66FPWND4DNZ5OZXAADJ3LOXK3IA5BA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.686429431 (600h24m33.557512476s) [noise=true , address=45.91.116.13:4153 , piece_id=AQGHSCVDMOODBX225O5SSOLNLJQYPAY2X7EJ7G5W6RP42ZJ4DGCQ, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.686489927 (600h24m33.557572962s) [noise=true , address=45.91.116.8:4047 , piece_id=TLKGZQ2VDRAGBZCULZ35ZR6GEJXQEE2HZ35Q52XBEO5C4FTRFICA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.686910547 (600h24m33.557993582s) [noise=true , address=145.79.227.37:4567 , piece_id=WSHVGPU7L73OET2M6WT7VPHGVRQV6ZCJRPAW7AEXBUQDFWQBNAUA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.686946536 (600h24m33.558029581s) [noise=true , address=145.79.227.14:4838 , piece_id=YB4VPIAQTBJ6G5XFXDVJ3OL6YFAWMEBHHS63SEDX5U4HYKXT3KJQ, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.687266755 (600h24m33.55834979s) [noise=true , address=145.79.227.6:4257 , piece_id=53MG7UDQR3MJYNMQXY3IOPPKOQEOI3KFUTGHQUEHOKCNG5NTAWXA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.687625419 (600h24m33.558708464s) [noise=true , address=45.91.116.11:4120 , piece_id=ZTXP5ESUQMWFEWTQUX2SMER7W2NJG5IISRWFAM3IKAKQFKJOGWSQ, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.688010913 (600h24m33.559093948s) [noise=true , address=145.79.227.19:4077 , piece_id=JEX52J3QTGUWMY3FO6BD5IJV5UPUUCKVV4MYZAOAU3BWMKFSN3KQ, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.688360248 (600h24m33.559443283s) [noise=true , address=145.79.227.16:4910 , piece_id=UHDFUXEQCMN73AUB7WEJBG5HY2G4E25MQGSBNK6AKHMQMDZCJYRA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.688673685 (600h24m33.55975672s) [noise=true , address=145.79.227.48:4303 , piece_id=IO7FUTELAP2L3VVZEQN6N6QG42WMAOS7H5JEPFAST5YEKMKHYLXA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.689057035 (600h24m33.56014007s) [noise=true , address=145.79.227.2:4114 , piece_id=3MJV2XXBY776ZUIO2KU4MEP37SP6P2NYFO255DCI6IFUZWUAGOWQ, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.690890025 (600h24m33.56197308s) [noise=true , address=145.79.227.3:4150 , piece_id=BUBUCPH3LTJO2CR3IVIXEOJOVS422KQT6L2Z22OBZP6DFPCK5KAA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:35.401137462 (600h24m35.272220507s) [upload=526231]: Next returned nil segment Dec 30 15:53:35.401154925 (600h24m35.27223797s) [upload=526231]: Waiting for error group managing segments... Dec 30 15:53:35.673442202 (600h24m35.544525237s) [noise=true , address=145.79.227.23:4545 , piece_id=NHTKPJFEFGAJOEXUNJJ7TU6N27JJP3O5TFSMWF5I2SMGEYXWEAKQ, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.674880822 (600h24m35.545963857s) [noise=true , address=145.79.227.2:4114 , piece_id=3MJV2XXBY776ZUIO2KU4MEP37SP6P2NYFO255DCI6IFUZWUAGOWQ, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.688801769 (600h24m35.559884814s) [noise=true , address=145.79.227.36:4519 , piece_id=5JAZZV45ZTNDSTJZIX2EI7RK7MCBXBQUV2PLSF2HBCRFCIJEA6OQ, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.689704678 (600h24m35.560787713s) [noise=true , address=145.79.227.6:4257 , piece_id=53MG7UDQR3MJYNMQXY3IOPPKOQEOI3KFUTGHQUEHOKCNG5NTAWXA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.690785346 (600h24m35.561868381s) [noise=true , address=145.79.227.14:4838 , piece_id=YB4VPIAQTBJ6G5XFXDVJ3OL6YFAWMEBHHS63SEDX5U4HYKXT3KJQ, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.692882529 (600h24m35.563965564s) [noise=true , address=145.79.227.37:4567 , piece_id=WSHVGPU7L73OET2M6WT7VPHGVRQV6ZCJRPAW7AEXBUQDFWQBNAUA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.696575461 (600h24m35.567658506s) [noise=true , address=145.79.227.16:4910 , piece_id=UHDFUXEQCMN73AUB7WEJBG5HY2G4E25MQGSBNK6AKHMQMDZCJYRA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.697864125 (600h24m35.56894716s) [noise=true , address=145.79.227.19:4077 , piece_id=JEX52J3QTGUWMY3FO6BD5IJV5UPUUCKVV4MYZAOAU3BWMKFSN3KQ, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.758530774 (600h24m35.629613819s) [noise=true , address=145.79.227.48:4303 , piece_id=IO7FUTELAP2L3VVZEQN6N6QG42WMAOS7H5JEPFAST5YEKMKHYLXA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.796785737 (600h24m35.667868772s) [noise=true , address=145.79.227.3:4150 , piece_id=BUBUCPH3LTJO2CR3IVIXEOJOVS422KQT6L2Z22OBZP6DFPCK5KAA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:36.029561278 (600h24m35.900644313s) [noise=true , address=77.90.31.10:4517 , piece_id=5Y32J2NPA664L4ZGRDDYYX66FPWND4DNZ5OZXAADJ3LOXK3IA5BA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:36.070210490 (600h24m35.941293536s) [noise=true , address=45.91.116.13:4153 , piece_id=AQGHSCVDMOODBX225O5SSOLNLJQYPAY2X7EJ7G5W6RP42ZJ4DGCQ, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:36.082591286 (600h24m35.953674331s) [noise=true , address=45.91.116.11:4120 , piece_id=ZTXP5ESUQMWFEWTQUX2SMER7W2NJG5IISRWFAM3IKAKQFKJOGWSQ, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:36.082610784 (600h24m35.953693819s) [seg_pos={0 0}, upload=526231]: Segment reached optimal threshold of 13 pieces. Dec 30 15:53:36.082858675 (600h24m35.953941711s) [noise=true , address=77.90.31.14:4664 , piece_id=YLXNEMZBXUYTFQ4P6ZFHPA6BIUH4TZ2OBXEQAJWYSCUX333L777A, seg_pos={0 0}, upload=526231]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: expected piece hash; context canceled; EOF Dec 30 15:53:36.082868053 (600h24m35.953951088s) [noise=true , address=77.90.31.14:4664 , piece_id=YLXNEMZBXUYTFQ4P6ZFHPA6BIUH4TZ2OBXEQAJWYSCUX333L777A, seg_pos={0 0}, upload=526231]: Long tail cancellation due to optimal threshold Dec 30 15:53:36.082953826 (600h24m35.954036871s) [noise=true , address=45.91.116.12:4227 , piece_id=5GQIOQC4VDIF3Z6J4TUPZG3XUC3AACZ53CPYB4JIBQD7F5BK2IFA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Dec 30 15:53:36.082969977 (600h24m35.954053012s) [noise=true , address=45.91.116.12:4227 , piece_id=5GQIOQC4VDIF3Z6J4TUPZG3XUC3AACZ53CPYB4JIBQD7F5BK2IFA, seg_pos={0 0}, upload=526231]: Long tail cancellation due to optimal threshold Dec 30 15:53:36.083266191 (600h24m35.954349226s) [noise=true , address=45.91.116.8:4047 , piece_id=TLKGZQ2VDRAGBZCULZ35ZR6GEJXQEE2HZ35Q52XBEO5C4FTRFICA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: read tcp 188.166.225.170:42816->45.91.116.8:4047: use of closed network connection; context canceled Dec 30 15:53:36.083274006 (600h24m35.954357041s) [noise=true , address=45.91.116.8:4047 , piece_id=TLKGZQ2VDRAGBZCULZ35ZR6GEJXQEE2HZ35Q52XBEO5C4FTRFICA, seg_pos={0 0}, upload=526231]: Long tail cancellation due to optimal threshold Dec 30 15:53:36.083296048 (600h24m35.954379083s) [seg_pos={0 0}, upload=526231]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 15:53:36.083385679 (600h24m35.954468714s) [upload=526231]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 15:53:36.083392341 (600h24m35.954475376s) [upload=526231]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 15:53:36.379743600 (600h24m36.250826635s) [upload=526231]: Done uploading segments... Dec 30 15:53:36.379758638 (600h24m36.250841673s) [upload=526231]: Upload finished. err: <nil> Duration: 27.711674466s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 44 out of 60 is failed (threshold=6) [XXXX.XX.X.X.X.XXXXXX.XXXX...XX.X.XXXXXXXXX.XXXXXXX...XXXXX.X] duration: 5.001315381s trace: https://jaeger.dev.storj.io/trace/3605fe9c3b54da5a FAIL (2025-12-30 15:57:41 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001397277s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 43 out of 60 is failed (threshold=6) [.XXXX.XX.X.X.X.XXXXXX.XXXX...XX.X.XXXXXXXXX.XXXXXXX...XXXXX.] duration: 3.639622486s trace: https://jaeger.dev.storj.io/trace/51613ef943cde7f1 FAIL (2025-12-30 15:57:22 UTC) orbiter warning: context deadline exceeded Duration: 3.639643617s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 11.391115549s trace: https://jaeger.dev.storj.io/trace/547648a2cfce50e7 FAIL (2025-12-30 15:57:20 UTC) orbiter warning: context deadline exceeded Duration: 11.3911426s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [..X............................X..XX..X..................X..] duration: 15.519909958s trace: https://jaeger.dev.storj.io/trace/2ead4e08bdf51a6a FAIL (2025-12-30 15:57:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 43:4696 , piece_id=3WM5R3KUHXN4WVDOKBAOFX7RAI6ZUTV7UKHUOYAKHUYPM4M24N2A, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.239224572 (600h25m38.110307617s) [noise=true , address=45.91.116.12:4241 , piece_id=FYZGK7TFHSQ5BDLEMJIOGXWVWSZ45NRVSYQ4DT6DHBANOCBOZZWA, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.239811830 (600h25m38.110894865s) [noise=true , address=145.79.227.43:4189 , piece_id=DYHLPLLBNG6B3FXERO33ZDDSHFUHQIIYGQZJFE2UMJXMLUBYC3KQ, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.241164706 (600h25m38.112247741s) [noise=true , address=77.90.31.12:4604 , piece_id=OKY6LAK735LFQ4MDC5K6LJPFQITQSGNOZUGNU6US25RELWLK33BQ, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.244508833 (600h25m38.115591878s) [noise=true , address=145.79.227.49:4350 , piece_id=MA66IA2WTN55SFQDQ4LF7H5LNVUK3IMIHHRUH4TXWHWZPTQDBDCA, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.244943361 (600h25m38.116026396s) [noise=true , address=77.90.31.17:4800 , piece_id=K3HNTJHOSIIUBSKISKWG5YRGLKNKHDOWHITXEDDLKPATPIC4IKTQ, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.250127493 (600h25m38.121210528s) [noise=true , address=145.79.227.35:4516 , piece_id=RDI4UFYB6YC22A5QOGDTLRGCG63TSV6XJFKGLKGSJBC5CH3KCQMQ, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.250776008 (600h25m38.121859053s) [noise=true , address=145.79.227.39:4033 , piece_id=BLTCPLYP5MIQO5Y7WWPT5QIDV6QM3YWAFMLOEWFOVWFLZOO67QSA, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.253191467 (600h25m38.124274512s) [noise=true , address=145.79.227.44:4196 , piece_id=ZUDLORSKFXXIGHZ3YNIE3T7JF7OG55NZKPTSFOUVHAISGF7V7J5Q, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.254554973 (600h25m38.125638018s) [noise=true , address=45.91.116.5:4314 , piece_id=XWWPTNVPYYZ6VL26MACQEJTB2FYN6523YJGZK4I5B4OFRJNUMY3A, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.255815333 (600h25m38.126898378s) [noise=true , address=145.79.227.40:4077 , piece_id=AWXM3BGQVCAARS5F5UISWWIFAVWLWV2KML4FAP6BQ6QNXJTG77NA, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.259122731 (600h25m38.130205776s) [noise=true , address=64.49.13.169:4713 , piece_id=KI4XABHC36ADSOANW3HVM3TCWZ4ZXXXNP7MWJVESDKD5WO26DJHQ, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.265517899 (600h25m38.136600944s) [noise=true , address=77.90.31.19:4854 , piece_id=4W27FHSBO3L2TSAZL2JFCTUPHYV7RI5XDQ4SJWOCRCD5V4K4IK2Q, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.266738002 (600h25m38.137821047s) [noise=true , address=145.79.227.33:4877 , piece_id=GPUOFM2E3KZGDCEZQYPKADLJFDML5DS7OTWFW7AFB5DYAHBVBQ6Q, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.267328907 (600h25m38.138411952s) [noise=true , address=77.90.31.21:4911 , piece_id=B25VZGBKAGIB4ZN6OICFQFZTRQWZIIYDBFUWVNAW4QW26AVYSRPA, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:39.692439689 (600h25m39.563522734s) [upload=526249]: Next returned nil segment Dec 30 15:54:39.692451583 (600h25m39.563534618s) [upload=526249]: Waiting for error group managing segments... Dec 30 15:54:40.135912060 (600h25m40.006995095s) [noise=true , address=145.79.227.39:4033 , piece_id=BLTCPLYP5MIQO5Y7WWPT5QIDV6QM3YWAFMLOEWFOVWFLZOO67QSA, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:40.285764442 (600h25m40.156847487s) [noise=true , address=5.231.73.3:4032 , piece_id=7MBVEHMVMT6XTZJZHHWA65RMFNIB7EH4HU63WTLBPPB6CILGMYAA, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:40.400071651 (600h25m40.271154696s) [noise=true , address=77.90.31.12:4604 , piece_id=OKY6LAK735LFQ4MDC5K6LJPFQITQSGNOZUGNU6US25RELWLK33BQ, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:40.423162570 (600h25m40.294245605s) [noise=true , address=45.91.116.12:4241 , piece_id=FYZGK7TFHSQ5BDLEMJIOGXWVWSZ45NRVSYQ4DT6DHBANOCBOZZWA, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:40.428962032 (600h25m40.300045067s) [noise=true , address=77.90.31.17:4800 , piece_id=K3HNTJHOSIIUBSKISKWG5YRGLKNKHDOWHITXEDDLKPATPIC4IKTQ, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:40.543712124 (600h25m40.414795169s) [noise=true , address=45.91.116.5:4314 , piece_id=XWWPTNVPYYZ6VL26MACQEJTB2FYN6523YJGZK4I5B4OFRJNUMY3A, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:40.993222051 (600h25m40.864305086s) [noise=true , address=145.79.227.44:4196 , piece_id=ZUDLORSKFXXIGHZ3YNIE3T7JF7OG55NZKPTSFOUVHAISGF7V7J5Q, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:41.006811979 (600h25m40.877895024s) [noise=true , address=193.149.164.143:4696 , piece_id=3WM5R3KUHXN4WVDOKBAOFX7RAI6ZUTV7UKHUOYAKHUYPM4M24N2A, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:41.024553631 (600h25m40.895636666s) [noise=true , address=145.79.227.40:4077 , piece_id=AWXM3BGQVCAARS5F5UISWWIFAVWLWV2KML4FAP6BQ6QNXJTG77NA, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:41.054002647 (600h25m40.925085692s) [noise=true , address=145.79.227.49:4350 , piece_id=MA66IA2WTN55SFQDQ4LF7H5LNVUK3IMIHHRUH4TXWHWZPTQDBDCA, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:41.074754852 (600h25m40.945837897s) [noise=true , address=77.90.31.21:4911 , piece_id=B25VZGBKAGIB4ZN6OICFQFZTRQWZIIYDBFUWVNAW4QW26AVYSRPA, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:41.076478905 (600h25m40.94756195s) [noise=true , address=145.79.227.35:4516 , piece_id=RDI4UFYB6YC22A5QOGDTLRGCG63TSV6XJFKGLKGSJBC5CH3KCQMQ, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:41.091294145 (600h25m40.96237718s) [noise=true , address=145.79.227.30:4789 , piece_id=5JZFEA3P6EICC2MGUYOQWY4OWOVHXWZLBJBBSVUAMPLV2EHZOZLA, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:41.138722376 (600h25m41.009805421s) [noise=true , address=145.79.227.49:4359 , piece_id=MCTYZULSW2Z2A6HXPREG3GOEYF5QLBPKEB7DDGT32FNTWHC646CQ, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:41.138745481 (600h25m41.009828516s) [seg_pos={0 0}, upload=526249]: Segment reached optimal threshold of 14 pieces. Dec 30 15:54:41.138965780 (600h25m41.010048815s) [noise=true , address=145.79.227.33:4877 , piece_id=GPUOFM2E3KZGDCEZQYPKADLJFDML5DS7OTWFW7AFB5DYAHBVBQ6Q, seg_pos={0 0}, upload=526249]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17TuauVrjKmehRpXqmK7bemU9V33GDZu9RGBs6fanXvJkV1gdH): protocol: expected piece hash; context canceled; EOF Dec 30 15:54:41.138974496 (600h25m41.010057531s) [noise=true , address=145.79.227.33:4877 , piece_id=GPUOFM2E3KZGDCEZQYPKADLJFDML5DS7OTWFW7AFB5DYAHBVBQ6Q, seg_pos={0 0}, upload=526249]: Long tail cancellation due to optimal threshold Dec 30 15:54:41.139084886 (600h25m41.010167921s) [noise=true , address=77.90.31.19:4854 , piece_id=4W27FHSBO3L2TSAZL2JFCTUPHYV7RI5XDQ4SJWOCRCD5V4K4IK2Q, seg_pos={0 0}, upload=526249]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16HwqYhJnhseWEhzzv4QreZ953rB5cKxqFX8swEqaqiRXCRsAE): protocol: write tcp 188.166.225.170:48144->77.90.31.19:4854: use of closed network connection; write tcp 188.166.225.170:48144->77.90.31.19:4854: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48144->77.90.31.19:4854: use of closed network connection Dec 30 15:54:41.139093533 (600h25m41.010176568s) [noise=true , address=77.90.31.19:4854 , piece_id=4W27FHSBO3L2TSAZL2JFCTUPHYV7RI5XDQ4SJWOCRCD5V4K4IK2Q, seg_pos={0 0}, upload=526249]: Long tail cancellation due to optimal threshold Dec 30 15:54:41.139129352 (600h25m41.010212397s) [noise=true , address=145.79.227.43:4189 , piece_id=DYHLPLLBNG6B3FXERO33ZDDSHFUHQIIYGQZJFE2UMJXMLUBYC3KQ, seg_pos={0 0}, upload=526249]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13mB9AoQJubhEJAAVqzd4o4eq8AfNvLkdqHTDAPN3UyXXwE4B7): protocol: expected piece hash; context canceled; EOF Dec 30 15:54:41.139138429 (600h25m41.010221484s) [noise=true , address=145.79.227.43:4189 , piece_id=DYHLPLLBNG6B3FXERO33ZDDSHFUHQIIYGQZJFE2UMJXMLUBYC3KQ, seg_pos={0 0}, upload=526249]: Long tail cancellation due to optimal threshold Dec 30 15:54:41.139238890 (600h25m41.010321925s) [noise=true , address=64.49.13.169:4713 , piece_id=KI4XABHC36ADSOANW3HVM3TCWZ4ZXXXNP7MWJVESDKD5WO26DJHQ, seg_pos={0 0}, upload=526249]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17z3AiUfbxBX2eMDnUnqo1bbgojuQxJro7PsEaPjYzQjHQ3D6K): protocol: write tcp 188.166.225.170:35558->64.49.13.169:4713: use of closed network connection; write tcp 188.166.225.170:35558->64.49.13.169:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35558->64.49.13.169:4713: use of closed network connection Dec 30 15:54:41.139244420 (600h25m41.010327455s) [noise=true , address=64.49.13.169:4713 , piece_id=KI4XABHC36ADSOANW3HVM3TCWZ4ZXXXNP7MWJVESDKD5WO26DJHQ, seg_pos={0 0}, upload=526249]: Long tail cancellation due to optimal threshold Dec 30 15:54:41.139323681 (600h25m41.010406716s) [seg_pos={0 0}, upload=526249]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 15:54:41.139392121 (600h25m41.010475166s) [upload=526249]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 15:54:41.139396971 (600h25m41.010480006s) [upload=526249]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 15:54:41.438011559 (600h25m41.309094604s) [upload=526249]: Done uploading segments... Dec 30 15:54:41.438036206 (600h25m41.309119251s) [upload=526249]: Upload finished. err: <nil> Duration: 15.519934464s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [....X.X..........XX.......XX.XX..X....X.......X......X.X....] duration: 5.956714898s trace: https://jaeger.dev.storj.io/trace/54fff38414d25b0f FAIL (2025-12-30 15:57:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: AQGHSCVDMOODBX225O5SSOLNLJQYPAY2X7EJ7G5W6RP42ZJ4DGCQ) context deadline exceeded; piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: ZTXP5ESUQMWFEWTQUX2SMER7W2NJG5IISRWFAM3IKAKQFKJOGWSQ) context deadline exceeded ======LOGS====== Dec 30 15:53:33.440030169 (600h24m33.311113204s) [upload=526231]: Starting upload Dec 30 15:53:33.440113437 (600h24m33.311196492s) [upload=526231]: Done starting upload Dec 30 15:53:33.440168081 (600h24m33.311251116s) [upload=526231]: Uploading segments... Dec 30 15:53:33.440357231 (600h24m33.311440276s) [upload=526231]: Got next segment. Inline: false Dec 30 15:53:33.440411114 (600h24m33.311494159s) [upload=526231]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 15:53:33.440416885 (600h24m33.31149992s) [upload=526231]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 15:53:33.684984430 (600h24m33.556067505s) [seg_pos={0 0}, upload=526231]: Begin upload segment... Dec 30 15:53:33.685033994 (600h24m33.556117029s) [seg_pos={0 0}, upload=526231]: Done begin upload segment. Dec 30 15:53:33.685081235 (600h24m33.55616427s) [noise=true , address=145.79.227.36:4519 , piece_id=5JAZZV45ZTNDSTJZIX2EI7RK7MCBXBQUV2PLSF2HBCRFCIJEA6OQ, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.685200161 (600h24m33.556283196s) [noise=true , address=145.79.227.23:4545 , piece_id=NHTKPJFEFGAJOEXUNJJ7TU6N27JJP3O5TFSMWF5I2SMGEYXWEAKQ, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.685709841 (600h24m33.556792887s) [noise=true , address=77.90.31.14:4664 , piece_id=YLXNEMZBXUYTFQ4P6ZFHPA6BIUH4TZ2OBXEQAJWYSCUX333L777A, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.686084574 (600h24m33.557167619s) [noise=true , address=45.91.116.12:4227 , piece_id=5GQIOQC4VDIF3Z6J4TUPZG3XUC3AACZ53CPYB4JIBQD7F5BK2IFA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.686094794 (600h24m33.55717783s) [noise=true , address=77.90.31.10:4517 , piece_id=5Y32J2NPA664L4ZGRDDYYX66FPWND4DNZ5OZXAADJ3LOXK3IA5BA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.686429431 (600h24m33.557512476s) [noise=true , address=45.91.116.13:4153 , piece_id=AQGHSCVDMOODBX225O5SSOLNLJQYPAY2X7EJ7G5W6RP42ZJ4DGCQ, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.686489927 (600h24m33.557572962s) [noise=true , address=45.91.116.8:4047 , piece_id=TLKGZQ2VDRAGBZCULZ35ZR6GEJXQEE2HZ35Q52XBEO5C4FTRFICA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.686910547 (600h24m33.557993582s) [noise=true , address=145.79.227.37:4567 , piece_id=WSHVGPU7L73OET2M6WT7VPHGVRQV6ZCJRPAW7AEXBUQDFWQBNAUA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.686946536 (600h24m33.558029581s) [noise=true , address=145.79.227.14:4838 , piece_id=YB4VPIAQTBJ6G5XFXDVJ3OL6YFAWMEBHHS63SEDX5U4HYKXT3KJQ, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.687266755 (600h24m33.55834979s) [noise=true , address=145.79.227.6:4257 , piece_id=53MG7UDQR3MJYNMQXY3IOPPKOQEOI3KFUTGHQUEHOKCNG5NTAWXA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.687625419 (600h24m33.558708464s) [noise=true , address=45.91.116.11:4120 , piece_id=ZTXP5ESUQMWFEWTQUX2SMER7W2NJG5IISRWFAM3IKAKQFKJOGWSQ, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.688010913 (600h24m33.559093948s) [noise=true , address=145.79.227.19:4077 , piece_id=JEX52J3QTGUWMY3FO6BD5IJV5UPUUCKVV4MYZAOAU3BWMKFSN3KQ, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.688360248 (600h24m33.559443283s) [noise=true , address=145.79.227.16:4910 , piece_id=UHDFUXEQCMN73AUB7WEJBG5HY2G4E25MQGSBNK6AKHMQMDZCJYRA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.688673685 (600h24m33.55975672s) [noise=true , address=145.79.227.48:4303 , piece_id=IO7FUTELAP2L3VVZEQN6N6QG42WMAOS7H5JEPFAST5YEKMKHYLXA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.689057035 (600h24m33.56014007s) [noise=true , address=145.79.227.2:4114 , piece_id=3MJV2XXBY776ZUIO2KU4MEP37SP6P2NYFO255DCI6IFUZWUAGOWQ, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.690890025 (600h24m33.56197308s) [noise=true , address=145.79.227.3:4150 , piece_id=BUBUCPH3LTJO2CR3IVIXEOJOVS422KQT6L2Z22OBZP6DFPCK5KAA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:35.401137462 (600h24m35.272220507s) [upload=526231]: Next returned nil segment Dec 30 15:53:35.401154925 (600h24m35.27223797s) [upload=526231]: Waiting for error group managing segments... Dec 30 15:53:35.673442202 (600h24m35.544525237s) [noise=true , address=145.79.227.23:4545 , piece_id=NHTKPJFEFGAJOEXUNJJ7TU6N27JJP3O5TFSMWF5I2SMGEYXWEAKQ, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.674880822 (600h24m35.545963857s) [noise=true , address=145.79.227.2:4114 , piece_id=3MJV2XXBY776ZUIO2KU4MEP37SP6P2NYFO255DCI6IFUZWUAGOWQ, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.688801769 (600h24m35.559884814s) [noise=true , address=145.79.227.36:4519 , piece_id=5JAZZV45ZTNDSTJZIX2EI7RK7MCBXBQUV2PLSF2HBCRFCIJEA6OQ, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.689704678 (600h24m35.560787713s) [noise=true , address=145.79.227.6:4257 , piece_id=53MG7UDQR3MJYNMQXY3IOPPKOQEOI3KFUTGHQUEHOKCNG5NTAWXA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.690785346 (600h24m35.561868381s) [noise=true , address=145.79.227.14:4838 , piece_id=YB4VPIAQTBJ6G5XFXDVJ3OL6YFAWMEBHHS63SEDX5U4HYKXT3KJQ, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.692882529 (600h24m35.563965564s) [noise=true , address=145.79.227.37:4567 , piece_id=WSHVGPU7L73OET2M6WT7VPHGVRQV6ZCJRPAW7AEXBUQDFWQBNAUA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.696575461 (600h24m35.567658506s) [noise=true , address=145.79.227.16:4910 , piece_id=UHDFUXEQCMN73AUB7WEJBG5HY2G4E25MQGSBNK6AKHMQMDZCJYRA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.697864125 (600h24m35.56894716s) [noise=true , address=145.79.227.19:4077 , piece_id=JEX52J3QTGUWMY3FO6BD5IJV5UPUUCKVV4MYZAOAU3BWMKFSN3KQ, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.758530774 (600h24m35.629613819s) [noise=true , address=145.79.227.48:4303 , piece_id=IO7FUTELAP2L3VVZEQN6N6QG42WMAOS7H5JEPFAST5YEKMKHYLXA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.796785737 (600h24m35.667868772s) [noise=true , address=145.79.227.3:4150 , piece_id=BUBUCPH3LTJO2CR3IVIXEOJOVS422KQT6L2Z22OBZP6DFPCK5KAA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:36.029561278 (600h24m35.900644313s) [noise=true , address=77.90.31.10:4517 , piece_id=5Y32J2NPA664L4ZGRDDYYX66FPWND4DNZ5OZXAADJ3LOXK3IA5BA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:36.070210490 (600h24m35.941293536s) [noise=true , address=45.91.116.13:4153 , piece_id=AQGHSCVDMOODBX225O5SSOLNLJQYPAY2X7EJ7G5W6RP42ZJ4DGCQ, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:36.082591286 (600h24m35.953674331s) [noise=true , address=45.91.116.11:4120 , piece_id=ZTXP5ESUQMWFEWTQUX2SMER7W2NJG5IISRWFAM3IKAKQFKJOGWSQ, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:36.082610784 (600h24m35.953693819s) [seg_pos={0 0}, upload=526231]: Segment reached optimal threshold of 13 pieces. Dec 30 15:53:36.082858675 (600h24m35.953941711s) [noise=true , address=77.90.31.14:4664 , piece_id=YLXNEMZBXUYTFQ4P6ZFHPA6BIUH4TZ2OBXEQAJWYSCUX333L777A, seg_pos={0 0}, upload=526231]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: expected piece hash; context canceled; EOF Dec 30 15:53:36.082868053 (600h24m35.953951088s) [noise=true , address=77.90.31.14:4664 , piece_id=YLXNEMZBXUYTFQ4P6ZFHPA6BIUH4TZ2OBXEQAJWYSCUX333L777A, seg_pos={0 0}, upload=526231]: Long tail cancellation due to optimal threshold Dec 30 15:53:36.082953826 (600h24m35.954036871s) [noise=true , address=45.91.116.12:4227 , piece_id=5GQIOQC4VDIF3Z6J4TUPZG3XUC3AACZ53CPYB4JIBQD7F5BK2IFA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Dec 30 15:53:36.082969977 (600h24m35.954053012s) [noise=true , address=45.91.116.12:4227 , piece_id=5GQIOQC4VDIF3Z6J4TUPZG3XUC3AACZ53CPYB4JIBQD7F5BK2IFA, seg_pos={0 0}, upload=526231]: Long tail cancellation due to optimal threshold Dec 30 15:53:36.083266191 (600h24m35.954349226s) [noise=true , address=45.91.116.8:4047 , piece_id=TLKGZQ2VDRAGBZCULZ35ZR6GEJXQEE2HZ35Q52XBEO5C4FTRFICA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: read tcp 188.166.225.170:42816->45.91.116.8:4047: use of closed network connection; context canceled Dec 30 15:53:36.083274006 (600h24m35.954357041s) [noise=true , address=45.91.116.8:4047 , piece_id=TLKGZQ2VDRAGBZCULZ35ZR6GEJXQEE2HZ35Q52XBEO5C4FTRFICA, seg_pos={0 0}, upload=526231]: Long tail cancellation due to optimal threshold Dec 30 15:53:36.083296048 (600h24m35.954379083s) [seg_pos={0 0}, upload=526231]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 15:53:36.083385679 (600h24m35.954468714s) [upload=526231]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 15:53:36.083392341 (600h24m35.954475376s) [upload=526231]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 15:53:36.379743600 (600h24m36.250826635s) [upload=526231]: Done uploading segments... Dec 30 15:53:36.379758638 (600h24m36.250841673s) [upload=526231]: Upload finished. err: <nil> Duration: 5.956746007s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 12.379093807s trace: https://jaeger.dev.storj.io/trace/547648a2cfce50e7 FAIL (2025-12-30 15:56:46 UTC) orbiter warning: context deadline exceeded Duration: 12.379120537s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 43 out of 60 is failed (threshold=6) [..XXXX.XX.X.X.X.XXXXXX.XXXX...XX.X.XXXXXXXXX.XXXXXXX...XXXXX] duration: 5.001177686s trace: https://jaeger.dev.storj.io/trace/51613ef943cde7f1 FAIL (2025-12-30 15:56:30 UTC) orbiter warning: context deadline exceeded Duration: 5.001200029s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 11.347210564s trace: https://jaeger.dev.storj.io/trace/547648a2cfce50e7 FAIL (2025-12-30 15:56:17 UTC) orbiter warning: context deadline exceeded Duration: 11.347285768s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.....X.X..........XX.......XX.XX..X....X.......X......X.X...] duration: 5.612959131s trace: https://jaeger.dev.storj.io/trace/54fff38414d25b0f FAIL (2025-12-30 15:56:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: AQGHSCVDMOODBX225O5SSOLNLJQYPAY2X7EJ7G5W6RP42ZJ4DGCQ) context deadline exceeded; piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: ZTXP5ESUQMWFEWTQUX2SMER7W2NJG5IISRWFAM3IKAKQFKJOGWSQ) context deadline exceeded ======LOGS====== Dec 30 15:53:33.440030169 (600h24m33.311113204s) [upload=526231]: Starting upload Dec 30 15:53:33.440113437 (600h24m33.311196492s) [upload=526231]: Done starting upload Dec 30 15:53:33.440168081 (600h24m33.311251116s) [upload=526231]: Uploading segments... Dec 30 15:53:33.440357231 (600h24m33.311440276s) [upload=526231]: Got next segment. Inline: false Dec 30 15:53:33.440411114 (600h24m33.311494159s) [upload=526231]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 15:53:33.440416885 (600h24m33.31149992s) [upload=526231]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 15:53:33.684984430 (600h24m33.556067505s) [seg_pos={0 0}, upload=526231]: Begin upload segment... Dec 30 15:53:33.685033994 (600h24m33.556117029s) [seg_pos={0 0}, upload=526231]: Done begin upload segment. Dec 30 15:53:33.685081235 (600h24m33.55616427s) [noise=true , address=145.79.227.36:4519 , piece_id=5JAZZV45ZTNDSTJZIX2EI7RK7MCBXBQUV2PLSF2HBCRFCIJEA6OQ, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.685200161 (600h24m33.556283196s) [noise=true , address=145.79.227.23:4545 , piece_id=NHTKPJFEFGAJOEXUNJJ7TU6N27JJP3O5TFSMWF5I2SMGEYXWEAKQ, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.685709841 (600h24m33.556792887s) [noise=true , address=77.90.31.14:4664 , piece_id=YLXNEMZBXUYTFQ4P6ZFHPA6BIUH4TZ2OBXEQAJWYSCUX333L777A, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.686084574 (600h24m33.557167619s) [noise=true , address=45.91.116.12:4227 , piece_id=5GQIOQC4VDIF3Z6J4TUPZG3XUC3AACZ53CPYB4JIBQD7F5BK2IFA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.686094794 (600h24m33.55717783s) [noise=true , address=77.90.31.10:4517 , piece_id=5Y32J2NPA664L4ZGRDDYYX66FPWND4DNZ5OZXAADJ3LOXK3IA5BA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.686429431 (600h24m33.557512476s) [noise=true , address=45.91.116.13:4153 , piece_id=AQGHSCVDMOODBX225O5SSOLNLJQYPAY2X7EJ7G5W6RP42ZJ4DGCQ, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.686489927 (600h24m33.557572962s) [noise=true , address=45.91.116.8:4047 , piece_id=TLKGZQ2VDRAGBZCULZ35ZR6GEJXQEE2HZ35Q52XBEO5C4FTRFICA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.686910547 (600h24m33.557993582s) [noise=true , address=145.79.227.37:4567 , piece_id=WSHVGPU7L73OET2M6WT7VPHGVRQV6ZCJRPAW7AEXBUQDFWQBNAUA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.686946536 (600h24m33.558029581s) [noise=true , address=145.79.227.14:4838 , piece_id=YB4VPIAQTBJ6G5XFXDVJ3OL6YFAWMEBHHS63SEDX5U4HYKXT3KJQ, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.687266755 (600h24m33.55834979s) [noise=true , address=145.79.227.6:4257 , piece_id=53MG7UDQR3MJYNMQXY3IOPPKOQEOI3KFUTGHQUEHOKCNG5NTAWXA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.687625419 (600h24m33.558708464s) [noise=true , address=45.91.116.11:4120 , piece_id=ZTXP5ESUQMWFEWTQUX2SMER7W2NJG5IISRWFAM3IKAKQFKJOGWSQ, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.688010913 (600h24m33.559093948s) [noise=true , address=145.79.227.19:4077 , piece_id=JEX52J3QTGUWMY3FO6BD5IJV5UPUUCKVV4MYZAOAU3BWMKFSN3KQ, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.688360248 (600h24m33.559443283s) [noise=true , address=145.79.227.16:4910 , piece_id=UHDFUXEQCMN73AUB7WEJBG5HY2G4E25MQGSBNK6AKHMQMDZCJYRA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.688673685 (600h24m33.55975672s) [noise=true , address=145.79.227.48:4303 , piece_id=IO7FUTELAP2L3VVZEQN6N6QG42WMAOS7H5JEPFAST5YEKMKHYLXA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.689057035 (600h24m33.56014007s) [noise=true , address=145.79.227.2:4114 , piece_id=3MJV2XXBY776ZUIO2KU4MEP37SP6P2NYFO255DCI6IFUZWUAGOWQ, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.690890025 (600h24m33.56197308s) [noise=true , address=145.79.227.3:4150 , piece_id=BUBUCPH3LTJO2CR3IVIXEOJOVS422KQT6L2Z22OBZP6DFPCK5KAA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:35.401137462 (600h24m35.272220507s) [upload=526231]: Next returned nil segment Dec 30 15:53:35.401154925 (600h24m35.27223797s) [upload=526231]: Waiting for error group managing segments... Dec 30 15:53:35.673442202 (600h24m35.544525237s) [noise=true , address=145.79.227.23:4545 , piece_id=NHTKPJFEFGAJOEXUNJJ7TU6N27JJP3O5TFSMWF5I2SMGEYXWEAKQ, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.674880822 (600h24m35.545963857s) [noise=true , address=145.79.227.2:4114 , piece_id=3MJV2XXBY776ZUIO2KU4MEP37SP6P2NYFO255DCI6IFUZWUAGOWQ, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.688801769 (600h24m35.559884814s) [noise=true , address=145.79.227.36:4519 , piece_id=5JAZZV45ZTNDSTJZIX2EI7RK7MCBXBQUV2PLSF2HBCRFCIJEA6OQ, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.689704678 (600h24m35.560787713s) [noise=true , address=145.79.227.6:4257 , piece_id=53MG7UDQR3MJYNMQXY3IOPPKOQEOI3KFUTGHQUEHOKCNG5NTAWXA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.690785346 (600h24m35.561868381s) [noise=true , address=145.79.227.14:4838 , piece_id=YB4VPIAQTBJ6G5XFXDVJ3OL6YFAWMEBHHS63SEDX5U4HYKXT3KJQ, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.692882529 (600h24m35.563965564s) [noise=true , address=145.79.227.37:4567 , piece_id=WSHVGPU7L73OET2M6WT7VPHGVRQV6ZCJRPAW7AEXBUQDFWQBNAUA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.696575461 (600h24m35.567658506s) [noise=true , address=145.79.227.16:4910 , piece_id=UHDFUXEQCMN73AUB7WEJBG5HY2G4E25MQGSBNK6AKHMQMDZCJYRA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.697864125 (600h24m35.56894716s) [noise=true , address=145.79.227.19:4077 , piece_id=JEX52J3QTGUWMY3FO6BD5IJV5UPUUCKVV4MYZAOAU3BWMKFSN3KQ, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.758530774 (600h24m35.629613819s) [noise=true , address=145.79.227.48:4303 , piece_id=IO7FUTELAP2L3VVZEQN6N6QG42WMAOS7H5JEPFAST5YEKMKHYLXA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.796785737 (600h24m35.667868772s) [noise=true , address=145.79.227.3:4150 , piece_id=BUBUCPH3LTJO2CR3IVIXEOJOVS422KQT6L2Z22OBZP6DFPCK5KAA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:36.029561278 (600h24m35.900644313s) [noise=true , address=77.90.31.10:4517 , piece_id=5Y32J2NPA664L4ZGRDDYYX66FPWND4DNZ5OZXAADJ3LOXK3IA5BA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:36.070210490 (600h24m35.941293536s) [noise=true , address=45.91.116.13:4153 , piece_id=AQGHSCVDMOODBX225O5SSOLNLJQYPAY2X7EJ7G5W6RP42ZJ4DGCQ, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:36.082591286 (600h24m35.953674331s) [noise=true , address=45.91.116.11:4120 , piece_id=ZTXP5ESUQMWFEWTQUX2SMER7W2NJG5IISRWFAM3IKAKQFKJOGWSQ, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:36.082610784 (600h24m35.953693819s) [seg_pos={0 0}, upload=526231]: Segment reached optimal threshold of 13 pieces. Dec 30 15:53:36.082858675 (600h24m35.953941711s) [noise=true , address=77.90.31.14:4664 , piece_id=YLXNEMZBXUYTFQ4P6ZFHPA6BIUH4TZ2OBXEQAJWYSCUX333L777A, seg_pos={0 0}, upload=526231]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: expected piece hash; context canceled; EOF Dec 30 15:53:36.082868053 (600h24m35.953951088s) [noise=true , address=77.90.31.14:4664 , piece_id=YLXNEMZBXUYTFQ4P6ZFHPA6BIUH4TZ2OBXEQAJWYSCUX333L777A, seg_pos={0 0}, upload=526231]: Long tail cancellation due to optimal threshold Dec 30 15:53:36.082953826 (600h24m35.954036871s) [noise=true , address=45.91.116.12:4227 , piece_id=5GQIOQC4VDIF3Z6J4TUPZG3XUC3AACZ53CPYB4JIBQD7F5BK2IFA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Dec 30 15:53:36.082969977 (600h24m35.954053012s) [noise=true , address=45.91.116.12:4227 , piece_id=5GQIOQC4VDIF3Z6J4TUPZG3XUC3AACZ53CPYB4JIBQD7F5BK2IFA, seg_pos={0 0}, upload=526231]: Long tail cancellation due to optimal threshold Dec 30 15:53:36.083266191 (600h24m35.954349226s) [noise=true , address=45.91.116.8:4047 , piece_id=TLKGZQ2VDRAGBZCULZ35ZR6GEJXQEE2HZ35Q52XBEO5C4FTRFICA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: read tcp 188.166.225.170:42816->45.91.116.8:4047: use of closed network connection; context canceled Dec 30 15:53:36.083274006 (600h24m35.954357041s) [noise=true , address=45.91.116.8:4047 , piece_id=TLKGZQ2VDRAGBZCULZ35ZR6GEJXQEE2HZ35Q52XBEO5C4FTRFICA, seg_pos={0 0}, upload=526231]: Long tail cancellation due to optimal threshold Dec 30 15:53:36.083296048 (600h24m35.954379083s) [seg_pos={0 0}, upload=526231]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 15:53:36.083385679 (600h24m35.954468714s) [upload=526231]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 15:53:36.083392341 (600h24m35.954475376s) [upload=526231]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 15:53:36.379743600 (600h24m36.250826635s) [upload=526231]: Done uploading segments... Dec 30 15:53:36.379758638 (600h24m36.250841673s) [upload=526231]: Upload finished. err: <nil> Duration: 5.6129892s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [...X............................X..XX..X..................X.] duration: 26.475847194s trace: https://jaeger.dev.storj.io/trace/2ead4e08bdf51a6a FAIL (2025-12-30 15:56:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 43:4696 , piece_id=3WM5R3KUHXN4WVDOKBAOFX7RAI6ZUTV7UKHUOYAKHUYPM4M24N2A, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.239224572 (600h25m38.110307617s) [noise=true , address=45.91.116.12:4241 , piece_id=FYZGK7TFHSQ5BDLEMJIOGXWVWSZ45NRVSYQ4DT6DHBANOCBOZZWA, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.239811830 (600h25m38.110894865s) [noise=true , address=145.79.227.43:4189 , piece_id=DYHLPLLBNG6B3FXERO33ZDDSHFUHQIIYGQZJFE2UMJXMLUBYC3KQ, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.241164706 (600h25m38.112247741s) [noise=true , address=77.90.31.12:4604 , piece_id=OKY6LAK735LFQ4MDC5K6LJPFQITQSGNOZUGNU6US25RELWLK33BQ, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.244508833 (600h25m38.115591878s) [noise=true , address=145.79.227.49:4350 , piece_id=MA66IA2WTN55SFQDQ4LF7H5LNVUK3IMIHHRUH4TXWHWZPTQDBDCA, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.244943361 (600h25m38.116026396s) [noise=true , address=77.90.31.17:4800 , piece_id=K3HNTJHOSIIUBSKISKWG5YRGLKNKHDOWHITXEDDLKPATPIC4IKTQ, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.250127493 (600h25m38.121210528s) [noise=true , address=145.79.227.35:4516 , piece_id=RDI4UFYB6YC22A5QOGDTLRGCG63TSV6XJFKGLKGSJBC5CH3KCQMQ, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.250776008 (600h25m38.121859053s) [noise=true , address=145.79.227.39:4033 , piece_id=BLTCPLYP5MIQO5Y7WWPT5QIDV6QM3YWAFMLOEWFOVWFLZOO67QSA, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.253191467 (600h25m38.124274512s) [noise=true , address=145.79.227.44:4196 , piece_id=ZUDLORSKFXXIGHZ3YNIE3T7JF7OG55NZKPTSFOUVHAISGF7V7J5Q, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.254554973 (600h25m38.125638018s) [noise=true , address=45.91.116.5:4314 , piece_id=XWWPTNVPYYZ6VL26MACQEJTB2FYN6523YJGZK4I5B4OFRJNUMY3A, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.255815333 (600h25m38.126898378s) [noise=true , address=145.79.227.40:4077 , piece_id=AWXM3BGQVCAARS5F5UISWWIFAVWLWV2KML4FAP6BQ6QNXJTG77NA, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.259122731 (600h25m38.130205776s) [noise=true , address=64.49.13.169:4713 , piece_id=KI4XABHC36ADSOANW3HVM3TCWZ4ZXXXNP7MWJVESDKD5WO26DJHQ, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.265517899 (600h25m38.136600944s) [noise=true , address=77.90.31.19:4854 , piece_id=4W27FHSBO3L2TSAZL2JFCTUPHYV7RI5XDQ4SJWOCRCD5V4K4IK2Q, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.266738002 (600h25m38.137821047s) [noise=true , address=145.79.227.33:4877 , piece_id=GPUOFM2E3KZGDCEZQYPKADLJFDML5DS7OTWFW7AFB5DYAHBVBQ6Q, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.267328907 (600h25m38.138411952s) [noise=true , address=77.90.31.21:4911 , piece_id=B25VZGBKAGIB4ZN6OICFQFZTRQWZIIYDBFUWVNAW4QW26AVYSRPA, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:39.692439689 (600h25m39.563522734s) [upload=526249]: Next returned nil segment Dec 30 15:54:39.692451583 (600h25m39.563534618s) [upload=526249]: Waiting for error group managing segments... Dec 30 15:54:40.135912060 (600h25m40.006995095s) [noise=true , address=145.79.227.39:4033 , piece_id=BLTCPLYP5MIQO5Y7WWPT5QIDV6QM3YWAFMLOEWFOVWFLZOO67QSA, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:40.285764442 (600h25m40.156847487s) [noise=true , address=5.231.73.3:4032 , piece_id=7MBVEHMVMT6XTZJZHHWA65RMFNIB7EH4HU63WTLBPPB6CILGMYAA, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:40.400071651 (600h25m40.271154696s) [noise=true , address=77.90.31.12:4604 , piece_id=OKY6LAK735LFQ4MDC5K6LJPFQITQSGNOZUGNU6US25RELWLK33BQ, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:40.423162570 (600h25m40.294245605s) [noise=true , address=45.91.116.12:4241 , piece_id=FYZGK7TFHSQ5BDLEMJIOGXWVWSZ45NRVSYQ4DT6DHBANOCBOZZWA, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:40.428962032 (600h25m40.300045067s) [noise=true , address=77.90.31.17:4800 , piece_id=K3HNTJHOSIIUBSKISKWG5YRGLKNKHDOWHITXEDDLKPATPIC4IKTQ, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:40.543712124 (600h25m40.414795169s) [noise=true , address=45.91.116.5:4314 , piece_id=XWWPTNVPYYZ6VL26MACQEJTB2FYN6523YJGZK4I5B4OFRJNUMY3A, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:40.993222051 (600h25m40.864305086s) [noise=true , address=145.79.227.44:4196 , piece_id=ZUDLORSKFXXIGHZ3YNIE3T7JF7OG55NZKPTSFOUVHAISGF7V7J5Q, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:41.006811979 (600h25m40.877895024s) [noise=true , address=193.149.164.143:4696 , piece_id=3WM5R3KUHXN4WVDOKBAOFX7RAI6ZUTV7UKHUOYAKHUYPM4M24N2A, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:41.024553631 (600h25m40.895636666s) [noise=true , address=145.79.227.40:4077 , piece_id=AWXM3BGQVCAARS5F5UISWWIFAVWLWV2KML4FAP6BQ6QNXJTG77NA, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:41.054002647 (600h25m40.925085692s) [noise=true , address=145.79.227.49:4350 , piece_id=MA66IA2WTN55SFQDQ4LF7H5LNVUK3IMIHHRUH4TXWHWZPTQDBDCA, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:41.074754852 (600h25m40.945837897s) [noise=true , address=77.90.31.21:4911 , piece_id=B25VZGBKAGIB4ZN6OICFQFZTRQWZIIYDBFUWVNAW4QW26AVYSRPA, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:41.076478905 (600h25m40.94756195s) [noise=true , address=145.79.227.35:4516 , piece_id=RDI4UFYB6YC22A5QOGDTLRGCG63TSV6XJFKGLKGSJBC5CH3KCQMQ, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:41.091294145 (600h25m40.96237718s) [noise=true , address=145.79.227.30:4789 , piece_id=5JZFEA3P6EICC2MGUYOQWY4OWOVHXWZLBJBBSVUAMPLV2EHZOZLA, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:41.138722376 (600h25m41.009805421s) [noise=true , address=145.79.227.49:4359 , piece_id=MCTYZULSW2Z2A6HXPREG3GOEYF5QLBPKEB7DDGT32FNTWHC646CQ, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:41.138745481 (600h25m41.009828516s) [seg_pos={0 0}, upload=526249]: Segment reached optimal threshold of 14 pieces. Dec 30 15:54:41.138965780 (600h25m41.010048815s) [noise=true , address=145.79.227.33:4877 , piece_id=GPUOFM2E3KZGDCEZQYPKADLJFDML5DS7OTWFW7AFB5DYAHBVBQ6Q, seg_pos={0 0}, upload=526249]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17TuauVrjKmehRpXqmK7bemU9V33GDZu9RGBs6fanXvJkV1gdH): protocol: expected piece hash; context canceled; EOF Dec 30 15:54:41.138974496 (600h25m41.010057531s) [noise=true , address=145.79.227.33:4877 , piece_id=GPUOFM2E3KZGDCEZQYPKADLJFDML5DS7OTWFW7AFB5DYAHBVBQ6Q, seg_pos={0 0}, upload=526249]: Long tail cancellation due to optimal threshold Dec 30 15:54:41.139084886 (600h25m41.010167921s) [noise=true , address=77.90.31.19:4854 , piece_id=4W27FHSBO3L2TSAZL2JFCTUPHYV7RI5XDQ4SJWOCRCD5V4K4IK2Q, seg_pos={0 0}, upload=526249]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16HwqYhJnhseWEhzzv4QreZ953rB5cKxqFX8swEqaqiRXCRsAE): protocol: write tcp 188.166.225.170:48144->77.90.31.19:4854: use of closed network connection; write tcp 188.166.225.170:48144->77.90.31.19:4854: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48144->77.90.31.19:4854: use of closed network connection Dec 30 15:54:41.139093533 (600h25m41.010176568s) [noise=true , address=77.90.31.19:4854 , piece_id=4W27FHSBO3L2TSAZL2JFCTUPHYV7RI5XDQ4SJWOCRCD5V4K4IK2Q, seg_pos={0 0}, upload=526249]: Long tail cancellation due to optimal threshold Dec 30 15:54:41.139129352 (600h25m41.010212397s) [noise=true , address=145.79.227.43:4189 , piece_id=DYHLPLLBNG6B3FXERO33ZDDSHFUHQIIYGQZJFE2UMJXMLUBYC3KQ, seg_pos={0 0}, upload=526249]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13mB9AoQJubhEJAAVqzd4o4eq8AfNvLkdqHTDAPN3UyXXwE4B7): protocol: expected piece hash; context canceled; EOF Dec 30 15:54:41.139138429 (600h25m41.010221484s) [noise=true , address=145.79.227.43:4189 , piece_id=DYHLPLLBNG6B3FXERO33ZDDSHFUHQIIYGQZJFE2UMJXMLUBYC3KQ, seg_pos={0 0}, upload=526249]: Long tail cancellation due to optimal threshold Dec 30 15:54:41.139238890 (600h25m41.010321925s) [noise=true , address=64.49.13.169:4713 , piece_id=KI4XABHC36ADSOANW3HVM3TCWZ4ZXXXNP7MWJVESDKD5WO26DJHQ, seg_pos={0 0}, upload=526249]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17z3AiUfbxBX2eMDnUnqo1bbgojuQxJro7PsEaPjYzQjHQ3D6K): protocol: write tcp 188.166.225.170:35558->64.49.13.169:4713: use of closed network connection; write tcp 188.166.225.170:35558->64.49.13.169:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35558->64.49.13.169:4713: use of closed network connection Dec 30 15:54:41.139244420 (600h25m41.010327455s) [noise=true , address=64.49.13.169:4713 , piece_id=KI4XABHC36ADSOANW3HVM3TCWZ4ZXXXNP7MWJVESDKD5WO26DJHQ, seg_pos={0 0}, upload=526249]: Long tail cancellation due to optimal threshold Dec 30 15:54:41.139323681 (600h25m41.010406716s) [seg_pos={0 0}, upload=526249]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 15:54:41.139392121 (600h25m41.010475166s) [upload=526249]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 15:54:41.139396971 (600h25m41.010480006s) [upload=526249]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 15:54:41.438011559 (600h25m41.309094604s) [upload=526249]: Done uploading segments... Dec 30 15:54:41.438036206 (600h25m41.309119251s) [upload=526249]: Upload finished. err: <nil> Duration: 26.475861562s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 9.759105695s trace: https://jaeger.dev.storj.io/trace/547648a2cfce50e7 FAIL (2025-12-30 15:55:41 UTC) orbiter warning: context deadline exceeded Duration: 9.759133278s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [....X............................X..XX..X..................X] duration: 1m0.003159686s trace: https://jaeger.dev.storj.io/trace/2ead4e08bdf51a6a FAIL (2025-12-30 15:55:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 43:4696 , piece_id=3WM5R3KUHXN4WVDOKBAOFX7RAI6ZUTV7UKHUOYAKHUYPM4M24N2A, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.239224572 (600h25m38.110307617s) [noise=true , address=45.91.116.12:4241 , piece_id=FYZGK7TFHSQ5BDLEMJIOGXWVWSZ45NRVSYQ4DT6DHBANOCBOZZWA, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.239811830 (600h25m38.110894865s) [noise=true , address=145.79.227.43:4189 , piece_id=DYHLPLLBNG6B3FXERO33ZDDSHFUHQIIYGQZJFE2UMJXMLUBYC3KQ, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.241164706 (600h25m38.112247741s) [noise=true , address=77.90.31.12:4604 , piece_id=OKY6LAK735LFQ4MDC5K6LJPFQITQSGNOZUGNU6US25RELWLK33BQ, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.244508833 (600h25m38.115591878s) [noise=true , address=145.79.227.49:4350 , piece_id=MA66IA2WTN55SFQDQ4LF7H5LNVUK3IMIHHRUH4TXWHWZPTQDBDCA, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.244943361 (600h25m38.116026396s) [noise=true , address=77.90.31.17:4800 , piece_id=K3HNTJHOSIIUBSKISKWG5YRGLKNKHDOWHITXEDDLKPATPIC4IKTQ, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.250127493 (600h25m38.121210528s) [noise=true , address=145.79.227.35:4516 , piece_id=RDI4UFYB6YC22A5QOGDTLRGCG63TSV6XJFKGLKGSJBC5CH3KCQMQ, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.250776008 (600h25m38.121859053s) [noise=true , address=145.79.227.39:4033 , piece_id=BLTCPLYP5MIQO5Y7WWPT5QIDV6QM3YWAFMLOEWFOVWFLZOO67QSA, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.253191467 (600h25m38.124274512s) [noise=true , address=145.79.227.44:4196 , piece_id=ZUDLORSKFXXIGHZ3YNIE3T7JF7OG55NZKPTSFOUVHAISGF7V7J5Q, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.254554973 (600h25m38.125638018s) [noise=true , address=45.91.116.5:4314 , piece_id=XWWPTNVPYYZ6VL26MACQEJTB2FYN6523YJGZK4I5B4OFRJNUMY3A, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.255815333 (600h25m38.126898378s) [noise=true , address=145.79.227.40:4077 , piece_id=AWXM3BGQVCAARS5F5UISWWIFAVWLWV2KML4FAP6BQ6QNXJTG77NA, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.259122731 (600h25m38.130205776s) [noise=true , address=64.49.13.169:4713 , piece_id=KI4XABHC36ADSOANW3HVM3TCWZ4ZXXXNP7MWJVESDKD5WO26DJHQ, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.265517899 (600h25m38.136600944s) [noise=true , address=77.90.31.19:4854 , piece_id=4W27FHSBO3L2TSAZL2JFCTUPHYV7RI5XDQ4SJWOCRCD5V4K4IK2Q, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.266738002 (600h25m38.137821047s) [noise=true , address=145.79.227.33:4877 , piece_id=GPUOFM2E3KZGDCEZQYPKADLJFDML5DS7OTWFW7AFB5DYAHBVBQ6Q, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.267328907 (600h25m38.138411952s) [noise=true , address=77.90.31.21:4911 , piece_id=B25VZGBKAGIB4ZN6OICFQFZTRQWZIIYDBFUWVNAW4QW26AVYSRPA, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:39.692439689 (600h25m39.563522734s) [upload=526249]: Next returned nil segment Dec 30 15:54:39.692451583 (600h25m39.563534618s) [upload=526249]: Waiting for error group managing segments... Dec 30 15:54:40.135912060 (600h25m40.006995095s) [noise=true , address=145.79.227.39:4033 , piece_id=BLTCPLYP5MIQO5Y7WWPT5QIDV6QM3YWAFMLOEWFOVWFLZOO67QSA, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:40.285764442 (600h25m40.156847487s) [noise=true , address=5.231.73.3:4032 , piece_id=7MBVEHMVMT6XTZJZHHWA65RMFNIB7EH4HU63WTLBPPB6CILGMYAA, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:40.400071651 (600h25m40.271154696s) [noise=true , address=77.90.31.12:4604 , piece_id=OKY6LAK735LFQ4MDC5K6LJPFQITQSGNOZUGNU6US25RELWLK33BQ, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:40.423162570 (600h25m40.294245605s) [noise=true , address=45.91.116.12:4241 , piece_id=FYZGK7TFHSQ5BDLEMJIOGXWVWSZ45NRVSYQ4DT6DHBANOCBOZZWA, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:40.428962032 (600h25m40.300045067s) [noise=true , address=77.90.31.17:4800 , piece_id=K3HNTJHOSIIUBSKISKWG5YRGLKNKHDOWHITXEDDLKPATPIC4IKTQ, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:40.543712124 (600h25m40.414795169s) [noise=true , address=45.91.116.5:4314 , piece_id=XWWPTNVPYYZ6VL26MACQEJTB2FYN6523YJGZK4I5B4OFRJNUMY3A, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:40.993222051 (600h25m40.864305086s) [noise=true , address=145.79.227.44:4196 , piece_id=ZUDLORSKFXXIGHZ3YNIE3T7JF7OG55NZKPTSFOUVHAISGF7V7J5Q, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:41.006811979 (600h25m40.877895024s) [noise=true , address=193.149.164.143:4696 , piece_id=3WM5R3KUHXN4WVDOKBAOFX7RAI6ZUTV7UKHUOYAKHUYPM4M24N2A, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:41.024553631 (600h25m40.895636666s) [noise=true , address=145.79.227.40:4077 , piece_id=AWXM3BGQVCAARS5F5UISWWIFAVWLWV2KML4FAP6BQ6QNXJTG77NA, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:41.054002647 (600h25m40.925085692s) [noise=true , address=145.79.227.49:4350 , piece_id=MA66IA2WTN55SFQDQ4LF7H5LNVUK3IMIHHRUH4TXWHWZPTQDBDCA, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:41.074754852 (600h25m40.945837897s) [noise=true , address=77.90.31.21:4911 , piece_id=B25VZGBKAGIB4ZN6OICFQFZTRQWZIIYDBFUWVNAW4QW26AVYSRPA, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:41.076478905 (600h25m40.94756195s) [noise=true , address=145.79.227.35:4516 , piece_id=RDI4UFYB6YC22A5QOGDTLRGCG63TSV6XJFKGLKGSJBC5CH3KCQMQ, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:41.091294145 (600h25m40.96237718s) [noise=true , address=145.79.227.30:4789 , piece_id=5JZFEA3P6EICC2MGUYOQWY4OWOVHXWZLBJBBSVUAMPLV2EHZOZLA, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:41.138722376 (600h25m41.009805421s) [noise=true , address=145.79.227.49:4359 , piece_id=MCTYZULSW2Z2A6HXPREG3GOEYF5QLBPKEB7DDGT32FNTWHC646CQ, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:41.138745481 (600h25m41.009828516s) [seg_pos={0 0}, upload=526249]: Segment reached optimal threshold of 14 pieces. Dec 30 15:54:41.138965780 (600h25m41.010048815s) [noise=true , address=145.79.227.33:4877 , piece_id=GPUOFM2E3KZGDCEZQYPKADLJFDML5DS7OTWFW7AFB5DYAHBVBQ6Q, seg_pos={0 0}, upload=526249]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17TuauVrjKmehRpXqmK7bemU9V33GDZu9RGBs6fanXvJkV1gdH): protocol: expected piece hash; context canceled; EOF Dec 30 15:54:41.138974496 (600h25m41.010057531s) [noise=true , address=145.79.227.33:4877 , piece_id=GPUOFM2E3KZGDCEZQYPKADLJFDML5DS7OTWFW7AFB5DYAHBVBQ6Q, seg_pos={0 0}, upload=526249]: Long tail cancellation due to optimal threshold Dec 30 15:54:41.139084886 (600h25m41.010167921s) [noise=true , address=77.90.31.19:4854 , piece_id=4W27FHSBO3L2TSAZL2JFCTUPHYV7RI5XDQ4SJWOCRCD5V4K4IK2Q, seg_pos={0 0}, upload=526249]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16HwqYhJnhseWEhzzv4QreZ953rB5cKxqFX8swEqaqiRXCRsAE): protocol: write tcp 188.166.225.170:48144->77.90.31.19:4854: use of closed network connection; write tcp 188.166.225.170:48144->77.90.31.19:4854: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48144->77.90.31.19:4854: use of closed network connection Dec 30 15:54:41.139093533 (600h25m41.010176568s) [noise=true , address=77.90.31.19:4854 , piece_id=4W27FHSBO3L2TSAZL2JFCTUPHYV7RI5XDQ4SJWOCRCD5V4K4IK2Q, seg_pos={0 0}, upload=526249]: Long tail cancellation due to optimal threshold Dec 30 15:54:41.139129352 (600h25m41.010212397s) [noise=true , address=145.79.227.43:4189 , piece_id=DYHLPLLBNG6B3FXERO33ZDDSHFUHQIIYGQZJFE2UMJXMLUBYC3KQ, seg_pos={0 0}, upload=526249]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13mB9AoQJubhEJAAVqzd4o4eq8AfNvLkdqHTDAPN3UyXXwE4B7): protocol: expected piece hash; context canceled; EOF Dec 30 15:54:41.139138429 (600h25m41.010221484s) [noise=true , address=145.79.227.43:4189 , piece_id=DYHLPLLBNG6B3FXERO33ZDDSHFUHQIIYGQZJFE2UMJXMLUBYC3KQ, seg_pos={0 0}, upload=526249]: Long tail cancellation due to optimal threshold Dec 30 15:54:41.139238890 (600h25m41.010321925s) [noise=true , address=64.49.13.169:4713 , piece_id=KI4XABHC36ADSOANW3HVM3TCWZ4ZXXXNP7MWJVESDKD5WO26DJHQ, seg_pos={0 0}, upload=526249]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17z3AiUfbxBX2eMDnUnqo1bbgojuQxJro7PsEaPjYzQjHQ3D6K): protocol: write tcp 188.166.225.170:35558->64.49.13.169:4713: use of closed network connection; write tcp 188.166.225.170:35558->64.49.13.169:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35558->64.49.13.169:4713: use of closed network connection Dec 30 15:54:41.139244420 (600h25m41.010327455s) [noise=true , address=64.49.13.169:4713 , piece_id=KI4XABHC36ADSOANW3HVM3TCWZ4ZXXXNP7MWJVESDKD5WO26DJHQ, seg_pos={0 0}, upload=526249]: Long tail cancellation due to optimal threshold Dec 30 15:54:41.139323681 (600h25m41.010406716s) [seg_pos={0 0}, upload=526249]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 15:54:41.139392121 (600h25m41.010475166s) [upload=526249]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 15:54:41.139396971 (600h25m41.010480006s) [upload=526249]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 15:54:41.438011559 (600h25m41.309094604s) [upload=526249]: Done uploading segments... Dec 30 15:54:41.438036206 (600h25m41.309119251s) [upload=526249]: Upload finished. err: <nil> Duration: 1m0.003169295s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 43 out of 60 is failed (threshold=6) [X..XXXX.XX.X.X.X.XXXXXX.XXXX...XX.X.XXXXXXXXX.XXXXXXX...XXXX] duration: 5.001083721s trace: https://jaeger.dev.storj.io/trace/237fc96070c8cc7b FAIL (2025-12-30 15:55:31 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001130199s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [......X.X..........XX.......XX.XX..X....X.......X......X.X..] duration: 5.868204115s trace: https://jaeger.dev.storj.io/trace/54fff38414d25b0f FAIL (2025-12-30 15:55:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: AQGHSCVDMOODBX225O5SSOLNLJQYPAY2X7EJ7G5W6RP42ZJ4DGCQ) context deadline exceeded; piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: ZTXP5ESUQMWFEWTQUX2SMER7W2NJG5IISRWFAM3IKAKQFKJOGWSQ) context deadline exceeded ======LOGS====== Dec 30 15:53:33.440030169 (600h24m33.311113204s) [upload=526231]: Starting upload Dec 30 15:53:33.440113437 (600h24m33.311196492s) [upload=526231]: Done starting upload Dec 30 15:53:33.440168081 (600h24m33.311251116s) [upload=526231]: Uploading segments... Dec 30 15:53:33.440357231 (600h24m33.311440276s) [upload=526231]: Got next segment. Inline: false Dec 30 15:53:33.440411114 (600h24m33.311494159s) [upload=526231]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 15:53:33.440416885 (600h24m33.31149992s) [upload=526231]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 15:53:33.684984430 (600h24m33.556067505s) [seg_pos={0 0}, upload=526231]: Begin upload segment... Dec 30 15:53:33.685033994 (600h24m33.556117029s) [seg_pos={0 0}, upload=526231]: Done begin upload segment. Dec 30 15:53:33.685081235 (600h24m33.55616427s) [noise=true , address=145.79.227.36:4519 , piece_id=5JAZZV45ZTNDSTJZIX2EI7RK7MCBXBQUV2PLSF2HBCRFCIJEA6OQ, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.685200161 (600h24m33.556283196s) [noise=true , address=145.79.227.23:4545 , piece_id=NHTKPJFEFGAJOEXUNJJ7TU6N27JJP3O5TFSMWF5I2SMGEYXWEAKQ, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.685709841 (600h24m33.556792887s) [noise=true , address=77.90.31.14:4664 , piece_id=YLXNEMZBXUYTFQ4P6ZFHPA6BIUH4TZ2OBXEQAJWYSCUX333L777A, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.686084574 (600h24m33.557167619s) [noise=true , address=45.91.116.12:4227 , piece_id=5GQIOQC4VDIF3Z6J4TUPZG3XUC3AACZ53CPYB4JIBQD7F5BK2IFA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.686094794 (600h24m33.55717783s) [noise=true , address=77.90.31.10:4517 , piece_id=5Y32J2NPA664L4ZGRDDYYX66FPWND4DNZ5OZXAADJ3LOXK3IA5BA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.686429431 (600h24m33.557512476s) [noise=true , address=45.91.116.13:4153 , piece_id=AQGHSCVDMOODBX225O5SSOLNLJQYPAY2X7EJ7G5W6RP42ZJ4DGCQ, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.686489927 (600h24m33.557572962s) [noise=true , address=45.91.116.8:4047 , piece_id=TLKGZQ2VDRAGBZCULZ35ZR6GEJXQEE2HZ35Q52XBEO5C4FTRFICA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.686910547 (600h24m33.557993582s) [noise=true , address=145.79.227.37:4567 , piece_id=WSHVGPU7L73OET2M6WT7VPHGVRQV6ZCJRPAW7AEXBUQDFWQBNAUA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.686946536 (600h24m33.558029581s) [noise=true , address=145.79.227.14:4838 , piece_id=YB4VPIAQTBJ6G5XFXDVJ3OL6YFAWMEBHHS63SEDX5U4HYKXT3KJQ, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.687266755 (600h24m33.55834979s) [noise=true , address=145.79.227.6:4257 , piece_id=53MG7UDQR3MJYNMQXY3IOPPKOQEOI3KFUTGHQUEHOKCNG5NTAWXA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.687625419 (600h24m33.558708464s) [noise=true , address=45.91.116.11:4120 , piece_id=ZTXP5ESUQMWFEWTQUX2SMER7W2NJG5IISRWFAM3IKAKQFKJOGWSQ, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.688010913 (600h24m33.559093948s) [noise=true , address=145.79.227.19:4077 , piece_id=JEX52J3QTGUWMY3FO6BD5IJV5UPUUCKVV4MYZAOAU3BWMKFSN3KQ, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.688360248 (600h24m33.559443283s) [noise=true , address=145.79.227.16:4910 , piece_id=UHDFUXEQCMN73AUB7WEJBG5HY2G4E25MQGSBNK6AKHMQMDZCJYRA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.688673685 (600h24m33.55975672s) [noise=true , address=145.79.227.48:4303 , piece_id=IO7FUTELAP2L3VVZEQN6N6QG42WMAOS7H5JEPFAST5YEKMKHYLXA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.689057035 (600h24m33.56014007s) [noise=true , address=145.79.227.2:4114 , piece_id=3MJV2XXBY776ZUIO2KU4MEP37SP6P2NYFO255DCI6IFUZWUAGOWQ, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.690890025 (600h24m33.56197308s) [noise=true , address=145.79.227.3:4150 , piece_id=BUBUCPH3LTJO2CR3IVIXEOJOVS422KQT6L2Z22OBZP6DFPCK5KAA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:35.401137462 (600h24m35.272220507s) [upload=526231]: Next returned nil segment Dec 30 15:53:35.401154925 (600h24m35.27223797s) [upload=526231]: Waiting for error group managing segments... Dec 30 15:53:35.673442202 (600h24m35.544525237s) [noise=true , address=145.79.227.23:4545 , piece_id=NHTKPJFEFGAJOEXUNJJ7TU6N27JJP3O5TFSMWF5I2SMGEYXWEAKQ, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.674880822 (600h24m35.545963857s) [noise=true , address=145.79.227.2:4114 , piece_id=3MJV2XXBY776ZUIO2KU4MEP37SP6P2NYFO255DCI6IFUZWUAGOWQ, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.688801769 (600h24m35.559884814s) [noise=true , address=145.79.227.36:4519 , piece_id=5JAZZV45ZTNDSTJZIX2EI7RK7MCBXBQUV2PLSF2HBCRFCIJEA6OQ, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.689704678 (600h24m35.560787713s) [noise=true , address=145.79.227.6:4257 , piece_id=53MG7UDQR3MJYNMQXY3IOPPKOQEOI3KFUTGHQUEHOKCNG5NTAWXA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.690785346 (600h24m35.561868381s) [noise=true , address=145.79.227.14:4838 , piece_id=YB4VPIAQTBJ6G5XFXDVJ3OL6YFAWMEBHHS63SEDX5U4HYKXT3KJQ, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.692882529 (600h24m35.563965564s) [noise=true , address=145.79.227.37:4567 , piece_id=WSHVGPU7L73OET2M6WT7VPHGVRQV6ZCJRPAW7AEXBUQDFWQBNAUA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.696575461 (600h24m35.567658506s) [noise=true , address=145.79.227.16:4910 , piece_id=UHDFUXEQCMN73AUB7WEJBG5HY2G4E25MQGSBNK6AKHMQMDZCJYRA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.697864125 (600h24m35.56894716s) [noise=true , address=145.79.227.19:4077 , piece_id=JEX52J3QTGUWMY3FO6BD5IJV5UPUUCKVV4MYZAOAU3BWMKFSN3KQ, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.758530774 (600h24m35.629613819s) [noise=true , address=145.79.227.48:4303 , piece_id=IO7FUTELAP2L3VVZEQN6N6QG42WMAOS7H5JEPFAST5YEKMKHYLXA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.796785737 (600h24m35.667868772s) [noise=true , address=145.79.227.3:4150 , piece_id=BUBUCPH3LTJO2CR3IVIXEOJOVS422KQT6L2Z22OBZP6DFPCK5KAA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:36.029561278 (600h24m35.900644313s) [noise=true , address=77.90.31.10:4517 , piece_id=5Y32J2NPA664L4ZGRDDYYX66FPWND4DNZ5OZXAADJ3LOXK3IA5BA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:36.070210490 (600h24m35.941293536s) [noise=true , address=45.91.116.13:4153 , piece_id=AQGHSCVDMOODBX225O5SSOLNLJQYPAY2X7EJ7G5W6RP42ZJ4DGCQ, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:36.082591286 (600h24m35.953674331s) [noise=true , address=45.91.116.11:4120 , piece_id=ZTXP5ESUQMWFEWTQUX2SMER7W2NJG5IISRWFAM3IKAKQFKJOGWSQ, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:36.082610784 (600h24m35.953693819s) [seg_pos={0 0}, upload=526231]: Segment reached optimal threshold of 13 pieces. Dec 30 15:53:36.082858675 (600h24m35.953941711s) [noise=true , address=77.90.31.14:4664 , piece_id=YLXNEMZBXUYTFQ4P6ZFHPA6BIUH4TZ2OBXEQAJWYSCUX333L777A, seg_pos={0 0}, upload=526231]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: expected piece hash; context canceled; EOF Dec 30 15:53:36.082868053 (600h24m35.953951088s) [noise=true , address=77.90.31.14:4664 , piece_id=YLXNEMZBXUYTFQ4P6ZFHPA6BIUH4TZ2OBXEQAJWYSCUX333L777A, seg_pos={0 0}, upload=526231]: Long tail cancellation due to optimal threshold Dec 30 15:53:36.082953826 (600h24m35.954036871s) [noise=true , address=45.91.116.12:4227 , piece_id=5GQIOQC4VDIF3Z6J4TUPZG3XUC3AACZ53CPYB4JIBQD7F5BK2IFA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Dec 30 15:53:36.082969977 (600h24m35.954053012s) [noise=true , address=45.91.116.12:4227 , piece_id=5GQIOQC4VDIF3Z6J4TUPZG3XUC3AACZ53CPYB4JIBQD7F5BK2IFA, seg_pos={0 0}, upload=526231]: Long tail cancellation due to optimal threshold Dec 30 15:53:36.083266191 (600h24m35.954349226s) [noise=true , address=45.91.116.8:4047 , piece_id=TLKGZQ2VDRAGBZCULZ35ZR6GEJXQEE2HZ35Q52XBEO5C4FTRFICA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: read tcp 188.166.225.170:42816->45.91.116.8:4047: use of closed network connection; context canceled Dec 30 15:53:36.083274006 (600h24m35.954357041s) [noise=true , address=45.91.116.8:4047 , piece_id=TLKGZQ2VDRAGBZCULZ35ZR6GEJXQEE2HZ35Q52XBEO5C4FTRFICA, seg_pos={0 0}, upload=526231]: Long tail cancellation due to optimal threshold Dec 30 15:53:36.083296048 (600h24m35.954379083s) [seg_pos={0 0}, upload=526231]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 15:53:36.083385679 (600h24m35.954468714s) [upload=526231]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 15:53:36.083392341 (600h24m35.954475376s) [upload=526231]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 15:53:36.379743600 (600h24m36.250826635s) [upload=526231]: Done uploading segments... Dec 30 15:53:36.379758638 (600h24m36.250841673s) [upload=526231]: Upload finished. err: <nil> Duration: 5.868230483s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 13.73727946s trace: https://jaeger.dev.storj.io/trace/547648a2cfce50e7 FAIL (2025-12-30 15:54:41 UTC) orbiter warning: context deadline exceeded Duration: 13.737328644s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....X............................X..XX..X..................] duration: 28.13734322s trace: https://jaeger.dev.storj.io/trace/3b27a789ef618091 FAIL (2025-12-30 15:54:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ise=true , address=145.79.227.5:4252 , piece_id=JRKV35AO742VMYZPN3WKPNRZKO2ZFUU53RS4MXF22HO5UC6BKCOQ, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.081840064 (600h10m41.952923099s) [noise=true , address=145.79.227.40:4065 , piece_id=6FSURDUGYJU4CE6LEWXTNFEIGP7X7GFXPUTO75PAQWNQKV3T2H2A, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.082191573 (600h10m41.953274608s) [noise=true , address=145.79.227.16:4942 , piece_id=G5I53KG36VHVW5LICAHOHRXTQVQ67ZFGJQNXRJMYNOIT7WB7XTEQ, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.082429897 (600h10m41.953512932s) [noise=true , address=145.79.227.29:4968 , piece_id=PMTNKRHI6YWMJPZ2R7RPIF7Q6CFLFKBH5OTHWCX4RNGIAFURVMOQ, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.082680584 (600h10m41.953763619s) [noise=true , address=145.79.227.33:4892 , piece_id=RSEJC3XHFACGLV6YBHYM7QBCU7MMYST53FJO4ITLA2C3IHBZCRTQ, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.082917976 (600h10m41.954001011s) [noise=true , address=77.90.31.12:4596 , piece_id=TQVNHTTAWVUCUIHSHWWH27MFJNOE2MWLJCRMTPUCREML7DT3RUYQ, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.083149276 (600h10m41.954232311s) [noise=true , address=64.49.13.169:4707 , piece_id=5NHKQJBQWSIMTFX47MKGNGAGMHKS5ENFOL3ROY45FLLXS7NSVJLQ, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.083393431 (600h10m41.954476466s) [noise=true , address=145.79.227.35:4508 , piece_id=KJJFNTPYUUHLFFCGHD3KDTN3DK6C2R5YRLWJDMF6C4UGSMFYVFBA, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.083842486 (600h10m41.954925521s) [noise=true , address=77.90.31.12:4612 , piece_id=W5IMLESMOEXZGSYXTUFENV5JHPDZKN4J2BYOZ3EEI5P4PDDQUM7A, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.086214362 (600h10m41.957297397s) [noise=true , address=145.79.227.49:4367 , piece_id=QUI5FSOTJVIHT67I3F2YDB5IPKSYEYJTDRKA44UM355S5FSNNLUA, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.086612852 (600h10m41.957695887s) [noise=true , address=77.90.31.13:4652 , piece_id=5VR2F6XUEYGX5KULBIL6V3YGH7Q3T5UYFJTS5RGWN3B7F7TVMHEQ, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.087061346 (600h10m41.958144381s) [noise=true , address=77.90.31.11:4588 , piece_id=WY5XS4EPFHDLCT3XEQQKBO2M5GLVTBNDZOKPTFPVZV6EALPMNSHA, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.087452310 (600h10m41.958535345s) [noise=true , address=145.79.227.30:4768 , piece_id=2KZKXIYGSDES5Q5W2PVAACJYU6ANKQNDP2VQEBXM7QKXOT6QLSBA, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.087903780 (600h10m41.958986825s) [noise=true , address=64.49.13.27:4957 , piece_id=ETFKCNNSFAVJWNB3J3U4LPRDGBILNNA5CK7FR7X5UHONPZHXFBLQ, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.088521366 (600h10m41.959604411s) [noise=true , address=193.149.164.143:4698 , piece_id=F6B3UPOKCSD7ZXQ4SYKOAALEBA4FSTQ2ELGMGCU7TMWVQZDZB2SQ, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.088618600 (600h10m41.959701645s) [noise=true , address=145.79.227.34:4918 , piece_id=6BBY7R4FCE7CW4EJG4AMINHO5YDA3KUJH4TAC7QNHXKBHL7C3M6A, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:43.511091675 (600h10m43.38217472s) [upload=526030]: Next returned nil segment Dec 30 15:39:43.511109379 (600h10m43.382192424s) [upload=526030]: Waiting for error group managing segments... Dec 30 15:39:44.017333248 (600h10m43.888416283s) [noise=true , address=145.79.227.5:4252 , piece_id=JRKV35AO742VMYZPN3WKPNRZKO2ZFUU53RS4MXF22HO5UC6BKCOQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil> Dec 30 15:39:44.069300150 (600h10m43.940383186s) [noise=true , address=64.49.13.27:4957 , piece_id=ETFKCNNSFAVJWNB3J3U4LPRDGBILNNA5CK7FR7X5UHONPZHXFBLQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil> Dec 30 15:39:44.126868228 (600h10m43.997951273s) [noise=true , address=77.90.31.17:4800 , piece_id=PYZVLAZZKWP6UOL6HJVVWSEJOXCJWK5IAMWJPRODOKCLRZE37BAQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil> Dec 30 15:39:44.146896235 (600h10m44.01797927s) [noise=true , address=145.79.227.29:4968 , piece_id=PMTNKRHI6YWMJPZ2R7RPIF7Q6CFLFKBH5OTHWCX4RNGIAFURVMOQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil> Dec 30 15:39:44.166221683 (600h10m44.037304718s) [noise=true , address=77.90.31.12:4596 , piece_id=TQVNHTTAWVUCUIHSHWWH27MFJNOE2MWLJCRMTPUCREML7DT3RUYQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil> Dec 30 15:39:44.234530763 (600h10m44.105613808s) [noise=true , address=77.90.31.13:4652 , piece_id=5VR2F6XUEYGX5KULBIL6V3YGH7Q3T5UYFJTS5RGWN3B7F7TVMHEQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil> Dec 30 15:39:44.489468910 (600h10m44.360551955s) [noise=true , address=145.79.227.16:4942 , piece_id=G5I53KG36VHVW5LICAHOHRXTQVQ67ZFGJQNXRJMYNOIT7WB7XTEQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil> Dec 30 15:39:44.802041331 (600h10m44.673124366s) [noise=true , address=193.149.164.143:4698 , piece_id=F6B3UPOKCSD7ZXQ4SYKOAALEBA4FSTQ2ELGMGCU7TMWVQZDZB2SQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil> Dec 30 15:39:44.822093824 (600h10m44.693176859s) [noise=true , address=145.79.227.49:4367 , piece_id=QUI5FSOTJVIHT67I3F2YDB5IPKSYEYJTDRKA44UM355S5FSNNLUA, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil> Dec 30 15:39:44.867276618 (600h10m44.738359653s) [noise=true , address=145.79.227.30:4768 , piece_id=2KZKXIYGSDES5Q5W2PVAACJYU6ANKQNDP2VQEBXM7QKXOT6QLSBA, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil> Dec 30 15:39:44.881025168 (600h10m44.752108203s) [noise=true , address=145.79.227.44:4199 , piece_id=AJKADWXHHMCPE6V52UMJJPCHOBBL7QDU7DF337LB6Q3YDHEYUUQQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil> Dec 30 15:39:44.899437537 (600h10m44.770520572s) [noise=true , address=77.90.31.11:4588 , piece_id=WY5XS4EPFHDLCT3XEQQKBO2M5GLVTBNDZOKPTFPVZV6EALPMNSHA, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil> Dec 30 15:39:44.941939747 (600h10m44.813022782s) [noise=true , address=145.79.227.33:4892 , piece_id=RSEJC3XHFACGLV6YBHYM7QBCU7MMYST53FJO4ITLA2C3IHBZCRTQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil> Dec 30 15:39:44.980395983 (600h10m44.851479029s) [noise=true , address=145.79.227.35:4508 , piece_id=KJJFNTPYUUHLFFCGHD3KDTN3DK6C2R5YRLWJDMF6C4UGSMFYVFBA, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil> Dec 30 15:39:44.980415871 (600h10m44.851498916s) [seg_pos={0 0}, upload=526030]: Segment reached optimal threshold of 14 pieces. Dec 30 15:39:44.980578861 (600h10m44.851661896s) [noise=true , address=64.49.13.169:4707 , piece_id=5NHKQJBQWSIMTFX47MKGNGAGMHKS5ENFOL3ROY45FLLXS7NSVJLQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:49762->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:49762->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49762->64.49.13.169:4707: use of closed network connection Dec 30 15:39:44.980584171 (600h10m44.851667206s) [noise=true , address=64.49.13.169:4707 , piece_id=5NHKQJBQWSIMTFX47MKGNGAGMHKS5ENFOL3ROY45FLLXS7NSVJLQ, seg_pos={0 0}, upload=526030]: Long tail cancellation due to optimal threshold Dec 30 15:39:44.980730901 (600h10m44.851813936s) [noise=true , address=145.79.227.34:4918 , piece_id=6BBY7R4FCE7CW4EJG4AMINHO5YDA3KUJH4TAC7QNHXKBHL7C3M6A, seg_pos={0 0}, upload=526030]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Nw78dd9WPnmCwv5bFQR3VpV39776Pye2QVGSNMTs3zq2BtaGJ): protocol: expected piece hash; context canceled; EOF Dec 30 15:39:44.980747002 (600h10m44.851830037s) [noise=true , address=145.79.227.34:4918 , piece_id=6BBY7R4FCE7CW4EJG4AMINHO5YDA3KUJH4TAC7QNHXKBHL7C3M6A, seg_pos={0 0}, upload=526030]: Long tail cancellation due to optimal threshold Dec 30 15:39:44.980793770 (600h10m44.851876806s) [noise=true , address=145.79.227.40:4065 , piece_id=6FSURDUGYJU4CE6LEWXTNFEIGP7X7GFXPUTO75PAQWNQKV3T2H2A, seg_pos={0 0}, upload=526030]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127NKToES711diH4f8TFJbwERq3NhhFEFrQyi7Mka3VVLsamuzp): protocol: expected piece hash; context canceled; EOF Dec 30 15:39:44.980799651 (600h10m44.851882687s) [noise=true , address=145.79.227.40:4065 , piece_id=6FSURDUGYJU4CE6LEWXTNFEIGP7X7GFXPUTO75PAQWNQKV3T2H2A, seg_pos={0 0}, upload=526030]: Long tail cancellation due to optimal threshold Dec 30 15:39:44.980930771 (600h10m44.852013806s) [noise=true , address=77.90.31.12:4612 , piece_id=W5IMLESMOEXZGSYXTUFENV5JHPDZKN4J2BYOZ3EEI5P4PDDQUM7A, seg_pos={0 0}, upload=526030]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12nnXd911ApLeiKnc4LgDEUNnnkxtnLEdwzP48BhuhyEykacV6W): protocol: expected piece hash; context canceled; EOF Dec 30 15:39:44.980937775 (600h10m44.85202081s) [noise=true , address=77.90.31.12:4612 , piece_id=W5IMLESMOEXZGSYXTUFENV5JHPDZKN4J2BYOZ3EEI5P4PDDQUM7A, seg_pos={0 0}, upload=526030]: Long tail cancellation due to optimal threshold Dec 30 15:39:44.980956481 (600h10m44.852039526s) [seg_pos={0 0}, upload=526030]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 15:39:44.981039398 (600h10m44.852122433s) [upload=526030]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 15:39:44.981056070 (600h10m44.852139105s) [upload=526030]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 15:39:45.290276710 (600h10m45.161359765s) [upload=526030]: Done uploading segments... Dec 30 15:39:45.290299182 (600h10m45.161382227s) [upload=526030]: Upload finished. err: <nil> Duration: 28.13735873s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 43 out of 60 is failed (threshold=6) [XX..XXXX.XX.X.X.X.XXXXXX.XXXX...XX.X.XXXXXXXXX.XXXXXXX...XXX] duration: 5.001342955s trace: https://jaeger.dev.storj.io/trace/5f18ac4558a1358d FAIL (2025-12-30 15:54:28 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001360888s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.......X.X..........XX.......XX.XX..X....X.......X......X.X.] duration: 5.800667553s trace: https://jaeger.dev.storj.io/trace/54fff38414d25b0f FAIL (2025-12-30 15:54:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: AQGHSCVDMOODBX225O5SSOLNLJQYPAY2X7EJ7G5W6RP42ZJ4DGCQ) context deadline exceeded; piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: ZTXP5ESUQMWFEWTQUX2SMER7W2NJG5IISRWFAM3IKAKQFKJOGWSQ) context deadline exceeded ======LOGS====== Dec 30 15:53:33.440030169 (600h24m33.311113204s) [upload=526231]: Starting upload Dec 30 15:53:33.440113437 (600h24m33.311196492s) [upload=526231]: Done starting upload Dec 30 15:53:33.440168081 (600h24m33.311251116s) [upload=526231]: Uploading segments... Dec 30 15:53:33.440357231 (600h24m33.311440276s) [upload=526231]: Got next segment. Inline: false Dec 30 15:53:33.440411114 (600h24m33.311494159s) [upload=526231]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 15:53:33.440416885 (600h24m33.31149992s) [upload=526231]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 15:53:33.684984430 (600h24m33.556067505s) [seg_pos={0 0}, upload=526231]: Begin upload segment... Dec 30 15:53:33.685033994 (600h24m33.556117029s) [seg_pos={0 0}, upload=526231]: Done begin upload segment. Dec 30 15:53:33.685081235 (600h24m33.55616427s) [noise=true , address=145.79.227.36:4519 , piece_id=5JAZZV45ZTNDSTJZIX2EI7RK7MCBXBQUV2PLSF2HBCRFCIJEA6OQ, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.685200161 (600h24m33.556283196s) [noise=true , address=145.79.227.23:4545 , piece_id=NHTKPJFEFGAJOEXUNJJ7TU6N27JJP3O5TFSMWF5I2SMGEYXWEAKQ, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.685709841 (600h24m33.556792887s) [noise=true , address=77.90.31.14:4664 , piece_id=YLXNEMZBXUYTFQ4P6ZFHPA6BIUH4TZ2OBXEQAJWYSCUX333L777A, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.686084574 (600h24m33.557167619s) [noise=true , address=45.91.116.12:4227 , piece_id=5GQIOQC4VDIF3Z6J4TUPZG3XUC3AACZ53CPYB4JIBQD7F5BK2IFA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.686094794 (600h24m33.55717783s) [noise=true , address=77.90.31.10:4517 , piece_id=5Y32J2NPA664L4ZGRDDYYX66FPWND4DNZ5OZXAADJ3LOXK3IA5BA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.686429431 (600h24m33.557512476s) [noise=true , address=45.91.116.13:4153 , piece_id=AQGHSCVDMOODBX225O5SSOLNLJQYPAY2X7EJ7G5W6RP42ZJ4DGCQ, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.686489927 (600h24m33.557572962s) [noise=true , address=45.91.116.8:4047 , piece_id=TLKGZQ2VDRAGBZCULZ35ZR6GEJXQEE2HZ35Q52XBEO5C4FTRFICA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.686910547 (600h24m33.557993582s) [noise=true , address=145.79.227.37:4567 , piece_id=WSHVGPU7L73OET2M6WT7VPHGVRQV6ZCJRPAW7AEXBUQDFWQBNAUA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.686946536 (600h24m33.558029581s) [noise=true , address=145.79.227.14:4838 , piece_id=YB4VPIAQTBJ6G5XFXDVJ3OL6YFAWMEBHHS63SEDX5U4HYKXT3KJQ, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.687266755 (600h24m33.55834979s) [noise=true , address=145.79.227.6:4257 , piece_id=53MG7UDQR3MJYNMQXY3IOPPKOQEOI3KFUTGHQUEHOKCNG5NTAWXA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.687625419 (600h24m33.558708464s) [noise=true , address=45.91.116.11:4120 , piece_id=ZTXP5ESUQMWFEWTQUX2SMER7W2NJG5IISRWFAM3IKAKQFKJOGWSQ, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.688010913 (600h24m33.559093948s) [noise=true , address=145.79.227.19:4077 , piece_id=JEX52J3QTGUWMY3FO6BD5IJV5UPUUCKVV4MYZAOAU3BWMKFSN3KQ, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.688360248 (600h24m33.559443283s) [noise=true , address=145.79.227.16:4910 , piece_id=UHDFUXEQCMN73AUB7WEJBG5HY2G4E25MQGSBNK6AKHMQMDZCJYRA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.688673685 (600h24m33.55975672s) [noise=true , address=145.79.227.48:4303 , piece_id=IO7FUTELAP2L3VVZEQN6N6QG42WMAOS7H5JEPFAST5YEKMKHYLXA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.689057035 (600h24m33.56014007s) [noise=true , address=145.79.227.2:4114 , piece_id=3MJV2XXBY776ZUIO2KU4MEP37SP6P2NYFO255DCI6IFUZWUAGOWQ, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.690890025 (600h24m33.56197308s) [noise=true , address=145.79.227.3:4150 , piece_id=BUBUCPH3LTJO2CR3IVIXEOJOVS422KQT6L2Z22OBZP6DFPCK5KAA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:35.401137462 (600h24m35.272220507s) [upload=526231]: Next returned nil segment Dec 30 15:53:35.401154925 (600h24m35.27223797s) [upload=526231]: Waiting for error group managing segments... Dec 30 15:53:35.673442202 (600h24m35.544525237s) [noise=true , address=145.79.227.23:4545 , piece_id=NHTKPJFEFGAJOEXUNJJ7TU6N27JJP3O5TFSMWF5I2SMGEYXWEAKQ, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.674880822 (600h24m35.545963857s) [noise=true , address=145.79.227.2:4114 , piece_id=3MJV2XXBY776ZUIO2KU4MEP37SP6P2NYFO255DCI6IFUZWUAGOWQ, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.688801769 (600h24m35.559884814s) [noise=true , address=145.79.227.36:4519 , piece_id=5JAZZV45ZTNDSTJZIX2EI7RK7MCBXBQUV2PLSF2HBCRFCIJEA6OQ, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.689704678 (600h24m35.560787713s) [noise=true , address=145.79.227.6:4257 , piece_id=53MG7UDQR3MJYNMQXY3IOPPKOQEOI3KFUTGHQUEHOKCNG5NTAWXA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.690785346 (600h24m35.561868381s) [noise=true , address=145.79.227.14:4838 , piece_id=YB4VPIAQTBJ6G5XFXDVJ3OL6YFAWMEBHHS63SEDX5U4HYKXT3KJQ, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.692882529 (600h24m35.563965564s) [noise=true , address=145.79.227.37:4567 , piece_id=WSHVGPU7L73OET2M6WT7VPHGVRQV6ZCJRPAW7AEXBUQDFWQBNAUA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.696575461 (600h24m35.567658506s) [noise=true , address=145.79.227.16:4910 , piece_id=UHDFUXEQCMN73AUB7WEJBG5HY2G4E25MQGSBNK6AKHMQMDZCJYRA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.697864125 (600h24m35.56894716s) [noise=true , address=145.79.227.19:4077 , piece_id=JEX52J3QTGUWMY3FO6BD5IJV5UPUUCKVV4MYZAOAU3BWMKFSN3KQ, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.758530774 (600h24m35.629613819s) [noise=true , address=145.79.227.48:4303 , piece_id=IO7FUTELAP2L3VVZEQN6N6QG42WMAOS7H5JEPFAST5YEKMKHYLXA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.796785737 (600h24m35.667868772s) [noise=true , address=145.79.227.3:4150 , piece_id=BUBUCPH3LTJO2CR3IVIXEOJOVS422KQT6L2Z22OBZP6DFPCK5KAA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:36.029561278 (600h24m35.900644313s) [noise=true , address=77.90.31.10:4517 , piece_id=5Y32J2NPA664L4ZGRDDYYX66FPWND4DNZ5OZXAADJ3LOXK3IA5BA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:36.070210490 (600h24m35.941293536s) [noise=true , address=45.91.116.13:4153 , piece_id=AQGHSCVDMOODBX225O5SSOLNLJQYPAY2X7EJ7G5W6RP42ZJ4DGCQ, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:36.082591286 (600h24m35.953674331s) [noise=true , address=45.91.116.11:4120 , piece_id=ZTXP5ESUQMWFEWTQUX2SMER7W2NJG5IISRWFAM3IKAKQFKJOGWSQ, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:36.082610784 (600h24m35.953693819s) [seg_pos={0 0}, upload=526231]: Segment reached optimal threshold of 13 pieces. Dec 30 15:53:36.082858675 (600h24m35.953941711s) [noise=true , address=77.90.31.14:4664 , piece_id=YLXNEMZBXUYTFQ4P6ZFHPA6BIUH4TZ2OBXEQAJWYSCUX333L777A, seg_pos={0 0}, upload=526231]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: expected piece hash; context canceled; EOF Dec 30 15:53:36.082868053 (600h24m35.953951088s) [noise=true , address=77.90.31.14:4664 , piece_id=YLXNEMZBXUYTFQ4P6ZFHPA6BIUH4TZ2OBXEQAJWYSCUX333L777A, seg_pos={0 0}, upload=526231]: Long tail cancellation due to optimal threshold Dec 30 15:53:36.082953826 (600h24m35.954036871s) [noise=true , address=45.91.116.12:4227 , piece_id=5GQIOQC4VDIF3Z6J4TUPZG3XUC3AACZ53CPYB4JIBQD7F5BK2IFA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Dec 30 15:53:36.082969977 (600h24m35.954053012s) [noise=true , address=45.91.116.12:4227 , piece_id=5GQIOQC4VDIF3Z6J4TUPZG3XUC3AACZ53CPYB4JIBQD7F5BK2IFA, seg_pos={0 0}, upload=526231]: Long tail cancellation due to optimal threshold Dec 30 15:53:36.083266191 (600h24m35.954349226s) [noise=true , address=45.91.116.8:4047 , piece_id=TLKGZQ2VDRAGBZCULZ35ZR6GEJXQEE2HZ35Q52XBEO5C4FTRFICA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: read tcp 188.166.225.170:42816->45.91.116.8:4047: use of closed network connection; context canceled Dec 30 15:53:36.083274006 (600h24m35.954357041s) [noise=true , address=45.91.116.8:4047 , piece_id=TLKGZQ2VDRAGBZCULZ35ZR6GEJXQEE2HZ35Q52XBEO5C4FTRFICA, seg_pos={0 0}, upload=526231]: Long tail cancellation due to optimal threshold Dec 30 15:53:36.083296048 (600h24m35.954379083s) [seg_pos={0 0}, upload=526231]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 15:53:36.083385679 (600h24m35.954468714s) [upload=526231]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 15:53:36.083392341 (600h24m35.954475376s) [upload=526231]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 15:53:36.379743600 (600h24m36.250826635s) [upload=526231]: Done uploading segments... Dec 30 15:53:36.379758638 (600h24m36.250841673s) [upload=526231]: Upload finished. err: <nil> Duration: 5.800695606s core uplink us-select-1

1
...
221
222
223
224
225
...
7139