Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 5.882180413s trace: https://jaeger.dev.storj.io/trace/49621e5e6beafe8e FAIL (2025-11-11 08:40:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 11 08:10:09.433180409 (48h6m7.98465087s) [upload=42408]: Starting upload Nov 11 08:10:09.433429884 (48h6m7.984900335s) [upload=42408]: Done starting upload Nov 11 08:10:09.433567967 (48h6m7.985038438s) [upload=42408]: Uploading segments... Nov 11 08:10:09.434191213 (48h6m7.985661674s) [upload=42408]: Got next segment. Inline: false Nov 11 08:10:09.434274032 (48h6m7.985744493s) [upload=42408]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 08:10:09.434282217 (48h6m7.985752688s) [upload=42408]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 08:10:09.714872160 (48h6m8.266342621s) [seg_pos={0 0}, upload=42408]: Begin upload segment... Nov 11 08:10:09.715084585 (48h6m8.266555046s) [seg_pos={0 0}, upload=42408]: Done begin upload segment. Nov 11 08:10:09.715464638 (48h6m8.2669351s) [noise=true , address=45.91.116.8:4047 , piece_id=VP6VW2ZTNSKS35R47GGJZI4YSHNNFUFB4YQ3W3QFT5ENN7F73VQA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.715555221 (48h6m8.267025692s) [noise=true , address=145.79.227.30:4761 , piece_id=ASO5O5LM6CEF6DXGBVOCI3NOCWLGKDU2T5JHOL5ZPWEXMTRMKZIA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.718085410 (48h6m8.269555901s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.718721901 (48h6m8.270192402s) [noise=true , address=145.79.227.11:4729 , piece_id=CDIU7FFPCVQGDK3K5ZDUARJ6VGB7KIY2DEVL7ZRLW5EASIFEHKHA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.719505724 (48h6m8.270976195s) [noise=true , address=145.79.227.27:4689 , piece_id=KLTJYVDWHEE2BGLUSP2XT6L5R2C5OB7TF4DZLYBF6R5OG3N6AEFQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.720136295 (48h6m8.271606756s) [noise=true , address=145.79.227.14:4840 , piece_id=QASWXZB3E6E7VYGF24KZ37Q3LKT6F4WVFKTBV3YS73KYAWTBV7YQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.721166056 (48h6m8.272636517s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.721883453 (48h6m8.273353994s) [noise=true , address=45.91.116.4:4939 , piece_id=KMHEQSQTA2WEWMIO7JDQZUDLSRV4CA6OJUN2CYYUSA7SU2ZDQBCQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.722398593 (48h6m8.273869054s) [noise=true , address=77.90.31.18:4806 , piece_id=LNL5LRHH3OOELKEO4HND2K3GNVDPCBCGKBNJ75DCLEKRAOMTQLKA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.724101306 (48h6m8.275571767s) [noise=true , address=145.79.227.17:4970 , piece_id=F2227LWTX5PRCQIUEJNRJ3QS3GRJOIEJIZFV3KRF7BAEAPSQZR7A, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.726385264 (48h6m8.277855776s) [noise=true , address=45.91.116.11:4117 , piece_id=UOEFPOPOFIO5KXY2JOX54YHUWNMTCAFMXEBKAMMR4GSR5ZE6J6PA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.728323225 (48h6m8.279793686s) [noise=true , address=145.79.227.20:4437 , piece_id=AOK6MTRZKPCXTNHZWGBYJGRBYNHFWO7GSG2PT4DFSMDAWK75GY4A, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.730321541 (48h6m8.281792002s) [noise=true , address=145.79.227.9:4366 , piece_id=JDBR4PTBZ5RN6NTWPERVLFANUAZQF3FSGIENVRHSYN56ED5NT2AQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.731324300 (48h6m8.282794832s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.732226989 (48h6m8.28369749s) [noise=true , address=77.90.31.19:4841 , piece_id=6JVB2WIEA2STMNBVTM6D7IYFHQY4QJGXAYHJAQILMYZZDH2MWIVA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.733026101 (48h6m8.284496562s) [noise=true , address=145.79.227.21:4474 , piece_id=RNB5T6DIUCY7IV2ULLO2UJRDKUJXI34UM4ZCHPJB22DNQ47VZJVA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:11.398415378 (48h6m9.949885839s) [upload=42408]: Next returned nil segment Nov 11 08:10:11.398434043 (48h6m9.949904504s) [upload=42408]: Waiting for error group managing segments... Nov 11 08:10:11.796154582 (48h6m10.347625043s) [noise=true , address=77.90.31.18:4806 , piece_id=LNL5LRHH3OOELKEO4HND2K3GNVDPCBCGKBNJ75DCLEKRAOMTQLKA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.809043214 (48h6m10.360513675s) [noise=true , address=45.91.116.8:4047 , piece_id=VP6VW2ZTNSKS35R47GGJZI4YSHNNFUFB4YQ3W3QFT5ENN7F73VQA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.866335992 (48h6m10.417806453s) [noise=true , address=45.91.116.11:4117 , piece_id=UOEFPOPOFIO5KXY2JOX54YHUWNMTCAFMXEBKAMMR4GSR5ZE6J6PA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.932922087 (48h6m10.484392538s) [noise=true , address=77.90.31.19:4841 , piece_id=6JVB2WIEA2STMNBVTM6D7IYFHQY4QJGXAYHJAQILMYZZDH2MWIVA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:12.005905333 (48h6m10.557375794s) [noise=true , address=45.91.116.4:4939 , piece_id=KMHEQSQTA2WEWMIO7JDQZUDLSRV4CA6OJUN2CYYUSA7SU2ZDQBCQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:16.703757864 (48h6m15.255228326s) [noise=true , address=145.79.227.9:4366 , piece_id=JDBR4PTBZ5RN6NTWPERVLFANUAZQF3FSGIENVRHSYN56ED5NT2AQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:17.540298352 (48h6m16.091768813s) [noise=true , address=145.79.227.21:4474 , piece_id=RNB5T6DIUCY7IV2ULLO2UJRDKUJXI34UM4ZCHPJB22DNQ47VZJVA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:19.805526866 (48h6m18.356997337s) [noise=true , address=145.79.227.27:4689 , piece_id=KLTJYVDWHEE2BGLUSP2XT6L5R2C5OB7TF4DZLYBF6R5OG3N6AEFQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:24.397542313 (48h6m22.949012764s) [noise=true , address=145.79.227.17:4970 , piece_id=F2227LWTX5PRCQIUEJNRJ3QS3GRJOIEJIZFV3KRF7BAEAPSQZR7A, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:24.779686050 (48h6m23.331156511s) [noise=true , address=145.79.227.14:4840 , piece_id=QASWXZB3E6E7VYGF24KZ37Q3LKT6F4WVFKTBV3YS73KYAWTBV7YQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:25.298168925 (48h6m23.849639387s) [noise=true , address=145.79.227.11:4729 , piece_id=CDIU7FFPCVQGDK3K5ZDUARJ6VGB7KIY2DEVL7ZRLW5EASIFEHKHA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:35.044925402 (48h6m33.596395853s) [noise=true , address=145.79.227.30:4761 , piece_id=ASO5O5LM6CEF6DXGBVOCI3NOCWLGKDU2T5JHOL5ZPWEXMTRMKZIA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:37.306959937 (48h6m35.858430398s) [noise=true , address=145.79.227.20:4437 , piece_id=AOK6MTRZKPCXTNHZWGBYJGRBYNHFWO7GSG2PT4DFSMDAWK75GY4A, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:37.306987550 (48h6m35.858458011s) [seg_pos={0 0}, upload=42408]: Segment reached optimal threshold of 13 pieces. Nov 11 08:10:37.307209231 (48h6m35.858679702s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4): protocol: expected piece hash; context canceled; EOF Nov 11 08:10:37.307218759 (48h6m35.85868922s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307368936 (48h6m35.858839397s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw): protocol: expected piece hash; context canceled; EOF Nov 11 08:10:37.307376741 (48h6m35.858847202s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307561282 (48h6m35.859031743s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9): protocol: write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection Nov 11 08:10:37.307585969 (48h6m35.85905643s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307629221 (48h6m35.859099682s) [seg_pos={0 0}, upload=42408]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 11 08:10:37.307741005 (48h6m35.859211466s) [upload=42408]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 11 08:10:37.307750052 (48h6m35.859220513s) [upload=42408]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 11 08:10:37.612339386 (48h6m36.163809848s) [upload=42408]: Done uploading segments... Nov 11 08:10:37.612366317 (48h6m36.163836788s) [upload=42408]: Upload finished. err: <nil> Duration: 5.882226071s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 5.77657196s trace: https://jaeger.dev.storj.io/trace/49621e5e6beafe8e FAIL (2025-11-11 08:39:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 11 08:10:09.433180409 (48h6m7.98465087s) [upload=42408]: Starting upload Nov 11 08:10:09.433429884 (48h6m7.984900335s) [upload=42408]: Done starting upload Nov 11 08:10:09.433567967 (48h6m7.985038438s) [upload=42408]: Uploading segments... Nov 11 08:10:09.434191213 (48h6m7.985661674s) [upload=42408]: Got next segment. Inline: false Nov 11 08:10:09.434274032 (48h6m7.985744493s) [upload=42408]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 08:10:09.434282217 (48h6m7.985752688s) [upload=42408]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 08:10:09.714872160 (48h6m8.266342621s) [seg_pos={0 0}, upload=42408]: Begin upload segment... Nov 11 08:10:09.715084585 (48h6m8.266555046s) [seg_pos={0 0}, upload=42408]: Done begin upload segment. Nov 11 08:10:09.715464638 (48h6m8.2669351s) [noise=true , address=45.91.116.8:4047 , piece_id=VP6VW2ZTNSKS35R47GGJZI4YSHNNFUFB4YQ3W3QFT5ENN7F73VQA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.715555221 (48h6m8.267025692s) [noise=true , address=145.79.227.30:4761 , piece_id=ASO5O5LM6CEF6DXGBVOCI3NOCWLGKDU2T5JHOL5ZPWEXMTRMKZIA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.718085410 (48h6m8.269555901s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.718721901 (48h6m8.270192402s) [noise=true , address=145.79.227.11:4729 , piece_id=CDIU7FFPCVQGDK3K5ZDUARJ6VGB7KIY2DEVL7ZRLW5EASIFEHKHA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.719505724 (48h6m8.270976195s) [noise=true , address=145.79.227.27:4689 , piece_id=KLTJYVDWHEE2BGLUSP2XT6L5R2C5OB7TF4DZLYBF6R5OG3N6AEFQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.720136295 (48h6m8.271606756s) [noise=true , address=145.79.227.14:4840 , piece_id=QASWXZB3E6E7VYGF24KZ37Q3LKT6F4WVFKTBV3YS73KYAWTBV7YQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.721166056 (48h6m8.272636517s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.721883453 (48h6m8.273353994s) [noise=true , address=45.91.116.4:4939 , piece_id=KMHEQSQTA2WEWMIO7JDQZUDLSRV4CA6OJUN2CYYUSA7SU2ZDQBCQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.722398593 (48h6m8.273869054s) [noise=true , address=77.90.31.18:4806 , piece_id=LNL5LRHH3OOELKEO4HND2K3GNVDPCBCGKBNJ75DCLEKRAOMTQLKA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.724101306 (48h6m8.275571767s) [noise=true , address=145.79.227.17:4970 , piece_id=F2227LWTX5PRCQIUEJNRJ3QS3GRJOIEJIZFV3KRF7BAEAPSQZR7A, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.726385264 (48h6m8.277855776s) [noise=true , address=45.91.116.11:4117 , piece_id=UOEFPOPOFIO5KXY2JOX54YHUWNMTCAFMXEBKAMMR4GSR5ZE6J6PA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.728323225 (48h6m8.279793686s) [noise=true , address=145.79.227.20:4437 , piece_id=AOK6MTRZKPCXTNHZWGBYJGRBYNHFWO7GSG2PT4DFSMDAWK75GY4A, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.730321541 (48h6m8.281792002s) [noise=true , address=145.79.227.9:4366 , piece_id=JDBR4PTBZ5RN6NTWPERVLFANUAZQF3FSGIENVRHSYN56ED5NT2AQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.731324300 (48h6m8.282794832s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.732226989 (48h6m8.28369749s) [noise=true , address=77.90.31.19:4841 , piece_id=6JVB2WIEA2STMNBVTM6D7IYFHQY4QJGXAYHJAQILMYZZDH2MWIVA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.733026101 (48h6m8.284496562s) [noise=true , address=145.79.227.21:4474 , piece_id=RNB5T6DIUCY7IV2ULLO2UJRDKUJXI34UM4ZCHPJB22DNQ47VZJVA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:11.398415378 (48h6m9.949885839s) [upload=42408]: Next returned nil segment Nov 11 08:10:11.398434043 (48h6m9.949904504s) [upload=42408]: Waiting for error group managing segments... Nov 11 08:10:11.796154582 (48h6m10.347625043s) [noise=true , address=77.90.31.18:4806 , piece_id=LNL5LRHH3OOELKEO4HND2K3GNVDPCBCGKBNJ75DCLEKRAOMTQLKA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.809043214 (48h6m10.360513675s) [noise=true , address=45.91.116.8:4047 , piece_id=VP6VW2ZTNSKS35R47GGJZI4YSHNNFUFB4YQ3W3QFT5ENN7F73VQA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.866335992 (48h6m10.417806453s) [noise=true , address=45.91.116.11:4117 , piece_id=UOEFPOPOFIO5KXY2JOX54YHUWNMTCAFMXEBKAMMR4GSR5ZE6J6PA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.932922087 (48h6m10.484392538s) [noise=true , address=77.90.31.19:4841 , piece_id=6JVB2WIEA2STMNBVTM6D7IYFHQY4QJGXAYHJAQILMYZZDH2MWIVA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:12.005905333 (48h6m10.557375794s) [noise=true , address=45.91.116.4:4939 , piece_id=KMHEQSQTA2WEWMIO7JDQZUDLSRV4CA6OJUN2CYYUSA7SU2ZDQBCQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:16.703757864 (48h6m15.255228326s) [noise=true , address=145.79.227.9:4366 , piece_id=JDBR4PTBZ5RN6NTWPERVLFANUAZQF3FSGIENVRHSYN56ED5NT2AQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:17.540298352 (48h6m16.091768813s) [noise=true , address=145.79.227.21:4474 , piece_id=RNB5T6DIUCY7IV2ULLO2UJRDKUJXI34UM4ZCHPJB22DNQ47VZJVA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:19.805526866 (48h6m18.356997337s) [noise=true , address=145.79.227.27:4689 , piece_id=KLTJYVDWHEE2BGLUSP2XT6L5R2C5OB7TF4DZLYBF6R5OG3N6AEFQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:24.397542313 (48h6m22.949012764s) [noise=true , address=145.79.227.17:4970 , piece_id=F2227LWTX5PRCQIUEJNRJ3QS3GRJOIEJIZFV3KRF7BAEAPSQZR7A, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:24.779686050 (48h6m23.331156511s) [noise=true , address=145.79.227.14:4840 , piece_id=QASWXZB3E6E7VYGF24KZ37Q3LKT6F4WVFKTBV3YS73KYAWTBV7YQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:25.298168925 (48h6m23.849639387s) [noise=true , address=145.79.227.11:4729 , piece_id=CDIU7FFPCVQGDK3K5ZDUARJ6VGB7KIY2DEVL7ZRLW5EASIFEHKHA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:35.044925402 (48h6m33.596395853s) [noise=true , address=145.79.227.30:4761 , piece_id=ASO5O5LM6CEF6DXGBVOCI3NOCWLGKDU2T5JHOL5ZPWEXMTRMKZIA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:37.306959937 (48h6m35.858430398s) [noise=true , address=145.79.227.20:4437 , piece_id=AOK6MTRZKPCXTNHZWGBYJGRBYNHFWO7GSG2PT4DFSMDAWK75GY4A, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:37.306987550 (48h6m35.858458011s) [seg_pos={0 0}, upload=42408]: Segment reached optimal threshold of 13 pieces. Nov 11 08:10:37.307209231 (48h6m35.858679702s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4): protocol: expected piece hash; context canceled; EOF Nov 11 08:10:37.307218759 (48h6m35.85868922s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307368936 (48h6m35.858839397s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw): protocol: expected piece hash; context canceled; EOF Nov 11 08:10:37.307376741 (48h6m35.858847202s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307561282 (48h6m35.859031743s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9): protocol: write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection Nov 11 08:10:37.307585969 (48h6m35.85905643s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307629221 (48h6m35.859099682s) [seg_pos={0 0}, upload=42408]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 11 08:10:37.307741005 (48h6m35.859211466s) [upload=42408]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 11 08:10:37.307750052 (48h6m35.859220513s) [upload=42408]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 11 08:10:37.612339386 (48h6m36.163809848s) [upload=42408]: Done uploading segments... Nov 11 08:10:37.612366317 (48h6m36.163836788s) [upload=42408]: Upload finished. err: <nil> Duration: 5.776599532s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 5.848817463s trace: https://jaeger.dev.storj.io/trace/49621e5e6beafe8e FAIL (2025-11-11 08:38:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 11 08:10:09.433180409 (48h6m7.98465087s) [upload=42408]: Starting upload Nov 11 08:10:09.433429884 (48h6m7.984900335s) [upload=42408]: Done starting upload Nov 11 08:10:09.433567967 (48h6m7.985038438s) [upload=42408]: Uploading segments... Nov 11 08:10:09.434191213 (48h6m7.985661674s) [upload=42408]: Got next segment. Inline: false Nov 11 08:10:09.434274032 (48h6m7.985744493s) [upload=42408]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 08:10:09.434282217 (48h6m7.985752688s) [upload=42408]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 08:10:09.714872160 (48h6m8.266342621s) [seg_pos={0 0}, upload=42408]: Begin upload segment... Nov 11 08:10:09.715084585 (48h6m8.266555046s) [seg_pos={0 0}, upload=42408]: Done begin upload segment. Nov 11 08:10:09.715464638 (48h6m8.2669351s) [noise=true , address=45.91.116.8:4047 , piece_id=VP6VW2ZTNSKS35R47GGJZI4YSHNNFUFB4YQ3W3QFT5ENN7F73VQA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.715555221 (48h6m8.267025692s) [noise=true , address=145.79.227.30:4761 , piece_id=ASO5O5LM6CEF6DXGBVOCI3NOCWLGKDU2T5JHOL5ZPWEXMTRMKZIA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.718085410 (48h6m8.269555901s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.718721901 (48h6m8.270192402s) [noise=true , address=145.79.227.11:4729 , piece_id=CDIU7FFPCVQGDK3K5ZDUARJ6VGB7KIY2DEVL7ZRLW5EASIFEHKHA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.719505724 (48h6m8.270976195s) [noise=true , address=145.79.227.27:4689 , piece_id=KLTJYVDWHEE2BGLUSP2XT6L5R2C5OB7TF4DZLYBF6R5OG3N6AEFQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.720136295 (48h6m8.271606756s) [noise=true , address=145.79.227.14:4840 , piece_id=QASWXZB3E6E7VYGF24KZ37Q3LKT6F4WVFKTBV3YS73KYAWTBV7YQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.721166056 (48h6m8.272636517s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.721883453 (48h6m8.273353994s) [noise=true , address=45.91.116.4:4939 , piece_id=KMHEQSQTA2WEWMIO7JDQZUDLSRV4CA6OJUN2CYYUSA7SU2ZDQBCQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.722398593 (48h6m8.273869054s) [noise=true , address=77.90.31.18:4806 , piece_id=LNL5LRHH3OOELKEO4HND2K3GNVDPCBCGKBNJ75DCLEKRAOMTQLKA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.724101306 (48h6m8.275571767s) [noise=true , address=145.79.227.17:4970 , piece_id=F2227LWTX5PRCQIUEJNRJ3QS3GRJOIEJIZFV3KRF7BAEAPSQZR7A, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.726385264 (48h6m8.277855776s) [noise=true , address=45.91.116.11:4117 , piece_id=UOEFPOPOFIO5KXY2JOX54YHUWNMTCAFMXEBKAMMR4GSR5ZE6J6PA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.728323225 (48h6m8.279793686s) [noise=true , address=145.79.227.20:4437 , piece_id=AOK6MTRZKPCXTNHZWGBYJGRBYNHFWO7GSG2PT4DFSMDAWK75GY4A, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.730321541 (48h6m8.281792002s) [noise=true , address=145.79.227.9:4366 , piece_id=JDBR4PTBZ5RN6NTWPERVLFANUAZQF3FSGIENVRHSYN56ED5NT2AQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.731324300 (48h6m8.282794832s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.732226989 (48h6m8.28369749s) [noise=true , address=77.90.31.19:4841 , piece_id=6JVB2WIEA2STMNBVTM6D7IYFHQY4QJGXAYHJAQILMYZZDH2MWIVA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.733026101 (48h6m8.284496562s) [noise=true , address=145.79.227.21:4474 , piece_id=RNB5T6DIUCY7IV2ULLO2UJRDKUJXI34UM4ZCHPJB22DNQ47VZJVA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:11.398415378 (48h6m9.949885839s) [upload=42408]: Next returned nil segment Nov 11 08:10:11.398434043 (48h6m9.949904504s) [upload=42408]: Waiting for error group managing segments... Nov 11 08:10:11.796154582 (48h6m10.347625043s) [noise=true , address=77.90.31.18:4806 , piece_id=LNL5LRHH3OOELKEO4HND2K3GNVDPCBCGKBNJ75DCLEKRAOMTQLKA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.809043214 (48h6m10.360513675s) [noise=true , address=45.91.116.8:4047 , piece_id=VP6VW2ZTNSKS35R47GGJZI4YSHNNFUFB4YQ3W3QFT5ENN7F73VQA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.866335992 (48h6m10.417806453s) [noise=true , address=45.91.116.11:4117 , piece_id=UOEFPOPOFIO5KXY2JOX54YHUWNMTCAFMXEBKAMMR4GSR5ZE6J6PA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.932922087 (48h6m10.484392538s) [noise=true , address=77.90.31.19:4841 , piece_id=6JVB2WIEA2STMNBVTM6D7IYFHQY4QJGXAYHJAQILMYZZDH2MWIVA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:12.005905333 (48h6m10.557375794s) [noise=true , address=45.91.116.4:4939 , piece_id=KMHEQSQTA2WEWMIO7JDQZUDLSRV4CA6OJUN2CYYUSA7SU2ZDQBCQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:16.703757864 (48h6m15.255228326s) [noise=true , address=145.79.227.9:4366 , piece_id=JDBR4PTBZ5RN6NTWPERVLFANUAZQF3FSGIENVRHSYN56ED5NT2AQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:17.540298352 (48h6m16.091768813s) [noise=true , address=145.79.227.21:4474 , piece_id=RNB5T6DIUCY7IV2ULLO2UJRDKUJXI34UM4ZCHPJB22DNQ47VZJVA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:19.805526866 (48h6m18.356997337s) [noise=true , address=145.79.227.27:4689 , piece_id=KLTJYVDWHEE2BGLUSP2XT6L5R2C5OB7TF4DZLYBF6R5OG3N6AEFQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:24.397542313 (48h6m22.949012764s) [noise=true , address=145.79.227.17:4970 , piece_id=F2227LWTX5PRCQIUEJNRJ3QS3GRJOIEJIZFV3KRF7BAEAPSQZR7A, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:24.779686050 (48h6m23.331156511s) [noise=true , address=145.79.227.14:4840 , piece_id=QASWXZB3E6E7VYGF24KZ37Q3LKT6F4WVFKTBV3YS73KYAWTBV7YQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:25.298168925 (48h6m23.849639387s) [noise=true , address=145.79.227.11:4729 , piece_id=CDIU7FFPCVQGDK3K5ZDUARJ6VGB7KIY2DEVL7ZRLW5EASIFEHKHA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:35.044925402 (48h6m33.596395853s) [noise=true , address=145.79.227.30:4761 , piece_id=ASO5O5LM6CEF6DXGBVOCI3NOCWLGKDU2T5JHOL5ZPWEXMTRMKZIA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:37.306959937 (48h6m35.858430398s) [noise=true , address=145.79.227.20:4437 , piece_id=AOK6MTRZKPCXTNHZWGBYJGRBYNHFWO7GSG2PT4DFSMDAWK75GY4A, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:37.306987550 (48h6m35.858458011s) [seg_pos={0 0}, upload=42408]: Segment reached optimal threshold of 13 pieces. Nov 11 08:10:37.307209231 (48h6m35.858679702s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4): protocol: expected piece hash; context canceled; EOF Nov 11 08:10:37.307218759 (48h6m35.85868922s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307368936 (48h6m35.858839397s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw): protocol: expected piece hash; context canceled; EOF Nov 11 08:10:37.307376741 (48h6m35.858847202s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307561282 (48h6m35.859031743s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9): protocol: write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection Nov 11 08:10:37.307585969 (48h6m35.85905643s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307629221 (48h6m35.859099682s) [seg_pos={0 0}, upload=42408]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 11 08:10:37.307741005 (48h6m35.859211466s) [upload=42408]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 11 08:10:37.307750052 (48h6m35.859220513s) [upload=42408]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 11 08:10:37.612339386 (48h6m36.163809848s) [upload=42408]: Done uploading segments... Nov 11 08:10:37.612366317 (48h6m36.163836788s) [upload=42408]: Upload finished. err: <nil> Duration: 5.848848562s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 6.014139898s trace: https://jaeger.dev.storj.io/trace/49621e5e6beafe8e FAIL (2025-11-11 08:37:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 11 08:10:09.433180409 (48h6m7.98465087s) [upload=42408]: Starting upload Nov 11 08:10:09.433429884 (48h6m7.984900335s) [upload=42408]: Done starting upload Nov 11 08:10:09.433567967 (48h6m7.985038438s) [upload=42408]: Uploading segments... Nov 11 08:10:09.434191213 (48h6m7.985661674s) [upload=42408]: Got next segment. Inline: false Nov 11 08:10:09.434274032 (48h6m7.985744493s) [upload=42408]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 08:10:09.434282217 (48h6m7.985752688s) [upload=42408]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 08:10:09.714872160 (48h6m8.266342621s) [seg_pos={0 0}, upload=42408]: Begin upload segment... Nov 11 08:10:09.715084585 (48h6m8.266555046s) [seg_pos={0 0}, upload=42408]: Done begin upload segment. Nov 11 08:10:09.715464638 (48h6m8.2669351s) [noise=true , address=45.91.116.8:4047 , piece_id=VP6VW2ZTNSKS35R47GGJZI4YSHNNFUFB4YQ3W3QFT5ENN7F73VQA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.715555221 (48h6m8.267025692s) [noise=true , address=145.79.227.30:4761 , piece_id=ASO5O5LM6CEF6DXGBVOCI3NOCWLGKDU2T5JHOL5ZPWEXMTRMKZIA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.718085410 (48h6m8.269555901s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.718721901 (48h6m8.270192402s) [noise=true , address=145.79.227.11:4729 , piece_id=CDIU7FFPCVQGDK3K5ZDUARJ6VGB7KIY2DEVL7ZRLW5EASIFEHKHA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.719505724 (48h6m8.270976195s) [noise=true , address=145.79.227.27:4689 , piece_id=KLTJYVDWHEE2BGLUSP2XT6L5R2C5OB7TF4DZLYBF6R5OG3N6AEFQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.720136295 (48h6m8.271606756s) [noise=true , address=145.79.227.14:4840 , piece_id=QASWXZB3E6E7VYGF24KZ37Q3LKT6F4WVFKTBV3YS73KYAWTBV7YQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.721166056 (48h6m8.272636517s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.721883453 (48h6m8.273353994s) [noise=true , address=45.91.116.4:4939 , piece_id=KMHEQSQTA2WEWMIO7JDQZUDLSRV4CA6OJUN2CYYUSA7SU2ZDQBCQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.722398593 (48h6m8.273869054s) [noise=true , address=77.90.31.18:4806 , piece_id=LNL5LRHH3OOELKEO4HND2K3GNVDPCBCGKBNJ75DCLEKRAOMTQLKA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.724101306 (48h6m8.275571767s) [noise=true , address=145.79.227.17:4970 , piece_id=F2227LWTX5PRCQIUEJNRJ3QS3GRJOIEJIZFV3KRF7BAEAPSQZR7A, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.726385264 (48h6m8.277855776s) [noise=true , address=45.91.116.11:4117 , piece_id=UOEFPOPOFIO5KXY2JOX54YHUWNMTCAFMXEBKAMMR4GSR5ZE6J6PA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.728323225 (48h6m8.279793686s) [noise=true , address=145.79.227.20:4437 , piece_id=AOK6MTRZKPCXTNHZWGBYJGRBYNHFWO7GSG2PT4DFSMDAWK75GY4A, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.730321541 (48h6m8.281792002s) [noise=true , address=145.79.227.9:4366 , piece_id=JDBR4PTBZ5RN6NTWPERVLFANUAZQF3FSGIENVRHSYN56ED5NT2AQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.731324300 (48h6m8.282794832s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.732226989 (48h6m8.28369749s) [noise=true , address=77.90.31.19:4841 , piece_id=6JVB2WIEA2STMNBVTM6D7IYFHQY4QJGXAYHJAQILMYZZDH2MWIVA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.733026101 (48h6m8.284496562s) [noise=true , address=145.79.227.21:4474 , piece_id=RNB5T6DIUCY7IV2ULLO2UJRDKUJXI34UM4ZCHPJB22DNQ47VZJVA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:11.398415378 (48h6m9.949885839s) [upload=42408]: Next returned nil segment Nov 11 08:10:11.398434043 (48h6m9.949904504s) [upload=42408]: Waiting for error group managing segments... Nov 11 08:10:11.796154582 (48h6m10.347625043s) [noise=true , address=77.90.31.18:4806 , piece_id=LNL5LRHH3OOELKEO4HND2K3GNVDPCBCGKBNJ75DCLEKRAOMTQLKA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.809043214 (48h6m10.360513675s) [noise=true , address=45.91.116.8:4047 , piece_id=VP6VW2ZTNSKS35R47GGJZI4YSHNNFUFB4YQ3W3QFT5ENN7F73VQA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.866335992 (48h6m10.417806453s) [noise=true , address=45.91.116.11:4117 , piece_id=UOEFPOPOFIO5KXY2JOX54YHUWNMTCAFMXEBKAMMR4GSR5ZE6J6PA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.932922087 (48h6m10.484392538s) [noise=true , address=77.90.31.19:4841 , piece_id=6JVB2WIEA2STMNBVTM6D7IYFHQY4QJGXAYHJAQILMYZZDH2MWIVA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:12.005905333 (48h6m10.557375794s) [noise=true , address=45.91.116.4:4939 , piece_id=KMHEQSQTA2WEWMIO7JDQZUDLSRV4CA6OJUN2CYYUSA7SU2ZDQBCQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:16.703757864 (48h6m15.255228326s) [noise=true , address=145.79.227.9:4366 , piece_id=JDBR4PTBZ5RN6NTWPERVLFANUAZQF3FSGIENVRHSYN56ED5NT2AQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:17.540298352 (48h6m16.091768813s) [noise=true , address=145.79.227.21:4474 , piece_id=RNB5T6DIUCY7IV2ULLO2UJRDKUJXI34UM4ZCHPJB22DNQ47VZJVA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:19.805526866 (48h6m18.356997337s) [noise=true , address=145.79.227.27:4689 , piece_id=KLTJYVDWHEE2BGLUSP2XT6L5R2C5OB7TF4DZLYBF6R5OG3N6AEFQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:24.397542313 (48h6m22.949012764s) [noise=true , address=145.79.227.17:4970 , piece_id=F2227LWTX5PRCQIUEJNRJ3QS3GRJOIEJIZFV3KRF7BAEAPSQZR7A, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:24.779686050 (48h6m23.331156511s) [noise=true , address=145.79.227.14:4840 , piece_id=QASWXZB3E6E7VYGF24KZ37Q3LKT6F4WVFKTBV3YS73KYAWTBV7YQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:25.298168925 (48h6m23.849639387s) [noise=true , address=145.79.227.11:4729 , piece_id=CDIU7FFPCVQGDK3K5ZDUARJ6VGB7KIY2DEVL7ZRLW5EASIFEHKHA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:35.044925402 (48h6m33.596395853s) [noise=true , address=145.79.227.30:4761 , piece_id=ASO5O5LM6CEF6DXGBVOCI3NOCWLGKDU2T5JHOL5ZPWEXMTRMKZIA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:37.306959937 (48h6m35.858430398s) [noise=true , address=145.79.227.20:4437 , piece_id=AOK6MTRZKPCXTNHZWGBYJGRBYNHFWO7GSG2PT4DFSMDAWK75GY4A, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:37.306987550 (48h6m35.858458011s) [seg_pos={0 0}, upload=42408]: Segment reached optimal threshold of 13 pieces. Nov 11 08:10:37.307209231 (48h6m35.858679702s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4): protocol: expected piece hash; context canceled; EOF Nov 11 08:10:37.307218759 (48h6m35.85868922s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307368936 (48h6m35.858839397s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw): protocol: expected piece hash; context canceled; EOF Nov 11 08:10:37.307376741 (48h6m35.858847202s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307561282 (48h6m35.859031743s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9): protocol: write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection Nov 11 08:10:37.307585969 (48h6m35.85905643s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307629221 (48h6m35.859099682s) [seg_pos={0 0}, upload=42408]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 11 08:10:37.307741005 (48h6m35.859211466s) [upload=42408]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 11 08:10:37.307750052 (48h6m35.859220513s) [upload=42408]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 11 08:10:37.612339386 (48h6m36.163809848s) [upload=42408]: Done uploading segments... Nov 11 08:10:37.612366317 (48h6m36.163836788s) [upload=42408]: Upload finished. err: <nil> Duration: 6.014170015s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 5.83627852s trace: https://jaeger.dev.storj.io/trace/49621e5e6beafe8e FAIL (2025-11-11 08:36:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 11 08:10:09.433180409 (48h6m7.98465087s) [upload=42408]: Starting upload Nov 11 08:10:09.433429884 (48h6m7.984900335s) [upload=42408]: Done starting upload Nov 11 08:10:09.433567967 (48h6m7.985038438s) [upload=42408]: Uploading segments... Nov 11 08:10:09.434191213 (48h6m7.985661674s) [upload=42408]: Got next segment. Inline: false Nov 11 08:10:09.434274032 (48h6m7.985744493s) [upload=42408]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 08:10:09.434282217 (48h6m7.985752688s) [upload=42408]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 08:10:09.714872160 (48h6m8.266342621s) [seg_pos={0 0}, upload=42408]: Begin upload segment... Nov 11 08:10:09.715084585 (48h6m8.266555046s) [seg_pos={0 0}, upload=42408]: Done begin upload segment. Nov 11 08:10:09.715464638 (48h6m8.2669351s) [noise=true , address=45.91.116.8:4047 , piece_id=VP6VW2ZTNSKS35R47GGJZI4YSHNNFUFB4YQ3W3QFT5ENN7F73VQA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.715555221 (48h6m8.267025692s) [noise=true , address=145.79.227.30:4761 , piece_id=ASO5O5LM6CEF6DXGBVOCI3NOCWLGKDU2T5JHOL5ZPWEXMTRMKZIA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.718085410 (48h6m8.269555901s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.718721901 (48h6m8.270192402s) [noise=true , address=145.79.227.11:4729 , piece_id=CDIU7FFPCVQGDK3K5ZDUARJ6VGB7KIY2DEVL7ZRLW5EASIFEHKHA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.719505724 (48h6m8.270976195s) [noise=true , address=145.79.227.27:4689 , piece_id=KLTJYVDWHEE2BGLUSP2XT6L5R2C5OB7TF4DZLYBF6R5OG3N6AEFQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.720136295 (48h6m8.271606756s) [noise=true , address=145.79.227.14:4840 , piece_id=QASWXZB3E6E7VYGF24KZ37Q3LKT6F4WVFKTBV3YS73KYAWTBV7YQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.721166056 (48h6m8.272636517s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.721883453 (48h6m8.273353994s) [noise=true , address=45.91.116.4:4939 , piece_id=KMHEQSQTA2WEWMIO7JDQZUDLSRV4CA6OJUN2CYYUSA7SU2ZDQBCQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.722398593 (48h6m8.273869054s) [noise=true , address=77.90.31.18:4806 , piece_id=LNL5LRHH3OOELKEO4HND2K3GNVDPCBCGKBNJ75DCLEKRAOMTQLKA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.724101306 (48h6m8.275571767s) [noise=true , address=145.79.227.17:4970 , piece_id=F2227LWTX5PRCQIUEJNRJ3QS3GRJOIEJIZFV3KRF7BAEAPSQZR7A, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.726385264 (48h6m8.277855776s) [noise=true , address=45.91.116.11:4117 , piece_id=UOEFPOPOFIO5KXY2JOX54YHUWNMTCAFMXEBKAMMR4GSR5ZE6J6PA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.728323225 (48h6m8.279793686s) [noise=true , address=145.79.227.20:4437 , piece_id=AOK6MTRZKPCXTNHZWGBYJGRBYNHFWO7GSG2PT4DFSMDAWK75GY4A, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.730321541 (48h6m8.281792002s) [noise=true , address=145.79.227.9:4366 , piece_id=JDBR4PTBZ5RN6NTWPERVLFANUAZQF3FSGIENVRHSYN56ED5NT2AQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.731324300 (48h6m8.282794832s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.732226989 (48h6m8.28369749s) [noise=true , address=77.90.31.19:4841 , piece_id=6JVB2WIEA2STMNBVTM6D7IYFHQY4QJGXAYHJAQILMYZZDH2MWIVA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.733026101 (48h6m8.284496562s) [noise=true , address=145.79.227.21:4474 , piece_id=RNB5T6DIUCY7IV2ULLO2UJRDKUJXI34UM4ZCHPJB22DNQ47VZJVA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:11.398415378 (48h6m9.949885839s) [upload=42408]: Next returned nil segment Nov 11 08:10:11.398434043 (48h6m9.949904504s) [upload=42408]: Waiting for error group managing segments... Nov 11 08:10:11.796154582 (48h6m10.347625043s) [noise=true , address=77.90.31.18:4806 , piece_id=LNL5LRHH3OOELKEO4HND2K3GNVDPCBCGKBNJ75DCLEKRAOMTQLKA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.809043214 (48h6m10.360513675s) [noise=true , address=45.91.116.8:4047 , piece_id=VP6VW2ZTNSKS35R47GGJZI4YSHNNFUFB4YQ3W3QFT5ENN7F73VQA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.866335992 (48h6m10.417806453s) [noise=true , address=45.91.116.11:4117 , piece_id=UOEFPOPOFIO5KXY2JOX54YHUWNMTCAFMXEBKAMMR4GSR5ZE6J6PA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.932922087 (48h6m10.484392538s) [noise=true , address=77.90.31.19:4841 , piece_id=6JVB2WIEA2STMNBVTM6D7IYFHQY4QJGXAYHJAQILMYZZDH2MWIVA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:12.005905333 (48h6m10.557375794s) [noise=true , address=45.91.116.4:4939 , piece_id=KMHEQSQTA2WEWMIO7JDQZUDLSRV4CA6OJUN2CYYUSA7SU2ZDQBCQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:16.703757864 (48h6m15.255228326s) [noise=true , address=145.79.227.9:4366 , piece_id=JDBR4PTBZ5RN6NTWPERVLFANUAZQF3FSGIENVRHSYN56ED5NT2AQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:17.540298352 (48h6m16.091768813s) [noise=true , address=145.79.227.21:4474 , piece_id=RNB5T6DIUCY7IV2ULLO2UJRDKUJXI34UM4ZCHPJB22DNQ47VZJVA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:19.805526866 (48h6m18.356997337s) [noise=true , address=145.79.227.27:4689 , piece_id=KLTJYVDWHEE2BGLUSP2XT6L5R2C5OB7TF4DZLYBF6R5OG3N6AEFQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:24.397542313 (48h6m22.949012764s) [noise=true , address=145.79.227.17:4970 , piece_id=F2227LWTX5PRCQIUEJNRJ3QS3GRJOIEJIZFV3KRF7BAEAPSQZR7A, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:24.779686050 (48h6m23.331156511s) [noise=true , address=145.79.227.14:4840 , piece_id=QASWXZB3E6E7VYGF24KZ37Q3LKT6F4WVFKTBV3YS73KYAWTBV7YQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:25.298168925 (48h6m23.849639387s) [noise=true , address=145.79.227.11:4729 , piece_id=CDIU7FFPCVQGDK3K5ZDUARJ6VGB7KIY2DEVL7ZRLW5EASIFEHKHA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:35.044925402 (48h6m33.596395853s) [noise=true , address=145.79.227.30:4761 , piece_id=ASO5O5LM6CEF6DXGBVOCI3NOCWLGKDU2T5JHOL5ZPWEXMTRMKZIA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:37.306959937 (48h6m35.858430398s) [noise=true , address=145.79.227.20:4437 , piece_id=AOK6MTRZKPCXTNHZWGBYJGRBYNHFWO7GSG2PT4DFSMDAWK75GY4A, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:37.306987550 (48h6m35.858458011s) [seg_pos={0 0}, upload=42408]: Segment reached optimal threshold of 13 pieces. Nov 11 08:10:37.307209231 (48h6m35.858679702s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4): protocol: expected piece hash; context canceled; EOF Nov 11 08:10:37.307218759 (48h6m35.85868922s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307368936 (48h6m35.858839397s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw): protocol: expected piece hash; context canceled; EOF Nov 11 08:10:37.307376741 (48h6m35.858847202s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307561282 (48h6m35.859031743s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9): protocol: write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection Nov 11 08:10:37.307585969 (48h6m35.85905643s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307629221 (48h6m35.859099682s) [seg_pos={0 0}, upload=42408]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 11 08:10:37.307741005 (48h6m35.859211466s) [upload=42408]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 11 08:10:37.307750052 (48h6m35.859220513s) [upload=42408]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 11 08:10:37.612339386 (48h6m36.163809848s) [upload=42408]: Done uploading segments... Nov 11 08:10:37.612366317 (48h6m36.163836788s) [upload=42408]: Upload finished. err: <nil> Duration: 5.836292025s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 5.910217995s trace: https://jaeger.dev.storj.io/trace/49621e5e6beafe8e FAIL (2025-11-11 08:35:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 11 08:10:09.433180409 (48h6m7.98465087s) [upload=42408]: Starting upload Nov 11 08:10:09.433429884 (48h6m7.984900335s) [upload=42408]: Done starting upload Nov 11 08:10:09.433567967 (48h6m7.985038438s) [upload=42408]: Uploading segments... Nov 11 08:10:09.434191213 (48h6m7.985661674s) [upload=42408]: Got next segment. Inline: false Nov 11 08:10:09.434274032 (48h6m7.985744493s) [upload=42408]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 08:10:09.434282217 (48h6m7.985752688s) [upload=42408]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 08:10:09.714872160 (48h6m8.266342621s) [seg_pos={0 0}, upload=42408]: Begin upload segment... Nov 11 08:10:09.715084585 (48h6m8.266555046s) [seg_pos={0 0}, upload=42408]: Done begin upload segment. Nov 11 08:10:09.715464638 (48h6m8.2669351s) [noise=true , address=45.91.116.8:4047 , piece_id=VP6VW2ZTNSKS35R47GGJZI4YSHNNFUFB4YQ3W3QFT5ENN7F73VQA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.715555221 (48h6m8.267025692s) [noise=true , address=145.79.227.30:4761 , piece_id=ASO5O5LM6CEF6DXGBVOCI3NOCWLGKDU2T5JHOL5ZPWEXMTRMKZIA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.718085410 (48h6m8.269555901s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.718721901 (48h6m8.270192402s) [noise=true , address=145.79.227.11:4729 , piece_id=CDIU7FFPCVQGDK3K5ZDUARJ6VGB7KIY2DEVL7ZRLW5EASIFEHKHA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.719505724 (48h6m8.270976195s) [noise=true , address=145.79.227.27:4689 , piece_id=KLTJYVDWHEE2BGLUSP2XT6L5R2C5OB7TF4DZLYBF6R5OG3N6AEFQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.720136295 (48h6m8.271606756s) [noise=true , address=145.79.227.14:4840 , piece_id=QASWXZB3E6E7VYGF24KZ37Q3LKT6F4WVFKTBV3YS73KYAWTBV7YQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.721166056 (48h6m8.272636517s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.721883453 (48h6m8.273353994s) [noise=true , address=45.91.116.4:4939 , piece_id=KMHEQSQTA2WEWMIO7JDQZUDLSRV4CA6OJUN2CYYUSA7SU2ZDQBCQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.722398593 (48h6m8.273869054s) [noise=true , address=77.90.31.18:4806 , piece_id=LNL5LRHH3OOELKEO4HND2K3GNVDPCBCGKBNJ75DCLEKRAOMTQLKA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.724101306 (48h6m8.275571767s) [noise=true , address=145.79.227.17:4970 , piece_id=F2227LWTX5PRCQIUEJNRJ3QS3GRJOIEJIZFV3KRF7BAEAPSQZR7A, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.726385264 (48h6m8.277855776s) [noise=true , address=45.91.116.11:4117 , piece_id=UOEFPOPOFIO5KXY2JOX54YHUWNMTCAFMXEBKAMMR4GSR5ZE6J6PA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.728323225 (48h6m8.279793686s) [noise=true , address=145.79.227.20:4437 , piece_id=AOK6MTRZKPCXTNHZWGBYJGRBYNHFWO7GSG2PT4DFSMDAWK75GY4A, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.730321541 (48h6m8.281792002s) [noise=true , address=145.79.227.9:4366 , piece_id=JDBR4PTBZ5RN6NTWPERVLFANUAZQF3FSGIENVRHSYN56ED5NT2AQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.731324300 (48h6m8.282794832s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.732226989 (48h6m8.28369749s) [noise=true , address=77.90.31.19:4841 , piece_id=6JVB2WIEA2STMNBVTM6D7IYFHQY4QJGXAYHJAQILMYZZDH2MWIVA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.733026101 (48h6m8.284496562s) [noise=true , address=145.79.227.21:4474 , piece_id=RNB5T6DIUCY7IV2ULLO2UJRDKUJXI34UM4ZCHPJB22DNQ47VZJVA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:11.398415378 (48h6m9.949885839s) [upload=42408]: Next returned nil segment Nov 11 08:10:11.398434043 (48h6m9.949904504s) [upload=42408]: Waiting for error group managing segments... Nov 11 08:10:11.796154582 (48h6m10.347625043s) [noise=true , address=77.90.31.18:4806 , piece_id=LNL5LRHH3OOELKEO4HND2K3GNVDPCBCGKBNJ75DCLEKRAOMTQLKA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.809043214 (48h6m10.360513675s) [noise=true , address=45.91.116.8:4047 , piece_id=VP6VW2ZTNSKS35R47GGJZI4YSHNNFUFB4YQ3W3QFT5ENN7F73VQA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.866335992 (48h6m10.417806453s) [noise=true , address=45.91.116.11:4117 , piece_id=UOEFPOPOFIO5KXY2JOX54YHUWNMTCAFMXEBKAMMR4GSR5ZE6J6PA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.932922087 (48h6m10.484392538s) [noise=true , address=77.90.31.19:4841 , piece_id=6JVB2WIEA2STMNBVTM6D7IYFHQY4QJGXAYHJAQILMYZZDH2MWIVA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:12.005905333 (48h6m10.557375794s) [noise=true , address=45.91.116.4:4939 , piece_id=KMHEQSQTA2WEWMIO7JDQZUDLSRV4CA6OJUN2CYYUSA7SU2ZDQBCQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:16.703757864 (48h6m15.255228326s) [noise=true , address=145.79.227.9:4366 , piece_id=JDBR4PTBZ5RN6NTWPERVLFANUAZQF3FSGIENVRHSYN56ED5NT2AQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:17.540298352 (48h6m16.091768813s) [noise=true , address=145.79.227.21:4474 , piece_id=RNB5T6DIUCY7IV2ULLO2UJRDKUJXI34UM4ZCHPJB22DNQ47VZJVA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:19.805526866 (48h6m18.356997337s) [noise=true , address=145.79.227.27:4689 , piece_id=KLTJYVDWHEE2BGLUSP2XT6L5R2C5OB7TF4DZLYBF6R5OG3N6AEFQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:24.397542313 (48h6m22.949012764s) [noise=true , address=145.79.227.17:4970 , piece_id=F2227LWTX5PRCQIUEJNRJ3QS3GRJOIEJIZFV3KRF7BAEAPSQZR7A, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:24.779686050 (48h6m23.331156511s) [noise=true , address=145.79.227.14:4840 , piece_id=QASWXZB3E6E7VYGF24KZ37Q3LKT6F4WVFKTBV3YS73KYAWTBV7YQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:25.298168925 (48h6m23.849639387s) [noise=true , address=145.79.227.11:4729 , piece_id=CDIU7FFPCVQGDK3K5ZDUARJ6VGB7KIY2DEVL7ZRLW5EASIFEHKHA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:35.044925402 (48h6m33.596395853s) [noise=true , address=145.79.227.30:4761 , piece_id=ASO5O5LM6CEF6DXGBVOCI3NOCWLGKDU2T5JHOL5ZPWEXMTRMKZIA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:37.306959937 (48h6m35.858430398s) [noise=true , address=145.79.227.20:4437 , piece_id=AOK6MTRZKPCXTNHZWGBYJGRBYNHFWO7GSG2PT4DFSMDAWK75GY4A, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:37.306987550 (48h6m35.858458011s) [seg_pos={0 0}, upload=42408]: Segment reached optimal threshold of 13 pieces. Nov 11 08:10:37.307209231 (48h6m35.858679702s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4): protocol: expected piece hash; context canceled; EOF Nov 11 08:10:37.307218759 (48h6m35.85868922s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307368936 (48h6m35.858839397s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw): protocol: expected piece hash; context canceled; EOF Nov 11 08:10:37.307376741 (48h6m35.858847202s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307561282 (48h6m35.859031743s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9): protocol: write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection Nov 11 08:10:37.307585969 (48h6m35.85905643s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307629221 (48h6m35.859099682s) [seg_pos={0 0}, upload=42408]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 11 08:10:37.307741005 (48h6m35.859211466s) [upload=42408]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 11 08:10:37.307750052 (48h6m35.859220513s) [upload=42408]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 11 08:10:37.612339386 (48h6m36.163809848s) [upload=42408]: Done uploading segments... Nov 11 08:10:37.612366317 (48h6m36.163836788s) [upload=42408]: Upload finished. err: <nil> Duration: 5.910258212s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 5.8211687s trace: https://jaeger.dev.storj.io/trace/49621e5e6beafe8e FAIL (2025-11-11 08:34:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 11 08:10:09.433180409 (48h6m7.98465087s) [upload=42408]: Starting upload Nov 11 08:10:09.433429884 (48h6m7.984900335s) [upload=42408]: Done starting upload Nov 11 08:10:09.433567967 (48h6m7.985038438s) [upload=42408]: Uploading segments... Nov 11 08:10:09.434191213 (48h6m7.985661674s) [upload=42408]: Got next segment. Inline: false Nov 11 08:10:09.434274032 (48h6m7.985744493s) [upload=42408]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 08:10:09.434282217 (48h6m7.985752688s) [upload=42408]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 08:10:09.714872160 (48h6m8.266342621s) [seg_pos={0 0}, upload=42408]: Begin upload segment... Nov 11 08:10:09.715084585 (48h6m8.266555046s) [seg_pos={0 0}, upload=42408]: Done begin upload segment. Nov 11 08:10:09.715464638 (48h6m8.2669351s) [noise=true , address=45.91.116.8:4047 , piece_id=VP6VW2ZTNSKS35R47GGJZI4YSHNNFUFB4YQ3W3QFT5ENN7F73VQA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.715555221 (48h6m8.267025692s) [noise=true , address=145.79.227.30:4761 , piece_id=ASO5O5LM6CEF6DXGBVOCI3NOCWLGKDU2T5JHOL5ZPWEXMTRMKZIA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.718085410 (48h6m8.269555901s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.718721901 (48h6m8.270192402s) [noise=true , address=145.79.227.11:4729 , piece_id=CDIU7FFPCVQGDK3K5ZDUARJ6VGB7KIY2DEVL7ZRLW5EASIFEHKHA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.719505724 (48h6m8.270976195s) [noise=true , address=145.79.227.27:4689 , piece_id=KLTJYVDWHEE2BGLUSP2XT6L5R2C5OB7TF4DZLYBF6R5OG3N6AEFQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.720136295 (48h6m8.271606756s) [noise=true , address=145.79.227.14:4840 , piece_id=QASWXZB3E6E7VYGF24KZ37Q3LKT6F4WVFKTBV3YS73KYAWTBV7YQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.721166056 (48h6m8.272636517s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.721883453 (48h6m8.273353994s) [noise=true , address=45.91.116.4:4939 , piece_id=KMHEQSQTA2WEWMIO7JDQZUDLSRV4CA6OJUN2CYYUSA7SU2ZDQBCQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.722398593 (48h6m8.273869054s) [noise=true , address=77.90.31.18:4806 , piece_id=LNL5LRHH3OOELKEO4HND2K3GNVDPCBCGKBNJ75DCLEKRAOMTQLKA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.724101306 (48h6m8.275571767s) [noise=true , address=145.79.227.17:4970 , piece_id=F2227LWTX5PRCQIUEJNRJ3QS3GRJOIEJIZFV3KRF7BAEAPSQZR7A, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.726385264 (48h6m8.277855776s) [noise=true , address=45.91.116.11:4117 , piece_id=UOEFPOPOFIO5KXY2JOX54YHUWNMTCAFMXEBKAMMR4GSR5ZE6J6PA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.728323225 (48h6m8.279793686s) [noise=true , address=145.79.227.20:4437 , piece_id=AOK6MTRZKPCXTNHZWGBYJGRBYNHFWO7GSG2PT4DFSMDAWK75GY4A, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.730321541 (48h6m8.281792002s) [noise=true , address=145.79.227.9:4366 , piece_id=JDBR4PTBZ5RN6NTWPERVLFANUAZQF3FSGIENVRHSYN56ED5NT2AQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.731324300 (48h6m8.282794832s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.732226989 (48h6m8.28369749s) [noise=true , address=77.90.31.19:4841 , piece_id=6JVB2WIEA2STMNBVTM6D7IYFHQY4QJGXAYHJAQILMYZZDH2MWIVA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.733026101 (48h6m8.284496562s) [noise=true , address=145.79.227.21:4474 , piece_id=RNB5T6DIUCY7IV2ULLO2UJRDKUJXI34UM4ZCHPJB22DNQ47VZJVA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:11.398415378 (48h6m9.949885839s) [upload=42408]: Next returned nil segment Nov 11 08:10:11.398434043 (48h6m9.949904504s) [upload=42408]: Waiting for error group managing segments... Nov 11 08:10:11.796154582 (48h6m10.347625043s) [noise=true , address=77.90.31.18:4806 , piece_id=LNL5LRHH3OOELKEO4HND2K3GNVDPCBCGKBNJ75DCLEKRAOMTQLKA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.809043214 (48h6m10.360513675s) [noise=true , address=45.91.116.8:4047 , piece_id=VP6VW2ZTNSKS35R47GGJZI4YSHNNFUFB4YQ3W3QFT5ENN7F73VQA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.866335992 (48h6m10.417806453s) [noise=true , address=45.91.116.11:4117 , piece_id=UOEFPOPOFIO5KXY2JOX54YHUWNMTCAFMXEBKAMMR4GSR5ZE6J6PA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.932922087 (48h6m10.484392538s) [noise=true , address=77.90.31.19:4841 , piece_id=6JVB2WIEA2STMNBVTM6D7IYFHQY4QJGXAYHJAQILMYZZDH2MWIVA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:12.005905333 (48h6m10.557375794s) [noise=true , address=45.91.116.4:4939 , piece_id=KMHEQSQTA2WEWMIO7JDQZUDLSRV4CA6OJUN2CYYUSA7SU2ZDQBCQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:16.703757864 (48h6m15.255228326s) [noise=true , address=145.79.227.9:4366 , piece_id=JDBR4PTBZ5RN6NTWPERVLFANUAZQF3FSGIENVRHSYN56ED5NT2AQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:17.540298352 (48h6m16.091768813s) [noise=true , address=145.79.227.21:4474 , piece_id=RNB5T6DIUCY7IV2ULLO2UJRDKUJXI34UM4ZCHPJB22DNQ47VZJVA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:19.805526866 (48h6m18.356997337s) [noise=true , address=145.79.227.27:4689 , piece_id=KLTJYVDWHEE2BGLUSP2XT6L5R2C5OB7TF4DZLYBF6R5OG3N6AEFQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:24.397542313 (48h6m22.949012764s) [noise=true , address=145.79.227.17:4970 , piece_id=F2227LWTX5PRCQIUEJNRJ3QS3GRJOIEJIZFV3KRF7BAEAPSQZR7A, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:24.779686050 (48h6m23.331156511s) [noise=true , address=145.79.227.14:4840 , piece_id=QASWXZB3E6E7VYGF24KZ37Q3LKT6F4WVFKTBV3YS73KYAWTBV7YQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:25.298168925 (48h6m23.849639387s) [noise=true , address=145.79.227.11:4729 , piece_id=CDIU7FFPCVQGDK3K5ZDUARJ6VGB7KIY2DEVL7ZRLW5EASIFEHKHA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:35.044925402 (48h6m33.596395853s) [noise=true , address=145.79.227.30:4761 , piece_id=ASO5O5LM6CEF6DXGBVOCI3NOCWLGKDU2T5JHOL5ZPWEXMTRMKZIA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:37.306959937 (48h6m35.858430398s) [noise=true , address=145.79.227.20:4437 , piece_id=AOK6MTRZKPCXTNHZWGBYJGRBYNHFWO7GSG2PT4DFSMDAWK75GY4A, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:37.306987550 (48h6m35.858458011s) [seg_pos={0 0}, upload=42408]: Segment reached optimal threshold of 13 pieces. Nov 11 08:10:37.307209231 (48h6m35.858679702s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4): protocol: expected piece hash; context canceled; EOF Nov 11 08:10:37.307218759 (48h6m35.85868922s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307368936 (48h6m35.858839397s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw): protocol: expected piece hash; context canceled; EOF Nov 11 08:10:37.307376741 (48h6m35.858847202s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307561282 (48h6m35.859031743s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9): protocol: write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection Nov 11 08:10:37.307585969 (48h6m35.85905643s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307629221 (48h6m35.859099682s) [seg_pos={0 0}, upload=42408]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 11 08:10:37.307741005 (48h6m35.859211466s) [upload=42408]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 11 08:10:37.307750052 (48h6m35.859220513s) [upload=42408]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 11 08:10:37.612339386 (48h6m36.163809848s) [upload=42408]: Done uploading segments... Nov 11 08:10:37.612366317 (48h6m36.163836788s) [upload=42408]: Upload finished. err: <nil> Duration: 5.821201964s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 5.891540823s trace: https://jaeger.dev.storj.io/trace/49621e5e6beafe8e FAIL (2025-11-11 08:34:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 11 08:10:09.433180409 (48h6m7.98465087s) [upload=42408]: Starting upload Nov 11 08:10:09.433429884 (48h6m7.984900335s) [upload=42408]: Done starting upload Nov 11 08:10:09.433567967 (48h6m7.985038438s) [upload=42408]: Uploading segments... Nov 11 08:10:09.434191213 (48h6m7.985661674s) [upload=42408]: Got next segment. Inline: false Nov 11 08:10:09.434274032 (48h6m7.985744493s) [upload=42408]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 08:10:09.434282217 (48h6m7.985752688s) [upload=42408]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 08:10:09.714872160 (48h6m8.266342621s) [seg_pos={0 0}, upload=42408]: Begin upload segment... Nov 11 08:10:09.715084585 (48h6m8.266555046s) [seg_pos={0 0}, upload=42408]: Done begin upload segment. Nov 11 08:10:09.715464638 (48h6m8.2669351s) [noise=true , address=45.91.116.8:4047 , piece_id=VP6VW2ZTNSKS35R47GGJZI4YSHNNFUFB4YQ3W3QFT5ENN7F73VQA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.715555221 (48h6m8.267025692s) [noise=true , address=145.79.227.30:4761 , piece_id=ASO5O5LM6CEF6DXGBVOCI3NOCWLGKDU2T5JHOL5ZPWEXMTRMKZIA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.718085410 (48h6m8.269555901s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.718721901 (48h6m8.270192402s) [noise=true , address=145.79.227.11:4729 , piece_id=CDIU7FFPCVQGDK3K5ZDUARJ6VGB7KIY2DEVL7ZRLW5EASIFEHKHA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.719505724 (48h6m8.270976195s) [noise=true , address=145.79.227.27:4689 , piece_id=KLTJYVDWHEE2BGLUSP2XT6L5R2C5OB7TF4DZLYBF6R5OG3N6AEFQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.720136295 (48h6m8.271606756s) [noise=true , address=145.79.227.14:4840 , piece_id=QASWXZB3E6E7VYGF24KZ37Q3LKT6F4WVFKTBV3YS73KYAWTBV7YQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.721166056 (48h6m8.272636517s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.721883453 (48h6m8.273353994s) [noise=true , address=45.91.116.4:4939 , piece_id=KMHEQSQTA2WEWMIO7JDQZUDLSRV4CA6OJUN2CYYUSA7SU2ZDQBCQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.722398593 (48h6m8.273869054s) [noise=true , address=77.90.31.18:4806 , piece_id=LNL5LRHH3OOELKEO4HND2K3GNVDPCBCGKBNJ75DCLEKRAOMTQLKA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.724101306 (48h6m8.275571767s) [noise=true , address=145.79.227.17:4970 , piece_id=F2227LWTX5PRCQIUEJNRJ3QS3GRJOIEJIZFV3KRF7BAEAPSQZR7A, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.726385264 (48h6m8.277855776s) [noise=true , address=45.91.116.11:4117 , piece_id=UOEFPOPOFIO5KXY2JOX54YHUWNMTCAFMXEBKAMMR4GSR5ZE6J6PA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.728323225 (48h6m8.279793686s) [noise=true , address=145.79.227.20:4437 , piece_id=AOK6MTRZKPCXTNHZWGBYJGRBYNHFWO7GSG2PT4DFSMDAWK75GY4A, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.730321541 (48h6m8.281792002s) [noise=true , address=145.79.227.9:4366 , piece_id=JDBR4PTBZ5RN6NTWPERVLFANUAZQF3FSGIENVRHSYN56ED5NT2AQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.731324300 (48h6m8.282794832s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.732226989 (48h6m8.28369749s) [noise=true , address=77.90.31.19:4841 , piece_id=6JVB2WIEA2STMNBVTM6D7IYFHQY4QJGXAYHJAQILMYZZDH2MWIVA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.733026101 (48h6m8.284496562s) [noise=true , address=145.79.227.21:4474 , piece_id=RNB5T6DIUCY7IV2ULLO2UJRDKUJXI34UM4ZCHPJB22DNQ47VZJVA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:11.398415378 (48h6m9.949885839s) [upload=42408]: Next returned nil segment Nov 11 08:10:11.398434043 (48h6m9.949904504s) [upload=42408]: Waiting for error group managing segments... Nov 11 08:10:11.796154582 (48h6m10.347625043s) [noise=true , address=77.90.31.18:4806 , piece_id=LNL5LRHH3OOELKEO4HND2K3GNVDPCBCGKBNJ75DCLEKRAOMTQLKA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.809043214 (48h6m10.360513675s) [noise=true , address=45.91.116.8:4047 , piece_id=VP6VW2ZTNSKS35R47GGJZI4YSHNNFUFB4YQ3W3QFT5ENN7F73VQA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.866335992 (48h6m10.417806453s) [noise=true , address=45.91.116.11:4117 , piece_id=UOEFPOPOFIO5KXY2JOX54YHUWNMTCAFMXEBKAMMR4GSR5ZE6J6PA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.932922087 (48h6m10.484392538s) [noise=true , address=77.90.31.19:4841 , piece_id=6JVB2WIEA2STMNBVTM6D7IYFHQY4QJGXAYHJAQILMYZZDH2MWIVA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:12.005905333 (48h6m10.557375794s) [noise=true , address=45.91.116.4:4939 , piece_id=KMHEQSQTA2WEWMIO7JDQZUDLSRV4CA6OJUN2CYYUSA7SU2ZDQBCQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:16.703757864 (48h6m15.255228326s) [noise=true , address=145.79.227.9:4366 , piece_id=JDBR4PTBZ5RN6NTWPERVLFANUAZQF3FSGIENVRHSYN56ED5NT2AQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:17.540298352 (48h6m16.091768813s) [noise=true , address=145.79.227.21:4474 , piece_id=RNB5T6DIUCY7IV2ULLO2UJRDKUJXI34UM4ZCHPJB22DNQ47VZJVA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:19.805526866 (48h6m18.356997337s) [noise=true , address=145.79.227.27:4689 , piece_id=KLTJYVDWHEE2BGLUSP2XT6L5R2C5OB7TF4DZLYBF6R5OG3N6AEFQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:24.397542313 (48h6m22.949012764s) [noise=true , address=145.79.227.17:4970 , piece_id=F2227LWTX5PRCQIUEJNRJ3QS3GRJOIEJIZFV3KRF7BAEAPSQZR7A, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:24.779686050 (48h6m23.331156511s) [noise=true , address=145.79.227.14:4840 , piece_id=QASWXZB3E6E7VYGF24KZ37Q3LKT6F4WVFKTBV3YS73KYAWTBV7YQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:25.298168925 (48h6m23.849639387s) [noise=true , address=145.79.227.11:4729 , piece_id=CDIU7FFPCVQGDK3K5ZDUARJ6VGB7KIY2DEVL7ZRLW5EASIFEHKHA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:35.044925402 (48h6m33.596395853s) [noise=true , address=145.79.227.30:4761 , piece_id=ASO5O5LM6CEF6DXGBVOCI3NOCWLGKDU2T5JHOL5ZPWEXMTRMKZIA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:37.306959937 (48h6m35.858430398s) [noise=true , address=145.79.227.20:4437 , piece_id=AOK6MTRZKPCXTNHZWGBYJGRBYNHFWO7GSG2PT4DFSMDAWK75GY4A, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:37.306987550 (48h6m35.858458011s) [seg_pos={0 0}, upload=42408]: Segment reached optimal threshold of 13 pieces. Nov 11 08:10:37.307209231 (48h6m35.858679702s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4): protocol: expected piece hash; context canceled; EOF Nov 11 08:10:37.307218759 (48h6m35.85868922s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307368936 (48h6m35.858839397s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw): protocol: expected piece hash; context canceled; EOF Nov 11 08:10:37.307376741 (48h6m35.858847202s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307561282 (48h6m35.859031743s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9): protocol: write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection Nov 11 08:10:37.307585969 (48h6m35.85905643s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307629221 (48h6m35.859099682s) [seg_pos={0 0}, upload=42408]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 11 08:10:37.307741005 (48h6m35.859211466s) [upload=42408]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 11 08:10:37.307750052 (48h6m35.859220513s) [upload=42408]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 11 08:10:37.612339386 (48h6m36.163809848s) [upload=42408]: Done uploading segments... Nov 11 08:10:37.612366317 (48h6m36.163836788s) [upload=42408]: Upload finished. err: <nil> Duration: 5.891569948s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 5.862994113s trace: https://jaeger.dev.storj.io/trace/49621e5e6beafe8e FAIL (2025-11-11 08:33:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 11 08:10:09.433180409 (48h6m7.98465087s) [upload=42408]: Starting upload Nov 11 08:10:09.433429884 (48h6m7.984900335s) [upload=42408]: Done starting upload Nov 11 08:10:09.433567967 (48h6m7.985038438s) [upload=42408]: Uploading segments... Nov 11 08:10:09.434191213 (48h6m7.985661674s) [upload=42408]: Got next segment. Inline: false Nov 11 08:10:09.434274032 (48h6m7.985744493s) [upload=42408]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 08:10:09.434282217 (48h6m7.985752688s) [upload=42408]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 08:10:09.714872160 (48h6m8.266342621s) [seg_pos={0 0}, upload=42408]: Begin upload segment... Nov 11 08:10:09.715084585 (48h6m8.266555046s) [seg_pos={0 0}, upload=42408]: Done begin upload segment. Nov 11 08:10:09.715464638 (48h6m8.2669351s) [noise=true , address=45.91.116.8:4047 , piece_id=VP6VW2ZTNSKS35R47GGJZI4YSHNNFUFB4YQ3W3QFT5ENN7F73VQA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.715555221 (48h6m8.267025692s) [noise=true , address=145.79.227.30:4761 , piece_id=ASO5O5LM6CEF6DXGBVOCI3NOCWLGKDU2T5JHOL5ZPWEXMTRMKZIA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.718085410 (48h6m8.269555901s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.718721901 (48h6m8.270192402s) [noise=true , address=145.79.227.11:4729 , piece_id=CDIU7FFPCVQGDK3K5ZDUARJ6VGB7KIY2DEVL7ZRLW5EASIFEHKHA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.719505724 (48h6m8.270976195s) [noise=true , address=145.79.227.27:4689 , piece_id=KLTJYVDWHEE2BGLUSP2XT6L5R2C5OB7TF4DZLYBF6R5OG3N6AEFQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.720136295 (48h6m8.271606756s) [noise=true , address=145.79.227.14:4840 , piece_id=QASWXZB3E6E7VYGF24KZ37Q3LKT6F4WVFKTBV3YS73KYAWTBV7YQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.721166056 (48h6m8.272636517s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.721883453 (48h6m8.273353994s) [noise=true , address=45.91.116.4:4939 , piece_id=KMHEQSQTA2WEWMIO7JDQZUDLSRV4CA6OJUN2CYYUSA7SU2ZDQBCQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.722398593 (48h6m8.273869054s) [noise=true , address=77.90.31.18:4806 , piece_id=LNL5LRHH3OOELKEO4HND2K3GNVDPCBCGKBNJ75DCLEKRAOMTQLKA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.724101306 (48h6m8.275571767s) [noise=true , address=145.79.227.17:4970 , piece_id=F2227LWTX5PRCQIUEJNRJ3QS3GRJOIEJIZFV3KRF7BAEAPSQZR7A, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.726385264 (48h6m8.277855776s) [noise=true , address=45.91.116.11:4117 , piece_id=UOEFPOPOFIO5KXY2JOX54YHUWNMTCAFMXEBKAMMR4GSR5ZE6J6PA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.728323225 (48h6m8.279793686s) [noise=true , address=145.79.227.20:4437 , piece_id=AOK6MTRZKPCXTNHZWGBYJGRBYNHFWO7GSG2PT4DFSMDAWK75GY4A, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.730321541 (48h6m8.281792002s) [noise=true , address=145.79.227.9:4366 , piece_id=JDBR4PTBZ5RN6NTWPERVLFANUAZQF3FSGIENVRHSYN56ED5NT2AQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.731324300 (48h6m8.282794832s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.732226989 (48h6m8.28369749s) [noise=true , address=77.90.31.19:4841 , piece_id=6JVB2WIEA2STMNBVTM6D7IYFHQY4QJGXAYHJAQILMYZZDH2MWIVA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.733026101 (48h6m8.284496562s) [noise=true , address=145.79.227.21:4474 , piece_id=RNB5T6DIUCY7IV2ULLO2UJRDKUJXI34UM4ZCHPJB22DNQ47VZJVA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:11.398415378 (48h6m9.949885839s) [upload=42408]: Next returned nil segment Nov 11 08:10:11.398434043 (48h6m9.949904504s) [upload=42408]: Waiting for error group managing segments... Nov 11 08:10:11.796154582 (48h6m10.347625043s) [noise=true , address=77.90.31.18:4806 , piece_id=LNL5LRHH3OOELKEO4HND2K3GNVDPCBCGKBNJ75DCLEKRAOMTQLKA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.809043214 (48h6m10.360513675s) [noise=true , address=45.91.116.8:4047 , piece_id=VP6VW2ZTNSKS35R47GGJZI4YSHNNFUFB4YQ3W3QFT5ENN7F73VQA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.866335992 (48h6m10.417806453s) [noise=true , address=45.91.116.11:4117 , piece_id=UOEFPOPOFIO5KXY2JOX54YHUWNMTCAFMXEBKAMMR4GSR5ZE6J6PA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.932922087 (48h6m10.484392538s) [noise=true , address=77.90.31.19:4841 , piece_id=6JVB2WIEA2STMNBVTM6D7IYFHQY4QJGXAYHJAQILMYZZDH2MWIVA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:12.005905333 (48h6m10.557375794s) [noise=true , address=45.91.116.4:4939 , piece_id=KMHEQSQTA2WEWMIO7JDQZUDLSRV4CA6OJUN2CYYUSA7SU2ZDQBCQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:16.703757864 (48h6m15.255228326s) [noise=true , address=145.79.227.9:4366 , piece_id=JDBR4PTBZ5RN6NTWPERVLFANUAZQF3FSGIENVRHSYN56ED5NT2AQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:17.540298352 (48h6m16.091768813s) [noise=true , address=145.79.227.21:4474 , piece_id=RNB5T6DIUCY7IV2ULLO2UJRDKUJXI34UM4ZCHPJB22DNQ47VZJVA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:19.805526866 (48h6m18.356997337s) [noise=true , address=145.79.227.27:4689 , piece_id=KLTJYVDWHEE2BGLUSP2XT6L5R2C5OB7TF4DZLYBF6R5OG3N6AEFQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:24.397542313 (48h6m22.949012764s) [noise=true , address=145.79.227.17:4970 , piece_id=F2227LWTX5PRCQIUEJNRJ3QS3GRJOIEJIZFV3KRF7BAEAPSQZR7A, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:24.779686050 (48h6m23.331156511s) [noise=true , address=145.79.227.14:4840 , piece_id=QASWXZB3E6E7VYGF24KZ37Q3LKT6F4WVFKTBV3YS73KYAWTBV7YQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:25.298168925 (48h6m23.849639387s) [noise=true , address=145.79.227.11:4729 , piece_id=CDIU7FFPCVQGDK3K5ZDUARJ6VGB7KIY2DEVL7ZRLW5EASIFEHKHA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:35.044925402 (48h6m33.596395853s) [noise=true , address=145.79.227.30:4761 , piece_id=ASO5O5LM6CEF6DXGBVOCI3NOCWLGKDU2T5JHOL5ZPWEXMTRMKZIA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:37.306959937 (48h6m35.858430398s) [noise=true , address=145.79.227.20:4437 , piece_id=AOK6MTRZKPCXTNHZWGBYJGRBYNHFWO7GSG2PT4DFSMDAWK75GY4A, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:37.306987550 (48h6m35.858458011s) [seg_pos={0 0}, upload=42408]: Segment reached optimal threshold of 13 pieces. Nov 11 08:10:37.307209231 (48h6m35.858679702s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4): protocol: expected piece hash; context canceled; EOF Nov 11 08:10:37.307218759 (48h6m35.85868922s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307368936 (48h6m35.858839397s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw): protocol: expected piece hash; context canceled; EOF Nov 11 08:10:37.307376741 (48h6m35.858847202s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307561282 (48h6m35.859031743s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9): protocol: write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection Nov 11 08:10:37.307585969 (48h6m35.85905643s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307629221 (48h6m35.859099682s) [seg_pos={0 0}, upload=42408]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 11 08:10:37.307741005 (48h6m35.859211466s) [upload=42408]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 11 08:10:37.307750052 (48h6m35.859220513s) [upload=42408]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 11 08:10:37.612339386 (48h6m36.163809848s) [upload=42408]: Done uploading segments... Nov 11 08:10:37.612366317 (48h6m36.163836788s) [upload=42408]: Upload finished. err: <nil> Duration: 5.863023618s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 5.911526257s trace: https://jaeger.dev.storj.io/trace/49621e5e6beafe8e FAIL (2025-11-11 08:33:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 11 08:10:09.433180409 (48h6m7.98465087s) [upload=42408]: Starting upload Nov 11 08:10:09.433429884 (48h6m7.984900335s) [upload=42408]: Done starting upload Nov 11 08:10:09.433567967 (48h6m7.985038438s) [upload=42408]: Uploading segments... Nov 11 08:10:09.434191213 (48h6m7.985661674s) [upload=42408]: Got next segment. Inline: false Nov 11 08:10:09.434274032 (48h6m7.985744493s) [upload=42408]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 08:10:09.434282217 (48h6m7.985752688s) [upload=42408]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 08:10:09.714872160 (48h6m8.266342621s) [seg_pos={0 0}, upload=42408]: Begin upload segment... Nov 11 08:10:09.715084585 (48h6m8.266555046s) [seg_pos={0 0}, upload=42408]: Done begin upload segment. Nov 11 08:10:09.715464638 (48h6m8.2669351s) [noise=true , address=45.91.116.8:4047 , piece_id=VP6VW2ZTNSKS35R47GGJZI4YSHNNFUFB4YQ3W3QFT5ENN7F73VQA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.715555221 (48h6m8.267025692s) [noise=true , address=145.79.227.30:4761 , piece_id=ASO5O5LM6CEF6DXGBVOCI3NOCWLGKDU2T5JHOL5ZPWEXMTRMKZIA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.718085410 (48h6m8.269555901s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.718721901 (48h6m8.270192402s) [noise=true , address=145.79.227.11:4729 , piece_id=CDIU7FFPCVQGDK3K5ZDUARJ6VGB7KIY2DEVL7ZRLW5EASIFEHKHA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.719505724 (48h6m8.270976195s) [noise=true , address=145.79.227.27:4689 , piece_id=KLTJYVDWHEE2BGLUSP2XT6L5R2C5OB7TF4DZLYBF6R5OG3N6AEFQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.720136295 (48h6m8.271606756s) [noise=true , address=145.79.227.14:4840 , piece_id=QASWXZB3E6E7VYGF24KZ37Q3LKT6F4WVFKTBV3YS73KYAWTBV7YQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.721166056 (48h6m8.272636517s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.721883453 (48h6m8.273353994s) [noise=true , address=45.91.116.4:4939 , piece_id=KMHEQSQTA2WEWMIO7JDQZUDLSRV4CA6OJUN2CYYUSA7SU2ZDQBCQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.722398593 (48h6m8.273869054s) [noise=true , address=77.90.31.18:4806 , piece_id=LNL5LRHH3OOELKEO4HND2K3GNVDPCBCGKBNJ75DCLEKRAOMTQLKA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.724101306 (48h6m8.275571767s) [noise=true , address=145.79.227.17:4970 , piece_id=F2227LWTX5PRCQIUEJNRJ3QS3GRJOIEJIZFV3KRF7BAEAPSQZR7A, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.726385264 (48h6m8.277855776s) [noise=true , address=45.91.116.11:4117 , piece_id=UOEFPOPOFIO5KXY2JOX54YHUWNMTCAFMXEBKAMMR4GSR5ZE6J6PA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.728323225 (48h6m8.279793686s) [noise=true , address=145.79.227.20:4437 , piece_id=AOK6MTRZKPCXTNHZWGBYJGRBYNHFWO7GSG2PT4DFSMDAWK75GY4A, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.730321541 (48h6m8.281792002s) [noise=true , address=145.79.227.9:4366 , piece_id=JDBR4PTBZ5RN6NTWPERVLFANUAZQF3FSGIENVRHSYN56ED5NT2AQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.731324300 (48h6m8.282794832s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.732226989 (48h6m8.28369749s) [noise=true , address=77.90.31.19:4841 , piece_id=6JVB2WIEA2STMNBVTM6D7IYFHQY4QJGXAYHJAQILMYZZDH2MWIVA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.733026101 (48h6m8.284496562s) [noise=true , address=145.79.227.21:4474 , piece_id=RNB5T6DIUCY7IV2ULLO2UJRDKUJXI34UM4ZCHPJB22DNQ47VZJVA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:11.398415378 (48h6m9.949885839s) [upload=42408]: Next returned nil segment Nov 11 08:10:11.398434043 (48h6m9.949904504s) [upload=42408]: Waiting for error group managing segments... Nov 11 08:10:11.796154582 (48h6m10.347625043s) [noise=true , address=77.90.31.18:4806 , piece_id=LNL5LRHH3OOELKEO4HND2K3GNVDPCBCGKBNJ75DCLEKRAOMTQLKA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.809043214 (48h6m10.360513675s) [noise=true , address=45.91.116.8:4047 , piece_id=VP6VW2ZTNSKS35R47GGJZI4YSHNNFUFB4YQ3W3QFT5ENN7F73VQA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.866335992 (48h6m10.417806453s) [noise=true , address=45.91.116.11:4117 , piece_id=UOEFPOPOFIO5KXY2JOX54YHUWNMTCAFMXEBKAMMR4GSR5ZE6J6PA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.932922087 (48h6m10.484392538s) [noise=true , address=77.90.31.19:4841 , piece_id=6JVB2WIEA2STMNBVTM6D7IYFHQY4QJGXAYHJAQILMYZZDH2MWIVA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:12.005905333 (48h6m10.557375794s) [noise=true , address=45.91.116.4:4939 , piece_id=KMHEQSQTA2WEWMIO7JDQZUDLSRV4CA6OJUN2CYYUSA7SU2ZDQBCQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:16.703757864 (48h6m15.255228326s) [noise=true , address=145.79.227.9:4366 , piece_id=JDBR4PTBZ5RN6NTWPERVLFANUAZQF3FSGIENVRHSYN56ED5NT2AQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:17.540298352 (48h6m16.091768813s) [noise=true , address=145.79.227.21:4474 , piece_id=RNB5T6DIUCY7IV2ULLO2UJRDKUJXI34UM4ZCHPJB22DNQ47VZJVA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:19.805526866 (48h6m18.356997337s) [noise=true , address=145.79.227.27:4689 , piece_id=KLTJYVDWHEE2BGLUSP2XT6L5R2C5OB7TF4DZLYBF6R5OG3N6AEFQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:24.397542313 (48h6m22.949012764s) [noise=true , address=145.79.227.17:4970 , piece_id=F2227LWTX5PRCQIUEJNRJ3QS3GRJOIEJIZFV3KRF7BAEAPSQZR7A, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:24.779686050 (48h6m23.331156511s) [noise=true , address=145.79.227.14:4840 , piece_id=QASWXZB3E6E7VYGF24KZ37Q3LKT6F4WVFKTBV3YS73KYAWTBV7YQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:25.298168925 (48h6m23.849639387s) [noise=true , address=145.79.227.11:4729 , piece_id=CDIU7FFPCVQGDK3K5ZDUARJ6VGB7KIY2DEVL7ZRLW5EASIFEHKHA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:35.044925402 (48h6m33.596395853s) [noise=true , address=145.79.227.30:4761 , piece_id=ASO5O5LM6CEF6DXGBVOCI3NOCWLGKDU2T5JHOL5ZPWEXMTRMKZIA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:37.306959937 (48h6m35.858430398s) [noise=true , address=145.79.227.20:4437 , piece_id=AOK6MTRZKPCXTNHZWGBYJGRBYNHFWO7GSG2PT4DFSMDAWK75GY4A, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:37.306987550 (48h6m35.858458011s) [seg_pos={0 0}, upload=42408]: Segment reached optimal threshold of 13 pieces. Nov 11 08:10:37.307209231 (48h6m35.858679702s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4): protocol: expected piece hash; context canceled; EOF Nov 11 08:10:37.307218759 (48h6m35.85868922s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307368936 (48h6m35.858839397s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw): protocol: expected piece hash; context canceled; EOF Nov 11 08:10:37.307376741 (48h6m35.858847202s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307561282 (48h6m35.859031743s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9): protocol: write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection Nov 11 08:10:37.307585969 (48h6m35.85905643s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307629221 (48h6m35.859099682s) [seg_pos={0 0}, upload=42408]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 11 08:10:37.307741005 (48h6m35.859211466s) [upload=42408]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 11 08:10:37.307750052 (48h6m35.859220513s) [upload=42408]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 11 08:10:37.612339386 (48h6m36.163809848s) [upload=42408]: Done uploading segments... Nov 11 08:10:37.612366317 (48h6m36.163836788s) [upload=42408]: Upload finished. err: <nil> Duration: 5.911555311s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 5.923878835s trace: https://jaeger.dev.storj.io/trace/49621e5e6beafe8e FAIL (2025-11-11 08:32:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 11 08:10:09.433180409 (48h6m7.98465087s) [upload=42408]: Starting upload Nov 11 08:10:09.433429884 (48h6m7.984900335s) [upload=42408]: Done starting upload Nov 11 08:10:09.433567967 (48h6m7.985038438s) [upload=42408]: Uploading segments... Nov 11 08:10:09.434191213 (48h6m7.985661674s) [upload=42408]: Got next segment. Inline: false Nov 11 08:10:09.434274032 (48h6m7.985744493s) [upload=42408]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 08:10:09.434282217 (48h6m7.985752688s) [upload=42408]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 08:10:09.714872160 (48h6m8.266342621s) [seg_pos={0 0}, upload=42408]: Begin upload segment... Nov 11 08:10:09.715084585 (48h6m8.266555046s) [seg_pos={0 0}, upload=42408]: Done begin upload segment. Nov 11 08:10:09.715464638 (48h6m8.2669351s) [noise=true , address=45.91.116.8:4047 , piece_id=VP6VW2ZTNSKS35R47GGJZI4YSHNNFUFB4YQ3W3QFT5ENN7F73VQA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.715555221 (48h6m8.267025692s) [noise=true , address=145.79.227.30:4761 , piece_id=ASO5O5LM6CEF6DXGBVOCI3NOCWLGKDU2T5JHOL5ZPWEXMTRMKZIA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.718085410 (48h6m8.269555901s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.718721901 (48h6m8.270192402s) [noise=true , address=145.79.227.11:4729 , piece_id=CDIU7FFPCVQGDK3K5ZDUARJ6VGB7KIY2DEVL7ZRLW5EASIFEHKHA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.719505724 (48h6m8.270976195s) [noise=true , address=145.79.227.27:4689 , piece_id=KLTJYVDWHEE2BGLUSP2XT6L5R2C5OB7TF4DZLYBF6R5OG3N6AEFQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.720136295 (48h6m8.271606756s) [noise=true , address=145.79.227.14:4840 , piece_id=QASWXZB3E6E7VYGF24KZ37Q3LKT6F4WVFKTBV3YS73KYAWTBV7YQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.721166056 (48h6m8.272636517s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.721883453 (48h6m8.273353994s) [noise=true , address=45.91.116.4:4939 , piece_id=KMHEQSQTA2WEWMIO7JDQZUDLSRV4CA6OJUN2CYYUSA7SU2ZDQBCQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.722398593 (48h6m8.273869054s) [noise=true , address=77.90.31.18:4806 , piece_id=LNL5LRHH3OOELKEO4HND2K3GNVDPCBCGKBNJ75DCLEKRAOMTQLKA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.724101306 (48h6m8.275571767s) [noise=true , address=145.79.227.17:4970 , piece_id=F2227LWTX5PRCQIUEJNRJ3QS3GRJOIEJIZFV3KRF7BAEAPSQZR7A, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.726385264 (48h6m8.277855776s) [noise=true , address=45.91.116.11:4117 , piece_id=UOEFPOPOFIO5KXY2JOX54YHUWNMTCAFMXEBKAMMR4GSR5ZE6J6PA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.728323225 (48h6m8.279793686s) [noise=true , address=145.79.227.20:4437 , piece_id=AOK6MTRZKPCXTNHZWGBYJGRBYNHFWO7GSG2PT4DFSMDAWK75GY4A, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.730321541 (48h6m8.281792002s) [noise=true , address=145.79.227.9:4366 , piece_id=JDBR4PTBZ5RN6NTWPERVLFANUAZQF3FSGIENVRHSYN56ED5NT2AQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.731324300 (48h6m8.282794832s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.732226989 (48h6m8.28369749s) [noise=true , address=77.90.31.19:4841 , piece_id=6JVB2WIEA2STMNBVTM6D7IYFHQY4QJGXAYHJAQILMYZZDH2MWIVA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.733026101 (48h6m8.284496562s) [noise=true , address=145.79.227.21:4474 , piece_id=RNB5T6DIUCY7IV2ULLO2UJRDKUJXI34UM4ZCHPJB22DNQ47VZJVA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:11.398415378 (48h6m9.949885839s) [upload=42408]: Next returned nil segment Nov 11 08:10:11.398434043 (48h6m9.949904504s) [upload=42408]: Waiting for error group managing segments... Nov 11 08:10:11.796154582 (48h6m10.347625043s) [noise=true , address=77.90.31.18:4806 , piece_id=LNL5LRHH3OOELKEO4HND2K3GNVDPCBCGKBNJ75DCLEKRAOMTQLKA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.809043214 (48h6m10.360513675s) [noise=true , address=45.91.116.8:4047 , piece_id=VP6VW2ZTNSKS35R47GGJZI4YSHNNFUFB4YQ3W3QFT5ENN7F73VQA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.866335992 (48h6m10.417806453s) [noise=true , address=45.91.116.11:4117 , piece_id=UOEFPOPOFIO5KXY2JOX54YHUWNMTCAFMXEBKAMMR4GSR5ZE6J6PA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.932922087 (48h6m10.484392538s) [noise=true , address=77.90.31.19:4841 , piece_id=6JVB2WIEA2STMNBVTM6D7IYFHQY4QJGXAYHJAQILMYZZDH2MWIVA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:12.005905333 (48h6m10.557375794s) [noise=true , address=45.91.116.4:4939 , piece_id=KMHEQSQTA2WEWMIO7JDQZUDLSRV4CA6OJUN2CYYUSA7SU2ZDQBCQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:16.703757864 (48h6m15.255228326s) [noise=true , address=145.79.227.9:4366 , piece_id=JDBR4PTBZ5RN6NTWPERVLFANUAZQF3FSGIENVRHSYN56ED5NT2AQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:17.540298352 (48h6m16.091768813s) [noise=true , address=145.79.227.21:4474 , piece_id=RNB5T6DIUCY7IV2ULLO2UJRDKUJXI34UM4ZCHPJB22DNQ47VZJVA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:19.805526866 (48h6m18.356997337s) [noise=true , address=145.79.227.27:4689 , piece_id=KLTJYVDWHEE2BGLUSP2XT6L5R2C5OB7TF4DZLYBF6R5OG3N6AEFQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:24.397542313 (48h6m22.949012764s) [noise=true , address=145.79.227.17:4970 , piece_id=F2227LWTX5PRCQIUEJNRJ3QS3GRJOIEJIZFV3KRF7BAEAPSQZR7A, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:24.779686050 (48h6m23.331156511s) [noise=true , address=145.79.227.14:4840 , piece_id=QASWXZB3E6E7VYGF24KZ37Q3LKT6F4WVFKTBV3YS73KYAWTBV7YQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:25.298168925 (48h6m23.849639387s) [noise=true , address=145.79.227.11:4729 , piece_id=CDIU7FFPCVQGDK3K5ZDUARJ6VGB7KIY2DEVL7ZRLW5EASIFEHKHA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:35.044925402 (48h6m33.596395853s) [noise=true , address=145.79.227.30:4761 , piece_id=ASO5O5LM6CEF6DXGBVOCI3NOCWLGKDU2T5JHOL5ZPWEXMTRMKZIA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:37.306959937 (48h6m35.858430398s) [noise=true , address=145.79.227.20:4437 , piece_id=AOK6MTRZKPCXTNHZWGBYJGRBYNHFWO7GSG2PT4DFSMDAWK75GY4A, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:37.306987550 (48h6m35.858458011s) [seg_pos={0 0}, upload=42408]: Segment reached optimal threshold of 13 pieces. Nov 11 08:10:37.307209231 (48h6m35.858679702s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4): protocol: expected piece hash; context canceled; EOF Nov 11 08:10:37.307218759 (48h6m35.85868922s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307368936 (48h6m35.858839397s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw): protocol: expected piece hash; context canceled; EOF Nov 11 08:10:37.307376741 (48h6m35.858847202s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307561282 (48h6m35.859031743s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9): protocol: write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection Nov 11 08:10:37.307585969 (48h6m35.85905643s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307629221 (48h6m35.859099682s) [seg_pos={0 0}, upload=42408]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 11 08:10:37.307741005 (48h6m35.859211466s) [upload=42408]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 11 08:10:37.307750052 (48h6m35.859220513s) [upload=42408]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 11 08:10:37.612339386 (48h6m36.163809848s) [upload=42408]: Done uploading segments... Nov 11 08:10:37.612366317 (48h6m36.163836788s) [upload=42408]: Upload finished. err: <nil> Duration: 5.923912961s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 5.82517799s trace: https://jaeger.dev.storj.io/trace/49621e5e6beafe8e FAIL (2025-11-11 08:31:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 11 08:10:09.433180409 (48h6m7.98465087s) [upload=42408]: Starting upload Nov 11 08:10:09.433429884 (48h6m7.984900335s) [upload=42408]: Done starting upload Nov 11 08:10:09.433567967 (48h6m7.985038438s) [upload=42408]: Uploading segments... Nov 11 08:10:09.434191213 (48h6m7.985661674s) [upload=42408]: Got next segment. Inline: false Nov 11 08:10:09.434274032 (48h6m7.985744493s) [upload=42408]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 08:10:09.434282217 (48h6m7.985752688s) [upload=42408]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 08:10:09.714872160 (48h6m8.266342621s) [seg_pos={0 0}, upload=42408]: Begin upload segment... Nov 11 08:10:09.715084585 (48h6m8.266555046s) [seg_pos={0 0}, upload=42408]: Done begin upload segment. Nov 11 08:10:09.715464638 (48h6m8.2669351s) [noise=true , address=45.91.116.8:4047 , piece_id=VP6VW2ZTNSKS35R47GGJZI4YSHNNFUFB4YQ3W3QFT5ENN7F73VQA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.715555221 (48h6m8.267025692s) [noise=true , address=145.79.227.30:4761 , piece_id=ASO5O5LM6CEF6DXGBVOCI3NOCWLGKDU2T5JHOL5ZPWEXMTRMKZIA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.718085410 (48h6m8.269555901s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.718721901 (48h6m8.270192402s) [noise=true , address=145.79.227.11:4729 , piece_id=CDIU7FFPCVQGDK3K5ZDUARJ6VGB7KIY2DEVL7ZRLW5EASIFEHKHA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.719505724 (48h6m8.270976195s) [noise=true , address=145.79.227.27:4689 , piece_id=KLTJYVDWHEE2BGLUSP2XT6L5R2C5OB7TF4DZLYBF6R5OG3N6AEFQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.720136295 (48h6m8.271606756s) [noise=true , address=145.79.227.14:4840 , piece_id=QASWXZB3E6E7VYGF24KZ37Q3LKT6F4WVFKTBV3YS73KYAWTBV7YQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.721166056 (48h6m8.272636517s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.721883453 (48h6m8.273353994s) [noise=true , address=45.91.116.4:4939 , piece_id=KMHEQSQTA2WEWMIO7JDQZUDLSRV4CA6OJUN2CYYUSA7SU2ZDQBCQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.722398593 (48h6m8.273869054s) [noise=true , address=77.90.31.18:4806 , piece_id=LNL5LRHH3OOELKEO4HND2K3GNVDPCBCGKBNJ75DCLEKRAOMTQLKA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.724101306 (48h6m8.275571767s) [noise=true , address=145.79.227.17:4970 , piece_id=F2227LWTX5PRCQIUEJNRJ3QS3GRJOIEJIZFV3KRF7BAEAPSQZR7A, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.726385264 (48h6m8.277855776s) [noise=true , address=45.91.116.11:4117 , piece_id=UOEFPOPOFIO5KXY2JOX54YHUWNMTCAFMXEBKAMMR4GSR5ZE6J6PA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.728323225 (48h6m8.279793686s) [noise=true , address=145.79.227.20:4437 , piece_id=AOK6MTRZKPCXTNHZWGBYJGRBYNHFWO7GSG2PT4DFSMDAWK75GY4A, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.730321541 (48h6m8.281792002s) [noise=true , address=145.79.227.9:4366 , piece_id=JDBR4PTBZ5RN6NTWPERVLFANUAZQF3FSGIENVRHSYN56ED5NT2AQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.731324300 (48h6m8.282794832s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.732226989 (48h6m8.28369749s) [noise=true , address=77.90.31.19:4841 , piece_id=6JVB2WIEA2STMNBVTM6D7IYFHQY4QJGXAYHJAQILMYZZDH2MWIVA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.733026101 (48h6m8.284496562s) [noise=true , address=145.79.227.21:4474 , piece_id=RNB5T6DIUCY7IV2ULLO2UJRDKUJXI34UM4ZCHPJB22DNQ47VZJVA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:11.398415378 (48h6m9.949885839s) [upload=42408]: Next returned nil segment Nov 11 08:10:11.398434043 (48h6m9.949904504s) [upload=42408]: Waiting for error group managing segments... Nov 11 08:10:11.796154582 (48h6m10.347625043s) [noise=true , address=77.90.31.18:4806 , piece_id=LNL5LRHH3OOELKEO4HND2K3GNVDPCBCGKBNJ75DCLEKRAOMTQLKA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.809043214 (48h6m10.360513675s) [noise=true , address=45.91.116.8:4047 , piece_id=VP6VW2ZTNSKS35R47GGJZI4YSHNNFUFB4YQ3W3QFT5ENN7F73VQA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.866335992 (48h6m10.417806453s) [noise=true , address=45.91.116.11:4117 , piece_id=UOEFPOPOFIO5KXY2JOX54YHUWNMTCAFMXEBKAMMR4GSR5ZE6J6PA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.932922087 (48h6m10.484392538s) [noise=true , address=77.90.31.19:4841 , piece_id=6JVB2WIEA2STMNBVTM6D7IYFHQY4QJGXAYHJAQILMYZZDH2MWIVA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:12.005905333 (48h6m10.557375794s) [noise=true , address=45.91.116.4:4939 , piece_id=KMHEQSQTA2WEWMIO7JDQZUDLSRV4CA6OJUN2CYYUSA7SU2ZDQBCQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:16.703757864 (48h6m15.255228326s) [noise=true , address=145.79.227.9:4366 , piece_id=JDBR4PTBZ5RN6NTWPERVLFANUAZQF3FSGIENVRHSYN56ED5NT2AQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:17.540298352 (48h6m16.091768813s) [noise=true , address=145.79.227.21:4474 , piece_id=RNB5T6DIUCY7IV2ULLO2UJRDKUJXI34UM4ZCHPJB22DNQ47VZJVA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:19.805526866 (48h6m18.356997337s) [noise=true , address=145.79.227.27:4689 , piece_id=KLTJYVDWHEE2BGLUSP2XT6L5R2C5OB7TF4DZLYBF6R5OG3N6AEFQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:24.397542313 (48h6m22.949012764s) [noise=true , address=145.79.227.17:4970 , piece_id=F2227LWTX5PRCQIUEJNRJ3QS3GRJOIEJIZFV3KRF7BAEAPSQZR7A, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:24.779686050 (48h6m23.331156511s) [noise=true , address=145.79.227.14:4840 , piece_id=QASWXZB3E6E7VYGF24KZ37Q3LKT6F4WVFKTBV3YS73KYAWTBV7YQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:25.298168925 (48h6m23.849639387s) [noise=true , address=145.79.227.11:4729 , piece_id=CDIU7FFPCVQGDK3K5ZDUARJ6VGB7KIY2DEVL7ZRLW5EASIFEHKHA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:35.044925402 (48h6m33.596395853s) [noise=true , address=145.79.227.30:4761 , piece_id=ASO5O5LM6CEF6DXGBVOCI3NOCWLGKDU2T5JHOL5ZPWEXMTRMKZIA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:37.306959937 (48h6m35.858430398s) [noise=true , address=145.79.227.20:4437 , piece_id=AOK6MTRZKPCXTNHZWGBYJGRBYNHFWO7GSG2PT4DFSMDAWK75GY4A, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:37.306987550 (48h6m35.858458011s) [seg_pos={0 0}, upload=42408]: Segment reached optimal threshold of 13 pieces. Nov 11 08:10:37.307209231 (48h6m35.858679702s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4): protocol: expected piece hash; context canceled; EOF Nov 11 08:10:37.307218759 (48h6m35.85868922s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307368936 (48h6m35.858839397s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw): protocol: expected piece hash; context canceled; EOF Nov 11 08:10:37.307376741 (48h6m35.858847202s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307561282 (48h6m35.859031743s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9): protocol: write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection Nov 11 08:10:37.307585969 (48h6m35.85905643s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307629221 (48h6m35.859099682s) [seg_pos={0 0}, upload=42408]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 11 08:10:37.307741005 (48h6m35.859211466s) [upload=42408]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 11 08:10:37.307750052 (48h6m35.859220513s) [upload=42408]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 11 08:10:37.612339386 (48h6m36.163809848s) [upload=42408]: Done uploading segments... Nov 11 08:10:37.612366317 (48h6m36.163836788s) [upload=42408]: Upload finished. err: <nil> Duration: 5.825207716s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [X..............] duration: 68.566917ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/77081cc731df4257 FAIL (2025-11-11 08:31:17 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:50034->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 68.597284ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 6.031649025s trace: https://jaeger.dev.storj.io/trace/49621e5e6beafe8e FAIL (2025-11-11 08:31:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 11 08:10:09.433180409 (48h6m7.98465087s) [upload=42408]: Starting upload Nov 11 08:10:09.433429884 (48h6m7.984900335s) [upload=42408]: Done starting upload Nov 11 08:10:09.433567967 (48h6m7.985038438s) [upload=42408]: Uploading segments... Nov 11 08:10:09.434191213 (48h6m7.985661674s) [upload=42408]: Got next segment. Inline: false Nov 11 08:10:09.434274032 (48h6m7.985744493s) [upload=42408]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 08:10:09.434282217 (48h6m7.985752688s) [upload=42408]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 08:10:09.714872160 (48h6m8.266342621s) [seg_pos={0 0}, upload=42408]: Begin upload segment... Nov 11 08:10:09.715084585 (48h6m8.266555046s) [seg_pos={0 0}, upload=42408]: Done begin upload segment. Nov 11 08:10:09.715464638 (48h6m8.2669351s) [noise=true , address=45.91.116.8:4047 , piece_id=VP6VW2ZTNSKS35R47GGJZI4YSHNNFUFB4YQ3W3QFT5ENN7F73VQA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.715555221 (48h6m8.267025692s) [noise=true , address=145.79.227.30:4761 , piece_id=ASO5O5LM6CEF6DXGBVOCI3NOCWLGKDU2T5JHOL5ZPWEXMTRMKZIA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.718085410 (48h6m8.269555901s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.718721901 (48h6m8.270192402s) [noise=true , address=145.79.227.11:4729 , piece_id=CDIU7FFPCVQGDK3K5ZDUARJ6VGB7KIY2DEVL7ZRLW5EASIFEHKHA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.719505724 (48h6m8.270976195s) [noise=true , address=145.79.227.27:4689 , piece_id=KLTJYVDWHEE2BGLUSP2XT6L5R2C5OB7TF4DZLYBF6R5OG3N6AEFQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.720136295 (48h6m8.271606756s) [noise=true , address=145.79.227.14:4840 , piece_id=QASWXZB3E6E7VYGF24KZ37Q3LKT6F4WVFKTBV3YS73KYAWTBV7YQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.721166056 (48h6m8.272636517s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.721883453 (48h6m8.273353994s) [noise=true , address=45.91.116.4:4939 , piece_id=KMHEQSQTA2WEWMIO7JDQZUDLSRV4CA6OJUN2CYYUSA7SU2ZDQBCQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.722398593 (48h6m8.273869054s) [noise=true , address=77.90.31.18:4806 , piece_id=LNL5LRHH3OOELKEO4HND2K3GNVDPCBCGKBNJ75DCLEKRAOMTQLKA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.724101306 (48h6m8.275571767s) [noise=true , address=145.79.227.17:4970 , piece_id=F2227LWTX5PRCQIUEJNRJ3QS3GRJOIEJIZFV3KRF7BAEAPSQZR7A, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.726385264 (48h6m8.277855776s) [noise=true , address=45.91.116.11:4117 , piece_id=UOEFPOPOFIO5KXY2JOX54YHUWNMTCAFMXEBKAMMR4GSR5ZE6J6PA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.728323225 (48h6m8.279793686s) [noise=true , address=145.79.227.20:4437 , piece_id=AOK6MTRZKPCXTNHZWGBYJGRBYNHFWO7GSG2PT4DFSMDAWK75GY4A, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.730321541 (48h6m8.281792002s) [noise=true , address=145.79.227.9:4366 , piece_id=JDBR4PTBZ5RN6NTWPERVLFANUAZQF3FSGIENVRHSYN56ED5NT2AQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.731324300 (48h6m8.282794832s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.732226989 (48h6m8.28369749s) [noise=true , address=77.90.31.19:4841 , piece_id=6JVB2WIEA2STMNBVTM6D7IYFHQY4QJGXAYHJAQILMYZZDH2MWIVA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.733026101 (48h6m8.284496562s) [noise=true , address=145.79.227.21:4474 , piece_id=RNB5T6DIUCY7IV2ULLO2UJRDKUJXI34UM4ZCHPJB22DNQ47VZJVA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:11.398415378 (48h6m9.949885839s) [upload=42408]: Next returned nil segment Nov 11 08:10:11.398434043 (48h6m9.949904504s) [upload=42408]: Waiting for error group managing segments... Nov 11 08:10:11.796154582 (48h6m10.347625043s) [noise=true , address=77.90.31.18:4806 , piece_id=LNL5LRHH3OOELKEO4HND2K3GNVDPCBCGKBNJ75DCLEKRAOMTQLKA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.809043214 (48h6m10.360513675s) [noise=true , address=45.91.116.8:4047 , piece_id=VP6VW2ZTNSKS35R47GGJZI4YSHNNFUFB4YQ3W3QFT5ENN7F73VQA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.866335992 (48h6m10.417806453s) [noise=true , address=45.91.116.11:4117 , piece_id=UOEFPOPOFIO5KXY2JOX54YHUWNMTCAFMXEBKAMMR4GSR5ZE6J6PA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.932922087 (48h6m10.484392538s) [noise=true , address=77.90.31.19:4841 , piece_id=6JVB2WIEA2STMNBVTM6D7IYFHQY4QJGXAYHJAQILMYZZDH2MWIVA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:12.005905333 (48h6m10.557375794s) [noise=true , address=45.91.116.4:4939 , piece_id=KMHEQSQTA2WEWMIO7JDQZUDLSRV4CA6OJUN2CYYUSA7SU2ZDQBCQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:16.703757864 (48h6m15.255228326s) [noise=true , address=145.79.227.9:4366 , piece_id=JDBR4PTBZ5RN6NTWPERVLFANUAZQF3FSGIENVRHSYN56ED5NT2AQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:17.540298352 (48h6m16.091768813s) [noise=true , address=145.79.227.21:4474 , piece_id=RNB5T6DIUCY7IV2ULLO2UJRDKUJXI34UM4ZCHPJB22DNQ47VZJVA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:19.805526866 (48h6m18.356997337s) [noise=true , address=145.79.227.27:4689 , piece_id=KLTJYVDWHEE2BGLUSP2XT6L5R2C5OB7TF4DZLYBF6R5OG3N6AEFQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:24.397542313 (48h6m22.949012764s) [noise=true , address=145.79.227.17:4970 , piece_id=F2227LWTX5PRCQIUEJNRJ3QS3GRJOIEJIZFV3KRF7BAEAPSQZR7A, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:24.779686050 (48h6m23.331156511s) [noise=true , address=145.79.227.14:4840 , piece_id=QASWXZB3E6E7VYGF24KZ37Q3LKT6F4WVFKTBV3YS73KYAWTBV7YQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:25.298168925 (48h6m23.849639387s) [noise=true , address=145.79.227.11:4729 , piece_id=CDIU7FFPCVQGDK3K5ZDUARJ6VGB7KIY2DEVL7ZRLW5EASIFEHKHA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:35.044925402 (48h6m33.596395853s) [noise=true , address=145.79.227.30:4761 , piece_id=ASO5O5LM6CEF6DXGBVOCI3NOCWLGKDU2T5JHOL5ZPWEXMTRMKZIA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:37.306959937 (48h6m35.858430398s) [noise=true , address=145.79.227.20:4437 , piece_id=AOK6MTRZKPCXTNHZWGBYJGRBYNHFWO7GSG2PT4DFSMDAWK75GY4A, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:37.306987550 (48h6m35.858458011s) [seg_pos={0 0}, upload=42408]: Segment reached optimal threshold of 13 pieces. Nov 11 08:10:37.307209231 (48h6m35.858679702s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4): protocol: expected piece hash; context canceled; EOF Nov 11 08:10:37.307218759 (48h6m35.85868922s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307368936 (48h6m35.858839397s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw): protocol: expected piece hash; context canceled; EOF Nov 11 08:10:37.307376741 (48h6m35.858847202s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307561282 (48h6m35.859031743s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9): protocol: write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection Nov 11 08:10:37.307585969 (48h6m35.85905643s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307629221 (48h6m35.859099682s) [seg_pos={0 0}, upload=42408]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 11 08:10:37.307741005 (48h6m35.859211466s) [upload=42408]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 11 08:10:37.307750052 (48h6m35.859220513s) [upload=42408]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 11 08:10:37.612339386 (48h6m36.163809848s) [upload=42408]: Done uploading segments... Nov 11 08:10:37.612366317 (48h6m36.163836788s) [upload=42408]: Upload finished. err: <nil> Duration: 6.031685454s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [.X.............] duration: 349.124027ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/77081cc731df4257 FAIL (2025-11-11 08:30:47 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:50034->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 349.157933ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 5.982239131s trace: https://jaeger.dev.storj.io/trace/49621e5e6beafe8e FAIL (2025-11-11 08:30:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 11 08:10:09.433180409 (48h6m7.98465087s) [upload=42408]: Starting upload Nov 11 08:10:09.433429884 (48h6m7.984900335s) [upload=42408]: Done starting upload Nov 11 08:10:09.433567967 (48h6m7.985038438s) [upload=42408]: Uploading segments... Nov 11 08:10:09.434191213 (48h6m7.985661674s) [upload=42408]: Got next segment. Inline: false Nov 11 08:10:09.434274032 (48h6m7.985744493s) [upload=42408]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 08:10:09.434282217 (48h6m7.985752688s) [upload=42408]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 08:10:09.714872160 (48h6m8.266342621s) [seg_pos={0 0}, upload=42408]: Begin upload segment... Nov 11 08:10:09.715084585 (48h6m8.266555046s) [seg_pos={0 0}, upload=42408]: Done begin upload segment. Nov 11 08:10:09.715464638 (48h6m8.2669351s) [noise=true , address=45.91.116.8:4047 , piece_id=VP6VW2ZTNSKS35R47GGJZI4YSHNNFUFB4YQ3W3QFT5ENN7F73VQA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.715555221 (48h6m8.267025692s) [noise=true , address=145.79.227.30:4761 , piece_id=ASO5O5LM6CEF6DXGBVOCI3NOCWLGKDU2T5JHOL5ZPWEXMTRMKZIA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.718085410 (48h6m8.269555901s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.718721901 (48h6m8.270192402s) [noise=true , address=145.79.227.11:4729 , piece_id=CDIU7FFPCVQGDK3K5ZDUARJ6VGB7KIY2DEVL7ZRLW5EASIFEHKHA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.719505724 (48h6m8.270976195s) [noise=true , address=145.79.227.27:4689 , piece_id=KLTJYVDWHEE2BGLUSP2XT6L5R2C5OB7TF4DZLYBF6R5OG3N6AEFQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.720136295 (48h6m8.271606756s) [noise=true , address=145.79.227.14:4840 , piece_id=QASWXZB3E6E7VYGF24KZ37Q3LKT6F4WVFKTBV3YS73KYAWTBV7YQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.721166056 (48h6m8.272636517s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.721883453 (48h6m8.273353994s) [noise=true , address=45.91.116.4:4939 , piece_id=KMHEQSQTA2WEWMIO7JDQZUDLSRV4CA6OJUN2CYYUSA7SU2ZDQBCQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.722398593 (48h6m8.273869054s) [noise=true , address=77.90.31.18:4806 , piece_id=LNL5LRHH3OOELKEO4HND2K3GNVDPCBCGKBNJ75DCLEKRAOMTQLKA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.724101306 (48h6m8.275571767s) [noise=true , address=145.79.227.17:4970 , piece_id=F2227LWTX5PRCQIUEJNRJ3QS3GRJOIEJIZFV3KRF7BAEAPSQZR7A, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.726385264 (48h6m8.277855776s) [noise=true , address=45.91.116.11:4117 , piece_id=UOEFPOPOFIO5KXY2JOX54YHUWNMTCAFMXEBKAMMR4GSR5ZE6J6PA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.728323225 (48h6m8.279793686s) [noise=true , address=145.79.227.20:4437 , piece_id=AOK6MTRZKPCXTNHZWGBYJGRBYNHFWO7GSG2PT4DFSMDAWK75GY4A, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.730321541 (48h6m8.281792002s) [noise=true , address=145.79.227.9:4366 , piece_id=JDBR4PTBZ5RN6NTWPERVLFANUAZQF3FSGIENVRHSYN56ED5NT2AQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.731324300 (48h6m8.282794832s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.732226989 (48h6m8.28369749s) [noise=true , address=77.90.31.19:4841 , piece_id=6JVB2WIEA2STMNBVTM6D7IYFHQY4QJGXAYHJAQILMYZZDH2MWIVA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.733026101 (48h6m8.284496562s) [noise=true , address=145.79.227.21:4474 , piece_id=RNB5T6DIUCY7IV2ULLO2UJRDKUJXI34UM4ZCHPJB22DNQ47VZJVA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:11.398415378 (48h6m9.949885839s) [upload=42408]: Next returned nil segment Nov 11 08:10:11.398434043 (48h6m9.949904504s) [upload=42408]: Waiting for error group managing segments... Nov 11 08:10:11.796154582 (48h6m10.347625043s) [noise=true , address=77.90.31.18:4806 , piece_id=LNL5LRHH3OOELKEO4HND2K3GNVDPCBCGKBNJ75DCLEKRAOMTQLKA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.809043214 (48h6m10.360513675s) [noise=true , address=45.91.116.8:4047 , piece_id=VP6VW2ZTNSKS35R47GGJZI4YSHNNFUFB4YQ3W3QFT5ENN7F73VQA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.866335992 (48h6m10.417806453s) [noise=true , address=45.91.116.11:4117 , piece_id=UOEFPOPOFIO5KXY2JOX54YHUWNMTCAFMXEBKAMMR4GSR5ZE6J6PA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.932922087 (48h6m10.484392538s) [noise=true , address=77.90.31.19:4841 , piece_id=6JVB2WIEA2STMNBVTM6D7IYFHQY4QJGXAYHJAQILMYZZDH2MWIVA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:12.005905333 (48h6m10.557375794s) [noise=true , address=45.91.116.4:4939 , piece_id=KMHEQSQTA2WEWMIO7JDQZUDLSRV4CA6OJUN2CYYUSA7SU2ZDQBCQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:16.703757864 (48h6m15.255228326s) [noise=true , address=145.79.227.9:4366 , piece_id=JDBR4PTBZ5RN6NTWPERVLFANUAZQF3FSGIENVRHSYN56ED5NT2AQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:17.540298352 (48h6m16.091768813s) [noise=true , address=145.79.227.21:4474 , piece_id=RNB5T6DIUCY7IV2ULLO2UJRDKUJXI34UM4ZCHPJB22DNQ47VZJVA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:19.805526866 (48h6m18.356997337s) [noise=true , address=145.79.227.27:4689 , piece_id=KLTJYVDWHEE2BGLUSP2XT6L5R2C5OB7TF4DZLYBF6R5OG3N6AEFQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:24.397542313 (48h6m22.949012764s) [noise=true , address=145.79.227.17:4970 , piece_id=F2227LWTX5PRCQIUEJNRJ3QS3GRJOIEJIZFV3KRF7BAEAPSQZR7A, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:24.779686050 (48h6m23.331156511s) [noise=true , address=145.79.227.14:4840 , piece_id=QASWXZB3E6E7VYGF24KZ37Q3LKT6F4WVFKTBV3YS73KYAWTBV7YQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:25.298168925 (48h6m23.849639387s) [noise=true , address=145.79.227.11:4729 , piece_id=CDIU7FFPCVQGDK3K5ZDUARJ6VGB7KIY2DEVL7ZRLW5EASIFEHKHA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:35.044925402 (48h6m33.596395853s) [noise=true , address=145.79.227.30:4761 , piece_id=ASO5O5LM6CEF6DXGBVOCI3NOCWLGKDU2T5JHOL5ZPWEXMTRMKZIA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:37.306959937 (48h6m35.858430398s) [noise=true , address=145.79.227.20:4437 , piece_id=AOK6MTRZKPCXTNHZWGBYJGRBYNHFWO7GSG2PT4DFSMDAWK75GY4A, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:37.306987550 (48h6m35.858458011s) [seg_pos={0 0}, upload=42408]: Segment reached optimal threshold of 13 pieces. Nov 11 08:10:37.307209231 (48h6m35.858679702s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4): protocol: expected piece hash; context canceled; EOF Nov 11 08:10:37.307218759 (48h6m35.85868922s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307368936 (48h6m35.858839397s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw): protocol: expected piece hash; context canceled; EOF Nov 11 08:10:37.307376741 (48h6m35.858847202s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307561282 (48h6m35.859031743s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9): protocol: write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection Nov 11 08:10:37.307585969 (48h6m35.85905643s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307629221 (48h6m35.859099682s) [seg_pos={0 0}, upload=42408]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 11 08:10:37.307741005 (48h6m35.859211466s) [upload=42408]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 11 08:10:37.307750052 (48h6m35.859220513s) [upload=42408]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 11 08:10:37.612339386 (48h6m36.163809848s) [upload=42408]: Done uploading segments... Nov 11 08:10:37.612366317 (48h6m36.163836788s) [upload=42408]: Upload finished. err: <nil> Duration: 5.982266393s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [..X............] duration: 88.240532ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/77081cc731df4257 FAIL (2025-11-11 08:29:43 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:50034->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 88.265629ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 6.584951908s trace: https://jaeger.dev.storj.io/trace/49621e5e6beafe8e FAIL (2025-11-11 08:29:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 11 08:10:09.433180409 (48h6m7.98465087s) [upload=42408]: Starting upload Nov 11 08:10:09.433429884 (48h6m7.984900335s) [upload=42408]: Done starting upload Nov 11 08:10:09.433567967 (48h6m7.985038438s) [upload=42408]: Uploading segments... Nov 11 08:10:09.434191213 (48h6m7.985661674s) [upload=42408]: Got next segment. Inline: false Nov 11 08:10:09.434274032 (48h6m7.985744493s) [upload=42408]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 08:10:09.434282217 (48h6m7.985752688s) [upload=42408]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 08:10:09.714872160 (48h6m8.266342621s) [seg_pos={0 0}, upload=42408]: Begin upload segment... Nov 11 08:10:09.715084585 (48h6m8.266555046s) [seg_pos={0 0}, upload=42408]: Done begin upload segment. Nov 11 08:10:09.715464638 (48h6m8.2669351s) [noise=true , address=45.91.116.8:4047 , piece_id=VP6VW2ZTNSKS35R47GGJZI4YSHNNFUFB4YQ3W3QFT5ENN7F73VQA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.715555221 (48h6m8.267025692s) [noise=true , address=145.79.227.30:4761 , piece_id=ASO5O5LM6CEF6DXGBVOCI3NOCWLGKDU2T5JHOL5ZPWEXMTRMKZIA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.718085410 (48h6m8.269555901s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.718721901 (48h6m8.270192402s) [noise=true , address=145.79.227.11:4729 , piece_id=CDIU7FFPCVQGDK3K5ZDUARJ6VGB7KIY2DEVL7ZRLW5EASIFEHKHA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.719505724 (48h6m8.270976195s) [noise=true , address=145.79.227.27:4689 , piece_id=KLTJYVDWHEE2BGLUSP2XT6L5R2C5OB7TF4DZLYBF6R5OG3N6AEFQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.720136295 (48h6m8.271606756s) [noise=true , address=145.79.227.14:4840 , piece_id=QASWXZB3E6E7VYGF24KZ37Q3LKT6F4WVFKTBV3YS73KYAWTBV7YQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.721166056 (48h6m8.272636517s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.721883453 (48h6m8.273353994s) [noise=true , address=45.91.116.4:4939 , piece_id=KMHEQSQTA2WEWMIO7JDQZUDLSRV4CA6OJUN2CYYUSA7SU2ZDQBCQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.722398593 (48h6m8.273869054s) [noise=true , address=77.90.31.18:4806 , piece_id=LNL5LRHH3OOELKEO4HND2K3GNVDPCBCGKBNJ75DCLEKRAOMTQLKA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.724101306 (48h6m8.275571767s) [noise=true , address=145.79.227.17:4970 , piece_id=F2227LWTX5PRCQIUEJNRJ3QS3GRJOIEJIZFV3KRF7BAEAPSQZR7A, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.726385264 (48h6m8.277855776s) [noise=true , address=45.91.116.11:4117 , piece_id=UOEFPOPOFIO5KXY2JOX54YHUWNMTCAFMXEBKAMMR4GSR5ZE6J6PA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.728323225 (48h6m8.279793686s) [noise=true , address=145.79.227.20:4437 , piece_id=AOK6MTRZKPCXTNHZWGBYJGRBYNHFWO7GSG2PT4DFSMDAWK75GY4A, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.730321541 (48h6m8.281792002s) [noise=true , address=145.79.227.9:4366 , piece_id=JDBR4PTBZ5RN6NTWPERVLFANUAZQF3FSGIENVRHSYN56ED5NT2AQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.731324300 (48h6m8.282794832s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.732226989 (48h6m8.28369749s) [noise=true , address=77.90.31.19:4841 , piece_id=6JVB2WIEA2STMNBVTM6D7IYFHQY4QJGXAYHJAQILMYZZDH2MWIVA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.733026101 (48h6m8.284496562s) [noise=true , address=145.79.227.21:4474 , piece_id=RNB5T6DIUCY7IV2ULLO2UJRDKUJXI34UM4ZCHPJB22DNQ47VZJVA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:11.398415378 (48h6m9.949885839s) [upload=42408]: Next returned nil segment Nov 11 08:10:11.398434043 (48h6m9.949904504s) [upload=42408]: Waiting for error group managing segments... Nov 11 08:10:11.796154582 (48h6m10.347625043s) [noise=true , address=77.90.31.18:4806 , piece_id=LNL5LRHH3OOELKEO4HND2K3GNVDPCBCGKBNJ75DCLEKRAOMTQLKA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.809043214 (48h6m10.360513675s) [noise=true , address=45.91.116.8:4047 , piece_id=VP6VW2ZTNSKS35R47GGJZI4YSHNNFUFB4YQ3W3QFT5ENN7F73VQA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.866335992 (48h6m10.417806453s) [noise=true , address=45.91.116.11:4117 , piece_id=UOEFPOPOFIO5KXY2JOX54YHUWNMTCAFMXEBKAMMR4GSR5ZE6J6PA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.932922087 (48h6m10.484392538s) [noise=true , address=77.90.31.19:4841 , piece_id=6JVB2WIEA2STMNBVTM6D7IYFHQY4QJGXAYHJAQILMYZZDH2MWIVA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:12.005905333 (48h6m10.557375794s) [noise=true , address=45.91.116.4:4939 , piece_id=KMHEQSQTA2WEWMIO7JDQZUDLSRV4CA6OJUN2CYYUSA7SU2ZDQBCQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:16.703757864 (48h6m15.255228326s) [noise=true , address=145.79.227.9:4366 , piece_id=JDBR4PTBZ5RN6NTWPERVLFANUAZQF3FSGIENVRHSYN56ED5NT2AQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:17.540298352 (48h6m16.091768813s) [noise=true , address=145.79.227.21:4474 , piece_id=RNB5T6DIUCY7IV2ULLO2UJRDKUJXI34UM4ZCHPJB22DNQ47VZJVA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:19.805526866 (48h6m18.356997337s) [noise=true , address=145.79.227.27:4689 , piece_id=KLTJYVDWHEE2BGLUSP2XT6L5R2C5OB7TF4DZLYBF6R5OG3N6AEFQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:24.397542313 (48h6m22.949012764s) [noise=true , address=145.79.227.17:4970 , piece_id=F2227LWTX5PRCQIUEJNRJ3QS3GRJOIEJIZFV3KRF7BAEAPSQZR7A, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:24.779686050 (48h6m23.331156511s) [noise=true , address=145.79.227.14:4840 , piece_id=QASWXZB3E6E7VYGF24KZ37Q3LKT6F4WVFKTBV3YS73KYAWTBV7YQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:25.298168925 (48h6m23.849639387s) [noise=true , address=145.79.227.11:4729 , piece_id=CDIU7FFPCVQGDK3K5ZDUARJ6VGB7KIY2DEVL7ZRLW5EASIFEHKHA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:35.044925402 (48h6m33.596395853s) [noise=true , address=145.79.227.30:4761 , piece_id=ASO5O5LM6CEF6DXGBVOCI3NOCWLGKDU2T5JHOL5ZPWEXMTRMKZIA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:37.306959937 (48h6m35.858430398s) [noise=true , address=145.79.227.20:4437 , piece_id=AOK6MTRZKPCXTNHZWGBYJGRBYNHFWO7GSG2PT4DFSMDAWK75GY4A, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:37.306987550 (48h6m35.858458011s) [seg_pos={0 0}, upload=42408]: Segment reached optimal threshold of 13 pieces. Nov 11 08:10:37.307209231 (48h6m35.858679702s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4): protocol: expected piece hash; context canceled; EOF Nov 11 08:10:37.307218759 (48h6m35.85868922s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307368936 (48h6m35.858839397s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw): protocol: expected piece hash; context canceled; EOF Nov 11 08:10:37.307376741 (48h6m35.858847202s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307561282 (48h6m35.859031743s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9): protocol: write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection Nov 11 08:10:37.307585969 (48h6m35.85905643s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307629221 (48h6m35.859099682s) [seg_pos={0 0}, upload=42408]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 11 08:10:37.307741005 (48h6m35.859211466s) [upload=42408]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 11 08:10:37.307750052 (48h6m35.859220513s) [upload=42408]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 11 08:10:37.612339386 (48h6m36.163809848s) [upload=42408]: Done uploading segments... Nov 11 08:10:37.612366317 (48h6m36.163836788s) [upload=42408]: Upload finished. err: <nil> Duration: 6.58498393s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 5.779245645s trace: https://jaeger.dev.storj.io/trace/49621e5e6beafe8e FAIL (2025-11-11 08:28:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 11 08:10:09.433180409 (48h6m7.98465087s) [upload=42408]: Starting upload Nov 11 08:10:09.433429884 (48h6m7.984900335s) [upload=42408]: Done starting upload Nov 11 08:10:09.433567967 (48h6m7.985038438s) [upload=42408]: Uploading segments... Nov 11 08:10:09.434191213 (48h6m7.985661674s) [upload=42408]: Got next segment. Inline: false Nov 11 08:10:09.434274032 (48h6m7.985744493s) [upload=42408]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 08:10:09.434282217 (48h6m7.985752688s) [upload=42408]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 08:10:09.714872160 (48h6m8.266342621s) [seg_pos={0 0}, upload=42408]: Begin upload segment... Nov 11 08:10:09.715084585 (48h6m8.266555046s) [seg_pos={0 0}, upload=42408]: Done begin upload segment. Nov 11 08:10:09.715464638 (48h6m8.2669351s) [noise=true , address=45.91.116.8:4047 , piece_id=VP6VW2ZTNSKS35R47GGJZI4YSHNNFUFB4YQ3W3QFT5ENN7F73VQA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.715555221 (48h6m8.267025692s) [noise=true , address=145.79.227.30:4761 , piece_id=ASO5O5LM6CEF6DXGBVOCI3NOCWLGKDU2T5JHOL5ZPWEXMTRMKZIA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.718085410 (48h6m8.269555901s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.718721901 (48h6m8.270192402s) [noise=true , address=145.79.227.11:4729 , piece_id=CDIU7FFPCVQGDK3K5ZDUARJ6VGB7KIY2DEVL7ZRLW5EASIFEHKHA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.719505724 (48h6m8.270976195s) [noise=true , address=145.79.227.27:4689 , piece_id=KLTJYVDWHEE2BGLUSP2XT6L5R2C5OB7TF4DZLYBF6R5OG3N6AEFQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.720136295 (48h6m8.271606756s) [noise=true , address=145.79.227.14:4840 , piece_id=QASWXZB3E6E7VYGF24KZ37Q3LKT6F4WVFKTBV3YS73KYAWTBV7YQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.721166056 (48h6m8.272636517s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.721883453 (48h6m8.273353994s) [noise=true , address=45.91.116.4:4939 , piece_id=KMHEQSQTA2WEWMIO7JDQZUDLSRV4CA6OJUN2CYYUSA7SU2ZDQBCQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.722398593 (48h6m8.273869054s) [noise=true , address=77.90.31.18:4806 , piece_id=LNL5LRHH3OOELKEO4HND2K3GNVDPCBCGKBNJ75DCLEKRAOMTQLKA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.724101306 (48h6m8.275571767s) [noise=true , address=145.79.227.17:4970 , piece_id=F2227LWTX5PRCQIUEJNRJ3QS3GRJOIEJIZFV3KRF7BAEAPSQZR7A, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.726385264 (48h6m8.277855776s) [noise=true , address=45.91.116.11:4117 , piece_id=UOEFPOPOFIO5KXY2JOX54YHUWNMTCAFMXEBKAMMR4GSR5ZE6J6PA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.728323225 (48h6m8.279793686s) [noise=true , address=145.79.227.20:4437 , piece_id=AOK6MTRZKPCXTNHZWGBYJGRBYNHFWO7GSG2PT4DFSMDAWK75GY4A, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.730321541 (48h6m8.281792002s) [noise=true , address=145.79.227.9:4366 , piece_id=JDBR4PTBZ5RN6NTWPERVLFANUAZQF3FSGIENVRHSYN56ED5NT2AQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.731324300 (48h6m8.282794832s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.732226989 (48h6m8.28369749s) [noise=true , address=77.90.31.19:4841 , piece_id=6JVB2WIEA2STMNBVTM6D7IYFHQY4QJGXAYHJAQILMYZZDH2MWIVA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.733026101 (48h6m8.284496562s) [noise=true , address=145.79.227.21:4474 , piece_id=RNB5T6DIUCY7IV2ULLO2UJRDKUJXI34UM4ZCHPJB22DNQ47VZJVA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:11.398415378 (48h6m9.949885839s) [upload=42408]: Next returned nil segment Nov 11 08:10:11.398434043 (48h6m9.949904504s) [upload=42408]: Waiting for error group managing segments... Nov 11 08:10:11.796154582 (48h6m10.347625043s) [noise=true , address=77.90.31.18:4806 , piece_id=LNL5LRHH3OOELKEO4HND2K3GNVDPCBCGKBNJ75DCLEKRAOMTQLKA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.809043214 (48h6m10.360513675s) [noise=true , address=45.91.116.8:4047 , piece_id=VP6VW2ZTNSKS35R47GGJZI4YSHNNFUFB4YQ3W3QFT5ENN7F73VQA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.866335992 (48h6m10.417806453s) [noise=true , address=45.91.116.11:4117 , piece_id=UOEFPOPOFIO5KXY2JOX54YHUWNMTCAFMXEBKAMMR4GSR5ZE6J6PA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.932922087 (48h6m10.484392538s) [noise=true , address=77.90.31.19:4841 , piece_id=6JVB2WIEA2STMNBVTM6D7IYFHQY4QJGXAYHJAQILMYZZDH2MWIVA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:12.005905333 (48h6m10.557375794s) [noise=true , address=45.91.116.4:4939 , piece_id=KMHEQSQTA2WEWMIO7JDQZUDLSRV4CA6OJUN2CYYUSA7SU2ZDQBCQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:16.703757864 (48h6m15.255228326s) [noise=true , address=145.79.227.9:4366 , piece_id=JDBR4PTBZ5RN6NTWPERVLFANUAZQF3FSGIENVRHSYN56ED5NT2AQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:17.540298352 (48h6m16.091768813s) [noise=true , address=145.79.227.21:4474 , piece_id=RNB5T6DIUCY7IV2ULLO2UJRDKUJXI34UM4ZCHPJB22DNQ47VZJVA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:19.805526866 (48h6m18.356997337s) [noise=true , address=145.79.227.27:4689 , piece_id=KLTJYVDWHEE2BGLUSP2XT6L5R2C5OB7TF4DZLYBF6R5OG3N6AEFQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:24.397542313 (48h6m22.949012764s) [noise=true , address=145.79.227.17:4970 , piece_id=F2227LWTX5PRCQIUEJNRJ3QS3GRJOIEJIZFV3KRF7BAEAPSQZR7A, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:24.779686050 (48h6m23.331156511s) [noise=true , address=145.79.227.14:4840 , piece_id=QASWXZB3E6E7VYGF24KZ37Q3LKT6F4WVFKTBV3YS73KYAWTBV7YQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:25.298168925 (48h6m23.849639387s) [noise=true , address=145.79.227.11:4729 , piece_id=CDIU7FFPCVQGDK3K5ZDUARJ6VGB7KIY2DEVL7ZRLW5EASIFEHKHA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:35.044925402 (48h6m33.596395853s) [noise=true , address=145.79.227.30:4761 , piece_id=ASO5O5LM6CEF6DXGBVOCI3NOCWLGKDU2T5JHOL5ZPWEXMTRMKZIA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:37.306959937 (48h6m35.858430398s) [noise=true , address=145.79.227.20:4437 , piece_id=AOK6MTRZKPCXTNHZWGBYJGRBYNHFWO7GSG2PT4DFSMDAWK75GY4A, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:37.306987550 (48h6m35.858458011s) [seg_pos={0 0}, upload=42408]: Segment reached optimal threshold of 13 pieces. Nov 11 08:10:37.307209231 (48h6m35.858679702s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4): protocol: expected piece hash; context canceled; EOF Nov 11 08:10:37.307218759 (48h6m35.85868922s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307368936 (48h6m35.858839397s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw): protocol: expected piece hash; context canceled; EOF Nov 11 08:10:37.307376741 (48h6m35.858847202s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307561282 (48h6m35.859031743s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9): protocol: write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection Nov 11 08:10:37.307585969 (48h6m35.85905643s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307629221 (48h6m35.859099682s) [seg_pos={0 0}, upload=42408]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 11 08:10:37.307741005 (48h6m35.859211466s) [upload=42408]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 11 08:10:37.307750052 (48h6m35.859220513s) [upload=42408]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 11 08:10:37.612339386 (48h6m36.163809848s) [upload=42408]: Done uploading segments... Nov 11 08:10:37.612366317 (48h6m36.163836788s) [upload=42408]: Upload finished. err: <nil> Duration: 5.77927475s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [...X...........] duration: 57.432432ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/77081cc731df4257 FAIL (2025-11-11 08:28:40 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:50034->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 57.460084ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 5.908437576s trace: https://jaeger.dev.storj.io/trace/49621e5e6beafe8e FAIL (2025-11-11 08:28:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 11 08:10:09.433180409 (48h6m7.98465087s) [upload=42408]: Starting upload Nov 11 08:10:09.433429884 (48h6m7.984900335s) [upload=42408]: Done starting upload Nov 11 08:10:09.433567967 (48h6m7.985038438s) [upload=42408]: Uploading segments... Nov 11 08:10:09.434191213 (48h6m7.985661674s) [upload=42408]: Got next segment. Inline: false Nov 11 08:10:09.434274032 (48h6m7.985744493s) [upload=42408]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 08:10:09.434282217 (48h6m7.985752688s) [upload=42408]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 08:10:09.714872160 (48h6m8.266342621s) [seg_pos={0 0}, upload=42408]: Begin upload segment... Nov 11 08:10:09.715084585 (48h6m8.266555046s) [seg_pos={0 0}, upload=42408]: Done begin upload segment. Nov 11 08:10:09.715464638 (48h6m8.2669351s) [noise=true , address=45.91.116.8:4047 , piece_id=VP6VW2ZTNSKS35R47GGJZI4YSHNNFUFB4YQ3W3QFT5ENN7F73VQA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.715555221 (48h6m8.267025692s) [noise=true , address=145.79.227.30:4761 , piece_id=ASO5O5LM6CEF6DXGBVOCI3NOCWLGKDU2T5JHOL5ZPWEXMTRMKZIA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.718085410 (48h6m8.269555901s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.718721901 (48h6m8.270192402s) [noise=true , address=145.79.227.11:4729 , piece_id=CDIU7FFPCVQGDK3K5ZDUARJ6VGB7KIY2DEVL7ZRLW5EASIFEHKHA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.719505724 (48h6m8.270976195s) [noise=true , address=145.79.227.27:4689 , piece_id=KLTJYVDWHEE2BGLUSP2XT6L5R2C5OB7TF4DZLYBF6R5OG3N6AEFQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.720136295 (48h6m8.271606756s) [noise=true , address=145.79.227.14:4840 , piece_id=QASWXZB3E6E7VYGF24KZ37Q3LKT6F4WVFKTBV3YS73KYAWTBV7YQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.721166056 (48h6m8.272636517s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.721883453 (48h6m8.273353994s) [noise=true , address=45.91.116.4:4939 , piece_id=KMHEQSQTA2WEWMIO7JDQZUDLSRV4CA6OJUN2CYYUSA7SU2ZDQBCQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.722398593 (48h6m8.273869054s) [noise=true , address=77.90.31.18:4806 , piece_id=LNL5LRHH3OOELKEO4HND2K3GNVDPCBCGKBNJ75DCLEKRAOMTQLKA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.724101306 (48h6m8.275571767s) [noise=true , address=145.79.227.17:4970 , piece_id=F2227LWTX5PRCQIUEJNRJ3QS3GRJOIEJIZFV3KRF7BAEAPSQZR7A, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.726385264 (48h6m8.277855776s) [noise=true , address=45.91.116.11:4117 , piece_id=UOEFPOPOFIO5KXY2JOX54YHUWNMTCAFMXEBKAMMR4GSR5ZE6J6PA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.728323225 (48h6m8.279793686s) [noise=true , address=145.79.227.20:4437 , piece_id=AOK6MTRZKPCXTNHZWGBYJGRBYNHFWO7GSG2PT4DFSMDAWK75GY4A, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.730321541 (48h6m8.281792002s) [noise=true , address=145.79.227.9:4366 , piece_id=JDBR4PTBZ5RN6NTWPERVLFANUAZQF3FSGIENVRHSYN56ED5NT2AQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.731324300 (48h6m8.282794832s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.732226989 (48h6m8.28369749s) [noise=true , address=77.90.31.19:4841 , piece_id=6JVB2WIEA2STMNBVTM6D7IYFHQY4QJGXAYHJAQILMYZZDH2MWIVA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.733026101 (48h6m8.284496562s) [noise=true , address=145.79.227.21:4474 , piece_id=RNB5T6DIUCY7IV2ULLO2UJRDKUJXI34UM4ZCHPJB22DNQ47VZJVA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:11.398415378 (48h6m9.949885839s) [upload=42408]: Next returned nil segment Nov 11 08:10:11.398434043 (48h6m9.949904504s) [upload=42408]: Waiting for error group managing segments... Nov 11 08:10:11.796154582 (48h6m10.347625043s) [noise=true , address=77.90.31.18:4806 , piece_id=LNL5LRHH3OOELKEO4HND2K3GNVDPCBCGKBNJ75DCLEKRAOMTQLKA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.809043214 (48h6m10.360513675s) [noise=true , address=45.91.116.8:4047 , piece_id=VP6VW2ZTNSKS35R47GGJZI4YSHNNFUFB4YQ3W3QFT5ENN7F73VQA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.866335992 (48h6m10.417806453s) [noise=true , address=45.91.116.11:4117 , piece_id=UOEFPOPOFIO5KXY2JOX54YHUWNMTCAFMXEBKAMMR4GSR5ZE6J6PA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.932922087 (48h6m10.484392538s) [noise=true , address=77.90.31.19:4841 , piece_id=6JVB2WIEA2STMNBVTM6D7IYFHQY4QJGXAYHJAQILMYZZDH2MWIVA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:12.005905333 (48h6m10.557375794s) [noise=true , address=45.91.116.4:4939 , piece_id=KMHEQSQTA2WEWMIO7JDQZUDLSRV4CA6OJUN2CYYUSA7SU2ZDQBCQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:16.703757864 (48h6m15.255228326s) [noise=true , address=145.79.227.9:4366 , piece_id=JDBR4PTBZ5RN6NTWPERVLFANUAZQF3FSGIENVRHSYN56ED5NT2AQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:17.540298352 (48h6m16.091768813s) [noise=true , address=145.79.227.21:4474 , piece_id=RNB5T6DIUCY7IV2ULLO2UJRDKUJXI34UM4ZCHPJB22DNQ47VZJVA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:19.805526866 (48h6m18.356997337s) [noise=true , address=145.79.227.27:4689 , piece_id=KLTJYVDWHEE2BGLUSP2XT6L5R2C5OB7TF4DZLYBF6R5OG3N6AEFQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:24.397542313 (48h6m22.949012764s) [noise=true , address=145.79.227.17:4970 , piece_id=F2227LWTX5PRCQIUEJNRJ3QS3GRJOIEJIZFV3KRF7BAEAPSQZR7A, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:24.779686050 (48h6m23.331156511s) [noise=true , address=145.79.227.14:4840 , piece_id=QASWXZB3E6E7VYGF24KZ37Q3LKT6F4WVFKTBV3YS73KYAWTBV7YQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:25.298168925 (48h6m23.849639387s) [noise=true , address=145.79.227.11:4729 , piece_id=CDIU7FFPCVQGDK3K5ZDUARJ6VGB7KIY2DEVL7ZRLW5EASIFEHKHA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:35.044925402 (48h6m33.596395853s) [noise=true , address=145.79.227.30:4761 , piece_id=ASO5O5LM6CEF6DXGBVOCI3NOCWLGKDU2T5JHOL5ZPWEXMTRMKZIA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:37.306959937 (48h6m35.858430398s) [noise=true , address=145.79.227.20:4437 , piece_id=AOK6MTRZKPCXTNHZWGBYJGRBYNHFWO7GSG2PT4DFSMDAWK75GY4A, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:37.306987550 (48h6m35.858458011s) [seg_pos={0 0}, upload=42408]: Segment reached optimal threshold of 13 pieces. Nov 11 08:10:37.307209231 (48h6m35.858679702s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4): protocol: expected piece hash; context canceled; EOF Nov 11 08:10:37.307218759 (48h6m35.85868922s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307368936 (48h6m35.858839397s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw): protocol: expected piece hash; context canceled; EOF Nov 11 08:10:37.307376741 (48h6m35.858847202s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307561282 (48h6m35.859031743s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9): protocol: write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection Nov 11 08:10:37.307585969 (48h6m35.85905643s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307629221 (48h6m35.859099682s) [seg_pos={0 0}, upload=42408]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 11 08:10:37.307741005 (48h6m35.859211466s) [upload=42408]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 11 08:10:37.307750052 (48h6m35.859220513s) [upload=42408]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 11 08:10:37.612339386 (48h6m36.163809848s) [upload=42408]: Done uploading segments... Nov 11 08:10:37.612366317 (48h6m36.163836788s) [upload=42408]: Upload finished. err: <nil> Duration: 5.908485366s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 5.888201359s trace: https://jaeger.dev.storj.io/trace/49621e5e6beafe8e FAIL (2025-11-11 08:27:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 11 08:10:09.433180409 (48h6m7.98465087s) [upload=42408]: Starting upload Nov 11 08:10:09.433429884 (48h6m7.984900335s) [upload=42408]: Done starting upload Nov 11 08:10:09.433567967 (48h6m7.985038438s) [upload=42408]: Uploading segments... Nov 11 08:10:09.434191213 (48h6m7.985661674s) [upload=42408]: Got next segment. Inline: false Nov 11 08:10:09.434274032 (48h6m7.985744493s) [upload=42408]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 08:10:09.434282217 (48h6m7.985752688s) [upload=42408]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 08:10:09.714872160 (48h6m8.266342621s) [seg_pos={0 0}, upload=42408]: Begin upload segment... Nov 11 08:10:09.715084585 (48h6m8.266555046s) [seg_pos={0 0}, upload=42408]: Done begin upload segment. Nov 11 08:10:09.715464638 (48h6m8.2669351s) [noise=true , address=45.91.116.8:4047 , piece_id=VP6VW2ZTNSKS35R47GGJZI4YSHNNFUFB4YQ3W3QFT5ENN7F73VQA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.715555221 (48h6m8.267025692s) [noise=true , address=145.79.227.30:4761 , piece_id=ASO5O5LM6CEF6DXGBVOCI3NOCWLGKDU2T5JHOL5ZPWEXMTRMKZIA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.718085410 (48h6m8.269555901s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.718721901 (48h6m8.270192402s) [noise=true , address=145.79.227.11:4729 , piece_id=CDIU7FFPCVQGDK3K5ZDUARJ6VGB7KIY2DEVL7ZRLW5EASIFEHKHA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.719505724 (48h6m8.270976195s) [noise=true , address=145.79.227.27:4689 , piece_id=KLTJYVDWHEE2BGLUSP2XT6L5R2C5OB7TF4DZLYBF6R5OG3N6AEFQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.720136295 (48h6m8.271606756s) [noise=true , address=145.79.227.14:4840 , piece_id=QASWXZB3E6E7VYGF24KZ37Q3LKT6F4WVFKTBV3YS73KYAWTBV7YQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.721166056 (48h6m8.272636517s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.721883453 (48h6m8.273353994s) [noise=true , address=45.91.116.4:4939 , piece_id=KMHEQSQTA2WEWMIO7JDQZUDLSRV4CA6OJUN2CYYUSA7SU2ZDQBCQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.722398593 (48h6m8.273869054s) [noise=true , address=77.90.31.18:4806 , piece_id=LNL5LRHH3OOELKEO4HND2K3GNVDPCBCGKBNJ75DCLEKRAOMTQLKA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.724101306 (48h6m8.275571767s) [noise=true , address=145.79.227.17:4970 , piece_id=F2227LWTX5PRCQIUEJNRJ3QS3GRJOIEJIZFV3KRF7BAEAPSQZR7A, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.726385264 (48h6m8.277855776s) [noise=true , address=45.91.116.11:4117 , piece_id=UOEFPOPOFIO5KXY2JOX54YHUWNMTCAFMXEBKAMMR4GSR5ZE6J6PA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.728323225 (48h6m8.279793686s) [noise=true , address=145.79.227.20:4437 , piece_id=AOK6MTRZKPCXTNHZWGBYJGRBYNHFWO7GSG2PT4DFSMDAWK75GY4A, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.730321541 (48h6m8.281792002s) [noise=true , address=145.79.227.9:4366 , piece_id=JDBR4PTBZ5RN6NTWPERVLFANUAZQF3FSGIENVRHSYN56ED5NT2AQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.731324300 (48h6m8.282794832s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.732226989 (48h6m8.28369749s) [noise=true , address=77.90.31.19:4841 , piece_id=6JVB2WIEA2STMNBVTM6D7IYFHQY4QJGXAYHJAQILMYZZDH2MWIVA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.733026101 (48h6m8.284496562s) [noise=true , address=145.79.227.21:4474 , piece_id=RNB5T6DIUCY7IV2ULLO2UJRDKUJXI34UM4ZCHPJB22DNQ47VZJVA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:11.398415378 (48h6m9.949885839s) [upload=42408]: Next returned nil segment Nov 11 08:10:11.398434043 (48h6m9.949904504s) [upload=42408]: Waiting for error group managing segments... Nov 11 08:10:11.796154582 (48h6m10.347625043s) [noise=true , address=77.90.31.18:4806 , piece_id=LNL5LRHH3OOELKEO4HND2K3GNVDPCBCGKBNJ75DCLEKRAOMTQLKA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.809043214 (48h6m10.360513675s) [noise=true , address=45.91.116.8:4047 , piece_id=VP6VW2ZTNSKS35R47GGJZI4YSHNNFUFB4YQ3W3QFT5ENN7F73VQA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.866335992 (48h6m10.417806453s) [noise=true , address=45.91.116.11:4117 , piece_id=UOEFPOPOFIO5KXY2JOX54YHUWNMTCAFMXEBKAMMR4GSR5ZE6J6PA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.932922087 (48h6m10.484392538s) [noise=true , address=77.90.31.19:4841 , piece_id=6JVB2WIEA2STMNBVTM6D7IYFHQY4QJGXAYHJAQILMYZZDH2MWIVA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:12.005905333 (48h6m10.557375794s) [noise=true , address=45.91.116.4:4939 , piece_id=KMHEQSQTA2WEWMIO7JDQZUDLSRV4CA6OJUN2CYYUSA7SU2ZDQBCQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:16.703757864 (48h6m15.255228326s) [noise=true , address=145.79.227.9:4366 , piece_id=JDBR4PTBZ5RN6NTWPERVLFANUAZQF3FSGIENVRHSYN56ED5NT2AQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:17.540298352 (48h6m16.091768813s) [noise=true , address=145.79.227.21:4474 , piece_id=RNB5T6DIUCY7IV2ULLO2UJRDKUJXI34UM4ZCHPJB22DNQ47VZJVA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:19.805526866 (48h6m18.356997337s) [noise=true , address=145.79.227.27:4689 , piece_id=KLTJYVDWHEE2BGLUSP2XT6L5R2C5OB7TF4DZLYBF6R5OG3N6AEFQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:24.397542313 (48h6m22.949012764s) [noise=true , address=145.79.227.17:4970 , piece_id=F2227LWTX5PRCQIUEJNRJ3QS3GRJOIEJIZFV3KRF7BAEAPSQZR7A, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:24.779686050 (48h6m23.331156511s) [noise=true , address=145.79.227.14:4840 , piece_id=QASWXZB3E6E7VYGF24KZ37Q3LKT6F4WVFKTBV3YS73KYAWTBV7YQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:25.298168925 (48h6m23.849639387s) [noise=true , address=145.79.227.11:4729 , piece_id=CDIU7FFPCVQGDK3K5ZDUARJ6VGB7KIY2DEVL7ZRLW5EASIFEHKHA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:35.044925402 (48h6m33.596395853s) [noise=true , address=145.79.227.30:4761 , piece_id=ASO5O5LM6CEF6DXGBVOCI3NOCWLGKDU2T5JHOL5ZPWEXMTRMKZIA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:37.306959937 (48h6m35.858430398s) [noise=true , address=145.79.227.20:4437 , piece_id=AOK6MTRZKPCXTNHZWGBYJGRBYNHFWO7GSG2PT4DFSMDAWK75GY4A, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:37.306987550 (48h6m35.858458011s) [seg_pos={0 0}, upload=42408]: Segment reached optimal threshold of 13 pieces. Nov 11 08:10:37.307209231 (48h6m35.858679702s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4): protocol: expected piece hash; context canceled; EOF Nov 11 08:10:37.307218759 (48h6m35.85868922s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307368936 (48h6m35.858839397s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw): protocol: expected piece hash; context canceled; EOF Nov 11 08:10:37.307376741 (48h6m35.858847202s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307561282 (48h6m35.859031743s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9): protocol: write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection Nov 11 08:10:37.307585969 (48h6m35.85905643s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307629221 (48h6m35.859099682s) [seg_pos={0 0}, upload=42408]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 11 08:10:37.307741005 (48h6m35.859211466s) [upload=42408]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 11 08:10:37.307750052 (48h6m35.859220513s) [upload=42408]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 11 08:10:37.612339386 (48h6m36.163809848s) [upload=42408]: Done uploading segments... Nov 11 08:10:37.612366317 (48h6m36.163836788s) [upload=42408]: Upload finished. err: <nil> Duration: 5.888232206s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [....X..........] duration: 105.90409ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/77081cc731df4257 FAIL (2025-11-11 08:27:30 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:50034->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 105.937242ms core website
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [.....X.........] duration: 50.080112ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/77081cc731df4257 FAIL (2025-11-11 08:26:55 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:50034->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 50.109757ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 5.836404612s trace: https://jaeger.dev.storj.io/trace/49621e5e6beafe8e FAIL (2025-11-11 08:26:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 11 08:10:09.433180409 (48h6m7.98465087s) [upload=42408]: Starting upload Nov 11 08:10:09.433429884 (48h6m7.984900335s) [upload=42408]: Done starting upload Nov 11 08:10:09.433567967 (48h6m7.985038438s) [upload=42408]: Uploading segments... Nov 11 08:10:09.434191213 (48h6m7.985661674s) [upload=42408]: Got next segment. Inline: false Nov 11 08:10:09.434274032 (48h6m7.985744493s) [upload=42408]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 08:10:09.434282217 (48h6m7.985752688s) [upload=42408]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 08:10:09.714872160 (48h6m8.266342621s) [seg_pos={0 0}, upload=42408]: Begin upload segment... Nov 11 08:10:09.715084585 (48h6m8.266555046s) [seg_pos={0 0}, upload=42408]: Done begin upload segment. Nov 11 08:10:09.715464638 (48h6m8.2669351s) [noise=true , address=45.91.116.8:4047 , piece_id=VP6VW2ZTNSKS35R47GGJZI4YSHNNFUFB4YQ3W3QFT5ENN7F73VQA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.715555221 (48h6m8.267025692s) [noise=true , address=145.79.227.30:4761 , piece_id=ASO5O5LM6CEF6DXGBVOCI3NOCWLGKDU2T5JHOL5ZPWEXMTRMKZIA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.718085410 (48h6m8.269555901s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.718721901 (48h6m8.270192402s) [noise=true , address=145.79.227.11:4729 , piece_id=CDIU7FFPCVQGDK3K5ZDUARJ6VGB7KIY2DEVL7ZRLW5EASIFEHKHA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.719505724 (48h6m8.270976195s) [noise=true , address=145.79.227.27:4689 , piece_id=KLTJYVDWHEE2BGLUSP2XT6L5R2C5OB7TF4DZLYBF6R5OG3N6AEFQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.720136295 (48h6m8.271606756s) [noise=true , address=145.79.227.14:4840 , piece_id=QASWXZB3E6E7VYGF24KZ37Q3LKT6F4WVFKTBV3YS73KYAWTBV7YQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.721166056 (48h6m8.272636517s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.721883453 (48h6m8.273353994s) [noise=true , address=45.91.116.4:4939 , piece_id=KMHEQSQTA2WEWMIO7JDQZUDLSRV4CA6OJUN2CYYUSA7SU2ZDQBCQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.722398593 (48h6m8.273869054s) [noise=true , address=77.90.31.18:4806 , piece_id=LNL5LRHH3OOELKEO4HND2K3GNVDPCBCGKBNJ75DCLEKRAOMTQLKA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.724101306 (48h6m8.275571767s) [noise=true , address=145.79.227.17:4970 , piece_id=F2227LWTX5PRCQIUEJNRJ3QS3GRJOIEJIZFV3KRF7BAEAPSQZR7A, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.726385264 (48h6m8.277855776s) [noise=true , address=45.91.116.11:4117 , piece_id=UOEFPOPOFIO5KXY2JOX54YHUWNMTCAFMXEBKAMMR4GSR5ZE6J6PA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.728323225 (48h6m8.279793686s) [noise=true , address=145.79.227.20:4437 , piece_id=AOK6MTRZKPCXTNHZWGBYJGRBYNHFWO7GSG2PT4DFSMDAWK75GY4A, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.730321541 (48h6m8.281792002s) [noise=true , address=145.79.227.9:4366 , piece_id=JDBR4PTBZ5RN6NTWPERVLFANUAZQF3FSGIENVRHSYN56ED5NT2AQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.731324300 (48h6m8.282794832s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.732226989 (48h6m8.28369749s) [noise=true , address=77.90.31.19:4841 , piece_id=6JVB2WIEA2STMNBVTM6D7IYFHQY4QJGXAYHJAQILMYZZDH2MWIVA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.733026101 (48h6m8.284496562s) [noise=true , address=145.79.227.21:4474 , piece_id=RNB5T6DIUCY7IV2ULLO2UJRDKUJXI34UM4ZCHPJB22DNQ47VZJVA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:11.398415378 (48h6m9.949885839s) [upload=42408]: Next returned nil segment Nov 11 08:10:11.398434043 (48h6m9.949904504s) [upload=42408]: Waiting for error group managing segments... Nov 11 08:10:11.796154582 (48h6m10.347625043s) [noise=true , address=77.90.31.18:4806 , piece_id=LNL5LRHH3OOELKEO4HND2K3GNVDPCBCGKBNJ75DCLEKRAOMTQLKA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.809043214 (48h6m10.360513675s) [noise=true , address=45.91.116.8:4047 , piece_id=VP6VW2ZTNSKS35R47GGJZI4YSHNNFUFB4YQ3W3QFT5ENN7F73VQA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.866335992 (48h6m10.417806453s) [noise=true , address=45.91.116.11:4117 , piece_id=UOEFPOPOFIO5KXY2JOX54YHUWNMTCAFMXEBKAMMR4GSR5ZE6J6PA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.932922087 (48h6m10.484392538s) [noise=true , address=77.90.31.19:4841 , piece_id=6JVB2WIEA2STMNBVTM6D7IYFHQY4QJGXAYHJAQILMYZZDH2MWIVA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:12.005905333 (48h6m10.557375794s) [noise=true , address=45.91.116.4:4939 , piece_id=KMHEQSQTA2WEWMIO7JDQZUDLSRV4CA6OJUN2CYYUSA7SU2ZDQBCQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:16.703757864 (48h6m15.255228326s) [noise=true , address=145.79.227.9:4366 , piece_id=JDBR4PTBZ5RN6NTWPERVLFANUAZQF3FSGIENVRHSYN56ED5NT2AQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:17.540298352 (48h6m16.091768813s) [noise=true , address=145.79.227.21:4474 , piece_id=RNB5T6DIUCY7IV2ULLO2UJRDKUJXI34UM4ZCHPJB22DNQ47VZJVA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:19.805526866 (48h6m18.356997337s) [noise=true , address=145.79.227.27:4689 , piece_id=KLTJYVDWHEE2BGLUSP2XT6L5R2C5OB7TF4DZLYBF6R5OG3N6AEFQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:24.397542313 (48h6m22.949012764s) [noise=true , address=145.79.227.17:4970 , piece_id=F2227LWTX5PRCQIUEJNRJ3QS3GRJOIEJIZFV3KRF7BAEAPSQZR7A, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:24.779686050 (48h6m23.331156511s) [noise=true , address=145.79.227.14:4840 , piece_id=QASWXZB3E6E7VYGF24KZ37Q3LKT6F4WVFKTBV3YS73KYAWTBV7YQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:25.298168925 (48h6m23.849639387s) [noise=true , address=145.79.227.11:4729 , piece_id=CDIU7FFPCVQGDK3K5ZDUARJ6VGB7KIY2DEVL7ZRLW5EASIFEHKHA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:35.044925402 (48h6m33.596395853s) [noise=true , address=145.79.227.30:4761 , piece_id=ASO5O5LM6CEF6DXGBVOCI3NOCWLGKDU2T5JHOL5ZPWEXMTRMKZIA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:37.306959937 (48h6m35.858430398s) [noise=true , address=145.79.227.20:4437 , piece_id=AOK6MTRZKPCXTNHZWGBYJGRBYNHFWO7GSG2PT4DFSMDAWK75GY4A, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:37.306987550 (48h6m35.858458011s) [seg_pos={0 0}, upload=42408]: Segment reached optimal threshold of 13 pieces. Nov 11 08:10:37.307209231 (48h6m35.858679702s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4): protocol: expected piece hash; context canceled; EOF Nov 11 08:10:37.307218759 (48h6m35.85868922s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307368936 (48h6m35.858839397s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw): protocol: expected piece hash; context canceled; EOF Nov 11 08:10:37.307376741 (48h6m35.858847202s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307561282 (48h6m35.859031743s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9): protocol: write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection Nov 11 08:10:37.307585969 (48h6m35.85905643s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307629221 (48h6m35.859099682s) [seg_pos={0 0}, upload=42408]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 11 08:10:37.307741005 (48h6m35.859211466s) [upload=42408]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 11 08:10:37.307750052 (48h6m35.859220513s) [upload=42408]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 11 08:10:37.612339386 (48h6m36.163809848s) [upload=42408]: Done uploading segments... Nov 11 08:10:37.612366317 (48h6m36.163836788s) [upload=42408]: Upload finished. err: <nil> Duration: 5.836435981s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [......X........] duration: 53.572493ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/77081cc731df4257 FAIL (2025-11-11 08:26:17 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:50034->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 53.593191ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 6.190202046s trace: https://jaeger.dev.storj.io/trace/49621e5e6beafe8e FAIL (2025-11-11 08:26:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 11 08:10:09.433180409 (48h6m7.98465087s) [upload=42408]: Starting upload Nov 11 08:10:09.433429884 (48h6m7.984900335s) [upload=42408]: Done starting upload Nov 11 08:10:09.433567967 (48h6m7.985038438s) [upload=42408]: Uploading segments... Nov 11 08:10:09.434191213 (48h6m7.985661674s) [upload=42408]: Got next segment. Inline: false Nov 11 08:10:09.434274032 (48h6m7.985744493s) [upload=42408]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 08:10:09.434282217 (48h6m7.985752688s) [upload=42408]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 08:10:09.714872160 (48h6m8.266342621s) [seg_pos={0 0}, upload=42408]: Begin upload segment... Nov 11 08:10:09.715084585 (48h6m8.266555046s) [seg_pos={0 0}, upload=42408]: Done begin upload segment. Nov 11 08:10:09.715464638 (48h6m8.2669351s) [noise=true , address=45.91.116.8:4047 , piece_id=VP6VW2ZTNSKS35R47GGJZI4YSHNNFUFB4YQ3W3QFT5ENN7F73VQA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.715555221 (48h6m8.267025692s) [noise=true , address=145.79.227.30:4761 , piece_id=ASO5O5LM6CEF6DXGBVOCI3NOCWLGKDU2T5JHOL5ZPWEXMTRMKZIA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.718085410 (48h6m8.269555901s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.718721901 (48h6m8.270192402s) [noise=true , address=145.79.227.11:4729 , piece_id=CDIU7FFPCVQGDK3K5ZDUARJ6VGB7KIY2DEVL7ZRLW5EASIFEHKHA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.719505724 (48h6m8.270976195s) [noise=true , address=145.79.227.27:4689 , piece_id=KLTJYVDWHEE2BGLUSP2XT6L5R2C5OB7TF4DZLYBF6R5OG3N6AEFQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.720136295 (48h6m8.271606756s) [noise=true , address=145.79.227.14:4840 , piece_id=QASWXZB3E6E7VYGF24KZ37Q3LKT6F4WVFKTBV3YS73KYAWTBV7YQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.721166056 (48h6m8.272636517s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.721883453 (48h6m8.273353994s) [noise=true , address=45.91.116.4:4939 , piece_id=KMHEQSQTA2WEWMIO7JDQZUDLSRV4CA6OJUN2CYYUSA7SU2ZDQBCQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.722398593 (48h6m8.273869054s) [noise=true , address=77.90.31.18:4806 , piece_id=LNL5LRHH3OOELKEO4HND2K3GNVDPCBCGKBNJ75DCLEKRAOMTQLKA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.724101306 (48h6m8.275571767s) [noise=true , address=145.79.227.17:4970 , piece_id=F2227LWTX5PRCQIUEJNRJ3QS3GRJOIEJIZFV3KRF7BAEAPSQZR7A, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.726385264 (48h6m8.277855776s) [noise=true , address=45.91.116.11:4117 , piece_id=UOEFPOPOFIO5KXY2JOX54YHUWNMTCAFMXEBKAMMR4GSR5ZE6J6PA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.728323225 (48h6m8.279793686s) [noise=true , address=145.79.227.20:4437 , piece_id=AOK6MTRZKPCXTNHZWGBYJGRBYNHFWO7GSG2PT4DFSMDAWK75GY4A, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.730321541 (48h6m8.281792002s) [noise=true , address=145.79.227.9:4366 , piece_id=JDBR4PTBZ5RN6NTWPERVLFANUAZQF3FSGIENVRHSYN56ED5NT2AQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.731324300 (48h6m8.282794832s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.732226989 (48h6m8.28369749s) [noise=true , address=77.90.31.19:4841 , piece_id=6JVB2WIEA2STMNBVTM6D7IYFHQY4QJGXAYHJAQILMYZZDH2MWIVA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.733026101 (48h6m8.284496562s) [noise=true , address=145.79.227.21:4474 , piece_id=RNB5T6DIUCY7IV2ULLO2UJRDKUJXI34UM4ZCHPJB22DNQ47VZJVA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:11.398415378 (48h6m9.949885839s) [upload=42408]: Next returned nil segment Nov 11 08:10:11.398434043 (48h6m9.949904504s) [upload=42408]: Waiting for error group managing segments... Nov 11 08:10:11.796154582 (48h6m10.347625043s) [noise=true , address=77.90.31.18:4806 , piece_id=LNL5LRHH3OOELKEO4HND2K3GNVDPCBCGKBNJ75DCLEKRAOMTQLKA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.809043214 (48h6m10.360513675s) [noise=true , address=45.91.116.8:4047 , piece_id=VP6VW2ZTNSKS35R47GGJZI4YSHNNFUFB4YQ3W3QFT5ENN7F73VQA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.866335992 (48h6m10.417806453s) [noise=true , address=45.91.116.11:4117 , piece_id=UOEFPOPOFIO5KXY2JOX54YHUWNMTCAFMXEBKAMMR4GSR5ZE6J6PA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.932922087 (48h6m10.484392538s) [noise=true , address=77.90.31.19:4841 , piece_id=6JVB2WIEA2STMNBVTM6D7IYFHQY4QJGXAYHJAQILMYZZDH2MWIVA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:12.005905333 (48h6m10.557375794s) [noise=true , address=45.91.116.4:4939 , piece_id=KMHEQSQTA2WEWMIO7JDQZUDLSRV4CA6OJUN2CYYUSA7SU2ZDQBCQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:16.703757864 (48h6m15.255228326s) [noise=true , address=145.79.227.9:4366 , piece_id=JDBR4PTBZ5RN6NTWPERVLFANUAZQF3FSGIENVRHSYN56ED5NT2AQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:17.540298352 (48h6m16.091768813s) [noise=true , address=145.79.227.21:4474 , piece_id=RNB5T6DIUCY7IV2ULLO2UJRDKUJXI34UM4ZCHPJB22DNQ47VZJVA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:19.805526866 (48h6m18.356997337s) [noise=true , address=145.79.227.27:4689 , piece_id=KLTJYVDWHEE2BGLUSP2XT6L5R2C5OB7TF4DZLYBF6R5OG3N6AEFQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:24.397542313 (48h6m22.949012764s) [noise=true , address=145.79.227.17:4970 , piece_id=F2227LWTX5PRCQIUEJNRJ3QS3GRJOIEJIZFV3KRF7BAEAPSQZR7A, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:24.779686050 (48h6m23.331156511s) [noise=true , address=145.79.227.14:4840 , piece_id=QASWXZB3E6E7VYGF24KZ37Q3LKT6F4WVFKTBV3YS73KYAWTBV7YQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:25.298168925 (48h6m23.849639387s) [noise=true , address=145.79.227.11:4729 , piece_id=CDIU7FFPCVQGDK3K5ZDUARJ6VGB7KIY2DEVL7ZRLW5EASIFEHKHA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:35.044925402 (48h6m33.596395853s) [noise=true , address=145.79.227.30:4761 , piece_id=ASO5O5LM6CEF6DXGBVOCI3NOCWLGKDU2T5JHOL5ZPWEXMTRMKZIA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:37.306959937 (48h6m35.858430398s) [noise=true , address=145.79.227.20:4437 , piece_id=AOK6MTRZKPCXTNHZWGBYJGRBYNHFWO7GSG2PT4DFSMDAWK75GY4A, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:37.306987550 (48h6m35.858458011s) [seg_pos={0 0}, upload=42408]: Segment reached optimal threshold of 13 pieces. Nov 11 08:10:37.307209231 (48h6m35.858679702s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4): protocol: expected piece hash; context canceled; EOF Nov 11 08:10:37.307218759 (48h6m35.85868922s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307368936 (48h6m35.858839397s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw): protocol: expected piece hash; context canceled; EOF Nov 11 08:10:37.307376741 (48h6m35.858847202s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307561282 (48h6m35.859031743s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9): protocol: write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection Nov 11 08:10:37.307585969 (48h6m35.85905643s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307629221 (48h6m35.859099682s) [seg_pos={0 0}, upload=42408]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 11 08:10:37.307741005 (48h6m35.859211466s) [upload=42408]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 11 08:10:37.307750052 (48h6m35.859220513s) [upload=42408]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 11 08:10:37.612339386 (48h6m36.163809848s) [upload=42408]: Done uploading segments... Nov 11 08:10:37.612366317 (48h6m36.163836788s) [upload=42408]: Upload finished. err: <nil> Duration: 6.190236652s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 5.696438495s trace: https://jaeger.dev.storj.io/trace/49621e5e6beafe8e FAIL (2025-11-11 08:25:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 11 08:10:09.433180409 (48h6m7.98465087s) [upload=42408]: Starting upload Nov 11 08:10:09.433429884 (48h6m7.984900335s) [upload=42408]: Done starting upload Nov 11 08:10:09.433567967 (48h6m7.985038438s) [upload=42408]: Uploading segments... Nov 11 08:10:09.434191213 (48h6m7.985661674s) [upload=42408]: Got next segment. Inline: false Nov 11 08:10:09.434274032 (48h6m7.985744493s) [upload=42408]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 08:10:09.434282217 (48h6m7.985752688s) [upload=42408]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 08:10:09.714872160 (48h6m8.266342621s) [seg_pos={0 0}, upload=42408]: Begin upload segment... Nov 11 08:10:09.715084585 (48h6m8.266555046s) [seg_pos={0 0}, upload=42408]: Done begin upload segment. Nov 11 08:10:09.715464638 (48h6m8.2669351s) [noise=true , address=45.91.116.8:4047 , piece_id=VP6VW2ZTNSKS35R47GGJZI4YSHNNFUFB4YQ3W3QFT5ENN7F73VQA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.715555221 (48h6m8.267025692s) [noise=true , address=145.79.227.30:4761 , piece_id=ASO5O5LM6CEF6DXGBVOCI3NOCWLGKDU2T5JHOL5ZPWEXMTRMKZIA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.718085410 (48h6m8.269555901s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.718721901 (48h6m8.270192402s) [noise=true , address=145.79.227.11:4729 , piece_id=CDIU7FFPCVQGDK3K5ZDUARJ6VGB7KIY2DEVL7ZRLW5EASIFEHKHA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.719505724 (48h6m8.270976195s) [noise=true , address=145.79.227.27:4689 , piece_id=KLTJYVDWHEE2BGLUSP2XT6L5R2C5OB7TF4DZLYBF6R5OG3N6AEFQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.720136295 (48h6m8.271606756s) [noise=true , address=145.79.227.14:4840 , piece_id=QASWXZB3E6E7VYGF24KZ37Q3LKT6F4WVFKTBV3YS73KYAWTBV7YQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.721166056 (48h6m8.272636517s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.721883453 (48h6m8.273353994s) [noise=true , address=45.91.116.4:4939 , piece_id=KMHEQSQTA2WEWMIO7JDQZUDLSRV4CA6OJUN2CYYUSA7SU2ZDQBCQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.722398593 (48h6m8.273869054s) [noise=true , address=77.90.31.18:4806 , piece_id=LNL5LRHH3OOELKEO4HND2K3GNVDPCBCGKBNJ75DCLEKRAOMTQLKA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.724101306 (48h6m8.275571767s) [noise=true , address=145.79.227.17:4970 , piece_id=F2227LWTX5PRCQIUEJNRJ3QS3GRJOIEJIZFV3KRF7BAEAPSQZR7A, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.726385264 (48h6m8.277855776s) [noise=true , address=45.91.116.11:4117 , piece_id=UOEFPOPOFIO5KXY2JOX54YHUWNMTCAFMXEBKAMMR4GSR5ZE6J6PA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.728323225 (48h6m8.279793686s) [noise=true , address=145.79.227.20:4437 , piece_id=AOK6MTRZKPCXTNHZWGBYJGRBYNHFWO7GSG2PT4DFSMDAWK75GY4A, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.730321541 (48h6m8.281792002s) [noise=true , address=145.79.227.9:4366 , piece_id=JDBR4PTBZ5RN6NTWPERVLFANUAZQF3FSGIENVRHSYN56ED5NT2AQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.731324300 (48h6m8.282794832s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.732226989 (48h6m8.28369749s) [noise=true , address=77.90.31.19:4841 , piece_id=6JVB2WIEA2STMNBVTM6D7IYFHQY4QJGXAYHJAQILMYZZDH2MWIVA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.733026101 (48h6m8.284496562s) [noise=true , address=145.79.227.21:4474 , piece_id=RNB5T6DIUCY7IV2ULLO2UJRDKUJXI34UM4ZCHPJB22DNQ47VZJVA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:11.398415378 (48h6m9.949885839s) [upload=42408]: Next returned nil segment Nov 11 08:10:11.398434043 (48h6m9.949904504s) [upload=42408]: Waiting for error group managing segments... Nov 11 08:10:11.796154582 (48h6m10.347625043s) [noise=true , address=77.90.31.18:4806 , piece_id=LNL5LRHH3OOELKEO4HND2K3GNVDPCBCGKBNJ75DCLEKRAOMTQLKA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.809043214 (48h6m10.360513675s) [noise=true , address=45.91.116.8:4047 , piece_id=VP6VW2ZTNSKS35R47GGJZI4YSHNNFUFB4YQ3W3QFT5ENN7F73VQA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.866335992 (48h6m10.417806453s) [noise=true , address=45.91.116.11:4117 , piece_id=UOEFPOPOFIO5KXY2JOX54YHUWNMTCAFMXEBKAMMR4GSR5ZE6J6PA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.932922087 (48h6m10.484392538s) [noise=true , address=77.90.31.19:4841 , piece_id=6JVB2WIEA2STMNBVTM6D7IYFHQY4QJGXAYHJAQILMYZZDH2MWIVA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:12.005905333 (48h6m10.557375794s) [noise=true , address=45.91.116.4:4939 , piece_id=KMHEQSQTA2WEWMIO7JDQZUDLSRV4CA6OJUN2CYYUSA7SU2ZDQBCQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:16.703757864 (48h6m15.255228326s) [noise=true , address=145.79.227.9:4366 , piece_id=JDBR4PTBZ5RN6NTWPERVLFANUAZQF3FSGIENVRHSYN56ED5NT2AQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:17.540298352 (48h6m16.091768813s) [noise=true , address=145.79.227.21:4474 , piece_id=RNB5T6DIUCY7IV2ULLO2UJRDKUJXI34UM4ZCHPJB22DNQ47VZJVA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:19.805526866 (48h6m18.356997337s) [noise=true , address=145.79.227.27:4689 , piece_id=KLTJYVDWHEE2BGLUSP2XT6L5R2C5OB7TF4DZLYBF6R5OG3N6AEFQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:24.397542313 (48h6m22.949012764s) [noise=true , address=145.79.227.17:4970 , piece_id=F2227LWTX5PRCQIUEJNRJ3QS3GRJOIEJIZFV3KRF7BAEAPSQZR7A, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:24.779686050 (48h6m23.331156511s) [noise=true , address=145.79.227.14:4840 , piece_id=QASWXZB3E6E7VYGF24KZ37Q3LKT6F4WVFKTBV3YS73KYAWTBV7YQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:25.298168925 (48h6m23.849639387s) [noise=true , address=145.79.227.11:4729 , piece_id=CDIU7FFPCVQGDK3K5ZDUARJ6VGB7KIY2DEVL7ZRLW5EASIFEHKHA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:35.044925402 (48h6m33.596395853s) [noise=true , address=145.79.227.30:4761 , piece_id=ASO5O5LM6CEF6DXGBVOCI3NOCWLGKDU2T5JHOL5ZPWEXMTRMKZIA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:37.306959937 (48h6m35.858430398s) [noise=true , address=145.79.227.20:4437 , piece_id=AOK6MTRZKPCXTNHZWGBYJGRBYNHFWO7GSG2PT4DFSMDAWK75GY4A, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:37.306987550 (48h6m35.858458011s) [seg_pos={0 0}, upload=42408]: Segment reached optimal threshold of 13 pieces. Nov 11 08:10:37.307209231 (48h6m35.858679702s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4): protocol: expected piece hash; context canceled; EOF Nov 11 08:10:37.307218759 (48h6m35.85868922s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307368936 (48h6m35.858839397s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw): protocol: expected piece hash; context canceled; EOF Nov 11 08:10:37.307376741 (48h6m35.858847202s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307561282 (48h6m35.859031743s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9): protocol: write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection Nov 11 08:10:37.307585969 (48h6m35.85905643s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307629221 (48h6m35.859099682s) [seg_pos={0 0}, upload=42408]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 11 08:10:37.307741005 (48h6m35.859211466s) [upload=42408]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 11 08:10:37.307750052 (48h6m35.859220513s) [upload=42408]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 11 08:10:37.612339386 (48h6m36.163809848s) [upload=42408]: Done uploading segments... Nov 11 08:10:37.612366317 (48h6m36.163836788s) [upload=42408]: Upload finished. err: <nil> Duration: 5.696473151s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [.......X.......] duration: 80.241259ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/77081cc731df4257 FAIL (2025-11-11 08:25:15 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:50034->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 80.272167ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 5.654448491s trace: https://jaeger.dev.storj.io/trace/49621e5e6beafe8e FAIL (2025-11-11 08:24:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 11 08:10:09.433180409 (48h6m7.98465087s) [upload=42408]: Starting upload Nov 11 08:10:09.433429884 (48h6m7.984900335s) [upload=42408]: Done starting upload Nov 11 08:10:09.433567967 (48h6m7.985038438s) [upload=42408]: Uploading segments... Nov 11 08:10:09.434191213 (48h6m7.985661674s) [upload=42408]: Got next segment. Inline: false Nov 11 08:10:09.434274032 (48h6m7.985744493s) [upload=42408]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 08:10:09.434282217 (48h6m7.985752688s) [upload=42408]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 08:10:09.714872160 (48h6m8.266342621s) [seg_pos={0 0}, upload=42408]: Begin upload segment... Nov 11 08:10:09.715084585 (48h6m8.266555046s) [seg_pos={0 0}, upload=42408]: Done begin upload segment. Nov 11 08:10:09.715464638 (48h6m8.2669351s) [noise=true , address=45.91.116.8:4047 , piece_id=VP6VW2ZTNSKS35R47GGJZI4YSHNNFUFB4YQ3W3QFT5ENN7F73VQA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.715555221 (48h6m8.267025692s) [noise=true , address=145.79.227.30:4761 , piece_id=ASO5O5LM6CEF6DXGBVOCI3NOCWLGKDU2T5JHOL5ZPWEXMTRMKZIA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.718085410 (48h6m8.269555901s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.718721901 (48h6m8.270192402s) [noise=true , address=145.79.227.11:4729 , piece_id=CDIU7FFPCVQGDK3K5ZDUARJ6VGB7KIY2DEVL7ZRLW5EASIFEHKHA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.719505724 (48h6m8.270976195s) [noise=true , address=145.79.227.27:4689 , piece_id=KLTJYVDWHEE2BGLUSP2XT6L5R2C5OB7TF4DZLYBF6R5OG3N6AEFQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.720136295 (48h6m8.271606756s) [noise=true , address=145.79.227.14:4840 , piece_id=QASWXZB3E6E7VYGF24KZ37Q3LKT6F4WVFKTBV3YS73KYAWTBV7YQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.721166056 (48h6m8.272636517s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.721883453 (48h6m8.273353994s) [noise=true , address=45.91.116.4:4939 , piece_id=KMHEQSQTA2WEWMIO7JDQZUDLSRV4CA6OJUN2CYYUSA7SU2ZDQBCQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.722398593 (48h6m8.273869054s) [noise=true , address=77.90.31.18:4806 , piece_id=LNL5LRHH3OOELKEO4HND2K3GNVDPCBCGKBNJ75DCLEKRAOMTQLKA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.724101306 (48h6m8.275571767s) [noise=true , address=145.79.227.17:4970 , piece_id=F2227LWTX5PRCQIUEJNRJ3QS3GRJOIEJIZFV3KRF7BAEAPSQZR7A, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.726385264 (48h6m8.277855776s) [noise=true , address=45.91.116.11:4117 , piece_id=UOEFPOPOFIO5KXY2JOX54YHUWNMTCAFMXEBKAMMR4GSR5ZE6J6PA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.728323225 (48h6m8.279793686s) [noise=true , address=145.79.227.20:4437 , piece_id=AOK6MTRZKPCXTNHZWGBYJGRBYNHFWO7GSG2PT4DFSMDAWK75GY4A, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.730321541 (48h6m8.281792002s) [noise=true , address=145.79.227.9:4366 , piece_id=JDBR4PTBZ5RN6NTWPERVLFANUAZQF3FSGIENVRHSYN56ED5NT2AQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.731324300 (48h6m8.282794832s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.732226989 (48h6m8.28369749s) [noise=true , address=77.90.31.19:4841 , piece_id=6JVB2WIEA2STMNBVTM6D7IYFHQY4QJGXAYHJAQILMYZZDH2MWIVA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.733026101 (48h6m8.284496562s) [noise=true , address=145.79.227.21:4474 , piece_id=RNB5T6DIUCY7IV2ULLO2UJRDKUJXI34UM4ZCHPJB22DNQ47VZJVA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:11.398415378 (48h6m9.949885839s) [upload=42408]: Next returned nil segment Nov 11 08:10:11.398434043 (48h6m9.949904504s) [upload=42408]: Waiting for error group managing segments... Nov 11 08:10:11.796154582 (48h6m10.347625043s) [noise=true , address=77.90.31.18:4806 , piece_id=LNL5LRHH3OOELKEO4HND2K3GNVDPCBCGKBNJ75DCLEKRAOMTQLKA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.809043214 (48h6m10.360513675s) [noise=true , address=45.91.116.8:4047 , piece_id=VP6VW2ZTNSKS35R47GGJZI4YSHNNFUFB4YQ3W3QFT5ENN7F73VQA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.866335992 (48h6m10.417806453s) [noise=true , address=45.91.116.11:4117 , piece_id=UOEFPOPOFIO5KXY2JOX54YHUWNMTCAFMXEBKAMMR4GSR5ZE6J6PA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.932922087 (48h6m10.484392538s) [noise=true , address=77.90.31.19:4841 , piece_id=6JVB2WIEA2STMNBVTM6D7IYFHQY4QJGXAYHJAQILMYZZDH2MWIVA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:12.005905333 (48h6m10.557375794s) [noise=true , address=45.91.116.4:4939 , piece_id=KMHEQSQTA2WEWMIO7JDQZUDLSRV4CA6OJUN2CYYUSA7SU2ZDQBCQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:16.703757864 (48h6m15.255228326s) [noise=true , address=145.79.227.9:4366 , piece_id=JDBR4PTBZ5RN6NTWPERVLFANUAZQF3FSGIENVRHSYN56ED5NT2AQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:17.540298352 (48h6m16.091768813s) [noise=true , address=145.79.227.21:4474 , piece_id=RNB5T6DIUCY7IV2ULLO2UJRDKUJXI34UM4ZCHPJB22DNQ47VZJVA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:19.805526866 (48h6m18.356997337s) [noise=true , address=145.79.227.27:4689 , piece_id=KLTJYVDWHEE2BGLUSP2XT6L5R2C5OB7TF4DZLYBF6R5OG3N6AEFQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:24.397542313 (48h6m22.949012764s) [noise=true , address=145.79.227.17:4970 , piece_id=F2227LWTX5PRCQIUEJNRJ3QS3GRJOIEJIZFV3KRF7BAEAPSQZR7A, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:24.779686050 (48h6m23.331156511s) [noise=true , address=145.79.227.14:4840 , piece_id=QASWXZB3E6E7VYGF24KZ37Q3LKT6F4WVFKTBV3YS73KYAWTBV7YQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:25.298168925 (48h6m23.849639387s) [noise=true , address=145.79.227.11:4729 , piece_id=CDIU7FFPCVQGDK3K5ZDUARJ6VGB7KIY2DEVL7ZRLW5EASIFEHKHA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:35.044925402 (48h6m33.596395853s) [noise=true , address=145.79.227.30:4761 , piece_id=ASO5O5LM6CEF6DXGBVOCI3NOCWLGKDU2T5JHOL5ZPWEXMTRMKZIA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:37.306959937 (48h6m35.858430398s) [noise=true , address=145.79.227.20:4437 , piece_id=AOK6MTRZKPCXTNHZWGBYJGRBYNHFWO7GSG2PT4DFSMDAWK75GY4A, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:37.306987550 (48h6m35.858458011s) [seg_pos={0 0}, upload=42408]: Segment reached optimal threshold of 13 pieces. Nov 11 08:10:37.307209231 (48h6m35.858679702s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4): protocol: expected piece hash; context canceled; EOF Nov 11 08:10:37.307218759 (48h6m35.85868922s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307368936 (48h6m35.858839397s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw): protocol: expected piece hash; context canceled; EOF Nov 11 08:10:37.307376741 (48h6m35.858847202s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307561282 (48h6m35.859031743s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9): protocol: write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection Nov 11 08:10:37.307585969 (48h6m35.85905643s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307629221 (48h6m35.859099682s) [seg_pos={0 0}, upload=42408]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 11 08:10:37.307741005 (48h6m35.859211466s) [upload=42408]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 11 08:10:37.307750052 (48h6m35.859220513s) [upload=42408]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 11 08:10:37.612339386 (48h6m36.163809848s) [upload=42408]: Done uploading segments... Nov 11 08:10:37.612366317 (48h6m36.163836788s) [upload=42408]: Upload finished. err: <nil> Duration: 5.654482605s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [........X......] duration: 65.511898ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/77081cc731df4257 FAIL (2025-11-11 08:24:06 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:50034->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 65.544101ms core website
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [.........X.....] duration: 85.402067ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/77081cc731df4257 FAIL (2025-11-11 08:23:50 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:50034->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 85.429449ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 5.808948572s trace: https://jaeger.dev.storj.io/trace/49621e5e6beafe8e FAIL (2025-11-11 08:23:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 11 08:10:09.433180409 (48h6m7.98465087s) [upload=42408]: Starting upload Nov 11 08:10:09.433429884 (48h6m7.984900335s) [upload=42408]: Done starting upload Nov 11 08:10:09.433567967 (48h6m7.985038438s) [upload=42408]: Uploading segments... Nov 11 08:10:09.434191213 (48h6m7.985661674s) [upload=42408]: Got next segment. Inline: false Nov 11 08:10:09.434274032 (48h6m7.985744493s) [upload=42408]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 08:10:09.434282217 (48h6m7.985752688s) [upload=42408]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 08:10:09.714872160 (48h6m8.266342621s) [seg_pos={0 0}, upload=42408]: Begin upload segment... Nov 11 08:10:09.715084585 (48h6m8.266555046s) [seg_pos={0 0}, upload=42408]: Done begin upload segment. Nov 11 08:10:09.715464638 (48h6m8.2669351s) [noise=true , address=45.91.116.8:4047 , piece_id=VP6VW2ZTNSKS35R47GGJZI4YSHNNFUFB4YQ3W3QFT5ENN7F73VQA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.715555221 (48h6m8.267025692s) [noise=true , address=145.79.227.30:4761 , piece_id=ASO5O5LM6CEF6DXGBVOCI3NOCWLGKDU2T5JHOL5ZPWEXMTRMKZIA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.718085410 (48h6m8.269555901s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.718721901 (48h6m8.270192402s) [noise=true , address=145.79.227.11:4729 , piece_id=CDIU7FFPCVQGDK3K5ZDUARJ6VGB7KIY2DEVL7ZRLW5EASIFEHKHA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.719505724 (48h6m8.270976195s) [noise=true , address=145.79.227.27:4689 , piece_id=KLTJYVDWHEE2BGLUSP2XT6L5R2C5OB7TF4DZLYBF6R5OG3N6AEFQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.720136295 (48h6m8.271606756s) [noise=true , address=145.79.227.14:4840 , piece_id=QASWXZB3E6E7VYGF24KZ37Q3LKT6F4WVFKTBV3YS73KYAWTBV7YQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.721166056 (48h6m8.272636517s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.721883453 (48h6m8.273353994s) [noise=true , address=45.91.116.4:4939 , piece_id=KMHEQSQTA2WEWMIO7JDQZUDLSRV4CA6OJUN2CYYUSA7SU2ZDQBCQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.722398593 (48h6m8.273869054s) [noise=true , address=77.90.31.18:4806 , piece_id=LNL5LRHH3OOELKEO4HND2K3GNVDPCBCGKBNJ75DCLEKRAOMTQLKA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.724101306 (48h6m8.275571767s) [noise=true , address=145.79.227.17:4970 , piece_id=F2227LWTX5PRCQIUEJNRJ3QS3GRJOIEJIZFV3KRF7BAEAPSQZR7A, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.726385264 (48h6m8.277855776s) [noise=true , address=45.91.116.11:4117 , piece_id=UOEFPOPOFIO5KXY2JOX54YHUWNMTCAFMXEBKAMMR4GSR5ZE6J6PA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.728323225 (48h6m8.279793686s) [noise=true , address=145.79.227.20:4437 , piece_id=AOK6MTRZKPCXTNHZWGBYJGRBYNHFWO7GSG2PT4DFSMDAWK75GY4A, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.730321541 (48h6m8.281792002s) [noise=true , address=145.79.227.9:4366 , piece_id=JDBR4PTBZ5RN6NTWPERVLFANUAZQF3FSGIENVRHSYN56ED5NT2AQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.731324300 (48h6m8.282794832s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.732226989 (48h6m8.28369749s) [noise=true , address=77.90.31.19:4841 , piece_id=6JVB2WIEA2STMNBVTM6D7IYFHQY4QJGXAYHJAQILMYZZDH2MWIVA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.733026101 (48h6m8.284496562s) [noise=true , address=145.79.227.21:4474 , piece_id=RNB5T6DIUCY7IV2ULLO2UJRDKUJXI34UM4ZCHPJB22DNQ47VZJVA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:11.398415378 (48h6m9.949885839s) [upload=42408]: Next returned nil segment Nov 11 08:10:11.398434043 (48h6m9.949904504s) [upload=42408]: Waiting for error group managing segments... Nov 11 08:10:11.796154582 (48h6m10.347625043s) [noise=true , address=77.90.31.18:4806 , piece_id=LNL5LRHH3OOELKEO4HND2K3GNVDPCBCGKBNJ75DCLEKRAOMTQLKA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.809043214 (48h6m10.360513675s) [noise=true , address=45.91.116.8:4047 , piece_id=VP6VW2ZTNSKS35R47GGJZI4YSHNNFUFB4YQ3W3QFT5ENN7F73VQA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.866335992 (48h6m10.417806453s) [noise=true , address=45.91.116.11:4117 , piece_id=UOEFPOPOFIO5KXY2JOX54YHUWNMTCAFMXEBKAMMR4GSR5ZE6J6PA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.932922087 (48h6m10.484392538s) [noise=true , address=77.90.31.19:4841 , piece_id=6JVB2WIEA2STMNBVTM6D7IYFHQY4QJGXAYHJAQILMYZZDH2MWIVA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:12.005905333 (48h6m10.557375794s) [noise=true , address=45.91.116.4:4939 , piece_id=KMHEQSQTA2WEWMIO7JDQZUDLSRV4CA6OJUN2CYYUSA7SU2ZDQBCQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:16.703757864 (48h6m15.255228326s) [noise=true , address=145.79.227.9:4366 , piece_id=JDBR4PTBZ5RN6NTWPERVLFANUAZQF3FSGIENVRHSYN56ED5NT2AQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:17.540298352 (48h6m16.091768813s) [noise=true , address=145.79.227.21:4474 , piece_id=RNB5T6DIUCY7IV2ULLO2UJRDKUJXI34UM4ZCHPJB22DNQ47VZJVA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:19.805526866 (48h6m18.356997337s) [noise=true , address=145.79.227.27:4689 , piece_id=KLTJYVDWHEE2BGLUSP2XT6L5R2C5OB7TF4DZLYBF6R5OG3N6AEFQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:24.397542313 (48h6m22.949012764s) [noise=true , address=145.79.227.17:4970 , piece_id=F2227LWTX5PRCQIUEJNRJ3QS3GRJOIEJIZFV3KRF7BAEAPSQZR7A, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:24.779686050 (48h6m23.331156511s) [noise=true , address=145.79.227.14:4840 , piece_id=QASWXZB3E6E7VYGF24KZ37Q3LKT6F4WVFKTBV3YS73KYAWTBV7YQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:25.298168925 (48h6m23.849639387s) [noise=true , address=145.79.227.11:4729 , piece_id=CDIU7FFPCVQGDK3K5ZDUARJ6VGB7KIY2DEVL7ZRLW5EASIFEHKHA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:35.044925402 (48h6m33.596395853s) [noise=true , address=145.79.227.30:4761 , piece_id=ASO5O5LM6CEF6DXGBVOCI3NOCWLGKDU2T5JHOL5ZPWEXMTRMKZIA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:37.306959937 (48h6m35.858430398s) [noise=true , address=145.79.227.20:4437 , piece_id=AOK6MTRZKPCXTNHZWGBYJGRBYNHFWO7GSG2PT4DFSMDAWK75GY4A, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:37.306987550 (48h6m35.858458011s) [seg_pos={0 0}, upload=42408]: Segment reached optimal threshold of 13 pieces. Nov 11 08:10:37.307209231 (48h6m35.858679702s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4): protocol: expected piece hash; context canceled; EOF Nov 11 08:10:37.307218759 (48h6m35.85868922s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307368936 (48h6m35.858839397s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw): protocol: expected piece hash; context canceled; EOF Nov 11 08:10:37.307376741 (48h6m35.858847202s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307561282 (48h6m35.859031743s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9): protocol: write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection Nov 11 08:10:37.307585969 (48h6m35.85905643s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307629221 (48h6m35.859099682s) [seg_pos={0 0}, upload=42408]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 11 08:10:37.307741005 (48h6m35.859211466s) [upload=42408]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 11 08:10:37.307750052 (48h6m35.859220513s) [upload=42408]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 11 08:10:37.612339386 (48h6m36.163809848s) [upload=42408]: Done uploading segments... Nov 11 08:10:37.612366317 (48h6m36.163836788s) [upload=42408]: Upload finished. err: <nil> Duration: 5.808987886s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [..........X....] duration: 51.238255ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/77081cc731df4257 FAIL (2025-11-11 08:23:19 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:50034->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 51.265759ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 5.904890919s trace: https://jaeger.dev.storj.io/trace/49621e5e6beafe8e FAIL (2025-11-11 08:23:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 11 08:10:09.433180409 (48h6m7.98465087s) [upload=42408]: Starting upload Nov 11 08:10:09.433429884 (48h6m7.984900335s) [upload=42408]: Done starting upload Nov 11 08:10:09.433567967 (48h6m7.985038438s) [upload=42408]: Uploading segments... Nov 11 08:10:09.434191213 (48h6m7.985661674s) [upload=42408]: Got next segment. Inline: false Nov 11 08:10:09.434274032 (48h6m7.985744493s) [upload=42408]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 08:10:09.434282217 (48h6m7.985752688s) [upload=42408]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 08:10:09.714872160 (48h6m8.266342621s) [seg_pos={0 0}, upload=42408]: Begin upload segment... Nov 11 08:10:09.715084585 (48h6m8.266555046s) [seg_pos={0 0}, upload=42408]: Done begin upload segment. Nov 11 08:10:09.715464638 (48h6m8.2669351s) [noise=true , address=45.91.116.8:4047 , piece_id=VP6VW2ZTNSKS35R47GGJZI4YSHNNFUFB4YQ3W3QFT5ENN7F73VQA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.715555221 (48h6m8.267025692s) [noise=true , address=145.79.227.30:4761 , piece_id=ASO5O5LM6CEF6DXGBVOCI3NOCWLGKDU2T5JHOL5ZPWEXMTRMKZIA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.718085410 (48h6m8.269555901s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.718721901 (48h6m8.270192402s) [noise=true , address=145.79.227.11:4729 , piece_id=CDIU7FFPCVQGDK3K5ZDUARJ6VGB7KIY2DEVL7ZRLW5EASIFEHKHA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.719505724 (48h6m8.270976195s) [noise=true , address=145.79.227.27:4689 , piece_id=KLTJYVDWHEE2BGLUSP2XT6L5R2C5OB7TF4DZLYBF6R5OG3N6AEFQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.720136295 (48h6m8.271606756s) [noise=true , address=145.79.227.14:4840 , piece_id=QASWXZB3E6E7VYGF24KZ37Q3LKT6F4WVFKTBV3YS73KYAWTBV7YQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.721166056 (48h6m8.272636517s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.721883453 (48h6m8.273353994s) [noise=true , address=45.91.116.4:4939 , piece_id=KMHEQSQTA2WEWMIO7JDQZUDLSRV4CA6OJUN2CYYUSA7SU2ZDQBCQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.722398593 (48h6m8.273869054s) [noise=true , address=77.90.31.18:4806 , piece_id=LNL5LRHH3OOELKEO4HND2K3GNVDPCBCGKBNJ75DCLEKRAOMTQLKA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.724101306 (48h6m8.275571767s) [noise=true , address=145.79.227.17:4970 , piece_id=F2227LWTX5PRCQIUEJNRJ3QS3GRJOIEJIZFV3KRF7BAEAPSQZR7A, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.726385264 (48h6m8.277855776s) [noise=true , address=45.91.116.11:4117 , piece_id=UOEFPOPOFIO5KXY2JOX54YHUWNMTCAFMXEBKAMMR4GSR5ZE6J6PA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.728323225 (48h6m8.279793686s) [noise=true , address=145.79.227.20:4437 , piece_id=AOK6MTRZKPCXTNHZWGBYJGRBYNHFWO7GSG2PT4DFSMDAWK75GY4A, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.730321541 (48h6m8.281792002s) [noise=true , address=145.79.227.9:4366 , piece_id=JDBR4PTBZ5RN6NTWPERVLFANUAZQF3FSGIENVRHSYN56ED5NT2AQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.731324300 (48h6m8.282794832s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.732226989 (48h6m8.28369749s) [noise=true , address=77.90.31.19:4841 , piece_id=6JVB2WIEA2STMNBVTM6D7IYFHQY4QJGXAYHJAQILMYZZDH2MWIVA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.733026101 (48h6m8.284496562s) [noise=true , address=145.79.227.21:4474 , piece_id=RNB5T6DIUCY7IV2ULLO2UJRDKUJXI34UM4ZCHPJB22DNQ47VZJVA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:11.398415378 (48h6m9.949885839s) [upload=42408]: Next returned nil segment Nov 11 08:10:11.398434043 (48h6m9.949904504s) [upload=42408]: Waiting for error group managing segments... Nov 11 08:10:11.796154582 (48h6m10.347625043s) [noise=true , address=77.90.31.18:4806 , piece_id=LNL5LRHH3OOELKEO4HND2K3GNVDPCBCGKBNJ75DCLEKRAOMTQLKA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.809043214 (48h6m10.360513675s) [noise=true , address=45.91.116.8:4047 , piece_id=VP6VW2ZTNSKS35R47GGJZI4YSHNNFUFB4YQ3W3QFT5ENN7F73VQA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.866335992 (48h6m10.417806453s) [noise=true , address=45.91.116.11:4117 , piece_id=UOEFPOPOFIO5KXY2JOX54YHUWNMTCAFMXEBKAMMR4GSR5ZE6J6PA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.932922087 (48h6m10.484392538s) [noise=true , address=77.90.31.19:4841 , piece_id=6JVB2WIEA2STMNBVTM6D7IYFHQY4QJGXAYHJAQILMYZZDH2MWIVA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:12.005905333 (48h6m10.557375794s) [noise=true , address=45.91.116.4:4939 , piece_id=KMHEQSQTA2WEWMIO7JDQZUDLSRV4CA6OJUN2CYYUSA7SU2ZDQBCQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:16.703757864 (48h6m15.255228326s) [noise=true , address=145.79.227.9:4366 , piece_id=JDBR4PTBZ5RN6NTWPERVLFANUAZQF3FSGIENVRHSYN56ED5NT2AQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:17.540298352 (48h6m16.091768813s) [noise=true , address=145.79.227.21:4474 , piece_id=RNB5T6DIUCY7IV2ULLO2UJRDKUJXI34UM4ZCHPJB22DNQ47VZJVA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:19.805526866 (48h6m18.356997337s) [noise=true , address=145.79.227.27:4689 , piece_id=KLTJYVDWHEE2BGLUSP2XT6L5R2C5OB7TF4DZLYBF6R5OG3N6AEFQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:24.397542313 (48h6m22.949012764s) [noise=true , address=145.79.227.17:4970 , piece_id=F2227LWTX5PRCQIUEJNRJ3QS3GRJOIEJIZFV3KRF7BAEAPSQZR7A, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:24.779686050 (48h6m23.331156511s) [noise=true , address=145.79.227.14:4840 , piece_id=QASWXZB3E6E7VYGF24KZ37Q3LKT6F4WVFKTBV3YS73KYAWTBV7YQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:25.298168925 (48h6m23.849639387s) [noise=true , address=145.79.227.11:4729 , piece_id=CDIU7FFPCVQGDK3K5ZDUARJ6VGB7KIY2DEVL7ZRLW5EASIFEHKHA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:35.044925402 (48h6m33.596395853s) [noise=true , address=145.79.227.30:4761 , piece_id=ASO5O5LM6CEF6DXGBVOCI3NOCWLGKDU2T5JHOL5ZPWEXMTRMKZIA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:37.306959937 (48h6m35.858430398s) [noise=true , address=145.79.227.20:4437 , piece_id=AOK6MTRZKPCXTNHZWGBYJGRBYNHFWO7GSG2PT4DFSMDAWK75GY4A, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:37.306987550 (48h6m35.858458011s) [seg_pos={0 0}, upload=42408]: Segment reached optimal threshold of 13 pieces. Nov 11 08:10:37.307209231 (48h6m35.858679702s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4): protocol: expected piece hash; context canceled; EOF Nov 11 08:10:37.307218759 (48h6m35.85868922s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307368936 (48h6m35.858839397s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw): protocol: expected piece hash; context canceled; EOF Nov 11 08:10:37.307376741 (48h6m35.858847202s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307561282 (48h6m35.859031743s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9): protocol: write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection Nov 11 08:10:37.307585969 (48h6m35.85905643s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307629221 (48h6m35.859099682s) [seg_pos={0 0}, upload=42408]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 11 08:10:37.307741005 (48h6m35.859211466s) [upload=42408]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 11 08:10:37.307750052 (48h6m35.859220513s) [upload=42408]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 11 08:10:37.612339386 (48h6m36.163809848s) [upload=42408]: Done uploading segments... Nov 11 08:10:37.612366317 (48h6m36.163836788s) [upload=42408]: Upload finished. err: <nil> Duration: 5.904918802s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [...........X...] duration: 52.122951ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/77081cc731df4257 FAIL (2025-11-11 08:22:30 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:50034->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 52.223131ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 5.819977107s trace: https://jaeger.dev.storj.io/trace/49621e5e6beafe8e FAIL (2025-11-11 08:22:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 11 08:10:09.433180409 (48h6m7.98465087s) [upload=42408]: Starting upload Nov 11 08:10:09.433429884 (48h6m7.984900335s) [upload=42408]: Done starting upload Nov 11 08:10:09.433567967 (48h6m7.985038438s) [upload=42408]: Uploading segments... Nov 11 08:10:09.434191213 (48h6m7.985661674s) [upload=42408]: Got next segment. Inline: false Nov 11 08:10:09.434274032 (48h6m7.985744493s) [upload=42408]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 08:10:09.434282217 (48h6m7.985752688s) [upload=42408]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 08:10:09.714872160 (48h6m8.266342621s) [seg_pos={0 0}, upload=42408]: Begin upload segment... Nov 11 08:10:09.715084585 (48h6m8.266555046s) [seg_pos={0 0}, upload=42408]: Done begin upload segment. Nov 11 08:10:09.715464638 (48h6m8.2669351s) [noise=true , address=45.91.116.8:4047 , piece_id=VP6VW2ZTNSKS35R47GGJZI4YSHNNFUFB4YQ3W3QFT5ENN7F73VQA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.715555221 (48h6m8.267025692s) [noise=true , address=145.79.227.30:4761 , piece_id=ASO5O5LM6CEF6DXGBVOCI3NOCWLGKDU2T5JHOL5ZPWEXMTRMKZIA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.718085410 (48h6m8.269555901s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.718721901 (48h6m8.270192402s) [noise=true , address=145.79.227.11:4729 , piece_id=CDIU7FFPCVQGDK3K5ZDUARJ6VGB7KIY2DEVL7ZRLW5EASIFEHKHA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.719505724 (48h6m8.270976195s) [noise=true , address=145.79.227.27:4689 , piece_id=KLTJYVDWHEE2BGLUSP2XT6L5R2C5OB7TF4DZLYBF6R5OG3N6AEFQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.720136295 (48h6m8.271606756s) [noise=true , address=145.79.227.14:4840 , piece_id=QASWXZB3E6E7VYGF24KZ37Q3LKT6F4WVFKTBV3YS73KYAWTBV7YQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.721166056 (48h6m8.272636517s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.721883453 (48h6m8.273353994s) [noise=true , address=45.91.116.4:4939 , piece_id=KMHEQSQTA2WEWMIO7JDQZUDLSRV4CA6OJUN2CYYUSA7SU2ZDQBCQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.722398593 (48h6m8.273869054s) [noise=true , address=77.90.31.18:4806 , piece_id=LNL5LRHH3OOELKEO4HND2K3GNVDPCBCGKBNJ75DCLEKRAOMTQLKA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.724101306 (48h6m8.275571767s) [noise=true , address=145.79.227.17:4970 , piece_id=F2227LWTX5PRCQIUEJNRJ3QS3GRJOIEJIZFV3KRF7BAEAPSQZR7A, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.726385264 (48h6m8.277855776s) [noise=true , address=45.91.116.11:4117 , piece_id=UOEFPOPOFIO5KXY2JOX54YHUWNMTCAFMXEBKAMMR4GSR5ZE6J6PA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.728323225 (48h6m8.279793686s) [noise=true , address=145.79.227.20:4437 , piece_id=AOK6MTRZKPCXTNHZWGBYJGRBYNHFWO7GSG2PT4DFSMDAWK75GY4A, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.730321541 (48h6m8.281792002s) [noise=true , address=145.79.227.9:4366 , piece_id=JDBR4PTBZ5RN6NTWPERVLFANUAZQF3FSGIENVRHSYN56ED5NT2AQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.731324300 (48h6m8.282794832s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.732226989 (48h6m8.28369749s) [noise=true , address=77.90.31.19:4841 , piece_id=6JVB2WIEA2STMNBVTM6D7IYFHQY4QJGXAYHJAQILMYZZDH2MWIVA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.733026101 (48h6m8.284496562s) [noise=true , address=145.79.227.21:4474 , piece_id=RNB5T6DIUCY7IV2ULLO2UJRDKUJXI34UM4ZCHPJB22DNQ47VZJVA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:11.398415378 (48h6m9.949885839s) [upload=42408]: Next returned nil segment Nov 11 08:10:11.398434043 (48h6m9.949904504s) [upload=42408]: Waiting for error group managing segments... Nov 11 08:10:11.796154582 (48h6m10.347625043s) [noise=true , address=77.90.31.18:4806 , piece_id=LNL5LRHH3OOELKEO4HND2K3GNVDPCBCGKBNJ75DCLEKRAOMTQLKA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.809043214 (48h6m10.360513675s) [noise=true , address=45.91.116.8:4047 , piece_id=VP6VW2ZTNSKS35R47GGJZI4YSHNNFUFB4YQ3W3QFT5ENN7F73VQA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.866335992 (48h6m10.417806453s) [noise=true , address=45.91.116.11:4117 , piece_id=UOEFPOPOFIO5KXY2JOX54YHUWNMTCAFMXEBKAMMR4GSR5ZE6J6PA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.932922087 (48h6m10.484392538s) [noise=true , address=77.90.31.19:4841 , piece_id=6JVB2WIEA2STMNBVTM6D7IYFHQY4QJGXAYHJAQILMYZZDH2MWIVA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:12.005905333 (48h6m10.557375794s) [noise=true , address=45.91.116.4:4939 , piece_id=KMHEQSQTA2WEWMIO7JDQZUDLSRV4CA6OJUN2CYYUSA7SU2ZDQBCQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:16.703757864 (48h6m15.255228326s) [noise=true , address=145.79.227.9:4366 , piece_id=JDBR4PTBZ5RN6NTWPERVLFANUAZQF3FSGIENVRHSYN56ED5NT2AQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:17.540298352 (48h6m16.091768813s) [noise=true , address=145.79.227.21:4474 , piece_id=RNB5T6DIUCY7IV2ULLO2UJRDKUJXI34UM4ZCHPJB22DNQ47VZJVA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:19.805526866 (48h6m18.356997337s) [noise=true , address=145.79.227.27:4689 , piece_id=KLTJYVDWHEE2BGLUSP2XT6L5R2C5OB7TF4DZLYBF6R5OG3N6AEFQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:24.397542313 (48h6m22.949012764s) [noise=true , address=145.79.227.17:4970 , piece_id=F2227LWTX5PRCQIUEJNRJ3QS3GRJOIEJIZFV3KRF7BAEAPSQZR7A, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:24.779686050 (48h6m23.331156511s) [noise=true , address=145.79.227.14:4840 , piece_id=QASWXZB3E6E7VYGF24KZ37Q3LKT6F4WVFKTBV3YS73KYAWTBV7YQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:25.298168925 (48h6m23.849639387s) [noise=true , address=145.79.227.11:4729 , piece_id=CDIU7FFPCVQGDK3K5ZDUARJ6VGB7KIY2DEVL7ZRLW5EASIFEHKHA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:35.044925402 (48h6m33.596395853s) [noise=true , address=145.79.227.30:4761 , piece_id=ASO5O5LM6CEF6DXGBVOCI3NOCWLGKDU2T5JHOL5ZPWEXMTRMKZIA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:37.306959937 (48h6m35.858430398s) [noise=true , address=145.79.227.20:4437 , piece_id=AOK6MTRZKPCXTNHZWGBYJGRBYNHFWO7GSG2PT4DFSMDAWK75GY4A, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:37.306987550 (48h6m35.858458011s) [seg_pos={0 0}, upload=42408]: Segment reached optimal threshold of 13 pieces. Nov 11 08:10:37.307209231 (48h6m35.858679702s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4): protocol: expected piece hash; context canceled; EOF Nov 11 08:10:37.307218759 (48h6m35.85868922s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307368936 (48h6m35.858839397s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw): protocol: expected piece hash; context canceled; EOF Nov 11 08:10:37.307376741 (48h6m35.858847202s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307561282 (48h6m35.859031743s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9): protocol: write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection Nov 11 08:10:37.307585969 (48h6m35.85905643s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307629221 (48h6m35.859099682s) [seg_pos={0 0}, upload=42408]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 11 08:10:37.307741005 (48h6m35.859211466s) [upload=42408]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 11 08:10:37.307750052 (48h6m35.859220513s) [upload=42408]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 11 08:10:37.612339386 (48h6m36.163809848s) [upload=42408]: Done uploading segments... Nov 11 08:10:37.612366317 (48h6m36.163836788s) [upload=42408]: Upload finished. err: <nil> Duration: 5.820007355s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [............X..] duration: 57.716022ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/77081cc731df4257 FAIL (2025-11-11 08:21:42 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:50034->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 57.748543ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 6.319391068s trace: https://jaeger.dev.storj.io/trace/49621e5e6beafe8e FAIL (2025-11-11 08:21:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 11 08:10:09.433180409 (48h6m7.98465087s) [upload=42408]: Starting upload Nov 11 08:10:09.433429884 (48h6m7.984900335s) [upload=42408]: Done starting upload Nov 11 08:10:09.433567967 (48h6m7.985038438s) [upload=42408]: Uploading segments... Nov 11 08:10:09.434191213 (48h6m7.985661674s) [upload=42408]: Got next segment. Inline: false Nov 11 08:10:09.434274032 (48h6m7.985744493s) [upload=42408]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 08:10:09.434282217 (48h6m7.985752688s) [upload=42408]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 08:10:09.714872160 (48h6m8.266342621s) [seg_pos={0 0}, upload=42408]: Begin upload segment... Nov 11 08:10:09.715084585 (48h6m8.266555046s) [seg_pos={0 0}, upload=42408]: Done begin upload segment. Nov 11 08:10:09.715464638 (48h6m8.2669351s) [noise=true , address=45.91.116.8:4047 , piece_id=VP6VW2ZTNSKS35R47GGJZI4YSHNNFUFB4YQ3W3QFT5ENN7F73VQA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.715555221 (48h6m8.267025692s) [noise=true , address=145.79.227.30:4761 , piece_id=ASO5O5LM6CEF6DXGBVOCI3NOCWLGKDU2T5JHOL5ZPWEXMTRMKZIA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.718085410 (48h6m8.269555901s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.718721901 (48h6m8.270192402s) [noise=true , address=145.79.227.11:4729 , piece_id=CDIU7FFPCVQGDK3K5ZDUARJ6VGB7KIY2DEVL7ZRLW5EASIFEHKHA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.719505724 (48h6m8.270976195s) [noise=true , address=145.79.227.27:4689 , piece_id=KLTJYVDWHEE2BGLUSP2XT6L5R2C5OB7TF4DZLYBF6R5OG3N6AEFQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.720136295 (48h6m8.271606756s) [noise=true , address=145.79.227.14:4840 , piece_id=QASWXZB3E6E7VYGF24KZ37Q3LKT6F4WVFKTBV3YS73KYAWTBV7YQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.721166056 (48h6m8.272636517s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.721883453 (48h6m8.273353994s) [noise=true , address=45.91.116.4:4939 , piece_id=KMHEQSQTA2WEWMIO7JDQZUDLSRV4CA6OJUN2CYYUSA7SU2ZDQBCQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.722398593 (48h6m8.273869054s) [noise=true , address=77.90.31.18:4806 , piece_id=LNL5LRHH3OOELKEO4HND2K3GNVDPCBCGKBNJ75DCLEKRAOMTQLKA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.724101306 (48h6m8.275571767s) [noise=true , address=145.79.227.17:4970 , piece_id=F2227LWTX5PRCQIUEJNRJ3QS3GRJOIEJIZFV3KRF7BAEAPSQZR7A, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.726385264 (48h6m8.277855776s) [noise=true , address=45.91.116.11:4117 , piece_id=UOEFPOPOFIO5KXY2JOX54YHUWNMTCAFMXEBKAMMR4GSR5ZE6J6PA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.728323225 (48h6m8.279793686s) [noise=true , address=145.79.227.20:4437 , piece_id=AOK6MTRZKPCXTNHZWGBYJGRBYNHFWO7GSG2PT4DFSMDAWK75GY4A, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.730321541 (48h6m8.281792002s) [noise=true , address=145.79.227.9:4366 , piece_id=JDBR4PTBZ5RN6NTWPERVLFANUAZQF3FSGIENVRHSYN56ED5NT2AQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.731324300 (48h6m8.282794832s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.732226989 (48h6m8.28369749s) [noise=true , address=77.90.31.19:4841 , piece_id=6JVB2WIEA2STMNBVTM6D7IYFHQY4QJGXAYHJAQILMYZZDH2MWIVA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.733026101 (48h6m8.284496562s) [noise=true , address=145.79.227.21:4474 , piece_id=RNB5T6DIUCY7IV2ULLO2UJRDKUJXI34UM4ZCHPJB22DNQ47VZJVA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:11.398415378 (48h6m9.949885839s) [upload=42408]: Next returned nil segment Nov 11 08:10:11.398434043 (48h6m9.949904504s) [upload=42408]: Waiting for error group managing segments... Nov 11 08:10:11.796154582 (48h6m10.347625043s) [noise=true , address=77.90.31.18:4806 , piece_id=LNL5LRHH3OOELKEO4HND2K3GNVDPCBCGKBNJ75DCLEKRAOMTQLKA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.809043214 (48h6m10.360513675s) [noise=true , address=45.91.116.8:4047 , piece_id=VP6VW2ZTNSKS35R47GGJZI4YSHNNFUFB4YQ3W3QFT5ENN7F73VQA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.866335992 (48h6m10.417806453s) [noise=true , address=45.91.116.11:4117 , piece_id=UOEFPOPOFIO5KXY2JOX54YHUWNMTCAFMXEBKAMMR4GSR5ZE6J6PA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.932922087 (48h6m10.484392538s) [noise=true , address=77.90.31.19:4841 , piece_id=6JVB2WIEA2STMNBVTM6D7IYFHQY4QJGXAYHJAQILMYZZDH2MWIVA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:12.005905333 (48h6m10.557375794s) [noise=true , address=45.91.116.4:4939 , piece_id=KMHEQSQTA2WEWMIO7JDQZUDLSRV4CA6OJUN2CYYUSA7SU2ZDQBCQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:16.703757864 (48h6m15.255228326s) [noise=true , address=145.79.227.9:4366 , piece_id=JDBR4PTBZ5RN6NTWPERVLFANUAZQF3FSGIENVRHSYN56ED5NT2AQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:17.540298352 (48h6m16.091768813s) [noise=true , address=145.79.227.21:4474 , piece_id=RNB5T6DIUCY7IV2ULLO2UJRDKUJXI34UM4ZCHPJB22DNQ47VZJVA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:19.805526866 (48h6m18.356997337s) [noise=true , address=145.79.227.27:4689 , piece_id=KLTJYVDWHEE2BGLUSP2XT6L5R2C5OB7TF4DZLYBF6R5OG3N6AEFQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:24.397542313 (48h6m22.949012764s) [noise=true , address=145.79.227.17:4970 , piece_id=F2227LWTX5PRCQIUEJNRJ3QS3GRJOIEJIZFV3KRF7BAEAPSQZR7A, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:24.779686050 (48h6m23.331156511s) [noise=true , address=145.79.227.14:4840 , piece_id=QASWXZB3E6E7VYGF24KZ37Q3LKT6F4WVFKTBV3YS73KYAWTBV7YQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:25.298168925 (48h6m23.849639387s) [noise=true , address=145.79.227.11:4729 , piece_id=CDIU7FFPCVQGDK3K5ZDUARJ6VGB7KIY2DEVL7ZRLW5EASIFEHKHA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:35.044925402 (48h6m33.596395853s) [noise=true , address=145.79.227.30:4761 , piece_id=ASO5O5LM6CEF6DXGBVOCI3NOCWLGKDU2T5JHOL5ZPWEXMTRMKZIA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:37.306959937 (48h6m35.858430398s) [noise=true , address=145.79.227.20:4437 , piece_id=AOK6MTRZKPCXTNHZWGBYJGRBYNHFWO7GSG2PT4DFSMDAWK75GY4A, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:37.306987550 (48h6m35.858458011s) [seg_pos={0 0}, upload=42408]: Segment reached optimal threshold of 13 pieces. Nov 11 08:10:37.307209231 (48h6m35.858679702s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4): protocol: expected piece hash; context canceled; EOF Nov 11 08:10:37.307218759 (48h6m35.85868922s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307368936 (48h6m35.858839397s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw): protocol: expected piece hash; context canceled; EOF Nov 11 08:10:37.307376741 (48h6m35.858847202s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307561282 (48h6m35.859031743s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9): protocol: write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection Nov 11 08:10:37.307585969 (48h6m35.85905643s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307629221 (48h6m35.859099682s) [seg_pos={0 0}, upload=42408]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 11 08:10:37.307741005 (48h6m35.859211466s) [upload=42408]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 11 08:10:37.307750052 (48h6m35.859220513s) [upload=42408]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 11 08:10:37.612339386 (48h6m36.163809848s) [upload=42408]: Done uploading segments... Nov 11 08:10:37.612366317 (48h6m36.163836788s) [upload=42408]: Upload finished. err: <nil> Duration: 6.319420925s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [.............X.] duration: 72.93234ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/77081cc731df4257 FAIL (2025-11-11 08:20:59 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:50034->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 72.964791ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 5.767347216s trace: https://jaeger.dev.storj.io/trace/49621e5e6beafe8e FAIL (2025-11-11 08:20:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 11 08:10:09.433180409 (48h6m7.98465087s) [upload=42408]: Starting upload Nov 11 08:10:09.433429884 (48h6m7.984900335s) [upload=42408]: Done starting upload Nov 11 08:10:09.433567967 (48h6m7.985038438s) [upload=42408]: Uploading segments... Nov 11 08:10:09.434191213 (48h6m7.985661674s) [upload=42408]: Got next segment. Inline: false Nov 11 08:10:09.434274032 (48h6m7.985744493s) [upload=42408]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 08:10:09.434282217 (48h6m7.985752688s) [upload=42408]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 08:10:09.714872160 (48h6m8.266342621s) [seg_pos={0 0}, upload=42408]: Begin upload segment... Nov 11 08:10:09.715084585 (48h6m8.266555046s) [seg_pos={0 0}, upload=42408]: Done begin upload segment. Nov 11 08:10:09.715464638 (48h6m8.2669351s) [noise=true , address=45.91.116.8:4047 , piece_id=VP6VW2ZTNSKS35R47GGJZI4YSHNNFUFB4YQ3W3QFT5ENN7F73VQA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.715555221 (48h6m8.267025692s) [noise=true , address=145.79.227.30:4761 , piece_id=ASO5O5LM6CEF6DXGBVOCI3NOCWLGKDU2T5JHOL5ZPWEXMTRMKZIA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.718085410 (48h6m8.269555901s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.718721901 (48h6m8.270192402s) [noise=true , address=145.79.227.11:4729 , piece_id=CDIU7FFPCVQGDK3K5ZDUARJ6VGB7KIY2DEVL7ZRLW5EASIFEHKHA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.719505724 (48h6m8.270976195s) [noise=true , address=145.79.227.27:4689 , piece_id=KLTJYVDWHEE2BGLUSP2XT6L5R2C5OB7TF4DZLYBF6R5OG3N6AEFQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.720136295 (48h6m8.271606756s) [noise=true , address=145.79.227.14:4840 , piece_id=QASWXZB3E6E7VYGF24KZ37Q3LKT6F4WVFKTBV3YS73KYAWTBV7YQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.721166056 (48h6m8.272636517s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.721883453 (48h6m8.273353994s) [noise=true , address=45.91.116.4:4939 , piece_id=KMHEQSQTA2WEWMIO7JDQZUDLSRV4CA6OJUN2CYYUSA7SU2ZDQBCQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.722398593 (48h6m8.273869054s) [noise=true , address=77.90.31.18:4806 , piece_id=LNL5LRHH3OOELKEO4HND2K3GNVDPCBCGKBNJ75DCLEKRAOMTQLKA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.724101306 (48h6m8.275571767s) [noise=true , address=145.79.227.17:4970 , piece_id=F2227LWTX5PRCQIUEJNRJ3QS3GRJOIEJIZFV3KRF7BAEAPSQZR7A, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.726385264 (48h6m8.277855776s) [noise=true , address=45.91.116.11:4117 , piece_id=UOEFPOPOFIO5KXY2JOX54YHUWNMTCAFMXEBKAMMR4GSR5ZE6J6PA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.728323225 (48h6m8.279793686s) [noise=true , address=145.79.227.20:4437 , piece_id=AOK6MTRZKPCXTNHZWGBYJGRBYNHFWO7GSG2PT4DFSMDAWK75GY4A, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.730321541 (48h6m8.281792002s) [noise=true , address=145.79.227.9:4366 , piece_id=JDBR4PTBZ5RN6NTWPERVLFANUAZQF3FSGIENVRHSYN56ED5NT2AQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.731324300 (48h6m8.282794832s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.732226989 (48h6m8.28369749s) [noise=true , address=77.90.31.19:4841 , piece_id=6JVB2WIEA2STMNBVTM6D7IYFHQY4QJGXAYHJAQILMYZZDH2MWIVA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.733026101 (48h6m8.284496562s) [noise=true , address=145.79.227.21:4474 , piece_id=RNB5T6DIUCY7IV2ULLO2UJRDKUJXI34UM4ZCHPJB22DNQ47VZJVA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:11.398415378 (48h6m9.949885839s) [upload=42408]: Next returned nil segment Nov 11 08:10:11.398434043 (48h6m9.949904504s) [upload=42408]: Waiting for error group managing segments... Nov 11 08:10:11.796154582 (48h6m10.347625043s) [noise=true , address=77.90.31.18:4806 , piece_id=LNL5LRHH3OOELKEO4HND2K3GNVDPCBCGKBNJ75DCLEKRAOMTQLKA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.809043214 (48h6m10.360513675s) [noise=true , address=45.91.116.8:4047 , piece_id=VP6VW2ZTNSKS35R47GGJZI4YSHNNFUFB4YQ3W3QFT5ENN7F73VQA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.866335992 (48h6m10.417806453s) [noise=true , address=45.91.116.11:4117 , piece_id=UOEFPOPOFIO5KXY2JOX54YHUWNMTCAFMXEBKAMMR4GSR5ZE6J6PA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.932922087 (48h6m10.484392538s) [noise=true , address=77.90.31.19:4841 , piece_id=6JVB2WIEA2STMNBVTM6D7IYFHQY4QJGXAYHJAQILMYZZDH2MWIVA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:12.005905333 (48h6m10.557375794s) [noise=true , address=45.91.116.4:4939 , piece_id=KMHEQSQTA2WEWMIO7JDQZUDLSRV4CA6OJUN2CYYUSA7SU2ZDQBCQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:16.703757864 (48h6m15.255228326s) [noise=true , address=145.79.227.9:4366 , piece_id=JDBR4PTBZ5RN6NTWPERVLFANUAZQF3FSGIENVRHSYN56ED5NT2AQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:17.540298352 (48h6m16.091768813s) [noise=true , address=145.79.227.21:4474 , piece_id=RNB5T6DIUCY7IV2ULLO2UJRDKUJXI34UM4ZCHPJB22DNQ47VZJVA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:19.805526866 (48h6m18.356997337s) [noise=true , address=145.79.227.27:4689 , piece_id=KLTJYVDWHEE2BGLUSP2XT6L5R2C5OB7TF4DZLYBF6R5OG3N6AEFQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:24.397542313 (48h6m22.949012764s) [noise=true , address=145.79.227.17:4970 , piece_id=F2227LWTX5PRCQIUEJNRJ3QS3GRJOIEJIZFV3KRF7BAEAPSQZR7A, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:24.779686050 (48h6m23.331156511s) [noise=true , address=145.79.227.14:4840 , piece_id=QASWXZB3E6E7VYGF24KZ37Q3LKT6F4WVFKTBV3YS73KYAWTBV7YQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:25.298168925 (48h6m23.849639387s) [noise=true , address=145.79.227.11:4729 , piece_id=CDIU7FFPCVQGDK3K5ZDUARJ6VGB7KIY2DEVL7ZRLW5EASIFEHKHA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:35.044925402 (48h6m33.596395853s) [noise=true , address=145.79.227.30:4761 , piece_id=ASO5O5LM6CEF6DXGBVOCI3NOCWLGKDU2T5JHOL5ZPWEXMTRMKZIA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:37.306959937 (48h6m35.858430398s) [noise=true , address=145.79.227.20:4437 , piece_id=AOK6MTRZKPCXTNHZWGBYJGRBYNHFWO7GSG2PT4DFSMDAWK75GY4A, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:37.306987550 (48h6m35.858458011s) [seg_pos={0 0}, upload=42408]: Segment reached optimal threshold of 13 pieces. Nov 11 08:10:37.307209231 (48h6m35.858679702s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4): protocol: expected piece hash; context canceled; EOF Nov 11 08:10:37.307218759 (48h6m35.85868922s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307368936 (48h6m35.858839397s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw): protocol: expected piece hash; context canceled; EOF Nov 11 08:10:37.307376741 (48h6m35.858847202s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307561282 (48h6m35.859031743s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9): protocol: write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection Nov 11 08:10:37.307585969 (48h6m35.85905643s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307629221 (48h6m35.859099682s) [seg_pos={0 0}, upload=42408]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 11 08:10:37.307741005 (48h6m35.859211466s) [upload=42408]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 11 08:10:37.307750052 (48h6m35.859220513s) [upload=42408]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 11 08:10:37.612339386 (48h6m36.163809848s) [upload=42408]: Done uploading segments... Nov 11 08:10:37.612366317 (48h6m36.163836788s) [upload=42408]: Upload finished. err: <nil> Duration: 5.767404585s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [..............X] duration: 442.764483ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/77081cc731df4257 FAIL (2025-11-11 08:20:16 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:50034->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 443.063753ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 5.874608264s trace: https://jaeger.dev.storj.io/trace/49621e5e6beafe8e FAIL (2025-11-11 08:19:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 11 08:10:09.433180409 (48h6m7.98465087s) [upload=42408]: Starting upload Nov 11 08:10:09.433429884 (48h6m7.984900335s) [upload=42408]: Done starting upload Nov 11 08:10:09.433567967 (48h6m7.985038438s) [upload=42408]: Uploading segments... Nov 11 08:10:09.434191213 (48h6m7.985661674s) [upload=42408]: Got next segment. Inline: false Nov 11 08:10:09.434274032 (48h6m7.985744493s) [upload=42408]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 08:10:09.434282217 (48h6m7.985752688s) [upload=42408]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 08:10:09.714872160 (48h6m8.266342621s) [seg_pos={0 0}, upload=42408]: Begin upload segment... Nov 11 08:10:09.715084585 (48h6m8.266555046s) [seg_pos={0 0}, upload=42408]: Done begin upload segment. Nov 11 08:10:09.715464638 (48h6m8.2669351s) [noise=true , address=45.91.116.8:4047 , piece_id=VP6VW2ZTNSKS35R47GGJZI4YSHNNFUFB4YQ3W3QFT5ENN7F73VQA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.715555221 (48h6m8.267025692s) [noise=true , address=145.79.227.30:4761 , piece_id=ASO5O5LM6CEF6DXGBVOCI3NOCWLGKDU2T5JHOL5ZPWEXMTRMKZIA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.718085410 (48h6m8.269555901s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.718721901 (48h6m8.270192402s) [noise=true , address=145.79.227.11:4729 , piece_id=CDIU7FFPCVQGDK3K5ZDUARJ6VGB7KIY2DEVL7ZRLW5EASIFEHKHA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.719505724 (48h6m8.270976195s) [noise=true , address=145.79.227.27:4689 , piece_id=KLTJYVDWHEE2BGLUSP2XT6L5R2C5OB7TF4DZLYBF6R5OG3N6AEFQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.720136295 (48h6m8.271606756s) [noise=true , address=145.79.227.14:4840 , piece_id=QASWXZB3E6E7VYGF24KZ37Q3LKT6F4WVFKTBV3YS73KYAWTBV7YQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.721166056 (48h6m8.272636517s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.721883453 (48h6m8.273353994s) [noise=true , address=45.91.116.4:4939 , piece_id=KMHEQSQTA2WEWMIO7JDQZUDLSRV4CA6OJUN2CYYUSA7SU2ZDQBCQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.722398593 (48h6m8.273869054s) [noise=true , address=77.90.31.18:4806 , piece_id=LNL5LRHH3OOELKEO4HND2K3GNVDPCBCGKBNJ75DCLEKRAOMTQLKA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.724101306 (48h6m8.275571767s) [noise=true , address=145.79.227.17:4970 , piece_id=F2227LWTX5PRCQIUEJNRJ3QS3GRJOIEJIZFV3KRF7BAEAPSQZR7A, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.726385264 (48h6m8.277855776s) [noise=true , address=45.91.116.11:4117 , piece_id=UOEFPOPOFIO5KXY2JOX54YHUWNMTCAFMXEBKAMMR4GSR5ZE6J6PA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.728323225 (48h6m8.279793686s) [noise=true , address=145.79.227.20:4437 , piece_id=AOK6MTRZKPCXTNHZWGBYJGRBYNHFWO7GSG2PT4DFSMDAWK75GY4A, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.730321541 (48h6m8.281792002s) [noise=true , address=145.79.227.9:4366 , piece_id=JDBR4PTBZ5RN6NTWPERVLFANUAZQF3FSGIENVRHSYN56ED5NT2AQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.731324300 (48h6m8.282794832s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.732226989 (48h6m8.28369749s) [noise=true , address=77.90.31.19:4841 , piece_id=6JVB2WIEA2STMNBVTM6D7IYFHQY4QJGXAYHJAQILMYZZDH2MWIVA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.733026101 (48h6m8.284496562s) [noise=true , address=145.79.227.21:4474 , piece_id=RNB5T6DIUCY7IV2ULLO2UJRDKUJXI34UM4ZCHPJB22DNQ47VZJVA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:11.398415378 (48h6m9.949885839s) [upload=42408]: Next returned nil segment Nov 11 08:10:11.398434043 (48h6m9.949904504s) [upload=42408]: Waiting for error group managing segments... Nov 11 08:10:11.796154582 (48h6m10.347625043s) [noise=true , address=77.90.31.18:4806 , piece_id=LNL5LRHH3OOELKEO4HND2K3GNVDPCBCGKBNJ75DCLEKRAOMTQLKA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.809043214 (48h6m10.360513675s) [noise=true , address=45.91.116.8:4047 , piece_id=VP6VW2ZTNSKS35R47GGJZI4YSHNNFUFB4YQ3W3QFT5ENN7F73VQA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.866335992 (48h6m10.417806453s) [noise=true , address=45.91.116.11:4117 , piece_id=UOEFPOPOFIO5KXY2JOX54YHUWNMTCAFMXEBKAMMR4GSR5ZE6J6PA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.932922087 (48h6m10.484392538s) [noise=true , address=77.90.31.19:4841 , piece_id=6JVB2WIEA2STMNBVTM6D7IYFHQY4QJGXAYHJAQILMYZZDH2MWIVA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:12.005905333 (48h6m10.557375794s) [noise=true , address=45.91.116.4:4939 , piece_id=KMHEQSQTA2WEWMIO7JDQZUDLSRV4CA6OJUN2CYYUSA7SU2ZDQBCQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:16.703757864 (48h6m15.255228326s) [noise=true , address=145.79.227.9:4366 , piece_id=JDBR4PTBZ5RN6NTWPERVLFANUAZQF3FSGIENVRHSYN56ED5NT2AQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:17.540298352 (48h6m16.091768813s) [noise=true , address=145.79.227.21:4474 , piece_id=RNB5T6DIUCY7IV2ULLO2UJRDKUJXI34UM4ZCHPJB22DNQ47VZJVA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:19.805526866 (48h6m18.356997337s) [noise=true , address=145.79.227.27:4689 , piece_id=KLTJYVDWHEE2BGLUSP2XT6L5R2C5OB7TF4DZLYBF6R5OG3N6AEFQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:24.397542313 (48h6m22.949012764s) [noise=true , address=145.79.227.17:4970 , piece_id=F2227LWTX5PRCQIUEJNRJ3QS3GRJOIEJIZFV3KRF7BAEAPSQZR7A, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:24.779686050 (48h6m23.331156511s) [noise=true , address=145.79.227.14:4840 , piece_id=QASWXZB3E6E7VYGF24KZ37Q3LKT6F4WVFKTBV3YS73KYAWTBV7YQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:25.298168925 (48h6m23.849639387s) [noise=true , address=145.79.227.11:4729 , piece_id=CDIU7FFPCVQGDK3K5ZDUARJ6VGB7KIY2DEVL7ZRLW5EASIFEHKHA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:35.044925402 (48h6m33.596395853s) [noise=true , address=145.79.227.30:4761 , piece_id=ASO5O5LM6CEF6DXGBVOCI3NOCWLGKDU2T5JHOL5ZPWEXMTRMKZIA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:37.306959937 (48h6m35.858430398s) [noise=true , address=145.79.227.20:4437 , piece_id=AOK6MTRZKPCXTNHZWGBYJGRBYNHFWO7GSG2PT4DFSMDAWK75GY4A, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:37.306987550 (48h6m35.858458011s) [seg_pos={0 0}, upload=42408]: Segment reached optimal threshold of 13 pieces. Nov 11 08:10:37.307209231 (48h6m35.858679702s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4): protocol: expected piece hash; context canceled; EOF Nov 11 08:10:37.307218759 (48h6m35.85868922s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307368936 (48h6m35.858839397s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw): protocol: expected piece hash; context canceled; EOF Nov 11 08:10:37.307376741 (48h6m35.858847202s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307561282 (48h6m35.859031743s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9): protocol: write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection Nov 11 08:10:37.307585969 (48h6m35.85905643s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307629221 (48h6m35.859099682s) [seg_pos={0 0}, upload=42408]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 11 08:10:37.307741005 (48h6m35.859211466s) [upload=42408]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 11 08:10:37.307750052 (48h6m35.859220513s) [upload=42408]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 11 08:10:37.612339386 (48h6m36.163809848s) [upload=42408]: Done uploading segments... Nov 11 08:10:37.612366317 (48h6m36.163836788s) [upload=42408]: Upload finished. err: <nil> Duration: 5.874639072s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 5.895165968s trace: https://jaeger.dev.storj.io/trace/49621e5e6beafe8e FAIL (2025-11-11 08:18:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 11 08:10:09.433180409 (48h6m7.98465087s) [upload=42408]: Starting upload Nov 11 08:10:09.433429884 (48h6m7.984900335s) [upload=42408]: Done starting upload Nov 11 08:10:09.433567967 (48h6m7.985038438s) [upload=42408]: Uploading segments... Nov 11 08:10:09.434191213 (48h6m7.985661674s) [upload=42408]: Got next segment. Inline: false Nov 11 08:10:09.434274032 (48h6m7.985744493s) [upload=42408]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 08:10:09.434282217 (48h6m7.985752688s) [upload=42408]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 08:10:09.714872160 (48h6m8.266342621s) [seg_pos={0 0}, upload=42408]: Begin upload segment... Nov 11 08:10:09.715084585 (48h6m8.266555046s) [seg_pos={0 0}, upload=42408]: Done begin upload segment. Nov 11 08:10:09.715464638 (48h6m8.2669351s) [noise=true , address=45.91.116.8:4047 , piece_id=VP6VW2ZTNSKS35R47GGJZI4YSHNNFUFB4YQ3W3QFT5ENN7F73VQA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.715555221 (48h6m8.267025692s) [noise=true , address=145.79.227.30:4761 , piece_id=ASO5O5LM6CEF6DXGBVOCI3NOCWLGKDU2T5JHOL5ZPWEXMTRMKZIA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.718085410 (48h6m8.269555901s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.718721901 (48h6m8.270192402s) [noise=true , address=145.79.227.11:4729 , piece_id=CDIU7FFPCVQGDK3K5ZDUARJ6VGB7KIY2DEVL7ZRLW5EASIFEHKHA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.719505724 (48h6m8.270976195s) [noise=true , address=145.79.227.27:4689 , piece_id=KLTJYVDWHEE2BGLUSP2XT6L5R2C5OB7TF4DZLYBF6R5OG3N6AEFQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.720136295 (48h6m8.271606756s) [noise=true , address=145.79.227.14:4840 , piece_id=QASWXZB3E6E7VYGF24KZ37Q3LKT6F4WVFKTBV3YS73KYAWTBV7YQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.721166056 (48h6m8.272636517s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.721883453 (48h6m8.273353994s) [noise=true , address=45.91.116.4:4939 , piece_id=KMHEQSQTA2WEWMIO7JDQZUDLSRV4CA6OJUN2CYYUSA7SU2ZDQBCQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.722398593 (48h6m8.273869054s) [noise=true , address=77.90.31.18:4806 , piece_id=LNL5LRHH3OOELKEO4HND2K3GNVDPCBCGKBNJ75DCLEKRAOMTQLKA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.724101306 (48h6m8.275571767s) [noise=true , address=145.79.227.17:4970 , piece_id=F2227LWTX5PRCQIUEJNRJ3QS3GRJOIEJIZFV3KRF7BAEAPSQZR7A, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.726385264 (48h6m8.277855776s) [noise=true , address=45.91.116.11:4117 , piece_id=UOEFPOPOFIO5KXY2JOX54YHUWNMTCAFMXEBKAMMR4GSR5ZE6J6PA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.728323225 (48h6m8.279793686s) [noise=true , address=145.79.227.20:4437 , piece_id=AOK6MTRZKPCXTNHZWGBYJGRBYNHFWO7GSG2PT4DFSMDAWK75GY4A, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.730321541 (48h6m8.281792002s) [noise=true , address=145.79.227.9:4366 , piece_id=JDBR4PTBZ5RN6NTWPERVLFANUAZQF3FSGIENVRHSYN56ED5NT2AQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.731324300 (48h6m8.282794832s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.732226989 (48h6m8.28369749s) [noise=true , address=77.90.31.19:4841 , piece_id=6JVB2WIEA2STMNBVTM6D7IYFHQY4QJGXAYHJAQILMYZZDH2MWIVA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.733026101 (48h6m8.284496562s) [noise=true , address=145.79.227.21:4474 , piece_id=RNB5T6DIUCY7IV2ULLO2UJRDKUJXI34UM4ZCHPJB22DNQ47VZJVA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:11.398415378 (48h6m9.949885839s) [upload=42408]: Next returned nil segment Nov 11 08:10:11.398434043 (48h6m9.949904504s) [upload=42408]: Waiting for error group managing segments... Nov 11 08:10:11.796154582 (48h6m10.347625043s) [noise=true , address=77.90.31.18:4806 , piece_id=LNL5LRHH3OOELKEO4HND2K3GNVDPCBCGKBNJ75DCLEKRAOMTQLKA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.809043214 (48h6m10.360513675s) [noise=true , address=45.91.116.8:4047 , piece_id=VP6VW2ZTNSKS35R47GGJZI4YSHNNFUFB4YQ3W3QFT5ENN7F73VQA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.866335992 (48h6m10.417806453s) [noise=true , address=45.91.116.11:4117 , piece_id=UOEFPOPOFIO5KXY2JOX54YHUWNMTCAFMXEBKAMMR4GSR5ZE6J6PA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.932922087 (48h6m10.484392538s) [noise=true , address=77.90.31.19:4841 , piece_id=6JVB2WIEA2STMNBVTM6D7IYFHQY4QJGXAYHJAQILMYZZDH2MWIVA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:12.005905333 (48h6m10.557375794s) [noise=true , address=45.91.116.4:4939 , piece_id=KMHEQSQTA2WEWMIO7JDQZUDLSRV4CA6OJUN2CYYUSA7SU2ZDQBCQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:16.703757864 (48h6m15.255228326s) [noise=true , address=145.79.227.9:4366 , piece_id=JDBR4PTBZ5RN6NTWPERVLFANUAZQF3FSGIENVRHSYN56ED5NT2AQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:17.540298352 (48h6m16.091768813s) [noise=true , address=145.79.227.21:4474 , piece_id=RNB5T6DIUCY7IV2ULLO2UJRDKUJXI34UM4ZCHPJB22DNQ47VZJVA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:19.805526866 (48h6m18.356997337s) [noise=true , address=145.79.227.27:4689 , piece_id=KLTJYVDWHEE2BGLUSP2XT6L5R2C5OB7TF4DZLYBF6R5OG3N6AEFQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:24.397542313 (48h6m22.949012764s) [noise=true , address=145.79.227.17:4970 , piece_id=F2227LWTX5PRCQIUEJNRJ3QS3GRJOIEJIZFV3KRF7BAEAPSQZR7A, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:24.779686050 (48h6m23.331156511s) [noise=true , address=145.79.227.14:4840 , piece_id=QASWXZB3E6E7VYGF24KZ37Q3LKT6F4WVFKTBV3YS73KYAWTBV7YQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:25.298168925 (48h6m23.849639387s) [noise=true , address=145.79.227.11:4729 , piece_id=CDIU7FFPCVQGDK3K5ZDUARJ6VGB7KIY2DEVL7ZRLW5EASIFEHKHA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:35.044925402 (48h6m33.596395853s) [noise=true , address=145.79.227.30:4761 , piece_id=ASO5O5LM6CEF6DXGBVOCI3NOCWLGKDU2T5JHOL5ZPWEXMTRMKZIA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:37.306959937 (48h6m35.858430398s) [noise=true , address=145.79.227.20:4437 , piece_id=AOK6MTRZKPCXTNHZWGBYJGRBYNHFWO7GSG2PT4DFSMDAWK75GY4A, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:37.306987550 (48h6m35.858458011s) [seg_pos={0 0}, upload=42408]: Segment reached optimal threshold of 13 pieces. Nov 11 08:10:37.307209231 (48h6m35.858679702s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4): protocol: expected piece hash; context canceled; EOF Nov 11 08:10:37.307218759 (48h6m35.85868922s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307368936 (48h6m35.858839397s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw): protocol: expected piece hash; context canceled; EOF Nov 11 08:10:37.307376741 (48h6m35.858847202s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307561282 (48h6m35.859031743s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9): protocol: write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection Nov 11 08:10:37.307585969 (48h6m35.85905643s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307629221 (48h6m35.859099682s) [seg_pos={0 0}, upload=42408]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 11 08:10:37.307741005 (48h6m35.859211466s) [upload=42408]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 11 08:10:37.307750052 (48h6m35.859220513s) [upload=42408]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 11 08:10:37.612339386 (48h6m36.163809848s) [upload=42408]: Done uploading segments... Nov 11 08:10:37.612366317 (48h6m36.163836788s) [upload=42408]: Upload finished. err: <nil> Duration: 5.895197027s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 6.090702227s trace: https://jaeger.dev.storj.io/trace/49621e5e6beafe8e FAIL (2025-11-11 08:17:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 11 08:10:09.433180409 (48h6m7.98465087s) [upload=42408]: Starting upload Nov 11 08:10:09.433429884 (48h6m7.984900335s) [upload=42408]: Done starting upload Nov 11 08:10:09.433567967 (48h6m7.985038438s) [upload=42408]: Uploading segments... Nov 11 08:10:09.434191213 (48h6m7.985661674s) [upload=42408]: Got next segment. Inline: false Nov 11 08:10:09.434274032 (48h6m7.985744493s) [upload=42408]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 08:10:09.434282217 (48h6m7.985752688s) [upload=42408]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 08:10:09.714872160 (48h6m8.266342621s) [seg_pos={0 0}, upload=42408]: Begin upload segment... Nov 11 08:10:09.715084585 (48h6m8.266555046s) [seg_pos={0 0}, upload=42408]: Done begin upload segment. Nov 11 08:10:09.715464638 (48h6m8.2669351s) [noise=true , address=45.91.116.8:4047 , piece_id=VP6VW2ZTNSKS35R47GGJZI4YSHNNFUFB4YQ3W3QFT5ENN7F73VQA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.715555221 (48h6m8.267025692s) [noise=true , address=145.79.227.30:4761 , piece_id=ASO5O5LM6CEF6DXGBVOCI3NOCWLGKDU2T5JHOL5ZPWEXMTRMKZIA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.718085410 (48h6m8.269555901s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.718721901 (48h6m8.270192402s) [noise=true , address=145.79.227.11:4729 , piece_id=CDIU7FFPCVQGDK3K5ZDUARJ6VGB7KIY2DEVL7ZRLW5EASIFEHKHA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.719505724 (48h6m8.270976195s) [noise=true , address=145.79.227.27:4689 , piece_id=KLTJYVDWHEE2BGLUSP2XT6L5R2C5OB7TF4DZLYBF6R5OG3N6AEFQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.720136295 (48h6m8.271606756s) [noise=true , address=145.79.227.14:4840 , piece_id=QASWXZB3E6E7VYGF24KZ37Q3LKT6F4WVFKTBV3YS73KYAWTBV7YQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.721166056 (48h6m8.272636517s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.721883453 (48h6m8.273353994s) [noise=true , address=45.91.116.4:4939 , piece_id=KMHEQSQTA2WEWMIO7JDQZUDLSRV4CA6OJUN2CYYUSA7SU2ZDQBCQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.722398593 (48h6m8.273869054s) [noise=true , address=77.90.31.18:4806 , piece_id=LNL5LRHH3OOELKEO4HND2K3GNVDPCBCGKBNJ75DCLEKRAOMTQLKA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.724101306 (48h6m8.275571767s) [noise=true , address=145.79.227.17:4970 , piece_id=F2227LWTX5PRCQIUEJNRJ3QS3GRJOIEJIZFV3KRF7BAEAPSQZR7A, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.726385264 (48h6m8.277855776s) [noise=true , address=45.91.116.11:4117 , piece_id=UOEFPOPOFIO5KXY2JOX54YHUWNMTCAFMXEBKAMMR4GSR5ZE6J6PA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.728323225 (48h6m8.279793686s) [noise=true , address=145.79.227.20:4437 , piece_id=AOK6MTRZKPCXTNHZWGBYJGRBYNHFWO7GSG2PT4DFSMDAWK75GY4A, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.730321541 (48h6m8.281792002s) [noise=true , address=145.79.227.9:4366 , piece_id=JDBR4PTBZ5RN6NTWPERVLFANUAZQF3FSGIENVRHSYN56ED5NT2AQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.731324300 (48h6m8.282794832s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.732226989 (48h6m8.28369749s) [noise=true , address=77.90.31.19:4841 , piece_id=6JVB2WIEA2STMNBVTM6D7IYFHQY4QJGXAYHJAQILMYZZDH2MWIVA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.733026101 (48h6m8.284496562s) [noise=true , address=145.79.227.21:4474 , piece_id=RNB5T6DIUCY7IV2ULLO2UJRDKUJXI34UM4ZCHPJB22DNQ47VZJVA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:11.398415378 (48h6m9.949885839s) [upload=42408]: Next returned nil segment Nov 11 08:10:11.398434043 (48h6m9.949904504s) [upload=42408]: Waiting for error group managing segments... Nov 11 08:10:11.796154582 (48h6m10.347625043s) [noise=true , address=77.90.31.18:4806 , piece_id=LNL5LRHH3OOELKEO4HND2K3GNVDPCBCGKBNJ75DCLEKRAOMTQLKA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.809043214 (48h6m10.360513675s) [noise=true , address=45.91.116.8:4047 , piece_id=VP6VW2ZTNSKS35R47GGJZI4YSHNNFUFB4YQ3W3QFT5ENN7F73VQA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.866335992 (48h6m10.417806453s) [noise=true , address=45.91.116.11:4117 , piece_id=UOEFPOPOFIO5KXY2JOX54YHUWNMTCAFMXEBKAMMR4GSR5ZE6J6PA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.932922087 (48h6m10.484392538s) [noise=true , address=77.90.31.19:4841 , piece_id=6JVB2WIEA2STMNBVTM6D7IYFHQY4QJGXAYHJAQILMYZZDH2MWIVA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:12.005905333 (48h6m10.557375794s) [noise=true , address=45.91.116.4:4939 , piece_id=KMHEQSQTA2WEWMIO7JDQZUDLSRV4CA6OJUN2CYYUSA7SU2ZDQBCQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:16.703757864 (48h6m15.255228326s) [noise=true , address=145.79.227.9:4366 , piece_id=JDBR4PTBZ5RN6NTWPERVLFANUAZQF3FSGIENVRHSYN56ED5NT2AQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:17.540298352 (48h6m16.091768813s) [noise=true , address=145.79.227.21:4474 , piece_id=RNB5T6DIUCY7IV2ULLO2UJRDKUJXI34UM4ZCHPJB22DNQ47VZJVA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:19.805526866 (48h6m18.356997337s) [noise=true , address=145.79.227.27:4689 , piece_id=KLTJYVDWHEE2BGLUSP2XT6L5R2C5OB7TF4DZLYBF6R5OG3N6AEFQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:24.397542313 (48h6m22.949012764s) [noise=true , address=145.79.227.17:4970 , piece_id=F2227LWTX5PRCQIUEJNRJ3QS3GRJOIEJIZFV3KRF7BAEAPSQZR7A, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:24.779686050 (48h6m23.331156511s) [noise=true , address=145.79.227.14:4840 , piece_id=QASWXZB3E6E7VYGF24KZ37Q3LKT6F4WVFKTBV3YS73KYAWTBV7YQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:25.298168925 (48h6m23.849639387s) [noise=true , address=145.79.227.11:4729 , piece_id=CDIU7FFPCVQGDK3K5ZDUARJ6VGB7KIY2DEVL7ZRLW5EASIFEHKHA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:35.044925402 (48h6m33.596395853s) [noise=true , address=145.79.227.30:4761 , piece_id=ASO5O5LM6CEF6DXGBVOCI3NOCWLGKDU2T5JHOL5ZPWEXMTRMKZIA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:37.306959937 (48h6m35.858430398s) [noise=true , address=145.79.227.20:4437 , piece_id=AOK6MTRZKPCXTNHZWGBYJGRBYNHFWO7GSG2PT4DFSMDAWK75GY4A, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:37.306987550 (48h6m35.858458011s) [seg_pos={0 0}, upload=42408]: Segment reached optimal threshold of 13 pieces. Nov 11 08:10:37.307209231 (48h6m35.858679702s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4): protocol: expected piece hash; context canceled; EOF Nov 11 08:10:37.307218759 (48h6m35.85868922s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307368936 (48h6m35.858839397s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw): protocol: expected piece hash; context canceled; EOF Nov 11 08:10:37.307376741 (48h6m35.858847202s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307561282 (48h6m35.859031743s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9): protocol: write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection Nov 11 08:10:37.307585969 (48h6m35.85905643s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307629221 (48h6m35.859099682s) [seg_pos={0 0}, upload=42408]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 11 08:10:37.307741005 (48h6m35.859211466s) [upload=42408]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 11 08:10:37.307750052 (48h6m35.859220513s) [upload=42408]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 11 08:10:37.612339386 (48h6m36.163809848s) [upload=42408]: Done uploading segments... Nov 11 08:10:37.612366317 (48h6m36.163836788s) [upload=42408]: Upload finished. err: <nil> Duration: 6.09073548s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 5.868755151s trace: https://jaeger.dev.storj.io/trace/49621e5e6beafe8e FAIL (2025-11-11 08:16:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 11 08:10:09.433180409 (48h6m7.98465087s) [upload=42408]: Starting upload Nov 11 08:10:09.433429884 (48h6m7.984900335s) [upload=42408]: Done starting upload Nov 11 08:10:09.433567967 (48h6m7.985038438s) [upload=42408]: Uploading segments... Nov 11 08:10:09.434191213 (48h6m7.985661674s) [upload=42408]: Got next segment. Inline: false Nov 11 08:10:09.434274032 (48h6m7.985744493s) [upload=42408]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 08:10:09.434282217 (48h6m7.985752688s) [upload=42408]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 08:10:09.714872160 (48h6m8.266342621s) [seg_pos={0 0}, upload=42408]: Begin upload segment... Nov 11 08:10:09.715084585 (48h6m8.266555046s) [seg_pos={0 0}, upload=42408]: Done begin upload segment. Nov 11 08:10:09.715464638 (48h6m8.2669351s) [noise=true , address=45.91.116.8:4047 , piece_id=VP6VW2ZTNSKS35R47GGJZI4YSHNNFUFB4YQ3W3QFT5ENN7F73VQA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.715555221 (48h6m8.267025692s) [noise=true , address=145.79.227.30:4761 , piece_id=ASO5O5LM6CEF6DXGBVOCI3NOCWLGKDU2T5JHOL5ZPWEXMTRMKZIA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.718085410 (48h6m8.269555901s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.718721901 (48h6m8.270192402s) [noise=true , address=145.79.227.11:4729 , piece_id=CDIU7FFPCVQGDK3K5ZDUARJ6VGB7KIY2DEVL7ZRLW5EASIFEHKHA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.719505724 (48h6m8.270976195s) [noise=true , address=145.79.227.27:4689 , piece_id=KLTJYVDWHEE2BGLUSP2XT6L5R2C5OB7TF4DZLYBF6R5OG3N6AEFQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.720136295 (48h6m8.271606756s) [noise=true , address=145.79.227.14:4840 , piece_id=QASWXZB3E6E7VYGF24KZ37Q3LKT6F4WVFKTBV3YS73KYAWTBV7YQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.721166056 (48h6m8.272636517s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.721883453 (48h6m8.273353994s) [noise=true , address=45.91.116.4:4939 , piece_id=KMHEQSQTA2WEWMIO7JDQZUDLSRV4CA6OJUN2CYYUSA7SU2ZDQBCQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.722398593 (48h6m8.273869054s) [noise=true , address=77.90.31.18:4806 , piece_id=LNL5LRHH3OOELKEO4HND2K3GNVDPCBCGKBNJ75DCLEKRAOMTQLKA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.724101306 (48h6m8.275571767s) [noise=true , address=145.79.227.17:4970 , piece_id=F2227LWTX5PRCQIUEJNRJ3QS3GRJOIEJIZFV3KRF7BAEAPSQZR7A, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.726385264 (48h6m8.277855776s) [noise=true , address=45.91.116.11:4117 , piece_id=UOEFPOPOFIO5KXY2JOX54YHUWNMTCAFMXEBKAMMR4GSR5ZE6J6PA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.728323225 (48h6m8.279793686s) [noise=true , address=145.79.227.20:4437 , piece_id=AOK6MTRZKPCXTNHZWGBYJGRBYNHFWO7GSG2PT4DFSMDAWK75GY4A, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.730321541 (48h6m8.281792002s) [noise=true , address=145.79.227.9:4366 , piece_id=JDBR4PTBZ5RN6NTWPERVLFANUAZQF3FSGIENVRHSYN56ED5NT2AQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.731324300 (48h6m8.282794832s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.732226989 (48h6m8.28369749s) [noise=true , address=77.90.31.19:4841 , piece_id=6JVB2WIEA2STMNBVTM6D7IYFHQY4QJGXAYHJAQILMYZZDH2MWIVA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.733026101 (48h6m8.284496562s) [noise=true , address=145.79.227.21:4474 , piece_id=RNB5T6DIUCY7IV2ULLO2UJRDKUJXI34UM4ZCHPJB22DNQ47VZJVA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:11.398415378 (48h6m9.949885839s) [upload=42408]: Next returned nil segment Nov 11 08:10:11.398434043 (48h6m9.949904504s) [upload=42408]: Waiting for error group managing segments... Nov 11 08:10:11.796154582 (48h6m10.347625043s) [noise=true , address=77.90.31.18:4806 , piece_id=LNL5LRHH3OOELKEO4HND2K3GNVDPCBCGKBNJ75DCLEKRAOMTQLKA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.809043214 (48h6m10.360513675s) [noise=true , address=45.91.116.8:4047 , piece_id=VP6VW2ZTNSKS35R47GGJZI4YSHNNFUFB4YQ3W3QFT5ENN7F73VQA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.866335992 (48h6m10.417806453s) [noise=true , address=45.91.116.11:4117 , piece_id=UOEFPOPOFIO5KXY2JOX54YHUWNMTCAFMXEBKAMMR4GSR5ZE6J6PA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.932922087 (48h6m10.484392538s) [noise=true , address=77.90.31.19:4841 , piece_id=6JVB2WIEA2STMNBVTM6D7IYFHQY4QJGXAYHJAQILMYZZDH2MWIVA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:12.005905333 (48h6m10.557375794s) [noise=true , address=45.91.116.4:4939 , piece_id=KMHEQSQTA2WEWMIO7JDQZUDLSRV4CA6OJUN2CYYUSA7SU2ZDQBCQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:16.703757864 (48h6m15.255228326s) [noise=true , address=145.79.227.9:4366 , piece_id=JDBR4PTBZ5RN6NTWPERVLFANUAZQF3FSGIENVRHSYN56ED5NT2AQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:17.540298352 (48h6m16.091768813s) [noise=true , address=145.79.227.21:4474 , piece_id=RNB5T6DIUCY7IV2ULLO2UJRDKUJXI34UM4ZCHPJB22DNQ47VZJVA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:19.805526866 (48h6m18.356997337s) [noise=true , address=145.79.227.27:4689 , piece_id=KLTJYVDWHEE2BGLUSP2XT6L5R2C5OB7TF4DZLYBF6R5OG3N6AEFQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:24.397542313 (48h6m22.949012764s) [noise=true , address=145.79.227.17:4970 , piece_id=F2227LWTX5PRCQIUEJNRJ3QS3GRJOIEJIZFV3KRF7BAEAPSQZR7A, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:24.779686050 (48h6m23.331156511s) [noise=true , address=145.79.227.14:4840 , piece_id=QASWXZB3E6E7VYGF24KZ37Q3LKT6F4WVFKTBV3YS73KYAWTBV7YQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:25.298168925 (48h6m23.849639387s) [noise=true , address=145.79.227.11:4729 , piece_id=CDIU7FFPCVQGDK3K5ZDUARJ6VGB7KIY2DEVL7ZRLW5EASIFEHKHA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:35.044925402 (48h6m33.596395853s) [noise=true , address=145.79.227.30:4761 , piece_id=ASO5O5LM6CEF6DXGBVOCI3NOCWLGKDU2T5JHOL5ZPWEXMTRMKZIA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:37.306959937 (48h6m35.858430398s) [noise=true , address=145.79.227.20:4437 , piece_id=AOK6MTRZKPCXTNHZWGBYJGRBYNHFWO7GSG2PT4DFSMDAWK75GY4A, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:37.306987550 (48h6m35.858458011s) [seg_pos={0 0}, upload=42408]: Segment reached optimal threshold of 13 pieces. Nov 11 08:10:37.307209231 (48h6m35.858679702s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4): protocol: expected piece hash; context canceled; EOF Nov 11 08:10:37.307218759 (48h6m35.85868922s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307368936 (48h6m35.858839397s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw): protocol: expected piece hash; context canceled; EOF Nov 11 08:10:37.307376741 (48h6m35.858847202s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307561282 (48h6m35.859031743s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9): protocol: write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection Nov 11 08:10:37.307585969 (48h6m35.85905643s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307629221 (48h6m35.859099682s) [seg_pos={0 0}, upload=42408]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 11 08:10:37.307741005 (48h6m35.859211466s) [upload=42408]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 11 08:10:37.307750052 (48h6m35.859220513s) [upload=42408]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 11 08:10:37.612339386 (48h6m36.163809848s) [upload=42408]: Done uploading segments... Nov 11 08:10:37.612366317 (48h6m36.163836788s) [upload=42408]: Upload finished. err: <nil> Duration: 5.868787222s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 5.780844913s trace: https://jaeger.dev.storj.io/trace/49621e5e6beafe8e FAIL (2025-11-11 08:16:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 11 08:10:09.433180409 (48h6m7.98465087s) [upload=42408]: Starting upload Nov 11 08:10:09.433429884 (48h6m7.984900335s) [upload=42408]: Done starting upload Nov 11 08:10:09.433567967 (48h6m7.985038438s) [upload=42408]: Uploading segments... Nov 11 08:10:09.434191213 (48h6m7.985661674s) [upload=42408]: Got next segment. Inline: false Nov 11 08:10:09.434274032 (48h6m7.985744493s) [upload=42408]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 08:10:09.434282217 (48h6m7.985752688s) [upload=42408]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 08:10:09.714872160 (48h6m8.266342621s) [seg_pos={0 0}, upload=42408]: Begin upload segment... Nov 11 08:10:09.715084585 (48h6m8.266555046s) [seg_pos={0 0}, upload=42408]: Done begin upload segment. Nov 11 08:10:09.715464638 (48h6m8.2669351s) [noise=true , address=45.91.116.8:4047 , piece_id=VP6VW2ZTNSKS35R47GGJZI4YSHNNFUFB4YQ3W3QFT5ENN7F73VQA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.715555221 (48h6m8.267025692s) [noise=true , address=145.79.227.30:4761 , piece_id=ASO5O5LM6CEF6DXGBVOCI3NOCWLGKDU2T5JHOL5ZPWEXMTRMKZIA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.718085410 (48h6m8.269555901s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.718721901 (48h6m8.270192402s) [noise=true , address=145.79.227.11:4729 , piece_id=CDIU7FFPCVQGDK3K5ZDUARJ6VGB7KIY2DEVL7ZRLW5EASIFEHKHA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.719505724 (48h6m8.270976195s) [noise=true , address=145.79.227.27:4689 , piece_id=KLTJYVDWHEE2BGLUSP2XT6L5R2C5OB7TF4DZLYBF6R5OG3N6AEFQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.720136295 (48h6m8.271606756s) [noise=true , address=145.79.227.14:4840 , piece_id=QASWXZB3E6E7VYGF24KZ37Q3LKT6F4WVFKTBV3YS73KYAWTBV7YQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.721166056 (48h6m8.272636517s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.721883453 (48h6m8.273353994s) [noise=true , address=45.91.116.4:4939 , piece_id=KMHEQSQTA2WEWMIO7JDQZUDLSRV4CA6OJUN2CYYUSA7SU2ZDQBCQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.722398593 (48h6m8.273869054s) [noise=true , address=77.90.31.18:4806 , piece_id=LNL5LRHH3OOELKEO4HND2K3GNVDPCBCGKBNJ75DCLEKRAOMTQLKA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.724101306 (48h6m8.275571767s) [noise=true , address=145.79.227.17:4970 , piece_id=F2227LWTX5PRCQIUEJNRJ3QS3GRJOIEJIZFV3KRF7BAEAPSQZR7A, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.726385264 (48h6m8.277855776s) [noise=true , address=45.91.116.11:4117 , piece_id=UOEFPOPOFIO5KXY2JOX54YHUWNMTCAFMXEBKAMMR4GSR5ZE6J6PA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.728323225 (48h6m8.279793686s) [noise=true , address=145.79.227.20:4437 , piece_id=AOK6MTRZKPCXTNHZWGBYJGRBYNHFWO7GSG2PT4DFSMDAWK75GY4A, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.730321541 (48h6m8.281792002s) [noise=true , address=145.79.227.9:4366 , piece_id=JDBR4PTBZ5RN6NTWPERVLFANUAZQF3FSGIENVRHSYN56ED5NT2AQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.731324300 (48h6m8.282794832s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.732226989 (48h6m8.28369749s) [noise=true , address=77.90.31.19:4841 , piece_id=6JVB2WIEA2STMNBVTM6D7IYFHQY4QJGXAYHJAQILMYZZDH2MWIVA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.733026101 (48h6m8.284496562s) [noise=true , address=145.79.227.21:4474 , piece_id=RNB5T6DIUCY7IV2ULLO2UJRDKUJXI34UM4ZCHPJB22DNQ47VZJVA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:11.398415378 (48h6m9.949885839s) [upload=42408]: Next returned nil segment Nov 11 08:10:11.398434043 (48h6m9.949904504s) [upload=42408]: Waiting for error group managing segments... Nov 11 08:10:11.796154582 (48h6m10.347625043s) [noise=true , address=77.90.31.18:4806 , piece_id=LNL5LRHH3OOELKEO4HND2K3GNVDPCBCGKBNJ75DCLEKRAOMTQLKA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.809043214 (48h6m10.360513675s) [noise=true , address=45.91.116.8:4047 , piece_id=VP6VW2ZTNSKS35R47GGJZI4YSHNNFUFB4YQ3W3QFT5ENN7F73VQA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.866335992 (48h6m10.417806453s) [noise=true , address=45.91.116.11:4117 , piece_id=UOEFPOPOFIO5KXY2JOX54YHUWNMTCAFMXEBKAMMR4GSR5ZE6J6PA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.932922087 (48h6m10.484392538s) [noise=true , address=77.90.31.19:4841 , piece_id=6JVB2WIEA2STMNBVTM6D7IYFHQY4QJGXAYHJAQILMYZZDH2MWIVA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:12.005905333 (48h6m10.557375794s) [noise=true , address=45.91.116.4:4939 , piece_id=KMHEQSQTA2WEWMIO7JDQZUDLSRV4CA6OJUN2CYYUSA7SU2ZDQBCQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:16.703757864 (48h6m15.255228326s) [noise=true , address=145.79.227.9:4366 , piece_id=JDBR4PTBZ5RN6NTWPERVLFANUAZQF3FSGIENVRHSYN56ED5NT2AQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:17.540298352 (48h6m16.091768813s) [noise=true , address=145.79.227.21:4474 , piece_id=RNB5T6DIUCY7IV2ULLO2UJRDKUJXI34UM4ZCHPJB22DNQ47VZJVA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:19.805526866 (48h6m18.356997337s) [noise=true , address=145.79.227.27:4689 , piece_id=KLTJYVDWHEE2BGLUSP2XT6L5R2C5OB7TF4DZLYBF6R5OG3N6AEFQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:24.397542313 (48h6m22.949012764s) [noise=true , address=145.79.227.17:4970 , piece_id=F2227LWTX5PRCQIUEJNRJ3QS3GRJOIEJIZFV3KRF7BAEAPSQZR7A, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:24.779686050 (48h6m23.331156511s) [noise=true , address=145.79.227.14:4840 , piece_id=QASWXZB3E6E7VYGF24KZ37Q3LKT6F4WVFKTBV3YS73KYAWTBV7YQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:25.298168925 (48h6m23.849639387s) [noise=true , address=145.79.227.11:4729 , piece_id=CDIU7FFPCVQGDK3K5ZDUARJ6VGB7KIY2DEVL7ZRLW5EASIFEHKHA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:35.044925402 (48h6m33.596395853s) [noise=true , address=145.79.227.30:4761 , piece_id=ASO5O5LM6CEF6DXGBVOCI3NOCWLGKDU2T5JHOL5ZPWEXMTRMKZIA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:37.306959937 (48h6m35.858430398s) [noise=true , address=145.79.227.20:4437 , piece_id=AOK6MTRZKPCXTNHZWGBYJGRBYNHFWO7GSG2PT4DFSMDAWK75GY4A, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:37.306987550 (48h6m35.858458011s) [seg_pos={0 0}, upload=42408]: Segment reached optimal threshold of 13 pieces. Nov 11 08:10:37.307209231 (48h6m35.858679702s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4): protocol: expected piece hash; context canceled; EOF Nov 11 08:10:37.307218759 (48h6m35.85868922s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307368936 (48h6m35.858839397s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw): protocol: expected piece hash; context canceled; EOF Nov 11 08:10:37.307376741 (48h6m35.858847202s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307561282 (48h6m35.859031743s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9): protocol: write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection Nov 11 08:10:37.307585969 (48h6m35.85905643s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307629221 (48h6m35.859099682s) [seg_pos={0 0}, upload=42408]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 11 08:10:37.307741005 (48h6m35.859211466s) [upload=42408]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 11 08:10:37.307750052 (48h6m35.859220513s) [upload=42408]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 11 08:10:37.612339386 (48h6m36.163809848s) [upload=42408]: Done uploading segments... Nov 11 08:10:37.612366317 (48h6m36.163836788s) [upload=42408]: Upload finished. err: <nil> Duration: 5.780872133s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 6.018861343s trace: https://jaeger.dev.storj.io/trace/49621e5e6beafe8e FAIL (2025-11-11 08:15:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 11 08:10:09.433180409 (48h6m7.98465087s) [upload=42408]: Starting upload Nov 11 08:10:09.433429884 (48h6m7.984900335s) [upload=42408]: Done starting upload Nov 11 08:10:09.433567967 (48h6m7.985038438s) [upload=42408]: Uploading segments... Nov 11 08:10:09.434191213 (48h6m7.985661674s) [upload=42408]: Got next segment. Inline: false Nov 11 08:10:09.434274032 (48h6m7.985744493s) [upload=42408]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 08:10:09.434282217 (48h6m7.985752688s) [upload=42408]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 08:10:09.714872160 (48h6m8.266342621s) [seg_pos={0 0}, upload=42408]: Begin upload segment... Nov 11 08:10:09.715084585 (48h6m8.266555046s) [seg_pos={0 0}, upload=42408]: Done begin upload segment. Nov 11 08:10:09.715464638 (48h6m8.2669351s) [noise=true , address=45.91.116.8:4047 , piece_id=VP6VW2ZTNSKS35R47GGJZI4YSHNNFUFB4YQ3W3QFT5ENN7F73VQA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.715555221 (48h6m8.267025692s) [noise=true , address=145.79.227.30:4761 , piece_id=ASO5O5LM6CEF6DXGBVOCI3NOCWLGKDU2T5JHOL5ZPWEXMTRMKZIA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.718085410 (48h6m8.269555901s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.718721901 (48h6m8.270192402s) [noise=true , address=145.79.227.11:4729 , piece_id=CDIU7FFPCVQGDK3K5ZDUARJ6VGB7KIY2DEVL7ZRLW5EASIFEHKHA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.719505724 (48h6m8.270976195s) [noise=true , address=145.79.227.27:4689 , piece_id=KLTJYVDWHEE2BGLUSP2XT6L5R2C5OB7TF4DZLYBF6R5OG3N6AEFQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.720136295 (48h6m8.271606756s) [noise=true , address=145.79.227.14:4840 , piece_id=QASWXZB3E6E7VYGF24KZ37Q3LKT6F4WVFKTBV3YS73KYAWTBV7YQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.721166056 (48h6m8.272636517s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.721883453 (48h6m8.273353994s) [noise=true , address=45.91.116.4:4939 , piece_id=KMHEQSQTA2WEWMIO7JDQZUDLSRV4CA6OJUN2CYYUSA7SU2ZDQBCQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.722398593 (48h6m8.273869054s) [noise=true , address=77.90.31.18:4806 , piece_id=LNL5LRHH3OOELKEO4HND2K3GNVDPCBCGKBNJ75DCLEKRAOMTQLKA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.724101306 (48h6m8.275571767s) [noise=true , address=145.79.227.17:4970 , piece_id=F2227LWTX5PRCQIUEJNRJ3QS3GRJOIEJIZFV3KRF7BAEAPSQZR7A, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.726385264 (48h6m8.277855776s) [noise=true , address=45.91.116.11:4117 , piece_id=UOEFPOPOFIO5KXY2JOX54YHUWNMTCAFMXEBKAMMR4GSR5ZE6J6PA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.728323225 (48h6m8.279793686s) [noise=true , address=145.79.227.20:4437 , piece_id=AOK6MTRZKPCXTNHZWGBYJGRBYNHFWO7GSG2PT4DFSMDAWK75GY4A, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.730321541 (48h6m8.281792002s) [noise=true , address=145.79.227.9:4366 , piece_id=JDBR4PTBZ5RN6NTWPERVLFANUAZQF3FSGIENVRHSYN56ED5NT2AQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.731324300 (48h6m8.282794832s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.732226989 (48h6m8.28369749s) [noise=true , address=77.90.31.19:4841 , piece_id=6JVB2WIEA2STMNBVTM6D7IYFHQY4QJGXAYHJAQILMYZZDH2MWIVA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.733026101 (48h6m8.284496562s) [noise=true , address=145.79.227.21:4474 , piece_id=RNB5T6DIUCY7IV2ULLO2UJRDKUJXI34UM4ZCHPJB22DNQ47VZJVA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:11.398415378 (48h6m9.949885839s) [upload=42408]: Next returned nil segment Nov 11 08:10:11.398434043 (48h6m9.949904504s) [upload=42408]: Waiting for error group managing segments... Nov 11 08:10:11.796154582 (48h6m10.347625043s) [noise=true , address=77.90.31.18:4806 , piece_id=LNL5LRHH3OOELKEO4HND2K3GNVDPCBCGKBNJ75DCLEKRAOMTQLKA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.809043214 (48h6m10.360513675s) [noise=true , address=45.91.116.8:4047 , piece_id=VP6VW2ZTNSKS35R47GGJZI4YSHNNFUFB4YQ3W3QFT5ENN7F73VQA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.866335992 (48h6m10.417806453s) [noise=true , address=45.91.116.11:4117 , piece_id=UOEFPOPOFIO5KXY2JOX54YHUWNMTCAFMXEBKAMMR4GSR5ZE6J6PA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.932922087 (48h6m10.484392538s) [noise=true , address=77.90.31.19:4841 , piece_id=6JVB2WIEA2STMNBVTM6D7IYFHQY4QJGXAYHJAQILMYZZDH2MWIVA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:12.005905333 (48h6m10.557375794s) [noise=true , address=45.91.116.4:4939 , piece_id=KMHEQSQTA2WEWMIO7JDQZUDLSRV4CA6OJUN2CYYUSA7SU2ZDQBCQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:16.703757864 (48h6m15.255228326s) [noise=true , address=145.79.227.9:4366 , piece_id=JDBR4PTBZ5RN6NTWPERVLFANUAZQF3FSGIENVRHSYN56ED5NT2AQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:17.540298352 (48h6m16.091768813s) [noise=true , address=145.79.227.21:4474 , piece_id=RNB5T6DIUCY7IV2ULLO2UJRDKUJXI34UM4ZCHPJB22DNQ47VZJVA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:19.805526866 (48h6m18.356997337s) [noise=true , address=145.79.227.27:4689 , piece_id=KLTJYVDWHEE2BGLUSP2XT6L5R2C5OB7TF4DZLYBF6R5OG3N6AEFQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:24.397542313 (48h6m22.949012764s) [noise=true , address=145.79.227.17:4970 , piece_id=F2227LWTX5PRCQIUEJNRJ3QS3GRJOIEJIZFV3KRF7BAEAPSQZR7A, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:24.779686050 (48h6m23.331156511s) [noise=true , address=145.79.227.14:4840 , piece_id=QASWXZB3E6E7VYGF24KZ37Q3LKT6F4WVFKTBV3YS73KYAWTBV7YQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:25.298168925 (48h6m23.849639387s) [noise=true , address=145.79.227.11:4729 , piece_id=CDIU7FFPCVQGDK3K5ZDUARJ6VGB7KIY2DEVL7ZRLW5EASIFEHKHA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:35.044925402 (48h6m33.596395853s) [noise=true , address=145.79.227.30:4761 , piece_id=ASO5O5LM6CEF6DXGBVOCI3NOCWLGKDU2T5JHOL5ZPWEXMTRMKZIA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:37.306959937 (48h6m35.858430398s) [noise=true , address=145.79.227.20:4437 , piece_id=AOK6MTRZKPCXTNHZWGBYJGRBYNHFWO7GSG2PT4DFSMDAWK75GY4A, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:37.306987550 (48h6m35.858458011s) [seg_pos={0 0}, upload=42408]: Segment reached optimal threshold of 13 pieces. Nov 11 08:10:37.307209231 (48h6m35.858679702s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4): protocol: expected piece hash; context canceled; EOF Nov 11 08:10:37.307218759 (48h6m35.85868922s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307368936 (48h6m35.858839397s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw): protocol: expected piece hash; context canceled; EOF Nov 11 08:10:37.307376741 (48h6m35.858847202s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307561282 (48h6m35.859031743s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9): protocol: write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection Nov 11 08:10:37.307585969 (48h6m35.85905643s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307629221 (48h6m35.859099682s) [seg_pos={0 0}, upload=42408]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 11 08:10:37.307741005 (48h6m35.859211466s) [upload=42408]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 11 08:10:37.307750052 (48h6m35.859220513s) [upload=42408]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 11 08:10:37.612339386 (48h6m36.163809848s) [upload=42408]: Done uploading segments... Nov 11 08:10:37.612366317 (48h6m36.163836788s) [upload=42408]: Upload finished. err: <nil> Duration: 6.018893263s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 5.968278708s trace: https://jaeger.dev.storj.io/trace/49621e5e6beafe8e FAIL (2025-11-11 08:14:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 11 08:10:09.433180409 (48h6m7.98465087s) [upload=42408]: Starting upload Nov 11 08:10:09.433429884 (48h6m7.984900335s) [upload=42408]: Done starting upload Nov 11 08:10:09.433567967 (48h6m7.985038438s) [upload=42408]: Uploading segments... Nov 11 08:10:09.434191213 (48h6m7.985661674s) [upload=42408]: Got next segment. Inline: false Nov 11 08:10:09.434274032 (48h6m7.985744493s) [upload=42408]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 08:10:09.434282217 (48h6m7.985752688s) [upload=42408]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 08:10:09.714872160 (48h6m8.266342621s) [seg_pos={0 0}, upload=42408]: Begin upload segment... Nov 11 08:10:09.715084585 (48h6m8.266555046s) [seg_pos={0 0}, upload=42408]: Done begin upload segment. Nov 11 08:10:09.715464638 (48h6m8.2669351s) [noise=true , address=45.91.116.8:4047 , piece_id=VP6VW2ZTNSKS35R47GGJZI4YSHNNFUFB4YQ3W3QFT5ENN7F73VQA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.715555221 (48h6m8.267025692s) [noise=true , address=145.79.227.30:4761 , piece_id=ASO5O5LM6CEF6DXGBVOCI3NOCWLGKDU2T5JHOL5ZPWEXMTRMKZIA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.718085410 (48h6m8.269555901s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.718721901 (48h6m8.270192402s) [noise=true , address=145.79.227.11:4729 , piece_id=CDIU7FFPCVQGDK3K5ZDUARJ6VGB7KIY2DEVL7ZRLW5EASIFEHKHA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.719505724 (48h6m8.270976195s) [noise=true , address=145.79.227.27:4689 , piece_id=KLTJYVDWHEE2BGLUSP2XT6L5R2C5OB7TF4DZLYBF6R5OG3N6AEFQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.720136295 (48h6m8.271606756s) [noise=true , address=145.79.227.14:4840 , piece_id=QASWXZB3E6E7VYGF24KZ37Q3LKT6F4WVFKTBV3YS73KYAWTBV7YQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.721166056 (48h6m8.272636517s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.721883453 (48h6m8.273353994s) [noise=true , address=45.91.116.4:4939 , piece_id=KMHEQSQTA2WEWMIO7JDQZUDLSRV4CA6OJUN2CYYUSA7SU2ZDQBCQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.722398593 (48h6m8.273869054s) [noise=true , address=77.90.31.18:4806 , piece_id=LNL5LRHH3OOELKEO4HND2K3GNVDPCBCGKBNJ75DCLEKRAOMTQLKA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.724101306 (48h6m8.275571767s) [noise=true , address=145.79.227.17:4970 , piece_id=F2227LWTX5PRCQIUEJNRJ3QS3GRJOIEJIZFV3KRF7BAEAPSQZR7A, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.726385264 (48h6m8.277855776s) [noise=true , address=45.91.116.11:4117 , piece_id=UOEFPOPOFIO5KXY2JOX54YHUWNMTCAFMXEBKAMMR4GSR5ZE6J6PA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.728323225 (48h6m8.279793686s) [noise=true , address=145.79.227.20:4437 , piece_id=AOK6MTRZKPCXTNHZWGBYJGRBYNHFWO7GSG2PT4DFSMDAWK75GY4A, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.730321541 (48h6m8.281792002s) [noise=true , address=145.79.227.9:4366 , piece_id=JDBR4PTBZ5RN6NTWPERVLFANUAZQF3FSGIENVRHSYN56ED5NT2AQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.731324300 (48h6m8.282794832s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.732226989 (48h6m8.28369749s) [noise=true , address=77.90.31.19:4841 , piece_id=6JVB2WIEA2STMNBVTM6D7IYFHQY4QJGXAYHJAQILMYZZDH2MWIVA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.733026101 (48h6m8.284496562s) [noise=true , address=145.79.227.21:4474 , piece_id=RNB5T6DIUCY7IV2ULLO2UJRDKUJXI34UM4ZCHPJB22DNQ47VZJVA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:11.398415378 (48h6m9.949885839s) [upload=42408]: Next returned nil segment Nov 11 08:10:11.398434043 (48h6m9.949904504s) [upload=42408]: Waiting for error group managing segments... Nov 11 08:10:11.796154582 (48h6m10.347625043s) [noise=true , address=77.90.31.18:4806 , piece_id=LNL5LRHH3OOELKEO4HND2K3GNVDPCBCGKBNJ75DCLEKRAOMTQLKA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.809043214 (48h6m10.360513675s) [noise=true , address=45.91.116.8:4047 , piece_id=VP6VW2ZTNSKS35R47GGJZI4YSHNNFUFB4YQ3W3QFT5ENN7F73VQA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.866335992 (48h6m10.417806453s) [noise=true , address=45.91.116.11:4117 , piece_id=UOEFPOPOFIO5KXY2JOX54YHUWNMTCAFMXEBKAMMR4GSR5ZE6J6PA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.932922087 (48h6m10.484392538s) [noise=true , address=77.90.31.19:4841 , piece_id=6JVB2WIEA2STMNBVTM6D7IYFHQY4QJGXAYHJAQILMYZZDH2MWIVA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:12.005905333 (48h6m10.557375794s) [noise=true , address=45.91.116.4:4939 , piece_id=KMHEQSQTA2WEWMIO7JDQZUDLSRV4CA6OJUN2CYYUSA7SU2ZDQBCQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:16.703757864 (48h6m15.255228326s) [noise=true , address=145.79.227.9:4366 , piece_id=JDBR4PTBZ5RN6NTWPERVLFANUAZQF3FSGIENVRHSYN56ED5NT2AQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:17.540298352 (48h6m16.091768813s) [noise=true , address=145.79.227.21:4474 , piece_id=RNB5T6DIUCY7IV2ULLO2UJRDKUJXI34UM4ZCHPJB22DNQ47VZJVA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:19.805526866 (48h6m18.356997337s) [noise=true , address=145.79.227.27:4689 , piece_id=KLTJYVDWHEE2BGLUSP2XT6L5R2C5OB7TF4DZLYBF6R5OG3N6AEFQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:24.397542313 (48h6m22.949012764s) [noise=true , address=145.79.227.17:4970 , piece_id=F2227LWTX5PRCQIUEJNRJ3QS3GRJOIEJIZFV3KRF7BAEAPSQZR7A, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:24.779686050 (48h6m23.331156511s) [noise=true , address=145.79.227.14:4840 , piece_id=QASWXZB3E6E7VYGF24KZ37Q3LKT6F4WVFKTBV3YS73KYAWTBV7YQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:25.298168925 (48h6m23.849639387s) [noise=true , address=145.79.227.11:4729 , piece_id=CDIU7FFPCVQGDK3K5ZDUARJ6VGB7KIY2DEVL7ZRLW5EASIFEHKHA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:35.044925402 (48h6m33.596395853s) [noise=true , address=145.79.227.30:4761 , piece_id=ASO5O5LM6CEF6DXGBVOCI3NOCWLGKDU2T5JHOL5ZPWEXMTRMKZIA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:37.306959937 (48h6m35.858430398s) [noise=true , address=145.79.227.20:4437 , piece_id=AOK6MTRZKPCXTNHZWGBYJGRBYNHFWO7GSG2PT4DFSMDAWK75GY4A, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:37.306987550 (48h6m35.858458011s) [seg_pos={0 0}, upload=42408]: Segment reached optimal threshold of 13 pieces. Nov 11 08:10:37.307209231 (48h6m35.858679702s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4): protocol: expected piece hash; context canceled; EOF Nov 11 08:10:37.307218759 (48h6m35.85868922s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307368936 (48h6m35.858839397s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw): protocol: expected piece hash; context canceled; EOF Nov 11 08:10:37.307376741 (48h6m35.858847202s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307561282 (48h6m35.859031743s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9): protocol: write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection Nov 11 08:10:37.307585969 (48h6m35.85905643s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307629221 (48h6m35.859099682s) [seg_pos={0 0}, upload=42408]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 11 08:10:37.307741005 (48h6m35.859211466s) [upload=42408]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 11 08:10:37.307750052 (48h6m35.859220513s) [upload=42408]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 11 08:10:37.612339386 (48h6m36.163809848s) [upload=42408]: Done uploading segments... Nov 11 08:10:37.612366317 (48h6m36.163836788s) [upload=42408]: Upload finished. err: <nil> Duration: 5.968325997s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 6.041553367s trace: https://jaeger.dev.storj.io/trace/49621e5e6beafe8e FAIL (2025-11-11 08:13:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 11 08:10:09.433180409 (48h6m7.98465087s) [upload=42408]: Starting upload Nov 11 08:10:09.433429884 (48h6m7.984900335s) [upload=42408]: Done starting upload Nov 11 08:10:09.433567967 (48h6m7.985038438s) [upload=42408]: Uploading segments... Nov 11 08:10:09.434191213 (48h6m7.985661674s) [upload=42408]: Got next segment. Inline: false Nov 11 08:10:09.434274032 (48h6m7.985744493s) [upload=42408]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 08:10:09.434282217 (48h6m7.985752688s) [upload=42408]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 08:10:09.714872160 (48h6m8.266342621s) [seg_pos={0 0}, upload=42408]: Begin upload segment... Nov 11 08:10:09.715084585 (48h6m8.266555046s) [seg_pos={0 0}, upload=42408]: Done begin upload segment. Nov 11 08:10:09.715464638 (48h6m8.2669351s) [noise=true , address=45.91.116.8:4047 , piece_id=VP6VW2ZTNSKS35R47GGJZI4YSHNNFUFB4YQ3W3QFT5ENN7F73VQA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.715555221 (48h6m8.267025692s) [noise=true , address=145.79.227.30:4761 , piece_id=ASO5O5LM6CEF6DXGBVOCI3NOCWLGKDU2T5JHOL5ZPWEXMTRMKZIA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.718085410 (48h6m8.269555901s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.718721901 (48h6m8.270192402s) [noise=true , address=145.79.227.11:4729 , piece_id=CDIU7FFPCVQGDK3K5ZDUARJ6VGB7KIY2DEVL7ZRLW5EASIFEHKHA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.719505724 (48h6m8.270976195s) [noise=true , address=145.79.227.27:4689 , piece_id=KLTJYVDWHEE2BGLUSP2XT6L5R2C5OB7TF4DZLYBF6R5OG3N6AEFQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.720136295 (48h6m8.271606756s) [noise=true , address=145.79.227.14:4840 , piece_id=QASWXZB3E6E7VYGF24KZ37Q3LKT6F4WVFKTBV3YS73KYAWTBV7YQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.721166056 (48h6m8.272636517s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.721883453 (48h6m8.273353994s) [noise=true , address=45.91.116.4:4939 , piece_id=KMHEQSQTA2WEWMIO7JDQZUDLSRV4CA6OJUN2CYYUSA7SU2ZDQBCQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.722398593 (48h6m8.273869054s) [noise=true , address=77.90.31.18:4806 , piece_id=LNL5LRHH3OOELKEO4HND2K3GNVDPCBCGKBNJ75DCLEKRAOMTQLKA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.724101306 (48h6m8.275571767s) [noise=true , address=145.79.227.17:4970 , piece_id=F2227LWTX5PRCQIUEJNRJ3QS3GRJOIEJIZFV3KRF7BAEAPSQZR7A, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.726385264 (48h6m8.277855776s) [noise=true , address=45.91.116.11:4117 , piece_id=UOEFPOPOFIO5KXY2JOX54YHUWNMTCAFMXEBKAMMR4GSR5ZE6J6PA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.728323225 (48h6m8.279793686s) [noise=true , address=145.79.227.20:4437 , piece_id=AOK6MTRZKPCXTNHZWGBYJGRBYNHFWO7GSG2PT4DFSMDAWK75GY4A, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.730321541 (48h6m8.281792002s) [noise=true , address=145.79.227.9:4366 , piece_id=JDBR4PTBZ5RN6NTWPERVLFANUAZQF3FSGIENVRHSYN56ED5NT2AQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.731324300 (48h6m8.282794832s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.732226989 (48h6m8.28369749s) [noise=true , address=77.90.31.19:4841 , piece_id=6JVB2WIEA2STMNBVTM6D7IYFHQY4QJGXAYHJAQILMYZZDH2MWIVA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.733026101 (48h6m8.284496562s) [noise=true , address=145.79.227.21:4474 , piece_id=RNB5T6DIUCY7IV2ULLO2UJRDKUJXI34UM4ZCHPJB22DNQ47VZJVA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:11.398415378 (48h6m9.949885839s) [upload=42408]: Next returned nil segment Nov 11 08:10:11.398434043 (48h6m9.949904504s) [upload=42408]: Waiting for error group managing segments... Nov 11 08:10:11.796154582 (48h6m10.347625043s) [noise=true , address=77.90.31.18:4806 , piece_id=LNL5LRHH3OOELKEO4HND2K3GNVDPCBCGKBNJ75DCLEKRAOMTQLKA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.809043214 (48h6m10.360513675s) [noise=true , address=45.91.116.8:4047 , piece_id=VP6VW2ZTNSKS35R47GGJZI4YSHNNFUFB4YQ3W3QFT5ENN7F73VQA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.866335992 (48h6m10.417806453s) [noise=true , address=45.91.116.11:4117 , piece_id=UOEFPOPOFIO5KXY2JOX54YHUWNMTCAFMXEBKAMMR4GSR5ZE6J6PA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.932922087 (48h6m10.484392538s) [noise=true , address=77.90.31.19:4841 , piece_id=6JVB2WIEA2STMNBVTM6D7IYFHQY4QJGXAYHJAQILMYZZDH2MWIVA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:12.005905333 (48h6m10.557375794s) [noise=true , address=45.91.116.4:4939 , piece_id=KMHEQSQTA2WEWMIO7JDQZUDLSRV4CA6OJUN2CYYUSA7SU2ZDQBCQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:16.703757864 (48h6m15.255228326s) [noise=true , address=145.79.227.9:4366 , piece_id=JDBR4PTBZ5RN6NTWPERVLFANUAZQF3FSGIENVRHSYN56ED5NT2AQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:17.540298352 (48h6m16.091768813s) [noise=true , address=145.79.227.21:4474 , piece_id=RNB5T6DIUCY7IV2ULLO2UJRDKUJXI34UM4ZCHPJB22DNQ47VZJVA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:19.805526866 (48h6m18.356997337s) [noise=true , address=145.79.227.27:4689 , piece_id=KLTJYVDWHEE2BGLUSP2XT6L5R2C5OB7TF4DZLYBF6R5OG3N6AEFQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:24.397542313 (48h6m22.949012764s) [noise=true , address=145.79.227.17:4970 , piece_id=F2227LWTX5PRCQIUEJNRJ3QS3GRJOIEJIZFV3KRF7BAEAPSQZR7A, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:24.779686050 (48h6m23.331156511s) [noise=true , address=145.79.227.14:4840 , piece_id=QASWXZB3E6E7VYGF24KZ37Q3LKT6F4WVFKTBV3YS73KYAWTBV7YQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:25.298168925 (48h6m23.849639387s) [noise=true , address=145.79.227.11:4729 , piece_id=CDIU7FFPCVQGDK3K5ZDUARJ6VGB7KIY2DEVL7ZRLW5EASIFEHKHA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:35.044925402 (48h6m33.596395853s) [noise=true , address=145.79.227.30:4761 , piece_id=ASO5O5LM6CEF6DXGBVOCI3NOCWLGKDU2T5JHOL5ZPWEXMTRMKZIA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:37.306959937 (48h6m35.858430398s) [noise=true , address=145.79.227.20:4437 , piece_id=AOK6MTRZKPCXTNHZWGBYJGRBYNHFWO7GSG2PT4DFSMDAWK75GY4A, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:37.306987550 (48h6m35.858458011s) [seg_pos={0 0}, upload=42408]: Segment reached optimal threshold of 13 pieces. Nov 11 08:10:37.307209231 (48h6m35.858679702s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4): protocol: expected piece hash; context canceled; EOF Nov 11 08:10:37.307218759 (48h6m35.85868922s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307368936 (48h6m35.858839397s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw): protocol: expected piece hash; context canceled; EOF Nov 11 08:10:37.307376741 (48h6m35.858847202s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307561282 (48h6m35.859031743s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9): protocol: write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection Nov 11 08:10:37.307585969 (48h6m35.85905643s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307629221 (48h6m35.859099682s) [seg_pos={0 0}, upload=42408]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 11 08:10:37.307741005 (48h6m35.859211466s) [upload=42408]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 11 08:10:37.307750052 (48h6m35.859220513s) [upload=42408]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 11 08:10:37.612339386 (48h6m36.163809848s) [upload=42408]: Done uploading segments... Nov 11 08:10:37.612366317 (48h6m36.163836788s) [upload=42408]: Upload finished. err: <nil> Duration: 6.041580609s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)