Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [.X.X........X.......X..X.XXXXXXX...X.........XXXX..XXX.XXXX.] duration: 24.921741694s trace: https://jaeger.dev.storj.io/trace/606e36023014ce41 FAIL (2025-12-27 18:41:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: HNUC4AKALPPNRE2LSDPEG2NZ5CQE36636XQHXDIFRSQR42EVO6KA) context deadline exceeded; piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: BJYRL52FMICD2B2YCG2GP3HUKO6GBW4G3VE7QGQBNWDWY3A4YGDQ) context deadline exceeded ======LOGS====== Dec 27 18:40:29.563290118 (531h11m29.434373163s) [upload=465855]: Starting upload Dec 27 18:40:29.563345193 (531h11m29.434428228s) [upload=465855]: Done starting upload Dec 27 18:40:29.563485801 (531h11m29.434568836s) [upload=465855]: Uploading segments... Dec 27 18:40:29.563587023 (531h11m29.434670068s) [upload=465855]: Got next segment. Inline: false Dec 27 18:40:29.563645925 (531h11m29.43472898s) [upload=465855]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 18:40:29.563655424 (531h11m29.434738469s) [upload=465855]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 18:40:29.811775988 (531h11m29.682859023s) [seg_pos={0 0}, upload=465855]: Begin upload segment... Dec 27 18:40:29.811836694 (531h11m29.682919739s) [seg_pos={0 0}, upload=465855]: Done begin upload segment. Dec 27 18:40:29.811933950 (531h11m29.683016985s) [noise=true , address=45.91.116.14:4189 , piece_id=TOXWDN4UODON25EMZURZXGQAAGCK32TEANRN6GVIWBXTTUVOMX6A, seg_pos={0 0}, upload=465855]: Uploading piece... Dec 27 18:40:29.811964738 (531h11m29.683047783s) [noise=true , address=145.79.227.41:4086 , piece_id=XPRMHYYSCSBAZBL55HYZTOJJVTIHD4VPVVJ24CUQ2FRVF5TVBZ4A, seg_pos={0 0}, upload=465855]: Uploading piece... Dec 27 18:40:29.812427900 (531h11m29.683510945s) [noise=true , address=145.79.227.42:4122 , piece_id=CNI3RHH7IPV5FITXI366RRMQJNWVQGYKN7A6QXRNTRURB73BK3GA, seg_pos={0 0}, upload=465855]: Uploading piece... Dec 27 18:40:29.812841097 (531h11m29.683924132s) [noise=true , address=145.79.227.15:4873 , piece_id=SBLCW4WOZ6IXK3SNVGLFMSIWHEGKW4EPLW65E52SWEL7NXYQWQQQ, seg_pos={0 0}, upload=465855]: Uploading piece... Dec 27 18:40:29.813306503 (531h11m29.684389538s) [noise=true , address=145.79.227.46:4602 , piece_id=KCLL26FUGLSHE2LR3ZQRXCZRF2PJL5LZUBNNQW3F7W52TI6JIOKQ, seg_pos={0 0}, upload=465855]: Uploading piece... Dec 27 18:40:29.813752933 (531h11m29.684835968s) [noise=true , address=45.91.116.10:4084 , piece_id=BJYRL52FMICD2B2YCG2GP3HUKO6GBW4G3VE7QGQBNWDWY3A4YGDQ, seg_pos={0 0}, upload=465855]: Uploading piece... Dec 27 18:40:29.814259458 (531h11m29.685342503s) [noise=true , address=77.90.31.21:4878 , piece_id=RD3NULS7IWTONXH53YLOUCEZAUX6V42WNJRQ345WFJE5N5LYPGRA, seg_pos={0 0}, upload=465855]: Uploading piece... Dec 27 18:40:29.814829623 (531h11m29.685912668s) [noise=true , address=145.79.227.16:4910 , piece_id=KX6EFMCV7LGXAS4YXVSMYECHJD7SN2SNTBNYGSOKZN4SSG5DQACA, seg_pos={0 0}, upload=465855]: Uploading piece... Dec 27 18:40:29.815877168 (531h11m29.686960203s) [noise=true , address=145.79.227.50:4375 , piece_id=Z5I2IFVSWCFIZEZ7BEJTPORMQG3RVLDBFW4ZUWR6E3M4TI7SQHLA, seg_pos={0 0}, upload=465855]: Uploading piece... Dec 27 18:40:29.817065270 (531h11m29.688148305s) [noise=true , address=145.79.227.24:4581 , piece_id=S3KMQT24OT2MLK77K5WNXGKU4YB3O5FJIRXONN7FHINM6GDWT4CA, seg_pos={0 0}, upload=465855]: Uploading piece... Dec 27 18:40:29.819017568 (531h11m29.690100613s) [noise=true , address=45.91.116.12:4228 , piece_id=TNPP2INNIO3IYTAJQK53FFKOFZVDZP5QWPFDZUHZLO3C3DEAG2CA, seg_pos={0 0}, upload=465855]: Uploading piece... Dec 27 18:40:29.820659485 (531h11m29.69174253s) [noise=true , address=45.91.116.8:4046 , piece_id=BEPXLRAMVWMBO443W33Y77MXVIJC4Q4XGCRELBL65QEASW6YOKKQ, seg_pos={0 0}, upload=465855]: Uploading piece... Dec 27 18:40:29.822226338 (531h11m29.693309383s) [noise=true , address=145.79.227.25:4619 , piece_id=52MTBOEJBUBTWD3TPIXNFII65F76HJ3CFRZCB3JAEFCYHQ2BOTMA, seg_pos={0 0}, upload=465855]: Uploading piece... Dec 27 18:40:29.823857914 (531h11m29.694940959s) [noise=true , address=145.79.227.9:4365 , piece_id=2FPIQV4FGURGE4NCE7CQHSR3H3ZT6N2DWV5OED4G2MJRLZ7RGZNA, seg_pos={0 0}, upload=465855]: Uploading piece... Dec 27 18:40:29.823966993 (531h11m29.695050028s) [noise=true , address=45.91.116.4:4939 , piece_id=Z74ZBVFQ52QGK4NQQVDCH5B3VQBXUQR2HE3UH5IR24SSEV5U5JBQ, seg_pos={0 0}, upload=465855]: Uploading piece... Dec 27 18:40:29.825101693 (531h11m29.696184738s) [noise=true , address=45.91.116.6:4974 , piece_id=HNUC4AKALPPNRE2LSDPEG2NZ5CQE36636XQHXDIFRSQR42EVO6KA, seg_pos={0 0}, upload=465855]: Uploading piece... Dec 27 18:40:31.504086507 (531h11m31.375169552s) [upload=465855]: Next returned nil segment Dec 27 18:40:31.504112516 (531h11m31.375195551s) [upload=465855]: Waiting for error group managing segments... Dec 27 18:40:31.802290870 (531h11m31.673373905s) [noise=true , address=145.79.227.15:4873 , piece_id=SBLCW4WOZ6IXK3SNVGLFMSIWHEGKW4EPLW65E52SWEL7NXYQWQQQ, seg_pos={0 0}, upload=465855]: Done uploading piece. err: <nil> Dec 27 18:40:31.808706406 (531h11m31.679789441s) [noise=true , address=145.79.227.16:4910 , piece_id=KX6EFMCV7LGXAS4YXVSMYECHJD7SN2SNTBNYGSOKZN4SSG5DQACA, seg_pos={0 0}, upload=465855]: Done uploading piece. err: <nil> Dec 27 18:40:31.833367526 (531h11m31.704450571s) [noise=true , address=145.79.227.42:4122 , piece_id=CNI3RHH7IPV5FITXI366RRMQJNWVQGYKN7A6QXRNTRURB73BK3GA, seg_pos={0 0}, upload=465855]: Done uploading piece. err: <nil> Dec 27 18:40:31.834781809 (531h11m31.705864844s) [noise=true , address=145.79.227.9:4365 , piece_id=2FPIQV4FGURGE4NCE7CQHSR3H3ZT6N2DWV5OED4G2MJRLZ7RGZNA, seg_pos={0 0}, upload=465855]: Done uploading piece. err: <nil> Dec 27 18:40:31.839940151 (531h11m31.711023197s) [noise=true , address=145.79.227.25:4619 , piece_id=52MTBOEJBUBTWD3TPIXNFII65F76HJ3CFRZCB3JAEFCYHQ2BOTMA, seg_pos={0 0}, upload=465855]: Done uploading piece. err: <nil> Dec 27 18:40:31.841439076 (531h11m31.712522111s) [noise=true , address=145.79.227.41:4086 , piece_id=XPRMHYYSCSBAZBL55HYZTOJJVTIHD4VPVVJ24CUQ2FRVF5TVBZ4A, seg_pos={0 0}, upload=465855]: Done uploading piece. err: <nil> Dec 27 18:40:31.843142389 (531h11m31.714225424s) [noise=true , address=145.79.227.50:4375 , piece_id=Z5I2IFVSWCFIZEZ7BEJTPORMQG3RVLDBFW4ZUWR6E3M4TI7SQHLA, seg_pos={0 0}, upload=465855]: Done uploading piece. err: <nil> Dec 27 18:40:31.855166656 (531h11m31.726249701s) [noise=true , address=145.79.227.24:4581 , piece_id=S3KMQT24OT2MLK77K5WNXGKU4YB3O5FJIRXONN7FHINM6GDWT4CA, seg_pos={0 0}, upload=465855]: Done uploading piece. err: <nil> Dec 27 18:40:31.858853947 (531h11m31.729936982s) [noise=true , address=145.79.227.46:4602 , piece_id=KCLL26FUGLSHE2LR3ZQRXCZRF2PJL5LZUBNNQW3F7W52TI6JIOKQ, seg_pos={0 0}, upload=465855]: Done uploading piece. err: <nil> Dec 27 18:40:32.040621204 (531h11m31.911704239s) [noise=true , address=77.90.31.21:4878 , piece_id=RD3NULS7IWTONXH53YLOUCEZAUX6V42WNJRQ345WFJE5N5LYPGRA, seg_pos={0 0}, upload=465855]: Done uploading piece. err: <nil> Dec 27 18:40:32.075657128 (531h11m31.946740163s) [noise=true , address=45.91.116.10:4084 , piece_id=BJYRL52FMICD2B2YCG2GP3HUKO6GBW4G3VE7QGQBNWDWY3A4YGDQ, seg_pos={0 0}, upload=465855]: Done uploading piece. err: <nil> Dec 27 18:40:32.086606037 (531h11m31.957689072s) [noise=true , address=45.91.116.6:4974 , piece_id=HNUC4AKALPPNRE2LSDPEG2NZ5CQE36636XQHXDIFRSQR42EVO6KA, seg_pos={0 0}, upload=465855]: Done uploading piece. err: <nil> Dec 27 18:40:32.100544979 (531h11m31.971628014s) [noise=true , address=45.91.116.12:4228 , piece_id=TNPP2INNIO3IYTAJQK53FFKOFZVDZP5QWPFDZUHZLO3C3DEAG2CA, seg_pos={0 0}, upload=465855]: Done uploading piece. err: <nil> Dec 27 18:40:32.100567292 (531h11m31.971650327s) [seg_pos={0 0}, upload=465855]: Segment reached optimal threshold of 13 pieces. Dec 27 18:40:32.100854248 (531h11m31.971937293s) [noise=true , address=45.91.116.8:4046 , piece_id=BEPXLRAMVWMBO443W33Y77MXVIJC4Q4XGCRELBL65QEASW6YOKKQ, seg_pos={0 0}, upload=465855]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 27 18:40:32.100864909 (531h11m31.971947954s) [noise=true , address=45.91.116.8:4046 , piece_id=BEPXLRAMVWMBO443W33Y77MXVIJC4Q4XGCRELBL65QEASW6YOKKQ, seg_pos={0 0}, upload=465855]: Long tail cancellation due to optimal threshold Dec 27 18:40:32.101085609 (531h11m31.972168674s) [noise=true , address=45.91.116.4:4939 , piece_id=Z74ZBVFQ52QGK4NQQVDCH5B3VQBXUQR2HE3UH5IR24SSEV5U5JBQ, seg_pos={0 0}, upload=465855]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 27 18:40:32.101095637 (531h11m31.972178672s) [noise=true , address=45.91.116.4:4939 , piece_id=Z74ZBVFQ52QGK4NQQVDCH5B3VQBXUQR2HE3UH5IR24SSEV5U5JBQ, seg_pos={0 0}, upload=465855]: Long tail cancellation due to optimal threshold Dec 27 18:40:32.101207340 (531h11m31.972290385s) [noise=true , address=45.91.116.14:4189 , piece_id=TOXWDN4UODON25EMZURZXGQAAGCK32TEANRN6GVIWBXTTUVOMX6A, seg_pos={0 0}, upload=465855]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 27 18:40:32.101229643 (531h11m31.972312688s) [noise=true , address=45.91.116.14:4189 , piece_id=TOXWDN4UODON25EMZURZXGQAAGCK32TEANRN6GVIWBXTTUVOMX6A, seg_pos={0 0}, upload=465855]: Long tail cancellation due to optimal threshold Dec 27 18:40:32.101279719 (531h11m31.972362754s) [seg_pos={0 0}, upload=465855]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 18:40:32.101457738 (531h11m31.972540773s) [upload=465855]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 18:40:32.101466835 (531h11m31.97254987s) [upload=465855]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 18:40:32.401875876 (531h11m32.272958972s) [upload=465855]: Done uploading segments... Dec 27 18:40:32.401904732 (531h11m32.272987817s) [upload=465855]: Upload finished. err: <nil> Duration: 24.921762794s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [..X.X........X.......X..X.XXXXXXX...X.........XXXX..XXX.XXXX] duration: 30.002935402s trace: https://jaeger.dev.storj.io/trace/606e36023014ce41 FAIL (2025-12-27 18:40:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: HNUC4AKALPPNRE2LSDPEG2NZ5CQE36636XQHXDIFRSQR42EVO6KA) context deadline exceeded; piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: BJYRL52FMICD2B2YCG2GP3HUKO6GBW4G3VE7QGQBNWDWY3A4YGDQ) context deadline exceeded ======LOGS====== Dec 27 18:40:29.563290118 (531h11m29.434373163s) [upload=465855]: Starting upload Dec 27 18:40:29.563345193 (531h11m29.434428228s) [upload=465855]: Done starting upload Dec 27 18:40:29.563485801 (531h11m29.434568836s) [upload=465855]: Uploading segments... Dec 27 18:40:29.563587023 (531h11m29.434670068s) [upload=465855]: Got next segment. Inline: false Dec 27 18:40:29.563645925 (531h11m29.43472898s) [upload=465855]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 18:40:29.563655424 (531h11m29.434738469s) [upload=465855]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 18:40:29.811775988 (531h11m29.682859023s) [seg_pos={0 0}, upload=465855]: Begin upload segment... Dec 27 18:40:29.811836694 (531h11m29.682919739s) [seg_pos={0 0}, upload=465855]: Done begin upload segment. Dec 27 18:40:29.811933950 (531h11m29.683016985s) [noise=true , address=45.91.116.14:4189 , piece_id=TOXWDN4UODON25EMZURZXGQAAGCK32TEANRN6GVIWBXTTUVOMX6A, seg_pos={0 0}, upload=465855]: Uploading piece... Dec 27 18:40:29.811964738 (531h11m29.683047783s) [noise=true , address=145.79.227.41:4086 , piece_id=XPRMHYYSCSBAZBL55HYZTOJJVTIHD4VPVVJ24CUQ2FRVF5TVBZ4A, seg_pos={0 0}, upload=465855]: Uploading piece... Dec 27 18:40:29.812427900 (531h11m29.683510945s) [noise=true , address=145.79.227.42:4122 , piece_id=CNI3RHH7IPV5FITXI366RRMQJNWVQGYKN7A6QXRNTRURB73BK3GA, seg_pos={0 0}, upload=465855]: Uploading piece... Dec 27 18:40:29.812841097 (531h11m29.683924132s) [noise=true , address=145.79.227.15:4873 , piece_id=SBLCW4WOZ6IXK3SNVGLFMSIWHEGKW4EPLW65E52SWEL7NXYQWQQQ, seg_pos={0 0}, upload=465855]: Uploading piece... Dec 27 18:40:29.813306503 (531h11m29.684389538s) [noise=true , address=145.79.227.46:4602 , piece_id=KCLL26FUGLSHE2LR3ZQRXCZRF2PJL5LZUBNNQW3F7W52TI6JIOKQ, seg_pos={0 0}, upload=465855]: Uploading piece... Dec 27 18:40:29.813752933 (531h11m29.684835968s) [noise=true , address=45.91.116.10:4084 , piece_id=BJYRL52FMICD2B2YCG2GP3HUKO6GBW4G3VE7QGQBNWDWY3A4YGDQ, seg_pos={0 0}, upload=465855]: Uploading piece... Dec 27 18:40:29.814259458 (531h11m29.685342503s) [noise=true , address=77.90.31.21:4878 , piece_id=RD3NULS7IWTONXH53YLOUCEZAUX6V42WNJRQ345WFJE5N5LYPGRA, seg_pos={0 0}, upload=465855]: Uploading piece... Dec 27 18:40:29.814829623 (531h11m29.685912668s) [noise=true , address=145.79.227.16:4910 , piece_id=KX6EFMCV7LGXAS4YXVSMYECHJD7SN2SNTBNYGSOKZN4SSG5DQACA, seg_pos={0 0}, upload=465855]: Uploading piece... Dec 27 18:40:29.815877168 (531h11m29.686960203s) [noise=true , address=145.79.227.50:4375 , piece_id=Z5I2IFVSWCFIZEZ7BEJTPORMQG3RVLDBFW4ZUWR6E3M4TI7SQHLA, seg_pos={0 0}, upload=465855]: Uploading piece... Dec 27 18:40:29.817065270 (531h11m29.688148305s) [noise=true , address=145.79.227.24:4581 , piece_id=S3KMQT24OT2MLK77K5WNXGKU4YB3O5FJIRXONN7FHINM6GDWT4CA, seg_pos={0 0}, upload=465855]: Uploading piece... Dec 27 18:40:29.819017568 (531h11m29.690100613s) [noise=true , address=45.91.116.12:4228 , piece_id=TNPP2INNIO3IYTAJQK53FFKOFZVDZP5QWPFDZUHZLO3C3DEAG2CA, seg_pos={0 0}, upload=465855]: Uploading piece... Dec 27 18:40:29.820659485 (531h11m29.69174253s) [noise=true , address=45.91.116.8:4046 , piece_id=BEPXLRAMVWMBO443W33Y77MXVIJC4Q4XGCRELBL65QEASW6YOKKQ, seg_pos={0 0}, upload=465855]: Uploading piece... Dec 27 18:40:29.822226338 (531h11m29.693309383s) [noise=true , address=145.79.227.25:4619 , piece_id=52MTBOEJBUBTWD3TPIXNFII65F76HJ3CFRZCB3JAEFCYHQ2BOTMA, seg_pos={0 0}, upload=465855]: Uploading piece... Dec 27 18:40:29.823857914 (531h11m29.694940959s) [noise=true , address=145.79.227.9:4365 , piece_id=2FPIQV4FGURGE4NCE7CQHSR3H3ZT6N2DWV5OED4G2MJRLZ7RGZNA, seg_pos={0 0}, upload=465855]: Uploading piece... Dec 27 18:40:29.823966993 (531h11m29.695050028s) [noise=true , address=45.91.116.4:4939 , piece_id=Z74ZBVFQ52QGK4NQQVDCH5B3VQBXUQR2HE3UH5IR24SSEV5U5JBQ, seg_pos={0 0}, upload=465855]: Uploading piece... Dec 27 18:40:29.825101693 (531h11m29.696184738s) [noise=true , address=45.91.116.6:4974 , piece_id=HNUC4AKALPPNRE2LSDPEG2NZ5CQE36636XQHXDIFRSQR42EVO6KA, seg_pos={0 0}, upload=465855]: Uploading piece... Dec 27 18:40:31.504086507 (531h11m31.375169552s) [upload=465855]: Next returned nil segment Dec 27 18:40:31.504112516 (531h11m31.375195551s) [upload=465855]: Waiting for error group managing segments... Dec 27 18:40:31.802290870 (531h11m31.673373905s) [noise=true , address=145.79.227.15:4873 , piece_id=SBLCW4WOZ6IXK3SNVGLFMSIWHEGKW4EPLW65E52SWEL7NXYQWQQQ, seg_pos={0 0}, upload=465855]: Done uploading piece. err: <nil> Dec 27 18:40:31.808706406 (531h11m31.679789441s) [noise=true , address=145.79.227.16:4910 , piece_id=KX6EFMCV7LGXAS4YXVSMYECHJD7SN2SNTBNYGSOKZN4SSG5DQACA, seg_pos={0 0}, upload=465855]: Done uploading piece. err: <nil> Dec 27 18:40:31.833367526 (531h11m31.704450571s) [noise=true , address=145.79.227.42:4122 , piece_id=CNI3RHH7IPV5FITXI366RRMQJNWVQGYKN7A6QXRNTRURB73BK3GA, seg_pos={0 0}, upload=465855]: Done uploading piece. err: <nil> Dec 27 18:40:31.834781809 (531h11m31.705864844s) [noise=true , address=145.79.227.9:4365 , piece_id=2FPIQV4FGURGE4NCE7CQHSR3H3ZT6N2DWV5OED4G2MJRLZ7RGZNA, seg_pos={0 0}, upload=465855]: Done uploading piece. err: <nil> Dec 27 18:40:31.839940151 (531h11m31.711023197s) [noise=true , address=145.79.227.25:4619 , piece_id=52MTBOEJBUBTWD3TPIXNFII65F76HJ3CFRZCB3JAEFCYHQ2BOTMA, seg_pos={0 0}, upload=465855]: Done uploading piece. err: <nil> Dec 27 18:40:31.841439076 (531h11m31.712522111s) [noise=true , address=145.79.227.41:4086 , piece_id=XPRMHYYSCSBAZBL55HYZTOJJVTIHD4VPVVJ24CUQ2FRVF5TVBZ4A, seg_pos={0 0}, upload=465855]: Done uploading piece. err: <nil> Dec 27 18:40:31.843142389 (531h11m31.714225424s) [noise=true , address=145.79.227.50:4375 , piece_id=Z5I2IFVSWCFIZEZ7BEJTPORMQG3RVLDBFW4ZUWR6E3M4TI7SQHLA, seg_pos={0 0}, upload=465855]: Done uploading piece. err: <nil> Dec 27 18:40:31.855166656 (531h11m31.726249701s) [noise=true , address=145.79.227.24:4581 , piece_id=S3KMQT24OT2MLK77K5WNXGKU4YB3O5FJIRXONN7FHINM6GDWT4CA, seg_pos={0 0}, upload=465855]: Done uploading piece. err: <nil> Dec 27 18:40:31.858853947 (531h11m31.729936982s) [noise=true , address=145.79.227.46:4602 , piece_id=KCLL26FUGLSHE2LR3ZQRXCZRF2PJL5LZUBNNQW3F7W52TI6JIOKQ, seg_pos={0 0}, upload=465855]: Done uploading piece. err: <nil> Dec 27 18:40:32.040621204 (531h11m31.911704239s) [noise=true , address=77.90.31.21:4878 , piece_id=RD3NULS7IWTONXH53YLOUCEZAUX6V42WNJRQ345WFJE5N5LYPGRA, seg_pos={0 0}, upload=465855]: Done uploading piece. err: <nil> Dec 27 18:40:32.075657128 (531h11m31.946740163s) [noise=true , address=45.91.116.10:4084 , piece_id=BJYRL52FMICD2B2YCG2GP3HUKO6GBW4G3VE7QGQBNWDWY3A4YGDQ, seg_pos={0 0}, upload=465855]: Done uploading piece. err: <nil> Dec 27 18:40:32.086606037 (531h11m31.957689072s) [noise=true , address=45.91.116.6:4974 , piece_id=HNUC4AKALPPNRE2LSDPEG2NZ5CQE36636XQHXDIFRSQR42EVO6KA, seg_pos={0 0}, upload=465855]: Done uploading piece. err: <nil> Dec 27 18:40:32.100544979 (531h11m31.971628014s) [noise=true , address=45.91.116.12:4228 , piece_id=TNPP2INNIO3IYTAJQK53FFKOFZVDZP5QWPFDZUHZLO3C3DEAG2CA, seg_pos={0 0}, upload=465855]: Done uploading piece. err: <nil> Dec 27 18:40:32.100567292 (531h11m31.971650327s) [seg_pos={0 0}, upload=465855]: Segment reached optimal threshold of 13 pieces. Dec 27 18:40:32.100854248 (531h11m31.971937293s) [noise=true , address=45.91.116.8:4046 , piece_id=BEPXLRAMVWMBO443W33Y77MXVIJC4Q4XGCRELBL65QEASW6YOKKQ, seg_pos={0 0}, upload=465855]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 27 18:40:32.100864909 (531h11m31.971947954s) [noise=true , address=45.91.116.8:4046 , piece_id=BEPXLRAMVWMBO443W33Y77MXVIJC4Q4XGCRELBL65QEASW6YOKKQ, seg_pos={0 0}, upload=465855]: Long tail cancellation due to optimal threshold Dec 27 18:40:32.101085609 (531h11m31.972168674s) [noise=true , address=45.91.116.4:4939 , piece_id=Z74ZBVFQ52QGK4NQQVDCH5B3VQBXUQR2HE3UH5IR24SSEV5U5JBQ, seg_pos={0 0}, upload=465855]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 27 18:40:32.101095637 (531h11m31.972178672s) [noise=true , address=45.91.116.4:4939 , piece_id=Z74ZBVFQ52QGK4NQQVDCH5B3VQBXUQR2HE3UH5IR24SSEV5U5JBQ, seg_pos={0 0}, upload=465855]: Long tail cancellation due to optimal threshold Dec 27 18:40:32.101207340 (531h11m31.972290385s) [noise=true , address=45.91.116.14:4189 , piece_id=TOXWDN4UODON25EMZURZXGQAAGCK32TEANRN6GVIWBXTTUVOMX6A, seg_pos={0 0}, upload=465855]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 27 18:40:32.101229643 (531h11m31.972312688s) [noise=true , address=45.91.116.14:4189 , piece_id=TOXWDN4UODON25EMZURZXGQAAGCK32TEANRN6GVIWBXTTUVOMX6A, seg_pos={0 0}, upload=465855]: Long tail cancellation due to optimal threshold Dec 27 18:40:32.101279719 (531h11m31.972362754s) [seg_pos={0 0}, upload=465855]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 18:40:32.101457738 (531h11m31.972540773s) [upload=465855]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 18:40:32.101466835 (531h11m31.97254987s) [upload=465855]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 18:40:32.401875876 (531h11m32.272958972s) [upload=465855]: Done uploading segments... Dec 27 18:40:32.401904732 (531h11m32.272987817s) [upload=465855]: Upload finished. err: <nil> Duration: 30.002957063s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [...X.X........X.......X..X.XXXXXXX...X.........XXXX..XXX.XXX] duration: 30.003383387s trace: https://jaeger.dev.storj.io/trace/13025240f4300ce5 FAIL (2025-12-27 18:40:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: 6SZH4UIPTEXSAHUBIPSXSK7DAIPHJCAWHTR5UMQMG2LN2RUXPBVA) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: YZ22UVAE746UDFHHJCDYFDJLZ7H2OI7YUOVUAL6G4X2OSODTO5XQ) context deadline exceeded ======LOGS====== Dec 27 18:39:39.228605071 (531h10m39.099688106s) [upload=465846]: Starting upload Dec 27 18:39:39.228656919 (531h10m39.099739954s) [upload=465846]: Done starting upload Dec 27 18:39:39.228718707 (531h10m39.099801742s) [upload=465846]: Uploading segments... Dec 27 18:39:39.228973622 (531h10m39.100056667s) [upload=465846]: Got next segment. Inline: false Dec 27 18:39:39.229009771 (531h10m39.100092806s) [upload=465846]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 18:39:39.229029258 (531h10m39.100112293s) [upload=465846]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 18:39:39.474814376 (531h10m39.345897421s) [seg_pos={0 0}, upload=465846]: Begin upload segment... Dec 27 18:39:39.474886804 (531h10m39.345969839s) [seg_pos={0 0}, upload=465846]: Done begin upload segment. Dec 27 18:39:39.474922913 (531h10m39.346005948s) [noise=true , address=145.79.227.11:4730 , piece_id=FW4HIVNUUM4PEEOOSBDBUPXKPGGJP4L3TH3YJ4TJCRU6RMQD5E6Q, seg_pos={0 0}, upload=465846]: Uploading piece... Dec 27 18:39:39.474995541 (531h10m39.346078586s) [noise=true , address=45.91.116.5:4300 , piece_id=YZ22UVAE746UDFHHJCDYFDJLZ7H2OI7YUOVUAL6G4X2OSODTO5XQ, seg_pos={0 0}, upload=465846]: Uploading piece... Dec 27 18:39:39.475298508 (531h10m39.346381553s) [noise=true , address=145.79.227.46:4603 , piece_id=X2QXUF2UUYEE2C6XBROTMZZMXTXTYALQWLBOM6NCCEHGLDLFN44Q, seg_pos={0 0}, upload=465846]: Uploading piece... Dec 27 18:39:39.475733347 (531h10m39.346816392s) [noise=true , address=45.91.116.6:4974 , piece_id=UHEGSZAKEJ4T6MNXILD2RPM2A5NK4LZZVTAC5DQF4GPYTIXR6ZNQ, seg_pos={0 0}, upload=465846]: Uploading piece... Dec 27 18:39:39.476125533 (531h10m39.347208568s) [noise=true , address=145.79.227.7:4293 , piece_id=A24NJD6NV32UVYUUCKOWUXMO3LL2GSQU7SFJ2UUTQHVNARTPKAGQ, seg_pos={0 0}, upload=465846]: Uploading piece... Dec 27 18:39:39.476533771 (531h10m39.347616816s) [noise=true , address=145.79.227.16:4912 , piece_id=5QIUUADTIGST5WZDMZPHTBMFSD7ZQ6AUNSBJ2HY2ZCEIBXOXJUOA, seg_pos={0 0}, upload=465846]: Uploading piece... Dec 27 18:39:39.476928793 (531h10m39.348011828s) [noise=true , address=45.91.116.12:4227 , piece_id=YXT4IND224A7Q52QDPVGSPHGCFOEA7TSZYHOKWI3B3YKW7PDE5GA, seg_pos={0 0}, upload=465846]: Uploading piece... Dec 27 18:39:39.477340246 (531h10m39.348423281s) [noise=true , address=145.79.227.3:4149 , piece_id=ES6CQVTOCR7V4PONFJ3J5ICN34H3NGQF3HTWTROBU3XHLG4NNIJQ, seg_pos={0 0}, upload=465846]: Uploading piece... Dec 27 18:39:39.477890334 (531h10m39.348973379s) [noise=true , address=145.79.227.14:4838 , piece_id=KYJHBB655ASCOIJM4DV55VGV77PMPXJEKAH4ZH2CW6FMPMG4PRWA, seg_pos={0 0}, upload=465846]: Uploading piece... Dec 27 18:39:39.478406075 (531h10m39.34948912s) [noise=true , address=145.79.227.41:4085 , piece_id=V5VV4UXIHQU6AGSQSWUK7NIYUM2VJV5ASHRLVY6L2O2FE5HNV2DA, seg_pos={0 0}, upload=465846]: Uploading piece... Dec 27 18:39:39.478919523 (531h10m39.350002558s) [noise=true , address=45.91.116.13:4155 , piece_id=6SZH4UIPTEXSAHUBIPSXSK7DAIPHJCAWHTR5UMQMG2LN2RUXPBVA, seg_pos={0 0}, upload=465846]: Uploading piece... Dec 27 18:39:39.480626934 (531h10m39.351709979s) [noise=true , address=145.79.227.38:4979 , piece_id=7JR5NABDEBHHZSX5UGJL4WHI4YS5PRY4XNAFZ3DGSICZOFNLW5ZQ, seg_pos={0 0}, upload=465846]: Uploading piece... Dec 27 18:39:39.483131203 (531h10m39.354214238s) [noise=true , address=45.91.116.14:4191 , piece_id=IAIWMZZUIUTG42MQ333BF2NEH2GZ2P7F4BD472QV6OK4JPRRTPOA, seg_pos={0 0}, upload=465846]: Uploading piece... Dec 27 18:39:39.484691735 (531h10m39.35577477s) [noise=true , address=145.79.227.22:4510 , piece_id=YUVLNWLWD3VFT2AZGNUUQJNXH5PJW2RF5BBITN5I2HRQCVGMGBIA, seg_pos={0 0}, upload=465846]: Uploading piece... Dec 27 18:39:39.486251124 (531h10m39.357334169s) [noise=true , address=145.79.227.24:4581 , piece_id=2XJ2HQJRBOPOUCPWY5A4DP4QUCXKT5B76PLNT67DG3KHQEU3PMMA, seg_pos={0 0}, upload=465846]: Uploading piece... Dec 27 18:39:39.487844258 (531h10m39.358927303s) [noise=true , address=45.91.116.11:4120 , piece_id=CKBFPMCNYOUG4DZ27WIRTOI25G64R2SBR7UIDECNGQS2OHH6YFWQ, seg_pos={0 0}, upload=465846]: Uploading piece... Dec 27 18:39:41.183220042 (531h10m41.054303087s) [upload=465846]: Next returned nil segment Dec 27 18:39:41.183231283 (531h10m41.054314319s) [upload=465846]: Waiting for error group managing segments... Dec 27 18:39:41.472366428 (531h10m41.343449463s) [noise=true , address=145.79.227.16:4912 , piece_id=5QIUUADTIGST5WZDMZPHTBMFSD7ZQ6AUNSBJ2HY2ZCEIBXOXJUOA, seg_pos={0 0}, upload=465846]: Done uploading piece. err: <nil> Dec 27 18:39:41.476272475 (531h10m41.34735552s) [noise=true , address=145.79.227.46:4603 , piece_id=X2QXUF2UUYEE2C6XBROTMZZMXTXTYALQWLBOM6NCCEHGLDLFN44Q, seg_pos={0 0}, upload=465846]: Done uploading piece. err: <nil> Dec 27 18:39:41.477085775 (531h10m41.34816881s) [noise=true , address=145.79.227.7:4293 , piece_id=A24NJD6NV32UVYUUCKOWUXMO3LL2GSQU7SFJ2UUTQHVNARTPKAGQ, seg_pos={0 0}, upload=465846]: Done uploading piece. err: <nil> Dec 27 18:39:41.479785284 (531h10m41.35086832s) [noise=true , address=145.79.227.3:4149 , piece_id=ES6CQVTOCR7V4PONFJ3J5ICN34H3NGQF3HTWTROBU3XHLG4NNIJQ, seg_pos={0 0}, upload=465846]: Done uploading piece. err: <nil> Dec 27 18:39:41.482982883 (531h10m41.354065918s) [noise=true , address=145.79.227.24:4581 , piece_id=2XJ2HQJRBOPOUCPWY5A4DP4QUCXKT5B76PLNT67DG3KHQEU3PMMA, seg_pos={0 0}, upload=465846]: Done uploading piece. err: <nil> Dec 27 18:39:41.487118999 (531h10m41.358202035s) [noise=true , address=145.79.227.11:4730 , piece_id=FW4HIVNUUM4PEEOOSBDBUPXKPGGJP4L3TH3YJ4TJCRU6RMQD5E6Q, seg_pos={0 0}, upload=465846]: Done uploading piece. err: <nil> Dec 27 18:39:41.489890177 (531h10m41.360973222s) [noise=true , address=145.79.227.22:4510 , piece_id=YUVLNWLWD3VFT2AZGNUUQJNXH5PJW2RF5BBITN5I2HRQCVGMGBIA, seg_pos={0 0}, upload=465846]: Done uploading piece. err: <nil> Dec 27 18:39:41.497956546 (531h10m41.369039591s) [noise=true , address=145.79.227.38:4979 , piece_id=7JR5NABDEBHHZSX5UGJL4WHI4YS5PRY4XNAFZ3DGSICZOFNLW5ZQ, seg_pos={0 0}, upload=465846]: Done uploading piece. err: <nil> Dec 27 18:39:41.514344722 (531h10m41.385427767s) [noise=true , address=145.79.227.14:4838 , piece_id=KYJHBB655ASCOIJM4DV55VGV77PMPXJEKAH4ZH2CW6FMPMG4PRWA, seg_pos={0 0}, upload=465846]: Done uploading piece. err: <nil> Dec 27 18:39:41.515387117 (531h10m41.386470152s) [noise=true , address=145.79.227.41:4085 , piece_id=V5VV4UXIHQU6AGSQSWUK7NIYUM2VJV5ASHRLVY6L2O2FE5HNV2DA, seg_pos={0 0}, upload=465846]: Done uploading piece. err: <nil> Dec 27 18:39:41.732857799 (531h10m41.603940834s) [noise=true , address=45.91.116.13:4155 , piece_id=6SZH4UIPTEXSAHUBIPSXSK7DAIPHJCAWHTR5UMQMG2LN2RUXPBVA, seg_pos={0 0}, upload=465846]: Done uploading piece. err: <nil> Dec 27 18:39:41.742788189 (531h10m41.613871224s) [noise=true , address=45.91.116.5:4300 , piece_id=YZ22UVAE746UDFHHJCDYFDJLZ7H2OI7YUOVUAL6G4X2OSODTO5XQ, seg_pos={0 0}, upload=465846]: Done uploading piece. err: <nil> Dec 27 18:39:41.943119154 (531h10m41.814202199s) [noise=true , address=45.91.116.6:4974 , piece_id=UHEGSZAKEJ4T6MNXILD2RPM2A5NK4LZZVTAC5DQF4GPYTIXR6ZNQ, seg_pos={0 0}, upload=465846]: Done uploading piece. err: <nil> Dec 27 18:39:41.943142729 (531h10m41.814225774s) [seg_pos={0 0}, upload=465846]: Segment reached optimal threshold of 13 pieces. Dec 27 18:39:41.943350585 (531h10m41.81443362s) [noise=true , address=45.91.116.14:4191 , piece_id=IAIWMZZUIUTG42MQ333BF2NEH2GZ2P7F4BD472QV6OK4JPRRTPOA, seg_pos={0 0}, upload=465846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 27 18:39:41.943360073 (531h10m41.814443118s) [noise=true , address=45.91.116.14:4191 , piece_id=IAIWMZZUIUTG42MQ333BF2NEH2GZ2P7F4BD472QV6OK4JPRRTPOA, seg_pos={0 0}, upload=465846]: Long tail cancellation due to optimal threshold Dec 27 18:39:41.943407414 (531h10m41.814490449s) [noise=true , address=45.91.116.11:4120 , piece_id=CKBFPMCNYOUG4DZ27WIRTOI25G64R2SBR7UIDECNGQS2OHH6YFWQ, seg_pos={0 0}, upload=465846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 27 18:39:41.943420438 (531h10m41.814503473s) [noise=true , address=45.91.116.11:4120 , piece_id=CKBFPMCNYOUG4DZ27WIRTOI25G64R2SBR7UIDECNGQS2OHH6YFWQ, seg_pos={0 0}, upload=465846]: Long tail cancellation due to optimal threshold Dec 27 18:39:41.943555976 (531h10m41.814639011s) [noise=true , address=45.91.116.12:4227 , piece_id=YXT4IND224A7Q52QDPVGSPHGCFOEA7TSZYHOKWI3B3YKW7PDE5GA, seg_pos={0 0}, upload=465846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Dec 27 18:39:41.943561887 (531h10m41.814644922s) [noise=true , address=45.91.116.12:4227 , piece_id=YXT4IND224A7Q52QDPVGSPHGCFOEA7TSZYHOKWI3B3YKW7PDE5GA, seg_pos={0 0}, upload=465846]: Long tail cancellation due to optimal threshold Dec 27 18:39:41.943576865 (531h10m41.814659901s) [seg_pos={0 0}, upload=465846]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 18:39:41.943642040 (531h10m41.814725075s) [upload=465846]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 18:39:41.943647350 (531h10m41.814730385s) [upload=465846]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 18:39:42.229875079 (531h10m42.100958124s) [upload=465846]: Done uploading segments... Dec 27 18:39:42.229900417 (531h10m42.100983462s) [upload=465846]: Upload finished. err: <nil> Duration: 30.003397805s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [X...X.X........X.......X..X.XXXXXXX...X.........XXXX..XXX.XX] duration: 30.002294304s trace: https://jaeger.dev.storj.io/trace/16a5b7390b94e96e FAIL (2025-12-27 18:39:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: LFU26RFFDWIJEG5GSEXP37WUV54K53KOWL6G7LU4GTF7WLRZEYSQ) context deadline exceeded; piecestore: (Node ID: 1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, Piece ID: 6F5SEPC4QOZCZG6KSKTIKTB4D4ONBHCR5HRZYTCHXVKBGKYKAYGA) context deadline exceeded ======LOGS====== Dec 27 18:39:09.012974507 (531h10m8.884057552s) [upload=465837]: Starting upload Dec 27 18:39:09.013073986 (531h10m8.884157041s) [upload=465837]: Done starting upload Dec 27 18:39:09.013145422 (531h10m8.884228457s) [upload=465837]: Uploading segments... Dec 27 18:39:09.013380220 (531h10m8.884463265s) [upload=465837]: Got next segment. Inline: false Dec 27 18:39:09.013430024 (531h10m8.884513069s) [upload=465837]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 18:39:09.013436437 (531h10m8.884519482s) [upload=465837]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 18:39:09.264540914 (531h10m9.135623979s) [seg_pos={0 0}, upload=465837]: Begin upload segment... Dec 27 18:39:09.264596650 (531h10m9.135679695s) [seg_pos={0 0}, upload=465837]: Done begin upload segment. Dec 27 18:39:09.264643860 (531h10m9.135726905s) [noise=true , address=45.91.116.6:4973 , piece_id=LFU26RFFDWIJEG5GSEXP37WUV54K53KOWL6G7LU4GTF7WLRZEYSQ, seg_pos={0 0}, upload=465837]: Uploading piece... Dec 27 18:39:09.264688394 (531h10m9.135771429s) [noise=true , address=145.79.227.3:4149 , piece_id=XVVH6AQ5OMU7UN6FJ4L4N5DLBPAJFZP5GEW5RVB4N5BRABK3CEEQ, seg_pos={0 0}, upload=465837]: Uploading piece... Dec 27 18:39:09.265308125 (531h10m9.13639116s) [noise=true , address=145.79.227.8:4329 , piece_id=FLVQ4UEVPHJPGJHT2T43U4J7F3UESICGG56E5AUH6QE2G3CNDEQA, seg_pos={0 0}, upload=465837]: Uploading piece... Dec 27 18:39:09.265394570 (531h10m9.136477615s) [noise=true , address=45.91.116.9:4262 , piece_id=LLZ6TRD3GOG77NC2FPXHL7V3PYPNC2WVCF5LZYO3K5R33IIIWAYQ, seg_pos={0 0}, upload=465837]: Uploading piece... Dec 27 18:39:09.265889302 (531h10m9.136972347s) [noise=true , address=145.79.227.26:4653 , piece_id=VMH46B734IUYDBTU6DBKZ7ANPQ4W23NPXR6FLLAIRMWAIZLY5QRQ, seg_pos={0 0}, upload=465837]: Uploading piece... Dec 27 18:39:09.266270958 (531h10m9.137354003s) [noise=true , address=145.79.227.14:4838 , piece_id=E7CBB7CYW7M5OB25K56FTH6I44RBI7PWHFM5V6HIKUN4LUAONHRQ, seg_pos={0 0}, upload=465837]: Uploading piece... Dec 27 18:39:09.267643852 (531h10m9.138726897s) [noise=true , address=45.91.116.10:4084 , piece_id=YGIR4NNLG3573KVQ3K5T36TPJ6PDJOGVW6ASQVFDDGERI4P4JIDQ, seg_pos={0 0}, upload=465837]: Uploading piece... Dec 27 18:39:09.268278611 (531h10m9.139361656s) [noise=true , address=45.91.116.4:4937 , piece_id=7ANDVD5CGNFDKAKHEMLFPTJVHQKNLXT4SYUEJXAWFWTG3SJBO64A, seg_pos={0 0}, upload=465837]: Uploading piece... Dec 27 18:39:09.271710496 (531h10m9.142793541s) [noise=true , address=145.79.227.22:4510 , piece_id=7EGXNZ6KG3G7QCUQAEM35ZCXWHOT2437Z3BM6EAE7KHLNDDN5BGQ, seg_pos={0 0}, upload=465837]: Uploading piece... Dec 27 18:39:09.272205198 (531h10m9.143288243s) [noise=true , address=145.79.227.50:4375 , piece_id=5YTML2EXERUYBS6NVEQ3PO2RTARXNWFWCPMYNXN5EWTNTAXV24BA, seg_pos={0 0}, upload=465837]: Uploading piece... Dec 27 18:39:09.274296810 (531h10m9.145379845s) [noise=true , address=145.79.227.4:4185 , piece_id=BOZPWSUCU3QGBO3KX4UNSW7FMJJPTRRQKMQH775OESDS375MRC4Q, seg_pos={0 0}, upload=465837]: Uploading piece... Dec 27 18:39:09.276318119 (531h10m9.147401164s) [noise=true , address=45.91.116.14:4189 , piece_id=5VVBRTW4JOXH3HKFDIJHP2UNJ34RG6ANX7IKFJQYLTRRZBYS3IEQ, seg_pos={0 0}, upload=465837]: Uploading piece... Dec 27 18:39:09.278048685 (531h10m9.14913173s) [noise=true , address=145.79.227.40:4051 , piece_id=WZCL44B2JI5TKO7RE2F6Q7S4PSGBNHTJWEE5BSNZOSCXPJG46GPA, seg_pos={0 0}, upload=465837]: Uploading piece... Dec 27 18:39:09.279605119 (531h10m9.150688154s) [noise=true , address=145.79.227.28:4725 , piece_id=OHCNMZUWMPUQSDAFFD5UOCIHJV6ZXYKR4PFWJ6Y2H6PRDPVALFEQ, seg_pos={0 0}, upload=465837]: Uploading piece... Dec 27 18:39:09.281209153 (531h10m9.152292198s) [noise=true , address=77.90.31.11:4555 , piece_id=6F5SEPC4QOZCZG6KSKTIKTB4D4ONBHCR5HRZYTCHXVKBGKYKAYGA, seg_pos={0 0}, upload=465837]: Uploading piece... Dec 27 18:39:09.282369653 (531h10m9.153452688s) [noise=true , address=145.79.227.51:4409 , piece_id=XE7LQ2YXRDSEI7PC3DBJGHQUCMEYB3ABTIGRQ7I4VJMKGEH3LWDQ, seg_pos={0 0}, upload=465837]: Uploading piece... Dec 27 18:39:10.981269338 (531h10m10.852352373s) [upload=465837]: Next returned nil segment Dec 27 18:39:10.981287893 (531h10m10.852370928s) [upload=465837]: Waiting for error group managing segments... Dec 27 18:39:11.265554247 (531h10m11.136637312s) [noise=true , address=145.79.227.50:4375 , piece_id=5YTML2EXERUYBS6NVEQ3PO2RTARXNWFWCPMYNXN5EWTNTAXV24BA, seg_pos={0 0}, upload=465837]: Done uploading piece. err: <nil> Dec 27 18:39:11.265631003 (531h10m11.136714048s) [noise=true , address=145.79.227.14:4838 , piece_id=E7CBB7CYW7M5OB25K56FTH6I44RBI7PWHFM5V6HIKUN4LUAONHRQ, seg_pos={0 0}, upload=465837]: Done uploading piece. err: <nil> Dec 27 18:39:11.267978643 (531h10m11.139061688s) [noise=true , address=145.79.227.3:4149 , piece_id=XVVH6AQ5OMU7UN6FJ4L4N5DLBPAJFZP5GEW5RVB4N5BRABK3CEEQ, seg_pos={0 0}, upload=465837]: Done uploading piece. err: <nil> Dec 27 18:39:11.279985696 (531h10m11.151068731s) [noise=true , address=145.79.227.4:4185 , piece_id=BOZPWSUCU3QGBO3KX4UNSW7FMJJPTRRQKMQH775OESDS375MRC4Q, seg_pos={0 0}, upload=465837]: Done uploading piece. err: <nil> Dec 27 18:39:11.282614092 (531h10m11.153697137s) [noise=true , address=145.79.227.22:4510 , piece_id=7EGXNZ6KG3G7QCUQAEM35ZCXWHOT2437Z3BM6EAE7KHLNDDN5BGQ, seg_pos={0 0}, upload=465837]: Done uploading piece. err: <nil> Dec 27 18:39:11.284415322 (531h10m11.155498367s) [noise=true , address=145.79.227.28:4725 , piece_id=OHCNMZUWMPUQSDAFFD5UOCIHJV6ZXYKR4PFWJ6Y2H6PRDPVALFEQ, seg_pos={0 0}, upload=465837]: Done uploading piece. err: <nil> Dec 27 18:39:11.294736545 (531h10m11.16581959s) [noise=true , address=145.79.227.26:4653 , piece_id=VMH46B734IUYDBTU6DBKZ7ANPQ4W23NPXR6FLLAIRMWAIZLY5QRQ, seg_pos={0 0}, upload=465837]: Done uploading piece. err: <nil> Dec 27 18:39:11.298947393 (531h10m11.170030438s) [noise=true , address=145.79.227.51:4409 , piece_id=XE7LQ2YXRDSEI7PC3DBJGHQUCMEYB3ABTIGRQ7I4VJMKGEH3LWDQ, seg_pos={0 0}, upload=465837]: Done uploading piece. err: <nil> Dec 27 18:39:11.304900930 (531h10m11.175983965s) [noise=true , address=145.79.227.8:4329 , piece_id=FLVQ4UEVPHJPGJHT2T43U4J7F3UESICGG56E5AUH6QE2G3CNDEQA, seg_pos={0 0}, upload=465837]: Done uploading piece. err: <nil> Dec 27 18:39:11.500123598 (531h10m11.371206643s) [noise=true , address=45.91.116.4:4937 , piece_id=7ANDVD5CGNFDKAKHEMLFPTJVHQKNLXT4SYUEJXAWFWTG3SJBO64A, seg_pos={0 0}, upload=465837]: Done uploading piece. err: <nil> Dec 27 18:39:11.540686076 (531h10m11.411769111s) [noise=true , address=77.90.31.11:4555 , piece_id=6F5SEPC4QOZCZG6KSKTIKTB4D4ONBHCR5HRZYTCHXVKBGKYKAYGA, seg_pos={0 0}, upload=465837]: Done uploading piece. err: <nil> Dec 27 18:39:11.553341444 (531h10m11.424424489s) [noise=true , address=45.91.116.6:4973 , piece_id=LFU26RFFDWIJEG5GSEXP37WUV54K53KOWL6G7LU4GTF7WLRZEYSQ, seg_pos={0 0}, upload=465837]: Done uploading piece. err: <nil> Dec 27 18:39:11.591232646 (531h10m11.462315691s) [noise=true , address=45.91.116.14:4189 , piece_id=5VVBRTW4JOXH3HKFDIJHP2UNJ34RG6ANX7IKFJQYLTRRZBYS3IEQ, seg_pos={0 0}, upload=465837]: Done uploading piece. err: <nil> Dec 27 18:39:11.591256471 (531h10m11.462339516s) [seg_pos={0 0}, upload=465837]: Segment reached optimal threshold of 13 pieces. Dec 27 18:39:11.591451443 (531h10m11.462534478s) [noise=true , address=145.79.227.40:4051 , piece_id=WZCL44B2JI5TKO7RE2F6Q7S4PSGBNHTJWEE5BSNZOSCXPJG46GPA, seg_pos={0 0}, upload=465837]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 27 18:39:11.591457764 (531h10m11.462540809s) [noise=true , address=145.79.227.40:4051 , piece_id=WZCL44B2JI5TKO7RE2F6Q7S4PSGBNHTJWEE5BSNZOSCXPJG46GPA, seg_pos={0 0}, upload=465837]: Long tail cancellation due to optimal threshold Dec 27 18:39:11.591660340 (531h10m11.462743375s) [noise=true , address=45.91.116.10:4084 , piece_id=YGIR4NNLG3573KVQ3K5T36TPJ6PDJOGVW6ASQVFDDGERI4P4JIDQ, seg_pos={0 0}, upload=465837]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 27 18:39:11.591678605 (531h10m11.46276164s) [noise=true , address=45.91.116.10:4084 , piece_id=YGIR4NNLG3573KVQ3K5T36TPJ6PDJOGVW6ASQVFDDGERI4P4JIDQ, seg_pos={0 0}, upload=465837]: Long tail cancellation due to optimal threshold Dec 27 18:39:11.591736706 (531h10m11.462819751s) [noise=true , address=45.91.116.9:4262 , piece_id=LLZ6TRD3GOG77NC2FPXHL7V3PYPNC2WVCF5LZYO3K5R33IIIWAYQ, seg_pos={0 0}, upload=465837]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: write tcp 188.166.225.170:34526->45.91.116.9:4262: use of closed network connection; write tcp 188.166.225.170:34526->45.91.116.9:4262: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34526->45.91.116.9:4262: use of closed network connection Dec 27 18:39:11.591753829 (531h10m11.462836874s) [noise=true , address=45.91.116.9:4262 , piece_id=LLZ6TRD3GOG77NC2FPXHL7V3PYPNC2WVCF5LZYO3K5R33IIIWAYQ, seg_pos={0 0}, upload=465837]: Long tail cancellation due to optimal threshold Dec 27 18:39:11.591780579 (531h10m11.462863624s) [seg_pos={0 0}, upload=465837]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 18:39:11.591894987 (531h10m11.462978022s) [upload=465837]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 18:39:11.591901279 (531h10m11.462984324s) [upload=465837]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 18:39:11.887309448 (531h10m11.758392493s) [upload=465837]: Done uploading segments... Dec 27 18:39:11.887341219 (531h10m11.758424264s) [upload=465837]: Upload finished. err: <nil> Duration: 30.002325312s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.X...X.X........X.......X..X.XXXXXXX...X.........XXXX..XXX.X] duration: 30.002530067s trace: https://jaeger.dev.storj.io/trace/e572081f0807bd2 FAIL (2025-12-27 18:39:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 18:38:38.399267990 (531h9m38.270351036s) [upload=465830]: Starting upload Dec 27 18:38:38.399321572 (531h9m38.270404607s) [upload=465830]: Done starting upload Dec 27 18:38:38.399471447 (531h9m38.270554492s) [upload=465830]: Uploading segments... Dec 27 18:38:38.399570856 (531h9m38.270653902s) [upload=465830]: Got next segment. Inline: false Dec 27 18:38:38.399612326 (531h9m38.270695361s) [upload=465830]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 18:38:38.399619420 (531h9m38.270702465s) [upload=465830]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 18:38:38.648164093 (531h9m38.519247128s) [seg_pos={0 0}, upload=465830]: Begin upload segment... Dec 27 18:38:38.648212514 (531h9m38.519295549s) [seg_pos={0 0}, upload=465830]: Done begin upload segment. Dec 27 18:38:38.648248413 (531h9m38.519331448s) [noise=true , address=77.90.31.21:4878 , piece_id=PSHO3SGQBPW7N5PEWYXCIQAIU5NUN3ML6777L2VAOKLXCTEE6K7Q, seg_pos={0 0}, upload=465830]: Uploading piece... Dec 27 18:38:38.648362270 (531h9m38.519445315s) [noise=true , address=145.79.227.18:4007 , piece_id=YLVJLBNA3CEHM5JJ3Z2MJNX5JKWODYODI4AGPHHILGVCKC4PHSMA, seg_pos={0 0}, upload=465830]: Uploading piece... Dec 27 18:38:38.648819170 (531h9m38.519902215s) [noise=true , address=145.79.227.11:4732 , piece_id=6P2IUVFGUGSTUWQM5ZW2GWDU53Q4QE22V3CPVGG7ICYX4ENIFCDA, seg_pos={0 0}, upload=465830]: Uploading piece... Dec 27 18:38:38.649100446 (531h9m38.520183491s) [noise=true , address=77.90.31.11:4553 , piece_id=CPPRGJXGVMZBYIH6QLAJJYM5ACOWAIMOXE7HZVYY7OSSZGIFRN4Q, seg_pos={0 0}, upload=465830]: Uploading piece... Dec 27 18:38:38.649381942 (531h9m38.520464977s) [noise=true , address=145.79.227.39:4015 , piece_id=IVEC6S32NVEOK75Q2PWRZGSQJRH4ACFMWR67UXPI6UWMRD3HNSCA, seg_pos={0 0}, upload=465830]: Uploading piece... Dec 27 18:38:38.649652146 (531h9m38.520735181s) [noise=true , address=145.79.227.12:4767 , piece_id=4UDOHYKLMLKIS4YX4OO77AEXOKNHDK7W5A62XGOYG4QVZMSY7HRQ, seg_pos={0 0}, upload=465830]: Uploading piece... Dec 27 18:38:38.649964591 (531h9m38.521047636s) [noise=true , address=145.79.227.4:4185 , piece_id=7Q3J4MELEL6KVBQWJREASRINMHYMK7SSCJWDIJZH3TZ3OPQFQZOA, seg_pos={0 0}, upload=465830]: Uploading piece... Dec 27 18:38:38.650267107 (531h9m38.521350142s) [noise=true , address=145.79.227.5:4223 , piece_id=OXFL3CNV2JFOKJKPFAMNOJCN7B3UFNSWE5EHAILCJVPOUGGQDQCA, seg_pos={0 0}, upload=465830]: Uploading piece... Dec 27 18:38:38.650692116 (531h9m38.521775151s) [noise=true , address=145.79.227.52:4447 , piece_id=7C6VLCPYFAXE4LIGHRBX52I6M5724GPEJS3BEB72FU6ZSIS7GW4A, seg_pos={0 0}, upload=465830]: Uploading piece... Dec 27 18:38:38.652003964 (531h9m38.523086999s) [noise=true , address=145.79.227.8:4329 , piece_id=VZDDPTGIESVHG3ZI7F3LUOR62N2IP5JRTIZ7MFQEFZECANS2BXJQ, seg_pos={0 0}, upload=465830]: Uploading piece... Dec 27 18:38:38.653206565 (531h9m38.52428961s) [noise=true , address=77.90.31.10:4517 , piece_id=3FU4CC5WSYA34E7LG5JIYNDJLF5LJXPTLTTLXC4UGEXCF6X53COQ, seg_pos={0 0}, upload=465830]: Uploading piece... Dec 27 18:38:38.654404816 (531h9m38.525487851s) [noise=true , address=145.79.227.32:4834 , piece_id=BSCPZ53B32QQJZJ5DT3CVARIEYNOAWFQKFU5KRCRZZHF2G42G43A, seg_pos={0 0}, upload=465830]: Uploading piece... Dec 27 18:38:38.655793560 (531h9m38.526876595s) [noise=true , address=45.91.116.8:4047 , piece_id=MC4CIOW6GABTIPOLG6CMGS4RQQFNGVS7T5AX7VMAXXFJLW6MNPHA, seg_pos={0 0}, upload=465830]: Uploading piece... Dec 27 18:38:38.657164791 (531h9m38.528247826s) [noise=true , address=77.90.31.16:4736 , piece_id=TZ4ZPDBJZ6DWGWAQUQOTW26V7PGAXM55IPML4XEPLONQMJQNSM7Q, seg_pos={0 0}, upload=465830]: Uploading piece... Dec 27 18:38:38.658353616 (531h9m38.529436651s) [noise=true , address=45.91.116.9:4263 , piece_id=MJKCR56NRQOHDOHFNNQ5XOUDWSMZCGIGUW4Z2RIEM44I32CQZO3A, seg_pos={0 0}, upload=465830]: Uploading piece... Dec 27 18:38:38.659281412 (531h9m38.530364447s) [noise=true , address=145.79.227.15:4876 , piece_id=37ZRWNOZ627LLJ6S5I2KKOZ4XGD4A4QZMDFSA55VURD7IQOJZD2A, seg_pos={0 0}, upload=465830]: Uploading piece... Dec 27 18:38:40.355687288 (531h9m40.226770333s) [upload=465830]: Next returned nil segment Dec 27 18:38:40.355704682 (531h9m40.226787717s) [upload=465830]: Waiting for error group managing segments... Dec 27 18:38:40.642881217 (531h9m40.513964262s) [noise=true , address=145.79.227.18:4007 , piece_id=YLVJLBNA3CEHM5JJ3Z2MJNX5JKWODYODI4AGPHHILGVCKC4PHSMA, seg_pos={0 0}, upload=465830]: Done uploading piece. err: <nil> Dec 27 18:38:40.655321775 (531h9m40.52640481s) [noise=true , address=145.79.227.39:4015 , piece_id=IVEC6S32NVEOK75Q2PWRZGSQJRH4ACFMWR67UXPI6UWMRD3HNSCA, seg_pos={0 0}, upload=465830]: Done uploading piece. err: <nil> Dec 27 18:38:40.656316680 (531h9m40.527399725s) [noise=true , address=145.79.227.4:4185 , piece_id=7Q3J4MELEL6KVBQWJREASRINMHYMK7SSCJWDIJZH3TZ3OPQFQZOA, seg_pos={0 0}, upload=465830]: Done uploading piece. err: <nil> Dec 27 18:38:40.662607097 (531h9m40.533690142s) [noise=true , address=145.79.227.32:4834 , piece_id=BSCPZ53B32QQJZJ5DT3CVARIEYNOAWFQKFU5KRCRZZHF2G42G43A, seg_pos={0 0}, upload=465830]: Done uploading piece. err: <nil> Dec 27 18:38:40.664986238 (531h9m40.536069283s) [noise=true , address=145.79.227.11:4732 , piece_id=6P2IUVFGUGSTUWQM5ZW2GWDU53Q4QE22V3CPVGG7ICYX4ENIFCDA, seg_pos={0 0}, upload=465830]: Done uploading piece. err: <nil> Dec 27 18:38:40.667446713 (531h9m40.538529758s) [noise=true , address=145.79.227.5:4223 , piece_id=OXFL3CNV2JFOKJKPFAMNOJCN7B3UFNSWE5EHAILCJVPOUGGQDQCA, seg_pos={0 0}, upload=465830]: Done uploading piece. err: <nil> Dec 27 18:38:40.668033701 (531h9m40.539116746s) [noise=true , address=145.79.227.12:4767 , piece_id=4UDOHYKLMLKIS4YX4OO77AEXOKNHDK7W5A62XGOYG4QVZMSY7HRQ, seg_pos={0 0}, upload=465830]: Done uploading piece. err: <nil> Dec 27 18:38:40.674658235 (531h9m40.54574127s) [noise=true , address=145.79.227.15:4876 , piece_id=37ZRWNOZ627LLJ6S5I2KKOZ4XGD4A4QZMDFSA55VURD7IQOJZD2A, seg_pos={0 0}, upload=465830]: Done uploading piece. err: <nil> Dec 27 18:38:40.675531609 (531h9m40.546614644s) [noise=true , address=145.79.227.8:4329 , piece_id=VZDDPTGIESVHG3ZI7F3LUOR62N2IP5JRTIZ7MFQEFZECANS2BXJQ, seg_pos={0 0}, upload=465830]: Done uploading piece. err: <nil> Dec 27 18:38:40.684556914 (531h9m40.555639959s) [noise=true , address=145.79.227.52:4447 , piece_id=7C6VLCPYFAXE4LIGHRBX52I6M5724GPEJS3BEB72FU6ZSIS7GW4A, seg_pos={0 0}, upload=465830]: Done uploading piece. err: <nil> Dec 27 18:38:40.806889778 (531h9m40.677972823s) [noise=true , address=77.90.31.11:4553 , piece_id=CPPRGJXGVMZBYIH6QLAJJYM5ACOWAIMOXE7HZVYY7OSSZGIFRN4Q, seg_pos={0 0}, upload=465830]: Done uploading piece. err: <nil> Dec 27 18:38:40.828987105 (531h9m40.70007014s) [noise=true , address=77.90.31.16:4736 , piece_id=TZ4ZPDBJZ6DWGWAQUQOTW26V7PGAXM55IPML4XEPLONQMJQNSM7Q, seg_pos={0 0}, upload=465830]: Done uploading piece. err: <nil> Dec 27 18:38:40.891165203 (531h9m40.762248248s) [noise=true , address=77.90.31.21:4878 , piece_id=PSHO3SGQBPW7N5PEWYXCIQAIU5NUN3ML6777L2VAOKLXCTEE6K7Q, seg_pos={0 0}, upload=465830]: Done uploading piece. err: <nil> Dec 27 18:38:40.891188427 (531h9m40.762271462s) [seg_pos={0 0}, upload=465830]: Segment reached optimal threshold of 13 pieces. Dec 27 18:38:40.891471847 (531h9m40.762554882s) [noise=true , address=45.91.116.9:4263 , piece_id=MJKCR56NRQOHDOHFNNQ5XOUDWSMZCGIGUW4Z2RIEM44I32CQZO3A, seg_pos={0 0}, upload=465830]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 27 18:38:40.891483028 (531h9m40.762566063s) [noise=true , address=45.91.116.9:4263 , piece_id=MJKCR56NRQOHDOHFNNQ5XOUDWSMZCGIGUW4Z2RIEM44I32CQZO3A, seg_pos={0 0}, upload=465830]: Long tail cancellation due to optimal threshold Dec 27 18:38:40.891852652 (531h9m40.762935697s) [noise=true , address=45.91.116.8:4047 , piece_id=MC4CIOW6GABTIPOLG6CMGS4RQQFNGVS7T5AX7VMAXXFJLW6MNPHA, seg_pos={0 0}, upload=465830]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 27 18:38:40.891861859 (531h9m40.762944904s) [noise=true , address=45.91.116.8:4047 , piece_id=MC4CIOW6GABTIPOLG6CMGS4RQQFNGVS7T5AX7VMAXXFJLW6MNPHA, seg_pos={0 0}, upload=465830]: Long tail cancellation due to optimal threshold Dec 27 18:38:40.892057232 (531h9m40.763140277s) [noise=true , address=77.90.31.10:4517 , piece_id=3FU4CC5WSYA34E7LG5JIYNDJLF5LJXPTLTTLXC4UGEXCF6X53COQ, seg_pos={0 0}, upload=465830]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 27 18:38:40.892067391 (531h9m40.763150436s) [noise=true , address=77.90.31.10:4517 , piece_id=3FU4CC5WSYA34E7LG5JIYNDJLF5LJXPTLTTLXC4UGEXCF6X53COQ, seg_pos={0 0}, upload=465830]: Long tail cancellation due to optimal threshold Dec 27 18:38:40.892100193 (531h9m40.763183248s) [seg_pos={0 0}, upload=465830]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 18:38:40.892191927 (531h9m40.763274972s) [upload=465830]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 18:38:40.892198451 (531h9m40.763281496s) [upload=465830]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 18:38:41.177641084 (531h9m41.048724129s) [upload=465830]: Done uploading segments... Dec 27 18:38:41.177660221 (531h9m41.048743266s) [upload=465830]: Upload finished. err: <nil> Duration: 30.002557178s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [..X...X.X........X.......X..X.XXXXXXX...X.........XXXX..XXX.] duration: 18.492434632s trace: https://jaeger.dev.storj.io/trace/73a3c8fc140bebab FAIL (2025-12-27 18:37:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 18:37:02.424684117 (531h8m2.295767163s) [upload=465802]: Starting upload Dec 27 18:37:02.424743982 (531h8m2.295827017s) [upload=465802]: Done starting upload Dec 27 18:37:02.424870984 (531h8m2.295954019s) [upload=465802]: Uploading segments... Dec 27 18:37:02.424980021 (531h8m2.296063066s) [upload=465802]: Got next segment. Inline: false Dec 27 18:37:02.425016992 (531h8m2.296100027s) [upload=465802]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 18:37:02.425022592 (531h8m2.296105637s) [upload=465802]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 18:37:02.672179384 (531h8m2.54326245s) [seg_pos={0 0}, upload=465802]: Begin upload segment... Dec 27 18:37:02.672240862 (531h8m2.543323907s) [seg_pos={0 0}, upload=465802]: Done begin upload segment. Dec 27 18:37:02.672277261 (531h8m2.543360306s) [noise=true , address=45.91.116.5:4297 , piece_id=ASCZI5WE3BVWLXVE6EUFWGWLK4JCRMPH2FWHLOXLNBSQ4QLENIEA, seg_pos={0 0}, upload=465802]: Uploading piece... Dec 27 18:37:02.672409191 (531h8m2.543492236s) [noise=true , address=145.79.227.14:4839 , piece_id=W5AZFHG7JBI27CR7JLEFK7XLURGM4F2BVAAU6QE43RRX3E22YF5Q, seg_pos={0 0}, upload=465802]: Uploading piece... Dec 27 18:37:02.673048930 (531h8m2.544131975s) [noise=true , address=145.79.227.35:4483 , piece_id=S3SCCSH2Q6WCIATY26OP5SKAUKU3VK67ZIUQG2E3TF3MNT4BIZOA, seg_pos={0 0}, upload=465802]: Uploading piece... Dec 27 18:37:02.673507654 (531h8m2.544590699s) [noise=true , address=45.91.116.7:4012 , piece_id=XQS343PYEPATYNGGFVI33MDQBVWHDBAM3MYC2Y5ZPZBV6F3ROK4Q, seg_pos={0 0}, upload=465802]: Uploading piece... Dec 27 18:37:02.674012164 (531h8m2.545095209s) [noise=true , address=145.79.227.25:4619 , piece_id=5LL7MPNI5N5NC7CKQ4IRHELA6VZYLCJW277UR3DQCBB2CTGCK2MA, seg_pos={0 0}, upload=465802]: Uploading piece... Dec 27 18:37:02.674560328 (531h8m2.545643373s) [noise=true , address=145.79.227.41:4087 , piece_id=XPJRDHHDQ6IKZILOSUYDTTI76YKVPGMWZV3GCZZWAYSYHU5UDPHA, seg_pos={0 0}, upload=465802]: Uploading piece... Dec 27 18:37:02.675458568 (531h8m2.546541623s) [noise=true , address=145.79.227.48:4303 , piece_id=P3QTZFEAFVCNOBNSCBZECXXFAKASACFSYPKOEHXESCRPHTC3HAAA, seg_pos={0 0}, upload=465802]: Uploading piece... Dec 27 18:37:02.675687425 (531h8m2.54677047s) [noise=true , address=145.79.227.36:4519 , piece_id=GL4ML7GPSL6KCPZ35RQHVY4R367P3UZOF2NRXRLCPEWLQEW3PVGA, seg_pos={0 0}, upload=465802]: Uploading piece... Dec 27 18:37:02.677035150 (531h8m2.548118195s) [noise=true , address=145.79.227.4:4186 , piece_id=NWVFHTEA2MCJHVOXIBTRH4TDI4MTQIYJBCDC62EMOICTF24MGOIQ, seg_pos={0 0}, upload=465802]: Uploading piece... Dec 27 18:37:02.682231345 (531h8m2.5533144s) [noise=true , address=145.79.227.11:4730 , piece_id=QKRYTTB6W7VTPCDC2I2STGJBDFRPQEPNTYFSWTK725MET6RE3XVQ, seg_pos={0 0}, upload=465802]: Uploading piece... Dec 27 18:37:02.682510687 (531h8m2.553593732s) [noise=true , address=45.91.116.14:4190 , piece_id=BVKOPS5DOCBV6MVHUXEDBBP4UZJSCQOEFXLWRZOYAYMQ25I4T4VA, seg_pos={0 0}, upload=465802]: Uploading piece... Dec 27 18:37:02.686033365 (531h8m2.55711641s) [noise=true , address=145.79.227.38:4979 , piece_id=SADDJUX4YMILZDQYWS3L56YAFKX4AZMSZ55CXW3I4NR2OQIOUGJA, seg_pos={0 0}, upload=465802]: Uploading piece... Dec 27 18:37:02.689074085 (531h8m2.56015713s) [noise=true , address=77.90.31.11:4556 , piece_id=7YVGJETUAATMNUSDN7QHQLAA7YMKYH5RA55YELHLFZ6MEDZB6XRA, seg_pos={0 0}, upload=465802]: Uploading piece... Dec 27 18:37:02.690158820 (531h8m2.561241865s) [noise=true , address=45.91.116.6:4976 , piece_id=AD7EFQ357KNIUWKHHMWK6VKZKL5FSF5ULUHMISRPLWCWXD4R2RNA, seg_pos={0 0}, upload=465802]: Uploading piece... Dec 27 18:37:02.691877363 (531h8m2.562960408s) [noise=true , address=145.79.227.3:4149 , piece_id=ZGT4W2LTT6F4TT33W3Z2JX5IEYE2523JYLUYR6SHFKQCOS5GME2Q, seg_pos={0 0}, upload=465802]: Uploading piece... Dec 27 18:37:02.700691536 (531h8m2.571774592s) [noise=true , address=145.79.227.39:4014 , piece_id=4WQ2GNXJZOH3VMCA2QVPDEOQT63XIAJA4P533MA7GERN7OOVWJTA, seg_pos={0 0}, upload=465802]: Uploading piece... Dec 27 18:37:04.292075532 (531h8m4.163158587s) [upload=465802]: Next returned nil segment Dec 27 18:37:04.292093526 (531h8m4.163176591s) [upload=465802]: Waiting for error group managing segments... Dec 27 18:37:04.676189002 (531h8m4.547272037s) [noise=true , address=145.79.227.36:4519 , piece_id=GL4ML7GPSL6KCPZ35RQHVY4R367P3UZOF2NRXRLCPEWLQEW3PVGA, seg_pos={0 0}, upload=465802]: Done uploading piece. err: <nil> Dec 27 18:37:04.682956829 (531h8m4.554039874s) [noise=true , address=145.79.227.11:4730 , piece_id=QKRYTTB6W7VTPCDC2I2STGJBDFRPQEPNTYFSWTK725MET6RE3XVQ, seg_pos={0 0}, upload=465802]: Done uploading piece. err: <nil> Dec 27 18:37:04.690984695 (531h8m4.562067741s) [noise=true , address=145.79.227.41:4087 , piece_id=XPJRDHHDQ6IKZILOSUYDTTI76YKVPGMWZV3GCZZWAYSYHU5UDPHA, seg_pos={0 0}, upload=465802]: Done uploading piece. err: <nil> Dec 27 18:37:04.692686806 (531h8m4.563769851s) [noise=true , address=145.79.227.48:4303 , piece_id=P3QTZFEAFVCNOBNSCBZECXXFAKASACFSYPKOEHXESCRPHTC3HAAA, seg_pos={0 0}, upload=465802]: Done uploading piece. err: <nil> Dec 27 18:37:04.696213512 (531h8m4.567296547s) [noise=true , address=145.79.227.14:4839 , piece_id=W5AZFHG7JBI27CR7JLEFK7XLURGM4F2BVAAU6QE43RRX3E22YF5Q, seg_pos={0 0}, upload=465802]: Done uploading piece. err: <nil> Dec 27 18:37:04.698768577 (531h8m4.569851622s) [noise=true , address=145.79.227.38:4979 , piece_id=SADDJUX4YMILZDQYWS3L56YAFKX4AZMSZ55CXW3I4NR2OQIOUGJA, seg_pos={0 0}, upload=465802]: Done uploading piece. err: <nil> Dec 27 18:37:04.699320859 (531h8m4.570403894s) [noise=true , address=145.79.227.4:4186 , piece_id=NWVFHTEA2MCJHVOXIBTRH4TDI4MTQIYJBCDC62EMOICTF24MGOIQ, seg_pos={0 0}, upload=465802]: Done uploading piece. err: <nil> Dec 27 18:37:04.703301408 (531h8m4.574384443s) [noise=true , address=145.79.227.25:4619 , piece_id=5LL7MPNI5N5NC7CKQ4IRHELA6VZYLCJW277UR3DQCBB2CTGCK2MA, seg_pos={0 0}, upload=465802]: Done uploading piece. err: <nil> Dec 27 18:37:04.705644741 (531h8m4.576727786s) [noise=true , address=145.79.227.39:4014 , piece_id=4WQ2GNXJZOH3VMCA2QVPDEOQT63XIAJA4P533MA7GERN7OOVWJTA, seg_pos={0 0}, upload=465802]: Done uploading piece. err: <nil> Dec 27 18:37:04.709300993 (531h8m4.580384038s) [noise=true , address=145.79.227.35:4483 , piece_id=S3SCCSH2Q6WCIATY26OP5SKAUKU3VK67ZIUQG2E3TF3MNT4BIZOA, seg_pos={0 0}, upload=465802]: Done uploading piece. err: <nil> Dec 27 18:37:04.724502079 (531h8m4.595585114s) [noise=true , address=145.79.227.3:4149 , piece_id=ZGT4W2LTT6F4TT33W3Z2JX5IEYE2523JYLUYR6SHFKQCOS5GME2Q, seg_pos={0 0}, upload=465802]: Done uploading piece. err: <nil> Dec 27 18:37:04.869318814 (531h8m4.74040186s) [noise=true , address=77.90.31.11:4556 , piece_id=7YVGJETUAATMNUSDN7QHQLAA7YMKYH5RA55YELHLFZ6MEDZB6XRA, seg_pos={0 0}, upload=465802]: Done uploading piece. err: <nil> Dec 27 18:37:04.958404830 (531h8m4.829487866s) [noise=true , address=45.91.116.5:4297 , piece_id=ASCZI5WE3BVWLXVE6EUFWGWLK4JCRMPH2FWHLOXLNBSQ4QLENIEA, seg_pos={0 0}, upload=465802]: Done uploading piece. err: <nil> Dec 27 18:37:04.958543004 (531h8m4.829626049s) [seg_pos={0 0}, upload=465802]: Segment reached optimal threshold of 13 pieces. Dec 27 18:37:04.958813237 (531h8m4.829896282s) [noise=true , address=45.91.116.7:4012 , piece_id=XQS343PYEPATYNGGFVI33MDQBVWHDBAM3MYC2Y5ZPZBV6F3ROK4Q, seg_pos={0 0}, upload=465802]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: expected piece hash; context canceled; EOF Dec 27 18:37:04.958826173 (531h8m4.829909218s) [noise=true , address=45.91.116.7:4012 , piece_id=XQS343PYEPATYNGGFVI33MDQBVWHDBAM3MYC2Y5ZPZBV6F3ROK4Q, seg_pos={0 0}, upload=465802]: Long tail cancellation due to optimal threshold Dec 27 18:37:04.958918368 (531h8m4.830001413s) [noise=true , address=45.91.116.6:4976 , piece_id=AD7EFQ357KNIUWKHHMWK6VKZKL5FSF5ULUHMISRPLWCWXD4R2RNA, seg_pos={0 0}, upload=465802]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: write tcp 188.166.225.170:48576->45.91.116.6:4976: use of closed network connection; write tcp 188.166.225.170:48576->45.91.116.6:4976: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48576->45.91.116.6:4976: use of closed network connection Dec 27 18:37:04.958939819 (531h8m4.830022874s) [noise=true , address=45.91.116.6:4976 , piece_id=AD7EFQ357KNIUWKHHMWK6VKZKL5FSF5ULUHMISRPLWCWXD4R2RNA, seg_pos={0 0}, upload=465802]: Long tail cancellation due to optimal threshold Dec 27 18:37:04.959166651 (531h8m4.830249696s) [noise=true , address=45.91.116.14:4190 , piece_id=BVKOPS5DOCBV6MVHUXEDBBP4UZJSCQOEFXLWRZOYAYMQ25I4T4VA, seg_pos={0 0}, upload=465802]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Dec 27 18:37:04.959175558 (531h8m4.830258603s) [noise=true , address=45.91.116.14:4190 , piece_id=BVKOPS5DOCBV6MVHUXEDBBP4UZJSCQOEFXLWRZOYAYMQ25I4T4VA, seg_pos={0 0}, upload=465802]: Long tail cancellation due to optimal threshold Dec 27 18:37:04.959198932 (531h8m4.830281977s) [seg_pos={0 0}, upload=465802]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 18:37:04.959288743 (531h8m4.830371778s) [upload=465802]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 18:37:04.959295386 (531h8m4.830378421s) [upload=465802]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 18:37:05.259099566 (531h8m5.130182611s) [upload=465802]: Done uploading segments... Dec 27 18:37:05.259122450 (531h8m5.130205495s) [upload=465802]: Upload finished. err: <nil> Duration: 18.492463147s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [...X...X.X........X.......X..X.XXXXXXX...X.........XXXX..XXX] duration: 30.003811833s trace: https://jaeger.dev.storj.io/trace/73a3c8fc140bebab FAIL (2025-12-27 18:37:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 18:37:02.424684117 (531h8m2.295767163s) [upload=465802]: Starting upload Dec 27 18:37:02.424743982 (531h8m2.295827017s) [upload=465802]: Done starting upload Dec 27 18:37:02.424870984 (531h8m2.295954019s) [upload=465802]: Uploading segments... Dec 27 18:37:02.424980021 (531h8m2.296063066s) [upload=465802]: Got next segment. Inline: false Dec 27 18:37:02.425016992 (531h8m2.296100027s) [upload=465802]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 18:37:02.425022592 (531h8m2.296105637s) [upload=465802]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 18:37:02.672179384 (531h8m2.54326245s) [seg_pos={0 0}, upload=465802]: Begin upload segment... Dec 27 18:37:02.672240862 (531h8m2.543323907s) [seg_pos={0 0}, upload=465802]: Done begin upload segment. Dec 27 18:37:02.672277261 (531h8m2.543360306s) [noise=true , address=45.91.116.5:4297 , piece_id=ASCZI5WE3BVWLXVE6EUFWGWLK4JCRMPH2FWHLOXLNBSQ4QLENIEA, seg_pos={0 0}, upload=465802]: Uploading piece... Dec 27 18:37:02.672409191 (531h8m2.543492236s) [noise=true , address=145.79.227.14:4839 , piece_id=W5AZFHG7JBI27CR7JLEFK7XLURGM4F2BVAAU6QE43RRX3E22YF5Q, seg_pos={0 0}, upload=465802]: Uploading piece... Dec 27 18:37:02.673048930 (531h8m2.544131975s) [noise=true , address=145.79.227.35:4483 , piece_id=S3SCCSH2Q6WCIATY26OP5SKAUKU3VK67ZIUQG2E3TF3MNT4BIZOA, seg_pos={0 0}, upload=465802]: Uploading piece... Dec 27 18:37:02.673507654 (531h8m2.544590699s) [noise=true , address=45.91.116.7:4012 , piece_id=XQS343PYEPATYNGGFVI33MDQBVWHDBAM3MYC2Y5ZPZBV6F3ROK4Q, seg_pos={0 0}, upload=465802]: Uploading piece... Dec 27 18:37:02.674012164 (531h8m2.545095209s) [noise=true , address=145.79.227.25:4619 , piece_id=5LL7MPNI5N5NC7CKQ4IRHELA6VZYLCJW277UR3DQCBB2CTGCK2MA, seg_pos={0 0}, upload=465802]: Uploading piece... Dec 27 18:37:02.674560328 (531h8m2.545643373s) [noise=true , address=145.79.227.41:4087 , piece_id=XPJRDHHDQ6IKZILOSUYDTTI76YKVPGMWZV3GCZZWAYSYHU5UDPHA, seg_pos={0 0}, upload=465802]: Uploading piece... Dec 27 18:37:02.675458568 (531h8m2.546541623s) [noise=true , address=145.79.227.48:4303 , piece_id=P3QTZFEAFVCNOBNSCBZECXXFAKASACFSYPKOEHXESCRPHTC3HAAA, seg_pos={0 0}, upload=465802]: Uploading piece... Dec 27 18:37:02.675687425 (531h8m2.54677047s) [noise=true , address=145.79.227.36:4519 , piece_id=GL4ML7GPSL6KCPZ35RQHVY4R367P3UZOF2NRXRLCPEWLQEW3PVGA, seg_pos={0 0}, upload=465802]: Uploading piece... Dec 27 18:37:02.677035150 (531h8m2.548118195s) [noise=true , address=145.79.227.4:4186 , piece_id=NWVFHTEA2MCJHVOXIBTRH4TDI4MTQIYJBCDC62EMOICTF24MGOIQ, seg_pos={0 0}, upload=465802]: Uploading piece... Dec 27 18:37:02.682231345 (531h8m2.5533144s) [noise=true , address=145.79.227.11:4730 , piece_id=QKRYTTB6W7VTPCDC2I2STGJBDFRPQEPNTYFSWTK725MET6RE3XVQ, seg_pos={0 0}, upload=465802]: Uploading piece... Dec 27 18:37:02.682510687 (531h8m2.553593732s) [noise=true , address=45.91.116.14:4190 , piece_id=BVKOPS5DOCBV6MVHUXEDBBP4UZJSCQOEFXLWRZOYAYMQ25I4T4VA, seg_pos={0 0}, upload=465802]: Uploading piece... Dec 27 18:37:02.686033365 (531h8m2.55711641s) [noise=true , address=145.79.227.38:4979 , piece_id=SADDJUX4YMILZDQYWS3L56YAFKX4AZMSZ55CXW3I4NR2OQIOUGJA, seg_pos={0 0}, upload=465802]: Uploading piece... Dec 27 18:37:02.689074085 (531h8m2.56015713s) [noise=true , address=77.90.31.11:4556 , piece_id=7YVGJETUAATMNUSDN7QHQLAA7YMKYH5RA55YELHLFZ6MEDZB6XRA, seg_pos={0 0}, upload=465802]: Uploading piece... Dec 27 18:37:02.690158820 (531h8m2.561241865s) [noise=true , address=45.91.116.6:4976 , piece_id=AD7EFQ357KNIUWKHHMWK6VKZKL5FSF5ULUHMISRPLWCWXD4R2RNA, seg_pos={0 0}, upload=465802]: Uploading piece... Dec 27 18:37:02.691877363 (531h8m2.562960408s) [noise=true , address=145.79.227.3:4149 , piece_id=ZGT4W2LTT6F4TT33W3Z2JX5IEYE2523JYLUYR6SHFKQCOS5GME2Q, seg_pos={0 0}, upload=465802]: Uploading piece... Dec 27 18:37:02.700691536 (531h8m2.571774592s) [noise=true , address=145.79.227.39:4014 , piece_id=4WQ2GNXJZOH3VMCA2QVPDEOQT63XIAJA4P533MA7GERN7OOVWJTA, seg_pos={0 0}, upload=465802]: Uploading piece... Dec 27 18:37:04.292075532 (531h8m4.163158587s) [upload=465802]: Next returned nil segment Dec 27 18:37:04.292093526 (531h8m4.163176591s) [upload=465802]: Waiting for error group managing segments... Dec 27 18:37:04.676189002 (531h8m4.547272037s) [noise=true , address=145.79.227.36:4519 , piece_id=GL4ML7GPSL6KCPZ35RQHVY4R367P3UZOF2NRXRLCPEWLQEW3PVGA, seg_pos={0 0}, upload=465802]: Done uploading piece. err: <nil> Dec 27 18:37:04.682956829 (531h8m4.554039874s) [noise=true , address=145.79.227.11:4730 , piece_id=QKRYTTB6W7VTPCDC2I2STGJBDFRPQEPNTYFSWTK725MET6RE3XVQ, seg_pos={0 0}, upload=465802]: Done uploading piece. err: <nil> Dec 27 18:37:04.690984695 (531h8m4.562067741s) [noise=true , address=145.79.227.41:4087 , piece_id=XPJRDHHDQ6IKZILOSUYDTTI76YKVPGMWZV3GCZZWAYSYHU5UDPHA, seg_pos={0 0}, upload=465802]: Done uploading piece. err: <nil> Dec 27 18:37:04.692686806 (531h8m4.563769851s) [noise=true , address=145.79.227.48:4303 , piece_id=P3QTZFEAFVCNOBNSCBZECXXFAKASACFSYPKOEHXESCRPHTC3HAAA, seg_pos={0 0}, upload=465802]: Done uploading piece. err: <nil> Dec 27 18:37:04.696213512 (531h8m4.567296547s) [noise=true , address=145.79.227.14:4839 , piece_id=W5AZFHG7JBI27CR7JLEFK7XLURGM4F2BVAAU6QE43RRX3E22YF5Q, seg_pos={0 0}, upload=465802]: Done uploading piece. err: <nil> Dec 27 18:37:04.698768577 (531h8m4.569851622s) [noise=true , address=145.79.227.38:4979 , piece_id=SADDJUX4YMILZDQYWS3L56YAFKX4AZMSZ55CXW3I4NR2OQIOUGJA, seg_pos={0 0}, upload=465802]: Done uploading piece. err: <nil> Dec 27 18:37:04.699320859 (531h8m4.570403894s) [noise=true , address=145.79.227.4:4186 , piece_id=NWVFHTEA2MCJHVOXIBTRH4TDI4MTQIYJBCDC62EMOICTF24MGOIQ, seg_pos={0 0}, upload=465802]: Done uploading piece. err: <nil> Dec 27 18:37:04.703301408 (531h8m4.574384443s) [noise=true , address=145.79.227.25:4619 , piece_id=5LL7MPNI5N5NC7CKQ4IRHELA6VZYLCJW277UR3DQCBB2CTGCK2MA, seg_pos={0 0}, upload=465802]: Done uploading piece. err: <nil> Dec 27 18:37:04.705644741 (531h8m4.576727786s) [noise=true , address=145.79.227.39:4014 , piece_id=4WQ2GNXJZOH3VMCA2QVPDEOQT63XIAJA4P533MA7GERN7OOVWJTA, seg_pos={0 0}, upload=465802]: Done uploading piece. err: <nil> Dec 27 18:37:04.709300993 (531h8m4.580384038s) [noise=true , address=145.79.227.35:4483 , piece_id=S3SCCSH2Q6WCIATY26OP5SKAUKU3VK67ZIUQG2E3TF3MNT4BIZOA, seg_pos={0 0}, upload=465802]: Done uploading piece. err: <nil> Dec 27 18:37:04.724502079 (531h8m4.595585114s) [noise=true , address=145.79.227.3:4149 , piece_id=ZGT4W2LTT6F4TT33W3Z2JX5IEYE2523JYLUYR6SHFKQCOS5GME2Q, seg_pos={0 0}, upload=465802]: Done uploading piece. err: <nil> Dec 27 18:37:04.869318814 (531h8m4.74040186s) [noise=true , address=77.90.31.11:4556 , piece_id=7YVGJETUAATMNUSDN7QHQLAA7YMKYH5RA55YELHLFZ6MEDZB6XRA, seg_pos={0 0}, upload=465802]: Done uploading piece. err: <nil> Dec 27 18:37:04.958404830 (531h8m4.829487866s) [noise=true , address=45.91.116.5:4297 , piece_id=ASCZI5WE3BVWLXVE6EUFWGWLK4JCRMPH2FWHLOXLNBSQ4QLENIEA, seg_pos={0 0}, upload=465802]: Done uploading piece. err: <nil> Dec 27 18:37:04.958543004 (531h8m4.829626049s) [seg_pos={0 0}, upload=465802]: Segment reached optimal threshold of 13 pieces. Dec 27 18:37:04.958813237 (531h8m4.829896282s) [noise=true , address=45.91.116.7:4012 , piece_id=XQS343PYEPATYNGGFVI33MDQBVWHDBAM3MYC2Y5ZPZBV6F3ROK4Q, seg_pos={0 0}, upload=465802]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: expected piece hash; context canceled; EOF Dec 27 18:37:04.958826173 (531h8m4.829909218s) [noise=true , address=45.91.116.7:4012 , piece_id=XQS343PYEPATYNGGFVI33MDQBVWHDBAM3MYC2Y5ZPZBV6F3ROK4Q, seg_pos={0 0}, upload=465802]: Long tail cancellation due to optimal threshold Dec 27 18:37:04.958918368 (531h8m4.830001413s) [noise=true , address=45.91.116.6:4976 , piece_id=AD7EFQ357KNIUWKHHMWK6VKZKL5FSF5ULUHMISRPLWCWXD4R2RNA, seg_pos={0 0}, upload=465802]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: write tcp 188.166.225.170:48576->45.91.116.6:4976: use of closed network connection; write tcp 188.166.225.170:48576->45.91.116.6:4976: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48576->45.91.116.6:4976: use of closed network connection Dec 27 18:37:04.958939819 (531h8m4.830022874s) [noise=true , address=45.91.116.6:4976 , piece_id=AD7EFQ357KNIUWKHHMWK6VKZKL5FSF5ULUHMISRPLWCWXD4R2RNA, seg_pos={0 0}, upload=465802]: Long tail cancellation due to optimal threshold Dec 27 18:37:04.959166651 (531h8m4.830249696s) [noise=true , address=45.91.116.14:4190 , piece_id=BVKOPS5DOCBV6MVHUXEDBBP4UZJSCQOEFXLWRZOYAYMQ25I4T4VA, seg_pos={0 0}, upload=465802]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Dec 27 18:37:04.959175558 (531h8m4.830258603s) [noise=true , address=45.91.116.14:4190 , piece_id=BVKOPS5DOCBV6MVHUXEDBBP4UZJSCQOEFXLWRZOYAYMQ25I4T4VA, seg_pos={0 0}, upload=465802]: Long tail cancellation due to optimal threshold Dec 27 18:37:04.959198932 (531h8m4.830281977s) [seg_pos={0 0}, upload=465802]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 18:37:04.959288743 (531h8m4.830371778s) [upload=465802]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 18:37:04.959295386 (531h8m4.830378421s) [upload=465802]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 18:37:05.259099566 (531h8m5.130182611s) [upload=465802]: Done uploading segments... Dec 27 18:37:05.259122450 (531h8m5.130205495s) [upload=465802]: Upload finished. err: <nil> Duration: 30.003834868s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [....X...X.X........X.......X..X.XXXXXXX...X.........XXXX..XX] duration: 30.002417744s trace: https://jaeger.dev.storj.io/trace/207ae3245d432d09 FAIL (2025-12-27 18:36:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: YVGRUAGMQ5HBLLIT7VC6U6ZBN4A7MYZNDST2YMCACOSBYBGIGTEQ) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: P5D4QL42O5Y7T7UZEFJDR6O4T7JBBXFTE23ZTFGO6QFY4LYRYF5Q) context deadline exceeded; piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: 7THNT2GAPOZVIXR7UMQWN7GWBL4JSJKAISYG5UVLZ55TY72OFUJQ) context deadline exceeded ======LOGS====== Dec 27 18:36:08.488379704 (531h7m8.359462739s) [upload=465789]: Starting upload Dec 27 18:36:08.488433066 (531h7m8.359516101s) [upload=465789]: Done starting upload Dec 27 18:36:08.488601536 (531h7m8.359684571s) [upload=465789]: Uploading segments... Dec 27 18:36:08.488633136 (531h7m8.359716171s) [upload=465789]: Got next segment. Inline: false Dec 27 18:36:08.488656932 (531h7m8.359739967s) [upload=465789]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 18:36:08.488660759 (531h7m8.359743794s) [upload=465789]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 18:36:08.741536368 (531h7m8.612619403s) [seg_pos={0 0}, upload=465789]: Begin upload segment... Dec 27 18:36:08.741594740 (531h7m8.612677775s) [seg_pos={0 0}, upload=465789]: Done begin upload segment. Dec 27 18:36:08.741635096 (531h7m8.612718131s) [noise=true , address=145.79.227.21:4474 , piece_id=TZDUIXMFBEZWQLHSRS4GPB23VWBYKZMVSBGDHOBFTYTCXAVTJ5VQ, seg_pos={0 0}, upload=465789]: Uploading piece... Dec 27 18:36:08.741746969 (531h7m8.612830024s) [noise=true , address=77.90.31.15:4700 , piece_id=P5D4QL42O5Y7T7UZEFJDR6O4T7JBBXFTE23ZTFGO6QFY4LYRYF5Q, seg_pos={0 0}, upload=465789]: Uploading piece... Dec 27 18:36:08.742322314 (531h7m8.61340537s) [noise=true , address=145.79.227.11:4730 , piece_id=HUEW2CG6E5RG566C3SDFAQN2BGRT6JC6LSUYY6CL6DJFCC34FOKQ, seg_pos={0 0}, upload=465789]: Uploading piece... Dec 27 18:36:08.742731724 (531h7m8.613814769s) [noise=true , address=145.79.227.38:4977 , piece_id=XNYFSKNOME52IGKZR7BIAG7G7ROM4G2HFDCDKVWTSQL7P6QM66WA, seg_pos={0 0}, upload=465789]: Uploading piece... Dec 27 18:36:08.743076521 (531h7m8.614159566s) [noise=true , address=77.90.31.13:4629 , piece_id=7THNT2GAPOZVIXR7UMQWN7GWBL4JSJKAISYG5UVLZ55TY72OFUJQ, seg_pos={0 0}, upload=465789]: Uploading piece... Dec 27 18:36:08.743418773 (531h7m8.614501808s) [noise=true , address=45.91.116.5:4298 , piece_id=GKVYRTZMQCH2R7TGKFUOBJU5LRQQ2MHZWUEWGVKBZDVIW357LHSA, seg_pos={0 0}, upload=465789]: Uploading piece... Dec 27 18:36:08.743731388 (531h7m8.614814433s) [noise=true , address=145.79.227.46:4601 , piece_id=LSCZFBUYWOWCLDOTIJI2534FEUJMUR3XNIC6JCFSVUECIL34NPOQ, seg_pos={0 0}, upload=465789]: Uploading piece... Dec 27 18:36:08.744097606 (531h7m8.615180641s) [noise=true , address=145.79.227.42:4122 , piece_id=YY5VKV6UN6A7DRPUVXAMTWLXETDAY7UPX4MJNTMHOQAMWXVRNCXA, seg_pos={0 0}, upload=465789]: Uploading piece... Dec 27 18:36:08.744481306 (531h7m8.615564341s) [noise=true , address=77.90.31.11:4553 , piece_id=YJKFA2WZBZNO74MSHUYBMPOLBGRI5H6JAZDT3BM2KLEGLS6WTXNA, seg_pos={0 0}, upload=465789]: Uploading piece... Dec 27 18:36:08.745909165 (531h7m8.61699221s) [noise=true , address=145.79.227.37:4565 , piece_id=3RYQM5NJ3FJ6JJCE3SRUTQAZUUXXU2CYMQYJPNTQVSJRG5XSX37Q, seg_pos={0 0}, upload=465789]: Uploading piece... Dec 27 18:36:08.746217171 (531h7m8.617300236s) [noise=true , address=45.91.116.6:4973 , piece_id=WXDWCYUJ3TO7XJBRZLFRGZH2ES3TIFON4D4FYX7M6FIPYHXZN4NA, seg_pos={0 0}, upload=465789]: Uploading piece... Dec 27 18:36:08.747736715 (531h7m8.61881975s) [noise=true , address=145.79.227.15:4874 , piece_id=EXBPENM7OOSYFT4G444UWK5VEAC3HITP65MFPZVRACRBWVCJOLNA, seg_pos={0 0}, upload=465789]: Uploading piece... Dec 27 18:36:08.749223015 (531h7m8.62030605s) [noise=true , address=145.79.227.48:4303 , piece_id=23UFGUYKFHJ5WD4MHBRFSQY5LBAVAWYJ6S262254T477S5VGWJGQ, seg_pos={0 0}, upload=465789]: Uploading piece... Dec 27 18:36:08.755873529 (531h7m8.626956564s) [noise=true , address=145.79.227.39:4015 , piece_id=E6AZSFHQ37II5XA64ZMHAOWJSFSW6KM7OETOLRBYFPSPKNG2QBFA, seg_pos={0 0}, upload=465789]: Uploading piece... Dec 27 18:36:08.756242280 (531h7m8.627325356s) [noise=true , address=45.91.116.13:4154 , piece_id=YVGRUAGMQ5HBLLIT7VC6U6ZBN4A7MYZNDST2YMCACOSBYBGIGTEQ, seg_pos={0 0}, upload=465789]: Uploading piece... Dec 27 18:36:08.757628019 (531h7m8.628711054s) [noise=true , address=145.79.227.45:4229 , piece_id=HDXRJ4LCHDMNCZXM75BH567FXFQGYVNMMRIHR7Z374AR3COPXZDA, seg_pos={0 0}, upload=465789]: Uploading piece... Dec 27 18:36:10.452745883 (531h7m10.323828918s) [upload=465789]: Next returned nil segment Dec 27 18:36:10.452765139 (531h7m10.323848184s) [upload=465789]: Waiting for error group managing segments... Dec 27 18:36:10.729978322 (531h7m10.601061368s) [noise=true , address=145.79.227.11:4730 , piece_id=HUEW2CG6E5RG566C3SDFAQN2BGRT6JC6LSUYY6CL6DJFCC34FOKQ, seg_pos={0 0}, upload=465789]: Done uploading piece. err: <nil> Dec 27 18:36:10.744947747 (531h7m10.616030792s) [noise=true , address=145.79.227.39:4015 , piece_id=E6AZSFHQ37II5XA64ZMHAOWJSFSW6KM7OETOLRBYFPSPKNG2QBFA, seg_pos={0 0}, upload=465789]: Done uploading piece. err: <nil> Dec 27 18:36:10.745258869 (531h7m10.616341904s) [noise=true , address=145.79.227.42:4122 , piece_id=YY5VKV6UN6A7DRPUVXAMTWLXETDAY7UPX4MJNTMHOQAMWXVRNCXA, seg_pos={0 0}, upload=465789]: Done uploading piece. err: <nil> Dec 27 18:36:10.754056121 (531h7m10.625139166s) [noise=true , address=145.79.227.45:4229 , piece_id=HDXRJ4LCHDMNCZXM75BH567FXFQGYVNMMRIHR7Z374AR3COPXZDA, seg_pos={0 0}, upload=465789]: Done uploading piece. err: <nil> Dec 27 18:36:10.756913031 (531h7m10.627996076s) [noise=true , address=145.79.227.38:4977 , piece_id=XNYFSKNOME52IGKZR7BIAG7G7ROM4G2HFDCDKVWTSQL7P6QM66WA, seg_pos={0 0}, upload=465789]: Done uploading piece. err: <nil> Dec 27 18:36:10.757680162 (531h7m10.628763207s) [noise=true , address=145.79.227.21:4474 , piece_id=TZDUIXMFBEZWQLHSRS4GPB23VWBYKZMVSBGDHOBFTYTCXAVTJ5VQ, seg_pos={0 0}, upload=465789]: Done uploading piece. err: <nil> Dec 27 18:36:10.760108356 (531h7m10.631191401s) [noise=true , address=145.79.227.48:4303 , piece_id=23UFGUYKFHJ5WD4MHBRFSQY5LBAVAWYJ6S262254T477S5VGWJGQ, seg_pos={0 0}, upload=465789]: Done uploading piece. err: <nil> Dec 27 18:36:10.765056798 (531h7m10.636139843s) [noise=true , address=145.79.227.46:4601 , piece_id=LSCZFBUYWOWCLDOTIJI2534FEUJMUR3XNIC6JCFSVUECIL34NPOQ, seg_pos={0 0}, upload=465789]: Done uploading piece. err: <nil> Dec 27 18:36:10.773488183 (531h7m10.644571218s) [noise=true , address=145.79.227.37:4565 , piece_id=3RYQM5NJ3FJ6JJCE3SRUTQAZUUXXU2CYMQYJPNTQVSJRG5XSX37Q, seg_pos={0 0}, upload=465789]: Done uploading piece. err: <nil> Dec 27 18:36:10.780588433 (531h7m10.651671468s) [noise=true , address=145.79.227.15:4874 , piece_id=EXBPENM7OOSYFT4G444UWK5VEAC3HITP65MFPZVRACRBWVCJOLNA, seg_pos={0 0}, upload=465789]: Done uploading piece. err: <nil> Dec 27 18:36:10.934580001 (531h7m10.805663046s) [noise=true , address=77.90.31.15:4700 , piece_id=P5D4QL42O5Y7T7UZEFJDR6O4T7JBBXFTE23ZTFGO6QFY4LYRYF5Q, seg_pos={0 0}, upload=465789]: Done uploading piece. err: <nil> Dec 27 18:36:10.985441129 (531h7m10.856524164s) [noise=true , address=77.90.31.13:4629 , piece_id=7THNT2GAPOZVIXR7UMQWN7GWBL4JSJKAISYG5UVLZ55TY72OFUJQ, seg_pos={0 0}, upload=465789]: Done uploading piece. err: <nil> Dec 27 18:36:11.017180577 (531h7m10.888263612s) [noise=true , address=45.91.116.13:4154 , piece_id=YVGRUAGMQ5HBLLIT7VC6U6ZBN4A7MYZNDST2YMCACOSBYBGIGTEQ, seg_pos={0 0}, upload=465789]: Done uploading piece. err: <nil> Dec 27 18:36:11.017201556 (531h7m10.888284591s) [seg_pos={0 0}, upload=465789]: Segment reached optimal threshold of 13 pieces. Dec 27 18:36:11.017419230 (531h7m10.888502275s) [noise=true , address=77.90.31.11:4553 , piece_id=YJKFA2WZBZNO74MSHUYBMPOLBGRI5H6JAZDT3BM2KLEGLS6WTXNA, seg_pos={0 0}, upload=465789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:47644->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:47644->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47644->77.90.31.11:4553: use of closed network connection Dec 27 18:36:11.017429511 (531h7m10.888512546s) [noise=true , address=77.90.31.11:4553 , piece_id=YJKFA2WZBZNO74MSHUYBMPOLBGRI5H6JAZDT3BM2KLEGLS6WTXNA, seg_pos={0 0}, upload=465789]: Long tail cancellation due to optimal threshold Dec 27 18:36:11.017579175 (531h7m10.88866224s) [noise=true , address=45.91.116.6:4973 , piece_id=WXDWCYUJ3TO7XJBRZLFRGZH2ES3TIFON4D4FYX7M6FIPYHXZN4NA, seg_pos={0 0}, upload=465789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Dec 27 18:36:11.017613831 (531h7m10.888696886s) [noise=true , address=45.91.116.6:4973 , piece_id=WXDWCYUJ3TO7XJBRZLFRGZH2ES3TIFON4D4FYX7M6FIPYHXZN4NA, seg_pos={0 0}, upload=465789]: Long tail cancellation due to optimal threshold Dec 27 18:36:11.017624401 (531h7m10.888707436s) [noise=true , address=45.91.116.5:4298 , piece_id=GKVYRTZMQCH2R7TGKFUOBJU5LRQQ2MHZWUEWGVKBZDVIW357LHSA, seg_pos={0 0}, upload=465789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 27 18:36:11.017634030 (531h7m10.888717065s) [noise=true , address=45.91.116.5:4298 , piece_id=GKVYRTZMQCH2R7TGKFUOBJU5LRQQ2MHZWUEWGVKBZDVIW357LHSA, seg_pos={0 0}, upload=465789]: Long tail cancellation due to optimal threshold Dec 27 18:36:11.017727127 (531h7m10.888810162s) [seg_pos={0 0}, upload=465789]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 18:36:11.017862565 (531h7m10.8889456s) [upload=465789]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 18:36:11.017868997 (531h7m10.888952052s) [upload=465789]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 18:36:11.308665775 (531h7m11.179748821s) [upload=465789]: Done uploading segments... Dec 27 18:36:11.308690192 (531h7m11.179773227s) [upload=465789]: Upload finished. err: <nil> Duration: 30.002442421s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [X....X...X.X........X.......X..X.XXXXXXX...X.........XXXX..X] duration: 30.002149484s trace: https://jaeger.dev.storj.io/trace/7c75d88dbfcfe426 FAIL (2025-12-27 18:35:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: GUPMFUQBHSKHVKR3J4Y7EZCBEHRMQ3UDACW6T73XVDSZYFAQJ3OA) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: PUA63GSD2C7NWJQOZQIIZ7F37NGC5J2E57UK4XVBJASDPQCIOQZA) context deadline exceeded ======LOGS====== 27 18:35:08.238804493 (531h6m8.109887538s) [upload=465774]: Uploading segments... Dec 27 18:35:08.238970550 (531h6m8.110053585s) [upload=465774]: Got next segment. Inline: false Dec 27 18:35:08.239018821 (531h6m8.110101866s) [upload=465774]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 18:35:08.239023992 (531h6m8.110107037s) [upload=465774]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 18:35:08.486211550 (531h6m8.357294595s) [seg_pos={0 0}, upload=465774]: Begin upload segment... Dec 27 18:35:08.486269370 (531h6m8.357352405s) [seg_pos={0 0}, upload=465774]: Done begin upload segment. Dec 27 18:35:08.486321118 (531h6m8.357404163s) [noise=true , address=145.79.227.24:4581 , piece_id=5ABRXLJKI5SBBB3HAJI5ER3AJFJ5NHZVQAJ4XUEI2G2MIY3SO3YA, seg_pos={0 0}, upload=465774]: Uploading piece... Dec 27 18:35:08.486388587 (531h6m8.357471632s) [noise=true , address=145.79.227.25:4618 , piece_id=QEKNJSH252DQKNBLPAEHC7HKPVRS7UKW3EK62U4UAU6F2FOTAE7Q, seg_pos={0 0}, upload=465774]: Uploading piece... Dec 27 18:35:08.487122975 (531h6m8.35820601s) [noise=true , address=145.79.227.17:4972 , piece_id=SARLRYE7USNNHQIYHM3RBKZTVSGVVGWNEYNVVJ4GRL2Y4XD4ABMA, seg_pos={0 0}, upload=465774]: Uploading piece... Dec 27 18:35:08.487582119 (531h6m8.358665164s) [noise=true , address=145.79.227.38:4979 , piece_id=Y7F2ZGWVIWIARLRVVMIINE3YPRCN2EOWVUBEOS7XP7HPKF4WGV3Q, seg_pos={0 0}, upload=465774]: Uploading piece... Dec 27 18:35:08.488018531 (531h6m8.359101576s) [noise=true , address=45.91.116.4:4940 , piece_id=5KUW3RSQGNU5N6PSQ6MFLPZAZHKAOVEZ7KBC5EEWNN2CZJIR4E7Q, seg_pos={0 0}, upload=465774]: Uploading piece... Dec 27 18:35:08.488459420 (531h6m8.359542465s) [noise=true , address=145.79.227.15:4874 , piece_id=YNHB4THKJ3USOYQGGTEDK7XF5PNZ6WGUBPX4KLWLXQHBTAHOINIQ, seg_pos={0 0}, upload=465774]: Uploading piece... Dec 27 18:35:08.488904497 (531h6m8.359987542s) [noise=true , address=145.79.227.34:4906 , piece_id=K3FDGUMH44QTYEQJ7CKHQZ7VLQWA3CLPZ47QU4PUL263TZV3U53A, seg_pos={0 0}, upload=465774]: Uploading piece... Dec 27 18:35:08.489368151 (531h6m8.360451196s) [noise=true , address=45.91.116.14:4192 , piece_id=DHH5SDL4KVBRV2CLY5CHMKPRQFFVQNWIGS6C5R3NEXTANU7YVHVA, seg_pos={0 0}, upload=465774]: Uploading piece... Dec 27 18:35:08.490777654 (531h6m8.361860699s) [noise=true , address=145.79.227.18:4007 , piece_id=UU4PBZ4MOHVJSXZHJ5YK6EQ43LG2CMTLLPVBTSEL6GCAYGCZBJUQ, seg_pos={0 0}, upload=465774]: Uploading piece... Dec 27 18:35:08.491245285 (531h6m8.36232832s) [noise=true , address=145.79.227.47:4266 , piece_id=FO2DL6Y6JRCN3IYCC32SWMC32FXSIC6VNH4HCQJA7P25UJMKIKLQ, seg_pos={0 0}, upload=465774]: Uploading piece... Dec 27 18:35:08.492239959 (531h6m8.363323004s) [noise=true , address=145.79.227.11:4732 , piece_id=YS7OPOPFVPHW2PKYJ3XFLBXZ2NPGLENB3UDBWZLHJSTH7W5IN4OQ, seg_pos={0 0}, upload=465774]: Uploading piece... Dec 27 18:35:08.493633743 (531h6m8.364716788s) [noise=true , address=77.90.31.10:4519 , piece_id=GUPMFUQBHSKHVKR3J4Y7EZCBEHRMQ3UDACW6T73XVDSZYFAQJ3OA, seg_pos={0 0}, upload=465774]: Uploading piece... Dec 27 18:35:08.495030572 (531h6m8.366113617s) [noise=true , address=45.91.116.7:4010 , piece_id=PUA63GSD2C7NWJQOZQIIZ7F37NGC5J2E57UK4XVBJASDPQCIOQZA, seg_pos={0 0}, upload=465774]: Uploading piece... Dec 27 18:35:08.496798679 (531h6m8.367881724s) [noise=true , address=145.79.227.12:4765 , piece_id=VISMRBWDJ2BBCJGDWB6IMKPGAIBSBRFTSFOMHGJO27LCPNYGZGYA, seg_pos={0 0}, upload=465774]: Uploading piece... Dec 27 18:35:08.500940837 (531h6m8.372023882s) [noise=true , address=145.79.227.9:4365 , piece_id=4NUDWZ4B4GKGO7WVB5VKHPOLGWGNEP6WFB3NBHHJ27OSIT5QF64A, seg_pos={0 0}, upload=465774]: Uploading piece... Dec 27 18:35:08.505121487 (531h6m8.376204532s) [noise=true , address=77.90.31.14:4664 , piece_id=5QUNP6U67WATL7SK5DPAPUOMKMTE2M5ZFPS4VAFUQ3ZBPQWXW2SQ, seg_pos={0 0}, upload=465774]: Uploading piece... Dec 27 18:35:10.110566599 (531h6m9.981649644s) [upload=465774]: Next returned nil segment Dec 27 18:35:10.110585285 (531h6m9.98166833s) [upload=465774]: Waiting for error group managing segments... Dec 27 18:35:10.495280874 (531h6m10.366363919s) [noise=true , address=145.79.227.17:4972 , piece_id=SARLRYE7USNNHQIYHM3RBKZTVSGVVGWNEYNVVJ4GRL2Y4XD4ABMA, seg_pos={0 0}, upload=465774]: Done uploading piece. err: <nil> Dec 27 18:35:10.495851189 (531h6m10.366934224s) [noise=true , address=145.79.227.18:4007 , piece_id=UU4PBZ4MOHVJSXZHJ5YK6EQ43LG2CMTLLPVBTSEL6GCAYGCZBJUQ, seg_pos={0 0}, upload=465774]: Done uploading piece. err: <nil> Dec 27 18:35:10.499044580 (531h6m10.370127615s) [noise=true , address=145.79.227.11:4732 , piece_id=YS7OPOPFVPHW2PKYJ3XFLBXZ2NPGLENB3UDBWZLHJSTH7W5IN4OQ, seg_pos={0 0}, upload=465774]: Done uploading piece. err: <nil> Dec 27 18:35:10.504516910 (531h6m10.375599955s) [noise=true , address=145.79.227.38:4979 , piece_id=Y7F2ZGWVIWIARLRVVMIINE3YPRCN2EOWVUBEOS7XP7HPKF4WGV3Q, seg_pos={0 0}, upload=465774]: Done uploading piece. err: <nil> Dec 27 18:35:10.512907297 (531h6m10.383990332s) [noise=true , address=145.79.227.12:4765 , piece_id=VISMRBWDJ2BBCJGDWB6IMKPGAIBSBRFTSFOMHGJO27LCPNYGZGYA, seg_pos={0 0}, upload=465774]: Done uploading piece. err: <nil> Dec 27 18:35:10.513607580 (531h6m10.384690625s) [noise=true , address=145.79.227.47:4266 , piece_id=FO2DL6Y6JRCN3IYCC32SWMC32FXSIC6VNH4HCQJA7P25UJMKIKLQ, seg_pos={0 0}, upload=465774]: Done uploading piece. err: <nil> Dec 27 18:35:10.515158554 (531h6m10.386241599s) [noise=true , address=145.79.227.24:4581 , piece_id=5ABRXLJKI5SBBB3HAJI5ER3AJFJ5NHZVQAJ4XUEI2G2MIY3SO3YA, seg_pos={0 0}, upload=465774]: Done uploading piece. err: <nil> Dec 27 18:35:10.515748057 (531h6m10.386831102s) [noise=true , address=145.79.227.9:4365 , piece_id=4NUDWZ4B4GKGO7WVB5VKHPOLGWGNEP6WFB3NBHHJ27OSIT5QF64A, seg_pos={0 0}, upload=465774]: Done uploading piece. err: <nil> Dec 27 18:35:10.516236036 (531h6m10.387319071s) [noise=true , address=145.79.227.25:4618 , piece_id=QEKNJSH252DQKNBLPAEHC7HKPVRS7UKW3EK62U4UAU6F2FOTAE7Q, seg_pos={0 0}, upload=465774]: Done uploading piece. err: <nil> Dec 27 18:35:10.519369963 (531h6m10.390452998s) [noise=true , address=145.79.227.15:4874 , piece_id=YNHB4THKJ3USOYQGGTEDK7XF5PNZ6WGUBPX4KLWLXQHBTAHOINIQ, seg_pos={0 0}, upload=465774]: Done uploading piece. err: <nil> Dec 27 18:35:10.664436865 (531h6m10.53551991s) [noise=true , address=77.90.31.10:4519 , piece_id=GUPMFUQBHSKHVKR3J4Y7EZCBEHRMQ3UDACW6T73XVDSZYFAQJ3OA, seg_pos={0 0}, upload=465774]: Done uploading piece. err: <nil> Dec 27 18:35:10.715523444 (531h6m10.586606489s) [noise=true , address=45.91.116.14:4192 , piece_id=DHH5SDL4KVBRV2CLY5CHMKPRQFFVQNWIGS6C5R3NEXTANU7YVHVA, seg_pos={0 0}, upload=465774]: Done uploading piece. err: <nil> Dec 27 18:35:10.796093974 (531h6m10.667177019s) [noise=true , address=45.91.116.7:4010 , piece_id=PUA63GSD2C7NWJQOZQIIZ7F37NGC5J2E57UK4XVBJASDPQCIOQZA, seg_pos={0 0}, upload=465774]: Done uploading piece. err: <nil> Dec 27 18:35:10.796123150 (531h6m10.667206185s) [seg_pos={0 0}, upload=465774]: Segment reached optimal threshold of 13 pieces. Dec 27 18:35:10.796344090 (531h6m10.667427135s) [noise=true , address=145.79.227.34:4906 , piece_id=K3FDGUMH44QTYEQJ7CKHQZ7VLQWA3CLPZ47QU4PUL263TZV3U53A, seg_pos={0 0}, upload=465774]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 27 18:35:10.796354329 (531h6m10.667437375s) [noise=true , address=145.79.227.34:4906 , piece_id=K3FDGUMH44QTYEQJ7CKHQZ7VLQWA3CLPZ47QU4PUL263TZV3U53A, seg_pos={0 0}, upload=465774]: Long tail cancellation due to optimal threshold Dec 27 18:35:10.796460812 (531h6m10.667543857s) [noise=true , address=45.91.116.4:4940 , piece_id=5KUW3RSQGNU5N6PSQ6MFLPZAZHKAOVEZ7KBC5EEWNN2CZJIR4E7Q, seg_pos={0 0}, upload=465774]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: write tcp 188.166.225.170:45192->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:45192->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45192->45.91.116.4:4940: use of closed network connection Dec 27 18:35:10.796470652 (531h6m10.667553697s) [noise=true , address=45.91.116.4:4940 , piece_id=5KUW3RSQGNU5N6PSQ6MFLPZAZHKAOVEZ7KBC5EEWNN2CZJIR4E7Q, seg_pos={0 0}, upload=465774]: Long tail cancellation due to optimal threshold Dec 27 18:35:10.796502903 (531h6m10.667585948s) [noise=true , address=77.90.31.14:4664 , piece_id=5QUNP6U67WATL7SK5DPAPUOMKMTE2M5ZFPS4VAFUQ3ZBPQWXW2SQ, seg_pos={0 0}, upload=465774]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: write tcp 188.166.225.170:44472->77.90.31.14:4664: use of closed network connection; write tcp 188.166.225.170:44472->77.90.31.14:4664: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44472->77.90.31.14:4664: use of closed network connection Dec 27 18:35:10.796511008 (531h6m10.667594053s) [noise=true , address=77.90.31.14:4664 , piece_id=5QUNP6U67WATL7SK5DPAPUOMKMTE2M5ZFPS4VAFUQ3ZBPQWXW2SQ, seg_pos={0 0}, upload=465774]: Long tail cancellation due to optimal threshold Dec 27 18:35:10.796532440 (531h6m10.667615485s) [seg_pos={0 0}, upload=465774]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 18:35:10.796676383 (531h6m10.667759429s) [upload=465774]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 18:35:10.796684238 (531h6m10.667767283s) [upload=465774]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 18:35:11.082699201 (531h6m10.953782246s) [upload=465774]: Done uploading segments... Dec 27 18:35:11.082746491 (531h6m10.953829537s) [upload=465774]: Upload finished. err: <nil> Duration: 30.002184802s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [XX....X...X.X........X.......X..X.XXXXXXX...X.........XXXX..] duration: 6.022083992s trace: https://jaeger.dev.storj.io/trace/77c06c284613bcdd FAIL (2025-12-27 18:34:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: GQT4KT6DXZIUP3RBB5CIFT5LULTJZSEXVUMQDNK3SDU5D2OYZL2Q) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: QQBSXSDLRBULRGZ5ZDJJ4N3PQVHSB5IUNEXOVMBBPTA2F5DGH3OQ) context deadline exceeded ======LOGS====== .498693041s) [upload=465736]: Uploading segments... Dec 27 18:32:25.627698233 (531h3m25.498781268s) [upload=465736]: Got next segment. Inline: false Dec 27 18:32:25.627816168 (531h3m25.498899213s) [upload=465736]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 18:32:25.627823151 (531h3m25.498906187s) [upload=465736]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 18:32:25.874268727 (531h3m25.745351772s) [seg_pos={0 0}, upload=465736]: Begin upload segment... Dec 27 18:32:25.874324954 (531h3m25.745407999s) [seg_pos={0 0}, upload=465736]: Done begin upload segment. Dec 27 18:32:25.874351565 (531h3m25.7454346s) [noise=true , address=145.79.227.51:4409 , piece_id=BA5JBEPAEHZJUAVLZVX66JZVKHVDSTMQU3767BTH732YI57OHYRQ, seg_pos={0 0}, upload=465736]: Uploading piece... Dec 27 18:32:25.874375831 (531h3m25.745458876s) [noise=true , address=145.79.227.22:4510 , piece_id=TYDRPSG2QPLKSDETT6OUV4BXC4DYQSIR5Y2ZIWMFZDFMTGXTLRIA, seg_pos={0 0}, upload=465736]: Uploading piece... Dec 27 18:32:25.874436847 (531h3m25.745519882s) [noise=true , address=45.91.116.9:4264 , piece_id=GQT4KT6DXZIUP3RBB5CIFT5LULTJZSEXVUMQDNK3SDU5D2OYZL2Q, seg_pos={0 0}, upload=465736]: Uploading piece... Dec 27 18:32:25.875075614 (531h3m25.746158659s) [noise=true , address=77.90.31.19:4844 , piece_id=IQ6DTEAUK3ZMVVTVEXB65YXIA4R3WLWDBNCUZRZHFHRJ4K3LJM7A, seg_pos={0 0}, upload=465736]: Uploading piece... Dec 27 18:32:25.875509329 (531h3m25.746592375s) [noise=true , address=145.79.227.39:4013 , piece_id=Y2FSILI3NNR7Q5XXGDDLHHTKE2IRFU6O5AMLU547RZPXKWYQRWAA, seg_pos={0 0}, upload=465736]: Uploading piece... Dec 27 18:32:25.875941221 (531h3m25.747024267s) [noise=true , address=145.79.227.25:4618 , piece_id=ZCPAXIEOBJ3BJ2MSKK2EXBCCG523JMCFFVTLKWQ55UEBRMJZVVHA, seg_pos={0 0}, upload=465736]: Uploading piece... Dec 27 18:32:25.876358175 (531h3m25.747441221s) [noise=true , address=145.79.227.2:4114 , piece_id=KCTKDAQCZTBI3GRWEUBHK4GTLQQLCL3FRFNTKTYPHIVYBTO5I3AQ, seg_pos={0 0}, upload=465736]: Uploading piece... Dec 27 18:32:25.876859811 (531h3m25.747942856s) [noise=true , address=45.91.116.14:4189 , piece_id=WALZZPSLAPU6ZDNYRPN4FKWFND73GYWSUIXBYFZ3LA7GC2OZDCQQ, seg_pos={0 0}, upload=465736]: Uploading piece... Dec 27 18:32:25.878331714 (531h3m25.749414759s) [noise=true , address=145.79.227.26:4655 , piece_id=4YBF3DJ4BIH3GHFFG3YI4L4FYTVLDQEXTT5U6XEGHMG4S46BDNCQ, seg_pos={0 0}, upload=465736]: Uploading piece... Dec 27 18:32:25.878408840 (531h3m25.749491875s) [noise=true , address=145.79.227.47:4265 , piece_id=RCA6L5QL7ZBKHRFY7C6KEIBU6Y4EMIO2GEFDYF43BKJQ356JYAGQ, seg_pos={0 0}, upload=465736]: Uploading piece... Dec 27 18:32:25.880110090 (531h3m25.751193125s) [noise=true , address=145.79.227.38:4979 , piece_id=NDHM36NVO7M2SXJ4SVNYHWOBAO2DF2PBQDQRNU4N2ROMAGOOV4MQ, seg_pos={0 0}, upload=465736]: Uploading piece... Dec 27 18:32:25.880593010 (531h3m25.751676055s) [noise=true , address=145.79.227.15:4873 , piece_id=ADWK2XVAE7WOEZSD2FBLN4I5HEQDOMUNADKIG7BQBBBVVDIBADXQ, seg_pos={0 0}, upload=465736]: Uploading piece... Dec 27 18:32:25.881686892 (531h3m25.752769937s) [noise=true , address=45.91.116.12:4228 , piece_id=DSI5RZ5EC3GZ3VWSHPOYLXTP355TDAXJUPFTU6U77N3AVR63DO7Q, seg_pos={0 0}, upload=465736]: Uploading piece... Dec 27 18:32:25.883346833 (531h3m25.754429878s) [noise=true , address=77.90.31.14:4663 , piece_id=QQBSXSDLRBULRGZ5ZDJJ4N3PQVHSB5IUNEXOVMBBPTA2F5DGH3OQ, seg_pos={0 0}, upload=465736]: Uploading piece... Dec 27 18:32:25.883745723 (531h3m25.754828768s) [noise=true , address=145.79.227.28:4725 , piece_id=RZOCXC2TOJK75HJUKBYG6ZWJF5VKLAIZF6KFISEIDQIA6PEBCC5A, seg_pos={0 0}, upload=465736]: Uploading piece... Dec 27 18:32:25.886654291 (531h3m25.757737336s) [noise=true , address=77.90.31.21:4878 , piece_id=B6J6LYWK6W3DUWJJ7AWXZXVD3CAVK2PLZRV7LPGF3EDQCPLIQ6UA, seg_pos={0 0}, upload=465736]: Uploading piece... Dec 27 18:32:27.491958225 (531h3m27.36304126s) [upload=465736]: Next returned nil segment Dec 27 18:32:27.491973453 (531h3m27.363056488s) [upload=465736]: Waiting for error group managing segments... Dec 27 18:32:27.883820980 (531h3m27.754904025s) [noise=true , address=145.79.227.25:4618 , piece_id=ZCPAXIEOBJ3BJ2MSKK2EXBCCG523JMCFFVTLKWQ55UEBRMJZVVHA, seg_pos={0 0}, upload=465736]: Done uploading piece. err: <nil> Dec 27 18:32:27.884654017 (531h3m27.755737052s) [noise=true , address=145.79.227.38:4979 , piece_id=NDHM36NVO7M2SXJ4SVNYHWOBAO2DF2PBQDQRNU4N2ROMAGOOV4MQ, seg_pos={0 0}, upload=465736]: Done uploading piece. err: <nil> Dec 27 18:32:27.892675351 (531h3m27.763758386s) [noise=true , address=145.79.227.39:4013 , piece_id=Y2FSILI3NNR7Q5XXGDDLHHTKE2IRFU6O5AMLU547RZPXKWYQRWAA, seg_pos={0 0}, upload=465736]: Done uploading piece. err: <nil> Dec 27 18:32:27.896075837 (531h3m27.767158872s) [noise=true , address=145.79.227.47:4265 , piece_id=RCA6L5QL7ZBKHRFY7C6KEIBU6Y4EMIO2GEFDYF43BKJQ356JYAGQ, seg_pos={0 0}, upload=465736]: Done uploading piece. err: <nil> Dec 27 18:32:27.897659301 (531h3m27.768742336s) [noise=true , address=145.79.227.15:4873 , piece_id=ADWK2XVAE7WOEZSD2FBLN4I5HEQDOMUNADKIG7BQBBBVVDIBADXQ, seg_pos={0 0}, upload=465736]: Done uploading piece. err: <nil> Dec 27 18:32:27.898201484 (531h3m27.769284519s) [noise=true , address=145.79.227.22:4510 , piece_id=TYDRPSG2QPLKSDETT6OUV4BXC4DYQSIR5Y2ZIWMFZDFMTGXTLRIA, seg_pos={0 0}, upload=465736]: Done uploading piece. err: <nil> Dec 27 18:32:27.904575391 (531h3m27.775658426s) [noise=true , address=145.79.227.28:4725 , piece_id=RZOCXC2TOJK75HJUKBYG6ZWJF5VKLAIZF6KFISEIDQIA6PEBCC5A, seg_pos={0 0}, upload=465736]: Done uploading piece. err: <nil> Dec 27 18:32:27.912159122 (531h3m27.783242167s) [noise=true , address=145.79.227.26:4655 , piece_id=4YBF3DJ4BIH3GHFFG3YI4L4FYTVLDQEXTT5U6XEGHMG4S46BDNCQ, seg_pos={0 0}, upload=465736]: Done uploading piece. err: <nil> Dec 27 18:32:28.081094634 (531h3m27.952177669s) [noise=true , address=77.90.31.14:4663 , piece_id=QQBSXSDLRBULRGZ5ZDJJ4N3PQVHSB5IUNEXOVMBBPTA2F5DGH3OQ, seg_pos={0 0}, upload=465736]: Done uploading piece. err: <nil> Dec 27 18:32:28.148951857 (531h3m28.020034902s) [noise=true , address=45.91.116.14:4189 , piece_id=WALZZPSLAPU6ZDNYRPN4FKWFND73GYWSUIXBYFZ3LA7GC2OZDCQQ, seg_pos={0 0}, upload=465736]: Done uploading piece. err: <nil> Dec 27 18:32:28.152422566 (531h3m28.023505611s) [noise=true , address=45.91.116.12:4228 , piece_id=DSI5RZ5EC3GZ3VWSHPOYLXTP355TDAXJUPFTU6U77N3AVR63DO7Q, seg_pos={0 0}, upload=465736]: Done uploading piece. err: <nil> Dec 27 18:32:28.258556914 (531h3m28.129639949s) [noise=true , address=45.91.116.9:4264 , piece_id=GQT4KT6DXZIUP3RBB5CIFT5LULTJZSEXVUMQDNK3SDU5D2OYZL2Q, seg_pos={0 0}, upload=465736]: Done uploading piece. err: <nil> Dec 27 18:32:28.785386931 (531h3m28.656469976s) [noise=true , address=77.90.31.19:4844 , piece_id=IQ6DTEAUK3ZMVVTVEXB65YXIA4R3WLWDBNCUZRZHFHRJ4K3LJM7A, seg_pos={0 0}, upload=465736]: Done uploading piece. err: <nil> Dec 27 18:32:28.785411759 (531h3m28.656494804s) [seg_pos={0 0}, upload=465736]: Segment reached optimal threshold of 13 pieces. Dec 27 18:32:28.785707543 (531h3m28.656790588s) [noise=true , address=145.79.227.2:4114 , piece_id=KCTKDAQCZTBI3GRWEUBHK4GTLQQLCL3FRFNTKTYPHIVYBTO5I3AQ, seg_pos={0 0}, upload=465736]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN): protocol: write tcp 188.166.225.170:34096->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:34096->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34096->145.79.227.2:4114: use of closed network connection Dec 27 18:32:28.785716920 (531h3m28.656799955s) [noise=true , address=145.79.227.2:4114 , piece_id=KCTKDAQCZTBI3GRWEUBHK4GTLQQLCL3FRFNTKTYPHIVYBTO5I3AQ, seg_pos={0 0}, upload=465736]: Long tail cancellation due to optimal threshold Dec 27 18:32:28.786070563 (531h3m28.657153609s) [noise=true , address=77.90.31.21:4878 , piece_id=B6J6LYWK6W3DUWJJ7AWXZXVD3CAVK2PLZRV7LPGF3EDQCPLIQ6UA, seg_pos={0 0}, upload=465736]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 27 18:32:28.786079079 (531h3m28.657162115s) [noise=true , address=77.90.31.21:4878 , piece_id=B6J6LYWK6W3DUWJJ7AWXZXVD3CAVK2PLZRV7LPGF3EDQCPLIQ6UA, seg_pos={0 0}, upload=465736]: Long tail cancellation due to optimal threshold Dec 27 18:32:28.786208105 (531h3m28.65729114s) [noise=true , address=145.79.227.51:4409 , piece_id=BA5JBEPAEHZJUAVLZVX66JZVKHVDSTMQU3767BTH732YI57OHYRQ, seg_pos={0 0}, upload=465736]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UoCGA4FjE4yADPXL1oifSPJssPgSGh81REZHwGv2eu1Y1kEaj): protocol: write tcp 188.166.225.170:51222->145.79.227.51:4409: use of closed network connection; write tcp 188.166.225.170:51222->145.79.227.51:4409: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51222->145.79.227.51:4409: use of closed network connection Dec 27 18:32:28.786216872 (531h3m28.657299917s) [noise=true , address=145.79.227.51:4409 , piece_id=BA5JBEPAEHZJUAVLZVX66JZVKHVDSTMQU3767BTH732YI57OHYRQ, seg_pos={0 0}, upload=465736]: Long tail cancellation due to optimal threshold Dec 27 18:32:28.786237081 (531h3m28.657320126s) [seg_pos={0 0}, upload=465736]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 18:32:28.786323595 (531h3m28.65740664s) [upload=465736]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 18:32:28.786329556 (531h3m28.657412601s) [upload=465736]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 18:32:29.083652482 (531h3m28.954735517s) [upload=465736]: Done uploading segments... Dec 27 18:32:29.083676638 (531h3m28.954759683s) [upload=465736]: Upload finished. err: <nil> Duration: 6.022110243s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [.XX....X...X.X........X.......X..X.XXXXXXX...X.........XXXX.] duration: 14.356854779s trace: https://jaeger.dev.storj.io/trace/77c06c284613bcdd FAIL (2025-12-27 18:33:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: GQT4KT6DXZIUP3RBB5CIFT5LULTJZSEXVUMQDNK3SDU5D2OYZL2Q) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: QQBSXSDLRBULRGZ5ZDJJ4N3PQVHSB5IUNEXOVMBBPTA2F5DGH3OQ) context deadline exceeded ======LOGS====== .498693041s) [upload=465736]: Uploading segments... Dec 27 18:32:25.627698233 (531h3m25.498781268s) [upload=465736]: Got next segment. Inline: false Dec 27 18:32:25.627816168 (531h3m25.498899213s) [upload=465736]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 18:32:25.627823151 (531h3m25.498906187s) [upload=465736]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 18:32:25.874268727 (531h3m25.745351772s) [seg_pos={0 0}, upload=465736]: Begin upload segment... Dec 27 18:32:25.874324954 (531h3m25.745407999s) [seg_pos={0 0}, upload=465736]: Done begin upload segment. Dec 27 18:32:25.874351565 (531h3m25.7454346s) [noise=true , address=145.79.227.51:4409 , piece_id=BA5JBEPAEHZJUAVLZVX66JZVKHVDSTMQU3767BTH732YI57OHYRQ, seg_pos={0 0}, upload=465736]: Uploading piece... Dec 27 18:32:25.874375831 (531h3m25.745458876s) [noise=true , address=145.79.227.22:4510 , piece_id=TYDRPSG2QPLKSDETT6OUV4BXC4DYQSIR5Y2ZIWMFZDFMTGXTLRIA, seg_pos={0 0}, upload=465736]: Uploading piece... Dec 27 18:32:25.874436847 (531h3m25.745519882s) [noise=true , address=45.91.116.9:4264 , piece_id=GQT4KT6DXZIUP3RBB5CIFT5LULTJZSEXVUMQDNK3SDU5D2OYZL2Q, seg_pos={0 0}, upload=465736]: Uploading piece... Dec 27 18:32:25.875075614 (531h3m25.746158659s) [noise=true , address=77.90.31.19:4844 , piece_id=IQ6DTEAUK3ZMVVTVEXB65YXIA4R3WLWDBNCUZRZHFHRJ4K3LJM7A, seg_pos={0 0}, upload=465736]: Uploading piece... Dec 27 18:32:25.875509329 (531h3m25.746592375s) [noise=true , address=145.79.227.39:4013 , piece_id=Y2FSILI3NNR7Q5XXGDDLHHTKE2IRFU6O5AMLU547RZPXKWYQRWAA, seg_pos={0 0}, upload=465736]: Uploading piece... Dec 27 18:32:25.875941221 (531h3m25.747024267s) [noise=true , address=145.79.227.25:4618 , piece_id=ZCPAXIEOBJ3BJ2MSKK2EXBCCG523JMCFFVTLKWQ55UEBRMJZVVHA, seg_pos={0 0}, upload=465736]: Uploading piece... Dec 27 18:32:25.876358175 (531h3m25.747441221s) [noise=true , address=145.79.227.2:4114 , piece_id=KCTKDAQCZTBI3GRWEUBHK4GTLQQLCL3FRFNTKTYPHIVYBTO5I3AQ, seg_pos={0 0}, upload=465736]: Uploading piece... Dec 27 18:32:25.876859811 (531h3m25.747942856s) [noise=true , address=45.91.116.14:4189 , piece_id=WALZZPSLAPU6ZDNYRPN4FKWFND73GYWSUIXBYFZ3LA7GC2OZDCQQ, seg_pos={0 0}, upload=465736]: Uploading piece... Dec 27 18:32:25.878331714 (531h3m25.749414759s) [noise=true , address=145.79.227.26:4655 , piece_id=4YBF3DJ4BIH3GHFFG3YI4L4FYTVLDQEXTT5U6XEGHMG4S46BDNCQ, seg_pos={0 0}, upload=465736]: Uploading piece... Dec 27 18:32:25.878408840 (531h3m25.749491875s) [noise=true , address=145.79.227.47:4265 , piece_id=RCA6L5QL7ZBKHRFY7C6KEIBU6Y4EMIO2GEFDYF43BKJQ356JYAGQ, seg_pos={0 0}, upload=465736]: Uploading piece... Dec 27 18:32:25.880110090 (531h3m25.751193125s) [noise=true , address=145.79.227.38:4979 , piece_id=NDHM36NVO7M2SXJ4SVNYHWOBAO2DF2PBQDQRNU4N2ROMAGOOV4MQ, seg_pos={0 0}, upload=465736]: Uploading piece... Dec 27 18:32:25.880593010 (531h3m25.751676055s) [noise=true , address=145.79.227.15:4873 , piece_id=ADWK2XVAE7WOEZSD2FBLN4I5HEQDOMUNADKIG7BQBBBVVDIBADXQ, seg_pos={0 0}, upload=465736]: Uploading piece... Dec 27 18:32:25.881686892 (531h3m25.752769937s) [noise=true , address=45.91.116.12:4228 , piece_id=DSI5RZ5EC3GZ3VWSHPOYLXTP355TDAXJUPFTU6U77N3AVR63DO7Q, seg_pos={0 0}, upload=465736]: Uploading piece... Dec 27 18:32:25.883346833 (531h3m25.754429878s) [noise=true , address=77.90.31.14:4663 , piece_id=QQBSXSDLRBULRGZ5ZDJJ4N3PQVHSB5IUNEXOVMBBPTA2F5DGH3OQ, seg_pos={0 0}, upload=465736]: Uploading piece... Dec 27 18:32:25.883745723 (531h3m25.754828768s) [noise=true , address=145.79.227.28:4725 , piece_id=RZOCXC2TOJK75HJUKBYG6ZWJF5VKLAIZF6KFISEIDQIA6PEBCC5A, seg_pos={0 0}, upload=465736]: Uploading piece... Dec 27 18:32:25.886654291 (531h3m25.757737336s) [noise=true , address=77.90.31.21:4878 , piece_id=B6J6LYWK6W3DUWJJ7AWXZXVD3CAVK2PLZRV7LPGF3EDQCPLIQ6UA, seg_pos={0 0}, upload=465736]: Uploading piece... Dec 27 18:32:27.491958225 (531h3m27.36304126s) [upload=465736]: Next returned nil segment Dec 27 18:32:27.491973453 (531h3m27.363056488s) [upload=465736]: Waiting for error group managing segments... Dec 27 18:32:27.883820980 (531h3m27.754904025s) [noise=true , address=145.79.227.25:4618 , piece_id=ZCPAXIEOBJ3BJ2MSKK2EXBCCG523JMCFFVTLKWQ55UEBRMJZVVHA, seg_pos={0 0}, upload=465736]: Done uploading piece. err: <nil> Dec 27 18:32:27.884654017 (531h3m27.755737052s) [noise=true , address=145.79.227.38:4979 , piece_id=NDHM36NVO7M2SXJ4SVNYHWOBAO2DF2PBQDQRNU4N2ROMAGOOV4MQ, seg_pos={0 0}, upload=465736]: Done uploading piece. err: <nil> Dec 27 18:32:27.892675351 (531h3m27.763758386s) [noise=true , address=145.79.227.39:4013 , piece_id=Y2FSILI3NNR7Q5XXGDDLHHTKE2IRFU6O5AMLU547RZPXKWYQRWAA, seg_pos={0 0}, upload=465736]: Done uploading piece. err: <nil> Dec 27 18:32:27.896075837 (531h3m27.767158872s) [noise=true , address=145.79.227.47:4265 , piece_id=RCA6L5QL7ZBKHRFY7C6KEIBU6Y4EMIO2GEFDYF43BKJQ356JYAGQ, seg_pos={0 0}, upload=465736]: Done uploading piece. err: <nil> Dec 27 18:32:27.897659301 (531h3m27.768742336s) [noise=true , address=145.79.227.15:4873 , piece_id=ADWK2XVAE7WOEZSD2FBLN4I5HEQDOMUNADKIG7BQBBBVVDIBADXQ, seg_pos={0 0}, upload=465736]: Done uploading piece. err: <nil> Dec 27 18:32:27.898201484 (531h3m27.769284519s) [noise=true , address=145.79.227.22:4510 , piece_id=TYDRPSG2QPLKSDETT6OUV4BXC4DYQSIR5Y2ZIWMFZDFMTGXTLRIA, seg_pos={0 0}, upload=465736]: Done uploading piece. err: <nil> Dec 27 18:32:27.904575391 (531h3m27.775658426s) [noise=true , address=145.79.227.28:4725 , piece_id=RZOCXC2TOJK75HJUKBYG6ZWJF5VKLAIZF6KFISEIDQIA6PEBCC5A, seg_pos={0 0}, upload=465736]: Done uploading piece. err: <nil> Dec 27 18:32:27.912159122 (531h3m27.783242167s) [noise=true , address=145.79.227.26:4655 , piece_id=4YBF3DJ4BIH3GHFFG3YI4L4FYTVLDQEXTT5U6XEGHMG4S46BDNCQ, seg_pos={0 0}, upload=465736]: Done uploading piece. err: <nil> Dec 27 18:32:28.081094634 (531h3m27.952177669s) [noise=true , address=77.90.31.14:4663 , piece_id=QQBSXSDLRBULRGZ5ZDJJ4N3PQVHSB5IUNEXOVMBBPTA2F5DGH3OQ, seg_pos={0 0}, upload=465736]: Done uploading piece. err: <nil> Dec 27 18:32:28.148951857 (531h3m28.020034902s) [noise=true , address=45.91.116.14:4189 , piece_id=WALZZPSLAPU6ZDNYRPN4FKWFND73GYWSUIXBYFZ3LA7GC2OZDCQQ, seg_pos={0 0}, upload=465736]: Done uploading piece. err: <nil> Dec 27 18:32:28.152422566 (531h3m28.023505611s) [noise=true , address=45.91.116.12:4228 , piece_id=DSI5RZ5EC3GZ3VWSHPOYLXTP355TDAXJUPFTU6U77N3AVR63DO7Q, seg_pos={0 0}, upload=465736]: Done uploading piece. err: <nil> Dec 27 18:32:28.258556914 (531h3m28.129639949s) [noise=true , address=45.91.116.9:4264 , piece_id=GQT4KT6DXZIUP3RBB5CIFT5LULTJZSEXVUMQDNK3SDU5D2OYZL2Q, seg_pos={0 0}, upload=465736]: Done uploading piece. err: <nil> Dec 27 18:32:28.785386931 (531h3m28.656469976s) [noise=true , address=77.90.31.19:4844 , piece_id=IQ6DTEAUK3ZMVVTVEXB65YXIA4R3WLWDBNCUZRZHFHRJ4K3LJM7A, seg_pos={0 0}, upload=465736]: Done uploading piece. err: <nil> Dec 27 18:32:28.785411759 (531h3m28.656494804s) [seg_pos={0 0}, upload=465736]: Segment reached optimal threshold of 13 pieces. Dec 27 18:32:28.785707543 (531h3m28.656790588s) [noise=true , address=145.79.227.2:4114 , piece_id=KCTKDAQCZTBI3GRWEUBHK4GTLQQLCL3FRFNTKTYPHIVYBTO5I3AQ, seg_pos={0 0}, upload=465736]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN): protocol: write tcp 188.166.225.170:34096->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:34096->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34096->145.79.227.2:4114: use of closed network connection Dec 27 18:32:28.785716920 (531h3m28.656799955s) [noise=true , address=145.79.227.2:4114 , piece_id=KCTKDAQCZTBI3GRWEUBHK4GTLQQLCL3FRFNTKTYPHIVYBTO5I3AQ, seg_pos={0 0}, upload=465736]: Long tail cancellation due to optimal threshold Dec 27 18:32:28.786070563 (531h3m28.657153609s) [noise=true , address=77.90.31.21:4878 , piece_id=B6J6LYWK6W3DUWJJ7AWXZXVD3CAVK2PLZRV7LPGF3EDQCPLIQ6UA, seg_pos={0 0}, upload=465736]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 27 18:32:28.786079079 (531h3m28.657162115s) [noise=true , address=77.90.31.21:4878 , piece_id=B6J6LYWK6W3DUWJJ7AWXZXVD3CAVK2PLZRV7LPGF3EDQCPLIQ6UA, seg_pos={0 0}, upload=465736]: Long tail cancellation due to optimal threshold Dec 27 18:32:28.786208105 (531h3m28.65729114s) [noise=true , address=145.79.227.51:4409 , piece_id=BA5JBEPAEHZJUAVLZVX66JZVKHVDSTMQU3767BTH732YI57OHYRQ, seg_pos={0 0}, upload=465736]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UoCGA4FjE4yADPXL1oifSPJssPgSGh81REZHwGv2eu1Y1kEaj): protocol: write tcp 188.166.225.170:51222->145.79.227.51:4409: use of closed network connection; write tcp 188.166.225.170:51222->145.79.227.51:4409: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51222->145.79.227.51:4409: use of closed network connection Dec 27 18:32:28.786216872 (531h3m28.657299917s) [noise=true , address=145.79.227.51:4409 , piece_id=BA5JBEPAEHZJUAVLZVX66JZVKHVDSTMQU3767BTH732YI57OHYRQ, seg_pos={0 0}, upload=465736]: Long tail cancellation due to optimal threshold Dec 27 18:32:28.786237081 (531h3m28.657320126s) [seg_pos={0 0}, upload=465736]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 18:32:28.786323595 (531h3m28.65740664s) [upload=465736]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 18:32:28.786329556 (531h3m28.657412601s) [upload=465736]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 18:32:29.083652482 (531h3m28.954735517s) [upload=465736]: Done uploading segments... Dec 27 18:32:29.083676638 (531h3m28.954759683s) [upload=465736]: Upload finished. err: <nil> Duration: 14.35688141s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [..XX....X...X.X........X.......X..X.XXXXXXX...X.........XXXX] duration: 30.003396882s trace: https://jaeger.dev.storj.io/trace/77c06c284613bcdd FAIL (2025-12-27 18:32:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: GQT4KT6DXZIUP3RBB5CIFT5LULTJZSEXVUMQDNK3SDU5D2OYZL2Q) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: QQBSXSDLRBULRGZ5ZDJJ4N3PQVHSB5IUNEXOVMBBPTA2F5DGH3OQ) context deadline exceeded ======LOGS====== .498693041s) [upload=465736]: Uploading segments... Dec 27 18:32:25.627698233 (531h3m25.498781268s) [upload=465736]: Got next segment. Inline: false Dec 27 18:32:25.627816168 (531h3m25.498899213s) [upload=465736]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 18:32:25.627823151 (531h3m25.498906187s) [upload=465736]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 18:32:25.874268727 (531h3m25.745351772s) [seg_pos={0 0}, upload=465736]: Begin upload segment... Dec 27 18:32:25.874324954 (531h3m25.745407999s) [seg_pos={0 0}, upload=465736]: Done begin upload segment. Dec 27 18:32:25.874351565 (531h3m25.7454346s) [noise=true , address=145.79.227.51:4409 , piece_id=BA5JBEPAEHZJUAVLZVX66JZVKHVDSTMQU3767BTH732YI57OHYRQ, seg_pos={0 0}, upload=465736]: Uploading piece... Dec 27 18:32:25.874375831 (531h3m25.745458876s) [noise=true , address=145.79.227.22:4510 , piece_id=TYDRPSG2QPLKSDETT6OUV4BXC4DYQSIR5Y2ZIWMFZDFMTGXTLRIA, seg_pos={0 0}, upload=465736]: Uploading piece... Dec 27 18:32:25.874436847 (531h3m25.745519882s) [noise=true , address=45.91.116.9:4264 , piece_id=GQT4KT6DXZIUP3RBB5CIFT5LULTJZSEXVUMQDNK3SDU5D2OYZL2Q, seg_pos={0 0}, upload=465736]: Uploading piece... Dec 27 18:32:25.875075614 (531h3m25.746158659s) [noise=true , address=77.90.31.19:4844 , piece_id=IQ6DTEAUK3ZMVVTVEXB65YXIA4R3WLWDBNCUZRZHFHRJ4K3LJM7A, seg_pos={0 0}, upload=465736]: Uploading piece... Dec 27 18:32:25.875509329 (531h3m25.746592375s) [noise=true , address=145.79.227.39:4013 , piece_id=Y2FSILI3NNR7Q5XXGDDLHHTKE2IRFU6O5AMLU547RZPXKWYQRWAA, seg_pos={0 0}, upload=465736]: Uploading piece... Dec 27 18:32:25.875941221 (531h3m25.747024267s) [noise=true , address=145.79.227.25:4618 , piece_id=ZCPAXIEOBJ3BJ2MSKK2EXBCCG523JMCFFVTLKWQ55UEBRMJZVVHA, seg_pos={0 0}, upload=465736]: Uploading piece... Dec 27 18:32:25.876358175 (531h3m25.747441221s) [noise=true , address=145.79.227.2:4114 , piece_id=KCTKDAQCZTBI3GRWEUBHK4GTLQQLCL3FRFNTKTYPHIVYBTO5I3AQ, seg_pos={0 0}, upload=465736]: Uploading piece... Dec 27 18:32:25.876859811 (531h3m25.747942856s) [noise=true , address=45.91.116.14:4189 , piece_id=WALZZPSLAPU6ZDNYRPN4FKWFND73GYWSUIXBYFZ3LA7GC2OZDCQQ, seg_pos={0 0}, upload=465736]: Uploading piece... Dec 27 18:32:25.878331714 (531h3m25.749414759s) [noise=true , address=145.79.227.26:4655 , piece_id=4YBF3DJ4BIH3GHFFG3YI4L4FYTVLDQEXTT5U6XEGHMG4S46BDNCQ, seg_pos={0 0}, upload=465736]: Uploading piece... Dec 27 18:32:25.878408840 (531h3m25.749491875s) [noise=true , address=145.79.227.47:4265 , piece_id=RCA6L5QL7ZBKHRFY7C6KEIBU6Y4EMIO2GEFDYF43BKJQ356JYAGQ, seg_pos={0 0}, upload=465736]: Uploading piece... Dec 27 18:32:25.880110090 (531h3m25.751193125s) [noise=true , address=145.79.227.38:4979 , piece_id=NDHM36NVO7M2SXJ4SVNYHWOBAO2DF2PBQDQRNU4N2ROMAGOOV4MQ, seg_pos={0 0}, upload=465736]: Uploading piece... Dec 27 18:32:25.880593010 (531h3m25.751676055s) [noise=true , address=145.79.227.15:4873 , piece_id=ADWK2XVAE7WOEZSD2FBLN4I5HEQDOMUNADKIG7BQBBBVVDIBADXQ, seg_pos={0 0}, upload=465736]: Uploading piece... Dec 27 18:32:25.881686892 (531h3m25.752769937s) [noise=true , address=45.91.116.12:4228 , piece_id=DSI5RZ5EC3GZ3VWSHPOYLXTP355TDAXJUPFTU6U77N3AVR63DO7Q, seg_pos={0 0}, upload=465736]: Uploading piece... Dec 27 18:32:25.883346833 (531h3m25.754429878s) [noise=true , address=77.90.31.14:4663 , piece_id=QQBSXSDLRBULRGZ5ZDJJ4N3PQVHSB5IUNEXOVMBBPTA2F5DGH3OQ, seg_pos={0 0}, upload=465736]: Uploading piece... Dec 27 18:32:25.883745723 (531h3m25.754828768s) [noise=true , address=145.79.227.28:4725 , piece_id=RZOCXC2TOJK75HJUKBYG6ZWJF5VKLAIZF6KFISEIDQIA6PEBCC5A, seg_pos={0 0}, upload=465736]: Uploading piece... Dec 27 18:32:25.886654291 (531h3m25.757737336s) [noise=true , address=77.90.31.21:4878 , piece_id=B6J6LYWK6W3DUWJJ7AWXZXVD3CAVK2PLZRV7LPGF3EDQCPLIQ6UA, seg_pos={0 0}, upload=465736]: Uploading piece... Dec 27 18:32:27.491958225 (531h3m27.36304126s) [upload=465736]: Next returned nil segment Dec 27 18:32:27.491973453 (531h3m27.363056488s) [upload=465736]: Waiting for error group managing segments... Dec 27 18:32:27.883820980 (531h3m27.754904025s) [noise=true , address=145.79.227.25:4618 , piece_id=ZCPAXIEOBJ3BJ2MSKK2EXBCCG523JMCFFVTLKWQ55UEBRMJZVVHA, seg_pos={0 0}, upload=465736]: Done uploading piece. err: <nil> Dec 27 18:32:27.884654017 (531h3m27.755737052s) [noise=true , address=145.79.227.38:4979 , piece_id=NDHM36NVO7M2SXJ4SVNYHWOBAO2DF2PBQDQRNU4N2ROMAGOOV4MQ, seg_pos={0 0}, upload=465736]: Done uploading piece. err: <nil> Dec 27 18:32:27.892675351 (531h3m27.763758386s) [noise=true , address=145.79.227.39:4013 , piece_id=Y2FSILI3NNR7Q5XXGDDLHHTKE2IRFU6O5AMLU547RZPXKWYQRWAA, seg_pos={0 0}, upload=465736]: Done uploading piece. err: <nil> Dec 27 18:32:27.896075837 (531h3m27.767158872s) [noise=true , address=145.79.227.47:4265 , piece_id=RCA6L5QL7ZBKHRFY7C6KEIBU6Y4EMIO2GEFDYF43BKJQ356JYAGQ, seg_pos={0 0}, upload=465736]: Done uploading piece. err: <nil> Dec 27 18:32:27.897659301 (531h3m27.768742336s) [noise=true , address=145.79.227.15:4873 , piece_id=ADWK2XVAE7WOEZSD2FBLN4I5HEQDOMUNADKIG7BQBBBVVDIBADXQ, seg_pos={0 0}, upload=465736]: Done uploading piece. err: <nil> Dec 27 18:32:27.898201484 (531h3m27.769284519s) [noise=true , address=145.79.227.22:4510 , piece_id=TYDRPSG2QPLKSDETT6OUV4BXC4DYQSIR5Y2ZIWMFZDFMTGXTLRIA, seg_pos={0 0}, upload=465736]: Done uploading piece. err: <nil> Dec 27 18:32:27.904575391 (531h3m27.775658426s) [noise=true , address=145.79.227.28:4725 , piece_id=RZOCXC2TOJK75HJUKBYG6ZWJF5VKLAIZF6KFISEIDQIA6PEBCC5A, seg_pos={0 0}, upload=465736]: Done uploading piece. err: <nil> Dec 27 18:32:27.912159122 (531h3m27.783242167s) [noise=true , address=145.79.227.26:4655 , piece_id=4YBF3DJ4BIH3GHFFG3YI4L4FYTVLDQEXTT5U6XEGHMG4S46BDNCQ, seg_pos={0 0}, upload=465736]: Done uploading piece. err: <nil> Dec 27 18:32:28.081094634 (531h3m27.952177669s) [noise=true , address=77.90.31.14:4663 , piece_id=QQBSXSDLRBULRGZ5ZDJJ4N3PQVHSB5IUNEXOVMBBPTA2F5DGH3OQ, seg_pos={0 0}, upload=465736]: Done uploading piece. err: <nil> Dec 27 18:32:28.148951857 (531h3m28.020034902s) [noise=true , address=45.91.116.14:4189 , piece_id=WALZZPSLAPU6ZDNYRPN4FKWFND73GYWSUIXBYFZ3LA7GC2OZDCQQ, seg_pos={0 0}, upload=465736]: Done uploading piece. err: <nil> Dec 27 18:32:28.152422566 (531h3m28.023505611s) [noise=true , address=45.91.116.12:4228 , piece_id=DSI5RZ5EC3GZ3VWSHPOYLXTP355TDAXJUPFTU6U77N3AVR63DO7Q, seg_pos={0 0}, upload=465736]: Done uploading piece. err: <nil> Dec 27 18:32:28.258556914 (531h3m28.129639949s) [noise=true , address=45.91.116.9:4264 , piece_id=GQT4KT6DXZIUP3RBB5CIFT5LULTJZSEXVUMQDNK3SDU5D2OYZL2Q, seg_pos={0 0}, upload=465736]: Done uploading piece. err: <nil> Dec 27 18:32:28.785386931 (531h3m28.656469976s) [noise=true , address=77.90.31.19:4844 , piece_id=IQ6DTEAUK3ZMVVTVEXB65YXIA4R3WLWDBNCUZRZHFHRJ4K3LJM7A, seg_pos={0 0}, upload=465736]: Done uploading piece. err: <nil> Dec 27 18:32:28.785411759 (531h3m28.656494804s) [seg_pos={0 0}, upload=465736]: Segment reached optimal threshold of 13 pieces. Dec 27 18:32:28.785707543 (531h3m28.656790588s) [noise=true , address=145.79.227.2:4114 , piece_id=KCTKDAQCZTBI3GRWEUBHK4GTLQQLCL3FRFNTKTYPHIVYBTO5I3AQ, seg_pos={0 0}, upload=465736]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN): protocol: write tcp 188.166.225.170:34096->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:34096->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34096->145.79.227.2:4114: use of closed network connection Dec 27 18:32:28.785716920 (531h3m28.656799955s) [noise=true , address=145.79.227.2:4114 , piece_id=KCTKDAQCZTBI3GRWEUBHK4GTLQQLCL3FRFNTKTYPHIVYBTO5I3AQ, seg_pos={0 0}, upload=465736]: Long tail cancellation due to optimal threshold Dec 27 18:32:28.786070563 (531h3m28.657153609s) [noise=true , address=77.90.31.21:4878 , piece_id=B6J6LYWK6W3DUWJJ7AWXZXVD3CAVK2PLZRV7LPGF3EDQCPLIQ6UA, seg_pos={0 0}, upload=465736]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 27 18:32:28.786079079 (531h3m28.657162115s) [noise=true , address=77.90.31.21:4878 , piece_id=B6J6LYWK6W3DUWJJ7AWXZXVD3CAVK2PLZRV7LPGF3EDQCPLIQ6UA, seg_pos={0 0}, upload=465736]: Long tail cancellation due to optimal threshold Dec 27 18:32:28.786208105 (531h3m28.65729114s) [noise=true , address=145.79.227.51:4409 , piece_id=BA5JBEPAEHZJUAVLZVX66JZVKHVDSTMQU3767BTH732YI57OHYRQ, seg_pos={0 0}, upload=465736]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UoCGA4FjE4yADPXL1oifSPJssPgSGh81REZHwGv2eu1Y1kEaj): protocol: write tcp 188.166.225.170:51222->145.79.227.51:4409: use of closed network connection; write tcp 188.166.225.170:51222->145.79.227.51:4409: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51222->145.79.227.51:4409: use of closed network connection Dec 27 18:32:28.786216872 (531h3m28.657299917s) [noise=true , address=145.79.227.51:4409 , piece_id=BA5JBEPAEHZJUAVLZVX66JZVKHVDSTMQU3767BTH732YI57OHYRQ, seg_pos={0 0}, upload=465736]: Long tail cancellation due to optimal threshold Dec 27 18:32:28.786237081 (531h3m28.657320126s) [seg_pos={0 0}, upload=465736]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 18:32:28.786323595 (531h3m28.65740664s) [upload=465736]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 18:32:28.786329556 (531h3m28.657412601s) [upload=465736]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 18:32:29.083652482 (531h3m28.954735517s) [upload=465736]: Done uploading segments... Dec 27 18:32:29.083676638 (531h3m28.954759683s) [upload=465736]: Upload finished. err: <nil> Duration: 30.003421268s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [X..XX....X...X.X........X.......X..X.XXXXXXX...X.........XXX] duration: 30.00316466s trace: https://jaeger.dev.storj.io/trace/18c9fa9df826a6a7 FAIL (2025-12-27 18:31:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: AF6M2V6UF7PXJTE7JKEAG4IYDNN4ADRENKH6UED6KEJSW5U7A7QQ) context deadline exceeded; piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: SCELXJGKMY67R2KJC3V6IQ4JD73V6Q5DQNVXMHWQKL2SKKEMVKUQ) context deadline exceeded ======LOGS====== Dec 27 18:31:12.241162786 (531h2m12.112245821s) [upload=465717]: Starting upload Dec 27 18:31:12.241220416 (531h2m12.112303461s) [upload=465717]: Done starting upload Dec 27 18:31:12.241302723 (531h2m12.112385758s) [upload=465717]: Uploading segments... Dec 27 18:31:12.241441787 (531h2m12.112524822s) [upload=465717]: Got next segment. Inline: false Dec 27 18:31:12.241486282 (531h2m12.112569327s) [upload=465717]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 18:31:12.241491301 (531h2m12.112574346s) [upload=465717]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 18:31:12.486978342 (531h2m12.358061387s) [seg_pos={0 0}, upload=465717]: Begin upload segment... Dec 27 18:31:12.487054757 (531h2m12.358137802s) [seg_pos={0 0}, upload=465717]: Done begin upload segment. Dec 27 18:31:12.487129960 (531h2m12.358213005s) [noise=true , address=145.79.227.4:4185 , piece_id=5PS232COVE6CDLHQI7YVHBQL6W43ZU7TFJGDF737XAAVEYUMW47A, seg_pos={0 0}, upload=465717]: Uploading piece... Dec 27 18:31:12.487137825 (531h2m12.35822086s) [noise=true , address=145.79.227.46:4603 , piece_id=NF7KPI3KJDK5JOI3WSZORE7T5PECXGARCPY7TKA6K2SOFDFEWVVA, seg_pos={0 0}, upload=465717]: Uploading piece... Dec 27 18:31:12.487957516 (531h2m12.359040561s) [noise=true , address=77.90.31.14:4661 , piece_id=W3VAJIGEYO75LYQW5F6DLCA2UWUN3X53H77VM62D6BJV73TWW6SQ, seg_pos={0 0}, upload=465717]: Uploading piece... Dec 27 18:31:12.488370352 (531h2m12.359453398s) [noise=true , address=77.90.31.10:4517 , piece_id=MSBXRZJWYO5P7EIW76IGPDP6BHGYHTQKI36IJF3AZX6WVO7UDS2A, seg_pos={0 0}, upload=465717]: Uploading piece... Dec 27 18:31:12.488746779 (531h2m12.359829825s) [noise=true , address=145.79.227.11:4732 , piece_id=2XM7I2G6ZFXF67KLXZQ5XDNUOWFWEGSMAJVVZML6DRDYUAHNIR7A, seg_pos={0 0}, upload=465717]: Uploading piece... Dec 27 18:31:12.489134337 (531h2m12.360217402s) [noise=true , address=145.79.227.52:4447 , piece_id=5W4GKTVPQRH55GLQ4BBPOUFHBIC632PFY5GNQ2FO4VM6T6XYMFDQ, seg_pos={0 0}, upload=465717]: Uploading piece... Dec 27 18:31:12.489997511 (531h2m12.361080556s) [noise=true , address=45.91.116.6:4976 , piece_id=SCELXJGKMY67R2KJC3V6IQ4JD73V6Q5DQNVXMHWQKL2SKKEMVKUQ, seg_pos={0 0}, upload=465717]: Uploading piece... Dec 27 18:31:12.490015996 (531h2m12.361099041s) [noise=true , address=145.79.227.17:4970 , piece_id=NMEN6FN3A42SCMLTNL2CUPBMJ6NYRZQHVCFQ2U46QSNCLQTMKUPQ, seg_pos={0 0}, upload=465717]: Uploading piece... Dec 27 18:31:12.490582604 (531h2m12.36166565s) [noise=true , address=145.79.227.45:4230 , piece_id=RLEBMRZB6DNUR6H4DPTPDN5KZWUFMDSXXUPD5TBZIQVUU6JJZG6A, seg_pos={0 0}, upload=465717]: Uploading piece... Dec 27 18:31:12.490982346 (531h2m12.362065381s) [noise=true , address=145.79.227.51:4409 , piece_id=X34PJCL5LG4W4FRT6ZOCGAJCPTQBTTTDVOK7A4N7F4SAXPRPBY3A, seg_pos={0 0}, upload=465717]: Uploading piece... Dec 27 18:31:12.490987897 (531h2m12.362070942s) [noise=true , address=145.79.227.43:4158 , piece_id=NJVDBXYTMMXEV4M3R5VJQ7FCOP2GSSNVDRZJW4IQNXC7Q4DQLGTA, seg_pos={0 0}, upload=465717]: Uploading piece... Dec 27 18:31:12.491383089 (531h2m12.362466124s) [noise=true , address=145.79.227.33:4869 , piece_id=5JHLQAQYWHRPID7HTMCJMTZS32Q5DWFQYVLZ5HSNXSMRIKVQASLA, seg_pos={0 0}, upload=465717]: Uploading piece... Dec 27 18:31:12.491443985 (531h2m12.36252703s) [noise=true , address=145.79.227.44:4193 , piece_id=7CLLX4XD7V4CDVYGV3QQEZD2THJ6MPF2NWOUHOV7HIJT523NMZZQ, seg_pos={0 0}, upload=465717]: Uploading piece... Dec 27 18:31:12.491791917 (531h2m12.362874973s) [noise=true , address=77.90.31.16:4736 , piece_id=AF6M2V6UF7PXJTE7JKEAG4IYDNN4ADRENKH6UED6KEJSW5U7A7QQ, seg_pos={0 0}, upload=465717]: Uploading piece... Dec 27 18:31:12.492192641 (531h2m12.363275776s) [noise=true , address=145.79.227.28:4725 , piece_id=MYRFVZYTQGKEFG4EOYYD2C7Q3VMMZ24PR2RA5QCNA6ZVIZXKJMUQ, seg_pos={0 0}, upload=465717]: Uploading piece... Dec 27 18:31:12.494145700 (531h2m12.365228745s) [noise=true , address=145.79.227.24:4581 , piece_id=MDEVVHI2CCFPHZ6HBFFDZK3W5FGLTICPMIJJHLHEXAQZPP5NUPHQ, seg_pos={0 0}, upload=465717]: Uploading piece... Dec 27 18:31:14.201427335 (531h2m14.07251038s) [upload=465717]: Next returned nil segment Dec 27 18:31:14.201491427 (531h2m14.072574492s) [upload=465717]: Waiting for error group managing segments... Dec 27 18:31:14.490956901 (531h2m14.362039946s) [noise=true , address=145.79.227.33:4869 , piece_id=5JHLQAQYWHRPID7HTMCJMTZS32Q5DWFQYVLZ5HSNXSMRIKVQASLA, seg_pos={0 0}, upload=465717]: Done uploading piece. err: <nil> Dec 27 18:31:14.502855628 (531h2m14.373938673s) [noise=true , address=145.79.227.24:4581 , piece_id=MDEVVHI2CCFPHZ6HBFFDZK3W5FGLTICPMIJJHLHEXAQZPP5NUPHQ, seg_pos={0 0}, upload=465717]: Done uploading piece. err: <nil> Dec 27 18:31:14.503101676 (531h2m14.374184722s) [noise=true , address=145.79.227.51:4409 , piece_id=X34PJCL5LG4W4FRT6ZOCGAJCPTQBTTTDVOK7A4N7F4SAXPRPBY3A, seg_pos={0 0}, upload=465717]: Done uploading piece. err: <nil> Dec 27 18:31:14.504753843 (531h2m14.375836888s) [noise=true , address=145.79.227.11:4732 , piece_id=2XM7I2G6ZFXF67KLXZQ5XDNUOWFWEGSMAJVVZML6DRDYUAHNIR7A, seg_pos={0 0}, upload=465717]: Done uploading piece. err: <nil> Dec 27 18:31:14.509392886 (531h2m14.380475921s) [noise=true , address=145.79.227.46:4603 , piece_id=NF7KPI3KJDK5JOI3WSZORE7T5PECXGARCPY7TKA6K2SOFDFEWVVA, seg_pos={0 0}, upload=465717]: Done uploading piece. err: <nil> Dec 27 18:31:14.517549268 (531h2m14.388632313s) [noise=true , address=145.79.227.17:4970 , piece_id=NMEN6FN3A42SCMLTNL2CUPBMJ6NYRZQHVCFQ2U46QSNCLQTMKUPQ, seg_pos={0 0}, upload=465717]: Done uploading piece. err: <nil> Dec 27 18:31:14.517866021 (531h2m14.388949066s) [noise=true , address=145.79.227.28:4725 , piece_id=MYRFVZYTQGKEFG4EOYYD2C7Q3VMMZ24PR2RA5QCNA6ZVIZXKJMUQ, seg_pos={0 0}, upload=465717]: Done uploading piece. err: <nil> Dec 27 18:31:14.523928384 (531h2m14.395011429s) [noise=true , address=145.79.227.45:4230 , piece_id=RLEBMRZB6DNUR6H4DPTPDN5KZWUFMDSXXUPD5TBZIQVUU6JJZG6A, seg_pos={0 0}, upload=465717]: Done uploading piece. err: <nil> Dec 27 18:31:14.532490268 (531h2m14.403573303s) [noise=true , address=145.79.227.4:4185 , piece_id=5PS232COVE6CDLHQI7YVHBQL6W43ZU7TFJGDF737XAAVEYUMW47A, seg_pos={0 0}, upload=465717]: Done uploading piece. err: <nil> Dec 27 18:31:14.548647844 (531h2m14.419730899s) [noise=true , address=145.79.227.52:4447 , piece_id=5W4GKTVPQRH55GLQ4BBPOUFHBIC632PFY5GNQ2FO4VM6T6XYMFDQ, seg_pos={0 0}, upload=465717]: Done uploading piece. err: <nil> Dec 27 18:31:14.654579307 (531h2m14.525662352s) [noise=true , address=77.90.31.16:4736 , piece_id=AF6M2V6UF7PXJTE7JKEAG4IYDNN4ADRENKH6UED6KEJSW5U7A7QQ, seg_pos={0 0}, upload=465717]: Done uploading piece. err: <nil> Dec 27 18:31:14.677271918 (531h2m14.548354953s) [noise=true , address=77.90.31.10:4517 , piece_id=MSBXRZJWYO5P7EIW76IGPDP6BHGYHTQKI36IJF3AZX6WVO7UDS2A, seg_pos={0 0}, upload=465717]: Done uploading piece. err: <nil> Dec 27 18:31:15.128556273 (531h2m14.999639308s) [noise=true , address=45.91.116.6:4976 , piece_id=SCELXJGKMY67R2KJC3V6IQ4JD73V6Q5DQNVXMHWQKL2SKKEMVKUQ, seg_pos={0 0}, upload=465717]: Done uploading piece. err: <nil> Dec 27 18:31:15.128593284 (531h2m14.999676329s) [seg_pos={0 0}, upload=465717]: Segment reached optimal threshold of 13 pieces. Dec 27 18:31:15.128833821 (531h2m14.999916856s) [noise=true , address=145.79.227.44:4193 , piece_id=7CLLX4XD7V4CDVYGV3QQEZD2THJ6MPF2NWOUHOV7HIJT523NMZZQ, seg_pos={0 0}, upload=465717]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 27 18:31:15.128845123 (531h2m14.999928158s) [noise=true , address=145.79.227.44:4193 , piece_id=7CLLX4XD7V4CDVYGV3QQEZD2THJ6MPF2NWOUHOV7HIJT523NMZZQ, seg_pos={0 0}, upload=465717]: Long tail cancellation due to optimal threshold Dec 27 18:31:15.129017692 (531h2m15.000100777s) [noise=true , address=145.79.227.43:4158 , piece_id=NJVDBXYTMMXEV4M3R5VJQ7FCOP2GSSNVDRZJW4IQNXC7Q4DQLGTA, seg_pos={0 0}, upload=465717]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF Dec 27 18:31:15.129041015 (531h2m15.00012405s) [noise=true , address=145.79.227.43:4158 , piece_id=NJVDBXYTMMXEV4M3R5VJQ7FCOP2GSSNVDRZJW4IQNXC7Q4DQLGTA, seg_pos={0 0}, upload=465717]: Long tail cancellation due to optimal threshold Dec 27 18:31:15.129115508 (531h2m15.000198543s) [noise=true , address=77.90.31.14:4661 , piece_id=W3VAJIGEYO75LYQW5F6DLCA2UWUN3X53H77VM62D6BJV73TWW6SQ, seg_pos={0 0}, upload=465717]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Dec 27 18:31:15.129126489 (531h2m15.000209524s) [noise=true , address=77.90.31.14:4661 , piece_id=W3VAJIGEYO75LYQW5F6DLCA2UWUN3X53H77VM62D6BJV73TWW6SQ, seg_pos={0 0}, upload=465717]: Long tail cancellation due to optimal threshold Dec 27 18:31:15.129167527 (531h2m15.000250572s) [seg_pos={0 0}, upload=465717]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 18:31:15.129306551 (531h2m15.000389586s) [upload=465717]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 18:31:15.129313074 (531h2m15.000396109s) [upload=465717]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 18:31:15.428428953 (531h2m15.299511998s) [upload=465717]: Done uploading segments... Dec 27 18:31:15.428453580 (531h2m15.299536625s) [upload=465717]: Upload finished. err: <nil> Duration: 30.003189467s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [XX..XX....X...X.X........X.......X..X.XXXXXXX...X.........XX] duration: 30.004224688s trace: https://jaeger.dev.storj.io/trace/5aac1552988aaa76 FAIL (2025-12-27 18:30:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: 5EUX7MSOOUW6IZHPIPYCTHOVZSCBIADTOQTQREPN7TD75VK6ROTA) context deadline exceeded; piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: QYG6XKO3CKHG6KUCTY7GQVINC2DRVWDEJE5QC33WFPKZ5UBJ5S3Q) context deadline exceeded ======LOGS====== Dec 27 18:30:00.748321107 (531h1m0.619404152s) [upload=465701]: Starting upload Dec 27 18:30:00.748401441 (531h1m0.619484476s) [upload=465701]: Done starting upload Dec 27 18:30:00.748633031 (531h1m0.619716066s) [upload=465701]: Uploading segments... Dec 27 18:30:00.748685241 (531h1m0.619768306s) [upload=465701]: Got next segment. Inline: false Dec 27 18:30:00.748721179 (531h1m0.619804225s) [upload=465701]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 18:30:00.748727201 (531h1m0.619810246s) [upload=465701]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 18:30:00.995831922 (531h1m0.866914957s) [seg_pos={0 0}, upload=465701]: Begin upload segment... Dec 27 18:30:00.995876637 (531h1m0.866959672s) [seg_pos={0 0}, upload=465701]: Done begin upload segment. Dec 27 18:30:00.995907626 (531h1m0.866990661s) [noise=true , address=145.79.227.33:4871 , piece_id=RKAN7KPKKLGXZIYSMVEPEG67CSKG653665W27WKZZJ3TJXWIZVTA, seg_pos={0 0}, upload=465701]: Uploading piece... Dec 27 18:30:00.995913848 (531h1m0.866996883s) [noise=true , address=145.79.227.10:4403 , piece_id=USDQ7IM6BIUHBOQ3DSSUB66BK2QG6QDAS4W7LMOTOU44TVT7LFEA, seg_pos={0 0}, upload=465701]: Uploading piece... Dec 27 18:30:00.996762554 (531h1m0.867845589s) [noise=true , address=45.91.116.13:4156 , piece_id=PQ5XPR6S64MBPUOO3OA5TQFXFBCD2GLMXAFV2DIKCIX4BGXTW5UQ, seg_pos={0 0}, upload=465701]: Uploading piece... Dec 27 18:30:00.997335555 (531h1m0.8684186s) [noise=true , address=145.79.227.41:4085 , piece_id=KWG2FDD6IQAJK7EBYXAAKSQIKDPV7WXDLFKC6Q5L63OLVVOYSBWQ, seg_pos={0 0}, upload=465701]: Uploading piece... Dec 27 18:30:00.997514305 (531h1m0.868597361s) [noise=true , address=145.79.227.14:4838 , piece_id=A2J73SQE374LQILPUINGDMWKBX43CDQ4GCKO6BVIHGCAKXPSACTA, seg_pos={0 0}, upload=465701]: Uploading piece... Dec 27 18:30:00.997645005 (531h1m0.86872804s) [noise=true , address=77.90.31.10:4518 , piece_id=5EUX7MSOOUW6IZHPIPYCTHOVZSCBIADTOQTQREPN7TD75VK6ROTA, seg_pos={0 0}, upload=465701]: Uploading piece... Dec 27 18:30:00.997974281 (531h1m0.869057326s) [noise=true , address=145.79.227.37:4566 , piece_id=FB2JIYCCUD7DAYMA77HHEKHJ4ETOZQEQGKSMFF5PGRZS2ZMWXOXA, seg_pos={0 0}, upload=465701]: Uploading piece... Dec 27 18:30:00.998306795 (531h1m0.86938983s) [noise=true , address=145.79.227.25:4618 , piece_id=FAS5PRM34HAV7M677ZIBT6T3CQQ6VV5WI4P6RF4GQ6PB775IC3WA, seg_pos={0 0}, upload=465701]: Uploading piece... Dec 27 18:30:00.998666569 (531h1m0.869749615s) [noise=true , address=145.79.227.9:4365 , piece_id=ZOD3S7HS6BEC3RVCHHE4CL7OIY32ESO76WBELTKJXU7J4AXJNFFQ, seg_pos={0 0}, upload=465701]: Uploading piece... Dec 27 18:30:00.998970889 (531h1m0.870053934s) [noise=true , address=45.91.116.12:4225 , piece_id=QYG6XKO3CKHG6KUCTY7GQVINC2DRVWDEJE5QC33WFPKZ5UBJ5S3Q, seg_pos={0 0}, upload=465701]: Uploading piece... Dec 27 18:30:00.999491020 (531h1m0.870574055s) [noise=true , address=45.91.116.5:4297 , piece_id=SSDKZVGCJRTMV2E52753PFMNZW5HJSTT5CWZDK7WEPCJVPAUYLFA, seg_pos={0 0}, upload=465701]: Uploading piece... Dec 27 18:30:00.999684578 (531h1m0.870767613s) [noise=true , address=145.79.227.18:4007 , piece_id=G7XMCZXLIKAIW3YPPWX27EHDV6TJOUJADQXZUDBXLH3N7CLCUKHA, seg_pos={0 0}, upload=465701]: Uploading piece... Dec 27 18:30:01.000071464 (531h1m0.871154519s) [noise=true , address=77.90.31.14:4664 , piece_id=L6EWIHSINFNBYSRSGXBSK6R4ZCKM27MKHNQAKN75JDKU4I67D3SQ, seg_pos={0 0}, upload=465701]: Uploading piece... Dec 27 18:30:01.000520760 (531h1m0.871603805s) [noise=true , address=145.79.227.39:4015 , piece_id=TXZC5HD6QQZXPEEKAAW6SUUOATHH55QUGDN65T4UMI7UZCIS2KHA, seg_pos={0 0}, upload=465701]: Uploading piece... Dec 27 18:30:01.000710632 (531h1m0.871793687s) [noise=true , address=77.90.31.21:4877 , piece_id=NXBNCEFP63NXUOHO5JSSEAO6HG4PCHRCQGMTILKQMOU4E3XBWZYQ, seg_pos={0 0}, upload=465701]: Uploading piece... Dec 27 18:30:01.001231083 (531h1m0.872314138s) [noise=true , address=145.79.227.5:4221 , piece_id=VSGIZHRBE6KQ3MFFINBMWJRCJYNKY4YUH5XMFFT2V2P6S4ZKPZ2Q, seg_pos={0 0}, upload=465701]: Uploading piece... Dec 27 18:30:02.673938103 (531h1m2.545021148s) [upload=465701]: Next returned nil segment Dec 27 18:30:02.673957680 (531h1m2.545040725s) [upload=465701]: Waiting for error group managing segments... Dec 27 18:30:03.061403025 (531h1m2.93248607s) [noise=true , address=145.79.227.37:4566 , piece_id=FB2JIYCCUD7DAYMA77HHEKHJ4ETOZQEQGKSMFF5PGRZS2ZMWXOXA, seg_pos={0 0}, upload=465701]: Done uploading piece. err: <nil> Dec 27 18:30:03.062026623 (531h1m2.933109668s) [noise=true , address=145.79.227.41:4085 , piece_id=KWG2FDD6IQAJK7EBYXAAKSQIKDPV7WXDLFKC6Q5L63OLVVOYSBWQ, seg_pos={0 0}, upload=465701]: Done uploading piece. err: <nil> Dec 27 18:30:03.076402427 (531h1m2.947485472s) [noise=true , address=145.79.227.14:4838 , piece_id=A2J73SQE374LQILPUINGDMWKBX43CDQ4GCKO6BVIHGCAKXPSACTA, seg_pos={0 0}, upload=465701]: Done uploading piece. err: <nil> Dec 27 18:30:03.082135252 (531h1m2.953218297s) [noise=true , address=145.79.227.10:4403 , piece_id=USDQ7IM6BIUHBOQ3DSSUB66BK2QG6QDAS4W7LMOTOU44TVT7LFEA, seg_pos={0 0}, upload=465701]: Done uploading piece. err: <nil> Dec 27 18:30:03.083830200 (531h1m2.954913255s) [noise=true , address=145.79.227.25:4618 , piece_id=FAS5PRM34HAV7M677ZIBT6T3CQQ6VV5WI4P6RF4GQ6PB775IC3WA, seg_pos={0 0}, upload=465701]: Done uploading piece. err: <nil> Dec 27 18:30:03.087014132 (531h1m2.958097197s) [noise=true , address=145.79.227.39:4015 , piece_id=TXZC5HD6QQZXPEEKAAW6SUUOATHH55QUGDN65T4UMI7UZCIS2KHA, seg_pos={0 0}, upload=465701]: Done uploading piece. err: <nil> Dec 27 18:30:03.088039986 (531h1m2.959123031s) [noise=true , address=145.79.227.18:4007 , piece_id=G7XMCZXLIKAIW3YPPWX27EHDV6TJOUJADQXZUDBXLH3N7CLCUKHA, seg_pos={0 0}, upload=465701]: Done uploading piece. err: <nil> Dec 27 18:30:03.101145882 (531h1m2.972228937s) [noise=true , address=145.79.227.9:4365 , piece_id=ZOD3S7HS6BEC3RVCHHE4CL7OIY32ESO76WBELTKJXU7J4AXJNFFQ, seg_pos={0 0}, upload=465701]: Done uploading piece. err: <nil> Dec 27 18:30:03.198429116 (531h1m3.069512171s) [noise=true , address=145.79.227.33:4871 , piece_id=RKAN7KPKKLGXZIYSMVEPEG67CSKG653665W27WKZZJ3TJXWIZVTA, seg_pos={0 0}, upload=465701]: Done uploading piece. err: <nil> Dec 27 18:30:03.273417959 (531h1m3.144501004s) [noise=true , address=77.90.31.10:4518 , piece_id=5EUX7MSOOUW6IZHPIPYCTHOVZSCBIADTOQTQREPN7TD75VK6ROTA, seg_pos={0 0}, upload=465701]: Done uploading piece. err: <nil> Dec 27 18:30:03.291514177 (531h1m3.162597232s) [noise=true , address=77.90.31.14:4664 , piece_id=L6EWIHSINFNBYSRSGXBSK6R4ZCKM27MKHNQAKN75JDKU4I67D3SQ, seg_pos={0 0}, upload=465701]: Done uploading piece. err: <nil> Dec 27 18:30:03.298402454 (531h1m3.169485499s) [noise=true , address=45.91.116.12:4225 , piece_id=QYG6XKO3CKHG6KUCTY7GQVINC2DRVWDEJE5QC33WFPKZ5UBJ5S3Q, seg_pos={0 0}, upload=465701]: Done uploading piece. err: <nil> Dec 27 18:30:03.359797911 (531h1m3.230880957s) [noise=true , address=77.90.31.21:4877 , piece_id=NXBNCEFP63NXUOHO5JSSEAO6HG4PCHRCQGMTILKQMOU4E3XBWZYQ, seg_pos={0 0}, upload=465701]: Done uploading piece. err: <nil> Dec 27 18:30:03.359827547 (531h1m3.230910592s) [seg_pos={0 0}, upload=465701]: Segment reached optimal threshold of 13 pieces. Dec 27 18:30:03.360083174 (531h1m3.23116623s) [noise=true , address=45.91.116.13:4156 , piece_id=PQ5XPR6S64MBPUOO3OA5TQFXFBCD2GLMXAFV2DIKCIX4BGXTW5UQ, seg_pos={0 0}, upload=465701]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 27 18:30:03.360094866 (531h1m3.231177911s) [noise=true , address=45.91.116.13:4156 , piece_id=PQ5XPR6S64MBPUOO3OA5TQFXFBCD2GLMXAFV2DIKCIX4BGXTW5UQ, seg_pos={0 0}, upload=465701]: Long tail cancellation due to optimal threshold Dec 27 18:30:03.360474840 (531h1m3.231557895s) [noise=true , address=145.79.227.5:4221 , piece_id=VSGIZHRBE6KQ3MFFINBMWJRCJYNKY4YUH5XMFFT2V2P6S4ZKPZ2Q, seg_pos={0 0}, upload=465701]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR): protocol: expected piece hash; context canceled; EOF Dec 27 18:30:03.360485209 (531h1m3.231568255s) [noise=true , address=145.79.227.5:4221 , piece_id=VSGIZHRBE6KQ3MFFINBMWJRCJYNKY4YUH5XMFFT2V2P6S4ZKPZ2Q, seg_pos={0 0}, upload=465701]: Long tail cancellation due to optimal threshold Dec 27 18:30:03.360660984 (531h1m3.231744029s) [noise=true , address=45.91.116.5:4297 , piece_id=SSDKZVGCJRTMV2E52753PFMNZW5HJSTT5CWZDK7WEPCJVPAUYLFA, seg_pos={0 0}, upload=465701]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 27 18:30:03.360671094 (531h1m3.231754129s) [noise=true , address=45.91.116.5:4297 , piece_id=SSDKZVGCJRTMV2E52753PFMNZW5HJSTT5CWZDK7WEPCJVPAUYLFA, seg_pos={0 0}, upload=465701]: Long tail cancellation due to optimal threshold Dec 27 18:30:03.360698546 (531h1m3.231781591s) [seg_pos={0 0}, upload=465701]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 18:30:03.360800019 (531h1m3.231883074s) [upload=465701]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 18:30:03.360807554 (531h1m3.231890599s) [upload=465701]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 18:30:03.644441543 (531h1m3.515524588s) [upload=465701]: Done uploading segments... Dec 27 18:30:03.644468425 (531h1m3.51555147s) [upload=465701]: Upload finished. err: <nil> Duration: 30.004252401s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [X...........................................................] duration: 3.595652169s trace: https://jaeger.dev.storj.io/trace/3bddfef0dfa96557 FAIL (2025-12-27 18:29:52 UTC) orbiter warning: context deadline exceeded Duration: 3.595671095s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [XXX..XX....X...X.X........X.......X..X.XXXXXXX...X.........X] duration: 30.002717853s trace: https://jaeger.dev.storj.io/trace/7dd759c97b4c0757 FAIL (2025-12-27 18:29:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: SVMVZ7XTIXVQLCUEOBYKPYBS3B735DJJ5WID4QQGFWWIS5MSLULQ) context deadline exceeded; piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: BTX5PTJDWGFVMJDSGFDVPGKRJG5D3XT25N2O3XX5MNQ5SBVQDE5Q) context deadline exceeded ======LOGS====== Dec 27 18:29:11.201244583 (531h0m11.072327618s) [upload=465690]: Starting upload Dec 27 18:29:11.201303445 (531h0m11.07238649s) [upload=465690]: Done starting upload Dec 27 18:29:11.201497744 (531h0m11.07258079s) [upload=465690]: Uploading segments... Dec 27 18:29:11.201548772 (531h0m11.072631827s) [upload=465690]: Got next segment. Inline: false Dec 27 18:29:11.201575683 (531h0m11.072658718s) [upload=465690]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 18:29:11.201579962 (531h0m11.072662997s) [upload=465690]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 18:29:11.446917667 (531h0m11.318000712s) [seg_pos={0 0}, upload=465690]: Begin upload segment... Dec 27 18:29:11.446986318 (531h0m11.318069363s) [seg_pos={0 0}, upload=465690]: Done begin upload segment. Dec 27 18:29:11.447045932 (531h0m11.318128987s) [noise=true , address=145.79.227.36:4519 , piece_id=GCEHLTPG7JAE7LJXNFIAQUVSOQXNKHCWM5XSYURVFQRQSWMW2H3Q, seg_pos={0 0}, upload=465690]: Uploading piece... Dec 27 18:29:11.447069076 (531h0m11.318152121s) [noise=true , address=145.79.227.11:4729 , piece_id=MBVTA3IKXV3HEXFLIF362ROULECHWBDIY2ZFALRBCMVQKS7EXCNA, seg_pos={0 0}, upload=465690]: Uploading piece... Dec 27 18:29:11.447199083 (531h0m11.318282128s) [noise=true , address=145.79.227.51:4409 , piece_id=JE7GRUEAZJ5YERU5UTTWPU7PD453WV3QO73KBRLIXM7PLZKPBYKA, seg_pos={0 0}, upload=465690]: Uploading piece... Dec 27 18:29:11.447620676 (531h0m11.318703711s) [noise=true , address=145.79.227.39:4015 , piece_id=JPZ2243MWXGOJXPG4VKAO3PIQKXBWDH3ZXJOH3JBIT64M7OTUOSQ, seg_pos={0 0}, upload=465690]: Uploading piece... Dec 27 18:29:11.448017091 (531h0m11.319100126s) [noise=true , address=145.79.227.37:4567 , piece_id=2LCXOJ7J2KGUDOKQDZ25LIQXIUMMCOEBVLXE6S6JBGMSYKZDTK2A, seg_pos={0 0}, upload=465690]: Uploading piece... Dec 27 18:29:11.448371436 (531h0m11.319454471s) [noise=true , address=145.79.227.28:4725 , piece_id=55ZRJ5KXBMUU4RXZANXSS3FBQK52QYUYQW6SVBLMQXQ7GMSCJJCA, seg_pos={0 0}, upload=465690]: Uploading piece... Dec 27 18:29:11.448714539 (531h0m11.319797574s) [noise=true , address=145.79.227.23:4546 , piece_id=QCK2MVYBKLD3FMAQDKG6GY7YRHYN75CCJNFO75F5EIEAJQQTMQAA, seg_pos={0 0}, upload=465690]: Uploading piece... Dec 27 18:29:11.448989262 (531h0m11.320072297s) [noise=true , address=145.79.227.32:4834 , piece_id=WBNN47URS7ZV6RS7ZDEAEZGSIOTF3NNGMYJ5BLEWU45MGMXO644Q, seg_pos={0 0}, upload=465690]: Uploading piece... Dec 27 18:29:11.449084233 (531h0m11.320167278s) [noise=true , address=145.79.227.34:4907 , piece_id=POFQB5NEJ6VILLEBFTRNG3IJYTNKBSIUAT5X4SGVQV5X7YIRCFUA, seg_pos={0 0}, upload=465690]: Uploading piece... Dec 27 18:29:11.449478614 (531h0m11.320561649s) [noise=true , address=77.90.31.15:4700 , piece_id=SVMVZ7XTIXVQLCUEOBYKPYBS3B735DJJ5WID4QQGFWWIS5MSLULQ, seg_pos={0 0}, upload=465690]: Uploading piece... Dec 27 18:29:11.449509503 (531h0m11.320592548s) [noise=true , address=145.79.227.50:4373 , piece_id=YWS7ALQY4WW6B342OV6FNAV65NTT2NOTLXCBW7V3EZGGO6XPFIFQ, seg_pos={0 0}, upload=465690]: Uploading piece... Dec 27 18:29:11.451116894 (531h0m11.322199929s) [noise=true , address=145.79.227.14:4838 , piece_id=OWM7XP7V4FHR7CHBJS4ENYYNPO33RTODEGRW32RXBDEBYVVFTVVQ, seg_pos={0 0}, upload=465690]: Uploading piece... Dec 27 18:29:11.451669195 (531h0m11.322752241s) [noise=true , address=77.90.31.11:4554 , piece_id=MG3ZZJPYV4VHTSEMZIQJ6I54DCEH3KK33ACKF66F3VJIUGA23YTQ, seg_pos={0 0}, upload=465690]: Uploading piece... Dec 27 18:29:11.454106296 (531h0m11.325189341s) [noise=true , address=45.91.116.14:4191 , piece_id=BTX5PTJDWGFVMJDSGFDVPGKRJG5D3XT25N2O3XX5MNQ5SBVQDE5Q, seg_pos={0 0}, upload=465690]: Uploading piece... Dec 27 18:29:11.455538714 (531h0m11.326621759s) [noise=true , address=145.79.227.49:4339 , piece_id=2WU24VSIN6V6L6DMH5GZILBETDON6FLPZ5FZGB3RQ3FIINOJEOQA, seg_pos={0 0}, upload=465690]: Uploading piece... Dec 27 18:29:11.456823580 (531h0m11.327906625s) [noise=true , address=45.91.116.9:4261 , piece_id=JAIZDO4EJV7N7P4AY4I3UOMYMI22HVD7FTAAZ42F6QXLRFW5TC5Q, seg_pos={0 0}, upload=465690]: Uploading piece... Dec 27 18:29:13.157309896 (531h0m13.028392941s) [upload=465690]: Next returned nil segment Dec 27 18:29:13.157324574 (531h0m13.028407609s) [upload=465690]: Waiting for error group managing segments... Dec 27 18:29:13.439491512 (531h0m13.310574547s) [noise=true , address=145.79.227.50:4373 , piece_id=YWS7ALQY4WW6B342OV6FNAV65NTT2NOTLXCBW7V3EZGGO6XPFIFQ, seg_pos={0 0}, upload=465690]: Done uploading piece. err: <nil> Dec 27 18:29:13.446129471 (531h0m13.317212506s) [noise=true , address=145.79.227.11:4729 , piece_id=MBVTA3IKXV3HEXFLIF362ROULECHWBDIY2ZFALRBCMVQKS7EXCNA, seg_pos={0 0}, upload=465690]: Done uploading piece. err: <nil> Dec 27 18:29:13.448688274 (531h0m13.319771319s) [noise=true , address=145.79.227.51:4409 , piece_id=JE7GRUEAZJ5YERU5UTTWPU7PD453WV3QO73KBRLIXM7PLZKPBYKA, seg_pos={0 0}, upload=465690]: Done uploading piece. err: <nil> Dec 27 18:29:13.452146659 (531h0m13.323229704s) [noise=true , address=145.79.227.39:4015 , piece_id=JPZ2243MWXGOJXPG4VKAO3PIQKXBWDH3ZXJOH3JBIT64M7OTUOSQ, seg_pos={0 0}, upload=465690]: Done uploading piece. err: <nil> Dec 27 18:29:13.460183964 (531h0m13.331266999s) [noise=true , address=145.79.227.23:4546 , piece_id=QCK2MVYBKLD3FMAQDKG6GY7YRHYN75CCJNFO75F5EIEAJQQTMQAA, seg_pos={0 0}, upload=465690]: Done uploading piece. err: <nil> Dec 27 18:29:13.468932072 (531h0m13.340015107s) [noise=true , address=145.79.227.28:4725 , piece_id=55ZRJ5KXBMUU4RXZANXSS3FBQK52QYUYQW6SVBLMQXQ7GMSCJJCA, seg_pos={0 0}, upload=465690]: Done uploading piece. err: <nil> Dec 27 18:29:13.469785857 (531h0m13.340868902s) [noise=true , address=145.79.227.36:4519 , piece_id=GCEHLTPG7JAE7LJXNFIAQUVSOQXNKHCWM5XSYURVFQRQSWMW2H3Q, seg_pos={0 0}, upload=465690]: Done uploading piece. err: <nil> Dec 27 18:29:13.481650389 (531h0m13.352733424s) [noise=true , address=145.79.227.37:4567 , piece_id=2LCXOJ7J2KGUDOKQDZ25LIQXIUMMCOEBVLXE6S6JBGMSYKZDTK2A, seg_pos={0 0}, upload=465690]: Done uploading piece. err: <nil> Dec 27 18:29:13.498795567 (531h0m13.369878612s) [noise=true , address=145.79.227.32:4834 , piece_id=WBNN47URS7ZV6RS7ZDEAEZGSIOTF3NNGMYJ5BLEWU45MGMXO644Q, seg_pos={0 0}, upload=465690]: Done uploading piece. err: <nil> Dec 27 18:29:13.649789487 (531h0m13.520872522s) [noise=true , address=77.90.31.15:4700 , piece_id=SVMVZ7XTIXVQLCUEOBYKPYBS3B735DJJ5WID4QQGFWWIS5MSLULQ, seg_pos={0 0}, upload=465690]: Done uploading piece. err: <nil> Dec 27 18:29:13.750424244 (531h0m13.621507289s) [noise=true , address=45.91.116.14:4191 , piece_id=BTX5PTJDWGFVMJDSGFDVPGKRJG5D3XT25N2O3XX5MNQ5SBVQDE5Q, seg_pos={0 0}, upload=465690]: Done uploading piece. err: <nil> Dec 27 18:29:13.752470150 (531h0m13.623553195s) [noise=true , address=145.79.227.14:4838 , piece_id=OWM7XP7V4FHR7CHBJS4ENYYNPO33RTODEGRW32RXBDEBYVVFTVVQ, seg_pos={0 0}, upload=465690]: Done uploading piece. err: <nil> Dec 27 18:29:13.763968274 (531h0m13.635051309s) [noise=true , address=77.90.31.11:4554 , piece_id=MG3ZZJPYV4VHTSEMZIQJ6I54DCEH3KK33ACKF66F3VJIUGA23YTQ, seg_pos={0 0}, upload=465690]: Done uploading piece. err: <nil> Dec 27 18:29:13.763996989 (531h0m13.635080034s) [seg_pos={0 0}, upload=465690]: Segment reached optimal threshold of 13 pieces. Dec 27 18:29:13.764239220 (531h0m13.635322265s) [noise=true , address=145.79.227.34:4907 , piece_id=POFQB5NEJ6VILLEBFTRNG3IJYTNKBSIUAT5X4SGVQV5X7YIRCFUA, seg_pos={0 0}, upload=465690]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 27 18:29:13.764251834 (531h0m13.635334879s) [noise=true , address=145.79.227.34:4907 , piece_id=POFQB5NEJ6VILLEBFTRNG3IJYTNKBSIUAT5X4SGVQV5X7YIRCFUA, seg_pos={0 0}, upload=465690]: Long tail cancellation due to optimal threshold Dec 27 18:29:13.764353908 (531h0m13.635436954s) [noise=true , address=145.79.227.49:4339 , piece_id=2WU24VSIN6V6L6DMH5GZILBETDON6FLPZ5FZGB3RQ3FIINOJEOQA, seg_pos={0 0}, upload=465690]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 27 18:29:13.764448829 (531h0m13.635531875s) [noise=true , address=145.79.227.49:4339 , piece_id=2WU24VSIN6V6L6DMH5GZILBETDON6FLPZ5FZGB3RQ3FIINOJEOQA, seg_pos={0 0}, upload=465690]: Long tail cancellation due to optimal threshold Dec 27 18:29:13.764597883 (531h0m13.635680928s) [noise=true , address=45.91.116.9:4261 , piece_id=JAIZDO4EJV7N7P4AY4I3UOMYMI22HVD7FTAAZ42F6QXLRFW5TC5Q, seg_pos={0 0}, upload=465690]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF Dec 27 18:29:13.764612061 (531h0m13.635695126s) [noise=true , address=45.91.116.9:4261 , piece_id=JAIZDO4EJV7N7P4AY4I3UOMYMI22HVD7FTAAZ42F6QXLRFW5TC5Q, seg_pos={0 0}, upload=465690]: Long tail cancellation due to optimal threshold Dec 27 18:29:13.764644432 (531h0m13.635727477s) [seg_pos={0 0}, upload=465690]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 18:29:13.764805138 (531h0m13.635888183s) [upload=465690]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 18:29:13.764813414 (531h0m13.635896459s) [upload=465690]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 18:29:14.053215415 (531h0m13.92429845s) [upload=465690]: Done uploading segments... Dec 27 18:29:14.053235483 (531h0m13.924318518s) [upload=465690]: Upload finished. err: <nil> Duration: 30.002750185s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.X..........................................................] duration: 3.52251362s trace: https://jaeger.dev.storj.io/trace/3bddfef0dfa96557 FAIL (2025-12-27 18:28:40 UTC) orbiter warning: context deadline exceeded Duration: 3.522536784s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [.XXX..XX....X...X.X........X.......X..X.XXXXXXX...X.........] duration: 6.163401353s trace: https://jaeger.dev.storj.io/trace/4981aef9a433490 FAIL (2025-12-27 18:28:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: VDRCSP2C5VIHZR6YLLR7WUGURUZOXC2G7MT56IE43PMVVXDD3HZQ) context deadline exceeded; piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: NVFVBNP6PWOJR3AT6ZB6LQWZ2JACMX5YK377GY2RE7GDM5FDPJUA) context deadline exceeded ======LOGS====== Dec 27 18:21:25.144885992 (530h52m25.015969028s) [upload=465581]: Starting upload Dec 27 18:21:25.144948482 (530h52m25.016031517s) [upload=465581]: Done starting upload Dec 27 18:21:25.145039054 (530h52m25.016122089s) [upload=465581]: Uploading segments... Dec 27 18:21:25.145258050 (530h52m25.016341095s) [upload=465581]: Got next segment. Inline: false Dec 27 18:21:25.145296154 (530h52m25.016379189s) [upload=465581]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 18:21:25.145301123 (530h52m25.016384158s) [upload=465581]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 18:21:25.389658082 (530h52m25.260741127s) [seg_pos={0 0}, upload=465581]: Begin upload segment... Dec 27 18:21:25.389719219 (530h52m25.260802264s) [seg_pos={0 0}, upload=465581]: Done begin upload segment. Dec 27 18:21:25.389751040 (530h52m25.260834095s) [noise=true , address=145.79.227.16:4910 , piece_id=PKF7JCLQKWL7DBRFMZQOTB2RURKDFJ6NMS7YB3AJYDJ2SC633JOQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.389827786 (530h52m25.260910821s) [noise=true , address=45.91.116.6:4974 , piece_id=VDRCSP2C5VIHZR6YLLR7WUGURUZOXC2G7MT56IE43PMVVXDD3HZQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.389925111 (530h52m25.261008146s) [noise=true , address=145.79.227.46:4602 , piece_id=OGGOAL326XC6BS7ULRPAM322APXHYL6FPWWVK32LHVANMFGYJWWQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.390583174 (530h52m25.261666239s) [noise=true , address=145.79.227.11:4732 , piece_id=WVWGLOP2LWAJXKXXMVSIXEXUMPYLU5UPROXM4XFGCDWYVBXTEKOA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.391844145 (530h52m25.26292719s) [noise=true , address=145.79.227.2:4113 , piece_id=B3FPIHAGBGSJN2I6BFIO6S2YPXSIHKF7ZBODNV6FGOI7DQRYTIQA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.392261891 (530h52m25.263344936s) [noise=true , address=145.79.227.4:4185 , piece_id=BSDWYICA7RMLTXX4URC3O6YC25FHVQ6ZRYL7OS2227KE3DUSWJDQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.392656372 (530h52m25.263739407s) [noise=true , address=45.91.116.13:4156 , piece_id=Y5TLN2KADJ7KUBGC6LFHRWSOSH3E2N2WLWM2O2X6O26YRL7LTDQA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.393053067 (530h52m25.264136112s) [noise=true , address=45.91.116.4:4939 , piece_id=EMKOBZFOAMID6S27D4UMCJL4JKCH2KGR4KPUKXG4P43BIJIIOGVA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.393516079 (530h52m25.264599114s) [noise=true , address=45.91.116.14:4192 , piece_id=4FEHZOQCMKHKWPHV6DCOTEOP7IYE7LQ22PX2WLFZ47IXQZ3IS36Q, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.394785246 (530h52m25.265868291s) [noise=true , address=145.79.227.10:4403 , piece_id=IZP3UFQYXDPVJQIMBMP6LWWZ6TR52FN755X7VYCFTLREMHOIJN3A, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.395302331 (530h52m25.266385376s) [noise=true , address=45.91.116.5:4297 , piece_id=NVFVBNP6PWOJR3AT6ZB6LQWZ2JACMX5YK377GY2RE7GDM5FDPJUA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.396222633 (530h52m25.267305678s) [noise=true , address=45.91.116.8:4046 , piece_id=6JWU3XRTODVGYUZBLZTMJJZFKYNTAHXGTJYQ2R7KXQKUBR7QSVKA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.397111025 (530h52m25.26819407s) [noise=true , address=145.79.227.17:4971 , piece_id=UO6L7APQZ46TBPJ5XOUHI2YL3Y5YZ2VAYFAZHOY7CL5C6BEO5G7Q, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.398240415 (530h52m25.26932346s) [noise=true , address=145.79.227.36:4517 , piece_id=RLIQWAGNKI5X6WGHYMN7FXPHWDYEVGWXLFMPN7IEZR4ZRDARFIBQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.402100956 (530h52m25.273184001s) [noise=true , address=145.79.227.48:4301 , piece_id=KKULR64B2HUQ7P4GSUX2XKQKW4DB3GVTD3PDJUOUPU7TOQ6ZKTJA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.405336728 (530h52m25.276419773s) [noise=true , address=145.79.227.45:4231 , piece_id=UX45A3K3LKYEG4YU7BVZKMJWYVUL47C5MMNQRSI32WE4Y62ILAOQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:27.033083485 (530h52m26.90416653s) [upload=465581]: Next returned nil segment Dec 27 18:21:27.033095617 (530h52m26.904178662s) [upload=465581]: Waiting for error group managing segments... Dec 27 18:21:27.422573865 (530h52m27.2936569s) [noise=true , address=145.79.227.11:4732 , piece_id=WVWGLOP2LWAJXKXXMVSIXEXUMPYLU5UPROXM4XFGCDWYVBXTEKOA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.425425164 (530h52m27.296508199s) [noise=true , address=145.79.227.46:4602 , piece_id=OGGOAL326XC6BS7ULRPAM322APXHYL6FPWWVK32LHVANMFGYJWWQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.442445785 (530h52m27.313528821s) [noise=true , address=145.79.227.45:4231 , piece_id=UX45A3K3LKYEG4YU7BVZKMJWYVUL47C5MMNQRSI32WE4Y62ILAOQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.449930007 (530h52m27.321013062s) [noise=true , address=145.79.227.17:4971 , piece_id=UO6L7APQZ46TBPJ5XOUHI2YL3Y5YZ2VAYFAZHOY7CL5C6BEO5G7Q, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.450121371 (530h52m27.321204406s) [noise=true , address=145.79.227.10:4403 , piece_id=IZP3UFQYXDPVJQIMBMP6LWWZ6TR52FN755X7VYCFTLREMHOIJN3A, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.454633623 (530h52m27.325716658s) [noise=true , address=145.79.227.48:4301 , piece_id=KKULR64B2HUQ7P4GSUX2XKQKW4DB3GVTD3PDJUOUPU7TOQ6ZKTJA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.455522034 (530h52m27.326605079s) [noise=true , address=145.79.227.2:4113 , piece_id=B3FPIHAGBGSJN2I6BFIO6S2YPXSIHKF7ZBODNV6FGOI7DQRYTIQA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.456210736 (530h52m27.327293781s) [noise=true , address=145.79.227.4:4185 , piece_id=BSDWYICA7RMLTXX4URC3O6YC25FHVQ6ZRYL7OS2227KE3DUSWJDQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.462096273 (530h52m27.333179308s) [noise=true , address=145.79.227.36:4517 , piece_id=RLIQWAGNKI5X6WGHYMN7FXPHWDYEVGWXLFMPN7IEZR4ZRDARFIBQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.509391311 (530h52m27.380474346s) [noise=true , address=145.79.227.16:4910 , piece_id=PKF7JCLQKWL7DBRFMZQOTB2RURKDFJ6NMS7YB3AJYDJ2SC633JOQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.640310134 (530h52m27.511393169s) [noise=true , address=45.91.116.6:4974 , piece_id=VDRCSP2C5VIHZR6YLLR7WUGURUZOXC2G7MT56IE43PMVVXDD3HZQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.671350691 (530h52m27.542433736s) [noise=true , address=45.91.116.5:4297 , piece_id=NVFVBNP6PWOJR3AT6ZB6LQWZ2JACMX5YK377GY2RE7GDM5FDPJUA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.671881432 (530h52m27.542964477s) [noise=true , address=45.91.116.13:4156 , piece_id=Y5TLN2KADJ7KUBGC6LFHRWSOSH3E2N2WLWM2O2X6O26YRL7LTDQA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.671891021 (530h52m27.542974056s) [seg_pos={0 0}, upload=465581]: Segment reached optimal threshold of 13 pieces. Dec 27 18:21:27.672486244 (530h52m27.543569279s) [noise=true , address=45.91.116.4:4939 , piece_id=EMKOBZFOAMID6S27D4UMCJL4JKCH2KGR4KPUKXG4P43BIJIIOGVA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: read tcp 188.166.225.170:39392->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 18:21:27.672494189 (530h52m27.543577234s) [noise=true , address=45.91.116.4:4939 , piece_id=EMKOBZFOAMID6S27D4UMCJL4JKCH2KGR4KPUKXG4P43BIJIIOGVA, seg_pos={0 0}, upload=465581]: Long tail cancellation due to optimal threshold Dec 27 18:21:27.672941320 (530h52m27.544024365s) [noise=true , address=45.91.116.14:4192 , piece_id=4FEHZOQCMKHKWPHV6DCOTEOP7IYE7LQ22PX2WLFZ47IXQZ3IS36Q, seg_pos={0 0}, upload=465581]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: read tcp 188.166.225.170:49760->45.91.116.14:4192: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 18:21:27.672949005 (530h52m27.54403204s) [noise=true , address=45.91.116.14:4192 , piece_id=4FEHZOQCMKHKWPHV6DCOTEOP7IYE7LQ22PX2WLFZ47IXQZ3IS36Q, seg_pos={0 0}, upload=465581]: Long tail cancellation due to optimal threshold Dec 27 18:21:27.673176889 (530h52m27.544259924s) [noise=true , address=45.91.116.8:4046 , piece_id=6JWU3XRTODVGYUZBLZTMJJZFKYNTAHXGTJYQ2R7KXQKUBR7QSVKA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 27 18:21:27.673184854 (530h52m27.544267899s) [noise=true , address=45.91.116.8:4046 , piece_id=6JWU3XRTODVGYUZBLZTMJJZFKYNTAHXGTJYQ2R7KXQKUBR7QSVKA, seg_pos={0 0}, upload=465581]: Long tail cancellation due to optimal threshold Dec 27 18:21:27.673203058 (530h52m27.544286093s) [seg_pos={0 0}, upload=465581]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 18:21:27.673307677 (530h52m27.544390713s) [upload=465581]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 18:21:27.673312197 (530h52m27.544395232s) [upload=465581]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 18:21:27.963559300 (530h52m27.834642355s) [upload=465581]: Done uploading segments... Dec 27 18:21:27.963583115 (530h52m27.83466616s) [upload=465581]: Upload finished. err: <nil> Duration: 6.163439494s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..X.........................................................] duration: 3.434775392s trace: https://jaeger.dev.storj.io/trace/3bddfef0dfa96557 FAIL (2025-12-27 18:27:33 UTC) orbiter warning: context deadline exceeded Duration: 3.434857096s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [..XXX..XX....X...X.X........X.......X..X.XXXXXXX...X........] duration: 13.569064724s trace: https://jaeger.dev.storj.io/trace/4981aef9a433490 FAIL (2025-12-27 18:27:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: VDRCSP2C5VIHZR6YLLR7WUGURUZOXC2G7MT56IE43PMVVXDD3HZQ) context deadline exceeded; piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: NVFVBNP6PWOJR3AT6ZB6LQWZ2JACMX5YK377GY2RE7GDM5FDPJUA) context deadline exceeded ======LOGS====== Dec 27 18:21:25.144885992 (530h52m25.015969028s) [upload=465581]: Starting upload Dec 27 18:21:25.144948482 (530h52m25.016031517s) [upload=465581]: Done starting upload Dec 27 18:21:25.145039054 (530h52m25.016122089s) [upload=465581]: Uploading segments... Dec 27 18:21:25.145258050 (530h52m25.016341095s) [upload=465581]: Got next segment. Inline: false Dec 27 18:21:25.145296154 (530h52m25.016379189s) [upload=465581]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 18:21:25.145301123 (530h52m25.016384158s) [upload=465581]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 18:21:25.389658082 (530h52m25.260741127s) [seg_pos={0 0}, upload=465581]: Begin upload segment... Dec 27 18:21:25.389719219 (530h52m25.260802264s) [seg_pos={0 0}, upload=465581]: Done begin upload segment. Dec 27 18:21:25.389751040 (530h52m25.260834095s) [noise=true , address=145.79.227.16:4910 , piece_id=PKF7JCLQKWL7DBRFMZQOTB2RURKDFJ6NMS7YB3AJYDJ2SC633JOQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.389827786 (530h52m25.260910821s) [noise=true , address=45.91.116.6:4974 , piece_id=VDRCSP2C5VIHZR6YLLR7WUGURUZOXC2G7MT56IE43PMVVXDD3HZQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.389925111 (530h52m25.261008146s) [noise=true , address=145.79.227.46:4602 , piece_id=OGGOAL326XC6BS7ULRPAM322APXHYL6FPWWVK32LHVANMFGYJWWQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.390583174 (530h52m25.261666239s) [noise=true , address=145.79.227.11:4732 , piece_id=WVWGLOP2LWAJXKXXMVSIXEXUMPYLU5UPROXM4XFGCDWYVBXTEKOA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.391844145 (530h52m25.26292719s) [noise=true , address=145.79.227.2:4113 , piece_id=B3FPIHAGBGSJN2I6BFIO6S2YPXSIHKF7ZBODNV6FGOI7DQRYTIQA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.392261891 (530h52m25.263344936s) [noise=true , address=145.79.227.4:4185 , piece_id=BSDWYICA7RMLTXX4URC3O6YC25FHVQ6ZRYL7OS2227KE3DUSWJDQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.392656372 (530h52m25.263739407s) [noise=true , address=45.91.116.13:4156 , piece_id=Y5TLN2KADJ7KUBGC6LFHRWSOSH3E2N2WLWM2O2X6O26YRL7LTDQA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.393053067 (530h52m25.264136112s) [noise=true , address=45.91.116.4:4939 , piece_id=EMKOBZFOAMID6S27D4UMCJL4JKCH2KGR4KPUKXG4P43BIJIIOGVA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.393516079 (530h52m25.264599114s) [noise=true , address=45.91.116.14:4192 , piece_id=4FEHZOQCMKHKWPHV6DCOTEOP7IYE7LQ22PX2WLFZ47IXQZ3IS36Q, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.394785246 (530h52m25.265868291s) [noise=true , address=145.79.227.10:4403 , piece_id=IZP3UFQYXDPVJQIMBMP6LWWZ6TR52FN755X7VYCFTLREMHOIJN3A, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.395302331 (530h52m25.266385376s) [noise=true , address=45.91.116.5:4297 , piece_id=NVFVBNP6PWOJR3AT6ZB6LQWZ2JACMX5YK377GY2RE7GDM5FDPJUA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.396222633 (530h52m25.267305678s) [noise=true , address=45.91.116.8:4046 , piece_id=6JWU3XRTODVGYUZBLZTMJJZFKYNTAHXGTJYQ2R7KXQKUBR7QSVKA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.397111025 (530h52m25.26819407s) [noise=true , address=145.79.227.17:4971 , piece_id=UO6L7APQZ46TBPJ5XOUHI2YL3Y5YZ2VAYFAZHOY7CL5C6BEO5G7Q, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.398240415 (530h52m25.26932346s) [noise=true , address=145.79.227.36:4517 , piece_id=RLIQWAGNKI5X6WGHYMN7FXPHWDYEVGWXLFMPN7IEZR4ZRDARFIBQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.402100956 (530h52m25.273184001s) [noise=true , address=145.79.227.48:4301 , piece_id=KKULR64B2HUQ7P4GSUX2XKQKW4DB3GVTD3PDJUOUPU7TOQ6ZKTJA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.405336728 (530h52m25.276419773s) [noise=true , address=145.79.227.45:4231 , piece_id=UX45A3K3LKYEG4YU7BVZKMJWYVUL47C5MMNQRSI32WE4Y62ILAOQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:27.033083485 (530h52m26.90416653s) [upload=465581]: Next returned nil segment Dec 27 18:21:27.033095617 (530h52m26.904178662s) [upload=465581]: Waiting for error group managing segments... Dec 27 18:21:27.422573865 (530h52m27.2936569s) [noise=true , address=145.79.227.11:4732 , piece_id=WVWGLOP2LWAJXKXXMVSIXEXUMPYLU5UPROXM4XFGCDWYVBXTEKOA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.425425164 (530h52m27.296508199s) [noise=true , address=145.79.227.46:4602 , piece_id=OGGOAL326XC6BS7ULRPAM322APXHYL6FPWWVK32LHVANMFGYJWWQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.442445785 (530h52m27.313528821s) [noise=true , address=145.79.227.45:4231 , piece_id=UX45A3K3LKYEG4YU7BVZKMJWYVUL47C5MMNQRSI32WE4Y62ILAOQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.449930007 (530h52m27.321013062s) [noise=true , address=145.79.227.17:4971 , piece_id=UO6L7APQZ46TBPJ5XOUHI2YL3Y5YZ2VAYFAZHOY7CL5C6BEO5G7Q, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.450121371 (530h52m27.321204406s) [noise=true , address=145.79.227.10:4403 , piece_id=IZP3UFQYXDPVJQIMBMP6LWWZ6TR52FN755X7VYCFTLREMHOIJN3A, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.454633623 (530h52m27.325716658s) [noise=true , address=145.79.227.48:4301 , piece_id=KKULR64B2HUQ7P4GSUX2XKQKW4DB3GVTD3PDJUOUPU7TOQ6ZKTJA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.455522034 (530h52m27.326605079s) [noise=true , address=145.79.227.2:4113 , piece_id=B3FPIHAGBGSJN2I6BFIO6S2YPXSIHKF7ZBODNV6FGOI7DQRYTIQA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.456210736 (530h52m27.327293781s) [noise=true , address=145.79.227.4:4185 , piece_id=BSDWYICA7RMLTXX4URC3O6YC25FHVQ6ZRYL7OS2227KE3DUSWJDQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.462096273 (530h52m27.333179308s) [noise=true , address=145.79.227.36:4517 , piece_id=RLIQWAGNKI5X6WGHYMN7FXPHWDYEVGWXLFMPN7IEZR4ZRDARFIBQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.509391311 (530h52m27.380474346s) [noise=true , address=145.79.227.16:4910 , piece_id=PKF7JCLQKWL7DBRFMZQOTB2RURKDFJ6NMS7YB3AJYDJ2SC633JOQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.640310134 (530h52m27.511393169s) [noise=true , address=45.91.116.6:4974 , piece_id=VDRCSP2C5VIHZR6YLLR7WUGURUZOXC2G7MT56IE43PMVVXDD3HZQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.671350691 (530h52m27.542433736s) [noise=true , address=45.91.116.5:4297 , piece_id=NVFVBNP6PWOJR3AT6ZB6LQWZ2JACMX5YK377GY2RE7GDM5FDPJUA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.671881432 (530h52m27.542964477s) [noise=true , address=45.91.116.13:4156 , piece_id=Y5TLN2KADJ7KUBGC6LFHRWSOSH3E2N2WLWM2O2X6O26YRL7LTDQA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.671891021 (530h52m27.542974056s) [seg_pos={0 0}, upload=465581]: Segment reached optimal threshold of 13 pieces. Dec 27 18:21:27.672486244 (530h52m27.543569279s) [noise=true , address=45.91.116.4:4939 , piece_id=EMKOBZFOAMID6S27D4UMCJL4JKCH2KGR4KPUKXG4P43BIJIIOGVA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: read tcp 188.166.225.170:39392->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 18:21:27.672494189 (530h52m27.543577234s) [noise=true , address=45.91.116.4:4939 , piece_id=EMKOBZFOAMID6S27D4UMCJL4JKCH2KGR4KPUKXG4P43BIJIIOGVA, seg_pos={0 0}, upload=465581]: Long tail cancellation due to optimal threshold Dec 27 18:21:27.672941320 (530h52m27.544024365s) [noise=true , address=45.91.116.14:4192 , piece_id=4FEHZOQCMKHKWPHV6DCOTEOP7IYE7LQ22PX2WLFZ47IXQZ3IS36Q, seg_pos={0 0}, upload=465581]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: read tcp 188.166.225.170:49760->45.91.116.14:4192: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 18:21:27.672949005 (530h52m27.54403204s) [noise=true , address=45.91.116.14:4192 , piece_id=4FEHZOQCMKHKWPHV6DCOTEOP7IYE7LQ22PX2WLFZ47IXQZ3IS36Q, seg_pos={0 0}, upload=465581]: Long tail cancellation due to optimal threshold Dec 27 18:21:27.673176889 (530h52m27.544259924s) [noise=true , address=45.91.116.8:4046 , piece_id=6JWU3XRTODVGYUZBLZTMJJZFKYNTAHXGTJYQ2R7KXQKUBR7QSVKA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 27 18:21:27.673184854 (530h52m27.544267899s) [noise=true , address=45.91.116.8:4046 , piece_id=6JWU3XRTODVGYUZBLZTMJJZFKYNTAHXGTJYQ2R7KXQKUBR7QSVKA, seg_pos={0 0}, upload=465581]: Long tail cancellation due to optimal threshold Dec 27 18:21:27.673203058 (530h52m27.544286093s) [seg_pos={0 0}, upload=465581]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 18:21:27.673307677 (530h52m27.544390713s) [upload=465581]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 18:21:27.673312197 (530h52m27.544395232s) [upload=465581]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 18:21:27.963559300 (530h52m27.834642355s) [upload=465581]: Done uploading segments... Dec 27 18:21:27.963583115 (530h52m27.83466616s) [upload=465581]: Upload finished. err: <nil> Duration: 13.569097467s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...X........................................................] duration: 3.558016323s trace: https://jaeger.dev.storj.io/trace/3bddfef0dfa96557 FAIL (2025-12-27 18:26:30 UTC) orbiter warning: context deadline exceeded Duration: 3.55807298s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [...XXX..XX....X...X.X........X.......X..X.XXXXXXX...X.......] duration: 15.249211185s trace: https://jaeger.dev.storj.io/trace/4981aef9a433490 FAIL (2025-12-27 18:25:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: VDRCSP2C5VIHZR6YLLR7WUGURUZOXC2G7MT56IE43PMVVXDD3HZQ) context deadline exceeded; piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: NVFVBNP6PWOJR3AT6ZB6LQWZ2JACMX5YK377GY2RE7GDM5FDPJUA) context deadline exceeded ======LOGS====== Dec 27 18:21:25.144885992 (530h52m25.015969028s) [upload=465581]: Starting upload Dec 27 18:21:25.144948482 (530h52m25.016031517s) [upload=465581]: Done starting upload Dec 27 18:21:25.145039054 (530h52m25.016122089s) [upload=465581]: Uploading segments... Dec 27 18:21:25.145258050 (530h52m25.016341095s) [upload=465581]: Got next segment. Inline: false Dec 27 18:21:25.145296154 (530h52m25.016379189s) [upload=465581]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 18:21:25.145301123 (530h52m25.016384158s) [upload=465581]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 18:21:25.389658082 (530h52m25.260741127s) [seg_pos={0 0}, upload=465581]: Begin upload segment... Dec 27 18:21:25.389719219 (530h52m25.260802264s) [seg_pos={0 0}, upload=465581]: Done begin upload segment. Dec 27 18:21:25.389751040 (530h52m25.260834095s) [noise=true , address=145.79.227.16:4910 , piece_id=PKF7JCLQKWL7DBRFMZQOTB2RURKDFJ6NMS7YB3AJYDJ2SC633JOQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.389827786 (530h52m25.260910821s) [noise=true , address=45.91.116.6:4974 , piece_id=VDRCSP2C5VIHZR6YLLR7WUGURUZOXC2G7MT56IE43PMVVXDD3HZQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.389925111 (530h52m25.261008146s) [noise=true , address=145.79.227.46:4602 , piece_id=OGGOAL326XC6BS7ULRPAM322APXHYL6FPWWVK32LHVANMFGYJWWQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.390583174 (530h52m25.261666239s) [noise=true , address=145.79.227.11:4732 , piece_id=WVWGLOP2LWAJXKXXMVSIXEXUMPYLU5UPROXM4XFGCDWYVBXTEKOA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.391844145 (530h52m25.26292719s) [noise=true , address=145.79.227.2:4113 , piece_id=B3FPIHAGBGSJN2I6BFIO6S2YPXSIHKF7ZBODNV6FGOI7DQRYTIQA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.392261891 (530h52m25.263344936s) [noise=true , address=145.79.227.4:4185 , piece_id=BSDWYICA7RMLTXX4URC3O6YC25FHVQ6ZRYL7OS2227KE3DUSWJDQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.392656372 (530h52m25.263739407s) [noise=true , address=45.91.116.13:4156 , piece_id=Y5TLN2KADJ7KUBGC6LFHRWSOSH3E2N2WLWM2O2X6O26YRL7LTDQA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.393053067 (530h52m25.264136112s) [noise=true , address=45.91.116.4:4939 , piece_id=EMKOBZFOAMID6S27D4UMCJL4JKCH2KGR4KPUKXG4P43BIJIIOGVA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.393516079 (530h52m25.264599114s) [noise=true , address=45.91.116.14:4192 , piece_id=4FEHZOQCMKHKWPHV6DCOTEOP7IYE7LQ22PX2WLFZ47IXQZ3IS36Q, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.394785246 (530h52m25.265868291s) [noise=true , address=145.79.227.10:4403 , piece_id=IZP3UFQYXDPVJQIMBMP6LWWZ6TR52FN755X7VYCFTLREMHOIJN3A, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.395302331 (530h52m25.266385376s) [noise=true , address=45.91.116.5:4297 , piece_id=NVFVBNP6PWOJR3AT6ZB6LQWZ2JACMX5YK377GY2RE7GDM5FDPJUA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.396222633 (530h52m25.267305678s) [noise=true , address=45.91.116.8:4046 , piece_id=6JWU3XRTODVGYUZBLZTMJJZFKYNTAHXGTJYQ2R7KXQKUBR7QSVKA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.397111025 (530h52m25.26819407s) [noise=true , address=145.79.227.17:4971 , piece_id=UO6L7APQZ46TBPJ5XOUHI2YL3Y5YZ2VAYFAZHOY7CL5C6BEO5G7Q, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.398240415 (530h52m25.26932346s) [noise=true , address=145.79.227.36:4517 , piece_id=RLIQWAGNKI5X6WGHYMN7FXPHWDYEVGWXLFMPN7IEZR4ZRDARFIBQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.402100956 (530h52m25.273184001s) [noise=true , address=145.79.227.48:4301 , piece_id=KKULR64B2HUQ7P4GSUX2XKQKW4DB3GVTD3PDJUOUPU7TOQ6ZKTJA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.405336728 (530h52m25.276419773s) [noise=true , address=145.79.227.45:4231 , piece_id=UX45A3K3LKYEG4YU7BVZKMJWYVUL47C5MMNQRSI32WE4Y62ILAOQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:27.033083485 (530h52m26.90416653s) [upload=465581]: Next returned nil segment Dec 27 18:21:27.033095617 (530h52m26.904178662s) [upload=465581]: Waiting for error group managing segments... Dec 27 18:21:27.422573865 (530h52m27.2936569s) [noise=true , address=145.79.227.11:4732 , piece_id=WVWGLOP2LWAJXKXXMVSIXEXUMPYLU5UPROXM4XFGCDWYVBXTEKOA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.425425164 (530h52m27.296508199s) [noise=true , address=145.79.227.46:4602 , piece_id=OGGOAL326XC6BS7ULRPAM322APXHYL6FPWWVK32LHVANMFGYJWWQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.442445785 (530h52m27.313528821s) [noise=true , address=145.79.227.45:4231 , piece_id=UX45A3K3LKYEG4YU7BVZKMJWYVUL47C5MMNQRSI32WE4Y62ILAOQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.449930007 (530h52m27.321013062s) [noise=true , address=145.79.227.17:4971 , piece_id=UO6L7APQZ46TBPJ5XOUHI2YL3Y5YZ2VAYFAZHOY7CL5C6BEO5G7Q, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.450121371 (530h52m27.321204406s) [noise=true , address=145.79.227.10:4403 , piece_id=IZP3UFQYXDPVJQIMBMP6LWWZ6TR52FN755X7VYCFTLREMHOIJN3A, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.454633623 (530h52m27.325716658s) [noise=true , address=145.79.227.48:4301 , piece_id=KKULR64B2HUQ7P4GSUX2XKQKW4DB3GVTD3PDJUOUPU7TOQ6ZKTJA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.455522034 (530h52m27.326605079s) [noise=true , address=145.79.227.2:4113 , piece_id=B3FPIHAGBGSJN2I6BFIO6S2YPXSIHKF7ZBODNV6FGOI7DQRYTIQA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.456210736 (530h52m27.327293781s) [noise=true , address=145.79.227.4:4185 , piece_id=BSDWYICA7RMLTXX4URC3O6YC25FHVQ6ZRYL7OS2227KE3DUSWJDQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.462096273 (530h52m27.333179308s) [noise=true , address=145.79.227.36:4517 , piece_id=RLIQWAGNKI5X6WGHYMN7FXPHWDYEVGWXLFMPN7IEZR4ZRDARFIBQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.509391311 (530h52m27.380474346s) [noise=true , address=145.79.227.16:4910 , piece_id=PKF7JCLQKWL7DBRFMZQOTB2RURKDFJ6NMS7YB3AJYDJ2SC633JOQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.640310134 (530h52m27.511393169s) [noise=true , address=45.91.116.6:4974 , piece_id=VDRCSP2C5VIHZR6YLLR7WUGURUZOXC2G7MT56IE43PMVVXDD3HZQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.671350691 (530h52m27.542433736s) [noise=true , address=45.91.116.5:4297 , piece_id=NVFVBNP6PWOJR3AT6ZB6LQWZ2JACMX5YK377GY2RE7GDM5FDPJUA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.671881432 (530h52m27.542964477s) [noise=true , address=45.91.116.13:4156 , piece_id=Y5TLN2KADJ7KUBGC6LFHRWSOSH3E2N2WLWM2O2X6O26YRL7LTDQA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.671891021 (530h52m27.542974056s) [seg_pos={0 0}, upload=465581]: Segment reached optimal threshold of 13 pieces. Dec 27 18:21:27.672486244 (530h52m27.543569279s) [noise=true , address=45.91.116.4:4939 , piece_id=EMKOBZFOAMID6S27D4UMCJL4JKCH2KGR4KPUKXG4P43BIJIIOGVA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: read tcp 188.166.225.170:39392->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 18:21:27.672494189 (530h52m27.543577234s) [noise=true , address=45.91.116.4:4939 , piece_id=EMKOBZFOAMID6S27D4UMCJL4JKCH2KGR4KPUKXG4P43BIJIIOGVA, seg_pos={0 0}, upload=465581]: Long tail cancellation due to optimal threshold Dec 27 18:21:27.672941320 (530h52m27.544024365s) [noise=true , address=45.91.116.14:4192 , piece_id=4FEHZOQCMKHKWPHV6DCOTEOP7IYE7LQ22PX2WLFZ47IXQZ3IS36Q, seg_pos={0 0}, upload=465581]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: read tcp 188.166.225.170:49760->45.91.116.14:4192: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 18:21:27.672949005 (530h52m27.54403204s) [noise=true , address=45.91.116.14:4192 , piece_id=4FEHZOQCMKHKWPHV6DCOTEOP7IYE7LQ22PX2WLFZ47IXQZ3IS36Q, seg_pos={0 0}, upload=465581]: Long tail cancellation due to optimal threshold Dec 27 18:21:27.673176889 (530h52m27.544259924s) [noise=true , address=45.91.116.8:4046 , piece_id=6JWU3XRTODVGYUZBLZTMJJZFKYNTAHXGTJYQ2R7KXQKUBR7QSVKA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 27 18:21:27.673184854 (530h52m27.544267899s) [noise=true , address=45.91.116.8:4046 , piece_id=6JWU3XRTODVGYUZBLZTMJJZFKYNTAHXGTJYQ2R7KXQKUBR7QSVKA, seg_pos={0 0}, upload=465581]: Long tail cancellation due to optimal threshold Dec 27 18:21:27.673203058 (530h52m27.544286093s) [seg_pos={0 0}, upload=465581]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 18:21:27.673307677 (530h52m27.544390713s) [upload=465581]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 18:21:27.673312197 (530h52m27.544395232s) [upload=465581]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 18:21:27.963559300 (530h52m27.834642355s) [upload=465581]: Done uploading segments... Dec 27 18:21:27.963583115 (530h52m27.83466616s) [upload=465581]: Upload finished. err: <nil> Duration: 15.249241113s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....X.......................................................] duration: 3.583378069s trace: https://jaeger.dev.storj.io/trace/3bddfef0dfa96557 FAIL (2025-12-27 18:25:36 UTC) orbiter warning: context deadline exceeded Duration: 3.583405521s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [....XXX..XX....X...X.X........X.......X..X.XXXXXXX...X......] duration: 6.138962526s trace: https://jaeger.dev.storj.io/trace/4981aef9a433490 FAIL (2025-12-27 18:25:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: VDRCSP2C5VIHZR6YLLR7WUGURUZOXC2G7MT56IE43PMVVXDD3HZQ) context deadline exceeded; piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: NVFVBNP6PWOJR3AT6ZB6LQWZ2JACMX5YK377GY2RE7GDM5FDPJUA) context deadline exceeded ======LOGS====== Dec 27 18:21:25.144885992 (530h52m25.015969028s) [upload=465581]: Starting upload Dec 27 18:21:25.144948482 (530h52m25.016031517s) [upload=465581]: Done starting upload Dec 27 18:21:25.145039054 (530h52m25.016122089s) [upload=465581]: Uploading segments... Dec 27 18:21:25.145258050 (530h52m25.016341095s) [upload=465581]: Got next segment. Inline: false Dec 27 18:21:25.145296154 (530h52m25.016379189s) [upload=465581]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 18:21:25.145301123 (530h52m25.016384158s) [upload=465581]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 18:21:25.389658082 (530h52m25.260741127s) [seg_pos={0 0}, upload=465581]: Begin upload segment... Dec 27 18:21:25.389719219 (530h52m25.260802264s) [seg_pos={0 0}, upload=465581]: Done begin upload segment. Dec 27 18:21:25.389751040 (530h52m25.260834095s) [noise=true , address=145.79.227.16:4910 , piece_id=PKF7JCLQKWL7DBRFMZQOTB2RURKDFJ6NMS7YB3AJYDJ2SC633JOQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.389827786 (530h52m25.260910821s) [noise=true , address=45.91.116.6:4974 , piece_id=VDRCSP2C5VIHZR6YLLR7WUGURUZOXC2G7MT56IE43PMVVXDD3HZQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.389925111 (530h52m25.261008146s) [noise=true , address=145.79.227.46:4602 , piece_id=OGGOAL326XC6BS7ULRPAM322APXHYL6FPWWVK32LHVANMFGYJWWQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.390583174 (530h52m25.261666239s) [noise=true , address=145.79.227.11:4732 , piece_id=WVWGLOP2LWAJXKXXMVSIXEXUMPYLU5UPROXM4XFGCDWYVBXTEKOA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.391844145 (530h52m25.26292719s) [noise=true , address=145.79.227.2:4113 , piece_id=B3FPIHAGBGSJN2I6BFIO6S2YPXSIHKF7ZBODNV6FGOI7DQRYTIQA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.392261891 (530h52m25.263344936s) [noise=true , address=145.79.227.4:4185 , piece_id=BSDWYICA7RMLTXX4URC3O6YC25FHVQ6ZRYL7OS2227KE3DUSWJDQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.392656372 (530h52m25.263739407s) [noise=true , address=45.91.116.13:4156 , piece_id=Y5TLN2KADJ7KUBGC6LFHRWSOSH3E2N2WLWM2O2X6O26YRL7LTDQA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.393053067 (530h52m25.264136112s) [noise=true , address=45.91.116.4:4939 , piece_id=EMKOBZFOAMID6S27D4UMCJL4JKCH2KGR4KPUKXG4P43BIJIIOGVA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.393516079 (530h52m25.264599114s) [noise=true , address=45.91.116.14:4192 , piece_id=4FEHZOQCMKHKWPHV6DCOTEOP7IYE7LQ22PX2WLFZ47IXQZ3IS36Q, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.394785246 (530h52m25.265868291s) [noise=true , address=145.79.227.10:4403 , piece_id=IZP3UFQYXDPVJQIMBMP6LWWZ6TR52FN755X7VYCFTLREMHOIJN3A, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.395302331 (530h52m25.266385376s) [noise=true , address=45.91.116.5:4297 , piece_id=NVFVBNP6PWOJR3AT6ZB6LQWZ2JACMX5YK377GY2RE7GDM5FDPJUA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.396222633 (530h52m25.267305678s) [noise=true , address=45.91.116.8:4046 , piece_id=6JWU3XRTODVGYUZBLZTMJJZFKYNTAHXGTJYQ2R7KXQKUBR7QSVKA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.397111025 (530h52m25.26819407s) [noise=true , address=145.79.227.17:4971 , piece_id=UO6L7APQZ46TBPJ5XOUHI2YL3Y5YZ2VAYFAZHOY7CL5C6BEO5G7Q, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.398240415 (530h52m25.26932346s) [noise=true , address=145.79.227.36:4517 , piece_id=RLIQWAGNKI5X6WGHYMN7FXPHWDYEVGWXLFMPN7IEZR4ZRDARFIBQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.402100956 (530h52m25.273184001s) [noise=true , address=145.79.227.48:4301 , piece_id=KKULR64B2HUQ7P4GSUX2XKQKW4DB3GVTD3PDJUOUPU7TOQ6ZKTJA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.405336728 (530h52m25.276419773s) [noise=true , address=145.79.227.45:4231 , piece_id=UX45A3K3LKYEG4YU7BVZKMJWYVUL47C5MMNQRSI32WE4Y62ILAOQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:27.033083485 (530h52m26.90416653s) [upload=465581]: Next returned nil segment Dec 27 18:21:27.033095617 (530h52m26.904178662s) [upload=465581]: Waiting for error group managing segments... Dec 27 18:21:27.422573865 (530h52m27.2936569s) [noise=true , address=145.79.227.11:4732 , piece_id=WVWGLOP2LWAJXKXXMVSIXEXUMPYLU5UPROXM4XFGCDWYVBXTEKOA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.425425164 (530h52m27.296508199s) [noise=true , address=145.79.227.46:4602 , piece_id=OGGOAL326XC6BS7ULRPAM322APXHYL6FPWWVK32LHVANMFGYJWWQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.442445785 (530h52m27.313528821s) [noise=true , address=145.79.227.45:4231 , piece_id=UX45A3K3LKYEG4YU7BVZKMJWYVUL47C5MMNQRSI32WE4Y62ILAOQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.449930007 (530h52m27.321013062s) [noise=true , address=145.79.227.17:4971 , piece_id=UO6L7APQZ46TBPJ5XOUHI2YL3Y5YZ2VAYFAZHOY7CL5C6BEO5G7Q, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.450121371 (530h52m27.321204406s) [noise=true , address=145.79.227.10:4403 , piece_id=IZP3UFQYXDPVJQIMBMP6LWWZ6TR52FN755X7VYCFTLREMHOIJN3A, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.454633623 (530h52m27.325716658s) [noise=true , address=145.79.227.48:4301 , piece_id=KKULR64B2HUQ7P4GSUX2XKQKW4DB3GVTD3PDJUOUPU7TOQ6ZKTJA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.455522034 (530h52m27.326605079s) [noise=true , address=145.79.227.2:4113 , piece_id=B3FPIHAGBGSJN2I6BFIO6S2YPXSIHKF7ZBODNV6FGOI7DQRYTIQA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.456210736 (530h52m27.327293781s) [noise=true , address=145.79.227.4:4185 , piece_id=BSDWYICA7RMLTXX4URC3O6YC25FHVQ6ZRYL7OS2227KE3DUSWJDQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.462096273 (530h52m27.333179308s) [noise=true , address=145.79.227.36:4517 , piece_id=RLIQWAGNKI5X6WGHYMN7FXPHWDYEVGWXLFMPN7IEZR4ZRDARFIBQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.509391311 (530h52m27.380474346s) [noise=true , address=145.79.227.16:4910 , piece_id=PKF7JCLQKWL7DBRFMZQOTB2RURKDFJ6NMS7YB3AJYDJ2SC633JOQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.640310134 (530h52m27.511393169s) [noise=true , address=45.91.116.6:4974 , piece_id=VDRCSP2C5VIHZR6YLLR7WUGURUZOXC2G7MT56IE43PMVVXDD3HZQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.671350691 (530h52m27.542433736s) [noise=true , address=45.91.116.5:4297 , piece_id=NVFVBNP6PWOJR3AT6ZB6LQWZ2JACMX5YK377GY2RE7GDM5FDPJUA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.671881432 (530h52m27.542964477s) [noise=true , address=45.91.116.13:4156 , piece_id=Y5TLN2KADJ7KUBGC6LFHRWSOSH3E2N2WLWM2O2X6O26YRL7LTDQA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.671891021 (530h52m27.542974056s) [seg_pos={0 0}, upload=465581]: Segment reached optimal threshold of 13 pieces. Dec 27 18:21:27.672486244 (530h52m27.543569279s) [noise=true , address=45.91.116.4:4939 , piece_id=EMKOBZFOAMID6S27D4UMCJL4JKCH2KGR4KPUKXG4P43BIJIIOGVA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: read tcp 188.166.225.170:39392->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 18:21:27.672494189 (530h52m27.543577234s) [noise=true , address=45.91.116.4:4939 , piece_id=EMKOBZFOAMID6S27D4UMCJL4JKCH2KGR4KPUKXG4P43BIJIIOGVA, seg_pos={0 0}, upload=465581]: Long tail cancellation due to optimal threshold Dec 27 18:21:27.672941320 (530h52m27.544024365s) [noise=true , address=45.91.116.14:4192 , piece_id=4FEHZOQCMKHKWPHV6DCOTEOP7IYE7LQ22PX2WLFZ47IXQZ3IS36Q, seg_pos={0 0}, upload=465581]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: read tcp 188.166.225.170:49760->45.91.116.14:4192: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 18:21:27.672949005 (530h52m27.54403204s) [noise=true , address=45.91.116.14:4192 , piece_id=4FEHZOQCMKHKWPHV6DCOTEOP7IYE7LQ22PX2WLFZ47IXQZ3IS36Q, seg_pos={0 0}, upload=465581]: Long tail cancellation due to optimal threshold Dec 27 18:21:27.673176889 (530h52m27.544259924s) [noise=true , address=45.91.116.8:4046 , piece_id=6JWU3XRTODVGYUZBLZTMJJZFKYNTAHXGTJYQ2R7KXQKUBR7QSVKA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 27 18:21:27.673184854 (530h52m27.544267899s) [noise=true , address=45.91.116.8:4046 , piece_id=6JWU3XRTODVGYUZBLZTMJJZFKYNTAHXGTJYQ2R7KXQKUBR7QSVKA, seg_pos={0 0}, upload=465581]: Long tail cancellation due to optimal threshold Dec 27 18:21:27.673203058 (530h52m27.544286093s) [seg_pos={0 0}, upload=465581]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 18:21:27.673307677 (530h52m27.544390713s) [upload=465581]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 18:21:27.673312197 (530h52m27.544395232s) [upload=465581]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 18:21:27.963559300 (530h52m27.834642355s) [upload=465581]: Done uploading segments... Dec 27 18:21:27.963583115 (530h52m27.83466616s) [upload=465581]: Upload finished. err: <nil> Duration: 6.138992303s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [X....XXX..XX....X...X.X........X.......X..X.XXXXXXX...X.....] duration: 6.001534146s trace: https://jaeger.dev.storj.io/trace/4981aef9a433490 FAIL (2025-12-27 18:24:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: VDRCSP2C5VIHZR6YLLR7WUGURUZOXC2G7MT56IE43PMVVXDD3HZQ) context deadline exceeded; piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: NVFVBNP6PWOJR3AT6ZB6LQWZ2JACMX5YK377GY2RE7GDM5FDPJUA) context deadline exceeded ======LOGS====== Dec 27 18:21:25.144885992 (530h52m25.015969028s) [upload=465581]: Starting upload Dec 27 18:21:25.144948482 (530h52m25.016031517s) [upload=465581]: Done starting upload Dec 27 18:21:25.145039054 (530h52m25.016122089s) [upload=465581]: Uploading segments... Dec 27 18:21:25.145258050 (530h52m25.016341095s) [upload=465581]: Got next segment. Inline: false Dec 27 18:21:25.145296154 (530h52m25.016379189s) [upload=465581]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 18:21:25.145301123 (530h52m25.016384158s) [upload=465581]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 18:21:25.389658082 (530h52m25.260741127s) [seg_pos={0 0}, upload=465581]: Begin upload segment... Dec 27 18:21:25.389719219 (530h52m25.260802264s) [seg_pos={0 0}, upload=465581]: Done begin upload segment. Dec 27 18:21:25.389751040 (530h52m25.260834095s) [noise=true , address=145.79.227.16:4910 , piece_id=PKF7JCLQKWL7DBRFMZQOTB2RURKDFJ6NMS7YB3AJYDJ2SC633JOQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.389827786 (530h52m25.260910821s) [noise=true , address=45.91.116.6:4974 , piece_id=VDRCSP2C5VIHZR6YLLR7WUGURUZOXC2G7MT56IE43PMVVXDD3HZQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.389925111 (530h52m25.261008146s) [noise=true , address=145.79.227.46:4602 , piece_id=OGGOAL326XC6BS7ULRPAM322APXHYL6FPWWVK32LHVANMFGYJWWQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.390583174 (530h52m25.261666239s) [noise=true , address=145.79.227.11:4732 , piece_id=WVWGLOP2LWAJXKXXMVSIXEXUMPYLU5UPROXM4XFGCDWYVBXTEKOA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.391844145 (530h52m25.26292719s) [noise=true , address=145.79.227.2:4113 , piece_id=B3FPIHAGBGSJN2I6BFIO6S2YPXSIHKF7ZBODNV6FGOI7DQRYTIQA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.392261891 (530h52m25.263344936s) [noise=true , address=145.79.227.4:4185 , piece_id=BSDWYICA7RMLTXX4URC3O6YC25FHVQ6ZRYL7OS2227KE3DUSWJDQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.392656372 (530h52m25.263739407s) [noise=true , address=45.91.116.13:4156 , piece_id=Y5TLN2KADJ7KUBGC6LFHRWSOSH3E2N2WLWM2O2X6O26YRL7LTDQA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.393053067 (530h52m25.264136112s) [noise=true , address=45.91.116.4:4939 , piece_id=EMKOBZFOAMID6S27D4UMCJL4JKCH2KGR4KPUKXG4P43BIJIIOGVA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.393516079 (530h52m25.264599114s) [noise=true , address=45.91.116.14:4192 , piece_id=4FEHZOQCMKHKWPHV6DCOTEOP7IYE7LQ22PX2WLFZ47IXQZ3IS36Q, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.394785246 (530h52m25.265868291s) [noise=true , address=145.79.227.10:4403 , piece_id=IZP3UFQYXDPVJQIMBMP6LWWZ6TR52FN755X7VYCFTLREMHOIJN3A, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.395302331 (530h52m25.266385376s) [noise=true , address=45.91.116.5:4297 , piece_id=NVFVBNP6PWOJR3AT6ZB6LQWZ2JACMX5YK377GY2RE7GDM5FDPJUA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.396222633 (530h52m25.267305678s) [noise=true , address=45.91.116.8:4046 , piece_id=6JWU3XRTODVGYUZBLZTMJJZFKYNTAHXGTJYQ2R7KXQKUBR7QSVKA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.397111025 (530h52m25.26819407s) [noise=true , address=145.79.227.17:4971 , piece_id=UO6L7APQZ46TBPJ5XOUHI2YL3Y5YZ2VAYFAZHOY7CL5C6BEO5G7Q, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.398240415 (530h52m25.26932346s) [noise=true , address=145.79.227.36:4517 , piece_id=RLIQWAGNKI5X6WGHYMN7FXPHWDYEVGWXLFMPN7IEZR4ZRDARFIBQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.402100956 (530h52m25.273184001s) [noise=true , address=145.79.227.48:4301 , piece_id=KKULR64B2HUQ7P4GSUX2XKQKW4DB3GVTD3PDJUOUPU7TOQ6ZKTJA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.405336728 (530h52m25.276419773s) [noise=true , address=145.79.227.45:4231 , piece_id=UX45A3K3LKYEG4YU7BVZKMJWYVUL47C5MMNQRSI32WE4Y62ILAOQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:27.033083485 (530h52m26.90416653s) [upload=465581]: Next returned nil segment Dec 27 18:21:27.033095617 (530h52m26.904178662s) [upload=465581]: Waiting for error group managing segments... Dec 27 18:21:27.422573865 (530h52m27.2936569s) [noise=true , address=145.79.227.11:4732 , piece_id=WVWGLOP2LWAJXKXXMVSIXEXUMPYLU5UPROXM4XFGCDWYVBXTEKOA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.425425164 (530h52m27.296508199s) [noise=true , address=145.79.227.46:4602 , piece_id=OGGOAL326XC6BS7ULRPAM322APXHYL6FPWWVK32LHVANMFGYJWWQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.442445785 (530h52m27.313528821s) [noise=true , address=145.79.227.45:4231 , piece_id=UX45A3K3LKYEG4YU7BVZKMJWYVUL47C5MMNQRSI32WE4Y62ILAOQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.449930007 (530h52m27.321013062s) [noise=true , address=145.79.227.17:4971 , piece_id=UO6L7APQZ46TBPJ5XOUHI2YL3Y5YZ2VAYFAZHOY7CL5C6BEO5G7Q, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.450121371 (530h52m27.321204406s) [noise=true , address=145.79.227.10:4403 , piece_id=IZP3UFQYXDPVJQIMBMP6LWWZ6TR52FN755X7VYCFTLREMHOIJN3A, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.454633623 (530h52m27.325716658s) [noise=true , address=145.79.227.48:4301 , piece_id=KKULR64B2HUQ7P4GSUX2XKQKW4DB3GVTD3PDJUOUPU7TOQ6ZKTJA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.455522034 (530h52m27.326605079s) [noise=true , address=145.79.227.2:4113 , piece_id=B3FPIHAGBGSJN2I6BFIO6S2YPXSIHKF7ZBODNV6FGOI7DQRYTIQA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.456210736 (530h52m27.327293781s) [noise=true , address=145.79.227.4:4185 , piece_id=BSDWYICA7RMLTXX4URC3O6YC25FHVQ6ZRYL7OS2227KE3DUSWJDQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.462096273 (530h52m27.333179308s) [noise=true , address=145.79.227.36:4517 , piece_id=RLIQWAGNKI5X6WGHYMN7FXPHWDYEVGWXLFMPN7IEZR4ZRDARFIBQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.509391311 (530h52m27.380474346s) [noise=true , address=145.79.227.16:4910 , piece_id=PKF7JCLQKWL7DBRFMZQOTB2RURKDFJ6NMS7YB3AJYDJ2SC633JOQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.640310134 (530h52m27.511393169s) [noise=true , address=45.91.116.6:4974 , piece_id=VDRCSP2C5VIHZR6YLLR7WUGURUZOXC2G7MT56IE43PMVVXDD3HZQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.671350691 (530h52m27.542433736s) [noise=true , address=45.91.116.5:4297 , piece_id=NVFVBNP6PWOJR3AT6ZB6LQWZ2JACMX5YK377GY2RE7GDM5FDPJUA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.671881432 (530h52m27.542964477s) [noise=true , address=45.91.116.13:4156 , piece_id=Y5TLN2KADJ7KUBGC6LFHRWSOSH3E2N2WLWM2O2X6O26YRL7LTDQA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.671891021 (530h52m27.542974056s) [seg_pos={0 0}, upload=465581]: Segment reached optimal threshold of 13 pieces. Dec 27 18:21:27.672486244 (530h52m27.543569279s) [noise=true , address=45.91.116.4:4939 , piece_id=EMKOBZFOAMID6S27D4UMCJL4JKCH2KGR4KPUKXG4P43BIJIIOGVA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: read tcp 188.166.225.170:39392->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 18:21:27.672494189 (530h52m27.543577234s) [noise=true , address=45.91.116.4:4939 , piece_id=EMKOBZFOAMID6S27D4UMCJL4JKCH2KGR4KPUKXG4P43BIJIIOGVA, seg_pos={0 0}, upload=465581]: Long tail cancellation due to optimal threshold Dec 27 18:21:27.672941320 (530h52m27.544024365s) [noise=true , address=45.91.116.14:4192 , piece_id=4FEHZOQCMKHKWPHV6DCOTEOP7IYE7LQ22PX2WLFZ47IXQZ3IS36Q, seg_pos={0 0}, upload=465581]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: read tcp 188.166.225.170:49760->45.91.116.14:4192: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 18:21:27.672949005 (530h52m27.54403204s) [noise=true , address=45.91.116.14:4192 , piece_id=4FEHZOQCMKHKWPHV6DCOTEOP7IYE7LQ22PX2WLFZ47IXQZ3IS36Q, seg_pos={0 0}, upload=465581]: Long tail cancellation due to optimal threshold Dec 27 18:21:27.673176889 (530h52m27.544259924s) [noise=true , address=45.91.116.8:4046 , piece_id=6JWU3XRTODVGYUZBLZTMJJZFKYNTAHXGTJYQ2R7KXQKUBR7QSVKA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 27 18:21:27.673184854 (530h52m27.544267899s) [noise=true , address=45.91.116.8:4046 , piece_id=6JWU3XRTODVGYUZBLZTMJJZFKYNTAHXGTJYQ2R7KXQKUBR7QSVKA, seg_pos={0 0}, upload=465581]: Long tail cancellation due to optimal threshold Dec 27 18:21:27.673203058 (530h52m27.544286093s) [seg_pos={0 0}, upload=465581]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 18:21:27.673307677 (530h52m27.544390713s) [upload=465581]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 18:21:27.673312197 (530h52m27.544395232s) [upload=465581]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 18:21:27.963559300 (530h52m27.834642355s) [upload=465581]: Done uploading segments... Dec 27 18:21:27.963583115 (530h52m27.83466616s) [upload=465581]: Upload finished. err: <nil> Duration: 6.0015675s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....X......................................................] duration: 3.505256485s trace: https://jaeger.dev.storj.io/trace/3bddfef0dfa96557 FAIL (2025-12-27 18:24:32 UTC) orbiter warning: context deadline exceeded Duration: 3.505284629s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [.X....XXX..XX....X...X.X........X.......X..X.XXXXXXX...X....] duration: 5.650115742s trace: https://jaeger.dev.storj.io/trace/4981aef9a433490 FAIL (2025-12-27 18:24:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: VDRCSP2C5VIHZR6YLLR7WUGURUZOXC2G7MT56IE43PMVVXDD3HZQ) context deadline exceeded; piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: NVFVBNP6PWOJR3AT6ZB6LQWZ2JACMX5YK377GY2RE7GDM5FDPJUA) context deadline exceeded ======LOGS====== Dec 27 18:21:25.144885992 (530h52m25.015969028s) [upload=465581]: Starting upload Dec 27 18:21:25.144948482 (530h52m25.016031517s) [upload=465581]: Done starting upload Dec 27 18:21:25.145039054 (530h52m25.016122089s) [upload=465581]: Uploading segments... Dec 27 18:21:25.145258050 (530h52m25.016341095s) [upload=465581]: Got next segment. Inline: false Dec 27 18:21:25.145296154 (530h52m25.016379189s) [upload=465581]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 18:21:25.145301123 (530h52m25.016384158s) [upload=465581]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 18:21:25.389658082 (530h52m25.260741127s) [seg_pos={0 0}, upload=465581]: Begin upload segment... Dec 27 18:21:25.389719219 (530h52m25.260802264s) [seg_pos={0 0}, upload=465581]: Done begin upload segment. Dec 27 18:21:25.389751040 (530h52m25.260834095s) [noise=true , address=145.79.227.16:4910 , piece_id=PKF7JCLQKWL7DBRFMZQOTB2RURKDFJ6NMS7YB3AJYDJ2SC633JOQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.389827786 (530h52m25.260910821s) [noise=true , address=45.91.116.6:4974 , piece_id=VDRCSP2C5VIHZR6YLLR7WUGURUZOXC2G7MT56IE43PMVVXDD3HZQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.389925111 (530h52m25.261008146s) [noise=true , address=145.79.227.46:4602 , piece_id=OGGOAL326XC6BS7ULRPAM322APXHYL6FPWWVK32LHVANMFGYJWWQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.390583174 (530h52m25.261666239s) [noise=true , address=145.79.227.11:4732 , piece_id=WVWGLOP2LWAJXKXXMVSIXEXUMPYLU5UPROXM4XFGCDWYVBXTEKOA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.391844145 (530h52m25.26292719s) [noise=true , address=145.79.227.2:4113 , piece_id=B3FPIHAGBGSJN2I6BFIO6S2YPXSIHKF7ZBODNV6FGOI7DQRYTIQA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.392261891 (530h52m25.263344936s) [noise=true , address=145.79.227.4:4185 , piece_id=BSDWYICA7RMLTXX4URC3O6YC25FHVQ6ZRYL7OS2227KE3DUSWJDQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.392656372 (530h52m25.263739407s) [noise=true , address=45.91.116.13:4156 , piece_id=Y5TLN2KADJ7KUBGC6LFHRWSOSH3E2N2WLWM2O2X6O26YRL7LTDQA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.393053067 (530h52m25.264136112s) [noise=true , address=45.91.116.4:4939 , piece_id=EMKOBZFOAMID6S27D4UMCJL4JKCH2KGR4KPUKXG4P43BIJIIOGVA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.393516079 (530h52m25.264599114s) [noise=true , address=45.91.116.14:4192 , piece_id=4FEHZOQCMKHKWPHV6DCOTEOP7IYE7LQ22PX2WLFZ47IXQZ3IS36Q, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.394785246 (530h52m25.265868291s) [noise=true , address=145.79.227.10:4403 , piece_id=IZP3UFQYXDPVJQIMBMP6LWWZ6TR52FN755X7VYCFTLREMHOIJN3A, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.395302331 (530h52m25.266385376s) [noise=true , address=45.91.116.5:4297 , piece_id=NVFVBNP6PWOJR3AT6ZB6LQWZ2JACMX5YK377GY2RE7GDM5FDPJUA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.396222633 (530h52m25.267305678s) [noise=true , address=45.91.116.8:4046 , piece_id=6JWU3XRTODVGYUZBLZTMJJZFKYNTAHXGTJYQ2R7KXQKUBR7QSVKA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.397111025 (530h52m25.26819407s) [noise=true , address=145.79.227.17:4971 , piece_id=UO6L7APQZ46TBPJ5XOUHI2YL3Y5YZ2VAYFAZHOY7CL5C6BEO5G7Q, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.398240415 (530h52m25.26932346s) [noise=true , address=145.79.227.36:4517 , piece_id=RLIQWAGNKI5X6WGHYMN7FXPHWDYEVGWXLFMPN7IEZR4ZRDARFIBQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.402100956 (530h52m25.273184001s) [noise=true , address=145.79.227.48:4301 , piece_id=KKULR64B2HUQ7P4GSUX2XKQKW4DB3GVTD3PDJUOUPU7TOQ6ZKTJA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.405336728 (530h52m25.276419773s) [noise=true , address=145.79.227.45:4231 , piece_id=UX45A3K3LKYEG4YU7BVZKMJWYVUL47C5MMNQRSI32WE4Y62ILAOQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:27.033083485 (530h52m26.90416653s) [upload=465581]: Next returned nil segment Dec 27 18:21:27.033095617 (530h52m26.904178662s) [upload=465581]: Waiting for error group managing segments... Dec 27 18:21:27.422573865 (530h52m27.2936569s) [noise=true , address=145.79.227.11:4732 , piece_id=WVWGLOP2LWAJXKXXMVSIXEXUMPYLU5UPROXM4XFGCDWYVBXTEKOA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.425425164 (530h52m27.296508199s) [noise=true , address=145.79.227.46:4602 , piece_id=OGGOAL326XC6BS7ULRPAM322APXHYL6FPWWVK32LHVANMFGYJWWQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.442445785 (530h52m27.313528821s) [noise=true , address=145.79.227.45:4231 , piece_id=UX45A3K3LKYEG4YU7BVZKMJWYVUL47C5MMNQRSI32WE4Y62ILAOQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.449930007 (530h52m27.321013062s) [noise=true , address=145.79.227.17:4971 , piece_id=UO6L7APQZ46TBPJ5XOUHI2YL3Y5YZ2VAYFAZHOY7CL5C6BEO5G7Q, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.450121371 (530h52m27.321204406s) [noise=true , address=145.79.227.10:4403 , piece_id=IZP3UFQYXDPVJQIMBMP6LWWZ6TR52FN755X7VYCFTLREMHOIJN3A, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.454633623 (530h52m27.325716658s) [noise=true , address=145.79.227.48:4301 , piece_id=KKULR64B2HUQ7P4GSUX2XKQKW4DB3GVTD3PDJUOUPU7TOQ6ZKTJA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.455522034 (530h52m27.326605079s) [noise=true , address=145.79.227.2:4113 , piece_id=B3FPIHAGBGSJN2I6BFIO6S2YPXSIHKF7ZBODNV6FGOI7DQRYTIQA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.456210736 (530h52m27.327293781s) [noise=true , address=145.79.227.4:4185 , piece_id=BSDWYICA7RMLTXX4URC3O6YC25FHVQ6ZRYL7OS2227KE3DUSWJDQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.462096273 (530h52m27.333179308s) [noise=true , address=145.79.227.36:4517 , piece_id=RLIQWAGNKI5X6WGHYMN7FXPHWDYEVGWXLFMPN7IEZR4ZRDARFIBQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.509391311 (530h52m27.380474346s) [noise=true , address=145.79.227.16:4910 , piece_id=PKF7JCLQKWL7DBRFMZQOTB2RURKDFJ6NMS7YB3AJYDJ2SC633JOQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.640310134 (530h52m27.511393169s) [noise=true , address=45.91.116.6:4974 , piece_id=VDRCSP2C5VIHZR6YLLR7WUGURUZOXC2G7MT56IE43PMVVXDD3HZQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.671350691 (530h52m27.542433736s) [noise=true , address=45.91.116.5:4297 , piece_id=NVFVBNP6PWOJR3AT6ZB6LQWZ2JACMX5YK377GY2RE7GDM5FDPJUA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.671881432 (530h52m27.542964477s) [noise=true , address=45.91.116.13:4156 , piece_id=Y5TLN2KADJ7KUBGC6LFHRWSOSH3E2N2WLWM2O2X6O26YRL7LTDQA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.671891021 (530h52m27.542974056s) [seg_pos={0 0}, upload=465581]: Segment reached optimal threshold of 13 pieces. Dec 27 18:21:27.672486244 (530h52m27.543569279s) [noise=true , address=45.91.116.4:4939 , piece_id=EMKOBZFOAMID6S27D4UMCJL4JKCH2KGR4KPUKXG4P43BIJIIOGVA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: read tcp 188.166.225.170:39392->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 18:21:27.672494189 (530h52m27.543577234s) [noise=true , address=45.91.116.4:4939 , piece_id=EMKOBZFOAMID6S27D4UMCJL4JKCH2KGR4KPUKXG4P43BIJIIOGVA, seg_pos={0 0}, upload=465581]: Long tail cancellation due to optimal threshold Dec 27 18:21:27.672941320 (530h52m27.544024365s) [noise=true , address=45.91.116.14:4192 , piece_id=4FEHZOQCMKHKWPHV6DCOTEOP7IYE7LQ22PX2WLFZ47IXQZ3IS36Q, seg_pos={0 0}, upload=465581]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: read tcp 188.166.225.170:49760->45.91.116.14:4192: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 18:21:27.672949005 (530h52m27.54403204s) [noise=true , address=45.91.116.14:4192 , piece_id=4FEHZOQCMKHKWPHV6DCOTEOP7IYE7LQ22PX2WLFZ47IXQZ3IS36Q, seg_pos={0 0}, upload=465581]: Long tail cancellation due to optimal threshold Dec 27 18:21:27.673176889 (530h52m27.544259924s) [noise=true , address=45.91.116.8:4046 , piece_id=6JWU3XRTODVGYUZBLZTMJJZFKYNTAHXGTJYQ2R7KXQKUBR7QSVKA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 27 18:21:27.673184854 (530h52m27.544267899s) [noise=true , address=45.91.116.8:4046 , piece_id=6JWU3XRTODVGYUZBLZTMJJZFKYNTAHXGTJYQ2R7KXQKUBR7QSVKA, seg_pos={0 0}, upload=465581]: Long tail cancellation due to optimal threshold Dec 27 18:21:27.673203058 (530h52m27.544286093s) [seg_pos={0 0}, upload=465581]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 18:21:27.673307677 (530h52m27.544390713s) [upload=465581]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 18:21:27.673312197 (530h52m27.544395232s) [upload=465581]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 18:21:27.963559300 (530h52m27.834642355s) [upload=465581]: Done uploading segments... Dec 27 18:21:27.963583115 (530h52m27.83466616s) [upload=465581]: Upload finished. err: <nil> Duration: 5.650146401s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [X.X....XXX..XX....X...X.X........X.......X..X.XXXXXXX...X...] duration: 6.0081466s trace: https://jaeger.dev.storj.io/trace/4981aef9a433490 FAIL (2025-12-27 18:23:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: VDRCSP2C5VIHZR6YLLR7WUGURUZOXC2G7MT56IE43PMVVXDD3HZQ) context deadline exceeded; piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: NVFVBNP6PWOJR3AT6ZB6LQWZ2JACMX5YK377GY2RE7GDM5FDPJUA) context deadline exceeded ======LOGS====== Dec 27 18:21:25.144885992 (530h52m25.015969028s) [upload=465581]: Starting upload Dec 27 18:21:25.144948482 (530h52m25.016031517s) [upload=465581]: Done starting upload Dec 27 18:21:25.145039054 (530h52m25.016122089s) [upload=465581]: Uploading segments... Dec 27 18:21:25.145258050 (530h52m25.016341095s) [upload=465581]: Got next segment. Inline: false Dec 27 18:21:25.145296154 (530h52m25.016379189s) [upload=465581]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 18:21:25.145301123 (530h52m25.016384158s) [upload=465581]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 18:21:25.389658082 (530h52m25.260741127s) [seg_pos={0 0}, upload=465581]: Begin upload segment... Dec 27 18:21:25.389719219 (530h52m25.260802264s) [seg_pos={0 0}, upload=465581]: Done begin upload segment. Dec 27 18:21:25.389751040 (530h52m25.260834095s) [noise=true , address=145.79.227.16:4910 , piece_id=PKF7JCLQKWL7DBRFMZQOTB2RURKDFJ6NMS7YB3AJYDJ2SC633JOQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.389827786 (530h52m25.260910821s) [noise=true , address=45.91.116.6:4974 , piece_id=VDRCSP2C5VIHZR6YLLR7WUGURUZOXC2G7MT56IE43PMVVXDD3HZQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.389925111 (530h52m25.261008146s) [noise=true , address=145.79.227.46:4602 , piece_id=OGGOAL326XC6BS7ULRPAM322APXHYL6FPWWVK32LHVANMFGYJWWQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.390583174 (530h52m25.261666239s) [noise=true , address=145.79.227.11:4732 , piece_id=WVWGLOP2LWAJXKXXMVSIXEXUMPYLU5UPROXM4XFGCDWYVBXTEKOA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.391844145 (530h52m25.26292719s) [noise=true , address=145.79.227.2:4113 , piece_id=B3FPIHAGBGSJN2I6BFIO6S2YPXSIHKF7ZBODNV6FGOI7DQRYTIQA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.392261891 (530h52m25.263344936s) [noise=true , address=145.79.227.4:4185 , piece_id=BSDWYICA7RMLTXX4URC3O6YC25FHVQ6ZRYL7OS2227KE3DUSWJDQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.392656372 (530h52m25.263739407s) [noise=true , address=45.91.116.13:4156 , piece_id=Y5TLN2KADJ7KUBGC6LFHRWSOSH3E2N2WLWM2O2X6O26YRL7LTDQA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.393053067 (530h52m25.264136112s) [noise=true , address=45.91.116.4:4939 , piece_id=EMKOBZFOAMID6S27D4UMCJL4JKCH2KGR4KPUKXG4P43BIJIIOGVA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.393516079 (530h52m25.264599114s) [noise=true , address=45.91.116.14:4192 , piece_id=4FEHZOQCMKHKWPHV6DCOTEOP7IYE7LQ22PX2WLFZ47IXQZ3IS36Q, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.394785246 (530h52m25.265868291s) [noise=true , address=145.79.227.10:4403 , piece_id=IZP3UFQYXDPVJQIMBMP6LWWZ6TR52FN755X7VYCFTLREMHOIJN3A, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.395302331 (530h52m25.266385376s) [noise=true , address=45.91.116.5:4297 , piece_id=NVFVBNP6PWOJR3AT6ZB6LQWZ2JACMX5YK377GY2RE7GDM5FDPJUA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.396222633 (530h52m25.267305678s) [noise=true , address=45.91.116.8:4046 , piece_id=6JWU3XRTODVGYUZBLZTMJJZFKYNTAHXGTJYQ2R7KXQKUBR7QSVKA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.397111025 (530h52m25.26819407s) [noise=true , address=145.79.227.17:4971 , piece_id=UO6L7APQZ46TBPJ5XOUHI2YL3Y5YZ2VAYFAZHOY7CL5C6BEO5G7Q, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.398240415 (530h52m25.26932346s) [noise=true , address=145.79.227.36:4517 , piece_id=RLIQWAGNKI5X6WGHYMN7FXPHWDYEVGWXLFMPN7IEZR4ZRDARFIBQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.402100956 (530h52m25.273184001s) [noise=true , address=145.79.227.48:4301 , piece_id=KKULR64B2HUQ7P4GSUX2XKQKW4DB3GVTD3PDJUOUPU7TOQ6ZKTJA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.405336728 (530h52m25.276419773s) [noise=true , address=145.79.227.45:4231 , piece_id=UX45A3K3LKYEG4YU7BVZKMJWYVUL47C5MMNQRSI32WE4Y62ILAOQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:27.033083485 (530h52m26.90416653s) [upload=465581]: Next returned nil segment Dec 27 18:21:27.033095617 (530h52m26.904178662s) [upload=465581]: Waiting for error group managing segments... Dec 27 18:21:27.422573865 (530h52m27.2936569s) [noise=true , address=145.79.227.11:4732 , piece_id=WVWGLOP2LWAJXKXXMVSIXEXUMPYLU5UPROXM4XFGCDWYVBXTEKOA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.425425164 (530h52m27.296508199s) [noise=true , address=145.79.227.46:4602 , piece_id=OGGOAL326XC6BS7ULRPAM322APXHYL6FPWWVK32LHVANMFGYJWWQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.442445785 (530h52m27.313528821s) [noise=true , address=145.79.227.45:4231 , piece_id=UX45A3K3LKYEG4YU7BVZKMJWYVUL47C5MMNQRSI32WE4Y62ILAOQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.449930007 (530h52m27.321013062s) [noise=true , address=145.79.227.17:4971 , piece_id=UO6L7APQZ46TBPJ5XOUHI2YL3Y5YZ2VAYFAZHOY7CL5C6BEO5G7Q, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.450121371 (530h52m27.321204406s) [noise=true , address=145.79.227.10:4403 , piece_id=IZP3UFQYXDPVJQIMBMP6LWWZ6TR52FN755X7VYCFTLREMHOIJN3A, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.454633623 (530h52m27.325716658s) [noise=true , address=145.79.227.48:4301 , piece_id=KKULR64B2HUQ7P4GSUX2XKQKW4DB3GVTD3PDJUOUPU7TOQ6ZKTJA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.455522034 (530h52m27.326605079s) [noise=true , address=145.79.227.2:4113 , piece_id=B3FPIHAGBGSJN2I6BFIO6S2YPXSIHKF7ZBODNV6FGOI7DQRYTIQA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.456210736 (530h52m27.327293781s) [noise=true , address=145.79.227.4:4185 , piece_id=BSDWYICA7RMLTXX4URC3O6YC25FHVQ6ZRYL7OS2227KE3DUSWJDQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.462096273 (530h52m27.333179308s) [noise=true , address=145.79.227.36:4517 , piece_id=RLIQWAGNKI5X6WGHYMN7FXPHWDYEVGWXLFMPN7IEZR4ZRDARFIBQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.509391311 (530h52m27.380474346s) [noise=true , address=145.79.227.16:4910 , piece_id=PKF7JCLQKWL7DBRFMZQOTB2RURKDFJ6NMS7YB3AJYDJ2SC633JOQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.640310134 (530h52m27.511393169s) [noise=true , address=45.91.116.6:4974 , piece_id=VDRCSP2C5VIHZR6YLLR7WUGURUZOXC2G7MT56IE43PMVVXDD3HZQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.671350691 (530h52m27.542433736s) [noise=true , address=45.91.116.5:4297 , piece_id=NVFVBNP6PWOJR3AT6ZB6LQWZ2JACMX5YK377GY2RE7GDM5FDPJUA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.671881432 (530h52m27.542964477s) [noise=true , address=45.91.116.13:4156 , piece_id=Y5TLN2KADJ7KUBGC6LFHRWSOSH3E2N2WLWM2O2X6O26YRL7LTDQA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.671891021 (530h52m27.542974056s) [seg_pos={0 0}, upload=465581]: Segment reached optimal threshold of 13 pieces. Dec 27 18:21:27.672486244 (530h52m27.543569279s) [noise=true , address=45.91.116.4:4939 , piece_id=EMKOBZFOAMID6S27D4UMCJL4JKCH2KGR4KPUKXG4P43BIJIIOGVA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: read tcp 188.166.225.170:39392->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 18:21:27.672494189 (530h52m27.543577234s) [noise=true , address=45.91.116.4:4939 , piece_id=EMKOBZFOAMID6S27D4UMCJL4JKCH2KGR4KPUKXG4P43BIJIIOGVA, seg_pos={0 0}, upload=465581]: Long tail cancellation due to optimal threshold Dec 27 18:21:27.672941320 (530h52m27.544024365s) [noise=true , address=45.91.116.14:4192 , piece_id=4FEHZOQCMKHKWPHV6DCOTEOP7IYE7LQ22PX2WLFZ47IXQZ3IS36Q, seg_pos={0 0}, upload=465581]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: read tcp 188.166.225.170:49760->45.91.116.14:4192: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 18:21:27.672949005 (530h52m27.54403204s) [noise=true , address=45.91.116.14:4192 , piece_id=4FEHZOQCMKHKWPHV6DCOTEOP7IYE7LQ22PX2WLFZ47IXQZ3IS36Q, seg_pos={0 0}, upload=465581]: Long tail cancellation due to optimal threshold Dec 27 18:21:27.673176889 (530h52m27.544259924s) [noise=true , address=45.91.116.8:4046 , piece_id=6JWU3XRTODVGYUZBLZTMJJZFKYNTAHXGTJYQ2R7KXQKUBR7QSVKA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 27 18:21:27.673184854 (530h52m27.544267899s) [noise=true , address=45.91.116.8:4046 , piece_id=6JWU3XRTODVGYUZBLZTMJJZFKYNTAHXGTJYQ2R7KXQKUBR7QSVKA, seg_pos={0 0}, upload=465581]: Long tail cancellation due to optimal threshold Dec 27 18:21:27.673203058 (530h52m27.544286093s) [seg_pos={0 0}, upload=465581]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 18:21:27.673307677 (530h52m27.544390713s) [upload=465581]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 18:21:27.673312197 (530h52m27.544395232s) [upload=465581]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 18:21:27.963559300 (530h52m27.834642355s) [upload=465581]: Done uploading segments... Dec 27 18:21:27.963583115 (530h52m27.83466616s) [upload=465581]: Upload finished. err: <nil> Duration: 6.0081737s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......X.....................................................] duration: 3.539940973s trace: https://jaeger.dev.storj.io/trace/3bddfef0dfa96557 FAIL (2025-12-27 18:23:24 UTC) orbiter warning: context deadline exceeded Duration: 3.539969597s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [XX.X....XXX..XX....X...X.X........X.......X..X.XXXXXXX...X..] duration: 5.949160817s trace: https://jaeger.dev.storj.io/trace/4981aef9a433490 FAIL (2025-12-27 18:23:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: VDRCSP2C5VIHZR6YLLR7WUGURUZOXC2G7MT56IE43PMVVXDD3HZQ) context deadline exceeded; piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: NVFVBNP6PWOJR3AT6ZB6LQWZ2JACMX5YK377GY2RE7GDM5FDPJUA) context deadline exceeded ======LOGS====== Dec 27 18:21:25.144885992 (530h52m25.015969028s) [upload=465581]: Starting upload Dec 27 18:21:25.144948482 (530h52m25.016031517s) [upload=465581]: Done starting upload Dec 27 18:21:25.145039054 (530h52m25.016122089s) [upload=465581]: Uploading segments... Dec 27 18:21:25.145258050 (530h52m25.016341095s) [upload=465581]: Got next segment. Inline: false Dec 27 18:21:25.145296154 (530h52m25.016379189s) [upload=465581]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 18:21:25.145301123 (530h52m25.016384158s) [upload=465581]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 18:21:25.389658082 (530h52m25.260741127s) [seg_pos={0 0}, upload=465581]: Begin upload segment... Dec 27 18:21:25.389719219 (530h52m25.260802264s) [seg_pos={0 0}, upload=465581]: Done begin upload segment. Dec 27 18:21:25.389751040 (530h52m25.260834095s) [noise=true , address=145.79.227.16:4910 , piece_id=PKF7JCLQKWL7DBRFMZQOTB2RURKDFJ6NMS7YB3AJYDJ2SC633JOQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.389827786 (530h52m25.260910821s) [noise=true , address=45.91.116.6:4974 , piece_id=VDRCSP2C5VIHZR6YLLR7WUGURUZOXC2G7MT56IE43PMVVXDD3HZQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.389925111 (530h52m25.261008146s) [noise=true , address=145.79.227.46:4602 , piece_id=OGGOAL326XC6BS7ULRPAM322APXHYL6FPWWVK32LHVANMFGYJWWQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.390583174 (530h52m25.261666239s) [noise=true , address=145.79.227.11:4732 , piece_id=WVWGLOP2LWAJXKXXMVSIXEXUMPYLU5UPROXM4XFGCDWYVBXTEKOA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.391844145 (530h52m25.26292719s) [noise=true , address=145.79.227.2:4113 , piece_id=B3FPIHAGBGSJN2I6BFIO6S2YPXSIHKF7ZBODNV6FGOI7DQRYTIQA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.392261891 (530h52m25.263344936s) [noise=true , address=145.79.227.4:4185 , piece_id=BSDWYICA7RMLTXX4URC3O6YC25FHVQ6ZRYL7OS2227KE3DUSWJDQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.392656372 (530h52m25.263739407s) [noise=true , address=45.91.116.13:4156 , piece_id=Y5TLN2KADJ7KUBGC6LFHRWSOSH3E2N2WLWM2O2X6O26YRL7LTDQA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.393053067 (530h52m25.264136112s) [noise=true , address=45.91.116.4:4939 , piece_id=EMKOBZFOAMID6S27D4UMCJL4JKCH2KGR4KPUKXG4P43BIJIIOGVA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.393516079 (530h52m25.264599114s) [noise=true , address=45.91.116.14:4192 , piece_id=4FEHZOQCMKHKWPHV6DCOTEOP7IYE7LQ22PX2WLFZ47IXQZ3IS36Q, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.394785246 (530h52m25.265868291s) [noise=true , address=145.79.227.10:4403 , piece_id=IZP3UFQYXDPVJQIMBMP6LWWZ6TR52FN755X7VYCFTLREMHOIJN3A, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.395302331 (530h52m25.266385376s) [noise=true , address=45.91.116.5:4297 , piece_id=NVFVBNP6PWOJR3AT6ZB6LQWZ2JACMX5YK377GY2RE7GDM5FDPJUA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.396222633 (530h52m25.267305678s) [noise=true , address=45.91.116.8:4046 , piece_id=6JWU3XRTODVGYUZBLZTMJJZFKYNTAHXGTJYQ2R7KXQKUBR7QSVKA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.397111025 (530h52m25.26819407s) [noise=true , address=145.79.227.17:4971 , piece_id=UO6L7APQZ46TBPJ5XOUHI2YL3Y5YZ2VAYFAZHOY7CL5C6BEO5G7Q, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.398240415 (530h52m25.26932346s) [noise=true , address=145.79.227.36:4517 , piece_id=RLIQWAGNKI5X6WGHYMN7FXPHWDYEVGWXLFMPN7IEZR4ZRDARFIBQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.402100956 (530h52m25.273184001s) [noise=true , address=145.79.227.48:4301 , piece_id=KKULR64B2HUQ7P4GSUX2XKQKW4DB3GVTD3PDJUOUPU7TOQ6ZKTJA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.405336728 (530h52m25.276419773s) [noise=true , address=145.79.227.45:4231 , piece_id=UX45A3K3LKYEG4YU7BVZKMJWYVUL47C5MMNQRSI32WE4Y62ILAOQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:27.033083485 (530h52m26.90416653s) [upload=465581]: Next returned nil segment Dec 27 18:21:27.033095617 (530h52m26.904178662s) [upload=465581]: Waiting for error group managing segments... Dec 27 18:21:27.422573865 (530h52m27.2936569s) [noise=true , address=145.79.227.11:4732 , piece_id=WVWGLOP2LWAJXKXXMVSIXEXUMPYLU5UPROXM4XFGCDWYVBXTEKOA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.425425164 (530h52m27.296508199s) [noise=true , address=145.79.227.46:4602 , piece_id=OGGOAL326XC6BS7ULRPAM322APXHYL6FPWWVK32LHVANMFGYJWWQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.442445785 (530h52m27.313528821s) [noise=true , address=145.79.227.45:4231 , piece_id=UX45A3K3LKYEG4YU7BVZKMJWYVUL47C5MMNQRSI32WE4Y62ILAOQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.449930007 (530h52m27.321013062s) [noise=true , address=145.79.227.17:4971 , piece_id=UO6L7APQZ46TBPJ5XOUHI2YL3Y5YZ2VAYFAZHOY7CL5C6BEO5G7Q, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.450121371 (530h52m27.321204406s) [noise=true , address=145.79.227.10:4403 , piece_id=IZP3UFQYXDPVJQIMBMP6LWWZ6TR52FN755X7VYCFTLREMHOIJN3A, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.454633623 (530h52m27.325716658s) [noise=true , address=145.79.227.48:4301 , piece_id=KKULR64B2HUQ7P4GSUX2XKQKW4DB3GVTD3PDJUOUPU7TOQ6ZKTJA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.455522034 (530h52m27.326605079s) [noise=true , address=145.79.227.2:4113 , piece_id=B3FPIHAGBGSJN2I6BFIO6S2YPXSIHKF7ZBODNV6FGOI7DQRYTIQA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.456210736 (530h52m27.327293781s) [noise=true , address=145.79.227.4:4185 , piece_id=BSDWYICA7RMLTXX4URC3O6YC25FHVQ6ZRYL7OS2227KE3DUSWJDQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.462096273 (530h52m27.333179308s) [noise=true , address=145.79.227.36:4517 , piece_id=RLIQWAGNKI5X6WGHYMN7FXPHWDYEVGWXLFMPN7IEZR4ZRDARFIBQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.509391311 (530h52m27.380474346s) [noise=true , address=145.79.227.16:4910 , piece_id=PKF7JCLQKWL7DBRFMZQOTB2RURKDFJ6NMS7YB3AJYDJ2SC633JOQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.640310134 (530h52m27.511393169s) [noise=true , address=45.91.116.6:4974 , piece_id=VDRCSP2C5VIHZR6YLLR7WUGURUZOXC2G7MT56IE43PMVVXDD3HZQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.671350691 (530h52m27.542433736s) [noise=true , address=45.91.116.5:4297 , piece_id=NVFVBNP6PWOJR3AT6ZB6LQWZ2JACMX5YK377GY2RE7GDM5FDPJUA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.671881432 (530h52m27.542964477s) [noise=true , address=45.91.116.13:4156 , piece_id=Y5TLN2KADJ7KUBGC6LFHRWSOSH3E2N2WLWM2O2X6O26YRL7LTDQA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.671891021 (530h52m27.542974056s) [seg_pos={0 0}, upload=465581]: Segment reached optimal threshold of 13 pieces. Dec 27 18:21:27.672486244 (530h52m27.543569279s) [noise=true , address=45.91.116.4:4939 , piece_id=EMKOBZFOAMID6S27D4UMCJL4JKCH2KGR4KPUKXG4P43BIJIIOGVA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: read tcp 188.166.225.170:39392->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 18:21:27.672494189 (530h52m27.543577234s) [noise=true , address=45.91.116.4:4939 , piece_id=EMKOBZFOAMID6S27D4UMCJL4JKCH2KGR4KPUKXG4P43BIJIIOGVA, seg_pos={0 0}, upload=465581]: Long tail cancellation due to optimal threshold Dec 27 18:21:27.672941320 (530h52m27.544024365s) [noise=true , address=45.91.116.14:4192 , piece_id=4FEHZOQCMKHKWPHV6DCOTEOP7IYE7LQ22PX2WLFZ47IXQZ3IS36Q, seg_pos={0 0}, upload=465581]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: read tcp 188.166.225.170:49760->45.91.116.14:4192: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 18:21:27.672949005 (530h52m27.54403204s) [noise=true , address=45.91.116.14:4192 , piece_id=4FEHZOQCMKHKWPHV6DCOTEOP7IYE7LQ22PX2WLFZ47IXQZ3IS36Q, seg_pos={0 0}, upload=465581]: Long tail cancellation due to optimal threshold Dec 27 18:21:27.673176889 (530h52m27.544259924s) [noise=true , address=45.91.116.8:4046 , piece_id=6JWU3XRTODVGYUZBLZTMJJZFKYNTAHXGTJYQ2R7KXQKUBR7QSVKA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 27 18:21:27.673184854 (530h52m27.544267899s) [noise=true , address=45.91.116.8:4046 , piece_id=6JWU3XRTODVGYUZBLZTMJJZFKYNTAHXGTJYQ2R7KXQKUBR7QSVKA, seg_pos={0 0}, upload=465581]: Long tail cancellation due to optimal threshold Dec 27 18:21:27.673203058 (530h52m27.544286093s) [seg_pos={0 0}, upload=465581]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 18:21:27.673307677 (530h52m27.544390713s) [upload=465581]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 18:21:27.673312197 (530h52m27.544395232s) [upload=465581]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 18:21:27.963559300 (530h52m27.834642355s) [upload=465581]: Done uploading segments... Dec 27 18:21:27.963583115 (530h52m27.83466616s) [upload=465581]: Upload finished. err: <nil> Duration: 5.949187538s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......X....................................................] duration: 3.587124239s trace: https://jaeger.dev.storj.io/trace/3bddfef0dfa96557 FAIL (2025-12-27 18:22:39 UTC) orbiter warning: context deadline exceeded Duration: 3.587166559s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [XXX.X....XXX..XX....X...X.X........X.......X..X.XXXXXXX...X.] duration: 6.503257439s trace: https://jaeger.dev.storj.io/trace/4981aef9a433490 FAIL (2025-12-27 18:22:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: VDRCSP2C5VIHZR6YLLR7WUGURUZOXC2G7MT56IE43PMVVXDD3HZQ) context deadline exceeded; piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: NVFVBNP6PWOJR3AT6ZB6LQWZ2JACMX5YK377GY2RE7GDM5FDPJUA) context deadline exceeded ======LOGS====== Dec 27 18:21:25.144885992 (530h52m25.015969028s) [upload=465581]: Starting upload Dec 27 18:21:25.144948482 (530h52m25.016031517s) [upload=465581]: Done starting upload Dec 27 18:21:25.145039054 (530h52m25.016122089s) [upload=465581]: Uploading segments... Dec 27 18:21:25.145258050 (530h52m25.016341095s) [upload=465581]: Got next segment. Inline: false Dec 27 18:21:25.145296154 (530h52m25.016379189s) [upload=465581]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 18:21:25.145301123 (530h52m25.016384158s) [upload=465581]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 18:21:25.389658082 (530h52m25.260741127s) [seg_pos={0 0}, upload=465581]: Begin upload segment... Dec 27 18:21:25.389719219 (530h52m25.260802264s) [seg_pos={0 0}, upload=465581]: Done begin upload segment. Dec 27 18:21:25.389751040 (530h52m25.260834095s) [noise=true , address=145.79.227.16:4910 , piece_id=PKF7JCLQKWL7DBRFMZQOTB2RURKDFJ6NMS7YB3AJYDJ2SC633JOQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.389827786 (530h52m25.260910821s) [noise=true , address=45.91.116.6:4974 , piece_id=VDRCSP2C5VIHZR6YLLR7WUGURUZOXC2G7MT56IE43PMVVXDD3HZQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.389925111 (530h52m25.261008146s) [noise=true , address=145.79.227.46:4602 , piece_id=OGGOAL326XC6BS7ULRPAM322APXHYL6FPWWVK32LHVANMFGYJWWQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.390583174 (530h52m25.261666239s) [noise=true , address=145.79.227.11:4732 , piece_id=WVWGLOP2LWAJXKXXMVSIXEXUMPYLU5UPROXM4XFGCDWYVBXTEKOA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.391844145 (530h52m25.26292719s) [noise=true , address=145.79.227.2:4113 , piece_id=B3FPIHAGBGSJN2I6BFIO6S2YPXSIHKF7ZBODNV6FGOI7DQRYTIQA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.392261891 (530h52m25.263344936s) [noise=true , address=145.79.227.4:4185 , piece_id=BSDWYICA7RMLTXX4URC3O6YC25FHVQ6ZRYL7OS2227KE3DUSWJDQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.392656372 (530h52m25.263739407s) [noise=true , address=45.91.116.13:4156 , piece_id=Y5TLN2KADJ7KUBGC6LFHRWSOSH3E2N2WLWM2O2X6O26YRL7LTDQA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.393053067 (530h52m25.264136112s) [noise=true , address=45.91.116.4:4939 , piece_id=EMKOBZFOAMID6S27D4UMCJL4JKCH2KGR4KPUKXG4P43BIJIIOGVA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.393516079 (530h52m25.264599114s) [noise=true , address=45.91.116.14:4192 , piece_id=4FEHZOQCMKHKWPHV6DCOTEOP7IYE7LQ22PX2WLFZ47IXQZ3IS36Q, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.394785246 (530h52m25.265868291s) [noise=true , address=145.79.227.10:4403 , piece_id=IZP3UFQYXDPVJQIMBMP6LWWZ6TR52FN755X7VYCFTLREMHOIJN3A, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.395302331 (530h52m25.266385376s) [noise=true , address=45.91.116.5:4297 , piece_id=NVFVBNP6PWOJR3AT6ZB6LQWZ2JACMX5YK377GY2RE7GDM5FDPJUA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.396222633 (530h52m25.267305678s) [noise=true , address=45.91.116.8:4046 , piece_id=6JWU3XRTODVGYUZBLZTMJJZFKYNTAHXGTJYQ2R7KXQKUBR7QSVKA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.397111025 (530h52m25.26819407s) [noise=true , address=145.79.227.17:4971 , piece_id=UO6L7APQZ46TBPJ5XOUHI2YL3Y5YZ2VAYFAZHOY7CL5C6BEO5G7Q, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.398240415 (530h52m25.26932346s) [noise=true , address=145.79.227.36:4517 , piece_id=RLIQWAGNKI5X6WGHYMN7FXPHWDYEVGWXLFMPN7IEZR4ZRDARFIBQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.402100956 (530h52m25.273184001s) [noise=true , address=145.79.227.48:4301 , piece_id=KKULR64B2HUQ7P4GSUX2XKQKW4DB3GVTD3PDJUOUPU7TOQ6ZKTJA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.405336728 (530h52m25.276419773s) [noise=true , address=145.79.227.45:4231 , piece_id=UX45A3K3LKYEG4YU7BVZKMJWYVUL47C5MMNQRSI32WE4Y62ILAOQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:27.033083485 (530h52m26.90416653s) [upload=465581]: Next returned nil segment Dec 27 18:21:27.033095617 (530h52m26.904178662s) [upload=465581]: Waiting for error group managing segments... Dec 27 18:21:27.422573865 (530h52m27.2936569s) [noise=true , address=145.79.227.11:4732 , piece_id=WVWGLOP2LWAJXKXXMVSIXEXUMPYLU5UPROXM4XFGCDWYVBXTEKOA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.425425164 (530h52m27.296508199s) [noise=true , address=145.79.227.46:4602 , piece_id=OGGOAL326XC6BS7ULRPAM322APXHYL6FPWWVK32LHVANMFGYJWWQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.442445785 (530h52m27.313528821s) [noise=true , address=145.79.227.45:4231 , piece_id=UX45A3K3LKYEG4YU7BVZKMJWYVUL47C5MMNQRSI32WE4Y62ILAOQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.449930007 (530h52m27.321013062s) [noise=true , address=145.79.227.17:4971 , piece_id=UO6L7APQZ46TBPJ5XOUHI2YL3Y5YZ2VAYFAZHOY7CL5C6BEO5G7Q, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.450121371 (530h52m27.321204406s) [noise=true , address=145.79.227.10:4403 , piece_id=IZP3UFQYXDPVJQIMBMP6LWWZ6TR52FN755X7VYCFTLREMHOIJN3A, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.454633623 (530h52m27.325716658s) [noise=true , address=145.79.227.48:4301 , piece_id=KKULR64B2HUQ7P4GSUX2XKQKW4DB3GVTD3PDJUOUPU7TOQ6ZKTJA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.455522034 (530h52m27.326605079s) [noise=true , address=145.79.227.2:4113 , piece_id=B3FPIHAGBGSJN2I6BFIO6S2YPXSIHKF7ZBODNV6FGOI7DQRYTIQA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.456210736 (530h52m27.327293781s) [noise=true , address=145.79.227.4:4185 , piece_id=BSDWYICA7RMLTXX4URC3O6YC25FHVQ6ZRYL7OS2227KE3DUSWJDQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.462096273 (530h52m27.333179308s) [noise=true , address=145.79.227.36:4517 , piece_id=RLIQWAGNKI5X6WGHYMN7FXPHWDYEVGWXLFMPN7IEZR4ZRDARFIBQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.509391311 (530h52m27.380474346s) [noise=true , address=145.79.227.16:4910 , piece_id=PKF7JCLQKWL7DBRFMZQOTB2RURKDFJ6NMS7YB3AJYDJ2SC633JOQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.640310134 (530h52m27.511393169s) [noise=true , address=45.91.116.6:4974 , piece_id=VDRCSP2C5VIHZR6YLLR7WUGURUZOXC2G7MT56IE43PMVVXDD3HZQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.671350691 (530h52m27.542433736s) [noise=true , address=45.91.116.5:4297 , piece_id=NVFVBNP6PWOJR3AT6ZB6LQWZ2JACMX5YK377GY2RE7GDM5FDPJUA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.671881432 (530h52m27.542964477s) [noise=true , address=45.91.116.13:4156 , piece_id=Y5TLN2KADJ7KUBGC6LFHRWSOSH3E2N2WLWM2O2X6O26YRL7LTDQA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.671891021 (530h52m27.542974056s) [seg_pos={0 0}, upload=465581]: Segment reached optimal threshold of 13 pieces. Dec 27 18:21:27.672486244 (530h52m27.543569279s) [noise=true , address=45.91.116.4:4939 , piece_id=EMKOBZFOAMID6S27D4UMCJL4JKCH2KGR4KPUKXG4P43BIJIIOGVA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: read tcp 188.166.225.170:39392->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 18:21:27.672494189 (530h52m27.543577234s) [noise=true , address=45.91.116.4:4939 , piece_id=EMKOBZFOAMID6S27D4UMCJL4JKCH2KGR4KPUKXG4P43BIJIIOGVA, seg_pos={0 0}, upload=465581]: Long tail cancellation due to optimal threshold Dec 27 18:21:27.672941320 (530h52m27.544024365s) [noise=true , address=45.91.116.14:4192 , piece_id=4FEHZOQCMKHKWPHV6DCOTEOP7IYE7LQ22PX2WLFZ47IXQZ3IS36Q, seg_pos={0 0}, upload=465581]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: read tcp 188.166.225.170:49760->45.91.116.14:4192: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 18:21:27.672949005 (530h52m27.54403204s) [noise=true , address=45.91.116.14:4192 , piece_id=4FEHZOQCMKHKWPHV6DCOTEOP7IYE7LQ22PX2WLFZ47IXQZ3IS36Q, seg_pos={0 0}, upload=465581]: Long tail cancellation due to optimal threshold Dec 27 18:21:27.673176889 (530h52m27.544259924s) [noise=true , address=45.91.116.8:4046 , piece_id=6JWU3XRTODVGYUZBLZTMJJZFKYNTAHXGTJYQ2R7KXQKUBR7QSVKA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 27 18:21:27.673184854 (530h52m27.544267899s) [noise=true , address=45.91.116.8:4046 , piece_id=6JWU3XRTODVGYUZBLZTMJJZFKYNTAHXGTJYQ2R7KXQKUBR7QSVKA, seg_pos={0 0}, upload=465581]: Long tail cancellation due to optimal threshold Dec 27 18:21:27.673203058 (530h52m27.544286093s) [seg_pos={0 0}, upload=465581]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 18:21:27.673307677 (530h52m27.544390713s) [upload=465581]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 18:21:27.673312197 (530h52m27.544395232s) [upload=465581]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 18:21:27.963559300 (530h52m27.834642355s) [upload=465581]: Done uploading segments... Dec 27 18:21:27.963583115 (530h52m27.83466616s) [upload=465581]: Upload finished. err: <nil> Duration: 6.503288949s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........X...................................................] duration: 3.588697074s trace: https://jaeger.dev.storj.io/trace/3bddfef0dfa96557 FAIL (2025-12-27 18:22:20 UTC) orbiter warning: context deadline exceeded Duration: 3.588731971s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........X..................................................] duration: 3.556671963s trace: https://jaeger.dev.storj.io/trace/3bddfef0dfa96557 FAIL (2025-12-27 18:21:56 UTC) orbiter warning: context deadline exceeded Duration: 3.556696328s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [XXXX.X....XXX..XX....X...X.X........X.......X..X.XXXXXXX...X] duration: 30.003264603s trace: https://jaeger.dev.storj.io/trace/4981aef9a433490 FAIL (2025-12-27 18:21:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: VDRCSP2C5VIHZR6YLLR7WUGURUZOXC2G7MT56IE43PMVVXDD3HZQ) context deadline exceeded; piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: NVFVBNP6PWOJR3AT6ZB6LQWZ2JACMX5YK377GY2RE7GDM5FDPJUA) context deadline exceeded ======LOGS====== Dec 27 18:21:25.144885992 (530h52m25.015969028s) [upload=465581]: Starting upload Dec 27 18:21:25.144948482 (530h52m25.016031517s) [upload=465581]: Done starting upload Dec 27 18:21:25.145039054 (530h52m25.016122089s) [upload=465581]: Uploading segments... Dec 27 18:21:25.145258050 (530h52m25.016341095s) [upload=465581]: Got next segment. Inline: false Dec 27 18:21:25.145296154 (530h52m25.016379189s) [upload=465581]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 18:21:25.145301123 (530h52m25.016384158s) [upload=465581]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 18:21:25.389658082 (530h52m25.260741127s) [seg_pos={0 0}, upload=465581]: Begin upload segment... Dec 27 18:21:25.389719219 (530h52m25.260802264s) [seg_pos={0 0}, upload=465581]: Done begin upload segment. Dec 27 18:21:25.389751040 (530h52m25.260834095s) [noise=true , address=145.79.227.16:4910 , piece_id=PKF7JCLQKWL7DBRFMZQOTB2RURKDFJ6NMS7YB3AJYDJ2SC633JOQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.389827786 (530h52m25.260910821s) [noise=true , address=45.91.116.6:4974 , piece_id=VDRCSP2C5VIHZR6YLLR7WUGURUZOXC2G7MT56IE43PMVVXDD3HZQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.389925111 (530h52m25.261008146s) [noise=true , address=145.79.227.46:4602 , piece_id=OGGOAL326XC6BS7ULRPAM322APXHYL6FPWWVK32LHVANMFGYJWWQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.390583174 (530h52m25.261666239s) [noise=true , address=145.79.227.11:4732 , piece_id=WVWGLOP2LWAJXKXXMVSIXEXUMPYLU5UPROXM4XFGCDWYVBXTEKOA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.391844145 (530h52m25.26292719s) [noise=true , address=145.79.227.2:4113 , piece_id=B3FPIHAGBGSJN2I6BFIO6S2YPXSIHKF7ZBODNV6FGOI7DQRYTIQA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.392261891 (530h52m25.263344936s) [noise=true , address=145.79.227.4:4185 , piece_id=BSDWYICA7RMLTXX4URC3O6YC25FHVQ6ZRYL7OS2227KE3DUSWJDQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.392656372 (530h52m25.263739407s) [noise=true , address=45.91.116.13:4156 , piece_id=Y5TLN2KADJ7KUBGC6LFHRWSOSH3E2N2WLWM2O2X6O26YRL7LTDQA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.393053067 (530h52m25.264136112s) [noise=true , address=45.91.116.4:4939 , piece_id=EMKOBZFOAMID6S27D4UMCJL4JKCH2KGR4KPUKXG4P43BIJIIOGVA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.393516079 (530h52m25.264599114s) [noise=true , address=45.91.116.14:4192 , piece_id=4FEHZOQCMKHKWPHV6DCOTEOP7IYE7LQ22PX2WLFZ47IXQZ3IS36Q, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.394785246 (530h52m25.265868291s) [noise=true , address=145.79.227.10:4403 , piece_id=IZP3UFQYXDPVJQIMBMP6LWWZ6TR52FN755X7VYCFTLREMHOIJN3A, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.395302331 (530h52m25.266385376s) [noise=true , address=45.91.116.5:4297 , piece_id=NVFVBNP6PWOJR3AT6ZB6LQWZ2JACMX5YK377GY2RE7GDM5FDPJUA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.396222633 (530h52m25.267305678s) [noise=true , address=45.91.116.8:4046 , piece_id=6JWU3XRTODVGYUZBLZTMJJZFKYNTAHXGTJYQ2R7KXQKUBR7QSVKA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.397111025 (530h52m25.26819407s) [noise=true , address=145.79.227.17:4971 , piece_id=UO6L7APQZ46TBPJ5XOUHI2YL3Y5YZ2VAYFAZHOY7CL5C6BEO5G7Q, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.398240415 (530h52m25.26932346s) [noise=true , address=145.79.227.36:4517 , piece_id=RLIQWAGNKI5X6WGHYMN7FXPHWDYEVGWXLFMPN7IEZR4ZRDARFIBQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.402100956 (530h52m25.273184001s) [noise=true , address=145.79.227.48:4301 , piece_id=KKULR64B2HUQ7P4GSUX2XKQKW4DB3GVTD3PDJUOUPU7TOQ6ZKTJA, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:25.405336728 (530h52m25.276419773s) [noise=true , address=145.79.227.45:4231 , piece_id=UX45A3K3LKYEG4YU7BVZKMJWYVUL47C5MMNQRSI32WE4Y62ILAOQ, seg_pos={0 0}, upload=465581]: Uploading piece... Dec 27 18:21:27.033083485 (530h52m26.90416653s) [upload=465581]: Next returned nil segment Dec 27 18:21:27.033095617 (530h52m26.904178662s) [upload=465581]: Waiting for error group managing segments... Dec 27 18:21:27.422573865 (530h52m27.2936569s) [noise=true , address=145.79.227.11:4732 , piece_id=WVWGLOP2LWAJXKXXMVSIXEXUMPYLU5UPROXM4XFGCDWYVBXTEKOA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.425425164 (530h52m27.296508199s) [noise=true , address=145.79.227.46:4602 , piece_id=OGGOAL326XC6BS7ULRPAM322APXHYL6FPWWVK32LHVANMFGYJWWQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.442445785 (530h52m27.313528821s) [noise=true , address=145.79.227.45:4231 , piece_id=UX45A3K3LKYEG4YU7BVZKMJWYVUL47C5MMNQRSI32WE4Y62ILAOQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.449930007 (530h52m27.321013062s) [noise=true , address=145.79.227.17:4971 , piece_id=UO6L7APQZ46TBPJ5XOUHI2YL3Y5YZ2VAYFAZHOY7CL5C6BEO5G7Q, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.450121371 (530h52m27.321204406s) [noise=true , address=145.79.227.10:4403 , piece_id=IZP3UFQYXDPVJQIMBMP6LWWZ6TR52FN755X7VYCFTLREMHOIJN3A, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.454633623 (530h52m27.325716658s) [noise=true , address=145.79.227.48:4301 , piece_id=KKULR64B2HUQ7P4GSUX2XKQKW4DB3GVTD3PDJUOUPU7TOQ6ZKTJA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.455522034 (530h52m27.326605079s) [noise=true , address=145.79.227.2:4113 , piece_id=B3FPIHAGBGSJN2I6BFIO6S2YPXSIHKF7ZBODNV6FGOI7DQRYTIQA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.456210736 (530h52m27.327293781s) [noise=true , address=145.79.227.4:4185 , piece_id=BSDWYICA7RMLTXX4URC3O6YC25FHVQ6ZRYL7OS2227KE3DUSWJDQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.462096273 (530h52m27.333179308s) [noise=true , address=145.79.227.36:4517 , piece_id=RLIQWAGNKI5X6WGHYMN7FXPHWDYEVGWXLFMPN7IEZR4ZRDARFIBQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.509391311 (530h52m27.380474346s) [noise=true , address=145.79.227.16:4910 , piece_id=PKF7JCLQKWL7DBRFMZQOTB2RURKDFJ6NMS7YB3AJYDJ2SC633JOQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.640310134 (530h52m27.511393169s) [noise=true , address=45.91.116.6:4974 , piece_id=VDRCSP2C5VIHZR6YLLR7WUGURUZOXC2G7MT56IE43PMVVXDD3HZQ, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.671350691 (530h52m27.542433736s) [noise=true , address=45.91.116.5:4297 , piece_id=NVFVBNP6PWOJR3AT6ZB6LQWZ2JACMX5YK377GY2RE7GDM5FDPJUA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.671881432 (530h52m27.542964477s) [noise=true , address=45.91.116.13:4156 , piece_id=Y5TLN2KADJ7KUBGC6LFHRWSOSH3E2N2WLWM2O2X6O26YRL7LTDQA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: <nil> Dec 27 18:21:27.671891021 (530h52m27.542974056s) [seg_pos={0 0}, upload=465581]: Segment reached optimal threshold of 13 pieces. Dec 27 18:21:27.672486244 (530h52m27.543569279s) [noise=true , address=45.91.116.4:4939 , piece_id=EMKOBZFOAMID6S27D4UMCJL4JKCH2KGR4KPUKXG4P43BIJIIOGVA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: read tcp 188.166.225.170:39392->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 18:21:27.672494189 (530h52m27.543577234s) [noise=true , address=45.91.116.4:4939 , piece_id=EMKOBZFOAMID6S27D4UMCJL4JKCH2KGR4KPUKXG4P43BIJIIOGVA, seg_pos={0 0}, upload=465581]: Long tail cancellation due to optimal threshold Dec 27 18:21:27.672941320 (530h52m27.544024365s) [noise=true , address=45.91.116.14:4192 , piece_id=4FEHZOQCMKHKWPHV6DCOTEOP7IYE7LQ22PX2WLFZ47IXQZ3IS36Q, seg_pos={0 0}, upload=465581]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: read tcp 188.166.225.170:49760->45.91.116.14:4192: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 18:21:27.672949005 (530h52m27.54403204s) [noise=true , address=45.91.116.14:4192 , piece_id=4FEHZOQCMKHKWPHV6DCOTEOP7IYE7LQ22PX2WLFZ47IXQZ3IS36Q, seg_pos={0 0}, upload=465581]: Long tail cancellation due to optimal threshold Dec 27 18:21:27.673176889 (530h52m27.544259924s) [noise=true , address=45.91.116.8:4046 , piece_id=6JWU3XRTODVGYUZBLZTMJJZFKYNTAHXGTJYQ2R7KXQKUBR7QSVKA, seg_pos={0 0}, upload=465581]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 27 18:21:27.673184854 (530h52m27.544267899s) [noise=true , address=45.91.116.8:4046 , piece_id=6JWU3XRTODVGYUZBLZTMJJZFKYNTAHXGTJYQ2R7KXQKUBR7QSVKA, seg_pos={0 0}, upload=465581]: Long tail cancellation due to optimal threshold Dec 27 18:21:27.673203058 (530h52m27.544286093s) [seg_pos={0 0}, upload=465581]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 18:21:27.673307677 (530h52m27.544390713s) [upload=465581]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 18:21:27.673312197 (530h52m27.544395232s) [upload=465581]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 18:21:27.963559300 (530h52m27.834642355s) [upload=465581]: Done uploading segments... Dec 27 18:21:27.963583115 (530h52m27.83466616s) [upload=465581]: Upload finished. err: <nil> Duration: 30.0032896s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [.XXXX.X....XXX..XX....X...X.X........X.......X..X.XXXXXXX...] duration: 5.663010263s trace: https://jaeger.dev.storj.io/trace/6aea6e6d013c1516 FAIL (2025-12-27 18:21:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: EUF7QLG6CJGERWWATE2N7QGNETUZYWDTFOC6LQVXUFQLKZKZSH5Q) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: 6KKONXYFBLEKXTJMNQ452QDLMGOFTGOML2W77L7AOCW6IB5D3ELQ) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: 57WUY7F7ERDXIXGF5LLR5V5SD3CFEMZHEA4TZY7N2PVHWORCO2GQ) context deadline exceeded ======LOGS====== g segments... Dec 27 18:18:48.727318184 (530h49m48.598401219s) [upload=465546]: Got next segment. Inline: false Dec 27 18:18:48.727380142 (530h49m48.598463187s) [upload=465546]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 18:18:48.727386154 (530h49m48.598469189s) [upload=465546]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 18:18:48.975492044 (530h49m48.846575079s) [seg_pos={0 0}, upload=465546]: Begin upload segment... Dec 27 18:18:48.975558320 (530h49m48.846641355s) [seg_pos={0 0}, upload=465546]: Done begin upload segment. Dec 27 18:18:48.975599548 (530h49m48.846682594s) [noise=true , address=77.90.31.12:4589 , piece_id=3MI7AOPYUTKLWMURIBUKOD7DNP4HYUPST63WDVZXLSEYPGFILXKA, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.975677116 (530h49m48.846760171s) [noise=true , address=145.79.227.7:4295 , piece_id=YUTDEZNBRFGICWDWL3UHOYHRIZH7RZSY43AAARQ7H72FVHV2TWRA, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.976537595 (530h49m48.84762064s) [noise=true , address=145.79.227.25:4618 , piece_id=657IUHGFLLOLMJVNEWFYZ55FLFQI5QKFO7ENAH6GKYDPI7GQVXFQ, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.977260301 (530h49m48.848343346s) [noise=true , address=145.79.227.47:4265 , piece_id=T4WQNGBMENUE32AYQ2MXFAKR3LRYIOVRCJJE37V2XOKOJ4VVIJYA, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.977697493 (530h49m48.848780528s) [noise=true , address=145.79.227.3:4151 , piece_id=V5F6ZDZ4S4EUAJI23OZTCWOD5QH5OHOWUENXFHIDNEJXNU6JA7AQ, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.977962107 (530h49m48.849045152s) [noise=true , address=77.90.31.10:4519 , piece_id=EUF7QLG6CJGERWWATE2N7QGNETUZYWDTFOC6LQVXUFQLKZKZSH5Q, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.978200521 (530h49m48.849283556s) [noise=true , address=45.91.116.7:4010 , piece_id=EY2J4FJ4JPQ2I2EK324SJ2Q43TOYQYPLJWXOUJNCRTCXXOZKMATA, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.978629147 (530h49m48.849712182s) [noise=true , address=145.79.227.45:4230 , piece_id=BHEUT4KRORKUD3CFOCIYSYJPISFV3EQMNQJ7Y3FB3KSOGIDUAIJQ, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.978701996 (530h49m48.849785031s) [noise=true , address=145.79.227.17:4972 , piece_id=NHQ6L2GHJA7UGARK2BA7HQNSNAFGILDJOCPQBDVWTOGLFXWDSERA, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.978951932 (530h49m48.850034967s) [noise=true , address=45.91.116.9:4261 , piece_id=6KKONXYFBLEKXTJMNQ452QDLMGOFTGOML2W77L7AOCW6IB5D3ELQ, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.979365720 (530h49m48.850448765s) [noise=true , address=77.90.31.14:4664 , piece_id=J42YQFOCQEVVX32GWSIDMOGZFXZ7KIO5KH5XUJ36XI3KVXYSKPNA, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.979489626 (530h49m48.850572681s) [noise=true , address=77.90.31.19:4844 , piece_id=57WUY7F7ERDXIXGF5LLR5V5SD3CFEMZHEA4TZY7N2PVHWORCO2GQ, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.979717109 (530h49m48.850800154s) [noise=true , address=77.90.31.13:4627 , piece_id=7IXYPYPEI77UG5VYCM6MGHA5JSEG6PGUUSJMDZ7BKWGB2V24MBSA, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.980035897 (530h49m48.851118932s) [noise=true , address=145.79.227.52:4447 , piece_id=KB2TWKDOQFH7JQ24QNMUHWZLIFMO2UEINGNIVSELXYFEY36F5SMA, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.980186343 (530h49m48.851269388s) [noise=true , address=45.91.116.4:4939 , piece_id=P2MFR7PJ6GJOENSTD2XRPYMTAUTGNQWNLCTVXLTFNZO3RMP7KDDQ, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.981159686 (530h49m48.852242731s) [noise=true , address=145.79.227.37:4566 , piece_id=JBT3U7FWWPQXN7TTXCKP7TVWO5C52RN3PADBCYAN2KHH7S7D7NTA, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:50.593473445 (530h49m50.46455649s) [upload=465546]: Next returned nil segment Dec 27 18:18:50.593485127 (530h49m50.464568162s) [upload=465546]: Waiting for error group managing segments... Dec 27 18:18:50.978438319 (530h49m50.849521364s) [noise=true , address=145.79.227.3:4151 , piece_id=V5F6ZDZ4S4EUAJI23OZTCWOD5QH5OHOWUENXFHIDNEJXNU6JA7AQ, seg_pos={0 0}, upload=465546]: Done uploading piece. err: <nil> Dec 27 18:18:50.983972487 (530h49m50.855055532s) [noise=true , address=145.79.227.37:4566 , piece_id=JBT3U7FWWPQXN7TTXCKP7TVWO5C52RN3PADBCYAN2KHH7S7D7NTA, seg_pos={0 0}, upload=465546]: Done uploading piece. err: <nil> Dec 27 18:18:50.987682160 (530h49m50.858765206s) [noise=true , address=145.79.227.7:4295 , piece_id=YUTDEZNBRFGICWDWL3UHOYHRIZH7RZSY43AAARQ7H72FVHV2TWRA, seg_pos={0 0}, upload=465546]: Done uploading piece. err: <nil> Dec 27 18:18:50.988931730 (530h49m50.860014775s) [noise=true , address=145.79.227.25:4618 , piece_id=657IUHGFLLOLMJVNEWFYZ55FLFQI5QKFO7ENAH6GKYDPI7GQVXFQ, seg_pos={0 0}, upload=465546]: Done uploading piece. err: <nil> Dec 27 18:18:50.991081203 (530h49m50.862164238s) [noise=true , address=145.79.227.52:4447 , piece_id=KB2TWKDOQFH7JQ24QNMUHWZLIFMO2UEINGNIVSELXYFEY36F5SMA, seg_pos={0 0}, upload=465546]: Done uploading piece. err: <nil> Dec 27 18:18:50.999632427 (530h49m50.870715472s) [noise=true , address=145.79.227.45:4230 , piece_id=BHEUT4KRORKUD3CFOCIYSYJPISFV3EQMNQJ7Y3FB3KSOGIDUAIJQ, seg_pos={0 0}, upload=465546]: Done uploading piece. err: <nil> Dec 27 18:18:51.006550861 (530h49m50.877633896s) [noise=true , address=145.79.227.17:4972 , piece_id=NHQ6L2GHJA7UGARK2BA7HQNSNAFGILDJOCPQBDVWTOGLFXWDSERA, seg_pos={0 0}, upload=465546]: Done uploading piece. err: <nil> Dec 27 18:18:51.027665085 (530h49m50.89874813s) [noise=true , address=145.79.227.47:4265 , piece_id=T4WQNGBMENUE32AYQ2MXFAKR3LRYIOVRCJJE37V2XOKOJ4VVIJYA, seg_pos={0 0}, upload=465546]: Done uploading piece. err: <nil> Dec 27 18:18:51.146239203 (530h49m51.017322248s) [noise=true , address=77.90.31.14:4664 , piece_id=J42YQFOCQEVVX32GWSIDMOGZFXZ7KIO5KH5XUJ36XI3KVXYSKPNA, seg_pos={0 0}, upload=465546]: Done uploading piece. err: <nil> Dec 27 18:18:51.164235901 (530h49m51.035318936s) [noise=true , address=77.90.31.10:4519 , piece_id=EUF7QLG6CJGERWWATE2N7QGNETUZYWDTFOC6LQVXUFQLKZKZSH5Q, seg_pos={0 0}, upload=465546]: Done uploading piece. err: <nil> Dec 27 18:18:51.231133206 (530h49m51.102216241s) [noise=true , address=45.91.116.9:4261 , piece_id=6KKONXYFBLEKXTJMNQ452QDLMGOFTGOML2W77L7AOCW6IB5D3ELQ, seg_pos={0 0}, upload=465546]: Done uploading piece. err: <nil> Dec 27 18:18:51.255307578 (530h49m51.126390623s) [noise=true , address=77.90.31.13:4627 , piece_id=7IXYPYPEI77UG5VYCM6MGHA5JSEG6PGUUSJMDZ7BKWGB2V24MBSA, seg_pos={0 0}, upload=465546]: Done uploading piece. err: <nil> Dec 27 18:18:51.282041364 (530h49m51.153124409s) [noise=true , address=77.90.31.19:4844 , piece_id=57WUY7F7ERDXIXGF5LLR5V5SD3CFEMZHEA4TZY7N2PVHWORCO2GQ, seg_pos={0 0}, upload=465546]: Done uploading piece. err: <nil> Dec 27 18:18:51.282068656 (530h49m51.153151701s) [seg_pos={0 0}, upload=465546]: Segment reached optimal threshold of 13 pieces. Dec 27 18:18:51.282286801 (530h49m51.153369847s) [noise=true , address=77.90.31.12:4589 , piece_id=3MI7AOPYUTKLWMURIBUKOD7DNP4HYUPST63WDVZXLSEYPGFILXKA, seg_pos={0 0}, upload=465546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: write tcp 188.166.225.170:39916->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:39916->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39916->77.90.31.12:4589: use of closed network connection Dec 27 18:18:51.282294697 (530h49m51.153377742s) [noise=true , address=77.90.31.12:4589 , piece_id=3MI7AOPYUTKLWMURIBUKOD7DNP4HYUPST63WDVZXLSEYPGFILXKA, seg_pos={0 0}, upload=465546]: Long tail cancellation due to optimal threshold Dec 27 18:18:51.282436467 (530h49m51.153519522s) [noise=true , address=45.91.116.4:4939 , piece_id=P2MFR7PJ6GJOENSTD2XRPYMTAUTGNQWNLCTVXLTFNZO3RMP7KDDQ, seg_pos={0 0}, upload=465546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: write tcp 188.166.225.170:55750->45.91.116.4:4939: use of closed network connection; write tcp 188.166.225.170:55750->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55750->45.91.116.4:4939: use of closed network connection Dec 27 18:18:51.282606351 (530h49m51.153689386s) [noise=true , address=45.91.116.4:4939 , piece_id=P2MFR7PJ6GJOENSTD2XRPYMTAUTGNQWNLCTVXLTFNZO3RMP7KDDQ, seg_pos={0 0}, upload=465546]: Long tail cancellation due to optimal threshold Dec 27 18:18:51.282667006 (530h49m51.153750051s) [noise=true , address=45.91.116.7:4010 , piece_id=EY2J4FJ4JPQ2I2EK324SJ2Q43TOYQYPLJWXOUJNCRTCXXOZKMATA, seg_pos={0 0}, upload=465546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 27 18:18:51.282676334 (530h49m51.153759379s) [noise=true , address=45.91.116.7:4010 , piece_id=EY2J4FJ4JPQ2I2EK324SJ2Q43TOYQYPLJWXOUJNCRTCXXOZKMATA, seg_pos={0 0}, upload=465546]: Long tail cancellation due to optimal threshold Dec 27 18:18:51.282764662 (530h49m51.153847707s) [seg_pos={0 0}, upload=465546]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 18:18:51.282917964 (530h49m51.154001009s) [upload=465546]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 18:18:51.282924527 (530h49m51.154007572s) [upload=465546]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 18:18:51.598820670 (530h49m51.469903705s) [upload=465546]: Done uploading segments... Dec 27 18:18:51.598840398 (530h49m51.469923433s) [upload=465546]: Upload finished. err: <nil> Duration: 5.663029902s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........X.................................................] duration: 3.591523516s trace: https://jaeger.dev.storj.io/trace/3bddfef0dfa96557 FAIL (2025-12-27 18:20:57 UTC) orbiter warning: context deadline exceeded Duration: 3.591558984s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [..XXXX.X....XXX..XX....X...X.X........X.......X..X.XXXXXXX..] duration: 26.406633823s trace: https://jaeger.dev.storj.io/trace/6aea6e6d013c1516 FAIL (2025-12-27 18:20:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: EUF7QLG6CJGERWWATE2N7QGNETUZYWDTFOC6LQVXUFQLKZKZSH5Q) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: 6KKONXYFBLEKXTJMNQ452QDLMGOFTGOML2W77L7AOCW6IB5D3ELQ) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: 57WUY7F7ERDXIXGF5LLR5V5SD3CFEMZHEA4TZY7N2PVHWORCO2GQ) context deadline exceeded ======LOGS====== g segments... Dec 27 18:18:48.727318184 (530h49m48.598401219s) [upload=465546]: Got next segment. Inline: false Dec 27 18:18:48.727380142 (530h49m48.598463187s) [upload=465546]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 18:18:48.727386154 (530h49m48.598469189s) [upload=465546]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 18:18:48.975492044 (530h49m48.846575079s) [seg_pos={0 0}, upload=465546]: Begin upload segment... Dec 27 18:18:48.975558320 (530h49m48.846641355s) [seg_pos={0 0}, upload=465546]: Done begin upload segment. Dec 27 18:18:48.975599548 (530h49m48.846682594s) [noise=true , address=77.90.31.12:4589 , piece_id=3MI7AOPYUTKLWMURIBUKOD7DNP4HYUPST63WDVZXLSEYPGFILXKA, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.975677116 (530h49m48.846760171s) [noise=true , address=145.79.227.7:4295 , piece_id=YUTDEZNBRFGICWDWL3UHOYHRIZH7RZSY43AAARQ7H72FVHV2TWRA, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.976537595 (530h49m48.84762064s) [noise=true , address=145.79.227.25:4618 , piece_id=657IUHGFLLOLMJVNEWFYZ55FLFQI5QKFO7ENAH6GKYDPI7GQVXFQ, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.977260301 (530h49m48.848343346s) [noise=true , address=145.79.227.47:4265 , piece_id=T4WQNGBMENUE32AYQ2MXFAKR3LRYIOVRCJJE37V2XOKOJ4VVIJYA, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.977697493 (530h49m48.848780528s) [noise=true , address=145.79.227.3:4151 , piece_id=V5F6ZDZ4S4EUAJI23OZTCWOD5QH5OHOWUENXFHIDNEJXNU6JA7AQ, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.977962107 (530h49m48.849045152s) [noise=true , address=77.90.31.10:4519 , piece_id=EUF7QLG6CJGERWWATE2N7QGNETUZYWDTFOC6LQVXUFQLKZKZSH5Q, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.978200521 (530h49m48.849283556s) [noise=true , address=45.91.116.7:4010 , piece_id=EY2J4FJ4JPQ2I2EK324SJ2Q43TOYQYPLJWXOUJNCRTCXXOZKMATA, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.978629147 (530h49m48.849712182s) [noise=true , address=145.79.227.45:4230 , piece_id=BHEUT4KRORKUD3CFOCIYSYJPISFV3EQMNQJ7Y3FB3KSOGIDUAIJQ, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.978701996 (530h49m48.849785031s) [noise=true , address=145.79.227.17:4972 , piece_id=NHQ6L2GHJA7UGARK2BA7HQNSNAFGILDJOCPQBDVWTOGLFXWDSERA, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.978951932 (530h49m48.850034967s) [noise=true , address=45.91.116.9:4261 , piece_id=6KKONXYFBLEKXTJMNQ452QDLMGOFTGOML2W77L7AOCW6IB5D3ELQ, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.979365720 (530h49m48.850448765s) [noise=true , address=77.90.31.14:4664 , piece_id=J42YQFOCQEVVX32GWSIDMOGZFXZ7KIO5KH5XUJ36XI3KVXYSKPNA, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.979489626 (530h49m48.850572681s) [noise=true , address=77.90.31.19:4844 , piece_id=57WUY7F7ERDXIXGF5LLR5V5SD3CFEMZHEA4TZY7N2PVHWORCO2GQ, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.979717109 (530h49m48.850800154s) [noise=true , address=77.90.31.13:4627 , piece_id=7IXYPYPEI77UG5VYCM6MGHA5JSEG6PGUUSJMDZ7BKWGB2V24MBSA, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.980035897 (530h49m48.851118932s) [noise=true , address=145.79.227.52:4447 , piece_id=KB2TWKDOQFH7JQ24QNMUHWZLIFMO2UEINGNIVSELXYFEY36F5SMA, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.980186343 (530h49m48.851269388s) [noise=true , address=45.91.116.4:4939 , piece_id=P2MFR7PJ6GJOENSTD2XRPYMTAUTGNQWNLCTVXLTFNZO3RMP7KDDQ, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.981159686 (530h49m48.852242731s) [noise=true , address=145.79.227.37:4566 , piece_id=JBT3U7FWWPQXN7TTXCKP7TVWO5C52RN3PADBCYAN2KHH7S7D7NTA, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:50.593473445 (530h49m50.46455649s) [upload=465546]: Next returned nil segment Dec 27 18:18:50.593485127 (530h49m50.464568162s) [upload=465546]: Waiting for error group managing segments... Dec 27 18:18:50.978438319 (530h49m50.849521364s) [noise=true , address=145.79.227.3:4151 , piece_id=V5F6ZDZ4S4EUAJI23OZTCWOD5QH5OHOWUENXFHIDNEJXNU6JA7AQ, seg_pos={0 0}, upload=465546]: Done uploading piece. err: <nil> Dec 27 18:18:50.983972487 (530h49m50.855055532s) [noise=true , address=145.79.227.37:4566 , piece_id=JBT3U7FWWPQXN7TTXCKP7TVWO5C52RN3PADBCYAN2KHH7S7D7NTA, seg_pos={0 0}, upload=465546]: Done uploading piece. err: <nil> Dec 27 18:18:50.987682160 (530h49m50.858765206s) [noise=true , address=145.79.227.7:4295 , piece_id=YUTDEZNBRFGICWDWL3UHOYHRIZH7RZSY43AAARQ7H72FVHV2TWRA, seg_pos={0 0}, upload=465546]: Done uploading piece. err: <nil> Dec 27 18:18:50.988931730 (530h49m50.860014775s) [noise=true , address=145.79.227.25:4618 , piece_id=657IUHGFLLOLMJVNEWFYZ55FLFQI5QKFO7ENAH6GKYDPI7GQVXFQ, seg_pos={0 0}, upload=465546]: Done uploading piece. err: <nil> Dec 27 18:18:50.991081203 (530h49m50.862164238s) [noise=true , address=145.79.227.52:4447 , piece_id=KB2TWKDOQFH7JQ24QNMUHWZLIFMO2UEINGNIVSELXYFEY36F5SMA, seg_pos={0 0}, upload=465546]: Done uploading piece. err: <nil> Dec 27 18:18:50.999632427 (530h49m50.870715472s) [noise=true , address=145.79.227.45:4230 , piece_id=BHEUT4KRORKUD3CFOCIYSYJPISFV3EQMNQJ7Y3FB3KSOGIDUAIJQ, seg_pos={0 0}, upload=465546]: Done uploading piece. err: <nil> Dec 27 18:18:51.006550861 (530h49m50.877633896s) [noise=true , address=145.79.227.17:4972 , piece_id=NHQ6L2GHJA7UGARK2BA7HQNSNAFGILDJOCPQBDVWTOGLFXWDSERA, seg_pos={0 0}, upload=465546]: Done uploading piece. err: <nil> Dec 27 18:18:51.027665085 (530h49m50.89874813s) [noise=true , address=145.79.227.47:4265 , piece_id=T4WQNGBMENUE32AYQ2MXFAKR3LRYIOVRCJJE37V2XOKOJ4VVIJYA, seg_pos={0 0}, upload=465546]: Done uploading piece. err: <nil> Dec 27 18:18:51.146239203 (530h49m51.017322248s) [noise=true , address=77.90.31.14:4664 , piece_id=J42YQFOCQEVVX32GWSIDMOGZFXZ7KIO5KH5XUJ36XI3KVXYSKPNA, seg_pos={0 0}, upload=465546]: Done uploading piece. err: <nil> Dec 27 18:18:51.164235901 (530h49m51.035318936s) [noise=true , address=77.90.31.10:4519 , piece_id=EUF7QLG6CJGERWWATE2N7QGNETUZYWDTFOC6LQVXUFQLKZKZSH5Q, seg_pos={0 0}, upload=465546]: Done uploading piece. err: <nil> Dec 27 18:18:51.231133206 (530h49m51.102216241s) [noise=true , address=45.91.116.9:4261 , piece_id=6KKONXYFBLEKXTJMNQ452QDLMGOFTGOML2W77L7AOCW6IB5D3ELQ, seg_pos={0 0}, upload=465546]: Done uploading piece. err: <nil> Dec 27 18:18:51.255307578 (530h49m51.126390623s) [noise=true , address=77.90.31.13:4627 , piece_id=7IXYPYPEI77UG5VYCM6MGHA5JSEG6PGUUSJMDZ7BKWGB2V24MBSA, seg_pos={0 0}, upload=465546]: Done uploading piece. err: <nil> Dec 27 18:18:51.282041364 (530h49m51.153124409s) [noise=true , address=77.90.31.19:4844 , piece_id=57WUY7F7ERDXIXGF5LLR5V5SD3CFEMZHEA4TZY7N2PVHWORCO2GQ, seg_pos={0 0}, upload=465546]: Done uploading piece. err: <nil> Dec 27 18:18:51.282068656 (530h49m51.153151701s) [seg_pos={0 0}, upload=465546]: Segment reached optimal threshold of 13 pieces. Dec 27 18:18:51.282286801 (530h49m51.153369847s) [noise=true , address=77.90.31.12:4589 , piece_id=3MI7AOPYUTKLWMURIBUKOD7DNP4HYUPST63WDVZXLSEYPGFILXKA, seg_pos={0 0}, upload=465546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: write tcp 188.166.225.170:39916->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:39916->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39916->77.90.31.12:4589: use of closed network connection Dec 27 18:18:51.282294697 (530h49m51.153377742s) [noise=true , address=77.90.31.12:4589 , piece_id=3MI7AOPYUTKLWMURIBUKOD7DNP4HYUPST63WDVZXLSEYPGFILXKA, seg_pos={0 0}, upload=465546]: Long tail cancellation due to optimal threshold Dec 27 18:18:51.282436467 (530h49m51.153519522s) [noise=true , address=45.91.116.4:4939 , piece_id=P2MFR7PJ6GJOENSTD2XRPYMTAUTGNQWNLCTVXLTFNZO3RMP7KDDQ, seg_pos={0 0}, upload=465546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: write tcp 188.166.225.170:55750->45.91.116.4:4939: use of closed network connection; write tcp 188.166.225.170:55750->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55750->45.91.116.4:4939: use of closed network connection Dec 27 18:18:51.282606351 (530h49m51.153689386s) [noise=true , address=45.91.116.4:4939 , piece_id=P2MFR7PJ6GJOENSTD2XRPYMTAUTGNQWNLCTVXLTFNZO3RMP7KDDQ, seg_pos={0 0}, upload=465546]: Long tail cancellation due to optimal threshold Dec 27 18:18:51.282667006 (530h49m51.153750051s) [noise=true , address=45.91.116.7:4010 , piece_id=EY2J4FJ4JPQ2I2EK324SJ2Q43TOYQYPLJWXOUJNCRTCXXOZKMATA, seg_pos={0 0}, upload=465546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 27 18:18:51.282676334 (530h49m51.153759379s) [noise=true , address=45.91.116.7:4010 , piece_id=EY2J4FJ4JPQ2I2EK324SJ2Q43TOYQYPLJWXOUJNCRTCXXOZKMATA, seg_pos={0 0}, upload=465546]: Long tail cancellation due to optimal threshold Dec 27 18:18:51.282764662 (530h49m51.153847707s) [seg_pos={0 0}, upload=465546]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 18:18:51.282917964 (530h49m51.154001009s) [upload=465546]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 18:18:51.282924527 (530h49m51.154007572s) [upload=465546]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 18:18:51.598820670 (530h49m51.469903705s) [upload=465546]: Done uploading segments... Dec 27 18:18:51.598840398 (530h49m51.469923433s) [upload=465546]: Upload finished. err: <nil> Duration: 26.406664221s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........X................................................] duration: 3.501916165s trace: https://jaeger.dev.storj.io/trace/3bddfef0dfa96557 FAIL (2025-12-27 18:19:54 UTC) orbiter warning: context deadline exceeded Duration: 3.501940321s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [X..XXXX.X....XXX..XX....X...X.X........X.......X..X.XXXXXXX.] duration: 5.955156284s trace: https://jaeger.dev.storj.io/trace/6aea6e6d013c1516 FAIL (2025-12-27 18:19:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: EUF7QLG6CJGERWWATE2N7QGNETUZYWDTFOC6LQVXUFQLKZKZSH5Q) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: 6KKONXYFBLEKXTJMNQ452QDLMGOFTGOML2W77L7AOCW6IB5D3ELQ) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: 57WUY7F7ERDXIXGF5LLR5V5SD3CFEMZHEA4TZY7N2PVHWORCO2GQ) context deadline exceeded ======LOGS====== g segments... Dec 27 18:18:48.727318184 (530h49m48.598401219s) [upload=465546]: Got next segment. Inline: false Dec 27 18:18:48.727380142 (530h49m48.598463187s) [upload=465546]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 18:18:48.727386154 (530h49m48.598469189s) [upload=465546]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 18:18:48.975492044 (530h49m48.846575079s) [seg_pos={0 0}, upload=465546]: Begin upload segment... Dec 27 18:18:48.975558320 (530h49m48.846641355s) [seg_pos={0 0}, upload=465546]: Done begin upload segment. Dec 27 18:18:48.975599548 (530h49m48.846682594s) [noise=true , address=77.90.31.12:4589 , piece_id=3MI7AOPYUTKLWMURIBUKOD7DNP4HYUPST63WDVZXLSEYPGFILXKA, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.975677116 (530h49m48.846760171s) [noise=true , address=145.79.227.7:4295 , piece_id=YUTDEZNBRFGICWDWL3UHOYHRIZH7RZSY43AAARQ7H72FVHV2TWRA, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.976537595 (530h49m48.84762064s) [noise=true , address=145.79.227.25:4618 , piece_id=657IUHGFLLOLMJVNEWFYZ55FLFQI5QKFO7ENAH6GKYDPI7GQVXFQ, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.977260301 (530h49m48.848343346s) [noise=true , address=145.79.227.47:4265 , piece_id=T4WQNGBMENUE32AYQ2MXFAKR3LRYIOVRCJJE37V2XOKOJ4VVIJYA, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.977697493 (530h49m48.848780528s) [noise=true , address=145.79.227.3:4151 , piece_id=V5F6ZDZ4S4EUAJI23OZTCWOD5QH5OHOWUENXFHIDNEJXNU6JA7AQ, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.977962107 (530h49m48.849045152s) [noise=true , address=77.90.31.10:4519 , piece_id=EUF7QLG6CJGERWWATE2N7QGNETUZYWDTFOC6LQVXUFQLKZKZSH5Q, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.978200521 (530h49m48.849283556s) [noise=true , address=45.91.116.7:4010 , piece_id=EY2J4FJ4JPQ2I2EK324SJ2Q43TOYQYPLJWXOUJNCRTCXXOZKMATA, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.978629147 (530h49m48.849712182s) [noise=true , address=145.79.227.45:4230 , piece_id=BHEUT4KRORKUD3CFOCIYSYJPISFV3EQMNQJ7Y3FB3KSOGIDUAIJQ, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.978701996 (530h49m48.849785031s) [noise=true , address=145.79.227.17:4972 , piece_id=NHQ6L2GHJA7UGARK2BA7HQNSNAFGILDJOCPQBDVWTOGLFXWDSERA, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.978951932 (530h49m48.850034967s) [noise=true , address=45.91.116.9:4261 , piece_id=6KKONXYFBLEKXTJMNQ452QDLMGOFTGOML2W77L7AOCW6IB5D3ELQ, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.979365720 (530h49m48.850448765s) [noise=true , address=77.90.31.14:4664 , piece_id=J42YQFOCQEVVX32GWSIDMOGZFXZ7KIO5KH5XUJ36XI3KVXYSKPNA, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.979489626 (530h49m48.850572681s) [noise=true , address=77.90.31.19:4844 , piece_id=57WUY7F7ERDXIXGF5LLR5V5SD3CFEMZHEA4TZY7N2PVHWORCO2GQ, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.979717109 (530h49m48.850800154s) [noise=true , address=77.90.31.13:4627 , piece_id=7IXYPYPEI77UG5VYCM6MGHA5JSEG6PGUUSJMDZ7BKWGB2V24MBSA, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.980035897 (530h49m48.851118932s) [noise=true , address=145.79.227.52:4447 , piece_id=KB2TWKDOQFH7JQ24QNMUHWZLIFMO2UEINGNIVSELXYFEY36F5SMA, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.980186343 (530h49m48.851269388s) [noise=true , address=45.91.116.4:4939 , piece_id=P2MFR7PJ6GJOENSTD2XRPYMTAUTGNQWNLCTVXLTFNZO3RMP7KDDQ, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.981159686 (530h49m48.852242731s) [noise=true , address=145.79.227.37:4566 , piece_id=JBT3U7FWWPQXN7TTXCKP7TVWO5C52RN3PADBCYAN2KHH7S7D7NTA, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:50.593473445 (530h49m50.46455649s) [upload=465546]: Next returned nil segment Dec 27 18:18:50.593485127 (530h49m50.464568162s) [upload=465546]: Waiting for error group managing segments... Dec 27 18:18:50.978438319 (530h49m50.849521364s) [noise=true , address=145.79.227.3:4151 , piece_id=V5F6ZDZ4S4EUAJI23OZTCWOD5QH5OHOWUENXFHIDNEJXNU6JA7AQ, seg_pos={0 0}, upload=465546]: Done uploading piece. err: <nil> Dec 27 18:18:50.983972487 (530h49m50.855055532s) [noise=true , address=145.79.227.37:4566 , piece_id=JBT3U7FWWPQXN7TTXCKP7TVWO5C52RN3PADBCYAN2KHH7S7D7NTA, seg_pos={0 0}, upload=465546]: Done uploading piece. err: <nil> Dec 27 18:18:50.987682160 (530h49m50.858765206s) [noise=true , address=145.79.227.7:4295 , piece_id=YUTDEZNBRFGICWDWL3UHOYHRIZH7RZSY43AAARQ7H72FVHV2TWRA, seg_pos={0 0}, upload=465546]: Done uploading piece. err: <nil> Dec 27 18:18:50.988931730 (530h49m50.860014775s) [noise=true , address=145.79.227.25:4618 , piece_id=657IUHGFLLOLMJVNEWFYZ55FLFQI5QKFO7ENAH6GKYDPI7GQVXFQ, seg_pos={0 0}, upload=465546]: Done uploading piece. err: <nil> Dec 27 18:18:50.991081203 (530h49m50.862164238s) [noise=true , address=145.79.227.52:4447 , piece_id=KB2TWKDOQFH7JQ24QNMUHWZLIFMO2UEINGNIVSELXYFEY36F5SMA, seg_pos={0 0}, upload=465546]: Done uploading piece. err: <nil> Dec 27 18:18:50.999632427 (530h49m50.870715472s) [noise=true , address=145.79.227.45:4230 , piece_id=BHEUT4KRORKUD3CFOCIYSYJPISFV3EQMNQJ7Y3FB3KSOGIDUAIJQ, seg_pos={0 0}, upload=465546]: Done uploading piece. err: <nil> Dec 27 18:18:51.006550861 (530h49m50.877633896s) [noise=true , address=145.79.227.17:4972 , piece_id=NHQ6L2GHJA7UGARK2BA7HQNSNAFGILDJOCPQBDVWTOGLFXWDSERA, seg_pos={0 0}, upload=465546]: Done uploading piece. err: <nil> Dec 27 18:18:51.027665085 (530h49m50.89874813s) [noise=true , address=145.79.227.47:4265 , piece_id=T4WQNGBMENUE32AYQ2MXFAKR3LRYIOVRCJJE37V2XOKOJ4VVIJYA, seg_pos={0 0}, upload=465546]: Done uploading piece. err: <nil> Dec 27 18:18:51.146239203 (530h49m51.017322248s) [noise=true , address=77.90.31.14:4664 , piece_id=J42YQFOCQEVVX32GWSIDMOGZFXZ7KIO5KH5XUJ36XI3KVXYSKPNA, seg_pos={0 0}, upload=465546]: Done uploading piece. err: <nil> Dec 27 18:18:51.164235901 (530h49m51.035318936s) [noise=true , address=77.90.31.10:4519 , piece_id=EUF7QLG6CJGERWWATE2N7QGNETUZYWDTFOC6LQVXUFQLKZKZSH5Q, seg_pos={0 0}, upload=465546]: Done uploading piece. err: <nil> Dec 27 18:18:51.231133206 (530h49m51.102216241s) [noise=true , address=45.91.116.9:4261 , piece_id=6KKONXYFBLEKXTJMNQ452QDLMGOFTGOML2W77L7AOCW6IB5D3ELQ, seg_pos={0 0}, upload=465546]: Done uploading piece. err: <nil> Dec 27 18:18:51.255307578 (530h49m51.126390623s) [noise=true , address=77.90.31.13:4627 , piece_id=7IXYPYPEI77UG5VYCM6MGHA5JSEG6PGUUSJMDZ7BKWGB2V24MBSA, seg_pos={0 0}, upload=465546]: Done uploading piece. err: <nil> Dec 27 18:18:51.282041364 (530h49m51.153124409s) [noise=true , address=77.90.31.19:4844 , piece_id=57WUY7F7ERDXIXGF5LLR5V5SD3CFEMZHEA4TZY7N2PVHWORCO2GQ, seg_pos={0 0}, upload=465546]: Done uploading piece. err: <nil> Dec 27 18:18:51.282068656 (530h49m51.153151701s) [seg_pos={0 0}, upload=465546]: Segment reached optimal threshold of 13 pieces. Dec 27 18:18:51.282286801 (530h49m51.153369847s) [noise=true , address=77.90.31.12:4589 , piece_id=3MI7AOPYUTKLWMURIBUKOD7DNP4HYUPST63WDVZXLSEYPGFILXKA, seg_pos={0 0}, upload=465546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: write tcp 188.166.225.170:39916->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:39916->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39916->77.90.31.12:4589: use of closed network connection Dec 27 18:18:51.282294697 (530h49m51.153377742s) [noise=true , address=77.90.31.12:4589 , piece_id=3MI7AOPYUTKLWMURIBUKOD7DNP4HYUPST63WDVZXLSEYPGFILXKA, seg_pos={0 0}, upload=465546]: Long tail cancellation due to optimal threshold Dec 27 18:18:51.282436467 (530h49m51.153519522s) [noise=true , address=45.91.116.4:4939 , piece_id=P2MFR7PJ6GJOENSTD2XRPYMTAUTGNQWNLCTVXLTFNZO3RMP7KDDQ, seg_pos={0 0}, upload=465546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: write tcp 188.166.225.170:55750->45.91.116.4:4939: use of closed network connection; write tcp 188.166.225.170:55750->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55750->45.91.116.4:4939: use of closed network connection Dec 27 18:18:51.282606351 (530h49m51.153689386s) [noise=true , address=45.91.116.4:4939 , piece_id=P2MFR7PJ6GJOENSTD2XRPYMTAUTGNQWNLCTVXLTFNZO3RMP7KDDQ, seg_pos={0 0}, upload=465546]: Long tail cancellation due to optimal threshold Dec 27 18:18:51.282667006 (530h49m51.153750051s) [noise=true , address=45.91.116.7:4010 , piece_id=EY2J4FJ4JPQ2I2EK324SJ2Q43TOYQYPLJWXOUJNCRTCXXOZKMATA, seg_pos={0 0}, upload=465546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 27 18:18:51.282676334 (530h49m51.153759379s) [noise=true , address=45.91.116.7:4010 , piece_id=EY2J4FJ4JPQ2I2EK324SJ2Q43TOYQYPLJWXOUJNCRTCXXOZKMATA, seg_pos={0 0}, upload=465546]: Long tail cancellation due to optimal threshold Dec 27 18:18:51.282764662 (530h49m51.153847707s) [seg_pos={0 0}, upload=465546]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 18:18:51.282917964 (530h49m51.154001009s) [upload=465546]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 18:18:51.282924527 (530h49m51.154007572s) [upload=465546]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 18:18:51.598820670 (530h49m51.469903705s) [upload=465546]: Done uploading segments... Dec 27 18:18:51.598840398 (530h49m51.469923433s) [upload=465546]: Upload finished. err: <nil> Duration: 5.955187164s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [.X..XXXX.X....XXX..XX....X...X.X........X.......X..X.XXXXXXX] duration: 30.002591872s trace: https://jaeger.dev.storj.io/trace/6aea6e6d013c1516 FAIL (2025-12-27 18:19:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: EUF7QLG6CJGERWWATE2N7QGNETUZYWDTFOC6LQVXUFQLKZKZSH5Q) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: 6KKONXYFBLEKXTJMNQ452QDLMGOFTGOML2W77L7AOCW6IB5D3ELQ) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: 57WUY7F7ERDXIXGF5LLR5V5SD3CFEMZHEA4TZY7N2PVHWORCO2GQ) context deadline exceeded ======LOGS====== g segments... Dec 27 18:18:48.727318184 (530h49m48.598401219s) [upload=465546]: Got next segment. Inline: false Dec 27 18:18:48.727380142 (530h49m48.598463187s) [upload=465546]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 18:18:48.727386154 (530h49m48.598469189s) [upload=465546]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 18:18:48.975492044 (530h49m48.846575079s) [seg_pos={0 0}, upload=465546]: Begin upload segment... Dec 27 18:18:48.975558320 (530h49m48.846641355s) [seg_pos={0 0}, upload=465546]: Done begin upload segment. Dec 27 18:18:48.975599548 (530h49m48.846682594s) [noise=true , address=77.90.31.12:4589 , piece_id=3MI7AOPYUTKLWMURIBUKOD7DNP4HYUPST63WDVZXLSEYPGFILXKA, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.975677116 (530h49m48.846760171s) [noise=true , address=145.79.227.7:4295 , piece_id=YUTDEZNBRFGICWDWL3UHOYHRIZH7RZSY43AAARQ7H72FVHV2TWRA, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.976537595 (530h49m48.84762064s) [noise=true , address=145.79.227.25:4618 , piece_id=657IUHGFLLOLMJVNEWFYZ55FLFQI5QKFO7ENAH6GKYDPI7GQVXFQ, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.977260301 (530h49m48.848343346s) [noise=true , address=145.79.227.47:4265 , piece_id=T4WQNGBMENUE32AYQ2MXFAKR3LRYIOVRCJJE37V2XOKOJ4VVIJYA, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.977697493 (530h49m48.848780528s) [noise=true , address=145.79.227.3:4151 , piece_id=V5F6ZDZ4S4EUAJI23OZTCWOD5QH5OHOWUENXFHIDNEJXNU6JA7AQ, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.977962107 (530h49m48.849045152s) [noise=true , address=77.90.31.10:4519 , piece_id=EUF7QLG6CJGERWWATE2N7QGNETUZYWDTFOC6LQVXUFQLKZKZSH5Q, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.978200521 (530h49m48.849283556s) [noise=true , address=45.91.116.7:4010 , piece_id=EY2J4FJ4JPQ2I2EK324SJ2Q43TOYQYPLJWXOUJNCRTCXXOZKMATA, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.978629147 (530h49m48.849712182s) [noise=true , address=145.79.227.45:4230 , piece_id=BHEUT4KRORKUD3CFOCIYSYJPISFV3EQMNQJ7Y3FB3KSOGIDUAIJQ, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.978701996 (530h49m48.849785031s) [noise=true , address=145.79.227.17:4972 , piece_id=NHQ6L2GHJA7UGARK2BA7HQNSNAFGILDJOCPQBDVWTOGLFXWDSERA, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.978951932 (530h49m48.850034967s) [noise=true , address=45.91.116.9:4261 , piece_id=6KKONXYFBLEKXTJMNQ452QDLMGOFTGOML2W77L7AOCW6IB5D3ELQ, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.979365720 (530h49m48.850448765s) [noise=true , address=77.90.31.14:4664 , piece_id=J42YQFOCQEVVX32GWSIDMOGZFXZ7KIO5KH5XUJ36XI3KVXYSKPNA, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.979489626 (530h49m48.850572681s) [noise=true , address=77.90.31.19:4844 , piece_id=57WUY7F7ERDXIXGF5LLR5V5SD3CFEMZHEA4TZY7N2PVHWORCO2GQ, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.979717109 (530h49m48.850800154s) [noise=true , address=77.90.31.13:4627 , piece_id=7IXYPYPEI77UG5VYCM6MGHA5JSEG6PGUUSJMDZ7BKWGB2V24MBSA, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.980035897 (530h49m48.851118932s) [noise=true , address=145.79.227.52:4447 , piece_id=KB2TWKDOQFH7JQ24QNMUHWZLIFMO2UEINGNIVSELXYFEY36F5SMA, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.980186343 (530h49m48.851269388s) [noise=true , address=45.91.116.4:4939 , piece_id=P2MFR7PJ6GJOENSTD2XRPYMTAUTGNQWNLCTVXLTFNZO3RMP7KDDQ, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:48.981159686 (530h49m48.852242731s) [noise=true , address=145.79.227.37:4566 , piece_id=JBT3U7FWWPQXN7TTXCKP7TVWO5C52RN3PADBCYAN2KHH7S7D7NTA, seg_pos={0 0}, upload=465546]: Uploading piece... Dec 27 18:18:50.593473445 (530h49m50.46455649s) [upload=465546]: Next returned nil segment Dec 27 18:18:50.593485127 (530h49m50.464568162s) [upload=465546]: Waiting for error group managing segments... Dec 27 18:18:50.978438319 (530h49m50.849521364s) [noise=true , address=145.79.227.3:4151 , piece_id=V5F6ZDZ4S4EUAJI23OZTCWOD5QH5OHOWUENXFHIDNEJXNU6JA7AQ, seg_pos={0 0}, upload=465546]: Done uploading piece. err: <nil> Dec 27 18:18:50.983972487 (530h49m50.855055532s) [noise=true , address=145.79.227.37:4566 , piece_id=JBT3U7FWWPQXN7TTXCKP7TVWO5C52RN3PADBCYAN2KHH7S7D7NTA, seg_pos={0 0}, upload=465546]: Done uploading piece. err: <nil> Dec 27 18:18:50.987682160 (530h49m50.858765206s) [noise=true , address=145.79.227.7:4295 , piece_id=YUTDEZNBRFGICWDWL3UHOYHRIZH7RZSY43AAARQ7H72FVHV2TWRA, seg_pos={0 0}, upload=465546]: Done uploading piece. err: <nil> Dec 27 18:18:50.988931730 (530h49m50.860014775s) [noise=true , address=145.79.227.25:4618 , piece_id=657IUHGFLLOLMJVNEWFYZ55FLFQI5QKFO7ENAH6GKYDPI7GQVXFQ, seg_pos={0 0}, upload=465546]: Done uploading piece. err: <nil> Dec 27 18:18:50.991081203 (530h49m50.862164238s) [noise=true , address=145.79.227.52:4447 , piece_id=KB2TWKDOQFH7JQ24QNMUHWZLIFMO2UEINGNIVSELXYFEY36F5SMA, seg_pos={0 0}, upload=465546]: Done uploading piece. err: <nil> Dec 27 18:18:50.999632427 (530h49m50.870715472s) [noise=true , address=145.79.227.45:4230 , piece_id=BHEUT4KRORKUD3CFOCIYSYJPISFV3EQMNQJ7Y3FB3KSOGIDUAIJQ, seg_pos={0 0}, upload=465546]: Done uploading piece. err: <nil> Dec 27 18:18:51.006550861 (530h49m50.877633896s) [noise=true , address=145.79.227.17:4972 , piece_id=NHQ6L2GHJA7UGARK2BA7HQNSNAFGILDJOCPQBDVWTOGLFXWDSERA, seg_pos={0 0}, upload=465546]: Done uploading piece. err: <nil> Dec 27 18:18:51.027665085 (530h49m50.89874813s) [noise=true , address=145.79.227.47:4265 , piece_id=T4WQNGBMENUE32AYQ2MXFAKR3LRYIOVRCJJE37V2XOKOJ4VVIJYA, seg_pos={0 0}, upload=465546]: Done uploading piece. err: <nil> Dec 27 18:18:51.146239203 (530h49m51.017322248s) [noise=true , address=77.90.31.14:4664 , piece_id=J42YQFOCQEVVX32GWSIDMOGZFXZ7KIO5KH5XUJ36XI3KVXYSKPNA, seg_pos={0 0}, upload=465546]: Done uploading piece. err: <nil> Dec 27 18:18:51.164235901 (530h49m51.035318936s) [noise=true , address=77.90.31.10:4519 , piece_id=EUF7QLG6CJGERWWATE2N7QGNETUZYWDTFOC6LQVXUFQLKZKZSH5Q, seg_pos={0 0}, upload=465546]: Done uploading piece. err: <nil> Dec 27 18:18:51.231133206 (530h49m51.102216241s) [noise=true , address=45.91.116.9:4261 , piece_id=6KKONXYFBLEKXTJMNQ452QDLMGOFTGOML2W77L7AOCW6IB5D3ELQ, seg_pos={0 0}, upload=465546]: Done uploading piece. err: <nil> Dec 27 18:18:51.255307578 (530h49m51.126390623s) [noise=true , address=77.90.31.13:4627 , piece_id=7IXYPYPEI77UG5VYCM6MGHA5JSEG6PGUUSJMDZ7BKWGB2V24MBSA, seg_pos={0 0}, upload=465546]: Done uploading piece. err: <nil> Dec 27 18:18:51.282041364 (530h49m51.153124409s) [noise=true , address=77.90.31.19:4844 , piece_id=57WUY7F7ERDXIXGF5LLR5V5SD3CFEMZHEA4TZY7N2PVHWORCO2GQ, seg_pos={0 0}, upload=465546]: Done uploading piece. err: <nil> Dec 27 18:18:51.282068656 (530h49m51.153151701s) [seg_pos={0 0}, upload=465546]: Segment reached optimal threshold of 13 pieces. Dec 27 18:18:51.282286801 (530h49m51.153369847s) [noise=true , address=77.90.31.12:4589 , piece_id=3MI7AOPYUTKLWMURIBUKOD7DNP4HYUPST63WDVZXLSEYPGFILXKA, seg_pos={0 0}, upload=465546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: write tcp 188.166.225.170:39916->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:39916->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39916->77.90.31.12:4589: use of closed network connection Dec 27 18:18:51.282294697 (530h49m51.153377742s) [noise=true , address=77.90.31.12:4589 , piece_id=3MI7AOPYUTKLWMURIBUKOD7DNP4HYUPST63WDVZXLSEYPGFILXKA, seg_pos={0 0}, upload=465546]: Long tail cancellation due to optimal threshold Dec 27 18:18:51.282436467 (530h49m51.153519522s) [noise=true , address=45.91.116.4:4939 , piece_id=P2MFR7PJ6GJOENSTD2XRPYMTAUTGNQWNLCTVXLTFNZO3RMP7KDDQ, seg_pos={0 0}, upload=465546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: write tcp 188.166.225.170:55750->45.91.116.4:4939: use of closed network connection; write tcp 188.166.225.170:55750->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55750->45.91.116.4:4939: use of closed network connection Dec 27 18:18:51.282606351 (530h49m51.153689386s) [noise=true , address=45.91.116.4:4939 , piece_id=P2MFR7PJ6GJOENSTD2XRPYMTAUTGNQWNLCTVXLTFNZO3RMP7KDDQ, seg_pos={0 0}, upload=465546]: Long tail cancellation due to optimal threshold Dec 27 18:18:51.282667006 (530h49m51.153750051s) [noise=true , address=45.91.116.7:4010 , piece_id=EY2J4FJ4JPQ2I2EK324SJ2Q43TOYQYPLJWXOUJNCRTCXXOZKMATA, seg_pos={0 0}, upload=465546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 27 18:18:51.282676334 (530h49m51.153759379s) [noise=true , address=45.91.116.7:4010 , piece_id=EY2J4FJ4JPQ2I2EK324SJ2Q43TOYQYPLJWXOUJNCRTCXXOZKMATA, seg_pos={0 0}, upload=465546]: Long tail cancellation due to optimal threshold Dec 27 18:18:51.282764662 (530h49m51.153847707s) [seg_pos={0 0}, upload=465546]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 18:18:51.282917964 (530h49m51.154001009s) [upload=465546]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 18:18:51.282924527 (530h49m51.154007572s) [upload=465546]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 18:18:51.598820670 (530h49m51.469903705s) [upload=465546]: Done uploading segments... Dec 27 18:18:51.598840398 (530h49m51.469923433s) [upload=465546]: Upload finished. err: <nil> Duration: 30.002606992s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............X...............................................] duration: 3.613767162s trace: https://jaeger.dev.storj.io/trace/3bddfef0dfa96557 FAIL (2025-12-27 18:18:55 UTC) orbiter warning: context deadline exceeded Duration: 3.61381345s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [X.X..XXXX.X....XXX..XX....X...X.X........X.......X..X.XXXXXX] duration: 30.002597784s trace: https://jaeger.dev.storj.io/trace/5b4659c7e613cd57 FAIL (2025-12-27 18:18:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: RIEDDTGKTNYGJZ5PKXS65FW4ZPNBRE37KSJY7MBMKXNEV7OJTFOQ) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: GYOCSOGUUUATXIZOSTBUFX33KOT767SBDTMOWM6MBJ2JVTH3PDLA) context deadline exceeded ======LOGS====== nts... Dec 27 18:18:18.491353699 (530h49m18.362436734s) [upload=465540]: Got next segment. Inline: false Dec 27 18:18:18.491402692 (530h49m18.362485757s) [upload=465540]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 18:18:18.491408723 (530h49m18.362491768s) [upload=465540]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 18:18:18.736247932 (530h49m18.607330977s) [seg_pos={0 0}, upload=465540]: Begin upload segment... Dec 27 18:18:18.736335028 (530h49m18.607418073s) [seg_pos={0 0}, upload=465540]: Done begin upload segment. Dec 27 18:18:18.736377780 (530h49m18.607460815s) [noise=true , address=145.79.227.20:4438 , piece_id=AGINRHCVMZKPFCODYFQLY2JYCJC6HIUJCBYS7IZH52YX4QNSD77A, seg_pos={0 0}, upload=465540]: Uploading piece... Dec 27 18:18:18.736405072 (530h49m18.607488117s) [noise=true , address=45.91.116.4:4939 , piece_id=WLWISOR624ZQIC5B6U3RIVBPELBMLIN6NFSDCROHMXEMAFYNBKEA, seg_pos={0 0}, upload=465540]: Uploading piece... Dec 27 18:18:18.737052955 (530h49m18.60813599s) [noise=true , address=45.91.116.10:4081 , piece_id=RIEDDTGKTNYGJZ5PKXS65FW4ZPNBRE37KSJY7MBMKXNEV7OJTFOQ, seg_pos={0 0}, upload=465540]: Uploading piece... Dec 27 18:18:18.737142406 (530h49m18.608225451s) [noise=true , address=45.91.116.8:4045 , piece_id=GYOCSOGUUUATXIZOSTBUFX33KOT767SBDTMOWM6MBJ2JVTH3PDLA, seg_pos={0 0}, upload=465540]: Uploading piece... Dec 27 18:18:18.737616077 (530h49m18.608699122s) [noise=true , address=145.79.227.14:4839 , piece_id=LZR55FEJOK4QGQQLJCPMC2K243LS7AIP2ZPI6N2LOVJ4N6J5P3HA, seg_pos={0 0}, upload=465540]: Uploading piece... Dec 27 18:18:18.738073247 (530h49m18.609156282s) [noise=true , address=145.79.227.28:4727 , piece_id=RSN7RCVQGONZN3OZL2IDRVQ4EFH3TOZZPY2OPVITFZKJKO4GVKBQ, seg_pos={0 0}, upload=465540]: Uploading piece... Dec 27 18:18:18.738322883 (530h49m18.609405929s) [noise=true , address=145.79.227.50:4375 , piece_id=DKJHVGZEFSGFYBYLNXTZDS5BCKDKUPI46W2IB2BXMAJ46DDT533Q, seg_pos={0 0}, upload=465540]: Uploading piece... Dec 27 18:18:18.738634226 (530h49m18.609717261s) [noise=true , address=77.90.31.19:4842 , piece_id=K44L4UJ3LNSHESWNV7LZ65TIJFB5PYFS3B76INRAW2HIQ5QHANHA, seg_pos={0 0}, upload=465540]: Uploading piece... Dec 27 18:18:18.738808568 (530h49m18.609891613s) [noise=true , address=145.79.227.8:4329 , piece_id=UBINTOUXOZCCJIR4VIQ7DU6536QJSUIJR2LDR2BG254MEHBEUPKQ, seg_pos={0 0}, upload=465540]: Uploading piece... Dec 27 18:18:18.739089363 (530h49m18.610172408s) [noise=true , address=145.79.227.11:4732 , piece_id=GIVU5AU4XFBTKGWUWPNYT4WHQEFHGFT55WSUDMSK5VIX7QTNKTTQ, seg_pos={0 0}, upload=465540]: Uploading piece... Dec 27 18:18:18.739130150 (530h49m18.610213185s) [noise=true , address=45.91.116.11:4117 , piece_id=FWKXGCKHDT5CHU2RXT5CBS7NIBHX5NZIB4L2IUJVU5WDXSRQCIPA, seg_pos={0 0}, upload=465540]: Uploading piece... Dec 27 18:18:18.739508321 (530h49m18.610591356s) [noise=true , address=145.79.227.17:4971 , piece_id=6QBLQDP5W7G7GP754PBTV3NXUSK5DF7F76UJWQVCZJXZZ6EVE6OQ, seg_pos={0 0}, upload=465540]: Uploading piece... Dec 27 18:18:18.739544770 (530h49m18.610627805s) [noise=true , address=145.79.227.2:4114 , piece_id=BPGZZMY64EUWH4IIB735XKRBNMQ3NN2WV6LQ2QKOH4HUSOPXGI5A, seg_pos={0 0}, upload=465540]: Uploading piece... Dec 27 18:18:18.739796189 (530h49m18.610879224s) [noise=true , address=45.91.116.6:4973 , piece_id=Q25VZBAUV7ZLHNKLHUBBAUCPELSN43NQPIRARPUK3GKCDYP5EUJA, seg_pos={0 0}, upload=465540]: Uploading piece... Dec 27 18:18:18.740012420 (530h49m18.611095455s) [noise=true , address=145.79.227.39:4014 , piece_id=6YBR6DPCGMGITVRGSK4P4XGXWSZINKYK5KOL275GIBPHA5WBIF7Q, seg_pos={0 0}, upload=465540]: Uploading piece... Dec 27 18:18:18.740106600 (530h49m18.611189645s) [noise=true , address=145.79.227.52:4445 , piece_id=OBMW7W5OUIK26KHFQ35P3ZZBRODCTPIQ3KI5OI3LSJEQHQFMTASQ, seg_pos={0 0}, upload=465540]: Uploading piece... Dec 27 18:18:20.444066038 (530h49m20.315149083s) [upload=465540]: Next returned nil segment Dec 27 18:18:20.444079033 (530h49m20.315162068s) [upload=465540]: Waiting for error group managing segments... Dec 27 18:18:20.727202522 (530h49m20.598285567s) [noise=true , address=145.79.227.14:4839 , piece_id=LZR55FEJOK4QGQQLJCPMC2K243LS7AIP2ZPI6N2LOVJ4N6J5P3HA, seg_pos={0 0}, upload=465540]: Done uploading piece. err: <nil> Dec 27 18:18:20.727469550 (530h49m20.598552585s) [noise=true , address=145.79.227.52:4445 , piece_id=OBMW7W5OUIK26KHFQ35P3ZZBRODCTPIQ3KI5OI3LSJEQHQFMTASQ, seg_pos={0 0}, upload=465540]: Done uploading piece. err: <nil> Dec 27 18:18:20.735557541 (530h49m20.606640586s) [noise=true , address=145.79.227.20:4438 , piece_id=AGINRHCVMZKPFCODYFQLY2JYCJC6HIUJCBYS7IZH52YX4QNSD77A, seg_pos={0 0}, upload=465540]: Done uploading piece. err: <nil> Dec 27 18:18:20.736738400 (530h49m20.607821445s) [noise=true , address=145.79.227.2:4114 , piece_id=BPGZZMY64EUWH4IIB735XKRBNMQ3NN2WV6LQ2QKOH4HUSOPXGI5A, seg_pos={0 0}, upload=465540]: Done uploading piece. err: <nil> Dec 27 18:18:20.742784853 (530h49m20.613867888s) [noise=true , address=145.79.227.28:4727 , piece_id=RSN7RCVQGONZN3OZL2IDRVQ4EFH3TOZZPY2OPVITFZKJKO4GVKBQ, seg_pos={0 0}, upload=465540]: Done uploading piece. err: <nil> Dec 27 18:18:20.743686721 (530h49m20.614769756s) [noise=true , address=145.79.227.39:4014 , piece_id=6YBR6DPCGMGITVRGSK4P4XGXWSZINKYK5KOL275GIBPHA5WBIF7Q, seg_pos={0 0}, upload=465540]: Done uploading piece. err: <nil> Dec 27 18:18:20.749983461 (530h49m20.621066506s) [noise=true , address=145.79.227.17:4971 , piece_id=6QBLQDP5W7G7GP754PBTV3NXUSK5DF7F76UJWQVCZJXZZ6EVE6OQ, seg_pos={0 0}, upload=465540]: Done uploading piece. err: <nil> Dec 27 18:18:20.766078810 (530h49m20.637161855s) [noise=true , address=145.79.227.50:4375 , piece_id=DKJHVGZEFSGFYBYLNXTZDS5BCKDKUPI46W2IB2BXMAJ46DDT533Q, seg_pos={0 0}, upload=465540]: Done uploading piece. err: <nil> Dec 27 18:18:20.778041508 (530h49m20.649124543s) [noise=true , address=145.79.227.8:4329 , piece_id=UBINTOUXOZCCJIR4VIQ7DU6536QJSUIJR2LDR2BG254MEHBEUPKQ, seg_pos={0 0}, upload=465540]: Done uploading piece. err: <nil> Dec 27 18:18:20.791689316 (530h49m20.662772361s) [noise=true , address=145.79.227.11:4732 , piece_id=GIVU5AU4XFBTKGWUWPNYT4WHQEFHGFT55WSUDMSK5VIX7QTNKTTQ, seg_pos={0 0}, upload=465540]: Done uploading piece. err: <nil> Dec 27 18:18:21.070354366 (530h49m20.941437411s) [noise=true , address=45.91.116.8:4045 , piece_id=GYOCSOGUUUATXIZOSTBUFX33KOT767SBDTMOWM6MBJ2JVTH3PDLA, seg_pos={0 0}, upload=465540]: Done uploading piece. err: <nil> Dec 27 18:18:21.083512792 (530h49m20.954595837s) [noise=true , address=45.91.116.10:4081 , piece_id=RIEDDTGKTNYGJZ5PKXS65FW4ZPNBRE37KSJY7MBMKXNEV7OJTFOQ, seg_pos={0 0}, upload=465540]: Done uploading piece. err: <nil> Dec 27 18:18:21.183765732 (530h49m21.054848777s) [noise=true , address=45.91.116.6:4973 , piece_id=Q25VZBAUV7ZLHNKLHUBBAUCPELSN43NQPIRARPUK3GKCDYP5EUJA, seg_pos={0 0}, upload=465540]: Done uploading piece. err: <nil> Dec 27 18:18:21.183785700 (530h49m21.054868745s) [seg_pos={0 0}, upload=465540]: Segment reached optimal threshold of 13 pieces. Dec 27 18:18:21.184085170 (530h49m21.055168225s) [noise=true , address=45.91.116.11:4117 , piece_id=FWKXGCKHDT5CHU2RXT5CBS7NIBHX5NZIB4L2IUJVU5WDXSRQCIPA, seg_pos={0 0}, upload=465540]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:54560->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:54560->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54560->45.91.116.11:4117: use of closed network connection Dec 27 18:18:21.184094017 (530h49m21.055177062s) [noise=true , address=45.91.116.11:4117 , piece_id=FWKXGCKHDT5CHU2RXT5CBS7NIBHX5NZIB4L2IUJVU5WDXSRQCIPA, seg_pos={0 0}, upload=465540]: Long tail cancellation due to optimal threshold Dec 27 18:18:21.184186964 (530h49m21.055270009s) [noise=true , address=77.90.31.19:4842 , piece_id=K44L4UJ3LNSHESWNV7LZ65TIJFB5PYFS3B76INRAW2HIQ5QHANHA, seg_pos={0 0}, upload=465540]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Dec 27 18:18:21.184200730 (530h49m21.055283765s) [noise=true , address=77.90.31.19:4842 , piece_id=K44L4UJ3LNSHESWNV7LZ65TIJFB5PYFS3B76INRAW2HIQ5QHANHA, seg_pos={0 0}, upload=465540]: Long tail cancellation due to optimal threshold Dec 27 18:18:21.184432932 (530h49m21.055515967s) [noise=true , address=45.91.116.4:4939 , piece_id=WLWISOR624ZQIC5B6U3RIVBPELBMLIN6NFSDCROHMXEMAFYNBKEA, seg_pos={0 0}, upload=465540]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: write tcp 188.166.225.170:52058->45.91.116.4:4939: use of closed network connection; write tcp 188.166.225.170:52058->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52058->45.91.116.4:4939: use of closed network connection Dec 27 18:18:21.184443983 (530h49m21.055527018s) [noise=true , address=45.91.116.4:4939 , piece_id=WLWISOR624ZQIC5B6U3RIVBPELBMLIN6NFSDCROHMXEMAFYNBKEA, seg_pos={0 0}, upload=465540]: Long tail cancellation due to optimal threshold Dec 27 18:18:21.184468320 (530h49m21.055551365s) [seg_pos={0 0}, upload=465540]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 18:18:21.184609388 (530h49m21.055692433s) [upload=465540]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 18:18:21.184616882 (530h49m21.055699927s) [upload=465540]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 18:18:21.483976558 (530h49m21.355059593s) [upload=465540]: Done uploading segments... Dec 27 18:18:21.484001997 (530h49m21.355085042s) [upload=465540]: Upload finished. err: <nil> Duration: 30.002620698s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............X..............................................] duration: 3.529841912s trace: https://jaeger.dev.storj.io/trace/3bddfef0dfa96557 FAIL (2025-12-27 18:18:20 UTC) orbiter warning: context deadline exceeded Duration: 3.52987719s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............X.............................................] duration: 3.596999995s trace: https://jaeger.dev.storj.io/trace/3bddfef0dfa96557 FAIL (2025-12-27 18:18:05 UTC) orbiter warning: context deadline exceeded Duration: 3.597029121s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [.X.X..XXXX.X....XXX..XX....X...X.X........X.......X..X.XXXXX] duration: 30.003293218s trace: https://jaeger.dev.storj.io/trace/1052af1d0e62e357 FAIL (2025-12-27 18:17:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 465521]: Starting upload Dec 27 18:17:07.015090265 (530h48m6.88617331s) [upload=465521]: Done starting upload Dec 27 18:17:07.015216264 (530h48m6.886299299s) [upload=465521]: Uploading segments... Dec 27 18:17:07.015322516 (530h48m6.886405551s) [upload=465521]: Got next segment. Inline: false Dec 27 18:17:07.015356591 (530h48m6.886439626s) [upload=465521]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 18:17:07.015363054 (530h48m6.886446099s) [upload=465521]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 18:17:07.262170781 (530h48m7.133253816s) [seg_pos={0 0}, upload=465521]: Begin upload segment... Dec 27 18:17:07.262216527 (530h48m7.133299573s) [seg_pos={0 0}, upload=465521]: Done begin upload segment. Dec 27 18:17:07.262233920 (530h48m7.133316965s) [noise=true , address=45.91.116.8:4048 , piece_id=EYXHVA2Z2RUREE6SATUWBW3NNN336GE743KRBDAUCEF4MTJBKWJA, seg_pos={0 0}, upload=465521]: Uploading piece... Dec 27 18:17:07.262245673 (530h48m7.133328708s) [noise=true , address=145.79.227.14:4838 , piece_id=TZ7BIFSPIRG3GOJ73VNYY44CSGKBTDKM72NDSNTO7UI7ZFEFGQ7Q, seg_pos={0 0}, upload=465521]: Uploading piece... Dec 27 18:17:07.262890721 (530h48m7.133973756s) [noise=true , address=45.91.116.7:4011 , piece_id=RTLANRN54FO3ONQXP3NKX5GFZ3KDXLSMDOHCH7EGT6Y67ETRBWHA, seg_pos={0 0}, upload=465521]: Uploading piece... Dec 27 18:17:07.263400332 (530h48m7.134483377s) [noise=true , address=145.79.227.50:4375 , piece_id=ABVM3DB2BPM5UR5CIVQ7LWR7777T2BP7IZBSN6ZEW32M6U6FFQZA, seg_pos={0 0}, upload=465521]: Uploading piece... Dec 27 18:17:07.263663253 (530h48m7.134746298s) [noise=true , address=77.90.31.16:4735 , piece_id=QPO4RPAYEROUVNVU234EHYISBEGIHIBD65FBCZ5XRVS6QCI2ZYWQ, seg_pos={0 0}, upload=465521]: Uploading piece... Dec 27 18:17:07.265395942 (530h48m7.136479007s) [noise=true , address=145.79.227.38:4977 , piece_id=6HXJI4YBX5KPUMAQSTMQAMJOPIL4P2ZQSZ5XLOCOLCJNXXHR25WA, seg_pos={0 0}, upload=465521]: Uploading piece... Dec 27 18:17:07.265878180 (530h48m7.136961215s) [noise=true , address=45.91.116.13:4154 , piece_id=5DZ7SAYKG5GQAXWL262WLFJXSR2XE2MUKR25O34H2FDHTIVI7YOQ, seg_pos={0 0}, upload=465521]: Uploading piece... Dec 27 18:17:07.267988629 (530h48m7.139071664s) [noise=true , address=45.91.116.5:4300 , piece_id=RYLJOW5TUWYXSORJGH3V22XMURNRTFDSBVN3JUD4OG4E5TIPI5TA, seg_pos={0 0}, upload=465521]: Uploading piece... Dec 27 18:17:07.269188344 (530h48m7.140271389s) [noise=true , address=145.79.227.28:4726 , piece_id=VSWGY6OLJCDWJSBCDQEHZS2YG32M5VTWSQ56T4VNODOMOWBT2YMQ, seg_pos={0 0}, upload=465521]: Uploading piece... Dec 27 18:17:07.270284761 (530h48m7.141367806s) [noise=true , address=145.79.227.11:4730 , piece_id=62GJCVVSEZI5GOOXWX6RZ7WJANBKHGW3ZCK5EWSQVYBXCCBJ5N2Q, seg_pos={0 0}, upload=465521]: Uploading piece... Dec 27 18:17:07.271407949 (530h48m7.142490995s) [noise=true , address=145.79.227.37:4566 , piece_id=SC4MJHWKP4AOBQG7B7NXLLJHA2ACRGDSAMC2WMXSOKTKP5THL2IA, seg_pos={0 0}, upload=465521]: Uploading piece... Dec 27 18:17:07.271965752 (530h48m7.143048817s) [noise=true , address=45.91.116.9:4263 , piece_id=R2RKSCBXN3DTPGFJSMDMUTXZFBGRTSED55WIB3CS67IREKYCUFKQ, seg_pos={0 0}, upload=465521]: Uploading piece... Dec 27 18:17:07.273364075 (530h48m7.14444712s) [noise=true , address=77.90.31.17:4773 , piece_id=IRWYVPKRBPCTQONR755HIFK6S6WCNICZAKYLL2XTVOHH45KFTZYA, seg_pos={0 0}, upload=465521]: Uploading piece... Dec 27 18:17:07.274248439 (530h48m7.145331484s) [noise=true , address=145.79.227.15:4874 , piece_id=52JPEKZDSO4PRW7QAWYYWCJKDWIZLIWA46TMDW6DMKG467G525SA, seg_pos={0 0}, upload=465521]: Uploading piece... Dec 27 18:17:07.275163982 (530h48m7.146247027s) [noise=true , address=145.79.227.42:4123 , piece_id=5Z7Y4LV4JZSMO2UND37YVRKR3SHVQBFS4K5XNOUGTTV6YPZBBQUQ, seg_pos={0 0}, upload=465521]: Uploading piece... Dec 27 18:17:07.276835434 (530h48m7.147918469s) [noise=true , address=145.79.227.32:4834 , piece_id=ZPOZLIFZ544XDTPZJ7N6ODBDPWDLX75E3N2L6WROGF5FBP3VDZOA, seg_pos={0 0}, upload=465521]: Uploading piece... Dec 27 18:17:08.877609861 (530h48m8.748692896s) [upload=465521]: Next returned nil segment Dec 27 18:17:08.877624570 (530h48m8.748707615s) [upload=465521]: Waiting for error group managing segments... Dec 27 18:17:09.262385694 (530h48m9.133468739s) [noise=true , address=145.79.227.14:4838 , piece_id=TZ7BIFSPIRG3GOJ73VNYY44CSGKBTDKM72NDSNTO7UI7ZFEFGQ7Q, seg_pos={0 0}, upload=465521]: Done uploading piece. err: <nil> Dec 27 18:17:09.266501982 (530h48m9.137585027s) [noise=true , address=145.79.227.42:4123 , piece_id=5Z7Y4LV4JZSMO2UND37YVRKR3SHVQBFS4K5XNOUGTTV6YPZBBQUQ, seg_pos={0 0}, upload=465521]: Done uploading piece. err: <nil> Dec 27 18:17:09.267329899 (530h48m9.138412934s) [noise=true , address=145.79.227.50:4375 , piece_id=ABVM3DB2BPM5UR5CIVQ7LWR7777T2BP7IZBSN6ZEW32M6U6FFQZA, seg_pos={0 0}, upload=465521]: Done uploading piece. err: <nil> Dec 27 18:17:09.267913531 (530h48m9.138996576s) [noise=true , address=145.79.227.15:4874 , piece_id=52JPEKZDSO4PRW7QAWYYWCJKDWIZLIWA46TMDW6DMKG467G525SA, seg_pos={0 0}, upload=465521]: Done uploading piece. err: <nil> Dec 27 18:17:09.269919189 (530h48m9.141002234s) [noise=true , address=145.79.227.32:4834 , piece_id=ZPOZLIFZ544XDTPZJ7N6ODBDPWDLX75E3N2L6WROGF5FBP3VDZOA, seg_pos={0 0}, upload=465521]: Done uploading piece. err: <nil> Dec 27 18:17:09.271956930 (530h48m9.143039975s) [noise=true , address=145.79.227.11:4730 , piece_id=62GJCVVSEZI5GOOXWX6RZ7WJANBKHGW3ZCK5EWSQVYBXCCBJ5N2Q, seg_pos={0 0}, upload=465521]: Done uploading piece. err: <nil> Dec 27 18:17:09.275666283 (530h48m9.146749329s) [noise=true , address=145.79.227.38:4977 , piece_id=6HXJI4YBX5KPUMAQSTMQAMJOPIL4P2ZQSZ5XLOCOLCJNXXHR25WA, seg_pos={0 0}, upload=465521]: Done uploading piece. err: <nil> Dec 27 18:17:09.293240187 (530h48m9.164323222s) [noise=true , address=145.79.227.37:4566 , piece_id=SC4MJHWKP4AOBQG7B7NXLLJHA2ACRGDSAMC2WMXSOKTKP5THL2IA, seg_pos={0 0}, upload=465521]: Done uploading piece. err: <nil> Dec 27 18:17:09.295782919 (530h48m9.166865954s) [noise=true , address=145.79.227.28:4726 , piece_id=VSWGY6OLJCDWJSBCDQEHZS2YG32M5VTWSQ56T4VNODOMOWBT2YMQ, seg_pos={0 0}, upload=465521]: Done uploading piece. err: <nil> Dec 27 18:17:09.495405466 (530h48m9.366488501s) [noise=true , address=45.91.116.9:4263 , piece_id=R2RKSCBXN3DTPGFJSMDMUTXZFBGRTSED55WIB3CS67IREKYCUFKQ, seg_pos={0 0}, upload=465521]: Done uploading piece. err: <nil> Dec 27 18:17:09.559514432 (530h48m9.430597467s) [noise=true , address=77.90.31.16:4735 , piece_id=QPO4RPAYEROUVNVU234EHYISBEGIHIBD65FBCZ5XRVS6QCI2ZYWQ, seg_pos={0 0}, upload=465521]: Done uploading piece. err: <nil> Dec 27 18:17:09.574652658 (530h48m9.445735693s) [noise=true , address=45.91.116.13:4154 , piece_id=5DZ7SAYKG5GQAXWL262WLFJXSR2XE2MUKR25O34H2FDHTIVI7YOQ, seg_pos={0 0}, upload=465521]: Done uploading piece. err: <nil> Dec 27 18:17:09.609507127 (530h48m9.480590172s) [noise=true , address=45.91.116.5:4300 , piece_id=RYLJOW5TUWYXSORJGH3V22XMURNRTFDSBVN3JUD4OG4E5TIPI5TA, seg_pos={0 0}, upload=465521]: Done uploading piece. err: <nil> Dec 27 18:17:09.609528308 (530h48m9.480611343s) [seg_pos={0 0}, upload=465521]: Segment reached optimal threshold of 13 pieces. Dec 27 18:17:09.609738017 (530h48m9.480821052s) [noise=true , address=77.90.31.17:4773 , piece_id=IRWYVPKRBPCTQONR755HIFK6S6WCNICZAKYLL2XTVOHH45KFTZYA, seg_pos={0 0}, upload=465521]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: write tcp 188.166.225.170:48428->77.90.31.17:4773: use of closed network connection; write tcp 188.166.225.170:48428->77.90.31.17:4773: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48428->77.90.31.17:4773: use of closed network connection Dec 27 18:17:09.609745571 (530h48m9.480828616s) [noise=true , address=77.90.31.17:4773 , piece_id=IRWYVPKRBPCTQONR755HIFK6S6WCNICZAKYLL2XTVOHH45KFTZYA, seg_pos={0 0}, upload=465521]: Long tail cancellation due to optimal threshold Dec 27 18:17:09.610085008 (530h48m9.481168053s) [noise=true , address=45.91.116.7:4011 , piece_id=RTLANRN54FO3ONQXP3NKX5GFZ3KDXLSMDOHCH7EGT6Y67ETRBWHA, seg_pos={0 0}, upload=465521]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: read tcp 188.166.225.170:37048->45.91.116.7:4011: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 18:17:09.610103041 (530h48m9.481186077s) [noise=true , address=45.91.116.7:4011 , piece_id=RTLANRN54FO3ONQXP3NKX5GFZ3KDXLSMDOHCH7EGT6Y67ETRBWHA, seg_pos={0 0}, upload=465521]: Long tail cancellation due to optimal threshold Dec 27 18:17:09.610167835 (530h48m9.48125087s) [noise=true , address=45.91.116.8:4048 , piece_id=EYXHVA2Z2RUREE6SATUWBW3NNN336GE743KRBDAUCEF4MTJBKWJA, seg_pos={0 0}, upload=465521]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 27 18:17:09.610176632 (530h48m9.481259678s) [noise=true , address=45.91.116.8:4048 , piece_id=EYXHVA2Z2RUREE6SATUWBW3NNN336GE743KRBDAUCEF4MTJBKWJA, seg_pos={0 0}, upload=465521]: Long tail cancellation due to optimal threshold Dec 27 18:17:09.610204645 (530h48m9.48128769s) [seg_pos={0 0}, upload=465521]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 18:17:09.610278435 (530h48m9.48136148s) [upload=465521]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 18:17:09.610282704 (530h48m9.481365749s) [upload=465521]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 18:17:09.896924453 (530h48m9.768007509s) [upload=465521]: Done uploading segments... Dec 27 18:17:09.896945012 (530h48m9.768028047s) [upload=465521]: Upload finished. err: <nil> Duration: 30.003311954s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............X............................................] duration: 3.573784618s trace: https://jaeger.dev.storj.io/trace/3bddfef0dfa96557 FAIL (2025-12-27 18:17:07 UTC) orbiter warning: context deadline exceeded Duration: 3.573842018s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [X.X.X..XXXX.X....XXX..XX....X...X.X........X.......X..X.XXXX] duration: 30.00423068s trace: https://jaeger.dev.storj.io/trace/757969d2cd84dd00 FAIL (2025-12-27 18:16:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, Piece ID: LJCIHM3FZNFVU4O4ZV4MWC6SKNGVDCYBXAVU7XD3C64ZLJGN3N3Q) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: DE3PLUNUBWJD6RE5AARM5SDPWP3U32MFFBEDPR2USHXAUCYPXI5A) context deadline exceeded ======LOGS====== Dec 27 18:16:05.344283394 (530h47m5.215366439s) [upload=465506]: Starting upload Dec 27 18:16:05.344338429 (530h47m5.215421464s) [upload=465506]: Done starting upload Dec 27 18:16:05.344483014 (530h47m5.215566059s) [upload=465506]: Uploading segments... Dec 27 18:16:05.344619965 (530h47m5.215703s) [upload=465506]: Got next segment. Inline: false Dec 27 18:16:05.344670161 (530h47m5.215753206s) [upload=465506]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 18:16:05.344675441 (530h47m5.215758496s) [upload=465506]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 18:16:05.590055691 (530h47m5.461138736s) [seg_pos={0 0}, upload=465506]: Begin upload segment... Dec 27 18:16:05.590113772 (530h47m5.461196807s) [seg_pos={0 0}, upload=465506]: Done begin upload segment. Dec 27 18:16:05.590151454 (530h47m5.461234489s) [noise=true , address=77.90.31.17:4773 , piece_id=LJCIHM3FZNFVU4O4ZV4MWC6SKNGVDCYBXAVU7XD3C64ZLJGN3N3Q, seg_pos={0 0}, upload=465506]: Uploading piece... Dec 27 18:16:05.590160130 (530h47m5.461243175s) [noise=true , address=145.79.227.48:4303 , piece_id=P5D5X5ORIZBYHDEIFC6TLCRVAGSPFNUKI5RTOUWULEYH6FTU4Y3Q, seg_pos={0 0}, upload=465506]: Uploading piece... Dec 27 18:16:05.590260161 (530h47m5.461343206s) [noise=true , address=145.79.227.15:4874 , piece_id=4FASBXBJPFNFFAETNGIKINVEHL76275PN2UXLWZS3WG3DCUO6RPA, seg_pos={0 0}, upload=465506]: Uploading piece... Dec 27 18:16:05.590798556 (530h47m5.461881601s) [noise=true , address=145.79.227.12:4765 , piece_id=7NIDT3MEPQBEPLIWYTX76YSDZ4RBCTNK7JNWRLHXIE3RJDHJ44PQ, seg_pos={0 0}, upload=465506]: Uploading piece... Dec 27 18:16:05.594884366 (530h47m5.465967401s) [noise=true , address=145.79.227.16:4911 , piece_id=EMTR4GXW55JY4E2RZ67AAF3IVAZZ2AR5T43BWCNCQ77XL5LRTW2A, seg_pos={0 0}, upload=465506]: Uploading piece... Dec 27 18:16:05.595319084 (530h47m5.466402129s) [noise=true , address=45.91.116.11:4120 , piece_id=Z5LI3ELXXRSIFZ33TOYWNXK6KFI6LB33SWZVBDIEQZA7WJKZQMAQ, seg_pos={0 0}, upload=465506]: Uploading piece... Dec 27 18:16:05.595701281 (530h47m5.466784326s) [noise=true , address=45.91.116.13:4153 , piece_id=RLOQLYRD5PCJ63NJZQ4OOZWMNJUPJTLY6GKQG4IMEKM75YGIA7HA, seg_pos={0 0}, upload=465506]: Uploading piece... Dec 27 18:16:05.596156879 (530h47m5.467239924s) [noise=true , address=145.79.227.5:4223 , piece_id=H7W3YYWUVSTHAJUKWRK74ADPKMKE4AFU6YUNXHHUIY5NZVILXCZA, seg_pos={0 0}, upload=465506]: Uploading piece... Dec 27 18:16:05.596551229 (530h47m5.467634275s) [noise=true , address=145.79.227.46:4602 , piece_id=EADSJHGMTKAO23CZPINBVQB7EAHQHJUKGR4OTNQPPTG2FGCVKK6Q, seg_pos={0 0}, upload=465506]: Uploading piece... Dec 27 18:16:05.596947114 (530h47m5.468030159s) [noise=true , address=145.79.227.4:4185 , piece_id=2YKY3G2AUAQDRNWVISS2HCAXK4QEL33MXPPRR6FN3NJZV6ZBX2LQ, seg_pos={0 0}, upload=465506]: Uploading piece... Dec 27 18:16:05.597367414 (530h47m5.468450459s) [noise=true , address=145.79.227.52:4445 , piece_id=YW254NXZDJKGZK32HZMZRA4RDB5QRWMEG74HDNM7OWISRV7ASDWQ, seg_pos={0 0}, upload=465506]: Uploading piece... Dec 27 18:16:05.598046316 (530h47m5.469129351s) [noise=true , address=77.90.31.10:4517 , piece_id=5UJOKHD6ZGY4XQWYAKTFHRBQZ6LHUWHFGQJZGLPHLGUQFTGLUTLA, seg_pos={0 0}, upload=465506]: Uploading piece... Dec 27 18:16:05.600463590 (530h47m5.471546625s) [noise=true , address=145.79.227.33:4869 , piece_id=KZRFP5EDEEFZYN5DWERCEGHIHLAXZDVNM6E4EF77ZZ6LIQTOWYEQ, seg_pos={0 0}, upload=465506]: Uploading piece... Dec 27 18:16:05.605348341 (530h47m5.476431376s) [noise=true , address=145.79.227.42:4123 , piece_id=6EQ5GKA3LAAHTW6KBS665BCVAT5DPU7LE7PEP2DEGO77AYOC5WHQ, seg_pos={0 0}, upload=465506]: Uploading piece... Dec 27 18:16:05.606974087 (530h47m5.478057172s) [noise=true , address=45.91.116.5:4299 , piece_id=DE3PLUNUBWJD6RE5AARM5SDPWP3U32MFFBEDPR2USHXAUCYPXI5A, seg_pos={0 0}, upload=465506]: Uploading piece... Dec 27 18:16:05.608368873 (530h47m5.479451908s) [noise=true , address=145.79.227.25:4619 , piece_id=KOMO72DG6NKMRJJCWK77R3EVK2MVP5CAGEQ4QV54DVTAVIKSDXYQ, seg_pos={0 0}, upload=465506]: Uploading piece... Dec 27 18:16:07.301127251 (530h47m7.172210297s) [upload=465506]: Next returned nil segment Dec 27 18:16:07.301150276 (530h47m7.172233321s) [upload=465506]: Waiting for error group managing segments... Dec 27 18:16:07.587049310 (530h47m7.458132355s) [noise=true , address=145.79.227.15:4874 , piece_id=4FASBXBJPFNFFAETNGIKINVEHL76275PN2UXLWZS3WG3DCUO6RPA, seg_pos={0 0}, upload=465506]: Done uploading piece. err: <nil> Dec 27 18:16:07.595408908 (530h47m7.466491953s) [noise=true , address=145.79.227.52:4445 , piece_id=YW254NXZDJKGZK32HZMZRA4RDB5QRWMEG74HDNM7OWISRV7ASDWQ, seg_pos={0 0}, upload=465506]: Done uploading piece. err: <nil> Dec 27 18:16:07.596246653 (530h47m7.467329698s) [noise=true , address=145.79.227.46:4602 , piece_id=EADSJHGMTKAO23CZPINBVQB7EAHQHJUKGR4OTNQPPTG2FGCVKK6Q, seg_pos={0 0}, upload=465506]: Done uploading piece. err: <nil> Dec 27 18:16:07.598299001 (530h47m7.469382036s) [noise=true , address=145.79.227.12:4765 , piece_id=7NIDT3MEPQBEPLIWYTX76YSDZ4RBCTNK7JNWRLHXIE3RJDHJ44PQ, seg_pos={0 0}, upload=465506]: Done uploading piece. err: <nil> Dec 27 18:16:07.607931092 (530h47m7.479014138s) [noise=true , address=145.79.227.48:4303 , piece_id=P5D5X5ORIZBYHDEIFC6TLCRVAGSPFNUKI5RTOUWULEYH6FTU4Y3Q, seg_pos={0 0}, upload=465506]: Done uploading piece. err: <nil> Dec 27 18:16:07.615121675 (530h47m7.48620472s) [noise=true , address=145.79.227.4:4185 , piece_id=2YKY3G2AUAQDRNWVISS2HCAXK4QEL33MXPPRR6FN3NJZV6ZBX2LQ, seg_pos={0 0}, upload=465506]: Done uploading piece. err: <nil> Dec 27 18:16:07.616187795 (530h47m7.487270831s) [noise=true , address=145.79.227.5:4223 , piece_id=H7W3YYWUVSTHAJUKWRK74ADPKMKE4AFU6YUNXHHUIY5NZVILXCZA, seg_pos={0 0}, upload=465506]: Done uploading piece. err: <nil> Dec 27 18:16:07.629081978 (530h47m7.500165023s) [noise=true , address=145.79.227.16:4911 , piece_id=EMTR4GXW55JY4E2RZ67AAF3IVAZZ2AR5T43BWCNCQ77XL5LRTW2A, seg_pos={0 0}, upload=465506]: Done uploading piece. err: <nil> Dec 27 18:16:07.630899488 (530h47m7.501982534s) [noise=true , address=145.79.227.25:4619 , piece_id=KOMO72DG6NKMRJJCWK77R3EVK2MVP5CAGEQ4QV54DVTAVIKSDXYQ, seg_pos={0 0}, upload=465506]: Done uploading piece. err: <nil> Dec 27 18:16:07.631569654 (530h47m7.502652699s) [noise=true , address=145.79.227.42:4123 , piece_id=6EQ5GKA3LAAHTW6KBS665BCVAT5DPU7LE7PEP2DEGO77AYOC5WHQ, seg_pos={0 0}, upload=465506]: Done uploading piece. err: <nil> Dec 27 18:16:07.791626823 (530h47m7.662709858s) [noise=true , address=77.90.31.17:4773 , piece_id=LJCIHM3FZNFVU4O4ZV4MWC6SKNGVDCYBXAVU7XD3C64ZLJGN3N3Q, seg_pos={0 0}, upload=465506]: Done uploading piece. err: <nil> Dec 27 18:16:07.803088648 (530h47m7.674171693s) [noise=true , address=77.90.31.10:4517 , piece_id=5UJOKHD6ZGY4XQWYAKTFHRBQZ6LHUWHFGQJZGLPHLGUQFTGLUTLA, seg_pos={0 0}, upload=465506]: Done uploading piece. err: <nil> Dec 27 18:16:07.863893813 (530h47m7.734976848s) [noise=true , address=45.91.116.5:4299 , piece_id=DE3PLUNUBWJD6RE5AARM5SDPWP3U32MFFBEDPR2USHXAUCYPXI5A, seg_pos={0 0}, upload=465506]: Done uploading piece. err: <nil> Dec 27 18:16:07.863919822 (530h47m7.735002868s) [seg_pos={0 0}, upload=465506]: Segment reached optimal threshold of 13 pieces. Dec 27 18:16:07.864162735 (530h47m7.73524579s) [noise=true , address=45.91.116.13:4153 , piece_id=RLOQLYRD5PCJ63NJZQ4OOZWMNJUPJTLY6GKQG4IMEKM75YGIA7HA, seg_pos={0 0}, upload=465506]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: write tcp 188.166.225.170:32792->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:32792->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32792->45.91.116.13:4153: use of closed network connection Dec 27 18:16:07.864173585 (530h47m7.735256661s) [noise=true , address=45.91.116.13:4153 , piece_id=RLOQLYRD5PCJ63NJZQ4OOZWMNJUPJTLY6GKQG4IMEKM75YGIA7HA, seg_pos={0 0}, upload=465506]: Long tail cancellation due to optimal threshold Dec 27 18:16:07.864395297 (530h47m7.735478352s) [noise=true , address=145.79.227.33:4869 , piece_id=KZRFP5EDEEFZYN5DWERCEGHIHLAXZDVNM6E4EF77ZZ6LIQTOWYEQ, seg_pos={0 0}, upload=465506]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF Dec 27 18:16:07.864428390 (530h47m7.735511445s) [noise=true , address=145.79.227.33:4869 , piece_id=KZRFP5EDEEFZYN5DWERCEGHIHLAXZDVNM6E4EF77ZZ6LIQTOWYEQ, seg_pos={0 0}, upload=465506]: Long tail cancellation due to optimal threshold Dec 27 18:16:07.864582033 (530h47m7.735665078s) [noise=true , address=45.91.116.11:4120 , piece_id=Z5LI3ELXXRSIFZ33TOYWNXK6KFI6LB33SWZVBDIEQZA7WJKZQMAQ, seg_pos={0 0}, upload=465506]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 27 18:16:07.864592393 (530h47m7.735675458s) [noise=true , address=45.91.116.11:4120 , piece_id=Z5LI3ELXXRSIFZ33TOYWNXK6KFI6LB33SWZVBDIEQZA7WJKZQMAQ, seg_pos={0 0}, upload=465506]: Long tail cancellation due to optimal threshold Dec 27 18:16:07.864621108 (530h47m7.735704153s) [seg_pos={0 0}, upload=465506]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 18:16:07.864716680 (530h47m7.735799725s) [upload=465506]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 18:16:07.864723713 (530h47m7.735806758s) [upload=465506]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 18:16:08.160809013 (530h47m8.031892058s) [upload=465506]: Done uploading segments... Dec 27 18:16:08.160830273 (530h47m8.031913308s) [upload=465506]: Upload finished. err: <nil> Duration: 30.004281306s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................X...........................................] duration: 3.580819726s trace: https://jaeger.dev.storj.io/trace/3bddfef0dfa96557 FAIL (2025-12-27 18:16:19 UTC) orbiter warning: context deadline exceeded Duration: 3.580848781s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.X.X.X..XXXX.X....XXX..XX....X...X.X........X.......X..X.XXX] duration: 30.003465693s trace: https://jaeger.dev.storj.io/trace/62eebb3fa7bb4444 FAIL (2025-12-27 18:15:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, Piece ID: USL3JMPWANZDRPOBSWECYE2BCB7BFY3IQZPO2VZBQFAZHYBF3FMA) context deadline exceeded; piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: 6ANM4YJOYQBVSYHKR7MOZNQWSQCPJHVRDLHV2IBHIMRXLED7USHQ) context deadline exceeded; piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: PRRHOOBWPLY3HNTNONSQRZQO7LZUOMUH23WYMXUEQGJN2X42MIRQ) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: MQ62SQ2ZJJWSVK45IBEIOUVJ6SMJTEV4S5LBHC5CFS2DQENM2QZQ) context deadline exceeded ======LOGS====== 5 (530h46m29.84069601s) [upload=465499]: Done starting upload Dec 27 18:15:29.969674562 (530h46m29.840757598s) [upload=465499]: Uploading segments... Dec 27 18:15:29.969911033 (530h46m29.840994078s) [upload=465499]: Got next segment. Inline: false Dec 27 18:15:29.969961779 (530h46m29.841044824s) [upload=465499]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 18:15:29.969968371 (530h46m29.841051417s) [upload=465499]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 18:15:30.213936822 (530h46m30.085019857s) [seg_pos={0 0}, upload=465499]: Begin upload segment... Dec 27 18:15:30.214005783 (530h46m30.085088819s) [seg_pos={0 0}, upload=465499]: Done begin upload segment. Dec 27 18:15:30.214049116 (530h46m30.085132151s) [noise=true , address=145.79.227.48:4302 , piece_id=SDWGVGGIGVUJG5QAFBBAH7G56WIEQKN6ZZURSPJPISESA33GO34Q, seg_pos={0 0}, upload=465499]: Uploading piece... Dec 27 18:15:30.214224991 (530h46m30.085308026s) [noise=true , address=145.79.227.7:4295 , piece_id=B44OXY5HHCHBXO6W2HGOOQTEBIOA5T3FQSGE5ETJHACGJRWFRDHQ, seg_pos={0 0}, upload=465499]: Uploading piece... Dec 27 18:15:30.214707079 (530h46m30.085790114s) [noise=true , address=145.79.227.25:4618 , piece_id=SYQTFNYWS5KTDGTCLLP6GF5QSFBOIGX2FFG7UAIAYRXB5AB7YJJA, seg_pos={0 0}, upload=465499]: Uploading piece... Dec 27 18:15:30.214717759 (530h46m30.085800804s) [noise=true , address=145.79.227.41:4086 , piece_id=UWWGOL7GBV5P5JWQHQU4BOCX4FRYZ3I5WWP4W72FVLDVAU7OPVQQ, seg_pos={0 0}, upload=465499]: Uploading piece... Dec 27 18:15:30.215069288 (530h46m30.086152323s) [noise=true , address=45.91.116.4:4940 , piece_id=AIP3R2LBG7LQ5K7UL7NYQGXA2TVHPGDKNHHUYQ2IJMFH5XMYRJPQ, seg_pos={0 0}, upload=465499]: Uploading piece... Dec 27 18:15:30.215100928 (530h46m30.086183993s) [noise=true , address=77.90.31.18:4805 , piece_id=6ANM4YJOYQBVSYHKR7MOZNQWSQCPJHVRDLHV2IBHIMRXLED7USHQ, seg_pos={0 0}, upload=465499]: Uploading piece... Dec 27 18:15:30.215409075 (530h46m30.08649212s) [noise=true , address=145.79.227.51:4410 , piece_id=M3RHFT3C7ZPJIN4D4R7NKLVGLNW7UMLQ5275MRB6V23T5KALMICA, seg_pos={0 0}, upload=465499]: Uploading piece... Dec 27 18:15:30.215451816 (530h46m30.086534851s) [noise=true , address=77.90.31.16:4735 , piece_id=PRRHOOBWPLY3HNTNONSQRZQO7LZUOMUH23WYMXUEQGJN2X42MIRQ, seg_pos={0 0}, upload=465499]: Uploading piece... Dec 27 18:15:30.215711111 (530h46m30.086794146s) [noise=true , address=45.91.116.10:4082 , piece_id=MQ62SQ2ZJJWSVK45IBEIOUVJ6SMJTEV4S5LBHC5CFS2DQENM2QZQ, seg_pos={0 0}, upload=465499]: Uploading piece... Dec 27 18:15:30.215785202 (530h46m30.086868237s) [noise=true , address=45.91.116.6:4973 , piece_id=FVGIYNX3LRUHLOQQUSQZJCGYB4YBHXMALK6WPEUWCLUIGDIA4SOA, seg_pos={0 0}, upload=465499]: Uploading piece... Dec 27 18:15:30.215975193 (530h46m30.087058228s) [noise=true , address=145.79.227.30:4762 , piece_id=3LADVONE63MPMXUSIFFFLV4IVKIYRVHFIFVOD5TYEV6MVBCG3AYA, seg_pos={0 0}, upload=465499]: Uploading piece... Dec 27 18:15:30.216137973 (530h46m30.087221018s) [noise=true , address=145.79.227.38:4979 , piece_id=ABSTDODLB7TPARN3KMJWUZAJBSY4A3MXF2T5LBDDDH5GXBH6MZCA, seg_pos={0 0}, upload=465499]: Uploading piece... Dec 27 18:15:30.216236541 (530h46m30.087319576s) [noise=true , address=145.79.227.12:4767 , piece_id=PFNI7ALXGAQ6STOHJUR62JED326OBRUL4KLVLI4PI4WUGMIC3TRQ, seg_pos={0 0}, upload=465499]: Uploading piece... Dec 27 18:15:30.216415291 (530h46m30.087498326s) [noise=true , address=145.79.227.21:4474 , piece_id=BY77R44FURZNFNV3YKYT4XNJTOSYBQOXAAZBXLG5RTH4MR5PQ55Q, seg_pos={0 0}, upload=465499]: Uploading piece... Dec 27 18:15:30.216553244 (530h46m30.087636279s) [noise=true , address=77.90.31.11:4556 , piece_id=USL3JMPWANZDRPOBSWECYE2BCB7BFY3IQZPO2VZBQFAZHYBF3FMA, seg_pos={0 0}, upload=465499]: Uploading piece... Dec 27 18:15:30.216795666 (530h46m30.087878711s) [noise=true , address=145.79.227.52:4445 , piece_id=VC6P2KBNSD4D3TX363JTTTPTPKY7GG2MUMZ3EOSZNCTSPJESIZZA, seg_pos={0 0}, upload=465499]: Uploading piece... Dec 27 18:15:31.923787379 (530h46m31.794870424s) [upload=465499]: Next returned nil segment Dec 27 18:15:31.923837425 (530h46m31.79492047s) [upload=465499]: Waiting for error group managing segments... Dec 27 18:15:32.205741582 (530h46m32.076824627s) [noise=true , address=145.79.227.41:4086 , piece_id=UWWGOL7GBV5P5JWQHQU4BOCX4FRYZ3I5WWP4W72FVLDVAU7OPVQQ, seg_pos={0 0}, upload=465499]: Done uploading piece. err: <nil> Dec 27 18:15:32.206089064 (530h46m32.077172099s) [noise=true , address=145.79.227.25:4618 , piece_id=SYQTFNYWS5KTDGTCLLP6GF5QSFBOIGX2FFG7UAIAYRXB5AB7YJJA, seg_pos={0 0}, upload=465499]: Done uploading piece. err: <nil> Dec 27 18:15:32.219951650 (530h46m32.091034685s) [noise=true , address=145.79.227.7:4295 , piece_id=B44OXY5HHCHBXO6W2HGOOQTEBIOA5T3FQSGE5ETJHACGJRWFRDHQ, seg_pos={0 0}, upload=465499]: Done uploading piece. err: <nil> Dec 27 18:15:32.225069085 (530h46m32.09615213s) [noise=true , address=145.79.227.51:4410 , piece_id=M3RHFT3C7ZPJIN4D4R7NKLVGLNW7UMLQ5275MRB6V23T5KALMICA, seg_pos={0 0}, upload=465499]: Done uploading piece. err: <nil> Dec 27 18:15:32.236712436 (530h46m32.107795481s) [noise=true , address=145.79.227.48:4302 , piece_id=SDWGVGGIGVUJG5QAFBBAH7G56WIEQKN6ZZURSPJPISESA33GO34Q, seg_pos={0 0}, upload=465499]: Done uploading piece. err: <nil> Dec 27 18:15:32.236937805 (530h46m32.10802085s) [noise=true , address=145.79.227.38:4979 , piece_id=ABSTDODLB7TPARN3KMJWUZAJBSY4A3MXF2T5LBDDDH5GXBH6MZCA, seg_pos={0 0}, upload=465499]: Done uploading piece. err: <nil> Dec 27 18:15:32.242082001 (530h46m32.113165036s) [noise=true , address=145.79.227.12:4767 , piece_id=PFNI7ALXGAQ6STOHJUR62JED326OBRUL4KLVLI4PI4WUGMIC3TRQ, seg_pos={0 0}, upload=465499]: Done uploading piece. err: <nil> Dec 27 18:15:32.258198369 (530h46m32.129281404s) [noise=true , address=145.79.227.21:4474 , piece_id=BY77R44FURZNFNV3YKYT4XNJTOSYBQOXAAZBXLG5RTH4MR5PQ55Q, seg_pos={0 0}, upload=465499]: Done uploading piece. err: <nil> Dec 27 18:15:32.268042664 (530h46m32.139125699s) [noise=true , address=145.79.227.52:4445 , piece_id=VC6P2KBNSD4D3TX363JTTTPTPKY7GG2MUMZ3EOSZNCTSPJESIZZA, seg_pos={0 0}, upload=465499]: Done uploading piece. err: <nil> Dec 27 18:15:32.387293951 (530h46m32.258376996s) [noise=true , address=77.90.31.18:4805 , piece_id=6ANM4YJOYQBVSYHKR7MOZNQWSQCPJHVRDLHV2IBHIMRXLED7USHQ, seg_pos={0 0}, upload=465499]: Done uploading piece. err: <nil> Dec 27 18:15:32.438742588 (530h46m32.309825633s) [noise=true , address=77.90.31.11:4556 , piece_id=USL3JMPWANZDRPOBSWECYE2BCB7BFY3IQZPO2VZBQFAZHYBF3FMA, seg_pos={0 0}, upload=465499]: Done uploading piece. err: <nil> Dec 27 18:15:32.465667879 (530h46m32.336750914s) [noise=true , address=45.91.116.10:4082 , piece_id=MQ62SQ2ZJJWSVK45IBEIOUVJ6SMJTEV4S5LBHC5CFS2DQENM2QZQ, seg_pos={0 0}, upload=465499]: Done uploading piece. err: <nil> Dec 27 18:15:32.518857222 (530h46m32.389940267s) [noise=true , address=77.90.31.16:4735 , piece_id=PRRHOOBWPLY3HNTNONSQRZQO7LZUOMUH23WYMXUEQGJN2X42MIRQ, seg_pos={0 0}, upload=465499]: Done uploading piece. err: <nil> Dec 27 18:15:32.518879123 (530h46m32.389962168s) [seg_pos={0 0}, upload=465499]: Segment reached optimal threshold of 13 pieces. Dec 27 18:15:32.519155629 (530h46m32.390238674s) [noise=true , address=45.91.116.6:4973 , piece_id=FVGIYNX3LRUHLOQQUSQZJCGYB4YBHXMALK6WPEUWCLUIGDIA4SOA, seg_pos={0 0}, upload=465499]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: read tcp 188.166.225.170:53650->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 18:15:32.519162001 (530h46m32.390245046s) [noise=true , address=45.91.116.6:4973 , piece_id=FVGIYNX3LRUHLOQQUSQZJCGYB4YBHXMALK6WPEUWCLUIGDIA4SOA, seg_pos={0 0}, upload=465499]: Long tail cancellation due to optimal threshold Dec 27 18:15:32.519269375 (530h46m32.390352411s) [noise=true , address=45.91.116.4:4940 , piece_id=AIP3R2LBG7LQ5K7UL7NYQGXA2TVHPGDKNHHUYQ2IJMFH5XMYRJPQ, seg_pos={0 0}, upload=465499]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: write tcp 188.166.225.170:58276->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:58276->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58276->45.91.116.4:4940: use of closed network connection Dec 27 18:15:32.519296818 (530h46m32.390379863s) [noise=true , address=45.91.116.4:4940 , piece_id=AIP3R2LBG7LQ5K7UL7NYQGXA2TVHPGDKNHHUYQ2IJMFH5XMYRJPQ, seg_pos={0 0}, upload=465499]: Long tail cancellation due to optimal threshold Dec 27 18:15:32.519394664 (530h46m32.39047771s) [noise=true , address=145.79.227.30:4762 , piece_id=3LADVONE63MPMXUSIFFFLV4IVKIYRVHFIFVOD5TYEV6MVBCG3AYA, seg_pos={0 0}, upload=465499]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 27 18:15:32.519402309 (530h46m32.390485354s) [noise=true , address=145.79.227.30:4762 , piece_id=3LADVONE63MPMXUSIFFFLV4IVKIYRVHFIFVOD5TYEV6MVBCG3AYA, seg_pos={0 0}, upload=465499]: Long tail cancellation due to optimal threshold Dec 27 18:15:32.519418680 (530h46m32.390501716s) [seg_pos={0 0}, upload=465499]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 18:15:32.519487091 (530h46m32.390570126s) [upload=465499]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 18:15:32.519492892 (530h46m32.390575927s) [upload=465499]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 18:15:32.825409242 (530h46m32.696492297s) [upload=465499]: Done uploading segments... Dec 27 18:15:32.825430032 (530h46m32.696513067s) [upload=465499]: Upload finished. err: <nil> Duration: 30.003487867s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)