Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................X.....................X.........] duration: 6.033918315s FAIL (2025-10-06 02:53:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: 2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: 7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA) context deadline exceeded ======LOGS====== Oct 6 02:45:41.756337907 (301h14m27.21843039s) [upload=265104]: Starting upload Oct 6 02:45:41.756430433 (301h14m27.218522916s) [upload=265104]: Done starting upload Oct 6 02:45:41.756513701 (301h14m27.218606185s) [upload=265104]: Uploading segments... Oct 6 02:45:41.756752276 (301h14m27.218844759s) [upload=265104]: Got next segment. Inline: false Oct 6 02:45:41.756794536 (301h14m27.218887019s) [upload=265104]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 02:45:41.756800898 (301h14m27.218893381s) [upload=265104]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 02:45:42.009327267 (301h14m27.47141975s) [seg_pos={0 0}, upload=265104]: Begin upload segment... Oct 6 02:45:42.009417419 (301h14m27.471509892s) [seg_pos={0 0}, upload=265104]: Done begin upload segment. Oct 6 02:45:42.009444500 (301h14m27.471536983s) [noise=true , address=77.90.31.14:4662 , piece_id=MPVBPBB63JILLOUF3N7SRLCZ2F74LX7Z4AH7ZFG3BDNBHT4UIXTQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.009561473 (301h14m27.471653956s) [noise=true , address=145.79.227.19:4077 , piece_id=EJIAPZSBMAO77E33V6MEBOQHUXVVYKMAVP53QAZXZSYBYDFP7MMQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.010197574 (301h14m27.472290057s) [noise=true , address=145.79.227.15:4875 , piece_id=PBDTNZ2NWRZZC3VCIQCVR2WLT2AP6662I2SSDNP6BLUDKCJMRK6Q, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.010634216 (301h14m27.472726699s) [noise=true , address=145.79.227.4:4185 , piece_id=3AD745KFD67M4XFNDXMJVEU34RRBQXXE4P3HVJJRDBHK3IIXNYTA, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011073742 (301h14m27.473166225s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011505996 (301h14m27.473598479s) [noise=true , address=45.91.116.10:4081 , piece_id=FTXZ2LOHQMY74KOX6XZC3TLHWXOGDTIY3SZLSPTE7THGKNLMPXRQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011982502 (301h14m27.474074985s) [noise=true , address=45.91.116.6:4974 , piece_id=7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.012488236 (301h14m27.474580719s) [noise=true , address=145.79.227.7:4294 , piece_id=7E3B4BGLCTW6C5DM6PPDJPWGVJS5HEGYPAPXRAMX7XDSG7USPQFQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.013242472 (301h14m27.475334955s) [noise=true , address=45.91.116.13:4154 , piece_id=2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.014529382 (301h14m27.476621855s) [noise=true , address=45.91.116.6:4975 , piece_id=T3VWH5OBQ7FE66GCKHZLEDSIXFVYEU3V52HI4CPD27QL3RH75K7Q, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.014632619 (301h14m27.476725102s) [noise=true , address=45.91.116.11:4117 , piece_id=QKEYHSK5JOL65H5T2KUEAEK6VKTNCZAYXIEZTMB76PV7P5PEVILQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.015990214 (301h14m27.478082697s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.016291928 (301h14m27.478384412s) [noise=true , address=45.91.116.12:4226 , piece_id=NM7SEIKZYUHZLNVSOGYVZWMI7U7X5SOXOJDAEGY5NF3M6QQFPZSQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.017455915 (301h14m27.479548399s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.018864467 (301h14m27.48095694s) [noise=true , address=77.90.31.10:4518 , piece_id=6FGNAMVK6OAGJTZJNKTUJWGRW7MLGMJMGJKMAINWMPW7BV74OO6A, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.020222022 (301h14m27.482314495s) [noise=true , address=45.91.116.11:4118 , piece_id=VUZB3QRFST2T3HJHMEYYZ6CV7CKJTFWQGNOBDTXHGET2MBUIC4HQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:43.755608897 (301h14m29.21770138s) [upload=265104]: Next returned nil segment Oct 6 02:45:43.755631300 (301h14m29.217723773s) [upload=265104]: Waiting for error group managing segments... Oct 6 02:45:44.043930124 (301h14m29.506022597s) [noise=true , address=145.79.227.19:4077 , piece_id=EJIAPZSBMAO77E33V6MEBOQHUXVVYKMAVP53QAZXZSYBYDFP7MMQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.045501296 (301h14m29.507593769s) [noise=true , address=145.79.227.4:4185 , piece_id=3AD745KFD67M4XFNDXMJVEU34RRBQXXE4P3HVJJRDBHK3IIXNYTA, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.057881199 (301h14m29.519973672s) [noise=true , address=145.79.227.15:4875 , piece_id=PBDTNZ2NWRZZC3VCIQCVR2WLT2AP6662I2SSDNP6BLUDKCJMRK6Q, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.068726540 (301h14m29.530819023s) [noise=true , address=145.79.227.7:4294 , piece_id=7E3B4BGLCTW6C5DM6PPDJPWGVJS5HEGYPAPXRAMX7XDSG7USPQFQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.897895712 (301h14m30.359988185s) [noise=true , address=45.91.116.12:4226 , piece_id=NM7SEIKZYUHZLNVSOGYVZWMI7U7X5SOXOJDAEGY5NF3M6QQFPZSQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.252859665 (301h14m30.714952148s) [noise=true , address=77.90.31.14:4662 , piece_id=MPVBPBB63JILLOUF3N7SRLCZ2F74LX7Z4AH7ZFG3BDNBHT4UIXTQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.318952216 (301h14m30.781044699s) [noise=true , address=45.91.116.6:4974 , piece_id=7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.836439779 (301h14m31.298532262s) [noise=true , address=45.91.116.10:4081 , piece_id=FTXZ2LOHQMY74KOX6XZC3TLHWXOGDTIY3SZLSPTE7THGKNLMPXRQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.975895173 (301h14m31.437987656s) [noise=true , address=45.91.116.6:4975 , piece_id=T3VWH5OBQ7FE66GCKHZLEDSIXFVYEU3V52HI4CPD27QL3RH75K7Q, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:46.826703981 (301h14m32.288796454s) [noise=true , address=77.90.31.10:4518 , piece_id=6FGNAMVK6OAGJTZJNKTUJWGRW7MLGMJMGJKMAINWMPW7BV74OO6A, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:48.069759484 (301h14m33.531851968s) [noise=true , address=45.91.116.13:4154 , piece_id=2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:54.791582362 (301h14m40.253674835s) [noise=true , address=45.91.116.11:4118 , piece_id=VUZB3QRFST2T3HJHMEYYZ6CV7CKJTFWQGNOBDTXHGET2MBUIC4HQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:56.329163998 (301h14m41.791256481s) [noise=true , address=45.91.116.11:4117 , piece_id=QKEYHSK5JOL65H5T2KUEAEK6VKTNCZAYXIEZTMB76PV7P5PEVILQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:56.329201379 (301h14m41.791293862s) [seg_pos={0 0}, upload=265104]: Segment reached optimal threshold of 13 pieces. Oct 6 02:45:56.329362146 (301h14m41.791454629s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Oct 6 02:45:56.329371974 (301h14m41.791464457s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329546396 (301h14m41.791638869s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: write tcp 188.166.225.170:57604->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 02:45:56.329556665 (301h14m41.791649148s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329567937 (301h14m41.791660421s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: rpc: dial tcp 77.90.31.19:4841: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 02:45:56.329579870 (301h14m41.791672353s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329599808 (301h14m41.791692291s) [seg_pos={0 0}, upload=265104]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 6 02:45:56.329683116 (301h14m41.7917756s) [upload=265104]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 6 02:45:56.329690771 (301h14m41.791783254s) [upload=265104]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 6 02:45:56.636341641 (301h14m42.098434134s) [upload=265104]: Done uploading segments... Oct 6 02:45:56.636368131 (301h14m42.098460614s) [upload=265104]: Upload finished. err: <nil> Duration: 6.033949404s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................X.....................X........] duration: 6.095605758s FAIL (2025-10-06 02:52:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: 2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: 7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA) context deadline exceeded ======LOGS====== Oct 6 02:45:41.756337907 (301h14m27.21843039s) [upload=265104]: Starting upload Oct 6 02:45:41.756430433 (301h14m27.218522916s) [upload=265104]: Done starting upload Oct 6 02:45:41.756513701 (301h14m27.218606185s) [upload=265104]: Uploading segments... Oct 6 02:45:41.756752276 (301h14m27.218844759s) [upload=265104]: Got next segment. Inline: false Oct 6 02:45:41.756794536 (301h14m27.218887019s) [upload=265104]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 02:45:41.756800898 (301h14m27.218893381s) [upload=265104]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 02:45:42.009327267 (301h14m27.47141975s) [seg_pos={0 0}, upload=265104]: Begin upload segment... Oct 6 02:45:42.009417419 (301h14m27.471509892s) [seg_pos={0 0}, upload=265104]: Done begin upload segment. Oct 6 02:45:42.009444500 (301h14m27.471536983s) [noise=true , address=77.90.31.14:4662 , piece_id=MPVBPBB63JILLOUF3N7SRLCZ2F74LX7Z4AH7ZFG3BDNBHT4UIXTQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.009561473 (301h14m27.471653956s) [noise=true , address=145.79.227.19:4077 , piece_id=EJIAPZSBMAO77E33V6MEBOQHUXVVYKMAVP53QAZXZSYBYDFP7MMQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.010197574 (301h14m27.472290057s) [noise=true , address=145.79.227.15:4875 , piece_id=PBDTNZ2NWRZZC3VCIQCVR2WLT2AP6662I2SSDNP6BLUDKCJMRK6Q, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.010634216 (301h14m27.472726699s) [noise=true , address=145.79.227.4:4185 , piece_id=3AD745KFD67M4XFNDXMJVEU34RRBQXXE4P3HVJJRDBHK3IIXNYTA, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011073742 (301h14m27.473166225s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011505996 (301h14m27.473598479s) [noise=true , address=45.91.116.10:4081 , piece_id=FTXZ2LOHQMY74KOX6XZC3TLHWXOGDTIY3SZLSPTE7THGKNLMPXRQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011982502 (301h14m27.474074985s) [noise=true , address=45.91.116.6:4974 , piece_id=7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.012488236 (301h14m27.474580719s) [noise=true , address=145.79.227.7:4294 , piece_id=7E3B4BGLCTW6C5DM6PPDJPWGVJS5HEGYPAPXRAMX7XDSG7USPQFQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.013242472 (301h14m27.475334955s) [noise=true , address=45.91.116.13:4154 , piece_id=2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.014529382 (301h14m27.476621855s) [noise=true , address=45.91.116.6:4975 , piece_id=T3VWH5OBQ7FE66GCKHZLEDSIXFVYEU3V52HI4CPD27QL3RH75K7Q, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.014632619 (301h14m27.476725102s) [noise=true , address=45.91.116.11:4117 , piece_id=QKEYHSK5JOL65H5T2KUEAEK6VKTNCZAYXIEZTMB76PV7P5PEVILQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.015990214 (301h14m27.478082697s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.016291928 (301h14m27.478384412s) [noise=true , address=45.91.116.12:4226 , piece_id=NM7SEIKZYUHZLNVSOGYVZWMI7U7X5SOXOJDAEGY5NF3M6QQFPZSQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.017455915 (301h14m27.479548399s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.018864467 (301h14m27.48095694s) [noise=true , address=77.90.31.10:4518 , piece_id=6FGNAMVK6OAGJTZJNKTUJWGRW7MLGMJMGJKMAINWMPW7BV74OO6A, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.020222022 (301h14m27.482314495s) [noise=true , address=45.91.116.11:4118 , piece_id=VUZB3QRFST2T3HJHMEYYZ6CV7CKJTFWQGNOBDTXHGET2MBUIC4HQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:43.755608897 (301h14m29.21770138s) [upload=265104]: Next returned nil segment Oct 6 02:45:43.755631300 (301h14m29.217723773s) [upload=265104]: Waiting for error group managing segments... Oct 6 02:45:44.043930124 (301h14m29.506022597s) [noise=true , address=145.79.227.19:4077 , piece_id=EJIAPZSBMAO77E33V6MEBOQHUXVVYKMAVP53QAZXZSYBYDFP7MMQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.045501296 (301h14m29.507593769s) [noise=true , address=145.79.227.4:4185 , piece_id=3AD745KFD67M4XFNDXMJVEU34RRBQXXE4P3HVJJRDBHK3IIXNYTA, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.057881199 (301h14m29.519973672s) [noise=true , address=145.79.227.15:4875 , piece_id=PBDTNZ2NWRZZC3VCIQCVR2WLT2AP6662I2SSDNP6BLUDKCJMRK6Q, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.068726540 (301h14m29.530819023s) [noise=true , address=145.79.227.7:4294 , piece_id=7E3B4BGLCTW6C5DM6PPDJPWGVJS5HEGYPAPXRAMX7XDSG7USPQFQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.897895712 (301h14m30.359988185s) [noise=true , address=45.91.116.12:4226 , piece_id=NM7SEIKZYUHZLNVSOGYVZWMI7U7X5SOXOJDAEGY5NF3M6QQFPZSQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.252859665 (301h14m30.714952148s) [noise=true , address=77.90.31.14:4662 , piece_id=MPVBPBB63JILLOUF3N7SRLCZ2F74LX7Z4AH7ZFG3BDNBHT4UIXTQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.318952216 (301h14m30.781044699s) [noise=true , address=45.91.116.6:4974 , piece_id=7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.836439779 (301h14m31.298532262s) [noise=true , address=45.91.116.10:4081 , piece_id=FTXZ2LOHQMY74KOX6XZC3TLHWXOGDTIY3SZLSPTE7THGKNLMPXRQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.975895173 (301h14m31.437987656s) [noise=true , address=45.91.116.6:4975 , piece_id=T3VWH5OBQ7FE66GCKHZLEDSIXFVYEU3V52HI4CPD27QL3RH75K7Q, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:46.826703981 (301h14m32.288796454s) [noise=true , address=77.90.31.10:4518 , piece_id=6FGNAMVK6OAGJTZJNKTUJWGRW7MLGMJMGJKMAINWMPW7BV74OO6A, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:48.069759484 (301h14m33.531851968s) [noise=true , address=45.91.116.13:4154 , piece_id=2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:54.791582362 (301h14m40.253674835s) [noise=true , address=45.91.116.11:4118 , piece_id=VUZB3QRFST2T3HJHMEYYZ6CV7CKJTFWQGNOBDTXHGET2MBUIC4HQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:56.329163998 (301h14m41.791256481s) [noise=true , address=45.91.116.11:4117 , piece_id=QKEYHSK5JOL65H5T2KUEAEK6VKTNCZAYXIEZTMB76PV7P5PEVILQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:56.329201379 (301h14m41.791293862s) [seg_pos={0 0}, upload=265104]: Segment reached optimal threshold of 13 pieces. Oct 6 02:45:56.329362146 (301h14m41.791454629s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Oct 6 02:45:56.329371974 (301h14m41.791464457s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329546396 (301h14m41.791638869s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: write tcp 188.166.225.170:57604->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 02:45:56.329556665 (301h14m41.791649148s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329567937 (301h14m41.791660421s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: rpc: dial tcp 77.90.31.19:4841: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 02:45:56.329579870 (301h14m41.791672353s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329599808 (301h14m41.791692291s) [seg_pos={0 0}, upload=265104]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 6 02:45:56.329683116 (301h14m41.7917756s) [upload=265104]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 6 02:45:56.329690771 (301h14m41.791783254s) [upload=265104]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 6 02:45:56.636341641 (301h14m42.098434134s) [upload=265104]: Done uploading segments... Oct 6 02:45:56.636368131 (301h14m42.098460614s) [upload=265104]: Upload finished. err: <nil> Duration: 6.095641406s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................X.....................X.......] duration: 6.267293873s FAIL (2025-10-06 02:51:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: 2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: 7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA) context deadline exceeded ======LOGS====== Oct 6 02:45:41.756337907 (301h14m27.21843039s) [upload=265104]: Starting upload Oct 6 02:45:41.756430433 (301h14m27.218522916s) [upload=265104]: Done starting upload Oct 6 02:45:41.756513701 (301h14m27.218606185s) [upload=265104]: Uploading segments... Oct 6 02:45:41.756752276 (301h14m27.218844759s) [upload=265104]: Got next segment. Inline: false Oct 6 02:45:41.756794536 (301h14m27.218887019s) [upload=265104]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 02:45:41.756800898 (301h14m27.218893381s) [upload=265104]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 02:45:42.009327267 (301h14m27.47141975s) [seg_pos={0 0}, upload=265104]: Begin upload segment... Oct 6 02:45:42.009417419 (301h14m27.471509892s) [seg_pos={0 0}, upload=265104]: Done begin upload segment. Oct 6 02:45:42.009444500 (301h14m27.471536983s) [noise=true , address=77.90.31.14:4662 , piece_id=MPVBPBB63JILLOUF3N7SRLCZ2F74LX7Z4AH7ZFG3BDNBHT4UIXTQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.009561473 (301h14m27.471653956s) [noise=true , address=145.79.227.19:4077 , piece_id=EJIAPZSBMAO77E33V6MEBOQHUXVVYKMAVP53QAZXZSYBYDFP7MMQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.010197574 (301h14m27.472290057s) [noise=true , address=145.79.227.15:4875 , piece_id=PBDTNZ2NWRZZC3VCIQCVR2WLT2AP6662I2SSDNP6BLUDKCJMRK6Q, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.010634216 (301h14m27.472726699s) [noise=true , address=145.79.227.4:4185 , piece_id=3AD745KFD67M4XFNDXMJVEU34RRBQXXE4P3HVJJRDBHK3IIXNYTA, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011073742 (301h14m27.473166225s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011505996 (301h14m27.473598479s) [noise=true , address=45.91.116.10:4081 , piece_id=FTXZ2LOHQMY74KOX6XZC3TLHWXOGDTIY3SZLSPTE7THGKNLMPXRQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011982502 (301h14m27.474074985s) [noise=true , address=45.91.116.6:4974 , piece_id=7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.012488236 (301h14m27.474580719s) [noise=true , address=145.79.227.7:4294 , piece_id=7E3B4BGLCTW6C5DM6PPDJPWGVJS5HEGYPAPXRAMX7XDSG7USPQFQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.013242472 (301h14m27.475334955s) [noise=true , address=45.91.116.13:4154 , piece_id=2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.014529382 (301h14m27.476621855s) [noise=true , address=45.91.116.6:4975 , piece_id=T3VWH5OBQ7FE66GCKHZLEDSIXFVYEU3V52HI4CPD27QL3RH75K7Q, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.014632619 (301h14m27.476725102s) [noise=true , address=45.91.116.11:4117 , piece_id=QKEYHSK5JOL65H5T2KUEAEK6VKTNCZAYXIEZTMB76PV7P5PEVILQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.015990214 (301h14m27.478082697s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.016291928 (301h14m27.478384412s) [noise=true , address=45.91.116.12:4226 , piece_id=NM7SEIKZYUHZLNVSOGYVZWMI7U7X5SOXOJDAEGY5NF3M6QQFPZSQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.017455915 (301h14m27.479548399s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.018864467 (301h14m27.48095694s) [noise=true , address=77.90.31.10:4518 , piece_id=6FGNAMVK6OAGJTZJNKTUJWGRW7MLGMJMGJKMAINWMPW7BV74OO6A, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.020222022 (301h14m27.482314495s) [noise=true , address=45.91.116.11:4118 , piece_id=VUZB3QRFST2T3HJHMEYYZ6CV7CKJTFWQGNOBDTXHGET2MBUIC4HQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:43.755608897 (301h14m29.21770138s) [upload=265104]: Next returned nil segment Oct 6 02:45:43.755631300 (301h14m29.217723773s) [upload=265104]: Waiting for error group managing segments... Oct 6 02:45:44.043930124 (301h14m29.506022597s) [noise=true , address=145.79.227.19:4077 , piece_id=EJIAPZSBMAO77E33V6MEBOQHUXVVYKMAVP53QAZXZSYBYDFP7MMQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.045501296 (301h14m29.507593769s) [noise=true , address=145.79.227.4:4185 , piece_id=3AD745KFD67M4XFNDXMJVEU34RRBQXXE4P3HVJJRDBHK3IIXNYTA, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.057881199 (301h14m29.519973672s) [noise=true , address=145.79.227.15:4875 , piece_id=PBDTNZ2NWRZZC3VCIQCVR2WLT2AP6662I2SSDNP6BLUDKCJMRK6Q, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.068726540 (301h14m29.530819023s) [noise=true , address=145.79.227.7:4294 , piece_id=7E3B4BGLCTW6C5DM6PPDJPWGVJS5HEGYPAPXRAMX7XDSG7USPQFQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.897895712 (301h14m30.359988185s) [noise=true , address=45.91.116.12:4226 , piece_id=NM7SEIKZYUHZLNVSOGYVZWMI7U7X5SOXOJDAEGY5NF3M6QQFPZSQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.252859665 (301h14m30.714952148s) [noise=true , address=77.90.31.14:4662 , piece_id=MPVBPBB63JILLOUF3N7SRLCZ2F74LX7Z4AH7ZFG3BDNBHT4UIXTQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.318952216 (301h14m30.781044699s) [noise=true , address=45.91.116.6:4974 , piece_id=7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.836439779 (301h14m31.298532262s) [noise=true , address=45.91.116.10:4081 , piece_id=FTXZ2LOHQMY74KOX6XZC3TLHWXOGDTIY3SZLSPTE7THGKNLMPXRQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.975895173 (301h14m31.437987656s) [noise=true , address=45.91.116.6:4975 , piece_id=T3VWH5OBQ7FE66GCKHZLEDSIXFVYEU3V52HI4CPD27QL3RH75K7Q, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:46.826703981 (301h14m32.288796454s) [noise=true , address=77.90.31.10:4518 , piece_id=6FGNAMVK6OAGJTZJNKTUJWGRW7MLGMJMGJKMAINWMPW7BV74OO6A, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:48.069759484 (301h14m33.531851968s) [noise=true , address=45.91.116.13:4154 , piece_id=2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:54.791582362 (301h14m40.253674835s) [noise=true , address=45.91.116.11:4118 , piece_id=VUZB3QRFST2T3HJHMEYYZ6CV7CKJTFWQGNOBDTXHGET2MBUIC4HQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:56.329163998 (301h14m41.791256481s) [noise=true , address=45.91.116.11:4117 , piece_id=QKEYHSK5JOL65H5T2KUEAEK6VKTNCZAYXIEZTMB76PV7P5PEVILQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:56.329201379 (301h14m41.791293862s) [seg_pos={0 0}, upload=265104]: Segment reached optimal threshold of 13 pieces. Oct 6 02:45:56.329362146 (301h14m41.791454629s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Oct 6 02:45:56.329371974 (301h14m41.791464457s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329546396 (301h14m41.791638869s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: write tcp 188.166.225.170:57604->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 02:45:56.329556665 (301h14m41.791649148s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329567937 (301h14m41.791660421s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: rpc: dial tcp 77.90.31.19:4841: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 02:45:56.329579870 (301h14m41.791672353s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329599808 (301h14m41.791692291s) [seg_pos={0 0}, upload=265104]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 6 02:45:56.329683116 (301h14m41.7917756s) [upload=265104]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 6 02:45:56.329690771 (301h14m41.791783254s) [upload=265104]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 6 02:45:56.636341641 (301h14m42.098434134s) [upload=265104]: Done uploading segments... Oct 6 02:45:56.636368131 (301h14m42.098460614s) [upload=265104]: Upload finished. err: <nil> Duration: 6.267328317s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................X.....................X......] duration: 6.059410115s FAIL (2025-10-06 02:50:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: 2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: 7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA) context deadline exceeded ======LOGS====== Oct 6 02:45:41.756337907 (301h14m27.21843039s) [upload=265104]: Starting upload Oct 6 02:45:41.756430433 (301h14m27.218522916s) [upload=265104]: Done starting upload Oct 6 02:45:41.756513701 (301h14m27.218606185s) [upload=265104]: Uploading segments... Oct 6 02:45:41.756752276 (301h14m27.218844759s) [upload=265104]: Got next segment. Inline: false Oct 6 02:45:41.756794536 (301h14m27.218887019s) [upload=265104]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 02:45:41.756800898 (301h14m27.218893381s) [upload=265104]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 02:45:42.009327267 (301h14m27.47141975s) [seg_pos={0 0}, upload=265104]: Begin upload segment... Oct 6 02:45:42.009417419 (301h14m27.471509892s) [seg_pos={0 0}, upload=265104]: Done begin upload segment. Oct 6 02:45:42.009444500 (301h14m27.471536983s) [noise=true , address=77.90.31.14:4662 , piece_id=MPVBPBB63JILLOUF3N7SRLCZ2F74LX7Z4AH7ZFG3BDNBHT4UIXTQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.009561473 (301h14m27.471653956s) [noise=true , address=145.79.227.19:4077 , piece_id=EJIAPZSBMAO77E33V6MEBOQHUXVVYKMAVP53QAZXZSYBYDFP7MMQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.010197574 (301h14m27.472290057s) [noise=true , address=145.79.227.15:4875 , piece_id=PBDTNZ2NWRZZC3VCIQCVR2WLT2AP6662I2SSDNP6BLUDKCJMRK6Q, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.010634216 (301h14m27.472726699s) [noise=true , address=145.79.227.4:4185 , piece_id=3AD745KFD67M4XFNDXMJVEU34RRBQXXE4P3HVJJRDBHK3IIXNYTA, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011073742 (301h14m27.473166225s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011505996 (301h14m27.473598479s) [noise=true , address=45.91.116.10:4081 , piece_id=FTXZ2LOHQMY74KOX6XZC3TLHWXOGDTIY3SZLSPTE7THGKNLMPXRQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011982502 (301h14m27.474074985s) [noise=true , address=45.91.116.6:4974 , piece_id=7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.012488236 (301h14m27.474580719s) [noise=true , address=145.79.227.7:4294 , piece_id=7E3B4BGLCTW6C5DM6PPDJPWGVJS5HEGYPAPXRAMX7XDSG7USPQFQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.013242472 (301h14m27.475334955s) [noise=true , address=45.91.116.13:4154 , piece_id=2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.014529382 (301h14m27.476621855s) [noise=true , address=45.91.116.6:4975 , piece_id=T3VWH5OBQ7FE66GCKHZLEDSIXFVYEU3V52HI4CPD27QL3RH75K7Q, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.014632619 (301h14m27.476725102s) [noise=true , address=45.91.116.11:4117 , piece_id=QKEYHSK5JOL65H5T2KUEAEK6VKTNCZAYXIEZTMB76PV7P5PEVILQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.015990214 (301h14m27.478082697s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.016291928 (301h14m27.478384412s) [noise=true , address=45.91.116.12:4226 , piece_id=NM7SEIKZYUHZLNVSOGYVZWMI7U7X5SOXOJDAEGY5NF3M6QQFPZSQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.017455915 (301h14m27.479548399s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.018864467 (301h14m27.48095694s) [noise=true , address=77.90.31.10:4518 , piece_id=6FGNAMVK6OAGJTZJNKTUJWGRW7MLGMJMGJKMAINWMPW7BV74OO6A, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.020222022 (301h14m27.482314495s) [noise=true , address=45.91.116.11:4118 , piece_id=VUZB3QRFST2T3HJHMEYYZ6CV7CKJTFWQGNOBDTXHGET2MBUIC4HQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:43.755608897 (301h14m29.21770138s) [upload=265104]: Next returned nil segment Oct 6 02:45:43.755631300 (301h14m29.217723773s) [upload=265104]: Waiting for error group managing segments... Oct 6 02:45:44.043930124 (301h14m29.506022597s) [noise=true , address=145.79.227.19:4077 , piece_id=EJIAPZSBMAO77E33V6MEBOQHUXVVYKMAVP53QAZXZSYBYDFP7MMQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.045501296 (301h14m29.507593769s) [noise=true , address=145.79.227.4:4185 , piece_id=3AD745KFD67M4XFNDXMJVEU34RRBQXXE4P3HVJJRDBHK3IIXNYTA, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.057881199 (301h14m29.519973672s) [noise=true , address=145.79.227.15:4875 , piece_id=PBDTNZ2NWRZZC3VCIQCVR2WLT2AP6662I2SSDNP6BLUDKCJMRK6Q, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.068726540 (301h14m29.530819023s) [noise=true , address=145.79.227.7:4294 , piece_id=7E3B4BGLCTW6C5DM6PPDJPWGVJS5HEGYPAPXRAMX7XDSG7USPQFQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.897895712 (301h14m30.359988185s) [noise=true , address=45.91.116.12:4226 , piece_id=NM7SEIKZYUHZLNVSOGYVZWMI7U7X5SOXOJDAEGY5NF3M6QQFPZSQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.252859665 (301h14m30.714952148s) [noise=true , address=77.90.31.14:4662 , piece_id=MPVBPBB63JILLOUF3N7SRLCZ2F74LX7Z4AH7ZFG3BDNBHT4UIXTQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.318952216 (301h14m30.781044699s) [noise=true , address=45.91.116.6:4974 , piece_id=7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.836439779 (301h14m31.298532262s) [noise=true , address=45.91.116.10:4081 , piece_id=FTXZ2LOHQMY74KOX6XZC3TLHWXOGDTIY3SZLSPTE7THGKNLMPXRQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.975895173 (301h14m31.437987656s) [noise=true , address=45.91.116.6:4975 , piece_id=T3VWH5OBQ7FE66GCKHZLEDSIXFVYEU3V52HI4CPD27QL3RH75K7Q, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:46.826703981 (301h14m32.288796454s) [noise=true , address=77.90.31.10:4518 , piece_id=6FGNAMVK6OAGJTZJNKTUJWGRW7MLGMJMGJKMAINWMPW7BV74OO6A, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:48.069759484 (301h14m33.531851968s) [noise=true , address=45.91.116.13:4154 , piece_id=2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:54.791582362 (301h14m40.253674835s) [noise=true , address=45.91.116.11:4118 , piece_id=VUZB3QRFST2T3HJHMEYYZ6CV7CKJTFWQGNOBDTXHGET2MBUIC4HQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:56.329163998 (301h14m41.791256481s) [noise=true , address=45.91.116.11:4117 , piece_id=QKEYHSK5JOL65H5T2KUEAEK6VKTNCZAYXIEZTMB76PV7P5PEVILQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:56.329201379 (301h14m41.791293862s) [seg_pos={0 0}, upload=265104]: Segment reached optimal threshold of 13 pieces. Oct 6 02:45:56.329362146 (301h14m41.791454629s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Oct 6 02:45:56.329371974 (301h14m41.791464457s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329546396 (301h14m41.791638869s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: write tcp 188.166.225.170:57604->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 02:45:56.329556665 (301h14m41.791649148s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329567937 (301h14m41.791660421s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: rpc: dial tcp 77.90.31.19:4841: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 02:45:56.329579870 (301h14m41.791672353s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329599808 (301h14m41.791692291s) [seg_pos={0 0}, upload=265104]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 6 02:45:56.329683116 (301h14m41.7917756s) [upload=265104]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 6 02:45:56.329690771 (301h14m41.791783254s) [upload=265104]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 6 02:45:56.636341641 (301h14m42.098434134s) [upload=265104]: Done uploading segments... Oct 6 02:45:56.636368131 (301h14m42.098460614s) [upload=265104]: Upload finished. err: <nil> Duration: 6.059445061s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................X.....................X.....] duration: 6.002496217s FAIL (2025-10-06 02:49:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: 2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: 7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA) context deadline exceeded ======LOGS====== Oct 6 02:45:41.756337907 (301h14m27.21843039s) [upload=265104]: Starting upload Oct 6 02:45:41.756430433 (301h14m27.218522916s) [upload=265104]: Done starting upload Oct 6 02:45:41.756513701 (301h14m27.218606185s) [upload=265104]: Uploading segments... Oct 6 02:45:41.756752276 (301h14m27.218844759s) [upload=265104]: Got next segment. Inline: false Oct 6 02:45:41.756794536 (301h14m27.218887019s) [upload=265104]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 02:45:41.756800898 (301h14m27.218893381s) [upload=265104]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 02:45:42.009327267 (301h14m27.47141975s) [seg_pos={0 0}, upload=265104]: Begin upload segment... Oct 6 02:45:42.009417419 (301h14m27.471509892s) [seg_pos={0 0}, upload=265104]: Done begin upload segment. Oct 6 02:45:42.009444500 (301h14m27.471536983s) [noise=true , address=77.90.31.14:4662 , piece_id=MPVBPBB63JILLOUF3N7SRLCZ2F74LX7Z4AH7ZFG3BDNBHT4UIXTQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.009561473 (301h14m27.471653956s) [noise=true , address=145.79.227.19:4077 , piece_id=EJIAPZSBMAO77E33V6MEBOQHUXVVYKMAVP53QAZXZSYBYDFP7MMQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.010197574 (301h14m27.472290057s) [noise=true , address=145.79.227.15:4875 , piece_id=PBDTNZ2NWRZZC3VCIQCVR2WLT2AP6662I2SSDNP6BLUDKCJMRK6Q, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.010634216 (301h14m27.472726699s) [noise=true , address=145.79.227.4:4185 , piece_id=3AD745KFD67M4XFNDXMJVEU34RRBQXXE4P3HVJJRDBHK3IIXNYTA, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011073742 (301h14m27.473166225s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011505996 (301h14m27.473598479s) [noise=true , address=45.91.116.10:4081 , piece_id=FTXZ2LOHQMY74KOX6XZC3TLHWXOGDTIY3SZLSPTE7THGKNLMPXRQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011982502 (301h14m27.474074985s) [noise=true , address=45.91.116.6:4974 , piece_id=7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.012488236 (301h14m27.474580719s) [noise=true , address=145.79.227.7:4294 , piece_id=7E3B4BGLCTW6C5DM6PPDJPWGVJS5HEGYPAPXRAMX7XDSG7USPQFQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.013242472 (301h14m27.475334955s) [noise=true , address=45.91.116.13:4154 , piece_id=2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.014529382 (301h14m27.476621855s) [noise=true , address=45.91.116.6:4975 , piece_id=T3VWH5OBQ7FE66GCKHZLEDSIXFVYEU3V52HI4CPD27QL3RH75K7Q, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.014632619 (301h14m27.476725102s) [noise=true , address=45.91.116.11:4117 , piece_id=QKEYHSK5JOL65H5T2KUEAEK6VKTNCZAYXIEZTMB76PV7P5PEVILQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.015990214 (301h14m27.478082697s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.016291928 (301h14m27.478384412s) [noise=true , address=45.91.116.12:4226 , piece_id=NM7SEIKZYUHZLNVSOGYVZWMI7U7X5SOXOJDAEGY5NF3M6QQFPZSQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.017455915 (301h14m27.479548399s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.018864467 (301h14m27.48095694s) [noise=true , address=77.90.31.10:4518 , piece_id=6FGNAMVK6OAGJTZJNKTUJWGRW7MLGMJMGJKMAINWMPW7BV74OO6A, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.020222022 (301h14m27.482314495s) [noise=true , address=45.91.116.11:4118 , piece_id=VUZB3QRFST2T3HJHMEYYZ6CV7CKJTFWQGNOBDTXHGET2MBUIC4HQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:43.755608897 (301h14m29.21770138s) [upload=265104]: Next returned nil segment Oct 6 02:45:43.755631300 (301h14m29.217723773s) [upload=265104]: Waiting for error group managing segments... Oct 6 02:45:44.043930124 (301h14m29.506022597s) [noise=true , address=145.79.227.19:4077 , piece_id=EJIAPZSBMAO77E33V6MEBOQHUXVVYKMAVP53QAZXZSYBYDFP7MMQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.045501296 (301h14m29.507593769s) [noise=true , address=145.79.227.4:4185 , piece_id=3AD745KFD67M4XFNDXMJVEU34RRBQXXE4P3HVJJRDBHK3IIXNYTA, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.057881199 (301h14m29.519973672s) [noise=true , address=145.79.227.15:4875 , piece_id=PBDTNZ2NWRZZC3VCIQCVR2WLT2AP6662I2SSDNP6BLUDKCJMRK6Q, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.068726540 (301h14m29.530819023s) [noise=true , address=145.79.227.7:4294 , piece_id=7E3B4BGLCTW6C5DM6PPDJPWGVJS5HEGYPAPXRAMX7XDSG7USPQFQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.897895712 (301h14m30.359988185s) [noise=true , address=45.91.116.12:4226 , piece_id=NM7SEIKZYUHZLNVSOGYVZWMI7U7X5SOXOJDAEGY5NF3M6QQFPZSQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.252859665 (301h14m30.714952148s) [noise=true , address=77.90.31.14:4662 , piece_id=MPVBPBB63JILLOUF3N7SRLCZ2F74LX7Z4AH7ZFG3BDNBHT4UIXTQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.318952216 (301h14m30.781044699s) [noise=true , address=45.91.116.6:4974 , piece_id=7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.836439779 (301h14m31.298532262s) [noise=true , address=45.91.116.10:4081 , piece_id=FTXZ2LOHQMY74KOX6XZC3TLHWXOGDTIY3SZLSPTE7THGKNLMPXRQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.975895173 (301h14m31.437987656s) [noise=true , address=45.91.116.6:4975 , piece_id=T3VWH5OBQ7FE66GCKHZLEDSIXFVYEU3V52HI4CPD27QL3RH75K7Q, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:46.826703981 (301h14m32.288796454s) [noise=true , address=77.90.31.10:4518 , piece_id=6FGNAMVK6OAGJTZJNKTUJWGRW7MLGMJMGJKMAINWMPW7BV74OO6A, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:48.069759484 (301h14m33.531851968s) [noise=true , address=45.91.116.13:4154 , piece_id=2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:54.791582362 (301h14m40.253674835s) [noise=true , address=45.91.116.11:4118 , piece_id=VUZB3QRFST2T3HJHMEYYZ6CV7CKJTFWQGNOBDTXHGET2MBUIC4HQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:56.329163998 (301h14m41.791256481s) [noise=true , address=45.91.116.11:4117 , piece_id=QKEYHSK5JOL65H5T2KUEAEK6VKTNCZAYXIEZTMB76PV7P5PEVILQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:56.329201379 (301h14m41.791293862s) [seg_pos={0 0}, upload=265104]: Segment reached optimal threshold of 13 pieces. Oct 6 02:45:56.329362146 (301h14m41.791454629s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Oct 6 02:45:56.329371974 (301h14m41.791464457s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329546396 (301h14m41.791638869s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: write tcp 188.166.225.170:57604->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 02:45:56.329556665 (301h14m41.791649148s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329567937 (301h14m41.791660421s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: rpc: dial tcp 77.90.31.19:4841: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 02:45:56.329579870 (301h14m41.791672353s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329599808 (301h14m41.791692291s) [seg_pos={0 0}, upload=265104]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 6 02:45:56.329683116 (301h14m41.7917756s) [upload=265104]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 6 02:45:56.329690771 (301h14m41.791783254s) [upload=265104]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 6 02:45:56.636341641 (301h14m42.098434134s) [upload=265104]: Done uploading segments... Oct 6 02:45:56.636368131 (301h14m42.098460614s) [upload=265104]: Upload finished. err: <nil> Duration: 6.002526344s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................X.....................X....] duration: 5.942141096s FAIL (2025-10-06 02:49:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: 2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: 7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA) context deadline exceeded ======LOGS====== Oct 6 02:45:41.756337907 (301h14m27.21843039s) [upload=265104]: Starting upload Oct 6 02:45:41.756430433 (301h14m27.218522916s) [upload=265104]: Done starting upload Oct 6 02:45:41.756513701 (301h14m27.218606185s) [upload=265104]: Uploading segments... Oct 6 02:45:41.756752276 (301h14m27.218844759s) [upload=265104]: Got next segment. Inline: false Oct 6 02:45:41.756794536 (301h14m27.218887019s) [upload=265104]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 02:45:41.756800898 (301h14m27.218893381s) [upload=265104]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 02:45:42.009327267 (301h14m27.47141975s) [seg_pos={0 0}, upload=265104]: Begin upload segment... Oct 6 02:45:42.009417419 (301h14m27.471509892s) [seg_pos={0 0}, upload=265104]: Done begin upload segment. Oct 6 02:45:42.009444500 (301h14m27.471536983s) [noise=true , address=77.90.31.14:4662 , piece_id=MPVBPBB63JILLOUF3N7SRLCZ2F74LX7Z4AH7ZFG3BDNBHT4UIXTQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.009561473 (301h14m27.471653956s) [noise=true , address=145.79.227.19:4077 , piece_id=EJIAPZSBMAO77E33V6MEBOQHUXVVYKMAVP53QAZXZSYBYDFP7MMQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.010197574 (301h14m27.472290057s) [noise=true , address=145.79.227.15:4875 , piece_id=PBDTNZ2NWRZZC3VCIQCVR2WLT2AP6662I2SSDNP6BLUDKCJMRK6Q, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.010634216 (301h14m27.472726699s) [noise=true , address=145.79.227.4:4185 , piece_id=3AD745KFD67M4XFNDXMJVEU34RRBQXXE4P3HVJJRDBHK3IIXNYTA, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011073742 (301h14m27.473166225s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011505996 (301h14m27.473598479s) [noise=true , address=45.91.116.10:4081 , piece_id=FTXZ2LOHQMY74KOX6XZC3TLHWXOGDTIY3SZLSPTE7THGKNLMPXRQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011982502 (301h14m27.474074985s) [noise=true , address=45.91.116.6:4974 , piece_id=7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.012488236 (301h14m27.474580719s) [noise=true , address=145.79.227.7:4294 , piece_id=7E3B4BGLCTW6C5DM6PPDJPWGVJS5HEGYPAPXRAMX7XDSG7USPQFQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.013242472 (301h14m27.475334955s) [noise=true , address=45.91.116.13:4154 , piece_id=2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.014529382 (301h14m27.476621855s) [noise=true , address=45.91.116.6:4975 , piece_id=T3VWH5OBQ7FE66GCKHZLEDSIXFVYEU3V52HI4CPD27QL3RH75K7Q, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.014632619 (301h14m27.476725102s) [noise=true , address=45.91.116.11:4117 , piece_id=QKEYHSK5JOL65H5T2KUEAEK6VKTNCZAYXIEZTMB76PV7P5PEVILQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.015990214 (301h14m27.478082697s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.016291928 (301h14m27.478384412s) [noise=true , address=45.91.116.12:4226 , piece_id=NM7SEIKZYUHZLNVSOGYVZWMI7U7X5SOXOJDAEGY5NF3M6QQFPZSQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.017455915 (301h14m27.479548399s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.018864467 (301h14m27.48095694s) [noise=true , address=77.90.31.10:4518 , piece_id=6FGNAMVK6OAGJTZJNKTUJWGRW7MLGMJMGJKMAINWMPW7BV74OO6A, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.020222022 (301h14m27.482314495s) [noise=true , address=45.91.116.11:4118 , piece_id=VUZB3QRFST2T3HJHMEYYZ6CV7CKJTFWQGNOBDTXHGET2MBUIC4HQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:43.755608897 (301h14m29.21770138s) [upload=265104]: Next returned nil segment Oct 6 02:45:43.755631300 (301h14m29.217723773s) [upload=265104]: Waiting for error group managing segments... Oct 6 02:45:44.043930124 (301h14m29.506022597s) [noise=true , address=145.79.227.19:4077 , piece_id=EJIAPZSBMAO77E33V6MEBOQHUXVVYKMAVP53QAZXZSYBYDFP7MMQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.045501296 (301h14m29.507593769s) [noise=true , address=145.79.227.4:4185 , piece_id=3AD745KFD67M4XFNDXMJVEU34RRBQXXE4P3HVJJRDBHK3IIXNYTA, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.057881199 (301h14m29.519973672s) [noise=true , address=145.79.227.15:4875 , piece_id=PBDTNZ2NWRZZC3VCIQCVR2WLT2AP6662I2SSDNP6BLUDKCJMRK6Q, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.068726540 (301h14m29.530819023s) [noise=true , address=145.79.227.7:4294 , piece_id=7E3B4BGLCTW6C5DM6PPDJPWGVJS5HEGYPAPXRAMX7XDSG7USPQFQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.897895712 (301h14m30.359988185s) [noise=true , address=45.91.116.12:4226 , piece_id=NM7SEIKZYUHZLNVSOGYVZWMI7U7X5SOXOJDAEGY5NF3M6QQFPZSQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.252859665 (301h14m30.714952148s) [noise=true , address=77.90.31.14:4662 , piece_id=MPVBPBB63JILLOUF3N7SRLCZ2F74LX7Z4AH7ZFG3BDNBHT4UIXTQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.318952216 (301h14m30.781044699s) [noise=true , address=45.91.116.6:4974 , piece_id=7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.836439779 (301h14m31.298532262s) [noise=true , address=45.91.116.10:4081 , piece_id=FTXZ2LOHQMY74KOX6XZC3TLHWXOGDTIY3SZLSPTE7THGKNLMPXRQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.975895173 (301h14m31.437987656s) [noise=true , address=45.91.116.6:4975 , piece_id=T3VWH5OBQ7FE66GCKHZLEDSIXFVYEU3V52HI4CPD27QL3RH75K7Q, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:46.826703981 (301h14m32.288796454s) [noise=true , address=77.90.31.10:4518 , piece_id=6FGNAMVK6OAGJTZJNKTUJWGRW7MLGMJMGJKMAINWMPW7BV74OO6A, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:48.069759484 (301h14m33.531851968s) [noise=true , address=45.91.116.13:4154 , piece_id=2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:54.791582362 (301h14m40.253674835s) [noise=true , address=45.91.116.11:4118 , piece_id=VUZB3QRFST2T3HJHMEYYZ6CV7CKJTFWQGNOBDTXHGET2MBUIC4HQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:56.329163998 (301h14m41.791256481s) [noise=true , address=45.91.116.11:4117 , piece_id=QKEYHSK5JOL65H5T2KUEAEK6VKTNCZAYXIEZTMB76PV7P5PEVILQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:56.329201379 (301h14m41.791293862s) [seg_pos={0 0}, upload=265104]: Segment reached optimal threshold of 13 pieces. Oct 6 02:45:56.329362146 (301h14m41.791454629s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Oct 6 02:45:56.329371974 (301h14m41.791464457s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329546396 (301h14m41.791638869s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: write tcp 188.166.225.170:57604->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 02:45:56.329556665 (301h14m41.791649148s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329567937 (301h14m41.791660421s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: rpc: dial tcp 77.90.31.19:4841: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 02:45:56.329579870 (301h14m41.791672353s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329599808 (301h14m41.791692291s) [seg_pos={0 0}, upload=265104]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 6 02:45:56.329683116 (301h14m41.7917756s) [upload=265104]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 6 02:45:56.329690771 (301h14m41.791783254s) [upload=265104]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 6 02:45:56.636341641 (301h14m42.098434134s) [upload=265104]: Done uploading segments... Oct 6 02:45:56.636368131 (301h14m42.098460614s) [upload=265104]: Upload finished. err: <nil> Duration: 5.94217019s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................X.....................X...] duration: 6.152001283s FAIL (2025-10-06 02:48:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: 2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: 7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA) context deadline exceeded ======LOGS====== Oct 6 02:45:41.756337907 (301h14m27.21843039s) [upload=265104]: Starting upload Oct 6 02:45:41.756430433 (301h14m27.218522916s) [upload=265104]: Done starting upload Oct 6 02:45:41.756513701 (301h14m27.218606185s) [upload=265104]: Uploading segments... Oct 6 02:45:41.756752276 (301h14m27.218844759s) [upload=265104]: Got next segment. Inline: false Oct 6 02:45:41.756794536 (301h14m27.218887019s) [upload=265104]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 02:45:41.756800898 (301h14m27.218893381s) [upload=265104]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 02:45:42.009327267 (301h14m27.47141975s) [seg_pos={0 0}, upload=265104]: Begin upload segment... Oct 6 02:45:42.009417419 (301h14m27.471509892s) [seg_pos={0 0}, upload=265104]: Done begin upload segment. Oct 6 02:45:42.009444500 (301h14m27.471536983s) [noise=true , address=77.90.31.14:4662 , piece_id=MPVBPBB63JILLOUF3N7SRLCZ2F74LX7Z4AH7ZFG3BDNBHT4UIXTQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.009561473 (301h14m27.471653956s) [noise=true , address=145.79.227.19:4077 , piece_id=EJIAPZSBMAO77E33V6MEBOQHUXVVYKMAVP53QAZXZSYBYDFP7MMQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.010197574 (301h14m27.472290057s) [noise=true , address=145.79.227.15:4875 , piece_id=PBDTNZ2NWRZZC3VCIQCVR2WLT2AP6662I2SSDNP6BLUDKCJMRK6Q, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.010634216 (301h14m27.472726699s) [noise=true , address=145.79.227.4:4185 , piece_id=3AD745KFD67M4XFNDXMJVEU34RRBQXXE4P3HVJJRDBHK3IIXNYTA, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011073742 (301h14m27.473166225s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011505996 (301h14m27.473598479s) [noise=true , address=45.91.116.10:4081 , piece_id=FTXZ2LOHQMY74KOX6XZC3TLHWXOGDTIY3SZLSPTE7THGKNLMPXRQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011982502 (301h14m27.474074985s) [noise=true , address=45.91.116.6:4974 , piece_id=7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.012488236 (301h14m27.474580719s) [noise=true , address=145.79.227.7:4294 , piece_id=7E3B4BGLCTW6C5DM6PPDJPWGVJS5HEGYPAPXRAMX7XDSG7USPQFQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.013242472 (301h14m27.475334955s) [noise=true , address=45.91.116.13:4154 , piece_id=2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.014529382 (301h14m27.476621855s) [noise=true , address=45.91.116.6:4975 , piece_id=T3VWH5OBQ7FE66GCKHZLEDSIXFVYEU3V52HI4CPD27QL3RH75K7Q, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.014632619 (301h14m27.476725102s) [noise=true , address=45.91.116.11:4117 , piece_id=QKEYHSK5JOL65H5T2KUEAEK6VKTNCZAYXIEZTMB76PV7P5PEVILQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.015990214 (301h14m27.478082697s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.016291928 (301h14m27.478384412s) [noise=true , address=45.91.116.12:4226 , piece_id=NM7SEIKZYUHZLNVSOGYVZWMI7U7X5SOXOJDAEGY5NF3M6QQFPZSQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.017455915 (301h14m27.479548399s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.018864467 (301h14m27.48095694s) [noise=true , address=77.90.31.10:4518 , piece_id=6FGNAMVK6OAGJTZJNKTUJWGRW7MLGMJMGJKMAINWMPW7BV74OO6A, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.020222022 (301h14m27.482314495s) [noise=true , address=45.91.116.11:4118 , piece_id=VUZB3QRFST2T3HJHMEYYZ6CV7CKJTFWQGNOBDTXHGET2MBUIC4HQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:43.755608897 (301h14m29.21770138s) [upload=265104]: Next returned nil segment Oct 6 02:45:43.755631300 (301h14m29.217723773s) [upload=265104]: Waiting for error group managing segments... Oct 6 02:45:44.043930124 (301h14m29.506022597s) [noise=true , address=145.79.227.19:4077 , piece_id=EJIAPZSBMAO77E33V6MEBOQHUXVVYKMAVP53QAZXZSYBYDFP7MMQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.045501296 (301h14m29.507593769s) [noise=true , address=145.79.227.4:4185 , piece_id=3AD745KFD67M4XFNDXMJVEU34RRBQXXE4P3HVJJRDBHK3IIXNYTA, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.057881199 (301h14m29.519973672s) [noise=true , address=145.79.227.15:4875 , piece_id=PBDTNZ2NWRZZC3VCIQCVR2WLT2AP6662I2SSDNP6BLUDKCJMRK6Q, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.068726540 (301h14m29.530819023s) [noise=true , address=145.79.227.7:4294 , piece_id=7E3B4BGLCTW6C5DM6PPDJPWGVJS5HEGYPAPXRAMX7XDSG7USPQFQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.897895712 (301h14m30.359988185s) [noise=true , address=45.91.116.12:4226 , piece_id=NM7SEIKZYUHZLNVSOGYVZWMI7U7X5SOXOJDAEGY5NF3M6QQFPZSQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.252859665 (301h14m30.714952148s) [noise=true , address=77.90.31.14:4662 , piece_id=MPVBPBB63JILLOUF3N7SRLCZ2F74LX7Z4AH7ZFG3BDNBHT4UIXTQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.318952216 (301h14m30.781044699s) [noise=true , address=45.91.116.6:4974 , piece_id=7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.836439779 (301h14m31.298532262s) [noise=true , address=45.91.116.10:4081 , piece_id=FTXZ2LOHQMY74KOX6XZC3TLHWXOGDTIY3SZLSPTE7THGKNLMPXRQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.975895173 (301h14m31.437987656s) [noise=true , address=45.91.116.6:4975 , piece_id=T3VWH5OBQ7FE66GCKHZLEDSIXFVYEU3V52HI4CPD27QL3RH75K7Q, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:46.826703981 (301h14m32.288796454s) [noise=true , address=77.90.31.10:4518 , piece_id=6FGNAMVK6OAGJTZJNKTUJWGRW7MLGMJMGJKMAINWMPW7BV74OO6A, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:48.069759484 (301h14m33.531851968s) [noise=true , address=45.91.116.13:4154 , piece_id=2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:54.791582362 (301h14m40.253674835s) [noise=true , address=45.91.116.11:4118 , piece_id=VUZB3QRFST2T3HJHMEYYZ6CV7CKJTFWQGNOBDTXHGET2MBUIC4HQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:56.329163998 (301h14m41.791256481s) [noise=true , address=45.91.116.11:4117 , piece_id=QKEYHSK5JOL65H5T2KUEAEK6VKTNCZAYXIEZTMB76PV7P5PEVILQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:56.329201379 (301h14m41.791293862s) [seg_pos={0 0}, upload=265104]: Segment reached optimal threshold of 13 pieces. Oct 6 02:45:56.329362146 (301h14m41.791454629s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Oct 6 02:45:56.329371974 (301h14m41.791464457s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329546396 (301h14m41.791638869s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: write tcp 188.166.225.170:57604->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 02:45:56.329556665 (301h14m41.791649148s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329567937 (301h14m41.791660421s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: rpc: dial tcp 77.90.31.19:4841: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 02:45:56.329579870 (301h14m41.791672353s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329599808 (301h14m41.791692291s) [seg_pos={0 0}, upload=265104]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 6 02:45:56.329683116 (301h14m41.7917756s) [upload=265104]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 6 02:45:56.329690771 (301h14m41.791783254s) [upload=265104]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 6 02:45:56.636341641 (301h14m42.098434134s) [upload=265104]: Done uploading segments... Oct 6 02:45:56.636368131 (301h14m42.098460614s) [upload=265104]: Upload finished. err: <nil> Duration: 6.15203098s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................X.....................X..] duration: 6.293772464s FAIL (2025-10-06 02:47:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: 2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: 7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA) context deadline exceeded ======LOGS====== Oct 6 02:45:41.756337907 (301h14m27.21843039s) [upload=265104]: Starting upload Oct 6 02:45:41.756430433 (301h14m27.218522916s) [upload=265104]: Done starting upload Oct 6 02:45:41.756513701 (301h14m27.218606185s) [upload=265104]: Uploading segments... Oct 6 02:45:41.756752276 (301h14m27.218844759s) [upload=265104]: Got next segment. Inline: false Oct 6 02:45:41.756794536 (301h14m27.218887019s) [upload=265104]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 02:45:41.756800898 (301h14m27.218893381s) [upload=265104]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 02:45:42.009327267 (301h14m27.47141975s) [seg_pos={0 0}, upload=265104]: Begin upload segment... Oct 6 02:45:42.009417419 (301h14m27.471509892s) [seg_pos={0 0}, upload=265104]: Done begin upload segment. Oct 6 02:45:42.009444500 (301h14m27.471536983s) [noise=true , address=77.90.31.14:4662 , piece_id=MPVBPBB63JILLOUF3N7SRLCZ2F74LX7Z4AH7ZFG3BDNBHT4UIXTQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.009561473 (301h14m27.471653956s) [noise=true , address=145.79.227.19:4077 , piece_id=EJIAPZSBMAO77E33V6MEBOQHUXVVYKMAVP53QAZXZSYBYDFP7MMQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.010197574 (301h14m27.472290057s) [noise=true , address=145.79.227.15:4875 , piece_id=PBDTNZ2NWRZZC3VCIQCVR2WLT2AP6662I2SSDNP6BLUDKCJMRK6Q, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.010634216 (301h14m27.472726699s) [noise=true , address=145.79.227.4:4185 , piece_id=3AD745KFD67M4XFNDXMJVEU34RRBQXXE4P3HVJJRDBHK3IIXNYTA, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011073742 (301h14m27.473166225s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011505996 (301h14m27.473598479s) [noise=true , address=45.91.116.10:4081 , piece_id=FTXZ2LOHQMY74KOX6XZC3TLHWXOGDTIY3SZLSPTE7THGKNLMPXRQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011982502 (301h14m27.474074985s) [noise=true , address=45.91.116.6:4974 , piece_id=7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.012488236 (301h14m27.474580719s) [noise=true , address=145.79.227.7:4294 , piece_id=7E3B4BGLCTW6C5DM6PPDJPWGVJS5HEGYPAPXRAMX7XDSG7USPQFQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.013242472 (301h14m27.475334955s) [noise=true , address=45.91.116.13:4154 , piece_id=2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.014529382 (301h14m27.476621855s) [noise=true , address=45.91.116.6:4975 , piece_id=T3VWH5OBQ7FE66GCKHZLEDSIXFVYEU3V52HI4CPD27QL3RH75K7Q, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.014632619 (301h14m27.476725102s) [noise=true , address=45.91.116.11:4117 , piece_id=QKEYHSK5JOL65H5T2KUEAEK6VKTNCZAYXIEZTMB76PV7P5PEVILQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.015990214 (301h14m27.478082697s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.016291928 (301h14m27.478384412s) [noise=true , address=45.91.116.12:4226 , piece_id=NM7SEIKZYUHZLNVSOGYVZWMI7U7X5SOXOJDAEGY5NF3M6QQFPZSQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.017455915 (301h14m27.479548399s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.018864467 (301h14m27.48095694s) [noise=true , address=77.90.31.10:4518 , piece_id=6FGNAMVK6OAGJTZJNKTUJWGRW7MLGMJMGJKMAINWMPW7BV74OO6A, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.020222022 (301h14m27.482314495s) [noise=true , address=45.91.116.11:4118 , piece_id=VUZB3QRFST2T3HJHMEYYZ6CV7CKJTFWQGNOBDTXHGET2MBUIC4HQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:43.755608897 (301h14m29.21770138s) [upload=265104]: Next returned nil segment Oct 6 02:45:43.755631300 (301h14m29.217723773s) [upload=265104]: Waiting for error group managing segments... Oct 6 02:45:44.043930124 (301h14m29.506022597s) [noise=true , address=145.79.227.19:4077 , piece_id=EJIAPZSBMAO77E33V6MEBOQHUXVVYKMAVP53QAZXZSYBYDFP7MMQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.045501296 (301h14m29.507593769s) [noise=true , address=145.79.227.4:4185 , piece_id=3AD745KFD67M4XFNDXMJVEU34RRBQXXE4P3HVJJRDBHK3IIXNYTA, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.057881199 (301h14m29.519973672s) [noise=true , address=145.79.227.15:4875 , piece_id=PBDTNZ2NWRZZC3VCIQCVR2WLT2AP6662I2SSDNP6BLUDKCJMRK6Q, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.068726540 (301h14m29.530819023s) [noise=true , address=145.79.227.7:4294 , piece_id=7E3B4BGLCTW6C5DM6PPDJPWGVJS5HEGYPAPXRAMX7XDSG7USPQFQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.897895712 (301h14m30.359988185s) [noise=true , address=45.91.116.12:4226 , piece_id=NM7SEIKZYUHZLNVSOGYVZWMI7U7X5SOXOJDAEGY5NF3M6QQFPZSQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.252859665 (301h14m30.714952148s) [noise=true , address=77.90.31.14:4662 , piece_id=MPVBPBB63JILLOUF3N7SRLCZ2F74LX7Z4AH7ZFG3BDNBHT4UIXTQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.318952216 (301h14m30.781044699s) [noise=true , address=45.91.116.6:4974 , piece_id=7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.836439779 (301h14m31.298532262s) [noise=true , address=45.91.116.10:4081 , piece_id=FTXZ2LOHQMY74KOX6XZC3TLHWXOGDTIY3SZLSPTE7THGKNLMPXRQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.975895173 (301h14m31.437987656s) [noise=true , address=45.91.116.6:4975 , piece_id=T3VWH5OBQ7FE66GCKHZLEDSIXFVYEU3V52HI4CPD27QL3RH75K7Q, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:46.826703981 (301h14m32.288796454s) [noise=true , address=77.90.31.10:4518 , piece_id=6FGNAMVK6OAGJTZJNKTUJWGRW7MLGMJMGJKMAINWMPW7BV74OO6A, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:48.069759484 (301h14m33.531851968s) [noise=true , address=45.91.116.13:4154 , piece_id=2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:54.791582362 (301h14m40.253674835s) [noise=true , address=45.91.116.11:4118 , piece_id=VUZB3QRFST2T3HJHMEYYZ6CV7CKJTFWQGNOBDTXHGET2MBUIC4HQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:56.329163998 (301h14m41.791256481s) [noise=true , address=45.91.116.11:4117 , piece_id=QKEYHSK5JOL65H5T2KUEAEK6VKTNCZAYXIEZTMB76PV7P5PEVILQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:56.329201379 (301h14m41.791293862s) [seg_pos={0 0}, upload=265104]: Segment reached optimal threshold of 13 pieces. Oct 6 02:45:56.329362146 (301h14m41.791454629s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Oct 6 02:45:56.329371974 (301h14m41.791464457s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329546396 (301h14m41.791638869s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: write tcp 188.166.225.170:57604->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 02:45:56.329556665 (301h14m41.791649148s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329567937 (301h14m41.791660421s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: rpc: dial tcp 77.90.31.19:4841: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 02:45:56.329579870 (301h14m41.791672353s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329599808 (301h14m41.791692291s) [seg_pos={0 0}, upload=265104]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 6 02:45:56.329683116 (301h14m41.7917756s) [upload=265104]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 6 02:45:56.329690771 (301h14m41.791783254s) [upload=265104]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 6 02:45:56.636341641 (301h14m42.098434134s) [upload=265104]: Done uploading segments... Oct 6 02:45:56.636368131 (301h14m42.098460614s) [upload=265104]: Upload finished. err: <nil> Duration: 6.293798372s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................X.....................X.] duration: 6.406812582s FAIL (2025-10-06 02:46:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: 2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: 7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA) context deadline exceeded ======LOGS====== Oct 6 02:45:41.756337907 (301h14m27.21843039s) [upload=265104]: Starting upload Oct 6 02:45:41.756430433 (301h14m27.218522916s) [upload=265104]: Done starting upload Oct 6 02:45:41.756513701 (301h14m27.218606185s) [upload=265104]: Uploading segments... Oct 6 02:45:41.756752276 (301h14m27.218844759s) [upload=265104]: Got next segment. Inline: false Oct 6 02:45:41.756794536 (301h14m27.218887019s) [upload=265104]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 02:45:41.756800898 (301h14m27.218893381s) [upload=265104]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 02:45:42.009327267 (301h14m27.47141975s) [seg_pos={0 0}, upload=265104]: Begin upload segment... Oct 6 02:45:42.009417419 (301h14m27.471509892s) [seg_pos={0 0}, upload=265104]: Done begin upload segment. Oct 6 02:45:42.009444500 (301h14m27.471536983s) [noise=true , address=77.90.31.14:4662 , piece_id=MPVBPBB63JILLOUF3N7SRLCZ2F74LX7Z4AH7ZFG3BDNBHT4UIXTQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.009561473 (301h14m27.471653956s) [noise=true , address=145.79.227.19:4077 , piece_id=EJIAPZSBMAO77E33V6MEBOQHUXVVYKMAVP53QAZXZSYBYDFP7MMQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.010197574 (301h14m27.472290057s) [noise=true , address=145.79.227.15:4875 , piece_id=PBDTNZ2NWRZZC3VCIQCVR2WLT2AP6662I2SSDNP6BLUDKCJMRK6Q, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.010634216 (301h14m27.472726699s) [noise=true , address=145.79.227.4:4185 , piece_id=3AD745KFD67M4XFNDXMJVEU34RRBQXXE4P3HVJJRDBHK3IIXNYTA, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011073742 (301h14m27.473166225s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011505996 (301h14m27.473598479s) [noise=true , address=45.91.116.10:4081 , piece_id=FTXZ2LOHQMY74KOX6XZC3TLHWXOGDTIY3SZLSPTE7THGKNLMPXRQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011982502 (301h14m27.474074985s) [noise=true , address=45.91.116.6:4974 , piece_id=7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.012488236 (301h14m27.474580719s) [noise=true , address=145.79.227.7:4294 , piece_id=7E3B4BGLCTW6C5DM6PPDJPWGVJS5HEGYPAPXRAMX7XDSG7USPQFQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.013242472 (301h14m27.475334955s) [noise=true , address=45.91.116.13:4154 , piece_id=2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.014529382 (301h14m27.476621855s) [noise=true , address=45.91.116.6:4975 , piece_id=T3VWH5OBQ7FE66GCKHZLEDSIXFVYEU3V52HI4CPD27QL3RH75K7Q, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.014632619 (301h14m27.476725102s) [noise=true , address=45.91.116.11:4117 , piece_id=QKEYHSK5JOL65H5T2KUEAEK6VKTNCZAYXIEZTMB76PV7P5PEVILQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.015990214 (301h14m27.478082697s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.016291928 (301h14m27.478384412s) [noise=true , address=45.91.116.12:4226 , piece_id=NM7SEIKZYUHZLNVSOGYVZWMI7U7X5SOXOJDAEGY5NF3M6QQFPZSQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.017455915 (301h14m27.479548399s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.018864467 (301h14m27.48095694s) [noise=true , address=77.90.31.10:4518 , piece_id=6FGNAMVK6OAGJTZJNKTUJWGRW7MLGMJMGJKMAINWMPW7BV74OO6A, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.020222022 (301h14m27.482314495s) [noise=true , address=45.91.116.11:4118 , piece_id=VUZB3QRFST2T3HJHMEYYZ6CV7CKJTFWQGNOBDTXHGET2MBUIC4HQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:43.755608897 (301h14m29.21770138s) [upload=265104]: Next returned nil segment Oct 6 02:45:43.755631300 (301h14m29.217723773s) [upload=265104]: Waiting for error group managing segments... Oct 6 02:45:44.043930124 (301h14m29.506022597s) [noise=true , address=145.79.227.19:4077 , piece_id=EJIAPZSBMAO77E33V6MEBOQHUXVVYKMAVP53QAZXZSYBYDFP7MMQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.045501296 (301h14m29.507593769s) [noise=true , address=145.79.227.4:4185 , piece_id=3AD745KFD67M4XFNDXMJVEU34RRBQXXE4P3HVJJRDBHK3IIXNYTA, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.057881199 (301h14m29.519973672s) [noise=true , address=145.79.227.15:4875 , piece_id=PBDTNZ2NWRZZC3VCIQCVR2WLT2AP6662I2SSDNP6BLUDKCJMRK6Q, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.068726540 (301h14m29.530819023s) [noise=true , address=145.79.227.7:4294 , piece_id=7E3B4BGLCTW6C5DM6PPDJPWGVJS5HEGYPAPXRAMX7XDSG7USPQFQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.897895712 (301h14m30.359988185s) [noise=true , address=45.91.116.12:4226 , piece_id=NM7SEIKZYUHZLNVSOGYVZWMI7U7X5SOXOJDAEGY5NF3M6QQFPZSQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.252859665 (301h14m30.714952148s) [noise=true , address=77.90.31.14:4662 , piece_id=MPVBPBB63JILLOUF3N7SRLCZ2F74LX7Z4AH7ZFG3BDNBHT4UIXTQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.318952216 (301h14m30.781044699s) [noise=true , address=45.91.116.6:4974 , piece_id=7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.836439779 (301h14m31.298532262s) [noise=true , address=45.91.116.10:4081 , piece_id=FTXZ2LOHQMY74KOX6XZC3TLHWXOGDTIY3SZLSPTE7THGKNLMPXRQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.975895173 (301h14m31.437987656s) [noise=true , address=45.91.116.6:4975 , piece_id=T3VWH5OBQ7FE66GCKHZLEDSIXFVYEU3V52HI4CPD27QL3RH75K7Q, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:46.826703981 (301h14m32.288796454s) [noise=true , address=77.90.31.10:4518 , piece_id=6FGNAMVK6OAGJTZJNKTUJWGRW7MLGMJMGJKMAINWMPW7BV74OO6A, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:48.069759484 (301h14m33.531851968s) [noise=true , address=45.91.116.13:4154 , piece_id=2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:54.791582362 (301h14m40.253674835s) [noise=true , address=45.91.116.11:4118 , piece_id=VUZB3QRFST2T3HJHMEYYZ6CV7CKJTFWQGNOBDTXHGET2MBUIC4HQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:56.329163998 (301h14m41.791256481s) [noise=true , address=45.91.116.11:4117 , piece_id=QKEYHSK5JOL65H5T2KUEAEK6VKTNCZAYXIEZTMB76PV7P5PEVILQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:56.329201379 (301h14m41.791293862s) [seg_pos={0 0}, upload=265104]: Segment reached optimal threshold of 13 pieces. Oct 6 02:45:56.329362146 (301h14m41.791454629s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Oct 6 02:45:56.329371974 (301h14m41.791464457s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329546396 (301h14m41.791638869s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: write tcp 188.166.225.170:57604->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 02:45:56.329556665 (301h14m41.791649148s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329567937 (301h14m41.791660421s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: rpc: dial tcp 77.90.31.19:4841: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 02:45:56.329579870 (301h14m41.791672353s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329599808 (301h14m41.791692291s) [seg_pos={0 0}, upload=265104]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 6 02:45:56.329683116 (301h14m41.7917756s) [upload=265104]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 6 02:45:56.329690771 (301h14m41.791783254s) [upload=265104]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 6 02:45:56.636341641 (301h14m42.098434134s) [upload=265104]: Done uploading segments... Oct 6 02:45:56.636368131 (301h14m42.098460614s) [upload=265104]: Upload finished. err: <nil> Duration: 6.406844172s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................X.....................X] duration: 30.006161626s FAIL (2025-10-06 02:46:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: 2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: 7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA) context deadline exceeded ======LOGS====== Oct 6 02:45:41.756337907 (301h14m27.21843039s) [upload=265104]: Starting upload Oct 6 02:45:41.756430433 (301h14m27.218522916s) [upload=265104]: Done starting upload Oct 6 02:45:41.756513701 (301h14m27.218606185s) [upload=265104]: Uploading segments... Oct 6 02:45:41.756752276 (301h14m27.218844759s) [upload=265104]: Got next segment. Inline: false Oct 6 02:45:41.756794536 (301h14m27.218887019s) [upload=265104]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 02:45:41.756800898 (301h14m27.218893381s) [upload=265104]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 02:45:42.009327267 (301h14m27.47141975s) [seg_pos={0 0}, upload=265104]: Begin upload segment... Oct 6 02:45:42.009417419 (301h14m27.471509892s) [seg_pos={0 0}, upload=265104]: Done begin upload segment. Oct 6 02:45:42.009444500 (301h14m27.471536983s) [noise=true , address=77.90.31.14:4662 , piece_id=MPVBPBB63JILLOUF3N7SRLCZ2F74LX7Z4AH7ZFG3BDNBHT4UIXTQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.009561473 (301h14m27.471653956s) [noise=true , address=145.79.227.19:4077 , piece_id=EJIAPZSBMAO77E33V6MEBOQHUXVVYKMAVP53QAZXZSYBYDFP7MMQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.010197574 (301h14m27.472290057s) [noise=true , address=145.79.227.15:4875 , piece_id=PBDTNZ2NWRZZC3VCIQCVR2WLT2AP6662I2SSDNP6BLUDKCJMRK6Q, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.010634216 (301h14m27.472726699s) [noise=true , address=145.79.227.4:4185 , piece_id=3AD745KFD67M4XFNDXMJVEU34RRBQXXE4P3HVJJRDBHK3IIXNYTA, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011073742 (301h14m27.473166225s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011505996 (301h14m27.473598479s) [noise=true , address=45.91.116.10:4081 , piece_id=FTXZ2LOHQMY74KOX6XZC3TLHWXOGDTIY3SZLSPTE7THGKNLMPXRQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011982502 (301h14m27.474074985s) [noise=true , address=45.91.116.6:4974 , piece_id=7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.012488236 (301h14m27.474580719s) [noise=true , address=145.79.227.7:4294 , piece_id=7E3B4BGLCTW6C5DM6PPDJPWGVJS5HEGYPAPXRAMX7XDSG7USPQFQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.013242472 (301h14m27.475334955s) [noise=true , address=45.91.116.13:4154 , piece_id=2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.014529382 (301h14m27.476621855s) [noise=true , address=45.91.116.6:4975 , piece_id=T3VWH5OBQ7FE66GCKHZLEDSIXFVYEU3V52HI4CPD27QL3RH75K7Q, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.014632619 (301h14m27.476725102s) [noise=true , address=45.91.116.11:4117 , piece_id=QKEYHSK5JOL65H5T2KUEAEK6VKTNCZAYXIEZTMB76PV7P5PEVILQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.015990214 (301h14m27.478082697s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.016291928 (301h14m27.478384412s) [noise=true , address=45.91.116.12:4226 , piece_id=NM7SEIKZYUHZLNVSOGYVZWMI7U7X5SOXOJDAEGY5NF3M6QQFPZSQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.017455915 (301h14m27.479548399s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.018864467 (301h14m27.48095694s) [noise=true , address=77.90.31.10:4518 , piece_id=6FGNAMVK6OAGJTZJNKTUJWGRW7MLGMJMGJKMAINWMPW7BV74OO6A, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.020222022 (301h14m27.482314495s) [noise=true , address=45.91.116.11:4118 , piece_id=VUZB3QRFST2T3HJHMEYYZ6CV7CKJTFWQGNOBDTXHGET2MBUIC4HQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:43.755608897 (301h14m29.21770138s) [upload=265104]: Next returned nil segment Oct 6 02:45:43.755631300 (301h14m29.217723773s) [upload=265104]: Waiting for error group managing segments... Oct 6 02:45:44.043930124 (301h14m29.506022597s) [noise=true , address=145.79.227.19:4077 , piece_id=EJIAPZSBMAO77E33V6MEBOQHUXVVYKMAVP53QAZXZSYBYDFP7MMQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.045501296 (301h14m29.507593769s) [noise=true , address=145.79.227.4:4185 , piece_id=3AD745KFD67M4XFNDXMJVEU34RRBQXXE4P3HVJJRDBHK3IIXNYTA, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.057881199 (301h14m29.519973672s) [noise=true , address=145.79.227.15:4875 , piece_id=PBDTNZ2NWRZZC3VCIQCVR2WLT2AP6662I2SSDNP6BLUDKCJMRK6Q, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.068726540 (301h14m29.530819023s) [noise=true , address=145.79.227.7:4294 , piece_id=7E3B4BGLCTW6C5DM6PPDJPWGVJS5HEGYPAPXRAMX7XDSG7USPQFQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.897895712 (301h14m30.359988185s) [noise=true , address=45.91.116.12:4226 , piece_id=NM7SEIKZYUHZLNVSOGYVZWMI7U7X5SOXOJDAEGY5NF3M6QQFPZSQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.252859665 (301h14m30.714952148s) [noise=true , address=77.90.31.14:4662 , piece_id=MPVBPBB63JILLOUF3N7SRLCZ2F74LX7Z4AH7ZFG3BDNBHT4UIXTQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.318952216 (301h14m30.781044699s) [noise=true , address=45.91.116.6:4974 , piece_id=7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.836439779 (301h14m31.298532262s) [noise=true , address=45.91.116.10:4081 , piece_id=FTXZ2LOHQMY74KOX6XZC3TLHWXOGDTIY3SZLSPTE7THGKNLMPXRQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.975895173 (301h14m31.437987656s) [noise=true , address=45.91.116.6:4975 , piece_id=T3VWH5OBQ7FE66GCKHZLEDSIXFVYEU3V52HI4CPD27QL3RH75K7Q, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:46.826703981 (301h14m32.288796454s) [noise=true , address=77.90.31.10:4518 , piece_id=6FGNAMVK6OAGJTZJNKTUJWGRW7MLGMJMGJKMAINWMPW7BV74OO6A, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:48.069759484 (301h14m33.531851968s) [noise=true , address=45.91.116.13:4154 , piece_id=2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:54.791582362 (301h14m40.253674835s) [noise=true , address=45.91.116.11:4118 , piece_id=VUZB3QRFST2T3HJHMEYYZ6CV7CKJTFWQGNOBDTXHGET2MBUIC4HQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:56.329163998 (301h14m41.791256481s) [noise=true , address=45.91.116.11:4117 , piece_id=QKEYHSK5JOL65H5T2KUEAEK6VKTNCZAYXIEZTMB76PV7P5PEVILQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:56.329201379 (301h14m41.791293862s) [seg_pos={0 0}, upload=265104]: Segment reached optimal threshold of 13 pieces. Oct 6 02:45:56.329362146 (301h14m41.791454629s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Oct 6 02:45:56.329371974 (301h14m41.791464457s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329546396 (301h14m41.791638869s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: write tcp 188.166.225.170:57604->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 02:45:56.329556665 (301h14m41.791649148s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329567937 (301h14m41.791660421s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: rpc: dial tcp 77.90.31.19:4841: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 02:45:56.329579870 (301h14m41.791672353s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329599808 (301h14m41.791692291s) [seg_pos={0 0}, upload=265104]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 6 02:45:56.329683116 (301h14m41.7917756s) [upload=265104]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 6 02:45:56.329690771 (301h14m41.791783254s) [upload=265104]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 6 02:45:56.636341641 (301h14m42.098434134s) [upload=265104]: Done uploading segments... Oct 6 02:45:56.636368131 (301h14m42.098460614s) [upload=265104]: Upload finished. err: <nil> Duration: 30.006236319s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 5.935492305s FAIL (2025-10-06 02:45:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, Piece ID: T254A3S7TTJEBOIBDDCS34DU7HV25TXDTNZWF2OGE4KRQ7YEA5ZA) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: OS3T2GGABRXYK6HNEXK4C5F5HTJFDEP7SHA5QV2R3SCMXDE2TD5A) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: S5NHKRILOZ47C4NXH4UXBG34ZF6IP2XHBR5JDDBNCBOZK26NALIA) context deadline exceeded; piecestore: (Node ID: 12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, Piece ID: CHJRUVVLWAR7627JOLKL2QHAK7EHW72ZTGE3BTOAAM4ZL5AZL3KQ) context deadline exceeded; piecestore: (Node ID: 12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, Piece ID: TLZVEBMQCCWINXB4AVRKHWKZ7WULCGUC6E5GGDJQLASKWNEHSJPA) context deadline exceeded; piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: LJKUWHGMINU2P7JTS2QN5VP25HPZBKRMFXRDBSO5OTVYTS47SB4A) context deadline exceeded; piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: VTE4WIBLUWV74XMSSPP37CEAWPRQHA3GFV2RJRQYGRQM5JGF3H3Q) context deadline exceeded; piecestore: (Node ID: 1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, Piece ID: 52Q5TEMXLZHLWJ5QWRFHGYWW6RCL2Y57AY6K2J7I3BUA22BNARQQ) context deadline exceeded ======LOGS====== 914003723 (300h58m34.376096196s) [upload=264873]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 02:29:49.215144679 (300h58m34.677237162s) [seg_pos={0 0}, upload=264873]: Begin upload segment... Oct 6 02:29:49.215212839 (300h58m34.677305312s) [seg_pos={0 0}, upload=264873]: Done begin upload segment. Oct 6 02:29:49.215241294 (300h58m34.677333777s) [noise=true , address=145.79.227.7:4294 , piece_id=TLZVEBMQCCWINXB4AVRKHWKZ7WULCGUC6E5GGDJQLASKWNEHSJPA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.215249739 (300h58m34.677342232s) [noise=true , address=45.91.116.4:4937 , piece_id=CHJRUVVLWAR7627JOLKL2QHAK7EHW72ZTGE3BTOAAM4ZL5AZL3KQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.215349820 (300h58m34.677442303s) [noise=true , address=77.90.31.18:4805 , piece_id=LJKUWHGMINU2P7JTS2QN5VP25HPZBKRMFXRDBSO5OTVYTS47SB4A, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.216239644 (300h58m34.678332127s) [noise=true , address=45.91.116.5:4298 , piece_id=VTE4WIBLUWV74XMSSPP37CEAWPRQHA3GFV2RJRQYGRQM5JGF3H3Q, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.218583898 (300h58m34.680676392s) [noise=true , address=45.91.116.10:4082 , piece_id=NX5W46AGWBSZ7EM2LZHZDPE6MEPSLZMNVFZEE2LQ5F5LV7YSQRKQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.219326442 (300h58m34.681418925s) [noise=true , address=45.91.116.6:4974 , piece_id=DBXKLTPEBNF5GVB4PC26BY4EPXOWHU2OWSLWGSHS52S6CDUZIMOA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.222198982 (300h58m34.684291465s) [noise=true , address=145.79.227.17:4972 , piece_id=I54FS5Q4CMMM46S6EP3C5MEUAJCVNWX23OHTBMNQNKP4TRERVZ7A, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.222520114 (300h58m34.684612597s) [noise=true , address=145.79.227.4:4185 , piece_id=ONU6YDQFAMJPRG3ZF45M7KUX5EGXZVQEGCTOJK3VGE2JXXLHCBAQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.225940076 (300h58m34.688032569s) [noise=true , address=77.90.31.13:4628 , piece_id=52Q5TEMXLZHLWJ5QWRFHGYWW6RCL2Y57AY6K2J7I3BUA22BNARQQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.227928832 (300h58m34.690021305s) [noise=true , address=145.79.227.16:4910 , piece_id=T254A3S7TTJEBOIBDDCS34DU7HV25TXDTNZWF2OGE4KRQ7YEA5ZA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.227998645 (300h58m34.690091128s) [noise=true , address=45.91.116.14:4190 , piece_id=OS3T2GGABRXYK6HNEXK4C5F5HTJFDEP7SHA5QV2R3SCMXDE2TD5A, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.231020309 (300h58m34.693112792s) [noise=true , address=145.79.227.18:4007 , piece_id=6PZE3N3QHDGV5K5HULQE5UJXWWG7RAIACP2OVEZVIOTTCWVNQO6Q, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.232908575 (300h58m34.695001068s) [noise=true , address=45.91.116.10:4081 , piece_id=PRCQUWMDPZTR2CSWMOODBECZOLAVJ4OPOUTGQHCPTWXUDQGY2MBQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.233037220 (300h58m34.695129693s) [noise=true , address=145.79.227.19:4079 , piece_id=2VBVLN2SNA6WUICKMNMNM7URPVUGWPB5SVZ335XZ7HIAUM3FY5VA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.234302058 (300h58m34.696394531s) [noise=true , address=45.91.116.7:4010 , piece_id=S5NHKRILOZ47C4NXH4UXBG34ZF6IP2XHBR5JDDBNCBOZK26NALIA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.235631800 (300h58m34.697724283s) [noise=true , address=77.90.31.15:4698 , piece_id=NOCO6KPTUBBKYCITDRFBKA3H5EJTKMDAO4AFQFFXC4PPH3D46X2A, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:50.988532254 (300h58m36.450624747s) [upload=264873]: Next returned nil segment Oct 6 02:29:50.988555589 (300h58m36.450648082s) [upload=264873]: Waiting for error group managing segments... Oct 6 02:29:51.282838267 (300h58m36.74493075s) [noise=true , address=45.91.116.4:4937 , piece_id=CHJRUVVLWAR7627JOLKL2QHAK7EHW72ZTGE3BTOAAM4ZL5AZL3KQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.318573943 (300h58m36.780666416s) [noise=true , address=77.90.31.18:4805 , piece_id=LJKUWHGMINU2P7JTS2QN5VP25HPZBKRMFXRDBSO5OTVYTS47SB4A, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.322112521 (300h58m36.784205004s) [noise=true , address=77.90.31.15:4698 , piece_id=NOCO6KPTUBBKYCITDRFBKA3H5EJTKMDAO4AFQFFXC4PPH3D46X2A, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.350536624 (300h58m36.812629108s) [noise=true , address=45.91.116.6:4974 , piece_id=DBXKLTPEBNF5GVB4PC26BY4EPXOWHU2OWSLWGSHS52S6CDUZIMOA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.362171679 (300h58m36.824264162s) [noise=true , address=45.91.116.10:4081 , piece_id=PRCQUWMDPZTR2CSWMOODBECZOLAVJ4OPOUTGQHCPTWXUDQGY2MBQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.366398738 (300h58m36.828491221s) [noise=true , address=45.91.116.10:4082 , piece_id=NX5W46AGWBSZ7EM2LZHZDPE6MEPSLZMNVFZEE2LQ5F5LV7YSQRKQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.375437720 (300h58m36.837530203s) [noise=true , address=45.91.116.14:4190 , piece_id=OS3T2GGABRXYK6HNEXK4C5F5HTJFDEP7SHA5QV2R3SCMXDE2TD5A, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.397600181 (300h58m36.859692664s) [noise=true , address=77.90.31.13:4628 , piece_id=52Q5TEMXLZHLWJ5QWRFHGYWW6RCL2Y57AY6K2J7I3BUA22BNARQQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.400132462 (300h58m36.862224945s) [noise=true , address=45.91.116.5:4298 , piece_id=VTE4WIBLUWV74XMSSPP37CEAWPRQHA3GFV2RJRQYGRQM5JGF3H3Q, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.412114018 (300h58m36.874206491s) [noise=true , address=45.91.116.7:4010 , piece_id=S5NHKRILOZ47C4NXH4UXBG34ZF6IP2XHBR5JDDBNCBOZK26NALIA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:30:14.302718308 (300h58m59.764810791s) [noise=true , address=145.79.227.18:4007 , piece_id=6PZE3N3QHDGV5K5HULQE5UJXWWG7RAIACP2OVEZVIOTTCWVNQO6Q, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:30:14.319938798 (300h58m59.782031281s) [noise=true , address=145.79.227.7:4294 , piece_id=TLZVEBMQCCWINXB4AVRKHWKZ7WULCGUC6E5GGDJQLASKWNEHSJPA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:30:16.104270571 (300h59m1.566363044s) [noise=true , address=145.79.227.16:4910 , piece_id=T254A3S7TTJEBOIBDDCS34DU7HV25TXDTNZWF2OGE4KRQ7YEA5ZA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:30:16.104295528 (300h59m1.566388012s) [seg_pos={0 0}, upload=264873]: Segment reached optimal threshold of 13 pieces. Oct 6 02:30:16.104442268 (300h59m1.566534751s) [noise=true , address=145.79.227.4:4185 , piece_id=ONU6YDQFAMJPRG3ZF45M7KUX5EGXZVQEGCTOJK3VGE2JXXLHCBAQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN): protocol: write tcp 188.166.225.170:34966->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:34966->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34966->145.79.227.4:4185: use of closed network connection Oct 6 02:30:16.104456395 (300h59m1.566548878s) [noise=true , address=145.79.227.4:4185 , piece_id=ONU6YDQFAMJPRG3ZF45M7KUX5EGXZVQEGCTOJK3VGE2JXXLHCBAQ, seg_pos={0 0}, upload=264873]: Long tail cancellation due to optimal threshold Oct 6 02:30:16.104598005 (300h59m1.566690478s) [noise=true , address=145.79.227.19:4079 , piece_id=2VBVLN2SNA6WUICKMNMNM7URPVUGWPB5SVZ335XZ7HIAUM3FY5VA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: write tcp 188.166.225.170:57162->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:57162->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57162->145.79.227.19:4079: use of closed network connection Oct 6 02:30:16.104604216 (300h59m1.566696699s) [noise=true , address=145.79.227.19:4079 , piece_id=2VBVLN2SNA6WUICKMNMNM7URPVUGWPB5SVZ335XZ7HIAUM3FY5VA, seg_pos={0 0}, upload=264873]: Long tail cancellation due to optimal threshold Oct 6 02:30:16.104672547 (300h59m1.56676503s) [noise=true , address=145.79.227.17:4972 , piece_id=I54FS5Q4CMMM46S6EP3C5MEUAJCVNWX23OHTBMNQNKP4TRERVZ7A, seg_pos={0 0}, upload=264873]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: write tcp 188.166.225.170:33630->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:33630->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33630->145.79.227.17:4972: use of closed network connection Oct 6 02:30:16.104678057 (300h59m1.56677053s) [noise=true , address=145.79.227.17:4972 , piece_id=I54FS5Q4CMMM46S6EP3C5MEUAJCVNWX23OHTBMNQNKP4TRERVZ7A, seg_pos={0 0}, upload=264873]: Long tail cancellation due to optimal threshold Oct 6 02:30:16.104695541 (300h59m1.566788024s) [seg_pos={0 0}, upload=264873]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 6 02:30:16.104762427 (300h59m1.56685491s) [upload=264873]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 6 02:30:16.104768159 (300h59m1.566860632s) [upload=264873]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 6 02:30:16.463894599 (300h59m1.925987092s) [upload=264873]: Done uploading segments... Oct 6 02:30:16.463937391 (300h59m1.926029874s) [upload=264873]: Upload finished. err: <nil> Duration:* 5.935523083s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 6.087614951s FAIL (2025-10-06 02:44:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, Piece ID: T254A3S7TTJEBOIBDDCS34DU7HV25TXDTNZWF2OGE4KRQ7YEA5ZA) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: OS3T2GGABRXYK6HNEXK4C5F5HTJFDEP7SHA5QV2R3SCMXDE2TD5A) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: S5NHKRILOZ47C4NXH4UXBG34ZF6IP2XHBR5JDDBNCBOZK26NALIA) context deadline exceeded; piecestore: (Node ID: 12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, Piece ID: CHJRUVVLWAR7627JOLKL2QHAK7EHW72ZTGE3BTOAAM4ZL5AZL3KQ) context deadline exceeded; piecestore: (Node ID: 12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, Piece ID: TLZVEBMQCCWINXB4AVRKHWKZ7WULCGUC6E5GGDJQLASKWNEHSJPA) context deadline exceeded; piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: LJKUWHGMINU2P7JTS2QN5VP25HPZBKRMFXRDBSO5OTVYTS47SB4A) context deadline exceeded; piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: VTE4WIBLUWV74XMSSPP37CEAWPRQHA3GFV2RJRQYGRQM5JGF3H3Q) context deadline exceeded; piecestore: (Node ID: 1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, Piece ID: 52Q5TEMXLZHLWJ5QWRFHGYWW6RCL2Y57AY6K2J7I3BUA22BNARQQ) context deadline exceeded ======LOGS====== 914003723 (300h58m34.376096196s) [upload=264873]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 02:29:49.215144679 (300h58m34.677237162s) [seg_pos={0 0}, upload=264873]: Begin upload segment... Oct 6 02:29:49.215212839 (300h58m34.677305312s) [seg_pos={0 0}, upload=264873]: Done begin upload segment. Oct 6 02:29:49.215241294 (300h58m34.677333777s) [noise=true , address=145.79.227.7:4294 , piece_id=TLZVEBMQCCWINXB4AVRKHWKZ7WULCGUC6E5GGDJQLASKWNEHSJPA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.215249739 (300h58m34.677342232s) [noise=true , address=45.91.116.4:4937 , piece_id=CHJRUVVLWAR7627JOLKL2QHAK7EHW72ZTGE3BTOAAM4ZL5AZL3KQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.215349820 (300h58m34.677442303s) [noise=true , address=77.90.31.18:4805 , piece_id=LJKUWHGMINU2P7JTS2QN5VP25HPZBKRMFXRDBSO5OTVYTS47SB4A, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.216239644 (300h58m34.678332127s) [noise=true , address=45.91.116.5:4298 , piece_id=VTE4WIBLUWV74XMSSPP37CEAWPRQHA3GFV2RJRQYGRQM5JGF3H3Q, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.218583898 (300h58m34.680676392s) [noise=true , address=45.91.116.10:4082 , piece_id=NX5W46AGWBSZ7EM2LZHZDPE6MEPSLZMNVFZEE2LQ5F5LV7YSQRKQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.219326442 (300h58m34.681418925s) [noise=true , address=45.91.116.6:4974 , piece_id=DBXKLTPEBNF5GVB4PC26BY4EPXOWHU2OWSLWGSHS52S6CDUZIMOA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.222198982 (300h58m34.684291465s) [noise=true , address=145.79.227.17:4972 , piece_id=I54FS5Q4CMMM46S6EP3C5MEUAJCVNWX23OHTBMNQNKP4TRERVZ7A, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.222520114 (300h58m34.684612597s) [noise=true , address=145.79.227.4:4185 , piece_id=ONU6YDQFAMJPRG3ZF45M7KUX5EGXZVQEGCTOJK3VGE2JXXLHCBAQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.225940076 (300h58m34.688032569s) [noise=true , address=77.90.31.13:4628 , piece_id=52Q5TEMXLZHLWJ5QWRFHGYWW6RCL2Y57AY6K2J7I3BUA22BNARQQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.227928832 (300h58m34.690021305s) [noise=true , address=145.79.227.16:4910 , piece_id=T254A3S7TTJEBOIBDDCS34DU7HV25TXDTNZWF2OGE4KRQ7YEA5ZA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.227998645 (300h58m34.690091128s) [noise=true , address=45.91.116.14:4190 , piece_id=OS3T2GGABRXYK6HNEXK4C5F5HTJFDEP7SHA5QV2R3SCMXDE2TD5A, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.231020309 (300h58m34.693112792s) [noise=true , address=145.79.227.18:4007 , piece_id=6PZE3N3QHDGV5K5HULQE5UJXWWG7RAIACP2OVEZVIOTTCWVNQO6Q, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.232908575 (300h58m34.695001068s) [noise=true , address=45.91.116.10:4081 , piece_id=PRCQUWMDPZTR2CSWMOODBECZOLAVJ4OPOUTGQHCPTWXUDQGY2MBQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.233037220 (300h58m34.695129693s) [noise=true , address=145.79.227.19:4079 , piece_id=2VBVLN2SNA6WUICKMNMNM7URPVUGWPB5SVZ335XZ7HIAUM3FY5VA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.234302058 (300h58m34.696394531s) [noise=true , address=45.91.116.7:4010 , piece_id=S5NHKRILOZ47C4NXH4UXBG34ZF6IP2XHBR5JDDBNCBOZK26NALIA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.235631800 (300h58m34.697724283s) [noise=true , address=77.90.31.15:4698 , piece_id=NOCO6KPTUBBKYCITDRFBKA3H5EJTKMDAO4AFQFFXC4PPH3D46X2A, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:50.988532254 (300h58m36.450624747s) [upload=264873]: Next returned nil segment Oct 6 02:29:50.988555589 (300h58m36.450648082s) [upload=264873]: Waiting for error group managing segments... Oct 6 02:29:51.282838267 (300h58m36.74493075s) [noise=true , address=45.91.116.4:4937 , piece_id=CHJRUVVLWAR7627JOLKL2QHAK7EHW72ZTGE3BTOAAM4ZL5AZL3KQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.318573943 (300h58m36.780666416s) [noise=true , address=77.90.31.18:4805 , piece_id=LJKUWHGMINU2P7JTS2QN5VP25HPZBKRMFXRDBSO5OTVYTS47SB4A, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.322112521 (300h58m36.784205004s) [noise=true , address=77.90.31.15:4698 , piece_id=NOCO6KPTUBBKYCITDRFBKA3H5EJTKMDAO4AFQFFXC4PPH3D46X2A, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.350536624 (300h58m36.812629108s) [noise=true , address=45.91.116.6:4974 , piece_id=DBXKLTPEBNF5GVB4PC26BY4EPXOWHU2OWSLWGSHS52S6CDUZIMOA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.362171679 (300h58m36.824264162s) [noise=true , address=45.91.116.10:4081 , piece_id=PRCQUWMDPZTR2CSWMOODBECZOLAVJ4OPOUTGQHCPTWXUDQGY2MBQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.366398738 (300h58m36.828491221s) [noise=true , address=45.91.116.10:4082 , piece_id=NX5W46AGWBSZ7EM2LZHZDPE6MEPSLZMNVFZEE2LQ5F5LV7YSQRKQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.375437720 (300h58m36.837530203s) [noise=true , address=45.91.116.14:4190 , piece_id=OS3T2GGABRXYK6HNEXK4C5F5HTJFDEP7SHA5QV2R3SCMXDE2TD5A, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.397600181 (300h58m36.859692664s) [noise=true , address=77.90.31.13:4628 , piece_id=52Q5TEMXLZHLWJ5QWRFHGYWW6RCL2Y57AY6K2J7I3BUA22BNARQQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.400132462 (300h58m36.862224945s) [noise=true , address=45.91.116.5:4298 , piece_id=VTE4WIBLUWV74XMSSPP37CEAWPRQHA3GFV2RJRQYGRQM5JGF3H3Q, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.412114018 (300h58m36.874206491s) [noise=true , address=45.91.116.7:4010 , piece_id=S5NHKRILOZ47C4NXH4UXBG34ZF6IP2XHBR5JDDBNCBOZK26NALIA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:30:14.302718308 (300h58m59.764810791s) [noise=true , address=145.79.227.18:4007 , piece_id=6PZE3N3QHDGV5K5HULQE5UJXWWG7RAIACP2OVEZVIOTTCWVNQO6Q, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:30:14.319938798 (300h58m59.782031281s) [noise=true , address=145.79.227.7:4294 , piece_id=TLZVEBMQCCWINXB4AVRKHWKZ7WULCGUC6E5GGDJQLASKWNEHSJPA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:30:16.104270571 (300h59m1.566363044s) [noise=true , address=145.79.227.16:4910 , piece_id=T254A3S7TTJEBOIBDDCS34DU7HV25TXDTNZWF2OGE4KRQ7YEA5ZA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:30:16.104295528 (300h59m1.566388012s) [seg_pos={0 0}, upload=264873]: Segment reached optimal threshold of 13 pieces. Oct 6 02:30:16.104442268 (300h59m1.566534751s) [noise=true , address=145.79.227.4:4185 , piece_id=ONU6YDQFAMJPRG3ZF45M7KUX5EGXZVQEGCTOJK3VGE2JXXLHCBAQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN): protocol: write tcp 188.166.225.170:34966->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:34966->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34966->145.79.227.4:4185: use of closed network connection Oct 6 02:30:16.104456395 (300h59m1.566548878s) [noise=true , address=145.79.227.4:4185 , piece_id=ONU6YDQFAMJPRG3ZF45M7KUX5EGXZVQEGCTOJK3VGE2JXXLHCBAQ, seg_pos={0 0}, upload=264873]: Long tail cancellation due to optimal threshold Oct 6 02:30:16.104598005 (300h59m1.566690478s) [noise=true , address=145.79.227.19:4079 , piece_id=2VBVLN2SNA6WUICKMNMNM7URPVUGWPB5SVZ335XZ7HIAUM3FY5VA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: write tcp 188.166.225.170:57162->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:57162->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57162->145.79.227.19:4079: use of closed network connection Oct 6 02:30:16.104604216 (300h59m1.566696699s) [noise=true , address=145.79.227.19:4079 , piece_id=2VBVLN2SNA6WUICKMNMNM7URPVUGWPB5SVZ335XZ7HIAUM3FY5VA, seg_pos={0 0}, upload=264873]: Long tail cancellation due to optimal threshold Oct 6 02:30:16.104672547 (300h59m1.56676503s) [noise=true , address=145.79.227.17:4972 , piece_id=I54FS5Q4CMMM46S6EP3C5MEUAJCVNWX23OHTBMNQNKP4TRERVZ7A, seg_pos={0 0}, upload=264873]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: write tcp 188.166.225.170:33630->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:33630->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33630->145.79.227.17:4972: use of closed network connection Oct 6 02:30:16.104678057 (300h59m1.56677053s) [noise=true , address=145.79.227.17:4972 , piece_id=I54FS5Q4CMMM46S6EP3C5MEUAJCVNWX23OHTBMNQNKP4TRERVZ7A, seg_pos={0 0}, upload=264873]: Long tail cancellation due to optimal threshold Oct 6 02:30:16.104695541 (300h59m1.566788024s) [seg_pos={0 0}, upload=264873]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 6 02:30:16.104762427 (300h59m1.56685491s) [upload=264873]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 6 02:30:16.104768159 (300h59m1.566860632s) [upload=264873]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 6 02:30:16.463894599 (300h59m1.925987092s) [upload=264873]: Done uploading segments... Oct 6 02:30:16.463937391 (300h59m1.926029874s) [upload=264873]: Upload finished. err: <nil> Duration:* 6.0879859s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 6.0560527s FAIL (2025-10-06 02:43:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, Piece ID: T254A3S7TTJEBOIBDDCS34DU7HV25TXDTNZWF2OGE4KRQ7YEA5ZA) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: OS3T2GGABRXYK6HNEXK4C5F5HTJFDEP7SHA5QV2R3SCMXDE2TD5A) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: S5NHKRILOZ47C4NXH4UXBG34ZF6IP2XHBR5JDDBNCBOZK26NALIA) context deadline exceeded; piecestore: (Node ID: 12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, Piece ID: CHJRUVVLWAR7627JOLKL2QHAK7EHW72ZTGE3BTOAAM4ZL5AZL3KQ) context deadline exceeded; piecestore: (Node ID: 12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, Piece ID: TLZVEBMQCCWINXB4AVRKHWKZ7WULCGUC6E5GGDJQLASKWNEHSJPA) context deadline exceeded; piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: LJKUWHGMINU2P7JTS2QN5VP25HPZBKRMFXRDBSO5OTVYTS47SB4A) context deadline exceeded; piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: VTE4WIBLUWV74XMSSPP37CEAWPRQHA3GFV2RJRQYGRQM5JGF3H3Q) context deadline exceeded; piecestore: (Node ID: 1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, Piece ID: 52Q5TEMXLZHLWJ5QWRFHGYWW6RCL2Y57AY6K2J7I3BUA22BNARQQ) context deadline exceeded ======LOGS====== 914003723 (300h58m34.376096196s) [upload=264873]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 02:29:49.215144679 (300h58m34.677237162s) [seg_pos={0 0}, upload=264873]: Begin upload segment... Oct 6 02:29:49.215212839 (300h58m34.677305312s) [seg_pos={0 0}, upload=264873]: Done begin upload segment. Oct 6 02:29:49.215241294 (300h58m34.677333777s) [noise=true , address=145.79.227.7:4294 , piece_id=TLZVEBMQCCWINXB4AVRKHWKZ7WULCGUC6E5GGDJQLASKWNEHSJPA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.215249739 (300h58m34.677342232s) [noise=true , address=45.91.116.4:4937 , piece_id=CHJRUVVLWAR7627JOLKL2QHAK7EHW72ZTGE3BTOAAM4ZL5AZL3KQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.215349820 (300h58m34.677442303s) [noise=true , address=77.90.31.18:4805 , piece_id=LJKUWHGMINU2P7JTS2QN5VP25HPZBKRMFXRDBSO5OTVYTS47SB4A, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.216239644 (300h58m34.678332127s) [noise=true , address=45.91.116.5:4298 , piece_id=VTE4WIBLUWV74XMSSPP37CEAWPRQHA3GFV2RJRQYGRQM5JGF3H3Q, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.218583898 (300h58m34.680676392s) [noise=true , address=45.91.116.10:4082 , piece_id=NX5W46AGWBSZ7EM2LZHZDPE6MEPSLZMNVFZEE2LQ5F5LV7YSQRKQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.219326442 (300h58m34.681418925s) [noise=true , address=45.91.116.6:4974 , piece_id=DBXKLTPEBNF5GVB4PC26BY4EPXOWHU2OWSLWGSHS52S6CDUZIMOA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.222198982 (300h58m34.684291465s) [noise=true , address=145.79.227.17:4972 , piece_id=I54FS5Q4CMMM46S6EP3C5MEUAJCVNWX23OHTBMNQNKP4TRERVZ7A, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.222520114 (300h58m34.684612597s) [noise=true , address=145.79.227.4:4185 , piece_id=ONU6YDQFAMJPRG3ZF45M7KUX5EGXZVQEGCTOJK3VGE2JXXLHCBAQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.225940076 (300h58m34.688032569s) [noise=true , address=77.90.31.13:4628 , piece_id=52Q5TEMXLZHLWJ5QWRFHGYWW6RCL2Y57AY6K2J7I3BUA22BNARQQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.227928832 (300h58m34.690021305s) [noise=true , address=145.79.227.16:4910 , piece_id=T254A3S7TTJEBOIBDDCS34DU7HV25TXDTNZWF2OGE4KRQ7YEA5ZA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.227998645 (300h58m34.690091128s) [noise=true , address=45.91.116.14:4190 , piece_id=OS3T2GGABRXYK6HNEXK4C5F5HTJFDEP7SHA5QV2R3SCMXDE2TD5A, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.231020309 (300h58m34.693112792s) [noise=true , address=145.79.227.18:4007 , piece_id=6PZE3N3QHDGV5K5HULQE5UJXWWG7RAIACP2OVEZVIOTTCWVNQO6Q, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.232908575 (300h58m34.695001068s) [noise=true , address=45.91.116.10:4081 , piece_id=PRCQUWMDPZTR2CSWMOODBECZOLAVJ4OPOUTGQHCPTWXUDQGY2MBQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.233037220 (300h58m34.695129693s) [noise=true , address=145.79.227.19:4079 , piece_id=2VBVLN2SNA6WUICKMNMNM7URPVUGWPB5SVZ335XZ7HIAUM3FY5VA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.234302058 (300h58m34.696394531s) [noise=true , address=45.91.116.7:4010 , piece_id=S5NHKRILOZ47C4NXH4UXBG34ZF6IP2XHBR5JDDBNCBOZK26NALIA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.235631800 (300h58m34.697724283s) [noise=true , address=77.90.31.15:4698 , piece_id=NOCO6KPTUBBKYCITDRFBKA3H5EJTKMDAO4AFQFFXC4PPH3D46X2A, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:50.988532254 (300h58m36.450624747s) [upload=264873]: Next returned nil segment Oct 6 02:29:50.988555589 (300h58m36.450648082s) [upload=264873]: Waiting for error group managing segments... Oct 6 02:29:51.282838267 (300h58m36.74493075s) [noise=true , address=45.91.116.4:4937 , piece_id=CHJRUVVLWAR7627JOLKL2QHAK7EHW72ZTGE3BTOAAM4ZL5AZL3KQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.318573943 (300h58m36.780666416s) [noise=true , address=77.90.31.18:4805 , piece_id=LJKUWHGMINU2P7JTS2QN5VP25HPZBKRMFXRDBSO5OTVYTS47SB4A, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.322112521 (300h58m36.784205004s) [noise=true , address=77.90.31.15:4698 , piece_id=NOCO6KPTUBBKYCITDRFBKA3H5EJTKMDAO4AFQFFXC4PPH3D46X2A, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.350536624 (300h58m36.812629108s) [noise=true , address=45.91.116.6:4974 , piece_id=DBXKLTPEBNF5GVB4PC26BY4EPXOWHU2OWSLWGSHS52S6CDUZIMOA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.362171679 (300h58m36.824264162s) [noise=true , address=45.91.116.10:4081 , piece_id=PRCQUWMDPZTR2CSWMOODBECZOLAVJ4OPOUTGQHCPTWXUDQGY2MBQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.366398738 (300h58m36.828491221s) [noise=true , address=45.91.116.10:4082 , piece_id=NX5W46AGWBSZ7EM2LZHZDPE6MEPSLZMNVFZEE2LQ5F5LV7YSQRKQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.375437720 (300h58m36.837530203s) [noise=true , address=45.91.116.14:4190 , piece_id=OS3T2GGABRXYK6HNEXK4C5F5HTJFDEP7SHA5QV2R3SCMXDE2TD5A, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.397600181 (300h58m36.859692664s) [noise=true , address=77.90.31.13:4628 , piece_id=52Q5TEMXLZHLWJ5QWRFHGYWW6RCL2Y57AY6K2J7I3BUA22BNARQQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.400132462 (300h58m36.862224945s) [noise=true , address=45.91.116.5:4298 , piece_id=VTE4WIBLUWV74XMSSPP37CEAWPRQHA3GFV2RJRQYGRQM5JGF3H3Q, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.412114018 (300h58m36.874206491s) [noise=true , address=45.91.116.7:4010 , piece_id=S5NHKRILOZ47C4NXH4UXBG34ZF6IP2XHBR5JDDBNCBOZK26NALIA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:30:14.302718308 (300h58m59.764810791s) [noise=true , address=145.79.227.18:4007 , piece_id=6PZE3N3QHDGV5K5HULQE5UJXWWG7RAIACP2OVEZVIOTTCWVNQO6Q, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:30:14.319938798 (300h58m59.782031281s) [noise=true , address=145.79.227.7:4294 , piece_id=TLZVEBMQCCWINXB4AVRKHWKZ7WULCGUC6E5GGDJQLASKWNEHSJPA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:30:16.104270571 (300h59m1.566363044s) [noise=true , address=145.79.227.16:4910 , piece_id=T254A3S7TTJEBOIBDDCS34DU7HV25TXDTNZWF2OGE4KRQ7YEA5ZA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:30:16.104295528 (300h59m1.566388012s) [seg_pos={0 0}, upload=264873]: Segment reached optimal threshold of 13 pieces. Oct 6 02:30:16.104442268 (300h59m1.566534751s) [noise=true , address=145.79.227.4:4185 , piece_id=ONU6YDQFAMJPRG3ZF45M7KUX5EGXZVQEGCTOJK3VGE2JXXLHCBAQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN): protocol: write tcp 188.166.225.170:34966->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:34966->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34966->145.79.227.4:4185: use of closed network connection Oct 6 02:30:16.104456395 (300h59m1.566548878s) [noise=true , address=145.79.227.4:4185 , piece_id=ONU6YDQFAMJPRG3ZF45M7KUX5EGXZVQEGCTOJK3VGE2JXXLHCBAQ, seg_pos={0 0}, upload=264873]: Long tail cancellation due to optimal threshold Oct 6 02:30:16.104598005 (300h59m1.566690478s) [noise=true , address=145.79.227.19:4079 , piece_id=2VBVLN2SNA6WUICKMNMNM7URPVUGWPB5SVZ335XZ7HIAUM3FY5VA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: write tcp 188.166.225.170:57162->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:57162->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57162->145.79.227.19:4079: use of closed network connection Oct 6 02:30:16.104604216 (300h59m1.566696699s) [noise=true , address=145.79.227.19:4079 , piece_id=2VBVLN2SNA6WUICKMNMNM7URPVUGWPB5SVZ335XZ7HIAUM3FY5VA, seg_pos={0 0}, upload=264873]: Long tail cancellation due to optimal threshold Oct 6 02:30:16.104672547 (300h59m1.56676503s) [noise=true , address=145.79.227.17:4972 , piece_id=I54FS5Q4CMMM46S6EP3C5MEUAJCVNWX23OHTBMNQNKP4TRERVZ7A, seg_pos={0 0}, upload=264873]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: write tcp 188.166.225.170:33630->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:33630->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33630->145.79.227.17:4972: use of closed network connection Oct 6 02:30:16.104678057 (300h59m1.56677053s) [noise=true , address=145.79.227.17:4972 , piece_id=I54FS5Q4CMMM46S6EP3C5MEUAJCVNWX23OHTBMNQNKP4TRERVZ7A, seg_pos={0 0}, upload=264873]: Long tail cancellation due to optimal threshold Oct 6 02:30:16.104695541 (300h59m1.566788024s) [seg_pos={0 0}, upload=264873]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 6 02:30:16.104762427 (300h59m1.56685491s) [upload=264873]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 6 02:30:16.104768159 (300h59m1.566860632s) [upload=264873]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 6 02:30:16.463894599 (300h59m1.925987092s) [upload=264873]: Done uploading segments... Oct 6 02:30:16.463937391 (300h59m1.926029874s) [upload=264873]: Upload finished. err: <nil> Duration:* 6.056083651s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 6.048916312s FAIL (2025-10-06 02:42:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, Piece ID: T254A3S7TTJEBOIBDDCS34DU7HV25TXDTNZWF2OGE4KRQ7YEA5ZA) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: OS3T2GGABRXYK6HNEXK4C5F5HTJFDEP7SHA5QV2R3SCMXDE2TD5A) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: S5NHKRILOZ47C4NXH4UXBG34ZF6IP2XHBR5JDDBNCBOZK26NALIA) context deadline exceeded; piecestore: (Node ID: 12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, Piece ID: CHJRUVVLWAR7627JOLKL2QHAK7EHW72ZTGE3BTOAAM4ZL5AZL3KQ) context deadline exceeded; piecestore: (Node ID: 12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, Piece ID: TLZVEBMQCCWINXB4AVRKHWKZ7WULCGUC6E5GGDJQLASKWNEHSJPA) context deadline exceeded; piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: LJKUWHGMINU2P7JTS2QN5VP25HPZBKRMFXRDBSO5OTVYTS47SB4A) context deadline exceeded; piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: VTE4WIBLUWV74XMSSPP37CEAWPRQHA3GFV2RJRQYGRQM5JGF3H3Q) context deadline exceeded; piecestore: (Node ID: 1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, Piece ID: 52Q5TEMXLZHLWJ5QWRFHGYWW6RCL2Y57AY6K2J7I3BUA22BNARQQ) context deadline exceeded ======LOGS====== 914003723 (300h58m34.376096196s) [upload=264873]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 02:29:49.215144679 (300h58m34.677237162s) [seg_pos={0 0}, upload=264873]: Begin upload segment... Oct 6 02:29:49.215212839 (300h58m34.677305312s) [seg_pos={0 0}, upload=264873]: Done begin upload segment. Oct 6 02:29:49.215241294 (300h58m34.677333777s) [noise=true , address=145.79.227.7:4294 , piece_id=TLZVEBMQCCWINXB4AVRKHWKZ7WULCGUC6E5GGDJQLASKWNEHSJPA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.215249739 (300h58m34.677342232s) [noise=true , address=45.91.116.4:4937 , piece_id=CHJRUVVLWAR7627JOLKL2QHAK7EHW72ZTGE3BTOAAM4ZL5AZL3KQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.215349820 (300h58m34.677442303s) [noise=true , address=77.90.31.18:4805 , piece_id=LJKUWHGMINU2P7JTS2QN5VP25HPZBKRMFXRDBSO5OTVYTS47SB4A, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.216239644 (300h58m34.678332127s) [noise=true , address=45.91.116.5:4298 , piece_id=VTE4WIBLUWV74XMSSPP37CEAWPRQHA3GFV2RJRQYGRQM5JGF3H3Q, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.218583898 (300h58m34.680676392s) [noise=true , address=45.91.116.10:4082 , piece_id=NX5W46AGWBSZ7EM2LZHZDPE6MEPSLZMNVFZEE2LQ5F5LV7YSQRKQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.219326442 (300h58m34.681418925s) [noise=true , address=45.91.116.6:4974 , piece_id=DBXKLTPEBNF5GVB4PC26BY4EPXOWHU2OWSLWGSHS52S6CDUZIMOA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.222198982 (300h58m34.684291465s) [noise=true , address=145.79.227.17:4972 , piece_id=I54FS5Q4CMMM46S6EP3C5MEUAJCVNWX23OHTBMNQNKP4TRERVZ7A, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.222520114 (300h58m34.684612597s) [noise=true , address=145.79.227.4:4185 , piece_id=ONU6YDQFAMJPRG3ZF45M7KUX5EGXZVQEGCTOJK3VGE2JXXLHCBAQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.225940076 (300h58m34.688032569s) [noise=true , address=77.90.31.13:4628 , piece_id=52Q5TEMXLZHLWJ5QWRFHGYWW6RCL2Y57AY6K2J7I3BUA22BNARQQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.227928832 (300h58m34.690021305s) [noise=true , address=145.79.227.16:4910 , piece_id=T254A3S7TTJEBOIBDDCS34DU7HV25TXDTNZWF2OGE4KRQ7YEA5ZA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.227998645 (300h58m34.690091128s) [noise=true , address=45.91.116.14:4190 , piece_id=OS3T2GGABRXYK6HNEXK4C5F5HTJFDEP7SHA5QV2R3SCMXDE2TD5A, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.231020309 (300h58m34.693112792s) [noise=true , address=145.79.227.18:4007 , piece_id=6PZE3N3QHDGV5K5HULQE5UJXWWG7RAIACP2OVEZVIOTTCWVNQO6Q, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.232908575 (300h58m34.695001068s) [noise=true , address=45.91.116.10:4081 , piece_id=PRCQUWMDPZTR2CSWMOODBECZOLAVJ4OPOUTGQHCPTWXUDQGY2MBQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.233037220 (300h58m34.695129693s) [noise=true , address=145.79.227.19:4079 , piece_id=2VBVLN2SNA6WUICKMNMNM7URPVUGWPB5SVZ335XZ7HIAUM3FY5VA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.234302058 (300h58m34.696394531s) [noise=true , address=45.91.116.7:4010 , piece_id=S5NHKRILOZ47C4NXH4UXBG34ZF6IP2XHBR5JDDBNCBOZK26NALIA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.235631800 (300h58m34.697724283s) [noise=true , address=77.90.31.15:4698 , piece_id=NOCO6KPTUBBKYCITDRFBKA3H5EJTKMDAO4AFQFFXC4PPH3D46X2A, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:50.988532254 (300h58m36.450624747s) [upload=264873]: Next returned nil segment Oct 6 02:29:50.988555589 (300h58m36.450648082s) [upload=264873]: Waiting for error group managing segments... Oct 6 02:29:51.282838267 (300h58m36.74493075s) [noise=true , address=45.91.116.4:4937 , piece_id=CHJRUVVLWAR7627JOLKL2QHAK7EHW72ZTGE3BTOAAM4ZL5AZL3KQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.318573943 (300h58m36.780666416s) [noise=true , address=77.90.31.18:4805 , piece_id=LJKUWHGMINU2P7JTS2QN5VP25HPZBKRMFXRDBSO5OTVYTS47SB4A, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.322112521 (300h58m36.784205004s) [noise=true , address=77.90.31.15:4698 , piece_id=NOCO6KPTUBBKYCITDRFBKA3H5EJTKMDAO4AFQFFXC4PPH3D46X2A, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.350536624 (300h58m36.812629108s) [noise=true , address=45.91.116.6:4974 , piece_id=DBXKLTPEBNF5GVB4PC26BY4EPXOWHU2OWSLWGSHS52S6CDUZIMOA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.362171679 (300h58m36.824264162s) [noise=true , address=45.91.116.10:4081 , piece_id=PRCQUWMDPZTR2CSWMOODBECZOLAVJ4OPOUTGQHCPTWXUDQGY2MBQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.366398738 (300h58m36.828491221s) [noise=true , address=45.91.116.10:4082 , piece_id=NX5W46AGWBSZ7EM2LZHZDPE6MEPSLZMNVFZEE2LQ5F5LV7YSQRKQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.375437720 (300h58m36.837530203s) [noise=true , address=45.91.116.14:4190 , piece_id=OS3T2GGABRXYK6HNEXK4C5F5HTJFDEP7SHA5QV2R3SCMXDE2TD5A, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.397600181 (300h58m36.859692664s) [noise=true , address=77.90.31.13:4628 , piece_id=52Q5TEMXLZHLWJ5QWRFHGYWW6RCL2Y57AY6K2J7I3BUA22BNARQQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.400132462 (300h58m36.862224945s) [noise=true , address=45.91.116.5:4298 , piece_id=VTE4WIBLUWV74XMSSPP37CEAWPRQHA3GFV2RJRQYGRQM5JGF3H3Q, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.412114018 (300h58m36.874206491s) [noise=true , address=45.91.116.7:4010 , piece_id=S5NHKRILOZ47C4NXH4UXBG34ZF6IP2XHBR5JDDBNCBOZK26NALIA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:30:14.302718308 (300h58m59.764810791s) [noise=true , address=145.79.227.18:4007 , piece_id=6PZE3N3QHDGV5K5HULQE5UJXWWG7RAIACP2OVEZVIOTTCWVNQO6Q, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:30:14.319938798 (300h58m59.782031281s) [noise=true , address=145.79.227.7:4294 , piece_id=TLZVEBMQCCWINXB4AVRKHWKZ7WULCGUC6E5GGDJQLASKWNEHSJPA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:30:16.104270571 (300h59m1.566363044s) [noise=true , address=145.79.227.16:4910 , piece_id=T254A3S7TTJEBOIBDDCS34DU7HV25TXDTNZWF2OGE4KRQ7YEA5ZA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:30:16.104295528 (300h59m1.566388012s) [seg_pos={0 0}, upload=264873]: Segment reached optimal threshold of 13 pieces. Oct 6 02:30:16.104442268 (300h59m1.566534751s) [noise=true , address=145.79.227.4:4185 , piece_id=ONU6YDQFAMJPRG3ZF45M7KUX5EGXZVQEGCTOJK3VGE2JXXLHCBAQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN): protocol: write tcp 188.166.225.170:34966->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:34966->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34966->145.79.227.4:4185: use of closed network connection Oct 6 02:30:16.104456395 (300h59m1.566548878s) [noise=true , address=145.79.227.4:4185 , piece_id=ONU6YDQFAMJPRG3ZF45M7KUX5EGXZVQEGCTOJK3VGE2JXXLHCBAQ, seg_pos={0 0}, upload=264873]: Long tail cancellation due to optimal threshold Oct 6 02:30:16.104598005 (300h59m1.566690478s) [noise=true , address=145.79.227.19:4079 , piece_id=2VBVLN2SNA6WUICKMNMNM7URPVUGWPB5SVZ335XZ7HIAUM3FY5VA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: write tcp 188.166.225.170:57162->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:57162->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57162->145.79.227.19:4079: use of closed network connection Oct 6 02:30:16.104604216 (300h59m1.566696699s) [noise=true , address=145.79.227.19:4079 , piece_id=2VBVLN2SNA6WUICKMNMNM7URPVUGWPB5SVZ335XZ7HIAUM3FY5VA, seg_pos={0 0}, upload=264873]: Long tail cancellation due to optimal threshold Oct 6 02:30:16.104672547 (300h59m1.56676503s) [noise=true , address=145.79.227.17:4972 , piece_id=I54FS5Q4CMMM46S6EP3C5MEUAJCVNWX23OHTBMNQNKP4TRERVZ7A, seg_pos={0 0}, upload=264873]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: write tcp 188.166.225.170:33630->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:33630->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33630->145.79.227.17:4972: use of closed network connection Oct 6 02:30:16.104678057 (300h59m1.56677053s) [noise=true , address=145.79.227.17:4972 , piece_id=I54FS5Q4CMMM46S6EP3C5MEUAJCVNWX23OHTBMNQNKP4TRERVZ7A, seg_pos={0 0}, upload=264873]: Long tail cancellation due to optimal threshold Oct 6 02:30:16.104695541 (300h59m1.566788024s) [seg_pos={0 0}, upload=264873]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 6 02:30:16.104762427 (300h59m1.56685491s) [upload=264873]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 6 02:30:16.104768159 (300h59m1.566860632s) [upload=264873]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 6 02:30:16.463894599 (300h59m1.925987092s) [upload=264873]: Done uploading segments... Oct 6 02:30:16.463937391 (300h59m1.926029874s) [upload=264873]: Upload finished. err: <nil> Duration:* 6.048948483s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 6.107760211s FAIL (2025-10-06 02:42:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, Piece ID: T254A3S7TTJEBOIBDDCS34DU7HV25TXDTNZWF2OGE4KRQ7YEA5ZA) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: OS3T2GGABRXYK6HNEXK4C5F5HTJFDEP7SHA5QV2R3SCMXDE2TD5A) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: S5NHKRILOZ47C4NXH4UXBG34ZF6IP2XHBR5JDDBNCBOZK26NALIA) context deadline exceeded; piecestore: (Node ID: 12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, Piece ID: CHJRUVVLWAR7627JOLKL2QHAK7EHW72ZTGE3BTOAAM4ZL5AZL3KQ) context deadline exceeded; piecestore: (Node ID: 12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, Piece ID: TLZVEBMQCCWINXB4AVRKHWKZ7WULCGUC6E5GGDJQLASKWNEHSJPA) context deadline exceeded; piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: LJKUWHGMINU2P7JTS2QN5VP25HPZBKRMFXRDBSO5OTVYTS47SB4A) context deadline exceeded; piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: VTE4WIBLUWV74XMSSPP37CEAWPRQHA3GFV2RJRQYGRQM5JGF3H3Q) context deadline exceeded; piecestore: (Node ID: 1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, Piece ID: 52Q5TEMXLZHLWJ5QWRFHGYWW6RCL2Y57AY6K2J7I3BUA22BNARQQ) context deadline exceeded ======LOGS====== 914003723 (300h58m34.376096196s) [upload=264873]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 02:29:49.215144679 (300h58m34.677237162s) [seg_pos={0 0}, upload=264873]: Begin upload segment... Oct 6 02:29:49.215212839 (300h58m34.677305312s) [seg_pos={0 0}, upload=264873]: Done begin upload segment. Oct 6 02:29:49.215241294 (300h58m34.677333777s) [noise=true , address=145.79.227.7:4294 , piece_id=TLZVEBMQCCWINXB4AVRKHWKZ7WULCGUC6E5GGDJQLASKWNEHSJPA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.215249739 (300h58m34.677342232s) [noise=true , address=45.91.116.4:4937 , piece_id=CHJRUVVLWAR7627JOLKL2QHAK7EHW72ZTGE3BTOAAM4ZL5AZL3KQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.215349820 (300h58m34.677442303s) [noise=true , address=77.90.31.18:4805 , piece_id=LJKUWHGMINU2P7JTS2QN5VP25HPZBKRMFXRDBSO5OTVYTS47SB4A, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.216239644 (300h58m34.678332127s) [noise=true , address=45.91.116.5:4298 , piece_id=VTE4WIBLUWV74XMSSPP37CEAWPRQHA3GFV2RJRQYGRQM5JGF3H3Q, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.218583898 (300h58m34.680676392s) [noise=true , address=45.91.116.10:4082 , piece_id=NX5W46AGWBSZ7EM2LZHZDPE6MEPSLZMNVFZEE2LQ5F5LV7YSQRKQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.219326442 (300h58m34.681418925s) [noise=true , address=45.91.116.6:4974 , piece_id=DBXKLTPEBNF5GVB4PC26BY4EPXOWHU2OWSLWGSHS52S6CDUZIMOA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.222198982 (300h58m34.684291465s) [noise=true , address=145.79.227.17:4972 , piece_id=I54FS5Q4CMMM46S6EP3C5MEUAJCVNWX23OHTBMNQNKP4TRERVZ7A, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.222520114 (300h58m34.684612597s) [noise=true , address=145.79.227.4:4185 , piece_id=ONU6YDQFAMJPRG3ZF45M7KUX5EGXZVQEGCTOJK3VGE2JXXLHCBAQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.225940076 (300h58m34.688032569s) [noise=true , address=77.90.31.13:4628 , piece_id=52Q5TEMXLZHLWJ5QWRFHGYWW6RCL2Y57AY6K2J7I3BUA22BNARQQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.227928832 (300h58m34.690021305s) [noise=true , address=145.79.227.16:4910 , piece_id=T254A3S7TTJEBOIBDDCS34DU7HV25TXDTNZWF2OGE4KRQ7YEA5ZA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.227998645 (300h58m34.690091128s) [noise=true , address=45.91.116.14:4190 , piece_id=OS3T2GGABRXYK6HNEXK4C5F5HTJFDEP7SHA5QV2R3SCMXDE2TD5A, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.231020309 (300h58m34.693112792s) [noise=true , address=145.79.227.18:4007 , piece_id=6PZE3N3QHDGV5K5HULQE5UJXWWG7RAIACP2OVEZVIOTTCWVNQO6Q, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.232908575 (300h58m34.695001068s) [noise=true , address=45.91.116.10:4081 , piece_id=PRCQUWMDPZTR2CSWMOODBECZOLAVJ4OPOUTGQHCPTWXUDQGY2MBQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.233037220 (300h58m34.695129693s) [noise=true , address=145.79.227.19:4079 , piece_id=2VBVLN2SNA6WUICKMNMNM7URPVUGWPB5SVZ335XZ7HIAUM3FY5VA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.234302058 (300h58m34.696394531s) [noise=true , address=45.91.116.7:4010 , piece_id=S5NHKRILOZ47C4NXH4UXBG34ZF6IP2XHBR5JDDBNCBOZK26NALIA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.235631800 (300h58m34.697724283s) [noise=true , address=77.90.31.15:4698 , piece_id=NOCO6KPTUBBKYCITDRFBKA3H5EJTKMDAO4AFQFFXC4PPH3D46X2A, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:50.988532254 (300h58m36.450624747s) [upload=264873]: Next returned nil segment Oct 6 02:29:50.988555589 (300h58m36.450648082s) [upload=264873]: Waiting for error group managing segments... Oct 6 02:29:51.282838267 (300h58m36.74493075s) [noise=true , address=45.91.116.4:4937 , piece_id=CHJRUVVLWAR7627JOLKL2QHAK7EHW72ZTGE3BTOAAM4ZL5AZL3KQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.318573943 (300h58m36.780666416s) [noise=true , address=77.90.31.18:4805 , piece_id=LJKUWHGMINU2P7JTS2QN5VP25HPZBKRMFXRDBSO5OTVYTS47SB4A, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.322112521 (300h58m36.784205004s) [noise=true , address=77.90.31.15:4698 , piece_id=NOCO6KPTUBBKYCITDRFBKA3H5EJTKMDAO4AFQFFXC4PPH3D46X2A, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.350536624 (300h58m36.812629108s) [noise=true , address=45.91.116.6:4974 , piece_id=DBXKLTPEBNF5GVB4PC26BY4EPXOWHU2OWSLWGSHS52S6CDUZIMOA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.362171679 (300h58m36.824264162s) [noise=true , address=45.91.116.10:4081 , piece_id=PRCQUWMDPZTR2CSWMOODBECZOLAVJ4OPOUTGQHCPTWXUDQGY2MBQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.366398738 (300h58m36.828491221s) [noise=true , address=45.91.116.10:4082 , piece_id=NX5W46AGWBSZ7EM2LZHZDPE6MEPSLZMNVFZEE2LQ5F5LV7YSQRKQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.375437720 (300h58m36.837530203s) [noise=true , address=45.91.116.14:4190 , piece_id=OS3T2GGABRXYK6HNEXK4C5F5HTJFDEP7SHA5QV2R3SCMXDE2TD5A, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.397600181 (300h58m36.859692664s) [noise=true , address=77.90.31.13:4628 , piece_id=52Q5TEMXLZHLWJ5QWRFHGYWW6RCL2Y57AY6K2J7I3BUA22BNARQQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.400132462 (300h58m36.862224945s) [noise=true , address=45.91.116.5:4298 , piece_id=VTE4WIBLUWV74XMSSPP37CEAWPRQHA3GFV2RJRQYGRQM5JGF3H3Q, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.412114018 (300h58m36.874206491s) [noise=true , address=45.91.116.7:4010 , piece_id=S5NHKRILOZ47C4NXH4UXBG34ZF6IP2XHBR5JDDBNCBOZK26NALIA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:30:14.302718308 (300h58m59.764810791s) [noise=true , address=145.79.227.18:4007 , piece_id=6PZE3N3QHDGV5K5HULQE5UJXWWG7RAIACP2OVEZVIOTTCWVNQO6Q, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:30:14.319938798 (300h58m59.782031281s) [noise=true , address=145.79.227.7:4294 , piece_id=TLZVEBMQCCWINXB4AVRKHWKZ7WULCGUC6E5GGDJQLASKWNEHSJPA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:30:16.104270571 (300h59m1.566363044s) [noise=true , address=145.79.227.16:4910 , piece_id=T254A3S7TTJEBOIBDDCS34DU7HV25TXDTNZWF2OGE4KRQ7YEA5ZA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:30:16.104295528 (300h59m1.566388012s) [seg_pos={0 0}, upload=264873]: Segment reached optimal threshold of 13 pieces. Oct 6 02:30:16.104442268 (300h59m1.566534751s) [noise=true , address=145.79.227.4:4185 , piece_id=ONU6YDQFAMJPRG3ZF45M7KUX5EGXZVQEGCTOJK3VGE2JXXLHCBAQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN): protocol: write tcp 188.166.225.170:34966->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:34966->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34966->145.79.227.4:4185: use of closed network connection Oct 6 02:30:16.104456395 (300h59m1.566548878s) [noise=true , address=145.79.227.4:4185 , piece_id=ONU6YDQFAMJPRG3ZF45M7KUX5EGXZVQEGCTOJK3VGE2JXXLHCBAQ, seg_pos={0 0}, upload=264873]: Long tail cancellation due to optimal threshold Oct 6 02:30:16.104598005 (300h59m1.566690478s) [noise=true , address=145.79.227.19:4079 , piece_id=2VBVLN2SNA6WUICKMNMNM7URPVUGWPB5SVZ335XZ7HIAUM3FY5VA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: write tcp 188.166.225.170:57162->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:57162->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57162->145.79.227.19:4079: use of closed network connection Oct 6 02:30:16.104604216 (300h59m1.566696699s) [noise=true , address=145.79.227.19:4079 , piece_id=2VBVLN2SNA6WUICKMNMNM7URPVUGWPB5SVZ335XZ7HIAUM3FY5VA, seg_pos={0 0}, upload=264873]: Long tail cancellation due to optimal threshold Oct 6 02:30:16.104672547 (300h59m1.56676503s) [noise=true , address=145.79.227.17:4972 , piece_id=I54FS5Q4CMMM46S6EP3C5MEUAJCVNWX23OHTBMNQNKP4TRERVZ7A, seg_pos={0 0}, upload=264873]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: write tcp 188.166.225.170:33630->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:33630->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33630->145.79.227.17:4972: use of closed network connection Oct 6 02:30:16.104678057 (300h59m1.56677053s) [noise=true , address=145.79.227.17:4972 , piece_id=I54FS5Q4CMMM46S6EP3C5MEUAJCVNWX23OHTBMNQNKP4TRERVZ7A, seg_pos={0 0}, upload=264873]: Long tail cancellation due to optimal threshold Oct 6 02:30:16.104695541 (300h59m1.566788024s) [seg_pos={0 0}, upload=264873]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 6 02:30:16.104762427 (300h59m1.56685491s) [upload=264873]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 6 02:30:16.104768159 (300h59m1.566860632s) [upload=264873]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 6 02:30:16.463894599 (300h59m1.925987092s) [upload=264873]: Done uploading segments... Oct 6 02:30:16.463937391 (300h59m1.926029874s) [upload=264873]: Upload finished. err: <nil> Duration:* 6.107793904s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 6.183287384s FAIL (2025-10-06 02:41:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, Piece ID: T254A3S7TTJEBOIBDDCS34DU7HV25TXDTNZWF2OGE4KRQ7YEA5ZA) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: OS3T2GGABRXYK6HNEXK4C5F5HTJFDEP7SHA5QV2R3SCMXDE2TD5A) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: S5NHKRILOZ47C4NXH4UXBG34ZF6IP2XHBR5JDDBNCBOZK26NALIA) context deadline exceeded; piecestore: (Node ID: 12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, Piece ID: CHJRUVVLWAR7627JOLKL2QHAK7EHW72ZTGE3BTOAAM4ZL5AZL3KQ) context deadline exceeded; piecestore: (Node ID: 12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, Piece ID: TLZVEBMQCCWINXB4AVRKHWKZ7WULCGUC6E5GGDJQLASKWNEHSJPA) context deadline exceeded; piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: LJKUWHGMINU2P7JTS2QN5VP25HPZBKRMFXRDBSO5OTVYTS47SB4A) context deadline exceeded; piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: VTE4WIBLUWV74XMSSPP37CEAWPRQHA3GFV2RJRQYGRQM5JGF3H3Q) context deadline exceeded; piecestore: (Node ID: 1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, Piece ID: 52Q5TEMXLZHLWJ5QWRFHGYWW6RCL2Y57AY6K2J7I3BUA22BNARQQ) context deadline exceeded ======LOGS====== 914003723 (300h58m34.376096196s) [upload=264873]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 02:29:49.215144679 (300h58m34.677237162s) [seg_pos={0 0}, upload=264873]: Begin upload segment... Oct 6 02:29:49.215212839 (300h58m34.677305312s) [seg_pos={0 0}, upload=264873]: Done begin upload segment. Oct 6 02:29:49.215241294 (300h58m34.677333777s) [noise=true , address=145.79.227.7:4294 , piece_id=TLZVEBMQCCWINXB4AVRKHWKZ7WULCGUC6E5GGDJQLASKWNEHSJPA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.215249739 (300h58m34.677342232s) [noise=true , address=45.91.116.4:4937 , piece_id=CHJRUVVLWAR7627JOLKL2QHAK7EHW72ZTGE3BTOAAM4ZL5AZL3KQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.215349820 (300h58m34.677442303s) [noise=true , address=77.90.31.18:4805 , piece_id=LJKUWHGMINU2P7JTS2QN5VP25HPZBKRMFXRDBSO5OTVYTS47SB4A, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.216239644 (300h58m34.678332127s) [noise=true , address=45.91.116.5:4298 , piece_id=VTE4WIBLUWV74XMSSPP37CEAWPRQHA3GFV2RJRQYGRQM5JGF3H3Q, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.218583898 (300h58m34.680676392s) [noise=true , address=45.91.116.10:4082 , piece_id=NX5W46AGWBSZ7EM2LZHZDPE6MEPSLZMNVFZEE2LQ5F5LV7YSQRKQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.219326442 (300h58m34.681418925s) [noise=true , address=45.91.116.6:4974 , piece_id=DBXKLTPEBNF5GVB4PC26BY4EPXOWHU2OWSLWGSHS52S6CDUZIMOA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.222198982 (300h58m34.684291465s) [noise=true , address=145.79.227.17:4972 , piece_id=I54FS5Q4CMMM46S6EP3C5MEUAJCVNWX23OHTBMNQNKP4TRERVZ7A, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.222520114 (300h58m34.684612597s) [noise=true , address=145.79.227.4:4185 , piece_id=ONU6YDQFAMJPRG3ZF45M7KUX5EGXZVQEGCTOJK3VGE2JXXLHCBAQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.225940076 (300h58m34.688032569s) [noise=true , address=77.90.31.13:4628 , piece_id=52Q5TEMXLZHLWJ5QWRFHGYWW6RCL2Y57AY6K2J7I3BUA22BNARQQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.227928832 (300h58m34.690021305s) [noise=true , address=145.79.227.16:4910 , piece_id=T254A3S7TTJEBOIBDDCS34DU7HV25TXDTNZWF2OGE4KRQ7YEA5ZA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.227998645 (300h58m34.690091128s) [noise=true , address=45.91.116.14:4190 , piece_id=OS3T2GGABRXYK6HNEXK4C5F5HTJFDEP7SHA5QV2R3SCMXDE2TD5A, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.231020309 (300h58m34.693112792s) [noise=true , address=145.79.227.18:4007 , piece_id=6PZE3N3QHDGV5K5HULQE5UJXWWG7RAIACP2OVEZVIOTTCWVNQO6Q, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.232908575 (300h58m34.695001068s) [noise=true , address=45.91.116.10:4081 , piece_id=PRCQUWMDPZTR2CSWMOODBECZOLAVJ4OPOUTGQHCPTWXUDQGY2MBQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.233037220 (300h58m34.695129693s) [noise=true , address=145.79.227.19:4079 , piece_id=2VBVLN2SNA6WUICKMNMNM7URPVUGWPB5SVZ335XZ7HIAUM3FY5VA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.234302058 (300h58m34.696394531s) [noise=true , address=45.91.116.7:4010 , piece_id=S5NHKRILOZ47C4NXH4UXBG34ZF6IP2XHBR5JDDBNCBOZK26NALIA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.235631800 (300h58m34.697724283s) [noise=true , address=77.90.31.15:4698 , piece_id=NOCO6KPTUBBKYCITDRFBKA3H5EJTKMDAO4AFQFFXC4PPH3D46X2A, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:50.988532254 (300h58m36.450624747s) [upload=264873]: Next returned nil segment Oct 6 02:29:50.988555589 (300h58m36.450648082s) [upload=264873]: Waiting for error group managing segments... Oct 6 02:29:51.282838267 (300h58m36.74493075s) [noise=true , address=45.91.116.4:4937 , piece_id=CHJRUVVLWAR7627JOLKL2QHAK7EHW72ZTGE3BTOAAM4ZL5AZL3KQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.318573943 (300h58m36.780666416s) [noise=true , address=77.90.31.18:4805 , piece_id=LJKUWHGMINU2P7JTS2QN5VP25HPZBKRMFXRDBSO5OTVYTS47SB4A, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.322112521 (300h58m36.784205004s) [noise=true , address=77.90.31.15:4698 , piece_id=NOCO6KPTUBBKYCITDRFBKA3H5EJTKMDAO4AFQFFXC4PPH3D46X2A, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.350536624 (300h58m36.812629108s) [noise=true , address=45.91.116.6:4974 , piece_id=DBXKLTPEBNF5GVB4PC26BY4EPXOWHU2OWSLWGSHS52S6CDUZIMOA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.362171679 (300h58m36.824264162s) [noise=true , address=45.91.116.10:4081 , piece_id=PRCQUWMDPZTR2CSWMOODBECZOLAVJ4OPOUTGQHCPTWXUDQGY2MBQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.366398738 (300h58m36.828491221s) [noise=true , address=45.91.116.10:4082 , piece_id=NX5W46AGWBSZ7EM2LZHZDPE6MEPSLZMNVFZEE2LQ5F5LV7YSQRKQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.375437720 (300h58m36.837530203s) [noise=true , address=45.91.116.14:4190 , piece_id=OS3T2GGABRXYK6HNEXK4C5F5HTJFDEP7SHA5QV2R3SCMXDE2TD5A, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.397600181 (300h58m36.859692664s) [noise=true , address=77.90.31.13:4628 , piece_id=52Q5TEMXLZHLWJ5QWRFHGYWW6RCL2Y57AY6K2J7I3BUA22BNARQQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.400132462 (300h58m36.862224945s) [noise=true , address=45.91.116.5:4298 , piece_id=VTE4WIBLUWV74XMSSPP37CEAWPRQHA3GFV2RJRQYGRQM5JGF3H3Q, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.412114018 (300h58m36.874206491s) [noise=true , address=45.91.116.7:4010 , piece_id=S5NHKRILOZ47C4NXH4UXBG34ZF6IP2XHBR5JDDBNCBOZK26NALIA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:30:14.302718308 (300h58m59.764810791s) [noise=true , address=145.79.227.18:4007 , piece_id=6PZE3N3QHDGV5K5HULQE5UJXWWG7RAIACP2OVEZVIOTTCWVNQO6Q, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:30:14.319938798 (300h58m59.782031281s) [noise=true , address=145.79.227.7:4294 , piece_id=TLZVEBMQCCWINXB4AVRKHWKZ7WULCGUC6E5GGDJQLASKWNEHSJPA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:30:16.104270571 (300h59m1.566363044s) [noise=true , address=145.79.227.16:4910 , piece_id=T254A3S7TTJEBOIBDDCS34DU7HV25TXDTNZWF2OGE4KRQ7YEA5ZA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:30:16.104295528 (300h59m1.566388012s) [seg_pos={0 0}, upload=264873]: Segment reached optimal threshold of 13 pieces. Oct 6 02:30:16.104442268 (300h59m1.566534751s) [noise=true , address=145.79.227.4:4185 , piece_id=ONU6YDQFAMJPRG3ZF45M7KUX5EGXZVQEGCTOJK3VGE2JXXLHCBAQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN): protocol: write tcp 188.166.225.170:34966->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:34966->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34966->145.79.227.4:4185: use of closed network connection Oct 6 02:30:16.104456395 (300h59m1.566548878s) [noise=true , address=145.79.227.4:4185 , piece_id=ONU6YDQFAMJPRG3ZF45M7KUX5EGXZVQEGCTOJK3VGE2JXXLHCBAQ, seg_pos={0 0}, upload=264873]: Long tail cancellation due to optimal threshold Oct 6 02:30:16.104598005 (300h59m1.566690478s) [noise=true , address=145.79.227.19:4079 , piece_id=2VBVLN2SNA6WUICKMNMNM7URPVUGWPB5SVZ335XZ7HIAUM3FY5VA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: write tcp 188.166.225.170:57162->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:57162->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57162->145.79.227.19:4079: use of closed network connection Oct 6 02:30:16.104604216 (300h59m1.566696699s) [noise=true , address=145.79.227.19:4079 , piece_id=2VBVLN2SNA6WUICKMNMNM7URPVUGWPB5SVZ335XZ7HIAUM3FY5VA, seg_pos={0 0}, upload=264873]: Long tail cancellation due to optimal threshold Oct 6 02:30:16.104672547 (300h59m1.56676503s) [noise=true , address=145.79.227.17:4972 , piece_id=I54FS5Q4CMMM46S6EP3C5MEUAJCVNWX23OHTBMNQNKP4TRERVZ7A, seg_pos={0 0}, upload=264873]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: write tcp 188.166.225.170:33630->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:33630->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33630->145.79.227.17:4972: use of closed network connection Oct 6 02:30:16.104678057 (300h59m1.56677053s) [noise=true , address=145.79.227.17:4972 , piece_id=I54FS5Q4CMMM46S6EP3C5MEUAJCVNWX23OHTBMNQNKP4TRERVZ7A, seg_pos={0 0}, upload=264873]: Long tail cancellation due to optimal threshold Oct 6 02:30:16.104695541 (300h59m1.566788024s) [seg_pos={0 0}, upload=264873]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 6 02:30:16.104762427 (300h59m1.56685491s) [upload=264873]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 6 02:30:16.104768159 (300h59m1.566860632s) [upload=264873]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 6 02:30:16.463894599 (300h59m1.925987092s) [upload=264873]: Done uploading segments... Oct 6 02:30:16.463937391 (300h59m1.926029874s) [upload=264873]: Upload finished. err: <nil> Duration:* 6.183325548s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 6.050064488s FAIL (2025-10-06 02:41:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, Piece ID: T254A3S7TTJEBOIBDDCS34DU7HV25TXDTNZWF2OGE4KRQ7YEA5ZA) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: OS3T2GGABRXYK6HNEXK4C5F5HTJFDEP7SHA5QV2R3SCMXDE2TD5A) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: S5NHKRILOZ47C4NXH4UXBG34ZF6IP2XHBR5JDDBNCBOZK26NALIA) context deadline exceeded; piecestore: (Node ID: 12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, Piece ID: CHJRUVVLWAR7627JOLKL2QHAK7EHW72ZTGE3BTOAAM4ZL5AZL3KQ) context deadline exceeded; piecestore: (Node ID: 12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, Piece ID: TLZVEBMQCCWINXB4AVRKHWKZ7WULCGUC6E5GGDJQLASKWNEHSJPA) context deadline exceeded; piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: LJKUWHGMINU2P7JTS2QN5VP25HPZBKRMFXRDBSO5OTVYTS47SB4A) context deadline exceeded; piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: VTE4WIBLUWV74XMSSPP37CEAWPRQHA3GFV2RJRQYGRQM5JGF3H3Q) context deadline exceeded; piecestore: (Node ID: 1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, Piece ID: 52Q5TEMXLZHLWJ5QWRFHGYWW6RCL2Y57AY6K2J7I3BUA22BNARQQ) context deadline exceeded ======LOGS====== 914003723 (300h58m34.376096196s) [upload=264873]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 02:29:49.215144679 (300h58m34.677237162s) [seg_pos={0 0}, upload=264873]: Begin upload segment... Oct 6 02:29:49.215212839 (300h58m34.677305312s) [seg_pos={0 0}, upload=264873]: Done begin upload segment. Oct 6 02:29:49.215241294 (300h58m34.677333777s) [noise=true , address=145.79.227.7:4294 , piece_id=TLZVEBMQCCWINXB4AVRKHWKZ7WULCGUC6E5GGDJQLASKWNEHSJPA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.215249739 (300h58m34.677342232s) [noise=true , address=45.91.116.4:4937 , piece_id=CHJRUVVLWAR7627JOLKL2QHAK7EHW72ZTGE3BTOAAM4ZL5AZL3KQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.215349820 (300h58m34.677442303s) [noise=true , address=77.90.31.18:4805 , piece_id=LJKUWHGMINU2P7JTS2QN5VP25HPZBKRMFXRDBSO5OTVYTS47SB4A, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.216239644 (300h58m34.678332127s) [noise=true , address=45.91.116.5:4298 , piece_id=VTE4WIBLUWV74XMSSPP37CEAWPRQHA3GFV2RJRQYGRQM5JGF3H3Q, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.218583898 (300h58m34.680676392s) [noise=true , address=45.91.116.10:4082 , piece_id=NX5W46AGWBSZ7EM2LZHZDPE6MEPSLZMNVFZEE2LQ5F5LV7YSQRKQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.219326442 (300h58m34.681418925s) [noise=true , address=45.91.116.6:4974 , piece_id=DBXKLTPEBNF5GVB4PC26BY4EPXOWHU2OWSLWGSHS52S6CDUZIMOA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.222198982 (300h58m34.684291465s) [noise=true , address=145.79.227.17:4972 , piece_id=I54FS5Q4CMMM46S6EP3C5MEUAJCVNWX23OHTBMNQNKP4TRERVZ7A, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.222520114 (300h58m34.684612597s) [noise=true , address=145.79.227.4:4185 , piece_id=ONU6YDQFAMJPRG3ZF45M7KUX5EGXZVQEGCTOJK3VGE2JXXLHCBAQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.225940076 (300h58m34.688032569s) [noise=true , address=77.90.31.13:4628 , piece_id=52Q5TEMXLZHLWJ5QWRFHGYWW6RCL2Y57AY6K2J7I3BUA22BNARQQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.227928832 (300h58m34.690021305s) [noise=true , address=145.79.227.16:4910 , piece_id=T254A3S7TTJEBOIBDDCS34DU7HV25TXDTNZWF2OGE4KRQ7YEA5ZA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.227998645 (300h58m34.690091128s) [noise=true , address=45.91.116.14:4190 , piece_id=OS3T2GGABRXYK6HNEXK4C5F5HTJFDEP7SHA5QV2R3SCMXDE2TD5A, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.231020309 (300h58m34.693112792s) [noise=true , address=145.79.227.18:4007 , piece_id=6PZE3N3QHDGV5K5HULQE5UJXWWG7RAIACP2OVEZVIOTTCWVNQO6Q, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.232908575 (300h58m34.695001068s) [noise=true , address=45.91.116.10:4081 , piece_id=PRCQUWMDPZTR2CSWMOODBECZOLAVJ4OPOUTGQHCPTWXUDQGY2MBQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.233037220 (300h58m34.695129693s) [noise=true , address=145.79.227.19:4079 , piece_id=2VBVLN2SNA6WUICKMNMNM7URPVUGWPB5SVZ335XZ7HIAUM3FY5VA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.234302058 (300h58m34.696394531s) [noise=true , address=45.91.116.7:4010 , piece_id=S5NHKRILOZ47C4NXH4UXBG34ZF6IP2XHBR5JDDBNCBOZK26NALIA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.235631800 (300h58m34.697724283s) [noise=true , address=77.90.31.15:4698 , piece_id=NOCO6KPTUBBKYCITDRFBKA3H5EJTKMDAO4AFQFFXC4PPH3D46X2A, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:50.988532254 (300h58m36.450624747s) [upload=264873]: Next returned nil segment Oct 6 02:29:50.988555589 (300h58m36.450648082s) [upload=264873]: Waiting for error group managing segments... Oct 6 02:29:51.282838267 (300h58m36.74493075s) [noise=true , address=45.91.116.4:4937 , piece_id=CHJRUVVLWAR7627JOLKL2QHAK7EHW72ZTGE3BTOAAM4ZL5AZL3KQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.318573943 (300h58m36.780666416s) [noise=true , address=77.90.31.18:4805 , piece_id=LJKUWHGMINU2P7JTS2QN5VP25HPZBKRMFXRDBSO5OTVYTS47SB4A, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.322112521 (300h58m36.784205004s) [noise=true , address=77.90.31.15:4698 , piece_id=NOCO6KPTUBBKYCITDRFBKA3H5EJTKMDAO4AFQFFXC4PPH3D46X2A, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.350536624 (300h58m36.812629108s) [noise=true , address=45.91.116.6:4974 , piece_id=DBXKLTPEBNF5GVB4PC26BY4EPXOWHU2OWSLWGSHS52S6CDUZIMOA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.362171679 (300h58m36.824264162s) [noise=true , address=45.91.116.10:4081 , piece_id=PRCQUWMDPZTR2CSWMOODBECZOLAVJ4OPOUTGQHCPTWXUDQGY2MBQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.366398738 (300h58m36.828491221s) [noise=true , address=45.91.116.10:4082 , piece_id=NX5W46AGWBSZ7EM2LZHZDPE6MEPSLZMNVFZEE2LQ5F5LV7YSQRKQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.375437720 (300h58m36.837530203s) [noise=true , address=45.91.116.14:4190 , piece_id=OS3T2GGABRXYK6HNEXK4C5F5HTJFDEP7SHA5QV2R3SCMXDE2TD5A, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.397600181 (300h58m36.859692664s) [noise=true , address=77.90.31.13:4628 , piece_id=52Q5TEMXLZHLWJ5QWRFHGYWW6RCL2Y57AY6K2J7I3BUA22BNARQQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.400132462 (300h58m36.862224945s) [noise=true , address=45.91.116.5:4298 , piece_id=VTE4WIBLUWV74XMSSPP37CEAWPRQHA3GFV2RJRQYGRQM5JGF3H3Q, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.412114018 (300h58m36.874206491s) [noise=true , address=45.91.116.7:4010 , piece_id=S5NHKRILOZ47C4NXH4UXBG34ZF6IP2XHBR5JDDBNCBOZK26NALIA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:30:14.302718308 (300h58m59.764810791s) [noise=true , address=145.79.227.18:4007 , piece_id=6PZE3N3QHDGV5K5HULQE5UJXWWG7RAIACP2OVEZVIOTTCWVNQO6Q, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:30:14.319938798 (300h58m59.782031281s) [noise=true , address=145.79.227.7:4294 , piece_id=TLZVEBMQCCWINXB4AVRKHWKZ7WULCGUC6E5GGDJQLASKWNEHSJPA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:30:16.104270571 (300h59m1.566363044s) [noise=true , address=145.79.227.16:4910 , piece_id=T254A3S7TTJEBOIBDDCS34DU7HV25TXDTNZWF2OGE4KRQ7YEA5ZA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:30:16.104295528 (300h59m1.566388012s) [seg_pos={0 0}, upload=264873]: Segment reached optimal threshold of 13 pieces. Oct 6 02:30:16.104442268 (300h59m1.566534751s) [noise=true , address=145.79.227.4:4185 , piece_id=ONU6YDQFAMJPRG3ZF45M7KUX5EGXZVQEGCTOJK3VGE2JXXLHCBAQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN): protocol: write tcp 188.166.225.170:34966->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:34966->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34966->145.79.227.4:4185: use of closed network connection Oct 6 02:30:16.104456395 (300h59m1.566548878s) [noise=true , address=145.79.227.4:4185 , piece_id=ONU6YDQFAMJPRG3ZF45M7KUX5EGXZVQEGCTOJK3VGE2JXXLHCBAQ, seg_pos={0 0}, upload=264873]: Long tail cancellation due to optimal threshold Oct 6 02:30:16.104598005 (300h59m1.566690478s) [noise=true , address=145.79.227.19:4079 , piece_id=2VBVLN2SNA6WUICKMNMNM7URPVUGWPB5SVZ335XZ7HIAUM3FY5VA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: write tcp 188.166.225.170:57162->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:57162->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57162->145.79.227.19:4079: use of closed network connection Oct 6 02:30:16.104604216 (300h59m1.566696699s) [noise=true , address=145.79.227.19:4079 , piece_id=2VBVLN2SNA6WUICKMNMNM7URPVUGWPB5SVZ335XZ7HIAUM3FY5VA, seg_pos={0 0}, upload=264873]: Long tail cancellation due to optimal threshold Oct 6 02:30:16.104672547 (300h59m1.56676503s) [noise=true , address=145.79.227.17:4972 , piece_id=I54FS5Q4CMMM46S6EP3C5MEUAJCVNWX23OHTBMNQNKP4TRERVZ7A, seg_pos={0 0}, upload=264873]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: write tcp 188.166.225.170:33630->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:33630->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33630->145.79.227.17:4972: use of closed network connection Oct 6 02:30:16.104678057 (300h59m1.56677053s) [noise=true , address=145.79.227.17:4972 , piece_id=I54FS5Q4CMMM46S6EP3C5MEUAJCVNWX23OHTBMNQNKP4TRERVZ7A, seg_pos={0 0}, upload=264873]: Long tail cancellation due to optimal threshold Oct 6 02:30:16.104695541 (300h59m1.566788024s) [seg_pos={0 0}, upload=264873]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 6 02:30:16.104762427 (300h59m1.56685491s) [upload=264873]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 6 02:30:16.104768159 (300h59m1.566860632s) [upload=264873]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 6 02:30:16.463894599 (300h59m1.925987092s) [upload=264873]: Done uploading segments... Oct 6 02:30:16.463937391 (300h59m1.926029874s) [upload=264873]: Upload finished. err: <nil> Duration:* 6.050108231s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 6.162240457s FAIL (2025-10-06 02:40:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, Piece ID: T254A3S7TTJEBOIBDDCS34DU7HV25TXDTNZWF2OGE4KRQ7YEA5ZA) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: OS3T2GGABRXYK6HNEXK4C5F5HTJFDEP7SHA5QV2R3SCMXDE2TD5A) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: S5NHKRILOZ47C4NXH4UXBG34ZF6IP2XHBR5JDDBNCBOZK26NALIA) context deadline exceeded; piecestore: (Node ID: 12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, Piece ID: CHJRUVVLWAR7627JOLKL2QHAK7EHW72ZTGE3BTOAAM4ZL5AZL3KQ) context deadline exceeded; piecestore: (Node ID: 12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, Piece ID: TLZVEBMQCCWINXB4AVRKHWKZ7WULCGUC6E5GGDJQLASKWNEHSJPA) context deadline exceeded; piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: LJKUWHGMINU2P7JTS2QN5VP25HPZBKRMFXRDBSO5OTVYTS47SB4A) context deadline exceeded; piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: VTE4WIBLUWV74XMSSPP37CEAWPRQHA3GFV2RJRQYGRQM5JGF3H3Q) context deadline exceeded; piecestore: (Node ID: 1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, Piece ID: 52Q5TEMXLZHLWJ5QWRFHGYWW6RCL2Y57AY6K2J7I3BUA22BNARQQ) context deadline exceeded ======LOGS====== 914003723 (300h58m34.376096196s) [upload=264873]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 02:29:49.215144679 (300h58m34.677237162s) [seg_pos={0 0}, upload=264873]: Begin upload segment... Oct 6 02:29:49.215212839 (300h58m34.677305312s) [seg_pos={0 0}, upload=264873]: Done begin upload segment. Oct 6 02:29:49.215241294 (300h58m34.677333777s) [noise=true , address=145.79.227.7:4294 , piece_id=TLZVEBMQCCWINXB4AVRKHWKZ7WULCGUC6E5GGDJQLASKWNEHSJPA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.215249739 (300h58m34.677342232s) [noise=true , address=45.91.116.4:4937 , piece_id=CHJRUVVLWAR7627JOLKL2QHAK7EHW72ZTGE3BTOAAM4ZL5AZL3KQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.215349820 (300h58m34.677442303s) [noise=true , address=77.90.31.18:4805 , piece_id=LJKUWHGMINU2P7JTS2QN5VP25HPZBKRMFXRDBSO5OTVYTS47SB4A, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.216239644 (300h58m34.678332127s) [noise=true , address=45.91.116.5:4298 , piece_id=VTE4WIBLUWV74XMSSPP37CEAWPRQHA3GFV2RJRQYGRQM5JGF3H3Q, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.218583898 (300h58m34.680676392s) [noise=true , address=45.91.116.10:4082 , piece_id=NX5W46AGWBSZ7EM2LZHZDPE6MEPSLZMNVFZEE2LQ5F5LV7YSQRKQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.219326442 (300h58m34.681418925s) [noise=true , address=45.91.116.6:4974 , piece_id=DBXKLTPEBNF5GVB4PC26BY4EPXOWHU2OWSLWGSHS52S6CDUZIMOA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.222198982 (300h58m34.684291465s) [noise=true , address=145.79.227.17:4972 , piece_id=I54FS5Q4CMMM46S6EP3C5MEUAJCVNWX23OHTBMNQNKP4TRERVZ7A, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.222520114 (300h58m34.684612597s) [noise=true , address=145.79.227.4:4185 , piece_id=ONU6YDQFAMJPRG3ZF45M7KUX5EGXZVQEGCTOJK3VGE2JXXLHCBAQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.225940076 (300h58m34.688032569s) [noise=true , address=77.90.31.13:4628 , piece_id=52Q5TEMXLZHLWJ5QWRFHGYWW6RCL2Y57AY6K2J7I3BUA22BNARQQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.227928832 (300h58m34.690021305s) [noise=true , address=145.79.227.16:4910 , piece_id=T254A3S7TTJEBOIBDDCS34DU7HV25TXDTNZWF2OGE4KRQ7YEA5ZA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.227998645 (300h58m34.690091128s) [noise=true , address=45.91.116.14:4190 , piece_id=OS3T2GGABRXYK6HNEXK4C5F5HTJFDEP7SHA5QV2R3SCMXDE2TD5A, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.231020309 (300h58m34.693112792s) [noise=true , address=145.79.227.18:4007 , piece_id=6PZE3N3QHDGV5K5HULQE5UJXWWG7RAIACP2OVEZVIOTTCWVNQO6Q, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.232908575 (300h58m34.695001068s) [noise=true , address=45.91.116.10:4081 , piece_id=PRCQUWMDPZTR2CSWMOODBECZOLAVJ4OPOUTGQHCPTWXUDQGY2MBQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.233037220 (300h58m34.695129693s) [noise=true , address=145.79.227.19:4079 , piece_id=2VBVLN2SNA6WUICKMNMNM7URPVUGWPB5SVZ335XZ7HIAUM3FY5VA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.234302058 (300h58m34.696394531s) [noise=true , address=45.91.116.7:4010 , piece_id=S5NHKRILOZ47C4NXH4UXBG34ZF6IP2XHBR5JDDBNCBOZK26NALIA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.235631800 (300h58m34.697724283s) [noise=true , address=77.90.31.15:4698 , piece_id=NOCO6KPTUBBKYCITDRFBKA3H5EJTKMDAO4AFQFFXC4PPH3D46X2A, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:50.988532254 (300h58m36.450624747s) [upload=264873]: Next returned nil segment Oct 6 02:29:50.988555589 (300h58m36.450648082s) [upload=264873]: Waiting for error group managing segments... Oct 6 02:29:51.282838267 (300h58m36.74493075s) [noise=true , address=45.91.116.4:4937 , piece_id=CHJRUVVLWAR7627JOLKL2QHAK7EHW72ZTGE3BTOAAM4ZL5AZL3KQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.318573943 (300h58m36.780666416s) [noise=true , address=77.90.31.18:4805 , piece_id=LJKUWHGMINU2P7JTS2QN5VP25HPZBKRMFXRDBSO5OTVYTS47SB4A, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.322112521 (300h58m36.784205004s) [noise=true , address=77.90.31.15:4698 , piece_id=NOCO6KPTUBBKYCITDRFBKA3H5EJTKMDAO4AFQFFXC4PPH3D46X2A, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.350536624 (300h58m36.812629108s) [noise=true , address=45.91.116.6:4974 , piece_id=DBXKLTPEBNF5GVB4PC26BY4EPXOWHU2OWSLWGSHS52S6CDUZIMOA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.362171679 (300h58m36.824264162s) [noise=true , address=45.91.116.10:4081 , piece_id=PRCQUWMDPZTR2CSWMOODBECZOLAVJ4OPOUTGQHCPTWXUDQGY2MBQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.366398738 (300h58m36.828491221s) [noise=true , address=45.91.116.10:4082 , piece_id=NX5W46AGWBSZ7EM2LZHZDPE6MEPSLZMNVFZEE2LQ5F5LV7YSQRKQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.375437720 (300h58m36.837530203s) [noise=true , address=45.91.116.14:4190 , piece_id=OS3T2GGABRXYK6HNEXK4C5F5HTJFDEP7SHA5QV2R3SCMXDE2TD5A, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.397600181 (300h58m36.859692664s) [noise=true , address=77.90.31.13:4628 , piece_id=52Q5TEMXLZHLWJ5QWRFHGYWW6RCL2Y57AY6K2J7I3BUA22BNARQQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.400132462 (300h58m36.862224945s) [noise=true , address=45.91.116.5:4298 , piece_id=VTE4WIBLUWV74XMSSPP37CEAWPRQHA3GFV2RJRQYGRQM5JGF3H3Q, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.412114018 (300h58m36.874206491s) [noise=true , address=45.91.116.7:4010 , piece_id=S5NHKRILOZ47C4NXH4UXBG34ZF6IP2XHBR5JDDBNCBOZK26NALIA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:30:14.302718308 (300h58m59.764810791s) [noise=true , address=145.79.227.18:4007 , piece_id=6PZE3N3QHDGV5K5HULQE5UJXWWG7RAIACP2OVEZVIOTTCWVNQO6Q, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:30:14.319938798 (300h58m59.782031281s) [noise=true , address=145.79.227.7:4294 , piece_id=TLZVEBMQCCWINXB4AVRKHWKZ7WULCGUC6E5GGDJQLASKWNEHSJPA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:30:16.104270571 (300h59m1.566363044s) [noise=true , address=145.79.227.16:4910 , piece_id=T254A3S7TTJEBOIBDDCS34DU7HV25TXDTNZWF2OGE4KRQ7YEA5ZA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:30:16.104295528 (300h59m1.566388012s) [seg_pos={0 0}, upload=264873]: Segment reached optimal threshold of 13 pieces. Oct 6 02:30:16.104442268 (300h59m1.566534751s) [noise=true , address=145.79.227.4:4185 , piece_id=ONU6YDQFAMJPRG3ZF45M7KUX5EGXZVQEGCTOJK3VGE2JXXLHCBAQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN): protocol: write tcp 188.166.225.170:34966->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:34966->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34966->145.79.227.4:4185: use of closed network connection Oct 6 02:30:16.104456395 (300h59m1.566548878s) [noise=true , address=145.79.227.4:4185 , piece_id=ONU6YDQFAMJPRG3ZF45M7KUX5EGXZVQEGCTOJK3VGE2JXXLHCBAQ, seg_pos={0 0}, upload=264873]: Long tail cancellation due to optimal threshold Oct 6 02:30:16.104598005 (300h59m1.566690478s) [noise=true , address=145.79.227.19:4079 , piece_id=2VBVLN2SNA6WUICKMNMNM7URPVUGWPB5SVZ335XZ7HIAUM3FY5VA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: write tcp 188.166.225.170:57162->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:57162->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57162->145.79.227.19:4079: use of closed network connection Oct 6 02:30:16.104604216 (300h59m1.566696699s) [noise=true , address=145.79.227.19:4079 , piece_id=2VBVLN2SNA6WUICKMNMNM7URPVUGWPB5SVZ335XZ7HIAUM3FY5VA, seg_pos={0 0}, upload=264873]: Long tail cancellation due to optimal threshold Oct 6 02:30:16.104672547 (300h59m1.56676503s) [noise=true , address=145.79.227.17:4972 , piece_id=I54FS5Q4CMMM46S6EP3C5MEUAJCVNWX23OHTBMNQNKP4TRERVZ7A, seg_pos={0 0}, upload=264873]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: write tcp 188.166.225.170:33630->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:33630->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33630->145.79.227.17:4972: use of closed network connection Oct 6 02:30:16.104678057 (300h59m1.56677053s) [noise=true , address=145.79.227.17:4972 , piece_id=I54FS5Q4CMMM46S6EP3C5MEUAJCVNWX23OHTBMNQNKP4TRERVZ7A, seg_pos={0 0}, upload=264873]: Long tail cancellation due to optimal threshold Oct 6 02:30:16.104695541 (300h59m1.566788024s) [seg_pos={0 0}, upload=264873]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 6 02:30:16.104762427 (300h59m1.56685491s) [upload=264873]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 6 02:30:16.104768159 (300h59m1.566860632s) [upload=264873]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 6 02:30:16.463894599 (300h59m1.925987092s) [upload=264873]: Done uploading segments... Oct 6 02:30:16.463937391 (300h59m1.926029874s) [upload=264873]: Upload finished. err: <nil> Duration:* 6.162273791s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 6.114729026s FAIL (2025-10-06 02:39:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, Piece ID: T254A3S7TTJEBOIBDDCS34DU7HV25TXDTNZWF2OGE4KRQ7YEA5ZA) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: OS3T2GGABRXYK6HNEXK4C5F5HTJFDEP7SHA5QV2R3SCMXDE2TD5A) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: S5NHKRILOZ47C4NXH4UXBG34ZF6IP2XHBR5JDDBNCBOZK26NALIA) context deadline exceeded; piecestore: (Node ID: 12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, Piece ID: CHJRUVVLWAR7627JOLKL2QHAK7EHW72ZTGE3BTOAAM4ZL5AZL3KQ) context deadline exceeded; piecestore: (Node ID: 12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, Piece ID: TLZVEBMQCCWINXB4AVRKHWKZ7WULCGUC6E5GGDJQLASKWNEHSJPA) context deadline exceeded; piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: LJKUWHGMINU2P7JTS2QN5VP25HPZBKRMFXRDBSO5OTVYTS47SB4A) context deadline exceeded; piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: VTE4WIBLUWV74XMSSPP37CEAWPRQHA3GFV2RJRQYGRQM5JGF3H3Q) context deadline exceeded; piecestore: (Node ID: 1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, Piece ID: 52Q5TEMXLZHLWJ5QWRFHGYWW6RCL2Y57AY6K2J7I3BUA22BNARQQ) context deadline exceeded ======LOGS====== 914003723 (300h58m34.376096196s) [upload=264873]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 02:29:49.215144679 (300h58m34.677237162s) [seg_pos={0 0}, upload=264873]: Begin upload segment... Oct 6 02:29:49.215212839 (300h58m34.677305312s) [seg_pos={0 0}, upload=264873]: Done begin upload segment. Oct 6 02:29:49.215241294 (300h58m34.677333777s) [noise=true , address=145.79.227.7:4294 , piece_id=TLZVEBMQCCWINXB4AVRKHWKZ7WULCGUC6E5GGDJQLASKWNEHSJPA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.215249739 (300h58m34.677342232s) [noise=true , address=45.91.116.4:4937 , piece_id=CHJRUVVLWAR7627JOLKL2QHAK7EHW72ZTGE3BTOAAM4ZL5AZL3KQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.215349820 (300h58m34.677442303s) [noise=true , address=77.90.31.18:4805 , piece_id=LJKUWHGMINU2P7JTS2QN5VP25HPZBKRMFXRDBSO5OTVYTS47SB4A, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.216239644 (300h58m34.678332127s) [noise=true , address=45.91.116.5:4298 , piece_id=VTE4WIBLUWV74XMSSPP37CEAWPRQHA3GFV2RJRQYGRQM5JGF3H3Q, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.218583898 (300h58m34.680676392s) [noise=true , address=45.91.116.10:4082 , piece_id=NX5W46AGWBSZ7EM2LZHZDPE6MEPSLZMNVFZEE2LQ5F5LV7YSQRKQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.219326442 (300h58m34.681418925s) [noise=true , address=45.91.116.6:4974 , piece_id=DBXKLTPEBNF5GVB4PC26BY4EPXOWHU2OWSLWGSHS52S6CDUZIMOA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.222198982 (300h58m34.684291465s) [noise=true , address=145.79.227.17:4972 , piece_id=I54FS5Q4CMMM46S6EP3C5MEUAJCVNWX23OHTBMNQNKP4TRERVZ7A, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.222520114 (300h58m34.684612597s) [noise=true , address=145.79.227.4:4185 , piece_id=ONU6YDQFAMJPRG3ZF45M7KUX5EGXZVQEGCTOJK3VGE2JXXLHCBAQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.225940076 (300h58m34.688032569s) [noise=true , address=77.90.31.13:4628 , piece_id=52Q5TEMXLZHLWJ5QWRFHGYWW6RCL2Y57AY6K2J7I3BUA22BNARQQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.227928832 (300h58m34.690021305s) [noise=true , address=145.79.227.16:4910 , piece_id=T254A3S7TTJEBOIBDDCS34DU7HV25TXDTNZWF2OGE4KRQ7YEA5ZA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.227998645 (300h58m34.690091128s) [noise=true , address=45.91.116.14:4190 , piece_id=OS3T2GGABRXYK6HNEXK4C5F5HTJFDEP7SHA5QV2R3SCMXDE2TD5A, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.231020309 (300h58m34.693112792s) [noise=true , address=145.79.227.18:4007 , piece_id=6PZE3N3QHDGV5K5HULQE5UJXWWG7RAIACP2OVEZVIOTTCWVNQO6Q, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.232908575 (300h58m34.695001068s) [noise=true , address=45.91.116.10:4081 , piece_id=PRCQUWMDPZTR2CSWMOODBECZOLAVJ4OPOUTGQHCPTWXUDQGY2MBQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.233037220 (300h58m34.695129693s) [noise=true , address=145.79.227.19:4079 , piece_id=2VBVLN2SNA6WUICKMNMNM7URPVUGWPB5SVZ335XZ7HIAUM3FY5VA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.234302058 (300h58m34.696394531s) [noise=true , address=45.91.116.7:4010 , piece_id=S5NHKRILOZ47C4NXH4UXBG34ZF6IP2XHBR5JDDBNCBOZK26NALIA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.235631800 (300h58m34.697724283s) [noise=true , address=77.90.31.15:4698 , piece_id=NOCO6KPTUBBKYCITDRFBKA3H5EJTKMDAO4AFQFFXC4PPH3D46X2A, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:50.988532254 (300h58m36.450624747s) [upload=264873]: Next returned nil segment Oct 6 02:29:50.988555589 (300h58m36.450648082s) [upload=264873]: Waiting for error group managing segments... Oct 6 02:29:51.282838267 (300h58m36.74493075s) [noise=true , address=45.91.116.4:4937 , piece_id=CHJRUVVLWAR7627JOLKL2QHAK7EHW72ZTGE3BTOAAM4ZL5AZL3KQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.318573943 (300h58m36.780666416s) [noise=true , address=77.90.31.18:4805 , piece_id=LJKUWHGMINU2P7JTS2QN5VP25HPZBKRMFXRDBSO5OTVYTS47SB4A, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.322112521 (300h58m36.784205004s) [noise=true , address=77.90.31.15:4698 , piece_id=NOCO6KPTUBBKYCITDRFBKA3H5EJTKMDAO4AFQFFXC4PPH3D46X2A, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.350536624 (300h58m36.812629108s) [noise=true , address=45.91.116.6:4974 , piece_id=DBXKLTPEBNF5GVB4PC26BY4EPXOWHU2OWSLWGSHS52S6CDUZIMOA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.362171679 (300h58m36.824264162s) [noise=true , address=45.91.116.10:4081 , piece_id=PRCQUWMDPZTR2CSWMOODBECZOLAVJ4OPOUTGQHCPTWXUDQGY2MBQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.366398738 (300h58m36.828491221s) [noise=true , address=45.91.116.10:4082 , piece_id=NX5W46AGWBSZ7EM2LZHZDPE6MEPSLZMNVFZEE2LQ5F5LV7YSQRKQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.375437720 (300h58m36.837530203s) [noise=true , address=45.91.116.14:4190 , piece_id=OS3T2GGABRXYK6HNEXK4C5F5HTJFDEP7SHA5QV2R3SCMXDE2TD5A, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.397600181 (300h58m36.859692664s) [noise=true , address=77.90.31.13:4628 , piece_id=52Q5TEMXLZHLWJ5QWRFHGYWW6RCL2Y57AY6K2J7I3BUA22BNARQQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.400132462 (300h58m36.862224945s) [noise=true , address=45.91.116.5:4298 , piece_id=VTE4WIBLUWV74XMSSPP37CEAWPRQHA3GFV2RJRQYGRQM5JGF3H3Q, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.412114018 (300h58m36.874206491s) [noise=true , address=45.91.116.7:4010 , piece_id=S5NHKRILOZ47C4NXH4UXBG34ZF6IP2XHBR5JDDBNCBOZK26NALIA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:30:14.302718308 (300h58m59.764810791s) [noise=true , address=145.79.227.18:4007 , piece_id=6PZE3N3QHDGV5K5HULQE5UJXWWG7RAIACP2OVEZVIOTTCWVNQO6Q, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:30:14.319938798 (300h58m59.782031281s) [noise=true , address=145.79.227.7:4294 , piece_id=TLZVEBMQCCWINXB4AVRKHWKZ7WULCGUC6E5GGDJQLASKWNEHSJPA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:30:16.104270571 (300h59m1.566363044s) [noise=true , address=145.79.227.16:4910 , piece_id=T254A3S7TTJEBOIBDDCS34DU7HV25TXDTNZWF2OGE4KRQ7YEA5ZA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:30:16.104295528 (300h59m1.566388012s) [seg_pos={0 0}, upload=264873]: Segment reached optimal threshold of 13 pieces. Oct 6 02:30:16.104442268 (300h59m1.566534751s) [noise=true , address=145.79.227.4:4185 , piece_id=ONU6YDQFAMJPRG3ZF45M7KUX5EGXZVQEGCTOJK3VGE2JXXLHCBAQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN): protocol: write tcp 188.166.225.170:34966->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:34966->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34966->145.79.227.4:4185: use of closed network connection Oct 6 02:30:16.104456395 (300h59m1.566548878s) [noise=true , address=145.79.227.4:4185 , piece_id=ONU6YDQFAMJPRG3ZF45M7KUX5EGXZVQEGCTOJK3VGE2JXXLHCBAQ, seg_pos={0 0}, upload=264873]: Long tail cancellation due to optimal threshold Oct 6 02:30:16.104598005 (300h59m1.566690478s) [noise=true , address=145.79.227.19:4079 , piece_id=2VBVLN2SNA6WUICKMNMNM7URPVUGWPB5SVZ335XZ7HIAUM3FY5VA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: write tcp 188.166.225.170:57162->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:57162->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57162->145.79.227.19:4079: use of closed network connection Oct 6 02:30:16.104604216 (300h59m1.566696699s) [noise=true , address=145.79.227.19:4079 , piece_id=2VBVLN2SNA6WUICKMNMNM7URPVUGWPB5SVZ335XZ7HIAUM3FY5VA, seg_pos={0 0}, upload=264873]: Long tail cancellation due to optimal threshold Oct 6 02:30:16.104672547 (300h59m1.56676503s) [noise=true , address=145.79.227.17:4972 , piece_id=I54FS5Q4CMMM46S6EP3C5MEUAJCVNWX23OHTBMNQNKP4TRERVZ7A, seg_pos={0 0}, upload=264873]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: write tcp 188.166.225.170:33630->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:33630->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33630->145.79.227.17:4972: use of closed network connection Oct 6 02:30:16.104678057 (300h59m1.56677053s) [noise=true , address=145.79.227.17:4972 , piece_id=I54FS5Q4CMMM46S6EP3C5MEUAJCVNWX23OHTBMNQNKP4TRERVZ7A, seg_pos={0 0}, upload=264873]: Long tail cancellation due to optimal threshold Oct 6 02:30:16.104695541 (300h59m1.566788024s) [seg_pos={0 0}, upload=264873]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 6 02:30:16.104762427 (300h59m1.56685491s) [upload=264873]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 6 02:30:16.104768159 (300h59m1.566860632s) [upload=264873]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 6 02:30:16.463894599 (300h59m1.925987092s) [upload=264873]: Done uploading segments... Oct 6 02:30:16.463937391 (300h59m1.926029874s) [upload=264873]: Upload finished. err: <nil> Duration:* 6.114769032s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 5.995528354s FAIL (2025-10-06 02:39:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, Piece ID: T254A3S7TTJEBOIBDDCS34DU7HV25TXDTNZWF2OGE4KRQ7YEA5ZA) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: OS3T2GGABRXYK6HNEXK4C5F5HTJFDEP7SHA5QV2R3SCMXDE2TD5A) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: S5NHKRILOZ47C4NXH4UXBG34ZF6IP2XHBR5JDDBNCBOZK26NALIA) context deadline exceeded; piecestore: (Node ID: 12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, Piece ID: CHJRUVVLWAR7627JOLKL2QHAK7EHW72ZTGE3BTOAAM4ZL5AZL3KQ) context deadline exceeded; piecestore: (Node ID: 12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, Piece ID: TLZVEBMQCCWINXB4AVRKHWKZ7WULCGUC6E5GGDJQLASKWNEHSJPA) context deadline exceeded; piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: LJKUWHGMINU2P7JTS2QN5VP25HPZBKRMFXRDBSO5OTVYTS47SB4A) context deadline exceeded; piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: VTE4WIBLUWV74XMSSPP37CEAWPRQHA3GFV2RJRQYGRQM5JGF3H3Q) context deadline exceeded; piecestore: (Node ID: 1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, Piece ID: 52Q5TEMXLZHLWJ5QWRFHGYWW6RCL2Y57AY6K2J7I3BUA22BNARQQ) context deadline exceeded ======LOGS====== 914003723 (300h58m34.376096196s) [upload=264873]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 02:29:49.215144679 (300h58m34.677237162s) [seg_pos={0 0}, upload=264873]: Begin upload segment... Oct 6 02:29:49.215212839 (300h58m34.677305312s) [seg_pos={0 0}, upload=264873]: Done begin upload segment. Oct 6 02:29:49.215241294 (300h58m34.677333777s) [noise=true , address=145.79.227.7:4294 , piece_id=TLZVEBMQCCWINXB4AVRKHWKZ7WULCGUC6E5GGDJQLASKWNEHSJPA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.215249739 (300h58m34.677342232s) [noise=true , address=45.91.116.4:4937 , piece_id=CHJRUVVLWAR7627JOLKL2QHAK7EHW72ZTGE3BTOAAM4ZL5AZL3KQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.215349820 (300h58m34.677442303s) [noise=true , address=77.90.31.18:4805 , piece_id=LJKUWHGMINU2P7JTS2QN5VP25HPZBKRMFXRDBSO5OTVYTS47SB4A, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.216239644 (300h58m34.678332127s) [noise=true , address=45.91.116.5:4298 , piece_id=VTE4WIBLUWV74XMSSPP37CEAWPRQHA3GFV2RJRQYGRQM5JGF3H3Q, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.218583898 (300h58m34.680676392s) [noise=true , address=45.91.116.10:4082 , piece_id=NX5W46AGWBSZ7EM2LZHZDPE6MEPSLZMNVFZEE2LQ5F5LV7YSQRKQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.219326442 (300h58m34.681418925s) [noise=true , address=45.91.116.6:4974 , piece_id=DBXKLTPEBNF5GVB4PC26BY4EPXOWHU2OWSLWGSHS52S6CDUZIMOA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.222198982 (300h58m34.684291465s) [noise=true , address=145.79.227.17:4972 , piece_id=I54FS5Q4CMMM46S6EP3C5MEUAJCVNWX23OHTBMNQNKP4TRERVZ7A, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.222520114 (300h58m34.684612597s) [noise=true , address=145.79.227.4:4185 , piece_id=ONU6YDQFAMJPRG3ZF45M7KUX5EGXZVQEGCTOJK3VGE2JXXLHCBAQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.225940076 (300h58m34.688032569s) [noise=true , address=77.90.31.13:4628 , piece_id=52Q5TEMXLZHLWJ5QWRFHGYWW6RCL2Y57AY6K2J7I3BUA22BNARQQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.227928832 (300h58m34.690021305s) [noise=true , address=145.79.227.16:4910 , piece_id=T254A3S7TTJEBOIBDDCS34DU7HV25TXDTNZWF2OGE4KRQ7YEA5ZA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.227998645 (300h58m34.690091128s) [noise=true , address=45.91.116.14:4190 , piece_id=OS3T2GGABRXYK6HNEXK4C5F5HTJFDEP7SHA5QV2R3SCMXDE2TD5A, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.231020309 (300h58m34.693112792s) [noise=true , address=145.79.227.18:4007 , piece_id=6PZE3N3QHDGV5K5HULQE5UJXWWG7RAIACP2OVEZVIOTTCWVNQO6Q, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.232908575 (300h58m34.695001068s) [noise=true , address=45.91.116.10:4081 , piece_id=PRCQUWMDPZTR2CSWMOODBECZOLAVJ4OPOUTGQHCPTWXUDQGY2MBQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.233037220 (300h58m34.695129693s) [noise=true , address=145.79.227.19:4079 , piece_id=2VBVLN2SNA6WUICKMNMNM7URPVUGWPB5SVZ335XZ7HIAUM3FY5VA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.234302058 (300h58m34.696394531s) [noise=true , address=45.91.116.7:4010 , piece_id=S5NHKRILOZ47C4NXH4UXBG34ZF6IP2XHBR5JDDBNCBOZK26NALIA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.235631800 (300h58m34.697724283s) [noise=true , address=77.90.31.15:4698 , piece_id=NOCO6KPTUBBKYCITDRFBKA3H5EJTKMDAO4AFQFFXC4PPH3D46X2A, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:50.988532254 (300h58m36.450624747s) [upload=264873]: Next returned nil segment Oct 6 02:29:50.988555589 (300h58m36.450648082s) [upload=264873]: Waiting for error group managing segments... Oct 6 02:29:51.282838267 (300h58m36.74493075s) [noise=true , address=45.91.116.4:4937 , piece_id=CHJRUVVLWAR7627JOLKL2QHAK7EHW72ZTGE3BTOAAM4ZL5AZL3KQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.318573943 (300h58m36.780666416s) [noise=true , address=77.90.31.18:4805 , piece_id=LJKUWHGMINU2P7JTS2QN5VP25HPZBKRMFXRDBSO5OTVYTS47SB4A, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.322112521 (300h58m36.784205004s) [noise=true , address=77.90.31.15:4698 , piece_id=NOCO6KPTUBBKYCITDRFBKA3H5EJTKMDAO4AFQFFXC4PPH3D46X2A, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.350536624 (300h58m36.812629108s) [noise=true , address=45.91.116.6:4974 , piece_id=DBXKLTPEBNF5GVB4PC26BY4EPXOWHU2OWSLWGSHS52S6CDUZIMOA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.362171679 (300h58m36.824264162s) [noise=true , address=45.91.116.10:4081 , piece_id=PRCQUWMDPZTR2CSWMOODBECZOLAVJ4OPOUTGQHCPTWXUDQGY2MBQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.366398738 (300h58m36.828491221s) [noise=true , address=45.91.116.10:4082 , piece_id=NX5W46AGWBSZ7EM2LZHZDPE6MEPSLZMNVFZEE2LQ5F5LV7YSQRKQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.375437720 (300h58m36.837530203s) [noise=true , address=45.91.116.14:4190 , piece_id=OS3T2GGABRXYK6HNEXK4C5F5HTJFDEP7SHA5QV2R3SCMXDE2TD5A, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.397600181 (300h58m36.859692664s) [noise=true , address=77.90.31.13:4628 , piece_id=52Q5TEMXLZHLWJ5QWRFHGYWW6RCL2Y57AY6K2J7I3BUA22BNARQQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.400132462 (300h58m36.862224945s) [noise=true , address=45.91.116.5:4298 , piece_id=VTE4WIBLUWV74XMSSPP37CEAWPRQHA3GFV2RJRQYGRQM5JGF3H3Q, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.412114018 (300h58m36.874206491s) [noise=true , address=45.91.116.7:4010 , piece_id=S5NHKRILOZ47C4NXH4UXBG34ZF6IP2XHBR5JDDBNCBOZK26NALIA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:30:14.302718308 (300h58m59.764810791s) [noise=true , address=145.79.227.18:4007 , piece_id=6PZE3N3QHDGV5K5HULQE5UJXWWG7RAIACP2OVEZVIOTTCWVNQO6Q, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:30:14.319938798 (300h58m59.782031281s) [noise=true , address=145.79.227.7:4294 , piece_id=TLZVEBMQCCWINXB4AVRKHWKZ7WULCGUC6E5GGDJQLASKWNEHSJPA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:30:16.104270571 (300h59m1.566363044s) [noise=true , address=145.79.227.16:4910 , piece_id=T254A3S7TTJEBOIBDDCS34DU7HV25TXDTNZWF2OGE4KRQ7YEA5ZA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:30:16.104295528 (300h59m1.566388012s) [seg_pos={0 0}, upload=264873]: Segment reached optimal threshold of 13 pieces. Oct 6 02:30:16.104442268 (300h59m1.566534751s) [noise=true , address=145.79.227.4:4185 , piece_id=ONU6YDQFAMJPRG3ZF45M7KUX5EGXZVQEGCTOJK3VGE2JXXLHCBAQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN): protocol: write tcp 188.166.225.170:34966->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:34966->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34966->145.79.227.4:4185: use of closed network connection Oct 6 02:30:16.104456395 (300h59m1.566548878s) [noise=true , address=145.79.227.4:4185 , piece_id=ONU6YDQFAMJPRG3ZF45M7KUX5EGXZVQEGCTOJK3VGE2JXXLHCBAQ, seg_pos={0 0}, upload=264873]: Long tail cancellation due to optimal threshold Oct 6 02:30:16.104598005 (300h59m1.566690478s) [noise=true , address=145.79.227.19:4079 , piece_id=2VBVLN2SNA6WUICKMNMNM7URPVUGWPB5SVZ335XZ7HIAUM3FY5VA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: write tcp 188.166.225.170:57162->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:57162->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57162->145.79.227.19:4079: use of closed network connection Oct 6 02:30:16.104604216 (300h59m1.566696699s) [noise=true , address=145.79.227.19:4079 , piece_id=2VBVLN2SNA6WUICKMNMNM7URPVUGWPB5SVZ335XZ7HIAUM3FY5VA, seg_pos={0 0}, upload=264873]: Long tail cancellation due to optimal threshold Oct 6 02:30:16.104672547 (300h59m1.56676503s) [noise=true , address=145.79.227.17:4972 , piece_id=I54FS5Q4CMMM46S6EP3C5MEUAJCVNWX23OHTBMNQNKP4TRERVZ7A, seg_pos={0 0}, upload=264873]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: write tcp 188.166.225.170:33630->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:33630->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33630->145.79.227.17:4972: use of closed network connection Oct 6 02:30:16.104678057 (300h59m1.56677053s) [noise=true , address=145.79.227.17:4972 , piece_id=I54FS5Q4CMMM46S6EP3C5MEUAJCVNWX23OHTBMNQNKP4TRERVZ7A, seg_pos={0 0}, upload=264873]: Long tail cancellation due to optimal threshold Oct 6 02:30:16.104695541 (300h59m1.566788024s) [seg_pos={0 0}, upload=264873]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 6 02:30:16.104762427 (300h59m1.56685491s) [upload=264873]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 6 02:30:16.104768159 (300h59m1.566860632s) [upload=264873]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 6 02:30:16.463894599 (300h59m1.925987092s) [upload=264873]: Done uploading segments... Oct 6 02:30:16.463937391 (300h59m1.926029874s) [upload=264873]: Upload finished. err: <nil> Duration:* 5.995569111s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 6.024895583s FAIL (2025-10-06 02:38:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, Piece ID: T254A3S7TTJEBOIBDDCS34DU7HV25TXDTNZWF2OGE4KRQ7YEA5ZA) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: OS3T2GGABRXYK6HNEXK4C5F5HTJFDEP7SHA5QV2R3SCMXDE2TD5A) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: S5NHKRILOZ47C4NXH4UXBG34ZF6IP2XHBR5JDDBNCBOZK26NALIA) context deadline exceeded; piecestore: (Node ID: 12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, Piece ID: CHJRUVVLWAR7627JOLKL2QHAK7EHW72ZTGE3BTOAAM4ZL5AZL3KQ) context deadline exceeded; piecestore: (Node ID: 12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, Piece ID: TLZVEBMQCCWINXB4AVRKHWKZ7WULCGUC6E5GGDJQLASKWNEHSJPA) context deadline exceeded; piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: LJKUWHGMINU2P7JTS2QN5VP25HPZBKRMFXRDBSO5OTVYTS47SB4A) context deadline exceeded; piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: VTE4WIBLUWV74XMSSPP37CEAWPRQHA3GFV2RJRQYGRQM5JGF3H3Q) context deadline exceeded; piecestore: (Node ID: 1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, Piece ID: 52Q5TEMXLZHLWJ5QWRFHGYWW6RCL2Y57AY6K2J7I3BUA22BNARQQ) context deadline exceeded ======LOGS====== 914003723 (300h58m34.376096196s) [upload=264873]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 02:29:49.215144679 (300h58m34.677237162s) [seg_pos={0 0}, upload=264873]: Begin upload segment... Oct 6 02:29:49.215212839 (300h58m34.677305312s) [seg_pos={0 0}, upload=264873]: Done begin upload segment. Oct 6 02:29:49.215241294 (300h58m34.677333777s) [noise=true , address=145.79.227.7:4294 , piece_id=TLZVEBMQCCWINXB4AVRKHWKZ7WULCGUC6E5GGDJQLASKWNEHSJPA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.215249739 (300h58m34.677342232s) [noise=true , address=45.91.116.4:4937 , piece_id=CHJRUVVLWAR7627JOLKL2QHAK7EHW72ZTGE3BTOAAM4ZL5AZL3KQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.215349820 (300h58m34.677442303s) [noise=true , address=77.90.31.18:4805 , piece_id=LJKUWHGMINU2P7JTS2QN5VP25HPZBKRMFXRDBSO5OTVYTS47SB4A, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.216239644 (300h58m34.678332127s) [noise=true , address=45.91.116.5:4298 , piece_id=VTE4WIBLUWV74XMSSPP37CEAWPRQHA3GFV2RJRQYGRQM5JGF3H3Q, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.218583898 (300h58m34.680676392s) [noise=true , address=45.91.116.10:4082 , piece_id=NX5W46AGWBSZ7EM2LZHZDPE6MEPSLZMNVFZEE2LQ5F5LV7YSQRKQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.219326442 (300h58m34.681418925s) [noise=true , address=45.91.116.6:4974 , piece_id=DBXKLTPEBNF5GVB4PC26BY4EPXOWHU2OWSLWGSHS52S6CDUZIMOA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.222198982 (300h58m34.684291465s) [noise=true , address=145.79.227.17:4972 , piece_id=I54FS5Q4CMMM46S6EP3C5MEUAJCVNWX23OHTBMNQNKP4TRERVZ7A, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.222520114 (300h58m34.684612597s) [noise=true , address=145.79.227.4:4185 , piece_id=ONU6YDQFAMJPRG3ZF45M7KUX5EGXZVQEGCTOJK3VGE2JXXLHCBAQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.225940076 (300h58m34.688032569s) [noise=true , address=77.90.31.13:4628 , piece_id=52Q5TEMXLZHLWJ5QWRFHGYWW6RCL2Y57AY6K2J7I3BUA22BNARQQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.227928832 (300h58m34.690021305s) [noise=true , address=145.79.227.16:4910 , piece_id=T254A3S7TTJEBOIBDDCS34DU7HV25TXDTNZWF2OGE4KRQ7YEA5ZA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.227998645 (300h58m34.690091128s) [noise=true , address=45.91.116.14:4190 , piece_id=OS3T2GGABRXYK6HNEXK4C5F5HTJFDEP7SHA5QV2R3SCMXDE2TD5A, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.231020309 (300h58m34.693112792s) [noise=true , address=145.79.227.18:4007 , piece_id=6PZE3N3QHDGV5K5HULQE5UJXWWG7RAIACP2OVEZVIOTTCWVNQO6Q, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.232908575 (300h58m34.695001068s) [noise=true , address=45.91.116.10:4081 , piece_id=PRCQUWMDPZTR2CSWMOODBECZOLAVJ4OPOUTGQHCPTWXUDQGY2MBQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.233037220 (300h58m34.695129693s) [noise=true , address=145.79.227.19:4079 , piece_id=2VBVLN2SNA6WUICKMNMNM7URPVUGWPB5SVZ335XZ7HIAUM3FY5VA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.234302058 (300h58m34.696394531s) [noise=true , address=45.91.116.7:4010 , piece_id=S5NHKRILOZ47C4NXH4UXBG34ZF6IP2XHBR5JDDBNCBOZK26NALIA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.235631800 (300h58m34.697724283s) [noise=true , address=77.90.31.15:4698 , piece_id=NOCO6KPTUBBKYCITDRFBKA3H5EJTKMDAO4AFQFFXC4PPH3D46X2A, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:50.988532254 (300h58m36.450624747s) [upload=264873]: Next returned nil segment Oct 6 02:29:50.988555589 (300h58m36.450648082s) [upload=264873]: Waiting for error group managing segments... Oct 6 02:29:51.282838267 (300h58m36.74493075s) [noise=true , address=45.91.116.4:4937 , piece_id=CHJRUVVLWAR7627JOLKL2QHAK7EHW72ZTGE3BTOAAM4ZL5AZL3KQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.318573943 (300h58m36.780666416s) [noise=true , address=77.90.31.18:4805 , piece_id=LJKUWHGMINU2P7JTS2QN5VP25HPZBKRMFXRDBSO5OTVYTS47SB4A, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.322112521 (300h58m36.784205004s) [noise=true , address=77.90.31.15:4698 , piece_id=NOCO6KPTUBBKYCITDRFBKA3H5EJTKMDAO4AFQFFXC4PPH3D46X2A, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.350536624 (300h58m36.812629108s) [noise=true , address=45.91.116.6:4974 , piece_id=DBXKLTPEBNF5GVB4PC26BY4EPXOWHU2OWSLWGSHS52S6CDUZIMOA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.362171679 (300h58m36.824264162s) [noise=true , address=45.91.116.10:4081 , piece_id=PRCQUWMDPZTR2CSWMOODBECZOLAVJ4OPOUTGQHCPTWXUDQGY2MBQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.366398738 (300h58m36.828491221s) [noise=true , address=45.91.116.10:4082 , piece_id=NX5W46AGWBSZ7EM2LZHZDPE6MEPSLZMNVFZEE2LQ5F5LV7YSQRKQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.375437720 (300h58m36.837530203s) [noise=true , address=45.91.116.14:4190 , piece_id=OS3T2GGABRXYK6HNEXK4C5F5HTJFDEP7SHA5QV2R3SCMXDE2TD5A, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.397600181 (300h58m36.859692664s) [noise=true , address=77.90.31.13:4628 , piece_id=52Q5TEMXLZHLWJ5QWRFHGYWW6RCL2Y57AY6K2J7I3BUA22BNARQQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.400132462 (300h58m36.862224945s) [noise=true , address=45.91.116.5:4298 , piece_id=VTE4WIBLUWV74XMSSPP37CEAWPRQHA3GFV2RJRQYGRQM5JGF3H3Q, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.412114018 (300h58m36.874206491s) [noise=true , address=45.91.116.7:4010 , piece_id=S5NHKRILOZ47C4NXH4UXBG34ZF6IP2XHBR5JDDBNCBOZK26NALIA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:30:14.302718308 (300h58m59.764810791s) [noise=true , address=145.79.227.18:4007 , piece_id=6PZE3N3QHDGV5K5HULQE5UJXWWG7RAIACP2OVEZVIOTTCWVNQO6Q, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:30:14.319938798 (300h58m59.782031281s) [noise=true , address=145.79.227.7:4294 , piece_id=TLZVEBMQCCWINXB4AVRKHWKZ7WULCGUC6E5GGDJQLASKWNEHSJPA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:30:16.104270571 (300h59m1.566363044s) [noise=true , address=145.79.227.16:4910 , piece_id=T254A3S7TTJEBOIBDDCS34DU7HV25TXDTNZWF2OGE4KRQ7YEA5ZA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:30:16.104295528 (300h59m1.566388012s) [seg_pos={0 0}, upload=264873]: Segment reached optimal threshold of 13 pieces. Oct 6 02:30:16.104442268 (300h59m1.566534751s) [noise=true , address=145.79.227.4:4185 , piece_id=ONU6YDQFAMJPRG3ZF45M7KUX5EGXZVQEGCTOJK3VGE2JXXLHCBAQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN): protocol: write tcp 188.166.225.170:34966->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:34966->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34966->145.79.227.4:4185: use of closed network connection Oct 6 02:30:16.104456395 (300h59m1.566548878s) [noise=true , address=145.79.227.4:4185 , piece_id=ONU6YDQFAMJPRG3ZF45M7KUX5EGXZVQEGCTOJK3VGE2JXXLHCBAQ, seg_pos={0 0}, upload=264873]: Long tail cancellation due to optimal threshold Oct 6 02:30:16.104598005 (300h59m1.566690478s) [noise=true , address=145.79.227.19:4079 , piece_id=2VBVLN2SNA6WUICKMNMNM7URPVUGWPB5SVZ335XZ7HIAUM3FY5VA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: write tcp 188.166.225.170:57162->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:57162->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57162->145.79.227.19:4079: use of closed network connection Oct 6 02:30:16.104604216 (300h59m1.566696699s) [noise=true , address=145.79.227.19:4079 , piece_id=2VBVLN2SNA6WUICKMNMNM7URPVUGWPB5SVZ335XZ7HIAUM3FY5VA, seg_pos={0 0}, upload=264873]: Long tail cancellation due to optimal threshold Oct 6 02:30:16.104672547 (300h59m1.56676503s) [noise=true , address=145.79.227.17:4972 , piece_id=I54FS5Q4CMMM46S6EP3C5MEUAJCVNWX23OHTBMNQNKP4TRERVZ7A, seg_pos={0 0}, upload=264873]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: write tcp 188.166.225.170:33630->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:33630->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33630->145.79.227.17:4972: use of closed network connection Oct 6 02:30:16.104678057 (300h59m1.56677053s) [noise=true , address=145.79.227.17:4972 , piece_id=I54FS5Q4CMMM46S6EP3C5MEUAJCVNWX23OHTBMNQNKP4TRERVZ7A, seg_pos={0 0}, upload=264873]: Long tail cancellation due to optimal threshold Oct 6 02:30:16.104695541 (300h59m1.566788024s) [seg_pos={0 0}, upload=264873]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 6 02:30:16.104762427 (300h59m1.56685491s) [upload=264873]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 6 02:30:16.104768159 (300h59m1.566860632s) [upload=264873]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 6 02:30:16.463894599 (300h59m1.925987092s) [upload=264873]: Done uploading segments... Oct 6 02:30:16.463937391 (300h59m1.926029874s) [upload=264873]: Upload finished. err: <nil> Duration:* 6.024930109s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 6.069057554s FAIL (2025-10-06 02:37:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, Piece ID: T254A3S7TTJEBOIBDDCS34DU7HV25TXDTNZWF2OGE4KRQ7YEA5ZA) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: OS3T2GGABRXYK6HNEXK4C5F5HTJFDEP7SHA5QV2R3SCMXDE2TD5A) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: S5NHKRILOZ47C4NXH4UXBG34ZF6IP2XHBR5JDDBNCBOZK26NALIA) context deadline exceeded; piecestore: (Node ID: 12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, Piece ID: CHJRUVVLWAR7627JOLKL2QHAK7EHW72ZTGE3BTOAAM4ZL5AZL3KQ) context deadline exceeded; piecestore: (Node ID: 12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, Piece ID: TLZVEBMQCCWINXB4AVRKHWKZ7WULCGUC6E5GGDJQLASKWNEHSJPA) context deadline exceeded; piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: LJKUWHGMINU2P7JTS2QN5VP25HPZBKRMFXRDBSO5OTVYTS47SB4A) context deadline exceeded; piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: VTE4WIBLUWV74XMSSPP37CEAWPRQHA3GFV2RJRQYGRQM5JGF3H3Q) context deadline exceeded; piecestore: (Node ID: 1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, Piece ID: 52Q5TEMXLZHLWJ5QWRFHGYWW6RCL2Y57AY6K2J7I3BUA22BNARQQ) context deadline exceeded ======LOGS====== 914003723 (300h58m34.376096196s) [upload=264873]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 02:29:49.215144679 (300h58m34.677237162s) [seg_pos={0 0}, upload=264873]: Begin upload segment... Oct 6 02:29:49.215212839 (300h58m34.677305312s) [seg_pos={0 0}, upload=264873]: Done begin upload segment. Oct 6 02:29:49.215241294 (300h58m34.677333777s) [noise=true , address=145.79.227.7:4294 , piece_id=TLZVEBMQCCWINXB4AVRKHWKZ7WULCGUC6E5GGDJQLASKWNEHSJPA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.215249739 (300h58m34.677342232s) [noise=true , address=45.91.116.4:4937 , piece_id=CHJRUVVLWAR7627JOLKL2QHAK7EHW72ZTGE3BTOAAM4ZL5AZL3KQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.215349820 (300h58m34.677442303s) [noise=true , address=77.90.31.18:4805 , piece_id=LJKUWHGMINU2P7JTS2QN5VP25HPZBKRMFXRDBSO5OTVYTS47SB4A, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.216239644 (300h58m34.678332127s) [noise=true , address=45.91.116.5:4298 , piece_id=VTE4WIBLUWV74XMSSPP37CEAWPRQHA3GFV2RJRQYGRQM5JGF3H3Q, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.218583898 (300h58m34.680676392s) [noise=true , address=45.91.116.10:4082 , piece_id=NX5W46AGWBSZ7EM2LZHZDPE6MEPSLZMNVFZEE2LQ5F5LV7YSQRKQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.219326442 (300h58m34.681418925s) [noise=true , address=45.91.116.6:4974 , piece_id=DBXKLTPEBNF5GVB4PC26BY4EPXOWHU2OWSLWGSHS52S6CDUZIMOA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.222198982 (300h58m34.684291465s) [noise=true , address=145.79.227.17:4972 , piece_id=I54FS5Q4CMMM46S6EP3C5MEUAJCVNWX23OHTBMNQNKP4TRERVZ7A, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.222520114 (300h58m34.684612597s) [noise=true , address=145.79.227.4:4185 , piece_id=ONU6YDQFAMJPRG3ZF45M7KUX5EGXZVQEGCTOJK3VGE2JXXLHCBAQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.225940076 (300h58m34.688032569s) [noise=true , address=77.90.31.13:4628 , piece_id=52Q5TEMXLZHLWJ5QWRFHGYWW6RCL2Y57AY6K2J7I3BUA22BNARQQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.227928832 (300h58m34.690021305s) [noise=true , address=145.79.227.16:4910 , piece_id=T254A3S7TTJEBOIBDDCS34DU7HV25TXDTNZWF2OGE4KRQ7YEA5ZA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.227998645 (300h58m34.690091128s) [noise=true , address=45.91.116.14:4190 , piece_id=OS3T2GGABRXYK6HNEXK4C5F5HTJFDEP7SHA5QV2R3SCMXDE2TD5A, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.231020309 (300h58m34.693112792s) [noise=true , address=145.79.227.18:4007 , piece_id=6PZE3N3QHDGV5K5HULQE5UJXWWG7RAIACP2OVEZVIOTTCWVNQO6Q, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.232908575 (300h58m34.695001068s) [noise=true , address=45.91.116.10:4081 , piece_id=PRCQUWMDPZTR2CSWMOODBECZOLAVJ4OPOUTGQHCPTWXUDQGY2MBQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.233037220 (300h58m34.695129693s) [noise=true , address=145.79.227.19:4079 , piece_id=2VBVLN2SNA6WUICKMNMNM7URPVUGWPB5SVZ335XZ7HIAUM3FY5VA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.234302058 (300h58m34.696394531s) [noise=true , address=45.91.116.7:4010 , piece_id=S5NHKRILOZ47C4NXH4UXBG34ZF6IP2XHBR5JDDBNCBOZK26NALIA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.235631800 (300h58m34.697724283s) [noise=true , address=77.90.31.15:4698 , piece_id=NOCO6KPTUBBKYCITDRFBKA3H5EJTKMDAO4AFQFFXC4PPH3D46X2A, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:50.988532254 (300h58m36.450624747s) [upload=264873]: Next returned nil segment Oct 6 02:29:50.988555589 (300h58m36.450648082s) [upload=264873]: Waiting for error group managing segments... Oct 6 02:29:51.282838267 (300h58m36.74493075s) [noise=true , address=45.91.116.4:4937 , piece_id=CHJRUVVLWAR7627JOLKL2QHAK7EHW72ZTGE3BTOAAM4ZL5AZL3KQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.318573943 (300h58m36.780666416s) [noise=true , address=77.90.31.18:4805 , piece_id=LJKUWHGMINU2P7JTS2QN5VP25HPZBKRMFXRDBSO5OTVYTS47SB4A, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.322112521 (300h58m36.784205004s) [noise=true , address=77.90.31.15:4698 , piece_id=NOCO6KPTUBBKYCITDRFBKA3H5EJTKMDAO4AFQFFXC4PPH3D46X2A, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.350536624 (300h58m36.812629108s) [noise=true , address=45.91.116.6:4974 , piece_id=DBXKLTPEBNF5GVB4PC26BY4EPXOWHU2OWSLWGSHS52S6CDUZIMOA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.362171679 (300h58m36.824264162s) [noise=true , address=45.91.116.10:4081 , piece_id=PRCQUWMDPZTR2CSWMOODBECZOLAVJ4OPOUTGQHCPTWXUDQGY2MBQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.366398738 (300h58m36.828491221s) [noise=true , address=45.91.116.10:4082 , piece_id=NX5W46AGWBSZ7EM2LZHZDPE6MEPSLZMNVFZEE2LQ5F5LV7YSQRKQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.375437720 (300h58m36.837530203s) [noise=true , address=45.91.116.14:4190 , piece_id=OS3T2GGABRXYK6HNEXK4C5F5HTJFDEP7SHA5QV2R3SCMXDE2TD5A, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.397600181 (300h58m36.859692664s) [noise=true , address=77.90.31.13:4628 , piece_id=52Q5TEMXLZHLWJ5QWRFHGYWW6RCL2Y57AY6K2J7I3BUA22BNARQQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.400132462 (300h58m36.862224945s) [noise=true , address=45.91.116.5:4298 , piece_id=VTE4WIBLUWV74XMSSPP37CEAWPRQHA3GFV2RJRQYGRQM5JGF3H3Q, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.412114018 (300h58m36.874206491s) [noise=true , address=45.91.116.7:4010 , piece_id=S5NHKRILOZ47C4NXH4UXBG34ZF6IP2XHBR5JDDBNCBOZK26NALIA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:30:14.302718308 (300h58m59.764810791s) [noise=true , address=145.79.227.18:4007 , piece_id=6PZE3N3QHDGV5K5HULQE5UJXWWG7RAIACP2OVEZVIOTTCWVNQO6Q, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:30:14.319938798 (300h58m59.782031281s) [noise=true , address=145.79.227.7:4294 , piece_id=TLZVEBMQCCWINXB4AVRKHWKZ7WULCGUC6E5GGDJQLASKWNEHSJPA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:30:16.104270571 (300h59m1.566363044s) [noise=true , address=145.79.227.16:4910 , piece_id=T254A3S7TTJEBOIBDDCS34DU7HV25TXDTNZWF2OGE4KRQ7YEA5ZA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:30:16.104295528 (300h59m1.566388012s) [seg_pos={0 0}, upload=264873]: Segment reached optimal threshold of 13 pieces. Oct 6 02:30:16.104442268 (300h59m1.566534751s) [noise=true , address=145.79.227.4:4185 , piece_id=ONU6YDQFAMJPRG3ZF45M7KUX5EGXZVQEGCTOJK3VGE2JXXLHCBAQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN): protocol: write tcp 188.166.225.170:34966->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:34966->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34966->145.79.227.4:4185: use of closed network connection Oct 6 02:30:16.104456395 (300h59m1.566548878s) [noise=true , address=145.79.227.4:4185 , piece_id=ONU6YDQFAMJPRG3ZF45M7KUX5EGXZVQEGCTOJK3VGE2JXXLHCBAQ, seg_pos={0 0}, upload=264873]: Long tail cancellation due to optimal threshold Oct 6 02:30:16.104598005 (300h59m1.566690478s) [noise=true , address=145.79.227.19:4079 , piece_id=2VBVLN2SNA6WUICKMNMNM7URPVUGWPB5SVZ335XZ7HIAUM3FY5VA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: write tcp 188.166.225.170:57162->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:57162->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57162->145.79.227.19:4079: use of closed network connection Oct 6 02:30:16.104604216 (300h59m1.566696699s) [noise=true , address=145.79.227.19:4079 , piece_id=2VBVLN2SNA6WUICKMNMNM7URPVUGWPB5SVZ335XZ7HIAUM3FY5VA, seg_pos={0 0}, upload=264873]: Long tail cancellation due to optimal threshold Oct 6 02:30:16.104672547 (300h59m1.56676503s) [noise=true , address=145.79.227.17:4972 , piece_id=I54FS5Q4CMMM46S6EP3C5MEUAJCVNWX23OHTBMNQNKP4TRERVZ7A, seg_pos={0 0}, upload=264873]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: write tcp 188.166.225.170:33630->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:33630->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33630->145.79.227.17:4972: use of closed network connection Oct 6 02:30:16.104678057 (300h59m1.56677053s) [noise=true , address=145.79.227.17:4972 , piece_id=I54FS5Q4CMMM46S6EP3C5MEUAJCVNWX23OHTBMNQNKP4TRERVZ7A, seg_pos={0 0}, upload=264873]: Long tail cancellation due to optimal threshold Oct 6 02:30:16.104695541 (300h59m1.566788024s) [seg_pos={0 0}, upload=264873]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 6 02:30:16.104762427 (300h59m1.56685491s) [upload=264873]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 6 02:30:16.104768159 (300h59m1.566860632s) [upload=264873]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 6 02:30:16.463894599 (300h59m1.925987092s) [upload=264873]: Done uploading segments... Oct 6 02:30:16.463937391 (300h59m1.926029874s) [upload=264873]: Upload finished. err: <nil> Duration:* 6.069090907s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 6.23277357s FAIL (2025-10-06 02:37:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, Piece ID: T254A3S7TTJEBOIBDDCS34DU7HV25TXDTNZWF2OGE4KRQ7YEA5ZA) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: OS3T2GGABRXYK6HNEXK4C5F5HTJFDEP7SHA5QV2R3SCMXDE2TD5A) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: S5NHKRILOZ47C4NXH4UXBG34ZF6IP2XHBR5JDDBNCBOZK26NALIA) context deadline exceeded; piecestore: (Node ID: 12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, Piece ID: CHJRUVVLWAR7627JOLKL2QHAK7EHW72ZTGE3BTOAAM4ZL5AZL3KQ) context deadline exceeded; piecestore: (Node ID: 12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, Piece ID: TLZVEBMQCCWINXB4AVRKHWKZ7WULCGUC6E5GGDJQLASKWNEHSJPA) context deadline exceeded; piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: LJKUWHGMINU2P7JTS2QN5VP25HPZBKRMFXRDBSO5OTVYTS47SB4A) context deadline exceeded; piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: VTE4WIBLUWV74XMSSPP37CEAWPRQHA3GFV2RJRQYGRQM5JGF3H3Q) context deadline exceeded; piecestore: (Node ID: 1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, Piece ID: 52Q5TEMXLZHLWJ5QWRFHGYWW6RCL2Y57AY6K2J7I3BUA22BNARQQ) context deadline exceeded ======LOGS====== 914003723 (300h58m34.376096196s) [upload=264873]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 02:29:49.215144679 (300h58m34.677237162s) [seg_pos={0 0}, upload=264873]: Begin upload segment... Oct 6 02:29:49.215212839 (300h58m34.677305312s) [seg_pos={0 0}, upload=264873]: Done begin upload segment. Oct 6 02:29:49.215241294 (300h58m34.677333777s) [noise=true , address=145.79.227.7:4294 , piece_id=TLZVEBMQCCWINXB4AVRKHWKZ7WULCGUC6E5GGDJQLASKWNEHSJPA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.215249739 (300h58m34.677342232s) [noise=true , address=45.91.116.4:4937 , piece_id=CHJRUVVLWAR7627JOLKL2QHAK7EHW72ZTGE3BTOAAM4ZL5AZL3KQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.215349820 (300h58m34.677442303s) [noise=true , address=77.90.31.18:4805 , piece_id=LJKUWHGMINU2P7JTS2QN5VP25HPZBKRMFXRDBSO5OTVYTS47SB4A, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.216239644 (300h58m34.678332127s) [noise=true , address=45.91.116.5:4298 , piece_id=VTE4WIBLUWV74XMSSPP37CEAWPRQHA3GFV2RJRQYGRQM5JGF3H3Q, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.218583898 (300h58m34.680676392s) [noise=true , address=45.91.116.10:4082 , piece_id=NX5W46AGWBSZ7EM2LZHZDPE6MEPSLZMNVFZEE2LQ5F5LV7YSQRKQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.219326442 (300h58m34.681418925s) [noise=true , address=45.91.116.6:4974 , piece_id=DBXKLTPEBNF5GVB4PC26BY4EPXOWHU2OWSLWGSHS52S6CDUZIMOA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.222198982 (300h58m34.684291465s) [noise=true , address=145.79.227.17:4972 , piece_id=I54FS5Q4CMMM46S6EP3C5MEUAJCVNWX23OHTBMNQNKP4TRERVZ7A, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.222520114 (300h58m34.684612597s) [noise=true , address=145.79.227.4:4185 , piece_id=ONU6YDQFAMJPRG3ZF45M7KUX5EGXZVQEGCTOJK3VGE2JXXLHCBAQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.225940076 (300h58m34.688032569s) [noise=true , address=77.90.31.13:4628 , piece_id=52Q5TEMXLZHLWJ5QWRFHGYWW6RCL2Y57AY6K2J7I3BUA22BNARQQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.227928832 (300h58m34.690021305s) [noise=true , address=145.79.227.16:4910 , piece_id=T254A3S7TTJEBOIBDDCS34DU7HV25TXDTNZWF2OGE4KRQ7YEA5ZA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.227998645 (300h58m34.690091128s) [noise=true , address=45.91.116.14:4190 , piece_id=OS3T2GGABRXYK6HNEXK4C5F5HTJFDEP7SHA5QV2R3SCMXDE2TD5A, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.231020309 (300h58m34.693112792s) [noise=true , address=145.79.227.18:4007 , piece_id=6PZE3N3QHDGV5K5HULQE5UJXWWG7RAIACP2OVEZVIOTTCWVNQO6Q, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.232908575 (300h58m34.695001068s) [noise=true , address=45.91.116.10:4081 , piece_id=PRCQUWMDPZTR2CSWMOODBECZOLAVJ4OPOUTGQHCPTWXUDQGY2MBQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.233037220 (300h58m34.695129693s) [noise=true , address=145.79.227.19:4079 , piece_id=2VBVLN2SNA6WUICKMNMNM7URPVUGWPB5SVZ335XZ7HIAUM3FY5VA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.234302058 (300h58m34.696394531s) [noise=true , address=45.91.116.7:4010 , piece_id=S5NHKRILOZ47C4NXH4UXBG34ZF6IP2XHBR5JDDBNCBOZK26NALIA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.235631800 (300h58m34.697724283s) [noise=true , address=77.90.31.15:4698 , piece_id=NOCO6KPTUBBKYCITDRFBKA3H5EJTKMDAO4AFQFFXC4PPH3D46X2A, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:50.988532254 (300h58m36.450624747s) [upload=264873]: Next returned nil segment Oct 6 02:29:50.988555589 (300h58m36.450648082s) [upload=264873]: Waiting for error group managing segments... Oct 6 02:29:51.282838267 (300h58m36.74493075s) [noise=true , address=45.91.116.4:4937 , piece_id=CHJRUVVLWAR7627JOLKL2QHAK7EHW72ZTGE3BTOAAM4ZL5AZL3KQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.318573943 (300h58m36.780666416s) [noise=true , address=77.90.31.18:4805 , piece_id=LJKUWHGMINU2P7JTS2QN5VP25HPZBKRMFXRDBSO5OTVYTS47SB4A, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.322112521 (300h58m36.784205004s) [noise=true , address=77.90.31.15:4698 , piece_id=NOCO6KPTUBBKYCITDRFBKA3H5EJTKMDAO4AFQFFXC4PPH3D46X2A, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.350536624 (300h58m36.812629108s) [noise=true , address=45.91.116.6:4974 , piece_id=DBXKLTPEBNF5GVB4PC26BY4EPXOWHU2OWSLWGSHS52S6CDUZIMOA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.362171679 (300h58m36.824264162s) [noise=true , address=45.91.116.10:4081 , piece_id=PRCQUWMDPZTR2CSWMOODBECZOLAVJ4OPOUTGQHCPTWXUDQGY2MBQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.366398738 (300h58m36.828491221s) [noise=true , address=45.91.116.10:4082 , piece_id=NX5W46AGWBSZ7EM2LZHZDPE6MEPSLZMNVFZEE2LQ5F5LV7YSQRKQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.375437720 (300h58m36.837530203s) [noise=true , address=45.91.116.14:4190 , piece_id=OS3T2GGABRXYK6HNEXK4C5F5HTJFDEP7SHA5QV2R3SCMXDE2TD5A, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.397600181 (300h58m36.859692664s) [noise=true , address=77.90.31.13:4628 , piece_id=52Q5TEMXLZHLWJ5QWRFHGYWW6RCL2Y57AY6K2J7I3BUA22BNARQQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.400132462 (300h58m36.862224945s) [noise=true , address=45.91.116.5:4298 , piece_id=VTE4WIBLUWV74XMSSPP37CEAWPRQHA3GFV2RJRQYGRQM5JGF3H3Q, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.412114018 (300h58m36.874206491s) [noise=true , address=45.91.116.7:4010 , piece_id=S5NHKRILOZ47C4NXH4UXBG34ZF6IP2XHBR5JDDBNCBOZK26NALIA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:30:14.302718308 (300h58m59.764810791s) [noise=true , address=145.79.227.18:4007 , piece_id=6PZE3N3QHDGV5K5HULQE5UJXWWG7RAIACP2OVEZVIOTTCWVNQO6Q, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:30:14.319938798 (300h58m59.782031281s) [noise=true , address=145.79.227.7:4294 , piece_id=TLZVEBMQCCWINXB4AVRKHWKZ7WULCGUC6E5GGDJQLASKWNEHSJPA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:30:16.104270571 (300h59m1.566363044s) [noise=true , address=145.79.227.16:4910 , piece_id=T254A3S7TTJEBOIBDDCS34DU7HV25TXDTNZWF2OGE4KRQ7YEA5ZA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:30:16.104295528 (300h59m1.566388012s) [seg_pos={0 0}, upload=264873]: Segment reached optimal threshold of 13 pieces. Oct 6 02:30:16.104442268 (300h59m1.566534751s) [noise=true , address=145.79.227.4:4185 , piece_id=ONU6YDQFAMJPRG3ZF45M7KUX5EGXZVQEGCTOJK3VGE2JXXLHCBAQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN): protocol: write tcp 188.166.225.170:34966->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:34966->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34966->145.79.227.4:4185: use of closed network connection Oct 6 02:30:16.104456395 (300h59m1.566548878s) [noise=true , address=145.79.227.4:4185 , piece_id=ONU6YDQFAMJPRG3ZF45M7KUX5EGXZVQEGCTOJK3VGE2JXXLHCBAQ, seg_pos={0 0}, upload=264873]: Long tail cancellation due to optimal threshold Oct 6 02:30:16.104598005 (300h59m1.566690478s) [noise=true , address=145.79.227.19:4079 , piece_id=2VBVLN2SNA6WUICKMNMNM7URPVUGWPB5SVZ335XZ7HIAUM3FY5VA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: write tcp 188.166.225.170:57162->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:57162->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57162->145.79.227.19:4079: use of closed network connection Oct 6 02:30:16.104604216 (300h59m1.566696699s) [noise=true , address=145.79.227.19:4079 , piece_id=2VBVLN2SNA6WUICKMNMNM7URPVUGWPB5SVZ335XZ7HIAUM3FY5VA, seg_pos={0 0}, upload=264873]: Long tail cancellation due to optimal threshold Oct 6 02:30:16.104672547 (300h59m1.56676503s) [noise=true , address=145.79.227.17:4972 , piece_id=I54FS5Q4CMMM46S6EP3C5MEUAJCVNWX23OHTBMNQNKP4TRERVZ7A, seg_pos={0 0}, upload=264873]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: write tcp 188.166.225.170:33630->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:33630->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33630->145.79.227.17:4972: use of closed network connection Oct 6 02:30:16.104678057 (300h59m1.56677053s) [noise=true , address=145.79.227.17:4972 , piece_id=I54FS5Q4CMMM46S6EP3C5MEUAJCVNWX23OHTBMNQNKP4TRERVZ7A, seg_pos={0 0}, upload=264873]: Long tail cancellation due to optimal threshold Oct 6 02:30:16.104695541 (300h59m1.566788024s) [seg_pos={0 0}, upload=264873]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 6 02:30:16.104762427 (300h59m1.56685491s) [upload=264873]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 6 02:30:16.104768159 (300h59m1.566860632s) [upload=264873]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 6 02:30:16.463894599 (300h59m1.925987092s) [upload=264873]: Done uploading segments... Oct 6 02:30:16.463937391 (300h59m1.926029874s) [upload=264873]: Upload finished. err: <nil> Duration:* 6.232804809s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 6.04553687s FAIL (2025-10-06 02:35:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, Piece ID: T254A3S7TTJEBOIBDDCS34DU7HV25TXDTNZWF2OGE4KRQ7YEA5ZA) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: OS3T2GGABRXYK6HNEXK4C5F5HTJFDEP7SHA5QV2R3SCMXDE2TD5A) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: S5NHKRILOZ47C4NXH4UXBG34ZF6IP2XHBR5JDDBNCBOZK26NALIA) context deadline exceeded; piecestore: (Node ID: 12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, Piece ID: CHJRUVVLWAR7627JOLKL2QHAK7EHW72ZTGE3BTOAAM4ZL5AZL3KQ) context deadline exceeded; piecestore: (Node ID: 12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, Piece ID: TLZVEBMQCCWINXB4AVRKHWKZ7WULCGUC6E5GGDJQLASKWNEHSJPA) context deadline exceeded; piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: LJKUWHGMINU2P7JTS2QN5VP25HPZBKRMFXRDBSO5OTVYTS47SB4A) context deadline exceeded; piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: VTE4WIBLUWV74XMSSPP37CEAWPRQHA3GFV2RJRQYGRQM5JGF3H3Q) context deadline exceeded; piecestore: (Node ID: 1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, Piece ID: 52Q5TEMXLZHLWJ5QWRFHGYWW6RCL2Y57AY6K2J7I3BUA22BNARQQ) context deadline exceeded ======LOGS====== 914003723 (300h58m34.376096196s) [upload=264873]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 02:29:49.215144679 (300h58m34.677237162s) [seg_pos={0 0}, upload=264873]: Begin upload segment... Oct 6 02:29:49.215212839 (300h58m34.677305312s) [seg_pos={0 0}, upload=264873]: Done begin upload segment. Oct 6 02:29:49.215241294 (300h58m34.677333777s) [noise=true , address=145.79.227.7:4294 , piece_id=TLZVEBMQCCWINXB4AVRKHWKZ7WULCGUC6E5GGDJQLASKWNEHSJPA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.215249739 (300h58m34.677342232s) [noise=true , address=45.91.116.4:4937 , piece_id=CHJRUVVLWAR7627JOLKL2QHAK7EHW72ZTGE3BTOAAM4ZL5AZL3KQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.215349820 (300h58m34.677442303s) [noise=true , address=77.90.31.18:4805 , piece_id=LJKUWHGMINU2P7JTS2QN5VP25HPZBKRMFXRDBSO5OTVYTS47SB4A, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.216239644 (300h58m34.678332127s) [noise=true , address=45.91.116.5:4298 , piece_id=VTE4WIBLUWV74XMSSPP37CEAWPRQHA3GFV2RJRQYGRQM5JGF3H3Q, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.218583898 (300h58m34.680676392s) [noise=true , address=45.91.116.10:4082 , piece_id=NX5W46AGWBSZ7EM2LZHZDPE6MEPSLZMNVFZEE2LQ5F5LV7YSQRKQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.219326442 (300h58m34.681418925s) [noise=true , address=45.91.116.6:4974 , piece_id=DBXKLTPEBNF5GVB4PC26BY4EPXOWHU2OWSLWGSHS52S6CDUZIMOA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.222198982 (300h58m34.684291465s) [noise=true , address=145.79.227.17:4972 , piece_id=I54FS5Q4CMMM46S6EP3C5MEUAJCVNWX23OHTBMNQNKP4TRERVZ7A, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.222520114 (300h58m34.684612597s) [noise=true , address=145.79.227.4:4185 , piece_id=ONU6YDQFAMJPRG3ZF45M7KUX5EGXZVQEGCTOJK3VGE2JXXLHCBAQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.225940076 (300h58m34.688032569s) [noise=true , address=77.90.31.13:4628 , piece_id=52Q5TEMXLZHLWJ5QWRFHGYWW6RCL2Y57AY6K2J7I3BUA22BNARQQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.227928832 (300h58m34.690021305s) [noise=true , address=145.79.227.16:4910 , piece_id=T254A3S7TTJEBOIBDDCS34DU7HV25TXDTNZWF2OGE4KRQ7YEA5ZA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.227998645 (300h58m34.690091128s) [noise=true , address=45.91.116.14:4190 , piece_id=OS3T2GGABRXYK6HNEXK4C5F5HTJFDEP7SHA5QV2R3SCMXDE2TD5A, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.231020309 (300h58m34.693112792s) [noise=true , address=145.79.227.18:4007 , piece_id=6PZE3N3QHDGV5K5HULQE5UJXWWG7RAIACP2OVEZVIOTTCWVNQO6Q, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.232908575 (300h58m34.695001068s) [noise=true , address=45.91.116.10:4081 , piece_id=PRCQUWMDPZTR2CSWMOODBECZOLAVJ4OPOUTGQHCPTWXUDQGY2MBQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.233037220 (300h58m34.695129693s) [noise=true , address=145.79.227.19:4079 , piece_id=2VBVLN2SNA6WUICKMNMNM7URPVUGWPB5SVZ335XZ7HIAUM3FY5VA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.234302058 (300h58m34.696394531s) [noise=true , address=45.91.116.7:4010 , piece_id=S5NHKRILOZ47C4NXH4UXBG34ZF6IP2XHBR5JDDBNCBOZK26NALIA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.235631800 (300h58m34.697724283s) [noise=true , address=77.90.31.15:4698 , piece_id=NOCO6KPTUBBKYCITDRFBKA3H5EJTKMDAO4AFQFFXC4PPH3D46X2A, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:50.988532254 (300h58m36.450624747s) [upload=264873]: Next returned nil segment Oct 6 02:29:50.988555589 (300h58m36.450648082s) [upload=264873]: Waiting for error group managing segments... Oct 6 02:29:51.282838267 (300h58m36.74493075s) [noise=true , address=45.91.116.4:4937 , piece_id=CHJRUVVLWAR7627JOLKL2QHAK7EHW72ZTGE3BTOAAM4ZL5AZL3KQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.318573943 (300h58m36.780666416s) [noise=true , address=77.90.31.18:4805 , piece_id=LJKUWHGMINU2P7JTS2QN5VP25HPZBKRMFXRDBSO5OTVYTS47SB4A, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.322112521 (300h58m36.784205004s) [noise=true , address=77.90.31.15:4698 , piece_id=NOCO6KPTUBBKYCITDRFBKA3H5EJTKMDAO4AFQFFXC4PPH3D46X2A, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.350536624 (300h58m36.812629108s) [noise=true , address=45.91.116.6:4974 , piece_id=DBXKLTPEBNF5GVB4PC26BY4EPXOWHU2OWSLWGSHS52S6CDUZIMOA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.362171679 (300h58m36.824264162s) [noise=true , address=45.91.116.10:4081 , piece_id=PRCQUWMDPZTR2CSWMOODBECZOLAVJ4OPOUTGQHCPTWXUDQGY2MBQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.366398738 (300h58m36.828491221s) [noise=true , address=45.91.116.10:4082 , piece_id=NX5W46AGWBSZ7EM2LZHZDPE6MEPSLZMNVFZEE2LQ5F5LV7YSQRKQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.375437720 (300h58m36.837530203s) [noise=true , address=45.91.116.14:4190 , piece_id=OS3T2GGABRXYK6HNEXK4C5F5HTJFDEP7SHA5QV2R3SCMXDE2TD5A, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.397600181 (300h58m36.859692664s) [noise=true , address=77.90.31.13:4628 , piece_id=52Q5TEMXLZHLWJ5QWRFHGYWW6RCL2Y57AY6K2J7I3BUA22BNARQQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.400132462 (300h58m36.862224945s) [noise=true , address=45.91.116.5:4298 , piece_id=VTE4WIBLUWV74XMSSPP37CEAWPRQHA3GFV2RJRQYGRQM5JGF3H3Q, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.412114018 (300h58m36.874206491s) [noise=true , address=45.91.116.7:4010 , piece_id=S5NHKRILOZ47C4NXH4UXBG34ZF6IP2XHBR5JDDBNCBOZK26NALIA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:30:14.302718308 (300h58m59.764810791s) [noise=true , address=145.79.227.18:4007 , piece_id=6PZE3N3QHDGV5K5HULQE5UJXWWG7RAIACP2OVEZVIOTTCWVNQO6Q, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:30:14.319938798 (300h58m59.782031281s) [noise=true , address=145.79.227.7:4294 , piece_id=TLZVEBMQCCWINXB4AVRKHWKZ7WULCGUC6E5GGDJQLASKWNEHSJPA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:30:16.104270571 (300h59m1.566363044s) [noise=true , address=145.79.227.16:4910 , piece_id=T254A3S7TTJEBOIBDDCS34DU7HV25TXDTNZWF2OGE4KRQ7YEA5ZA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:30:16.104295528 (300h59m1.566388012s) [seg_pos={0 0}, upload=264873]: Segment reached optimal threshold of 13 pieces. Oct 6 02:30:16.104442268 (300h59m1.566534751s) [noise=true , address=145.79.227.4:4185 , piece_id=ONU6YDQFAMJPRG3ZF45M7KUX5EGXZVQEGCTOJK3VGE2JXXLHCBAQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN): protocol: write tcp 188.166.225.170:34966->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:34966->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34966->145.79.227.4:4185: use of closed network connection Oct 6 02:30:16.104456395 (300h59m1.566548878s) [noise=true , address=145.79.227.4:4185 , piece_id=ONU6YDQFAMJPRG3ZF45M7KUX5EGXZVQEGCTOJK3VGE2JXXLHCBAQ, seg_pos={0 0}, upload=264873]: Long tail cancellation due to optimal threshold Oct 6 02:30:16.104598005 (300h59m1.566690478s) [noise=true , address=145.79.227.19:4079 , piece_id=2VBVLN2SNA6WUICKMNMNM7URPVUGWPB5SVZ335XZ7HIAUM3FY5VA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: write tcp 188.166.225.170:57162->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:57162->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57162->145.79.227.19:4079: use of closed network connection Oct 6 02:30:16.104604216 (300h59m1.566696699s) [noise=true , address=145.79.227.19:4079 , piece_id=2VBVLN2SNA6WUICKMNMNM7URPVUGWPB5SVZ335XZ7HIAUM3FY5VA, seg_pos={0 0}, upload=264873]: Long tail cancellation due to optimal threshold Oct 6 02:30:16.104672547 (300h59m1.56676503s) [noise=true , address=145.79.227.17:4972 , piece_id=I54FS5Q4CMMM46S6EP3C5MEUAJCVNWX23OHTBMNQNKP4TRERVZ7A, seg_pos={0 0}, upload=264873]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: write tcp 188.166.225.170:33630->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:33630->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33630->145.79.227.17:4972: use of closed network connection Oct 6 02:30:16.104678057 (300h59m1.56677053s) [noise=true , address=145.79.227.17:4972 , piece_id=I54FS5Q4CMMM46S6EP3C5MEUAJCVNWX23OHTBMNQNKP4TRERVZ7A, seg_pos={0 0}, upload=264873]: Long tail cancellation due to optimal threshold Oct 6 02:30:16.104695541 (300h59m1.566788024s) [seg_pos={0 0}, upload=264873]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 6 02:30:16.104762427 (300h59m1.56685491s) [upload=264873]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 6 02:30:16.104768159 (300h59m1.566860632s) [upload=264873]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 6 02:30:16.463894599 (300h59m1.925987092s) [upload=264873]: Done uploading segments... Oct 6 02:30:16.463937391 (300h59m1.926029874s) [upload=264873]: Upload finished. err: <nil> Duration:* 6.045565102s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 6.033857091s FAIL (2025-10-06 02:35:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, Piece ID: T254A3S7TTJEBOIBDDCS34DU7HV25TXDTNZWF2OGE4KRQ7YEA5ZA) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: OS3T2GGABRXYK6HNEXK4C5F5HTJFDEP7SHA5QV2R3SCMXDE2TD5A) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: S5NHKRILOZ47C4NXH4UXBG34ZF6IP2XHBR5JDDBNCBOZK26NALIA) context deadline exceeded; piecestore: (Node ID: 12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, Piece ID: CHJRUVVLWAR7627JOLKL2QHAK7EHW72ZTGE3BTOAAM4ZL5AZL3KQ) context deadline exceeded; piecestore: (Node ID: 12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, Piece ID: TLZVEBMQCCWINXB4AVRKHWKZ7WULCGUC6E5GGDJQLASKWNEHSJPA) context deadline exceeded; piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: LJKUWHGMINU2P7JTS2QN5VP25HPZBKRMFXRDBSO5OTVYTS47SB4A) context deadline exceeded; piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: VTE4WIBLUWV74XMSSPP37CEAWPRQHA3GFV2RJRQYGRQM5JGF3H3Q) context deadline exceeded; piecestore: (Node ID: 1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, Piece ID: 52Q5TEMXLZHLWJ5QWRFHGYWW6RCL2Y57AY6K2J7I3BUA22BNARQQ) context deadline exceeded ======LOGS====== 914003723 (300h58m34.376096196s) [upload=264873]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 02:29:49.215144679 (300h58m34.677237162s) [seg_pos={0 0}, upload=264873]: Begin upload segment... Oct 6 02:29:49.215212839 (300h58m34.677305312s) [seg_pos={0 0}, upload=264873]: Done begin upload segment. Oct 6 02:29:49.215241294 (300h58m34.677333777s) [noise=true , address=145.79.227.7:4294 , piece_id=TLZVEBMQCCWINXB4AVRKHWKZ7WULCGUC6E5GGDJQLASKWNEHSJPA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.215249739 (300h58m34.677342232s) [noise=true , address=45.91.116.4:4937 , piece_id=CHJRUVVLWAR7627JOLKL2QHAK7EHW72ZTGE3BTOAAM4ZL5AZL3KQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.215349820 (300h58m34.677442303s) [noise=true , address=77.90.31.18:4805 , piece_id=LJKUWHGMINU2P7JTS2QN5VP25HPZBKRMFXRDBSO5OTVYTS47SB4A, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.216239644 (300h58m34.678332127s) [noise=true , address=45.91.116.5:4298 , piece_id=VTE4WIBLUWV74XMSSPP37CEAWPRQHA3GFV2RJRQYGRQM5JGF3H3Q, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.218583898 (300h58m34.680676392s) [noise=true , address=45.91.116.10:4082 , piece_id=NX5W46AGWBSZ7EM2LZHZDPE6MEPSLZMNVFZEE2LQ5F5LV7YSQRKQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.219326442 (300h58m34.681418925s) [noise=true , address=45.91.116.6:4974 , piece_id=DBXKLTPEBNF5GVB4PC26BY4EPXOWHU2OWSLWGSHS52S6CDUZIMOA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.222198982 (300h58m34.684291465s) [noise=true , address=145.79.227.17:4972 , piece_id=I54FS5Q4CMMM46S6EP3C5MEUAJCVNWX23OHTBMNQNKP4TRERVZ7A, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.222520114 (300h58m34.684612597s) [noise=true , address=145.79.227.4:4185 , piece_id=ONU6YDQFAMJPRG3ZF45M7KUX5EGXZVQEGCTOJK3VGE2JXXLHCBAQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.225940076 (300h58m34.688032569s) [noise=true , address=77.90.31.13:4628 , piece_id=52Q5TEMXLZHLWJ5QWRFHGYWW6RCL2Y57AY6K2J7I3BUA22BNARQQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.227928832 (300h58m34.690021305s) [noise=true , address=145.79.227.16:4910 , piece_id=T254A3S7TTJEBOIBDDCS34DU7HV25TXDTNZWF2OGE4KRQ7YEA5ZA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.227998645 (300h58m34.690091128s) [noise=true , address=45.91.116.14:4190 , piece_id=OS3T2GGABRXYK6HNEXK4C5F5HTJFDEP7SHA5QV2R3SCMXDE2TD5A, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.231020309 (300h58m34.693112792s) [noise=true , address=145.79.227.18:4007 , piece_id=6PZE3N3QHDGV5K5HULQE5UJXWWG7RAIACP2OVEZVIOTTCWVNQO6Q, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.232908575 (300h58m34.695001068s) [noise=true , address=45.91.116.10:4081 , piece_id=PRCQUWMDPZTR2CSWMOODBECZOLAVJ4OPOUTGQHCPTWXUDQGY2MBQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.233037220 (300h58m34.695129693s) [noise=true , address=145.79.227.19:4079 , piece_id=2VBVLN2SNA6WUICKMNMNM7URPVUGWPB5SVZ335XZ7HIAUM3FY5VA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.234302058 (300h58m34.696394531s) [noise=true , address=45.91.116.7:4010 , piece_id=S5NHKRILOZ47C4NXH4UXBG34ZF6IP2XHBR5JDDBNCBOZK26NALIA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.235631800 (300h58m34.697724283s) [noise=true , address=77.90.31.15:4698 , piece_id=NOCO6KPTUBBKYCITDRFBKA3H5EJTKMDAO4AFQFFXC4PPH3D46X2A, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:50.988532254 (300h58m36.450624747s) [upload=264873]: Next returned nil segment Oct 6 02:29:50.988555589 (300h58m36.450648082s) [upload=264873]: Waiting for error group managing segments... Oct 6 02:29:51.282838267 (300h58m36.74493075s) [noise=true , address=45.91.116.4:4937 , piece_id=CHJRUVVLWAR7627JOLKL2QHAK7EHW72ZTGE3BTOAAM4ZL5AZL3KQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.318573943 (300h58m36.780666416s) [noise=true , address=77.90.31.18:4805 , piece_id=LJKUWHGMINU2P7JTS2QN5VP25HPZBKRMFXRDBSO5OTVYTS47SB4A, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.322112521 (300h58m36.784205004s) [noise=true , address=77.90.31.15:4698 , piece_id=NOCO6KPTUBBKYCITDRFBKA3H5EJTKMDAO4AFQFFXC4PPH3D46X2A, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.350536624 (300h58m36.812629108s) [noise=true , address=45.91.116.6:4974 , piece_id=DBXKLTPEBNF5GVB4PC26BY4EPXOWHU2OWSLWGSHS52S6CDUZIMOA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.362171679 (300h58m36.824264162s) [noise=true , address=45.91.116.10:4081 , piece_id=PRCQUWMDPZTR2CSWMOODBECZOLAVJ4OPOUTGQHCPTWXUDQGY2MBQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.366398738 (300h58m36.828491221s) [noise=true , address=45.91.116.10:4082 , piece_id=NX5W46AGWBSZ7EM2LZHZDPE6MEPSLZMNVFZEE2LQ5F5LV7YSQRKQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.375437720 (300h58m36.837530203s) [noise=true , address=45.91.116.14:4190 , piece_id=OS3T2GGABRXYK6HNEXK4C5F5HTJFDEP7SHA5QV2R3SCMXDE2TD5A, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.397600181 (300h58m36.859692664s) [noise=true , address=77.90.31.13:4628 , piece_id=52Q5TEMXLZHLWJ5QWRFHGYWW6RCL2Y57AY6K2J7I3BUA22BNARQQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.400132462 (300h58m36.862224945s) [noise=true , address=45.91.116.5:4298 , piece_id=VTE4WIBLUWV74XMSSPP37CEAWPRQHA3GFV2RJRQYGRQM5JGF3H3Q, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.412114018 (300h58m36.874206491s) [noise=true , address=45.91.116.7:4010 , piece_id=S5NHKRILOZ47C4NXH4UXBG34ZF6IP2XHBR5JDDBNCBOZK26NALIA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:30:14.302718308 (300h58m59.764810791s) [noise=true , address=145.79.227.18:4007 , piece_id=6PZE3N3QHDGV5K5HULQE5UJXWWG7RAIACP2OVEZVIOTTCWVNQO6Q, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:30:14.319938798 (300h58m59.782031281s) [noise=true , address=145.79.227.7:4294 , piece_id=TLZVEBMQCCWINXB4AVRKHWKZ7WULCGUC6E5GGDJQLASKWNEHSJPA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:30:16.104270571 (300h59m1.566363044s) [noise=true , address=145.79.227.16:4910 , piece_id=T254A3S7TTJEBOIBDDCS34DU7HV25TXDTNZWF2OGE4KRQ7YEA5ZA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:30:16.104295528 (300h59m1.566388012s) [seg_pos={0 0}, upload=264873]: Segment reached optimal threshold of 13 pieces. Oct 6 02:30:16.104442268 (300h59m1.566534751s) [noise=true , address=145.79.227.4:4185 , piece_id=ONU6YDQFAMJPRG3ZF45M7KUX5EGXZVQEGCTOJK3VGE2JXXLHCBAQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN): protocol: write tcp 188.166.225.170:34966->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:34966->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34966->145.79.227.4:4185: use of closed network connection Oct 6 02:30:16.104456395 (300h59m1.566548878s) [noise=true , address=145.79.227.4:4185 , piece_id=ONU6YDQFAMJPRG3ZF45M7KUX5EGXZVQEGCTOJK3VGE2JXXLHCBAQ, seg_pos={0 0}, upload=264873]: Long tail cancellation due to optimal threshold Oct 6 02:30:16.104598005 (300h59m1.566690478s) [noise=true , address=145.79.227.19:4079 , piece_id=2VBVLN2SNA6WUICKMNMNM7URPVUGWPB5SVZ335XZ7HIAUM3FY5VA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: write tcp 188.166.225.170:57162->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:57162->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57162->145.79.227.19:4079: use of closed network connection Oct 6 02:30:16.104604216 (300h59m1.566696699s) [noise=true , address=145.79.227.19:4079 , piece_id=2VBVLN2SNA6WUICKMNMNM7URPVUGWPB5SVZ335XZ7HIAUM3FY5VA, seg_pos={0 0}, upload=264873]: Long tail cancellation due to optimal threshold Oct 6 02:30:16.104672547 (300h59m1.56676503s) [noise=true , address=145.79.227.17:4972 , piece_id=I54FS5Q4CMMM46S6EP3C5MEUAJCVNWX23OHTBMNQNKP4TRERVZ7A, seg_pos={0 0}, upload=264873]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: write tcp 188.166.225.170:33630->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:33630->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33630->145.79.227.17:4972: use of closed network connection Oct 6 02:30:16.104678057 (300h59m1.56677053s) [noise=true , address=145.79.227.17:4972 , piece_id=I54FS5Q4CMMM46S6EP3C5MEUAJCVNWX23OHTBMNQNKP4TRERVZ7A, seg_pos={0 0}, upload=264873]: Long tail cancellation due to optimal threshold Oct 6 02:30:16.104695541 (300h59m1.566788024s) [seg_pos={0 0}, upload=264873]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 6 02:30:16.104762427 (300h59m1.56685491s) [upload=264873]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 6 02:30:16.104768159 (300h59m1.566860632s) [upload=264873]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 6 02:30:16.463894599 (300h59m1.925987092s) [upload=264873]: Done uploading segments... Oct 6 02:30:16.463937391 (300h59m1.926029874s) [upload=264873]: Upload finished. err: <nil> Duration:* 6.033894142s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 6.110339423s FAIL (2025-10-06 02:34:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, Piece ID: T254A3S7TTJEBOIBDDCS34DU7HV25TXDTNZWF2OGE4KRQ7YEA5ZA) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: OS3T2GGABRXYK6HNEXK4C5F5HTJFDEP7SHA5QV2R3SCMXDE2TD5A) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: S5NHKRILOZ47C4NXH4UXBG34ZF6IP2XHBR5JDDBNCBOZK26NALIA) context deadline exceeded; piecestore: (Node ID: 12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, Piece ID: CHJRUVVLWAR7627JOLKL2QHAK7EHW72ZTGE3BTOAAM4ZL5AZL3KQ) context deadline exceeded; piecestore: (Node ID: 12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, Piece ID: TLZVEBMQCCWINXB4AVRKHWKZ7WULCGUC6E5GGDJQLASKWNEHSJPA) context deadline exceeded; piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: LJKUWHGMINU2P7JTS2QN5VP25HPZBKRMFXRDBSO5OTVYTS47SB4A) context deadline exceeded; piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: VTE4WIBLUWV74XMSSPP37CEAWPRQHA3GFV2RJRQYGRQM5JGF3H3Q) context deadline exceeded; piecestore: (Node ID: 1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, Piece ID: 52Q5TEMXLZHLWJ5QWRFHGYWW6RCL2Y57AY6K2J7I3BUA22BNARQQ) context deadline exceeded ======LOGS====== 914003723 (300h58m34.376096196s) [upload=264873]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 02:29:49.215144679 (300h58m34.677237162s) [seg_pos={0 0}, upload=264873]: Begin upload segment... Oct 6 02:29:49.215212839 (300h58m34.677305312s) [seg_pos={0 0}, upload=264873]: Done begin upload segment. Oct 6 02:29:49.215241294 (300h58m34.677333777s) [noise=true , address=145.79.227.7:4294 , piece_id=TLZVEBMQCCWINXB4AVRKHWKZ7WULCGUC6E5GGDJQLASKWNEHSJPA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.215249739 (300h58m34.677342232s) [noise=true , address=45.91.116.4:4937 , piece_id=CHJRUVVLWAR7627JOLKL2QHAK7EHW72ZTGE3BTOAAM4ZL5AZL3KQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.215349820 (300h58m34.677442303s) [noise=true , address=77.90.31.18:4805 , piece_id=LJKUWHGMINU2P7JTS2QN5VP25HPZBKRMFXRDBSO5OTVYTS47SB4A, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.216239644 (300h58m34.678332127s) [noise=true , address=45.91.116.5:4298 , piece_id=VTE4WIBLUWV74XMSSPP37CEAWPRQHA3GFV2RJRQYGRQM5JGF3H3Q, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.218583898 (300h58m34.680676392s) [noise=true , address=45.91.116.10:4082 , piece_id=NX5W46AGWBSZ7EM2LZHZDPE6MEPSLZMNVFZEE2LQ5F5LV7YSQRKQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.219326442 (300h58m34.681418925s) [noise=true , address=45.91.116.6:4974 , piece_id=DBXKLTPEBNF5GVB4PC26BY4EPXOWHU2OWSLWGSHS52S6CDUZIMOA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.222198982 (300h58m34.684291465s) [noise=true , address=145.79.227.17:4972 , piece_id=I54FS5Q4CMMM46S6EP3C5MEUAJCVNWX23OHTBMNQNKP4TRERVZ7A, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.222520114 (300h58m34.684612597s) [noise=true , address=145.79.227.4:4185 , piece_id=ONU6YDQFAMJPRG3ZF45M7KUX5EGXZVQEGCTOJK3VGE2JXXLHCBAQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.225940076 (300h58m34.688032569s) [noise=true , address=77.90.31.13:4628 , piece_id=52Q5TEMXLZHLWJ5QWRFHGYWW6RCL2Y57AY6K2J7I3BUA22BNARQQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.227928832 (300h58m34.690021305s) [noise=true , address=145.79.227.16:4910 , piece_id=T254A3S7TTJEBOIBDDCS34DU7HV25TXDTNZWF2OGE4KRQ7YEA5ZA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.227998645 (300h58m34.690091128s) [noise=true , address=45.91.116.14:4190 , piece_id=OS3T2GGABRXYK6HNEXK4C5F5HTJFDEP7SHA5QV2R3SCMXDE2TD5A, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.231020309 (300h58m34.693112792s) [noise=true , address=145.79.227.18:4007 , piece_id=6PZE3N3QHDGV5K5HULQE5UJXWWG7RAIACP2OVEZVIOTTCWVNQO6Q, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.232908575 (300h58m34.695001068s) [noise=true , address=45.91.116.10:4081 , piece_id=PRCQUWMDPZTR2CSWMOODBECZOLAVJ4OPOUTGQHCPTWXUDQGY2MBQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.233037220 (300h58m34.695129693s) [noise=true , address=145.79.227.19:4079 , piece_id=2VBVLN2SNA6WUICKMNMNM7URPVUGWPB5SVZ335XZ7HIAUM3FY5VA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.234302058 (300h58m34.696394531s) [noise=true , address=45.91.116.7:4010 , piece_id=S5NHKRILOZ47C4NXH4UXBG34ZF6IP2XHBR5JDDBNCBOZK26NALIA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.235631800 (300h58m34.697724283s) [noise=true , address=77.90.31.15:4698 , piece_id=NOCO6KPTUBBKYCITDRFBKA3H5EJTKMDAO4AFQFFXC4PPH3D46X2A, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:50.988532254 (300h58m36.450624747s) [upload=264873]: Next returned nil segment Oct 6 02:29:50.988555589 (300h58m36.450648082s) [upload=264873]: Waiting for error group managing segments... Oct 6 02:29:51.282838267 (300h58m36.74493075s) [noise=true , address=45.91.116.4:4937 , piece_id=CHJRUVVLWAR7627JOLKL2QHAK7EHW72ZTGE3BTOAAM4ZL5AZL3KQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.318573943 (300h58m36.780666416s) [noise=true , address=77.90.31.18:4805 , piece_id=LJKUWHGMINU2P7JTS2QN5VP25HPZBKRMFXRDBSO5OTVYTS47SB4A, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.322112521 (300h58m36.784205004s) [noise=true , address=77.90.31.15:4698 , piece_id=NOCO6KPTUBBKYCITDRFBKA3H5EJTKMDAO4AFQFFXC4PPH3D46X2A, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.350536624 (300h58m36.812629108s) [noise=true , address=45.91.116.6:4974 , piece_id=DBXKLTPEBNF5GVB4PC26BY4EPXOWHU2OWSLWGSHS52S6CDUZIMOA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.362171679 (300h58m36.824264162s) [noise=true , address=45.91.116.10:4081 , piece_id=PRCQUWMDPZTR2CSWMOODBECZOLAVJ4OPOUTGQHCPTWXUDQGY2MBQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.366398738 (300h58m36.828491221s) [noise=true , address=45.91.116.10:4082 , piece_id=NX5W46AGWBSZ7EM2LZHZDPE6MEPSLZMNVFZEE2LQ5F5LV7YSQRKQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.375437720 (300h58m36.837530203s) [noise=true , address=45.91.116.14:4190 , piece_id=OS3T2GGABRXYK6HNEXK4C5F5HTJFDEP7SHA5QV2R3SCMXDE2TD5A, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.397600181 (300h58m36.859692664s) [noise=true , address=77.90.31.13:4628 , piece_id=52Q5TEMXLZHLWJ5QWRFHGYWW6RCL2Y57AY6K2J7I3BUA22BNARQQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.400132462 (300h58m36.862224945s) [noise=true , address=45.91.116.5:4298 , piece_id=VTE4WIBLUWV74XMSSPP37CEAWPRQHA3GFV2RJRQYGRQM5JGF3H3Q, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.412114018 (300h58m36.874206491s) [noise=true , address=45.91.116.7:4010 , piece_id=S5NHKRILOZ47C4NXH4UXBG34ZF6IP2XHBR5JDDBNCBOZK26NALIA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:30:14.302718308 (300h58m59.764810791s) [noise=true , address=145.79.227.18:4007 , piece_id=6PZE3N3QHDGV5K5HULQE5UJXWWG7RAIACP2OVEZVIOTTCWVNQO6Q, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:30:14.319938798 (300h58m59.782031281s) [noise=true , address=145.79.227.7:4294 , piece_id=TLZVEBMQCCWINXB4AVRKHWKZ7WULCGUC6E5GGDJQLASKWNEHSJPA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:30:16.104270571 (300h59m1.566363044s) [noise=true , address=145.79.227.16:4910 , piece_id=T254A3S7TTJEBOIBDDCS34DU7HV25TXDTNZWF2OGE4KRQ7YEA5ZA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:30:16.104295528 (300h59m1.566388012s) [seg_pos={0 0}, upload=264873]: Segment reached optimal threshold of 13 pieces. Oct 6 02:30:16.104442268 (300h59m1.566534751s) [noise=true , address=145.79.227.4:4185 , piece_id=ONU6YDQFAMJPRG3ZF45M7KUX5EGXZVQEGCTOJK3VGE2JXXLHCBAQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN): protocol: write tcp 188.166.225.170:34966->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:34966->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34966->145.79.227.4:4185: use of closed network connection Oct 6 02:30:16.104456395 (300h59m1.566548878s) [noise=true , address=145.79.227.4:4185 , piece_id=ONU6YDQFAMJPRG3ZF45M7KUX5EGXZVQEGCTOJK3VGE2JXXLHCBAQ, seg_pos={0 0}, upload=264873]: Long tail cancellation due to optimal threshold Oct 6 02:30:16.104598005 (300h59m1.566690478s) [noise=true , address=145.79.227.19:4079 , piece_id=2VBVLN2SNA6WUICKMNMNM7URPVUGWPB5SVZ335XZ7HIAUM3FY5VA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: write tcp 188.166.225.170:57162->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:57162->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57162->145.79.227.19:4079: use of closed network connection Oct 6 02:30:16.104604216 (300h59m1.566696699s) [noise=true , address=145.79.227.19:4079 , piece_id=2VBVLN2SNA6WUICKMNMNM7URPVUGWPB5SVZ335XZ7HIAUM3FY5VA, seg_pos={0 0}, upload=264873]: Long tail cancellation due to optimal threshold Oct 6 02:30:16.104672547 (300h59m1.56676503s) [noise=true , address=145.79.227.17:4972 , piece_id=I54FS5Q4CMMM46S6EP3C5MEUAJCVNWX23OHTBMNQNKP4TRERVZ7A, seg_pos={0 0}, upload=264873]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: write tcp 188.166.225.170:33630->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:33630->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33630->145.79.227.17:4972: use of closed network connection Oct 6 02:30:16.104678057 (300h59m1.56677053s) [noise=true , address=145.79.227.17:4972 , piece_id=I54FS5Q4CMMM46S6EP3C5MEUAJCVNWX23OHTBMNQNKP4TRERVZ7A, seg_pos={0 0}, upload=264873]: Long tail cancellation due to optimal threshold Oct 6 02:30:16.104695541 (300h59m1.566788024s) [seg_pos={0 0}, upload=264873]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 6 02:30:16.104762427 (300h59m1.56685491s) [upload=264873]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 6 02:30:16.104768159 (300h59m1.566860632s) [upload=264873]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 6 02:30:16.463894599 (300h59m1.925987092s) [upload=264873]: Done uploading segments... Oct 6 02:30:16.463937391 (300h59m1.926029874s) [upload=264873]: Upload finished. err: <nil> Duration:* 6.110370163s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 5.943150162s FAIL (2025-10-06 02:33:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, Piece ID: T254A3S7TTJEBOIBDDCS34DU7HV25TXDTNZWF2OGE4KRQ7YEA5ZA) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: OS3T2GGABRXYK6HNEXK4C5F5HTJFDEP7SHA5QV2R3SCMXDE2TD5A) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: S5NHKRILOZ47C4NXH4UXBG34ZF6IP2XHBR5JDDBNCBOZK26NALIA) context deadline exceeded; piecestore: (Node ID: 12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, Piece ID: CHJRUVVLWAR7627JOLKL2QHAK7EHW72ZTGE3BTOAAM4ZL5AZL3KQ) context deadline exceeded; piecestore: (Node ID: 12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, Piece ID: TLZVEBMQCCWINXB4AVRKHWKZ7WULCGUC6E5GGDJQLASKWNEHSJPA) context deadline exceeded; piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: LJKUWHGMINU2P7JTS2QN5VP25HPZBKRMFXRDBSO5OTVYTS47SB4A) context deadline exceeded; piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: VTE4WIBLUWV74XMSSPP37CEAWPRQHA3GFV2RJRQYGRQM5JGF3H3Q) context deadline exceeded; piecestore: (Node ID: 1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, Piece ID: 52Q5TEMXLZHLWJ5QWRFHGYWW6RCL2Y57AY6K2J7I3BUA22BNARQQ) context deadline exceeded ======LOGS====== 914003723 (300h58m34.376096196s) [upload=264873]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 02:29:49.215144679 (300h58m34.677237162s) [seg_pos={0 0}, upload=264873]: Begin upload segment... Oct 6 02:29:49.215212839 (300h58m34.677305312s) [seg_pos={0 0}, upload=264873]: Done begin upload segment. Oct 6 02:29:49.215241294 (300h58m34.677333777s) [noise=true , address=145.79.227.7:4294 , piece_id=TLZVEBMQCCWINXB4AVRKHWKZ7WULCGUC6E5GGDJQLASKWNEHSJPA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.215249739 (300h58m34.677342232s) [noise=true , address=45.91.116.4:4937 , piece_id=CHJRUVVLWAR7627JOLKL2QHAK7EHW72ZTGE3BTOAAM4ZL5AZL3KQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.215349820 (300h58m34.677442303s) [noise=true , address=77.90.31.18:4805 , piece_id=LJKUWHGMINU2P7JTS2QN5VP25HPZBKRMFXRDBSO5OTVYTS47SB4A, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.216239644 (300h58m34.678332127s) [noise=true , address=45.91.116.5:4298 , piece_id=VTE4WIBLUWV74XMSSPP37CEAWPRQHA3GFV2RJRQYGRQM5JGF3H3Q, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.218583898 (300h58m34.680676392s) [noise=true , address=45.91.116.10:4082 , piece_id=NX5W46AGWBSZ7EM2LZHZDPE6MEPSLZMNVFZEE2LQ5F5LV7YSQRKQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.219326442 (300h58m34.681418925s) [noise=true , address=45.91.116.6:4974 , piece_id=DBXKLTPEBNF5GVB4PC26BY4EPXOWHU2OWSLWGSHS52S6CDUZIMOA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.222198982 (300h58m34.684291465s) [noise=true , address=145.79.227.17:4972 , piece_id=I54FS5Q4CMMM46S6EP3C5MEUAJCVNWX23OHTBMNQNKP4TRERVZ7A, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.222520114 (300h58m34.684612597s) [noise=true , address=145.79.227.4:4185 , piece_id=ONU6YDQFAMJPRG3ZF45M7KUX5EGXZVQEGCTOJK3VGE2JXXLHCBAQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.225940076 (300h58m34.688032569s) [noise=true , address=77.90.31.13:4628 , piece_id=52Q5TEMXLZHLWJ5QWRFHGYWW6RCL2Y57AY6K2J7I3BUA22BNARQQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.227928832 (300h58m34.690021305s) [noise=true , address=145.79.227.16:4910 , piece_id=T254A3S7TTJEBOIBDDCS34DU7HV25TXDTNZWF2OGE4KRQ7YEA5ZA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.227998645 (300h58m34.690091128s) [noise=true , address=45.91.116.14:4190 , piece_id=OS3T2GGABRXYK6HNEXK4C5F5HTJFDEP7SHA5QV2R3SCMXDE2TD5A, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.231020309 (300h58m34.693112792s) [noise=true , address=145.79.227.18:4007 , piece_id=6PZE3N3QHDGV5K5HULQE5UJXWWG7RAIACP2OVEZVIOTTCWVNQO6Q, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.232908575 (300h58m34.695001068s) [noise=true , address=45.91.116.10:4081 , piece_id=PRCQUWMDPZTR2CSWMOODBECZOLAVJ4OPOUTGQHCPTWXUDQGY2MBQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.233037220 (300h58m34.695129693s) [noise=true , address=145.79.227.19:4079 , piece_id=2VBVLN2SNA6WUICKMNMNM7URPVUGWPB5SVZ335XZ7HIAUM3FY5VA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.234302058 (300h58m34.696394531s) [noise=true , address=45.91.116.7:4010 , piece_id=S5NHKRILOZ47C4NXH4UXBG34ZF6IP2XHBR5JDDBNCBOZK26NALIA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.235631800 (300h58m34.697724283s) [noise=true , address=77.90.31.15:4698 , piece_id=NOCO6KPTUBBKYCITDRFBKA3H5EJTKMDAO4AFQFFXC4PPH3D46X2A, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:50.988532254 (300h58m36.450624747s) [upload=264873]: Next returned nil segment Oct 6 02:29:50.988555589 (300h58m36.450648082s) [upload=264873]: Waiting for error group managing segments... Oct 6 02:29:51.282838267 (300h58m36.74493075s) [noise=true , address=45.91.116.4:4937 , piece_id=CHJRUVVLWAR7627JOLKL2QHAK7EHW72ZTGE3BTOAAM4ZL5AZL3KQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.318573943 (300h58m36.780666416s) [noise=true , address=77.90.31.18:4805 , piece_id=LJKUWHGMINU2P7JTS2QN5VP25HPZBKRMFXRDBSO5OTVYTS47SB4A, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.322112521 (300h58m36.784205004s) [noise=true , address=77.90.31.15:4698 , piece_id=NOCO6KPTUBBKYCITDRFBKA3H5EJTKMDAO4AFQFFXC4PPH3D46X2A, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.350536624 (300h58m36.812629108s) [noise=true , address=45.91.116.6:4974 , piece_id=DBXKLTPEBNF5GVB4PC26BY4EPXOWHU2OWSLWGSHS52S6CDUZIMOA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.362171679 (300h58m36.824264162s) [noise=true , address=45.91.116.10:4081 , piece_id=PRCQUWMDPZTR2CSWMOODBECZOLAVJ4OPOUTGQHCPTWXUDQGY2MBQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.366398738 (300h58m36.828491221s) [noise=true , address=45.91.116.10:4082 , piece_id=NX5W46AGWBSZ7EM2LZHZDPE6MEPSLZMNVFZEE2LQ5F5LV7YSQRKQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.375437720 (300h58m36.837530203s) [noise=true , address=45.91.116.14:4190 , piece_id=OS3T2GGABRXYK6HNEXK4C5F5HTJFDEP7SHA5QV2R3SCMXDE2TD5A, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.397600181 (300h58m36.859692664s) [noise=true , address=77.90.31.13:4628 , piece_id=52Q5TEMXLZHLWJ5QWRFHGYWW6RCL2Y57AY6K2J7I3BUA22BNARQQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.400132462 (300h58m36.862224945s) [noise=true , address=45.91.116.5:4298 , piece_id=VTE4WIBLUWV74XMSSPP37CEAWPRQHA3GFV2RJRQYGRQM5JGF3H3Q, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.412114018 (300h58m36.874206491s) [noise=true , address=45.91.116.7:4010 , piece_id=S5NHKRILOZ47C4NXH4UXBG34ZF6IP2XHBR5JDDBNCBOZK26NALIA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:30:14.302718308 (300h58m59.764810791s) [noise=true , address=145.79.227.18:4007 , piece_id=6PZE3N3QHDGV5K5HULQE5UJXWWG7RAIACP2OVEZVIOTTCWVNQO6Q, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:30:14.319938798 (300h58m59.782031281s) [noise=true , address=145.79.227.7:4294 , piece_id=TLZVEBMQCCWINXB4AVRKHWKZ7WULCGUC6E5GGDJQLASKWNEHSJPA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:30:16.104270571 (300h59m1.566363044s) [noise=true , address=145.79.227.16:4910 , piece_id=T254A3S7TTJEBOIBDDCS34DU7HV25TXDTNZWF2OGE4KRQ7YEA5ZA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:30:16.104295528 (300h59m1.566388012s) [seg_pos={0 0}, upload=264873]: Segment reached optimal threshold of 13 pieces. Oct 6 02:30:16.104442268 (300h59m1.566534751s) [noise=true , address=145.79.227.4:4185 , piece_id=ONU6YDQFAMJPRG3ZF45M7KUX5EGXZVQEGCTOJK3VGE2JXXLHCBAQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN): protocol: write tcp 188.166.225.170:34966->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:34966->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34966->145.79.227.4:4185: use of closed network connection Oct 6 02:30:16.104456395 (300h59m1.566548878s) [noise=true , address=145.79.227.4:4185 , piece_id=ONU6YDQFAMJPRG3ZF45M7KUX5EGXZVQEGCTOJK3VGE2JXXLHCBAQ, seg_pos={0 0}, upload=264873]: Long tail cancellation due to optimal threshold Oct 6 02:30:16.104598005 (300h59m1.566690478s) [noise=true , address=145.79.227.19:4079 , piece_id=2VBVLN2SNA6WUICKMNMNM7URPVUGWPB5SVZ335XZ7HIAUM3FY5VA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: write tcp 188.166.225.170:57162->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:57162->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57162->145.79.227.19:4079: use of closed network connection Oct 6 02:30:16.104604216 (300h59m1.566696699s) [noise=true , address=145.79.227.19:4079 , piece_id=2VBVLN2SNA6WUICKMNMNM7URPVUGWPB5SVZ335XZ7HIAUM3FY5VA, seg_pos={0 0}, upload=264873]: Long tail cancellation due to optimal threshold Oct 6 02:30:16.104672547 (300h59m1.56676503s) [noise=true , address=145.79.227.17:4972 , piece_id=I54FS5Q4CMMM46S6EP3C5MEUAJCVNWX23OHTBMNQNKP4TRERVZ7A, seg_pos={0 0}, upload=264873]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: write tcp 188.166.225.170:33630->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:33630->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33630->145.79.227.17:4972: use of closed network connection Oct 6 02:30:16.104678057 (300h59m1.56677053s) [noise=true , address=145.79.227.17:4972 , piece_id=I54FS5Q4CMMM46S6EP3C5MEUAJCVNWX23OHTBMNQNKP4TRERVZ7A, seg_pos={0 0}, upload=264873]: Long tail cancellation due to optimal threshold Oct 6 02:30:16.104695541 (300h59m1.566788024s) [seg_pos={0 0}, upload=264873]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 6 02:30:16.104762427 (300h59m1.56685491s) [upload=264873]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 6 02:30:16.104768159 (300h59m1.566860632s) [upload=264873]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 6 02:30:16.463894599 (300h59m1.925987092s) [upload=264873]: Done uploading segments... Oct 6 02:30:16.463937391 (300h59m1.926029874s) [upload=264873]: Upload finished. err: <nil> Duration:* 5.943181823s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 6.112980943s FAIL (2025-10-06 02:32:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, Piece ID: T254A3S7TTJEBOIBDDCS34DU7HV25TXDTNZWF2OGE4KRQ7YEA5ZA) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: OS3T2GGABRXYK6HNEXK4C5F5HTJFDEP7SHA5QV2R3SCMXDE2TD5A) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: S5NHKRILOZ47C4NXH4UXBG34ZF6IP2XHBR5JDDBNCBOZK26NALIA) context deadline exceeded; piecestore: (Node ID: 12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, Piece ID: CHJRUVVLWAR7627JOLKL2QHAK7EHW72ZTGE3BTOAAM4ZL5AZL3KQ) context deadline exceeded; piecestore: (Node ID: 12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, Piece ID: TLZVEBMQCCWINXB4AVRKHWKZ7WULCGUC6E5GGDJQLASKWNEHSJPA) context deadline exceeded; piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: LJKUWHGMINU2P7JTS2QN5VP25HPZBKRMFXRDBSO5OTVYTS47SB4A) context deadline exceeded; piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: VTE4WIBLUWV74XMSSPP37CEAWPRQHA3GFV2RJRQYGRQM5JGF3H3Q) context deadline exceeded; piecestore: (Node ID: 1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, Piece ID: 52Q5TEMXLZHLWJ5QWRFHGYWW6RCL2Y57AY6K2J7I3BUA22BNARQQ) context deadline exceeded ======LOGS====== 914003723 (300h58m34.376096196s) [upload=264873]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 02:29:49.215144679 (300h58m34.677237162s) [seg_pos={0 0}, upload=264873]: Begin upload segment... Oct 6 02:29:49.215212839 (300h58m34.677305312s) [seg_pos={0 0}, upload=264873]: Done begin upload segment. Oct 6 02:29:49.215241294 (300h58m34.677333777s) [noise=true , address=145.79.227.7:4294 , piece_id=TLZVEBMQCCWINXB4AVRKHWKZ7WULCGUC6E5GGDJQLASKWNEHSJPA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.215249739 (300h58m34.677342232s) [noise=true , address=45.91.116.4:4937 , piece_id=CHJRUVVLWAR7627JOLKL2QHAK7EHW72ZTGE3BTOAAM4ZL5AZL3KQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.215349820 (300h58m34.677442303s) [noise=true , address=77.90.31.18:4805 , piece_id=LJKUWHGMINU2P7JTS2QN5VP25HPZBKRMFXRDBSO5OTVYTS47SB4A, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.216239644 (300h58m34.678332127s) [noise=true , address=45.91.116.5:4298 , piece_id=VTE4WIBLUWV74XMSSPP37CEAWPRQHA3GFV2RJRQYGRQM5JGF3H3Q, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.218583898 (300h58m34.680676392s) [noise=true , address=45.91.116.10:4082 , piece_id=NX5W46AGWBSZ7EM2LZHZDPE6MEPSLZMNVFZEE2LQ5F5LV7YSQRKQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.219326442 (300h58m34.681418925s) [noise=true , address=45.91.116.6:4974 , piece_id=DBXKLTPEBNF5GVB4PC26BY4EPXOWHU2OWSLWGSHS52S6CDUZIMOA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.222198982 (300h58m34.684291465s) [noise=true , address=145.79.227.17:4972 , piece_id=I54FS5Q4CMMM46S6EP3C5MEUAJCVNWX23OHTBMNQNKP4TRERVZ7A, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.222520114 (300h58m34.684612597s) [noise=true , address=145.79.227.4:4185 , piece_id=ONU6YDQFAMJPRG3ZF45M7KUX5EGXZVQEGCTOJK3VGE2JXXLHCBAQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.225940076 (300h58m34.688032569s) [noise=true , address=77.90.31.13:4628 , piece_id=52Q5TEMXLZHLWJ5QWRFHGYWW6RCL2Y57AY6K2J7I3BUA22BNARQQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.227928832 (300h58m34.690021305s) [noise=true , address=145.79.227.16:4910 , piece_id=T254A3S7TTJEBOIBDDCS34DU7HV25TXDTNZWF2OGE4KRQ7YEA5ZA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.227998645 (300h58m34.690091128s) [noise=true , address=45.91.116.14:4190 , piece_id=OS3T2GGABRXYK6HNEXK4C5F5HTJFDEP7SHA5QV2R3SCMXDE2TD5A, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.231020309 (300h58m34.693112792s) [noise=true , address=145.79.227.18:4007 , piece_id=6PZE3N3QHDGV5K5HULQE5UJXWWG7RAIACP2OVEZVIOTTCWVNQO6Q, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.232908575 (300h58m34.695001068s) [noise=true , address=45.91.116.10:4081 , piece_id=PRCQUWMDPZTR2CSWMOODBECZOLAVJ4OPOUTGQHCPTWXUDQGY2MBQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.233037220 (300h58m34.695129693s) [noise=true , address=145.79.227.19:4079 , piece_id=2VBVLN2SNA6WUICKMNMNM7URPVUGWPB5SVZ335XZ7HIAUM3FY5VA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.234302058 (300h58m34.696394531s) [noise=true , address=45.91.116.7:4010 , piece_id=S5NHKRILOZ47C4NXH4UXBG34ZF6IP2XHBR5JDDBNCBOZK26NALIA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.235631800 (300h58m34.697724283s) [noise=true , address=77.90.31.15:4698 , piece_id=NOCO6KPTUBBKYCITDRFBKA3H5EJTKMDAO4AFQFFXC4PPH3D46X2A, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:50.988532254 (300h58m36.450624747s) [upload=264873]: Next returned nil segment Oct 6 02:29:50.988555589 (300h58m36.450648082s) [upload=264873]: Waiting for error group managing segments... Oct 6 02:29:51.282838267 (300h58m36.74493075s) [noise=true , address=45.91.116.4:4937 , piece_id=CHJRUVVLWAR7627JOLKL2QHAK7EHW72ZTGE3BTOAAM4ZL5AZL3KQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.318573943 (300h58m36.780666416s) [noise=true , address=77.90.31.18:4805 , piece_id=LJKUWHGMINU2P7JTS2QN5VP25HPZBKRMFXRDBSO5OTVYTS47SB4A, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.322112521 (300h58m36.784205004s) [noise=true , address=77.90.31.15:4698 , piece_id=NOCO6KPTUBBKYCITDRFBKA3H5EJTKMDAO4AFQFFXC4PPH3D46X2A, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.350536624 (300h58m36.812629108s) [noise=true , address=45.91.116.6:4974 , piece_id=DBXKLTPEBNF5GVB4PC26BY4EPXOWHU2OWSLWGSHS52S6CDUZIMOA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.362171679 (300h58m36.824264162s) [noise=true , address=45.91.116.10:4081 , piece_id=PRCQUWMDPZTR2CSWMOODBECZOLAVJ4OPOUTGQHCPTWXUDQGY2MBQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.366398738 (300h58m36.828491221s) [noise=true , address=45.91.116.10:4082 , piece_id=NX5W46AGWBSZ7EM2LZHZDPE6MEPSLZMNVFZEE2LQ5F5LV7YSQRKQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.375437720 (300h58m36.837530203s) [noise=true , address=45.91.116.14:4190 , piece_id=OS3T2GGABRXYK6HNEXK4C5F5HTJFDEP7SHA5QV2R3SCMXDE2TD5A, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.397600181 (300h58m36.859692664s) [noise=true , address=77.90.31.13:4628 , piece_id=52Q5TEMXLZHLWJ5QWRFHGYWW6RCL2Y57AY6K2J7I3BUA22BNARQQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.400132462 (300h58m36.862224945s) [noise=true , address=45.91.116.5:4298 , piece_id=VTE4WIBLUWV74XMSSPP37CEAWPRQHA3GFV2RJRQYGRQM5JGF3H3Q, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.412114018 (300h58m36.874206491s) [noise=true , address=45.91.116.7:4010 , piece_id=S5NHKRILOZ47C4NXH4UXBG34ZF6IP2XHBR5JDDBNCBOZK26NALIA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:30:14.302718308 (300h58m59.764810791s) [noise=true , address=145.79.227.18:4007 , piece_id=6PZE3N3QHDGV5K5HULQE5UJXWWG7RAIACP2OVEZVIOTTCWVNQO6Q, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:30:14.319938798 (300h58m59.782031281s) [noise=true , address=145.79.227.7:4294 , piece_id=TLZVEBMQCCWINXB4AVRKHWKZ7WULCGUC6E5GGDJQLASKWNEHSJPA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:30:16.104270571 (300h59m1.566363044s) [noise=true , address=145.79.227.16:4910 , piece_id=T254A3S7TTJEBOIBDDCS34DU7HV25TXDTNZWF2OGE4KRQ7YEA5ZA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:30:16.104295528 (300h59m1.566388012s) [seg_pos={0 0}, upload=264873]: Segment reached optimal threshold of 13 pieces. Oct 6 02:30:16.104442268 (300h59m1.566534751s) [noise=true , address=145.79.227.4:4185 , piece_id=ONU6YDQFAMJPRG3ZF45M7KUX5EGXZVQEGCTOJK3VGE2JXXLHCBAQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN): protocol: write tcp 188.166.225.170:34966->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:34966->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34966->145.79.227.4:4185: use of closed network connection Oct 6 02:30:16.104456395 (300h59m1.566548878s) [noise=true , address=145.79.227.4:4185 , piece_id=ONU6YDQFAMJPRG3ZF45M7KUX5EGXZVQEGCTOJK3VGE2JXXLHCBAQ, seg_pos={0 0}, upload=264873]: Long tail cancellation due to optimal threshold Oct 6 02:30:16.104598005 (300h59m1.566690478s) [noise=true , address=145.79.227.19:4079 , piece_id=2VBVLN2SNA6WUICKMNMNM7URPVUGWPB5SVZ335XZ7HIAUM3FY5VA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: write tcp 188.166.225.170:57162->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:57162->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57162->145.79.227.19:4079: use of closed network connection Oct 6 02:30:16.104604216 (300h59m1.566696699s) [noise=true , address=145.79.227.19:4079 , piece_id=2VBVLN2SNA6WUICKMNMNM7URPVUGWPB5SVZ335XZ7HIAUM3FY5VA, seg_pos={0 0}, upload=264873]: Long tail cancellation due to optimal threshold Oct 6 02:30:16.104672547 (300h59m1.56676503s) [noise=true , address=145.79.227.17:4972 , piece_id=I54FS5Q4CMMM46S6EP3C5MEUAJCVNWX23OHTBMNQNKP4TRERVZ7A, seg_pos={0 0}, upload=264873]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: write tcp 188.166.225.170:33630->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:33630->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33630->145.79.227.17:4972: use of closed network connection Oct 6 02:30:16.104678057 (300h59m1.56677053s) [noise=true , address=145.79.227.17:4972 , piece_id=I54FS5Q4CMMM46S6EP3C5MEUAJCVNWX23OHTBMNQNKP4TRERVZ7A, seg_pos={0 0}, upload=264873]: Long tail cancellation due to optimal threshold Oct 6 02:30:16.104695541 (300h59m1.566788024s) [seg_pos={0 0}, upload=264873]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 6 02:30:16.104762427 (300h59m1.56685491s) [upload=264873]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 6 02:30:16.104768159 (300h59m1.566860632s) [upload=264873]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 6 02:30:16.463894599 (300h59m1.925987092s) [upload=264873]: Done uploading segments... Oct 6 02:30:16.463937391 (300h59m1.926029874s) [upload=264873]: Upload finished. err: <nil> Duration:* 6.113012955s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 6.45920744s FAIL (2025-10-06 02:32:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, Piece ID: T254A3S7TTJEBOIBDDCS34DU7HV25TXDTNZWF2OGE4KRQ7YEA5ZA) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: OS3T2GGABRXYK6HNEXK4C5F5HTJFDEP7SHA5QV2R3SCMXDE2TD5A) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: S5NHKRILOZ47C4NXH4UXBG34ZF6IP2XHBR5JDDBNCBOZK26NALIA) context deadline exceeded; piecestore: (Node ID: 12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, Piece ID: CHJRUVVLWAR7627JOLKL2QHAK7EHW72ZTGE3BTOAAM4ZL5AZL3KQ) context deadline exceeded; piecestore: (Node ID: 12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, Piece ID: TLZVEBMQCCWINXB4AVRKHWKZ7WULCGUC6E5GGDJQLASKWNEHSJPA) context deadline exceeded; piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: LJKUWHGMINU2P7JTS2QN5VP25HPZBKRMFXRDBSO5OTVYTS47SB4A) context deadline exceeded; piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: VTE4WIBLUWV74XMSSPP37CEAWPRQHA3GFV2RJRQYGRQM5JGF3H3Q) context deadline exceeded; piecestore: (Node ID: 1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, Piece ID: 52Q5TEMXLZHLWJ5QWRFHGYWW6RCL2Y57AY6K2J7I3BUA22BNARQQ) context deadline exceeded ======LOGS====== 914003723 (300h58m34.376096196s) [upload=264873]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 02:29:49.215144679 (300h58m34.677237162s) [seg_pos={0 0}, upload=264873]: Begin upload segment... Oct 6 02:29:49.215212839 (300h58m34.677305312s) [seg_pos={0 0}, upload=264873]: Done begin upload segment. Oct 6 02:29:49.215241294 (300h58m34.677333777s) [noise=true , address=145.79.227.7:4294 , piece_id=TLZVEBMQCCWINXB4AVRKHWKZ7WULCGUC6E5GGDJQLASKWNEHSJPA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.215249739 (300h58m34.677342232s) [noise=true , address=45.91.116.4:4937 , piece_id=CHJRUVVLWAR7627JOLKL2QHAK7EHW72ZTGE3BTOAAM4ZL5AZL3KQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.215349820 (300h58m34.677442303s) [noise=true , address=77.90.31.18:4805 , piece_id=LJKUWHGMINU2P7JTS2QN5VP25HPZBKRMFXRDBSO5OTVYTS47SB4A, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.216239644 (300h58m34.678332127s) [noise=true , address=45.91.116.5:4298 , piece_id=VTE4WIBLUWV74XMSSPP37CEAWPRQHA3GFV2RJRQYGRQM5JGF3H3Q, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.218583898 (300h58m34.680676392s) [noise=true , address=45.91.116.10:4082 , piece_id=NX5W46AGWBSZ7EM2LZHZDPE6MEPSLZMNVFZEE2LQ5F5LV7YSQRKQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.219326442 (300h58m34.681418925s) [noise=true , address=45.91.116.6:4974 , piece_id=DBXKLTPEBNF5GVB4PC26BY4EPXOWHU2OWSLWGSHS52S6CDUZIMOA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.222198982 (300h58m34.684291465s) [noise=true , address=145.79.227.17:4972 , piece_id=I54FS5Q4CMMM46S6EP3C5MEUAJCVNWX23OHTBMNQNKP4TRERVZ7A, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.222520114 (300h58m34.684612597s) [noise=true , address=145.79.227.4:4185 , piece_id=ONU6YDQFAMJPRG3ZF45M7KUX5EGXZVQEGCTOJK3VGE2JXXLHCBAQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.225940076 (300h58m34.688032569s) [noise=true , address=77.90.31.13:4628 , piece_id=52Q5TEMXLZHLWJ5QWRFHGYWW6RCL2Y57AY6K2J7I3BUA22BNARQQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.227928832 (300h58m34.690021305s) [noise=true , address=145.79.227.16:4910 , piece_id=T254A3S7TTJEBOIBDDCS34DU7HV25TXDTNZWF2OGE4KRQ7YEA5ZA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.227998645 (300h58m34.690091128s) [noise=true , address=45.91.116.14:4190 , piece_id=OS3T2GGABRXYK6HNEXK4C5F5HTJFDEP7SHA5QV2R3SCMXDE2TD5A, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.231020309 (300h58m34.693112792s) [noise=true , address=145.79.227.18:4007 , piece_id=6PZE3N3QHDGV5K5HULQE5UJXWWG7RAIACP2OVEZVIOTTCWVNQO6Q, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.232908575 (300h58m34.695001068s) [noise=true , address=45.91.116.10:4081 , piece_id=PRCQUWMDPZTR2CSWMOODBECZOLAVJ4OPOUTGQHCPTWXUDQGY2MBQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.233037220 (300h58m34.695129693s) [noise=true , address=145.79.227.19:4079 , piece_id=2VBVLN2SNA6WUICKMNMNM7URPVUGWPB5SVZ335XZ7HIAUM3FY5VA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.234302058 (300h58m34.696394531s) [noise=true , address=45.91.116.7:4010 , piece_id=S5NHKRILOZ47C4NXH4UXBG34ZF6IP2XHBR5JDDBNCBOZK26NALIA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.235631800 (300h58m34.697724283s) [noise=true , address=77.90.31.15:4698 , piece_id=NOCO6KPTUBBKYCITDRFBKA3H5EJTKMDAO4AFQFFXC4PPH3D46X2A, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:50.988532254 (300h58m36.450624747s) [upload=264873]: Next returned nil segment Oct 6 02:29:50.988555589 (300h58m36.450648082s) [upload=264873]: Waiting for error group managing segments... Oct 6 02:29:51.282838267 (300h58m36.74493075s) [noise=true , address=45.91.116.4:4937 , piece_id=CHJRUVVLWAR7627JOLKL2QHAK7EHW72ZTGE3BTOAAM4ZL5AZL3KQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.318573943 (300h58m36.780666416s) [noise=true , address=77.90.31.18:4805 , piece_id=LJKUWHGMINU2P7JTS2QN5VP25HPZBKRMFXRDBSO5OTVYTS47SB4A, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.322112521 (300h58m36.784205004s) [noise=true , address=77.90.31.15:4698 , piece_id=NOCO6KPTUBBKYCITDRFBKA3H5EJTKMDAO4AFQFFXC4PPH3D46X2A, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.350536624 (300h58m36.812629108s) [noise=true , address=45.91.116.6:4974 , piece_id=DBXKLTPEBNF5GVB4PC26BY4EPXOWHU2OWSLWGSHS52S6CDUZIMOA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.362171679 (300h58m36.824264162s) [noise=true , address=45.91.116.10:4081 , piece_id=PRCQUWMDPZTR2CSWMOODBECZOLAVJ4OPOUTGQHCPTWXUDQGY2MBQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.366398738 (300h58m36.828491221s) [noise=true , address=45.91.116.10:4082 , piece_id=NX5W46AGWBSZ7EM2LZHZDPE6MEPSLZMNVFZEE2LQ5F5LV7YSQRKQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.375437720 (300h58m36.837530203s) [noise=true , address=45.91.116.14:4190 , piece_id=OS3T2GGABRXYK6HNEXK4C5F5HTJFDEP7SHA5QV2R3SCMXDE2TD5A, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.397600181 (300h58m36.859692664s) [noise=true , address=77.90.31.13:4628 , piece_id=52Q5TEMXLZHLWJ5QWRFHGYWW6RCL2Y57AY6K2J7I3BUA22BNARQQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.400132462 (300h58m36.862224945s) [noise=true , address=45.91.116.5:4298 , piece_id=VTE4WIBLUWV74XMSSPP37CEAWPRQHA3GFV2RJRQYGRQM5JGF3H3Q, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.412114018 (300h58m36.874206491s) [noise=true , address=45.91.116.7:4010 , piece_id=S5NHKRILOZ47C4NXH4UXBG34ZF6IP2XHBR5JDDBNCBOZK26NALIA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:30:14.302718308 (300h58m59.764810791s) [noise=true , address=145.79.227.18:4007 , piece_id=6PZE3N3QHDGV5K5HULQE5UJXWWG7RAIACP2OVEZVIOTTCWVNQO6Q, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:30:14.319938798 (300h58m59.782031281s) [noise=true , address=145.79.227.7:4294 , piece_id=TLZVEBMQCCWINXB4AVRKHWKZ7WULCGUC6E5GGDJQLASKWNEHSJPA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:30:16.104270571 (300h59m1.566363044s) [noise=true , address=145.79.227.16:4910 , piece_id=T254A3S7TTJEBOIBDDCS34DU7HV25TXDTNZWF2OGE4KRQ7YEA5ZA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:30:16.104295528 (300h59m1.566388012s) [seg_pos={0 0}, upload=264873]: Segment reached optimal threshold of 13 pieces. Oct 6 02:30:16.104442268 (300h59m1.566534751s) [noise=true , address=145.79.227.4:4185 , piece_id=ONU6YDQFAMJPRG3ZF45M7KUX5EGXZVQEGCTOJK3VGE2JXXLHCBAQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN): protocol: write tcp 188.166.225.170:34966->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:34966->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34966->145.79.227.4:4185: use of closed network connection Oct 6 02:30:16.104456395 (300h59m1.566548878s) [noise=true , address=145.79.227.4:4185 , piece_id=ONU6YDQFAMJPRG3ZF45M7KUX5EGXZVQEGCTOJK3VGE2JXXLHCBAQ, seg_pos={0 0}, upload=264873]: Long tail cancellation due to optimal threshold Oct 6 02:30:16.104598005 (300h59m1.566690478s) [noise=true , address=145.79.227.19:4079 , piece_id=2VBVLN2SNA6WUICKMNMNM7URPVUGWPB5SVZ335XZ7HIAUM3FY5VA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: write tcp 188.166.225.170:57162->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:57162->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57162->145.79.227.19:4079: use of closed network connection Oct 6 02:30:16.104604216 (300h59m1.566696699s) [noise=true , address=145.79.227.19:4079 , piece_id=2VBVLN2SNA6WUICKMNMNM7URPVUGWPB5SVZ335XZ7HIAUM3FY5VA, seg_pos={0 0}, upload=264873]: Long tail cancellation due to optimal threshold Oct 6 02:30:16.104672547 (300h59m1.56676503s) [noise=true , address=145.79.227.17:4972 , piece_id=I54FS5Q4CMMM46S6EP3C5MEUAJCVNWX23OHTBMNQNKP4TRERVZ7A, seg_pos={0 0}, upload=264873]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: write tcp 188.166.225.170:33630->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:33630->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33630->145.79.227.17:4972: use of closed network connection Oct 6 02:30:16.104678057 (300h59m1.56677053s) [noise=true , address=145.79.227.17:4972 , piece_id=I54FS5Q4CMMM46S6EP3C5MEUAJCVNWX23OHTBMNQNKP4TRERVZ7A, seg_pos={0 0}, upload=264873]: Long tail cancellation due to optimal threshold Oct 6 02:30:16.104695541 (300h59m1.566788024s) [seg_pos={0 0}, upload=264873]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 6 02:30:16.104762427 (300h59m1.56685491s) [upload=264873]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 6 02:30:16.104768159 (300h59m1.566860632s) [upload=264873]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 6 02:30:16.463894599 (300h59m1.925987092s) [upload=264873]: Done uploading segments... Oct 6 02:30:16.463937391 (300h59m1.926029874s) [upload=264873]: Upload finished. err: <nil> Duration:* 6.459268677s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 5.958285208s FAIL (2025-10-06 02:31:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, Piece ID: T254A3S7TTJEBOIBDDCS34DU7HV25TXDTNZWF2OGE4KRQ7YEA5ZA) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: OS3T2GGABRXYK6HNEXK4C5F5HTJFDEP7SHA5QV2R3SCMXDE2TD5A) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: S5NHKRILOZ47C4NXH4UXBG34ZF6IP2XHBR5JDDBNCBOZK26NALIA) context deadline exceeded; piecestore: (Node ID: 12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, Piece ID: CHJRUVVLWAR7627JOLKL2QHAK7EHW72ZTGE3BTOAAM4ZL5AZL3KQ) context deadline exceeded; piecestore: (Node ID: 12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, Piece ID: TLZVEBMQCCWINXB4AVRKHWKZ7WULCGUC6E5GGDJQLASKWNEHSJPA) context deadline exceeded; piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: LJKUWHGMINU2P7JTS2QN5VP25HPZBKRMFXRDBSO5OTVYTS47SB4A) context deadline exceeded; piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: VTE4WIBLUWV74XMSSPP37CEAWPRQHA3GFV2RJRQYGRQM5JGF3H3Q) context deadline exceeded; piecestore: (Node ID: 1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, Piece ID: 52Q5TEMXLZHLWJ5QWRFHGYWW6RCL2Y57AY6K2J7I3BUA22BNARQQ) context deadline exceeded ======LOGS====== 914003723 (300h58m34.376096196s) [upload=264873]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 02:29:49.215144679 (300h58m34.677237162s) [seg_pos={0 0}, upload=264873]: Begin upload segment... Oct 6 02:29:49.215212839 (300h58m34.677305312s) [seg_pos={0 0}, upload=264873]: Done begin upload segment. Oct 6 02:29:49.215241294 (300h58m34.677333777s) [noise=true , address=145.79.227.7:4294 , piece_id=TLZVEBMQCCWINXB4AVRKHWKZ7WULCGUC6E5GGDJQLASKWNEHSJPA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.215249739 (300h58m34.677342232s) [noise=true , address=45.91.116.4:4937 , piece_id=CHJRUVVLWAR7627JOLKL2QHAK7EHW72ZTGE3BTOAAM4ZL5AZL3KQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.215349820 (300h58m34.677442303s) [noise=true , address=77.90.31.18:4805 , piece_id=LJKUWHGMINU2P7JTS2QN5VP25HPZBKRMFXRDBSO5OTVYTS47SB4A, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.216239644 (300h58m34.678332127s) [noise=true , address=45.91.116.5:4298 , piece_id=VTE4WIBLUWV74XMSSPP37CEAWPRQHA3GFV2RJRQYGRQM5JGF3H3Q, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.218583898 (300h58m34.680676392s) [noise=true , address=45.91.116.10:4082 , piece_id=NX5W46AGWBSZ7EM2LZHZDPE6MEPSLZMNVFZEE2LQ5F5LV7YSQRKQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.219326442 (300h58m34.681418925s) [noise=true , address=45.91.116.6:4974 , piece_id=DBXKLTPEBNF5GVB4PC26BY4EPXOWHU2OWSLWGSHS52S6CDUZIMOA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.222198982 (300h58m34.684291465s) [noise=true , address=145.79.227.17:4972 , piece_id=I54FS5Q4CMMM46S6EP3C5MEUAJCVNWX23OHTBMNQNKP4TRERVZ7A, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.222520114 (300h58m34.684612597s) [noise=true , address=145.79.227.4:4185 , piece_id=ONU6YDQFAMJPRG3ZF45M7KUX5EGXZVQEGCTOJK3VGE2JXXLHCBAQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.225940076 (300h58m34.688032569s) [noise=true , address=77.90.31.13:4628 , piece_id=52Q5TEMXLZHLWJ5QWRFHGYWW6RCL2Y57AY6K2J7I3BUA22BNARQQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.227928832 (300h58m34.690021305s) [noise=true , address=145.79.227.16:4910 , piece_id=T254A3S7TTJEBOIBDDCS34DU7HV25TXDTNZWF2OGE4KRQ7YEA5ZA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.227998645 (300h58m34.690091128s) [noise=true , address=45.91.116.14:4190 , piece_id=OS3T2GGABRXYK6HNEXK4C5F5HTJFDEP7SHA5QV2R3SCMXDE2TD5A, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.231020309 (300h58m34.693112792s) [noise=true , address=145.79.227.18:4007 , piece_id=6PZE3N3QHDGV5K5HULQE5UJXWWG7RAIACP2OVEZVIOTTCWVNQO6Q, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.232908575 (300h58m34.695001068s) [noise=true , address=45.91.116.10:4081 , piece_id=PRCQUWMDPZTR2CSWMOODBECZOLAVJ4OPOUTGQHCPTWXUDQGY2MBQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.233037220 (300h58m34.695129693s) [noise=true , address=145.79.227.19:4079 , piece_id=2VBVLN2SNA6WUICKMNMNM7URPVUGWPB5SVZ335XZ7HIAUM3FY5VA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.234302058 (300h58m34.696394531s) [noise=true , address=45.91.116.7:4010 , piece_id=S5NHKRILOZ47C4NXH4UXBG34ZF6IP2XHBR5JDDBNCBOZK26NALIA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.235631800 (300h58m34.697724283s) [noise=true , address=77.90.31.15:4698 , piece_id=NOCO6KPTUBBKYCITDRFBKA3H5EJTKMDAO4AFQFFXC4PPH3D46X2A, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:50.988532254 (300h58m36.450624747s) [upload=264873]: Next returned nil segment Oct 6 02:29:50.988555589 (300h58m36.450648082s) [upload=264873]: Waiting for error group managing segments... Oct 6 02:29:51.282838267 (300h58m36.74493075s) [noise=true , address=45.91.116.4:4937 , piece_id=CHJRUVVLWAR7627JOLKL2QHAK7EHW72ZTGE3BTOAAM4ZL5AZL3KQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.318573943 (300h58m36.780666416s) [noise=true , address=77.90.31.18:4805 , piece_id=LJKUWHGMINU2P7JTS2QN5VP25HPZBKRMFXRDBSO5OTVYTS47SB4A, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.322112521 (300h58m36.784205004s) [noise=true , address=77.90.31.15:4698 , piece_id=NOCO6KPTUBBKYCITDRFBKA3H5EJTKMDAO4AFQFFXC4PPH3D46X2A, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.350536624 (300h58m36.812629108s) [noise=true , address=45.91.116.6:4974 , piece_id=DBXKLTPEBNF5GVB4PC26BY4EPXOWHU2OWSLWGSHS52S6CDUZIMOA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.362171679 (300h58m36.824264162s) [noise=true , address=45.91.116.10:4081 , piece_id=PRCQUWMDPZTR2CSWMOODBECZOLAVJ4OPOUTGQHCPTWXUDQGY2MBQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.366398738 (300h58m36.828491221s) [noise=true , address=45.91.116.10:4082 , piece_id=NX5W46AGWBSZ7EM2LZHZDPE6MEPSLZMNVFZEE2LQ5F5LV7YSQRKQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.375437720 (300h58m36.837530203s) [noise=true , address=45.91.116.14:4190 , piece_id=OS3T2GGABRXYK6HNEXK4C5F5HTJFDEP7SHA5QV2R3SCMXDE2TD5A, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.397600181 (300h58m36.859692664s) [noise=true , address=77.90.31.13:4628 , piece_id=52Q5TEMXLZHLWJ5QWRFHGYWW6RCL2Y57AY6K2J7I3BUA22BNARQQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.400132462 (300h58m36.862224945s) [noise=true , address=45.91.116.5:4298 , piece_id=VTE4WIBLUWV74XMSSPP37CEAWPRQHA3GFV2RJRQYGRQM5JGF3H3Q, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.412114018 (300h58m36.874206491s) [noise=true , address=45.91.116.7:4010 , piece_id=S5NHKRILOZ47C4NXH4UXBG34ZF6IP2XHBR5JDDBNCBOZK26NALIA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:30:14.302718308 (300h58m59.764810791s) [noise=true , address=145.79.227.18:4007 , piece_id=6PZE3N3QHDGV5K5HULQE5UJXWWG7RAIACP2OVEZVIOTTCWVNQO6Q, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:30:14.319938798 (300h58m59.782031281s) [noise=true , address=145.79.227.7:4294 , piece_id=TLZVEBMQCCWINXB4AVRKHWKZ7WULCGUC6E5GGDJQLASKWNEHSJPA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:30:16.104270571 (300h59m1.566363044s) [noise=true , address=145.79.227.16:4910 , piece_id=T254A3S7TTJEBOIBDDCS34DU7HV25TXDTNZWF2OGE4KRQ7YEA5ZA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:30:16.104295528 (300h59m1.566388012s) [seg_pos={0 0}, upload=264873]: Segment reached optimal threshold of 13 pieces. Oct 6 02:30:16.104442268 (300h59m1.566534751s) [noise=true , address=145.79.227.4:4185 , piece_id=ONU6YDQFAMJPRG3ZF45M7KUX5EGXZVQEGCTOJK3VGE2JXXLHCBAQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN): protocol: write tcp 188.166.225.170:34966->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:34966->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34966->145.79.227.4:4185: use of closed network connection Oct 6 02:30:16.104456395 (300h59m1.566548878s) [noise=true , address=145.79.227.4:4185 , piece_id=ONU6YDQFAMJPRG3ZF45M7KUX5EGXZVQEGCTOJK3VGE2JXXLHCBAQ, seg_pos={0 0}, upload=264873]: Long tail cancellation due to optimal threshold Oct 6 02:30:16.104598005 (300h59m1.566690478s) [noise=true , address=145.79.227.19:4079 , piece_id=2VBVLN2SNA6WUICKMNMNM7URPVUGWPB5SVZ335XZ7HIAUM3FY5VA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: write tcp 188.166.225.170:57162->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:57162->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57162->145.79.227.19:4079: use of closed network connection Oct 6 02:30:16.104604216 (300h59m1.566696699s) [noise=true , address=145.79.227.19:4079 , piece_id=2VBVLN2SNA6WUICKMNMNM7URPVUGWPB5SVZ335XZ7HIAUM3FY5VA, seg_pos={0 0}, upload=264873]: Long tail cancellation due to optimal threshold Oct 6 02:30:16.104672547 (300h59m1.56676503s) [noise=true , address=145.79.227.17:4972 , piece_id=I54FS5Q4CMMM46S6EP3C5MEUAJCVNWX23OHTBMNQNKP4TRERVZ7A, seg_pos={0 0}, upload=264873]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: write tcp 188.166.225.170:33630->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:33630->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33630->145.79.227.17:4972: use of closed network connection Oct 6 02:30:16.104678057 (300h59m1.56677053s) [noise=true , address=145.79.227.17:4972 , piece_id=I54FS5Q4CMMM46S6EP3C5MEUAJCVNWX23OHTBMNQNKP4TRERVZ7A, seg_pos={0 0}, upload=264873]: Long tail cancellation due to optimal threshold Oct 6 02:30:16.104695541 (300h59m1.566788024s) [seg_pos={0 0}, upload=264873]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 6 02:30:16.104762427 (300h59m1.56685491s) [upload=264873]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 6 02:30:16.104768159 (300h59m1.566860632s) [upload=264873]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 6 02:30:16.463894599 (300h59m1.925987092s) [upload=264873]: Done uploading segments... Oct 6 02:30:16.463937391 (300h59m1.926029874s) [upload=264873]: Upload finished. err: <nil> Duration:* 5.958319434s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 5.901601237s FAIL (2025-10-06 02:30:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, Piece ID: T254A3S7TTJEBOIBDDCS34DU7HV25TXDTNZWF2OGE4KRQ7YEA5ZA) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: OS3T2GGABRXYK6HNEXK4C5F5HTJFDEP7SHA5QV2R3SCMXDE2TD5A) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: S5NHKRILOZ47C4NXH4UXBG34ZF6IP2XHBR5JDDBNCBOZK26NALIA) context deadline exceeded; piecestore: (Node ID: 12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, Piece ID: CHJRUVVLWAR7627JOLKL2QHAK7EHW72ZTGE3BTOAAM4ZL5AZL3KQ) context deadline exceeded; piecestore: (Node ID: 12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, Piece ID: TLZVEBMQCCWINXB4AVRKHWKZ7WULCGUC6E5GGDJQLASKWNEHSJPA) context deadline exceeded; piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: LJKUWHGMINU2P7JTS2QN5VP25HPZBKRMFXRDBSO5OTVYTS47SB4A) context deadline exceeded; piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: VTE4WIBLUWV74XMSSPP37CEAWPRQHA3GFV2RJRQYGRQM5JGF3H3Q) context deadline exceeded; piecestore: (Node ID: 1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, Piece ID: 52Q5TEMXLZHLWJ5QWRFHGYWW6RCL2Y57AY6K2J7I3BUA22BNARQQ) context deadline exceeded ======LOGS====== 914003723 (300h58m34.376096196s) [upload=264873]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 02:29:49.215144679 (300h58m34.677237162s) [seg_pos={0 0}, upload=264873]: Begin upload segment... Oct 6 02:29:49.215212839 (300h58m34.677305312s) [seg_pos={0 0}, upload=264873]: Done begin upload segment. Oct 6 02:29:49.215241294 (300h58m34.677333777s) [noise=true , address=145.79.227.7:4294 , piece_id=TLZVEBMQCCWINXB4AVRKHWKZ7WULCGUC6E5GGDJQLASKWNEHSJPA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.215249739 (300h58m34.677342232s) [noise=true , address=45.91.116.4:4937 , piece_id=CHJRUVVLWAR7627JOLKL2QHAK7EHW72ZTGE3BTOAAM4ZL5AZL3KQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.215349820 (300h58m34.677442303s) [noise=true , address=77.90.31.18:4805 , piece_id=LJKUWHGMINU2P7JTS2QN5VP25HPZBKRMFXRDBSO5OTVYTS47SB4A, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.216239644 (300h58m34.678332127s) [noise=true , address=45.91.116.5:4298 , piece_id=VTE4WIBLUWV74XMSSPP37CEAWPRQHA3GFV2RJRQYGRQM5JGF3H3Q, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.218583898 (300h58m34.680676392s) [noise=true , address=45.91.116.10:4082 , piece_id=NX5W46AGWBSZ7EM2LZHZDPE6MEPSLZMNVFZEE2LQ5F5LV7YSQRKQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.219326442 (300h58m34.681418925s) [noise=true , address=45.91.116.6:4974 , piece_id=DBXKLTPEBNF5GVB4PC26BY4EPXOWHU2OWSLWGSHS52S6CDUZIMOA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.222198982 (300h58m34.684291465s) [noise=true , address=145.79.227.17:4972 , piece_id=I54FS5Q4CMMM46S6EP3C5MEUAJCVNWX23OHTBMNQNKP4TRERVZ7A, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.222520114 (300h58m34.684612597s) [noise=true , address=145.79.227.4:4185 , piece_id=ONU6YDQFAMJPRG3ZF45M7KUX5EGXZVQEGCTOJK3VGE2JXXLHCBAQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.225940076 (300h58m34.688032569s) [noise=true , address=77.90.31.13:4628 , piece_id=52Q5TEMXLZHLWJ5QWRFHGYWW6RCL2Y57AY6K2J7I3BUA22BNARQQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.227928832 (300h58m34.690021305s) [noise=true , address=145.79.227.16:4910 , piece_id=T254A3S7TTJEBOIBDDCS34DU7HV25TXDTNZWF2OGE4KRQ7YEA5ZA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.227998645 (300h58m34.690091128s) [noise=true , address=45.91.116.14:4190 , piece_id=OS3T2GGABRXYK6HNEXK4C5F5HTJFDEP7SHA5QV2R3SCMXDE2TD5A, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.231020309 (300h58m34.693112792s) [noise=true , address=145.79.227.18:4007 , piece_id=6PZE3N3QHDGV5K5HULQE5UJXWWG7RAIACP2OVEZVIOTTCWVNQO6Q, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.232908575 (300h58m34.695001068s) [noise=true , address=45.91.116.10:4081 , piece_id=PRCQUWMDPZTR2CSWMOODBECZOLAVJ4OPOUTGQHCPTWXUDQGY2MBQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.233037220 (300h58m34.695129693s) [noise=true , address=145.79.227.19:4079 , piece_id=2VBVLN2SNA6WUICKMNMNM7URPVUGWPB5SVZ335XZ7HIAUM3FY5VA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.234302058 (300h58m34.696394531s) [noise=true , address=45.91.116.7:4010 , piece_id=S5NHKRILOZ47C4NXH4UXBG34ZF6IP2XHBR5JDDBNCBOZK26NALIA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.235631800 (300h58m34.697724283s) [noise=true , address=77.90.31.15:4698 , piece_id=NOCO6KPTUBBKYCITDRFBKA3H5EJTKMDAO4AFQFFXC4PPH3D46X2A, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:50.988532254 (300h58m36.450624747s) [upload=264873]: Next returned nil segment Oct 6 02:29:50.988555589 (300h58m36.450648082s) [upload=264873]: Waiting for error group managing segments... Oct 6 02:29:51.282838267 (300h58m36.74493075s) [noise=true , address=45.91.116.4:4937 , piece_id=CHJRUVVLWAR7627JOLKL2QHAK7EHW72ZTGE3BTOAAM4ZL5AZL3KQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.318573943 (300h58m36.780666416s) [noise=true , address=77.90.31.18:4805 , piece_id=LJKUWHGMINU2P7JTS2QN5VP25HPZBKRMFXRDBSO5OTVYTS47SB4A, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.322112521 (300h58m36.784205004s) [noise=true , address=77.90.31.15:4698 , piece_id=NOCO6KPTUBBKYCITDRFBKA3H5EJTKMDAO4AFQFFXC4PPH3D46X2A, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.350536624 (300h58m36.812629108s) [noise=true , address=45.91.116.6:4974 , piece_id=DBXKLTPEBNF5GVB4PC26BY4EPXOWHU2OWSLWGSHS52S6CDUZIMOA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.362171679 (300h58m36.824264162s) [noise=true , address=45.91.116.10:4081 , piece_id=PRCQUWMDPZTR2CSWMOODBECZOLAVJ4OPOUTGQHCPTWXUDQGY2MBQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.366398738 (300h58m36.828491221s) [noise=true , address=45.91.116.10:4082 , piece_id=NX5W46AGWBSZ7EM2LZHZDPE6MEPSLZMNVFZEE2LQ5F5LV7YSQRKQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.375437720 (300h58m36.837530203s) [noise=true , address=45.91.116.14:4190 , piece_id=OS3T2GGABRXYK6HNEXK4C5F5HTJFDEP7SHA5QV2R3SCMXDE2TD5A, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.397600181 (300h58m36.859692664s) [noise=true , address=77.90.31.13:4628 , piece_id=52Q5TEMXLZHLWJ5QWRFHGYWW6RCL2Y57AY6K2J7I3BUA22BNARQQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.400132462 (300h58m36.862224945s) [noise=true , address=45.91.116.5:4298 , piece_id=VTE4WIBLUWV74XMSSPP37CEAWPRQHA3GFV2RJRQYGRQM5JGF3H3Q, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.412114018 (300h58m36.874206491s) [noise=true , address=45.91.116.7:4010 , piece_id=S5NHKRILOZ47C4NXH4UXBG34ZF6IP2XHBR5JDDBNCBOZK26NALIA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:30:14.302718308 (300h58m59.764810791s) [noise=true , address=145.79.227.18:4007 , piece_id=6PZE3N3QHDGV5K5HULQE5UJXWWG7RAIACP2OVEZVIOTTCWVNQO6Q, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:30:14.319938798 (300h58m59.782031281s) [noise=true , address=145.79.227.7:4294 , piece_id=TLZVEBMQCCWINXB4AVRKHWKZ7WULCGUC6E5GGDJQLASKWNEHSJPA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:30:16.104270571 (300h59m1.566363044s) [noise=true , address=145.79.227.16:4910 , piece_id=T254A3S7TTJEBOIBDDCS34DU7HV25TXDTNZWF2OGE4KRQ7YEA5ZA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:30:16.104295528 (300h59m1.566388012s) [seg_pos={0 0}, upload=264873]: Segment reached optimal threshold of 13 pieces. Oct 6 02:30:16.104442268 (300h59m1.566534751s) [noise=true , address=145.79.227.4:4185 , piece_id=ONU6YDQFAMJPRG3ZF45M7KUX5EGXZVQEGCTOJK3VGE2JXXLHCBAQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN): protocol: write tcp 188.166.225.170:34966->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:34966->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34966->145.79.227.4:4185: use of closed network connection Oct 6 02:30:16.104456395 (300h59m1.566548878s) [noise=true , address=145.79.227.4:4185 , piece_id=ONU6YDQFAMJPRG3ZF45M7KUX5EGXZVQEGCTOJK3VGE2JXXLHCBAQ, seg_pos={0 0}, upload=264873]: Long tail cancellation due to optimal threshold Oct 6 02:30:16.104598005 (300h59m1.566690478s) [noise=true , address=145.79.227.19:4079 , piece_id=2VBVLN2SNA6WUICKMNMNM7URPVUGWPB5SVZ335XZ7HIAUM3FY5VA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: write tcp 188.166.225.170:57162->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:57162->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57162->145.79.227.19:4079: use of closed network connection Oct 6 02:30:16.104604216 (300h59m1.566696699s) [noise=true , address=145.79.227.19:4079 , piece_id=2VBVLN2SNA6WUICKMNMNM7URPVUGWPB5SVZ335XZ7HIAUM3FY5VA, seg_pos={0 0}, upload=264873]: Long tail cancellation due to optimal threshold Oct 6 02:30:16.104672547 (300h59m1.56676503s) [noise=true , address=145.79.227.17:4972 , piece_id=I54FS5Q4CMMM46S6EP3C5MEUAJCVNWX23OHTBMNQNKP4TRERVZ7A, seg_pos={0 0}, upload=264873]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: write tcp 188.166.225.170:33630->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:33630->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33630->145.79.227.17:4972: use of closed network connection Oct 6 02:30:16.104678057 (300h59m1.56677053s) [noise=true , address=145.79.227.17:4972 , piece_id=I54FS5Q4CMMM46S6EP3C5MEUAJCVNWX23OHTBMNQNKP4TRERVZ7A, seg_pos={0 0}, upload=264873]: Long tail cancellation due to optimal threshold Oct 6 02:30:16.104695541 (300h59m1.566788024s) [seg_pos={0 0}, upload=264873]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 6 02:30:16.104762427 (300h59m1.56685491s) [upload=264873]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 6 02:30:16.104768159 (300h59m1.566860632s) [upload=264873]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 6 02:30:16.463894599 (300h59m1.925987092s) [upload=264873]: Done uploading segments... Oct 6 02:30:16.463937391 (300h59m1.926029874s) [upload=264873]: Upload finished. err: <nil> Duration:* 5.901647644s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.003069733s FAIL (2025-10-06 02:30:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, Piece ID: T254A3S7TTJEBOIBDDCS34DU7HV25TXDTNZWF2OGE4KRQ7YEA5ZA) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: OS3T2GGABRXYK6HNEXK4C5F5HTJFDEP7SHA5QV2R3SCMXDE2TD5A) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: S5NHKRILOZ47C4NXH4UXBG34ZF6IP2XHBR5JDDBNCBOZK26NALIA) context deadline exceeded; piecestore: (Node ID: 12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, Piece ID: CHJRUVVLWAR7627JOLKL2QHAK7EHW72ZTGE3BTOAAM4ZL5AZL3KQ) context deadline exceeded; piecestore: (Node ID: 12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, Piece ID: TLZVEBMQCCWINXB4AVRKHWKZ7WULCGUC6E5GGDJQLASKWNEHSJPA) context deadline exceeded; piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: LJKUWHGMINU2P7JTS2QN5VP25HPZBKRMFXRDBSO5OTVYTS47SB4A) context deadline exceeded; piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: VTE4WIBLUWV74XMSSPP37CEAWPRQHA3GFV2RJRQYGRQM5JGF3H3Q) context deadline exceeded; piecestore: (Node ID: 1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, Piece ID: 52Q5TEMXLZHLWJ5QWRFHGYWW6RCL2Y57AY6K2J7I3BUA22BNARQQ) context deadline exceeded ======LOGS====== 914003723 (300h58m34.376096196s) [upload=264873]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 02:29:49.215144679 (300h58m34.677237162s) [seg_pos={0 0}, upload=264873]: Begin upload segment... Oct 6 02:29:49.215212839 (300h58m34.677305312s) [seg_pos={0 0}, upload=264873]: Done begin upload segment. Oct 6 02:29:49.215241294 (300h58m34.677333777s) [noise=true , address=145.79.227.7:4294 , piece_id=TLZVEBMQCCWINXB4AVRKHWKZ7WULCGUC6E5GGDJQLASKWNEHSJPA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.215249739 (300h58m34.677342232s) [noise=true , address=45.91.116.4:4937 , piece_id=CHJRUVVLWAR7627JOLKL2QHAK7EHW72ZTGE3BTOAAM4ZL5AZL3KQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.215349820 (300h58m34.677442303s) [noise=true , address=77.90.31.18:4805 , piece_id=LJKUWHGMINU2P7JTS2QN5VP25HPZBKRMFXRDBSO5OTVYTS47SB4A, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.216239644 (300h58m34.678332127s) [noise=true , address=45.91.116.5:4298 , piece_id=VTE4WIBLUWV74XMSSPP37CEAWPRQHA3GFV2RJRQYGRQM5JGF3H3Q, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.218583898 (300h58m34.680676392s) [noise=true , address=45.91.116.10:4082 , piece_id=NX5W46AGWBSZ7EM2LZHZDPE6MEPSLZMNVFZEE2LQ5F5LV7YSQRKQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.219326442 (300h58m34.681418925s) [noise=true , address=45.91.116.6:4974 , piece_id=DBXKLTPEBNF5GVB4PC26BY4EPXOWHU2OWSLWGSHS52S6CDUZIMOA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.222198982 (300h58m34.684291465s) [noise=true , address=145.79.227.17:4972 , piece_id=I54FS5Q4CMMM46S6EP3C5MEUAJCVNWX23OHTBMNQNKP4TRERVZ7A, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.222520114 (300h58m34.684612597s) [noise=true , address=145.79.227.4:4185 , piece_id=ONU6YDQFAMJPRG3ZF45M7KUX5EGXZVQEGCTOJK3VGE2JXXLHCBAQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.225940076 (300h58m34.688032569s) [noise=true , address=77.90.31.13:4628 , piece_id=52Q5TEMXLZHLWJ5QWRFHGYWW6RCL2Y57AY6K2J7I3BUA22BNARQQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.227928832 (300h58m34.690021305s) [noise=true , address=145.79.227.16:4910 , piece_id=T254A3S7TTJEBOIBDDCS34DU7HV25TXDTNZWF2OGE4KRQ7YEA5ZA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.227998645 (300h58m34.690091128s) [noise=true , address=45.91.116.14:4190 , piece_id=OS3T2GGABRXYK6HNEXK4C5F5HTJFDEP7SHA5QV2R3SCMXDE2TD5A, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.231020309 (300h58m34.693112792s) [noise=true , address=145.79.227.18:4007 , piece_id=6PZE3N3QHDGV5K5HULQE5UJXWWG7RAIACP2OVEZVIOTTCWVNQO6Q, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.232908575 (300h58m34.695001068s) [noise=true , address=45.91.116.10:4081 , piece_id=PRCQUWMDPZTR2CSWMOODBECZOLAVJ4OPOUTGQHCPTWXUDQGY2MBQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.233037220 (300h58m34.695129693s) [noise=true , address=145.79.227.19:4079 , piece_id=2VBVLN2SNA6WUICKMNMNM7URPVUGWPB5SVZ335XZ7HIAUM3FY5VA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.234302058 (300h58m34.696394531s) [noise=true , address=45.91.116.7:4010 , piece_id=S5NHKRILOZ47C4NXH4UXBG34ZF6IP2XHBR5JDDBNCBOZK26NALIA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.235631800 (300h58m34.697724283s) [noise=true , address=77.90.31.15:4698 , piece_id=NOCO6KPTUBBKYCITDRFBKA3H5EJTKMDAO4AFQFFXC4PPH3D46X2A, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:50.988532254 (300h58m36.450624747s) [upload=264873]: Next returned nil segment Oct 6 02:29:50.988555589 (300h58m36.450648082s) [upload=264873]: Waiting for error group managing segments... Oct 6 02:29:51.282838267 (300h58m36.74493075s) [noise=true , address=45.91.116.4:4937 , piece_id=CHJRUVVLWAR7627JOLKL2QHAK7EHW72ZTGE3BTOAAM4ZL5AZL3KQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.318573943 (300h58m36.780666416s) [noise=true , address=77.90.31.18:4805 , piece_id=LJKUWHGMINU2P7JTS2QN5VP25HPZBKRMFXRDBSO5OTVYTS47SB4A, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.322112521 (300h58m36.784205004s) [noise=true , address=77.90.31.15:4698 , piece_id=NOCO6KPTUBBKYCITDRFBKA3H5EJTKMDAO4AFQFFXC4PPH3D46X2A, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.350536624 (300h58m36.812629108s) [noise=true , address=45.91.116.6:4974 , piece_id=DBXKLTPEBNF5GVB4PC26BY4EPXOWHU2OWSLWGSHS52S6CDUZIMOA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.362171679 (300h58m36.824264162s) [noise=true , address=45.91.116.10:4081 , piece_id=PRCQUWMDPZTR2CSWMOODBECZOLAVJ4OPOUTGQHCPTWXUDQGY2MBQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.366398738 (300h58m36.828491221s) [noise=true , address=45.91.116.10:4082 , piece_id=NX5W46AGWBSZ7EM2LZHZDPE6MEPSLZMNVFZEE2LQ5F5LV7YSQRKQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.375437720 (300h58m36.837530203s) [noise=true , address=45.91.116.14:4190 , piece_id=OS3T2GGABRXYK6HNEXK4C5F5HTJFDEP7SHA5QV2R3SCMXDE2TD5A, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.397600181 (300h58m36.859692664s) [noise=true , address=77.90.31.13:4628 , piece_id=52Q5TEMXLZHLWJ5QWRFHGYWW6RCL2Y57AY6K2J7I3BUA22BNARQQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.400132462 (300h58m36.862224945s) [noise=true , address=45.91.116.5:4298 , piece_id=VTE4WIBLUWV74XMSSPP37CEAWPRQHA3GFV2RJRQYGRQM5JGF3H3Q, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.412114018 (300h58m36.874206491s) [noise=true , address=45.91.116.7:4010 , piece_id=S5NHKRILOZ47C4NXH4UXBG34ZF6IP2XHBR5JDDBNCBOZK26NALIA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:30:14.302718308 (300h58m59.764810791s) [noise=true , address=145.79.227.18:4007 , piece_id=6PZE3N3QHDGV5K5HULQE5UJXWWG7RAIACP2OVEZVIOTTCWVNQO6Q, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:30:14.319938798 (300h58m59.782031281s) [noise=true , address=145.79.227.7:4294 , piece_id=TLZVEBMQCCWINXB4AVRKHWKZ7WULCGUC6E5GGDJQLASKWNEHSJPA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:30:16.104270571 (300h59m1.566363044s) [noise=true , address=145.79.227.16:4910 , piece_id=T254A3S7TTJEBOIBDDCS34DU7HV25TXDTNZWF2OGE4KRQ7YEA5ZA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:30:16.104295528 (300h59m1.566388012s) [seg_pos={0 0}, upload=264873]: Segment reached optimal threshold of 13 pieces. Oct 6 02:30:16.104442268 (300h59m1.566534751s) [noise=true , address=145.79.227.4:4185 , piece_id=ONU6YDQFAMJPRG3ZF45M7KUX5EGXZVQEGCTOJK3VGE2JXXLHCBAQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN): protocol: write tcp 188.166.225.170:34966->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:34966->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34966->145.79.227.4:4185: use of closed network connection Oct 6 02:30:16.104456395 (300h59m1.566548878s) [noise=true , address=145.79.227.4:4185 , piece_id=ONU6YDQFAMJPRG3ZF45M7KUX5EGXZVQEGCTOJK3VGE2JXXLHCBAQ, seg_pos={0 0}, upload=264873]: Long tail cancellation due to optimal threshold Oct 6 02:30:16.104598005 (300h59m1.566690478s) [noise=true , address=145.79.227.19:4079 , piece_id=2VBVLN2SNA6WUICKMNMNM7URPVUGWPB5SVZ335XZ7HIAUM3FY5VA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: write tcp 188.166.225.170:57162->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:57162->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57162->145.79.227.19:4079: use of closed network connection Oct 6 02:30:16.104604216 (300h59m1.566696699s) [noise=true , address=145.79.227.19:4079 , piece_id=2VBVLN2SNA6WUICKMNMNM7URPVUGWPB5SVZ335XZ7HIAUM3FY5VA, seg_pos={0 0}, upload=264873]: Long tail cancellation due to optimal threshold Oct 6 02:30:16.104672547 (300h59m1.56676503s) [noise=true , address=145.79.227.17:4972 , piece_id=I54FS5Q4CMMM46S6EP3C5MEUAJCVNWX23OHTBMNQNKP4TRERVZ7A, seg_pos={0 0}, upload=264873]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: write tcp 188.166.225.170:33630->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:33630->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33630->145.79.227.17:4972: use of closed network connection Oct 6 02:30:16.104678057 (300h59m1.56677053s) [noise=true , address=145.79.227.17:4972 , piece_id=I54FS5Q4CMMM46S6EP3C5MEUAJCVNWX23OHTBMNQNKP4TRERVZ7A, seg_pos={0 0}, upload=264873]: Long tail cancellation due to optimal threshold Oct 6 02:30:16.104695541 (300h59m1.566788024s) [seg_pos={0 0}, upload=264873]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 6 02:30:16.104762427 (300h59m1.56685491s) [upload=264873]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 6 02:30:16.104768159 (300h59m1.566860632s) [upload=264873]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 6 02:30:16.463894599 (300h59m1.925987092s) [upload=264873]: Done uploading segments... Oct 6 02:30:16.463937391 (300h59m1.926029874s) [upload=264873]: Upload finished. err: <nil> Duration:* 30.003094699s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 6.09091967s FAIL (2025-10-06 00:35:47 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.090949959s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 5.814295031s FAIL (2025-10-06 00:35:14 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.814322864s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 6.491867885s FAIL (2025-10-06 00:34:32 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.49189714s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 6.066306955s FAIL (2025-10-06 00:33:24 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.06633625s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 6.469054806s FAIL (2025-10-06 00:32:10 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.469084592s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 5.725649695s FAIL (2025-10-06 00:31:12 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.725681677s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 6.301602766s FAIL (2025-10-06 00:30:16 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.301623417s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 6.250106521s FAIL (2025-10-06 00:29:41 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.250132941s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 6.320235947s FAIL (2025-10-06 00:28:53 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.320262568s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 6.493583901s FAIL (2025-10-06 00:27:57 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.493611134s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 6.487946147s FAIL (2025-10-06 00:27:29 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.48797372s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 6.146034578s FAIL (2025-10-06 00:26:44 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.14606692s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 6.27898565s FAIL (2025-10-06 00:25:46 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.27902285s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 6.152112001s FAIL (2025-10-06 00:24:35 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.152139102s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 6.166560915s FAIL (2025-10-06 00:23:22 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.166588768s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 6.151872707s FAIL (2025-10-06 00:22:30 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.151898025s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 6.253606054s FAIL (2025-10-06 00:22:14 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.253642524s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 6.146386219s FAIL (2025-10-06 00:21:09 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.146441003s edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)