Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 6.266222709s FAIL (2025-10-06 03:20:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: 2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: 7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA) context deadline exceeded ======LOGS====== Oct 6 02:45:41.756337907 (301h14m27.21843039s) [upload=265104]: Starting upload Oct 6 02:45:41.756430433 (301h14m27.218522916s) [upload=265104]: Done starting upload Oct 6 02:45:41.756513701 (301h14m27.218606185s) [upload=265104]: Uploading segments... Oct 6 02:45:41.756752276 (301h14m27.218844759s) [upload=265104]: Got next segment. Inline: false Oct 6 02:45:41.756794536 (301h14m27.218887019s) [upload=265104]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 02:45:41.756800898 (301h14m27.218893381s) [upload=265104]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 02:45:42.009327267 (301h14m27.47141975s) [seg_pos={0 0}, upload=265104]: Begin upload segment... Oct 6 02:45:42.009417419 (301h14m27.471509892s) [seg_pos={0 0}, upload=265104]: Done begin upload segment. Oct 6 02:45:42.009444500 (301h14m27.471536983s) [noise=true , address=77.90.31.14:4662 , piece_id=MPVBPBB63JILLOUF3N7SRLCZ2F74LX7Z4AH7ZFG3BDNBHT4UIXTQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.009561473 (301h14m27.471653956s) [noise=true , address=145.79.227.19:4077 , piece_id=EJIAPZSBMAO77E33V6MEBOQHUXVVYKMAVP53QAZXZSYBYDFP7MMQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.010197574 (301h14m27.472290057s) [noise=true , address=145.79.227.15:4875 , piece_id=PBDTNZ2NWRZZC3VCIQCVR2WLT2AP6662I2SSDNP6BLUDKCJMRK6Q, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.010634216 (301h14m27.472726699s) [noise=true , address=145.79.227.4:4185 , piece_id=3AD745KFD67M4XFNDXMJVEU34RRBQXXE4P3HVJJRDBHK3IIXNYTA, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011073742 (301h14m27.473166225s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011505996 (301h14m27.473598479s) [noise=true , address=45.91.116.10:4081 , piece_id=FTXZ2LOHQMY74KOX6XZC3TLHWXOGDTIY3SZLSPTE7THGKNLMPXRQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011982502 (301h14m27.474074985s) [noise=true , address=45.91.116.6:4974 , piece_id=7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.012488236 (301h14m27.474580719s) [noise=true , address=145.79.227.7:4294 , piece_id=7E3B4BGLCTW6C5DM6PPDJPWGVJS5HEGYPAPXRAMX7XDSG7USPQFQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.013242472 (301h14m27.475334955s) [noise=true , address=45.91.116.13:4154 , piece_id=2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.014529382 (301h14m27.476621855s) [noise=true , address=45.91.116.6:4975 , piece_id=T3VWH5OBQ7FE66GCKHZLEDSIXFVYEU3V52HI4CPD27QL3RH75K7Q, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.014632619 (301h14m27.476725102s) [noise=true , address=45.91.116.11:4117 , piece_id=QKEYHSK5JOL65H5T2KUEAEK6VKTNCZAYXIEZTMB76PV7P5PEVILQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.015990214 (301h14m27.478082697s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.016291928 (301h14m27.478384412s) [noise=true , address=45.91.116.12:4226 , piece_id=NM7SEIKZYUHZLNVSOGYVZWMI7U7X5SOXOJDAEGY5NF3M6QQFPZSQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.017455915 (301h14m27.479548399s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.018864467 (301h14m27.48095694s) [noise=true , address=77.90.31.10:4518 , piece_id=6FGNAMVK6OAGJTZJNKTUJWGRW7MLGMJMGJKMAINWMPW7BV74OO6A, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.020222022 (301h14m27.482314495s) [noise=true , address=45.91.116.11:4118 , piece_id=VUZB3QRFST2T3HJHMEYYZ6CV7CKJTFWQGNOBDTXHGET2MBUIC4HQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:43.755608897 (301h14m29.21770138s) [upload=265104]: Next returned nil segment Oct 6 02:45:43.755631300 (301h14m29.217723773s) [upload=265104]: Waiting for error group managing segments... Oct 6 02:45:44.043930124 (301h14m29.506022597s) [noise=true , address=145.79.227.19:4077 , piece_id=EJIAPZSBMAO77E33V6MEBOQHUXVVYKMAVP53QAZXZSYBYDFP7MMQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.045501296 (301h14m29.507593769s) [noise=true , address=145.79.227.4:4185 , piece_id=3AD745KFD67M4XFNDXMJVEU34RRBQXXE4P3HVJJRDBHK3IIXNYTA, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.057881199 (301h14m29.519973672s) [noise=true , address=145.79.227.15:4875 , piece_id=PBDTNZ2NWRZZC3VCIQCVR2WLT2AP6662I2SSDNP6BLUDKCJMRK6Q, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.068726540 (301h14m29.530819023s) [noise=true , address=145.79.227.7:4294 , piece_id=7E3B4BGLCTW6C5DM6PPDJPWGVJS5HEGYPAPXRAMX7XDSG7USPQFQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.897895712 (301h14m30.359988185s) [noise=true , address=45.91.116.12:4226 , piece_id=NM7SEIKZYUHZLNVSOGYVZWMI7U7X5SOXOJDAEGY5NF3M6QQFPZSQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.252859665 (301h14m30.714952148s) [noise=true , address=77.90.31.14:4662 , piece_id=MPVBPBB63JILLOUF3N7SRLCZ2F74LX7Z4AH7ZFG3BDNBHT4UIXTQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.318952216 (301h14m30.781044699s) [noise=true , address=45.91.116.6:4974 , piece_id=7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.836439779 (301h14m31.298532262s) [noise=true , address=45.91.116.10:4081 , piece_id=FTXZ2LOHQMY74KOX6XZC3TLHWXOGDTIY3SZLSPTE7THGKNLMPXRQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.975895173 (301h14m31.437987656s) [noise=true , address=45.91.116.6:4975 , piece_id=T3VWH5OBQ7FE66GCKHZLEDSIXFVYEU3V52HI4CPD27QL3RH75K7Q, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:46.826703981 (301h14m32.288796454s) [noise=true , address=77.90.31.10:4518 , piece_id=6FGNAMVK6OAGJTZJNKTUJWGRW7MLGMJMGJKMAINWMPW7BV74OO6A, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:48.069759484 (301h14m33.531851968s) [noise=true , address=45.91.116.13:4154 , piece_id=2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:54.791582362 (301h14m40.253674835s) [noise=true , address=45.91.116.11:4118 , piece_id=VUZB3QRFST2T3HJHMEYYZ6CV7CKJTFWQGNOBDTXHGET2MBUIC4HQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:56.329163998 (301h14m41.791256481s) [noise=true , address=45.91.116.11:4117 , piece_id=QKEYHSK5JOL65H5T2KUEAEK6VKTNCZAYXIEZTMB76PV7P5PEVILQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:56.329201379 (301h14m41.791293862s) [seg_pos={0 0}, upload=265104]: Segment reached optimal threshold of 13 pieces. Oct 6 02:45:56.329362146 (301h14m41.791454629s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Oct 6 02:45:56.329371974 (301h14m41.791464457s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329546396 (301h14m41.791638869s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: write tcp 188.166.225.170:57604->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 02:45:56.329556665 (301h14m41.791649148s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329567937 (301h14m41.791660421s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: rpc: dial tcp 77.90.31.19:4841: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 02:45:56.329579870 (301h14m41.791672353s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329599808 (301h14m41.791692291s) [seg_pos={0 0}, upload=265104]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 6 02:45:56.329683116 (301h14m41.7917756s) [upload=265104]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 6 02:45:56.329690771 (301h14m41.791783254s) [upload=265104]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 6 02:45:56.636341641 (301h14m42.098434134s) [upload=265104]: Done uploading segments... Oct 6 02:45:56.636368131 (301h14m42.098460614s) [upload=265104]: Upload finished. err: <nil> Duration: 6.266259989s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 6.081426628s FAIL (2025-10-06 03:19:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: 2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: 7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA) context deadline exceeded ======LOGS====== Oct 6 02:45:41.756337907 (301h14m27.21843039s) [upload=265104]: Starting upload Oct 6 02:45:41.756430433 (301h14m27.218522916s) [upload=265104]: Done starting upload Oct 6 02:45:41.756513701 (301h14m27.218606185s) [upload=265104]: Uploading segments... Oct 6 02:45:41.756752276 (301h14m27.218844759s) [upload=265104]: Got next segment. Inline: false Oct 6 02:45:41.756794536 (301h14m27.218887019s) [upload=265104]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 02:45:41.756800898 (301h14m27.218893381s) [upload=265104]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 02:45:42.009327267 (301h14m27.47141975s) [seg_pos={0 0}, upload=265104]: Begin upload segment... Oct 6 02:45:42.009417419 (301h14m27.471509892s) [seg_pos={0 0}, upload=265104]: Done begin upload segment. Oct 6 02:45:42.009444500 (301h14m27.471536983s) [noise=true , address=77.90.31.14:4662 , piece_id=MPVBPBB63JILLOUF3N7SRLCZ2F74LX7Z4AH7ZFG3BDNBHT4UIXTQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.009561473 (301h14m27.471653956s) [noise=true , address=145.79.227.19:4077 , piece_id=EJIAPZSBMAO77E33V6MEBOQHUXVVYKMAVP53QAZXZSYBYDFP7MMQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.010197574 (301h14m27.472290057s) [noise=true , address=145.79.227.15:4875 , piece_id=PBDTNZ2NWRZZC3VCIQCVR2WLT2AP6662I2SSDNP6BLUDKCJMRK6Q, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.010634216 (301h14m27.472726699s) [noise=true , address=145.79.227.4:4185 , piece_id=3AD745KFD67M4XFNDXMJVEU34RRBQXXE4P3HVJJRDBHK3IIXNYTA, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011073742 (301h14m27.473166225s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011505996 (301h14m27.473598479s) [noise=true , address=45.91.116.10:4081 , piece_id=FTXZ2LOHQMY74KOX6XZC3TLHWXOGDTIY3SZLSPTE7THGKNLMPXRQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011982502 (301h14m27.474074985s) [noise=true , address=45.91.116.6:4974 , piece_id=7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.012488236 (301h14m27.474580719s) [noise=true , address=145.79.227.7:4294 , piece_id=7E3B4BGLCTW6C5DM6PPDJPWGVJS5HEGYPAPXRAMX7XDSG7USPQFQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.013242472 (301h14m27.475334955s) [noise=true , address=45.91.116.13:4154 , piece_id=2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.014529382 (301h14m27.476621855s) [noise=true , address=45.91.116.6:4975 , piece_id=T3VWH5OBQ7FE66GCKHZLEDSIXFVYEU3V52HI4CPD27QL3RH75K7Q, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.014632619 (301h14m27.476725102s) [noise=true , address=45.91.116.11:4117 , piece_id=QKEYHSK5JOL65H5T2KUEAEK6VKTNCZAYXIEZTMB76PV7P5PEVILQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.015990214 (301h14m27.478082697s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.016291928 (301h14m27.478384412s) [noise=true , address=45.91.116.12:4226 , piece_id=NM7SEIKZYUHZLNVSOGYVZWMI7U7X5SOXOJDAEGY5NF3M6QQFPZSQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.017455915 (301h14m27.479548399s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.018864467 (301h14m27.48095694s) [noise=true , address=77.90.31.10:4518 , piece_id=6FGNAMVK6OAGJTZJNKTUJWGRW7MLGMJMGJKMAINWMPW7BV74OO6A, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.020222022 (301h14m27.482314495s) [noise=true , address=45.91.116.11:4118 , piece_id=VUZB3QRFST2T3HJHMEYYZ6CV7CKJTFWQGNOBDTXHGET2MBUIC4HQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:43.755608897 (301h14m29.21770138s) [upload=265104]: Next returned nil segment Oct 6 02:45:43.755631300 (301h14m29.217723773s) [upload=265104]: Waiting for error group managing segments... Oct 6 02:45:44.043930124 (301h14m29.506022597s) [noise=true , address=145.79.227.19:4077 , piece_id=EJIAPZSBMAO77E33V6MEBOQHUXVVYKMAVP53QAZXZSYBYDFP7MMQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.045501296 (301h14m29.507593769s) [noise=true , address=145.79.227.4:4185 , piece_id=3AD745KFD67M4XFNDXMJVEU34RRBQXXE4P3HVJJRDBHK3IIXNYTA, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.057881199 (301h14m29.519973672s) [noise=true , address=145.79.227.15:4875 , piece_id=PBDTNZ2NWRZZC3VCIQCVR2WLT2AP6662I2SSDNP6BLUDKCJMRK6Q, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.068726540 (301h14m29.530819023s) [noise=true , address=145.79.227.7:4294 , piece_id=7E3B4BGLCTW6C5DM6PPDJPWGVJS5HEGYPAPXRAMX7XDSG7USPQFQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.897895712 (301h14m30.359988185s) [noise=true , address=45.91.116.12:4226 , piece_id=NM7SEIKZYUHZLNVSOGYVZWMI7U7X5SOXOJDAEGY5NF3M6QQFPZSQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.252859665 (301h14m30.714952148s) [noise=true , address=77.90.31.14:4662 , piece_id=MPVBPBB63JILLOUF3N7SRLCZ2F74LX7Z4AH7ZFG3BDNBHT4UIXTQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.318952216 (301h14m30.781044699s) [noise=true , address=45.91.116.6:4974 , piece_id=7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.836439779 (301h14m31.298532262s) [noise=true , address=45.91.116.10:4081 , piece_id=FTXZ2LOHQMY74KOX6XZC3TLHWXOGDTIY3SZLSPTE7THGKNLMPXRQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.975895173 (301h14m31.437987656s) [noise=true , address=45.91.116.6:4975 , piece_id=T3VWH5OBQ7FE66GCKHZLEDSIXFVYEU3V52HI4CPD27QL3RH75K7Q, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:46.826703981 (301h14m32.288796454s) [noise=true , address=77.90.31.10:4518 , piece_id=6FGNAMVK6OAGJTZJNKTUJWGRW7MLGMJMGJKMAINWMPW7BV74OO6A, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:48.069759484 (301h14m33.531851968s) [noise=true , address=45.91.116.13:4154 , piece_id=2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:54.791582362 (301h14m40.253674835s) [noise=true , address=45.91.116.11:4118 , piece_id=VUZB3QRFST2T3HJHMEYYZ6CV7CKJTFWQGNOBDTXHGET2MBUIC4HQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:56.329163998 (301h14m41.791256481s) [noise=true , address=45.91.116.11:4117 , piece_id=QKEYHSK5JOL65H5T2KUEAEK6VKTNCZAYXIEZTMB76PV7P5PEVILQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:56.329201379 (301h14m41.791293862s) [seg_pos={0 0}, upload=265104]: Segment reached optimal threshold of 13 pieces. Oct 6 02:45:56.329362146 (301h14m41.791454629s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Oct 6 02:45:56.329371974 (301h14m41.791464457s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329546396 (301h14m41.791638869s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: write tcp 188.166.225.170:57604->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 02:45:56.329556665 (301h14m41.791649148s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329567937 (301h14m41.791660421s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: rpc: dial tcp 77.90.31.19:4841: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 02:45:56.329579870 (301h14m41.791672353s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329599808 (301h14m41.791692291s) [seg_pos={0 0}, upload=265104]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 6 02:45:56.329683116 (301h14m41.7917756s) [upload=265104]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 6 02:45:56.329690771 (301h14m41.791783254s) [upload=265104]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 6 02:45:56.636341641 (301h14m42.098434134s) [upload=265104]: Done uploading segments... Oct 6 02:45:56.636368131 (301h14m42.098460614s) [upload=265104]: Upload finished. err: <nil> Duration: 6.081454151s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 5.92516517s FAIL (2025-10-06 03:18:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: 2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: 7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA) context deadline exceeded ======LOGS====== Oct 6 02:45:41.756337907 (301h14m27.21843039s) [upload=265104]: Starting upload Oct 6 02:45:41.756430433 (301h14m27.218522916s) [upload=265104]: Done starting upload Oct 6 02:45:41.756513701 (301h14m27.218606185s) [upload=265104]: Uploading segments... Oct 6 02:45:41.756752276 (301h14m27.218844759s) [upload=265104]: Got next segment. Inline: false Oct 6 02:45:41.756794536 (301h14m27.218887019s) [upload=265104]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 02:45:41.756800898 (301h14m27.218893381s) [upload=265104]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 02:45:42.009327267 (301h14m27.47141975s) [seg_pos={0 0}, upload=265104]: Begin upload segment... Oct 6 02:45:42.009417419 (301h14m27.471509892s) [seg_pos={0 0}, upload=265104]: Done begin upload segment. Oct 6 02:45:42.009444500 (301h14m27.471536983s) [noise=true , address=77.90.31.14:4662 , piece_id=MPVBPBB63JILLOUF3N7SRLCZ2F74LX7Z4AH7ZFG3BDNBHT4UIXTQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.009561473 (301h14m27.471653956s) [noise=true , address=145.79.227.19:4077 , piece_id=EJIAPZSBMAO77E33V6MEBOQHUXVVYKMAVP53QAZXZSYBYDFP7MMQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.010197574 (301h14m27.472290057s) [noise=true , address=145.79.227.15:4875 , piece_id=PBDTNZ2NWRZZC3VCIQCVR2WLT2AP6662I2SSDNP6BLUDKCJMRK6Q, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.010634216 (301h14m27.472726699s) [noise=true , address=145.79.227.4:4185 , piece_id=3AD745KFD67M4XFNDXMJVEU34RRBQXXE4P3HVJJRDBHK3IIXNYTA, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011073742 (301h14m27.473166225s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011505996 (301h14m27.473598479s) [noise=true , address=45.91.116.10:4081 , piece_id=FTXZ2LOHQMY74KOX6XZC3TLHWXOGDTIY3SZLSPTE7THGKNLMPXRQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011982502 (301h14m27.474074985s) [noise=true , address=45.91.116.6:4974 , piece_id=7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.012488236 (301h14m27.474580719s) [noise=true , address=145.79.227.7:4294 , piece_id=7E3B4BGLCTW6C5DM6PPDJPWGVJS5HEGYPAPXRAMX7XDSG7USPQFQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.013242472 (301h14m27.475334955s) [noise=true , address=45.91.116.13:4154 , piece_id=2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.014529382 (301h14m27.476621855s) [noise=true , address=45.91.116.6:4975 , piece_id=T3VWH5OBQ7FE66GCKHZLEDSIXFVYEU3V52HI4CPD27QL3RH75K7Q, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.014632619 (301h14m27.476725102s) [noise=true , address=45.91.116.11:4117 , piece_id=QKEYHSK5JOL65H5T2KUEAEK6VKTNCZAYXIEZTMB76PV7P5PEVILQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.015990214 (301h14m27.478082697s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.016291928 (301h14m27.478384412s) [noise=true , address=45.91.116.12:4226 , piece_id=NM7SEIKZYUHZLNVSOGYVZWMI7U7X5SOXOJDAEGY5NF3M6QQFPZSQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.017455915 (301h14m27.479548399s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.018864467 (301h14m27.48095694s) [noise=true , address=77.90.31.10:4518 , piece_id=6FGNAMVK6OAGJTZJNKTUJWGRW7MLGMJMGJKMAINWMPW7BV74OO6A, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.020222022 (301h14m27.482314495s) [noise=true , address=45.91.116.11:4118 , piece_id=VUZB3QRFST2T3HJHMEYYZ6CV7CKJTFWQGNOBDTXHGET2MBUIC4HQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:43.755608897 (301h14m29.21770138s) [upload=265104]: Next returned nil segment Oct 6 02:45:43.755631300 (301h14m29.217723773s) [upload=265104]: Waiting for error group managing segments... Oct 6 02:45:44.043930124 (301h14m29.506022597s) [noise=true , address=145.79.227.19:4077 , piece_id=EJIAPZSBMAO77E33V6MEBOQHUXVVYKMAVP53QAZXZSYBYDFP7MMQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.045501296 (301h14m29.507593769s) [noise=true , address=145.79.227.4:4185 , piece_id=3AD745KFD67M4XFNDXMJVEU34RRBQXXE4P3HVJJRDBHK3IIXNYTA, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.057881199 (301h14m29.519973672s) [noise=true , address=145.79.227.15:4875 , piece_id=PBDTNZ2NWRZZC3VCIQCVR2WLT2AP6662I2SSDNP6BLUDKCJMRK6Q, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.068726540 (301h14m29.530819023s) [noise=true , address=145.79.227.7:4294 , piece_id=7E3B4BGLCTW6C5DM6PPDJPWGVJS5HEGYPAPXRAMX7XDSG7USPQFQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.897895712 (301h14m30.359988185s) [noise=true , address=45.91.116.12:4226 , piece_id=NM7SEIKZYUHZLNVSOGYVZWMI7U7X5SOXOJDAEGY5NF3M6QQFPZSQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.252859665 (301h14m30.714952148s) [noise=true , address=77.90.31.14:4662 , piece_id=MPVBPBB63JILLOUF3N7SRLCZ2F74LX7Z4AH7ZFG3BDNBHT4UIXTQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.318952216 (301h14m30.781044699s) [noise=true , address=45.91.116.6:4974 , piece_id=7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.836439779 (301h14m31.298532262s) [noise=true , address=45.91.116.10:4081 , piece_id=FTXZ2LOHQMY74KOX6XZC3TLHWXOGDTIY3SZLSPTE7THGKNLMPXRQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.975895173 (301h14m31.437987656s) [noise=true , address=45.91.116.6:4975 , piece_id=T3VWH5OBQ7FE66GCKHZLEDSIXFVYEU3V52HI4CPD27QL3RH75K7Q, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:46.826703981 (301h14m32.288796454s) [noise=true , address=77.90.31.10:4518 , piece_id=6FGNAMVK6OAGJTZJNKTUJWGRW7MLGMJMGJKMAINWMPW7BV74OO6A, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:48.069759484 (301h14m33.531851968s) [noise=true , address=45.91.116.13:4154 , piece_id=2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:54.791582362 (301h14m40.253674835s) [noise=true , address=45.91.116.11:4118 , piece_id=VUZB3QRFST2T3HJHMEYYZ6CV7CKJTFWQGNOBDTXHGET2MBUIC4HQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:56.329163998 (301h14m41.791256481s) [noise=true , address=45.91.116.11:4117 , piece_id=QKEYHSK5JOL65H5T2KUEAEK6VKTNCZAYXIEZTMB76PV7P5PEVILQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:56.329201379 (301h14m41.791293862s) [seg_pos={0 0}, upload=265104]: Segment reached optimal threshold of 13 pieces. Oct 6 02:45:56.329362146 (301h14m41.791454629s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Oct 6 02:45:56.329371974 (301h14m41.791464457s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329546396 (301h14m41.791638869s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: write tcp 188.166.225.170:57604->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 02:45:56.329556665 (301h14m41.791649148s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329567937 (301h14m41.791660421s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: rpc: dial tcp 77.90.31.19:4841: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 02:45:56.329579870 (301h14m41.791672353s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329599808 (301h14m41.791692291s) [seg_pos={0 0}, upload=265104]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 6 02:45:56.329683116 (301h14m41.7917756s) [upload=265104]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 6 02:45:56.329690771 (301h14m41.791783254s) [upload=265104]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 6 02:45:56.636341641 (301h14m42.098434134s) [upload=265104]: Done uploading segments... Oct 6 02:45:56.636368131 (301h14m42.098460614s) [upload=265104]: Upload finished. err: <nil> Duration: 5.925205538s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 6.40383753s FAIL (2025-10-06 03:17:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: 2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: 7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA) context deadline exceeded ======LOGS====== Oct 6 02:45:41.756337907 (301h14m27.21843039s) [upload=265104]: Starting upload Oct 6 02:45:41.756430433 (301h14m27.218522916s) [upload=265104]: Done starting upload Oct 6 02:45:41.756513701 (301h14m27.218606185s) [upload=265104]: Uploading segments... Oct 6 02:45:41.756752276 (301h14m27.218844759s) [upload=265104]: Got next segment. Inline: false Oct 6 02:45:41.756794536 (301h14m27.218887019s) [upload=265104]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 02:45:41.756800898 (301h14m27.218893381s) [upload=265104]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 02:45:42.009327267 (301h14m27.47141975s) [seg_pos={0 0}, upload=265104]: Begin upload segment... Oct 6 02:45:42.009417419 (301h14m27.471509892s) [seg_pos={0 0}, upload=265104]: Done begin upload segment. Oct 6 02:45:42.009444500 (301h14m27.471536983s) [noise=true , address=77.90.31.14:4662 , piece_id=MPVBPBB63JILLOUF3N7SRLCZ2F74LX7Z4AH7ZFG3BDNBHT4UIXTQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.009561473 (301h14m27.471653956s) [noise=true , address=145.79.227.19:4077 , piece_id=EJIAPZSBMAO77E33V6MEBOQHUXVVYKMAVP53QAZXZSYBYDFP7MMQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.010197574 (301h14m27.472290057s) [noise=true , address=145.79.227.15:4875 , piece_id=PBDTNZ2NWRZZC3VCIQCVR2WLT2AP6662I2SSDNP6BLUDKCJMRK6Q, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.010634216 (301h14m27.472726699s) [noise=true , address=145.79.227.4:4185 , piece_id=3AD745KFD67M4XFNDXMJVEU34RRBQXXE4P3HVJJRDBHK3IIXNYTA, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011073742 (301h14m27.473166225s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011505996 (301h14m27.473598479s) [noise=true , address=45.91.116.10:4081 , piece_id=FTXZ2LOHQMY74KOX6XZC3TLHWXOGDTIY3SZLSPTE7THGKNLMPXRQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011982502 (301h14m27.474074985s) [noise=true , address=45.91.116.6:4974 , piece_id=7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.012488236 (301h14m27.474580719s) [noise=true , address=145.79.227.7:4294 , piece_id=7E3B4BGLCTW6C5DM6PPDJPWGVJS5HEGYPAPXRAMX7XDSG7USPQFQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.013242472 (301h14m27.475334955s) [noise=true , address=45.91.116.13:4154 , piece_id=2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.014529382 (301h14m27.476621855s) [noise=true , address=45.91.116.6:4975 , piece_id=T3VWH5OBQ7FE66GCKHZLEDSIXFVYEU3V52HI4CPD27QL3RH75K7Q, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.014632619 (301h14m27.476725102s) [noise=true , address=45.91.116.11:4117 , piece_id=QKEYHSK5JOL65H5T2KUEAEK6VKTNCZAYXIEZTMB76PV7P5PEVILQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.015990214 (301h14m27.478082697s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.016291928 (301h14m27.478384412s) [noise=true , address=45.91.116.12:4226 , piece_id=NM7SEIKZYUHZLNVSOGYVZWMI7U7X5SOXOJDAEGY5NF3M6QQFPZSQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.017455915 (301h14m27.479548399s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.018864467 (301h14m27.48095694s) [noise=true , address=77.90.31.10:4518 , piece_id=6FGNAMVK6OAGJTZJNKTUJWGRW7MLGMJMGJKMAINWMPW7BV74OO6A, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.020222022 (301h14m27.482314495s) [noise=true , address=45.91.116.11:4118 , piece_id=VUZB3QRFST2T3HJHMEYYZ6CV7CKJTFWQGNOBDTXHGET2MBUIC4HQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:43.755608897 (301h14m29.21770138s) [upload=265104]: Next returned nil segment Oct 6 02:45:43.755631300 (301h14m29.217723773s) [upload=265104]: Waiting for error group managing segments... Oct 6 02:45:44.043930124 (301h14m29.506022597s) [noise=true , address=145.79.227.19:4077 , piece_id=EJIAPZSBMAO77E33V6MEBOQHUXVVYKMAVP53QAZXZSYBYDFP7MMQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.045501296 (301h14m29.507593769s) [noise=true , address=145.79.227.4:4185 , piece_id=3AD745KFD67M4XFNDXMJVEU34RRBQXXE4P3HVJJRDBHK3IIXNYTA, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.057881199 (301h14m29.519973672s) [noise=true , address=145.79.227.15:4875 , piece_id=PBDTNZ2NWRZZC3VCIQCVR2WLT2AP6662I2SSDNP6BLUDKCJMRK6Q, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.068726540 (301h14m29.530819023s) [noise=true , address=145.79.227.7:4294 , piece_id=7E3B4BGLCTW6C5DM6PPDJPWGVJS5HEGYPAPXRAMX7XDSG7USPQFQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.897895712 (301h14m30.359988185s) [noise=true , address=45.91.116.12:4226 , piece_id=NM7SEIKZYUHZLNVSOGYVZWMI7U7X5SOXOJDAEGY5NF3M6QQFPZSQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.252859665 (301h14m30.714952148s) [noise=true , address=77.90.31.14:4662 , piece_id=MPVBPBB63JILLOUF3N7SRLCZ2F74LX7Z4AH7ZFG3BDNBHT4UIXTQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.318952216 (301h14m30.781044699s) [noise=true , address=45.91.116.6:4974 , piece_id=7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.836439779 (301h14m31.298532262s) [noise=true , address=45.91.116.10:4081 , piece_id=FTXZ2LOHQMY74KOX6XZC3TLHWXOGDTIY3SZLSPTE7THGKNLMPXRQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.975895173 (301h14m31.437987656s) [noise=true , address=45.91.116.6:4975 , piece_id=T3VWH5OBQ7FE66GCKHZLEDSIXFVYEU3V52HI4CPD27QL3RH75K7Q, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:46.826703981 (301h14m32.288796454s) [noise=true , address=77.90.31.10:4518 , piece_id=6FGNAMVK6OAGJTZJNKTUJWGRW7MLGMJMGJKMAINWMPW7BV74OO6A, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:48.069759484 (301h14m33.531851968s) [noise=true , address=45.91.116.13:4154 , piece_id=2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:54.791582362 (301h14m40.253674835s) [noise=true , address=45.91.116.11:4118 , piece_id=VUZB3QRFST2T3HJHMEYYZ6CV7CKJTFWQGNOBDTXHGET2MBUIC4HQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:56.329163998 (301h14m41.791256481s) [noise=true , address=45.91.116.11:4117 , piece_id=QKEYHSK5JOL65H5T2KUEAEK6VKTNCZAYXIEZTMB76PV7P5PEVILQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:56.329201379 (301h14m41.791293862s) [seg_pos={0 0}, upload=265104]: Segment reached optimal threshold of 13 pieces. Oct 6 02:45:56.329362146 (301h14m41.791454629s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Oct 6 02:45:56.329371974 (301h14m41.791464457s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329546396 (301h14m41.791638869s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: write tcp 188.166.225.170:57604->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 02:45:56.329556665 (301h14m41.791649148s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329567937 (301h14m41.791660421s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: rpc: dial tcp 77.90.31.19:4841: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 02:45:56.329579870 (301h14m41.791672353s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329599808 (301h14m41.791692291s) [seg_pos={0 0}, upload=265104]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 6 02:45:56.329683116 (301h14m41.7917756s) [upload=265104]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 6 02:45:56.329690771 (301h14m41.791783254s) [upload=265104]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 6 02:45:56.636341641 (301h14m42.098434134s) [upload=265104]: Done uploading segments... Oct 6 02:45:56.636368131 (301h14m42.098460614s) [upload=265104]: Upload finished. err: <nil> Duration: 6.403868179s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 5.967059935s FAIL (2025-10-06 03:17:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: 2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: 7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA) context deadline exceeded ======LOGS====== Oct 6 02:45:41.756337907 (301h14m27.21843039s) [upload=265104]: Starting upload Oct 6 02:45:41.756430433 (301h14m27.218522916s) [upload=265104]: Done starting upload Oct 6 02:45:41.756513701 (301h14m27.218606185s) [upload=265104]: Uploading segments... Oct 6 02:45:41.756752276 (301h14m27.218844759s) [upload=265104]: Got next segment. Inline: false Oct 6 02:45:41.756794536 (301h14m27.218887019s) [upload=265104]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 02:45:41.756800898 (301h14m27.218893381s) [upload=265104]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 02:45:42.009327267 (301h14m27.47141975s) [seg_pos={0 0}, upload=265104]: Begin upload segment... Oct 6 02:45:42.009417419 (301h14m27.471509892s) [seg_pos={0 0}, upload=265104]: Done begin upload segment. Oct 6 02:45:42.009444500 (301h14m27.471536983s) [noise=true , address=77.90.31.14:4662 , piece_id=MPVBPBB63JILLOUF3N7SRLCZ2F74LX7Z4AH7ZFG3BDNBHT4UIXTQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.009561473 (301h14m27.471653956s) [noise=true , address=145.79.227.19:4077 , piece_id=EJIAPZSBMAO77E33V6MEBOQHUXVVYKMAVP53QAZXZSYBYDFP7MMQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.010197574 (301h14m27.472290057s) [noise=true , address=145.79.227.15:4875 , piece_id=PBDTNZ2NWRZZC3VCIQCVR2WLT2AP6662I2SSDNP6BLUDKCJMRK6Q, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.010634216 (301h14m27.472726699s) [noise=true , address=145.79.227.4:4185 , piece_id=3AD745KFD67M4XFNDXMJVEU34RRBQXXE4P3HVJJRDBHK3IIXNYTA, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011073742 (301h14m27.473166225s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011505996 (301h14m27.473598479s) [noise=true , address=45.91.116.10:4081 , piece_id=FTXZ2LOHQMY74KOX6XZC3TLHWXOGDTIY3SZLSPTE7THGKNLMPXRQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011982502 (301h14m27.474074985s) [noise=true , address=45.91.116.6:4974 , piece_id=7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.012488236 (301h14m27.474580719s) [noise=true , address=145.79.227.7:4294 , piece_id=7E3B4BGLCTW6C5DM6PPDJPWGVJS5HEGYPAPXRAMX7XDSG7USPQFQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.013242472 (301h14m27.475334955s) [noise=true , address=45.91.116.13:4154 , piece_id=2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.014529382 (301h14m27.476621855s) [noise=true , address=45.91.116.6:4975 , piece_id=T3VWH5OBQ7FE66GCKHZLEDSIXFVYEU3V52HI4CPD27QL3RH75K7Q, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.014632619 (301h14m27.476725102s) [noise=true , address=45.91.116.11:4117 , piece_id=QKEYHSK5JOL65H5T2KUEAEK6VKTNCZAYXIEZTMB76PV7P5PEVILQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.015990214 (301h14m27.478082697s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.016291928 (301h14m27.478384412s) [noise=true , address=45.91.116.12:4226 , piece_id=NM7SEIKZYUHZLNVSOGYVZWMI7U7X5SOXOJDAEGY5NF3M6QQFPZSQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.017455915 (301h14m27.479548399s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.018864467 (301h14m27.48095694s) [noise=true , address=77.90.31.10:4518 , piece_id=6FGNAMVK6OAGJTZJNKTUJWGRW7MLGMJMGJKMAINWMPW7BV74OO6A, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.020222022 (301h14m27.482314495s) [noise=true , address=45.91.116.11:4118 , piece_id=VUZB3QRFST2T3HJHMEYYZ6CV7CKJTFWQGNOBDTXHGET2MBUIC4HQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:43.755608897 (301h14m29.21770138s) [upload=265104]: Next returned nil segment Oct 6 02:45:43.755631300 (301h14m29.217723773s) [upload=265104]: Waiting for error group managing segments... Oct 6 02:45:44.043930124 (301h14m29.506022597s) [noise=true , address=145.79.227.19:4077 , piece_id=EJIAPZSBMAO77E33V6MEBOQHUXVVYKMAVP53QAZXZSYBYDFP7MMQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.045501296 (301h14m29.507593769s) [noise=true , address=145.79.227.4:4185 , piece_id=3AD745KFD67M4XFNDXMJVEU34RRBQXXE4P3HVJJRDBHK3IIXNYTA, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.057881199 (301h14m29.519973672s) [noise=true , address=145.79.227.15:4875 , piece_id=PBDTNZ2NWRZZC3VCIQCVR2WLT2AP6662I2SSDNP6BLUDKCJMRK6Q, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.068726540 (301h14m29.530819023s) [noise=true , address=145.79.227.7:4294 , piece_id=7E3B4BGLCTW6C5DM6PPDJPWGVJS5HEGYPAPXRAMX7XDSG7USPQFQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.897895712 (301h14m30.359988185s) [noise=true , address=45.91.116.12:4226 , piece_id=NM7SEIKZYUHZLNVSOGYVZWMI7U7X5SOXOJDAEGY5NF3M6QQFPZSQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.252859665 (301h14m30.714952148s) [noise=true , address=77.90.31.14:4662 , piece_id=MPVBPBB63JILLOUF3N7SRLCZ2F74LX7Z4AH7ZFG3BDNBHT4UIXTQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.318952216 (301h14m30.781044699s) [noise=true , address=45.91.116.6:4974 , piece_id=7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.836439779 (301h14m31.298532262s) [noise=true , address=45.91.116.10:4081 , piece_id=FTXZ2LOHQMY74KOX6XZC3TLHWXOGDTIY3SZLSPTE7THGKNLMPXRQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.975895173 (301h14m31.437987656s) [noise=true , address=45.91.116.6:4975 , piece_id=T3VWH5OBQ7FE66GCKHZLEDSIXFVYEU3V52HI4CPD27QL3RH75K7Q, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:46.826703981 (301h14m32.288796454s) [noise=true , address=77.90.31.10:4518 , piece_id=6FGNAMVK6OAGJTZJNKTUJWGRW7MLGMJMGJKMAINWMPW7BV74OO6A, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:48.069759484 (301h14m33.531851968s) [noise=true , address=45.91.116.13:4154 , piece_id=2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:54.791582362 (301h14m40.253674835s) [noise=true , address=45.91.116.11:4118 , piece_id=VUZB3QRFST2T3HJHMEYYZ6CV7CKJTFWQGNOBDTXHGET2MBUIC4HQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:56.329163998 (301h14m41.791256481s) [noise=true , address=45.91.116.11:4117 , piece_id=QKEYHSK5JOL65H5T2KUEAEK6VKTNCZAYXIEZTMB76PV7P5PEVILQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:56.329201379 (301h14m41.791293862s) [seg_pos={0 0}, upload=265104]: Segment reached optimal threshold of 13 pieces. Oct 6 02:45:56.329362146 (301h14m41.791454629s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Oct 6 02:45:56.329371974 (301h14m41.791464457s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329546396 (301h14m41.791638869s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: write tcp 188.166.225.170:57604->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 02:45:56.329556665 (301h14m41.791649148s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329567937 (301h14m41.791660421s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: rpc: dial tcp 77.90.31.19:4841: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 02:45:56.329579870 (301h14m41.791672353s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329599808 (301h14m41.791692291s) [seg_pos={0 0}, upload=265104]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 6 02:45:56.329683116 (301h14m41.7917756s) [upload=265104]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 6 02:45:56.329690771 (301h14m41.791783254s) [upload=265104]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 6 02:45:56.636341641 (301h14m42.098434134s) [upload=265104]: Done uploading segments... Oct 6 02:45:56.636368131 (301h14m42.098460614s) [upload=265104]: Upload finished. err: <nil> Duration: 5.96708932s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 6.067753883s FAIL (2025-10-06 03:17:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: 2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: 7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA) context deadline exceeded ======LOGS====== Oct 6 02:45:41.756337907 (301h14m27.21843039s) [upload=265104]: Starting upload Oct 6 02:45:41.756430433 (301h14m27.218522916s) [upload=265104]: Done starting upload Oct 6 02:45:41.756513701 (301h14m27.218606185s) [upload=265104]: Uploading segments... Oct 6 02:45:41.756752276 (301h14m27.218844759s) [upload=265104]: Got next segment. Inline: false Oct 6 02:45:41.756794536 (301h14m27.218887019s) [upload=265104]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 02:45:41.756800898 (301h14m27.218893381s) [upload=265104]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 02:45:42.009327267 (301h14m27.47141975s) [seg_pos={0 0}, upload=265104]: Begin upload segment... Oct 6 02:45:42.009417419 (301h14m27.471509892s) [seg_pos={0 0}, upload=265104]: Done begin upload segment. Oct 6 02:45:42.009444500 (301h14m27.471536983s) [noise=true , address=77.90.31.14:4662 , piece_id=MPVBPBB63JILLOUF3N7SRLCZ2F74LX7Z4AH7ZFG3BDNBHT4UIXTQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.009561473 (301h14m27.471653956s) [noise=true , address=145.79.227.19:4077 , piece_id=EJIAPZSBMAO77E33V6MEBOQHUXVVYKMAVP53QAZXZSYBYDFP7MMQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.010197574 (301h14m27.472290057s) [noise=true , address=145.79.227.15:4875 , piece_id=PBDTNZ2NWRZZC3VCIQCVR2WLT2AP6662I2SSDNP6BLUDKCJMRK6Q, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.010634216 (301h14m27.472726699s) [noise=true , address=145.79.227.4:4185 , piece_id=3AD745KFD67M4XFNDXMJVEU34RRBQXXE4P3HVJJRDBHK3IIXNYTA, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011073742 (301h14m27.473166225s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011505996 (301h14m27.473598479s) [noise=true , address=45.91.116.10:4081 , piece_id=FTXZ2LOHQMY74KOX6XZC3TLHWXOGDTIY3SZLSPTE7THGKNLMPXRQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011982502 (301h14m27.474074985s) [noise=true , address=45.91.116.6:4974 , piece_id=7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.012488236 (301h14m27.474580719s) [noise=true , address=145.79.227.7:4294 , piece_id=7E3B4BGLCTW6C5DM6PPDJPWGVJS5HEGYPAPXRAMX7XDSG7USPQFQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.013242472 (301h14m27.475334955s) [noise=true , address=45.91.116.13:4154 , piece_id=2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.014529382 (301h14m27.476621855s) [noise=true , address=45.91.116.6:4975 , piece_id=T3VWH5OBQ7FE66GCKHZLEDSIXFVYEU3V52HI4CPD27QL3RH75K7Q, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.014632619 (301h14m27.476725102s) [noise=true , address=45.91.116.11:4117 , piece_id=QKEYHSK5JOL65H5T2KUEAEK6VKTNCZAYXIEZTMB76PV7P5PEVILQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.015990214 (301h14m27.478082697s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.016291928 (301h14m27.478384412s) [noise=true , address=45.91.116.12:4226 , piece_id=NM7SEIKZYUHZLNVSOGYVZWMI7U7X5SOXOJDAEGY5NF3M6QQFPZSQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.017455915 (301h14m27.479548399s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.018864467 (301h14m27.48095694s) [noise=true , address=77.90.31.10:4518 , piece_id=6FGNAMVK6OAGJTZJNKTUJWGRW7MLGMJMGJKMAINWMPW7BV74OO6A, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.020222022 (301h14m27.482314495s) [noise=true , address=45.91.116.11:4118 , piece_id=VUZB3QRFST2T3HJHMEYYZ6CV7CKJTFWQGNOBDTXHGET2MBUIC4HQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:43.755608897 (301h14m29.21770138s) [upload=265104]: Next returned nil segment Oct 6 02:45:43.755631300 (301h14m29.217723773s) [upload=265104]: Waiting for error group managing segments... Oct 6 02:45:44.043930124 (301h14m29.506022597s) [noise=true , address=145.79.227.19:4077 , piece_id=EJIAPZSBMAO77E33V6MEBOQHUXVVYKMAVP53QAZXZSYBYDFP7MMQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.045501296 (301h14m29.507593769s) [noise=true , address=145.79.227.4:4185 , piece_id=3AD745KFD67M4XFNDXMJVEU34RRBQXXE4P3HVJJRDBHK3IIXNYTA, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.057881199 (301h14m29.519973672s) [noise=true , address=145.79.227.15:4875 , piece_id=PBDTNZ2NWRZZC3VCIQCVR2WLT2AP6662I2SSDNP6BLUDKCJMRK6Q, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.068726540 (301h14m29.530819023s) [noise=true , address=145.79.227.7:4294 , piece_id=7E3B4BGLCTW6C5DM6PPDJPWGVJS5HEGYPAPXRAMX7XDSG7USPQFQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.897895712 (301h14m30.359988185s) [noise=true , address=45.91.116.12:4226 , piece_id=NM7SEIKZYUHZLNVSOGYVZWMI7U7X5SOXOJDAEGY5NF3M6QQFPZSQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.252859665 (301h14m30.714952148s) [noise=true , address=77.90.31.14:4662 , piece_id=MPVBPBB63JILLOUF3N7SRLCZ2F74LX7Z4AH7ZFG3BDNBHT4UIXTQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.318952216 (301h14m30.781044699s) [noise=true , address=45.91.116.6:4974 , piece_id=7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.836439779 (301h14m31.298532262s) [noise=true , address=45.91.116.10:4081 , piece_id=FTXZ2LOHQMY74KOX6XZC3TLHWXOGDTIY3SZLSPTE7THGKNLMPXRQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.975895173 (301h14m31.437987656s) [noise=true , address=45.91.116.6:4975 , piece_id=T3VWH5OBQ7FE66GCKHZLEDSIXFVYEU3V52HI4CPD27QL3RH75K7Q, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:46.826703981 (301h14m32.288796454s) [noise=true , address=77.90.31.10:4518 , piece_id=6FGNAMVK6OAGJTZJNKTUJWGRW7MLGMJMGJKMAINWMPW7BV74OO6A, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:48.069759484 (301h14m33.531851968s) [noise=true , address=45.91.116.13:4154 , piece_id=2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:54.791582362 (301h14m40.253674835s) [noise=true , address=45.91.116.11:4118 , piece_id=VUZB3QRFST2T3HJHMEYYZ6CV7CKJTFWQGNOBDTXHGET2MBUIC4HQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:56.329163998 (301h14m41.791256481s) [noise=true , address=45.91.116.11:4117 , piece_id=QKEYHSK5JOL65H5T2KUEAEK6VKTNCZAYXIEZTMB76PV7P5PEVILQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:56.329201379 (301h14m41.791293862s) [seg_pos={0 0}, upload=265104]: Segment reached optimal threshold of 13 pieces. Oct 6 02:45:56.329362146 (301h14m41.791454629s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Oct 6 02:45:56.329371974 (301h14m41.791464457s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329546396 (301h14m41.791638869s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: write tcp 188.166.225.170:57604->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 02:45:56.329556665 (301h14m41.791649148s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329567937 (301h14m41.791660421s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: rpc: dial tcp 77.90.31.19:4841: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 02:45:56.329579870 (301h14m41.791672353s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329599808 (301h14m41.791692291s) [seg_pos={0 0}, upload=265104]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 6 02:45:56.329683116 (301h14m41.7917756s) [upload=265104]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 6 02:45:56.329690771 (301h14m41.791783254s) [upload=265104]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 6 02:45:56.636341641 (301h14m42.098434134s) [upload=265104]: Done uploading segments... Oct 6 02:45:56.636368131 (301h14m42.098460614s) [upload=265104]: Upload finished. err: <nil> Duration: 6.067789761s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 5.987181962s FAIL (2025-10-06 03:16:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: 2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: 7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA) context deadline exceeded ======LOGS====== Oct 6 02:45:41.756337907 (301h14m27.21843039s) [upload=265104]: Starting upload Oct 6 02:45:41.756430433 (301h14m27.218522916s) [upload=265104]: Done starting upload Oct 6 02:45:41.756513701 (301h14m27.218606185s) [upload=265104]: Uploading segments... Oct 6 02:45:41.756752276 (301h14m27.218844759s) [upload=265104]: Got next segment. Inline: false Oct 6 02:45:41.756794536 (301h14m27.218887019s) [upload=265104]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 02:45:41.756800898 (301h14m27.218893381s) [upload=265104]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 02:45:42.009327267 (301h14m27.47141975s) [seg_pos={0 0}, upload=265104]: Begin upload segment... Oct 6 02:45:42.009417419 (301h14m27.471509892s) [seg_pos={0 0}, upload=265104]: Done begin upload segment. Oct 6 02:45:42.009444500 (301h14m27.471536983s) [noise=true , address=77.90.31.14:4662 , piece_id=MPVBPBB63JILLOUF3N7SRLCZ2F74LX7Z4AH7ZFG3BDNBHT4UIXTQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.009561473 (301h14m27.471653956s) [noise=true , address=145.79.227.19:4077 , piece_id=EJIAPZSBMAO77E33V6MEBOQHUXVVYKMAVP53QAZXZSYBYDFP7MMQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.010197574 (301h14m27.472290057s) [noise=true , address=145.79.227.15:4875 , piece_id=PBDTNZ2NWRZZC3VCIQCVR2WLT2AP6662I2SSDNP6BLUDKCJMRK6Q, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.010634216 (301h14m27.472726699s) [noise=true , address=145.79.227.4:4185 , piece_id=3AD745KFD67M4XFNDXMJVEU34RRBQXXE4P3HVJJRDBHK3IIXNYTA, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011073742 (301h14m27.473166225s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011505996 (301h14m27.473598479s) [noise=true , address=45.91.116.10:4081 , piece_id=FTXZ2LOHQMY74KOX6XZC3TLHWXOGDTIY3SZLSPTE7THGKNLMPXRQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011982502 (301h14m27.474074985s) [noise=true , address=45.91.116.6:4974 , piece_id=7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.012488236 (301h14m27.474580719s) [noise=true , address=145.79.227.7:4294 , piece_id=7E3B4BGLCTW6C5DM6PPDJPWGVJS5HEGYPAPXRAMX7XDSG7USPQFQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.013242472 (301h14m27.475334955s) [noise=true , address=45.91.116.13:4154 , piece_id=2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.014529382 (301h14m27.476621855s) [noise=true , address=45.91.116.6:4975 , piece_id=T3VWH5OBQ7FE66GCKHZLEDSIXFVYEU3V52HI4CPD27QL3RH75K7Q, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.014632619 (301h14m27.476725102s) [noise=true , address=45.91.116.11:4117 , piece_id=QKEYHSK5JOL65H5T2KUEAEK6VKTNCZAYXIEZTMB76PV7P5PEVILQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.015990214 (301h14m27.478082697s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.016291928 (301h14m27.478384412s) [noise=true , address=45.91.116.12:4226 , piece_id=NM7SEIKZYUHZLNVSOGYVZWMI7U7X5SOXOJDAEGY5NF3M6QQFPZSQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.017455915 (301h14m27.479548399s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.018864467 (301h14m27.48095694s) [noise=true , address=77.90.31.10:4518 , piece_id=6FGNAMVK6OAGJTZJNKTUJWGRW7MLGMJMGJKMAINWMPW7BV74OO6A, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.020222022 (301h14m27.482314495s) [noise=true , address=45.91.116.11:4118 , piece_id=VUZB3QRFST2T3HJHMEYYZ6CV7CKJTFWQGNOBDTXHGET2MBUIC4HQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:43.755608897 (301h14m29.21770138s) [upload=265104]: Next returned nil segment Oct 6 02:45:43.755631300 (301h14m29.217723773s) [upload=265104]: Waiting for error group managing segments... Oct 6 02:45:44.043930124 (301h14m29.506022597s) [noise=true , address=145.79.227.19:4077 , piece_id=EJIAPZSBMAO77E33V6MEBOQHUXVVYKMAVP53QAZXZSYBYDFP7MMQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.045501296 (301h14m29.507593769s) [noise=true , address=145.79.227.4:4185 , piece_id=3AD745KFD67M4XFNDXMJVEU34RRBQXXE4P3HVJJRDBHK3IIXNYTA, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.057881199 (301h14m29.519973672s) [noise=true , address=145.79.227.15:4875 , piece_id=PBDTNZ2NWRZZC3VCIQCVR2WLT2AP6662I2SSDNP6BLUDKCJMRK6Q, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.068726540 (301h14m29.530819023s) [noise=true , address=145.79.227.7:4294 , piece_id=7E3B4BGLCTW6C5DM6PPDJPWGVJS5HEGYPAPXRAMX7XDSG7USPQFQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.897895712 (301h14m30.359988185s) [noise=true , address=45.91.116.12:4226 , piece_id=NM7SEIKZYUHZLNVSOGYVZWMI7U7X5SOXOJDAEGY5NF3M6QQFPZSQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.252859665 (301h14m30.714952148s) [noise=true , address=77.90.31.14:4662 , piece_id=MPVBPBB63JILLOUF3N7SRLCZ2F74LX7Z4AH7ZFG3BDNBHT4UIXTQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.318952216 (301h14m30.781044699s) [noise=true , address=45.91.116.6:4974 , piece_id=7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.836439779 (301h14m31.298532262s) [noise=true , address=45.91.116.10:4081 , piece_id=FTXZ2LOHQMY74KOX6XZC3TLHWXOGDTIY3SZLSPTE7THGKNLMPXRQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.975895173 (301h14m31.437987656s) [noise=true , address=45.91.116.6:4975 , piece_id=T3VWH5OBQ7FE66GCKHZLEDSIXFVYEU3V52HI4CPD27QL3RH75K7Q, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:46.826703981 (301h14m32.288796454s) [noise=true , address=77.90.31.10:4518 , piece_id=6FGNAMVK6OAGJTZJNKTUJWGRW7MLGMJMGJKMAINWMPW7BV74OO6A, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:48.069759484 (301h14m33.531851968s) [noise=true , address=45.91.116.13:4154 , piece_id=2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:54.791582362 (301h14m40.253674835s) [noise=true , address=45.91.116.11:4118 , piece_id=VUZB3QRFST2T3HJHMEYYZ6CV7CKJTFWQGNOBDTXHGET2MBUIC4HQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:56.329163998 (301h14m41.791256481s) [noise=true , address=45.91.116.11:4117 , piece_id=QKEYHSK5JOL65H5T2KUEAEK6VKTNCZAYXIEZTMB76PV7P5PEVILQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:56.329201379 (301h14m41.791293862s) [seg_pos={0 0}, upload=265104]: Segment reached optimal threshold of 13 pieces. Oct 6 02:45:56.329362146 (301h14m41.791454629s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Oct 6 02:45:56.329371974 (301h14m41.791464457s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329546396 (301h14m41.791638869s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: write tcp 188.166.225.170:57604->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 02:45:56.329556665 (301h14m41.791649148s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329567937 (301h14m41.791660421s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: rpc: dial tcp 77.90.31.19:4841: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 02:45:56.329579870 (301h14m41.791672353s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329599808 (301h14m41.791692291s) [seg_pos={0 0}, upload=265104]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 6 02:45:56.329683116 (301h14m41.7917756s) [upload=265104]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 6 02:45:56.329690771 (301h14m41.791783254s) [upload=265104]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 6 02:45:56.636341641 (301h14m42.098434134s) [upload=265104]: Done uploading segments... Oct 6 02:45:56.636368131 (301h14m42.098460614s) [upload=265104]: Upload finished. err: <nil> Duration: 5.987213821s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 5.978547387s FAIL (2025-10-06 03:15:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: 2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: 7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA) context deadline exceeded ======LOGS====== Oct 6 02:45:41.756337907 (301h14m27.21843039s) [upload=265104]: Starting upload Oct 6 02:45:41.756430433 (301h14m27.218522916s) [upload=265104]: Done starting upload Oct 6 02:45:41.756513701 (301h14m27.218606185s) [upload=265104]: Uploading segments... Oct 6 02:45:41.756752276 (301h14m27.218844759s) [upload=265104]: Got next segment. Inline: false Oct 6 02:45:41.756794536 (301h14m27.218887019s) [upload=265104]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 02:45:41.756800898 (301h14m27.218893381s) [upload=265104]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 02:45:42.009327267 (301h14m27.47141975s) [seg_pos={0 0}, upload=265104]: Begin upload segment... Oct 6 02:45:42.009417419 (301h14m27.471509892s) [seg_pos={0 0}, upload=265104]: Done begin upload segment. Oct 6 02:45:42.009444500 (301h14m27.471536983s) [noise=true , address=77.90.31.14:4662 , piece_id=MPVBPBB63JILLOUF3N7SRLCZ2F74LX7Z4AH7ZFG3BDNBHT4UIXTQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.009561473 (301h14m27.471653956s) [noise=true , address=145.79.227.19:4077 , piece_id=EJIAPZSBMAO77E33V6MEBOQHUXVVYKMAVP53QAZXZSYBYDFP7MMQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.010197574 (301h14m27.472290057s) [noise=true , address=145.79.227.15:4875 , piece_id=PBDTNZ2NWRZZC3VCIQCVR2WLT2AP6662I2SSDNP6BLUDKCJMRK6Q, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.010634216 (301h14m27.472726699s) [noise=true , address=145.79.227.4:4185 , piece_id=3AD745KFD67M4XFNDXMJVEU34RRBQXXE4P3HVJJRDBHK3IIXNYTA, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011073742 (301h14m27.473166225s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011505996 (301h14m27.473598479s) [noise=true , address=45.91.116.10:4081 , piece_id=FTXZ2LOHQMY74KOX6XZC3TLHWXOGDTIY3SZLSPTE7THGKNLMPXRQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011982502 (301h14m27.474074985s) [noise=true , address=45.91.116.6:4974 , piece_id=7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.012488236 (301h14m27.474580719s) [noise=true , address=145.79.227.7:4294 , piece_id=7E3B4BGLCTW6C5DM6PPDJPWGVJS5HEGYPAPXRAMX7XDSG7USPQFQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.013242472 (301h14m27.475334955s) [noise=true , address=45.91.116.13:4154 , piece_id=2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.014529382 (301h14m27.476621855s) [noise=true , address=45.91.116.6:4975 , piece_id=T3VWH5OBQ7FE66GCKHZLEDSIXFVYEU3V52HI4CPD27QL3RH75K7Q, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.014632619 (301h14m27.476725102s) [noise=true , address=45.91.116.11:4117 , piece_id=QKEYHSK5JOL65H5T2KUEAEK6VKTNCZAYXIEZTMB76PV7P5PEVILQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.015990214 (301h14m27.478082697s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.016291928 (301h14m27.478384412s) [noise=true , address=45.91.116.12:4226 , piece_id=NM7SEIKZYUHZLNVSOGYVZWMI7U7X5SOXOJDAEGY5NF3M6QQFPZSQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.017455915 (301h14m27.479548399s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.018864467 (301h14m27.48095694s) [noise=true , address=77.90.31.10:4518 , piece_id=6FGNAMVK6OAGJTZJNKTUJWGRW7MLGMJMGJKMAINWMPW7BV74OO6A, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.020222022 (301h14m27.482314495s) [noise=true , address=45.91.116.11:4118 , piece_id=VUZB3QRFST2T3HJHMEYYZ6CV7CKJTFWQGNOBDTXHGET2MBUIC4HQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:43.755608897 (301h14m29.21770138s) [upload=265104]: Next returned nil segment Oct 6 02:45:43.755631300 (301h14m29.217723773s) [upload=265104]: Waiting for error group managing segments... Oct 6 02:45:44.043930124 (301h14m29.506022597s) [noise=true , address=145.79.227.19:4077 , piece_id=EJIAPZSBMAO77E33V6MEBOQHUXVVYKMAVP53QAZXZSYBYDFP7MMQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.045501296 (301h14m29.507593769s) [noise=true , address=145.79.227.4:4185 , piece_id=3AD745KFD67M4XFNDXMJVEU34RRBQXXE4P3HVJJRDBHK3IIXNYTA, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.057881199 (301h14m29.519973672s) [noise=true , address=145.79.227.15:4875 , piece_id=PBDTNZ2NWRZZC3VCIQCVR2WLT2AP6662I2SSDNP6BLUDKCJMRK6Q, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.068726540 (301h14m29.530819023s) [noise=true , address=145.79.227.7:4294 , piece_id=7E3B4BGLCTW6C5DM6PPDJPWGVJS5HEGYPAPXRAMX7XDSG7USPQFQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.897895712 (301h14m30.359988185s) [noise=true , address=45.91.116.12:4226 , piece_id=NM7SEIKZYUHZLNVSOGYVZWMI7U7X5SOXOJDAEGY5NF3M6QQFPZSQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.252859665 (301h14m30.714952148s) [noise=true , address=77.90.31.14:4662 , piece_id=MPVBPBB63JILLOUF3N7SRLCZ2F74LX7Z4AH7ZFG3BDNBHT4UIXTQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.318952216 (301h14m30.781044699s) [noise=true , address=45.91.116.6:4974 , piece_id=7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.836439779 (301h14m31.298532262s) [noise=true , address=45.91.116.10:4081 , piece_id=FTXZ2LOHQMY74KOX6XZC3TLHWXOGDTIY3SZLSPTE7THGKNLMPXRQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.975895173 (301h14m31.437987656s) [noise=true , address=45.91.116.6:4975 , piece_id=T3VWH5OBQ7FE66GCKHZLEDSIXFVYEU3V52HI4CPD27QL3RH75K7Q, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:46.826703981 (301h14m32.288796454s) [noise=true , address=77.90.31.10:4518 , piece_id=6FGNAMVK6OAGJTZJNKTUJWGRW7MLGMJMGJKMAINWMPW7BV74OO6A, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:48.069759484 (301h14m33.531851968s) [noise=true , address=45.91.116.13:4154 , piece_id=2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:54.791582362 (301h14m40.253674835s) [noise=true , address=45.91.116.11:4118 , piece_id=VUZB3QRFST2T3HJHMEYYZ6CV7CKJTFWQGNOBDTXHGET2MBUIC4HQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:56.329163998 (301h14m41.791256481s) [noise=true , address=45.91.116.11:4117 , piece_id=QKEYHSK5JOL65H5T2KUEAEK6VKTNCZAYXIEZTMB76PV7P5PEVILQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:56.329201379 (301h14m41.791293862s) [seg_pos={0 0}, upload=265104]: Segment reached optimal threshold of 13 pieces. Oct 6 02:45:56.329362146 (301h14m41.791454629s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Oct 6 02:45:56.329371974 (301h14m41.791464457s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329546396 (301h14m41.791638869s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: write tcp 188.166.225.170:57604->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 02:45:56.329556665 (301h14m41.791649148s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329567937 (301h14m41.791660421s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: rpc: dial tcp 77.90.31.19:4841: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 02:45:56.329579870 (301h14m41.791672353s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329599808 (301h14m41.791692291s) [seg_pos={0 0}, upload=265104]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 6 02:45:56.329683116 (301h14m41.7917756s) [upload=265104]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 6 02:45:56.329690771 (301h14m41.791783254s) [upload=265104]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 6 02:45:56.636341641 (301h14m42.098434134s) [upload=265104]: Done uploading segments... Oct 6 02:45:56.636368131 (301h14m42.098460614s) [upload=265104]: Upload finished. err: <nil> Duration: 5.978582355s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [X.....................X.....................................] duration: 6.372881024s FAIL (2025-10-06 03:15:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: 2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: 7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA) context deadline exceeded ======LOGS====== Oct 6 02:45:41.756337907 (301h14m27.21843039s) [upload=265104]: Starting upload Oct 6 02:45:41.756430433 (301h14m27.218522916s) [upload=265104]: Done starting upload Oct 6 02:45:41.756513701 (301h14m27.218606185s) [upload=265104]: Uploading segments... Oct 6 02:45:41.756752276 (301h14m27.218844759s) [upload=265104]: Got next segment. Inline: false Oct 6 02:45:41.756794536 (301h14m27.218887019s) [upload=265104]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 02:45:41.756800898 (301h14m27.218893381s) [upload=265104]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 02:45:42.009327267 (301h14m27.47141975s) [seg_pos={0 0}, upload=265104]: Begin upload segment... Oct 6 02:45:42.009417419 (301h14m27.471509892s) [seg_pos={0 0}, upload=265104]: Done begin upload segment. Oct 6 02:45:42.009444500 (301h14m27.471536983s) [noise=true , address=77.90.31.14:4662 , piece_id=MPVBPBB63JILLOUF3N7SRLCZ2F74LX7Z4AH7ZFG3BDNBHT4UIXTQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.009561473 (301h14m27.471653956s) [noise=true , address=145.79.227.19:4077 , piece_id=EJIAPZSBMAO77E33V6MEBOQHUXVVYKMAVP53QAZXZSYBYDFP7MMQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.010197574 (301h14m27.472290057s) [noise=true , address=145.79.227.15:4875 , piece_id=PBDTNZ2NWRZZC3VCIQCVR2WLT2AP6662I2SSDNP6BLUDKCJMRK6Q, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.010634216 (301h14m27.472726699s) [noise=true , address=145.79.227.4:4185 , piece_id=3AD745KFD67M4XFNDXMJVEU34RRBQXXE4P3HVJJRDBHK3IIXNYTA, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011073742 (301h14m27.473166225s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011505996 (301h14m27.473598479s) [noise=true , address=45.91.116.10:4081 , piece_id=FTXZ2LOHQMY74KOX6XZC3TLHWXOGDTIY3SZLSPTE7THGKNLMPXRQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011982502 (301h14m27.474074985s) [noise=true , address=45.91.116.6:4974 , piece_id=7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.012488236 (301h14m27.474580719s) [noise=true , address=145.79.227.7:4294 , piece_id=7E3B4BGLCTW6C5DM6PPDJPWGVJS5HEGYPAPXRAMX7XDSG7USPQFQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.013242472 (301h14m27.475334955s) [noise=true , address=45.91.116.13:4154 , piece_id=2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.014529382 (301h14m27.476621855s) [noise=true , address=45.91.116.6:4975 , piece_id=T3VWH5OBQ7FE66GCKHZLEDSIXFVYEU3V52HI4CPD27QL3RH75K7Q, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.014632619 (301h14m27.476725102s) [noise=true , address=45.91.116.11:4117 , piece_id=QKEYHSK5JOL65H5T2KUEAEK6VKTNCZAYXIEZTMB76PV7P5PEVILQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.015990214 (301h14m27.478082697s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.016291928 (301h14m27.478384412s) [noise=true , address=45.91.116.12:4226 , piece_id=NM7SEIKZYUHZLNVSOGYVZWMI7U7X5SOXOJDAEGY5NF3M6QQFPZSQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.017455915 (301h14m27.479548399s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.018864467 (301h14m27.48095694s) [noise=true , address=77.90.31.10:4518 , piece_id=6FGNAMVK6OAGJTZJNKTUJWGRW7MLGMJMGJKMAINWMPW7BV74OO6A, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.020222022 (301h14m27.482314495s) [noise=true , address=45.91.116.11:4118 , piece_id=VUZB3QRFST2T3HJHMEYYZ6CV7CKJTFWQGNOBDTXHGET2MBUIC4HQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:43.755608897 (301h14m29.21770138s) [upload=265104]: Next returned nil segment Oct 6 02:45:43.755631300 (301h14m29.217723773s) [upload=265104]: Waiting for error group managing segments... Oct 6 02:45:44.043930124 (301h14m29.506022597s) [noise=true , address=145.79.227.19:4077 , piece_id=EJIAPZSBMAO77E33V6MEBOQHUXVVYKMAVP53QAZXZSYBYDFP7MMQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.045501296 (301h14m29.507593769s) [noise=true , address=145.79.227.4:4185 , piece_id=3AD745KFD67M4XFNDXMJVEU34RRBQXXE4P3HVJJRDBHK3IIXNYTA, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.057881199 (301h14m29.519973672s) [noise=true , address=145.79.227.15:4875 , piece_id=PBDTNZ2NWRZZC3VCIQCVR2WLT2AP6662I2SSDNP6BLUDKCJMRK6Q, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.068726540 (301h14m29.530819023s) [noise=true , address=145.79.227.7:4294 , piece_id=7E3B4BGLCTW6C5DM6PPDJPWGVJS5HEGYPAPXRAMX7XDSG7USPQFQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.897895712 (301h14m30.359988185s) [noise=true , address=45.91.116.12:4226 , piece_id=NM7SEIKZYUHZLNVSOGYVZWMI7U7X5SOXOJDAEGY5NF3M6QQFPZSQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.252859665 (301h14m30.714952148s) [noise=true , address=77.90.31.14:4662 , piece_id=MPVBPBB63JILLOUF3N7SRLCZ2F74LX7Z4AH7ZFG3BDNBHT4UIXTQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.318952216 (301h14m30.781044699s) [noise=true , address=45.91.116.6:4974 , piece_id=7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.836439779 (301h14m31.298532262s) [noise=true , address=45.91.116.10:4081 , piece_id=FTXZ2LOHQMY74KOX6XZC3TLHWXOGDTIY3SZLSPTE7THGKNLMPXRQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.975895173 (301h14m31.437987656s) [noise=true , address=45.91.116.6:4975 , piece_id=T3VWH5OBQ7FE66GCKHZLEDSIXFVYEU3V52HI4CPD27QL3RH75K7Q, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:46.826703981 (301h14m32.288796454s) [noise=true , address=77.90.31.10:4518 , piece_id=6FGNAMVK6OAGJTZJNKTUJWGRW7MLGMJMGJKMAINWMPW7BV74OO6A, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:48.069759484 (301h14m33.531851968s) [noise=true , address=45.91.116.13:4154 , piece_id=2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:54.791582362 (301h14m40.253674835s) [noise=true , address=45.91.116.11:4118 , piece_id=VUZB3QRFST2T3HJHMEYYZ6CV7CKJTFWQGNOBDTXHGET2MBUIC4HQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:56.329163998 (301h14m41.791256481s) [noise=true , address=45.91.116.11:4117 , piece_id=QKEYHSK5JOL65H5T2KUEAEK6VKTNCZAYXIEZTMB76PV7P5PEVILQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:56.329201379 (301h14m41.791293862s) [seg_pos={0 0}, upload=265104]: Segment reached optimal threshold of 13 pieces. Oct 6 02:45:56.329362146 (301h14m41.791454629s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Oct 6 02:45:56.329371974 (301h14m41.791464457s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329546396 (301h14m41.791638869s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: write tcp 188.166.225.170:57604->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 02:45:56.329556665 (301h14m41.791649148s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329567937 (301h14m41.791660421s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: rpc: dial tcp 77.90.31.19:4841: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 02:45:56.329579870 (301h14m41.791672353s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329599808 (301h14m41.791692291s) [seg_pos={0 0}, upload=265104]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 6 02:45:56.329683116 (301h14m41.7917756s) [upload=265104]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 6 02:45:56.329690771 (301h14m41.791783254s) [upload=265104]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 6 02:45:56.636341641 (301h14m42.098434134s) [upload=265104]: Done uploading segments... Oct 6 02:45:56.636368131 (301h14m42.098460614s) [upload=265104]: Upload finished. err: <nil> Duration: 6.372948122s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X.....................X....................................] duration: 5.97167723s FAIL (2025-10-06 03:14:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: 2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: 7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA) context deadline exceeded ======LOGS====== Oct 6 02:45:41.756337907 (301h14m27.21843039s) [upload=265104]: Starting upload Oct 6 02:45:41.756430433 (301h14m27.218522916s) [upload=265104]: Done starting upload Oct 6 02:45:41.756513701 (301h14m27.218606185s) [upload=265104]: Uploading segments... Oct 6 02:45:41.756752276 (301h14m27.218844759s) [upload=265104]: Got next segment. Inline: false Oct 6 02:45:41.756794536 (301h14m27.218887019s) [upload=265104]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 02:45:41.756800898 (301h14m27.218893381s) [upload=265104]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 02:45:42.009327267 (301h14m27.47141975s) [seg_pos={0 0}, upload=265104]: Begin upload segment... Oct 6 02:45:42.009417419 (301h14m27.471509892s) [seg_pos={0 0}, upload=265104]: Done begin upload segment. Oct 6 02:45:42.009444500 (301h14m27.471536983s) [noise=true , address=77.90.31.14:4662 , piece_id=MPVBPBB63JILLOUF3N7SRLCZ2F74LX7Z4AH7ZFG3BDNBHT4UIXTQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.009561473 (301h14m27.471653956s) [noise=true , address=145.79.227.19:4077 , piece_id=EJIAPZSBMAO77E33V6MEBOQHUXVVYKMAVP53QAZXZSYBYDFP7MMQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.010197574 (301h14m27.472290057s) [noise=true , address=145.79.227.15:4875 , piece_id=PBDTNZ2NWRZZC3VCIQCVR2WLT2AP6662I2SSDNP6BLUDKCJMRK6Q, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.010634216 (301h14m27.472726699s) [noise=true , address=145.79.227.4:4185 , piece_id=3AD745KFD67M4XFNDXMJVEU34RRBQXXE4P3HVJJRDBHK3IIXNYTA, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011073742 (301h14m27.473166225s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011505996 (301h14m27.473598479s) [noise=true , address=45.91.116.10:4081 , piece_id=FTXZ2LOHQMY74KOX6XZC3TLHWXOGDTIY3SZLSPTE7THGKNLMPXRQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011982502 (301h14m27.474074985s) [noise=true , address=45.91.116.6:4974 , piece_id=7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.012488236 (301h14m27.474580719s) [noise=true , address=145.79.227.7:4294 , piece_id=7E3B4BGLCTW6C5DM6PPDJPWGVJS5HEGYPAPXRAMX7XDSG7USPQFQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.013242472 (301h14m27.475334955s) [noise=true , address=45.91.116.13:4154 , piece_id=2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.014529382 (301h14m27.476621855s) [noise=true , address=45.91.116.6:4975 , piece_id=T3VWH5OBQ7FE66GCKHZLEDSIXFVYEU3V52HI4CPD27QL3RH75K7Q, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.014632619 (301h14m27.476725102s) [noise=true , address=45.91.116.11:4117 , piece_id=QKEYHSK5JOL65H5T2KUEAEK6VKTNCZAYXIEZTMB76PV7P5PEVILQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.015990214 (301h14m27.478082697s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.016291928 (301h14m27.478384412s) [noise=true , address=45.91.116.12:4226 , piece_id=NM7SEIKZYUHZLNVSOGYVZWMI7U7X5SOXOJDAEGY5NF3M6QQFPZSQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.017455915 (301h14m27.479548399s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.018864467 (301h14m27.48095694s) [noise=true , address=77.90.31.10:4518 , piece_id=6FGNAMVK6OAGJTZJNKTUJWGRW7MLGMJMGJKMAINWMPW7BV74OO6A, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.020222022 (301h14m27.482314495s) [noise=true , address=45.91.116.11:4118 , piece_id=VUZB3QRFST2T3HJHMEYYZ6CV7CKJTFWQGNOBDTXHGET2MBUIC4HQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:43.755608897 (301h14m29.21770138s) [upload=265104]: Next returned nil segment Oct 6 02:45:43.755631300 (301h14m29.217723773s) [upload=265104]: Waiting for error group managing segments... Oct 6 02:45:44.043930124 (301h14m29.506022597s) [noise=true , address=145.79.227.19:4077 , piece_id=EJIAPZSBMAO77E33V6MEBOQHUXVVYKMAVP53QAZXZSYBYDFP7MMQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.045501296 (301h14m29.507593769s) [noise=true , address=145.79.227.4:4185 , piece_id=3AD745KFD67M4XFNDXMJVEU34RRBQXXE4P3HVJJRDBHK3IIXNYTA, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.057881199 (301h14m29.519973672s) [noise=true , address=145.79.227.15:4875 , piece_id=PBDTNZ2NWRZZC3VCIQCVR2WLT2AP6662I2SSDNP6BLUDKCJMRK6Q, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.068726540 (301h14m29.530819023s) [noise=true , address=145.79.227.7:4294 , piece_id=7E3B4BGLCTW6C5DM6PPDJPWGVJS5HEGYPAPXRAMX7XDSG7USPQFQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.897895712 (301h14m30.359988185s) [noise=true , address=45.91.116.12:4226 , piece_id=NM7SEIKZYUHZLNVSOGYVZWMI7U7X5SOXOJDAEGY5NF3M6QQFPZSQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.252859665 (301h14m30.714952148s) [noise=true , address=77.90.31.14:4662 , piece_id=MPVBPBB63JILLOUF3N7SRLCZ2F74LX7Z4AH7ZFG3BDNBHT4UIXTQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.318952216 (301h14m30.781044699s) [noise=true , address=45.91.116.6:4974 , piece_id=7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.836439779 (301h14m31.298532262s) [noise=true , address=45.91.116.10:4081 , piece_id=FTXZ2LOHQMY74KOX6XZC3TLHWXOGDTIY3SZLSPTE7THGKNLMPXRQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.975895173 (301h14m31.437987656s) [noise=true , address=45.91.116.6:4975 , piece_id=T3VWH5OBQ7FE66GCKHZLEDSIXFVYEU3V52HI4CPD27QL3RH75K7Q, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:46.826703981 (301h14m32.288796454s) [noise=true , address=77.90.31.10:4518 , piece_id=6FGNAMVK6OAGJTZJNKTUJWGRW7MLGMJMGJKMAINWMPW7BV74OO6A, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:48.069759484 (301h14m33.531851968s) [noise=true , address=45.91.116.13:4154 , piece_id=2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:54.791582362 (301h14m40.253674835s) [noise=true , address=45.91.116.11:4118 , piece_id=VUZB3QRFST2T3HJHMEYYZ6CV7CKJTFWQGNOBDTXHGET2MBUIC4HQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:56.329163998 (301h14m41.791256481s) [noise=true , address=45.91.116.11:4117 , piece_id=QKEYHSK5JOL65H5T2KUEAEK6VKTNCZAYXIEZTMB76PV7P5PEVILQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:56.329201379 (301h14m41.791293862s) [seg_pos={0 0}, upload=265104]: Segment reached optimal threshold of 13 pieces. Oct 6 02:45:56.329362146 (301h14m41.791454629s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Oct 6 02:45:56.329371974 (301h14m41.791464457s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329546396 (301h14m41.791638869s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: write tcp 188.166.225.170:57604->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 02:45:56.329556665 (301h14m41.791649148s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329567937 (301h14m41.791660421s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: rpc: dial tcp 77.90.31.19:4841: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 02:45:56.329579870 (301h14m41.791672353s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329599808 (301h14m41.791692291s) [seg_pos={0 0}, upload=265104]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 6 02:45:56.329683116 (301h14m41.7917756s) [upload=265104]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 6 02:45:56.329690771 (301h14m41.791783254s) [upload=265104]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 6 02:45:56.636341641 (301h14m42.098434134s) [upload=265104]: Done uploading segments... Oct 6 02:45:56.636368131 (301h14m42.098460614s) [upload=265104]: Upload finished. err: <nil> Duration: 5.97171971s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..X.....................X...................................] duration: 6.428100543s FAIL (2025-10-06 03:13:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: 2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: 7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA) context deadline exceeded ======LOGS====== Oct 6 02:45:41.756337907 (301h14m27.21843039s) [upload=265104]: Starting upload Oct 6 02:45:41.756430433 (301h14m27.218522916s) [upload=265104]: Done starting upload Oct 6 02:45:41.756513701 (301h14m27.218606185s) [upload=265104]: Uploading segments... Oct 6 02:45:41.756752276 (301h14m27.218844759s) [upload=265104]: Got next segment. Inline: false Oct 6 02:45:41.756794536 (301h14m27.218887019s) [upload=265104]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 02:45:41.756800898 (301h14m27.218893381s) [upload=265104]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 02:45:42.009327267 (301h14m27.47141975s) [seg_pos={0 0}, upload=265104]: Begin upload segment... Oct 6 02:45:42.009417419 (301h14m27.471509892s) [seg_pos={0 0}, upload=265104]: Done begin upload segment. Oct 6 02:45:42.009444500 (301h14m27.471536983s) [noise=true , address=77.90.31.14:4662 , piece_id=MPVBPBB63JILLOUF3N7SRLCZ2F74LX7Z4AH7ZFG3BDNBHT4UIXTQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.009561473 (301h14m27.471653956s) [noise=true , address=145.79.227.19:4077 , piece_id=EJIAPZSBMAO77E33V6MEBOQHUXVVYKMAVP53QAZXZSYBYDFP7MMQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.010197574 (301h14m27.472290057s) [noise=true , address=145.79.227.15:4875 , piece_id=PBDTNZ2NWRZZC3VCIQCVR2WLT2AP6662I2SSDNP6BLUDKCJMRK6Q, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.010634216 (301h14m27.472726699s) [noise=true , address=145.79.227.4:4185 , piece_id=3AD745KFD67M4XFNDXMJVEU34RRBQXXE4P3HVJJRDBHK3IIXNYTA, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011073742 (301h14m27.473166225s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011505996 (301h14m27.473598479s) [noise=true , address=45.91.116.10:4081 , piece_id=FTXZ2LOHQMY74KOX6XZC3TLHWXOGDTIY3SZLSPTE7THGKNLMPXRQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011982502 (301h14m27.474074985s) [noise=true , address=45.91.116.6:4974 , piece_id=7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.012488236 (301h14m27.474580719s) [noise=true , address=145.79.227.7:4294 , piece_id=7E3B4BGLCTW6C5DM6PPDJPWGVJS5HEGYPAPXRAMX7XDSG7USPQFQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.013242472 (301h14m27.475334955s) [noise=true , address=45.91.116.13:4154 , piece_id=2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.014529382 (301h14m27.476621855s) [noise=true , address=45.91.116.6:4975 , piece_id=T3VWH5OBQ7FE66GCKHZLEDSIXFVYEU3V52HI4CPD27QL3RH75K7Q, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.014632619 (301h14m27.476725102s) [noise=true , address=45.91.116.11:4117 , piece_id=QKEYHSK5JOL65H5T2KUEAEK6VKTNCZAYXIEZTMB76PV7P5PEVILQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.015990214 (301h14m27.478082697s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.016291928 (301h14m27.478384412s) [noise=true , address=45.91.116.12:4226 , piece_id=NM7SEIKZYUHZLNVSOGYVZWMI7U7X5SOXOJDAEGY5NF3M6QQFPZSQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.017455915 (301h14m27.479548399s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.018864467 (301h14m27.48095694s) [noise=true , address=77.90.31.10:4518 , piece_id=6FGNAMVK6OAGJTZJNKTUJWGRW7MLGMJMGJKMAINWMPW7BV74OO6A, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.020222022 (301h14m27.482314495s) [noise=true , address=45.91.116.11:4118 , piece_id=VUZB3QRFST2T3HJHMEYYZ6CV7CKJTFWQGNOBDTXHGET2MBUIC4HQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:43.755608897 (301h14m29.21770138s) [upload=265104]: Next returned nil segment Oct 6 02:45:43.755631300 (301h14m29.217723773s) [upload=265104]: Waiting for error group managing segments... Oct 6 02:45:44.043930124 (301h14m29.506022597s) [noise=true , address=145.79.227.19:4077 , piece_id=EJIAPZSBMAO77E33V6MEBOQHUXVVYKMAVP53QAZXZSYBYDFP7MMQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.045501296 (301h14m29.507593769s) [noise=true , address=145.79.227.4:4185 , piece_id=3AD745KFD67M4XFNDXMJVEU34RRBQXXE4P3HVJJRDBHK3IIXNYTA, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.057881199 (301h14m29.519973672s) [noise=true , address=145.79.227.15:4875 , piece_id=PBDTNZ2NWRZZC3VCIQCVR2WLT2AP6662I2SSDNP6BLUDKCJMRK6Q, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.068726540 (301h14m29.530819023s) [noise=true , address=145.79.227.7:4294 , piece_id=7E3B4BGLCTW6C5DM6PPDJPWGVJS5HEGYPAPXRAMX7XDSG7USPQFQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.897895712 (301h14m30.359988185s) [noise=true , address=45.91.116.12:4226 , piece_id=NM7SEIKZYUHZLNVSOGYVZWMI7U7X5SOXOJDAEGY5NF3M6QQFPZSQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.252859665 (301h14m30.714952148s) [noise=true , address=77.90.31.14:4662 , piece_id=MPVBPBB63JILLOUF3N7SRLCZ2F74LX7Z4AH7ZFG3BDNBHT4UIXTQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.318952216 (301h14m30.781044699s) [noise=true , address=45.91.116.6:4974 , piece_id=7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.836439779 (301h14m31.298532262s) [noise=true , address=45.91.116.10:4081 , piece_id=FTXZ2LOHQMY74KOX6XZC3TLHWXOGDTIY3SZLSPTE7THGKNLMPXRQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.975895173 (301h14m31.437987656s) [noise=true , address=45.91.116.6:4975 , piece_id=T3VWH5OBQ7FE66GCKHZLEDSIXFVYEU3V52HI4CPD27QL3RH75K7Q, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:46.826703981 (301h14m32.288796454s) [noise=true , address=77.90.31.10:4518 , piece_id=6FGNAMVK6OAGJTZJNKTUJWGRW7MLGMJMGJKMAINWMPW7BV74OO6A, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:48.069759484 (301h14m33.531851968s) [noise=true , address=45.91.116.13:4154 , piece_id=2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:54.791582362 (301h14m40.253674835s) [noise=true , address=45.91.116.11:4118 , piece_id=VUZB3QRFST2T3HJHMEYYZ6CV7CKJTFWQGNOBDTXHGET2MBUIC4HQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:56.329163998 (301h14m41.791256481s) [noise=true , address=45.91.116.11:4117 , piece_id=QKEYHSK5JOL65H5T2KUEAEK6VKTNCZAYXIEZTMB76PV7P5PEVILQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:56.329201379 (301h14m41.791293862s) [seg_pos={0 0}, upload=265104]: Segment reached optimal threshold of 13 pieces. Oct 6 02:45:56.329362146 (301h14m41.791454629s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Oct 6 02:45:56.329371974 (301h14m41.791464457s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329546396 (301h14m41.791638869s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: write tcp 188.166.225.170:57604->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 02:45:56.329556665 (301h14m41.791649148s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329567937 (301h14m41.791660421s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: rpc: dial tcp 77.90.31.19:4841: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 02:45:56.329579870 (301h14m41.791672353s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329599808 (301h14m41.791692291s) [seg_pos={0 0}, upload=265104]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 6 02:45:56.329683116 (301h14m41.7917756s) [upload=265104]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 6 02:45:56.329690771 (301h14m41.791783254s) [upload=265104]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 6 02:45:56.636341641 (301h14m42.098434134s) [upload=265104]: Done uploading segments... Oct 6 02:45:56.636368131 (301h14m42.098460614s) [upload=265104]: Upload finished. err: <nil> Duration: 6.428128395s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...X.....................X..................................] duration: 6.345281624s FAIL (2025-10-06 03:13:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: 2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: 7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA) context deadline exceeded ======LOGS====== Oct 6 02:45:41.756337907 (301h14m27.21843039s) [upload=265104]: Starting upload Oct 6 02:45:41.756430433 (301h14m27.218522916s) [upload=265104]: Done starting upload Oct 6 02:45:41.756513701 (301h14m27.218606185s) [upload=265104]: Uploading segments... Oct 6 02:45:41.756752276 (301h14m27.218844759s) [upload=265104]: Got next segment. Inline: false Oct 6 02:45:41.756794536 (301h14m27.218887019s) [upload=265104]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 02:45:41.756800898 (301h14m27.218893381s) [upload=265104]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 02:45:42.009327267 (301h14m27.47141975s) [seg_pos={0 0}, upload=265104]: Begin upload segment... Oct 6 02:45:42.009417419 (301h14m27.471509892s) [seg_pos={0 0}, upload=265104]: Done begin upload segment. Oct 6 02:45:42.009444500 (301h14m27.471536983s) [noise=true , address=77.90.31.14:4662 , piece_id=MPVBPBB63JILLOUF3N7SRLCZ2F74LX7Z4AH7ZFG3BDNBHT4UIXTQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.009561473 (301h14m27.471653956s) [noise=true , address=145.79.227.19:4077 , piece_id=EJIAPZSBMAO77E33V6MEBOQHUXVVYKMAVP53QAZXZSYBYDFP7MMQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.010197574 (301h14m27.472290057s) [noise=true , address=145.79.227.15:4875 , piece_id=PBDTNZ2NWRZZC3VCIQCVR2WLT2AP6662I2SSDNP6BLUDKCJMRK6Q, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.010634216 (301h14m27.472726699s) [noise=true , address=145.79.227.4:4185 , piece_id=3AD745KFD67M4XFNDXMJVEU34RRBQXXE4P3HVJJRDBHK3IIXNYTA, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011073742 (301h14m27.473166225s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011505996 (301h14m27.473598479s) [noise=true , address=45.91.116.10:4081 , piece_id=FTXZ2LOHQMY74KOX6XZC3TLHWXOGDTIY3SZLSPTE7THGKNLMPXRQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011982502 (301h14m27.474074985s) [noise=true , address=45.91.116.6:4974 , piece_id=7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.012488236 (301h14m27.474580719s) [noise=true , address=145.79.227.7:4294 , piece_id=7E3B4BGLCTW6C5DM6PPDJPWGVJS5HEGYPAPXRAMX7XDSG7USPQFQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.013242472 (301h14m27.475334955s) [noise=true , address=45.91.116.13:4154 , piece_id=2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.014529382 (301h14m27.476621855s) [noise=true , address=45.91.116.6:4975 , piece_id=T3VWH5OBQ7FE66GCKHZLEDSIXFVYEU3V52HI4CPD27QL3RH75K7Q, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.014632619 (301h14m27.476725102s) [noise=true , address=45.91.116.11:4117 , piece_id=QKEYHSK5JOL65H5T2KUEAEK6VKTNCZAYXIEZTMB76PV7P5PEVILQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.015990214 (301h14m27.478082697s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.016291928 (301h14m27.478384412s) [noise=true , address=45.91.116.12:4226 , piece_id=NM7SEIKZYUHZLNVSOGYVZWMI7U7X5SOXOJDAEGY5NF3M6QQFPZSQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.017455915 (301h14m27.479548399s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.018864467 (301h14m27.48095694s) [noise=true , address=77.90.31.10:4518 , piece_id=6FGNAMVK6OAGJTZJNKTUJWGRW7MLGMJMGJKMAINWMPW7BV74OO6A, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.020222022 (301h14m27.482314495s) [noise=true , address=45.91.116.11:4118 , piece_id=VUZB3QRFST2T3HJHMEYYZ6CV7CKJTFWQGNOBDTXHGET2MBUIC4HQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:43.755608897 (301h14m29.21770138s) [upload=265104]: Next returned nil segment Oct 6 02:45:43.755631300 (301h14m29.217723773s) [upload=265104]: Waiting for error group managing segments... Oct 6 02:45:44.043930124 (301h14m29.506022597s) [noise=true , address=145.79.227.19:4077 , piece_id=EJIAPZSBMAO77E33V6MEBOQHUXVVYKMAVP53QAZXZSYBYDFP7MMQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.045501296 (301h14m29.507593769s) [noise=true , address=145.79.227.4:4185 , piece_id=3AD745KFD67M4XFNDXMJVEU34RRBQXXE4P3HVJJRDBHK3IIXNYTA, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.057881199 (301h14m29.519973672s) [noise=true , address=145.79.227.15:4875 , piece_id=PBDTNZ2NWRZZC3VCIQCVR2WLT2AP6662I2SSDNP6BLUDKCJMRK6Q, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.068726540 (301h14m29.530819023s) [noise=true , address=145.79.227.7:4294 , piece_id=7E3B4BGLCTW6C5DM6PPDJPWGVJS5HEGYPAPXRAMX7XDSG7USPQFQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.897895712 (301h14m30.359988185s) [noise=true , address=45.91.116.12:4226 , piece_id=NM7SEIKZYUHZLNVSOGYVZWMI7U7X5SOXOJDAEGY5NF3M6QQFPZSQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.252859665 (301h14m30.714952148s) [noise=true , address=77.90.31.14:4662 , piece_id=MPVBPBB63JILLOUF3N7SRLCZ2F74LX7Z4AH7ZFG3BDNBHT4UIXTQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.318952216 (301h14m30.781044699s) [noise=true , address=45.91.116.6:4974 , piece_id=7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.836439779 (301h14m31.298532262s) [noise=true , address=45.91.116.10:4081 , piece_id=FTXZ2LOHQMY74KOX6XZC3TLHWXOGDTIY3SZLSPTE7THGKNLMPXRQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.975895173 (301h14m31.437987656s) [noise=true , address=45.91.116.6:4975 , piece_id=T3VWH5OBQ7FE66GCKHZLEDSIXFVYEU3V52HI4CPD27QL3RH75K7Q, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:46.826703981 (301h14m32.288796454s) [noise=true , address=77.90.31.10:4518 , piece_id=6FGNAMVK6OAGJTZJNKTUJWGRW7MLGMJMGJKMAINWMPW7BV74OO6A, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:48.069759484 (301h14m33.531851968s) [noise=true , address=45.91.116.13:4154 , piece_id=2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:54.791582362 (301h14m40.253674835s) [noise=true , address=45.91.116.11:4118 , piece_id=VUZB3QRFST2T3HJHMEYYZ6CV7CKJTFWQGNOBDTXHGET2MBUIC4HQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:56.329163998 (301h14m41.791256481s) [noise=true , address=45.91.116.11:4117 , piece_id=QKEYHSK5JOL65H5T2KUEAEK6VKTNCZAYXIEZTMB76PV7P5PEVILQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:56.329201379 (301h14m41.791293862s) [seg_pos={0 0}, upload=265104]: Segment reached optimal threshold of 13 pieces. Oct 6 02:45:56.329362146 (301h14m41.791454629s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Oct 6 02:45:56.329371974 (301h14m41.791464457s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329546396 (301h14m41.791638869s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: write tcp 188.166.225.170:57604->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 02:45:56.329556665 (301h14m41.791649148s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329567937 (301h14m41.791660421s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: rpc: dial tcp 77.90.31.19:4841: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 02:45:56.329579870 (301h14m41.791672353s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329599808 (301h14m41.791692291s) [seg_pos={0 0}, upload=265104]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 6 02:45:56.329683116 (301h14m41.7917756s) [upload=265104]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 6 02:45:56.329690771 (301h14m41.791783254s) [upload=265104]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 6 02:45:56.636341641 (301h14m42.098434134s) [upload=265104]: Done uploading segments... Oct 6 02:45:56.636368131 (301h14m42.098460614s) [upload=265104]: Upload finished. err: <nil> Duration: 6.345309728s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....X.....................X.................................] duration: 6.069808092s FAIL (2025-10-06 03:12:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: 2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: 7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA) context deadline exceeded ======LOGS====== Oct 6 02:45:41.756337907 (301h14m27.21843039s) [upload=265104]: Starting upload Oct 6 02:45:41.756430433 (301h14m27.218522916s) [upload=265104]: Done starting upload Oct 6 02:45:41.756513701 (301h14m27.218606185s) [upload=265104]: Uploading segments... Oct 6 02:45:41.756752276 (301h14m27.218844759s) [upload=265104]: Got next segment. Inline: false Oct 6 02:45:41.756794536 (301h14m27.218887019s) [upload=265104]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 02:45:41.756800898 (301h14m27.218893381s) [upload=265104]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 02:45:42.009327267 (301h14m27.47141975s) [seg_pos={0 0}, upload=265104]: Begin upload segment... Oct 6 02:45:42.009417419 (301h14m27.471509892s) [seg_pos={0 0}, upload=265104]: Done begin upload segment. Oct 6 02:45:42.009444500 (301h14m27.471536983s) [noise=true , address=77.90.31.14:4662 , piece_id=MPVBPBB63JILLOUF3N7SRLCZ2F74LX7Z4AH7ZFG3BDNBHT4UIXTQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.009561473 (301h14m27.471653956s) [noise=true , address=145.79.227.19:4077 , piece_id=EJIAPZSBMAO77E33V6MEBOQHUXVVYKMAVP53QAZXZSYBYDFP7MMQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.010197574 (301h14m27.472290057s) [noise=true , address=145.79.227.15:4875 , piece_id=PBDTNZ2NWRZZC3VCIQCVR2WLT2AP6662I2SSDNP6BLUDKCJMRK6Q, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.010634216 (301h14m27.472726699s) [noise=true , address=145.79.227.4:4185 , piece_id=3AD745KFD67M4XFNDXMJVEU34RRBQXXE4P3HVJJRDBHK3IIXNYTA, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011073742 (301h14m27.473166225s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011505996 (301h14m27.473598479s) [noise=true , address=45.91.116.10:4081 , piece_id=FTXZ2LOHQMY74KOX6XZC3TLHWXOGDTIY3SZLSPTE7THGKNLMPXRQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011982502 (301h14m27.474074985s) [noise=true , address=45.91.116.6:4974 , piece_id=7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.012488236 (301h14m27.474580719s) [noise=true , address=145.79.227.7:4294 , piece_id=7E3B4BGLCTW6C5DM6PPDJPWGVJS5HEGYPAPXRAMX7XDSG7USPQFQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.013242472 (301h14m27.475334955s) [noise=true , address=45.91.116.13:4154 , piece_id=2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.014529382 (301h14m27.476621855s) [noise=true , address=45.91.116.6:4975 , piece_id=T3VWH5OBQ7FE66GCKHZLEDSIXFVYEU3V52HI4CPD27QL3RH75K7Q, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.014632619 (301h14m27.476725102s) [noise=true , address=45.91.116.11:4117 , piece_id=QKEYHSK5JOL65H5T2KUEAEK6VKTNCZAYXIEZTMB76PV7P5PEVILQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.015990214 (301h14m27.478082697s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.016291928 (301h14m27.478384412s) [noise=true , address=45.91.116.12:4226 , piece_id=NM7SEIKZYUHZLNVSOGYVZWMI7U7X5SOXOJDAEGY5NF3M6QQFPZSQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.017455915 (301h14m27.479548399s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.018864467 (301h14m27.48095694s) [noise=true , address=77.90.31.10:4518 , piece_id=6FGNAMVK6OAGJTZJNKTUJWGRW7MLGMJMGJKMAINWMPW7BV74OO6A, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.020222022 (301h14m27.482314495s) [noise=true , address=45.91.116.11:4118 , piece_id=VUZB3QRFST2T3HJHMEYYZ6CV7CKJTFWQGNOBDTXHGET2MBUIC4HQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:43.755608897 (301h14m29.21770138s) [upload=265104]: Next returned nil segment Oct 6 02:45:43.755631300 (301h14m29.217723773s) [upload=265104]: Waiting for error group managing segments... Oct 6 02:45:44.043930124 (301h14m29.506022597s) [noise=true , address=145.79.227.19:4077 , piece_id=EJIAPZSBMAO77E33V6MEBOQHUXVVYKMAVP53QAZXZSYBYDFP7MMQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.045501296 (301h14m29.507593769s) [noise=true , address=145.79.227.4:4185 , piece_id=3AD745KFD67M4XFNDXMJVEU34RRBQXXE4P3HVJJRDBHK3IIXNYTA, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.057881199 (301h14m29.519973672s) [noise=true , address=145.79.227.15:4875 , piece_id=PBDTNZ2NWRZZC3VCIQCVR2WLT2AP6662I2SSDNP6BLUDKCJMRK6Q, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.068726540 (301h14m29.530819023s) [noise=true , address=145.79.227.7:4294 , piece_id=7E3B4BGLCTW6C5DM6PPDJPWGVJS5HEGYPAPXRAMX7XDSG7USPQFQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.897895712 (301h14m30.359988185s) [noise=true , address=45.91.116.12:4226 , piece_id=NM7SEIKZYUHZLNVSOGYVZWMI7U7X5SOXOJDAEGY5NF3M6QQFPZSQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.252859665 (301h14m30.714952148s) [noise=true , address=77.90.31.14:4662 , piece_id=MPVBPBB63JILLOUF3N7SRLCZ2F74LX7Z4AH7ZFG3BDNBHT4UIXTQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.318952216 (301h14m30.781044699s) [noise=true , address=45.91.116.6:4974 , piece_id=7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.836439779 (301h14m31.298532262s) [noise=true , address=45.91.116.10:4081 , piece_id=FTXZ2LOHQMY74KOX6XZC3TLHWXOGDTIY3SZLSPTE7THGKNLMPXRQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.975895173 (301h14m31.437987656s) [noise=true , address=45.91.116.6:4975 , piece_id=T3VWH5OBQ7FE66GCKHZLEDSIXFVYEU3V52HI4CPD27QL3RH75K7Q, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:46.826703981 (301h14m32.288796454s) [noise=true , address=77.90.31.10:4518 , piece_id=6FGNAMVK6OAGJTZJNKTUJWGRW7MLGMJMGJKMAINWMPW7BV74OO6A, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:48.069759484 (301h14m33.531851968s) [noise=true , address=45.91.116.13:4154 , piece_id=2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:54.791582362 (301h14m40.253674835s) [noise=true , address=45.91.116.11:4118 , piece_id=VUZB3QRFST2T3HJHMEYYZ6CV7CKJTFWQGNOBDTXHGET2MBUIC4HQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:56.329163998 (301h14m41.791256481s) [noise=true , address=45.91.116.11:4117 , piece_id=QKEYHSK5JOL65H5T2KUEAEK6VKTNCZAYXIEZTMB76PV7P5PEVILQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:56.329201379 (301h14m41.791293862s) [seg_pos={0 0}, upload=265104]: Segment reached optimal threshold of 13 pieces. Oct 6 02:45:56.329362146 (301h14m41.791454629s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Oct 6 02:45:56.329371974 (301h14m41.791464457s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329546396 (301h14m41.791638869s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: write tcp 188.166.225.170:57604->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 02:45:56.329556665 (301h14m41.791649148s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329567937 (301h14m41.791660421s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: rpc: dial tcp 77.90.31.19:4841: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 02:45:56.329579870 (301h14m41.791672353s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329599808 (301h14m41.791692291s) [seg_pos={0 0}, upload=265104]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 6 02:45:56.329683116 (301h14m41.7917756s) [upload=265104]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 6 02:45:56.329690771 (301h14m41.791783254s) [upload=265104]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 6 02:45:56.636341641 (301h14m42.098434134s) [upload=265104]: Done uploading segments... Oct 6 02:45:56.636368131 (301h14m42.098460614s) [upload=265104]: Upload finished. err: <nil> Duration: 6.069840905s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....X.....................X................................] duration: 6.292299367s FAIL (2025-10-06 03:11:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: 2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: 7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA) context deadline exceeded ======LOGS====== Oct 6 02:45:41.756337907 (301h14m27.21843039s) [upload=265104]: Starting upload Oct 6 02:45:41.756430433 (301h14m27.218522916s) [upload=265104]: Done starting upload Oct 6 02:45:41.756513701 (301h14m27.218606185s) [upload=265104]: Uploading segments... Oct 6 02:45:41.756752276 (301h14m27.218844759s) [upload=265104]: Got next segment. Inline: false Oct 6 02:45:41.756794536 (301h14m27.218887019s) [upload=265104]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 02:45:41.756800898 (301h14m27.218893381s) [upload=265104]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 02:45:42.009327267 (301h14m27.47141975s) [seg_pos={0 0}, upload=265104]: Begin upload segment... Oct 6 02:45:42.009417419 (301h14m27.471509892s) [seg_pos={0 0}, upload=265104]: Done begin upload segment. Oct 6 02:45:42.009444500 (301h14m27.471536983s) [noise=true , address=77.90.31.14:4662 , piece_id=MPVBPBB63JILLOUF3N7SRLCZ2F74LX7Z4AH7ZFG3BDNBHT4UIXTQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.009561473 (301h14m27.471653956s) [noise=true , address=145.79.227.19:4077 , piece_id=EJIAPZSBMAO77E33V6MEBOQHUXVVYKMAVP53QAZXZSYBYDFP7MMQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.010197574 (301h14m27.472290057s) [noise=true , address=145.79.227.15:4875 , piece_id=PBDTNZ2NWRZZC3VCIQCVR2WLT2AP6662I2SSDNP6BLUDKCJMRK6Q, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.010634216 (301h14m27.472726699s) [noise=true , address=145.79.227.4:4185 , piece_id=3AD745KFD67M4XFNDXMJVEU34RRBQXXE4P3HVJJRDBHK3IIXNYTA, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011073742 (301h14m27.473166225s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011505996 (301h14m27.473598479s) [noise=true , address=45.91.116.10:4081 , piece_id=FTXZ2LOHQMY74KOX6XZC3TLHWXOGDTIY3SZLSPTE7THGKNLMPXRQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011982502 (301h14m27.474074985s) [noise=true , address=45.91.116.6:4974 , piece_id=7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.012488236 (301h14m27.474580719s) [noise=true , address=145.79.227.7:4294 , piece_id=7E3B4BGLCTW6C5DM6PPDJPWGVJS5HEGYPAPXRAMX7XDSG7USPQFQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.013242472 (301h14m27.475334955s) [noise=true , address=45.91.116.13:4154 , piece_id=2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.014529382 (301h14m27.476621855s) [noise=true , address=45.91.116.6:4975 , piece_id=T3VWH5OBQ7FE66GCKHZLEDSIXFVYEU3V52HI4CPD27QL3RH75K7Q, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.014632619 (301h14m27.476725102s) [noise=true , address=45.91.116.11:4117 , piece_id=QKEYHSK5JOL65H5T2KUEAEK6VKTNCZAYXIEZTMB76PV7P5PEVILQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.015990214 (301h14m27.478082697s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.016291928 (301h14m27.478384412s) [noise=true , address=45.91.116.12:4226 , piece_id=NM7SEIKZYUHZLNVSOGYVZWMI7U7X5SOXOJDAEGY5NF3M6QQFPZSQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.017455915 (301h14m27.479548399s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.018864467 (301h14m27.48095694s) [noise=true , address=77.90.31.10:4518 , piece_id=6FGNAMVK6OAGJTZJNKTUJWGRW7MLGMJMGJKMAINWMPW7BV74OO6A, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.020222022 (301h14m27.482314495s) [noise=true , address=45.91.116.11:4118 , piece_id=VUZB3QRFST2T3HJHMEYYZ6CV7CKJTFWQGNOBDTXHGET2MBUIC4HQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:43.755608897 (301h14m29.21770138s) [upload=265104]: Next returned nil segment Oct 6 02:45:43.755631300 (301h14m29.217723773s) [upload=265104]: Waiting for error group managing segments... Oct 6 02:45:44.043930124 (301h14m29.506022597s) [noise=true , address=145.79.227.19:4077 , piece_id=EJIAPZSBMAO77E33V6MEBOQHUXVVYKMAVP53QAZXZSYBYDFP7MMQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.045501296 (301h14m29.507593769s) [noise=true , address=145.79.227.4:4185 , piece_id=3AD745KFD67M4XFNDXMJVEU34RRBQXXE4P3HVJJRDBHK3IIXNYTA, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.057881199 (301h14m29.519973672s) [noise=true , address=145.79.227.15:4875 , piece_id=PBDTNZ2NWRZZC3VCIQCVR2WLT2AP6662I2SSDNP6BLUDKCJMRK6Q, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.068726540 (301h14m29.530819023s) [noise=true , address=145.79.227.7:4294 , piece_id=7E3B4BGLCTW6C5DM6PPDJPWGVJS5HEGYPAPXRAMX7XDSG7USPQFQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.897895712 (301h14m30.359988185s) [noise=true , address=45.91.116.12:4226 , piece_id=NM7SEIKZYUHZLNVSOGYVZWMI7U7X5SOXOJDAEGY5NF3M6QQFPZSQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.252859665 (301h14m30.714952148s) [noise=true , address=77.90.31.14:4662 , piece_id=MPVBPBB63JILLOUF3N7SRLCZ2F74LX7Z4AH7ZFG3BDNBHT4UIXTQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.318952216 (301h14m30.781044699s) [noise=true , address=45.91.116.6:4974 , piece_id=7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.836439779 (301h14m31.298532262s) [noise=true , address=45.91.116.10:4081 , piece_id=FTXZ2LOHQMY74KOX6XZC3TLHWXOGDTIY3SZLSPTE7THGKNLMPXRQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.975895173 (301h14m31.437987656s) [noise=true , address=45.91.116.6:4975 , piece_id=T3VWH5OBQ7FE66GCKHZLEDSIXFVYEU3V52HI4CPD27QL3RH75K7Q, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:46.826703981 (301h14m32.288796454s) [noise=true , address=77.90.31.10:4518 , piece_id=6FGNAMVK6OAGJTZJNKTUJWGRW7MLGMJMGJKMAINWMPW7BV74OO6A, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:48.069759484 (301h14m33.531851968s) [noise=true , address=45.91.116.13:4154 , piece_id=2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:54.791582362 (301h14m40.253674835s) [noise=true , address=45.91.116.11:4118 , piece_id=VUZB3QRFST2T3HJHMEYYZ6CV7CKJTFWQGNOBDTXHGET2MBUIC4HQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:56.329163998 (301h14m41.791256481s) [noise=true , address=45.91.116.11:4117 , piece_id=QKEYHSK5JOL65H5T2KUEAEK6VKTNCZAYXIEZTMB76PV7P5PEVILQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:56.329201379 (301h14m41.791293862s) [seg_pos={0 0}, upload=265104]: Segment reached optimal threshold of 13 pieces. Oct 6 02:45:56.329362146 (301h14m41.791454629s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Oct 6 02:45:56.329371974 (301h14m41.791464457s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329546396 (301h14m41.791638869s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: write tcp 188.166.225.170:57604->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 02:45:56.329556665 (301h14m41.791649148s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329567937 (301h14m41.791660421s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: rpc: dial tcp 77.90.31.19:4841: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 02:45:56.329579870 (301h14m41.791672353s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329599808 (301h14m41.791692291s) [seg_pos={0 0}, upload=265104]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 6 02:45:56.329683116 (301h14m41.7917756s) [upload=265104]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 6 02:45:56.329690771 (301h14m41.791783254s) [upload=265104]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 6 02:45:56.636341641 (301h14m42.098434134s) [upload=265104]: Done uploading segments... Oct 6 02:45:56.636368131 (301h14m42.098460614s) [upload=265104]: Upload finished. err: <nil> Duration: 6.292330497s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......X.....................X...............................] duration: 6.136674023s FAIL (2025-10-06 03:11:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: 2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: 7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA) context deadline exceeded ======LOGS====== Oct 6 02:45:41.756337907 (301h14m27.21843039s) [upload=265104]: Starting upload Oct 6 02:45:41.756430433 (301h14m27.218522916s) [upload=265104]: Done starting upload Oct 6 02:45:41.756513701 (301h14m27.218606185s) [upload=265104]: Uploading segments... Oct 6 02:45:41.756752276 (301h14m27.218844759s) [upload=265104]: Got next segment. Inline: false Oct 6 02:45:41.756794536 (301h14m27.218887019s) [upload=265104]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 02:45:41.756800898 (301h14m27.218893381s) [upload=265104]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 02:45:42.009327267 (301h14m27.47141975s) [seg_pos={0 0}, upload=265104]: Begin upload segment... Oct 6 02:45:42.009417419 (301h14m27.471509892s) [seg_pos={0 0}, upload=265104]: Done begin upload segment. Oct 6 02:45:42.009444500 (301h14m27.471536983s) [noise=true , address=77.90.31.14:4662 , piece_id=MPVBPBB63JILLOUF3N7SRLCZ2F74LX7Z4AH7ZFG3BDNBHT4UIXTQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.009561473 (301h14m27.471653956s) [noise=true , address=145.79.227.19:4077 , piece_id=EJIAPZSBMAO77E33V6MEBOQHUXVVYKMAVP53QAZXZSYBYDFP7MMQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.010197574 (301h14m27.472290057s) [noise=true , address=145.79.227.15:4875 , piece_id=PBDTNZ2NWRZZC3VCIQCVR2WLT2AP6662I2SSDNP6BLUDKCJMRK6Q, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.010634216 (301h14m27.472726699s) [noise=true , address=145.79.227.4:4185 , piece_id=3AD745KFD67M4XFNDXMJVEU34RRBQXXE4P3HVJJRDBHK3IIXNYTA, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011073742 (301h14m27.473166225s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011505996 (301h14m27.473598479s) [noise=true , address=45.91.116.10:4081 , piece_id=FTXZ2LOHQMY74KOX6XZC3TLHWXOGDTIY3SZLSPTE7THGKNLMPXRQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011982502 (301h14m27.474074985s) [noise=true , address=45.91.116.6:4974 , piece_id=7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.012488236 (301h14m27.474580719s) [noise=true , address=145.79.227.7:4294 , piece_id=7E3B4BGLCTW6C5DM6PPDJPWGVJS5HEGYPAPXRAMX7XDSG7USPQFQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.013242472 (301h14m27.475334955s) [noise=true , address=45.91.116.13:4154 , piece_id=2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.014529382 (301h14m27.476621855s) [noise=true , address=45.91.116.6:4975 , piece_id=T3VWH5OBQ7FE66GCKHZLEDSIXFVYEU3V52HI4CPD27QL3RH75K7Q, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.014632619 (301h14m27.476725102s) [noise=true , address=45.91.116.11:4117 , piece_id=QKEYHSK5JOL65H5T2KUEAEK6VKTNCZAYXIEZTMB76PV7P5PEVILQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.015990214 (301h14m27.478082697s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.016291928 (301h14m27.478384412s) [noise=true , address=45.91.116.12:4226 , piece_id=NM7SEIKZYUHZLNVSOGYVZWMI7U7X5SOXOJDAEGY5NF3M6QQFPZSQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.017455915 (301h14m27.479548399s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.018864467 (301h14m27.48095694s) [noise=true , address=77.90.31.10:4518 , piece_id=6FGNAMVK6OAGJTZJNKTUJWGRW7MLGMJMGJKMAINWMPW7BV74OO6A, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.020222022 (301h14m27.482314495s) [noise=true , address=45.91.116.11:4118 , piece_id=VUZB3QRFST2T3HJHMEYYZ6CV7CKJTFWQGNOBDTXHGET2MBUIC4HQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:43.755608897 (301h14m29.21770138s) [upload=265104]: Next returned nil segment Oct 6 02:45:43.755631300 (301h14m29.217723773s) [upload=265104]: Waiting for error group managing segments... Oct 6 02:45:44.043930124 (301h14m29.506022597s) [noise=true , address=145.79.227.19:4077 , piece_id=EJIAPZSBMAO77E33V6MEBOQHUXVVYKMAVP53QAZXZSYBYDFP7MMQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.045501296 (301h14m29.507593769s) [noise=true , address=145.79.227.4:4185 , piece_id=3AD745KFD67M4XFNDXMJVEU34RRBQXXE4P3HVJJRDBHK3IIXNYTA, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.057881199 (301h14m29.519973672s) [noise=true , address=145.79.227.15:4875 , piece_id=PBDTNZ2NWRZZC3VCIQCVR2WLT2AP6662I2SSDNP6BLUDKCJMRK6Q, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.068726540 (301h14m29.530819023s) [noise=true , address=145.79.227.7:4294 , piece_id=7E3B4BGLCTW6C5DM6PPDJPWGVJS5HEGYPAPXRAMX7XDSG7USPQFQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.897895712 (301h14m30.359988185s) [noise=true , address=45.91.116.12:4226 , piece_id=NM7SEIKZYUHZLNVSOGYVZWMI7U7X5SOXOJDAEGY5NF3M6QQFPZSQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.252859665 (301h14m30.714952148s) [noise=true , address=77.90.31.14:4662 , piece_id=MPVBPBB63JILLOUF3N7SRLCZ2F74LX7Z4AH7ZFG3BDNBHT4UIXTQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.318952216 (301h14m30.781044699s) [noise=true , address=45.91.116.6:4974 , piece_id=7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.836439779 (301h14m31.298532262s) [noise=true , address=45.91.116.10:4081 , piece_id=FTXZ2LOHQMY74KOX6XZC3TLHWXOGDTIY3SZLSPTE7THGKNLMPXRQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.975895173 (301h14m31.437987656s) [noise=true , address=45.91.116.6:4975 , piece_id=T3VWH5OBQ7FE66GCKHZLEDSIXFVYEU3V52HI4CPD27QL3RH75K7Q, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:46.826703981 (301h14m32.288796454s) [noise=true , address=77.90.31.10:4518 , piece_id=6FGNAMVK6OAGJTZJNKTUJWGRW7MLGMJMGJKMAINWMPW7BV74OO6A, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:48.069759484 (301h14m33.531851968s) [noise=true , address=45.91.116.13:4154 , piece_id=2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:54.791582362 (301h14m40.253674835s) [noise=true , address=45.91.116.11:4118 , piece_id=VUZB3QRFST2T3HJHMEYYZ6CV7CKJTFWQGNOBDTXHGET2MBUIC4HQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:56.329163998 (301h14m41.791256481s) [noise=true , address=45.91.116.11:4117 , piece_id=QKEYHSK5JOL65H5T2KUEAEK6VKTNCZAYXIEZTMB76PV7P5PEVILQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:56.329201379 (301h14m41.791293862s) [seg_pos={0 0}, upload=265104]: Segment reached optimal threshold of 13 pieces. Oct 6 02:45:56.329362146 (301h14m41.791454629s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Oct 6 02:45:56.329371974 (301h14m41.791464457s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329546396 (301h14m41.791638869s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: write tcp 188.166.225.170:57604->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 02:45:56.329556665 (301h14m41.791649148s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329567937 (301h14m41.791660421s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: rpc: dial tcp 77.90.31.19:4841: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 02:45:56.329579870 (301h14m41.791672353s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329599808 (301h14m41.791692291s) [seg_pos={0 0}, upload=265104]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 6 02:45:56.329683116 (301h14m41.7917756s) [upload=265104]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 6 02:45:56.329690771 (301h14m41.791783254s) [upload=265104]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 6 02:45:56.636341641 (301h14m42.098434134s) [upload=265104]: Done uploading segments... Oct 6 02:45:56.636368131 (301h14m42.098460614s) [upload=265104]: Upload finished. err: <nil> Duration: 6.136712066s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......X.....................X..............................] duration: 6.155118451s FAIL (2025-10-06 03:10:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: 2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: 7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA) context deadline exceeded ======LOGS====== Oct 6 02:45:41.756337907 (301h14m27.21843039s) [upload=265104]: Starting upload Oct 6 02:45:41.756430433 (301h14m27.218522916s) [upload=265104]: Done starting upload Oct 6 02:45:41.756513701 (301h14m27.218606185s) [upload=265104]: Uploading segments... Oct 6 02:45:41.756752276 (301h14m27.218844759s) [upload=265104]: Got next segment. Inline: false Oct 6 02:45:41.756794536 (301h14m27.218887019s) [upload=265104]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 02:45:41.756800898 (301h14m27.218893381s) [upload=265104]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 02:45:42.009327267 (301h14m27.47141975s) [seg_pos={0 0}, upload=265104]: Begin upload segment... Oct 6 02:45:42.009417419 (301h14m27.471509892s) [seg_pos={0 0}, upload=265104]: Done begin upload segment. Oct 6 02:45:42.009444500 (301h14m27.471536983s) [noise=true , address=77.90.31.14:4662 , piece_id=MPVBPBB63JILLOUF3N7SRLCZ2F74LX7Z4AH7ZFG3BDNBHT4UIXTQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.009561473 (301h14m27.471653956s) [noise=true , address=145.79.227.19:4077 , piece_id=EJIAPZSBMAO77E33V6MEBOQHUXVVYKMAVP53QAZXZSYBYDFP7MMQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.010197574 (301h14m27.472290057s) [noise=true , address=145.79.227.15:4875 , piece_id=PBDTNZ2NWRZZC3VCIQCVR2WLT2AP6662I2SSDNP6BLUDKCJMRK6Q, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.010634216 (301h14m27.472726699s) [noise=true , address=145.79.227.4:4185 , piece_id=3AD745KFD67M4XFNDXMJVEU34RRBQXXE4P3HVJJRDBHK3IIXNYTA, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011073742 (301h14m27.473166225s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011505996 (301h14m27.473598479s) [noise=true , address=45.91.116.10:4081 , piece_id=FTXZ2LOHQMY74KOX6XZC3TLHWXOGDTIY3SZLSPTE7THGKNLMPXRQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011982502 (301h14m27.474074985s) [noise=true , address=45.91.116.6:4974 , piece_id=7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.012488236 (301h14m27.474580719s) [noise=true , address=145.79.227.7:4294 , piece_id=7E3B4BGLCTW6C5DM6PPDJPWGVJS5HEGYPAPXRAMX7XDSG7USPQFQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.013242472 (301h14m27.475334955s) [noise=true , address=45.91.116.13:4154 , piece_id=2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.014529382 (301h14m27.476621855s) [noise=true , address=45.91.116.6:4975 , piece_id=T3VWH5OBQ7FE66GCKHZLEDSIXFVYEU3V52HI4CPD27QL3RH75K7Q, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.014632619 (301h14m27.476725102s) [noise=true , address=45.91.116.11:4117 , piece_id=QKEYHSK5JOL65H5T2KUEAEK6VKTNCZAYXIEZTMB76PV7P5PEVILQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.015990214 (301h14m27.478082697s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.016291928 (301h14m27.478384412s) [noise=true , address=45.91.116.12:4226 , piece_id=NM7SEIKZYUHZLNVSOGYVZWMI7U7X5SOXOJDAEGY5NF3M6QQFPZSQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.017455915 (301h14m27.479548399s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.018864467 (301h14m27.48095694s) [noise=true , address=77.90.31.10:4518 , piece_id=6FGNAMVK6OAGJTZJNKTUJWGRW7MLGMJMGJKMAINWMPW7BV74OO6A, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.020222022 (301h14m27.482314495s) [noise=true , address=45.91.116.11:4118 , piece_id=VUZB3QRFST2T3HJHMEYYZ6CV7CKJTFWQGNOBDTXHGET2MBUIC4HQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:43.755608897 (301h14m29.21770138s) [upload=265104]: Next returned nil segment Oct 6 02:45:43.755631300 (301h14m29.217723773s) [upload=265104]: Waiting for error group managing segments... Oct 6 02:45:44.043930124 (301h14m29.506022597s) [noise=true , address=145.79.227.19:4077 , piece_id=EJIAPZSBMAO77E33V6MEBOQHUXVVYKMAVP53QAZXZSYBYDFP7MMQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.045501296 (301h14m29.507593769s) [noise=true , address=145.79.227.4:4185 , piece_id=3AD745KFD67M4XFNDXMJVEU34RRBQXXE4P3HVJJRDBHK3IIXNYTA, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.057881199 (301h14m29.519973672s) [noise=true , address=145.79.227.15:4875 , piece_id=PBDTNZ2NWRZZC3VCIQCVR2WLT2AP6662I2SSDNP6BLUDKCJMRK6Q, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.068726540 (301h14m29.530819023s) [noise=true , address=145.79.227.7:4294 , piece_id=7E3B4BGLCTW6C5DM6PPDJPWGVJS5HEGYPAPXRAMX7XDSG7USPQFQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.897895712 (301h14m30.359988185s) [noise=true , address=45.91.116.12:4226 , piece_id=NM7SEIKZYUHZLNVSOGYVZWMI7U7X5SOXOJDAEGY5NF3M6QQFPZSQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.252859665 (301h14m30.714952148s) [noise=true , address=77.90.31.14:4662 , piece_id=MPVBPBB63JILLOUF3N7SRLCZ2F74LX7Z4AH7ZFG3BDNBHT4UIXTQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.318952216 (301h14m30.781044699s) [noise=true , address=45.91.116.6:4974 , piece_id=7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.836439779 (301h14m31.298532262s) [noise=true , address=45.91.116.10:4081 , piece_id=FTXZ2LOHQMY74KOX6XZC3TLHWXOGDTIY3SZLSPTE7THGKNLMPXRQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.975895173 (301h14m31.437987656s) [noise=true , address=45.91.116.6:4975 , piece_id=T3VWH5OBQ7FE66GCKHZLEDSIXFVYEU3V52HI4CPD27QL3RH75K7Q, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:46.826703981 (301h14m32.288796454s) [noise=true , address=77.90.31.10:4518 , piece_id=6FGNAMVK6OAGJTZJNKTUJWGRW7MLGMJMGJKMAINWMPW7BV74OO6A, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:48.069759484 (301h14m33.531851968s) [noise=true , address=45.91.116.13:4154 , piece_id=2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:54.791582362 (301h14m40.253674835s) [noise=true , address=45.91.116.11:4118 , piece_id=VUZB3QRFST2T3HJHMEYYZ6CV7CKJTFWQGNOBDTXHGET2MBUIC4HQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:56.329163998 (301h14m41.791256481s) [noise=true , address=45.91.116.11:4117 , piece_id=QKEYHSK5JOL65H5T2KUEAEK6VKTNCZAYXIEZTMB76PV7P5PEVILQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:56.329201379 (301h14m41.791293862s) [seg_pos={0 0}, upload=265104]: Segment reached optimal threshold of 13 pieces. Oct 6 02:45:56.329362146 (301h14m41.791454629s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Oct 6 02:45:56.329371974 (301h14m41.791464457s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329546396 (301h14m41.791638869s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: write tcp 188.166.225.170:57604->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 02:45:56.329556665 (301h14m41.791649148s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329567937 (301h14m41.791660421s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: rpc: dial tcp 77.90.31.19:4841: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 02:45:56.329579870 (301h14m41.791672353s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329599808 (301h14m41.791692291s) [seg_pos={0 0}, upload=265104]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 6 02:45:56.329683116 (301h14m41.7917756s) [upload=265104]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 6 02:45:56.329690771 (301h14m41.791783254s) [upload=265104]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 6 02:45:56.636341641 (301h14m42.098434134s) [upload=265104]: Done uploading segments... Oct 6 02:45:56.636368131 (301h14m42.098460614s) [upload=265104]: Upload finished. err: <nil> Duration: 6.155145982s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........X.....................X.............................] duration: 6.021066252s FAIL (2025-10-06 03:10:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: 2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: 7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA) context deadline exceeded ======LOGS====== Oct 6 02:45:41.756337907 (301h14m27.21843039s) [upload=265104]: Starting upload Oct 6 02:45:41.756430433 (301h14m27.218522916s) [upload=265104]: Done starting upload Oct 6 02:45:41.756513701 (301h14m27.218606185s) [upload=265104]: Uploading segments... Oct 6 02:45:41.756752276 (301h14m27.218844759s) [upload=265104]: Got next segment. Inline: false Oct 6 02:45:41.756794536 (301h14m27.218887019s) [upload=265104]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 02:45:41.756800898 (301h14m27.218893381s) [upload=265104]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 02:45:42.009327267 (301h14m27.47141975s) [seg_pos={0 0}, upload=265104]: Begin upload segment... Oct 6 02:45:42.009417419 (301h14m27.471509892s) [seg_pos={0 0}, upload=265104]: Done begin upload segment. Oct 6 02:45:42.009444500 (301h14m27.471536983s) [noise=true , address=77.90.31.14:4662 , piece_id=MPVBPBB63JILLOUF3N7SRLCZ2F74LX7Z4AH7ZFG3BDNBHT4UIXTQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.009561473 (301h14m27.471653956s) [noise=true , address=145.79.227.19:4077 , piece_id=EJIAPZSBMAO77E33V6MEBOQHUXVVYKMAVP53QAZXZSYBYDFP7MMQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.010197574 (301h14m27.472290057s) [noise=true , address=145.79.227.15:4875 , piece_id=PBDTNZ2NWRZZC3VCIQCVR2WLT2AP6662I2SSDNP6BLUDKCJMRK6Q, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.010634216 (301h14m27.472726699s) [noise=true , address=145.79.227.4:4185 , piece_id=3AD745KFD67M4XFNDXMJVEU34RRBQXXE4P3HVJJRDBHK3IIXNYTA, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011073742 (301h14m27.473166225s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011505996 (301h14m27.473598479s) [noise=true , address=45.91.116.10:4081 , piece_id=FTXZ2LOHQMY74KOX6XZC3TLHWXOGDTIY3SZLSPTE7THGKNLMPXRQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011982502 (301h14m27.474074985s) [noise=true , address=45.91.116.6:4974 , piece_id=7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.012488236 (301h14m27.474580719s) [noise=true , address=145.79.227.7:4294 , piece_id=7E3B4BGLCTW6C5DM6PPDJPWGVJS5HEGYPAPXRAMX7XDSG7USPQFQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.013242472 (301h14m27.475334955s) [noise=true , address=45.91.116.13:4154 , piece_id=2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.014529382 (301h14m27.476621855s) [noise=true , address=45.91.116.6:4975 , piece_id=T3VWH5OBQ7FE66GCKHZLEDSIXFVYEU3V52HI4CPD27QL3RH75K7Q, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.014632619 (301h14m27.476725102s) [noise=true , address=45.91.116.11:4117 , piece_id=QKEYHSK5JOL65H5T2KUEAEK6VKTNCZAYXIEZTMB76PV7P5PEVILQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.015990214 (301h14m27.478082697s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.016291928 (301h14m27.478384412s) [noise=true , address=45.91.116.12:4226 , piece_id=NM7SEIKZYUHZLNVSOGYVZWMI7U7X5SOXOJDAEGY5NF3M6QQFPZSQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.017455915 (301h14m27.479548399s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.018864467 (301h14m27.48095694s) [noise=true , address=77.90.31.10:4518 , piece_id=6FGNAMVK6OAGJTZJNKTUJWGRW7MLGMJMGJKMAINWMPW7BV74OO6A, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.020222022 (301h14m27.482314495s) [noise=true , address=45.91.116.11:4118 , piece_id=VUZB3QRFST2T3HJHMEYYZ6CV7CKJTFWQGNOBDTXHGET2MBUIC4HQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:43.755608897 (301h14m29.21770138s) [upload=265104]: Next returned nil segment Oct 6 02:45:43.755631300 (301h14m29.217723773s) [upload=265104]: Waiting for error group managing segments... Oct 6 02:45:44.043930124 (301h14m29.506022597s) [noise=true , address=145.79.227.19:4077 , piece_id=EJIAPZSBMAO77E33V6MEBOQHUXVVYKMAVP53QAZXZSYBYDFP7MMQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.045501296 (301h14m29.507593769s) [noise=true , address=145.79.227.4:4185 , piece_id=3AD745KFD67M4XFNDXMJVEU34RRBQXXE4P3HVJJRDBHK3IIXNYTA, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.057881199 (301h14m29.519973672s) [noise=true , address=145.79.227.15:4875 , piece_id=PBDTNZ2NWRZZC3VCIQCVR2WLT2AP6662I2SSDNP6BLUDKCJMRK6Q, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.068726540 (301h14m29.530819023s) [noise=true , address=145.79.227.7:4294 , piece_id=7E3B4BGLCTW6C5DM6PPDJPWGVJS5HEGYPAPXRAMX7XDSG7USPQFQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.897895712 (301h14m30.359988185s) [noise=true , address=45.91.116.12:4226 , piece_id=NM7SEIKZYUHZLNVSOGYVZWMI7U7X5SOXOJDAEGY5NF3M6QQFPZSQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.252859665 (301h14m30.714952148s) [noise=true , address=77.90.31.14:4662 , piece_id=MPVBPBB63JILLOUF3N7SRLCZ2F74LX7Z4AH7ZFG3BDNBHT4UIXTQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.318952216 (301h14m30.781044699s) [noise=true , address=45.91.116.6:4974 , piece_id=7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.836439779 (301h14m31.298532262s) [noise=true , address=45.91.116.10:4081 , piece_id=FTXZ2LOHQMY74KOX6XZC3TLHWXOGDTIY3SZLSPTE7THGKNLMPXRQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.975895173 (301h14m31.437987656s) [noise=true , address=45.91.116.6:4975 , piece_id=T3VWH5OBQ7FE66GCKHZLEDSIXFVYEU3V52HI4CPD27QL3RH75K7Q, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:46.826703981 (301h14m32.288796454s) [noise=true , address=77.90.31.10:4518 , piece_id=6FGNAMVK6OAGJTZJNKTUJWGRW7MLGMJMGJKMAINWMPW7BV74OO6A, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:48.069759484 (301h14m33.531851968s) [noise=true , address=45.91.116.13:4154 , piece_id=2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:54.791582362 (301h14m40.253674835s) [noise=true , address=45.91.116.11:4118 , piece_id=VUZB3QRFST2T3HJHMEYYZ6CV7CKJTFWQGNOBDTXHGET2MBUIC4HQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:56.329163998 (301h14m41.791256481s) [noise=true , address=45.91.116.11:4117 , piece_id=QKEYHSK5JOL65H5T2KUEAEK6VKTNCZAYXIEZTMB76PV7P5PEVILQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:56.329201379 (301h14m41.791293862s) [seg_pos={0 0}, upload=265104]: Segment reached optimal threshold of 13 pieces. Oct 6 02:45:56.329362146 (301h14m41.791454629s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Oct 6 02:45:56.329371974 (301h14m41.791464457s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329546396 (301h14m41.791638869s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: write tcp 188.166.225.170:57604->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 02:45:56.329556665 (301h14m41.791649148s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329567937 (301h14m41.791660421s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: rpc: dial tcp 77.90.31.19:4841: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 02:45:56.329579870 (301h14m41.791672353s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329599808 (301h14m41.791692291s) [seg_pos={0 0}, upload=265104]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 6 02:45:56.329683116 (301h14m41.7917756s) [upload=265104]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 6 02:45:56.329690771 (301h14m41.791783254s) [upload=265104]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 6 02:45:56.636341641 (301h14m42.098434134s) [upload=265104]: Done uploading segments... Oct 6 02:45:56.636368131 (301h14m42.098460614s) [upload=265104]: Upload finished. err: <nil> Duration: 6.021098603s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........X.....................X............................] duration: 6.070368679s FAIL (2025-10-06 03:08:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: 2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: 7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA) context deadline exceeded ======LOGS====== Oct 6 02:45:41.756337907 (301h14m27.21843039s) [upload=265104]: Starting upload Oct 6 02:45:41.756430433 (301h14m27.218522916s) [upload=265104]: Done starting upload Oct 6 02:45:41.756513701 (301h14m27.218606185s) [upload=265104]: Uploading segments... Oct 6 02:45:41.756752276 (301h14m27.218844759s) [upload=265104]: Got next segment. Inline: false Oct 6 02:45:41.756794536 (301h14m27.218887019s) [upload=265104]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 02:45:41.756800898 (301h14m27.218893381s) [upload=265104]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 02:45:42.009327267 (301h14m27.47141975s) [seg_pos={0 0}, upload=265104]: Begin upload segment... Oct 6 02:45:42.009417419 (301h14m27.471509892s) [seg_pos={0 0}, upload=265104]: Done begin upload segment. Oct 6 02:45:42.009444500 (301h14m27.471536983s) [noise=true , address=77.90.31.14:4662 , piece_id=MPVBPBB63JILLOUF3N7SRLCZ2F74LX7Z4AH7ZFG3BDNBHT4UIXTQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.009561473 (301h14m27.471653956s) [noise=true , address=145.79.227.19:4077 , piece_id=EJIAPZSBMAO77E33V6MEBOQHUXVVYKMAVP53QAZXZSYBYDFP7MMQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.010197574 (301h14m27.472290057s) [noise=true , address=145.79.227.15:4875 , piece_id=PBDTNZ2NWRZZC3VCIQCVR2WLT2AP6662I2SSDNP6BLUDKCJMRK6Q, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.010634216 (301h14m27.472726699s) [noise=true , address=145.79.227.4:4185 , piece_id=3AD745KFD67M4XFNDXMJVEU34RRBQXXE4P3HVJJRDBHK3IIXNYTA, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011073742 (301h14m27.473166225s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011505996 (301h14m27.473598479s) [noise=true , address=45.91.116.10:4081 , piece_id=FTXZ2LOHQMY74KOX6XZC3TLHWXOGDTIY3SZLSPTE7THGKNLMPXRQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011982502 (301h14m27.474074985s) [noise=true , address=45.91.116.6:4974 , piece_id=7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.012488236 (301h14m27.474580719s) [noise=true , address=145.79.227.7:4294 , piece_id=7E3B4BGLCTW6C5DM6PPDJPWGVJS5HEGYPAPXRAMX7XDSG7USPQFQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.013242472 (301h14m27.475334955s) [noise=true , address=45.91.116.13:4154 , piece_id=2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.014529382 (301h14m27.476621855s) [noise=true , address=45.91.116.6:4975 , piece_id=T3VWH5OBQ7FE66GCKHZLEDSIXFVYEU3V52HI4CPD27QL3RH75K7Q, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.014632619 (301h14m27.476725102s) [noise=true , address=45.91.116.11:4117 , piece_id=QKEYHSK5JOL65H5T2KUEAEK6VKTNCZAYXIEZTMB76PV7P5PEVILQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.015990214 (301h14m27.478082697s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.016291928 (301h14m27.478384412s) [noise=true , address=45.91.116.12:4226 , piece_id=NM7SEIKZYUHZLNVSOGYVZWMI7U7X5SOXOJDAEGY5NF3M6QQFPZSQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.017455915 (301h14m27.479548399s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.018864467 (301h14m27.48095694s) [noise=true , address=77.90.31.10:4518 , piece_id=6FGNAMVK6OAGJTZJNKTUJWGRW7MLGMJMGJKMAINWMPW7BV74OO6A, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.020222022 (301h14m27.482314495s) [noise=true , address=45.91.116.11:4118 , piece_id=VUZB3QRFST2T3HJHMEYYZ6CV7CKJTFWQGNOBDTXHGET2MBUIC4HQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:43.755608897 (301h14m29.21770138s) [upload=265104]: Next returned nil segment Oct 6 02:45:43.755631300 (301h14m29.217723773s) [upload=265104]: Waiting for error group managing segments... Oct 6 02:45:44.043930124 (301h14m29.506022597s) [noise=true , address=145.79.227.19:4077 , piece_id=EJIAPZSBMAO77E33V6MEBOQHUXVVYKMAVP53QAZXZSYBYDFP7MMQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.045501296 (301h14m29.507593769s) [noise=true , address=145.79.227.4:4185 , piece_id=3AD745KFD67M4XFNDXMJVEU34RRBQXXE4P3HVJJRDBHK3IIXNYTA, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.057881199 (301h14m29.519973672s) [noise=true , address=145.79.227.15:4875 , piece_id=PBDTNZ2NWRZZC3VCIQCVR2WLT2AP6662I2SSDNP6BLUDKCJMRK6Q, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.068726540 (301h14m29.530819023s) [noise=true , address=145.79.227.7:4294 , piece_id=7E3B4BGLCTW6C5DM6PPDJPWGVJS5HEGYPAPXRAMX7XDSG7USPQFQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.897895712 (301h14m30.359988185s) [noise=true , address=45.91.116.12:4226 , piece_id=NM7SEIKZYUHZLNVSOGYVZWMI7U7X5SOXOJDAEGY5NF3M6QQFPZSQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.252859665 (301h14m30.714952148s) [noise=true , address=77.90.31.14:4662 , piece_id=MPVBPBB63JILLOUF3N7SRLCZ2F74LX7Z4AH7ZFG3BDNBHT4UIXTQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.318952216 (301h14m30.781044699s) [noise=true , address=45.91.116.6:4974 , piece_id=7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.836439779 (301h14m31.298532262s) [noise=true , address=45.91.116.10:4081 , piece_id=FTXZ2LOHQMY74KOX6XZC3TLHWXOGDTIY3SZLSPTE7THGKNLMPXRQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.975895173 (301h14m31.437987656s) [noise=true , address=45.91.116.6:4975 , piece_id=T3VWH5OBQ7FE66GCKHZLEDSIXFVYEU3V52HI4CPD27QL3RH75K7Q, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:46.826703981 (301h14m32.288796454s) [noise=true , address=77.90.31.10:4518 , piece_id=6FGNAMVK6OAGJTZJNKTUJWGRW7MLGMJMGJKMAINWMPW7BV74OO6A, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:48.069759484 (301h14m33.531851968s) [noise=true , address=45.91.116.13:4154 , piece_id=2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:54.791582362 (301h14m40.253674835s) [noise=true , address=45.91.116.11:4118 , piece_id=VUZB3QRFST2T3HJHMEYYZ6CV7CKJTFWQGNOBDTXHGET2MBUIC4HQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:56.329163998 (301h14m41.791256481s) [noise=true , address=45.91.116.11:4117 , piece_id=QKEYHSK5JOL65H5T2KUEAEK6VKTNCZAYXIEZTMB76PV7P5PEVILQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:56.329201379 (301h14m41.791293862s) [seg_pos={0 0}, upload=265104]: Segment reached optimal threshold of 13 pieces. Oct 6 02:45:56.329362146 (301h14m41.791454629s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Oct 6 02:45:56.329371974 (301h14m41.791464457s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329546396 (301h14m41.791638869s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: write tcp 188.166.225.170:57604->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 02:45:56.329556665 (301h14m41.791649148s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329567937 (301h14m41.791660421s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: rpc: dial tcp 77.90.31.19:4841: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 02:45:56.329579870 (301h14m41.791672353s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329599808 (301h14m41.791692291s) [seg_pos={0 0}, upload=265104]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 6 02:45:56.329683116 (301h14m41.7917756s) [upload=265104]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 6 02:45:56.329690771 (301h14m41.791783254s) [upload=265104]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 6 02:45:56.636341641 (301h14m42.098434134s) [upload=265104]: Done uploading segments... Oct 6 02:45:56.636368131 (301h14m42.098460614s) [upload=265104]: Upload finished. err: <nil> Duration: 6.070395769s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........X.....................X...........................] duration: 6.079347718s FAIL (2025-10-06 03:07:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: 2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: 7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA) context deadline exceeded ======LOGS====== Oct 6 02:45:41.756337907 (301h14m27.21843039s) [upload=265104]: Starting upload Oct 6 02:45:41.756430433 (301h14m27.218522916s) [upload=265104]: Done starting upload Oct 6 02:45:41.756513701 (301h14m27.218606185s) [upload=265104]: Uploading segments... Oct 6 02:45:41.756752276 (301h14m27.218844759s) [upload=265104]: Got next segment. Inline: false Oct 6 02:45:41.756794536 (301h14m27.218887019s) [upload=265104]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 02:45:41.756800898 (301h14m27.218893381s) [upload=265104]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 02:45:42.009327267 (301h14m27.47141975s) [seg_pos={0 0}, upload=265104]: Begin upload segment... Oct 6 02:45:42.009417419 (301h14m27.471509892s) [seg_pos={0 0}, upload=265104]: Done begin upload segment. Oct 6 02:45:42.009444500 (301h14m27.471536983s) [noise=true , address=77.90.31.14:4662 , piece_id=MPVBPBB63JILLOUF3N7SRLCZ2F74LX7Z4AH7ZFG3BDNBHT4UIXTQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.009561473 (301h14m27.471653956s) [noise=true , address=145.79.227.19:4077 , piece_id=EJIAPZSBMAO77E33V6MEBOQHUXVVYKMAVP53QAZXZSYBYDFP7MMQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.010197574 (301h14m27.472290057s) [noise=true , address=145.79.227.15:4875 , piece_id=PBDTNZ2NWRZZC3VCIQCVR2WLT2AP6662I2SSDNP6BLUDKCJMRK6Q, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.010634216 (301h14m27.472726699s) [noise=true , address=145.79.227.4:4185 , piece_id=3AD745KFD67M4XFNDXMJVEU34RRBQXXE4P3HVJJRDBHK3IIXNYTA, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011073742 (301h14m27.473166225s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011505996 (301h14m27.473598479s) [noise=true , address=45.91.116.10:4081 , piece_id=FTXZ2LOHQMY74KOX6XZC3TLHWXOGDTIY3SZLSPTE7THGKNLMPXRQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011982502 (301h14m27.474074985s) [noise=true , address=45.91.116.6:4974 , piece_id=7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.012488236 (301h14m27.474580719s) [noise=true , address=145.79.227.7:4294 , piece_id=7E3B4BGLCTW6C5DM6PPDJPWGVJS5HEGYPAPXRAMX7XDSG7USPQFQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.013242472 (301h14m27.475334955s) [noise=true , address=45.91.116.13:4154 , piece_id=2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.014529382 (301h14m27.476621855s) [noise=true , address=45.91.116.6:4975 , piece_id=T3VWH5OBQ7FE66GCKHZLEDSIXFVYEU3V52HI4CPD27QL3RH75K7Q, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.014632619 (301h14m27.476725102s) [noise=true , address=45.91.116.11:4117 , piece_id=QKEYHSK5JOL65H5T2KUEAEK6VKTNCZAYXIEZTMB76PV7P5PEVILQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.015990214 (301h14m27.478082697s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.016291928 (301h14m27.478384412s) [noise=true , address=45.91.116.12:4226 , piece_id=NM7SEIKZYUHZLNVSOGYVZWMI7U7X5SOXOJDAEGY5NF3M6QQFPZSQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.017455915 (301h14m27.479548399s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.018864467 (301h14m27.48095694s) [noise=true , address=77.90.31.10:4518 , piece_id=6FGNAMVK6OAGJTZJNKTUJWGRW7MLGMJMGJKMAINWMPW7BV74OO6A, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.020222022 (301h14m27.482314495s) [noise=true , address=45.91.116.11:4118 , piece_id=VUZB3QRFST2T3HJHMEYYZ6CV7CKJTFWQGNOBDTXHGET2MBUIC4HQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:43.755608897 (301h14m29.21770138s) [upload=265104]: Next returned nil segment Oct 6 02:45:43.755631300 (301h14m29.217723773s) [upload=265104]: Waiting for error group managing segments... Oct 6 02:45:44.043930124 (301h14m29.506022597s) [noise=true , address=145.79.227.19:4077 , piece_id=EJIAPZSBMAO77E33V6MEBOQHUXVVYKMAVP53QAZXZSYBYDFP7MMQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.045501296 (301h14m29.507593769s) [noise=true , address=145.79.227.4:4185 , piece_id=3AD745KFD67M4XFNDXMJVEU34RRBQXXE4P3HVJJRDBHK3IIXNYTA, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.057881199 (301h14m29.519973672s) [noise=true , address=145.79.227.15:4875 , piece_id=PBDTNZ2NWRZZC3VCIQCVR2WLT2AP6662I2SSDNP6BLUDKCJMRK6Q, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.068726540 (301h14m29.530819023s) [noise=true , address=145.79.227.7:4294 , piece_id=7E3B4BGLCTW6C5DM6PPDJPWGVJS5HEGYPAPXRAMX7XDSG7USPQFQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.897895712 (301h14m30.359988185s) [noise=true , address=45.91.116.12:4226 , piece_id=NM7SEIKZYUHZLNVSOGYVZWMI7U7X5SOXOJDAEGY5NF3M6QQFPZSQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.252859665 (301h14m30.714952148s) [noise=true , address=77.90.31.14:4662 , piece_id=MPVBPBB63JILLOUF3N7SRLCZ2F74LX7Z4AH7ZFG3BDNBHT4UIXTQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.318952216 (301h14m30.781044699s) [noise=true , address=45.91.116.6:4974 , piece_id=7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.836439779 (301h14m31.298532262s) [noise=true , address=45.91.116.10:4081 , piece_id=FTXZ2LOHQMY74KOX6XZC3TLHWXOGDTIY3SZLSPTE7THGKNLMPXRQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.975895173 (301h14m31.437987656s) [noise=true , address=45.91.116.6:4975 , piece_id=T3VWH5OBQ7FE66GCKHZLEDSIXFVYEU3V52HI4CPD27QL3RH75K7Q, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:46.826703981 (301h14m32.288796454s) [noise=true , address=77.90.31.10:4518 , piece_id=6FGNAMVK6OAGJTZJNKTUJWGRW7MLGMJMGJKMAINWMPW7BV74OO6A, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:48.069759484 (301h14m33.531851968s) [noise=true , address=45.91.116.13:4154 , piece_id=2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:54.791582362 (301h14m40.253674835s) [noise=true , address=45.91.116.11:4118 , piece_id=VUZB3QRFST2T3HJHMEYYZ6CV7CKJTFWQGNOBDTXHGET2MBUIC4HQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:56.329163998 (301h14m41.791256481s) [noise=true , address=45.91.116.11:4117 , piece_id=QKEYHSK5JOL65H5T2KUEAEK6VKTNCZAYXIEZTMB76PV7P5PEVILQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:56.329201379 (301h14m41.791293862s) [seg_pos={0 0}, upload=265104]: Segment reached optimal threshold of 13 pieces. Oct 6 02:45:56.329362146 (301h14m41.791454629s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Oct 6 02:45:56.329371974 (301h14m41.791464457s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329546396 (301h14m41.791638869s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: write tcp 188.166.225.170:57604->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 02:45:56.329556665 (301h14m41.791649148s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329567937 (301h14m41.791660421s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: rpc: dial tcp 77.90.31.19:4841: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 02:45:56.329579870 (301h14m41.791672353s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329599808 (301h14m41.791692291s) [seg_pos={0 0}, upload=265104]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 6 02:45:56.329683116 (301h14m41.7917756s) [upload=265104]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 6 02:45:56.329690771 (301h14m41.791783254s) [upload=265104]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 6 02:45:56.636341641 (301h14m42.098434134s) [upload=265104]: Done uploading segments... Oct 6 02:45:56.636368131 (301h14m42.098460614s) [upload=265104]: Upload finished. err: <nil> Duration: 6.079372365s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........X.....................X..........................] duration: 6.315735182s FAIL (2025-10-06 03:06:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: 2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: 7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA) context deadline exceeded ======LOGS====== Oct 6 02:45:41.756337907 (301h14m27.21843039s) [upload=265104]: Starting upload Oct 6 02:45:41.756430433 (301h14m27.218522916s) [upload=265104]: Done starting upload Oct 6 02:45:41.756513701 (301h14m27.218606185s) [upload=265104]: Uploading segments... Oct 6 02:45:41.756752276 (301h14m27.218844759s) [upload=265104]: Got next segment. Inline: false Oct 6 02:45:41.756794536 (301h14m27.218887019s) [upload=265104]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 02:45:41.756800898 (301h14m27.218893381s) [upload=265104]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 02:45:42.009327267 (301h14m27.47141975s) [seg_pos={0 0}, upload=265104]: Begin upload segment... Oct 6 02:45:42.009417419 (301h14m27.471509892s) [seg_pos={0 0}, upload=265104]: Done begin upload segment. Oct 6 02:45:42.009444500 (301h14m27.471536983s) [noise=true , address=77.90.31.14:4662 , piece_id=MPVBPBB63JILLOUF3N7SRLCZ2F74LX7Z4AH7ZFG3BDNBHT4UIXTQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.009561473 (301h14m27.471653956s) [noise=true , address=145.79.227.19:4077 , piece_id=EJIAPZSBMAO77E33V6MEBOQHUXVVYKMAVP53QAZXZSYBYDFP7MMQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.010197574 (301h14m27.472290057s) [noise=true , address=145.79.227.15:4875 , piece_id=PBDTNZ2NWRZZC3VCIQCVR2WLT2AP6662I2SSDNP6BLUDKCJMRK6Q, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.010634216 (301h14m27.472726699s) [noise=true , address=145.79.227.4:4185 , piece_id=3AD745KFD67M4XFNDXMJVEU34RRBQXXE4P3HVJJRDBHK3IIXNYTA, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011073742 (301h14m27.473166225s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011505996 (301h14m27.473598479s) [noise=true , address=45.91.116.10:4081 , piece_id=FTXZ2LOHQMY74KOX6XZC3TLHWXOGDTIY3SZLSPTE7THGKNLMPXRQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011982502 (301h14m27.474074985s) [noise=true , address=45.91.116.6:4974 , piece_id=7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.012488236 (301h14m27.474580719s) [noise=true , address=145.79.227.7:4294 , piece_id=7E3B4BGLCTW6C5DM6PPDJPWGVJS5HEGYPAPXRAMX7XDSG7USPQFQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.013242472 (301h14m27.475334955s) [noise=true , address=45.91.116.13:4154 , piece_id=2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.014529382 (301h14m27.476621855s) [noise=true , address=45.91.116.6:4975 , piece_id=T3VWH5OBQ7FE66GCKHZLEDSIXFVYEU3V52HI4CPD27QL3RH75K7Q, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.014632619 (301h14m27.476725102s) [noise=true , address=45.91.116.11:4117 , piece_id=QKEYHSK5JOL65H5T2KUEAEK6VKTNCZAYXIEZTMB76PV7P5PEVILQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.015990214 (301h14m27.478082697s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.016291928 (301h14m27.478384412s) [noise=true , address=45.91.116.12:4226 , piece_id=NM7SEIKZYUHZLNVSOGYVZWMI7U7X5SOXOJDAEGY5NF3M6QQFPZSQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.017455915 (301h14m27.479548399s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.018864467 (301h14m27.48095694s) [noise=true , address=77.90.31.10:4518 , piece_id=6FGNAMVK6OAGJTZJNKTUJWGRW7MLGMJMGJKMAINWMPW7BV74OO6A, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.020222022 (301h14m27.482314495s) [noise=true , address=45.91.116.11:4118 , piece_id=VUZB3QRFST2T3HJHMEYYZ6CV7CKJTFWQGNOBDTXHGET2MBUIC4HQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:43.755608897 (301h14m29.21770138s) [upload=265104]: Next returned nil segment Oct 6 02:45:43.755631300 (301h14m29.217723773s) [upload=265104]: Waiting for error group managing segments... Oct 6 02:45:44.043930124 (301h14m29.506022597s) [noise=true , address=145.79.227.19:4077 , piece_id=EJIAPZSBMAO77E33V6MEBOQHUXVVYKMAVP53QAZXZSYBYDFP7MMQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.045501296 (301h14m29.507593769s) [noise=true , address=145.79.227.4:4185 , piece_id=3AD745KFD67M4XFNDXMJVEU34RRBQXXE4P3HVJJRDBHK3IIXNYTA, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.057881199 (301h14m29.519973672s) [noise=true , address=145.79.227.15:4875 , piece_id=PBDTNZ2NWRZZC3VCIQCVR2WLT2AP6662I2SSDNP6BLUDKCJMRK6Q, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.068726540 (301h14m29.530819023s) [noise=true , address=145.79.227.7:4294 , piece_id=7E3B4BGLCTW6C5DM6PPDJPWGVJS5HEGYPAPXRAMX7XDSG7USPQFQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.897895712 (301h14m30.359988185s) [noise=true , address=45.91.116.12:4226 , piece_id=NM7SEIKZYUHZLNVSOGYVZWMI7U7X5SOXOJDAEGY5NF3M6QQFPZSQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.252859665 (301h14m30.714952148s) [noise=true , address=77.90.31.14:4662 , piece_id=MPVBPBB63JILLOUF3N7SRLCZ2F74LX7Z4AH7ZFG3BDNBHT4UIXTQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.318952216 (301h14m30.781044699s) [noise=true , address=45.91.116.6:4974 , piece_id=7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.836439779 (301h14m31.298532262s) [noise=true , address=45.91.116.10:4081 , piece_id=FTXZ2LOHQMY74KOX6XZC3TLHWXOGDTIY3SZLSPTE7THGKNLMPXRQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.975895173 (301h14m31.437987656s) [noise=true , address=45.91.116.6:4975 , piece_id=T3VWH5OBQ7FE66GCKHZLEDSIXFVYEU3V52HI4CPD27QL3RH75K7Q, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:46.826703981 (301h14m32.288796454s) [noise=true , address=77.90.31.10:4518 , piece_id=6FGNAMVK6OAGJTZJNKTUJWGRW7MLGMJMGJKMAINWMPW7BV74OO6A, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:48.069759484 (301h14m33.531851968s) [noise=true , address=45.91.116.13:4154 , piece_id=2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:54.791582362 (301h14m40.253674835s) [noise=true , address=45.91.116.11:4118 , piece_id=VUZB3QRFST2T3HJHMEYYZ6CV7CKJTFWQGNOBDTXHGET2MBUIC4HQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:56.329163998 (301h14m41.791256481s) [noise=true , address=45.91.116.11:4117 , piece_id=QKEYHSK5JOL65H5T2KUEAEK6VKTNCZAYXIEZTMB76PV7P5PEVILQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:56.329201379 (301h14m41.791293862s) [seg_pos={0 0}, upload=265104]: Segment reached optimal threshold of 13 pieces. Oct 6 02:45:56.329362146 (301h14m41.791454629s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Oct 6 02:45:56.329371974 (301h14m41.791464457s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329546396 (301h14m41.791638869s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: write tcp 188.166.225.170:57604->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 02:45:56.329556665 (301h14m41.791649148s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329567937 (301h14m41.791660421s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: rpc: dial tcp 77.90.31.19:4841: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 02:45:56.329579870 (301h14m41.791672353s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329599808 (301h14m41.791692291s) [seg_pos={0 0}, upload=265104]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 6 02:45:56.329683116 (301h14m41.7917756s) [upload=265104]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 6 02:45:56.329690771 (301h14m41.791783254s) [upload=265104]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 6 02:45:56.636341641 (301h14m42.098434134s) [upload=265104]: Done uploading segments... Oct 6 02:45:56.636368131 (301h14m42.098460614s) [upload=265104]: Upload finished. err: <nil> Duration: 6.31576572s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............X.....................X.........................] duration: 5.908520696s FAIL (2025-10-06 03:06:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: 2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: 7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA) context deadline exceeded ======LOGS====== Oct 6 02:45:41.756337907 (301h14m27.21843039s) [upload=265104]: Starting upload Oct 6 02:45:41.756430433 (301h14m27.218522916s) [upload=265104]: Done starting upload Oct 6 02:45:41.756513701 (301h14m27.218606185s) [upload=265104]: Uploading segments... Oct 6 02:45:41.756752276 (301h14m27.218844759s) [upload=265104]: Got next segment. Inline: false Oct 6 02:45:41.756794536 (301h14m27.218887019s) [upload=265104]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 02:45:41.756800898 (301h14m27.218893381s) [upload=265104]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 02:45:42.009327267 (301h14m27.47141975s) [seg_pos={0 0}, upload=265104]: Begin upload segment... Oct 6 02:45:42.009417419 (301h14m27.471509892s) [seg_pos={0 0}, upload=265104]: Done begin upload segment. Oct 6 02:45:42.009444500 (301h14m27.471536983s) [noise=true , address=77.90.31.14:4662 , piece_id=MPVBPBB63JILLOUF3N7SRLCZ2F74LX7Z4AH7ZFG3BDNBHT4UIXTQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.009561473 (301h14m27.471653956s) [noise=true , address=145.79.227.19:4077 , piece_id=EJIAPZSBMAO77E33V6MEBOQHUXVVYKMAVP53QAZXZSYBYDFP7MMQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.010197574 (301h14m27.472290057s) [noise=true , address=145.79.227.15:4875 , piece_id=PBDTNZ2NWRZZC3VCIQCVR2WLT2AP6662I2SSDNP6BLUDKCJMRK6Q, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.010634216 (301h14m27.472726699s) [noise=true , address=145.79.227.4:4185 , piece_id=3AD745KFD67M4XFNDXMJVEU34RRBQXXE4P3HVJJRDBHK3IIXNYTA, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011073742 (301h14m27.473166225s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011505996 (301h14m27.473598479s) [noise=true , address=45.91.116.10:4081 , piece_id=FTXZ2LOHQMY74KOX6XZC3TLHWXOGDTIY3SZLSPTE7THGKNLMPXRQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011982502 (301h14m27.474074985s) [noise=true , address=45.91.116.6:4974 , piece_id=7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.012488236 (301h14m27.474580719s) [noise=true , address=145.79.227.7:4294 , piece_id=7E3B4BGLCTW6C5DM6PPDJPWGVJS5HEGYPAPXRAMX7XDSG7USPQFQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.013242472 (301h14m27.475334955s) [noise=true , address=45.91.116.13:4154 , piece_id=2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.014529382 (301h14m27.476621855s) [noise=true , address=45.91.116.6:4975 , piece_id=T3VWH5OBQ7FE66GCKHZLEDSIXFVYEU3V52HI4CPD27QL3RH75K7Q, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.014632619 (301h14m27.476725102s) [noise=true , address=45.91.116.11:4117 , piece_id=QKEYHSK5JOL65H5T2KUEAEK6VKTNCZAYXIEZTMB76PV7P5PEVILQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.015990214 (301h14m27.478082697s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.016291928 (301h14m27.478384412s) [noise=true , address=45.91.116.12:4226 , piece_id=NM7SEIKZYUHZLNVSOGYVZWMI7U7X5SOXOJDAEGY5NF3M6QQFPZSQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.017455915 (301h14m27.479548399s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.018864467 (301h14m27.48095694s) [noise=true , address=77.90.31.10:4518 , piece_id=6FGNAMVK6OAGJTZJNKTUJWGRW7MLGMJMGJKMAINWMPW7BV74OO6A, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.020222022 (301h14m27.482314495s) [noise=true , address=45.91.116.11:4118 , piece_id=VUZB3QRFST2T3HJHMEYYZ6CV7CKJTFWQGNOBDTXHGET2MBUIC4HQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:43.755608897 (301h14m29.21770138s) [upload=265104]: Next returned nil segment Oct 6 02:45:43.755631300 (301h14m29.217723773s) [upload=265104]: Waiting for error group managing segments... Oct 6 02:45:44.043930124 (301h14m29.506022597s) [noise=true , address=145.79.227.19:4077 , piece_id=EJIAPZSBMAO77E33V6MEBOQHUXVVYKMAVP53QAZXZSYBYDFP7MMQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.045501296 (301h14m29.507593769s) [noise=true , address=145.79.227.4:4185 , piece_id=3AD745KFD67M4XFNDXMJVEU34RRBQXXE4P3HVJJRDBHK3IIXNYTA, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.057881199 (301h14m29.519973672s) [noise=true , address=145.79.227.15:4875 , piece_id=PBDTNZ2NWRZZC3VCIQCVR2WLT2AP6662I2SSDNP6BLUDKCJMRK6Q, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.068726540 (301h14m29.530819023s) [noise=true , address=145.79.227.7:4294 , piece_id=7E3B4BGLCTW6C5DM6PPDJPWGVJS5HEGYPAPXRAMX7XDSG7USPQFQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.897895712 (301h14m30.359988185s) [noise=true , address=45.91.116.12:4226 , piece_id=NM7SEIKZYUHZLNVSOGYVZWMI7U7X5SOXOJDAEGY5NF3M6QQFPZSQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.252859665 (301h14m30.714952148s) [noise=true , address=77.90.31.14:4662 , piece_id=MPVBPBB63JILLOUF3N7SRLCZ2F74LX7Z4AH7ZFG3BDNBHT4UIXTQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.318952216 (301h14m30.781044699s) [noise=true , address=45.91.116.6:4974 , piece_id=7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.836439779 (301h14m31.298532262s) [noise=true , address=45.91.116.10:4081 , piece_id=FTXZ2LOHQMY74KOX6XZC3TLHWXOGDTIY3SZLSPTE7THGKNLMPXRQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.975895173 (301h14m31.437987656s) [noise=true , address=45.91.116.6:4975 , piece_id=T3VWH5OBQ7FE66GCKHZLEDSIXFVYEU3V52HI4CPD27QL3RH75K7Q, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:46.826703981 (301h14m32.288796454s) [noise=true , address=77.90.31.10:4518 , piece_id=6FGNAMVK6OAGJTZJNKTUJWGRW7MLGMJMGJKMAINWMPW7BV74OO6A, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:48.069759484 (301h14m33.531851968s) [noise=true , address=45.91.116.13:4154 , piece_id=2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:54.791582362 (301h14m40.253674835s) [noise=true , address=45.91.116.11:4118 , piece_id=VUZB3QRFST2T3HJHMEYYZ6CV7CKJTFWQGNOBDTXHGET2MBUIC4HQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:56.329163998 (301h14m41.791256481s) [noise=true , address=45.91.116.11:4117 , piece_id=QKEYHSK5JOL65H5T2KUEAEK6VKTNCZAYXIEZTMB76PV7P5PEVILQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:56.329201379 (301h14m41.791293862s) [seg_pos={0 0}, upload=265104]: Segment reached optimal threshold of 13 pieces. Oct 6 02:45:56.329362146 (301h14m41.791454629s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Oct 6 02:45:56.329371974 (301h14m41.791464457s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329546396 (301h14m41.791638869s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: write tcp 188.166.225.170:57604->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 02:45:56.329556665 (301h14m41.791649148s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329567937 (301h14m41.791660421s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: rpc: dial tcp 77.90.31.19:4841: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 02:45:56.329579870 (301h14m41.791672353s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329599808 (301h14m41.791692291s) [seg_pos={0 0}, upload=265104]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 6 02:45:56.329683116 (301h14m41.7917756s) [upload=265104]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 6 02:45:56.329690771 (301h14m41.791783254s) [upload=265104]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 6 02:45:56.636341641 (301h14m42.098434134s) [upload=265104]: Done uploading segments... Oct 6 02:45:56.636368131 (301h14m42.098460614s) [upload=265104]: Upload finished. err: <nil> Duration: 5.908556143s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............X.....................X........................] duration: 5.971711198s FAIL (2025-10-06 03:05:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: 2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: 7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA) context deadline exceeded ======LOGS====== Oct 6 02:45:41.756337907 (301h14m27.21843039s) [upload=265104]: Starting upload Oct 6 02:45:41.756430433 (301h14m27.218522916s) [upload=265104]: Done starting upload Oct 6 02:45:41.756513701 (301h14m27.218606185s) [upload=265104]: Uploading segments... Oct 6 02:45:41.756752276 (301h14m27.218844759s) [upload=265104]: Got next segment. Inline: false Oct 6 02:45:41.756794536 (301h14m27.218887019s) [upload=265104]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 02:45:41.756800898 (301h14m27.218893381s) [upload=265104]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 02:45:42.009327267 (301h14m27.47141975s) [seg_pos={0 0}, upload=265104]: Begin upload segment... Oct 6 02:45:42.009417419 (301h14m27.471509892s) [seg_pos={0 0}, upload=265104]: Done begin upload segment. Oct 6 02:45:42.009444500 (301h14m27.471536983s) [noise=true , address=77.90.31.14:4662 , piece_id=MPVBPBB63JILLOUF3N7SRLCZ2F74LX7Z4AH7ZFG3BDNBHT4UIXTQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.009561473 (301h14m27.471653956s) [noise=true , address=145.79.227.19:4077 , piece_id=EJIAPZSBMAO77E33V6MEBOQHUXVVYKMAVP53QAZXZSYBYDFP7MMQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.010197574 (301h14m27.472290057s) [noise=true , address=145.79.227.15:4875 , piece_id=PBDTNZ2NWRZZC3VCIQCVR2WLT2AP6662I2SSDNP6BLUDKCJMRK6Q, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.010634216 (301h14m27.472726699s) [noise=true , address=145.79.227.4:4185 , piece_id=3AD745KFD67M4XFNDXMJVEU34RRBQXXE4P3HVJJRDBHK3IIXNYTA, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011073742 (301h14m27.473166225s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011505996 (301h14m27.473598479s) [noise=true , address=45.91.116.10:4081 , piece_id=FTXZ2LOHQMY74KOX6XZC3TLHWXOGDTIY3SZLSPTE7THGKNLMPXRQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011982502 (301h14m27.474074985s) [noise=true , address=45.91.116.6:4974 , piece_id=7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.012488236 (301h14m27.474580719s) [noise=true , address=145.79.227.7:4294 , piece_id=7E3B4BGLCTW6C5DM6PPDJPWGVJS5HEGYPAPXRAMX7XDSG7USPQFQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.013242472 (301h14m27.475334955s) [noise=true , address=45.91.116.13:4154 , piece_id=2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.014529382 (301h14m27.476621855s) [noise=true , address=45.91.116.6:4975 , piece_id=T3VWH5OBQ7FE66GCKHZLEDSIXFVYEU3V52HI4CPD27QL3RH75K7Q, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.014632619 (301h14m27.476725102s) [noise=true , address=45.91.116.11:4117 , piece_id=QKEYHSK5JOL65H5T2KUEAEK6VKTNCZAYXIEZTMB76PV7P5PEVILQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.015990214 (301h14m27.478082697s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.016291928 (301h14m27.478384412s) [noise=true , address=45.91.116.12:4226 , piece_id=NM7SEIKZYUHZLNVSOGYVZWMI7U7X5SOXOJDAEGY5NF3M6QQFPZSQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.017455915 (301h14m27.479548399s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.018864467 (301h14m27.48095694s) [noise=true , address=77.90.31.10:4518 , piece_id=6FGNAMVK6OAGJTZJNKTUJWGRW7MLGMJMGJKMAINWMPW7BV74OO6A, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.020222022 (301h14m27.482314495s) [noise=true , address=45.91.116.11:4118 , piece_id=VUZB3QRFST2T3HJHMEYYZ6CV7CKJTFWQGNOBDTXHGET2MBUIC4HQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:43.755608897 (301h14m29.21770138s) [upload=265104]: Next returned nil segment Oct 6 02:45:43.755631300 (301h14m29.217723773s) [upload=265104]: Waiting for error group managing segments... Oct 6 02:45:44.043930124 (301h14m29.506022597s) [noise=true , address=145.79.227.19:4077 , piece_id=EJIAPZSBMAO77E33V6MEBOQHUXVVYKMAVP53QAZXZSYBYDFP7MMQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.045501296 (301h14m29.507593769s) [noise=true , address=145.79.227.4:4185 , piece_id=3AD745KFD67M4XFNDXMJVEU34RRBQXXE4P3HVJJRDBHK3IIXNYTA, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.057881199 (301h14m29.519973672s) [noise=true , address=145.79.227.15:4875 , piece_id=PBDTNZ2NWRZZC3VCIQCVR2WLT2AP6662I2SSDNP6BLUDKCJMRK6Q, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.068726540 (301h14m29.530819023s) [noise=true , address=145.79.227.7:4294 , piece_id=7E3B4BGLCTW6C5DM6PPDJPWGVJS5HEGYPAPXRAMX7XDSG7USPQFQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.897895712 (301h14m30.359988185s) [noise=true , address=45.91.116.12:4226 , piece_id=NM7SEIKZYUHZLNVSOGYVZWMI7U7X5SOXOJDAEGY5NF3M6QQFPZSQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.252859665 (301h14m30.714952148s) [noise=true , address=77.90.31.14:4662 , piece_id=MPVBPBB63JILLOUF3N7SRLCZ2F74LX7Z4AH7ZFG3BDNBHT4UIXTQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.318952216 (301h14m30.781044699s) [noise=true , address=45.91.116.6:4974 , piece_id=7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.836439779 (301h14m31.298532262s) [noise=true , address=45.91.116.10:4081 , piece_id=FTXZ2LOHQMY74KOX6XZC3TLHWXOGDTIY3SZLSPTE7THGKNLMPXRQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.975895173 (301h14m31.437987656s) [noise=true , address=45.91.116.6:4975 , piece_id=T3VWH5OBQ7FE66GCKHZLEDSIXFVYEU3V52HI4CPD27QL3RH75K7Q, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:46.826703981 (301h14m32.288796454s) [noise=true , address=77.90.31.10:4518 , piece_id=6FGNAMVK6OAGJTZJNKTUJWGRW7MLGMJMGJKMAINWMPW7BV74OO6A, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:48.069759484 (301h14m33.531851968s) [noise=true , address=45.91.116.13:4154 , piece_id=2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:54.791582362 (301h14m40.253674835s) [noise=true , address=45.91.116.11:4118 , piece_id=VUZB3QRFST2T3HJHMEYYZ6CV7CKJTFWQGNOBDTXHGET2MBUIC4HQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:56.329163998 (301h14m41.791256481s) [noise=true , address=45.91.116.11:4117 , piece_id=QKEYHSK5JOL65H5T2KUEAEK6VKTNCZAYXIEZTMB76PV7P5PEVILQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:56.329201379 (301h14m41.791293862s) [seg_pos={0 0}, upload=265104]: Segment reached optimal threshold of 13 pieces. Oct 6 02:45:56.329362146 (301h14m41.791454629s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Oct 6 02:45:56.329371974 (301h14m41.791464457s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329546396 (301h14m41.791638869s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: write tcp 188.166.225.170:57604->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 02:45:56.329556665 (301h14m41.791649148s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329567937 (301h14m41.791660421s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: rpc: dial tcp 77.90.31.19:4841: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 02:45:56.329579870 (301h14m41.791672353s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329599808 (301h14m41.791692291s) [seg_pos={0 0}, upload=265104]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 6 02:45:56.329683116 (301h14m41.7917756s) [upload=265104]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 6 02:45:56.329690771 (301h14m41.791783254s) [upload=265104]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 6 02:45:56.636341641 (301h14m42.098434134s) [upload=265104]: Done uploading segments... Oct 6 02:45:56.636368131 (301h14m42.098460614s) [upload=265104]: Upload finished. err: <nil> Duration: 5.971744983s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............X.....................X.......................] duration: 6.270393676s FAIL (2025-10-06 03:05:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: 2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: 7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA) context deadline exceeded ======LOGS====== Oct 6 02:45:41.756337907 (301h14m27.21843039s) [upload=265104]: Starting upload Oct 6 02:45:41.756430433 (301h14m27.218522916s) [upload=265104]: Done starting upload Oct 6 02:45:41.756513701 (301h14m27.218606185s) [upload=265104]: Uploading segments... Oct 6 02:45:41.756752276 (301h14m27.218844759s) [upload=265104]: Got next segment. Inline: false Oct 6 02:45:41.756794536 (301h14m27.218887019s) [upload=265104]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 02:45:41.756800898 (301h14m27.218893381s) [upload=265104]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 02:45:42.009327267 (301h14m27.47141975s) [seg_pos={0 0}, upload=265104]: Begin upload segment... Oct 6 02:45:42.009417419 (301h14m27.471509892s) [seg_pos={0 0}, upload=265104]: Done begin upload segment. Oct 6 02:45:42.009444500 (301h14m27.471536983s) [noise=true , address=77.90.31.14:4662 , piece_id=MPVBPBB63JILLOUF3N7SRLCZ2F74LX7Z4AH7ZFG3BDNBHT4UIXTQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.009561473 (301h14m27.471653956s) [noise=true , address=145.79.227.19:4077 , piece_id=EJIAPZSBMAO77E33V6MEBOQHUXVVYKMAVP53QAZXZSYBYDFP7MMQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.010197574 (301h14m27.472290057s) [noise=true , address=145.79.227.15:4875 , piece_id=PBDTNZ2NWRZZC3VCIQCVR2WLT2AP6662I2SSDNP6BLUDKCJMRK6Q, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.010634216 (301h14m27.472726699s) [noise=true , address=145.79.227.4:4185 , piece_id=3AD745KFD67M4XFNDXMJVEU34RRBQXXE4P3HVJJRDBHK3IIXNYTA, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011073742 (301h14m27.473166225s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011505996 (301h14m27.473598479s) [noise=true , address=45.91.116.10:4081 , piece_id=FTXZ2LOHQMY74KOX6XZC3TLHWXOGDTIY3SZLSPTE7THGKNLMPXRQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011982502 (301h14m27.474074985s) [noise=true , address=45.91.116.6:4974 , piece_id=7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.012488236 (301h14m27.474580719s) [noise=true , address=145.79.227.7:4294 , piece_id=7E3B4BGLCTW6C5DM6PPDJPWGVJS5HEGYPAPXRAMX7XDSG7USPQFQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.013242472 (301h14m27.475334955s) [noise=true , address=45.91.116.13:4154 , piece_id=2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.014529382 (301h14m27.476621855s) [noise=true , address=45.91.116.6:4975 , piece_id=T3VWH5OBQ7FE66GCKHZLEDSIXFVYEU3V52HI4CPD27QL3RH75K7Q, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.014632619 (301h14m27.476725102s) [noise=true , address=45.91.116.11:4117 , piece_id=QKEYHSK5JOL65H5T2KUEAEK6VKTNCZAYXIEZTMB76PV7P5PEVILQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.015990214 (301h14m27.478082697s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.016291928 (301h14m27.478384412s) [noise=true , address=45.91.116.12:4226 , piece_id=NM7SEIKZYUHZLNVSOGYVZWMI7U7X5SOXOJDAEGY5NF3M6QQFPZSQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.017455915 (301h14m27.479548399s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.018864467 (301h14m27.48095694s) [noise=true , address=77.90.31.10:4518 , piece_id=6FGNAMVK6OAGJTZJNKTUJWGRW7MLGMJMGJKMAINWMPW7BV74OO6A, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.020222022 (301h14m27.482314495s) [noise=true , address=45.91.116.11:4118 , piece_id=VUZB3QRFST2T3HJHMEYYZ6CV7CKJTFWQGNOBDTXHGET2MBUIC4HQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:43.755608897 (301h14m29.21770138s) [upload=265104]: Next returned nil segment Oct 6 02:45:43.755631300 (301h14m29.217723773s) [upload=265104]: Waiting for error group managing segments... Oct 6 02:45:44.043930124 (301h14m29.506022597s) [noise=true , address=145.79.227.19:4077 , piece_id=EJIAPZSBMAO77E33V6MEBOQHUXVVYKMAVP53QAZXZSYBYDFP7MMQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.045501296 (301h14m29.507593769s) [noise=true , address=145.79.227.4:4185 , piece_id=3AD745KFD67M4XFNDXMJVEU34RRBQXXE4P3HVJJRDBHK3IIXNYTA, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.057881199 (301h14m29.519973672s) [noise=true , address=145.79.227.15:4875 , piece_id=PBDTNZ2NWRZZC3VCIQCVR2WLT2AP6662I2SSDNP6BLUDKCJMRK6Q, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.068726540 (301h14m29.530819023s) [noise=true , address=145.79.227.7:4294 , piece_id=7E3B4BGLCTW6C5DM6PPDJPWGVJS5HEGYPAPXRAMX7XDSG7USPQFQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.897895712 (301h14m30.359988185s) [noise=true , address=45.91.116.12:4226 , piece_id=NM7SEIKZYUHZLNVSOGYVZWMI7U7X5SOXOJDAEGY5NF3M6QQFPZSQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.252859665 (301h14m30.714952148s) [noise=true , address=77.90.31.14:4662 , piece_id=MPVBPBB63JILLOUF3N7SRLCZ2F74LX7Z4AH7ZFG3BDNBHT4UIXTQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.318952216 (301h14m30.781044699s) [noise=true , address=45.91.116.6:4974 , piece_id=7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.836439779 (301h14m31.298532262s) [noise=true , address=45.91.116.10:4081 , piece_id=FTXZ2LOHQMY74KOX6XZC3TLHWXOGDTIY3SZLSPTE7THGKNLMPXRQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.975895173 (301h14m31.437987656s) [noise=true , address=45.91.116.6:4975 , piece_id=T3VWH5OBQ7FE66GCKHZLEDSIXFVYEU3V52HI4CPD27QL3RH75K7Q, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:46.826703981 (301h14m32.288796454s) [noise=true , address=77.90.31.10:4518 , piece_id=6FGNAMVK6OAGJTZJNKTUJWGRW7MLGMJMGJKMAINWMPW7BV74OO6A, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:48.069759484 (301h14m33.531851968s) [noise=true , address=45.91.116.13:4154 , piece_id=2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:54.791582362 (301h14m40.253674835s) [noise=true , address=45.91.116.11:4118 , piece_id=VUZB3QRFST2T3HJHMEYYZ6CV7CKJTFWQGNOBDTXHGET2MBUIC4HQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:56.329163998 (301h14m41.791256481s) [noise=true , address=45.91.116.11:4117 , piece_id=QKEYHSK5JOL65H5T2KUEAEK6VKTNCZAYXIEZTMB76PV7P5PEVILQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:56.329201379 (301h14m41.791293862s) [seg_pos={0 0}, upload=265104]: Segment reached optimal threshold of 13 pieces. Oct 6 02:45:56.329362146 (301h14m41.791454629s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Oct 6 02:45:56.329371974 (301h14m41.791464457s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329546396 (301h14m41.791638869s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: write tcp 188.166.225.170:57604->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 02:45:56.329556665 (301h14m41.791649148s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329567937 (301h14m41.791660421s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: rpc: dial tcp 77.90.31.19:4841: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 02:45:56.329579870 (301h14m41.791672353s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329599808 (301h14m41.791692291s) [seg_pos={0 0}, upload=265104]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 6 02:45:56.329683116 (301h14m41.7917756s) [upload=265104]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 6 02:45:56.329690771 (301h14m41.791783254s) [upload=265104]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 6 02:45:56.636341641 (301h14m42.098434134s) [upload=265104]: Done uploading segments... Oct 6 02:45:56.636368131 (301h14m42.098460614s) [upload=265104]: Upload finished. err: <nil> Duration: 6.270435927s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............X.....................X......................] duration: 6.232958304s FAIL (2025-10-06 03:04:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: 2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: 7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA) context deadline exceeded ======LOGS====== Oct 6 02:45:41.756337907 (301h14m27.21843039s) [upload=265104]: Starting upload Oct 6 02:45:41.756430433 (301h14m27.218522916s) [upload=265104]: Done starting upload Oct 6 02:45:41.756513701 (301h14m27.218606185s) [upload=265104]: Uploading segments... Oct 6 02:45:41.756752276 (301h14m27.218844759s) [upload=265104]: Got next segment. Inline: false Oct 6 02:45:41.756794536 (301h14m27.218887019s) [upload=265104]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 02:45:41.756800898 (301h14m27.218893381s) [upload=265104]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 02:45:42.009327267 (301h14m27.47141975s) [seg_pos={0 0}, upload=265104]: Begin upload segment... Oct 6 02:45:42.009417419 (301h14m27.471509892s) [seg_pos={0 0}, upload=265104]: Done begin upload segment. Oct 6 02:45:42.009444500 (301h14m27.471536983s) [noise=true , address=77.90.31.14:4662 , piece_id=MPVBPBB63JILLOUF3N7SRLCZ2F74LX7Z4AH7ZFG3BDNBHT4UIXTQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.009561473 (301h14m27.471653956s) [noise=true , address=145.79.227.19:4077 , piece_id=EJIAPZSBMAO77E33V6MEBOQHUXVVYKMAVP53QAZXZSYBYDFP7MMQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.010197574 (301h14m27.472290057s) [noise=true , address=145.79.227.15:4875 , piece_id=PBDTNZ2NWRZZC3VCIQCVR2WLT2AP6662I2SSDNP6BLUDKCJMRK6Q, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.010634216 (301h14m27.472726699s) [noise=true , address=145.79.227.4:4185 , piece_id=3AD745KFD67M4XFNDXMJVEU34RRBQXXE4P3HVJJRDBHK3IIXNYTA, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011073742 (301h14m27.473166225s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011505996 (301h14m27.473598479s) [noise=true , address=45.91.116.10:4081 , piece_id=FTXZ2LOHQMY74KOX6XZC3TLHWXOGDTIY3SZLSPTE7THGKNLMPXRQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011982502 (301h14m27.474074985s) [noise=true , address=45.91.116.6:4974 , piece_id=7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.012488236 (301h14m27.474580719s) [noise=true , address=145.79.227.7:4294 , piece_id=7E3B4BGLCTW6C5DM6PPDJPWGVJS5HEGYPAPXRAMX7XDSG7USPQFQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.013242472 (301h14m27.475334955s) [noise=true , address=45.91.116.13:4154 , piece_id=2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.014529382 (301h14m27.476621855s) [noise=true , address=45.91.116.6:4975 , piece_id=T3VWH5OBQ7FE66GCKHZLEDSIXFVYEU3V52HI4CPD27QL3RH75K7Q, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.014632619 (301h14m27.476725102s) [noise=true , address=45.91.116.11:4117 , piece_id=QKEYHSK5JOL65H5T2KUEAEK6VKTNCZAYXIEZTMB76PV7P5PEVILQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.015990214 (301h14m27.478082697s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.016291928 (301h14m27.478384412s) [noise=true , address=45.91.116.12:4226 , piece_id=NM7SEIKZYUHZLNVSOGYVZWMI7U7X5SOXOJDAEGY5NF3M6QQFPZSQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.017455915 (301h14m27.479548399s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.018864467 (301h14m27.48095694s) [noise=true , address=77.90.31.10:4518 , piece_id=6FGNAMVK6OAGJTZJNKTUJWGRW7MLGMJMGJKMAINWMPW7BV74OO6A, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.020222022 (301h14m27.482314495s) [noise=true , address=45.91.116.11:4118 , piece_id=VUZB3QRFST2T3HJHMEYYZ6CV7CKJTFWQGNOBDTXHGET2MBUIC4HQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:43.755608897 (301h14m29.21770138s) [upload=265104]: Next returned nil segment Oct 6 02:45:43.755631300 (301h14m29.217723773s) [upload=265104]: Waiting for error group managing segments... Oct 6 02:45:44.043930124 (301h14m29.506022597s) [noise=true , address=145.79.227.19:4077 , piece_id=EJIAPZSBMAO77E33V6MEBOQHUXVVYKMAVP53QAZXZSYBYDFP7MMQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.045501296 (301h14m29.507593769s) [noise=true , address=145.79.227.4:4185 , piece_id=3AD745KFD67M4XFNDXMJVEU34RRBQXXE4P3HVJJRDBHK3IIXNYTA, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.057881199 (301h14m29.519973672s) [noise=true , address=145.79.227.15:4875 , piece_id=PBDTNZ2NWRZZC3VCIQCVR2WLT2AP6662I2SSDNP6BLUDKCJMRK6Q, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.068726540 (301h14m29.530819023s) [noise=true , address=145.79.227.7:4294 , piece_id=7E3B4BGLCTW6C5DM6PPDJPWGVJS5HEGYPAPXRAMX7XDSG7USPQFQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.897895712 (301h14m30.359988185s) [noise=true , address=45.91.116.12:4226 , piece_id=NM7SEIKZYUHZLNVSOGYVZWMI7U7X5SOXOJDAEGY5NF3M6QQFPZSQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.252859665 (301h14m30.714952148s) [noise=true , address=77.90.31.14:4662 , piece_id=MPVBPBB63JILLOUF3N7SRLCZ2F74LX7Z4AH7ZFG3BDNBHT4UIXTQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.318952216 (301h14m30.781044699s) [noise=true , address=45.91.116.6:4974 , piece_id=7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.836439779 (301h14m31.298532262s) [noise=true , address=45.91.116.10:4081 , piece_id=FTXZ2LOHQMY74KOX6XZC3TLHWXOGDTIY3SZLSPTE7THGKNLMPXRQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.975895173 (301h14m31.437987656s) [noise=true , address=45.91.116.6:4975 , piece_id=T3VWH5OBQ7FE66GCKHZLEDSIXFVYEU3V52HI4CPD27QL3RH75K7Q, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:46.826703981 (301h14m32.288796454s) [noise=true , address=77.90.31.10:4518 , piece_id=6FGNAMVK6OAGJTZJNKTUJWGRW7MLGMJMGJKMAINWMPW7BV74OO6A, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:48.069759484 (301h14m33.531851968s) [noise=true , address=45.91.116.13:4154 , piece_id=2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:54.791582362 (301h14m40.253674835s) [noise=true , address=45.91.116.11:4118 , piece_id=VUZB3QRFST2T3HJHMEYYZ6CV7CKJTFWQGNOBDTXHGET2MBUIC4HQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:56.329163998 (301h14m41.791256481s) [noise=true , address=45.91.116.11:4117 , piece_id=QKEYHSK5JOL65H5T2KUEAEK6VKTNCZAYXIEZTMB76PV7P5PEVILQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:56.329201379 (301h14m41.791293862s) [seg_pos={0 0}, upload=265104]: Segment reached optimal threshold of 13 pieces. Oct 6 02:45:56.329362146 (301h14m41.791454629s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Oct 6 02:45:56.329371974 (301h14m41.791464457s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329546396 (301h14m41.791638869s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: write tcp 188.166.225.170:57604->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 02:45:56.329556665 (301h14m41.791649148s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329567937 (301h14m41.791660421s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: rpc: dial tcp 77.90.31.19:4841: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 02:45:56.329579870 (301h14m41.791672353s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329599808 (301h14m41.791692291s) [seg_pos={0 0}, upload=265104]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 6 02:45:56.329683116 (301h14m41.7917756s) [upload=265104]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 6 02:45:56.329690771 (301h14m41.791783254s) [upload=265104]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 6 02:45:56.636341641 (301h14m42.098434134s) [upload=265104]: Done uploading segments... Oct 6 02:45:56.636368131 (301h14m42.098460614s) [upload=265104]: Upload finished. err: <nil> Duration: 6.23299782s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................X.....................X.....................] duration: 6.102813125s FAIL (2025-10-06 03:03:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: 2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: 7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA) context deadline exceeded ======LOGS====== Oct 6 02:45:41.756337907 (301h14m27.21843039s) [upload=265104]: Starting upload Oct 6 02:45:41.756430433 (301h14m27.218522916s) [upload=265104]: Done starting upload Oct 6 02:45:41.756513701 (301h14m27.218606185s) [upload=265104]: Uploading segments... Oct 6 02:45:41.756752276 (301h14m27.218844759s) [upload=265104]: Got next segment. Inline: false Oct 6 02:45:41.756794536 (301h14m27.218887019s) [upload=265104]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 02:45:41.756800898 (301h14m27.218893381s) [upload=265104]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 02:45:42.009327267 (301h14m27.47141975s) [seg_pos={0 0}, upload=265104]: Begin upload segment... Oct 6 02:45:42.009417419 (301h14m27.471509892s) [seg_pos={0 0}, upload=265104]: Done begin upload segment. Oct 6 02:45:42.009444500 (301h14m27.471536983s) [noise=true , address=77.90.31.14:4662 , piece_id=MPVBPBB63JILLOUF3N7SRLCZ2F74LX7Z4AH7ZFG3BDNBHT4UIXTQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.009561473 (301h14m27.471653956s) [noise=true , address=145.79.227.19:4077 , piece_id=EJIAPZSBMAO77E33V6MEBOQHUXVVYKMAVP53QAZXZSYBYDFP7MMQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.010197574 (301h14m27.472290057s) [noise=true , address=145.79.227.15:4875 , piece_id=PBDTNZ2NWRZZC3VCIQCVR2WLT2AP6662I2SSDNP6BLUDKCJMRK6Q, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.010634216 (301h14m27.472726699s) [noise=true , address=145.79.227.4:4185 , piece_id=3AD745KFD67M4XFNDXMJVEU34RRBQXXE4P3HVJJRDBHK3IIXNYTA, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011073742 (301h14m27.473166225s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011505996 (301h14m27.473598479s) [noise=true , address=45.91.116.10:4081 , piece_id=FTXZ2LOHQMY74KOX6XZC3TLHWXOGDTIY3SZLSPTE7THGKNLMPXRQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011982502 (301h14m27.474074985s) [noise=true , address=45.91.116.6:4974 , piece_id=7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.012488236 (301h14m27.474580719s) [noise=true , address=145.79.227.7:4294 , piece_id=7E3B4BGLCTW6C5DM6PPDJPWGVJS5HEGYPAPXRAMX7XDSG7USPQFQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.013242472 (301h14m27.475334955s) [noise=true , address=45.91.116.13:4154 , piece_id=2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.014529382 (301h14m27.476621855s) [noise=true , address=45.91.116.6:4975 , piece_id=T3VWH5OBQ7FE66GCKHZLEDSIXFVYEU3V52HI4CPD27QL3RH75K7Q, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.014632619 (301h14m27.476725102s) [noise=true , address=45.91.116.11:4117 , piece_id=QKEYHSK5JOL65H5T2KUEAEK6VKTNCZAYXIEZTMB76PV7P5PEVILQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.015990214 (301h14m27.478082697s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.016291928 (301h14m27.478384412s) [noise=true , address=45.91.116.12:4226 , piece_id=NM7SEIKZYUHZLNVSOGYVZWMI7U7X5SOXOJDAEGY5NF3M6QQFPZSQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.017455915 (301h14m27.479548399s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.018864467 (301h14m27.48095694s) [noise=true , address=77.90.31.10:4518 , piece_id=6FGNAMVK6OAGJTZJNKTUJWGRW7MLGMJMGJKMAINWMPW7BV74OO6A, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.020222022 (301h14m27.482314495s) [noise=true , address=45.91.116.11:4118 , piece_id=VUZB3QRFST2T3HJHMEYYZ6CV7CKJTFWQGNOBDTXHGET2MBUIC4HQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:43.755608897 (301h14m29.21770138s) [upload=265104]: Next returned nil segment Oct 6 02:45:43.755631300 (301h14m29.217723773s) [upload=265104]: Waiting for error group managing segments... Oct 6 02:45:44.043930124 (301h14m29.506022597s) [noise=true , address=145.79.227.19:4077 , piece_id=EJIAPZSBMAO77E33V6MEBOQHUXVVYKMAVP53QAZXZSYBYDFP7MMQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.045501296 (301h14m29.507593769s) [noise=true , address=145.79.227.4:4185 , piece_id=3AD745KFD67M4XFNDXMJVEU34RRBQXXE4P3HVJJRDBHK3IIXNYTA, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.057881199 (301h14m29.519973672s) [noise=true , address=145.79.227.15:4875 , piece_id=PBDTNZ2NWRZZC3VCIQCVR2WLT2AP6662I2SSDNP6BLUDKCJMRK6Q, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.068726540 (301h14m29.530819023s) [noise=true , address=145.79.227.7:4294 , piece_id=7E3B4BGLCTW6C5DM6PPDJPWGVJS5HEGYPAPXRAMX7XDSG7USPQFQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.897895712 (301h14m30.359988185s) [noise=true , address=45.91.116.12:4226 , piece_id=NM7SEIKZYUHZLNVSOGYVZWMI7U7X5SOXOJDAEGY5NF3M6QQFPZSQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.252859665 (301h14m30.714952148s) [noise=true , address=77.90.31.14:4662 , piece_id=MPVBPBB63JILLOUF3N7SRLCZ2F74LX7Z4AH7ZFG3BDNBHT4UIXTQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.318952216 (301h14m30.781044699s) [noise=true , address=45.91.116.6:4974 , piece_id=7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.836439779 (301h14m31.298532262s) [noise=true , address=45.91.116.10:4081 , piece_id=FTXZ2LOHQMY74KOX6XZC3TLHWXOGDTIY3SZLSPTE7THGKNLMPXRQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.975895173 (301h14m31.437987656s) [noise=true , address=45.91.116.6:4975 , piece_id=T3VWH5OBQ7FE66GCKHZLEDSIXFVYEU3V52HI4CPD27QL3RH75K7Q, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:46.826703981 (301h14m32.288796454s) [noise=true , address=77.90.31.10:4518 , piece_id=6FGNAMVK6OAGJTZJNKTUJWGRW7MLGMJMGJKMAINWMPW7BV74OO6A, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:48.069759484 (301h14m33.531851968s) [noise=true , address=45.91.116.13:4154 , piece_id=2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:54.791582362 (301h14m40.253674835s) [noise=true , address=45.91.116.11:4118 , piece_id=VUZB3QRFST2T3HJHMEYYZ6CV7CKJTFWQGNOBDTXHGET2MBUIC4HQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:56.329163998 (301h14m41.791256481s) [noise=true , address=45.91.116.11:4117 , piece_id=QKEYHSK5JOL65H5T2KUEAEK6VKTNCZAYXIEZTMB76PV7P5PEVILQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:56.329201379 (301h14m41.791293862s) [seg_pos={0 0}, upload=265104]: Segment reached optimal threshold of 13 pieces. Oct 6 02:45:56.329362146 (301h14m41.791454629s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Oct 6 02:45:56.329371974 (301h14m41.791464457s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329546396 (301h14m41.791638869s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: write tcp 188.166.225.170:57604->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 02:45:56.329556665 (301h14m41.791649148s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329567937 (301h14m41.791660421s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: rpc: dial tcp 77.90.31.19:4841: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 02:45:56.329579870 (301h14m41.791672353s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329599808 (301h14m41.791692291s) [seg_pos={0 0}, upload=265104]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 6 02:45:56.329683116 (301h14m41.7917756s) [upload=265104]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 6 02:45:56.329690771 (301h14m41.791783254s) [upload=265104]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 6 02:45:56.636341641 (301h14m42.098434134s) [upload=265104]: Done uploading segments... Oct 6 02:45:56.636368131 (301h14m42.098460614s) [upload=265104]: Upload finished. err: <nil> Duration: 6.102846748s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................X.....................X....................] duration: 6.013953016s FAIL (2025-10-06 03:02:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: 2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: 7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA) context deadline exceeded ======LOGS====== Oct 6 02:45:41.756337907 (301h14m27.21843039s) [upload=265104]: Starting upload Oct 6 02:45:41.756430433 (301h14m27.218522916s) [upload=265104]: Done starting upload Oct 6 02:45:41.756513701 (301h14m27.218606185s) [upload=265104]: Uploading segments... Oct 6 02:45:41.756752276 (301h14m27.218844759s) [upload=265104]: Got next segment. Inline: false Oct 6 02:45:41.756794536 (301h14m27.218887019s) [upload=265104]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 02:45:41.756800898 (301h14m27.218893381s) [upload=265104]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 02:45:42.009327267 (301h14m27.47141975s) [seg_pos={0 0}, upload=265104]: Begin upload segment... Oct 6 02:45:42.009417419 (301h14m27.471509892s) [seg_pos={0 0}, upload=265104]: Done begin upload segment. Oct 6 02:45:42.009444500 (301h14m27.471536983s) [noise=true , address=77.90.31.14:4662 , piece_id=MPVBPBB63JILLOUF3N7SRLCZ2F74LX7Z4AH7ZFG3BDNBHT4UIXTQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.009561473 (301h14m27.471653956s) [noise=true , address=145.79.227.19:4077 , piece_id=EJIAPZSBMAO77E33V6MEBOQHUXVVYKMAVP53QAZXZSYBYDFP7MMQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.010197574 (301h14m27.472290057s) [noise=true , address=145.79.227.15:4875 , piece_id=PBDTNZ2NWRZZC3VCIQCVR2WLT2AP6662I2SSDNP6BLUDKCJMRK6Q, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.010634216 (301h14m27.472726699s) [noise=true , address=145.79.227.4:4185 , piece_id=3AD745KFD67M4XFNDXMJVEU34RRBQXXE4P3HVJJRDBHK3IIXNYTA, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011073742 (301h14m27.473166225s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011505996 (301h14m27.473598479s) [noise=true , address=45.91.116.10:4081 , piece_id=FTXZ2LOHQMY74KOX6XZC3TLHWXOGDTIY3SZLSPTE7THGKNLMPXRQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011982502 (301h14m27.474074985s) [noise=true , address=45.91.116.6:4974 , piece_id=7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.012488236 (301h14m27.474580719s) [noise=true , address=145.79.227.7:4294 , piece_id=7E3B4BGLCTW6C5DM6PPDJPWGVJS5HEGYPAPXRAMX7XDSG7USPQFQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.013242472 (301h14m27.475334955s) [noise=true , address=45.91.116.13:4154 , piece_id=2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.014529382 (301h14m27.476621855s) [noise=true , address=45.91.116.6:4975 , piece_id=T3VWH5OBQ7FE66GCKHZLEDSIXFVYEU3V52HI4CPD27QL3RH75K7Q, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.014632619 (301h14m27.476725102s) [noise=true , address=45.91.116.11:4117 , piece_id=QKEYHSK5JOL65H5T2KUEAEK6VKTNCZAYXIEZTMB76PV7P5PEVILQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.015990214 (301h14m27.478082697s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.016291928 (301h14m27.478384412s) [noise=true , address=45.91.116.12:4226 , piece_id=NM7SEIKZYUHZLNVSOGYVZWMI7U7X5SOXOJDAEGY5NF3M6QQFPZSQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.017455915 (301h14m27.479548399s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.018864467 (301h14m27.48095694s) [noise=true , address=77.90.31.10:4518 , piece_id=6FGNAMVK6OAGJTZJNKTUJWGRW7MLGMJMGJKMAINWMPW7BV74OO6A, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.020222022 (301h14m27.482314495s) [noise=true , address=45.91.116.11:4118 , piece_id=VUZB3QRFST2T3HJHMEYYZ6CV7CKJTFWQGNOBDTXHGET2MBUIC4HQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:43.755608897 (301h14m29.21770138s) [upload=265104]: Next returned nil segment Oct 6 02:45:43.755631300 (301h14m29.217723773s) [upload=265104]: Waiting for error group managing segments... Oct 6 02:45:44.043930124 (301h14m29.506022597s) [noise=true , address=145.79.227.19:4077 , piece_id=EJIAPZSBMAO77E33V6MEBOQHUXVVYKMAVP53QAZXZSYBYDFP7MMQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.045501296 (301h14m29.507593769s) [noise=true , address=145.79.227.4:4185 , piece_id=3AD745KFD67M4XFNDXMJVEU34RRBQXXE4P3HVJJRDBHK3IIXNYTA, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.057881199 (301h14m29.519973672s) [noise=true , address=145.79.227.15:4875 , piece_id=PBDTNZ2NWRZZC3VCIQCVR2WLT2AP6662I2SSDNP6BLUDKCJMRK6Q, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.068726540 (301h14m29.530819023s) [noise=true , address=145.79.227.7:4294 , piece_id=7E3B4BGLCTW6C5DM6PPDJPWGVJS5HEGYPAPXRAMX7XDSG7USPQFQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.897895712 (301h14m30.359988185s) [noise=true , address=45.91.116.12:4226 , piece_id=NM7SEIKZYUHZLNVSOGYVZWMI7U7X5SOXOJDAEGY5NF3M6QQFPZSQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.252859665 (301h14m30.714952148s) [noise=true , address=77.90.31.14:4662 , piece_id=MPVBPBB63JILLOUF3N7SRLCZ2F74LX7Z4AH7ZFG3BDNBHT4UIXTQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.318952216 (301h14m30.781044699s) [noise=true , address=45.91.116.6:4974 , piece_id=7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.836439779 (301h14m31.298532262s) [noise=true , address=45.91.116.10:4081 , piece_id=FTXZ2LOHQMY74KOX6XZC3TLHWXOGDTIY3SZLSPTE7THGKNLMPXRQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.975895173 (301h14m31.437987656s) [noise=true , address=45.91.116.6:4975 , piece_id=T3VWH5OBQ7FE66GCKHZLEDSIXFVYEU3V52HI4CPD27QL3RH75K7Q, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:46.826703981 (301h14m32.288796454s) [noise=true , address=77.90.31.10:4518 , piece_id=6FGNAMVK6OAGJTZJNKTUJWGRW7MLGMJMGJKMAINWMPW7BV74OO6A, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:48.069759484 (301h14m33.531851968s) [noise=true , address=45.91.116.13:4154 , piece_id=2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:54.791582362 (301h14m40.253674835s) [noise=true , address=45.91.116.11:4118 , piece_id=VUZB3QRFST2T3HJHMEYYZ6CV7CKJTFWQGNOBDTXHGET2MBUIC4HQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:56.329163998 (301h14m41.791256481s) [noise=true , address=45.91.116.11:4117 , piece_id=QKEYHSK5JOL65H5T2KUEAEK6VKTNCZAYXIEZTMB76PV7P5PEVILQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:56.329201379 (301h14m41.791293862s) [seg_pos={0 0}, upload=265104]: Segment reached optimal threshold of 13 pieces. Oct 6 02:45:56.329362146 (301h14m41.791454629s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Oct 6 02:45:56.329371974 (301h14m41.791464457s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329546396 (301h14m41.791638869s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: write tcp 188.166.225.170:57604->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 02:45:56.329556665 (301h14m41.791649148s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329567937 (301h14m41.791660421s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: rpc: dial tcp 77.90.31.19:4841: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 02:45:56.329579870 (301h14m41.791672353s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329599808 (301h14m41.791692291s) [seg_pos={0 0}, upload=265104]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 6 02:45:56.329683116 (301h14m41.7917756s) [upload=265104]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 6 02:45:56.329690771 (301h14m41.791783254s) [upload=265104]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 6 02:45:56.636341641 (301h14m42.098434134s) [upload=265104]: Done uploading segments... Oct 6 02:45:56.636368131 (301h14m42.098460614s) [upload=265104]: Upload finished. err: <nil> Duration: 6.013999254s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................X.....................X...................] duration: 6.015286031s FAIL (2025-10-06 03:02:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: 2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: 7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA) context deadline exceeded ======LOGS====== Oct 6 02:45:41.756337907 (301h14m27.21843039s) [upload=265104]: Starting upload Oct 6 02:45:41.756430433 (301h14m27.218522916s) [upload=265104]: Done starting upload Oct 6 02:45:41.756513701 (301h14m27.218606185s) [upload=265104]: Uploading segments... Oct 6 02:45:41.756752276 (301h14m27.218844759s) [upload=265104]: Got next segment. Inline: false Oct 6 02:45:41.756794536 (301h14m27.218887019s) [upload=265104]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 02:45:41.756800898 (301h14m27.218893381s) [upload=265104]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 02:45:42.009327267 (301h14m27.47141975s) [seg_pos={0 0}, upload=265104]: Begin upload segment... Oct 6 02:45:42.009417419 (301h14m27.471509892s) [seg_pos={0 0}, upload=265104]: Done begin upload segment. Oct 6 02:45:42.009444500 (301h14m27.471536983s) [noise=true , address=77.90.31.14:4662 , piece_id=MPVBPBB63JILLOUF3N7SRLCZ2F74LX7Z4AH7ZFG3BDNBHT4UIXTQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.009561473 (301h14m27.471653956s) [noise=true , address=145.79.227.19:4077 , piece_id=EJIAPZSBMAO77E33V6MEBOQHUXVVYKMAVP53QAZXZSYBYDFP7MMQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.010197574 (301h14m27.472290057s) [noise=true , address=145.79.227.15:4875 , piece_id=PBDTNZ2NWRZZC3VCIQCVR2WLT2AP6662I2SSDNP6BLUDKCJMRK6Q, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.010634216 (301h14m27.472726699s) [noise=true , address=145.79.227.4:4185 , piece_id=3AD745KFD67M4XFNDXMJVEU34RRBQXXE4P3HVJJRDBHK3IIXNYTA, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011073742 (301h14m27.473166225s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011505996 (301h14m27.473598479s) [noise=true , address=45.91.116.10:4081 , piece_id=FTXZ2LOHQMY74KOX6XZC3TLHWXOGDTIY3SZLSPTE7THGKNLMPXRQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011982502 (301h14m27.474074985s) [noise=true , address=45.91.116.6:4974 , piece_id=7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.012488236 (301h14m27.474580719s) [noise=true , address=145.79.227.7:4294 , piece_id=7E3B4BGLCTW6C5DM6PPDJPWGVJS5HEGYPAPXRAMX7XDSG7USPQFQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.013242472 (301h14m27.475334955s) [noise=true , address=45.91.116.13:4154 , piece_id=2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.014529382 (301h14m27.476621855s) [noise=true , address=45.91.116.6:4975 , piece_id=T3VWH5OBQ7FE66GCKHZLEDSIXFVYEU3V52HI4CPD27QL3RH75K7Q, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.014632619 (301h14m27.476725102s) [noise=true , address=45.91.116.11:4117 , piece_id=QKEYHSK5JOL65H5T2KUEAEK6VKTNCZAYXIEZTMB76PV7P5PEVILQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.015990214 (301h14m27.478082697s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.016291928 (301h14m27.478384412s) [noise=true , address=45.91.116.12:4226 , piece_id=NM7SEIKZYUHZLNVSOGYVZWMI7U7X5SOXOJDAEGY5NF3M6QQFPZSQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.017455915 (301h14m27.479548399s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.018864467 (301h14m27.48095694s) [noise=true , address=77.90.31.10:4518 , piece_id=6FGNAMVK6OAGJTZJNKTUJWGRW7MLGMJMGJKMAINWMPW7BV74OO6A, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.020222022 (301h14m27.482314495s) [noise=true , address=45.91.116.11:4118 , piece_id=VUZB3QRFST2T3HJHMEYYZ6CV7CKJTFWQGNOBDTXHGET2MBUIC4HQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:43.755608897 (301h14m29.21770138s) [upload=265104]: Next returned nil segment Oct 6 02:45:43.755631300 (301h14m29.217723773s) [upload=265104]: Waiting for error group managing segments... Oct 6 02:45:44.043930124 (301h14m29.506022597s) [noise=true , address=145.79.227.19:4077 , piece_id=EJIAPZSBMAO77E33V6MEBOQHUXVVYKMAVP53QAZXZSYBYDFP7MMQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.045501296 (301h14m29.507593769s) [noise=true , address=145.79.227.4:4185 , piece_id=3AD745KFD67M4XFNDXMJVEU34RRBQXXE4P3HVJJRDBHK3IIXNYTA, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.057881199 (301h14m29.519973672s) [noise=true , address=145.79.227.15:4875 , piece_id=PBDTNZ2NWRZZC3VCIQCVR2WLT2AP6662I2SSDNP6BLUDKCJMRK6Q, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.068726540 (301h14m29.530819023s) [noise=true , address=145.79.227.7:4294 , piece_id=7E3B4BGLCTW6C5DM6PPDJPWGVJS5HEGYPAPXRAMX7XDSG7USPQFQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.897895712 (301h14m30.359988185s) [noise=true , address=45.91.116.12:4226 , piece_id=NM7SEIKZYUHZLNVSOGYVZWMI7U7X5SOXOJDAEGY5NF3M6QQFPZSQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.252859665 (301h14m30.714952148s) [noise=true , address=77.90.31.14:4662 , piece_id=MPVBPBB63JILLOUF3N7SRLCZ2F74LX7Z4AH7ZFG3BDNBHT4UIXTQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.318952216 (301h14m30.781044699s) [noise=true , address=45.91.116.6:4974 , piece_id=7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.836439779 (301h14m31.298532262s) [noise=true , address=45.91.116.10:4081 , piece_id=FTXZ2LOHQMY74KOX6XZC3TLHWXOGDTIY3SZLSPTE7THGKNLMPXRQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.975895173 (301h14m31.437987656s) [noise=true , address=45.91.116.6:4975 , piece_id=T3VWH5OBQ7FE66GCKHZLEDSIXFVYEU3V52HI4CPD27QL3RH75K7Q, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:46.826703981 (301h14m32.288796454s) [noise=true , address=77.90.31.10:4518 , piece_id=6FGNAMVK6OAGJTZJNKTUJWGRW7MLGMJMGJKMAINWMPW7BV74OO6A, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:48.069759484 (301h14m33.531851968s) [noise=true , address=45.91.116.13:4154 , piece_id=2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:54.791582362 (301h14m40.253674835s) [noise=true , address=45.91.116.11:4118 , piece_id=VUZB3QRFST2T3HJHMEYYZ6CV7CKJTFWQGNOBDTXHGET2MBUIC4HQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:56.329163998 (301h14m41.791256481s) [noise=true , address=45.91.116.11:4117 , piece_id=QKEYHSK5JOL65H5T2KUEAEK6VKTNCZAYXIEZTMB76PV7P5PEVILQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:56.329201379 (301h14m41.791293862s) [seg_pos={0 0}, upload=265104]: Segment reached optimal threshold of 13 pieces. Oct 6 02:45:56.329362146 (301h14m41.791454629s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Oct 6 02:45:56.329371974 (301h14m41.791464457s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329546396 (301h14m41.791638869s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: write tcp 188.166.225.170:57604->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 02:45:56.329556665 (301h14m41.791649148s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329567937 (301h14m41.791660421s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: rpc: dial tcp 77.90.31.19:4841: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 02:45:56.329579870 (301h14m41.791672353s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329599808 (301h14m41.791692291s) [seg_pos={0 0}, upload=265104]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 6 02:45:56.329683116 (301h14m41.7917756s) [upload=265104]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 6 02:45:56.329690771 (301h14m41.791783254s) [upload=265104]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 6 02:45:56.636341641 (301h14m42.098434134s) [upload=265104]: Done uploading segments... Oct 6 02:45:56.636368131 (301h14m42.098460614s) [upload=265104]: Upload finished. err: <nil> Duration: 6.015339514s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................X.....................X..................] duration: 5.982337115s FAIL (2025-10-06 03:00:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: 2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: 7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA) context deadline exceeded ======LOGS====== Oct 6 02:45:41.756337907 (301h14m27.21843039s) [upload=265104]: Starting upload Oct 6 02:45:41.756430433 (301h14m27.218522916s) [upload=265104]: Done starting upload Oct 6 02:45:41.756513701 (301h14m27.218606185s) [upload=265104]: Uploading segments... Oct 6 02:45:41.756752276 (301h14m27.218844759s) [upload=265104]: Got next segment. Inline: false Oct 6 02:45:41.756794536 (301h14m27.218887019s) [upload=265104]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 02:45:41.756800898 (301h14m27.218893381s) [upload=265104]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 02:45:42.009327267 (301h14m27.47141975s) [seg_pos={0 0}, upload=265104]: Begin upload segment... Oct 6 02:45:42.009417419 (301h14m27.471509892s) [seg_pos={0 0}, upload=265104]: Done begin upload segment. Oct 6 02:45:42.009444500 (301h14m27.471536983s) [noise=true , address=77.90.31.14:4662 , piece_id=MPVBPBB63JILLOUF3N7SRLCZ2F74LX7Z4AH7ZFG3BDNBHT4UIXTQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.009561473 (301h14m27.471653956s) [noise=true , address=145.79.227.19:4077 , piece_id=EJIAPZSBMAO77E33V6MEBOQHUXVVYKMAVP53QAZXZSYBYDFP7MMQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.010197574 (301h14m27.472290057s) [noise=true , address=145.79.227.15:4875 , piece_id=PBDTNZ2NWRZZC3VCIQCVR2WLT2AP6662I2SSDNP6BLUDKCJMRK6Q, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.010634216 (301h14m27.472726699s) [noise=true , address=145.79.227.4:4185 , piece_id=3AD745KFD67M4XFNDXMJVEU34RRBQXXE4P3HVJJRDBHK3IIXNYTA, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011073742 (301h14m27.473166225s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011505996 (301h14m27.473598479s) [noise=true , address=45.91.116.10:4081 , piece_id=FTXZ2LOHQMY74KOX6XZC3TLHWXOGDTIY3SZLSPTE7THGKNLMPXRQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011982502 (301h14m27.474074985s) [noise=true , address=45.91.116.6:4974 , piece_id=7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.012488236 (301h14m27.474580719s) [noise=true , address=145.79.227.7:4294 , piece_id=7E3B4BGLCTW6C5DM6PPDJPWGVJS5HEGYPAPXRAMX7XDSG7USPQFQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.013242472 (301h14m27.475334955s) [noise=true , address=45.91.116.13:4154 , piece_id=2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.014529382 (301h14m27.476621855s) [noise=true , address=45.91.116.6:4975 , piece_id=T3VWH5OBQ7FE66GCKHZLEDSIXFVYEU3V52HI4CPD27QL3RH75K7Q, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.014632619 (301h14m27.476725102s) [noise=true , address=45.91.116.11:4117 , piece_id=QKEYHSK5JOL65H5T2KUEAEK6VKTNCZAYXIEZTMB76PV7P5PEVILQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.015990214 (301h14m27.478082697s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.016291928 (301h14m27.478384412s) [noise=true , address=45.91.116.12:4226 , piece_id=NM7SEIKZYUHZLNVSOGYVZWMI7U7X5SOXOJDAEGY5NF3M6QQFPZSQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.017455915 (301h14m27.479548399s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.018864467 (301h14m27.48095694s) [noise=true , address=77.90.31.10:4518 , piece_id=6FGNAMVK6OAGJTZJNKTUJWGRW7MLGMJMGJKMAINWMPW7BV74OO6A, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.020222022 (301h14m27.482314495s) [noise=true , address=45.91.116.11:4118 , piece_id=VUZB3QRFST2T3HJHMEYYZ6CV7CKJTFWQGNOBDTXHGET2MBUIC4HQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:43.755608897 (301h14m29.21770138s) [upload=265104]: Next returned nil segment Oct 6 02:45:43.755631300 (301h14m29.217723773s) [upload=265104]: Waiting for error group managing segments... Oct 6 02:45:44.043930124 (301h14m29.506022597s) [noise=true , address=145.79.227.19:4077 , piece_id=EJIAPZSBMAO77E33V6MEBOQHUXVVYKMAVP53QAZXZSYBYDFP7MMQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.045501296 (301h14m29.507593769s) [noise=true , address=145.79.227.4:4185 , piece_id=3AD745KFD67M4XFNDXMJVEU34RRBQXXE4P3HVJJRDBHK3IIXNYTA, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.057881199 (301h14m29.519973672s) [noise=true , address=145.79.227.15:4875 , piece_id=PBDTNZ2NWRZZC3VCIQCVR2WLT2AP6662I2SSDNP6BLUDKCJMRK6Q, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.068726540 (301h14m29.530819023s) [noise=true , address=145.79.227.7:4294 , piece_id=7E3B4BGLCTW6C5DM6PPDJPWGVJS5HEGYPAPXRAMX7XDSG7USPQFQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.897895712 (301h14m30.359988185s) [noise=true , address=45.91.116.12:4226 , piece_id=NM7SEIKZYUHZLNVSOGYVZWMI7U7X5SOXOJDAEGY5NF3M6QQFPZSQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.252859665 (301h14m30.714952148s) [noise=true , address=77.90.31.14:4662 , piece_id=MPVBPBB63JILLOUF3N7SRLCZ2F74LX7Z4AH7ZFG3BDNBHT4UIXTQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.318952216 (301h14m30.781044699s) [noise=true , address=45.91.116.6:4974 , piece_id=7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.836439779 (301h14m31.298532262s) [noise=true , address=45.91.116.10:4081 , piece_id=FTXZ2LOHQMY74KOX6XZC3TLHWXOGDTIY3SZLSPTE7THGKNLMPXRQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.975895173 (301h14m31.437987656s) [noise=true , address=45.91.116.6:4975 , piece_id=T3VWH5OBQ7FE66GCKHZLEDSIXFVYEU3V52HI4CPD27QL3RH75K7Q, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:46.826703981 (301h14m32.288796454s) [noise=true , address=77.90.31.10:4518 , piece_id=6FGNAMVK6OAGJTZJNKTUJWGRW7MLGMJMGJKMAINWMPW7BV74OO6A, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:48.069759484 (301h14m33.531851968s) [noise=true , address=45.91.116.13:4154 , piece_id=2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:54.791582362 (301h14m40.253674835s) [noise=true , address=45.91.116.11:4118 , piece_id=VUZB3QRFST2T3HJHMEYYZ6CV7CKJTFWQGNOBDTXHGET2MBUIC4HQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:56.329163998 (301h14m41.791256481s) [noise=true , address=45.91.116.11:4117 , piece_id=QKEYHSK5JOL65H5T2KUEAEK6VKTNCZAYXIEZTMB76PV7P5PEVILQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:56.329201379 (301h14m41.791293862s) [seg_pos={0 0}, upload=265104]: Segment reached optimal threshold of 13 pieces. Oct 6 02:45:56.329362146 (301h14m41.791454629s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Oct 6 02:45:56.329371974 (301h14m41.791464457s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329546396 (301h14m41.791638869s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: write tcp 188.166.225.170:57604->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 02:45:56.329556665 (301h14m41.791649148s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329567937 (301h14m41.791660421s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: rpc: dial tcp 77.90.31.19:4841: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 02:45:56.329579870 (301h14m41.791672353s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329599808 (301h14m41.791692291s) [seg_pos={0 0}, upload=265104]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 6 02:45:56.329683116 (301h14m41.7917756s) [upload=265104]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 6 02:45:56.329690771 (301h14m41.791783254s) [upload=265104]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 6 02:45:56.636341641 (301h14m42.098434134s) [upload=265104]: Done uploading segments... Oct 6 02:45:56.636368131 (301h14m42.098460614s) [upload=265104]: Upload finished. err: <nil> Duration: 5.982372553s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................X.....................X.................] duration: 5.863454111s FAIL (2025-10-06 02:59:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: 2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: 7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA) context deadline exceeded ======LOGS====== Oct 6 02:45:41.756337907 (301h14m27.21843039s) [upload=265104]: Starting upload Oct 6 02:45:41.756430433 (301h14m27.218522916s) [upload=265104]: Done starting upload Oct 6 02:45:41.756513701 (301h14m27.218606185s) [upload=265104]: Uploading segments... Oct 6 02:45:41.756752276 (301h14m27.218844759s) [upload=265104]: Got next segment. Inline: false Oct 6 02:45:41.756794536 (301h14m27.218887019s) [upload=265104]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 02:45:41.756800898 (301h14m27.218893381s) [upload=265104]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 02:45:42.009327267 (301h14m27.47141975s) [seg_pos={0 0}, upload=265104]: Begin upload segment... Oct 6 02:45:42.009417419 (301h14m27.471509892s) [seg_pos={0 0}, upload=265104]: Done begin upload segment. Oct 6 02:45:42.009444500 (301h14m27.471536983s) [noise=true , address=77.90.31.14:4662 , piece_id=MPVBPBB63JILLOUF3N7SRLCZ2F74LX7Z4AH7ZFG3BDNBHT4UIXTQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.009561473 (301h14m27.471653956s) [noise=true , address=145.79.227.19:4077 , piece_id=EJIAPZSBMAO77E33V6MEBOQHUXVVYKMAVP53QAZXZSYBYDFP7MMQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.010197574 (301h14m27.472290057s) [noise=true , address=145.79.227.15:4875 , piece_id=PBDTNZ2NWRZZC3VCIQCVR2WLT2AP6662I2SSDNP6BLUDKCJMRK6Q, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.010634216 (301h14m27.472726699s) [noise=true , address=145.79.227.4:4185 , piece_id=3AD745KFD67M4XFNDXMJVEU34RRBQXXE4P3HVJJRDBHK3IIXNYTA, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011073742 (301h14m27.473166225s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011505996 (301h14m27.473598479s) [noise=true , address=45.91.116.10:4081 , piece_id=FTXZ2LOHQMY74KOX6XZC3TLHWXOGDTIY3SZLSPTE7THGKNLMPXRQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011982502 (301h14m27.474074985s) [noise=true , address=45.91.116.6:4974 , piece_id=7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.012488236 (301h14m27.474580719s) [noise=true , address=145.79.227.7:4294 , piece_id=7E3B4BGLCTW6C5DM6PPDJPWGVJS5HEGYPAPXRAMX7XDSG7USPQFQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.013242472 (301h14m27.475334955s) [noise=true , address=45.91.116.13:4154 , piece_id=2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.014529382 (301h14m27.476621855s) [noise=true , address=45.91.116.6:4975 , piece_id=T3VWH5OBQ7FE66GCKHZLEDSIXFVYEU3V52HI4CPD27QL3RH75K7Q, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.014632619 (301h14m27.476725102s) [noise=true , address=45.91.116.11:4117 , piece_id=QKEYHSK5JOL65H5T2KUEAEK6VKTNCZAYXIEZTMB76PV7P5PEVILQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.015990214 (301h14m27.478082697s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.016291928 (301h14m27.478384412s) [noise=true , address=45.91.116.12:4226 , piece_id=NM7SEIKZYUHZLNVSOGYVZWMI7U7X5SOXOJDAEGY5NF3M6QQFPZSQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.017455915 (301h14m27.479548399s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.018864467 (301h14m27.48095694s) [noise=true , address=77.90.31.10:4518 , piece_id=6FGNAMVK6OAGJTZJNKTUJWGRW7MLGMJMGJKMAINWMPW7BV74OO6A, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.020222022 (301h14m27.482314495s) [noise=true , address=45.91.116.11:4118 , piece_id=VUZB3QRFST2T3HJHMEYYZ6CV7CKJTFWQGNOBDTXHGET2MBUIC4HQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:43.755608897 (301h14m29.21770138s) [upload=265104]: Next returned nil segment Oct 6 02:45:43.755631300 (301h14m29.217723773s) [upload=265104]: Waiting for error group managing segments... Oct 6 02:45:44.043930124 (301h14m29.506022597s) [noise=true , address=145.79.227.19:4077 , piece_id=EJIAPZSBMAO77E33V6MEBOQHUXVVYKMAVP53QAZXZSYBYDFP7MMQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.045501296 (301h14m29.507593769s) [noise=true , address=145.79.227.4:4185 , piece_id=3AD745KFD67M4XFNDXMJVEU34RRBQXXE4P3HVJJRDBHK3IIXNYTA, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.057881199 (301h14m29.519973672s) [noise=true , address=145.79.227.15:4875 , piece_id=PBDTNZ2NWRZZC3VCIQCVR2WLT2AP6662I2SSDNP6BLUDKCJMRK6Q, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.068726540 (301h14m29.530819023s) [noise=true , address=145.79.227.7:4294 , piece_id=7E3B4BGLCTW6C5DM6PPDJPWGVJS5HEGYPAPXRAMX7XDSG7USPQFQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.897895712 (301h14m30.359988185s) [noise=true , address=45.91.116.12:4226 , piece_id=NM7SEIKZYUHZLNVSOGYVZWMI7U7X5SOXOJDAEGY5NF3M6QQFPZSQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.252859665 (301h14m30.714952148s) [noise=true , address=77.90.31.14:4662 , piece_id=MPVBPBB63JILLOUF3N7SRLCZ2F74LX7Z4AH7ZFG3BDNBHT4UIXTQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.318952216 (301h14m30.781044699s) [noise=true , address=45.91.116.6:4974 , piece_id=7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.836439779 (301h14m31.298532262s) [noise=true , address=45.91.116.10:4081 , piece_id=FTXZ2LOHQMY74KOX6XZC3TLHWXOGDTIY3SZLSPTE7THGKNLMPXRQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.975895173 (301h14m31.437987656s) [noise=true , address=45.91.116.6:4975 , piece_id=T3VWH5OBQ7FE66GCKHZLEDSIXFVYEU3V52HI4CPD27QL3RH75K7Q, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:46.826703981 (301h14m32.288796454s) [noise=true , address=77.90.31.10:4518 , piece_id=6FGNAMVK6OAGJTZJNKTUJWGRW7MLGMJMGJKMAINWMPW7BV74OO6A, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:48.069759484 (301h14m33.531851968s) [noise=true , address=45.91.116.13:4154 , piece_id=2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:54.791582362 (301h14m40.253674835s) [noise=true , address=45.91.116.11:4118 , piece_id=VUZB3QRFST2T3HJHMEYYZ6CV7CKJTFWQGNOBDTXHGET2MBUIC4HQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:56.329163998 (301h14m41.791256481s) [noise=true , address=45.91.116.11:4117 , piece_id=QKEYHSK5JOL65H5T2KUEAEK6VKTNCZAYXIEZTMB76PV7P5PEVILQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:56.329201379 (301h14m41.791293862s) [seg_pos={0 0}, upload=265104]: Segment reached optimal threshold of 13 pieces. Oct 6 02:45:56.329362146 (301h14m41.791454629s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Oct 6 02:45:56.329371974 (301h14m41.791464457s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329546396 (301h14m41.791638869s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: write tcp 188.166.225.170:57604->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 02:45:56.329556665 (301h14m41.791649148s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329567937 (301h14m41.791660421s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: rpc: dial tcp 77.90.31.19:4841: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 02:45:56.329579870 (301h14m41.791672353s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329599808 (301h14m41.791692291s) [seg_pos={0 0}, upload=265104]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 6 02:45:56.329683116 (301h14m41.7917756s) [upload=265104]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 6 02:45:56.329690771 (301h14m41.791783254s) [upload=265104]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 6 02:45:56.636341641 (301h14m42.098434134s) [upload=265104]: Done uploading segments... Oct 6 02:45:56.636368131 (301h14m42.098460614s) [upload=265104]: Upload finished. err: <nil> Duration: 5.863485912s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................X.....................X................] duration: 6.020804428s FAIL (2025-10-06 02:58:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: 2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: 7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA) context deadline exceeded ======LOGS====== Oct 6 02:45:41.756337907 (301h14m27.21843039s) [upload=265104]: Starting upload Oct 6 02:45:41.756430433 (301h14m27.218522916s) [upload=265104]: Done starting upload Oct 6 02:45:41.756513701 (301h14m27.218606185s) [upload=265104]: Uploading segments... Oct 6 02:45:41.756752276 (301h14m27.218844759s) [upload=265104]: Got next segment. Inline: false Oct 6 02:45:41.756794536 (301h14m27.218887019s) [upload=265104]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 02:45:41.756800898 (301h14m27.218893381s) [upload=265104]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 02:45:42.009327267 (301h14m27.47141975s) [seg_pos={0 0}, upload=265104]: Begin upload segment... Oct 6 02:45:42.009417419 (301h14m27.471509892s) [seg_pos={0 0}, upload=265104]: Done begin upload segment. Oct 6 02:45:42.009444500 (301h14m27.471536983s) [noise=true , address=77.90.31.14:4662 , piece_id=MPVBPBB63JILLOUF3N7SRLCZ2F74LX7Z4AH7ZFG3BDNBHT4UIXTQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.009561473 (301h14m27.471653956s) [noise=true , address=145.79.227.19:4077 , piece_id=EJIAPZSBMAO77E33V6MEBOQHUXVVYKMAVP53QAZXZSYBYDFP7MMQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.010197574 (301h14m27.472290057s) [noise=true , address=145.79.227.15:4875 , piece_id=PBDTNZ2NWRZZC3VCIQCVR2WLT2AP6662I2SSDNP6BLUDKCJMRK6Q, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.010634216 (301h14m27.472726699s) [noise=true , address=145.79.227.4:4185 , piece_id=3AD745KFD67M4XFNDXMJVEU34RRBQXXE4P3HVJJRDBHK3IIXNYTA, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011073742 (301h14m27.473166225s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011505996 (301h14m27.473598479s) [noise=true , address=45.91.116.10:4081 , piece_id=FTXZ2LOHQMY74KOX6XZC3TLHWXOGDTIY3SZLSPTE7THGKNLMPXRQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011982502 (301h14m27.474074985s) [noise=true , address=45.91.116.6:4974 , piece_id=7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.012488236 (301h14m27.474580719s) [noise=true , address=145.79.227.7:4294 , piece_id=7E3B4BGLCTW6C5DM6PPDJPWGVJS5HEGYPAPXRAMX7XDSG7USPQFQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.013242472 (301h14m27.475334955s) [noise=true , address=45.91.116.13:4154 , piece_id=2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.014529382 (301h14m27.476621855s) [noise=true , address=45.91.116.6:4975 , piece_id=T3VWH5OBQ7FE66GCKHZLEDSIXFVYEU3V52HI4CPD27QL3RH75K7Q, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.014632619 (301h14m27.476725102s) [noise=true , address=45.91.116.11:4117 , piece_id=QKEYHSK5JOL65H5T2KUEAEK6VKTNCZAYXIEZTMB76PV7P5PEVILQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.015990214 (301h14m27.478082697s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.016291928 (301h14m27.478384412s) [noise=true , address=45.91.116.12:4226 , piece_id=NM7SEIKZYUHZLNVSOGYVZWMI7U7X5SOXOJDAEGY5NF3M6QQFPZSQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.017455915 (301h14m27.479548399s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.018864467 (301h14m27.48095694s) [noise=true , address=77.90.31.10:4518 , piece_id=6FGNAMVK6OAGJTZJNKTUJWGRW7MLGMJMGJKMAINWMPW7BV74OO6A, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.020222022 (301h14m27.482314495s) [noise=true , address=45.91.116.11:4118 , piece_id=VUZB3QRFST2T3HJHMEYYZ6CV7CKJTFWQGNOBDTXHGET2MBUIC4HQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:43.755608897 (301h14m29.21770138s) [upload=265104]: Next returned nil segment Oct 6 02:45:43.755631300 (301h14m29.217723773s) [upload=265104]: Waiting for error group managing segments... Oct 6 02:45:44.043930124 (301h14m29.506022597s) [noise=true , address=145.79.227.19:4077 , piece_id=EJIAPZSBMAO77E33V6MEBOQHUXVVYKMAVP53QAZXZSYBYDFP7MMQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.045501296 (301h14m29.507593769s) [noise=true , address=145.79.227.4:4185 , piece_id=3AD745KFD67M4XFNDXMJVEU34RRBQXXE4P3HVJJRDBHK3IIXNYTA, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.057881199 (301h14m29.519973672s) [noise=true , address=145.79.227.15:4875 , piece_id=PBDTNZ2NWRZZC3VCIQCVR2WLT2AP6662I2SSDNP6BLUDKCJMRK6Q, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.068726540 (301h14m29.530819023s) [noise=true , address=145.79.227.7:4294 , piece_id=7E3B4BGLCTW6C5DM6PPDJPWGVJS5HEGYPAPXRAMX7XDSG7USPQFQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.897895712 (301h14m30.359988185s) [noise=true , address=45.91.116.12:4226 , piece_id=NM7SEIKZYUHZLNVSOGYVZWMI7U7X5SOXOJDAEGY5NF3M6QQFPZSQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.252859665 (301h14m30.714952148s) [noise=true , address=77.90.31.14:4662 , piece_id=MPVBPBB63JILLOUF3N7SRLCZ2F74LX7Z4AH7ZFG3BDNBHT4UIXTQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.318952216 (301h14m30.781044699s) [noise=true , address=45.91.116.6:4974 , piece_id=7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.836439779 (301h14m31.298532262s) [noise=true , address=45.91.116.10:4081 , piece_id=FTXZ2LOHQMY74KOX6XZC3TLHWXOGDTIY3SZLSPTE7THGKNLMPXRQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.975895173 (301h14m31.437987656s) [noise=true , address=45.91.116.6:4975 , piece_id=T3VWH5OBQ7FE66GCKHZLEDSIXFVYEU3V52HI4CPD27QL3RH75K7Q, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:46.826703981 (301h14m32.288796454s) [noise=true , address=77.90.31.10:4518 , piece_id=6FGNAMVK6OAGJTZJNKTUJWGRW7MLGMJMGJKMAINWMPW7BV74OO6A, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:48.069759484 (301h14m33.531851968s) [noise=true , address=45.91.116.13:4154 , piece_id=2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:54.791582362 (301h14m40.253674835s) [noise=true , address=45.91.116.11:4118 , piece_id=VUZB3QRFST2T3HJHMEYYZ6CV7CKJTFWQGNOBDTXHGET2MBUIC4HQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:56.329163998 (301h14m41.791256481s) [noise=true , address=45.91.116.11:4117 , piece_id=QKEYHSK5JOL65H5T2KUEAEK6VKTNCZAYXIEZTMB76PV7P5PEVILQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:56.329201379 (301h14m41.791293862s) [seg_pos={0 0}, upload=265104]: Segment reached optimal threshold of 13 pieces. Oct 6 02:45:56.329362146 (301h14m41.791454629s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Oct 6 02:45:56.329371974 (301h14m41.791464457s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329546396 (301h14m41.791638869s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: write tcp 188.166.225.170:57604->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 02:45:56.329556665 (301h14m41.791649148s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329567937 (301h14m41.791660421s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: rpc: dial tcp 77.90.31.19:4841: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 02:45:56.329579870 (301h14m41.791672353s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329599808 (301h14m41.791692291s) [seg_pos={0 0}, upload=265104]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 6 02:45:56.329683116 (301h14m41.7917756s) [upload=265104]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 6 02:45:56.329690771 (301h14m41.791783254s) [upload=265104]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 6 02:45:56.636341641 (301h14m42.098434134s) [upload=265104]: Done uploading segments... Oct 6 02:45:56.636368131 (301h14m42.098460614s) [upload=265104]: Upload finished. err: <nil> Duration: 6.020850515s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................X.....................X...............] duration: 6.00653892s FAIL (2025-10-06 02:57:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: 2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: 7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA) context deadline exceeded ======LOGS====== Oct 6 02:45:41.756337907 (301h14m27.21843039s) [upload=265104]: Starting upload Oct 6 02:45:41.756430433 (301h14m27.218522916s) [upload=265104]: Done starting upload Oct 6 02:45:41.756513701 (301h14m27.218606185s) [upload=265104]: Uploading segments... Oct 6 02:45:41.756752276 (301h14m27.218844759s) [upload=265104]: Got next segment. Inline: false Oct 6 02:45:41.756794536 (301h14m27.218887019s) [upload=265104]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 02:45:41.756800898 (301h14m27.218893381s) [upload=265104]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 02:45:42.009327267 (301h14m27.47141975s) [seg_pos={0 0}, upload=265104]: Begin upload segment... Oct 6 02:45:42.009417419 (301h14m27.471509892s) [seg_pos={0 0}, upload=265104]: Done begin upload segment. Oct 6 02:45:42.009444500 (301h14m27.471536983s) [noise=true , address=77.90.31.14:4662 , piece_id=MPVBPBB63JILLOUF3N7SRLCZ2F74LX7Z4AH7ZFG3BDNBHT4UIXTQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.009561473 (301h14m27.471653956s) [noise=true , address=145.79.227.19:4077 , piece_id=EJIAPZSBMAO77E33V6MEBOQHUXVVYKMAVP53QAZXZSYBYDFP7MMQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.010197574 (301h14m27.472290057s) [noise=true , address=145.79.227.15:4875 , piece_id=PBDTNZ2NWRZZC3VCIQCVR2WLT2AP6662I2SSDNP6BLUDKCJMRK6Q, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.010634216 (301h14m27.472726699s) [noise=true , address=145.79.227.4:4185 , piece_id=3AD745KFD67M4XFNDXMJVEU34RRBQXXE4P3HVJJRDBHK3IIXNYTA, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011073742 (301h14m27.473166225s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011505996 (301h14m27.473598479s) [noise=true , address=45.91.116.10:4081 , piece_id=FTXZ2LOHQMY74KOX6XZC3TLHWXOGDTIY3SZLSPTE7THGKNLMPXRQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011982502 (301h14m27.474074985s) [noise=true , address=45.91.116.6:4974 , piece_id=7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.012488236 (301h14m27.474580719s) [noise=true , address=145.79.227.7:4294 , piece_id=7E3B4BGLCTW6C5DM6PPDJPWGVJS5HEGYPAPXRAMX7XDSG7USPQFQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.013242472 (301h14m27.475334955s) [noise=true , address=45.91.116.13:4154 , piece_id=2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.014529382 (301h14m27.476621855s) [noise=true , address=45.91.116.6:4975 , piece_id=T3VWH5OBQ7FE66GCKHZLEDSIXFVYEU3V52HI4CPD27QL3RH75K7Q, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.014632619 (301h14m27.476725102s) [noise=true , address=45.91.116.11:4117 , piece_id=QKEYHSK5JOL65H5T2KUEAEK6VKTNCZAYXIEZTMB76PV7P5PEVILQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.015990214 (301h14m27.478082697s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.016291928 (301h14m27.478384412s) [noise=true , address=45.91.116.12:4226 , piece_id=NM7SEIKZYUHZLNVSOGYVZWMI7U7X5SOXOJDAEGY5NF3M6QQFPZSQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.017455915 (301h14m27.479548399s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.018864467 (301h14m27.48095694s) [noise=true , address=77.90.31.10:4518 , piece_id=6FGNAMVK6OAGJTZJNKTUJWGRW7MLGMJMGJKMAINWMPW7BV74OO6A, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.020222022 (301h14m27.482314495s) [noise=true , address=45.91.116.11:4118 , piece_id=VUZB3QRFST2T3HJHMEYYZ6CV7CKJTFWQGNOBDTXHGET2MBUIC4HQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:43.755608897 (301h14m29.21770138s) [upload=265104]: Next returned nil segment Oct 6 02:45:43.755631300 (301h14m29.217723773s) [upload=265104]: Waiting for error group managing segments... Oct 6 02:45:44.043930124 (301h14m29.506022597s) [noise=true , address=145.79.227.19:4077 , piece_id=EJIAPZSBMAO77E33V6MEBOQHUXVVYKMAVP53QAZXZSYBYDFP7MMQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.045501296 (301h14m29.507593769s) [noise=true , address=145.79.227.4:4185 , piece_id=3AD745KFD67M4XFNDXMJVEU34RRBQXXE4P3HVJJRDBHK3IIXNYTA, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.057881199 (301h14m29.519973672s) [noise=true , address=145.79.227.15:4875 , piece_id=PBDTNZ2NWRZZC3VCIQCVR2WLT2AP6662I2SSDNP6BLUDKCJMRK6Q, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.068726540 (301h14m29.530819023s) [noise=true , address=145.79.227.7:4294 , piece_id=7E3B4BGLCTW6C5DM6PPDJPWGVJS5HEGYPAPXRAMX7XDSG7USPQFQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.897895712 (301h14m30.359988185s) [noise=true , address=45.91.116.12:4226 , piece_id=NM7SEIKZYUHZLNVSOGYVZWMI7U7X5SOXOJDAEGY5NF3M6QQFPZSQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.252859665 (301h14m30.714952148s) [noise=true , address=77.90.31.14:4662 , piece_id=MPVBPBB63JILLOUF3N7SRLCZ2F74LX7Z4AH7ZFG3BDNBHT4UIXTQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.318952216 (301h14m30.781044699s) [noise=true , address=45.91.116.6:4974 , piece_id=7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.836439779 (301h14m31.298532262s) [noise=true , address=45.91.116.10:4081 , piece_id=FTXZ2LOHQMY74KOX6XZC3TLHWXOGDTIY3SZLSPTE7THGKNLMPXRQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.975895173 (301h14m31.437987656s) [noise=true , address=45.91.116.6:4975 , piece_id=T3VWH5OBQ7FE66GCKHZLEDSIXFVYEU3V52HI4CPD27QL3RH75K7Q, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:46.826703981 (301h14m32.288796454s) [noise=true , address=77.90.31.10:4518 , piece_id=6FGNAMVK6OAGJTZJNKTUJWGRW7MLGMJMGJKMAINWMPW7BV74OO6A, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:48.069759484 (301h14m33.531851968s) [noise=true , address=45.91.116.13:4154 , piece_id=2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:54.791582362 (301h14m40.253674835s) [noise=true , address=45.91.116.11:4118 , piece_id=VUZB3QRFST2T3HJHMEYYZ6CV7CKJTFWQGNOBDTXHGET2MBUIC4HQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:56.329163998 (301h14m41.791256481s) [noise=true , address=45.91.116.11:4117 , piece_id=QKEYHSK5JOL65H5T2KUEAEK6VKTNCZAYXIEZTMB76PV7P5PEVILQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:56.329201379 (301h14m41.791293862s) [seg_pos={0 0}, upload=265104]: Segment reached optimal threshold of 13 pieces. Oct 6 02:45:56.329362146 (301h14m41.791454629s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Oct 6 02:45:56.329371974 (301h14m41.791464457s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329546396 (301h14m41.791638869s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: write tcp 188.166.225.170:57604->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 02:45:56.329556665 (301h14m41.791649148s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329567937 (301h14m41.791660421s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: rpc: dial tcp 77.90.31.19:4841: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 02:45:56.329579870 (301h14m41.791672353s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329599808 (301h14m41.791692291s) [seg_pos={0 0}, upload=265104]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 6 02:45:56.329683116 (301h14m41.7917756s) [upload=265104]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 6 02:45:56.329690771 (301h14m41.791783254s) [upload=265104]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 6 02:45:56.636341641 (301h14m42.098434134s) [upload=265104]: Done uploading segments... Oct 6 02:45:56.636368131 (301h14m42.098460614s) [upload=265104]: Upload finished. err: <nil> Duration: 6.00659658s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................X.....................X..............] duration: 6.229766296s FAIL (2025-10-06 02:56:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: 2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: 7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA) context deadline exceeded ======LOGS====== Oct 6 02:45:41.756337907 (301h14m27.21843039s) [upload=265104]: Starting upload Oct 6 02:45:41.756430433 (301h14m27.218522916s) [upload=265104]: Done starting upload Oct 6 02:45:41.756513701 (301h14m27.218606185s) [upload=265104]: Uploading segments... Oct 6 02:45:41.756752276 (301h14m27.218844759s) [upload=265104]: Got next segment. Inline: false Oct 6 02:45:41.756794536 (301h14m27.218887019s) [upload=265104]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 02:45:41.756800898 (301h14m27.218893381s) [upload=265104]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 02:45:42.009327267 (301h14m27.47141975s) [seg_pos={0 0}, upload=265104]: Begin upload segment... Oct 6 02:45:42.009417419 (301h14m27.471509892s) [seg_pos={0 0}, upload=265104]: Done begin upload segment. Oct 6 02:45:42.009444500 (301h14m27.471536983s) [noise=true , address=77.90.31.14:4662 , piece_id=MPVBPBB63JILLOUF3N7SRLCZ2F74LX7Z4AH7ZFG3BDNBHT4UIXTQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.009561473 (301h14m27.471653956s) [noise=true , address=145.79.227.19:4077 , piece_id=EJIAPZSBMAO77E33V6MEBOQHUXVVYKMAVP53QAZXZSYBYDFP7MMQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.010197574 (301h14m27.472290057s) [noise=true , address=145.79.227.15:4875 , piece_id=PBDTNZ2NWRZZC3VCIQCVR2WLT2AP6662I2SSDNP6BLUDKCJMRK6Q, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.010634216 (301h14m27.472726699s) [noise=true , address=145.79.227.4:4185 , piece_id=3AD745KFD67M4XFNDXMJVEU34RRBQXXE4P3HVJJRDBHK3IIXNYTA, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011073742 (301h14m27.473166225s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011505996 (301h14m27.473598479s) [noise=true , address=45.91.116.10:4081 , piece_id=FTXZ2LOHQMY74KOX6XZC3TLHWXOGDTIY3SZLSPTE7THGKNLMPXRQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011982502 (301h14m27.474074985s) [noise=true , address=45.91.116.6:4974 , piece_id=7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.012488236 (301h14m27.474580719s) [noise=true , address=145.79.227.7:4294 , piece_id=7E3B4BGLCTW6C5DM6PPDJPWGVJS5HEGYPAPXRAMX7XDSG7USPQFQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.013242472 (301h14m27.475334955s) [noise=true , address=45.91.116.13:4154 , piece_id=2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.014529382 (301h14m27.476621855s) [noise=true , address=45.91.116.6:4975 , piece_id=T3VWH5OBQ7FE66GCKHZLEDSIXFVYEU3V52HI4CPD27QL3RH75K7Q, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.014632619 (301h14m27.476725102s) [noise=true , address=45.91.116.11:4117 , piece_id=QKEYHSK5JOL65H5T2KUEAEK6VKTNCZAYXIEZTMB76PV7P5PEVILQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.015990214 (301h14m27.478082697s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.016291928 (301h14m27.478384412s) [noise=true , address=45.91.116.12:4226 , piece_id=NM7SEIKZYUHZLNVSOGYVZWMI7U7X5SOXOJDAEGY5NF3M6QQFPZSQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.017455915 (301h14m27.479548399s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.018864467 (301h14m27.48095694s) [noise=true , address=77.90.31.10:4518 , piece_id=6FGNAMVK6OAGJTZJNKTUJWGRW7MLGMJMGJKMAINWMPW7BV74OO6A, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.020222022 (301h14m27.482314495s) [noise=true , address=45.91.116.11:4118 , piece_id=VUZB3QRFST2T3HJHMEYYZ6CV7CKJTFWQGNOBDTXHGET2MBUIC4HQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:43.755608897 (301h14m29.21770138s) [upload=265104]: Next returned nil segment Oct 6 02:45:43.755631300 (301h14m29.217723773s) [upload=265104]: Waiting for error group managing segments... Oct 6 02:45:44.043930124 (301h14m29.506022597s) [noise=true , address=145.79.227.19:4077 , piece_id=EJIAPZSBMAO77E33V6MEBOQHUXVVYKMAVP53QAZXZSYBYDFP7MMQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.045501296 (301h14m29.507593769s) [noise=true , address=145.79.227.4:4185 , piece_id=3AD745KFD67M4XFNDXMJVEU34RRBQXXE4P3HVJJRDBHK3IIXNYTA, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.057881199 (301h14m29.519973672s) [noise=true , address=145.79.227.15:4875 , piece_id=PBDTNZ2NWRZZC3VCIQCVR2WLT2AP6662I2SSDNP6BLUDKCJMRK6Q, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.068726540 (301h14m29.530819023s) [noise=true , address=145.79.227.7:4294 , piece_id=7E3B4BGLCTW6C5DM6PPDJPWGVJS5HEGYPAPXRAMX7XDSG7USPQFQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.897895712 (301h14m30.359988185s) [noise=true , address=45.91.116.12:4226 , piece_id=NM7SEIKZYUHZLNVSOGYVZWMI7U7X5SOXOJDAEGY5NF3M6QQFPZSQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.252859665 (301h14m30.714952148s) [noise=true , address=77.90.31.14:4662 , piece_id=MPVBPBB63JILLOUF3N7SRLCZ2F74LX7Z4AH7ZFG3BDNBHT4UIXTQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.318952216 (301h14m30.781044699s) [noise=true , address=45.91.116.6:4974 , piece_id=7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.836439779 (301h14m31.298532262s) [noise=true , address=45.91.116.10:4081 , piece_id=FTXZ2LOHQMY74KOX6XZC3TLHWXOGDTIY3SZLSPTE7THGKNLMPXRQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.975895173 (301h14m31.437987656s) [noise=true , address=45.91.116.6:4975 , piece_id=T3VWH5OBQ7FE66GCKHZLEDSIXFVYEU3V52HI4CPD27QL3RH75K7Q, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:46.826703981 (301h14m32.288796454s) [noise=true , address=77.90.31.10:4518 , piece_id=6FGNAMVK6OAGJTZJNKTUJWGRW7MLGMJMGJKMAINWMPW7BV74OO6A, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:48.069759484 (301h14m33.531851968s) [noise=true , address=45.91.116.13:4154 , piece_id=2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:54.791582362 (301h14m40.253674835s) [noise=true , address=45.91.116.11:4118 , piece_id=VUZB3QRFST2T3HJHMEYYZ6CV7CKJTFWQGNOBDTXHGET2MBUIC4HQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:56.329163998 (301h14m41.791256481s) [noise=true , address=45.91.116.11:4117 , piece_id=QKEYHSK5JOL65H5T2KUEAEK6VKTNCZAYXIEZTMB76PV7P5PEVILQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:56.329201379 (301h14m41.791293862s) [seg_pos={0 0}, upload=265104]: Segment reached optimal threshold of 13 pieces. Oct 6 02:45:56.329362146 (301h14m41.791454629s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Oct 6 02:45:56.329371974 (301h14m41.791464457s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329546396 (301h14m41.791638869s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: write tcp 188.166.225.170:57604->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 02:45:56.329556665 (301h14m41.791649148s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329567937 (301h14m41.791660421s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: rpc: dial tcp 77.90.31.19:4841: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 02:45:56.329579870 (301h14m41.791672353s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329599808 (301h14m41.791692291s) [seg_pos={0 0}, upload=265104]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 6 02:45:56.329683116 (301h14m41.7917756s) [upload=265104]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 6 02:45:56.329690771 (301h14m41.791783254s) [upload=265104]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 6 02:45:56.636341641 (301h14m42.098434134s) [upload=265104]: Done uploading segments... Oct 6 02:45:56.636368131 (301h14m42.098460614s) [upload=265104]: Upload finished. err: <nil> Duration: 6.229807004s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................X.....................X.............] duration: 6.048483567s FAIL (2025-10-06 02:56:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: 2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: 7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA) context deadline exceeded ======LOGS====== Oct 6 02:45:41.756337907 (301h14m27.21843039s) [upload=265104]: Starting upload Oct 6 02:45:41.756430433 (301h14m27.218522916s) [upload=265104]: Done starting upload Oct 6 02:45:41.756513701 (301h14m27.218606185s) [upload=265104]: Uploading segments... Oct 6 02:45:41.756752276 (301h14m27.218844759s) [upload=265104]: Got next segment. Inline: false Oct 6 02:45:41.756794536 (301h14m27.218887019s) [upload=265104]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 02:45:41.756800898 (301h14m27.218893381s) [upload=265104]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 02:45:42.009327267 (301h14m27.47141975s) [seg_pos={0 0}, upload=265104]: Begin upload segment... Oct 6 02:45:42.009417419 (301h14m27.471509892s) [seg_pos={0 0}, upload=265104]: Done begin upload segment. Oct 6 02:45:42.009444500 (301h14m27.471536983s) [noise=true , address=77.90.31.14:4662 , piece_id=MPVBPBB63JILLOUF3N7SRLCZ2F74LX7Z4AH7ZFG3BDNBHT4UIXTQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.009561473 (301h14m27.471653956s) [noise=true , address=145.79.227.19:4077 , piece_id=EJIAPZSBMAO77E33V6MEBOQHUXVVYKMAVP53QAZXZSYBYDFP7MMQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.010197574 (301h14m27.472290057s) [noise=true , address=145.79.227.15:4875 , piece_id=PBDTNZ2NWRZZC3VCIQCVR2WLT2AP6662I2SSDNP6BLUDKCJMRK6Q, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.010634216 (301h14m27.472726699s) [noise=true , address=145.79.227.4:4185 , piece_id=3AD745KFD67M4XFNDXMJVEU34RRBQXXE4P3HVJJRDBHK3IIXNYTA, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011073742 (301h14m27.473166225s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011505996 (301h14m27.473598479s) [noise=true , address=45.91.116.10:4081 , piece_id=FTXZ2LOHQMY74KOX6XZC3TLHWXOGDTIY3SZLSPTE7THGKNLMPXRQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011982502 (301h14m27.474074985s) [noise=true , address=45.91.116.6:4974 , piece_id=7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.012488236 (301h14m27.474580719s) [noise=true , address=145.79.227.7:4294 , piece_id=7E3B4BGLCTW6C5DM6PPDJPWGVJS5HEGYPAPXRAMX7XDSG7USPQFQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.013242472 (301h14m27.475334955s) [noise=true , address=45.91.116.13:4154 , piece_id=2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.014529382 (301h14m27.476621855s) [noise=true , address=45.91.116.6:4975 , piece_id=T3VWH5OBQ7FE66GCKHZLEDSIXFVYEU3V52HI4CPD27QL3RH75K7Q, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.014632619 (301h14m27.476725102s) [noise=true , address=45.91.116.11:4117 , piece_id=QKEYHSK5JOL65H5T2KUEAEK6VKTNCZAYXIEZTMB76PV7P5PEVILQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.015990214 (301h14m27.478082697s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.016291928 (301h14m27.478384412s) [noise=true , address=45.91.116.12:4226 , piece_id=NM7SEIKZYUHZLNVSOGYVZWMI7U7X5SOXOJDAEGY5NF3M6QQFPZSQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.017455915 (301h14m27.479548399s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.018864467 (301h14m27.48095694s) [noise=true , address=77.90.31.10:4518 , piece_id=6FGNAMVK6OAGJTZJNKTUJWGRW7MLGMJMGJKMAINWMPW7BV74OO6A, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.020222022 (301h14m27.482314495s) [noise=true , address=45.91.116.11:4118 , piece_id=VUZB3QRFST2T3HJHMEYYZ6CV7CKJTFWQGNOBDTXHGET2MBUIC4HQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:43.755608897 (301h14m29.21770138s) [upload=265104]: Next returned nil segment Oct 6 02:45:43.755631300 (301h14m29.217723773s) [upload=265104]: Waiting for error group managing segments... Oct 6 02:45:44.043930124 (301h14m29.506022597s) [noise=true , address=145.79.227.19:4077 , piece_id=EJIAPZSBMAO77E33V6MEBOQHUXVVYKMAVP53QAZXZSYBYDFP7MMQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.045501296 (301h14m29.507593769s) [noise=true , address=145.79.227.4:4185 , piece_id=3AD745KFD67M4XFNDXMJVEU34RRBQXXE4P3HVJJRDBHK3IIXNYTA, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.057881199 (301h14m29.519973672s) [noise=true , address=145.79.227.15:4875 , piece_id=PBDTNZ2NWRZZC3VCIQCVR2WLT2AP6662I2SSDNP6BLUDKCJMRK6Q, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.068726540 (301h14m29.530819023s) [noise=true , address=145.79.227.7:4294 , piece_id=7E3B4BGLCTW6C5DM6PPDJPWGVJS5HEGYPAPXRAMX7XDSG7USPQFQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.897895712 (301h14m30.359988185s) [noise=true , address=45.91.116.12:4226 , piece_id=NM7SEIKZYUHZLNVSOGYVZWMI7U7X5SOXOJDAEGY5NF3M6QQFPZSQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.252859665 (301h14m30.714952148s) [noise=true , address=77.90.31.14:4662 , piece_id=MPVBPBB63JILLOUF3N7SRLCZ2F74LX7Z4AH7ZFG3BDNBHT4UIXTQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.318952216 (301h14m30.781044699s) [noise=true , address=45.91.116.6:4974 , piece_id=7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.836439779 (301h14m31.298532262s) [noise=true , address=45.91.116.10:4081 , piece_id=FTXZ2LOHQMY74KOX6XZC3TLHWXOGDTIY3SZLSPTE7THGKNLMPXRQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.975895173 (301h14m31.437987656s) [noise=true , address=45.91.116.6:4975 , piece_id=T3VWH5OBQ7FE66GCKHZLEDSIXFVYEU3V52HI4CPD27QL3RH75K7Q, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:46.826703981 (301h14m32.288796454s) [noise=true , address=77.90.31.10:4518 , piece_id=6FGNAMVK6OAGJTZJNKTUJWGRW7MLGMJMGJKMAINWMPW7BV74OO6A, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:48.069759484 (301h14m33.531851968s) [noise=true , address=45.91.116.13:4154 , piece_id=2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:54.791582362 (301h14m40.253674835s) [noise=true , address=45.91.116.11:4118 , piece_id=VUZB3QRFST2T3HJHMEYYZ6CV7CKJTFWQGNOBDTXHGET2MBUIC4HQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:56.329163998 (301h14m41.791256481s) [noise=true , address=45.91.116.11:4117 , piece_id=QKEYHSK5JOL65H5T2KUEAEK6VKTNCZAYXIEZTMB76PV7P5PEVILQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:56.329201379 (301h14m41.791293862s) [seg_pos={0 0}, upload=265104]: Segment reached optimal threshold of 13 pieces. Oct 6 02:45:56.329362146 (301h14m41.791454629s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Oct 6 02:45:56.329371974 (301h14m41.791464457s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329546396 (301h14m41.791638869s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: write tcp 188.166.225.170:57604->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 02:45:56.329556665 (301h14m41.791649148s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329567937 (301h14m41.791660421s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: rpc: dial tcp 77.90.31.19:4841: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 02:45:56.329579870 (301h14m41.791672353s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329599808 (301h14m41.791692291s) [seg_pos={0 0}, upload=265104]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 6 02:45:56.329683116 (301h14m41.7917756s) [upload=265104]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 6 02:45:56.329690771 (301h14m41.791783254s) [upload=265104]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 6 02:45:56.636341641 (301h14m42.098434134s) [upload=265104]: Done uploading segments... Oct 6 02:45:56.636368131 (301h14m42.098460614s) [upload=265104]: Upload finished. err: <nil> Duration: 6.04851177s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................X.....................X............] duration: 6.080662402s FAIL (2025-10-06 02:56:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: 2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: 7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA) context deadline exceeded ======LOGS====== Oct 6 02:45:41.756337907 (301h14m27.21843039s) [upload=265104]: Starting upload Oct 6 02:45:41.756430433 (301h14m27.218522916s) [upload=265104]: Done starting upload Oct 6 02:45:41.756513701 (301h14m27.218606185s) [upload=265104]: Uploading segments... Oct 6 02:45:41.756752276 (301h14m27.218844759s) [upload=265104]: Got next segment. Inline: false Oct 6 02:45:41.756794536 (301h14m27.218887019s) [upload=265104]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 02:45:41.756800898 (301h14m27.218893381s) [upload=265104]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 02:45:42.009327267 (301h14m27.47141975s) [seg_pos={0 0}, upload=265104]: Begin upload segment... Oct 6 02:45:42.009417419 (301h14m27.471509892s) [seg_pos={0 0}, upload=265104]: Done begin upload segment. Oct 6 02:45:42.009444500 (301h14m27.471536983s) [noise=true , address=77.90.31.14:4662 , piece_id=MPVBPBB63JILLOUF3N7SRLCZ2F74LX7Z4AH7ZFG3BDNBHT4UIXTQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.009561473 (301h14m27.471653956s) [noise=true , address=145.79.227.19:4077 , piece_id=EJIAPZSBMAO77E33V6MEBOQHUXVVYKMAVP53QAZXZSYBYDFP7MMQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.010197574 (301h14m27.472290057s) [noise=true , address=145.79.227.15:4875 , piece_id=PBDTNZ2NWRZZC3VCIQCVR2WLT2AP6662I2SSDNP6BLUDKCJMRK6Q, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.010634216 (301h14m27.472726699s) [noise=true , address=145.79.227.4:4185 , piece_id=3AD745KFD67M4XFNDXMJVEU34RRBQXXE4P3HVJJRDBHK3IIXNYTA, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011073742 (301h14m27.473166225s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011505996 (301h14m27.473598479s) [noise=true , address=45.91.116.10:4081 , piece_id=FTXZ2LOHQMY74KOX6XZC3TLHWXOGDTIY3SZLSPTE7THGKNLMPXRQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011982502 (301h14m27.474074985s) [noise=true , address=45.91.116.6:4974 , piece_id=7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.012488236 (301h14m27.474580719s) [noise=true , address=145.79.227.7:4294 , piece_id=7E3B4BGLCTW6C5DM6PPDJPWGVJS5HEGYPAPXRAMX7XDSG7USPQFQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.013242472 (301h14m27.475334955s) [noise=true , address=45.91.116.13:4154 , piece_id=2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.014529382 (301h14m27.476621855s) [noise=true , address=45.91.116.6:4975 , piece_id=T3VWH5OBQ7FE66GCKHZLEDSIXFVYEU3V52HI4CPD27QL3RH75K7Q, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.014632619 (301h14m27.476725102s) [noise=true , address=45.91.116.11:4117 , piece_id=QKEYHSK5JOL65H5T2KUEAEK6VKTNCZAYXIEZTMB76PV7P5PEVILQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.015990214 (301h14m27.478082697s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.016291928 (301h14m27.478384412s) [noise=true , address=45.91.116.12:4226 , piece_id=NM7SEIKZYUHZLNVSOGYVZWMI7U7X5SOXOJDAEGY5NF3M6QQFPZSQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.017455915 (301h14m27.479548399s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.018864467 (301h14m27.48095694s) [noise=true , address=77.90.31.10:4518 , piece_id=6FGNAMVK6OAGJTZJNKTUJWGRW7MLGMJMGJKMAINWMPW7BV74OO6A, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.020222022 (301h14m27.482314495s) [noise=true , address=45.91.116.11:4118 , piece_id=VUZB3QRFST2T3HJHMEYYZ6CV7CKJTFWQGNOBDTXHGET2MBUIC4HQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:43.755608897 (301h14m29.21770138s) [upload=265104]: Next returned nil segment Oct 6 02:45:43.755631300 (301h14m29.217723773s) [upload=265104]: Waiting for error group managing segments... Oct 6 02:45:44.043930124 (301h14m29.506022597s) [noise=true , address=145.79.227.19:4077 , piece_id=EJIAPZSBMAO77E33V6MEBOQHUXVVYKMAVP53QAZXZSYBYDFP7MMQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.045501296 (301h14m29.507593769s) [noise=true , address=145.79.227.4:4185 , piece_id=3AD745KFD67M4XFNDXMJVEU34RRBQXXE4P3HVJJRDBHK3IIXNYTA, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.057881199 (301h14m29.519973672s) [noise=true , address=145.79.227.15:4875 , piece_id=PBDTNZ2NWRZZC3VCIQCVR2WLT2AP6662I2SSDNP6BLUDKCJMRK6Q, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.068726540 (301h14m29.530819023s) [noise=true , address=145.79.227.7:4294 , piece_id=7E3B4BGLCTW6C5DM6PPDJPWGVJS5HEGYPAPXRAMX7XDSG7USPQFQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.897895712 (301h14m30.359988185s) [noise=true , address=45.91.116.12:4226 , piece_id=NM7SEIKZYUHZLNVSOGYVZWMI7U7X5SOXOJDAEGY5NF3M6QQFPZSQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.252859665 (301h14m30.714952148s) [noise=true , address=77.90.31.14:4662 , piece_id=MPVBPBB63JILLOUF3N7SRLCZ2F74LX7Z4AH7ZFG3BDNBHT4UIXTQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.318952216 (301h14m30.781044699s) [noise=true , address=45.91.116.6:4974 , piece_id=7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.836439779 (301h14m31.298532262s) [noise=true , address=45.91.116.10:4081 , piece_id=FTXZ2LOHQMY74KOX6XZC3TLHWXOGDTIY3SZLSPTE7THGKNLMPXRQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.975895173 (301h14m31.437987656s) [noise=true , address=45.91.116.6:4975 , piece_id=T3VWH5OBQ7FE66GCKHZLEDSIXFVYEU3V52HI4CPD27QL3RH75K7Q, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:46.826703981 (301h14m32.288796454s) [noise=true , address=77.90.31.10:4518 , piece_id=6FGNAMVK6OAGJTZJNKTUJWGRW7MLGMJMGJKMAINWMPW7BV74OO6A, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:48.069759484 (301h14m33.531851968s) [noise=true , address=45.91.116.13:4154 , piece_id=2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:54.791582362 (301h14m40.253674835s) [noise=true , address=45.91.116.11:4118 , piece_id=VUZB3QRFST2T3HJHMEYYZ6CV7CKJTFWQGNOBDTXHGET2MBUIC4HQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:56.329163998 (301h14m41.791256481s) [noise=true , address=45.91.116.11:4117 , piece_id=QKEYHSK5JOL65H5T2KUEAEK6VKTNCZAYXIEZTMB76PV7P5PEVILQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:56.329201379 (301h14m41.791293862s) [seg_pos={0 0}, upload=265104]: Segment reached optimal threshold of 13 pieces. Oct 6 02:45:56.329362146 (301h14m41.791454629s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Oct 6 02:45:56.329371974 (301h14m41.791464457s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329546396 (301h14m41.791638869s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: write tcp 188.166.225.170:57604->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 02:45:56.329556665 (301h14m41.791649148s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329567937 (301h14m41.791660421s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: rpc: dial tcp 77.90.31.19:4841: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 02:45:56.329579870 (301h14m41.791672353s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329599808 (301h14m41.791692291s) [seg_pos={0 0}, upload=265104]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 6 02:45:56.329683116 (301h14m41.7917756s) [upload=265104]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 6 02:45:56.329690771 (301h14m41.791783254s) [upload=265104]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 6 02:45:56.636341641 (301h14m42.098434134s) [upload=265104]: Done uploading segments... Oct 6 02:45:56.636368131 (301h14m42.098460614s) [upload=265104]: Upload finished. err: <nil> Duration: 6.080696559s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................X.....................X...........] duration: 6.071597861s FAIL (2025-10-06 02:55:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: 2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: 7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA) context deadline exceeded ======LOGS====== Oct 6 02:45:41.756337907 (301h14m27.21843039s) [upload=265104]: Starting upload Oct 6 02:45:41.756430433 (301h14m27.218522916s) [upload=265104]: Done starting upload Oct 6 02:45:41.756513701 (301h14m27.218606185s) [upload=265104]: Uploading segments... Oct 6 02:45:41.756752276 (301h14m27.218844759s) [upload=265104]: Got next segment. Inline: false Oct 6 02:45:41.756794536 (301h14m27.218887019s) [upload=265104]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 02:45:41.756800898 (301h14m27.218893381s) [upload=265104]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 02:45:42.009327267 (301h14m27.47141975s) [seg_pos={0 0}, upload=265104]: Begin upload segment... Oct 6 02:45:42.009417419 (301h14m27.471509892s) [seg_pos={0 0}, upload=265104]: Done begin upload segment. Oct 6 02:45:42.009444500 (301h14m27.471536983s) [noise=true , address=77.90.31.14:4662 , piece_id=MPVBPBB63JILLOUF3N7SRLCZ2F74LX7Z4AH7ZFG3BDNBHT4UIXTQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.009561473 (301h14m27.471653956s) [noise=true , address=145.79.227.19:4077 , piece_id=EJIAPZSBMAO77E33V6MEBOQHUXVVYKMAVP53QAZXZSYBYDFP7MMQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.010197574 (301h14m27.472290057s) [noise=true , address=145.79.227.15:4875 , piece_id=PBDTNZ2NWRZZC3VCIQCVR2WLT2AP6662I2SSDNP6BLUDKCJMRK6Q, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.010634216 (301h14m27.472726699s) [noise=true , address=145.79.227.4:4185 , piece_id=3AD745KFD67M4XFNDXMJVEU34RRBQXXE4P3HVJJRDBHK3IIXNYTA, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011073742 (301h14m27.473166225s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011505996 (301h14m27.473598479s) [noise=true , address=45.91.116.10:4081 , piece_id=FTXZ2LOHQMY74KOX6XZC3TLHWXOGDTIY3SZLSPTE7THGKNLMPXRQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011982502 (301h14m27.474074985s) [noise=true , address=45.91.116.6:4974 , piece_id=7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.012488236 (301h14m27.474580719s) [noise=true , address=145.79.227.7:4294 , piece_id=7E3B4BGLCTW6C5DM6PPDJPWGVJS5HEGYPAPXRAMX7XDSG7USPQFQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.013242472 (301h14m27.475334955s) [noise=true , address=45.91.116.13:4154 , piece_id=2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.014529382 (301h14m27.476621855s) [noise=true , address=45.91.116.6:4975 , piece_id=T3VWH5OBQ7FE66GCKHZLEDSIXFVYEU3V52HI4CPD27QL3RH75K7Q, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.014632619 (301h14m27.476725102s) [noise=true , address=45.91.116.11:4117 , piece_id=QKEYHSK5JOL65H5T2KUEAEK6VKTNCZAYXIEZTMB76PV7P5PEVILQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.015990214 (301h14m27.478082697s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.016291928 (301h14m27.478384412s) [noise=true , address=45.91.116.12:4226 , piece_id=NM7SEIKZYUHZLNVSOGYVZWMI7U7X5SOXOJDAEGY5NF3M6QQFPZSQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.017455915 (301h14m27.479548399s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.018864467 (301h14m27.48095694s) [noise=true , address=77.90.31.10:4518 , piece_id=6FGNAMVK6OAGJTZJNKTUJWGRW7MLGMJMGJKMAINWMPW7BV74OO6A, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.020222022 (301h14m27.482314495s) [noise=true , address=45.91.116.11:4118 , piece_id=VUZB3QRFST2T3HJHMEYYZ6CV7CKJTFWQGNOBDTXHGET2MBUIC4HQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:43.755608897 (301h14m29.21770138s) [upload=265104]: Next returned nil segment Oct 6 02:45:43.755631300 (301h14m29.217723773s) [upload=265104]: Waiting for error group managing segments... Oct 6 02:45:44.043930124 (301h14m29.506022597s) [noise=true , address=145.79.227.19:4077 , piece_id=EJIAPZSBMAO77E33V6MEBOQHUXVVYKMAVP53QAZXZSYBYDFP7MMQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.045501296 (301h14m29.507593769s) [noise=true , address=145.79.227.4:4185 , piece_id=3AD745KFD67M4XFNDXMJVEU34RRBQXXE4P3HVJJRDBHK3IIXNYTA, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.057881199 (301h14m29.519973672s) [noise=true , address=145.79.227.15:4875 , piece_id=PBDTNZ2NWRZZC3VCIQCVR2WLT2AP6662I2SSDNP6BLUDKCJMRK6Q, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.068726540 (301h14m29.530819023s) [noise=true , address=145.79.227.7:4294 , piece_id=7E3B4BGLCTW6C5DM6PPDJPWGVJS5HEGYPAPXRAMX7XDSG7USPQFQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.897895712 (301h14m30.359988185s) [noise=true , address=45.91.116.12:4226 , piece_id=NM7SEIKZYUHZLNVSOGYVZWMI7U7X5SOXOJDAEGY5NF3M6QQFPZSQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.252859665 (301h14m30.714952148s) [noise=true , address=77.90.31.14:4662 , piece_id=MPVBPBB63JILLOUF3N7SRLCZ2F74LX7Z4AH7ZFG3BDNBHT4UIXTQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.318952216 (301h14m30.781044699s) [noise=true , address=45.91.116.6:4974 , piece_id=7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.836439779 (301h14m31.298532262s) [noise=true , address=45.91.116.10:4081 , piece_id=FTXZ2LOHQMY74KOX6XZC3TLHWXOGDTIY3SZLSPTE7THGKNLMPXRQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.975895173 (301h14m31.437987656s) [noise=true , address=45.91.116.6:4975 , piece_id=T3VWH5OBQ7FE66GCKHZLEDSIXFVYEU3V52HI4CPD27QL3RH75K7Q, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:46.826703981 (301h14m32.288796454s) [noise=true , address=77.90.31.10:4518 , piece_id=6FGNAMVK6OAGJTZJNKTUJWGRW7MLGMJMGJKMAINWMPW7BV74OO6A, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:48.069759484 (301h14m33.531851968s) [noise=true , address=45.91.116.13:4154 , piece_id=2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:54.791582362 (301h14m40.253674835s) [noise=true , address=45.91.116.11:4118 , piece_id=VUZB3QRFST2T3HJHMEYYZ6CV7CKJTFWQGNOBDTXHGET2MBUIC4HQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:56.329163998 (301h14m41.791256481s) [noise=true , address=45.91.116.11:4117 , piece_id=QKEYHSK5JOL65H5T2KUEAEK6VKTNCZAYXIEZTMB76PV7P5PEVILQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:56.329201379 (301h14m41.791293862s) [seg_pos={0 0}, upload=265104]: Segment reached optimal threshold of 13 pieces. Oct 6 02:45:56.329362146 (301h14m41.791454629s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Oct 6 02:45:56.329371974 (301h14m41.791464457s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329546396 (301h14m41.791638869s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: write tcp 188.166.225.170:57604->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 02:45:56.329556665 (301h14m41.791649148s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329567937 (301h14m41.791660421s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: rpc: dial tcp 77.90.31.19:4841: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 02:45:56.329579870 (301h14m41.791672353s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329599808 (301h14m41.791692291s) [seg_pos={0 0}, upload=265104]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 6 02:45:56.329683116 (301h14m41.7917756s) [upload=265104]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 6 02:45:56.329690771 (301h14m41.791783254s) [upload=265104]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 6 02:45:56.636341641 (301h14m42.098434134s) [upload=265104]: Done uploading segments... Oct 6 02:45:56.636368131 (301h14m42.098460614s) [upload=265104]: Upload finished. err: <nil> Duration: 6.071620897s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................X.....................X..........] duration: 5.907688983s FAIL (2025-10-06 02:54:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: 2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: 7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA) context deadline exceeded ======LOGS====== Oct 6 02:45:41.756337907 (301h14m27.21843039s) [upload=265104]: Starting upload Oct 6 02:45:41.756430433 (301h14m27.218522916s) [upload=265104]: Done starting upload Oct 6 02:45:41.756513701 (301h14m27.218606185s) [upload=265104]: Uploading segments... Oct 6 02:45:41.756752276 (301h14m27.218844759s) [upload=265104]: Got next segment. Inline: false Oct 6 02:45:41.756794536 (301h14m27.218887019s) [upload=265104]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 02:45:41.756800898 (301h14m27.218893381s) [upload=265104]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 02:45:42.009327267 (301h14m27.47141975s) [seg_pos={0 0}, upload=265104]: Begin upload segment... Oct 6 02:45:42.009417419 (301h14m27.471509892s) [seg_pos={0 0}, upload=265104]: Done begin upload segment. Oct 6 02:45:42.009444500 (301h14m27.471536983s) [noise=true , address=77.90.31.14:4662 , piece_id=MPVBPBB63JILLOUF3N7SRLCZ2F74LX7Z4AH7ZFG3BDNBHT4UIXTQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.009561473 (301h14m27.471653956s) [noise=true , address=145.79.227.19:4077 , piece_id=EJIAPZSBMAO77E33V6MEBOQHUXVVYKMAVP53QAZXZSYBYDFP7MMQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.010197574 (301h14m27.472290057s) [noise=true , address=145.79.227.15:4875 , piece_id=PBDTNZ2NWRZZC3VCIQCVR2WLT2AP6662I2SSDNP6BLUDKCJMRK6Q, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.010634216 (301h14m27.472726699s) [noise=true , address=145.79.227.4:4185 , piece_id=3AD745KFD67M4XFNDXMJVEU34RRBQXXE4P3HVJJRDBHK3IIXNYTA, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011073742 (301h14m27.473166225s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011505996 (301h14m27.473598479s) [noise=true , address=45.91.116.10:4081 , piece_id=FTXZ2LOHQMY74KOX6XZC3TLHWXOGDTIY3SZLSPTE7THGKNLMPXRQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011982502 (301h14m27.474074985s) [noise=true , address=45.91.116.6:4974 , piece_id=7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.012488236 (301h14m27.474580719s) [noise=true , address=145.79.227.7:4294 , piece_id=7E3B4BGLCTW6C5DM6PPDJPWGVJS5HEGYPAPXRAMX7XDSG7USPQFQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.013242472 (301h14m27.475334955s) [noise=true , address=45.91.116.13:4154 , piece_id=2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.014529382 (301h14m27.476621855s) [noise=true , address=45.91.116.6:4975 , piece_id=T3VWH5OBQ7FE66GCKHZLEDSIXFVYEU3V52HI4CPD27QL3RH75K7Q, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.014632619 (301h14m27.476725102s) [noise=true , address=45.91.116.11:4117 , piece_id=QKEYHSK5JOL65H5T2KUEAEK6VKTNCZAYXIEZTMB76PV7P5PEVILQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.015990214 (301h14m27.478082697s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.016291928 (301h14m27.478384412s) [noise=true , address=45.91.116.12:4226 , piece_id=NM7SEIKZYUHZLNVSOGYVZWMI7U7X5SOXOJDAEGY5NF3M6QQFPZSQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.017455915 (301h14m27.479548399s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.018864467 (301h14m27.48095694s) [noise=true , address=77.90.31.10:4518 , piece_id=6FGNAMVK6OAGJTZJNKTUJWGRW7MLGMJMGJKMAINWMPW7BV74OO6A, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.020222022 (301h14m27.482314495s) [noise=true , address=45.91.116.11:4118 , piece_id=VUZB3QRFST2T3HJHMEYYZ6CV7CKJTFWQGNOBDTXHGET2MBUIC4HQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:43.755608897 (301h14m29.21770138s) [upload=265104]: Next returned nil segment Oct 6 02:45:43.755631300 (301h14m29.217723773s) [upload=265104]: Waiting for error group managing segments... Oct 6 02:45:44.043930124 (301h14m29.506022597s) [noise=true , address=145.79.227.19:4077 , piece_id=EJIAPZSBMAO77E33V6MEBOQHUXVVYKMAVP53QAZXZSYBYDFP7MMQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.045501296 (301h14m29.507593769s) [noise=true , address=145.79.227.4:4185 , piece_id=3AD745KFD67M4XFNDXMJVEU34RRBQXXE4P3HVJJRDBHK3IIXNYTA, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.057881199 (301h14m29.519973672s) [noise=true , address=145.79.227.15:4875 , piece_id=PBDTNZ2NWRZZC3VCIQCVR2WLT2AP6662I2SSDNP6BLUDKCJMRK6Q, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.068726540 (301h14m29.530819023s) [noise=true , address=145.79.227.7:4294 , piece_id=7E3B4BGLCTW6C5DM6PPDJPWGVJS5HEGYPAPXRAMX7XDSG7USPQFQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.897895712 (301h14m30.359988185s) [noise=true , address=45.91.116.12:4226 , piece_id=NM7SEIKZYUHZLNVSOGYVZWMI7U7X5SOXOJDAEGY5NF3M6QQFPZSQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.252859665 (301h14m30.714952148s) [noise=true , address=77.90.31.14:4662 , piece_id=MPVBPBB63JILLOUF3N7SRLCZ2F74LX7Z4AH7ZFG3BDNBHT4UIXTQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.318952216 (301h14m30.781044699s) [noise=true , address=45.91.116.6:4974 , piece_id=7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.836439779 (301h14m31.298532262s) [noise=true , address=45.91.116.10:4081 , piece_id=FTXZ2LOHQMY74KOX6XZC3TLHWXOGDTIY3SZLSPTE7THGKNLMPXRQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.975895173 (301h14m31.437987656s) [noise=true , address=45.91.116.6:4975 , piece_id=T3VWH5OBQ7FE66GCKHZLEDSIXFVYEU3V52HI4CPD27QL3RH75K7Q, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:46.826703981 (301h14m32.288796454s) [noise=true , address=77.90.31.10:4518 , piece_id=6FGNAMVK6OAGJTZJNKTUJWGRW7MLGMJMGJKMAINWMPW7BV74OO6A, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:48.069759484 (301h14m33.531851968s) [noise=true , address=45.91.116.13:4154 , piece_id=2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:54.791582362 (301h14m40.253674835s) [noise=true , address=45.91.116.11:4118 , piece_id=VUZB3QRFST2T3HJHMEYYZ6CV7CKJTFWQGNOBDTXHGET2MBUIC4HQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:56.329163998 (301h14m41.791256481s) [noise=true , address=45.91.116.11:4117 , piece_id=QKEYHSK5JOL65H5T2KUEAEK6VKTNCZAYXIEZTMB76PV7P5PEVILQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:56.329201379 (301h14m41.791293862s) [seg_pos={0 0}, upload=265104]: Segment reached optimal threshold of 13 pieces. Oct 6 02:45:56.329362146 (301h14m41.791454629s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Oct 6 02:45:56.329371974 (301h14m41.791464457s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329546396 (301h14m41.791638869s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: write tcp 188.166.225.170:57604->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 02:45:56.329556665 (301h14m41.791649148s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329567937 (301h14m41.791660421s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: rpc: dial tcp 77.90.31.19:4841: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 02:45:56.329579870 (301h14m41.791672353s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329599808 (301h14m41.791692291s) [seg_pos={0 0}, upload=265104]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 6 02:45:56.329683116 (301h14m41.7917756s) [upload=265104]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 6 02:45:56.329690771 (301h14m41.791783254s) [upload=265104]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 6 02:45:56.636341641 (301h14m42.098434134s) [upload=265104]: Done uploading segments... Oct 6 02:45:56.636368131 (301h14m42.098460614s) [upload=265104]: Upload finished. err: <nil> Duration: 5.907726935s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................X.....................X.........] duration: 6.033918315s FAIL (2025-10-06 02:53:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: 2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: 7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA) context deadline exceeded ======LOGS====== Oct 6 02:45:41.756337907 (301h14m27.21843039s) [upload=265104]: Starting upload Oct 6 02:45:41.756430433 (301h14m27.218522916s) [upload=265104]: Done starting upload Oct 6 02:45:41.756513701 (301h14m27.218606185s) [upload=265104]: Uploading segments... Oct 6 02:45:41.756752276 (301h14m27.218844759s) [upload=265104]: Got next segment. Inline: false Oct 6 02:45:41.756794536 (301h14m27.218887019s) [upload=265104]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 02:45:41.756800898 (301h14m27.218893381s) [upload=265104]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 02:45:42.009327267 (301h14m27.47141975s) [seg_pos={0 0}, upload=265104]: Begin upload segment... Oct 6 02:45:42.009417419 (301h14m27.471509892s) [seg_pos={0 0}, upload=265104]: Done begin upload segment. Oct 6 02:45:42.009444500 (301h14m27.471536983s) [noise=true , address=77.90.31.14:4662 , piece_id=MPVBPBB63JILLOUF3N7SRLCZ2F74LX7Z4AH7ZFG3BDNBHT4UIXTQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.009561473 (301h14m27.471653956s) [noise=true , address=145.79.227.19:4077 , piece_id=EJIAPZSBMAO77E33V6MEBOQHUXVVYKMAVP53QAZXZSYBYDFP7MMQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.010197574 (301h14m27.472290057s) [noise=true , address=145.79.227.15:4875 , piece_id=PBDTNZ2NWRZZC3VCIQCVR2WLT2AP6662I2SSDNP6BLUDKCJMRK6Q, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.010634216 (301h14m27.472726699s) [noise=true , address=145.79.227.4:4185 , piece_id=3AD745KFD67M4XFNDXMJVEU34RRBQXXE4P3HVJJRDBHK3IIXNYTA, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011073742 (301h14m27.473166225s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011505996 (301h14m27.473598479s) [noise=true , address=45.91.116.10:4081 , piece_id=FTXZ2LOHQMY74KOX6XZC3TLHWXOGDTIY3SZLSPTE7THGKNLMPXRQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011982502 (301h14m27.474074985s) [noise=true , address=45.91.116.6:4974 , piece_id=7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.012488236 (301h14m27.474580719s) [noise=true , address=145.79.227.7:4294 , piece_id=7E3B4BGLCTW6C5DM6PPDJPWGVJS5HEGYPAPXRAMX7XDSG7USPQFQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.013242472 (301h14m27.475334955s) [noise=true , address=45.91.116.13:4154 , piece_id=2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.014529382 (301h14m27.476621855s) [noise=true , address=45.91.116.6:4975 , piece_id=T3VWH5OBQ7FE66GCKHZLEDSIXFVYEU3V52HI4CPD27QL3RH75K7Q, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.014632619 (301h14m27.476725102s) [noise=true , address=45.91.116.11:4117 , piece_id=QKEYHSK5JOL65H5T2KUEAEK6VKTNCZAYXIEZTMB76PV7P5PEVILQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.015990214 (301h14m27.478082697s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.016291928 (301h14m27.478384412s) [noise=true , address=45.91.116.12:4226 , piece_id=NM7SEIKZYUHZLNVSOGYVZWMI7U7X5SOXOJDAEGY5NF3M6QQFPZSQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.017455915 (301h14m27.479548399s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.018864467 (301h14m27.48095694s) [noise=true , address=77.90.31.10:4518 , piece_id=6FGNAMVK6OAGJTZJNKTUJWGRW7MLGMJMGJKMAINWMPW7BV74OO6A, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.020222022 (301h14m27.482314495s) [noise=true , address=45.91.116.11:4118 , piece_id=VUZB3QRFST2T3HJHMEYYZ6CV7CKJTFWQGNOBDTXHGET2MBUIC4HQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:43.755608897 (301h14m29.21770138s) [upload=265104]: Next returned nil segment Oct 6 02:45:43.755631300 (301h14m29.217723773s) [upload=265104]: Waiting for error group managing segments... Oct 6 02:45:44.043930124 (301h14m29.506022597s) [noise=true , address=145.79.227.19:4077 , piece_id=EJIAPZSBMAO77E33V6MEBOQHUXVVYKMAVP53QAZXZSYBYDFP7MMQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.045501296 (301h14m29.507593769s) [noise=true , address=145.79.227.4:4185 , piece_id=3AD745KFD67M4XFNDXMJVEU34RRBQXXE4P3HVJJRDBHK3IIXNYTA, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.057881199 (301h14m29.519973672s) [noise=true , address=145.79.227.15:4875 , piece_id=PBDTNZ2NWRZZC3VCIQCVR2WLT2AP6662I2SSDNP6BLUDKCJMRK6Q, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.068726540 (301h14m29.530819023s) [noise=true , address=145.79.227.7:4294 , piece_id=7E3B4BGLCTW6C5DM6PPDJPWGVJS5HEGYPAPXRAMX7XDSG7USPQFQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.897895712 (301h14m30.359988185s) [noise=true , address=45.91.116.12:4226 , piece_id=NM7SEIKZYUHZLNVSOGYVZWMI7U7X5SOXOJDAEGY5NF3M6QQFPZSQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.252859665 (301h14m30.714952148s) [noise=true , address=77.90.31.14:4662 , piece_id=MPVBPBB63JILLOUF3N7SRLCZ2F74LX7Z4AH7ZFG3BDNBHT4UIXTQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.318952216 (301h14m30.781044699s) [noise=true , address=45.91.116.6:4974 , piece_id=7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.836439779 (301h14m31.298532262s) [noise=true , address=45.91.116.10:4081 , piece_id=FTXZ2LOHQMY74KOX6XZC3TLHWXOGDTIY3SZLSPTE7THGKNLMPXRQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.975895173 (301h14m31.437987656s) [noise=true , address=45.91.116.6:4975 , piece_id=T3VWH5OBQ7FE66GCKHZLEDSIXFVYEU3V52HI4CPD27QL3RH75K7Q, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:46.826703981 (301h14m32.288796454s) [noise=true , address=77.90.31.10:4518 , piece_id=6FGNAMVK6OAGJTZJNKTUJWGRW7MLGMJMGJKMAINWMPW7BV74OO6A, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:48.069759484 (301h14m33.531851968s) [noise=true , address=45.91.116.13:4154 , piece_id=2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:54.791582362 (301h14m40.253674835s) [noise=true , address=45.91.116.11:4118 , piece_id=VUZB3QRFST2T3HJHMEYYZ6CV7CKJTFWQGNOBDTXHGET2MBUIC4HQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:56.329163998 (301h14m41.791256481s) [noise=true , address=45.91.116.11:4117 , piece_id=QKEYHSK5JOL65H5T2KUEAEK6VKTNCZAYXIEZTMB76PV7P5PEVILQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:56.329201379 (301h14m41.791293862s) [seg_pos={0 0}, upload=265104]: Segment reached optimal threshold of 13 pieces. Oct 6 02:45:56.329362146 (301h14m41.791454629s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Oct 6 02:45:56.329371974 (301h14m41.791464457s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329546396 (301h14m41.791638869s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: write tcp 188.166.225.170:57604->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 02:45:56.329556665 (301h14m41.791649148s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329567937 (301h14m41.791660421s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: rpc: dial tcp 77.90.31.19:4841: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 02:45:56.329579870 (301h14m41.791672353s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329599808 (301h14m41.791692291s) [seg_pos={0 0}, upload=265104]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 6 02:45:56.329683116 (301h14m41.7917756s) [upload=265104]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 6 02:45:56.329690771 (301h14m41.791783254s) [upload=265104]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 6 02:45:56.636341641 (301h14m42.098434134s) [upload=265104]: Done uploading segments... Oct 6 02:45:56.636368131 (301h14m42.098460614s) [upload=265104]: Upload finished. err: <nil> Duration: 6.033949404s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................X.....................X........] duration: 6.095605758s FAIL (2025-10-06 02:52:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: 2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: 7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA) context deadline exceeded ======LOGS====== Oct 6 02:45:41.756337907 (301h14m27.21843039s) [upload=265104]: Starting upload Oct 6 02:45:41.756430433 (301h14m27.218522916s) [upload=265104]: Done starting upload Oct 6 02:45:41.756513701 (301h14m27.218606185s) [upload=265104]: Uploading segments... Oct 6 02:45:41.756752276 (301h14m27.218844759s) [upload=265104]: Got next segment. Inline: false Oct 6 02:45:41.756794536 (301h14m27.218887019s) [upload=265104]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 02:45:41.756800898 (301h14m27.218893381s) [upload=265104]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 02:45:42.009327267 (301h14m27.47141975s) [seg_pos={0 0}, upload=265104]: Begin upload segment... Oct 6 02:45:42.009417419 (301h14m27.471509892s) [seg_pos={0 0}, upload=265104]: Done begin upload segment. Oct 6 02:45:42.009444500 (301h14m27.471536983s) [noise=true , address=77.90.31.14:4662 , piece_id=MPVBPBB63JILLOUF3N7SRLCZ2F74LX7Z4AH7ZFG3BDNBHT4UIXTQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.009561473 (301h14m27.471653956s) [noise=true , address=145.79.227.19:4077 , piece_id=EJIAPZSBMAO77E33V6MEBOQHUXVVYKMAVP53QAZXZSYBYDFP7MMQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.010197574 (301h14m27.472290057s) [noise=true , address=145.79.227.15:4875 , piece_id=PBDTNZ2NWRZZC3VCIQCVR2WLT2AP6662I2SSDNP6BLUDKCJMRK6Q, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.010634216 (301h14m27.472726699s) [noise=true , address=145.79.227.4:4185 , piece_id=3AD745KFD67M4XFNDXMJVEU34RRBQXXE4P3HVJJRDBHK3IIXNYTA, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011073742 (301h14m27.473166225s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011505996 (301h14m27.473598479s) [noise=true , address=45.91.116.10:4081 , piece_id=FTXZ2LOHQMY74KOX6XZC3TLHWXOGDTIY3SZLSPTE7THGKNLMPXRQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011982502 (301h14m27.474074985s) [noise=true , address=45.91.116.6:4974 , piece_id=7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.012488236 (301h14m27.474580719s) [noise=true , address=145.79.227.7:4294 , piece_id=7E3B4BGLCTW6C5DM6PPDJPWGVJS5HEGYPAPXRAMX7XDSG7USPQFQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.013242472 (301h14m27.475334955s) [noise=true , address=45.91.116.13:4154 , piece_id=2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.014529382 (301h14m27.476621855s) [noise=true , address=45.91.116.6:4975 , piece_id=T3VWH5OBQ7FE66GCKHZLEDSIXFVYEU3V52HI4CPD27QL3RH75K7Q, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.014632619 (301h14m27.476725102s) [noise=true , address=45.91.116.11:4117 , piece_id=QKEYHSK5JOL65H5T2KUEAEK6VKTNCZAYXIEZTMB76PV7P5PEVILQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.015990214 (301h14m27.478082697s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.016291928 (301h14m27.478384412s) [noise=true , address=45.91.116.12:4226 , piece_id=NM7SEIKZYUHZLNVSOGYVZWMI7U7X5SOXOJDAEGY5NF3M6QQFPZSQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.017455915 (301h14m27.479548399s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.018864467 (301h14m27.48095694s) [noise=true , address=77.90.31.10:4518 , piece_id=6FGNAMVK6OAGJTZJNKTUJWGRW7MLGMJMGJKMAINWMPW7BV74OO6A, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.020222022 (301h14m27.482314495s) [noise=true , address=45.91.116.11:4118 , piece_id=VUZB3QRFST2T3HJHMEYYZ6CV7CKJTFWQGNOBDTXHGET2MBUIC4HQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:43.755608897 (301h14m29.21770138s) [upload=265104]: Next returned nil segment Oct 6 02:45:43.755631300 (301h14m29.217723773s) [upload=265104]: Waiting for error group managing segments... Oct 6 02:45:44.043930124 (301h14m29.506022597s) [noise=true , address=145.79.227.19:4077 , piece_id=EJIAPZSBMAO77E33V6MEBOQHUXVVYKMAVP53QAZXZSYBYDFP7MMQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.045501296 (301h14m29.507593769s) [noise=true , address=145.79.227.4:4185 , piece_id=3AD745KFD67M4XFNDXMJVEU34RRBQXXE4P3HVJJRDBHK3IIXNYTA, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.057881199 (301h14m29.519973672s) [noise=true , address=145.79.227.15:4875 , piece_id=PBDTNZ2NWRZZC3VCIQCVR2WLT2AP6662I2SSDNP6BLUDKCJMRK6Q, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.068726540 (301h14m29.530819023s) [noise=true , address=145.79.227.7:4294 , piece_id=7E3B4BGLCTW6C5DM6PPDJPWGVJS5HEGYPAPXRAMX7XDSG7USPQFQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.897895712 (301h14m30.359988185s) [noise=true , address=45.91.116.12:4226 , piece_id=NM7SEIKZYUHZLNVSOGYVZWMI7U7X5SOXOJDAEGY5NF3M6QQFPZSQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.252859665 (301h14m30.714952148s) [noise=true , address=77.90.31.14:4662 , piece_id=MPVBPBB63JILLOUF3N7SRLCZ2F74LX7Z4AH7ZFG3BDNBHT4UIXTQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.318952216 (301h14m30.781044699s) [noise=true , address=45.91.116.6:4974 , piece_id=7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.836439779 (301h14m31.298532262s) [noise=true , address=45.91.116.10:4081 , piece_id=FTXZ2LOHQMY74KOX6XZC3TLHWXOGDTIY3SZLSPTE7THGKNLMPXRQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.975895173 (301h14m31.437987656s) [noise=true , address=45.91.116.6:4975 , piece_id=T3VWH5OBQ7FE66GCKHZLEDSIXFVYEU3V52HI4CPD27QL3RH75K7Q, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:46.826703981 (301h14m32.288796454s) [noise=true , address=77.90.31.10:4518 , piece_id=6FGNAMVK6OAGJTZJNKTUJWGRW7MLGMJMGJKMAINWMPW7BV74OO6A, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:48.069759484 (301h14m33.531851968s) [noise=true , address=45.91.116.13:4154 , piece_id=2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:54.791582362 (301h14m40.253674835s) [noise=true , address=45.91.116.11:4118 , piece_id=VUZB3QRFST2T3HJHMEYYZ6CV7CKJTFWQGNOBDTXHGET2MBUIC4HQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:56.329163998 (301h14m41.791256481s) [noise=true , address=45.91.116.11:4117 , piece_id=QKEYHSK5JOL65H5T2KUEAEK6VKTNCZAYXIEZTMB76PV7P5PEVILQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:56.329201379 (301h14m41.791293862s) [seg_pos={0 0}, upload=265104]: Segment reached optimal threshold of 13 pieces. Oct 6 02:45:56.329362146 (301h14m41.791454629s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Oct 6 02:45:56.329371974 (301h14m41.791464457s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329546396 (301h14m41.791638869s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: write tcp 188.166.225.170:57604->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 02:45:56.329556665 (301h14m41.791649148s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329567937 (301h14m41.791660421s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: rpc: dial tcp 77.90.31.19:4841: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 02:45:56.329579870 (301h14m41.791672353s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329599808 (301h14m41.791692291s) [seg_pos={0 0}, upload=265104]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 6 02:45:56.329683116 (301h14m41.7917756s) [upload=265104]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 6 02:45:56.329690771 (301h14m41.791783254s) [upload=265104]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 6 02:45:56.636341641 (301h14m42.098434134s) [upload=265104]: Done uploading segments... Oct 6 02:45:56.636368131 (301h14m42.098460614s) [upload=265104]: Upload finished. err: <nil> Duration: 6.095641406s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................X.....................X.......] duration: 6.267293873s FAIL (2025-10-06 02:51:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: 2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: 7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA) context deadline exceeded ======LOGS====== Oct 6 02:45:41.756337907 (301h14m27.21843039s) [upload=265104]: Starting upload Oct 6 02:45:41.756430433 (301h14m27.218522916s) [upload=265104]: Done starting upload Oct 6 02:45:41.756513701 (301h14m27.218606185s) [upload=265104]: Uploading segments... Oct 6 02:45:41.756752276 (301h14m27.218844759s) [upload=265104]: Got next segment. Inline: false Oct 6 02:45:41.756794536 (301h14m27.218887019s) [upload=265104]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 02:45:41.756800898 (301h14m27.218893381s) [upload=265104]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 02:45:42.009327267 (301h14m27.47141975s) [seg_pos={0 0}, upload=265104]: Begin upload segment... Oct 6 02:45:42.009417419 (301h14m27.471509892s) [seg_pos={0 0}, upload=265104]: Done begin upload segment. Oct 6 02:45:42.009444500 (301h14m27.471536983s) [noise=true , address=77.90.31.14:4662 , piece_id=MPVBPBB63JILLOUF3N7SRLCZ2F74LX7Z4AH7ZFG3BDNBHT4UIXTQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.009561473 (301h14m27.471653956s) [noise=true , address=145.79.227.19:4077 , piece_id=EJIAPZSBMAO77E33V6MEBOQHUXVVYKMAVP53QAZXZSYBYDFP7MMQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.010197574 (301h14m27.472290057s) [noise=true , address=145.79.227.15:4875 , piece_id=PBDTNZ2NWRZZC3VCIQCVR2WLT2AP6662I2SSDNP6BLUDKCJMRK6Q, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.010634216 (301h14m27.472726699s) [noise=true , address=145.79.227.4:4185 , piece_id=3AD745KFD67M4XFNDXMJVEU34RRBQXXE4P3HVJJRDBHK3IIXNYTA, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011073742 (301h14m27.473166225s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011505996 (301h14m27.473598479s) [noise=true , address=45.91.116.10:4081 , piece_id=FTXZ2LOHQMY74KOX6XZC3TLHWXOGDTIY3SZLSPTE7THGKNLMPXRQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011982502 (301h14m27.474074985s) [noise=true , address=45.91.116.6:4974 , piece_id=7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.012488236 (301h14m27.474580719s) [noise=true , address=145.79.227.7:4294 , piece_id=7E3B4BGLCTW6C5DM6PPDJPWGVJS5HEGYPAPXRAMX7XDSG7USPQFQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.013242472 (301h14m27.475334955s) [noise=true , address=45.91.116.13:4154 , piece_id=2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.014529382 (301h14m27.476621855s) [noise=true , address=45.91.116.6:4975 , piece_id=T3VWH5OBQ7FE66GCKHZLEDSIXFVYEU3V52HI4CPD27QL3RH75K7Q, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.014632619 (301h14m27.476725102s) [noise=true , address=45.91.116.11:4117 , piece_id=QKEYHSK5JOL65H5T2KUEAEK6VKTNCZAYXIEZTMB76PV7P5PEVILQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.015990214 (301h14m27.478082697s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.016291928 (301h14m27.478384412s) [noise=true , address=45.91.116.12:4226 , piece_id=NM7SEIKZYUHZLNVSOGYVZWMI7U7X5SOXOJDAEGY5NF3M6QQFPZSQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.017455915 (301h14m27.479548399s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.018864467 (301h14m27.48095694s) [noise=true , address=77.90.31.10:4518 , piece_id=6FGNAMVK6OAGJTZJNKTUJWGRW7MLGMJMGJKMAINWMPW7BV74OO6A, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.020222022 (301h14m27.482314495s) [noise=true , address=45.91.116.11:4118 , piece_id=VUZB3QRFST2T3HJHMEYYZ6CV7CKJTFWQGNOBDTXHGET2MBUIC4HQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:43.755608897 (301h14m29.21770138s) [upload=265104]: Next returned nil segment Oct 6 02:45:43.755631300 (301h14m29.217723773s) [upload=265104]: Waiting for error group managing segments... Oct 6 02:45:44.043930124 (301h14m29.506022597s) [noise=true , address=145.79.227.19:4077 , piece_id=EJIAPZSBMAO77E33V6MEBOQHUXVVYKMAVP53QAZXZSYBYDFP7MMQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.045501296 (301h14m29.507593769s) [noise=true , address=145.79.227.4:4185 , piece_id=3AD745KFD67M4XFNDXMJVEU34RRBQXXE4P3HVJJRDBHK3IIXNYTA, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.057881199 (301h14m29.519973672s) [noise=true , address=145.79.227.15:4875 , piece_id=PBDTNZ2NWRZZC3VCIQCVR2WLT2AP6662I2SSDNP6BLUDKCJMRK6Q, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.068726540 (301h14m29.530819023s) [noise=true , address=145.79.227.7:4294 , piece_id=7E3B4BGLCTW6C5DM6PPDJPWGVJS5HEGYPAPXRAMX7XDSG7USPQFQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.897895712 (301h14m30.359988185s) [noise=true , address=45.91.116.12:4226 , piece_id=NM7SEIKZYUHZLNVSOGYVZWMI7U7X5SOXOJDAEGY5NF3M6QQFPZSQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.252859665 (301h14m30.714952148s) [noise=true , address=77.90.31.14:4662 , piece_id=MPVBPBB63JILLOUF3N7SRLCZ2F74LX7Z4AH7ZFG3BDNBHT4UIXTQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.318952216 (301h14m30.781044699s) [noise=true , address=45.91.116.6:4974 , piece_id=7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.836439779 (301h14m31.298532262s) [noise=true , address=45.91.116.10:4081 , piece_id=FTXZ2LOHQMY74KOX6XZC3TLHWXOGDTIY3SZLSPTE7THGKNLMPXRQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.975895173 (301h14m31.437987656s) [noise=true , address=45.91.116.6:4975 , piece_id=T3VWH5OBQ7FE66GCKHZLEDSIXFVYEU3V52HI4CPD27QL3RH75K7Q, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:46.826703981 (301h14m32.288796454s) [noise=true , address=77.90.31.10:4518 , piece_id=6FGNAMVK6OAGJTZJNKTUJWGRW7MLGMJMGJKMAINWMPW7BV74OO6A, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:48.069759484 (301h14m33.531851968s) [noise=true , address=45.91.116.13:4154 , piece_id=2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:54.791582362 (301h14m40.253674835s) [noise=true , address=45.91.116.11:4118 , piece_id=VUZB3QRFST2T3HJHMEYYZ6CV7CKJTFWQGNOBDTXHGET2MBUIC4HQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:56.329163998 (301h14m41.791256481s) [noise=true , address=45.91.116.11:4117 , piece_id=QKEYHSK5JOL65H5T2KUEAEK6VKTNCZAYXIEZTMB76PV7P5PEVILQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:56.329201379 (301h14m41.791293862s) [seg_pos={0 0}, upload=265104]: Segment reached optimal threshold of 13 pieces. Oct 6 02:45:56.329362146 (301h14m41.791454629s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Oct 6 02:45:56.329371974 (301h14m41.791464457s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329546396 (301h14m41.791638869s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: write tcp 188.166.225.170:57604->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 02:45:56.329556665 (301h14m41.791649148s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329567937 (301h14m41.791660421s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: rpc: dial tcp 77.90.31.19:4841: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 02:45:56.329579870 (301h14m41.791672353s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329599808 (301h14m41.791692291s) [seg_pos={0 0}, upload=265104]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 6 02:45:56.329683116 (301h14m41.7917756s) [upload=265104]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 6 02:45:56.329690771 (301h14m41.791783254s) [upload=265104]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 6 02:45:56.636341641 (301h14m42.098434134s) [upload=265104]: Done uploading segments... Oct 6 02:45:56.636368131 (301h14m42.098460614s) [upload=265104]: Upload finished. err: <nil> Duration: 6.267328317s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................X.....................X......] duration: 6.059410115s FAIL (2025-10-06 02:50:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: 2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: 7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA) context deadline exceeded ======LOGS====== Oct 6 02:45:41.756337907 (301h14m27.21843039s) [upload=265104]: Starting upload Oct 6 02:45:41.756430433 (301h14m27.218522916s) [upload=265104]: Done starting upload Oct 6 02:45:41.756513701 (301h14m27.218606185s) [upload=265104]: Uploading segments... Oct 6 02:45:41.756752276 (301h14m27.218844759s) [upload=265104]: Got next segment. Inline: false Oct 6 02:45:41.756794536 (301h14m27.218887019s) [upload=265104]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 02:45:41.756800898 (301h14m27.218893381s) [upload=265104]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 02:45:42.009327267 (301h14m27.47141975s) [seg_pos={0 0}, upload=265104]: Begin upload segment... Oct 6 02:45:42.009417419 (301h14m27.471509892s) [seg_pos={0 0}, upload=265104]: Done begin upload segment. Oct 6 02:45:42.009444500 (301h14m27.471536983s) [noise=true , address=77.90.31.14:4662 , piece_id=MPVBPBB63JILLOUF3N7SRLCZ2F74LX7Z4AH7ZFG3BDNBHT4UIXTQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.009561473 (301h14m27.471653956s) [noise=true , address=145.79.227.19:4077 , piece_id=EJIAPZSBMAO77E33V6MEBOQHUXVVYKMAVP53QAZXZSYBYDFP7MMQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.010197574 (301h14m27.472290057s) [noise=true , address=145.79.227.15:4875 , piece_id=PBDTNZ2NWRZZC3VCIQCVR2WLT2AP6662I2SSDNP6BLUDKCJMRK6Q, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.010634216 (301h14m27.472726699s) [noise=true , address=145.79.227.4:4185 , piece_id=3AD745KFD67M4XFNDXMJVEU34RRBQXXE4P3HVJJRDBHK3IIXNYTA, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011073742 (301h14m27.473166225s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011505996 (301h14m27.473598479s) [noise=true , address=45.91.116.10:4081 , piece_id=FTXZ2LOHQMY74KOX6XZC3TLHWXOGDTIY3SZLSPTE7THGKNLMPXRQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011982502 (301h14m27.474074985s) [noise=true , address=45.91.116.6:4974 , piece_id=7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.012488236 (301h14m27.474580719s) [noise=true , address=145.79.227.7:4294 , piece_id=7E3B4BGLCTW6C5DM6PPDJPWGVJS5HEGYPAPXRAMX7XDSG7USPQFQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.013242472 (301h14m27.475334955s) [noise=true , address=45.91.116.13:4154 , piece_id=2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.014529382 (301h14m27.476621855s) [noise=true , address=45.91.116.6:4975 , piece_id=T3VWH5OBQ7FE66GCKHZLEDSIXFVYEU3V52HI4CPD27QL3RH75K7Q, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.014632619 (301h14m27.476725102s) [noise=true , address=45.91.116.11:4117 , piece_id=QKEYHSK5JOL65H5T2KUEAEK6VKTNCZAYXIEZTMB76PV7P5PEVILQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.015990214 (301h14m27.478082697s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.016291928 (301h14m27.478384412s) [noise=true , address=45.91.116.12:4226 , piece_id=NM7SEIKZYUHZLNVSOGYVZWMI7U7X5SOXOJDAEGY5NF3M6QQFPZSQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.017455915 (301h14m27.479548399s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.018864467 (301h14m27.48095694s) [noise=true , address=77.90.31.10:4518 , piece_id=6FGNAMVK6OAGJTZJNKTUJWGRW7MLGMJMGJKMAINWMPW7BV74OO6A, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.020222022 (301h14m27.482314495s) [noise=true , address=45.91.116.11:4118 , piece_id=VUZB3QRFST2T3HJHMEYYZ6CV7CKJTFWQGNOBDTXHGET2MBUIC4HQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:43.755608897 (301h14m29.21770138s) [upload=265104]: Next returned nil segment Oct 6 02:45:43.755631300 (301h14m29.217723773s) [upload=265104]: Waiting for error group managing segments... Oct 6 02:45:44.043930124 (301h14m29.506022597s) [noise=true , address=145.79.227.19:4077 , piece_id=EJIAPZSBMAO77E33V6MEBOQHUXVVYKMAVP53QAZXZSYBYDFP7MMQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.045501296 (301h14m29.507593769s) [noise=true , address=145.79.227.4:4185 , piece_id=3AD745KFD67M4XFNDXMJVEU34RRBQXXE4P3HVJJRDBHK3IIXNYTA, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.057881199 (301h14m29.519973672s) [noise=true , address=145.79.227.15:4875 , piece_id=PBDTNZ2NWRZZC3VCIQCVR2WLT2AP6662I2SSDNP6BLUDKCJMRK6Q, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.068726540 (301h14m29.530819023s) [noise=true , address=145.79.227.7:4294 , piece_id=7E3B4BGLCTW6C5DM6PPDJPWGVJS5HEGYPAPXRAMX7XDSG7USPQFQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.897895712 (301h14m30.359988185s) [noise=true , address=45.91.116.12:4226 , piece_id=NM7SEIKZYUHZLNVSOGYVZWMI7U7X5SOXOJDAEGY5NF3M6QQFPZSQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.252859665 (301h14m30.714952148s) [noise=true , address=77.90.31.14:4662 , piece_id=MPVBPBB63JILLOUF3N7SRLCZ2F74LX7Z4AH7ZFG3BDNBHT4UIXTQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.318952216 (301h14m30.781044699s) [noise=true , address=45.91.116.6:4974 , piece_id=7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.836439779 (301h14m31.298532262s) [noise=true , address=45.91.116.10:4081 , piece_id=FTXZ2LOHQMY74KOX6XZC3TLHWXOGDTIY3SZLSPTE7THGKNLMPXRQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.975895173 (301h14m31.437987656s) [noise=true , address=45.91.116.6:4975 , piece_id=T3VWH5OBQ7FE66GCKHZLEDSIXFVYEU3V52HI4CPD27QL3RH75K7Q, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:46.826703981 (301h14m32.288796454s) [noise=true , address=77.90.31.10:4518 , piece_id=6FGNAMVK6OAGJTZJNKTUJWGRW7MLGMJMGJKMAINWMPW7BV74OO6A, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:48.069759484 (301h14m33.531851968s) [noise=true , address=45.91.116.13:4154 , piece_id=2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:54.791582362 (301h14m40.253674835s) [noise=true , address=45.91.116.11:4118 , piece_id=VUZB3QRFST2T3HJHMEYYZ6CV7CKJTFWQGNOBDTXHGET2MBUIC4HQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:56.329163998 (301h14m41.791256481s) [noise=true , address=45.91.116.11:4117 , piece_id=QKEYHSK5JOL65H5T2KUEAEK6VKTNCZAYXIEZTMB76PV7P5PEVILQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:56.329201379 (301h14m41.791293862s) [seg_pos={0 0}, upload=265104]: Segment reached optimal threshold of 13 pieces. Oct 6 02:45:56.329362146 (301h14m41.791454629s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Oct 6 02:45:56.329371974 (301h14m41.791464457s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329546396 (301h14m41.791638869s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: write tcp 188.166.225.170:57604->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 02:45:56.329556665 (301h14m41.791649148s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329567937 (301h14m41.791660421s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: rpc: dial tcp 77.90.31.19:4841: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 02:45:56.329579870 (301h14m41.791672353s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329599808 (301h14m41.791692291s) [seg_pos={0 0}, upload=265104]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 6 02:45:56.329683116 (301h14m41.7917756s) [upload=265104]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 6 02:45:56.329690771 (301h14m41.791783254s) [upload=265104]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 6 02:45:56.636341641 (301h14m42.098434134s) [upload=265104]: Done uploading segments... Oct 6 02:45:56.636368131 (301h14m42.098460614s) [upload=265104]: Upload finished. err: <nil> Duration: 6.059445061s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................X.....................X.....] duration: 6.002496217s FAIL (2025-10-06 02:49:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: 2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: 7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA) context deadline exceeded ======LOGS====== Oct 6 02:45:41.756337907 (301h14m27.21843039s) [upload=265104]: Starting upload Oct 6 02:45:41.756430433 (301h14m27.218522916s) [upload=265104]: Done starting upload Oct 6 02:45:41.756513701 (301h14m27.218606185s) [upload=265104]: Uploading segments... Oct 6 02:45:41.756752276 (301h14m27.218844759s) [upload=265104]: Got next segment. Inline: false Oct 6 02:45:41.756794536 (301h14m27.218887019s) [upload=265104]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 02:45:41.756800898 (301h14m27.218893381s) [upload=265104]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 02:45:42.009327267 (301h14m27.47141975s) [seg_pos={0 0}, upload=265104]: Begin upload segment... Oct 6 02:45:42.009417419 (301h14m27.471509892s) [seg_pos={0 0}, upload=265104]: Done begin upload segment. Oct 6 02:45:42.009444500 (301h14m27.471536983s) [noise=true , address=77.90.31.14:4662 , piece_id=MPVBPBB63JILLOUF3N7SRLCZ2F74LX7Z4AH7ZFG3BDNBHT4UIXTQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.009561473 (301h14m27.471653956s) [noise=true , address=145.79.227.19:4077 , piece_id=EJIAPZSBMAO77E33V6MEBOQHUXVVYKMAVP53QAZXZSYBYDFP7MMQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.010197574 (301h14m27.472290057s) [noise=true , address=145.79.227.15:4875 , piece_id=PBDTNZ2NWRZZC3VCIQCVR2WLT2AP6662I2SSDNP6BLUDKCJMRK6Q, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.010634216 (301h14m27.472726699s) [noise=true , address=145.79.227.4:4185 , piece_id=3AD745KFD67M4XFNDXMJVEU34RRBQXXE4P3HVJJRDBHK3IIXNYTA, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011073742 (301h14m27.473166225s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011505996 (301h14m27.473598479s) [noise=true , address=45.91.116.10:4081 , piece_id=FTXZ2LOHQMY74KOX6XZC3TLHWXOGDTIY3SZLSPTE7THGKNLMPXRQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011982502 (301h14m27.474074985s) [noise=true , address=45.91.116.6:4974 , piece_id=7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.012488236 (301h14m27.474580719s) [noise=true , address=145.79.227.7:4294 , piece_id=7E3B4BGLCTW6C5DM6PPDJPWGVJS5HEGYPAPXRAMX7XDSG7USPQFQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.013242472 (301h14m27.475334955s) [noise=true , address=45.91.116.13:4154 , piece_id=2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.014529382 (301h14m27.476621855s) [noise=true , address=45.91.116.6:4975 , piece_id=T3VWH5OBQ7FE66GCKHZLEDSIXFVYEU3V52HI4CPD27QL3RH75K7Q, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.014632619 (301h14m27.476725102s) [noise=true , address=45.91.116.11:4117 , piece_id=QKEYHSK5JOL65H5T2KUEAEK6VKTNCZAYXIEZTMB76PV7P5PEVILQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.015990214 (301h14m27.478082697s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.016291928 (301h14m27.478384412s) [noise=true , address=45.91.116.12:4226 , piece_id=NM7SEIKZYUHZLNVSOGYVZWMI7U7X5SOXOJDAEGY5NF3M6QQFPZSQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.017455915 (301h14m27.479548399s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.018864467 (301h14m27.48095694s) [noise=true , address=77.90.31.10:4518 , piece_id=6FGNAMVK6OAGJTZJNKTUJWGRW7MLGMJMGJKMAINWMPW7BV74OO6A, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.020222022 (301h14m27.482314495s) [noise=true , address=45.91.116.11:4118 , piece_id=VUZB3QRFST2T3HJHMEYYZ6CV7CKJTFWQGNOBDTXHGET2MBUIC4HQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:43.755608897 (301h14m29.21770138s) [upload=265104]: Next returned nil segment Oct 6 02:45:43.755631300 (301h14m29.217723773s) [upload=265104]: Waiting for error group managing segments... Oct 6 02:45:44.043930124 (301h14m29.506022597s) [noise=true , address=145.79.227.19:4077 , piece_id=EJIAPZSBMAO77E33V6MEBOQHUXVVYKMAVP53QAZXZSYBYDFP7MMQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.045501296 (301h14m29.507593769s) [noise=true , address=145.79.227.4:4185 , piece_id=3AD745KFD67M4XFNDXMJVEU34RRBQXXE4P3HVJJRDBHK3IIXNYTA, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.057881199 (301h14m29.519973672s) [noise=true , address=145.79.227.15:4875 , piece_id=PBDTNZ2NWRZZC3VCIQCVR2WLT2AP6662I2SSDNP6BLUDKCJMRK6Q, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.068726540 (301h14m29.530819023s) [noise=true , address=145.79.227.7:4294 , piece_id=7E3B4BGLCTW6C5DM6PPDJPWGVJS5HEGYPAPXRAMX7XDSG7USPQFQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.897895712 (301h14m30.359988185s) [noise=true , address=45.91.116.12:4226 , piece_id=NM7SEIKZYUHZLNVSOGYVZWMI7U7X5SOXOJDAEGY5NF3M6QQFPZSQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.252859665 (301h14m30.714952148s) [noise=true , address=77.90.31.14:4662 , piece_id=MPVBPBB63JILLOUF3N7SRLCZ2F74LX7Z4AH7ZFG3BDNBHT4UIXTQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.318952216 (301h14m30.781044699s) [noise=true , address=45.91.116.6:4974 , piece_id=7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.836439779 (301h14m31.298532262s) [noise=true , address=45.91.116.10:4081 , piece_id=FTXZ2LOHQMY74KOX6XZC3TLHWXOGDTIY3SZLSPTE7THGKNLMPXRQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.975895173 (301h14m31.437987656s) [noise=true , address=45.91.116.6:4975 , piece_id=T3VWH5OBQ7FE66GCKHZLEDSIXFVYEU3V52HI4CPD27QL3RH75K7Q, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:46.826703981 (301h14m32.288796454s) [noise=true , address=77.90.31.10:4518 , piece_id=6FGNAMVK6OAGJTZJNKTUJWGRW7MLGMJMGJKMAINWMPW7BV74OO6A, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:48.069759484 (301h14m33.531851968s) [noise=true , address=45.91.116.13:4154 , piece_id=2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:54.791582362 (301h14m40.253674835s) [noise=true , address=45.91.116.11:4118 , piece_id=VUZB3QRFST2T3HJHMEYYZ6CV7CKJTFWQGNOBDTXHGET2MBUIC4HQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:56.329163998 (301h14m41.791256481s) [noise=true , address=45.91.116.11:4117 , piece_id=QKEYHSK5JOL65H5T2KUEAEK6VKTNCZAYXIEZTMB76PV7P5PEVILQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:56.329201379 (301h14m41.791293862s) [seg_pos={0 0}, upload=265104]: Segment reached optimal threshold of 13 pieces. Oct 6 02:45:56.329362146 (301h14m41.791454629s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Oct 6 02:45:56.329371974 (301h14m41.791464457s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329546396 (301h14m41.791638869s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: write tcp 188.166.225.170:57604->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 02:45:56.329556665 (301h14m41.791649148s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329567937 (301h14m41.791660421s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: rpc: dial tcp 77.90.31.19:4841: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 02:45:56.329579870 (301h14m41.791672353s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329599808 (301h14m41.791692291s) [seg_pos={0 0}, upload=265104]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 6 02:45:56.329683116 (301h14m41.7917756s) [upload=265104]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 6 02:45:56.329690771 (301h14m41.791783254s) [upload=265104]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 6 02:45:56.636341641 (301h14m42.098434134s) [upload=265104]: Done uploading segments... Oct 6 02:45:56.636368131 (301h14m42.098460614s) [upload=265104]: Upload finished. err: <nil> Duration: 6.002526344s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................X.....................X....] duration: 5.942141096s FAIL (2025-10-06 02:49:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: 2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: 7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA) context deadline exceeded ======LOGS====== Oct 6 02:45:41.756337907 (301h14m27.21843039s) [upload=265104]: Starting upload Oct 6 02:45:41.756430433 (301h14m27.218522916s) [upload=265104]: Done starting upload Oct 6 02:45:41.756513701 (301h14m27.218606185s) [upload=265104]: Uploading segments... Oct 6 02:45:41.756752276 (301h14m27.218844759s) [upload=265104]: Got next segment. Inline: false Oct 6 02:45:41.756794536 (301h14m27.218887019s) [upload=265104]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 02:45:41.756800898 (301h14m27.218893381s) [upload=265104]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 02:45:42.009327267 (301h14m27.47141975s) [seg_pos={0 0}, upload=265104]: Begin upload segment... Oct 6 02:45:42.009417419 (301h14m27.471509892s) [seg_pos={0 0}, upload=265104]: Done begin upload segment. Oct 6 02:45:42.009444500 (301h14m27.471536983s) [noise=true , address=77.90.31.14:4662 , piece_id=MPVBPBB63JILLOUF3N7SRLCZ2F74LX7Z4AH7ZFG3BDNBHT4UIXTQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.009561473 (301h14m27.471653956s) [noise=true , address=145.79.227.19:4077 , piece_id=EJIAPZSBMAO77E33V6MEBOQHUXVVYKMAVP53QAZXZSYBYDFP7MMQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.010197574 (301h14m27.472290057s) [noise=true , address=145.79.227.15:4875 , piece_id=PBDTNZ2NWRZZC3VCIQCVR2WLT2AP6662I2SSDNP6BLUDKCJMRK6Q, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.010634216 (301h14m27.472726699s) [noise=true , address=145.79.227.4:4185 , piece_id=3AD745KFD67M4XFNDXMJVEU34RRBQXXE4P3HVJJRDBHK3IIXNYTA, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011073742 (301h14m27.473166225s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011505996 (301h14m27.473598479s) [noise=true , address=45.91.116.10:4081 , piece_id=FTXZ2LOHQMY74KOX6XZC3TLHWXOGDTIY3SZLSPTE7THGKNLMPXRQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011982502 (301h14m27.474074985s) [noise=true , address=45.91.116.6:4974 , piece_id=7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.012488236 (301h14m27.474580719s) [noise=true , address=145.79.227.7:4294 , piece_id=7E3B4BGLCTW6C5DM6PPDJPWGVJS5HEGYPAPXRAMX7XDSG7USPQFQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.013242472 (301h14m27.475334955s) [noise=true , address=45.91.116.13:4154 , piece_id=2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.014529382 (301h14m27.476621855s) [noise=true , address=45.91.116.6:4975 , piece_id=T3VWH5OBQ7FE66GCKHZLEDSIXFVYEU3V52HI4CPD27QL3RH75K7Q, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.014632619 (301h14m27.476725102s) [noise=true , address=45.91.116.11:4117 , piece_id=QKEYHSK5JOL65H5T2KUEAEK6VKTNCZAYXIEZTMB76PV7P5PEVILQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.015990214 (301h14m27.478082697s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.016291928 (301h14m27.478384412s) [noise=true , address=45.91.116.12:4226 , piece_id=NM7SEIKZYUHZLNVSOGYVZWMI7U7X5SOXOJDAEGY5NF3M6QQFPZSQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.017455915 (301h14m27.479548399s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.018864467 (301h14m27.48095694s) [noise=true , address=77.90.31.10:4518 , piece_id=6FGNAMVK6OAGJTZJNKTUJWGRW7MLGMJMGJKMAINWMPW7BV74OO6A, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.020222022 (301h14m27.482314495s) [noise=true , address=45.91.116.11:4118 , piece_id=VUZB3QRFST2T3HJHMEYYZ6CV7CKJTFWQGNOBDTXHGET2MBUIC4HQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:43.755608897 (301h14m29.21770138s) [upload=265104]: Next returned nil segment Oct 6 02:45:43.755631300 (301h14m29.217723773s) [upload=265104]: Waiting for error group managing segments... Oct 6 02:45:44.043930124 (301h14m29.506022597s) [noise=true , address=145.79.227.19:4077 , piece_id=EJIAPZSBMAO77E33V6MEBOQHUXVVYKMAVP53QAZXZSYBYDFP7MMQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.045501296 (301h14m29.507593769s) [noise=true , address=145.79.227.4:4185 , piece_id=3AD745KFD67M4XFNDXMJVEU34RRBQXXE4P3HVJJRDBHK3IIXNYTA, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.057881199 (301h14m29.519973672s) [noise=true , address=145.79.227.15:4875 , piece_id=PBDTNZ2NWRZZC3VCIQCVR2WLT2AP6662I2SSDNP6BLUDKCJMRK6Q, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.068726540 (301h14m29.530819023s) [noise=true , address=145.79.227.7:4294 , piece_id=7E3B4BGLCTW6C5DM6PPDJPWGVJS5HEGYPAPXRAMX7XDSG7USPQFQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.897895712 (301h14m30.359988185s) [noise=true , address=45.91.116.12:4226 , piece_id=NM7SEIKZYUHZLNVSOGYVZWMI7U7X5SOXOJDAEGY5NF3M6QQFPZSQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.252859665 (301h14m30.714952148s) [noise=true , address=77.90.31.14:4662 , piece_id=MPVBPBB63JILLOUF3N7SRLCZ2F74LX7Z4AH7ZFG3BDNBHT4UIXTQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.318952216 (301h14m30.781044699s) [noise=true , address=45.91.116.6:4974 , piece_id=7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.836439779 (301h14m31.298532262s) [noise=true , address=45.91.116.10:4081 , piece_id=FTXZ2LOHQMY74KOX6XZC3TLHWXOGDTIY3SZLSPTE7THGKNLMPXRQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.975895173 (301h14m31.437987656s) [noise=true , address=45.91.116.6:4975 , piece_id=T3VWH5OBQ7FE66GCKHZLEDSIXFVYEU3V52HI4CPD27QL3RH75K7Q, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:46.826703981 (301h14m32.288796454s) [noise=true , address=77.90.31.10:4518 , piece_id=6FGNAMVK6OAGJTZJNKTUJWGRW7MLGMJMGJKMAINWMPW7BV74OO6A, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:48.069759484 (301h14m33.531851968s) [noise=true , address=45.91.116.13:4154 , piece_id=2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:54.791582362 (301h14m40.253674835s) [noise=true , address=45.91.116.11:4118 , piece_id=VUZB3QRFST2T3HJHMEYYZ6CV7CKJTFWQGNOBDTXHGET2MBUIC4HQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:56.329163998 (301h14m41.791256481s) [noise=true , address=45.91.116.11:4117 , piece_id=QKEYHSK5JOL65H5T2KUEAEK6VKTNCZAYXIEZTMB76PV7P5PEVILQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:56.329201379 (301h14m41.791293862s) [seg_pos={0 0}, upload=265104]: Segment reached optimal threshold of 13 pieces. Oct 6 02:45:56.329362146 (301h14m41.791454629s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Oct 6 02:45:56.329371974 (301h14m41.791464457s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329546396 (301h14m41.791638869s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: write tcp 188.166.225.170:57604->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 02:45:56.329556665 (301h14m41.791649148s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329567937 (301h14m41.791660421s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: rpc: dial tcp 77.90.31.19:4841: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 02:45:56.329579870 (301h14m41.791672353s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329599808 (301h14m41.791692291s) [seg_pos={0 0}, upload=265104]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 6 02:45:56.329683116 (301h14m41.7917756s) [upload=265104]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 6 02:45:56.329690771 (301h14m41.791783254s) [upload=265104]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 6 02:45:56.636341641 (301h14m42.098434134s) [upload=265104]: Done uploading segments... Oct 6 02:45:56.636368131 (301h14m42.098460614s) [upload=265104]: Upload finished. err: <nil> Duration: 5.94217019s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................X.....................X...] duration: 6.152001283s FAIL (2025-10-06 02:48:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: 2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: 7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA) context deadline exceeded ======LOGS====== Oct 6 02:45:41.756337907 (301h14m27.21843039s) [upload=265104]: Starting upload Oct 6 02:45:41.756430433 (301h14m27.218522916s) [upload=265104]: Done starting upload Oct 6 02:45:41.756513701 (301h14m27.218606185s) [upload=265104]: Uploading segments... Oct 6 02:45:41.756752276 (301h14m27.218844759s) [upload=265104]: Got next segment. Inline: false Oct 6 02:45:41.756794536 (301h14m27.218887019s) [upload=265104]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 02:45:41.756800898 (301h14m27.218893381s) [upload=265104]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 02:45:42.009327267 (301h14m27.47141975s) [seg_pos={0 0}, upload=265104]: Begin upload segment... Oct 6 02:45:42.009417419 (301h14m27.471509892s) [seg_pos={0 0}, upload=265104]: Done begin upload segment. Oct 6 02:45:42.009444500 (301h14m27.471536983s) [noise=true , address=77.90.31.14:4662 , piece_id=MPVBPBB63JILLOUF3N7SRLCZ2F74LX7Z4AH7ZFG3BDNBHT4UIXTQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.009561473 (301h14m27.471653956s) [noise=true , address=145.79.227.19:4077 , piece_id=EJIAPZSBMAO77E33V6MEBOQHUXVVYKMAVP53QAZXZSYBYDFP7MMQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.010197574 (301h14m27.472290057s) [noise=true , address=145.79.227.15:4875 , piece_id=PBDTNZ2NWRZZC3VCIQCVR2WLT2AP6662I2SSDNP6BLUDKCJMRK6Q, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.010634216 (301h14m27.472726699s) [noise=true , address=145.79.227.4:4185 , piece_id=3AD745KFD67M4XFNDXMJVEU34RRBQXXE4P3HVJJRDBHK3IIXNYTA, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011073742 (301h14m27.473166225s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011505996 (301h14m27.473598479s) [noise=true , address=45.91.116.10:4081 , piece_id=FTXZ2LOHQMY74KOX6XZC3TLHWXOGDTIY3SZLSPTE7THGKNLMPXRQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011982502 (301h14m27.474074985s) [noise=true , address=45.91.116.6:4974 , piece_id=7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.012488236 (301h14m27.474580719s) [noise=true , address=145.79.227.7:4294 , piece_id=7E3B4BGLCTW6C5DM6PPDJPWGVJS5HEGYPAPXRAMX7XDSG7USPQFQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.013242472 (301h14m27.475334955s) [noise=true , address=45.91.116.13:4154 , piece_id=2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.014529382 (301h14m27.476621855s) [noise=true , address=45.91.116.6:4975 , piece_id=T3VWH5OBQ7FE66GCKHZLEDSIXFVYEU3V52HI4CPD27QL3RH75K7Q, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.014632619 (301h14m27.476725102s) [noise=true , address=45.91.116.11:4117 , piece_id=QKEYHSK5JOL65H5T2KUEAEK6VKTNCZAYXIEZTMB76PV7P5PEVILQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.015990214 (301h14m27.478082697s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.016291928 (301h14m27.478384412s) [noise=true , address=45.91.116.12:4226 , piece_id=NM7SEIKZYUHZLNVSOGYVZWMI7U7X5SOXOJDAEGY5NF3M6QQFPZSQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.017455915 (301h14m27.479548399s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.018864467 (301h14m27.48095694s) [noise=true , address=77.90.31.10:4518 , piece_id=6FGNAMVK6OAGJTZJNKTUJWGRW7MLGMJMGJKMAINWMPW7BV74OO6A, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.020222022 (301h14m27.482314495s) [noise=true , address=45.91.116.11:4118 , piece_id=VUZB3QRFST2T3HJHMEYYZ6CV7CKJTFWQGNOBDTXHGET2MBUIC4HQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:43.755608897 (301h14m29.21770138s) [upload=265104]: Next returned nil segment Oct 6 02:45:43.755631300 (301h14m29.217723773s) [upload=265104]: Waiting for error group managing segments... Oct 6 02:45:44.043930124 (301h14m29.506022597s) [noise=true , address=145.79.227.19:4077 , piece_id=EJIAPZSBMAO77E33V6MEBOQHUXVVYKMAVP53QAZXZSYBYDFP7MMQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.045501296 (301h14m29.507593769s) [noise=true , address=145.79.227.4:4185 , piece_id=3AD745KFD67M4XFNDXMJVEU34RRBQXXE4P3HVJJRDBHK3IIXNYTA, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.057881199 (301h14m29.519973672s) [noise=true , address=145.79.227.15:4875 , piece_id=PBDTNZ2NWRZZC3VCIQCVR2WLT2AP6662I2SSDNP6BLUDKCJMRK6Q, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.068726540 (301h14m29.530819023s) [noise=true , address=145.79.227.7:4294 , piece_id=7E3B4BGLCTW6C5DM6PPDJPWGVJS5HEGYPAPXRAMX7XDSG7USPQFQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.897895712 (301h14m30.359988185s) [noise=true , address=45.91.116.12:4226 , piece_id=NM7SEIKZYUHZLNVSOGYVZWMI7U7X5SOXOJDAEGY5NF3M6QQFPZSQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.252859665 (301h14m30.714952148s) [noise=true , address=77.90.31.14:4662 , piece_id=MPVBPBB63JILLOUF3N7SRLCZ2F74LX7Z4AH7ZFG3BDNBHT4UIXTQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.318952216 (301h14m30.781044699s) [noise=true , address=45.91.116.6:4974 , piece_id=7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.836439779 (301h14m31.298532262s) [noise=true , address=45.91.116.10:4081 , piece_id=FTXZ2LOHQMY74KOX6XZC3TLHWXOGDTIY3SZLSPTE7THGKNLMPXRQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.975895173 (301h14m31.437987656s) [noise=true , address=45.91.116.6:4975 , piece_id=T3VWH5OBQ7FE66GCKHZLEDSIXFVYEU3V52HI4CPD27QL3RH75K7Q, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:46.826703981 (301h14m32.288796454s) [noise=true , address=77.90.31.10:4518 , piece_id=6FGNAMVK6OAGJTZJNKTUJWGRW7MLGMJMGJKMAINWMPW7BV74OO6A, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:48.069759484 (301h14m33.531851968s) [noise=true , address=45.91.116.13:4154 , piece_id=2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:54.791582362 (301h14m40.253674835s) [noise=true , address=45.91.116.11:4118 , piece_id=VUZB3QRFST2T3HJHMEYYZ6CV7CKJTFWQGNOBDTXHGET2MBUIC4HQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:56.329163998 (301h14m41.791256481s) [noise=true , address=45.91.116.11:4117 , piece_id=QKEYHSK5JOL65H5T2KUEAEK6VKTNCZAYXIEZTMB76PV7P5PEVILQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:56.329201379 (301h14m41.791293862s) [seg_pos={0 0}, upload=265104]: Segment reached optimal threshold of 13 pieces. Oct 6 02:45:56.329362146 (301h14m41.791454629s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Oct 6 02:45:56.329371974 (301h14m41.791464457s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329546396 (301h14m41.791638869s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: write tcp 188.166.225.170:57604->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 02:45:56.329556665 (301h14m41.791649148s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329567937 (301h14m41.791660421s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: rpc: dial tcp 77.90.31.19:4841: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 02:45:56.329579870 (301h14m41.791672353s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329599808 (301h14m41.791692291s) [seg_pos={0 0}, upload=265104]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 6 02:45:56.329683116 (301h14m41.7917756s) [upload=265104]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 6 02:45:56.329690771 (301h14m41.791783254s) [upload=265104]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 6 02:45:56.636341641 (301h14m42.098434134s) [upload=265104]: Done uploading segments... Oct 6 02:45:56.636368131 (301h14m42.098460614s) [upload=265104]: Upload finished. err: <nil> Duration: 6.15203098s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................X.....................X..] duration: 6.293772464s FAIL (2025-10-06 02:47:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: 2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: 7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA) context deadline exceeded ======LOGS====== Oct 6 02:45:41.756337907 (301h14m27.21843039s) [upload=265104]: Starting upload Oct 6 02:45:41.756430433 (301h14m27.218522916s) [upload=265104]: Done starting upload Oct 6 02:45:41.756513701 (301h14m27.218606185s) [upload=265104]: Uploading segments... Oct 6 02:45:41.756752276 (301h14m27.218844759s) [upload=265104]: Got next segment. Inline: false Oct 6 02:45:41.756794536 (301h14m27.218887019s) [upload=265104]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 02:45:41.756800898 (301h14m27.218893381s) [upload=265104]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 02:45:42.009327267 (301h14m27.47141975s) [seg_pos={0 0}, upload=265104]: Begin upload segment... Oct 6 02:45:42.009417419 (301h14m27.471509892s) [seg_pos={0 0}, upload=265104]: Done begin upload segment. Oct 6 02:45:42.009444500 (301h14m27.471536983s) [noise=true , address=77.90.31.14:4662 , piece_id=MPVBPBB63JILLOUF3N7SRLCZ2F74LX7Z4AH7ZFG3BDNBHT4UIXTQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.009561473 (301h14m27.471653956s) [noise=true , address=145.79.227.19:4077 , piece_id=EJIAPZSBMAO77E33V6MEBOQHUXVVYKMAVP53QAZXZSYBYDFP7MMQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.010197574 (301h14m27.472290057s) [noise=true , address=145.79.227.15:4875 , piece_id=PBDTNZ2NWRZZC3VCIQCVR2WLT2AP6662I2SSDNP6BLUDKCJMRK6Q, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.010634216 (301h14m27.472726699s) [noise=true , address=145.79.227.4:4185 , piece_id=3AD745KFD67M4XFNDXMJVEU34RRBQXXE4P3HVJJRDBHK3IIXNYTA, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011073742 (301h14m27.473166225s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011505996 (301h14m27.473598479s) [noise=true , address=45.91.116.10:4081 , piece_id=FTXZ2LOHQMY74KOX6XZC3TLHWXOGDTIY3SZLSPTE7THGKNLMPXRQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011982502 (301h14m27.474074985s) [noise=true , address=45.91.116.6:4974 , piece_id=7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.012488236 (301h14m27.474580719s) [noise=true , address=145.79.227.7:4294 , piece_id=7E3B4BGLCTW6C5DM6PPDJPWGVJS5HEGYPAPXRAMX7XDSG7USPQFQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.013242472 (301h14m27.475334955s) [noise=true , address=45.91.116.13:4154 , piece_id=2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.014529382 (301h14m27.476621855s) [noise=true , address=45.91.116.6:4975 , piece_id=T3VWH5OBQ7FE66GCKHZLEDSIXFVYEU3V52HI4CPD27QL3RH75K7Q, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.014632619 (301h14m27.476725102s) [noise=true , address=45.91.116.11:4117 , piece_id=QKEYHSK5JOL65H5T2KUEAEK6VKTNCZAYXIEZTMB76PV7P5PEVILQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.015990214 (301h14m27.478082697s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.016291928 (301h14m27.478384412s) [noise=true , address=45.91.116.12:4226 , piece_id=NM7SEIKZYUHZLNVSOGYVZWMI7U7X5SOXOJDAEGY5NF3M6QQFPZSQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.017455915 (301h14m27.479548399s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.018864467 (301h14m27.48095694s) [noise=true , address=77.90.31.10:4518 , piece_id=6FGNAMVK6OAGJTZJNKTUJWGRW7MLGMJMGJKMAINWMPW7BV74OO6A, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.020222022 (301h14m27.482314495s) [noise=true , address=45.91.116.11:4118 , piece_id=VUZB3QRFST2T3HJHMEYYZ6CV7CKJTFWQGNOBDTXHGET2MBUIC4HQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:43.755608897 (301h14m29.21770138s) [upload=265104]: Next returned nil segment Oct 6 02:45:43.755631300 (301h14m29.217723773s) [upload=265104]: Waiting for error group managing segments... Oct 6 02:45:44.043930124 (301h14m29.506022597s) [noise=true , address=145.79.227.19:4077 , piece_id=EJIAPZSBMAO77E33V6MEBOQHUXVVYKMAVP53QAZXZSYBYDFP7MMQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.045501296 (301h14m29.507593769s) [noise=true , address=145.79.227.4:4185 , piece_id=3AD745KFD67M4XFNDXMJVEU34RRBQXXE4P3HVJJRDBHK3IIXNYTA, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.057881199 (301h14m29.519973672s) [noise=true , address=145.79.227.15:4875 , piece_id=PBDTNZ2NWRZZC3VCIQCVR2WLT2AP6662I2SSDNP6BLUDKCJMRK6Q, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.068726540 (301h14m29.530819023s) [noise=true , address=145.79.227.7:4294 , piece_id=7E3B4BGLCTW6C5DM6PPDJPWGVJS5HEGYPAPXRAMX7XDSG7USPQFQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.897895712 (301h14m30.359988185s) [noise=true , address=45.91.116.12:4226 , piece_id=NM7SEIKZYUHZLNVSOGYVZWMI7U7X5SOXOJDAEGY5NF3M6QQFPZSQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.252859665 (301h14m30.714952148s) [noise=true , address=77.90.31.14:4662 , piece_id=MPVBPBB63JILLOUF3N7SRLCZ2F74LX7Z4AH7ZFG3BDNBHT4UIXTQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.318952216 (301h14m30.781044699s) [noise=true , address=45.91.116.6:4974 , piece_id=7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.836439779 (301h14m31.298532262s) [noise=true , address=45.91.116.10:4081 , piece_id=FTXZ2LOHQMY74KOX6XZC3TLHWXOGDTIY3SZLSPTE7THGKNLMPXRQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.975895173 (301h14m31.437987656s) [noise=true , address=45.91.116.6:4975 , piece_id=T3VWH5OBQ7FE66GCKHZLEDSIXFVYEU3V52HI4CPD27QL3RH75K7Q, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:46.826703981 (301h14m32.288796454s) [noise=true , address=77.90.31.10:4518 , piece_id=6FGNAMVK6OAGJTZJNKTUJWGRW7MLGMJMGJKMAINWMPW7BV74OO6A, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:48.069759484 (301h14m33.531851968s) [noise=true , address=45.91.116.13:4154 , piece_id=2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:54.791582362 (301h14m40.253674835s) [noise=true , address=45.91.116.11:4118 , piece_id=VUZB3QRFST2T3HJHMEYYZ6CV7CKJTFWQGNOBDTXHGET2MBUIC4HQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:56.329163998 (301h14m41.791256481s) [noise=true , address=45.91.116.11:4117 , piece_id=QKEYHSK5JOL65H5T2KUEAEK6VKTNCZAYXIEZTMB76PV7P5PEVILQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:56.329201379 (301h14m41.791293862s) [seg_pos={0 0}, upload=265104]: Segment reached optimal threshold of 13 pieces. Oct 6 02:45:56.329362146 (301h14m41.791454629s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Oct 6 02:45:56.329371974 (301h14m41.791464457s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329546396 (301h14m41.791638869s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: write tcp 188.166.225.170:57604->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 02:45:56.329556665 (301h14m41.791649148s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329567937 (301h14m41.791660421s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: rpc: dial tcp 77.90.31.19:4841: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 02:45:56.329579870 (301h14m41.791672353s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329599808 (301h14m41.791692291s) [seg_pos={0 0}, upload=265104]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 6 02:45:56.329683116 (301h14m41.7917756s) [upload=265104]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 6 02:45:56.329690771 (301h14m41.791783254s) [upload=265104]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 6 02:45:56.636341641 (301h14m42.098434134s) [upload=265104]: Done uploading segments... Oct 6 02:45:56.636368131 (301h14m42.098460614s) [upload=265104]: Upload finished. err: <nil> Duration: 6.293798372s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................X.....................X.] duration: 6.406812582s FAIL (2025-10-06 02:46:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: 2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: 7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA) context deadline exceeded ======LOGS====== Oct 6 02:45:41.756337907 (301h14m27.21843039s) [upload=265104]: Starting upload Oct 6 02:45:41.756430433 (301h14m27.218522916s) [upload=265104]: Done starting upload Oct 6 02:45:41.756513701 (301h14m27.218606185s) [upload=265104]: Uploading segments... Oct 6 02:45:41.756752276 (301h14m27.218844759s) [upload=265104]: Got next segment. Inline: false Oct 6 02:45:41.756794536 (301h14m27.218887019s) [upload=265104]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 02:45:41.756800898 (301h14m27.218893381s) [upload=265104]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 02:45:42.009327267 (301h14m27.47141975s) [seg_pos={0 0}, upload=265104]: Begin upload segment... Oct 6 02:45:42.009417419 (301h14m27.471509892s) [seg_pos={0 0}, upload=265104]: Done begin upload segment. Oct 6 02:45:42.009444500 (301h14m27.471536983s) [noise=true , address=77.90.31.14:4662 , piece_id=MPVBPBB63JILLOUF3N7SRLCZ2F74LX7Z4AH7ZFG3BDNBHT4UIXTQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.009561473 (301h14m27.471653956s) [noise=true , address=145.79.227.19:4077 , piece_id=EJIAPZSBMAO77E33V6MEBOQHUXVVYKMAVP53QAZXZSYBYDFP7MMQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.010197574 (301h14m27.472290057s) [noise=true , address=145.79.227.15:4875 , piece_id=PBDTNZ2NWRZZC3VCIQCVR2WLT2AP6662I2SSDNP6BLUDKCJMRK6Q, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.010634216 (301h14m27.472726699s) [noise=true , address=145.79.227.4:4185 , piece_id=3AD745KFD67M4XFNDXMJVEU34RRBQXXE4P3HVJJRDBHK3IIXNYTA, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011073742 (301h14m27.473166225s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011505996 (301h14m27.473598479s) [noise=true , address=45.91.116.10:4081 , piece_id=FTXZ2LOHQMY74KOX6XZC3TLHWXOGDTIY3SZLSPTE7THGKNLMPXRQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011982502 (301h14m27.474074985s) [noise=true , address=45.91.116.6:4974 , piece_id=7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.012488236 (301h14m27.474580719s) [noise=true , address=145.79.227.7:4294 , piece_id=7E3B4BGLCTW6C5DM6PPDJPWGVJS5HEGYPAPXRAMX7XDSG7USPQFQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.013242472 (301h14m27.475334955s) [noise=true , address=45.91.116.13:4154 , piece_id=2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.014529382 (301h14m27.476621855s) [noise=true , address=45.91.116.6:4975 , piece_id=T3VWH5OBQ7FE66GCKHZLEDSIXFVYEU3V52HI4CPD27QL3RH75K7Q, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.014632619 (301h14m27.476725102s) [noise=true , address=45.91.116.11:4117 , piece_id=QKEYHSK5JOL65H5T2KUEAEK6VKTNCZAYXIEZTMB76PV7P5PEVILQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.015990214 (301h14m27.478082697s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.016291928 (301h14m27.478384412s) [noise=true , address=45.91.116.12:4226 , piece_id=NM7SEIKZYUHZLNVSOGYVZWMI7U7X5SOXOJDAEGY5NF3M6QQFPZSQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.017455915 (301h14m27.479548399s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.018864467 (301h14m27.48095694s) [noise=true , address=77.90.31.10:4518 , piece_id=6FGNAMVK6OAGJTZJNKTUJWGRW7MLGMJMGJKMAINWMPW7BV74OO6A, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.020222022 (301h14m27.482314495s) [noise=true , address=45.91.116.11:4118 , piece_id=VUZB3QRFST2T3HJHMEYYZ6CV7CKJTFWQGNOBDTXHGET2MBUIC4HQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:43.755608897 (301h14m29.21770138s) [upload=265104]: Next returned nil segment Oct 6 02:45:43.755631300 (301h14m29.217723773s) [upload=265104]: Waiting for error group managing segments... Oct 6 02:45:44.043930124 (301h14m29.506022597s) [noise=true , address=145.79.227.19:4077 , piece_id=EJIAPZSBMAO77E33V6MEBOQHUXVVYKMAVP53QAZXZSYBYDFP7MMQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.045501296 (301h14m29.507593769s) [noise=true , address=145.79.227.4:4185 , piece_id=3AD745KFD67M4XFNDXMJVEU34RRBQXXE4P3HVJJRDBHK3IIXNYTA, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.057881199 (301h14m29.519973672s) [noise=true , address=145.79.227.15:4875 , piece_id=PBDTNZ2NWRZZC3VCIQCVR2WLT2AP6662I2SSDNP6BLUDKCJMRK6Q, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.068726540 (301h14m29.530819023s) [noise=true , address=145.79.227.7:4294 , piece_id=7E3B4BGLCTW6C5DM6PPDJPWGVJS5HEGYPAPXRAMX7XDSG7USPQFQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.897895712 (301h14m30.359988185s) [noise=true , address=45.91.116.12:4226 , piece_id=NM7SEIKZYUHZLNVSOGYVZWMI7U7X5SOXOJDAEGY5NF3M6QQFPZSQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.252859665 (301h14m30.714952148s) [noise=true , address=77.90.31.14:4662 , piece_id=MPVBPBB63JILLOUF3N7SRLCZ2F74LX7Z4AH7ZFG3BDNBHT4UIXTQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.318952216 (301h14m30.781044699s) [noise=true , address=45.91.116.6:4974 , piece_id=7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.836439779 (301h14m31.298532262s) [noise=true , address=45.91.116.10:4081 , piece_id=FTXZ2LOHQMY74KOX6XZC3TLHWXOGDTIY3SZLSPTE7THGKNLMPXRQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.975895173 (301h14m31.437987656s) [noise=true , address=45.91.116.6:4975 , piece_id=T3VWH5OBQ7FE66GCKHZLEDSIXFVYEU3V52HI4CPD27QL3RH75K7Q, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:46.826703981 (301h14m32.288796454s) [noise=true , address=77.90.31.10:4518 , piece_id=6FGNAMVK6OAGJTZJNKTUJWGRW7MLGMJMGJKMAINWMPW7BV74OO6A, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:48.069759484 (301h14m33.531851968s) [noise=true , address=45.91.116.13:4154 , piece_id=2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:54.791582362 (301h14m40.253674835s) [noise=true , address=45.91.116.11:4118 , piece_id=VUZB3QRFST2T3HJHMEYYZ6CV7CKJTFWQGNOBDTXHGET2MBUIC4HQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:56.329163998 (301h14m41.791256481s) [noise=true , address=45.91.116.11:4117 , piece_id=QKEYHSK5JOL65H5T2KUEAEK6VKTNCZAYXIEZTMB76PV7P5PEVILQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:56.329201379 (301h14m41.791293862s) [seg_pos={0 0}, upload=265104]: Segment reached optimal threshold of 13 pieces. Oct 6 02:45:56.329362146 (301h14m41.791454629s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Oct 6 02:45:56.329371974 (301h14m41.791464457s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329546396 (301h14m41.791638869s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: write tcp 188.166.225.170:57604->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 02:45:56.329556665 (301h14m41.791649148s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329567937 (301h14m41.791660421s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: rpc: dial tcp 77.90.31.19:4841: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 02:45:56.329579870 (301h14m41.791672353s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329599808 (301h14m41.791692291s) [seg_pos={0 0}, upload=265104]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 6 02:45:56.329683116 (301h14m41.7917756s) [upload=265104]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 6 02:45:56.329690771 (301h14m41.791783254s) [upload=265104]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 6 02:45:56.636341641 (301h14m42.098434134s) [upload=265104]: Done uploading segments... Oct 6 02:45:56.636368131 (301h14m42.098460614s) [upload=265104]: Upload finished. err: <nil> Duration: 6.406844172s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................X.....................X] duration: 30.006161626s FAIL (2025-10-06 02:46:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: 2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: 7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA) context deadline exceeded ======LOGS====== Oct 6 02:45:41.756337907 (301h14m27.21843039s) [upload=265104]: Starting upload Oct 6 02:45:41.756430433 (301h14m27.218522916s) [upload=265104]: Done starting upload Oct 6 02:45:41.756513701 (301h14m27.218606185s) [upload=265104]: Uploading segments... Oct 6 02:45:41.756752276 (301h14m27.218844759s) [upload=265104]: Got next segment. Inline: false Oct 6 02:45:41.756794536 (301h14m27.218887019s) [upload=265104]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 02:45:41.756800898 (301h14m27.218893381s) [upload=265104]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 02:45:42.009327267 (301h14m27.47141975s) [seg_pos={0 0}, upload=265104]: Begin upload segment... Oct 6 02:45:42.009417419 (301h14m27.471509892s) [seg_pos={0 0}, upload=265104]: Done begin upload segment. Oct 6 02:45:42.009444500 (301h14m27.471536983s) [noise=true , address=77.90.31.14:4662 , piece_id=MPVBPBB63JILLOUF3N7SRLCZ2F74LX7Z4AH7ZFG3BDNBHT4UIXTQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.009561473 (301h14m27.471653956s) [noise=true , address=145.79.227.19:4077 , piece_id=EJIAPZSBMAO77E33V6MEBOQHUXVVYKMAVP53QAZXZSYBYDFP7MMQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.010197574 (301h14m27.472290057s) [noise=true , address=145.79.227.15:4875 , piece_id=PBDTNZ2NWRZZC3VCIQCVR2WLT2AP6662I2SSDNP6BLUDKCJMRK6Q, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.010634216 (301h14m27.472726699s) [noise=true , address=145.79.227.4:4185 , piece_id=3AD745KFD67M4XFNDXMJVEU34RRBQXXE4P3HVJJRDBHK3IIXNYTA, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011073742 (301h14m27.473166225s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011505996 (301h14m27.473598479s) [noise=true , address=45.91.116.10:4081 , piece_id=FTXZ2LOHQMY74KOX6XZC3TLHWXOGDTIY3SZLSPTE7THGKNLMPXRQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.011982502 (301h14m27.474074985s) [noise=true , address=45.91.116.6:4974 , piece_id=7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.012488236 (301h14m27.474580719s) [noise=true , address=145.79.227.7:4294 , piece_id=7E3B4BGLCTW6C5DM6PPDJPWGVJS5HEGYPAPXRAMX7XDSG7USPQFQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.013242472 (301h14m27.475334955s) [noise=true , address=45.91.116.13:4154 , piece_id=2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.014529382 (301h14m27.476621855s) [noise=true , address=45.91.116.6:4975 , piece_id=T3VWH5OBQ7FE66GCKHZLEDSIXFVYEU3V52HI4CPD27QL3RH75K7Q, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.014632619 (301h14m27.476725102s) [noise=true , address=45.91.116.11:4117 , piece_id=QKEYHSK5JOL65H5T2KUEAEK6VKTNCZAYXIEZTMB76PV7P5PEVILQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.015990214 (301h14m27.478082697s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.016291928 (301h14m27.478384412s) [noise=true , address=45.91.116.12:4226 , piece_id=NM7SEIKZYUHZLNVSOGYVZWMI7U7X5SOXOJDAEGY5NF3M6QQFPZSQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.017455915 (301h14m27.479548399s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.018864467 (301h14m27.48095694s) [noise=true , address=77.90.31.10:4518 , piece_id=6FGNAMVK6OAGJTZJNKTUJWGRW7MLGMJMGJKMAINWMPW7BV74OO6A, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:42.020222022 (301h14m27.482314495s) [noise=true , address=45.91.116.11:4118 , piece_id=VUZB3QRFST2T3HJHMEYYZ6CV7CKJTFWQGNOBDTXHGET2MBUIC4HQ, seg_pos={0 0}, upload=265104]: Uploading piece... Oct 6 02:45:43.755608897 (301h14m29.21770138s) [upload=265104]: Next returned nil segment Oct 6 02:45:43.755631300 (301h14m29.217723773s) [upload=265104]: Waiting for error group managing segments... Oct 6 02:45:44.043930124 (301h14m29.506022597s) [noise=true , address=145.79.227.19:4077 , piece_id=EJIAPZSBMAO77E33V6MEBOQHUXVVYKMAVP53QAZXZSYBYDFP7MMQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.045501296 (301h14m29.507593769s) [noise=true , address=145.79.227.4:4185 , piece_id=3AD745KFD67M4XFNDXMJVEU34RRBQXXE4P3HVJJRDBHK3IIXNYTA, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.057881199 (301h14m29.519973672s) [noise=true , address=145.79.227.15:4875 , piece_id=PBDTNZ2NWRZZC3VCIQCVR2WLT2AP6662I2SSDNP6BLUDKCJMRK6Q, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.068726540 (301h14m29.530819023s) [noise=true , address=145.79.227.7:4294 , piece_id=7E3B4BGLCTW6C5DM6PPDJPWGVJS5HEGYPAPXRAMX7XDSG7USPQFQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:44.897895712 (301h14m30.359988185s) [noise=true , address=45.91.116.12:4226 , piece_id=NM7SEIKZYUHZLNVSOGYVZWMI7U7X5SOXOJDAEGY5NF3M6QQFPZSQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.252859665 (301h14m30.714952148s) [noise=true , address=77.90.31.14:4662 , piece_id=MPVBPBB63JILLOUF3N7SRLCZ2F74LX7Z4AH7ZFG3BDNBHT4UIXTQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.318952216 (301h14m30.781044699s) [noise=true , address=45.91.116.6:4974 , piece_id=7GB3M7QBFROHDC473T6HRRBIBS365BTTXXSJ43NRZFO26VI6FQNA, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.836439779 (301h14m31.298532262s) [noise=true , address=45.91.116.10:4081 , piece_id=FTXZ2LOHQMY74KOX6XZC3TLHWXOGDTIY3SZLSPTE7THGKNLMPXRQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:45.975895173 (301h14m31.437987656s) [noise=true , address=45.91.116.6:4975 , piece_id=T3VWH5OBQ7FE66GCKHZLEDSIXFVYEU3V52HI4CPD27QL3RH75K7Q, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:46.826703981 (301h14m32.288796454s) [noise=true , address=77.90.31.10:4518 , piece_id=6FGNAMVK6OAGJTZJNKTUJWGRW7MLGMJMGJKMAINWMPW7BV74OO6A, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:48.069759484 (301h14m33.531851968s) [noise=true , address=45.91.116.13:4154 , piece_id=2X3CRKGIY4SGS3PZRGN42SGAV2GODMYOIORJLXQQXVUZETDHLFAQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:54.791582362 (301h14m40.253674835s) [noise=true , address=45.91.116.11:4118 , piece_id=VUZB3QRFST2T3HJHMEYYZ6CV7CKJTFWQGNOBDTXHGET2MBUIC4HQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:56.329163998 (301h14m41.791256481s) [noise=true , address=45.91.116.11:4117 , piece_id=QKEYHSK5JOL65H5T2KUEAEK6VKTNCZAYXIEZTMB76PV7P5PEVILQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: <nil> Oct 6 02:45:56.329201379 (301h14m41.791293862s) [seg_pos={0 0}, upload=265104]: Segment reached optimal threshold of 13 pieces. Oct 6 02:45:56.329362146 (301h14m41.791454629s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Oct 6 02:45:56.329371974 (301h14m41.791464457s) [noise=true , address=45.91.116.8:4048 , piece_id=6566HGINBEPR4CQKAHALXBV55HTARQPJQXFZCIMN4H5PCIBNSNIQ, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329546396 (301h14m41.791638869s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: write tcp 188.166.225.170:57604->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 02:45:56.329556665 (301h14m41.791649148s) [noise=true , address=45.91.116.12:4225 , piece_id=K2I55ZGQRH5U7D3RAT4QCTNZJ2XNTU7DOGFLJ3RUXUHH7MYNACUQ, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329567937 (301h14m41.791660421s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: rpc: dial tcp 77.90.31.19:4841: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 02:45:56.329579870 (301h14m41.791672353s) [noise=true , address=77.90.31.19:4841 , piece_id=PS4CVE2ZJAUJ6CHXWVGSL4T6CXMOGR2NK6CAU6DLBR45YBR3MH5A, seg_pos={0 0}, upload=265104]: Long tail cancellation due to optimal threshold Oct 6 02:45:56.329599808 (301h14m41.791692291s) [seg_pos={0 0}, upload=265104]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 6 02:45:56.329683116 (301h14m41.7917756s) [upload=265104]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 6 02:45:56.329690771 (301h14m41.791783254s) [upload=265104]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 6 02:45:56.636341641 (301h14m42.098434134s) [upload=265104]: Done uploading segments... Oct 6 02:45:56.636368131 (301h14m42.098460614s) [upload=265104]: Upload finished. err: <nil> Duration: 30.006236319s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 5.935492305s FAIL (2025-10-06 02:45:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, Piece ID: T254A3S7TTJEBOIBDDCS34DU7HV25TXDTNZWF2OGE4KRQ7YEA5ZA) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: OS3T2GGABRXYK6HNEXK4C5F5HTJFDEP7SHA5QV2R3SCMXDE2TD5A) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: S5NHKRILOZ47C4NXH4UXBG34ZF6IP2XHBR5JDDBNCBOZK26NALIA) context deadline exceeded; piecestore: (Node ID: 12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, Piece ID: CHJRUVVLWAR7627JOLKL2QHAK7EHW72ZTGE3BTOAAM4ZL5AZL3KQ) context deadline exceeded; piecestore: (Node ID: 12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, Piece ID: TLZVEBMQCCWINXB4AVRKHWKZ7WULCGUC6E5GGDJQLASKWNEHSJPA) context deadline exceeded; piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: LJKUWHGMINU2P7JTS2QN5VP25HPZBKRMFXRDBSO5OTVYTS47SB4A) context deadline exceeded; piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: VTE4WIBLUWV74XMSSPP37CEAWPRQHA3GFV2RJRQYGRQM5JGF3H3Q) context deadline exceeded; piecestore: (Node ID: 1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, Piece ID: 52Q5TEMXLZHLWJ5QWRFHGYWW6RCL2Y57AY6K2J7I3BUA22BNARQQ) context deadline exceeded ======LOGS====== 914003723 (300h58m34.376096196s) [upload=264873]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 02:29:49.215144679 (300h58m34.677237162s) [seg_pos={0 0}, upload=264873]: Begin upload segment... Oct 6 02:29:49.215212839 (300h58m34.677305312s) [seg_pos={0 0}, upload=264873]: Done begin upload segment. Oct 6 02:29:49.215241294 (300h58m34.677333777s) [noise=true , address=145.79.227.7:4294 , piece_id=TLZVEBMQCCWINXB4AVRKHWKZ7WULCGUC6E5GGDJQLASKWNEHSJPA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.215249739 (300h58m34.677342232s) [noise=true , address=45.91.116.4:4937 , piece_id=CHJRUVVLWAR7627JOLKL2QHAK7EHW72ZTGE3BTOAAM4ZL5AZL3KQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.215349820 (300h58m34.677442303s) [noise=true , address=77.90.31.18:4805 , piece_id=LJKUWHGMINU2P7JTS2QN5VP25HPZBKRMFXRDBSO5OTVYTS47SB4A, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.216239644 (300h58m34.678332127s) [noise=true , address=45.91.116.5:4298 , piece_id=VTE4WIBLUWV74XMSSPP37CEAWPRQHA3GFV2RJRQYGRQM5JGF3H3Q, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.218583898 (300h58m34.680676392s) [noise=true , address=45.91.116.10:4082 , piece_id=NX5W46AGWBSZ7EM2LZHZDPE6MEPSLZMNVFZEE2LQ5F5LV7YSQRKQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.219326442 (300h58m34.681418925s) [noise=true , address=45.91.116.6:4974 , piece_id=DBXKLTPEBNF5GVB4PC26BY4EPXOWHU2OWSLWGSHS52S6CDUZIMOA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.222198982 (300h58m34.684291465s) [noise=true , address=145.79.227.17:4972 , piece_id=I54FS5Q4CMMM46S6EP3C5MEUAJCVNWX23OHTBMNQNKP4TRERVZ7A, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.222520114 (300h58m34.684612597s) [noise=true , address=145.79.227.4:4185 , piece_id=ONU6YDQFAMJPRG3ZF45M7KUX5EGXZVQEGCTOJK3VGE2JXXLHCBAQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.225940076 (300h58m34.688032569s) [noise=true , address=77.90.31.13:4628 , piece_id=52Q5TEMXLZHLWJ5QWRFHGYWW6RCL2Y57AY6K2J7I3BUA22BNARQQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.227928832 (300h58m34.690021305s) [noise=true , address=145.79.227.16:4910 , piece_id=T254A3S7TTJEBOIBDDCS34DU7HV25TXDTNZWF2OGE4KRQ7YEA5ZA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.227998645 (300h58m34.690091128s) [noise=true , address=45.91.116.14:4190 , piece_id=OS3T2GGABRXYK6HNEXK4C5F5HTJFDEP7SHA5QV2R3SCMXDE2TD5A, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.231020309 (300h58m34.693112792s) [noise=true , address=145.79.227.18:4007 , piece_id=6PZE3N3QHDGV5K5HULQE5UJXWWG7RAIACP2OVEZVIOTTCWVNQO6Q, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.232908575 (300h58m34.695001068s) [noise=true , address=45.91.116.10:4081 , piece_id=PRCQUWMDPZTR2CSWMOODBECZOLAVJ4OPOUTGQHCPTWXUDQGY2MBQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.233037220 (300h58m34.695129693s) [noise=true , address=145.79.227.19:4079 , piece_id=2VBVLN2SNA6WUICKMNMNM7URPVUGWPB5SVZ335XZ7HIAUM3FY5VA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.234302058 (300h58m34.696394531s) [noise=true , address=45.91.116.7:4010 , piece_id=S5NHKRILOZ47C4NXH4UXBG34ZF6IP2XHBR5JDDBNCBOZK26NALIA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.235631800 (300h58m34.697724283s) [noise=true , address=77.90.31.15:4698 , piece_id=NOCO6KPTUBBKYCITDRFBKA3H5EJTKMDAO4AFQFFXC4PPH3D46X2A, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:50.988532254 (300h58m36.450624747s) [upload=264873]: Next returned nil segment Oct 6 02:29:50.988555589 (300h58m36.450648082s) [upload=264873]: Waiting for error group managing segments... Oct 6 02:29:51.282838267 (300h58m36.74493075s) [noise=true , address=45.91.116.4:4937 , piece_id=CHJRUVVLWAR7627JOLKL2QHAK7EHW72ZTGE3BTOAAM4ZL5AZL3KQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.318573943 (300h58m36.780666416s) [noise=true , address=77.90.31.18:4805 , piece_id=LJKUWHGMINU2P7JTS2QN5VP25HPZBKRMFXRDBSO5OTVYTS47SB4A, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.322112521 (300h58m36.784205004s) [noise=true , address=77.90.31.15:4698 , piece_id=NOCO6KPTUBBKYCITDRFBKA3H5EJTKMDAO4AFQFFXC4PPH3D46X2A, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.350536624 (300h58m36.812629108s) [noise=true , address=45.91.116.6:4974 , piece_id=DBXKLTPEBNF5GVB4PC26BY4EPXOWHU2OWSLWGSHS52S6CDUZIMOA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.362171679 (300h58m36.824264162s) [noise=true , address=45.91.116.10:4081 , piece_id=PRCQUWMDPZTR2CSWMOODBECZOLAVJ4OPOUTGQHCPTWXUDQGY2MBQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.366398738 (300h58m36.828491221s) [noise=true , address=45.91.116.10:4082 , piece_id=NX5W46AGWBSZ7EM2LZHZDPE6MEPSLZMNVFZEE2LQ5F5LV7YSQRKQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.375437720 (300h58m36.837530203s) [noise=true , address=45.91.116.14:4190 , piece_id=OS3T2GGABRXYK6HNEXK4C5F5HTJFDEP7SHA5QV2R3SCMXDE2TD5A, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.397600181 (300h58m36.859692664s) [noise=true , address=77.90.31.13:4628 , piece_id=52Q5TEMXLZHLWJ5QWRFHGYWW6RCL2Y57AY6K2J7I3BUA22BNARQQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.400132462 (300h58m36.862224945s) [noise=true , address=45.91.116.5:4298 , piece_id=VTE4WIBLUWV74XMSSPP37CEAWPRQHA3GFV2RJRQYGRQM5JGF3H3Q, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.412114018 (300h58m36.874206491s) [noise=true , address=45.91.116.7:4010 , piece_id=S5NHKRILOZ47C4NXH4UXBG34ZF6IP2XHBR5JDDBNCBOZK26NALIA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:30:14.302718308 (300h58m59.764810791s) [noise=true , address=145.79.227.18:4007 , piece_id=6PZE3N3QHDGV5K5HULQE5UJXWWG7RAIACP2OVEZVIOTTCWVNQO6Q, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:30:14.319938798 (300h58m59.782031281s) [noise=true , address=145.79.227.7:4294 , piece_id=TLZVEBMQCCWINXB4AVRKHWKZ7WULCGUC6E5GGDJQLASKWNEHSJPA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:30:16.104270571 (300h59m1.566363044s) [noise=true , address=145.79.227.16:4910 , piece_id=T254A3S7TTJEBOIBDDCS34DU7HV25TXDTNZWF2OGE4KRQ7YEA5ZA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:30:16.104295528 (300h59m1.566388012s) [seg_pos={0 0}, upload=264873]: Segment reached optimal threshold of 13 pieces. Oct 6 02:30:16.104442268 (300h59m1.566534751s) [noise=true , address=145.79.227.4:4185 , piece_id=ONU6YDQFAMJPRG3ZF45M7KUX5EGXZVQEGCTOJK3VGE2JXXLHCBAQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN): protocol: write tcp 188.166.225.170:34966->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:34966->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34966->145.79.227.4:4185: use of closed network connection Oct 6 02:30:16.104456395 (300h59m1.566548878s) [noise=true , address=145.79.227.4:4185 , piece_id=ONU6YDQFAMJPRG3ZF45M7KUX5EGXZVQEGCTOJK3VGE2JXXLHCBAQ, seg_pos={0 0}, upload=264873]: Long tail cancellation due to optimal threshold Oct 6 02:30:16.104598005 (300h59m1.566690478s) [noise=true , address=145.79.227.19:4079 , piece_id=2VBVLN2SNA6WUICKMNMNM7URPVUGWPB5SVZ335XZ7HIAUM3FY5VA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: write tcp 188.166.225.170:57162->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:57162->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57162->145.79.227.19:4079: use of closed network connection Oct 6 02:30:16.104604216 (300h59m1.566696699s) [noise=true , address=145.79.227.19:4079 , piece_id=2VBVLN2SNA6WUICKMNMNM7URPVUGWPB5SVZ335XZ7HIAUM3FY5VA, seg_pos={0 0}, upload=264873]: Long tail cancellation due to optimal threshold Oct 6 02:30:16.104672547 (300h59m1.56676503s) [noise=true , address=145.79.227.17:4972 , piece_id=I54FS5Q4CMMM46S6EP3C5MEUAJCVNWX23OHTBMNQNKP4TRERVZ7A, seg_pos={0 0}, upload=264873]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: write tcp 188.166.225.170:33630->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:33630->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33630->145.79.227.17:4972: use of closed network connection Oct 6 02:30:16.104678057 (300h59m1.56677053s) [noise=true , address=145.79.227.17:4972 , piece_id=I54FS5Q4CMMM46S6EP3C5MEUAJCVNWX23OHTBMNQNKP4TRERVZ7A, seg_pos={0 0}, upload=264873]: Long tail cancellation due to optimal threshold Oct 6 02:30:16.104695541 (300h59m1.566788024s) [seg_pos={0 0}, upload=264873]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 6 02:30:16.104762427 (300h59m1.56685491s) [upload=264873]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 6 02:30:16.104768159 (300h59m1.566860632s) [upload=264873]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 6 02:30:16.463894599 (300h59m1.925987092s) [upload=264873]: Done uploading segments... Oct 6 02:30:16.463937391 (300h59m1.926029874s) [upload=264873]: Upload finished. err: <nil> Duration:* 5.935523083s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 6.087614951s FAIL (2025-10-06 02:44:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, Piece ID: T254A3S7TTJEBOIBDDCS34DU7HV25TXDTNZWF2OGE4KRQ7YEA5ZA) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: OS3T2GGABRXYK6HNEXK4C5F5HTJFDEP7SHA5QV2R3SCMXDE2TD5A) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: S5NHKRILOZ47C4NXH4UXBG34ZF6IP2XHBR5JDDBNCBOZK26NALIA) context deadline exceeded; piecestore: (Node ID: 12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, Piece ID: CHJRUVVLWAR7627JOLKL2QHAK7EHW72ZTGE3BTOAAM4ZL5AZL3KQ) context deadline exceeded; piecestore: (Node ID: 12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, Piece ID: TLZVEBMQCCWINXB4AVRKHWKZ7WULCGUC6E5GGDJQLASKWNEHSJPA) context deadline exceeded; piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: LJKUWHGMINU2P7JTS2QN5VP25HPZBKRMFXRDBSO5OTVYTS47SB4A) context deadline exceeded; piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: VTE4WIBLUWV74XMSSPP37CEAWPRQHA3GFV2RJRQYGRQM5JGF3H3Q) context deadline exceeded; piecestore: (Node ID: 1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, Piece ID: 52Q5TEMXLZHLWJ5QWRFHGYWW6RCL2Y57AY6K2J7I3BUA22BNARQQ) context deadline exceeded ======LOGS====== 914003723 (300h58m34.376096196s) [upload=264873]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 02:29:49.215144679 (300h58m34.677237162s) [seg_pos={0 0}, upload=264873]: Begin upload segment... Oct 6 02:29:49.215212839 (300h58m34.677305312s) [seg_pos={0 0}, upload=264873]: Done begin upload segment. Oct 6 02:29:49.215241294 (300h58m34.677333777s) [noise=true , address=145.79.227.7:4294 , piece_id=TLZVEBMQCCWINXB4AVRKHWKZ7WULCGUC6E5GGDJQLASKWNEHSJPA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.215249739 (300h58m34.677342232s) [noise=true , address=45.91.116.4:4937 , piece_id=CHJRUVVLWAR7627JOLKL2QHAK7EHW72ZTGE3BTOAAM4ZL5AZL3KQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.215349820 (300h58m34.677442303s) [noise=true , address=77.90.31.18:4805 , piece_id=LJKUWHGMINU2P7JTS2QN5VP25HPZBKRMFXRDBSO5OTVYTS47SB4A, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.216239644 (300h58m34.678332127s) [noise=true , address=45.91.116.5:4298 , piece_id=VTE4WIBLUWV74XMSSPP37CEAWPRQHA3GFV2RJRQYGRQM5JGF3H3Q, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.218583898 (300h58m34.680676392s) [noise=true , address=45.91.116.10:4082 , piece_id=NX5W46AGWBSZ7EM2LZHZDPE6MEPSLZMNVFZEE2LQ5F5LV7YSQRKQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.219326442 (300h58m34.681418925s) [noise=true , address=45.91.116.6:4974 , piece_id=DBXKLTPEBNF5GVB4PC26BY4EPXOWHU2OWSLWGSHS52S6CDUZIMOA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.222198982 (300h58m34.684291465s) [noise=true , address=145.79.227.17:4972 , piece_id=I54FS5Q4CMMM46S6EP3C5MEUAJCVNWX23OHTBMNQNKP4TRERVZ7A, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.222520114 (300h58m34.684612597s) [noise=true , address=145.79.227.4:4185 , piece_id=ONU6YDQFAMJPRG3ZF45M7KUX5EGXZVQEGCTOJK3VGE2JXXLHCBAQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.225940076 (300h58m34.688032569s) [noise=true , address=77.90.31.13:4628 , piece_id=52Q5TEMXLZHLWJ5QWRFHGYWW6RCL2Y57AY6K2J7I3BUA22BNARQQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.227928832 (300h58m34.690021305s) [noise=true , address=145.79.227.16:4910 , piece_id=T254A3S7TTJEBOIBDDCS34DU7HV25TXDTNZWF2OGE4KRQ7YEA5ZA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.227998645 (300h58m34.690091128s) [noise=true , address=45.91.116.14:4190 , piece_id=OS3T2GGABRXYK6HNEXK4C5F5HTJFDEP7SHA5QV2R3SCMXDE2TD5A, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.231020309 (300h58m34.693112792s) [noise=true , address=145.79.227.18:4007 , piece_id=6PZE3N3QHDGV5K5HULQE5UJXWWG7RAIACP2OVEZVIOTTCWVNQO6Q, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.232908575 (300h58m34.695001068s) [noise=true , address=45.91.116.10:4081 , piece_id=PRCQUWMDPZTR2CSWMOODBECZOLAVJ4OPOUTGQHCPTWXUDQGY2MBQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.233037220 (300h58m34.695129693s) [noise=true , address=145.79.227.19:4079 , piece_id=2VBVLN2SNA6WUICKMNMNM7URPVUGWPB5SVZ335XZ7HIAUM3FY5VA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.234302058 (300h58m34.696394531s) [noise=true , address=45.91.116.7:4010 , piece_id=S5NHKRILOZ47C4NXH4UXBG34ZF6IP2XHBR5JDDBNCBOZK26NALIA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.235631800 (300h58m34.697724283s) [noise=true , address=77.90.31.15:4698 , piece_id=NOCO6KPTUBBKYCITDRFBKA3H5EJTKMDAO4AFQFFXC4PPH3D46X2A, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:50.988532254 (300h58m36.450624747s) [upload=264873]: Next returned nil segment Oct 6 02:29:50.988555589 (300h58m36.450648082s) [upload=264873]: Waiting for error group managing segments... Oct 6 02:29:51.282838267 (300h58m36.74493075s) [noise=true , address=45.91.116.4:4937 , piece_id=CHJRUVVLWAR7627JOLKL2QHAK7EHW72ZTGE3BTOAAM4ZL5AZL3KQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.318573943 (300h58m36.780666416s) [noise=true , address=77.90.31.18:4805 , piece_id=LJKUWHGMINU2P7JTS2QN5VP25HPZBKRMFXRDBSO5OTVYTS47SB4A, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.322112521 (300h58m36.784205004s) [noise=true , address=77.90.31.15:4698 , piece_id=NOCO6KPTUBBKYCITDRFBKA3H5EJTKMDAO4AFQFFXC4PPH3D46X2A, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.350536624 (300h58m36.812629108s) [noise=true , address=45.91.116.6:4974 , piece_id=DBXKLTPEBNF5GVB4PC26BY4EPXOWHU2OWSLWGSHS52S6CDUZIMOA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.362171679 (300h58m36.824264162s) [noise=true , address=45.91.116.10:4081 , piece_id=PRCQUWMDPZTR2CSWMOODBECZOLAVJ4OPOUTGQHCPTWXUDQGY2MBQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.366398738 (300h58m36.828491221s) [noise=true , address=45.91.116.10:4082 , piece_id=NX5W46AGWBSZ7EM2LZHZDPE6MEPSLZMNVFZEE2LQ5F5LV7YSQRKQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.375437720 (300h58m36.837530203s) [noise=true , address=45.91.116.14:4190 , piece_id=OS3T2GGABRXYK6HNEXK4C5F5HTJFDEP7SHA5QV2R3SCMXDE2TD5A, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.397600181 (300h58m36.859692664s) [noise=true , address=77.90.31.13:4628 , piece_id=52Q5TEMXLZHLWJ5QWRFHGYWW6RCL2Y57AY6K2J7I3BUA22BNARQQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.400132462 (300h58m36.862224945s) [noise=true , address=45.91.116.5:4298 , piece_id=VTE4WIBLUWV74XMSSPP37CEAWPRQHA3GFV2RJRQYGRQM5JGF3H3Q, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.412114018 (300h58m36.874206491s) [noise=true , address=45.91.116.7:4010 , piece_id=S5NHKRILOZ47C4NXH4UXBG34ZF6IP2XHBR5JDDBNCBOZK26NALIA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:30:14.302718308 (300h58m59.764810791s) [noise=true , address=145.79.227.18:4007 , piece_id=6PZE3N3QHDGV5K5HULQE5UJXWWG7RAIACP2OVEZVIOTTCWVNQO6Q, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:30:14.319938798 (300h58m59.782031281s) [noise=true , address=145.79.227.7:4294 , piece_id=TLZVEBMQCCWINXB4AVRKHWKZ7WULCGUC6E5GGDJQLASKWNEHSJPA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:30:16.104270571 (300h59m1.566363044s) [noise=true , address=145.79.227.16:4910 , piece_id=T254A3S7TTJEBOIBDDCS34DU7HV25TXDTNZWF2OGE4KRQ7YEA5ZA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:30:16.104295528 (300h59m1.566388012s) [seg_pos={0 0}, upload=264873]: Segment reached optimal threshold of 13 pieces. Oct 6 02:30:16.104442268 (300h59m1.566534751s) [noise=true , address=145.79.227.4:4185 , piece_id=ONU6YDQFAMJPRG3ZF45M7KUX5EGXZVQEGCTOJK3VGE2JXXLHCBAQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN): protocol: write tcp 188.166.225.170:34966->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:34966->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34966->145.79.227.4:4185: use of closed network connection Oct 6 02:30:16.104456395 (300h59m1.566548878s) [noise=true , address=145.79.227.4:4185 , piece_id=ONU6YDQFAMJPRG3ZF45M7KUX5EGXZVQEGCTOJK3VGE2JXXLHCBAQ, seg_pos={0 0}, upload=264873]: Long tail cancellation due to optimal threshold Oct 6 02:30:16.104598005 (300h59m1.566690478s) [noise=true , address=145.79.227.19:4079 , piece_id=2VBVLN2SNA6WUICKMNMNM7URPVUGWPB5SVZ335XZ7HIAUM3FY5VA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: write tcp 188.166.225.170:57162->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:57162->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57162->145.79.227.19:4079: use of closed network connection Oct 6 02:30:16.104604216 (300h59m1.566696699s) [noise=true , address=145.79.227.19:4079 , piece_id=2VBVLN2SNA6WUICKMNMNM7URPVUGWPB5SVZ335XZ7HIAUM3FY5VA, seg_pos={0 0}, upload=264873]: Long tail cancellation due to optimal threshold Oct 6 02:30:16.104672547 (300h59m1.56676503s) [noise=true , address=145.79.227.17:4972 , piece_id=I54FS5Q4CMMM46S6EP3C5MEUAJCVNWX23OHTBMNQNKP4TRERVZ7A, seg_pos={0 0}, upload=264873]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: write tcp 188.166.225.170:33630->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:33630->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33630->145.79.227.17:4972: use of closed network connection Oct 6 02:30:16.104678057 (300h59m1.56677053s) [noise=true , address=145.79.227.17:4972 , piece_id=I54FS5Q4CMMM46S6EP3C5MEUAJCVNWX23OHTBMNQNKP4TRERVZ7A, seg_pos={0 0}, upload=264873]: Long tail cancellation due to optimal threshold Oct 6 02:30:16.104695541 (300h59m1.566788024s) [seg_pos={0 0}, upload=264873]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 6 02:30:16.104762427 (300h59m1.56685491s) [upload=264873]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 6 02:30:16.104768159 (300h59m1.566860632s) [upload=264873]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 6 02:30:16.463894599 (300h59m1.925987092s) [upload=264873]: Done uploading segments... Oct 6 02:30:16.463937391 (300h59m1.926029874s) [upload=264873]: Upload finished. err: <nil> Duration:* 6.0879859s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 6.0560527s FAIL (2025-10-06 02:43:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, Piece ID: T254A3S7TTJEBOIBDDCS34DU7HV25TXDTNZWF2OGE4KRQ7YEA5ZA) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: OS3T2GGABRXYK6HNEXK4C5F5HTJFDEP7SHA5QV2R3SCMXDE2TD5A) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: S5NHKRILOZ47C4NXH4UXBG34ZF6IP2XHBR5JDDBNCBOZK26NALIA) context deadline exceeded; piecestore: (Node ID: 12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, Piece ID: CHJRUVVLWAR7627JOLKL2QHAK7EHW72ZTGE3BTOAAM4ZL5AZL3KQ) context deadline exceeded; piecestore: (Node ID: 12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, Piece ID: TLZVEBMQCCWINXB4AVRKHWKZ7WULCGUC6E5GGDJQLASKWNEHSJPA) context deadline exceeded; piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: LJKUWHGMINU2P7JTS2QN5VP25HPZBKRMFXRDBSO5OTVYTS47SB4A) context deadline exceeded; piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: VTE4WIBLUWV74XMSSPP37CEAWPRQHA3GFV2RJRQYGRQM5JGF3H3Q) context deadline exceeded; piecestore: (Node ID: 1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, Piece ID: 52Q5TEMXLZHLWJ5QWRFHGYWW6RCL2Y57AY6K2J7I3BUA22BNARQQ) context deadline exceeded ======LOGS====== 914003723 (300h58m34.376096196s) [upload=264873]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 02:29:49.215144679 (300h58m34.677237162s) [seg_pos={0 0}, upload=264873]: Begin upload segment... Oct 6 02:29:49.215212839 (300h58m34.677305312s) [seg_pos={0 0}, upload=264873]: Done begin upload segment. Oct 6 02:29:49.215241294 (300h58m34.677333777s) [noise=true , address=145.79.227.7:4294 , piece_id=TLZVEBMQCCWINXB4AVRKHWKZ7WULCGUC6E5GGDJQLASKWNEHSJPA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.215249739 (300h58m34.677342232s) [noise=true , address=45.91.116.4:4937 , piece_id=CHJRUVVLWAR7627JOLKL2QHAK7EHW72ZTGE3BTOAAM4ZL5AZL3KQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.215349820 (300h58m34.677442303s) [noise=true , address=77.90.31.18:4805 , piece_id=LJKUWHGMINU2P7JTS2QN5VP25HPZBKRMFXRDBSO5OTVYTS47SB4A, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.216239644 (300h58m34.678332127s) [noise=true , address=45.91.116.5:4298 , piece_id=VTE4WIBLUWV74XMSSPP37CEAWPRQHA3GFV2RJRQYGRQM5JGF3H3Q, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.218583898 (300h58m34.680676392s) [noise=true , address=45.91.116.10:4082 , piece_id=NX5W46AGWBSZ7EM2LZHZDPE6MEPSLZMNVFZEE2LQ5F5LV7YSQRKQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.219326442 (300h58m34.681418925s) [noise=true , address=45.91.116.6:4974 , piece_id=DBXKLTPEBNF5GVB4PC26BY4EPXOWHU2OWSLWGSHS52S6CDUZIMOA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.222198982 (300h58m34.684291465s) [noise=true , address=145.79.227.17:4972 , piece_id=I54FS5Q4CMMM46S6EP3C5MEUAJCVNWX23OHTBMNQNKP4TRERVZ7A, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.222520114 (300h58m34.684612597s) [noise=true , address=145.79.227.4:4185 , piece_id=ONU6YDQFAMJPRG3ZF45M7KUX5EGXZVQEGCTOJK3VGE2JXXLHCBAQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.225940076 (300h58m34.688032569s) [noise=true , address=77.90.31.13:4628 , piece_id=52Q5TEMXLZHLWJ5QWRFHGYWW6RCL2Y57AY6K2J7I3BUA22BNARQQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.227928832 (300h58m34.690021305s) [noise=true , address=145.79.227.16:4910 , piece_id=T254A3S7TTJEBOIBDDCS34DU7HV25TXDTNZWF2OGE4KRQ7YEA5ZA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.227998645 (300h58m34.690091128s) [noise=true , address=45.91.116.14:4190 , piece_id=OS3T2GGABRXYK6HNEXK4C5F5HTJFDEP7SHA5QV2R3SCMXDE2TD5A, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.231020309 (300h58m34.693112792s) [noise=true , address=145.79.227.18:4007 , piece_id=6PZE3N3QHDGV5K5HULQE5UJXWWG7RAIACP2OVEZVIOTTCWVNQO6Q, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.232908575 (300h58m34.695001068s) [noise=true , address=45.91.116.10:4081 , piece_id=PRCQUWMDPZTR2CSWMOODBECZOLAVJ4OPOUTGQHCPTWXUDQGY2MBQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.233037220 (300h58m34.695129693s) [noise=true , address=145.79.227.19:4079 , piece_id=2VBVLN2SNA6WUICKMNMNM7URPVUGWPB5SVZ335XZ7HIAUM3FY5VA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.234302058 (300h58m34.696394531s) [noise=true , address=45.91.116.7:4010 , piece_id=S5NHKRILOZ47C4NXH4UXBG34ZF6IP2XHBR5JDDBNCBOZK26NALIA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.235631800 (300h58m34.697724283s) [noise=true , address=77.90.31.15:4698 , piece_id=NOCO6KPTUBBKYCITDRFBKA3H5EJTKMDAO4AFQFFXC4PPH3D46X2A, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:50.988532254 (300h58m36.450624747s) [upload=264873]: Next returned nil segment Oct 6 02:29:50.988555589 (300h58m36.450648082s) [upload=264873]: Waiting for error group managing segments... Oct 6 02:29:51.282838267 (300h58m36.74493075s) [noise=true , address=45.91.116.4:4937 , piece_id=CHJRUVVLWAR7627JOLKL2QHAK7EHW72ZTGE3BTOAAM4ZL5AZL3KQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.318573943 (300h58m36.780666416s) [noise=true , address=77.90.31.18:4805 , piece_id=LJKUWHGMINU2P7JTS2QN5VP25HPZBKRMFXRDBSO5OTVYTS47SB4A, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.322112521 (300h58m36.784205004s) [noise=true , address=77.90.31.15:4698 , piece_id=NOCO6KPTUBBKYCITDRFBKA3H5EJTKMDAO4AFQFFXC4PPH3D46X2A, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.350536624 (300h58m36.812629108s) [noise=true , address=45.91.116.6:4974 , piece_id=DBXKLTPEBNF5GVB4PC26BY4EPXOWHU2OWSLWGSHS52S6CDUZIMOA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.362171679 (300h58m36.824264162s) [noise=true , address=45.91.116.10:4081 , piece_id=PRCQUWMDPZTR2CSWMOODBECZOLAVJ4OPOUTGQHCPTWXUDQGY2MBQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.366398738 (300h58m36.828491221s) [noise=true , address=45.91.116.10:4082 , piece_id=NX5W46AGWBSZ7EM2LZHZDPE6MEPSLZMNVFZEE2LQ5F5LV7YSQRKQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.375437720 (300h58m36.837530203s) [noise=true , address=45.91.116.14:4190 , piece_id=OS3T2GGABRXYK6HNEXK4C5F5HTJFDEP7SHA5QV2R3SCMXDE2TD5A, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.397600181 (300h58m36.859692664s) [noise=true , address=77.90.31.13:4628 , piece_id=52Q5TEMXLZHLWJ5QWRFHGYWW6RCL2Y57AY6K2J7I3BUA22BNARQQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.400132462 (300h58m36.862224945s) [noise=true , address=45.91.116.5:4298 , piece_id=VTE4WIBLUWV74XMSSPP37CEAWPRQHA3GFV2RJRQYGRQM5JGF3H3Q, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.412114018 (300h58m36.874206491s) [noise=true , address=45.91.116.7:4010 , piece_id=S5NHKRILOZ47C4NXH4UXBG34ZF6IP2XHBR5JDDBNCBOZK26NALIA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:30:14.302718308 (300h58m59.764810791s) [noise=true , address=145.79.227.18:4007 , piece_id=6PZE3N3QHDGV5K5HULQE5UJXWWG7RAIACP2OVEZVIOTTCWVNQO6Q, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:30:14.319938798 (300h58m59.782031281s) [noise=true , address=145.79.227.7:4294 , piece_id=TLZVEBMQCCWINXB4AVRKHWKZ7WULCGUC6E5GGDJQLASKWNEHSJPA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:30:16.104270571 (300h59m1.566363044s) [noise=true , address=145.79.227.16:4910 , piece_id=T254A3S7TTJEBOIBDDCS34DU7HV25TXDTNZWF2OGE4KRQ7YEA5ZA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:30:16.104295528 (300h59m1.566388012s) [seg_pos={0 0}, upload=264873]: Segment reached optimal threshold of 13 pieces. Oct 6 02:30:16.104442268 (300h59m1.566534751s) [noise=true , address=145.79.227.4:4185 , piece_id=ONU6YDQFAMJPRG3ZF45M7KUX5EGXZVQEGCTOJK3VGE2JXXLHCBAQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN): protocol: write tcp 188.166.225.170:34966->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:34966->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34966->145.79.227.4:4185: use of closed network connection Oct 6 02:30:16.104456395 (300h59m1.566548878s) [noise=true , address=145.79.227.4:4185 , piece_id=ONU6YDQFAMJPRG3ZF45M7KUX5EGXZVQEGCTOJK3VGE2JXXLHCBAQ, seg_pos={0 0}, upload=264873]: Long tail cancellation due to optimal threshold Oct 6 02:30:16.104598005 (300h59m1.566690478s) [noise=true , address=145.79.227.19:4079 , piece_id=2VBVLN2SNA6WUICKMNMNM7URPVUGWPB5SVZ335XZ7HIAUM3FY5VA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: write tcp 188.166.225.170:57162->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:57162->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57162->145.79.227.19:4079: use of closed network connection Oct 6 02:30:16.104604216 (300h59m1.566696699s) [noise=true , address=145.79.227.19:4079 , piece_id=2VBVLN2SNA6WUICKMNMNM7URPVUGWPB5SVZ335XZ7HIAUM3FY5VA, seg_pos={0 0}, upload=264873]: Long tail cancellation due to optimal threshold Oct 6 02:30:16.104672547 (300h59m1.56676503s) [noise=true , address=145.79.227.17:4972 , piece_id=I54FS5Q4CMMM46S6EP3C5MEUAJCVNWX23OHTBMNQNKP4TRERVZ7A, seg_pos={0 0}, upload=264873]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: write tcp 188.166.225.170:33630->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:33630->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33630->145.79.227.17:4972: use of closed network connection Oct 6 02:30:16.104678057 (300h59m1.56677053s) [noise=true , address=145.79.227.17:4972 , piece_id=I54FS5Q4CMMM46S6EP3C5MEUAJCVNWX23OHTBMNQNKP4TRERVZ7A, seg_pos={0 0}, upload=264873]: Long tail cancellation due to optimal threshold Oct 6 02:30:16.104695541 (300h59m1.566788024s) [seg_pos={0 0}, upload=264873]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 6 02:30:16.104762427 (300h59m1.56685491s) [upload=264873]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 6 02:30:16.104768159 (300h59m1.566860632s) [upload=264873]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 6 02:30:16.463894599 (300h59m1.925987092s) [upload=264873]: Done uploading segments... Oct 6 02:30:16.463937391 (300h59m1.926029874s) [upload=264873]: Upload finished. err: <nil> Duration:* 6.056083651s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 6.048916312s FAIL (2025-10-06 02:42:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, Piece ID: T254A3S7TTJEBOIBDDCS34DU7HV25TXDTNZWF2OGE4KRQ7YEA5ZA) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: OS3T2GGABRXYK6HNEXK4C5F5HTJFDEP7SHA5QV2R3SCMXDE2TD5A) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: S5NHKRILOZ47C4NXH4UXBG34ZF6IP2XHBR5JDDBNCBOZK26NALIA) context deadline exceeded; piecestore: (Node ID: 12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq, Piece ID: CHJRUVVLWAR7627JOLKL2QHAK7EHW72ZTGE3BTOAAM4ZL5AZL3KQ) context deadline exceeded; piecestore: (Node ID: 12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, Piece ID: TLZVEBMQCCWINXB4AVRKHWKZ7WULCGUC6E5GGDJQLASKWNEHSJPA) context deadline exceeded; piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: LJKUWHGMINU2P7JTS2QN5VP25HPZBKRMFXRDBSO5OTVYTS47SB4A) context deadline exceeded; piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: VTE4WIBLUWV74XMSSPP37CEAWPRQHA3GFV2RJRQYGRQM5JGF3H3Q) context deadline exceeded; piecestore: (Node ID: 1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, Piece ID: 52Q5TEMXLZHLWJ5QWRFHGYWW6RCL2Y57AY6K2J7I3BUA22BNARQQ) context deadline exceeded ======LOGS====== 914003723 (300h58m34.376096196s) [upload=264873]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 02:29:49.215144679 (300h58m34.677237162s) [seg_pos={0 0}, upload=264873]: Begin upload segment... Oct 6 02:29:49.215212839 (300h58m34.677305312s) [seg_pos={0 0}, upload=264873]: Done begin upload segment. Oct 6 02:29:49.215241294 (300h58m34.677333777s) [noise=true , address=145.79.227.7:4294 , piece_id=TLZVEBMQCCWINXB4AVRKHWKZ7WULCGUC6E5GGDJQLASKWNEHSJPA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.215249739 (300h58m34.677342232s) [noise=true , address=45.91.116.4:4937 , piece_id=CHJRUVVLWAR7627JOLKL2QHAK7EHW72ZTGE3BTOAAM4ZL5AZL3KQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.215349820 (300h58m34.677442303s) [noise=true , address=77.90.31.18:4805 , piece_id=LJKUWHGMINU2P7JTS2QN5VP25HPZBKRMFXRDBSO5OTVYTS47SB4A, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.216239644 (300h58m34.678332127s) [noise=true , address=45.91.116.5:4298 , piece_id=VTE4WIBLUWV74XMSSPP37CEAWPRQHA3GFV2RJRQYGRQM5JGF3H3Q, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.218583898 (300h58m34.680676392s) [noise=true , address=45.91.116.10:4082 , piece_id=NX5W46AGWBSZ7EM2LZHZDPE6MEPSLZMNVFZEE2LQ5F5LV7YSQRKQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.219326442 (300h58m34.681418925s) [noise=true , address=45.91.116.6:4974 , piece_id=DBXKLTPEBNF5GVB4PC26BY4EPXOWHU2OWSLWGSHS52S6CDUZIMOA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.222198982 (300h58m34.684291465s) [noise=true , address=145.79.227.17:4972 , piece_id=I54FS5Q4CMMM46S6EP3C5MEUAJCVNWX23OHTBMNQNKP4TRERVZ7A, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.222520114 (300h58m34.684612597s) [noise=true , address=145.79.227.4:4185 , piece_id=ONU6YDQFAMJPRG3ZF45M7KUX5EGXZVQEGCTOJK3VGE2JXXLHCBAQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.225940076 (300h58m34.688032569s) [noise=true , address=77.90.31.13:4628 , piece_id=52Q5TEMXLZHLWJ5QWRFHGYWW6RCL2Y57AY6K2J7I3BUA22BNARQQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.227928832 (300h58m34.690021305s) [noise=true , address=145.79.227.16:4910 , piece_id=T254A3S7TTJEBOIBDDCS34DU7HV25TXDTNZWF2OGE4KRQ7YEA5ZA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.227998645 (300h58m34.690091128s) [noise=true , address=45.91.116.14:4190 , piece_id=OS3T2GGABRXYK6HNEXK4C5F5HTJFDEP7SHA5QV2R3SCMXDE2TD5A, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.231020309 (300h58m34.693112792s) [noise=true , address=145.79.227.18:4007 , piece_id=6PZE3N3QHDGV5K5HULQE5UJXWWG7RAIACP2OVEZVIOTTCWVNQO6Q, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.232908575 (300h58m34.695001068s) [noise=true , address=45.91.116.10:4081 , piece_id=PRCQUWMDPZTR2CSWMOODBECZOLAVJ4OPOUTGQHCPTWXUDQGY2MBQ, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.233037220 (300h58m34.695129693s) [noise=true , address=145.79.227.19:4079 , piece_id=2VBVLN2SNA6WUICKMNMNM7URPVUGWPB5SVZ335XZ7HIAUM3FY5VA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.234302058 (300h58m34.696394531s) [noise=true , address=45.91.116.7:4010 , piece_id=S5NHKRILOZ47C4NXH4UXBG34ZF6IP2XHBR5JDDBNCBOZK26NALIA, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:49.235631800 (300h58m34.697724283s) [noise=true , address=77.90.31.15:4698 , piece_id=NOCO6KPTUBBKYCITDRFBKA3H5EJTKMDAO4AFQFFXC4PPH3D46X2A, seg_pos={0 0}, upload=264873]: Uploading piece... Oct 6 02:29:50.988532254 (300h58m36.450624747s) [upload=264873]: Next returned nil segment Oct 6 02:29:50.988555589 (300h58m36.450648082s) [upload=264873]: Waiting for error group managing segments... Oct 6 02:29:51.282838267 (300h58m36.74493075s) [noise=true , address=45.91.116.4:4937 , piece_id=CHJRUVVLWAR7627JOLKL2QHAK7EHW72ZTGE3BTOAAM4ZL5AZL3KQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.318573943 (300h58m36.780666416s) [noise=true , address=77.90.31.18:4805 , piece_id=LJKUWHGMINU2P7JTS2QN5VP25HPZBKRMFXRDBSO5OTVYTS47SB4A, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.322112521 (300h58m36.784205004s) [noise=true , address=77.90.31.15:4698 , piece_id=NOCO6KPTUBBKYCITDRFBKA3H5EJTKMDAO4AFQFFXC4PPH3D46X2A, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.350536624 (300h58m36.812629108s) [noise=true , address=45.91.116.6:4974 , piece_id=DBXKLTPEBNF5GVB4PC26BY4EPXOWHU2OWSLWGSHS52S6CDUZIMOA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.362171679 (300h58m36.824264162s) [noise=true , address=45.91.116.10:4081 , piece_id=PRCQUWMDPZTR2CSWMOODBECZOLAVJ4OPOUTGQHCPTWXUDQGY2MBQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.366398738 (300h58m36.828491221s) [noise=true , address=45.91.116.10:4082 , piece_id=NX5W46AGWBSZ7EM2LZHZDPE6MEPSLZMNVFZEE2LQ5F5LV7YSQRKQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.375437720 (300h58m36.837530203s) [noise=true , address=45.91.116.14:4190 , piece_id=OS3T2GGABRXYK6HNEXK4C5F5HTJFDEP7SHA5QV2R3SCMXDE2TD5A, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.397600181 (300h58m36.859692664s) [noise=true , address=77.90.31.13:4628 , piece_id=52Q5TEMXLZHLWJ5QWRFHGYWW6RCL2Y57AY6K2J7I3BUA22BNARQQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.400132462 (300h58m36.862224945s) [noise=true , address=45.91.116.5:4298 , piece_id=VTE4WIBLUWV74XMSSPP37CEAWPRQHA3GFV2RJRQYGRQM5JGF3H3Q, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:29:51.412114018 (300h58m36.874206491s) [noise=true , address=45.91.116.7:4010 , piece_id=S5NHKRILOZ47C4NXH4UXBG34ZF6IP2XHBR5JDDBNCBOZK26NALIA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:30:14.302718308 (300h58m59.764810791s) [noise=true , address=145.79.227.18:4007 , piece_id=6PZE3N3QHDGV5K5HULQE5UJXWWG7RAIACP2OVEZVIOTTCWVNQO6Q, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:30:14.319938798 (300h58m59.782031281s) [noise=true , address=145.79.227.7:4294 , piece_id=TLZVEBMQCCWINXB4AVRKHWKZ7WULCGUC6E5GGDJQLASKWNEHSJPA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:30:16.104270571 (300h59m1.566363044s) [noise=true , address=145.79.227.16:4910 , piece_id=T254A3S7TTJEBOIBDDCS34DU7HV25TXDTNZWF2OGE4KRQ7YEA5ZA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: <nil> Oct 6 02:30:16.104295528 (300h59m1.566388012s) [seg_pos={0 0}, upload=264873]: Segment reached optimal threshold of 13 pieces. Oct 6 02:30:16.104442268 (300h59m1.566534751s) [noise=true , address=145.79.227.4:4185 , piece_id=ONU6YDQFAMJPRG3ZF45M7KUX5EGXZVQEGCTOJK3VGE2JXXLHCBAQ, seg_pos={0 0}, upload=264873]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN): protocol: write tcp 188.166.225.170:34966->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:34966->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34966->145.79.227.4:4185: use of closed network connection Oct 6 02:30:16.104456395 (300h59m1.566548878s) [noise=true , address=145.79.227.4:4185 , piece_id=ONU6YDQFAMJPRG3ZF45M7KUX5EGXZVQEGCTOJK3VGE2JXXLHCBAQ, seg_pos={0 0}, upload=264873]: Long tail cancellation due to optimal threshold Oct 6 02:30:16.104598005 (300h59m1.566690478s) [noise=true , address=145.79.227.19:4079 , piece_id=2VBVLN2SNA6WUICKMNMNM7URPVUGWPB5SVZ335XZ7HIAUM3FY5VA, seg_pos={0 0}, upload=264873]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: write tcp 188.166.225.170:57162->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:57162->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57162->145.79.227.19:4079: use of closed network connection Oct 6 02:30:16.104604216 (300h59m1.566696699s) [noise=true , address=145.79.227.19:4079 , piece_id=2VBVLN2SNA6WUICKMNMNM7URPVUGWPB5SVZ335XZ7HIAUM3FY5VA, seg_pos={0 0}, upload=264873]: Long tail cancellation due to optimal threshold Oct 6 02:30:16.104672547 (300h59m1.56676503s) [noise=true , address=145.79.227.17:4972 , piece_id=I54FS5Q4CMMM46S6EP3C5MEUAJCVNWX23OHTBMNQNKP4TRERVZ7A, seg_pos={0 0}, upload=264873]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: write tcp 188.166.225.170:33630->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:33630->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33630->145.79.227.17:4972: use of closed network connection Oct 6 02:30:16.104678057 (300h59m1.56677053s) [noise=true , address=145.79.227.17:4972 , piece_id=I54FS5Q4CMMM46S6EP3C5MEUAJCVNWX23OHTBMNQNKP4TRERVZ7A, seg_pos={0 0}, upload=264873]: Long tail cancellation due to optimal threshold Oct 6 02:30:16.104695541 (300h59m1.566788024s) [seg_pos={0 0}, upload=264873]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 6 02:30:16.104762427 (300h59m1.56685491s) [upload=264873]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 6 02:30:16.104768159 (300h59m1.566860632s) [upload=264873]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 6 02:30:16.463894599 (300h59m1.925987092s) [upload=264873]: Done uploading segments... Oct 6 02:30:16.463937391 (300h59m1.926029874s) [upload=264873]: Upload finished. err: <nil> Duration:* 6.048948483s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)