Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 32 out of 60 is failed (threshold=6) [X.X.XX.X.XX..X..XX.XXXX.XXXX...XXX...XXX..XX.X..X.X.X..X..X.] duration: 5.002162025s trace: https://jaeger.dev.storj.io/trace/39e5ac5a0dfb8461 FAIL (2025-12-29 12:08:26 UTC) orbiter warning: context deadline exceeded Duration: 5.002188576s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [....X..X....................................................] duration: 35.480546132s trace: https://jaeger.dev.storj.io/trace/7e7354d8023608a8 FAIL (2025-12-29 12:07:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== ) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480097317 (571h53m39.351180352s) [noise=true , address=77.90.31.21:4889 , piece_id=BR6Z55P3A7YJ5FJRDSZFUIF66FGIJQY7WCEQ7PL5DOYWL73OMKTQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480405053 (571h53m39.351488098s) [noise=true , address=77.90.31.16:4750 , piece_id=MVVZPJEGDCBSNVOB6CD2WUZGIJTL3VBHX2G4L3TVZ6JF5ZEJ5U6Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480712969 (571h53m39.351796014s) [noise=true , address=77.90.31.18:4834 , piece_id=BTWNGNQVEUEVV6HLBDM63SGNTKV4RDU4J4XFLG52WHKP53H3L5KA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481019453 (571h53m39.352102488s) [noise=true , address=145.79.227.30:4777 , piece_id=VSCXLXGIBSFE6E4OHLWNZIZ75PO7WQO2VFHUYHW7JUNH62KGGU7Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481029532 (571h53m39.352112567s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481559832 (571h53m39.352642877s) [noise=true , address=145.79.227.35:4515 , piece_id=L7LFZKZEZERUNX4HUNDNQ2PWHXNJRJ7OBFRNMFIICPZEUIBHGP7A, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481731919 (571h53m39.352814954s) [noise=true , address=64.49.13.26:4953 , piece_id=4YSW5Y3A6HYG3XJ2XAB7RGVHAOHMP7G3QIZYMYTSXVXYJ3ANLNXA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482078329 (571h53m39.353161364s) [noise=true , address=145.79.227.44:4205 , piece_id=UE4QSHDQBZ3XBWQ6I6MCRVTLMU7U6W6OZDA2ZGQCH6ZYRAGPD7WA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482099830 (571h53m39.353182865s) [noise=true , address=145.79.227.40:4067 , piece_id=WSYDRX5YJ6OCFW52ZVMM6WZYNRFUXMR4XJ6C5727BOFMR4RRH33Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482434487 (571h53m39.353517532s) [noise=true , address=145.79.227.3:4164 , piece_id=U6F7VDPOH7K6R5EH5UUXHFP55GXOEXO5BLBJHZRTITPDQI4LP7KQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482480845 (571h53m39.35356388s) [noise=true , address=64.49.13.140:4516 , piece_id=4GUYOYOAEASUK2W25S3NILWIIIHGOZX76L4EGFLUUUEDWRALPP2A, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482841442 (571h53m39.353924477s) [noise=true , address=145.79.227.34:4910 , piece_id=ZH4DCNXET2Q6VRAUH3NQ3KKSV4OYI575X7YS72XD236EEOMOG3LA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483163645 (571h53m39.35424668s) [noise=true , address=145.79.227.29:4974 , piece_id=KEKREOD2B35XQNI45JBHE2SZZC3MQQBQUR5I6WFDAQYJQGJBXSMA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483532478 (571h53m39.354615513s) [noise=true , address=77.90.31.17:4769 , piece_id=3MKUMOQIUVERKQLLIFX52XQPTQ5LSFMHIPD5YW2YTO5YEFR36IYA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483914596 (571h53m39.354997631s) [noise=true , address=145.79.227.13:4807 , piece_id=VKXB5MWAG2YYH2AWWJEGTLBZ2QUPLI25VTHI6AMMALSM23Z3AWZQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:40.761721106 (571h53m40.632804141s) [upload=501384]: Next returned nil segment Dec 29 11:22:40.761733349 (571h53m40.632816384s) [upload=501384]: Waiting for error group managing segments... Dec 29 11:22:41.121222862 (571h53m40.992305897s) [noise=true , address=64.49.13.140:4516 , piece_id=4GUYOYOAEASUK2W25S3NILWIIIHGOZX76L4EGFLUUUEDWRALPP2A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.197311646 (571h53m41.068394691s) [noise=true , address=64.49.13.26:4953 , piece_id=4YSW5Y3A6HYG3XJ2XAB7RGVHAOHMP7G3QIZYMYTSXVXYJ3ANLNXA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.347247568 (571h53m41.218330613s) [noise=true , address=145.79.227.13:4807 , piece_id=VKXB5MWAG2YYH2AWWJEGTLBZ2QUPLI25VTHI6AMMALSM23Z3AWZQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.347917313 (571h53m41.219000348s) [noise=true , address=145.79.227.3:4164 , piece_id=U6F7VDPOH7K6R5EH5UUXHFP55GXOEXO5BLBJHZRTITPDQI4LP7KQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.348510613 (571h53m41.219593648s) [noise=true , address=145.79.227.29:4974 , piece_id=KEKREOD2B35XQNI45JBHE2SZZC3MQQBQUR5I6WFDAQYJQGJBXSMA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.503950985 (571h53m41.37503402s) [noise=true , address=77.90.31.17:4769 , piece_id=3MKUMOQIUVERKQLLIFX52XQPTQ5LSFMHIPD5YW2YTO5YEFR36IYA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.561352025 (571h53m41.43243506s) [noise=true , address=77.90.31.18:4834 , piece_id=BTWNGNQVEUEVV6HLBDM63SGNTKV4RDU4J4XFLG52WHKP53H3L5KA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.622447380 (571h53m41.493530425s) [noise=true , address=77.90.31.16:4750 , piece_id=MVVZPJEGDCBSNVOB6CD2WUZGIJTL3VBHX2G4L3TVZ6JF5ZEJ5U6Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.645494685 (571h53m41.51657773s) [noise=true , address=77.90.31.21:4889 , piece_id=BR6Z55P3A7YJ5FJRDSZFUIF66FGIJQY7WCEQ7PL5DOYWL73OMKTQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.247930045 (571h53m42.11901308s) [noise=true , address=145.79.227.30:4777 , piece_id=VSCXLXGIBSFE6E4OHLWNZIZ75PO7WQO2VFHUYHW7JUNH62KGGU7Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.262221890 (571h53m42.133304925s) [noise=true , address=145.79.227.44:4205 , piece_id=UE4QSHDQBZ3XBWQ6I6MCRVTLMU7U6W6OZDA2ZGQCH6ZYRAGPD7WA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.295598123 (571h53m42.166681158s) [noise=true , address=145.79.227.35:4515 , piece_id=L7LFZKZEZERUNX4HUNDNQ2PWHXNJRJ7OBFRNMFIICPZEUIBHGP7A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.307481351 (571h53m42.178564386s) [noise=true , address=145.79.227.34:4910 , piece_id=ZH4DCNXET2Q6VRAUH3NQ3KKSV4OYI575X7YS72XD236EEOMOG3LA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.325775906 (571h53m42.196858941s) [noise=true , address=145.79.227.40:4067 , piece_id=WSYDRX5YJ6OCFW52ZVMM6WZYNRFUXMR4XJ6C5727BOFMR4RRH33Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.325865978 (571h53m42.196949023s) [seg_pos={0 0}, upload=501384]: Segment reached optimal threshold of 14 pieces. Dec 29 11:22:42.326137074 (571h53m42.197220109s) [noise=true , address=64.49.13.169:4708 , piece_id=B2X2RWRELXWJPF6AVJJZCNU46LM5ASGRE4EWGPKL3SCLQ4VKUBDQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection Dec 29 11:22:42.326153074 (571h53m42.197236109s) [noise=true , address=64.49.13.169:4708 , piece_id=B2X2RWRELXWJPF6AVJJZCNU46LM5ASGRE4EWGPKL3SCLQ4VKUBDQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326193401 (571h53m42.197276446s) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12h5esfg6mw2c8y3AVpeFBkL5FfLAwYdjhcnQ9CJ8fjh61WLYjx): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326205705 (571h53m42.19728874s) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326407739 (571h53m42.197490784s) [noise=true , address=77.90.31.10:4524 , piece_id=Q2JGEUV5EDFTVY2CSNPZOVJIIIAFBA3IHA7S5N7YZRH3E7TSTE5A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uN7CznoK4AD8FqFkNUndawdGJ9rmjAe7oNnjJkV844ScY87qe): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326415374 (571h53m42.197498419s) [noise=true , address=77.90.31.10:4524 , piece_id=Q2JGEUV5EDFTVY2CSNPZOVJIIIAFBA3IHA7S5N7YZRH3E7TSTE5A, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326505105 (571h53m42.19758815s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tnTEYvqJ6PB2DsuXA28sEuEiGj11StA4qdhcYH1kuoaMkTVdY): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326512188 (571h53m42.197595223s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326532357 (571h53m42.197615392s) [seg_pos={0 0}, upload=501384]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 11:22:42.326671562 (571h53m42.197754597s) [upload=501384]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 11:22:42.326677843 (571h53m42.197760879s) [upload=501384]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 11:22:42.610391438 (571h53m42.481474493s) [upload=501384]: Done uploading segments... Dec 29 11:22:42.610416826 (571h53m42.481499871s) [upload=501384]: Upload finished. err: <nil> Duration: 35.480562191s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 32 out of 60 is failed (threshold=6) [.X.X.XX.X.XX..X..XX.XXXX.XXXX...XXX...XXX..XX.X..X.X.X..X..X] duration: 5.001346484s trace: https://jaeger.dev.storj.io/trace/39e5ac5a0dfb8461 FAIL (2025-12-29 12:07:56 UTC) orbiter warning: context deadline exceeded Duration: 5.001368946s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 27 out of 60 is failed (threshold=3) [XXX.X.......X.X..X.XXX...XX.XXX.X.X....XXXXXX..X.X.......X.X] duration: 30.002078402s trace: https://jaeger.dev.storj.io/trace/4c97c679614316eb FAIL (2025-12-29 12:07:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: PMGS2FWBT5FDZYNHIYO4RDJ7FBBDRZQVCSLFBPCJLROPKQE2OW3Q) context deadline exceeded; piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: NAJUSCYEUF4BAEFGHB4D5UG26VIEDDWX7AYPK5VCNSQFEKQLL25A) context deadline exceeded ======LOGS====== Dec 29 12:07:02.340851259 (572h38m2.211934304s) [upload=502026]: Starting upload Dec 29 12:07:02.340930660 (572h38m2.212013725s) [upload=502026]: Done starting upload Dec 29 12:07:02.341349728 (572h38m2.212432773s) [upload=502026]: Uploading segments... Dec 29 12:07:02.341413470 (572h38m2.212496525s) [upload=502026]: Got next segment. Inline: false Dec 29 12:07:02.341464217 (572h38m2.212547262s) [upload=502026]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 12:07:02.341470819 (572h38m2.212553864s) [upload=502026]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 12:07:02.596288162 (572h38m2.467371208s) [seg_pos={0 0}, upload=502026]: Begin upload segment... Dec 29 12:07:02.596350921 (572h38m2.467433966s) [seg_pos={0 0}, upload=502026]: Done begin upload segment. Dec 29 12:07:02.596384024 (572h38m2.467467069s) [noise=true , address=145.79.227.52:4445 , piece_id=GQ4BICIFZEO5NP6FZLCKXECYTSFIRLPNDRAMLK32DBF3SIOMV6VA, seg_pos={0 0}, upload=502026]: Uploading piece... Dec 29 12:07:02.596429081 (572h38m2.467512126s) [noise=true , address=45.91.116.12:4225 , piece_id=TD5ALXKV4GGJXMPBW4IUGPLIKZSW2PDQ6SHXUB4XHBKXFV4IIZXQ, seg_pos={0 0}, upload=502026]: Uploading piece... Dec 29 12:07:02.596952367 (572h38m2.468035412s) [noise=true , address=45.91.116.14:4192 , piece_id=H47E3USGQG3LZ3LMKOK7NQRSXTMOH5Z26DOVVMIEBPCQMXAJBBKQ, seg_pos={0 0}, upload=502026]: Uploading piece... Dec 29 12:07:02.597501583 (572h38m2.468584628s) [noise=true , address=145.79.227.46:4601 , piece_id=KE7HSW5QFFK47TZVGKHHIZ5WQL5IWU4E7BCRO3ODZ7KXETOIQXNA, seg_pos={0 0}, upload=502026]: Uploading piece... Dec 29 12:07:02.597999601 (572h38m2.469082646s) [noise=true , address=145.79.227.30:4761 , piece_id=OCSM2D6ZIUIOBHMVZ4IJ264YNU4T2Y236FGHXCKN74EOCOHFS7HQ, seg_pos={0 0}, upload=502026]: Uploading piece... Dec 29 12:07:02.598446943 (572h38m2.469529988s) [noise=true , address=145.79.227.28:4726 , piece_id=WLDTJSDJMTAHZ6VOWNFBTIHJMGF7YLG2TLNISH7TOQG3LP4RNKOA, seg_pos={0 0}, upload=502026]: Uploading piece... Dec 29 12:07:02.598926466 (572h38m2.470009512s) [noise=true , address=45.91.116.9:4262 , piece_id=NAJUSCYEUF4BAEFGHB4D5UG26VIEDDWX7AYPK5VCNSQFEKQLL25A, seg_pos={0 0}, upload=502026]: Uploading piece... Dec 29 12:07:02.599488466 (572h38m2.470571512s) [noise=true , address=145.79.227.19:4077 , piece_id=HO6ZWRGGJ36YAQXCQIRNI4KLNYEDGEQ6SYXT6JY6SBDVIKHZLZUA, seg_pos={0 0}, upload=502026]: Uploading piece... Dec 29 12:07:02.600053873 (572h38m2.471136918s) [noise=true , address=145.79.227.2:4114 , piece_id=3CM6USYYMXQOA7DFVZGSVZXUWXTM7V4EITS673TMR7OCK4UFXHGA, seg_pos={0 0}, upload=502026]: Uploading piece... Dec 29 12:07:02.601083734 (572h38m2.472166779s) [noise=true , address=77.90.31.10:4519 , piece_id=PMGS2FWBT5FDZYNHIYO4RDJ7FBBDRZQVCSLFBPCJLROPKQE2OW3Q, seg_pos={0 0}, upload=502026]: Uploading piece... Dec 29 12:07:02.602480524 (572h38m2.473563569s) [noise=true , address=45.91.116.13:4154 , piece_id=OIC65IMOMTBAXXDPHSCXTOA7WZT5BRZDEUN5NY3ZAO2JBURPNO2A, seg_pos={0 0}, upload=502026]: Uploading piece... Dec 29 12:07:02.603711307 (572h38m2.474794362s) [noise=true , address=145.79.227.4:4185 , piece_id=BO6SEMS2RX7CBRGLBFHG5AT7JLXF2HMTLQXFXFHJWCNUG4CJCU4A, seg_pos={0 0}, upload=502026]: Uploading piece... Dec 29 12:07:02.605222756 (572h38m2.476305801s) [noise=true , address=145.79.227.37:4566 , piece_id=CV6XOIXXA243KNYH2PD4MW2QMOG63GAHV3W6L3QV43RG5RBWQCGA, seg_pos={0 0}, upload=502026]: Uploading piece... Dec 29 12:07:02.606728904 (572h38m2.477811939s) [noise=true , address=145.79.227.44:4193 , piece_id=P3YSDY4ICXXXKT32RVK5PKQ4F6UBRF7UE56HHM7JN2ADMI2CCBVA, seg_pos={0 0}, upload=502026]: Uploading piece... Dec 29 12:07:02.608077452 (572h38m2.479160487s) [noise=true , address=145.79.227.38:4979 , piece_id=QSGBFNQWNAOK2VEWJQAFOW2U5ABAQRB4CDLHHBMIBU5ROUCQ5FHA, seg_pos={0 0}, upload=502026]: Uploading piece... Dec 29 12:07:02.609471316 (572h38m2.480554371s) [noise=true , address=145.79.227.12:4765 , piece_id=AINDDPLEMFD6OMRJKOZ2RN52OAGYYECA5AAJKGRY5TH2TBJTF2KA, seg_pos={0 0}, upload=502026]: Uploading piece... Dec 29 12:07:04.290216954 (572h38m4.161300029s) [upload=502026]: Next returned nil segment Dec 29 12:07:04.290242753 (572h38m4.161325819s) [upload=502026]: Waiting for error group managing segments... Dec 29 12:07:04.575742372 (572h38m4.446825408s) [noise=true , address=145.79.227.46:4601 , piece_id=KE7HSW5QFFK47TZVGKHHIZ5WQL5IWU4E7BCRO3ODZ7KXETOIQXNA, seg_pos={0 0}, upload=502026]: Done uploading piece. err: <nil> Dec 29 12:07:04.577076322 (572h38m4.448159367s) [noise=true , address=145.79.227.19:4077 , piece_id=HO6ZWRGGJ36YAQXCQIRNI4KLNYEDGEQ6SYXT6JY6SBDVIKHZLZUA, seg_pos={0 0}, upload=502026]: Done uploading piece. err: <nil> Dec 29 12:07:04.583899906 (572h38m4.454982952s) [noise=true , address=145.79.227.28:4726 , piece_id=WLDTJSDJMTAHZ6VOWNFBTIHJMGF7YLG2TLNISH7TOQG3LP4RNKOA, seg_pos={0 0}, upload=502026]: Done uploading piece. err: <nil> Dec 29 12:07:04.592820953 (572h38m4.463903998s) [noise=true , address=145.79.227.52:4445 , piece_id=GQ4BICIFZEO5NP6FZLCKXECYTSFIRLPNDRAMLK32DBF3SIOMV6VA, seg_pos={0 0}, upload=502026]: Done uploading piece. err: <nil> Dec 29 12:07:04.600198723 (572h38m4.471281768s) [noise=true , address=145.79.227.12:4765 , piece_id=AINDDPLEMFD6OMRJKOZ2RN52OAGYYECA5AAJKGRY5TH2TBJTF2KA, seg_pos={0 0}, upload=502026]: Done uploading piece. err: <nil> Dec 29 12:07:04.602984047 (572h38m4.474067092s) [noise=true , address=145.79.227.4:4185 , piece_id=BO6SEMS2RX7CBRGLBFHG5AT7JLXF2HMTLQXFXFHJWCNUG4CJCU4A, seg_pos={0 0}, upload=502026]: Done uploading piece. err: <nil> Dec 29 12:07:04.606202135 (572h38m4.47728518s) [noise=true , address=145.79.227.2:4114 , piece_id=3CM6USYYMXQOA7DFVZGSVZXUWXTM7V4EITS673TMR7OCK4UFXHGA, seg_pos={0 0}, upload=502026]: Done uploading piece. err: <nil> Dec 29 12:07:04.608901495 (572h38m4.47998454s) [noise=true , address=145.79.227.38:4979 , piece_id=QSGBFNQWNAOK2VEWJQAFOW2U5ABAQRB4CDLHHBMIBU5ROUCQ5FHA, seg_pos={0 0}, upload=502026]: Done uploading piece. err: <nil> Dec 29 12:07:04.610871907 (572h38m4.481954952s) [noise=true , address=145.79.227.37:4566 , piece_id=CV6XOIXXA243KNYH2PD4MW2QMOG63GAHV3W6L3QV43RG5RBWQCGA, seg_pos={0 0}, upload=502026]: Done uploading piece. err: <nil> Dec 29 12:07:04.880421784 (572h38m4.751504829s) [noise=true , address=45.91.116.9:4262 , piece_id=NAJUSCYEUF4BAEFGHB4D5UG26VIEDDWX7AYPK5VCNSQFEKQLL25A, seg_pos={0 0}, upload=502026]: Done uploading piece. err: <nil> Dec 29 12:07:04.884249222 (572h38m4.755332307s) [noise=true , address=77.90.31.10:4519 , piece_id=PMGS2FWBT5FDZYNHIYO4RDJ7FBBDRZQVCSLFBPCJLROPKQE2OW3Q, seg_pos={0 0}, upload=502026]: Done uploading piece. err: <nil> Dec 29 12:07:04.970404301 (572h38m4.841487337s) [noise=true , address=45.91.116.13:4154 , piece_id=OIC65IMOMTBAXXDPHSCXTOA7WZT5BRZDEUN5NY3ZAO2JBURPNO2A, seg_pos={0 0}, upload=502026]: Done uploading piece. err: <nil> Dec 29 12:07:04.979038193 (572h38m4.850121238s) [noise=true , address=45.91.116.12:4225 , piece_id=TD5ALXKV4GGJXMPBW4IUGPLIKZSW2PDQ6SHXUB4XHBKXFV4IIZXQ, seg_pos={0 0}, upload=502026]: Done uploading piece. err: <nil> Dec 29 12:07:04.979064814 (572h38m4.850147859s) [seg_pos={0 0}, upload=502026]: Segment reached optimal threshold of 13 pieces. Dec 29 12:07:04.979257150 (572h38m4.850340195s) [noise=true , address=145.79.227.30:4761 , piece_id=OCSM2D6ZIUIOBHMVZ4IJ264YNU4T2Y236FGHXCKN74EOCOHFS7HQ, seg_pos={0 0}, upload=502026]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 29 12:07:04.979263632 (572h38m4.850346677s) [noise=true , address=145.79.227.30:4761 , piece_id=OCSM2D6ZIUIOBHMVZ4IJ264YNU4T2Y236FGHXCKN74EOCOHFS7HQ, seg_pos={0 0}, upload=502026]: Long tail cancellation due to optimal threshold Dec 29 12:07:04.979498309 (572h38m4.850581354s) [noise=true , address=145.79.227.44:4193 , piece_id=P3YSDY4ICXXXKT32RVK5PKQ4F6UBRF7UE56HHM7JN2ADMI2CCBVA, seg_pos={0 0}, upload=502026]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 29 12:07:04.979515081 (572h38m4.850598126s) [noise=true , address=145.79.227.44:4193 , piece_id=P3YSDY4ICXXXKT32RVK5PKQ4F6UBRF7UE56HHM7JN2ADMI2CCBVA, seg_pos={0 0}, upload=502026]: Long tail cancellation due to optimal threshold Dec 29 12:07:04.979522505 (572h38m4.85060555s) [noise=true , address=45.91.116.14:4192 , piece_id=H47E3USGQG3LZ3LMKOK7NQRSXTMOH5Z26DOVVMIEBPCQMXAJBBKQ, seg_pos={0 0}, upload=502026]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF Dec 29 12:07:04.979534918 (572h38m4.850617964s) [noise=true , address=45.91.116.14:4192 , piece_id=H47E3USGQG3LZ3LMKOK7NQRSXTMOH5Z26DOVVMIEBPCQMXAJBBKQ, seg_pos={0 0}, upload=502026]: Long tail cancellation due to optimal threshold Dec 29 12:07:04.979593140 (572h38m4.850676185s) [seg_pos={0 0}, upload=502026]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 12:07:04.979681297 (572h38m4.850764342s) [upload=502026]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 12:07:04.979687650 (572h38m4.850770685s) [upload=502026]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 12:07:05.273361574 (572h38m5.144444619s) [upload=502026]: Done uploading segments... Dec 29 12:07:05.273386793 (572h38m5.144469828s) [upload=502026]: Upload finished. err: <nil> Duration: 30.002102158s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....X..X...................................................] duration: 51.669749515s trace: https://jaeger.dev.storj.io/trace/7e7354d8023608a8 FAIL (2025-12-29 12:07:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== ) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480097317 (571h53m39.351180352s) [noise=true , address=77.90.31.21:4889 , piece_id=BR6Z55P3A7YJ5FJRDSZFUIF66FGIJQY7WCEQ7PL5DOYWL73OMKTQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480405053 (571h53m39.351488098s) [noise=true , address=77.90.31.16:4750 , piece_id=MVVZPJEGDCBSNVOB6CD2WUZGIJTL3VBHX2G4L3TVZ6JF5ZEJ5U6Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480712969 (571h53m39.351796014s) [noise=true , address=77.90.31.18:4834 , piece_id=BTWNGNQVEUEVV6HLBDM63SGNTKV4RDU4J4XFLG52WHKP53H3L5KA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481019453 (571h53m39.352102488s) [noise=true , address=145.79.227.30:4777 , piece_id=VSCXLXGIBSFE6E4OHLWNZIZ75PO7WQO2VFHUYHW7JUNH62KGGU7Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481029532 (571h53m39.352112567s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481559832 (571h53m39.352642877s) [noise=true , address=145.79.227.35:4515 , piece_id=L7LFZKZEZERUNX4HUNDNQ2PWHXNJRJ7OBFRNMFIICPZEUIBHGP7A, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481731919 (571h53m39.352814954s) [noise=true , address=64.49.13.26:4953 , piece_id=4YSW5Y3A6HYG3XJ2XAB7RGVHAOHMP7G3QIZYMYTSXVXYJ3ANLNXA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482078329 (571h53m39.353161364s) [noise=true , address=145.79.227.44:4205 , piece_id=UE4QSHDQBZ3XBWQ6I6MCRVTLMU7U6W6OZDA2ZGQCH6ZYRAGPD7WA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482099830 (571h53m39.353182865s) [noise=true , address=145.79.227.40:4067 , piece_id=WSYDRX5YJ6OCFW52ZVMM6WZYNRFUXMR4XJ6C5727BOFMR4RRH33Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482434487 (571h53m39.353517532s) [noise=true , address=145.79.227.3:4164 , piece_id=U6F7VDPOH7K6R5EH5UUXHFP55GXOEXO5BLBJHZRTITPDQI4LP7KQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482480845 (571h53m39.35356388s) [noise=true , address=64.49.13.140:4516 , piece_id=4GUYOYOAEASUK2W25S3NILWIIIHGOZX76L4EGFLUUUEDWRALPP2A, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482841442 (571h53m39.353924477s) [noise=true , address=145.79.227.34:4910 , piece_id=ZH4DCNXET2Q6VRAUH3NQ3KKSV4OYI575X7YS72XD236EEOMOG3LA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483163645 (571h53m39.35424668s) [noise=true , address=145.79.227.29:4974 , piece_id=KEKREOD2B35XQNI45JBHE2SZZC3MQQBQUR5I6WFDAQYJQGJBXSMA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483532478 (571h53m39.354615513s) [noise=true , address=77.90.31.17:4769 , piece_id=3MKUMOQIUVERKQLLIFX52XQPTQ5LSFMHIPD5YW2YTO5YEFR36IYA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483914596 (571h53m39.354997631s) [noise=true , address=145.79.227.13:4807 , piece_id=VKXB5MWAG2YYH2AWWJEGTLBZ2QUPLI25VTHI6AMMALSM23Z3AWZQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:40.761721106 (571h53m40.632804141s) [upload=501384]: Next returned nil segment Dec 29 11:22:40.761733349 (571h53m40.632816384s) [upload=501384]: Waiting for error group managing segments... Dec 29 11:22:41.121222862 (571h53m40.992305897s) [noise=true , address=64.49.13.140:4516 , piece_id=4GUYOYOAEASUK2W25S3NILWIIIHGOZX76L4EGFLUUUEDWRALPP2A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.197311646 (571h53m41.068394691s) [noise=true , address=64.49.13.26:4953 , piece_id=4YSW5Y3A6HYG3XJ2XAB7RGVHAOHMP7G3QIZYMYTSXVXYJ3ANLNXA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.347247568 (571h53m41.218330613s) [noise=true , address=145.79.227.13:4807 , piece_id=VKXB5MWAG2YYH2AWWJEGTLBZ2QUPLI25VTHI6AMMALSM23Z3AWZQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.347917313 (571h53m41.219000348s) [noise=true , address=145.79.227.3:4164 , piece_id=U6F7VDPOH7K6R5EH5UUXHFP55GXOEXO5BLBJHZRTITPDQI4LP7KQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.348510613 (571h53m41.219593648s) [noise=true , address=145.79.227.29:4974 , piece_id=KEKREOD2B35XQNI45JBHE2SZZC3MQQBQUR5I6WFDAQYJQGJBXSMA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.503950985 (571h53m41.37503402s) [noise=true , address=77.90.31.17:4769 , piece_id=3MKUMOQIUVERKQLLIFX52XQPTQ5LSFMHIPD5YW2YTO5YEFR36IYA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.561352025 (571h53m41.43243506s) [noise=true , address=77.90.31.18:4834 , piece_id=BTWNGNQVEUEVV6HLBDM63SGNTKV4RDU4J4XFLG52WHKP53H3L5KA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.622447380 (571h53m41.493530425s) [noise=true , address=77.90.31.16:4750 , piece_id=MVVZPJEGDCBSNVOB6CD2WUZGIJTL3VBHX2G4L3TVZ6JF5ZEJ5U6Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.645494685 (571h53m41.51657773s) [noise=true , address=77.90.31.21:4889 , piece_id=BR6Z55P3A7YJ5FJRDSZFUIF66FGIJQY7WCEQ7PL5DOYWL73OMKTQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.247930045 (571h53m42.11901308s) [noise=true , address=145.79.227.30:4777 , piece_id=VSCXLXGIBSFE6E4OHLWNZIZ75PO7WQO2VFHUYHW7JUNH62KGGU7Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.262221890 (571h53m42.133304925s) [noise=true , address=145.79.227.44:4205 , piece_id=UE4QSHDQBZ3XBWQ6I6MCRVTLMU7U6W6OZDA2ZGQCH6ZYRAGPD7WA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.295598123 (571h53m42.166681158s) [noise=true , address=145.79.227.35:4515 , piece_id=L7LFZKZEZERUNX4HUNDNQ2PWHXNJRJ7OBFRNMFIICPZEUIBHGP7A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.307481351 (571h53m42.178564386s) [noise=true , address=145.79.227.34:4910 , piece_id=ZH4DCNXET2Q6VRAUH3NQ3KKSV4OYI575X7YS72XD236EEOMOG3LA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.325775906 (571h53m42.196858941s) [noise=true , address=145.79.227.40:4067 , piece_id=WSYDRX5YJ6OCFW52ZVMM6WZYNRFUXMR4XJ6C5727BOFMR4RRH33Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.325865978 (571h53m42.196949023s) [seg_pos={0 0}, upload=501384]: Segment reached optimal threshold of 14 pieces. Dec 29 11:22:42.326137074 (571h53m42.197220109s) [noise=true , address=64.49.13.169:4708 , piece_id=B2X2RWRELXWJPF6AVJJZCNU46LM5ASGRE4EWGPKL3SCLQ4VKUBDQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection Dec 29 11:22:42.326153074 (571h53m42.197236109s) [noise=true , address=64.49.13.169:4708 , piece_id=B2X2RWRELXWJPF6AVJJZCNU46LM5ASGRE4EWGPKL3SCLQ4VKUBDQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326193401 (571h53m42.197276446s) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12h5esfg6mw2c8y3AVpeFBkL5FfLAwYdjhcnQ9CJ8fjh61WLYjx): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326205705 (571h53m42.19728874s) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326407739 (571h53m42.197490784s) [noise=true , address=77.90.31.10:4524 , piece_id=Q2JGEUV5EDFTVY2CSNPZOVJIIIAFBA3IHA7S5N7YZRH3E7TSTE5A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uN7CznoK4AD8FqFkNUndawdGJ9rmjAe7oNnjJkV844ScY87qe): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326415374 (571h53m42.197498419s) [noise=true , address=77.90.31.10:4524 , piece_id=Q2JGEUV5EDFTVY2CSNPZOVJIIIAFBA3IHA7S5N7YZRH3E7TSTE5A, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326505105 (571h53m42.19758815s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tnTEYvqJ6PB2DsuXA28sEuEiGj11StA4qdhcYH1kuoaMkTVdY): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326512188 (571h53m42.197595223s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326532357 (571h53m42.197615392s) [seg_pos={0 0}, upload=501384]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 11:22:42.326671562 (571h53m42.197754597s) [upload=501384]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 11:22:42.326677843 (571h53m42.197760879s) [upload=501384]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 11:22:42.610391438 (571h53m42.481474493s) [upload=501384]: Done uploading segments... Dec 29 11:22:42.610416826 (571h53m42.481499871s) [upload=501384]: Upload finished. err: <nil> Duration: 51.669774012s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 31 out of 60 is failed (threshold=6) [..X.X.XX.X.XX..X..XX.XXXX.XXXX...XXX...XXX..XX.X..X.X.X..X..] duration: 4.92175587s trace: https://jaeger.dev.storj.io/trace/17faabc7c8ad3dec FAIL (2025-12-29 12:06:49 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.921801938s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [......X..X..................................................] duration: 23.898732335s trace: https://jaeger.dev.storj.io/trace/7e7354d8023608a8 FAIL (2025-12-29 12:06:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== ) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480097317 (571h53m39.351180352s) [noise=true , address=77.90.31.21:4889 , piece_id=BR6Z55P3A7YJ5FJRDSZFUIF66FGIJQY7WCEQ7PL5DOYWL73OMKTQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480405053 (571h53m39.351488098s) [noise=true , address=77.90.31.16:4750 , piece_id=MVVZPJEGDCBSNVOB6CD2WUZGIJTL3VBHX2G4L3TVZ6JF5ZEJ5U6Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480712969 (571h53m39.351796014s) [noise=true , address=77.90.31.18:4834 , piece_id=BTWNGNQVEUEVV6HLBDM63SGNTKV4RDU4J4XFLG52WHKP53H3L5KA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481019453 (571h53m39.352102488s) [noise=true , address=145.79.227.30:4777 , piece_id=VSCXLXGIBSFE6E4OHLWNZIZ75PO7WQO2VFHUYHW7JUNH62KGGU7Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481029532 (571h53m39.352112567s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481559832 (571h53m39.352642877s) [noise=true , address=145.79.227.35:4515 , piece_id=L7LFZKZEZERUNX4HUNDNQ2PWHXNJRJ7OBFRNMFIICPZEUIBHGP7A, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481731919 (571h53m39.352814954s) [noise=true , address=64.49.13.26:4953 , piece_id=4YSW5Y3A6HYG3XJ2XAB7RGVHAOHMP7G3QIZYMYTSXVXYJ3ANLNXA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482078329 (571h53m39.353161364s) [noise=true , address=145.79.227.44:4205 , piece_id=UE4QSHDQBZ3XBWQ6I6MCRVTLMU7U6W6OZDA2ZGQCH6ZYRAGPD7WA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482099830 (571h53m39.353182865s) [noise=true , address=145.79.227.40:4067 , piece_id=WSYDRX5YJ6OCFW52ZVMM6WZYNRFUXMR4XJ6C5727BOFMR4RRH33Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482434487 (571h53m39.353517532s) [noise=true , address=145.79.227.3:4164 , piece_id=U6F7VDPOH7K6R5EH5UUXHFP55GXOEXO5BLBJHZRTITPDQI4LP7KQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482480845 (571h53m39.35356388s) [noise=true , address=64.49.13.140:4516 , piece_id=4GUYOYOAEASUK2W25S3NILWIIIHGOZX76L4EGFLUUUEDWRALPP2A, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482841442 (571h53m39.353924477s) [noise=true , address=145.79.227.34:4910 , piece_id=ZH4DCNXET2Q6VRAUH3NQ3KKSV4OYI575X7YS72XD236EEOMOG3LA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483163645 (571h53m39.35424668s) [noise=true , address=145.79.227.29:4974 , piece_id=KEKREOD2B35XQNI45JBHE2SZZC3MQQBQUR5I6WFDAQYJQGJBXSMA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483532478 (571h53m39.354615513s) [noise=true , address=77.90.31.17:4769 , piece_id=3MKUMOQIUVERKQLLIFX52XQPTQ5LSFMHIPD5YW2YTO5YEFR36IYA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483914596 (571h53m39.354997631s) [noise=true , address=145.79.227.13:4807 , piece_id=VKXB5MWAG2YYH2AWWJEGTLBZ2QUPLI25VTHI6AMMALSM23Z3AWZQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:40.761721106 (571h53m40.632804141s) [upload=501384]: Next returned nil segment Dec 29 11:22:40.761733349 (571h53m40.632816384s) [upload=501384]: Waiting for error group managing segments... Dec 29 11:22:41.121222862 (571h53m40.992305897s) [noise=true , address=64.49.13.140:4516 , piece_id=4GUYOYOAEASUK2W25S3NILWIIIHGOZX76L4EGFLUUUEDWRALPP2A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.197311646 (571h53m41.068394691s) [noise=true , address=64.49.13.26:4953 , piece_id=4YSW5Y3A6HYG3XJ2XAB7RGVHAOHMP7G3QIZYMYTSXVXYJ3ANLNXA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.347247568 (571h53m41.218330613s) [noise=true , address=145.79.227.13:4807 , piece_id=VKXB5MWAG2YYH2AWWJEGTLBZ2QUPLI25VTHI6AMMALSM23Z3AWZQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.347917313 (571h53m41.219000348s) [noise=true , address=145.79.227.3:4164 , piece_id=U6F7VDPOH7K6R5EH5UUXHFP55GXOEXO5BLBJHZRTITPDQI4LP7KQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.348510613 (571h53m41.219593648s) [noise=true , address=145.79.227.29:4974 , piece_id=KEKREOD2B35XQNI45JBHE2SZZC3MQQBQUR5I6WFDAQYJQGJBXSMA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.503950985 (571h53m41.37503402s) [noise=true , address=77.90.31.17:4769 , piece_id=3MKUMOQIUVERKQLLIFX52XQPTQ5LSFMHIPD5YW2YTO5YEFR36IYA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.561352025 (571h53m41.43243506s) [noise=true , address=77.90.31.18:4834 , piece_id=BTWNGNQVEUEVV6HLBDM63SGNTKV4RDU4J4XFLG52WHKP53H3L5KA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.622447380 (571h53m41.493530425s) [noise=true , address=77.90.31.16:4750 , piece_id=MVVZPJEGDCBSNVOB6CD2WUZGIJTL3VBHX2G4L3TVZ6JF5ZEJ5U6Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.645494685 (571h53m41.51657773s) [noise=true , address=77.90.31.21:4889 , piece_id=BR6Z55P3A7YJ5FJRDSZFUIF66FGIJQY7WCEQ7PL5DOYWL73OMKTQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.247930045 (571h53m42.11901308s) [noise=true , address=145.79.227.30:4777 , piece_id=VSCXLXGIBSFE6E4OHLWNZIZ75PO7WQO2VFHUYHW7JUNH62KGGU7Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.262221890 (571h53m42.133304925s) [noise=true , address=145.79.227.44:4205 , piece_id=UE4QSHDQBZ3XBWQ6I6MCRVTLMU7U6W6OZDA2ZGQCH6ZYRAGPD7WA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.295598123 (571h53m42.166681158s) [noise=true , address=145.79.227.35:4515 , piece_id=L7LFZKZEZERUNX4HUNDNQ2PWHXNJRJ7OBFRNMFIICPZEUIBHGP7A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.307481351 (571h53m42.178564386s) [noise=true , address=145.79.227.34:4910 , piece_id=ZH4DCNXET2Q6VRAUH3NQ3KKSV4OYI575X7YS72XD236EEOMOG3LA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.325775906 (571h53m42.196858941s) [noise=true , address=145.79.227.40:4067 , piece_id=WSYDRX5YJ6OCFW52ZVMM6WZYNRFUXMR4XJ6C5727BOFMR4RRH33Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.325865978 (571h53m42.196949023s) [seg_pos={0 0}, upload=501384]: Segment reached optimal threshold of 14 pieces. Dec 29 11:22:42.326137074 (571h53m42.197220109s) [noise=true , address=64.49.13.169:4708 , piece_id=B2X2RWRELXWJPF6AVJJZCNU46LM5ASGRE4EWGPKL3SCLQ4VKUBDQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection Dec 29 11:22:42.326153074 (571h53m42.197236109s) [noise=true , address=64.49.13.169:4708 , piece_id=B2X2RWRELXWJPF6AVJJZCNU46LM5ASGRE4EWGPKL3SCLQ4VKUBDQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326193401 (571h53m42.197276446s) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12h5esfg6mw2c8y3AVpeFBkL5FfLAwYdjhcnQ9CJ8fjh61WLYjx): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326205705 (571h53m42.19728874s) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326407739 (571h53m42.197490784s) [noise=true , address=77.90.31.10:4524 , piece_id=Q2JGEUV5EDFTVY2CSNPZOVJIIIAFBA3IHA7S5N7YZRH3E7TSTE5A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uN7CznoK4AD8FqFkNUndawdGJ9rmjAe7oNnjJkV844ScY87qe): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326415374 (571h53m42.197498419s) [noise=true , address=77.90.31.10:4524 , piece_id=Q2JGEUV5EDFTVY2CSNPZOVJIIIAFBA3IHA7S5N7YZRH3E7TSTE5A, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326505105 (571h53m42.19758815s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tnTEYvqJ6PB2DsuXA28sEuEiGj11StA4qdhcYH1kuoaMkTVdY): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326512188 (571h53m42.197595223s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326532357 (571h53m42.197615392s) [seg_pos={0 0}, upload=501384]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 11:22:42.326671562 (571h53m42.197754597s) [upload=501384]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 11:22:42.326677843 (571h53m42.197760879s) [upload=501384]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 11:22:42.610391438 (571h53m42.481474493s) [upload=501384]: Done uploading segments... Dec 29 11:22:42.610416826 (571h53m42.481499871s) [upload=501384]: Upload finished. err: <nil> Duration: 23.898760588s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 31 out of 60 is failed (threshold=6) [...X.X.XX.X.XX..X..XX.XXXX.XXXX...XXX...XXX..XX.X..X.X.X..X.] duration: 4.715658122s trace: https://jaeger.dev.storj.io/trace/17faabc7c8ad3dec FAIL (2025-12-29 12:06:03 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.715693719s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [.XXX.X.......X.X..X.XXX...XX.XXX.X.X....XXXXXX..X.X.......X.] duration: 5.913205186s trace: https://jaeger.dev.storj.io/trace/3a5a7136cd60ee2d FAIL (2025-12-29 12:06:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: QONUFHDVW4LPDBKIES45X2TBDVIAEBYG3DZ7UBC6H6W4UNXNO26A) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: QX52CIP3NIFD4GCPNDMKKWFDNI6I77HTQMB5ARFJDZMNMGUWOGYA) context deadline exceeded ======LOGS====== c 29 12:04:43.570366187 (572h35m43.441449232s) [upload=501994]: Starting upload Dec 29 12:04:43.570425691 (572h35m43.441508726s) [upload=501994]: Done starting upload Dec 29 12:04:43.570628457 (572h35m43.441711492s) [upload=501994]: Uploading segments... Dec 29 12:04:43.570670217 (572h35m43.441753252s) [upload=501994]: Got next segment. Inline: false Dec 29 12:04:43.570700514 (572h35m43.44178356s) [upload=501994]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 12:04:43.570706726 (572h35m43.441789761s) [upload=501994]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 12:04:43.817729527 (572h35m43.688812572s) [seg_pos={0 0}, upload=501994]: Begin upload segment... Dec 29 12:04:43.817905662 (572h35m43.688988697s) [seg_pos={0 0}, upload=501994]: Done begin upload segment. Dec 29 12:04:43.817949526 (572h35m43.689032571s) [noise=true , address=145.79.227.23:4545 , piece_id=W5I2OO4D5ERE4QJ2JDKSFRJPFS3C2LPL2OBKROF3YGQM3RS7QPOA, seg_pos={0 0}, upload=501994]: Uploading piece... Dec 29 12:04:43.817957130 (572h35m43.689040175s) [noise=true , address=145.79.227.6:4257 , piece_id=HHCCUY345JXXFYMICEH7KUO55PVUV3CQWLJKKSBJP7PSB45AWNMA, seg_pos={0 0}, upload=501994]: Uploading piece... Dec 29 12:04:43.818138254 (572h35m43.6892213s) [noise=true , address=145.79.227.21:4473 , piece_id=U7S4AOB77RVHDNUTO5MLIGMPB4WE5JUOJZ66RMLNOQZRVZXXFQRQ, seg_pos={0 0}, upload=501994]: Uploading piece... Dec 29 12:04:43.818195494 (572h35m43.689278539s) [noise=true , address=145.79.227.33:4870 , piece_id=NSBMWLDYF5H37JDSLQN7Q6OFMK4TB4AWW5ZLESF6LX5L3LAYPL2A, seg_pos={0 0}, upload=501994]: Uploading piece... Dec 29 12:04:43.818725102 (572h35m43.689808137s) [noise=true , address=45.91.116.14:4190 , piece_id=QX52CIP3NIFD4GCPNDMKKWFDNI6I77HTQMB5ARFJDZMNMGUWOGYA, seg_pos={0 0}, upload=501994]: Uploading piece... Dec 29 12:04:43.818895988 (572h35m43.689979033s) [noise=true , address=145.79.227.32:4834 , piece_id=YOQVWY42RZYBJ5WPETXY67XEM6LO3VTT2IX6NTTXNJC3IIIRG3VQ, seg_pos={0 0}, upload=501994]: Uploading piece... Dec 29 12:04:43.819247697 (572h35m43.690330732s) [noise=true , address=77.90.31.14:4662 , piece_id=F2BVPFUDPWTFS4O5OOGF723WVQFFMMNTRMBOC3UD4KRKKKNHYMLA, seg_pos={0 0}, upload=501994]: Uploading piece... Dec 29 12:04:43.819301921 (572h35m43.690384966s) [noise=true , address=145.79.227.10:4401 , piece_id=UPNXMGZYKMIAY5R5NS6UMI7CE6ZLRBYFCRMIGYORGYOGPMLMU7VA, seg_pos={0 0}, upload=501994]: Uploading piece... Dec 29 12:04:43.819718204 (572h35m43.690801249s) [noise=true , address=77.90.31.21:4880 , piece_id=QONUFHDVW4LPDBKIES45X2TBDVIAEBYG3DZ7UBC6H6W4UNXNO26A, seg_pos={0 0}, upload=501994]: Uploading piece... Dec 29 12:04:43.819795661 (572h35m43.690878696s) [noise=true , address=145.79.227.41:4085 , piece_id=MXN73ZLSFOGHJ4BGFYAQCUUUQJ7OQQCANF5VRLTCKA6YDEDRDPKQ, seg_pos={0 0}, upload=501994]: Uploading piece... Dec 29 12:04:43.820144415 (572h35m43.69122747s) [noise=true , address=145.79.227.14:4840 , piece_id=LPCBY3R5LSNOWPI3KW73PNR4URSFM7GNBGUSUJZ3OTRNXW73B4SA, seg_pos={0 0}, upload=501994]: Uploading piece... Dec 29 12:04:43.820310762 (572h35m43.691393807s) [noise=true , address=145.79.227.19:4079 , piece_id=3RDDVZ4KEMXFW2IV4ASMC34DRMBSSPOSOX3Y2OFLXQWOVWOUD7WA, seg_pos={0 0}, upload=501994]: Uploading piece... Dec 29 12:04:43.820500773 (572h35m43.691583818s) [noise=true , address=145.79.227.18:4006 , piece_id=VXUNUHJSNEXJEX4GFVP7RNUWOMAKMKUBXWUC5ZLNQO7LJAUMOKXA, seg_pos={0 0}, upload=501994]: Uploading piece... Dec 29 12:04:43.821021756 (572h35m43.692104801s) [noise=true , address=145.79.227.5:4223 , piece_id=LI4RDVFM7UQMSGZWZNFMURCDAIPJTG3CVR4MKX5HXX3AFBBYU33A, seg_pos={0 0}, upload=501994]: Uploading piece... Dec 29 12:04:43.821748419 (572h35m43.692831464s) [noise=true , address=45.91.116.8:4046 , piece_id=Q4ONADO5N553QD2B4VJFZDM7KNCIASXOVDJIT7CRP7QHOX5FMYMA, seg_pos={0 0}, upload=501994]: Uploading piece... Dec 29 12:04:43.822316331 (572h35m43.693399376s) [noise=true , address=145.79.227.36:4518 , piece_id=QGEFQB6LGW3ZS2MSPFMRMM2SDFPKKHT5GQNSR77HPATMR7G4P6RA, seg_pos={0 0}, upload=501994]: Uploading piece... Dec 29 12:04:45.520085014 (572h35m45.391168059s) [upload=501994]: Next returned nil segment Dec 29 12:04:45.520117246 (572h35m45.391200291s) [upload=501994]: Waiting for error group managing segments... Dec 29 12:04:45.792811009 (572h35m45.663894044s) [noise=true , address=145.79.227.6:4257 , piece_id=HHCCUY345JXXFYMICEH7KUO55PVUV3CQWLJKKSBJP7PSB45AWNMA, seg_pos={0 0}, upload=501994]: Done uploading piece. err: <nil> Dec 29 12:04:45.802839376 (572h35m45.673922411s) [noise=true , address=145.79.227.18:4006 , piece_id=VXUNUHJSNEXJEX4GFVP7RNUWOMAKMKUBXWUC5ZLNQO7LJAUMOKXA, seg_pos={0 0}, upload=501994]: Done uploading piece. err: <nil> Dec 29 12:04:45.802871507 (572h35m45.673954542s) [noise=true , address=145.79.227.14:4840 , piece_id=LPCBY3R5LSNOWPI3KW73PNR4URSFM7GNBGUSUJZ3OTRNXW73B4SA, seg_pos={0 0}, upload=501994]: Done uploading piece. err: <nil> Dec 29 12:04:45.804713525 (572h35m45.67579657s) [noise=true , address=145.79.227.10:4401 , piece_id=UPNXMGZYKMIAY5R5NS6UMI7CE6ZLRBYFCRMIGYORGYOGPMLMU7VA, seg_pos={0 0}, upload=501994]: Done uploading piece. err: <nil> Dec 29 12:04:45.808339880 (572h35m45.679422925s) [noise=true , address=145.79.227.19:4079 , piece_id=3RDDVZ4KEMXFW2IV4ASMC34DRMBSSPOSOX3Y2OFLXQWOVWOUD7WA, seg_pos={0 0}, upload=501994]: Done uploading piece. err: <nil> Dec 29 12:04:45.820139890 (572h35m45.691222935s) [noise=true , address=145.79.227.21:4473 , piece_id=U7S4AOB77RVHDNUTO5MLIGMPB4WE5JUOJZ66RMLNOQZRVZXXFQRQ, seg_pos={0 0}, upload=501994]: Done uploading piece. err: <nil> Dec 29 12:04:45.822479585 (572h35m45.69356263s) [noise=true , address=145.79.227.23:4545 , piece_id=W5I2OO4D5ERE4QJ2JDKSFRJPFS3C2LPL2OBKROF3YGQM3RS7QPOA, seg_pos={0 0}, upload=501994]: Done uploading piece. err: <nil> Dec 29 12:04:45.826875486 (572h35m45.697958531s) [noise=true , address=145.79.227.41:4085 , piece_id=MXN73ZLSFOGHJ4BGFYAQCUUUQJ7OQQCANF5VRLTCKA6YDEDRDPKQ, seg_pos={0 0}, upload=501994]: Done uploading piece. err: <nil> Dec 29 12:04:45.829106404 (572h35m45.700189449s) [noise=true , address=145.79.227.32:4834 , piece_id=YOQVWY42RZYBJ5WPETXY67XEM6LO3VTT2IX6NTTXNJC3IIIRG3VQ, seg_pos={0 0}, upload=501994]: Done uploading piece. err: <nil> Dec 29 12:04:45.832455172 (572h35m45.703538217s) [noise=true , address=145.79.227.5:4223 , piece_id=LI4RDVFM7UQMSGZWZNFMURCDAIPJTG3CVR4MKX5HXX3AFBBYU33A, seg_pos={0 0}, upload=501994]: Done uploading piece. err: <nil> Dec 29 12:04:45.845923287 (572h35m45.717006332s) [noise=true , address=145.79.227.36:4518 , piece_id=QGEFQB6LGW3ZS2MSPFMRMM2SDFPKKHT5GQNSR77HPATMR7G4P6RA, seg_pos={0 0}, upload=501994]: Done uploading piece. err: <nil> Dec 29 12:04:46.133251038 (572h35m46.004334073s) [noise=true , address=77.90.31.21:4880 , piece_id=QONUFHDVW4LPDBKIES45X2TBDVIAEBYG3DZ7UBC6H6W4UNXNO26A, seg_pos={0 0}, upload=501994]: Done uploading piece. err: <nil> Dec 29 12:04:46.186356834 (572h35m46.057439869s) [noise=true , address=45.91.116.14:4190 , piece_id=QX52CIP3NIFD4GCPNDMKKWFDNI6I77HTQMB5ARFJDZMNMGUWOGYA, seg_pos={0 0}, upload=501994]: Done uploading piece. err: <nil> Dec 29 12:04:46.186375739 (572h35m46.057458774s) [seg_pos={0 0}, upload=501994]: Segment reached optimal threshold of 13 pieces. Dec 29 12:04:46.186615686 (572h35m46.057698721s) [noise=true , address=145.79.227.33:4870 , piece_id=NSBMWLDYF5H37JDSLQN7Q6OFMK4TB4AWW5ZLESF6LX5L3LAYPL2A, seg_pos={0 0}, upload=501994]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 29 12:04:46.186624183 (572h35m46.057707228s) [noise=true , address=145.79.227.33:4870 , piece_id=NSBMWLDYF5H37JDSLQN7Q6OFMK4TB4AWW5ZLESF6LX5L3LAYPL2A, seg_pos={0 0}, upload=501994]: Long tail cancellation due to optimal threshold Dec 29 12:04:46.186668157 (572h35m46.057751192s) [noise=true , address=77.90.31.14:4662 , piece_id=F2BVPFUDPWTFS4O5OOGF723WVQFFMMNTRMBOC3UD4KRKKKNHYMLA, seg_pos={0 0}, upload=501994]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: write tcp 188.166.225.170:43478->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:43478->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43478->77.90.31.14:4662: use of closed network connection Dec 29 12:04:46.186681692 (572h35m46.057764737s) [noise=true , address=77.90.31.14:4662 , piece_id=F2BVPFUDPWTFS4O5OOGF723WVQFFMMNTRMBOC3UD4KRKKKNHYMLA, seg_pos={0 0}, upload=501994]: Long tail cancellation due to optimal threshold Dec 29 12:04:46.186824344 (572h35m46.057907379s) [noise=true , address=45.91.116.8:4046 , piece_id=Q4ONADO5N553QD2B4VJFZDM7KNCIASXOVDJIT7CRP7QHOX5FMYMA, seg_pos={0 0}, upload=501994]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 29 12:04:46.186830707 (572h35m46.057913742s) [noise=true , address=45.91.116.8:4046 , piece_id=Q4ONADO5N553QD2B4VJFZDM7KNCIASXOVDJIT7CRP7QHOX5FMYMA, seg_pos={0 0}, upload=501994]: Long tail cancellation due to optimal threshold Dec 29 12:04:46.186847508 (572h35m46.057930543s) [seg_pos={0 0}, upload=501994]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 12:04:46.186908414 (572h35m46.057991449s) [upload=501994]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 12:04:46.186913694 (572h35m46.057996729s) [upload=501994]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 12:04:46.477343211 (572h35m46.348426256s) [upload=501994]: Done uploading segments... Dec 29 12:04:46.477362888 (572h35m46.348445923s) [upload=501994]: Upload finished. err: <nil> Duration: 5.913237095s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 31 out of 60 is failed (threshold=6) [....X.X.XX.X.XX..X..XX.XXXX.XXXX...XXX...XXX..XX.X..X.X.X..X] duration: 5.001932547s trace: https://jaeger.dev.storj.io/trace/17faabc7c8ad3dec FAIL (2025-12-29 12:05:33 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001963775s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [..XXX.X.......X.X..X.XXX...XX.XXX.X.X....XXXXXX..X.X.......X] duration: 30.0020929s trace: https://jaeger.dev.storj.io/trace/3a5a7136cd60ee2d FAIL (2025-12-29 12:05:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: QONUFHDVW4LPDBKIES45X2TBDVIAEBYG3DZ7UBC6H6W4UNXNO26A) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: QX52CIP3NIFD4GCPNDMKKWFDNI6I77HTQMB5ARFJDZMNMGUWOGYA) context deadline exceeded ======LOGS====== c 29 12:04:43.570366187 (572h35m43.441449232s) [upload=501994]: Starting upload Dec 29 12:04:43.570425691 (572h35m43.441508726s) [upload=501994]: Done starting upload Dec 29 12:04:43.570628457 (572h35m43.441711492s) [upload=501994]: Uploading segments... Dec 29 12:04:43.570670217 (572h35m43.441753252s) [upload=501994]: Got next segment. Inline: false Dec 29 12:04:43.570700514 (572h35m43.44178356s) [upload=501994]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 12:04:43.570706726 (572h35m43.441789761s) [upload=501994]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 12:04:43.817729527 (572h35m43.688812572s) [seg_pos={0 0}, upload=501994]: Begin upload segment... Dec 29 12:04:43.817905662 (572h35m43.688988697s) [seg_pos={0 0}, upload=501994]: Done begin upload segment. Dec 29 12:04:43.817949526 (572h35m43.689032571s) [noise=true , address=145.79.227.23:4545 , piece_id=W5I2OO4D5ERE4QJ2JDKSFRJPFS3C2LPL2OBKROF3YGQM3RS7QPOA, seg_pos={0 0}, upload=501994]: Uploading piece... Dec 29 12:04:43.817957130 (572h35m43.689040175s) [noise=true , address=145.79.227.6:4257 , piece_id=HHCCUY345JXXFYMICEH7KUO55PVUV3CQWLJKKSBJP7PSB45AWNMA, seg_pos={0 0}, upload=501994]: Uploading piece... Dec 29 12:04:43.818138254 (572h35m43.6892213s) [noise=true , address=145.79.227.21:4473 , piece_id=U7S4AOB77RVHDNUTO5MLIGMPB4WE5JUOJZ66RMLNOQZRVZXXFQRQ, seg_pos={0 0}, upload=501994]: Uploading piece... Dec 29 12:04:43.818195494 (572h35m43.689278539s) [noise=true , address=145.79.227.33:4870 , piece_id=NSBMWLDYF5H37JDSLQN7Q6OFMK4TB4AWW5ZLESF6LX5L3LAYPL2A, seg_pos={0 0}, upload=501994]: Uploading piece... Dec 29 12:04:43.818725102 (572h35m43.689808137s) [noise=true , address=45.91.116.14:4190 , piece_id=QX52CIP3NIFD4GCPNDMKKWFDNI6I77HTQMB5ARFJDZMNMGUWOGYA, seg_pos={0 0}, upload=501994]: Uploading piece... Dec 29 12:04:43.818895988 (572h35m43.689979033s) [noise=true , address=145.79.227.32:4834 , piece_id=YOQVWY42RZYBJ5WPETXY67XEM6LO3VTT2IX6NTTXNJC3IIIRG3VQ, seg_pos={0 0}, upload=501994]: Uploading piece... Dec 29 12:04:43.819247697 (572h35m43.690330732s) [noise=true , address=77.90.31.14:4662 , piece_id=F2BVPFUDPWTFS4O5OOGF723WVQFFMMNTRMBOC3UD4KRKKKNHYMLA, seg_pos={0 0}, upload=501994]: Uploading piece... Dec 29 12:04:43.819301921 (572h35m43.690384966s) [noise=true , address=145.79.227.10:4401 , piece_id=UPNXMGZYKMIAY5R5NS6UMI7CE6ZLRBYFCRMIGYORGYOGPMLMU7VA, seg_pos={0 0}, upload=501994]: Uploading piece... Dec 29 12:04:43.819718204 (572h35m43.690801249s) [noise=true , address=77.90.31.21:4880 , piece_id=QONUFHDVW4LPDBKIES45X2TBDVIAEBYG3DZ7UBC6H6W4UNXNO26A, seg_pos={0 0}, upload=501994]: Uploading piece... Dec 29 12:04:43.819795661 (572h35m43.690878696s) [noise=true , address=145.79.227.41:4085 , piece_id=MXN73ZLSFOGHJ4BGFYAQCUUUQJ7OQQCANF5VRLTCKA6YDEDRDPKQ, seg_pos={0 0}, upload=501994]: Uploading piece... Dec 29 12:04:43.820144415 (572h35m43.69122747s) [noise=true , address=145.79.227.14:4840 , piece_id=LPCBY3R5LSNOWPI3KW73PNR4URSFM7GNBGUSUJZ3OTRNXW73B4SA, seg_pos={0 0}, upload=501994]: Uploading piece... Dec 29 12:04:43.820310762 (572h35m43.691393807s) [noise=true , address=145.79.227.19:4079 , piece_id=3RDDVZ4KEMXFW2IV4ASMC34DRMBSSPOSOX3Y2OFLXQWOVWOUD7WA, seg_pos={0 0}, upload=501994]: Uploading piece... Dec 29 12:04:43.820500773 (572h35m43.691583818s) [noise=true , address=145.79.227.18:4006 , piece_id=VXUNUHJSNEXJEX4GFVP7RNUWOMAKMKUBXWUC5ZLNQO7LJAUMOKXA, seg_pos={0 0}, upload=501994]: Uploading piece... Dec 29 12:04:43.821021756 (572h35m43.692104801s) [noise=true , address=145.79.227.5:4223 , piece_id=LI4RDVFM7UQMSGZWZNFMURCDAIPJTG3CVR4MKX5HXX3AFBBYU33A, seg_pos={0 0}, upload=501994]: Uploading piece... Dec 29 12:04:43.821748419 (572h35m43.692831464s) [noise=true , address=45.91.116.8:4046 , piece_id=Q4ONADO5N553QD2B4VJFZDM7KNCIASXOVDJIT7CRP7QHOX5FMYMA, seg_pos={0 0}, upload=501994]: Uploading piece... Dec 29 12:04:43.822316331 (572h35m43.693399376s) [noise=true , address=145.79.227.36:4518 , piece_id=QGEFQB6LGW3ZS2MSPFMRMM2SDFPKKHT5GQNSR77HPATMR7G4P6RA, seg_pos={0 0}, upload=501994]: Uploading piece... Dec 29 12:04:45.520085014 (572h35m45.391168059s) [upload=501994]: Next returned nil segment Dec 29 12:04:45.520117246 (572h35m45.391200291s) [upload=501994]: Waiting for error group managing segments... Dec 29 12:04:45.792811009 (572h35m45.663894044s) [noise=true , address=145.79.227.6:4257 , piece_id=HHCCUY345JXXFYMICEH7KUO55PVUV3CQWLJKKSBJP7PSB45AWNMA, seg_pos={0 0}, upload=501994]: Done uploading piece. err: <nil> Dec 29 12:04:45.802839376 (572h35m45.673922411s) [noise=true , address=145.79.227.18:4006 , piece_id=VXUNUHJSNEXJEX4GFVP7RNUWOMAKMKUBXWUC5ZLNQO7LJAUMOKXA, seg_pos={0 0}, upload=501994]: Done uploading piece. err: <nil> Dec 29 12:04:45.802871507 (572h35m45.673954542s) [noise=true , address=145.79.227.14:4840 , piece_id=LPCBY3R5LSNOWPI3KW73PNR4URSFM7GNBGUSUJZ3OTRNXW73B4SA, seg_pos={0 0}, upload=501994]: Done uploading piece. err: <nil> Dec 29 12:04:45.804713525 (572h35m45.67579657s) [noise=true , address=145.79.227.10:4401 , piece_id=UPNXMGZYKMIAY5R5NS6UMI7CE6ZLRBYFCRMIGYORGYOGPMLMU7VA, seg_pos={0 0}, upload=501994]: Done uploading piece. err: <nil> Dec 29 12:04:45.808339880 (572h35m45.679422925s) [noise=true , address=145.79.227.19:4079 , piece_id=3RDDVZ4KEMXFW2IV4ASMC34DRMBSSPOSOX3Y2OFLXQWOVWOUD7WA, seg_pos={0 0}, upload=501994]: Done uploading piece. err: <nil> Dec 29 12:04:45.820139890 (572h35m45.691222935s) [noise=true , address=145.79.227.21:4473 , piece_id=U7S4AOB77RVHDNUTO5MLIGMPB4WE5JUOJZ66RMLNOQZRVZXXFQRQ, seg_pos={0 0}, upload=501994]: Done uploading piece. err: <nil> Dec 29 12:04:45.822479585 (572h35m45.69356263s) [noise=true , address=145.79.227.23:4545 , piece_id=W5I2OO4D5ERE4QJ2JDKSFRJPFS3C2LPL2OBKROF3YGQM3RS7QPOA, seg_pos={0 0}, upload=501994]: Done uploading piece. err: <nil> Dec 29 12:04:45.826875486 (572h35m45.697958531s) [noise=true , address=145.79.227.41:4085 , piece_id=MXN73ZLSFOGHJ4BGFYAQCUUUQJ7OQQCANF5VRLTCKA6YDEDRDPKQ, seg_pos={0 0}, upload=501994]: Done uploading piece. err: <nil> Dec 29 12:04:45.829106404 (572h35m45.700189449s) [noise=true , address=145.79.227.32:4834 , piece_id=YOQVWY42RZYBJ5WPETXY67XEM6LO3VTT2IX6NTTXNJC3IIIRG3VQ, seg_pos={0 0}, upload=501994]: Done uploading piece. err: <nil> Dec 29 12:04:45.832455172 (572h35m45.703538217s) [noise=true , address=145.79.227.5:4223 , piece_id=LI4RDVFM7UQMSGZWZNFMURCDAIPJTG3CVR4MKX5HXX3AFBBYU33A, seg_pos={0 0}, upload=501994]: Done uploading piece. err: <nil> Dec 29 12:04:45.845923287 (572h35m45.717006332s) [noise=true , address=145.79.227.36:4518 , piece_id=QGEFQB6LGW3ZS2MSPFMRMM2SDFPKKHT5GQNSR77HPATMR7G4P6RA, seg_pos={0 0}, upload=501994]: Done uploading piece. err: <nil> Dec 29 12:04:46.133251038 (572h35m46.004334073s) [noise=true , address=77.90.31.21:4880 , piece_id=QONUFHDVW4LPDBKIES45X2TBDVIAEBYG3DZ7UBC6H6W4UNXNO26A, seg_pos={0 0}, upload=501994]: Done uploading piece. err: <nil> Dec 29 12:04:46.186356834 (572h35m46.057439869s) [noise=true , address=45.91.116.14:4190 , piece_id=QX52CIP3NIFD4GCPNDMKKWFDNI6I77HTQMB5ARFJDZMNMGUWOGYA, seg_pos={0 0}, upload=501994]: Done uploading piece. err: <nil> Dec 29 12:04:46.186375739 (572h35m46.057458774s) [seg_pos={0 0}, upload=501994]: Segment reached optimal threshold of 13 pieces. Dec 29 12:04:46.186615686 (572h35m46.057698721s) [noise=true , address=145.79.227.33:4870 , piece_id=NSBMWLDYF5H37JDSLQN7Q6OFMK4TB4AWW5ZLESF6LX5L3LAYPL2A, seg_pos={0 0}, upload=501994]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 29 12:04:46.186624183 (572h35m46.057707228s) [noise=true , address=145.79.227.33:4870 , piece_id=NSBMWLDYF5H37JDSLQN7Q6OFMK4TB4AWW5ZLESF6LX5L3LAYPL2A, seg_pos={0 0}, upload=501994]: Long tail cancellation due to optimal threshold Dec 29 12:04:46.186668157 (572h35m46.057751192s) [noise=true , address=77.90.31.14:4662 , piece_id=F2BVPFUDPWTFS4O5OOGF723WVQFFMMNTRMBOC3UD4KRKKKNHYMLA, seg_pos={0 0}, upload=501994]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: write tcp 188.166.225.170:43478->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:43478->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43478->77.90.31.14:4662: use of closed network connection Dec 29 12:04:46.186681692 (572h35m46.057764737s) [noise=true , address=77.90.31.14:4662 , piece_id=F2BVPFUDPWTFS4O5OOGF723WVQFFMMNTRMBOC3UD4KRKKKNHYMLA, seg_pos={0 0}, upload=501994]: Long tail cancellation due to optimal threshold Dec 29 12:04:46.186824344 (572h35m46.057907379s) [noise=true , address=45.91.116.8:4046 , piece_id=Q4ONADO5N553QD2B4VJFZDM7KNCIASXOVDJIT7CRP7QHOX5FMYMA, seg_pos={0 0}, upload=501994]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 29 12:04:46.186830707 (572h35m46.057913742s) [noise=true , address=45.91.116.8:4046 , piece_id=Q4ONADO5N553QD2B4VJFZDM7KNCIASXOVDJIT7CRP7QHOX5FMYMA, seg_pos={0 0}, upload=501994]: Long tail cancellation due to optimal threshold Dec 29 12:04:46.186847508 (572h35m46.057930543s) [seg_pos={0 0}, upload=501994]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 12:04:46.186908414 (572h35m46.057991449s) [upload=501994]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 12:04:46.186913694 (572h35m46.057996729s) [upload=501994]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 12:04:46.477343211 (572h35m46.348426256s) [upload=501994]: Done uploading segments... Dec 29 12:04:46.477362888 (572h35m46.348445923s) [upload=501994]: Upload finished. err: <nil> Duration: 30.002116115s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......X..X.................................................] duration: 33.001151889s trace: https://jaeger.dev.storj.io/trace/7e7354d8023608a8 FAIL (2025-12-29 12:05:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== ) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480097317 (571h53m39.351180352s) [noise=true , address=77.90.31.21:4889 , piece_id=BR6Z55P3A7YJ5FJRDSZFUIF66FGIJQY7WCEQ7PL5DOYWL73OMKTQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480405053 (571h53m39.351488098s) [noise=true , address=77.90.31.16:4750 , piece_id=MVVZPJEGDCBSNVOB6CD2WUZGIJTL3VBHX2G4L3TVZ6JF5ZEJ5U6Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480712969 (571h53m39.351796014s) [noise=true , address=77.90.31.18:4834 , piece_id=BTWNGNQVEUEVV6HLBDM63SGNTKV4RDU4J4XFLG52WHKP53H3L5KA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481019453 (571h53m39.352102488s) [noise=true , address=145.79.227.30:4777 , piece_id=VSCXLXGIBSFE6E4OHLWNZIZ75PO7WQO2VFHUYHW7JUNH62KGGU7Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481029532 (571h53m39.352112567s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481559832 (571h53m39.352642877s) [noise=true , address=145.79.227.35:4515 , piece_id=L7LFZKZEZERUNX4HUNDNQ2PWHXNJRJ7OBFRNMFIICPZEUIBHGP7A, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481731919 (571h53m39.352814954s) [noise=true , address=64.49.13.26:4953 , piece_id=4YSW5Y3A6HYG3XJ2XAB7RGVHAOHMP7G3QIZYMYTSXVXYJ3ANLNXA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482078329 (571h53m39.353161364s) [noise=true , address=145.79.227.44:4205 , piece_id=UE4QSHDQBZ3XBWQ6I6MCRVTLMU7U6W6OZDA2ZGQCH6ZYRAGPD7WA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482099830 (571h53m39.353182865s) [noise=true , address=145.79.227.40:4067 , piece_id=WSYDRX5YJ6OCFW52ZVMM6WZYNRFUXMR4XJ6C5727BOFMR4RRH33Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482434487 (571h53m39.353517532s) [noise=true , address=145.79.227.3:4164 , piece_id=U6F7VDPOH7K6R5EH5UUXHFP55GXOEXO5BLBJHZRTITPDQI4LP7KQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482480845 (571h53m39.35356388s) [noise=true , address=64.49.13.140:4516 , piece_id=4GUYOYOAEASUK2W25S3NILWIIIHGOZX76L4EGFLUUUEDWRALPP2A, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482841442 (571h53m39.353924477s) [noise=true , address=145.79.227.34:4910 , piece_id=ZH4DCNXET2Q6VRAUH3NQ3KKSV4OYI575X7YS72XD236EEOMOG3LA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483163645 (571h53m39.35424668s) [noise=true , address=145.79.227.29:4974 , piece_id=KEKREOD2B35XQNI45JBHE2SZZC3MQQBQUR5I6WFDAQYJQGJBXSMA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483532478 (571h53m39.354615513s) [noise=true , address=77.90.31.17:4769 , piece_id=3MKUMOQIUVERKQLLIFX52XQPTQ5LSFMHIPD5YW2YTO5YEFR36IYA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483914596 (571h53m39.354997631s) [noise=true , address=145.79.227.13:4807 , piece_id=VKXB5MWAG2YYH2AWWJEGTLBZ2QUPLI25VTHI6AMMALSM23Z3AWZQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:40.761721106 (571h53m40.632804141s) [upload=501384]: Next returned nil segment Dec 29 11:22:40.761733349 (571h53m40.632816384s) [upload=501384]: Waiting for error group managing segments... Dec 29 11:22:41.121222862 (571h53m40.992305897s) [noise=true , address=64.49.13.140:4516 , piece_id=4GUYOYOAEASUK2W25S3NILWIIIHGOZX76L4EGFLUUUEDWRALPP2A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.197311646 (571h53m41.068394691s) [noise=true , address=64.49.13.26:4953 , piece_id=4YSW5Y3A6HYG3XJ2XAB7RGVHAOHMP7G3QIZYMYTSXVXYJ3ANLNXA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.347247568 (571h53m41.218330613s) [noise=true , address=145.79.227.13:4807 , piece_id=VKXB5MWAG2YYH2AWWJEGTLBZ2QUPLI25VTHI6AMMALSM23Z3AWZQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.347917313 (571h53m41.219000348s) [noise=true , address=145.79.227.3:4164 , piece_id=U6F7VDPOH7K6R5EH5UUXHFP55GXOEXO5BLBJHZRTITPDQI4LP7KQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.348510613 (571h53m41.219593648s) [noise=true , address=145.79.227.29:4974 , piece_id=KEKREOD2B35XQNI45JBHE2SZZC3MQQBQUR5I6WFDAQYJQGJBXSMA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.503950985 (571h53m41.37503402s) [noise=true , address=77.90.31.17:4769 , piece_id=3MKUMOQIUVERKQLLIFX52XQPTQ5LSFMHIPD5YW2YTO5YEFR36IYA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.561352025 (571h53m41.43243506s) [noise=true , address=77.90.31.18:4834 , piece_id=BTWNGNQVEUEVV6HLBDM63SGNTKV4RDU4J4XFLG52WHKP53H3L5KA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.622447380 (571h53m41.493530425s) [noise=true , address=77.90.31.16:4750 , piece_id=MVVZPJEGDCBSNVOB6CD2WUZGIJTL3VBHX2G4L3TVZ6JF5ZEJ5U6Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.645494685 (571h53m41.51657773s) [noise=true , address=77.90.31.21:4889 , piece_id=BR6Z55P3A7YJ5FJRDSZFUIF66FGIJQY7WCEQ7PL5DOYWL73OMKTQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.247930045 (571h53m42.11901308s) [noise=true , address=145.79.227.30:4777 , piece_id=VSCXLXGIBSFE6E4OHLWNZIZ75PO7WQO2VFHUYHW7JUNH62KGGU7Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.262221890 (571h53m42.133304925s) [noise=true , address=145.79.227.44:4205 , piece_id=UE4QSHDQBZ3XBWQ6I6MCRVTLMU7U6W6OZDA2ZGQCH6ZYRAGPD7WA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.295598123 (571h53m42.166681158s) [noise=true , address=145.79.227.35:4515 , piece_id=L7LFZKZEZERUNX4HUNDNQ2PWHXNJRJ7OBFRNMFIICPZEUIBHGP7A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.307481351 (571h53m42.178564386s) [noise=true , address=145.79.227.34:4910 , piece_id=ZH4DCNXET2Q6VRAUH3NQ3KKSV4OYI575X7YS72XD236EEOMOG3LA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.325775906 (571h53m42.196858941s) [noise=true , address=145.79.227.40:4067 , piece_id=WSYDRX5YJ6OCFW52ZVMM6WZYNRFUXMR4XJ6C5727BOFMR4RRH33Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.325865978 (571h53m42.196949023s) [seg_pos={0 0}, upload=501384]: Segment reached optimal threshold of 14 pieces. Dec 29 11:22:42.326137074 (571h53m42.197220109s) [noise=true , address=64.49.13.169:4708 , piece_id=B2X2RWRELXWJPF6AVJJZCNU46LM5ASGRE4EWGPKL3SCLQ4VKUBDQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection Dec 29 11:22:42.326153074 (571h53m42.197236109s) [noise=true , address=64.49.13.169:4708 , piece_id=B2X2RWRELXWJPF6AVJJZCNU46LM5ASGRE4EWGPKL3SCLQ4VKUBDQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326193401 (571h53m42.197276446s) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12h5esfg6mw2c8y3AVpeFBkL5FfLAwYdjhcnQ9CJ8fjh61WLYjx): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326205705 (571h53m42.19728874s) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326407739 (571h53m42.197490784s) [noise=true , address=77.90.31.10:4524 , piece_id=Q2JGEUV5EDFTVY2CSNPZOVJIIIAFBA3IHA7S5N7YZRH3E7TSTE5A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uN7CznoK4AD8FqFkNUndawdGJ9rmjAe7oNnjJkV844ScY87qe): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326415374 (571h53m42.197498419s) [noise=true , address=77.90.31.10:4524 , piece_id=Q2JGEUV5EDFTVY2CSNPZOVJIIIAFBA3IHA7S5N7YZRH3E7TSTE5A, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326505105 (571h53m42.19758815s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tnTEYvqJ6PB2DsuXA28sEuEiGj11StA4qdhcYH1kuoaMkTVdY): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326512188 (571h53m42.197595223s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326532357 (571h53m42.197615392s) [seg_pos={0 0}, upload=501384]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 11:22:42.326671562 (571h53m42.197754597s) [upload=501384]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 11:22:42.326677843 (571h53m42.197760879s) [upload=501384]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 11:22:42.610391438 (571h53m42.481474493s) [upload=501384]: Done uploading segments... Dec 29 11:22:42.610416826 (571h53m42.481499871s) [upload=501384]: Upload finished. err: <nil> Duration: 33.001182627s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 30 out of 60 is failed (threshold=6) [.....X.X.XX.X.XX..X..XX.XXXX.XXXX...XXX...XXX..XX.X..X.X.X..] duration: 3.713671524s trace: https://jaeger.dev.storj.io/trace/7f0c58f8d9eccf8c FAIL (2025-12-29 12:04:26 UTC) orbiter warning: context deadline exceeded Duration: 3.713695338s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [........X..X................................................] duration: 38.195646098s trace: https://jaeger.dev.storj.io/trace/7e7354d8023608a8 FAIL (2025-12-29 12:04:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== ) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480097317 (571h53m39.351180352s) [noise=true , address=77.90.31.21:4889 , piece_id=BR6Z55P3A7YJ5FJRDSZFUIF66FGIJQY7WCEQ7PL5DOYWL73OMKTQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480405053 (571h53m39.351488098s) [noise=true , address=77.90.31.16:4750 , piece_id=MVVZPJEGDCBSNVOB6CD2WUZGIJTL3VBHX2G4L3TVZ6JF5ZEJ5U6Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480712969 (571h53m39.351796014s) [noise=true , address=77.90.31.18:4834 , piece_id=BTWNGNQVEUEVV6HLBDM63SGNTKV4RDU4J4XFLG52WHKP53H3L5KA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481019453 (571h53m39.352102488s) [noise=true , address=145.79.227.30:4777 , piece_id=VSCXLXGIBSFE6E4OHLWNZIZ75PO7WQO2VFHUYHW7JUNH62KGGU7Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481029532 (571h53m39.352112567s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481559832 (571h53m39.352642877s) [noise=true , address=145.79.227.35:4515 , piece_id=L7LFZKZEZERUNX4HUNDNQ2PWHXNJRJ7OBFRNMFIICPZEUIBHGP7A, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481731919 (571h53m39.352814954s) [noise=true , address=64.49.13.26:4953 , piece_id=4YSW5Y3A6HYG3XJ2XAB7RGVHAOHMP7G3QIZYMYTSXVXYJ3ANLNXA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482078329 (571h53m39.353161364s) [noise=true , address=145.79.227.44:4205 , piece_id=UE4QSHDQBZ3XBWQ6I6MCRVTLMU7U6W6OZDA2ZGQCH6ZYRAGPD7WA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482099830 (571h53m39.353182865s) [noise=true , address=145.79.227.40:4067 , piece_id=WSYDRX5YJ6OCFW52ZVMM6WZYNRFUXMR4XJ6C5727BOFMR4RRH33Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482434487 (571h53m39.353517532s) [noise=true , address=145.79.227.3:4164 , piece_id=U6F7VDPOH7K6R5EH5UUXHFP55GXOEXO5BLBJHZRTITPDQI4LP7KQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482480845 (571h53m39.35356388s) [noise=true , address=64.49.13.140:4516 , piece_id=4GUYOYOAEASUK2W25S3NILWIIIHGOZX76L4EGFLUUUEDWRALPP2A, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482841442 (571h53m39.353924477s) [noise=true , address=145.79.227.34:4910 , piece_id=ZH4DCNXET2Q6VRAUH3NQ3KKSV4OYI575X7YS72XD236EEOMOG3LA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483163645 (571h53m39.35424668s) [noise=true , address=145.79.227.29:4974 , piece_id=KEKREOD2B35XQNI45JBHE2SZZC3MQQBQUR5I6WFDAQYJQGJBXSMA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483532478 (571h53m39.354615513s) [noise=true , address=77.90.31.17:4769 , piece_id=3MKUMOQIUVERKQLLIFX52XQPTQ5LSFMHIPD5YW2YTO5YEFR36IYA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483914596 (571h53m39.354997631s) [noise=true , address=145.79.227.13:4807 , piece_id=VKXB5MWAG2YYH2AWWJEGTLBZ2QUPLI25VTHI6AMMALSM23Z3AWZQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:40.761721106 (571h53m40.632804141s) [upload=501384]: Next returned nil segment Dec 29 11:22:40.761733349 (571h53m40.632816384s) [upload=501384]: Waiting for error group managing segments... Dec 29 11:22:41.121222862 (571h53m40.992305897s) [noise=true , address=64.49.13.140:4516 , piece_id=4GUYOYOAEASUK2W25S3NILWIIIHGOZX76L4EGFLUUUEDWRALPP2A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.197311646 (571h53m41.068394691s) [noise=true , address=64.49.13.26:4953 , piece_id=4YSW5Y3A6HYG3XJ2XAB7RGVHAOHMP7G3QIZYMYTSXVXYJ3ANLNXA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.347247568 (571h53m41.218330613s) [noise=true , address=145.79.227.13:4807 , piece_id=VKXB5MWAG2YYH2AWWJEGTLBZ2QUPLI25VTHI6AMMALSM23Z3AWZQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.347917313 (571h53m41.219000348s) [noise=true , address=145.79.227.3:4164 , piece_id=U6F7VDPOH7K6R5EH5UUXHFP55GXOEXO5BLBJHZRTITPDQI4LP7KQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.348510613 (571h53m41.219593648s) [noise=true , address=145.79.227.29:4974 , piece_id=KEKREOD2B35XQNI45JBHE2SZZC3MQQBQUR5I6WFDAQYJQGJBXSMA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.503950985 (571h53m41.37503402s) [noise=true , address=77.90.31.17:4769 , piece_id=3MKUMOQIUVERKQLLIFX52XQPTQ5LSFMHIPD5YW2YTO5YEFR36IYA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.561352025 (571h53m41.43243506s) [noise=true , address=77.90.31.18:4834 , piece_id=BTWNGNQVEUEVV6HLBDM63SGNTKV4RDU4J4XFLG52WHKP53H3L5KA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.622447380 (571h53m41.493530425s) [noise=true , address=77.90.31.16:4750 , piece_id=MVVZPJEGDCBSNVOB6CD2WUZGIJTL3VBHX2G4L3TVZ6JF5ZEJ5U6Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.645494685 (571h53m41.51657773s) [noise=true , address=77.90.31.21:4889 , piece_id=BR6Z55P3A7YJ5FJRDSZFUIF66FGIJQY7WCEQ7PL5DOYWL73OMKTQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.247930045 (571h53m42.11901308s) [noise=true , address=145.79.227.30:4777 , piece_id=VSCXLXGIBSFE6E4OHLWNZIZ75PO7WQO2VFHUYHW7JUNH62KGGU7Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.262221890 (571h53m42.133304925s) [noise=true , address=145.79.227.44:4205 , piece_id=UE4QSHDQBZ3XBWQ6I6MCRVTLMU7U6W6OZDA2ZGQCH6ZYRAGPD7WA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.295598123 (571h53m42.166681158s) [noise=true , address=145.79.227.35:4515 , piece_id=L7LFZKZEZERUNX4HUNDNQ2PWHXNJRJ7OBFRNMFIICPZEUIBHGP7A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.307481351 (571h53m42.178564386s) [noise=true , address=145.79.227.34:4910 , piece_id=ZH4DCNXET2Q6VRAUH3NQ3KKSV4OYI575X7YS72XD236EEOMOG3LA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.325775906 (571h53m42.196858941s) [noise=true , address=145.79.227.40:4067 , piece_id=WSYDRX5YJ6OCFW52ZVMM6WZYNRFUXMR4XJ6C5727BOFMR4RRH33Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.325865978 (571h53m42.196949023s) [seg_pos={0 0}, upload=501384]: Segment reached optimal threshold of 14 pieces. Dec 29 11:22:42.326137074 (571h53m42.197220109s) [noise=true , address=64.49.13.169:4708 , piece_id=B2X2RWRELXWJPF6AVJJZCNU46LM5ASGRE4EWGPKL3SCLQ4VKUBDQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection Dec 29 11:22:42.326153074 (571h53m42.197236109s) [noise=true , address=64.49.13.169:4708 , piece_id=B2X2RWRELXWJPF6AVJJZCNU46LM5ASGRE4EWGPKL3SCLQ4VKUBDQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326193401 (571h53m42.197276446s) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12h5esfg6mw2c8y3AVpeFBkL5FfLAwYdjhcnQ9CJ8fjh61WLYjx): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326205705 (571h53m42.19728874s) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326407739 (571h53m42.197490784s) [noise=true , address=77.90.31.10:4524 , piece_id=Q2JGEUV5EDFTVY2CSNPZOVJIIIAFBA3IHA7S5N7YZRH3E7TSTE5A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uN7CznoK4AD8FqFkNUndawdGJ9rmjAe7oNnjJkV844ScY87qe): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326415374 (571h53m42.197498419s) [noise=true , address=77.90.31.10:4524 , piece_id=Q2JGEUV5EDFTVY2CSNPZOVJIIIAFBA3IHA7S5N7YZRH3E7TSTE5A, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326505105 (571h53m42.19758815s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tnTEYvqJ6PB2DsuXA28sEuEiGj11StA4qdhcYH1kuoaMkTVdY): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326512188 (571h53m42.197595223s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326532357 (571h53m42.197615392s) [seg_pos={0 0}, upload=501384]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 11:22:42.326671562 (571h53m42.197754597s) [upload=501384]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 11:22:42.326677843 (571h53m42.197760879s) [upload=501384]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 11:22:42.610391438 (571h53m42.481474493s) [upload=501384]: Done uploading segments... Dec 29 11:22:42.610416826 (571h53m42.481499871s) [upload=501384]: Upload finished. err: <nil> Duration: 38.1956735s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 30 out of 60 is failed (threshold=6) [......X.X.XX.X.XX..X..XX.XXXX.XXXX...XXX...XXX..XX.X..X.X.X.] duration: 3.544884409s trace: https://jaeger.dev.storj.io/trace/7f0c58f8d9eccf8c FAIL (2025-12-29 12:03:58 UTC) orbiter warning: context deadline exceeded Duration: 3.544905337s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [X..XXX.X.......X.X..X.XXX...XX.XXX.X.X....XXXXXX..X.X.......] duration: 6.224021027s trace: https://jaeger.dev.storj.io/trace/78a810c1312d39a9 FAIL (2025-12-29 12:03:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: TXY6KE2F3OOGRMOT4ZF3MERPEEXFOQONOJUJ2G5TDQQTT5TFGANA) context deadline exceeded; piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: 2PTAVS33GMBNXYULC2YFT6YTVLDSPBQOIYM2BHCXGYTTWLF2FNDQ) context deadline exceeded ======LOGS====== Dec 29 11:59:08.361853165 (572h30m8.2329362s) [upload=501910]: Starting upload Dec 29 11:59:08.361997270 (572h30m8.233080335s) [upload=501910]: Done starting upload Dec 29 11:59:08.362093513 (572h30m8.233176548s) [upload=501910]: Uploading segments... Dec 29 11:59:08.362292753 (572h30m8.233375798s) [upload=501910]: Got next segment. Inline: false Dec 29 11:59:08.362335234 (572h30m8.233418269s) [upload=501910]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 11:59:08.362340063 (572h30m8.233423098s) [upload=501910]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 11:59:08.611633275 (572h30m8.482716321s) [seg_pos={0 0}, upload=501910]: Begin upload segment... Dec 29 11:59:08.611679954 (572h30m8.482762999s) [seg_pos={0 0}, upload=501910]: Done begin upload segment. Dec 29 11:59:08.611705703 (572h30m8.482788738s) [noise=true , address=145.79.227.30:4761 , piece_id=FWOGG3AETOEAFIFEBVASAVOTUITY7TVF3JVWTOHDCTXDGQMLQGVA, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.611729619 (572h30m8.482812664s) [noise=true , address=45.91.116.14:4192 , piece_id=TXY6KE2F3OOGRMOT4ZF3MERPEEXFOQONOJUJ2G5TDQQTT5TFGANA, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.612625375 (572h30m8.48370841s) [noise=true , address=145.79.227.47:4266 , piece_id=XAZDWB62CWNFH7N5D4TGNXRLRT7YF2ZP4L6MNATUCZDDUXA2QDGQ, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.612648729 (572h30m8.483731774s) [noise=true , address=45.91.116.5:4298 , piece_id=TTGXECJ2OUJFDRSXM53VW6V2KJGT5ZTWKGEN6WOKN7RQZURKK62Q, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.613025066 (572h30m8.484108111s) [noise=true , address=45.91.116.9:4262 , piece_id=KXI3CRGDXPFIQMETUUKKJZS5YX3TY7XZJ6A2HCHO4JW3DNB2I43A, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.613521331 (572h30m8.484604376s) [noise=true , address=145.79.227.17:4970 , piece_id=PZIKWOOFEAAWSI3DDG6MN44EJ6RVL7HYC6HXLP3JGDVDRPPGORFQ, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.613528916 (572h30m8.484611961s) [noise=true , address=145.79.227.2:4115 , piece_id=VVEW5N7PMXNFLYMEIWFB6ZBTJWMBFK2PRWTFZKTUB5AUIPSOMCZA, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.613977430 (572h30m8.485060465s) [noise=true , address=145.79.227.46:4602 , piece_id=UO7XPJAI7VQXCF7SVBV43WTI6JPOK4DFQK3WMJ54EPXVSRWLA6QA, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.614275347 (572h30m8.485358382s) [noise=true , address=145.79.227.23:4545 , piece_id=XI7C54VR2CYZIE6OLAXP6UYWH2BQI5IFZLL6XFGMDPIF5UKKJUKA, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.614415964 (572h30m8.485499009s) [noise=true , address=77.90.31.14:4661 , piece_id=XLSIJ2ZO6ZB7AFRAGBG7B3K5T4Q4EEGXKRUJ3QLQ5CNPRKSQ5N4A, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.614940583 (572h30m8.486023618s) [noise=true , address=145.79.227.5:4222 , piece_id=UBA5YYPYYNUYRYHYURWMRXXB7Q6AMYZ3EZ3HZDLNF77OG4MDKEUQ, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.614969919 (572h30m8.486052964s) [noise=true , address=77.90.31.19:4842 , piece_id=2PTAVS33GMBNXYULC2YFT6YTVLDSPBQOIYM2BHCXGYTTWLF2FNDQ, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.615383867 (572h30m8.486466903s) [noise=true , address=145.79.227.14:4837 , piece_id=TE4VAA7KHS4YYQRGKMOMBM2WLVEMNB5NSML4JC7OY2EKHMDZQ6SA, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.615607825 (572h30m8.48669087s) [noise=true , address=145.79.227.21:4473 , piece_id=IDGJS4QYGBIORV26G45FWDSOA3H63UEEGPPPLLDN5RCFE3W4HAXA, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.615881095 (572h30m8.48696414s) [noise=true , address=145.79.227.26:4654 , piece_id=45IHGFKRVEDQDGQTSFALHRIE6IVQZHY7E3DCIZI6TNHLJFITPGQQ, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.616226894 (572h30m8.487309939s) [noise=true , address=145.79.227.50:4374 , piece_id=L2USZXC5XKVWUDSZ5MDLOZVAFJOMVGYF7SM2E5Q4EAU4EEZEF23Q, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:10.218180372 (572h30m10.089263407s) [upload=501910]: Next returned nil segment Dec 29 11:59:10.218197525 (572h30m10.08928056s) [upload=501910]: Waiting for error group managing segments... Dec 29 11:59:10.596756931 (572h30m10.467839966s) [noise=true , address=145.79.227.17:4970 , piece_id=PZIKWOOFEAAWSI3DDG6MN44EJ6RVL7HYC6HXLP3JGDVDRPPGORFQ, seg_pos={0 0}, upload=501910]: Done uploading piece. err: <nil> Dec 29 11:59:10.597714194 (572h30m10.46879724s) [noise=true , address=145.79.227.47:4266 , piece_id=XAZDWB62CWNFH7N5D4TGNXRLRT7YF2ZP4L6MNATUCZDDUXA2QDGQ, seg_pos={0 0}, upload=501910]: Done uploading piece. err: <nil> Dec 29 11:59:10.600236358 (572h30m10.471319393s) [noise=true , address=145.79.227.14:4837 , piece_id=TE4VAA7KHS4YYQRGKMOMBM2WLVEMNB5NSML4JC7OY2EKHMDZQ6SA, seg_pos={0 0}, upload=501910]: Done uploading piece. err: <nil> Dec 29 11:59:10.602320044 (572h30m10.47340308s) [noise=true , address=145.79.227.5:4222 , piece_id=UBA5YYPYYNUYRYHYURWMRXXB7Q6AMYZ3EZ3HZDLNF77OG4MDKEUQ, seg_pos={0 0}, upload=501910]: Done uploading piece. err: <nil> Dec 29 11:59:10.602912464 (572h30m10.473995499s) [noise=true , address=145.79.227.26:4654 , piece_id=45IHGFKRVEDQDGQTSFALHRIE6IVQZHY7E3DCIZI6TNHLJFITPGQQ, seg_pos={0 0}, upload=501910]: Done uploading piece. err: <nil> Dec 29 11:59:10.611761013 (572h30m10.482844048s) [noise=true , address=145.79.227.23:4545 , piece_id=XI7C54VR2CYZIE6OLAXP6UYWH2BQI5IFZLL6XFGMDPIF5UKKJUKA, seg_pos={0 0}, upload=501910]: Done uploading piece. err: <nil> Dec 29 11:59:10.623036624 (572h30m10.494119669s) [noise=true , address=145.79.227.46:4602 , piece_id=UO7XPJAI7VQXCF7SVBV43WTI6JPOK4DFQK3WMJ54EPXVSRWLA6QA, seg_pos={0 0}, upload=501910]: Done uploading piece. err: <nil> Dec 29 11:59:10.626994862 (572h30m10.498077907s) [noise=true , address=145.79.227.50:4374 , piece_id=L2USZXC5XKVWUDSZ5MDLOZVAFJOMVGYF7SM2E5Q4EAU4EEZEF23Q, seg_pos={0 0}, upload=501910]: Done uploading piece. err: <nil> Dec 29 11:59:10.627968646 (572h30m10.499051691s) [noise=true , address=145.79.227.21:4473 , piece_id=IDGJS4QYGBIORV26G45FWDSOA3H63UEEGPPPLLDN5RCFE3W4HAXA, seg_pos={0 0}, upload=501910]: Done uploading piece. err: <nil> Dec 29 11:59:10.629745359 (572h30m10.500828404s) [noise=true , address=145.79.227.2:4115 , piece_id=VVEW5N7PMXNFLYMEIWFB6ZBTJWMBFK2PRWTFZKTUB5AUIPSOMCZA, seg_pos={0 0}, upload=501910]: Done uploading piece. err: <nil> Dec 29 11:59:10.847903124 (572h30m10.718986159s) [noise=true , address=45.91.116.14:4192 , piece_id=TXY6KE2F3OOGRMOT4ZF3MERPEEXFOQONOJUJ2G5TDQQTT5TFGANA, seg_pos={0 0}, upload=501910]: Done uploading piece. err: <nil> Dec 29 11:59:10.901405164 (572h30m10.772488199s) [noise=true , address=77.90.31.19:4842 , piece_id=2PTAVS33GMBNXYULC2YFT6YTVLDSPBQOIYM2BHCXGYTTWLF2FNDQ, seg_pos={0 0}, upload=501910]: Done uploading piece. err: <nil> Dec 29 11:59:11.144788475 (572h30m11.01587151s) [noise=true , address=45.91.116.9:4262 , piece_id=KXI3CRGDXPFIQMETUUKKJZS5YX3TY7XZJ6A2HCHO4JW3DNB2I43A, seg_pos={0 0}, upload=501910]: Done uploading piece. err: <nil> Dec 29 11:59:11.144812380 (572h30m11.015895415s) [seg_pos={0 0}, upload=501910]: Segment reached optimal threshold of 13 pieces. Dec 29 11:59:11.145162818 (572h30m11.016245863s) [noise=true , address=45.91.116.5:4298 , piece_id=TTGXECJ2OUJFDRSXM53VW6V2KJGT5ZTWKGEN6WOKN7RQZURKK62Q, seg_pos={0 0}, upload=501910]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 29 11:59:11.145173508 (572h30m11.016256543s) [noise=true , address=45.91.116.5:4298 , piece_id=TTGXECJ2OUJFDRSXM53VW6V2KJGT5ZTWKGEN6WOKN7RQZURKK62Q, seg_pos={0 0}, upload=501910]: Long tail cancellation due to optimal threshold Dec 29 11:59:11.145253922 (572h30m11.016336967s) [noise=true , address=145.79.227.30:4761 , piece_id=FWOGG3AETOEAFIFEBVASAVOTUITY7TVF3JVWTOHDCTXDGQMLQGVA, seg_pos={0 0}, upload=501910]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 29 11:59:11.145265453 (572h30m11.016348498s) [noise=true , address=145.79.227.30:4761 , piece_id=FWOGG3AETOEAFIFEBVASAVOTUITY7TVF3JVWTOHDCTXDGQMLQGVA, seg_pos={0 0}, upload=501910]: Long tail cancellation due to optimal threshold Dec 29 11:59:11.145333914 (572h30m11.016416949s) [noise=true , address=77.90.31.14:4661 , piece_id=XLSIJ2ZO6ZB7AFRAGBG7B3K5T4Q4EEGXKRUJ3QLQ5CNPRKSQ5N4A, seg_pos={0 0}, upload=501910]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Dec 29 11:59:11.145342329 (572h30m11.016425374s) [noise=true , address=77.90.31.14:4661 , piece_id=XLSIJ2ZO6ZB7AFRAGBG7B3K5T4Q4EEGXKRUJ3QLQ5CNPRKSQ5N4A, seg_pos={0 0}, upload=501910]: Long tail cancellation due to optimal threshold Dec 29 11:59:11.145359503 (572h30m11.016442538s) [seg_pos={0 0}, upload=501910]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 11:59:11.145451607 (572h30m11.016534652s) [upload=501910]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 11:59:11.145457439 (572h30m11.016540484s) [upload=501910]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 11:59:11.445524688 (572h30m11.316607733s) [upload=501910]: Done uploading segments... Dec 29 11:59:11.445548383 (572h30m11.316631429s) [upload=501910]: Upload finished. err: <nil> Duration: 6.224049281s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 31 out of 60 is failed (threshold=6) [X......X.X.XX.X.XX..X..XX.XXXX.XXXX...XXX...XXX..XX.X..X.X.X] duration: 5.000746517s trace: https://jaeger.dev.storj.io/trace/7f0c58f8d9eccf8c FAIL (2025-12-29 12:03:24 UTC) orbiter warning: context deadline exceeded Duration: 5.000766195s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........X..X...............................................] duration: 33.836841502s trace: https://jaeger.dev.storj.io/trace/7e7354d8023608a8 FAIL (2025-12-29 12:02:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== ) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480097317 (571h53m39.351180352s) [noise=true , address=77.90.31.21:4889 , piece_id=BR6Z55P3A7YJ5FJRDSZFUIF66FGIJQY7WCEQ7PL5DOYWL73OMKTQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480405053 (571h53m39.351488098s) [noise=true , address=77.90.31.16:4750 , piece_id=MVVZPJEGDCBSNVOB6CD2WUZGIJTL3VBHX2G4L3TVZ6JF5ZEJ5U6Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480712969 (571h53m39.351796014s) [noise=true , address=77.90.31.18:4834 , piece_id=BTWNGNQVEUEVV6HLBDM63SGNTKV4RDU4J4XFLG52WHKP53H3L5KA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481019453 (571h53m39.352102488s) [noise=true , address=145.79.227.30:4777 , piece_id=VSCXLXGIBSFE6E4OHLWNZIZ75PO7WQO2VFHUYHW7JUNH62KGGU7Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481029532 (571h53m39.352112567s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481559832 (571h53m39.352642877s) [noise=true , address=145.79.227.35:4515 , piece_id=L7LFZKZEZERUNX4HUNDNQ2PWHXNJRJ7OBFRNMFIICPZEUIBHGP7A, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481731919 (571h53m39.352814954s) [noise=true , address=64.49.13.26:4953 , piece_id=4YSW5Y3A6HYG3XJ2XAB7RGVHAOHMP7G3QIZYMYTSXVXYJ3ANLNXA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482078329 (571h53m39.353161364s) [noise=true , address=145.79.227.44:4205 , piece_id=UE4QSHDQBZ3XBWQ6I6MCRVTLMU7U6W6OZDA2ZGQCH6ZYRAGPD7WA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482099830 (571h53m39.353182865s) [noise=true , address=145.79.227.40:4067 , piece_id=WSYDRX5YJ6OCFW52ZVMM6WZYNRFUXMR4XJ6C5727BOFMR4RRH33Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482434487 (571h53m39.353517532s) [noise=true , address=145.79.227.3:4164 , piece_id=U6F7VDPOH7K6R5EH5UUXHFP55GXOEXO5BLBJHZRTITPDQI4LP7KQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482480845 (571h53m39.35356388s) [noise=true , address=64.49.13.140:4516 , piece_id=4GUYOYOAEASUK2W25S3NILWIIIHGOZX76L4EGFLUUUEDWRALPP2A, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482841442 (571h53m39.353924477s) [noise=true , address=145.79.227.34:4910 , piece_id=ZH4DCNXET2Q6VRAUH3NQ3KKSV4OYI575X7YS72XD236EEOMOG3LA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483163645 (571h53m39.35424668s) [noise=true , address=145.79.227.29:4974 , piece_id=KEKREOD2B35XQNI45JBHE2SZZC3MQQBQUR5I6WFDAQYJQGJBXSMA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483532478 (571h53m39.354615513s) [noise=true , address=77.90.31.17:4769 , piece_id=3MKUMOQIUVERKQLLIFX52XQPTQ5LSFMHIPD5YW2YTO5YEFR36IYA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483914596 (571h53m39.354997631s) [noise=true , address=145.79.227.13:4807 , piece_id=VKXB5MWAG2YYH2AWWJEGTLBZ2QUPLI25VTHI6AMMALSM23Z3AWZQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:40.761721106 (571h53m40.632804141s) [upload=501384]: Next returned nil segment Dec 29 11:22:40.761733349 (571h53m40.632816384s) [upload=501384]: Waiting for error group managing segments... Dec 29 11:22:41.121222862 (571h53m40.992305897s) [noise=true , address=64.49.13.140:4516 , piece_id=4GUYOYOAEASUK2W25S3NILWIIIHGOZX76L4EGFLUUUEDWRALPP2A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.197311646 (571h53m41.068394691s) [noise=true , address=64.49.13.26:4953 , piece_id=4YSW5Y3A6HYG3XJ2XAB7RGVHAOHMP7G3QIZYMYTSXVXYJ3ANLNXA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.347247568 (571h53m41.218330613s) [noise=true , address=145.79.227.13:4807 , piece_id=VKXB5MWAG2YYH2AWWJEGTLBZ2QUPLI25VTHI6AMMALSM23Z3AWZQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.347917313 (571h53m41.219000348s) [noise=true , address=145.79.227.3:4164 , piece_id=U6F7VDPOH7K6R5EH5UUXHFP55GXOEXO5BLBJHZRTITPDQI4LP7KQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.348510613 (571h53m41.219593648s) [noise=true , address=145.79.227.29:4974 , piece_id=KEKREOD2B35XQNI45JBHE2SZZC3MQQBQUR5I6WFDAQYJQGJBXSMA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.503950985 (571h53m41.37503402s) [noise=true , address=77.90.31.17:4769 , piece_id=3MKUMOQIUVERKQLLIFX52XQPTQ5LSFMHIPD5YW2YTO5YEFR36IYA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.561352025 (571h53m41.43243506s) [noise=true , address=77.90.31.18:4834 , piece_id=BTWNGNQVEUEVV6HLBDM63SGNTKV4RDU4J4XFLG52WHKP53H3L5KA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.622447380 (571h53m41.493530425s) [noise=true , address=77.90.31.16:4750 , piece_id=MVVZPJEGDCBSNVOB6CD2WUZGIJTL3VBHX2G4L3TVZ6JF5ZEJ5U6Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.645494685 (571h53m41.51657773s) [noise=true , address=77.90.31.21:4889 , piece_id=BR6Z55P3A7YJ5FJRDSZFUIF66FGIJQY7WCEQ7PL5DOYWL73OMKTQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.247930045 (571h53m42.11901308s) [noise=true , address=145.79.227.30:4777 , piece_id=VSCXLXGIBSFE6E4OHLWNZIZ75PO7WQO2VFHUYHW7JUNH62KGGU7Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.262221890 (571h53m42.133304925s) [noise=true , address=145.79.227.44:4205 , piece_id=UE4QSHDQBZ3XBWQ6I6MCRVTLMU7U6W6OZDA2ZGQCH6ZYRAGPD7WA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.295598123 (571h53m42.166681158s) [noise=true , address=145.79.227.35:4515 , piece_id=L7LFZKZEZERUNX4HUNDNQ2PWHXNJRJ7OBFRNMFIICPZEUIBHGP7A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.307481351 (571h53m42.178564386s) [noise=true , address=145.79.227.34:4910 , piece_id=ZH4DCNXET2Q6VRAUH3NQ3KKSV4OYI575X7YS72XD236EEOMOG3LA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.325775906 (571h53m42.196858941s) [noise=true , address=145.79.227.40:4067 , piece_id=WSYDRX5YJ6OCFW52ZVMM6WZYNRFUXMR4XJ6C5727BOFMR4RRH33Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.325865978 (571h53m42.196949023s) [seg_pos={0 0}, upload=501384]: Segment reached optimal threshold of 14 pieces. Dec 29 11:22:42.326137074 (571h53m42.197220109s) [noise=true , address=64.49.13.169:4708 , piece_id=B2X2RWRELXWJPF6AVJJZCNU46LM5ASGRE4EWGPKL3SCLQ4VKUBDQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection Dec 29 11:22:42.326153074 (571h53m42.197236109s) [noise=true , address=64.49.13.169:4708 , piece_id=B2X2RWRELXWJPF6AVJJZCNU46LM5ASGRE4EWGPKL3SCLQ4VKUBDQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326193401 (571h53m42.197276446s) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12h5esfg6mw2c8y3AVpeFBkL5FfLAwYdjhcnQ9CJ8fjh61WLYjx): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326205705 (571h53m42.19728874s) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326407739 (571h53m42.197490784s) [noise=true , address=77.90.31.10:4524 , piece_id=Q2JGEUV5EDFTVY2CSNPZOVJIIIAFBA3IHA7S5N7YZRH3E7TSTE5A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uN7CznoK4AD8FqFkNUndawdGJ9rmjAe7oNnjJkV844ScY87qe): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326415374 (571h53m42.197498419s) [noise=true , address=77.90.31.10:4524 , piece_id=Q2JGEUV5EDFTVY2CSNPZOVJIIIAFBA3IHA7S5N7YZRH3E7TSTE5A, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326505105 (571h53m42.19758815s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tnTEYvqJ6PB2DsuXA28sEuEiGj11StA4qdhcYH1kuoaMkTVdY): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326512188 (571h53m42.197595223s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326532357 (571h53m42.197615392s) [seg_pos={0 0}, upload=501384]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 11:22:42.326671562 (571h53m42.197754597s) [upload=501384]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 11:22:42.326677843 (571h53m42.197760879s) [upload=501384]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 11:22:42.610391438 (571h53m42.481474493s) [upload=501384]: Done uploading segments... Dec 29 11:22:42.610416826 (571h53m42.481499871s) [upload=501384]: Upload finished. err: <nil> Duration: 33.836862973s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 31 out of 60 is failed (threshold=6) [XX......X.X.XX.X.XX..X..XX.XXXX.XXXX...XXX...XXX..XX.X..X.X.] duration: 5.002054247s trace: https://jaeger.dev.storj.io/trace/6946781033b6e6de FAIL (2025-12-29 12:02:51 UTC) orbiter warning: context deadline exceeded Duration: 5.002077992s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [.X..XXX.X.......X.X..X.XXX...XX.XXX.X.X....XXXXXX..X.X......] duration: 6.207252602s trace: https://jaeger.dev.storj.io/trace/78a810c1312d39a9 FAIL (2025-12-29 12:02:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: TXY6KE2F3OOGRMOT4ZF3MERPEEXFOQONOJUJ2G5TDQQTT5TFGANA) context deadline exceeded; piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: 2PTAVS33GMBNXYULC2YFT6YTVLDSPBQOIYM2BHCXGYTTWLF2FNDQ) context deadline exceeded ======LOGS====== Dec 29 11:59:08.361853165 (572h30m8.2329362s) [upload=501910]: Starting upload Dec 29 11:59:08.361997270 (572h30m8.233080335s) [upload=501910]: Done starting upload Dec 29 11:59:08.362093513 (572h30m8.233176548s) [upload=501910]: Uploading segments... Dec 29 11:59:08.362292753 (572h30m8.233375798s) [upload=501910]: Got next segment. Inline: false Dec 29 11:59:08.362335234 (572h30m8.233418269s) [upload=501910]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 11:59:08.362340063 (572h30m8.233423098s) [upload=501910]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 11:59:08.611633275 (572h30m8.482716321s) [seg_pos={0 0}, upload=501910]: Begin upload segment... Dec 29 11:59:08.611679954 (572h30m8.482762999s) [seg_pos={0 0}, upload=501910]: Done begin upload segment. Dec 29 11:59:08.611705703 (572h30m8.482788738s) [noise=true , address=145.79.227.30:4761 , piece_id=FWOGG3AETOEAFIFEBVASAVOTUITY7TVF3JVWTOHDCTXDGQMLQGVA, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.611729619 (572h30m8.482812664s) [noise=true , address=45.91.116.14:4192 , piece_id=TXY6KE2F3OOGRMOT4ZF3MERPEEXFOQONOJUJ2G5TDQQTT5TFGANA, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.612625375 (572h30m8.48370841s) [noise=true , address=145.79.227.47:4266 , piece_id=XAZDWB62CWNFH7N5D4TGNXRLRT7YF2ZP4L6MNATUCZDDUXA2QDGQ, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.612648729 (572h30m8.483731774s) [noise=true , address=45.91.116.5:4298 , piece_id=TTGXECJ2OUJFDRSXM53VW6V2KJGT5ZTWKGEN6WOKN7RQZURKK62Q, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.613025066 (572h30m8.484108111s) [noise=true , address=45.91.116.9:4262 , piece_id=KXI3CRGDXPFIQMETUUKKJZS5YX3TY7XZJ6A2HCHO4JW3DNB2I43A, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.613521331 (572h30m8.484604376s) [noise=true , address=145.79.227.17:4970 , piece_id=PZIKWOOFEAAWSI3DDG6MN44EJ6RVL7HYC6HXLP3JGDVDRPPGORFQ, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.613528916 (572h30m8.484611961s) [noise=true , address=145.79.227.2:4115 , piece_id=VVEW5N7PMXNFLYMEIWFB6ZBTJWMBFK2PRWTFZKTUB5AUIPSOMCZA, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.613977430 (572h30m8.485060465s) [noise=true , address=145.79.227.46:4602 , piece_id=UO7XPJAI7VQXCF7SVBV43WTI6JPOK4DFQK3WMJ54EPXVSRWLA6QA, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.614275347 (572h30m8.485358382s) [noise=true , address=145.79.227.23:4545 , piece_id=XI7C54VR2CYZIE6OLAXP6UYWH2BQI5IFZLL6XFGMDPIF5UKKJUKA, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.614415964 (572h30m8.485499009s) [noise=true , address=77.90.31.14:4661 , piece_id=XLSIJ2ZO6ZB7AFRAGBG7B3K5T4Q4EEGXKRUJ3QLQ5CNPRKSQ5N4A, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.614940583 (572h30m8.486023618s) [noise=true , address=145.79.227.5:4222 , piece_id=UBA5YYPYYNUYRYHYURWMRXXB7Q6AMYZ3EZ3HZDLNF77OG4MDKEUQ, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.614969919 (572h30m8.486052964s) [noise=true , address=77.90.31.19:4842 , piece_id=2PTAVS33GMBNXYULC2YFT6YTVLDSPBQOIYM2BHCXGYTTWLF2FNDQ, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.615383867 (572h30m8.486466903s) [noise=true , address=145.79.227.14:4837 , piece_id=TE4VAA7KHS4YYQRGKMOMBM2WLVEMNB5NSML4JC7OY2EKHMDZQ6SA, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.615607825 (572h30m8.48669087s) [noise=true , address=145.79.227.21:4473 , piece_id=IDGJS4QYGBIORV26G45FWDSOA3H63UEEGPPPLLDN5RCFE3W4HAXA, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.615881095 (572h30m8.48696414s) [noise=true , address=145.79.227.26:4654 , piece_id=45IHGFKRVEDQDGQTSFALHRIE6IVQZHY7E3DCIZI6TNHLJFITPGQQ, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.616226894 (572h30m8.487309939s) [noise=true , address=145.79.227.50:4374 , piece_id=L2USZXC5XKVWUDSZ5MDLOZVAFJOMVGYF7SM2E5Q4EAU4EEZEF23Q, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:10.218180372 (572h30m10.089263407s) [upload=501910]: Next returned nil segment Dec 29 11:59:10.218197525 (572h30m10.08928056s) [upload=501910]: Waiting for error group managing segments... Dec 29 11:59:10.596756931 (572h30m10.467839966s) [noise=true , address=145.79.227.17:4970 , piece_id=PZIKWOOFEAAWSI3DDG6MN44EJ6RVL7HYC6HXLP3JGDVDRPPGORFQ, seg_pos={0 0}, upload=501910]: Done uploading piece. err: <nil> Dec 29 11:59:10.597714194 (572h30m10.46879724s) [noise=true , address=145.79.227.47:4266 , piece_id=XAZDWB62CWNFH7N5D4TGNXRLRT7YF2ZP4L6MNATUCZDDUXA2QDGQ, seg_pos={0 0}, upload=501910]: Done uploading piece. err: <nil> Dec 29 11:59:10.600236358 (572h30m10.471319393s) [noise=true , address=145.79.227.14:4837 , piece_id=TE4VAA7KHS4YYQRGKMOMBM2WLVEMNB5NSML4JC7OY2EKHMDZQ6SA, seg_pos={0 0}, upload=501910]: Done uploading piece. err: <nil> Dec 29 11:59:10.602320044 (572h30m10.47340308s) [noise=true , address=145.79.227.5:4222 , piece_id=UBA5YYPYYNUYRYHYURWMRXXB7Q6AMYZ3EZ3HZDLNF77OG4MDKEUQ, seg_pos={0 0}, upload=501910]: Done uploading piece. err: <nil> Dec 29 11:59:10.602912464 (572h30m10.473995499s) [noise=true , address=145.79.227.26:4654 , piece_id=45IHGFKRVEDQDGQTSFALHRIE6IVQZHY7E3DCIZI6TNHLJFITPGQQ, seg_pos={0 0}, upload=501910]: Done uploading piece. err: <nil> Dec 29 11:59:10.611761013 (572h30m10.482844048s) [noise=true , address=145.79.227.23:4545 , piece_id=XI7C54VR2CYZIE6OLAXP6UYWH2BQI5IFZLL6XFGMDPIF5UKKJUKA, seg_pos={0 0}, upload=501910]: Done uploading piece. err: <nil> Dec 29 11:59:10.623036624 (572h30m10.494119669s) [noise=true , address=145.79.227.46:4602 , piece_id=UO7XPJAI7VQXCF7SVBV43WTI6JPOK4DFQK3WMJ54EPXVSRWLA6QA, seg_pos={0 0}, upload=501910]: Done uploading piece. err: <nil> Dec 29 11:59:10.626994862 (572h30m10.498077907s) [noise=true , address=145.79.227.50:4374 , piece_id=L2USZXC5XKVWUDSZ5MDLOZVAFJOMVGYF7SM2E5Q4EAU4EEZEF23Q, seg_pos={0 0}, upload=501910]: Done uploading piece. err: <nil> Dec 29 11:59:10.627968646 (572h30m10.499051691s) [noise=true , address=145.79.227.21:4473 , piece_id=IDGJS4QYGBIORV26G45FWDSOA3H63UEEGPPPLLDN5RCFE3W4HAXA, seg_pos={0 0}, upload=501910]: Done uploading piece. err: <nil> Dec 29 11:59:10.629745359 (572h30m10.500828404s) [noise=true , address=145.79.227.2:4115 , piece_id=VVEW5N7PMXNFLYMEIWFB6ZBTJWMBFK2PRWTFZKTUB5AUIPSOMCZA, seg_pos={0 0}, upload=501910]: Done uploading piece. err: <nil> Dec 29 11:59:10.847903124 (572h30m10.718986159s) [noise=true , address=45.91.116.14:4192 , piece_id=TXY6KE2F3OOGRMOT4ZF3MERPEEXFOQONOJUJ2G5TDQQTT5TFGANA, seg_pos={0 0}, upload=501910]: Done uploading piece. err: <nil> Dec 29 11:59:10.901405164 (572h30m10.772488199s) [noise=true , address=77.90.31.19:4842 , piece_id=2PTAVS33GMBNXYULC2YFT6YTVLDSPBQOIYM2BHCXGYTTWLF2FNDQ, seg_pos={0 0}, upload=501910]: Done uploading piece. err: <nil> Dec 29 11:59:11.144788475 (572h30m11.01587151s) [noise=true , address=45.91.116.9:4262 , piece_id=KXI3CRGDXPFIQMETUUKKJZS5YX3TY7XZJ6A2HCHO4JW3DNB2I43A, seg_pos={0 0}, upload=501910]: Done uploading piece. err: <nil> Dec 29 11:59:11.144812380 (572h30m11.015895415s) [seg_pos={0 0}, upload=501910]: Segment reached optimal threshold of 13 pieces. Dec 29 11:59:11.145162818 (572h30m11.016245863s) [noise=true , address=45.91.116.5:4298 , piece_id=TTGXECJ2OUJFDRSXM53VW6V2KJGT5ZTWKGEN6WOKN7RQZURKK62Q, seg_pos={0 0}, upload=501910]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 29 11:59:11.145173508 (572h30m11.016256543s) [noise=true , address=45.91.116.5:4298 , piece_id=TTGXECJ2OUJFDRSXM53VW6V2KJGT5ZTWKGEN6WOKN7RQZURKK62Q, seg_pos={0 0}, upload=501910]: Long tail cancellation due to optimal threshold Dec 29 11:59:11.145253922 (572h30m11.016336967s) [noise=true , address=145.79.227.30:4761 , piece_id=FWOGG3AETOEAFIFEBVASAVOTUITY7TVF3JVWTOHDCTXDGQMLQGVA, seg_pos={0 0}, upload=501910]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 29 11:59:11.145265453 (572h30m11.016348498s) [noise=true , address=145.79.227.30:4761 , piece_id=FWOGG3AETOEAFIFEBVASAVOTUITY7TVF3JVWTOHDCTXDGQMLQGVA, seg_pos={0 0}, upload=501910]: Long tail cancellation due to optimal threshold Dec 29 11:59:11.145333914 (572h30m11.016416949s) [noise=true , address=77.90.31.14:4661 , piece_id=XLSIJ2ZO6ZB7AFRAGBG7B3K5T4Q4EEGXKRUJ3QLQ5CNPRKSQ5N4A, seg_pos={0 0}, upload=501910]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Dec 29 11:59:11.145342329 (572h30m11.016425374s) [noise=true , address=77.90.31.14:4661 , piece_id=XLSIJ2ZO6ZB7AFRAGBG7B3K5T4Q4EEGXKRUJ3QLQ5CNPRKSQ5N4A, seg_pos={0 0}, upload=501910]: Long tail cancellation due to optimal threshold Dec 29 11:59:11.145359503 (572h30m11.016442538s) [seg_pos={0 0}, upload=501910]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 11:59:11.145451607 (572h30m11.016534652s) [upload=501910]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 11:59:11.145457439 (572h30m11.016540484s) [upload=501910]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 11:59:11.445524688 (572h30m11.316607733s) [upload=501910]: Done uploading segments... Dec 29 11:59:11.445548383 (572h30m11.316631429s) [upload=501910]: Upload finished. err: <nil> Duration: 6.207280475s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........X..X..............................................] duration: 39.706816344s trace: https://jaeger.dev.storj.io/trace/7e7354d8023608a8 FAIL (2025-12-29 12:02:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== ) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480097317 (571h53m39.351180352s) [noise=true , address=77.90.31.21:4889 , piece_id=BR6Z55P3A7YJ5FJRDSZFUIF66FGIJQY7WCEQ7PL5DOYWL73OMKTQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480405053 (571h53m39.351488098s) [noise=true , address=77.90.31.16:4750 , piece_id=MVVZPJEGDCBSNVOB6CD2WUZGIJTL3VBHX2G4L3TVZ6JF5ZEJ5U6Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480712969 (571h53m39.351796014s) [noise=true , address=77.90.31.18:4834 , piece_id=BTWNGNQVEUEVV6HLBDM63SGNTKV4RDU4J4XFLG52WHKP53H3L5KA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481019453 (571h53m39.352102488s) [noise=true , address=145.79.227.30:4777 , piece_id=VSCXLXGIBSFE6E4OHLWNZIZ75PO7WQO2VFHUYHW7JUNH62KGGU7Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481029532 (571h53m39.352112567s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481559832 (571h53m39.352642877s) [noise=true , address=145.79.227.35:4515 , piece_id=L7LFZKZEZERUNX4HUNDNQ2PWHXNJRJ7OBFRNMFIICPZEUIBHGP7A, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481731919 (571h53m39.352814954s) [noise=true , address=64.49.13.26:4953 , piece_id=4YSW5Y3A6HYG3XJ2XAB7RGVHAOHMP7G3QIZYMYTSXVXYJ3ANLNXA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482078329 (571h53m39.353161364s) [noise=true , address=145.79.227.44:4205 , piece_id=UE4QSHDQBZ3XBWQ6I6MCRVTLMU7U6W6OZDA2ZGQCH6ZYRAGPD7WA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482099830 (571h53m39.353182865s) [noise=true , address=145.79.227.40:4067 , piece_id=WSYDRX5YJ6OCFW52ZVMM6WZYNRFUXMR4XJ6C5727BOFMR4RRH33Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482434487 (571h53m39.353517532s) [noise=true , address=145.79.227.3:4164 , piece_id=U6F7VDPOH7K6R5EH5UUXHFP55GXOEXO5BLBJHZRTITPDQI4LP7KQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482480845 (571h53m39.35356388s) [noise=true , address=64.49.13.140:4516 , piece_id=4GUYOYOAEASUK2W25S3NILWIIIHGOZX76L4EGFLUUUEDWRALPP2A, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482841442 (571h53m39.353924477s) [noise=true , address=145.79.227.34:4910 , piece_id=ZH4DCNXET2Q6VRAUH3NQ3KKSV4OYI575X7YS72XD236EEOMOG3LA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483163645 (571h53m39.35424668s) [noise=true , address=145.79.227.29:4974 , piece_id=KEKREOD2B35XQNI45JBHE2SZZC3MQQBQUR5I6WFDAQYJQGJBXSMA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483532478 (571h53m39.354615513s) [noise=true , address=77.90.31.17:4769 , piece_id=3MKUMOQIUVERKQLLIFX52XQPTQ5LSFMHIPD5YW2YTO5YEFR36IYA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483914596 (571h53m39.354997631s) [noise=true , address=145.79.227.13:4807 , piece_id=VKXB5MWAG2YYH2AWWJEGTLBZ2QUPLI25VTHI6AMMALSM23Z3AWZQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:40.761721106 (571h53m40.632804141s) [upload=501384]: Next returned nil segment Dec 29 11:22:40.761733349 (571h53m40.632816384s) [upload=501384]: Waiting for error group managing segments... Dec 29 11:22:41.121222862 (571h53m40.992305897s) [noise=true , address=64.49.13.140:4516 , piece_id=4GUYOYOAEASUK2W25S3NILWIIIHGOZX76L4EGFLUUUEDWRALPP2A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.197311646 (571h53m41.068394691s) [noise=true , address=64.49.13.26:4953 , piece_id=4YSW5Y3A6HYG3XJ2XAB7RGVHAOHMP7G3QIZYMYTSXVXYJ3ANLNXA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.347247568 (571h53m41.218330613s) [noise=true , address=145.79.227.13:4807 , piece_id=VKXB5MWAG2YYH2AWWJEGTLBZ2QUPLI25VTHI6AMMALSM23Z3AWZQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.347917313 (571h53m41.219000348s) [noise=true , address=145.79.227.3:4164 , piece_id=U6F7VDPOH7K6R5EH5UUXHFP55GXOEXO5BLBJHZRTITPDQI4LP7KQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.348510613 (571h53m41.219593648s) [noise=true , address=145.79.227.29:4974 , piece_id=KEKREOD2B35XQNI45JBHE2SZZC3MQQBQUR5I6WFDAQYJQGJBXSMA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.503950985 (571h53m41.37503402s) [noise=true , address=77.90.31.17:4769 , piece_id=3MKUMOQIUVERKQLLIFX52XQPTQ5LSFMHIPD5YW2YTO5YEFR36IYA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.561352025 (571h53m41.43243506s) [noise=true , address=77.90.31.18:4834 , piece_id=BTWNGNQVEUEVV6HLBDM63SGNTKV4RDU4J4XFLG52WHKP53H3L5KA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.622447380 (571h53m41.493530425s) [noise=true , address=77.90.31.16:4750 , piece_id=MVVZPJEGDCBSNVOB6CD2WUZGIJTL3VBHX2G4L3TVZ6JF5ZEJ5U6Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.645494685 (571h53m41.51657773s) [noise=true , address=77.90.31.21:4889 , piece_id=BR6Z55P3A7YJ5FJRDSZFUIF66FGIJQY7WCEQ7PL5DOYWL73OMKTQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.247930045 (571h53m42.11901308s) [noise=true , address=145.79.227.30:4777 , piece_id=VSCXLXGIBSFE6E4OHLWNZIZ75PO7WQO2VFHUYHW7JUNH62KGGU7Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.262221890 (571h53m42.133304925s) [noise=true , address=145.79.227.44:4205 , piece_id=UE4QSHDQBZ3XBWQ6I6MCRVTLMU7U6W6OZDA2ZGQCH6ZYRAGPD7WA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.295598123 (571h53m42.166681158s) [noise=true , address=145.79.227.35:4515 , piece_id=L7LFZKZEZERUNX4HUNDNQ2PWHXNJRJ7OBFRNMFIICPZEUIBHGP7A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.307481351 (571h53m42.178564386s) [noise=true , address=145.79.227.34:4910 , piece_id=ZH4DCNXET2Q6VRAUH3NQ3KKSV4OYI575X7YS72XD236EEOMOG3LA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.325775906 (571h53m42.196858941s) [noise=true , address=145.79.227.40:4067 , piece_id=WSYDRX5YJ6OCFW52ZVMM6WZYNRFUXMR4XJ6C5727BOFMR4RRH33Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.325865978 (571h53m42.196949023s) [seg_pos={0 0}, upload=501384]: Segment reached optimal threshold of 14 pieces. Dec 29 11:22:42.326137074 (571h53m42.197220109s) [noise=true , address=64.49.13.169:4708 , piece_id=B2X2RWRELXWJPF6AVJJZCNU46LM5ASGRE4EWGPKL3SCLQ4VKUBDQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection Dec 29 11:22:42.326153074 (571h53m42.197236109s) [noise=true , address=64.49.13.169:4708 , piece_id=B2X2RWRELXWJPF6AVJJZCNU46LM5ASGRE4EWGPKL3SCLQ4VKUBDQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326193401 (571h53m42.197276446s) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12h5esfg6mw2c8y3AVpeFBkL5FfLAwYdjhcnQ9CJ8fjh61WLYjx): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326205705 (571h53m42.19728874s) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326407739 (571h53m42.197490784s) [noise=true , address=77.90.31.10:4524 , piece_id=Q2JGEUV5EDFTVY2CSNPZOVJIIIAFBA3IHA7S5N7YZRH3E7TSTE5A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uN7CznoK4AD8FqFkNUndawdGJ9rmjAe7oNnjJkV844ScY87qe): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326415374 (571h53m42.197498419s) [noise=true , address=77.90.31.10:4524 , piece_id=Q2JGEUV5EDFTVY2CSNPZOVJIIIAFBA3IHA7S5N7YZRH3E7TSTE5A, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326505105 (571h53m42.19758815s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tnTEYvqJ6PB2DsuXA28sEuEiGj11StA4qdhcYH1kuoaMkTVdY): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326512188 (571h53m42.197595223s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326532357 (571h53m42.197615392s) [seg_pos={0 0}, upload=501384]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 11:22:42.326671562 (571h53m42.197754597s) [upload=501384]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 11:22:42.326677843 (571h53m42.197760879s) [upload=501384]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 11:22:42.610391438 (571h53m42.481474493s) [upload=501384]: Done uploading segments... Dec 29 11:22:42.610416826 (571h53m42.481499871s) [upload=501384]: Upload finished. err: <nil> Duration: 39.706833016s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [..X..XXX.X.......X.X..X.XXX...XX.XXX.X.X....XXXXXX..X.X.....] duration: 6.779996513s trace: https://jaeger.dev.storj.io/trace/78a810c1312d39a9 FAIL (2025-12-29 12:02:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: TXY6KE2F3OOGRMOT4ZF3MERPEEXFOQONOJUJ2G5TDQQTT5TFGANA) context deadline exceeded; piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: 2PTAVS33GMBNXYULC2YFT6YTVLDSPBQOIYM2BHCXGYTTWLF2FNDQ) context deadline exceeded ======LOGS====== Dec 29 11:59:08.361853165 (572h30m8.2329362s) [upload=501910]: Starting upload Dec 29 11:59:08.361997270 (572h30m8.233080335s) [upload=501910]: Done starting upload Dec 29 11:59:08.362093513 (572h30m8.233176548s) [upload=501910]: Uploading segments... Dec 29 11:59:08.362292753 (572h30m8.233375798s) [upload=501910]: Got next segment. Inline: false Dec 29 11:59:08.362335234 (572h30m8.233418269s) [upload=501910]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 11:59:08.362340063 (572h30m8.233423098s) [upload=501910]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 11:59:08.611633275 (572h30m8.482716321s) [seg_pos={0 0}, upload=501910]: Begin upload segment... Dec 29 11:59:08.611679954 (572h30m8.482762999s) [seg_pos={0 0}, upload=501910]: Done begin upload segment. Dec 29 11:59:08.611705703 (572h30m8.482788738s) [noise=true , address=145.79.227.30:4761 , piece_id=FWOGG3AETOEAFIFEBVASAVOTUITY7TVF3JVWTOHDCTXDGQMLQGVA, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.611729619 (572h30m8.482812664s) [noise=true , address=45.91.116.14:4192 , piece_id=TXY6KE2F3OOGRMOT4ZF3MERPEEXFOQONOJUJ2G5TDQQTT5TFGANA, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.612625375 (572h30m8.48370841s) [noise=true , address=145.79.227.47:4266 , piece_id=XAZDWB62CWNFH7N5D4TGNXRLRT7YF2ZP4L6MNATUCZDDUXA2QDGQ, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.612648729 (572h30m8.483731774s) [noise=true , address=45.91.116.5:4298 , piece_id=TTGXECJ2OUJFDRSXM53VW6V2KJGT5ZTWKGEN6WOKN7RQZURKK62Q, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.613025066 (572h30m8.484108111s) [noise=true , address=45.91.116.9:4262 , piece_id=KXI3CRGDXPFIQMETUUKKJZS5YX3TY7XZJ6A2HCHO4JW3DNB2I43A, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.613521331 (572h30m8.484604376s) [noise=true , address=145.79.227.17:4970 , piece_id=PZIKWOOFEAAWSI3DDG6MN44EJ6RVL7HYC6HXLP3JGDVDRPPGORFQ, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.613528916 (572h30m8.484611961s) [noise=true , address=145.79.227.2:4115 , piece_id=VVEW5N7PMXNFLYMEIWFB6ZBTJWMBFK2PRWTFZKTUB5AUIPSOMCZA, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.613977430 (572h30m8.485060465s) [noise=true , address=145.79.227.46:4602 , piece_id=UO7XPJAI7VQXCF7SVBV43WTI6JPOK4DFQK3WMJ54EPXVSRWLA6QA, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.614275347 (572h30m8.485358382s) [noise=true , address=145.79.227.23:4545 , piece_id=XI7C54VR2CYZIE6OLAXP6UYWH2BQI5IFZLL6XFGMDPIF5UKKJUKA, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.614415964 (572h30m8.485499009s) [noise=true , address=77.90.31.14:4661 , piece_id=XLSIJ2ZO6ZB7AFRAGBG7B3K5T4Q4EEGXKRUJ3QLQ5CNPRKSQ5N4A, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.614940583 (572h30m8.486023618s) [noise=true , address=145.79.227.5:4222 , piece_id=UBA5YYPYYNUYRYHYURWMRXXB7Q6AMYZ3EZ3HZDLNF77OG4MDKEUQ, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.614969919 (572h30m8.486052964s) [noise=true , address=77.90.31.19:4842 , piece_id=2PTAVS33GMBNXYULC2YFT6YTVLDSPBQOIYM2BHCXGYTTWLF2FNDQ, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.615383867 (572h30m8.486466903s) [noise=true , address=145.79.227.14:4837 , piece_id=TE4VAA7KHS4YYQRGKMOMBM2WLVEMNB5NSML4JC7OY2EKHMDZQ6SA, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.615607825 (572h30m8.48669087s) [noise=true , address=145.79.227.21:4473 , piece_id=IDGJS4QYGBIORV26G45FWDSOA3H63UEEGPPPLLDN5RCFE3W4HAXA, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.615881095 (572h30m8.48696414s) [noise=true , address=145.79.227.26:4654 , piece_id=45IHGFKRVEDQDGQTSFALHRIE6IVQZHY7E3DCIZI6TNHLJFITPGQQ, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.616226894 (572h30m8.487309939s) [noise=true , address=145.79.227.50:4374 , piece_id=L2USZXC5XKVWUDSZ5MDLOZVAFJOMVGYF7SM2E5Q4EAU4EEZEF23Q, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:10.218180372 (572h30m10.089263407s) [upload=501910]: Next returned nil segment Dec 29 11:59:10.218197525 (572h30m10.08928056s) [upload=501910]: Waiting for error group managing segments... Dec 29 11:59:10.596756931 (572h30m10.467839966s) [noise=true , address=145.79.227.17:4970 , piece_id=PZIKWOOFEAAWSI3DDG6MN44EJ6RVL7HYC6HXLP3JGDVDRPPGORFQ, seg_pos={0 0}, upload=501910]: Done uploading piece. err: <nil> Dec 29 11:59:10.597714194 (572h30m10.46879724s) [noise=true , address=145.79.227.47:4266 , piece_id=XAZDWB62CWNFH7N5D4TGNXRLRT7YF2ZP4L6MNATUCZDDUXA2QDGQ, seg_pos={0 0}, upload=501910]: Done uploading piece. err: <nil> Dec 29 11:59:10.600236358 (572h30m10.471319393s) [noise=true , address=145.79.227.14:4837 , piece_id=TE4VAA7KHS4YYQRGKMOMBM2WLVEMNB5NSML4JC7OY2EKHMDZQ6SA, seg_pos={0 0}, upload=501910]: Done uploading piece. err: <nil> Dec 29 11:59:10.602320044 (572h30m10.47340308s) [noise=true , address=145.79.227.5:4222 , piece_id=UBA5YYPYYNUYRYHYURWMRXXB7Q6AMYZ3EZ3HZDLNF77OG4MDKEUQ, seg_pos={0 0}, upload=501910]: Done uploading piece. err: <nil> Dec 29 11:59:10.602912464 (572h30m10.473995499s) [noise=true , address=145.79.227.26:4654 , piece_id=45IHGFKRVEDQDGQTSFALHRIE6IVQZHY7E3DCIZI6TNHLJFITPGQQ, seg_pos={0 0}, upload=501910]: Done uploading piece. err: <nil> Dec 29 11:59:10.611761013 (572h30m10.482844048s) [noise=true , address=145.79.227.23:4545 , piece_id=XI7C54VR2CYZIE6OLAXP6UYWH2BQI5IFZLL6XFGMDPIF5UKKJUKA, seg_pos={0 0}, upload=501910]: Done uploading piece. err: <nil> Dec 29 11:59:10.623036624 (572h30m10.494119669s) [noise=true , address=145.79.227.46:4602 , piece_id=UO7XPJAI7VQXCF7SVBV43WTI6JPOK4DFQK3WMJ54EPXVSRWLA6QA, seg_pos={0 0}, upload=501910]: Done uploading piece. err: <nil> Dec 29 11:59:10.626994862 (572h30m10.498077907s) [noise=true , address=145.79.227.50:4374 , piece_id=L2USZXC5XKVWUDSZ5MDLOZVAFJOMVGYF7SM2E5Q4EAU4EEZEF23Q, seg_pos={0 0}, upload=501910]: Done uploading piece. err: <nil> Dec 29 11:59:10.627968646 (572h30m10.499051691s) [noise=true , address=145.79.227.21:4473 , piece_id=IDGJS4QYGBIORV26G45FWDSOA3H63UEEGPPPLLDN5RCFE3W4HAXA, seg_pos={0 0}, upload=501910]: Done uploading piece. err: <nil> Dec 29 11:59:10.629745359 (572h30m10.500828404s) [noise=true , address=145.79.227.2:4115 , piece_id=VVEW5N7PMXNFLYMEIWFB6ZBTJWMBFK2PRWTFZKTUB5AUIPSOMCZA, seg_pos={0 0}, upload=501910]: Done uploading piece. err: <nil> Dec 29 11:59:10.847903124 (572h30m10.718986159s) [noise=true , address=45.91.116.14:4192 , piece_id=TXY6KE2F3OOGRMOT4ZF3MERPEEXFOQONOJUJ2G5TDQQTT5TFGANA, seg_pos={0 0}, upload=501910]: Done uploading piece. err: <nil> Dec 29 11:59:10.901405164 (572h30m10.772488199s) [noise=true , address=77.90.31.19:4842 , piece_id=2PTAVS33GMBNXYULC2YFT6YTVLDSPBQOIYM2BHCXGYTTWLF2FNDQ, seg_pos={0 0}, upload=501910]: Done uploading piece. err: <nil> Dec 29 11:59:11.144788475 (572h30m11.01587151s) [noise=true , address=45.91.116.9:4262 , piece_id=KXI3CRGDXPFIQMETUUKKJZS5YX3TY7XZJ6A2HCHO4JW3DNB2I43A, seg_pos={0 0}, upload=501910]: Done uploading piece. err: <nil> Dec 29 11:59:11.144812380 (572h30m11.015895415s) [seg_pos={0 0}, upload=501910]: Segment reached optimal threshold of 13 pieces. Dec 29 11:59:11.145162818 (572h30m11.016245863s) [noise=true , address=45.91.116.5:4298 , piece_id=TTGXECJ2OUJFDRSXM53VW6V2KJGT5ZTWKGEN6WOKN7RQZURKK62Q, seg_pos={0 0}, upload=501910]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 29 11:59:11.145173508 (572h30m11.016256543s) [noise=true , address=45.91.116.5:4298 , piece_id=TTGXECJ2OUJFDRSXM53VW6V2KJGT5ZTWKGEN6WOKN7RQZURKK62Q, seg_pos={0 0}, upload=501910]: Long tail cancellation due to optimal threshold Dec 29 11:59:11.145253922 (572h30m11.016336967s) [noise=true , address=145.79.227.30:4761 , piece_id=FWOGG3AETOEAFIFEBVASAVOTUITY7TVF3JVWTOHDCTXDGQMLQGVA, seg_pos={0 0}, upload=501910]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 29 11:59:11.145265453 (572h30m11.016348498s) [noise=true , address=145.79.227.30:4761 , piece_id=FWOGG3AETOEAFIFEBVASAVOTUITY7TVF3JVWTOHDCTXDGQMLQGVA, seg_pos={0 0}, upload=501910]: Long tail cancellation due to optimal threshold Dec 29 11:59:11.145333914 (572h30m11.016416949s) [noise=true , address=77.90.31.14:4661 , piece_id=XLSIJ2ZO6ZB7AFRAGBG7B3K5T4Q4EEGXKRUJ3QLQ5CNPRKSQ5N4A, seg_pos={0 0}, upload=501910]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Dec 29 11:59:11.145342329 (572h30m11.016425374s) [noise=true , address=77.90.31.14:4661 , piece_id=XLSIJ2ZO6ZB7AFRAGBG7B3K5T4Q4EEGXKRUJ3QLQ5CNPRKSQ5N4A, seg_pos={0 0}, upload=501910]: Long tail cancellation due to optimal threshold Dec 29 11:59:11.145359503 (572h30m11.016442538s) [seg_pos={0 0}, upload=501910]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 11:59:11.145451607 (572h30m11.016534652s) [upload=501910]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 11:59:11.145457439 (572h30m11.016540484s) [upload=501910]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 11:59:11.445524688 (572h30m11.316607733s) [upload=501910]: Done uploading segments... Dec 29 11:59:11.445548383 (572h30m11.316631429s) [upload=501910]: Upload finished. err: <nil> Duration: 6.780025678s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 31 out of 60 is failed (threshold=6) [.XX......X.X.XX.X.XX..X..XX.XXXX.XXXX...XXX...XXX..XX.X..X.X] duration: 5.000812575s trace: https://jaeger.dev.storj.io/trace/6946781033b6e6de FAIL (2025-12-29 12:01:48 UTC) orbiter warning: context deadline exceeded Duration: 5.000827923s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........X..X.............................................] duration: 35.847739138s trace: https://jaeger.dev.storj.io/trace/7e7354d8023608a8 FAIL (2025-12-29 12:01:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== ) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480097317 (571h53m39.351180352s) [noise=true , address=77.90.31.21:4889 , piece_id=BR6Z55P3A7YJ5FJRDSZFUIF66FGIJQY7WCEQ7PL5DOYWL73OMKTQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480405053 (571h53m39.351488098s) [noise=true , address=77.90.31.16:4750 , piece_id=MVVZPJEGDCBSNVOB6CD2WUZGIJTL3VBHX2G4L3TVZ6JF5ZEJ5U6Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480712969 (571h53m39.351796014s) [noise=true , address=77.90.31.18:4834 , piece_id=BTWNGNQVEUEVV6HLBDM63SGNTKV4RDU4J4XFLG52WHKP53H3L5KA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481019453 (571h53m39.352102488s) [noise=true , address=145.79.227.30:4777 , piece_id=VSCXLXGIBSFE6E4OHLWNZIZ75PO7WQO2VFHUYHW7JUNH62KGGU7Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481029532 (571h53m39.352112567s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481559832 (571h53m39.352642877s) [noise=true , address=145.79.227.35:4515 , piece_id=L7LFZKZEZERUNX4HUNDNQ2PWHXNJRJ7OBFRNMFIICPZEUIBHGP7A, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481731919 (571h53m39.352814954s) [noise=true , address=64.49.13.26:4953 , piece_id=4YSW5Y3A6HYG3XJ2XAB7RGVHAOHMP7G3QIZYMYTSXVXYJ3ANLNXA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482078329 (571h53m39.353161364s) [noise=true , address=145.79.227.44:4205 , piece_id=UE4QSHDQBZ3XBWQ6I6MCRVTLMU7U6W6OZDA2ZGQCH6ZYRAGPD7WA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482099830 (571h53m39.353182865s) [noise=true , address=145.79.227.40:4067 , piece_id=WSYDRX5YJ6OCFW52ZVMM6WZYNRFUXMR4XJ6C5727BOFMR4RRH33Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482434487 (571h53m39.353517532s) [noise=true , address=145.79.227.3:4164 , piece_id=U6F7VDPOH7K6R5EH5UUXHFP55GXOEXO5BLBJHZRTITPDQI4LP7KQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482480845 (571h53m39.35356388s) [noise=true , address=64.49.13.140:4516 , piece_id=4GUYOYOAEASUK2W25S3NILWIIIHGOZX76L4EGFLUUUEDWRALPP2A, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482841442 (571h53m39.353924477s) [noise=true , address=145.79.227.34:4910 , piece_id=ZH4DCNXET2Q6VRAUH3NQ3KKSV4OYI575X7YS72XD236EEOMOG3LA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483163645 (571h53m39.35424668s) [noise=true , address=145.79.227.29:4974 , piece_id=KEKREOD2B35XQNI45JBHE2SZZC3MQQBQUR5I6WFDAQYJQGJBXSMA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483532478 (571h53m39.354615513s) [noise=true , address=77.90.31.17:4769 , piece_id=3MKUMOQIUVERKQLLIFX52XQPTQ5LSFMHIPD5YW2YTO5YEFR36IYA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483914596 (571h53m39.354997631s) [noise=true , address=145.79.227.13:4807 , piece_id=VKXB5MWAG2YYH2AWWJEGTLBZ2QUPLI25VTHI6AMMALSM23Z3AWZQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:40.761721106 (571h53m40.632804141s) [upload=501384]: Next returned nil segment Dec 29 11:22:40.761733349 (571h53m40.632816384s) [upload=501384]: Waiting for error group managing segments... Dec 29 11:22:41.121222862 (571h53m40.992305897s) [noise=true , address=64.49.13.140:4516 , piece_id=4GUYOYOAEASUK2W25S3NILWIIIHGOZX76L4EGFLUUUEDWRALPP2A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.197311646 (571h53m41.068394691s) [noise=true , address=64.49.13.26:4953 , piece_id=4YSW5Y3A6HYG3XJ2XAB7RGVHAOHMP7G3QIZYMYTSXVXYJ3ANLNXA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.347247568 (571h53m41.218330613s) [noise=true , address=145.79.227.13:4807 , piece_id=VKXB5MWAG2YYH2AWWJEGTLBZ2QUPLI25VTHI6AMMALSM23Z3AWZQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.347917313 (571h53m41.219000348s) [noise=true , address=145.79.227.3:4164 , piece_id=U6F7VDPOH7K6R5EH5UUXHFP55GXOEXO5BLBJHZRTITPDQI4LP7KQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.348510613 (571h53m41.219593648s) [noise=true , address=145.79.227.29:4974 , piece_id=KEKREOD2B35XQNI45JBHE2SZZC3MQQBQUR5I6WFDAQYJQGJBXSMA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.503950985 (571h53m41.37503402s) [noise=true , address=77.90.31.17:4769 , piece_id=3MKUMOQIUVERKQLLIFX52XQPTQ5LSFMHIPD5YW2YTO5YEFR36IYA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.561352025 (571h53m41.43243506s) [noise=true , address=77.90.31.18:4834 , piece_id=BTWNGNQVEUEVV6HLBDM63SGNTKV4RDU4J4XFLG52WHKP53H3L5KA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.622447380 (571h53m41.493530425s) [noise=true , address=77.90.31.16:4750 , piece_id=MVVZPJEGDCBSNVOB6CD2WUZGIJTL3VBHX2G4L3TVZ6JF5ZEJ5U6Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.645494685 (571h53m41.51657773s) [noise=true , address=77.90.31.21:4889 , piece_id=BR6Z55P3A7YJ5FJRDSZFUIF66FGIJQY7WCEQ7PL5DOYWL73OMKTQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.247930045 (571h53m42.11901308s) [noise=true , address=145.79.227.30:4777 , piece_id=VSCXLXGIBSFE6E4OHLWNZIZ75PO7WQO2VFHUYHW7JUNH62KGGU7Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.262221890 (571h53m42.133304925s) [noise=true , address=145.79.227.44:4205 , piece_id=UE4QSHDQBZ3XBWQ6I6MCRVTLMU7U6W6OZDA2ZGQCH6ZYRAGPD7WA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.295598123 (571h53m42.166681158s) [noise=true , address=145.79.227.35:4515 , piece_id=L7LFZKZEZERUNX4HUNDNQ2PWHXNJRJ7OBFRNMFIICPZEUIBHGP7A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.307481351 (571h53m42.178564386s) [noise=true , address=145.79.227.34:4910 , piece_id=ZH4DCNXET2Q6VRAUH3NQ3KKSV4OYI575X7YS72XD236EEOMOG3LA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.325775906 (571h53m42.196858941s) [noise=true , address=145.79.227.40:4067 , piece_id=WSYDRX5YJ6OCFW52ZVMM6WZYNRFUXMR4XJ6C5727BOFMR4RRH33Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.325865978 (571h53m42.196949023s) [seg_pos={0 0}, upload=501384]: Segment reached optimal threshold of 14 pieces. Dec 29 11:22:42.326137074 (571h53m42.197220109s) [noise=true , address=64.49.13.169:4708 , piece_id=B2X2RWRELXWJPF6AVJJZCNU46LM5ASGRE4EWGPKL3SCLQ4VKUBDQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection Dec 29 11:22:42.326153074 (571h53m42.197236109s) [noise=true , address=64.49.13.169:4708 , piece_id=B2X2RWRELXWJPF6AVJJZCNU46LM5ASGRE4EWGPKL3SCLQ4VKUBDQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326193401 (571h53m42.197276446s) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12h5esfg6mw2c8y3AVpeFBkL5FfLAwYdjhcnQ9CJ8fjh61WLYjx): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326205705 (571h53m42.19728874s) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326407739 (571h53m42.197490784s) [noise=true , address=77.90.31.10:4524 , piece_id=Q2JGEUV5EDFTVY2CSNPZOVJIIIAFBA3IHA7S5N7YZRH3E7TSTE5A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uN7CznoK4AD8FqFkNUndawdGJ9rmjAe7oNnjJkV844ScY87qe): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326415374 (571h53m42.197498419s) [noise=true , address=77.90.31.10:4524 , piece_id=Q2JGEUV5EDFTVY2CSNPZOVJIIIAFBA3IHA7S5N7YZRH3E7TSTE5A, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326505105 (571h53m42.19758815s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tnTEYvqJ6PB2DsuXA28sEuEiGj11StA4qdhcYH1kuoaMkTVdY): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326512188 (571h53m42.197595223s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326532357 (571h53m42.197615392s) [seg_pos={0 0}, upload=501384]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 11:22:42.326671562 (571h53m42.197754597s) [upload=501384]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 11:22:42.326677843 (571h53m42.197760879s) [upload=501384]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 11:22:42.610391438 (571h53m42.481474493s) [upload=501384]: Done uploading segments... Dec 29 11:22:42.610416826 (571h53m42.481499871s) [upload=501384]: Upload finished. err: <nil> Duration: 35.847756302s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 27 out of 60 is failed (threshold=3) [X..X..XXX.X.......X.X..X.XXX...XX.XXX.X.X....XXXXXX..X.X....] duration: 5.940386604s trace: https://jaeger.dev.storj.io/trace/78a810c1312d39a9 FAIL (2025-12-29 12:01:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: TXY6KE2F3OOGRMOT4ZF3MERPEEXFOQONOJUJ2G5TDQQTT5TFGANA) context deadline exceeded; piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: 2PTAVS33GMBNXYULC2YFT6YTVLDSPBQOIYM2BHCXGYTTWLF2FNDQ) context deadline exceeded ======LOGS====== Dec 29 11:59:08.361853165 (572h30m8.2329362s) [upload=501910]: Starting upload Dec 29 11:59:08.361997270 (572h30m8.233080335s) [upload=501910]: Done starting upload Dec 29 11:59:08.362093513 (572h30m8.233176548s) [upload=501910]: Uploading segments... Dec 29 11:59:08.362292753 (572h30m8.233375798s) [upload=501910]: Got next segment. Inline: false Dec 29 11:59:08.362335234 (572h30m8.233418269s) [upload=501910]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 11:59:08.362340063 (572h30m8.233423098s) [upload=501910]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 11:59:08.611633275 (572h30m8.482716321s) [seg_pos={0 0}, upload=501910]: Begin upload segment... Dec 29 11:59:08.611679954 (572h30m8.482762999s) [seg_pos={0 0}, upload=501910]: Done begin upload segment. Dec 29 11:59:08.611705703 (572h30m8.482788738s) [noise=true , address=145.79.227.30:4761 , piece_id=FWOGG3AETOEAFIFEBVASAVOTUITY7TVF3JVWTOHDCTXDGQMLQGVA, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.611729619 (572h30m8.482812664s) [noise=true , address=45.91.116.14:4192 , piece_id=TXY6KE2F3OOGRMOT4ZF3MERPEEXFOQONOJUJ2G5TDQQTT5TFGANA, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.612625375 (572h30m8.48370841s) [noise=true , address=145.79.227.47:4266 , piece_id=XAZDWB62CWNFH7N5D4TGNXRLRT7YF2ZP4L6MNATUCZDDUXA2QDGQ, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.612648729 (572h30m8.483731774s) [noise=true , address=45.91.116.5:4298 , piece_id=TTGXECJ2OUJFDRSXM53VW6V2KJGT5ZTWKGEN6WOKN7RQZURKK62Q, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.613025066 (572h30m8.484108111s) [noise=true , address=45.91.116.9:4262 , piece_id=KXI3CRGDXPFIQMETUUKKJZS5YX3TY7XZJ6A2HCHO4JW3DNB2I43A, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.613521331 (572h30m8.484604376s) [noise=true , address=145.79.227.17:4970 , piece_id=PZIKWOOFEAAWSI3DDG6MN44EJ6RVL7HYC6HXLP3JGDVDRPPGORFQ, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.613528916 (572h30m8.484611961s) [noise=true , address=145.79.227.2:4115 , piece_id=VVEW5N7PMXNFLYMEIWFB6ZBTJWMBFK2PRWTFZKTUB5AUIPSOMCZA, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.613977430 (572h30m8.485060465s) [noise=true , address=145.79.227.46:4602 , piece_id=UO7XPJAI7VQXCF7SVBV43WTI6JPOK4DFQK3WMJ54EPXVSRWLA6QA, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.614275347 (572h30m8.485358382s) [noise=true , address=145.79.227.23:4545 , piece_id=XI7C54VR2CYZIE6OLAXP6UYWH2BQI5IFZLL6XFGMDPIF5UKKJUKA, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.614415964 (572h30m8.485499009s) [noise=true , address=77.90.31.14:4661 , piece_id=XLSIJ2ZO6ZB7AFRAGBG7B3K5T4Q4EEGXKRUJ3QLQ5CNPRKSQ5N4A, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.614940583 (572h30m8.486023618s) [noise=true , address=145.79.227.5:4222 , piece_id=UBA5YYPYYNUYRYHYURWMRXXB7Q6AMYZ3EZ3HZDLNF77OG4MDKEUQ, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.614969919 (572h30m8.486052964s) [noise=true , address=77.90.31.19:4842 , piece_id=2PTAVS33GMBNXYULC2YFT6YTVLDSPBQOIYM2BHCXGYTTWLF2FNDQ, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.615383867 (572h30m8.486466903s) [noise=true , address=145.79.227.14:4837 , piece_id=TE4VAA7KHS4YYQRGKMOMBM2WLVEMNB5NSML4JC7OY2EKHMDZQ6SA, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.615607825 (572h30m8.48669087s) [noise=true , address=145.79.227.21:4473 , piece_id=IDGJS4QYGBIORV26G45FWDSOA3H63UEEGPPPLLDN5RCFE3W4HAXA, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.615881095 (572h30m8.48696414s) [noise=true , address=145.79.227.26:4654 , piece_id=45IHGFKRVEDQDGQTSFALHRIE6IVQZHY7E3DCIZI6TNHLJFITPGQQ, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.616226894 (572h30m8.487309939s) [noise=true , address=145.79.227.50:4374 , piece_id=L2USZXC5XKVWUDSZ5MDLOZVAFJOMVGYF7SM2E5Q4EAU4EEZEF23Q, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:10.218180372 (572h30m10.089263407s) [upload=501910]: Next returned nil segment Dec 29 11:59:10.218197525 (572h30m10.08928056s) [upload=501910]: Waiting for error group managing segments... Dec 29 11:59:10.596756931 (572h30m10.467839966s) [noise=true , address=145.79.227.17:4970 , piece_id=PZIKWOOFEAAWSI3DDG6MN44EJ6RVL7HYC6HXLP3JGDVDRPPGORFQ, seg_pos={0 0}, upload=501910]: Done uploading piece. err: <nil> Dec 29 11:59:10.597714194 (572h30m10.46879724s) [noise=true , address=145.79.227.47:4266 , piece_id=XAZDWB62CWNFH7N5D4TGNXRLRT7YF2ZP4L6MNATUCZDDUXA2QDGQ, seg_pos={0 0}, upload=501910]: Done uploading piece. err: <nil> Dec 29 11:59:10.600236358 (572h30m10.471319393s) [noise=true , address=145.79.227.14:4837 , piece_id=TE4VAA7KHS4YYQRGKMOMBM2WLVEMNB5NSML4JC7OY2EKHMDZQ6SA, seg_pos={0 0}, upload=501910]: Done uploading piece. err: <nil> Dec 29 11:59:10.602320044 (572h30m10.47340308s) [noise=true , address=145.79.227.5:4222 , piece_id=UBA5YYPYYNUYRYHYURWMRXXB7Q6AMYZ3EZ3HZDLNF77OG4MDKEUQ, seg_pos={0 0}, upload=501910]: Done uploading piece. err: <nil> Dec 29 11:59:10.602912464 (572h30m10.473995499s) [noise=true , address=145.79.227.26:4654 , piece_id=45IHGFKRVEDQDGQTSFALHRIE6IVQZHY7E3DCIZI6TNHLJFITPGQQ, seg_pos={0 0}, upload=501910]: Done uploading piece. err: <nil> Dec 29 11:59:10.611761013 (572h30m10.482844048s) [noise=true , address=145.79.227.23:4545 , piece_id=XI7C54VR2CYZIE6OLAXP6UYWH2BQI5IFZLL6XFGMDPIF5UKKJUKA, seg_pos={0 0}, upload=501910]: Done uploading piece. err: <nil> Dec 29 11:59:10.623036624 (572h30m10.494119669s) [noise=true , address=145.79.227.46:4602 , piece_id=UO7XPJAI7VQXCF7SVBV43WTI6JPOK4DFQK3WMJ54EPXVSRWLA6QA, seg_pos={0 0}, upload=501910]: Done uploading piece. err: <nil> Dec 29 11:59:10.626994862 (572h30m10.498077907s) [noise=true , address=145.79.227.50:4374 , piece_id=L2USZXC5XKVWUDSZ5MDLOZVAFJOMVGYF7SM2E5Q4EAU4EEZEF23Q, seg_pos={0 0}, upload=501910]: Done uploading piece. err: <nil> Dec 29 11:59:10.627968646 (572h30m10.499051691s) [noise=true , address=145.79.227.21:4473 , piece_id=IDGJS4QYGBIORV26G45FWDSOA3H63UEEGPPPLLDN5RCFE3W4HAXA, seg_pos={0 0}, upload=501910]: Done uploading piece. err: <nil> Dec 29 11:59:10.629745359 (572h30m10.500828404s) [noise=true , address=145.79.227.2:4115 , piece_id=VVEW5N7PMXNFLYMEIWFB6ZBTJWMBFK2PRWTFZKTUB5AUIPSOMCZA, seg_pos={0 0}, upload=501910]: Done uploading piece. err: <nil> Dec 29 11:59:10.847903124 (572h30m10.718986159s) [noise=true , address=45.91.116.14:4192 , piece_id=TXY6KE2F3OOGRMOT4ZF3MERPEEXFOQONOJUJ2G5TDQQTT5TFGANA, seg_pos={0 0}, upload=501910]: Done uploading piece. err: <nil> Dec 29 11:59:10.901405164 (572h30m10.772488199s) [noise=true , address=77.90.31.19:4842 , piece_id=2PTAVS33GMBNXYULC2YFT6YTVLDSPBQOIYM2BHCXGYTTWLF2FNDQ, seg_pos={0 0}, upload=501910]: Done uploading piece. err: <nil> Dec 29 11:59:11.144788475 (572h30m11.01587151s) [noise=true , address=45.91.116.9:4262 , piece_id=KXI3CRGDXPFIQMETUUKKJZS5YX3TY7XZJ6A2HCHO4JW3DNB2I43A, seg_pos={0 0}, upload=501910]: Done uploading piece. err: <nil> Dec 29 11:59:11.144812380 (572h30m11.015895415s) [seg_pos={0 0}, upload=501910]: Segment reached optimal threshold of 13 pieces. Dec 29 11:59:11.145162818 (572h30m11.016245863s) [noise=true , address=45.91.116.5:4298 , piece_id=TTGXECJ2OUJFDRSXM53VW6V2KJGT5ZTWKGEN6WOKN7RQZURKK62Q, seg_pos={0 0}, upload=501910]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 29 11:59:11.145173508 (572h30m11.016256543s) [noise=true , address=45.91.116.5:4298 , piece_id=TTGXECJ2OUJFDRSXM53VW6V2KJGT5ZTWKGEN6WOKN7RQZURKK62Q, seg_pos={0 0}, upload=501910]: Long tail cancellation due to optimal threshold Dec 29 11:59:11.145253922 (572h30m11.016336967s) [noise=true , address=145.79.227.30:4761 , piece_id=FWOGG3AETOEAFIFEBVASAVOTUITY7TVF3JVWTOHDCTXDGQMLQGVA, seg_pos={0 0}, upload=501910]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 29 11:59:11.145265453 (572h30m11.016348498s) [noise=true , address=145.79.227.30:4761 , piece_id=FWOGG3AETOEAFIFEBVASAVOTUITY7TVF3JVWTOHDCTXDGQMLQGVA, seg_pos={0 0}, upload=501910]: Long tail cancellation due to optimal threshold Dec 29 11:59:11.145333914 (572h30m11.016416949s) [noise=true , address=77.90.31.14:4661 , piece_id=XLSIJ2ZO6ZB7AFRAGBG7B3K5T4Q4EEGXKRUJ3QLQ5CNPRKSQ5N4A, seg_pos={0 0}, upload=501910]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Dec 29 11:59:11.145342329 (572h30m11.016425374s) [noise=true , address=77.90.31.14:4661 , piece_id=XLSIJ2ZO6ZB7AFRAGBG7B3K5T4Q4EEGXKRUJ3QLQ5CNPRKSQ5N4A, seg_pos={0 0}, upload=501910]: Long tail cancellation due to optimal threshold Dec 29 11:59:11.145359503 (572h30m11.016442538s) [seg_pos={0 0}, upload=501910]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 11:59:11.145451607 (572h30m11.016534652s) [upload=501910]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 11:59:11.145457439 (572h30m11.016540484s) [upload=501910]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 11:59:11.445524688 (572h30m11.316607733s) [upload=501910]: Done uploading segments... Dec 29 11:59:11.445548383 (572h30m11.316631429s) [upload=501910]: Upload finished. err: <nil> Duration: 5.940415738s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [XX..X..XXX.X.......X.X..X.XXX...XX.XXX.X.X....XXXXXX..X.X...] duration: 5.651005452s trace: https://jaeger.dev.storj.io/trace/78a810c1312d39a9 FAIL (2025-12-29 12:01:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: TXY6KE2F3OOGRMOT4ZF3MERPEEXFOQONOJUJ2G5TDQQTT5TFGANA) context deadline exceeded; piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: 2PTAVS33GMBNXYULC2YFT6YTVLDSPBQOIYM2BHCXGYTTWLF2FNDQ) context deadline exceeded ======LOGS====== Dec 29 11:59:08.361853165 (572h30m8.2329362s) [upload=501910]: Starting upload Dec 29 11:59:08.361997270 (572h30m8.233080335s) [upload=501910]: Done starting upload Dec 29 11:59:08.362093513 (572h30m8.233176548s) [upload=501910]: Uploading segments... Dec 29 11:59:08.362292753 (572h30m8.233375798s) [upload=501910]: Got next segment. Inline: false Dec 29 11:59:08.362335234 (572h30m8.233418269s) [upload=501910]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 11:59:08.362340063 (572h30m8.233423098s) [upload=501910]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 11:59:08.611633275 (572h30m8.482716321s) [seg_pos={0 0}, upload=501910]: Begin upload segment... Dec 29 11:59:08.611679954 (572h30m8.482762999s) [seg_pos={0 0}, upload=501910]: Done begin upload segment. Dec 29 11:59:08.611705703 (572h30m8.482788738s) [noise=true , address=145.79.227.30:4761 , piece_id=FWOGG3AETOEAFIFEBVASAVOTUITY7TVF3JVWTOHDCTXDGQMLQGVA, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.611729619 (572h30m8.482812664s) [noise=true , address=45.91.116.14:4192 , piece_id=TXY6KE2F3OOGRMOT4ZF3MERPEEXFOQONOJUJ2G5TDQQTT5TFGANA, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.612625375 (572h30m8.48370841s) [noise=true , address=145.79.227.47:4266 , piece_id=XAZDWB62CWNFH7N5D4TGNXRLRT7YF2ZP4L6MNATUCZDDUXA2QDGQ, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.612648729 (572h30m8.483731774s) [noise=true , address=45.91.116.5:4298 , piece_id=TTGXECJ2OUJFDRSXM53VW6V2KJGT5ZTWKGEN6WOKN7RQZURKK62Q, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.613025066 (572h30m8.484108111s) [noise=true , address=45.91.116.9:4262 , piece_id=KXI3CRGDXPFIQMETUUKKJZS5YX3TY7XZJ6A2HCHO4JW3DNB2I43A, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.613521331 (572h30m8.484604376s) [noise=true , address=145.79.227.17:4970 , piece_id=PZIKWOOFEAAWSI3DDG6MN44EJ6RVL7HYC6HXLP3JGDVDRPPGORFQ, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.613528916 (572h30m8.484611961s) [noise=true , address=145.79.227.2:4115 , piece_id=VVEW5N7PMXNFLYMEIWFB6ZBTJWMBFK2PRWTFZKTUB5AUIPSOMCZA, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.613977430 (572h30m8.485060465s) [noise=true , address=145.79.227.46:4602 , piece_id=UO7XPJAI7VQXCF7SVBV43WTI6JPOK4DFQK3WMJ54EPXVSRWLA6QA, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.614275347 (572h30m8.485358382s) [noise=true , address=145.79.227.23:4545 , piece_id=XI7C54VR2CYZIE6OLAXP6UYWH2BQI5IFZLL6XFGMDPIF5UKKJUKA, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.614415964 (572h30m8.485499009s) [noise=true , address=77.90.31.14:4661 , piece_id=XLSIJ2ZO6ZB7AFRAGBG7B3K5T4Q4EEGXKRUJ3QLQ5CNPRKSQ5N4A, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.614940583 (572h30m8.486023618s) [noise=true , address=145.79.227.5:4222 , piece_id=UBA5YYPYYNUYRYHYURWMRXXB7Q6AMYZ3EZ3HZDLNF77OG4MDKEUQ, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.614969919 (572h30m8.486052964s) [noise=true , address=77.90.31.19:4842 , piece_id=2PTAVS33GMBNXYULC2YFT6YTVLDSPBQOIYM2BHCXGYTTWLF2FNDQ, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.615383867 (572h30m8.486466903s) [noise=true , address=145.79.227.14:4837 , piece_id=TE4VAA7KHS4YYQRGKMOMBM2WLVEMNB5NSML4JC7OY2EKHMDZQ6SA, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.615607825 (572h30m8.48669087s) [noise=true , address=145.79.227.21:4473 , piece_id=IDGJS4QYGBIORV26G45FWDSOA3H63UEEGPPPLLDN5RCFE3W4HAXA, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.615881095 (572h30m8.48696414s) [noise=true , address=145.79.227.26:4654 , piece_id=45IHGFKRVEDQDGQTSFALHRIE6IVQZHY7E3DCIZI6TNHLJFITPGQQ, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.616226894 (572h30m8.487309939s) [noise=true , address=145.79.227.50:4374 , piece_id=L2USZXC5XKVWUDSZ5MDLOZVAFJOMVGYF7SM2E5Q4EAU4EEZEF23Q, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:10.218180372 (572h30m10.089263407s) [upload=501910]: Next returned nil segment Dec 29 11:59:10.218197525 (572h30m10.08928056s) [upload=501910]: Waiting for error group managing segments... Dec 29 11:59:10.596756931 (572h30m10.467839966s) [noise=true , address=145.79.227.17:4970 , piece_id=PZIKWOOFEAAWSI3DDG6MN44EJ6RVL7HYC6HXLP3JGDVDRPPGORFQ, seg_pos={0 0}, upload=501910]: Done uploading piece. err: <nil> Dec 29 11:59:10.597714194 (572h30m10.46879724s) [noise=true , address=145.79.227.47:4266 , piece_id=XAZDWB62CWNFH7N5D4TGNXRLRT7YF2ZP4L6MNATUCZDDUXA2QDGQ, seg_pos={0 0}, upload=501910]: Done uploading piece. err: <nil> Dec 29 11:59:10.600236358 (572h30m10.471319393s) [noise=true , address=145.79.227.14:4837 , piece_id=TE4VAA7KHS4YYQRGKMOMBM2WLVEMNB5NSML4JC7OY2EKHMDZQ6SA, seg_pos={0 0}, upload=501910]: Done uploading piece. err: <nil> Dec 29 11:59:10.602320044 (572h30m10.47340308s) [noise=true , address=145.79.227.5:4222 , piece_id=UBA5YYPYYNUYRYHYURWMRXXB7Q6AMYZ3EZ3HZDLNF77OG4MDKEUQ, seg_pos={0 0}, upload=501910]: Done uploading piece. err: <nil> Dec 29 11:59:10.602912464 (572h30m10.473995499s) [noise=true , address=145.79.227.26:4654 , piece_id=45IHGFKRVEDQDGQTSFALHRIE6IVQZHY7E3DCIZI6TNHLJFITPGQQ, seg_pos={0 0}, upload=501910]: Done uploading piece. err: <nil> Dec 29 11:59:10.611761013 (572h30m10.482844048s) [noise=true , address=145.79.227.23:4545 , piece_id=XI7C54VR2CYZIE6OLAXP6UYWH2BQI5IFZLL6XFGMDPIF5UKKJUKA, seg_pos={0 0}, upload=501910]: Done uploading piece. err: <nil> Dec 29 11:59:10.623036624 (572h30m10.494119669s) [noise=true , address=145.79.227.46:4602 , piece_id=UO7XPJAI7VQXCF7SVBV43WTI6JPOK4DFQK3WMJ54EPXVSRWLA6QA, seg_pos={0 0}, upload=501910]: Done uploading piece. err: <nil> Dec 29 11:59:10.626994862 (572h30m10.498077907s) [noise=true , address=145.79.227.50:4374 , piece_id=L2USZXC5XKVWUDSZ5MDLOZVAFJOMVGYF7SM2E5Q4EAU4EEZEF23Q, seg_pos={0 0}, upload=501910]: Done uploading piece. err: <nil> Dec 29 11:59:10.627968646 (572h30m10.499051691s) [noise=true , address=145.79.227.21:4473 , piece_id=IDGJS4QYGBIORV26G45FWDSOA3H63UEEGPPPLLDN5RCFE3W4HAXA, seg_pos={0 0}, upload=501910]: Done uploading piece. err: <nil> Dec 29 11:59:10.629745359 (572h30m10.500828404s) [noise=true , address=145.79.227.2:4115 , piece_id=VVEW5N7PMXNFLYMEIWFB6ZBTJWMBFK2PRWTFZKTUB5AUIPSOMCZA, seg_pos={0 0}, upload=501910]: Done uploading piece. err: <nil> Dec 29 11:59:10.847903124 (572h30m10.718986159s) [noise=true , address=45.91.116.14:4192 , piece_id=TXY6KE2F3OOGRMOT4ZF3MERPEEXFOQONOJUJ2G5TDQQTT5TFGANA, seg_pos={0 0}, upload=501910]: Done uploading piece. err: <nil> Dec 29 11:59:10.901405164 (572h30m10.772488199s) [noise=true , address=77.90.31.19:4842 , piece_id=2PTAVS33GMBNXYULC2YFT6YTVLDSPBQOIYM2BHCXGYTTWLF2FNDQ, seg_pos={0 0}, upload=501910]: Done uploading piece. err: <nil> Dec 29 11:59:11.144788475 (572h30m11.01587151s) [noise=true , address=45.91.116.9:4262 , piece_id=KXI3CRGDXPFIQMETUUKKJZS5YX3TY7XZJ6A2HCHO4JW3DNB2I43A, seg_pos={0 0}, upload=501910]: Done uploading piece. err: <nil> Dec 29 11:59:11.144812380 (572h30m11.015895415s) [seg_pos={0 0}, upload=501910]: Segment reached optimal threshold of 13 pieces. Dec 29 11:59:11.145162818 (572h30m11.016245863s) [noise=true , address=45.91.116.5:4298 , piece_id=TTGXECJ2OUJFDRSXM53VW6V2KJGT5ZTWKGEN6WOKN7RQZURKK62Q, seg_pos={0 0}, upload=501910]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 29 11:59:11.145173508 (572h30m11.016256543s) [noise=true , address=45.91.116.5:4298 , piece_id=TTGXECJ2OUJFDRSXM53VW6V2KJGT5ZTWKGEN6WOKN7RQZURKK62Q, seg_pos={0 0}, upload=501910]: Long tail cancellation due to optimal threshold Dec 29 11:59:11.145253922 (572h30m11.016336967s) [noise=true , address=145.79.227.30:4761 , piece_id=FWOGG3AETOEAFIFEBVASAVOTUITY7TVF3JVWTOHDCTXDGQMLQGVA, seg_pos={0 0}, upload=501910]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 29 11:59:11.145265453 (572h30m11.016348498s) [noise=true , address=145.79.227.30:4761 , piece_id=FWOGG3AETOEAFIFEBVASAVOTUITY7TVF3JVWTOHDCTXDGQMLQGVA, seg_pos={0 0}, upload=501910]: Long tail cancellation due to optimal threshold Dec 29 11:59:11.145333914 (572h30m11.016416949s) [noise=true , address=77.90.31.14:4661 , piece_id=XLSIJ2ZO6ZB7AFRAGBG7B3K5T4Q4EEGXKRUJ3QLQ5CNPRKSQ5N4A, seg_pos={0 0}, upload=501910]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Dec 29 11:59:11.145342329 (572h30m11.016425374s) [noise=true , address=77.90.31.14:4661 , piece_id=XLSIJ2ZO6ZB7AFRAGBG7B3K5T4Q4EEGXKRUJ3QLQ5CNPRKSQ5N4A, seg_pos={0 0}, upload=501910]: Long tail cancellation due to optimal threshold Dec 29 11:59:11.145359503 (572h30m11.016442538s) [seg_pos={0 0}, upload=501910]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 11:59:11.145451607 (572h30m11.016534652s) [upload=501910]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 11:59:11.145457439 (572h30m11.016540484s) [upload=501910]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 11:59:11.445524688 (572h30m11.316607733s) [upload=501910]: Done uploading segments... Dec 29 11:59:11.445548383 (572h30m11.316631429s) [upload=501910]: Upload finished. err: <nil> Duration: 5.651026603s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 30 out of 60 is failed (threshold=6) [..XX......X.X.XX.X.XX..X..XX.XXXX.XXXX...XXX...XXX..XX.X..X.] duration: 3.418139147s trace: https://jaeger.dev.storj.io/trace/36a8212af98f1204 FAIL (2025-12-29 12:01:11 UTC) orbiter warning: context deadline exceeded Duration: 3.418173382s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [............X..X............................................] duration: 45.537981273s trace: https://jaeger.dev.storj.io/trace/7e7354d8023608a8 FAIL (2025-12-29 12:01:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== ) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480097317 (571h53m39.351180352s) [noise=true , address=77.90.31.21:4889 , piece_id=BR6Z55P3A7YJ5FJRDSZFUIF66FGIJQY7WCEQ7PL5DOYWL73OMKTQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480405053 (571h53m39.351488098s) [noise=true , address=77.90.31.16:4750 , piece_id=MVVZPJEGDCBSNVOB6CD2WUZGIJTL3VBHX2G4L3TVZ6JF5ZEJ5U6Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480712969 (571h53m39.351796014s) [noise=true , address=77.90.31.18:4834 , piece_id=BTWNGNQVEUEVV6HLBDM63SGNTKV4RDU4J4XFLG52WHKP53H3L5KA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481019453 (571h53m39.352102488s) [noise=true , address=145.79.227.30:4777 , piece_id=VSCXLXGIBSFE6E4OHLWNZIZ75PO7WQO2VFHUYHW7JUNH62KGGU7Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481029532 (571h53m39.352112567s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481559832 (571h53m39.352642877s) [noise=true , address=145.79.227.35:4515 , piece_id=L7LFZKZEZERUNX4HUNDNQ2PWHXNJRJ7OBFRNMFIICPZEUIBHGP7A, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481731919 (571h53m39.352814954s) [noise=true , address=64.49.13.26:4953 , piece_id=4YSW5Y3A6HYG3XJ2XAB7RGVHAOHMP7G3QIZYMYTSXVXYJ3ANLNXA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482078329 (571h53m39.353161364s) [noise=true , address=145.79.227.44:4205 , piece_id=UE4QSHDQBZ3XBWQ6I6MCRVTLMU7U6W6OZDA2ZGQCH6ZYRAGPD7WA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482099830 (571h53m39.353182865s) [noise=true , address=145.79.227.40:4067 , piece_id=WSYDRX5YJ6OCFW52ZVMM6WZYNRFUXMR4XJ6C5727BOFMR4RRH33Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482434487 (571h53m39.353517532s) [noise=true , address=145.79.227.3:4164 , piece_id=U6F7VDPOH7K6R5EH5UUXHFP55GXOEXO5BLBJHZRTITPDQI4LP7KQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482480845 (571h53m39.35356388s) [noise=true , address=64.49.13.140:4516 , piece_id=4GUYOYOAEASUK2W25S3NILWIIIHGOZX76L4EGFLUUUEDWRALPP2A, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482841442 (571h53m39.353924477s) [noise=true , address=145.79.227.34:4910 , piece_id=ZH4DCNXET2Q6VRAUH3NQ3KKSV4OYI575X7YS72XD236EEOMOG3LA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483163645 (571h53m39.35424668s) [noise=true , address=145.79.227.29:4974 , piece_id=KEKREOD2B35XQNI45JBHE2SZZC3MQQBQUR5I6WFDAQYJQGJBXSMA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483532478 (571h53m39.354615513s) [noise=true , address=77.90.31.17:4769 , piece_id=3MKUMOQIUVERKQLLIFX52XQPTQ5LSFMHIPD5YW2YTO5YEFR36IYA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483914596 (571h53m39.354997631s) [noise=true , address=145.79.227.13:4807 , piece_id=VKXB5MWAG2YYH2AWWJEGTLBZ2QUPLI25VTHI6AMMALSM23Z3AWZQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:40.761721106 (571h53m40.632804141s) [upload=501384]: Next returned nil segment Dec 29 11:22:40.761733349 (571h53m40.632816384s) [upload=501384]: Waiting for error group managing segments... Dec 29 11:22:41.121222862 (571h53m40.992305897s) [noise=true , address=64.49.13.140:4516 , piece_id=4GUYOYOAEASUK2W25S3NILWIIIHGOZX76L4EGFLUUUEDWRALPP2A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.197311646 (571h53m41.068394691s) [noise=true , address=64.49.13.26:4953 , piece_id=4YSW5Y3A6HYG3XJ2XAB7RGVHAOHMP7G3QIZYMYTSXVXYJ3ANLNXA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.347247568 (571h53m41.218330613s) [noise=true , address=145.79.227.13:4807 , piece_id=VKXB5MWAG2YYH2AWWJEGTLBZ2QUPLI25VTHI6AMMALSM23Z3AWZQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.347917313 (571h53m41.219000348s) [noise=true , address=145.79.227.3:4164 , piece_id=U6F7VDPOH7K6R5EH5UUXHFP55GXOEXO5BLBJHZRTITPDQI4LP7KQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.348510613 (571h53m41.219593648s) [noise=true , address=145.79.227.29:4974 , piece_id=KEKREOD2B35XQNI45JBHE2SZZC3MQQBQUR5I6WFDAQYJQGJBXSMA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.503950985 (571h53m41.37503402s) [noise=true , address=77.90.31.17:4769 , piece_id=3MKUMOQIUVERKQLLIFX52XQPTQ5LSFMHIPD5YW2YTO5YEFR36IYA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.561352025 (571h53m41.43243506s) [noise=true , address=77.90.31.18:4834 , piece_id=BTWNGNQVEUEVV6HLBDM63SGNTKV4RDU4J4XFLG52WHKP53H3L5KA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.622447380 (571h53m41.493530425s) [noise=true , address=77.90.31.16:4750 , piece_id=MVVZPJEGDCBSNVOB6CD2WUZGIJTL3VBHX2G4L3TVZ6JF5ZEJ5U6Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.645494685 (571h53m41.51657773s) [noise=true , address=77.90.31.21:4889 , piece_id=BR6Z55P3A7YJ5FJRDSZFUIF66FGIJQY7WCEQ7PL5DOYWL73OMKTQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.247930045 (571h53m42.11901308s) [noise=true , address=145.79.227.30:4777 , piece_id=VSCXLXGIBSFE6E4OHLWNZIZ75PO7WQO2VFHUYHW7JUNH62KGGU7Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.262221890 (571h53m42.133304925s) [noise=true , address=145.79.227.44:4205 , piece_id=UE4QSHDQBZ3XBWQ6I6MCRVTLMU7U6W6OZDA2ZGQCH6ZYRAGPD7WA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.295598123 (571h53m42.166681158s) [noise=true , address=145.79.227.35:4515 , piece_id=L7LFZKZEZERUNX4HUNDNQ2PWHXNJRJ7OBFRNMFIICPZEUIBHGP7A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.307481351 (571h53m42.178564386s) [noise=true , address=145.79.227.34:4910 , piece_id=ZH4DCNXET2Q6VRAUH3NQ3KKSV4OYI575X7YS72XD236EEOMOG3LA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.325775906 (571h53m42.196858941s) [noise=true , address=145.79.227.40:4067 , piece_id=WSYDRX5YJ6OCFW52ZVMM6WZYNRFUXMR4XJ6C5727BOFMR4RRH33Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.325865978 (571h53m42.196949023s) [seg_pos={0 0}, upload=501384]: Segment reached optimal threshold of 14 pieces. Dec 29 11:22:42.326137074 (571h53m42.197220109s) [noise=true , address=64.49.13.169:4708 , piece_id=B2X2RWRELXWJPF6AVJJZCNU46LM5ASGRE4EWGPKL3SCLQ4VKUBDQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection Dec 29 11:22:42.326153074 (571h53m42.197236109s) [noise=true , address=64.49.13.169:4708 , piece_id=B2X2RWRELXWJPF6AVJJZCNU46LM5ASGRE4EWGPKL3SCLQ4VKUBDQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326193401 (571h53m42.197276446s) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12h5esfg6mw2c8y3AVpeFBkL5FfLAwYdjhcnQ9CJ8fjh61WLYjx): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326205705 (571h53m42.19728874s) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326407739 (571h53m42.197490784s) [noise=true , address=77.90.31.10:4524 , piece_id=Q2JGEUV5EDFTVY2CSNPZOVJIIIAFBA3IHA7S5N7YZRH3E7TSTE5A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uN7CznoK4AD8FqFkNUndawdGJ9rmjAe7oNnjJkV844ScY87qe): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326415374 (571h53m42.197498419s) [noise=true , address=77.90.31.10:4524 , piece_id=Q2JGEUV5EDFTVY2CSNPZOVJIIIAFBA3IHA7S5N7YZRH3E7TSTE5A, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326505105 (571h53m42.19758815s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tnTEYvqJ6PB2DsuXA28sEuEiGj11StA4qdhcYH1kuoaMkTVdY): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326512188 (571h53m42.197595223s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326532357 (571h53m42.197615392s) [seg_pos={0 0}, upload=501384]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 11:22:42.326671562 (571h53m42.197754597s) [upload=501384]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 11:22:42.326677843 (571h53m42.197760879s) [upload=501384]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 11:22:42.610391438 (571h53m42.481474493s) [upload=501384]: Done uploading segments... Dec 29 11:22:42.610416826 (571h53m42.481499871s) [upload=501384]: Upload finished. err: <nil> Duration: 45.538022893s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 31 out of 60 is failed (threshold=6) [X..XX......X.X.XX.X.XX..X..XX.XXXX.XXXX...XXX...XXX..XX.X..X] duration: 5.001759919s trace: https://jaeger.dev.storj.io/trace/36a8212af98f1204 FAIL (2025-12-29 12:00:52 UTC) orbiter warning: context deadline exceeded Duration: 5.001784846s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [.XX..X..XXX.X.......X.X..X.XXX...XX.XXX.X.X....XXXXXX..X.X..] duration: 5.756147511s trace: https://jaeger.dev.storj.io/trace/78a810c1312d39a9 FAIL (2025-12-29 12:00:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: TXY6KE2F3OOGRMOT4ZF3MERPEEXFOQONOJUJ2G5TDQQTT5TFGANA) context deadline exceeded; piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: 2PTAVS33GMBNXYULC2YFT6YTVLDSPBQOIYM2BHCXGYTTWLF2FNDQ) context deadline exceeded ======LOGS====== Dec 29 11:59:08.361853165 (572h30m8.2329362s) [upload=501910]: Starting upload Dec 29 11:59:08.361997270 (572h30m8.233080335s) [upload=501910]: Done starting upload Dec 29 11:59:08.362093513 (572h30m8.233176548s) [upload=501910]: Uploading segments... Dec 29 11:59:08.362292753 (572h30m8.233375798s) [upload=501910]: Got next segment. Inline: false Dec 29 11:59:08.362335234 (572h30m8.233418269s) [upload=501910]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 11:59:08.362340063 (572h30m8.233423098s) [upload=501910]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 11:59:08.611633275 (572h30m8.482716321s) [seg_pos={0 0}, upload=501910]: Begin upload segment... Dec 29 11:59:08.611679954 (572h30m8.482762999s) [seg_pos={0 0}, upload=501910]: Done begin upload segment. Dec 29 11:59:08.611705703 (572h30m8.482788738s) [noise=true , address=145.79.227.30:4761 , piece_id=FWOGG3AETOEAFIFEBVASAVOTUITY7TVF3JVWTOHDCTXDGQMLQGVA, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.611729619 (572h30m8.482812664s) [noise=true , address=45.91.116.14:4192 , piece_id=TXY6KE2F3OOGRMOT4ZF3MERPEEXFOQONOJUJ2G5TDQQTT5TFGANA, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.612625375 (572h30m8.48370841s) [noise=true , address=145.79.227.47:4266 , piece_id=XAZDWB62CWNFH7N5D4TGNXRLRT7YF2ZP4L6MNATUCZDDUXA2QDGQ, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.612648729 (572h30m8.483731774s) [noise=true , address=45.91.116.5:4298 , piece_id=TTGXECJ2OUJFDRSXM53VW6V2KJGT5ZTWKGEN6WOKN7RQZURKK62Q, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.613025066 (572h30m8.484108111s) [noise=true , address=45.91.116.9:4262 , piece_id=KXI3CRGDXPFIQMETUUKKJZS5YX3TY7XZJ6A2HCHO4JW3DNB2I43A, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.613521331 (572h30m8.484604376s) [noise=true , address=145.79.227.17:4970 , piece_id=PZIKWOOFEAAWSI3DDG6MN44EJ6RVL7HYC6HXLP3JGDVDRPPGORFQ, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.613528916 (572h30m8.484611961s) [noise=true , address=145.79.227.2:4115 , piece_id=VVEW5N7PMXNFLYMEIWFB6ZBTJWMBFK2PRWTFZKTUB5AUIPSOMCZA, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.613977430 (572h30m8.485060465s) [noise=true , address=145.79.227.46:4602 , piece_id=UO7XPJAI7VQXCF7SVBV43WTI6JPOK4DFQK3WMJ54EPXVSRWLA6QA, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.614275347 (572h30m8.485358382s) [noise=true , address=145.79.227.23:4545 , piece_id=XI7C54VR2CYZIE6OLAXP6UYWH2BQI5IFZLL6XFGMDPIF5UKKJUKA, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.614415964 (572h30m8.485499009s) [noise=true , address=77.90.31.14:4661 , piece_id=XLSIJ2ZO6ZB7AFRAGBG7B3K5T4Q4EEGXKRUJ3QLQ5CNPRKSQ5N4A, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.614940583 (572h30m8.486023618s) [noise=true , address=145.79.227.5:4222 , piece_id=UBA5YYPYYNUYRYHYURWMRXXB7Q6AMYZ3EZ3HZDLNF77OG4MDKEUQ, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.614969919 (572h30m8.486052964s) [noise=true , address=77.90.31.19:4842 , piece_id=2PTAVS33GMBNXYULC2YFT6YTVLDSPBQOIYM2BHCXGYTTWLF2FNDQ, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.615383867 (572h30m8.486466903s) [noise=true , address=145.79.227.14:4837 , piece_id=TE4VAA7KHS4YYQRGKMOMBM2WLVEMNB5NSML4JC7OY2EKHMDZQ6SA, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.615607825 (572h30m8.48669087s) [noise=true , address=145.79.227.21:4473 , piece_id=IDGJS4QYGBIORV26G45FWDSOA3H63UEEGPPPLLDN5RCFE3W4HAXA, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.615881095 (572h30m8.48696414s) [noise=true , address=145.79.227.26:4654 , piece_id=45IHGFKRVEDQDGQTSFALHRIE6IVQZHY7E3DCIZI6TNHLJFITPGQQ, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.616226894 (572h30m8.487309939s) [noise=true , address=145.79.227.50:4374 , piece_id=L2USZXC5XKVWUDSZ5MDLOZVAFJOMVGYF7SM2E5Q4EAU4EEZEF23Q, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:10.218180372 (572h30m10.089263407s) [upload=501910]: Next returned nil segment Dec 29 11:59:10.218197525 (572h30m10.08928056s) [upload=501910]: Waiting for error group managing segments... Dec 29 11:59:10.596756931 (572h30m10.467839966s) [noise=true , address=145.79.227.17:4970 , piece_id=PZIKWOOFEAAWSI3DDG6MN44EJ6RVL7HYC6HXLP3JGDVDRPPGORFQ, seg_pos={0 0}, upload=501910]: Done uploading piece. err: <nil> Dec 29 11:59:10.597714194 (572h30m10.46879724s) [noise=true , address=145.79.227.47:4266 , piece_id=XAZDWB62CWNFH7N5D4TGNXRLRT7YF2ZP4L6MNATUCZDDUXA2QDGQ, seg_pos={0 0}, upload=501910]: Done uploading piece. err: <nil> Dec 29 11:59:10.600236358 (572h30m10.471319393s) [noise=true , address=145.79.227.14:4837 , piece_id=TE4VAA7KHS4YYQRGKMOMBM2WLVEMNB5NSML4JC7OY2EKHMDZQ6SA, seg_pos={0 0}, upload=501910]: Done uploading piece. err: <nil> Dec 29 11:59:10.602320044 (572h30m10.47340308s) [noise=true , address=145.79.227.5:4222 , piece_id=UBA5YYPYYNUYRYHYURWMRXXB7Q6AMYZ3EZ3HZDLNF77OG4MDKEUQ, seg_pos={0 0}, upload=501910]: Done uploading piece. err: <nil> Dec 29 11:59:10.602912464 (572h30m10.473995499s) [noise=true , address=145.79.227.26:4654 , piece_id=45IHGFKRVEDQDGQTSFALHRIE6IVQZHY7E3DCIZI6TNHLJFITPGQQ, seg_pos={0 0}, upload=501910]: Done uploading piece. err: <nil> Dec 29 11:59:10.611761013 (572h30m10.482844048s) [noise=true , address=145.79.227.23:4545 , piece_id=XI7C54VR2CYZIE6OLAXP6UYWH2BQI5IFZLL6XFGMDPIF5UKKJUKA, seg_pos={0 0}, upload=501910]: Done uploading piece. err: <nil> Dec 29 11:59:10.623036624 (572h30m10.494119669s) [noise=true , address=145.79.227.46:4602 , piece_id=UO7XPJAI7VQXCF7SVBV43WTI6JPOK4DFQK3WMJ54EPXVSRWLA6QA, seg_pos={0 0}, upload=501910]: Done uploading piece. err: <nil> Dec 29 11:59:10.626994862 (572h30m10.498077907s) [noise=true , address=145.79.227.50:4374 , piece_id=L2USZXC5XKVWUDSZ5MDLOZVAFJOMVGYF7SM2E5Q4EAU4EEZEF23Q, seg_pos={0 0}, upload=501910]: Done uploading piece. err: <nil> Dec 29 11:59:10.627968646 (572h30m10.499051691s) [noise=true , address=145.79.227.21:4473 , piece_id=IDGJS4QYGBIORV26G45FWDSOA3H63UEEGPPPLLDN5RCFE3W4HAXA, seg_pos={0 0}, upload=501910]: Done uploading piece. err: <nil> Dec 29 11:59:10.629745359 (572h30m10.500828404s) [noise=true , address=145.79.227.2:4115 , piece_id=VVEW5N7PMXNFLYMEIWFB6ZBTJWMBFK2PRWTFZKTUB5AUIPSOMCZA, seg_pos={0 0}, upload=501910]: Done uploading piece. err: <nil> Dec 29 11:59:10.847903124 (572h30m10.718986159s) [noise=true , address=45.91.116.14:4192 , piece_id=TXY6KE2F3OOGRMOT4ZF3MERPEEXFOQONOJUJ2G5TDQQTT5TFGANA, seg_pos={0 0}, upload=501910]: Done uploading piece. err: <nil> Dec 29 11:59:10.901405164 (572h30m10.772488199s) [noise=true , address=77.90.31.19:4842 , piece_id=2PTAVS33GMBNXYULC2YFT6YTVLDSPBQOIYM2BHCXGYTTWLF2FNDQ, seg_pos={0 0}, upload=501910]: Done uploading piece. err: <nil> Dec 29 11:59:11.144788475 (572h30m11.01587151s) [noise=true , address=45.91.116.9:4262 , piece_id=KXI3CRGDXPFIQMETUUKKJZS5YX3TY7XZJ6A2HCHO4JW3DNB2I43A, seg_pos={0 0}, upload=501910]: Done uploading piece. err: <nil> Dec 29 11:59:11.144812380 (572h30m11.015895415s) [seg_pos={0 0}, upload=501910]: Segment reached optimal threshold of 13 pieces. Dec 29 11:59:11.145162818 (572h30m11.016245863s) [noise=true , address=45.91.116.5:4298 , piece_id=TTGXECJ2OUJFDRSXM53VW6V2KJGT5ZTWKGEN6WOKN7RQZURKK62Q, seg_pos={0 0}, upload=501910]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 29 11:59:11.145173508 (572h30m11.016256543s) [noise=true , address=45.91.116.5:4298 , piece_id=TTGXECJ2OUJFDRSXM53VW6V2KJGT5ZTWKGEN6WOKN7RQZURKK62Q, seg_pos={0 0}, upload=501910]: Long tail cancellation due to optimal threshold Dec 29 11:59:11.145253922 (572h30m11.016336967s) [noise=true , address=145.79.227.30:4761 , piece_id=FWOGG3AETOEAFIFEBVASAVOTUITY7TVF3JVWTOHDCTXDGQMLQGVA, seg_pos={0 0}, upload=501910]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 29 11:59:11.145265453 (572h30m11.016348498s) [noise=true , address=145.79.227.30:4761 , piece_id=FWOGG3AETOEAFIFEBVASAVOTUITY7TVF3JVWTOHDCTXDGQMLQGVA, seg_pos={0 0}, upload=501910]: Long tail cancellation due to optimal threshold Dec 29 11:59:11.145333914 (572h30m11.016416949s) [noise=true , address=77.90.31.14:4661 , piece_id=XLSIJ2ZO6ZB7AFRAGBG7B3K5T4Q4EEGXKRUJ3QLQ5CNPRKSQ5N4A, seg_pos={0 0}, upload=501910]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Dec 29 11:59:11.145342329 (572h30m11.016425374s) [noise=true , address=77.90.31.14:4661 , piece_id=XLSIJ2ZO6ZB7AFRAGBG7B3K5T4Q4EEGXKRUJ3QLQ5CNPRKSQ5N4A, seg_pos={0 0}, upload=501910]: Long tail cancellation due to optimal threshold Dec 29 11:59:11.145359503 (572h30m11.016442538s) [seg_pos={0 0}, upload=501910]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 11:59:11.145451607 (572h30m11.016534652s) [upload=501910]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 11:59:11.145457439 (572h30m11.016540484s) [upload=501910]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 11:59:11.445524688 (572h30m11.316607733s) [upload=501910]: Done uploading segments... Dec 29 11:59:11.445548383 (572h30m11.316631429s) [upload=501910]: Upload finished. err: <nil> Duration: 5.75619466s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............X..X...........................................] duration: 46.193980844s trace: https://jaeger.dev.storj.io/trace/7e7354d8023608a8 FAIL (2025-12-29 12:00:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== ) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480097317 (571h53m39.351180352s) [noise=true , address=77.90.31.21:4889 , piece_id=BR6Z55P3A7YJ5FJRDSZFUIF66FGIJQY7WCEQ7PL5DOYWL73OMKTQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480405053 (571h53m39.351488098s) [noise=true , address=77.90.31.16:4750 , piece_id=MVVZPJEGDCBSNVOB6CD2WUZGIJTL3VBHX2G4L3TVZ6JF5ZEJ5U6Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480712969 (571h53m39.351796014s) [noise=true , address=77.90.31.18:4834 , piece_id=BTWNGNQVEUEVV6HLBDM63SGNTKV4RDU4J4XFLG52WHKP53H3L5KA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481019453 (571h53m39.352102488s) [noise=true , address=145.79.227.30:4777 , piece_id=VSCXLXGIBSFE6E4OHLWNZIZ75PO7WQO2VFHUYHW7JUNH62KGGU7Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481029532 (571h53m39.352112567s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481559832 (571h53m39.352642877s) [noise=true , address=145.79.227.35:4515 , piece_id=L7LFZKZEZERUNX4HUNDNQ2PWHXNJRJ7OBFRNMFIICPZEUIBHGP7A, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481731919 (571h53m39.352814954s) [noise=true , address=64.49.13.26:4953 , piece_id=4YSW5Y3A6HYG3XJ2XAB7RGVHAOHMP7G3QIZYMYTSXVXYJ3ANLNXA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482078329 (571h53m39.353161364s) [noise=true , address=145.79.227.44:4205 , piece_id=UE4QSHDQBZ3XBWQ6I6MCRVTLMU7U6W6OZDA2ZGQCH6ZYRAGPD7WA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482099830 (571h53m39.353182865s) [noise=true , address=145.79.227.40:4067 , piece_id=WSYDRX5YJ6OCFW52ZVMM6WZYNRFUXMR4XJ6C5727BOFMR4RRH33Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482434487 (571h53m39.353517532s) [noise=true , address=145.79.227.3:4164 , piece_id=U6F7VDPOH7K6R5EH5UUXHFP55GXOEXO5BLBJHZRTITPDQI4LP7KQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482480845 (571h53m39.35356388s) [noise=true , address=64.49.13.140:4516 , piece_id=4GUYOYOAEASUK2W25S3NILWIIIHGOZX76L4EGFLUUUEDWRALPP2A, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482841442 (571h53m39.353924477s) [noise=true , address=145.79.227.34:4910 , piece_id=ZH4DCNXET2Q6VRAUH3NQ3KKSV4OYI575X7YS72XD236EEOMOG3LA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483163645 (571h53m39.35424668s) [noise=true , address=145.79.227.29:4974 , piece_id=KEKREOD2B35XQNI45JBHE2SZZC3MQQBQUR5I6WFDAQYJQGJBXSMA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483532478 (571h53m39.354615513s) [noise=true , address=77.90.31.17:4769 , piece_id=3MKUMOQIUVERKQLLIFX52XQPTQ5LSFMHIPD5YW2YTO5YEFR36IYA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483914596 (571h53m39.354997631s) [noise=true , address=145.79.227.13:4807 , piece_id=VKXB5MWAG2YYH2AWWJEGTLBZ2QUPLI25VTHI6AMMALSM23Z3AWZQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:40.761721106 (571h53m40.632804141s) [upload=501384]: Next returned nil segment Dec 29 11:22:40.761733349 (571h53m40.632816384s) [upload=501384]: Waiting for error group managing segments... Dec 29 11:22:41.121222862 (571h53m40.992305897s) [noise=true , address=64.49.13.140:4516 , piece_id=4GUYOYOAEASUK2W25S3NILWIIIHGOZX76L4EGFLUUUEDWRALPP2A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.197311646 (571h53m41.068394691s) [noise=true , address=64.49.13.26:4953 , piece_id=4YSW5Y3A6HYG3XJ2XAB7RGVHAOHMP7G3QIZYMYTSXVXYJ3ANLNXA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.347247568 (571h53m41.218330613s) [noise=true , address=145.79.227.13:4807 , piece_id=VKXB5MWAG2YYH2AWWJEGTLBZ2QUPLI25VTHI6AMMALSM23Z3AWZQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.347917313 (571h53m41.219000348s) [noise=true , address=145.79.227.3:4164 , piece_id=U6F7VDPOH7K6R5EH5UUXHFP55GXOEXO5BLBJHZRTITPDQI4LP7KQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.348510613 (571h53m41.219593648s) [noise=true , address=145.79.227.29:4974 , piece_id=KEKREOD2B35XQNI45JBHE2SZZC3MQQBQUR5I6WFDAQYJQGJBXSMA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.503950985 (571h53m41.37503402s) [noise=true , address=77.90.31.17:4769 , piece_id=3MKUMOQIUVERKQLLIFX52XQPTQ5LSFMHIPD5YW2YTO5YEFR36IYA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.561352025 (571h53m41.43243506s) [noise=true , address=77.90.31.18:4834 , piece_id=BTWNGNQVEUEVV6HLBDM63SGNTKV4RDU4J4XFLG52WHKP53H3L5KA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.622447380 (571h53m41.493530425s) [noise=true , address=77.90.31.16:4750 , piece_id=MVVZPJEGDCBSNVOB6CD2WUZGIJTL3VBHX2G4L3TVZ6JF5ZEJ5U6Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.645494685 (571h53m41.51657773s) [noise=true , address=77.90.31.21:4889 , piece_id=BR6Z55P3A7YJ5FJRDSZFUIF66FGIJQY7WCEQ7PL5DOYWL73OMKTQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.247930045 (571h53m42.11901308s) [noise=true , address=145.79.227.30:4777 , piece_id=VSCXLXGIBSFE6E4OHLWNZIZ75PO7WQO2VFHUYHW7JUNH62KGGU7Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.262221890 (571h53m42.133304925s) [noise=true , address=145.79.227.44:4205 , piece_id=UE4QSHDQBZ3XBWQ6I6MCRVTLMU7U6W6OZDA2ZGQCH6ZYRAGPD7WA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.295598123 (571h53m42.166681158s) [noise=true , address=145.79.227.35:4515 , piece_id=L7LFZKZEZERUNX4HUNDNQ2PWHXNJRJ7OBFRNMFIICPZEUIBHGP7A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.307481351 (571h53m42.178564386s) [noise=true , address=145.79.227.34:4910 , piece_id=ZH4DCNXET2Q6VRAUH3NQ3KKSV4OYI575X7YS72XD236EEOMOG3LA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.325775906 (571h53m42.196858941s) [noise=true , address=145.79.227.40:4067 , piece_id=WSYDRX5YJ6OCFW52ZVMM6WZYNRFUXMR4XJ6C5727BOFMR4RRH33Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.325865978 (571h53m42.196949023s) [seg_pos={0 0}, upload=501384]: Segment reached optimal threshold of 14 pieces. Dec 29 11:22:42.326137074 (571h53m42.197220109s) [noise=true , address=64.49.13.169:4708 , piece_id=B2X2RWRELXWJPF6AVJJZCNU46LM5ASGRE4EWGPKL3SCLQ4VKUBDQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection Dec 29 11:22:42.326153074 (571h53m42.197236109s) [noise=true , address=64.49.13.169:4708 , piece_id=B2X2RWRELXWJPF6AVJJZCNU46LM5ASGRE4EWGPKL3SCLQ4VKUBDQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326193401 (571h53m42.197276446s) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12h5esfg6mw2c8y3AVpeFBkL5FfLAwYdjhcnQ9CJ8fjh61WLYjx): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326205705 (571h53m42.19728874s) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326407739 (571h53m42.197490784s) [noise=true , address=77.90.31.10:4524 , piece_id=Q2JGEUV5EDFTVY2CSNPZOVJIIIAFBA3IHA7S5N7YZRH3E7TSTE5A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uN7CznoK4AD8FqFkNUndawdGJ9rmjAe7oNnjJkV844ScY87qe): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326415374 (571h53m42.197498419s) [noise=true , address=77.90.31.10:4524 , piece_id=Q2JGEUV5EDFTVY2CSNPZOVJIIIAFBA3IHA7S5N7YZRH3E7TSTE5A, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326505105 (571h53m42.19758815s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tnTEYvqJ6PB2DsuXA28sEuEiGj11StA4qdhcYH1kuoaMkTVdY): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326512188 (571h53m42.197595223s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326532357 (571h53m42.197615392s) [seg_pos={0 0}, upload=501384]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 11:22:42.326671562 (571h53m42.197754597s) [upload=501384]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 11:22:42.326677843 (571h53m42.197760879s) [upload=501384]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 11:22:42.610391438 (571h53m42.481474493s) [upload=501384]: Done uploading segments... Dec 29 11:22:42.610416826 (571h53m42.481499871s) [upload=501384]: Upload finished. err: <nil> Duration: 46.194125689s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 31 out of 60 is failed (threshold=6) [XX..XX......X.X.XX.X.XX..X..XX.XXXX.XXXX...XXX...XXX..XX.X..] duration: 4.685914291s trace: https://jaeger.dev.storj.io/trace/34cfb808f43065bf FAIL (2025-12-29 11:59:51 UTC) orbiter warning: context deadline exceeded Duration: 4.685938687s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [..XX..X..XXX.X.......X.X..X.XXX...XX.XXX.X.X....XXXXXX..X.X.] duration: 5.972032794s trace: https://jaeger.dev.storj.io/trace/78a810c1312d39a9 FAIL (2025-12-29 11:59:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: TXY6KE2F3OOGRMOT4ZF3MERPEEXFOQONOJUJ2G5TDQQTT5TFGANA) context deadline exceeded; piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: 2PTAVS33GMBNXYULC2YFT6YTVLDSPBQOIYM2BHCXGYTTWLF2FNDQ) context deadline exceeded ======LOGS====== Dec 29 11:59:08.361853165 (572h30m8.2329362s) [upload=501910]: Starting upload Dec 29 11:59:08.361997270 (572h30m8.233080335s) [upload=501910]: Done starting upload Dec 29 11:59:08.362093513 (572h30m8.233176548s) [upload=501910]: Uploading segments... Dec 29 11:59:08.362292753 (572h30m8.233375798s) [upload=501910]: Got next segment. Inline: false Dec 29 11:59:08.362335234 (572h30m8.233418269s) [upload=501910]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 11:59:08.362340063 (572h30m8.233423098s) [upload=501910]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 11:59:08.611633275 (572h30m8.482716321s) [seg_pos={0 0}, upload=501910]: Begin upload segment... Dec 29 11:59:08.611679954 (572h30m8.482762999s) [seg_pos={0 0}, upload=501910]: Done begin upload segment. Dec 29 11:59:08.611705703 (572h30m8.482788738s) [noise=true , address=145.79.227.30:4761 , piece_id=FWOGG3AETOEAFIFEBVASAVOTUITY7TVF3JVWTOHDCTXDGQMLQGVA, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.611729619 (572h30m8.482812664s) [noise=true , address=45.91.116.14:4192 , piece_id=TXY6KE2F3OOGRMOT4ZF3MERPEEXFOQONOJUJ2G5TDQQTT5TFGANA, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.612625375 (572h30m8.48370841s) [noise=true , address=145.79.227.47:4266 , piece_id=XAZDWB62CWNFH7N5D4TGNXRLRT7YF2ZP4L6MNATUCZDDUXA2QDGQ, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.612648729 (572h30m8.483731774s) [noise=true , address=45.91.116.5:4298 , piece_id=TTGXECJ2OUJFDRSXM53VW6V2KJGT5ZTWKGEN6WOKN7RQZURKK62Q, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.613025066 (572h30m8.484108111s) [noise=true , address=45.91.116.9:4262 , piece_id=KXI3CRGDXPFIQMETUUKKJZS5YX3TY7XZJ6A2HCHO4JW3DNB2I43A, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.613521331 (572h30m8.484604376s) [noise=true , address=145.79.227.17:4970 , piece_id=PZIKWOOFEAAWSI3DDG6MN44EJ6RVL7HYC6HXLP3JGDVDRPPGORFQ, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.613528916 (572h30m8.484611961s) [noise=true , address=145.79.227.2:4115 , piece_id=VVEW5N7PMXNFLYMEIWFB6ZBTJWMBFK2PRWTFZKTUB5AUIPSOMCZA, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.613977430 (572h30m8.485060465s) [noise=true , address=145.79.227.46:4602 , piece_id=UO7XPJAI7VQXCF7SVBV43WTI6JPOK4DFQK3WMJ54EPXVSRWLA6QA, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.614275347 (572h30m8.485358382s) [noise=true , address=145.79.227.23:4545 , piece_id=XI7C54VR2CYZIE6OLAXP6UYWH2BQI5IFZLL6XFGMDPIF5UKKJUKA, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.614415964 (572h30m8.485499009s) [noise=true , address=77.90.31.14:4661 , piece_id=XLSIJ2ZO6ZB7AFRAGBG7B3K5T4Q4EEGXKRUJ3QLQ5CNPRKSQ5N4A, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.614940583 (572h30m8.486023618s) [noise=true , address=145.79.227.5:4222 , piece_id=UBA5YYPYYNUYRYHYURWMRXXB7Q6AMYZ3EZ3HZDLNF77OG4MDKEUQ, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.614969919 (572h30m8.486052964s) [noise=true , address=77.90.31.19:4842 , piece_id=2PTAVS33GMBNXYULC2YFT6YTVLDSPBQOIYM2BHCXGYTTWLF2FNDQ, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.615383867 (572h30m8.486466903s) [noise=true , address=145.79.227.14:4837 , piece_id=TE4VAA7KHS4YYQRGKMOMBM2WLVEMNB5NSML4JC7OY2EKHMDZQ6SA, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.615607825 (572h30m8.48669087s) [noise=true , address=145.79.227.21:4473 , piece_id=IDGJS4QYGBIORV26G45FWDSOA3H63UEEGPPPLLDN5RCFE3W4HAXA, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.615881095 (572h30m8.48696414s) [noise=true , address=145.79.227.26:4654 , piece_id=45IHGFKRVEDQDGQTSFALHRIE6IVQZHY7E3DCIZI6TNHLJFITPGQQ, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.616226894 (572h30m8.487309939s) [noise=true , address=145.79.227.50:4374 , piece_id=L2USZXC5XKVWUDSZ5MDLOZVAFJOMVGYF7SM2E5Q4EAU4EEZEF23Q, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:10.218180372 (572h30m10.089263407s) [upload=501910]: Next returned nil segment Dec 29 11:59:10.218197525 (572h30m10.08928056s) [upload=501910]: Waiting for error group managing segments... Dec 29 11:59:10.596756931 (572h30m10.467839966s) [noise=true , address=145.79.227.17:4970 , piece_id=PZIKWOOFEAAWSI3DDG6MN44EJ6RVL7HYC6HXLP3JGDVDRPPGORFQ, seg_pos={0 0}, upload=501910]: Done uploading piece. err: <nil> Dec 29 11:59:10.597714194 (572h30m10.46879724s) [noise=true , address=145.79.227.47:4266 , piece_id=XAZDWB62CWNFH7N5D4TGNXRLRT7YF2ZP4L6MNATUCZDDUXA2QDGQ, seg_pos={0 0}, upload=501910]: Done uploading piece. err: <nil> Dec 29 11:59:10.600236358 (572h30m10.471319393s) [noise=true , address=145.79.227.14:4837 , piece_id=TE4VAA7KHS4YYQRGKMOMBM2WLVEMNB5NSML4JC7OY2EKHMDZQ6SA, seg_pos={0 0}, upload=501910]: Done uploading piece. err: <nil> Dec 29 11:59:10.602320044 (572h30m10.47340308s) [noise=true , address=145.79.227.5:4222 , piece_id=UBA5YYPYYNUYRYHYURWMRXXB7Q6AMYZ3EZ3HZDLNF77OG4MDKEUQ, seg_pos={0 0}, upload=501910]: Done uploading piece. err: <nil> Dec 29 11:59:10.602912464 (572h30m10.473995499s) [noise=true , address=145.79.227.26:4654 , piece_id=45IHGFKRVEDQDGQTSFALHRIE6IVQZHY7E3DCIZI6TNHLJFITPGQQ, seg_pos={0 0}, upload=501910]: Done uploading piece. err: <nil> Dec 29 11:59:10.611761013 (572h30m10.482844048s) [noise=true , address=145.79.227.23:4545 , piece_id=XI7C54VR2CYZIE6OLAXP6UYWH2BQI5IFZLL6XFGMDPIF5UKKJUKA, seg_pos={0 0}, upload=501910]: Done uploading piece. err: <nil> Dec 29 11:59:10.623036624 (572h30m10.494119669s) [noise=true , address=145.79.227.46:4602 , piece_id=UO7XPJAI7VQXCF7SVBV43WTI6JPOK4DFQK3WMJ54EPXVSRWLA6QA, seg_pos={0 0}, upload=501910]: Done uploading piece. err: <nil> Dec 29 11:59:10.626994862 (572h30m10.498077907s) [noise=true , address=145.79.227.50:4374 , piece_id=L2USZXC5XKVWUDSZ5MDLOZVAFJOMVGYF7SM2E5Q4EAU4EEZEF23Q, seg_pos={0 0}, upload=501910]: Done uploading piece. err: <nil> Dec 29 11:59:10.627968646 (572h30m10.499051691s) [noise=true , address=145.79.227.21:4473 , piece_id=IDGJS4QYGBIORV26G45FWDSOA3H63UEEGPPPLLDN5RCFE3W4HAXA, seg_pos={0 0}, upload=501910]: Done uploading piece. err: <nil> Dec 29 11:59:10.629745359 (572h30m10.500828404s) [noise=true , address=145.79.227.2:4115 , piece_id=VVEW5N7PMXNFLYMEIWFB6ZBTJWMBFK2PRWTFZKTUB5AUIPSOMCZA, seg_pos={0 0}, upload=501910]: Done uploading piece. err: <nil> Dec 29 11:59:10.847903124 (572h30m10.718986159s) [noise=true , address=45.91.116.14:4192 , piece_id=TXY6KE2F3OOGRMOT4ZF3MERPEEXFOQONOJUJ2G5TDQQTT5TFGANA, seg_pos={0 0}, upload=501910]: Done uploading piece. err: <nil> Dec 29 11:59:10.901405164 (572h30m10.772488199s) [noise=true , address=77.90.31.19:4842 , piece_id=2PTAVS33GMBNXYULC2YFT6YTVLDSPBQOIYM2BHCXGYTTWLF2FNDQ, seg_pos={0 0}, upload=501910]: Done uploading piece. err: <nil> Dec 29 11:59:11.144788475 (572h30m11.01587151s) [noise=true , address=45.91.116.9:4262 , piece_id=KXI3CRGDXPFIQMETUUKKJZS5YX3TY7XZJ6A2HCHO4JW3DNB2I43A, seg_pos={0 0}, upload=501910]: Done uploading piece. err: <nil> Dec 29 11:59:11.144812380 (572h30m11.015895415s) [seg_pos={0 0}, upload=501910]: Segment reached optimal threshold of 13 pieces. Dec 29 11:59:11.145162818 (572h30m11.016245863s) [noise=true , address=45.91.116.5:4298 , piece_id=TTGXECJ2OUJFDRSXM53VW6V2KJGT5ZTWKGEN6WOKN7RQZURKK62Q, seg_pos={0 0}, upload=501910]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 29 11:59:11.145173508 (572h30m11.016256543s) [noise=true , address=45.91.116.5:4298 , piece_id=TTGXECJ2OUJFDRSXM53VW6V2KJGT5ZTWKGEN6WOKN7RQZURKK62Q, seg_pos={0 0}, upload=501910]: Long tail cancellation due to optimal threshold Dec 29 11:59:11.145253922 (572h30m11.016336967s) [noise=true , address=145.79.227.30:4761 , piece_id=FWOGG3AETOEAFIFEBVASAVOTUITY7TVF3JVWTOHDCTXDGQMLQGVA, seg_pos={0 0}, upload=501910]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 29 11:59:11.145265453 (572h30m11.016348498s) [noise=true , address=145.79.227.30:4761 , piece_id=FWOGG3AETOEAFIFEBVASAVOTUITY7TVF3JVWTOHDCTXDGQMLQGVA, seg_pos={0 0}, upload=501910]: Long tail cancellation due to optimal threshold Dec 29 11:59:11.145333914 (572h30m11.016416949s) [noise=true , address=77.90.31.14:4661 , piece_id=XLSIJ2ZO6ZB7AFRAGBG7B3K5T4Q4EEGXKRUJ3QLQ5CNPRKSQ5N4A, seg_pos={0 0}, upload=501910]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Dec 29 11:59:11.145342329 (572h30m11.016425374s) [noise=true , address=77.90.31.14:4661 , piece_id=XLSIJ2ZO6ZB7AFRAGBG7B3K5T4Q4EEGXKRUJ3QLQ5CNPRKSQ5N4A, seg_pos={0 0}, upload=501910]: Long tail cancellation due to optimal threshold Dec 29 11:59:11.145359503 (572h30m11.016442538s) [seg_pos={0 0}, upload=501910]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 11:59:11.145451607 (572h30m11.016534652s) [upload=501910]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 11:59:11.145457439 (572h30m11.016540484s) [upload=501910]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 11:59:11.445524688 (572h30m11.316607733s) [upload=501910]: Done uploading segments... Dec 29 11:59:11.445548383 (572h30m11.316631429s) [upload=501910]: Upload finished. err: <nil> Duration: 5.972060316s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 29 out of 60 is failed (threshold=3) [X..XX..X..XXX.X.......X.X..X.XXX...XX.XXX.X.X....XXXXXX..X.X] duration: 30.003386462s trace: https://jaeger.dev.storj.io/trace/78a810c1312d39a9 FAIL (2025-12-29 11:59:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: TXY6KE2F3OOGRMOT4ZF3MERPEEXFOQONOJUJ2G5TDQQTT5TFGANA) context deadline exceeded; piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: 2PTAVS33GMBNXYULC2YFT6YTVLDSPBQOIYM2BHCXGYTTWLF2FNDQ) context deadline exceeded ======LOGS====== Dec 29 11:59:08.361853165 (572h30m8.2329362s) [upload=501910]: Starting upload Dec 29 11:59:08.361997270 (572h30m8.233080335s) [upload=501910]: Done starting upload Dec 29 11:59:08.362093513 (572h30m8.233176548s) [upload=501910]: Uploading segments... Dec 29 11:59:08.362292753 (572h30m8.233375798s) [upload=501910]: Got next segment. Inline: false Dec 29 11:59:08.362335234 (572h30m8.233418269s) [upload=501910]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 11:59:08.362340063 (572h30m8.233423098s) [upload=501910]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 11:59:08.611633275 (572h30m8.482716321s) [seg_pos={0 0}, upload=501910]: Begin upload segment... Dec 29 11:59:08.611679954 (572h30m8.482762999s) [seg_pos={0 0}, upload=501910]: Done begin upload segment. Dec 29 11:59:08.611705703 (572h30m8.482788738s) [noise=true , address=145.79.227.30:4761 , piece_id=FWOGG3AETOEAFIFEBVASAVOTUITY7TVF3JVWTOHDCTXDGQMLQGVA, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.611729619 (572h30m8.482812664s) [noise=true , address=45.91.116.14:4192 , piece_id=TXY6KE2F3OOGRMOT4ZF3MERPEEXFOQONOJUJ2G5TDQQTT5TFGANA, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.612625375 (572h30m8.48370841s) [noise=true , address=145.79.227.47:4266 , piece_id=XAZDWB62CWNFH7N5D4TGNXRLRT7YF2ZP4L6MNATUCZDDUXA2QDGQ, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.612648729 (572h30m8.483731774s) [noise=true , address=45.91.116.5:4298 , piece_id=TTGXECJ2OUJFDRSXM53VW6V2KJGT5ZTWKGEN6WOKN7RQZURKK62Q, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.613025066 (572h30m8.484108111s) [noise=true , address=45.91.116.9:4262 , piece_id=KXI3CRGDXPFIQMETUUKKJZS5YX3TY7XZJ6A2HCHO4JW3DNB2I43A, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.613521331 (572h30m8.484604376s) [noise=true , address=145.79.227.17:4970 , piece_id=PZIKWOOFEAAWSI3DDG6MN44EJ6RVL7HYC6HXLP3JGDVDRPPGORFQ, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.613528916 (572h30m8.484611961s) [noise=true , address=145.79.227.2:4115 , piece_id=VVEW5N7PMXNFLYMEIWFB6ZBTJWMBFK2PRWTFZKTUB5AUIPSOMCZA, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.613977430 (572h30m8.485060465s) [noise=true , address=145.79.227.46:4602 , piece_id=UO7XPJAI7VQXCF7SVBV43WTI6JPOK4DFQK3WMJ54EPXVSRWLA6QA, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.614275347 (572h30m8.485358382s) [noise=true , address=145.79.227.23:4545 , piece_id=XI7C54VR2CYZIE6OLAXP6UYWH2BQI5IFZLL6XFGMDPIF5UKKJUKA, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.614415964 (572h30m8.485499009s) [noise=true , address=77.90.31.14:4661 , piece_id=XLSIJ2ZO6ZB7AFRAGBG7B3K5T4Q4EEGXKRUJ3QLQ5CNPRKSQ5N4A, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.614940583 (572h30m8.486023618s) [noise=true , address=145.79.227.5:4222 , piece_id=UBA5YYPYYNUYRYHYURWMRXXB7Q6AMYZ3EZ3HZDLNF77OG4MDKEUQ, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.614969919 (572h30m8.486052964s) [noise=true , address=77.90.31.19:4842 , piece_id=2PTAVS33GMBNXYULC2YFT6YTVLDSPBQOIYM2BHCXGYTTWLF2FNDQ, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.615383867 (572h30m8.486466903s) [noise=true , address=145.79.227.14:4837 , piece_id=TE4VAA7KHS4YYQRGKMOMBM2WLVEMNB5NSML4JC7OY2EKHMDZQ6SA, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.615607825 (572h30m8.48669087s) [noise=true , address=145.79.227.21:4473 , piece_id=IDGJS4QYGBIORV26G45FWDSOA3H63UEEGPPPLLDN5RCFE3W4HAXA, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.615881095 (572h30m8.48696414s) [noise=true , address=145.79.227.26:4654 , piece_id=45IHGFKRVEDQDGQTSFALHRIE6IVQZHY7E3DCIZI6TNHLJFITPGQQ, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:08.616226894 (572h30m8.487309939s) [noise=true , address=145.79.227.50:4374 , piece_id=L2USZXC5XKVWUDSZ5MDLOZVAFJOMVGYF7SM2E5Q4EAU4EEZEF23Q, seg_pos={0 0}, upload=501910]: Uploading piece... Dec 29 11:59:10.218180372 (572h30m10.089263407s) [upload=501910]: Next returned nil segment Dec 29 11:59:10.218197525 (572h30m10.08928056s) [upload=501910]: Waiting for error group managing segments... Dec 29 11:59:10.596756931 (572h30m10.467839966s) [noise=true , address=145.79.227.17:4970 , piece_id=PZIKWOOFEAAWSI3DDG6MN44EJ6RVL7HYC6HXLP3JGDVDRPPGORFQ, seg_pos={0 0}, upload=501910]: Done uploading piece. err: <nil> Dec 29 11:59:10.597714194 (572h30m10.46879724s) [noise=true , address=145.79.227.47:4266 , piece_id=XAZDWB62CWNFH7N5D4TGNXRLRT7YF2ZP4L6MNATUCZDDUXA2QDGQ, seg_pos={0 0}, upload=501910]: Done uploading piece. err: <nil> Dec 29 11:59:10.600236358 (572h30m10.471319393s) [noise=true , address=145.79.227.14:4837 , piece_id=TE4VAA7KHS4YYQRGKMOMBM2WLVEMNB5NSML4JC7OY2EKHMDZQ6SA, seg_pos={0 0}, upload=501910]: Done uploading piece. err: <nil> Dec 29 11:59:10.602320044 (572h30m10.47340308s) [noise=true , address=145.79.227.5:4222 , piece_id=UBA5YYPYYNUYRYHYURWMRXXB7Q6AMYZ3EZ3HZDLNF77OG4MDKEUQ, seg_pos={0 0}, upload=501910]: Done uploading piece. err: <nil> Dec 29 11:59:10.602912464 (572h30m10.473995499s) [noise=true , address=145.79.227.26:4654 , piece_id=45IHGFKRVEDQDGQTSFALHRIE6IVQZHY7E3DCIZI6TNHLJFITPGQQ, seg_pos={0 0}, upload=501910]: Done uploading piece. err: <nil> Dec 29 11:59:10.611761013 (572h30m10.482844048s) [noise=true , address=145.79.227.23:4545 , piece_id=XI7C54VR2CYZIE6OLAXP6UYWH2BQI5IFZLL6XFGMDPIF5UKKJUKA, seg_pos={0 0}, upload=501910]: Done uploading piece. err: <nil> Dec 29 11:59:10.623036624 (572h30m10.494119669s) [noise=true , address=145.79.227.46:4602 , piece_id=UO7XPJAI7VQXCF7SVBV43WTI6JPOK4DFQK3WMJ54EPXVSRWLA6QA, seg_pos={0 0}, upload=501910]: Done uploading piece. err: <nil> Dec 29 11:59:10.626994862 (572h30m10.498077907s) [noise=true , address=145.79.227.50:4374 , piece_id=L2USZXC5XKVWUDSZ5MDLOZVAFJOMVGYF7SM2E5Q4EAU4EEZEF23Q, seg_pos={0 0}, upload=501910]: Done uploading piece. err: <nil> Dec 29 11:59:10.627968646 (572h30m10.499051691s) [noise=true , address=145.79.227.21:4473 , piece_id=IDGJS4QYGBIORV26G45FWDSOA3H63UEEGPPPLLDN5RCFE3W4HAXA, seg_pos={0 0}, upload=501910]: Done uploading piece. err: <nil> Dec 29 11:59:10.629745359 (572h30m10.500828404s) [noise=true , address=145.79.227.2:4115 , piece_id=VVEW5N7PMXNFLYMEIWFB6ZBTJWMBFK2PRWTFZKTUB5AUIPSOMCZA, seg_pos={0 0}, upload=501910]: Done uploading piece. err: <nil> Dec 29 11:59:10.847903124 (572h30m10.718986159s) [noise=true , address=45.91.116.14:4192 , piece_id=TXY6KE2F3OOGRMOT4ZF3MERPEEXFOQONOJUJ2G5TDQQTT5TFGANA, seg_pos={0 0}, upload=501910]: Done uploading piece. err: <nil> Dec 29 11:59:10.901405164 (572h30m10.772488199s) [noise=true , address=77.90.31.19:4842 , piece_id=2PTAVS33GMBNXYULC2YFT6YTVLDSPBQOIYM2BHCXGYTTWLF2FNDQ, seg_pos={0 0}, upload=501910]: Done uploading piece. err: <nil> Dec 29 11:59:11.144788475 (572h30m11.01587151s) [noise=true , address=45.91.116.9:4262 , piece_id=KXI3CRGDXPFIQMETUUKKJZS5YX3TY7XZJ6A2HCHO4JW3DNB2I43A, seg_pos={0 0}, upload=501910]: Done uploading piece. err: <nil> Dec 29 11:59:11.144812380 (572h30m11.015895415s) [seg_pos={0 0}, upload=501910]: Segment reached optimal threshold of 13 pieces. Dec 29 11:59:11.145162818 (572h30m11.016245863s) [noise=true , address=45.91.116.5:4298 , piece_id=TTGXECJ2OUJFDRSXM53VW6V2KJGT5ZTWKGEN6WOKN7RQZURKK62Q, seg_pos={0 0}, upload=501910]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 29 11:59:11.145173508 (572h30m11.016256543s) [noise=true , address=45.91.116.5:4298 , piece_id=TTGXECJ2OUJFDRSXM53VW6V2KJGT5ZTWKGEN6WOKN7RQZURKK62Q, seg_pos={0 0}, upload=501910]: Long tail cancellation due to optimal threshold Dec 29 11:59:11.145253922 (572h30m11.016336967s) [noise=true , address=145.79.227.30:4761 , piece_id=FWOGG3AETOEAFIFEBVASAVOTUITY7TVF3JVWTOHDCTXDGQMLQGVA, seg_pos={0 0}, upload=501910]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 29 11:59:11.145265453 (572h30m11.016348498s) [noise=true , address=145.79.227.30:4761 , piece_id=FWOGG3AETOEAFIFEBVASAVOTUITY7TVF3JVWTOHDCTXDGQMLQGVA, seg_pos={0 0}, upload=501910]: Long tail cancellation due to optimal threshold Dec 29 11:59:11.145333914 (572h30m11.016416949s) [noise=true , address=77.90.31.14:4661 , piece_id=XLSIJ2ZO6ZB7AFRAGBG7B3K5T4Q4EEGXKRUJ3QLQ5CNPRKSQ5N4A, seg_pos={0 0}, upload=501910]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Dec 29 11:59:11.145342329 (572h30m11.016425374s) [noise=true , address=77.90.31.14:4661 , piece_id=XLSIJ2ZO6ZB7AFRAGBG7B3K5T4Q4EEGXKRUJ3QLQ5CNPRKSQ5N4A, seg_pos={0 0}, upload=501910]: Long tail cancellation due to optimal threshold Dec 29 11:59:11.145359503 (572h30m11.016442538s) [seg_pos={0 0}, upload=501910]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 11:59:11.145451607 (572h30m11.016534652s) [upload=501910]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 11:59:11.145457439 (572h30m11.016540484s) [upload=501910]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 11:59:11.445524688 (572h30m11.316607733s) [upload=501910]: Done uploading segments... Dec 29 11:59:11.445548383 (572h30m11.316631429s) [upload=501910]: Upload finished. err: <nil> Duration: 30.003417823s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............X..X..........................................] duration: 32.2843507s trace: https://jaeger.dev.storj.io/trace/7e7354d8023608a8 FAIL (2025-12-29 11:59:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== ) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480097317 (571h53m39.351180352s) [noise=true , address=77.90.31.21:4889 , piece_id=BR6Z55P3A7YJ5FJRDSZFUIF66FGIJQY7WCEQ7PL5DOYWL73OMKTQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480405053 (571h53m39.351488098s) [noise=true , address=77.90.31.16:4750 , piece_id=MVVZPJEGDCBSNVOB6CD2WUZGIJTL3VBHX2G4L3TVZ6JF5ZEJ5U6Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480712969 (571h53m39.351796014s) [noise=true , address=77.90.31.18:4834 , piece_id=BTWNGNQVEUEVV6HLBDM63SGNTKV4RDU4J4XFLG52WHKP53H3L5KA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481019453 (571h53m39.352102488s) [noise=true , address=145.79.227.30:4777 , piece_id=VSCXLXGIBSFE6E4OHLWNZIZ75PO7WQO2VFHUYHW7JUNH62KGGU7Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481029532 (571h53m39.352112567s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481559832 (571h53m39.352642877s) [noise=true , address=145.79.227.35:4515 , piece_id=L7LFZKZEZERUNX4HUNDNQ2PWHXNJRJ7OBFRNMFIICPZEUIBHGP7A, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481731919 (571h53m39.352814954s) [noise=true , address=64.49.13.26:4953 , piece_id=4YSW5Y3A6HYG3XJ2XAB7RGVHAOHMP7G3QIZYMYTSXVXYJ3ANLNXA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482078329 (571h53m39.353161364s) [noise=true , address=145.79.227.44:4205 , piece_id=UE4QSHDQBZ3XBWQ6I6MCRVTLMU7U6W6OZDA2ZGQCH6ZYRAGPD7WA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482099830 (571h53m39.353182865s) [noise=true , address=145.79.227.40:4067 , piece_id=WSYDRX5YJ6OCFW52ZVMM6WZYNRFUXMR4XJ6C5727BOFMR4RRH33Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482434487 (571h53m39.353517532s) [noise=true , address=145.79.227.3:4164 , piece_id=U6F7VDPOH7K6R5EH5UUXHFP55GXOEXO5BLBJHZRTITPDQI4LP7KQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482480845 (571h53m39.35356388s) [noise=true , address=64.49.13.140:4516 , piece_id=4GUYOYOAEASUK2W25S3NILWIIIHGOZX76L4EGFLUUUEDWRALPP2A, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482841442 (571h53m39.353924477s) [noise=true , address=145.79.227.34:4910 , piece_id=ZH4DCNXET2Q6VRAUH3NQ3KKSV4OYI575X7YS72XD236EEOMOG3LA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483163645 (571h53m39.35424668s) [noise=true , address=145.79.227.29:4974 , piece_id=KEKREOD2B35XQNI45JBHE2SZZC3MQQBQUR5I6WFDAQYJQGJBXSMA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483532478 (571h53m39.354615513s) [noise=true , address=77.90.31.17:4769 , piece_id=3MKUMOQIUVERKQLLIFX52XQPTQ5LSFMHIPD5YW2YTO5YEFR36IYA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483914596 (571h53m39.354997631s) [noise=true , address=145.79.227.13:4807 , piece_id=VKXB5MWAG2YYH2AWWJEGTLBZ2QUPLI25VTHI6AMMALSM23Z3AWZQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:40.761721106 (571h53m40.632804141s) [upload=501384]: Next returned nil segment Dec 29 11:22:40.761733349 (571h53m40.632816384s) [upload=501384]: Waiting for error group managing segments... Dec 29 11:22:41.121222862 (571h53m40.992305897s) [noise=true , address=64.49.13.140:4516 , piece_id=4GUYOYOAEASUK2W25S3NILWIIIHGOZX76L4EGFLUUUEDWRALPP2A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.197311646 (571h53m41.068394691s) [noise=true , address=64.49.13.26:4953 , piece_id=4YSW5Y3A6HYG3XJ2XAB7RGVHAOHMP7G3QIZYMYTSXVXYJ3ANLNXA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.347247568 (571h53m41.218330613s) [noise=true , address=145.79.227.13:4807 , piece_id=VKXB5MWAG2YYH2AWWJEGTLBZ2QUPLI25VTHI6AMMALSM23Z3AWZQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.347917313 (571h53m41.219000348s) [noise=true , address=145.79.227.3:4164 , piece_id=U6F7VDPOH7K6R5EH5UUXHFP55GXOEXO5BLBJHZRTITPDQI4LP7KQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.348510613 (571h53m41.219593648s) [noise=true , address=145.79.227.29:4974 , piece_id=KEKREOD2B35XQNI45JBHE2SZZC3MQQBQUR5I6WFDAQYJQGJBXSMA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.503950985 (571h53m41.37503402s) [noise=true , address=77.90.31.17:4769 , piece_id=3MKUMOQIUVERKQLLIFX52XQPTQ5LSFMHIPD5YW2YTO5YEFR36IYA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.561352025 (571h53m41.43243506s) [noise=true , address=77.90.31.18:4834 , piece_id=BTWNGNQVEUEVV6HLBDM63SGNTKV4RDU4J4XFLG52WHKP53H3L5KA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.622447380 (571h53m41.493530425s) [noise=true , address=77.90.31.16:4750 , piece_id=MVVZPJEGDCBSNVOB6CD2WUZGIJTL3VBHX2G4L3TVZ6JF5ZEJ5U6Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.645494685 (571h53m41.51657773s) [noise=true , address=77.90.31.21:4889 , piece_id=BR6Z55P3A7YJ5FJRDSZFUIF66FGIJQY7WCEQ7PL5DOYWL73OMKTQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.247930045 (571h53m42.11901308s) [noise=true , address=145.79.227.30:4777 , piece_id=VSCXLXGIBSFE6E4OHLWNZIZ75PO7WQO2VFHUYHW7JUNH62KGGU7Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.262221890 (571h53m42.133304925s) [noise=true , address=145.79.227.44:4205 , piece_id=UE4QSHDQBZ3XBWQ6I6MCRVTLMU7U6W6OZDA2ZGQCH6ZYRAGPD7WA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.295598123 (571h53m42.166681158s) [noise=true , address=145.79.227.35:4515 , piece_id=L7LFZKZEZERUNX4HUNDNQ2PWHXNJRJ7OBFRNMFIICPZEUIBHGP7A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.307481351 (571h53m42.178564386s) [noise=true , address=145.79.227.34:4910 , piece_id=ZH4DCNXET2Q6VRAUH3NQ3KKSV4OYI575X7YS72XD236EEOMOG3LA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.325775906 (571h53m42.196858941s) [noise=true , address=145.79.227.40:4067 , piece_id=WSYDRX5YJ6OCFW52ZVMM6WZYNRFUXMR4XJ6C5727BOFMR4RRH33Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.325865978 (571h53m42.196949023s) [seg_pos={0 0}, upload=501384]: Segment reached optimal threshold of 14 pieces. Dec 29 11:22:42.326137074 (571h53m42.197220109s) [noise=true , address=64.49.13.169:4708 , piece_id=B2X2RWRELXWJPF6AVJJZCNU46LM5ASGRE4EWGPKL3SCLQ4VKUBDQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection Dec 29 11:22:42.326153074 (571h53m42.197236109s) [noise=true , address=64.49.13.169:4708 , piece_id=B2X2RWRELXWJPF6AVJJZCNU46LM5ASGRE4EWGPKL3SCLQ4VKUBDQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326193401 (571h53m42.197276446s) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12h5esfg6mw2c8y3AVpeFBkL5FfLAwYdjhcnQ9CJ8fjh61WLYjx): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326205705 (571h53m42.19728874s) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326407739 (571h53m42.197490784s) [noise=true , address=77.90.31.10:4524 , piece_id=Q2JGEUV5EDFTVY2CSNPZOVJIIIAFBA3IHA7S5N7YZRH3E7TSTE5A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uN7CznoK4AD8FqFkNUndawdGJ9rmjAe7oNnjJkV844ScY87qe): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326415374 (571h53m42.197498419s) [noise=true , address=77.90.31.10:4524 , piece_id=Q2JGEUV5EDFTVY2CSNPZOVJIIIAFBA3IHA7S5N7YZRH3E7TSTE5A, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326505105 (571h53m42.19758815s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tnTEYvqJ6PB2DsuXA28sEuEiGj11StA4qdhcYH1kuoaMkTVdY): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326512188 (571h53m42.197595223s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326532357 (571h53m42.197615392s) [seg_pos={0 0}, upload=501384]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 11:22:42.326671562 (571h53m42.197754597s) [upload=501384]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 11:22:42.326677843 (571h53m42.197760879s) [upload=501384]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 11:22:42.610391438 (571h53m42.481474493s) [upload=501384]: Done uploading segments... Dec 29 11:22:42.610416826 (571h53m42.481499871s) [upload=501384]: Upload finished. err: <nil> Duration: 32.284370618s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 31 out of 60 is failed (threshold=6) [.XX..XX......X.X.XX.X.XX..X..XX.XXXX.XXXX...XXX...XXX..XX.X.] duration: 3.782657122s trace: https://jaeger.dev.storj.io/trace/34cfb808f43065bf FAIL (2025-12-29 11:59:07 UTC) orbiter warning: context deadline exceeded Duration: 3.782684184s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 31 out of 60 is failed (threshold=6) [..XX..XX......X.X.XX.X.XX..X..XX.XXXX.XXXX...XXX...XXX..XX.X] duration: 5.00114696s trace: https://jaeger.dev.storj.io/trace/34cfb808f43065bf FAIL (2025-12-29 11:58:39 UTC) orbiter warning: context deadline exceeded Duration: 5.001180584s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............X..X.........................................] duration: 45.354617202s trace: https://jaeger.dev.storj.io/trace/7e7354d8023608a8 FAIL (2025-12-29 11:58:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== ) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480097317 (571h53m39.351180352s) [noise=true , address=77.90.31.21:4889 , piece_id=BR6Z55P3A7YJ5FJRDSZFUIF66FGIJQY7WCEQ7PL5DOYWL73OMKTQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480405053 (571h53m39.351488098s) [noise=true , address=77.90.31.16:4750 , piece_id=MVVZPJEGDCBSNVOB6CD2WUZGIJTL3VBHX2G4L3TVZ6JF5ZEJ5U6Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480712969 (571h53m39.351796014s) [noise=true , address=77.90.31.18:4834 , piece_id=BTWNGNQVEUEVV6HLBDM63SGNTKV4RDU4J4XFLG52WHKP53H3L5KA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481019453 (571h53m39.352102488s) [noise=true , address=145.79.227.30:4777 , piece_id=VSCXLXGIBSFE6E4OHLWNZIZ75PO7WQO2VFHUYHW7JUNH62KGGU7Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481029532 (571h53m39.352112567s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481559832 (571h53m39.352642877s) [noise=true , address=145.79.227.35:4515 , piece_id=L7LFZKZEZERUNX4HUNDNQ2PWHXNJRJ7OBFRNMFIICPZEUIBHGP7A, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481731919 (571h53m39.352814954s) [noise=true , address=64.49.13.26:4953 , piece_id=4YSW5Y3A6HYG3XJ2XAB7RGVHAOHMP7G3QIZYMYTSXVXYJ3ANLNXA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482078329 (571h53m39.353161364s) [noise=true , address=145.79.227.44:4205 , piece_id=UE4QSHDQBZ3XBWQ6I6MCRVTLMU7U6W6OZDA2ZGQCH6ZYRAGPD7WA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482099830 (571h53m39.353182865s) [noise=true , address=145.79.227.40:4067 , piece_id=WSYDRX5YJ6OCFW52ZVMM6WZYNRFUXMR4XJ6C5727BOFMR4RRH33Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482434487 (571h53m39.353517532s) [noise=true , address=145.79.227.3:4164 , piece_id=U6F7VDPOH7K6R5EH5UUXHFP55GXOEXO5BLBJHZRTITPDQI4LP7KQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482480845 (571h53m39.35356388s) [noise=true , address=64.49.13.140:4516 , piece_id=4GUYOYOAEASUK2W25S3NILWIIIHGOZX76L4EGFLUUUEDWRALPP2A, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482841442 (571h53m39.353924477s) [noise=true , address=145.79.227.34:4910 , piece_id=ZH4DCNXET2Q6VRAUH3NQ3KKSV4OYI575X7YS72XD236EEOMOG3LA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483163645 (571h53m39.35424668s) [noise=true , address=145.79.227.29:4974 , piece_id=KEKREOD2B35XQNI45JBHE2SZZC3MQQBQUR5I6WFDAQYJQGJBXSMA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483532478 (571h53m39.354615513s) [noise=true , address=77.90.31.17:4769 , piece_id=3MKUMOQIUVERKQLLIFX52XQPTQ5LSFMHIPD5YW2YTO5YEFR36IYA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483914596 (571h53m39.354997631s) [noise=true , address=145.79.227.13:4807 , piece_id=VKXB5MWAG2YYH2AWWJEGTLBZ2QUPLI25VTHI6AMMALSM23Z3AWZQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:40.761721106 (571h53m40.632804141s) [upload=501384]: Next returned nil segment Dec 29 11:22:40.761733349 (571h53m40.632816384s) [upload=501384]: Waiting for error group managing segments... Dec 29 11:22:41.121222862 (571h53m40.992305897s) [noise=true , address=64.49.13.140:4516 , piece_id=4GUYOYOAEASUK2W25S3NILWIIIHGOZX76L4EGFLUUUEDWRALPP2A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.197311646 (571h53m41.068394691s) [noise=true , address=64.49.13.26:4953 , piece_id=4YSW5Y3A6HYG3XJ2XAB7RGVHAOHMP7G3QIZYMYTSXVXYJ3ANLNXA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.347247568 (571h53m41.218330613s) [noise=true , address=145.79.227.13:4807 , piece_id=VKXB5MWAG2YYH2AWWJEGTLBZ2QUPLI25VTHI6AMMALSM23Z3AWZQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.347917313 (571h53m41.219000348s) [noise=true , address=145.79.227.3:4164 , piece_id=U6F7VDPOH7K6R5EH5UUXHFP55GXOEXO5BLBJHZRTITPDQI4LP7KQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.348510613 (571h53m41.219593648s) [noise=true , address=145.79.227.29:4974 , piece_id=KEKREOD2B35XQNI45JBHE2SZZC3MQQBQUR5I6WFDAQYJQGJBXSMA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.503950985 (571h53m41.37503402s) [noise=true , address=77.90.31.17:4769 , piece_id=3MKUMOQIUVERKQLLIFX52XQPTQ5LSFMHIPD5YW2YTO5YEFR36IYA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.561352025 (571h53m41.43243506s) [noise=true , address=77.90.31.18:4834 , piece_id=BTWNGNQVEUEVV6HLBDM63SGNTKV4RDU4J4XFLG52WHKP53H3L5KA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.622447380 (571h53m41.493530425s) [noise=true , address=77.90.31.16:4750 , piece_id=MVVZPJEGDCBSNVOB6CD2WUZGIJTL3VBHX2G4L3TVZ6JF5ZEJ5U6Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.645494685 (571h53m41.51657773s) [noise=true , address=77.90.31.21:4889 , piece_id=BR6Z55P3A7YJ5FJRDSZFUIF66FGIJQY7WCEQ7PL5DOYWL73OMKTQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.247930045 (571h53m42.11901308s) [noise=true , address=145.79.227.30:4777 , piece_id=VSCXLXGIBSFE6E4OHLWNZIZ75PO7WQO2VFHUYHW7JUNH62KGGU7Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.262221890 (571h53m42.133304925s) [noise=true , address=145.79.227.44:4205 , piece_id=UE4QSHDQBZ3XBWQ6I6MCRVTLMU7U6W6OZDA2ZGQCH6ZYRAGPD7WA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.295598123 (571h53m42.166681158s) [noise=true , address=145.79.227.35:4515 , piece_id=L7LFZKZEZERUNX4HUNDNQ2PWHXNJRJ7OBFRNMFIICPZEUIBHGP7A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.307481351 (571h53m42.178564386s) [noise=true , address=145.79.227.34:4910 , piece_id=ZH4DCNXET2Q6VRAUH3NQ3KKSV4OYI575X7YS72XD236EEOMOG3LA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.325775906 (571h53m42.196858941s) [noise=true , address=145.79.227.40:4067 , piece_id=WSYDRX5YJ6OCFW52ZVMM6WZYNRFUXMR4XJ6C5727BOFMR4RRH33Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.325865978 (571h53m42.196949023s) [seg_pos={0 0}, upload=501384]: Segment reached optimal threshold of 14 pieces. Dec 29 11:22:42.326137074 (571h53m42.197220109s) [noise=true , address=64.49.13.169:4708 , piece_id=B2X2RWRELXWJPF6AVJJZCNU46LM5ASGRE4EWGPKL3SCLQ4VKUBDQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection Dec 29 11:22:42.326153074 (571h53m42.197236109s) [noise=true , address=64.49.13.169:4708 , piece_id=B2X2RWRELXWJPF6AVJJZCNU46LM5ASGRE4EWGPKL3SCLQ4VKUBDQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326193401 (571h53m42.197276446s) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12h5esfg6mw2c8y3AVpeFBkL5FfLAwYdjhcnQ9CJ8fjh61WLYjx): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326205705 (571h53m42.19728874s) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326407739 (571h53m42.197490784s) [noise=true , address=77.90.31.10:4524 , piece_id=Q2JGEUV5EDFTVY2CSNPZOVJIIIAFBA3IHA7S5N7YZRH3E7TSTE5A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uN7CznoK4AD8FqFkNUndawdGJ9rmjAe7oNnjJkV844ScY87qe): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326415374 (571h53m42.197498419s) [noise=true , address=77.90.31.10:4524 , piece_id=Q2JGEUV5EDFTVY2CSNPZOVJIIIAFBA3IHA7S5N7YZRH3E7TSTE5A, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326505105 (571h53m42.19758815s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tnTEYvqJ6PB2DsuXA28sEuEiGj11StA4qdhcYH1kuoaMkTVdY): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326512188 (571h53m42.197595223s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326532357 (571h53m42.197615392s) [seg_pos={0 0}, upload=501384]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 11:22:42.326671562 (571h53m42.197754597s) [upload=501384]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 11:22:42.326677843 (571h53m42.197760879s) [upload=501384]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 11:22:42.610391438 (571h53m42.481474493s) [upload=501384]: Done uploading segments... Dec 29 11:22:42.610416826 (571h53m42.481499871s) [upload=501384]: Upload finished. err: <nil> Duration: 45.354641329s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 29 out of 60 is failed (threshold=3) [XX..XX..X..XXX.X.......X.X..X.XXX...XX.XXX.X.X....XXXXXX..X.] duration: 6.057867736s trace: https://jaeger.dev.storj.io/trace/513f576711610517 FAIL (2025-12-29 11:58:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 800 (572h28m6.999214835s) [upload=501879]: Uploading segments... Dec 29 11:57:07.128698610 (572h28m6.999781655s) [upload=501879]: Got next segment. Inline: false Dec 29 11:57:07.128761249 (572h28m6.999844294s) [upload=501879]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 11:57:07.128767080 (572h28m6.999850115s) [upload=501879]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 11:57:07.372566733 (572h28m7.243649778s) [seg_pos={0 0}, upload=501879]: Begin upload segment... Dec 29 11:57:07.372640152 (572h28m7.243723188s) [seg_pos={0 0}, upload=501879]: Done begin upload segment. Dec 29 11:57:07.372689095 (572h28m7.24377214s) [noise=true , address=145.79.227.5:4221 , piece_id=CPFJ33EJBGQJBLD33PRRNPFO4CY3FISHHKYKHK3FAG6H7ZJD7X7A, seg_pos={0 0}, upload=501879]: Uploading piece... Dec 29 11:57:07.372709935 (572h28m7.24379297s) [noise=true , address=145.79.227.26:4654 , piece_id=SR4BSFUJONKDA5COOUFQERVBUFOIKBIKR7USOESBHONMUAE2V4EQ, seg_pos={0 0}, upload=501879]: Uploading piece... Dec 29 11:57:07.373240525 (572h28m7.24432356s) [noise=true , address=77.90.31.11:4555 , piece_id=RYE7FEXDOKN2XWVM6LCPXHUNIRQG3UDOADQTJ2EVRPCUGFBXDYGA, seg_pos={0 0}, upload=501879]: Uploading piece... Dec 29 11:57:07.373538603 (572h28m7.244621639s) [noise=true , address=77.90.31.13:4627 , piece_id=4KFSDWAF5F7NLREPSJGLKIBQO337IXZ2LCZ6RKH5HF22WITCJ4RA, seg_pos={0 0}, upload=501879]: Uploading piece... Dec 29 11:57:07.373557519 (572h28m7.244640554s) [noise=true , address=145.79.227.4:4185 , piece_id=QCKCVPJWJN7HCHIFR3FCQY2YMLWPCQX6IHFAB3M6UNG3F2MHRAJQ, seg_pos={0 0}, upload=501879]: Uploading piece... Dec 29 11:57:07.373899170 (572h28m7.244982215s) [noise=true , address=145.79.227.17:4972 , piece_id=YKEVNKIYDMEKANB7K6CTOEY5I5CIF6GRX36CH25RTLEPS5TH4ZQA, seg_pos={0 0}, upload=501879]: Uploading piece... Dec 29 11:57:07.374034808 (572h28m7.245117843s) [noise=true , address=145.79.227.45:4231 , piece_id=JRRZ7BNBMSDT4NIHLYF675FQ3BOPWJ3255GPJ4BVKQT6RLLADHIQ, seg_pos={0 0}, upload=501879]: Uploading piece... Dec 29 11:57:07.374251901 (572h28m7.245334936s) [noise=true , address=145.79.227.50:4375 , piece_id=IIBYARTEZ4DJENYXCDBVYJUTG634SBFKBXN55WH7MS44PBWLN67Q, seg_pos={0 0}, upload=501879]: Uploading piece... Dec 29 11:57:07.374318177 (572h28m7.245401213s) [noise=true , address=145.79.227.51:4410 , piece_id=LA7ERJSLBONUJ7DUUVYRRWPH53MQ2HLUC2SABKBIODSFFSQ4X2JQ, seg_pos={0 0}, upload=501879]: Uploading piece... Dec 29 11:57:07.374531033 (572h28m7.245614068s) [noise=true , address=145.79.227.15:4874 , piece_id=IUI7NKWR7DFF6JWVSU2KU2RAWLEGK3BVBQQZWJESEYE3B5D5QJCA, seg_pos={0 0}, upload=501879]: Uploading piece... Dec 29 11:57:07.374586298 (572h28m7.245669333s) [noise=true , address=45.91.116.14:4191 , piece_id=JA6JRFY36WUCCPLOGFNC5NVCPAYUU5LZ4SN4YESPJZF7454PIQIA, seg_pos={0 0}, upload=501879]: Uploading piece... Dec 29 11:57:07.374811959 (572h28m7.245894994s) [noise=true , address=77.90.31.14:4661 , piece_id=YGK5O774HGX4E4MFSKCAMRNOESERQVOT4P6GUZQILKMM5RHUMLMQ, seg_pos={0 0}, upload=501879]: Uploading piece... Dec 29 11:57:07.375074087 (572h28m7.246157122s) [noise=true , address=145.79.227.42:4122 , piece_id=3CGCHILBDC3NVP2A2ZFL32TF3JALJLS3L6SHS7SOK7RSDZ32WG3Q, seg_pos={0 0}, upload=501879]: Uploading piece... Dec 29 11:57:07.375330105 (572h28m7.24641314s) [noise=true , address=145.79.227.47:4267 , piece_id=5ESJZA6WOA7MFH5OA5EKAHFYAN24L6QCU5NUWQZRJJ2B2DDDFJRQ, seg_pos={0 0}, upload=501879]: Uploading piece... Dec 29 11:57:07.375590691 (572h28m7.246673726s) [noise=true , address=145.79.227.10:4401 , piece_id=HXRP4FZNXUX4W2P5U4AEL2BTFQIEHOOAI2DG5HQMZBLEG454DD5A, seg_pos={0 0}, upload=501879]: Uploading piece... Dec 29 11:57:07.375882987 (572h28m7.246966022s) [noise=true , address=45.91.116.7:4011 , piece_id=22QCHII2VIBUNJL25BOZWFSIONACKCP2P67ZOKFAEEEJDOZ3RYYA, seg_pos={0 0}, upload=501879]: Uploading piece... Dec 29 11:57:08.982588455 (572h28m8.8536715s) [upload=501879]: Next returned nil segment Dec 29 11:57:08.982600729 (572h28m8.853683774s) [upload=501879]: Waiting for error group managing segments... Dec 29 11:57:09.361465868 (572h28m9.232548913s) [noise=true , address=145.79.227.45:4231 , piece_id=JRRZ7BNBMSDT4NIHLYF675FQ3BOPWJ3255GPJ4BVKQT6RLLADHIQ, seg_pos={0 0}, upload=501879]: Done uploading piece. err: <nil> Dec 29 11:57:09.362339351 (572h28m9.233422396s) [noise=true , address=145.79.227.10:4401 , piece_id=HXRP4FZNXUX4W2P5U4AEL2BTFQIEHOOAI2DG5HQMZBLEG454DD5A, seg_pos={0 0}, upload=501879]: Done uploading piece. err: <nil> Dec 29 11:57:09.365289920 (572h28m9.236372995s) [noise=true , address=145.79.227.50:4375 , piece_id=IIBYARTEZ4DJENYXCDBVYJUTG634SBFKBXN55WH7MS44PBWLN67Q, seg_pos={0 0}, upload=501879]: Done uploading piece. err: <nil> Dec 29 11:57:09.365970737 (572h28m9.237053782s) [noise=true , address=145.79.227.17:4972 , piece_id=YKEVNKIYDMEKANB7K6CTOEY5I5CIF6GRX36CH25RTLEPS5TH4ZQA, seg_pos={0 0}, upload=501879]: Done uploading piece. err: <nil> Dec 29 11:57:09.368034907 (572h28m9.239117942s) [noise=true , address=145.79.227.26:4654 , piece_id=SR4BSFUJONKDA5COOUFQERVBUFOIKBIKR7USOESBHONMUAE2V4EQ, seg_pos={0 0}, upload=501879]: Done uploading piece. err: <nil> Dec 29 11:57:09.369782424 (572h28m9.240865459s) [noise=true , address=145.79.227.15:4874 , piece_id=IUI7NKWR7DFF6JWVSU2KU2RAWLEGK3BVBQQZWJESEYE3B5D5QJCA, seg_pos={0 0}, upload=501879]: Done uploading piece. err: <nil> Dec 29 11:57:09.371692482 (572h28m9.242775517s) [noise=true , address=145.79.227.47:4267 , piece_id=5ESJZA6WOA7MFH5OA5EKAHFYAN24L6QCU5NUWQZRJJ2B2DDDFJRQ, seg_pos={0 0}, upload=501879]: Done uploading piece. err: <nil> Dec 29 11:57:09.384896074 (572h28m9.255979109s) [noise=true , address=145.79.227.51:4410 , piece_id=LA7ERJSLBONUJ7DUUVYRRWPH53MQ2HLUC2SABKBIODSFFSQ4X2JQ, seg_pos={0 0}, upload=501879]: Done uploading piece. err: <nil> Dec 29 11:57:09.385752005 (572h28m9.25683504s) [noise=true , address=145.79.227.42:4122 , piece_id=3CGCHILBDC3NVP2A2ZFL32TF3JALJLS3L6SHS7SOK7RSDZ32WG3Q, seg_pos={0 0}, upload=501879]: Done uploading piece. err: <nil> Dec 29 11:57:09.387512547 (572h28m9.258595582s) [noise=true , address=145.79.227.5:4221 , piece_id=CPFJ33EJBGQJBLD33PRRNPFO4CY3FISHHKYKHK3FAG6H7ZJD7X7A, seg_pos={0 0}, upload=501879]: Done uploading piece. err: <nil> Dec 29 11:57:09.405882007 (572h28m9.276965052s) [noise=true , address=145.79.227.4:4185 , piece_id=QCKCVPJWJN7HCHIFR3FCQY2YMLWPCQX6IHFAB3M6UNG3F2MHRAJQ, seg_pos={0 0}, upload=501879]: Done uploading piece. err: <nil> Dec 29 11:57:09.641314977 (572h28m9.512398012s) [noise=true , address=45.91.116.14:4191 , piece_id=JA6JRFY36WUCCPLOGFNC5NVCPAYUU5LZ4SN4YESPJZF7454PIQIA, seg_pos={0 0}, upload=501879]: Done uploading piece. err: <nil> Dec 29 11:57:09.650671725 (572h28m9.52175477s) [noise=true , address=77.90.31.11:4555 , piece_id=RYE7FEXDOKN2XWVM6LCPXHUNIRQG3UDOADQTJ2EVRPCUGFBXDYGA, seg_pos={0 0}, upload=501879]: Done uploading piece. err: <nil> Dec 29 11:57:09.650692604 (572h28m9.521775639s) [seg_pos={0 0}, upload=501879]: Segment reached optimal threshold of 13 pieces. Dec 29 11:57:09.650870935 (572h28m9.52195397s) [noise=true , address=45.91.116.7:4011 , piece_id=22QCHII2VIBUNJL25BOZWFSIONACKCP2P67ZOKFAEEEJDOZ3RYYA, seg_pos={0 0}, upload=501879]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 29 11:57:09.650877417 (572h28m9.521960462s) [noise=true , address=45.91.116.7:4011 , piece_id=22QCHII2VIBUNJL25BOZWFSIONACKCP2P67ZOKFAEEEJDOZ3RYYA, seg_pos={0 0}, upload=501879]: Long tail cancellation due to optimal threshold Dec 29 11:57:09.651132001 (572h28m9.522215036s) [noise=true , address=77.90.31.14:4661 , piece_id=YGK5O774HGX4E4MFSKCAMRNOESERQVOT4P6GUZQILKMM5RHUMLMQ, seg_pos={0 0}, upload=501879]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: write tcp 188.166.225.170:59046->77.90.31.14:4661: use of closed network connection; write tcp 188.166.225.170:59046->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59046->77.90.31.14:4661: use of closed network connection Dec 29 11:57:09.651139826 (572h28m9.522222871s) [noise=true , address=77.90.31.14:4661 , piece_id=YGK5O774HGX4E4MFSKCAMRNOESERQVOT4P6GUZQILKMM5RHUMLMQ, seg_pos={0 0}, upload=501879]: Long tail cancellation due to optimal threshold Dec 29 11:57:09.651175023 (572h28m9.522258078s) [noise=true , address=77.90.31.13:4627 , piece_id=4KFSDWAF5F7NLREPSJGLKIBQO337IXZ2LCZ6RKH5HF22WITCJ4RA, seg_pos={0 0}, upload=501879]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:38342->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:38342->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38342->77.90.31.13:4627: use of closed network connection Dec 29 11:57:09.651201173 (572h28m9.522284228s) [noise=true , address=77.90.31.13:4627 , piece_id=4KFSDWAF5F7NLREPSJGLKIBQO337IXZ2LCZ6RKH5HF22WITCJ4RA, seg_pos={0 0}, upload=501879]: Long tail cancellation due to optimal threshold Dec 29 11:57:09.651239816 (572h28m9.522322862s) [seg_pos={0 0}, upload=501879]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 11:57:09.651308948 (572h28m9.522391993s) [upload=501879]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 11:57:09.651314058 (572h28m9.522397093s) [upload=501879]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 11:57:09.946164994 (572h28m9.817248029s) [upload=501879]: Done uploading segments... Dec 29 11:57:09.946186836 (572h28m9.817269881s) [upload=501879]: Upload finished. err: <nil> Duration: 6.05789598s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 30 out of 60 is failed (threshold=6) [...XX..XX......X.X.XX.X.XX..X..XX.XXXX.XXXX...XXX...XXX..XX.] duration: 3.436475739s trace: https://jaeger.dev.storj.io/trace/6f7e362292525c6d FAIL (2025-12-29 11:57:43 UTC) orbiter warning: context deadline exceeded Duration: 3.437079017s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 29 out of 60 is failed (threshold=3) [.XX..XX..X..XXX.X.......X.X..X.XXX...XX.XXX.X.X....XXXXXX..X] duration: 30.002658517s trace: https://jaeger.dev.storj.io/trace/513f576711610517 FAIL (2025-12-29 11:57:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 800 (572h28m6.999214835s) [upload=501879]: Uploading segments... Dec 29 11:57:07.128698610 (572h28m6.999781655s) [upload=501879]: Got next segment. Inline: false Dec 29 11:57:07.128761249 (572h28m6.999844294s) [upload=501879]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 11:57:07.128767080 (572h28m6.999850115s) [upload=501879]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 11:57:07.372566733 (572h28m7.243649778s) [seg_pos={0 0}, upload=501879]: Begin upload segment... Dec 29 11:57:07.372640152 (572h28m7.243723188s) [seg_pos={0 0}, upload=501879]: Done begin upload segment. Dec 29 11:57:07.372689095 (572h28m7.24377214s) [noise=true , address=145.79.227.5:4221 , piece_id=CPFJ33EJBGQJBLD33PRRNPFO4CY3FISHHKYKHK3FAG6H7ZJD7X7A, seg_pos={0 0}, upload=501879]: Uploading piece... Dec 29 11:57:07.372709935 (572h28m7.24379297s) [noise=true , address=145.79.227.26:4654 , piece_id=SR4BSFUJONKDA5COOUFQERVBUFOIKBIKR7USOESBHONMUAE2V4EQ, seg_pos={0 0}, upload=501879]: Uploading piece... Dec 29 11:57:07.373240525 (572h28m7.24432356s) [noise=true , address=77.90.31.11:4555 , piece_id=RYE7FEXDOKN2XWVM6LCPXHUNIRQG3UDOADQTJ2EVRPCUGFBXDYGA, seg_pos={0 0}, upload=501879]: Uploading piece... Dec 29 11:57:07.373538603 (572h28m7.244621639s) [noise=true , address=77.90.31.13:4627 , piece_id=4KFSDWAF5F7NLREPSJGLKIBQO337IXZ2LCZ6RKH5HF22WITCJ4RA, seg_pos={0 0}, upload=501879]: Uploading piece... Dec 29 11:57:07.373557519 (572h28m7.244640554s) [noise=true , address=145.79.227.4:4185 , piece_id=QCKCVPJWJN7HCHIFR3FCQY2YMLWPCQX6IHFAB3M6UNG3F2MHRAJQ, seg_pos={0 0}, upload=501879]: Uploading piece... Dec 29 11:57:07.373899170 (572h28m7.244982215s) [noise=true , address=145.79.227.17:4972 , piece_id=YKEVNKIYDMEKANB7K6CTOEY5I5CIF6GRX36CH25RTLEPS5TH4ZQA, seg_pos={0 0}, upload=501879]: Uploading piece... Dec 29 11:57:07.374034808 (572h28m7.245117843s) [noise=true , address=145.79.227.45:4231 , piece_id=JRRZ7BNBMSDT4NIHLYF675FQ3BOPWJ3255GPJ4BVKQT6RLLADHIQ, seg_pos={0 0}, upload=501879]: Uploading piece... Dec 29 11:57:07.374251901 (572h28m7.245334936s) [noise=true , address=145.79.227.50:4375 , piece_id=IIBYARTEZ4DJENYXCDBVYJUTG634SBFKBXN55WH7MS44PBWLN67Q, seg_pos={0 0}, upload=501879]: Uploading piece... Dec 29 11:57:07.374318177 (572h28m7.245401213s) [noise=true , address=145.79.227.51:4410 , piece_id=LA7ERJSLBONUJ7DUUVYRRWPH53MQ2HLUC2SABKBIODSFFSQ4X2JQ, seg_pos={0 0}, upload=501879]: Uploading piece... Dec 29 11:57:07.374531033 (572h28m7.245614068s) [noise=true , address=145.79.227.15:4874 , piece_id=IUI7NKWR7DFF6JWVSU2KU2RAWLEGK3BVBQQZWJESEYE3B5D5QJCA, seg_pos={0 0}, upload=501879]: Uploading piece... Dec 29 11:57:07.374586298 (572h28m7.245669333s) [noise=true , address=45.91.116.14:4191 , piece_id=JA6JRFY36WUCCPLOGFNC5NVCPAYUU5LZ4SN4YESPJZF7454PIQIA, seg_pos={0 0}, upload=501879]: Uploading piece... Dec 29 11:57:07.374811959 (572h28m7.245894994s) [noise=true , address=77.90.31.14:4661 , piece_id=YGK5O774HGX4E4MFSKCAMRNOESERQVOT4P6GUZQILKMM5RHUMLMQ, seg_pos={0 0}, upload=501879]: Uploading piece... Dec 29 11:57:07.375074087 (572h28m7.246157122s) [noise=true , address=145.79.227.42:4122 , piece_id=3CGCHILBDC3NVP2A2ZFL32TF3JALJLS3L6SHS7SOK7RSDZ32WG3Q, seg_pos={0 0}, upload=501879]: Uploading piece... Dec 29 11:57:07.375330105 (572h28m7.24641314s) [noise=true , address=145.79.227.47:4267 , piece_id=5ESJZA6WOA7MFH5OA5EKAHFYAN24L6QCU5NUWQZRJJ2B2DDDFJRQ, seg_pos={0 0}, upload=501879]: Uploading piece... Dec 29 11:57:07.375590691 (572h28m7.246673726s) [noise=true , address=145.79.227.10:4401 , piece_id=HXRP4FZNXUX4W2P5U4AEL2BTFQIEHOOAI2DG5HQMZBLEG454DD5A, seg_pos={0 0}, upload=501879]: Uploading piece... Dec 29 11:57:07.375882987 (572h28m7.246966022s) [noise=true , address=45.91.116.7:4011 , piece_id=22QCHII2VIBUNJL25BOZWFSIONACKCP2P67ZOKFAEEEJDOZ3RYYA, seg_pos={0 0}, upload=501879]: Uploading piece... Dec 29 11:57:08.982588455 (572h28m8.8536715s) [upload=501879]: Next returned nil segment Dec 29 11:57:08.982600729 (572h28m8.853683774s) [upload=501879]: Waiting for error group managing segments... Dec 29 11:57:09.361465868 (572h28m9.232548913s) [noise=true , address=145.79.227.45:4231 , piece_id=JRRZ7BNBMSDT4NIHLYF675FQ3BOPWJ3255GPJ4BVKQT6RLLADHIQ, seg_pos={0 0}, upload=501879]: Done uploading piece. err: <nil> Dec 29 11:57:09.362339351 (572h28m9.233422396s) [noise=true , address=145.79.227.10:4401 , piece_id=HXRP4FZNXUX4W2P5U4AEL2BTFQIEHOOAI2DG5HQMZBLEG454DD5A, seg_pos={0 0}, upload=501879]: Done uploading piece. err: <nil> Dec 29 11:57:09.365289920 (572h28m9.236372995s) [noise=true , address=145.79.227.50:4375 , piece_id=IIBYARTEZ4DJENYXCDBVYJUTG634SBFKBXN55WH7MS44PBWLN67Q, seg_pos={0 0}, upload=501879]: Done uploading piece. err: <nil> Dec 29 11:57:09.365970737 (572h28m9.237053782s) [noise=true , address=145.79.227.17:4972 , piece_id=YKEVNKIYDMEKANB7K6CTOEY5I5CIF6GRX36CH25RTLEPS5TH4ZQA, seg_pos={0 0}, upload=501879]: Done uploading piece. err: <nil> Dec 29 11:57:09.368034907 (572h28m9.239117942s) [noise=true , address=145.79.227.26:4654 , piece_id=SR4BSFUJONKDA5COOUFQERVBUFOIKBIKR7USOESBHONMUAE2V4EQ, seg_pos={0 0}, upload=501879]: Done uploading piece. err: <nil> Dec 29 11:57:09.369782424 (572h28m9.240865459s) [noise=true , address=145.79.227.15:4874 , piece_id=IUI7NKWR7DFF6JWVSU2KU2RAWLEGK3BVBQQZWJESEYE3B5D5QJCA, seg_pos={0 0}, upload=501879]: Done uploading piece. err: <nil> Dec 29 11:57:09.371692482 (572h28m9.242775517s) [noise=true , address=145.79.227.47:4267 , piece_id=5ESJZA6WOA7MFH5OA5EKAHFYAN24L6QCU5NUWQZRJJ2B2DDDFJRQ, seg_pos={0 0}, upload=501879]: Done uploading piece. err: <nil> Dec 29 11:57:09.384896074 (572h28m9.255979109s) [noise=true , address=145.79.227.51:4410 , piece_id=LA7ERJSLBONUJ7DUUVYRRWPH53MQ2HLUC2SABKBIODSFFSQ4X2JQ, seg_pos={0 0}, upload=501879]: Done uploading piece. err: <nil> Dec 29 11:57:09.385752005 (572h28m9.25683504s) [noise=true , address=145.79.227.42:4122 , piece_id=3CGCHILBDC3NVP2A2ZFL32TF3JALJLS3L6SHS7SOK7RSDZ32WG3Q, seg_pos={0 0}, upload=501879]: Done uploading piece. err: <nil> Dec 29 11:57:09.387512547 (572h28m9.258595582s) [noise=true , address=145.79.227.5:4221 , piece_id=CPFJ33EJBGQJBLD33PRRNPFO4CY3FISHHKYKHK3FAG6H7ZJD7X7A, seg_pos={0 0}, upload=501879]: Done uploading piece. err: <nil> Dec 29 11:57:09.405882007 (572h28m9.276965052s) [noise=true , address=145.79.227.4:4185 , piece_id=QCKCVPJWJN7HCHIFR3FCQY2YMLWPCQX6IHFAB3M6UNG3F2MHRAJQ, seg_pos={0 0}, upload=501879]: Done uploading piece. err: <nil> Dec 29 11:57:09.641314977 (572h28m9.512398012s) [noise=true , address=45.91.116.14:4191 , piece_id=JA6JRFY36WUCCPLOGFNC5NVCPAYUU5LZ4SN4YESPJZF7454PIQIA, seg_pos={0 0}, upload=501879]: Done uploading piece. err: <nil> Dec 29 11:57:09.650671725 (572h28m9.52175477s) [noise=true , address=77.90.31.11:4555 , piece_id=RYE7FEXDOKN2XWVM6LCPXHUNIRQG3UDOADQTJ2EVRPCUGFBXDYGA, seg_pos={0 0}, upload=501879]: Done uploading piece. err: <nil> Dec 29 11:57:09.650692604 (572h28m9.521775639s) [seg_pos={0 0}, upload=501879]: Segment reached optimal threshold of 13 pieces. Dec 29 11:57:09.650870935 (572h28m9.52195397s) [noise=true , address=45.91.116.7:4011 , piece_id=22QCHII2VIBUNJL25BOZWFSIONACKCP2P67ZOKFAEEEJDOZ3RYYA, seg_pos={0 0}, upload=501879]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 29 11:57:09.650877417 (572h28m9.521960462s) [noise=true , address=45.91.116.7:4011 , piece_id=22QCHII2VIBUNJL25BOZWFSIONACKCP2P67ZOKFAEEEJDOZ3RYYA, seg_pos={0 0}, upload=501879]: Long tail cancellation due to optimal threshold Dec 29 11:57:09.651132001 (572h28m9.522215036s) [noise=true , address=77.90.31.14:4661 , piece_id=YGK5O774HGX4E4MFSKCAMRNOESERQVOT4P6GUZQILKMM5RHUMLMQ, seg_pos={0 0}, upload=501879]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: write tcp 188.166.225.170:59046->77.90.31.14:4661: use of closed network connection; write tcp 188.166.225.170:59046->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59046->77.90.31.14:4661: use of closed network connection Dec 29 11:57:09.651139826 (572h28m9.522222871s) [noise=true , address=77.90.31.14:4661 , piece_id=YGK5O774HGX4E4MFSKCAMRNOESERQVOT4P6GUZQILKMM5RHUMLMQ, seg_pos={0 0}, upload=501879]: Long tail cancellation due to optimal threshold Dec 29 11:57:09.651175023 (572h28m9.522258078s) [noise=true , address=77.90.31.13:4627 , piece_id=4KFSDWAF5F7NLREPSJGLKIBQO337IXZ2LCZ6RKH5HF22WITCJ4RA, seg_pos={0 0}, upload=501879]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:38342->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:38342->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38342->77.90.31.13:4627: use of closed network connection Dec 29 11:57:09.651201173 (572h28m9.522284228s) [noise=true , address=77.90.31.13:4627 , piece_id=4KFSDWAF5F7NLREPSJGLKIBQO337IXZ2LCZ6RKH5HF22WITCJ4RA, seg_pos={0 0}, upload=501879]: Long tail cancellation due to optimal threshold Dec 29 11:57:09.651239816 (572h28m9.522322862s) [seg_pos={0 0}, upload=501879]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 11:57:09.651308948 (572h28m9.522391993s) [upload=501879]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 11:57:09.651314058 (572h28m9.522397093s) [upload=501879]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 11:57:09.946164994 (572h28m9.817248029s) [upload=501879]: Done uploading segments... Dec 29 11:57:09.946186836 (572h28m9.817269881s) [upload=501879]: Upload finished. err: <nil> Duration: 30.002682404s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [X...............X..X........................................] duration: 41.804268457s trace: https://jaeger.dev.storj.io/trace/7e7354d8023608a8 FAIL (2025-12-29 11:57:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== ) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480097317 (571h53m39.351180352s) [noise=true , address=77.90.31.21:4889 , piece_id=BR6Z55P3A7YJ5FJRDSZFUIF66FGIJQY7WCEQ7PL5DOYWL73OMKTQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480405053 (571h53m39.351488098s) [noise=true , address=77.90.31.16:4750 , piece_id=MVVZPJEGDCBSNVOB6CD2WUZGIJTL3VBHX2G4L3TVZ6JF5ZEJ5U6Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480712969 (571h53m39.351796014s) [noise=true , address=77.90.31.18:4834 , piece_id=BTWNGNQVEUEVV6HLBDM63SGNTKV4RDU4J4XFLG52WHKP53H3L5KA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481019453 (571h53m39.352102488s) [noise=true , address=145.79.227.30:4777 , piece_id=VSCXLXGIBSFE6E4OHLWNZIZ75PO7WQO2VFHUYHW7JUNH62KGGU7Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481029532 (571h53m39.352112567s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481559832 (571h53m39.352642877s) [noise=true , address=145.79.227.35:4515 , piece_id=L7LFZKZEZERUNX4HUNDNQ2PWHXNJRJ7OBFRNMFIICPZEUIBHGP7A, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481731919 (571h53m39.352814954s) [noise=true , address=64.49.13.26:4953 , piece_id=4YSW5Y3A6HYG3XJ2XAB7RGVHAOHMP7G3QIZYMYTSXVXYJ3ANLNXA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482078329 (571h53m39.353161364s) [noise=true , address=145.79.227.44:4205 , piece_id=UE4QSHDQBZ3XBWQ6I6MCRVTLMU7U6W6OZDA2ZGQCH6ZYRAGPD7WA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482099830 (571h53m39.353182865s) [noise=true , address=145.79.227.40:4067 , piece_id=WSYDRX5YJ6OCFW52ZVMM6WZYNRFUXMR4XJ6C5727BOFMR4RRH33Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482434487 (571h53m39.353517532s) [noise=true , address=145.79.227.3:4164 , piece_id=U6F7VDPOH7K6R5EH5UUXHFP55GXOEXO5BLBJHZRTITPDQI4LP7KQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482480845 (571h53m39.35356388s) [noise=true , address=64.49.13.140:4516 , piece_id=4GUYOYOAEASUK2W25S3NILWIIIHGOZX76L4EGFLUUUEDWRALPP2A, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482841442 (571h53m39.353924477s) [noise=true , address=145.79.227.34:4910 , piece_id=ZH4DCNXET2Q6VRAUH3NQ3KKSV4OYI575X7YS72XD236EEOMOG3LA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483163645 (571h53m39.35424668s) [noise=true , address=145.79.227.29:4974 , piece_id=KEKREOD2B35XQNI45JBHE2SZZC3MQQBQUR5I6WFDAQYJQGJBXSMA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483532478 (571h53m39.354615513s) [noise=true , address=77.90.31.17:4769 , piece_id=3MKUMOQIUVERKQLLIFX52XQPTQ5LSFMHIPD5YW2YTO5YEFR36IYA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483914596 (571h53m39.354997631s) [noise=true , address=145.79.227.13:4807 , piece_id=VKXB5MWAG2YYH2AWWJEGTLBZ2QUPLI25VTHI6AMMALSM23Z3AWZQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:40.761721106 (571h53m40.632804141s) [upload=501384]: Next returned nil segment Dec 29 11:22:40.761733349 (571h53m40.632816384s) [upload=501384]: Waiting for error group managing segments... Dec 29 11:22:41.121222862 (571h53m40.992305897s) [noise=true , address=64.49.13.140:4516 , piece_id=4GUYOYOAEASUK2W25S3NILWIIIHGOZX76L4EGFLUUUEDWRALPP2A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.197311646 (571h53m41.068394691s) [noise=true , address=64.49.13.26:4953 , piece_id=4YSW5Y3A6HYG3XJ2XAB7RGVHAOHMP7G3QIZYMYTSXVXYJ3ANLNXA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.347247568 (571h53m41.218330613s) [noise=true , address=145.79.227.13:4807 , piece_id=VKXB5MWAG2YYH2AWWJEGTLBZ2QUPLI25VTHI6AMMALSM23Z3AWZQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.347917313 (571h53m41.219000348s) [noise=true , address=145.79.227.3:4164 , piece_id=U6F7VDPOH7K6R5EH5UUXHFP55GXOEXO5BLBJHZRTITPDQI4LP7KQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.348510613 (571h53m41.219593648s) [noise=true , address=145.79.227.29:4974 , piece_id=KEKREOD2B35XQNI45JBHE2SZZC3MQQBQUR5I6WFDAQYJQGJBXSMA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.503950985 (571h53m41.37503402s) [noise=true , address=77.90.31.17:4769 , piece_id=3MKUMOQIUVERKQLLIFX52XQPTQ5LSFMHIPD5YW2YTO5YEFR36IYA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.561352025 (571h53m41.43243506s) [noise=true , address=77.90.31.18:4834 , piece_id=BTWNGNQVEUEVV6HLBDM63SGNTKV4RDU4J4XFLG52WHKP53H3L5KA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.622447380 (571h53m41.493530425s) [noise=true , address=77.90.31.16:4750 , piece_id=MVVZPJEGDCBSNVOB6CD2WUZGIJTL3VBHX2G4L3TVZ6JF5ZEJ5U6Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.645494685 (571h53m41.51657773s) [noise=true , address=77.90.31.21:4889 , piece_id=BR6Z55P3A7YJ5FJRDSZFUIF66FGIJQY7WCEQ7PL5DOYWL73OMKTQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.247930045 (571h53m42.11901308s) [noise=true , address=145.79.227.30:4777 , piece_id=VSCXLXGIBSFE6E4OHLWNZIZ75PO7WQO2VFHUYHW7JUNH62KGGU7Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.262221890 (571h53m42.133304925s) [noise=true , address=145.79.227.44:4205 , piece_id=UE4QSHDQBZ3XBWQ6I6MCRVTLMU7U6W6OZDA2ZGQCH6ZYRAGPD7WA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.295598123 (571h53m42.166681158s) [noise=true , address=145.79.227.35:4515 , piece_id=L7LFZKZEZERUNX4HUNDNQ2PWHXNJRJ7OBFRNMFIICPZEUIBHGP7A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.307481351 (571h53m42.178564386s) [noise=true , address=145.79.227.34:4910 , piece_id=ZH4DCNXET2Q6VRAUH3NQ3KKSV4OYI575X7YS72XD236EEOMOG3LA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.325775906 (571h53m42.196858941s) [noise=true , address=145.79.227.40:4067 , piece_id=WSYDRX5YJ6OCFW52ZVMM6WZYNRFUXMR4XJ6C5727BOFMR4RRH33Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.325865978 (571h53m42.196949023s) [seg_pos={0 0}, upload=501384]: Segment reached optimal threshold of 14 pieces. Dec 29 11:22:42.326137074 (571h53m42.197220109s) [noise=true , address=64.49.13.169:4708 , piece_id=B2X2RWRELXWJPF6AVJJZCNU46LM5ASGRE4EWGPKL3SCLQ4VKUBDQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection Dec 29 11:22:42.326153074 (571h53m42.197236109s) [noise=true , address=64.49.13.169:4708 , piece_id=B2X2RWRELXWJPF6AVJJZCNU46LM5ASGRE4EWGPKL3SCLQ4VKUBDQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326193401 (571h53m42.197276446s) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12h5esfg6mw2c8y3AVpeFBkL5FfLAwYdjhcnQ9CJ8fjh61WLYjx): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326205705 (571h53m42.19728874s) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326407739 (571h53m42.197490784s) [noise=true , address=77.90.31.10:4524 , piece_id=Q2JGEUV5EDFTVY2CSNPZOVJIIIAFBA3IHA7S5N7YZRH3E7TSTE5A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uN7CznoK4AD8FqFkNUndawdGJ9rmjAe7oNnjJkV844ScY87qe): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326415374 (571h53m42.197498419s) [noise=true , address=77.90.31.10:4524 , piece_id=Q2JGEUV5EDFTVY2CSNPZOVJIIIAFBA3IHA7S5N7YZRH3E7TSTE5A, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326505105 (571h53m42.19758815s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tnTEYvqJ6PB2DsuXA28sEuEiGj11StA4qdhcYH1kuoaMkTVdY): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326512188 (571h53m42.197595223s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326532357 (571h53m42.197615392s) [seg_pos={0 0}, upload=501384]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 11:22:42.326671562 (571h53m42.197754597s) [upload=501384]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 11:22:42.326677843 (571h53m42.197760879s) [upload=501384]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 11:22:42.610391438 (571h53m42.481474493s) [upload=501384]: Done uploading segments... Dec 29 11:22:42.610416826 (571h53m42.481499871s) [upload=501384]: Upload finished. err: <nil> Duration: 41.804293183s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 31 out of 60 is failed (threshold=6) [X...XX..XX......X.X.XX.X.XX..X..XX.XXXX.XXXX...XXX...XXX..XX] duration: 5.001510464s trace: https://jaeger.dev.storj.io/trace/6f7e362292525c6d FAIL (2025-12-29 11:57:00 UTC) orbiter warning: context deadline exceeded Duration: 5.001523267s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 29 out of 60 is failed (threshold=3) [X.XX..XX..X..XXX.X.......X.X..X.XXX...XX.XXX.X.X....XXXXXX..] duration: 5.663698886s trace: https://jaeger.dev.storj.io/trace/7fce40104016cac9 FAIL (2025-12-29 11:56:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: S3VTHYQBEDUEOQKOPZ5FCL34OYGMDJKJ7VXCVXTJPYOZY2QIAAPA) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: SJBQMXH4AJGJJMX64XVKZS6D4GFGR4TLAKF3UCMIDDQCVYXOSAJQ) context deadline exceeded ======LOGS====== ec 29 11:54:12.183378237 (572h25m12.054461282s) [upload=501838]: Starting upload Dec 29 11:54:12.183434364 (572h25m12.054517399s) [upload=501838]: Done starting upload Dec 29 11:54:12.183616762 (572h25m12.054699807s) [upload=501838]: Uploading segments... Dec 29 11:54:12.183672798 (572h25m12.054755843s) [upload=501838]: Got next segment. Inline: false Dec 29 11:54:12.183700170 (572h25m12.054783215s) [upload=501838]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 11:54:12.183704618 (572h25m12.054787654s) [upload=501838]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 11:54:12.429960929 (572h25m12.301043964s) [seg_pos={0 0}, upload=501838]: Begin upload segment... Dec 29 11:54:12.429999843 (572h25m12.301082888s) [seg_pos={0 0}, upload=501838]: Done begin upload segment. Dec 29 11:54:12.430024149 (572h25m12.301107184s) [noise=true , address=145.79.227.41:4087 , piece_id=5AUNCE5E6BLUDKLK57BMI7F6EAV6VJ7MT3MNWS7ERTPPVNPATPEA, seg_pos={0 0}, upload=501838]: Uploading piece... Dec 29 11:54:12.430057042 (572h25m12.301140077s) [noise=true , address=45.91.116.14:4190 , piece_id=ZMWNVW4LQFA5FOQUZOWR73F62RTDZCALYWJP5JDOFBL54KO7SAZQ, seg_pos={0 0}, upload=501838]: Uploading piece... Dec 29 11:54:12.430416957 (572h25m12.301500002s) [noise=true , address=145.79.227.3:4149 , piece_id=UWY7EWF2FPD4DXIU6XPKK7KKM5AZKWEXZEWZR5YD7PFA37BMN7HQ, seg_pos={0 0}, upload=501838]: Uploading piece... Dec 29 11:54:12.430690799 (572h25m12.301773844s) [noise=true , address=145.79.227.28:4726 , piece_id=PHSWWSLPTVNIQF53QA6F2PD5R5PNINLN7NIOX7K4XC5FDNFTGHXA, seg_pos={0 0}, upload=501838]: Uploading piece... Dec 29 11:54:12.430962045 (572h25m12.30204509s) [noise=true , address=45.91.116.10:4083 , piece_id=S3VTHYQBEDUEOQKOPZ5FCL34OYGMDJKJ7VXCVXTJPYOZY2QIAAPA, seg_pos={0 0}, upload=501838]: Uploading piece... Dec 29 11:54:12.431212773 (572h25m12.302295808s) [noise=true , address=145.79.227.2:4113 , piece_id=X5V3AHPH7BYYSSOU7NFXUOEIV4EOHTZEY3NUV5CKEQ6NO3V7T2ZQ, seg_pos={0 0}, upload=501838]: Uploading piece... Dec 29 11:54:12.431467138 (572h25m12.302550183s) [noise=true , address=45.91.116.13:4155 , piece_id=SJBQMXH4AJGJJMX64XVKZS6D4GFGR4TLAKF3UCMIDDQCVYXOSAJQ, seg_pos={0 0}, upload=501838]: Uploading piece... Dec 29 11:54:12.431739235 (572h25m12.30282227s) [noise=true , address=145.79.227.35:4482 , piece_id=MR3WEBRCUYTNXHI67Q6KPYANVDHEOBZQGRJ4REJWHCGLU5RPWTUA, seg_pos={0 0}, upload=501838]: Uploading piece... Dec 29 11:54:12.432084863 (572h25m12.303167898s) [noise=true , address=145.79.227.14:4838 , piece_id=6WAL32VWR7GARCLAE6OVTPDF7RLC2ESQSMPFMAGC2PZ3SZMRNJGA, seg_pos={0 0}, upload=501838]: Uploading piece... Dec 29 11:54:12.433595931 (572h25m12.304678976s) [noise=true , address=45.91.116.6:4974 , piece_id=HOUKX5QELY46CMRL7235KVSUBHWZDS4YBCSLHCQSAZTR7X2TI2PA, seg_pos={0 0}, upload=501838]: Uploading piece... Dec 29 11:54:12.433697615 (572h25m12.30478065s) [noise=true , address=145.79.227.18:4007 , piece_id=HNWZSCIZAJ4VXOA3CFZWIHRXOQ5LMDG4RHIWHVI65KBYA5JVUX3Q, seg_pos={0 0}, upload=501838]: Uploading piece... Dec 29 11:54:12.434811366 (572h25m12.305894411s) [noise=true , address=145.79.227.12:4767 , piece_id=ADE5EYRHL55MQHDXUA54OLLWEBUR4CEJ7CHPPAC3ZWL342UZAGIA, seg_pos={0 0}, upload=501838]: Uploading piece... Dec 29 11:54:12.435948301 (572h25m12.307031346s) [noise=true , address=145.79.227.37:4565 , piece_id=MH2V4AX6AOOMKHONMKEUAHADFWJPMUOXXVRVRFEWLCCLRHQ3EFHA, seg_pos={0 0}, upload=501838]: Uploading piece... Dec 29 11:54:12.437024890 (572h25m12.308107925s) [noise=true , address=145.79.227.26:4654 , piece_id=GCJLKD2VIAXJYHH3T6FCMMIALNRYX4PQB6E73PM33KJKSXIKCGYA, seg_pos={0 0}, upload=501838]: Uploading piece... Dec 29 11:54:12.438235165 (572h25m12.3093182s) [noise=true , address=45.91.116.11:4120 , piece_id=GTCROTJ3COLZPTIH6OY6SER2PNCMZQNAKIGA46PYCCHFCLWGG5UQ, seg_pos={0 0}, upload=501838]: Uploading piece... Dec 29 11:54:12.439556451 (572h25m12.310639496s) [noise=true , address=77.90.31.10:4519 , piece_id=AYAQKRRH736HVTI4CYH6UJILIPMDT5RYEH7NORLWPOJHR4A5CYPA, seg_pos={0 0}, upload=501838]: Uploading piece... Dec 29 11:54:14.136140791 (572h25m14.007223826s) [upload=501838]: Next returned nil segment Dec 29 11:54:14.136153355 (572h25m14.00723639s) [upload=501838]: Waiting for error group managing segments... Dec 29 11:54:14.419035379 (572h25m14.290118414s) [noise=true , address=145.79.227.3:4149 , piece_id=UWY7EWF2FPD4DXIU6XPKK7KKM5AZKWEXZEWZR5YD7PFA37BMN7HQ, seg_pos={0 0}, upload=501838]: Done uploading piece. err: <nil> Dec 29 11:54:14.419338256 (572h25m14.290421301s) [noise=true , address=145.79.227.2:4113 , piece_id=X5V3AHPH7BYYSSOU7NFXUOEIV4EOHTZEY3NUV5CKEQ6NO3V7T2ZQ, seg_pos={0 0}, upload=501838]: Done uploading piece. err: <nil> Dec 29 11:54:14.427199976 (572h25m14.298283011s) [noise=true , address=145.79.227.12:4767 , piece_id=ADE5EYRHL55MQHDXUA54OLLWEBUR4CEJ7CHPPAC3ZWL342UZAGIA, seg_pos={0 0}, upload=501838]: Done uploading piece. err: <nil> Dec 29 11:54:14.438348766 (572h25m14.309431801s) [noise=true , address=145.79.227.14:4838 , piece_id=6WAL32VWR7GARCLAE6OVTPDF7RLC2ESQSMPFMAGC2PZ3SZMRNJGA, seg_pos={0 0}, upload=501838]: Done uploading piece. err: <nil> Dec 29 11:54:14.439031396 (572h25m14.310114432s) [noise=true , address=145.79.227.28:4726 , piece_id=PHSWWSLPTVNIQF53QA6F2PD5R5PNINLN7NIOX7K4XC5FDNFTGHXA, seg_pos={0 0}, upload=501838]: Done uploading piece. err: <nil> Dec 29 11:54:14.439411270 (572h25m14.310494315s) [noise=true , address=145.79.227.41:4087 , piece_id=5AUNCE5E6BLUDKLK57BMI7F6EAV6VJ7MT3MNWS7ERTPPVNPATPEA, seg_pos={0 0}, upload=501838]: Done uploading piece. err: <nil> Dec 29 11:54:14.444124194 (572h25m14.315207229s) [noise=true , address=145.79.227.18:4007 , piece_id=HNWZSCIZAJ4VXOA3CFZWIHRXOQ5LMDG4RHIWHVI65KBYA5JVUX3Q, seg_pos={0 0}, upload=501838]: Done uploading piece. err: <nil> Dec 29 11:54:14.452632195 (572h25m14.32371523s) [noise=true , address=145.79.227.26:4654 , piece_id=GCJLKD2VIAXJYHH3T6FCMMIALNRYX4PQB6E73PM33KJKSXIKCGYA, seg_pos={0 0}, upload=501838]: Done uploading piece. err: <nil> Dec 29 11:54:14.460803867 (572h25m14.331886912s) [noise=true , address=145.79.227.37:4565 , piece_id=MH2V4AX6AOOMKHONMKEUAHADFWJPMUOXXVRVRFEWLCCLRHQ3EFHA, seg_pos={0 0}, upload=501838]: Done uploading piece. err: <nil> Dec 29 11:54:14.700913472 (572h25m14.571996507s) [noise=true , address=45.91.116.6:4974 , piece_id=HOUKX5QELY46CMRL7235KVSUBHWZDS4YBCSLHCQSAZTR7X2TI2PA, seg_pos={0 0}, upload=501838]: Done uploading piece. err: <nil> Dec 29 11:54:14.756984224 (572h25m14.628067259s) [noise=true , address=45.91.116.10:4083 , piece_id=S3VTHYQBEDUEOQKOPZ5FCL34OYGMDJKJ7VXCVXTJPYOZY2QIAAPA, seg_pos={0 0}, upload=501838]: Done uploading piece. err: <nil> Dec 29 11:54:14.760267275 (572h25m14.63135032s) [noise=true , address=45.91.116.14:4190 , piece_id=ZMWNVW4LQFA5FOQUZOWR73F62RTDZCALYWJP5JDOFBL54KO7SAZQ, seg_pos={0 0}, upload=501838]: Done uploading piece. err: <nil> Dec 29 11:54:15.074515317 (572h25m14.945598352s) [noise=true , address=45.91.116.13:4155 , piece_id=SJBQMXH4AJGJJMX64XVKZS6D4GFGR4TLAKF3UCMIDDQCVYXOSAJQ, seg_pos={0 0}, upload=501838]: Done uploading piece. err: <nil> Dec 29 11:54:15.074540726 (572h25m14.945623761s) [seg_pos={0 0}, upload=501838]: Segment reached optimal threshold of 13 pieces. Dec 29 11:54:15.074774802 (572h25m14.945857847s) [noise=true , address=77.90.31.10:4519 , piece_id=AYAQKRRH736HVTI4CYH6UJILIPMDT5RYEH7NORLWPOJHR4A5CYPA, seg_pos={0 0}, upload=501838]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Dec 29 11:54:15.074784831 (572h25m14.945867876s) [noise=true , address=77.90.31.10:4519 , piece_id=AYAQKRRH736HVTI4CYH6UJILIPMDT5RYEH7NORLWPOJHR4A5CYPA, seg_pos={0 0}, upload=501838]: Long tail cancellation due to optimal threshold Dec 29 11:54:15.074792485 (572h25m14.94587553s) [noise=true , address=145.79.227.35:4482 , piece_id=MR3WEBRCUYTNXHI67Q6KPYANVDHEOBZQGRJ4REJWHCGLU5RPWTUA, seg_pos={0 0}, upload=501838]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: expected piece hash; context canceled; EOF Dec 29 11:54:15.074817082 (572h25m14.945900127s) [noise=true , address=145.79.227.35:4482 , piece_id=MR3WEBRCUYTNXHI67Q6KPYANVDHEOBZQGRJ4REJWHCGLU5RPWTUA, seg_pos={0 0}, upload=501838]: Long tail cancellation due to optimal threshold Dec 29 11:54:15.074949394 (572h25m14.946032429s) [noise=true , address=45.91.116.11:4120 , piece_id=GTCROTJ3COLZPTIH6OY6SER2PNCMZQNAKIGA46PYCCHFCLWGG5UQ, seg_pos={0 0}, upload=501838]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 188.166.225.170:58174->45.91.116.11:4120: use of closed network connection; write tcp 188.166.225.170:58174->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58174->45.91.116.11:4120: use of closed network connection Dec 29 11:54:15.074958461 (572h25m14.946041496s) [noise=true , address=45.91.116.11:4120 , piece_id=GTCROTJ3COLZPTIH6OY6SER2PNCMZQNAKIGA46PYCCHFCLWGG5UQ, seg_pos={0 0}, upload=501838]: Long tail cancellation due to optimal threshold Dec 29 11:54:15.074992276 (572h25m14.946075311s) [seg_pos={0 0}, upload=501838]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 11:54:15.075079723 (572h25m14.946162768s) [upload=501838]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 11:54:15.075086215 (572h25m14.94616926s) [upload=501838]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 11:54:15.375404280 (572h25m15.246487315s) [upload=501838]: Done uploading segments... Dec 29 11:54:15.375423086 (572h25m15.246506131s) [upload=501838]: Upload finished. err: <nil> Duration: 5.663726318s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 30 out of 60 is failed (threshold=6) [.X...XX..XX......X.X.XX.X.XX..X..XX.XXXX.XXXX...XXX...XXX..X] duration: 5.00178212s trace: https://jaeger.dev.storj.io/trace/2877e14add1ed605 FAIL (2025-12-29 11:56:18 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001803399s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.X...............X..X.......................................] duration: 35.910847738s trace: https://jaeger.dev.storj.io/trace/7e7354d8023608a8 FAIL (2025-12-29 11:56:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== ) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480097317 (571h53m39.351180352s) [noise=true , address=77.90.31.21:4889 , piece_id=BR6Z55P3A7YJ5FJRDSZFUIF66FGIJQY7WCEQ7PL5DOYWL73OMKTQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480405053 (571h53m39.351488098s) [noise=true , address=77.90.31.16:4750 , piece_id=MVVZPJEGDCBSNVOB6CD2WUZGIJTL3VBHX2G4L3TVZ6JF5ZEJ5U6Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480712969 (571h53m39.351796014s) [noise=true , address=77.90.31.18:4834 , piece_id=BTWNGNQVEUEVV6HLBDM63SGNTKV4RDU4J4XFLG52WHKP53H3L5KA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481019453 (571h53m39.352102488s) [noise=true , address=145.79.227.30:4777 , piece_id=VSCXLXGIBSFE6E4OHLWNZIZ75PO7WQO2VFHUYHW7JUNH62KGGU7Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481029532 (571h53m39.352112567s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481559832 (571h53m39.352642877s) [noise=true , address=145.79.227.35:4515 , piece_id=L7LFZKZEZERUNX4HUNDNQ2PWHXNJRJ7OBFRNMFIICPZEUIBHGP7A, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481731919 (571h53m39.352814954s) [noise=true , address=64.49.13.26:4953 , piece_id=4YSW5Y3A6HYG3XJ2XAB7RGVHAOHMP7G3QIZYMYTSXVXYJ3ANLNXA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482078329 (571h53m39.353161364s) [noise=true , address=145.79.227.44:4205 , piece_id=UE4QSHDQBZ3XBWQ6I6MCRVTLMU7U6W6OZDA2ZGQCH6ZYRAGPD7WA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482099830 (571h53m39.353182865s) [noise=true , address=145.79.227.40:4067 , piece_id=WSYDRX5YJ6OCFW52ZVMM6WZYNRFUXMR4XJ6C5727BOFMR4RRH33Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482434487 (571h53m39.353517532s) [noise=true , address=145.79.227.3:4164 , piece_id=U6F7VDPOH7K6R5EH5UUXHFP55GXOEXO5BLBJHZRTITPDQI4LP7KQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482480845 (571h53m39.35356388s) [noise=true , address=64.49.13.140:4516 , piece_id=4GUYOYOAEASUK2W25S3NILWIIIHGOZX76L4EGFLUUUEDWRALPP2A, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482841442 (571h53m39.353924477s) [noise=true , address=145.79.227.34:4910 , piece_id=ZH4DCNXET2Q6VRAUH3NQ3KKSV4OYI575X7YS72XD236EEOMOG3LA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483163645 (571h53m39.35424668s) [noise=true , address=145.79.227.29:4974 , piece_id=KEKREOD2B35XQNI45JBHE2SZZC3MQQBQUR5I6WFDAQYJQGJBXSMA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483532478 (571h53m39.354615513s) [noise=true , address=77.90.31.17:4769 , piece_id=3MKUMOQIUVERKQLLIFX52XQPTQ5LSFMHIPD5YW2YTO5YEFR36IYA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483914596 (571h53m39.354997631s) [noise=true , address=145.79.227.13:4807 , piece_id=VKXB5MWAG2YYH2AWWJEGTLBZ2QUPLI25VTHI6AMMALSM23Z3AWZQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:40.761721106 (571h53m40.632804141s) [upload=501384]: Next returned nil segment Dec 29 11:22:40.761733349 (571h53m40.632816384s) [upload=501384]: Waiting for error group managing segments... Dec 29 11:22:41.121222862 (571h53m40.992305897s) [noise=true , address=64.49.13.140:4516 , piece_id=4GUYOYOAEASUK2W25S3NILWIIIHGOZX76L4EGFLUUUEDWRALPP2A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.197311646 (571h53m41.068394691s) [noise=true , address=64.49.13.26:4953 , piece_id=4YSW5Y3A6HYG3XJ2XAB7RGVHAOHMP7G3QIZYMYTSXVXYJ3ANLNXA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.347247568 (571h53m41.218330613s) [noise=true , address=145.79.227.13:4807 , piece_id=VKXB5MWAG2YYH2AWWJEGTLBZ2QUPLI25VTHI6AMMALSM23Z3AWZQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.347917313 (571h53m41.219000348s) [noise=true , address=145.79.227.3:4164 , piece_id=U6F7VDPOH7K6R5EH5UUXHFP55GXOEXO5BLBJHZRTITPDQI4LP7KQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.348510613 (571h53m41.219593648s) [noise=true , address=145.79.227.29:4974 , piece_id=KEKREOD2B35XQNI45JBHE2SZZC3MQQBQUR5I6WFDAQYJQGJBXSMA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.503950985 (571h53m41.37503402s) [noise=true , address=77.90.31.17:4769 , piece_id=3MKUMOQIUVERKQLLIFX52XQPTQ5LSFMHIPD5YW2YTO5YEFR36IYA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.561352025 (571h53m41.43243506s) [noise=true , address=77.90.31.18:4834 , piece_id=BTWNGNQVEUEVV6HLBDM63SGNTKV4RDU4J4XFLG52WHKP53H3L5KA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.622447380 (571h53m41.493530425s) [noise=true , address=77.90.31.16:4750 , piece_id=MVVZPJEGDCBSNVOB6CD2WUZGIJTL3VBHX2G4L3TVZ6JF5ZEJ5U6Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.645494685 (571h53m41.51657773s) [noise=true , address=77.90.31.21:4889 , piece_id=BR6Z55P3A7YJ5FJRDSZFUIF66FGIJQY7WCEQ7PL5DOYWL73OMKTQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.247930045 (571h53m42.11901308s) [noise=true , address=145.79.227.30:4777 , piece_id=VSCXLXGIBSFE6E4OHLWNZIZ75PO7WQO2VFHUYHW7JUNH62KGGU7Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.262221890 (571h53m42.133304925s) [noise=true , address=145.79.227.44:4205 , piece_id=UE4QSHDQBZ3XBWQ6I6MCRVTLMU7U6W6OZDA2ZGQCH6ZYRAGPD7WA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.295598123 (571h53m42.166681158s) [noise=true , address=145.79.227.35:4515 , piece_id=L7LFZKZEZERUNX4HUNDNQ2PWHXNJRJ7OBFRNMFIICPZEUIBHGP7A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.307481351 (571h53m42.178564386s) [noise=true , address=145.79.227.34:4910 , piece_id=ZH4DCNXET2Q6VRAUH3NQ3KKSV4OYI575X7YS72XD236EEOMOG3LA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.325775906 (571h53m42.196858941s) [noise=true , address=145.79.227.40:4067 , piece_id=WSYDRX5YJ6OCFW52ZVMM6WZYNRFUXMR4XJ6C5727BOFMR4RRH33Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.325865978 (571h53m42.196949023s) [seg_pos={0 0}, upload=501384]: Segment reached optimal threshold of 14 pieces. Dec 29 11:22:42.326137074 (571h53m42.197220109s) [noise=true , address=64.49.13.169:4708 , piece_id=B2X2RWRELXWJPF6AVJJZCNU46LM5ASGRE4EWGPKL3SCLQ4VKUBDQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection Dec 29 11:22:42.326153074 (571h53m42.197236109s) [noise=true , address=64.49.13.169:4708 , piece_id=B2X2RWRELXWJPF6AVJJZCNU46LM5ASGRE4EWGPKL3SCLQ4VKUBDQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326193401 (571h53m42.197276446s) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12h5esfg6mw2c8y3AVpeFBkL5FfLAwYdjhcnQ9CJ8fjh61WLYjx): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326205705 (571h53m42.19728874s) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326407739 (571h53m42.197490784s) [noise=true , address=77.90.31.10:4524 , piece_id=Q2JGEUV5EDFTVY2CSNPZOVJIIIAFBA3IHA7S5N7YZRH3E7TSTE5A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uN7CznoK4AD8FqFkNUndawdGJ9rmjAe7oNnjJkV844ScY87qe): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326415374 (571h53m42.197498419s) [noise=true , address=77.90.31.10:4524 , piece_id=Q2JGEUV5EDFTVY2CSNPZOVJIIIAFBA3IHA7S5N7YZRH3E7TSTE5A, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326505105 (571h53m42.19758815s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tnTEYvqJ6PB2DsuXA28sEuEiGj11StA4qdhcYH1kuoaMkTVdY): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326512188 (571h53m42.197595223s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326532357 (571h53m42.197615392s) [seg_pos={0 0}, upload=501384]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 11:22:42.326671562 (571h53m42.197754597s) [upload=501384]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 11:22:42.326677843 (571h53m42.197760879s) [upload=501384]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 11:22:42.610391438 (571h53m42.481474493s) [upload=501384]: Done uploading segments... Dec 29 11:22:42.610416826 (571h53m42.481499871s) [upload=501384]: Upload finished. err: <nil> Duration: 35.910869119s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [..X...............X..X......................................] duration: 29.039745758s trace: https://jaeger.dev.storj.io/trace/7e7354d8023608a8 FAIL (2025-12-29 11:55:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== ) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480097317 (571h53m39.351180352s) [noise=true , address=77.90.31.21:4889 , piece_id=BR6Z55P3A7YJ5FJRDSZFUIF66FGIJQY7WCEQ7PL5DOYWL73OMKTQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480405053 (571h53m39.351488098s) [noise=true , address=77.90.31.16:4750 , piece_id=MVVZPJEGDCBSNVOB6CD2WUZGIJTL3VBHX2G4L3TVZ6JF5ZEJ5U6Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480712969 (571h53m39.351796014s) [noise=true , address=77.90.31.18:4834 , piece_id=BTWNGNQVEUEVV6HLBDM63SGNTKV4RDU4J4XFLG52WHKP53H3L5KA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481019453 (571h53m39.352102488s) [noise=true , address=145.79.227.30:4777 , piece_id=VSCXLXGIBSFE6E4OHLWNZIZ75PO7WQO2VFHUYHW7JUNH62KGGU7Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481029532 (571h53m39.352112567s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481559832 (571h53m39.352642877s) [noise=true , address=145.79.227.35:4515 , piece_id=L7LFZKZEZERUNX4HUNDNQ2PWHXNJRJ7OBFRNMFIICPZEUIBHGP7A, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481731919 (571h53m39.352814954s) [noise=true , address=64.49.13.26:4953 , piece_id=4YSW5Y3A6HYG3XJ2XAB7RGVHAOHMP7G3QIZYMYTSXVXYJ3ANLNXA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482078329 (571h53m39.353161364s) [noise=true , address=145.79.227.44:4205 , piece_id=UE4QSHDQBZ3XBWQ6I6MCRVTLMU7U6W6OZDA2ZGQCH6ZYRAGPD7WA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482099830 (571h53m39.353182865s) [noise=true , address=145.79.227.40:4067 , piece_id=WSYDRX5YJ6OCFW52ZVMM6WZYNRFUXMR4XJ6C5727BOFMR4RRH33Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482434487 (571h53m39.353517532s) [noise=true , address=145.79.227.3:4164 , piece_id=U6F7VDPOH7K6R5EH5UUXHFP55GXOEXO5BLBJHZRTITPDQI4LP7KQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482480845 (571h53m39.35356388s) [noise=true , address=64.49.13.140:4516 , piece_id=4GUYOYOAEASUK2W25S3NILWIIIHGOZX76L4EGFLUUUEDWRALPP2A, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482841442 (571h53m39.353924477s) [noise=true , address=145.79.227.34:4910 , piece_id=ZH4DCNXET2Q6VRAUH3NQ3KKSV4OYI575X7YS72XD236EEOMOG3LA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483163645 (571h53m39.35424668s) [noise=true , address=145.79.227.29:4974 , piece_id=KEKREOD2B35XQNI45JBHE2SZZC3MQQBQUR5I6WFDAQYJQGJBXSMA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483532478 (571h53m39.354615513s) [noise=true , address=77.90.31.17:4769 , piece_id=3MKUMOQIUVERKQLLIFX52XQPTQ5LSFMHIPD5YW2YTO5YEFR36IYA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483914596 (571h53m39.354997631s) [noise=true , address=145.79.227.13:4807 , piece_id=VKXB5MWAG2YYH2AWWJEGTLBZ2QUPLI25VTHI6AMMALSM23Z3AWZQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:40.761721106 (571h53m40.632804141s) [upload=501384]: Next returned nil segment Dec 29 11:22:40.761733349 (571h53m40.632816384s) [upload=501384]: Waiting for error group managing segments... Dec 29 11:22:41.121222862 (571h53m40.992305897s) [noise=true , address=64.49.13.140:4516 , piece_id=4GUYOYOAEASUK2W25S3NILWIIIHGOZX76L4EGFLUUUEDWRALPP2A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.197311646 (571h53m41.068394691s) [noise=true , address=64.49.13.26:4953 , piece_id=4YSW5Y3A6HYG3XJ2XAB7RGVHAOHMP7G3QIZYMYTSXVXYJ3ANLNXA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.347247568 (571h53m41.218330613s) [noise=true , address=145.79.227.13:4807 , piece_id=VKXB5MWAG2YYH2AWWJEGTLBZ2QUPLI25VTHI6AMMALSM23Z3AWZQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.347917313 (571h53m41.219000348s) [noise=true , address=145.79.227.3:4164 , piece_id=U6F7VDPOH7K6R5EH5UUXHFP55GXOEXO5BLBJHZRTITPDQI4LP7KQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.348510613 (571h53m41.219593648s) [noise=true , address=145.79.227.29:4974 , piece_id=KEKREOD2B35XQNI45JBHE2SZZC3MQQBQUR5I6WFDAQYJQGJBXSMA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.503950985 (571h53m41.37503402s) [noise=true , address=77.90.31.17:4769 , piece_id=3MKUMOQIUVERKQLLIFX52XQPTQ5LSFMHIPD5YW2YTO5YEFR36IYA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.561352025 (571h53m41.43243506s) [noise=true , address=77.90.31.18:4834 , piece_id=BTWNGNQVEUEVV6HLBDM63SGNTKV4RDU4J4XFLG52WHKP53H3L5KA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.622447380 (571h53m41.493530425s) [noise=true , address=77.90.31.16:4750 , piece_id=MVVZPJEGDCBSNVOB6CD2WUZGIJTL3VBHX2G4L3TVZ6JF5ZEJ5U6Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.645494685 (571h53m41.51657773s) [noise=true , address=77.90.31.21:4889 , piece_id=BR6Z55P3A7YJ5FJRDSZFUIF66FGIJQY7WCEQ7PL5DOYWL73OMKTQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.247930045 (571h53m42.11901308s) [noise=true , address=145.79.227.30:4777 , piece_id=VSCXLXGIBSFE6E4OHLWNZIZ75PO7WQO2VFHUYHW7JUNH62KGGU7Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.262221890 (571h53m42.133304925s) [noise=true , address=145.79.227.44:4205 , piece_id=UE4QSHDQBZ3XBWQ6I6MCRVTLMU7U6W6OZDA2ZGQCH6ZYRAGPD7WA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.295598123 (571h53m42.166681158s) [noise=true , address=145.79.227.35:4515 , piece_id=L7LFZKZEZERUNX4HUNDNQ2PWHXNJRJ7OBFRNMFIICPZEUIBHGP7A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.307481351 (571h53m42.178564386s) [noise=true , address=145.79.227.34:4910 , piece_id=ZH4DCNXET2Q6VRAUH3NQ3KKSV4OYI575X7YS72XD236EEOMOG3LA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.325775906 (571h53m42.196858941s) [noise=true , address=145.79.227.40:4067 , piece_id=WSYDRX5YJ6OCFW52ZVMM6WZYNRFUXMR4XJ6C5727BOFMR4RRH33Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.325865978 (571h53m42.196949023s) [seg_pos={0 0}, upload=501384]: Segment reached optimal threshold of 14 pieces. Dec 29 11:22:42.326137074 (571h53m42.197220109s) [noise=true , address=64.49.13.169:4708 , piece_id=B2X2RWRELXWJPF6AVJJZCNU46LM5ASGRE4EWGPKL3SCLQ4VKUBDQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection Dec 29 11:22:42.326153074 (571h53m42.197236109s) [noise=true , address=64.49.13.169:4708 , piece_id=B2X2RWRELXWJPF6AVJJZCNU46LM5ASGRE4EWGPKL3SCLQ4VKUBDQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326193401 (571h53m42.197276446s) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12h5esfg6mw2c8y3AVpeFBkL5FfLAwYdjhcnQ9CJ8fjh61WLYjx): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326205705 (571h53m42.19728874s) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326407739 (571h53m42.197490784s) [noise=true , address=77.90.31.10:4524 , piece_id=Q2JGEUV5EDFTVY2CSNPZOVJIIIAFBA3IHA7S5N7YZRH3E7TSTE5A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uN7CznoK4AD8FqFkNUndawdGJ9rmjAe7oNnjJkV844ScY87qe): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326415374 (571h53m42.197498419s) [noise=true , address=77.90.31.10:4524 , piece_id=Q2JGEUV5EDFTVY2CSNPZOVJIIIAFBA3IHA7S5N7YZRH3E7TSTE5A, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326505105 (571h53m42.19758815s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tnTEYvqJ6PB2DsuXA28sEuEiGj11StA4qdhcYH1kuoaMkTVdY): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326512188 (571h53m42.197595223s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326532357 (571h53m42.197615392s) [seg_pos={0 0}, upload=501384]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 11:22:42.326671562 (571h53m42.197754597s) [upload=501384]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 11:22:42.326677843 (571h53m42.197760879s) [upload=501384]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 11:22:42.610391438 (571h53m42.481474493s) [upload=501384]: Done uploading segments... Dec 29 11:22:42.610416826 (571h53m42.481499871s) [upload=501384]: Upload finished. err: <nil> Duration: 29.039772861s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 29 out of 60 is failed (threshold=3) [.X.XX..XX..X..XXX.X.......X.X..X.XXX...XX.XXX.X.X....XXXXXX.] duration: 6.680517351s trace: https://jaeger.dev.storj.io/trace/7fce40104016cac9 FAIL (2025-12-29 11:55:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: S3VTHYQBEDUEOQKOPZ5FCL34OYGMDJKJ7VXCVXTJPYOZY2QIAAPA) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: SJBQMXH4AJGJJMX64XVKZS6D4GFGR4TLAKF3UCMIDDQCVYXOSAJQ) context deadline exceeded ======LOGS====== ec 29 11:54:12.183378237 (572h25m12.054461282s) [upload=501838]: Starting upload Dec 29 11:54:12.183434364 (572h25m12.054517399s) [upload=501838]: Done starting upload Dec 29 11:54:12.183616762 (572h25m12.054699807s) [upload=501838]: Uploading segments... Dec 29 11:54:12.183672798 (572h25m12.054755843s) [upload=501838]: Got next segment. Inline: false Dec 29 11:54:12.183700170 (572h25m12.054783215s) [upload=501838]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 11:54:12.183704618 (572h25m12.054787654s) [upload=501838]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 11:54:12.429960929 (572h25m12.301043964s) [seg_pos={0 0}, upload=501838]: Begin upload segment... Dec 29 11:54:12.429999843 (572h25m12.301082888s) [seg_pos={0 0}, upload=501838]: Done begin upload segment. Dec 29 11:54:12.430024149 (572h25m12.301107184s) [noise=true , address=145.79.227.41:4087 , piece_id=5AUNCE5E6BLUDKLK57BMI7F6EAV6VJ7MT3MNWS7ERTPPVNPATPEA, seg_pos={0 0}, upload=501838]: Uploading piece... Dec 29 11:54:12.430057042 (572h25m12.301140077s) [noise=true , address=45.91.116.14:4190 , piece_id=ZMWNVW4LQFA5FOQUZOWR73F62RTDZCALYWJP5JDOFBL54KO7SAZQ, seg_pos={0 0}, upload=501838]: Uploading piece... Dec 29 11:54:12.430416957 (572h25m12.301500002s) [noise=true , address=145.79.227.3:4149 , piece_id=UWY7EWF2FPD4DXIU6XPKK7KKM5AZKWEXZEWZR5YD7PFA37BMN7HQ, seg_pos={0 0}, upload=501838]: Uploading piece... Dec 29 11:54:12.430690799 (572h25m12.301773844s) [noise=true , address=145.79.227.28:4726 , piece_id=PHSWWSLPTVNIQF53QA6F2PD5R5PNINLN7NIOX7K4XC5FDNFTGHXA, seg_pos={0 0}, upload=501838]: Uploading piece... Dec 29 11:54:12.430962045 (572h25m12.30204509s) [noise=true , address=45.91.116.10:4083 , piece_id=S3VTHYQBEDUEOQKOPZ5FCL34OYGMDJKJ7VXCVXTJPYOZY2QIAAPA, seg_pos={0 0}, upload=501838]: Uploading piece... Dec 29 11:54:12.431212773 (572h25m12.302295808s) [noise=true , address=145.79.227.2:4113 , piece_id=X5V3AHPH7BYYSSOU7NFXUOEIV4EOHTZEY3NUV5CKEQ6NO3V7T2ZQ, seg_pos={0 0}, upload=501838]: Uploading piece... Dec 29 11:54:12.431467138 (572h25m12.302550183s) [noise=true , address=45.91.116.13:4155 , piece_id=SJBQMXH4AJGJJMX64XVKZS6D4GFGR4TLAKF3UCMIDDQCVYXOSAJQ, seg_pos={0 0}, upload=501838]: Uploading piece... Dec 29 11:54:12.431739235 (572h25m12.30282227s) [noise=true , address=145.79.227.35:4482 , piece_id=MR3WEBRCUYTNXHI67Q6KPYANVDHEOBZQGRJ4REJWHCGLU5RPWTUA, seg_pos={0 0}, upload=501838]: Uploading piece... Dec 29 11:54:12.432084863 (572h25m12.303167898s) [noise=true , address=145.79.227.14:4838 , piece_id=6WAL32VWR7GARCLAE6OVTPDF7RLC2ESQSMPFMAGC2PZ3SZMRNJGA, seg_pos={0 0}, upload=501838]: Uploading piece... Dec 29 11:54:12.433595931 (572h25m12.304678976s) [noise=true , address=45.91.116.6:4974 , piece_id=HOUKX5QELY46CMRL7235KVSUBHWZDS4YBCSLHCQSAZTR7X2TI2PA, seg_pos={0 0}, upload=501838]: Uploading piece... Dec 29 11:54:12.433697615 (572h25m12.30478065s) [noise=true , address=145.79.227.18:4007 , piece_id=HNWZSCIZAJ4VXOA3CFZWIHRXOQ5LMDG4RHIWHVI65KBYA5JVUX3Q, seg_pos={0 0}, upload=501838]: Uploading piece... Dec 29 11:54:12.434811366 (572h25m12.305894411s) [noise=true , address=145.79.227.12:4767 , piece_id=ADE5EYRHL55MQHDXUA54OLLWEBUR4CEJ7CHPPAC3ZWL342UZAGIA, seg_pos={0 0}, upload=501838]: Uploading piece... Dec 29 11:54:12.435948301 (572h25m12.307031346s) [noise=true , address=145.79.227.37:4565 , piece_id=MH2V4AX6AOOMKHONMKEUAHADFWJPMUOXXVRVRFEWLCCLRHQ3EFHA, seg_pos={0 0}, upload=501838]: Uploading piece... Dec 29 11:54:12.437024890 (572h25m12.308107925s) [noise=true , address=145.79.227.26:4654 , piece_id=GCJLKD2VIAXJYHH3T6FCMMIALNRYX4PQB6E73PM33KJKSXIKCGYA, seg_pos={0 0}, upload=501838]: Uploading piece... Dec 29 11:54:12.438235165 (572h25m12.3093182s) [noise=true , address=45.91.116.11:4120 , piece_id=GTCROTJ3COLZPTIH6OY6SER2PNCMZQNAKIGA46PYCCHFCLWGG5UQ, seg_pos={0 0}, upload=501838]: Uploading piece... Dec 29 11:54:12.439556451 (572h25m12.310639496s) [noise=true , address=77.90.31.10:4519 , piece_id=AYAQKRRH736HVTI4CYH6UJILIPMDT5RYEH7NORLWPOJHR4A5CYPA, seg_pos={0 0}, upload=501838]: Uploading piece... Dec 29 11:54:14.136140791 (572h25m14.007223826s) [upload=501838]: Next returned nil segment Dec 29 11:54:14.136153355 (572h25m14.00723639s) [upload=501838]: Waiting for error group managing segments... Dec 29 11:54:14.419035379 (572h25m14.290118414s) [noise=true , address=145.79.227.3:4149 , piece_id=UWY7EWF2FPD4DXIU6XPKK7KKM5AZKWEXZEWZR5YD7PFA37BMN7HQ, seg_pos={0 0}, upload=501838]: Done uploading piece. err: <nil> Dec 29 11:54:14.419338256 (572h25m14.290421301s) [noise=true , address=145.79.227.2:4113 , piece_id=X5V3AHPH7BYYSSOU7NFXUOEIV4EOHTZEY3NUV5CKEQ6NO3V7T2ZQ, seg_pos={0 0}, upload=501838]: Done uploading piece. err: <nil> Dec 29 11:54:14.427199976 (572h25m14.298283011s) [noise=true , address=145.79.227.12:4767 , piece_id=ADE5EYRHL55MQHDXUA54OLLWEBUR4CEJ7CHPPAC3ZWL342UZAGIA, seg_pos={0 0}, upload=501838]: Done uploading piece. err: <nil> Dec 29 11:54:14.438348766 (572h25m14.309431801s) [noise=true , address=145.79.227.14:4838 , piece_id=6WAL32VWR7GARCLAE6OVTPDF7RLC2ESQSMPFMAGC2PZ3SZMRNJGA, seg_pos={0 0}, upload=501838]: Done uploading piece. err: <nil> Dec 29 11:54:14.439031396 (572h25m14.310114432s) [noise=true , address=145.79.227.28:4726 , piece_id=PHSWWSLPTVNIQF53QA6F2PD5R5PNINLN7NIOX7K4XC5FDNFTGHXA, seg_pos={0 0}, upload=501838]: Done uploading piece. err: <nil> Dec 29 11:54:14.439411270 (572h25m14.310494315s) [noise=true , address=145.79.227.41:4087 , piece_id=5AUNCE5E6BLUDKLK57BMI7F6EAV6VJ7MT3MNWS7ERTPPVNPATPEA, seg_pos={0 0}, upload=501838]: Done uploading piece. err: <nil> Dec 29 11:54:14.444124194 (572h25m14.315207229s) [noise=true , address=145.79.227.18:4007 , piece_id=HNWZSCIZAJ4VXOA3CFZWIHRXOQ5LMDG4RHIWHVI65KBYA5JVUX3Q, seg_pos={0 0}, upload=501838]: Done uploading piece. err: <nil> Dec 29 11:54:14.452632195 (572h25m14.32371523s) [noise=true , address=145.79.227.26:4654 , piece_id=GCJLKD2VIAXJYHH3T6FCMMIALNRYX4PQB6E73PM33KJKSXIKCGYA, seg_pos={0 0}, upload=501838]: Done uploading piece. err: <nil> Dec 29 11:54:14.460803867 (572h25m14.331886912s) [noise=true , address=145.79.227.37:4565 , piece_id=MH2V4AX6AOOMKHONMKEUAHADFWJPMUOXXVRVRFEWLCCLRHQ3EFHA, seg_pos={0 0}, upload=501838]: Done uploading piece. err: <nil> Dec 29 11:54:14.700913472 (572h25m14.571996507s) [noise=true , address=45.91.116.6:4974 , piece_id=HOUKX5QELY46CMRL7235KVSUBHWZDS4YBCSLHCQSAZTR7X2TI2PA, seg_pos={0 0}, upload=501838]: Done uploading piece. err: <nil> Dec 29 11:54:14.756984224 (572h25m14.628067259s) [noise=true , address=45.91.116.10:4083 , piece_id=S3VTHYQBEDUEOQKOPZ5FCL34OYGMDJKJ7VXCVXTJPYOZY2QIAAPA, seg_pos={0 0}, upload=501838]: Done uploading piece. err: <nil> Dec 29 11:54:14.760267275 (572h25m14.63135032s) [noise=true , address=45.91.116.14:4190 , piece_id=ZMWNVW4LQFA5FOQUZOWR73F62RTDZCALYWJP5JDOFBL54KO7SAZQ, seg_pos={0 0}, upload=501838]: Done uploading piece. err: <nil> Dec 29 11:54:15.074515317 (572h25m14.945598352s) [noise=true , address=45.91.116.13:4155 , piece_id=SJBQMXH4AJGJJMX64XVKZS6D4GFGR4TLAKF3UCMIDDQCVYXOSAJQ, seg_pos={0 0}, upload=501838]: Done uploading piece. err: <nil> Dec 29 11:54:15.074540726 (572h25m14.945623761s) [seg_pos={0 0}, upload=501838]: Segment reached optimal threshold of 13 pieces. Dec 29 11:54:15.074774802 (572h25m14.945857847s) [noise=true , address=77.90.31.10:4519 , piece_id=AYAQKRRH736HVTI4CYH6UJILIPMDT5RYEH7NORLWPOJHR4A5CYPA, seg_pos={0 0}, upload=501838]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Dec 29 11:54:15.074784831 (572h25m14.945867876s) [noise=true , address=77.90.31.10:4519 , piece_id=AYAQKRRH736HVTI4CYH6UJILIPMDT5RYEH7NORLWPOJHR4A5CYPA, seg_pos={0 0}, upload=501838]: Long tail cancellation due to optimal threshold Dec 29 11:54:15.074792485 (572h25m14.94587553s) [noise=true , address=145.79.227.35:4482 , piece_id=MR3WEBRCUYTNXHI67Q6KPYANVDHEOBZQGRJ4REJWHCGLU5RPWTUA, seg_pos={0 0}, upload=501838]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: expected piece hash; context canceled; EOF Dec 29 11:54:15.074817082 (572h25m14.945900127s) [noise=true , address=145.79.227.35:4482 , piece_id=MR3WEBRCUYTNXHI67Q6KPYANVDHEOBZQGRJ4REJWHCGLU5RPWTUA, seg_pos={0 0}, upload=501838]: Long tail cancellation due to optimal threshold Dec 29 11:54:15.074949394 (572h25m14.946032429s) [noise=true , address=45.91.116.11:4120 , piece_id=GTCROTJ3COLZPTIH6OY6SER2PNCMZQNAKIGA46PYCCHFCLWGG5UQ, seg_pos={0 0}, upload=501838]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 188.166.225.170:58174->45.91.116.11:4120: use of closed network connection; write tcp 188.166.225.170:58174->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58174->45.91.116.11:4120: use of closed network connection Dec 29 11:54:15.074958461 (572h25m14.946041496s) [noise=true , address=45.91.116.11:4120 , piece_id=GTCROTJ3COLZPTIH6OY6SER2PNCMZQNAKIGA46PYCCHFCLWGG5UQ, seg_pos={0 0}, upload=501838]: Long tail cancellation due to optimal threshold Dec 29 11:54:15.074992276 (572h25m14.946075311s) [seg_pos={0 0}, upload=501838]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 11:54:15.075079723 (572h25m14.946162768s) [upload=501838]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 11:54:15.075086215 (572h25m14.94616926s) [upload=501838]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 11:54:15.375404280 (572h25m15.246487315s) [upload=501838]: Done uploading segments... Dec 29 11:54:15.375423086 (572h25m15.246506131s) [upload=501838]: Upload finished. err: <nil> Duration: 6.680552679s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 29 out of 60 is failed (threshold=6) [..X...XX..XX......X.X.XX.X.XX..X..XX.XXXX.XXXX...XXX...XXX..] duration: 3.541274597s trace: https://jaeger.dev.storj.io/trace/663575c087ceb96b FAIL (2025-12-29 11:55:16 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.541302842s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 30 out of 60 is failed (threshold=3) [X.X.XX..XX..X..XXX.X.......X.X..X.XXX...XX.XXX.X.X....XXXXXX] duration: 30.003566348s trace: https://jaeger.dev.storj.io/trace/7fce40104016cac9 FAIL (2025-12-29 11:54:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: S3VTHYQBEDUEOQKOPZ5FCL34OYGMDJKJ7VXCVXTJPYOZY2QIAAPA) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: SJBQMXH4AJGJJMX64XVKZS6D4GFGR4TLAKF3UCMIDDQCVYXOSAJQ) context deadline exceeded ======LOGS====== ec 29 11:54:12.183378237 (572h25m12.054461282s) [upload=501838]: Starting upload Dec 29 11:54:12.183434364 (572h25m12.054517399s) [upload=501838]: Done starting upload Dec 29 11:54:12.183616762 (572h25m12.054699807s) [upload=501838]: Uploading segments... Dec 29 11:54:12.183672798 (572h25m12.054755843s) [upload=501838]: Got next segment. Inline: false Dec 29 11:54:12.183700170 (572h25m12.054783215s) [upload=501838]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 11:54:12.183704618 (572h25m12.054787654s) [upload=501838]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 11:54:12.429960929 (572h25m12.301043964s) [seg_pos={0 0}, upload=501838]: Begin upload segment... Dec 29 11:54:12.429999843 (572h25m12.301082888s) [seg_pos={0 0}, upload=501838]: Done begin upload segment. Dec 29 11:54:12.430024149 (572h25m12.301107184s) [noise=true , address=145.79.227.41:4087 , piece_id=5AUNCE5E6BLUDKLK57BMI7F6EAV6VJ7MT3MNWS7ERTPPVNPATPEA, seg_pos={0 0}, upload=501838]: Uploading piece... Dec 29 11:54:12.430057042 (572h25m12.301140077s) [noise=true , address=45.91.116.14:4190 , piece_id=ZMWNVW4LQFA5FOQUZOWR73F62RTDZCALYWJP5JDOFBL54KO7SAZQ, seg_pos={0 0}, upload=501838]: Uploading piece... Dec 29 11:54:12.430416957 (572h25m12.301500002s) [noise=true , address=145.79.227.3:4149 , piece_id=UWY7EWF2FPD4DXIU6XPKK7KKM5AZKWEXZEWZR5YD7PFA37BMN7HQ, seg_pos={0 0}, upload=501838]: Uploading piece... Dec 29 11:54:12.430690799 (572h25m12.301773844s) [noise=true , address=145.79.227.28:4726 , piece_id=PHSWWSLPTVNIQF53QA6F2PD5R5PNINLN7NIOX7K4XC5FDNFTGHXA, seg_pos={0 0}, upload=501838]: Uploading piece... Dec 29 11:54:12.430962045 (572h25m12.30204509s) [noise=true , address=45.91.116.10:4083 , piece_id=S3VTHYQBEDUEOQKOPZ5FCL34OYGMDJKJ7VXCVXTJPYOZY2QIAAPA, seg_pos={0 0}, upload=501838]: Uploading piece... Dec 29 11:54:12.431212773 (572h25m12.302295808s) [noise=true , address=145.79.227.2:4113 , piece_id=X5V3AHPH7BYYSSOU7NFXUOEIV4EOHTZEY3NUV5CKEQ6NO3V7T2ZQ, seg_pos={0 0}, upload=501838]: Uploading piece... Dec 29 11:54:12.431467138 (572h25m12.302550183s) [noise=true , address=45.91.116.13:4155 , piece_id=SJBQMXH4AJGJJMX64XVKZS6D4GFGR4TLAKF3UCMIDDQCVYXOSAJQ, seg_pos={0 0}, upload=501838]: Uploading piece... Dec 29 11:54:12.431739235 (572h25m12.30282227s) [noise=true , address=145.79.227.35:4482 , piece_id=MR3WEBRCUYTNXHI67Q6KPYANVDHEOBZQGRJ4REJWHCGLU5RPWTUA, seg_pos={0 0}, upload=501838]: Uploading piece... Dec 29 11:54:12.432084863 (572h25m12.303167898s) [noise=true , address=145.79.227.14:4838 , piece_id=6WAL32VWR7GARCLAE6OVTPDF7RLC2ESQSMPFMAGC2PZ3SZMRNJGA, seg_pos={0 0}, upload=501838]: Uploading piece... Dec 29 11:54:12.433595931 (572h25m12.304678976s) [noise=true , address=45.91.116.6:4974 , piece_id=HOUKX5QELY46CMRL7235KVSUBHWZDS4YBCSLHCQSAZTR7X2TI2PA, seg_pos={0 0}, upload=501838]: Uploading piece... Dec 29 11:54:12.433697615 (572h25m12.30478065s) [noise=true , address=145.79.227.18:4007 , piece_id=HNWZSCIZAJ4VXOA3CFZWIHRXOQ5LMDG4RHIWHVI65KBYA5JVUX3Q, seg_pos={0 0}, upload=501838]: Uploading piece... Dec 29 11:54:12.434811366 (572h25m12.305894411s) [noise=true , address=145.79.227.12:4767 , piece_id=ADE5EYRHL55MQHDXUA54OLLWEBUR4CEJ7CHPPAC3ZWL342UZAGIA, seg_pos={0 0}, upload=501838]: Uploading piece... Dec 29 11:54:12.435948301 (572h25m12.307031346s) [noise=true , address=145.79.227.37:4565 , piece_id=MH2V4AX6AOOMKHONMKEUAHADFWJPMUOXXVRVRFEWLCCLRHQ3EFHA, seg_pos={0 0}, upload=501838]: Uploading piece... Dec 29 11:54:12.437024890 (572h25m12.308107925s) [noise=true , address=145.79.227.26:4654 , piece_id=GCJLKD2VIAXJYHH3T6FCMMIALNRYX4PQB6E73PM33KJKSXIKCGYA, seg_pos={0 0}, upload=501838]: Uploading piece... Dec 29 11:54:12.438235165 (572h25m12.3093182s) [noise=true , address=45.91.116.11:4120 , piece_id=GTCROTJ3COLZPTIH6OY6SER2PNCMZQNAKIGA46PYCCHFCLWGG5UQ, seg_pos={0 0}, upload=501838]: Uploading piece... Dec 29 11:54:12.439556451 (572h25m12.310639496s) [noise=true , address=77.90.31.10:4519 , piece_id=AYAQKRRH736HVTI4CYH6UJILIPMDT5RYEH7NORLWPOJHR4A5CYPA, seg_pos={0 0}, upload=501838]: Uploading piece... Dec 29 11:54:14.136140791 (572h25m14.007223826s) [upload=501838]: Next returned nil segment Dec 29 11:54:14.136153355 (572h25m14.00723639s) [upload=501838]: Waiting for error group managing segments... Dec 29 11:54:14.419035379 (572h25m14.290118414s) [noise=true , address=145.79.227.3:4149 , piece_id=UWY7EWF2FPD4DXIU6XPKK7KKM5AZKWEXZEWZR5YD7PFA37BMN7HQ, seg_pos={0 0}, upload=501838]: Done uploading piece. err: <nil> Dec 29 11:54:14.419338256 (572h25m14.290421301s) [noise=true , address=145.79.227.2:4113 , piece_id=X5V3AHPH7BYYSSOU7NFXUOEIV4EOHTZEY3NUV5CKEQ6NO3V7T2ZQ, seg_pos={0 0}, upload=501838]: Done uploading piece. err: <nil> Dec 29 11:54:14.427199976 (572h25m14.298283011s) [noise=true , address=145.79.227.12:4767 , piece_id=ADE5EYRHL55MQHDXUA54OLLWEBUR4CEJ7CHPPAC3ZWL342UZAGIA, seg_pos={0 0}, upload=501838]: Done uploading piece. err: <nil> Dec 29 11:54:14.438348766 (572h25m14.309431801s) [noise=true , address=145.79.227.14:4838 , piece_id=6WAL32VWR7GARCLAE6OVTPDF7RLC2ESQSMPFMAGC2PZ3SZMRNJGA, seg_pos={0 0}, upload=501838]: Done uploading piece. err: <nil> Dec 29 11:54:14.439031396 (572h25m14.310114432s) [noise=true , address=145.79.227.28:4726 , piece_id=PHSWWSLPTVNIQF53QA6F2PD5R5PNINLN7NIOX7K4XC5FDNFTGHXA, seg_pos={0 0}, upload=501838]: Done uploading piece. err: <nil> Dec 29 11:54:14.439411270 (572h25m14.310494315s) [noise=true , address=145.79.227.41:4087 , piece_id=5AUNCE5E6BLUDKLK57BMI7F6EAV6VJ7MT3MNWS7ERTPPVNPATPEA, seg_pos={0 0}, upload=501838]: Done uploading piece. err: <nil> Dec 29 11:54:14.444124194 (572h25m14.315207229s) [noise=true , address=145.79.227.18:4007 , piece_id=HNWZSCIZAJ4VXOA3CFZWIHRXOQ5LMDG4RHIWHVI65KBYA5JVUX3Q, seg_pos={0 0}, upload=501838]: Done uploading piece. err: <nil> Dec 29 11:54:14.452632195 (572h25m14.32371523s) [noise=true , address=145.79.227.26:4654 , piece_id=GCJLKD2VIAXJYHH3T6FCMMIALNRYX4PQB6E73PM33KJKSXIKCGYA, seg_pos={0 0}, upload=501838]: Done uploading piece. err: <nil> Dec 29 11:54:14.460803867 (572h25m14.331886912s) [noise=true , address=145.79.227.37:4565 , piece_id=MH2V4AX6AOOMKHONMKEUAHADFWJPMUOXXVRVRFEWLCCLRHQ3EFHA, seg_pos={0 0}, upload=501838]: Done uploading piece. err: <nil> Dec 29 11:54:14.700913472 (572h25m14.571996507s) [noise=true , address=45.91.116.6:4974 , piece_id=HOUKX5QELY46CMRL7235KVSUBHWZDS4YBCSLHCQSAZTR7X2TI2PA, seg_pos={0 0}, upload=501838]: Done uploading piece. err: <nil> Dec 29 11:54:14.756984224 (572h25m14.628067259s) [noise=true , address=45.91.116.10:4083 , piece_id=S3VTHYQBEDUEOQKOPZ5FCL34OYGMDJKJ7VXCVXTJPYOZY2QIAAPA, seg_pos={0 0}, upload=501838]: Done uploading piece. err: <nil> Dec 29 11:54:14.760267275 (572h25m14.63135032s) [noise=true , address=45.91.116.14:4190 , piece_id=ZMWNVW4LQFA5FOQUZOWR73F62RTDZCALYWJP5JDOFBL54KO7SAZQ, seg_pos={0 0}, upload=501838]: Done uploading piece. err: <nil> Dec 29 11:54:15.074515317 (572h25m14.945598352s) [noise=true , address=45.91.116.13:4155 , piece_id=SJBQMXH4AJGJJMX64XVKZS6D4GFGR4TLAKF3UCMIDDQCVYXOSAJQ, seg_pos={0 0}, upload=501838]: Done uploading piece. err: <nil> Dec 29 11:54:15.074540726 (572h25m14.945623761s) [seg_pos={0 0}, upload=501838]: Segment reached optimal threshold of 13 pieces. Dec 29 11:54:15.074774802 (572h25m14.945857847s) [noise=true , address=77.90.31.10:4519 , piece_id=AYAQKRRH736HVTI4CYH6UJILIPMDT5RYEH7NORLWPOJHR4A5CYPA, seg_pos={0 0}, upload=501838]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Dec 29 11:54:15.074784831 (572h25m14.945867876s) [noise=true , address=77.90.31.10:4519 , piece_id=AYAQKRRH736HVTI4CYH6UJILIPMDT5RYEH7NORLWPOJHR4A5CYPA, seg_pos={0 0}, upload=501838]: Long tail cancellation due to optimal threshold Dec 29 11:54:15.074792485 (572h25m14.94587553s) [noise=true , address=145.79.227.35:4482 , piece_id=MR3WEBRCUYTNXHI67Q6KPYANVDHEOBZQGRJ4REJWHCGLU5RPWTUA, seg_pos={0 0}, upload=501838]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: expected piece hash; context canceled; EOF Dec 29 11:54:15.074817082 (572h25m14.945900127s) [noise=true , address=145.79.227.35:4482 , piece_id=MR3WEBRCUYTNXHI67Q6KPYANVDHEOBZQGRJ4REJWHCGLU5RPWTUA, seg_pos={0 0}, upload=501838]: Long tail cancellation due to optimal threshold Dec 29 11:54:15.074949394 (572h25m14.946032429s) [noise=true , address=45.91.116.11:4120 , piece_id=GTCROTJ3COLZPTIH6OY6SER2PNCMZQNAKIGA46PYCCHFCLWGG5UQ, seg_pos={0 0}, upload=501838]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 188.166.225.170:58174->45.91.116.11:4120: use of closed network connection; write tcp 188.166.225.170:58174->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58174->45.91.116.11:4120: use of closed network connection Dec 29 11:54:15.074958461 (572h25m14.946041496s) [noise=true , address=45.91.116.11:4120 , piece_id=GTCROTJ3COLZPTIH6OY6SER2PNCMZQNAKIGA46PYCCHFCLWGG5UQ, seg_pos={0 0}, upload=501838]: Long tail cancellation due to optimal threshold Dec 29 11:54:15.074992276 (572h25m14.946075311s) [seg_pos={0 0}, upload=501838]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 11:54:15.075079723 (572h25m14.946162768s) [upload=501838]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 11:54:15.075086215 (572h25m14.94616926s) [upload=501838]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 11:54:15.375404280 (572h25m15.246487315s) [upload=501838]: Done uploading segments... Dec 29 11:54:15.375423086 (572h25m15.246506131s) [upload=501838]: Upload finished. err: <nil> Duration: 30.0035881s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)