Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [X.X...X.X.X.X.XX..XX..X..XXX.X.......X.X..X.XXX...XX.XXX.X.X] duration: 30.003010696s trace: https://jaeger.dev.storj.io/trace/35d74c7413b86052 FAIL (2025-12-29 11:45:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: GT477FKHU6WDZO3VWVRUAAS3LA22EWSSVJI4OFFQGL3KOPZ2LY7A) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: 2M5RVXZ6FCMTFWGURVDKW42STWTFYKQ3QQSH5CXP3S4NY2EK6WAA) context deadline exceeded ======LOGS====== 11:45:29.178357526 (572h16m29.049440571s) [upload=501713]: Starting upload Dec 29 11:45:29.178448629 (572h16m29.049531664s) [upload=501713]: Done starting upload Dec 29 11:45:29.178651686 (572h16m29.049734731s) [upload=501713]: Uploading segments... Dec 29 11:45:29.178745815 (572h16m29.04982885s) [upload=501713]: Got next segment. Inline: false Dec 29 11:45:29.178799217 (572h16m29.049882262s) [upload=501713]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 11:45:29.178806712 (572h16m29.049889747s) [upload=501713]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 11:45:29.426206698 (572h16m29.297289733s) [seg_pos={0 0}, upload=501713]: Begin upload segment... Dec 29 11:45:29.426275820 (572h16m29.297358865s) [seg_pos={0 0}, upload=501713]: Done begin upload segment. Dec 29 11:45:29.426331074 (572h16m29.297414119s) [noise=true , address=145.79.227.40:4051 , piece_id=G3FBYTY3KKRHFBTQNHCLRS3C5BS5H6ALDPR37FLF22Y6SG67TCNQ, seg_pos={0 0}, upload=501713]: Uploading piece... Dec 29 11:45:29.426388314 (572h16m29.297471359s) [noise=true , address=45.91.116.9:4262 , piece_id=UNL3IUL5ZY2HGHP2EX3PMDEHB7YGZP7CA2ZQIM5BEJWDND7BCUPA, seg_pos={0 0}, upload=501713]: Uploading piece... Dec 29 11:45:29.426900329 (572h16m29.297983364s) [noise=true , address=145.79.227.50:4374 , piece_id=RDQ6XPJS5QXHVV3A53XPWRO4COAXAEXXBT3JGLOZF7RLL6OUFW2A, seg_pos={0 0}, upload=501713]: Uploading piece... Dec 29 11:45:29.427399218 (572h16m29.298482263s) [noise=true , address=145.79.227.19:4079 , piece_id=ILNKHDUOPJTH2PZ4W5QDMDD7BRQ4AI2ACXWIC5MZIBB32OXFXW6Q, seg_pos={0 0}, upload=501713]: Uploading piece... Dec 29 11:45:29.428268133 (572h16m29.299351168s) [noise=true , address=145.79.227.39:4013 , piece_id=DN5EETJP2OHSLZYOXCKZJVUEFPG74HA2TUIN7H4ZPUYJGGOUWRGQ, seg_pos={0 0}, upload=501713]: Uploading piece... Dec 29 11:45:29.428980379 (572h16m29.300063414s) [noise=true , address=45.91.116.12:4226 , piece_id=WDLPSOUH2NA7AFDHRR2POG6DFEP22KBD4QW345N5LEXHIEXDUK5Q, seg_pos={0 0}, upload=501713]: Uploading piece... Dec 29 11:45:29.429585061 (572h16m29.300668096s) [noise=true , address=45.91.116.4:4939 , piece_id=GL2DOU6EDSH663L6EMJ4O5APIAUHVRTOUXMZ7VPFSG5RI7ZJOVNQ, seg_pos={0 0}, upload=501713]: Uploading piece... Dec 29 11:45:29.430106474 (572h16m29.301189519s) [noise=true , address=145.79.227.3:4151 , piece_id=XK4DTIX2OY7XS6JJAUSOZ7MKTDJR7ZSFDM2PPCNJF3GJC5YIKX4A, seg_pos={0 0}, upload=501713]: Uploading piece... Dec 29 11:45:29.430224608 (572h16m29.301307643s) [noise=true , address=145.79.227.6:4257 , piece_id=V5Y4L3E2KU3XBGAGF3OBICHX7URDTE4K6GCRBVZQQ7KWAAMCXTDQ, seg_pos={0 0}, upload=501713]: Uploading piece... Dec 29 11:45:29.430836905 (572h16m29.30191994s) [noise=true , address=145.79.227.5:4222 , piece_id=5DQJR6WCRMUC7ZH44XGJJESJZOJ5NDOKN2CK74AL6HFMVU7YVKSA, seg_pos={0 0}, upload=501713]: Uploading piece... Dec 29 11:45:29.430912017 (572h16m29.301995063s) [noise=true , address=145.79.227.17:4971 , piece_id=6I7CWECJBSRPXO3EQJTFKQ2ZGGA3TI44ZX55O46SYEOHOZ3PWGBA, seg_pos={0 0}, upload=501713]: Uploading piece... Dec 29 11:45:29.431221126 (572h16m29.302304171s) [noise=true , address=145.79.227.12:4765 , piece_id=FQ4AGT4JPB5YUSSI45ZIQKQHM2JLVZUD76BHYXQTUUIQARTJCBHQ, seg_pos={0 0}, upload=501713]: Uploading piece... Dec 29 11:45:29.432519037 (572h16m29.303602072s) [noise=true , address=45.91.116.14:4191 , piece_id=GT477FKHU6WDZO3VWVRUAAS3LA22EWSSVJI4OFFQGL3KOPZ2LY7A, seg_pos={0 0}, upload=501713]: Uploading piece... Dec 29 11:45:29.433140721 (572h16m29.304223756s) [noise=true , address=145.79.227.24:4581 , piece_id=QGAZSLGMO3SKRESBGWA2XL2F66MEBTGREQUTI2YHZZMC4E6H25BQ, seg_pos={0 0}, upload=501713]: Uploading piece... Dec 29 11:45:29.433295315 (572h16m29.30437835s) [noise=true , address=145.79.227.49:4337 , piece_id=HMCZPDT3WJ7JEF4FSWV7KSHVR6NLR3QL5FLQXJQXEGLCD3AWUBCA, seg_pos={0 0}, upload=501713]: Uploading piece... Dec 29 11:45:29.434083847 (572h16m29.305166882s) [noise=true , address=45.91.116.13:4155 , piece_id=2M5RVXZ6FCMTFWGURVDKW42STWTFYKQ3QQSH5CXP3S4NY2EK6WAA, seg_pos={0 0}, upload=501713]: Uploading piece... Dec 29 11:45:31.025242349 (572h16m30.896325394s) [upload=501713]: Next returned nil segment Dec 29 11:45:31.025257788 (572h16m30.896340823s) [upload=501713]: Waiting for error group managing segments... Dec 29 11:45:31.408678267 (572h16m31.279761302s) [noise=true , address=145.79.227.19:4079 , piece_id=ILNKHDUOPJTH2PZ4W5QDMDD7BRQ4AI2ACXWIC5MZIBB32OXFXW6Q, seg_pos={0 0}, upload=501713]: Done uploading piece. err: <nil> Dec 29 11:45:31.415898847 (572h16m31.286981882s) [noise=true , address=145.79.227.5:4222 , piece_id=5DQJR6WCRMUC7ZH44XGJJESJZOJ5NDOKN2CK74AL6HFMVU7YVKSA, seg_pos={0 0}, upload=501713]: Done uploading piece. err: <nil> Dec 29 11:45:31.416047149 (572h16m31.287130194s) [noise=true , address=145.79.227.17:4971 , piece_id=6I7CWECJBSRPXO3EQJTFKQ2ZGGA3TI44ZX55O46SYEOHOZ3PWGBA, seg_pos={0 0}, upload=501713]: Done uploading piece. err: <nil> Dec 29 11:45:31.416601555 (572h16m31.28768459s) [noise=true , address=145.79.227.3:4151 , piece_id=XK4DTIX2OY7XS6JJAUSOZ7MKTDJR7ZSFDM2PPCNJF3GJC5YIKX4A, seg_pos={0 0}, upload=501713]: Done uploading piece. err: <nil> Dec 29 11:45:31.426722616 (572h16m31.297805651s) [noise=true , address=145.79.227.39:4013 , piece_id=DN5EETJP2OHSLZYOXCKZJVUEFPG74HA2TUIN7H4ZPUYJGGOUWRGQ, seg_pos={0 0}, upload=501713]: Done uploading piece. err: <nil> Dec 29 11:45:31.427125364 (572h16m31.298208399s) [noise=true , address=145.79.227.24:4581 , piece_id=QGAZSLGMO3SKRESBGWA2XL2F66MEBTGREQUTI2YHZZMC4E6H25BQ, seg_pos={0 0}, upload=501713]: Done uploading piece. err: <nil> Dec 29 11:45:31.434883457 (572h16m31.305966492s) [noise=true , address=145.79.227.6:4257 , piece_id=V5Y4L3E2KU3XBGAGF3OBICHX7URDTE4K6GCRBVZQQ7KWAAMCXTDQ, seg_pos={0 0}, upload=501713]: Done uploading piece. err: <nil> Dec 29 11:45:31.439186129 (572h16m31.310269174s) [noise=true , address=145.79.227.12:4765 , piece_id=FQ4AGT4JPB5YUSSI45ZIQKQHM2JLVZUD76BHYXQTUUIQARTJCBHQ, seg_pos={0 0}, upload=501713]: Done uploading piece. err: <nil> Dec 29 11:45:31.442123592 (572h16m31.313206627s) [noise=true , address=145.79.227.50:4374 , piece_id=RDQ6XPJS5QXHVV3A53XPWRO4COAXAEXXBT3JGLOZF7RLL6OUFW2A, seg_pos={0 0}, upload=501713]: Done uploading piece. err: <nil> Dec 29 11:45:31.667324117 (572h16m31.538407152s) [noise=true , address=45.91.116.4:4939 , piece_id=GL2DOU6EDSH663L6EMJ4O5APIAUHVRTOUXMZ7VPFSG5RI7ZJOVNQ, seg_pos={0 0}, upload=501713]: Done uploading piece. err: <nil> Dec 29 11:45:31.679063640 (572h16m31.550146675s) [noise=true , address=45.91.116.9:4262 , piece_id=UNL3IUL5ZY2HGHP2EX3PMDEHB7YGZP7CA2ZQIM5BEJWDND7BCUPA, seg_pos={0 0}, upload=501713]: Done uploading piece. err: <nil> Dec 29 11:45:31.707638261 (572h16m31.578721296s) [noise=true , address=45.91.116.13:4155 , piece_id=2M5RVXZ6FCMTFWGURVDKW42STWTFYKQ3QQSH5CXP3S4NY2EK6WAA, seg_pos={0 0}, upload=501713]: Done uploading piece. err: <nil> Dec 29 11:45:31.771282238 (572h16m31.642365283s) [noise=true , address=45.91.116.14:4191 , piece_id=GT477FKHU6WDZO3VWVRUAAS3LA22EWSSVJI4OFFQGL3KOPZ2LY7A, seg_pos={0 0}, upload=501713]: Done uploading piece. err: <nil> Dec 29 11:45:31.771309921 (572h16m31.642392956s) [seg_pos={0 0}, upload=501713]: Segment reached optimal threshold of 13 pieces. Dec 29 11:45:31.771500173 (572h16m31.642583208s) [noise=true , address=145.79.227.40:4051 , piece_id=G3FBYTY3KKRHFBTQNHCLRS3C5BS5H6ALDPR37FLF22Y6SG67TCNQ, seg_pos={0 0}, upload=501713]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 29 11:45:31.771506866 (572h16m31.642589911s) [noise=true , address=145.79.227.40:4051 , piece_id=G3FBYTY3KKRHFBTQNHCLRS3C5BS5H6ALDPR37FLF22Y6SG67TCNQ, seg_pos={0 0}, upload=501713]: Long tail cancellation due to optimal threshold Dec 29 11:45:31.771680747 (572h16m31.642763782s) [noise=true , address=145.79.227.49:4337 , piece_id=HMCZPDT3WJ7JEF4FSWV7KSHVR6NLR3QL5FLQXJQXEGLCD3AWUBCA, seg_pos={0 0}, upload=501713]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE): protocol: expected piece hash; context canceled; EOF Dec 29 11:45:31.771686438 (572h16m31.642769473s) [noise=true , address=145.79.227.49:4337 , piece_id=HMCZPDT3WJ7JEF4FSWV7KSHVR6NLR3QL5FLQXJQXEGLCD3AWUBCA, seg_pos={0 0}, upload=501713]: Long tail cancellation due to optimal threshold Dec 29 11:45:31.771924651 (572h16m31.643007696s) [noise=true , address=45.91.116.12:4226 , piece_id=WDLPSOUH2NA7AFDHRR2POG6DFEP22KBD4QW345N5LEXHIEXDUK5Q, seg_pos={0 0}, upload=501713]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: write tcp 188.166.225.170:46212->45.91.116.12:4226: use of closed network connection; write tcp 188.166.225.170:46212->45.91.116.12:4226: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46212->45.91.116.12:4226: use of closed network connection Dec 29 11:45:31.771946623 (572h16m31.643029678s) [noise=true , address=45.91.116.12:4226 , piece_id=WDLPSOUH2NA7AFDHRR2POG6DFEP22KBD4QW345N5LEXHIEXDUK5Q, seg_pos={0 0}, upload=501713]: Long tail cancellation due to optimal threshold Dec 29 11:45:31.771976160 (572h16m31.643059205s) [seg_pos={0 0}, upload=501713]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 11:45:31.772089485 (572h16m31.64317252s) [upload=501713]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 11:45:31.772098242 (572h16m31.643181287s) [upload=501713]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 11:45:32.063478402 (572h16m31.934561437s) [upload=501713]: Done uploading segments... Dec 29 11:45:32.063497779 (572h16m31.934580814s) [upload=501713]: Upload finished. err: <nil> Duration: 30.003036235s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [.X....X.......X..X...XX..XX......X.X.XX.X.XX..X..XX.XXXX.XXX] duration: 5.000517101s trace: https://jaeger.dev.storj.io/trace/756abb622643ca70 FAIL (2025-12-29 11:45:50 UTC) orbiter warning: context deadline exceeded Duration: 5.000559502s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [............X...............X..X............................] duration: 51.241718727s trace: https://jaeger.dev.storj.io/trace/7e7354d8023608a8 FAIL (2025-12-29 11:45:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== ) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480097317 (571h53m39.351180352s) [noise=true , address=77.90.31.21:4889 , piece_id=BR6Z55P3A7YJ5FJRDSZFUIF66FGIJQY7WCEQ7PL5DOYWL73OMKTQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480405053 (571h53m39.351488098s) [noise=true , address=77.90.31.16:4750 , piece_id=MVVZPJEGDCBSNVOB6CD2WUZGIJTL3VBHX2G4L3TVZ6JF5ZEJ5U6Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480712969 (571h53m39.351796014s) [noise=true , address=77.90.31.18:4834 , piece_id=BTWNGNQVEUEVV6HLBDM63SGNTKV4RDU4J4XFLG52WHKP53H3L5KA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481019453 (571h53m39.352102488s) [noise=true , address=145.79.227.30:4777 , piece_id=VSCXLXGIBSFE6E4OHLWNZIZ75PO7WQO2VFHUYHW7JUNH62KGGU7Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481029532 (571h53m39.352112567s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481559832 (571h53m39.352642877s) [noise=true , address=145.79.227.35:4515 , piece_id=L7LFZKZEZERUNX4HUNDNQ2PWHXNJRJ7OBFRNMFIICPZEUIBHGP7A, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481731919 (571h53m39.352814954s) [noise=true , address=64.49.13.26:4953 , piece_id=4YSW5Y3A6HYG3XJ2XAB7RGVHAOHMP7G3QIZYMYTSXVXYJ3ANLNXA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482078329 (571h53m39.353161364s) [noise=true , address=145.79.227.44:4205 , piece_id=UE4QSHDQBZ3XBWQ6I6MCRVTLMU7U6W6OZDA2ZGQCH6ZYRAGPD7WA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482099830 (571h53m39.353182865s) [noise=true , address=145.79.227.40:4067 , piece_id=WSYDRX5YJ6OCFW52ZVMM6WZYNRFUXMR4XJ6C5727BOFMR4RRH33Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482434487 (571h53m39.353517532s) [noise=true , address=145.79.227.3:4164 , piece_id=U6F7VDPOH7K6R5EH5UUXHFP55GXOEXO5BLBJHZRTITPDQI4LP7KQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482480845 (571h53m39.35356388s) [noise=true , address=64.49.13.140:4516 , piece_id=4GUYOYOAEASUK2W25S3NILWIIIHGOZX76L4EGFLUUUEDWRALPP2A, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482841442 (571h53m39.353924477s) [noise=true , address=145.79.227.34:4910 , piece_id=ZH4DCNXET2Q6VRAUH3NQ3KKSV4OYI575X7YS72XD236EEOMOG3LA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483163645 (571h53m39.35424668s) [noise=true , address=145.79.227.29:4974 , piece_id=KEKREOD2B35XQNI45JBHE2SZZC3MQQBQUR5I6WFDAQYJQGJBXSMA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483532478 (571h53m39.354615513s) [noise=true , address=77.90.31.17:4769 , piece_id=3MKUMOQIUVERKQLLIFX52XQPTQ5LSFMHIPD5YW2YTO5YEFR36IYA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483914596 (571h53m39.354997631s) [noise=true , address=145.79.227.13:4807 , piece_id=VKXB5MWAG2YYH2AWWJEGTLBZ2QUPLI25VTHI6AMMALSM23Z3AWZQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:40.761721106 (571h53m40.632804141s) [upload=501384]: Next returned nil segment Dec 29 11:22:40.761733349 (571h53m40.632816384s) [upload=501384]: Waiting for error group managing segments... Dec 29 11:22:41.121222862 (571h53m40.992305897s) [noise=true , address=64.49.13.140:4516 , piece_id=4GUYOYOAEASUK2W25S3NILWIIIHGOZX76L4EGFLUUUEDWRALPP2A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.197311646 (571h53m41.068394691s) [noise=true , address=64.49.13.26:4953 , piece_id=4YSW5Y3A6HYG3XJ2XAB7RGVHAOHMP7G3QIZYMYTSXVXYJ3ANLNXA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.347247568 (571h53m41.218330613s) [noise=true , address=145.79.227.13:4807 , piece_id=VKXB5MWAG2YYH2AWWJEGTLBZ2QUPLI25VTHI6AMMALSM23Z3AWZQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.347917313 (571h53m41.219000348s) [noise=true , address=145.79.227.3:4164 , piece_id=U6F7VDPOH7K6R5EH5UUXHFP55GXOEXO5BLBJHZRTITPDQI4LP7KQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.348510613 (571h53m41.219593648s) [noise=true , address=145.79.227.29:4974 , piece_id=KEKREOD2B35XQNI45JBHE2SZZC3MQQBQUR5I6WFDAQYJQGJBXSMA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.503950985 (571h53m41.37503402s) [noise=true , address=77.90.31.17:4769 , piece_id=3MKUMOQIUVERKQLLIFX52XQPTQ5LSFMHIPD5YW2YTO5YEFR36IYA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.561352025 (571h53m41.43243506s) [noise=true , address=77.90.31.18:4834 , piece_id=BTWNGNQVEUEVV6HLBDM63SGNTKV4RDU4J4XFLG52WHKP53H3L5KA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.622447380 (571h53m41.493530425s) [noise=true , address=77.90.31.16:4750 , piece_id=MVVZPJEGDCBSNVOB6CD2WUZGIJTL3VBHX2G4L3TVZ6JF5ZEJ5U6Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.645494685 (571h53m41.51657773s) [noise=true , address=77.90.31.21:4889 , piece_id=BR6Z55P3A7YJ5FJRDSZFUIF66FGIJQY7WCEQ7PL5DOYWL73OMKTQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.247930045 (571h53m42.11901308s) [noise=true , address=145.79.227.30:4777 , piece_id=VSCXLXGIBSFE6E4OHLWNZIZ75PO7WQO2VFHUYHW7JUNH62KGGU7Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.262221890 (571h53m42.133304925s) [noise=true , address=145.79.227.44:4205 , piece_id=UE4QSHDQBZ3XBWQ6I6MCRVTLMU7U6W6OZDA2ZGQCH6ZYRAGPD7WA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.295598123 (571h53m42.166681158s) [noise=true , address=145.79.227.35:4515 , piece_id=L7LFZKZEZERUNX4HUNDNQ2PWHXNJRJ7OBFRNMFIICPZEUIBHGP7A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.307481351 (571h53m42.178564386s) [noise=true , address=145.79.227.34:4910 , piece_id=ZH4DCNXET2Q6VRAUH3NQ3KKSV4OYI575X7YS72XD236EEOMOG3LA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.325775906 (571h53m42.196858941s) [noise=true , address=145.79.227.40:4067 , piece_id=WSYDRX5YJ6OCFW52ZVMM6WZYNRFUXMR4XJ6C5727BOFMR4RRH33Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.325865978 (571h53m42.196949023s) [seg_pos={0 0}, upload=501384]: Segment reached optimal threshold of 14 pieces. Dec 29 11:22:42.326137074 (571h53m42.197220109s) [noise=true , address=64.49.13.169:4708 , piece_id=B2X2RWRELXWJPF6AVJJZCNU46LM5ASGRE4EWGPKL3SCLQ4VKUBDQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection Dec 29 11:22:42.326153074 (571h53m42.197236109s) [noise=true , address=64.49.13.169:4708 , piece_id=B2X2RWRELXWJPF6AVJJZCNU46LM5ASGRE4EWGPKL3SCLQ4VKUBDQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326193401 (571h53m42.197276446s) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12h5esfg6mw2c8y3AVpeFBkL5FfLAwYdjhcnQ9CJ8fjh61WLYjx): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326205705 (571h53m42.19728874s) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326407739 (571h53m42.197490784s) [noise=true , address=77.90.31.10:4524 , piece_id=Q2JGEUV5EDFTVY2CSNPZOVJIIIAFBA3IHA7S5N7YZRH3E7TSTE5A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uN7CznoK4AD8FqFkNUndawdGJ9rmjAe7oNnjJkV844ScY87qe): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326415374 (571h53m42.197498419s) [noise=true , address=77.90.31.10:4524 , piece_id=Q2JGEUV5EDFTVY2CSNPZOVJIIIAFBA3IHA7S5N7YZRH3E7TSTE5A, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326505105 (571h53m42.19758815s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tnTEYvqJ6PB2DsuXA28sEuEiGj11StA4qdhcYH1kuoaMkTVdY): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326512188 (571h53m42.197595223s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326532357 (571h53m42.197615392s) [seg_pos={0 0}, upload=501384]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 11:22:42.326671562 (571h53m42.197754597s) [upload=501384]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 11:22:42.326677843 (571h53m42.197760879s) [upload=501384]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 11:22:42.610391438 (571h53m42.481474493s) [upload=501384]: Done uploading segments... Dec 29 11:22:42.610416826 (571h53m42.481499871s) [upload=501384]: Upload finished. err: <nil> Duration: 51.241733937s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 24 out of 60 is failed (threshold=6) [..X....X.......X..X...XX..XX......X.X.XX.X.XX..X..XX.XXXX.XX] duration: 5.001105767s trace: https://jaeger.dev.storj.io/trace/2692e4a8cdb3711f FAIL (2025-12-29 11:44:59 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001116667s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 27 out of 60 is failed (threshold=3) [.X.X...X.X.X.X.XX..XX..X..XXX.X.......X.X..X.XXX...XX.XXX.X.] duration: 5.769311079s trace: https://jaeger.dev.storj.io/trace/8aa890280477e0f FAIL (2025-12-29 11:44:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: ZYYGPYH5QLSCNARST3UWR7SLR5ZTVQBNPBIPU3J6LWK3OMUFRFIA) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: CCH64MF2LQZPIOGGWDFJK2XQE4WBJ4RKRZNAIUOY3O3VODL7FLOA) context deadline exceeded ======LOGS====== egments... Dec 29 11:44:18.394194329 (572h15m18.265277375s) [upload=501696]: Got next segment. Inline: false Dec 29 11:44:18.394221692 (572h15m18.265304727s) [upload=501696]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 11:44:18.394228394 (572h15m18.265311429s) [upload=501696]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 11:44:18.642640749 (572h15m18.513723804s) [seg_pos={0 0}, upload=501696]: Begin upload segment... Dec 29 11:44:18.642704190 (572h15m18.513787235s) [seg_pos={0 0}, upload=501696]: Done begin upload segment. Dec 29 11:44:18.642740539 (572h15m18.513823585s) [noise=true , address=145.79.227.15:4873 , piece_id=Z47TGEBRNDMXQ2W7BTJFR6EH2FQRYNMXVFZTP467GMMVTTWCRR7Q, seg_pos={0 0}, upload=501696]: Uploading piece... Dec 29 11:44:18.646040594 (572h15m18.517123639s) [noise=true , address=45.91.116.4:4937 , piece_id=JXS64WOJLDLYDM2ON6MTDA2WFMD5NV3IGT56AHDGZLZK7HQOYAOQ, seg_pos={0 0}, upload=501696]: Uploading piece... Dec 29 11:44:18.646184748 (572h15m18.517267793s) [noise=true , address=77.90.31.10:4517 , piece_id=2VDT6R5VOE45RWZSTHUJ22TVESUFOA44P7JWT3VHHQS3UAIDGPDQ, seg_pos={0 0}, upload=501696]: Uploading piece... Dec 29 11:44:18.646697804 (572h15m18.517780849s) [noise=true , address=45.91.116.7:4011 , piece_id=B77JNPESCHEP43VRWTO33HAVRAYB4GNYKK3WMMB3QFVQ4ES7OI4Q, seg_pos={0 0}, upload=501696]: Uploading piece... Dec 29 11:44:18.647055215 (572h15m18.51813825s) [noise=true , address=145.79.227.4:4186 , piece_id=6QTFJZYG5CQS4SHL7PRDHWN2OTLV5L5B5XCMHE3JBISFACRVRH7Q, seg_pos={0 0}, upload=501696]: Uploading piece... Dec 29 11:44:18.647840450 (572h15m18.518923485s) [noise=true , address=145.79.227.33:4871 , piece_id=YJHMGZ2USMZIAHHL4EX3L4BOXW3M5RFHNBMDXJZSTF5SCGBLYPOQ, seg_pos={0 0}, upload=501696]: Uploading piece... Dec 29 11:44:18.648855152 (572h15m18.519938197s) [noise=true , address=77.90.31.21:4877 , piece_id=2ZDEBXHNIK7565QUWEWXJS2ZCQS5IRMHVJS32LX6TD6SLQJHT7LQ, seg_pos={0 0}, upload=501696]: Uploading piece... Dec 29 11:44:18.650226753 (572h15m18.521309798s) [noise=true , address=45.91.116.12:4226 , piece_id=ZYYGPYH5QLSCNARST3UWR7SLR5ZTVQBNPBIPU3J6LWK3OMUFRFIA, seg_pos={0 0}, upload=501696]: Uploading piece... Dec 29 11:44:18.650366359 (572h15m18.521449394s) [noise=true , address=45.91.116.14:4189 , piece_id=CCH64MF2LQZPIOGGWDFJK2XQE4WBJ4RKRZNAIUOY3O3VODL7FLOA, seg_pos={0 0}, upload=501696]: Uploading piece... Dec 29 11:44:18.653136264 (572h15m18.524219309s) [noise=true , address=145.79.227.48:4301 , piece_id=RLK7JDVKG3VTYY6L522DCMJMBO44ABO65DMNAXSOTZIW3V67FFRQ, seg_pos={0 0}, upload=501696]: Uploading piece... Dec 29 11:44:18.659426051 (572h15m18.530509096s) [noise=true , address=145.79.227.16:4910 , piece_id=AMUWAX5TUDSS455FDR4T74AJN63UH5E4G3UMOVACIH5RWUSQYKAQ, seg_pos={0 0}, upload=501696]: Uploading piece... Dec 29 11:44:18.662520614 (572h15m18.533603649s) [noise=true , address=45.91.116.9:4262 , piece_id=BYRP5SVZ7VSEWSOHBDUDID5YFKYKGE5OH5U4EYWEMH3A27LJFGAQ, seg_pos={0 0}, upload=501696]: Uploading piece... Dec 29 11:44:18.664257390 (572h15m18.535340425s) [noise=true , address=145.79.227.12:4765 , piece_id=6ZA4W2RI45ZNOSLQOKUAAXZIXFFJLFXI6NM6RO4XSMHWXMZWECDA, seg_pos={0 0}, upload=501696]: Uploading piece... Dec 29 11:44:18.668847731 (572h15m18.539930776s) [noise=true , address=145.79.227.18:4006 , piece_id=ZZTRMSWXYWG77KFA57DITLGHGBONEIWQQ6S6JJA236L4GVSLPT5A, seg_pos={0 0}, upload=501696]: Uploading piece... Dec 29 11:44:18.668952320 (572h15m18.540035355s) [noise=true , address=145.79.227.13:4802 , piece_id=TYBPHG3KFBEMLPEPNB4GAGLO3I7VPXM3RGUI3I3O7LVSGJGVY3AA, seg_pos={0 0}, upload=501696]: Uploading piece... Dec 29 11:44:18.670779409 (572h15m18.541862444s) [noise=true , address=145.79.227.37:4565 , piece_id=JP4U2Q7LPWJJ4VVI2Y7GFI7LIB4UGUOINPA3KITUOUFPPGNLSMYQ, seg_pos={0 0}, upload=501696]: Uploading piece... Dec 29 11:44:20.342972051 (572h15m20.214055086s) [upload=501696]: Next returned nil segment Dec 29 11:44:20.343021264 (572h15m20.214104299s) [upload=501696]: Waiting for error group managing segments... Dec 29 11:44:20.626687767 (572h15m20.497770802s) [noise=true , address=145.79.227.4:4186 , piece_id=6QTFJZYG5CQS4SHL7PRDHWN2OTLV5L5B5XCMHE3JBISFACRVRH7Q, seg_pos={0 0}, upload=501696]: Done uploading piece. err: <nil> Dec 29 11:44:20.654559369 (572h15m20.525642404s) [noise=true , address=145.79.227.12:4765 , piece_id=6ZA4W2RI45ZNOSLQOKUAAXZIXFFJLFXI6NM6RO4XSMHWXMZWECDA, seg_pos={0 0}, upload=501696]: Done uploading piece. err: <nil> Dec 29 11:44:20.655084560 (572h15m20.526167595s) [noise=true , address=145.79.227.18:4006 , piece_id=ZZTRMSWXYWG77KFA57DITLGHGBONEIWQQ6S6JJA236L4GVSLPT5A, seg_pos={0 0}, upload=501696]: Done uploading piece. err: <nil> Dec 29 11:44:20.661859690 (572h15m20.532942725s) [noise=true , address=145.79.227.15:4873 , piece_id=Z47TGEBRNDMXQ2W7BTJFR6EH2FQRYNMXVFZTP467GMMVTTWCRR7Q, seg_pos={0 0}, upload=501696]: Done uploading piece. err: <nil> Dec 29 11:44:20.671405627 (572h15m20.542488662s) [noise=true , address=145.79.227.37:4565 , piece_id=JP4U2Q7LPWJJ4VVI2Y7GFI7LIB4UGUOINPA3KITUOUFPPGNLSMYQ, seg_pos={0 0}, upload=501696]: Done uploading piece. err: <nil> Dec 29 11:44:20.674772999 (572h15m20.545856034s) [noise=true , address=145.79.227.48:4301 , piece_id=RLK7JDVKG3VTYY6L522DCMJMBO44ABO65DMNAXSOTZIW3V67FFRQ, seg_pos={0 0}, upload=501696]: Done uploading piece. err: <nil> Dec 29 11:44:20.678909896 (572h15m20.549992941s) [noise=true , address=145.79.227.16:4910 , piece_id=AMUWAX5TUDSS455FDR4T74AJN63UH5E4G3UMOVACIH5RWUSQYKAQ, seg_pos={0 0}, upload=501696]: Done uploading piece. err: <nil> Dec 29 11:44:20.688215485 (572h15m20.55929853s) [noise=true , address=145.79.227.13:4802 , piece_id=TYBPHG3KFBEMLPEPNB4GAGLO3I7VPXM3RGUI3I3O7LVSGJGVY3AA, seg_pos={0 0}, upload=501696]: Done uploading piece. err: <nil> Dec 29 11:44:20.966711934 (572h15m20.837794989s) [noise=true , address=45.91.116.14:4189 , piece_id=CCH64MF2LQZPIOGGWDFJK2XQE4WBJ4RKRZNAIUOY3O3VODL7FLOA, seg_pos={0 0}, upload=501696]: Done uploading piece. err: <nil> Dec 29 11:44:20.967025020 (572h15m20.838108055s) [noise=true , address=77.90.31.10:4517 , piece_id=2VDT6R5VOE45RWZSTHUJ22TVESUFOA44P7JWT3VHHQS3UAIDGPDQ, seg_pos={0 0}, upload=501696]: Done uploading piece. err: <nil> Dec 29 11:44:20.969266067 (572h15m20.840349102s) [noise=true , address=45.91.116.9:4262 , piece_id=BYRP5SVZ7VSEWSOHBDUDID5YFKYKGE5OH5U4EYWEMH3A27LJFGAQ, seg_pos={0 0}, upload=501696]: Done uploading piece. err: <nil> Dec 29 11:44:21.070654546 (572h15m20.941737581s) [noise=true , address=45.91.116.12:4226 , piece_id=ZYYGPYH5QLSCNARST3UWR7SLR5ZTVQBNPBIPU3J6LWK3OMUFRFIA, seg_pos={0 0}, upload=501696]: Done uploading piece. err: <nil> Dec 29 11:44:21.182140943 (572h15m21.053223988s) [noise=true , address=45.91.116.4:4937 , piece_id=JXS64WOJLDLYDM2ON6MTDA2WFMD5NV3IGT56AHDGZLZK7HQOYAOQ, seg_pos={0 0}, upload=501696]: Done uploading piece. err: <nil> Dec 29 11:44:21.182160159 (572h15m21.053243204s) [seg_pos={0 0}, upload=501696]: Segment reached optimal threshold of 13 pieces. Dec 29 11:44:21.182335353 (572h15m21.053418388s) [noise=true , address=45.91.116.7:4011 , piece_id=B77JNPESCHEP43VRWTO33HAVRAYB4GNYKK3WMMB3QFVQ4ES7OI4Q, seg_pos={0 0}, upload=501696]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: write tcp 188.166.225.170:41580->45.91.116.7:4011: use of closed network connection; write tcp 188.166.225.170:41580->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41580->45.91.116.7:4011: use of closed network connection Dec 29 11:44:21.182341274 (572h15m21.053424309s) [noise=true , address=45.91.116.7:4011 , piece_id=B77JNPESCHEP43VRWTO33HAVRAYB4GNYKK3WMMB3QFVQ4ES7OI4Q, seg_pos={0 0}, upload=501696]: Long tail cancellation due to optimal threshold Dec 29 11:44:21.182474307 (572h15m21.053557342s) [noise=true , address=145.79.227.33:4871 , piece_id=YJHMGZ2USMZIAHHL4EX3L4BOXW3M5RFHNBMDXJZSTF5SCGBLYPOQ, seg_pos={0 0}, upload=501696]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF Dec 29 11:44:21.182480910 (572h15m21.053563945s) [noise=true , address=145.79.227.33:4871 , piece_id=YJHMGZ2USMZIAHHL4EX3L4BOXW3M5RFHNBMDXJZSTF5SCGBLYPOQ, seg_pos={0 0}, upload=501696]: Long tail cancellation due to optimal threshold Dec 29 11:44:21.182599796 (572h15m21.053682871s) [noise=true , address=77.90.31.21:4877 , piece_id=2ZDEBXHNIK7565QUWEWXJS2ZCQS5IRMHVJS32LX6TD6SLQJHT7LQ, seg_pos={0 0}, upload=501696]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: write tcp 188.166.225.170:60812->77.90.31.21:4877: use of closed network connection; write tcp 188.166.225.170:60812->77.90.31.21:4877: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60812->77.90.31.21:4877: use of closed network connection Dec 29 11:44:21.182620195 (572h15m21.05370325s) [noise=true , address=77.90.31.21:4877 , piece_id=2ZDEBXHNIK7565QUWEWXJS2ZCQS5IRMHVJS32LX6TD6SLQJHT7LQ, seg_pos={0 0}, upload=501696]: Long tail cancellation due to optimal threshold Dec 29 11:44:21.182656144 (572h15m21.053739199s) [seg_pos={0 0}, upload=501696]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 11:44:21.182787895 (572h15m21.05387094s) [upload=501696]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 11:44:21.182799867 (572h15m21.053882942s) [upload=501696]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 11:44:21.474188034 (572h15m21.345271069s) [upload=501696]: Done uploading segments... Dec 29 11:44:21.474207340 (572h15m21.345290385s) [upload=501696]: Upload finished. err: <nil> Duration: 5.76932673s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............X...............X..X...........................] duration: 40.289425773s trace: https://jaeger.dev.storj.io/trace/7e7354d8023608a8 FAIL (2025-12-29 11:44:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== ) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480097317 (571h53m39.351180352s) [noise=true , address=77.90.31.21:4889 , piece_id=BR6Z55P3A7YJ5FJRDSZFUIF66FGIJQY7WCEQ7PL5DOYWL73OMKTQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480405053 (571h53m39.351488098s) [noise=true , address=77.90.31.16:4750 , piece_id=MVVZPJEGDCBSNVOB6CD2WUZGIJTL3VBHX2G4L3TVZ6JF5ZEJ5U6Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480712969 (571h53m39.351796014s) [noise=true , address=77.90.31.18:4834 , piece_id=BTWNGNQVEUEVV6HLBDM63SGNTKV4RDU4J4XFLG52WHKP53H3L5KA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481019453 (571h53m39.352102488s) [noise=true , address=145.79.227.30:4777 , piece_id=VSCXLXGIBSFE6E4OHLWNZIZ75PO7WQO2VFHUYHW7JUNH62KGGU7Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481029532 (571h53m39.352112567s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481559832 (571h53m39.352642877s) [noise=true , address=145.79.227.35:4515 , piece_id=L7LFZKZEZERUNX4HUNDNQ2PWHXNJRJ7OBFRNMFIICPZEUIBHGP7A, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481731919 (571h53m39.352814954s) [noise=true , address=64.49.13.26:4953 , piece_id=4YSW5Y3A6HYG3XJ2XAB7RGVHAOHMP7G3QIZYMYTSXVXYJ3ANLNXA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482078329 (571h53m39.353161364s) [noise=true , address=145.79.227.44:4205 , piece_id=UE4QSHDQBZ3XBWQ6I6MCRVTLMU7U6W6OZDA2ZGQCH6ZYRAGPD7WA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482099830 (571h53m39.353182865s) [noise=true , address=145.79.227.40:4067 , piece_id=WSYDRX5YJ6OCFW52ZVMM6WZYNRFUXMR4XJ6C5727BOFMR4RRH33Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482434487 (571h53m39.353517532s) [noise=true , address=145.79.227.3:4164 , piece_id=U6F7VDPOH7K6R5EH5UUXHFP55GXOEXO5BLBJHZRTITPDQI4LP7KQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482480845 (571h53m39.35356388s) [noise=true , address=64.49.13.140:4516 , piece_id=4GUYOYOAEASUK2W25S3NILWIIIHGOZX76L4EGFLUUUEDWRALPP2A, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482841442 (571h53m39.353924477s) [noise=true , address=145.79.227.34:4910 , piece_id=ZH4DCNXET2Q6VRAUH3NQ3KKSV4OYI575X7YS72XD236EEOMOG3LA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483163645 (571h53m39.35424668s) [noise=true , address=145.79.227.29:4974 , piece_id=KEKREOD2B35XQNI45JBHE2SZZC3MQQBQUR5I6WFDAQYJQGJBXSMA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483532478 (571h53m39.354615513s) [noise=true , address=77.90.31.17:4769 , piece_id=3MKUMOQIUVERKQLLIFX52XQPTQ5LSFMHIPD5YW2YTO5YEFR36IYA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483914596 (571h53m39.354997631s) [noise=true , address=145.79.227.13:4807 , piece_id=VKXB5MWAG2YYH2AWWJEGTLBZ2QUPLI25VTHI6AMMALSM23Z3AWZQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:40.761721106 (571h53m40.632804141s) [upload=501384]: Next returned nil segment Dec 29 11:22:40.761733349 (571h53m40.632816384s) [upload=501384]: Waiting for error group managing segments... Dec 29 11:22:41.121222862 (571h53m40.992305897s) [noise=true , address=64.49.13.140:4516 , piece_id=4GUYOYOAEASUK2W25S3NILWIIIHGOZX76L4EGFLUUUEDWRALPP2A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.197311646 (571h53m41.068394691s) [noise=true , address=64.49.13.26:4953 , piece_id=4YSW5Y3A6HYG3XJ2XAB7RGVHAOHMP7G3QIZYMYTSXVXYJ3ANLNXA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.347247568 (571h53m41.218330613s) [noise=true , address=145.79.227.13:4807 , piece_id=VKXB5MWAG2YYH2AWWJEGTLBZ2QUPLI25VTHI6AMMALSM23Z3AWZQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.347917313 (571h53m41.219000348s) [noise=true , address=145.79.227.3:4164 , piece_id=U6F7VDPOH7K6R5EH5UUXHFP55GXOEXO5BLBJHZRTITPDQI4LP7KQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.348510613 (571h53m41.219593648s) [noise=true , address=145.79.227.29:4974 , piece_id=KEKREOD2B35XQNI45JBHE2SZZC3MQQBQUR5I6WFDAQYJQGJBXSMA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.503950985 (571h53m41.37503402s) [noise=true , address=77.90.31.17:4769 , piece_id=3MKUMOQIUVERKQLLIFX52XQPTQ5LSFMHIPD5YW2YTO5YEFR36IYA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.561352025 (571h53m41.43243506s) [noise=true , address=77.90.31.18:4834 , piece_id=BTWNGNQVEUEVV6HLBDM63SGNTKV4RDU4J4XFLG52WHKP53H3L5KA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.622447380 (571h53m41.493530425s) [noise=true , address=77.90.31.16:4750 , piece_id=MVVZPJEGDCBSNVOB6CD2WUZGIJTL3VBHX2G4L3TVZ6JF5ZEJ5U6Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.645494685 (571h53m41.51657773s) [noise=true , address=77.90.31.21:4889 , piece_id=BR6Z55P3A7YJ5FJRDSZFUIF66FGIJQY7WCEQ7PL5DOYWL73OMKTQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.247930045 (571h53m42.11901308s) [noise=true , address=145.79.227.30:4777 , piece_id=VSCXLXGIBSFE6E4OHLWNZIZ75PO7WQO2VFHUYHW7JUNH62KGGU7Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.262221890 (571h53m42.133304925s) [noise=true , address=145.79.227.44:4205 , piece_id=UE4QSHDQBZ3XBWQ6I6MCRVTLMU7U6W6OZDA2ZGQCH6ZYRAGPD7WA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.295598123 (571h53m42.166681158s) [noise=true , address=145.79.227.35:4515 , piece_id=L7LFZKZEZERUNX4HUNDNQ2PWHXNJRJ7OBFRNMFIICPZEUIBHGP7A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.307481351 (571h53m42.178564386s) [noise=true , address=145.79.227.34:4910 , piece_id=ZH4DCNXET2Q6VRAUH3NQ3KKSV4OYI575X7YS72XD236EEOMOG3LA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.325775906 (571h53m42.196858941s) [noise=true , address=145.79.227.40:4067 , piece_id=WSYDRX5YJ6OCFW52ZVMM6WZYNRFUXMR4XJ6C5727BOFMR4RRH33Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.325865978 (571h53m42.196949023s) [seg_pos={0 0}, upload=501384]: Segment reached optimal threshold of 14 pieces. Dec 29 11:22:42.326137074 (571h53m42.197220109s) [noise=true , address=64.49.13.169:4708 , piece_id=B2X2RWRELXWJPF6AVJJZCNU46LM5ASGRE4EWGPKL3SCLQ4VKUBDQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection Dec 29 11:22:42.326153074 (571h53m42.197236109s) [noise=true , address=64.49.13.169:4708 , piece_id=B2X2RWRELXWJPF6AVJJZCNU46LM5ASGRE4EWGPKL3SCLQ4VKUBDQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326193401 (571h53m42.197276446s) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12h5esfg6mw2c8y3AVpeFBkL5FfLAwYdjhcnQ9CJ8fjh61WLYjx): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326205705 (571h53m42.19728874s) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326407739 (571h53m42.197490784s) [noise=true , address=77.90.31.10:4524 , piece_id=Q2JGEUV5EDFTVY2CSNPZOVJIIIAFBA3IHA7S5N7YZRH3E7TSTE5A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uN7CznoK4AD8FqFkNUndawdGJ9rmjAe7oNnjJkV844ScY87qe): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326415374 (571h53m42.197498419s) [noise=true , address=77.90.31.10:4524 , piece_id=Q2JGEUV5EDFTVY2CSNPZOVJIIIAFBA3IHA7S5N7YZRH3E7TSTE5A, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326505105 (571h53m42.19758815s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tnTEYvqJ6PB2DsuXA28sEuEiGj11StA4qdhcYH1kuoaMkTVdY): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326512188 (571h53m42.197595223s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326532357 (571h53m42.197615392s) [seg_pos={0 0}, upload=501384]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 11:22:42.326671562 (571h53m42.197754597s) [upload=501384]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 11:22:42.326677843 (571h53m42.197760879s) [upload=501384]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 11:22:42.610391438 (571h53m42.481474493s) [upload=501384]: Done uploading segments... Dec 29 11:22:42.610416826 (571h53m42.481499871s) [upload=501384]: Upload finished. err: <nil> Duration: 40.289443747s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 27 out of 60 is failed (threshold=3) [..X.X...X.X.X.X.XX..XX..X..XXX.X.......X.X..X.XXX...XX.XXX.X] duration: 30.002160864s trace: https://jaeger.dev.storj.io/trace/8aa890280477e0f FAIL (2025-12-29 11:44:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: ZYYGPYH5QLSCNARST3UWR7SLR5ZTVQBNPBIPU3J6LWK3OMUFRFIA) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: CCH64MF2LQZPIOGGWDFJK2XQE4WBJ4RKRZNAIUOY3O3VODL7FLOA) context deadline exceeded ======LOGS====== egments... Dec 29 11:44:18.394194329 (572h15m18.265277375s) [upload=501696]: Got next segment. Inline: false Dec 29 11:44:18.394221692 (572h15m18.265304727s) [upload=501696]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 11:44:18.394228394 (572h15m18.265311429s) [upload=501696]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 11:44:18.642640749 (572h15m18.513723804s) [seg_pos={0 0}, upload=501696]: Begin upload segment... Dec 29 11:44:18.642704190 (572h15m18.513787235s) [seg_pos={0 0}, upload=501696]: Done begin upload segment. Dec 29 11:44:18.642740539 (572h15m18.513823585s) [noise=true , address=145.79.227.15:4873 , piece_id=Z47TGEBRNDMXQ2W7BTJFR6EH2FQRYNMXVFZTP467GMMVTTWCRR7Q, seg_pos={0 0}, upload=501696]: Uploading piece... Dec 29 11:44:18.646040594 (572h15m18.517123639s) [noise=true , address=45.91.116.4:4937 , piece_id=JXS64WOJLDLYDM2ON6MTDA2WFMD5NV3IGT56AHDGZLZK7HQOYAOQ, seg_pos={0 0}, upload=501696]: Uploading piece... Dec 29 11:44:18.646184748 (572h15m18.517267793s) [noise=true , address=77.90.31.10:4517 , piece_id=2VDT6R5VOE45RWZSTHUJ22TVESUFOA44P7JWT3VHHQS3UAIDGPDQ, seg_pos={0 0}, upload=501696]: Uploading piece... Dec 29 11:44:18.646697804 (572h15m18.517780849s) [noise=true , address=45.91.116.7:4011 , piece_id=B77JNPESCHEP43VRWTO33HAVRAYB4GNYKK3WMMB3QFVQ4ES7OI4Q, seg_pos={0 0}, upload=501696]: Uploading piece... Dec 29 11:44:18.647055215 (572h15m18.51813825s) [noise=true , address=145.79.227.4:4186 , piece_id=6QTFJZYG5CQS4SHL7PRDHWN2OTLV5L5B5XCMHE3JBISFACRVRH7Q, seg_pos={0 0}, upload=501696]: Uploading piece... Dec 29 11:44:18.647840450 (572h15m18.518923485s) [noise=true , address=145.79.227.33:4871 , piece_id=YJHMGZ2USMZIAHHL4EX3L4BOXW3M5RFHNBMDXJZSTF5SCGBLYPOQ, seg_pos={0 0}, upload=501696]: Uploading piece... Dec 29 11:44:18.648855152 (572h15m18.519938197s) [noise=true , address=77.90.31.21:4877 , piece_id=2ZDEBXHNIK7565QUWEWXJS2ZCQS5IRMHVJS32LX6TD6SLQJHT7LQ, seg_pos={0 0}, upload=501696]: Uploading piece... Dec 29 11:44:18.650226753 (572h15m18.521309798s) [noise=true , address=45.91.116.12:4226 , piece_id=ZYYGPYH5QLSCNARST3UWR7SLR5ZTVQBNPBIPU3J6LWK3OMUFRFIA, seg_pos={0 0}, upload=501696]: Uploading piece... Dec 29 11:44:18.650366359 (572h15m18.521449394s) [noise=true , address=45.91.116.14:4189 , piece_id=CCH64MF2LQZPIOGGWDFJK2XQE4WBJ4RKRZNAIUOY3O3VODL7FLOA, seg_pos={0 0}, upload=501696]: Uploading piece... Dec 29 11:44:18.653136264 (572h15m18.524219309s) [noise=true , address=145.79.227.48:4301 , piece_id=RLK7JDVKG3VTYY6L522DCMJMBO44ABO65DMNAXSOTZIW3V67FFRQ, seg_pos={0 0}, upload=501696]: Uploading piece... Dec 29 11:44:18.659426051 (572h15m18.530509096s) [noise=true , address=145.79.227.16:4910 , piece_id=AMUWAX5TUDSS455FDR4T74AJN63UH5E4G3UMOVACIH5RWUSQYKAQ, seg_pos={0 0}, upload=501696]: Uploading piece... Dec 29 11:44:18.662520614 (572h15m18.533603649s) [noise=true , address=45.91.116.9:4262 , piece_id=BYRP5SVZ7VSEWSOHBDUDID5YFKYKGE5OH5U4EYWEMH3A27LJFGAQ, seg_pos={0 0}, upload=501696]: Uploading piece... Dec 29 11:44:18.664257390 (572h15m18.535340425s) [noise=true , address=145.79.227.12:4765 , piece_id=6ZA4W2RI45ZNOSLQOKUAAXZIXFFJLFXI6NM6RO4XSMHWXMZWECDA, seg_pos={0 0}, upload=501696]: Uploading piece... Dec 29 11:44:18.668847731 (572h15m18.539930776s) [noise=true , address=145.79.227.18:4006 , piece_id=ZZTRMSWXYWG77KFA57DITLGHGBONEIWQQ6S6JJA236L4GVSLPT5A, seg_pos={0 0}, upload=501696]: Uploading piece... Dec 29 11:44:18.668952320 (572h15m18.540035355s) [noise=true , address=145.79.227.13:4802 , piece_id=TYBPHG3KFBEMLPEPNB4GAGLO3I7VPXM3RGUI3I3O7LVSGJGVY3AA, seg_pos={0 0}, upload=501696]: Uploading piece... Dec 29 11:44:18.670779409 (572h15m18.541862444s) [noise=true , address=145.79.227.37:4565 , piece_id=JP4U2Q7LPWJJ4VVI2Y7GFI7LIB4UGUOINPA3KITUOUFPPGNLSMYQ, seg_pos={0 0}, upload=501696]: Uploading piece... Dec 29 11:44:20.342972051 (572h15m20.214055086s) [upload=501696]: Next returned nil segment Dec 29 11:44:20.343021264 (572h15m20.214104299s) [upload=501696]: Waiting for error group managing segments... Dec 29 11:44:20.626687767 (572h15m20.497770802s) [noise=true , address=145.79.227.4:4186 , piece_id=6QTFJZYG5CQS4SHL7PRDHWN2OTLV5L5B5XCMHE3JBISFACRVRH7Q, seg_pos={0 0}, upload=501696]: Done uploading piece. err: <nil> Dec 29 11:44:20.654559369 (572h15m20.525642404s) [noise=true , address=145.79.227.12:4765 , piece_id=6ZA4W2RI45ZNOSLQOKUAAXZIXFFJLFXI6NM6RO4XSMHWXMZWECDA, seg_pos={0 0}, upload=501696]: Done uploading piece. err: <nil> Dec 29 11:44:20.655084560 (572h15m20.526167595s) [noise=true , address=145.79.227.18:4006 , piece_id=ZZTRMSWXYWG77KFA57DITLGHGBONEIWQQ6S6JJA236L4GVSLPT5A, seg_pos={0 0}, upload=501696]: Done uploading piece. err: <nil> Dec 29 11:44:20.661859690 (572h15m20.532942725s) [noise=true , address=145.79.227.15:4873 , piece_id=Z47TGEBRNDMXQ2W7BTJFR6EH2FQRYNMXVFZTP467GMMVTTWCRR7Q, seg_pos={0 0}, upload=501696]: Done uploading piece. err: <nil> Dec 29 11:44:20.671405627 (572h15m20.542488662s) [noise=true , address=145.79.227.37:4565 , piece_id=JP4U2Q7LPWJJ4VVI2Y7GFI7LIB4UGUOINPA3KITUOUFPPGNLSMYQ, seg_pos={0 0}, upload=501696]: Done uploading piece. err: <nil> Dec 29 11:44:20.674772999 (572h15m20.545856034s) [noise=true , address=145.79.227.48:4301 , piece_id=RLK7JDVKG3VTYY6L522DCMJMBO44ABO65DMNAXSOTZIW3V67FFRQ, seg_pos={0 0}, upload=501696]: Done uploading piece. err: <nil> Dec 29 11:44:20.678909896 (572h15m20.549992941s) [noise=true , address=145.79.227.16:4910 , piece_id=AMUWAX5TUDSS455FDR4T74AJN63UH5E4G3UMOVACIH5RWUSQYKAQ, seg_pos={0 0}, upload=501696]: Done uploading piece. err: <nil> Dec 29 11:44:20.688215485 (572h15m20.55929853s) [noise=true , address=145.79.227.13:4802 , piece_id=TYBPHG3KFBEMLPEPNB4GAGLO3I7VPXM3RGUI3I3O7LVSGJGVY3AA, seg_pos={0 0}, upload=501696]: Done uploading piece. err: <nil> Dec 29 11:44:20.966711934 (572h15m20.837794989s) [noise=true , address=45.91.116.14:4189 , piece_id=CCH64MF2LQZPIOGGWDFJK2XQE4WBJ4RKRZNAIUOY3O3VODL7FLOA, seg_pos={0 0}, upload=501696]: Done uploading piece. err: <nil> Dec 29 11:44:20.967025020 (572h15m20.838108055s) [noise=true , address=77.90.31.10:4517 , piece_id=2VDT6R5VOE45RWZSTHUJ22TVESUFOA44P7JWT3VHHQS3UAIDGPDQ, seg_pos={0 0}, upload=501696]: Done uploading piece. err: <nil> Dec 29 11:44:20.969266067 (572h15m20.840349102s) [noise=true , address=45.91.116.9:4262 , piece_id=BYRP5SVZ7VSEWSOHBDUDID5YFKYKGE5OH5U4EYWEMH3A27LJFGAQ, seg_pos={0 0}, upload=501696]: Done uploading piece. err: <nil> Dec 29 11:44:21.070654546 (572h15m20.941737581s) [noise=true , address=45.91.116.12:4226 , piece_id=ZYYGPYH5QLSCNARST3UWR7SLR5ZTVQBNPBIPU3J6LWK3OMUFRFIA, seg_pos={0 0}, upload=501696]: Done uploading piece. err: <nil> Dec 29 11:44:21.182140943 (572h15m21.053223988s) [noise=true , address=45.91.116.4:4937 , piece_id=JXS64WOJLDLYDM2ON6MTDA2WFMD5NV3IGT56AHDGZLZK7HQOYAOQ, seg_pos={0 0}, upload=501696]: Done uploading piece. err: <nil> Dec 29 11:44:21.182160159 (572h15m21.053243204s) [seg_pos={0 0}, upload=501696]: Segment reached optimal threshold of 13 pieces. Dec 29 11:44:21.182335353 (572h15m21.053418388s) [noise=true , address=45.91.116.7:4011 , piece_id=B77JNPESCHEP43VRWTO33HAVRAYB4GNYKK3WMMB3QFVQ4ES7OI4Q, seg_pos={0 0}, upload=501696]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: write tcp 188.166.225.170:41580->45.91.116.7:4011: use of closed network connection; write tcp 188.166.225.170:41580->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41580->45.91.116.7:4011: use of closed network connection Dec 29 11:44:21.182341274 (572h15m21.053424309s) [noise=true , address=45.91.116.7:4011 , piece_id=B77JNPESCHEP43VRWTO33HAVRAYB4GNYKK3WMMB3QFVQ4ES7OI4Q, seg_pos={0 0}, upload=501696]: Long tail cancellation due to optimal threshold Dec 29 11:44:21.182474307 (572h15m21.053557342s) [noise=true , address=145.79.227.33:4871 , piece_id=YJHMGZ2USMZIAHHL4EX3L4BOXW3M5RFHNBMDXJZSTF5SCGBLYPOQ, seg_pos={0 0}, upload=501696]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF Dec 29 11:44:21.182480910 (572h15m21.053563945s) [noise=true , address=145.79.227.33:4871 , piece_id=YJHMGZ2USMZIAHHL4EX3L4BOXW3M5RFHNBMDXJZSTF5SCGBLYPOQ, seg_pos={0 0}, upload=501696]: Long tail cancellation due to optimal threshold Dec 29 11:44:21.182599796 (572h15m21.053682871s) [noise=true , address=77.90.31.21:4877 , piece_id=2ZDEBXHNIK7565QUWEWXJS2ZCQS5IRMHVJS32LX6TD6SLQJHT7LQ, seg_pos={0 0}, upload=501696]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: write tcp 188.166.225.170:60812->77.90.31.21:4877: use of closed network connection; write tcp 188.166.225.170:60812->77.90.31.21:4877: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60812->77.90.31.21:4877: use of closed network connection Dec 29 11:44:21.182620195 (572h15m21.05370325s) [noise=true , address=77.90.31.21:4877 , piece_id=2ZDEBXHNIK7565QUWEWXJS2ZCQS5IRMHVJS32LX6TD6SLQJHT7LQ, seg_pos={0 0}, upload=501696]: Long tail cancellation due to optimal threshold Dec 29 11:44:21.182656144 (572h15m21.053739199s) [seg_pos={0 0}, upload=501696]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 11:44:21.182787895 (572h15m21.05387094s) [upload=501696]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 11:44:21.182799867 (572h15m21.053882942s) [upload=501696]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 11:44:21.474188034 (572h15m21.345271069s) [upload=501696]: Done uploading segments... Dec 29 11:44:21.474207340 (572h15m21.345290385s) [upload=501696]: Upload finished. err: <nil> Duration: 30.002183998s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 23 out of 60 is failed (threshold=6) [...X....X.......X..X...XX..XX......X.X.XX.X.XX..X..XX.XXXX.X] duration: 5.000982649s trace: https://jaeger.dev.storj.io/trace/27f023a31914e801 FAIL (2025-12-29 11:44:36 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00100392s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............X...............X..X..........................] duration: 42.646902411s trace: https://jaeger.dev.storj.io/trace/7e7354d8023608a8 FAIL (2025-12-29 11:44:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== ) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480097317 (571h53m39.351180352s) [noise=true , address=77.90.31.21:4889 , piece_id=BR6Z55P3A7YJ5FJRDSZFUIF66FGIJQY7WCEQ7PL5DOYWL73OMKTQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480405053 (571h53m39.351488098s) [noise=true , address=77.90.31.16:4750 , piece_id=MVVZPJEGDCBSNVOB6CD2WUZGIJTL3VBHX2G4L3TVZ6JF5ZEJ5U6Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480712969 (571h53m39.351796014s) [noise=true , address=77.90.31.18:4834 , piece_id=BTWNGNQVEUEVV6HLBDM63SGNTKV4RDU4J4XFLG52WHKP53H3L5KA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481019453 (571h53m39.352102488s) [noise=true , address=145.79.227.30:4777 , piece_id=VSCXLXGIBSFE6E4OHLWNZIZ75PO7WQO2VFHUYHW7JUNH62KGGU7Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481029532 (571h53m39.352112567s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481559832 (571h53m39.352642877s) [noise=true , address=145.79.227.35:4515 , piece_id=L7LFZKZEZERUNX4HUNDNQ2PWHXNJRJ7OBFRNMFIICPZEUIBHGP7A, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481731919 (571h53m39.352814954s) [noise=true , address=64.49.13.26:4953 , piece_id=4YSW5Y3A6HYG3XJ2XAB7RGVHAOHMP7G3QIZYMYTSXVXYJ3ANLNXA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482078329 (571h53m39.353161364s) [noise=true , address=145.79.227.44:4205 , piece_id=UE4QSHDQBZ3XBWQ6I6MCRVTLMU7U6W6OZDA2ZGQCH6ZYRAGPD7WA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482099830 (571h53m39.353182865s) [noise=true , address=145.79.227.40:4067 , piece_id=WSYDRX5YJ6OCFW52ZVMM6WZYNRFUXMR4XJ6C5727BOFMR4RRH33Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482434487 (571h53m39.353517532s) [noise=true , address=145.79.227.3:4164 , piece_id=U6F7VDPOH7K6R5EH5UUXHFP55GXOEXO5BLBJHZRTITPDQI4LP7KQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482480845 (571h53m39.35356388s) [noise=true , address=64.49.13.140:4516 , piece_id=4GUYOYOAEASUK2W25S3NILWIIIHGOZX76L4EGFLUUUEDWRALPP2A, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482841442 (571h53m39.353924477s) [noise=true , address=145.79.227.34:4910 , piece_id=ZH4DCNXET2Q6VRAUH3NQ3KKSV4OYI575X7YS72XD236EEOMOG3LA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483163645 (571h53m39.35424668s) [noise=true , address=145.79.227.29:4974 , piece_id=KEKREOD2B35XQNI45JBHE2SZZC3MQQBQUR5I6WFDAQYJQGJBXSMA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483532478 (571h53m39.354615513s) [noise=true , address=77.90.31.17:4769 , piece_id=3MKUMOQIUVERKQLLIFX52XQPTQ5LSFMHIPD5YW2YTO5YEFR36IYA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483914596 (571h53m39.354997631s) [noise=true , address=145.79.227.13:4807 , piece_id=VKXB5MWAG2YYH2AWWJEGTLBZ2QUPLI25VTHI6AMMALSM23Z3AWZQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:40.761721106 (571h53m40.632804141s) [upload=501384]: Next returned nil segment Dec 29 11:22:40.761733349 (571h53m40.632816384s) [upload=501384]: Waiting for error group managing segments... Dec 29 11:22:41.121222862 (571h53m40.992305897s) [noise=true , address=64.49.13.140:4516 , piece_id=4GUYOYOAEASUK2W25S3NILWIIIHGOZX76L4EGFLUUUEDWRALPP2A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.197311646 (571h53m41.068394691s) [noise=true , address=64.49.13.26:4953 , piece_id=4YSW5Y3A6HYG3XJ2XAB7RGVHAOHMP7G3QIZYMYTSXVXYJ3ANLNXA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.347247568 (571h53m41.218330613s) [noise=true , address=145.79.227.13:4807 , piece_id=VKXB5MWAG2YYH2AWWJEGTLBZ2QUPLI25VTHI6AMMALSM23Z3AWZQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.347917313 (571h53m41.219000348s) [noise=true , address=145.79.227.3:4164 , piece_id=U6F7VDPOH7K6R5EH5UUXHFP55GXOEXO5BLBJHZRTITPDQI4LP7KQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.348510613 (571h53m41.219593648s) [noise=true , address=145.79.227.29:4974 , piece_id=KEKREOD2B35XQNI45JBHE2SZZC3MQQBQUR5I6WFDAQYJQGJBXSMA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.503950985 (571h53m41.37503402s) [noise=true , address=77.90.31.17:4769 , piece_id=3MKUMOQIUVERKQLLIFX52XQPTQ5LSFMHIPD5YW2YTO5YEFR36IYA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.561352025 (571h53m41.43243506s) [noise=true , address=77.90.31.18:4834 , piece_id=BTWNGNQVEUEVV6HLBDM63SGNTKV4RDU4J4XFLG52WHKP53H3L5KA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.622447380 (571h53m41.493530425s) [noise=true , address=77.90.31.16:4750 , piece_id=MVVZPJEGDCBSNVOB6CD2WUZGIJTL3VBHX2G4L3TVZ6JF5ZEJ5U6Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.645494685 (571h53m41.51657773s) [noise=true , address=77.90.31.21:4889 , piece_id=BR6Z55P3A7YJ5FJRDSZFUIF66FGIJQY7WCEQ7PL5DOYWL73OMKTQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.247930045 (571h53m42.11901308s) [noise=true , address=145.79.227.30:4777 , piece_id=VSCXLXGIBSFE6E4OHLWNZIZ75PO7WQO2VFHUYHW7JUNH62KGGU7Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.262221890 (571h53m42.133304925s) [noise=true , address=145.79.227.44:4205 , piece_id=UE4QSHDQBZ3XBWQ6I6MCRVTLMU7U6W6OZDA2ZGQCH6ZYRAGPD7WA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.295598123 (571h53m42.166681158s) [noise=true , address=145.79.227.35:4515 , piece_id=L7LFZKZEZERUNX4HUNDNQ2PWHXNJRJ7OBFRNMFIICPZEUIBHGP7A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.307481351 (571h53m42.178564386s) [noise=true , address=145.79.227.34:4910 , piece_id=ZH4DCNXET2Q6VRAUH3NQ3KKSV4OYI575X7YS72XD236EEOMOG3LA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.325775906 (571h53m42.196858941s) [noise=true , address=145.79.227.40:4067 , piece_id=WSYDRX5YJ6OCFW52ZVMM6WZYNRFUXMR4XJ6C5727BOFMR4RRH33Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.325865978 (571h53m42.196949023s) [seg_pos={0 0}, upload=501384]: Segment reached optimal threshold of 14 pieces. Dec 29 11:22:42.326137074 (571h53m42.197220109s) [noise=true , address=64.49.13.169:4708 , piece_id=B2X2RWRELXWJPF6AVJJZCNU46LM5ASGRE4EWGPKL3SCLQ4VKUBDQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection Dec 29 11:22:42.326153074 (571h53m42.197236109s) [noise=true , address=64.49.13.169:4708 , piece_id=B2X2RWRELXWJPF6AVJJZCNU46LM5ASGRE4EWGPKL3SCLQ4VKUBDQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326193401 (571h53m42.197276446s) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12h5esfg6mw2c8y3AVpeFBkL5FfLAwYdjhcnQ9CJ8fjh61WLYjx): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326205705 (571h53m42.19728874s) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326407739 (571h53m42.197490784s) [noise=true , address=77.90.31.10:4524 , piece_id=Q2JGEUV5EDFTVY2CSNPZOVJIIIAFBA3IHA7S5N7YZRH3E7TSTE5A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uN7CznoK4AD8FqFkNUndawdGJ9rmjAe7oNnjJkV844ScY87qe): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326415374 (571h53m42.197498419s) [noise=true , address=77.90.31.10:4524 , piece_id=Q2JGEUV5EDFTVY2CSNPZOVJIIIAFBA3IHA7S5N7YZRH3E7TSTE5A, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326505105 (571h53m42.19758815s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tnTEYvqJ6PB2DsuXA28sEuEiGj11StA4qdhcYH1kuoaMkTVdY): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326512188 (571h53m42.197595223s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326532357 (571h53m42.197615392s) [seg_pos={0 0}, upload=501384]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 11:22:42.326671562 (571h53m42.197754597s) [upload=501384]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 11:22:42.326677843 (571h53m42.197760879s) [upload=501384]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 11:22:42.610391438 (571h53m42.481474493s) [upload=501384]: Done uploading segments... Dec 29 11:22:42.610416826 (571h53m42.481499871s) [upload=501384]: Upload finished. err: <nil> Duration: 42.646927608s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [...X.X...X.X.X.X.XX..XX..X..XXX.X.......X.X..X.XXX...XX.XXX.] duration: 5.973909244s trace: https://jaeger.dev.storj.io/trace/5f0cec5203452a77 FAIL (2025-12-29 11:43:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ding segments... Dec 29 11:42:38.893269934 (572h13m38.764352979s) [upload=501675]: Got next segment. Inline: false Dec 29 11:42:38.893304609 (572h13m38.764387644s) [upload=501675]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 11:42:38.893309399 (572h13m38.764392444s) [upload=501675]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 11:42:39.140853510 (572h13m39.011936555s) [seg_pos={0 0}, upload=501675]: Begin upload segment... Dec 29 11:42:39.140901751 (572h13m39.011984786s) [seg_pos={0 0}, upload=501675]: Done begin upload segment. Dec 29 11:42:39.140929935 (572h13m39.01201297s) [noise=true , address=77.90.31.13:4626 , piece_id=CR36M4G6P3M7YJKVLW3HTCWO2ZTDVD3IBJASDN74BGKYXNB4QHIQ, seg_pos={0 0}, upload=501675]: Uploading piece... Dec 29 11:42:39.141382377 (572h13m39.012465412s) [noise=true , address=145.79.227.37:4566 , piece_id=UDB32B2SM3BKLNWCFS3MAVFFDRPIVEXOGY2NIZVVJ3ZZS62JPEOQ, seg_pos={0 0}, upload=501675]: Uploading piece... Dec 29 11:42:39.141958102 (572h13m39.013041138s) [noise=true , address=45.91.116.14:4190 , piece_id=VAZA4SJ7RWGRPZTPDJNNZU4YOQQILOZSM7FFH5TGXTF5FKC6KMUQ, seg_pos={0 0}, upload=501675]: Uploading piece... Dec 29 11:42:39.142334950 (572h13m39.013417985s) [noise=true , address=45.91.116.8:4047 , piece_id=KZKD3H7LCDDI5U3PTRQP2SPMZU64ERJLM2X7XLQXGUX56OD33B5Q, seg_pos={0 0}, upload=501675]: Uploading piece... Dec 29 11:42:39.142641865 (572h13m39.0137249s) [noise=true , address=45.91.116.11:4117 , piece_id=MCQDNW6SNNTZNAT5PBDBAIHLAKQOAKK5UDXJEYPMYYYYGGILGAIA, seg_pos={0 0}, upload=501675]: Uploading piece... Dec 29 11:42:39.142937377 (572h13m39.014020412s) [noise=true , address=145.79.227.11:4731 , piece_id=DUVVPQMXPGKYA3WLL6EFJV55YDWO7NY47RTMGKSCASE2CN6Z5MNA, seg_pos={0 0}, upload=501675]: Uploading piece... Dec 29 11:42:39.143274980 (572h13m39.014358015s) [noise=true , address=145.79.227.20:4437 , piece_id=EPJHXPCW2AGWRIGTGREFYIANKM2OKDY3DFOJTFGAN2AKGQRVJSEA, seg_pos={0 0}, upload=501675]: Uploading piece... Dec 29 11:42:39.143556115 (572h13m39.014639161s) [noise=true , address=145.79.227.50:4374 , piece_id=YQHGIAMXGJH47ZPTLO3GKWBL7LXN77XE7JYWVMN4OYIQUQY5SFMQ, seg_pos={0 0}, upload=501675]: Uploading piece... Dec 29 11:42:39.143812985 (572h13m39.01489603s) [noise=true , address=145.79.227.16:4909 , piece_id=VT3CD62H6IJUI65NR3IFYX3Z2MJT72JM2DAUEV7JFZT2LGOAXZTA, seg_pos={0 0}, upload=501675]: Uploading piece... Dec 29 11:42:39.146164312 (572h13m39.017247357s) [noise=true , address=145.79.227.48:4302 , piece_id=H2IDFCNUSUW7B253SMFRCDDH4YKK4N4KVGFALRWBDXANJBIH5KPA, seg_pos={0 0}, upload=501675]: Uploading piece... Dec 29 11:42:39.146745779 (572h13m39.017828815s) [noise=true , address=77.90.31.21:4878 , piece_id=ZQE56GCTRQCQXWNFRZW7N7DR4ILYVFNOPUPRAMBMXLH2UBH5UPYA, seg_pos={0 0}, upload=501675]: Uploading piece... Dec 29 11:42:39.147314181 (572h13m39.018397217s) [noise=true , address=77.90.31.12:4589 , piece_id=S2LBJVKVLYEPOX3BLNCX7LQGZ7JQH7E46MQG6RNAXZZEAEBDYD5Q, seg_pos={0 0}, upload=501675]: Uploading piece... Dec 29 11:42:39.148531600 (572h13m39.019614635s) [noise=true , address=145.79.227.46:4603 , piece_id=IC4H4VK7LJEO6EBD64FXEQISMW5QDYEBE6RIOIZ442BFGRQZ53PA, seg_pos={0 0}, upload=501675]: Uploading piece... Dec 29 11:42:39.150024432 (572h13m39.021107498s) [noise=true , address=145.79.227.32:4836 , piece_id=3LS2S2EH75OOLITN5WK3GV4CB7EFKQRUZK2LXC7EVARBA4XCLQGQ, seg_pos={0 0}, upload=501675]: Uploading piece... Dec 29 11:42:39.151379823 (572h13m39.022462858s) [noise=true , address=145.79.227.3:4150 , piece_id=2W5HD5746TADCDZNV6CQP2QF6OXNRNWW3RVTELM4V5ZMVOORXBZA, seg_pos={0 0}, upload=501675]: Uploading piece... Dec 29 11:42:39.152861635 (572h13m39.023944681s) [noise=true , address=77.90.31.10:4518 , piece_id=L4C3ZT47WY3UMRFAAHODPONRWHGCGPIQD6UHPIZG64BD7BQNADYQ, seg_pos={0 0}, upload=501675]: Uploading piece... Dec 29 11:42:40.834309560 (572h13m40.705392595s) [upload=501675]: Next returned nil segment Dec 29 11:42:40.834322545 (572h13m40.70540558s) [upload=501675]: Waiting for error group managing segments... Dec 29 11:42:41.119214070 (572h13m40.990297115s) [noise=true , address=145.79.227.50:4374 , piece_id=YQHGIAMXGJH47ZPTLO3GKWBL7LXN77XE7JYWVMN4OYIQUQY5SFMQ, seg_pos={0 0}, upload=501675]: Done uploading piece. err: <nil> Dec 29 11:42:41.134018821 (572h13m41.005101856s) [noise=true , address=145.79.227.3:4150 , piece_id=2W5HD5746TADCDZNV6CQP2QF6OXNRNWW3RVTELM4V5ZMVOORXBZA, seg_pos={0 0}, upload=501675]: Done uploading piece. err: <nil> Dec 29 11:42:41.136707762 (572h13m41.007790797s) [noise=true , address=145.79.227.11:4731 , piece_id=DUVVPQMXPGKYA3WLL6EFJV55YDWO7NY47RTMGKSCASE2CN6Z5MNA, seg_pos={0 0}, upload=501675]: Done uploading piece. err: <nil> Dec 29 11:42:41.137409337 (572h13m41.008492372s) [noise=true , address=145.79.227.20:4437 , piece_id=EPJHXPCW2AGWRIGTGREFYIANKM2OKDY3DFOJTFGAN2AKGQRVJSEA, seg_pos={0 0}, upload=501675]: Done uploading piece. err: <nil> Dec 29 11:42:41.144200238 (572h13m41.015283283s) [noise=true , address=145.79.227.16:4909 , piece_id=VT3CD62H6IJUI65NR3IFYX3Z2MJT72JM2DAUEV7JFZT2LGOAXZTA, seg_pos={0 0}, upload=501675]: Done uploading piece. err: <nil> Dec 29 11:42:41.149422142 (572h13m41.020505177s) [noise=true , address=145.79.227.48:4302 , piece_id=H2IDFCNUSUW7B253SMFRCDDH4YKK4N4KVGFALRWBDXANJBIH5KPA, seg_pos={0 0}, upload=501675]: Done uploading piece. err: <nil> Dec 29 11:42:41.152081605 (572h13m41.02316465s) [noise=true , address=145.79.227.46:4603 , piece_id=IC4H4VK7LJEO6EBD64FXEQISMW5QDYEBE6RIOIZ442BFGRQZ53PA, seg_pos={0 0}, upload=501675]: Done uploading piece. err: <nil> Dec 29 11:42:41.156549282 (572h13m41.027632317s) [noise=true , address=145.79.227.32:4836 , piece_id=3LS2S2EH75OOLITN5WK3GV4CB7EFKQRUZK2LXC7EVARBA4XCLQGQ, seg_pos={0 0}, upload=501675]: Done uploading piece. err: <nil> Dec 29 11:42:41.161490281 (572h13m41.032573316s) [noise=true , address=145.79.227.37:4566 , piece_id=UDB32B2SM3BKLNWCFS3MAVFFDRPIVEXOGY2NIZVVJ3ZZS62JPEOQ, seg_pos={0 0}, upload=501675]: Done uploading piece. err: <nil> Dec 29 11:42:41.385423282 (572h13m41.256506317s) [noise=true , address=45.91.116.8:4047 , piece_id=KZKD3H7LCDDI5U3PTRQP2SPMZU64ERJLM2X7XLQXGUX56OD33B5Q, seg_pos={0 0}, upload=501675]: Done uploading piece. err: <nil> Dec 29 11:42:41.416208352 (572h13m41.287291397s) [noise=true , address=45.91.116.11:4117 , piece_id=MCQDNW6SNNTZNAT5PBDBAIHLAKQOAKK5UDXJEYPMYYYYGGILGAIA, seg_pos={0 0}, upload=501675]: Done uploading piece. err: <nil> Dec 29 11:42:41.457373016 (572h13m41.328456061s) [noise=true , address=45.91.116.14:4190 , piece_id=VAZA4SJ7RWGRPZTPDJNNZU4YOQQILOZSM7FFH5TGXTF5FKC6KMUQ, seg_pos={0 0}, upload=501675]: Done uploading piece. err: <nil> Dec 29 11:42:41.683286518 (572h13m41.554369564s) [noise=true , address=77.90.31.10:4518 , piece_id=L4C3ZT47WY3UMRFAAHODPONRWHGCGPIQD6UHPIZG64BD7BQNADYQ, seg_pos={0 0}, upload=501675]: Done uploading piece. err: <nil> Dec 29 11:42:41.683325673 (572h13m41.554408718s) [seg_pos={0 0}, upload=501675]: Segment reached optimal threshold of 13 pieces. Dec 29 11:42:41.683635783 (572h13m41.554718829s) [noise=true , address=77.90.31.21:4878 , piece_id=ZQE56GCTRQCQXWNFRZW7N7DR4ILYVFNOPUPRAMBMXLH2UBH5UPYA, seg_pos={0 0}, upload=501675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: write tcp 188.166.225.170:59810->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:59810->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59810->77.90.31.21:4878: use of closed network connection Dec 29 11:42:41.683647425 (572h13m41.55473046s) [noise=true , address=77.90.31.21:4878 , piece_id=ZQE56GCTRQCQXWNFRZW7N7DR4ILYVFNOPUPRAMBMXLH2UBH5UPYA, seg_pos={0 0}, upload=501675]: Long tail cancellation due to optimal threshold Dec 29 11:42:41.684282655 (572h13m41.55536571s) [noise=true , address=77.90.31.12:4589 , piece_id=S2LBJVKVLYEPOX3BLNCX7LQGZ7JQH7E46MQG6RNAXZZEAEBDYD5Q, seg_pos={0 0}, upload=501675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 29 11:42:41.684293817 (572h13m41.555376882s) [noise=true , address=77.90.31.12:4589 , piece_id=S2LBJVKVLYEPOX3BLNCX7LQGZ7JQH7E46MQG6RNAXZZEAEBDYD5Q, seg_pos={0 0}, upload=501675]: Long tail cancellation due to optimal threshold Dec 29 11:42:41.684368859 (572h13m41.555451894s) [noise=true , address=77.90.31.13:4626 , piece_id=CR36M4G6P3M7YJKVLW3HTCWO2ZTDVD3IBJASDN74BGKYXNB4QHIQ, seg_pos={0 0}, upload=501675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: write tcp 188.166.225.170:51330->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:51330->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51330->77.90.31.13:4626: use of closed network connection Dec 29 11:42:41.684401321 (572h13m41.555484356s) [noise=true , address=77.90.31.13:4626 , piece_id=CR36M4G6P3M7YJKVLW3HTCWO2ZTDVD3IBJASDN74BGKYXNB4QHIQ, seg_pos={0 0}, upload=501675]: Long tail cancellation due to optimal threshold Dec 29 11:42:41.684499427 (572h13m41.555582462s) [seg_pos={0 0}, upload=501675]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 11:42:41.684772618 (572h13m41.555855683s) [upload=501675]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 11:42:41.684782617 (572h13m41.555865662s) [upload=501675]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 11:42:42.004727708 (572h13m41.875810763s) [upload=501675]: Done uploading segments... Dec 29 11:42:42.004748278 (572h13m41.875831313s) [upload=501675]: Upload finished. err: <nil> Duration: 5.973941355s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 22 out of 60 is failed (threshold=6) [....X....X.......X..X...XX..XX......X.X.XX.X.XX..X..XX.XXXX.] duration: 3.418908387s trace: https://jaeger.dev.storj.io/trace/17e744d0edc2fc99 FAIL (2025-12-29 11:43:37 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.419111183s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [....X.X...X.X.X.X.XX..XX..X..XXX.X.......X.X..X.XXX...XX.XXX] duration: 30.002840433s trace: https://jaeger.dev.storj.io/trace/5f0cec5203452a77 FAIL (2025-12-29 11:43:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ding segments... Dec 29 11:42:38.893269934 (572h13m38.764352979s) [upload=501675]: Got next segment. Inline: false Dec 29 11:42:38.893304609 (572h13m38.764387644s) [upload=501675]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 11:42:38.893309399 (572h13m38.764392444s) [upload=501675]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 11:42:39.140853510 (572h13m39.011936555s) [seg_pos={0 0}, upload=501675]: Begin upload segment... Dec 29 11:42:39.140901751 (572h13m39.011984786s) [seg_pos={0 0}, upload=501675]: Done begin upload segment. Dec 29 11:42:39.140929935 (572h13m39.01201297s) [noise=true , address=77.90.31.13:4626 , piece_id=CR36M4G6P3M7YJKVLW3HTCWO2ZTDVD3IBJASDN74BGKYXNB4QHIQ, seg_pos={0 0}, upload=501675]: Uploading piece... Dec 29 11:42:39.141382377 (572h13m39.012465412s) [noise=true , address=145.79.227.37:4566 , piece_id=UDB32B2SM3BKLNWCFS3MAVFFDRPIVEXOGY2NIZVVJ3ZZS62JPEOQ, seg_pos={0 0}, upload=501675]: Uploading piece... Dec 29 11:42:39.141958102 (572h13m39.013041138s) [noise=true , address=45.91.116.14:4190 , piece_id=VAZA4SJ7RWGRPZTPDJNNZU4YOQQILOZSM7FFH5TGXTF5FKC6KMUQ, seg_pos={0 0}, upload=501675]: Uploading piece... Dec 29 11:42:39.142334950 (572h13m39.013417985s) [noise=true , address=45.91.116.8:4047 , piece_id=KZKD3H7LCDDI5U3PTRQP2SPMZU64ERJLM2X7XLQXGUX56OD33B5Q, seg_pos={0 0}, upload=501675]: Uploading piece... Dec 29 11:42:39.142641865 (572h13m39.0137249s) [noise=true , address=45.91.116.11:4117 , piece_id=MCQDNW6SNNTZNAT5PBDBAIHLAKQOAKK5UDXJEYPMYYYYGGILGAIA, seg_pos={0 0}, upload=501675]: Uploading piece... Dec 29 11:42:39.142937377 (572h13m39.014020412s) [noise=true , address=145.79.227.11:4731 , piece_id=DUVVPQMXPGKYA3WLL6EFJV55YDWO7NY47RTMGKSCASE2CN6Z5MNA, seg_pos={0 0}, upload=501675]: Uploading piece... Dec 29 11:42:39.143274980 (572h13m39.014358015s) [noise=true , address=145.79.227.20:4437 , piece_id=EPJHXPCW2AGWRIGTGREFYIANKM2OKDY3DFOJTFGAN2AKGQRVJSEA, seg_pos={0 0}, upload=501675]: Uploading piece... Dec 29 11:42:39.143556115 (572h13m39.014639161s) [noise=true , address=145.79.227.50:4374 , piece_id=YQHGIAMXGJH47ZPTLO3GKWBL7LXN77XE7JYWVMN4OYIQUQY5SFMQ, seg_pos={0 0}, upload=501675]: Uploading piece... Dec 29 11:42:39.143812985 (572h13m39.01489603s) [noise=true , address=145.79.227.16:4909 , piece_id=VT3CD62H6IJUI65NR3IFYX3Z2MJT72JM2DAUEV7JFZT2LGOAXZTA, seg_pos={0 0}, upload=501675]: Uploading piece... Dec 29 11:42:39.146164312 (572h13m39.017247357s) [noise=true , address=145.79.227.48:4302 , piece_id=H2IDFCNUSUW7B253SMFRCDDH4YKK4N4KVGFALRWBDXANJBIH5KPA, seg_pos={0 0}, upload=501675]: Uploading piece... Dec 29 11:42:39.146745779 (572h13m39.017828815s) [noise=true , address=77.90.31.21:4878 , piece_id=ZQE56GCTRQCQXWNFRZW7N7DR4ILYVFNOPUPRAMBMXLH2UBH5UPYA, seg_pos={0 0}, upload=501675]: Uploading piece... Dec 29 11:42:39.147314181 (572h13m39.018397217s) [noise=true , address=77.90.31.12:4589 , piece_id=S2LBJVKVLYEPOX3BLNCX7LQGZ7JQH7E46MQG6RNAXZZEAEBDYD5Q, seg_pos={0 0}, upload=501675]: Uploading piece... Dec 29 11:42:39.148531600 (572h13m39.019614635s) [noise=true , address=145.79.227.46:4603 , piece_id=IC4H4VK7LJEO6EBD64FXEQISMW5QDYEBE6RIOIZ442BFGRQZ53PA, seg_pos={0 0}, upload=501675]: Uploading piece... Dec 29 11:42:39.150024432 (572h13m39.021107498s) [noise=true , address=145.79.227.32:4836 , piece_id=3LS2S2EH75OOLITN5WK3GV4CB7EFKQRUZK2LXC7EVARBA4XCLQGQ, seg_pos={0 0}, upload=501675]: Uploading piece... Dec 29 11:42:39.151379823 (572h13m39.022462858s) [noise=true , address=145.79.227.3:4150 , piece_id=2W5HD5746TADCDZNV6CQP2QF6OXNRNWW3RVTELM4V5ZMVOORXBZA, seg_pos={0 0}, upload=501675]: Uploading piece... Dec 29 11:42:39.152861635 (572h13m39.023944681s) [noise=true , address=77.90.31.10:4518 , piece_id=L4C3ZT47WY3UMRFAAHODPONRWHGCGPIQD6UHPIZG64BD7BQNADYQ, seg_pos={0 0}, upload=501675]: Uploading piece... Dec 29 11:42:40.834309560 (572h13m40.705392595s) [upload=501675]: Next returned nil segment Dec 29 11:42:40.834322545 (572h13m40.70540558s) [upload=501675]: Waiting for error group managing segments... Dec 29 11:42:41.119214070 (572h13m40.990297115s) [noise=true , address=145.79.227.50:4374 , piece_id=YQHGIAMXGJH47ZPTLO3GKWBL7LXN77XE7JYWVMN4OYIQUQY5SFMQ, seg_pos={0 0}, upload=501675]: Done uploading piece. err: <nil> Dec 29 11:42:41.134018821 (572h13m41.005101856s) [noise=true , address=145.79.227.3:4150 , piece_id=2W5HD5746TADCDZNV6CQP2QF6OXNRNWW3RVTELM4V5ZMVOORXBZA, seg_pos={0 0}, upload=501675]: Done uploading piece. err: <nil> Dec 29 11:42:41.136707762 (572h13m41.007790797s) [noise=true , address=145.79.227.11:4731 , piece_id=DUVVPQMXPGKYA3WLL6EFJV55YDWO7NY47RTMGKSCASE2CN6Z5MNA, seg_pos={0 0}, upload=501675]: Done uploading piece. err: <nil> Dec 29 11:42:41.137409337 (572h13m41.008492372s) [noise=true , address=145.79.227.20:4437 , piece_id=EPJHXPCW2AGWRIGTGREFYIANKM2OKDY3DFOJTFGAN2AKGQRVJSEA, seg_pos={0 0}, upload=501675]: Done uploading piece. err: <nil> Dec 29 11:42:41.144200238 (572h13m41.015283283s) [noise=true , address=145.79.227.16:4909 , piece_id=VT3CD62H6IJUI65NR3IFYX3Z2MJT72JM2DAUEV7JFZT2LGOAXZTA, seg_pos={0 0}, upload=501675]: Done uploading piece. err: <nil> Dec 29 11:42:41.149422142 (572h13m41.020505177s) [noise=true , address=145.79.227.48:4302 , piece_id=H2IDFCNUSUW7B253SMFRCDDH4YKK4N4KVGFALRWBDXANJBIH5KPA, seg_pos={0 0}, upload=501675]: Done uploading piece. err: <nil> Dec 29 11:42:41.152081605 (572h13m41.02316465s) [noise=true , address=145.79.227.46:4603 , piece_id=IC4H4VK7LJEO6EBD64FXEQISMW5QDYEBE6RIOIZ442BFGRQZ53PA, seg_pos={0 0}, upload=501675]: Done uploading piece. err: <nil> Dec 29 11:42:41.156549282 (572h13m41.027632317s) [noise=true , address=145.79.227.32:4836 , piece_id=3LS2S2EH75OOLITN5WK3GV4CB7EFKQRUZK2LXC7EVARBA4XCLQGQ, seg_pos={0 0}, upload=501675]: Done uploading piece. err: <nil> Dec 29 11:42:41.161490281 (572h13m41.032573316s) [noise=true , address=145.79.227.37:4566 , piece_id=UDB32B2SM3BKLNWCFS3MAVFFDRPIVEXOGY2NIZVVJ3ZZS62JPEOQ, seg_pos={0 0}, upload=501675]: Done uploading piece. err: <nil> Dec 29 11:42:41.385423282 (572h13m41.256506317s) [noise=true , address=45.91.116.8:4047 , piece_id=KZKD3H7LCDDI5U3PTRQP2SPMZU64ERJLM2X7XLQXGUX56OD33B5Q, seg_pos={0 0}, upload=501675]: Done uploading piece. err: <nil> Dec 29 11:42:41.416208352 (572h13m41.287291397s) [noise=true , address=45.91.116.11:4117 , piece_id=MCQDNW6SNNTZNAT5PBDBAIHLAKQOAKK5UDXJEYPMYYYYGGILGAIA, seg_pos={0 0}, upload=501675]: Done uploading piece. err: <nil> Dec 29 11:42:41.457373016 (572h13m41.328456061s) [noise=true , address=45.91.116.14:4190 , piece_id=VAZA4SJ7RWGRPZTPDJNNZU4YOQQILOZSM7FFH5TGXTF5FKC6KMUQ, seg_pos={0 0}, upload=501675]: Done uploading piece. err: <nil> Dec 29 11:42:41.683286518 (572h13m41.554369564s) [noise=true , address=77.90.31.10:4518 , piece_id=L4C3ZT47WY3UMRFAAHODPONRWHGCGPIQD6UHPIZG64BD7BQNADYQ, seg_pos={0 0}, upload=501675]: Done uploading piece. err: <nil> Dec 29 11:42:41.683325673 (572h13m41.554408718s) [seg_pos={0 0}, upload=501675]: Segment reached optimal threshold of 13 pieces. Dec 29 11:42:41.683635783 (572h13m41.554718829s) [noise=true , address=77.90.31.21:4878 , piece_id=ZQE56GCTRQCQXWNFRZW7N7DR4ILYVFNOPUPRAMBMXLH2UBH5UPYA, seg_pos={0 0}, upload=501675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: write tcp 188.166.225.170:59810->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:59810->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59810->77.90.31.21:4878: use of closed network connection Dec 29 11:42:41.683647425 (572h13m41.55473046s) [noise=true , address=77.90.31.21:4878 , piece_id=ZQE56GCTRQCQXWNFRZW7N7DR4ILYVFNOPUPRAMBMXLH2UBH5UPYA, seg_pos={0 0}, upload=501675]: Long tail cancellation due to optimal threshold Dec 29 11:42:41.684282655 (572h13m41.55536571s) [noise=true , address=77.90.31.12:4589 , piece_id=S2LBJVKVLYEPOX3BLNCX7LQGZ7JQH7E46MQG6RNAXZZEAEBDYD5Q, seg_pos={0 0}, upload=501675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 29 11:42:41.684293817 (572h13m41.555376882s) [noise=true , address=77.90.31.12:4589 , piece_id=S2LBJVKVLYEPOX3BLNCX7LQGZ7JQH7E46MQG6RNAXZZEAEBDYD5Q, seg_pos={0 0}, upload=501675]: Long tail cancellation due to optimal threshold Dec 29 11:42:41.684368859 (572h13m41.555451894s) [noise=true , address=77.90.31.13:4626 , piece_id=CR36M4G6P3M7YJKVLW3HTCWO2ZTDVD3IBJASDN74BGKYXNB4QHIQ, seg_pos={0 0}, upload=501675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: write tcp 188.166.225.170:51330->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:51330->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51330->77.90.31.13:4626: use of closed network connection Dec 29 11:42:41.684401321 (572h13m41.555484356s) [noise=true , address=77.90.31.13:4626 , piece_id=CR36M4G6P3M7YJKVLW3HTCWO2ZTDVD3IBJASDN74BGKYXNB4QHIQ, seg_pos={0 0}, upload=501675]: Long tail cancellation due to optimal threshold Dec 29 11:42:41.684499427 (572h13m41.555582462s) [seg_pos={0 0}, upload=501675]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 11:42:41.684772618 (572h13m41.555855683s) [upload=501675]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 11:42:41.684782617 (572h13m41.555865662s) [upload=501675]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 11:42:42.004727708 (572h13m41.875810763s) [upload=501675]: Done uploading segments... Dec 29 11:42:42.004748278 (572h13m41.875831313s) [upload=501675]: Upload finished. err: <nil> Duration: 30.002851905s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 22 out of 60 is failed (threshold=6) [.....X....X.......X..X...XX..XX......X.X.XX.X.XX..X..XX.XXXX] duration: 5.001378251s trace: https://jaeger.dev.storj.io/trace/17e744d0edc2fc99 FAIL (2025-12-29 11:42:56 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001400083s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............X...............X..X.........................] duration: 25.47726012s trace: https://jaeger.dev.storj.io/trace/7e7354d8023608a8 FAIL (2025-12-29 11:42:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== ) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480097317 (571h53m39.351180352s) [noise=true , address=77.90.31.21:4889 , piece_id=BR6Z55P3A7YJ5FJRDSZFUIF66FGIJQY7WCEQ7PL5DOYWL73OMKTQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480405053 (571h53m39.351488098s) [noise=true , address=77.90.31.16:4750 , piece_id=MVVZPJEGDCBSNVOB6CD2WUZGIJTL3VBHX2G4L3TVZ6JF5ZEJ5U6Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480712969 (571h53m39.351796014s) [noise=true , address=77.90.31.18:4834 , piece_id=BTWNGNQVEUEVV6HLBDM63SGNTKV4RDU4J4XFLG52WHKP53H3L5KA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481019453 (571h53m39.352102488s) [noise=true , address=145.79.227.30:4777 , piece_id=VSCXLXGIBSFE6E4OHLWNZIZ75PO7WQO2VFHUYHW7JUNH62KGGU7Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481029532 (571h53m39.352112567s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481559832 (571h53m39.352642877s) [noise=true , address=145.79.227.35:4515 , piece_id=L7LFZKZEZERUNX4HUNDNQ2PWHXNJRJ7OBFRNMFIICPZEUIBHGP7A, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481731919 (571h53m39.352814954s) [noise=true , address=64.49.13.26:4953 , piece_id=4YSW5Y3A6HYG3XJ2XAB7RGVHAOHMP7G3QIZYMYTSXVXYJ3ANLNXA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482078329 (571h53m39.353161364s) [noise=true , address=145.79.227.44:4205 , piece_id=UE4QSHDQBZ3XBWQ6I6MCRVTLMU7U6W6OZDA2ZGQCH6ZYRAGPD7WA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482099830 (571h53m39.353182865s) [noise=true , address=145.79.227.40:4067 , piece_id=WSYDRX5YJ6OCFW52ZVMM6WZYNRFUXMR4XJ6C5727BOFMR4RRH33Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482434487 (571h53m39.353517532s) [noise=true , address=145.79.227.3:4164 , piece_id=U6F7VDPOH7K6R5EH5UUXHFP55GXOEXO5BLBJHZRTITPDQI4LP7KQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482480845 (571h53m39.35356388s) [noise=true , address=64.49.13.140:4516 , piece_id=4GUYOYOAEASUK2W25S3NILWIIIHGOZX76L4EGFLUUUEDWRALPP2A, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482841442 (571h53m39.353924477s) [noise=true , address=145.79.227.34:4910 , piece_id=ZH4DCNXET2Q6VRAUH3NQ3KKSV4OYI575X7YS72XD236EEOMOG3LA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483163645 (571h53m39.35424668s) [noise=true , address=145.79.227.29:4974 , piece_id=KEKREOD2B35XQNI45JBHE2SZZC3MQQBQUR5I6WFDAQYJQGJBXSMA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483532478 (571h53m39.354615513s) [noise=true , address=77.90.31.17:4769 , piece_id=3MKUMOQIUVERKQLLIFX52XQPTQ5LSFMHIPD5YW2YTO5YEFR36IYA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483914596 (571h53m39.354997631s) [noise=true , address=145.79.227.13:4807 , piece_id=VKXB5MWAG2YYH2AWWJEGTLBZ2QUPLI25VTHI6AMMALSM23Z3AWZQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:40.761721106 (571h53m40.632804141s) [upload=501384]: Next returned nil segment Dec 29 11:22:40.761733349 (571h53m40.632816384s) [upload=501384]: Waiting for error group managing segments... Dec 29 11:22:41.121222862 (571h53m40.992305897s) [noise=true , address=64.49.13.140:4516 , piece_id=4GUYOYOAEASUK2W25S3NILWIIIHGOZX76L4EGFLUUUEDWRALPP2A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.197311646 (571h53m41.068394691s) [noise=true , address=64.49.13.26:4953 , piece_id=4YSW5Y3A6HYG3XJ2XAB7RGVHAOHMP7G3QIZYMYTSXVXYJ3ANLNXA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.347247568 (571h53m41.218330613s) [noise=true , address=145.79.227.13:4807 , piece_id=VKXB5MWAG2YYH2AWWJEGTLBZ2QUPLI25VTHI6AMMALSM23Z3AWZQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.347917313 (571h53m41.219000348s) [noise=true , address=145.79.227.3:4164 , piece_id=U6F7VDPOH7K6R5EH5UUXHFP55GXOEXO5BLBJHZRTITPDQI4LP7KQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.348510613 (571h53m41.219593648s) [noise=true , address=145.79.227.29:4974 , piece_id=KEKREOD2B35XQNI45JBHE2SZZC3MQQBQUR5I6WFDAQYJQGJBXSMA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.503950985 (571h53m41.37503402s) [noise=true , address=77.90.31.17:4769 , piece_id=3MKUMOQIUVERKQLLIFX52XQPTQ5LSFMHIPD5YW2YTO5YEFR36IYA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.561352025 (571h53m41.43243506s) [noise=true , address=77.90.31.18:4834 , piece_id=BTWNGNQVEUEVV6HLBDM63SGNTKV4RDU4J4XFLG52WHKP53H3L5KA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.622447380 (571h53m41.493530425s) [noise=true , address=77.90.31.16:4750 , piece_id=MVVZPJEGDCBSNVOB6CD2WUZGIJTL3VBHX2G4L3TVZ6JF5ZEJ5U6Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.645494685 (571h53m41.51657773s) [noise=true , address=77.90.31.21:4889 , piece_id=BR6Z55P3A7YJ5FJRDSZFUIF66FGIJQY7WCEQ7PL5DOYWL73OMKTQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.247930045 (571h53m42.11901308s) [noise=true , address=145.79.227.30:4777 , piece_id=VSCXLXGIBSFE6E4OHLWNZIZ75PO7WQO2VFHUYHW7JUNH62KGGU7Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.262221890 (571h53m42.133304925s) [noise=true , address=145.79.227.44:4205 , piece_id=UE4QSHDQBZ3XBWQ6I6MCRVTLMU7U6W6OZDA2ZGQCH6ZYRAGPD7WA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.295598123 (571h53m42.166681158s) [noise=true , address=145.79.227.35:4515 , piece_id=L7LFZKZEZERUNX4HUNDNQ2PWHXNJRJ7OBFRNMFIICPZEUIBHGP7A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.307481351 (571h53m42.178564386s) [noise=true , address=145.79.227.34:4910 , piece_id=ZH4DCNXET2Q6VRAUH3NQ3KKSV4OYI575X7YS72XD236EEOMOG3LA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.325775906 (571h53m42.196858941s) [noise=true , address=145.79.227.40:4067 , piece_id=WSYDRX5YJ6OCFW52ZVMM6WZYNRFUXMR4XJ6C5727BOFMR4RRH33Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.325865978 (571h53m42.196949023s) [seg_pos={0 0}, upload=501384]: Segment reached optimal threshold of 14 pieces. Dec 29 11:22:42.326137074 (571h53m42.197220109s) [noise=true , address=64.49.13.169:4708 , piece_id=B2X2RWRELXWJPF6AVJJZCNU46LM5ASGRE4EWGPKL3SCLQ4VKUBDQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection Dec 29 11:22:42.326153074 (571h53m42.197236109s) [noise=true , address=64.49.13.169:4708 , piece_id=B2X2RWRELXWJPF6AVJJZCNU46LM5ASGRE4EWGPKL3SCLQ4VKUBDQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326193401 (571h53m42.197276446s) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12h5esfg6mw2c8y3AVpeFBkL5FfLAwYdjhcnQ9CJ8fjh61WLYjx): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326205705 (571h53m42.19728874s) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326407739 (571h53m42.197490784s) [noise=true , address=77.90.31.10:4524 , piece_id=Q2JGEUV5EDFTVY2CSNPZOVJIIIAFBA3IHA7S5N7YZRH3E7TSTE5A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uN7CznoK4AD8FqFkNUndawdGJ9rmjAe7oNnjJkV844ScY87qe): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326415374 (571h53m42.197498419s) [noise=true , address=77.90.31.10:4524 , piece_id=Q2JGEUV5EDFTVY2CSNPZOVJIIIAFBA3IHA7S5N7YZRH3E7TSTE5A, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326505105 (571h53m42.19758815s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tnTEYvqJ6PB2DsuXA28sEuEiGj11StA4qdhcYH1kuoaMkTVdY): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326512188 (571h53m42.197595223s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326532357 (571h53m42.197615392s) [seg_pos={0 0}, upload=501384]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 11:22:42.326671562 (571h53m42.197754597s) [upload=501384]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 11:22:42.326677843 (571h53m42.197760879s) [upload=501384]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 11:22:42.610391438 (571h53m42.481474493s) [upload=501384]: Done uploading segments... Dec 29 11:22:42.610416826 (571h53m42.481499871s) [upload=501384]: Upload finished. err: <nil> Duration: 25.477275569s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [.....X.X...X.X.X.X.XX..XX..X..XXX.X.......X.X..X.XXX...XX.XX] duration: 30.002381214s trace: https://jaeger.dev.storj.io/trace/42a344130a66999b FAIL (2025-12-29 11:42:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: 3CGQCULIL4PHZ5AZLRGY5OX6LPWIPRJJVIQGGOG6CR4GSLNS6JTA) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: QJM5SK3I2GG5XOIC7DZC45XVGEFITZG6RLB37JALZCSNIIWPP2PQ) context deadline exceeded ======LOGS====== Dec 29 11:42:08.647350940 (572h13m8.518433975s) [upload=501665]: Starting upload Dec 29 11:42:08.647422697 (572h13m8.518505732s) [upload=501665]: Done starting upload Dec 29 11:42:08.647505263 (572h13m8.518588299s) [upload=501665]: Uploading segments... Dec 29 11:42:08.647669717 (572h13m8.518752762s) [upload=501665]: Got next segment. Inline: false Dec 29 11:42:08.647710184 (572h13m8.518793229s) [upload=501665]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 11:42:08.647716846 (572h13m8.518799881s) [upload=501665]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 11:42:08.891364949 (572h13m8.762447984s) [seg_pos={0 0}, upload=501665]: Begin upload segment... Dec 29 11:42:08.891420725 (572h13m8.76250376s) [seg_pos={0 0}, upload=501665]: Done begin upload segment. Dec 29 11:42:08.891509885 (572h13m8.76259292s) [noise=true , address=145.79.227.52:4446 , piece_id=SHAJ2N627KM2ETWH6AXYQ2N2JWAYKF5UEBHY7WD7EXJKX5XNQYJA, seg_pos={0 0}, upload=501665]: Uploading piece... Dec 29 11:42:08.891525574 (572h13m8.762608609s) [noise=true , address=45.91.116.9:4261 , piece_id=QJM5SK3I2GG5XOIC7DZC45XVGEFITZG6RLB37JALZCSNIIWPP2PQ, seg_pos={0 0}, upload=501665]: Uploading piece... Dec 29 11:42:08.892031578 (572h13m8.763114623s) [noise=true , address=145.79.227.12:4767 , piece_id=VZRTNGI2WZA7DDNBUN4JXUUAAPNKTC4IKM6UYFUVTS2UR4QL3N6A, seg_pos={0 0}, upload=501665]: Uploading piece... Dec 29 11:42:08.892341088 (572h13m8.763424133s) [noise=true , address=145.79.227.37:4567 , piece_id=2KPQ6RJUY7CDVJH77OZFNXBNAJZ3SYXHAMDIKL3T3FYGJLV5IOAA, seg_pos={0 0}, upload=501665]: Uploading piece... Dec 29 11:42:08.892754735 (572h13m8.76383777s) [noise=true , address=45.91.116.12:4228 , piece_id=YRMBVP4NYM7KBY535RVO5FHLSG2TACPP2DQCVZPJQEYVBHRCZSBA, seg_pos={0 0}, upload=501665]: Uploading piece... Dec 29 11:42:08.893185946 (572h13m8.764268981s) [noise=true , address=145.79.227.13:4804 , piece_id=H26S6IITCESUK55VVVWMVX6DADQ4KDXUG2FFI24BBE66D6OCESSA, seg_pos={0 0}, upload=501665]: Uploading piece... Dec 29 11:42:08.893482320 (572h13m8.764565365s) [noise=true , address=45.91.116.10:4083 , piece_id=UMGJ7MJ7TS5V4DMQ6V2WVZWEGAAKSYKKARW6NDSMPI47ZSL7EZDA, seg_pos={0 0}, upload=501665]: Uploading piece... Dec 29 11:42:08.893832066 (572h13m8.764915111s) [noise=true , address=77.90.31.10:4520 , piece_id=6VKFN7VABP6AWDT4I3D7VT66GFX3FPMZ3D4W637JO2TSPUJPNYIQ, seg_pos={0 0}, upload=501665]: Uploading piece... Dec 29 11:42:08.895178530 (572h13m8.766261565s) [noise=true , address=145.79.227.11:4731 , piece_id=HDCENBYXK6KUUKB6SDALFX35XD7EMQZXNHFWZXEJ7QV4EMGPVEKQ, seg_pos={0 0}, upload=501665]: Uploading piece... Dec 29 11:42:08.895452783 (572h13m8.766535818s) [noise=true , address=45.91.116.4:4940 , piece_id=QXB6EIXYPS4H6IJVQZB4DRCPMBHNE5TWEJ4CAWXRS5SV2BEMKMUA, seg_pos={0 0}, upload=501665]: Uploading piece... Dec 29 11:42:08.896850394 (572h13m8.767933439s) [noise=true , address=145.79.227.51:4409 , piece_id=EALRNXBXHFBZ2O2TJVTQNCRUA44AUWWNMRDCZNFKRFKEGFORFGDA, seg_pos={0 0}, upload=501665]: Uploading piece... Dec 29 11:42:08.898488142 (572h13m8.769571187s) [noise=true , address=145.79.227.47:4266 , piece_id=UOUX3QHFXKSBRPVP6DLQ6SGZXEOCD7OBBSRZHJG3ID3UDGQCIK2A, seg_pos={0 0}, upload=501665]: Uploading piece... Dec 29 11:42:08.900181637 (572h13m8.771264682s) [noise=true , address=145.79.227.26:4655 , piece_id=ZIZTQMXLM3NILRV5OSBZDWIN3MPXDW2M2MPX3TRV5VIR7M2DINEQ, seg_pos={0 0}, upload=501665]: Uploading piece... Dec 29 11:42:08.901609756 (572h13m8.772692791s) [noise=true , address=45.91.116.7:4010 , piece_id=WXP2B2D5E4TWWIX2X7UNOLKRXGWMU6XIGKJQOKKVBKSAS46WMTCQ, seg_pos={0 0}, upload=501665]: Uploading piece... Dec 29 11:42:08.904516842 (572h13m8.775599877s) [noise=true , address=45.91.116.8:4045 , piece_id=2BTFNH4LWUKO25O3AQIG7HLE2FPQTV5J4C5IXU24AE5PL6OMZ7BA, seg_pos={0 0}, upload=501665]: Uploading piece... Dec 29 11:42:08.908672354 (572h13m8.779755389s) [noise=true , address=45.91.116.14:4190 , piece_id=3CGQCULIL4PHZ5AZLRGY5OX6LPWIPRJJVIQGGOG6CR4GSLNS6JTA, seg_pos={0 0}, upload=501665]: Uploading piece... Dec 29 11:42:10.595039157 (572h13m10.466122192s) [upload=501665]: Next returned nil segment Dec 29 11:42:10.595052943 (572h13m10.466135978s) [upload=501665]: Waiting for error group managing segments... Dec 29 11:42:10.878478558 (572h13m10.749561593s) [noise=true , address=145.79.227.37:4567 , piece_id=2KPQ6RJUY7CDVJH77OZFNXBNAJZ3SYXHAMDIKL3T3FYGJLV5IOAA, seg_pos={0 0}, upload=501665]: Done uploading piece. err: <nil> Dec 29 11:42:10.879098548 (572h13m10.750181583s) [noise=true , address=145.79.227.12:4767 , piece_id=VZRTNGI2WZA7DDNBUN4JXUUAAPNKTC4IKM6UYFUVTS2UR4QL3N6A, seg_pos={0 0}, upload=501665]: Done uploading piece. err: <nil> Dec 29 11:42:10.880102891 (572h13m10.751185936s) [noise=true , address=145.79.227.52:4446 , piece_id=SHAJ2N627KM2ETWH6AXYQ2N2JWAYKF5UEBHY7WD7EXJKX5XNQYJA, seg_pos={0 0}, upload=501665]: Done uploading piece. err: <nil> Dec 29 11:42:10.892515687 (572h13m10.763598732s) [noise=true , address=145.79.227.47:4266 , piece_id=UOUX3QHFXKSBRPVP6DLQ6SGZXEOCD7OBBSRZHJG3ID3UDGQCIK2A, seg_pos={0 0}, upload=501665]: Done uploading piece. err: <nil> Dec 29 11:42:10.899782433 (572h13m10.770865469s) [noise=true , address=145.79.227.51:4409 , piece_id=EALRNXBXHFBZ2O2TJVTQNCRUA44AUWWNMRDCZNFKRFKEGFORFGDA, seg_pos={0 0}, upload=501665]: Done uploading piece. err: <nil> Dec 29 11:42:10.904882705 (572h13m10.77596575s) [noise=true , address=145.79.227.26:4655 , piece_id=ZIZTQMXLM3NILRV5OSBZDWIN3MPXDW2M2MPX3TRV5VIR7M2DINEQ, seg_pos={0 0}, upload=501665]: Done uploading piece. err: <nil> Dec 29 11:42:10.917006801 (572h13m10.788089836s) [noise=true , address=145.79.227.13:4804 , piece_id=H26S6IITCESUK55VVVWMVX6DADQ4KDXUG2FFI24BBE66D6OCESSA, seg_pos={0 0}, upload=501665]: Done uploading piece. err: <nil> Dec 29 11:42:10.924353089 (572h13m10.795436124s) [noise=true , address=145.79.227.11:4731 , piece_id=HDCENBYXK6KUUKB6SDALFX35XD7EMQZXNHFWZXEJ7QV4EMGPVEKQ, seg_pos={0 0}, upload=501665]: Done uploading piece. err: <nil> Dec 29 11:42:11.096123005 (572h13m10.96720605s) [noise=true , address=45.91.116.10:4083 , piece_id=UMGJ7MJ7TS5V4DMQ6V2WVZWEGAAKSYKKARW6NDSMPI47ZSL7EZDA, seg_pos={0 0}, upload=501665]: Done uploading piece. err: <nil> Dec 29 11:42:11.146088486 (572h13m11.017171531s) [noise=true , address=45.91.116.14:4190 , piece_id=3CGQCULIL4PHZ5AZLRGY5OX6LPWIPRJJVIQGGOG6CR4GSLNS6JTA, seg_pos={0 0}, upload=501665]: Done uploading piece. err: <nil> Dec 29 11:42:11.151102814 (572h13m11.022185869s) [noise=true , address=45.91.116.9:4261 , piece_id=QJM5SK3I2GG5XOIC7DZC45XVGEFITZG6RLB37JALZCSNIIWPP2PQ, seg_pos={0 0}, upload=501665]: Done uploading piece. err: <nil> Dec 29 11:42:11.168477048 (572h13m11.039560083s) [noise=true , address=45.91.116.4:4940 , piece_id=QXB6EIXYPS4H6IJVQZB4DRCPMBHNE5TWEJ4CAWXRS5SV2BEMKMUA, seg_pos={0 0}, upload=501665]: Done uploading piece. err: <nil> Dec 29 11:42:11.278134922 (572h13m11.149217957s) [noise=true , address=45.91.116.7:4010 , piece_id=WXP2B2D5E4TWWIX2X7UNOLKRXGWMU6XIGKJQOKKVBKSAS46WMTCQ, seg_pos={0 0}, upload=501665]: Done uploading piece. err: <nil> Dec 29 11:42:11.278154560 (572h13m11.149237595s) [seg_pos={0 0}, upload=501665]: Segment reached optimal threshold of 13 pieces. Dec 29 11:42:11.278385871 (572h13m11.149468906s) [noise=true , address=45.91.116.8:4045 , piece_id=2BTFNH4LWUKO25O3AQIG7HLE2FPQTV5J4C5IXU24AE5PL6OMZ7BA, seg_pos={0 0}, upload=501665]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: write tcp 188.166.225.170:47196->45.91.116.8:4045: use of closed network connection; write tcp 188.166.225.170:47196->45.91.116.8:4045: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47196->45.91.116.8:4045: use of closed network connection Dec 29 11:42:11.278393455 (572h13m11.14947649s) [noise=true , address=45.91.116.8:4045 , piece_id=2BTFNH4LWUKO25O3AQIG7HLE2FPQTV5J4C5IXU24AE5PL6OMZ7BA, seg_pos={0 0}, upload=501665]: Long tail cancellation due to optimal threshold Dec 29 11:42:11.278523884 (572h13m11.149606929s) [noise=true , address=45.91.116.12:4228 , piece_id=YRMBVP4NYM7KBY535RVO5FHLSG2TACPP2DQCVZPJQEYVBHRCZSBA, seg_pos={0 0}, upload=501665]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 29 11:42:11.278543510 (572h13m11.149626545s) [noise=true , address=45.91.116.12:4228 , piece_id=YRMBVP4NYM7KBY535RVO5FHLSG2TACPP2DQCVZPJQEYVBHRCZSBA, seg_pos={0 0}, upload=501665]: Long tail cancellation due to optimal threshold Dec 29 11:42:11.278700450 (572h13m11.149783485s) [noise=true , address=77.90.31.10:4520 , piece_id=6VKFN7VABP6AWDT4I3D7VT66GFX3FPMZ3D4W637JO2TSPUJPNYIQ, seg_pos={0 0}, upload=501665]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Dec 29 11:42:11.278708224 (572h13m11.149791259s) [noise=true , address=77.90.31.10:4520 , piece_id=6VKFN7VABP6AWDT4I3D7VT66GFX3FPMZ3D4W637JO2TSPUJPNYIQ, seg_pos={0 0}, upload=501665]: Long tail cancellation due to optimal threshold Dec 29 11:42:11.278728603 (572h13m11.149811648s) [seg_pos={0 0}, upload=501665]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 11:42:11.278827341 (572h13m11.149910386s) [upload=501665]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 11:42:11.278833953 (572h13m11.149916988s) [upload=501665]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 11:42:11.571635311 (572h13m11.442718366s) [upload=501665]: Done uploading segments... Dec 29 11:42:11.571659377 (572h13m11.442742422s) [upload=501665]: Upload finished. err: <nil> Duration: 30.002405899s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [................X...............X..X........................] duration: 37.581350274s trace: https://jaeger.dev.storj.io/trace/7e7354d8023608a8 FAIL (2025-12-29 11:42:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== ) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480097317 (571h53m39.351180352s) [noise=true , address=77.90.31.21:4889 , piece_id=BR6Z55P3A7YJ5FJRDSZFUIF66FGIJQY7WCEQ7PL5DOYWL73OMKTQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480405053 (571h53m39.351488098s) [noise=true , address=77.90.31.16:4750 , piece_id=MVVZPJEGDCBSNVOB6CD2WUZGIJTL3VBHX2G4L3TVZ6JF5ZEJ5U6Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480712969 (571h53m39.351796014s) [noise=true , address=77.90.31.18:4834 , piece_id=BTWNGNQVEUEVV6HLBDM63SGNTKV4RDU4J4XFLG52WHKP53H3L5KA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481019453 (571h53m39.352102488s) [noise=true , address=145.79.227.30:4777 , piece_id=VSCXLXGIBSFE6E4OHLWNZIZ75PO7WQO2VFHUYHW7JUNH62KGGU7Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481029532 (571h53m39.352112567s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481559832 (571h53m39.352642877s) [noise=true , address=145.79.227.35:4515 , piece_id=L7LFZKZEZERUNX4HUNDNQ2PWHXNJRJ7OBFRNMFIICPZEUIBHGP7A, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481731919 (571h53m39.352814954s) [noise=true , address=64.49.13.26:4953 , piece_id=4YSW5Y3A6HYG3XJ2XAB7RGVHAOHMP7G3QIZYMYTSXVXYJ3ANLNXA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482078329 (571h53m39.353161364s) [noise=true , address=145.79.227.44:4205 , piece_id=UE4QSHDQBZ3XBWQ6I6MCRVTLMU7U6W6OZDA2ZGQCH6ZYRAGPD7WA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482099830 (571h53m39.353182865s) [noise=true , address=145.79.227.40:4067 , piece_id=WSYDRX5YJ6OCFW52ZVMM6WZYNRFUXMR4XJ6C5727BOFMR4RRH33Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482434487 (571h53m39.353517532s) [noise=true , address=145.79.227.3:4164 , piece_id=U6F7VDPOH7K6R5EH5UUXHFP55GXOEXO5BLBJHZRTITPDQI4LP7KQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482480845 (571h53m39.35356388s) [noise=true , address=64.49.13.140:4516 , piece_id=4GUYOYOAEASUK2W25S3NILWIIIHGOZX76L4EGFLUUUEDWRALPP2A, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482841442 (571h53m39.353924477s) [noise=true , address=145.79.227.34:4910 , piece_id=ZH4DCNXET2Q6VRAUH3NQ3KKSV4OYI575X7YS72XD236EEOMOG3LA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483163645 (571h53m39.35424668s) [noise=true , address=145.79.227.29:4974 , piece_id=KEKREOD2B35XQNI45JBHE2SZZC3MQQBQUR5I6WFDAQYJQGJBXSMA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483532478 (571h53m39.354615513s) [noise=true , address=77.90.31.17:4769 , piece_id=3MKUMOQIUVERKQLLIFX52XQPTQ5LSFMHIPD5YW2YTO5YEFR36IYA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483914596 (571h53m39.354997631s) [noise=true , address=145.79.227.13:4807 , piece_id=VKXB5MWAG2YYH2AWWJEGTLBZ2QUPLI25VTHI6AMMALSM23Z3AWZQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:40.761721106 (571h53m40.632804141s) [upload=501384]: Next returned nil segment Dec 29 11:22:40.761733349 (571h53m40.632816384s) [upload=501384]: Waiting for error group managing segments... Dec 29 11:22:41.121222862 (571h53m40.992305897s) [noise=true , address=64.49.13.140:4516 , piece_id=4GUYOYOAEASUK2W25S3NILWIIIHGOZX76L4EGFLUUUEDWRALPP2A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.197311646 (571h53m41.068394691s) [noise=true , address=64.49.13.26:4953 , piece_id=4YSW5Y3A6HYG3XJ2XAB7RGVHAOHMP7G3QIZYMYTSXVXYJ3ANLNXA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.347247568 (571h53m41.218330613s) [noise=true , address=145.79.227.13:4807 , piece_id=VKXB5MWAG2YYH2AWWJEGTLBZ2QUPLI25VTHI6AMMALSM23Z3AWZQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.347917313 (571h53m41.219000348s) [noise=true , address=145.79.227.3:4164 , piece_id=U6F7VDPOH7K6R5EH5UUXHFP55GXOEXO5BLBJHZRTITPDQI4LP7KQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.348510613 (571h53m41.219593648s) [noise=true , address=145.79.227.29:4974 , piece_id=KEKREOD2B35XQNI45JBHE2SZZC3MQQBQUR5I6WFDAQYJQGJBXSMA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.503950985 (571h53m41.37503402s) [noise=true , address=77.90.31.17:4769 , piece_id=3MKUMOQIUVERKQLLIFX52XQPTQ5LSFMHIPD5YW2YTO5YEFR36IYA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.561352025 (571h53m41.43243506s) [noise=true , address=77.90.31.18:4834 , piece_id=BTWNGNQVEUEVV6HLBDM63SGNTKV4RDU4J4XFLG52WHKP53H3L5KA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.622447380 (571h53m41.493530425s) [noise=true , address=77.90.31.16:4750 , piece_id=MVVZPJEGDCBSNVOB6CD2WUZGIJTL3VBHX2G4L3TVZ6JF5ZEJ5U6Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.645494685 (571h53m41.51657773s) [noise=true , address=77.90.31.21:4889 , piece_id=BR6Z55P3A7YJ5FJRDSZFUIF66FGIJQY7WCEQ7PL5DOYWL73OMKTQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.247930045 (571h53m42.11901308s) [noise=true , address=145.79.227.30:4777 , piece_id=VSCXLXGIBSFE6E4OHLWNZIZ75PO7WQO2VFHUYHW7JUNH62KGGU7Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.262221890 (571h53m42.133304925s) [noise=true , address=145.79.227.44:4205 , piece_id=UE4QSHDQBZ3XBWQ6I6MCRVTLMU7U6W6OZDA2ZGQCH6ZYRAGPD7WA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.295598123 (571h53m42.166681158s) [noise=true , address=145.79.227.35:4515 , piece_id=L7LFZKZEZERUNX4HUNDNQ2PWHXNJRJ7OBFRNMFIICPZEUIBHGP7A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.307481351 (571h53m42.178564386s) [noise=true , address=145.79.227.34:4910 , piece_id=ZH4DCNXET2Q6VRAUH3NQ3KKSV4OYI575X7YS72XD236EEOMOG3LA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.325775906 (571h53m42.196858941s) [noise=true , address=145.79.227.40:4067 , piece_id=WSYDRX5YJ6OCFW52ZVMM6WZYNRFUXMR4XJ6C5727BOFMR4RRH33Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.325865978 (571h53m42.196949023s) [seg_pos={0 0}, upload=501384]: Segment reached optimal threshold of 14 pieces. Dec 29 11:22:42.326137074 (571h53m42.197220109s) [noise=true , address=64.49.13.169:4708 , piece_id=B2X2RWRELXWJPF6AVJJZCNU46LM5ASGRE4EWGPKL3SCLQ4VKUBDQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection Dec 29 11:22:42.326153074 (571h53m42.197236109s) [noise=true , address=64.49.13.169:4708 , piece_id=B2X2RWRELXWJPF6AVJJZCNU46LM5ASGRE4EWGPKL3SCLQ4VKUBDQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326193401 (571h53m42.197276446s) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12h5esfg6mw2c8y3AVpeFBkL5FfLAwYdjhcnQ9CJ8fjh61WLYjx): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326205705 (571h53m42.19728874s) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326407739 (571h53m42.197490784s) [noise=true , address=77.90.31.10:4524 , piece_id=Q2JGEUV5EDFTVY2CSNPZOVJIIIAFBA3IHA7S5N7YZRH3E7TSTE5A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uN7CznoK4AD8FqFkNUndawdGJ9rmjAe7oNnjJkV844ScY87qe): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326415374 (571h53m42.197498419s) [noise=true , address=77.90.31.10:4524 , piece_id=Q2JGEUV5EDFTVY2CSNPZOVJIIIAFBA3IHA7S5N7YZRH3E7TSTE5A, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326505105 (571h53m42.19758815s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tnTEYvqJ6PB2DsuXA28sEuEiGj11StA4qdhcYH1kuoaMkTVdY): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326512188 (571h53m42.197595223s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326532357 (571h53m42.197615392s) [seg_pos={0 0}, upload=501384]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 11:22:42.326671562 (571h53m42.197754597s) [upload=501384]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 11:22:42.326677843 (571h53m42.197760879s) [upload=501384]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 11:22:42.610391438 (571h53m42.481474493s) [upload=501384]: Done uploading segments... Dec 29 11:22:42.610416826 (571h53m42.481499871s) [upload=501384]: Upload finished. err: <nil> Duration: 37.581376304s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 21 out of 60 is failed (threshold=6) [......X....X.......X..X...XX..XX......X.X.XX.X.XX..X..XX.XXX] duration: 5.00100275s trace: https://jaeger.dev.storj.io/trace/17d602297ef5b5f9 FAIL (2025-12-29 11:42:07 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001026474s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [X.....X.X...X.X.X.X.XX..XX..X..XXX.X.......X.X..X.XXX...XX.X] duration: 30.004718397s trace: https://jaeger.dev.storj.io/trace/3bb887713ece719b FAIL (2025-12-29 11:41:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 11:41:19.452732839 (572h12m19.323815874s) [upload=501652]: Starting upload Dec 29 11:41:19.452799064 (572h12m19.323882099s) [upload=501652]: Done starting upload Dec 29 11:41:19.452889607 (572h12m19.323972642s) [upload=501652]: Uploading segments... Dec 29 11:41:19.453009995 (572h12m19.32409307s) [upload=501652]: Got next segment. Inline: false Dec 29 11:41:19.453164620 (572h12m19.324247665s) [upload=501652]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 11:41:19.453172906 (572h12m19.324255951s) [upload=501652]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 11:41:19.698948089 (572h12m19.570031134s) [seg_pos={0 0}, upload=501652]: Begin upload segment... Dec 29 11:41:19.699000558 (572h12m19.570083603s) [seg_pos={0 0}, upload=501652]: Done begin upload segment. Dec 29 11:41:19.699044111 (572h12m19.570127146s) [noise=true , address=45.91.116.7:4011 , piece_id=H7JCCSIORPN2N2G4ZLCS3UATACLSVE2JA4WDIZL5WXI6ANWXVPTQ, seg_pos={0 0}, upload=501652]: Uploading piece... Dec 29 11:41:19.699073247 (572h12m19.570156282s) [noise=true , address=45.91.116.12:4225 , piece_id=I6A2OXIOHCPOHBR36EDCSN6VG2XXF3ECGZSZNKDCHD4D2MFA6TIA, seg_pos={0 0}, upload=501652]: Uploading piece... Dec 29 11:41:19.699572317 (572h12m19.570655352s) [noise=true , address=77.90.31.11:4556 , piece_id=UYPTL2JSX4FYPRFAD6SALSD4K5HE5SQ2ZRXYQN6OHKN56PPYOBYA, seg_pos={0 0}, upload=501652]: Uploading piece... Dec 29 11:41:19.699864193 (572h12m19.570947238s) [noise=true , address=145.79.227.6:4257 , piece_id=R5DHSZ2GGUUHPZ7KKEPWV3IOKZ6R7EV54VO43Z6H3WJ4BX3LQODA, seg_pos={0 0}, upload=501652]: Uploading piece... Dec 29 11:41:19.700328216 (572h12m19.571411251s) [noise=true , address=145.79.227.17:4971 , piece_id=DZUXY7632ARGYJPSPO56DPGK4DDOQ5IHFPGAOSJLBWJVZ4WNPEZA, seg_pos={0 0}, upload=501652]: Uploading piece... Dec 29 11:41:19.700687861 (572h12m19.571770896s) [noise=true , address=145.79.227.14:4837 , piece_id=6U2HDAWT2K25Y75YI4IS4UA6NKPSEQY5QYWRV6NISZ6GJALRUDXA, seg_pos={0 0}, upload=501652]: Uploading piece... Dec 29 11:41:19.700960771 (572h12m19.572043806s) [noise=true , address=145.79.227.52:4445 , piece_id=D46AHN6PBRHSJ2F6IQA6I2GELK2GGNDCAW5DQ6CKXG4W2Z5WA5AQ, seg_pos={0 0}, upload=501652]: Uploading piece... Dec 29 11:41:19.701367796 (572h12m19.572450841s) [noise=true , address=45.91.116.6:4973 , piece_id=45OSKBLMB6BGFKU3MCETVIAWTDQMH77U3TYHUES35WT4LVFVZ53Q, seg_pos={0 0}, upload=501652]: Uploading piece... Dec 29 11:41:19.701707061 (572h12m19.572790096s) [noise=true , address=145.79.227.48:4303 , piece_id=7FZCZ3LSR7MQT7VIDJJQ5L5WR7KZDJM3WOPXME6XGISB7JEEPWOA, seg_pos={0 0}, upload=501652]: Uploading piece... Dec 29 11:41:19.703706990 (572h12m19.574790035s) [noise=true , address=77.90.31.14:4662 , piece_id=N6HIF5VGU4SHQGGF3M2K5HPEVW4PPBXTVVNWUQ2GUYNP5I53SHXA, seg_pos={0 0}, upload=501652]: Uploading piece... Dec 29 11:41:19.703870542 (572h12m19.574953587s) [noise=true , address=145.79.227.44:4195 , piece_id=NDSXMHUAIBEFPJVHIJ6HCLNATAA7T6B4UCTP4UOHDV2JWNXDGQIQ, seg_pos={0 0}, upload=501652]: Uploading piece... Dec 29 11:41:19.705138886 (572h12m19.576221921s) [noise=true , address=45.91.116.14:4189 , piece_id=46GHE7IJC7M2H5ZKRHAKPLZ3ECSSSIWTUDJCBBZRZI6W4FKNX7XA, seg_pos={0 0}, upload=501652]: Uploading piece... Dec 29 11:41:19.706395098 (572h12m19.577478133s) [noise=true , address=77.90.31.19:4842 , piece_id=6EDCIQKOLFQD4PSBOZP77DQ75TCKYFC4CUAG5FJJZMN55ZMETZDA, seg_pos={0 0}, upload=501652]: Uploading piece... Dec 29 11:41:19.707765237 (572h12m19.578848272s) [noise=true , address=145.79.227.20:4437 , piece_id=ABXV75SMM6VWUO3CY2VJ4F3IFBGNUFENXW3QPY64HBJIH2MXMF5Q, seg_pos={0 0}, upload=501652]: Uploading piece... Dec 29 11:41:19.709279761 (572h12m19.580362796s) [noise=true , address=145.79.227.46:4601 , piece_id=NYKGBXXQ6GGFOYUXP563XD5V2Z3VC7VXFPINOOCOM2YMI6LVIVUQ, seg_pos={0 0}, upload=501652]: Uploading piece... Dec 29 11:41:19.710750722 (572h12m19.581833757s) [noise=true , address=145.79.227.12:4767 , piece_id=HU567SALHW5BKR4RTQX7AMQMQMDDURNCDFLRRVJ3QN46CTXD2TSA, seg_pos={0 0}, upload=501652]: Uploading piece... Dec 29 11:41:21.406808216 (572h12m21.277891261s) [upload=501652]: Next returned nil segment Dec 29 11:41:21.406824096 (572h12m21.277907131s) [upload=501652]: Waiting for error group managing segments... Dec 29 11:41:21.691988982 (572h12m21.563072017s) [noise=true , address=145.79.227.6:4257 , piece_id=R5DHSZ2GGUUHPZ7KKEPWV3IOKZ6R7EV54VO43Z6H3WJ4BX3LQODA, seg_pos={0 0}, upload=501652]: Done uploading piece. err: <nil> Dec 29 11:41:21.692371240 (572h12m21.563454285s) [noise=true , address=145.79.227.20:4437 , piece_id=ABXV75SMM6VWUO3CY2VJ4F3IFBGNUFENXW3QPY64HBJIH2MXMF5Q, seg_pos={0 0}, upload=501652]: Done uploading piece. err: <nil> Dec 29 11:41:21.692599384 (572h12m21.563682429s) [noise=true , address=145.79.227.17:4971 , piece_id=DZUXY7632ARGYJPSPO56DPGK4DDOQ5IHFPGAOSJLBWJVZ4WNPEZA, seg_pos={0 0}, upload=501652]: Done uploading piece. err: <nil> Dec 29 11:41:21.694270697 (572h12m21.565353742s) [noise=true , address=145.79.227.52:4445 , piece_id=D46AHN6PBRHSJ2F6IQA6I2GELK2GGNDCAW5DQ6CKXG4W2Z5WA5AQ, seg_pos={0 0}, upload=501652]: Done uploading piece. err: <nil> Dec 29 11:41:21.694976601 (572h12m21.566059646s) [noise=true , address=145.79.227.46:4601 , piece_id=NYKGBXXQ6GGFOYUXP563XD5V2Z3VC7VXFPINOOCOM2YMI6LVIVUQ, seg_pos={0 0}, upload=501652]: Done uploading piece. err: <nil> Dec 29 11:41:21.703111942 (572h12m21.574194977s) [noise=true , address=145.79.227.12:4767 , piece_id=HU567SALHW5BKR4RTQX7AMQMQMDDURNCDFLRRVJ3QN46CTXD2TSA, seg_pos={0 0}, upload=501652]: Done uploading piece. err: <nil> Dec 29 11:41:21.703497536 (572h12m21.574580581s) [noise=true , address=145.79.227.14:4837 , piece_id=6U2HDAWT2K25Y75YI4IS4UA6NKPSEQY5QYWRV6NISZ6GJALRUDXA, seg_pos={0 0}, upload=501652]: Done uploading piece. err: <nil> Dec 29 11:41:21.716390536 (572h12m21.587473571s) [noise=true , address=145.79.227.48:4303 , piece_id=7FZCZ3LSR7MQT7VIDJJQ5L5WR7KZDJM3WOPXME6XGISB7JEEPWOA, seg_pos={0 0}, upload=501652]: Done uploading piece. err: <nil> Dec 29 11:41:22.023501140 (572h12m21.894584185s) [noise=true , address=77.90.31.11:4556 , piece_id=UYPTL2JSX4FYPRFAD6SALSD4K5HE5SQ2ZRXYQN6OHKN56PPYOBYA, seg_pos={0 0}, upload=501652]: Done uploading piece. err: <nil> Dec 29 11:41:22.029183751 (572h12m21.900266796s) [noise=true , address=45.91.116.7:4011 , piece_id=H7JCCSIORPN2N2G4ZLCS3UATACLSVE2JA4WDIZL5WXI6ANWXVPTQ, seg_pos={0 0}, upload=501652]: Done uploading piece. err: <nil> Dec 29 11:41:22.040569852 (572h12m21.911652887s) [noise=true , address=45.91.116.14:4189 , piece_id=46GHE7IJC7M2H5ZKRHAKPLZ3ECSSSIWTUDJCBBZRZI6W4FKNX7XA, seg_pos={0 0}, upload=501652]: Done uploading piece. err: <nil> Dec 29 11:41:22.046380836 (572h12m21.917463871s) [noise=true , address=77.90.31.19:4842 , piece_id=6EDCIQKOLFQD4PSBOZP77DQ75TCKYFC4CUAG5FJJZMN55ZMETZDA, seg_pos={0 0}, upload=501652]: Done uploading piece. err: <nil> Dec 29 11:41:22.256379786 (572h12m22.127462821s) [noise=true , address=45.91.116.12:4225 , piece_id=I6A2OXIOHCPOHBR36EDCSN6VG2XXF3ECGZSZNKDCHD4D2MFA6TIA, seg_pos={0 0}, upload=501652]: Done uploading piece. err: <nil> Dec 29 11:41:22.256403772 (572h12m22.127486807s) [seg_pos={0 0}, upload=501652]: Segment reached optimal threshold of 13 pieces. Dec 29 11:41:22.256630984 (572h12m22.127714029s) [noise=true , address=77.90.31.14:4662 , piece_id=N6HIF5VGU4SHQGGF3M2K5HPEVW4PPBXTVVNWUQ2GUYNP5I53SHXA, seg_pos={0 0}, upload=501652]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Dec 29 11:41:22.256640954 (572h12m22.127724009s) [noise=true , address=77.90.31.14:4662 , piece_id=N6HIF5VGU4SHQGGF3M2K5HPEVW4PPBXTVVNWUQ2GUYNP5I53SHXA, seg_pos={0 0}, upload=501652]: Long tail cancellation due to optimal threshold Dec 29 11:41:22.256815275 (572h12m22.127898311s) [noise=true , address=145.79.227.44:4195 , piece_id=NDSXMHUAIBEFPJVHIJ6HCLNATAA7T6B4UCTP4UOHDV2JWNXDGQIQ, seg_pos={0 0}, upload=501652]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 29 11:41:22.256836105 (572h12m22.12791915s) [noise=true , address=145.79.227.44:4195 , piece_id=NDSXMHUAIBEFPJVHIJ6HCLNATAA7T6B4UCTP4UOHDV2JWNXDGQIQ, seg_pos={0 0}, upload=501652]: Long tail cancellation due to optimal threshold Dec 29 11:41:22.256891120 (572h12m22.127974155s) [noise=true , address=45.91.116.6:4973 , piece_id=45OSKBLMB6BGFKU3MCETVIAWTDQMH77U3TYHUES35WT4LVFVZ53Q, seg_pos={0 0}, upload=501652]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Dec 29 11:41:22.256898924 (572h12m22.127981969s) [noise=true , address=45.91.116.6:4973 , piece_id=45OSKBLMB6BGFKU3MCETVIAWTDQMH77U3TYHUES35WT4LVFVZ53Q, seg_pos={0 0}, upload=501652]: Long tail cancellation due to optimal threshold Dec 29 11:41:22.256965692 (572h12m22.128048727s) [seg_pos={0 0}, upload=501652]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 11:41:22.257151766 (572h12m22.128234831s) [upload=501652]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 11:41:22.257161144 (572h12m22.128244179s) [upload=501652]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 11:41:22.543128058 (572h12m22.414211093s) [upload=501652]: Done uploading segments... Dec 29 11:41:22.543145881 (572h12m22.414228926s) [upload=501652]: Upload finished. err: <nil> Duration: 30.004744236s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 20 out of 60 is failed (threshold=6) [.......X....X.......X..X...XX..XX......X.X.XX.X.XX..X..XX.XX] duration: 5.001295494s trace: https://jaeger.dev.storj.io/trace/6e72248e72ad44be FAIL (2025-12-29 11:41:30 UTC) orbiter warning: context deadline exceeded Duration: 5.001309721s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................X...............X..X.......................] duration: 45.678365731s trace: https://jaeger.dev.storj.io/trace/7e7354d8023608a8 FAIL (2025-12-29 11:41:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== ) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480097317 (571h53m39.351180352s) [noise=true , address=77.90.31.21:4889 , piece_id=BR6Z55P3A7YJ5FJRDSZFUIF66FGIJQY7WCEQ7PL5DOYWL73OMKTQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480405053 (571h53m39.351488098s) [noise=true , address=77.90.31.16:4750 , piece_id=MVVZPJEGDCBSNVOB6CD2WUZGIJTL3VBHX2G4L3TVZ6JF5ZEJ5U6Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480712969 (571h53m39.351796014s) [noise=true , address=77.90.31.18:4834 , piece_id=BTWNGNQVEUEVV6HLBDM63SGNTKV4RDU4J4XFLG52WHKP53H3L5KA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481019453 (571h53m39.352102488s) [noise=true , address=145.79.227.30:4777 , piece_id=VSCXLXGIBSFE6E4OHLWNZIZ75PO7WQO2VFHUYHW7JUNH62KGGU7Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481029532 (571h53m39.352112567s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481559832 (571h53m39.352642877s) [noise=true , address=145.79.227.35:4515 , piece_id=L7LFZKZEZERUNX4HUNDNQ2PWHXNJRJ7OBFRNMFIICPZEUIBHGP7A, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481731919 (571h53m39.352814954s) [noise=true , address=64.49.13.26:4953 , piece_id=4YSW5Y3A6HYG3XJ2XAB7RGVHAOHMP7G3QIZYMYTSXVXYJ3ANLNXA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482078329 (571h53m39.353161364s) [noise=true , address=145.79.227.44:4205 , piece_id=UE4QSHDQBZ3XBWQ6I6MCRVTLMU7U6W6OZDA2ZGQCH6ZYRAGPD7WA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482099830 (571h53m39.353182865s) [noise=true , address=145.79.227.40:4067 , piece_id=WSYDRX5YJ6OCFW52ZVMM6WZYNRFUXMR4XJ6C5727BOFMR4RRH33Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482434487 (571h53m39.353517532s) [noise=true , address=145.79.227.3:4164 , piece_id=U6F7VDPOH7K6R5EH5UUXHFP55GXOEXO5BLBJHZRTITPDQI4LP7KQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482480845 (571h53m39.35356388s) [noise=true , address=64.49.13.140:4516 , piece_id=4GUYOYOAEASUK2W25S3NILWIIIHGOZX76L4EGFLUUUEDWRALPP2A, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482841442 (571h53m39.353924477s) [noise=true , address=145.79.227.34:4910 , piece_id=ZH4DCNXET2Q6VRAUH3NQ3KKSV4OYI575X7YS72XD236EEOMOG3LA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483163645 (571h53m39.35424668s) [noise=true , address=145.79.227.29:4974 , piece_id=KEKREOD2B35XQNI45JBHE2SZZC3MQQBQUR5I6WFDAQYJQGJBXSMA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483532478 (571h53m39.354615513s) [noise=true , address=77.90.31.17:4769 , piece_id=3MKUMOQIUVERKQLLIFX52XQPTQ5LSFMHIPD5YW2YTO5YEFR36IYA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483914596 (571h53m39.354997631s) [noise=true , address=145.79.227.13:4807 , piece_id=VKXB5MWAG2YYH2AWWJEGTLBZ2QUPLI25VTHI6AMMALSM23Z3AWZQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:40.761721106 (571h53m40.632804141s) [upload=501384]: Next returned nil segment Dec 29 11:22:40.761733349 (571h53m40.632816384s) [upload=501384]: Waiting for error group managing segments... Dec 29 11:22:41.121222862 (571h53m40.992305897s) [noise=true , address=64.49.13.140:4516 , piece_id=4GUYOYOAEASUK2W25S3NILWIIIHGOZX76L4EGFLUUUEDWRALPP2A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.197311646 (571h53m41.068394691s) [noise=true , address=64.49.13.26:4953 , piece_id=4YSW5Y3A6HYG3XJ2XAB7RGVHAOHMP7G3QIZYMYTSXVXYJ3ANLNXA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.347247568 (571h53m41.218330613s) [noise=true , address=145.79.227.13:4807 , piece_id=VKXB5MWAG2YYH2AWWJEGTLBZ2QUPLI25VTHI6AMMALSM23Z3AWZQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.347917313 (571h53m41.219000348s) [noise=true , address=145.79.227.3:4164 , piece_id=U6F7VDPOH7K6R5EH5UUXHFP55GXOEXO5BLBJHZRTITPDQI4LP7KQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.348510613 (571h53m41.219593648s) [noise=true , address=145.79.227.29:4974 , piece_id=KEKREOD2B35XQNI45JBHE2SZZC3MQQBQUR5I6WFDAQYJQGJBXSMA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.503950985 (571h53m41.37503402s) [noise=true , address=77.90.31.17:4769 , piece_id=3MKUMOQIUVERKQLLIFX52XQPTQ5LSFMHIPD5YW2YTO5YEFR36IYA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.561352025 (571h53m41.43243506s) [noise=true , address=77.90.31.18:4834 , piece_id=BTWNGNQVEUEVV6HLBDM63SGNTKV4RDU4J4XFLG52WHKP53H3L5KA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.622447380 (571h53m41.493530425s) [noise=true , address=77.90.31.16:4750 , piece_id=MVVZPJEGDCBSNVOB6CD2WUZGIJTL3VBHX2G4L3TVZ6JF5ZEJ5U6Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.645494685 (571h53m41.51657773s) [noise=true , address=77.90.31.21:4889 , piece_id=BR6Z55P3A7YJ5FJRDSZFUIF66FGIJQY7WCEQ7PL5DOYWL73OMKTQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.247930045 (571h53m42.11901308s) [noise=true , address=145.79.227.30:4777 , piece_id=VSCXLXGIBSFE6E4OHLWNZIZ75PO7WQO2VFHUYHW7JUNH62KGGU7Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.262221890 (571h53m42.133304925s) [noise=true , address=145.79.227.44:4205 , piece_id=UE4QSHDQBZ3XBWQ6I6MCRVTLMU7U6W6OZDA2ZGQCH6ZYRAGPD7WA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.295598123 (571h53m42.166681158s) [noise=true , address=145.79.227.35:4515 , piece_id=L7LFZKZEZERUNX4HUNDNQ2PWHXNJRJ7OBFRNMFIICPZEUIBHGP7A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.307481351 (571h53m42.178564386s) [noise=true , address=145.79.227.34:4910 , piece_id=ZH4DCNXET2Q6VRAUH3NQ3KKSV4OYI575X7YS72XD236EEOMOG3LA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.325775906 (571h53m42.196858941s) [noise=true , address=145.79.227.40:4067 , piece_id=WSYDRX5YJ6OCFW52ZVMM6WZYNRFUXMR4XJ6C5727BOFMR4RRH33Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.325865978 (571h53m42.196949023s) [seg_pos={0 0}, upload=501384]: Segment reached optimal threshold of 14 pieces. Dec 29 11:22:42.326137074 (571h53m42.197220109s) [noise=true , address=64.49.13.169:4708 , piece_id=B2X2RWRELXWJPF6AVJJZCNU46LM5ASGRE4EWGPKL3SCLQ4VKUBDQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection Dec 29 11:22:42.326153074 (571h53m42.197236109s) [noise=true , address=64.49.13.169:4708 , piece_id=B2X2RWRELXWJPF6AVJJZCNU46LM5ASGRE4EWGPKL3SCLQ4VKUBDQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326193401 (571h53m42.197276446s) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12h5esfg6mw2c8y3AVpeFBkL5FfLAwYdjhcnQ9CJ8fjh61WLYjx): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326205705 (571h53m42.19728874s) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326407739 (571h53m42.197490784s) [noise=true , address=77.90.31.10:4524 , piece_id=Q2JGEUV5EDFTVY2CSNPZOVJIIIAFBA3IHA7S5N7YZRH3E7TSTE5A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uN7CznoK4AD8FqFkNUndawdGJ9rmjAe7oNnjJkV844ScY87qe): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326415374 (571h53m42.197498419s) [noise=true , address=77.90.31.10:4524 , piece_id=Q2JGEUV5EDFTVY2CSNPZOVJIIIAFBA3IHA7S5N7YZRH3E7TSTE5A, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326505105 (571h53m42.19758815s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tnTEYvqJ6PB2DsuXA28sEuEiGj11StA4qdhcYH1kuoaMkTVdY): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326512188 (571h53m42.197595223s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326532357 (571h53m42.197615392s) [seg_pos={0 0}, upload=501384]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 11:22:42.326671562 (571h53m42.197754597s) [upload=501384]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 11:22:42.326677843 (571h53m42.197760879s) [upload=501384]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 11:22:42.610391438 (571h53m42.481474493s) [upload=501384]: Done uploading segments... Dec 29 11:22:42.610416826 (571h53m42.481499871s) [upload=501384]: Upload finished. err: <nil> Duration: 45.678392832s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [..................X...............X..X......................] duration: 37.663809033s trace: https://jaeger.dev.storj.io/trace/7e7354d8023608a8 FAIL (2025-12-29 11:40:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== ) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480097317 (571h53m39.351180352s) [noise=true , address=77.90.31.21:4889 , piece_id=BR6Z55P3A7YJ5FJRDSZFUIF66FGIJQY7WCEQ7PL5DOYWL73OMKTQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480405053 (571h53m39.351488098s) [noise=true , address=77.90.31.16:4750 , piece_id=MVVZPJEGDCBSNVOB6CD2WUZGIJTL3VBHX2G4L3TVZ6JF5ZEJ5U6Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480712969 (571h53m39.351796014s) [noise=true , address=77.90.31.18:4834 , piece_id=BTWNGNQVEUEVV6HLBDM63SGNTKV4RDU4J4XFLG52WHKP53H3L5KA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481019453 (571h53m39.352102488s) [noise=true , address=145.79.227.30:4777 , piece_id=VSCXLXGIBSFE6E4OHLWNZIZ75PO7WQO2VFHUYHW7JUNH62KGGU7Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481029532 (571h53m39.352112567s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481559832 (571h53m39.352642877s) [noise=true , address=145.79.227.35:4515 , piece_id=L7LFZKZEZERUNX4HUNDNQ2PWHXNJRJ7OBFRNMFIICPZEUIBHGP7A, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481731919 (571h53m39.352814954s) [noise=true , address=64.49.13.26:4953 , piece_id=4YSW5Y3A6HYG3XJ2XAB7RGVHAOHMP7G3QIZYMYTSXVXYJ3ANLNXA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482078329 (571h53m39.353161364s) [noise=true , address=145.79.227.44:4205 , piece_id=UE4QSHDQBZ3XBWQ6I6MCRVTLMU7U6W6OZDA2ZGQCH6ZYRAGPD7WA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482099830 (571h53m39.353182865s) [noise=true , address=145.79.227.40:4067 , piece_id=WSYDRX5YJ6OCFW52ZVMM6WZYNRFUXMR4XJ6C5727BOFMR4RRH33Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482434487 (571h53m39.353517532s) [noise=true , address=145.79.227.3:4164 , piece_id=U6F7VDPOH7K6R5EH5UUXHFP55GXOEXO5BLBJHZRTITPDQI4LP7KQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482480845 (571h53m39.35356388s) [noise=true , address=64.49.13.140:4516 , piece_id=4GUYOYOAEASUK2W25S3NILWIIIHGOZX76L4EGFLUUUEDWRALPP2A, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482841442 (571h53m39.353924477s) [noise=true , address=145.79.227.34:4910 , piece_id=ZH4DCNXET2Q6VRAUH3NQ3KKSV4OYI575X7YS72XD236EEOMOG3LA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483163645 (571h53m39.35424668s) [noise=true , address=145.79.227.29:4974 , piece_id=KEKREOD2B35XQNI45JBHE2SZZC3MQQBQUR5I6WFDAQYJQGJBXSMA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483532478 (571h53m39.354615513s) [noise=true , address=77.90.31.17:4769 , piece_id=3MKUMOQIUVERKQLLIFX52XQPTQ5LSFMHIPD5YW2YTO5YEFR36IYA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483914596 (571h53m39.354997631s) [noise=true , address=145.79.227.13:4807 , piece_id=VKXB5MWAG2YYH2AWWJEGTLBZ2QUPLI25VTHI6AMMALSM23Z3AWZQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:40.761721106 (571h53m40.632804141s) [upload=501384]: Next returned nil segment Dec 29 11:22:40.761733349 (571h53m40.632816384s) [upload=501384]: Waiting for error group managing segments... Dec 29 11:22:41.121222862 (571h53m40.992305897s) [noise=true , address=64.49.13.140:4516 , piece_id=4GUYOYOAEASUK2W25S3NILWIIIHGOZX76L4EGFLUUUEDWRALPP2A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.197311646 (571h53m41.068394691s) [noise=true , address=64.49.13.26:4953 , piece_id=4YSW5Y3A6HYG3XJ2XAB7RGVHAOHMP7G3QIZYMYTSXVXYJ3ANLNXA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.347247568 (571h53m41.218330613s) [noise=true , address=145.79.227.13:4807 , piece_id=VKXB5MWAG2YYH2AWWJEGTLBZ2QUPLI25VTHI6AMMALSM23Z3AWZQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.347917313 (571h53m41.219000348s) [noise=true , address=145.79.227.3:4164 , piece_id=U6F7VDPOH7K6R5EH5UUXHFP55GXOEXO5BLBJHZRTITPDQI4LP7KQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.348510613 (571h53m41.219593648s) [noise=true , address=145.79.227.29:4974 , piece_id=KEKREOD2B35XQNI45JBHE2SZZC3MQQBQUR5I6WFDAQYJQGJBXSMA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.503950985 (571h53m41.37503402s) [noise=true , address=77.90.31.17:4769 , piece_id=3MKUMOQIUVERKQLLIFX52XQPTQ5LSFMHIPD5YW2YTO5YEFR36IYA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.561352025 (571h53m41.43243506s) [noise=true , address=77.90.31.18:4834 , piece_id=BTWNGNQVEUEVV6HLBDM63SGNTKV4RDU4J4XFLG52WHKP53H3L5KA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.622447380 (571h53m41.493530425s) [noise=true , address=77.90.31.16:4750 , piece_id=MVVZPJEGDCBSNVOB6CD2WUZGIJTL3VBHX2G4L3TVZ6JF5ZEJ5U6Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.645494685 (571h53m41.51657773s) [noise=true , address=77.90.31.21:4889 , piece_id=BR6Z55P3A7YJ5FJRDSZFUIF66FGIJQY7WCEQ7PL5DOYWL73OMKTQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.247930045 (571h53m42.11901308s) [noise=true , address=145.79.227.30:4777 , piece_id=VSCXLXGIBSFE6E4OHLWNZIZ75PO7WQO2VFHUYHW7JUNH62KGGU7Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.262221890 (571h53m42.133304925s) [noise=true , address=145.79.227.44:4205 , piece_id=UE4QSHDQBZ3XBWQ6I6MCRVTLMU7U6W6OZDA2ZGQCH6ZYRAGPD7WA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.295598123 (571h53m42.166681158s) [noise=true , address=145.79.227.35:4515 , piece_id=L7LFZKZEZERUNX4HUNDNQ2PWHXNJRJ7OBFRNMFIICPZEUIBHGP7A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.307481351 (571h53m42.178564386s) [noise=true , address=145.79.227.34:4910 , piece_id=ZH4DCNXET2Q6VRAUH3NQ3KKSV4OYI575X7YS72XD236EEOMOG3LA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.325775906 (571h53m42.196858941s) [noise=true , address=145.79.227.40:4067 , piece_id=WSYDRX5YJ6OCFW52ZVMM6WZYNRFUXMR4XJ6C5727BOFMR4RRH33Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.325865978 (571h53m42.196949023s) [seg_pos={0 0}, upload=501384]: Segment reached optimal threshold of 14 pieces. Dec 29 11:22:42.326137074 (571h53m42.197220109s) [noise=true , address=64.49.13.169:4708 , piece_id=B2X2RWRELXWJPF6AVJJZCNU46LM5ASGRE4EWGPKL3SCLQ4VKUBDQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection Dec 29 11:22:42.326153074 (571h53m42.197236109s) [noise=true , address=64.49.13.169:4708 , piece_id=B2X2RWRELXWJPF6AVJJZCNU46LM5ASGRE4EWGPKL3SCLQ4VKUBDQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326193401 (571h53m42.197276446s) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12h5esfg6mw2c8y3AVpeFBkL5FfLAwYdjhcnQ9CJ8fjh61WLYjx): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326205705 (571h53m42.19728874s) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326407739 (571h53m42.197490784s) [noise=true , address=77.90.31.10:4524 , piece_id=Q2JGEUV5EDFTVY2CSNPZOVJIIIAFBA3IHA7S5N7YZRH3E7TSTE5A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uN7CznoK4AD8FqFkNUndawdGJ9rmjAe7oNnjJkV844ScY87qe): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326415374 (571h53m42.197498419s) [noise=true , address=77.90.31.10:4524 , piece_id=Q2JGEUV5EDFTVY2CSNPZOVJIIIAFBA3IHA7S5N7YZRH3E7TSTE5A, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326505105 (571h53m42.19758815s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tnTEYvqJ6PB2DsuXA28sEuEiGj11StA4qdhcYH1kuoaMkTVdY): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326512188 (571h53m42.197595223s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326532357 (571h53m42.197615392s) [seg_pos={0 0}, upload=501384]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 11:22:42.326671562 (571h53m42.197754597s) [upload=501384]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 11:22:42.326677843 (571h53m42.197760879s) [upload=501384]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 11:22:42.610391438 (571h53m42.481474493s) [upload=501384]: Done uploading segments... Dec 29 11:22:42.610416826 (571h53m42.481499871s) [upload=501384]: Upload finished. err: <nil> Duration: 37.663832248s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 19 out of 60 is failed (threshold=6) [........X....X.......X..X...XX..XX......X.X.XX.X.XX..X..XX.X] duration: 5.001747748s trace: https://jaeger.dev.storj.io/trace/6494a8aa68acd03c FAIL (2025-12-29 11:40:22 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001870833s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [.X.....X.X...X.X.X.X.XX..XX..X..XXX.X.......X.X..X.XXX...XX.] duration: 5.964353964s trace: https://jaeger.dev.storj.io/trace/139faac8fffa8c5 FAIL (2025-12-29 11:40:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: 2TRSPS4SR2ZD64PXTRZY6X2QLW7OE33ZLDYW7PEIUVTYDYAEDGEQ) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: SXT24VJURZOAWBSKU5GBNZGDTPCE4OLDU3BXBLKMWIQDU63ZYHYA) context deadline exceeded; piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: 36FUDIJUW7QTRYSOR7JQFEJT5ROQFDVDNNWBKH3KLWMYKJBTZF7A) context deadline exceeded ======LOGS====== Dec 29 11:39:34.431716226 (572h10m34.302799261s) [upload=501635]: Starting upload Dec 29 11:39:34.431785748 (572h10m34.302868783s) [upload=501635]: Done starting upload Dec 29 11:39:34.431830503 (572h10m34.302913538s) [upload=501635]: Uploading segments... Dec 29 11:39:34.431973696 (572h10m34.303056731s) [upload=501635]: Got next segment. Inline: false Dec 29 11:39:34.432009684 (572h10m34.303092719s) [upload=501635]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 11:39:34.432014143 (572h10m34.303097178s) [upload=501635]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 11:39:34.678261985 (572h10m34.54934502s) [seg_pos={0 0}, upload=501635]: Begin upload segment... Dec 29 11:39:34.678335806 (572h10m34.549418841s) [seg_pos={0 0}, upload=501635]: Done begin upload segment. Dec 29 11:39:34.678948232 (572h10m34.550031267s) [noise=true , address=77.90.31.10:4519 , piece_id=URU5TODWEPATKFHYTSRA74MAPFPPQUDEFNIOEQGBQFF7DEM3MOZA, seg_pos={0 0}, upload=501635]: Uploading piece... Dec 29 11:39:34.679188039 (572h10m34.550271074s) [noise=true , address=45.91.116.12:4225 , piece_id=36FUDIJUW7QTRYSOR7JQFEJT5ROQFDVDNNWBKH3KLWMYKJBTZF7A, seg_pos={0 0}, upload=501635]: Uploading piece... Dec 29 11:39:34.679748386 (572h10m34.550831421s) [noise=true , address=45.91.116.6:4974 , piece_id=QZDYHQOBU6S52KGEKFWR6OV2VRWVKITSNL4BYGDOROIFQQQ2ZNTQ, seg_pos={0 0}, upload=501635]: Uploading piece... Dec 29 11:39:34.680071831 (572h10m34.551154866s) [noise=true , address=145.79.227.50:4374 , piece_id=6GEJPR3WZI3MRDYMLRDO3UDAWU3ZYEWSVFZSVAJKR4A5ILMARZ2A, seg_pos={0 0}, upload=501635]: Uploading piece... Dec 29 11:39:34.680254759 (572h10m34.551337795s) [noise=true , address=145.79.227.21:4473 , piece_id=ZA3R3AQYXK3VB2FT3SZMQ6NM4INN4JGTNKMQ6TQTCHME225EKSHQ, seg_pos={0 0}, upload=501635]: Uploading piece... Dec 29 11:39:34.680554250 (572h10m34.551637285s) [noise=true , address=145.79.227.17:4971 , piece_id=Y2DQYWVFKMASQCZCQMOGB2RWSESIQJTNXMUGJSZOEVPF3AI4CORQ, seg_pos={0 0}, upload=501635]: Uploading piece... Dec 29 11:39:34.681912266 (572h10m34.552995301s) [noise=true , address=45.91.116.5:4300 , piece_id=2TRSPS4SR2ZD64PXTRZY6X2QLW7OE33ZLDYW7PEIUVTYDYAEDGEQ, seg_pos={0 0}, upload=501635]: Uploading piece... Dec 29 11:39:34.682312468 (572h10m34.553395503s) [noise=true , address=45.91.116.4:4937 , piece_id=I5SO7BWBKIXRDHJELN4MOQDRKLZZEEZUPVHWGE4D5SHU3FCPC2BQ, seg_pos={0 0}, upload=501635]: Uploading piece... Dec 29 11:39:34.682651373 (572h10m34.553734418s) [noise=true , address=45.91.116.11:4120 , piece_id=ZZY2YEHBPH2R2E4EL4MR6A6KS4JQ7OPIYHY4TRGRGV4PDR6FQWJQ, seg_pos={0 0}, upload=501635]: Uploading piece... Dec 29 11:39:34.682976423 (572h10m34.554059458s) [noise=true , address=145.79.227.18:4006 , piece_id=YULFZJ4JM4A7D5TH3PM2436LKQPFFE2T4FJ3P7SPNB7D5H3HO2JA, seg_pos={0 0}, upload=501635]: Uploading piece... Dec 29 11:39:34.684098740 (572h10m34.555181775s) [noise=true , address=145.79.227.28:4726 , piece_id=X77YPCOFOWVOCFNCXQH24VPGYKMYX3FV6GHT3JIJ6RMPJ6ATX5YA, seg_pos={0 0}, upload=501635]: Uploading piece... Dec 29 11:39:34.684604813 (572h10m34.555687848s) [noise=true , address=145.79.227.14:4839 , piece_id=UB4DX77DFXN25KKOWFM4KIN277A5AMHTQF347UYQKGGZM3LCIMZQ, seg_pos={0 0}, upload=501635]: Uploading piece... Dec 29 11:39:34.685344402 (572h10m34.556427437s) [noise=true , address=45.91.116.14:4189 , piece_id=SXT24VJURZOAWBSKU5GBNZGDTPCE4OLDU3BXBLKMWIQDU63ZYHYA, seg_pos={0 0}, upload=501635]: Uploading piece... Dec 29 11:39:34.685719265 (572h10m34.55680231s) [noise=true , address=77.90.31.12:4589 , piece_id=LWK2TNHUNBIFEE7LIBXBSGAK37EWKHCJNZI4JEVDZTSV4T4I6TNA, seg_pos={0 0}, upload=501635]: Uploading piece... Dec 29 11:39:34.686263271 (572h10m34.557346316s) [noise=true , address=45.91.116.8:4047 , piece_id=GZFD5ZKJ2PELRAZQIDCXKV5WVKIUHBTCYD2ORELMBEVTU76H7AKQ, seg_pos={0 0}, upload=501635]: Uploading piece... Dec 29 11:39:34.686661379 (572h10m34.557744424s) [noise=true , address=145.79.227.51:4411 , piece_id=2TMSIIBD5NTREN5JECTFA3DO36MKDZT4YRFGRLBWTKYVBKNFICYA, seg_pos={0 0}, upload=501635]: Uploading piece... Dec 29 11:39:36.328305722 (572h10m36.199388757s) [upload=501635]: Next returned nil segment Dec 29 11:39:36.328318156 (572h10m36.199401191s) [upload=501635]: Waiting for error group managing segments... Dec 29 11:39:36.659952647 (572h10m36.531035692s) [noise=true , address=145.79.227.21:4473 , piece_id=ZA3R3AQYXK3VB2FT3SZMQ6NM4INN4JGTNKMQ6TQTCHME225EKSHQ, seg_pos={0 0}, upload=501635]: Done uploading piece. err: <nil> Dec 29 11:39:36.662702633 (572h10m36.533785678s) [noise=true , address=145.79.227.17:4971 , piece_id=Y2DQYWVFKMASQCZCQMOGB2RWSESIQJTNXMUGJSZOEVPF3AI4CORQ, seg_pos={0 0}, upload=501635]: Done uploading piece. err: <nil> Dec 29 11:39:36.666756883 (572h10m36.537839918s) [noise=true , address=145.79.227.14:4839 , piece_id=UB4DX77DFXN25KKOWFM4KIN277A5AMHTQF347UYQKGGZM3LCIMZQ, seg_pos={0 0}, upload=501635]: Done uploading piece. err: <nil> Dec 29 11:39:36.670531641 (572h10m36.541614676s) [noise=true , address=145.79.227.28:4726 , piece_id=X77YPCOFOWVOCFNCXQH24VPGYKMYX3FV6GHT3JIJ6RMPJ6ATX5YA, seg_pos={0 0}, upload=501635]: Done uploading piece. err: <nil> Dec 29 11:39:36.688738770 (572h10m36.559821805s) [noise=true , address=145.79.227.18:4006 , piece_id=YULFZJ4JM4A7D5TH3PM2436LKQPFFE2T4FJ3P7SPNB7D5H3HO2JA, seg_pos={0 0}, upload=501635]: Done uploading piece. err: <nil> Dec 29 11:39:36.689978400 (572h10m36.561061436s) [noise=true , address=145.79.227.51:4411 , piece_id=2TMSIIBD5NTREN5JECTFA3DO36MKDZT4YRFGRLBWTKYVBKNFICYA, seg_pos={0 0}, upload=501635]: Done uploading piece. err: <nil> Dec 29 11:39:36.697216082 (572h10m36.568299117s) [noise=true , address=145.79.227.50:4374 , piece_id=6GEJPR3WZI3MRDYMLRDO3UDAWU3ZYEWSVFZSVAJKR4A5ILMARZ2A, seg_pos={0 0}, upload=501635]: Done uploading piece. err: <nil> Dec 29 11:39:36.851792368 (572h10m36.722875403s) [noise=true , address=77.90.31.10:4519 , piece_id=URU5TODWEPATKFHYTSRA74MAPFPPQUDEFNIOEQGBQFF7DEM3MOZA, seg_pos={0 0}, upload=501635]: Done uploading piece. err: <nil> Dec 29 11:39:36.924015341 (572h10m36.795098376s) [noise=true , address=45.91.116.11:4120 , piece_id=ZZY2YEHBPH2R2E4EL4MR6A6KS4JQ7OPIYHY4TRGRGV4PDR6FQWJQ, seg_pos={0 0}, upload=501635]: Done uploading piece. err: <nil> Dec 29 11:39:36.932420455 (572h10m36.80350349s) [noise=true , address=45.91.116.12:4225 , piece_id=36FUDIJUW7QTRYSOR7JQFEJT5ROQFDVDNNWBKH3KLWMYKJBTZF7A, seg_pos={0 0}, upload=501635]: Done uploading piece. err: <nil> Dec 29 11:39:36.977302929 (572h10m36.848385964s) [noise=true , address=45.91.116.14:4189 , piece_id=SXT24VJURZOAWBSKU5GBNZGDTPCE4OLDU3BXBLKMWIQDU63ZYHYA, seg_pos={0 0}, upload=501635]: Done uploading piece. err: <nil> Dec 29 11:39:37.004664870 (572h10m36.875747905s) [noise=true , address=45.91.116.5:4300 , piece_id=2TRSPS4SR2ZD64PXTRZY6X2QLW7OE33ZLDYW7PEIUVTYDYAEDGEQ, seg_pos={0 0}, upload=501635]: Done uploading piece. err: <nil> Dec 29 11:39:37.017713517 (572h10m36.888796592s) [noise=true , address=45.91.116.4:4937 , piece_id=I5SO7BWBKIXRDHJELN4MOQDRKLZZEEZUPVHWGE4D5SHU3FCPC2BQ, seg_pos={0 0}, upload=501635]: Done uploading piece. err: <nil> Dec 29 11:39:37.017739627 (572h10m36.888822682s) [seg_pos={0 0}, upload=501635]: Segment reached optimal threshold of 13 pieces. Dec 29 11:39:37.018015793 (572h10m36.889098828s) [noise=true , address=45.91.116.6:4974 , piece_id=QZDYHQOBU6S52KGEKFWR6OV2VRWVKITSNL4BYGDOROIFQQQ2ZNTQ, seg_pos={0 0}, upload=501635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 29 11:39:37.018026653 (572h10m36.889109688s) [noise=true , address=45.91.116.6:4974 , piece_id=QZDYHQOBU6S52KGEKFWR6OV2VRWVKITSNL4BYGDOROIFQQQ2ZNTQ, seg_pos={0 0}, upload=501635]: Long tail cancellation due to optimal threshold Dec 29 11:39:37.018045018 (572h10m36.889128053s) [noise=true , address=45.91.116.8:4047 , piece_id=GZFD5ZKJ2PELRAZQIDCXKV5WVKIUHBTCYD2ORELMBEVTU76H7AKQ, seg_pos={0 0}, upload=501635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 29 11:39:37.018050329 (572h10m36.889133374s) [noise=true , address=45.91.116.8:4047 , piece_id=GZFD5ZKJ2PELRAZQIDCXKV5WVKIUHBTCYD2ORELMBEVTU76H7AKQ, seg_pos={0 0}, upload=501635]: Long tail cancellation due to optimal threshold Dec 29 11:39:37.018131302 (572h10m36.889214338s) [noise=true , address=77.90.31.12:4589 , piece_id=LWK2TNHUNBIFEE7LIBXBSGAK37EWKHCJNZI4JEVDZTSV4T4I6TNA, seg_pos={0 0}, upload=501635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 29 11:39:37.018145098 (572h10m36.889228133s) [noise=true , address=77.90.31.12:4589 , piece_id=LWK2TNHUNBIFEE7LIBXBSGAK37EWKHCJNZI4JEVDZTSV4T4I6TNA, seg_pos={0 0}, upload=501635]: Long tail cancellation due to optimal threshold Dec 29 11:39:37.018161640 (572h10m36.889244685s) [seg_pos={0 0}, upload=501635]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 11:39:37.018241773 (572h10m36.889324808s) [upload=501635]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 11:39:37.018246882 (572h10m36.889329917s) [upload=501635]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 11:39:37.308715998 (572h10m37.179799043s) [upload=501635]: Done uploading segments... Dec 29 11:39:37.308762196 (572h10m37.179845231s) [upload=501635]: Upload finished. err: <nil> Duration: 5.964375134s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [..X.....X.X...X.X.X.X.XX..XX..X..XXX.X.......X.X..X.XXX...XX] duration: 30.00563106s trace: https://jaeger.dev.storj.io/trace/139faac8fffa8c5 FAIL (2025-12-29 11:40:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: 2TRSPS4SR2ZD64PXTRZY6X2QLW7OE33ZLDYW7PEIUVTYDYAEDGEQ) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: SXT24VJURZOAWBSKU5GBNZGDTPCE4OLDU3BXBLKMWIQDU63ZYHYA) context deadline exceeded; piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: 36FUDIJUW7QTRYSOR7JQFEJT5ROQFDVDNNWBKH3KLWMYKJBTZF7A) context deadline exceeded ======LOGS====== Dec 29 11:39:34.431716226 (572h10m34.302799261s) [upload=501635]: Starting upload Dec 29 11:39:34.431785748 (572h10m34.302868783s) [upload=501635]: Done starting upload Dec 29 11:39:34.431830503 (572h10m34.302913538s) [upload=501635]: Uploading segments... Dec 29 11:39:34.431973696 (572h10m34.303056731s) [upload=501635]: Got next segment. Inline: false Dec 29 11:39:34.432009684 (572h10m34.303092719s) [upload=501635]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 11:39:34.432014143 (572h10m34.303097178s) [upload=501635]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 11:39:34.678261985 (572h10m34.54934502s) [seg_pos={0 0}, upload=501635]: Begin upload segment... Dec 29 11:39:34.678335806 (572h10m34.549418841s) [seg_pos={0 0}, upload=501635]: Done begin upload segment. Dec 29 11:39:34.678948232 (572h10m34.550031267s) [noise=true , address=77.90.31.10:4519 , piece_id=URU5TODWEPATKFHYTSRA74MAPFPPQUDEFNIOEQGBQFF7DEM3MOZA, seg_pos={0 0}, upload=501635]: Uploading piece... Dec 29 11:39:34.679188039 (572h10m34.550271074s) [noise=true , address=45.91.116.12:4225 , piece_id=36FUDIJUW7QTRYSOR7JQFEJT5ROQFDVDNNWBKH3KLWMYKJBTZF7A, seg_pos={0 0}, upload=501635]: Uploading piece... Dec 29 11:39:34.679748386 (572h10m34.550831421s) [noise=true , address=45.91.116.6:4974 , piece_id=QZDYHQOBU6S52KGEKFWR6OV2VRWVKITSNL4BYGDOROIFQQQ2ZNTQ, seg_pos={0 0}, upload=501635]: Uploading piece... Dec 29 11:39:34.680071831 (572h10m34.551154866s) [noise=true , address=145.79.227.50:4374 , piece_id=6GEJPR3WZI3MRDYMLRDO3UDAWU3ZYEWSVFZSVAJKR4A5ILMARZ2A, seg_pos={0 0}, upload=501635]: Uploading piece... Dec 29 11:39:34.680254759 (572h10m34.551337795s) [noise=true , address=145.79.227.21:4473 , piece_id=ZA3R3AQYXK3VB2FT3SZMQ6NM4INN4JGTNKMQ6TQTCHME225EKSHQ, seg_pos={0 0}, upload=501635]: Uploading piece... Dec 29 11:39:34.680554250 (572h10m34.551637285s) [noise=true , address=145.79.227.17:4971 , piece_id=Y2DQYWVFKMASQCZCQMOGB2RWSESIQJTNXMUGJSZOEVPF3AI4CORQ, seg_pos={0 0}, upload=501635]: Uploading piece... Dec 29 11:39:34.681912266 (572h10m34.552995301s) [noise=true , address=45.91.116.5:4300 , piece_id=2TRSPS4SR2ZD64PXTRZY6X2QLW7OE33ZLDYW7PEIUVTYDYAEDGEQ, seg_pos={0 0}, upload=501635]: Uploading piece... Dec 29 11:39:34.682312468 (572h10m34.553395503s) [noise=true , address=45.91.116.4:4937 , piece_id=I5SO7BWBKIXRDHJELN4MOQDRKLZZEEZUPVHWGE4D5SHU3FCPC2BQ, seg_pos={0 0}, upload=501635]: Uploading piece... Dec 29 11:39:34.682651373 (572h10m34.553734418s) [noise=true , address=45.91.116.11:4120 , piece_id=ZZY2YEHBPH2R2E4EL4MR6A6KS4JQ7OPIYHY4TRGRGV4PDR6FQWJQ, seg_pos={0 0}, upload=501635]: Uploading piece... Dec 29 11:39:34.682976423 (572h10m34.554059458s) [noise=true , address=145.79.227.18:4006 , piece_id=YULFZJ4JM4A7D5TH3PM2436LKQPFFE2T4FJ3P7SPNB7D5H3HO2JA, seg_pos={0 0}, upload=501635]: Uploading piece... Dec 29 11:39:34.684098740 (572h10m34.555181775s) [noise=true , address=145.79.227.28:4726 , piece_id=X77YPCOFOWVOCFNCXQH24VPGYKMYX3FV6GHT3JIJ6RMPJ6ATX5YA, seg_pos={0 0}, upload=501635]: Uploading piece... Dec 29 11:39:34.684604813 (572h10m34.555687848s) [noise=true , address=145.79.227.14:4839 , piece_id=UB4DX77DFXN25KKOWFM4KIN277A5AMHTQF347UYQKGGZM3LCIMZQ, seg_pos={0 0}, upload=501635]: Uploading piece... Dec 29 11:39:34.685344402 (572h10m34.556427437s) [noise=true , address=45.91.116.14:4189 , piece_id=SXT24VJURZOAWBSKU5GBNZGDTPCE4OLDU3BXBLKMWIQDU63ZYHYA, seg_pos={0 0}, upload=501635]: Uploading piece... Dec 29 11:39:34.685719265 (572h10m34.55680231s) [noise=true , address=77.90.31.12:4589 , piece_id=LWK2TNHUNBIFEE7LIBXBSGAK37EWKHCJNZI4JEVDZTSV4T4I6TNA, seg_pos={0 0}, upload=501635]: Uploading piece... Dec 29 11:39:34.686263271 (572h10m34.557346316s) [noise=true , address=45.91.116.8:4047 , piece_id=GZFD5ZKJ2PELRAZQIDCXKV5WVKIUHBTCYD2ORELMBEVTU76H7AKQ, seg_pos={0 0}, upload=501635]: Uploading piece... Dec 29 11:39:34.686661379 (572h10m34.557744424s) [noise=true , address=145.79.227.51:4411 , piece_id=2TMSIIBD5NTREN5JECTFA3DO36MKDZT4YRFGRLBWTKYVBKNFICYA, seg_pos={0 0}, upload=501635]: Uploading piece... Dec 29 11:39:36.328305722 (572h10m36.199388757s) [upload=501635]: Next returned nil segment Dec 29 11:39:36.328318156 (572h10m36.199401191s) [upload=501635]: Waiting for error group managing segments... Dec 29 11:39:36.659952647 (572h10m36.531035692s) [noise=true , address=145.79.227.21:4473 , piece_id=ZA3R3AQYXK3VB2FT3SZMQ6NM4INN4JGTNKMQ6TQTCHME225EKSHQ, seg_pos={0 0}, upload=501635]: Done uploading piece. err: <nil> Dec 29 11:39:36.662702633 (572h10m36.533785678s) [noise=true , address=145.79.227.17:4971 , piece_id=Y2DQYWVFKMASQCZCQMOGB2RWSESIQJTNXMUGJSZOEVPF3AI4CORQ, seg_pos={0 0}, upload=501635]: Done uploading piece. err: <nil> Dec 29 11:39:36.666756883 (572h10m36.537839918s) [noise=true , address=145.79.227.14:4839 , piece_id=UB4DX77DFXN25KKOWFM4KIN277A5AMHTQF347UYQKGGZM3LCIMZQ, seg_pos={0 0}, upload=501635]: Done uploading piece. err: <nil> Dec 29 11:39:36.670531641 (572h10m36.541614676s) [noise=true , address=145.79.227.28:4726 , piece_id=X77YPCOFOWVOCFNCXQH24VPGYKMYX3FV6GHT3JIJ6RMPJ6ATX5YA, seg_pos={0 0}, upload=501635]: Done uploading piece. err: <nil> Dec 29 11:39:36.688738770 (572h10m36.559821805s) [noise=true , address=145.79.227.18:4006 , piece_id=YULFZJ4JM4A7D5TH3PM2436LKQPFFE2T4FJ3P7SPNB7D5H3HO2JA, seg_pos={0 0}, upload=501635]: Done uploading piece. err: <nil> Dec 29 11:39:36.689978400 (572h10m36.561061436s) [noise=true , address=145.79.227.51:4411 , piece_id=2TMSIIBD5NTREN5JECTFA3DO36MKDZT4YRFGRLBWTKYVBKNFICYA, seg_pos={0 0}, upload=501635]: Done uploading piece. err: <nil> Dec 29 11:39:36.697216082 (572h10m36.568299117s) [noise=true , address=145.79.227.50:4374 , piece_id=6GEJPR3WZI3MRDYMLRDO3UDAWU3ZYEWSVFZSVAJKR4A5ILMARZ2A, seg_pos={0 0}, upload=501635]: Done uploading piece. err: <nil> Dec 29 11:39:36.851792368 (572h10m36.722875403s) [noise=true , address=77.90.31.10:4519 , piece_id=URU5TODWEPATKFHYTSRA74MAPFPPQUDEFNIOEQGBQFF7DEM3MOZA, seg_pos={0 0}, upload=501635]: Done uploading piece. err: <nil> Dec 29 11:39:36.924015341 (572h10m36.795098376s) [noise=true , address=45.91.116.11:4120 , piece_id=ZZY2YEHBPH2R2E4EL4MR6A6KS4JQ7OPIYHY4TRGRGV4PDR6FQWJQ, seg_pos={0 0}, upload=501635]: Done uploading piece. err: <nil> Dec 29 11:39:36.932420455 (572h10m36.80350349s) [noise=true , address=45.91.116.12:4225 , piece_id=36FUDIJUW7QTRYSOR7JQFEJT5ROQFDVDNNWBKH3KLWMYKJBTZF7A, seg_pos={0 0}, upload=501635]: Done uploading piece. err: <nil> Dec 29 11:39:36.977302929 (572h10m36.848385964s) [noise=true , address=45.91.116.14:4189 , piece_id=SXT24VJURZOAWBSKU5GBNZGDTPCE4OLDU3BXBLKMWIQDU63ZYHYA, seg_pos={0 0}, upload=501635]: Done uploading piece. err: <nil> Dec 29 11:39:37.004664870 (572h10m36.875747905s) [noise=true , address=45.91.116.5:4300 , piece_id=2TRSPS4SR2ZD64PXTRZY6X2QLW7OE33ZLDYW7PEIUVTYDYAEDGEQ, seg_pos={0 0}, upload=501635]: Done uploading piece. err: <nil> Dec 29 11:39:37.017713517 (572h10m36.888796592s) [noise=true , address=45.91.116.4:4937 , piece_id=I5SO7BWBKIXRDHJELN4MOQDRKLZZEEZUPVHWGE4D5SHU3FCPC2BQ, seg_pos={0 0}, upload=501635]: Done uploading piece. err: <nil> Dec 29 11:39:37.017739627 (572h10m36.888822682s) [seg_pos={0 0}, upload=501635]: Segment reached optimal threshold of 13 pieces. Dec 29 11:39:37.018015793 (572h10m36.889098828s) [noise=true , address=45.91.116.6:4974 , piece_id=QZDYHQOBU6S52KGEKFWR6OV2VRWVKITSNL4BYGDOROIFQQQ2ZNTQ, seg_pos={0 0}, upload=501635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 29 11:39:37.018026653 (572h10m36.889109688s) [noise=true , address=45.91.116.6:4974 , piece_id=QZDYHQOBU6S52KGEKFWR6OV2VRWVKITSNL4BYGDOROIFQQQ2ZNTQ, seg_pos={0 0}, upload=501635]: Long tail cancellation due to optimal threshold Dec 29 11:39:37.018045018 (572h10m36.889128053s) [noise=true , address=45.91.116.8:4047 , piece_id=GZFD5ZKJ2PELRAZQIDCXKV5WVKIUHBTCYD2ORELMBEVTU76H7AKQ, seg_pos={0 0}, upload=501635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 29 11:39:37.018050329 (572h10m36.889133374s) [noise=true , address=45.91.116.8:4047 , piece_id=GZFD5ZKJ2PELRAZQIDCXKV5WVKIUHBTCYD2ORELMBEVTU76H7AKQ, seg_pos={0 0}, upload=501635]: Long tail cancellation due to optimal threshold Dec 29 11:39:37.018131302 (572h10m36.889214338s) [noise=true , address=77.90.31.12:4589 , piece_id=LWK2TNHUNBIFEE7LIBXBSGAK37EWKHCJNZI4JEVDZTSV4T4I6TNA, seg_pos={0 0}, upload=501635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 29 11:39:37.018145098 (572h10m36.889228133s) [noise=true , address=77.90.31.12:4589 , piece_id=LWK2TNHUNBIFEE7LIBXBSGAK37EWKHCJNZI4JEVDZTSV4T4I6TNA, seg_pos={0 0}, upload=501635]: Long tail cancellation due to optimal threshold Dec 29 11:39:37.018161640 (572h10m36.889244685s) [seg_pos={0 0}, upload=501635]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 11:39:37.018241773 (572h10m36.889324808s) [upload=501635]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 11:39:37.018246882 (572h10m36.889329917s) [upload=501635]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 11:39:37.308715998 (572h10m37.179799043s) [upload=501635]: Done uploading segments... Dec 29 11:39:37.308762196 (572h10m37.179845231s) [upload=501635]: Upload finished. err: <nil> Duration: 30.005655928s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [...................X...............X..X.....................] duration: 43.534779306s trace: https://jaeger.dev.storj.io/trace/7e7354d8023608a8 FAIL (2025-12-29 11:39:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== ) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480097317 (571h53m39.351180352s) [noise=true , address=77.90.31.21:4889 , piece_id=BR6Z55P3A7YJ5FJRDSZFUIF66FGIJQY7WCEQ7PL5DOYWL73OMKTQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480405053 (571h53m39.351488098s) [noise=true , address=77.90.31.16:4750 , piece_id=MVVZPJEGDCBSNVOB6CD2WUZGIJTL3VBHX2G4L3TVZ6JF5ZEJ5U6Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480712969 (571h53m39.351796014s) [noise=true , address=77.90.31.18:4834 , piece_id=BTWNGNQVEUEVV6HLBDM63SGNTKV4RDU4J4XFLG52WHKP53H3L5KA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481019453 (571h53m39.352102488s) [noise=true , address=145.79.227.30:4777 , piece_id=VSCXLXGIBSFE6E4OHLWNZIZ75PO7WQO2VFHUYHW7JUNH62KGGU7Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481029532 (571h53m39.352112567s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481559832 (571h53m39.352642877s) [noise=true , address=145.79.227.35:4515 , piece_id=L7LFZKZEZERUNX4HUNDNQ2PWHXNJRJ7OBFRNMFIICPZEUIBHGP7A, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481731919 (571h53m39.352814954s) [noise=true , address=64.49.13.26:4953 , piece_id=4YSW5Y3A6HYG3XJ2XAB7RGVHAOHMP7G3QIZYMYTSXVXYJ3ANLNXA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482078329 (571h53m39.353161364s) [noise=true , address=145.79.227.44:4205 , piece_id=UE4QSHDQBZ3XBWQ6I6MCRVTLMU7U6W6OZDA2ZGQCH6ZYRAGPD7WA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482099830 (571h53m39.353182865s) [noise=true , address=145.79.227.40:4067 , piece_id=WSYDRX5YJ6OCFW52ZVMM6WZYNRFUXMR4XJ6C5727BOFMR4RRH33Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482434487 (571h53m39.353517532s) [noise=true , address=145.79.227.3:4164 , piece_id=U6F7VDPOH7K6R5EH5UUXHFP55GXOEXO5BLBJHZRTITPDQI4LP7KQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482480845 (571h53m39.35356388s) [noise=true , address=64.49.13.140:4516 , piece_id=4GUYOYOAEASUK2W25S3NILWIIIHGOZX76L4EGFLUUUEDWRALPP2A, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482841442 (571h53m39.353924477s) [noise=true , address=145.79.227.34:4910 , piece_id=ZH4DCNXET2Q6VRAUH3NQ3KKSV4OYI575X7YS72XD236EEOMOG3LA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483163645 (571h53m39.35424668s) [noise=true , address=145.79.227.29:4974 , piece_id=KEKREOD2B35XQNI45JBHE2SZZC3MQQBQUR5I6WFDAQYJQGJBXSMA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483532478 (571h53m39.354615513s) [noise=true , address=77.90.31.17:4769 , piece_id=3MKUMOQIUVERKQLLIFX52XQPTQ5LSFMHIPD5YW2YTO5YEFR36IYA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483914596 (571h53m39.354997631s) [noise=true , address=145.79.227.13:4807 , piece_id=VKXB5MWAG2YYH2AWWJEGTLBZ2QUPLI25VTHI6AMMALSM23Z3AWZQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:40.761721106 (571h53m40.632804141s) [upload=501384]: Next returned nil segment Dec 29 11:22:40.761733349 (571h53m40.632816384s) [upload=501384]: Waiting for error group managing segments... Dec 29 11:22:41.121222862 (571h53m40.992305897s) [noise=true , address=64.49.13.140:4516 , piece_id=4GUYOYOAEASUK2W25S3NILWIIIHGOZX76L4EGFLUUUEDWRALPP2A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.197311646 (571h53m41.068394691s) [noise=true , address=64.49.13.26:4953 , piece_id=4YSW5Y3A6HYG3XJ2XAB7RGVHAOHMP7G3QIZYMYTSXVXYJ3ANLNXA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.347247568 (571h53m41.218330613s) [noise=true , address=145.79.227.13:4807 , piece_id=VKXB5MWAG2YYH2AWWJEGTLBZ2QUPLI25VTHI6AMMALSM23Z3AWZQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.347917313 (571h53m41.219000348s) [noise=true , address=145.79.227.3:4164 , piece_id=U6F7VDPOH7K6R5EH5UUXHFP55GXOEXO5BLBJHZRTITPDQI4LP7KQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.348510613 (571h53m41.219593648s) [noise=true , address=145.79.227.29:4974 , piece_id=KEKREOD2B35XQNI45JBHE2SZZC3MQQBQUR5I6WFDAQYJQGJBXSMA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.503950985 (571h53m41.37503402s) [noise=true , address=77.90.31.17:4769 , piece_id=3MKUMOQIUVERKQLLIFX52XQPTQ5LSFMHIPD5YW2YTO5YEFR36IYA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.561352025 (571h53m41.43243506s) [noise=true , address=77.90.31.18:4834 , piece_id=BTWNGNQVEUEVV6HLBDM63SGNTKV4RDU4J4XFLG52WHKP53H3L5KA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.622447380 (571h53m41.493530425s) [noise=true , address=77.90.31.16:4750 , piece_id=MVVZPJEGDCBSNVOB6CD2WUZGIJTL3VBHX2G4L3TVZ6JF5ZEJ5U6Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.645494685 (571h53m41.51657773s) [noise=true , address=77.90.31.21:4889 , piece_id=BR6Z55P3A7YJ5FJRDSZFUIF66FGIJQY7WCEQ7PL5DOYWL73OMKTQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.247930045 (571h53m42.11901308s) [noise=true , address=145.79.227.30:4777 , piece_id=VSCXLXGIBSFE6E4OHLWNZIZ75PO7WQO2VFHUYHW7JUNH62KGGU7Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.262221890 (571h53m42.133304925s) [noise=true , address=145.79.227.44:4205 , piece_id=UE4QSHDQBZ3XBWQ6I6MCRVTLMU7U6W6OZDA2ZGQCH6ZYRAGPD7WA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.295598123 (571h53m42.166681158s) [noise=true , address=145.79.227.35:4515 , piece_id=L7LFZKZEZERUNX4HUNDNQ2PWHXNJRJ7OBFRNMFIICPZEUIBHGP7A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.307481351 (571h53m42.178564386s) [noise=true , address=145.79.227.34:4910 , piece_id=ZH4DCNXET2Q6VRAUH3NQ3KKSV4OYI575X7YS72XD236EEOMOG3LA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.325775906 (571h53m42.196858941s) [noise=true , address=145.79.227.40:4067 , piece_id=WSYDRX5YJ6OCFW52ZVMM6WZYNRFUXMR4XJ6C5727BOFMR4RRH33Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.325865978 (571h53m42.196949023s) [seg_pos={0 0}, upload=501384]: Segment reached optimal threshold of 14 pieces. Dec 29 11:22:42.326137074 (571h53m42.197220109s) [noise=true , address=64.49.13.169:4708 , piece_id=B2X2RWRELXWJPF6AVJJZCNU46LM5ASGRE4EWGPKL3SCLQ4VKUBDQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection Dec 29 11:22:42.326153074 (571h53m42.197236109s) [noise=true , address=64.49.13.169:4708 , piece_id=B2X2RWRELXWJPF6AVJJZCNU46LM5ASGRE4EWGPKL3SCLQ4VKUBDQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326193401 (571h53m42.197276446s) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12h5esfg6mw2c8y3AVpeFBkL5FfLAwYdjhcnQ9CJ8fjh61WLYjx): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326205705 (571h53m42.19728874s) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326407739 (571h53m42.197490784s) [noise=true , address=77.90.31.10:4524 , piece_id=Q2JGEUV5EDFTVY2CSNPZOVJIIIAFBA3IHA7S5N7YZRH3E7TSTE5A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uN7CznoK4AD8FqFkNUndawdGJ9rmjAe7oNnjJkV844ScY87qe): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326415374 (571h53m42.197498419s) [noise=true , address=77.90.31.10:4524 , piece_id=Q2JGEUV5EDFTVY2CSNPZOVJIIIAFBA3IHA7S5N7YZRH3E7TSTE5A, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326505105 (571h53m42.19758815s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tnTEYvqJ6PB2DsuXA28sEuEiGj11StA4qdhcYH1kuoaMkTVdY): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326512188 (571h53m42.197595223s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326532357 (571h53m42.197615392s) [seg_pos={0 0}, upload=501384]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 11:22:42.326671562 (571h53m42.197754597s) [upload=501384]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 11:22:42.326677843 (571h53m42.197760879s) [upload=501384]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 11:22:42.610391438 (571h53m42.481474493s) [upload=501384]: Done uploading segments... Dec 29 11:22:42.610416826 (571h53m42.481499871s) [upload=501384]: Upload finished. err: <nil> Duration: 43.534805435s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 18 out of 60 is failed (threshold=6) [.........X....X.......X..X...XX..XX......X.X.XX.X.XX..X..XX.] duration: 5.001273134s trace: https://jaeger.dev.storj.io/trace/13c922971106cc5e FAIL (2025-12-29 11:39:44 UTC) orbiter warning: context deadline exceeded Duration: 5.001296568s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [X..X.....X.X...X.X.X.X.XX..XX..X..XXX.X.......X.X..X.XXX...X] duration: 30.003347634s trace: https://jaeger.dev.storj.io/trace/75e4667c358ca0ca FAIL (2025-12-29 11:39:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 11:39:00.389131848 (572h10m0.260214893s) [upload=501624]: Starting upload Dec 29 11:39:00.389377867 (572h10m0.260460902s) [upload=501624]: Done starting upload Dec 29 11:39:00.389480632 (572h10m0.260563677s) [upload=501624]: Uploading segments... Dec 29 11:39:00.389639736 (572h10m0.260722771s) [upload=501624]: Got next segment. Inline: false Dec 29 11:39:00.389690632 (572h10m0.260773677s) [upload=501624]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 11:39:00.389696233 (572h10m0.260779278s) [upload=501624]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 11:39:00.633132020 (572h10m0.504215055s) [seg_pos={0 0}, upload=501624]: Begin upload segment... Dec 29 11:39:00.633195662 (572h10m0.504278697s) [seg_pos={0 0}, upload=501624]: Done begin upload segment. Dec 29 11:39:00.633273180 (572h10m0.504356215s) [noise=true , address=145.79.227.16:4909 , piece_id=4SCXTDORKR6YW4RLDODXN7WEOFSTVHGJE3HDJ37CCT3IQB7RMIOQ, seg_pos={0 0}, upload=501624]: Uploading piece... Dec 29 11:39:00.633312234 (572h10m0.504395269s) [noise=true , address=145.79.227.45:4230 , piece_id=45BQDOKA3M7DXRE4YVRMITKXZIRB7QRKBRABRHGIMCLYA6PO5MGQ, seg_pos={0 0}, upload=501624]: Uploading piece... Dec 29 11:39:00.633408166 (572h10m0.504491201s) [noise=true , address=145.79.227.48:4302 , piece_id=JLXWLOZNXIN7D4CRXQAJQE3Z2DR3XLB3IIAUXHRRNXFPL2NPMAZQ, seg_pos={0 0}, upload=501624]: Uploading piece... Dec 29 11:39:00.633457060 (572h10m0.504540105s) [noise=true , address=145.79.227.30:4761 , piece_id=VPI5YEL65D725AJDXMELRFIC4RHCIJZOXW7V5ZO6FWIHBQ3M6HTQ, seg_pos={0 0}, upload=501624]: Uploading piece... Dec 29 11:39:00.634029519 (572h10m0.505112565s) [noise=true , address=145.79.227.50:4374 , piece_id=H3UNHWHVX3IIQSA54Z7UJWAVGA2GCS2URBU4P36RO5OMCU2LLJXA, seg_pos={0 0}, upload=501624]: Uploading piece... Dec 29 11:39:00.635576195 (572h10m0.50665924s) [noise=true , address=145.79.227.38:4979 , piece_id=QB6M25NVAZ4JJK4H6Z7EM7KTXZUYV576JBFXCIHAQ3KQ5NOZLYEA, seg_pos={0 0}, upload=501624]: Uploading piece... Dec 29 11:39:00.636017876 (572h10m0.507100911s) [noise=true , address=77.90.31.21:4878 , piece_id=W7EQURSFDT56NYINZFLQOH3WWANBESY3KUN2HZVSZUGPKES7Q3OQ, seg_pos={0 0}, upload=501624]: Uploading piece... Dec 29 11:39:00.636453125 (572h10m0.50753616s) [noise=true , address=145.79.227.4:4186 , piece_id=OCDV2QVERBE73UC4HJUKUHIPLQI23GISRWFQFLPRSAKB4ACZMOLA, seg_pos={0 0}, upload=501624]: Uploading piece... Dec 29 11:39:00.637885753 (572h10m0.508968798s) [noise=true , address=45.91.116.11:4119 , piece_id=EDYYP5JZWLPOEYM72ESFKR36JVFMKIQORDS7QK4MU25XLE6XGIQQ, seg_pos={0 0}, upload=501624]: Uploading piece... Dec 29 11:39:00.638088429 (572h10m0.509171474s) [noise=true , address=77.90.31.13:4626 , piece_id=ZPH2HQN2ZGGY2JAHFZQXPCWDRG7GGKZIUIQUM55JTN5EXV66EDOA, seg_pos={0 0}, upload=501624]: Uploading piece... Dec 29 11:39:00.639774980 (572h10m0.510858015s) [noise=true , address=45.91.116.13:4156 , piece_id=T4ULJFF43ML4KLN7BCAROCTAXIQDQBK4GMN2NB54JZ4MKU3CGLJQ, seg_pos={0 0}, upload=501624]: Uploading piece... Dec 29 11:39:00.641456001 (572h10m0.512539046s) [noise=true , address=145.79.227.51:4409 , piece_id=7GVIZVW3VGO3IXD3OZV44ITJGGAB6F3MK4NUKOMLZP55HJ3QCMTQ, seg_pos={0 0}, upload=501624]: Uploading piece... Dec 29 11:39:00.641894927 (572h10m0.512977962s) [noise=true , address=145.79.227.15:4874 , piece_id=62VYSEPXNJCZBFUWASUCMH4TZ5PDJ7V25H7QH5ZLHRHCAHDHP3MA, seg_pos={0 0}, upload=501624]: Uploading piece... Dec 29 11:39:00.643277830 (572h10m0.514360875s) [noise=true , address=145.79.227.26:4654 , piece_id=MCN6R3ZH6PYI4LVBRBU5GA724QQHPKAEYTV7NNBOG7QAUPSHAFYQ, seg_pos={0 0}, upload=501624]: Uploading piece... Dec 29 11:39:00.644649171 (572h10m0.515732226s) [noise=true , address=145.79.227.37:4565 , piece_id=3CHFHTGPUZMLXFQ74PETDJK4YKELU53KGPZR75MVNYKG5XZILUFA, seg_pos={0 0}, upload=501624]: Uploading piece... Dec 29 11:39:00.646294905 (572h10m0.51737795s) [noise=true , address=45.91.116.10:4083 , piece_id=HJO4SI2RR72JPQD5V37YVPF3E3Y2IVQGGG7TTPL5APMQPKHDUEPQ, seg_pos={0 0}, upload=501624]: Uploading piece... Dec 29 11:39:02.256214029 (572h10m2.127297064s) [upload=501624]: Next returned nil segment Dec 29 11:39:02.256228968 (572h10m2.127312013s) [upload=501624]: Waiting for error group managing segments... Dec 29 11:39:02.643306891 (572h10m2.514389947s) [noise=true , address=145.79.227.50:4374 , piece_id=H3UNHWHVX3IIQSA54Z7UJWAVGA2GCS2URBU4P36RO5OMCU2LLJXA, seg_pos={0 0}, upload=501624]: Done uploading piece. err: <nil> Dec 29 11:39:02.646322483 (572h10m2.517405518s) [noise=true , address=145.79.227.37:4565 , piece_id=3CHFHTGPUZMLXFQ74PETDJK4YKELU53KGPZR75MVNYKG5XZILUFA, seg_pos={0 0}, upload=501624]: Done uploading piece. err: <nil> Dec 29 11:39:02.649859008 (572h10m2.520942053s) [noise=true , address=145.79.227.48:4302 , piece_id=JLXWLOZNXIN7D4CRXQAJQE3Z2DR3XLB3IIAUXHRRNXFPL2NPMAZQ, seg_pos={0 0}, upload=501624]: Done uploading piece. err: <nil> Dec 29 11:39:02.652143668 (572h10m2.523226713s) [noise=true , address=145.79.227.45:4230 , piece_id=45BQDOKA3M7DXRE4YVRMITKXZIRB7QRKBRABRHGIMCLYA6PO5MGQ, seg_pos={0 0}, upload=501624]: Done uploading piece. err: <nil> Dec 29 11:39:02.652282753 (572h10m2.523365798s) [noise=true , address=145.79.227.26:4654 , piece_id=MCN6R3ZH6PYI4LVBRBU5GA724QQHPKAEYTV7NNBOG7QAUPSHAFYQ, seg_pos={0 0}, upload=501624]: Done uploading piece. err: <nil> Dec 29 11:39:02.672587466 (572h10m2.543670511s) [noise=true , address=145.79.227.4:4186 , piece_id=OCDV2QVERBE73UC4HJUKUHIPLQI23GISRWFQFLPRSAKB4ACZMOLA, seg_pos={0 0}, upload=501624]: Done uploading piece. err: <nil> Dec 29 11:39:02.687847283 (572h10m2.558930328s) [noise=true , address=145.79.227.15:4874 , piece_id=62VYSEPXNJCZBFUWASUCMH4TZ5PDJ7V25H7QH5ZLHRHCAHDHP3MA, seg_pos={0 0}, upload=501624]: Done uploading piece. err: <nil> Dec 29 11:39:02.688752707 (572h10m2.559835752s) [noise=true , address=145.79.227.51:4409 , piece_id=7GVIZVW3VGO3IXD3OZV44ITJGGAB6F3MK4NUKOMLZP55HJ3QCMTQ, seg_pos={0 0}, upload=501624]: Done uploading piece. err: <nil> Dec 29 11:39:02.689498157 (572h10m2.560581202s) [noise=true , address=145.79.227.16:4909 , piece_id=4SCXTDORKR6YW4RLDODXN7WEOFSTVHGJE3HDJ37CCT3IQB7RMIOQ, seg_pos={0 0}, upload=501624]: Done uploading piece. err: <nil> Dec 29 11:39:02.693367865 (572h10m2.56445091s) [noise=true , address=145.79.227.38:4979 , piece_id=QB6M25NVAZ4JJK4H6Z7EM7KTXZUYV576JBFXCIHAQ3KQ5NOZLYEA, seg_pos={0 0}, upload=501624]: Done uploading piece. err: <nil> Dec 29 11:39:02.877904900 (572h10m2.748987945s) [noise=true , address=45.91.116.11:4119 , piece_id=EDYYP5JZWLPOEYM72ESFKR36JVFMKIQORDS7QK4MU25XLE6XGIQQ, seg_pos={0 0}, upload=501624]: Done uploading piece. err: <nil> Dec 29 11:39:02.898166251 (572h10m2.769249286s) [noise=true , address=45.91.116.10:4083 , piece_id=HJO4SI2RR72JPQD5V37YVPF3E3Y2IVQGGG7TTPL5APMQPKHDUEPQ, seg_pos={0 0}, upload=501624]: Done uploading piece. err: <nil> Dec 29 11:39:02.996083670 (572h10m2.867166715s) [noise=true , address=45.91.116.13:4156 , piece_id=T4ULJFF43ML4KLN7BCAROCTAXIQDQBK4GMN2NB54JZ4MKU3CGLJQ, seg_pos={0 0}, upload=501624]: Done uploading piece. err: <nil> Dec 29 11:39:02.996158753 (572h10m2.867241798s) [seg_pos={0 0}, upload=501624]: Segment reached optimal threshold of 13 pieces. Dec 29 11:39:02.996379584 (572h10m2.867462629s) [noise=true , address=77.90.31.13:4626 , piece_id=ZPH2HQN2ZGGY2JAHFZQXPCWDRG7GGKZIUIQUM55JTN5EXV66EDOA, seg_pos={0 0}, upload=501624]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF Dec 29 11:39:02.996386557 (572h10m2.867469602s) [noise=true , address=77.90.31.13:4626 , piece_id=ZPH2HQN2ZGGY2JAHFZQXPCWDRG7GGKZIUIQUM55JTN5EXV66EDOA, seg_pos={0 0}, upload=501624]: Long tail cancellation due to optimal threshold Dec 29 11:39:02.996507086 (572h10m2.867590131s) [noise=true , address=145.79.227.30:4761 , piece_id=VPI5YEL65D725AJDXMELRFIC4RHCIJZOXW7V5ZO6FWIHBQ3M6HTQ, seg_pos={0 0}, upload=501624]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 29 11:39:02.996513518 (572h10m2.867596563s) [noise=true , address=145.79.227.30:4761 , piece_id=VPI5YEL65D725AJDXMELRFIC4RHCIJZOXW7V5ZO6FWIHBQ3M6HTQ, seg_pos={0 0}, upload=501624]: Long tail cancellation due to optimal threshold Dec 29 11:39:02.996629720 (572h10m2.867712765s) [noise=true , address=77.90.31.21:4878 , piece_id=W7EQURSFDT56NYINZFLQOH3WWANBESY3KUN2HZVSZUGPKES7Q3OQ, seg_pos={0 0}, upload=501624]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 29 11:39:02.996636002 (572h10m2.867719037s) [noise=true , address=77.90.31.21:4878 , piece_id=W7EQURSFDT56NYINZFLQOH3WWANBESY3KUN2HZVSZUGPKES7Q3OQ, seg_pos={0 0}, upload=501624]: Long tail cancellation due to optimal threshold Dec 29 11:39:02.996653264 (572h10m2.86773631s) [seg_pos={0 0}, upload=501624]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 11:39:02.996737455 (572h10m2.8678205s) [upload=501624]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 11:39:02.996743857 (572h10m2.867826902s) [upload=501624]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 11:39:03.283859597 (572h10m3.154942632s) [upload=501624]: Done uploading segments... Dec 29 11:39:03.283883452 (572h10m3.154966497s) [upload=501624]: Upload finished. err: <nil> Duration: 30.00337237s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [....................X...............X..X....................] duration: 53.022297624s trace: https://jaeger.dev.storj.io/trace/7e7354d8023608a8 FAIL (2025-12-29 11:38:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== ) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480097317 (571h53m39.351180352s) [noise=true , address=77.90.31.21:4889 , piece_id=BR6Z55P3A7YJ5FJRDSZFUIF66FGIJQY7WCEQ7PL5DOYWL73OMKTQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480405053 (571h53m39.351488098s) [noise=true , address=77.90.31.16:4750 , piece_id=MVVZPJEGDCBSNVOB6CD2WUZGIJTL3VBHX2G4L3TVZ6JF5ZEJ5U6Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480712969 (571h53m39.351796014s) [noise=true , address=77.90.31.18:4834 , piece_id=BTWNGNQVEUEVV6HLBDM63SGNTKV4RDU4J4XFLG52WHKP53H3L5KA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481019453 (571h53m39.352102488s) [noise=true , address=145.79.227.30:4777 , piece_id=VSCXLXGIBSFE6E4OHLWNZIZ75PO7WQO2VFHUYHW7JUNH62KGGU7Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481029532 (571h53m39.352112567s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481559832 (571h53m39.352642877s) [noise=true , address=145.79.227.35:4515 , piece_id=L7LFZKZEZERUNX4HUNDNQ2PWHXNJRJ7OBFRNMFIICPZEUIBHGP7A, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481731919 (571h53m39.352814954s) [noise=true , address=64.49.13.26:4953 , piece_id=4YSW5Y3A6HYG3XJ2XAB7RGVHAOHMP7G3QIZYMYTSXVXYJ3ANLNXA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482078329 (571h53m39.353161364s) [noise=true , address=145.79.227.44:4205 , piece_id=UE4QSHDQBZ3XBWQ6I6MCRVTLMU7U6W6OZDA2ZGQCH6ZYRAGPD7WA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482099830 (571h53m39.353182865s) [noise=true , address=145.79.227.40:4067 , piece_id=WSYDRX5YJ6OCFW52ZVMM6WZYNRFUXMR4XJ6C5727BOFMR4RRH33Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482434487 (571h53m39.353517532s) [noise=true , address=145.79.227.3:4164 , piece_id=U6F7VDPOH7K6R5EH5UUXHFP55GXOEXO5BLBJHZRTITPDQI4LP7KQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482480845 (571h53m39.35356388s) [noise=true , address=64.49.13.140:4516 , piece_id=4GUYOYOAEASUK2W25S3NILWIIIHGOZX76L4EGFLUUUEDWRALPP2A, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482841442 (571h53m39.353924477s) [noise=true , address=145.79.227.34:4910 , piece_id=ZH4DCNXET2Q6VRAUH3NQ3KKSV4OYI575X7YS72XD236EEOMOG3LA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483163645 (571h53m39.35424668s) [noise=true , address=145.79.227.29:4974 , piece_id=KEKREOD2B35XQNI45JBHE2SZZC3MQQBQUR5I6WFDAQYJQGJBXSMA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483532478 (571h53m39.354615513s) [noise=true , address=77.90.31.17:4769 , piece_id=3MKUMOQIUVERKQLLIFX52XQPTQ5LSFMHIPD5YW2YTO5YEFR36IYA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483914596 (571h53m39.354997631s) [noise=true , address=145.79.227.13:4807 , piece_id=VKXB5MWAG2YYH2AWWJEGTLBZ2QUPLI25VTHI6AMMALSM23Z3AWZQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:40.761721106 (571h53m40.632804141s) [upload=501384]: Next returned nil segment Dec 29 11:22:40.761733349 (571h53m40.632816384s) [upload=501384]: Waiting for error group managing segments... Dec 29 11:22:41.121222862 (571h53m40.992305897s) [noise=true , address=64.49.13.140:4516 , piece_id=4GUYOYOAEASUK2W25S3NILWIIIHGOZX76L4EGFLUUUEDWRALPP2A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.197311646 (571h53m41.068394691s) [noise=true , address=64.49.13.26:4953 , piece_id=4YSW5Y3A6HYG3XJ2XAB7RGVHAOHMP7G3QIZYMYTSXVXYJ3ANLNXA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.347247568 (571h53m41.218330613s) [noise=true , address=145.79.227.13:4807 , piece_id=VKXB5MWAG2YYH2AWWJEGTLBZ2QUPLI25VTHI6AMMALSM23Z3AWZQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.347917313 (571h53m41.219000348s) [noise=true , address=145.79.227.3:4164 , piece_id=U6F7VDPOH7K6R5EH5UUXHFP55GXOEXO5BLBJHZRTITPDQI4LP7KQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.348510613 (571h53m41.219593648s) [noise=true , address=145.79.227.29:4974 , piece_id=KEKREOD2B35XQNI45JBHE2SZZC3MQQBQUR5I6WFDAQYJQGJBXSMA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.503950985 (571h53m41.37503402s) [noise=true , address=77.90.31.17:4769 , piece_id=3MKUMOQIUVERKQLLIFX52XQPTQ5LSFMHIPD5YW2YTO5YEFR36IYA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.561352025 (571h53m41.43243506s) [noise=true , address=77.90.31.18:4834 , piece_id=BTWNGNQVEUEVV6HLBDM63SGNTKV4RDU4J4XFLG52WHKP53H3L5KA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.622447380 (571h53m41.493530425s) [noise=true , address=77.90.31.16:4750 , piece_id=MVVZPJEGDCBSNVOB6CD2WUZGIJTL3VBHX2G4L3TVZ6JF5ZEJ5U6Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.645494685 (571h53m41.51657773s) [noise=true , address=77.90.31.21:4889 , piece_id=BR6Z55P3A7YJ5FJRDSZFUIF66FGIJQY7WCEQ7PL5DOYWL73OMKTQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.247930045 (571h53m42.11901308s) [noise=true , address=145.79.227.30:4777 , piece_id=VSCXLXGIBSFE6E4OHLWNZIZ75PO7WQO2VFHUYHW7JUNH62KGGU7Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.262221890 (571h53m42.133304925s) [noise=true , address=145.79.227.44:4205 , piece_id=UE4QSHDQBZ3XBWQ6I6MCRVTLMU7U6W6OZDA2ZGQCH6ZYRAGPD7WA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.295598123 (571h53m42.166681158s) [noise=true , address=145.79.227.35:4515 , piece_id=L7LFZKZEZERUNX4HUNDNQ2PWHXNJRJ7OBFRNMFIICPZEUIBHGP7A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.307481351 (571h53m42.178564386s) [noise=true , address=145.79.227.34:4910 , piece_id=ZH4DCNXET2Q6VRAUH3NQ3KKSV4OYI575X7YS72XD236EEOMOG3LA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.325775906 (571h53m42.196858941s) [noise=true , address=145.79.227.40:4067 , piece_id=WSYDRX5YJ6OCFW52ZVMM6WZYNRFUXMR4XJ6C5727BOFMR4RRH33Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.325865978 (571h53m42.196949023s) [seg_pos={0 0}, upload=501384]: Segment reached optimal threshold of 14 pieces. Dec 29 11:22:42.326137074 (571h53m42.197220109s) [noise=true , address=64.49.13.169:4708 , piece_id=B2X2RWRELXWJPF6AVJJZCNU46LM5ASGRE4EWGPKL3SCLQ4VKUBDQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection Dec 29 11:22:42.326153074 (571h53m42.197236109s) [noise=true , address=64.49.13.169:4708 , piece_id=B2X2RWRELXWJPF6AVJJZCNU46LM5ASGRE4EWGPKL3SCLQ4VKUBDQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326193401 (571h53m42.197276446s) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12h5esfg6mw2c8y3AVpeFBkL5FfLAwYdjhcnQ9CJ8fjh61WLYjx): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326205705 (571h53m42.19728874s) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326407739 (571h53m42.197490784s) [noise=true , address=77.90.31.10:4524 , piece_id=Q2JGEUV5EDFTVY2CSNPZOVJIIIAFBA3IHA7S5N7YZRH3E7TSTE5A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uN7CznoK4AD8FqFkNUndawdGJ9rmjAe7oNnjJkV844ScY87qe): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326415374 (571h53m42.197498419s) [noise=true , address=77.90.31.10:4524 , piece_id=Q2JGEUV5EDFTVY2CSNPZOVJIIIAFBA3IHA7S5N7YZRH3E7TSTE5A, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326505105 (571h53m42.19758815s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tnTEYvqJ6PB2DsuXA28sEuEiGj11StA4qdhcYH1kuoaMkTVdY): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326512188 (571h53m42.197595223s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326532357 (571h53m42.197615392s) [seg_pos={0 0}, upload=501384]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 11:22:42.326671562 (571h53m42.197754597s) [upload=501384]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 11:22:42.326677843 (571h53m42.197760879s) [upload=501384]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 11:22:42.610391438 (571h53m42.481474493s) [upload=501384]: Done uploading segments... Dec 29 11:22:42.610416826 (571h53m42.481499871s) [upload=501384]: Upload finished. err: <nil> Duration: 53.022314184s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 18 out of 60 is failed (threshold=6) [..........X....X.......X..X...XX..XX......X.X.XX.X.XX..X..XX] duration: 5.001335119s trace: https://jaeger.dev.storj.io/trace/13c922971106cc5e FAIL (2025-12-29 11:38:32 UTC) orbiter warning: context deadline exceeded Duration: 5.001359335s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....................X...............X..X...................] duration: 34.612574907s trace: https://jaeger.dev.storj.io/trace/7e7354d8023608a8 FAIL (2025-12-29 11:38:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== ) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480097317 (571h53m39.351180352s) [noise=true , address=77.90.31.21:4889 , piece_id=BR6Z55P3A7YJ5FJRDSZFUIF66FGIJQY7WCEQ7PL5DOYWL73OMKTQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480405053 (571h53m39.351488098s) [noise=true , address=77.90.31.16:4750 , piece_id=MVVZPJEGDCBSNVOB6CD2WUZGIJTL3VBHX2G4L3TVZ6JF5ZEJ5U6Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480712969 (571h53m39.351796014s) [noise=true , address=77.90.31.18:4834 , piece_id=BTWNGNQVEUEVV6HLBDM63SGNTKV4RDU4J4XFLG52WHKP53H3L5KA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481019453 (571h53m39.352102488s) [noise=true , address=145.79.227.30:4777 , piece_id=VSCXLXGIBSFE6E4OHLWNZIZ75PO7WQO2VFHUYHW7JUNH62KGGU7Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481029532 (571h53m39.352112567s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481559832 (571h53m39.352642877s) [noise=true , address=145.79.227.35:4515 , piece_id=L7LFZKZEZERUNX4HUNDNQ2PWHXNJRJ7OBFRNMFIICPZEUIBHGP7A, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481731919 (571h53m39.352814954s) [noise=true , address=64.49.13.26:4953 , piece_id=4YSW5Y3A6HYG3XJ2XAB7RGVHAOHMP7G3QIZYMYTSXVXYJ3ANLNXA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482078329 (571h53m39.353161364s) [noise=true , address=145.79.227.44:4205 , piece_id=UE4QSHDQBZ3XBWQ6I6MCRVTLMU7U6W6OZDA2ZGQCH6ZYRAGPD7WA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482099830 (571h53m39.353182865s) [noise=true , address=145.79.227.40:4067 , piece_id=WSYDRX5YJ6OCFW52ZVMM6WZYNRFUXMR4XJ6C5727BOFMR4RRH33Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482434487 (571h53m39.353517532s) [noise=true , address=145.79.227.3:4164 , piece_id=U6F7VDPOH7K6R5EH5UUXHFP55GXOEXO5BLBJHZRTITPDQI4LP7KQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482480845 (571h53m39.35356388s) [noise=true , address=64.49.13.140:4516 , piece_id=4GUYOYOAEASUK2W25S3NILWIIIHGOZX76L4EGFLUUUEDWRALPP2A, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482841442 (571h53m39.353924477s) [noise=true , address=145.79.227.34:4910 , piece_id=ZH4DCNXET2Q6VRAUH3NQ3KKSV4OYI575X7YS72XD236EEOMOG3LA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483163645 (571h53m39.35424668s) [noise=true , address=145.79.227.29:4974 , piece_id=KEKREOD2B35XQNI45JBHE2SZZC3MQQBQUR5I6WFDAQYJQGJBXSMA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483532478 (571h53m39.354615513s) [noise=true , address=77.90.31.17:4769 , piece_id=3MKUMOQIUVERKQLLIFX52XQPTQ5LSFMHIPD5YW2YTO5YEFR36IYA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483914596 (571h53m39.354997631s) [noise=true , address=145.79.227.13:4807 , piece_id=VKXB5MWAG2YYH2AWWJEGTLBZ2QUPLI25VTHI6AMMALSM23Z3AWZQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:40.761721106 (571h53m40.632804141s) [upload=501384]: Next returned nil segment Dec 29 11:22:40.761733349 (571h53m40.632816384s) [upload=501384]: Waiting for error group managing segments... Dec 29 11:22:41.121222862 (571h53m40.992305897s) [noise=true , address=64.49.13.140:4516 , piece_id=4GUYOYOAEASUK2W25S3NILWIIIHGOZX76L4EGFLUUUEDWRALPP2A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.197311646 (571h53m41.068394691s) [noise=true , address=64.49.13.26:4953 , piece_id=4YSW5Y3A6HYG3XJ2XAB7RGVHAOHMP7G3QIZYMYTSXVXYJ3ANLNXA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.347247568 (571h53m41.218330613s) [noise=true , address=145.79.227.13:4807 , piece_id=VKXB5MWAG2YYH2AWWJEGTLBZ2QUPLI25VTHI6AMMALSM23Z3AWZQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.347917313 (571h53m41.219000348s) [noise=true , address=145.79.227.3:4164 , piece_id=U6F7VDPOH7K6R5EH5UUXHFP55GXOEXO5BLBJHZRTITPDQI4LP7KQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.348510613 (571h53m41.219593648s) [noise=true , address=145.79.227.29:4974 , piece_id=KEKREOD2B35XQNI45JBHE2SZZC3MQQBQUR5I6WFDAQYJQGJBXSMA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.503950985 (571h53m41.37503402s) [noise=true , address=77.90.31.17:4769 , piece_id=3MKUMOQIUVERKQLLIFX52XQPTQ5LSFMHIPD5YW2YTO5YEFR36IYA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.561352025 (571h53m41.43243506s) [noise=true , address=77.90.31.18:4834 , piece_id=BTWNGNQVEUEVV6HLBDM63SGNTKV4RDU4J4XFLG52WHKP53H3L5KA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.622447380 (571h53m41.493530425s) [noise=true , address=77.90.31.16:4750 , piece_id=MVVZPJEGDCBSNVOB6CD2WUZGIJTL3VBHX2G4L3TVZ6JF5ZEJ5U6Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.645494685 (571h53m41.51657773s) [noise=true , address=77.90.31.21:4889 , piece_id=BR6Z55P3A7YJ5FJRDSZFUIF66FGIJQY7WCEQ7PL5DOYWL73OMKTQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.247930045 (571h53m42.11901308s) [noise=true , address=145.79.227.30:4777 , piece_id=VSCXLXGIBSFE6E4OHLWNZIZ75PO7WQO2VFHUYHW7JUNH62KGGU7Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.262221890 (571h53m42.133304925s) [noise=true , address=145.79.227.44:4205 , piece_id=UE4QSHDQBZ3XBWQ6I6MCRVTLMU7U6W6OZDA2ZGQCH6ZYRAGPD7WA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.295598123 (571h53m42.166681158s) [noise=true , address=145.79.227.35:4515 , piece_id=L7LFZKZEZERUNX4HUNDNQ2PWHXNJRJ7OBFRNMFIICPZEUIBHGP7A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.307481351 (571h53m42.178564386s) [noise=true , address=145.79.227.34:4910 , piece_id=ZH4DCNXET2Q6VRAUH3NQ3KKSV4OYI575X7YS72XD236EEOMOG3LA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.325775906 (571h53m42.196858941s) [noise=true , address=145.79.227.40:4067 , piece_id=WSYDRX5YJ6OCFW52ZVMM6WZYNRFUXMR4XJ6C5727BOFMR4RRH33Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.325865978 (571h53m42.196949023s) [seg_pos={0 0}, upload=501384]: Segment reached optimal threshold of 14 pieces. Dec 29 11:22:42.326137074 (571h53m42.197220109s) [noise=true , address=64.49.13.169:4708 , piece_id=B2X2RWRELXWJPF6AVJJZCNU46LM5ASGRE4EWGPKL3SCLQ4VKUBDQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection Dec 29 11:22:42.326153074 (571h53m42.197236109s) [noise=true , address=64.49.13.169:4708 , piece_id=B2X2RWRELXWJPF6AVJJZCNU46LM5ASGRE4EWGPKL3SCLQ4VKUBDQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326193401 (571h53m42.197276446s) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12h5esfg6mw2c8y3AVpeFBkL5FfLAwYdjhcnQ9CJ8fjh61WLYjx): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326205705 (571h53m42.19728874s) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326407739 (571h53m42.197490784s) [noise=true , address=77.90.31.10:4524 , piece_id=Q2JGEUV5EDFTVY2CSNPZOVJIIIAFBA3IHA7S5N7YZRH3E7TSTE5A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uN7CznoK4AD8FqFkNUndawdGJ9rmjAe7oNnjJkV844ScY87qe): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326415374 (571h53m42.197498419s) [noise=true , address=77.90.31.10:4524 , piece_id=Q2JGEUV5EDFTVY2CSNPZOVJIIIAFBA3IHA7S5N7YZRH3E7TSTE5A, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326505105 (571h53m42.19758815s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tnTEYvqJ6PB2DsuXA28sEuEiGj11StA4qdhcYH1kuoaMkTVdY): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326512188 (571h53m42.197595223s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326532357 (571h53m42.197615392s) [seg_pos={0 0}, upload=501384]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 11:22:42.326671562 (571h53m42.197754597s) [upload=501384]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 11:22:42.326677843 (571h53m42.197760879s) [upload=501384]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 11:22:42.610391438 (571h53m42.481474493s) [upload=501384]: Done uploading segments... Dec 29 11:22:42.610416826 (571h53m42.481499871s) [upload=501384]: Upload finished. err: <nil> Duration: 34.612604694s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [.X..X.....X.X...X.X.X.X.XX..XX..X..XXX.X.......X.X..X.XXX...] duration: 6.266650637s trace: https://jaeger.dev.storj.io/trace/3dd25a5fabd0957b FAIL (2025-12-29 11:37:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: WWAWENADBGDTJ6WDNLIG2DGQDGPE6JDWSUKDZRD2JI353CRA3ZEA) context deadline exceeded; piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: 5QLVW6DOT4KWQYBS6RUR4CCZ67V3LQGGMXJHIVEQHNUEI5FOW7GA) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: CXOPOZBQC6R6Q5G44EDXJBXFQBNBZ426PCIWLGXP7SS3HURY56KQ) context deadline exceeded ======LOGS====== Dec 29 11:35:56.459074447 (572h6m56.330157492s) [upload=501579]: Starting upload Dec 29 11:35:56.459138019 (572h6m56.330221054s) [upload=501579]: Done starting upload Dec 29 11:35:56.459205287 (572h6m56.330288322s) [upload=501579]: Uploading segments... Dec 29 11:35:56.459388165 (572h6m56.33047121s) [upload=501579]: Got next segment. Inline: false Dec 29 11:35:56.459425846 (572h6m56.330508891s) [upload=501579]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 11:35:56.459431608 (572h6m56.330514653s) [upload=501579]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 11:35:56.711093519 (572h6m56.582176554s) [seg_pos={0 0}, upload=501579]: Begin upload segment... Dec 29 11:35:56.711138224 (572h6m56.582221259s) [seg_pos={0 0}, upload=501579]: Done begin upload segment. Dec 29 11:35:56.711174664 (572h6m56.582257699s) [noise=true , address=145.79.227.51:4410 , piece_id=YXJGHLBQWCOIJCYOBR62YZOIOBVM2MFZ46HMGAKXQBDV6HQJEAMA, seg_pos={0 0}, upload=501579]: Uploading piece... Dec 29 11:35:56.711269364 (572h6m56.582352409s) [noise=true , address=145.79.227.48:4301 , piece_id=CYYRHB527AIKYVHLOGQDYUZKIBQ57DFBPZGMUFSOX4Z7FI7RP2CA, seg_pos={0 0}, upload=501579]: Uploading piece... Dec 29 11:35:56.711907739 (572h6m56.582990785s) [noise=true , address=145.79.227.18:4007 , piece_id=AHHHEKGANPKWUF36MOETE7RROQDB5NDYBIATQKX32LXSB627UGSQ, seg_pos={0 0}, upload=501579]: Uploading piece... Dec 29 11:35:56.712185939 (572h6m56.583268984s) [noise=true , address=45.91.116.10:4083 , piece_id=ZJB5XCBP3NHKUTCR7PTPU27CIZCAYNS75YLAPTIJOEUARUWAEY2Q, seg_pos={0 0}, upload=501579]: Uploading piece... Dec 29 11:35:56.713315771 (572h6m56.584398806s) [noise=true , address=45.91.116.4:4939 , piece_id=5B5EB3KIC4XXBW2HVTGSYWXI6O4AR7LCKP3CHRSRLTTQNB3QI3DA, seg_pos={0 0}, upload=501579]: Uploading piece... Dec 29 11:35:56.713357170 (572h6m56.584440205s) [noise=true , address=145.79.227.14:4838 , piece_id=I324MYN64UEHSH6WLDDSJSDIT5QG5JTD3MVGVCS5QCDKEK3TAI4Q, seg_pos={0 0}, upload=501579]: Uploading piece... Dec 29 11:35:56.714384246 (572h6m56.585467291s) [noise=true , address=45.91.116.8:4048 , piece_id=WWAWENADBGDTJ6WDNLIG2DGQDGPE6JDWSUKDZRD2JI353CRA3ZEA, seg_pos={0 0}, upload=501579]: Uploading piece... Dec 29 11:35:56.714462304 (572h6m56.585545359s) [noise=true , address=45.91.116.5:4297 , piece_id=5QLVW6DOT4KWQYBS6RUR4CCZ67V3LQGGMXJHIVEQHNUEI5FOW7GA, seg_pos={0 0}, upload=501579]: Uploading piece... Dec 29 11:35:56.715383739 (572h6m56.586466774s) [noise=true , address=45.91.116.6:4975 , piece_id=CXOPOZBQC6R6Q5G44EDXJBXFQBNBZ426PCIWLGXP7SS3HURY56KQ, seg_pos={0 0}, upload=501579]: Uploading piece... Dec 29 11:35:56.715475003 (572h6m56.586558038s) [noise=true , address=145.79.227.49:4337 , piece_id=7PF7B4RWEHDZBS6GD6XFYTHIR5JTPJBZCUXE2HG7D4TJD4GQD6UA, seg_pos={0 0}, upload=501579]: Uploading piece... Dec 29 11:35:56.716386869 (572h6m56.587469904s) [noise=true , address=45.91.116.13:4155 , piece_id=JJHI4DZMMVCQZHHWLLS6WMTCGZDFCZOYNX53TKOXAB2YRFJPHJ2A, seg_pos={0 0}, upload=501579]: Uploading piece... Dec 29 11:35:56.716459126 (572h6m56.587542161s) [noise=true , address=145.79.227.4:4185 , piece_id=V4FK2BIHYDHHP3UBLHOFMHSH6NCL7ZYRAEO2XRB726X32HM75E2A, seg_pos={0 0}, upload=501579]: Uploading piece... Dec 29 11:35:56.717277685 (572h6m56.588360731s) [noise=true , address=145.79.227.52:4446 , piece_id=OKHNVKLC55SA3VPJ4OFBMKJKCQ2E3TSCZXRZHVGO5K2Y5PH6FWBQ, seg_pos={0 0}, upload=501579]: Uploading piece... Dec 29 11:35:56.717673769 (572h6m56.588756804s) [noise=true , address=45.91.116.7:4009 , piece_id=6IVOZQ5AOE45W5ZWIQXA6KC4YU775DHAOQBH6ZO77YJPLZPGHQGA, seg_pos={0 0}, upload=501579]: Uploading piece... Dec 29 11:35:56.717895341 (572h6m56.588978426s) [noise=true , address=145.79.227.26:4655 , piece_id=XXJMUKL5PFEHXZG6JLMLNO63GQUJCO2HS6GEQAU6RL36IL2M2SWQ, seg_pos={0 0}, upload=501579]: Uploading piece... Dec 29 11:35:56.718323988 (572h6m56.589407023s) [noise=true , address=45.91.116.12:4225 , piece_id=NZS3EUF2BFLLEJ734IBT5RRZF2ZMRC4YYPSL2QD7K7X5MD765BJA, seg_pos={0 0}, upload=501579]: Uploading piece... Dec 29 11:35:58.324261742 (572h6m58.195344777s) [upload=501579]: Next returned nil segment Dec 29 11:35:58.324275217 (572h6m58.195358253s) [upload=501579]: Waiting for error group managing segments... Dec 29 11:35:58.706737693 (572h6m58.577820728s) [noise=true , address=145.79.227.14:4838 , piece_id=I324MYN64UEHSH6WLDDSJSDIT5QG5JTD3MVGVCS5QCDKEK3TAI4Q, seg_pos={0 0}, upload=501579]: Done uploading piece. err: <nil> Dec 29 11:35:58.706821933 (572h6m58.577904978s) [noise=true , address=145.79.227.18:4007 , piece_id=AHHHEKGANPKWUF36MOETE7RROQDB5NDYBIATQKX32LXSB627UGSQ, seg_pos={0 0}, upload=501579]: Done uploading piece. err: <nil> Dec 29 11:35:58.707328368 (572h6m58.578411403s) [noise=true , address=145.79.227.4:4185 , piece_id=V4FK2BIHYDHHP3UBLHOFMHSH6NCL7ZYRAEO2XRB726X32HM75E2A, seg_pos={0 0}, upload=501579]: Done uploading piece. err: <nil> Dec 29 11:35:58.710997875 (572h6m58.58208091s) [noise=true , address=145.79.227.52:4446 , piece_id=OKHNVKLC55SA3VPJ4OFBMKJKCQ2E3TSCZXRZHVGO5K2Y5PH6FWBQ, seg_pos={0 0}, upload=501579]: Done uploading piece. err: <nil> Dec 29 11:35:58.711306833 (572h6m58.582389868s) [noise=true , address=145.79.227.26:4655 , piece_id=XXJMUKL5PFEHXZG6JLMLNO63GQUJCO2HS6GEQAU6RL36IL2M2SWQ, seg_pos={0 0}, upload=501579]: Done uploading piece. err: <nil> Dec 29 11:35:58.725093245 (572h6m58.59617629s) [noise=true , address=145.79.227.51:4410 , piece_id=YXJGHLBQWCOIJCYOBR62YZOIOBVM2MFZ46HMGAKXQBDV6HQJEAMA, seg_pos={0 0}, upload=501579]: Done uploading piece. err: <nil> Dec 29 11:35:58.726268482 (572h6m58.597351517s) [noise=true , address=145.79.227.48:4301 , piece_id=CYYRHB527AIKYVHLOGQDYUZKIBQ57DFBPZGMUFSOX4Z7FI7RP2CA, seg_pos={0 0}, upload=501579]: Done uploading piece. err: <nil> Dec 29 11:35:58.973990231 (572h6m58.845073266s) [noise=true , address=45.91.116.8:4048 , piece_id=WWAWENADBGDTJ6WDNLIG2DGQDGPE6JDWSUKDZRD2JI353CRA3ZEA, seg_pos={0 0}, upload=501579]: Done uploading piece. err: <nil> Dec 29 11:35:58.974561478 (572h6m58.845644523s) [noise=true , address=45.91.116.4:4939 , piece_id=5B5EB3KIC4XXBW2HVTGSYWXI6O4AR7LCKP3CHRSRLTTQNB3QI3DA, seg_pos={0 0}, upload=501579]: Done uploading piece. err: <nil> Dec 29 11:35:59.003043162 (572h6m58.874126207s) [noise=true , address=45.91.116.5:4297 , piece_id=5QLVW6DOT4KWQYBS6RUR4CCZ67V3LQGGMXJHIVEQHNUEI5FOW7GA, seg_pos={0 0}, upload=501579]: Done uploading piece. err: <nil> Dec 29 11:35:59.122220202 (572h6m58.993303237s) [noise=true , address=45.91.116.12:4225 , piece_id=NZS3EUF2BFLLEJ734IBT5RRZF2ZMRC4YYPSL2QD7K7X5MD765BJA, seg_pos={0 0}, upload=501579]: Done uploading piece. err: <nil> Dec 29 11:35:59.215783370 (572h6m59.086866405s) [noise=true , address=45.91.116.7:4009 , piece_id=6IVOZQ5AOE45W5ZWIQXA6KC4YU775DHAOQBH6ZO77YJPLZPGHQGA, seg_pos={0 0}, upload=501579]: Done uploading piece. err: <nil> Dec 29 11:35:59.686524179 (572h6m59.557607214s) [noise=true , address=45.91.116.6:4975 , piece_id=CXOPOZBQC6R6Q5G44EDXJBXFQBNBZ426PCIWLGXP7SS3HURY56KQ, seg_pos={0 0}, upload=501579]: Done uploading piece. err: <nil> Dec 29 11:35:59.686542454 (572h6m59.557625499s) [seg_pos={0 0}, upload=501579]: Segment reached optimal threshold of 13 pieces. Dec 29 11:35:59.686724450 (572h6m59.557807495s) [noise=true , address=145.79.227.49:4337 , piece_id=7PF7B4RWEHDZBS6GD6XFYTHIR5JTPJBZCUXE2HG7D4TJD4GQD6UA, seg_pos={0 0}, upload=501579]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE): protocol: expected piece hash; context canceled; EOF Dec 29 11:35:59.686730231 (572h6m59.557813276s) [noise=true , address=145.79.227.49:4337 , piece_id=7PF7B4RWEHDZBS6GD6XFYTHIR5JTPJBZCUXE2HG7D4TJD4GQD6UA, seg_pos={0 0}, upload=501579]: Long tail cancellation due to optimal threshold Dec 29 11:35:59.686832947 (572h6m59.557915982s) [noise=true , address=45.91.116.13:4155 , piece_id=JJHI4DZMMVCQZHHWLLS6WMTCGZDFCZOYNX53TKOXAB2YRFJPHJ2A, seg_pos={0 0}, upload=501579]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 29 11:35:59.686837585 (572h6m59.557920621s) [noise=true , address=45.91.116.13:4155 , piece_id=JJHI4DZMMVCQZHHWLLS6WMTCGZDFCZOYNX53TKOXAB2YRFJPHJ2A, seg_pos={0 0}, upload=501579]: Long tail cancellation due to optimal threshold Dec 29 11:35:59.686944549 (572h6m59.558027594s) [noise=true , address=45.91.116.10:4083 , piece_id=ZJB5XCBP3NHKUTCR7PTPU27CIZCAYNS75YLAPTIJOEUARUWAEY2Q, seg_pos={0 0}, upload=501579]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 29 11:35:59.686949909 (572h6m59.558032944s) [noise=true , address=45.91.116.10:4083 , piece_id=ZJB5XCBP3NHKUTCR7PTPU27CIZCAYNS75YLAPTIJOEUARUWAEY2Q, seg_pos={0 0}, upload=501579]: Long tail cancellation due to optimal threshold Dec 29 11:35:59.686983924 (572h6m59.558066959s) [seg_pos={0 0}, upload=501579]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 11:35:59.687060260 (572h6m59.558143305s) [upload=501579]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 11:35:59.687064809 (572h6m59.558147844s) [upload=501579]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 11:36:00.006171134 (572h6m59.877254169s) [upload=501579]: Done uploading segments... Dec 29 11:36:00.006192696 (572h6m59.877275741s) [upload=501579]: Upload finished. err: <nil> Duration: 6.266682127s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 18 out of 60 is failed (threshold=6) [X..........X....X.......X..X...XX..XX......X.X.XX.X.XX..X..X] duration: 5.000435659s trace: https://jaeger.dev.storj.io/trace/6104dded72758154 FAIL (2025-12-29 11:37:54 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000458552s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 17 out of 60 is failed (threshold=6) [.X..........X....X.......X..X...XX..XX......X.X.XX.X.XX..X..] duration: 3.800422216s trace: https://jaeger.dev.storj.io/trace/7cbeba427ba3458a FAIL (2025-12-29 11:37:26 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.800447553s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [..X..X.....X.X...X.X.X.X.XX..XX..X..XXX.X.......X.X..X.XXX..] duration: 6.552481633s trace: https://jaeger.dev.storj.io/trace/3dd25a5fabd0957b FAIL (2025-12-29 11:37:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: WWAWENADBGDTJ6WDNLIG2DGQDGPE6JDWSUKDZRD2JI353CRA3ZEA) context deadline exceeded; piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: 5QLVW6DOT4KWQYBS6RUR4CCZ67V3LQGGMXJHIVEQHNUEI5FOW7GA) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: CXOPOZBQC6R6Q5G44EDXJBXFQBNBZ426PCIWLGXP7SS3HURY56KQ) context deadline exceeded ======LOGS====== Dec 29 11:35:56.459074447 (572h6m56.330157492s) [upload=501579]: Starting upload Dec 29 11:35:56.459138019 (572h6m56.330221054s) [upload=501579]: Done starting upload Dec 29 11:35:56.459205287 (572h6m56.330288322s) [upload=501579]: Uploading segments... Dec 29 11:35:56.459388165 (572h6m56.33047121s) [upload=501579]: Got next segment. Inline: false Dec 29 11:35:56.459425846 (572h6m56.330508891s) [upload=501579]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 11:35:56.459431608 (572h6m56.330514653s) [upload=501579]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 11:35:56.711093519 (572h6m56.582176554s) [seg_pos={0 0}, upload=501579]: Begin upload segment... Dec 29 11:35:56.711138224 (572h6m56.582221259s) [seg_pos={0 0}, upload=501579]: Done begin upload segment. Dec 29 11:35:56.711174664 (572h6m56.582257699s) [noise=true , address=145.79.227.51:4410 , piece_id=YXJGHLBQWCOIJCYOBR62YZOIOBVM2MFZ46HMGAKXQBDV6HQJEAMA, seg_pos={0 0}, upload=501579]: Uploading piece... Dec 29 11:35:56.711269364 (572h6m56.582352409s) [noise=true , address=145.79.227.48:4301 , piece_id=CYYRHB527AIKYVHLOGQDYUZKIBQ57DFBPZGMUFSOX4Z7FI7RP2CA, seg_pos={0 0}, upload=501579]: Uploading piece... Dec 29 11:35:56.711907739 (572h6m56.582990785s) [noise=true , address=145.79.227.18:4007 , piece_id=AHHHEKGANPKWUF36MOETE7RROQDB5NDYBIATQKX32LXSB627UGSQ, seg_pos={0 0}, upload=501579]: Uploading piece... Dec 29 11:35:56.712185939 (572h6m56.583268984s) [noise=true , address=45.91.116.10:4083 , piece_id=ZJB5XCBP3NHKUTCR7PTPU27CIZCAYNS75YLAPTIJOEUARUWAEY2Q, seg_pos={0 0}, upload=501579]: Uploading piece... Dec 29 11:35:56.713315771 (572h6m56.584398806s) [noise=true , address=45.91.116.4:4939 , piece_id=5B5EB3KIC4XXBW2HVTGSYWXI6O4AR7LCKP3CHRSRLTTQNB3QI3DA, seg_pos={0 0}, upload=501579]: Uploading piece... Dec 29 11:35:56.713357170 (572h6m56.584440205s) [noise=true , address=145.79.227.14:4838 , piece_id=I324MYN64UEHSH6WLDDSJSDIT5QG5JTD3MVGVCS5QCDKEK3TAI4Q, seg_pos={0 0}, upload=501579]: Uploading piece... Dec 29 11:35:56.714384246 (572h6m56.585467291s) [noise=true , address=45.91.116.8:4048 , piece_id=WWAWENADBGDTJ6WDNLIG2DGQDGPE6JDWSUKDZRD2JI353CRA3ZEA, seg_pos={0 0}, upload=501579]: Uploading piece... Dec 29 11:35:56.714462304 (572h6m56.585545359s) [noise=true , address=45.91.116.5:4297 , piece_id=5QLVW6DOT4KWQYBS6RUR4CCZ67V3LQGGMXJHIVEQHNUEI5FOW7GA, seg_pos={0 0}, upload=501579]: Uploading piece... Dec 29 11:35:56.715383739 (572h6m56.586466774s) [noise=true , address=45.91.116.6:4975 , piece_id=CXOPOZBQC6R6Q5G44EDXJBXFQBNBZ426PCIWLGXP7SS3HURY56KQ, seg_pos={0 0}, upload=501579]: Uploading piece... Dec 29 11:35:56.715475003 (572h6m56.586558038s) [noise=true , address=145.79.227.49:4337 , piece_id=7PF7B4RWEHDZBS6GD6XFYTHIR5JTPJBZCUXE2HG7D4TJD4GQD6UA, seg_pos={0 0}, upload=501579]: Uploading piece... Dec 29 11:35:56.716386869 (572h6m56.587469904s) [noise=true , address=45.91.116.13:4155 , piece_id=JJHI4DZMMVCQZHHWLLS6WMTCGZDFCZOYNX53TKOXAB2YRFJPHJ2A, seg_pos={0 0}, upload=501579]: Uploading piece... Dec 29 11:35:56.716459126 (572h6m56.587542161s) [noise=true , address=145.79.227.4:4185 , piece_id=V4FK2BIHYDHHP3UBLHOFMHSH6NCL7ZYRAEO2XRB726X32HM75E2A, seg_pos={0 0}, upload=501579]: Uploading piece... Dec 29 11:35:56.717277685 (572h6m56.588360731s) [noise=true , address=145.79.227.52:4446 , piece_id=OKHNVKLC55SA3VPJ4OFBMKJKCQ2E3TSCZXRZHVGO5K2Y5PH6FWBQ, seg_pos={0 0}, upload=501579]: Uploading piece... Dec 29 11:35:56.717673769 (572h6m56.588756804s) [noise=true , address=45.91.116.7:4009 , piece_id=6IVOZQ5AOE45W5ZWIQXA6KC4YU775DHAOQBH6ZO77YJPLZPGHQGA, seg_pos={0 0}, upload=501579]: Uploading piece... Dec 29 11:35:56.717895341 (572h6m56.588978426s) [noise=true , address=145.79.227.26:4655 , piece_id=XXJMUKL5PFEHXZG6JLMLNO63GQUJCO2HS6GEQAU6RL36IL2M2SWQ, seg_pos={0 0}, upload=501579]: Uploading piece... Dec 29 11:35:56.718323988 (572h6m56.589407023s) [noise=true , address=45.91.116.12:4225 , piece_id=NZS3EUF2BFLLEJ734IBT5RRZF2ZMRC4YYPSL2QD7K7X5MD765BJA, seg_pos={0 0}, upload=501579]: Uploading piece... Dec 29 11:35:58.324261742 (572h6m58.195344777s) [upload=501579]: Next returned nil segment Dec 29 11:35:58.324275217 (572h6m58.195358253s) [upload=501579]: Waiting for error group managing segments... Dec 29 11:35:58.706737693 (572h6m58.577820728s) [noise=true , address=145.79.227.14:4838 , piece_id=I324MYN64UEHSH6WLDDSJSDIT5QG5JTD3MVGVCS5QCDKEK3TAI4Q, seg_pos={0 0}, upload=501579]: Done uploading piece. err: <nil> Dec 29 11:35:58.706821933 (572h6m58.577904978s) [noise=true , address=145.79.227.18:4007 , piece_id=AHHHEKGANPKWUF36MOETE7RROQDB5NDYBIATQKX32LXSB627UGSQ, seg_pos={0 0}, upload=501579]: Done uploading piece. err: <nil> Dec 29 11:35:58.707328368 (572h6m58.578411403s) [noise=true , address=145.79.227.4:4185 , piece_id=V4FK2BIHYDHHP3UBLHOFMHSH6NCL7ZYRAEO2XRB726X32HM75E2A, seg_pos={0 0}, upload=501579]: Done uploading piece. err: <nil> Dec 29 11:35:58.710997875 (572h6m58.58208091s) [noise=true , address=145.79.227.52:4446 , piece_id=OKHNVKLC55SA3VPJ4OFBMKJKCQ2E3TSCZXRZHVGO5K2Y5PH6FWBQ, seg_pos={0 0}, upload=501579]: Done uploading piece. err: <nil> Dec 29 11:35:58.711306833 (572h6m58.582389868s) [noise=true , address=145.79.227.26:4655 , piece_id=XXJMUKL5PFEHXZG6JLMLNO63GQUJCO2HS6GEQAU6RL36IL2M2SWQ, seg_pos={0 0}, upload=501579]: Done uploading piece. err: <nil> Dec 29 11:35:58.725093245 (572h6m58.59617629s) [noise=true , address=145.79.227.51:4410 , piece_id=YXJGHLBQWCOIJCYOBR62YZOIOBVM2MFZ46HMGAKXQBDV6HQJEAMA, seg_pos={0 0}, upload=501579]: Done uploading piece. err: <nil> Dec 29 11:35:58.726268482 (572h6m58.597351517s) [noise=true , address=145.79.227.48:4301 , piece_id=CYYRHB527AIKYVHLOGQDYUZKIBQ57DFBPZGMUFSOX4Z7FI7RP2CA, seg_pos={0 0}, upload=501579]: Done uploading piece. err: <nil> Dec 29 11:35:58.973990231 (572h6m58.845073266s) [noise=true , address=45.91.116.8:4048 , piece_id=WWAWENADBGDTJ6WDNLIG2DGQDGPE6JDWSUKDZRD2JI353CRA3ZEA, seg_pos={0 0}, upload=501579]: Done uploading piece. err: <nil> Dec 29 11:35:58.974561478 (572h6m58.845644523s) [noise=true , address=45.91.116.4:4939 , piece_id=5B5EB3KIC4XXBW2HVTGSYWXI6O4AR7LCKP3CHRSRLTTQNB3QI3DA, seg_pos={0 0}, upload=501579]: Done uploading piece. err: <nil> Dec 29 11:35:59.003043162 (572h6m58.874126207s) [noise=true , address=45.91.116.5:4297 , piece_id=5QLVW6DOT4KWQYBS6RUR4CCZ67V3LQGGMXJHIVEQHNUEI5FOW7GA, seg_pos={0 0}, upload=501579]: Done uploading piece. err: <nil> Dec 29 11:35:59.122220202 (572h6m58.993303237s) [noise=true , address=45.91.116.12:4225 , piece_id=NZS3EUF2BFLLEJ734IBT5RRZF2ZMRC4YYPSL2QD7K7X5MD765BJA, seg_pos={0 0}, upload=501579]: Done uploading piece. err: <nil> Dec 29 11:35:59.215783370 (572h6m59.086866405s) [noise=true , address=45.91.116.7:4009 , piece_id=6IVOZQ5AOE45W5ZWIQXA6KC4YU775DHAOQBH6ZO77YJPLZPGHQGA, seg_pos={0 0}, upload=501579]: Done uploading piece. err: <nil> Dec 29 11:35:59.686524179 (572h6m59.557607214s) [noise=true , address=45.91.116.6:4975 , piece_id=CXOPOZBQC6R6Q5G44EDXJBXFQBNBZ426PCIWLGXP7SS3HURY56KQ, seg_pos={0 0}, upload=501579]: Done uploading piece. err: <nil> Dec 29 11:35:59.686542454 (572h6m59.557625499s) [seg_pos={0 0}, upload=501579]: Segment reached optimal threshold of 13 pieces. Dec 29 11:35:59.686724450 (572h6m59.557807495s) [noise=true , address=145.79.227.49:4337 , piece_id=7PF7B4RWEHDZBS6GD6XFYTHIR5JTPJBZCUXE2HG7D4TJD4GQD6UA, seg_pos={0 0}, upload=501579]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE): protocol: expected piece hash; context canceled; EOF Dec 29 11:35:59.686730231 (572h6m59.557813276s) [noise=true , address=145.79.227.49:4337 , piece_id=7PF7B4RWEHDZBS6GD6XFYTHIR5JTPJBZCUXE2HG7D4TJD4GQD6UA, seg_pos={0 0}, upload=501579]: Long tail cancellation due to optimal threshold Dec 29 11:35:59.686832947 (572h6m59.557915982s) [noise=true , address=45.91.116.13:4155 , piece_id=JJHI4DZMMVCQZHHWLLS6WMTCGZDFCZOYNX53TKOXAB2YRFJPHJ2A, seg_pos={0 0}, upload=501579]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 29 11:35:59.686837585 (572h6m59.557920621s) [noise=true , address=45.91.116.13:4155 , piece_id=JJHI4DZMMVCQZHHWLLS6WMTCGZDFCZOYNX53TKOXAB2YRFJPHJ2A, seg_pos={0 0}, upload=501579]: Long tail cancellation due to optimal threshold Dec 29 11:35:59.686944549 (572h6m59.558027594s) [noise=true , address=45.91.116.10:4083 , piece_id=ZJB5XCBP3NHKUTCR7PTPU27CIZCAYNS75YLAPTIJOEUARUWAEY2Q, seg_pos={0 0}, upload=501579]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 29 11:35:59.686949909 (572h6m59.558032944s) [noise=true , address=45.91.116.10:4083 , piece_id=ZJB5XCBP3NHKUTCR7PTPU27CIZCAYNS75YLAPTIJOEUARUWAEY2Q, seg_pos={0 0}, upload=501579]: Long tail cancellation due to optimal threshold Dec 29 11:35:59.686983924 (572h6m59.558066959s) [seg_pos={0 0}, upload=501579]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 11:35:59.687060260 (572h6m59.558143305s) [upload=501579]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 11:35:59.687064809 (572h6m59.558147844s) [upload=501579]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 11:36:00.006171134 (572h6m59.877254169s) [upload=501579]: Done uploading segments... Dec 29 11:36:00.006192696 (572h6m59.877275741s) [upload=501579]: Upload finished. err: <nil> Duration: 6.552515507s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [......................X...............X..X..................] duration: 15.429856994s trace: https://jaeger.dev.storj.io/trace/7e7354d8023608a8 FAIL (2025-12-29 11:36:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== ) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480097317 (571h53m39.351180352s) [noise=true , address=77.90.31.21:4889 , piece_id=BR6Z55P3A7YJ5FJRDSZFUIF66FGIJQY7WCEQ7PL5DOYWL73OMKTQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480405053 (571h53m39.351488098s) [noise=true , address=77.90.31.16:4750 , piece_id=MVVZPJEGDCBSNVOB6CD2WUZGIJTL3VBHX2G4L3TVZ6JF5ZEJ5U6Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480712969 (571h53m39.351796014s) [noise=true , address=77.90.31.18:4834 , piece_id=BTWNGNQVEUEVV6HLBDM63SGNTKV4RDU4J4XFLG52WHKP53H3L5KA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481019453 (571h53m39.352102488s) [noise=true , address=145.79.227.30:4777 , piece_id=VSCXLXGIBSFE6E4OHLWNZIZ75PO7WQO2VFHUYHW7JUNH62KGGU7Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481029532 (571h53m39.352112567s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481559832 (571h53m39.352642877s) [noise=true , address=145.79.227.35:4515 , piece_id=L7LFZKZEZERUNX4HUNDNQ2PWHXNJRJ7OBFRNMFIICPZEUIBHGP7A, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481731919 (571h53m39.352814954s) [noise=true , address=64.49.13.26:4953 , piece_id=4YSW5Y3A6HYG3XJ2XAB7RGVHAOHMP7G3QIZYMYTSXVXYJ3ANLNXA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482078329 (571h53m39.353161364s) [noise=true , address=145.79.227.44:4205 , piece_id=UE4QSHDQBZ3XBWQ6I6MCRVTLMU7U6W6OZDA2ZGQCH6ZYRAGPD7WA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482099830 (571h53m39.353182865s) [noise=true , address=145.79.227.40:4067 , piece_id=WSYDRX5YJ6OCFW52ZVMM6WZYNRFUXMR4XJ6C5727BOFMR4RRH33Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482434487 (571h53m39.353517532s) [noise=true , address=145.79.227.3:4164 , piece_id=U6F7VDPOH7K6R5EH5UUXHFP55GXOEXO5BLBJHZRTITPDQI4LP7KQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482480845 (571h53m39.35356388s) [noise=true , address=64.49.13.140:4516 , piece_id=4GUYOYOAEASUK2W25S3NILWIIIHGOZX76L4EGFLUUUEDWRALPP2A, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482841442 (571h53m39.353924477s) [noise=true , address=145.79.227.34:4910 , piece_id=ZH4DCNXET2Q6VRAUH3NQ3KKSV4OYI575X7YS72XD236EEOMOG3LA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483163645 (571h53m39.35424668s) [noise=true , address=145.79.227.29:4974 , piece_id=KEKREOD2B35XQNI45JBHE2SZZC3MQQBQUR5I6WFDAQYJQGJBXSMA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483532478 (571h53m39.354615513s) [noise=true , address=77.90.31.17:4769 , piece_id=3MKUMOQIUVERKQLLIFX52XQPTQ5LSFMHIPD5YW2YTO5YEFR36IYA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483914596 (571h53m39.354997631s) [noise=true , address=145.79.227.13:4807 , piece_id=VKXB5MWAG2YYH2AWWJEGTLBZ2QUPLI25VTHI6AMMALSM23Z3AWZQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:40.761721106 (571h53m40.632804141s) [upload=501384]: Next returned nil segment Dec 29 11:22:40.761733349 (571h53m40.632816384s) [upload=501384]: Waiting for error group managing segments... Dec 29 11:22:41.121222862 (571h53m40.992305897s) [noise=true , address=64.49.13.140:4516 , piece_id=4GUYOYOAEASUK2W25S3NILWIIIHGOZX76L4EGFLUUUEDWRALPP2A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.197311646 (571h53m41.068394691s) [noise=true , address=64.49.13.26:4953 , piece_id=4YSW5Y3A6HYG3XJ2XAB7RGVHAOHMP7G3QIZYMYTSXVXYJ3ANLNXA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.347247568 (571h53m41.218330613s) [noise=true , address=145.79.227.13:4807 , piece_id=VKXB5MWAG2YYH2AWWJEGTLBZ2QUPLI25VTHI6AMMALSM23Z3AWZQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.347917313 (571h53m41.219000348s) [noise=true , address=145.79.227.3:4164 , piece_id=U6F7VDPOH7K6R5EH5UUXHFP55GXOEXO5BLBJHZRTITPDQI4LP7KQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.348510613 (571h53m41.219593648s) [noise=true , address=145.79.227.29:4974 , piece_id=KEKREOD2B35XQNI45JBHE2SZZC3MQQBQUR5I6WFDAQYJQGJBXSMA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.503950985 (571h53m41.37503402s) [noise=true , address=77.90.31.17:4769 , piece_id=3MKUMOQIUVERKQLLIFX52XQPTQ5LSFMHIPD5YW2YTO5YEFR36IYA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.561352025 (571h53m41.43243506s) [noise=true , address=77.90.31.18:4834 , piece_id=BTWNGNQVEUEVV6HLBDM63SGNTKV4RDU4J4XFLG52WHKP53H3L5KA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.622447380 (571h53m41.493530425s) [noise=true , address=77.90.31.16:4750 , piece_id=MVVZPJEGDCBSNVOB6CD2WUZGIJTL3VBHX2G4L3TVZ6JF5ZEJ5U6Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.645494685 (571h53m41.51657773s) [noise=true , address=77.90.31.21:4889 , piece_id=BR6Z55P3A7YJ5FJRDSZFUIF66FGIJQY7WCEQ7PL5DOYWL73OMKTQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.247930045 (571h53m42.11901308s) [noise=true , address=145.79.227.30:4777 , piece_id=VSCXLXGIBSFE6E4OHLWNZIZ75PO7WQO2VFHUYHW7JUNH62KGGU7Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.262221890 (571h53m42.133304925s) [noise=true , address=145.79.227.44:4205 , piece_id=UE4QSHDQBZ3XBWQ6I6MCRVTLMU7U6W6OZDA2ZGQCH6ZYRAGPD7WA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.295598123 (571h53m42.166681158s) [noise=true , address=145.79.227.35:4515 , piece_id=L7LFZKZEZERUNX4HUNDNQ2PWHXNJRJ7OBFRNMFIICPZEUIBHGP7A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.307481351 (571h53m42.178564386s) [noise=true , address=145.79.227.34:4910 , piece_id=ZH4DCNXET2Q6VRAUH3NQ3KKSV4OYI575X7YS72XD236EEOMOG3LA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.325775906 (571h53m42.196858941s) [noise=true , address=145.79.227.40:4067 , piece_id=WSYDRX5YJ6OCFW52ZVMM6WZYNRFUXMR4XJ6C5727BOFMR4RRH33Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.325865978 (571h53m42.196949023s) [seg_pos={0 0}, upload=501384]: Segment reached optimal threshold of 14 pieces. Dec 29 11:22:42.326137074 (571h53m42.197220109s) [noise=true , address=64.49.13.169:4708 , piece_id=B2X2RWRELXWJPF6AVJJZCNU46LM5ASGRE4EWGPKL3SCLQ4VKUBDQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection Dec 29 11:22:42.326153074 (571h53m42.197236109s) [noise=true , address=64.49.13.169:4708 , piece_id=B2X2RWRELXWJPF6AVJJZCNU46LM5ASGRE4EWGPKL3SCLQ4VKUBDQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326193401 (571h53m42.197276446s) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12h5esfg6mw2c8y3AVpeFBkL5FfLAwYdjhcnQ9CJ8fjh61WLYjx): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326205705 (571h53m42.19728874s) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326407739 (571h53m42.197490784s) [noise=true , address=77.90.31.10:4524 , piece_id=Q2JGEUV5EDFTVY2CSNPZOVJIIIAFBA3IHA7S5N7YZRH3E7TSTE5A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uN7CznoK4AD8FqFkNUndawdGJ9rmjAe7oNnjJkV844ScY87qe): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326415374 (571h53m42.197498419s) [noise=true , address=77.90.31.10:4524 , piece_id=Q2JGEUV5EDFTVY2CSNPZOVJIIIAFBA3IHA7S5N7YZRH3E7TSTE5A, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326505105 (571h53m42.19758815s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tnTEYvqJ6PB2DsuXA28sEuEiGj11StA4qdhcYH1kuoaMkTVdY): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326512188 (571h53m42.197595223s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326532357 (571h53m42.197615392s) [seg_pos={0 0}, upload=501384]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 11:22:42.326671562 (571h53m42.197754597s) [upload=501384]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 11:22:42.326677843 (571h53m42.197760879s) [upload=501384]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 11:22:42.610391438 (571h53m42.481474493s) [upload=501384]: Done uploading segments... Dec 29 11:22:42.610416826 (571h53m42.481499871s) [upload=501384]: Upload finished. err: <nil> Duration: 15.429881462s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......................X...............X..X.................] duration: 36.766504326s trace: https://jaeger.dev.storj.io/trace/7e7354d8023608a8 FAIL (2025-12-29 11:36:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== ) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480097317 (571h53m39.351180352s) [noise=true , address=77.90.31.21:4889 , piece_id=BR6Z55P3A7YJ5FJRDSZFUIF66FGIJQY7WCEQ7PL5DOYWL73OMKTQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480405053 (571h53m39.351488098s) [noise=true , address=77.90.31.16:4750 , piece_id=MVVZPJEGDCBSNVOB6CD2WUZGIJTL3VBHX2G4L3TVZ6JF5ZEJ5U6Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480712969 (571h53m39.351796014s) [noise=true , address=77.90.31.18:4834 , piece_id=BTWNGNQVEUEVV6HLBDM63SGNTKV4RDU4J4XFLG52WHKP53H3L5KA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481019453 (571h53m39.352102488s) [noise=true , address=145.79.227.30:4777 , piece_id=VSCXLXGIBSFE6E4OHLWNZIZ75PO7WQO2VFHUYHW7JUNH62KGGU7Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481029532 (571h53m39.352112567s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481559832 (571h53m39.352642877s) [noise=true , address=145.79.227.35:4515 , piece_id=L7LFZKZEZERUNX4HUNDNQ2PWHXNJRJ7OBFRNMFIICPZEUIBHGP7A, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481731919 (571h53m39.352814954s) [noise=true , address=64.49.13.26:4953 , piece_id=4YSW5Y3A6HYG3XJ2XAB7RGVHAOHMP7G3QIZYMYTSXVXYJ3ANLNXA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482078329 (571h53m39.353161364s) [noise=true , address=145.79.227.44:4205 , piece_id=UE4QSHDQBZ3XBWQ6I6MCRVTLMU7U6W6OZDA2ZGQCH6ZYRAGPD7WA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482099830 (571h53m39.353182865s) [noise=true , address=145.79.227.40:4067 , piece_id=WSYDRX5YJ6OCFW52ZVMM6WZYNRFUXMR4XJ6C5727BOFMR4RRH33Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482434487 (571h53m39.353517532s) [noise=true , address=145.79.227.3:4164 , piece_id=U6F7VDPOH7K6R5EH5UUXHFP55GXOEXO5BLBJHZRTITPDQI4LP7KQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482480845 (571h53m39.35356388s) [noise=true , address=64.49.13.140:4516 , piece_id=4GUYOYOAEASUK2W25S3NILWIIIHGOZX76L4EGFLUUUEDWRALPP2A, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482841442 (571h53m39.353924477s) [noise=true , address=145.79.227.34:4910 , piece_id=ZH4DCNXET2Q6VRAUH3NQ3KKSV4OYI575X7YS72XD236EEOMOG3LA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483163645 (571h53m39.35424668s) [noise=true , address=145.79.227.29:4974 , piece_id=KEKREOD2B35XQNI45JBHE2SZZC3MQQBQUR5I6WFDAQYJQGJBXSMA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483532478 (571h53m39.354615513s) [noise=true , address=77.90.31.17:4769 , piece_id=3MKUMOQIUVERKQLLIFX52XQPTQ5LSFMHIPD5YW2YTO5YEFR36IYA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483914596 (571h53m39.354997631s) [noise=true , address=145.79.227.13:4807 , piece_id=VKXB5MWAG2YYH2AWWJEGTLBZ2QUPLI25VTHI6AMMALSM23Z3AWZQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:40.761721106 (571h53m40.632804141s) [upload=501384]: Next returned nil segment Dec 29 11:22:40.761733349 (571h53m40.632816384s) [upload=501384]: Waiting for error group managing segments... Dec 29 11:22:41.121222862 (571h53m40.992305897s) [noise=true , address=64.49.13.140:4516 , piece_id=4GUYOYOAEASUK2W25S3NILWIIIHGOZX76L4EGFLUUUEDWRALPP2A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.197311646 (571h53m41.068394691s) [noise=true , address=64.49.13.26:4953 , piece_id=4YSW5Y3A6HYG3XJ2XAB7RGVHAOHMP7G3QIZYMYTSXVXYJ3ANLNXA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.347247568 (571h53m41.218330613s) [noise=true , address=145.79.227.13:4807 , piece_id=VKXB5MWAG2YYH2AWWJEGTLBZ2QUPLI25VTHI6AMMALSM23Z3AWZQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.347917313 (571h53m41.219000348s) [noise=true , address=145.79.227.3:4164 , piece_id=U6F7VDPOH7K6R5EH5UUXHFP55GXOEXO5BLBJHZRTITPDQI4LP7KQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.348510613 (571h53m41.219593648s) [noise=true , address=145.79.227.29:4974 , piece_id=KEKREOD2B35XQNI45JBHE2SZZC3MQQBQUR5I6WFDAQYJQGJBXSMA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.503950985 (571h53m41.37503402s) [noise=true , address=77.90.31.17:4769 , piece_id=3MKUMOQIUVERKQLLIFX52XQPTQ5LSFMHIPD5YW2YTO5YEFR36IYA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.561352025 (571h53m41.43243506s) [noise=true , address=77.90.31.18:4834 , piece_id=BTWNGNQVEUEVV6HLBDM63SGNTKV4RDU4J4XFLG52WHKP53H3L5KA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.622447380 (571h53m41.493530425s) [noise=true , address=77.90.31.16:4750 , piece_id=MVVZPJEGDCBSNVOB6CD2WUZGIJTL3VBHX2G4L3TVZ6JF5ZEJ5U6Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.645494685 (571h53m41.51657773s) [noise=true , address=77.90.31.21:4889 , piece_id=BR6Z55P3A7YJ5FJRDSZFUIF66FGIJQY7WCEQ7PL5DOYWL73OMKTQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.247930045 (571h53m42.11901308s) [noise=true , address=145.79.227.30:4777 , piece_id=VSCXLXGIBSFE6E4OHLWNZIZ75PO7WQO2VFHUYHW7JUNH62KGGU7Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.262221890 (571h53m42.133304925s) [noise=true , address=145.79.227.44:4205 , piece_id=UE4QSHDQBZ3XBWQ6I6MCRVTLMU7U6W6OZDA2ZGQCH6ZYRAGPD7WA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.295598123 (571h53m42.166681158s) [noise=true , address=145.79.227.35:4515 , piece_id=L7LFZKZEZERUNX4HUNDNQ2PWHXNJRJ7OBFRNMFIICPZEUIBHGP7A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.307481351 (571h53m42.178564386s) [noise=true , address=145.79.227.34:4910 , piece_id=ZH4DCNXET2Q6VRAUH3NQ3KKSV4OYI575X7YS72XD236EEOMOG3LA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.325775906 (571h53m42.196858941s) [noise=true , address=145.79.227.40:4067 , piece_id=WSYDRX5YJ6OCFW52ZVMM6WZYNRFUXMR4XJ6C5727BOFMR4RRH33Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.325865978 (571h53m42.196949023s) [seg_pos={0 0}, upload=501384]: Segment reached optimal threshold of 14 pieces. Dec 29 11:22:42.326137074 (571h53m42.197220109s) [noise=true , address=64.49.13.169:4708 , piece_id=B2X2RWRELXWJPF6AVJJZCNU46LM5ASGRE4EWGPKL3SCLQ4VKUBDQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection Dec 29 11:22:42.326153074 (571h53m42.197236109s) [noise=true , address=64.49.13.169:4708 , piece_id=B2X2RWRELXWJPF6AVJJZCNU46LM5ASGRE4EWGPKL3SCLQ4VKUBDQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326193401 (571h53m42.197276446s) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12h5esfg6mw2c8y3AVpeFBkL5FfLAwYdjhcnQ9CJ8fjh61WLYjx): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326205705 (571h53m42.19728874s) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326407739 (571h53m42.197490784s) [noise=true , address=77.90.31.10:4524 , piece_id=Q2JGEUV5EDFTVY2CSNPZOVJIIIAFBA3IHA7S5N7YZRH3E7TSTE5A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uN7CznoK4AD8FqFkNUndawdGJ9rmjAe7oNnjJkV844ScY87qe): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326415374 (571h53m42.197498419s) [noise=true , address=77.90.31.10:4524 , piece_id=Q2JGEUV5EDFTVY2CSNPZOVJIIIAFBA3IHA7S5N7YZRH3E7TSTE5A, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326505105 (571h53m42.19758815s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tnTEYvqJ6PB2DsuXA28sEuEiGj11StA4qdhcYH1kuoaMkTVdY): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326512188 (571h53m42.197595223s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326532357 (571h53m42.197615392s) [seg_pos={0 0}, upload=501384]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 11:22:42.326671562 (571h53m42.197754597s) [upload=501384]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 11:22:42.326677843 (571h53m42.197760879s) [upload=501384]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 11:22:42.610391438 (571h53m42.481474493s) [upload=501384]: Done uploading segments... Dec 29 11:22:42.610416826 (571h53m42.481499871s) [upload=501384]: Upload finished. err: <nil> Duration: 36.766531709s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [...X..X.....X.X...X.X.X.X.XX..XX..X..XXX.X.......X.X..X.XXX.] duration: 5.880737441s trace: https://jaeger.dev.storj.io/trace/3dd25a5fabd0957b FAIL (2025-12-29 11:36:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: WWAWENADBGDTJ6WDNLIG2DGQDGPE6JDWSUKDZRD2JI353CRA3ZEA) context deadline exceeded; piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: 5QLVW6DOT4KWQYBS6RUR4CCZ67V3LQGGMXJHIVEQHNUEI5FOW7GA) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: CXOPOZBQC6R6Q5G44EDXJBXFQBNBZ426PCIWLGXP7SS3HURY56KQ) context deadline exceeded ======LOGS====== Dec 29 11:35:56.459074447 (572h6m56.330157492s) [upload=501579]: Starting upload Dec 29 11:35:56.459138019 (572h6m56.330221054s) [upload=501579]: Done starting upload Dec 29 11:35:56.459205287 (572h6m56.330288322s) [upload=501579]: Uploading segments... Dec 29 11:35:56.459388165 (572h6m56.33047121s) [upload=501579]: Got next segment. Inline: false Dec 29 11:35:56.459425846 (572h6m56.330508891s) [upload=501579]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 11:35:56.459431608 (572h6m56.330514653s) [upload=501579]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 11:35:56.711093519 (572h6m56.582176554s) [seg_pos={0 0}, upload=501579]: Begin upload segment... Dec 29 11:35:56.711138224 (572h6m56.582221259s) [seg_pos={0 0}, upload=501579]: Done begin upload segment. Dec 29 11:35:56.711174664 (572h6m56.582257699s) [noise=true , address=145.79.227.51:4410 , piece_id=YXJGHLBQWCOIJCYOBR62YZOIOBVM2MFZ46HMGAKXQBDV6HQJEAMA, seg_pos={0 0}, upload=501579]: Uploading piece... Dec 29 11:35:56.711269364 (572h6m56.582352409s) [noise=true , address=145.79.227.48:4301 , piece_id=CYYRHB527AIKYVHLOGQDYUZKIBQ57DFBPZGMUFSOX4Z7FI7RP2CA, seg_pos={0 0}, upload=501579]: Uploading piece... Dec 29 11:35:56.711907739 (572h6m56.582990785s) [noise=true , address=145.79.227.18:4007 , piece_id=AHHHEKGANPKWUF36MOETE7RROQDB5NDYBIATQKX32LXSB627UGSQ, seg_pos={0 0}, upload=501579]: Uploading piece... Dec 29 11:35:56.712185939 (572h6m56.583268984s) [noise=true , address=45.91.116.10:4083 , piece_id=ZJB5XCBP3NHKUTCR7PTPU27CIZCAYNS75YLAPTIJOEUARUWAEY2Q, seg_pos={0 0}, upload=501579]: Uploading piece... Dec 29 11:35:56.713315771 (572h6m56.584398806s) [noise=true , address=45.91.116.4:4939 , piece_id=5B5EB3KIC4XXBW2HVTGSYWXI6O4AR7LCKP3CHRSRLTTQNB3QI3DA, seg_pos={0 0}, upload=501579]: Uploading piece... Dec 29 11:35:56.713357170 (572h6m56.584440205s) [noise=true , address=145.79.227.14:4838 , piece_id=I324MYN64UEHSH6WLDDSJSDIT5QG5JTD3MVGVCS5QCDKEK3TAI4Q, seg_pos={0 0}, upload=501579]: Uploading piece... Dec 29 11:35:56.714384246 (572h6m56.585467291s) [noise=true , address=45.91.116.8:4048 , piece_id=WWAWENADBGDTJ6WDNLIG2DGQDGPE6JDWSUKDZRD2JI353CRA3ZEA, seg_pos={0 0}, upload=501579]: Uploading piece... Dec 29 11:35:56.714462304 (572h6m56.585545359s) [noise=true , address=45.91.116.5:4297 , piece_id=5QLVW6DOT4KWQYBS6RUR4CCZ67V3LQGGMXJHIVEQHNUEI5FOW7GA, seg_pos={0 0}, upload=501579]: Uploading piece... Dec 29 11:35:56.715383739 (572h6m56.586466774s) [noise=true , address=45.91.116.6:4975 , piece_id=CXOPOZBQC6R6Q5G44EDXJBXFQBNBZ426PCIWLGXP7SS3HURY56KQ, seg_pos={0 0}, upload=501579]: Uploading piece... Dec 29 11:35:56.715475003 (572h6m56.586558038s) [noise=true , address=145.79.227.49:4337 , piece_id=7PF7B4RWEHDZBS6GD6XFYTHIR5JTPJBZCUXE2HG7D4TJD4GQD6UA, seg_pos={0 0}, upload=501579]: Uploading piece... Dec 29 11:35:56.716386869 (572h6m56.587469904s) [noise=true , address=45.91.116.13:4155 , piece_id=JJHI4DZMMVCQZHHWLLS6WMTCGZDFCZOYNX53TKOXAB2YRFJPHJ2A, seg_pos={0 0}, upload=501579]: Uploading piece... Dec 29 11:35:56.716459126 (572h6m56.587542161s) [noise=true , address=145.79.227.4:4185 , piece_id=V4FK2BIHYDHHP3UBLHOFMHSH6NCL7ZYRAEO2XRB726X32HM75E2A, seg_pos={0 0}, upload=501579]: Uploading piece... Dec 29 11:35:56.717277685 (572h6m56.588360731s) [noise=true , address=145.79.227.52:4446 , piece_id=OKHNVKLC55SA3VPJ4OFBMKJKCQ2E3TSCZXRZHVGO5K2Y5PH6FWBQ, seg_pos={0 0}, upload=501579]: Uploading piece... Dec 29 11:35:56.717673769 (572h6m56.588756804s) [noise=true , address=45.91.116.7:4009 , piece_id=6IVOZQ5AOE45W5ZWIQXA6KC4YU775DHAOQBH6ZO77YJPLZPGHQGA, seg_pos={0 0}, upload=501579]: Uploading piece... Dec 29 11:35:56.717895341 (572h6m56.588978426s) [noise=true , address=145.79.227.26:4655 , piece_id=XXJMUKL5PFEHXZG6JLMLNO63GQUJCO2HS6GEQAU6RL36IL2M2SWQ, seg_pos={0 0}, upload=501579]: Uploading piece... Dec 29 11:35:56.718323988 (572h6m56.589407023s) [noise=true , address=45.91.116.12:4225 , piece_id=NZS3EUF2BFLLEJ734IBT5RRZF2ZMRC4YYPSL2QD7K7X5MD765BJA, seg_pos={0 0}, upload=501579]: Uploading piece... Dec 29 11:35:58.324261742 (572h6m58.195344777s) [upload=501579]: Next returned nil segment Dec 29 11:35:58.324275217 (572h6m58.195358253s) [upload=501579]: Waiting for error group managing segments... Dec 29 11:35:58.706737693 (572h6m58.577820728s) [noise=true , address=145.79.227.14:4838 , piece_id=I324MYN64UEHSH6WLDDSJSDIT5QG5JTD3MVGVCS5QCDKEK3TAI4Q, seg_pos={0 0}, upload=501579]: Done uploading piece. err: <nil> Dec 29 11:35:58.706821933 (572h6m58.577904978s) [noise=true , address=145.79.227.18:4007 , piece_id=AHHHEKGANPKWUF36MOETE7RROQDB5NDYBIATQKX32LXSB627UGSQ, seg_pos={0 0}, upload=501579]: Done uploading piece. err: <nil> Dec 29 11:35:58.707328368 (572h6m58.578411403s) [noise=true , address=145.79.227.4:4185 , piece_id=V4FK2BIHYDHHP3UBLHOFMHSH6NCL7ZYRAEO2XRB726X32HM75E2A, seg_pos={0 0}, upload=501579]: Done uploading piece. err: <nil> Dec 29 11:35:58.710997875 (572h6m58.58208091s) [noise=true , address=145.79.227.52:4446 , piece_id=OKHNVKLC55SA3VPJ4OFBMKJKCQ2E3TSCZXRZHVGO5K2Y5PH6FWBQ, seg_pos={0 0}, upload=501579]: Done uploading piece. err: <nil> Dec 29 11:35:58.711306833 (572h6m58.582389868s) [noise=true , address=145.79.227.26:4655 , piece_id=XXJMUKL5PFEHXZG6JLMLNO63GQUJCO2HS6GEQAU6RL36IL2M2SWQ, seg_pos={0 0}, upload=501579]: Done uploading piece. err: <nil> Dec 29 11:35:58.725093245 (572h6m58.59617629s) [noise=true , address=145.79.227.51:4410 , piece_id=YXJGHLBQWCOIJCYOBR62YZOIOBVM2MFZ46HMGAKXQBDV6HQJEAMA, seg_pos={0 0}, upload=501579]: Done uploading piece. err: <nil> Dec 29 11:35:58.726268482 (572h6m58.597351517s) [noise=true , address=145.79.227.48:4301 , piece_id=CYYRHB527AIKYVHLOGQDYUZKIBQ57DFBPZGMUFSOX4Z7FI7RP2CA, seg_pos={0 0}, upload=501579]: Done uploading piece. err: <nil> Dec 29 11:35:58.973990231 (572h6m58.845073266s) [noise=true , address=45.91.116.8:4048 , piece_id=WWAWENADBGDTJ6WDNLIG2DGQDGPE6JDWSUKDZRD2JI353CRA3ZEA, seg_pos={0 0}, upload=501579]: Done uploading piece. err: <nil> Dec 29 11:35:58.974561478 (572h6m58.845644523s) [noise=true , address=45.91.116.4:4939 , piece_id=5B5EB3KIC4XXBW2HVTGSYWXI6O4AR7LCKP3CHRSRLTTQNB3QI3DA, seg_pos={0 0}, upload=501579]: Done uploading piece. err: <nil> Dec 29 11:35:59.003043162 (572h6m58.874126207s) [noise=true , address=45.91.116.5:4297 , piece_id=5QLVW6DOT4KWQYBS6RUR4CCZ67V3LQGGMXJHIVEQHNUEI5FOW7GA, seg_pos={0 0}, upload=501579]: Done uploading piece. err: <nil> Dec 29 11:35:59.122220202 (572h6m58.993303237s) [noise=true , address=45.91.116.12:4225 , piece_id=NZS3EUF2BFLLEJ734IBT5RRZF2ZMRC4YYPSL2QD7K7X5MD765BJA, seg_pos={0 0}, upload=501579]: Done uploading piece. err: <nil> Dec 29 11:35:59.215783370 (572h6m59.086866405s) [noise=true , address=45.91.116.7:4009 , piece_id=6IVOZQ5AOE45W5ZWIQXA6KC4YU775DHAOQBH6ZO77YJPLZPGHQGA, seg_pos={0 0}, upload=501579]: Done uploading piece. err: <nil> Dec 29 11:35:59.686524179 (572h6m59.557607214s) [noise=true , address=45.91.116.6:4975 , piece_id=CXOPOZBQC6R6Q5G44EDXJBXFQBNBZ426PCIWLGXP7SS3HURY56KQ, seg_pos={0 0}, upload=501579]: Done uploading piece. err: <nil> Dec 29 11:35:59.686542454 (572h6m59.557625499s) [seg_pos={0 0}, upload=501579]: Segment reached optimal threshold of 13 pieces. Dec 29 11:35:59.686724450 (572h6m59.557807495s) [noise=true , address=145.79.227.49:4337 , piece_id=7PF7B4RWEHDZBS6GD6XFYTHIR5JTPJBZCUXE2HG7D4TJD4GQD6UA, seg_pos={0 0}, upload=501579]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE): protocol: expected piece hash; context canceled; EOF Dec 29 11:35:59.686730231 (572h6m59.557813276s) [noise=true , address=145.79.227.49:4337 , piece_id=7PF7B4RWEHDZBS6GD6XFYTHIR5JTPJBZCUXE2HG7D4TJD4GQD6UA, seg_pos={0 0}, upload=501579]: Long tail cancellation due to optimal threshold Dec 29 11:35:59.686832947 (572h6m59.557915982s) [noise=true , address=45.91.116.13:4155 , piece_id=JJHI4DZMMVCQZHHWLLS6WMTCGZDFCZOYNX53TKOXAB2YRFJPHJ2A, seg_pos={0 0}, upload=501579]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 29 11:35:59.686837585 (572h6m59.557920621s) [noise=true , address=45.91.116.13:4155 , piece_id=JJHI4DZMMVCQZHHWLLS6WMTCGZDFCZOYNX53TKOXAB2YRFJPHJ2A, seg_pos={0 0}, upload=501579]: Long tail cancellation due to optimal threshold Dec 29 11:35:59.686944549 (572h6m59.558027594s) [noise=true , address=45.91.116.10:4083 , piece_id=ZJB5XCBP3NHKUTCR7PTPU27CIZCAYNS75YLAPTIJOEUARUWAEY2Q, seg_pos={0 0}, upload=501579]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 29 11:35:59.686949909 (572h6m59.558032944s) [noise=true , address=45.91.116.10:4083 , piece_id=ZJB5XCBP3NHKUTCR7PTPU27CIZCAYNS75YLAPTIJOEUARUWAEY2Q, seg_pos={0 0}, upload=501579]: Long tail cancellation due to optimal threshold Dec 29 11:35:59.686983924 (572h6m59.558066959s) [seg_pos={0 0}, upload=501579]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 11:35:59.687060260 (572h6m59.558143305s) [upload=501579]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 11:35:59.687064809 (572h6m59.558147844s) [upload=501579]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 11:36:00.006171134 (572h6m59.877254169s) [upload=501579]: Done uploading segments... Dec 29 11:36:00.006192696 (572h6m59.877275741s) [upload=501579]: Upload finished. err: <nil> Duration: 5.880752199s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 17 out of 60 is failed (threshold=6) [..X..........X....X.......X..X...XX..XX......X.X.XX.X.XX..X.] duration: 3.475496347s trace: https://jaeger.dev.storj.io/trace/7cbeba427ba3458a FAIL (2025-12-29 11:36:26 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.475524361s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [....X..X.....X.X...X.X.X.X.XX..XX..X..XXX.X.......X.X..X.XXX] duration: 30.002857574s trace: https://jaeger.dev.storj.io/trace/3dd25a5fabd0957b FAIL (2025-12-29 11:36:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: WWAWENADBGDTJ6WDNLIG2DGQDGPE6JDWSUKDZRD2JI353CRA3ZEA) context deadline exceeded; piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: 5QLVW6DOT4KWQYBS6RUR4CCZ67V3LQGGMXJHIVEQHNUEI5FOW7GA) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: CXOPOZBQC6R6Q5G44EDXJBXFQBNBZ426PCIWLGXP7SS3HURY56KQ) context deadline exceeded ======LOGS====== Dec 29 11:35:56.459074447 (572h6m56.330157492s) [upload=501579]: Starting upload Dec 29 11:35:56.459138019 (572h6m56.330221054s) [upload=501579]: Done starting upload Dec 29 11:35:56.459205287 (572h6m56.330288322s) [upload=501579]: Uploading segments... Dec 29 11:35:56.459388165 (572h6m56.33047121s) [upload=501579]: Got next segment. Inline: false Dec 29 11:35:56.459425846 (572h6m56.330508891s) [upload=501579]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 11:35:56.459431608 (572h6m56.330514653s) [upload=501579]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 11:35:56.711093519 (572h6m56.582176554s) [seg_pos={0 0}, upload=501579]: Begin upload segment... Dec 29 11:35:56.711138224 (572h6m56.582221259s) [seg_pos={0 0}, upload=501579]: Done begin upload segment. Dec 29 11:35:56.711174664 (572h6m56.582257699s) [noise=true , address=145.79.227.51:4410 , piece_id=YXJGHLBQWCOIJCYOBR62YZOIOBVM2MFZ46HMGAKXQBDV6HQJEAMA, seg_pos={0 0}, upload=501579]: Uploading piece... Dec 29 11:35:56.711269364 (572h6m56.582352409s) [noise=true , address=145.79.227.48:4301 , piece_id=CYYRHB527AIKYVHLOGQDYUZKIBQ57DFBPZGMUFSOX4Z7FI7RP2CA, seg_pos={0 0}, upload=501579]: Uploading piece... Dec 29 11:35:56.711907739 (572h6m56.582990785s) [noise=true , address=145.79.227.18:4007 , piece_id=AHHHEKGANPKWUF36MOETE7RROQDB5NDYBIATQKX32LXSB627UGSQ, seg_pos={0 0}, upload=501579]: Uploading piece... Dec 29 11:35:56.712185939 (572h6m56.583268984s) [noise=true , address=45.91.116.10:4083 , piece_id=ZJB5XCBP3NHKUTCR7PTPU27CIZCAYNS75YLAPTIJOEUARUWAEY2Q, seg_pos={0 0}, upload=501579]: Uploading piece... Dec 29 11:35:56.713315771 (572h6m56.584398806s) [noise=true , address=45.91.116.4:4939 , piece_id=5B5EB3KIC4XXBW2HVTGSYWXI6O4AR7LCKP3CHRSRLTTQNB3QI3DA, seg_pos={0 0}, upload=501579]: Uploading piece... Dec 29 11:35:56.713357170 (572h6m56.584440205s) [noise=true , address=145.79.227.14:4838 , piece_id=I324MYN64UEHSH6WLDDSJSDIT5QG5JTD3MVGVCS5QCDKEK3TAI4Q, seg_pos={0 0}, upload=501579]: Uploading piece... Dec 29 11:35:56.714384246 (572h6m56.585467291s) [noise=true , address=45.91.116.8:4048 , piece_id=WWAWENADBGDTJ6WDNLIG2DGQDGPE6JDWSUKDZRD2JI353CRA3ZEA, seg_pos={0 0}, upload=501579]: Uploading piece... Dec 29 11:35:56.714462304 (572h6m56.585545359s) [noise=true , address=45.91.116.5:4297 , piece_id=5QLVW6DOT4KWQYBS6RUR4CCZ67V3LQGGMXJHIVEQHNUEI5FOW7GA, seg_pos={0 0}, upload=501579]: Uploading piece... Dec 29 11:35:56.715383739 (572h6m56.586466774s) [noise=true , address=45.91.116.6:4975 , piece_id=CXOPOZBQC6R6Q5G44EDXJBXFQBNBZ426PCIWLGXP7SS3HURY56KQ, seg_pos={0 0}, upload=501579]: Uploading piece... Dec 29 11:35:56.715475003 (572h6m56.586558038s) [noise=true , address=145.79.227.49:4337 , piece_id=7PF7B4RWEHDZBS6GD6XFYTHIR5JTPJBZCUXE2HG7D4TJD4GQD6UA, seg_pos={0 0}, upload=501579]: Uploading piece... Dec 29 11:35:56.716386869 (572h6m56.587469904s) [noise=true , address=45.91.116.13:4155 , piece_id=JJHI4DZMMVCQZHHWLLS6WMTCGZDFCZOYNX53TKOXAB2YRFJPHJ2A, seg_pos={0 0}, upload=501579]: Uploading piece... Dec 29 11:35:56.716459126 (572h6m56.587542161s) [noise=true , address=145.79.227.4:4185 , piece_id=V4FK2BIHYDHHP3UBLHOFMHSH6NCL7ZYRAEO2XRB726X32HM75E2A, seg_pos={0 0}, upload=501579]: Uploading piece... Dec 29 11:35:56.717277685 (572h6m56.588360731s) [noise=true , address=145.79.227.52:4446 , piece_id=OKHNVKLC55SA3VPJ4OFBMKJKCQ2E3TSCZXRZHVGO5K2Y5PH6FWBQ, seg_pos={0 0}, upload=501579]: Uploading piece... Dec 29 11:35:56.717673769 (572h6m56.588756804s) [noise=true , address=45.91.116.7:4009 , piece_id=6IVOZQ5AOE45W5ZWIQXA6KC4YU775DHAOQBH6ZO77YJPLZPGHQGA, seg_pos={0 0}, upload=501579]: Uploading piece... Dec 29 11:35:56.717895341 (572h6m56.588978426s) [noise=true , address=145.79.227.26:4655 , piece_id=XXJMUKL5PFEHXZG6JLMLNO63GQUJCO2HS6GEQAU6RL36IL2M2SWQ, seg_pos={0 0}, upload=501579]: Uploading piece... Dec 29 11:35:56.718323988 (572h6m56.589407023s) [noise=true , address=45.91.116.12:4225 , piece_id=NZS3EUF2BFLLEJ734IBT5RRZF2ZMRC4YYPSL2QD7K7X5MD765BJA, seg_pos={0 0}, upload=501579]: Uploading piece... Dec 29 11:35:58.324261742 (572h6m58.195344777s) [upload=501579]: Next returned nil segment Dec 29 11:35:58.324275217 (572h6m58.195358253s) [upload=501579]: Waiting for error group managing segments... Dec 29 11:35:58.706737693 (572h6m58.577820728s) [noise=true , address=145.79.227.14:4838 , piece_id=I324MYN64UEHSH6WLDDSJSDIT5QG5JTD3MVGVCS5QCDKEK3TAI4Q, seg_pos={0 0}, upload=501579]: Done uploading piece. err: <nil> Dec 29 11:35:58.706821933 (572h6m58.577904978s) [noise=true , address=145.79.227.18:4007 , piece_id=AHHHEKGANPKWUF36MOETE7RROQDB5NDYBIATQKX32LXSB627UGSQ, seg_pos={0 0}, upload=501579]: Done uploading piece. err: <nil> Dec 29 11:35:58.707328368 (572h6m58.578411403s) [noise=true , address=145.79.227.4:4185 , piece_id=V4FK2BIHYDHHP3UBLHOFMHSH6NCL7ZYRAEO2XRB726X32HM75E2A, seg_pos={0 0}, upload=501579]: Done uploading piece. err: <nil> Dec 29 11:35:58.710997875 (572h6m58.58208091s) [noise=true , address=145.79.227.52:4446 , piece_id=OKHNVKLC55SA3VPJ4OFBMKJKCQ2E3TSCZXRZHVGO5K2Y5PH6FWBQ, seg_pos={0 0}, upload=501579]: Done uploading piece. err: <nil> Dec 29 11:35:58.711306833 (572h6m58.582389868s) [noise=true , address=145.79.227.26:4655 , piece_id=XXJMUKL5PFEHXZG6JLMLNO63GQUJCO2HS6GEQAU6RL36IL2M2SWQ, seg_pos={0 0}, upload=501579]: Done uploading piece. err: <nil> Dec 29 11:35:58.725093245 (572h6m58.59617629s) [noise=true , address=145.79.227.51:4410 , piece_id=YXJGHLBQWCOIJCYOBR62YZOIOBVM2MFZ46HMGAKXQBDV6HQJEAMA, seg_pos={0 0}, upload=501579]: Done uploading piece. err: <nil> Dec 29 11:35:58.726268482 (572h6m58.597351517s) [noise=true , address=145.79.227.48:4301 , piece_id=CYYRHB527AIKYVHLOGQDYUZKIBQ57DFBPZGMUFSOX4Z7FI7RP2CA, seg_pos={0 0}, upload=501579]: Done uploading piece. err: <nil> Dec 29 11:35:58.973990231 (572h6m58.845073266s) [noise=true , address=45.91.116.8:4048 , piece_id=WWAWENADBGDTJ6WDNLIG2DGQDGPE6JDWSUKDZRD2JI353CRA3ZEA, seg_pos={0 0}, upload=501579]: Done uploading piece. err: <nil> Dec 29 11:35:58.974561478 (572h6m58.845644523s) [noise=true , address=45.91.116.4:4939 , piece_id=5B5EB3KIC4XXBW2HVTGSYWXI6O4AR7LCKP3CHRSRLTTQNB3QI3DA, seg_pos={0 0}, upload=501579]: Done uploading piece. err: <nil> Dec 29 11:35:59.003043162 (572h6m58.874126207s) [noise=true , address=45.91.116.5:4297 , piece_id=5QLVW6DOT4KWQYBS6RUR4CCZ67V3LQGGMXJHIVEQHNUEI5FOW7GA, seg_pos={0 0}, upload=501579]: Done uploading piece. err: <nil> Dec 29 11:35:59.122220202 (572h6m58.993303237s) [noise=true , address=45.91.116.12:4225 , piece_id=NZS3EUF2BFLLEJ734IBT5RRZF2ZMRC4YYPSL2QD7K7X5MD765BJA, seg_pos={0 0}, upload=501579]: Done uploading piece. err: <nil> Dec 29 11:35:59.215783370 (572h6m59.086866405s) [noise=true , address=45.91.116.7:4009 , piece_id=6IVOZQ5AOE45W5ZWIQXA6KC4YU775DHAOQBH6ZO77YJPLZPGHQGA, seg_pos={0 0}, upload=501579]: Done uploading piece. err: <nil> Dec 29 11:35:59.686524179 (572h6m59.557607214s) [noise=true , address=45.91.116.6:4975 , piece_id=CXOPOZBQC6R6Q5G44EDXJBXFQBNBZ426PCIWLGXP7SS3HURY56KQ, seg_pos={0 0}, upload=501579]: Done uploading piece. err: <nil> Dec 29 11:35:59.686542454 (572h6m59.557625499s) [seg_pos={0 0}, upload=501579]: Segment reached optimal threshold of 13 pieces. Dec 29 11:35:59.686724450 (572h6m59.557807495s) [noise=true , address=145.79.227.49:4337 , piece_id=7PF7B4RWEHDZBS6GD6XFYTHIR5JTPJBZCUXE2HG7D4TJD4GQD6UA, seg_pos={0 0}, upload=501579]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE): protocol: expected piece hash; context canceled; EOF Dec 29 11:35:59.686730231 (572h6m59.557813276s) [noise=true , address=145.79.227.49:4337 , piece_id=7PF7B4RWEHDZBS6GD6XFYTHIR5JTPJBZCUXE2HG7D4TJD4GQD6UA, seg_pos={0 0}, upload=501579]: Long tail cancellation due to optimal threshold Dec 29 11:35:59.686832947 (572h6m59.557915982s) [noise=true , address=45.91.116.13:4155 , piece_id=JJHI4DZMMVCQZHHWLLS6WMTCGZDFCZOYNX53TKOXAB2YRFJPHJ2A, seg_pos={0 0}, upload=501579]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 29 11:35:59.686837585 (572h6m59.557920621s) [noise=true , address=45.91.116.13:4155 , piece_id=JJHI4DZMMVCQZHHWLLS6WMTCGZDFCZOYNX53TKOXAB2YRFJPHJ2A, seg_pos={0 0}, upload=501579]: Long tail cancellation due to optimal threshold Dec 29 11:35:59.686944549 (572h6m59.558027594s) [noise=true , address=45.91.116.10:4083 , piece_id=ZJB5XCBP3NHKUTCR7PTPU27CIZCAYNS75YLAPTIJOEUARUWAEY2Q, seg_pos={0 0}, upload=501579]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 29 11:35:59.686949909 (572h6m59.558032944s) [noise=true , address=45.91.116.10:4083 , piece_id=ZJB5XCBP3NHKUTCR7PTPU27CIZCAYNS75YLAPTIJOEUARUWAEY2Q, seg_pos={0 0}, upload=501579]: Long tail cancellation due to optimal threshold Dec 29 11:35:59.686983924 (572h6m59.558066959s) [seg_pos={0 0}, upload=501579]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 11:35:59.687060260 (572h6m59.558143305s) [upload=501579]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 11:35:59.687064809 (572h6m59.558147844s) [upload=501579]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 11:36:00.006171134 (572h6m59.877254169s) [upload=501579]: Done uploading segments... Dec 29 11:36:00.006192696 (572h6m59.877275741s) [upload=501579]: Upload finished. err: <nil> Duration: 30.002868695s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 17 out of 60 is failed (threshold=6) [...X..........X....X.......X..X...XX..XX......X.X.XX.X.XX..X] duration: 5.00190475s trace: https://jaeger.dev.storj.io/trace/7cbeba427ba3458a FAIL (2025-12-29 11:36:05 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001927033s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [X....X..X.....X.X...X.X.X.X.XX..XX..X..XXX.X.......X.X..X.XX] duration: 30.005062492s trace: https://jaeger.dev.storj.io/trace/14da4ef2a8727853 FAIL (2025-12-29 11:35:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 11:35:26.196771203 (572h6m26.067854248s) [upload=501571]: Starting upload Dec 29 11:35:26.196825296 (572h6m26.067908331s) [upload=501571]: Done starting upload Dec 29 11:35:26.196936998 (572h6m26.068020033s) [upload=501571]: Uploading segments... Dec 29 11:35:26.197185111 (572h6m26.068268156s) [upload=501571]: Got next segment. Inline: false Dec 29 11:35:26.197273710 (572h6m26.068356745s) [upload=501571]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 11:35:26.197281294 (572h6m26.068364339s) [upload=501571]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 11:35:26.443311092 (572h6m26.314394137s) [seg_pos={0 0}, upload=501571]: Begin upload segment... Dec 29 11:35:26.443367999 (572h6m26.314451034s) [seg_pos={0 0}, upload=501571]: Done begin upload segment. Dec 29 11:35:26.443384100 (572h6m26.314467146s) [noise=true , address=145.79.227.43:4159 , piece_id=WZVWYC542MB4SVCCTSB32MBLS5BRFLJXRGFUCV25LBCQZIOVONAQ, seg_pos={0 0}, upload=501571]: Uploading piece... Dec 29 11:35:26.443404649 (572h6m26.314487714s) [noise=true , address=145.79.227.52:4446 , piece_id=55QKFHNYXJUATMRIUHKNW3OBOOGANFVAG3GR7M6JJPIUFHMDPOTA, seg_pos={0 0}, upload=501571]: Uploading piece... Dec 29 11:35:26.443468761 (572h6m26.314551796s) [noise=true , address=45.91.116.7:4010 , piece_id=JODJLSJG32TIR34N6TQ6MX224WVKY362X5OPYBER7UNSJRQOGQRA, seg_pos={0 0}, upload=501571]: Uploading piece... Dec 29 11:35:26.444035039 (572h6m26.315118084s) [noise=true , address=145.79.227.6:4258 , piece_id=WPQMQUVY46PDXNK6V2PQRWA6RHRP6ZKRVJVZQL2QRFMW4O7UQZNQ, seg_pos={0 0}, upload=501571]: Uploading piece... Dec 29 11:35:26.444557614 (572h6m26.315640659s) [noise=true , address=145.79.227.41:4085 , piece_id=OTLVVQLBFV2QOYE2FYMPG7ZZYGPHTY366TO5QKYP3WCPBQQYP3QA, seg_pos={0 0}, upload=501571]: Uploading piece... Dec 29 11:35:26.445094157 (572h6m26.316177202s) [noise=true , address=145.79.227.38:4977 , piece_id=PXTQOPK7FODIJVGYLDNOETJ7D74MK4AHL74VF35YJP4FSH3IJXFA, seg_pos={0 0}, upload=501571]: Uploading piece... Dec 29 11:35:26.445606331 (572h6m26.316689757s) [noise=true , address=145.79.227.11:4731 , piece_id=JIOIQETWHH3AMF5VR3QXYTYZBKIRUTYIDQ7KJ6ZOKGOQDIMPMOZA, seg_pos={0 0}, upload=501571]: Uploading piece... Dec 29 11:35:26.446166808 (572h6m26.317249843s) [noise=true , address=145.79.227.50:4373 , piece_id=47HCC4XOXOKTR2RII6AZ22NI5T5456SPNAG3U2XHDH3AH2F7CJKQ, seg_pos={0 0}, upload=501571]: Uploading piece... Dec 29 11:35:26.446827857 (572h6m26.317910912s) [noise=true , address=145.79.227.44:4194 , piece_id=HFWKL5N5XHLWJPI663KPYZX2OAP3U4V76632JACPBBX3LPZU6LAQ, seg_pos={0 0}, upload=501571]: Uploading piece... Dec 29 11:35:26.448743145 (572h6m26.3198262s) [noise=true , address=77.90.31.21:4877 , piece_id=B76KG5YPHZ4WZ7IXVNMHSZ3J2UD2UAXKGSFWLFYJYTOJOTRQZLHQ, seg_pos={0 0}, upload=501571]: Uploading piece... Dec 29 11:35:26.450975426 (572h6m26.322058481s) [noise=true , address=145.79.227.34:4906 , piece_id=LFPYSZR3L3BFFWTOZL5XBUGSD7ZTFPYYMQUX7DTFN3V5SOYMZRIA, seg_pos={0 0}, upload=501571]: Uploading piece... Dec 29 11:35:26.453208958 (572h6m26.324292224s) [noise=true , address=145.79.227.17:4971 , piece_id=XLMVQENGUGBSAIRLN2RWCB4VTWTGK4XPX5PHWAZJ4T3AUI7WUG5Q, seg_pos={0 0}, upload=501571]: Uploading piece... Dec 29 11:35:26.454045010 (572h6m26.325128055s) [noise=true , address=145.79.227.14:4840 , piece_id=R67MJHEH6SOYJWX7H4372OEOSW3UHKKFOBS2HKQXWJX5ZINWGRXQ, seg_pos={0 0}, upload=501571]: Uploading piece... Dec 29 11:35:26.455479221 (572h6m26.326562266s) [noise=true , address=145.79.227.4:4186 , piece_id=MDYGNWTJH7D63EFZBXZ7ENHT4F5UJT5G5DKPWUEVEFWH62NGOXSA, seg_pos={0 0}, upload=501571]: Uploading piece... Dec 29 11:35:26.456623140 (572h6m26.327706175s) [noise=true , address=45.91.116.5:4299 , piece_id=NE45NIPY3QC75VUPUARIYMTVVINFZUVOYXHQVF5WXJZCLASEVK7Q, seg_pos={0 0}, upload=501571]: Uploading piece... Dec 29 11:35:26.457654634 (572h6m26.328737669s) [noise=true , address=145.79.227.51:4411 , piece_id=2ZZ5CKC6CSPCVH3LC7QJBES6YDR4O5UIZ76UCRB3DUTQG6VORPZQ, seg_pos={0 0}, upload=501571]: Uploading piece... Dec 29 11:35:28.142238912 (572h6m28.013321947s) [upload=501571]: Next returned nil segment Dec 29 11:35:28.142252959 (572h6m28.013335994s) [upload=501571]: Waiting for error group managing segments... Dec 29 11:35:28.424963582 (572h6m28.296046617s) [noise=true , address=145.79.227.17:4971 , piece_id=XLMVQENGUGBSAIRLN2RWCB4VTWTGK4XPX5PHWAZJ4T3AUI7WUG5Q, seg_pos={0 0}, upload=501571]: Done uploading piece. err: <nil> Dec 29 11:35:28.434247971 (572h6m28.305331006s) [noise=true , address=145.79.227.41:4085 , piece_id=OTLVVQLBFV2QOYE2FYMPG7ZZYGPHTY366TO5QKYP3WCPBQQYP3QA, seg_pos={0 0}, upload=501571]: Done uploading piece. err: <nil> Dec 29 11:35:28.440045510 (572h6m28.311128555s) [noise=true , address=145.79.227.51:4411 , piece_id=2ZZ5CKC6CSPCVH3LC7QJBES6YDR4O5UIZ76UCRB3DUTQG6VORPZQ, seg_pos={0 0}, upload=501571]: Done uploading piece. err: <nil> Dec 29 11:35:28.440164546 (572h6m28.311247581s) [noise=true , address=145.79.227.50:4373 , piece_id=47HCC4XOXOKTR2RII6AZ22NI5T5456SPNAG3U2XHDH3AH2F7CJKQ, seg_pos={0 0}, upload=501571]: Done uploading piece. err: <nil> Dec 29 11:35:28.441986866 (572h6m28.313069901s) [noise=true , address=145.79.227.52:4446 , piece_id=55QKFHNYXJUATMRIUHKNW3OBOOGANFVAG3GR7M6JJPIUFHMDPOTA, seg_pos={0 0}, upload=501571]: Done uploading piece. err: <nil> Dec 29 11:35:28.443239772 (572h6m28.314322817s) [noise=true , address=145.79.227.6:4258 , piece_id=WPQMQUVY46PDXNK6V2PQRWA6RHRP6ZKRVJVZQL2QRFMW4O7UQZNQ, seg_pos={0 0}, upload=501571]: Done uploading piece. err: <nil> Dec 29 11:35:28.457680199 (572h6m28.328763244s) [noise=true , address=145.79.227.14:4840 , piece_id=R67MJHEH6SOYJWX7H4372OEOSW3UHKKFOBS2HKQXWJX5ZINWGRXQ, seg_pos={0 0}, upload=501571]: Done uploading piece. err: <nil> Dec 29 11:35:28.458497385 (572h6m28.32958042s) [noise=true , address=145.79.227.4:4186 , piece_id=MDYGNWTJH7D63EFZBXZ7ENHT4F5UJT5G5DKPWUEVEFWH62NGOXSA, seg_pos={0 0}, upload=501571]: Done uploading piece. err: <nil> Dec 29 11:35:28.462093503 (572h6m28.333176538s) [noise=true , address=145.79.227.11:4731 , piece_id=JIOIQETWHH3AMF5VR3QXYTYZBKIRUTYIDQ7KJ6ZOKGOQDIMPMOZA, seg_pos={0 0}, upload=501571]: Done uploading piece. err: <nil> Dec 29 11:35:28.462968048 (572h6m28.334051113s) [noise=true , address=145.79.227.38:4977 , piece_id=PXTQOPK7FODIJVGYLDNOETJ7D74MK4AHL74VF35YJP4FSH3IJXFA, seg_pos={0 0}, upload=501571]: Done uploading piece. err: <nil> Dec 29 11:35:28.691707401 (572h6m28.562790446s) [noise=true , address=77.90.31.21:4877 , piece_id=B76KG5YPHZ4WZ7IXVNMHSZ3J2UD2UAXKGSFWLFYJYTOJOTRQZLHQ, seg_pos={0 0}, upload=501571]: Done uploading piece. err: <nil> Dec 29 11:35:28.717389013 (572h6m28.588472058s) [noise=true , address=45.91.116.7:4010 , piece_id=JODJLSJG32TIR34N6TQ6MX224WVKY362X5OPYBER7UNSJRQOGQRA, seg_pos={0 0}, upload=501571]: Done uploading piece. err: <nil> Dec 29 11:35:29.743693769 (572h6m29.614776814s) [noise=true , address=45.91.116.5:4299 , piece_id=NE45NIPY3QC75VUPUARIYMTVVINFZUVOYXHQVF5WXJZCLASEVK7Q, seg_pos={0 0}, upload=501571]: Done uploading piece. err: <nil> Dec 29 11:35:29.743718536 (572h6m29.614801571s) [seg_pos={0 0}, upload=501571]: Segment reached optimal threshold of 13 pieces. Dec 29 11:35:29.743981516 (572h6m29.615064551s) [noise=true , address=145.79.227.43:4159 , piece_id=WZVWYC542MB4SVCCTSB32MBLS5BRFLJXRGFUCV25LBCQZIOVONAQ, seg_pos={0 0}, upload=501571]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Dec 29 11:35:29.743992618 (572h6m29.615075673s) [noise=true , address=145.79.227.43:4159 , piece_id=WZVWYC542MB4SVCCTSB32MBLS5BRFLJXRGFUCV25LBCQZIOVONAQ, seg_pos={0 0}, upload=501571]: Long tail cancellation due to optimal threshold Dec 29 11:35:29.744091126 (572h6m29.615174171s) [noise=true , address=145.79.227.34:4906 , piece_id=LFPYSZR3L3BFFWTOZL5XBUGSD7ZTFPYYMQUX7DTFN3V5SOYMZRIA, seg_pos={0 0}, upload=501571]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 29 11:35:29.744106835 (572h6m29.61518987s) [noise=true , address=145.79.227.34:4906 , piece_id=LFPYSZR3L3BFFWTOZL5XBUGSD7ZTFPYYMQUX7DTFN3V5SOYMZRIA, seg_pos={0 0}, upload=501571]: Long tail cancellation due to optimal threshold Dec 29 11:35:29.744266911 (572h6m29.615349956s) [noise=true , address=145.79.227.44:4194 , piece_id=HFWKL5N5XHLWJPI663KPYZX2OAP3U4V76632JACPBBX3LPZU6LAQ, seg_pos={0 0}, upload=501571]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 29 11:35:29.744273283 (572h6m29.615356328s) [noise=true , address=145.79.227.44:4194 , piece_id=HFWKL5N5XHLWJPI663KPYZX2OAP3U4V76632JACPBBX3LPZU6LAQ, seg_pos={0 0}, upload=501571]: Long tail cancellation due to optimal threshold Dec 29 11:35:29.744303731 (572h6m29.615386766s) [seg_pos={0 0}, upload=501571]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 11:35:29.744383122 (572h6m29.615466167s) [upload=501571]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 11:35:29.744388001 (572h6m29.615471036s) [upload=501571]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 11:35:30.044010086 (572h6m29.915093121s) [upload=501571]: Done uploading segments... Dec 29 11:35:30.044030093 (572h6m29.915113128s) [upload=501571]: Upload finished. err: <nil> Duration: 30.005076039s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [........................X...............X..X................] duration: 36.916355068s trace: https://jaeger.dev.storj.io/trace/7e7354d8023608a8 FAIL (2025-12-29 11:35:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== ) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480097317 (571h53m39.351180352s) [noise=true , address=77.90.31.21:4889 , piece_id=BR6Z55P3A7YJ5FJRDSZFUIF66FGIJQY7WCEQ7PL5DOYWL73OMKTQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480405053 (571h53m39.351488098s) [noise=true , address=77.90.31.16:4750 , piece_id=MVVZPJEGDCBSNVOB6CD2WUZGIJTL3VBHX2G4L3TVZ6JF5ZEJ5U6Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480712969 (571h53m39.351796014s) [noise=true , address=77.90.31.18:4834 , piece_id=BTWNGNQVEUEVV6HLBDM63SGNTKV4RDU4J4XFLG52WHKP53H3L5KA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481019453 (571h53m39.352102488s) [noise=true , address=145.79.227.30:4777 , piece_id=VSCXLXGIBSFE6E4OHLWNZIZ75PO7WQO2VFHUYHW7JUNH62KGGU7Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481029532 (571h53m39.352112567s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481559832 (571h53m39.352642877s) [noise=true , address=145.79.227.35:4515 , piece_id=L7LFZKZEZERUNX4HUNDNQ2PWHXNJRJ7OBFRNMFIICPZEUIBHGP7A, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481731919 (571h53m39.352814954s) [noise=true , address=64.49.13.26:4953 , piece_id=4YSW5Y3A6HYG3XJ2XAB7RGVHAOHMP7G3QIZYMYTSXVXYJ3ANLNXA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482078329 (571h53m39.353161364s) [noise=true , address=145.79.227.44:4205 , piece_id=UE4QSHDQBZ3XBWQ6I6MCRVTLMU7U6W6OZDA2ZGQCH6ZYRAGPD7WA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482099830 (571h53m39.353182865s) [noise=true , address=145.79.227.40:4067 , piece_id=WSYDRX5YJ6OCFW52ZVMM6WZYNRFUXMR4XJ6C5727BOFMR4RRH33Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482434487 (571h53m39.353517532s) [noise=true , address=145.79.227.3:4164 , piece_id=U6F7VDPOH7K6R5EH5UUXHFP55GXOEXO5BLBJHZRTITPDQI4LP7KQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482480845 (571h53m39.35356388s) [noise=true , address=64.49.13.140:4516 , piece_id=4GUYOYOAEASUK2W25S3NILWIIIHGOZX76L4EGFLUUUEDWRALPP2A, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482841442 (571h53m39.353924477s) [noise=true , address=145.79.227.34:4910 , piece_id=ZH4DCNXET2Q6VRAUH3NQ3KKSV4OYI575X7YS72XD236EEOMOG3LA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483163645 (571h53m39.35424668s) [noise=true , address=145.79.227.29:4974 , piece_id=KEKREOD2B35XQNI45JBHE2SZZC3MQQBQUR5I6WFDAQYJQGJBXSMA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483532478 (571h53m39.354615513s) [noise=true , address=77.90.31.17:4769 , piece_id=3MKUMOQIUVERKQLLIFX52XQPTQ5LSFMHIPD5YW2YTO5YEFR36IYA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483914596 (571h53m39.354997631s) [noise=true , address=145.79.227.13:4807 , piece_id=VKXB5MWAG2YYH2AWWJEGTLBZ2QUPLI25VTHI6AMMALSM23Z3AWZQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:40.761721106 (571h53m40.632804141s) [upload=501384]: Next returned nil segment Dec 29 11:22:40.761733349 (571h53m40.632816384s) [upload=501384]: Waiting for error group managing segments... Dec 29 11:22:41.121222862 (571h53m40.992305897s) [noise=true , address=64.49.13.140:4516 , piece_id=4GUYOYOAEASUK2W25S3NILWIIIHGOZX76L4EGFLUUUEDWRALPP2A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.197311646 (571h53m41.068394691s) [noise=true , address=64.49.13.26:4953 , piece_id=4YSW5Y3A6HYG3XJ2XAB7RGVHAOHMP7G3QIZYMYTSXVXYJ3ANLNXA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.347247568 (571h53m41.218330613s) [noise=true , address=145.79.227.13:4807 , piece_id=VKXB5MWAG2YYH2AWWJEGTLBZ2QUPLI25VTHI6AMMALSM23Z3AWZQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.347917313 (571h53m41.219000348s) [noise=true , address=145.79.227.3:4164 , piece_id=U6F7VDPOH7K6R5EH5UUXHFP55GXOEXO5BLBJHZRTITPDQI4LP7KQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.348510613 (571h53m41.219593648s) [noise=true , address=145.79.227.29:4974 , piece_id=KEKREOD2B35XQNI45JBHE2SZZC3MQQBQUR5I6WFDAQYJQGJBXSMA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.503950985 (571h53m41.37503402s) [noise=true , address=77.90.31.17:4769 , piece_id=3MKUMOQIUVERKQLLIFX52XQPTQ5LSFMHIPD5YW2YTO5YEFR36IYA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.561352025 (571h53m41.43243506s) [noise=true , address=77.90.31.18:4834 , piece_id=BTWNGNQVEUEVV6HLBDM63SGNTKV4RDU4J4XFLG52WHKP53H3L5KA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.622447380 (571h53m41.493530425s) [noise=true , address=77.90.31.16:4750 , piece_id=MVVZPJEGDCBSNVOB6CD2WUZGIJTL3VBHX2G4L3TVZ6JF5ZEJ5U6Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.645494685 (571h53m41.51657773s) [noise=true , address=77.90.31.21:4889 , piece_id=BR6Z55P3A7YJ5FJRDSZFUIF66FGIJQY7WCEQ7PL5DOYWL73OMKTQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.247930045 (571h53m42.11901308s) [noise=true , address=145.79.227.30:4777 , piece_id=VSCXLXGIBSFE6E4OHLWNZIZ75PO7WQO2VFHUYHW7JUNH62KGGU7Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.262221890 (571h53m42.133304925s) [noise=true , address=145.79.227.44:4205 , piece_id=UE4QSHDQBZ3XBWQ6I6MCRVTLMU7U6W6OZDA2ZGQCH6ZYRAGPD7WA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.295598123 (571h53m42.166681158s) [noise=true , address=145.79.227.35:4515 , piece_id=L7LFZKZEZERUNX4HUNDNQ2PWHXNJRJ7OBFRNMFIICPZEUIBHGP7A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.307481351 (571h53m42.178564386s) [noise=true , address=145.79.227.34:4910 , piece_id=ZH4DCNXET2Q6VRAUH3NQ3KKSV4OYI575X7YS72XD236EEOMOG3LA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.325775906 (571h53m42.196858941s) [noise=true , address=145.79.227.40:4067 , piece_id=WSYDRX5YJ6OCFW52ZVMM6WZYNRFUXMR4XJ6C5727BOFMR4RRH33Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.325865978 (571h53m42.196949023s) [seg_pos={0 0}, upload=501384]: Segment reached optimal threshold of 14 pieces. Dec 29 11:22:42.326137074 (571h53m42.197220109s) [noise=true , address=64.49.13.169:4708 , piece_id=B2X2RWRELXWJPF6AVJJZCNU46LM5ASGRE4EWGPKL3SCLQ4VKUBDQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection Dec 29 11:22:42.326153074 (571h53m42.197236109s) [noise=true , address=64.49.13.169:4708 , piece_id=B2X2RWRELXWJPF6AVJJZCNU46LM5ASGRE4EWGPKL3SCLQ4VKUBDQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326193401 (571h53m42.197276446s) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12h5esfg6mw2c8y3AVpeFBkL5FfLAwYdjhcnQ9CJ8fjh61WLYjx): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326205705 (571h53m42.19728874s) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326407739 (571h53m42.197490784s) [noise=true , address=77.90.31.10:4524 , piece_id=Q2JGEUV5EDFTVY2CSNPZOVJIIIAFBA3IHA7S5N7YZRH3E7TSTE5A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uN7CznoK4AD8FqFkNUndawdGJ9rmjAe7oNnjJkV844ScY87qe): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326415374 (571h53m42.197498419s) [noise=true , address=77.90.31.10:4524 , piece_id=Q2JGEUV5EDFTVY2CSNPZOVJIIIAFBA3IHA7S5N7YZRH3E7TSTE5A, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326505105 (571h53m42.19758815s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tnTEYvqJ6PB2DsuXA28sEuEiGj11StA4qdhcYH1kuoaMkTVdY): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326512188 (571h53m42.197595223s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326532357 (571h53m42.197615392s) [seg_pos={0 0}, upload=501384]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 11:22:42.326671562 (571h53m42.197754597s) [upload=501384]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 11:22:42.326677843 (571h53m42.197760879s) [upload=501384]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 11:22:42.610391438 (571h53m42.481474493s) [upload=501384]: Done uploading segments... Dec 29 11:22:42.610416826 (571h53m42.481499871s) [upload=501384]: Upload finished. err: <nil> Duration: 36.916386537s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [XX....X..X.....X.X...X.X.X.X.XX..XX..X..XXX.X.......X.X..X.X] duration: 30.002432903s trace: https://jaeger.dev.storj.io/trace/e9aadcd862ab38e FAIL (2025-12-29 11:35:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 62]: Starting upload Dec 29 11:34:56.000001888 (572h5m55.871084943s) [upload=501562]: Done starting upload Dec 29 11:34:56.000178123 (572h5m55.871261168s) [upload=501562]: Uploading segments... Dec 29 11:34:56.000217889 (572h5m55.871300924s) [upload=501562]: Got next segment. Inline: false Dec 29 11:34:56.000243347 (572h5m55.871326383s) [upload=501562]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 11:34:56.000247515 (572h5m55.87133056s) [upload=501562]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 11:34:56.248472732 (572h5m56.119555777s) [seg_pos={0 0}, upload=501562]: Begin upload segment... Dec 29 11:34:56.248530152 (572h5m56.119613187s) [seg_pos={0 0}, upload=501562]: Done begin upload segment. Dec 29 11:34:56.248564307 (572h5m56.119647342s) [noise=true , address=45.91.116.13:4156 , piece_id=6N7FG7XNHYEG5TDYD4D6HWR53CBXMYPMGOPSVZOJUYGPGEU3DTVA, seg_pos={0 0}, upload=501562]: Uploading piece... Dec 29 11:34:56.248576249 (572h5m56.119659294s) [noise=true , address=45.91.116.5:4300 , piece_id=FKA57MCHUX7OS646EMYFISXEQF3EF4OEJDCZOA46WQ3Q2KUAFKRQ, seg_pos={0 0}, upload=501562]: Uploading piece... Dec 29 11:34:56.248623861 (572h5m56.119706896s) [noise=true , address=145.79.227.38:4978 , piece_id=ETIRBDVHSYLBTRJYYMUCEPNWPCJH4GZOF2GRUXRTZBLBLFUOEV2Q, seg_pos={0 0}, upload=501562]: Uploading piece... Dec 29 11:34:56.249383356 (572h5m56.120466421s) [noise=true , address=145.79.227.24:4581 , piece_id=HE6DBPKAOU2Q2UR66NU6A7GWQUEZ652425IDU3ESRGVICWAI6TJQ, seg_pos={0 0}, upload=501562]: Uploading piece... Dec 29 11:34:56.250033494 (572h5m56.121116539s) [noise=true , address=145.79.227.47:4267 , piece_id=M2AGKXSL5DCZ52VH2RH2AVBNIY3OU7BG6KQ7E6WOICIWC76JMOYA, seg_pos={0 0}, upload=501562]: Uploading piece... Dec 29 11:34:56.250395374 (572h5m56.121478409s) [noise=true , address=45.91.116.8:4048 , piece_id=PR5LQYRQZNKSGXY2BJY2ZJJCOP4X3MZ3ZDR7TTIZELBS7UL66KYQ, seg_pos={0 0}, upload=501562]: Uploading piece... Dec 29 11:34:56.250791899 (572h5m56.121874944s) [noise=true , address=145.79.227.2:4114 , piece_id=SKSGFARV3EVPKXXJZV3NFTNNOWIQKMA556OQPVNBP6CDIBASGPAA, seg_pos={0 0}, upload=501562]: Uploading piece... Dec 29 11:34:56.251181540 (572h5m56.122264586s) [noise=true , address=145.79.227.13:4803 , piece_id=EZB2VAFV7LRS54XTBXL2PAPDF7XFTADLV7SXRL77AFZ2V5TRL3GA, seg_pos={0 0}, upload=501562]: Uploading piece... Dec 29 11:34:56.251609705 (572h5m56.122692751s) [noise=true , address=77.90.31.18:4805 , piece_id=TTTSIETCHPZAD6GBAOXCHSWRVCKBW4YJW4XID7DSG6KMQ4V4QTWQ, seg_pos={0 0}, upload=501562]: Uploading piece... Dec 29 11:34:56.252064713 (572h5m56.123147748s) [noise=true , address=45.91.116.14:4189 , piece_id=NQFDB3XBR5JWZRE26S3PG3BZZPJ377DAI3F4RHSSMQUJYJUYNKBA, seg_pos={0 0}, upload=501562]: Uploading piece... Dec 29 11:34:56.252583660 (572h5m56.123666695s) [noise=true , address=145.79.227.6:4259 , piece_id=JGLUXLLLJ4GBMDD4JU76FVDG2JJU4R4K2D4BHMHIBXY7BV4EQWKQ, seg_pos={0 0}, upload=501562]: Uploading piece... Dec 29 11:34:56.254328763 (572h5m56.125411808s) [noise=true , address=145.79.227.16:4910 , piece_id=RQUJAMXAGIGAWCL3GQWELFQKG2JCZX3W7OWHO5BW4XXDVNUSFIUA, seg_pos={0 0}, upload=501562]: Uploading piece... Dec 29 11:34:56.256136806 (572h5m56.127219851s) [noise=true , address=145.79.227.50:4374 , piece_id=R2Y6U6GCIV2VB24EVETJXE3EMXQ5J3NXDYHSHJFYDVTEQWQXY7YQ, seg_pos={0 0}, upload=501562]: Uploading piece... Dec 29 11:34:56.257886327 (572h5m56.128969372s) [noise=true , address=145.79.227.15:4876 , piece_id=6AG26KYI7YWUFOOF3DFVDZLMAX2LQZ73C2K7QTBGV7AZJ6IHODPQ, seg_pos={0 0}, upload=501562]: Uploading piece... Dec 29 11:34:56.259636380 (572h5m56.130719425s) [noise=true , address=145.79.227.12:4765 , piece_id=VCMQKCYRSYDO6UVQ7X3EJ4TI2AZ2XHT7S5ETJAFPYTI3FFRRPYVA, seg_pos={0 0}, upload=501562]: Uploading piece... Dec 29 11:34:56.261390570 (572h5m56.132473605s) [noise=true , address=45.91.116.9:4262 , piece_id=ACW6FXQS3AAO4YUEKSL6VZMDYQNNW4P75WTN6R347K37262X2O3Q, seg_pos={0 0}, upload=501562]: Uploading piece... Dec 29 11:34:57.874144650 (572h5m57.745227695s) [upload=501562]: Next returned nil segment Dec 29 11:34:57.874171441 (572h5m57.745254476s) [upload=501562]: Waiting for error group managing segments... Dec 29 11:34:58.249856812 (572h5m58.120939857s) [noise=true , address=145.79.227.2:4114 , piece_id=SKSGFARV3EVPKXXJZV3NFTNNOWIQKMA556OQPVNBP6CDIBASGPAA, seg_pos={0 0}, upload=501562]: Done uploading piece. err: <nil> Dec 29 11:34:58.250276130 (572h5m58.121359175s) [noise=true , address=145.79.227.13:4803 , piece_id=EZB2VAFV7LRS54XTBXL2PAPDF7XFTADLV7SXRL77AFZ2V5TRL3GA, seg_pos={0 0}, upload=501562]: Done uploading piece. err: <nil> Dec 29 11:34:58.253518334 (572h5m58.124601379s) [noise=true , address=145.79.227.12:4765 , piece_id=VCMQKCYRSYDO6UVQ7X3EJ4TI2AZ2XHT7S5ETJAFPYTI3FFRRPYVA, seg_pos={0 0}, upload=501562]: Done uploading piece. err: <nil> Dec 29 11:34:58.254288360 (572h5m58.125371425s) [noise=true , address=145.79.227.16:4910 , piece_id=RQUJAMXAGIGAWCL3GQWELFQKG2JCZX3W7OWHO5BW4XXDVNUSFIUA, seg_pos={0 0}, upload=501562]: Done uploading piece. err: <nil> Dec 29 11:34:58.255102460 (572h5m58.126185495s) [noise=true , address=145.79.227.50:4374 , piece_id=R2Y6U6GCIV2VB24EVETJXE3EMXQ5J3NXDYHSHJFYDVTEQWQXY7YQ, seg_pos={0 0}, upload=501562]: Done uploading piece. err: <nil> Dec 29 11:34:58.256880576 (572h5m58.127963611s) [noise=true , address=145.79.227.38:4978 , piece_id=ETIRBDVHSYLBTRJYYMUCEPNWPCJH4GZOF2GRUXRTZBLBLFUOEV2Q, seg_pos={0 0}, upload=501562]: Done uploading piece. err: <nil> Dec 29 11:34:58.274419101 (572h5m58.145502136s) [noise=true , address=145.79.227.15:4876 , piece_id=6AG26KYI7YWUFOOF3DFVDZLMAX2LQZ73C2K7QTBGV7AZJ6IHODPQ, seg_pos={0 0}, upload=501562]: Done uploading piece. err: <nil> Dec 29 11:34:58.275881436 (572h5m58.146964471s) [noise=true , address=145.79.227.24:4581 , piece_id=HE6DBPKAOU2Q2UR66NU6A7GWQUEZ652425IDU3ESRGVICWAI6TJQ, seg_pos={0 0}, upload=501562]: Done uploading piece. err: <nil> Dec 29 11:34:58.278356780 (572h5m58.149439825s) [noise=true , address=145.79.227.6:4259 , piece_id=JGLUXLLLJ4GBMDD4JU76FVDG2JJU4R4K2D4BHMHIBXY7BV4EQWKQ, seg_pos={0 0}, upload=501562]: Done uploading piece. err: <nil> Dec 29 11:34:58.289878128 (572h5m58.160961193s) [noise=true , address=145.79.227.47:4267 , piece_id=M2AGKXSL5DCZ52VH2RH2AVBNIY3OU7BG6KQ7E6WOICIWC76JMOYA, seg_pos={0 0}, upload=501562]: Done uploading piece. err: <nil> Dec 29 11:34:58.568495957 (572h5m58.439578992s) [noise=true , address=77.90.31.18:4805 , piece_id=TTTSIETCHPZAD6GBAOXCHSWRVCKBW4YJW4XID7DSG6KMQ4V4QTWQ, seg_pos={0 0}, upload=501562]: Done uploading piece. err: <nil> Dec 29 11:34:58.586763492 (572h5m58.457846527s) [noise=true , address=45.91.116.13:4156 , piece_id=6N7FG7XNHYEG5TDYD4D6HWR53CBXMYPMGOPSVZOJUYGPGEU3DTVA, seg_pos={0 0}, upload=501562]: Done uploading piece. err: <nil> Dec 29 11:34:58.609889376 (572h5m58.480972412s) [noise=true , address=45.91.116.9:4262 , piece_id=ACW6FXQS3AAO4YUEKSL6VZMDYQNNW4P75WTN6R347K37262X2O3Q, seg_pos={0 0}, upload=501562]: Done uploading piece. err: <nil> Dec 29 11:34:58.609909135 (572h5m58.48099217s) [seg_pos={0 0}, upload=501562]: Segment reached optimal threshold of 13 pieces. Dec 29 11:34:58.610083717 (572h5m58.481166762s) [noise=true , address=45.91.116.8:4048 , piece_id=PR5LQYRQZNKSGXY2BJY2ZJJCOP4X3MZ3ZDR7TTIZELBS7UL66KYQ, seg_pos={0 0}, upload=501562]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 29 11:34:58.610090339 (572h5m58.481173394s) [noise=true , address=45.91.116.8:4048 , piece_id=PR5LQYRQZNKSGXY2BJY2ZJJCOP4X3MZ3ZDR7TTIZELBS7UL66KYQ, seg_pos={0 0}, upload=501562]: Long tail cancellation due to optimal threshold Dec 29 11:34:58.610306260 (572h5m58.481389295s) [noise=true , address=45.91.116.14:4189 , piece_id=NQFDB3XBR5JWZRE26S3PG3BZZPJ377DAI3F4RHSSMQUJYJUYNKBA, seg_pos={0 0}, upload=501562]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: read tcp 188.166.225.170:53398->45.91.116.14:4189: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 11:34:58.610311891 (572h5m58.481394926s) [noise=true , address=45.91.116.14:4189 , piece_id=NQFDB3XBR5JWZRE26S3PG3BZZPJ377DAI3F4RHSSMQUJYJUYNKBA, seg_pos={0 0}, upload=501562]: Long tail cancellation due to optimal threshold Dec 29 11:34:58.610509257 (572h5m58.481592302s) [noise=true , address=45.91.116.5:4300 , piece_id=FKA57MCHUX7OS646EMYFISXEQF3EF4OEJDCZOA46WQ3Q2KUAFKRQ, seg_pos={0 0}, upload=501562]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: write tcp 188.166.225.170:50282->45.91.116.5:4300: use of closed network connection; write tcp 188.166.225.170:50282->45.91.116.5:4300: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50282->45.91.116.5:4300: use of closed network connection Dec 29 11:34:58.610524496 (572h5m58.481607531s) [noise=true , address=45.91.116.5:4300 , piece_id=FKA57MCHUX7OS646EMYFISXEQF3EF4OEJDCZOA46WQ3Q2KUAFKRQ, seg_pos={0 0}, upload=501562]: Long tail cancellation due to optimal threshold Dec 29 11:34:58.610581224 (572h5m58.481664259s) [seg_pos={0 0}, upload=501562]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 11:34:58.610720489 (572h5m58.481803524s) [upload=501562]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 11:34:58.610728634 (572h5m58.481811669s) [upload=501562]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 11:34:58.915512829 (572h5m58.786595864s) [upload=501562]: Done uploading segments... Dec 29 11:34:58.915546002 (572h5m58.786629047s) [upload=501562]: Upload finished. err: <nil> Duration: 30.002452971s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 17 out of 60 is failed (threshold=6) [X...X..........X....X.......X..X...XX..XX......X.X.XX.X.XX..] duration: 4.912923833s trace: https://jaeger.dev.storj.io/trace/102364c565a1cd6b FAIL (2025-12-29 11:35:02 UTC) orbiter warning: context deadline exceeded Duration: 4.912951196s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........................X...............X..X...............] duration: 40.167617133s trace: https://jaeger.dev.storj.io/trace/7e7354d8023608a8 FAIL (2025-12-29 11:34:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== ) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480097317 (571h53m39.351180352s) [noise=true , address=77.90.31.21:4889 , piece_id=BR6Z55P3A7YJ5FJRDSZFUIF66FGIJQY7WCEQ7PL5DOYWL73OMKTQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480405053 (571h53m39.351488098s) [noise=true , address=77.90.31.16:4750 , piece_id=MVVZPJEGDCBSNVOB6CD2WUZGIJTL3VBHX2G4L3TVZ6JF5ZEJ5U6Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480712969 (571h53m39.351796014s) [noise=true , address=77.90.31.18:4834 , piece_id=BTWNGNQVEUEVV6HLBDM63SGNTKV4RDU4J4XFLG52WHKP53H3L5KA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481019453 (571h53m39.352102488s) [noise=true , address=145.79.227.30:4777 , piece_id=VSCXLXGIBSFE6E4OHLWNZIZ75PO7WQO2VFHUYHW7JUNH62KGGU7Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481029532 (571h53m39.352112567s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481559832 (571h53m39.352642877s) [noise=true , address=145.79.227.35:4515 , piece_id=L7LFZKZEZERUNX4HUNDNQ2PWHXNJRJ7OBFRNMFIICPZEUIBHGP7A, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481731919 (571h53m39.352814954s) [noise=true , address=64.49.13.26:4953 , piece_id=4YSW5Y3A6HYG3XJ2XAB7RGVHAOHMP7G3QIZYMYTSXVXYJ3ANLNXA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482078329 (571h53m39.353161364s) [noise=true , address=145.79.227.44:4205 , piece_id=UE4QSHDQBZ3XBWQ6I6MCRVTLMU7U6W6OZDA2ZGQCH6ZYRAGPD7WA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482099830 (571h53m39.353182865s) [noise=true , address=145.79.227.40:4067 , piece_id=WSYDRX5YJ6OCFW52ZVMM6WZYNRFUXMR4XJ6C5727BOFMR4RRH33Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482434487 (571h53m39.353517532s) [noise=true , address=145.79.227.3:4164 , piece_id=U6F7VDPOH7K6R5EH5UUXHFP55GXOEXO5BLBJHZRTITPDQI4LP7KQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482480845 (571h53m39.35356388s) [noise=true , address=64.49.13.140:4516 , piece_id=4GUYOYOAEASUK2W25S3NILWIIIHGOZX76L4EGFLUUUEDWRALPP2A, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482841442 (571h53m39.353924477s) [noise=true , address=145.79.227.34:4910 , piece_id=ZH4DCNXET2Q6VRAUH3NQ3KKSV4OYI575X7YS72XD236EEOMOG3LA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483163645 (571h53m39.35424668s) [noise=true , address=145.79.227.29:4974 , piece_id=KEKREOD2B35XQNI45JBHE2SZZC3MQQBQUR5I6WFDAQYJQGJBXSMA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483532478 (571h53m39.354615513s) [noise=true , address=77.90.31.17:4769 , piece_id=3MKUMOQIUVERKQLLIFX52XQPTQ5LSFMHIPD5YW2YTO5YEFR36IYA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483914596 (571h53m39.354997631s) [noise=true , address=145.79.227.13:4807 , piece_id=VKXB5MWAG2YYH2AWWJEGTLBZ2QUPLI25VTHI6AMMALSM23Z3AWZQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:40.761721106 (571h53m40.632804141s) [upload=501384]: Next returned nil segment Dec 29 11:22:40.761733349 (571h53m40.632816384s) [upload=501384]: Waiting for error group managing segments... Dec 29 11:22:41.121222862 (571h53m40.992305897s) [noise=true , address=64.49.13.140:4516 , piece_id=4GUYOYOAEASUK2W25S3NILWIIIHGOZX76L4EGFLUUUEDWRALPP2A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.197311646 (571h53m41.068394691s) [noise=true , address=64.49.13.26:4953 , piece_id=4YSW5Y3A6HYG3XJ2XAB7RGVHAOHMP7G3QIZYMYTSXVXYJ3ANLNXA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.347247568 (571h53m41.218330613s) [noise=true , address=145.79.227.13:4807 , piece_id=VKXB5MWAG2YYH2AWWJEGTLBZ2QUPLI25VTHI6AMMALSM23Z3AWZQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.347917313 (571h53m41.219000348s) [noise=true , address=145.79.227.3:4164 , piece_id=U6F7VDPOH7K6R5EH5UUXHFP55GXOEXO5BLBJHZRTITPDQI4LP7KQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.348510613 (571h53m41.219593648s) [noise=true , address=145.79.227.29:4974 , piece_id=KEKREOD2B35XQNI45JBHE2SZZC3MQQBQUR5I6WFDAQYJQGJBXSMA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.503950985 (571h53m41.37503402s) [noise=true , address=77.90.31.17:4769 , piece_id=3MKUMOQIUVERKQLLIFX52XQPTQ5LSFMHIPD5YW2YTO5YEFR36IYA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.561352025 (571h53m41.43243506s) [noise=true , address=77.90.31.18:4834 , piece_id=BTWNGNQVEUEVV6HLBDM63SGNTKV4RDU4J4XFLG52WHKP53H3L5KA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.622447380 (571h53m41.493530425s) [noise=true , address=77.90.31.16:4750 , piece_id=MVVZPJEGDCBSNVOB6CD2WUZGIJTL3VBHX2G4L3TVZ6JF5ZEJ5U6Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.645494685 (571h53m41.51657773s) [noise=true , address=77.90.31.21:4889 , piece_id=BR6Z55P3A7YJ5FJRDSZFUIF66FGIJQY7WCEQ7PL5DOYWL73OMKTQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.247930045 (571h53m42.11901308s) [noise=true , address=145.79.227.30:4777 , piece_id=VSCXLXGIBSFE6E4OHLWNZIZ75PO7WQO2VFHUYHW7JUNH62KGGU7Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.262221890 (571h53m42.133304925s) [noise=true , address=145.79.227.44:4205 , piece_id=UE4QSHDQBZ3XBWQ6I6MCRVTLMU7U6W6OZDA2ZGQCH6ZYRAGPD7WA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.295598123 (571h53m42.166681158s) [noise=true , address=145.79.227.35:4515 , piece_id=L7LFZKZEZERUNX4HUNDNQ2PWHXNJRJ7OBFRNMFIICPZEUIBHGP7A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.307481351 (571h53m42.178564386s) [noise=true , address=145.79.227.34:4910 , piece_id=ZH4DCNXET2Q6VRAUH3NQ3KKSV4OYI575X7YS72XD236EEOMOG3LA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.325775906 (571h53m42.196858941s) [noise=true , address=145.79.227.40:4067 , piece_id=WSYDRX5YJ6OCFW52ZVMM6WZYNRFUXMR4XJ6C5727BOFMR4RRH33Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.325865978 (571h53m42.196949023s) [seg_pos={0 0}, upload=501384]: Segment reached optimal threshold of 14 pieces. Dec 29 11:22:42.326137074 (571h53m42.197220109s) [noise=true , address=64.49.13.169:4708 , piece_id=B2X2RWRELXWJPF6AVJJZCNU46LM5ASGRE4EWGPKL3SCLQ4VKUBDQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection Dec 29 11:22:42.326153074 (571h53m42.197236109s) [noise=true , address=64.49.13.169:4708 , piece_id=B2X2RWRELXWJPF6AVJJZCNU46LM5ASGRE4EWGPKL3SCLQ4VKUBDQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326193401 (571h53m42.197276446s) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12h5esfg6mw2c8y3AVpeFBkL5FfLAwYdjhcnQ9CJ8fjh61WLYjx): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326205705 (571h53m42.19728874s) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326407739 (571h53m42.197490784s) [noise=true , address=77.90.31.10:4524 , piece_id=Q2JGEUV5EDFTVY2CSNPZOVJIIIAFBA3IHA7S5N7YZRH3E7TSTE5A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uN7CznoK4AD8FqFkNUndawdGJ9rmjAe7oNnjJkV844ScY87qe): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326415374 (571h53m42.197498419s) [noise=true , address=77.90.31.10:4524 , piece_id=Q2JGEUV5EDFTVY2CSNPZOVJIIIAFBA3IHA7S5N7YZRH3E7TSTE5A, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326505105 (571h53m42.19758815s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tnTEYvqJ6PB2DsuXA28sEuEiGj11StA4qdhcYH1kuoaMkTVdY): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326512188 (571h53m42.197595223s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326532357 (571h53m42.197615392s) [seg_pos={0 0}, upload=501384]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 11:22:42.326671562 (571h53m42.197754597s) [upload=501384]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 11:22:42.326677843 (571h53m42.197760879s) [upload=501384]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 11:22:42.610391438 (571h53m42.481474493s) [upload=501384]: Done uploading segments... Dec 29 11:22:42.610416826 (571h53m42.481499871s) [upload=501384]: Upload finished. err: <nil> Duration: 40.167642049s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.XX....X..X.....X.X...X.X.X.X.XX..XX..X..XXX.X.......X.X..X.] duration: 5.683291146s trace: https://jaeger.dev.storj.io/trace/7d5264e2a6b1751c FAIL (2025-12-29 11:34:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: 3NV4SDM7V5GJ5EEZQPBDM6XFK354G7LBAYCH444KJ5ICJXYDPEUQ) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: BNWSSQOGSBAT3YCZAHAJJQLVOEEPZYUNW64LHR5U75RIXVA7DDMQ) context deadline exceeded ======LOGS====== Dec 29 11:33:32.949900962 (572h4m32.820983997s) [upload=501542]: Starting upload Dec 29 11:33:32.949964402 (572h4m32.821047457s) [upload=501542]: Done starting upload Dec 29 11:33:32.950094059 (572h4m32.821177104s) [upload=501542]: Uploading segments... Dec 29 11:33:32.950220661 (572h4m32.821303696s) [upload=501542]: Got next segment. Inline: false Dec 29 11:33:32.950263332 (572h4m32.821346367s) [upload=501542]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 11:33:32.950268091 (572h4m32.821351136s) [upload=501542]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 11:33:33.197681415 (572h4m33.06876446s) [seg_pos={0 0}, upload=501542]: Begin upload segment... Dec 29 11:33:33.197744164 (572h4m33.068827199s) [seg_pos={0 0}, upload=501542]: Done begin upload segment. Dec 29 11:33:33.197763862 (572h4m33.068846937s) [noise=true , address=45.91.116.8:4048 , piece_id=5SM3PJXUVKE3ES55YZH6AUT5KBQ3AAKDKBRRKUX3ZX3OTSHVH4UA, seg_pos={0 0}, upload=501542]: Uploading piece... Dec 29 11:33:33.197863512 (572h4m33.068946557s) [noise=true , address=77.90.31.19:4842 , piece_id=VCBLKXBJLPLCAVPJPR777TCTNRWSYOEKEYN5H5YNAEZ7ZSVSB2RA, seg_pos={0 0}, upload=501542]: Uploading piece... Dec 29 11:33:33.198626615 (572h4m33.06970965s) [noise=true , address=77.90.31.18:4808 , piece_id=JFVH4LZULNIB6PLY2F5ELRH7RS2DFYMG7BXUESFY2SGQRVOFFJDA, seg_pos={0 0}, upload=501542]: Uploading piece... Dec 29 11:33:33.198932015 (572h4m33.070015051s) [noise=true , address=77.90.31.14:4661 , piece_id=X6FGJQA2H37TZBS7A7OBBG2LK6YSKGEAD4FVEPPDMRQ3U3M5MJYA, seg_pos={0 0}, upload=501542]: Uploading piece... Dec 29 11:33:33.199220636 (572h4m33.070303671s) [noise=true , address=145.79.227.38:4978 , piece_id=QNLRZOWXIVXBK24MHB56SUP5F5S5TQDVT2R64MMKGAHJMQMLO6EQ, seg_pos={0 0}, upload=501542]: Uploading piece... Dec 29 11:33:33.199482143 (572h4m33.070565188s) [noise=true , address=145.79.227.5:4223 , piece_id=QA235O4TAV2XOJXK36S27KTSLJWVHA7SGPBKLEOPYJPX62D4KVDA, seg_pos={0 0}, upload=501542]: Uploading piece... Dec 29 11:33:33.199737439 (572h4m33.070820484s) [noise=true , address=145.79.227.14:4839 , piece_id=KOZ47QJGCFCF3KNJ44VS2OSUDNXHUAJUCG73B4I3UHNPBMP66OTA, seg_pos={0 0}, upload=501542]: Uploading piece... Dec 29 11:33:33.200003616 (572h4m33.071086651s) [noise=true , address=145.79.227.45:4231 , piece_id=5TZR2KVOUSCTCBUIXDXUJRXMO4ZVPY7F4JNHSDP4NROO7WQEWR6A, seg_pos={0 0}, upload=501542]: Uploading piece... Dec 29 11:33:33.200418486 (572h4m33.071501521s) [noise=true , address=145.79.227.48:4302 , piece_id=3CPKMM7TXSPO2QXKQGOA53BTDETU6MPKAVMME6BNSFLMUGYJEVLA, seg_pos={0 0}, upload=501542]: Uploading piece... Dec 29 11:33:33.200847092 (572h4m33.071930127s) [noise=true , address=77.90.31.21:4877 , piece_id=6EHY77KDZINTESJN4IFO6MIN23HDTN4O7T5IGEJETDAON6NRLVTQ, seg_pos={0 0}, upload=501542]: Uploading piece... Dec 29 11:33:33.201382422 (572h4m33.072465467s) [noise=true , address=45.91.116.13:4153 , piece_id=3NV4SDM7V5GJ5EEZQPBDM6XFK354G7LBAYCH444KJ5ICJXYDPEUQ, seg_pos={0 0}, upload=501542]: Uploading piece... Dec 29 11:33:33.202452268 (572h4m33.073535304s) [noise=true , address=77.90.31.17:4773 , piece_id=VATYJS6YI3OT5TAG3MTRCUYHBVI2AQED7YB3U7ZWKCKKGZJKBV4A, seg_pos={0 0}, upload=501542]: Uploading piece... Dec 29 11:33:33.202754043 (572h4m33.073837079s) [noise=true , address=45.91.116.14:4190 , piece_id=BNWSSQOGSBAT3YCZAHAJJQLVOEEPZYUNW64LHR5U75RIXVA7DDMQ, seg_pos={0 0}, upload=501542]: Uploading piece... Dec 29 11:33:33.203943117 (572h4m33.075026163s) [noise=true , address=45.91.116.11:4119 , piece_id=LMJZCONZWHTFUYYTIENW672XWCEECRPEX32EC24SFOGZ62VKHD3Q, seg_pos={0 0}, upload=501542]: Uploading piece... Dec 29 11:33:33.205378330 (572h4m33.076461375s) [noise=true , address=45.91.116.5:4297 , piece_id=DNNMNO4JTRNNLPZKWP5BVQKW5YDBUCHOJ6LXJJ5WZGSPFANOI3EA, seg_pos={0 0}, upload=501542]: Uploading piece... Dec 29 11:33:33.207053991 (572h4m33.078137026s) [noise=true , address=145.79.227.4:4187 , piece_id=MWM6HCFA7J4W25ZFLOA3UL32BFXTIB5VPVNXOY2XRNM6TK5V2NBA, seg_pos={0 0}, upload=501542]: Uploading piece... Dec 29 11:33:34.899018782 (572h4m34.770101827s) [upload=501542]: Next returned nil segment Dec 29 11:33:34.899031206 (572h4m34.770114251s) [upload=501542]: Waiting for error group managing segments... Dec 29 11:33:35.183350333 (572h4m35.054433378s) [noise=true , address=145.79.227.48:4302 , piece_id=3CPKMM7TXSPO2QXKQGOA53BTDETU6MPKAVMME6BNSFLMUGYJEVLA, seg_pos={0 0}, upload=501542]: Done uploading piece. err: <nil> Dec 29 11:33:35.183532509 (572h4m35.054615544s) [noise=true , address=145.79.227.5:4223 , piece_id=QA235O4TAV2XOJXK36S27KTSLJWVHA7SGPBKLEOPYJPX62D4KVDA, seg_pos={0 0}, upload=501542]: Done uploading piece. err: <nil> Dec 29 11:33:35.190911029 (572h4m35.061994064s) [noise=true , address=145.79.227.45:4231 , piece_id=5TZR2KVOUSCTCBUIXDXUJRXMO4ZVPY7F4JNHSDP4NROO7WQEWR6A, seg_pos={0 0}, upload=501542]: Done uploading piece. err: <nil> Dec 29 11:33:35.199529030 (572h4m35.070612065s) [noise=true , address=145.79.227.4:4187 , piece_id=MWM6HCFA7J4W25ZFLOA3UL32BFXTIB5VPVNXOY2XRNM6TK5V2NBA, seg_pos={0 0}, upload=501542]: Done uploading piece. err: <nil> Dec 29 11:33:35.209616958 (572h4m35.080699993s) [noise=true , address=145.79.227.14:4839 , piece_id=KOZ47QJGCFCF3KNJ44VS2OSUDNXHUAJUCG73B4I3UHNPBMP66OTA, seg_pos={0 0}, upload=501542]: Done uploading piece. err: <nil> Dec 29 11:33:35.220204839 (572h4m35.091287874s) [noise=true , address=145.79.227.38:4978 , piece_id=QNLRZOWXIVXBK24MHB56SUP5F5S5TQDVT2R64MMKGAHJMQMLO6EQ, seg_pos={0 0}, upload=501542]: Done uploading piece. err: <nil> Dec 29 11:33:35.405746028 (572h4m35.276829063s) [noise=true , address=77.90.31.14:4661 , piece_id=X6FGJQA2H37TZBS7A7OBBG2LK6YSKGEAD4FVEPPDMRQ3U3M5MJYA, seg_pos={0 0}, upload=501542]: Done uploading piece. err: <nil> Dec 29 11:33:35.471805645 (572h4m35.34288868s) [noise=true , address=45.91.116.13:4153 , piece_id=3NV4SDM7V5GJ5EEZQPBDM6XFK354G7LBAYCH444KJ5ICJXYDPEUQ, seg_pos={0 0}, upload=501542]: Done uploading piece. err: <nil> Dec 29 11:33:35.482728244 (572h4m35.353811279s) [noise=true , address=45.91.116.8:4048 , piece_id=5SM3PJXUVKE3ES55YZH6AUT5KBQ3AAKDKBRRKUX3ZX3OTSHVH4UA, seg_pos={0 0}, upload=501542]: Done uploading piece. err: <nil> Dec 29 11:33:35.562364463 (572h4m35.433447498s) [noise=true , address=77.90.31.21:4877 , piece_id=6EHY77KDZINTESJN4IFO6MIN23HDTN4O7T5IGEJETDAON6NRLVTQ, seg_pos={0 0}, upload=501542]: Done uploading piece. err: <nil> Dec 29 11:33:35.574257479 (572h4m35.445340514s) [noise=true , address=45.91.116.14:4190 , piece_id=BNWSSQOGSBAT3YCZAHAJJQLVOEEPZYUNW64LHR5U75RIXVA7DDMQ, seg_pos={0 0}, upload=501542]: Done uploading piece. err: <nil> Dec 29 11:33:35.666488982 (572h4m35.537572027s) [noise=true , address=45.91.116.11:4119 , piece_id=LMJZCONZWHTFUYYTIENW672XWCEECRPEX32EC24SFOGZ62VKHD3Q, seg_pos={0 0}, upload=501542]: Done uploading piece. err: <nil> Dec 29 11:33:36.047396097 (572h4m35.918479132s) [noise=true , address=77.90.31.19:4842 , piece_id=VCBLKXBJLPLCAVPJPR777TCTNRWSYOEKEYN5H5YNAEZ7ZSVSB2RA, seg_pos={0 0}, upload=501542]: Done uploading piece. err: <nil> Dec 29 11:33:36.047416907 (572h4m35.918499942s) [seg_pos={0 0}, upload=501542]: Segment reached optimal threshold of 13 pieces. Dec 29 11:33:36.047623811 (572h4m35.918706846s) [noise=true , address=77.90.31.18:4808 , piece_id=JFVH4LZULNIB6PLY2F5ELRH7RS2DFYMG7BXUESFY2SGQRVOFFJDA, seg_pos={0 0}, upload=501542]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: write tcp 188.166.225.170:41492->77.90.31.18:4808: use of closed network connection; write tcp 188.166.225.170:41492->77.90.31.18:4808: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41492->77.90.31.18:4808: use of closed network connection Dec 29 11:33:36.047630654 (572h4m35.918713689s) [noise=true , address=77.90.31.18:4808 , piece_id=JFVH4LZULNIB6PLY2F5ELRH7RS2DFYMG7BXUESFY2SGQRVOFFJDA, seg_pos={0 0}, upload=501542]: Long tail cancellation due to optimal threshold Dec 29 11:33:36.047748979 (572h4m35.918832024s) [noise=true , address=45.91.116.5:4297 , piece_id=DNNMNO4JTRNNLPZKWP5BVQKW5YDBUCHOJ6LXJJ5WZGSPFANOI3EA, seg_pos={0 0}, upload=501542]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 29 11:33:36.047765430 (572h4m35.918848465s) [noise=true , address=45.91.116.5:4297 , piece_id=DNNMNO4JTRNNLPZKWP5BVQKW5YDBUCHOJ6LXJJ5WZGSPFANOI3EA, seg_pos={0 0}, upload=501542]: Long tail cancellation due to optimal threshold Dec 29 11:33:36.047871973 (572h4m35.918955008s) [noise=true , address=77.90.31.17:4773 , piece_id=VATYJS6YI3OT5TAG3MTRCUYHBVI2AQED7YB3U7ZWKCKKGZJKBV4A, seg_pos={0 0}, upload=501542]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: expected piece hash; context canceled; EOF Dec 29 11:33:36.047878786 (572h4m35.918961821s) [noise=true , address=77.90.31.17:4773 , piece_id=VATYJS6YI3OT5TAG3MTRCUYHBVI2AQED7YB3U7ZWKCKKGZJKBV4A, seg_pos={0 0}, upload=501542]: Long tail cancellation due to optimal threshold Dec 29 11:33:36.047896871 (572h4m35.918979906s) [seg_pos={0 0}, upload=501542]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 11:33:36.047961684 (572h4m35.91904472s) [upload=501542]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 11:33:36.047990919 (572h4m35.919073954s) [upload=501542]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 11:33:36.342340696 (572h4m36.213423741s) [upload=501542]: Done uploading segments... Dec 29 11:33:36.342399549 (572h4m36.213482624s) [upload=501542]: Upload finished. err: <nil> Duration: 5.683315562s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........................X...............X..X..............] duration: 35.145031625s trace: https://jaeger.dev.storj.io/trace/7e7354d8023608a8 FAIL (2025-12-29 11:34:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== ) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480097317 (571h53m39.351180352s) [noise=true , address=77.90.31.21:4889 , piece_id=BR6Z55P3A7YJ5FJRDSZFUIF66FGIJQY7WCEQ7PL5DOYWL73OMKTQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480405053 (571h53m39.351488098s) [noise=true , address=77.90.31.16:4750 , piece_id=MVVZPJEGDCBSNVOB6CD2WUZGIJTL3VBHX2G4L3TVZ6JF5ZEJ5U6Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480712969 (571h53m39.351796014s) [noise=true , address=77.90.31.18:4834 , piece_id=BTWNGNQVEUEVV6HLBDM63SGNTKV4RDU4J4XFLG52WHKP53H3L5KA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481019453 (571h53m39.352102488s) [noise=true , address=145.79.227.30:4777 , piece_id=VSCXLXGIBSFE6E4OHLWNZIZ75PO7WQO2VFHUYHW7JUNH62KGGU7Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481029532 (571h53m39.352112567s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481559832 (571h53m39.352642877s) [noise=true , address=145.79.227.35:4515 , piece_id=L7LFZKZEZERUNX4HUNDNQ2PWHXNJRJ7OBFRNMFIICPZEUIBHGP7A, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481731919 (571h53m39.352814954s) [noise=true , address=64.49.13.26:4953 , piece_id=4YSW5Y3A6HYG3XJ2XAB7RGVHAOHMP7G3QIZYMYTSXVXYJ3ANLNXA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482078329 (571h53m39.353161364s) [noise=true , address=145.79.227.44:4205 , piece_id=UE4QSHDQBZ3XBWQ6I6MCRVTLMU7U6W6OZDA2ZGQCH6ZYRAGPD7WA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482099830 (571h53m39.353182865s) [noise=true , address=145.79.227.40:4067 , piece_id=WSYDRX5YJ6OCFW52ZVMM6WZYNRFUXMR4XJ6C5727BOFMR4RRH33Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482434487 (571h53m39.353517532s) [noise=true , address=145.79.227.3:4164 , piece_id=U6F7VDPOH7K6R5EH5UUXHFP55GXOEXO5BLBJHZRTITPDQI4LP7KQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482480845 (571h53m39.35356388s) [noise=true , address=64.49.13.140:4516 , piece_id=4GUYOYOAEASUK2W25S3NILWIIIHGOZX76L4EGFLUUUEDWRALPP2A, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482841442 (571h53m39.353924477s) [noise=true , address=145.79.227.34:4910 , piece_id=ZH4DCNXET2Q6VRAUH3NQ3KKSV4OYI575X7YS72XD236EEOMOG3LA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483163645 (571h53m39.35424668s) [noise=true , address=145.79.227.29:4974 , piece_id=KEKREOD2B35XQNI45JBHE2SZZC3MQQBQUR5I6WFDAQYJQGJBXSMA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483532478 (571h53m39.354615513s) [noise=true , address=77.90.31.17:4769 , piece_id=3MKUMOQIUVERKQLLIFX52XQPTQ5LSFMHIPD5YW2YTO5YEFR36IYA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483914596 (571h53m39.354997631s) [noise=true , address=145.79.227.13:4807 , piece_id=VKXB5MWAG2YYH2AWWJEGTLBZ2QUPLI25VTHI6AMMALSM23Z3AWZQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:40.761721106 (571h53m40.632804141s) [upload=501384]: Next returned nil segment Dec 29 11:22:40.761733349 (571h53m40.632816384s) [upload=501384]: Waiting for error group managing segments... Dec 29 11:22:41.121222862 (571h53m40.992305897s) [noise=true , address=64.49.13.140:4516 , piece_id=4GUYOYOAEASUK2W25S3NILWIIIHGOZX76L4EGFLUUUEDWRALPP2A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.197311646 (571h53m41.068394691s) [noise=true , address=64.49.13.26:4953 , piece_id=4YSW5Y3A6HYG3XJ2XAB7RGVHAOHMP7G3QIZYMYTSXVXYJ3ANLNXA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.347247568 (571h53m41.218330613s) [noise=true , address=145.79.227.13:4807 , piece_id=VKXB5MWAG2YYH2AWWJEGTLBZ2QUPLI25VTHI6AMMALSM23Z3AWZQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.347917313 (571h53m41.219000348s) [noise=true , address=145.79.227.3:4164 , piece_id=U6F7VDPOH7K6R5EH5UUXHFP55GXOEXO5BLBJHZRTITPDQI4LP7KQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.348510613 (571h53m41.219593648s) [noise=true , address=145.79.227.29:4974 , piece_id=KEKREOD2B35XQNI45JBHE2SZZC3MQQBQUR5I6WFDAQYJQGJBXSMA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.503950985 (571h53m41.37503402s) [noise=true , address=77.90.31.17:4769 , piece_id=3MKUMOQIUVERKQLLIFX52XQPTQ5LSFMHIPD5YW2YTO5YEFR36IYA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.561352025 (571h53m41.43243506s) [noise=true , address=77.90.31.18:4834 , piece_id=BTWNGNQVEUEVV6HLBDM63SGNTKV4RDU4J4XFLG52WHKP53H3L5KA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.622447380 (571h53m41.493530425s) [noise=true , address=77.90.31.16:4750 , piece_id=MVVZPJEGDCBSNVOB6CD2WUZGIJTL3VBHX2G4L3TVZ6JF5ZEJ5U6Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.645494685 (571h53m41.51657773s) [noise=true , address=77.90.31.21:4889 , piece_id=BR6Z55P3A7YJ5FJRDSZFUIF66FGIJQY7WCEQ7PL5DOYWL73OMKTQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.247930045 (571h53m42.11901308s) [noise=true , address=145.79.227.30:4777 , piece_id=VSCXLXGIBSFE6E4OHLWNZIZ75PO7WQO2VFHUYHW7JUNH62KGGU7Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.262221890 (571h53m42.133304925s) [noise=true , address=145.79.227.44:4205 , piece_id=UE4QSHDQBZ3XBWQ6I6MCRVTLMU7U6W6OZDA2ZGQCH6ZYRAGPD7WA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.295598123 (571h53m42.166681158s) [noise=true , address=145.79.227.35:4515 , piece_id=L7LFZKZEZERUNX4HUNDNQ2PWHXNJRJ7OBFRNMFIICPZEUIBHGP7A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.307481351 (571h53m42.178564386s) [noise=true , address=145.79.227.34:4910 , piece_id=ZH4DCNXET2Q6VRAUH3NQ3KKSV4OYI575X7YS72XD236EEOMOG3LA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.325775906 (571h53m42.196858941s) [noise=true , address=145.79.227.40:4067 , piece_id=WSYDRX5YJ6OCFW52ZVMM6WZYNRFUXMR4XJ6C5727BOFMR4RRH33Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.325865978 (571h53m42.196949023s) [seg_pos={0 0}, upload=501384]: Segment reached optimal threshold of 14 pieces. Dec 29 11:22:42.326137074 (571h53m42.197220109s) [noise=true , address=64.49.13.169:4708 , piece_id=B2X2RWRELXWJPF6AVJJZCNU46LM5ASGRE4EWGPKL3SCLQ4VKUBDQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection Dec 29 11:22:42.326153074 (571h53m42.197236109s) [noise=true , address=64.49.13.169:4708 , piece_id=B2X2RWRELXWJPF6AVJJZCNU46LM5ASGRE4EWGPKL3SCLQ4VKUBDQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326193401 (571h53m42.197276446s) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12h5esfg6mw2c8y3AVpeFBkL5FfLAwYdjhcnQ9CJ8fjh61WLYjx): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326205705 (571h53m42.19728874s) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326407739 (571h53m42.197490784s) [noise=true , address=77.90.31.10:4524 , piece_id=Q2JGEUV5EDFTVY2CSNPZOVJIIIAFBA3IHA7S5N7YZRH3E7TSTE5A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uN7CznoK4AD8FqFkNUndawdGJ9rmjAe7oNnjJkV844ScY87qe): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326415374 (571h53m42.197498419s) [noise=true , address=77.90.31.10:4524 , piece_id=Q2JGEUV5EDFTVY2CSNPZOVJIIIAFBA3IHA7S5N7YZRH3E7TSTE5A, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326505105 (571h53m42.19758815s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tnTEYvqJ6PB2DsuXA28sEuEiGj11StA4qdhcYH1kuoaMkTVdY): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326512188 (571h53m42.197595223s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326532357 (571h53m42.197615392s) [seg_pos={0 0}, upload=501384]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 11:22:42.326671562 (571h53m42.197754597s) [upload=501384]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 11:22:42.326677843 (571h53m42.197760879s) [upload=501384]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 11:22:42.610391438 (571h53m42.481474493s) [upload=501384]: Done uploading segments... Dec 29 11:22:42.610416826 (571h53m42.481499871s) [upload=501384]: Upload finished. err: <nil> Duration: 35.145061821s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 17 out of 60 is failed (threshold=6) [.X...X..........X....X.......X..X...XX..XX......X.X.XX.X.XX.] duration: 5.000452922s trace: https://jaeger.dev.storj.io/trace/102364c565a1cd6b FAIL (2025-12-29 11:34:08 UTC) orbiter warning: context deadline exceeded Duration: 5.000470645s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [X.XX....X..X.....X.X...X.X.X.X.XX..XX..X..XXX.X.......X.X..X] duration: 30.003904763s trace: https://jaeger.dev.storj.io/trace/7d5264e2a6b1751c FAIL (2025-12-29 11:34:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: 3NV4SDM7V5GJ5EEZQPBDM6XFK354G7LBAYCH444KJ5ICJXYDPEUQ) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: BNWSSQOGSBAT3YCZAHAJJQLVOEEPZYUNW64LHR5U75RIXVA7DDMQ) context deadline exceeded ======LOGS====== Dec 29 11:33:32.949900962 (572h4m32.820983997s) [upload=501542]: Starting upload Dec 29 11:33:32.949964402 (572h4m32.821047457s) [upload=501542]: Done starting upload Dec 29 11:33:32.950094059 (572h4m32.821177104s) [upload=501542]: Uploading segments... Dec 29 11:33:32.950220661 (572h4m32.821303696s) [upload=501542]: Got next segment. Inline: false Dec 29 11:33:32.950263332 (572h4m32.821346367s) [upload=501542]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 11:33:32.950268091 (572h4m32.821351136s) [upload=501542]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 11:33:33.197681415 (572h4m33.06876446s) [seg_pos={0 0}, upload=501542]: Begin upload segment... Dec 29 11:33:33.197744164 (572h4m33.068827199s) [seg_pos={0 0}, upload=501542]: Done begin upload segment. Dec 29 11:33:33.197763862 (572h4m33.068846937s) [noise=true , address=45.91.116.8:4048 , piece_id=5SM3PJXUVKE3ES55YZH6AUT5KBQ3AAKDKBRRKUX3ZX3OTSHVH4UA, seg_pos={0 0}, upload=501542]: Uploading piece... Dec 29 11:33:33.197863512 (572h4m33.068946557s) [noise=true , address=77.90.31.19:4842 , piece_id=VCBLKXBJLPLCAVPJPR777TCTNRWSYOEKEYN5H5YNAEZ7ZSVSB2RA, seg_pos={0 0}, upload=501542]: Uploading piece... Dec 29 11:33:33.198626615 (572h4m33.06970965s) [noise=true , address=77.90.31.18:4808 , piece_id=JFVH4LZULNIB6PLY2F5ELRH7RS2DFYMG7BXUESFY2SGQRVOFFJDA, seg_pos={0 0}, upload=501542]: Uploading piece... Dec 29 11:33:33.198932015 (572h4m33.070015051s) [noise=true , address=77.90.31.14:4661 , piece_id=X6FGJQA2H37TZBS7A7OBBG2LK6YSKGEAD4FVEPPDMRQ3U3M5MJYA, seg_pos={0 0}, upload=501542]: Uploading piece... Dec 29 11:33:33.199220636 (572h4m33.070303671s) [noise=true , address=145.79.227.38:4978 , piece_id=QNLRZOWXIVXBK24MHB56SUP5F5S5TQDVT2R64MMKGAHJMQMLO6EQ, seg_pos={0 0}, upload=501542]: Uploading piece... Dec 29 11:33:33.199482143 (572h4m33.070565188s) [noise=true , address=145.79.227.5:4223 , piece_id=QA235O4TAV2XOJXK36S27KTSLJWVHA7SGPBKLEOPYJPX62D4KVDA, seg_pos={0 0}, upload=501542]: Uploading piece... Dec 29 11:33:33.199737439 (572h4m33.070820484s) [noise=true , address=145.79.227.14:4839 , piece_id=KOZ47QJGCFCF3KNJ44VS2OSUDNXHUAJUCG73B4I3UHNPBMP66OTA, seg_pos={0 0}, upload=501542]: Uploading piece... Dec 29 11:33:33.200003616 (572h4m33.071086651s) [noise=true , address=145.79.227.45:4231 , piece_id=5TZR2KVOUSCTCBUIXDXUJRXMO4ZVPY7F4JNHSDP4NROO7WQEWR6A, seg_pos={0 0}, upload=501542]: Uploading piece... Dec 29 11:33:33.200418486 (572h4m33.071501521s) [noise=true , address=145.79.227.48:4302 , piece_id=3CPKMM7TXSPO2QXKQGOA53BTDETU6MPKAVMME6BNSFLMUGYJEVLA, seg_pos={0 0}, upload=501542]: Uploading piece... Dec 29 11:33:33.200847092 (572h4m33.071930127s) [noise=true , address=77.90.31.21:4877 , piece_id=6EHY77KDZINTESJN4IFO6MIN23HDTN4O7T5IGEJETDAON6NRLVTQ, seg_pos={0 0}, upload=501542]: Uploading piece... Dec 29 11:33:33.201382422 (572h4m33.072465467s) [noise=true , address=45.91.116.13:4153 , piece_id=3NV4SDM7V5GJ5EEZQPBDM6XFK354G7LBAYCH444KJ5ICJXYDPEUQ, seg_pos={0 0}, upload=501542]: Uploading piece... Dec 29 11:33:33.202452268 (572h4m33.073535304s) [noise=true , address=77.90.31.17:4773 , piece_id=VATYJS6YI3OT5TAG3MTRCUYHBVI2AQED7YB3U7ZWKCKKGZJKBV4A, seg_pos={0 0}, upload=501542]: Uploading piece... Dec 29 11:33:33.202754043 (572h4m33.073837079s) [noise=true , address=45.91.116.14:4190 , piece_id=BNWSSQOGSBAT3YCZAHAJJQLVOEEPZYUNW64LHR5U75RIXVA7DDMQ, seg_pos={0 0}, upload=501542]: Uploading piece... Dec 29 11:33:33.203943117 (572h4m33.075026163s) [noise=true , address=45.91.116.11:4119 , piece_id=LMJZCONZWHTFUYYTIENW672XWCEECRPEX32EC24SFOGZ62VKHD3Q, seg_pos={0 0}, upload=501542]: Uploading piece... Dec 29 11:33:33.205378330 (572h4m33.076461375s) [noise=true , address=45.91.116.5:4297 , piece_id=DNNMNO4JTRNNLPZKWP5BVQKW5YDBUCHOJ6LXJJ5WZGSPFANOI3EA, seg_pos={0 0}, upload=501542]: Uploading piece... Dec 29 11:33:33.207053991 (572h4m33.078137026s) [noise=true , address=145.79.227.4:4187 , piece_id=MWM6HCFA7J4W25ZFLOA3UL32BFXTIB5VPVNXOY2XRNM6TK5V2NBA, seg_pos={0 0}, upload=501542]: Uploading piece... Dec 29 11:33:34.899018782 (572h4m34.770101827s) [upload=501542]: Next returned nil segment Dec 29 11:33:34.899031206 (572h4m34.770114251s) [upload=501542]: Waiting for error group managing segments... Dec 29 11:33:35.183350333 (572h4m35.054433378s) [noise=true , address=145.79.227.48:4302 , piece_id=3CPKMM7TXSPO2QXKQGOA53BTDETU6MPKAVMME6BNSFLMUGYJEVLA, seg_pos={0 0}, upload=501542]: Done uploading piece. err: <nil> Dec 29 11:33:35.183532509 (572h4m35.054615544s) [noise=true , address=145.79.227.5:4223 , piece_id=QA235O4TAV2XOJXK36S27KTSLJWVHA7SGPBKLEOPYJPX62D4KVDA, seg_pos={0 0}, upload=501542]: Done uploading piece. err: <nil> Dec 29 11:33:35.190911029 (572h4m35.061994064s) [noise=true , address=145.79.227.45:4231 , piece_id=5TZR2KVOUSCTCBUIXDXUJRXMO4ZVPY7F4JNHSDP4NROO7WQEWR6A, seg_pos={0 0}, upload=501542]: Done uploading piece. err: <nil> Dec 29 11:33:35.199529030 (572h4m35.070612065s) [noise=true , address=145.79.227.4:4187 , piece_id=MWM6HCFA7J4W25ZFLOA3UL32BFXTIB5VPVNXOY2XRNM6TK5V2NBA, seg_pos={0 0}, upload=501542]: Done uploading piece. err: <nil> Dec 29 11:33:35.209616958 (572h4m35.080699993s) [noise=true , address=145.79.227.14:4839 , piece_id=KOZ47QJGCFCF3KNJ44VS2OSUDNXHUAJUCG73B4I3UHNPBMP66OTA, seg_pos={0 0}, upload=501542]: Done uploading piece. err: <nil> Dec 29 11:33:35.220204839 (572h4m35.091287874s) [noise=true , address=145.79.227.38:4978 , piece_id=QNLRZOWXIVXBK24MHB56SUP5F5S5TQDVT2R64MMKGAHJMQMLO6EQ, seg_pos={0 0}, upload=501542]: Done uploading piece. err: <nil> Dec 29 11:33:35.405746028 (572h4m35.276829063s) [noise=true , address=77.90.31.14:4661 , piece_id=X6FGJQA2H37TZBS7A7OBBG2LK6YSKGEAD4FVEPPDMRQ3U3M5MJYA, seg_pos={0 0}, upload=501542]: Done uploading piece. err: <nil> Dec 29 11:33:35.471805645 (572h4m35.34288868s) [noise=true , address=45.91.116.13:4153 , piece_id=3NV4SDM7V5GJ5EEZQPBDM6XFK354G7LBAYCH444KJ5ICJXYDPEUQ, seg_pos={0 0}, upload=501542]: Done uploading piece. err: <nil> Dec 29 11:33:35.482728244 (572h4m35.353811279s) [noise=true , address=45.91.116.8:4048 , piece_id=5SM3PJXUVKE3ES55YZH6AUT5KBQ3AAKDKBRRKUX3ZX3OTSHVH4UA, seg_pos={0 0}, upload=501542]: Done uploading piece. err: <nil> Dec 29 11:33:35.562364463 (572h4m35.433447498s) [noise=true , address=77.90.31.21:4877 , piece_id=6EHY77KDZINTESJN4IFO6MIN23HDTN4O7T5IGEJETDAON6NRLVTQ, seg_pos={0 0}, upload=501542]: Done uploading piece. err: <nil> Dec 29 11:33:35.574257479 (572h4m35.445340514s) [noise=true , address=45.91.116.14:4190 , piece_id=BNWSSQOGSBAT3YCZAHAJJQLVOEEPZYUNW64LHR5U75RIXVA7DDMQ, seg_pos={0 0}, upload=501542]: Done uploading piece. err: <nil> Dec 29 11:33:35.666488982 (572h4m35.537572027s) [noise=true , address=45.91.116.11:4119 , piece_id=LMJZCONZWHTFUYYTIENW672XWCEECRPEX32EC24SFOGZ62VKHD3Q, seg_pos={0 0}, upload=501542]: Done uploading piece. err: <nil> Dec 29 11:33:36.047396097 (572h4m35.918479132s) [noise=true , address=77.90.31.19:4842 , piece_id=VCBLKXBJLPLCAVPJPR777TCTNRWSYOEKEYN5H5YNAEZ7ZSVSB2RA, seg_pos={0 0}, upload=501542]: Done uploading piece. err: <nil> Dec 29 11:33:36.047416907 (572h4m35.918499942s) [seg_pos={0 0}, upload=501542]: Segment reached optimal threshold of 13 pieces. Dec 29 11:33:36.047623811 (572h4m35.918706846s) [noise=true , address=77.90.31.18:4808 , piece_id=JFVH4LZULNIB6PLY2F5ELRH7RS2DFYMG7BXUESFY2SGQRVOFFJDA, seg_pos={0 0}, upload=501542]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: write tcp 188.166.225.170:41492->77.90.31.18:4808: use of closed network connection; write tcp 188.166.225.170:41492->77.90.31.18:4808: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41492->77.90.31.18:4808: use of closed network connection Dec 29 11:33:36.047630654 (572h4m35.918713689s) [noise=true , address=77.90.31.18:4808 , piece_id=JFVH4LZULNIB6PLY2F5ELRH7RS2DFYMG7BXUESFY2SGQRVOFFJDA, seg_pos={0 0}, upload=501542]: Long tail cancellation due to optimal threshold Dec 29 11:33:36.047748979 (572h4m35.918832024s) [noise=true , address=45.91.116.5:4297 , piece_id=DNNMNO4JTRNNLPZKWP5BVQKW5YDBUCHOJ6LXJJ5WZGSPFANOI3EA, seg_pos={0 0}, upload=501542]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 29 11:33:36.047765430 (572h4m35.918848465s) [noise=true , address=45.91.116.5:4297 , piece_id=DNNMNO4JTRNNLPZKWP5BVQKW5YDBUCHOJ6LXJJ5WZGSPFANOI3EA, seg_pos={0 0}, upload=501542]: Long tail cancellation due to optimal threshold Dec 29 11:33:36.047871973 (572h4m35.918955008s) [noise=true , address=77.90.31.17:4773 , piece_id=VATYJS6YI3OT5TAG3MTRCUYHBVI2AQED7YB3U7ZWKCKKGZJKBV4A, seg_pos={0 0}, upload=501542]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: expected piece hash; context canceled; EOF Dec 29 11:33:36.047878786 (572h4m35.918961821s) [noise=true , address=77.90.31.17:4773 , piece_id=VATYJS6YI3OT5TAG3MTRCUYHBVI2AQED7YB3U7ZWKCKKGZJKBV4A, seg_pos={0 0}, upload=501542]: Long tail cancellation due to optimal threshold Dec 29 11:33:36.047896871 (572h4m35.918979906s) [seg_pos={0 0}, upload=501542]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 11:33:36.047961684 (572h4m35.91904472s) [upload=501542]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 11:33:36.047990919 (572h4m35.919073954s) [upload=501542]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 11:33:36.342340696 (572h4m36.213423741s) [upload=501542]: Done uploading segments... Dec 29 11:33:36.342399549 (572h4m36.213482624s) [upload=501542]: Upload finished. err: <nil> Duration: 30.003918599s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........................X...............X..X.............] duration: 33.614047184s trace: https://jaeger.dev.storj.io/trace/7e7354d8023608a8 FAIL (2025-12-29 11:33:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== ) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480097317 (571h53m39.351180352s) [noise=true , address=77.90.31.21:4889 , piece_id=BR6Z55P3A7YJ5FJRDSZFUIF66FGIJQY7WCEQ7PL5DOYWL73OMKTQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480405053 (571h53m39.351488098s) [noise=true , address=77.90.31.16:4750 , piece_id=MVVZPJEGDCBSNVOB6CD2WUZGIJTL3VBHX2G4L3TVZ6JF5ZEJ5U6Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480712969 (571h53m39.351796014s) [noise=true , address=77.90.31.18:4834 , piece_id=BTWNGNQVEUEVV6HLBDM63SGNTKV4RDU4J4XFLG52WHKP53H3L5KA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481019453 (571h53m39.352102488s) [noise=true , address=145.79.227.30:4777 , piece_id=VSCXLXGIBSFE6E4OHLWNZIZ75PO7WQO2VFHUYHW7JUNH62KGGU7Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481029532 (571h53m39.352112567s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481559832 (571h53m39.352642877s) [noise=true , address=145.79.227.35:4515 , piece_id=L7LFZKZEZERUNX4HUNDNQ2PWHXNJRJ7OBFRNMFIICPZEUIBHGP7A, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481731919 (571h53m39.352814954s) [noise=true , address=64.49.13.26:4953 , piece_id=4YSW5Y3A6HYG3XJ2XAB7RGVHAOHMP7G3QIZYMYTSXVXYJ3ANLNXA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482078329 (571h53m39.353161364s) [noise=true , address=145.79.227.44:4205 , piece_id=UE4QSHDQBZ3XBWQ6I6MCRVTLMU7U6W6OZDA2ZGQCH6ZYRAGPD7WA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482099830 (571h53m39.353182865s) [noise=true , address=145.79.227.40:4067 , piece_id=WSYDRX5YJ6OCFW52ZVMM6WZYNRFUXMR4XJ6C5727BOFMR4RRH33Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482434487 (571h53m39.353517532s) [noise=true , address=145.79.227.3:4164 , piece_id=U6F7VDPOH7K6R5EH5UUXHFP55GXOEXO5BLBJHZRTITPDQI4LP7KQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482480845 (571h53m39.35356388s) [noise=true , address=64.49.13.140:4516 , piece_id=4GUYOYOAEASUK2W25S3NILWIIIHGOZX76L4EGFLUUUEDWRALPP2A, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482841442 (571h53m39.353924477s) [noise=true , address=145.79.227.34:4910 , piece_id=ZH4DCNXET2Q6VRAUH3NQ3KKSV4OYI575X7YS72XD236EEOMOG3LA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483163645 (571h53m39.35424668s) [noise=true , address=145.79.227.29:4974 , piece_id=KEKREOD2B35XQNI45JBHE2SZZC3MQQBQUR5I6WFDAQYJQGJBXSMA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483532478 (571h53m39.354615513s) [noise=true , address=77.90.31.17:4769 , piece_id=3MKUMOQIUVERKQLLIFX52XQPTQ5LSFMHIPD5YW2YTO5YEFR36IYA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483914596 (571h53m39.354997631s) [noise=true , address=145.79.227.13:4807 , piece_id=VKXB5MWAG2YYH2AWWJEGTLBZ2QUPLI25VTHI6AMMALSM23Z3AWZQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:40.761721106 (571h53m40.632804141s) [upload=501384]: Next returned nil segment Dec 29 11:22:40.761733349 (571h53m40.632816384s) [upload=501384]: Waiting for error group managing segments... Dec 29 11:22:41.121222862 (571h53m40.992305897s) [noise=true , address=64.49.13.140:4516 , piece_id=4GUYOYOAEASUK2W25S3NILWIIIHGOZX76L4EGFLUUUEDWRALPP2A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.197311646 (571h53m41.068394691s) [noise=true , address=64.49.13.26:4953 , piece_id=4YSW5Y3A6HYG3XJ2XAB7RGVHAOHMP7G3QIZYMYTSXVXYJ3ANLNXA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.347247568 (571h53m41.218330613s) [noise=true , address=145.79.227.13:4807 , piece_id=VKXB5MWAG2YYH2AWWJEGTLBZ2QUPLI25VTHI6AMMALSM23Z3AWZQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.347917313 (571h53m41.219000348s) [noise=true , address=145.79.227.3:4164 , piece_id=U6F7VDPOH7K6R5EH5UUXHFP55GXOEXO5BLBJHZRTITPDQI4LP7KQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.348510613 (571h53m41.219593648s) [noise=true , address=145.79.227.29:4974 , piece_id=KEKREOD2B35XQNI45JBHE2SZZC3MQQBQUR5I6WFDAQYJQGJBXSMA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.503950985 (571h53m41.37503402s) [noise=true , address=77.90.31.17:4769 , piece_id=3MKUMOQIUVERKQLLIFX52XQPTQ5LSFMHIPD5YW2YTO5YEFR36IYA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.561352025 (571h53m41.43243506s) [noise=true , address=77.90.31.18:4834 , piece_id=BTWNGNQVEUEVV6HLBDM63SGNTKV4RDU4J4XFLG52WHKP53H3L5KA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.622447380 (571h53m41.493530425s) [noise=true , address=77.90.31.16:4750 , piece_id=MVVZPJEGDCBSNVOB6CD2WUZGIJTL3VBHX2G4L3TVZ6JF5ZEJ5U6Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.645494685 (571h53m41.51657773s) [noise=true , address=77.90.31.21:4889 , piece_id=BR6Z55P3A7YJ5FJRDSZFUIF66FGIJQY7WCEQ7PL5DOYWL73OMKTQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.247930045 (571h53m42.11901308s) [noise=true , address=145.79.227.30:4777 , piece_id=VSCXLXGIBSFE6E4OHLWNZIZ75PO7WQO2VFHUYHW7JUNH62KGGU7Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.262221890 (571h53m42.133304925s) [noise=true , address=145.79.227.44:4205 , piece_id=UE4QSHDQBZ3XBWQ6I6MCRVTLMU7U6W6OZDA2ZGQCH6ZYRAGPD7WA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.295598123 (571h53m42.166681158s) [noise=true , address=145.79.227.35:4515 , piece_id=L7LFZKZEZERUNX4HUNDNQ2PWHXNJRJ7OBFRNMFIICPZEUIBHGP7A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.307481351 (571h53m42.178564386s) [noise=true , address=145.79.227.34:4910 , piece_id=ZH4DCNXET2Q6VRAUH3NQ3KKSV4OYI575X7YS72XD236EEOMOG3LA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.325775906 (571h53m42.196858941s) [noise=true , address=145.79.227.40:4067 , piece_id=WSYDRX5YJ6OCFW52ZVMM6WZYNRFUXMR4XJ6C5727BOFMR4RRH33Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.325865978 (571h53m42.196949023s) [seg_pos={0 0}, upload=501384]: Segment reached optimal threshold of 14 pieces. Dec 29 11:22:42.326137074 (571h53m42.197220109s) [noise=true , address=64.49.13.169:4708 , piece_id=B2X2RWRELXWJPF6AVJJZCNU46LM5ASGRE4EWGPKL3SCLQ4VKUBDQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection Dec 29 11:22:42.326153074 (571h53m42.197236109s) [noise=true , address=64.49.13.169:4708 , piece_id=B2X2RWRELXWJPF6AVJJZCNU46LM5ASGRE4EWGPKL3SCLQ4VKUBDQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326193401 (571h53m42.197276446s) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12h5esfg6mw2c8y3AVpeFBkL5FfLAwYdjhcnQ9CJ8fjh61WLYjx): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326205705 (571h53m42.19728874s) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326407739 (571h53m42.197490784s) [noise=true , address=77.90.31.10:4524 , piece_id=Q2JGEUV5EDFTVY2CSNPZOVJIIIAFBA3IHA7S5N7YZRH3E7TSTE5A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uN7CznoK4AD8FqFkNUndawdGJ9rmjAe7oNnjJkV844ScY87qe): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326415374 (571h53m42.197498419s) [noise=true , address=77.90.31.10:4524 , piece_id=Q2JGEUV5EDFTVY2CSNPZOVJIIIAFBA3IHA7S5N7YZRH3E7TSTE5A, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326505105 (571h53m42.19758815s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tnTEYvqJ6PB2DsuXA28sEuEiGj11StA4qdhcYH1kuoaMkTVdY): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326512188 (571h53m42.197595223s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326532357 (571h53m42.197615392s) [seg_pos={0 0}, upload=501384]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 11:22:42.326671562 (571h53m42.197754597s) [upload=501384]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 11:22:42.326677843 (571h53m42.197760879s) [upload=501384]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 11:22:42.610391438 (571h53m42.481474493s) [upload=501384]: Done uploading segments... Dec 29 11:22:42.610416826 (571h53m42.481499871s) [upload=501384]: Upload finished. err: <nil> Duration: 33.614062104s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)