Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [......X.....X..................X..............X.X.........X.] duration: 3.587707754s trace: https://jaeger.dev.storj.io/trace/42d15caac6419534 FAIL (2025-12-28 10:41:08 UTC) orbiter warning: context deadline exceeded Duration: 3.587795632s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [..X...................X..XX.....X..X............XX..........] duration: 5.718886215s trace: https://jaeger.dev.storj.io/trace/12fdeb4fc6be55f2 FAIL (2025-12-28 10:40:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: JRQ4W24BKCX2ELV7C6P3QXHLF4BWUO7FEI7CMDMV43YYTK7FB3HQ) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: UMHHU3Y6JA7GBPWMG3NZRBNJ6QSP762N3LZUH4HEOJXEQBO5NP4A) context deadline exceeded ======LOGS====== Dec 28 10:32:09.171162516 (547h3m9.042245551s) [upload=479693]: Starting upload Dec 28 10:32:09.171211930 (547h3m9.042294965s) [upload=479693]: Done starting upload Dec 28 10:32:09.171389138 (547h3m9.042472173s) [upload=479693]: Uploading segments... Dec 28 10:32:09.171420808 (547h3m9.042503843s) [upload=479693]: Got next segment. Inline: false Dec 28 10:32:09.171446587 (547h3m9.042529622s) [upload=479693]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 10:32:09.171451076 (547h3m9.042534121s) [upload=479693]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 10:32:09.415778928 (547h3m9.286861973s) [seg_pos={0 0}, upload=479693]: Begin upload segment... Dec 28 10:32:09.415823763 (547h3m9.286906798s) [seg_pos={0 0}, upload=479693]: Done begin upload segment. Dec 28 10:32:09.415852628 (547h3m9.286935673s) [noise=true , address=145.79.227.5:4223 , piece_id=2DUTU22HULS2ERHZZ2TYB2YTL5ZUUAL5Z65V5VSNDKAUNWOYLFWQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.415915609 (547h3m9.286998644s) [noise=true , address=45.91.116.14:4192 , piece_id=UMHHU3Y6JA7GBPWMG3NZRBNJ6QSP762N3LZUH4HEOJXEQBO5NP4A, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.416427983 (547h3m9.287511019s) [noise=true , address=145.79.227.41:4086 , piece_id=OTV62ELETLJDI25ANYVI5TOAQKAMO5TRHWSGFLQSHBBRGBO3PLVQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.416839006 (547h3m9.287922041s) [noise=true , address=145.79.227.14:4840 , piece_id=6FSN47UNI7MQJBWBIV5THQD4WUXF52NM7M6BP7VHRB4J6E2F4VKQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.416935741 (547h3m9.288018786s) [noise=true , address=77.90.31.18:4808 , piece_id=UZP6F2ITW5Q3AFHKITHSUTPKZZVK2ZBLFF5DHH5UA5URTSPCP7JA, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.417242896 (547h3m9.288325941s) [noise=true , address=77.90.31.21:4877 , piece_id=3ELH72FJ2GAD7CFLEMWFNKERUTYJFB6FE2V7S7YGOONGYU6WPBNQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.417527407 (547h3m9.288610442s) [noise=true , address=45.91.116.13:4154 , piece_id=4VHHRCQCEPZUWD2QT6V62WVKBRXMFY7NYAQCAF4DVB4OFCRSFGUA, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.417826456 (547h3m9.288909491s) [noise=true , address=145.79.227.3:4149 , piece_id=GUIDXMU3Q5RTZTK2IOIJ2CXIMQ5HQGNDSEN5WVVZMRR336IM5FGQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.417930104 (547h3m9.289013149s) [noise=true , address=145.79.227.2:4113 , piece_id=7VXXE267VCB2OXILXW6TV4AJGNHW4LQVXEA4HOSVNMI6UAEMS3VA, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.418096461 (547h3m9.289179506s) [noise=true , address=145.79.227.47:4265 , piece_id=TDVXCOFNRIOSRS2OETJPUM3EELE4YN73KLHWDSPL6XPX5TACZCWQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.418451897 (547h3m9.289534942s) [noise=true , address=145.79.227.37:4567 , piece_id=ZY7KVVWYGKO6X2OONS7LXBBXDYKLSNIGL33U72HP6INZOUDUIS5A, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.419025139 (547h3m9.290108184s) [noise=true , address=77.90.31.15:4700 , piece_id=CTHN5F73XF6BQ2ATYAUFBRGTD6YWEOK3U7OKAFXDMDLJ7DUZPGYQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.419682581 (547h3m9.290765626s) [noise=true , address=45.91.116.6:4975 , piece_id=JRQ4W24BKCX2ELV7C6P3QXHLF4BWUO7FEI7CMDMV43YYTK7FB3HQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.420365671 (547h3m9.291448706s) [noise=true , address=145.79.227.46:4603 , piece_id=HUJL4BAY7U2G57TOMZVHQWTFOTN2RV2XKDK2MIUILVD6HR6Y27NQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.421011522 (547h3m9.292094557s) [noise=true , address=145.79.227.10:4403 , piece_id=FECST4PG6RGH6TUCOJZCSIOVRESXJMABM235ZOO5M4GYZTY75LTQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.421697077 (547h3m9.292780122s) [noise=true , address=145.79.227.29:4941 , piece_id=U5QPSY5M32O3ZGTK4YNIQNLEXHF72OPVCXWPRCDTFEIPWKGVNV2A, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:11.133534367 (547h3m11.004617412s) [upload=479693]: Next returned nil segment Dec 28 10:32:11.133569233 (547h3m11.004652278s) [upload=479693]: Waiting for error group managing segments... Dec 28 10:32:11.421476958 (547h3m11.292559993s) [noise=true , address=145.79.227.3:4149 , piece_id=GUIDXMU3Q5RTZTK2IOIJ2CXIMQ5HQGNDSEN5WVVZMRR336IM5FGQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.423163530 (547h3m11.294246575s) [noise=true , address=145.79.227.14:4840 , piece_id=6FSN47UNI7MQJBWBIV5THQD4WUXF52NM7M6BP7VHRB4J6E2F4VKQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.428115299 (547h3m11.299198334s) [noise=true , address=145.79.227.41:4086 , piece_id=OTV62ELETLJDI25ANYVI5TOAQKAMO5TRHWSGFLQSHBBRGBO3PLVQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.436758778 (547h3m11.307841823s) [noise=true , address=145.79.227.10:4403 , piece_id=FECST4PG6RGH6TUCOJZCSIOVRESXJMABM235ZOO5M4GYZTY75LTQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.439394376 (547h3m11.310477411s) [noise=true , address=145.79.227.46:4603 , piece_id=HUJL4BAY7U2G57TOMZVHQWTFOTN2RV2XKDK2MIUILVD6HR6Y27NQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.439861225 (547h3m11.31094427s) [noise=true , address=145.79.227.47:4265 , piece_id=TDVXCOFNRIOSRS2OETJPUM3EELE4YN73KLHWDSPL6XPX5TACZCWQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.442937283 (547h3m11.314020318s) [noise=true , address=145.79.227.37:4567 , piece_id=ZY7KVVWYGKO6X2OONS7LXBBXDYKLSNIGL33U72HP6INZOUDUIS5A, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.444228622 (547h3m11.315311667s) [noise=true , address=145.79.227.5:4223 , piece_id=2DUTU22HULS2ERHZZ2TYB2YTL5ZUUAL5Z65V5VSNDKAUNWOYLFWQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.452679474 (547h3m11.323762519s) [noise=true , address=145.79.227.2:4113 , piece_id=7VXXE267VCB2OXILXW6TV4AJGNHW4LQVXEA4HOSVNMI6UAEMS3VA, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.457542493 (547h3m11.328625538s) [noise=true , address=145.79.227.29:4941 , piece_id=U5QPSY5M32O3ZGTK4YNIQNLEXHF72OPVCXWPRCDTFEIPWKGVNV2A, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.708367176 (547h3m11.579450221s) [noise=true , address=45.91.116.13:4154 , piece_id=4VHHRCQCEPZUWD2QT6V62WVKBRXMFY7NYAQCAF4DVB4OFCRSFGUA, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.716026030 (547h3m11.587109075s) [noise=true , address=45.91.116.14:4192 , piece_id=UMHHU3Y6JA7GBPWMG3NZRBNJ6QSP762N3LZUH4HEOJXEQBO5NP4A, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:12.434215195 (547h3m12.30529824s) [noise=true , address=45.91.116.6:4975 , piece_id=JRQ4W24BKCX2ELV7C6P3QXHLF4BWUO7FEI7CMDMV43YYTK7FB3HQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:12.434324403 (547h3m12.305407438s) [seg_pos={0 0}, upload=479693]: Segment reached optimal threshold of 13 pieces. Dec 28 10:32:12.434515456 (547h3m12.305598501s) [noise=true , address=77.90.31.21:4877 , piece_id=3ELH72FJ2GAD7CFLEMWFNKERUTYJFB6FE2V7S7YGOONGYU6WPBNQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Dec 28 10:32:12.434521548 (547h3m12.305604593s) [noise=true , address=77.90.31.21:4877 , piece_id=3ELH72FJ2GAD7CFLEMWFNKERUTYJFB6FE2V7S7YGOONGYU6WPBNQ, seg_pos={0 0}, upload=479693]: Long tail cancellation due to optimal threshold Dec 28 10:32:12.434710859 (547h3m12.305793894s) [noise=true , address=77.90.31.15:4700 , piece_id=CTHN5F73XF6BQ2ATYAUFBRGTD6YWEOK3U7OKAFXDMDLJ7DUZPGYQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 28 10:32:12.434716900 (547h3m12.305799945s) [noise=true , address=77.90.31.15:4700 , piece_id=CTHN5F73XF6BQ2ATYAUFBRGTD6YWEOK3U7OKAFXDMDLJ7DUZPGYQ, seg_pos={0 0}, upload=479693]: Long tail cancellation due to optimal threshold Dec 28 10:32:12.434797904 (547h3m12.305880949s) [noise=true , address=77.90.31.18:4808 , piece_id=UZP6F2ITW5Q3AFHKITHSUTPKZZVK2ZBLFF5DHH5UA5URTSPCP7JA, seg_pos={0 0}, upload=479693]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF Dec 28 10:32:12.434813053 (547h3m12.305896088s) [noise=true , address=77.90.31.18:4808 , piece_id=UZP6F2ITW5Q3AFHKITHSUTPKZZVK2ZBLFF5DHH5UA5URTSPCP7JA, seg_pos={0 0}, upload=479693]: Long tail cancellation due to optimal threshold Dec 28 10:32:12.434845295 (547h3m12.30592833s) [seg_pos={0 0}, upload=479693]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 10:32:12.434957198 (547h3m12.306040233s) [upload=479693]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 10:32:12.434962488 (547h3m12.306045523s) [upload=479693]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 10:32:12.725143372 (547h3m12.596226407s) [upload=479693]: Done uploading segments... Dec 28 10:32:12.725163410 (547h3m12.596246445s) [upload=479693]: Upload finished. err: <nil> Duration: 5.71891431s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [.......X.....X..................X..............X.X.........X] duration: 5.001242753s trace: https://jaeger.dev.storj.io/trace/42d15caac6419534 FAIL (2025-12-28 10:40:14 UTC) orbiter warning: context deadline exceeded Duration: 5.001268192s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [...X...................X..XX.....X..X............XX.........] duration: 5.637291508s trace: https://jaeger.dev.storj.io/trace/12fdeb4fc6be55f2 FAIL (2025-12-28 10:39:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: JRQ4W24BKCX2ELV7C6P3QXHLF4BWUO7FEI7CMDMV43YYTK7FB3HQ) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: UMHHU3Y6JA7GBPWMG3NZRBNJ6QSP762N3LZUH4HEOJXEQBO5NP4A) context deadline exceeded ======LOGS====== Dec 28 10:32:09.171162516 (547h3m9.042245551s) [upload=479693]: Starting upload Dec 28 10:32:09.171211930 (547h3m9.042294965s) [upload=479693]: Done starting upload Dec 28 10:32:09.171389138 (547h3m9.042472173s) [upload=479693]: Uploading segments... Dec 28 10:32:09.171420808 (547h3m9.042503843s) [upload=479693]: Got next segment. Inline: false Dec 28 10:32:09.171446587 (547h3m9.042529622s) [upload=479693]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 10:32:09.171451076 (547h3m9.042534121s) [upload=479693]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 10:32:09.415778928 (547h3m9.286861973s) [seg_pos={0 0}, upload=479693]: Begin upload segment... Dec 28 10:32:09.415823763 (547h3m9.286906798s) [seg_pos={0 0}, upload=479693]: Done begin upload segment. Dec 28 10:32:09.415852628 (547h3m9.286935673s) [noise=true , address=145.79.227.5:4223 , piece_id=2DUTU22HULS2ERHZZ2TYB2YTL5ZUUAL5Z65V5VSNDKAUNWOYLFWQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.415915609 (547h3m9.286998644s) [noise=true , address=45.91.116.14:4192 , piece_id=UMHHU3Y6JA7GBPWMG3NZRBNJ6QSP762N3LZUH4HEOJXEQBO5NP4A, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.416427983 (547h3m9.287511019s) [noise=true , address=145.79.227.41:4086 , piece_id=OTV62ELETLJDI25ANYVI5TOAQKAMO5TRHWSGFLQSHBBRGBO3PLVQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.416839006 (547h3m9.287922041s) [noise=true , address=145.79.227.14:4840 , piece_id=6FSN47UNI7MQJBWBIV5THQD4WUXF52NM7M6BP7VHRB4J6E2F4VKQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.416935741 (547h3m9.288018786s) [noise=true , address=77.90.31.18:4808 , piece_id=UZP6F2ITW5Q3AFHKITHSUTPKZZVK2ZBLFF5DHH5UA5URTSPCP7JA, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.417242896 (547h3m9.288325941s) [noise=true , address=77.90.31.21:4877 , piece_id=3ELH72FJ2GAD7CFLEMWFNKERUTYJFB6FE2V7S7YGOONGYU6WPBNQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.417527407 (547h3m9.288610442s) [noise=true , address=45.91.116.13:4154 , piece_id=4VHHRCQCEPZUWD2QT6V62WVKBRXMFY7NYAQCAF4DVB4OFCRSFGUA, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.417826456 (547h3m9.288909491s) [noise=true , address=145.79.227.3:4149 , piece_id=GUIDXMU3Q5RTZTK2IOIJ2CXIMQ5HQGNDSEN5WVVZMRR336IM5FGQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.417930104 (547h3m9.289013149s) [noise=true , address=145.79.227.2:4113 , piece_id=7VXXE267VCB2OXILXW6TV4AJGNHW4LQVXEA4HOSVNMI6UAEMS3VA, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.418096461 (547h3m9.289179506s) [noise=true , address=145.79.227.47:4265 , piece_id=TDVXCOFNRIOSRS2OETJPUM3EELE4YN73KLHWDSPL6XPX5TACZCWQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.418451897 (547h3m9.289534942s) [noise=true , address=145.79.227.37:4567 , piece_id=ZY7KVVWYGKO6X2OONS7LXBBXDYKLSNIGL33U72HP6INZOUDUIS5A, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.419025139 (547h3m9.290108184s) [noise=true , address=77.90.31.15:4700 , piece_id=CTHN5F73XF6BQ2ATYAUFBRGTD6YWEOK3U7OKAFXDMDLJ7DUZPGYQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.419682581 (547h3m9.290765626s) [noise=true , address=45.91.116.6:4975 , piece_id=JRQ4W24BKCX2ELV7C6P3QXHLF4BWUO7FEI7CMDMV43YYTK7FB3HQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.420365671 (547h3m9.291448706s) [noise=true , address=145.79.227.46:4603 , piece_id=HUJL4BAY7U2G57TOMZVHQWTFOTN2RV2XKDK2MIUILVD6HR6Y27NQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.421011522 (547h3m9.292094557s) [noise=true , address=145.79.227.10:4403 , piece_id=FECST4PG6RGH6TUCOJZCSIOVRESXJMABM235ZOO5M4GYZTY75LTQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.421697077 (547h3m9.292780122s) [noise=true , address=145.79.227.29:4941 , piece_id=U5QPSY5M32O3ZGTK4YNIQNLEXHF72OPVCXWPRCDTFEIPWKGVNV2A, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:11.133534367 (547h3m11.004617412s) [upload=479693]: Next returned nil segment Dec 28 10:32:11.133569233 (547h3m11.004652278s) [upload=479693]: Waiting for error group managing segments... Dec 28 10:32:11.421476958 (547h3m11.292559993s) [noise=true , address=145.79.227.3:4149 , piece_id=GUIDXMU3Q5RTZTK2IOIJ2CXIMQ5HQGNDSEN5WVVZMRR336IM5FGQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.423163530 (547h3m11.294246575s) [noise=true , address=145.79.227.14:4840 , piece_id=6FSN47UNI7MQJBWBIV5THQD4WUXF52NM7M6BP7VHRB4J6E2F4VKQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.428115299 (547h3m11.299198334s) [noise=true , address=145.79.227.41:4086 , piece_id=OTV62ELETLJDI25ANYVI5TOAQKAMO5TRHWSGFLQSHBBRGBO3PLVQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.436758778 (547h3m11.307841823s) [noise=true , address=145.79.227.10:4403 , piece_id=FECST4PG6RGH6TUCOJZCSIOVRESXJMABM235ZOO5M4GYZTY75LTQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.439394376 (547h3m11.310477411s) [noise=true , address=145.79.227.46:4603 , piece_id=HUJL4BAY7U2G57TOMZVHQWTFOTN2RV2XKDK2MIUILVD6HR6Y27NQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.439861225 (547h3m11.31094427s) [noise=true , address=145.79.227.47:4265 , piece_id=TDVXCOFNRIOSRS2OETJPUM3EELE4YN73KLHWDSPL6XPX5TACZCWQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.442937283 (547h3m11.314020318s) [noise=true , address=145.79.227.37:4567 , piece_id=ZY7KVVWYGKO6X2OONS7LXBBXDYKLSNIGL33U72HP6INZOUDUIS5A, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.444228622 (547h3m11.315311667s) [noise=true , address=145.79.227.5:4223 , piece_id=2DUTU22HULS2ERHZZ2TYB2YTL5ZUUAL5Z65V5VSNDKAUNWOYLFWQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.452679474 (547h3m11.323762519s) [noise=true , address=145.79.227.2:4113 , piece_id=7VXXE267VCB2OXILXW6TV4AJGNHW4LQVXEA4HOSVNMI6UAEMS3VA, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.457542493 (547h3m11.328625538s) [noise=true , address=145.79.227.29:4941 , piece_id=U5QPSY5M32O3ZGTK4YNIQNLEXHF72OPVCXWPRCDTFEIPWKGVNV2A, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.708367176 (547h3m11.579450221s) [noise=true , address=45.91.116.13:4154 , piece_id=4VHHRCQCEPZUWD2QT6V62WVKBRXMFY7NYAQCAF4DVB4OFCRSFGUA, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.716026030 (547h3m11.587109075s) [noise=true , address=45.91.116.14:4192 , piece_id=UMHHU3Y6JA7GBPWMG3NZRBNJ6QSP762N3LZUH4HEOJXEQBO5NP4A, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:12.434215195 (547h3m12.30529824s) [noise=true , address=45.91.116.6:4975 , piece_id=JRQ4W24BKCX2ELV7C6P3QXHLF4BWUO7FEI7CMDMV43YYTK7FB3HQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:12.434324403 (547h3m12.305407438s) [seg_pos={0 0}, upload=479693]: Segment reached optimal threshold of 13 pieces. Dec 28 10:32:12.434515456 (547h3m12.305598501s) [noise=true , address=77.90.31.21:4877 , piece_id=3ELH72FJ2GAD7CFLEMWFNKERUTYJFB6FE2V7S7YGOONGYU6WPBNQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Dec 28 10:32:12.434521548 (547h3m12.305604593s) [noise=true , address=77.90.31.21:4877 , piece_id=3ELH72FJ2GAD7CFLEMWFNKERUTYJFB6FE2V7S7YGOONGYU6WPBNQ, seg_pos={0 0}, upload=479693]: Long tail cancellation due to optimal threshold Dec 28 10:32:12.434710859 (547h3m12.305793894s) [noise=true , address=77.90.31.15:4700 , piece_id=CTHN5F73XF6BQ2ATYAUFBRGTD6YWEOK3U7OKAFXDMDLJ7DUZPGYQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 28 10:32:12.434716900 (547h3m12.305799945s) [noise=true , address=77.90.31.15:4700 , piece_id=CTHN5F73XF6BQ2ATYAUFBRGTD6YWEOK3U7OKAFXDMDLJ7DUZPGYQ, seg_pos={0 0}, upload=479693]: Long tail cancellation due to optimal threshold Dec 28 10:32:12.434797904 (547h3m12.305880949s) [noise=true , address=77.90.31.18:4808 , piece_id=UZP6F2ITW5Q3AFHKITHSUTPKZZVK2ZBLFF5DHH5UA5URTSPCP7JA, seg_pos={0 0}, upload=479693]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF Dec 28 10:32:12.434813053 (547h3m12.305896088s) [noise=true , address=77.90.31.18:4808 , piece_id=UZP6F2ITW5Q3AFHKITHSUTPKZZVK2ZBLFF5DHH5UA5URTSPCP7JA, seg_pos={0 0}, upload=479693]: Long tail cancellation due to optimal threshold Dec 28 10:32:12.434845295 (547h3m12.30592833s) [seg_pos={0 0}, upload=479693]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 10:32:12.434957198 (547h3m12.306040233s) [upload=479693]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 10:32:12.434962488 (547h3m12.306045523s) [upload=479693]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 10:32:12.725143372 (547h3m12.596226407s) [upload=479693]: Done uploading segments... Dec 28 10:32:12.725163410 (547h3m12.596246445s) [upload=479693]: Upload finished. err: <nil> Duration: 5.63732428s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [........X.....X..................X..............X.X.........] duration: 3.66170653s trace: https://jaeger.dev.storj.io/trace/57df8e96b91a6205 FAIL (2025-12-28 10:39:24 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.661732339s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [....X...................X..XX.....X..X............XX........] duration: 5.837708598s trace: https://jaeger.dev.storj.io/trace/12fdeb4fc6be55f2 FAIL (2025-12-28 10:39:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: JRQ4W24BKCX2ELV7C6P3QXHLF4BWUO7FEI7CMDMV43YYTK7FB3HQ) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: UMHHU3Y6JA7GBPWMG3NZRBNJ6QSP762N3LZUH4HEOJXEQBO5NP4A) context deadline exceeded ======LOGS====== Dec 28 10:32:09.171162516 (547h3m9.042245551s) [upload=479693]: Starting upload Dec 28 10:32:09.171211930 (547h3m9.042294965s) [upload=479693]: Done starting upload Dec 28 10:32:09.171389138 (547h3m9.042472173s) [upload=479693]: Uploading segments... Dec 28 10:32:09.171420808 (547h3m9.042503843s) [upload=479693]: Got next segment. Inline: false Dec 28 10:32:09.171446587 (547h3m9.042529622s) [upload=479693]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 10:32:09.171451076 (547h3m9.042534121s) [upload=479693]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 10:32:09.415778928 (547h3m9.286861973s) [seg_pos={0 0}, upload=479693]: Begin upload segment... Dec 28 10:32:09.415823763 (547h3m9.286906798s) [seg_pos={0 0}, upload=479693]: Done begin upload segment. Dec 28 10:32:09.415852628 (547h3m9.286935673s) [noise=true , address=145.79.227.5:4223 , piece_id=2DUTU22HULS2ERHZZ2TYB2YTL5ZUUAL5Z65V5VSNDKAUNWOYLFWQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.415915609 (547h3m9.286998644s) [noise=true , address=45.91.116.14:4192 , piece_id=UMHHU3Y6JA7GBPWMG3NZRBNJ6QSP762N3LZUH4HEOJXEQBO5NP4A, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.416427983 (547h3m9.287511019s) [noise=true , address=145.79.227.41:4086 , piece_id=OTV62ELETLJDI25ANYVI5TOAQKAMO5TRHWSGFLQSHBBRGBO3PLVQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.416839006 (547h3m9.287922041s) [noise=true , address=145.79.227.14:4840 , piece_id=6FSN47UNI7MQJBWBIV5THQD4WUXF52NM7M6BP7VHRB4J6E2F4VKQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.416935741 (547h3m9.288018786s) [noise=true , address=77.90.31.18:4808 , piece_id=UZP6F2ITW5Q3AFHKITHSUTPKZZVK2ZBLFF5DHH5UA5URTSPCP7JA, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.417242896 (547h3m9.288325941s) [noise=true , address=77.90.31.21:4877 , piece_id=3ELH72FJ2GAD7CFLEMWFNKERUTYJFB6FE2V7S7YGOONGYU6WPBNQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.417527407 (547h3m9.288610442s) [noise=true , address=45.91.116.13:4154 , piece_id=4VHHRCQCEPZUWD2QT6V62WVKBRXMFY7NYAQCAF4DVB4OFCRSFGUA, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.417826456 (547h3m9.288909491s) [noise=true , address=145.79.227.3:4149 , piece_id=GUIDXMU3Q5RTZTK2IOIJ2CXIMQ5HQGNDSEN5WVVZMRR336IM5FGQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.417930104 (547h3m9.289013149s) [noise=true , address=145.79.227.2:4113 , piece_id=7VXXE267VCB2OXILXW6TV4AJGNHW4LQVXEA4HOSVNMI6UAEMS3VA, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.418096461 (547h3m9.289179506s) [noise=true , address=145.79.227.47:4265 , piece_id=TDVXCOFNRIOSRS2OETJPUM3EELE4YN73KLHWDSPL6XPX5TACZCWQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.418451897 (547h3m9.289534942s) [noise=true , address=145.79.227.37:4567 , piece_id=ZY7KVVWYGKO6X2OONS7LXBBXDYKLSNIGL33U72HP6INZOUDUIS5A, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.419025139 (547h3m9.290108184s) [noise=true , address=77.90.31.15:4700 , piece_id=CTHN5F73XF6BQ2ATYAUFBRGTD6YWEOK3U7OKAFXDMDLJ7DUZPGYQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.419682581 (547h3m9.290765626s) [noise=true , address=45.91.116.6:4975 , piece_id=JRQ4W24BKCX2ELV7C6P3QXHLF4BWUO7FEI7CMDMV43YYTK7FB3HQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.420365671 (547h3m9.291448706s) [noise=true , address=145.79.227.46:4603 , piece_id=HUJL4BAY7U2G57TOMZVHQWTFOTN2RV2XKDK2MIUILVD6HR6Y27NQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.421011522 (547h3m9.292094557s) [noise=true , address=145.79.227.10:4403 , piece_id=FECST4PG6RGH6TUCOJZCSIOVRESXJMABM235ZOO5M4GYZTY75LTQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.421697077 (547h3m9.292780122s) [noise=true , address=145.79.227.29:4941 , piece_id=U5QPSY5M32O3ZGTK4YNIQNLEXHF72OPVCXWPRCDTFEIPWKGVNV2A, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:11.133534367 (547h3m11.004617412s) [upload=479693]: Next returned nil segment Dec 28 10:32:11.133569233 (547h3m11.004652278s) [upload=479693]: Waiting for error group managing segments... Dec 28 10:32:11.421476958 (547h3m11.292559993s) [noise=true , address=145.79.227.3:4149 , piece_id=GUIDXMU3Q5RTZTK2IOIJ2CXIMQ5HQGNDSEN5WVVZMRR336IM5FGQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.423163530 (547h3m11.294246575s) [noise=true , address=145.79.227.14:4840 , piece_id=6FSN47UNI7MQJBWBIV5THQD4WUXF52NM7M6BP7VHRB4J6E2F4VKQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.428115299 (547h3m11.299198334s) [noise=true , address=145.79.227.41:4086 , piece_id=OTV62ELETLJDI25ANYVI5TOAQKAMO5TRHWSGFLQSHBBRGBO3PLVQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.436758778 (547h3m11.307841823s) [noise=true , address=145.79.227.10:4403 , piece_id=FECST4PG6RGH6TUCOJZCSIOVRESXJMABM235ZOO5M4GYZTY75LTQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.439394376 (547h3m11.310477411s) [noise=true , address=145.79.227.46:4603 , piece_id=HUJL4BAY7U2G57TOMZVHQWTFOTN2RV2XKDK2MIUILVD6HR6Y27NQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.439861225 (547h3m11.31094427s) [noise=true , address=145.79.227.47:4265 , piece_id=TDVXCOFNRIOSRS2OETJPUM3EELE4YN73KLHWDSPL6XPX5TACZCWQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.442937283 (547h3m11.314020318s) [noise=true , address=145.79.227.37:4567 , piece_id=ZY7KVVWYGKO6X2OONS7LXBBXDYKLSNIGL33U72HP6INZOUDUIS5A, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.444228622 (547h3m11.315311667s) [noise=true , address=145.79.227.5:4223 , piece_id=2DUTU22HULS2ERHZZ2TYB2YTL5ZUUAL5Z65V5VSNDKAUNWOYLFWQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.452679474 (547h3m11.323762519s) [noise=true , address=145.79.227.2:4113 , piece_id=7VXXE267VCB2OXILXW6TV4AJGNHW4LQVXEA4HOSVNMI6UAEMS3VA, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.457542493 (547h3m11.328625538s) [noise=true , address=145.79.227.29:4941 , piece_id=U5QPSY5M32O3ZGTK4YNIQNLEXHF72OPVCXWPRCDTFEIPWKGVNV2A, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.708367176 (547h3m11.579450221s) [noise=true , address=45.91.116.13:4154 , piece_id=4VHHRCQCEPZUWD2QT6V62WVKBRXMFY7NYAQCAF4DVB4OFCRSFGUA, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.716026030 (547h3m11.587109075s) [noise=true , address=45.91.116.14:4192 , piece_id=UMHHU3Y6JA7GBPWMG3NZRBNJ6QSP762N3LZUH4HEOJXEQBO5NP4A, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:12.434215195 (547h3m12.30529824s) [noise=true , address=45.91.116.6:4975 , piece_id=JRQ4W24BKCX2ELV7C6P3QXHLF4BWUO7FEI7CMDMV43YYTK7FB3HQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:12.434324403 (547h3m12.305407438s) [seg_pos={0 0}, upload=479693]: Segment reached optimal threshold of 13 pieces. Dec 28 10:32:12.434515456 (547h3m12.305598501s) [noise=true , address=77.90.31.21:4877 , piece_id=3ELH72FJ2GAD7CFLEMWFNKERUTYJFB6FE2V7S7YGOONGYU6WPBNQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Dec 28 10:32:12.434521548 (547h3m12.305604593s) [noise=true , address=77.90.31.21:4877 , piece_id=3ELH72FJ2GAD7CFLEMWFNKERUTYJFB6FE2V7S7YGOONGYU6WPBNQ, seg_pos={0 0}, upload=479693]: Long tail cancellation due to optimal threshold Dec 28 10:32:12.434710859 (547h3m12.305793894s) [noise=true , address=77.90.31.15:4700 , piece_id=CTHN5F73XF6BQ2ATYAUFBRGTD6YWEOK3U7OKAFXDMDLJ7DUZPGYQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 28 10:32:12.434716900 (547h3m12.305799945s) [noise=true , address=77.90.31.15:4700 , piece_id=CTHN5F73XF6BQ2ATYAUFBRGTD6YWEOK3U7OKAFXDMDLJ7DUZPGYQ, seg_pos={0 0}, upload=479693]: Long tail cancellation due to optimal threshold Dec 28 10:32:12.434797904 (547h3m12.305880949s) [noise=true , address=77.90.31.18:4808 , piece_id=UZP6F2ITW5Q3AFHKITHSUTPKZZVK2ZBLFF5DHH5UA5URTSPCP7JA, seg_pos={0 0}, upload=479693]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF Dec 28 10:32:12.434813053 (547h3m12.305896088s) [noise=true , address=77.90.31.18:4808 , piece_id=UZP6F2ITW5Q3AFHKITHSUTPKZZVK2ZBLFF5DHH5UA5URTSPCP7JA, seg_pos={0 0}, upload=479693]: Long tail cancellation due to optimal threshold Dec 28 10:32:12.434845295 (547h3m12.30592833s) [seg_pos={0 0}, upload=479693]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 10:32:12.434957198 (547h3m12.306040233s) [upload=479693]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 10:32:12.434962488 (547h3m12.306045523s) [upload=479693]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 10:32:12.725143372 (547h3m12.596226407s) [upload=479693]: Done uploading segments... Dec 28 10:32:12.725163410 (547h3m12.596246445s) [upload=479693]: Upload finished. err: <nil> Duration: 5.83773659s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [X........X.....X..................X..............X.X........] duration: 3.413997399s trace: https://jaeger.dev.storj.io/trace/57df8e96b91a6205 FAIL (2025-12-28 10:38:34 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.414249369s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.....X...................X..XX.....X..X............XX.......] duration: 5.929829585s trace: https://jaeger.dev.storj.io/trace/12fdeb4fc6be55f2 FAIL (2025-12-28 10:38:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: JRQ4W24BKCX2ELV7C6P3QXHLF4BWUO7FEI7CMDMV43YYTK7FB3HQ) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: UMHHU3Y6JA7GBPWMG3NZRBNJ6QSP762N3LZUH4HEOJXEQBO5NP4A) context deadline exceeded ======LOGS====== Dec 28 10:32:09.171162516 (547h3m9.042245551s) [upload=479693]: Starting upload Dec 28 10:32:09.171211930 (547h3m9.042294965s) [upload=479693]: Done starting upload Dec 28 10:32:09.171389138 (547h3m9.042472173s) [upload=479693]: Uploading segments... Dec 28 10:32:09.171420808 (547h3m9.042503843s) [upload=479693]: Got next segment. Inline: false Dec 28 10:32:09.171446587 (547h3m9.042529622s) [upload=479693]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 10:32:09.171451076 (547h3m9.042534121s) [upload=479693]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 10:32:09.415778928 (547h3m9.286861973s) [seg_pos={0 0}, upload=479693]: Begin upload segment... Dec 28 10:32:09.415823763 (547h3m9.286906798s) [seg_pos={0 0}, upload=479693]: Done begin upload segment. Dec 28 10:32:09.415852628 (547h3m9.286935673s) [noise=true , address=145.79.227.5:4223 , piece_id=2DUTU22HULS2ERHZZ2TYB2YTL5ZUUAL5Z65V5VSNDKAUNWOYLFWQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.415915609 (547h3m9.286998644s) [noise=true , address=45.91.116.14:4192 , piece_id=UMHHU3Y6JA7GBPWMG3NZRBNJ6QSP762N3LZUH4HEOJXEQBO5NP4A, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.416427983 (547h3m9.287511019s) [noise=true , address=145.79.227.41:4086 , piece_id=OTV62ELETLJDI25ANYVI5TOAQKAMO5TRHWSGFLQSHBBRGBO3PLVQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.416839006 (547h3m9.287922041s) [noise=true , address=145.79.227.14:4840 , piece_id=6FSN47UNI7MQJBWBIV5THQD4WUXF52NM7M6BP7VHRB4J6E2F4VKQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.416935741 (547h3m9.288018786s) [noise=true , address=77.90.31.18:4808 , piece_id=UZP6F2ITW5Q3AFHKITHSUTPKZZVK2ZBLFF5DHH5UA5URTSPCP7JA, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.417242896 (547h3m9.288325941s) [noise=true , address=77.90.31.21:4877 , piece_id=3ELH72FJ2GAD7CFLEMWFNKERUTYJFB6FE2V7S7YGOONGYU6WPBNQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.417527407 (547h3m9.288610442s) [noise=true , address=45.91.116.13:4154 , piece_id=4VHHRCQCEPZUWD2QT6V62WVKBRXMFY7NYAQCAF4DVB4OFCRSFGUA, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.417826456 (547h3m9.288909491s) [noise=true , address=145.79.227.3:4149 , piece_id=GUIDXMU3Q5RTZTK2IOIJ2CXIMQ5HQGNDSEN5WVVZMRR336IM5FGQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.417930104 (547h3m9.289013149s) [noise=true , address=145.79.227.2:4113 , piece_id=7VXXE267VCB2OXILXW6TV4AJGNHW4LQVXEA4HOSVNMI6UAEMS3VA, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.418096461 (547h3m9.289179506s) [noise=true , address=145.79.227.47:4265 , piece_id=TDVXCOFNRIOSRS2OETJPUM3EELE4YN73KLHWDSPL6XPX5TACZCWQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.418451897 (547h3m9.289534942s) [noise=true , address=145.79.227.37:4567 , piece_id=ZY7KVVWYGKO6X2OONS7LXBBXDYKLSNIGL33U72HP6INZOUDUIS5A, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.419025139 (547h3m9.290108184s) [noise=true , address=77.90.31.15:4700 , piece_id=CTHN5F73XF6BQ2ATYAUFBRGTD6YWEOK3U7OKAFXDMDLJ7DUZPGYQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.419682581 (547h3m9.290765626s) [noise=true , address=45.91.116.6:4975 , piece_id=JRQ4W24BKCX2ELV7C6P3QXHLF4BWUO7FEI7CMDMV43YYTK7FB3HQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.420365671 (547h3m9.291448706s) [noise=true , address=145.79.227.46:4603 , piece_id=HUJL4BAY7U2G57TOMZVHQWTFOTN2RV2XKDK2MIUILVD6HR6Y27NQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.421011522 (547h3m9.292094557s) [noise=true , address=145.79.227.10:4403 , piece_id=FECST4PG6RGH6TUCOJZCSIOVRESXJMABM235ZOO5M4GYZTY75LTQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.421697077 (547h3m9.292780122s) [noise=true , address=145.79.227.29:4941 , piece_id=U5QPSY5M32O3ZGTK4YNIQNLEXHF72OPVCXWPRCDTFEIPWKGVNV2A, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:11.133534367 (547h3m11.004617412s) [upload=479693]: Next returned nil segment Dec 28 10:32:11.133569233 (547h3m11.004652278s) [upload=479693]: Waiting for error group managing segments... Dec 28 10:32:11.421476958 (547h3m11.292559993s) [noise=true , address=145.79.227.3:4149 , piece_id=GUIDXMU3Q5RTZTK2IOIJ2CXIMQ5HQGNDSEN5WVVZMRR336IM5FGQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.423163530 (547h3m11.294246575s) [noise=true , address=145.79.227.14:4840 , piece_id=6FSN47UNI7MQJBWBIV5THQD4WUXF52NM7M6BP7VHRB4J6E2F4VKQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.428115299 (547h3m11.299198334s) [noise=true , address=145.79.227.41:4086 , piece_id=OTV62ELETLJDI25ANYVI5TOAQKAMO5TRHWSGFLQSHBBRGBO3PLVQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.436758778 (547h3m11.307841823s) [noise=true , address=145.79.227.10:4403 , piece_id=FECST4PG6RGH6TUCOJZCSIOVRESXJMABM235ZOO5M4GYZTY75LTQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.439394376 (547h3m11.310477411s) [noise=true , address=145.79.227.46:4603 , piece_id=HUJL4BAY7U2G57TOMZVHQWTFOTN2RV2XKDK2MIUILVD6HR6Y27NQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.439861225 (547h3m11.31094427s) [noise=true , address=145.79.227.47:4265 , piece_id=TDVXCOFNRIOSRS2OETJPUM3EELE4YN73KLHWDSPL6XPX5TACZCWQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.442937283 (547h3m11.314020318s) [noise=true , address=145.79.227.37:4567 , piece_id=ZY7KVVWYGKO6X2OONS7LXBBXDYKLSNIGL33U72HP6INZOUDUIS5A, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.444228622 (547h3m11.315311667s) [noise=true , address=145.79.227.5:4223 , piece_id=2DUTU22HULS2ERHZZ2TYB2YTL5ZUUAL5Z65V5VSNDKAUNWOYLFWQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.452679474 (547h3m11.323762519s) [noise=true , address=145.79.227.2:4113 , piece_id=7VXXE267VCB2OXILXW6TV4AJGNHW4LQVXEA4HOSVNMI6UAEMS3VA, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.457542493 (547h3m11.328625538s) [noise=true , address=145.79.227.29:4941 , piece_id=U5QPSY5M32O3ZGTK4YNIQNLEXHF72OPVCXWPRCDTFEIPWKGVNV2A, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.708367176 (547h3m11.579450221s) [noise=true , address=45.91.116.13:4154 , piece_id=4VHHRCQCEPZUWD2QT6V62WVKBRXMFY7NYAQCAF4DVB4OFCRSFGUA, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.716026030 (547h3m11.587109075s) [noise=true , address=45.91.116.14:4192 , piece_id=UMHHU3Y6JA7GBPWMG3NZRBNJ6QSP762N3LZUH4HEOJXEQBO5NP4A, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:12.434215195 (547h3m12.30529824s) [noise=true , address=45.91.116.6:4975 , piece_id=JRQ4W24BKCX2ELV7C6P3QXHLF4BWUO7FEI7CMDMV43YYTK7FB3HQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:12.434324403 (547h3m12.305407438s) [seg_pos={0 0}, upload=479693]: Segment reached optimal threshold of 13 pieces. Dec 28 10:32:12.434515456 (547h3m12.305598501s) [noise=true , address=77.90.31.21:4877 , piece_id=3ELH72FJ2GAD7CFLEMWFNKERUTYJFB6FE2V7S7YGOONGYU6WPBNQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Dec 28 10:32:12.434521548 (547h3m12.305604593s) [noise=true , address=77.90.31.21:4877 , piece_id=3ELH72FJ2GAD7CFLEMWFNKERUTYJFB6FE2V7S7YGOONGYU6WPBNQ, seg_pos={0 0}, upload=479693]: Long tail cancellation due to optimal threshold Dec 28 10:32:12.434710859 (547h3m12.305793894s) [noise=true , address=77.90.31.15:4700 , piece_id=CTHN5F73XF6BQ2ATYAUFBRGTD6YWEOK3U7OKAFXDMDLJ7DUZPGYQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 28 10:32:12.434716900 (547h3m12.305799945s) [noise=true , address=77.90.31.15:4700 , piece_id=CTHN5F73XF6BQ2ATYAUFBRGTD6YWEOK3U7OKAFXDMDLJ7DUZPGYQ, seg_pos={0 0}, upload=479693]: Long tail cancellation due to optimal threshold Dec 28 10:32:12.434797904 (547h3m12.305880949s) [noise=true , address=77.90.31.18:4808 , piece_id=UZP6F2ITW5Q3AFHKITHSUTPKZZVK2ZBLFF5DHH5UA5URTSPCP7JA, seg_pos={0 0}, upload=479693]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF Dec 28 10:32:12.434813053 (547h3m12.305896088s) [noise=true , address=77.90.31.18:4808 , piece_id=UZP6F2ITW5Q3AFHKITHSUTPKZZVK2ZBLFF5DHH5UA5URTSPCP7JA, seg_pos={0 0}, upload=479693]: Long tail cancellation due to optimal threshold Dec 28 10:32:12.434845295 (547h3m12.30592833s) [seg_pos={0 0}, upload=479693]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 10:32:12.434957198 (547h3m12.306040233s) [upload=479693]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 10:32:12.434962488 (547h3m12.306045523s) [upload=479693]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 10:32:12.725143372 (547h3m12.596226407s) [upload=479693]: Done uploading segments... Dec 28 10:32:12.725163410 (547h3m12.596246445s) [upload=479693]: Upload finished. err: <nil> Duration: 5.929856097s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [.X........X.....X..................X..............X.X.......] duration: 5.000778147s trace: https://jaeger.dev.storj.io/trace/57df8e96b91a6205 FAIL (2025-12-28 10:38:15 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000815609s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [..X........X.....X..................X..............X.X......] duration: 3.467988517s trace: https://jaeger.dev.storj.io/trace/57df8e96b91a6205 FAIL (2025-12-28 10:37:37 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.468144865s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [......X...................X..XX.....X..X............XX......] duration: 6.198266809s trace: https://jaeger.dev.storj.io/trace/12fdeb4fc6be55f2 FAIL (2025-12-28 10:37:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: JRQ4W24BKCX2ELV7C6P3QXHLF4BWUO7FEI7CMDMV43YYTK7FB3HQ) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: UMHHU3Y6JA7GBPWMG3NZRBNJ6QSP762N3LZUH4HEOJXEQBO5NP4A) context deadline exceeded ======LOGS====== Dec 28 10:32:09.171162516 (547h3m9.042245551s) [upload=479693]: Starting upload Dec 28 10:32:09.171211930 (547h3m9.042294965s) [upload=479693]: Done starting upload Dec 28 10:32:09.171389138 (547h3m9.042472173s) [upload=479693]: Uploading segments... Dec 28 10:32:09.171420808 (547h3m9.042503843s) [upload=479693]: Got next segment. Inline: false Dec 28 10:32:09.171446587 (547h3m9.042529622s) [upload=479693]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 10:32:09.171451076 (547h3m9.042534121s) [upload=479693]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 10:32:09.415778928 (547h3m9.286861973s) [seg_pos={0 0}, upload=479693]: Begin upload segment... Dec 28 10:32:09.415823763 (547h3m9.286906798s) [seg_pos={0 0}, upload=479693]: Done begin upload segment. Dec 28 10:32:09.415852628 (547h3m9.286935673s) [noise=true , address=145.79.227.5:4223 , piece_id=2DUTU22HULS2ERHZZ2TYB2YTL5ZUUAL5Z65V5VSNDKAUNWOYLFWQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.415915609 (547h3m9.286998644s) [noise=true , address=45.91.116.14:4192 , piece_id=UMHHU3Y6JA7GBPWMG3NZRBNJ6QSP762N3LZUH4HEOJXEQBO5NP4A, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.416427983 (547h3m9.287511019s) [noise=true , address=145.79.227.41:4086 , piece_id=OTV62ELETLJDI25ANYVI5TOAQKAMO5TRHWSGFLQSHBBRGBO3PLVQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.416839006 (547h3m9.287922041s) [noise=true , address=145.79.227.14:4840 , piece_id=6FSN47UNI7MQJBWBIV5THQD4WUXF52NM7M6BP7VHRB4J6E2F4VKQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.416935741 (547h3m9.288018786s) [noise=true , address=77.90.31.18:4808 , piece_id=UZP6F2ITW5Q3AFHKITHSUTPKZZVK2ZBLFF5DHH5UA5URTSPCP7JA, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.417242896 (547h3m9.288325941s) [noise=true , address=77.90.31.21:4877 , piece_id=3ELH72FJ2GAD7CFLEMWFNKERUTYJFB6FE2V7S7YGOONGYU6WPBNQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.417527407 (547h3m9.288610442s) [noise=true , address=45.91.116.13:4154 , piece_id=4VHHRCQCEPZUWD2QT6V62WVKBRXMFY7NYAQCAF4DVB4OFCRSFGUA, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.417826456 (547h3m9.288909491s) [noise=true , address=145.79.227.3:4149 , piece_id=GUIDXMU3Q5RTZTK2IOIJ2CXIMQ5HQGNDSEN5WVVZMRR336IM5FGQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.417930104 (547h3m9.289013149s) [noise=true , address=145.79.227.2:4113 , piece_id=7VXXE267VCB2OXILXW6TV4AJGNHW4LQVXEA4HOSVNMI6UAEMS3VA, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.418096461 (547h3m9.289179506s) [noise=true , address=145.79.227.47:4265 , piece_id=TDVXCOFNRIOSRS2OETJPUM3EELE4YN73KLHWDSPL6XPX5TACZCWQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.418451897 (547h3m9.289534942s) [noise=true , address=145.79.227.37:4567 , piece_id=ZY7KVVWYGKO6X2OONS7LXBBXDYKLSNIGL33U72HP6INZOUDUIS5A, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.419025139 (547h3m9.290108184s) [noise=true , address=77.90.31.15:4700 , piece_id=CTHN5F73XF6BQ2ATYAUFBRGTD6YWEOK3U7OKAFXDMDLJ7DUZPGYQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.419682581 (547h3m9.290765626s) [noise=true , address=45.91.116.6:4975 , piece_id=JRQ4W24BKCX2ELV7C6P3QXHLF4BWUO7FEI7CMDMV43YYTK7FB3HQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.420365671 (547h3m9.291448706s) [noise=true , address=145.79.227.46:4603 , piece_id=HUJL4BAY7U2G57TOMZVHQWTFOTN2RV2XKDK2MIUILVD6HR6Y27NQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.421011522 (547h3m9.292094557s) [noise=true , address=145.79.227.10:4403 , piece_id=FECST4PG6RGH6TUCOJZCSIOVRESXJMABM235ZOO5M4GYZTY75LTQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.421697077 (547h3m9.292780122s) [noise=true , address=145.79.227.29:4941 , piece_id=U5QPSY5M32O3ZGTK4YNIQNLEXHF72OPVCXWPRCDTFEIPWKGVNV2A, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:11.133534367 (547h3m11.004617412s) [upload=479693]: Next returned nil segment Dec 28 10:32:11.133569233 (547h3m11.004652278s) [upload=479693]: Waiting for error group managing segments... Dec 28 10:32:11.421476958 (547h3m11.292559993s) [noise=true , address=145.79.227.3:4149 , piece_id=GUIDXMU3Q5RTZTK2IOIJ2CXIMQ5HQGNDSEN5WVVZMRR336IM5FGQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.423163530 (547h3m11.294246575s) [noise=true , address=145.79.227.14:4840 , piece_id=6FSN47UNI7MQJBWBIV5THQD4WUXF52NM7M6BP7VHRB4J6E2F4VKQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.428115299 (547h3m11.299198334s) [noise=true , address=145.79.227.41:4086 , piece_id=OTV62ELETLJDI25ANYVI5TOAQKAMO5TRHWSGFLQSHBBRGBO3PLVQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.436758778 (547h3m11.307841823s) [noise=true , address=145.79.227.10:4403 , piece_id=FECST4PG6RGH6TUCOJZCSIOVRESXJMABM235ZOO5M4GYZTY75LTQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.439394376 (547h3m11.310477411s) [noise=true , address=145.79.227.46:4603 , piece_id=HUJL4BAY7U2G57TOMZVHQWTFOTN2RV2XKDK2MIUILVD6HR6Y27NQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.439861225 (547h3m11.31094427s) [noise=true , address=145.79.227.47:4265 , piece_id=TDVXCOFNRIOSRS2OETJPUM3EELE4YN73KLHWDSPL6XPX5TACZCWQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.442937283 (547h3m11.314020318s) [noise=true , address=145.79.227.37:4567 , piece_id=ZY7KVVWYGKO6X2OONS7LXBBXDYKLSNIGL33U72HP6INZOUDUIS5A, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.444228622 (547h3m11.315311667s) [noise=true , address=145.79.227.5:4223 , piece_id=2DUTU22HULS2ERHZZ2TYB2YTL5ZUUAL5Z65V5VSNDKAUNWOYLFWQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.452679474 (547h3m11.323762519s) [noise=true , address=145.79.227.2:4113 , piece_id=7VXXE267VCB2OXILXW6TV4AJGNHW4LQVXEA4HOSVNMI6UAEMS3VA, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.457542493 (547h3m11.328625538s) [noise=true , address=145.79.227.29:4941 , piece_id=U5QPSY5M32O3ZGTK4YNIQNLEXHF72OPVCXWPRCDTFEIPWKGVNV2A, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.708367176 (547h3m11.579450221s) [noise=true , address=45.91.116.13:4154 , piece_id=4VHHRCQCEPZUWD2QT6V62WVKBRXMFY7NYAQCAF4DVB4OFCRSFGUA, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.716026030 (547h3m11.587109075s) [noise=true , address=45.91.116.14:4192 , piece_id=UMHHU3Y6JA7GBPWMG3NZRBNJ6QSP762N3LZUH4HEOJXEQBO5NP4A, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:12.434215195 (547h3m12.30529824s) [noise=true , address=45.91.116.6:4975 , piece_id=JRQ4W24BKCX2ELV7C6P3QXHLF4BWUO7FEI7CMDMV43YYTK7FB3HQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:12.434324403 (547h3m12.305407438s) [seg_pos={0 0}, upload=479693]: Segment reached optimal threshold of 13 pieces. Dec 28 10:32:12.434515456 (547h3m12.305598501s) [noise=true , address=77.90.31.21:4877 , piece_id=3ELH72FJ2GAD7CFLEMWFNKERUTYJFB6FE2V7S7YGOONGYU6WPBNQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Dec 28 10:32:12.434521548 (547h3m12.305604593s) [noise=true , address=77.90.31.21:4877 , piece_id=3ELH72FJ2GAD7CFLEMWFNKERUTYJFB6FE2V7S7YGOONGYU6WPBNQ, seg_pos={0 0}, upload=479693]: Long tail cancellation due to optimal threshold Dec 28 10:32:12.434710859 (547h3m12.305793894s) [noise=true , address=77.90.31.15:4700 , piece_id=CTHN5F73XF6BQ2ATYAUFBRGTD6YWEOK3U7OKAFXDMDLJ7DUZPGYQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 28 10:32:12.434716900 (547h3m12.305799945s) [noise=true , address=77.90.31.15:4700 , piece_id=CTHN5F73XF6BQ2ATYAUFBRGTD6YWEOK3U7OKAFXDMDLJ7DUZPGYQ, seg_pos={0 0}, upload=479693]: Long tail cancellation due to optimal threshold Dec 28 10:32:12.434797904 (547h3m12.305880949s) [noise=true , address=77.90.31.18:4808 , piece_id=UZP6F2ITW5Q3AFHKITHSUTPKZZVK2ZBLFF5DHH5UA5URTSPCP7JA, seg_pos={0 0}, upload=479693]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF Dec 28 10:32:12.434813053 (547h3m12.305896088s) [noise=true , address=77.90.31.18:4808 , piece_id=UZP6F2ITW5Q3AFHKITHSUTPKZZVK2ZBLFF5DHH5UA5URTSPCP7JA, seg_pos={0 0}, upload=479693]: Long tail cancellation due to optimal threshold Dec 28 10:32:12.434845295 (547h3m12.30592833s) [seg_pos={0 0}, upload=479693]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 10:32:12.434957198 (547h3m12.306040233s) [upload=479693]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 10:32:12.434962488 (547h3m12.306045523s) [upload=479693]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 10:32:12.725143372 (547h3m12.596226407s) [upload=479693]: Done uploading segments... Dec 28 10:32:12.725163410 (547h3m12.596246445s) [upload=479693]: Upload finished. err: <nil> Duration: 6.198294221s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [...X........X.....X..................X..............X.X.....] duration: 3.97057648s trace: https://jaeger.dev.storj.io/trace/57df8e96b91a6205 FAIL (2025-12-28 10:36:38 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.970601518s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.......X...................X..XX.....X..X............XX.....] duration: 6.139021024s trace: https://jaeger.dev.storj.io/trace/12fdeb4fc6be55f2 FAIL (2025-12-28 10:36:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: JRQ4W24BKCX2ELV7C6P3QXHLF4BWUO7FEI7CMDMV43YYTK7FB3HQ) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: UMHHU3Y6JA7GBPWMG3NZRBNJ6QSP762N3LZUH4HEOJXEQBO5NP4A) context deadline exceeded ======LOGS====== Dec 28 10:32:09.171162516 (547h3m9.042245551s) [upload=479693]: Starting upload Dec 28 10:32:09.171211930 (547h3m9.042294965s) [upload=479693]: Done starting upload Dec 28 10:32:09.171389138 (547h3m9.042472173s) [upload=479693]: Uploading segments... Dec 28 10:32:09.171420808 (547h3m9.042503843s) [upload=479693]: Got next segment. Inline: false Dec 28 10:32:09.171446587 (547h3m9.042529622s) [upload=479693]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 10:32:09.171451076 (547h3m9.042534121s) [upload=479693]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 10:32:09.415778928 (547h3m9.286861973s) [seg_pos={0 0}, upload=479693]: Begin upload segment... Dec 28 10:32:09.415823763 (547h3m9.286906798s) [seg_pos={0 0}, upload=479693]: Done begin upload segment. Dec 28 10:32:09.415852628 (547h3m9.286935673s) [noise=true , address=145.79.227.5:4223 , piece_id=2DUTU22HULS2ERHZZ2TYB2YTL5ZUUAL5Z65V5VSNDKAUNWOYLFWQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.415915609 (547h3m9.286998644s) [noise=true , address=45.91.116.14:4192 , piece_id=UMHHU3Y6JA7GBPWMG3NZRBNJ6QSP762N3LZUH4HEOJXEQBO5NP4A, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.416427983 (547h3m9.287511019s) [noise=true , address=145.79.227.41:4086 , piece_id=OTV62ELETLJDI25ANYVI5TOAQKAMO5TRHWSGFLQSHBBRGBO3PLVQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.416839006 (547h3m9.287922041s) [noise=true , address=145.79.227.14:4840 , piece_id=6FSN47UNI7MQJBWBIV5THQD4WUXF52NM7M6BP7VHRB4J6E2F4VKQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.416935741 (547h3m9.288018786s) [noise=true , address=77.90.31.18:4808 , piece_id=UZP6F2ITW5Q3AFHKITHSUTPKZZVK2ZBLFF5DHH5UA5URTSPCP7JA, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.417242896 (547h3m9.288325941s) [noise=true , address=77.90.31.21:4877 , piece_id=3ELH72FJ2GAD7CFLEMWFNKERUTYJFB6FE2V7S7YGOONGYU6WPBNQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.417527407 (547h3m9.288610442s) [noise=true , address=45.91.116.13:4154 , piece_id=4VHHRCQCEPZUWD2QT6V62WVKBRXMFY7NYAQCAF4DVB4OFCRSFGUA, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.417826456 (547h3m9.288909491s) [noise=true , address=145.79.227.3:4149 , piece_id=GUIDXMU3Q5RTZTK2IOIJ2CXIMQ5HQGNDSEN5WVVZMRR336IM5FGQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.417930104 (547h3m9.289013149s) [noise=true , address=145.79.227.2:4113 , piece_id=7VXXE267VCB2OXILXW6TV4AJGNHW4LQVXEA4HOSVNMI6UAEMS3VA, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.418096461 (547h3m9.289179506s) [noise=true , address=145.79.227.47:4265 , piece_id=TDVXCOFNRIOSRS2OETJPUM3EELE4YN73KLHWDSPL6XPX5TACZCWQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.418451897 (547h3m9.289534942s) [noise=true , address=145.79.227.37:4567 , piece_id=ZY7KVVWYGKO6X2OONS7LXBBXDYKLSNIGL33U72HP6INZOUDUIS5A, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.419025139 (547h3m9.290108184s) [noise=true , address=77.90.31.15:4700 , piece_id=CTHN5F73XF6BQ2ATYAUFBRGTD6YWEOK3U7OKAFXDMDLJ7DUZPGYQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.419682581 (547h3m9.290765626s) [noise=true , address=45.91.116.6:4975 , piece_id=JRQ4W24BKCX2ELV7C6P3QXHLF4BWUO7FEI7CMDMV43YYTK7FB3HQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.420365671 (547h3m9.291448706s) [noise=true , address=145.79.227.46:4603 , piece_id=HUJL4BAY7U2G57TOMZVHQWTFOTN2RV2XKDK2MIUILVD6HR6Y27NQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.421011522 (547h3m9.292094557s) [noise=true , address=145.79.227.10:4403 , piece_id=FECST4PG6RGH6TUCOJZCSIOVRESXJMABM235ZOO5M4GYZTY75LTQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.421697077 (547h3m9.292780122s) [noise=true , address=145.79.227.29:4941 , piece_id=U5QPSY5M32O3ZGTK4YNIQNLEXHF72OPVCXWPRCDTFEIPWKGVNV2A, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:11.133534367 (547h3m11.004617412s) [upload=479693]: Next returned nil segment Dec 28 10:32:11.133569233 (547h3m11.004652278s) [upload=479693]: Waiting for error group managing segments... Dec 28 10:32:11.421476958 (547h3m11.292559993s) [noise=true , address=145.79.227.3:4149 , piece_id=GUIDXMU3Q5RTZTK2IOIJ2CXIMQ5HQGNDSEN5WVVZMRR336IM5FGQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.423163530 (547h3m11.294246575s) [noise=true , address=145.79.227.14:4840 , piece_id=6FSN47UNI7MQJBWBIV5THQD4WUXF52NM7M6BP7VHRB4J6E2F4VKQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.428115299 (547h3m11.299198334s) [noise=true , address=145.79.227.41:4086 , piece_id=OTV62ELETLJDI25ANYVI5TOAQKAMO5TRHWSGFLQSHBBRGBO3PLVQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.436758778 (547h3m11.307841823s) [noise=true , address=145.79.227.10:4403 , piece_id=FECST4PG6RGH6TUCOJZCSIOVRESXJMABM235ZOO5M4GYZTY75LTQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.439394376 (547h3m11.310477411s) [noise=true , address=145.79.227.46:4603 , piece_id=HUJL4BAY7U2G57TOMZVHQWTFOTN2RV2XKDK2MIUILVD6HR6Y27NQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.439861225 (547h3m11.31094427s) [noise=true , address=145.79.227.47:4265 , piece_id=TDVXCOFNRIOSRS2OETJPUM3EELE4YN73KLHWDSPL6XPX5TACZCWQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.442937283 (547h3m11.314020318s) [noise=true , address=145.79.227.37:4567 , piece_id=ZY7KVVWYGKO6X2OONS7LXBBXDYKLSNIGL33U72HP6INZOUDUIS5A, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.444228622 (547h3m11.315311667s) [noise=true , address=145.79.227.5:4223 , piece_id=2DUTU22HULS2ERHZZ2TYB2YTL5ZUUAL5Z65V5VSNDKAUNWOYLFWQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.452679474 (547h3m11.323762519s) [noise=true , address=145.79.227.2:4113 , piece_id=7VXXE267VCB2OXILXW6TV4AJGNHW4LQVXEA4HOSVNMI6UAEMS3VA, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.457542493 (547h3m11.328625538s) [noise=true , address=145.79.227.29:4941 , piece_id=U5QPSY5M32O3ZGTK4YNIQNLEXHF72OPVCXWPRCDTFEIPWKGVNV2A, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.708367176 (547h3m11.579450221s) [noise=true , address=45.91.116.13:4154 , piece_id=4VHHRCQCEPZUWD2QT6V62WVKBRXMFY7NYAQCAF4DVB4OFCRSFGUA, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.716026030 (547h3m11.587109075s) [noise=true , address=45.91.116.14:4192 , piece_id=UMHHU3Y6JA7GBPWMG3NZRBNJ6QSP762N3LZUH4HEOJXEQBO5NP4A, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:12.434215195 (547h3m12.30529824s) [noise=true , address=45.91.116.6:4975 , piece_id=JRQ4W24BKCX2ELV7C6P3QXHLF4BWUO7FEI7CMDMV43YYTK7FB3HQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:12.434324403 (547h3m12.305407438s) [seg_pos={0 0}, upload=479693]: Segment reached optimal threshold of 13 pieces. Dec 28 10:32:12.434515456 (547h3m12.305598501s) [noise=true , address=77.90.31.21:4877 , piece_id=3ELH72FJ2GAD7CFLEMWFNKERUTYJFB6FE2V7S7YGOONGYU6WPBNQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Dec 28 10:32:12.434521548 (547h3m12.305604593s) [noise=true , address=77.90.31.21:4877 , piece_id=3ELH72FJ2GAD7CFLEMWFNKERUTYJFB6FE2V7S7YGOONGYU6WPBNQ, seg_pos={0 0}, upload=479693]: Long tail cancellation due to optimal threshold Dec 28 10:32:12.434710859 (547h3m12.305793894s) [noise=true , address=77.90.31.15:4700 , piece_id=CTHN5F73XF6BQ2ATYAUFBRGTD6YWEOK3U7OKAFXDMDLJ7DUZPGYQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 28 10:32:12.434716900 (547h3m12.305799945s) [noise=true , address=77.90.31.15:4700 , piece_id=CTHN5F73XF6BQ2ATYAUFBRGTD6YWEOK3U7OKAFXDMDLJ7DUZPGYQ, seg_pos={0 0}, upload=479693]: Long tail cancellation due to optimal threshold Dec 28 10:32:12.434797904 (547h3m12.305880949s) [noise=true , address=77.90.31.18:4808 , piece_id=UZP6F2ITW5Q3AFHKITHSUTPKZZVK2ZBLFF5DHH5UA5URTSPCP7JA, seg_pos={0 0}, upload=479693]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF Dec 28 10:32:12.434813053 (547h3m12.305896088s) [noise=true , address=77.90.31.18:4808 , piece_id=UZP6F2ITW5Q3AFHKITHSUTPKZZVK2ZBLFF5DHH5UA5URTSPCP7JA, seg_pos={0 0}, upload=479693]: Long tail cancellation due to optimal threshold Dec 28 10:32:12.434845295 (547h3m12.30592833s) [seg_pos={0 0}, upload=479693]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 10:32:12.434957198 (547h3m12.306040233s) [upload=479693]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 10:32:12.434962488 (547h3m12.306045523s) [upload=479693]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 10:32:12.725143372 (547h3m12.596226407s) [upload=479693]: Done uploading segments... Dec 28 10:32:12.725163410 (547h3m12.596246445s) [upload=479693]: Upload finished. err: <nil> Duration: 6.13904464s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [X...X........X.....X..................X..............X.X....] duration: 3.557512646s trace: https://jaeger.dev.storj.io/trace/57df8e96b91a6205 FAIL (2025-12-28 10:36:19 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.557531382s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [........X...................X..XX.....X..X............XX....] duration: 5.94265528s trace: https://jaeger.dev.storj.io/trace/12fdeb4fc6be55f2 FAIL (2025-12-28 10:35:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: JRQ4W24BKCX2ELV7C6P3QXHLF4BWUO7FEI7CMDMV43YYTK7FB3HQ) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: UMHHU3Y6JA7GBPWMG3NZRBNJ6QSP762N3LZUH4HEOJXEQBO5NP4A) context deadline exceeded ======LOGS====== Dec 28 10:32:09.171162516 (547h3m9.042245551s) [upload=479693]: Starting upload Dec 28 10:32:09.171211930 (547h3m9.042294965s) [upload=479693]: Done starting upload Dec 28 10:32:09.171389138 (547h3m9.042472173s) [upload=479693]: Uploading segments... Dec 28 10:32:09.171420808 (547h3m9.042503843s) [upload=479693]: Got next segment. Inline: false Dec 28 10:32:09.171446587 (547h3m9.042529622s) [upload=479693]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 10:32:09.171451076 (547h3m9.042534121s) [upload=479693]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 10:32:09.415778928 (547h3m9.286861973s) [seg_pos={0 0}, upload=479693]: Begin upload segment... Dec 28 10:32:09.415823763 (547h3m9.286906798s) [seg_pos={0 0}, upload=479693]: Done begin upload segment. Dec 28 10:32:09.415852628 (547h3m9.286935673s) [noise=true , address=145.79.227.5:4223 , piece_id=2DUTU22HULS2ERHZZ2TYB2YTL5ZUUAL5Z65V5VSNDKAUNWOYLFWQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.415915609 (547h3m9.286998644s) [noise=true , address=45.91.116.14:4192 , piece_id=UMHHU3Y6JA7GBPWMG3NZRBNJ6QSP762N3LZUH4HEOJXEQBO5NP4A, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.416427983 (547h3m9.287511019s) [noise=true , address=145.79.227.41:4086 , piece_id=OTV62ELETLJDI25ANYVI5TOAQKAMO5TRHWSGFLQSHBBRGBO3PLVQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.416839006 (547h3m9.287922041s) [noise=true , address=145.79.227.14:4840 , piece_id=6FSN47UNI7MQJBWBIV5THQD4WUXF52NM7M6BP7VHRB4J6E2F4VKQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.416935741 (547h3m9.288018786s) [noise=true , address=77.90.31.18:4808 , piece_id=UZP6F2ITW5Q3AFHKITHSUTPKZZVK2ZBLFF5DHH5UA5URTSPCP7JA, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.417242896 (547h3m9.288325941s) [noise=true , address=77.90.31.21:4877 , piece_id=3ELH72FJ2GAD7CFLEMWFNKERUTYJFB6FE2V7S7YGOONGYU6WPBNQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.417527407 (547h3m9.288610442s) [noise=true , address=45.91.116.13:4154 , piece_id=4VHHRCQCEPZUWD2QT6V62WVKBRXMFY7NYAQCAF4DVB4OFCRSFGUA, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.417826456 (547h3m9.288909491s) [noise=true , address=145.79.227.3:4149 , piece_id=GUIDXMU3Q5RTZTK2IOIJ2CXIMQ5HQGNDSEN5WVVZMRR336IM5FGQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.417930104 (547h3m9.289013149s) [noise=true , address=145.79.227.2:4113 , piece_id=7VXXE267VCB2OXILXW6TV4AJGNHW4LQVXEA4HOSVNMI6UAEMS3VA, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.418096461 (547h3m9.289179506s) [noise=true , address=145.79.227.47:4265 , piece_id=TDVXCOFNRIOSRS2OETJPUM3EELE4YN73KLHWDSPL6XPX5TACZCWQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.418451897 (547h3m9.289534942s) [noise=true , address=145.79.227.37:4567 , piece_id=ZY7KVVWYGKO6X2OONS7LXBBXDYKLSNIGL33U72HP6INZOUDUIS5A, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.419025139 (547h3m9.290108184s) [noise=true , address=77.90.31.15:4700 , piece_id=CTHN5F73XF6BQ2ATYAUFBRGTD6YWEOK3U7OKAFXDMDLJ7DUZPGYQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.419682581 (547h3m9.290765626s) [noise=true , address=45.91.116.6:4975 , piece_id=JRQ4W24BKCX2ELV7C6P3QXHLF4BWUO7FEI7CMDMV43YYTK7FB3HQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.420365671 (547h3m9.291448706s) [noise=true , address=145.79.227.46:4603 , piece_id=HUJL4BAY7U2G57TOMZVHQWTFOTN2RV2XKDK2MIUILVD6HR6Y27NQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.421011522 (547h3m9.292094557s) [noise=true , address=145.79.227.10:4403 , piece_id=FECST4PG6RGH6TUCOJZCSIOVRESXJMABM235ZOO5M4GYZTY75LTQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.421697077 (547h3m9.292780122s) [noise=true , address=145.79.227.29:4941 , piece_id=U5QPSY5M32O3ZGTK4YNIQNLEXHF72OPVCXWPRCDTFEIPWKGVNV2A, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:11.133534367 (547h3m11.004617412s) [upload=479693]: Next returned nil segment Dec 28 10:32:11.133569233 (547h3m11.004652278s) [upload=479693]: Waiting for error group managing segments... Dec 28 10:32:11.421476958 (547h3m11.292559993s) [noise=true , address=145.79.227.3:4149 , piece_id=GUIDXMU3Q5RTZTK2IOIJ2CXIMQ5HQGNDSEN5WVVZMRR336IM5FGQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.423163530 (547h3m11.294246575s) [noise=true , address=145.79.227.14:4840 , piece_id=6FSN47UNI7MQJBWBIV5THQD4WUXF52NM7M6BP7VHRB4J6E2F4VKQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.428115299 (547h3m11.299198334s) [noise=true , address=145.79.227.41:4086 , piece_id=OTV62ELETLJDI25ANYVI5TOAQKAMO5TRHWSGFLQSHBBRGBO3PLVQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.436758778 (547h3m11.307841823s) [noise=true , address=145.79.227.10:4403 , piece_id=FECST4PG6RGH6TUCOJZCSIOVRESXJMABM235ZOO5M4GYZTY75LTQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.439394376 (547h3m11.310477411s) [noise=true , address=145.79.227.46:4603 , piece_id=HUJL4BAY7U2G57TOMZVHQWTFOTN2RV2XKDK2MIUILVD6HR6Y27NQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.439861225 (547h3m11.31094427s) [noise=true , address=145.79.227.47:4265 , piece_id=TDVXCOFNRIOSRS2OETJPUM3EELE4YN73KLHWDSPL6XPX5TACZCWQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.442937283 (547h3m11.314020318s) [noise=true , address=145.79.227.37:4567 , piece_id=ZY7KVVWYGKO6X2OONS7LXBBXDYKLSNIGL33U72HP6INZOUDUIS5A, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.444228622 (547h3m11.315311667s) [noise=true , address=145.79.227.5:4223 , piece_id=2DUTU22HULS2ERHZZ2TYB2YTL5ZUUAL5Z65V5VSNDKAUNWOYLFWQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.452679474 (547h3m11.323762519s) [noise=true , address=145.79.227.2:4113 , piece_id=7VXXE267VCB2OXILXW6TV4AJGNHW4LQVXEA4HOSVNMI6UAEMS3VA, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.457542493 (547h3m11.328625538s) [noise=true , address=145.79.227.29:4941 , piece_id=U5QPSY5M32O3ZGTK4YNIQNLEXHF72OPVCXWPRCDTFEIPWKGVNV2A, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.708367176 (547h3m11.579450221s) [noise=true , address=45.91.116.13:4154 , piece_id=4VHHRCQCEPZUWD2QT6V62WVKBRXMFY7NYAQCAF4DVB4OFCRSFGUA, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.716026030 (547h3m11.587109075s) [noise=true , address=45.91.116.14:4192 , piece_id=UMHHU3Y6JA7GBPWMG3NZRBNJ6QSP762N3LZUH4HEOJXEQBO5NP4A, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:12.434215195 (547h3m12.30529824s) [noise=true , address=45.91.116.6:4975 , piece_id=JRQ4W24BKCX2ELV7C6P3QXHLF4BWUO7FEI7CMDMV43YYTK7FB3HQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:12.434324403 (547h3m12.305407438s) [seg_pos={0 0}, upload=479693]: Segment reached optimal threshold of 13 pieces. Dec 28 10:32:12.434515456 (547h3m12.305598501s) [noise=true , address=77.90.31.21:4877 , piece_id=3ELH72FJ2GAD7CFLEMWFNKERUTYJFB6FE2V7S7YGOONGYU6WPBNQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Dec 28 10:32:12.434521548 (547h3m12.305604593s) [noise=true , address=77.90.31.21:4877 , piece_id=3ELH72FJ2GAD7CFLEMWFNKERUTYJFB6FE2V7S7YGOONGYU6WPBNQ, seg_pos={0 0}, upload=479693]: Long tail cancellation due to optimal threshold Dec 28 10:32:12.434710859 (547h3m12.305793894s) [noise=true , address=77.90.31.15:4700 , piece_id=CTHN5F73XF6BQ2ATYAUFBRGTD6YWEOK3U7OKAFXDMDLJ7DUZPGYQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 28 10:32:12.434716900 (547h3m12.305799945s) [noise=true , address=77.90.31.15:4700 , piece_id=CTHN5F73XF6BQ2ATYAUFBRGTD6YWEOK3U7OKAFXDMDLJ7DUZPGYQ, seg_pos={0 0}, upload=479693]: Long tail cancellation due to optimal threshold Dec 28 10:32:12.434797904 (547h3m12.305880949s) [noise=true , address=77.90.31.18:4808 , piece_id=UZP6F2ITW5Q3AFHKITHSUTPKZZVK2ZBLFF5DHH5UA5URTSPCP7JA, seg_pos={0 0}, upload=479693]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF Dec 28 10:32:12.434813053 (547h3m12.305896088s) [noise=true , address=77.90.31.18:4808 , piece_id=UZP6F2ITW5Q3AFHKITHSUTPKZZVK2ZBLFF5DHH5UA5URTSPCP7JA, seg_pos={0 0}, upload=479693]: Long tail cancellation due to optimal threshold Dec 28 10:32:12.434845295 (547h3m12.30592833s) [seg_pos={0 0}, upload=479693]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 10:32:12.434957198 (547h3m12.306040233s) [upload=479693]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 10:32:12.434962488 (547h3m12.306045523s) [upload=479693]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 10:32:12.725143372 (547h3m12.596226407s) [upload=479693]: Done uploading segments... Dec 28 10:32:12.725163410 (547h3m12.596246445s) [upload=479693]: Upload finished. err: <nil> Duration: 5.942685869s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [XX...X........X.....X..................X..............X.X...] duration: 3.552244204s trace: https://jaeger.dev.storj.io/trace/57df8e96b91a6205 FAIL (2025-12-28 10:35:27 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.552334595s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [XXX...X........X.....X..................X..............X.X..] duration: 3.515278259s trace: https://jaeger.dev.storj.io/trace/57df8e96b91a6205 FAIL (2025-12-28 10:34:36 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.515295832s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.........X...................X..XX.....X..X............XX...] duration: 10.016981018s trace: https://jaeger.dev.storj.io/trace/12fdeb4fc6be55f2 FAIL (2025-12-28 10:34:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: JRQ4W24BKCX2ELV7C6P3QXHLF4BWUO7FEI7CMDMV43YYTK7FB3HQ) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: UMHHU3Y6JA7GBPWMG3NZRBNJ6QSP762N3LZUH4HEOJXEQBO5NP4A) context deadline exceeded ======LOGS====== Dec 28 10:32:09.171162516 (547h3m9.042245551s) [upload=479693]: Starting upload Dec 28 10:32:09.171211930 (547h3m9.042294965s) [upload=479693]: Done starting upload Dec 28 10:32:09.171389138 (547h3m9.042472173s) [upload=479693]: Uploading segments... Dec 28 10:32:09.171420808 (547h3m9.042503843s) [upload=479693]: Got next segment. Inline: false Dec 28 10:32:09.171446587 (547h3m9.042529622s) [upload=479693]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 10:32:09.171451076 (547h3m9.042534121s) [upload=479693]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 10:32:09.415778928 (547h3m9.286861973s) [seg_pos={0 0}, upload=479693]: Begin upload segment... Dec 28 10:32:09.415823763 (547h3m9.286906798s) [seg_pos={0 0}, upload=479693]: Done begin upload segment. Dec 28 10:32:09.415852628 (547h3m9.286935673s) [noise=true , address=145.79.227.5:4223 , piece_id=2DUTU22HULS2ERHZZ2TYB2YTL5ZUUAL5Z65V5VSNDKAUNWOYLFWQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.415915609 (547h3m9.286998644s) [noise=true , address=45.91.116.14:4192 , piece_id=UMHHU3Y6JA7GBPWMG3NZRBNJ6QSP762N3LZUH4HEOJXEQBO5NP4A, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.416427983 (547h3m9.287511019s) [noise=true , address=145.79.227.41:4086 , piece_id=OTV62ELETLJDI25ANYVI5TOAQKAMO5TRHWSGFLQSHBBRGBO3PLVQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.416839006 (547h3m9.287922041s) [noise=true , address=145.79.227.14:4840 , piece_id=6FSN47UNI7MQJBWBIV5THQD4WUXF52NM7M6BP7VHRB4J6E2F4VKQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.416935741 (547h3m9.288018786s) [noise=true , address=77.90.31.18:4808 , piece_id=UZP6F2ITW5Q3AFHKITHSUTPKZZVK2ZBLFF5DHH5UA5URTSPCP7JA, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.417242896 (547h3m9.288325941s) [noise=true , address=77.90.31.21:4877 , piece_id=3ELH72FJ2GAD7CFLEMWFNKERUTYJFB6FE2V7S7YGOONGYU6WPBNQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.417527407 (547h3m9.288610442s) [noise=true , address=45.91.116.13:4154 , piece_id=4VHHRCQCEPZUWD2QT6V62WVKBRXMFY7NYAQCAF4DVB4OFCRSFGUA, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.417826456 (547h3m9.288909491s) [noise=true , address=145.79.227.3:4149 , piece_id=GUIDXMU3Q5RTZTK2IOIJ2CXIMQ5HQGNDSEN5WVVZMRR336IM5FGQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.417930104 (547h3m9.289013149s) [noise=true , address=145.79.227.2:4113 , piece_id=7VXXE267VCB2OXILXW6TV4AJGNHW4LQVXEA4HOSVNMI6UAEMS3VA, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.418096461 (547h3m9.289179506s) [noise=true , address=145.79.227.47:4265 , piece_id=TDVXCOFNRIOSRS2OETJPUM3EELE4YN73KLHWDSPL6XPX5TACZCWQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.418451897 (547h3m9.289534942s) [noise=true , address=145.79.227.37:4567 , piece_id=ZY7KVVWYGKO6X2OONS7LXBBXDYKLSNIGL33U72HP6INZOUDUIS5A, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.419025139 (547h3m9.290108184s) [noise=true , address=77.90.31.15:4700 , piece_id=CTHN5F73XF6BQ2ATYAUFBRGTD6YWEOK3U7OKAFXDMDLJ7DUZPGYQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.419682581 (547h3m9.290765626s) [noise=true , address=45.91.116.6:4975 , piece_id=JRQ4W24BKCX2ELV7C6P3QXHLF4BWUO7FEI7CMDMV43YYTK7FB3HQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.420365671 (547h3m9.291448706s) [noise=true , address=145.79.227.46:4603 , piece_id=HUJL4BAY7U2G57TOMZVHQWTFOTN2RV2XKDK2MIUILVD6HR6Y27NQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.421011522 (547h3m9.292094557s) [noise=true , address=145.79.227.10:4403 , piece_id=FECST4PG6RGH6TUCOJZCSIOVRESXJMABM235ZOO5M4GYZTY75LTQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.421697077 (547h3m9.292780122s) [noise=true , address=145.79.227.29:4941 , piece_id=U5QPSY5M32O3ZGTK4YNIQNLEXHF72OPVCXWPRCDTFEIPWKGVNV2A, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:11.133534367 (547h3m11.004617412s) [upload=479693]: Next returned nil segment Dec 28 10:32:11.133569233 (547h3m11.004652278s) [upload=479693]: Waiting for error group managing segments... Dec 28 10:32:11.421476958 (547h3m11.292559993s) [noise=true , address=145.79.227.3:4149 , piece_id=GUIDXMU3Q5RTZTK2IOIJ2CXIMQ5HQGNDSEN5WVVZMRR336IM5FGQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.423163530 (547h3m11.294246575s) [noise=true , address=145.79.227.14:4840 , piece_id=6FSN47UNI7MQJBWBIV5THQD4WUXF52NM7M6BP7VHRB4J6E2F4VKQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.428115299 (547h3m11.299198334s) [noise=true , address=145.79.227.41:4086 , piece_id=OTV62ELETLJDI25ANYVI5TOAQKAMO5TRHWSGFLQSHBBRGBO3PLVQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.436758778 (547h3m11.307841823s) [noise=true , address=145.79.227.10:4403 , piece_id=FECST4PG6RGH6TUCOJZCSIOVRESXJMABM235ZOO5M4GYZTY75LTQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.439394376 (547h3m11.310477411s) [noise=true , address=145.79.227.46:4603 , piece_id=HUJL4BAY7U2G57TOMZVHQWTFOTN2RV2XKDK2MIUILVD6HR6Y27NQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.439861225 (547h3m11.31094427s) [noise=true , address=145.79.227.47:4265 , piece_id=TDVXCOFNRIOSRS2OETJPUM3EELE4YN73KLHWDSPL6XPX5TACZCWQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.442937283 (547h3m11.314020318s) [noise=true , address=145.79.227.37:4567 , piece_id=ZY7KVVWYGKO6X2OONS7LXBBXDYKLSNIGL33U72HP6INZOUDUIS5A, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.444228622 (547h3m11.315311667s) [noise=true , address=145.79.227.5:4223 , piece_id=2DUTU22HULS2ERHZZ2TYB2YTL5ZUUAL5Z65V5VSNDKAUNWOYLFWQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.452679474 (547h3m11.323762519s) [noise=true , address=145.79.227.2:4113 , piece_id=7VXXE267VCB2OXILXW6TV4AJGNHW4LQVXEA4HOSVNMI6UAEMS3VA, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.457542493 (547h3m11.328625538s) [noise=true , address=145.79.227.29:4941 , piece_id=U5QPSY5M32O3ZGTK4YNIQNLEXHF72OPVCXWPRCDTFEIPWKGVNV2A, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.708367176 (547h3m11.579450221s) [noise=true , address=45.91.116.13:4154 , piece_id=4VHHRCQCEPZUWD2QT6V62WVKBRXMFY7NYAQCAF4DVB4OFCRSFGUA, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.716026030 (547h3m11.587109075s) [noise=true , address=45.91.116.14:4192 , piece_id=UMHHU3Y6JA7GBPWMG3NZRBNJ6QSP762N3LZUH4HEOJXEQBO5NP4A, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:12.434215195 (547h3m12.30529824s) [noise=true , address=45.91.116.6:4975 , piece_id=JRQ4W24BKCX2ELV7C6P3QXHLF4BWUO7FEI7CMDMV43YYTK7FB3HQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:12.434324403 (547h3m12.305407438s) [seg_pos={0 0}, upload=479693]: Segment reached optimal threshold of 13 pieces. Dec 28 10:32:12.434515456 (547h3m12.305598501s) [noise=true , address=77.90.31.21:4877 , piece_id=3ELH72FJ2GAD7CFLEMWFNKERUTYJFB6FE2V7S7YGOONGYU6WPBNQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Dec 28 10:32:12.434521548 (547h3m12.305604593s) [noise=true , address=77.90.31.21:4877 , piece_id=3ELH72FJ2GAD7CFLEMWFNKERUTYJFB6FE2V7S7YGOONGYU6WPBNQ, seg_pos={0 0}, upload=479693]: Long tail cancellation due to optimal threshold Dec 28 10:32:12.434710859 (547h3m12.305793894s) [noise=true , address=77.90.31.15:4700 , piece_id=CTHN5F73XF6BQ2ATYAUFBRGTD6YWEOK3U7OKAFXDMDLJ7DUZPGYQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 28 10:32:12.434716900 (547h3m12.305799945s) [noise=true , address=77.90.31.15:4700 , piece_id=CTHN5F73XF6BQ2ATYAUFBRGTD6YWEOK3U7OKAFXDMDLJ7DUZPGYQ, seg_pos={0 0}, upload=479693]: Long tail cancellation due to optimal threshold Dec 28 10:32:12.434797904 (547h3m12.305880949s) [noise=true , address=77.90.31.18:4808 , piece_id=UZP6F2ITW5Q3AFHKITHSUTPKZZVK2ZBLFF5DHH5UA5URTSPCP7JA, seg_pos={0 0}, upload=479693]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF Dec 28 10:32:12.434813053 (547h3m12.305896088s) [noise=true , address=77.90.31.18:4808 , piece_id=UZP6F2ITW5Q3AFHKITHSUTPKZZVK2ZBLFF5DHH5UA5URTSPCP7JA, seg_pos={0 0}, upload=479693]: Long tail cancellation due to optimal threshold Dec 28 10:32:12.434845295 (547h3m12.30592833s) [seg_pos={0 0}, upload=479693]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 10:32:12.434957198 (547h3m12.306040233s) [upload=479693]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 10:32:12.434962488 (547h3m12.306045523s) [upload=479693]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 10:32:12.725143372 (547h3m12.596226407s) [upload=479693]: Done uploading segments... Dec 28 10:32:12.725163410 (547h3m12.596246445s) [upload=479693]: Upload finished. err: <nil> Duration: 10.017016938s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [XXXX...X........X.....X..................X..............X.X.] duration: 3.57167581s trace: https://jaeger.dev.storj.io/trace/57df8e96b91a6205 FAIL (2025-12-28 10:34:07 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.571705587s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [XXXXX...X........X.....X..................X..............X.X] duration: 5.000980009s trace: https://jaeger.dev.storj.io/trace/57df8e96b91a6205 FAIL (2025-12-28 10:33:32 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001017349s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [X.........X...................X..XX.....X..X............XX..] duration: 5.79354645s trace: https://jaeger.dev.storj.io/trace/12fdeb4fc6be55f2 FAIL (2025-12-28 10:33:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: JRQ4W24BKCX2ELV7C6P3QXHLF4BWUO7FEI7CMDMV43YYTK7FB3HQ) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: UMHHU3Y6JA7GBPWMG3NZRBNJ6QSP762N3LZUH4HEOJXEQBO5NP4A) context deadline exceeded ======LOGS====== Dec 28 10:32:09.171162516 (547h3m9.042245551s) [upload=479693]: Starting upload Dec 28 10:32:09.171211930 (547h3m9.042294965s) [upload=479693]: Done starting upload Dec 28 10:32:09.171389138 (547h3m9.042472173s) [upload=479693]: Uploading segments... Dec 28 10:32:09.171420808 (547h3m9.042503843s) [upload=479693]: Got next segment. Inline: false Dec 28 10:32:09.171446587 (547h3m9.042529622s) [upload=479693]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 10:32:09.171451076 (547h3m9.042534121s) [upload=479693]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 10:32:09.415778928 (547h3m9.286861973s) [seg_pos={0 0}, upload=479693]: Begin upload segment... Dec 28 10:32:09.415823763 (547h3m9.286906798s) [seg_pos={0 0}, upload=479693]: Done begin upload segment. Dec 28 10:32:09.415852628 (547h3m9.286935673s) [noise=true , address=145.79.227.5:4223 , piece_id=2DUTU22HULS2ERHZZ2TYB2YTL5ZUUAL5Z65V5VSNDKAUNWOYLFWQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.415915609 (547h3m9.286998644s) [noise=true , address=45.91.116.14:4192 , piece_id=UMHHU3Y6JA7GBPWMG3NZRBNJ6QSP762N3LZUH4HEOJXEQBO5NP4A, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.416427983 (547h3m9.287511019s) [noise=true , address=145.79.227.41:4086 , piece_id=OTV62ELETLJDI25ANYVI5TOAQKAMO5TRHWSGFLQSHBBRGBO3PLVQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.416839006 (547h3m9.287922041s) [noise=true , address=145.79.227.14:4840 , piece_id=6FSN47UNI7MQJBWBIV5THQD4WUXF52NM7M6BP7VHRB4J6E2F4VKQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.416935741 (547h3m9.288018786s) [noise=true , address=77.90.31.18:4808 , piece_id=UZP6F2ITW5Q3AFHKITHSUTPKZZVK2ZBLFF5DHH5UA5URTSPCP7JA, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.417242896 (547h3m9.288325941s) [noise=true , address=77.90.31.21:4877 , piece_id=3ELH72FJ2GAD7CFLEMWFNKERUTYJFB6FE2V7S7YGOONGYU6WPBNQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.417527407 (547h3m9.288610442s) [noise=true , address=45.91.116.13:4154 , piece_id=4VHHRCQCEPZUWD2QT6V62WVKBRXMFY7NYAQCAF4DVB4OFCRSFGUA, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.417826456 (547h3m9.288909491s) [noise=true , address=145.79.227.3:4149 , piece_id=GUIDXMU3Q5RTZTK2IOIJ2CXIMQ5HQGNDSEN5WVVZMRR336IM5FGQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.417930104 (547h3m9.289013149s) [noise=true , address=145.79.227.2:4113 , piece_id=7VXXE267VCB2OXILXW6TV4AJGNHW4LQVXEA4HOSVNMI6UAEMS3VA, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.418096461 (547h3m9.289179506s) [noise=true , address=145.79.227.47:4265 , piece_id=TDVXCOFNRIOSRS2OETJPUM3EELE4YN73KLHWDSPL6XPX5TACZCWQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.418451897 (547h3m9.289534942s) [noise=true , address=145.79.227.37:4567 , piece_id=ZY7KVVWYGKO6X2OONS7LXBBXDYKLSNIGL33U72HP6INZOUDUIS5A, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.419025139 (547h3m9.290108184s) [noise=true , address=77.90.31.15:4700 , piece_id=CTHN5F73XF6BQ2ATYAUFBRGTD6YWEOK3U7OKAFXDMDLJ7DUZPGYQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.419682581 (547h3m9.290765626s) [noise=true , address=45.91.116.6:4975 , piece_id=JRQ4W24BKCX2ELV7C6P3QXHLF4BWUO7FEI7CMDMV43YYTK7FB3HQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.420365671 (547h3m9.291448706s) [noise=true , address=145.79.227.46:4603 , piece_id=HUJL4BAY7U2G57TOMZVHQWTFOTN2RV2XKDK2MIUILVD6HR6Y27NQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.421011522 (547h3m9.292094557s) [noise=true , address=145.79.227.10:4403 , piece_id=FECST4PG6RGH6TUCOJZCSIOVRESXJMABM235ZOO5M4GYZTY75LTQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.421697077 (547h3m9.292780122s) [noise=true , address=145.79.227.29:4941 , piece_id=U5QPSY5M32O3ZGTK4YNIQNLEXHF72OPVCXWPRCDTFEIPWKGVNV2A, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:11.133534367 (547h3m11.004617412s) [upload=479693]: Next returned nil segment Dec 28 10:32:11.133569233 (547h3m11.004652278s) [upload=479693]: Waiting for error group managing segments... Dec 28 10:32:11.421476958 (547h3m11.292559993s) [noise=true , address=145.79.227.3:4149 , piece_id=GUIDXMU3Q5RTZTK2IOIJ2CXIMQ5HQGNDSEN5WVVZMRR336IM5FGQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.423163530 (547h3m11.294246575s) [noise=true , address=145.79.227.14:4840 , piece_id=6FSN47UNI7MQJBWBIV5THQD4WUXF52NM7M6BP7VHRB4J6E2F4VKQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.428115299 (547h3m11.299198334s) [noise=true , address=145.79.227.41:4086 , piece_id=OTV62ELETLJDI25ANYVI5TOAQKAMO5TRHWSGFLQSHBBRGBO3PLVQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.436758778 (547h3m11.307841823s) [noise=true , address=145.79.227.10:4403 , piece_id=FECST4PG6RGH6TUCOJZCSIOVRESXJMABM235ZOO5M4GYZTY75LTQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.439394376 (547h3m11.310477411s) [noise=true , address=145.79.227.46:4603 , piece_id=HUJL4BAY7U2G57TOMZVHQWTFOTN2RV2XKDK2MIUILVD6HR6Y27NQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.439861225 (547h3m11.31094427s) [noise=true , address=145.79.227.47:4265 , piece_id=TDVXCOFNRIOSRS2OETJPUM3EELE4YN73KLHWDSPL6XPX5TACZCWQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.442937283 (547h3m11.314020318s) [noise=true , address=145.79.227.37:4567 , piece_id=ZY7KVVWYGKO6X2OONS7LXBBXDYKLSNIGL33U72HP6INZOUDUIS5A, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.444228622 (547h3m11.315311667s) [noise=true , address=145.79.227.5:4223 , piece_id=2DUTU22HULS2ERHZZ2TYB2YTL5ZUUAL5Z65V5VSNDKAUNWOYLFWQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.452679474 (547h3m11.323762519s) [noise=true , address=145.79.227.2:4113 , piece_id=7VXXE267VCB2OXILXW6TV4AJGNHW4LQVXEA4HOSVNMI6UAEMS3VA, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.457542493 (547h3m11.328625538s) [noise=true , address=145.79.227.29:4941 , piece_id=U5QPSY5M32O3ZGTK4YNIQNLEXHF72OPVCXWPRCDTFEIPWKGVNV2A, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.708367176 (547h3m11.579450221s) [noise=true , address=45.91.116.13:4154 , piece_id=4VHHRCQCEPZUWD2QT6V62WVKBRXMFY7NYAQCAF4DVB4OFCRSFGUA, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.716026030 (547h3m11.587109075s) [noise=true , address=45.91.116.14:4192 , piece_id=UMHHU3Y6JA7GBPWMG3NZRBNJ6QSP762N3LZUH4HEOJXEQBO5NP4A, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:12.434215195 (547h3m12.30529824s) [noise=true , address=45.91.116.6:4975 , piece_id=JRQ4W24BKCX2ELV7C6P3QXHLF4BWUO7FEI7CMDMV43YYTK7FB3HQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:12.434324403 (547h3m12.305407438s) [seg_pos={0 0}, upload=479693]: Segment reached optimal threshold of 13 pieces. Dec 28 10:32:12.434515456 (547h3m12.305598501s) [noise=true , address=77.90.31.21:4877 , piece_id=3ELH72FJ2GAD7CFLEMWFNKERUTYJFB6FE2V7S7YGOONGYU6WPBNQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Dec 28 10:32:12.434521548 (547h3m12.305604593s) [noise=true , address=77.90.31.21:4877 , piece_id=3ELH72FJ2GAD7CFLEMWFNKERUTYJFB6FE2V7S7YGOONGYU6WPBNQ, seg_pos={0 0}, upload=479693]: Long tail cancellation due to optimal threshold Dec 28 10:32:12.434710859 (547h3m12.305793894s) [noise=true , address=77.90.31.15:4700 , piece_id=CTHN5F73XF6BQ2ATYAUFBRGTD6YWEOK3U7OKAFXDMDLJ7DUZPGYQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 28 10:32:12.434716900 (547h3m12.305799945s) [noise=true , address=77.90.31.15:4700 , piece_id=CTHN5F73XF6BQ2ATYAUFBRGTD6YWEOK3U7OKAFXDMDLJ7DUZPGYQ, seg_pos={0 0}, upload=479693]: Long tail cancellation due to optimal threshold Dec 28 10:32:12.434797904 (547h3m12.305880949s) [noise=true , address=77.90.31.18:4808 , piece_id=UZP6F2ITW5Q3AFHKITHSUTPKZZVK2ZBLFF5DHH5UA5URTSPCP7JA, seg_pos={0 0}, upload=479693]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF Dec 28 10:32:12.434813053 (547h3m12.305896088s) [noise=true , address=77.90.31.18:4808 , piece_id=UZP6F2ITW5Q3AFHKITHSUTPKZZVK2ZBLFF5DHH5UA5URTSPCP7JA, seg_pos={0 0}, upload=479693]: Long tail cancellation due to optimal threshold Dec 28 10:32:12.434845295 (547h3m12.30592833s) [seg_pos={0 0}, upload=479693]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 10:32:12.434957198 (547h3m12.306040233s) [upload=479693]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 10:32:12.434962488 (547h3m12.306045523s) [upload=479693]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 10:32:12.725143372 (547h3m12.596226407s) [upload=479693]: Done uploading segments... Dec 28 10:32:12.725163410 (547h3m12.596246445s) [upload=479693]: Upload finished. err: <nil> Duration: 5.793599682s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [.XXXXX...X........X.....X..................X..............X.] duration: 3.453263893s trace: https://jaeger.dev.storj.io/trace/5ea2a8966c11b1a9 FAIL (2025-12-28 10:32:47 UTC) orbiter warning: context deadline exceeded Duration: 3.453288551s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.X.........X...................X..XX.....X..X............XX.] duration: 5.817214127s trace: https://jaeger.dev.storj.io/trace/12fdeb4fc6be55f2 FAIL (2025-12-28 10:32:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: JRQ4W24BKCX2ELV7C6P3QXHLF4BWUO7FEI7CMDMV43YYTK7FB3HQ) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: UMHHU3Y6JA7GBPWMG3NZRBNJ6QSP762N3LZUH4HEOJXEQBO5NP4A) context deadline exceeded ======LOGS====== Dec 28 10:32:09.171162516 (547h3m9.042245551s) [upload=479693]: Starting upload Dec 28 10:32:09.171211930 (547h3m9.042294965s) [upload=479693]: Done starting upload Dec 28 10:32:09.171389138 (547h3m9.042472173s) [upload=479693]: Uploading segments... Dec 28 10:32:09.171420808 (547h3m9.042503843s) [upload=479693]: Got next segment. Inline: false Dec 28 10:32:09.171446587 (547h3m9.042529622s) [upload=479693]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 10:32:09.171451076 (547h3m9.042534121s) [upload=479693]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 10:32:09.415778928 (547h3m9.286861973s) [seg_pos={0 0}, upload=479693]: Begin upload segment... Dec 28 10:32:09.415823763 (547h3m9.286906798s) [seg_pos={0 0}, upload=479693]: Done begin upload segment. Dec 28 10:32:09.415852628 (547h3m9.286935673s) [noise=true , address=145.79.227.5:4223 , piece_id=2DUTU22HULS2ERHZZ2TYB2YTL5ZUUAL5Z65V5VSNDKAUNWOYLFWQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.415915609 (547h3m9.286998644s) [noise=true , address=45.91.116.14:4192 , piece_id=UMHHU3Y6JA7GBPWMG3NZRBNJ6QSP762N3LZUH4HEOJXEQBO5NP4A, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.416427983 (547h3m9.287511019s) [noise=true , address=145.79.227.41:4086 , piece_id=OTV62ELETLJDI25ANYVI5TOAQKAMO5TRHWSGFLQSHBBRGBO3PLVQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.416839006 (547h3m9.287922041s) [noise=true , address=145.79.227.14:4840 , piece_id=6FSN47UNI7MQJBWBIV5THQD4WUXF52NM7M6BP7VHRB4J6E2F4VKQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.416935741 (547h3m9.288018786s) [noise=true , address=77.90.31.18:4808 , piece_id=UZP6F2ITW5Q3AFHKITHSUTPKZZVK2ZBLFF5DHH5UA5URTSPCP7JA, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.417242896 (547h3m9.288325941s) [noise=true , address=77.90.31.21:4877 , piece_id=3ELH72FJ2GAD7CFLEMWFNKERUTYJFB6FE2V7S7YGOONGYU6WPBNQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.417527407 (547h3m9.288610442s) [noise=true , address=45.91.116.13:4154 , piece_id=4VHHRCQCEPZUWD2QT6V62WVKBRXMFY7NYAQCAF4DVB4OFCRSFGUA, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.417826456 (547h3m9.288909491s) [noise=true , address=145.79.227.3:4149 , piece_id=GUIDXMU3Q5RTZTK2IOIJ2CXIMQ5HQGNDSEN5WVVZMRR336IM5FGQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.417930104 (547h3m9.289013149s) [noise=true , address=145.79.227.2:4113 , piece_id=7VXXE267VCB2OXILXW6TV4AJGNHW4LQVXEA4HOSVNMI6UAEMS3VA, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.418096461 (547h3m9.289179506s) [noise=true , address=145.79.227.47:4265 , piece_id=TDVXCOFNRIOSRS2OETJPUM3EELE4YN73KLHWDSPL6XPX5TACZCWQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.418451897 (547h3m9.289534942s) [noise=true , address=145.79.227.37:4567 , piece_id=ZY7KVVWYGKO6X2OONS7LXBBXDYKLSNIGL33U72HP6INZOUDUIS5A, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.419025139 (547h3m9.290108184s) [noise=true , address=77.90.31.15:4700 , piece_id=CTHN5F73XF6BQ2ATYAUFBRGTD6YWEOK3U7OKAFXDMDLJ7DUZPGYQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.419682581 (547h3m9.290765626s) [noise=true , address=45.91.116.6:4975 , piece_id=JRQ4W24BKCX2ELV7C6P3QXHLF4BWUO7FEI7CMDMV43YYTK7FB3HQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.420365671 (547h3m9.291448706s) [noise=true , address=145.79.227.46:4603 , piece_id=HUJL4BAY7U2G57TOMZVHQWTFOTN2RV2XKDK2MIUILVD6HR6Y27NQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.421011522 (547h3m9.292094557s) [noise=true , address=145.79.227.10:4403 , piece_id=FECST4PG6RGH6TUCOJZCSIOVRESXJMABM235ZOO5M4GYZTY75LTQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.421697077 (547h3m9.292780122s) [noise=true , address=145.79.227.29:4941 , piece_id=U5QPSY5M32O3ZGTK4YNIQNLEXHF72OPVCXWPRCDTFEIPWKGVNV2A, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:11.133534367 (547h3m11.004617412s) [upload=479693]: Next returned nil segment Dec 28 10:32:11.133569233 (547h3m11.004652278s) [upload=479693]: Waiting for error group managing segments... Dec 28 10:32:11.421476958 (547h3m11.292559993s) [noise=true , address=145.79.227.3:4149 , piece_id=GUIDXMU3Q5RTZTK2IOIJ2CXIMQ5HQGNDSEN5WVVZMRR336IM5FGQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.423163530 (547h3m11.294246575s) [noise=true , address=145.79.227.14:4840 , piece_id=6FSN47UNI7MQJBWBIV5THQD4WUXF52NM7M6BP7VHRB4J6E2F4VKQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.428115299 (547h3m11.299198334s) [noise=true , address=145.79.227.41:4086 , piece_id=OTV62ELETLJDI25ANYVI5TOAQKAMO5TRHWSGFLQSHBBRGBO3PLVQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.436758778 (547h3m11.307841823s) [noise=true , address=145.79.227.10:4403 , piece_id=FECST4PG6RGH6TUCOJZCSIOVRESXJMABM235ZOO5M4GYZTY75LTQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.439394376 (547h3m11.310477411s) [noise=true , address=145.79.227.46:4603 , piece_id=HUJL4BAY7U2G57TOMZVHQWTFOTN2RV2XKDK2MIUILVD6HR6Y27NQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.439861225 (547h3m11.31094427s) [noise=true , address=145.79.227.47:4265 , piece_id=TDVXCOFNRIOSRS2OETJPUM3EELE4YN73KLHWDSPL6XPX5TACZCWQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.442937283 (547h3m11.314020318s) [noise=true , address=145.79.227.37:4567 , piece_id=ZY7KVVWYGKO6X2OONS7LXBBXDYKLSNIGL33U72HP6INZOUDUIS5A, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.444228622 (547h3m11.315311667s) [noise=true , address=145.79.227.5:4223 , piece_id=2DUTU22HULS2ERHZZ2TYB2YTL5ZUUAL5Z65V5VSNDKAUNWOYLFWQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.452679474 (547h3m11.323762519s) [noise=true , address=145.79.227.2:4113 , piece_id=7VXXE267VCB2OXILXW6TV4AJGNHW4LQVXEA4HOSVNMI6UAEMS3VA, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.457542493 (547h3m11.328625538s) [noise=true , address=145.79.227.29:4941 , piece_id=U5QPSY5M32O3ZGTK4YNIQNLEXHF72OPVCXWPRCDTFEIPWKGVNV2A, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.708367176 (547h3m11.579450221s) [noise=true , address=45.91.116.13:4154 , piece_id=4VHHRCQCEPZUWD2QT6V62WVKBRXMFY7NYAQCAF4DVB4OFCRSFGUA, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.716026030 (547h3m11.587109075s) [noise=true , address=45.91.116.14:4192 , piece_id=UMHHU3Y6JA7GBPWMG3NZRBNJ6QSP762N3LZUH4HEOJXEQBO5NP4A, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:12.434215195 (547h3m12.30529824s) [noise=true , address=45.91.116.6:4975 , piece_id=JRQ4W24BKCX2ELV7C6P3QXHLF4BWUO7FEI7CMDMV43YYTK7FB3HQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:12.434324403 (547h3m12.305407438s) [seg_pos={0 0}, upload=479693]: Segment reached optimal threshold of 13 pieces. Dec 28 10:32:12.434515456 (547h3m12.305598501s) [noise=true , address=77.90.31.21:4877 , piece_id=3ELH72FJ2GAD7CFLEMWFNKERUTYJFB6FE2V7S7YGOONGYU6WPBNQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Dec 28 10:32:12.434521548 (547h3m12.305604593s) [noise=true , address=77.90.31.21:4877 , piece_id=3ELH72FJ2GAD7CFLEMWFNKERUTYJFB6FE2V7S7YGOONGYU6WPBNQ, seg_pos={0 0}, upload=479693]: Long tail cancellation due to optimal threshold Dec 28 10:32:12.434710859 (547h3m12.305793894s) [noise=true , address=77.90.31.15:4700 , piece_id=CTHN5F73XF6BQ2ATYAUFBRGTD6YWEOK3U7OKAFXDMDLJ7DUZPGYQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 28 10:32:12.434716900 (547h3m12.305799945s) [noise=true , address=77.90.31.15:4700 , piece_id=CTHN5F73XF6BQ2ATYAUFBRGTD6YWEOK3U7OKAFXDMDLJ7DUZPGYQ, seg_pos={0 0}, upload=479693]: Long tail cancellation due to optimal threshold Dec 28 10:32:12.434797904 (547h3m12.305880949s) [noise=true , address=77.90.31.18:4808 , piece_id=UZP6F2ITW5Q3AFHKITHSUTPKZZVK2ZBLFF5DHH5UA5URTSPCP7JA, seg_pos={0 0}, upload=479693]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF Dec 28 10:32:12.434813053 (547h3m12.305896088s) [noise=true , address=77.90.31.18:4808 , piece_id=UZP6F2ITW5Q3AFHKITHSUTPKZZVK2ZBLFF5DHH5UA5URTSPCP7JA, seg_pos={0 0}, upload=479693]: Long tail cancellation due to optimal threshold Dec 28 10:32:12.434845295 (547h3m12.30592833s) [seg_pos={0 0}, upload=479693]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 10:32:12.434957198 (547h3m12.306040233s) [upload=479693]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 10:32:12.434962488 (547h3m12.306045523s) [upload=479693]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 10:32:12.725143372 (547h3m12.596226407s) [upload=479693]: Done uploading segments... Dec 28 10:32:12.725163410 (547h3m12.596246445s) [upload=479693]: Upload finished. err: <nil> Duration: 5.817229267s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [..X.........X...................X..XX.....X..X............XX] duration: 30.003752641s trace: https://jaeger.dev.storj.io/trace/12fdeb4fc6be55f2 FAIL (2025-12-28 10:32:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: JRQ4W24BKCX2ELV7C6P3QXHLF4BWUO7FEI7CMDMV43YYTK7FB3HQ) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: UMHHU3Y6JA7GBPWMG3NZRBNJ6QSP762N3LZUH4HEOJXEQBO5NP4A) context deadline exceeded ======LOGS====== Dec 28 10:32:09.171162516 (547h3m9.042245551s) [upload=479693]: Starting upload Dec 28 10:32:09.171211930 (547h3m9.042294965s) [upload=479693]: Done starting upload Dec 28 10:32:09.171389138 (547h3m9.042472173s) [upload=479693]: Uploading segments... Dec 28 10:32:09.171420808 (547h3m9.042503843s) [upload=479693]: Got next segment. Inline: false Dec 28 10:32:09.171446587 (547h3m9.042529622s) [upload=479693]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 10:32:09.171451076 (547h3m9.042534121s) [upload=479693]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 10:32:09.415778928 (547h3m9.286861973s) [seg_pos={0 0}, upload=479693]: Begin upload segment... Dec 28 10:32:09.415823763 (547h3m9.286906798s) [seg_pos={0 0}, upload=479693]: Done begin upload segment. Dec 28 10:32:09.415852628 (547h3m9.286935673s) [noise=true , address=145.79.227.5:4223 , piece_id=2DUTU22HULS2ERHZZ2TYB2YTL5ZUUAL5Z65V5VSNDKAUNWOYLFWQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.415915609 (547h3m9.286998644s) [noise=true , address=45.91.116.14:4192 , piece_id=UMHHU3Y6JA7GBPWMG3NZRBNJ6QSP762N3LZUH4HEOJXEQBO5NP4A, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.416427983 (547h3m9.287511019s) [noise=true , address=145.79.227.41:4086 , piece_id=OTV62ELETLJDI25ANYVI5TOAQKAMO5TRHWSGFLQSHBBRGBO3PLVQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.416839006 (547h3m9.287922041s) [noise=true , address=145.79.227.14:4840 , piece_id=6FSN47UNI7MQJBWBIV5THQD4WUXF52NM7M6BP7VHRB4J6E2F4VKQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.416935741 (547h3m9.288018786s) [noise=true , address=77.90.31.18:4808 , piece_id=UZP6F2ITW5Q3AFHKITHSUTPKZZVK2ZBLFF5DHH5UA5URTSPCP7JA, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.417242896 (547h3m9.288325941s) [noise=true , address=77.90.31.21:4877 , piece_id=3ELH72FJ2GAD7CFLEMWFNKERUTYJFB6FE2V7S7YGOONGYU6WPBNQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.417527407 (547h3m9.288610442s) [noise=true , address=45.91.116.13:4154 , piece_id=4VHHRCQCEPZUWD2QT6V62WVKBRXMFY7NYAQCAF4DVB4OFCRSFGUA, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.417826456 (547h3m9.288909491s) [noise=true , address=145.79.227.3:4149 , piece_id=GUIDXMU3Q5RTZTK2IOIJ2CXIMQ5HQGNDSEN5WVVZMRR336IM5FGQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.417930104 (547h3m9.289013149s) [noise=true , address=145.79.227.2:4113 , piece_id=7VXXE267VCB2OXILXW6TV4AJGNHW4LQVXEA4HOSVNMI6UAEMS3VA, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.418096461 (547h3m9.289179506s) [noise=true , address=145.79.227.47:4265 , piece_id=TDVXCOFNRIOSRS2OETJPUM3EELE4YN73KLHWDSPL6XPX5TACZCWQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.418451897 (547h3m9.289534942s) [noise=true , address=145.79.227.37:4567 , piece_id=ZY7KVVWYGKO6X2OONS7LXBBXDYKLSNIGL33U72HP6INZOUDUIS5A, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.419025139 (547h3m9.290108184s) [noise=true , address=77.90.31.15:4700 , piece_id=CTHN5F73XF6BQ2ATYAUFBRGTD6YWEOK3U7OKAFXDMDLJ7DUZPGYQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.419682581 (547h3m9.290765626s) [noise=true , address=45.91.116.6:4975 , piece_id=JRQ4W24BKCX2ELV7C6P3QXHLF4BWUO7FEI7CMDMV43YYTK7FB3HQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.420365671 (547h3m9.291448706s) [noise=true , address=145.79.227.46:4603 , piece_id=HUJL4BAY7U2G57TOMZVHQWTFOTN2RV2XKDK2MIUILVD6HR6Y27NQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.421011522 (547h3m9.292094557s) [noise=true , address=145.79.227.10:4403 , piece_id=FECST4PG6RGH6TUCOJZCSIOVRESXJMABM235ZOO5M4GYZTY75LTQ, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:09.421697077 (547h3m9.292780122s) [noise=true , address=145.79.227.29:4941 , piece_id=U5QPSY5M32O3ZGTK4YNIQNLEXHF72OPVCXWPRCDTFEIPWKGVNV2A, seg_pos={0 0}, upload=479693]: Uploading piece... Dec 28 10:32:11.133534367 (547h3m11.004617412s) [upload=479693]: Next returned nil segment Dec 28 10:32:11.133569233 (547h3m11.004652278s) [upload=479693]: Waiting for error group managing segments... Dec 28 10:32:11.421476958 (547h3m11.292559993s) [noise=true , address=145.79.227.3:4149 , piece_id=GUIDXMU3Q5RTZTK2IOIJ2CXIMQ5HQGNDSEN5WVVZMRR336IM5FGQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.423163530 (547h3m11.294246575s) [noise=true , address=145.79.227.14:4840 , piece_id=6FSN47UNI7MQJBWBIV5THQD4WUXF52NM7M6BP7VHRB4J6E2F4VKQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.428115299 (547h3m11.299198334s) [noise=true , address=145.79.227.41:4086 , piece_id=OTV62ELETLJDI25ANYVI5TOAQKAMO5TRHWSGFLQSHBBRGBO3PLVQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.436758778 (547h3m11.307841823s) [noise=true , address=145.79.227.10:4403 , piece_id=FECST4PG6RGH6TUCOJZCSIOVRESXJMABM235ZOO5M4GYZTY75LTQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.439394376 (547h3m11.310477411s) [noise=true , address=145.79.227.46:4603 , piece_id=HUJL4BAY7U2G57TOMZVHQWTFOTN2RV2XKDK2MIUILVD6HR6Y27NQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.439861225 (547h3m11.31094427s) [noise=true , address=145.79.227.47:4265 , piece_id=TDVXCOFNRIOSRS2OETJPUM3EELE4YN73KLHWDSPL6XPX5TACZCWQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.442937283 (547h3m11.314020318s) [noise=true , address=145.79.227.37:4567 , piece_id=ZY7KVVWYGKO6X2OONS7LXBBXDYKLSNIGL33U72HP6INZOUDUIS5A, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.444228622 (547h3m11.315311667s) [noise=true , address=145.79.227.5:4223 , piece_id=2DUTU22HULS2ERHZZ2TYB2YTL5ZUUAL5Z65V5VSNDKAUNWOYLFWQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.452679474 (547h3m11.323762519s) [noise=true , address=145.79.227.2:4113 , piece_id=7VXXE267VCB2OXILXW6TV4AJGNHW4LQVXEA4HOSVNMI6UAEMS3VA, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.457542493 (547h3m11.328625538s) [noise=true , address=145.79.227.29:4941 , piece_id=U5QPSY5M32O3ZGTK4YNIQNLEXHF72OPVCXWPRCDTFEIPWKGVNV2A, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.708367176 (547h3m11.579450221s) [noise=true , address=45.91.116.13:4154 , piece_id=4VHHRCQCEPZUWD2QT6V62WVKBRXMFY7NYAQCAF4DVB4OFCRSFGUA, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:11.716026030 (547h3m11.587109075s) [noise=true , address=45.91.116.14:4192 , piece_id=UMHHU3Y6JA7GBPWMG3NZRBNJ6QSP762N3LZUH4HEOJXEQBO5NP4A, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:12.434215195 (547h3m12.30529824s) [noise=true , address=45.91.116.6:4975 , piece_id=JRQ4W24BKCX2ELV7C6P3QXHLF4BWUO7FEI7CMDMV43YYTK7FB3HQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: <nil> Dec 28 10:32:12.434324403 (547h3m12.305407438s) [seg_pos={0 0}, upload=479693]: Segment reached optimal threshold of 13 pieces. Dec 28 10:32:12.434515456 (547h3m12.305598501s) [noise=true , address=77.90.31.21:4877 , piece_id=3ELH72FJ2GAD7CFLEMWFNKERUTYJFB6FE2V7S7YGOONGYU6WPBNQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Dec 28 10:32:12.434521548 (547h3m12.305604593s) [noise=true , address=77.90.31.21:4877 , piece_id=3ELH72FJ2GAD7CFLEMWFNKERUTYJFB6FE2V7S7YGOONGYU6WPBNQ, seg_pos={0 0}, upload=479693]: Long tail cancellation due to optimal threshold Dec 28 10:32:12.434710859 (547h3m12.305793894s) [noise=true , address=77.90.31.15:4700 , piece_id=CTHN5F73XF6BQ2ATYAUFBRGTD6YWEOK3U7OKAFXDMDLJ7DUZPGYQ, seg_pos={0 0}, upload=479693]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 28 10:32:12.434716900 (547h3m12.305799945s) [noise=true , address=77.90.31.15:4700 , piece_id=CTHN5F73XF6BQ2ATYAUFBRGTD6YWEOK3U7OKAFXDMDLJ7DUZPGYQ, seg_pos={0 0}, upload=479693]: Long tail cancellation due to optimal threshold Dec 28 10:32:12.434797904 (547h3m12.305880949s) [noise=true , address=77.90.31.18:4808 , piece_id=UZP6F2ITW5Q3AFHKITHSUTPKZZVK2ZBLFF5DHH5UA5URTSPCP7JA, seg_pos={0 0}, upload=479693]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF Dec 28 10:32:12.434813053 (547h3m12.305896088s) [noise=true , address=77.90.31.18:4808 , piece_id=UZP6F2ITW5Q3AFHKITHSUTPKZZVK2ZBLFF5DHH5UA5URTSPCP7JA, seg_pos={0 0}, upload=479693]: Long tail cancellation due to optimal threshold Dec 28 10:32:12.434845295 (547h3m12.30592833s) [seg_pos={0 0}, upload=479693]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 10:32:12.434957198 (547h3m12.306040233s) [upload=479693]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 10:32:12.434962488 (547h3m12.306045523s) [upload=479693]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 10:32:12.725143372 (547h3m12.596226407s) [upload=479693]: Done uploading segments... Dec 28 10:32:12.725163410 (547h3m12.596246445s) [upload=479693]: Upload finished. err: <nil> Duration: 30.003765756s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [...X.........X...................X..XX.....X..X............X] duration: 30.004034658s trace: https://jaeger.dev.storj.io/trace/3a7e4b566e4fc151 FAIL (2025-12-28 10:32:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 10:31:38.943089524 (547h2m38.814172559s) [upload=479684]: Starting upload Dec 28 10:31:38.943170588 (547h2m38.814253623s) [upload=479684]: Done starting upload Dec 28 10:31:38.943310123 (547h2m38.814393158s) [upload=479684]: Uploading segments... Dec 28 10:31:38.943413500 (547h2m38.814496535s) [upload=479684]: Got next segment. Inline: false Dec 28 10:31:38.943475769 (547h2m38.814558814s) [upload=479684]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 10:31:38.943482151 (547h2m38.814565186s) [upload=479684]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 10:31:39.190897532 (547h2m39.061980577s) [seg_pos={0 0}, upload=479684]: Begin upload segment... Dec 28 10:31:39.190959470 (547h2m39.062042505s) [seg_pos={0 0}, upload=479684]: Done begin upload segment. Dec 28 10:31:39.190988045 (547h2m39.06207108s) [noise=true , address=145.79.227.44:4195 , piece_id=6225HC5DLQEZ3DUXAXMK65EYZJ4XCGFZFKM5T35FOCDCV7WSKL7Q, seg_pos={0 0}, upload=479684]: Uploading piece... Dec 28 10:31:39.191196933 (547h2m39.062279978s) [noise=true , address=45.91.116.6:4974 , piece_id=QR5A32ONWD53H5UZOAXMTA3WOQ3CKJCVHI6XK2QFZ53FFNFUN7JQ, seg_pos={0 0}, upload=479684]: Uploading piece... Dec 28 10:31:39.191862119 (547h2m39.062945164s) [noise=true , address=45.91.116.10:4082 , piece_id=JD3SEIMXWTFWZTIUIWTFIUUJHOJWT2PPLB37Q33MJF3R6COAYDNQ, seg_pos={0 0}, upload=479684]: Uploading piece... Dec 28 10:31:39.191961749 (547h2m39.063044794s) [noise=true , address=45.91.116.5:4299 , piece_id=ROZTD34SSENZTXOEJ6SCIMYOZMJZ7VX6WZ2E2QBN2PGMJM2M7NYQ, seg_pos={0 0}, upload=479684]: Uploading piece... Dec 28 10:31:39.192321303 (547h2m39.063404338s) [noise=true , address=45.91.116.14:4191 , piece_id=A7XY7VNABAJLB7UQ2TDZ6EZLFRKCTC2NXTOKS7IMLCQSF5ZOA6WQ, seg_pos={0 0}, upload=479684]: Uploading piece... Dec 28 10:31:39.192368703 (547h2m39.063451979s) [noise=true , address=145.79.227.3:4150 , piece_id=5GPGG2BLTYWY3GUESIOMRQFMIJC4367ZPUCGW3TI4NCY3L6LPW4A, seg_pos={0 0}, upload=479684]: Uploading piece... Dec 28 10:31:39.192745170 (547h2m39.063828205s) [noise=true , address=145.79.227.16:4909 , piece_id=SOUMG6ZDPUGHMANPPMWHFRAWATI4L4Z3RYPSJPVLVOVM577LBHNQ, seg_pos={0 0}, upload=479684]: Uploading piece... Dec 28 10:31:39.192799614 (547h2m39.063882659s) [noise=true , address=145.79.227.51:4411 , piece_id=J2UL5PJCTH767CXJ5PH42SENIG2KQSUO2VIFF6WQIGS77BCDAJXQ, seg_pos={0 0}, upload=479684]: Uploading piece... Dec 28 10:31:39.193120315 (547h2m39.06420335s) [noise=true , address=77.90.31.11:4555 , piece_id=UTW5G2KDZRHZ3EHFGFHCFTZE3UD7UVRYMRPTXHRQNSZVPBNEZT7Q, seg_pos={0 0}, upload=479684]: Uploading piece... Dec 28 10:31:39.193170300 (547h2m39.064253335s) [noise=true , address=145.79.227.39:4015 , piece_id=P6JCV2TH6FVOSXSHKDOSLW27OVQOEXD6DIJ5QFNSA5T5TABMG46A, seg_pos={0 0}, upload=479684]: Uploading piece... Dec 28 10:31:39.193523884 (547h2m39.064606919s) [noise=true , address=145.79.227.14:4840 , piece_id=345UYNWJFFEO557RTUHE3YABZKKGY5BGWBIBEXNAVRAGFLAFDPQA, seg_pos={0 0}, upload=479684]: Uploading piece... Dec 28 10:31:39.193576954 (547h2m39.064659989s) [noise=true , address=145.79.227.28:4725 , piece_id=35Z4LRH4VSUWIIXGSOJGYFHMGGM7XUAY45TLBK3PKXDM26RW4UDQ, seg_pos={0 0}, upload=479684]: Uploading piece... Dec 28 10:31:39.193889459 (547h2m39.064972494s) [noise=true , address=145.79.227.52:4445 , piece_id=M7JILWX67E4IZ3RLE6QVQEZ7FKZOID2GOL7LRYVMPHPGSTPCS7WQ, seg_pos={0 0}, upload=479684]: Uploading piece... Dec 28 10:31:39.193943392 (547h2m39.065026438s) [noise=true , address=77.90.31.18:4805 , piece_id=6DNYCVD5EFIHPVUXFE6AIM3OEOOZVH5Z6WAOGZP7FIBTRW76R4PA, seg_pos={0 0}, upload=479684]: Uploading piece... Dec 28 10:31:39.194398599 (547h2m39.065481634s) [noise=true , address=145.79.227.17:4971 , piece_id=IJCDDKSPUAZE5ODHJFO7NC4F6P2LQCNRXUD7JJSN63GCQLDMXLTQ, seg_pos={0 0}, upload=479684]: Uploading piece... Dec 28 10:31:39.200393394 (547h2m39.071476439s) [noise=true , address=145.79.227.11:4730 , piece_id=R7WENADVKOVH6VQ5IOEPVFYGP7K7QOWVDDNTKMRJN3AR6JY2AQLA, seg_pos={0 0}, upload=479684]: Uploading piece... Dec 28 10:31:40.809781584 (547h2m40.680864629s) [upload=479684]: Next returned nil segment Dec 28 10:31:40.809833141 (547h2m40.680916186s) [upload=479684]: Waiting for error group managing segments... Dec 28 10:31:41.192252172 (547h2m41.063335207s) [noise=true , address=145.79.227.11:4730 , piece_id=R7WENADVKOVH6VQ5IOEPVFYGP7K7QOWVDDNTKMRJN3AR6JY2AQLA, seg_pos={0 0}, upload=479684]: Done uploading piece. err: <nil> Dec 28 10:31:41.206657603 (547h2m41.077740648s) [noise=true , address=145.79.227.52:4445 , piece_id=M7JILWX67E4IZ3RLE6QVQEZ7FKZOID2GOL7LRYVMPHPGSTPCS7WQ, seg_pos={0 0}, upload=479684]: Done uploading piece. err: <nil> Dec 28 10:31:41.206924290 (547h2m41.078007335s) [noise=true , address=145.79.227.51:4411 , piece_id=J2UL5PJCTH767CXJ5PH42SENIG2KQSUO2VIFF6WQIGS77BCDAJXQ, seg_pos={0 0}, upload=479684]: Done uploading piece. err: <nil> Dec 28 10:31:41.209704715 (547h2m41.08078776s) [noise=true , address=145.79.227.39:4015 , piece_id=P6JCV2TH6FVOSXSHKDOSLW27OVQOEXD6DIJ5QFNSA5T5TABMG46A, seg_pos={0 0}, upload=479684]: Done uploading piece. err: <nil> Dec 28 10:31:41.216949831 (547h2m41.088032876s) [noise=true , address=145.79.227.17:4971 , piece_id=IJCDDKSPUAZE5ODHJFO7NC4F6P2LQCNRXUD7JJSN63GCQLDMXLTQ, seg_pos={0 0}, upload=479684]: Done uploading piece. err: <nil> Dec 28 10:31:41.221689155 (547h2m41.09277219s) [noise=true , address=145.79.227.3:4150 , piece_id=5GPGG2BLTYWY3GUESIOMRQFMIJC4367ZPUCGW3TI4NCY3L6LPW4A, seg_pos={0 0}, upload=479684]: Done uploading piece. err: <nil> Dec 28 10:31:41.224928273 (547h2m41.096011308s) [noise=true , address=145.79.227.14:4840 , piece_id=345UYNWJFFEO557RTUHE3YABZKKGY5BGWBIBEXNAVRAGFLAFDPQA, seg_pos={0 0}, upload=479684]: Done uploading piece. err: <nil> Dec 28 10:31:41.228895707 (547h2m41.099978742s) [noise=true , address=145.79.227.28:4725 , piece_id=35Z4LRH4VSUWIIXGSOJGYFHMGGM7XUAY45TLBK3PKXDM26RW4UDQ, seg_pos={0 0}, upload=479684]: Done uploading piece. err: <nil> Dec 28 10:31:41.231142556 (547h2m41.102225591s) [noise=true , address=145.79.227.16:4909 , piece_id=SOUMG6ZDPUGHMANPPMWHFRAWATI4L4Z3RYPSJPVLVOVM577LBHNQ, seg_pos={0 0}, upload=479684]: Done uploading piece. err: <nil> Dec 28 10:31:41.446529660 (547h2m41.317612705s) [noise=true , address=45.91.116.14:4191 , piece_id=A7XY7VNABAJLB7UQ2TDZ6EZLFRKCTC2NXTOKS7IMLCQSF5ZOA6WQ, seg_pos={0 0}, upload=479684]: Done uploading piece. err: <nil> Dec 28 10:31:41.535604725 (547h2m41.40668776s) [noise=true , address=77.90.31.18:4805 , piece_id=6DNYCVD5EFIHPVUXFE6AIM3OEOOZVH5Z6WAOGZP7FIBTRW76R4PA, seg_pos={0 0}, upload=479684]: Done uploading piece. err: <nil> Dec 28 10:31:41.657905236 (547h2m41.528988281s) [noise=true , address=45.91.116.10:4082 , piece_id=JD3SEIMXWTFWZTIUIWTFIUUJHOJWT2PPLB37Q33MJF3R6COAYDNQ, seg_pos={0 0}, upload=479684]: Done uploading piece. err: <nil> Dec 28 10:31:41.825772482 (547h2m41.696855518s) [noise=true , address=45.91.116.6:4974 , piece_id=QR5A32ONWD53H5UZOAXMTA3WOQ3CKJCVHI6XK2QFZ53FFNFUN7JQ, seg_pos={0 0}, upload=479684]: Done uploading piece. err: <nil> Dec 28 10:31:41.825828860 (547h2m41.696911905s) [seg_pos={0 0}, upload=479684]: Segment reached optimal threshold of 13 pieces. Dec 28 10:31:41.826081160 (547h2m41.697164215s) [noise=true , address=45.91.116.5:4299 , piece_id=ROZTD34SSENZTXOEJ6SCIMYOZMJZ7VX6WZ2E2QBN2PGMJM2M7NYQ, seg_pos={0 0}, upload=479684]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Dec 28 10:31:41.826090038 (547h2m41.697173093s) [noise=true , address=45.91.116.5:4299 , piece_id=ROZTD34SSENZTXOEJ6SCIMYOZMJZ7VX6WZ2E2QBN2PGMJM2M7NYQ, seg_pos={0 0}, upload=479684]: Long tail cancellation due to optimal threshold Dec 28 10:31:41.826382344 (547h2m41.697465389s) [noise=true , address=77.90.31.11:4555 , piece_id=UTW5G2KDZRHZ3EHFGFHCFTZE3UD7UVRYMRPTXHRQNSZVPBNEZT7Q, seg_pos={0 0}, upload=479684]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: write tcp 188.166.225.170:54858->77.90.31.11:4555: use of closed network connection; write tcp 188.166.225.170:54858->77.90.31.11:4555: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54858->77.90.31.11:4555: use of closed network connection Dec 28 10:31:41.826395098 (547h2m41.697478133s) [noise=true , address=77.90.31.11:4555 , piece_id=UTW5G2KDZRHZ3EHFGFHCFTZE3UD7UVRYMRPTXHRQNSZVPBNEZT7Q, seg_pos={0 0}, upload=479684]: Long tail cancellation due to optimal threshold Dec 28 10:31:41.826597905 (547h2m41.69768096s) [noise=true , address=145.79.227.44:4195 , piece_id=6225HC5DLQEZ3DUXAXMK65EYZJ4XCGFZFKM5T35FOCDCV7WSKL7Q, seg_pos={0 0}, upload=479684]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 28 10:31:41.826607773 (547h2m41.697690808s) [noise=true , address=145.79.227.44:4195 , piece_id=6225HC5DLQEZ3DUXAXMK65EYZJ4XCGFZFKM5T35FOCDCV7WSKL7Q, seg_pos={0 0}, upload=479684]: Long tail cancellation due to optimal threshold Dec 28 10:31:41.826639724 (547h2m41.697722769s) [seg_pos={0 0}, upload=479684]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 10:31:41.826735116 (547h2m41.697818161s) [upload=479684]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 10:31:41.826742259 (547h2m41.697825324s) [upload=479684]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 10:31:42.111155741 (547h2m41.982238776s) [upload=479684]: Done uploading segments... Dec 28 10:31:42.111180208 (547h2m41.982263253s) [upload=479684]: Upload finished. err: <nil> Duration: 30.004062971s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [..XXXXX...X........X.....X..................X..............X] duration: 5.001510819s trace: https://jaeger.dev.storj.io/trace/5ea2a8966c11b1a9 FAIL (2025-12-28 10:31:36 UTC) orbiter warning: context deadline exceeded Duration: 5.0015326s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [...XXXXX...X........X.....X..................X..............] duration: 3.648303782s trace: https://jaeger.dev.storj.io/trace/49dbd650238e8f85 FAIL (2025-12-28 10:31:14 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.648353928s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [....X.........X...................X..XX.....X..X............] duration: 5.904192617s trace: https://jaeger.dev.storj.io/trace/37d347f90dc836a FAIL (2025-12-28 10:30:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== segments... Dec 28 10:20:52.978430940 (546h51m52.849513975s) [upload=479529]: Got next segment. Inline: false Dec 28 10:20:52.978465887 (546h51m52.849548922s) [upload=479529]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 10:20:52.978471527 (546h51m52.849554562s) [upload=479529]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 10:20:53.224037597 (546h51m53.095120632s) [seg_pos={0 0}, upload=479529]: Begin upload segment... Dec 28 10:20:53.224084497 (546h51m53.095167532s) [seg_pos={0 0}, upload=479529]: Done begin upload segment. Dec 28 10:20:53.224140062 (546h51m53.095223117s) [noise=true , address=77.90.31.11:4556 , piece_id=ZHXRF3QCUU4FV7P3OJ7TLFEFLMR7F26K4UHG4OFWYIZFTJOJGD6A, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.224148639 (546h51m53.095231685s) [noise=true , address=145.79.227.28:4725 , piece_id=BJ32VJWVH66R5T57UVSE76M227T3PND3IS4GYTVJCN62NRDFK2ZA, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.224718384 (546h51m53.095801419s) [noise=true , address=145.79.227.42:4123 , piece_id=CG5UCA4G3BNPI3BAFBTMOAGUZJEU7FW2MSNRYSHV77LPLR4CK5SA, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.225234898 (546h51m53.096317933s) [noise=true , address=145.79.227.3:4151 , piece_id=SUWYBG7TPG7BGBOS5G4UKXR2Z753Q4GJ2AN3M3WCRYPN3IKVEHQQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.225504882 (546h51m53.096587917s) [noise=true , address=145.79.227.21:4474 , piece_id=EEGOXLAYIZVCT4NF34CFLAVESLW2WPEOKJPVPRYIUSHKRIC5VR3Q, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.226212750 (546h51m53.097295785s) [noise=true , address=145.79.227.14:4840 , piece_id=YNOHADAVEJGWSG5VERF7HWN7GHHIPDWWECQ5HTPSS7HKC7DMVOWQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.226608573 (546h51m53.097691618s) [noise=true , address=145.79.227.6:4258 , piece_id=2DFJ2B7ETKXW76LASWPT4LGUAUV7I2OEV3N5JGOVPW74QVK6F4TQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.226767135 (546h51m53.09785018s) [noise=true , address=45.91.116.7:4009 , piece_id=FWZA5D3EYV77RAPKAZOBR4Z4OOSNWXHXXE4WJ4QNEETMZPJHYMAQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.227078098 (546h51m53.098161143s) [noise=true , address=45.91.116.9:4262 , piece_id=6JXAFCBODMHO5OYB5WGZNB762V5JUL32RXMFYHP6E3DCTWSSQYFA, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.227238392 (546h51m53.098321427s) [noise=true , address=145.79.227.16:4911 , piece_id=TZOOFWAQ33QB5B6MOUJ3XH7EB3IALWYGN43QUGNXOG6EXOJTB4AA, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.227466998 (546h51m53.098550033s) [noise=true , address=145.79.227.48:4301 , piece_id=NZ24TJ7I42YFMDOUF5SHORFQIHOKJBDILK3SGMGFHJ46OO6DGX5Q, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.227646309 (546h51m53.098729354s) [noise=true , address=77.90.31.15:4697 , piece_id=3FQIGY4VHPZIQ7ONVHGW7Z4UIV7EAECOAPGQEUVRYHXOLEOLHCTA, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.228003639 (546h51m53.099086684s) [noise=true , address=145.79.227.2:4115 , piece_id=XKLFZEWHLNY6AA5RVLWWPCLVUMWDU4YBJJQH2UFQ4U27YGVZGOXQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.228329440 (546h51m53.099412475s) [noise=true , address=145.79.227.32:4835 , piece_id=K5FO3OYDESYAUI3QUQ65WTEFJ2N77PPJL5X3FFNKAYU4X5PWKXBQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.228632276 (546h51m53.099715311s) [noise=true , address=45.91.116.8:4047 , piece_id=ATZ3XPE3S3AZSYXQZQWBDLBZQCVAMVIDX3Q2PDKQETLXWYEWMKPQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.228942016 (546h51m53.100025061s) [noise=true , address=145.79.227.36:4519 , piece_id=WVXG7E2MKVEYO4MHELTZ5HNRIXPJWLF2ZL4AU46X32W62SW5V3IQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:54.851414651 (546h51m54.722497686s) [upload=479529]: Next returned nil segment Dec 28 10:20:54.851429859 (546h51m54.722512894s) [upload=479529]: Waiting for error group managing segments... Dec 28 10:20:55.232471268 (546h51m55.103554303s) [noise=true , address=145.79.227.16:4911 , piece_id=TZOOFWAQ33QB5B6MOUJ3XH7EB3IALWYGN43QUGNXOG6EXOJTB4AA, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.232529519 (546h51m55.103612554s) [noise=true , address=145.79.227.2:4115 , piece_id=XKLFZEWHLNY6AA5RVLWWPCLVUMWDU4YBJJQH2UFQ4U27YGVZGOXQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.234914169 (546h51m55.105997204s) [noise=true , address=145.79.227.32:4835 , piece_id=K5FO3OYDESYAUI3QUQ65WTEFJ2N77PPJL5X3FFNKAYU4X5PWKXBQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.235465209 (546h51m55.106548244s) [noise=true , address=145.79.227.28:4725 , piece_id=BJ32VJWVH66R5T57UVSE76M227T3PND3IS4GYTVJCN62NRDFK2ZA, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.235615435 (546h51m55.10669847s) [noise=true , address=145.79.227.21:4474 , piece_id=EEGOXLAYIZVCT4NF34CFLAVESLW2WPEOKJPVPRYIUSHKRIC5VR3Q, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.236221559 (546h51m55.107304594s) [noise=true , address=145.79.227.6:4258 , piece_id=2DFJ2B7ETKXW76LASWPT4LGUAUV7I2OEV3N5JGOVPW74QVK6F4TQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.246296964 (546h51m55.117379999s) [noise=true , address=145.79.227.42:4123 , piece_id=CG5UCA4G3BNPI3BAFBTMOAGUZJEU7FW2MSNRYSHV77LPLR4CK5SA, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.246947282 (546h51m55.118030317s) [noise=true , address=145.79.227.3:4151 , piece_id=SUWYBG7TPG7BGBOS5G4UKXR2Z753Q4GJ2AN3M3WCRYPN3IKVEHQQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.251425650 (546h51m55.122508695s) [noise=true , address=145.79.227.14:4840 , piece_id=YNOHADAVEJGWSG5VERF7HWN7GHHIPDWWECQ5HTPSS7HKC7DMVOWQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.256324468 (546h51m55.127407513s) [noise=true , address=145.79.227.48:4301 , piece_id=NZ24TJ7I42YFMDOUF5SHORFQIHOKJBDILK3SGMGFHJ46OO6DGX5Q, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.259608812 (546h51m55.130691847s) [noise=true , address=145.79.227.36:4519 , piece_id=WVXG7E2MKVEYO4MHELTZ5HNRIXPJWLF2ZL4AU46X32W62SW5V3IQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.483932663 (546h51m55.355015698s) [noise=true , address=45.91.116.8:4047 , piece_id=ATZ3XPE3S3AZSYXQZQWBDLBZQCVAMVIDX3Q2PDKQETLXWYEWMKPQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.551135548 (546h51m55.422218594s) [noise=true , address=45.91.116.9:4262 , piece_id=6JXAFCBODMHO5OYB5WGZNB762V5JUL32RXMFYHP6E3DCTWSSQYFA, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.551161338 (546h51m55.422244383s) [seg_pos={0 0}, upload=479529]: Segment reached optimal threshold of 13 pieces. Dec 28 10:20:55.551510112 (546h51m55.422593147s) [noise=true , address=77.90.31.11:4556 , piece_id=ZHXRF3QCUU4FV7P3OJ7TLFEFLMR7F26K4UHG4OFWYIZFTJOJGD6A, seg_pos={0 0}, upload=479529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: write tcp 188.166.225.170:33006->77.90.31.11:4556: use of closed network connection; write tcp 188.166.225.170:33006->77.90.31.11:4556: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33006->77.90.31.11:4556: use of closed network connection Dec 28 10:20:55.551527014 (546h51m55.422610059s) [noise=true , address=77.90.31.11:4556 , piece_id=ZHXRF3QCUU4FV7P3OJ7TLFEFLMR7F26K4UHG4OFWYIZFTJOJGD6A, seg_pos={0 0}, upload=479529]: Long tail cancellation due to optimal threshold Dec 28 10:20:55.551578462 (546h51m55.422661507s) [noise=true , address=45.91.116.7:4009 , piece_id=FWZA5D3EYV77RAPKAZOBR4Z4OOSNWXHXXE4WJ4QNEETMZPJHYMAQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 28 10:20:55.551591066 (546h51m55.422674121s) [noise=true , address=45.91.116.7:4009 , piece_id=FWZA5D3EYV77RAPKAZOBR4Z4OOSNWXHXXE4WJ4QNEETMZPJHYMAQ, seg_pos={0 0}, upload=479529]: Long tail cancellation due to optimal threshold Dec 28 10:20:55.551835822 (546h51m55.422918867s) [noise=true , address=77.90.31.15:4697 , piece_id=3FQIGY4VHPZIQ7ONVHGW7Z4UIV7EAECOAPGQEUVRYHXOLEOLHCTA, seg_pos={0 0}, upload=479529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:58702->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:58702->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58702->77.90.31.15:4697: use of closed network connection Dec 28 10:20:55.551845641 (546h51m55.422928676s) [noise=true , address=77.90.31.15:4697 , piece_id=3FQIGY4VHPZIQ7ONVHGW7Z4UIV7EAECOAPGQEUVRYHXOLEOLHCTA, seg_pos={0 0}, upload=479529]: Long tail cancellation due to optimal threshold Dec 28 10:20:55.551895075 (546h51m55.42297812s) [seg_pos={0 0}, upload=479529]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 10:20:55.552135894 (546h51m55.423218929s) [upload=479529]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 10:20:55.552144850 (546h51m55.423227885s) [upload=479529]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 10:20:55.854005394 (546h51m55.72508845s) [upload=479529]: Done uploading segments... Dec 28 10:20:55.854081920 (546h51m55.725164965s) [upload=479529]: Upload finished. err: <nil> Duration: 5.904224838s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [....XXXXX...X........X.....X..................X.............] duration: 5.001542198s trace: https://jaeger.dev.storj.io/trace/49dbd650238e8f85 FAIL (2025-12-28 10:30:29 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001571273s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.....X.........X...................X..XX.....X..X...........] duration: 5.990165366s trace: https://jaeger.dev.storj.io/trace/37d347f90dc836a FAIL (2025-12-28 10:30:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== segments... Dec 28 10:20:52.978430940 (546h51m52.849513975s) [upload=479529]: Got next segment. Inline: false Dec 28 10:20:52.978465887 (546h51m52.849548922s) [upload=479529]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 10:20:52.978471527 (546h51m52.849554562s) [upload=479529]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 10:20:53.224037597 (546h51m53.095120632s) [seg_pos={0 0}, upload=479529]: Begin upload segment... Dec 28 10:20:53.224084497 (546h51m53.095167532s) [seg_pos={0 0}, upload=479529]: Done begin upload segment. Dec 28 10:20:53.224140062 (546h51m53.095223117s) [noise=true , address=77.90.31.11:4556 , piece_id=ZHXRF3QCUU4FV7P3OJ7TLFEFLMR7F26K4UHG4OFWYIZFTJOJGD6A, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.224148639 (546h51m53.095231685s) [noise=true , address=145.79.227.28:4725 , piece_id=BJ32VJWVH66R5T57UVSE76M227T3PND3IS4GYTVJCN62NRDFK2ZA, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.224718384 (546h51m53.095801419s) [noise=true , address=145.79.227.42:4123 , piece_id=CG5UCA4G3BNPI3BAFBTMOAGUZJEU7FW2MSNRYSHV77LPLR4CK5SA, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.225234898 (546h51m53.096317933s) [noise=true , address=145.79.227.3:4151 , piece_id=SUWYBG7TPG7BGBOS5G4UKXR2Z753Q4GJ2AN3M3WCRYPN3IKVEHQQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.225504882 (546h51m53.096587917s) [noise=true , address=145.79.227.21:4474 , piece_id=EEGOXLAYIZVCT4NF34CFLAVESLW2WPEOKJPVPRYIUSHKRIC5VR3Q, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.226212750 (546h51m53.097295785s) [noise=true , address=145.79.227.14:4840 , piece_id=YNOHADAVEJGWSG5VERF7HWN7GHHIPDWWECQ5HTPSS7HKC7DMVOWQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.226608573 (546h51m53.097691618s) [noise=true , address=145.79.227.6:4258 , piece_id=2DFJ2B7ETKXW76LASWPT4LGUAUV7I2OEV3N5JGOVPW74QVK6F4TQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.226767135 (546h51m53.09785018s) [noise=true , address=45.91.116.7:4009 , piece_id=FWZA5D3EYV77RAPKAZOBR4Z4OOSNWXHXXE4WJ4QNEETMZPJHYMAQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.227078098 (546h51m53.098161143s) [noise=true , address=45.91.116.9:4262 , piece_id=6JXAFCBODMHO5OYB5WGZNB762V5JUL32RXMFYHP6E3DCTWSSQYFA, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.227238392 (546h51m53.098321427s) [noise=true , address=145.79.227.16:4911 , piece_id=TZOOFWAQ33QB5B6MOUJ3XH7EB3IALWYGN43QUGNXOG6EXOJTB4AA, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.227466998 (546h51m53.098550033s) [noise=true , address=145.79.227.48:4301 , piece_id=NZ24TJ7I42YFMDOUF5SHORFQIHOKJBDILK3SGMGFHJ46OO6DGX5Q, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.227646309 (546h51m53.098729354s) [noise=true , address=77.90.31.15:4697 , piece_id=3FQIGY4VHPZIQ7ONVHGW7Z4UIV7EAECOAPGQEUVRYHXOLEOLHCTA, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.228003639 (546h51m53.099086684s) [noise=true , address=145.79.227.2:4115 , piece_id=XKLFZEWHLNY6AA5RVLWWPCLVUMWDU4YBJJQH2UFQ4U27YGVZGOXQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.228329440 (546h51m53.099412475s) [noise=true , address=145.79.227.32:4835 , piece_id=K5FO3OYDESYAUI3QUQ65WTEFJ2N77PPJL5X3FFNKAYU4X5PWKXBQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.228632276 (546h51m53.099715311s) [noise=true , address=45.91.116.8:4047 , piece_id=ATZ3XPE3S3AZSYXQZQWBDLBZQCVAMVIDX3Q2PDKQETLXWYEWMKPQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.228942016 (546h51m53.100025061s) [noise=true , address=145.79.227.36:4519 , piece_id=WVXG7E2MKVEYO4MHELTZ5HNRIXPJWLF2ZL4AU46X32W62SW5V3IQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:54.851414651 (546h51m54.722497686s) [upload=479529]: Next returned nil segment Dec 28 10:20:54.851429859 (546h51m54.722512894s) [upload=479529]: Waiting for error group managing segments... Dec 28 10:20:55.232471268 (546h51m55.103554303s) [noise=true , address=145.79.227.16:4911 , piece_id=TZOOFWAQ33QB5B6MOUJ3XH7EB3IALWYGN43QUGNXOG6EXOJTB4AA, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.232529519 (546h51m55.103612554s) [noise=true , address=145.79.227.2:4115 , piece_id=XKLFZEWHLNY6AA5RVLWWPCLVUMWDU4YBJJQH2UFQ4U27YGVZGOXQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.234914169 (546h51m55.105997204s) [noise=true , address=145.79.227.32:4835 , piece_id=K5FO3OYDESYAUI3QUQ65WTEFJ2N77PPJL5X3FFNKAYU4X5PWKXBQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.235465209 (546h51m55.106548244s) [noise=true , address=145.79.227.28:4725 , piece_id=BJ32VJWVH66R5T57UVSE76M227T3PND3IS4GYTVJCN62NRDFK2ZA, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.235615435 (546h51m55.10669847s) [noise=true , address=145.79.227.21:4474 , piece_id=EEGOXLAYIZVCT4NF34CFLAVESLW2WPEOKJPVPRYIUSHKRIC5VR3Q, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.236221559 (546h51m55.107304594s) [noise=true , address=145.79.227.6:4258 , piece_id=2DFJ2B7ETKXW76LASWPT4LGUAUV7I2OEV3N5JGOVPW74QVK6F4TQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.246296964 (546h51m55.117379999s) [noise=true , address=145.79.227.42:4123 , piece_id=CG5UCA4G3BNPI3BAFBTMOAGUZJEU7FW2MSNRYSHV77LPLR4CK5SA, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.246947282 (546h51m55.118030317s) [noise=true , address=145.79.227.3:4151 , piece_id=SUWYBG7TPG7BGBOS5G4UKXR2Z753Q4GJ2AN3M3WCRYPN3IKVEHQQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.251425650 (546h51m55.122508695s) [noise=true , address=145.79.227.14:4840 , piece_id=YNOHADAVEJGWSG5VERF7HWN7GHHIPDWWECQ5HTPSS7HKC7DMVOWQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.256324468 (546h51m55.127407513s) [noise=true , address=145.79.227.48:4301 , piece_id=NZ24TJ7I42YFMDOUF5SHORFQIHOKJBDILK3SGMGFHJ46OO6DGX5Q, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.259608812 (546h51m55.130691847s) [noise=true , address=145.79.227.36:4519 , piece_id=WVXG7E2MKVEYO4MHELTZ5HNRIXPJWLF2ZL4AU46X32W62SW5V3IQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.483932663 (546h51m55.355015698s) [noise=true , address=45.91.116.8:4047 , piece_id=ATZ3XPE3S3AZSYXQZQWBDLBZQCVAMVIDX3Q2PDKQETLXWYEWMKPQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.551135548 (546h51m55.422218594s) [noise=true , address=45.91.116.9:4262 , piece_id=6JXAFCBODMHO5OYB5WGZNB762V5JUL32RXMFYHP6E3DCTWSSQYFA, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.551161338 (546h51m55.422244383s) [seg_pos={0 0}, upload=479529]: Segment reached optimal threshold of 13 pieces. Dec 28 10:20:55.551510112 (546h51m55.422593147s) [noise=true , address=77.90.31.11:4556 , piece_id=ZHXRF3QCUU4FV7P3OJ7TLFEFLMR7F26K4UHG4OFWYIZFTJOJGD6A, seg_pos={0 0}, upload=479529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: write tcp 188.166.225.170:33006->77.90.31.11:4556: use of closed network connection; write tcp 188.166.225.170:33006->77.90.31.11:4556: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33006->77.90.31.11:4556: use of closed network connection Dec 28 10:20:55.551527014 (546h51m55.422610059s) [noise=true , address=77.90.31.11:4556 , piece_id=ZHXRF3QCUU4FV7P3OJ7TLFEFLMR7F26K4UHG4OFWYIZFTJOJGD6A, seg_pos={0 0}, upload=479529]: Long tail cancellation due to optimal threshold Dec 28 10:20:55.551578462 (546h51m55.422661507s) [noise=true , address=45.91.116.7:4009 , piece_id=FWZA5D3EYV77RAPKAZOBR4Z4OOSNWXHXXE4WJ4QNEETMZPJHYMAQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 28 10:20:55.551591066 (546h51m55.422674121s) [noise=true , address=45.91.116.7:4009 , piece_id=FWZA5D3EYV77RAPKAZOBR4Z4OOSNWXHXXE4WJ4QNEETMZPJHYMAQ, seg_pos={0 0}, upload=479529]: Long tail cancellation due to optimal threshold Dec 28 10:20:55.551835822 (546h51m55.422918867s) [noise=true , address=77.90.31.15:4697 , piece_id=3FQIGY4VHPZIQ7ONVHGW7Z4UIV7EAECOAPGQEUVRYHXOLEOLHCTA, seg_pos={0 0}, upload=479529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:58702->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:58702->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58702->77.90.31.15:4697: use of closed network connection Dec 28 10:20:55.551845641 (546h51m55.422928676s) [noise=true , address=77.90.31.15:4697 , piece_id=3FQIGY4VHPZIQ7ONVHGW7Z4UIV7EAECOAPGQEUVRYHXOLEOLHCTA, seg_pos={0 0}, upload=479529]: Long tail cancellation due to optimal threshold Dec 28 10:20:55.551895075 (546h51m55.42297812s) [seg_pos={0 0}, upload=479529]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 10:20:55.552135894 (546h51m55.423218929s) [upload=479529]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 10:20:55.552144850 (546h51m55.423227885s) [upload=479529]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 10:20:55.854005394 (546h51m55.72508845s) [upload=479529]: Done uploading segments... Dec 28 10:20:55.854081920 (546h51m55.725164965s) [upload=479529]: Upload finished. err: <nil> Duration: 5.990204961s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [X....XXXXX...X........X.....X..................X............] duration: 3.377798846s trace: https://jaeger.dev.storj.io/trace/49dbd650238e8f85 FAIL (2025-12-28 10:29:39 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.377823192s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [......X.........X...................X..XX.....X..X..........] duration: 5.962314664s trace: https://jaeger.dev.storj.io/trace/37d347f90dc836a FAIL (2025-12-28 10:29:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== segments... Dec 28 10:20:52.978430940 (546h51m52.849513975s) [upload=479529]: Got next segment. Inline: false Dec 28 10:20:52.978465887 (546h51m52.849548922s) [upload=479529]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 10:20:52.978471527 (546h51m52.849554562s) [upload=479529]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 10:20:53.224037597 (546h51m53.095120632s) [seg_pos={0 0}, upload=479529]: Begin upload segment... Dec 28 10:20:53.224084497 (546h51m53.095167532s) [seg_pos={0 0}, upload=479529]: Done begin upload segment. Dec 28 10:20:53.224140062 (546h51m53.095223117s) [noise=true , address=77.90.31.11:4556 , piece_id=ZHXRF3QCUU4FV7P3OJ7TLFEFLMR7F26K4UHG4OFWYIZFTJOJGD6A, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.224148639 (546h51m53.095231685s) [noise=true , address=145.79.227.28:4725 , piece_id=BJ32VJWVH66R5T57UVSE76M227T3PND3IS4GYTVJCN62NRDFK2ZA, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.224718384 (546h51m53.095801419s) [noise=true , address=145.79.227.42:4123 , piece_id=CG5UCA4G3BNPI3BAFBTMOAGUZJEU7FW2MSNRYSHV77LPLR4CK5SA, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.225234898 (546h51m53.096317933s) [noise=true , address=145.79.227.3:4151 , piece_id=SUWYBG7TPG7BGBOS5G4UKXR2Z753Q4GJ2AN3M3WCRYPN3IKVEHQQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.225504882 (546h51m53.096587917s) [noise=true , address=145.79.227.21:4474 , piece_id=EEGOXLAYIZVCT4NF34CFLAVESLW2WPEOKJPVPRYIUSHKRIC5VR3Q, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.226212750 (546h51m53.097295785s) [noise=true , address=145.79.227.14:4840 , piece_id=YNOHADAVEJGWSG5VERF7HWN7GHHIPDWWECQ5HTPSS7HKC7DMVOWQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.226608573 (546h51m53.097691618s) [noise=true , address=145.79.227.6:4258 , piece_id=2DFJ2B7ETKXW76LASWPT4LGUAUV7I2OEV3N5JGOVPW74QVK6F4TQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.226767135 (546h51m53.09785018s) [noise=true , address=45.91.116.7:4009 , piece_id=FWZA5D3EYV77RAPKAZOBR4Z4OOSNWXHXXE4WJ4QNEETMZPJHYMAQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.227078098 (546h51m53.098161143s) [noise=true , address=45.91.116.9:4262 , piece_id=6JXAFCBODMHO5OYB5WGZNB762V5JUL32RXMFYHP6E3DCTWSSQYFA, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.227238392 (546h51m53.098321427s) [noise=true , address=145.79.227.16:4911 , piece_id=TZOOFWAQ33QB5B6MOUJ3XH7EB3IALWYGN43QUGNXOG6EXOJTB4AA, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.227466998 (546h51m53.098550033s) [noise=true , address=145.79.227.48:4301 , piece_id=NZ24TJ7I42YFMDOUF5SHORFQIHOKJBDILK3SGMGFHJ46OO6DGX5Q, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.227646309 (546h51m53.098729354s) [noise=true , address=77.90.31.15:4697 , piece_id=3FQIGY4VHPZIQ7ONVHGW7Z4UIV7EAECOAPGQEUVRYHXOLEOLHCTA, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.228003639 (546h51m53.099086684s) [noise=true , address=145.79.227.2:4115 , piece_id=XKLFZEWHLNY6AA5RVLWWPCLVUMWDU4YBJJQH2UFQ4U27YGVZGOXQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.228329440 (546h51m53.099412475s) [noise=true , address=145.79.227.32:4835 , piece_id=K5FO3OYDESYAUI3QUQ65WTEFJ2N77PPJL5X3FFNKAYU4X5PWKXBQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.228632276 (546h51m53.099715311s) [noise=true , address=45.91.116.8:4047 , piece_id=ATZ3XPE3S3AZSYXQZQWBDLBZQCVAMVIDX3Q2PDKQETLXWYEWMKPQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.228942016 (546h51m53.100025061s) [noise=true , address=145.79.227.36:4519 , piece_id=WVXG7E2MKVEYO4MHELTZ5HNRIXPJWLF2ZL4AU46X32W62SW5V3IQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:54.851414651 (546h51m54.722497686s) [upload=479529]: Next returned nil segment Dec 28 10:20:54.851429859 (546h51m54.722512894s) [upload=479529]: Waiting for error group managing segments... Dec 28 10:20:55.232471268 (546h51m55.103554303s) [noise=true , address=145.79.227.16:4911 , piece_id=TZOOFWAQ33QB5B6MOUJ3XH7EB3IALWYGN43QUGNXOG6EXOJTB4AA, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.232529519 (546h51m55.103612554s) [noise=true , address=145.79.227.2:4115 , piece_id=XKLFZEWHLNY6AA5RVLWWPCLVUMWDU4YBJJQH2UFQ4U27YGVZGOXQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.234914169 (546h51m55.105997204s) [noise=true , address=145.79.227.32:4835 , piece_id=K5FO3OYDESYAUI3QUQ65WTEFJ2N77PPJL5X3FFNKAYU4X5PWKXBQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.235465209 (546h51m55.106548244s) [noise=true , address=145.79.227.28:4725 , piece_id=BJ32VJWVH66R5T57UVSE76M227T3PND3IS4GYTVJCN62NRDFK2ZA, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.235615435 (546h51m55.10669847s) [noise=true , address=145.79.227.21:4474 , piece_id=EEGOXLAYIZVCT4NF34CFLAVESLW2WPEOKJPVPRYIUSHKRIC5VR3Q, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.236221559 (546h51m55.107304594s) [noise=true , address=145.79.227.6:4258 , piece_id=2DFJ2B7ETKXW76LASWPT4LGUAUV7I2OEV3N5JGOVPW74QVK6F4TQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.246296964 (546h51m55.117379999s) [noise=true , address=145.79.227.42:4123 , piece_id=CG5UCA4G3BNPI3BAFBTMOAGUZJEU7FW2MSNRYSHV77LPLR4CK5SA, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.246947282 (546h51m55.118030317s) [noise=true , address=145.79.227.3:4151 , piece_id=SUWYBG7TPG7BGBOS5G4UKXR2Z753Q4GJ2AN3M3WCRYPN3IKVEHQQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.251425650 (546h51m55.122508695s) [noise=true , address=145.79.227.14:4840 , piece_id=YNOHADAVEJGWSG5VERF7HWN7GHHIPDWWECQ5HTPSS7HKC7DMVOWQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.256324468 (546h51m55.127407513s) [noise=true , address=145.79.227.48:4301 , piece_id=NZ24TJ7I42YFMDOUF5SHORFQIHOKJBDILK3SGMGFHJ46OO6DGX5Q, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.259608812 (546h51m55.130691847s) [noise=true , address=145.79.227.36:4519 , piece_id=WVXG7E2MKVEYO4MHELTZ5HNRIXPJWLF2ZL4AU46X32W62SW5V3IQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.483932663 (546h51m55.355015698s) [noise=true , address=45.91.116.8:4047 , piece_id=ATZ3XPE3S3AZSYXQZQWBDLBZQCVAMVIDX3Q2PDKQETLXWYEWMKPQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.551135548 (546h51m55.422218594s) [noise=true , address=45.91.116.9:4262 , piece_id=6JXAFCBODMHO5OYB5WGZNB762V5JUL32RXMFYHP6E3DCTWSSQYFA, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.551161338 (546h51m55.422244383s) [seg_pos={0 0}, upload=479529]: Segment reached optimal threshold of 13 pieces. Dec 28 10:20:55.551510112 (546h51m55.422593147s) [noise=true , address=77.90.31.11:4556 , piece_id=ZHXRF3QCUU4FV7P3OJ7TLFEFLMR7F26K4UHG4OFWYIZFTJOJGD6A, seg_pos={0 0}, upload=479529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: write tcp 188.166.225.170:33006->77.90.31.11:4556: use of closed network connection; write tcp 188.166.225.170:33006->77.90.31.11:4556: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33006->77.90.31.11:4556: use of closed network connection Dec 28 10:20:55.551527014 (546h51m55.422610059s) [noise=true , address=77.90.31.11:4556 , piece_id=ZHXRF3QCUU4FV7P3OJ7TLFEFLMR7F26K4UHG4OFWYIZFTJOJGD6A, seg_pos={0 0}, upload=479529]: Long tail cancellation due to optimal threshold Dec 28 10:20:55.551578462 (546h51m55.422661507s) [noise=true , address=45.91.116.7:4009 , piece_id=FWZA5D3EYV77RAPKAZOBR4Z4OOSNWXHXXE4WJ4QNEETMZPJHYMAQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 28 10:20:55.551591066 (546h51m55.422674121s) [noise=true , address=45.91.116.7:4009 , piece_id=FWZA5D3EYV77RAPKAZOBR4Z4OOSNWXHXXE4WJ4QNEETMZPJHYMAQ, seg_pos={0 0}, upload=479529]: Long tail cancellation due to optimal threshold Dec 28 10:20:55.551835822 (546h51m55.422918867s) [noise=true , address=77.90.31.15:4697 , piece_id=3FQIGY4VHPZIQ7ONVHGW7Z4UIV7EAECOAPGQEUVRYHXOLEOLHCTA, seg_pos={0 0}, upload=479529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:58702->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:58702->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58702->77.90.31.15:4697: use of closed network connection Dec 28 10:20:55.551845641 (546h51m55.422928676s) [noise=true , address=77.90.31.15:4697 , piece_id=3FQIGY4VHPZIQ7ONVHGW7Z4UIV7EAECOAPGQEUVRYHXOLEOLHCTA, seg_pos={0 0}, upload=479529]: Long tail cancellation due to optimal threshold Dec 28 10:20:55.551895075 (546h51m55.42297812s) [seg_pos={0 0}, upload=479529]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 10:20:55.552135894 (546h51m55.423218929s) [upload=479529]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 10:20:55.552144850 (546h51m55.423227885s) [upload=479529]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 10:20:55.854005394 (546h51m55.72508845s) [upload=479529]: Done uploading segments... Dec 28 10:20:55.854081920 (546h51m55.725164965s) [upload=479529]: Upload finished. err: <nil> Duration: 5.962338047s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [.X....XXXXX...X........X.....X..................X...........] duration: 3.694809968s trace: https://jaeger.dev.storj.io/trace/49dbd650238e8f85 FAIL (2025-12-28 10:28:43 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.694837891s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [X.X....XXXXX...X........X.....X..................X..........] duration: 4.060766814s trace: https://jaeger.dev.storj.io/trace/49dbd650238e8f85 FAIL (2025-12-28 10:28:17 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.060810607s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.......X.........X...................X..XX.....X..X.........] duration: 5.732544972s trace: https://jaeger.dev.storj.io/trace/37d347f90dc836a FAIL (2025-12-28 10:28:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== segments... Dec 28 10:20:52.978430940 (546h51m52.849513975s) [upload=479529]: Got next segment. Inline: false Dec 28 10:20:52.978465887 (546h51m52.849548922s) [upload=479529]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 10:20:52.978471527 (546h51m52.849554562s) [upload=479529]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 10:20:53.224037597 (546h51m53.095120632s) [seg_pos={0 0}, upload=479529]: Begin upload segment... Dec 28 10:20:53.224084497 (546h51m53.095167532s) [seg_pos={0 0}, upload=479529]: Done begin upload segment. Dec 28 10:20:53.224140062 (546h51m53.095223117s) [noise=true , address=77.90.31.11:4556 , piece_id=ZHXRF3QCUU4FV7P3OJ7TLFEFLMR7F26K4UHG4OFWYIZFTJOJGD6A, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.224148639 (546h51m53.095231685s) [noise=true , address=145.79.227.28:4725 , piece_id=BJ32VJWVH66R5T57UVSE76M227T3PND3IS4GYTVJCN62NRDFK2ZA, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.224718384 (546h51m53.095801419s) [noise=true , address=145.79.227.42:4123 , piece_id=CG5UCA4G3BNPI3BAFBTMOAGUZJEU7FW2MSNRYSHV77LPLR4CK5SA, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.225234898 (546h51m53.096317933s) [noise=true , address=145.79.227.3:4151 , piece_id=SUWYBG7TPG7BGBOS5G4UKXR2Z753Q4GJ2AN3M3WCRYPN3IKVEHQQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.225504882 (546h51m53.096587917s) [noise=true , address=145.79.227.21:4474 , piece_id=EEGOXLAYIZVCT4NF34CFLAVESLW2WPEOKJPVPRYIUSHKRIC5VR3Q, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.226212750 (546h51m53.097295785s) [noise=true , address=145.79.227.14:4840 , piece_id=YNOHADAVEJGWSG5VERF7HWN7GHHIPDWWECQ5HTPSS7HKC7DMVOWQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.226608573 (546h51m53.097691618s) [noise=true , address=145.79.227.6:4258 , piece_id=2DFJ2B7ETKXW76LASWPT4LGUAUV7I2OEV3N5JGOVPW74QVK6F4TQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.226767135 (546h51m53.09785018s) [noise=true , address=45.91.116.7:4009 , piece_id=FWZA5D3EYV77RAPKAZOBR4Z4OOSNWXHXXE4WJ4QNEETMZPJHYMAQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.227078098 (546h51m53.098161143s) [noise=true , address=45.91.116.9:4262 , piece_id=6JXAFCBODMHO5OYB5WGZNB762V5JUL32RXMFYHP6E3DCTWSSQYFA, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.227238392 (546h51m53.098321427s) [noise=true , address=145.79.227.16:4911 , piece_id=TZOOFWAQ33QB5B6MOUJ3XH7EB3IALWYGN43QUGNXOG6EXOJTB4AA, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.227466998 (546h51m53.098550033s) [noise=true , address=145.79.227.48:4301 , piece_id=NZ24TJ7I42YFMDOUF5SHORFQIHOKJBDILK3SGMGFHJ46OO6DGX5Q, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.227646309 (546h51m53.098729354s) [noise=true , address=77.90.31.15:4697 , piece_id=3FQIGY4VHPZIQ7ONVHGW7Z4UIV7EAECOAPGQEUVRYHXOLEOLHCTA, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.228003639 (546h51m53.099086684s) [noise=true , address=145.79.227.2:4115 , piece_id=XKLFZEWHLNY6AA5RVLWWPCLVUMWDU4YBJJQH2UFQ4U27YGVZGOXQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.228329440 (546h51m53.099412475s) [noise=true , address=145.79.227.32:4835 , piece_id=K5FO3OYDESYAUI3QUQ65WTEFJ2N77PPJL5X3FFNKAYU4X5PWKXBQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.228632276 (546h51m53.099715311s) [noise=true , address=45.91.116.8:4047 , piece_id=ATZ3XPE3S3AZSYXQZQWBDLBZQCVAMVIDX3Q2PDKQETLXWYEWMKPQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.228942016 (546h51m53.100025061s) [noise=true , address=145.79.227.36:4519 , piece_id=WVXG7E2MKVEYO4MHELTZ5HNRIXPJWLF2ZL4AU46X32W62SW5V3IQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:54.851414651 (546h51m54.722497686s) [upload=479529]: Next returned nil segment Dec 28 10:20:54.851429859 (546h51m54.722512894s) [upload=479529]: Waiting for error group managing segments... Dec 28 10:20:55.232471268 (546h51m55.103554303s) [noise=true , address=145.79.227.16:4911 , piece_id=TZOOFWAQ33QB5B6MOUJ3XH7EB3IALWYGN43QUGNXOG6EXOJTB4AA, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.232529519 (546h51m55.103612554s) [noise=true , address=145.79.227.2:4115 , piece_id=XKLFZEWHLNY6AA5RVLWWPCLVUMWDU4YBJJQH2UFQ4U27YGVZGOXQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.234914169 (546h51m55.105997204s) [noise=true , address=145.79.227.32:4835 , piece_id=K5FO3OYDESYAUI3QUQ65WTEFJ2N77PPJL5X3FFNKAYU4X5PWKXBQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.235465209 (546h51m55.106548244s) [noise=true , address=145.79.227.28:4725 , piece_id=BJ32VJWVH66R5T57UVSE76M227T3PND3IS4GYTVJCN62NRDFK2ZA, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.235615435 (546h51m55.10669847s) [noise=true , address=145.79.227.21:4474 , piece_id=EEGOXLAYIZVCT4NF34CFLAVESLW2WPEOKJPVPRYIUSHKRIC5VR3Q, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.236221559 (546h51m55.107304594s) [noise=true , address=145.79.227.6:4258 , piece_id=2DFJ2B7ETKXW76LASWPT4LGUAUV7I2OEV3N5JGOVPW74QVK6F4TQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.246296964 (546h51m55.117379999s) [noise=true , address=145.79.227.42:4123 , piece_id=CG5UCA4G3BNPI3BAFBTMOAGUZJEU7FW2MSNRYSHV77LPLR4CK5SA, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.246947282 (546h51m55.118030317s) [noise=true , address=145.79.227.3:4151 , piece_id=SUWYBG7TPG7BGBOS5G4UKXR2Z753Q4GJ2AN3M3WCRYPN3IKVEHQQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.251425650 (546h51m55.122508695s) [noise=true , address=145.79.227.14:4840 , piece_id=YNOHADAVEJGWSG5VERF7HWN7GHHIPDWWECQ5HTPSS7HKC7DMVOWQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.256324468 (546h51m55.127407513s) [noise=true , address=145.79.227.48:4301 , piece_id=NZ24TJ7I42YFMDOUF5SHORFQIHOKJBDILK3SGMGFHJ46OO6DGX5Q, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.259608812 (546h51m55.130691847s) [noise=true , address=145.79.227.36:4519 , piece_id=WVXG7E2MKVEYO4MHELTZ5HNRIXPJWLF2ZL4AU46X32W62SW5V3IQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.483932663 (546h51m55.355015698s) [noise=true , address=45.91.116.8:4047 , piece_id=ATZ3XPE3S3AZSYXQZQWBDLBZQCVAMVIDX3Q2PDKQETLXWYEWMKPQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.551135548 (546h51m55.422218594s) [noise=true , address=45.91.116.9:4262 , piece_id=6JXAFCBODMHO5OYB5WGZNB762V5JUL32RXMFYHP6E3DCTWSSQYFA, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.551161338 (546h51m55.422244383s) [seg_pos={0 0}, upload=479529]: Segment reached optimal threshold of 13 pieces. Dec 28 10:20:55.551510112 (546h51m55.422593147s) [noise=true , address=77.90.31.11:4556 , piece_id=ZHXRF3QCUU4FV7P3OJ7TLFEFLMR7F26K4UHG4OFWYIZFTJOJGD6A, seg_pos={0 0}, upload=479529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: write tcp 188.166.225.170:33006->77.90.31.11:4556: use of closed network connection; write tcp 188.166.225.170:33006->77.90.31.11:4556: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33006->77.90.31.11:4556: use of closed network connection Dec 28 10:20:55.551527014 (546h51m55.422610059s) [noise=true , address=77.90.31.11:4556 , piece_id=ZHXRF3QCUU4FV7P3OJ7TLFEFLMR7F26K4UHG4OFWYIZFTJOJGD6A, seg_pos={0 0}, upload=479529]: Long tail cancellation due to optimal threshold Dec 28 10:20:55.551578462 (546h51m55.422661507s) [noise=true , address=45.91.116.7:4009 , piece_id=FWZA5D3EYV77RAPKAZOBR4Z4OOSNWXHXXE4WJ4QNEETMZPJHYMAQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 28 10:20:55.551591066 (546h51m55.422674121s) [noise=true , address=45.91.116.7:4009 , piece_id=FWZA5D3EYV77RAPKAZOBR4Z4OOSNWXHXXE4WJ4QNEETMZPJHYMAQ, seg_pos={0 0}, upload=479529]: Long tail cancellation due to optimal threshold Dec 28 10:20:55.551835822 (546h51m55.422918867s) [noise=true , address=77.90.31.15:4697 , piece_id=3FQIGY4VHPZIQ7ONVHGW7Z4UIV7EAECOAPGQEUVRYHXOLEOLHCTA, seg_pos={0 0}, upload=479529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:58702->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:58702->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58702->77.90.31.15:4697: use of closed network connection Dec 28 10:20:55.551845641 (546h51m55.422928676s) [noise=true , address=77.90.31.15:4697 , piece_id=3FQIGY4VHPZIQ7ONVHGW7Z4UIV7EAECOAPGQEUVRYHXOLEOLHCTA, seg_pos={0 0}, upload=479529]: Long tail cancellation due to optimal threshold Dec 28 10:20:55.551895075 (546h51m55.42297812s) [seg_pos={0 0}, upload=479529]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 10:20:55.552135894 (546h51m55.423218929s) [upload=479529]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 10:20:55.552144850 (546h51m55.423227885s) [upload=479529]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 10:20:55.854005394 (546h51m55.72508845s) [upload=479529]: Done uploading segments... Dec 28 10:20:55.854081920 (546h51m55.725164965s) [upload=479529]: Upload finished. err: <nil> Duration: 5.732582404s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [.X.X....XXXXX...X........X.....X..................X.........] duration: 3.642689203s trace: https://jaeger.dev.storj.io/trace/49dbd650238e8f85 FAIL (2025-12-28 10:27:42 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.642717126s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [........X.........X...................X..XX.....X..X........] duration: 6.042706984s trace: https://jaeger.dev.storj.io/trace/37d347f90dc836a FAIL (2025-12-28 10:27:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== segments... Dec 28 10:20:52.978430940 (546h51m52.849513975s) [upload=479529]: Got next segment. Inline: false Dec 28 10:20:52.978465887 (546h51m52.849548922s) [upload=479529]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 10:20:52.978471527 (546h51m52.849554562s) [upload=479529]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 10:20:53.224037597 (546h51m53.095120632s) [seg_pos={0 0}, upload=479529]: Begin upload segment... Dec 28 10:20:53.224084497 (546h51m53.095167532s) [seg_pos={0 0}, upload=479529]: Done begin upload segment. Dec 28 10:20:53.224140062 (546h51m53.095223117s) [noise=true , address=77.90.31.11:4556 , piece_id=ZHXRF3QCUU4FV7P3OJ7TLFEFLMR7F26K4UHG4OFWYIZFTJOJGD6A, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.224148639 (546h51m53.095231685s) [noise=true , address=145.79.227.28:4725 , piece_id=BJ32VJWVH66R5T57UVSE76M227T3PND3IS4GYTVJCN62NRDFK2ZA, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.224718384 (546h51m53.095801419s) [noise=true , address=145.79.227.42:4123 , piece_id=CG5UCA4G3BNPI3BAFBTMOAGUZJEU7FW2MSNRYSHV77LPLR4CK5SA, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.225234898 (546h51m53.096317933s) [noise=true , address=145.79.227.3:4151 , piece_id=SUWYBG7TPG7BGBOS5G4UKXR2Z753Q4GJ2AN3M3WCRYPN3IKVEHQQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.225504882 (546h51m53.096587917s) [noise=true , address=145.79.227.21:4474 , piece_id=EEGOXLAYIZVCT4NF34CFLAVESLW2WPEOKJPVPRYIUSHKRIC5VR3Q, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.226212750 (546h51m53.097295785s) [noise=true , address=145.79.227.14:4840 , piece_id=YNOHADAVEJGWSG5VERF7HWN7GHHIPDWWECQ5HTPSS7HKC7DMVOWQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.226608573 (546h51m53.097691618s) [noise=true , address=145.79.227.6:4258 , piece_id=2DFJ2B7ETKXW76LASWPT4LGUAUV7I2OEV3N5JGOVPW74QVK6F4TQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.226767135 (546h51m53.09785018s) [noise=true , address=45.91.116.7:4009 , piece_id=FWZA5D3EYV77RAPKAZOBR4Z4OOSNWXHXXE4WJ4QNEETMZPJHYMAQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.227078098 (546h51m53.098161143s) [noise=true , address=45.91.116.9:4262 , piece_id=6JXAFCBODMHO5OYB5WGZNB762V5JUL32RXMFYHP6E3DCTWSSQYFA, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.227238392 (546h51m53.098321427s) [noise=true , address=145.79.227.16:4911 , piece_id=TZOOFWAQ33QB5B6MOUJ3XH7EB3IALWYGN43QUGNXOG6EXOJTB4AA, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.227466998 (546h51m53.098550033s) [noise=true , address=145.79.227.48:4301 , piece_id=NZ24TJ7I42YFMDOUF5SHORFQIHOKJBDILK3SGMGFHJ46OO6DGX5Q, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.227646309 (546h51m53.098729354s) [noise=true , address=77.90.31.15:4697 , piece_id=3FQIGY4VHPZIQ7ONVHGW7Z4UIV7EAECOAPGQEUVRYHXOLEOLHCTA, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.228003639 (546h51m53.099086684s) [noise=true , address=145.79.227.2:4115 , piece_id=XKLFZEWHLNY6AA5RVLWWPCLVUMWDU4YBJJQH2UFQ4U27YGVZGOXQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.228329440 (546h51m53.099412475s) [noise=true , address=145.79.227.32:4835 , piece_id=K5FO3OYDESYAUI3QUQ65WTEFJ2N77PPJL5X3FFNKAYU4X5PWKXBQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.228632276 (546h51m53.099715311s) [noise=true , address=45.91.116.8:4047 , piece_id=ATZ3XPE3S3AZSYXQZQWBDLBZQCVAMVIDX3Q2PDKQETLXWYEWMKPQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.228942016 (546h51m53.100025061s) [noise=true , address=145.79.227.36:4519 , piece_id=WVXG7E2MKVEYO4MHELTZ5HNRIXPJWLF2ZL4AU46X32W62SW5V3IQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:54.851414651 (546h51m54.722497686s) [upload=479529]: Next returned nil segment Dec 28 10:20:54.851429859 (546h51m54.722512894s) [upload=479529]: Waiting for error group managing segments... Dec 28 10:20:55.232471268 (546h51m55.103554303s) [noise=true , address=145.79.227.16:4911 , piece_id=TZOOFWAQ33QB5B6MOUJ3XH7EB3IALWYGN43QUGNXOG6EXOJTB4AA, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.232529519 (546h51m55.103612554s) [noise=true , address=145.79.227.2:4115 , piece_id=XKLFZEWHLNY6AA5RVLWWPCLVUMWDU4YBJJQH2UFQ4U27YGVZGOXQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.234914169 (546h51m55.105997204s) [noise=true , address=145.79.227.32:4835 , piece_id=K5FO3OYDESYAUI3QUQ65WTEFJ2N77PPJL5X3FFNKAYU4X5PWKXBQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.235465209 (546h51m55.106548244s) [noise=true , address=145.79.227.28:4725 , piece_id=BJ32VJWVH66R5T57UVSE76M227T3PND3IS4GYTVJCN62NRDFK2ZA, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.235615435 (546h51m55.10669847s) [noise=true , address=145.79.227.21:4474 , piece_id=EEGOXLAYIZVCT4NF34CFLAVESLW2WPEOKJPVPRYIUSHKRIC5VR3Q, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.236221559 (546h51m55.107304594s) [noise=true , address=145.79.227.6:4258 , piece_id=2DFJ2B7ETKXW76LASWPT4LGUAUV7I2OEV3N5JGOVPW74QVK6F4TQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.246296964 (546h51m55.117379999s) [noise=true , address=145.79.227.42:4123 , piece_id=CG5UCA4G3BNPI3BAFBTMOAGUZJEU7FW2MSNRYSHV77LPLR4CK5SA, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.246947282 (546h51m55.118030317s) [noise=true , address=145.79.227.3:4151 , piece_id=SUWYBG7TPG7BGBOS5G4UKXR2Z753Q4GJ2AN3M3WCRYPN3IKVEHQQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.251425650 (546h51m55.122508695s) [noise=true , address=145.79.227.14:4840 , piece_id=YNOHADAVEJGWSG5VERF7HWN7GHHIPDWWECQ5HTPSS7HKC7DMVOWQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.256324468 (546h51m55.127407513s) [noise=true , address=145.79.227.48:4301 , piece_id=NZ24TJ7I42YFMDOUF5SHORFQIHOKJBDILK3SGMGFHJ46OO6DGX5Q, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.259608812 (546h51m55.130691847s) [noise=true , address=145.79.227.36:4519 , piece_id=WVXG7E2MKVEYO4MHELTZ5HNRIXPJWLF2ZL4AU46X32W62SW5V3IQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.483932663 (546h51m55.355015698s) [noise=true , address=45.91.116.8:4047 , piece_id=ATZ3XPE3S3AZSYXQZQWBDLBZQCVAMVIDX3Q2PDKQETLXWYEWMKPQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.551135548 (546h51m55.422218594s) [noise=true , address=45.91.116.9:4262 , piece_id=6JXAFCBODMHO5OYB5WGZNB762V5JUL32RXMFYHP6E3DCTWSSQYFA, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.551161338 (546h51m55.422244383s) [seg_pos={0 0}, upload=479529]: Segment reached optimal threshold of 13 pieces. Dec 28 10:20:55.551510112 (546h51m55.422593147s) [noise=true , address=77.90.31.11:4556 , piece_id=ZHXRF3QCUU4FV7P3OJ7TLFEFLMR7F26K4UHG4OFWYIZFTJOJGD6A, seg_pos={0 0}, upload=479529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: write tcp 188.166.225.170:33006->77.90.31.11:4556: use of closed network connection; write tcp 188.166.225.170:33006->77.90.31.11:4556: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33006->77.90.31.11:4556: use of closed network connection Dec 28 10:20:55.551527014 (546h51m55.422610059s) [noise=true , address=77.90.31.11:4556 , piece_id=ZHXRF3QCUU4FV7P3OJ7TLFEFLMR7F26K4UHG4OFWYIZFTJOJGD6A, seg_pos={0 0}, upload=479529]: Long tail cancellation due to optimal threshold Dec 28 10:20:55.551578462 (546h51m55.422661507s) [noise=true , address=45.91.116.7:4009 , piece_id=FWZA5D3EYV77RAPKAZOBR4Z4OOSNWXHXXE4WJ4QNEETMZPJHYMAQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 28 10:20:55.551591066 (546h51m55.422674121s) [noise=true , address=45.91.116.7:4009 , piece_id=FWZA5D3EYV77RAPKAZOBR4Z4OOSNWXHXXE4WJ4QNEETMZPJHYMAQ, seg_pos={0 0}, upload=479529]: Long tail cancellation due to optimal threshold Dec 28 10:20:55.551835822 (546h51m55.422918867s) [noise=true , address=77.90.31.15:4697 , piece_id=3FQIGY4VHPZIQ7ONVHGW7Z4UIV7EAECOAPGQEUVRYHXOLEOLHCTA, seg_pos={0 0}, upload=479529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:58702->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:58702->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58702->77.90.31.15:4697: use of closed network connection Dec 28 10:20:55.551845641 (546h51m55.422928676s) [noise=true , address=77.90.31.15:4697 , piece_id=3FQIGY4VHPZIQ7ONVHGW7Z4UIV7EAECOAPGQEUVRYHXOLEOLHCTA, seg_pos={0 0}, upload=479529]: Long tail cancellation due to optimal threshold Dec 28 10:20:55.551895075 (546h51m55.42297812s) [seg_pos={0 0}, upload=479529]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 10:20:55.552135894 (546h51m55.423218929s) [upload=479529]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 10:20:55.552144850 (546h51m55.423227885s) [upload=479529]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 10:20:55.854005394 (546h51m55.72508845s) [upload=479529]: Done uploading segments... Dec 28 10:20:55.854081920 (546h51m55.725164965s) [upload=479529]: Upload finished. err: <nil> Duration: 6.042734827s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [..X.X....XXXXX...X........X.....X..................X........] duration: 3.954679955s trace: https://jaeger.dev.storj.io/trace/49dbd650238e8f85 FAIL (2025-12-28 10:26:59 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.954705954s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.........X.........X...................X..XX.....X..X.......] duration: 5.807320389s trace: https://jaeger.dev.storj.io/trace/37d347f90dc836a FAIL (2025-12-28 10:26:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== segments... Dec 28 10:20:52.978430940 (546h51m52.849513975s) [upload=479529]: Got next segment. Inline: false Dec 28 10:20:52.978465887 (546h51m52.849548922s) [upload=479529]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 10:20:52.978471527 (546h51m52.849554562s) [upload=479529]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 10:20:53.224037597 (546h51m53.095120632s) [seg_pos={0 0}, upload=479529]: Begin upload segment... Dec 28 10:20:53.224084497 (546h51m53.095167532s) [seg_pos={0 0}, upload=479529]: Done begin upload segment. Dec 28 10:20:53.224140062 (546h51m53.095223117s) [noise=true , address=77.90.31.11:4556 , piece_id=ZHXRF3QCUU4FV7P3OJ7TLFEFLMR7F26K4UHG4OFWYIZFTJOJGD6A, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.224148639 (546h51m53.095231685s) [noise=true , address=145.79.227.28:4725 , piece_id=BJ32VJWVH66R5T57UVSE76M227T3PND3IS4GYTVJCN62NRDFK2ZA, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.224718384 (546h51m53.095801419s) [noise=true , address=145.79.227.42:4123 , piece_id=CG5UCA4G3BNPI3BAFBTMOAGUZJEU7FW2MSNRYSHV77LPLR4CK5SA, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.225234898 (546h51m53.096317933s) [noise=true , address=145.79.227.3:4151 , piece_id=SUWYBG7TPG7BGBOS5G4UKXR2Z753Q4GJ2AN3M3WCRYPN3IKVEHQQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.225504882 (546h51m53.096587917s) [noise=true , address=145.79.227.21:4474 , piece_id=EEGOXLAYIZVCT4NF34CFLAVESLW2WPEOKJPVPRYIUSHKRIC5VR3Q, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.226212750 (546h51m53.097295785s) [noise=true , address=145.79.227.14:4840 , piece_id=YNOHADAVEJGWSG5VERF7HWN7GHHIPDWWECQ5HTPSS7HKC7DMVOWQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.226608573 (546h51m53.097691618s) [noise=true , address=145.79.227.6:4258 , piece_id=2DFJ2B7ETKXW76LASWPT4LGUAUV7I2OEV3N5JGOVPW74QVK6F4TQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.226767135 (546h51m53.09785018s) [noise=true , address=45.91.116.7:4009 , piece_id=FWZA5D3EYV77RAPKAZOBR4Z4OOSNWXHXXE4WJ4QNEETMZPJHYMAQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.227078098 (546h51m53.098161143s) [noise=true , address=45.91.116.9:4262 , piece_id=6JXAFCBODMHO5OYB5WGZNB762V5JUL32RXMFYHP6E3DCTWSSQYFA, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.227238392 (546h51m53.098321427s) [noise=true , address=145.79.227.16:4911 , piece_id=TZOOFWAQ33QB5B6MOUJ3XH7EB3IALWYGN43QUGNXOG6EXOJTB4AA, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.227466998 (546h51m53.098550033s) [noise=true , address=145.79.227.48:4301 , piece_id=NZ24TJ7I42YFMDOUF5SHORFQIHOKJBDILK3SGMGFHJ46OO6DGX5Q, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.227646309 (546h51m53.098729354s) [noise=true , address=77.90.31.15:4697 , piece_id=3FQIGY4VHPZIQ7ONVHGW7Z4UIV7EAECOAPGQEUVRYHXOLEOLHCTA, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.228003639 (546h51m53.099086684s) [noise=true , address=145.79.227.2:4115 , piece_id=XKLFZEWHLNY6AA5RVLWWPCLVUMWDU4YBJJQH2UFQ4U27YGVZGOXQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.228329440 (546h51m53.099412475s) [noise=true , address=145.79.227.32:4835 , piece_id=K5FO3OYDESYAUI3QUQ65WTEFJ2N77PPJL5X3FFNKAYU4X5PWKXBQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.228632276 (546h51m53.099715311s) [noise=true , address=45.91.116.8:4047 , piece_id=ATZ3XPE3S3AZSYXQZQWBDLBZQCVAMVIDX3Q2PDKQETLXWYEWMKPQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.228942016 (546h51m53.100025061s) [noise=true , address=145.79.227.36:4519 , piece_id=WVXG7E2MKVEYO4MHELTZ5HNRIXPJWLF2ZL4AU46X32W62SW5V3IQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:54.851414651 (546h51m54.722497686s) [upload=479529]: Next returned nil segment Dec 28 10:20:54.851429859 (546h51m54.722512894s) [upload=479529]: Waiting for error group managing segments... Dec 28 10:20:55.232471268 (546h51m55.103554303s) [noise=true , address=145.79.227.16:4911 , piece_id=TZOOFWAQ33QB5B6MOUJ3XH7EB3IALWYGN43QUGNXOG6EXOJTB4AA, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.232529519 (546h51m55.103612554s) [noise=true , address=145.79.227.2:4115 , piece_id=XKLFZEWHLNY6AA5RVLWWPCLVUMWDU4YBJJQH2UFQ4U27YGVZGOXQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.234914169 (546h51m55.105997204s) [noise=true , address=145.79.227.32:4835 , piece_id=K5FO3OYDESYAUI3QUQ65WTEFJ2N77PPJL5X3FFNKAYU4X5PWKXBQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.235465209 (546h51m55.106548244s) [noise=true , address=145.79.227.28:4725 , piece_id=BJ32VJWVH66R5T57UVSE76M227T3PND3IS4GYTVJCN62NRDFK2ZA, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.235615435 (546h51m55.10669847s) [noise=true , address=145.79.227.21:4474 , piece_id=EEGOXLAYIZVCT4NF34CFLAVESLW2WPEOKJPVPRYIUSHKRIC5VR3Q, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.236221559 (546h51m55.107304594s) [noise=true , address=145.79.227.6:4258 , piece_id=2DFJ2B7ETKXW76LASWPT4LGUAUV7I2OEV3N5JGOVPW74QVK6F4TQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.246296964 (546h51m55.117379999s) [noise=true , address=145.79.227.42:4123 , piece_id=CG5UCA4G3BNPI3BAFBTMOAGUZJEU7FW2MSNRYSHV77LPLR4CK5SA, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.246947282 (546h51m55.118030317s) [noise=true , address=145.79.227.3:4151 , piece_id=SUWYBG7TPG7BGBOS5G4UKXR2Z753Q4GJ2AN3M3WCRYPN3IKVEHQQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.251425650 (546h51m55.122508695s) [noise=true , address=145.79.227.14:4840 , piece_id=YNOHADAVEJGWSG5VERF7HWN7GHHIPDWWECQ5HTPSS7HKC7DMVOWQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.256324468 (546h51m55.127407513s) [noise=true , address=145.79.227.48:4301 , piece_id=NZ24TJ7I42YFMDOUF5SHORFQIHOKJBDILK3SGMGFHJ46OO6DGX5Q, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.259608812 (546h51m55.130691847s) [noise=true , address=145.79.227.36:4519 , piece_id=WVXG7E2MKVEYO4MHELTZ5HNRIXPJWLF2ZL4AU46X32W62SW5V3IQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.483932663 (546h51m55.355015698s) [noise=true , address=45.91.116.8:4047 , piece_id=ATZ3XPE3S3AZSYXQZQWBDLBZQCVAMVIDX3Q2PDKQETLXWYEWMKPQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.551135548 (546h51m55.422218594s) [noise=true , address=45.91.116.9:4262 , piece_id=6JXAFCBODMHO5OYB5WGZNB762V5JUL32RXMFYHP6E3DCTWSSQYFA, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.551161338 (546h51m55.422244383s) [seg_pos={0 0}, upload=479529]: Segment reached optimal threshold of 13 pieces. Dec 28 10:20:55.551510112 (546h51m55.422593147s) [noise=true , address=77.90.31.11:4556 , piece_id=ZHXRF3QCUU4FV7P3OJ7TLFEFLMR7F26K4UHG4OFWYIZFTJOJGD6A, seg_pos={0 0}, upload=479529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: write tcp 188.166.225.170:33006->77.90.31.11:4556: use of closed network connection; write tcp 188.166.225.170:33006->77.90.31.11:4556: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33006->77.90.31.11:4556: use of closed network connection Dec 28 10:20:55.551527014 (546h51m55.422610059s) [noise=true , address=77.90.31.11:4556 , piece_id=ZHXRF3QCUU4FV7P3OJ7TLFEFLMR7F26K4UHG4OFWYIZFTJOJGD6A, seg_pos={0 0}, upload=479529]: Long tail cancellation due to optimal threshold Dec 28 10:20:55.551578462 (546h51m55.422661507s) [noise=true , address=45.91.116.7:4009 , piece_id=FWZA5D3EYV77RAPKAZOBR4Z4OOSNWXHXXE4WJ4QNEETMZPJHYMAQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 28 10:20:55.551591066 (546h51m55.422674121s) [noise=true , address=45.91.116.7:4009 , piece_id=FWZA5D3EYV77RAPKAZOBR4Z4OOSNWXHXXE4WJ4QNEETMZPJHYMAQ, seg_pos={0 0}, upload=479529]: Long tail cancellation due to optimal threshold Dec 28 10:20:55.551835822 (546h51m55.422918867s) [noise=true , address=77.90.31.15:4697 , piece_id=3FQIGY4VHPZIQ7ONVHGW7Z4UIV7EAECOAPGQEUVRYHXOLEOLHCTA, seg_pos={0 0}, upload=479529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:58702->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:58702->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58702->77.90.31.15:4697: use of closed network connection Dec 28 10:20:55.551845641 (546h51m55.422928676s) [noise=true , address=77.90.31.15:4697 , piece_id=3FQIGY4VHPZIQ7ONVHGW7Z4UIV7EAECOAPGQEUVRYHXOLEOLHCTA, seg_pos={0 0}, upload=479529]: Long tail cancellation due to optimal threshold Dec 28 10:20:55.551895075 (546h51m55.42297812s) [seg_pos={0 0}, upload=479529]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 10:20:55.552135894 (546h51m55.423218929s) [upload=479529]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 10:20:55.552144850 (546h51m55.423227885s) [upload=479529]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 10:20:55.854005394 (546h51m55.72508845s) [upload=479529]: Done uploading segments... Dec 28 10:20:55.854081920 (546h51m55.725164965s) [upload=479529]: Upload finished. err: <nil> Duration: 5.807352078s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [...X.X....XXXXX...X........X.....X..................X.......] duration: 3.577011523s trace: https://jaeger.dev.storj.io/trace/49dbd650238e8f85 FAIL (2025-12-28 10:26:01 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.577033875s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [X.........X.........X...................X..XX.....X..X......] duration: 5.693173504s trace: https://jaeger.dev.storj.io/trace/37d347f90dc836a FAIL (2025-12-28 10:25:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== segments... Dec 28 10:20:52.978430940 (546h51m52.849513975s) [upload=479529]: Got next segment. Inline: false Dec 28 10:20:52.978465887 (546h51m52.849548922s) [upload=479529]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 10:20:52.978471527 (546h51m52.849554562s) [upload=479529]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 10:20:53.224037597 (546h51m53.095120632s) [seg_pos={0 0}, upload=479529]: Begin upload segment... Dec 28 10:20:53.224084497 (546h51m53.095167532s) [seg_pos={0 0}, upload=479529]: Done begin upload segment. Dec 28 10:20:53.224140062 (546h51m53.095223117s) [noise=true , address=77.90.31.11:4556 , piece_id=ZHXRF3QCUU4FV7P3OJ7TLFEFLMR7F26K4UHG4OFWYIZFTJOJGD6A, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.224148639 (546h51m53.095231685s) [noise=true , address=145.79.227.28:4725 , piece_id=BJ32VJWVH66R5T57UVSE76M227T3PND3IS4GYTVJCN62NRDFK2ZA, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.224718384 (546h51m53.095801419s) [noise=true , address=145.79.227.42:4123 , piece_id=CG5UCA4G3BNPI3BAFBTMOAGUZJEU7FW2MSNRYSHV77LPLR4CK5SA, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.225234898 (546h51m53.096317933s) [noise=true , address=145.79.227.3:4151 , piece_id=SUWYBG7TPG7BGBOS5G4UKXR2Z753Q4GJ2AN3M3WCRYPN3IKVEHQQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.225504882 (546h51m53.096587917s) [noise=true , address=145.79.227.21:4474 , piece_id=EEGOXLAYIZVCT4NF34CFLAVESLW2WPEOKJPVPRYIUSHKRIC5VR3Q, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.226212750 (546h51m53.097295785s) [noise=true , address=145.79.227.14:4840 , piece_id=YNOHADAVEJGWSG5VERF7HWN7GHHIPDWWECQ5HTPSS7HKC7DMVOWQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.226608573 (546h51m53.097691618s) [noise=true , address=145.79.227.6:4258 , piece_id=2DFJ2B7ETKXW76LASWPT4LGUAUV7I2OEV3N5JGOVPW74QVK6F4TQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.226767135 (546h51m53.09785018s) [noise=true , address=45.91.116.7:4009 , piece_id=FWZA5D3EYV77RAPKAZOBR4Z4OOSNWXHXXE4WJ4QNEETMZPJHYMAQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.227078098 (546h51m53.098161143s) [noise=true , address=45.91.116.9:4262 , piece_id=6JXAFCBODMHO5OYB5WGZNB762V5JUL32RXMFYHP6E3DCTWSSQYFA, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.227238392 (546h51m53.098321427s) [noise=true , address=145.79.227.16:4911 , piece_id=TZOOFWAQ33QB5B6MOUJ3XH7EB3IALWYGN43QUGNXOG6EXOJTB4AA, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.227466998 (546h51m53.098550033s) [noise=true , address=145.79.227.48:4301 , piece_id=NZ24TJ7I42YFMDOUF5SHORFQIHOKJBDILK3SGMGFHJ46OO6DGX5Q, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.227646309 (546h51m53.098729354s) [noise=true , address=77.90.31.15:4697 , piece_id=3FQIGY4VHPZIQ7ONVHGW7Z4UIV7EAECOAPGQEUVRYHXOLEOLHCTA, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.228003639 (546h51m53.099086684s) [noise=true , address=145.79.227.2:4115 , piece_id=XKLFZEWHLNY6AA5RVLWWPCLVUMWDU4YBJJQH2UFQ4U27YGVZGOXQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.228329440 (546h51m53.099412475s) [noise=true , address=145.79.227.32:4835 , piece_id=K5FO3OYDESYAUI3QUQ65WTEFJ2N77PPJL5X3FFNKAYU4X5PWKXBQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.228632276 (546h51m53.099715311s) [noise=true , address=45.91.116.8:4047 , piece_id=ATZ3XPE3S3AZSYXQZQWBDLBZQCVAMVIDX3Q2PDKQETLXWYEWMKPQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.228942016 (546h51m53.100025061s) [noise=true , address=145.79.227.36:4519 , piece_id=WVXG7E2MKVEYO4MHELTZ5HNRIXPJWLF2ZL4AU46X32W62SW5V3IQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:54.851414651 (546h51m54.722497686s) [upload=479529]: Next returned nil segment Dec 28 10:20:54.851429859 (546h51m54.722512894s) [upload=479529]: Waiting for error group managing segments... Dec 28 10:20:55.232471268 (546h51m55.103554303s) [noise=true , address=145.79.227.16:4911 , piece_id=TZOOFWAQ33QB5B6MOUJ3XH7EB3IALWYGN43QUGNXOG6EXOJTB4AA, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.232529519 (546h51m55.103612554s) [noise=true , address=145.79.227.2:4115 , piece_id=XKLFZEWHLNY6AA5RVLWWPCLVUMWDU4YBJJQH2UFQ4U27YGVZGOXQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.234914169 (546h51m55.105997204s) [noise=true , address=145.79.227.32:4835 , piece_id=K5FO3OYDESYAUI3QUQ65WTEFJ2N77PPJL5X3FFNKAYU4X5PWKXBQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.235465209 (546h51m55.106548244s) [noise=true , address=145.79.227.28:4725 , piece_id=BJ32VJWVH66R5T57UVSE76M227T3PND3IS4GYTVJCN62NRDFK2ZA, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.235615435 (546h51m55.10669847s) [noise=true , address=145.79.227.21:4474 , piece_id=EEGOXLAYIZVCT4NF34CFLAVESLW2WPEOKJPVPRYIUSHKRIC5VR3Q, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.236221559 (546h51m55.107304594s) [noise=true , address=145.79.227.6:4258 , piece_id=2DFJ2B7ETKXW76LASWPT4LGUAUV7I2OEV3N5JGOVPW74QVK6F4TQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.246296964 (546h51m55.117379999s) [noise=true , address=145.79.227.42:4123 , piece_id=CG5UCA4G3BNPI3BAFBTMOAGUZJEU7FW2MSNRYSHV77LPLR4CK5SA, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.246947282 (546h51m55.118030317s) [noise=true , address=145.79.227.3:4151 , piece_id=SUWYBG7TPG7BGBOS5G4UKXR2Z753Q4GJ2AN3M3WCRYPN3IKVEHQQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.251425650 (546h51m55.122508695s) [noise=true , address=145.79.227.14:4840 , piece_id=YNOHADAVEJGWSG5VERF7HWN7GHHIPDWWECQ5HTPSS7HKC7DMVOWQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.256324468 (546h51m55.127407513s) [noise=true , address=145.79.227.48:4301 , piece_id=NZ24TJ7I42YFMDOUF5SHORFQIHOKJBDILK3SGMGFHJ46OO6DGX5Q, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.259608812 (546h51m55.130691847s) [noise=true , address=145.79.227.36:4519 , piece_id=WVXG7E2MKVEYO4MHELTZ5HNRIXPJWLF2ZL4AU46X32W62SW5V3IQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.483932663 (546h51m55.355015698s) [noise=true , address=45.91.116.8:4047 , piece_id=ATZ3XPE3S3AZSYXQZQWBDLBZQCVAMVIDX3Q2PDKQETLXWYEWMKPQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.551135548 (546h51m55.422218594s) [noise=true , address=45.91.116.9:4262 , piece_id=6JXAFCBODMHO5OYB5WGZNB762V5JUL32RXMFYHP6E3DCTWSSQYFA, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.551161338 (546h51m55.422244383s) [seg_pos={0 0}, upload=479529]: Segment reached optimal threshold of 13 pieces. Dec 28 10:20:55.551510112 (546h51m55.422593147s) [noise=true , address=77.90.31.11:4556 , piece_id=ZHXRF3QCUU4FV7P3OJ7TLFEFLMR7F26K4UHG4OFWYIZFTJOJGD6A, seg_pos={0 0}, upload=479529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: write tcp 188.166.225.170:33006->77.90.31.11:4556: use of closed network connection; write tcp 188.166.225.170:33006->77.90.31.11:4556: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33006->77.90.31.11:4556: use of closed network connection Dec 28 10:20:55.551527014 (546h51m55.422610059s) [noise=true , address=77.90.31.11:4556 , piece_id=ZHXRF3QCUU4FV7P3OJ7TLFEFLMR7F26K4UHG4OFWYIZFTJOJGD6A, seg_pos={0 0}, upload=479529]: Long tail cancellation due to optimal threshold Dec 28 10:20:55.551578462 (546h51m55.422661507s) [noise=true , address=45.91.116.7:4009 , piece_id=FWZA5D3EYV77RAPKAZOBR4Z4OOSNWXHXXE4WJ4QNEETMZPJHYMAQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 28 10:20:55.551591066 (546h51m55.422674121s) [noise=true , address=45.91.116.7:4009 , piece_id=FWZA5D3EYV77RAPKAZOBR4Z4OOSNWXHXXE4WJ4QNEETMZPJHYMAQ, seg_pos={0 0}, upload=479529]: Long tail cancellation due to optimal threshold Dec 28 10:20:55.551835822 (546h51m55.422918867s) [noise=true , address=77.90.31.15:4697 , piece_id=3FQIGY4VHPZIQ7ONVHGW7Z4UIV7EAECOAPGQEUVRYHXOLEOLHCTA, seg_pos={0 0}, upload=479529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:58702->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:58702->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58702->77.90.31.15:4697: use of closed network connection Dec 28 10:20:55.551845641 (546h51m55.422928676s) [noise=true , address=77.90.31.15:4697 , piece_id=3FQIGY4VHPZIQ7ONVHGW7Z4UIV7EAECOAPGQEUVRYHXOLEOLHCTA, seg_pos={0 0}, upload=479529]: Long tail cancellation due to optimal threshold Dec 28 10:20:55.551895075 (546h51m55.42297812s) [seg_pos={0 0}, upload=479529]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 10:20:55.552135894 (546h51m55.423218929s) [upload=479529]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 10:20:55.552144850 (546h51m55.423227885s) [upload=479529]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 10:20:55.854005394 (546h51m55.72508845s) [upload=479529]: Done uploading segments... Dec 28 10:20:55.854081920 (546h51m55.725164965s) [upload=479529]: Upload finished. err: <nil> Duration: 5.693200675s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [X...X.X....XXXXX...X........X.....X..................X......] duration: 5.000566401s trace: https://jaeger.dev.storj.io/trace/49dbd650238e8f85 FAIL (2025-12-28 10:25:24 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.0005979s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.X.........X.........X...................X..XX.....X..X.....] duration: 6.064192947s trace: https://jaeger.dev.storj.io/trace/37d347f90dc836a FAIL (2025-12-28 10:24:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== segments... Dec 28 10:20:52.978430940 (546h51m52.849513975s) [upload=479529]: Got next segment. Inline: false Dec 28 10:20:52.978465887 (546h51m52.849548922s) [upload=479529]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 10:20:52.978471527 (546h51m52.849554562s) [upload=479529]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 10:20:53.224037597 (546h51m53.095120632s) [seg_pos={0 0}, upload=479529]: Begin upload segment... Dec 28 10:20:53.224084497 (546h51m53.095167532s) [seg_pos={0 0}, upload=479529]: Done begin upload segment. Dec 28 10:20:53.224140062 (546h51m53.095223117s) [noise=true , address=77.90.31.11:4556 , piece_id=ZHXRF3QCUU4FV7P3OJ7TLFEFLMR7F26K4UHG4OFWYIZFTJOJGD6A, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.224148639 (546h51m53.095231685s) [noise=true , address=145.79.227.28:4725 , piece_id=BJ32VJWVH66R5T57UVSE76M227T3PND3IS4GYTVJCN62NRDFK2ZA, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.224718384 (546h51m53.095801419s) [noise=true , address=145.79.227.42:4123 , piece_id=CG5UCA4G3BNPI3BAFBTMOAGUZJEU7FW2MSNRYSHV77LPLR4CK5SA, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.225234898 (546h51m53.096317933s) [noise=true , address=145.79.227.3:4151 , piece_id=SUWYBG7TPG7BGBOS5G4UKXR2Z753Q4GJ2AN3M3WCRYPN3IKVEHQQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.225504882 (546h51m53.096587917s) [noise=true , address=145.79.227.21:4474 , piece_id=EEGOXLAYIZVCT4NF34CFLAVESLW2WPEOKJPVPRYIUSHKRIC5VR3Q, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.226212750 (546h51m53.097295785s) [noise=true , address=145.79.227.14:4840 , piece_id=YNOHADAVEJGWSG5VERF7HWN7GHHIPDWWECQ5HTPSS7HKC7DMVOWQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.226608573 (546h51m53.097691618s) [noise=true , address=145.79.227.6:4258 , piece_id=2DFJ2B7ETKXW76LASWPT4LGUAUV7I2OEV3N5JGOVPW74QVK6F4TQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.226767135 (546h51m53.09785018s) [noise=true , address=45.91.116.7:4009 , piece_id=FWZA5D3EYV77RAPKAZOBR4Z4OOSNWXHXXE4WJ4QNEETMZPJHYMAQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.227078098 (546h51m53.098161143s) [noise=true , address=45.91.116.9:4262 , piece_id=6JXAFCBODMHO5OYB5WGZNB762V5JUL32RXMFYHP6E3DCTWSSQYFA, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.227238392 (546h51m53.098321427s) [noise=true , address=145.79.227.16:4911 , piece_id=TZOOFWAQ33QB5B6MOUJ3XH7EB3IALWYGN43QUGNXOG6EXOJTB4AA, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.227466998 (546h51m53.098550033s) [noise=true , address=145.79.227.48:4301 , piece_id=NZ24TJ7I42YFMDOUF5SHORFQIHOKJBDILK3SGMGFHJ46OO6DGX5Q, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.227646309 (546h51m53.098729354s) [noise=true , address=77.90.31.15:4697 , piece_id=3FQIGY4VHPZIQ7ONVHGW7Z4UIV7EAECOAPGQEUVRYHXOLEOLHCTA, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.228003639 (546h51m53.099086684s) [noise=true , address=145.79.227.2:4115 , piece_id=XKLFZEWHLNY6AA5RVLWWPCLVUMWDU4YBJJQH2UFQ4U27YGVZGOXQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.228329440 (546h51m53.099412475s) [noise=true , address=145.79.227.32:4835 , piece_id=K5FO3OYDESYAUI3QUQ65WTEFJ2N77PPJL5X3FFNKAYU4X5PWKXBQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.228632276 (546h51m53.099715311s) [noise=true , address=45.91.116.8:4047 , piece_id=ATZ3XPE3S3AZSYXQZQWBDLBZQCVAMVIDX3Q2PDKQETLXWYEWMKPQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.228942016 (546h51m53.100025061s) [noise=true , address=145.79.227.36:4519 , piece_id=WVXG7E2MKVEYO4MHELTZ5HNRIXPJWLF2ZL4AU46X32W62SW5V3IQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:54.851414651 (546h51m54.722497686s) [upload=479529]: Next returned nil segment Dec 28 10:20:54.851429859 (546h51m54.722512894s) [upload=479529]: Waiting for error group managing segments... Dec 28 10:20:55.232471268 (546h51m55.103554303s) [noise=true , address=145.79.227.16:4911 , piece_id=TZOOFWAQ33QB5B6MOUJ3XH7EB3IALWYGN43QUGNXOG6EXOJTB4AA, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.232529519 (546h51m55.103612554s) [noise=true , address=145.79.227.2:4115 , piece_id=XKLFZEWHLNY6AA5RVLWWPCLVUMWDU4YBJJQH2UFQ4U27YGVZGOXQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.234914169 (546h51m55.105997204s) [noise=true , address=145.79.227.32:4835 , piece_id=K5FO3OYDESYAUI3QUQ65WTEFJ2N77PPJL5X3FFNKAYU4X5PWKXBQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.235465209 (546h51m55.106548244s) [noise=true , address=145.79.227.28:4725 , piece_id=BJ32VJWVH66R5T57UVSE76M227T3PND3IS4GYTVJCN62NRDFK2ZA, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.235615435 (546h51m55.10669847s) [noise=true , address=145.79.227.21:4474 , piece_id=EEGOXLAYIZVCT4NF34CFLAVESLW2WPEOKJPVPRYIUSHKRIC5VR3Q, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.236221559 (546h51m55.107304594s) [noise=true , address=145.79.227.6:4258 , piece_id=2DFJ2B7ETKXW76LASWPT4LGUAUV7I2OEV3N5JGOVPW74QVK6F4TQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.246296964 (546h51m55.117379999s) [noise=true , address=145.79.227.42:4123 , piece_id=CG5UCA4G3BNPI3BAFBTMOAGUZJEU7FW2MSNRYSHV77LPLR4CK5SA, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.246947282 (546h51m55.118030317s) [noise=true , address=145.79.227.3:4151 , piece_id=SUWYBG7TPG7BGBOS5G4UKXR2Z753Q4GJ2AN3M3WCRYPN3IKVEHQQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.251425650 (546h51m55.122508695s) [noise=true , address=145.79.227.14:4840 , piece_id=YNOHADAVEJGWSG5VERF7HWN7GHHIPDWWECQ5HTPSS7HKC7DMVOWQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.256324468 (546h51m55.127407513s) [noise=true , address=145.79.227.48:4301 , piece_id=NZ24TJ7I42YFMDOUF5SHORFQIHOKJBDILK3SGMGFHJ46OO6DGX5Q, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.259608812 (546h51m55.130691847s) [noise=true , address=145.79.227.36:4519 , piece_id=WVXG7E2MKVEYO4MHELTZ5HNRIXPJWLF2ZL4AU46X32W62SW5V3IQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.483932663 (546h51m55.355015698s) [noise=true , address=45.91.116.8:4047 , piece_id=ATZ3XPE3S3AZSYXQZQWBDLBZQCVAMVIDX3Q2PDKQETLXWYEWMKPQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.551135548 (546h51m55.422218594s) [noise=true , address=45.91.116.9:4262 , piece_id=6JXAFCBODMHO5OYB5WGZNB762V5JUL32RXMFYHP6E3DCTWSSQYFA, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.551161338 (546h51m55.422244383s) [seg_pos={0 0}, upload=479529]: Segment reached optimal threshold of 13 pieces. Dec 28 10:20:55.551510112 (546h51m55.422593147s) [noise=true , address=77.90.31.11:4556 , piece_id=ZHXRF3QCUU4FV7P3OJ7TLFEFLMR7F26K4UHG4OFWYIZFTJOJGD6A, seg_pos={0 0}, upload=479529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: write tcp 188.166.225.170:33006->77.90.31.11:4556: use of closed network connection; write tcp 188.166.225.170:33006->77.90.31.11:4556: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33006->77.90.31.11:4556: use of closed network connection Dec 28 10:20:55.551527014 (546h51m55.422610059s) [noise=true , address=77.90.31.11:4556 , piece_id=ZHXRF3QCUU4FV7P3OJ7TLFEFLMR7F26K4UHG4OFWYIZFTJOJGD6A, seg_pos={0 0}, upload=479529]: Long tail cancellation due to optimal threshold Dec 28 10:20:55.551578462 (546h51m55.422661507s) [noise=true , address=45.91.116.7:4009 , piece_id=FWZA5D3EYV77RAPKAZOBR4Z4OOSNWXHXXE4WJ4QNEETMZPJHYMAQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 28 10:20:55.551591066 (546h51m55.422674121s) [noise=true , address=45.91.116.7:4009 , piece_id=FWZA5D3EYV77RAPKAZOBR4Z4OOSNWXHXXE4WJ4QNEETMZPJHYMAQ, seg_pos={0 0}, upload=479529]: Long tail cancellation due to optimal threshold Dec 28 10:20:55.551835822 (546h51m55.422918867s) [noise=true , address=77.90.31.15:4697 , piece_id=3FQIGY4VHPZIQ7ONVHGW7Z4UIV7EAECOAPGQEUVRYHXOLEOLHCTA, seg_pos={0 0}, upload=479529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:58702->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:58702->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58702->77.90.31.15:4697: use of closed network connection Dec 28 10:20:55.551845641 (546h51m55.422928676s) [noise=true , address=77.90.31.15:4697 , piece_id=3FQIGY4VHPZIQ7ONVHGW7Z4UIV7EAECOAPGQEUVRYHXOLEOLHCTA, seg_pos={0 0}, upload=479529]: Long tail cancellation due to optimal threshold Dec 28 10:20:55.551895075 (546h51m55.42297812s) [seg_pos={0 0}, upload=479529]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 10:20:55.552135894 (546h51m55.423218929s) [upload=479529]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 10:20:55.552144850 (546h51m55.423227885s) [upload=479529]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 10:20:55.854005394 (546h51m55.72508845s) [upload=479529]: Done uploading segments... Dec 28 10:20:55.854081920 (546h51m55.725164965s) [upload=479529]: Upload finished. err: <nil> Duration: 6.06422018s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [.X...X.X....XXXXX...X........X.....X..................X.....] duration: 3.758628108s trace: https://jaeger.dev.storj.io/trace/49dbd650238e8f85 FAIL (2025-12-28 10:24:10 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.758660692s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [..X.........X.........X...................X..XX.....X..X....] duration: 6.291490529s trace: https://jaeger.dev.storj.io/trace/37d347f90dc836a FAIL (2025-12-28 10:23:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== segments... Dec 28 10:20:52.978430940 (546h51m52.849513975s) [upload=479529]: Got next segment. Inline: false Dec 28 10:20:52.978465887 (546h51m52.849548922s) [upload=479529]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 10:20:52.978471527 (546h51m52.849554562s) [upload=479529]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 10:20:53.224037597 (546h51m53.095120632s) [seg_pos={0 0}, upload=479529]: Begin upload segment... Dec 28 10:20:53.224084497 (546h51m53.095167532s) [seg_pos={0 0}, upload=479529]: Done begin upload segment. Dec 28 10:20:53.224140062 (546h51m53.095223117s) [noise=true , address=77.90.31.11:4556 , piece_id=ZHXRF3QCUU4FV7P3OJ7TLFEFLMR7F26K4UHG4OFWYIZFTJOJGD6A, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.224148639 (546h51m53.095231685s) [noise=true , address=145.79.227.28:4725 , piece_id=BJ32VJWVH66R5T57UVSE76M227T3PND3IS4GYTVJCN62NRDFK2ZA, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.224718384 (546h51m53.095801419s) [noise=true , address=145.79.227.42:4123 , piece_id=CG5UCA4G3BNPI3BAFBTMOAGUZJEU7FW2MSNRYSHV77LPLR4CK5SA, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.225234898 (546h51m53.096317933s) [noise=true , address=145.79.227.3:4151 , piece_id=SUWYBG7TPG7BGBOS5G4UKXR2Z753Q4GJ2AN3M3WCRYPN3IKVEHQQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.225504882 (546h51m53.096587917s) [noise=true , address=145.79.227.21:4474 , piece_id=EEGOXLAYIZVCT4NF34CFLAVESLW2WPEOKJPVPRYIUSHKRIC5VR3Q, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.226212750 (546h51m53.097295785s) [noise=true , address=145.79.227.14:4840 , piece_id=YNOHADAVEJGWSG5VERF7HWN7GHHIPDWWECQ5HTPSS7HKC7DMVOWQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.226608573 (546h51m53.097691618s) [noise=true , address=145.79.227.6:4258 , piece_id=2DFJ2B7ETKXW76LASWPT4LGUAUV7I2OEV3N5JGOVPW74QVK6F4TQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.226767135 (546h51m53.09785018s) [noise=true , address=45.91.116.7:4009 , piece_id=FWZA5D3EYV77RAPKAZOBR4Z4OOSNWXHXXE4WJ4QNEETMZPJHYMAQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.227078098 (546h51m53.098161143s) [noise=true , address=45.91.116.9:4262 , piece_id=6JXAFCBODMHO5OYB5WGZNB762V5JUL32RXMFYHP6E3DCTWSSQYFA, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.227238392 (546h51m53.098321427s) [noise=true , address=145.79.227.16:4911 , piece_id=TZOOFWAQ33QB5B6MOUJ3XH7EB3IALWYGN43QUGNXOG6EXOJTB4AA, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.227466998 (546h51m53.098550033s) [noise=true , address=145.79.227.48:4301 , piece_id=NZ24TJ7I42YFMDOUF5SHORFQIHOKJBDILK3SGMGFHJ46OO6DGX5Q, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.227646309 (546h51m53.098729354s) [noise=true , address=77.90.31.15:4697 , piece_id=3FQIGY4VHPZIQ7ONVHGW7Z4UIV7EAECOAPGQEUVRYHXOLEOLHCTA, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.228003639 (546h51m53.099086684s) [noise=true , address=145.79.227.2:4115 , piece_id=XKLFZEWHLNY6AA5RVLWWPCLVUMWDU4YBJJQH2UFQ4U27YGVZGOXQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.228329440 (546h51m53.099412475s) [noise=true , address=145.79.227.32:4835 , piece_id=K5FO3OYDESYAUI3QUQ65WTEFJ2N77PPJL5X3FFNKAYU4X5PWKXBQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.228632276 (546h51m53.099715311s) [noise=true , address=45.91.116.8:4047 , piece_id=ATZ3XPE3S3AZSYXQZQWBDLBZQCVAMVIDX3Q2PDKQETLXWYEWMKPQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.228942016 (546h51m53.100025061s) [noise=true , address=145.79.227.36:4519 , piece_id=WVXG7E2MKVEYO4MHELTZ5HNRIXPJWLF2ZL4AU46X32W62SW5V3IQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:54.851414651 (546h51m54.722497686s) [upload=479529]: Next returned nil segment Dec 28 10:20:54.851429859 (546h51m54.722512894s) [upload=479529]: Waiting for error group managing segments... Dec 28 10:20:55.232471268 (546h51m55.103554303s) [noise=true , address=145.79.227.16:4911 , piece_id=TZOOFWAQ33QB5B6MOUJ3XH7EB3IALWYGN43QUGNXOG6EXOJTB4AA, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.232529519 (546h51m55.103612554s) [noise=true , address=145.79.227.2:4115 , piece_id=XKLFZEWHLNY6AA5RVLWWPCLVUMWDU4YBJJQH2UFQ4U27YGVZGOXQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.234914169 (546h51m55.105997204s) [noise=true , address=145.79.227.32:4835 , piece_id=K5FO3OYDESYAUI3QUQ65WTEFJ2N77PPJL5X3FFNKAYU4X5PWKXBQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.235465209 (546h51m55.106548244s) [noise=true , address=145.79.227.28:4725 , piece_id=BJ32VJWVH66R5T57UVSE76M227T3PND3IS4GYTVJCN62NRDFK2ZA, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.235615435 (546h51m55.10669847s) [noise=true , address=145.79.227.21:4474 , piece_id=EEGOXLAYIZVCT4NF34CFLAVESLW2WPEOKJPVPRYIUSHKRIC5VR3Q, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.236221559 (546h51m55.107304594s) [noise=true , address=145.79.227.6:4258 , piece_id=2DFJ2B7ETKXW76LASWPT4LGUAUV7I2OEV3N5JGOVPW74QVK6F4TQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.246296964 (546h51m55.117379999s) [noise=true , address=145.79.227.42:4123 , piece_id=CG5UCA4G3BNPI3BAFBTMOAGUZJEU7FW2MSNRYSHV77LPLR4CK5SA, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.246947282 (546h51m55.118030317s) [noise=true , address=145.79.227.3:4151 , piece_id=SUWYBG7TPG7BGBOS5G4UKXR2Z753Q4GJ2AN3M3WCRYPN3IKVEHQQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.251425650 (546h51m55.122508695s) [noise=true , address=145.79.227.14:4840 , piece_id=YNOHADAVEJGWSG5VERF7HWN7GHHIPDWWECQ5HTPSS7HKC7DMVOWQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.256324468 (546h51m55.127407513s) [noise=true , address=145.79.227.48:4301 , piece_id=NZ24TJ7I42YFMDOUF5SHORFQIHOKJBDILK3SGMGFHJ46OO6DGX5Q, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.259608812 (546h51m55.130691847s) [noise=true , address=145.79.227.36:4519 , piece_id=WVXG7E2MKVEYO4MHELTZ5HNRIXPJWLF2ZL4AU46X32W62SW5V3IQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.483932663 (546h51m55.355015698s) [noise=true , address=45.91.116.8:4047 , piece_id=ATZ3XPE3S3AZSYXQZQWBDLBZQCVAMVIDX3Q2PDKQETLXWYEWMKPQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.551135548 (546h51m55.422218594s) [noise=true , address=45.91.116.9:4262 , piece_id=6JXAFCBODMHO5OYB5WGZNB762V5JUL32RXMFYHP6E3DCTWSSQYFA, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.551161338 (546h51m55.422244383s) [seg_pos={0 0}, upload=479529]: Segment reached optimal threshold of 13 pieces. Dec 28 10:20:55.551510112 (546h51m55.422593147s) [noise=true , address=77.90.31.11:4556 , piece_id=ZHXRF3QCUU4FV7P3OJ7TLFEFLMR7F26K4UHG4OFWYIZFTJOJGD6A, seg_pos={0 0}, upload=479529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: write tcp 188.166.225.170:33006->77.90.31.11:4556: use of closed network connection; write tcp 188.166.225.170:33006->77.90.31.11:4556: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33006->77.90.31.11:4556: use of closed network connection Dec 28 10:20:55.551527014 (546h51m55.422610059s) [noise=true , address=77.90.31.11:4556 , piece_id=ZHXRF3QCUU4FV7P3OJ7TLFEFLMR7F26K4UHG4OFWYIZFTJOJGD6A, seg_pos={0 0}, upload=479529]: Long tail cancellation due to optimal threshold Dec 28 10:20:55.551578462 (546h51m55.422661507s) [noise=true , address=45.91.116.7:4009 , piece_id=FWZA5D3EYV77RAPKAZOBR4Z4OOSNWXHXXE4WJ4QNEETMZPJHYMAQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 28 10:20:55.551591066 (546h51m55.422674121s) [noise=true , address=45.91.116.7:4009 , piece_id=FWZA5D3EYV77RAPKAZOBR4Z4OOSNWXHXXE4WJ4QNEETMZPJHYMAQ, seg_pos={0 0}, upload=479529]: Long tail cancellation due to optimal threshold Dec 28 10:20:55.551835822 (546h51m55.422918867s) [noise=true , address=77.90.31.15:4697 , piece_id=3FQIGY4VHPZIQ7ONVHGW7Z4UIV7EAECOAPGQEUVRYHXOLEOLHCTA, seg_pos={0 0}, upload=479529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:58702->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:58702->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58702->77.90.31.15:4697: use of closed network connection Dec 28 10:20:55.551845641 (546h51m55.422928676s) [noise=true , address=77.90.31.15:4697 , piece_id=3FQIGY4VHPZIQ7ONVHGW7Z4UIV7EAECOAPGQEUVRYHXOLEOLHCTA, seg_pos={0 0}, upload=479529]: Long tail cancellation due to optimal threshold Dec 28 10:20:55.551895075 (546h51m55.42297812s) [seg_pos={0 0}, upload=479529]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 10:20:55.552135894 (546h51m55.423218929s) [upload=479529]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 10:20:55.552144850 (546h51m55.423227885s) [upload=479529]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 10:20:55.854005394 (546h51m55.72508845s) [upload=479529]: Done uploading segments... Dec 28 10:20:55.854081920 (546h51m55.725164965s) [upload=479529]: Upload finished. err: <nil> Duration: 6.291517692s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [X..X.........X.........X...................X..XX.....X..X...] duration: 5.659040658s trace: https://jaeger.dev.storj.io/trace/37d347f90dc836a FAIL (2025-12-28 10:22:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== segments... Dec 28 10:20:52.978430940 (546h51m52.849513975s) [upload=479529]: Got next segment. Inline: false Dec 28 10:20:52.978465887 (546h51m52.849548922s) [upload=479529]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 10:20:52.978471527 (546h51m52.849554562s) [upload=479529]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 10:20:53.224037597 (546h51m53.095120632s) [seg_pos={0 0}, upload=479529]: Begin upload segment... Dec 28 10:20:53.224084497 (546h51m53.095167532s) [seg_pos={0 0}, upload=479529]: Done begin upload segment. Dec 28 10:20:53.224140062 (546h51m53.095223117s) [noise=true , address=77.90.31.11:4556 , piece_id=ZHXRF3QCUU4FV7P3OJ7TLFEFLMR7F26K4UHG4OFWYIZFTJOJGD6A, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.224148639 (546h51m53.095231685s) [noise=true , address=145.79.227.28:4725 , piece_id=BJ32VJWVH66R5T57UVSE76M227T3PND3IS4GYTVJCN62NRDFK2ZA, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.224718384 (546h51m53.095801419s) [noise=true , address=145.79.227.42:4123 , piece_id=CG5UCA4G3BNPI3BAFBTMOAGUZJEU7FW2MSNRYSHV77LPLR4CK5SA, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.225234898 (546h51m53.096317933s) [noise=true , address=145.79.227.3:4151 , piece_id=SUWYBG7TPG7BGBOS5G4UKXR2Z753Q4GJ2AN3M3WCRYPN3IKVEHQQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.225504882 (546h51m53.096587917s) [noise=true , address=145.79.227.21:4474 , piece_id=EEGOXLAYIZVCT4NF34CFLAVESLW2WPEOKJPVPRYIUSHKRIC5VR3Q, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.226212750 (546h51m53.097295785s) [noise=true , address=145.79.227.14:4840 , piece_id=YNOHADAVEJGWSG5VERF7HWN7GHHIPDWWECQ5HTPSS7HKC7DMVOWQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.226608573 (546h51m53.097691618s) [noise=true , address=145.79.227.6:4258 , piece_id=2DFJ2B7ETKXW76LASWPT4LGUAUV7I2OEV3N5JGOVPW74QVK6F4TQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.226767135 (546h51m53.09785018s) [noise=true , address=45.91.116.7:4009 , piece_id=FWZA5D3EYV77RAPKAZOBR4Z4OOSNWXHXXE4WJ4QNEETMZPJHYMAQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.227078098 (546h51m53.098161143s) [noise=true , address=45.91.116.9:4262 , piece_id=6JXAFCBODMHO5OYB5WGZNB762V5JUL32RXMFYHP6E3DCTWSSQYFA, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.227238392 (546h51m53.098321427s) [noise=true , address=145.79.227.16:4911 , piece_id=TZOOFWAQ33QB5B6MOUJ3XH7EB3IALWYGN43QUGNXOG6EXOJTB4AA, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.227466998 (546h51m53.098550033s) [noise=true , address=145.79.227.48:4301 , piece_id=NZ24TJ7I42YFMDOUF5SHORFQIHOKJBDILK3SGMGFHJ46OO6DGX5Q, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.227646309 (546h51m53.098729354s) [noise=true , address=77.90.31.15:4697 , piece_id=3FQIGY4VHPZIQ7ONVHGW7Z4UIV7EAECOAPGQEUVRYHXOLEOLHCTA, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.228003639 (546h51m53.099086684s) [noise=true , address=145.79.227.2:4115 , piece_id=XKLFZEWHLNY6AA5RVLWWPCLVUMWDU4YBJJQH2UFQ4U27YGVZGOXQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.228329440 (546h51m53.099412475s) [noise=true , address=145.79.227.32:4835 , piece_id=K5FO3OYDESYAUI3QUQ65WTEFJ2N77PPJL5X3FFNKAYU4X5PWKXBQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.228632276 (546h51m53.099715311s) [noise=true , address=45.91.116.8:4047 , piece_id=ATZ3XPE3S3AZSYXQZQWBDLBZQCVAMVIDX3Q2PDKQETLXWYEWMKPQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.228942016 (546h51m53.100025061s) [noise=true , address=145.79.227.36:4519 , piece_id=WVXG7E2MKVEYO4MHELTZ5HNRIXPJWLF2ZL4AU46X32W62SW5V3IQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:54.851414651 (546h51m54.722497686s) [upload=479529]: Next returned nil segment Dec 28 10:20:54.851429859 (546h51m54.722512894s) [upload=479529]: Waiting for error group managing segments... Dec 28 10:20:55.232471268 (546h51m55.103554303s) [noise=true , address=145.79.227.16:4911 , piece_id=TZOOFWAQ33QB5B6MOUJ3XH7EB3IALWYGN43QUGNXOG6EXOJTB4AA, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.232529519 (546h51m55.103612554s) [noise=true , address=145.79.227.2:4115 , piece_id=XKLFZEWHLNY6AA5RVLWWPCLVUMWDU4YBJJQH2UFQ4U27YGVZGOXQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.234914169 (546h51m55.105997204s) [noise=true , address=145.79.227.32:4835 , piece_id=K5FO3OYDESYAUI3QUQ65WTEFJ2N77PPJL5X3FFNKAYU4X5PWKXBQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.235465209 (546h51m55.106548244s) [noise=true , address=145.79.227.28:4725 , piece_id=BJ32VJWVH66R5T57UVSE76M227T3PND3IS4GYTVJCN62NRDFK2ZA, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.235615435 (546h51m55.10669847s) [noise=true , address=145.79.227.21:4474 , piece_id=EEGOXLAYIZVCT4NF34CFLAVESLW2WPEOKJPVPRYIUSHKRIC5VR3Q, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.236221559 (546h51m55.107304594s) [noise=true , address=145.79.227.6:4258 , piece_id=2DFJ2B7ETKXW76LASWPT4LGUAUV7I2OEV3N5JGOVPW74QVK6F4TQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.246296964 (546h51m55.117379999s) [noise=true , address=145.79.227.42:4123 , piece_id=CG5UCA4G3BNPI3BAFBTMOAGUZJEU7FW2MSNRYSHV77LPLR4CK5SA, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.246947282 (546h51m55.118030317s) [noise=true , address=145.79.227.3:4151 , piece_id=SUWYBG7TPG7BGBOS5G4UKXR2Z753Q4GJ2AN3M3WCRYPN3IKVEHQQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.251425650 (546h51m55.122508695s) [noise=true , address=145.79.227.14:4840 , piece_id=YNOHADAVEJGWSG5VERF7HWN7GHHIPDWWECQ5HTPSS7HKC7DMVOWQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.256324468 (546h51m55.127407513s) [noise=true , address=145.79.227.48:4301 , piece_id=NZ24TJ7I42YFMDOUF5SHORFQIHOKJBDILK3SGMGFHJ46OO6DGX5Q, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.259608812 (546h51m55.130691847s) [noise=true , address=145.79.227.36:4519 , piece_id=WVXG7E2MKVEYO4MHELTZ5HNRIXPJWLF2ZL4AU46X32W62SW5V3IQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.483932663 (546h51m55.355015698s) [noise=true , address=45.91.116.8:4047 , piece_id=ATZ3XPE3S3AZSYXQZQWBDLBZQCVAMVIDX3Q2PDKQETLXWYEWMKPQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.551135548 (546h51m55.422218594s) [noise=true , address=45.91.116.9:4262 , piece_id=6JXAFCBODMHO5OYB5WGZNB762V5JUL32RXMFYHP6E3DCTWSSQYFA, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.551161338 (546h51m55.422244383s) [seg_pos={0 0}, upload=479529]: Segment reached optimal threshold of 13 pieces. Dec 28 10:20:55.551510112 (546h51m55.422593147s) [noise=true , address=77.90.31.11:4556 , piece_id=ZHXRF3QCUU4FV7P3OJ7TLFEFLMR7F26K4UHG4OFWYIZFTJOJGD6A, seg_pos={0 0}, upload=479529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: write tcp 188.166.225.170:33006->77.90.31.11:4556: use of closed network connection; write tcp 188.166.225.170:33006->77.90.31.11:4556: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33006->77.90.31.11:4556: use of closed network connection Dec 28 10:20:55.551527014 (546h51m55.422610059s) [noise=true , address=77.90.31.11:4556 , piece_id=ZHXRF3QCUU4FV7P3OJ7TLFEFLMR7F26K4UHG4OFWYIZFTJOJGD6A, seg_pos={0 0}, upload=479529]: Long tail cancellation due to optimal threshold Dec 28 10:20:55.551578462 (546h51m55.422661507s) [noise=true , address=45.91.116.7:4009 , piece_id=FWZA5D3EYV77RAPKAZOBR4Z4OOSNWXHXXE4WJ4QNEETMZPJHYMAQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 28 10:20:55.551591066 (546h51m55.422674121s) [noise=true , address=45.91.116.7:4009 , piece_id=FWZA5D3EYV77RAPKAZOBR4Z4OOSNWXHXXE4WJ4QNEETMZPJHYMAQ, seg_pos={0 0}, upload=479529]: Long tail cancellation due to optimal threshold Dec 28 10:20:55.551835822 (546h51m55.422918867s) [noise=true , address=77.90.31.15:4697 , piece_id=3FQIGY4VHPZIQ7ONVHGW7Z4UIV7EAECOAPGQEUVRYHXOLEOLHCTA, seg_pos={0 0}, upload=479529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:58702->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:58702->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58702->77.90.31.15:4697: use of closed network connection Dec 28 10:20:55.551845641 (546h51m55.422928676s) [noise=true , address=77.90.31.15:4697 , piece_id=3FQIGY4VHPZIQ7ONVHGW7Z4UIV7EAECOAPGQEUVRYHXOLEOLHCTA, seg_pos={0 0}, upload=479529]: Long tail cancellation due to optimal threshold Dec 28 10:20:55.551895075 (546h51m55.42297812s) [seg_pos={0 0}, upload=479529]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 10:20:55.552135894 (546h51m55.423218929s) [upload=479529]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 10:20:55.552144850 (546h51m55.423227885s) [upload=479529]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 10:20:55.854005394 (546h51m55.72508845s) [upload=479529]: Done uploading segments... Dec 28 10:20:55.854081920 (546h51m55.725164965s) [upload=479529]: Upload finished. err: <nil> Duration: 5.65907854s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [X.X...X.X....XXXXX...X........X.....X..................X....] duration: 3.383932415s trace: https://jaeger.dev.storj.io/trace/49dbd650238e8f85 FAIL (2025-12-28 10:22:59 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.383955088s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [XX.X...X.X....XXXXX...X........X.....X..................X...] duration: 3.96605189s trace: https://jaeger.dev.storj.io/trace/49dbd650238e8f85 FAIL (2025-12-28 10:21:46 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.96607902s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.X..X.........X.........X...................X..XX.....X..X..] duration: 5.6355302s trace: https://jaeger.dev.storj.io/trace/37d347f90dc836a FAIL (2025-12-28 10:21:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== segments... Dec 28 10:20:52.978430940 (546h51m52.849513975s) [upload=479529]: Got next segment. Inline: false Dec 28 10:20:52.978465887 (546h51m52.849548922s) [upload=479529]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 10:20:52.978471527 (546h51m52.849554562s) [upload=479529]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 10:20:53.224037597 (546h51m53.095120632s) [seg_pos={0 0}, upload=479529]: Begin upload segment... Dec 28 10:20:53.224084497 (546h51m53.095167532s) [seg_pos={0 0}, upload=479529]: Done begin upload segment. Dec 28 10:20:53.224140062 (546h51m53.095223117s) [noise=true , address=77.90.31.11:4556 , piece_id=ZHXRF3QCUU4FV7P3OJ7TLFEFLMR7F26K4UHG4OFWYIZFTJOJGD6A, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.224148639 (546h51m53.095231685s) [noise=true , address=145.79.227.28:4725 , piece_id=BJ32VJWVH66R5T57UVSE76M227T3PND3IS4GYTVJCN62NRDFK2ZA, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.224718384 (546h51m53.095801419s) [noise=true , address=145.79.227.42:4123 , piece_id=CG5UCA4G3BNPI3BAFBTMOAGUZJEU7FW2MSNRYSHV77LPLR4CK5SA, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.225234898 (546h51m53.096317933s) [noise=true , address=145.79.227.3:4151 , piece_id=SUWYBG7TPG7BGBOS5G4UKXR2Z753Q4GJ2AN3M3WCRYPN3IKVEHQQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.225504882 (546h51m53.096587917s) [noise=true , address=145.79.227.21:4474 , piece_id=EEGOXLAYIZVCT4NF34CFLAVESLW2WPEOKJPVPRYIUSHKRIC5VR3Q, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.226212750 (546h51m53.097295785s) [noise=true , address=145.79.227.14:4840 , piece_id=YNOHADAVEJGWSG5VERF7HWN7GHHIPDWWECQ5HTPSS7HKC7DMVOWQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.226608573 (546h51m53.097691618s) [noise=true , address=145.79.227.6:4258 , piece_id=2DFJ2B7ETKXW76LASWPT4LGUAUV7I2OEV3N5JGOVPW74QVK6F4TQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.226767135 (546h51m53.09785018s) [noise=true , address=45.91.116.7:4009 , piece_id=FWZA5D3EYV77RAPKAZOBR4Z4OOSNWXHXXE4WJ4QNEETMZPJHYMAQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.227078098 (546h51m53.098161143s) [noise=true , address=45.91.116.9:4262 , piece_id=6JXAFCBODMHO5OYB5WGZNB762V5JUL32RXMFYHP6E3DCTWSSQYFA, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.227238392 (546h51m53.098321427s) [noise=true , address=145.79.227.16:4911 , piece_id=TZOOFWAQ33QB5B6MOUJ3XH7EB3IALWYGN43QUGNXOG6EXOJTB4AA, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.227466998 (546h51m53.098550033s) [noise=true , address=145.79.227.48:4301 , piece_id=NZ24TJ7I42YFMDOUF5SHORFQIHOKJBDILK3SGMGFHJ46OO6DGX5Q, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.227646309 (546h51m53.098729354s) [noise=true , address=77.90.31.15:4697 , piece_id=3FQIGY4VHPZIQ7ONVHGW7Z4UIV7EAECOAPGQEUVRYHXOLEOLHCTA, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.228003639 (546h51m53.099086684s) [noise=true , address=145.79.227.2:4115 , piece_id=XKLFZEWHLNY6AA5RVLWWPCLVUMWDU4YBJJQH2UFQ4U27YGVZGOXQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.228329440 (546h51m53.099412475s) [noise=true , address=145.79.227.32:4835 , piece_id=K5FO3OYDESYAUI3QUQ65WTEFJ2N77PPJL5X3FFNKAYU4X5PWKXBQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.228632276 (546h51m53.099715311s) [noise=true , address=45.91.116.8:4047 , piece_id=ATZ3XPE3S3AZSYXQZQWBDLBZQCVAMVIDX3Q2PDKQETLXWYEWMKPQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.228942016 (546h51m53.100025061s) [noise=true , address=145.79.227.36:4519 , piece_id=WVXG7E2MKVEYO4MHELTZ5HNRIXPJWLF2ZL4AU46X32W62SW5V3IQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:54.851414651 (546h51m54.722497686s) [upload=479529]: Next returned nil segment Dec 28 10:20:54.851429859 (546h51m54.722512894s) [upload=479529]: Waiting for error group managing segments... Dec 28 10:20:55.232471268 (546h51m55.103554303s) [noise=true , address=145.79.227.16:4911 , piece_id=TZOOFWAQ33QB5B6MOUJ3XH7EB3IALWYGN43QUGNXOG6EXOJTB4AA, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.232529519 (546h51m55.103612554s) [noise=true , address=145.79.227.2:4115 , piece_id=XKLFZEWHLNY6AA5RVLWWPCLVUMWDU4YBJJQH2UFQ4U27YGVZGOXQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.234914169 (546h51m55.105997204s) [noise=true , address=145.79.227.32:4835 , piece_id=K5FO3OYDESYAUI3QUQ65WTEFJ2N77PPJL5X3FFNKAYU4X5PWKXBQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.235465209 (546h51m55.106548244s) [noise=true , address=145.79.227.28:4725 , piece_id=BJ32VJWVH66R5T57UVSE76M227T3PND3IS4GYTVJCN62NRDFK2ZA, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.235615435 (546h51m55.10669847s) [noise=true , address=145.79.227.21:4474 , piece_id=EEGOXLAYIZVCT4NF34CFLAVESLW2WPEOKJPVPRYIUSHKRIC5VR3Q, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.236221559 (546h51m55.107304594s) [noise=true , address=145.79.227.6:4258 , piece_id=2DFJ2B7ETKXW76LASWPT4LGUAUV7I2OEV3N5JGOVPW74QVK6F4TQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.246296964 (546h51m55.117379999s) [noise=true , address=145.79.227.42:4123 , piece_id=CG5UCA4G3BNPI3BAFBTMOAGUZJEU7FW2MSNRYSHV77LPLR4CK5SA, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.246947282 (546h51m55.118030317s) [noise=true , address=145.79.227.3:4151 , piece_id=SUWYBG7TPG7BGBOS5G4UKXR2Z753Q4GJ2AN3M3WCRYPN3IKVEHQQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.251425650 (546h51m55.122508695s) [noise=true , address=145.79.227.14:4840 , piece_id=YNOHADAVEJGWSG5VERF7HWN7GHHIPDWWECQ5HTPSS7HKC7DMVOWQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.256324468 (546h51m55.127407513s) [noise=true , address=145.79.227.48:4301 , piece_id=NZ24TJ7I42YFMDOUF5SHORFQIHOKJBDILK3SGMGFHJ46OO6DGX5Q, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.259608812 (546h51m55.130691847s) [noise=true , address=145.79.227.36:4519 , piece_id=WVXG7E2MKVEYO4MHELTZ5HNRIXPJWLF2ZL4AU46X32W62SW5V3IQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.483932663 (546h51m55.355015698s) [noise=true , address=45.91.116.8:4047 , piece_id=ATZ3XPE3S3AZSYXQZQWBDLBZQCVAMVIDX3Q2PDKQETLXWYEWMKPQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.551135548 (546h51m55.422218594s) [noise=true , address=45.91.116.9:4262 , piece_id=6JXAFCBODMHO5OYB5WGZNB762V5JUL32RXMFYHP6E3DCTWSSQYFA, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.551161338 (546h51m55.422244383s) [seg_pos={0 0}, upload=479529]: Segment reached optimal threshold of 13 pieces. Dec 28 10:20:55.551510112 (546h51m55.422593147s) [noise=true , address=77.90.31.11:4556 , piece_id=ZHXRF3QCUU4FV7P3OJ7TLFEFLMR7F26K4UHG4OFWYIZFTJOJGD6A, seg_pos={0 0}, upload=479529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: write tcp 188.166.225.170:33006->77.90.31.11:4556: use of closed network connection; write tcp 188.166.225.170:33006->77.90.31.11:4556: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33006->77.90.31.11:4556: use of closed network connection Dec 28 10:20:55.551527014 (546h51m55.422610059s) [noise=true , address=77.90.31.11:4556 , piece_id=ZHXRF3QCUU4FV7P3OJ7TLFEFLMR7F26K4UHG4OFWYIZFTJOJGD6A, seg_pos={0 0}, upload=479529]: Long tail cancellation due to optimal threshold Dec 28 10:20:55.551578462 (546h51m55.422661507s) [noise=true , address=45.91.116.7:4009 , piece_id=FWZA5D3EYV77RAPKAZOBR4Z4OOSNWXHXXE4WJ4QNEETMZPJHYMAQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 28 10:20:55.551591066 (546h51m55.422674121s) [noise=true , address=45.91.116.7:4009 , piece_id=FWZA5D3EYV77RAPKAZOBR4Z4OOSNWXHXXE4WJ4QNEETMZPJHYMAQ, seg_pos={0 0}, upload=479529]: Long tail cancellation due to optimal threshold Dec 28 10:20:55.551835822 (546h51m55.422918867s) [noise=true , address=77.90.31.15:4697 , piece_id=3FQIGY4VHPZIQ7ONVHGW7Z4UIV7EAECOAPGQEUVRYHXOLEOLHCTA, seg_pos={0 0}, upload=479529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:58702->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:58702->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58702->77.90.31.15:4697: use of closed network connection Dec 28 10:20:55.551845641 (546h51m55.422928676s) [noise=true , address=77.90.31.15:4697 , piece_id=3FQIGY4VHPZIQ7ONVHGW7Z4UIV7EAECOAPGQEUVRYHXOLEOLHCTA, seg_pos={0 0}, upload=479529]: Long tail cancellation due to optimal threshold Dec 28 10:20:55.551895075 (546h51m55.42297812s) [seg_pos={0 0}, upload=479529]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 10:20:55.552135894 (546h51m55.423218929s) [upload=479529]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 10:20:55.552144850 (546h51m55.423227885s) [upload=479529]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 10:20:55.854005394 (546h51m55.72508845s) [upload=479529]: Done uploading segments... Dec 28 10:20:55.854081920 (546h51m55.725164965s) [upload=479529]: Upload finished. err: <nil> Duration: 5.635560497s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [..X..X.........X.........X...................X..XX.....X..X.] duration: 6.018281723s trace: https://jaeger.dev.storj.io/trace/37d347f90dc836a FAIL (2025-12-28 10:21:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== segments... Dec 28 10:20:52.978430940 (546h51m52.849513975s) [upload=479529]: Got next segment. Inline: false Dec 28 10:20:52.978465887 (546h51m52.849548922s) [upload=479529]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 10:20:52.978471527 (546h51m52.849554562s) [upload=479529]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 10:20:53.224037597 (546h51m53.095120632s) [seg_pos={0 0}, upload=479529]: Begin upload segment... Dec 28 10:20:53.224084497 (546h51m53.095167532s) [seg_pos={0 0}, upload=479529]: Done begin upload segment. Dec 28 10:20:53.224140062 (546h51m53.095223117s) [noise=true , address=77.90.31.11:4556 , piece_id=ZHXRF3QCUU4FV7P3OJ7TLFEFLMR7F26K4UHG4OFWYIZFTJOJGD6A, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.224148639 (546h51m53.095231685s) [noise=true , address=145.79.227.28:4725 , piece_id=BJ32VJWVH66R5T57UVSE76M227T3PND3IS4GYTVJCN62NRDFK2ZA, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.224718384 (546h51m53.095801419s) [noise=true , address=145.79.227.42:4123 , piece_id=CG5UCA4G3BNPI3BAFBTMOAGUZJEU7FW2MSNRYSHV77LPLR4CK5SA, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.225234898 (546h51m53.096317933s) [noise=true , address=145.79.227.3:4151 , piece_id=SUWYBG7TPG7BGBOS5G4UKXR2Z753Q4GJ2AN3M3WCRYPN3IKVEHQQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.225504882 (546h51m53.096587917s) [noise=true , address=145.79.227.21:4474 , piece_id=EEGOXLAYIZVCT4NF34CFLAVESLW2WPEOKJPVPRYIUSHKRIC5VR3Q, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.226212750 (546h51m53.097295785s) [noise=true , address=145.79.227.14:4840 , piece_id=YNOHADAVEJGWSG5VERF7HWN7GHHIPDWWECQ5HTPSS7HKC7DMVOWQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.226608573 (546h51m53.097691618s) [noise=true , address=145.79.227.6:4258 , piece_id=2DFJ2B7ETKXW76LASWPT4LGUAUV7I2OEV3N5JGOVPW74QVK6F4TQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.226767135 (546h51m53.09785018s) [noise=true , address=45.91.116.7:4009 , piece_id=FWZA5D3EYV77RAPKAZOBR4Z4OOSNWXHXXE4WJ4QNEETMZPJHYMAQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.227078098 (546h51m53.098161143s) [noise=true , address=45.91.116.9:4262 , piece_id=6JXAFCBODMHO5OYB5WGZNB762V5JUL32RXMFYHP6E3DCTWSSQYFA, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.227238392 (546h51m53.098321427s) [noise=true , address=145.79.227.16:4911 , piece_id=TZOOFWAQ33QB5B6MOUJ3XH7EB3IALWYGN43QUGNXOG6EXOJTB4AA, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.227466998 (546h51m53.098550033s) [noise=true , address=145.79.227.48:4301 , piece_id=NZ24TJ7I42YFMDOUF5SHORFQIHOKJBDILK3SGMGFHJ46OO6DGX5Q, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.227646309 (546h51m53.098729354s) [noise=true , address=77.90.31.15:4697 , piece_id=3FQIGY4VHPZIQ7ONVHGW7Z4UIV7EAECOAPGQEUVRYHXOLEOLHCTA, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.228003639 (546h51m53.099086684s) [noise=true , address=145.79.227.2:4115 , piece_id=XKLFZEWHLNY6AA5RVLWWPCLVUMWDU4YBJJQH2UFQ4U27YGVZGOXQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.228329440 (546h51m53.099412475s) [noise=true , address=145.79.227.32:4835 , piece_id=K5FO3OYDESYAUI3QUQ65WTEFJ2N77PPJL5X3FFNKAYU4X5PWKXBQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.228632276 (546h51m53.099715311s) [noise=true , address=45.91.116.8:4047 , piece_id=ATZ3XPE3S3AZSYXQZQWBDLBZQCVAMVIDX3Q2PDKQETLXWYEWMKPQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:53.228942016 (546h51m53.100025061s) [noise=true , address=145.79.227.36:4519 , piece_id=WVXG7E2MKVEYO4MHELTZ5HNRIXPJWLF2ZL4AU46X32W62SW5V3IQ, seg_pos={0 0}, upload=479529]: Uploading piece... Dec 28 10:20:54.851414651 (546h51m54.722497686s) [upload=479529]: Next returned nil segment Dec 28 10:20:54.851429859 (546h51m54.722512894s) [upload=479529]: Waiting for error group managing segments... Dec 28 10:20:55.232471268 (546h51m55.103554303s) [noise=true , address=145.79.227.16:4911 , piece_id=TZOOFWAQ33QB5B6MOUJ3XH7EB3IALWYGN43QUGNXOG6EXOJTB4AA, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.232529519 (546h51m55.103612554s) [noise=true , address=145.79.227.2:4115 , piece_id=XKLFZEWHLNY6AA5RVLWWPCLVUMWDU4YBJJQH2UFQ4U27YGVZGOXQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.234914169 (546h51m55.105997204s) [noise=true , address=145.79.227.32:4835 , piece_id=K5FO3OYDESYAUI3QUQ65WTEFJ2N77PPJL5X3FFNKAYU4X5PWKXBQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.235465209 (546h51m55.106548244s) [noise=true , address=145.79.227.28:4725 , piece_id=BJ32VJWVH66R5T57UVSE76M227T3PND3IS4GYTVJCN62NRDFK2ZA, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.235615435 (546h51m55.10669847s) [noise=true , address=145.79.227.21:4474 , piece_id=EEGOXLAYIZVCT4NF34CFLAVESLW2WPEOKJPVPRYIUSHKRIC5VR3Q, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.236221559 (546h51m55.107304594s) [noise=true , address=145.79.227.6:4258 , piece_id=2DFJ2B7ETKXW76LASWPT4LGUAUV7I2OEV3N5JGOVPW74QVK6F4TQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.246296964 (546h51m55.117379999s) [noise=true , address=145.79.227.42:4123 , piece_id=CG5UCA4G3BNPI3BAFBTMOAGUZJEU7FW2MSNRYSHV77LPLR4CK5SA, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.246947282 (546h51m55.118030317s) [noise=true , address=145.79.227.3:4151 , piece_id=SUWYBG7TPG7BGBOS5G4UKXR2Z753Q4GJ2AN3M3WCRYPN3IKVEHQQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.251425650 (546h51m55.122508695s) [noise=true , address=145.79.227.14:4840 , piece_id=YNOHADAVEJGWSG5VERF7HWN7GHHIPDWWECQ5HTPSS7HKC7DMVOWQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.256324468 (546h51m55.127407513s) [noise=true , address=145.79.227.48:4301 , piece_id=NZ24TJ7I42YFMDOUF5SHORFQIHOKJBDILK3SGMGFHJ46OO6DGX5Q, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.259608812 (546h51m55.130691847s) [noise=true , address=145.79.227.36:4519 , piece_id=WVXG7E2MKVEYO4MHELTZ5HNRIXPJWLF2ZL4AU46X32W62SW5V3IQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.483932663 (546h51m55.355015698s) [noise=true , address=45.91.116.8:4047 , piece_id=ATZ3XPE3S3AZSYXQZQWBDLBZQCVAMVIDX3Q2PDKQETLXWYEWMKPQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.551135548 (546h51m55.422218594s) [noise=true , address=45.91.116.9:4262 , piece_id=6JXAFCBODMHO5OYB5WGZNB762V5JUL32RXMFYHP6E3DCTWSSQYFA, seg_pos={0 0}, upload=479529]: Done uploading piece. err: <nil> Dec 28 10:20:55.551161338 (546h51m55.422244383s) [seg_pos={0 0}, upload=479529]: Segment reached optimal threshold of 13 pieces. Dec 28 10:20:55.551510112 (546h51m55.422593147s) [noise=true , address=77.90.31.11:4556 , piece_id=ZHXRF3QCUU4FV7P3OJ7TLFEFLMR7F26K4UHG4OFWYIZFTJOJGD6A, seg_pos={0 0}, upload=479529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: write tcp 188.166.225.170:33006->77.90.31.11:4556: use of closed network connection; write tcp 188.166.225.170:33006->77.90.31.11:4556: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33006->77.90.31.11:4556: use of closed network connection Dec 28 10:20:55.551527014 (546h51m55.422610059s) [noise=true , address=77.90.31.11:4556 , piece_id=ZHXRF3QCUU4FV7P3OJ7TLFEFLMR7F26K4UHG4OFWYIZFTJOJGD6A, seg_pos={0 0}, upload=479529]: Long tail cancellation due to optimal threshold Dec 28 10:20:55.551578462 (546h51m55.422661507s) [noise=true , address=45.91.116.7:4009 , piece_id=FWZA5D3EYV77RAPKAZOBR4Z4OOSNWXHXXE4WJ4QNEETMZPJHYMAQ, seg_pos={0 0}, upload=479529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 28 10:20:55.551591066 (546h51m55.422674121s) [noise=true , address=45.91.116.7:4009 , piece_id=FWZA5D3EYV77RAPKAZOBR4Z4OOSNWXHXXE4WJ4QNEETMZPJHYMAQ, seg_pos={0 0}, upload=479529]: Long tail cancellation due to optimal threshold Dec 28 10:20:55.551835822 (546h51m55.422918867s) [noise=true , address=77.90.31.15:4697 , piece_id=3FQIGY4VHPZIQ7ONVHGW7Z4UIV7EAECOAPGQEUVRYHXOLEOLHCTA, seg_pos={0 0}, upload=479529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:58702->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:58702->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58702->77.90.31.15:4697: use of closed network connection Dec 28 10:20:55.551845641 (546h51m55.422928676s) [noise=true , address=77.90.31.15:4697 , piece_id=3FQIGY4VHPZIQ7ONVHGW7Z4UIV7EAECOAPGQEUVRYHXOLEOLHCTA, seg_pos={0 0}, upload=479529]: Long tail cancellation due to optimal threshold Dec 28 10:20:55.551895075 (546h51m55.42297812s) [seg_pos={0 0}, upload=479529]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 10:20:55.552135894 (546h51m55.423218929s) [upload=479529]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 10:20:55.552144850 (546h51m55.423227885s) [upload=479529]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 10:20:55.854005394 (546h51m55.72508845s) [upload=479529]: Done uploading segments... Dec 28 10:20:55.854081920 (546h51m55.725164965s) [upload=479529]: Upload finished. err: <nil> Duration: 6.018323062s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)