Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [............................X...................X.....X.....] duration: 47.279507118s trace: https://jaeger.dev.storj.io/trace/1822be52d200e5b5 FAIL (2025-12-28 02:30:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1ZAfYZ4wgpZVvatj6wtHona3hTaPcLxAmXC1DT2uKQyo8nLQQ7, Piece ID: 5MMQ44MXRT4XEL4X6WO54WRG2XYMX3AN3FNQ2H2XIUTKBXI3EEPA) context deadline exceeded; piecestore: (Node ID: 124guAM82oibYZsMho67Em6Ke6yEZXwT6LPMLLSZZBLqwoFGUkn, Piece ID: 22WJIPX4SM5NTIAW5ZRM2UKX32CTDBFCXPCCHRRWRCUCOF5CRE5Q) context deadline exceeded; piecestore: (Node ID: 1e6p1TD5CqUyWdZjkrxh2kRqMuyDiMUr9eimfeayDAKwhtBrZ8, Piece ID: XRI5EJFN7YYUG4QKLDGEFT6YAUHCUTTSXF735UDR6LZZBGMAZDJQ) context deadline exceeded ======LOGS====== 5.234848204 (538h55m35.105931239s) [noise=true , address=77.90.31.11:4572 , piece_id=5MMQ44MXRT4XEL4X6WO54WRG2XYMX3AN3FNQ2H2XIUTKBXI3EEPA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.235412057 (538h55m35.106495102s) [noise=true , address=145.79.227.22:4512 , piece_id=D46YLTIKGHG36DQ4YBKBSUDV4GGB4WVDWNOJS27IYYLN6AYZ5QFQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.235820946 (538h55m35.106903981s) [noise=true , address=77.90.31.19:4874 , piece_id=RHZUXUYMJU6RKQGSDZM5WDWIN5SQ7YIRSRATNNKT5GEMU2QH444A, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.236192192 (538h55m35.107275237s) [noise=true , address=77.90.31.16:4760 , piece_id=XRI5EJFN7YYUG4QKLDGEFT6YAUHCUTTSXF735UDR6LZZBGMAZDJQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.236564772 (538h55m35.107647817s) [noise=true , address=145.79.227.40:4062 , piece_id=PW6QWIPQWVUETLC6DJ624G6IB4P6EM4SYB72NAMQ4V6NUKNLUYLQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.236929766 (538h55m35.108012812s) [noise=true , address=64.49.13.139:4480 , piece_id=JAJY3F7IDFFMFKCK2QJV6W74RIESLRKBLXMN27Q3C45JWEPLBZLQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.237916846 (538h55m35.108999881s) [noise=true , address=145.79.227.34:4932 , piece_id=UPXSCQ45QDGPKNJXRFJ33EAA3QENVHRVQARU5TFJTEWICZNR253Q, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.238852948 (538h55m35.109935983s) [noise=true , address=45.91.116.12:4257 , piece_id=ZI67LIAHZW4G357GWKDPTB5SMXYGNXZTTTDMNAETDS2B6SWVDAEQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239152910 (538h55m35.110235945s) [noise=true , address=145.79.227.44:4222 , piece_id=CZ2CS62CTHQFAIE3S3JO2N3VJ4XIR5YJ2LMRVLLX7I4VM2VZVQVA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239409969 (538h55m35.110493004s) [noise=true , address=145.79.227.30:4773 , piece_id=KUGJXZCNMUE7WSK3VLY4AUBSOT4H6IY4XB4NCBGAGCNUG7F7IODQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239681136 (538h55m35.110764171s) [noise=true , address=77.90.31.13:4635 , piece_id=IIOY5X2W3XW4C2OFEBRCABMYFBI24RRTYNUSKN6KJU7CTUW4KONQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239954676 (538h55m35.111037711s) [noise=true , address=45.91.116.9:4281 , piece_id=JZLDW777CBX2BP7MLOQV4NFP3H4XJVCNQXTHNSUWPVDOJLPZK36A, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.240189914 (538h55m35.111272949s) [noise=true , address=145.79.227.35:4512 , piece_id=3E3VCCC45XVT5FHFDKXSTGGAGYBEML3BKU6JJTT33LCD2YMOMBZQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.240449147 (538h55m35.111532182s) [noise=true , address=45.91.116.8:4064 , piece_id=22WJIPX4SM5NTIAW5ZRM2UKX32CTDBFCXPCCHRRWRCUCOF5CRE5Q, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.240639640 (538h55m35.111722675s) [noise=true , address=145.79.227.32:4853 , piece_id=MBNCFDBQ4SZZXV4F7J5TRMOUYUQ5IPKQCGYGGM5SOIFJ6ZYCAUYA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.241137798 (538h55m35.112220833s) [noise=true , address=77.90.31.16:4751 , piece_id=N3Z2LLMCIA2F42VXZA7IDRU4XEKGOICH6TRVERODU76MFFE4QSHA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.241383797 (538h55m35.112466842s) [noise=true , address=77.90.31.18:4836 , piece_id=BSVEFI52EKN44CMINFUSZBNOH53IW5TXLVTF5IH6ZUTNII6RRLVA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:36.592880816 (538h55m36.463963871s) [upload=472631]: Next returned nil segment Dec 28 02:24:36.592901625 (538h55m36.46398467s) [upload=472631]: Waiting for error group managing segments... Dec 28 02:24:36.946933741 (538h55m36.818016786s) [noise=true , address=64.49.13.139:4480 , piece_id=JAJY3F7IDFFMFKCK2QJV6W74RIESLRKBLXMN27Q3C45JWEPLBZLQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.128189692 (538h55m36.999272737s) [noise=true , address=145.79.227.35:4512 , piece_id=3E3VCCC45XVT5FHFDKXSTGGAGYBEML3BKU6JJTT33LCD2YMOMBZQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.160760084 (538h55m37.031843119s) [noise=true , address=145.79.227.34:4932 , piece_id=UPXSCQ45QDGPKNJXRFJ33EAA3QENVHRVQARU5TFJTEWICZNR253Q, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.288838207 (538h55m37.159921252s) [noise=true , address=77.90.31.11:4572 , piece_id=5MMQ44MXRT4XEL4X6WO54WRG2XYMX3AN3FNQ2H2XIUTKBXI3EEPA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.319996310 (538h55m37.191079345s) [noise=true , address=145.79.227.22:4512 , piece_id=D46YLTIKGHG36DQ4YBKBSUDV4GGB4WVDWNOJS27IYYLN6AYZ5QFQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.320513273 (538h55m37.191596319s) [noise=true , address=77.90.31.18:4836 , piece_id=BSVEFI52EKN44CMINFUSZBNOH53IW5TXLVTF5IH6ZUTNII6RRLVA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.336522926 (538h55m37.207605971s) [noise=true , address=145.79.227.32:4853 , piece_id=MBNCFDBQ4SZZXV4F7J5TRMOUYUQ5IPKQCGYGGM5SOIFJ6ZYCAUYA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.384868031 (538h55m37.255951066s) [noise=true , address=77.90.31.16:4751 , piece_id=N3Z2LLMCIA2F42VXZA7IDRU4XEKGOICH6TRVERODU76MFFE4QSHA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.453563554 (538h55m37.324646599s) [noise=true , address=45.91.116.8:4064 , piece_id=22WJIPX4SM5NTIAW5ZRM2UKX32CTDBFCXPCCHRRWRCUCOF5CRE5Q, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.479454839 (538h55m37.350537884s) [noise=true , address=45.91.116.12:4257 , piece_id=ZI67LIAHZW4G357GWKDPTB5SMXYGNXZTTTDMNAETDS2B6SWVDAEQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.659710174 (538h55m37.530793219s) [noise=true , address=77.90.31.13:4635 , piece_id=IIOY5X2W3XW4C2OFEBRCABMYFBI24RRTYNUSKN6KJU7CTUW4KONQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.715039128 (538h55m37.586122163s) [noise=true , address=77.90.31.16:4760 , piece_id=XRI5EJFN7YYUG4QKLDGEFT6YAUHCUTTSXF735UDR6LZZBGMAZDJQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.977473406 (538h55m37.848556441s) [noise=true , address=77.90.31.19:4874 , piece_id=RHZUXUYMJU6RKQGSDZM5WDWIN5SQ7YIRSRATNNKT5GEMU2QH444A, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:38.016302745 (538h55m37.88738578s) [noise=true , address=145.79.227.30:4773 , piece_id=KUGJXZCNMUE7WSK3VLY4AUBSOT4H6IY4XB4NCBGAGCNUG7F7IODQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:38.016326982 (538h55m37.887410027s) [seg_pos={0 0}, upload=472631]: Segment reached optimal threshold of 14 pieces. Dec 28 02:24:38.016585353 (538h55m37.887668398s) [noise=true , address=145.79.227.30:4766 , piece_id=67M2DMPAVWIQKQSWUAG73D4E4NRRSG4XEVPOUVIJPEQOUV7A6VVA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1k6zdamuKPkBdzyNZNWx4FCmaTkv33PchcwvTRR8bgcETBs4K7): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016594450 (538h55m37.887677495s) [noise=true , address=145.79.227.30:4766 , piece_id=67M2DMPAVWIQKQSWUAG73D4E4NRRSG4XEVPOUVIJPEQOUV7A6VVA, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016627093 (538h55m37.887710148s) [noise=true , address=45.91.116.9:4281 , piece_id=JZLDW777CBX2BP7MLOQV4NFP3H4XJVCNQXTHNSUWPVDOJLPZK36A, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gxf7DpJZthyR6f1PExX56tJavjhPw9Gxb4h8mtmeYKW3t3tS1): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016635258 (538h55m37.887718293s) [noise=true , address=45.91.116.9:4281 , piece_id=JZLDW777CBX2BP7MLOQV4NFP3H4XJVCNQXTHNSUWPVDOJLPZK36A, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016871298 (538h55m37.887954363s) [noise=true , address=145.79.227.44:4222 , piece_id=CZ2CS62CTHQFAIE3S3JO2N3VJ4XIR5YJ2LMRVLLX7I4VM2VZVQVA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZKW35KvrsoYRefbYeUFZo8bh4XZYycNKGEvTFjzibezjXgj3e): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016880124 (538h55m37.887963169s) [noise=true , address=145.79.227.44:4222 , piece_id=CZ2CS62CTHQFAIE3S3JO2N3VJ4XIR5YJ2LMRVLLX7I4VM2VZVQVA, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016934839 (538h55m37.888017884s) [noise=true , address=145.79.227.40:4062 , piece_id=PW6QWIPQWVUETLC6DJ624G6IB4P6EM4SYB72NAMQ4V6NUKNLUYLQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016943425 (538h55m37.88802649s) [noise=true , address=145.79.227.40:4062 , piece_id=PW6QWIPQWVUETLC6DJ624G6IB4P6EM4SYB72NAMQ4V6NUKNLUYLQ, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016968252 (538h55m37.888051297s) [seg_pos={0 0}, upload=472631]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 02:24:38.017067822 (538h55m37.888150867s) [upload=472631]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 02:24:38.017074244 (538h55m37.888157289s) [upload=472631]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 02:24:38.304306667 (538h55m38.175389712s) [upload=472631]: Done uploading segments... Dec 28 02:24:38.304329641 (538h55m38.175412686s) [upload=472631]: Upload finished. err: <nil> Duration: 47.279529942s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 33 out of 60 is failed (threshold=3) [XX.X.XX.XX.X..X.X.X.X.X..X...XX..X.XXX.XXX.X.....XXX.XXXXXX.] duration: 6.075099403s trace: https://jaeger.dev.storj.io/trace/dfb1bc3b364e701 FAIL (2025-12-28 02:30:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, Piece ID: EBAXKZZCWMMC7GIVXHBRGNF5IILE726WZDK64FBYD4ROWKIDW4IA) context deadline exceeded; piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: WNH5SYEGHKFTKFN4C6D6T67TH3NADBI3KJWDL24LQFAI7DEYYQFA) context deadline exceeded ======LOGS====== Dec 28 02:28:58.855004942 (538h59m58.726087987s) [upload=472691]: Starting upload Dec 28 02:28:58.855059025 (538h59m58.72614206s) [upload=472691]: Done starting upload Dec 28 02:28:58.855109451 (538h59m58.726192486s) [upload=472691]: Uploading segments... Dec 28 02:28:58.855329971 (538h59m58.726413006s) [upload=472691]: Got next segment. Inline: false Dec 28 02:28:58.855367172 (538h59m58.726450217s) [upload=472691]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 02:28:58.855374065 (538h59m58.7264571s) [upload=472691]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 02:28:59.099231384 (538h59m58.970314419s) [seg_pos={0 0}, upload=472691]: Begin upload segment... Dec 28 02:28:59.099280277 (538h59m58.970363312s) [seg_pos={0 0}, upload=472691]: Done begin upload segment. Dec 28 02:28:59.099305996 (538h59m58.970389031s) [noise=true , address=45.91.116.4:4940 , piece_id=NRXFZMSOKTHKO4OWZO37F6VDLNE3IWCGWEEBIDZFYRORR6RXLPLQ, seg_pos={0 0}, upload=472691]: Uploading piece... Dec 28 02:28:59.099334120 (538h59m58.970417165s) [noise=true , address=77.90.31.13:4627 , piece_id=HV26DBCARVJYFTLV5EVSTQ745IWROPHVJ5W5TGBJR27BASJU2RLQ, seg_pos={0 0}, upload=472691]: Uploading piece... Dec 28 02:28:59.099725425 (538h59m58.97080847s) [noise=true , address=145.79.227.47:4265 , piece_id=7MXGUKZ6LVYSWCJ36UMHQUQX4SXMRSYIW7ZRZIJAZGJKK46QJ47A, seg_pos={0 0}, upload=472691]: Uploading piece... Dec 28 02:28:59.100162867 (538h59m58.971245912s) [noise=true , address=145.79.227.32:4836 , piece_id=IM75V7UENSIBLUFMSYX4JOGESCCM44USLGIX5TKIX2FW4GULF7PA, seg_pos={0 0}, upload=472691]: Uploading piece... Dec 28 02:28:59.100456527 (538h59m58.971539562s) [noise=true , address=145.79.227.20:4438 , piece_id=U72NA6GHPLII3RK2L2NHCMI7CZC5DBQMH3DSLX2LXXTA3XS3T77Q, seg_pos={0 0}, upload=472691]: Uploading piece... Dec 28 02:28:59.100792576 (538h59m58.971875611s) [noise=true , address=145.79.227.39:4014 , piece_id=NEV7VGBK54NR6757SFGZ4YQZAYKDJ5ZVSREJX5UY4URLX2YQGIWA, seg_pos={0 0}, upload=472691]: Uploading piece... Dec 28 02:28:59.101089492 (538h59m58.972172527s) [noise=true , address=145.79.227.36:4519 , piece_id=XYNHOSHH7FQW6OGIWRBG2C2ZP7WMWNYSU5MB2VQGF23RIFGZ334Q, seg_pos={0 0}, upload=472691]: Uploading piece... Dec 28 02:28:59.101150938 (538h59m58.972233973s) [noise=true , address=45.91.116.9:4262 , piece_id=5ICN2BBOMYZOW6YNGHDWZ64QHURGHFNGE3XS4XFM5VLMBVLJNUGQ, seg_pos={0 0}, upload=472691]: Uploading piece... Dec 28 02:28:59.101358704 (538h59m58.972441739s) [noise=true , address=145.79.227.42:4123 , piece_id=4AE5GYXAK2DVWXEKFGMQ7RT3DZMNX7C7E2AGESBYIUCNYXHKZYZQ, seg_pos={0 0}, upload=472691]: Uploading piece... Dec 28 02:28:59.101525532 (538h59m58.972608567s) [noise=true , address=45.91.116.14:4189 , piece_id=HARIV2LDOHJEG3ST5MAHIWES7LGA63KEQW4I3VUD2HQ2OGDM4OJQ, seg_pos={0 0}, upload=472691]: Uploading piece... Dec 28 02:28:59.101631564 (538h59m58.972714599s) [noise=true , address=145.79.227.15:4873 , piece_id=GX5CU47WFDNAEDMNX2PTGBPEHYU5BL7SRALQEMDCJR2UX3LBVQVA, seg_pos={0 0}, upload=472691]: Uploading piece... Dec 28 02:28:59.101684204 (538h59m58.972767239s) [noise=true , address=145.79.227.9:4365 , piece_id=VVQ6CFNSIP2OTZLB5PGUANRW2ZVRSSYL3ZQECDDZVJH2AJWYFXOA, seg_pos={0 0}, upload=472691]: Uploading piece... Dec 28 02:28:59.101727136 (538h59m58.972810171s) [noise=true , address=145.79.227.11:4729 , piece_id=JRY36DMXMJJ6PLLHON4GJKOKQTQT43J24TWMLWKOQ5HQKAHI4HGQ, seg_pos={0 0}, upload=472691]: Uploading piece... Dec 28 02:28:59.102179377 (538h59m58.973262422s) [noise=true , address=77.90.31.12:4590 , piece_id=WNH5SYEGHKFTKFN4C6D6T67TH3NADBI3KJWDL24LQFAI7DEYYQFA, seg_pos={0 0}, upload=472691]: Uploading piece... Dec 28 02:28:59.102663779 (538h59m58.973746815s) [noise=true , address=77.90.31.11:4556 , piece_id=EBAXKZZCWMMC7GIVXHBRGNF5IILE726WZDK64FBYD4ROWKIDW4IA, seg_pos={0 0}, upload=472691]: Uploading piece... Dec 28 02:28:59.104033427 (538h59m58.975116462s) [noise=true , address=45.91.116.12:4225 , piece_id=SNWLUQPSRQN3GLVI3TFR4IHNU4ZYNPQB3B7FZ5U3AHU2ZN5RKNHQ, seg_pos={0 0}, upload=472691]: Uploading piece... Dec 28 02:29:00.832744238 (539h0m0.703827274s) [upload=472691]: Next returned nil segment Dec 28 02:29:00.832760890 (539h0m0.703843925s) [upload=472691]: Waiting for error group managing segments... Dec 28 02:29:01.111841663 (539h0m0.982924698s) [noise=true , address=145.79.227.39:4014 , piece_id=NEV7VGBK54NR6757SFGZ4YQZAYKDJ5ZVSREJX5UY4URLX2YQGIWA, seg_pos={0 0}, upload=472691]: Done uploading piece. err: <nil> Dec 28 02:29:01.120488677 (539h0m0.991571722s) [noise=true , address=145.79.227.20:4438 , piece_id=U72NA6GHPLII3RK2L2NHCMI7CZC5DBQMH3DSLX2LXXTA3XS3T77Q, seg_pos={0 0}, upload=472691]: Done uploading piece. err: <nil> Dec 28 02:29:01.126427143 (539h0m0.997510188s) [noise=true , address=145.79.227.47:4265 , piece_id=7MXGUKZ6LVYSWCJ36UMHQUQX4SXMRSYIW7ZRZIJAZGJKK46QJ47A, seg_pos={0 0}, upload=472691]: Done uploading piece. err: <nil> Dec 28 02:29:01.230392336 (539h0m1.101475371s) [noise=true , address=145.79.227.42:4123 , piece_id=4AE5GYXAK2DVWXEKFGMQ7RT3DZMNX7C7E2AGESBYIUCNYXHKZYZQ, seg_pos={0 0}, upload=472691]: Done uploading piece. err: <nil> Dec 28 02:29:01.234844102 (539h0m1.105927137s) [noise=true , address=145.79.227.11:4729 , piece_id=JRY36DMXMJJ6PLLHON4GJKOKQTQT43J24TWMLWKOQ5HQKAHI4HGQ, seg_pos={0 0}, upload=472691]: Done uploading piece. err: <nil> Dec 28 02:29:01.236119871 (539h0m1.107202916s) [noise=true , address=145.79.227.15:4873 , piece_id=GX5CU47WFDNAEDMNX2PTGBPEHYU5BL7SRALQEMDCJR2UX3LBVQVA, seg_pos={0 0}, upload=472691]: Done uploading piece. err: <nil> Dec 28 02:29:01.255026739 (539h0m1.126109774s) [noise=true , address=145.79.227.32:4836 , piece_id=IM75V7UENSIBLUFMSYX4JOGESCCM44USLGIX5TKIX2FW4GULF7PA, seg_pos={0 0}, upload=472691]: Done uploading piece. err: <nil> Dec 28 02:29:01.256349948 (539h0m1.127432993s) [noise=true , address=145.79.227.36:4519 , piece_id=XYNHOSHH7FQW6OGIWRBG2C2ZP7WMWNYSU5MB2VQGF23RIFGZ334Q, seg_pos={0 0}, upload=472691]: Done uploading piece. err: <nil> Dec 28 02:29:01.260568300 (539h0m1.131651345s) [noise=true , address=145.79.227.9:4365 , piece_id=VVQ6CFNSIP2OTZLB5PGUANRW2ZVRSSYL3ZQECDDZVJH2AJWYFXOA, seg_pos={0 0}, upload=472691]: Done uploading piece. err: <nil> Dec 28 02:29:01.366463447 (539h0m1.237546492s) [noise=true , address=77.90.31.12:4590 , piece_id=WNH5SYEGHKFTKFN4C6D6T67TH3NADBI3KJWDL24LQFAI7DEYYQFA, seg_pos={0 0}, upload=472691]: Done uploading piece. err: <nil> Dec 28 02:29:01.453658953 (539h0m1.324741988s) [noise=true , address=45.91.116.4:4940 , piece_id=NRXFZMSOKTHKO4OWZO37F6VDLNE3IWCGWEEBIDZFYRORR6RXLPLQ, seg_pos={0 0}, upload=472691]: Done uploading piece. err: <nil> Dec 28 02:29:02.051278664 (539h0m1.922361709s) [noise=true , address=45.91.116.12:4225 , piece_id=SNWLUQPSRQN3GLVI3TFR4IHNU4ZYNPQB3B7FZ5U3AHU2ZN5RKNHQ, seg_pos={0 0}, upload=472691]: Done uploading piece. err: <nil> Dec 28 02:29:02.068449226 (539h0m1.939532271s) [noise=true , address=77.90.31.11:4556 , piece_id=EBAXKZZCWMMC7GIVXHBRGNF5IILE726WZDK64FBYD4ROWKIDW4IA, seg_pos={0 0}, upload=472691]: Done uploading piece. err: <nil> Dec 28 02:29:02.068475426 (539h0m1.939558471s) [seg_pos={0 0}, upload=472691]: Segment reached optimal threshold of 13 pieces. Dec 28 02:29:02.068738647 (539h0m1.939821712s) [noise=true , address=77.90.31.13:4627 , piece_id=HV26DBCARVJYFTLV5EVSTQ745IWROPHVJ5W5TGBJR27BASJU2RLQ, seg_pos={0 0}, upload=472691]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: expected piece hash; context canceled; EOF Dec 28 02:29:02.068749789 (539h0m1.939832834s) [noise=true , address=77.90.31.13:4627 , piece_id=HV26DBCARVJYFTLV5EVSTQ745IWROPHVJ5W5TGBJR27BASJU2RLQ, seg_pos={0 0}, upload=472691]: Long tail cancellation due to optimal threshold Dec 28 02:29:02.068957234 (539h0m1.940040269s) [noise=true , address=45.91.116.14:4189 , piece_id=HARIV2LDOHJEG3ST5MAHIWES7LGA63KEQW4I3VUD2HQ2OGDM4OJQ, seg_pos={0 0}, upload=472691]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: write tcp 188.166.225.170:37276->45.91.116.14:4189: use of closed network connection; write tcp 188.166.225.170:37276->45.91.116.14:4189: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37276->45.91.116.14:4189: use of closed network connection Dec 28 02:29:02.068976740 (539h0m1.940059805s) [noise=true , address=45.91.116.14:4189 , piece_id=HARIV2LDOHJEG3ST5MAHIWES7LGA63KEQW4I3VUD2HQ2OGDM4OJQ, seg_pos={0 0}, upload=472691]: Long tail cancellation due to optimal threshold Dec 28 02:29:02.069260851 (539h0m1.940343896s) [noise=true , address=45.91.116.9:4262 , piece_id=5ICN2BBOMYZOW6YNGHDWZ64QHURGHFNGE3XS4XFM5VLMBVLJNUGQ, seg_pos={0 0}, upload=472691]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 28 02:29:02.069270099 (539h0m1.940353144s) [noise=true , address=45.91.116.9:4262 , piece_id=5ICN2BBOMYZOW6YNGHDWZ64QHURGHFNGE3XS4XFM5VLMBVLJNUGQ, seg_pos={0 0}, upload=472691]: Long tail cancellation due to optimal threshold Dec 28 02:29:02.069297280 (539h0m1.940380325s) [seg_pos={0 0}, upload=472691]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 02:29:02.069399585 (539h0m1.94048263s) [upload=472691]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 02:29:02.069406850 (539h0m1.940489935s) [upload=472691]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 02:29:02.358322848 (539h0m2.229405893s) [upload=472691]: Done uploading segments... Dec 28 02:29:02.358345360 (539h0m2.229428396s) [upload=472691]: Upload finished. err: <nil> Duration: 6.075129871s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.........................X.........................X........] duration: 3.524282341s trace: https://jaeger.dev.storj.io/trace/4aff4da4d82027c9 FAIL (2025-12-28 02:30:01 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.524307749s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 34 out of 60 is failed (threshold=3) [XXX.X.XX.XX.X..X.X.X.X.X..X...XX..X.XXX.XXX.X.....XXX.XXXXXX] duration: 30.004227611s trace: https://jaeger.dev.storj.io/trace/dfb1bc3b364e701 FAIL (2025-12-28 02:29:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, Piece ID: EBAXKZZCWMMC7GIVXHBRGNF5IILE726WZDK64FBYD4ROWKIDW4IA) context deadline exceeded; piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: WNH5SYEGHKFTKFN4C6D6T67TH3NADBI3KJWDL24LQFAI7DEYYQFA) context deadline exceeded ======LOGS====== Dec 28 02:28:58.855004942 (538h59m58.726087987s) [upload=472691]: Starting upload Dec 28 02:28:58.855059025 (538h59m58.72614206s) [upload=472691]: Done starting upload Dec 28 02:28:58.855109451 (538h59m58.726192486s) [upload=472691]: Uploading segments... Dec 28 02:28:58.855329971 (538h59m58.726413006s) [upload=472691]: Got next segment. Inline: false Dec 28 02:28:58.855367172 (538h59m58.726450217s) [upload=472691]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 02:28:58.855374065 (538h59m58.7264571s) [upload=472691]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 02:28:59.099231384 (538h59m58.970314419s) [seg_pos={0 0}, upload=472691]: Begin upload segment... Dec 28 02:28:59.099280277 (538h59m58.970363312s) [seg_pos={0 0}, upload=472691]: Done begin upload segment. Dec 28 02:28:59.099305996 (538h59m58.970389031s) [noise=true , address=45.91.116.4:4940 , piece_id=NRXFZMSOKTHKO4OWZO37F6VDLNE3IWCGWEEBIDZFYRORR6RXLPLQ, seg_pos={0 0}, upload=472691]: Uploading piece... Dec 28 02:28:59.099334120 (538h59m58.970417165s) [noise=true , address=77.90.31.13:4627 , piece_id=HV26DBCARVJYFTLV5EVSTQ745IWROPHVJ5W5TGBJR27BASJU2RLQ, seg_pos={0 0}, upload=472691]: Uploading piece... Dec 28 02:28:59.099725425 (538h59m58.97080847s) [noise=true , address=145.79.227.47:4265 , piece_id=7MXGUKZ6LVYSWCJ36UMHQUQX4SXMRSYIW7ZRZIJAZGJKK46QJ47A, seg_pos={0 0}, upload=472691]: Uploading piece... Dec 28 02:28:59.100162867 (538h59m58.971245912s) [noise=true , address=145.79.227.32:4836 , piece_id=IM75V7UENSIBLUFMSYX4JOGESCCM44USLGIX5TKIX2FW4GULF7PA, seg_pos={0 0}, upload=472691]: Uploading piece... Dec 28 02:28:59.100456527 (538h59m58.971539562s) [noise=true , address=145.79.227.20:4438 , piece_id=U72NA6GHPLII3RK2L2NHCMI7CZC5DBQMH3DSLX2LXXTA3XS3T77Q, seg_pos={0 0}, upload=472691]: Uploading piece... Dec 28 02:28:59.100792576 (538h59m58.971875611s) [noise=true , address=145.79.227.39:4014 , piece_id=NEV7VGBK54NR6757SFGZ4YQZAYKDJ5ZVSREJX5UY4URLX2YQGIWA, seg_pos={0 0}, upload=472691]: Uploading piece... Dec 28 02:28:59.101089492 (538h59m58.972172527s) [noise=true , address=145.79.227.36:4519 , piece_id=XYNHOSHH7FQW6OGIWRBG2C2ZP7WMWNYSU5MB2VQGF23RIFGZ334Q, seg_pos={0 0}, upload=472691]: Uploading piece... Dec 28 02:28:59.101150938 (538h59m58.972233973s) [noise=true , address=45.91.116.9:4262 , piece_id=5ICN2BBOMYZOW6YNGHDWZ64QHURGHFNGE3XS4XFM5VLMBVLJNUGQ, seg_pos={0 0}, upload=472691]: Uploading piece... Dec 28 02:28:59.101358704 (538h59m58.972441739s) [noise=true , address=145.79.227.42:4123 , piece_id=4AE5GYXAK2DVWXEKFGMQ7RT3DZMNX7C7E2AGESBYIUCNYXHKZYZQ, seg_pos={0 0}, upload=472691]: Uploading piece... Dec 28 02:28:59.101525532 (538h59m58.972608567s) [noise=true , address=45.91.116.14:4189 , piece_id=HARIV2LDOHJEG3ST5MAHIWES7LGA63KEQW4I3VUD2HQ2OGDM4OJQ, seg_pos={0 0}, upload=472691]: Uploading piece... Dec 28 02:28:59.101631564 (538h59m58.972714599s) [noise=true , address=145.79.227.15:4873 , piece_id=GX5CU47WFDNAEDMNX2PTGBPEHYU5BL7SRALQEMDCJR2UX3LBVQVA, seg_pos={0 0}, upload=472691]: Uploading piece... Dec 28 02:28:59.101684204 (538h59m58.972767239s) [noise=true , address=145.79.227.9:4365 , piece_id=VVQ6CFNSIP2OTZLB5PGUANRW2ZVRSSYL3ZQECDDZVJH2AJWYFXOA, seg_pos={0 0}, upload=472691]: Uploading piece... Dec 28 02:28:59.101727136 (538h59m58.972810171s) [noise=true , address=145.79.227.11:4729 , piece_id=JRY36DMXMJJ6PLLHON4GJKOKQTQT43J24TWMLWKOQ5HQKAHI4HGQ, seg_pos={0 0}, upload=472691]: Uploading piece... Dec 28 02:28:59.102179377 (538h59m58.973262422s) [noise=true , address=77.90.31.12:4590 , piece_id=WNH5SYEGHKFTKFN4C6D6T67TH3NADBI3KJWDL24LQFAI7DEYYQFA, seg_pos={0 0}, upload=472691]: Uploading piece... Dec 28 02:28:59.102663779 (538h59m58.973746815s) [noise=true , address=77.90.31.11:4556 , piece_id=EBAXKZZCWMMC7GIVXHBRGNF5IILE726WZDK64FBYD4ROWKIDW4IA, seg_pos={0 0}, upload=472691]: Uploading piece... Dec 28 02:28:59.104033427 (538h59m58.975116462s) [noise=true , address=45.91.116.12:4225 , piece_id=SNWLUQPSRQN3GLVI3TFR4IHNU4ZYNPQB3B7FZ5U3AHU2ZN5RKNHQ, seg_pos={0 0}, upload=472691]: Uploading piece... Dec 28 02:29:00.832744238 (539h0m0.703827274s) [upload=472691]: Next returned nil segment Dec 28 02:29:00.832760890 (539h0m0.703843925s) [upload=472691]: Waiting for error group managing segments... Dec 28 02:29:01.111841663 (539h0m0.982924698s) [noise=true , address=145.79.227.39:4014 , piece_id=NEV7VGBK54NR6757SFGZ4YQZAYKDJ5ZVSREJX5UY4URLX2YQGIWA, seg_pos={0 0}, upload=472691]: Done uploading piece. err: <nil> Dec 28 02:29:01.120488677 (539h0m0.991571722s) [noise=true , address=145.79.227.20:4438 , piece_id=U72NA6GHPLII3RK2L2NHCMI7CZC5DBQMH3DSLX2LXXTA3XS3T77Q, seg_pos={0 0}, upload=472691]: Done uploading piece. err: <nil> Dec 28 02:29:01.126427143 (539h0m0.997510188s) [noise=true , address=145.79.227.47:4265 , piece_id=7MXGUKZ6LVYSWCJ36UMHQUQX4SXMRSYIW7ZRZIJAZGJKK46QJ47A, seg_pos={0 0}, upload=472691]: Done uploading piece. err: <nil> Dec 28 02:29:01.230392336 (539h0m1.101475371s) [noise=true , address=145.79.227.42:4123 , piece_id=4AE5GYXAK2DVWXEKFGMQ7RT3DZMNX7C7E2AGESBYIUCNYXHKZYZQ, seg_pos={0 0}, upload=472691]: Done uploading piece. err: <nil> Dec 28 02:29:01.234844102 (539h0m1.105927137s) [noise=true , address=145.79.227.11:4729 , piece_id=JRY36DMXMJJ6PLLHON4GJKOKQTQT43J24TWMLWKOQ5HQKAHI4HGQ, seg_pos={0 0}, upload=472691]: Done uploading piece. err: <nil> Dec 28 02:29:01.236119871 (539h0m1.107202916s) [noise=true , address=145.79.227.15:4873 , piece_id=GX5CU47WFDNAEDMNX2PTGBPEHYU5BL7SRALQEMDCJR2UX3LBVQVA, seg_pos={0 0}, upload=472691]: Done uploading piece. err: <nil> Dec 28 02:29:01.255026739 (539h0m1.126109774s) [noise=true , address=145.79.227.32:4836 , piece_id=IM75V7UENSIBLUFMSYX4JOGESCCM44USLGIX5TKIX2FW4GULF7PA, seg_pos={0 0}, upload=472691]: Done uploading piece. err: <nil> Dec 28 02:29:01.256349948 (539h0m1.127432993s) [noise=true , address=145.79.227.36:4519 , piece_id=XYNHOSHH7FQW6OGIWRBG2C2ZP7WMWNYSU5MB2VQGF23RIFGZ334Q, seg_pos={0 0}, upload=472691]: Done uploading piece. err: <nil> Dec 28 02:29:01.260568300 (539h0m1.131651345s) [noise=true , address=145.79.227.9:4365 , piece_id=VVQ6CFNSIP2OTZLB5PGUANRW2ZVRSSYL3ZQECDDZVJH2AJWYFXOA, seg_pos={0 0}, upload=472691]: Done uploading piece. err: <nil> Dec 28 02:29:01.366463447 (539h0m1.237546492s) [noise=true , address=77.90.31.12:4590 , piece_id=WNH5SYEGHKFTKFN4C6D6T67TH3NADBI3KJWDL24LQFAI7DEYYQFA, seg_pos={0 0}, upload=472691]: Done uploading piece. err: <nil> Dec 28 02:29:01.453658953 (539h0m1.324741988s) [noise=true , address=45.91.116.4:4940 , piece_id=NRXFZMSOKTHKO4OWZO37F6VDLNE3IWCGWEEBIDZFYRORR6RXLPLQ, seg_pos={0 0}, upload=472691]: Done uploading piece. err: <nil> Dec 28 02:29:02.051278664 (539h0m1.922361709s) [noise=true , address=45.91.116.12:4225 , piece_id=SNWLUQPSRQN3GLVI3TFR4IHNU4ZYNPQB3B7FZ5U3AHU2ZN5RKNHQ, seg_pos={0 0}, upload=472691]: Done uploading piece. err: <nil> Dec 28 02:29:02.068449226 (539h0m1.939532271s) [noise=true , address=77.90.31.11:4556 , piece_id=EBAXKZZCWMMC7GIVXHBRGNF5IILE726WZDK64FBYD4ROWKIDW4IA, seg_pos={0 0}, upload=472691]: Done uploading piece. err: <nil> Dec 28 02:29:02.068475426 (539h0m1.939558471s) [seg_pos={0 0}, upload=472691]: Segment reached optimal threshold of 13 pieces. Dec 28 02:29:02.068738647 (539h0m1.939821712s) [noise=true , address=77.90.31.13:4627 , piece_id=HV26DBCARVJYFTLV5EVSTQ745IWROPHVJ5W5TGBJR27BASJU2RLQ, seg_pos={0 0}, upload=472691]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: expected piece hash; context canceled; EOF Dec 28 02:29:02.068749789 (539h0m1.939832834s) [noise=true , address=77.90.31.13:4627 , piece_id=HV26DBCARVJYFTLV5EVSTQ745IWROPHVJ5W5TGBJR27BASJU2RLQ, seg_pos={0 0}, upload=472691]: Long tail cancellation due to optimal threshold Dec 28 02:29:02.068957234 (539h0m1.940040269s) [noise=true , address=45.91.116.14:4189 , piece_id=HARIV2LDOHJEG3ST5MAHIWES7LGA63KEQW4I3VUD2HQ2OGDM4OJQ, seg_pos={0 0}, upload=472691]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: write tcp 188.166.225.170:37276->45.91.116.14:4189: use of closed network connection; write tcp 188.166.225.170:37276->45.91.116.14:4189: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37276->45.91.116.14:4189: use of closed network connection Dec 28 02:29:02.068976740 (539h0m1.940059805s) [noise=true , address=45.91.116.14:4189 , piece_id=HARIV2LDOHJEG3ST5MAHIWES7LGA63KEQW4I3VUD2HQ2OGDM4OJQ, seg_pos={0 0}, upload=472691]: Long tail cancellation due to optimal threshold Dec 28 02:29:02.069260851 (539h0m1.940343896s) [noise=true , address=45.91.116.9:4262 , piece_id=5ICN2BBOMYZOW6YNGHDWZ64QHURGHFNGE3XS4XFM5VLMBVLJNUGQ, seg_pos={0 0}, upload=472691]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 28 02:29:02.069270099 (539h0m1.940353144s) [noise=true , address=45.91.116.9:4262 , piece_id=5ICN2BBOMYZOW6YNGHDWZ64QHURGHFNGE3XS4XFM5VLMBVLJNUGQ, seg_pos={0 0}, upload=472691]: Long tail cancellation due to optimal threshold Dec 28 02:29:02.069297280 (539h0m1.940380325s) [seg_pos={0 0}, upload=472691]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 02:29:02.069399585 (539h0m1.94048263s) [upload=472691]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 02:29:02.069406850 (539h0m1.940489935s) [upload=472691]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 02:29:02.358322848 (539h0m2.229405893s) [upload=472691]: Done uploading segments... Dec 28 02:29:02.358345360 (539h0m2.229428396s) [upload=472691]: Upload finished. err: <nil> Duration: 30.004249914s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............................X...................X.....X....] duration: 51.98033267s trace: https://jaeger.dev.storj.io/trace/1822be52d200e5b5 FAIL (2025-12-28 02:29:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1ZAfYZ4wgpZVvatj6wtHona3hTaPcLxAmXC1DT2uKQyo8nLQQ7, Piece ID: 5MMQ44MXRT4XEL4X6WO54WRG2XYMX3AN3FNQ2H2XIUTKBXI3EEPA) context deadline exceeded; piecestore: (Node ID: 124guAM82oibYZsMho67Em6Ke6yEZXwT6LPMLLSZZBLqwoFGUkn, Piece ID: 22WJIPX4SM5NTIAW5ZRM2UKX32CTDBFCXPCCHRRWRCUCOF5CRE5Q) context deadline exceeded; piecestore: (Node ID: 1e6p1TD5CqUyWdZjkrxh2kRqMuyDiMUr9eimfeayDAKwhtBrZ8, Piece ID: XRI5EJFN7YYUG4QKLDGEFT6YAUHCUTTSXF735UDR6LZZBGMAZDJQ) context deadline exceeded ======LOGS====== 5.234848204 (538h55m35.105931239s) [noise=true , address=77.90.31.11:4572 , piece_id=5MMQ44MXRT4XEL4X6WO54WRG2XYMX3AN3FNQ2H2XIUTKBXI3EEPA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.235412057 (538h55m35.106495102s) [noise=true , address=145.79.227.22:4512 , piece_id=D46YLTIKGHG36DQ4YBKBSUDV4GGB4WVDWNOJS27IYYLN6AYZ5QFQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.235820946 (538h55m35.106903981s) [noise=true , address=77.90.31.19:4874 , piece_id=RHZUXUYMJU6RKQGSDZM5WDWIN5SQ7YIRSRATNNKT5GEMU2QH444A, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.236192192 (538h55m35.107275237s) [noise=true , address=77.90.31.16:4760 , piece_id=XRI5EJFN7YYUG4QKLDGEFT6YAUHCUTTSXF735UDR6LZZBGMAZDJQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.236564772 (538h55m35.107647817s) [noise=true , address=145.79.227.40:4062 , piece_id=PW6QWIPQWVUETLC6DJ624G6IB4P6EM4SYB72NAMQ4V6NUKNLUYLQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.236929766 (538h55m35.108012812s) [noise=true , address=64.49.13.139:4480 , piece_id=JAJY3F7IDFFMFKCK2QJV6W74RIESLRKBLXMN27Q3C45JWEPLBZLQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.237916846 (538h55m35.108999881s) [noise=true , address=145.79.227.34:4932 , piece_id=UPXSCQ45QDGPKNJXRFJ33EAA3QENVHRVQARU5TFJTEWICZNR253Q, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.238852948 (538h55m35.109935983s) [noise=true , address=45.91.116.12:4257 , piece_id=ZI67LIAHZW4G357GWKDPTB5SMXYGNXZTTTDMNAETDS2B6SWVDAEQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239152910 (538h55m35.110235945s) [noise=true , address=145.79.227.44:4222 , piece_id=CZ2CS62CTHQFAIE3S3JO2N3VJ4XIR5YJ2LMRVLLX7I4VM2VZVQVA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239409969 (538h55m35.110493004s) [noise=true , address=145.79.227.30:4773 , piece_id=KUGJXZCNMUE7WSK3VLY4AUBSOT4H6IY4XB4NCBGAGCNUG7F7IODQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239681136 (538h55m35.110764171s) [noise=true , address=77.90.31.13:4635 , piece_id=IIOY5X2W3XW4C2OFEBRCABMYFBI24RRTYNUSKN6KJU7CTUW4KONQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239954676 (538h55m35.111037711s) [noise=true , address=45.91.116.9:4281 , piece_id=JZLDW777CBX2BP7MLOQV4NFP3H4XJVCNQXTHNSUWPVDOJLPZK36A, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.240189914 (538h55m35.111272949s) [noise=true , address=145.79.227.35:4512 , piece_id=3E3VCCC45XVT5FHFDKXSTGGAGYBEML3BKU6JJTT33LCD2YMOMBZQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.240449147 (538h55m35.111532182s) [noise=true , address=45.91.116.8:4064 , piece_id=22WJIPX4SM5NTIAW5ZRM2UKX32CTDBFCXPCCHRRWRCUCOF5CRE5Q, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.240639640 (538h55m35.111722675s) [noise=true , address=145.79.227.32:4853 , piece_id=MBNCFDBQ4SZZXV4F7J5TRMOUYUQ5IPKQCGYGGM5SOIFJ6ZYCAUYA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.241137798 (538h55m35.112220833s) [noise=true , address=77.90.31.16:4751 , piece_id=N3Z2LLMCIA2F42VXZA7IDRU4XEKGOICH6TRVERODU76MFFE4QSHA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.241383797 (538h55m35.112466842s) [noise=true , address=77.90.31.18:4836 , piece_id=BSVEFI52EKN44CMINFUSZBNOH53IW5TXLVTF5IH6ZUTNII6RRLVA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:36.592880816 (538h55m36.463963871s) [upload=472631]: Next returned nil segment Dec 28 02:24:36.592901625 (538h55m36.46398467s) [upload=472631]: Waiting for error group managing segments... Dec 28 02:24:36.946933741 (538h55m36.818016786s) [noise=true , address=64.49.13.139:4480 , piece_id=JAJY3F7IDFFMFKCK2QJV6W74RIESLRKBLXMN27Q3C45JWEPLBZLQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.128189692 (538h55m36.999272737s) [noise=true , address=145.79.227.35:4512 , piece_id=3E3VCCC45XVT5FHFDKXSTGGAGYBEML3BKU6JJTT33LCD2YMOMBZQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.160760084 (538h55m37.031843119s) [noise=true , address=145.79.227.34:4932 , piece_id=UPXSCQ45QDGPKNJXRFJ33EAA3QENVHRVQARU5TFJTEWICZNR253Q, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.288838207 (538h55m37.159921252s) [noise=true , address=77.90.31.11:4572 , piece_id=5MMQ44MXRT4XEL4X6WO54WRG2XYMX3AN3FNQ2H2XIUTKBXI3EEPA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.319996310 (538h55m37.191079345s) [noise=true , address=145.79.227.22:4512 , piece_id=D46YLTIKGHG36DQ4YBKBSUDV4GGB4WVDWNOJS27IYYLN6AYZ5QFQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.320513273 (538h55m37.191596319s) [noise=true , address=77.90.31.18:4836 , piece_id=BSVEFI52EKN44CMINFUSZBNOH53IW5TXLVTF5IH6ZUTNII6RRLVA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.336522926 (538h55m37.207605971s) [noise=true , address=145.79.227.32:4853 , piece_id=MBNCFDBQ4SZZXV4F7J5TRMOUYUQ5IPKQCGYGGM5SOIFJ6ZYCAUYA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.384868031 (538h55m37.255951066s) [noise=true , address=77.90.31.16:4751 , piece_id=N3Z2LLMCIA2F42VXZA7IDRU4XEKGOICH6TRVERODU76MFFE4QSHA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.453563554 (538h55m37.324646599s) [noise=true , address=45.91.116.8:4064 , piece_id=22WJIPX4SM5NTIAW5ZRM2UKX32CTDBFCXPCCHRRWRCUCOF5CRE5Q, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.479454839 (538h55m37.350537884s) [noise=true , address=45.91.116.12:4257 , piece_id=ZI67LIAHZW4G357GWKDPTB5SMXYGNXZTTTDMNAETDS2B6SWVDAEQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.659710174 (538h55m37.530793219s) [noise=true , address=77.90.31.13:4635 , piece_id=IIOY5X2W3XW4C2OFEBRCABMYFBI24RRTYNUSKN6KJU7CTUW4KONQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.715039128 (538h55m37.586122163s) [noise=true , address=77.90.31.16:4760 , piece_id=XRI5EJFN7YYUG4QKLDGEFT6YAUHCUTTSXF735UDR6LZZBGMAZDJQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.977473406 (538h55m37.848556441s) [noise=true , address=77.90.31.19:4874 , piece_id=RHZUXUYMJU6RKQGSDZM5WDWIN5SQ7YIRSRATNNKT5GEMU2QH444A, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:38.016302745 (538h55m37.88738578s) [noise=true , address=145.79.227.30:4773 , piece_id=KUGJXZCNMUE7WSK3VLY4AUBSOT4H6IY4XB4NCBGAGCNUG7F7IODQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:38.016326982 (538h55m37.887410027s) [seg_pos={0 0}, upload=472631]: Segment reached optimal threshold of 14 pieces. Dec 28 02:24:38.016585353 (538h55m37.887668398s) [noise=true , address=145.79.227.30:4766 , piece_id=67M2DMPAVWIQKQSWUAG73D4E4NRRSG4XEVPOUVIJPEQOUV7A6VVA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1k6zdamuKPkBdzyNZNWx4FCmaTkv33PchcwvTRR8bgcETBs4K7): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016594450 (538h55m37.887677495s) [noise=true , address=145.79.227.30:4766 , piece_id=67M2DMPAVWIQKQSWUAG73D4E4NRRSG4XEVPOUVIJPEQOUV7A6VVA, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016627093 (538h55m37.887710148s) [noise=true , address=45.91.116.9:4281 , piece_id=JZLDW777CBX2BP7MLOQV4NFP3H4XJVCNQXTHNSUWPVDOJLPZK36A, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gxf7DpJZthyR6f1PExX56tJavjhPw9Gxb4h8mtmeYKW3t3tS1): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016635258 (538h55m37.887718293s) [noise=true , address=45.91.116.9:4281 , piece_id=JZLDW777CBX2BP7MLOQV4NFP3H4XJVCNQXTHNSUWPVDOJLPZK36A, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016871298 (538h55m37.887954363s) [noise=true , address=145.79.227.44:4222 , piece_id=CZ2CS62CTHQFAIE3S3JO2N3VJ4XIR5YJ2LMRVLLX7I4VM2VZVQVA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZKW35KvrsoYRefbYeUFZo8bh4XZYycNKGEvTFjzibezjXgj3e): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016880124 (538h55m37.887963169s) [noise=true , address=145.79.227.44:4222 , piece_id=CZ2CS62CTHQFAIE3S3JO2N3VJ4XIR5YJ2LMRVLLX7I4VM2VZVQVA, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016934839 (538h55m37.888017884s) [noise=true , address=145.79.227.40:4062 , piece_id=PW6QWIPQWVUETLC6DJ624G6IB4P6EM4SYB72NAMQ4V6NUKNLUYLQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016943425 (538h55m37.88802649s) [noise=true , address=145.79.227.40:4062 , piece_id=PW6QWIPQWVUETLC6DJ624G6IB4P6EM4SYB72NAMQ4V6NUKNLUYLQ, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016968252 (538h55m37.888051297s) [seg_pos={0 0}, upload=472631]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 02:24:38.017067822 (538h55m37.888150867s) [upload=472631]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 02:24:38.017074244 (538h55m37.888157289s) [upload=472631]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 02:24:38.304306667 (538h55m38.175389712s) [upload=472631]: Done uploading segments... Dec 28 02:24:38.304329641 (538h55m38.175412686s) [upload=472631]: Upload finished. err: <nil> Duration: 51.980348639s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..........................X.........................X.......] duration: 3.556776235s trace: https://jaeger.dev.storj.io/trace/4aff4da4d82027c9 FAIL (2025-12-28 02:29:11 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.556805591s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............................X...................X.....X...] duration: 39.328326625s trace: https://jaeger.dev.storj.io/trace/1822be52d200e5b5 FAIL (2025-12-28 02:28:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1ZAfYZ4wgpZVvatj6wtHona3hTaPcLxAmXC1DT2uKQyo8nLQQ7, Piece ID: 5MMQ44MXRT4XEL4X6WO54WRG2XYMX3AN3FNQ2H2XIUTKBXI3EEPA) context deadline exceeded; piecestore: (Node ID: 124guAM82oibYZsMho67Em6Ke6yEZXwT6LPMLLSZZBLqwoFGUkn, Piece ID: 22WJIPX4SM5NTIAW5ZRM2UKX32CTDBFCXPCCHRRWRCUCOF5CRE5Q) context deadline exceeded; piecestore: (Node ID: 1e6p1TD5CqUyWdZjkrxh2kRqMuyDiMUr9eimfeayDAKwhtBrZ8, Piece ID: XRI5EJFN7YYUG4QKLDGEFT6YAUHCUTTSXF735UDR6LZZBGMAZDJQ) context deadline exceeded ======LOGS====== 5.234848204 (538h55m35.105931239s) [noise=true , address=77.90.31.11:4572 , piece_id=5MMQ44MXRT4XEL4X6WO54WRG2XYMX3AN3FNQ2H2XIUTKBXI3EEPA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.235412057 (538h55m35.106495102s) [noise=true , address=145.79.227.22:4512 , piece_id=D46YLTIKGHG36DQ4YBKBSUDV4GGB4WVDWNOJS27IYYLN6AYZ5QFQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.235820946 (538h55m35.106903981s) [noise=true , address=77.90.31.19:4874 , piece_id=RHZUXUYMJU6RKQGSDZM5WDWIN5SQ7YIRSRATNNKT5GEMU2QH444A, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.236192192 (538h55m35.107275237s) [noise=true , address=77.90.31.16:4760 , piece_id=XRI5EJFN7YYUG4QKLDGEFT6YAUHCUTTSXF735UDR6LZZBGMAZDJQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.236564772 (538h55m35.107647817s) [noise=true , address=145.79.227.40:4062 , piece_id=PW6QWIPQWVUETLC6DJ624G6IB4P6EM4SYB72NAMQ4V6NUKNLUYLQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.236929766 (538h55m35.108012812s) [noise=true , address=64.49.13.139:4480 , piece_id=JAJY3F7IDFFMFKCK2QJV6W74RIESLRKBLXMN27Q3C45JWEPLBZLQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.237916846 (538h55m35.108999881s) [noise=true , address=145.79.227.34:4932 , piece_id=UPXSCQ45QDGPKNJXRFJ33EAA3QENVHRVQARU5TFJTEWICZNR253Q, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.238852948 (538h55m35.109935983s) [noise=true , address=45.91.116.12:4257 , piece_id=ZI67LIAHZW4G357GWKDPTB5SMXYGNXZTTTDMNAETDS2B6SWVDAEQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239152910 (538h55m35.110235945s) [noise=true , address=145.79.227.44:4222 , piece_id=CZ2CS62CTHQFAIE3S3JO2N3VJ4XIR5YJ2LMRVLLX7I4VM2VZVQVA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239409969 (538h55m35.110493004s) [noise=true , address=145.79.227.30:4773 , piece_id=KUGJXZCNMUE7WSK3VLY4AUBSOT4H6IY4XB4NCBGAGCNUG7F7IODQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239681136 (538h55m35.110764171s) [noise=true , address=77.90.31.13:4635 , piece_id=IIOY5X2W3XW4C2OFEBRCABMYFBI24RRTYNUSKN6KJU7CTUW4KONQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239954676 (538h55m35.111037711s) [noise=true , address=45.91.116.9:4281 , piece_id=JZLDW777CBX2BP7MLOQV4NFP3H4XJVCNQXTHNSUWPVDOJLPZK36A, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.240189914 (538h55m35.111272949s) [noise=true , address=145.79.227.35:4512 , piece_id=3E3VCCC45XVT5FHFDKXSTGGAGYBEML3BKU6JJTT33LCD2YMOMBZQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.240449147 (538h55m35.111532182s) [noise=true , address=45.91.116.8:4064 , piece_id=22WJIPX4SM5NTIAW5ZRM2UKX32CTDBFCXPCCHRRWRCUCOF5CRE5Q, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.240639640 (538h55m35.111722675s) [noise=true , address=145.79.227.32:4853 , piece_id=MBNCFDBQ4SZZXV4F7J5TRMOUYUQ5IPKQCGYGGM5SOIFJ6ZYCAUYA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.241137798 (538h55m35.112220833s) [noise=true , address=77.90.31.16:4751 , piece_id=N3Z2LLMCIA2F42VXZA7IDRU4XEKGOICH6TRVERODU76MFFE4QSHA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.241383797 (538h55m35.112466842s) [noise=true , address=77.90.31.18:4836 , piece_id=BSVEFI52EKN44CMINFUSZBNOH53IW5TXLVTF5IH6ZUTNII6RRLVA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:36.592880816 (538h55m36.463963871s) [upload=472631]: Next returned nil segment Dec 28 02:24:36.592901625 (538h55m36.46398467s) [upload=472631]: Waiting for error group managing segments... Dec 28 02:24:36.946933741 (538h55m36.818016786s) [noise=true , address=64.49.13.139:4480 , piece_id=JAJY3F7IDFFMFKCK2QJV6W74RIESLRKBLXMN27Q3C45JWEPLBZLQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.128189692 (538h55m36.999272737s) [noise=true , address=145.79.227.35:4512 , piece_id=3E3VCCC45XVT5FHFDKXSTGGAGYBEML3BKU6JJTT33LCD2YMOMBZQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.160760084 (538h55m37.031843119s) [noise=true , address=145.79.227.34:4932 , piece_id=UPXSCQ45QDGPKNJXRFJ33EAA3QENVHRVQARU5TFJTEWICZNR253Q, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.288838207 (538h55m37.159921252s) [noise=true , address=77.90.31.11:4572 , piece_id=5MMQ44MXRT4XEL4X6WO54WRG2XYMX3AN3FNQ2H2XIUTKBXI3EEPA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.319996310 (538h55m37.191079345s) [noise=true , address=145.79.227.22:4512 , piece_id=D46YLTIKGHG36DQ4YBKBSUDV4GGB4WVDWNOJS27IYYLN6AYZ5QFQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.320513273 (538h55m37.191596319s) [noise=true , address=77.90.31.18:4836 , piece_id=BSVEFI52EKN44CMINFUSZBNOH53IW5TXLVTF5IH6ZUTNII6RRLVA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.336522926 (538h55m37.207605971s) [noise=true , address=145.79.227.32:4853 , piece_id=MBNCFDBQ4SZZXV4F7J5TRMOUYUQ5IPKQCGYGGM5SOIFJ6ZYCAUYA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.384868031 (538h55m37.255951066s) [noise=true , address=77.90.31.16:4751 , piece_id=N3Z2LLMCIA2F42VXZA7IDRU4XEKGOICH6TRVERODU76MFFE4QSHA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.453563554 (538h55m37.324646599s) [noise=true , address=45.91.116.8:4064 , piece_id=22WJIPX4SM5NTIAW5ZRM2UKX32CTDBFCXPCCHRRWRCUCOF5CRE5Q, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.479454839 (538h55m37.350537884s) [noise=true , address=45.91.116.12:4257 , piece_id=ZI67LIAHZW4G357GWKDPTB5SMXYGNXZTTTDMNAETDS2B6SWVDAEQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.659710174 (538h55m37.530793219s) [noise=true , address=77.90.31.13:4635 , piece_id=IIOY5X2W3XW4C2OFEBRCABMYFBI24RRTYNUSKN6KJU7CTUW4KONQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.715039128 (538h55m37.586122163s) [noise=true , address=77.90.31.16:4760 , piece_id=XRI5EJFN7YYUG4QKLDGEFT6YAUHCUTTSXF735UDR6LZZBGMAZDJQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.977473406 (538h55m37.848556441s) [noise=true , address=77.90.31.19:4874 , piece_id=RHZUXUYMJU6RKQGSDZM5WDWIN5SQ7YIRSRATNNKT5GEMU2QH444A, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:38.016302745 (538h55m37.88738578s) [noise=true , address=145.79.227.30:4773 , piece_id=KUGJXZCNMUE7WSK3VLY4AUBSOT4H6IY4XB4NCBGAGCNUG7F7IODQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:38.016326982 (538h55m37.887410027s) [seg_pos={0 0}, upload=472631]: Segment reached optimal threshold of 14 pieces. Dec 28 02:24:38.016585353 (538h55m37.887668398s) [noise=true , address=145.79.227.30:4766 , piece_id=67M2DMPAVWIQKQSWUAG73D4E4NRRSG4XEVPOUVIJPEQOUV7A6VVA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1k6zdamuKPkBdzyNZNWx4FCmaTkv33PchcwvTRR8bgcETBs4K7): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016594450 (538h55m37.887677495s) [noise=true , address=145.79.227.30:4766 , piece_id=67M2DMPAVWIQKQSWUAG73D4E4NRRSG4XEVPOUVIJPEQOUV7A6VVA, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016627093 (538h55m37.887710148s) [noise=true , address=45.91.116.9:4281 , piece_id=JZLDW777CBX2BP7MLOQV4NFP3H4XJVCNQXTHNSUWPVDOJLPZK36A, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gxf7DpJZthyR6f1PExX56tJavjhPw9Gxb4h8mtmeYKW3t3tS1): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016635258 (538h55m37.887718293s) [noise=true , address=45.91.116.9:4281 , piece_id=JZLDW777CBX2BP7MLOQV4NFP3H4XJVCNQXTHNSUWPVDOJLPZK36A, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016871298 (538h55m37.887954363s) [noise=true , address=145.79.227.44:4222 , piece_id=CZ2CS62CTHQFAIE3S3JO2N3VJ4XIR5YJ2LMRVLLX7I4VM2VZVQVA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZKW35KvrsoYRefbYeUFZo8bh4XZYycNKGEvTFjzibezjXgj3e): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016880124 (538h55m37.887963169s) [noise=true , address=145.79.227.44:4222 , piece_id=CZ2CS62CTHQFAIE3S3JO2N3VJ4XIR5YJ2LMRVLLX7I4VM2VZVQVA, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016934839 (538h55m37.888017884s) [noise=true , address=145.79.227.40:4062 , piece_id=PW6QWIPQWVUETLC6DJ624G6IB4P6EM4SYB72NAMQ4V6NUKNLUYLQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016943425 (538h55m37.88802649s) [noise=true , address=145.79.227.40:4062 , piece_id=PW6QWIPQWVUETLC6DJ624G6IB4P6EM4SYB72NAMQ4V6NUKNLUYLQ, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016968252 (538h55m37.888051297s) [seg_pos={0 0}, upload=472631]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 02:24:38.017067822 (538h55m37.888150867s) [upload=472631]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 02:24:38.017074244 (538h55m37.888157289s) [upload=472631]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 02:24:38.304306667 (538h55m38.175389712s) [upload=472631]: Done uploading segments... Dec 28 02:24:38.304329641 (538h55m38.175412686s) [upload=472631]: Upload finished. err: <nil> Duration: 39.328349508s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 33 out of 60 is failed (threshold=3) [.XXX.X.XX.XX.X..X.X.X.X.X..X...XX..X.XXX.XXX.X.....XXX.XXXXX] duration: 30.005135653s trace: https://jaeger.dev.storj.io/trace/5945b669b34116f7 FAIL (2025-12-28 02:28:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 02:27:45.764153255 (538h58m45.63523629s) [upload=472675]: Starting upload Dec 28 02:27:45.764206967 (538h58m45.635290002s) [upload=472675]: Done starting upload Dec 28 02:27:45.764297199 (538h58m45.635380244s) [upload=472675]: Uploading segments... Dec 28 02:27:45.764481199 (538h58m45.635564244s) [upload=472675]: Got next segment. Inline: false Dec 28 02:27:45.764530072 (538h58m45.635613117s) [upload=472675]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 02:27:45.764536765 (538h58m45.6356198s) [upload=472675]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 02:27:46.010241883 (538h58m45.881324928s) [seg_pos={0 0}, upload=472675]: Begin upload segment... Dec 28 02:27:46.010292839 (538h58m45.881375875s) [seg_pos={0 0}, upload=472675]: Done begin upload segment. Dec 28 02:27:46.010346392 (538h58m45.881429437s) [noise=true , address=45.91.116.7:4012 , piece_id=GE7G55ZM5Z2UTP6PPZQVIHGEN4CWBZZQFWFY5EDLK35NXFQDEDCA, seg_pos={0 0}, upload=472675]: Uploading piece... Dec 28 02:27:46.010383833 (538h58m45.881466888s) [noise=true , address=45.91.116.4:4940 , piece_id=HZLTIHXAVHRJF3CFO45DL762M77JCWOIEA7EY3SMRO5VQFQTXBNQ, seg_pos={0 0}, upload=472675]: Uploading piece... Dec 28 02:27:46.011021046 (538h58m45.882104091s) [noise=true , address=145.79.227.13:4804 , piece_id=LNUZDE6KKINL2W3F6QCINLP6G3OY3QUH3MER6P3ONBUZ6M4M4BAQ, seg_pos={0 0}, upload=472675]: Uploading piece... Dec 28 02:27:46.011462497 (538h58m45.882545532s) [noise=true , address=145.79.227.43:4159 , piece_id=MHNIHNU77BYT5LKNVPLARHB53T5G427OXZWWKJ5LNJKMHYOTPFXA, seg_pos={0 0}, upload=472675]: Uploading piece... Dec 28 02:27:46.011908926 (538h58m45.882991971s) [noise=true , address=145.79.227.46:4601 , piece_id=L2I2PEXRCZ7GAYO73NJMYVBU2CNP36H6PHAN7FQRAVYZBTSTL4OA, seg_pos={0 0}, upload=472675]: Uploading piece... Dec 28 02:27:46.012405903 (538h58m45.883488948s) [noise=true , address=145.79.227.9:4365 , piece_id=AKHFF7PQYR223SIIUYBL6M3SLTD6ETLWHCZWKRPNHER4PEKSEF5Q, seg_pos={0 0}, upload=472675]: Uploading piece... Dec 28 02:27:46.012914692 (538h58m45.883997737s) [noise=true , address=45.91.116.10:4082 , piece_id=WLSEPJA3N4ZL73CPSA2E6AJMVHIHDTZ6524LPQYICHCONPVGENUQ, seg_pos={0 0}, upload=472675]: Uploading piece... Dec 28 02:27:46.014228032 (538h58m45.885311067s) [noise=true , address=145.79.227.31:4797 , piece_id=SYCUZGZER6HCSVD2WRR3UXIGDNRL74MQWJXRMNK4BBWV4EAFB4RA, seg_pos={0 0}, upload=472675]: Uploading piece... Dec 28 02:27:46.014911784 (538h58m45.885994819s) [noise=true , address=45.91.116.5:4298 , piece_id=XI72RNHESQMVBTKFAN4GVE2Y7FCG6X4OPYRZ4R7HGPSJ7AV2QJWQ, seg_pos={0 0}, upload=472675]: Uploading piece... Dec 28 02:27:46.015672573 (538h58m45.886755608s) [noise=true , address=145.79.227.36:4517 , piece_id=DLUHQRIAHNNMOUPE7UT4MUWVAFJ23ZXBFVDNEN6VJVTMGVKUNTIA, seg_pos={0 0}, upload=472675]: Uploading piece... Dec 28 02:27:46.017060064 (538h58m45.888143099s) [noise=true , address=145.79.227.11:4730 , piece_id=IQ4N2UP3M4HHIJNJQL4TGJQV3MF7KZSMA6OSDH67KQEZ2UMERBUA, seg_pos={0 0}, upload=472675]: Uploading piece... Dec 28 02:27:46.017178690 (538h58m45.888261736s) [noise=true , address=145.79.227.19:4077 , piece_id=ELOT7V5WJX5YQ45IBUUTCJ2WBZTPTKNAKFQHFCYAC2ZZIGF4AOQA, seg_pos={0 0}, upload=472675]: Uploading piece... Dec 28 02:27:46.018159918 (538h58m45.889242953s) [noise=true , address=45.91.116.8:4048 , piece_id=N7XFCFSDID7LTLLLCYF2M4ZGLXR6B3ITAWAM6DNQMZRS2D2D6FXQ, seg_pos={0 0}, upload=472675]: Uploading piece... Dec 28 02:27:46.021447548 (538h58m45.892530583s) [noise=true , address=45.91.116.6:4973 , piece_id=DZ6H3TYWLWN6LTTFUU3CM2GNOXRJLPHS2Z2JHMTBT7ZHKOOWEAMA, seg_pos={0 0}, upload=472675]: Uploading piece... Dec 28 02:27:46.022470204 (538h58m45.89355325s) [noise=true , address=145.79.227.41:4086 , piece_id=4EKVCE7KGHRZUPZK4NQSTD47GXRKDWIIKS37LM2F6HSHVKVYM7EA, seg_pos={0 0}, upload=472675]: Uploading piece... Dec 28 02:27:46.024865866 (538h58m45.895948911s) [noise=true , address=145.79.227.51:4411 , piece_id=2OLYCWCRU3M3DE46WO7LPIXVWAOGW4STR5SWFA5RZ67IMXJ4AKBQ, seg_pos={0 0}, upload=472675]: Uploading piece... Dec 28 02:27:47.747039370 (538h58m47.618122415s) [upload=472675]: Next returned nil segment Dec 28 02:27:47.747053477 (538h58m47.618136512s) [upload=472675]: Waiting for error group managing segments... Dec 28 02:27:48.031904617 (538h58m47.902987662s) [noise=true , address=145.79.227.51:4411 , piece_id=2OLYCWCRU3M3DE46WO7LPIXVWAOGW4STR5SWFA5RZ67IMXJ4AKBQ, seg_pos={0 0}, upload=472675]: Done uploading piece. err: <nil> Dec 28 02:27:48.034862538 (538h58m47.905945573s) [noise=true , address=145.79.227.19:4077 , piece_id=ELOT7V5WJX5YQ45IBUUTCJ2WBZTPTKNAKFQHFCYAC2ZZIGF4AOQA, seg_pos={0 0}, upload=472675]: Done uploading piece. err: <nil> Dec 28 02:27:48.040514920 (538h58m47.911597955s) [noise=true , address=145.79.227.31:4797 , piece_id=SYCUZGZER6HCSVD2WRR3UXIGDNRL74MQWJXRMNK4BBWV4EAFB4RA, seg_pos={0 0}, upload=472675]: Done uploading piece. err: <nil> Dec 28 02:27:48.053299792 (538h58m47.924382837s) [noise=true , address=145.79.227.13:4804 , piece_id=LNUZDE6KKINL2W3F6QCINLP6G3OY3QUH3MER6P3ONBUZ6M4M4BAQ, seg_pos={0 0}, upload=472675]: Done uploading piece. err: <nil> Dec 28 02:27:48.065741490 (538h58m47.936824525s) [noise=true , address=145.79.227.36:4517 , piece_id=DLUHQRIAHNNMOUPE7UT4MUWVAFJ23ZXBFVDNEN6VJVTMGVKUNTIA, seg_pos={0 0}, upload=472675]: Done uploading piece. err: <nil> Dec 28 02:27:48.356659991 (538h58m48.227743026s) [noise=true , address=45.91.116.5:4298 , piece_id=XI72RNHESQMVBTKFAN4GVE2Y7FCG6X4OPYRZ4R7HGPSJ7AV2QJWQ, seg_pos={0 0}, upload=472675]: Done uploading piece. err: <nil> Dec 28 02:27:48.371562204 (538h58m48.242645249s) [noise=true , address=45.91.116.4:4940 , piece_id=HZLTIHXAVHRJF3CFO45DL762M77JCWOIEA7EY3SMRO5VQFQTXBNQ, seg_pos={0 0}, upload=472675]: Done uploading piece. err: <nil> Dec 28 02:27:48.385231069 (538h58m48.256314114s) [noise=true , address=45.91.116.6:4973 , piece_id=DZ6H3TYWLWN6LTTFUU3CM2GNOXRJLPHS2Z2JHMTBT7ZHKOOWEAMA, seg_pos={0 0}, upload=472675]: Done uploading piece. err: <nil> Dec 28 02:27:48.600519596 (538h58m48.471602631s) [noise=true , address=145.79.227.9:4365 , piece_id=AKHFF7PQYR223SIIUYBL6M3SLTD6ETLWHCZWKRPNHER4PEKSEF5Q, seg_pos={0 0}, upload=472675]: Done uploading piece. err: <nil> Dec 28 02:27:48.635828942 (538h58m48.506911977s) [noise=true , address=145.79.227.41:4086 , piece_id=4EKVCE7KGHRZUPZK4NQSTD47GXRKDWIIKS37LM2F6HSHVKVYM7EA, seg_pos={0 0}, upload=472675]: Done uploading piece. err: <nil> Dec 28 02:27:48.755298896 (538h58m48.626381931s) [noise=true , address=45.91.116.7:4012 , piece_id=GE7G55ZM5Z2UTP6PPZQVIHGEN4CWBZZQFWFY5EDLK35NXFQDEDCA, seg_pos={0 0}, upload=472675]: Done uploading piece. err: <nil> Dec 28 02:27:48.901596106 (538h58m48.772679152s) [noise=true , address=145.79.227.46:4601 , piece_id=L2I2PEXRCZ7GAYO73NJMYVBU2CNP36H6PHAN7FQRAVYZBTSTL4OA, seg_pos={0 0}, upload=472675]: Done uploading piece. err: <nil> Dec 28 02:27:48.929057752 (538h58m48.800140787s) [noise=true , address=145.79.227.11:4730 , piece_id=IQ4N2UP3M4HHIJNJQL4TGJQV3MF7KZSMA6OSDH67KQEZ2UMERBUA, seg_pos={0 0}, upload=472675]: Done uploading piece. err: <nil> Dec 28 02:27:48.929078892 (538h58m48.800161927s) [seg_pos={0 0}, upload=472675]: Segment reached optimal threshold of 13 pieces. Dec 28 02:27:48.929296516 (538h58m48.800379551s) [noise=true , address=45.91.116.8:4048 , piece_id=N7XFCFSDID7LTLLLCYF2M4ZGLXR6B3ITAWAM6DNQMZRS2D2D6FXQ, seg_pos={0 0}, upload=472675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 28 02:27:48.929302108 (538h58m48.800385143s) [noise=true , address=45.91.116.8:4048 , piece_id=N7XFCFSDID7LTLLLCYF2M4ZGLXR6B3ITAWAM6DNQMZRS2D2D6FXQ, seg_pos={0 0}, upload=472675]: Long tail cancellation due to optimal threshold Dec 28 02:27:48.929379224 (538h58m48.800462269s) [noise=true , address=45.91.116.10:4082 , piece_id=WLSEPJA3N4ZL73CPSA2E6AJMVHIHDTZ6524LPQYICHCONPVGENUQ, seg_pos={0 0}, upload=472675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 28 02:27:48.929394092 (538h58m48.800477127s) [noise=true , address=45.91.116.10:4082 , piece_id=WLSEPJA3N4ZL73CPSA2E6AJMVHIHDTZ6524LPQYICHCONPVGENUQ, seg_pos={0 0}, upload=472675]: Long tail cancellation due to optimal threshold Dec 28 02:27:48.929510765 (538h58m48.8005938s) [noise=true , address=145.79.227.43:4159 , piece_id=MHNIHNU77BYT5LKNVPLARHB53T5G427OXZWWKJ5LNJKMHYOTPFXA, seg_pos={0 0}, upload=472675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Dec 28 02:27:48.929516095 (538h58m48.80059913s) [noise=true , address=145.79.227.43:4159 , piece_id=MHNIHNU77BYT5LKNVPLARHB53T5G427OXZWWKJ5LNJKMHYOTPFXA, seg_pos={0 0}, upload=472675]: Long tail cancellation due to optimal threshold Dec 28 02:27:48.929541412 (538h58m48.800624447s) [seg_pos={0 0}, upload=472675]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 02:27:48.929614151 (538h58m48.800697186s) [upload=472675]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 02:27:48.929618821 (538h58m48.800701856s) [upload=472675]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 02:27:49.221464326 (538h58m49.092547361s) [upload=472675]: Done uploading segments... Dec 28 02:27:49.221485787 (538h58m49.092568822s) [upload=472675]: Upload finished. err: <nil> Duration: 30.005154068s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...........................X.........................X......] duration: 3.550332381s trace: https://jaeger.dev.storj.io/trace/4aff4da4d82027c9 FAIL (2025-12-28 02:28:01 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.550361536s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [............................X.........................X.....] duration: 3.339660652s trace: https://jaeger.dev.storj.io/trace/4aff4da4d82027c9 FAIL (2025-12-28 02:27:42 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.339687103s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.............................X.........................X....] duration: 3.398490722s trace: https://jaeger.dev.storj.io/trace/4aff4da4d82027c9 FAIL (2025-12-28 02:27:26 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.398544934s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............................X...................X.....X..] duration: 42.120311072s trace: https://jaeger.dev.storj.io/trace/1822be52d200e5b5 FAIL (2025-12-28 02:27:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1ZAfYZ4wgpZVvatj6wtHona3hTaPcLxAmXC1DT2uKQyo8nLQQ7, Piece ID: 5MMQ44MXRT4XEL4X6WO54WRG2XYMX3AN3FNQ2H2XIUTKBXI3EEPA) context deadline exceeded; piecestore: (Node ID: 124guAM82oibYZsMho67Em6Ke6yEZXwT6LPMLLSZZBLqwoFGUkn, Piece ID: 22WJIPX4SM5NTIAW5ZRM2UKX32CTDBFCXPCCHRRWRCUCOF5CRE5Q) context deadline exceeded; piecestore: (Node ID: 1e6p1TD5CqUyWdZjkrxh2kRqMuyDiMUr9eimfeayDAKwhtBrZ8, Piece ID: XRI5EJFN7YYUG4QKLDGEFT6YAUHCUTTSXF735UDR6LZZBGMAZDJQ) context deadline exceeded ======LOGS====== 5.234848204 (538h55m35.105931239s) [noise=true , address=77.90.31.11:4572 , piece_id=5MMQ44MXRT4XEL4X6WO54WRG2XYMX3AN3FNQ2H2XIUTKBXI3EEPA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.235412057 (538h55m35.106495102s) [noise=true , address=145.79.227.22:4512 , piece_id=D46YLTIKGHG36DQ4YBKBSUDV4GGB4WVDWNOJS27IYYLN6AYZ5QFQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.235820946 (538h55m35.106903981s) [noise=true , address=77.90.31.19:4874 , piece_id=RHZUXUYMJU6RKQGSDZM5WDWIN5SQ7YIRSRATNNKT5GEMU2QH444A, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.236192192 (538h55m35.107275237s) [noise=true , address=77.90.31.16:4760 , piece_id=XRI5EJFN7YYUG4QKLDGEFT6YAUHCUTTSXF735UDR6LZZBGMAZDJQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.236564772 (538h55m35.107647817s) [noise=true , address=145.79.227.40:4062 , piece_id=PW6QWIPQWVUETLC6DJ624G6IB4P6EM4SYB72NAMQ4V6NUKNLUYLQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.236929766 (538h55m35.108012812s) [noise=true , address=64.49.13.139:4480 , piece_id=JAJY3F7IDFFMFKCK2QJV6W74RIESLRKBLXMN27Q3C45JWEPLBZLQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.237916846 (538h55m35.108999881s) [noise=true , address=145.79.227.34:4932 , piece_id=UPXSCQ45QDGPKNJXRFJ33EAA3QENVHRVQARU5TFJTEWICZNR253Q, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.238852948 (538h55m35.109935983s) [noise=true , address=45.91.116.12:4257 , piece_id=ZI67LIAHZW4G357GWKDPTB5SMXYGNXZTTTDMNAETDS2B6SWVDAEQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239152910 (538h55m35.110235945s) [noise=true , address=145.79.227.44:4222 , piece_id=CZ2CS62CTHQFAIE3S3JO2N3VJ4XIR5YJ2LMRVLLX7I4VM2VZVQVA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239409969 (538h55m35.110493004s) [noise=true , address=145.79.227.30:4773 , piece_id=KUGJXZCNMUE7WSK3VLY4AUBSOT4H6IY4XB4NCBGAGCNUG7F7IODQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239681136 (538h55m35.110764171s) [noise=true , address=77.90.31.13:4635 , piece_id=IIOY5X2W3XW4C2OFEBRCABMYFBI24RRTYNUSKN6KJU7CTUW4KONQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239954676 (538h55m35.111037711s) [noise=true , address=45.91.116.9:4281 , piece_id=JZLDW777CBX2BP7MLOQV4NFP3H4XJVCNQXTHNSUWPVDOJLPZK36A, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.240189914 (538h55m35.111272949s) [noise=true , address=145.79.227.35:4512 , piece_id=3E3VCCC45XVT5FHFDKXSTGGAGYBEML3BKU6JJTT33LCD2YMOMBZQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.240449147 (538h55m35.111532182s) [noise=true , address=45.91.116.8:4064 , piece_id=22WJIPX4SM5NTIAW5ZRM2UKX32CTDBFCXPCCHRRWRCUCOF5CRE5Q, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.240639640 (538h55m35.111722675s) [noise=true , address=145.79.227.32:4853 , piece_id=MBNCFDBQ4SZZXV4F7J5TRMOUYUQ5IPKQCGYGGM5SOIFJ6ZYCAUYA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.241137798 (538h55m35.112220833s) [noise=true , address=77.90.31.16:4751 , piece_id=N3Z2LLMCIA2F42VXZA7IDRU4XEKGOICH6TRVERODU76MFFE4QSHA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.241383797 (538h55m35.112466842s) [noise=true , address=77.90.31.18:4836 , piece_id=BSVEFI52EKN44CMINFUSZBNOH53IW5TXLVTF5IH6ZUTNII6RRLVA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:36.592880816 (538h55m36.463963871s) [upload=472631]: Next returned nil segment Dec 28 02:24:36.592901625 (538h55m36.46398467s) [upload=472631]: Waiting for error group managing segments... Dec 28 02:24:36.946933741 (538h55m36.818016786s) [noise=true , address=64.49.13.139:4480 , piece_id=JAJY3F7IDFFMFKCK2QJV6W74RIESLRKBLXMN27Q3C45JWEPLBZLQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.128189692 (538h55m36.999272737s) [noise=true , address=145.79.227.35:4512 , piece_id=3E3VCCC45XVT5FHFDKXSTGGAGYBEML3BKU6JJTT33LCD2YMOMBZQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.160760084 (538h55m37.031843119s) [noise=true , address=145.79.227.34:4932 , piece_id=UPXSCQ45QDGPKNJXRFJ33EAA3QENVHRVQARU5TFJTEWICZNR253Q, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.288838207 (538h55m37.159921252s) [noise=true , address=77.90.31.11:4572 , piece_id=5MMQ44MXRT4XEL4X6WO54WRG2XYMX3AN3FNQ2H2XIUTKBXI3EEPA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.319996310 (538h55m37.191079345s) [noise=true , address=145.79.227.22:4512 , piece_id=D46YLTIKGHG36DQ4YBKBSUDV4GGB4WVDWNOJS27IYYLN6AYZ5QFQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.320513273 (538h55m37.191596319s) [noise=true , address=77.90.31.18:4836 , piece_id=BSVEFI52EKN44CMINFUSZBNOH53IW5TXLVTF5IH6ZUTNII6RRLVA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.336522926 (538h55m37.207605971s) [noise=true , address=145.79.227.32:4853 , piece_id=MBNCFDBQ4SZZXV4F7J5TRMOUYUQ5IPKQCGYGGM5SOIFJ6ZYCAUYA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.384868031 (538h55m37.255951066s) [noise=true , address=77.90.31.16:4751 , piece_id=N3Z2LLMCIA2F42VXZA7IDRU4XEKGOICH6TRVERODU76MFFE4QSHA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.453563554 (538h55m37.324646599s) [noise=true , address=45.91.116.8:4064 , piece_id=22WJIPX4SM5NTIAW5ZRM2UKX32CTDBFCXPCCHRRWRCUCOF5CRE5Q, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.479454839 (538h55m37.350537884s) [noise=true , address=45.91.116.12:4257 , piece_id=ZI67LIAHZW4G357GWKDPTB5SMXYGNXZTTTDMNAETDS2B6SWVDAEQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.659710174 (538h55m37.530793219s) [noise=true , address=77.90.31.13:4635 , piece_id=IIOY5X2W3XW4C2OFEBRCABMYFBI24RRTYNUSKN6KJU7CTUW4KONQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.715039128 (538h55m37.586122163s) [noise=true , address=77.90.31.16:4760 , piece_id=XRI5EJFN7YYUG4QKLDGEFT6YAUHCUTTSXF735UDR6LZZBGMAZDJQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.977473406 (538h55m37.848556441s) [noise=true , address=77.90.31.19:4874 , piece_id=RHZUXUYMJU6RKQGSDZM5WDWIN5SQ7YIRSRATNNKT5GEMU2QH444A, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:38.016302745 (538h55m37.88738578s) [noise=true , address=145.79.227.30:4773 , piece_id=KUGJXZCNMUE7WSK3VLY4AUBSOT4H6IY4XB4NCBGAGCNUG7F7IODQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:38.016326982 (538h55m37.887410027s) [seg_pos={0 0}, upload=472631]: Segment reached optimal threshold of 14 pieces. Dec 28 02:24:38.016585353 (538h55m37.887668398s) [noise=true , address=145.79.227.30:4766 , piece_id=67M2DMPAVWIQKQSWUAG73D4E4NRRSG4XEVPOUVIJPEQOUV7A6VVA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1k6zdamuKPkBdzyNZNWx4FCmaTkv33PchcwvTRR8bgcETBs4K7): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016594450 (538h55m37.887677495s) [noise=true , address=145.79.227.30:4766 , piece_id=67M2DMPAVWIQKQSWUAG73D4E4NRRSG4XEVPOUVIJPEQOUV7A6VVA, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016627093 (538h55m37.887710148s) [noise=true , address=45.91.116.9:4281 , piece_id=JZLDW777CBX2BP7MLOQV4NFP3H4XJVCNQXTHNSUWPVDOJLPZK36A, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gxf7DpJZthyR6f1PExX56tJavjhPw9Gxb4h8mtmeYKW3t3tS1): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016635258 (538h55m37.887718293s) [noise=true , address=45.91.116.9:4281 , piece_id=JZLDW777CBX2BP7MLOQV4NFP3H4XJVCNQXTHNSUWPVDOJLPZK36A, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016871298 (538h55m37.887954363s) [noise=true , address=145.79.227.44:4222 , piece_id=CZ2CS62CTHQFAIE3S3JO2N3VJ4XIR5YJ2LMRVLLX7I4VM2VZVQVA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZKW35KvrsoYRefbYeUFZo8bh4XZYycNKGEvTFjzibezjXgj3e): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016880124 (538h55m37.887963169s) [noise=true , address=145.79.227.44:4222 , piece_id=CZ2CS62CTHQFAIE3S3JO2N3VJ4XIR5YJ2LMRVLLX7I4VM2VZVQVA, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016934839 (538h55m37.888017884s) [noise=true , address=145.79.227.40:4062 , piece_id=PW6QWIPQWVUETLC6DJ624G6IB4P6EM4SYB72NAMQ4V6NUKNLUYLQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016943425 (538h55m37.88802649s) [noise=true , address=145.79.227.40:4062 , piece_id=PW6QWIPQWVUETLC6DJ624G6IB4P6EM4SYB72NAMQ4V6NUKNLUYLQ, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016968252 (538h55m37.888051297s) [seg_pos={0 0}, upload=472631]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 02:24:38.017067822 (538h55m37.888150867s) [upload=472631]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 02:24:38.017074244 (538h55m37.888157289s) [upload=472631]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 02:24:38.304306667 (538h55m38.175389712s) [upload=472631]: Done uploading segments... Dec 28 02:24:38.304329641 (538h55m38.175412686s) [upload=472631]: Upload finished. err: <nil> Duration: 42.120348023s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 33 out of 60 is failed (threshold=3) [X.XXX.X.XX.XX.X..X.X.X.X.X..X...XX..X.XXX.XXX.X.....XXX.XXXX] duration: 30.004627903s trace: https://jaeger.dev.storj.io/trace/68764ce3ecf599fb FAIL (2025-12-28 02:27:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== c 28 02:26:47.196166002 (538h57m47.067249037s) [upload=472662]: Starting upload Dec 28 02:26:47.196226747 (538h57m47.067309783s) [upload=472662]: Done starting upload Dec 28 02:26:47.196345112 (538h57m47.067428158s) [upload=472662]: Uploading segments... Dec 28 02:26:47.196433872 (538h57m47.067516917s) [upload=472662]: Got next segment. Inline: false Dec 28 02:26:47.196506260 (538h57m47.067589305s) [upload=472662]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 02:26:47.196512111 (538h57m47.067595156s) [upload=472662]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 02:26:47.436015993 (538h57m47.307099028s) [seg_pos={0 0}, upload=472662]: Begin upload segment... Dec 28 02:26:47.436071979 (538h57m47.307155024s) [seg_pos={0 0}, upload=472662]: Done begin upload segment. Dec 28 02:26:47.436103039 (538h57m47.307186074s) [noise=true , address=145.79.227.2:4114 , piece_id=4PFLUU4AKGCM2OJCTR4OOOOHX4YJM5J4VKHCDU2SS4IAZEI4U22A, seg_pos={0 0}, upload=472662]: Uploading piece... Dec 28 02:26:47.436226564 (538h57m47.307309599s) [noise=true , address=145.79.227.7:4295 , piece_id=OVLLLNVXYUTVWDITBCAECE7POFOT5BNBCPIAE27V4GNMB2NVQDXQ, seg_pos={0 0}, upload=472662]: Uploading piece... Dec 28 02:26:47.436821216 (538h57m47.307904282s) [noise=true , address=77.90.31.11:4553 , piece_id=OKW2XGVS45O276JX2PDXZOESXBWXBWB6P7COHOZF5LRUDUPCSRFA, seg_pos={0 0}, upload=472662]: Uploading piece... Dec 28 02:26:47.437290159 (538h57m47.308373194s) [noise=true , address=145.79.227.4:4186 , piece_id=3HUD4SNDNV4BGGJN2SEFXOO3GY7ZHJ2IK5LYRSM573ZP4EGL6WKQ, seg_pos={0 0}, upload=472662]: Uploading piece... Dec 28 02:26:47.437609166 (538h57m47.308692201s) [noise=true , address=45.91.116.13:4154 , piece_id=Y3JKBJWZER5Q34WVMTQ7MIGIT6AIX7TD6DPXZXZ6ZUU3HGRZOW6Q, seg_pos={0 0}, upload=472662]: Uploading piece... Dec 28 02:26:47.438069462 (538h57m47.309152498s) [noise=true , address=45.91.116.11:4118 , piece_id=KPD5TZAPY4VBNTL5P4YLHNLTFH4IOSDZF7YGDNWE2ESHZ4JPKNHA, seg_pos={0 0}, upload=472662]: Uploading piece... Dec 28 02:26:47.438308838 (538h57m47.309391873s) [noise=true , address=145.79.227.41:4086 , piece_id=FV6GTIKBNIJQFWOWUS7E6LS3J5V675VPL2JVJHN5RZSVYHPJOBKQ, seg_pos={0 0}, upload=472662]: Uploading piece... Dec 28 02:26:47.438997470 (538h57m47.310080505s) [noise=true , address=45.91.116.14:4191 , piece_id=YGJIYP6S2T3YI7YLB2KSAHJTXYO6LEQA423MPQOJSCBTVDPISN2A, seg_pos={0 0}, upload=472662]: Uploading piece... Dec 28 02:26:47.439278525 (538h57m47.31036156s) [noise=true , address=145.79.227.9:4365 , piece_id=CHS62KWTEREH7A3VK7MQBFVZ2JIOQANBYSLGH7NMO36PMRRVMOJA, seg_pos={0 0}, upload=472662]: Uploading piece... Dec 28 02:26:47.439880000 (538h57m47.310963035s) [noise=true , address=145.79.227.15:4874 , piece_id=XBHDKLDXT2YKNE2Q6DYL2CWFBGO6ESRHBAR7AMG3UTQTDTVFVT5Q, seg_pos={0 0}, upload=472662]: Uploading piece... Dec 28 02:26:47.440146889 (538h57m47.311229934s) [noise=true , address=145.79.227.52:4445 , piece_id=6VV7RXEZGMXQEUHHNTYVMHHNN7NMAVIEQEGR5BV5Q3NGD3WZ4JIA, seg_pos={0 0}, upload=472662]: Uploading piece... Dec 28 02:26:47.440842863 (538h57m47.311925908s) [noise=true , address=145.79.227.11:4729 , piece_id=RYLED7ZV45T6FDXV5MESNLZHGTHXHAWDVZSAR7CFB5ZJALZ6MX4Q, seg_pos={0 0}, upload=472662]: Uploading piece... Dec 28 02:26:47.440998830 (538h57m47.312081865s) [noise=true , address=45.91.116.7:4011 , piece_id=DB2X67NJV5VEP2YQAQX7Q6HRDZGKOKJFESMBGTPJVY5RNBPS2IZA, seg_pos={0 0}, upload=472662]: Uploading piece... Dec 28 02:26:47.441453105 (538h57m47.31253614s) [noise=true , address=45.91.116.5:4297 , piece_id=ZNLGXAUC3EXBGG43SHWQAET2DP2UL74VYZAREZBPNWAUVCDMWCIA, seg_pos={0 0}, upload=472662]: Uploading piece... Dec 28 02:26:47.441654498 (538h57m47.312737533s) [noise=true , address=145.79.227.14:4837 , piece_id=3T76IFQAB4HG7QX6IMXF3A5MNDGCH4ZT4ENMIP4WMHX56N3QA4PA, seg_pos={0 0}, upload=472662]: Uploading piece... Dec 28 02:26:47.442956047 (538h57m47.314039082s) [noise=true , address=77.90.31.10:4518 , piece_id=SPWCNIRCIDWPWYSU45II3P2NT3DRSJ3KSZAICJHI2RZO4SK3MOBA, seg_pos={0 0}, upload=472662]: Uploading piece... Dec 28 02:26:49.169098577 (538h57m49.040181612s) [upload=472662]: Next returned nil segment Dec 28 02:26:49.169128995 (538h57m49.04021203s) [upload=472662]: Waiting for error group managing segments... Dec 28 02:26:49.446462801 (538h57m49.317545836s) [noise=true , address=145.79.227.4:4186 , piece_id=3HUD4SNDNV4BGGJN2SEFXOO3GY7ZHJ2IK5LYRSM573ZP4EGL6WKQ, seg_pos={0 0}, upload=472662]: Done uploading piece. err: <nil> Dec 28 02:26:49.457373344 (538h57m49.328456379s) [noise=true , address=145.79.227.15:4874 , piece_id=XBHDKLDXT2YKNE2Q6DYL2CWFBGO6ESRHBAR7AMG3UTQTDTVFVT5Q, seg_pos={0 0}, upload=472662]: Done uploading piece. err: <nil> Dec 28 02:26:49.467686019 (538h57m49.338769064s) [noise=true , address=145.79.227.9:4365 , piece_id=CHS62KWTEREH7A3VK7MQBFVZ2JIOQANBYSLGH7NMO36PMRRVMOJA, seg_pos={0 0}, upload=472662]: Done uploading piece. err: <nil> Dec 28 02:26:49.491126058 (538h57m49.362209093s) [noise=true , address=145.79.227.41:4086 , piece_id=FV6GTIKBNIJQFWOWUS7E6LS3J5V675VPL2JVJHN5RZSVYHPJOBKQ, seg_pos={0 0}, upload=472662]: Done uploading piece. err: <nil> Dec 28 02:26:49.500579496 (538h57m49.371662541s) [noise=true , address=145.79.227.52:4445 , piece_id=6VV7RXEZGMXQEUHHNTYVMHHNN7NMAVIEQEGR5BV5Q3NGD3WZ4JIA, seg_pos={0 0}, upload=472662]: Done uploading piece. err: <nil> Dec 28 02:26:49.508940695 (538h57m49.38002374s) [noise=true , address=145.79.227.7:4295 , piece_id=OVLLLNVXYUTVWDITBCAECE7POFOT5BNBCPIAE27V4GNMB2NVQDXQ, seg_pos={0 0}, upload=472662]: Done uploading piece. err: <nil> Dec 28 02:26:49.683021635 (538h57m49.55410467s) [noise=true , address=45.91.116.5:4297 , piece_id=ZNLGXAUC3EXBGG43SHWQAET2DP2UL74VYZAREZBPNWAUVCDMWCIA, seg_pos={0 0}, upload=472662]: Done uploading piece. err: <nil> Dec 28 02:26:49.685365357 (538h57m49.556448402s) [noise=true , address=45.91.116.11:4118 , piece_id=KPD5TZAPY4VBNTL5P4YLHNLTFH4IOSDZF7YGDNWE2ESHZ4JPKNHA, seg_pos={0 0}, upload=472662]: Done uploading piece. err: <nil> Dec 28 02:26:50.013338819 (538h57m49.884421864s) [noise=true , address=77.90.31.10:4518 , piece_id=SPWCNIRCIDWPWYSU45II3P2NT3DRSJ3KSZAICJHI2RZO4SK3MOBA, seg_pos={0 0}, upload=472662]: Done uploading piece. err: <nil> Dec 28 02:26:50.416372892 (538h57m50.287455937s) [noise=true , address=45.91.116.14:4191 , piece_id=YGJIYP6S2T3YI7YLB2KSAHJTXYO6LEQA423MPQOJSCBTVDPISN2A, seg_pos={0 0}, upload=472662]: Done uploading piece. err: <nil> Dec 28 02:26:50.604365408 (538h57m50.475448453s) [noise=true , address=145.79.227.11:4729 , piece_id=RYLED7ZV45T6FDXV5MESNLZHGTHXHAWDVZSAR7CFB5ZJALZ6MX4Q, seg_pos={0 0}, upload=472662]: Done uploading piece. err: <nil> Dec 28 02:26:50.612968557 (538h57m50.484051592s) [noise=true , address=77.90.31.11:4553 , piece_id=OKW2XGVS45O276JX2PDXZOESXBWXBWB6P7COHOZF5LRUDUPCSRFA, seg_pos={0 0}, upload=472662]: Done uploading piece. err: <nil> Dec 28 02:26:50.630594306 (538h57m50.501677372s) [noise=true , address=145.79.227.14:4837 , piece_id=3T76IFQAB4HG7QX6IMXF3A5MNDGCH4ZT4ENMIP4WMHX56N3QA4PA, seg_pos={0 0}, upload=472662]: Done uploading piece. err: <nil> Dec 28 02:26:50.630622960 (538h57m50.501705995s) [seg_pos={0 0}, upload=472662]: Segment reached optimal threshold of 13 pieces. Dec 28 02:26:50.630812411 (538h57m50.501895447s) [noise=true , address=45.91.116.13:4154 , piece_id=Y3JKBJWZER5Q34WVMTQ7MIGIT6AIX7TD6DPXZXZ6ZUU3HGRZOW6Q, seg_pos={0 0}, upload=472662]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: write tcp 188.166.225.170:44068->45.91.116.13:4154: use of closed network connection; write tcp 188.166.225.170:44068->45.91.116.13:4154: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44068->45.91.116.13:4154: use of closed network connection Dec 28 02:26:50.630819785 (538h57m50.50190282s) [noise=true , address=45.91.116.13:4154 , piece_id=Y3JKBJWZER5Q34WVMTQ7MIGIT6AIX7TD6DPXZXZ6ZUU3HGRZOW6Q, seg_pos={0 0}, upload=472662]: Long tail cancellation due to optimal threshold Dec 28 02:26:50.631094999 (538h57m50.502178035s) [noise=true , address=145.79.227.2:4114 , piece_id=4PFLUU4AKGCM2OJCTR4OOOOHX4YJM5J4VKHCDU2SS4IAZEI4U22A, seg_pos={0 0}, upload=472662]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN): protocol: expected piece hash; context canceled; EOF Dec 28 02:26:50.631101612 (538h57m50.502184647s) [noise=true , address=145.79.227.2:4114 , piece_id=4PFLUU4AKGCM2OJCTR4OOOOHX4YJM5J4VKHCDU2SS4IAZEI4U22A, seg_pos={0 0}, upload=472662]: Long tail cancellation due to optimal threshold Dec 28 02:26:50.631207363 (538h57m50.502290398s) [noise=true , address=45.91.116.7:4011 , piece_id=DB2X67NJV5VEP2YQAQX7Q6HRDZGKOKJFESMBGTPJVY5RNBPS2IZA, seg_pos={0 0}, upload=472662]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 28 02:26:50.631219296 (538h57m50.502302331s) [noise=true , address=45.91.116.7:4011 , piece_id=DB2X67NJV5VEP2YQAQX7Q6HRDZGKOKJFESMBGTPJVY5RNBPS2IZA, seg_pos={0 0}, upload=472662]: Long tail cancellation due to optimal threshold Dec 28 02:26:50.631236278 (538h57m50.502319313s) [seg_pos={0 0}, upload=472662]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 02:26:50.631301062 (538h57m50.502384107s) [upload=472662]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 02:26:50.631305701 (538h57m50.502388736s) [upload=472662]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 02:26:50.923055765 (538h57m50.7941388s) [upload=472662]: Done uploading segments... Dec 28 02:26:50.923075482 (538h57m50.794158527s) [upload=472662]: Upload finished. err: <nil> Duration: 30.004654663s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..............................X.........................X...] duration: 3.453056652s trace: https://jaeger.dev.storj.io/trace/4aff4da4d82027c9 FAIL (2025-12-28 02:27:09 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.453100736s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 32 out of 60 is failed (threshold=3) [.X.XXX.X.XX.XX.X..X.X.X.X.X..X...XX..X.XXX.XXX.X.....XXX.XXX] duration: 30.003753127s trace: https://jaeger.dev.storj.io/trace/73b691fb7203b22a FAIL (2025-12-28 02:26:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: OFFXE56DY744AQ25MOFWSENMASTKDKT4IBUVZL3PK3MT7CHI2CUA) context deadline exceeded; piecestore: (Node ID: 1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, Piece ID: SYEX7HEURWETLW7XF4AOBADEMWGEQPVGLTN3TCMJRJRZ5JF7RKDA) context deadline exceeded ======LOGS====== Dec 28 02:26:02.278777784 (538h57m2.149860829s) [upload=472653]: Starting upload Dec 28 02:26:02.278867865 (538h57m2.14995091s) [upload=472653]: Done starting upload Dec 28 02:26:02.278944401 (538h57m2.150027446s) [upload=472653]: Uploading segments... Dec 28 02:26:02.279110306 (538h57m2.150193351s) [upload=472653]: Got next segment. Inline: false Dec 28 02:26:02.279144271 (538h57m2.150227306s) [upload=472653]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 02:26:02.279149381 (538h57m2.150232426s) [upload=472653]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 02:26:02.519896970 (538h57m2.390980015s) [seg_pos={0 0}, upload=472653]: Begin upload segment... Dec 28 02:26:02.519963246 (538h57m2.391046291s) [seg_pos={0 0}, upload=472653]: Done begin upload segment. Dec 28 02:26:02.520005627 (538h57m2.391088672s) [noise=true , address=145.79.227.15:4876 , piece_id=SRMRAUYZPBJ752TLRLV5VS4OFVZEY3JTP4EZKNPI7YJQFNUYDDUA, seg_pos={0 0}, upload=472653]: Uploading piece... Dec 28 02:26:02.520016578 (538h57m2.391099633s) [noise=true , address=77.90.31.14:4661 , piece_id=A2MMEKMK6HPLM2PXNVQKEHVJANO25C5ISAU6A22PYUQKD3CBQDSQ, seg_pos={0 0}, upload=472653]: Uploading piece... Dec 28 02:26:02.520526088 (538h57m2.391609133s) [noise=true , address=145.79.227.44:4193 , piece_id=EN3DVFP5B576L626SA5IKU42Z36B2NQIUC3GLNYRENVJ7V67A7DA, seg_pos={0 0}, upload=472653]: Uploading piece... Dec 28 02:26:02.520927573 (538h57m2.392010608s) [noise=true , address=145.79.227.4:4187 , piece_id=SOZPFCTTGAT4KDR67LXWWGQRGFAK4RKH2RMOVWQ47Y3QXRH26Q3A, seg_pos={0 0}, upload=472653]: Uploading piece... Dec 28 02:26:02.521284302 (538h57m2.392367337s) [noise=true , address=45.91.116.14:4189 , piece_id=XSUY6HZG7RWBOI6BZVBTOCTMI675IUBKA2HRZPL2XAMM4QG46TVQ, seg_pos={0 0}, upload=472653]: Uploading piece... Dec 28 02:26:02.521700214 (538h57m2.392783249s) [noise=true , address=145.79.227.50:4375 , piece_id=EWOCRQISWPHZTGBEKJEQL6TRFQLICBFKV7DBOQYKYYDAEAENH4GA, seg_pos={0 0}, upload=472653]: Uploading piece... Dec 28 02:26:02.522155030 (538h57m2.393238075s) [noise=true , address=77.90.31.18:4805 , piece_id=I2DG3BETDNJCDNVFR4DP2KXW3EXY3DJXYFQ4IFZPDEKDM5HV74FQ, seg_pos={0 0}, upload=472653]: Uploading piece... Dec 28 02:26:02.522737007 (538h57m2.393820042s) [noise=true , address=145.79.227.16:4909 , piece_id=BDNVTYFUIG7Y6JPYZBPKKD3CG33H7QCD3PLND7TLHS4NZA6RAHDQ, seg_pos={0 0}, upload=472653]: Uploading piece... Dec 28 02:26:02.523223293 (538h57m2.394306328s) [noise=true , address=145.79.227.51:4411 , piece_id=SK4732SRMX3ZCA6OOG2YDQFH2PMSHGPP4O7MRMNIFVUFRWXDIUXA, seg_pos={0 0}, upload=472653]: Uploading piece... Dec 28 02:26:02.524266379 (538h57m2.395349425s) [noise=true , address=145.79.227.19:4079 , piece_id=AD73DSK6EYN2WHIPPBXY4XDCY3MDKQZ4E7F6BAJXRD24KUEU62YQ, seg_pos={0 0}, upload=472653]: Uploading piece... Dec 28 02:26:02.525675552 (538h57m2.396758587s) [noise=true , address=145.79.227.47:4265 , piece_id=ERAMZWILIBHOH66HYHZLT4WB22SBSK5ND4EFUIFMVABAVCH6MJ2Q, seg_pos={0 0}, upload=472653]: Uploading piece... Dec 28 02:26:02.527831817 (538h57m2.398914852s) [noise=true , address=145.79.227.7:4293 , piece_id=LBUSFEPH7GPSYBDOSVCHBBJZ2YL3PHWR5WIOCIMZEXBEZLMJFNQA, seg_pos={0 0}, upload=472653]: Uploading piece... Dec 28 02:26:02.529651171 (538h57m2.400734216s) [noise=true , address=45.91.116.8:4045 , piece_id=OFFXE56DY744AQ25MOFWSENMASTKDKT4IBUVZL3PK3MT7CHI2CUA, seg_pos={0 0}, upload=472653]: Uploading piece... Dec 28 02:26:02.530906572 (538h57m2.401989617s) [noise=true , address=77.90.31.17:4773 , piece_id=SYEX7HEURWETLW7XF4AOBADEMWGEQPVGLTN3TCMJRJRZ5JF7RKDA, seg_pos={0 0}, upload=472653]: Uploading piece... Dec 28 02:26:02.532246493 (538h57m2.403329538s) [noise=true , address=45.91.116.7:4012 , piece_id=ED4PHZZSJYOFSKKXZHV5BNPO3JOYSUDOHTOVJ72NW6PKFKFBOZCQ, seg_pos={0 0}, upload=472653]: Uploading piece... Dec 28 02:26:02.533678640 (538h57m2.404761675s) [noise=true , address=145.79.227.38:4978 , piece_id=WJM7BZJL5CKFWJGZFX654WQZP4HNNYD76RYQZFZLWWZYDPNTFNOQ, seg_pos={0 0}, upload=472653]: Uploading piece... Dec 28 02:26:04.237323757 (538h57m4.108406792s) [upload=472653]: Next returned nil segment Dec 28 02:26:04.237343114 (538h57m4.108426159s) [upload=472653]: Waiting for error group managing segments... Dec 28 02:26:04.525347867 (538h57m4.396430912s) [noise=true , address=145.79.227.50:4375 , piece_id=EWOCRQISWPHZTGBEKJEQL6TRFQLICBFKV7DBOQYKYYDAEAENH4GA, seg_pos={0 0}, upload=472653]: Done uploading piece. err: <nil> Dec 28 02:26:04.525868338 (538h57m4.396951384s) [noise=true , address=145.79.227.7:4293 , piece_id=LBUSFEPH7GPSYBDOSVCHBBJZ2YL3PHWR5WIOCIMZEXBEZLMJFNQA, seg_pos={0 0}, upload=472653]: Done uploading piece. err: <nil> Dec 28 02:26:04.541017551 (538h57m4.412100596s) [noise=true , address=145.79.227.38:4978 , piece_id=WJM7BZJL5CKFWJGZFX654WQZP4HNNYD76RYQZFZLWWZYDPNTFNOQ, seg_pos={0 0}, upload=472653]: Done uploading piece. err: <nil> Dec 28 02:26:04.561608776 (538h57m4.432691821s) [noise=true , address=145.79.227.19:4079 , piece_id=AD73DSK6EYN2WHIPPBXY4XDCY3MDKQZ4E7F6BAJXRD24KUEU62YQ, seg_pos={0 0}, upload=472653]: Done uploading piece. err: <nil> Dec 28 02:26:04.564288849 (538h57m4.435371894s) [noise=true , address=145.79.227.4:4187 , piece_id=SOZPFCTTGAT4KDR67LXWWGQRGFAK4RKH2RMOVWQ47Y3QXRH26Q3A, seg_pos={0 0}, upload=472653]: Done uploading piece. err: <nil> Dec 28 02:26:04.578057393 (538h57m4.449140428s) [noise=true , address=145.79.227.47:4265 , piece_id=ERAMZWILIBHOH66HYHZLT4WB22SBSK5ND4EFUIFMVABAVCH6MJ2Q, seg_pos={0 0}, upload=472653]: Done uploading piece. err: <nil> Dec 28 02:26:04.713494577 (538h57m4.584577622s) [noise=true , address=145.79.227.15:4876 , piece_id=SRMRAUYZPBJ752TLRLV5VS4OFVZEY3JTP4EZKNPI7YJQFNUYDDUA, seg_pos={0 0}, upload=472653]: Done uploading piece. err: <nil> Dec 28 02:26:04.721789771 (538h57m4.592872806s) [noise=true , address=77.90.31.17:4773 , piece_id=SYEX7HEURWETLW7XF4AOBADEMWGEQPVGLTN3TCMJRJRZ5JF7RKDA, seg_pos={0 0}, upload=472653]: Done uploading piece. err: <nil> Dec 28 02:26:04.757417064 (538h57m4.628500099s) [noise=true , address=145.79.227.51:4411 , piece_id=SK4732SRMX3ZCA6OOG2YDQFH2PMSHGPP4O7MRMNIFVUFRWXDIUXA, seg_pos={0 0}, upload=472653]: Done uploading piece. err: <nil> Dec 28 02:26:04.828450075 (538h57m4.69953312s) [noise=true , address=45.91.116.14:4189 , piece_id=XSUY6HZG7RWBOI6BZVBTOCTMI675IUBKA2HRZPL2XAMM4QG46TVQ, seg_pos={0 0}, upload=472653]: Done uploading piece. err: <nil> Dec 28 02:26:04.838985074 (538h57m4.710068119s) [noise=true , address=77.90.31.14:4661 , piece_id=A2MMEKMK6HPLM2PXNVQKEHVJANO25C5ISAU6A22PYUQKD3CBQDSQ, seg_pos={0 0}, upload=472653]: Done uploading piece. err: <nil> Dec 28 02:26:04.863626660 (538h57m4.734709695s) [noise=true , address=45.91.116.7:4012 , piece_id=ED4PHZZSJYOFSKKXZHV5BNPO3JOYSUDOHTOVJ72NW6PKFKFBOZCQ, seg_pos={0 0}, upload=472653]: Done uploading piece. err: <nil> Dec 28 02:26:04.941565371 (538h57m4.812648416s) [noise=true , address=45.91.116.8:4045 , piece_id=OFFXE56DY744AQ25MOFWSENMASTKDKT4IBUVZL3PK3MT7CHI2CUA, seg_pos={0 0}, upload=472653]: Done uploading piece. err: <nil> Dec 28 02:26:04.941589416 (538h57m4.812672451s) [seg_pos={0 0}, upload=472653]: Segment reached optimal threshold of 13 pieces. Dec 28 02:26:04.941880301 (538h57m4.812963336s) [noise=true , address=145.79.227.16:4909 , piece_id=BDNVTYFUIG7Y6JPYZBPKKD3CG33H7QCD3PLND7TLHS4NZA6RAHDQ, seg_pos={0 0}, upload=472653]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x): protocol: expected piece hash; context canceled; EOF Dec 28 02:26:04.941888686 (538h57m4.812971721s) [noise=true , address=145.79.227.16:4909 , piece_id=BDNVTYFUIG7Y6JPYZBPKKD3CG33H7QCD3PLND7TLHS4NZA6RAHDQ, seg_pos={0 0}, upload=472653]: Long tail cancellation due to optimal threshold Dec 28 02:26:04.942010608 (538h57m4.813093653s) [noise=true , address=145.79.227.44:4193 , piece_id=EN3DVFP5B576L626SA5IKU42Z36B2NQIUC3GLNYRENVJ7V67A7DA, seg_pos={0 0}, upload=472653]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 28 02:26:04.942016059 (538h57m4.813099104s) [noise=true , address=145.79.227.44:4193 , piece_id=EN3DVFP5B576L626SA5IKU42Z36B2NQIUC3GLNYRENVJ7V67A7DA, seg_pos={0 0}, upload=472653]: Long tail cancellation due to optimal threshold Dec 28 02:26:04.942028913 (538h57m4.813111958s) [noise=true , address=77.90.31.18:4805 , piece_id=I2DG3BETDNJCDNVFR4DP2KXW3EXY3DJXYFQ4IFZPDEKDM5HV74FQ, seg_pos={0 0}, upload=472653]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF Dec 28 02:26:04.942040004 (538h57m4.813123049s) [noise=true , address=77.90.31.18:4805 , piece_id=I2DG3BETDNJCDNVFR4DP2KXW3EXY3DJXYFQ4IFZPDEKDM5HV74FQ, seg_pos={0 0}, upload=472653]: Long tail cancellation due to optimal threshold Dec 28 02:26:04.942075742 (538h57m4.813158787s) [seg_pos={0 0}, upload=472653]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 02:26:04.942196883 (538h57m4.813279928s) [upload=472653]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 02:26:04.942211210 (538h57m4.813294255s) [upload=472653]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 02:26:05.227048714 (538h57m5.098131749s) [upload=472653]: Done uploading segments... Dec 28 02:26:05.227069744 (538h57m5.098152779s) [upload=472653]: Upload finished. err: <nil> Duration: 30.003777793s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [................................X...................X.....X.] duration: 46.236641698s trace: https://jaeger.dev.storj.io/trace/1822be52d200e5b5 FAIL (2025-12-28 02:26:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1ZAfYZ4wgpZVvatj6wtHona3hTaPcLxAmXC1DT2uKQyo8nLQQ7, Piece ID: 5MMQ44MXRT4XEL4X6WO54WRG2XYMX3AN3FNQ2H2XIUTKBXI3EEPA) context deadline exceeded; piecestore: (Node ID: 124guAM82oibYZsMho67Em6Ke6yEZXwT6LPMLLSZZBLqwoFGUkn, Piece ID: 22WJIPX4SM5NTIAW5ZRM2UKX32CTDBFCXPCCHRRWRCUCOF5CRE5Q) context deadline exceeded; piecestore: (Node ID: 1e6p1TD5CqUyWdZjkrxh2kRqMuyDiMUr9eimfeayDAKwhtBrZ8, Piece ID: XRI5EJFN7YYUG4QKLDGEFT6YAUHCUTTSXF735UDR6LZZBGMAZDJQ) context deadline exceeded ======LOGS====== 5.234848204 (538h55m35.105931239s) [noise=true , address=77.90.31.11:4572 , piece_id=5MMQ44MXRT4XEL4X6WO54WRG2XYMX3AN3FNQ2H2XIUTKBXI3EEPA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.235412057 (538h55m35.106495102s) [noise=true , address=145.79.227.22:4512 , piece_id=D46YLTIKGHG36DQ4YBKBSUDV4GGB4WVDWNOJS27IYYLN6AYZ5QFQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.235820946 (538h55m35.106903981s) [noise=true , address=77.90.31.19:4874 , piece_id=RHZUXUYMJU6RKQGSDZM5WDWIN5SQ7YIRSRATNNKT5GEMU2QH444A, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.236192192 (538h55m35.107275237s) [noise=true , address=77.90.31.16:4760 , piece_id=XRI5EJFN7YYUG4QKLDGEFT6YAUHCUTTSXF735UDR6LZZBGMAZDJQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.236564772 (538h55m35.107647817s) [noise=true , address=145.79.227.40:4062 , piece_id=PW6QWIPQWVUETLC6DJ624G6IB4P6EM4SYB72NAMQ4V6NUKNLUYLQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.236929766 (538h55m35.108012812s) [noise=true , address=64.49.13.139:4480 , piece_id=JAJY3F7IDFFMFKCK2QJV6W74RIESLRKBLXMN27Q3C45JWEPLBZLQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.237916846 (538h55m35.108999881s) [noise=true , address=145.79.227.34:4932 , piece_id=UPXSCQ45QDGPKNJXRFJ33EAA3QENVHRVQARU5TFJTEWICZNR253Q, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.238852948 (538h55m35.109935983s) [noise=true , address=45.91.116.12:4257 , piece_id=ZI67LIAHZW4G357GWKDPTB5SMXYGNXZTTTDMNAETDS2B6SWVDAEQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239152910 (538h55m35.110235945s) [noise=true , address=145.79.227.44:4222 , piece_id=CZ2CS62CTHQFAIE3S3JO2N3VJ4XIR5YJ2LMRVLLX7I4VM2VZVQVA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239409969 (538h55m35.110493004s) [noise=true , address=145.79.227.30:4773 , piece_id=KUGJXZCNMUE7WSK3VLY4AUBSOT4H6IY4XB4NCBGAGCNUG7F7IODQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239681136 (538h55m35.110764171s) [noise=true , address=77.90.31.13:4635 , piece_id=IIOY5X2W3XW4C2OFEBRCABMYFBI24RRTYNUSKN6KJU7CTUW4KONQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239954676 (538h55m35.111037711s) [noise=true , address=45.91.116.9:4281 , piece_id=JZLDW777CBX2BP7MLOQV4NFP3H4XJVCNQXTHNSUWPVDOJLPZK36A, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.240189914 (538h55m35.111272949s) [noise=true , address=145.79.227.35:4512 , piece_id=3E3VCCC45XVT5FHFDKXSTGGAGYBEML3BKU6JJTT33LCD2YMOMBZQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.240449147 (538h55m35.111532182s) [noise=true , address=45.91.116.8:4064 , piece_id=22WJIPX4SM5NTIAW5ZRM2UKX32CTDBFCXPCCHRRWRCUCOF5CRE5Q, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.240639640 (538h55m35.111722675s) [noise=true , address=145.79.227.32:4853 , piece_id=MBNCFDBQ4SZZXV4F7J5TRMOUYUQ5IPKQCGYGGM5SOIFJ6ZYCAUYA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.241137798 (538h55m35.112220833s) [noise=true , address=77.90.31.16:4751 , piece_id=N3Z2LLMCIA2F42VXZA7IDRU4XEKGOICH6TRVERODU76MFFE4QSHA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.241383797 (538h55m35.112466842s) [noise=true , address=77.90.31.18:4836 , piece_id=BSVEFI52EKN44CMINFUSZBNOH53IW5TXLVTF5IH6ZUTNII6RRLVA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:36.592880816 (538h55m36.463963871s) [upload=472631]: Next returned nil segment Dec 28 02:24:36.592901625 (538h55m36.46398467s) [upload=472631]: Waiting for error group managing segments... Dec 28 02:24:36.946933741 (538h55m36.818016786s) [noise=true , address=64.49.13.139:4480 , piece_id=JAJY3F7IDFFMFKCK2QJV6W74RIESLRKBLXMN27Q3C45JWEPLBZLQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.128189692 (538h55m36.999272737s) [noise=true , address=145.79.227.35:4512 , piece_id=3E3VCCC45XVT5FHFDKXSTGGAGYBEML3BKU6JJTT33LCD2YMOMBZQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.160760084 (538h55m37.031843119s) [noise=true , address=145.79.227.34:4932 , piece_id=UPXSCQ45QDGPKNJXRFJ33EAA3QENVHRVQARU5TFJTEWICZNR253Q, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.288838207 (538h55m37.159921252s) [noise=true , address=77.90.31.11:4572 , piece_id=5MMQ44MXRT4XEL4X6WO54WRG2XYMX3AN3FNQ2H2XIUTKBXI3EEPA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.319996310 (538h55m37.191079345s) [noise=true , address=145.79.227.22:4512 , piece_id=D46YLTIKGHG36DQ4YBKBSUDV4GGB4WVDWNOJS27IYYLN6AYZ5QFQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.320513273 (538h55m37.191596319s) [noise=true , address=77.90.31.18:4836 , piece_id=BSVEFI52EKN44CMINFUSZBNOH53IW5TXLVTF5IH6ZUTNII6RRLVA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.336522926 (538h55m37.207605971s) [noise=true , address=145.79.227.32:4853 , piece_id=MBNCFDBQ4SZZXV4F7J5TRMOUYUQ5IPKQCGYGGM5SOIFJ6ZYCAUYA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.384868031 (538h55m37.255951066s) [noise=true , address=77.90.31.16:4751 , piece_id=N3Z2LLMCIA2F42VXZA7IDRU4XEKGOICH6TRVERODU76MFFE4QSHA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.453563554 (538h55m37.324646599s) [noise=true , address=45.91.116.8:4064 , piece_id=22WJIPX4SM5NTIAW5ZRM2UKX32CTDBFCXPCCHRRWRCUCOF5CRE5Q, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.479454839 (538h55m37.350537884s) [noise=true , address=45.91.116.12:4257 , piece_id=ZI67LIAHZW4G357GWKDPTB5SMXYGNXZTTTDMNAETDS2B6SWVDAEQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.659710174 (538h55m37.530793219s) [noise=true , address=77.90.31.13:4635 , piece_id=IIOY5X2W3XW4C2OFEBRCABMYFBI24RRTYNUSKN6KJU7CTUW4KONQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.715039128 (538h55m37.586122163s) [noise=true , address=77.90.31.16:4760 , piece_id=XRI5EJFN7YYUG4QKLDGEFT6YAUHCUTTSXF735UDR6LZZBGMAZDJQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.977473406 (538h55m37.848556441s) [noise=true , address=77.90.31.19:4874 , piece_id=RHZUXUYMJU6RKQGSDZM5WDWIN5SQ7YIRSRATNNKT5GEMU2QH444A, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:38.016302745 (538h55m37.88738578s) [noise=true , address=145.79.227.30:4773 , piece_id=KUGJXZCNMUE7WSK3VLY4AUBSOT4H6IY4XB4NCBGAGCNUG7F7IODQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:38.016326982 (538h55m37.887410027s) [seg_pos={0 0}, upload=472631]: Segment reached optimal threshold of 14 pieces. Dec 28 02:24:38.016585353 (538h55m37.887668398s) [noise=true , address=145.79.227.30:4766 , piece_id=67M2DMPAVWIQKQSWUAG73D4E4NRRSG4XEVPOUVIJPEQOUV7A6VVA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1k6zdamuKPkBdzyNZNWx4FCmaTkv33PchcwvTRR8bgcETBs4K7): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016594450 (538h55m37.887677495s) [noise=true , address=145.79.227.30:4766 , piece_id=67M2DMPAVWIQKQSWUAG73D4E4NRRSG4XEVPOUVIJPEQOUV7A6VVA, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016627093 (538h55m37.887710148s) [noise=true , address=45.91.116.9:4281 , piece_id=JZLDW777CBX2BP7MLOQV4NFP3H4XJVCNQXTHNSUWPVDOJLPZK36A, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gxf7DpJZthyR6f1PExX56tJavjhPw9Gxb4h8mtmeYKW3t3tS1): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016635258 (538h55m37.887718293s) [noise=true , address=45.91.116.9:4281 , piece_id=JZLDW777CBX2BP7MLOQV4NFP3H4XJVCNQXTHNSUWPVDOJLPZK36A, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016871298 (538h55m37.887954363s) [noise=true , address=145.79.227.44:4222 , piece_id=CZ2CS62CTHQFAIE3S3JO2N3VJ4XIR5YJ2LMRVLLX7I4VM2VZVQVA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZKW35KvrsoYRefbYeUFZo8bh4XZYycNKGEvTFjzibezjXgj3e): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016880124 (538h55m37.887963169s) [noise=true , address=145.79.227.44:4222 , piece_id=CZ2CS62CTHQFAIE3S3JO2N3VJ4XIR5YJ2LMRVLLX7I4VM2VZVQVA, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016934839 (538h55m37.888017884s) [noise=true , address=145.79.227.40:4062 , piece_id=PW6QWIPQWVUETLC6DJ624G6IB4P6EM4SYB72NAMQ4V6NUKNLUYLQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016943425 (538h55m37.88802649s) [noise=true , address=145.79.227.40:4062 , piece_id=PW6QWIPQWVUETLC6DJ624G6IB4P6EM4SYB72NAMQ4V6NUKNLUYLQ, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016968252 (538h55m37.888051297s) [seg_pos={0 0}, upload=472631]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 02:24:38.017067822 (538h55m37.888150867s) [upload=472631]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 02:24:38.017074244 (538h55m37.888157289s) [upload=472631]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 02:24:38.304306667 (538h55m38.175389712s) [upload=472631]: Done uploading segments... Dec 28 02:24:38.304329641 (538h55m38.175412686s) [upload=472631]: Upload finished. err: <nil> Duration: 46.236661727s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...............................X.........................X..] duration: 3.485392078s trace: https://jaeger.dev.storj.io/trace/4aff4da4d82027c9 FAIL (2025-12-28 02:26:11 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.48547152s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................................X...................X.....X] duration: 1m0.001846713s trace: https://jaeger.dev.storj.io/trace/1822be52d200e5b5 FAIL (2025-12-28 02:25:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1ZAfYZ4wgpZVvatj6wtHona3hTaPcLxAmXC1DT2uKQyo8nLQQ7, Piece ID: 5MMQ44MXRT4XEL4X6WO54WRG2XYMX3AN3FNQ2H2XIUTKBXI3EEPA) context deadline exceeded; piecestore: (Node ID: 124guAM82oibYZsMho67Em6Ke6yEZXwT6LPMLLSZZBLqwoFGUkn, Piece ID: 22WJIPX4SM5NTIAW5ZRM2UKX32CTDBFCXPCCHRRWRCUCOF5CRE5Q) context deadline exceeded; piecestore: (Node ID: 1e6p1TD5CqUyWdZjkrxh2kRqMuyDiMUr9eimfeayDAKwhtBrZ8, Piece ID: XRI5EJFN7YYUG4QKLDGEFT6YAUHCUTTSXF735UDR6LZZBGMAZDJQ) context deadline exceeded ======LOGS====== 5.234848204 (538h55m35.105931239s) [noise=true , address=77.90.31.11:4572 , piece_id=5MMQ44MXRT4XEL4X6WO54WRG2XYMX3AN3FNQ2H2XIUTKBXI3EEPA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.235412057 (538h55m35.106495102s) [noise=true , address=145.79.227.22:4512 , piece_id=D46YLTIKGHG36DQ4YBKBSUDV4GGB4WVDWNOJS27IYYLN6AYZ5QFQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.235820946 (538h55m35.106903981s) [noise=true , address=77.90.31.19:4874 , piece_id=RHZUXUYMJU6RKQGSDZM5WDWIN5SQ7YIRSRATNNKT5GEMU2QH444A, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.236192192 (538h55m35.107275237s) [noise=true , address=77.90.31.16:4760 , piece_id=XRI5EJFN7YYUG4QKLDGEFT6YAUHCUTTSXF735UDR6LZZBGMAZDJQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.236564772 (538h55m35.107647817s) [noise=true , address=145.79.227.40:4062 , piece_id=PW6QWIPQWVUETLC6DJ624G6IB4P6EM4SYB72NAMQ4V6NUKNLUYLQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.236929766 (538h55m35.108012812s) [noise=true , address=64.49.13.139:4480 , piece_id=JAJY3F7IDFFMFKCK2QJV6W74RIESLRKBLXMN27Q3C45JWEPLBZLQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.237916846 (538h55m35.108999881s) [noise=true , address=145.79.227.34:4932 , piece_id=UPXSCQ45QDGPKNJXRFJ33EAA3QENVHRVQARU5TFJTEWICZNR253Q, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.238852948 (538h55m35.109935983s) [noise=true , address=45.91.116.12:4257 , piece_id=ZI67LIAHZW4G357GWKDPTB5SMXYGNXZTTTDMNAETDS2B6SWVDAEQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239152910 (538h55m35.110235945s) [noise=true , address=145.79.227.44:4222 , piece_id=CZ2CS62CTHQFAIE3S3JO2N3VJ4XIR5YJ2LMRVLLX7I4VM2VZVQVA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239409969 (538h55m35.110493004s) [noise=true , address=145.79.227.30:4773 , piece_id=KUGJXZCNMUE7WSK3VLY4AUBSOT4H6IY4XB4NCBGAGCNUG7F7IODQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239681136 (538h55m35.110764171s) [noise=true , address=77.90.31.13:4635 , piece_id=IIOY5X2W3XW4C2OFEBRCABMYFBI24RRTYNUSKN6KJU7CTUW4KONQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239954676 (538h55m35.111037711s) [noise=true , address=45.91.116.9:4281 , piece_id=JZLDW777CBX2BP7MLOQV4NFP3H4XJVCNQXTHNSUWPVDOJLPZK36A, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.240189914 (538h55m35.111272949s) [noise=true , address=145.79.227.35:4512 , piece_id=3E3VCCC45XVT5FHFDKXSTGGAGYBEML3BKU6JJTT33LCD2YMOMBZQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.240449147 (538h55m35.111532182s) [noise=true , address=45.91.116.8:4064 , piece_id=22WJIPX4SM5NTIAW5ZRM2UKX32CTDBFCXPCCHRRWRCUCOF5CRE5Q, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.240639640 (538h55m35.111722675s) [noise=true , address=145.79.227.32:4853 , piece_id=MBNCFDBQ4SZZXV4F7J5TRMOUYUQ5IPKQCGYGGM5SOIFJ6ZYCAUYA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.241137798 (538h55m35.112220833s) [noise=true , address=77.90.31.16:4751 , piece_id=N3Z2LLMCIA2F42VXZA7IDRU4XEKGOICH6TRVERODU76MFFE4QSHA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.241383797 (538h55m35.112466842s) [noise=true , address=77.90.31.18:4836 , piece_id=BSVEFI52EKN44CMINFUSZBNOH53IW5TXLVTF5IH6ZUTNII6RRLVA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:36.592880816 (538h55m36.463963871s) [upload=472631]: Next returned nil segment Dec 28 02:24:36.592901625 (538h55m36.46398467s) [upload=472631]: Waiting for error group managing segments... Dec 28 02:24:36.946933741 (538h55m36.818016786s) [noise=true , address=64.49.13.139:4480 , piece_id=JAJY3F7IDFFMFKCK2QJV6W74RIESLRKBLXMN27Q3C45JWEPLBZLQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.128189692 (538h55m36.999272737s) [noise=true , address=145.79.227.35:4512 , piece_id=3E3VCCC45XVT5FHFDKXSTGGAGYBEML3BKU6JJTT33LCD2YMOMBZQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.160760084 (538h55m37.031843119s) [noise=true , address=145.79.227.34:4932 , piece_id=UPXSCQ45QDGPKNJXRFJ33EAA3QENVHRVQARU5TFJTEWICZNR253Q, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.288838207 (538h55m37.159921252s) [noise=true , address=77.90.31.11:4572 , piece_id=5MMQ44MXRT4XEL4X6WO54WRG2XYMX3AN3FNQ2H2XIUTKBXI3EEPA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.319996310 (538h55m37.191079345s) [noise=true , address=145.79.227.22:4512 , piece_id=D46YLTIKGHG36DQ4YBKBSUDV4GGB4WVDWNOJS27IYYLN6AYZ5QFQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.320513273 (538h55m37.191596319s) [noise=true , address=77.90.31.18:4836 , piece_id=BSVEFI52EKN44CMINFUSZBNOH53IW5TXLVTF5IH6ZUTNII6RRLVA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.336522926 (538h55m37.207605971s) [noise=true , address=145.79.227.32:4853 , piece_id=MBNCFDBQ4SZZXV4F7J5TRMOUYUQ5IPKQCGYGGM5SOIFJ6ZYCAUYA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.384868031 (538h55m37.255951066s) [noise=true , address=77.90.31.16:4751 , piece_id=N3Z2LLMCIA2F42VXZA7IDRU4XEKGOICH6TRVERODU76MFFE4QSHA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.453563554 (538h55m37.324646599s) [noise=true , address=45.91.116.8:4064 , piece_id=22WJIPX4SM5NTIAW5ZRM2UKX32CTDBFCXPCCHRRWRCUCOF5CRE5Q, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.479454839 (538h55m37.350537884s) [noise=true , address=45.91.116.12:4257 , piece_id=ZI67LIAHZW4G357GWKDPTB5SMXYGNXZTTTDMNAETDS2B6SWVDAEQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.659710174 (538h55m37.530793219s) [noise=true , address=77.90.31.13:4635 , piece_id=IIOY5X2W3XW4C2OFEBRCABMYFBI24RRTYNUSKN6KJU7CTUW4KONQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.715039128 (538h55m37.586122163s) [noise=true , address=77.90.31.16:4760 , piece_id=XRI5EJFN7YYUG4QKLDGEFT6YAUHCUTTSXF735UDR6LZZBGMAZDJQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.977473406 (538h55m37.848556441s) [noise=true , address=77.90.31.19:4874 , piece_id=RHZUXUYMJU6RKQGSDZM5WDWIN5SQ7YIRSRATNNKT5GEMU2QH444A, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:38.016302745 (538h55m37.88738578s) [noise=true , address=145.79.227.30:4773 , piece_id=KUGJXZCNMUE7WSK3VLY4AUBSOT4H6IY4XB4NCBGAGCNUG7F7IODQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:38.016326982 (538h55m37.887410027s) [seg_pos={0 0}, upload=472631]: Segment reached optimal threshold of 14 pieces. Dec 28 02:24:38.016585353 (538h55m37.887668398s) [noise=true , address=145.79.227.30:4766 , piece_id=67M2DMPAVWIQKQSWUAG73D4E4NRRSG4XEVPOUVIJPEQOUV7A6VVA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1k6zdamuKPkBdzyNZNWx4FCmaTkv33PchcwvTRR8bgcETBs4K7): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016594450 (538h55m37.887677495s) [noise=true , address=145.79.227.30:4766 , piece_id=67M2DMPAVWIQKQSWUAG73D4E4NRRSG4XEVPOUVIJPEQOUV7A6VVA, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016627093 (538h55m37.887710148s) [noise=true , address=45.91.116.9:4281 , piece_id=JZLDW777CBX2BP7MLOQV4NFP3H4XJVCNQXTHNSUWPVDOJLPZK36A, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gxf7DpJZthyR6f1PExX56tJavjhPw9Gxb4h8mtmeYKW3t3tS1): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016635258 (538h55m37.887718293s) [noise=true , address=45.91.116.9:4281 , piece_id=JZLDW777CBX2BP7MLOQV4NFP3H4XJVCNQXTHNSUWPVDOJLPZK36A, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016871298 (538h55m37.887954363s) [noise=true , address=145.79.227.44:4222 , piece_id=CZ2CS62CTHQFAIE3S3JO2N3VJ4XIR5YJ2LMRVLLX7I4VM2VZVQVA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZKW35KvrsoYRefbYeUFZo8bh4XZYycNKGEvTFjzibezjXgj3e): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016880124 (538h55m37.887963169s) [noise=true , address=145.79.227.44:4222 , piece_id=CZ2CS62CTHQFAIE3S3JO2N3VJ4XIR5YJ2LMRVLLX7I4VM2VZVQVA, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016934839 (538h55m37.888017884s) [noise=true , address=145.79.227.40:4062 , piece_id=PW6QWIPQWVUETLC6DJ624G6IB4P6EM4SYB72NAMQ4V6NUKNLUYLQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016943425 (538h55m37.88802649s) [noise=true , address=145.79.227.40:4062 , piece_id=PW6QWIPQWVUETLC6DJ624G6IB4P6EM4SYB72NAMQ4V6NUKNLUYLQ, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016968252 (538h55m37.888051297s) [seg_pos={0 0}, upload=472631]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 02:24:38.017067822 (538h55m37.888150867s) [upload=472631]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 02:24:38.017074244 (538h55m37.888157289s) [upload=472631]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 02:24:38.304306667 (538h55m38.175389712s) [upload=472631]: Done uploading segments... Dec 28 02:24:38.304329641 (538h55m38.175412686s) [upload=472631]: Upload finished. err: <nil> Duration: 1m0.001871921s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 31 out of 60 is failed (threshold=3) [..X.XXX.X.XX.XX.X..X.X.X.X.X..X...XX..X.XXX.XXX.X.....XXX.XX] duration: 30.002655384s trace: https://jaeger.dev.storj.io/trace/b33308329497d3c FAIL (2025-12-28 02:25:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: NFQSJSHRNYXKDY5IWKRLBZ6PGNNAEZYIXBG45HHPIMDBYXHQUV3Q) context deadline exceeded; piecestore: (Node ID: 12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, Piece ID: YYJZWPEEYCTURKA2G77QLY7HIHUZG7AU4P2KXEHBLFVHNAQKOJ5Q) context deadline exceeded ======LOGS====== Dec 28 02:25:01.928184726 (538h56m1.799267771s) [upload=472638]: Starting upload Dec 28 02:25:01.928256883 (538h56m1.799339928s) [upload=472638]: Done starting upload Dec 28 02:25:01.928514404 (538h56m1.799597439s) [upload=472638]: Uploading segments... Dec 28 02:25:01.928558247 (538h56m1.799641293s) [upload=472638]: Got next segment. Inline: false Dec 28 02:25:01.928595398 (538h56m1.799678443s) [upload=472638]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 02:25:01.928600678 (538h56m1.799683723s) [upload=472638]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 02:25:02.174969891 (538h56m2.046052936s) [seg_pos={0 0}, upload=472638]: Begin upload segment... Dec 28 02:25:02.175038412 (538h56m2.046121497s) [seg_pos={0 0}, upload=472638]: Done begin upload segment. Dec 28 02:25:02.175092715 (538h56m2.04617577s) [noise=true , address=145.79.227.15:4874 , piece_id=4MLUNHLW44OJDNXFYPDAPVCNA6IA7VSLBHS7EH6ET6KUJGWFIRVQ, seg_pos={0 0}, upload=472638]: Uploading piece... Dec 28 02:25:02.175145215 (538h56m2.04622826s) [noise=true , address=45.91.116.14:4192 , piece_id=KJ5TRZIXDCJITTG6ISFBKG5VDSVFR2KIPG74MZTC36G56CC57TDA, seg_pos={0 0}, upload=472638]: Uploading piece... Dec 28 02:25:02.175226880 (538h56m2.046309925s) [noise=true , address=45.91.116.4:4939 , piece_id=5D5BATVNTNMNWWR2BPICRBU5PAI2MSITD2UH3YAZ23XXILW3H24Q, seg_pos={0 0}, upload=472638]: Uploading piece... Dec 28 02:25:02.175708888 (538h56m2.046791933s) [noise=true , address=77.90.31.14:4661 , piece_id=YYJZWPEEYCTURKA2G77QLY7HIHUZG7AU4P2KXEHBLFVHNAQKOJ5Q, seg_pos={0 0}, upload=472638]: Uploading piece... Dec 28 02:25:02.176098811 (538h56m2.047181856s) [noise=true , address=77.90.31.10:4519 , piece_id=57THD7AISKOYJWZSCWR6LKU2D2GV5CLTFUY2PDVIVHKARFMDBVNA, seg_pos={0 0}, upload=472638]: Uploading piece... Dec 28 02:25:02.176454347 (538h56m2.047537392s) [noise=true , address=145.79.227.36:4518 , piece_id=HILGHKIS3FEPKJQD4ZSLPYDIPNRHGUMJB5KVPZ7EBZCHURPTQ5NQ, seg_pos={0 0}, upload=472638]: Uploading piece... Dec 28 02:25:02.176837407 (538h56m2.047920462s) [noise=true , address=145.79.227.2:4115 , piece_id=IF5KHN5ULB2M5MCKRF7FSFQ7DRTWIBWDVV45KCCIVSYNKIO7BBSQ, seg_pos={0 0}, upload=472638]: Uploading piece... Dec 28 02:25:02.177375811 (538h56m2.048458867s) [noise=true , address=145.79.227.11:4732 , piece_id=RYYB3DLNDZ3X3UDY7E2XC3PFON26C6JALJMAC4RFNFRUP53CITEQ, seg_pos={0 0}, upload=472638]: Uploading piece... Dec 28 02:25:02.177993538 (538h56m2.049076583s) [noise=true , address=45.91.116.5:4297 , piece_id=7PEGDQUHFYYZLQKMYYF3AGUIJXG7ZKBGZ5TGEVIOWG7RPDSSGWJA, seg_pos={0 0}, upload=472638]: Uploading piece... Dec 28 02:25:02.178557692 (538h56m2.049640737s) [noise=true , address=45.91.116.7:4010 , piece_id=NFQSJSHRNYXKDY5IWKRLBZ6PGNNAEZYIXBG45HHPIMDBYXHQUV3Q, seg_pos={0 0}, upload=472638]: Uploading piece... Dec 28 02:25:02.179134079 (538h56m2.050217134s) [noise=true , address=145.79.227.51:4409 , piece_id=3GJUEQA526TZ7HBZSYLADSRYA4QT5QOXZSVALMNQT3PJCAR5EMKQ, seg_pos={0 0}, upload=472638]: Uploading piece... Dec 28 02:25:02.180513907 (538h56m2.051596952s) [noise=true , address=145.79.227.7:4293 , piece_id=5E7KXS6G3KPFX5F3NLZJ6KQN2AQ6N3GFDFIU2HOXK6UJOMBNEUTQ, seg_pos={0 0}, upload=472638]: Uploading piece... Dec 28 02:25:02.182052465 (538h56m2.05313551s) [noise=true , address=145.79.227.50:4375 , piece_id=TJJEXYTFY77ISKFQFV3UQF7GBC7QUFLUC7UAHO6DJTPBJYX5Y4QQ, seg_pos={0 0}, upload=472638]: Uploading piece... Dec 28 02:25:02.184302690 (538h56m2.055385736s) [noise=true , address=145.79.227.20:4438 , piece_id=N4RAOFVG5R4EPPMRYU4QB25TFWPHUZYQKJQWX7D4CORDJYPJOILQ, seg_pos={0 0}, upload=472638]: Uploading piece... Dec 28 02:25:02.185249032 (538h56m2.056332077s) [noise=true , address=45.91.116.11:4118 , piece_id=DFV2ZKQZSS3MS7NXSFZUXF5RPGAQCSVCV3PVBKENI4SMHRU4E3KQ, seg_pos={0 0}, upload=472638]: Uploading piece... Dec 28 02:25:02.189587582 (538h56m2.060670637s) [noise=true , address=145.79.227.45:4230 , piece_id=XOC6E5VSNO2P7NZEVNRX2JI6ZPMTEDZA3ETHNCEJ53OSABIWYCSQ, seg_pos={0 0}, upload=472638]: Uploading piece... Dec 28 02:25:03.833845039 (538h56m3.704928094s) [upload=472638]: Next returned nil segment Dec 28 02:25:03.833860978 (538h56m3.704944024s) [upload=472638]: Waiting for error group managing segments... Dec 28 02:25:04.202128649 (538h56m4.073211694s) [noise=true , address=145.79.227.2:4115 , piece_id=IF5KHN5ULB2M5MCKRF7FSFQ7DRTWIBWDVV45KCCIVSYNKIO7BBSQ, seg_pos={0 0}, upload=472638]: Done uploading piece. err: <nil> Dec 28 02:25:04.211761128 (538h56m4.082844173s) [noise=true , address=145.79.227.51:4409 , piece_id=3GJUEQA526TZ7HBZSYLADSRYA4QT5QOXZSVALMNQT3PJCAR5EMKQ, seg_pos={0 0}, upload=472638]: Done uploading piece. err: <nil> Dec 28 02:25:04.214521644 (538h56m4.085604689s) [noise=true , address=145.79.227.11:4732 , piece_id=RYYB3DLNDZ3X3UDY7E2XC3PFON26C6JALJMAC4RFNFRUP53CITEQ, seg_pos={0 0}, upload=472638]: Done uploading piece. err: <nil> Dec 28 02:25:04.215348508 (538h56m4.086431543s) [noise=true , address=145.79.227.50:4375 , piece_id=TJJEXYTFY77ISKFQFV3UQF7GBC7QUFLUC7UAHO6DJTPBJYX5Y4QQ, seg_pos={0 0}, upload=472638]: Done uploading piece. err: <nil> Dec 28 02:25:04.219124578 (538h56m4.090207623s) [noise=true , address=145.79.227.45:4230 , piece_id=XOC6E5VSNO2P7NZEVNRX2JI6ZPMTEDZA3ETHNCEJ53OSABIWYCSQ, seg_pos={0 0}, upload=472638]: Done uploading piece. err: <nil> Dec 28 02:25:04.220968539 (538h56m4.092051584s) [noise=true , address=145.79.227.36:4518 , piece_id=HILGHKIS3FEPKJQD4ZSLPYDIPNRHGUMJB5KVPZ7EBZCHURPTQ5NQ, seg_pos={0 0}, upload=472638]: Done uploading piece. err: <nil> Dec 28 02:25:04.227872954 (538h56m4.098955999s) [noise=true , address=145.79.227.15:4874 , piece_id=4MLUNHLW44OJDNXFYPDAPVCNA6IA7VSLBHS7EH6ET6KUJGWFIRVQ, seg_pos={0 0}, upload=472638]: Done uploading piece. err: <nil> Dec 28 02:25:04.244820751 (538h56m4.115903797s) [noise=true , address=145.79.227.20:4438 , piece_id=N4RAOFVG5R4EPPMRYU4QB25TFWPHUZYQKJQWX7D4CORDJYPJOILQ, seg_pos={0 0}, upload=472638]: Done uploading piece. err: <nil> Dec 28 02:25:04.285383571 (538h56m4.156466616s) [noise=true , address=145.79.227.7:4293 , piece_id=5E7KXS6G3KPFX5F3NLZJ6KQN2AQ6N3GFDFIU2HOXK6UJOMBNEUTQ, seg_pos={0 0}, upload=472638]: Done uploading piece. err: <nil> Dec 28 02:25:04.447648052 (538h56m4.318731087s) [noise=true , address=45.91.116.7:4010 , piece_id=NFQSJSHRNYXKDY5IWKRLBZ6PGNNAEZYIXBG45HHPIMDBYXHQUV3Q, seg_pos={0 0}, upload=472638]: Done uploading piece. err: <nil> Dec 28 02:25:04.467486162 (538h56m4.338569208s) [noise=true , address=77.90.31.14:4661 , piece_id=YYJZWPEEYCTURKA2G77QLY7HIHUZG7AU4P2KXEHBLFVHNAQKOJ5Q, seg_pos={0 0}, upload=472638]: Done uploading piece. err: <nil> Dec 28 02:25:04.524147493 (538h56m4.395230548s) [noise=true , address=77.90.31.10:4519 , piece_id=57THD7AISKOYJWZSCWR6LKU2D2GV5CLTFUY2PDVIVHKARFMDBVNA, seg_pos={0 0}, upload=472638]: Done uploading piece. err: <nil> Dec 28 02:25:04.537582943 (538h56m4.408665988s) [noise=true , address=45.91.116.4:4939 , piece_id=5D5BATVNTNMNWWR2BPICRBU5PAI2MSITD2UH3YAZ23XXILW3H24Q, seg_pos={0 0}, upload=472638]: Done uploading piece. err: <nil> Dec 28 02:25:04.537614022 (538h56m4.408697068s) [seg_pos={0 0}, upload=472638]: Segment reached optimal threshold of 13 pieces. Dec 28 02:25:04.537946204 (538h56m4.40902925s) [noise=true , address=45.91.116.11:4118 , piece_id=DFV2ZKQZSS3MS7NXSFZUXF5RPGAQCSVCV3PVBKENI4SMHRU4E3KQ, seg_pos={0 0}, upload=472638]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF Dec 28 02:25:04.537956505 (538h56m4.40903955s) [noise=true , address=45.91.116.11:4118 , piece_id=DFV2ZKQZSS3MS7NXSFZUXF5RPGAQCSVCV3PVBKENI4SMHRU4E3KQ, seg_pos={0 0}, upload=472638]: Long tail cancellation due to optimal threshold Dec 28 02:25:04.538156876 (538h56m4.409239921s) [noise=true , address=45.91.116.14:4192 , piece_id=KJ5TRZIXDCJITTG6ISFBKG5VDSVFR2KIPG74MZTC36G56CC57TDA, seg_pos={0 0}, upload=472638]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: write tcp 188.166.225.170:57712->45.91.116.14:4192: use of closed network connection; write tcp 188.166.225.170:57712->45.91.116.14:4192: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57712->45.91.116.14:4192: use of closed network connection Dec 28 02:25:04.538166604 (538h56m4.409249669s) [noise=true , address=45.91.116.14:4192 , piece_id=KJ5TRZIXDCJITTG6ISFBKG5VDSVFR2KIPG74MZTC36G56CC57TDA, seg_pos={0 0}, upload=472638]: Long tail cancellation due to optimal threshold Dec 28 02:25:04.539139897 (538h56m4.410222952s) [noise=true , address=45.91.116.5:4297 , piece_id=7PEGDQUHFYYZLQKMYYF3AGUIJXG7ZKBGZ5TGEVIOWG7RPDSSGWJA, seg_pos={0 0}, upload=472638]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 28 02:25:04.539166298 (538h56m4.410249353s) [noise=true , address=45.91.116.5:4297 , piece_id=7PEGDQUHFYYZLQKMYYF3AGUIJXG7ZKBGZ5TGEVIOWG7RPDSSGWJA, seg_pos={0 0}, upload=472638]: Long tail cancellation due to optimal threshold Dec 28 02:25:04.539209860 (538h56m4.410292905s) [seg_pos={0 0}, upload=472638]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 02:25:04.539335009 (538h56m4.410418054s) [upload=472638]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 02:25:04.539343595 (538h56m4.41042664s) [upload=472638]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 02:25:04.844931847 (538h56m4.716014932s) [upload=472638]: Done uploading segments... Dec 28 02:25:04.844958798 (538h56m4.716041843s) [upload=472638]: Upload finished. err: <nil> Duration: 30.002684841s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [................................X.........................X.] duration: 4.045786463s trace: https://jaeger.dev.storj.io/trace/4aff4da4d82027c9 FAIL (2025-12-28 02:25:22 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.045824805s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.................................X.........................X] duration: 5.001354609s trace: https://jaeger.dev.storj.io/trace/4aff4da4d82027c9 FAIL (2025-12-28 02:25:02 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001382511s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................................X.........................] duration: 3.458363384s trace: https://jaeger.dev.storj.io/trace/7fc142ad3d711e34 FAIL (2025-12-28 02:24:33 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.458391568s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 31 out of 60 is failed (threshold=3) [X..X.XXX.X.XX.XX.X..X.X.X.X.X..X...XX..X.XXX.XXX.X.....XXX.X] duration: 30.003534963s trace: https://jaeger.dev.storj.io/trace/21f0755736489735 FAIL (2025-12-28 02:24:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: T5ERZRHDFHNK6SE7V5IJN6VPSGMHKQTBCE7E5SAYSAD7BNT4PQ3Q) context deadline exceeded; piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: Q4PO4H5VP5AOOE5A72L42DOM4ERRLYFRZAEV5LKW75FIMUAAVOWQ) context deadline exceeded ======LOGS====== 28 02:23:51.365530267 (538h54m51.236613312s) [upload=472620]: Starting upload Dec 28 02:23:51.365603587 (538h54m51.236686632s) [upload=472620]: Done starting upload Dec 28 02:23:51.365652651 (538h54m51.236735696s) [upload=472620]: Uploading segments... Dec 28 02:23:51.365875385 (538h54m51.23695842s) [upload=472620]: Got next segment. Inline: false Dec 28 02:23:51.365905552 (538h54m51.236988587s) [upload=472620]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 02:23:51.365912465 (538h54m51.23699551s) [upload=472620]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 02:23:51.613500980 (538h54m51.484584015s) [seg_pos={0 0}, upload=472620]: Begin upload segment... Dec 28 02:23:51.613555594 (538h54m51.484638639s) [seg_pos={0 0}, upload=472620]: Done begin upload segment. Dec 28 02:23:51.613596292 (538h54m51.484679327s) [noise=true , address=145.79.227.24:4583 , piece_id=JKQT7WLOKUTB3BPVZ5EBGABTEB6B7Z5X7L76JFHR3ZSFIK3P3YYQ, seg_pos={0 0}, upload=472620]: Uploading piece... Dec 28 02:23:51.613738362 (538h54m51.484821407s) [noise=true , address=45.91.116.14:4191 , piece_id=IIJ46FNSY5KB2CLOACJ2V6WGBWWZHQVY62CDOBXEP2JRR2I2CGNQ, seg_pos={0 0}, upload=472620]: Uploading piece... Dec 28 02:23:51.614280304 (538h54m51.485363349s) [noise=true , address=145.79.227.14:4838 , piece_id=ZTGZOWOO747I5LGBWZQN56WXGKDOPU2AWFM27R7GBOQML5WQG47Q, seg_pos={0 0}, upload=472620]: Uploading piece... Dec 28 02:23:51.614841843 (538h54m51.485924888s) [noise=true , address=77.90.31.16:4735 , piece_id=T5ERZRHDFHNK6SE7V5IJN6VPSGMHKQTBCE7E5SAYSAD7BNT4PQ3Q, seg_pos={0 0}, upload=472620]: Uploading piece... Dec 28 02:23:51.614933959 (538h54m51.486017004s) [noise=true , address=145.79.227.47:4265 , piece_id=ZHI6LY2M3FIZWMDLRAGCWAKWZHOM4X62Z4APV3MI3IWMEANTA4XQ, seg_pos={0 0}, upload=472620]: Uploading piece... Dec 28 02:23:51.615249269 (538h54m51.486332314s) [noise=true , address=145.79.227.39:4014 , piece_id=Y76GG6PEEUONGBEDBFNVUGR5DT352PCOXD7D6QMGS5TJQRP76V5Q, seg_pos={0 0}, upload=472620]: Uploading piece... Dec 28 02:23:51.615419082 (538h54m51.486502127s) [noise=true , address=145.79.227.42:4122 , piece_id=4MZLCZSTKUZ33TKERXUB6HAIK4RCEUG2JCP4KDU2AGIMACW72WVQ, seg_pos={0 0}, upload=472620]: Uploading piece... Dec 28 02:23:51.615633851 (538h54m51.486716886s) [noise=true , address=77.90.31.13:4627 , piece_id=TZCGEGUS37SLTSS76DLXS6FMYVTVYR47EYYVPYMVMXJRZEZGZ4MQ, seg_pos={0 0}, upload=472620]: Uploading piece... Dec 28 02:23:51.615921980 (538h54m51.487005015s) [noise=true , address=145.79.227.15:4873 , piece_id=DJY54LVF7TPINJWNKYCP4G6FC4KTT6LCKDF3SANOQW3MNJ7UVLWQ, seg_pos={0 0}, upload=472620]: Uploading piece... Dec 28 02:23:51.616246197 (538h54m51.487329242s) [noise=true , address=45.91.116.7:4011 , piece_id=Q4PO4H5VP5AOOE5A72L42DOM4ERRLYFRZAEV5LKW75FIMUAAVOWQ, seg_pos={0 0}, upload=472620]: Uploading piece... Dec 28 02:23:51.616618315 (538h54m51.487701361s) [noise=true , address=145.79.227.36:4518 , piece_id=P2YQDHT4TSSWAO6RHNECPEAGVNEPY5ABJYBH4VOENUEMMASPHGHA, seg_pos={0 0}, upload=472620]: Uploading piece... Dec 28 02:23:51.617006705 (538h54m51.48808975s) [noise=true , address=45.91.116.10:4082 , piece_id=AD7PEUFCVXGHV6VACCZE5CGM4G4UJRUTVZUEQTMRW6SY5R2ELCCA, seg_pos={0 0}, upload=472620]: Uploading piece... Dec 28 02:23:51.617019359 (538h54m51.488102404s) [noise=true , address=145.79.227.16:4911 , piece_id=S3UVZC7CC2D73HF6HDTVRIJR7GASTKJKTXAMY6GQC772KPSTTWIA, seg_pos={0 0}, upload=472620]: Uploading piece... Dec 28 02:23:51.617363995 (538h54m51.48844704s) [noise=true , address=145.79.227.6:4257 , piece_id=PN6LYFDXVTYFCMVSR6U2UXCWNNEPCEAOVLMMQYN6H7MVA6EJTTJQ, seg_pos={0 0}, upload=472620]: Uploading piece... Dec 28 02:23:51.617567633 (538h54m51.488650678s) [noise=true , address=45.91.116.13:4153 , piece_id=CMYHM36AAMUKIBJH7DNAUTCIYYWXNKQ7N4TIGNAEREKCVPG72CDA, seg_pos={0 0}, upload=472620]: Uploading piece... Dec 28 02:23:51.617712138 (538h54m51.488795183s) [noise=true , address=145.79.227.11:4732 , piece_id=4MS72WGIFJD4VKJHRX75PHML3SO6A5D72MVSDCEFLKNSS4SRK3SQ, seg_pos={0 0}, upload=472620]: Uploading piece... Dec 28 02:23:53.337907525 (538h54m53.20899057s) [upload=472620]: Next returned nil segment Dec 28 02:23:53.337933764 (538h54m53.209016809s) [upload=472620]: Waiting for error group managing segments... Dec 28 02:23:53.617786056 (538h54m53.488869252s) [noise=true , address=145.79.227.14:4838 , piece_id=ZTGZOWOO747I5LGBWZQN56WXGKDOPU2AWFM27R7GBOQML5WQG47Q, seg_pos={0 0}, upload=472620]: Done uploading piece. err: <nil> Dec 28 02:23:53.627682869 (538h54m53.498765904s) [noise=true , address=145.79.227.42:4122 , piece_id=4MZLCZSTKUZ33TKERXUB6HAIK4RCEUG2JCP4KDU2AGIMACW72WVQ, seg_pos={0 0}, upload=472620]: Done uploading piece. err: <nil> Dec 28 02:23:53.638987172 (538h54m53.510070217s) [noise=true , address=145.79.227.36:4518 , piece_id=P2YQDHT4TSSWAO6RHNECPEAGVNEPY5ABJYBH4VOENUEMMASPHGHA, seg_pos={0 0}, upload=472620]: Done uploading piece. err: <nil> Dec 28 02:23:53.639562969 (538h54m53.510646014s) [noise=true , address=145.79.227.11:4732 , piece_id=4MS72WGIFJD4VKJHRX75PHML3SO6A5D72MVSDCEFLKNSS4SRK3SQ, seg_pos={0 0}, upload=472620]: Done uploading piece. err: <nil> Dec 28 02:23:53.645642193 (538h54m53.516725268s) [noise=true , address=145.79.227.15:4873 , piece_id=DJY54LVF7TPINJWNKYCP4G6FC4KTT6LCKDF3SANOQW3MNJ7UVLWQ, seg_pos={0 0}, upload=472620]: Done uploading piece. err: <nil> Dec 28 02:23:53.650570816 (538h54m53.521653851s) [noise=true , address=145.79.227.47:4265 , piece_id=ZHI6LY2M3FIZWMDLRAGCWAKWZHOM4X62Z4APV3MI3IWMEANTA4XQ, seg_pos={0 0}, upload=472620]: Done uploading piece. err: <nil> Dec 28 02:23:53.653406906 (538h54m53.524489941s) [noise=true , address=145.79.227.16:4911 , piece_id=S3UVZC7CC2D73HF6HDTVRIJR7GASTKJKTXAMY6GQC772KPSTTWIA, seg_pos={0 0}, upload=472620]: Done uploading piece. err: <nil> Dec 28 02:23:53.736298317 (538h54m53.607381352s) [noise=true , address=145.79.227.39:4014 , piece_id=Y76GG6PEEUONGBEDBFNVUGR5DT352PCOXD7D6QMGS5TJQRP76V5Q, seg_pos={0 0}, upload=472620]: Done uploading piece. err: <nil> Dec 28 02:23:53.738681915 (538h54m53.60976495s) [noise=true , address=145.79.227.24:4583 , piece_id=JKQT7WLOKUTB3BPVZ5EBGABTEB6B7Z5X7L76JFHR3ZSFIK3P3YYQ, seg_pos={0 0}, upload=472620]: Done uploading piece. err: <nil> Dec 28 02:23:53.804576174 (538h54m53.675659209s) [noise=true , address=77.90.31.13:4627 , piece_id=TZCGEGUS37SLTSS76DLXS6FMYVTVYR47EYYVPYMVMXJRZEZGZ4MQ, seg_pos={0 0}, upload=472620]: Done uploading piece. err: <nil> Dec 28 02:23:53.852768990 (538h54m53.723852035s) [noise=true , address=45.91.116.7:4011 , piece_id=Q4PO4H5VP5AOOE5A72L42DOM4ERRLYFRZAEV5LKW75FIMUAAVOWQ, seg_pos={0 0}, upload=472620]: Done uploading piece. err: <nil> Dec 28 02:23:53.871818379 (538h54m53.742901414s) [noise=true , address=45.91.116.13:4153 , piece_id=CMYHM36AAMUKIBJH7DNAUTCIYYWXNKQ7N4TIGNAEREKCVPG72CDA, seg_pos={0 0}, upload=472620]: Done uploading piece. err: <nil> Dec 28 02:23:53.913652779 (538h54m53.784735824s) [noise=true , address=77.90.31.16:4735 , piece_id=T5ERZRHDFHNK6SE7V5IJN6VPSGMHKQTBCE7E5SAYSAD7BNT4PQ3Q, seg_pos={0 0}, upload=472620]: Done uploading piece. err: <nil> Dec 28 02:23:53.913685261 (538h54m53.784768296s) [seg_pos={0 0}, upload=472620]: Segment reached optimal threshold of 13 pieces. Dec 28 02:23:53.914073190 (538h54m53.785156265s) [noise=true , address=145.79.227.6:4257 , piece_id=PN6LYFDXVTYFCMVSR6U2UXCWNNEPCEAOVLMMQYN6H7MVA6EJTTJQ, seg_pos={0 0}, upload=472620]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK): protocol: write tcp 188.166.225.170:52404->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:52404->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52404->145.79.227.6:4257: use of closed network connection Dec 28 02:23:53.914089380 (538h54m53.785172425s) [noise=true , address=145.79.227.6:4257 , piece_id=PN6LYFDXVTYFCMVSR6U2UXCWNNEPCEAOVLMMQYN6H7MVA6EJTTJQ, seg_pos={0 0}, upload=472620]: Long tail cancellation due to optimal threshold Dec 28 02:23:53.914731653 (538h54m53.785814708s) [noise=true , address=45.91.116.14:4191 , piece_id=IIJ46FNSY5KB2CLOACJ2V6WGBWWZHQVY62CDOBXEP2JRR2I2CGNQ, seg_pos={0 0}, upload=472620]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 28 02:23:53.914741863 (538h54m53.785824908s) [noise=true , address=45.91.116.14:4191 , piece_id=IIJ46FNSY5KB2CLOACJ2V6WGBWWZHQVY62CDOBXEP2JRR2I2CGNQ, seg_pos={0 0}, upload=472620]: Long tail cancellation due to optimal threshold Dec 28 02:23:53.914993903 (538h54m53.786076958s) [noise=true , address=45.91.116.10:4082 , piece_id=AD7PEUFCVXGHV6VACCZE5CGM4G4UJRUTVZUEQTMRW6SY5R2ELCCA, seg_pos={0 0}, upload=472620]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 28 02:23:53.915004532 (538h54m53.786087588s) [noise=true , address=45.91.116.10:4082 , piece_id=AD7PEUFCVXGHV6VACCZE5CGM4G4UJRUTVZUEQTMRW6SY5R2ELCCA, seg_pos={0 0}, upload=472620]: Long tail cancellation due to optimal threshold Dec 28 02:23:53.915024902 (538h54m53.786107947s) [seg_pos={0 0}, upload=472620]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 02:23:53.915106907 (538h54m53.786189953s) [upload=472620]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 02:23:53.915113069 (538h54m53.786196114s) [upload=472620]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 02:23:54.200352007 (538h54m54.071435042s) [upload=472620]: Done uploading segments... Dec 28 02:23:54.200370702 (538h54m54.071453737s) [upload=472620]: Upload finished. err: <nil> Duration: 30.003559329s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................................X........................] duration: 3.464770174s trace: https://jaeger.dev.storj.io/trace/7fc142ad3d711e34 FAIL (2025-12-28 02:24:16 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.464792305s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................X...................X.....] duration: 15.374123745s trace: https://jaeger.dev.storj.io/trace/1f68404b41391e4a FAIL (2025-12-28 02:23:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12CaBSQgwTru9NXaznMJ2XExKe3yhLoUCvDKFvqvLJGJev4mEPc, Piece ID: ULPCWUUHAL35F5WZYHDQLO7VBIGAUH2XHH4I2NYYHG7OAC5VJFOA) context deadline exceeded; piecestore: (Node ID: 1H1XywjaRt5ZRGqmZBgzfSowuCnPqFesL1Ch2jYYS1gV2oBgQX, Piece ID: EMRORAB74LXEB6222NUCRM5ZENJC2AZNK7Q2V2NSQMVD5TIRGVGQ) context deadline exceeded; piecestore: (Node ID: 129YE2Wd6LmDABJFaGVtYDwCHmStVDnPRCagD2cKPWA5PDTtZyh, Piece ID: UEKG3FOTTORJVQRPLAFCJEP2ZAVA5N7M2DFGH4BUTB2OGVC2KIZA) context deadline exceeded ======LOGS====== upload=472552]: Uploading piece... Dec 28 02:19:02.432783751 (538h50m2.303866796s) [noise=true , address=145.79.227.33:4896 , piece_id=D44WQB3G3NBNSB6ITIBHVV67V2EKPGXXONQU25IE6RVSGGTUJ44Q, seg_pos={0 0}, upload=472552]: Uploading piece... Dec 28 02:19:02.432842112 (538h50m2.303925147s) [noise=true , address=77.90.31.21:4900 , piece_id=UEKG3FOTTORJVQRPLAFCJEP2ZAVA5N7M2DFGH4BUTB2OGVC2KIZA, seg_pos={0 0}, upload=472552]: Uploading piece... Dec 28 02:19:02.433283032 (538h50m2.304366077s) [noise=true , address=77.90.31.15:4703 , piece_id=BERLD7G2DBBDBNE3MCS57VXAKXL3P4RQIV47ZOCU6VXMPJU4QITA, seg_pos={0 0}, upload=472552]: Uploading piece... Dec 28 02:19:02.433734562 (538h50m2.304817617s) [noise=true , address=77.90.31.16:4741 , piece_id=CX66NBDSLZWQ34Z2UDYM6HM73TABJSVHSGBQRONAQ3S6ENVTY3KQ, seg_pos={0 0}, upload=472552]: Uploading piece... Dec 28 02:19:02.434178757 (538h50m2.305261802s) [noise=true , address=77.90.31.13:4646 , piece_id=V5HTB4HVJVO7OG745JBRQSISFZU6UOORKLHEVDUDUR24EWRSSVOQ, seg_pos={0 0}, upload=472552]: Uploading piece... Dec 28 02:19:02.434595160 (538h50m2.305678195s) [noise=true , address=145.79.227.37:4596 , piece_id=XBEKEUWGAAAWZGOV3PBH7NE2CX5VID75MFGWMPHTNR7MV66U3XNA, seg_pos={0 0}, upload=472552]: Uploading piece... Dec 28 02:19:02.435217555 (538h50m2.3063006s) [noise=true , address=45.91.116.11:4136 , piece_id=JD6QHJCQ6D53BGKDPSPYBBHNCP7XDZCSV7H6ARD3JNBL75NULT6Q, seg_pos={0 0}, upload=472552]: Uploading piece... Dec 28 02:19:02.435684624 (538h50m2.306767669s) [noise=true , address=145.79.227.32:4856 , piece_id=II5XB6TRHNQAAHBPP5PRMMA7BCBGR5QO4HUBL4YJ7ZLLMK4DJB5Q, seg_pos={0 0}, upload=472552]: Uploading piece... Dec 28 02:19:02.436105054 (538h50m2.307188099s) [noise=true , address=145.79.227.14:4842 , piece_id=3F6EV4TB5TQNK5W5CJHRCCNORCEEAA6AXCOJQTQ6WJNRZG3YILOQ, seg_pos={0 0}, upload=472552]: Uploading piece... Dec 28 02:19:02.436528130 (538h50m2.307611175s) [noise=true , address=77.90.31.12:4610 , piece_id=C7IRMHAHNXDZLKFBL7OVTNVI2KWMFREMFE5GR4USVMNT5QRB3HXQ, seg_pos={0 0}, upload=472552]: Uploading piece... Dec 28 02:19:02.437497977 (538h50m2.308581022s) [noise=true , address=145.79.227.49:4340 , piece_id=5C75E4QGOX6G7QNOUUKX4GC7YL5MESWBOFSNCA6PO4PPSMQKIVFQ, seg_pos={0 0}, upload=472552]: Uploading piece... Dec 28 02:19:02.439092573 (538h50m2.310175618s) [noise=true , address=45.91.116.11:4134 , piece_id=D4PXHXGBZGH3UU2CNKDRCWD44IJQR5BQFWHCNQRPZDNYVV755ICQ, seg_pos={0 0}, upload=472552]: Uploading piece... Dec 28 02:19:02.440580235 (538h50m2.31166328s) [noise=true , address=45.91.116.5:4318 , piece_id=ULPCWUUHAL35F5WZYHDQLO7VBIGAUH2XHH4I2NYYHG7OAC5VJFOA, seg_pos={0 0}, upload=472552]: Uploading piece... Dec 28 02:19:02.442091032 (538h50m2.313174077s) [noise=true , address=77.90.31.11:4558 , piece_id=TWOTPEDKIRIOR7NQNCEFM3Z5KGWVOFUYKIQYMEGHED5FKRDCB54Q, seg_pos={0 0}, upload=472552]: Uploading piece... Dec 28 02:19:02.443305184 (538h50m2.314388229s) [noise=true , address=145.79.227.40:4078 , piece_id=3ICUQQR7NS6O2QYHBFCAQZS7A7UAWHTD3O2LOWLRZ3OCMU3WWBSA, seg_pos={0 0}, upload=472552]: Uploading piece... Dec 28 02:19:02.447829959 (538h50m2.318913014s) [noise=true , address=145.79.227.14:4863 , piece_id=N3XS43U6I4WJYUCC5NVUBRLTTHVPYKDR3KRCNGJRYU3E26WDO3HA, seg_pos={0 0}, upload=472552]: Uploading piece... Dec 28 02:19:02.451586160 (538h50m2.322669225s) [noise=true , address=45.91.116.9:4288 , piece_id=EMRORAB74LXEB6222NUCRM5ZENJC2AZNK7Q2V2NSQMVD5TIRGVGQ, seg_pos={0 0}, upload=472552]: Uploading piece... Dec 28 02:19:03.923272933 (538h50m3.794355978s) [upload=472552]: Next returned nil segment Dec 28 02:19:03.923289645 (538h50m3.79437269s) [upload=472552]: Waiting for error group managing segments... Dec 28 02:19:04.324829202 (538h50m4.195912237s) [noise=true , address=145.79.227.49:4340 , piece_id=5C75E4QGOX6G7QNOUUKX4GC7YL5MESWBOFSNCA6PO4PPSMQKIVFQ, seg_pos={0 0}, upload=472552]: Done uploading piece. err: <nil> Dec 28 02:19:04.335200317 (538h50m4.206283363s) [noise=true , address=145.79.227.14:4842 , piece_id=3F6EV4TB5TQNK5W5CJHRCCNORCEEAA6AXCOJQTQ6WJNRZG3YILOQ, seg_pos={0 0}, upload=472552]: Done uploading piece. err: <nil> Dec 28 02:19:04.343899841 (538h50m4.214982886s) [noise=true , address=145.79.227.14:4863 , piece_id=N3XS43U6I4WJYUCC5NVUBRLTTHVPYKDR3KRCNGJRYU3E26WDO3HA, seg_pos={0 0}, upload=472552]: Done uploading piece. err: <nil> Dec 28 02:19:04.378111328 (538h50m4.249194373s) [noise=true , address=145.79.227.37:4596 , piece_id=XBEKEUWGAAAWZGOV3PBH7NE2CX5VID75MFGWMPHTNR7MV66U3XNA, seg_pos={0 0}, upload=472552]: Done uploading piece. err: <nil> Dec 28 02:19:04.381135876 (538h50m4.252218921s) [noise=true , address=145.79.227.32:4856 , piece_id=II5XB6TRHNQAAHBPP5PRMMA7BCBGR5QO4HUBL4YJ7ZLLMK4DJB5Q, seg_pos={0 0}, upload=472552]: Done uploading piece. err: <nil> Dec 28 02:19:04.476390445 (538h50m4.34747348s) [noise=true , address=145.79.227.6:4290 , piece_id=K4OGXS2MXEFVQVW7HEAZJ6267HWETOUXWJAC77JNZADW76K2XMJA, seg_pos={0 0}, upload=472552]: Done uploading piece. err: <nil> Dec 28 02:19:04.533625848 (538h50m4.404708893s) [noise=true , address=145.79.227.33:4896 , piece_id=D44WQB3G3NBNSB6ITIBHVV67V2EKPGXXONQU25IE6RVSGGTUJ44Q, seg_pos={0 0}, upload=472552]: Done uploading piece. err: <nil> Dec 28 02:19:04.562532124 (538h50m4.433615159s) [noise=true , address=77.90.31.11:4558 , piece_id=TWOTPEDKIRIOR7NQNCEFM3Z5KGWVOFUYKIQYMEGHED5FKRDCB54Q, seg_pos={0 0}, upload=472552]: Done uploading piece. err: <nil> Dec 28 02:19:04.643414509 (538h50m4.514497544s) [noise=true , address=77.90.31.21:4900 , piece_id=UEKG3FOTTORJVQRPLAFCJEP2ZAVA5N7M2DFGH4BUTB2OGVC2KIZA, seg_pos={0 0}, upload=472552]: Done uploading piece. err: <nil> Dec 28 02:19:04.651013327 (538h50m4.522096372s) [noise=true , address=77.90.31.13:4646 , piece_id=V5HTB4HVJVO7OG745JBRQSISFZU6UOORKLHEVDUDUR24EWRSSVOQ, seg_pos={0 0}, upload=472552]: Done uploading piece. err: <nil> Dec 28 02:19:04.776667164 (538h50m4.647750209s) [noise=true , address=45.91.116.11:4136 , piece_id=JD6QHJCQ6D53BGKDPSPYBBHNCP7XDZCSV7H6ARD3JNBL75NULT6Q, seg_pos={0 0}, upload=472552]: Done uploading piece. err: <nil> Dec 28 02:19:04.919205529 (538h50m4.790288574s) [noise=true , address=45.91.116.11:4134 , piece_id=D4PXHXGBZGH3UU2CNKDRCWD44IJQR5BQFWHCNQRPZDNYVV755ICQ, seg_pos={0 0}, upload=472552]: Done uploading piece. err: <nil> Dec 28 02:19:04.927993490 (538h50m4.799076535s) [noise=true , address=45.91.116.9:4288 , piece_id=EMRORAB74LXEB6222NUCRM5ZENJC2AZNK7Q2V2NSQMVD5TIRGVGQ, seg_pos={0 0}, upload=472552]: Done uploading piece. err: <nil> Dec 28 02:19:05.041729347 (538h50m4.912812392s) [noise=true , address=45.91.116.5:4318 , piece_id=ULPCWUUHAL35F5WZYHDQLO7VBIGAUH2XHH4I2NYYHG7OAC5VJFOA, seg_pos={0 0}, upload=472552]: Done uploading piece. err: <nil> Dec 28 02:19:05.041751558 (538h50m4.912834593s) [seg_pos={0 0}, upload=472552]: Segment reached optimal threshold of 14 pieces. Dec 28 02:19:05.041986606 (538h50m4.913069651s) [noise=true , address=77.90.31.12:4610 , piece_id=C7IRMHAHNXDZLKFBL7OVTNVI2KWMFREMFE5GR4USVMNT5QRB3HXQ, seg_pos={0 0}, upload=472552]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi): protocol: expected piece hash; context canceled; EOF Dec 28 02:19:05.041995954 (538h50m4.913078999s) [noise=true , address=77.90.31.12:4610 , piece_id=C7IRMHAHNXDZLKFBL7OVTNVI2KWMFREMFE5GR4USVMNT5QRB3HXQ, seg_pos={0 0}, upload=472552]: Long tail cancellation due to optimal threshold Dec 28 02:19:05.042149837 (538h50m4.913232872s) [noise=true , address=77.90.31.15:4703 , piece_id=BERLD7G2DBBDBNE3MCS57VXAKXL3P4RQIV47ZOCU6VXMPJU4QITA, seg_pos={0 0}, upload=472552]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hMDD5K7WDjvN4ZgcpwpnPFb6X1ncCfyucfFrpTUshd8Ye1B6D): protocol: expected piece hash; context canceled; EOF Dec 28 02:19:05.042158714 (538h50m4.913241759s) [noise=true , address=77.90.31.15:4703 , piece_id=BERLD7G2DBBDBNE3MCS57VXAKXL3P4RQIV47ZOCU6VXMPJU4QITA, seg_pos={0 0}, upload=472552]: Long tail cancellation due to optimal threshold Dec 28 02:19:05.042299752 (538h50m4.913382788s) [noise=true , address=77.90.31.16:4741 , piece_id=CX66NBDSLZWQ34Z2UDYM6HM73TABJSVHSGBQRONAQ3S6ENVTY3KQ, seg_pos={0 0}, upload=472552]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR): protocol: expected piece hash; context canceled; EOF Dec 28 02:19:05.042304702 (538h50m4.913387737s) [noise=true , address=77.90.31.16:4741 , piece_id=CX66NBDSLZWQ34Z2UDYM6HM73TABJSVHSGBQRONAQ3S6ENVTY3KQ, seg_pos={0 0}, upload=472552]: Long tail cancellation due to optimal threshold Dec 28 02:19:05.042396276 (538h50m4.913479352s) [noise=true , address=145.79.227.40:4078 , piece_id=3ICUQQR7NS6O2QYHBFCAQZS7A7UAWHTD3O2LOWLRZ3OCMU3WWBSA, seg_pos={0 0}, upload=472552]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tzgtenV6us5ssD7xWGpZmVUoWEfbFhBNewbK9irHtZPkyDtRd): protocol: expected piece hash; context canceled; EOF Dec 28 02:19:05.042417406 (538h50m4.913500461s) [noise=true , address=145.79.227.40:4078 , piece_id=3ICUQQR7NS6O2QYHBFCAQZS7A7UAWHTD3O2LOWLRZ3OCMU3WWBSA, seg_pos={0 0}, upload=472552]: Long tail cancellation due to optimal threshold Dec 28 02:19:05.042612779 (538h50m4.913695814s) [seg_pos={0 0}, upload=472552]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 02:19:05.042821726 (538h50m4.913904771s) [upload=472552]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 02:19:05.042916818 (538h50m4.913999863s) [upload=472552]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 02:19:05.336925913 (538h50m5.208008958s) [upload=472552]: Done uploading segments... Dec 28 02:19:05.336953856 (538h50m5.208036901s) [upload=472552]: Upload finished. err: <nil> Duration: 15.374150348s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................................X.......................] duration: 3.536074102s trace: https://jaeger.dev.storj.io/trace/7fc142ad3d711e34 FAIL (2025-12-28 02:23:33 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.536100222s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................X...................X....] duration: 50.861984922s trace: https://jaeger.dev.storj.io/trace/1f68404b41391e4a FAIL (2025-12-28 02:23:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12CaBSQgwTru9NXaznMJ2XExKe3yhLoUCvDKFvqvLJGJev4mEPc, Piece ID: ULPCWUUHAL35F5WZYHDQLO7VBIGAUH2XHH4I2NYYHG7OAC5VJFOA) context deadline exceeded; piecestore: (Node ID: 1H1XywjaRt5ZRGqmZBgzfSowuCnPqFesL1Ch2jYYS1gV2oBgQX, Piece ID: EMRORAB74LXEB6222NUCRM5ZENJC2AZNK7Q2V2NSQMVD5TIRGVGQ) context deadline exceeded; piecestore: (Node ID: 129YE2Wd6LmDABJFaGVtYDwCHmStVDnPRCagD2cKPWA5PDTtZyh, Piece ID: UEKG3FOTTORJVQRPLAFCJEP2ZAVA5N7M2DFGH4BUTB2OGVC2KIZA) context deadline exceeded ======LOGS====== upload=472552]: Uploading piece... Dec 28 02:19:02.432783751 (538h50m2.303866796s) [noise=true , address=145.79.227.33:4896 , piece_id=D44WQB3G3NBNSB6ITIBHVV67V2EKPGXXONQU25IE6RVSGGTUJ44Q, seg_pos={0 0}, upload=472552]: Uploading piece... Dec 28 02:19:02.432842112 (538h50m2.303925147s) [noise=true , address=77.90.31.21:4900 , piece_id=UEKG3FOTTORJVQRPLAFCJEP2ZAVA5N7M2DFGH4BUTB2OGVC2KIZA, seg_pos={0 0}, upload=472552]: Uploading piece... Dec 28 02:19:02.433283032 (538h50m2.304366077s) [noise=true , address=77.90.31.15:4703 , piece_id=BERLD7G2DBBDBNE3MCS57VXAKXL3P4RQIV47ZOCU6VXMPJU4QITA, seg_pos={0 0}, upload=472552]: Uploading piece... Dec 28 02:19:02.433734562 (538h50m2.304817617s) [noise=true , address=77.90.31.16:4741 , piece_id=CX66NBDSLZWQ34Z2UDYM6HM73TABJSVHSGBQRONAQ3S6ENVTY3KQ, seg_pos={0 0}, upload=472552]: Uploading piece... Dec 28 02:19:02.434178757 (538h50m2.305261802s) [noise=true , address=77.90.31.13:4646 , piece_id=V5HTB4HVJVO7OG745JBRQSISFZU6UOORKLHEVDUDUR24EWRSSVOQ, seg_pos={0 0}, upload=472552]: Uploading piece... Dec 28 02:19:02.434595160 (538h50m2.305678195s) [noise=true , address=145.79.227.37:4596 , piece_id=XBEKEUWGAAAWZGOV3PBH7NE2CX5VID75MFGWMPHTNR7MV66U3XNA, seg_pos={0 0}, upload=472552]: Uploading piece... Dec 28 02:19:02.435217555 (538h50m2.3063006s) [noise=true , address=45.91.116.11:4136 , piece_id=JD6QHJCQ6D53BGKDPSPYBBHNCP7XDZCSV7H6ARD3JNBL75NULT6Q, seg_pos={0 0}, upload=472552]: Uploading piece... Dec 28 02:19:02.435684624 (538h50m2.306767669s) [noise=true , address=145.79.227.32:4856 , piece_id=II5XB6TRHNQAAHBPP5PRMMA7BCBGR5QO4HUBL4YJ7ZLLMK4DJB5Q, seg_pos={0 0}, upload=472552]: Uploading piece... Dec 28 02:19:02.436105054 (538h50m2.307188099s) [noise=true , address=145.79.227.14:4842 , piece_id=3F6EV4TB5TQNK5W5CJHRCCNORCEEAA6AXCOJQTQ6WJNRZG3YILOQ, seg_pos={0 0}, upload=472552]: Uploading piece... Dec 28 02:19:02.436528130 (538h50m2.307611175s) [noise=true , address=77.90.31.12:4610 , piece_id=C7IRMHAHNXDZLKFBL7OVTNVI2KWMFREMFE5GR4USVMNT5QRB3HXQ, seg_pos={0 0}, upload=472552]: Uploading piece... Dec 28 02:19:02.437497977 (538h50m2.308581022s) [noise=true , address=145.79.227.49:4340 , piece_id=5C75E4QGOX6G7QNOUUKX4GC7YL5MESWBOFSNCA6PO4PPSMQKIVFQ, seg_pos={0 0}, upload=472552]: Uploading piece... Dec 28 02:19:02.439092573 (538h50m2.310175618s) [noise=true , address=45.91.116.11:4134 , piece_id=D4PXHXGBZGH3UU2CNKDRCWD44IJQR5BQFWHCNQRPZDNYVV755ICQ, seg_pos={0 0}, upload=472552]: Uploading piece... Dec 28 02:19:02.440580235 (538h50m2.31166328s) [noise=true , address=45.91.116.5:4318 , piece_id=ULPCWUUHAL35F5WZYHDQLO7VBIGAUH2XHH4I2NYYHG7OAC5VJFOA, seg_pos={0 0}, upload=472552]: Uploading piece... Dec 28 02:19:02.442091032 (538h50m2.313174077s) [noise=true , address=77.90.31.11:4558 , piece_id=TWOTPEDKIRIOR7NQNCEFM3Z5KGWVOFUYKIQYMEGHED5FKRDCB54Q, seg_pos={0 0}, upload=472552]: Uploading piece... Dec 28 02:19:02.443305184 (538h50m2.314388229s) [noise=true , address=145.79.227.40:4078 , piece_id=3ICUQQR7NS6O2QYHBFCAQZS7A7UAWHTD3O2LOWLRZ3OCMU3WWBSA, seg_pos={0 0}, upload=472552]: Uploading piece... Dec 28 02:19:02.447829959 (538h50m2.318913014s) [noise=true , address=145.79.227.14:4863 , piece_id=N3XS43U6I4WJYUCC5NVUBRLTTHVPYKDR3KRCNGJRYU3E26WDO3HA, seg_pos={0 0}, upload=472552]: Uploading piece... Dec 28 02:19:02.451586160 (538h50m2.322669225s) [noise=true , address=45.91.116.9:4288 , piece_id=EMRORAB74LXEB6222NUCRM5ZENJC2AZNK7Q2V2NSQMVD5TIRGVGQ, seg_pos={0 0}, upload=472552]: Uploading piece... Dec 28 02:19:03.923272933 (538h50m3.794355978s) [upload=472552]: Next returned nil segment Dec 28 02:19:03.923289645 (538h50m3.79437269s) [upload=472552]: Waiting for error group managing segments... Dec 28 02:19:04.324829202 (538h50m4.195912237s) [noise=true , address=145.79.227.49:4340 , piece_id=5C75E4QGOX6G7QNOUUKX4GC7YL5MESWBOFSNCA6PO4PPSMQKIVFQ, seg_pos={0 0}, upload=472552]: Done uploading piece. err: <nil> Dec 28 02:19:04.335200317 (538h50m4.206283363s) [noise=true , address=145.79.227.14:4842 , piece_id=3F6EV4TB5TQNK5W5CJHRCCNORCEEAA6AXCOJQTQ6WJNRZG3YILOQ, seg_pos={0 0}, upload=472552]: Done uploading piece. err: <nil> Dec 28 02:19:04.343899841 (538h50m4.214982886s) [noise=true , address=145.79.227.14:4863 , piece_id=N3XS43U6I4WJYUCC5NVUBRLTTHVPYKDR3KRCNGJRYU3E26WDO3HA, seg_pos={0 0}, upload=472552]: Done uploading piece. err: <nil> Dec 28 02:19:04.378111328 (538h50m4.249194373s) [noise=true , address=145.79.227.37:4596 , piece_id=XBEKEUWGAAAWZGOV3PBH7NE2CX5VID75MFGWMPHTNR7MV66U3XNA, seg_pos={0 0}, upload=472552]: Done uploading piece. err: <nil> Dec 28 02:19:04.381135876 (538h50m4.252218921s) [noise=true , address=145.79.227.32:4856 , piece_id=II5XB6TRHNQAAHBPP5PRMMA7BCBGR5QO4HUBL4YJ7ZLLMK4DJB5Q, seg_pos={0 0}, upload=472552]: Done uploading piece. err: <nil> Dec 28 02:19:04.476390445 (538h50m4.34747348s) [noise=true , address=145.79.227.6:4290 , piece_id=K4OGXS2MXEFVQVW7HEAZJ6267HWETOUXWJAC77JNZADW76K2XMJA, seg_pos={0 0}, upload=472552]: Done uploading piece. err: <nil> Dec 28 02:19:04.533625848 (538h50m4.404708893s) [noise=true , address=145.79.227.33:4896 , piece_id=D44WQB3G3NBNSB6ITIBHVV67V2EKPGXXONQU25IE6RVSGGTUJ44Q, seg_pos={0 0}, upload=472552]: Done uploading piece. err: <nil> Dec 28 02:19:04.562532124 (538h50m4.433615159s) [noise=true , address=77.90.31.11:4558 , piece_id=TWOTPEDKIRIOR7NQNCEFM3Z5KGWVOFUYKIQYMEGHED5FKRDCB54Q, seg_pos={0 0}, upload=472552]: Done uploading piece. err: <nil> Dec 28 02:19:04.643414509 (538h50m4.514497544s) [noise=true , address=77.90.31.21:4900 , piece_id=UEKG3FOTTORJVQRPLAFCJEP2ZAVA5N7M2DFGH4BUTB2OGVC2KIZA, seg_pos={0 0}, upload=472552]: Done uploading piece. err: <nil> Dec 28 02:19:04.651013327 (538h50m4.522096372s) [noise=true , address=77.90.31.13:4646 , piece_id=V5HTB4HVJVO7OG745JBRQSISFZU6UOORKLHEVDUDUR24EWRSSVOQ, seg_pos={0 0}, upload=472552]: Done uploading piece. err: <nil> Dec 28 02:19:04.776667164 (538h50m4.647750209s) [noise=true , address=45.91.116.11:4136 , piece_id=JD6QHJCQ6D53BGKDPSPYBBHNCP7XDZCSV7H6ARD3JNBL75NULT6Q, seg_pos={0 0}, upload=472552]: Done uploading piece. err: <nil> Dec 28 02:19:04.919205529 (538h50m4.790288574s) [noise=true , address=45.91.116.11:4134 , piece_id=D4PXHXGBZGH3UU2CNKDRCWD44IJQR5BQFWHCNQRPZDNYVV755ICQ, seg_pos={0 0}, upload=472552]: Done uploading piece. err: <nil> Dec 28 02:19:04.927993490 (538h50m4.799076535s) [noise=true , address=45.91.116.9:4288 , piece_id=EMRORAB74LXEB6222NUCRM5ZENJC2AZNK7Q2V2NSQMVD5TIRGVGQ, seg_pos={0 0}, upload=472552]: Done uploading piece. err: <nil> Dec 28 02:19:05.041729347 (538h50m4.912812392s) [noise=true , address=45.91.116.5:4318 , piece_id=ULPCWUUHAL35F5WZYHDQLO7VBIGAUH2XHH4I2NYYHG7OAC5VJFOA, seg_pos={0 0}, upload=472552]: Done uploading piece. err: <nil> Dec 28 02:19:05.041751558 (538h50m4.912834593s) [seg_pos={0 0}, upload=472552]: Segment reached optimal threshold of 14 pieces. Dec 28 02:19:05.041986606 (538h50m4.913069651s) [noise=true , address=77.90.31.12:4610 , piece_id=C7IRMHAHNXDZLKFBL7OVTNVI2KWMFREMFE5GR4USVMNT5QRB3HXQ, seg_pos={0 0}, upload=472552]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi): protocol: expected piece hash; context canceled; EOF Dec 28 02:19:05.041995954 (538h50m4.913078999s) [noise=true , address=77.90.31.12:4610 , piece_id=C7IRMHAHNXDZLKFBL7OVTNVI2KWMFREMFE5GR4USVMNT5QRB3HXQ, seg_pos={0 0}, upload=472552]: Long tail cancellation due to optimal threshold Dec 28 02:19:05.042149837 (538h50m4.913232872s) [noise=true , address=77.90.31.15:4703 , piece_id=BERLD7G2DBBDBNE3MCS57VXAKXL3P4RQIV47ZOCU6VXMPJU4QITA, seg_pos={0 0}, upload=472552]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hMDD5K7WDjvN4ZgcpwpnPFb6X1ncCfyucfFrpTUshd8Ye1B6D): protocol: expected piece hash; context canceled; EOF Dec 28 02:19:05.042158714 (538h50m4.913241759s) [noise=true , address=77.90.31.15:4703 , piece_id=BERLD7G2DBBDBNE3MCS57VXAKXL3P4RQIV47ZOCU6VXMPJU4QITA, seg_pos={0 0}, upload=472552]: Long tail cancellation due to optimal threshold Dec 28 02:19:05.042299752 (538h50m4.913382788s) [noise=true , address=77.90.31.16:4741 , piece_id=CX66NBDSLZWQ34Z2UDYM6HM73TABJSVHSGBQRONAQ3S6ENVTY3KQ, seg_pos={0 0}, upload=472552]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR): protocol: expected piece hash; context canceled; EOF Dec 28 02:19:05.042304702 (538h50m4.913387737s) [noise=true , address=77.90.31.16:4741 , piece_id=CX66NBDSLZWQ34Z2UDYM6HM73TABJSVHSGBQRONAQ3S6ENVTY3KQ, seg_pos={0 0}, upload=472552]: Long tail cancellation due to optimal threshold Dec 28 02:19:05.042396276 (538h50m4.913479352s) [noise=true , address=145.79.227.40:4078 , piece_id=3ICUQQR7NS6O2QYHBFCAQZS7A7UAWHTD3O2LOWLRZ3OCMU3WWBSA, seg_pos={0 0}, upload=472552]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tzgtenV6us5ssD7xWGpZmVUoWEfbFhBNewbK9irHtZPkyDtRd): protocol: expected piece hash; context canceled; EOF Dec 28 02:19:05.042417406 (538h50m4.913500461s) [noise=true , address=145.79.227.40:4078 , piece_id=3ICUQQR7NS6O2QYHBFCAQZS7A7UAWHTD3O2LOWLRZ3OCMU3WWBSA, seg_pos={0 0}, upload=472552]: Long tail cancellation due to optimal threshold Dec 28 02:19:05.042612779 (538h50m4.913695814s) [seg_pos={0 0}, upload=472552]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 02:19:05.042821726 (538h50m4.913904771s) [upload=472552]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 02:19:05.042916818 (538h50m4.913999863s) [upload=472552]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 02:19:05.336925913 (538h50m5.208008958s) [upload=472552]: Done uploading segments... Dec 28 02:19:05.336953856 (538h50m5.208036901s) [upload=472552]: Upload finished. err: <nil> Duration: 50.862002866s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................................X......................] duration: 3.642691024s trace: https://jaeger.dev.storj.io/trace/7fc142ad3d711e34 FAIL (2025-12-28 02:23:18 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.642715069s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 31 out of 60 is failed (threshold=3) [XX..X.XXX.X.XX.XX.X..X.X.X.X.X..X...XX..X.XXX.XXX.X.....XXX.] duration: 8.518671828s trace: https://jaeger.dev.storj.io/trace/585bc3668fb646a FAIL (2025-12-28 02:23:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2:22.361988511 (538h53m22.233071556s) [upload=472599]: Starting upload Dec 28 02:22:22.362084634 (538h53m22.233167679s) [upload=472599]: Done starting upload Dec 28 02:22:22.362190385 (538h53m22.23327345s) [upload=472599]: Uploading segments... Dec 28 02:22:22.362403721 (538h53m22.233486767s) [upload=472599]: Got next segment. Inline: false Dec 28 02:22:22.362449959 (538h53m22.233533014s) [upload=472599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 02:22:22.362456692 (538h53m22.233539747s) [upload=472599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 02:22:22.607170793 (538h53m22.478253828s) [seg_pos={0 0}, upload=472599]: Begin upload segment... Dec 28 02:22:22.607215508 (538h53m22.478298553s) [seg_pos={0 0}, upload=472599]: Done begin upload segment. Dec 28 02:22:22.607253891 (538h53m22.478336936s) [noise=true , address=45.91.116.9:4261 , piece_id=AG3QWJKZKGKG75SSTKXPB6GIJ5DEWNVQ72UTCQ6NJQEJA2X54VHQ, seg_pos={0 0}, upload=472599]: Uploading piece... Dec 28 02:22:22.607288507 (538h53m22.478371612s) [noise=true , address=145.79.227.45:4229 , piece_id=7NASM7KG3RPFKRBY2623QYLWMXDVGSL2HSIKHRGGSJHJUKT7XWIQ, seg_pos={0 0}, upload=472599]: Uploading piece... Dec 28 02:22:22.607963743 (538h53m22.479046788s) [noise=true , address=77.90.31.13:4626 , piece_id=KBNYZ4KGNXQC72GQFVZ2QHZI2LQFIOWPHXXVLAEELVLMB7UWR7UQ, seg_pos={0 0}, upload=472599]: Uploading piece... Dec 28 02:22:22.608270116 (538h53m22.479353151s) [noise=true , address=45.91.116.14:4192 , piece_id=3HQITFQ5LNNNHPJFUW64G6BDBKVSBMV7K3IFTPEPT6WD2KTL7Z7Q, seg_pos={0 0}, upload=472599]: Uploading piece... Dec 28 02:22:22.608544959 (538h53m22.479628004s) [noise=true , address=45.91.116.5:4299 , piece_id=VVD6MOMC2MTD6QKVOJWRLEZM7QIQOIUVFBC4RMZFWYUDFXQDZD3Q, seg_pos={0 0}, upload=472599]: Uploading piece... Dec 28 02:22:22.608849109 (538h53m22.479932144s) [noise=true , address=145.79.227.47:4265 , piece_id=EB577QZCBLJVQPMXDQU6IQJRPVFZPFYMOQSFHABQ5E4VC56MVUOQ, seg_pos={0 0}, upload=472599]: Uploading piece... Dec 28 02:22:22.609127970 (538h53m22.480211015s) [noise=true , address=45.91.116.13:4156 , piece_id=3OMW7OEVL2RE6ZZGEQCCOZXJ7TG72AJPG3HVXFHO4R4WPQ5BAHQA, seg_pos={0 0}, upload=472599]: Uploading piece... Dec 28 02:22:22.609416008 (538h53m22.480499053s) [noise=true , address=145.79.227.48:4302 , piece_id=UQQ5HNPRA4LPZXIE6LECTCZBG7GILULH4IU3SGEGF5KZUD4FKBNA, seg_pos={0 0}, upload=472599]: Uploading piece... Dec 28 02:22:22.609955394 (538h53m22.481038429s) [noise=true , address=145.79.227.41:4085 , piece_id=Z5A2ZRBECXB4DK7XVYFAIQRUK3SOT6TLGVGPBKOMDWEADNEYDNTQ, seg_pos={0 0}, upload=472599]: Uploading piece... Dec 28 02:22:22.611717640 (538h53m22.482800675s) [noise=true , address=145.79.227.2:4114 , piece_id=S6IAHYA7FDEVXQX3OAA26U2JBNHLXEQOPFH55OGKEN45IDEU2JPA, seg_pos={0 0}, upload=472599]: Uploading piece... Dec 28 02:22:22.611944041 (538h53m22.483027086s) [noise=true , address=77.90.31.12:4590 , piece_id=OIRUX7W7KPAHKEPI6DOARLN6CZQZWRCVFNRZXLUQ46TD4NEOTXCQ, seg_pos={0 0}, upload=472599]: Uploading piece... Dec 28 02:22:22.613114168 (538h53m22.484197214s) [noise=true , address=145.79.227.27:4691 , piece_id=WV7TK72ECL63NAF7ZEV6CWRPRL7H3432CPVLDCBS3ISPBIWGFXNQ, seg_pos={0 0}, upload=472599]: Uploading piece... Dec 28 02:22:22.614569189 (538h53m22.485652234s) [noise=true , address=145.79.227.39:4014 , piece_id=CMG724TG5S7TMP6Y5YHICAFRLW3UOZOTK6OIN3W6YU4ES5KIQZXA, seg_pos={0 0}, upload=472599]: Uploading piece... Dec 28 02:22:22.616242084 (538h53m22.487325119s) [noise=true , address=145.79.227.4:4185 , piece_id=653IBL7FKUVSTRSZKBCOZ4Q3IKSTBVFFJ27ZG3SUGN76RDQ5UE4A, seg_pos={0 0}, upload=472599]: Uploading piece... Dec 28 02:22:22.617769502 (538h53m22.488852537s) [noise=true , address=145.79.227.37:4567 , piece_id=ROZG3NVNSZ6QA6L7OYEZE2BHUI5QNRLWSDVHV7DJYXRVBY4725HA, seg_pos={0 0}, upload=472599]: Uploading piece... Dec 28 02:22:22.619046834 (538h53m22.490129879s) [noise=true , address=145.79.227.46:4602 , piece_id=2BIOA327IUM2NKIKIVMMUZQ4NPT4FNRWQ3J3V4QBSW6DBTLY26BA, seg_pos={0 0}, upload=472599]: Uploading piece... Dec 28 02:22:24.323983119 (538h53m24.195066154s) [upload=472599]: Next returned nil segment Dec 28 02:22:24.324008929 (538h53m24.195091984s) [upload=472599]: Waiting for error group managing segments... Dec 28 02:22:24.629082611 (538h53m24.500165646s) [noise=true , address=145.79.227.27:4691 , piece_id=WV7TK72ECL63NAF7ZEV6CWRPRL7H3432CPVLDCBS3ISPBIWGFXNQ, seg_pos={0 0}, upload=472599]: Done uploading piece. err: <nil> Dec 28 02:22:24.635804368 (538h53m24.506887403s) [noise=true , address=145.79.227.4:4185 , piece_id=653IBL7FKUVSTRSZKBCOZ4Q3IKSTBVFFJ27ZG3SUGN76RDQ5UE4A, seg_pos={0 0}, upload=472599]: Done uploading piece. err: <nil> Dec 28 02:22:24.642160340 (538h53m24.513243385s) [noise=true , address=145.79.227.46:4602 , piece_id=2BIOA327IUM2NKIKIVMMUZQ4NPT4FNRWQ3J3V4QBSW6DBTLY26BA, seg_pos={0 0}, upload=472599]: Done uploading piece. err: <nil> Dec 28 02:22:24.643438123 (538h53m24.514521158s) [noise=true , address=145.79.227.39:4014 , piece_id=CMG724TG5S7TMP6Y5YHICAFRLW3UOZOTK6OIN3W6YU4ES5KIQZXA, seg_pos={0 0}, upload=472599]: Done uploading piece. err: <nil> Dec 28 02:22:24.657099102 (538h53m24.528182138s) [noise=true , address=145.79.227.2:4114 , piece_id=S6IAHYA7FDEVXQX3OAA26U2JBNHLXEQOPFH55OGKEN45IDEU2JPA, seg_pos={0 0}, upload=472599]: Done uploading piece. err: <nil> Dec 28 02:22:24.665599446 (538h53m24.536682481s) [noise=true , address=145.79.227.41:4085 , piece_id=Z5A2ZRBECXB4DK7XVYFAIQRUK3SOT6TLGVGPBKOMDWEADNEYDNTQ, seg_pos={0 0}, upload=472599]: Done uploading piece. err: <nil> Dec 28 02:22:24.727896035 (538h53m24.59897908s) [noise=true , address=145.79.227.47:4265 , piece_id=EB577QZCBLJVQPMXDQU6IQJRPVFZPFYMOQSFHABQ5E4VC56MVUOQ, seg_pos={0 0}, upload=472599]: Done uploading piece. err: <nil> Dec 28 02:22:24.854262951 (538h53m24.725345986s) [noise=true , address=45.91.116.9:4261 , piece_id=AG3QWJKZKGKG75SSTKXPB6GIJ5DEWNVQ72UTCQ6NJQEJA2X54VHQ, seg_pos={0 0}, upload=472599]: Done uploading piece. err: <nil> Dec 28 02:22:24.992124691 (538h53m24.863207736s) [noise=true , address=145.79.227.37:4567 , piece_id=ROZG3NVNSZ6QA6L7OYEZE2BHUI5QNRLWSDVHV7DJYXRVBY4725HA, seg_pos={0 0}, upload=472599]: Done uploading piece. err: <nil> Dec 28 02:22:25.010569950 (538h53m24.881652995s) [noise=true , address=45.91.116.14:4192 , piece_id=3HQITFQ5LNNNHPJFUW64G6BDBKVSBMV7K3IFTPEPT6WD2KTL7Z7Q, seg_pos={0 0}, upload=472599]: Done uploading piece. err: <nil> Dec 28 02:22:25.348627070 (538h53m25.219710105s) [noise=true , address=145.79.227.45:4229 , piece_id=7NASM7KG3RPFKRBY2623QYLWMXDVGSL2HSIKHRGGSJHJUKT7XWIQ, seg_pos={0 0}, upload=472599]: Done uploading piece. err: <nil> Dec 28 02:22:25.491610973 (538h53m25.362694008s) [noise=true , address=45.91.116.13:4156 , piece_id=3OMW7OEVL2RE6ZZGEQCCOZXJ7TG72AJPG3HVXFHO4R4WPQ5BAHQA, seg_pos={0 0}, upload=472599]: Done uploading piece. err: <nil> Dec 28 02:22:25.570441721 (538h53m25.441524757s) [noise=true , address=77.90.31.12:4590 , piece_id=OIRUX7W7KPAHKEPI6DOARLN6CZQZWRCVFNRZXLUQ46TD4NEOTXCQ, seg_pos={0 0}, upload=472599]: Done uploading piece. err: <nil> Dec 28 02:22:25.570462441 (538h53m25.441545486s) [seg_pos={0 0}, upload=472599]: Segment reached optimal threshold of 13 pieces. Dec 28 02:22:25.570687540 (538h53m25.441770585s) [noise=true , address=45.91.116.5:4299 , piece_id=VVD6MOMC2MTD6QKVOJWRLEZM7QIQOIUVFBC4RMZFWYUDFXQDZD3Q, seg_pos={0 0}, upload=472599]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Dec 28 02:22:25.570693972 (538h53m25.441777017s) [noise=true , address=45.91.116.5:4299 , piece_id=VVD6MOMC2MTD6QKVOJWRLEZM7QIQOIUVFBC4RMZFWYUDFXQDZD3Q, seg_pos={0 0}, upload=472599]: Long tail cancellation due to optimal threshold Dec 28 02:22:25.571024812 (538h53m25.442107847s) [noise=true , address=77.90.31.13:4626 , piece_id=KBNYZ4KGNXQC72GQFVZ2QHZI2LQFIOWPHXXVLAEELVLMB7UWR7UQ, seg_pos={0 0}, upload=472599]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF Dec 28 02:22:25.571032527 (538h53m25.442115572s) [noise=true , address=77.90.31.13:4626 , piece_id=KBNYZ4KGNXQC72GQFVZ2QHZI2LQFIOWPHXXVLAEELVLMB7UWR7UQ, seg_pos={0 0}, upload=472599]: Long tail cancellation due to optimal threshold Dec 28 02:22:25.571192200 (538h53m25.442275235s) [noise=true , address=145.79.227.48:4302 , piece_id=UQQ5HNPRA4LPZXIE6LECTCZBG7GILULH4IU3SGEGF5KZUD4FKBNA, seg_pos={0 0}, upload=472599]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tGpuYdPLiy4ncDjxrZbQfp7Bq79NVBHDmr4vjaL4qMpudrvgR): protocol: write tcp 188.166.225.170:50138->145.79.227.48:4302: use of closed network connection; write tcp 188.166.225.170:50138->145.79.227.48:4302: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50138->145.79.227.48:4302: use of closed network connection Dec 28 02:22:25.571199925 (538h53m25.442282971s) [noise=true , address=145.79.227.48:4302 , piece_id=UQQ5HNPRA4LPZXIE6LECTCZBG7GILULH4IU3SGEGF5KZUD4FKBNA, seg_pos={0 0}, upload=472599]: Long tail cancellation due to optimal threshold Dec 28 02:22:25.571218450 (538h53m25.442301495s) [seg_pos={0 0}, upload=472599]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 02:22:25.571296919 (538h53m25.442379964s) [upload=472599]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 02:22:25.571302249 (538h53m25.442385284s) [upload=472599]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 02:22:25.852548115 (538h53m25.72363116s) [upload=472599]: Done uploading segments... Dec 28 02:22:25.852569416 (538h53m25.723652461s) [upload=472599]: Upload finished. err: <nil> Duration: 8.51870408s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 32 out of 60 is failed (threshold=3) [XXX..X.XXX.X.XX.XX.X..X.X.X.X.X..X...XX..X.XXX.XXX.X.....XXX] duration: 30.003276125s trace: https://jaeger.dev.storj.io/trace/585bc3668fb646a FAIL (2025-12-28 02:22:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2:22.361988511 (538h53m22.233071556s) [upload=472599]: Starting upload Dec 28 02:22:22.362084634 (538h53m22.233167679s) [upload=472599]: Done starting upload Dec 28 02:22:22.362190385 (538h53m22.23327345s) [upload=472599]: Uploading segments... Dec 28 02:22:22.362403721 (538h53m22.233486767s) [upload=472599]: Got next segment. Inline: false Dec 28 02:22:22.362449959 (538h53m22.233533014s) [upload=472599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 02:22:22.362456692 (538h53m22.233539747s) [upload=472599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 02:22:22.607170793 (538h53m22.478253828s) [seg_pos={0 0}, upload=472599]: Begin upload segment... Dec 28 02:22:22.607215508 (538h53m22.478298553s) [seg_pos={0 0}, upload=472599]: Done begin upload segment. Dec 28 02:22:22.607253891 (538h53m22.478336936s) [noise=true , address=45.91.116.9:4261 , piece_id=AG3QWJKZKGKG75SSTKXPB6GIJ5DEWNVQ72UTCQ6NJQEJA2X54VHQ, seg_pos={0 0}, upload=472599]: Uploading piece... Dec 28 02:22:22.607288507 (538h53m22.478371612s) [noise=true , address=145.79.227.45:4229 , piece_id=7NASM7KG3RPFKRBY2623QYLWMXDVGSL2HSIKHRGGSJHJUKT7XWIQ, seg_pos={0 0}, upload=472599]: Uploading piece... Dec 28 02:22:22.607963743 (538h53m22.479046788s) [noise=true , address=77.90.31.13:4626 , piece_id=KBNYZ4KGNXQC72GQFVZ2QHZI2LQFIOWPHXXVLAEELVLMB7UWR7UQ, seg_pos={0 0}, upload=472599]: Uploading piece... Dec 28 02:22:22.608270116 (538h53m22.479353151s) [noise=true , address=45.91.116.14:4192 , piece_id=3HQITFQ5LNNNHPJFUW64G6BDBKVSBMV7K3IFTPEPT6WD2KTL7Z7Q, seg_pos={0 0}, upload=472599]: Uploading piece... Dec 28 02:22:22.608544959 (538h53m22.479628004s) [noise=true , address=45.91.116.5:4299 , piece_id=VVD6MOMC2MTD6QKVOJWRLEZM7QIQOIUVFBC4RMZFWYUDFXQDZD3Q, seg_pos={0 0}, upload=472599]: Uploading piece... Dec 28 02:22:22.608849109 (538h53m22.479932144s) [noise=true , address=145.79.227.47:4265 , piece_id=EB577QZCBLJVQPMXDQU6IQJRPVFZPFYMOQSFHABQ5E4VC56MVUOQ, seg_pos={0 0}, upload=472599]: Uploading piece... Dec 28 02:22:22.609127970 (538h53m22.480211015s) [noise=true , address=45.91.116.13:4156 , piece_id=3OMW7OEVL2RE6ZZGEQCCOZXJ7TG72AJPG3HVXFHO4R4WPQ5BAHQA, seg_pos={0 0}, upload=472599]: Uploading piece... Dec 28 02:22:22.609416008 (538h53m22.480499053s) [noise=true , address=145.79.227.48:4302 , piece_id=UQQ5HNPRA4LPZXIE6LECTCZBG7GILULH4IU3SGEGF5KZUD4FKBNA, seg_pos={0 0}, upload=472599]: Uploading piece... Dec 28 02:22:22.609955394 (538h53m22.481038429s) [noise=true , address=145.79.227.41:4085 , piece_id=Z5A2ZRBECXB4DK7XVYFAIQRUK3SOT6TLGVGPBKOMDWEADNEYDNTQ, seg_pos={0 0}, upload=472599]: Uploading piece... Dec 28 02:22:22.611717640 (538h53m22.482800675s) [noise=true , address=145.79.227.2:4114 , piece_id=S6IAHYA7FDEVXQX3OAA26U2JBNHLXEQOPFH55OGKEN45IDEU2JPA, seg_pos={0 0}, upload=472599]: Uploading piece... Dec 28 02:22:22.611944041 (538h53m22.483027086s) [noise=true , address=77.90.31.12:4590 , piece_id=OIRUX7W7KPAHKEPI6DOARLN6CZQZWRCVFNRZXLUQ46TD4NEOTXCQ, seg_pos={0 0}, upload=472599]: Uploading piece... Dec 28 02:22:22.613114168 (538h53m22.484197214s) [noise=true , address=145.79.227.27:4691 , piece_id=WV7TK72ECL63NAF7ZEV6CWRPRL7H3432CPVLDCBS3ISPBIWGFXNQ, seg_pos={0 0}, upload=472599]: Uploading piece... Dec 28 02:22:22.614569189 (538h53m22.485652234s) [noise=true , address=145.79.227.39:4014 , piece_id=CMG724TG5S7TMP6Y5YHICAFRLW3UOZOTK6OIN3W6YU4ES5KIQZXA, seg_pos={0 0}, upload=472599]: Uploading piece... Dec 28 02:22:22.616242084 (538h53m22.487325119s) [noise=true , address=145.79.227.4:4185 , piece_id=653IBL7FKUVSTRSZKBCOZ4Q3IKSTBVFFJ27ZG3SUGN76RDQ5UE4A, seg_pos={0 0}, upload=472599]: Uploading piece... Dec 28 02:22:22.617769502 (538h53m22.488852537s) [noise=true , address=145.79.227.37:4567 , piece_id=ROZG3NVNSZ6QA6L7OYEZE2BHUI5QNRLWSDVHV7DJYXRVBY4725HA, seg_pos={0 0}, upload=472599]: Uploading piece... Dec 28 02:22:22.619046834 (538h53m22.490129879s) [noise=true , address=145.79.227.46:4602 , piece_id=2BIOA327IUM2NKIKIVMMUZQ4NPT4FNRWQ3J3V4QBSW6DBTLY26BA, seg_pos={0 0}, upload=472599]: Uploading piece... Dec 28 02:22:24.323983119 (538h53m24.195066154s) [upload=472599]: Next returned nil segment Dec 28 02:22:24.324008929 (538h53m24.195091984s) [upload=472599]: Waiting for error group managing segments... Dec 28 02:22:24.629082611 (538h53m24.500165646s) [noise=true , address=145.79.227.27:4691 , piece_id=WV7TK72ECL63NAF7ZEV6CWRPRL7H3432CPVLDCBS3ISPBIWGFXNQ, seg_pos={0 0}, upload=472599]: Done uploading piece. err: <nil> Dec 28 02:22:24.635804368 (538h53m24.506887403s) [noise=true , address=145.79.227.4:4185 , piece_id=653IBL7FKUVSTRSZKBCOZ4Q3IKSTBVFFJ27ZG3SUGN76RDQ5UE4A, seg_pos={0 0}, upload=472599]: Done uploading piece. err: <nil> Dec 28 02:22:24.642160340 (538h53m24.513243385s) [noise=true , address=145.79.227.46:4602 , piece_id=2BIOA327IUM2NKIKIVMMUZQ4NPT4FNRWQ3J3V4QBSW6DBTLY26BA, seg_pos={0 0}, upload=472599]: Done uploading piece. err: <nil> Dec 28 02:22:24.643438123 (538h53m24.514521158s) [noise=true , address=145.79.227.39:4014 , piece_id=CMG724TG5S7TMP6Y5YHICAFRLW3UOZOTK6OIN3W6YU4ES5KIQZXA, seg_pos={0 0}, upload=472599]: Done uploading piece. err: <nil> Dec 28 02:22:24.657099102 (538h53m24.528182138s) [noise=true , address=145.79.227.2:4114 , piece_id=S6IAHYA7FDEVXQX3OAA26U2JBNHLXEQOPFH55OGKEN45IDEU2JPA, seg_pos={0 0}, upload=472599]: Done uploading piece. err: <nil> Dec 28 02:22:24.665599446 (538h53m24.536682481s) [noise=true , address=145.79.227.41:4085 , piece_id=Z5A2ZRBECXB4DK7XVYFAIQRUK3SOT6TLGVGPBKOMDWEADNEYDNTQ, seg_pos={0 0}, upload=472599]: Done uploading piece. err: <nil> Dec 28 02:22:24.727896035 (538h53m24.59897908s) [noise=true , address=145.79.227.47:4265 , piece_id=EB577QZCBLJVQPMXDQU6IQJRPVFZPFYMOQSFHABQ5E4VC56MVUOQ, seg_pos={0 0}, upload=472599]: Done uploading piece. err: <nil> Dec 28 02:22:24.854262951 (538h53m24.725345986s) [noise=true , address=45.91.116.9:4261 , piece_id=AG3QWJKZKGKG75SSTKXPB6GIJ5DEWNVQ72UTCQ6NJQEJA2X54VHQ, seg_pos={0 0}, upload=472599]: Done uploading piece. err: <nil> Dec 28 02:22:24.992124691 (538h53m24.863207736s) [noise=true , address=145.79.227.37:4567 , piece_id=ROZG3NVNSZ6QA6L7OYEZE2BHUI5QNRLWSDVHV7DJYXRVBY4725HA, seg_pos={0 0}, upload=472599]: Done uploading piece. err: <nil> Dec 28 02:22:25.010569950 (538h53m24.881652995s) [noise=true , address=45.91.116.14:4192 , piece_id=3HQITFQ5LNNNHPJFUW64G6BDBKVSBMV7K3IFTPEPT6WD2KTL7Z7Q, seg_pos={0 0}, upload=472599]: Done uploading piece. err: <nil> Dec 28 02:22:25.348627070 (538h53m25.219710105s) [noise=true , address=145.79.227.45:4229 , piece_id=7NASM7KG3RPFKRBY2623QYLWMXDVGSL2HSIKHRGGSJHJUKT7XWIQ, seg_pos={0 0}, upload=472599]: Done uploading piece. err: <nil> Dec 28 02:22:25.491610973 (538h53m25.362694008s) [noise=true , address=45.91.116.13:4156 , piece_id=3OMW7OEVL2RE6ZZGEQCCOZXJ7TG72AJPG3HVXFHO4R4WPQ5BAHQA, seg_pos={0 0}, upload=472599]: Done uploading piece. err: <nil> Dec 28 02:22:25.570441721 (538h53m25.441524757s) [noise=true , address=77.90.31.12:4590 , piece_id=OIRUX7W7KPAHKEPI6DOARLN6CZQZWRCVFNRZXLUQ46TD4NEOTXCQ, seg_pos={0 0}, upload=472599]: Done uploading piece. err: <nil> Dec 28 02:22:25.570462441 (538h53m25.441545486s) [seg_pos={0 0}, upload=472599]: Segment reached optimal threshold of 13 pieces. Dec 28 02:22:25.570687540 (538h53m25.441770585s) [noise=true , address=45.91.116.5:4299 , piece_id=VVD6MOMC2MTD6QKVOJWRLEZM7QIQOIUVFBC4RMZFWYUDFXQDZD3Q, seg_pos={0 0}, upload=472599]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Dec 28 02:22:25.570693972 (538h53m25.441777017s) [noise=true , address=45.91.116.5:4299 , piece_id=VVD6MOMC2MTD6QKVOJWRLEZM7QIQOIUVFBC4RMZFWYUDFXQDZD3Q, seg_pos={0 0}, upload=472599]: Long tail cancellation due to optimal threshold Dec 28 02:22:25.571024812 (538h53m25.442107847s) [noise=true , address=77.90.31.13:4626 , piece_id=KBNYZ4KGNXQC72GQFVZ2QHZI2LQFIOWPHXXVLAEELVLMB7UWR7UQ, seg_pos={0 0}, upload=472599]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF Dec 28 02:22:25.571032527 (538h53m25.442115572s) [noise=true , address=77.90.31.13:4626 , piece_id=KBNYZ4KGNXQC72GQFVZ2QHZI2LQFIOWPHXXVLAEELVLMB7UWR7UQ, seg_pos={0 0}, upload=472599]: Long tail cancellation due to optimal threshold Dec 28 02:22:25.571192200 (538h53m25.442275235s) [noise=true , address=145.79.227.48:4302 , piece_id=UQQ5HNPRA4LPZXIE6LECTCZBG7GILULH4IU3SGEGF5KZUD4FKBNA, seg_pos={0 0}, upload=472599]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tGpuYdPLiy4ncDjxrZbQfp7Bq79NVBHDmr4vjaL4qMpudrvgR): protocol: write tcp 188.166.225.170:50138->145.79.227.48:4302: use of closed network connection; write tcp 188.166.225.170:50138->145.79.227.48:4302: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50138->145.79.227.48:4302: use of closed network connection Dec 28 02:22:25.571199925 (538h53m25.442282971s) [noise=true , address=145.79.227.48:4302 , piece_id=UQQ5HNPRA4LPZXIE6LECTCZBG7GILULH4IU3SGEGF5KZUD4FKBNA, seg_pos={0 0}, upload=472599]: Long tail cancellation due to optimal threshold Dec 28 02:22:25.571218450 (538h53m25.442301495s) [seg_pos={0 0}, upload=472599]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 02:22:25.571296919 (538h53m25.442379964s) [upload=472599]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 02:22:25.571302249 (538h53m25.442385284s) [upload=472599]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 02:22:25.852548115 (538h53m25.72363116s) [upload=472599]: Done uploading segments... Dec 28 02:22:25.852569416 (538h53m25.723652461s) [upload=472599]: Upload finished. err: <nil> Duration: 30.003298868s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................................X.....................] duration: 5.001358235s trace: https://jaeger.dev.storj.io/trace/7fc142ad3d711e34 FAIL (2025-12-28 02:22:50 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001384936s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................X...................X...] duration: 49.778227692s trace: https://jaeger.dev.storj.io/trace/1f68404b41391e4a FAIL (2025-12-28 02:22:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12CaBSQgwTru9NXaznMJ2XExKe3yhLoUCvDKFvqvLJGJev4mEPc, Piece ID: ULPCWUUHAL35F5WZYHDQLO7VBIGAUH2XHH4I2NYYHG7OAC5VJFOA) context deadline exceeded; piecestore: (Node ID: 1H1XywjaRt5ZRGqmZBgzfSowuCnPqFesL1Ch2jYYS1gV2oBgQX, Piece ID: EMRORAB74LXEB6222NUCRM5ZENJC2AZNK7Q2V2NSQMVD5TIRGVGQ) context deadline exceeded; piecestore: (Node ID: 129YE2Wd6LmDABJFaGVtYDwCHmStVDnPRCagD2cKPWA5PDTtZyh, Piece ID: UEKG3FOTTORJVQRPLAFCJEP2ZAVA5N7M2DFGH4BUTB2OGVC2KIZA) context deadline exceeded ======LOGS====== upload=472552]: Uploading piece... Dec 28 02:19:02.432783751 (538h50m2.303866796s) [noise=true , address=145.79.227.33:4896 , piece_id=D44WQB3G3NBNSB6ITIBHVV67V2EKPGXXONQU25IE6RVSGGTUJ44Q, seg_pos={0 0}, upload=472552]: Uploading piece... Dec 28 02:19:02.432842112 (538h50m2.303925147s) [noise=true , address=77.90.31.21:4900 , piece_id=UEKG3FOTTORJVQRPLAFCJEP2ZAVA5N7M2DFGH4BUTB2OGVC2KIZA, seg_pos={0 0}, upload=472552]: Uploading piece... Dec 28 02:19:02.433283032 (538h50m2.304366077s) [noise=true , address=77.90.31.15:4703 , piece_id=BERLD7G2DBBDBNE3MCS57VXAKXL3P4RQIV47ZOCU6VXMPJU4QITA, seg_pos={0 0}, upload=472552]: Uploading piece... Dec 28 02:19:02.433734562 (538h50m2.304817617s) [noise=true , address=77.90.31.16:4741 , piece_id=CX66NBDSLZWQ34Z2UDYM6HM73TABJSVHSGBQRONAQ3S6ENVTY3KQ, seg_pos={0 0}, upload=472552]: Uploading piece... Dec 28 02:19:02.434178757 (538h50m2.305261802s) [noise=true , address=77.90.31.13:4646 , piece_id=V5HTB4HVJVO7OG745JBRQSISFZU6UOORKLHEVDUDUR24EWRSSVOQ, seg_pos={0 0}, upload=472552]: Uploading piece... Dec 28 02:19:02.434595160 (538h50m2.305678195s) [noise=true , address=145.79.227.37:4596 , piece_id=XBEKEUWGAAAWZGOV3PBH7NE2CX5VID75MFGWMPHTNR7MV66U3XNA, seg_pos={0 0}, upload=472552]: Uploading piece... Dec 28 02:19:02.435217555 (538h50m2.3063006s) [noise=true , address=45.91.116.11:4136 , piece_id=JD6QHJCQ6D53BGKDPSPYBBHNCP7XDZCSV7H6ARD3JNBL75NULT6Q, seg_pos={0 0}, upload=472552]: Uploading piece... Dec 28 02:19:02.435684624 (538h50m2.306767669s) [noise=true , address=145.79.227.32:4856 , piece_id=II5XB6TRHNQAAHBPP5PRMMA7BCBGR5QO4HUBL4YJ7ZLLMK4DJB5Q, seg_pos={0 0}, upload=472552]: Uploading piece... Dec 28 02:19:02.436105054 (538h50m2.307188099s) [noise=true , address=145.79.227.14:4842 , piece_id=3F6EV4TB5TQNK5W5CJHRCCNORCEEAA6AXCOJQTQ6WJNRZG3YILOQ, seg_pos={0 0}, upload=472552]: Uploading piece... Dec 28 02:19:02.436528130 (538h50m2.307611175s) [noise=true , address=77.90.31.12:4610 , piece_id=C7IRMHAHNXDZLKFBL7OVTNVI2KWMFREMFE5GR4USVMNT5QRB3HXQ, seg_pos={0 0}, upload=472552]: Uploading piece... Dec 28 02:19:02.437497977 (538h50m2.308581022s) [noise=true , address=145.79.227.49:4340 , piece_id=5C75E4QGOX6G7QNOUUKX4GC7YL5MESWBOFSNCA6PO4PPSMQKIVFQ, seg_pos={0 0}, upload=472552]: Uploading piece... Dec 28 02:19:02.439092573 (538h50m2.310175618s) [noise=true , address=45.91.116.11:4134 , piece_id=D4PXHXGBZGH3UU2CNKDRCWD44IJQR5BQFWHCNQRPZDNYVV755ICQ, seg_pos={0 0}, upload=472552]: Uploading piece... Dec 28 02:19:02.440580235 (538h50m2.31166328s) [noise=true , address=45.91.116.5:4318 , piece_id=ULPCWUUHAL35F5WZYHDQLO7VBIGAUH2XHH4I2NYYHG7OAC5VJFOA, seg_pos={0 0}, upload=472552]: Uploading piece... Dec 28 02:19:02.442091032 (538h50m2.313174077s) [noise=true , address=77.90.31.11:4558 , piece_id=TWOTPEDKIRIOR7NQNCEFM3Z5KGWVOFUYKIQYMEGHED5FKRDCB54Q, seg_pos={0 0}, upload=472552]: Uploading piece... Dec 28 02:19:02.443305184 (538h50m2.314388229s) [noise=true , address=145.79.227.40:4078 , piece_id=3ICUQQR7NS6O2QYHBFCAQZS7A7UAWHTD3O2LOWLRZ3OCMU3WWBSA, seg_pos={0 0}, upload=472552]: Uploading piece... Dec 28 02:19:02.447829959 (538h50m2.318913014s) [noise=true , address=145.79.227.14:4863 , piece_id=N3XS43U6I4WJYUCC5NVUBRLTTHVPYKDR3KRCNGJRYU3E26WDO3HA, seg_pos={0 0}, upload=472552]: Uploading piece... Dec 28 02:19:02.451586160 (538h50m2.322669225s) [noise=true , address=45.91.116.9:4288 , piece_id=EMRORAB74LXEB6222NUCRM5ZENJC2AZNK7Q2V2NSQMVD5TIRGVGQ, seg_pos={0 0}, upload=472552]: Uploading piece... Dec 28 02:19:03.923272933 (538h50m3.794355978s) [upload=472552]: Next returned nil segment Dec 28 02:19:03.923289645 (538h50m3.79437269s) [upload=472552]: Waiting for error group managing segments... Dec 28 02:19:04.324829202 (538h50m4.195912237s) [noise=true , address=145.79.227.49:4340 , piece_id=5C75E4QGOX6G7QNOUUKX4GC7YL5MESWBOFSNCA6PO4PPSMQKIVFQ, seg_pos={0 0}, upload=472552]: Done uploading piece. err: <nil> Dec 28 02:19:04.335200317 (538h50m4.206283363s) [noise=true , address=145.79.227.14:4842 , piece_id=3F6EV4TB5TQNK5W5CJHRCCNORCEEAA6AXCOJQTQ6WJNRZG3YILOQ, seg_pos={0 0}, upload=472552]: Done uploading piece. err: <nil> Dec 28 02:19:04.343899841 (538h50m4.214982886s) [noise=true , address=145.79.227.14:4863 , piece_id=N3XS43U6I4WJYUCC5NVUBRLTTHVPYKDR3KRCNGJRYU3E26WDO3HA, seg_pos={0 0}, upload=472552]: Done uploading piece. err: <nil> Dec 28 02:19:04.378111328 (538h50m4.249194373s) [noise=true , address=145.79.227.37:4596 , piece_id=XBEKEUWGAAAWZGOV3PBH7NE2CX5VID75MFGWMPHTNR7MV66U3XNA, seg_pos={0 0}, upload=472552]: Done uploading piece. err: <nil> Dec 28 02:19:04.381135876 (538h50m4.252218921s) [noise=true , address=145.79.227.32:4856 , piece_id=II5XB6TRHNQAAHBPP5PRMMA7BCBGR5QO4HUBL4YJ7ZLLMK4DJB5Q, seg_pos={0 0}, upload=472552]: Done uploading piece. err: <nil> Dec 28 02:19:04.476390445 (538h50m4.34747348s) [noise=true , address=145.79.227.6:4290 , piece_id=K4OGXS2MXEFVQVW7HEAZJ6267HWETOUXWJAC77JNZADW76K2XMJA, seg_pos={0 0}, upload=472552]: Done uploading piece. err: <nil> Dec 28 02:19:04.533625848 (538h50m4.404708893s) [noise=true , address=145.79.227.33:4896 , piece_id=D44WQB3G3NBNSB6ITIBHVV67V2EKPGXXONQU25IE6RVSGGTUJ44Q, seg_pos={0 0}, upload=472552]: Done uploading piece. err: <nil> Dec 28 02:19:04.562532124 (538h50m4.433615159s) [noise=true , address=77.90.31.11:4558 , piece_id=TWOTPEDKIRIOR7NQNCEFM3Z5KGWVOFUYKIQYMEGHED5FKRDCB54Q, seg_pos={0 0}, upload=472552]: Done uploading piece. err: <nil> Dec 28 02:19:04.643414509 (538h50m4.514497544s) [noise=true , address=77.90.31.21:4900 , piece_id=UEKG3FOTTORJVQRPLAFCJEP2ZAVA5N7M2DFGH4BUTB2OGVC2KIZA, seg_pos={0 0}, upload=472552]: Done uploading piece. err: <nil> Dec 28 02:19:04.651013327 (538h50m4.522096372s) [noise=true , address=77.90.31.13:4646 , piece_id=V5HTB4HVJVO7OG745JBRQSISFZU6UOORKLHEVDUDUR24EWRSSVOQ, seg_pos={0 0}, upload=472552]: Done uploading piece. err: <nil> Dec 28 02:19:04.776667164 (538h50m4.647750209s) [noise=true , address=45.91.116.11:4136 , piece_id=JD6QHJCQ6D53BGKDPSPYBBHNCP7XDZCSV7H6ARD3JNBL75NULT6Q, seg_pos={0 0}, upload=472552]: Done uploading piece. err: <nil> Dec 28 02:19:04.919205529 (538h50m4.790288574s) [noise=true , address=45.91.116.11:4134 , piece_id=D4PXHXGBZGH3UU2CNKDRCWD44IJQR5BQFWHCNQRPZDNYVV755ICQ, seg_pos={0 0}, upload=472552]: Done uploading piece. err: <nil> Dec 28 02:19:04.927993490 (538h50m4.799076535s) [noise=true , address=45.91.116.9:4288 , piece_id=EMRORAB74LXEB6222NUCRM5ZENJC2AZNK7Q2V2NSQMVD5TIRGVGQ, seg_pos={0 0}, upload=472552]: Done uploading piece. err: <nil> Dec 28 02:19:05.041729347 (538h50m4.912812392s) [noise=true , address=45.91.116.5:4318 , piece_id=ULPCWUUHAL35F5WZYHDQLO7VBIGAUH2XHH4I2NYYHG7OAC5VJFOA, seg_pos={0 0}, upload=472552]: Done uploading piece. err: <nil> Dec 28 02:19:05.041751558 (538h50m4.912834593s) [seg_pos={0 0}, upload=472552]: Segment reached optimal threshold of 14 pieces. Dec 28 02:19:05.041986606 (538h50m4.913069651s) [noise=true , address=77.90.31.12:4610 , piece_id=C7IRMHAHNXDZLKFBL7OVTNVI2KWMFREMFE5GR4USVMNT5QRB3HXQ, seg_pos={0 0}, upload=472552]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi): protocol: expected piece hash; context canceled; EOF Dec 28 02:19:05.041995954 (538h50m4.913078999s) [noise=true , address=77.90.31.12:4610 , piece_id=C7IRMHAHNXDZLKFBL7OVTNVI2KWMFREMFE5GR4USVMNT5QRB3HXQ, seg_pos={0 0}, upload=472552]: Long tail cancellation due to optimal threshold Dec 28 02:19:05.042149837 (538h50m4.913232872s) [noise=true , address=77.90.31.15:4703 , piece_id=BERLD7G2DBBDBNE3MCS57VXAKXL3P4RQIV47ZOCU6VXMPJU4QITA, seg_pos={0 0}, upload=472552]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hMDD5K7WDjvN4ZgcpwpnPFb6X1ncCfyucfFrpTUshd8Ye1B6D): protocol: expected piece hash; context canceled; EOF Dec 28 02:19:05.042158714 (538h50m4.913241759s) [noise=true , address=77.90.31.15:4703 , piece_id=BERLD7G2DBBDBNE3MCS57VXAKXL3P4RQIV47ZOCU6VXMPJU4QITA, seg_pos={0 0}, upload=472552]: Long tail cancellation due to optimal threshold Dec 28 02:19:05.042299752 (538h50m4.913382788s) [noise=true , address=77.90.31.16:4741 , piece_id=CX66NBDSLZWQ34Z2UDYM6HM73TABJSVHSGBQRONAQ3S6ENVTY3KQ, seg_pos={0 0}, upload=472552]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR): protocol: expected piece hash; context canceled; EOF Dec 28 02:19:05.042304702 (538h50m4.913387737s) [noise=true , address=77.90.31.16:4741 , piece_id=CX66NBDSLZWQ34Z2UDYM6HM73TABJSVHSGBQRONAQ3S6ENVTY3KQ, seg_pos={0 0}, upload=472552]: Long tail cancellation due to optimal threshold Dec 28 02:19:05.042396276 (538h50m4.913479352s) [noise=true , address=145.79.227.40:4078 , piece_id=3ICUQQR7NS6O2QYHBFCAQZS7A7UAWHTD3O2LOWLRZ3OCMU3WWBSA, seg_pos={0 0}, upload=472552]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tzgtenV6us5ssD7xWGpZmVUoWEfbFhBNewbK9irHtZPkyDtRd): protocol: expected piece hash; context canceled; EOF Dec 28 02:19:05.042417406 (538h50m4.913500461s) [noise=true , address=145.79.227.40:4078 , piece_id=3ICUQQR7NS6O2QYHBFCAQZS7A7UAWHTD3O2LOWLRZ3OCMU3WWBSA, seg_pos={0 0}, upload=472552]: Long tail cancellation due to optimal threshold Dec 28 02:19:05.042612779 (538h50m4.913695814s) [seg_pos={0 0}, upload=472552]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 02:19:05.042821726 (538h50m4.913904771s) [upload=472552]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 02:19:05.042916818 (538h50m4.913999863s) [upload=472552]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 02:19:05.336925913 (538h50m5.208008958s) [upload=472552]: Done uploading segments... Dec 28 02:19:05.336953856 (538h50m5.208036901s) [upload=472552]: Upload finished. err: <nil> Duration: 49.778254234s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................................X....................] duration: 3.455621555s trace: https://jaeger.dev.storj.io/trace/7fc142ad3d711e34 FAIL (2025-12-28 02:22:11 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.45565079s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 32 out of 60 is failed (threshold=3) [XXXX..X.XXX.X.XX.XX.X..X.X.X.X.X..X...XX..X.XXX.XXX.X.....XX] duration: 30.002900023s trace: https://jaeger.dev.storj.io/trace/30d1fda329de480f FAIL (2025-12-28 02:21:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 02:21:15.010699307 (538h52m14.881782352s) [upload=472584]: Starting upload Dec 28 02:21:15.010771194 (538h52m14.881854239s) [upload=472584]: Done starting upload Dec 28 02:21:15.010855544 (538h52m14.881938579s) [upload=472584]: Uploading segments... Dec 28 02:21:15.010986253 (538h52m14.882069298s) [upload=472584]: Got next segment. Inline: false Dec 28 02:21:15.011024286 (538h52m14.882107331s) [upload=472584]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 02:21:15.011029586 (538h52m14.882112621s) [upload=472584]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 02:21:15.253996610 (538h52m15.125079665s) [seg_pos={0 0}, upload=472584]: Begin upload segment... Dec 28 02:21:15.254057155 (538h52m15.1251402s) [seg_pos={0 0}, upload=472584]: Done begin upload segment. Dec 28 02:21:15.254092903 (538h52m15.125175978s) [noise=true , address=145.79.227.18:4007 , piece_id=YZD3LFDT5OJY6J3YRB7KA3ZBBIHKTQBAKE5H5BGMQLDNJ6HITLOA, seg_pos={0 0}, upload=472584]: Uploading piece... Dec 28 02:21:15.254106670 (538h52m15.125189715s) [noise=true , address=145.79.227.14:4838 , piece_id=K2FSYPUEHT6MFYABRBLLWO3WXIN4SSU3TMSNFWSB3VNSJ5AG2QKQ, seg_pos={0 0}, upload=472584]: Uploading piece... Dec 28 02:21:15.254169419 (538h52m15.125252464s) [noise=true , address=45.91.116.13:4156 , piece_id=G3X4VH3XF7ZIS67EVUGDZHIEGDD3ZEDHAYRKUSK6QKJSMEZHTXXQ, seg_pos={0 0}, upload=472584]: Uploading piece... Dec 28 02:21:15.254686373 (538h52m15.125769418s) [noise=true , address=145.79.227.47:4265 , piece_id=CU2JE3PSM5KXO3GUDS3A64AQL4IRMULPNWVLDFQCRRVOW4XBKZUA, seg_pos={0 0}, upload=472584]: Uploading piece... Dec 28 02:21:15.255117745 (538h52m15.12620078s) [noise=true , address=145.79.227.11:4732 , piece_id=RJGAPRXG5Z4GAIKK4FLBZ52SWZIU74B525ZHBYZMVSCXQVE22DEA, seg_pos={0 0}, upload=472584]: Uploading piece... Dec 28 02:21:15.255526352 (538h52m15.126609397s) [noise=true , address=45.91.116.12:4227 , piece_id=UESW2GB5GZ6Y2XXPZVZOCYX7KRA4BSCGKSCMDM2F6BC6BFMSKB2Q, seg_pos={0 0}, upload=472584]: Uploading piece... Dec 28 02:21:15.255953075 (538h52m15.12703612s) [noise=true , address=145.79.227.41:4087 , piece_id=JU6LPEA6A7DIKLSNRQOMDE2SIM6KEDVDHZ4IQKHHXMUEUL7KEL7A, seg_pos={0 0}, upload=472584]: Uploading piece... Dec 28 02:21:15.256441965 (538h52m15.127525s) [noise=true , address=45.91.116.8:4046 , piece_id=7T4INYGP5W5A2AMEZ6RF6VOTEHH3XIYQVPESVYFQHQQGNSPXQSCQ, seg_pos={0 0}, upload=472584]: Uploading piece... Dec 28 02:21:15.256835946 (538h52m15.127918991s) [noise=true , address=77.90.31.12:4590 , piece_id=SJCDQTPPRVAJTSGYGT5IXXVOVH2LMG6PI6PB4TBHL55SUU2QO2IA, seg_pos={0 0}, upload=472584]: Uploading piece... Dec 28 02:21:15.258263734 (538h52m15.129346779s) [noise=true , address=145.79.227.45:4229 , piece_id=F2QOVMQZXYPXYGGGZYUQO54LG3DPXIKOSFFWGKAP6PVIFCYSLJCA, seg_pos={0 0}, upload=472584]: Uploading piece... Dec 28 02:21:15.258733840 (538h52m15.129816885s) [noise=true , address=45.91.116.10:4083 , piece_id=OU25WCCTVNZJY3HTTT4BUO4WIRQQBXVE2QGRFBXEOJQIRUWBQQJA, seg_pos={0 0}, upload=472584]: Uploading piece... Dec 28 02:21:15.260301113 (538h52m15.131384158s) [noise=true , address=45.91.116.14:4190 , piece_id=Q72E7T4PMHAVGMI74R3PEBARBIK7TRTT4FLLV4HLYZYQTXN7DGGA, seg_pos={0 0}, upload=472584]: Uploading piece... Dec 28 02:21:15.261968057 (538h52m15.133051092s) [noise=true , address=145.79.227.7:4295 , piece_id=EEUERRE6UANWXYFFQ5AXK6V4ZS2ZDVHMNQAULIKDRFBON6P6D7AQ, seg_pos={0 0}, upload=472584]: Uploading piece... Dec 28 02:21:15.263047403 (538h52m15.134130448s) [noise=true , address=77.90.31.13:4627 , piece_id=HO3TVUS3GRFRPPMWVFD7XXQG6CFUGVQ5MGM2YFCCELYOAZQSDLUA, seg_pos={0 0}, upload=472584]: Uploading piece... Dec 28 02:21:15.264546666 (538h52m15.135629711s) [noise=true , address=45.91.116.6:4974 , piece_id=NYP5FE5F5NHI5K4GTC3IUQ74SA6JODB4DA7V2TMJRLHUVVVDGIRQ, seg_pos={0 0}, upload=472584]: Uploading piece... Dec 28 02:21:15.264609616 (538h52m15.135692661s) [noise=true , address=145.79.227.15:4874 , piece_id=6TLXJ6DSILVGIVREKLBHX3EQLSWT7FBTPLV6IK2K47Z2JTARFJRA, seg_pos={0 0}, upload=472584]: Uploading piece... Dec 28 02:21:16.949278245 (538h52m16.82036128s) [upload=472584]: Next returned nil segment Dec 28 02:21:16.949294345 (538h52m16.82037738s) [upload=472584]: Waiting for error group managing segments... Dec 28 02:21:17.266577753 (538h52m17.137660788s) [noise=true , address=145.79.227.11:4732 , piece_id=RJGAPRXG5Z4GAIKK4FLBZ52SWZIU74B525ZHBYZMVSCXQVE22DEA, seg_pos={0 0}, upload=472584]: Done uploading piece. err: <nil> Dec 28 02:21:17.299768827 (538h52m17.170851862s) [noise=true , address=145.79.227.41:4087 , piece_id=JU6LPEA6A7DIKLSNRQOMDE2SIM6KEDVDHZ4IQKHHXMUEUL7KEL7A, seg_pos={0 0}, upload=472584]: Done uploading piece. err: <nil> Dec 28 02:21:17.308858794 (538h52m17.179941839s) [noise=true , address=145.79.227.14:4838 , piece_id=K2FSYPUEHT6MFYABRBLLWO3WXIN4SSU3TMSNFWSB3VNSJ5AG2QKQ, seg_pos={0 0}, upload=472584]: Done uploading piece. err: <nil> Dec 28 02:21:17.332057293 (538h52m17.203140328s) [noise=true , address=145.79.227.47:4265 , piece_id=CU2JE3PSM5KXO3GUDS3A64AQL4IRMULPNWVLDFQCRRVOW4XBKZUA, seg_pos={0 0}, upload=472584]: Done uploading piece. err: <nil> Dec 28 02:21:17.442578508 (538h52m17.313661543s) [noise=true , address=77.90.31.13:4627 , piece_id=HO3TVUS3GRFRPPMWVFD7XXQG6CFUGVQ5MGM2YFCCELYOAZQSDLUA, seg_pos={0 0}, upload=472584]: Done uploading piece. err: <nil> Dec 28 02:21:17.471623579 (538h52m17.342706614s) [noise=true , address=77.90.31.12:4590 , piece_id=SJCDQTPPRVAJTSGYGT5IXXVOVH2LMG6PI6PB4TBHL55SUU2QO2IA, seg_pos={0 0}, upload=472584]: Done uploading piece. err: <nil> Dec 28 02:21:17.496659185 (538h52m17.36774222s) [noise=true , address=45.91.116.12:4227 , piece_id=UESW2GB5GZ6Y2XXPZVZOCYX7KRA4BSCGKSCMDM2F6BC6BFMSKB2Q, seg_pos={0 0}, upload=472584]: Done uploading piece. err: <nil> Dec 28 02:21:17.533215185 (538h52m17.40429822s) [noise=true , address=45.91.116.8:4046 , piece_id=7T4INYGP5W5A2AMEZ6RF6VOTEHH3XIYQVPESVYFQHQQGNSPXQSCQ, seg_pos={0 0}, upload=472584]: Done uploading piece. err: <nil> Dec 28 02:21:17.550042884 (538h52m17.421125919s) [noise=true , address=45.91.116.10:4083 , piece_id=OU25WCCTVNZJY3HTTT4BUO4WIRQQBXVE2QGRFBXEOJQIRUWBQQJA, seg_pos={0 0}, upload=472584]: Done uploading piece. err: <nil> Dec 28 02:21:18.223692472 (538h52m18.094775507s) [noise=true , address=45.91.116.13:4156 , piece_id=G3X4VH3XF7ZIS67EVUGDZHIEGDD3ZEDHAYRKUSK6QKJSMEZHTXXQ, seg_pos={0 0}, upload=472584]: Done uploading piece. err: <nil> Dec 28 02:21:18.251465470 (538h52m18.122548515s) [noise=true , address=45.91.116.6:4974 , piece_id=NYP5FE5F5NHI5K4GTC3IUQ74SA6JODB4DA7V2TMJRLHUVVVDGIRQ, seg_pos={0 0}, upload=472584]: Done uploading piece. err: <nil> Dec 28 02:21:18.426501187 (538h52m18.297584222s) [noise=true , address=145.79.227.7:4295 , piece_id=EEUERRE6UANWXYFFQ5AXK6V4ZS2ZDVHMNQAULIKDRFBON6P6D7AQ, seg_pos={0 0}, upload=472584]: Done uploading piece. err: <nil> Dec 28 02:21:18.428572902 (538h52m18.299655937s) [noise=true , address=145.79.227.15:4874 , piece_id=6TLXJ6DSILVGIVREKLBHX3EQLSWT7FBTPLV6IK2K47Z2JTARFJRA, seg_pos={0 0}, upload=472584]: Done uploading piece. err: <nil> Dec 28 02:21:18.428586458 (538h52m18.299669493s) [seg_pos={0 0}, upload=472584]: Segment reached optimal threshold of 13 pieces. Dec 28 02:21:18.428767112 (538h52m18.299850157s) [noise=true , address=145.79.227.18:4007 , piece_id=YZD3LFDT5OJY6J3YRB7KA3ZBBIHKTQBAKE5H5BGMQLDNJ6HITLOA, seg_pos={0 0}, upload=472584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp): protocol: expected piece hash; context canceled; EOF Dec 28 02:21:18.428773243 (538h52m18.299856278s) [noise=true , address=145.79.227.18:4007 , piece_id=YZD3LFDT5OJY6J3YRB7KA3ZBBIHKTQBAKE5H5BGMQLDNJ6HITLOA, seg_pos={0 0}, upload=472584]: Long tail cancellation due to optimal threshold Dec 28 02:21:18.428902830 (538h52m18.299985876s) [noise=true , address=45.91.116.14:4190 , piece_id=Q72E7T4PMHAVGMI74R3PEBARBIK7TRTT4FLLV4HLYZYQTXN7DGGA, seg_pos={0 0}, upload=472584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Dec 28 02:21:18.428908561 (538h52m18.299991596s) [noise=true , address=45.91.116.14:4190 , piece_id=Q72E7T4PMHAVGMI74R3PEBARBIK7TRTT4FLLV4HLYZYQTXN7DGGA, seg_pos={0 0}, upload=472584]: Long tail cancellation due to optimal threshold Dec 28 02:21:18.429089185 (538h52m18.30017223s) [noise=true , address=145.79.227.45:4229 , piece_id=F2QOVMQZXYPXYGGGZYUQO54LG3DPXIKOSFFWGKAP6PVIFCYSLJCA, seg_pos={0 0}, upload=472584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka): protocol: expected piece hash; context canceled; EOF Dec 28 02:21:18.429107058 (538h52m18.300190094s) [noise=true , address=145.79.227.45:4229 , piece_id=F2QOVMQZXYPXYGGGZYUQO54LG3DPXIKOSFFWGKAP6PVIFCYSLJCA, seg_pos={0 0}, upload=472584]: Long tail cancellation due to optimal threshold Dec 28 02:21:18.429150402 (538h52m18.300233447s) [seg_pos={0 0}, upload=472584]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 02:21:18.429241314 (538h52m18.300324349s) [upload=472584]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 02:21:18.429246815 (538h52m18.30032986s) [upload=472584]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 02:21:18.722160166 (538h52m18.593243211s) [upload=472584]: Done uploading segments... Dec 28 02:21:18.722184062 (538h52m18.593267107s) [upload=472584]: Upload finished. err: <nil> Duration: 30.002926963s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................X...................X..] duration: 41.43241978s trace: https://jaeger.dev.storj.io/trace/1f68404b41391e4a FAIL (2025-12-28 02:21:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12CaBSQgwTru9NXaznMJ2XExKe3yhLoUCvDKFvqvLJGJev4mEPc, Piece ID: ULPCWUUHAL35F5WZYHDQLO7VBIGAUH2XHH4I2NYYHG7OAC5VJFOA) context deadline exceeded; piecestore: (Node ID: 1H1XywjaRt5ZRGqmZBgzfSowuCnPqFesL1Ch2jYYS1gV2oBgQX, Piece ID: EMRORAB74LXEB6222NUCRM5ZENJC2AZNK7Q2V2NSQMVD5TIRGVGQ) context deadline exceeded; piecestore: (Node ID: 129YE2Wd6LmDABJFaGVtYDwCHmStVDnPRCagD2cKPWA5PDTtZyh, Piece ID: UEKG3FOTTORJVQRPLAFCJEP2ZAVA5N7M2DFGH4BUTB2OGVC2KIZA) context deadline exceeded ======LOGS====== upload=472552]: Uploading piece... Dec 28 02:19:02.432783751 (538h50m2.303866796s) [noise=true , address=145.79.227.33:4896 , piece_id=D44WQB3G3NBNSB6ITIBHVV67V2EKPGXXONQU25IE6RVSGGTUJ44Q, seg_pos={0 0}, upload=472552]: Uploading piece... Dec 28 02:19:02.432842112 (538h50m2.303925147s) [noise=true , address=77.90.31.21:4900 , piece_id=UEKG3FOTTORJVQRPLAFCJEP2ZAVA5N7M2DFGH4BUTB2OGVC2KIZA, seg_pos={0 0}, upload=472552]: Uploading piece... Dec 28 02:19:02.433283032 (538h50m2.304366077s) [noise=true , address=77.90.31.15:4703 , piece_id=BERLD7G2DBBDBNE3MCS57VXAKXL3P4RQIV47ZOCU6VXMPJU4QITA, seg_pos={0 0}, upload=472552]: Uploading piece... Dec 28 02:19:02.433734562 (538h50m2.304817617s) [noise=true , address=77.90.31.16:4741 , piece_id=CX66NBDSLZWQ34Z2UDYM6HM73TABJSVHSGBQRONAQ3S6ENVTY3KQ, seg_pos={0 0}, upload=472552]: Uploading piece... Dec 28 02:19:02.434178757 (538h50m2.305261802s) [noise=true , address=77.90.31.13:4646 , piece_id=V5HTB4HVJVO7OG745JBRQSISFZU6UOORKLHEVDUDUR24EWRSSVOQ, seg_pos={0 0}, upload=472552]: Uploading piece... Dec 28 02:19:02.434595160 (538h50m2.305678195s) [noise=true , address=145.79.227.37:4596 , piece_id=XBEKEUWGAAAWZGOV3PBH7NE2CX5VID75MFGWMPHTNR7MV66U3XNA, seg_pos={0 0}, upload=472552]: Uploading piece... Dec 28 02:19:02.435217555 (538h50m2.3063006s) [noise=true , address=45.91.116.11:4136 , piece_id=JD6QHJCQ6D53BGKDPSPYBBHNCP7XDZCSV7H6ARD3JNBL75NULT6Q, seg_pos={0 0}, upload=472552]: Uploading piece... Dec 28 02:19:02.435684624 (538h50m2.306767669s) [noise=true , address=145.79.227.32:4856 , piece_id=II5XB6TRHNQAAHBPP5PRMMA7BCBGR5QO4HUBL4YJ7ZLLMK4DJB5Q, seg_pos={0 0}, upload=472552]: Uploading piece... Dec 28 02:19:02.436105054 (538h50m2.307188099s) [noise=true , address=145.79.227.14:4842 , piece_id=3F6EV4TB5TQNK5W5CJHRCCNORCEEAA6AXCOJQTQ6WJNRZG3YILOQ, seg_pos={0 0}, upload=472552]: Uploading piece... Dec 28 02:19:02.436528130 (538h50m2.307611175s) [noise=true , address=77.90.31.12:4610 , piece_id=C7IRMHAHNXDZLKFBL7OVTNVI2KWMFREMFE5GR4USVMNT5QRB3HXQ, seg_pos={0 0}, upload=472552]: Uploading piece... Dec 28 02:19:02.437497977 (538h50m2.308581022s) [noise=true , address=145.79.227.49:4340 , piece_id=5C75E4QGOX6G7QNOUUKX4GC7YL5MESWBOFSNCA6PO4PPSMQKIVFQ, seg_pos={0 0}, upload=472552]: Uploading piece... Dec 28 02:19:02.439092573 (538h50m2.310175618s) [noise=true , address=45.91.116.11:4134 , piece_id=D4PXHXGBZGH3UU2CNKDRCWD44IJQR5BQFWHCNQRPZDNYVV755ICQ, seg_pos={0 0}, upload=472552]: Uploading piece... Dec 28 02:19:02.440580235 (538h50m2.31166328s) [noise=true , address=45.91.116.5:4318 , piece_id=ULPCWUUHAL35F5WZYHDQLO7VBIGAUH2XHH4I2NYYHG7OAC5VJFOA, seg_pos={0 0}, upload=472552]: Uploading piece... Dec 28 02:19:02.442091032 (538h50m2.313174077s) [noise=true , address=77.90.31.11:4558 , piece_id=TWOTPEDKIRIOR7NQNCEFM3Z5KGWVOFUYKIQYMEGHED5FKRDCB54Q, seg_pos={0 0}, upload=472552]: Uploading piece... Dec 28 02:19:02.443305184 (538h50m2.314388229s) [noise=true , address=145.79.227.40:4078 , piece_id=3ICUQQR7NS6O2QYHBFCAQZS7A7UAWHTD3O2LOWLRZ3OCMU3WWBSA, seg_pos={0 0}, upload=472552]: Uploading piece... Dec 28 02:19:02.447829959 (538h50m2.318913014s) [noise=true , address=145.79.227.14:4863 , piece_id=N3XS43U6I4WJYUCC5NVUBRLTTHVPYKDR3KRCNGJRYU3E26WDO3HA, seg_pos={0 0}, upload=472552]: Uploading piece... Dec 28 02:19:02.451586160 (538h50m2.322669225s) [noise=true , address=45.91.116.9:4288 , piece_id=EMRORAB74LXEB6222NUCRM5ZENJC2AZNK7Q2V2NSQMVD5TIRGVGQ, seg_pos={0 0}, upload=472552]: Uploading piece... Dec 28 02:19:03.923272933 (538h50m3.794355978s) [upload=472552]: Next returned nil segment Dec 28 02:19:03.923289645 (538h50m3.79437269s) [upload=472552]: Waiting for error group managing segments... Dec 28 02:19:04.324829202 (538h50m4.195912237s) [noise=true , address=145.79.227.49:4340 , piece_id=5C75E4QGOX6G7QNOUUKX4GC7YL5MESWBOFSNCA6PO4PPSMQKIVFQ, seg_pos={0 0}, upload=472552]: Done uploading piece. err: <nil> Dec 28 02:19:04.335200317 (538h50m4.206283363s) [noise=true , address=145.79.227.14:4842 , piece_id=3F6EV4TB5TQNK5W5CJHRCCNORCEEAA6AXCOJQTQ6WJNRZG3YILOQ, seg_pos={0 0}, upload=472552]: Done uploading piece. err: <nil> Dec 28 02:19:04.343899841 (538h50m4.214982886s) [noise=true , address=145.79.227.14:4863 , piece_id=N3XS43U6I4WJYUCC5NVUBRLTTHVPYKDR3KRCNGJRYU3E26WDO3HA, seg_pos={0 0}, upload=472552]: Done uploading piece. err: <nil> Dec 28 02:19:04.378111328 (538h50m4.249194373s) [noise=true , address=145.79.227.37:4596 , piece_id=XBEKEUWGAAAWZGOV3PBH7NE2CX5VID75MFGWMPHTNR7MV66U3XNA, seg_pos={0 0}, upload=472552]: Done uploading piece. err: <nil> Dec 28 02:19:04.381135876 (538h50m4.252218921s) [noise=true , address=145.79.227.32:4856 , piece_id=II5XB6TRHNQAAHBPP5PRMMA7BCBGR5QO4HUBL4YJ7ZLLMK4DJB5Q, seg_pos={0 0}, upload=472552]: Done uploading piece. err: <nil> Dec 28 02:19:04.476390445 (538h50m4.34747348s) [noise=true , address=145.79.227.6:4290 , piece_id=K4OGXS2MXEFVQVW7HEAZJ6267HWETOUXWJAC77JNZADW76K2XMJA, seg_pos={0 0}, upload=472552]: Done uploading piece. err: <nil> Dec 28 02:19:04.533625848 (538h50m4.404708893s) [noise=true , address=145.79.227.33:4896 , piece_id=D44WQB3G3NBNSB6ITIBHVV67V2EKPGXXONQU25IE6RVSGGTUJ44Q, seg_pos={0 0}, upload=472552]: Done uploading piece. err: <nil> Dec 28 02:19:04.562532124 (538h50m4.433615159s) [noise=true , address=77.90.31.11:4558 , piece_id=TWOTPEDKIRIOR7NQNCEFM3Z5KGWVOFUYKIQYMEGHED5FKRDCB54Q, seg_pos={0 0}, upload=472552]: Done uploading piece. err: <nil> Dec 28 02:19:04.643414509 (538h50m4.514497544s) [noise=true , address=77.90.31.21:4900 , piece_id=UEKG3FOTTORJVQRPLAFCJEP2ZAVA5N7M2DFGH4BUTB2OGVC2KIZA, seg_pos={0 0}, upload=472552]: Done uploading piece. err: <nil> Dec 28 02:19:04.651013327 (538h50m4.522096372s) [noise=true , address=77.90.31.13:4646 , piece_id=V5HTB4HVJVO7OG745JBRQSISFZU6UOORKLHEVDUDUR24EWRSSVOQ, seg_pos={0 0}, upload=472552]: Done uploading piece. err: <nil> Dec 28 02:19:04.776667164 (538h50m4.647750209s) [noise=true , address=45.91.116.11:4136 , piece_id=JD6QHJCQ6D53BGKDPSPYBBHNCP7XDZCSV7H6ARD3JNBL75NULT6Q, seg_pos={0 0}, upload=472552]: Done uploading piece. err: <nil> Dec 28 02:19:04.919205529 (538h50m4.790288574s) [noise=true , address=45.91.116.11:4134 , piece_id=D4PXHXGBZGH3UU2CNKDRCWD44IJQR5BQFWHCNQRPZDNYVV755ICQ, seg_pos={0 0}, upload=472552]: Done uploading piece. err: <nil> Dec 28 02:19:04.927993490 (538h50m4.799076535s) [noise=true , address=45.91.116.9:4288 , piece_id=EMRORAB74LXEB6222NUCRM5ZENJC2AZNK7Q2V2NSQMVD5TIRGVGQ, seg_pos={0 0}, upload=472552]: Done uploading piece. err: <nil> Dec 28 02:19:05.041729347 (538h50m4.912812392s) [noise=true , address=45.91.116.5:4318 , piece_id=ULPCWUUHAL35F5WZYHDQLO7VBIGAUH2XHH4I2NYYHG7OAC5VJFOA, seg_pos={0 0}, upload=472552]: Done uploading piece. err: <nil> Dec 28 02:19:05.041751558 (538h50m4.912834593s) [seg_pos={0 0}, upload=472552]: Segment reached optimal threshold of 14 pieces. Dec 28 02:19:05.041986606 (538h50m4.913069651s) [noise=true , address=77.90.31.12:4610 , piece_id=C7IRMHAHNXDZLKFBL7OVTNVI2KWMFREMFE5GR4USVMNT5QRB3HXQ, seg_pos={0 0}, upload=472552]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi): protocol: expected piece hash; context canceled; EOF Dec 28 02:19:05.041995954 (538h50m4.913078999s) [noise=true , address=77.90.31.12:4610 , piece_id=C7IRMHAHNXDZLKFBL7OVTNVI2KWMFREMFE5GR4USVMNT5QRB3HXQ, seg_pos={0 0}, upload=472552]: Long tail cancellation due to optimal threshold Dec 28 02:19:05.042149837 (538h50m4.913232872s) [noise=true , address=77.90.31.15:4703 , piece_id=BERLD7G2DBBDBNE3MCS57VXAKXL3P4RQIV47ZOCU6VXMPJU4QITA, seg_pos={0 0}, upload=472552]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hMDD5K7WDjvN4ZgcpwpnPFb6X1ncCfyucfFrpTUshd8Ye1B6D): protocol: expected piece hash; context canceled; EOF Dec 28 02:19:05.042158714 (538h50m4.913241759s) [noise=true , address=77.90.31.15:4703 , piece_id=BERLD7G2DBBDBNE3MCS57VXAKXL3P4RQIV47ZOCU6VXMPJU4QITA, seg_pos={0 0}, upload=472552]: Long tail cancellation due to optimal threshold Dec 28 02:19:05.042299752 (538h50m4.913382788s) [noise=true , address=77.90.31.16:4741 , piece_id=CX66NBDSLZWQ34Z2UDYM6HM73TABJSVHSGBQRONAQ3S6ENVTY3KQ, seg_pos={0 0}, upload=472552]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR): protocol: expected piece hash; context canceled; EOF Dec 28 02:19:05.042304702 (538h50m4.913387737s) [noise=true , address=77.90.31.16:4741 , piece_id=CX66NBDSLZWQ34Z2UDYM6HM73TABJSVHSGBQRONAQ3S6ENVTY3KQ, seg_pos={0 0}, upload=472552]: Long tail cancellation due to optimal threshold Dec 28 02:19:05.042396276 (538h50m4.913479352s) [noise=true , address=145.79.227.40:4078 , piece_id=3ICUQQR7NS6O2QYHBFCAQZS7A7UAWHTD3O2LOWLRZ3OCMU3WWBSA, seg_pos={0 0}, upload=472552]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tzgtenV6us5ssD7xWGpZmVUoWEfbFhBNewbK9irHtZPkyDtRd): protocol: expected piece hash; context canceled; EOF Dec 28 02:19:05.042417406 (538h50m4.913500461s) [noise=true , address=145.79.227.40:4078 , piece_id=3ICUQQR7NS6O2QYHBFCAQZS7A7UAWHTD3O2LOWLRZ3OCMU3WWBSA, seg_pos={0 0}, upload=472552]: Long tail cancellation due to optimal threshold Dec 28 02:19:05.042612779 (538h50m4.913695814s) [seg_pos={0 0}, upload=472552]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 02:19:05.042821726 (538h50m4.913904771s) [upload=472552]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 02:19:05.042916818 (538h50m4.913999863s) [upload=472552]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 02:19:05.336925913 (538h50m5.208008958s) [upload=472552]: Done uploading segments... Dec 28 02:19:05.336953856 (538h50m5.208036901s) [upload=472552]: Upload finished. err: <nil> Duration: 41.432470405s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................................X...................] duration: 3.576985334s trace: https://jaeger.dev.storj.io/trace/7fc142ad3d711e34 FAIL (2025-12-28 02:21:05 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.577022876s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................X...................X.] duration: 47.763377649s trace: https://jaeger.dev.storj.io/trace/1f68404b41391e4a FAIL (2025-12-28 02:20:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12CaBSQgwTru9NXaznMJ2XExKe3yhLoUCvDKFvqvLJGJev4mEPc, Piece ID: ULPCWUUHAL35F5WZYHDQLO7VBIGAUH2XHH4I2NYYHG7OAC5VJFOA) context deadline exceeded; piecestore: (Node ID: 1H1XywjaRt5ZRGqmZBgzfSowuCnPqFesL1Ch2jYYS1gV2oBgQX, Piece ID: EMRORAB74LXEB6222NUCRM5ZENJC2AZNK7Q2V2NSQMVD5TIRGVGQ) context deadline exceeded; piecestore: (Node ID: 129YE2Wd6LmDABJFaGVtYDwCHmStVDnPRCagD2cKPWA5PDTtZyh, Piece ID: UEKG3FOTTORJVQRPLAFCJEP2ZAVA5N7M2DFGH4BUTB2OGVC2KIZA) context deadline exceeded ======LOGS====== upload=472552]: Uploading piece... Dec 28 02:19:02.432783751 (538h50m2.303866796s) [noise=true , address=145.79.227.33:4896 , piece_id=D44WQB3G3NBNSB6ITIBHVV67V2EKPGXXONQU25IE6RVSGGTUJ44Q, seg_pos={0 0}, upload=472552]: Uploading piece... Dec 28 02:19:02.432842112 (538h50m2.303925147s) [noise=true , address=77.90.31.21:4900 , piece_id=UEKG3FOTTORJVQRPLAFCJEP2ZAVA5N7M2DFGH4BUTB2OGVC2KIZA, seg_pos={0 0}, upload=472552]: Uploading piece... Dec 28 02:19:02.433283032 (538h50m2.304366077s) [noise=true , address=77.90.31.15:4703 , piece_id=BERLD7G2DBBDBNE3MCS57VXAKXL3P4RQIV47ZOCU6VXMPJU4QITA, seg_pos={0 0}, upload=472552]: Uploading piece... Dec 28 02:19:02.433734562 (538h50m2.304817617s) [noise=true , address=77.90.31.16:4741 , piece_id=CX66NBDSLZWQ34Z2UDYM6HM73TABJSVHSGBQRONAQ3S6ENVTY3KQ, seg_pos={0 0}, upload=472552]: Uploading piece... Dec 28 02:19:02.434178757 (538h50m2.305261802s) [noise=true , address=77.90.31.13:4646 , piece_id=V5HTB4HVJVO7OG745JBRQSISFZU6UOORKLHEVDUDUR24EWRSSVOQ, seg_pos={0 0}, upload=472552]: Uploading piece... Dec 28 02:19:02.434595160 (538h50m2.305678195s) [noise=true , address=145.79.227.37:4596 , piece_id=XBEKEUWGAAAWZGOV3PBH7NE2CX5VID75MFGWMPHTNR7MV66U3XNA, seg_pos={0 0}, upload=472552]: Uploading piece... Dec 28 02:19:02.435217555 (538h50m2.3063006s) [noise=true , address=45.91.116.11:4136 , piece_id=JD6QHJCQ6D53BGKDPSPYBBHNCP7XDZCSV7H6ARD3JNBL75NULT6Q, seg_pos={0 0}, upload=472552]: Uploading piece... Dec 28 02:19:02.435684624 (538h50m2.306767669s) [noise=true , address=145.79.227.32:4856 , piece_id=II5XB6TRHNQAAHBPP5PRMMA7BCBGR5QO4HUBL4YJ7ZLLMK4DJB5Q, seg_pos={0 0}, upload=472552]: Uploading piece... Dec 28 02:19:02.436105054 (538h50m2.307188099s) [noise=true , address=145.79.227.14:4842 , piece_id=3F6EV4TB5TQNK5W5CJHRCCNORCEEAA6AXCOJQTQ6WJNRZG3YILOQ, seg_pos={0 0}, upload=472552]: Uploading piece... Dec 28 02:19:02.436528130 (538h50m2.307611175s) [noise=true , address=77.90.31.12:4610 , piece_id=C7IRMHAHNXDZLKFBL7OVTNVI2KWMFREMFE5GR4USVMNT5QRB3HXQ, seg_pos={0 0}, upload=472552]: Uploading piece... Dec 28 02:19:02.437497977 (538h50m2.308581022s) [noise=true , address=145.79.227.49:4340 , piece_id=5C75E4QGOX6G7QNOUUKX4GC7YL5MESWBOFSNCA6PO4PPSMQKIVFQ, seg_pos={0 0}, upload=472552]: Uploading piece... Dec 28 02:19:02.439092573 (538h50m2.310175618s) [noise=true , address=45.91.116.11:4134 , piece_id=D4PXHXGBZGH3UU2CNKDRCWD44IJQR5BQFWHCNQRPZDNYVV755ICQ, seg_pos={0 0}, upload=472552]: Uploading piece... Dec 28 02:19:02.440580235 (538h50m2.31166328s) [noise=true , address=45.91.116.5:4318 , piece_id=ULPCWUUHAL35F5WZYHDQLO7VBIGAUH2XHH4I2NYYHG7OAC5VJFOA, seg_pos={0 0}, upload=472552]: Uploading piece... Dec 28 02:19:02.442091032 (538h50m2.313174077s) [noise=true , address=77.90.31.11:4558 , piece_id=TWOTPEDKIRIOR7NQNCEFM3Z5KGWVOFUYKIQYMEGHED5FKRDCB54Q, seg_pos={0 0}, upload=472552]: Uploading piece... Dec 28 02:19:02.443305184 (538h50m2.314388229s) [noise=true , address=145.79.227.40:4078 , piece_id=3ICUQQR7NS6O2QYHBFCAQZS7A7UAWHTD3O2LOWLRZ3OCMU3WWBSA, seg_pos={0 0}, upload=472552]: Uploading piece... Dec 28 02:19:02.447829959 (538h50m2.318913014s) [noise=true , address=145.79.227.14:4863 , piece_id=N3XS43U6I4WJYUCC5NVUBRLTTHVPYKDR3KRCNGJRYU3E26WDO3HA, seg_pos={0 0}, upload=472552]: Uploading piece... Dec 28 02:19:02.451586160 (538h50m2.322669225s) [noise=true , address=45.91.116.9:4288 , piece_id=EMRORAB74LXEB6222NUCRM5ZENJC2AZNK7Q2V2NSQMVD5TIRGVGQ, seg_pos={0 0}, upload=472552]: Uploading piece... Dec 28 02:19:03.923272933 (538h50m3.794355978s) [upload=472552]: Next returned nil segment Dec 28 02:19:03.923289645 (538h50m3.79437269s) [upload=472552]: Waiting for error group managing segments... Dec 28 02:19:04.324829202 (538h50m4.195912237s) [noise=true , address=145.79.227.49:4340 , piece_id=5C75E4QGOX6G7QNOUUKX4GC7YL5MESWBOFSNCA6PO4PPSMQKIVFQ, seg_pos={0 0}, upload=472552]: Done uploading piece. err: <nil> Dec 28 02:19:04.335200317 (538h50m4.206283363s) [noise=true , address=145.79.227.14:4842 , piece_id=3F6EV4TB5TQNK5W5CJHRCCNORCEEAA6AXCOJQTQ6WJNRZG3YILOQ, seg_pos={0 0}, upload=472552]: Done uploading piece. err: <nil> Dec 28 02:19:04.343899841 (538h50m4.214982886s) [noise=true , address=145.79.227.14:4863 , piece_id=N3XS43U6I4WJYUCC5NVUBRLTTHVPYKDR3KRCNGJRYU3E26WDO3HA, seg_pos={0 0}, upload=472552]: Done uploading piece. err: <nil> Dec 28 02:19:04.378111328 (538h50m4.249194373s) [noise=true , address=145.79.227.37:4596 , piece_id=XBEKEUWGAAAWZGOV3PBH7NE2CX5VID75MFGWMPHTNR7MV66U3XNA, seg_pos={0 0}, upload=472552]: Done uploading piece. err: <nil> Dec 28 02:19:04.381135876 (538h50m4.252218921s) [noise=true , address=145.79.227.32:4856 , piece_id=II5XB6TRHNQAAHBPP5PRMMA7BCBGR5QO4HUBL4YJ7ZLLMK4DJB5Q, seg_pos={0 0}, upload=472552]: Done uploading piece. err: <nil> Dec 28 02:19:04.476390445 (538h50m4.34747348s) [noise=true , address=145.79.227.6:4290 , piece_id=K4OGXS2MXEFVQVW7HEAZJ6267HWETOUXWJAC77JNZADW76K2XMJA, seg_pos={0 0}, upload=472552]: Done uploading piece. err: <nil> Dec 28 02:19:04.533625848 (538h50m4.404708893s) [noise=true , address=145.79.227.33:4896 , piece_id=D44WQB3G3NBNSB6ITIBHVV67V2EKPGXXONQU25IE6RVSGGTUJ44Q, seg_pos={0 0}, upload=472552]: Done uploading piece. err: <nil> Dec 28 02:19:04.562532124 (538h50m4.433615159s) [noise=true , address=77.90.31.11:4558 , piece_id=TWOTPEDKIRIOR7NQNCEFM3Z5KGWVOFUYKIQYMEGHED5FKRDCB54Q, seg_pos={0 0}, upload=472552]: Done uploading piece. err: <nil> Dec 28 02:19:04.643414509 (538h50m4.514497544s) [noise=true , address=77.90.31.21:4900 , piece_id=UEKG3FOTTORJVQRPLAFCJEP2ZAVA5N7M2DFGH4BUTB2OGVC2KIZA, seg_pos={0 0}, upload=472552]: Done uploading piece. err: <nil> Dec 28 02:19:04.651013327 (538h50m4.522096372s) [noise=true , address=77.90.31.13:4646 , piece_id=V5HTB4HVJVO7OG745JBRQSISFZU6UOORKLHEVDUDUR24EWRSSVOQ, seg_pos={0 0}, upload=472552]: Done uploading piece. err: <nil> Dec 28 02:19:04.776667164 (538h50m4.647750209s) [noise=true , address=45.91.116.11:4136 , piece_id=JD6QHJCQ6D53BGKDPSPYBBHNCP7XDZCSV7H6ARD3JNBL75NULT6Q, seg_pos={0 0}, upload=472552]: Done uploading piece. err: <nil> Dec 28 02:19:04.919205529 (538h50m4.790288574s) [noise=true , address=45.91.116.11:4134 , piece_id=D4PXHXGBZGH3UU2CNKDRCWD44IJQR5BQFWHCNQRPZDNYVV755ICQ, seg_pos={0 0}, upload=472552]: Done uploading piece. err: <nil> Dec 28 02:19:04.927993490 (538h50m4.799076535s) [noise=true , address=45.91.116.9:4288 , piece_id=EMRORAB74LXEB6222NUCRM5ZENJC2AZNK7Q2V2NSQMVD5TIRGVGQ, seg_pos={0 0}, upload=472552]: Done uploading piece. err: <nil> Dec 28 02:19:05.041729347 (538h50m4.912812392s) [noise=true , address=45.91.116.5:4318 , piece_id=ULPCWUUHAL35F5WZYHDQLO7VBIGAUH2XHH4I2NYYHG7OAC5VJFOA, seg_pos={0 0}, upload=472552]: Done uploading piece. err: <nil> Dec 28 02:19:05.041751558 (538h50m4.912834593s) [seg_pos={0 0}, upload=472552]: Segment reached optimal threshold of 14 pieces. Dec 28 02:19:05.041986606 (538h50m4.913069651s) [noise=true , address=77.90.31.12:4610 , piece_id=C7IRMHAHNXDZLKFBL7OVTNVI2KWMFREMFE5GR4USVMNT5QRB3HXQ, seg_pos={0 0}, upload=472552]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi): protocol: expected piece hash; context canceled; EOF Dec 28 02:19:05.041995954 (538h50m4.913078999s) [noise=true , address=77.90.31.12:4610 , piece_id=C7IRMHAHNXDZLKFBL7OVTNVI2KWMFREMFE5GR4USVMNT5QRB3HXQ, seg_pos={0 0}, upload=472552]: Long tail cancellation due to optimal threshold Dec 28 02:19:05.042149837 (538h50m4.913232872s) [noise=true , address=77.90.31.15:4703 , piece_id=BERLD7G2DBBDBNE3MCS57VXAKXL3P4RQIV47ZOCU6VXMPJU4QITA, seg_pos={0 0}, upload=472552]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hMDD5K7WDjvN4ZgcpwpnPFb6X1ncCfyucfFrpTUshd8Ye1B6D): protocol: expected piece hash; context canceled; EOF Dec 28 02:19:05.042158714 (538h50m4.913241759s) [noise=true , address=77.90.31.15:4703 , piece_id=BERLD7G2DBBDBNE3MCS57VXAKXL3P4RQIV47ZOCU6VXMPJU4QITA, seg_pos={0 0}, upload=472552]: Long tail cancellation due to optimal threshold Dec 28 02:19:05.042299752 (538h50m4.913382788s) [noise=true , address=77.90.31.16:4741 , piece_id=CX66NBDSLZWQ34Z2UDYM6HM73TABJSVHSGBQRONAQ3S6ENVTY3KQ, seg_pos={0 0}, upload=472552]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR): protocol: expected piece hash; context canceled; EOF Dec 28 02:19:05.042304702 (538h50m4.913387737s) [noise=true , address=77.90.31.16:4741 , piece_id=CX66NBDSLZWQ34Z2UDYM6HM73TABJSVHSGBQRONAQ3S6ENVTY3KQ, seg_pos={0 0}, upload=472552]: Long tail cancellation due to optimal threshold Dec 28 02:19:05.042396276 (538h50m4.913479352s) [noise=true , address=145.79.227.40:4078 , piece_id=3ICUQQR7NS6O2QYHBFCAQZS7A7UAWHTD3O2LOWLRZ3OCMU3WWBSA, seg_pos={0 0}, upload=472552]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tzgtenV6us5ssD7xWGpZmVUoWEfbFhBNewbK9irHtZPkyDtRd): protocol: expected piece hash; context canceled; EOF Dec 28 02:19:05.042417406 (538h50m4.913500461s) [noise=true , address=145.79.227.40:4078 , piece_id=3ICUQQR7NS6O2QYHBFCAQZS7A7UAWHTD3O2LOWLRZ3OCMU3WWBSA, seg_pos={0 0}, upload=472552]: Long tail cancellation due to optimal threshold Dec 28 02:19:05.042612779 (538h50m4.913695814s) [seg_pos={0 0}, upload=472552]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 02:19:05.042821726 (538h50m4.913904771s) [upload=472552]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 02:19:05.042916818 (538h50m4.913999863s) [upload=472552]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 02:19:05.336925913 (538h50m5.208008958s) [upload=472552]: Done uploading segments... Dec 28 02:19:05.336953856 (538h50m5.208036901s) [upload=472552]: Upload finished. err: <nil> Duration: 47.763392929s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 32 out of 60 is failed (threshold=3) [XXXXX..X.XXX.X.XX.XX.X..X.X.X.X.X..X...XX..X.XXX.XXX.X.....X] duration: 30.003223578s trace: https://jaeger.dev.storj.io/trace/34bfc05c41018d57 FAIL (2025-12-28 02:20:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: NRZ6EW5LGJ5KBFMIQTCBYTQR4PPTWU7V2EKNC5HRNVHDD23HXPVQ) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: XV7ZZEQPAEZ63WSGJC2QHITQA75UX7TCIAVDAKYIJQWLW65QXGHA) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: PFARERA2YZ4SAYZKCHHBGWASO5PXSGYI2QYELIJSWLVIHLYC7BWQ) context deadline exceeded ======LOGS====== 2:20:10.438235560 (538h51m10.309318595s) [upload=472569]: Starting upload Dec 28 02:20:10.438300133 (538h51m10.309383168s) [upload=472569]: Done starting upload Dec 28 02:20:10.438366860 (538h51m10.309449895s) [upload=472569]: Uploading segments... Dec 28 02:20:10.438504151 (538h51m10.309587186s) [upload=472569]: Got next segment. Inline: false Dec 28 02:20:10.438540731 (538h51m10.309623776s) [upload=472569]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 02:20:10.438545390 (538h51m10.309628435s) [upload=472569]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 02:20:10.689544007 (538h51m10.560627062s) [seg_pos={0 0}, upload=472569]: Begin upload segment... Dec 28 02:20:10.689604482 (538h51m10.560687527s) [seg_pos={0 0}, upload=472569]: Done begin upload segment. Dec 28 02:20:10.689642704 (538h51m10.560725749s) [noise=true , address=145.79.227.47:4266 , piece_id=2JHVUZPYFLQIVJRTJB2QBHQA6E5KF62OVHOFGKIJMY4N6MJLGXFQ, seg_pos={0 0}, upload=472569]: Uploading piece... Dec 28 02:20:10.689694473 (538h51m10.560777508s) [noise=true , address=45.91.116.4:4939 , piece_id=PW6A23TVKXIB4LT7RH2J5JMUSEZFGTJNKOBY2V4MB7OKBA7UAOEA, seg_pos={0 0}, upload=472569]: Uploading piece... Dec 28 02:20:10.690241404 (538h51m10.561324439s) [noise=true , address=145.79.227.45:4229 , piece_id=YNWPHWZGXLHRYBPV7CNLFCL7BBJMDPZZ7X2524PE2ZT6S6TT5SZA, seg_pos={0 0}, upload=472569]: Uploading piece... Dec 28 02:20:10.690779189 (538h51m10.561862234s) [noise=true , address=77.90.31.14:4662 , piece_id=MV2B4MXLHGCDUW4TQC2NGLCJ7XZNNK5AFRJYVYJDJDEKTFOSMBMQ, seg_pos={0 0}, upload=472569]: Uploading piece... Dec 28 02:20:10.691333815 (538h51m10.56241686s) [noise=true , address=145.79.227.38:4978 , piece_id=IAGO56T4YQ5WY7TVNV2RVJKY3XI4OFN5FQDDRUJ2Q7PSBPQ4XIOA, seg_pos={0 0}, upload=472569]: Uploading piece... Dec 28 02:20:10.693665213 (538h51m10.564748248s) [noise=true , address=45.91.116.11:4120 , piece_id=NRZ6EW5LGJ5KBFMIQTCBYTQR4PPTWU7V2EKNC5HRNVHDD23HXPVQ, seg_pos={0 0}, upload=472569]: Uploading piece... Dec 28 02:20:10.694087247 (538h51m10.565170292s) [noise=true , address=145.79.227.34:4907 , piece_id=SD57MI4S5AKIJIKQNYA4BGASGVZA3RTRDQRH22TC45HQCFNW7U6A, seg_pos={0 0}, upload=472569]: Uploading piece... Dec 28 02:20:10.695480490 (538h51m10.566563535s) [noise=true , address=145.79.227.2:4115 , piece_id=HTCFBKEY5S3DV2VX5M363KEAZSIFJCOBEJ6N34BVAHYZESSRV4NA, seg_pos={0 0}, upload=472569]: Uploading piece... Dec 28 02:20:10.696016009 (538h51m10.567099054s) [noise=true , address=145.79.227.7:4293 , piece_id=NZ3IVEEEN4BOL4ZHGA4RELOWZNEUY4UELNARMYPJUDA6W2FEVBDQ, seg_pos={0 0}, upload=472569]: Uploading piece... Dec 28 02:20:10.697440421 (538h51m10.568523467s) [noise=true , address=45.91.116.7:4010 , piece_id=XV7ZZEQPAEZ63WSGJC2QHITQA75UX7TCIAVDAKYIJQWLW65QXGHA, seg_pos={0 0}, upload=472569]: Uploading piece... Dec 28 02:20:10.697937197 (538h51m10.569020242s) [noise=true , address=145.79.227.50:4373 , piece_id=JPUFWXM26VZ7NNMQ5X7BJEZZDIVT3ISZOK2E22HNHO7QAOUULMVA, seg_pos={0 0}, upload=472569]: Uploading piece... Dec 28 02:20:10.699022584 (538h51m10.570105629s) [noise=true , address=77.90.31.15:4700 , piece_id=QAUZQP3R4CSW2R4E6I36CYRMCL4PF532UQ3U36DP37CBMRTZK2CA, seg_pos={0 0}, upload=472569]: Uploading piece... Dec 28 02:20:10.699681898 (538h51m10.570764943s) [noise=true , address=145.79.227.5:4223 , piece_id=ARMGJ3H65L2QOQO5MM43D6OFJZK2Y5XEHFOGJDZC4HMOZAVYBNWQ, seg_pos={0 0}, upload=472569]: Uploading piece... Dec 28 02:20:10.700369969 (538h51m10.571453014s) [noise=true , address=145.79.227.52:4447 , piece_id=EA3CNFLJUUAEWHRDESNJWD7MA4LDNSXIYEY6363HPDFGG6WP5E6A, seg_pos={0 0}, upload=472569]: Uploading piece... Dec 28 02:20:10.701858743 (538h51m10.572941788s) [noise=true , address=145.79.227.36:4518 , piece_id=NKYCFZIPW7L5ELURKXXWN3ZA6PSM3SSLMFQIFTEKHGE7IBAJAY2Q, seg_pos={0 0}, upload=472569]: Uploading piece... Dec 28 02:20:10.703512422 (538h51m10.574595457s) [noise=true , address=45.91.116.8:4045 , piece_id=PFARERA2YZ4SAYZKCHHBGWASO5PXSGYI2QYELIJSWLVIHLYC7BWQ, seg_pos={0 0}, upload=472569]: Uploading piece... Dec 28 02:20:12.531362918 (538h51m12.402445953s) [upload=472569]: Next returned nil segment Dec 28 02:20:12.531382576 (538h51m12.402465611s) [upload=472569]: Waiting for error group managing segments... Dec 28 02:20:12.833720969 (538h51m12.704804014s) [noise=true , address=145.79.227.7:4293 , piece_id=NZ3IVEEEN4BOL4ZHGA4RELOWZNEUY4UELNARMYPJUDA6W2FEVBDQ, seg_pos={0 0}, upload=472569]: Done uploading piece. err: <nil> Dec 28 02:20:12.860224621 (538h51m12.731307656s) [noise=true , address=145.79.227.38:4978 , piece_id=IAGO56T4YQ5WY7TVNV2RVJKY3XI4OFN5FQDDRUJ2Q7PSBPQ4XIOA, seg_pos={0 0}, upload=472569]: Done uploading piece. err: <nil> Dec 28 02:20:12.868864721 (538h51m12.739947756s) [noise=true , address=145.79.227.50:4373 , piece_id=JPUFWXM26VZ7NNMQ5X7BJEZZDIVT3ISZOK2E22HNHO7QAOUULMVA, seg_pos={0 0}, upload=472569]: Done uploading piece. err: <nil> Dec 28 02:20:12.871542629 (538h51m12.742625664s) [noise=true , address=145.79.227.45:4229 , piece_id=YNWPHWZGXLHRYBPV7CNLFCL7BBJMDPZZ7X2524PE2ZT6S6TT5SZA, seg_pos={0 0}, upload=472569]: Done uploading piece. err: <nil> Dec 28 02:20:12.877556689 (538h51m12.748639735s) [noise=true , address=145.79.227.5:4223 , piece_id=ARMGJ3H65L2QOQO5MM43D6OFJZK2Y5XEHFOGJDZC4HMOZAVYBNWQ, seg_pos={0 0}, upload=472569]: Done uploading piece. err: <nil> Dec 28 02:20:12.916746525 (538h51m12.78782956s) [noise=true , address=145.79.227.52:4447 , piece_id=EA3CNFLJUUAEWHRDESNJWD7MA4LDNSXIYEY6363HPDFGG6WP5E6A, seg_pos={0 0}, upload=472569]: Done uploading piece. err: <nil> Dec 28 02:20:13.086035692 (538h51m12.957118737s) [noise=true , address=45.91.116.8:4045 , piece_id=PFARERA2YZ4SAYZKCHHBGWASO5PXSGYI2QYELIJSWLVIHLYC7BWQ, seg_pos={0 0}, upload=472569]: Done uploading piece. err: <nil> Dec 28 02:20:13.198936748 (538h51m13.070019793s) [noise=true , address=145.79.227.47:4266 , piece_id=2JHVUZPYFLQIVJRTJB2QBHQA6E5KF62OVHOFGKIJMY4N6MJLGXFQ, seg_pos={0 0}, upload=472569]: Done uploading piece. err: <nil> Dec 28 02:20:13.209416541 (538h51m13.080499576s) [noise=true , address=45.91.116.11:4120 , piece_id=NRZ6EW5LGJ5KBFMIQTCBYTQR4PPTWU7V2EKNC5HRNVHDD23HXPVQ, seg_pos={0 0}, upload=472569]: Done uploading piece. err: <nil> Dec 28 02:20:13.628338714 (538h51m13.499421749s) [noise=true , address=77.90.31.15:4700 , piece_id=QAUZQP3R4CSW2R4E6I36CYRMCL4PF532UQ3U36DP37CBMRTZK2CA, seg_pos={0 0}, upload=472569]: Done uploading piece. err: <nil> Dec 28 02:20:13.638193536 (538h51m13.509276581s) [noise=true , address=145.79.227.36:4518 , piece_id=NKYCFZIPW7L5ELURKXXWN3ZA6PSM3SSLMFQIFTEKHGE7IBAJAY2Q, seg_pos={0 0}, upload=472569]: Done uploading piece. err: <nil> Dec 28 02:20:13.728729511 (538h51m13.599812556s) [noise=true , address=45.91.116.4:4939 , piece_id=PW6A23TVKXIB4LT7RH2J5JMUSEZFGTJNKOBY2V4MB7OKBA7UAOEA, seg_pos={0 0}, upload=472569]: Done uploading piece. err: <nil> Dec 28 02:20:14.008342538 (538h51m13.879425573s) [noise=true , address=45.91.116.7:4010 , piece_id=XV7ZZEQPAEZ63WSGJC2QHITQA75UX7TCIAVDAKYIJQWLW65QXGHA, seg_pos={0 0}, upload=472569]: Done uploading piece. err: <nil> Dec 28 02:20:14.008362015 (538h51m13.87944505s) [seg_pos={0 0}, upload=472569]: Segment reached optimal threshold of 13 pieces. Dec 28 02:20:14.008538842 (538h51m13.879621887s) [noise=true , address=145.79.227.2:4115 , piece_id=HTCFBKEY5S3DV2VX5M363KEAZSIFJCOBEJ6N34BVAHYZESSRV4NA, seg_pos={0 0}, upload=472569]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:54740->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:54740->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54740->145.79.227.2:4115: use of closed network connection Dec 28 02:20:14.008544492 (538h51m13.879627537s) [noise=true , address=145.79.227.2:4115 , piece_id=HTCFBKEY5S3DV2VX5M363KEAZSIFJCOBEJ6N34BVAHYZESSRV4NA, seg_pos={0 0}, upload=472569]: Long tail cancellation due to optimal threshold Dec 28 02:20:14.008709317 (538h51m13.879792372s) [noise=true , address=145.79.227.34:4907 , piece_id=SD57MI4S5AKIJIKQNYA4BGASGVZA3RTRDQRH22TC45HQCFNW7U6A, seg_pos={0 0}, upload=472569]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 28 02:20:14.008721349 (538h51m13.879804404s) [noise=true , address=145.79.227.34:4907 , piece_id=SD57MI4S5AKIJIKQNYA4BGASGVZA3RTRDQRH22TC45HQCFNW7U6A, seg_pos={0 0}, upload=472569]: Long tail cancellation due to optimal threshold Dec 28 02:20:14.008737530 (538h51m13.879820575s) [noise=true , address=77.90.31.14:4662 , piece_id=MV2B4MXLHGCDUW4TQC2NGLCJ7XZNNK5AFRJYVYJDJDEKTFOSMBMQ, seg_pos={0 0}, upload=472569]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Dec 28 02:20:14.008747279 (538h51m13.879830334s) [noise=true , address=77.90.31.14:4662 , piece_id=MV2B4MXLHGCDUW4TQC2NGLCJ7XZNNK5AFRJYVYJDJDEKTFOSMBMQ, seg_pos={0 0}, upload=472569]: Long tail cancellation due to optimal threshold Dec 28 02:20:14.008842671 (538h51m13.879925716s) [seg_pos={0 0}, upload=472569]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 02:20:14.008911241 (538h51m13.879994276s) [upload=472569]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 02:20:14.008916411 (538h51m13.879999446s) [upload=472569]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 02:20:14.302197630 (538h51m14.173280675s) [upload=472569]: Done uploading segments... Dec 28 02:20:14.302220043 (538h51m14.173303088s) [upload=472569]: Upload finished. err: <nil> Duration: 30.003244599s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................................X..................] duration: 3.478337537s trace: https://jaeger.dev.storj.io/trace/7fc142ad3d711e34 FAIL (2025-12-28 02:20:06 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.47850255s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................X...................X] duration: 1m0.003367954s trace: https://jaeger.dev.storj.io/trace/1f68404b41391e4a FAIL (2025-12-28 02:20:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12CaBSQgwTru9NXaznMJ2XExKe3yhLoUCvDKFvqvLJGJev4mEPc, Piece ID: ULPCWUUHAL35F5WZYHDQLO7VBIGAUH2XHH4I2NYYHG7OAC5VJFOA) context deadline exceeded; piecestore: (Node ID: 1H1XywjaRt5ZRGqmZBgzfSowuCnPqFesL1Ch2jYYS1gV2oBgQX, Piece ID: EMRORAB74LXEB6222NUCRM5ZENJC2AZNK7Q2V2NSQMVD5TIRGVGQ) context deadline exceeded; piecestore: (Node ID: 129YE2Wd6LmDABJFaGVtYDwCHmStVDnPRCagD2cKPWA5PDTtZyh, Piece ID: UEKG3FOTTORJVQRPLAFCJEP2ZAVA5N7M2DFGH4BUTB2OGVC2KIZA) context deadline exceeded ======LOGS====== upload=472552]: Uploading piece... Dec 28 02:19:02.432783751 (538h50m2.303866796s) [noise=true , address=145.79.227.33:4896 , piece_id=D44WQB3G3NBNSB6ITIBHVV67V2EKPGXXONQU25IE6RVSGGTUJ44Q, seg_pos={0 0}, upload=472552]: Uploading piece... Dec 28 02:19:02.432842112 (538h50m2.303925147s) [noise=true , address=77.90.31.21:4900 , piece_id=UEKG3FOTTORJVQRPLAFCJEP2ZAVA5N7M2DFGH4BUTB2OGVC2KIZA, seg_pos={0 0}, upload=472552]: Uploading piece... Dec 28 02:19:02.433283032 (538h50m2.304366077s) [noise=true , address=77.90.31.15:4703 , piece_id=BERLD7G2DBBDBNE3MCS57VXAKXL3P4RQIV47ZOCU6VXMPJU4QITA, seg_pos={0 0}, upload=472552]: Uploading piece... Dec 28 02:19:02.433734562 (538h50m2.304817617s) [noise=true , address=77.90.31.16:4741 , piece_id=CX66NBDSLZWQ34Z2UDYM6HM73TABJSVHSGBQRONAQ3S6ENVTY3KQ, seg_pos={0 0}, upload=472552]: Uploading piece... Dec 28 02:19:02.434178757 (538h50m2.305261802s) [noise=true , address=77.90.31.13:4646 , piece_id=V5HTB4HVJVO7OG745JBRQSISFZU6UOORKLHEVDUDUR24EWRSSVOQ, seg_pos={0 0}, upload=472552]: Uploading piece... Dec 28 02:19:02.434595160 (538h50m2.305678195s) [noise=true , address=145.79.227.37:4596 , piece_id=XBEKEUWGAAAWZGOV3PBH7NE2CX5VID75MFGWMPHTNR7MV66U3XNA, seg_pos={0 0}, upload=472552]: Uploading piece... Dec 28 02:19:02.435217555 (538h50m2.3063006s) [noise=true , address=45.91.116.11:4136 , piece_id=JD6QHJCQ6D53BGKDPSPYBBHNCP7XDZCSV7H6ARD3JNBL75NULT6Q, seg_pos={0 0}, upload=472552]: Uploading piece... Dec 28 02:19:02.435684624 (538h50m2.306767669s) [noise=true , address=145.79.227.32:4856 , piece_id=II5XB6TRHNQAAHBPP5PRMMA7BCBGR5QO4HUBL4YJ7ZLLMK4DJB5Q, seg_pos={0 0}, upload=472552]: Uploading piece... Dec 28 02:19:02.436105054 (538h50m2.307188099s) [noise=true , address=145.79.227.14:4842 , piece_id=3F6EV4TB5TQNK5W5CJHRCCNORCEEAA6AXCOJQTQ6WJNRZG3YILOQ, seg_pos={0 0}, upload=472552]: Uploading piece... Dec 28 02:19:02.436528130 (538h50m2.307611175s) [noise=true , address=77.90.31.12:4610 , piece_id=C7IRMHAHNXDZLKFBL7OVTNVI2KWMFREMFE5GR4USVMNT5QRB3HXQ, seg_pos={0 0}, upload=472552]: Uploading piece... Dec 28 02:19:02.437497977 (538h50m2.308581022s) [noise=true , address=145.79.227.49:4340 , piece_id=5C75E4QGOX6G7QNOUUKX4GC7YL5MESWBOFSNCA6PO4PPSMQKIVFQ, seg_pos={0 0}, upload=472552]: Uploading piece... Dec 28 02:19:02.439092573 (538h50m2.310175618s) [noise=true , address=45.91.116.11:4134 , piece_id=D4PXHXGBZGH3UU2CNKDRCWD44IJQR5BQFWHCNQRPZDNYVV755ICQ, seg_pos={0 0}, upload=472552]: Uploading piece... Dec 28 02:19:02.440580235 (538h50m2.31166328s) [noise=true , address=45.91.116.5:4318 , piece_id=ULPCWUUHAL35F5WZYHDQLO7VBIGAUH2XHH4I2NYYHG7OAC5VJFOA, seg_pos={0 0}, upload=472552]: Uploading piece... Dec 28 02:19:02.442091032 (538h50m2.313174077s) [noise=true , address=77.90.31.11:4558 , piece_id=TWOTPEDKIRIOR7NQNCEFM3Z5KGWVOFUYKIQYMEGHED5FKRDCB54Q, seg_pos={0 0}, upload=472552]: Uploading piece... Dec 28 02:19:02.443305184 (538h50m2.314388229s) [noise=true , address=145.79.227.40:4078 , piece_id=3ICUQQR7NS6O2QYHBFCAQZS7A7UAWHTD3O2LOWLRZ3OCMU3WWBSA, seg_pos={0 0}, upload=472552]: Uploading piece... Dec 28 02:19:02.447829959 (538h50m2.318913014s) [noise=true , address=145.79.227.14:4863 , piece_id=N3XS43U6I4WJYUCC5NVUBRLTTHVPYKDR3KRCNGJRYU3E26WDO3HA, seg_pos={0 0}, upload=472552]: Uploading piece... Dec 28 02:19:02.451586160 (538h50m2.322669225s) [noise=true , address=45.91.116.9:4288 , piece_id=EMRORAB74LXEB6222NUCRM5ZENJC2AZNK7Q2V2NSQMVD5TIRGVGQ, seg_pos={0 0}, upload=472552]: Uploading piece... Dec 28 02:19:03.923272933 (538h50m3.794355978s) [upload=472552]: Next returned nil segment Dec 28 02:19:03.923289645 (538h50m3.79437269s) [upload=472552]: Waiting for error group managing segments... Dec 28 02:19:04.324829202 (538h50m4.195912237s) [noise=true , address=145.79.227.49:4340 , piece_id=5C75E4QGOX6G7QNOUUKX4GC7YL5MESWBOFSNCA6PO4PPSMQKIVFQ, seg_pos={0 0}, upload=472552]: Done uploading piece. err: <nil> Dec 28 02:19:04.335200317 (538h50m4.206283363s) [noise=true , address=145.79.227.14:4842 , piece_id=3F6EV4TB5TQNK5W5CJHRCCNORCEEAA6AXCOJQTQ6WJNRZG3YILOQ, seg_pos={0 0}, upload=472552]: Done uploading piece. err: <nil> Dec 28 02:19:04.343899841 (538h50m4.214982886s) [noise=true , address=145.79.227.14:4863 , piece_id=N3XS43U6I4WJYUCC5NVUBRLTTHVPYKDR3KRCNGJRYU3E26WDO3HA, seg_pos={0 0}, upload=472552]: Done uploading piece. err: <nil> Dec 28 02:19:04.378111328 (538h50m4.249194373s) [noise=true , address=145.79.227.37:4596 , piece_id=XBEKEUWGAAAWZGOV3PBH7NE2CX5VID75MFGWMPHTNR7MV66U3XNA, seg_pos={0 0}, upload=472552]: Done uploading piece. err: <nil> Dec 28 02:19:04.381135876 (538h50m4.252218921s) [noise=true , address=145.79.227.32:4856 , piece_id=II5XB6TRHNQAAHBPP5PRMMA7BCBGR5QO4HUBL4YJ7ZLLMK4DJB5Q, seg_pos={0 0}, upload=472552]: Done uploading piece. err: <nil> Dec 28 02:19:04.476390445 (538h50m4.34747348s) [noise=true , address=145.79.227.6:4290 , piece_id=K4OGXS2MXEFVQVW7HEAZJ6267HWETOUXWJAC77JNZADW76K2XMJA, seg_pos={0 0}, upload=472552]: Done uploading piece. err: <nil> Dec 28 02:19:04.533625848 (538h50m4.404708893s) [noise=true , address=145.79.227.33:4896 , piece_id=D44WQB3G3NBNSB6ITIBHVV67V2EKPGXXONQU25IE6RVSGGTUJ44Q, seg_pos={0 0}, upload=472552]: Done uploading piece. err: <nil> Dec 28 02:19:04.562532124 (538h50m4.433615159s) [noise=true , address=77.90.31.11:4558 , piece_id=TWOTPEDKIRIOR7NQNCEFM3Z5KGWVOFUYKIQYMEGHED5FKRDCB54Q, seg_pos={0 0}, upload=472552]: Done uploading piece. err: <nil> Dec 28 02:19:04.643414509 (538h50m4.514497544s) [noise=true , address=77.90.31.21:4900 , piece_id=UEKG3FOTTORJVQRPLAFCJEP2ZAVA5N7M2DFGH4BUTB2OGVC2KIZA, seg_pos={0 0}, upload=472552]: Done uploading piece. err: <nil> Dec 28 02:19:04.651013327 (538h50m4.522096372s) [noise=true , address=77.90.31.13:4646 , piece_id=V5HTB4HVJVO7OG745JBRQSISFZU6UOORKLHEVDUDUR24EWRSSVOQ, seg_pos={0 0}, upload=472552]: Done uploading piece. err: <nil> Dec 28 02:19:04.776667164 (538h50m4.647750209s) [noise=true , address=45.91.116.11:4136 , piece_id=JD6QHJCQ6D53BGKDPSPYBBHNCP7XDZCSV7H6ARD3JNBL75NULT6Q, seg_pos={0 0}, upload=472552]: Done uploading piece. err: <nil> Dec 28 02:19:04.919205529 (538h50m4.790288574s) [noise=true , address=45.91.116.11:4134 , piece_id=D4PXHXGBZGH3UU2CNKDRCWD44IJQR5BQFWHCNQRPZDNYVV755ICQ, seg_pos={0 0}, upload=472552]: Done uploading piece. err: <nil> Dec 28 02:19:04.927993490 (538h50m4.799076535s) [noise=true , address=45.91.116.9:4288 , piece_id=EMRORAB74LXEB6222NUCRM5ZENJC2AZNK7Q2V2NSQMVD5TIRGVGQ, seg_pos={0 0}, upload=472552]: Done uploading piece. err: <nil> Dec 28 02:19:05.041729347 (538h50m4.912812392s) [noise=true , address=45.91.116.5:4318 , piece_id=ULPCWUUHAL35F5WZYHDQLO7VBIGAUH2XHH4I2NYYHG7OAC5VJFOA, seg_pos={0 0}, upload=472552]: Done uploading piece. err: <nil> Dec 28 02:19:05.041751558 (538h50m4.912834593s) [seg_pos={0 0}, upload=472552]: Segment reached optimal threshold of 14 pieces. Dec 28 02:19:05.041986606 (538h50m4.913069651s) [noise=true , address=77.90.31.12:4610 , piece_id=C7IRMHAHNXDZLKFBL7OVTNVI2KWMFREMFE5GR4USVMNT5QRB3HXQ, seg_pos={0 0}, upload=472552]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi): protocol: expected piece hash; context canceled; EOF Dec 28 02:19:05.041995954 (538h50m4.913078999s) [noise=true , address=77.90.31.12:4610 , piece_id=C7IRMHAHNXDZLKFBL7OVTNVI2KWMFREMFE5GR4USVMNT5QRB3HXQ, seg_pos={0 0}, upload=472552]: Long tail cancellation due to optimal threshold Dec 28 02:19:05.042149837 (538h50m4.913232872s) [noise=true , address=77.90.31.15:4703 , piece_id=BERLD7G2DBBDBNE3MCS57VXAKXL3P4RQIV47ZOCU6VXMPJU4QITA, seg_pos={0 0}, upload=472552]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hMDD5K7WDjvN4ZgcpwpnPFb6X1ncCfyucfFrpTUshd8Ye1B6D): protocol: expected piece hash; context canceled; EOF Dec 28 02:19:05.042158714 (538h50m4.913241759s) [noise=true , address=77.90.31.15:4703 , piece_id=BERLD7G2DBBDBNE3MCS57VXAKXL3P4RQIV47ZOCU6VXMPJU4QITA, seg_pos={0 0}, upload=472552]: Long tail cancellation due to optimal threshold Dec 28 02:19:05.042299752 (538h50m4.913382788s) [noise=true , address=77.90.31.16:4741 , piece_id=CX66NBDSLZWQ34Z2UDYM6HM73TABJSVHSGBQRONAQ3S6ENVTY3KQ, seg_pos={0 0}, upload=472552]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR): protocol: expected piece hash; context canceled; EOF Dec 28 02:19:05.042304702 (538h50m4.913387737s) [noise=true , address=77.90.31.16:4741 , piece_id=CX66NBDSLZWQ34Z2UDYM6HM73TABJSVHSGBQRONAQ3S6ENVTY3KQ, seg_pos={0 0}, upload=472552]: Long tail cancellation due to optimal threshold Dec 28 02:19:05.042396276 (538h50m4.913479352s) [noise=true , address=145.79.227.40:4078 , piece_id=3ICUQQR7NS6O2QYHBFCAQZS7A7UAWHTD3O2LOWLRZ3OCMU3WWBSA, seg_pos={0 0}, upload=472552]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tzgtenV6us5ssD7xWGpZmVUoWEfbFhBNewbK9irHtZPkyDtRd): protocol: expected piece hash; context canceled; EOF Dec 28 02:19:05.042417406 (538h50m4.913500461s) [noise=true , address=145.79.227.40:4078 , piece_id=3ICUQQR7NS6O2QYHBFCAQZS7A7UAWHTD3O2LOWLRZ3OCMU3WWBSA, seg_pos={0 0}, upload=472552]: Long tail cancellation due to optimal threshold Dec 28 02:19:05.042612779 (538h50m4.913695814s) [seg_pos={0 0}, upload=472552]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 02:19:05.042821726 (538h50m4.913904771s) [upload=472552]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 02:19:05.042916818 (538h50m4.913999863s) [upload=472552]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 02:19:05.336925913 (538h50m5.208008958s) [upload=472552]: Done uploading segments... Dec 28 02:19:05.336953856 (538h50m5.208036901s) [upload=472552]: Upload finished. err: <nil> Duration: 1m0.003380327s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 31 out of 60 is failed (threshold=3) [.XXXXX..X.XXX.X.XX.XX.X..X.X.X.X.X..X...XX..X.XXX.XXX.X.....] duration: 6.014814775s trace: https://jaeger.dev.storj.io/trace/1c22d34f4ac03822 FAIL (2025-12-28 02:19:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: GU4X73Q6AFIBRIYAG4ASNF22MDMCKUOKIJCS3TTJT3FKDWLAMIKA) context deadline exceeded; piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: DQ4IHJQYEPBEW63HH3ELSFGDFZUHJ7HL6UHBLGQRHKOHTD7HP2SQ) context deadline exceeded ======LOGS====== Dec 28 02:15:53.759588852 (538h46m53.630671907s) [upload=472506]: Starting upload Dec 28 02:15:53.759803180 (538h46m53.630886235s) [upload=472506]: Done starting upload Dec 28 02:15:53.759860660 (538h46m53.630943705s) [upload=472506]: Uploading segments... Dec 28 02:15:53.760077182 (538h46m53.631160227s) [upload=472506]: Got next segment. Inline: false Dec 28 02:15:53.760126496 (538h46m53.631209541s) [upload=472506]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 02:15:53.760133219 (538h46m53.631216264s) [upload=472506]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 02:15:54.003115733 (538h46m53.874198788s) [seg_pos={0 0}, upload=472506]: Begin upload segment... Dec 28 02:15:54.003896579 (538h46m53.874979644s) [seg_pos={0 0}, upload=472506]: Done begin upload segment. Dec 28 02:15:54.003960000 (538h46m53.875043055s) [noise=true , address=145.79.227.16:4909 , piece_id=KSGRHS6G6JZ7IY23T277NMHN6HGG7DUODYBIXENFS2PSTYNUX37Q, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.003967354 (538h46m53.875050399s) [noise=true , address=145.79.227.52:4445 , piece_id=ZEY2QW4N6ZNGBILKIDQ7YBFSB7U2JVICZJV2W3GQU7D3QACB32FQ, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.004662528 (538h46m53.875745573s) [noise=true , address=145.79.227.31:4797 , piece_id=J5OPI7TBATORMGHB245L3YCWH2I56AVGHVAWJRLALMYDQIW7YAGQ, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.004851216 (538h46m53.875934261s) [noise=true , address=45.91.116.11:4118 , piece_id=LA7UXPHDDM25MJ4IKX4IGJXKZFA37TTXYBAGMA25C3Y57X6YFQ3Q, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.005240899 (538h46m53.876323944s) [noise=true , address=145.79.227.4:4185 , piece_id=SKBIGBFEIKWVJJFKWFM6TO5CYQ6RCM6RLV6AXCM4TKG7TS6TAVXQ, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.005594032 (538h46m53.876677067s) [noise=true , address=77.90.31.16:4735 , piece_id=VD6RDROPI6XQA2TYQUB4ENXFD2B6OCM2YH5AXP3OVGPQ5YBI37BA, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.005750930 (538h46m53.876833975s) [noise=true , address=145.79.227.18:4006 , piece_id=GYNQ6GGPHVJJYRV2RHRL63SFS4UPSONZY6PPPVC4ZTSOK5SVWVLQ, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.006059718 (538h46m53.877142793s) [noise=true , address=45.91.116.14:4190 , piece_id=ZDWBLJMWUZN4ZMK5X6FWN24VEWAFU2KODQJ7URBZBWJHIML4GKKA, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.006277232 (538h46m53.877360277s) [noise=true , address=145.79.227.17:4972 , piece_id=64UKV76NMUYNSLPZA3WLTQ6F7W2XP5Y564DFNQMLBFB3SG6S3IFA, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.006515977 (538h46m53.877599022s) [noise=true , address=45.91.116.6:4973 , piece_id=GU4X73Q6AFIBRIYAG4ASNF22MDMCKUOKIJCS3TTJT3FKDWLAMIKA, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.007002723 (538h46m53.878085789s) [noise=true , address=145.79.227.19:4077 , piece_id=4LCN23MUWAX2HK3J2OFAFLEGUQM4YYTJWZ7IPCN3D6QROORY5O6A, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.007095439 (538h46m53.878178484s) [noise=true , address=77.90.31.14:4661 , piece_id=U2S5KQX5UAKMMMIOZH6CP7SE7PR2DZIKHKX7OXBCJMSZKG4RUMMQ, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.007443692 (538h46m53.878526728s) [noise=true , address=45.91.116.7:4009 , piece_id=DQ4IHJQYEPBEW63HH3ELSFGDFZUHJ7HL6UHBLGQRHKOHTD7HP2SQ, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.007675033 (538h46m53.878758078s) [noise=true , address=145.79.227.32:4836 , piece_id=AZ5NCPW2VX2VSH7XGAUEWC2B25ZEQ2NVQVZZUL6CANSCBWIIEY3A, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.007858693 (538h46m53.878941738s) [noise=true , address=145.79.227.27:4691 , piece_id=MJMBV5LR4RA5ZP7Q7U34DXO2TWJ6LDKH4YMADM3K5CU2KRHMQWBA, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.008204601 (538h46m53.879287637s) [noise=true , address=145.79.227.12:4767 , piece_id=A3L6CAJZWVR4RYOZMKQ5C5UALP6FKW367UGSOCW3YEVLWMYJBBTQ, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:55.745297151 (538h46m55.616380186s) [upload=472506]: Next returned nil segment Dec 28 02:15:55.745351053 (538h46m55.616434108s) [upload=472506]: Waiting for error group managing segments... Dec 28 02:15:56.032845416 (538h46m55.903928451s) [noise=true , address=145.79.227.18:4006 , piece_id=GYNQ6GGPHVJJYRV2RHRL63SFS4UPSONZY6PPPVC4ZTSOK5SVWVLQ, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.202191721 (538h46m56.073274756s) [noise=true , address=145.79.227.19:4077 , piece_id=4LCN23MUWAX2HK3J2OFAFLEGUQM4YYTJWZ7IPCN3D6QROORY5O6A, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.205182144 (538h46m56.076265189s) [noise=true , address=145.79.227.31:4797 , piece_id=J5OPI7TBATORMGHB245L3YCWH2I56AVGHVAWJRLALMYDQIW7YAGQ, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.205433684 (538h46m56.076516719s) [noise=true , address=145.79.227.27:4691 , piece_id=MJMBV5LR4RA5ZP7Q7U34DXO2TWJ6LDKH4YMADM3K5CU2KRHMQWBA, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.214467002 (538h46m56.085550047s) [noise=true , address=145.79.227.16:4909 , piece_id=KSGRHS6G6JZ7IY23T277NMHN6HGG7DUODYBIXENFS2PSTYNUX37Q, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.214875340 (538h46m56.085958375s) [noise=true , address=145.79.227.4:4185 , piece_id=SKBIGBFEIKWVJJFKWFM6TO5CYQ6RCM6RLV6AXCM4TKG7TS6TAVXQ, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.217932621 (538h46m56.089015656s) [noise=true , address=145.79.227.12:4767 , piece_id=A3L6CAJZWVR4RYOZMKQ5C5UALP6FKW367UGSOCW3YEVLWMYJBBTQ, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.306475972 (538h46m56.177559007s) [noise=true , address=45.91.116.6:4973 , piece_id=GU4X73Q6AFIBRIYAG4ASNF22MDMCKUOKIJCS3TTJT3FKDWLAMIKA, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.385055362 (538h46m56.256138408s) [noise=true , address=45.91.116.7:4009 , piece_id=DQ4IHJQYEPBEW63HH3ELSFGDFZUHJ7HL6UHBLGQRHKOHTD7HP2SQ, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.395589739 (538h46m56.266672784s) [noise=true , address=45.91.116.14:4190 , piece_id=ZDWBLJMWUZN4ZMK5X6FWN24VEWAFU2KODQJ7URBZBWJHIML4GKKA, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.395681062 (538h46m56.266764107s) [noise=true , address=145.79.227.52:4445 , piece_id=ZEY2QW4N6ZNGBILKIDQ7YBFSB7U2JVICZJV2W3GQU7D3QACB32FQ, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.403044182 (538h46m56.274127217s) [noise=true , address=145.79.227.32:4836 , piece_id=AZ5NCPW2VX2VSH7XGAUEWC2B25ZEQ2NVQVZZUL6CANSCBWIIEY3A, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:57.110570880 (538h46m56.981653925s) [noise=true , address=45.91.116.11:4118 , piece_id=LA7UXPHDDM25MJ4IKX4IGJXKZFA37TTXYBAGMA25C3Y57X6YFQ3Q, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:57.110788534 (538h46m56.981871569s) [seg_pos={0 0}, upload=472506]: Segment reached optimal threshold of 13 pieces. Dec 28 02:15:57.111013823 (538h46m56.982096858s) [noise=true , address=77.90.31.14:4661 , piece_id=U2S5KQX5UAKMMMIOZH6CP7SE7PR2DZIKHKX7OXBCJMSZKG4RUMMQ, seg_pos={0 0}, upload=472506]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Dec 28 02:15:57.111022289 (538h46m56.982105335s) [noise=true , address=77.90.31.14:4661 , piece_id=U2S5KQX5UAKMMMIOZH6CP7SE7PR2DZIKHKX7OXBCJMSZKG4RUMMQ, seg_pos={0 0}, upload=472506]: Long tail cancellation due to optimal threshold Dec 28 02:15:57.111365041 (538h46m56.982448076s) [noise=true , address=77.90.31.16:4735 , piece_id=VD6RDROPI6XQA2TYQUB4ENXFD2B6OCM2YH5AXP3OVGPQ5YBI37BA, seg_pos={0 0}, upload=472506]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: read tcp 188.166.225.170:47380->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 02:15:57.111372196 (538h46m56.982455241s) [noise=true , address=77.90.31.16:4735 , piece_id=VD6RDROPI6XQA2TYQUB4ENXFD2B6OCM2YH5AXP3OVGPQ5YBI37BA, seg_pos={0 0}, upload=472506]: Long tail cancellation due to optimal threshold Dec 28 02:15:57.111468879 (538h46m56.982551924s) [noise=true , address=145.79.227.17:4972 , piece_id=64UKV76NMUYNSLPZA3WLTQ6F7W2XP5Y564DFNQMLBFB3SG6S3IFA, seg_pos={0 0}, upload=472506]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: expected piece hash; context canceled; EOF Dec 28 02:15:57.111474270 (538h46m56.982557315s) [noise=true , address=145.79.227.17:4972 , piece_id=64UKV76NMUYNSLPZA3WLTQ6F7W2XP5Y564DFNQMLBFB3SG6S3IFA, seg_pos={0 0}, upload=472506]: Long tail cancellation due to optimal threshold Dec 28 02:15:57.111493035 (538h46m56.98257607s) [seg_pos={0 0}, upload=472506]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 02:15:57.111624576 (538h46m56.982707611s) [upload=472506]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 02:15:57.111630667 (538h46m56.982713713s) [upload=472506]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 02:15:57.409023835 (538h46m57.28010687s) [upload=472506]: Done uploading segments... Dec 28 02:15:57.409043784 (538h46m57.280126819s) [upload=472506]: Upload finished. err: <nil> Duration: 6.014947757s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 32 out of 60 is failed (threshold=3) [X.XXXXX..X.XXX.X.XX.XX.X..X.X.X.X.X..X...XX..X.XXX.XXX.X....] duration: 18.547683991s trace: https://jaeger.dev.storj.io/trace/1c22d34f4ac03822 FAIL (2025-12-28 02:19:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: GU4X73Q6AFIBRIYAG4ASNF22MDMCKUOKIJCS3TTJT3FKDWLAMIKA) context deadline exceeded; piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: DQ4IHJQYEPBEW63HH3ELSFGDFZUHJ7HL6UHBLGQRHKOHTD7HP2SQ) context deadline exceeded ======LOGS====== Dec 28 02:15:53.759588852 (538h46m53.630671907s) [upload=472506]: Starting upload Dec 28 02:15:53.759803180 (538h46m53.630886235s) [upload=472506]: Done starting upload Dec 28 02:15:53.759860660 (538h46m53.630943705s) [upload=472506]: Uploading segments... Dec 28 02:15:53.760077182 (538h46m53.631160227s) [upload=472506]: Got next segment. Inline: false Dec 28 02:15:53.760126496 (538h46m53.631209541s) [upload=472506]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 02:15:53.760133219 (538h46m53.631216264s) [upload=472506]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 02:15:54.003115733 (538h46m53.874198788s) [seg_pos={0 0}, upload=472506]: Begin upload segment... Dec 28 02:15:54.003896579 (538h46m53.874979644s) [seg_pos={0 0}, upload=472506]: Done begin upload segment. Dec 28 02:15:54.003960000 (538h46m53.875043055s) [noise=true , address=145.79.227.16:4909 , piece_id=KSGRHS6G6JZ7IY23T277NMHN6HGG7DUODYBIXENFS2PSTYNUX37Q, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.003967354 (538h46m53.875050399s) [noise=true , address=145.79.227.52:4445 , piece_id=ZEY2QW4N6ZNGBILKIDQ7YBFSB7U2JVICZJV2W3GQU7D3QACB32FQ, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.004662528 (538h46m53.875745573s) [noise=true , address=145.79.227.31:4797 , piece_id=J5OPI7TBATORMGHB245L3YCWH2I56AVGHVAWJRLALMYDQIW7YAGQ, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.004851216 (538h46m53.875934261s) [noise=true , address=45.91.116.11:4118 , piece_id=LA7UXPHDDM25MJ4IKX4IGJXKZFA37TTXYBAGMA25C3Y57X6YFQ3Q, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.005240899 (538h46m53.876323944s) [noise=true , address=145.79.227.4:4185 , piece_id=SKBIGBFEIKWVJJFKWFM6TO5CYQ6RCM6RLV6AXCM4TKG7TS6TAVXQ, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.005594032 (538h46m53.876677067s) [noise=true , address=77.90.31.16:4735 , piece_id=VD6RDROPI6XQA2TYQUB4ENXFD2B6OCM2YH5AXP3OVGPQ5YBI37BA, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.005750930 (538h46m53.876833975s) [noise=true , address=145.79.227.18:4006 , piece_id=GYNQ6GGPHVJJYRV2RHRL63SFS4UPSONZY6PPPVC4ZTSOK5SVWVLQ, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.006059718 (538h46m53.877142793s) [noise=true , address=45.91.116.14:4190 , piece_id=ZDWBLJMWUZN4ZMK5X6FWN24VEWAFU2KODQJ7URBZBWJHIML4GKKA, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.006277232 (538h46m53.877360277s) [noise=true , address=145.79.227.17:4972 , piece_id=64UKV76NMUYNSLPZA3WLTQ6F7W2XP5Y564DFNQMLBFB3SG6S3IFA, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.006515977 (538h46m53.877599022s) [noise=true , address=45.91.116.6:4973 , piece_id=GU4X73Q6AFIBRIYAG4ASNF22MDMCKUOKIJCS3TTJT3FKDWLAMIKA, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.007002723 (538h46m53.878085789s) [noise=true , address=145.79.227.19:4077 , piece_id=4LCN23MUWAX2HK3J2OFAFLEGUQM4YYTJWZ7IPCN3D6QROORY5O6A, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.007095439 (538h46m53.878178484s) [noise=true , address=77.90.31.14:4661 , piece_id=U2S5KQX5UAKMMMIOZH6CP7SE7PR2DZIKHKX7OXBCJMSZKG4RUMMQ, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.007443692 (538h46m53.878526728s) [noise=true , address=45.91.116.7:4009 , piece_id=DQ4IHJQYEPBEW63HH3ELSFGDFZUHJ7HL6UHBLGQRHKOHTD7HP2SQ, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.007675033 (538h46m53.878758078s) [noise=true , address=145.79.227.32:4836 , piece_id=AZ5NCPW2VX2VSH7XGAUEWC2B25ZEQ2NVQVZZUL6CANSCBWIIEY3A, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.007858693 (538h46m53.878941738s) [noise=true , address=145.79.227.27:4691 , piece_id=MJMBV5LR4RA5ZP7Q7U34DXO2TWJ6LDKH4YMADM3K5CU2KRHMQWBA, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.008204601 (538h46m53.879287637s) [noise=true , address=145.79.227.12:4767 , piece_id=A3L6CAJZWVR4RYOZMKQ5C5UALP6FKW367UGSOCW3YEVLWMYJBBTQ, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:55.745297151 (538h46m55.616380186s) [upload=472506]: Next returned nil segment Dec 28 02:15:55.745351053 (538h46m55.616434108s) [upload=472506]: Waiting for error group managing segments... Dec 28 02:15:56.032845416 (538h46m55.903928451s) [noise=true , address=145.79.227.18:4006 , piece_id=GYNQ6GGPHVJJYRV2RHRL63SFS4UPSONZY6PPPVC4ZTSOK5SVWVLQ, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.202191721 (538h46m56.073274756s) [noise=true , address=145.79.227.19:4077 , piece_id=4LCN23MUWAX2HK3J2OFAFLEGUQM4YYTJWZ7IPCN3D6QROORY5O6A, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.205182144 (538h46m56.076265189s) [noise=true , address=145.79.227.31:4797 , piece_id=J5OPI7TBATORMGHB245L3YCWH2I56AVGHVAWJRLALMYDQIW7YAGQ, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.205433684 (538h46m56.076516719s) [noise=true , address=145.79.227.27:4691 , piece_id=MJMBV5LR4RA5ZP7Q7U34DXO2TWJ6LDKH4YMADM3K5CU2KRHMQWBA, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.214467002 (538h46m56.085550047s) [noise=true , address=145.79.227.16:4909 , piece_id=KSGRHS6G6JZ7IY23T277NMHN6HGG7DUODYBIXENFS2PSTYNUX37Q, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.214875340 (538h46m56.085958375s) [noise=true , address=145.79.227.4:4185 , piece_id=SKBIGBFEIKWVJJFKWFM6TO5CYQ6RCM6RLV6AXCM4TKG7TS6TAVXQ, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.217932621 (538h46m56.089015656s) [noise=true , address=145.79.227.12:4767 , piece_id=A3L6CAJZWVR4RYOZMKQ5C5UALP6FKW367UGSOCW3YEVLWMYJBBTQ, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.306475972 (538h46m56.177559007s) [noise=true , address=45.91.116.6:4973 , piece_id=GU4X73Q6AFIBRIYAG4ASNF22MDMCKUOKIJCS3TTJT3FKDWLAMIKA, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.385055362 (538h46m56.256138408s) [noise=true , address=45.91.116.7:4009 , piece_id=DQ4IHJQYEPBEW63HH3ELSFGDFZUHJ7HL6UHBLGQRHKOHTD7HP2SQ, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.395589739 (538h46m56.266672784s) [noise=true , address=45.91.116.14:4190 , piece_id=ZDWBLJMWUZN4ZMK5X6FWN24VEWAFU2KODQJ7URBZBWJHIML4GKKA, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.395681062 (538h46m56.266764107s) [noise=true , address=145.79.227.52:4445 , piece_id=ZEY2QW4N6ZNGBILKIDQ7YBFSB7U2JVICZJV2W3GQU7D3QACB32FQ, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.403044182 (538h46m56.274127217s) [noise=true , address=145.79.227.32:4836 , piece_id=AZ5NCPW2VX2VSH7XGAUEWC2B25ZEQ2NVQVZZUL6CANSCBWIIEY3A, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:57.110570880 (538h46m56.981653925s) [noise=true , address=45.91.116.11:4118 , piece_id=LA7UXPHDDM25MJ4IKX4IGJXKZFA37TTXYBAGMA25C3Y57X6YFQ3Q, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:57.110788534 (538h46m56.981871569s) [seg_pos={0 0}, upload=472506]: Segment reached optimal threshold of 13 pieces. Dec 28 02:15:57.111013823 (538h46m56.982096858s) [noise=true , address=77.90.31.14:4661 , piece_id=U2S5KQX5UAKMMMIOZH6CP7SE7PR2DZIKHKX7OXBCJMSZKG4RUMMQ, seg_pos={0 0}, upload=472506]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Dec 28 02:15:57.111022289 (538h46m56.982105335s) [noise=true , address=77.90.31.14:4661 , piece_id=U2S5KQX5UAKMMMIOZH6CP7SE7PR2DZIKHKX7OXBCJMSZKG4RUMMQ, seg_pos={0 0}, upload=472506]: Long tail cancellation due to optimal threshold Dec 28 02:15:57.111365041 (538h46m56.982448076s) [noise=true , address=77.90.31.16:4735 , piece_id=VD6RDROPI6XQA2TYQUB4ENXFD2B6OCM2YH5AXP3OVGPQ5YBI37BA, seg_pos={0 0}, upload=472506]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: read tcp 188.166.225.170:47380->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 02:15:57.111372196 (538h46m56.982455241s) [noise=true , address=77.90.31.16:4735 , piece_id=VD6RDROPI6XQA2TYQUB4ENXFD2B6OCM2YH5AXP3OVGPQ5YBI37BA, seg_pos={0 0}, upload=472506]: Long tail cancellation due to optimal threshold Dec 28 02:15:57.111468879 (538h46m56.982551924s) [noise=true , address=145.79.227.17:4972 , piece_id=64UKV76NMUYNSLPZA3WLTQ6F7W2XP5Y564DFNQMLBFB3SG6S3IFA, seg_pos={0 0}, upload=472506]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: expected piece hash; context canceled; EOF Dec 28 02:15:57.111474270 (538h46m56.982557315s) [noise=true , address=145.79.227.17:4972 , piece_id=64UKV76NMUYNSLPZA3WLTQ6F7W2XP5Y564DFNQMLBFB3SG6S3IFA, seg_pos={0 0}, upload=472506]: Long tail cancellation due to optimal threshold Dec 28 02:15:57.111493035 (538h46m56.98257607s) [seg_pos={0 0}, upload=472506]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 02:15:57.111624576 (538h46m56.982707611s) [upload=472506]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 02:15:57.111630667 (538h46m56.982713713s) [upload=472506]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 02:15:57.409023835 (538h46m57.28010687s) [upload=472506]: Done uploading segments... Dec 28 02:15:57.409043784 (538h46m57.280126819s) [upload=472506]: Upload finished. err: <nil> Duration: 18.547747333s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 43.400881464s trace: https://jaeger.dev.storj.io/trace/12a1d059b6f75af7 FAIL (2025-12-28 02:19:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1.145865281 (538h33m41.016948316s) [noise=true , address=77.90.31.10:4551 , piece_id=KBPO4MUDASHKMPRNJELQWLNMJNQBB3PZUOVNCEPP2CYVHU6X2PPA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.146408725 (538h33m41.01749177s) [noise=true , address=145.79.227.15:4908 , piece_id=I2FSDNVYBFZKRXZNUHH2APY7N2PI7DPAKFKGXRJW2UFU2PBKSSNQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.146778139 (538h33m41.017861184s) [noise=true , address=145.79.227.43:4161 , piece_id=ZN3T6JSPHJ57X4YYG5MHRJYYOFPH2T2CSGJ34MJQU55HDC7CMOXQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.147137874 (538h33m41.018220919s) [noise=true , address=77.90.31.21:4882 , piece_id=WQAGQTGLU5HRKEET36W34MQP4NPGSVCA4QZ6YEGVAJWBBPLZOUQQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.147426784 (538h33m41.018509819s) [noise=true , address=145.79.227.35:4503 , piece_id=MDYJ2TROT4FHLX4XHKHEQACC7ZXHZCQ5MUPJL5EDS3F2YMXSUGUA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.147466109 (538h33m41.018549154s) [noise=true , address=77.90.31.15:4730 , piece_id=EN7IGRJZ2WFN4HXJPMNLLXDUZPIN7KDUGKPPLJH7CSA4SDOUM2JA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.147817368 (538h33m41.018900413s) [noise=true , address=77.90.31.18:4833 , piece_id=3E57LWLI3WHROFGMXVZ7LHDHO5O7453LWQXA3OEKM52B2NIYM2JA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.147860690 (538h33m41.018943745s) [noise=true , address=145.79.227.44:4212 , piece_id=EFTTQM6323MUQ5WTGY6DFBTF3TGRM6XZWXIGT6DM4VS7GRGM4C2Q, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.148163427 (538h33m41.019246472s) [noise=true , address=77.90.31.19:4862 , piece_id=PSBBBSGY7THVLMXJQCADIE7VYZ33OIIDVAEQWG3RWMNA6RSKELNQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.148189276 (538h33m41.019272321s) [noise=true , address=77.90.31.10:4524 , piece_id=X2VCBS4PTPPWI6U64T4FVIULRLCDY3XUOJTXJNP34OHSO54ONIWA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.148496531 (538h33m41.019579566s) [noise=true , address=145.79.227.23:4556 , piece_id=AKBXRSU5PM5QLHGHE3VXTJ3ZH23BECPGBLQSVZK6XVIYJGVDXT4Q, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.148527380 (538h33m41.019610425s) [noise=true , address=145.79.227.33:4899 , piece_id=JFEXXK22EATV7UUH7LLXTFEPIUBW7EWSNYFXLE4AO5F6SLYFTGBA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.148801512 (538h33m41.019884547s) [noise=true , address=77.90.31.14:4672 , piece_id=BZG2Y5X47J2WVFF5LBHEJZNUD2OQ3NOTVPTCAVT4TLFQB5FWESLA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.148894920 (538h33m41.019977965s) [noise=true , address=5.231.73.5:4016 , piece_id=YCNKQEKJQSJHKKGCYWLOLEBKIVPJHZIJ6TYEG6DF4TRXMZ3ZTTIQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.149118806 (538h33m41.020201851s) [noise=true , address=145.79.227.47:4295 , piece_id=OM7B75MCPM7L3WP5MLQ7MAVBZ2NJUSISOWD56RSWI2AV2OWVDRTA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.149274993 (538h33m41.020358038s) [noise=true , address=145.79.227.50:4400 , piece_id=HZAT7YAVZYWK52FE4TIC6OA37U4VX5B35H55WMBO3MMWNIKTJLJQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.149625842 (538h33m41.020708877s) [noise=true , address=145.79.227.34:4915 , piece_id=RMXYZGGJA5EPJLQD6SQG677HFLOQWRY2HFTGEOICYAQG6QR557RQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:42.618098549 (538h33m42.489181584s) [upload=472313]: Next returned nil segment Dec 28 02:02:42.618120911 (538h33m42.489203946s) [upload=472313]: Waiting for error group managing segments... Dec 28 02:02:43.021704832 (538h33m42.892787877s) [noise=true , address=145.79.227.23:4556 , piece_id=AKBXRSU5PM5QLHGHE3VXTJ3ZH23BECPGBLQSVZK6XVIYJGVDXT4Q, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.027980492 (538h33m42.899063537s) [noise=true , address=145.79.227.50:4400 , piece_id=HZAT7YAVZYWK52FE4TIC6OA37U4VX5B35H55WMBO3MMWNIKTJLJQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.043329239 (538h33m42.914412284s) [noise=true , address=145.79.227.15:4908 , piece_id=I2FSDNVYBFZKRXZNUHH2APY7N2PI7DPAKFKGXRJW2UFU2PBKSSNQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.056308314 (538h33m42.927391349s) [noise=true , address=145.79.227.49:4347 , piece_id=WURIHGQASBLSVLZJVM5EJHWD7U4VFLLPGJIFYNOK6ALA5LYEQDSA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.074628196 (538h33m42.945711252s) [noise=true , address=145.79.227.47:4295 , piece_id=OM7B75MCPM7L3WP5MLQ7MAVBZ2NJUSISOWD56RSWI2AV2OWVDRTA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.080495740 (538h33m42.951578775s) [noise=true , address=5.231.73.5:4016 , piece_id=YCNKQEKJQSJHKKGCYWLOLEBKIVPJHZIJ6TYEG6DF4TRXMZ3ZTTIQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.193955176 (538h33m43.065038241s) [noise=true , address=77.90.31.10:4551 , piece_id=KBPO4MUDASHKMPRNJELQWLNMJNQBB3PZUOVNCEPP2CYVHU6X2PPA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.301513865 (538h33m43.1725969s) [noise=true , address=77.90.31.15:4730 , piece_id=EN7IGRJZ2WFN4HXJPMNLLXDUZPIN7KDUGKPPLJH7CSA4SDOUM2JA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.305020622 (538h33m43.176103657s) [noise=true , address=77.90.31.19:4862 , piece_id=PSBBBSGY7THVLMXJQCADIE7VYZ33OIIDVAEQWG3RWMNA6RSKELNQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.312187619 (538h33m43.183270654s) [noise=true , address=77.90.31.14:4672 , piece_id=BZG2Y5X47J2WVFF5LBHEJZNUD2OQ3NOTVPTCAVT4TLFQB5FWESLA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.319277390 (538h33m43.190360425s) [noise=true , address=77.90.31.21:4882 , piece_id=WQAGQTGLU5HRKEET36W34MQP4NPGSVCA4QZ6YEGVAJWBBPLZOUQQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.948916959 (538h33m43.820000004s) [noise=true , address=145.79.227.34:4915 , piece_id=RMXYZGGJA5EPJLQD6SQG677HFLOQWRY2HFTGEOICYAQG6QR557RQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.989823331 (538h33m43.860906366s) [noise=true , address=145.79.227.44:4212 , piece_id=EFTTQM6323MUQ5WTGY6DFBTF3TGRM6XZWXIGT6DM4VS7GRGM4C2Q, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:44.123218949 (538h33m43.994301984s) [noise=true , address=145.79.227.43:4161 , piece_id=ZN3T6JSPHJ57X4YYG5MHRJYYOFPH2T2CSGJ34MJQU55HDC7CMOXQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:44.123238386 (538h33m43.994321421s) [seg_pos={0 0}, upload=472313]: Segment reached optimal threshold of 14 pieces. Dec 28 02:02:44.123431754 (538h33m43.994514789s) [noise=true , address=145.79.227.33:4899 , piece_id=JFEXXK22EATV7UUH7LLXTFEPIUBW7EWSNYFXLE4AO5F6SLYFTGBA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b4pZVQRnjcQNn45VTgdsz5fpjYPMAzSXGR5KJ5BP9XHMkRnRw): protocol: expected piece hash; context canceled; EOF Dec 28 02:02:44.123437586 (538h33m43.994520621s) [noise=true , address=145.79.227.33:4899 , piece_id=JFEXXK22EATV7UUH7LLXTFEPIUBW7EWSNYFXLE4AO5F6SLYFTGBA, seg_pos={0 0}, upload=472313]: Long tail cancellation due to optimal threshold Dec 28 02:02:44.123631064 (538h33m43.994714109s) [noise=true , address=77.90.31.18:4833 , piece_id=3E57LWLI3WHROFGMXVZ7LHDHO5O7453LWQXA3OEKM52B2NIYM2JA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FbVbo8kGy1aaFyCk2noiXNUJHtosFJHFykBMRecGd5Kj5gjEp): protocol: expected piece hash; context canceled; EOF Dec 28 02:02:44.123675267 (538h33m43.994758313s) [noise=true , address=77.90.31.18:4833 , piece_id=3E57LWLI3WHROFGMXVZ7LHDHO5O7453LWQXA3OEKM52B2NIYM2JA, seg_pos={0 0}, upload=472313]: Long tail cancellation due to optimal threshold Dec 28 02:02:44.123774066 (538h33m43.994857101s) [noise=true , address=77.90.31.10:4524 , piece_id=X2VCBS4PTPPWI6U64T4FVIULRLCDY3XUOJTXJNP34OHSO54ONIWA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uN7CznoK4AD8FqFkNUndawdGJ9rmjAe7oNnjJkV844ScY87qe): protocol: expected piece hash; context canceled; EOF Dec 28 02:02:44.123781761 (538h33m43.994864796s) [noise=true , address=77.90.31.10:4524 , piece_id=X2VCBS4PTPPWI6U64T4FVIULRLCDY3XUOJTXJNP34OHSO54ONIWA, seg_pos={0 0}, upload=472313]: Long tail cancellation due to optimal threshold Dec 28 02:02:44.123923341 (538h33m43.995006376s) [noise=true , address=145.79.227.35:4503 , piece_id=MDYJ2TROT4FHLX4XHKHEQACC7ZXHZCQ5MUPJL5EDS3F2YMXSUGUA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ctmSF1aMQasuXByFW8DppDantREmmWL2WUjmKGW5ZULVePyvm): protocol: expected piece hash; context canceled; EOF Dec 28 02:02:44.123928591 (538h33m43.995011636s) [noise=true , address=145.79.227.35:4503 , piece_id=MDYJ2TROT4FHLX4XHKHEQACC7ZXHZCQ5MUPJL5EDS3F2YMXSUGUA, seg_pos={0 0}, upload=472313]: Long tail cancellation due to optimal threshold Dec 28 02:02:44.123944731 (538h33m43.995027766s) [seg_pos={0 0}, upload=472313]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 02:02:44.124017179 (538h33m43.995100214s) [upload=472313]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 02:02:44.124022469 (538h33m43.995105504s) [upload=472313]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 02:02:44.409047274 (538h33m44.280130349s) [upload=472313]: Done uploading segments... Dec 28 02:02:44.409073354 (538h33m44.280156389s) [upload=472313]: Upload finished. err: <nil> Duration: 43.400901101s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................................X.................] duration: 3.629032461s trace: https://jaeger.dev.storj.io/trace/7fc142ad3d711e34 FAIL (2025-12-28 02:18:54 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.629060835s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 51.46678748s trace: https://jaeger.dev.storj.io/trace/12a1d059b6f75af7 FAIL (2025-12-28 02:18:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1.145865281 (538h33m41.016948316s) [noise=true , address=77.90.31.10:4551 , piece_id=KBPO4MUDASHKMPRNJELQWLNMJNQBB3PZUOVNCEPP2CYVHU6X2PPA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.146408725 (538h33m41.01749177s) [noise=true , address=145.79.227.15:4908 , piece_id=I2FSDNVYBFZKRXZNUHH2APY7N2PI7DPAKFKGXRJW2UFU2PBKSSNQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.146778139 (538h33m41.017861184s) [noise=true , address=145.79.227.43:4161 , piece_id=ZN3T6JSPHJ57X4YYG5MHRJYYOFPH2T2CSGJ34MJQU55HDC7CMOXQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.147137874 (538h33m41.018220919s) [noise=true , address=77.90.31.21:4882 , piece_id=WQAGQTGLU5HRKEET36W34MQP4NPGSVCA4QZ6YEGVAJWBBPLZOUQQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.147426784 (538h33m41.018509819s) [noise=true , address=145.79.227.35:4503 , piece_id=MDYJ2TROT4FHLX4XHKHEQACC7ZXHZCQ5MUPJL5EDS3F2YMXSUGUA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.147466109 (538h33m41.018549154s) [noise=true , address=77.90.31.15:4730 , piece_id=EN7IGRJZ2WFN4HXJPMNLLXDUZPIN7KDUGKPPLJH7CSA4SDOUM2JA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.147817368 (538h33m41.018900413s) [noise=true , address=77.90.31.18:4833 , piece_id=3E57LWLI3WHROFGMXVZ7LHDHO5O7453LWQXA3OEKM52B2NIYM2JA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.147860690 (538h33m41.018943745s) [noise=true , address=145.79.227.44:4212 , piece_id=EFTTQM6323MUQ5WTGY6DFBTF3TGRM6XZWXIGT6DM4VS7GRGM4C2Q, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.148163427 (538h33m41.019246472s) [noise=true , address=77.90.31.19:4862 , piece_id=PSBBBSGY7THVLMXJQCADIE7VYZ33OIIDVAEQWG3RWMNA6RSKELNQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.148189276 (538h33m41.019272321s) [noise=true , address=77.90.31.10:4524 , piece_id=X2VCBS4PTPPWI6U64T4FVIULRLCDY3XUOJTXJNP34OHSO54ONIWA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.148496531 (538h33m41.019579566s) [noise=true , address=145.79.227.23:4556 , piece_id=AKBXRSU5PM5QLHGHE3VXTJ3ZH23BECPGBLQSVZK6XVIYJGVDXT4Q, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.148527380 (538h33m41.019610425s) [noise=true , address=145.79.227.33:4899 , piece_id=JFEXXK22EATV7UUH7LLXTFEPIUBW7EWSNYFXLE4AO5F6SLYFTGBA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.148801512 (538h33m41.019884547s) [noise=true , address=77.90.31.14:4672 , piece_id=BZG2Y5X47J2WVFF5LBHEJZNUD2OQ3NOTVPTCAVT4TLFQB5FWESLA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.148894920 (538h33m41.019977965s) [noise=true , address=5.231.73.5:4016 , piece_id=YCNKQEKJQSJHKKGCYWLOLEBKIVPJHZIJ6TYEG6DF4TRXMZ3ZTTIQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.149118806 (538h33m41.020201851s) [noise=true , address=145.79.227.47:4295 , piece_id=OM7B75MCPM7L3WP5MLQ7MAVBZ2NJUSISOWD56RSWI2AV2OWVDRTA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.149274993 (538h33m41.020358038s) [noise=true , address=145.79.227.50:4400 , piece_id=HZAT7YAVZYWK52FE4TIC6OA37U4VX5B35H55WMBO3MMWNIKTJLJQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.149625842 (538h33m41.020708877s) [noise=true , address=145.79.227.34:4915 , piece_id=RMXYZGGJA5EPJLQD6SQG677HFLOQWRY2HFTGEOICYAQG6QR557RQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:42.618098549 (538h33m42.489181584s) [upload=472313]: Next returned nil segment Dec 28 02:02:42.618120911 (538h33m42.489203946s) [upload=472313]: Waiting for error group managing segments... Dec 28 02:02:43.021704832 (538h33m42.892787877s) [noise=true , address=145.79.227.23:4556 , piece_id=AKBXRSU5PM5QLHGHE3VXTJ3ZH23BECPGBLQSVZK6XVIYJGVDXT4Q, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.027980492 (538h33m42.899063537s) [noise=true , address=145.79.227.50:4400 , piece_id=HZAT7YAVZYWK52FE4TIC6OA37U4VX5B35H55WMBO3MMWNIKTJLJQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.043329239 (538h33m42.914412284s) [noise=true , address=145.79.227.15:4908 , piece_id=I2FSDNVYBFZKRXZNUHH2APY7N2PI7DPAKFKGXRJW2UFU2PBKSSNQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.056308314 (538h33m42.927391349s) [noise=true , address=145.79.227.49:4347 , piece_id=WURIHGQASBLSVLZJVM5EJHWD7U4VFLLPGJIFYNOK6ALA5LYEQDSA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.074628196 (538h33m42.945711252s) [noise=true , address=145.79.227.47:4295 , piece_id=OM7B75MCPM7L3WP5MLQ7MAVBZ2NJUSISOWD56RSWI2AV2OWVDRTA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.080495740 (538h33m42.951578775s) [noise=true , address=5.231.73.5:4016 , piece_id=YCNKQEKJQSJHKKGCYWLOLEBKIVPJHZIJ6TYEG6DF4TRXMZ3ZTTIQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.193955176 (538h33m43.065038241s) [noise=true , address=77.90.31.10:4551 , piece_id=KBPO4MUDASHKMPRNJELQWLNMJNQBB3PZUOVNCEPP2CYVHU6X2PPA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.301513865 (538h33m43.1725969s) [noise=true , address=77.90.31.15:4730 , piece_id=EN7IGRJZ2WFN4HXJPMNLLXDUZPIN7KDUGKPPLJH7CSA4SDOUM2JA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.305020622 (538h33m43.176103657s) [noise=true , address=77.90.31.19:4862 , piece_id=PSBBBSGY7THVLMXJQCADIE7VYZ33OIIDVAEQWG3RWMNA6RSKELNQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.312187619 (538h33m43.183270654s) [noise=true , address=77.90.31.14:4672 , piece_id=BZG2Y5X47J2WVFF5LBHEJZNUD2OQ3NOTVPTCAVT4TLFQB5FWESLA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.319277390 (538h33m43.190360425s) [noise=true , address=77.90.31.21:4882 , piece_id=WQAGQTGLU5HRKEET36W34MQP4NPGSVCA4QZ6YEGVAJWBBPLZOUQQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.948916959 (538h33m43.820000004s) [noise=true , address=145.79.227.34:4915 , piece_id=RMXYZGGJA5EPJLQD6SQG677HFLOQWRY2HFTGEOICYAQG6QR557RQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.989823331 (538h33m43.860906366s) [noise=true , address=145.79.227.44:4212 , piece_id=EFTTQM6323MUQ5WTGY6DFBTF3TGRM6XZWXIGT6DM4VS7GRGM4C2Q, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:44.123218949 (538h33m43.994301984s) [noise=true , address=145.79.227.43:4161 , piece_id=ZN3T6JSPHJ57X4YYG5MHRJYYOFPH2T2CSGJ34MJQU55HDC7CMOXQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:44.123238386 (538h33m43.994321421s) [seg_pos={0 0}, upload=472313]: Segment reached optimal threshold of 14 pieces. Dec 28 02:02:44.123431754 (538h33m43.994514789s) [noise=true , address=145.79.227.33:4899 , piece_id=JFEXXK22EATV7UUH7LLXTFEPIUBW7EWSNYFXLE4AO5F6SLYFTGBA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b4pZVQRnjcQNn45VTgdsz5fpjYPMAzSXGR5KJ5BP9XHMkRnRw): protocol: expected piece hash; context canceled; EOF Dec 28 02:02:44.123437586 (538h33m43.994520621s) [noise=true , address=145.79.227.33:4899 , piece_id=JFEXXK22EATV7UUH7LLXTFEPIUBW7EWSNYFXLE4AO5F6SLYFTGBA, seg_pos={0 0}, upload=472313]: Long tail cancellation due to optimal threshold Dec 28 02:02:44.123631064 (538h33m43.994714109s) [noise=true , address=77.90.31.18:4833 , piece_id=3E57LWLI3WHROFGMXVZ7LHDHO5O7453LWQXA3OEKM52B2NIYM2JA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FbVbo8kGy1aaFyCk2noiXNUJHtosFJHFykBMRecGd5Kj5gjEp): protocol: expected piece hash; context canceled; EOF Dec 28 02:02:44.123675267 (538h33m43.994758313s) [noise=true , address=77.90.31.18:4833 , piece_id=3E57LWLI3WHROFGMXVZ7LHDHO5O7453LWQXA3OEKM52B2NIYM2JA, seg_pos={0 0}, upload=472313]: Long tail cancellation due to optimal threshold Dec 28 02:02:44.123774066 (538h33m43.994857101s) [noise=true , address=77.90.31.10:4524 , piece_id=X2VCBS4PTPPWI6U64T4FVIULRLCDY3XUOJTXJNP34OHSO54ONIWA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uN7CznoK4AD8FqFkNUndawdGJ9rmjAe7oNnjJkV844ScY87qe): protocol: expected piece hash; context canceled; EOF Dec 28 02:02:44.123781761 (538h33m43.994864796s) [noise=true , address=77.90.31.10:4524 , piece_id=X2VCBS4PTPPWI6U64T4FVIULRLCDY3XUOJTXJNP34OHSO54ONIWA, seg_pos={0 0}, upload=472313]: Long tail cancellation due to optimal threshold Dec 28 02:02:44.123923341 (538h33m43.995006376s) [noise=true , address=145.79.227.35:4503 , piece_id=MDYJ2TROT4FHLX4XHKHEQACC7ZXHZCQ5MUPJL5EDS3F2YMXSUGUA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ctmSF1aMQasuXByFW8DppDantREmmWL2WUjmKGW5ZULVePyvm): protocol: expected piece hash; context canceled; EOF Dec 28 02:02:44.123928591 (538h33m43.995011636s) [noise=true , address=145.79.227.35:4503 , piece_id=MDYJ2TROT4FHLX4XHKHEQACC7ZXHZCQ5MUPJL5EDS3F2YMXSUGUA, seg_pos={0 0}, upload=472313]: Long tail cancellation due to optimal threshold Dec 28 02:02:44.123944731 (538h33m43.995027766s) [seg_pos={0 0}, upload=472313]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 02:02:44.124017179 (538h33m43.995100214s) [upload=472313]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 02:02:44.124022469 (538h33m43.995105504s) [upload=472313]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 02:02:44.409047274 (538h33m44.280130349s) [upload=472313]: Done uploading segments... Dec 28 02:02:44.409073354 (538h33m44.280156389s) [upload=472313]: Upload finished. err: <nil> Duration: 51.466814892s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 33 out of 60 is failed (threshold=3) [XX.XXXXX..X.XXX.X.XX.XX.X..X.X.X.X.X..X...XX..X.XXX.XXX.X...] duration: 8.004482162s trace: https://jaeger.dev.storj.io/trace/1c22d34f4ac03822 FAIL (2025-12-28 02:18:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: GU4X73Q6AFIBRIYAG4ASNF22MDMCKUOKIJCS3TTJT3FKDWLAMIKA) context deadline exceeded; piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: DQ4IHJQYEPBEW63HH3ELSFGDFZUHJ7HL6UHBLGQRHKOHTD7HP2SQ) context deadline exceeded ======LOGS====== Dec 28 02:15:53.759588852 (538h46m53.630671907s) [upload=472506]: Starting upload Dec 28 02:15:53.759803180 (538h46m53.630886235s) [upload=472506]: Done starting upload Dec 28 02:15:53.759860660 (538h46m53.630943705s) [upload=472506]: Uploading segments... Dec 28 02:15:53.760077182 (538h46m53.631160227s) [upload=472506]: Got next segment. Inline: false Dec 28 02:15:53.760126496 (538h46m53.631209541s) [upload=472506]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 02:15:53.760133219 (538h46m53.631216264s) [upload=472506]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 02:15:54.003115733 (538h46m53.874198788s) [seg_pos={0 0}, upload=472506]: Begin upload segment... Dec 28 02:15:54.003896579 (538h46m53.874979644s) [seg_pos={0 0}, upload=472506]: Done begin upload segment. Dec 28 02:15:54.003960000 (538h46m53.875043055s) [noise=true , address=145.79.227.16:4909 , piece_id=KSGRHS6G6JZ7IY23T277NMHN6HGG7DUODYBIXENFS2PSTYNUX37Q, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.003967354 (538h46m53.875050399s) [noise=true , address=145.79.227.52:4445 , piece_id=ZEY2QW4N6ZNGBILKIDQ7YBFSB7U2JVICZJV2W3GQU7D3QACB32FQ, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.004662528 (538h46m53.875745573s) [noise=true , address=145.79.227.31:4797 , piece_id=J5OPI7TBATORMGHB245L3YCWH2I56AVGHVAWJRLALMYDQIW7YAGQ, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.004851216 (538h46m53.875934261s) [noise=true , address=45.91.116.11:4118 , piece_id=LA7UXPHDDM25MJ4IKX4IGJXKZFA37TTXYBAGMA25C3Y57X6YFQ3Q, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.005240899 (538h46m53.876323944s) [noise=true , address=145.79.227.4:4185 , piece_id=SKBIGBFEIKWVJJFKWFM6TO5CYQ6RCM6RLV6AXCM4TKG7TS6TAVXQ, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.005594032 (538h46m53.876677067s) [noise=true , address=77.90.31.16:4735 , piece_id=VD6RDROPI6XQA2TYQUB4ENXFD2B6OCM2YH5AXP3OVGPQ5YBI37BA, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.005750930 (538h46m53.876833975s) [noise=true , address=145.79.227.18:4006 , piece_id=GYNQ6GGPHVJJYRV2RHRL63SFS4UPSONZY6PPPVC4ZTSOK5SVWVLQ, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.006059718 (538h46m53.877142793s) [noise=true , address=45.91.116.14:4190 , piece_id=ZDWBLJMWUZN4ZMK5X6FWN24VEWAFU2KODQJ7URBZBWJHIML4GKKA, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.006277232 (538h46m53.877360277s) [noise=true , address=145.79.227.17:4972 , piece_id=64UKV76NMUYNSLPZA3WLTQ6F7W2XP5Y564DFNQMLBFB3SG6S3IFA, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.006515977 (538h46m53.877599022s) [noise=true , address=45.91.116.6:4973 , piece_id=GU4X73Q6AFIBRIYAG4ASNF22MDMCKUOKIJCS3TTJT3FKDWLAMIKA, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.007002723 (538h46m53.878085789s) [noise=true , address=145.79.227.19:4077 , piece_id=4LCN23MUWAX2HK3J2OFAFLEGUQM4YYTJWZ7IPCN3D6QROORY5O6A, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.007095439 (538h46m53.878178484s) [noise=true , address=77.90.31.14:4661 , piece_id=U2S5KQX5UAKMMMIOZH6CP7SE7PR2DZIKHKX7OXBCJMSZKG4RUMMQ, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.007443692 (538h46m53.878526728s) [noise=true , address=45.91.116.7:4009 , piece_id=DQ4IHJQYEPBEW63HH3ELSFGDFZUHJ7HL6UHBLGQRHKOHTD7HP2SQ, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.007675033 (538h46m53.878758078s) [noise=true , address=145.79.227.32:4836 , piece_id=AZ5NCPW2VX2VSH7XGAUEWC2B25ZEQ2NVQVZZUL6CANSCBWIIEY3A, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.007858693 (538h46m53.878941738s) [noise=true , address=145.79.227.27:4691 , piece_id=MJMBV5LR4RA5ZP7Q7U34DXO2TWJ6LDKH4YMADM3K5CU2KRHMQWBA, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.008204601 (538h46m53.879287637s) [noise=true , address=145.79.227.12:4767 , piece_id=A3L6CAJZWVR4RYOZMKQ5C5UALP6FKW367UGSOCW3YEVLWMYJBBTQ, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:55.745297151 (538h46m55.616380186s) [upload=472506]: Next returned nil segment Dec 28 02:15:55.745351053 (538h46m55.616434108s) [upload=472506]: Waiting for error group managing segments... Dec 28 02:15:56.032845416 (538h46m55.903928451s) [noise=true , address=145.79.227.18:4006 , piece_id=GYNQ6GGPHVJJYRV2RHRL63SFS4UPSONZY6PPPVC4ZTSOK5SVWVLQ, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.202191721 (538h46m56.073274756s) [noise=true , address=145.79.227.19:4077 , piece_id=4LCN23MUWAX2HK3J2OFAFLEGUQM4YYTJWZ7IPCN3D6QROORY5O6A, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.205182144 (538h46m56.076265189s) [noise=true , address=145.79.227.31:4797 , piece_id=J5OPI7TBATORMGHB245L3YCWH2I56AVGHVAWJRLALMYDQIW7YAGQ, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.205433684 (538h46m56.076516719s) [noise=true , address=145.79.227.27:4691 , piece_id=MJMBV5LR4RA5ZP7Q7U34DXO2TWJ6LDKH4YMADM3K5CU2KRHMQWBA, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.214467002 (538h46m56.085550047s) [noise=true , address=145.79.227.16:4909 , piece_id=KSGRHS6G6JZ7IY23T277NMHN6HGG7DUODYBIXENFS2PSTYNUX37Q, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.214875340 (538h46m56.085958375s) [noise=true , address=145.79.227.4:4185 , piece_id=SKBIGBFEIKWVJJFKWFM6TO5CYQ6RCM6RLV6AXCM4TKG7TS6TAVXQ, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.217932621 (538h46m56.089015656s) [noise=true , address=145.79.227.12:4767 , piece_id=A3L6CAJZWVR4RYOZMKQ5C5UALP6FKW367UGSOCW3YEVLWMYJBBTQ, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.306475972 (538h46m56.177559007s) [noise=true , address=45.91.116.6:4973 , piece_id=GU4X73Q6AFIBRIYAG4ASNF22MDMCKUOKIJCS3TTJT3FKDWLAMIKA, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.385055362 (538h46m56.256138408s) [noise=true , address=45.91.116.7:4009 , piece_id=DQ4IHJQYEPBEW63HH3ELSFGDFZUHJ7HL6UHBLGQRHKOHTD7HP2SQ, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.395589739 (538h46m56.266672784s) [noise=true , address=45.91.116.14:4190 , piece_id=ZDWBLJMWUZN4ZMK5X6FWN24VEWAFU2KODQJ7URBZBWJHIML4GKKA, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.395681062 (538h46m56.266764107s) [noise=true , address=145.79.227.52:4445 , piece_id=ZEY2QW4N6ZNGBILKIDQ7YBFSB7U2JVICZJV2W3GQU7D3QACB32FQ, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.403044182 (538h46m56.274127217s) [noise=true , address=145.79.227.32:4836 , piece_id=AZ5NCPW2VX2VSH7XGAUEWC2B25ZEQ2NVQVZZUL6CANSCBWIIEY3A, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:57.110570880 (538h46m56.981653925s) [noise=true , address=45.91.116.11:4118 , piece_id=LA7UXPHDDM25MJ4IKX4IGJXKZFA37TTXYBAGMA25C3Y57X6YFQ3Q, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:57.110788534 (538h46m56.981871569s) [seg_pos={0 0}, upload=472506]: Segment reached optimal threshold of 13 pieces. Dec 28 02:15:57.111013823 (538h46m56.982096858s) [noise=true , address=77.90.31.14:4661 , piece_id=U2S5KQX5UAKMMMIOZH6CP7SE7PR2DZIKHKX7OXBCJMSZKG4RUMMQ, seg_pos={0 0}, upload=472506]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Dec 28 02:15:57.111022289 (538h46m56.982105335s) [noise=true , address=77.90.31.14:4661 , piece_id=U2S5KQX5UAKMMMIOZH6CP7SE7PR2DZIKHKX7OXBCJMSZKG4RUMMQ, seg_pos={0 0}, upload=472506]: Long tail cancellation due to optimal threshold Dec 28 02:15:57.111365041 (538h46m56.982448076s) [noise=true , address=77.90.31.16:4735 , piece_id=VD6RDROPI6XQA2TYQUB4ENXFD2B6OCM2YH5AXP3OVGPQ5YBI37BA, seg_pos={0 0}, upload=472506]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: read tcp 188.166.225.170:47380->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 02:15:57.111372196 (538h46m56.982455241s) [noise=true , address=77.90.31.16:4735 , piece_id=VD6RDROPI6XQA2TYQUB4ENXFD2B6OCM2YH5AXP3OVGPQ5YBI37BA, seg_pos={0 0}, upload=472506]: Long tail cancellation due to optimal threshold Dec 28 02:15:57.111468879 (538h46m56.982551924s) [noise=true , address=145.79.227.17:4972 , piece_id=64UKV76NMUYNSLPZA3WLTQ6F7W2XP5Y564DFNQMLBFB3SG6S3IFA, seg_pos={0 0}, upload=472506]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: expected piece hash; context canceled; EOF Dec 28 02:15:57.111474270 (538h46m56.982557315s) [noise=true , address=145.79.227.17:4972 , piece_id=64UKV76NMUYNSLPZA3WLTQ6F7W2XP5Y564DFNQMLBFB3SG6S3IFA, seg_pos={0 0}, upload=472506]: Long tail cancellation due to optimal threshold Dec 28 02:15:57.111493035 (538h46m56.98257607s) [seg_pos={0 0}, upload=472506]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 02:15:57.111624576 (538h46m56.982707611s) [upload=472506]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 02:15:57.111630667 (538h46m56.982713713s) [upload=472506]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 02:15:57.409023835 (538h46m57.28010687s) [upload=472506]: Done uploading segments... Dec 28 02:15:57.409043784 (538h46m57.280126819s) [upload=472506]: Upload finished. err: <nil> Duration: 8.004507841s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................................X................] duration: 3.773630435s trace: https://jaeger.dev.storj.io/trace/7fc142ad3d711e34 FAIL (2025-12-28 02:18:02 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.773667907s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 33 out of 60 is failed (threshold=3) [.XX.XXXXX..X.XXX.X.XX.XX.X..X.X.X.X.X..X...XX..X.XXX.XXX.X..] duration: 6.152997356s trace: https://jaeger.dev.storj.io/trace/1c22d34f4ac03822 FAIL (2025-12-28 02:17:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: GU4X73Q6AFIBRIYAG4ASNF22MDMCKUOKIJCS3TTJT3FKDWLAMIKA) context deadline exceeded; piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: DQ4IHJQYEPBEW63HH3ELSFGDFZUHJ7HL6UHBLGQRHKOHTD7HP2SQ) context deadline exceeded ======LOGS====== Dec 28 02:15:53.759588852 (538h46m53.630671907s) [upload=472506]: Starting upload Dec 28 02:15:53.759803180 (538h46m53.630886235s) [upload=472506]: Done starting upload Dec 28 02:15:53.759860660 (538h46m53.630943705s) [upload=472506]: Uploading segments... Dec 28 02:15:53.760077182 (538h46m53.631160227s) [upload=472506]: Got next segment. Inline: false Dec 28 02:15:53.760126496 (538h46m53.631209541s) [upload=472506]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 02:15:53.760133219 (538h46m53.631216264s) [upload=472506]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 02:15:54.003115733 (538h46m53.874198788s) [seg_pos={0 0}, upload=472506]: Begin upload segment... Dec 28 02:15:54.003896579 (538h46m53.874979644s) [seg_pos={0 0}, upload=472506]: Done begin upload segment. Dec 28 02:15:54.003960000 (538h46m53.875043055s) [noise=true , address=145.79.227.16:4909 , piece_id=KSGRHS6G6JZ7IY23T277NMHN6HGG7DUODYBIXENFS2PSTYNUX37Q, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.003967354 (538h46m53.875050399s) [noise=true , address=145.79.227.52:4445 , piece_id=ZEY2QW4N6ZNGBILKIDQ7YBFSB7U2JVICZJV2W3GQU7D3QACB32FQ, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.004662528 (538h46m53.875745573s) [noise=true , address=145.79.227.31:4797 , piece_id=J5OPI7TBATORMGHB245L3YCWH2I56AVGHVAWJRLALMYDQIW7YAGQ, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.004851216 (538h46m53.875934261s) [noise=true , address=45.91.116.11:4118 , piece_id=LA7UXPHDDM25MJ4IKX4IGJXKZFA37TTXYBAGMA25C3Y57X6YFQ3Q, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.005240899 (538h46m53.876323944s) [noise=true , address=145.79.227.4:4185 , piece_id=SKBIGBFEIKWVJJFKWFM6TO5CYQ6RCM6RLV6AXCM4TKG7TS6TAVXQ, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.005594032 (538h46m53.876677067s) [noise=true , address=77.90.31.16:4735 , piece_id=VD6RDROPI6XQA2TYQUB4ENXFD2B6OCM2YH5AXP3OVGPQ5YBI37BA, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.005750930 (538h46m53.876833975s) [noise=true , address=145.79.227.18:4006 , piece_id=GYNQ6GGPHVJJYRV2RHRL63SFS4UPSONZY6PPPVC4ZTSOK5SVWVLQ, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.006059718 (538h46m53.877142793s) [noise=true , address=45.91.116.14:4190 , piece_id=ZDWBLJMWUZN4ZMK5X6FWN24VEWAFU2KODQJ7URBZBWJHIML4GKKA, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.006277232 (538h46m53.877360277s) [noise=true , address=145.79.227.17:4972 , piece_id=64UKV76NMUYNSLPZA3WLTQ6F7W2XP5Y564DFNQMLBFB3SG6S3IFA, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.006515977 (538h46m53.877599022s) [noise=true , address=45.91.116.6:4973 , piece_id=GU4X73Q6AFIBRIYAG4ASNF22MDMCKUOKIJCS3TTJT3FKDWLAMIKA, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.007002723 (538h46m53.878085789s) [noise=true , address=145.79.227.19:4077 , piece_id=4LCN23MUWAX2HK3J2OFAFLEGUQM4YYTJWZ7IPCN3D6QROORY5O6A, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.007095439 (538h46m53.878178484s) [noise=true , address=77.90.31.14:4661 , piece_id=U2S5KQX5UAKMMMIOZH6CP7SE7PR2DZIKHKX7OXBCJMSZKG4RUMMQ, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.007443692 (538h46m53.878526728s) [noise=true , address=45.91.116.7:4009 , piece_id=DQ4IHJQYEPBEW63HH3ELSFGDFZUHJ7HL6UHBLGQRHKOHTD7HP2SQ, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.007675033 (538h46m53.878758078s) [noise=true , address=145.79.227.32:4836 , piece_id=AZ5NCPW2VX2VSH7XGAUEWC2B25ZEQ2NVQVZZUL6CANSCBWIIEY3A, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.007858693 (538h46m53.878941738s) [noise=true , address=145.79.227.27:4691 , piece_id=MJMBV5LR4RA5ZP7Q7U34DXO2TWJ6LDKH4YMADM3K5CU2KRHMQWBA, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.008204601 (538h46m53.879287637s) [noise=true , address=145.79.227.12:4767 , piece_id=A3L6CAJZWVR4RYOZMKQ5C5UALP6FKW367UGSOCW3YEVLWMYJBBTQ, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:55.745297151 (538h46m55.616380186s) [upload=472506]: Next returned nil segment Dec 28 02:15:55.745351053 (538h46m55.616434108s) [upload=472506]: Waiting for error group managing segments... Dec 28 02:15:56.032845416 (538h46m55.903928451s) [noise=true , address=145.79.227.18:4006 , piece_id=GYNQ6GGPHVJJYRV2RHRL63SFS4UPSONZY6PPPVC4ZTSOK5SVWVLQ, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.202191721 (538h46m56.073274756s) [noise=true , address=145.79.227.19:4077 , piece_id=4LCN23MUWAX2HK3J2OFAFLEGUQM4YYTJWZ7IPCN3D6QROORY5O6A, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.205182144 (538h46m56.076265189s) [noise=true , address=145.79.227.31:4797 , piece_id=J5OPI7TBATORMGHB245L3YCWH2I56AVGHVAWJRLALMYDQIW7YAGQ, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.205433684 (538h46m56.076516719s) [noise=true , address=145.79.227.27:4691 , piece_id=MJMBV5LR4RA5ZP7Q7U34DXO2TWJ6LDKH4YMADM3K5CU2KRHMQWBA, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.214467002 (538h46m56.085550047s) [noise=true , address=145.79.227.16:4909 , piece_id=KSGRHS6G6JZ7IY23T277NMHN6HGG7DUODYBIXENFS2PSTYNUX37Q, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.214875340 (538h46m56.085958375s) [noise=true , address=145.79.227.4:4185 , piece_id=SKBIGBFEIKWVJJFKWFM6TO5CYQ6RCM6RLV6AXCM4TKG7TS6TAVXQ, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.217932621 (538h46m56.089015656s) [noise=true , address=145.79.227.12:4767 , piece_id=A3L6CAJZWVR4RYOZMKQ5C5UALP6FKW367UGSOCW3YEVLWMYJBBTQ, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.306475972 (538h46m56.177559007s) [noise=true , address=45.91.116.6:4973 , piece_id=GU4X73Q6AFIBRIYAG4ASNF22MDMCKUOKIJCS3TTJT3FKDWLAMIKA, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.385055362 (538h46m56.256138408s) [noise=true , address=45.91.116.7:4009 , piece_id=DQ4IHJQYEPBEW63HH3ELSFGDFZUHJ7HL6UHBLGQRHKOHTD7HP2SQ, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.395589739 (538h46m56.266672784s) [noise=true , address=45.91.116.14:4190 , piece_id=ZDWBLJMWUZN4ZMK5X6FWN24VEWAFU2KODQJ7URBZBWJHIML4GKKA, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.395681062 (538h46m56.266764107s) [noise=true , address=145.79.227.52:4445 , piece_id=ZEY2QW4N6ZNGBILKIDQ7YBFSB7U2JVICZJV2W3GQU7D3QACB32FQ, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.403044182 (538h46m56.274127217s) [noise=true , address=145.79.227.32:4836 , piece_id=AZ5NCPW2VX2VSH7XGAUEWC2B25ZEQ2NVQVZZUL6CANSCBWIIEY3A, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:57.110570880 (538h46m56.981653925s) [noise=true , address=45.91.116.11:4118 , piece_id=LA7UXPHDDM25MJ4IKX4IGJXKZFA37TTXYBAGMA25C3Y57X6YFQ3Q, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:57.110788534 (538h46m56.981871569s) [seg_pos={0 0}, upload=472506]: Segment reached optimal threshold of 13 pieces. Dec 28 02:15:57.111013823 (538h46m56.982096858s) [noise=true , address=77.90.31.14:4661 , piece_id=U2S5KQX5UAKMMMIOZH6CP7SE7PR2DZIKHKX7OXBCJMSZKG4RUMMQ, seg_pos={0 0}, upload=472506]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Dec 28 02:15:57.111022289 (538h46m56.982105335s) [noise=true , address=77.90.31.14:4661 , piece_id=U2S5KQX5UAKMMMIOZH6CP7SE7PR2DZIKHKX7OXBCJMSZKG4RUMMQ, seg_pos={0 0}, upload=472506]: Long tail cancellation due to optimal threshold Dec 28 02:15:57.111365041 (538h46m56.982448076s) [noise=true , address=77.90.31.16:4735 , piece_id=VD6RDROPI6XQA2TYQUB4ENXFD2B6OCM2YH5AXP3OVGPQ5YBI37BA, seg_pos={0 0}, upload=472506]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: read tcp 188.166.225.170:47380->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 02:15:57.111372196 (538h46m56.982455241s) [noise=true , address=77.90.31.16:4735 , piece_id=VD6RDROPI6XQA2TYQUB4ENXFD2B6OCM2YH5AXP3OVGPQ5YBI37BA, seg_pos={0 0}, upload=472506]: Long tail cancellation due to optimal threshold Dec 28 02:15:57.111468879 (538h46m56.982551924s) [noise=true , address=145.79.227.17:4972 , piece_id=64UKV76NMUYNSLPZA3WLTQ6F7W2XP5Y564DFNQMLBFB3SG6S3IFA, seg_pos={0 0}, upload=472506]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: expected piece hash; context canceled; EOF Dec 28 02:15:57.111474270 (538h46m56.982557315s) [noise=true , address=145.79.227.17:4972 , piece_id=64UKV76NMUYNSLPZA3WLTQ6F7W2XP5Y564DFNQMLBFB3SG6S3IFA, seg_pos={0 0}, upload=472506]: Long tail cancellation due to optimal threshold Dec 28 02:15:57.111493035 (538h46m56.98257607s) [seg_pos={0 0}, upload=472506]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 02:15:57.111624576 (538h46m56.982707611s) [upload=472506]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 02:15:57.111630667 (538h46m56.982713713s) [upload=472506]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 02:15:57.409023835 (538h46m57.28010687s) [upload=472506]: Done uploading segments... Dec 28 02:15:57.409043784 (538h46m57.280126819s) [upload=472506]: Upload finished. err: <nil> Duration: 6.153021833s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............................................X...............] duration: 4.299293537s trace: https://jaeger.dev.storj.io/trace/7fc142ad3d711e34 FAIL (2025-12-28 02:17:12 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.299320498s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 34 out of 60 is failed (threshold=3) [X.XX.XXXXX..X.XXX.X.XX.XX.X..X.X.X.X.X..X...XX..X.XXX.XXX.X.] duration: 9.662157017s trace: https://jaeger.dev.storj.io/trace/1c22d34f4ac03822 FAIL (2025-12-28 02:17:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: GU4X73Q6AFIBRIYAG4ASNF22MDMCKUOKIJCS3TTJT3FKDWLAMIKA) context deadline exceeded; piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: DQ4IHJQYEPBEW63HH3ELSFGDFZUHJ7HL6UHBLGQRHKOHTD7HP2SQ) context deadline exceeded ======LOGS====== Dec 28 02:15:53.759588852 (538h46m53.630671907s) [upload=472506]: Starting upload Dec 28 02:15:53.759803180 (538h46m53.630886235s) [upload=472506]: Done starting upload Dec 28 02:15:53.759860660 (538h46m53.630943705s) [upload=472506]: Uploading segments... Dec 28 02:15:53.760077182 (538h46m53.631160227s) [upload=472506]: Got next segment. Inline: false Dec 28 02:15:53.760126496 (538h46m53.631209541s) [upload=472506]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 02:15:53.760133219 (538h46m53.631216264s) [upload=472506]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 02:15:54.003115733 (538h46m53.874198788s) [seg_pos={0 0}, upload=472506]: Begin upload segment... Dec 28 02:15:54.003896579 (538h46m53.874979644s) [seg_pos={0 0}, upload=472506]: Done begin upload segment. Dec 28 02:15:54.003960000 (538h46m53.875043055s) [noise=true , address=145.79.227.16:4909 , piece_id=KSGRHS6G6JZ7IY23T277NMHN6HGG7DUODYBIXENFS2PSTYNUX37Q, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.003967354 (538h46m53.875050399s) [noise=true , address=145.79.227.52:4445 , piece_id=ZEY2QW4N6ZNGBILKIDQ7YBFSB7U2JVICZJV2W3GQU7D3QACB32FQ, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.004662528 (538h46m53.875745573s) [noise=true , address=145.79.227.31:4797 , piece_id=J5OPI7TBATORMGHB245L3YCWH2I56AVGHVAWJRLALMYDQIW7YAGQ, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.004851216 (538h46m53.875934261s) [noise=true , address=45.91.116.11:4118 , piece_id=LA7UXPHDDM25MJ4IKX4IGJXKZFA37TTXYBAGMA25C3Y57X6YFQ3Q, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.005240899 (538h46m53.876323944s) [noise=true , address=145.79.227.4:4185 , piece_id=SKBIGBFEIKWVJJFKWFM6TO5CYQ6RCM6RLV6AXCM4TKG7TS6TAVXQ, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.005594032 (538h46m53.876677067s) [noise=true , address=77.90.31.16:4735 , piece_id=VD6RDROPI6XQA2TYQUB4ENXFD2B6OCM2YH5AXP3OVGPQ5YBI37BA, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.005750930 (538h46m53.876833975s) [noise=true , address=145.79.227.18:4006 , piece_id=GYNQ6GGPHVJJYRV2RHRL63SFS4UPSONZY6PPPVC4ZTSOK5SVWVLQ, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.006059718 (538h46m53.877142793s) [noise=true , address=45.91.116.14:4190 , piece_id=ZDWBLJMWUZN4ZMK5X6FWN24VEWAFU2KODQJ7URBZBWJHIML4GKKA, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.006277232 (538h46m53.877360277s) [noise=true , address=145.79.227.17:4972 , piece_id=64UKV76NMUYNSLPZA3WLTQ6F7W2XP5Y564DFNQMLBFB3SG6S3IFA, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.006515977 (538h46m53.877599022s) [noise=true , address=45.91.116.6:4973 , piece_id=GU4X73Q6AFIBRIYAG4ASNF22MDMCKUOKIJCS3TTJT3FKDWLAMIKA, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.007002723 (538h46m53.878085789s) [noise=true , address=145.79.227.19:4077 , piece_id=4LCN23MUWAX2HK3J2OFAFLEGUQM4YYTJWZ7IPCN3D6QROORY5O6A, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.007095439 (538h46m53.878178484s) [noise=true , address=77.90.31.14:4661 , piece_id=U2S5KQX5UAKMMMIOZH6CP7SE7PR2DZIKHKX7OXBCJMSZKG4RUMMQ, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.007443692 (538h46m53.878526728s) [noise=true , address=45.91.116.7:4009 , piece_id=DQ4IHJQYEPBEW63HH3ELSFGDFZUHJ7HL6UHBLGQRHKOHTD7HP2SQ, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.007675033 (538h46m53.878758078s) [noise=true , address=145.79.227.32:4836 , piece_id=AZ5NCPW2VX2VSH7XGAUEWC2B25ZEQ2NVQVZZUL6CANSCBWIIEY3A, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.007858693 (538h46m53.878941738s) [noise=true , address=145.79.227.27:4691 , piece_id=MJMBV5LR4RA5ZP7Q7U34DXO2TWJ6LDKH4YMADM3K5CU2KRHMQWBA, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.008204601 (538h46m53.879287637s) [noise=true , address=145.79.227.12:4767 , piece_id=A3L6CAJZWVR4RYOZMKQ5C5UALP6FKW367UGSOCW3YEVLWMYJBBTQ, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:55.745297151 (538h46m55.616380186s) [upload=472506]: Next returned nil segment Dec 28 02:15:55.745351053 (538h46m55.616434108s) [upload=472506]: Waiting for error group managing segments... Dec 28 02:15:56.032845416 (538h46m55.903928451s) [noise=true , address=145.79.227.18:4006 , piece_id=GYNQ6GGPHVJJYRV2RHRL63SFS4UPSONZY6PPPVC4ZTSOK5SVWVLQ, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.202191721 (538h46m56.073274756s) [noise=true , address=145.79.227.19:4077 , piece_id=4LCN23MUWAX2HK3J2OFAFLEGUQM4YYTJWZ7IPCN3D6QROORY5O6A, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.205182144 (538h46m56.076265189s) [noise=true , address=145.79.227.31:4797 , piece_id=J5OPI7TBATORMGHB245L3YCWH2I56AVGHVAWJRLALMYDQIW7YAGQ, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.205433684 (538h46m56.076516719s) [noise=true , address=145.79.227.27:4691 , piece_id=MJMBV5LR4RA5ZP7Q7U34DXO2TWJ6LDKH4YMADM3K5CU2KRHMQWBA, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.214467002 (538h46m56.085550047s) [noise=true , address=145.79.227.16:4909 , piece_id=KSGRHS6G6JZ7IY23T277NMHN6HGG7DUODYBIXENFS2PSTYNUX37Q, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.214875340 (538h46m56.085958375s) [noise=true , address=145.79.227.4:4185 , piece_id=SKBIGBFEIKWVJJFKWFM6TO5CYQ6RCM6RLV6AXCM4TKG7TS6TAVXQ, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.217932621 (538h46m56.089015656s) [noise=true , address=145.79.227.12:4767 , piece_id=A3L6CAJZWVR4RYOZMKQ5C5UALP6FKW367UGSOCW3YEVLWMYJBBTQ, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.306475972 (538h46m56.177559007s) [noise=true , address=45.91.116.6:4973 , piece_id=GU4X73Q6AFIBRIYAG4ASNF22MDMCKUOKIJCS3TTJT3FKDWLAMIKA, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.385055362 (538h46m56.256138408s) [noise=true , address=45.91.116.7:4009 , piece_id=DQ4IHJQYEPBEW63HH3ELSFGDFZUHJ7HL6UHBLGQRHKOHTD7HP2SQ, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.395589739 (538h46m56.266672784s) [noise=true , address=45.91.116.14:4190 , piece_id=ZDWBLJMWUZN4ZMK5X6FWN24VEWAFU2KODQJ7URBZBWJHIML4GKKA, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.395681062 (538h46m56.266764107s) [noise=true , address=145.79.227.52:4445 , piece_id=ZEY2QW4N6ZNGBILKIDQ7YBFSB7U2JVICZJV2W3GQU7D3QACB32FQ, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.403044182 (538h46m56.274127217s) [noise=true , address=145.79.227.32:4836 , piece_id=AZ5NCPW2VX2VSH7XGAUEWC2B25ZEQ2NVQVZZUL6CANSCBWIIEY3A, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:57.110570880 (538h46m56.981653925s) [noise=true , address=45.91.116.11:4118 , piece_id=LA7UXPHDDM25MJ4IKX4IGJXKZFA37TTXYBAGMA25C3Y57X6YFQ3Q, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:57.110788534 (538h46m56.981871569s) [seg_pos={0 0}, upload=472506]: Segment reached optimal threshold of 13 pieces. Dec 28 02:15:57.111013823 (538h46m56.982096858s) [noise=true , address=77.90.31.14:4661 , piece_id=U2S5KQX5UAKMMMIOZH6CP7SE7PR2DZIKHKX7OXBCJMSZKG4RUMMQ, seg_pos={0 0}, upload=472506]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Dec 28 02:15:57.111022289 (538h46m56.982105335s) [noise=true , address=77.90.31.14:4661 , piece_id=U2S5KQX5UAKMMMIOZH6CP7SE7PR2DZIKHKX7OXBCJMSZKG4RUMMQ, seg_pos={0 0}, upload=472506]: Long tail cancellation due to optimal threshold Dec 28 02:15:57.111365041 (538h46m56.982448076s) [noise=true , address=77.90.31.16:4735 , piece_id=VD6RDROPI6XQA2TYQUB4ENXFD2B6OCM2YH5AXP3OVGPQ5YBI37BA, seg_pos={0 0}, upload=472506]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: read tcp 188.166.225.170:47380->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 02:15:57.111372196 (538h46m56.982455241s) [noise=true , address=77.90.31.16:4735 , piece_id=VD6RDROPI6XQA2TYQUB4ENXFD2B6OCM2YH5AXP3OVGPQ5YBI37BA, seg_pos={0 0}, upload=472506]: Long tail cancellation due to optimal threshold Dec 28 02:15:57.111468879 (538h46m56.982551924s) [noise=true , address=145.79.227.17:4972 , piece_id=64UKV76NMUYNSLPZA3WLTQ6F7W2XP5Y564DFNQMLBFB3SG6S3IFA, seg_pos={0 0}, upload=472506]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: expected piece hash; context canceled; EOF Dec 28 02:15:57.111474270 (538h46m56.982557315s) [noise=true , address=145.79.227.17:4972 , piece_id=64UKV76NMUYNSLPZA3WLTQ6F7W2XP5Y564DFNQMLBFB3SG6S3IFA, seg_pos={0 0}, upload=472506]: Long tail cancellation due to optimal threshold Dec 28 02:15:57.111493035 (538h46m56.98257607s) [seg_pos={0 0}, upload=472506]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 02:15:57.111624576 (538h46m56.982707611s) [upload=472506]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 02:15:57.111630667 (538h46m56.982713713s) [upload=472506]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 02:15:57.409023835 (538h46m57.28010687s) [upload=472506]: Done uploading segments... Dec 28 02:15:57.409043784 (538h46m57.280126819s) [upload=472506]: Upload finished. err: <nil> Duration: 9.66218474s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)