Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [......X....X............X........X....X...........X.....XX..] duration: 3.50623593s trace: https://jaeger.dev.storj.io/trace/75a936d104ca03de FAIL (2025-12-28 03:15:16 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.506264435s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 47 out of 60 is failed (threshold=3) [XXXX.XXXX.XXXX..X..XXXXX.XXXXXXXXXXXXX.XXXXXXXXX.X..XX..XXXX] duration: 30.004317965s trace: https://jaeger.dev.storj.io/trace/2041d6f85a87503d FAIL (2025-12-28 03:15:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: YRLD2DNU2XW2OI7S5SKRSYD2BQSEYTXLCPHI7IZBLZNMGHLJQO5A) context deadline exceeded; piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: WSS4GWV6X76NC5RYDWJGV6JQX7MZDPJBQE5UJA2V3RBXDCFDW2ZQ) context deadline exceeded ======LOGS====== Dec 28 03:14:34.098010659 (539h45m33.969093694s) [upload=473358]: Starting upload Dec 28 03:14:34.098063249 (539h45m33.969146284s) [upload=473358]: Done starting upload Dec 28 03:14:34.098234254 (539h45m33.96931729s) [upload=473358]: Uploading segments... Dec 28 03:14:34.098271145 (539h45m33.96935418s) [upload=473358]: Got next segment. Inline: false Dec 28 03:14:34.098299659 (539h45m33.969382694s) [upload=473358]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 03:14:34.098304388 (539h45m33.969387423s) [upload=473358]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 03:14:34.357073825 (539h45m34.22815686s) [seg_pos={0 0}, upload=473358]: Begin upload segment... Dec 28 03:14:34.357165900 (539h45m34.228248955s) [seg_pos={0 0}, upload=473358]: Done begin upload segment. Dec 28 03:14:34.357250521 (539h45m34.228333557s) [noise=true , address=145.79.227.6:4259 , piece_id=MFEBLZTLZ3A4VAIY3KARNNCPSAOGEUSQIDQY5HWGKG7YPV2PA3WA, seg_pos={0 0}, upload=473358]: Uploading piece... Dec 28 03:14:34.357293914 (539h45m34.228376959s) [noise=true , address=77.90.31.17:4770 , piece_id=DJAHWROHFNM5SMULEMVYYJVPKDSAOX7PFQCAAST5L4L7SOQSDKFA, seg_pos={0 0}, upload=473358]: Uploading piece... Dec 28 03:14:34.357853189 (539h45m34.228936224s) [noise=true , address=45.91.116.12:4226 , piece_id=YRLD2DNU2XW2OI7S5SKRSYD2BQSEYTXLCPHI7IZBLZNMGHLJQO5A, seg_pos={0 0}, upload=473358]: Uploading piece... Dec 28 03:14:34.358265153 (539h45m34.229348198s) [noise=true , address=45.91.116.14:4190 , piece_id=HRUJHBNPR4L6AKG4JJASQ4ZF73WYXBWJVFVSICAXZ2GJ5UF2BNZQ, seg_pos={0 0}, upload=473358]: Uploading piece... Dec 28 03:14:34.358736250 (539h45m34.229819285s) [noise=true , address=77.90.31.12:4590 , piece_id=WSS4GWV6X76NC5RYDWJGV6JQX7MZDPJBQE5UJA2V3RBXDCFDW2ZQ, seg_pos={0 0}, upload=473358]: Uploading piece... Dec 28 03:14:34.359187319 (539h45m34.230270354s) [noise=true , address=145.79.227.50:4374 , piece_id=JHE4UCFTG3SWFJS6XW6QJXM42VKJH4QUEEIK4ZZWNSPECF6YHS5A, seg_pos={0 0}, upload=473358]: Uploading piece... Dec 28 03:14:34.359637737 (539h45m34.230720772s) [noise=true , address=145.79.227.13:4804 , piece_id=TUOW6OQNXVQJ6OE745ZBG26QVYSZUL7SPCZQBGT5JZVBNZGC5NUA, seg_pos={0 0}, upload=473358]: Uploading piece... Dec 28 03:14:34.360111319 (539h45m34.231194354s) [noise=true , address=45.91.116.5:4298 , piece_id=C6YYFE66ZPKPE44TI5MY3GXF2URBY5TJ437XJUII4VCNPFOU7XWQ, seg_pos={0 0}, upload=473358]: Uploading piece... Dec 28 03:14:34.361134917 (539h45m34.232217952s) [noise=true , address=145.79.227.34:4907 , piece_id=FTC6HKTIFEXUXRVS2YF3LRV36H6572M3FXH7KEOG5JXFI7AXYDJA, seg_pos={0 0}, upload=473358]: Uploading piece... Dec 28 03:14:34.361679824 (539h45m34.232762859s) [noise=true , address=145.79.227.18:4006 , piece_id=GLVWUJEBEJMYA3WZJTALLNFEQZFCJZD4GKHKU2GFEKD5DQLBGQMA, seg_pos={0 0}, upload=473358]: Uploading piece... Dec 28 03:14:34.362314524 (539h45m34.233397559s) [noise=true , address=77.90.31.14:4663 , piece_id=DCENHYL7LGJZ37YYN7ZLCXGA7L6DNSZ4EV5GIZ43PXWXRPGR7UZQ, seg_pos={0 0}, upload=473358]: Uploading piece... Dec 28 03:14:34.363393428 (539h45m34.234476463s) [noise=true , address=145.79.227.52:4446 , piece_id=H7E36GL3LLMFXSELU27JPYZKGH522NUKNUKKBU4Z5RFE2LCJF57Q, seg_pos={0 0}, upload=473358]: Uploading piece... Dec 28 03:14:34.366179273 (539h45m34.237262308s) [noise=true , address=145.79.227.51:4409 , piece_id=EMEWGDXSOQSJBVNZFGGEETK5CQPHZDSLQ6PJDGVXD42WTAOJCJ2Q, seg_pos={0 0}, upload=473358]: Uploading piece... Dec 28 03:14:34.370306391 (539h45m34.241389436s) [noise=true , address=77.90.31.10:4518 , piece_id=2YGWF7AZUWBGKHUUJGEQCZIDC7XGUY7DHAOGBOGZ2ASWKRSCNTJA, seg_pos={0 0}, upload=473358]: Uploading piece... Dec 28 03:14:34.375128343 (539h45m34.246211378s) [noise=true , address=145.79.227.35:4482 , piece_id=FRY75Z7NHHMSEN4SNIB3ROYTSIMT23LT7SHCSTRZ44DPMTSWUJIA, seg_pos={0 0}, upload=473358]: Uploading piece... Dec 28 03:14:34.378458604 (539h45m34.249541649s) [noise=true , address=145.79.227.16:4912 , piece_id=ODBCD6NNU7AJOEHT2Z4OHIBCIIN3KAKHAT4ZUYVXM6U4OBDU4LJA, seg_pos={0 0}, upload=473358]: Uploading piece... Dec 28 03:14:36.217621632 (539h45m36.088704677s) [upload=473358]: Next returned nil segment Dec 28 03:14:36.217642963 (539h45m36.088726008s) [upload=473358]: Waiting for error group managing segments... Dec 28 03:14:36.529714034 (539h45m36.400797069s) [noise=true , address=145.79.227.50:4374 , piece_id=JHE4UCFTG3SWFJS6XW6QJXM42VKJH4QUEEIK4ZZWNSPECF6YHS5A, seg_pos={0 0}, upload=473358]: Done uploading piece. err: <nil> Dec 28 03:14:36.532476785 (539h45m36.40355982s) [noise=true , address=145.79.227.18:4006 , piece_id=GLVWUJEBEJMYA3WZJTALLNFEQZFCJZD4GKHKU2GFEKD5DQLBGQMA, seg_pos={0 0}, upload=473358]: Done uploading piece. err: <nil> Dec 28 03:14:36.541282482 (539h45m36.412365517s) [noise=true , address=145.79.227.13:4804 , piece_id=TUOW6OQNXVQJ6OE745ZBG26QVYSZUL7SPCZQBGT5JZVBNZGC5NUA, seg_pos={0 0}, upload=473358]: Done uploading piece. err: <nil> Dec 28 03:14:36.549881867 (539h45m36.420964902s) [noise=true , address=145.79.227.52:4446 , piece_id=H7E36GL3LLMFXSELU27JPYZKGH522NUKNUKKBU4Z5RFE2LCJF57Q, seg_pos={0 0}, upload=473358]: Done uploading piece. err: <nil> Dec 28 03:14:36.563540305 (539h45m36.43462334s) [noise=true , address=145.79.227.16:4912 , piece_id=ODBCD6NNU7AJOEHT2Z4OHIBCIIN3KAKHAT4ZUYVXM6U4OBDU4LJA, seg_pos={0 0}, upload=473358]: Done uploading piece. err: <nil> Dec 28 03:14:36.659576112 (539h45m36.530659147s) [noise=true , address=77.90.31.17:4770 , piece_id=DJAHWROHFNM5SMULEMVYYJVPKDSAOX7PFQCAAST5L4L7SOQSDKFA, seg_pos={0 0}, upload=473358]: Done uploading piece. err: <nil> Dec 28 03:14:36.709018207 (539h45m36.580101242s) [noise=true , address=45.91.116.5:4298 , piece_id=C6YYFE66ZPKPE44TI5MY3GXF2URBY5TJ437XJUII4VCNPFOU7XWQ, seg_pos={0 0}, upload=473358]: Done uploading piece. err: <nil> Dec 28 03:14:36.754480815 (539h45m36.62556385s) [noise=true , address=45.91.116.14:4190 , piece_id=HRUJHBNPR4L6AKG4JJASQ4ZF73WYXBWJVFVSICAXZ2GJ5UF2BNZQ, seg_pos={0 0}, upload=473358]: Done uploading piece. err: <nil> Dec 28 03:14:36.778907237 (539h45m36.649990272s) [noise=true , address=45.91.116.12:4226 , piece_id=YRLD2DNU2XW2OI7S5SKRSYD2BQSEYTXLCPHI7IZBLZNMGHLJQO5A, seg_pos={0 0}, upload=473358]: Done uploading piece. err: <nil> Dec 28 03:14:36.883590279 (539h45m36.754673324s) [noise=true , address=145.79.227.51:4409 , piece_id=EMEWGDXSOQSJBVNZFGGEETK5CQPHZDSLQ6PJDGVXD42WTAOJCJ2Q, seg_pos={0 0}, upload=473358]: Done uploading piece. err: <nil> Dec 28 03:14:36.916614422 (539h45m36.787697457s) [noise=true , address=145.79.227.6:4259 , piece_id=MFEBLZTLZ3A4VAIY3KARNNCPSAOGEUSQIDQY5HWGKG7YPV2PA3WA, seg_pos={0 0}, upload=473358]: Done uploading piece. err: <nil> Dec 28 03:14:36.945551062 (539h45m36.816634097s) [noise=true , address=77.90.31.14:4663 , piece_id=DCENHYL7LGJZ37YYN7ZLCXGA7L6DNSZ4EV5GIZ43PXWXRPGR7UZQ, seg_pos={0 0}, upload=473358]: Done uploading piece. err: <nil> Dec 28 03:14:37.661008792 (539h45m37.532091827s) [noise=true , address=77.90.31.12:4590 , piece_id=WSS4GWV6X76NC5RYDWJGV6JQX7MZDPJBQE5UJA2V3RBXDCFDW2ZQ, seg_pos={0 0}, upload=473358]: Done uploading piece. err: <nil> Dec 28 03:14:37.661069328 (539h45m37.532152363s) [seg_pos={0 0}, upload=473358]: Segment reached optimal threshold of 13 pieces. Dec 28 03:14:37.661551485 (539h45m37.53263455s) [noise=true , address=145.79.227.35:4482 , piece_id=FRY75Z7NHHMSEN4SNIB3ROYTSIMT23LT7SHCSTRZ44DPMTSWUJIA, seg_pos={0 0}, upload=473358]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: expected piece hash; context canceled; EOF Dec 28 03:14:37.661566785 (539h45m37.53264986s) [noise=true , address=145.79.227.35:4482 , piece_id=FRY75Z7NHHMSEN4SNIB3ROYTSIMT23LT7SHCSTRZ44DPMTSWUJIA, seg_pos={0 0}, upload=473358]: Long tail cancellation due to optimal threshold Dec 28 03:14:37.662200852 (539h45m37.533283907s) [noise=true , address=145.79.227.34:4907 , piece_id=FTC6HKTIFEXUXRVS2YF3LRV36H6572M3FXH7KEOG5JXFI7AXYDJA, seg_pos={0 0}, upload=473358]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 28 03:14:37.662241990 (539h45m37.533325035s) [noise=true , address=145.79.227.34:4907 , piece_id=FTC6HKTIFEXUXRVS2YF3LRV36H6572M3FXH7KEOG5JXFI7AXYDJA, seg_pos={0 0}, upload=473358]: Long tail cancellation due to optimal threshold Dec 28 03:14:37.662459926 (539h45m37.533542961s) [noise=true , address=77.90.31.10:4518 , piece_id=2YGWF7AZUWBGKHUUJGEQCZIDC7XGUY7DHAOGBOGZ2ASWKRSCNTJA, seg_pos={0 0}, upload=473358]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 28 03:14:37.662468793 (539h45m37.533551838s) [noise=true , address=77.90.31.10:4518 , piece_id=2YGWF7AZUWBGKHUUJGEQCZIDC7XGUY7DHAOGBOGZ2ASWKRSCNTJA, seg_pos={0 0}, upload=473358]: Long tail cancellation due to optimal threshold Dec 28 03:14:37.662511464 (539h45m37.533594509s) [seg_pos={0 0}, upload=473358]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 03:14:37.662655207 (539h45m37.533738252s) [upload=473358]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 03:14:37.662683672 (539h45m37.533766737s) [upload=473358]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 03:14:37.982864414 (539h45m37.853947449s) [upload=473358]: Done uploading segments... Dec 28 03:14:37.982884201 (539h45m37.853967246s) [upload=473358]: Upload finished. err: <nil> Duration: 30.00436813s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 34.347594875s trace: https://jaeger.dev.storj.io/trace/1822be52d200e5b5 FAIL (2025-12-28 03:15:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1ZAfYZ4wgpZVvatj6wtHona3hTaPcLxAmXC1DT2uKQyo8nLQQ7, Piece ID: 5MMQ44MXRT4XEL4X6WO54WRG2XYMX3AN3FNQ2H2XIUTKBXI3EEPA) context deadline exceeded; piecestore: (Node ID: 124guAM82oibYZsMho67Em6Ke6yEZXwT6LPMLLSZZBLqwoFGUkn, Piece ID: 22WJIPX4SM5NTIAW5ZRM2UKX32CTDBFCXPCCHRRWRCUCOF5CRE5Q) context deadline exceeded; piecestore: (Node ID: 1e6p1TD5CqUyWdZjkrxh2kRqMuyDiMUr9eimfeayDAKwhtBrZ8, Piece ID: XRI5EJFN7YYUG4QKLDGEFT6YAUHCUTTSXF735UDR6LZZBGMAZDJQ) context deadline exceeded ======LOGS====== 5.234848204 (538h55m35.105931239s) [noise=true , address=77.90.31.11:4572 , piece_id=5MMQ44MXRT4XEL4X6WO54WRG2XYMX3AN3FNQ2H2XIUTKBXI3EEPA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.235412057 (538h55m35.106495102s) [noise=true , address=145.79.227.22:4512 , piece_id=D46YLTIKGHG36DQ4YBKBSUDV4GGB4WVDWNOJS27IYYLN6AYZ5QFQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.235820946 (538h55m35.106903981s) [noise=true , address=77.90.31.19:4874 , piece_id=RHZUXUYMJU6RKQGSDZM5WDWIN5SQ7YIRSRATNNKT5GEMU2QH444A, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.236192192 (538h55m35.107275237s) [noise=true , address=77.90.31.16:4760 , piece_id=XRI5EJFN7YYUG4QKLDGEFT6YAUHCUTTSXF735UDR6LZZBGMAZDJQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.236564772 (538h55m35.107647817s) [noise=true , address=145.79.227.40:4062 , piece_id=PW6QWIPQWVUETLC6DJ624G6IB4P6EM4SYB72NAMQ4V6NUKNLUYLQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.236929766 (538h55m35.108012812s) [noise=true , address=64.49.13.139:4480 , piece_id=JAJY3F7IDFFMFKCK2QJV6W74RIESLRKBLXMN27Q3C45JWEPLBZLQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.237916846 (538h55m35.108999881s) [noise=true , address=145.79.227.34:4932 , piece_id=UPXSCQ45QDGPKNJXRFJ33EAA3QENVHRVQARU5TFJTEWICZNR253Q, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.238852948 (538h55m35.109935983s) [noise=true , address=45.91.116.12:4257 , piece_id=ZI67LIAHZW4G357GWKDPTB5SMXYGNXZTTTDMNAETDS2B6SWVDAEQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239152910 (538h55m35.110235945s) [noise=true , address=145.79.227.44:4222 , piece_id=CZ2CS62CTHQFAIE3S3JO2N3VJ4XIR5YJ2LMRVLLX7I4VM2VZVQVA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239409969 (538h55m35.110493004s) [noise=true , address=145.79.227.30:4773 , piece_id=KUGJXZCNMUE7WSK3VLY4AUBSOT4H6IY4XB4NCBGAGCNUG7F7IODQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239681136 (538h55m35.110764171s) [noise=true , address=77.90.31.13:4635 , piece_id=IIOY5X2W3XW4C2OFEBRCABMYFBI24RRTYNUSKN6KJU7CTUW4KONQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239954676 (538h55m35.111037711s) [noise=true , address=45.91.116.9:4281 , piece_id=JZLDW777CBX2BP7MLOQV4NFP3H4XJVCNQXTHNSUWPVDOJLPZK36A, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.240189914 (538h55m35.111272949s) [noise=true , address=145.79.227.35:4512 , piece_id=3E3VCCC45XVT5FHFDKXSTGGAGYBEML3BKU6JJTT33LCD2YMOMBZQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.240449147 (538h55m35.111532182s) [noise=true , address=45.91.116.8:4064 , piece_id=22WJIPX4SM5NTIAW5ZRM2UKX32CTDBFCXPCCHRRWRCUCOF5CRE5Q, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.240639640 (538h55m35.111722675s) [noise=true , address=145.79.227.32:4853 , piece_id=MBNCFDBQ4SZZXV4F7J5TRMOUYUQ5IPKQCGYGGM5SOIFJ6ZYCAUYA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.241137798 (538h55m35.112220833s) [noise=true , address=77.90.31.16:4751 , piece_id=N3Z2LLMCIA2F42VXZA7IDRU4XEKGOICH6TRVERODU76MFFE4QSHA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.241383797 (538h55m35.112466842s) [noise=true , address=77.90.31.18:4836 , piece_id=BSVEFI52EKN44CMINFUSZBNOH53IW5TXLVTF5IH6ZUTNII6RRLVA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:36.592880816 (538h55m36.463963871s) [upload=472631]: Next returned nil segment Dec 28 02:24:36.592901625 (538h55m36.46398467s) [upload=472631]: Waiting for error group managing segments... Dec 28 02:24:36.946933741 (538h55m36.818016786s) [noise=true , address=64.49.13.139:4480 , piece_id=JAJY3F7IDFFMFKCK2QJV6W74RIESLRKBLXMN27Q3C45JWEPLBZLQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.128189692 (538h55m36.999272737s) [noise=true , address=145.79.227.35:4512 , piece_id=3E3VCCC45XVT5FHFDKXSTGGAGYBEML3BKU6JJTT33LCD2YMOMBZQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.160760084 (538h55m37.031843119s) [noise=true , address=145.79.227.34:4932 , piece_id=UPXSCQ45QDGPKNJXRFJ33EAA3QENVHRVQARU5TFJTEWICZNR253Q, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.288838207 (538h55m37.159921252s) [noise=true , address=77.90.31.11:4572 , piece_id=5MMQ44MXRT4XEL4X6WO54WRG2XYMX3AN3FNQ2H2XIUTKBXI3EEPA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.319996310 (538h55m37.191079345s) [noise=true , address=145.79.227.22:4512 , piece_id=D46YLTIKGHG36DQ4YBKBSUDV4GGB4WVDWNOJS27IYYLN6AYZ5QFQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.320513273 (538h55m37.191596319s) [noise=true , address=77.90.31.18:4836 , piece_id=BSVEFI52EKN44CMINFUSZBNOH53IW5TXLVTF5IH6ZUTNII6RRLVA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.336522926 (538h55m37.207605971s) [noise=true , address=145.79.227.32:4853 , piece_id=MBNCFDBQ4SZZXV4F7J5TRMOUYUQ5IPKQCGYGGM5SOIFJ6ZYCAUYA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.384868031 (538h55m37.255951066s) [noise=true , address=77.90.31.16:4751 , piece_id=N3Z2LLMCIA2F42VXZA7IDRU4XEKGOICH6TRVERODU76MFFE4QSHA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.453563554 (538h55m37.324646599s) [noise=true , address=45.91.116.8:4064 , piece_id=22WJIPX4SM5NTIAW5ZRM2UKX32CTDBFCXPCCHRRWRCUCOF5CRE5Q, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.479454839 (538h55m37.350537884s) [noise=true , address=45.91.116.12:4257 , piece_id=ZI67LIAHZW4G357GWKDPTB5SMXYGNXZTTTDMNAETDS2B6SWVDAEQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.659710174 (538h55m37.530793219s) [noise=true , address=77.90.31.13:4635 , piece_id=IIOY5X2W3XW4C2OFEBRCABMYFBI24RRTYNUSKN6KJU7CTUW4KONQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.715039128 (538h55m37.586122163s) [noise=true , address=77.90.31.16:4760 , piece_id=XRI5EJFN7YYUG4QKLDGEFT6YAUHCUTTSXF735UDR6LZZBGMAZDJQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.977473406 (538h55m37.848556441s) [noise=true , address=77.90.31.19:4874 , piece_id=RHZUXUYMJU6RKQGSDZM5WDWIN5SQ7YIRSRATNNKT5GEMU2QH444A, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:38.016302745 (538h55m37.88738578s) [noise=true , address=145.79.227.30:4773 , piece_id=KUGJXZCNMUE7WSK3VLY4AUBSOT4H6IY4XB4NCBGAGCNUG7F7IODQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:38.016326982 (538h55m37.887410027s) [seg_pos={0 0}, upload=472631]: Segment reached optimal threshold of 14 pieces. Dec 28 02:24:38.016585353 (538h55m37.887668398s) [noise=true , address=145.79.227.30:4766 , piece_id=67M2DMPAVWIQKQSWUAG73D4E4NRRSG4XEVPOUVIJPEQOUV7A6VVA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1k6zdamuKPkBdzyNZNWx4FCmaTkv33PchcwvTRR8bgcETBs4K7): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016594450 (538h55m37.887677495s) [noise=true , address=145.79.227.30:4766 , piece_id=67M2DMPAVWIQKQSWUAG73D4E4NRRSG4XEVPOUVIJPEQOUV7A6VVA, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016627093 (538h55m37.887710148s) [noise=true , address=45.91.116.9:4281 , piece_id=JZLDW777CBX2BP7MLOQV4NFP3H4XJVCNQXTHNSUWPVDOJLPZK36A, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gxf7DpJZthyR6f1PExX56tJavjhPw9Gxb4h8mtmeYKW3t3tS1): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016635258 (538h55m37.887718293s) [noise=true , address=45.91.116.9:4281 , piece_id=JZLDW777CBX2BP7MLOQV4NFP3H4XJVCNQXTHNSUWPVDOJLPZK36A, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016871298 (538h55m37.887954363s) [noise=true , address=145.79.227.44:4222 , piece_id=CZ2CS62CTHQFAIE3S3JO2N3VJ4XIR5YJ2LMRVLLX7I4VM2VZVQVA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZKW35KvrsoYRefbYeUFZo8bh4XZYycNKGEvTFjzibezjXgj3e): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016880124 (538h55m37.887963169s) [noise=true , address=145.79.227.44:4222 , piece_id=CZ2CS62CTHQFAIE3S3JO2N3VJ4XIR5YJ2LMRVLLX7I4VM2VZVQVA, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016934839 (538h55m37.888017884s) [noise=true , address=145.79.227.40:4062 , piece_id=PW6QWIPQWVUETLC6DJ624G6IB4P6EM4SYB72NAMQ4V6NUKNLUYLQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016943425 (538h55m37.88802649s) [noise=true , address=145.79.227.40:4062 , piece_id=PW6QWIPQWVUETLC6DJ624G6IB4P6EM4SYB72NAMQ4V6NUKNLUYLQ, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016968252 (538h55m37.888051297s) [seg_pos={0 0}, upload=472631]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 02:24:38.017067822 (538h55m37.888150867s) [upload=472631]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 02:24:38.017074244 (538h55m37.888157289s) [upload=472631]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 02:24:38.304306667 (538h55m38.175389712s) [upload=472631]: Done uploading segments... Dec 28 02:24:38.304329641 (538h55m38.175412686s) [upload=472631]: Upload finished. err: <nil> Duration: 34.34763428s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [.......X....X............X........X....X...........X.....XX.] duration: 3.930061308s trace: https://jaeger.dev.storj.io/trace/75a936d104ca03de FAIL (2025-12-28 03:14:52 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.930081486s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 55.815708658s trace: https://jaeger.dev.storj.io/trace/1822be52d200e5b5 FAIL (2025-12-28 03:14:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1ZAfYZ4wgpZVvatj6wtHona3hTaPcLxAmXC1DT2uKQyo8nLQQ7, Piece ID: 5MMQ44MXRT4XEL4X6WO54WRG2XYMX3AN3FNQ2H2XIUTKBXI3EEPA) context deadline exceeded; piecestore: (Node ID: 124guAM82oibYZsMho67Em6Ke6yEZXwT6LPMLLSZZBLqwoFGUkn, Piece ID: 22WJIPX4SM5NTIAW5ZRM2UKX32CTDBFCXPCCHRRWRCUCOF5CRE5Q) context deadline exceeded; piecestore: (Node ID: 1e6p1TD5CqUyWdZjkrxh2kRqMuyDiMUr9eimfeayDAKwhtBrZ8, Piece ID: XRI5EJFN7YYUG4QKLDGEFT6YAUHCUTTSXF735UDR6LZZBGMAZDJQ) context deadline exceeded ======LOGS====== 5.234848204 (538h55m35.105931239s) [noise=true , address=77.90.31.11:4572 , piece_id=5MMQ44MXRT4XEL4X6WO54WRG2XYMX3AN3FNQ2H2XIUTKBXI3EEPA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.235412057 (538h55m35.106495102s) [noise=true , address=145.79.227.22:4512 , piece_id=D46YLTIKGHG36DQ4YBKBSUDV4GGB4WVDWNOJS27IYYLN6AYZ5QFQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.235820946 (538h55m35.106903981s) [noise=true , address=77.90.31.19:4874 , piece_id=RHZUXUYMJU6RKQGSDZM5WDWIN5SQ7YIRSRATNNKT5GEMU2QH444A, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.236192192 (538h55m35.107275237s) [noise=true , address=77.90.31.16:4760 , piece_id=XRI5EJFN7YYUG4QKLDGEFT6YAUHCUTTSXF735UDR6LZZBGMAZDJQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.236564772 (538h55m35.107647817s) [noise=true , address=145.79.227.40:4062 , piece_id=PW6QWIPQWVUETLC6DJ624G6IB4P6EM4SYB72NAMQ4V6NUKNLUYLQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.236929766 (538h55m35.108012812s) [noise=true , address=64.49.13.139:4480 , piece_id=JAJY3F7IDFFMFKCK2QJV6W74RIESLRKBLXMN27Q3C45JWEPLBZLQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.237916846 (538h55m35.108999881s) [noise=true , address=145.79.227.34:4932 , piece_id=UPXSCQ45QDGPKNJXRFJ33EAA3QENVHRVQARU5TFJTEWICZNR253Q, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.238852948 (538h55m35.109935983s) [noise=true , address=45.91.116.12:4257 , piece_id=ZI67LIAHZW4G357GWKDPTB5SMXYGNXZTTTDMNAETDS2B6SWVDAEQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239152910 (538h55m35.110235945s) [noise=true , address=145.79.227.44:4222 , piece_id=CZ2CS62CTHQFAIE3S3JO2N3VJ4XIR5YJ2LMRVLLX7I4VM2VZVQVA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239409969 (538h55m35.110493004s) [noise=true , address=145.79.227.30:4773 , piece_id=KUGJXZCNMUE7WSK3VLY4AUBSOT4H6IY4XB4NCBGAGCNUG7F7IODQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239681136 (538h55m35.110764171s) [noise=true , address=77.90.31.13:4635 , piece_id=IIOY5X2W3XW4C2OFEBRCABMYFBI24RRTYNUSKN6KJU7CTUW4KONQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239954676 (538h55m35.111037711s) [noise=true , address=45.91.116.9:4281 , piece_id=JZLDW777CBX2BP7MLOQV4NFP3H4XJVCNQXTHNSUWPVDOJLPZK36A, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.240189914 (538h55m35.111272949s) [noise=true , address=145.79.227.35:4512 , piece_id=3E3VCCC45XVT5FHFDKXSTGGAGYBEML3BKU6JJTT33LCD2YMOMBZQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.240449147 (538h55m35.111532182s) [noise=true , address=45.91.116.8:4064 , piece_id=22WJIPX4SM5NTIAW5ZRM2UKX32CTDBFCXPCCHRRWRCUCOF5CRE5Q, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.240639640 (538h55m35.111722675s) [noise=true , address=145.79.227.32:4853 , piece_id=MBNCFDBQ4SZZXV4F7J5TRMOUYUQ5IPKQCGYGGM5SOIFJ6ZYCAUYA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.241137798 (538h55m35.112220833s) [noise=true , address=77.90.31.16:4751 , piece_id=N3Z2LLMCIA2F42VXZA7IDRU4XEKGOICH6TRVERODU76MFFE4QSHA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.241383797 (538h55m35.112466842s) [noise=true , address=77.90.31.18:4836 , piece_id=BSVEFI52EKN44CMINFUSZBNOH53IW5TXLVTF5IH6ZUTNII6RRLVA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:36.592880816 (538h55m36.463963871s) [upload=472631]: Next returned nil segment Dec 28 02:24:36.592901625 (538h55m36.46398467s) [upload=472631]: Waiting for error group managing segments... Dec 28 02:24:36.946933741 (538h55m36.818016786s) [noise=true , address=64.49.13.139:4480 , piece_id=JAJY3F7IDFFMFKCK2QJV6W74RIESLRKBLXMN27Q3C45JWEPLBZLQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.128189692 (538h55m36.999272737s) [noise=true , address=145.79.227.35:4512 , piece_id=3E3VCCC45XVT5FHFDKXSTGGAGYBEML3BKU6JJTT33LCD2YMOMBZQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.160760084 (538h55m37.031843119s) [noise=true , address=145.79.227.34:4932 , piece_id=UPXSCQ45QDGPKNJXRFJ33EAA3QENVHRVQARU5TFJTEWICZNR253Q, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.288838207 (538h55m37.159921252s) [noise=true , address=77.90.31.11:4572 , piece_id=5MMQ44MXRT4XEL4X6WO54WRG2XYMX3AN3FNQ2H2XIUTKBXI3EEPA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.319996310 (538h55m37.191079345s) [noise=true , address=145.79.227.22:4512 , piece_id=D46YLTIKGHG36DQ4YBKBSUDV4GGB4WVDWNOJS27IYYLN6AYZ5QFQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.320513273 (538h55m37.191596319s) [noise=true , address=77.90.31.18:4836 , piece_id=BSVEFI52EKN44CMINFUSZBNOH53IW5TXLVTF5IH6ZUTNII6RRLVA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.336522926 (538h55m37.207605971s) [noise=true , address=145.79.227.32:4853 , piece_id=MBNCFDBQ4SZZXV4F7J5TRMOUYUQ5IPKQCGYGGM5SOIFJ6ZYCAUYA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.384868031 (538h55m37.255951066s) [noise=true , address=77.90.31.16:4751 , piece_id=N3Z2LLMCIA2F42VXZA7IDRU4XEKGOICH6TRVERODU76MFFE4QSHA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.453563554 (538h55m37.324646599s) [noise=true , address=45.91.116.8:4064 , piece_id=22WJIPX4SM5NTIAW5ZRM2UKX32CTDBFCXPCCHRRWRCUCOF5CRE5Q, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.479454839 (538h55m37.350537884s) [noise=true , address=45.91.116.12:4257 , piece_id=ZI67LIAHZW4G357GWKDPTB5SMXYGNXZTTTDMNAETDS2B6SWVDAEQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.659710174 (538h55m37.530793219s) [noise=true , address=77.90.31.13:4635 , piece_id=IIOY5X2W3XW4C2OFEBRCABMYFBI24RRTYNUSKN6KJU7CTUW4KONQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.715039128 (538h55m37.586122163s) [noise=true , address=77.90.31.16:4760 , piece_id=XRI5EJFN7YYUG4QKLDGEFT6YAUHCUTTSXF735UDR6LZZBGMAZDJQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.977473406 (538h55m37.848556441s) [noise=true , address=77.90.31.19:4874 , piece_id=RHZUXUYMJU6RKQGSDZM5WDWIN5SQ7YIRSRATNNKT5GEMU2QH444A, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:38.016302745 (538h55m37.88738578s) [noise=true , address=145.79.227.30:4773 , piece_id=KUGJXZCNMUE7WSK3VLY4AUBSOT4H6IY4XB4NCBGAGCNUG7F7IODQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:38.016326982 (538h55m37.887410027s) [seg_pos={0 0}, upload=472631]: Segment reached optimal threshold of 14 pieces. Dec 28 02:24:38.016585353 (538h55m37.887668398s) [noise=true , address=145.79.227.30:4766 , piece_id=67M2DMPAVWIQKQSWUAG73D4E4NRRSG4XEVPOUVIJPEQOUV7A6VVA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1k6zdamuKPkBdzyNZNWx4FCmaTkv33PchcwvTRR8bgcETBs4K7): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016594450 (538h55m37.887677495s) [noise=true , address=145.79.227.30:4766 , piece_id=67M2DMPAVWIQKQSWUAG73D4E4NRRSG4XEVPOUVIJPEQOUV7A6VVA, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016627093 (538h55m37.887710148s) [noise=true , address=45.91.116.9:4281 , piece_id=JZLDW777CBX2BP7MLOQV4NFP3H4XJVCNQXTHNSUWPVDOJLPZK36A, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gxf7DpJZthyR6f1PExX56tJavjhPw9Gxb4h8mtmeYKW3t3tS1): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016635258 (538h55m37.887718293s) [noise=true , address=45.91.116.9:4281 , piece_id=JZLDW777CBX2BP7MLOQV4NFP3H4XJVCNQXTHNSUWPVDOJLPZK36A, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016871298 (538h55m37.887954363s) [noise=true , address=145.79.227.44:4222 , piece_id=CZ2CS62CTHQFAIE3S3JO2N3VJ4XIR5YJ2LMRVLLX7I4VM2VZVQVA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZKW35KvrsoYRefbYeUFZo8bh4XZYycNKGEvTFjzibezjXgj3e): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016880124 (538h55m37.887963169s) [noise=true , address=145.79.227.44:4222 , piece_id=CZ2CS62CTHQFAIE3S3JO2N3VJ4XIR5YJ2LMRVLLX7I4VM2VZVQVA, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016934839 (538h55m37.888017884s) [noise=true , address=145.79.227.40:4062 , piece_id=PW6QWIPQWVUETLC6DJ624G6IB4P6EM4SYB72NAMQ4V6NUKNLUYLQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016943425 (538h55m37.88802649s) [noise=true , address=145.79.227.40:4062 , piece_id=PW6QWIPQWVUETLC6DJ624G6IB4P6EM4SYB72NAMQ4V6NUKNLUYLQ, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016968252 (538h55m37.888051297s) [seg_pos={0 0}, upload=472631]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 02:24:38.017067822 (538h55m37.888150867s) [upload=472631]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 02:24:38.017074244 (538h55m37.888157289s) [upload=472631]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 02:24:38.304306667 (538h55m38.175389712s) [upload=472631]: Done uploading segments... Dec 28 02:24:38.304329641 (538h55m38.175412686s) [upload=472631]: Upload finished. err: <nil> Duration: 55.815721893s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 47 out of 60 is failed (threshold=3) [XXXXX.XXXX.XXXX..X..XXXXX.XXXXXXXXXXXXX.XXXXXXXXX.X..XX..XXX] duration: 30.002656919s trace: https://jaeger.dev.storj.io/trace/767e4c83791b8675 FAIL (2025-12-28 03:14:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: PCZGL4H3U4GD2UGBFZGAV2MGQS5JFLPB6BWVPKVUCBS6GKMMNCPA) context deadline exceeded; piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: 7FVP3G4XOLL7UMW7J73BPLYAQCP4BV32IT6YLHGH7HZSGBADE6YA) context deadline exceeded ======LOGS====== Dec 28 03:13:55.381200782 (539h44m55.252283827s) [upload=473346]: Starting upload Dec 28 03:13:55.381286355 (539h44m55.25236939s) [upload=473346]: Done starting upload Dec 28 03:13:55.381323045 (539h44m55.25240608s) [upload=473346]: Uploading segments... Dec 28 03:13:55.381590534 (539h44m55.252673589s) [upload=473346]: Got next segment. Inline: false Dec 28 03:13:55.381680736 (539h44m55.252763781s) [upload=473346]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 03:13:55.381689482 (539h44m55.252772527s) [upload=473346]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 03:13:55.779907050 (539h44m55.650990095s) [seg_pos={0 0}, upload=473346]: Begin upload segment... Dec 28 03:13:55.779988035 (539h44m55.65107107s) [noise=true , address=145.79.227.17:4971 , piece_id=XMOL5BEM34M43GNTCOJFVJMVSSCKETV4Y46S7ETQWMY6IIGV4KUA, seg_pos={0 0}, upload=473346]: Uploading piece... Dec 28 03:13:55.780004686 (539h44m55.651087731s) [seg_pos={0 0}, upload=473346]: Done begin upload segment. Dec 28 03:13:55.780063919 (539h44m55.651146954s) [noise=true , address=145.79.227.46:4602 , piece_id=VYIRUDSACFAGAXMPIIXJEIUGXOSGS7ECL2WFZTT74WS3E4DFKMLA, seg_pos={0 0}, upload=473346]: Uploading piece... Dec 28 03:13:55.780690993 (539h44m55.651774038s) [noise=true , address=145.79.227.50:4373 , piece_id=2MAIXIOBYRISWEM33KRQICY4OHOMYFRSBUVL43LEGJVAMTKVWDZA, seg_pos={0 0}, upload=473346]: Uploading piece... Dec 28 03:13:55.782404105 (539h44m55.65348714s) [noise=true , address=145.79.227.7:4295 , piece_id=MD4XPRJ3F7PEBKLSPMGYCVGIAI3BYUORS2MNBIQ5Z7HQXBQ3TJZQ, seg_pos={0 0}, upload=473346]: Uploading piece... Dec 28 03:13:55.785200119 (539h44m55.656283154s) [noise=true , address=145.79.227.35:4482 , piece_id=54DPK5EUKV4WGKHKINZWROCEYDW4WWLNMCF2RV2BQOXOQYIUXH4Q, seg_pos={0 0}, upload=473346]: Uploading piece... Dec 28 03:13:55.786820745 (539h44m55.65790379s) [noise=true , address=45.91.116.4:4940 , piece_id=FLQR4H4XHVTXDRMGKNI5QOAGN26A6LXBMBRLL5RII7WTFG4PS55Q, seg_pos={0 0}, upload=473346]: Uploading piece... Dec 28 03:13:55.787778678 (539h44m55.658861714s) [noise=true , address=145.79.227.51:4409 , piece_id=EQVX62AHEJ2TOGPNCGVCJZYLE6AI7BAO3WOS67BH4NZXYXUWX36Q, seg_pos={0 0}, upload=473346]: Uploading piece... Dec 28 03:13:55.789009742 (539h44m55.660092777s) [noise=true , address=45.91.116.14:4191 , piece_id=7FVP3G4XOLL7UMW7J73BPLYAQCP4BV32IT6YLHGH7HZSGBADE6YA, seg_pos={0 0}, upload=473346]: Uploading piece... Dec 28 03:13:55.790994231 (539h44m55.662077266s) [noise=true , address=145.79.227.3:4150 , piece_id=IGHNMKYADQS6QPR7OA54HTFAAYG37CETVN6ATCQMCHSIHCJWDH6Q, seg_pos={0 0}, upload=473346]: Uploading piece... Dec 28 03:13:55.791539930 (539h44m55.662622976s) [noise=true , address=45.91.116.12:4227 , piece_id=UVDO6BOJYAGKBT366CCJBFFSHPUBPFSSYXLFIOG6X7NKNI376GZQ, seg_pos={0 0}, upload=473346]: Uploading piece... Dec 28 03:13:55.792305418 (539h44m55.663388463s) [noise=true , address=45.91.116.6:4976 , piece_id=RCZS5PJYJXMZTWZE23LRUKWWQNLH52Y5K3QZGSOH4TIMIMMQQIYQ, seg_pos={0 0}, upload=473346]: Uploading piece... Dec 28 03:13:55.793880577 (539h44m55.664963612s) [noise=true , address=45.91.116.9:4262 , piece_id=PCZGL4H3U4GD2UGBFZGAV2MGQS5JFLPB6BWVPKVUCBS6GKMMNCPA, seg_pos={0 0}, upload=473346]: Uploading piece... Dec 28 03:13:55.794720256 (539h44m55.665803301s) [noise=true , address=77.90.31.17:4773 , piece_id=AI5UFYZPSRGLDKTV2I5K5JJRJX2WP5C5NJIQWNVZ6XMXJA5FZRBA, seg_pos={0 0}, upload=473346]: Uploading piece... Dec 28 03:13:55.795830701 (539h44m55.666913736s) [noise=true , address=145.79.227.36:4518 , piece_id=Q6WLEDKRF2NG77LFCUPX5JYPT5B6DOAHZONOORZTBQ7BQ2ACVMCQ, seg_pos={0 0}, upload=473346]: Uploading piece... Dec 28 03:13:55.798381438 (539h44m55.669464473s) [noise=true , address=145.79.227.34:4906 , piece_id=OZU5J55PAU5WJQDQSJLFJUAJ52FSVFF2NP4UKF4K72BBB4MJ3O3Q, seg_pos={0 0}, upload=473346]: Uploading piece... Dec 28 03:13:55.799620808 (539h44m55.670703843s) [noise=true , address=145.79.227.38:4978 , piece_id=SHSGHIYFOKU5P6GNK3DMEBXVRICBHLB5S7ISQ2BBJN5FGXDFCPIQ, seg_pos={0 0}, upload=473346]: Uploading piece... Dec 28 03:13:57.645309913 (539h44m57.516392948s) [upload=473346]: Next returned nil segment Dec 28 03:13:57.645335141 (539h44m57.516418176s) [upload=473346]: Waiting for error group managing segments... Dec 28 03:13:58.076527387 (539h44m57.947610422s) [noise=true , address=145.79.227.46:4602 , piece_id=VYIRUDSACFAGAXMPIIXJEIUGXOSGS7ECL2WFZTT74WS3E4DFKMLA, seg_pos={0 0}, upload=473346]: Done uploading piece. err: <nil> Dec 28 03:13:58.086541987 (539h44m57.957625052s) [noise=true , address=145.79.227.50:4373 , piece_id=2MAIXIOBYRISWEM33KRQICY4OHOMYFRSBUVL43LEGJVAMTKVWDZA, seg_pos={0 0}, upload=473346]: Done uploading piece. err: <nil> Dec 28 03:13:58.086923864 (539h44m57.958006899s) [noise=true , address=45.91.116.4:4940 , piece_id=FLQR4H4XHVTXDRMGKNI5QOAGN26A6LXBMBRLL5RII7WTFG4PS55Q, seg_pos={0 0}, upload=473346]: Done uploading piece. err: <nil> Dec 28 03:13:58.126653004 (539h44m57.997736039s) [noise=true , address=77.90.31.17:4773 , piece_id=AI5UFYZPSRGLDKTV2I5K5JJRJX2WP5C5NJIQWNVZ6XMXJA5FZRBA, seg_pos={0 0}, upload=473346]: Done uploading piece. err: <nil> Dec 28 03:13:58.146922280 (539h44m58.018005315s) [noise=true , address=45.91.116.6:4976 , piece_id=RCZS5PJYJXMZTWZE23LRUKWWQNLH52Y5K3QZGSOH4TIMIMMQQIYQ, seg_pos={0 0}, upload=473346]: Done uploading piece. err: <nil> Dec 28 03:13:58.207669502 (539h44m58.078752547s) [noise=true , address=45.91.116.12:4227 , piece_id=UVDO6BOJYAGKBT366CCJBFFSHPUBPFSSYXLFIOG6X7NKNI376GZQ, seg_pos={0 0}, upload=473346]: Done uploading piece. err: <nil> Dec 28 03:13:58.389566261 (539h44m58.260649296s) [noise=true , address=45.91.116.9:4262 , piece_id=PCZGL4H3U4GD2UGBFZGAV2MGQS5JFLPB6BWVPKVUCBS6GKMMNCPA, seg_pos={0 0}, upload=473346]: Done uploading piece. err: <nil> Dec 28 03:13:59.035367757 (539h44m58.906450802s) [noise=true , address=145.79.227.7:4295 , piece_id=MD4XPRJ3F7PEBKLSPMGYCVGIAI3BYUORS2MNBIQ5Z7HQXBQ3TJZQ, seg_pos={0 0}, upload=473346]: Done uploading piece. err: <nil> Dec 28 03:13:59.049092480 (539h44m58.920175515s) [noise=true , address=145.79.227.38:4978 , piece_id=SHSGHIYFOKU5P6GNK3DMEBXVRICBHLB5S7ISQ2BBJN5FGXDFCPIQ, seg_pos={0 0}, upload=473346]: Done uploading piece. err: <nil> Dec 28 03:13:59.594115858 (539h44m59.465198903s) [noise=true , address=145.79.227.3:4150 , piece_id=IGHNMKYADQS6QPR7OA54HTFAAYG37CETVN6ATCQMCHSIHCJWDH6Q, seg_pos={0 0}, upload=473346]: Done uploading piece. err: <nil> Dec 28 03:13:59.806948946 (539h44m59.678031981s) [noise=true , address=45.91.116.14:4191 , piece_id=7FVP3G4XOLL7UMW7J73BPLYAQCP4BV32IT6YLHGH7HZSGBADE6YA, seg_pos={0 0}, upload=473346]: Done uploading piece. err: <nil> Dec 28 03:14:00.010493015 (539h44m59.88157605s) [noise=true , address=145.79.227.36:4518 , piece_id=Q6WLEDKRF2NG77LFCUPX5JYPT5B6DOAHZONOORZTBQ7BQ2ACVMCQ, seg_pos={0 0}, upload=473346]: Done uploading piece. err: <nil> Dec 28 03:14:00.243085747 (539h45m0.114168792s) [noise=true , address=145.79.227.34:4906 , piece_id=OZU5J55PAU5WJQDQSJLFJUAJ52FSVFF2NP4UKF4K72BBB4MJ3O3Q, seg_pos={0 0}, upload=473346]: Done uploading piece. err: <nil> Dec 28 03:14:00.243113008 (539h45m0.114196053s) [seg_pos={0 0}, upload=473346]: Segment reached optimal threshold of 13 pieces. Dec 28 03:14:00.243334490 (539h45m0.114417525s) [noise=true , address=145.79.227.51:4409 , piece_id=EQVX62AHEJ2TOGPNCGVCJZYLE6AI7BAO3WOS67BH4NZXYXUWX36Q, seg_pos={0 0}, upload=473346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UoCGA4FjE4yADPXL1oifSPJssPgSGh81REZHwGv2eu1Y1kEaj): protocol: expected piece hash; context canceled; EOF Dec 28 03:14:00.243342495 (539h45m0.11442553s) [noise=true , address=145.79.227.51:4409 , piece_id=EQVX62AHEJ2TOGPNCGVCJZYLE6AI7BAO3WOS67BH4NZXYXUWX36Q, seg_pos={0 0}, upload=473346]: Long tail cancellation due to optimal threshold Dec 28 03:14:00.243525634 (539h45m0.114608699s) [noise=true , address=145.79.227.35:4482 , piece_id=54DPK5EUKV4WGKHKINZWROCEYDW4WWLNMCF2RV2BQOXOQYIUXH4Q, seg_pos={0 0}, upload=473346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: expected piece hash; context canceled; EOF Dec 28 03:14:00.243535764 (539h45m0.114618819s) [noise=true , address=145.79.227.35:4482 , piece_id=54DPK5EUKV4WGKHKINZWROCEYDW4WWLNMCF2RV2BQOXOQYIUXH4Q, seg_pos={0 0}, upload=473346]: Long tail cancellation due to optimal threshold Dec 28 03:14:00.243574417 (539h45m0.114657452s) [noise=true , address=145.79.227.17:4971 , piece_id=XMOL5BEM34M43GNTCOJFVJMVSSCKETV4Y46S7ETQWMY6IIGV4KUA, seg_pos={0 0}, upload=473346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: write tcp 188.166.225.170:58258->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:58258->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58258->145.79.227.17:4971: use of closed network connection Dec 28 03:14:00.243594876 (539h45m0.114677921s) [noise=true , address=145.79.227.17:4971 , piece_id=XMOL5BEM34M43GNTCOJFVJMVSSCKETV4Y46S7ETQWMY6IIGV4KUA, seg_pos={0 0}, upload=473346]: Long tail cancellation due to optimal threshold Dec 28 03:14:00.243651865 (539h45m0.1147349s) [seg_pos={0 0}, upload=473346]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 03:14:00.243838309 (539h45m0.114921344s) [upload=473346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 03:14:00.243845883 (539h45m0.114928929s) [upload=473346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 03:14:00.706573989 (539h45m0.577657034s) [upload=473346]: Done uploading segments... Dec 28 03:14:00.706601922 (539h45m0.577684967s) [upload=473346]: Upload finished. err: <nil> Duration: 30.00266759s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [........X....X............X........X....X...........X.....XX] duration: 5.001769685s trace: https://jaeger.dev.storj.io/trace/75a936d104ca03de FAIL (2025-12-28 03:14:21 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001815262s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 47 out of 60 is failed (threshold=3) [XXXXXX.XXXX.XXXX..X..XXXXX.XXXXXXXXXXXXX.XXXXXXXXX.X..XX..XX] duration: 30.003286438s trace: https://jaeger.dev.storj.io/trace/4bd76317058e6ab FAIL (2025-12-28 03:13:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: 4NADR64IPV4DF7WAEV3XBSVJTUBBTAP3DT6YMEJ6JMR65I7CU6AA) context deadline exceeded; piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: K2DIVO3V3666IDCZKMOESIDUZCTBLJRBJ2DWFWKQUDKPYO52IAMQ) context deadline exceeded ======LOGS====== :13:24.756133236 (539h44m24.627216281s) [upload=473338]: Starting upload Dec 28 03:13:24.756185876 (539h44m24.627268911s) [upload=473338]: Done starting upload Dec 28 03:13:24.756312006 (539h44m24.627395041s) [upload=473338]: Uploading segments... Dec 28 03:13:24.756465588 (539h44m24.627548623s) [upload=473338]: Got next segment. Inline: false Dec 28 03:13:24.756510384 (539h44m24.62759343s) [upload=473338]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 03:13:24.756517358 (539h44m24.627600403s) [upload=473338]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 03:13:25.020711373 (539h44m24.891794418s) [seg_pos={0 0}, upload=473338]: Begin upload segment... Dec 28 03:13:25.020771277 (539h44m24.891854313s) [seg_pos={0 0}, upload=473338]: Done begin upload segment. Dec 28 03:13:25.020807607 (539h44m24.891890642s) [noise=true , address=145.79.227.45:4229 , piece_id=XTAQTKMGBIQZKUMPKBPHGUA6CLB6IG4IV7QDR7TN5OYR3V3LCZSQ, seg_pos={0 0}, upload=473338]: Uploading piece... Dec 28 03:13:25.020814990 (539h44m24.891898025s) [noise=true , address=145.79.227.46:4602 , piece_id=2Y7EFFRBFEWYDYDIXJTJPMLQMPXUSR2WUBXFKGNLJS6LDAA2GMNQ, seg_pos={0 0}, upload=473338]: Uploading piece... Dec 28 03:13:25.020972250 (539h44m24.892055285s) [noise=true , address=45.91.116.6:4974 , piece_id=5Q5MUZXLI3R73RHT7S3NABINZMV6RAAXMZ4OAE735HGS3H6IU6RA, seg_pos={0 0}, upload=473338]: Uploading piece... Dec 28 03:13:25.021311196 (539h44m24.892394241s) [noise=true , address=145.79.227.47:4265 , piece_id=4XSSHNNPBAEWQAQYQRYKJLGCYBESL7UZJB6WMHQ4TXMFVI74MPOQ, seg_pos={0 0}, upload=473338]: Uploading piece... Dec 28 03:13:25.021621848 (539h44m24.892704883s) [noise=true , address=45.91.116.10:4084 , piece_id=EOV6JLWAUEG4DB74VDZO2DJEEGKL5GNFH6XJK2C3CPO42YD66PLA, seg_pos={0 0}, upload=473338]: Uploading piece... Dec 28 03:13:25.021915627 (539h44m24.892998662s) [noise=true , address=77.90.31.10:4519 , piece_id=4NADR64IPV4DF7WAEV3XBSVJTUBBTAP3DT6YMEJ6JMR65I7CU6AA, seg_pos={0 0}, upload=473338]: Uploading piece... Dec 28 03:13:25.022166424 (539h44m24.893249459s) [noise=true , address=145.79.227.48:4302 , piece_id=YLNEL2JBXOYG5FF3XCFFGOTERETDWQH5VRMGD35N5B6POR25EQBQ, seg_pos={0 0}, upload=473338]: Uploading piece... Dec 28 03:13:25.022443582 (539h44m24.893526617s) [noise=true , address=45.91.116.13:4156 , piece_id=K2DIVO3V3666IDCZKMOESIDUZCTBLJRBJ2DWFWKQUDKPYO52IAMQ, seg_pos={0 0}, upload=473338]: Uploading piece... Dec 28 03:13:25.022861768 (539h44m24.893944803s) [noise=true , address=145.79.227.9:4365 , piece_id=LBAQP24LWFED5FVWM3O4AOQG5FEJV4QEJMN4TXWQEGMFUO5ZPXWA, seg_pos={0 0}, upload=473338]: Uploading piece... Dec 28 03:13:25.024200537 (539h44m24.895283582s) [noise=true , address=145.79.227.33:4870 , piece_id=Q7TCXIJELJXOMKWDGIAIFHASVMNLED2BHXYLY6S747DA7MWYJH4A, seg_pos={0 0}, upload=473338]: Uploading piece... Dec 28 03:13:25.025678211 (539h44m24.896761246s) [noise=true , address=145.79.227.7:4295 , piece_id=PBJ3THOXMIH2VV774XKN5MRHU3V7Y4U4XDJ44ZIYAIXFOBQPXNMQ, seg_pos={0 0}, upload=473338]: Uploading piece... Dec 28 03:13:25.029524224 (539h44m24.900607269s) [noise=true , address=145.79.227.19:4078 , piece_id=SJPKATYGSBIUFAF4UVVOQ52G5O5MMDGO32YXINHSYU3CFPZNYKZQ, seg_pos={0 0}, upload=473338]: Uploading piece... Dec 28 03:13:25.032146888 (539h44m24.903229923s) [noise=true , address=145.79.227.37:4566 , piece_id=K2RNZSSF7J4KOLFL4OKXO4BISMTLHMVHCF24RBSYRTEMF34EZQIA, seg_pos={0 0}, upload=473338]: Uploading piece... Dec 28 03:13:25.034344724 (539h44m24.905427759s) [noise=true , address=145.79.227.49:4338 , piece_id=5A33ZMUQPVGJCEFA55YVRXVYFDRT2OEKQIBHMJTAEOS3GV6LWXUA, seg_pos={0 0}, upload=473338]: Uploading piece... Dec 28 03:13:25.036677997 (539h44m24.907761032s) [noise=true , address=77.90.31.19:4844 , piece_id=P4F3BTTOO6PBRTTOVFPQWDA4AGY5NJGN2NL2367NWTWL3M5LH7VA, seg_pos={0 0}, upload=473338]: Uploading piece... Dec 28 03:13:25.037884944 (539h44m24.908967979s) [noise=true , address=145.79.227.36:4518 , piece_id=AIY3T2CZ3QACP6AZSMIDOUE67FQQX2G7GX4FDYLOVSHIKA4PG5UQ, seg_pos={0 0}, upload=473338]: Uploading piece... Dec 28 03:13:26.747185946 (539h44m26.618268981s) [upload=473338]: Next returned nil segment Dec 28 03:13:26.747214861 (539h44m26.618297896s) [upload=473338]: Waiting for error group managing segments... Dec 28 03:13:27.040312213 (539h44m26.911395248s) [noise=true , address=145.79.227.7:4295 , piece_id=PBJ3THOXMIH2VV774XKN5MRHU3V7Y4U4XDJ44ZIYAIXFOBQPXNMQ, seg_pos={0 0}, upload=473338]: Done uploading piece. err: <nil> Dec 28 03:13:27.042317661 (539h44m26.913400706s) [noise=true , address=145.79.227.45:4229 , piece_id=XTAQTKMGBIQZKUMPKBPHGUA6CLB6IG4IV7QDR7TN5OYR3V3LCZSQ, seg_pos={0 0}, upload=473338]: Done uploading piece. err: <nil> Dec 28 03:13:27.045111051 (539h44m26.916194086s) [noise=true , address=145.79.227.9:4365 , piece_id=LBAQP24LWFED5FVWM3O4AOQG5FEJV4QEJMN4TXWQEGMFUO5ZPXWA, seg_pos={0 0}, upload=473338]: Done uploading piece. err: <nil> Dec 28 03:13:27.046221555 (539h44m26.9173046s) [noise=true , address=145.79.227.47:4265 , piece_id=4XSSHNNPBAEWQAQYQRYKJLGCYBESL7UZJB6WMHQ4TXMFVI74MPOQ, seg_pos={0 0}, upload=473338]: Done uploading piece. err: <nil> Dec 28 03:13:27.050783793 (539h44m26.921866828s) [noise=true , address=145.79.227.46:4602 , piece_id=2Y7EFFRBFEWYDYDIXJTJPMLQMPXUSR2WUBXFKGNLJS6LDAA2GMNQ, seg_pos={0 0}, upload=473338]: Done uploading piece. err: <nil> Dec 28 03:13:27.058134860 (539h44m26.929217905s) [noise=true , address=145.79.227.48:4302 , piece_id=YLNEL2JBXOYG5FF3XCFFGOTERETDWQH5VRMGD35N5B6POR25EQBQ, seg_pos={0 0}, upload=473338]: Done uploading piece. err: <nil> Dec 28 03:13:27.117068719 (539h44m26.988151764s) [noise=true , address=145.79.227.36:4518 , piece_id=AIY3T2CZ3QACP6AZSMIDOUE67FQQX2G7GX4FDYLOVSHIKA4PG5UQ, seg_pos={0 0}, upload=473338]: Done uploading piece. err: <nil> Dec 28 03:13:27.166645712 (539h44m27.037728747s) [noise=true , address=145.79.227.49:4338 , piece_id=5A33ZMUQPVGJCEFA55YVRXVYFDRT2OEKQIBHMJTAEOS3GV6LWXUA, seg_pos={0 0}, upload=473338]: Done uploading piece. err: <nil> Dec 28 03:13:27.220210707 (539h44m27.091293762s) [noise=true , address=77.90.31.10:4519 , piece_id=4NADR64IPV4DF7WAEV3XBSVJTUBBTAP3DT6YMEJ6JMR65I7CU6AA, seg_pos={0 0}, upload=473338]: Done uploading piece. err: <nil> Dec 28 03:13:27.310015118 (539h44m27.181098153s) [noise=true , address=45.91.116.6:4974 , piece_id=5Q5MUZXLI3R73RHT7S3NABINZMV6RAAXMZ4OAE735HGS3H6IU6RA, seg_pos={0 0}, upload=473338]: Done uploading piece. err: <nil> Dec 28 03:13:27.339789784 (539h44m27.210872839s) [noise=true , address=77.90.31.19:4844 , piece_id=P4F3BTTOO6PBRTTOVFPQWDA4AGY5NJGN2NL2367NWTWL3M5LH7VA, seg_pos={0 0}, upload=473338]: Done uploading piece. err: <nil> Dec 28 03:13:27.367435306 (539h44m27.238518351s) [noise=true , address=145.79.227.19:4078 , piece_id=SJPKATYGSBIUFAF4UVVOQ52G5O5MMDGO32YXINHSYU3CFPZNYKZQ, seg_pos={0 0}, upload=473338]: Done uploading piece. err: <nil> Dec 28 03:13:27.973514564 (539h44m27.844597609s) [noise=true , address=45.91.116.13:4156 , piece_id=K2DIVO3V3666IDCZKMOESIDUZCTBLJRBJ2DWFWKQUDKPYO52IAMQ, seg_pos={0 0}, upload=473338]: Done uploading piece. err: <nil> Dec 28 03:13:27.973539121 (539h44m27.844622156s) [seg_pos={0 0}, upload=473338]: Segment reached optimal threshold of 13 pieces. Dec 28 03:13:27.973778365 (539h44m27.844861411s) [noise=true , address=145.79.227.33:4870 , piece_id=Q7TCXIJELJXOMKWDGIAIFHASVMNLED2BHXYLY6S747DA7MWYJH4A, seg_pos={0 0}, upload=473338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 28 03:13:27.973787884 (539h44m27.844870929s) [noise=true , address=145.79.227.33:4870 , piece_id=Q7TCXIJELJXOMKWDGIAIFHASVMNLED2BHXYLY6S747DA7MWYJH4A, seg_pos={0 0}, upload=473338]: Long tail cancellation due to optimal threshold Dec 28 03:13:27.973815656 (539h44m27.844898702s) [noise=true , address=145.79.227.37:4566 , piece_id=K2RNZSSF7J4KOLFL4OKXO4BISMTLHMVHCF24RBSYRTEMF34EZQIA, seg_pos={0 0}, upload=473338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SaNJZaVix7BSbEWhu2Vmt3XjDDWCsQ8wTSiFR7MQ8oGwD9N2F): protocol: write tcp 188.166.225.170:36272->145.79.227.37:4566: use of closed network connection; write tcp 188.166.225.170:36272->145.79.227.37:4566: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36272->145.79.227.37:4566: use of closed network connection Dec 28 03:13:27.973824183 (539h44m27.844907218s) [noise=true , address=145.79.227.37:4566 , piece_id=K2RNZSSF7J4KOLFL4OKXO4BISMTLHMVHCF24RBSYRTEMF34EZQIA, seg_pos={0 0}, upload=473338]: Long tail cancellation due to optimal threshold Dec 28 03:13:27.973990980 (539h44m27.845074015s) [noise=true , address=45.91.116.10:4084 , piece_id=EOV6JLWAUEG4DB74VDZO2DJEEGKL5GNFH6XJK2C3CPO42YD66PLA, seg_pos={0 0}, upload=473338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 28 03:13:27.973998024 (539h44m27.84508106s) [noise=true , address=45.91.116.10:4084 , piece_id=EOV6JLWAUEG4DB74VDZO2DJEEGKL5GNFH6XJK2C3CPO42YD66PLA, seg_pos={0 0}, upload=473338]: Long tail cancellation due to optimal threshold Dec 28 03:13:27.974013163 (539h44m27.845096208s) [seg_pos={0 0}, upload=473338]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 03:13:27.974088616 (539h44m27.845171651s) [upload=473338]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 03:13:27.974094608 (539h44m27.845177643s) [upload=473338]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 03:13:28.262843260 (539h44m28.133926305s) [upload=473338]: Done uploading segments... Dec 28 03:13:28.262867547 (539h44m28.133950592s) [upload=473338]: Upload finished. err: <nil> Duration: 30.003312208s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [.........X....X............X........X....X...........X.....X] duration: 5.001298017s trace: https://jaeger.dev.storj.io/trace/2a8b91f06217fa82 FAIL (2025-12-28 03:13:53 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.0013203s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 37.215024071s trace: https://jaeger.dev.storj.io/trace/1822be52d200e5b5 FAIL (2025-12-28 03:13:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1ZAfYZ4wgpZVvatj6wtHona3hTaPcLxAmXC1DT2uKQyo8nLQQ7, Piece ID: 5MMQ44MXRT4XEL4X6WO54WRG2XYMX3AN3FNQ2H2XIUTKBXI3EEPA) context deadline exceeded; piecestore: (Node ID: 124guAM82oibYZsMho67Em6Ke6yEZXwT6LPMLLSZZBLqwoFGUkn, Piece ID: 22WJIPX4SM5NTIAW5ZRM2UKX32CTDBFCXPCCHRRWRCUCOF5CRE5Q) context deadline exceeded; piecestore: (Node ID: 1e6p1TD5CqUyWdZjkrxh2kRqMuyDiMUr9eimfeayDAKwhtBrZ8, Piece ID: XRI5EJFN7YYUG4QKLDGEFT6YAUHCUTTSXF735UDR6LZZBGMAZDJQ) context deadline exceeded ======LOGS====== 5.234848204 (538h55m35.105931239s) [noise=true , address=77.90.31.11:4572 , piece_id=5MMQ44MXRT4XEL4X6WO54WRG2XYMX3AN3FNQ2H2XIUTKBXI3EEPA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.235412057 (538h55m35.106495102s) [noise=true , address=145.79.227.22:4512 , piece_id=D46YLTIKGHG36DQ4YBKBSUDV4GGB4WVDWNOJS27IYYLN6AYZ5QFQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.235820946 (538h55m35.106903981s) [noise=true , address=77.90.31.19:4874 , piece_id=RHZUXUYMJU6RKQGSDZM5WDWIN5SQ7YIRSRATNNKT5GEMU2QH444A, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.236192192 (538h55m35.107275237s) [noise=true , address=77.90.31.16:4760 , piece_id=XRI5EJFN7YYUG4QKLDGEFT6YAUHCUTTSXF735UDR6LZZBGMAZDJQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.236564772 (538h55m35.107647817s) [noise=true , address=145.79.227.40:4062 , piece_id=PW6QWIPQWVUETLC6DJ624G6IB4P6EM4SYB72NAMQ4V6NUKNLUYLQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.236929766 (538h55m35.108012812s) [noise=true , address=64.49.13.139:4480 , piece_id=JAJY3F7IDFFMFKCK2QJV6W74RIESLRKBLXMN27Q3C45JWEPLBZLQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.237916846 (538h55m35.108999881s) [noise=true , address=145.79.227.34:4932 , piece_id=UPXSCQ45QDGPKNJXRFJ33EAA3QENVHRVQARU5TFJTEWICZNR253Q, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.238852948 (538h55m35.109935983s) [noise=true , address=45.91.116.12:4257 , piece_id=ZI67LIAHZW4G357GWKDPTB5SMXYGNXZTTTDMNAETDS2B6SWVDAEQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239152910 (538h55m35.110235945s) [noise=true , address=145.79.227.44:4222 , piece_id=CZ2CS62CTHQFAIE3S3JO2N3VJ4XIR5YJ2LMRVLLX7I4VM2VZVQVA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239409969 (538h55m35.110493004s) [noise=true , address=145.79.227.30:4773 , piece_id=KUGJXZCNMUE7WSK3VLY4AUBSOT4H6IY4XB4NCBGAGCNUG7F7IODQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239681136 (538h55m35.110764171s) [noise=true , address=77.90.31.13:4635 , piece_id=IIOY5X2W3XW4C2OFEBRCABMYFBI24RRTYNUSKN6KJU7CTUW4KONQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239954676 (538h55m35.111037711s) [noise=true , address=45.91.116.9:4281 , piece_id=JZLDW777CBX2BP7MLOQV4NFP3H4XJVCNQXTHNSUWPVDOJLPZK36A, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.240189914 (538h55m35.111272949s) [noise=true , address=145.79.227.35:4512 , piece_id=3E3VCCC45XVT5FHFDKXSTGGAGYBEML3BKU6JJTT33LCD2YMOMBZQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.240449147 (538h55m35.111532182s) [noise=true , address=45.91.116.8:4064 , piece_id=22WJIPX4SM5NTIAW5ZRM2UKX32CTDBFCXPCCHRRWRCUCOF5CRE5Q, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.240639640 (538h55m35.111722675s) [noise=true , address=145.79.227.32:4853 , piece_id=MBNCFDBQ4SZZXV4F7J5TRMOUYUQ5IPKQCGYGGM5SOIFJ6ZYCAUYA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.241137798 (538h55m35.112220833s) [noise=true , address=77.90.31.16:4751 , piece_id=N3Z2LLMCIA2F42VXZA7IDRU4XEKGOICH6TRVERODU76MFFE4QSHA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.241383797 (538h55m35.112466842s) [noise=true , address=77.90.31.18:4836 , piece_id=BSVEFI52EKN44CMINFUSZBNOH53IW5TXLVTF5IH6ZUTNII6RRLVA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:36.592880816 (538h55m36.463963871s) [upload=472631]: Next returned nil segment Dec 28 02:24:36.592901625 (538h55m36.46398467s) [upload=472631]: Waiting for error group managing segments... Dec 28 02:24:36.946933741 (538h55m36.818016786s) [noise=true , address=64.49.13.139:4480 , piece_id=JAJY3F7IDFFMFKCK2QJV6W74RIESLRKBLXMN27Q3C45JWEPLBZLQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.128189692 (538h55m36.999272737s) [noise=true , address=145.79.227.35:4512 , piece_id=3E3VCCC45XVT5FHFDKXSTGGAGYBEML3BKU6JJTT33LCD2YMOMBZQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.160760084 (538h55m37.031843119s) [noise=true , address=145.79.227.34:4932 , piece_id=UPXSCQ45QDGPKNJXRFJ33EAA3QENVHRVQARU5TFJTEWICZNR253Q, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.288838207 (538h55m37.159921252s) [noise=true , address=77.90.31.11:4572 , piece_id=5MMQ44MXRT4XEL4X6WO54WRG2XYMX3AN3FNQ2H2XIUTKBXI3EEPA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.319996310 (538h55m37.191079345s) [noise=true , address=145.79.227.22:4512 , piece_id=D46YLTIKGHG36DQ4YBKBSUDV4GGB4WVDWNOJS27IYYLN6AYZ5QFQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.320513273 (538h55m37.191596319s) [noise=true , address=77.90.31.18:4836 , piece_id=BSVEFI52EKN44CMINFUSZBNOH53IW5TXLVTF5IH6ZUTNII6RRLVA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.336522926 (538h55m37.207605971s) [noise=true , address=145.79.227.32:4853 , piece_id=MBNCFDBQ4SZZXV4F7J5TRMOUYUQ5IPKQCGYGGM5SOIFJ6ZYCAUYA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.384868031 (538h55m37.255951066s) [noise=true , address=77.90.31.16:4751 , piece_id=N3Z2LLMCIA2F42VXZA7IDRU4XEKGOICH6TRVERODU76MFFE4QSHA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.453563554 (538h55m37.324646599s) [noise=true , address=45.91.116.8:4064 , piece_id=22WJIPX4SM5NTIAW5ZRM2UKX32CTDBFCXPCCHRRWRCUCOF5CRE5Q, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.479454839 (538h55m37.350537884s) [noise=true , address=45.91.116.12:4257 , piece_id=ZI67LIAHZW4G357GWKDPTB5SMXYGNXZTTTDMNAETDS2B6SWVDAEQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.659710174 (538h55m37.530793219s) [noise=true , address=77.90.31.13:4635 , piece_id=IIOY5X2W3XW4C2OFEBRCABMYFBI24RRTYNUSKN6KJU7CTUW4KONQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.715039128 (538h55m37.586122163s) [noise=true , address=77.90.31.16:4760 , piece_id=XRI5EJFN7YYUG4QKLDGEFT6YAUHCUTTSXF735UDR6LZZBGMAZDJQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.977473406 (538h55m37.848556441s) [noise=true , address=77.90.31.19:4874 , piece_id=RHZUXUYMJU6RKQGSDZM5WDWIN5SQ7YIRSRATNNKT5GEMU2QH444A, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:38.016302745 (538h55m37.88738578s) [noise=true , address=145.79.227.30:4773 , piece_id=KUGJXZCNMUE7WSK3VLY4AUBSOT4H6IY4XB4NCBGAGCNUG7F7IODQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:38.016326982 (538h55m37.887410027s) [seg_pos={0 0}, upload=472631]: Segment reached optimal threshold of 14 pieces. Dec 28 02:24:38.016585353 (538h55m37.887668398s) [noise=true , address=145.79.227.30:4766 , piece_id=67M2DMPAVWIQKQSWUAG73D4E4NRRSG4XEVPOUVIJPEQOUV7A6VVA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1k6zdamuKPkBdzyNZNWx4FCmaTkv33PchcwvTRR8bgcETBs4K7): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016594450 (538h55m37.887677495s) [noise=true , address=145.79.227.30:4766 , piece_id=67M2DMPAVWIQKQSWUAG73D4E4NRRSG4XEVPOUVIJPEQOUV7A6VVA, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016627093 (538h55m37.887710148s) [noise=true , address=45.91.116.9:4281 , piece_id=JZLDW777CBX2BP7MLOQV4NFP3H4XJVCNQXTHNSUWPVDOJLPZK36A, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gxf7DpJZthyR6f1PExX56tJavjhPw9Gxb4h8mtmeYKW3t3tS1): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016635258 (538h55m37.887718293s) [noise=true , address=45.91.116.9:4281 , piece_id=JZLDW777CBX2BP7MLOQV4NFP3H4XJVCNQXTHNSUWPVDOJLPZK36A, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016871298 (538h55m37.887954363s) [noise=true , address=145.79.227.44:4222 , piece_id=CZ2CS62CTHQFAIE3S3JO2N3VJ4XIR5YJ2LMRVLLX7I4VM2VZVQVA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZKW35KvrsoYRefbYeUFZo8bh4XZYycNKGEvTFjzibezjXgj3e): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016880124 (538h55m37.887963169s) [noise=true , address=145.79.227.44:4222 , piece_id=CZ2CS62CTHQFAIE3S3JO2N3VJ4XIR5YJ2LMRVLLX7I4VM2VZVQVA, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016934839 (538h55m37.888017884s) [noise=true , address=145.79.227.40:4062 , piece_id=PW6QWIPQWVUETLC6DJ624G6IB4P6EM4SYB72NAMQ4V6NUKNLUYLQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016943425 (538h55m37.88802649s) [noise=true , address=145.79.227.40:4062 , piece_id=PW6QWIPQWVUETLC6DJ624G6IB4P6EM4SYB72NAMQ4V6NUKNLUYLQ, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016968252 (538h55m37.888051297s) [seg_pos={0 0}, upload=472631]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 02:24:38.017067822 (538h55m37.888150867s) [upload=472631]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 02:24:38.017074244 (538h55m37.888157289s) [upload=472631]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 02:24:38.304306667 (538h55m38.175389712s) [upload=472631]: Done uploading segments... Dec 28 02:24:38.304329641 (538h55m38.175412686s) [upload=472631]: Upload finished. err: <nil> Duration: 37.215053838s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [..........X....X............X........X....X...........X.....] duration: 3.457708998s trace: https://jaeger.dev.storj.io/trace/235003f053bdab FAIL (2025-12-28 03:13:01 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.45773661s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 46 out of 60 is failed (threshold=3) [.XXXXXX.XXXX.XXXX..X..XXXXX.XXXXXXXXXXXXX.XXXXXXXXX.X..XX..X] duration: 30.004082765s trace: https://jaeger.dev.storj.io/trace/4b03ffb68234937 FAIL (2025-12-28 03:12:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: 3DIWCIENFZADNJZKZF7LN32PFFG3RIWA63HINHZXL6PROJVSPVAA) context deadline exceeded; piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: N3UHS5S7PGDD3RHIKIOFD4DUKT63KG6SBGXVWXLAEAB3NVOIVEVQ) context deadline exceeded ======LOGS====== egments... Dec 28 03:12:30.213469870 (539h43m30.084552916s) [upload=473321]: Got next segment. Inline: false Dec 28 03:12:30.213542228 (539h43m30.084625263s) [upload=473321]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 03:12:30.213547809 (539h43m30.084630844s) [upload=473321]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 03:12:30.490611890 (539h43m30.361694925s) [seg_pos={0 0}, upload=473321]: Begin upload segment... Dec 28 03:12:30.490672907 (539h43m30.361755942s) [seg_pos={0 0}, upload=473321]: Done begin upload segment. Dec 28 03:12:30.490704717 (539h43m30.361787753s) [noise=true , address=145.79.227.41:4086 , piece_id=3OF7QEXJAMMLFNNNEAIW4JEHPPWFFZ3FN2EXWXUSPLJNODOB7QPQ, seg_pos={0 0}, upload=473321]: Uploading piece... Dec 28 03:12:30.490758540 (539h43m30.361841585s) [noise=true , address=145.79.227.22:4511 , piece_id=EGYGR7C3AUCZRWAC5GV46POLSOYRFGA3URKTREKPOW273SGHCTZA, seg_pos={0 0}, upload=473321]: Uploading piece... Dec 28 03:12:30.490877867 (539h43m30.361960902s) [noise=true , address=45.91.116.8:4047 , piece_id=N3UHS5S7PGDD3RHIKIOFD4DUKT63KG6SBGXVWXLAEAB3NVOIVEVQ, seg_pos={0 0}, upload=473321]: Uploading piece... Dec 28 03:12:30.491567010 (539h43m30.362650045s) [noise=true , address=145.79.227.11:4732 , piece_id=HWCRDYX5QFNH3TQY4MUQXWODJRSZ2R35JGYWQIPPOVJN46CKPGUQ, seg_pos={0 0}, upload=473321]: Uploading piece... Dec 28 03:12:30.491728788 (539h43m30.362811823s) [noise=true , address=145.79.227.3:4150 , piece_id=PJ3RU7VR2EM2KE7FN4XBHBB663I6F3HFCE2GJP734YFSFZISP54A, seg_pos={0 0}, upload=473321]: Uploading piece... Dec 28 03:12:30.492059026 (539h43m30.363142061s) [noise=true , address=145.79.227.5:4223 , piece_id=CNQHX6CK42QQTDG2VQXBTQZQOWQP2TH6XRXN6F23YBCFLFES7UCA, seg_pos={0 0}, upload=473321]: Uploading piece... Dec 28 03:12:30.492385788 (539h43m30.363468823s) [noise=true , address=77.90.31.10:4518 , piece_id=JR3QKKJGURJH2IX47XWURGIUVPGPZ2K3EQANS45IFPRXKLQ3V67Q, seg_pos={0 0}, upload=473321]: Uploading piece... Dec 28 03:12:30.492714515 (539h43m30.36379755s) [noise=true , address=45.91.116.13:4156 , piece_id=3DIWCIENFZADNJZKZF7LN32PFFG3RIWA63HINHZXL6PROJVSPVAA, seg_pos={0 0}, upload=473321]: Uploading piece... Dec 28 03:12:30.493328965 (539h43m30.364412s) [noise=true , address=145.79.227.17:4971 , piece_id=BRO6LDVMKNQ2A547SCXN6AG6PWKIT7IZF2SENK7GSMWUG54GGCAQ, seg_pos={0 0}, upload=473321]: Uploading piece... Dec 28 03:12:30.493991577 (539h43m30.365074612s) [noise=true , address=145.79.227.9:4365 , piece_id=VNHXYQMYUQ3QF5NZKFISZ2SATTF6TJBOAUVIYBRA5VJOM2CHPRMQ, seg_pos={0 0}, upload=473321]: Uploading piece... Dec 28 03:12:30.499417237 (539h43m30.370500292s) [noise=true , address=145.79.227.15:4873 , piece_id=OFMCB7QUVNZXGN3HVB3LQHABJT6AYNT7UXSZKLMM5DNI6QG7ROMQ, seg_pos={0 0}, upload=473321]: Uploading piece... Dec 28 03:12:30.500999180 (539h43m30.372082225s) [noise=true , address=145.79.227.42:4122 , piece_id=TQOOX4AUROCJA24CXOUV7RFN5LWARSMYHMCJXT2S5DFQHFQ2RFQA, seg_pos={0 0}, upload=473321]: Uploading piece... Dec 28 03:12:30.502636688 (539h43m30.373719733s) [noise=true , address=145.79.227.13:4802 , piece_id=CZJ4NBP2YJBNGPJ2UK7ZEPD7JVNWWFCPVSSR2OOFYAFJZEQNS3NQ, seg_pos={0 0}, upload=473321]: Uploading piece... Dec 28 03:12:30.504210415 (539h43m30.37529346s) [noise=true , address=145.79.227.51:4411 , piece_id=LJETTY3C3JQBRRJBSUBMDJBWZJ7LWZCRCAECUD5FLK2DMTPYMBLA, seg_pos={0 0}, upload=473321]: Uploading piece... Dec 28 03:12:30.505853704 (539h43m30.376936749s) [noise=true , address=145.79.227.16:4909 , piece_id=FM7XOJ3VKTB4W53QWEE4TQXIWT5ZQXLKFJMMYJV37YPICPURSGNA, seg_pos={0 0}, upload=473321]: Uploading piece... Dec 28 03:12:30.506917729 (539h43m30.378000764s) [noise=true , address=77.90.31.21:4877 , piece_id=Y7SF6C2LOWHXK5YJQT3WQKLIB4OKJVZ2KKIJFVI4QI7HLXPCUSTQ, seg_pos={0 0}, upload=473321]: Uploading piece... Dec 28 03:12:32.286773639 (539h43m32.157856674s) [upload=473321]: Next returned nil segment Dec 28 03:12:32.286799838 (539h43m32.157882873s) [upload=473321]: Waiting for error group managing segments... Dec 28 03:12:32.563814937 (539h43m32.434897982s) [noise=true , address=145.79.227.41:4086 , piece_id=3OF7QEXJAMMLFNNNEAIW4JEHPPWFFZ3FN2EXWXUSPLJNODOB7QPQ, seg_pos={0 0}, upload=473321]: Done uploading piece. err: <nil> Dec 28 03:12:32.571206322 (539h43m32.442289357s) [noise=true , address=145.79.227.16:4909 , piece_id=FM7XOJ3VKTB4W53QWEE4TQXIWT5ZQXLKFJMMYJV37YPICPURSGNA, seg_pos={0 0}, upload=473321]: Done uploading piece. err: <nil> Dec 28 03:12:32.578367497 (539h43m32.449450532s) [noise=true , address=145.79.227.42:4122 , piece_id=TQOOX4AUROCJA24CXOUV7RFN5LWARSMYHMCJXT2S5DFQHFQ2RFQA, seg_pos={0 0}, upload=473321]: Done uploading piece. err: <nil> Dec 28 03:12:32.579598351 (539h43m32.450681387s) [noise=true , address=145.79.227.11:4732 , piece_id=HWCRDYX5QFNH3TQY4MUQXWODJRSZ2R35JGYWQIPPOVJN46CKPGUQ, seg_pos={0 0}, upload=473321]: Done uploading piece. err: <nil> Dec 28 03:12:32.580631518 (539h43m32.451714563s) [noise=true , address=145.79.227.15:4873 , piece_id=OFMCB7QUVNZXGN3HVB3LQHABJT6AYNT7UXSZKLMM5DNI6QG7ROMQ, seg_pos={0 0}, upload=473321]: Done uploading piece. err: <nil> Dec 28 03:12:32.583737041 (539h43m32.454820076s) [noise=true , address=145.79.227.13:4802 , piece_id=CZJ4NBP2YJBNGPJ2UK7ZEPD7JVNWWFCPVSSR2OOFYAFJZEQNS3NQ, seg_pos={0 0}, upload=473321]: Done uploading piece. err: <nil> Dec 28 03:12:32.587554521 (539h43m32.458637556s) [noise=true , address=145.79.227.51:4411 , piece_id=LJETTY3C3JQBRRJBSUBMDJBWZJ7LWZCRCAECUD5FLK2DMTPYMBLA, seg_pos={0 0}, upload=473321]: Done uploading piece. err: <nil> Dec 28 03:12:32.607197144 (539h43m32.478280179s) [noise=true , address=145.79.227.5:4223 , piece_id=CNQHX6CK42QQTDG2VQXBTQZQOWQP2TH6XRXN6F23YBCFLFES7UCA, seg_pos={0 0}, upload=473321]: Done uploading piece. err: <nil> Dec 28 03:12:32.632768726 (539h43m32.503851771s) [noise=true , address=145.79.227.22:4511 , piece_id=EGYGR7C3AUCZRWAC5GV46POLSOYRFGA3URKTREKPOW273SGHCTZA, seg_pos={0 0}, upload=473321]: Done uploading piece. err: <nil> Dec 28 03:12:32.634145167 (539h43m32.505228212s) [noise=true , address=145.79.227.3:4150 , piece_id=PJ3RU7VR2EM2KE7FN4XBHBB663I6F3HFCE2GJP734YFSFZISP54A, seg_pos={0 0}, upload=473321]: Done uploading piece. err: <nil> Dec 28 03:12:32.638311149 (539h43m32.509394184s) [noise=true , address=145.79.227.9:4365 , piece_id=VNHXYQMYUQ3QF5NZKFISZ2SATTF6TJBOAUVIYBRA5VJOM2CHPRMQ, seg_pos={0 0}, upload=473321]: Done uploading piece. err: <nil> Dec 28 03:12:32.732150364 (539h43m32.603233399s) [noise=true , address=45.91.116.13:4156 , piece_id=3DIWCIENFZADNJZKZF7LN32PFFG3RIWA63HINHZXL6PROJVSPVAA, seg_pos={0 0}, upload=473321]: Done uploading piece. err: <nil> Dec 28 03:12:32.750896349 (539h43m32.621979384s) [noise=true , address=45.91.116.8:4047 , piece_id=N3UHS5S7PGDD3RHIKIOFD4DUKT63KG6SBGXVWXLAEAB3NVOIVEVQ, seg_pos={0 0}, upload=473321]: Done uploading piece. err: <nil> Dec 28 03:12:32.750921457 (539h43m32.622004492s) [seg_pos={0 0}, upload=473321]: Segment reached optimal threshold of 13 pieces. Dec 28 03:12:32.751141034 (539h43m32.62222407s) [noise=true , address=145.79.227.17:4971 , piece_id=BRO6LDVMKNQ2A547SCXN6AG6PWKIT7IZF2SENK7GSMWUG54GGCAQ, seg_pos={0 0}, upload=473321]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: write tcp 188.166.225.170:55952->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:55952->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55952->145.79.227.17:4971: use of closed network connection Dec 28 03:12:32.751147277 (539h43m32.622230322s) [noise=true , address=145.79.227.17:4971 , piece_id=BRO6LDVMKNQ2A547SCXN6AG6PWKIT7IZF2SENK7GSMWUG54GGCAQ, seg_pos={0 0}, upload=473321]: Long tail cancellation due to optimal threshold Dec 28 03:12:32.751313693 (539h43m32.622396728s) [noise=true , address=77.90.31.21:4877 , piece_id=Y7SF6C2LOWHXK5YJQT3WQKLIB4OKJVZ2KKIJFVI4QI7HLXPCUSTQ, seg_pos={0 0}, upload=473321]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Dec 28 03:12:32.751330015 (539h43m32.62241305s) [noise=true , address=77.90.31.21:4877 , piece_id=Y7SF6C2LOWHXK5YJQT3WQKLIB4OKJVZ2KKIJFVI4QI7HLXPCUSTQ, seg_pos={0 0}, upload=473321]: Long tail cancellation due to optimal threshold Dec 28 03:12:32.751423754 (539h43m32.622506789s) [noise=true , address=77.90.31.10:4518 , piece_id=JR3QKKJGURJH2IX47XWURGIUVPGPZ2K3EQANS45IFPRXKLQ3V67Q, seg_pos={0 0}, upload=473321]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: write tcp 188.166.225.170:49402->77.90.31.10:4518: use of closed network connection; write tcp 188.166.225.170:49402->77.90.31.10:4518: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49402->77.90.31.10:4518: use of closed network connection Dec 28 03:12:32.751441838 (539h43m32.622524883s) [noise=true , address=77.90.31.10:4518 , piece_id=JR3QKKJGURJH2IX47XWURGIUVPGPZ2K3EQANS45IFPRXKLQ3V67Q, seg_pos={0 0}, upload=473321]: Long tail cancellation due to optimal threshold Dec 28 03:12:32.751492735 (539h43m32.62257577s) [seg_pos={0 0}, upload=473321]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 03:12:32.751594959 (539h43m32.622677994s) [upload=473321]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 03:12:32.751615208 (539h43m32.622698243s) [upload=473321]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 03:12:33.049992563 (539h43m32.921075608s) [upload=473321]: Done uploading segments... Dec 28 03:12:33.050019415 (539h43m32.92110246s) [upload=473321]: Upload finished. err: <nil> Duration: 30.004106761s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 35.035223347s trace: https://jaeger.dev.storj.io/trace/1822be52d200e5b5 FAIL (2025-12-28 03:12:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1ZAfYZ4wgpZVvatj6wtHona3hTaPcLxAmXC1DT2uKQyo8nLQQ7, Piece ID: 5MMQ44MXRT4XEL4X6WO54WRG2XYMX3AN3FNQ2H2XIUTKBXI3EEPA) context deadline exceeded; piecestore: (Node ID: 124guAM82oibYZsMho67Em6Ke6yEZXwT6LPMLLSZZBLqwoFGUkn, Piece ID: 22WJIPX4SM5NTIAW5ZRM2UKX32CTDBFCXPCCHRRWRCUCOF5CRE5Q) context deadline exceeded; piecestore: (Node ID: 1e6p1TD5CqUyWdZjkrxh2kRqMuyDiMUr9eimfeayDAKwhtBrZ8, Piece ID: XRI5EJFN7YYUG4QKLDGEFT6YAUHCUTTSXF735UDR6LZZBGMAZDJQ) context deadline exceeded ======LOGS====== 5.234848204 (538h55m35.105931239s) [noise=true , address=77.90.31.11:4572 , piece_id=5MMQ44MXRT4XEL4X6WO54WRG2XYMX3AN3FNQ2H2XIUTKBXI3EEPA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.235412057 (538h55m35.106495102s) [noise=true , address=145.79.227.22:4512 , piece_id=D46YLTIKGHG36DQ4YBKBSUDV4GGB4WVDWNOJS27IYYLN6AYZ5QFQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.235820946 (538h55m35.106903981s) [noise=true , address=77.90.31.19:4874 , piece_id=RHZUXUYMJU6RKQGSDZM5WDWIN5SQ7YIRSRATNNKT5GEMU2QH444A, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.236192192 (538h55m35.107275237s) [noise=true , address=77.90.31.16:4760 , piece_id=XRI5EJFN7YYUG4QKLDGEFT6YAUHCUTTSXF735UDR6LZZBGMAZDJQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.236564772 (538h55m35.107647817s) [noise=true , address=145.79.227.40:4062 , piece_id=PW6QWIPQWVUETLC6DJ624G6IB4P6EM4SYB72NAMQ4V6NUKNLUYLQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.236929766 (538h55m35.108012812s) [noise=true , address=64.49.13.139:4480 , piece_id=JAJY3F7IDFFMFKCK2QJV6W74RIESLRKBLXMN27Q3C45JWEPLBZLQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.237916846 (538h55m35.108999881s) [noise=true , address=145.79.227.34:4932 , piece_id=UPXSCQ45QDGPKNJXRFJ33EAA3QENVHRVQARU5TFJTEWICZNR253Q, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.238852948 (538h55m35.109935983s) [noise=true , address=45.91.116.12:4257 , piece_id=ZI67LIAHZW4G357GWKDPTB5SMXYGNXZTTTDMNAETDS2B6SWVDAEQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239152910 (538h55m35.110235945s) [noise=true , address=145.79.227.44:4222 , piece_id=CZ2CS62CTHQFAIE3S3JO2N3VJ4XIR5YJ2LMRVLLX7I4VM2VZVQVA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239409969 (538h55m35.110493004s) [noise=true , address=145.79.227.30:4773 , piece_id=KUGJXZCNMUE7WSK3VLY4AUBSOT4H6IY4XB4NCBGAGCNUG7F7IODQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239681136 (538h55m35.110764171s) [noise=true , address=77.90.31.13:4635 , piece_id=IIOY5X2W3XW4C2OFEBRCABMYFBI24RRTYNUSKN6KJU7CTUW4KONQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239954676 (538h55m35.111037711s) [noise=true , address=45.91.116.9:4281 , piece_id=JZLDW777CBX2BP7MLOQV4NFP3H4XJVCNQXTHNSUWPVDOJLPZK36A, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.240189914 (538h55m35.111272949s) [noise=true , address=145.79.227.35:4512 , piece_id=3E3VCCC45XVT5FHFDKXSTGGAGYBEML3BKU6JJTT33LCD2YMOMBZQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.240449147 (538h55m35.111532182s) [noise=true , address=45.91.116.8:4064 , piece_id=22WJIPX4SM5NTIAW5ZRM2UKX32CTDBFCXPCCHRRWRCUCOF5CRE5Q, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.240639640 (538h55m35.111722675s) [noise=true , address=145.79.227.32:4853 , piece_id=MBNCFDBQ4SZZXV4F7J5TRMOUYUQ5IPKQCGYGGM5SOIFJ6ZYCAUYA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.241137798 (538h55m35.112220833s) [noise=true , address=77.90.31.16:4751 , piece_id=N3Z2LLMCIA2F42VXZA7IDRU4XEKGOICH6TRVERODU76MFFE4QSHA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.241383797 (538h55m35.112466842s) [noise=true , address=77.90.31.18:4836 , piece_id=BSVEFI52EKN44CMINFUSZBNOH53IW5TXLVTF5IH6ZUTNII6RRLVA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:36.592880816 (538h55m36.463963871s) [upload=472631]: Next returned nil segment Dec 28 02:24:36.592901625 (538h55m36.46398467s) [upload=472631]: Waiting for error group managing segments... Dec 28 02:24:36.946933741 (538h55m36.818016786s) [noise=true , address=64.49.13.139:4480 , piece_id=JAJY3F7IDFFMFKCK2QJV6W74RIESLRKBLXMN27Q3C45JWEPLBZLQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.128189692 (538h55m36.999272737s) [noise=true , address=145.79.227.35:4512 , piece_id=3E3VCCC45XVT5FHFDKXSTGGAGYBEML3BKU6JJTT33LCD2YMOMBZQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.160760084 (538h55m37.031843119s) [noise=true , address=145.79.227.34:4932 , piece_id=UPXSCQ45QDGPKNJXRFJ33EAA3QENVHRVQARU5TFJTEWICZNR253Q, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.288838207 (538h55m37.159921252s) [noise=true , address=77.90.31.11:4572 , piece_id=5MMQ44MXRT4XEL4X6WO54WRG2XYMX3AN3FNQ2H2XIUTKBXI3EEPA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.319996310 (538h55m37.191079345s) [noise=true , address=145.79.227.22:4512 , piece_id=D46YLTIKGHG36DQ4YBKBSUDV4GGB4WVDWNOJS27IYYLN6AYZ5QFQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.320513273 (538h55m37.191596319s) [noise=true , address=77.90.31.18:4836 , piece_id=BSVEFI52EKN44CMINFUSZBNOH53IW5TXLVTF5IH6ZUTNII6RRLVA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.336522926 (538h55m37.207605971s) [noise=true , address=145.79.227.32:4853 , piece_id=MBNCFDBQ4SZZXV4F7J5TRMOUYUQ5IPKQCGYGGM5SOIFJ6ZYCAUYA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.384868031 (538h55m37.255951066s) [noise=true , address=77.90.31.16:4751 , piece_id=N3Z2LLMCIA2F42VXZA7IDRU4XEKGOICH6TRVERODU76MFFE4QSHA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.453563554 (538h55m37.324646599s) [noise=true , address=45.91.116.8:4064 , piece_id=22WJIPX4SM5NTIAW5ZRM2UKX32CTDBFCXPCCHRRWRCUCOF5CRE5Q, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.479454839 (538h55m37.350537884s) [noise=true , address=45.91.116.12:4257 , piece_id=ZI67LIAHZW4G357GWKDPTB5SMXYGNXZTTTDMNAETDS2B6SWVDAEQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.659710174 (538h55m37.530793219s) [noise=true , address=77.90.31.13:4635 , piece_id=IIOY5X2W3XW4C2OFEBRCABMYFBI24RRTYNUSKN6KJU7CTUW4KONQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.715039128 (538h55m37.586122163s) [noise=true , address=77.90.31.16:4760 , piece_id=XRI5EJFN7YYUG4QKLDGEFT6YAUHCUTTSXF735UDR6LZZBGMAZDJQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.977473406 (538h55m37.848556441s) [noise=true , address=77.90.31.19:4874 , piece_id=RHZUXUYMJU6RKQGSDZM5WDWIN5SQ7YIRSRATNNKT5GEMU2QH444A, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:38.016302745 (538h55m37.88738578s) [noise=true , address=145.79.227.30:4773 , piece_id=KUGJXZCNMUE7WSK3VLY4AUBSOT4H6IY4XB4NCBGAGCNUG7F7IODQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:38.016326982 (538h55m37.887410027s) [seg_pos={0 0}, upload=472631]: Segment reached optimal threshold of 14 pieces. Dec 28 02:24:38.016585353 (538h55m37.887668398s) [noise=true , address=145.79.227.30:4766 , piece_id=67M2DMPAVWIQKQSWUAG73D4E4NRRSG4XEVPOUVIJPEQOUV7A6VVA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1k6zdamuKPkBdzyNZNWx4FCmaTkv33PchcwvTRR8bgcETBs4K7): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016594450 (538h55m37.887677495s) [noise=true , address=145.79.227.30:4766 , piece_id=67M2DMPAVWIQKQSWUAG73D4E4NRRSG4XEVPOUVIJPEQOUV7A6VVA, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016627093 (538h55m37.887710148s) [noise=true , address=45.91.116.9:4281 , piece_id=JZLDW777CBX2BP7MLOQV4NFP3H4XJVCNQXTHNSUWPVDOJLPZK36A, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gxf7DpJZthyR6f1PExX56tJavjhPw9Gxb4h8mtmeYKW3t3tS1): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016635258 (538h55m37.887718293s) [noise=true , address=45.91.116.9:4281 , piece_id=JZLDW777CBX2BP7MLOQV4NFP3H4XJVCNQXTHNSUWPVDOJLPZK36A, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016871298 (538h55m37.887954363s) [noise=true , address=145.79.227.44:4222 , piece_id=CZ2CS62CTHQFAIE3S3JO2N3VJ4XIR5YJ2LMRVLLX7I4VM2VZVQVA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZKW35KvrsoYRefbYeUFZo8bh4XZYycNKGEvTFjzibezjXgj3e): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016880124 (538h55m37.887963169s) [noise=true , address=145.79.227.44:4222 , piece_id=CZ2CS62CTHQFAIE3S3JO2N3VJ4XIR5YJ2LMRVLLX7I4VM2VZVQVA, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016934839 (538h55m37.888017884s) [noise=true , address=145.79.227.40:4062 , piece_id=PW6QWIPQWVUETLC6DJ624G6IB4P6EM4SYB72NAMQ4V6NUKNLUYLQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016943425 (538h55m37.88802649s) [noise=true , address=145.79.227.40:4062 , piece_id=PW6QWIPQWVUETLC6DJ624G6IB4P6EM4SYB72NAMQ4V6NUKNLUYLQ, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016968252 (538h55m37.888051297s) [seg_pos={0 0}, upload=472631]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 02:24:38.017067822 (538h55m37.888150867s) [upload=472631]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 02:24:38.017074244 (538h55m37.888157289s) [upload=472631]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 02:24:38.304306667 (538h55m38.175389712s) [upload=472631]: Done uploading segments... Dec 28 02:24:38.304329641 (538h55m38.175412686s) [upload=472631]: Upload finished. err: <nil> Duration: 35.035253083s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [...........X....X............X........X....X...........X....] duration: 3.480383439s trace: https://jaeger.dev.storj.io/trace/235003f053bdab FAIL (2025-12-28 03:12:07 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.480411132s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 46 out of 60 is failed (threshold=3) [X.XXXXXX.XXXX.XXXX..X..XXXXX.XXXXXXXXXXXXX.XXXXXXXXX.X..XX..] duration: 18.726340195s trace: https://jaeger.dev.storj.io/trace/30fe8cdffa81d60d FAIL (2025-12-28 03:11:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: QE4BN77KQCTWG6O4VIEN4JMWBRJ27DLOWX4BCJDJMKRLJITIPK5Q) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: J3A5ANOJLZBBBX5EOFPWRTVLJ4MOPJWO2DAUHSQRYJSJNKOKUBPQ) context deadline exceeded ======LOGS====== 28 03:09:31.842900787 (539h40m31.713983822s) [upload=473276]: Starting upload Dec 28 03:09:31.842985860 (539h40m31.714068915s) [upload=473276]: Done starting upload Dec 28 03:09:31.843136727 (539h40m31.714219782s) [upload=473276]: Uploading segments... Dec 28 03:09:31.843263529 (539h40m31.714346574s) [upload=473276]: Got next segment. Inline: false Dec 28 03:09:31.843309576 (539h40m31.714392641s) [upload=473276]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 03:09:31.843316148 (539h40m31.714399193s) [upload=473276]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 03:09:32.091638863 (539h40m31.962721908s) [seg_pos={0 0}, upload=473276]: Begin upload segment... Dec 28 03:09:32.091684179 (539h40m31.962767214s) [seg_pos={0 0}, upload=473276]: Done begin upload segment. Dec 28 03:09:32.091709558 (539h40m31.962792593s) [noise=true , address=45.91.116.9:4261 , piece_id=QE4BN77KQCTWG6O4VIEN4JMWBRJ27DLOWX4BCJDJMKRLJITIPK5Q, seg_pos={0 0}, upload=473276]: Uploading piece... Dec 28 03:09:32.091732532 (539h40m31.962815577s) [noise=true , address=145.79.227.13:4802 , piece_id=IDDNJMKDGWBL53RTXHZ27W7M6WANVSC76T46MDMNOSVBKYLER34Q, seg_pos={0 0}, upload=473276]: Uploading piece... Dec 28 03:09:32.092360778 (539h40m31.963443813s) [noise=true , address=45.91.116.11:4117 , piece_id=J3A5ANOJLZBBBX5EOFPWRTVLJ4MOPJWO2DAUHSQRYJSJNKOKUBPQ, seg_pos={0 0}, upload=473276]: Uploading piece... Dec 28 03:09:32.092821214 (539h40m31.96390426s) [noise=true , address=145.79.227.7:4295 , piece_id=ZS55RLW2XXOXUK64T4ABWCZJ5J6WX32Z3PYG7NZW7XDH2DQGQVEA, seg_pos={0 0}, upload=473276]: Uploading piece... Dec 28 03:09:32.093210165 (539h40m31.9642932s) [noise=true , address=145.79.227.36:4517 , piece_id=W2DV45TQYAZL3V4ZKGFXUXYVMG7DHHU4QTZ4WWAMMDYQ5OLEFKDQ, seg_pos={0 0}, upload=473276]: Uploading piece... Dec 28 03:09:32.093484267 (539h40m31.964567302s) [noise=true , address=77.90.31.21:4880 , piece_id=QFJVHJIC3UTM67CEQHYMHKEG6AOGBAM5P6T475S4BXIZ2O752SLQ, seg_pos={0 0}, upload=473276]: Uploading piece... Dec 28 03:09:32.093886552 (539h40m31.964969598s) [noise=true , address=145.79.227.41:4085 , piece_id=X72JKDXT6S7CZOE3MDNRVDHF5TIO2V2MGS6GPQ2EEW3DSX7EFZQQ, seg_pos={0 0}, upload=473276]: Uploading piece... Dec 28 03:09:32.095140079 (539h40m31.966223114s) [noise=true , address=77.90.31.10:4519 , piece_id=BTQVCUIINZPJQGB7EM4ZTEYACDOFN7IWKEI3L6ONYGH2IEOTZSNA, seg_pos={0 0}, upload=473276]: Uploading piece... Dec 28 03:09:32.095358536 (539h40m31.966441571s) [noise=true , address=145.79.227.45:4231 , piece_id=3VCTP37YAVDZTCLW7R62D6JHDOS62YJOW4FNL2RS36LERGCMXOJA, seg_pos={0 0}, upload=473276]: Uploading piece... Dec 28 03:09:32.096249953 (539h40m31.967332988s) [noise=true , address=45.91.116.5:4298 , piece_id=4T7UVHYLPRJLUUHLUPSQUXAEUXH2Y2ZWENWV7VFEWZDM76HOHK3Q, seg_pos={0 0}, upload=473276]: Uploading piece... Dec 28 03:09:32.097530952 (539h40m31.968613987s) [noise=true , address=145.79.227.42:4121 , piece_id=7QF7YSXAF6F4AGXTTXYBFZ4UXXBAQZCEKI2TAUMQJDXOQ3MDEILQ, seg_pos={0 0}, upload=473276]: Uploading piece... Dec 28 03:09:32.101540176 (539h40m31.972623222s) [noise=true , address=145.79.227.38:4978 , piece_id=VN6FCKACQRZ7MWPUIAUURPHCNML6DBRHUQ4LXNCTBFSVSNXMKLWA, seg_pos={0 0}, upload=473276]: Uploading piece... Dec 28 03:09:32.103185900 (539h40m31.974268935s) [noise=true , address=77.90.31.13:4629 , piece_id=ZAQCKXGM45ODLTTLH4GARCW3RRMY3PN3DYUKIPYKGLFY2AWJLYVA, seg_pos={0 0}, upload=473276]: Uploading piece... Dec 28 03:09:32.104285855 (539h40m31.97536889s) [noise=true , address=45.91.116.14:4189 , piece_id=E6GZY3XPIL4ZPOBA273UAM2O45OB5A2ALGK2ODWNCE7WDJU3HYCA, seg_pos={0 0}, upload=473276]: Uploading piece... Dec 28 03:09:32.105979329 (539h40m31.977062364s) [noise=true , address=145.79.227.15:4874 , piece_id=M5G7FSNIBFQ7HMPFUEQ5JD6FL53G6LQADAWGK5Q2CWPZISZUT3TA, seg_pos={0 0}, upload=473276]: Uploading piece... Dec 28 03:09:32.106435507 (539h40m31.977518542s) [noise=true , address=145.79.227.37:4566 , piece_id=56OEILXQJTPRZMGD6J5C3FJAJHHATMPBGSXZSXEO5GHRXOSDHMLA, seg_pos={0 0}, upload=473276]: Uploading piece... Dec 28 03:09:33.820927223 (539h40m33.692010258s) [upload=473276]: Next returned nil segment Dec 28 03:09:33.820942673 (539h40m33.692025708s) [upload=473276]: Waiting for error group managing segments... Dec 28 03:09:34.116226980 (539h40m33.987310025s) [noise=true , address=145.79.227.45:4231 , piece_id=3VCTP37YAVDZTCLW7R62D6JHDOS62YJOW4FNL2RS36LERGCMXOJA, seg_pos={0 0}, upload=473276]: Done uploading piece. err: <nil> Dec 28 03:09:34.118383466 (539h40m33.989466511s) [noise=true , address=145.79.227.36:4517 , piece_id=W2DV45TQYAZL3V4ZKGFXUXYVMG7DHHU4QTZ4WWAMMDYQ5OLEFKDQ, seg_pos={0 0}, upload=473276]: Done uploading piece. err: <nil> Dec 28 03:09:34.120008690 (539h40m33.991091725s) [noise=true , address=145.79.227.15:4874 , piece_id=M5G7FSNIBFQ7HMPFUEQ5JD6FL53G6LQADAWGK5Q2CWPZISZUT3TA, seg_pos={0 0}, upload=473276]: Done uploading piece. err: <nil> Dec 28 03:09:34.149025624 (539h40m34.020108659s) [noise=true , address=145.79.227.13:4802 , piece_id=IDDNJMKDGWBL53RTXHZ27W7M6WANVSC76T46MDMNOSVBKYLER34Q, seg_pos={0 0}, upload=473276]: Done uploading piece. err: <nil> Dec 28 03:09:34.301762367 (539h40m34.172845402s) [noise=true , address=77.90.31.13:4629 , piece_id=ZAQCKXGM45ODLTTLH4GARCW3RRMY3PN3DYUKIPYKGLFY2AWJLYVA, seg_pos={0 0}, upload=473276]: Done uploading piece. err: <nil> Dec 28 03:09:34.342051454 (539h40m34.213134489s) [noise=true , address=45.91.116.9:4261 , piece_id=QE4BN77KQCTWG6O4VIEN4JMWBRJ27DLOWX4BCJDJMKRLJITIPK5Q, seg_pos={0 0}, upload=473276]: Done uploading piece. err: <nil> Dec 28 03:09:34.377776209 (539h40m34.248859244s) [noise=true , address=45.91.116.5:4298 , piece_id=4T7UVHYLPRJLUUHLUPSQUXAEUXH2Y2ZWENWV7VFEWZDM76HOHK3Q, seg_pos={0 0}, upload=473276]: Done uploading piece. err: <nil> Dec 28 03:09:34.408185212 (539h40m34.279268247s) [noise=true , address=77.90.31.21:4880 , piece_id=QFJVHJIC3UTM67CEQHYMHKEG6AOGBAM5P6T475S4BXIZ2O752SLQ, seg_pos={0 0}, upload=473276]: Done uploading piece. err: <nil> Dec 28 03:09:34.636060612 (539h40m34.507143667s) [noise=true , address=45.91.116.11:4117 , piece_id=J3A5ANOJLZBBBX5EOFPWRTVLJ4MOPJWO2DAUHSQRYJSJNKOKUBPQ, seg_pos={0 0}, upload=473276]: Done uploading piece. err: <nil> Dec 28 03:09:34.751811039 (539h40m34.622894074s) [noise=true , address=145.79.227.38:4978 , piece_id=VN6FCKACQRZ7MWPUIAUURPHCNML6DBRHUQ4LXNCTBFSVSNXMKLWA, seg_pos={0 0}, upload=473276]: Done uploading piece. err: <nil> Dec 28 03:09:34.756647709 (539h40m34.627730744s) [noise=true , address=145.79.227.37:4566 , piece_id=56OEILXQJTPRZMGD6J5C3FJAJHHATMPBGSXZSXEO5GHRXOSDHMLA, seg_pos={0 0}, upload=473276]: Done uploading piece. err: <nil> Dec 28 03:09:34.759230737 (539h40m34.630313772s) [noise=true , address=145.79.227.42:4121 , piece_id=7QF7YSXAF6F4AGXTTXYBFZ4UXXBAQZCEKI2TAUMQJDXOQ3MDEILQ, seg_pos={0 0}, upload=473276]: Done uploading piece. err: <nil> Dec 28 03:09:34.967953718 (539h40m34.839036753s) [noise=true , address=145.79.227.41:4085 , piece_id=X72JKDXT6S7CZOE3MDNRVDHF5TIO2V2MGS6GPQ2EEW3DSX7EFZQQ, seg_pos={0 0}, upload=473276]: Done uploading piece. err: <nil> Dec 28 03:09:34.967973807 (539h40m34.839056842s) [seg_pos={0 0}, upload=473276]: Segment reached optimal threshold of 13 pieces. Dec 28 03:09:34.968177494 (539h40m34.839260539s) [noise=true , address=45.91.116.14:4189 , piece_id=E6GZY3XPIL4ZPOBA273UAM2O45OB5A2ALGK2ODWNCE7WDJU3HYCA, seg_pos={0 0}, upload=473276]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 28 03:09:34.968188125 (539h40m34.83927117s) [noise=true , address=45.91.116.14:4189 , piece_id=E6GZY3XPIL4ZPOBA273UAM2O45OB5A2ALGK2ODWNCE7WDJU3HYCA, seg_pos={0 0}, upload=473276]: Long tail cancellation due to optimal threshold Dec 28 03:09:34.968193855 (539h40m34.8392769s) [noise=true , address=77.90.31.10:4519 , piece_id=BTQVCUIINZPJQGB7EM4ZTEYACDOFN7IWKEI3L6ONYGH2IEOTZSNA, seg_pos={0 0}, upload=473276]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Dec 28 03:09:34.968202542 (539h40m34.839285577s) [noise=true , address=77.90.31.10:4519 , piece_id=BTQVCUIINZPJQGB7EM4ZTEYACDOFN7IWKEI3L6ONYGH2IEOTZSNA, seg_pos={0 0}, upload=473276]: Long tail cancellation due to optimal threshold Dec 28 03:09:34.968349261 (539h40m34.839432306s) [noise=true , address=145.79.227.7:4295 , piece_id=ZS55RLW2XXOXUK64T4ABWCZJ5J6WX32Z3PYG7NZW7XDH2DQGQVEA, seg_pos={0 0}, upload=473276]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT): protocol: write tcp 188.166.225.170:38156->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:38156->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38156->145.79.227.7:4295: use of closed network connection Dec 28 03:09:34.968355733 (539h40m34.839438768s) [noise=true , address=145.79.227.7:4295 , piece_id=ZS55RLW2XXOXUK64T4ABWCZJ5J6WX32Z3PYG7NZW7XDH2DQGQVEA, seg_pos={0 0}, upload=473276]: Long tail cancellation due to optimal threshold Dec 28 03:09:34.968378056 (539h40m34.839461121s) [seg_pos={0 0}, upload=473276]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 03:09:34.968481032 (539h40m34.839564077s) [upload=473276]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 03:09:34.968487644 (539h40m34.839570679s) [upload=473276]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 03:09:35.262134002 (539h40m35.133217037s) [upload=473276]: Done uploading segments... Dec 28 03:09:35.262152909 (539h40m35.133235944s) [upload=473276]: Upload finished. err: <nil> Duration: 18.726370962s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 44.730203983s trace: https://jaeger.dev.storj.io/trace/1822be52d200e5b5 FAIL (2025-12-28 03:11:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1ZAfYZ4wgpZVvatj6wtHona3hTaPcLxAmXC1DT2uKQyo8nLQQ7, Piece ID: 5MMQ44MXRT4XEL4X6WO54WRG2XYMX3AN3FNQ2H2XIUTKBXI3EEPA) context deadline exceeded; piecestore: (Node ID: 124guAM82oibYZsMho67Em6Ke6yEZXwT6LPMLLSZZBLqwoFGUkn, Piece ID: 22WJIPX4SM5NTIAW5ZRM2UKX32CTDBFCXPCCHRRWRCUCOF5CRE5Q) context deadline exceeded; piecestore: (Node ID: 1e6p1TD5CqUyWdZjkrxh2kRqMuyDiMUr9eimfeayDAKwhtBrZ8, Piece ID: XRI5EJFN7YYUG4QKLDGEFT6YAUHCUTTSXF735UDR6LZZBGMAZDJQ) context deadline exceeded ======LOGS====== 5.234848204 (538h55m35.105931239s) [noise=true , address=77.90.31.11:4572 , piece_id=5MMQ44MXRT4XEL4X6WO54WRG2XYMX3AN3FNQ2H2XIUTKBXI3EEPA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.235412057 (538h55m35.106495102s) [noise=true , address=145.79.227.22:4512 , piece_id=D46YLTIKGHG36DQ4YBKBSUDV4GGB4WVDWNOJS27IYYLN6AYZ5QFQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.235820946 (538h55m35.106903981s) [noise=true , address=77.90.31.19:4874 , piece_id=RHZUXUYMJU6RKQGSDZM5WDWIN5SQ7YIRSRATNNKT5GEMU2QH444A, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.236192192 (538h55m35.107275237s) [noise=true , address=77.90.31.16:4760 , piece_id=XRI5EJFN7YYUG4QKLDGEFT6YAUHCUTTSXF735UDR6LZZBGMAZDJQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.236564772 (538h55m35.107647817s) [noise=true , address=145.79.227.40:4062 , piece_id=PW6QWIPQWVUETLC6DJ624G6IB4P6EM4SYB72NAMQ4V6NUKNLUYLQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.236929766 (538h55m35.108012812s) [noise=true , address=64.49.13.139:4480 , piece_id=JAJY3F7IDFFMFKCK2QJV6W74RIESLRKBLXMN27Q3C45JWEPLBZLQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.237916846 (538h55m35.108999881s) [noise=true , address=145.79.227.34:4932 , piece_id=UPXSCQ45QDGPKNJXRFJ33EAA3QENVHRVQARU5TFJTEWICZNR253Q, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.238852948 (538h55m35.109935983s) [noise=true , address=45.91.116.12:4257 , piece_id=ZI67LIAHZW4G357GWKDPTB5SMXYGNXZTTTDMNAETDS2B6SWVDAEQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239152910 (538h55m35.110235945s) [noise=true , address=145.79.227.44:4222 , piece_id=CZ2CS62CTHQFAIE3S3JO2N3VJ4XIR5YJ2LMRVLLX7I4VM2VZVQVA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239409969 (538h55m35.110493004s) [noise=true , address=145.79.227.30:4773 , piece_id=KUGJXZCNMUE7WSK3VLY4AUBSOT4H6IY4XB4NCBGAGCNUG7F7IODQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239681136 (538h55m35.110764171s) [noise=true , address=77.90.31.13:4635 , piece_id=IIOY5X2W3XW4C2OFEBRCABMYFBI24RRTYNUSKN6KJU7CTUW4KONQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239954676 (538h55m35.111037711s) [noise=true , address=45.91.116.9:4281 , piece_id=JZLDW777CBX2BP7MLOQV4NFP3H4XJVCNQXTHNSUWPVDOJLPZK36A, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.240189914 (538h55m35.111272949s) [noise=true , address=145.79.227.35:4512 , piece_id=3E3VCCC45XVT5FHFDKXSTGGAGYBEML3BKU6JJTT33LCD2YMOMBZQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.240449147 (538h55m35.111532182s) [noise=true , address=45.91.116.8:4064 , piece_id=22WJIPX4SM5NTIAW5ZRM2UKX32CTDBFCXPCCHRRWRCUCOF5CRE5Q, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.240639640 (538h55m35.111722675s) [noise=true , address=145.79.227.32:4853 , piece_id=MBNCFDBQ4SZZXV4F7J5TRMOUYUQ5IPKQCGYGGM5SOIFJ6ZYCAUYA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.241137798 (538h55m35.112220833s) [noise=true , address=77.90.31.16:4751 , piece_id=N3Z2LLMCIA2F42VXZA7IDRU4XEKGOICH6TRVERODU76MFFE4QSHA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.241383797 (538h55m35.112466842s) [noise=true , address=77.90.31.18:4836 , piece_id=BSVEFI52EKN44CMINFUSZBNOH53IW5TXLVTF5IH6ZUTNII6RRLVA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:36.592880816 (538h55m36.463963871s) [upload=472631]: Next returned nil segment Dec 28 02:24:36.592901625 (538h55m36.46398467s) [upload=472631]: Waiting for error group managing segments... Dec 28 02:24:36.946933741 (538h55m36.818016786s) [noise=true , address=64.49.13.139:4480 , piece_id=JAJY3F7IDFFMFKCK2QJV6W74RIESLRKBLXMN27Q3C45JWEPLBZLQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.128189692 (538h55m36.999272737s) [noise=true , address=145.79.227.35:4512 , piece_id=3E3VCCC45XVT5FHFDKXSTGGAGYBEML3BKU6JJTT33LCD2YMOMBZQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.160760084 (538h55m37.031843119s) [noise=true , address=145.79.227.34:4932 , piece_id=UPXSCQ45QDGPKNJXRFJ33EAA3QENVHRVQARU5TFJTEWICZNR253Q, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.288838207 (538h55m37.159921252s) [noise=true , address=77.90.31.11:4572 , piece_id=5MMQ44MXRT4XEL4X6WO54WRG2XYMX3AN3FNQ2H2XIUTKBXI3EEPA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.319996310 (538h55m37.191079345s) [noise=true , address=145.79.227.22:4512 , piece_id=D46YLTIKGHG36DQ4YBKBSUDV4GGB4WVDWNOJS27IYYLN6AYZ5QFQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.320513273 (538h55m37.191596319s) [noise=true , address=77.90.31.18:4836 , piece_id=BSVEFI52EKN44CMINFUSZBNOH53IW5TXLVTF5IH6ZUTNII6RRLVA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.336522926 (538h55m37.207605971s) [noise=true , address=145.79.227.32:4853 , piece_id=MBNCFDBQ4SZZXV4F7J5TRMOUYUQ5IPKQCGYGGM5SOIFJ6ZYCAUYA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.384868031 (538h55m37.255951066s) [noise=true , address=77.90.31.16:4751 , piece_id=N3Z2LLMCIA2F42VXZA7IDRU4XEKGOICH6TRVERODU76MFFE4QSHA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.453563554 (538h55m37.324646599s) [noise=true , address=45.91.116.8:4064 , piece_id=22WJIPX4SM5NTIAW5ZRM2UKX32CTDBFCXPCCHRRWRCUCOF5CRE5Q, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.479454839 (538h55m37.350537884s) [noise=true , address=45.91.116.12:4257 , piece_id=ZI67LIAHZW4G357GWKDPTB5SMXYGNXZTTTDMNAETDS2B6SWVDAEQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.659710174 (538h55m37.530793219s) [noise=true , address=77.90.31.13:4635 , piece_id=IIOY5X2W3XW4C2OFEBRCABMYFBI24RRTYNUSKN6KJU7CTUW4KONQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.715039128 (538h55m37.586122163s) [noise=true , address=77.90.31.16:4760 , piece_id=XRI5EJFN7YYUG4QKLDGEFT6YAUHCUTTSXF735UDR6LZZBGMAZDJQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.977473406 (538h55m37.848556441s) [noise=true , address=77.90.31.19:4874 , piece_id=RHZUXUYMJU6RKQGSDZM5WDWIN5SQ7YIRSRATNNKT5GEMU2QH444A, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:38.016302745 (538h55m37.88738578s) [noise=true , address=145.79.227.30:4773 , piece_id=KUGJXZCNMUE7WSK3VLY4AUBSOT4H6IY4XB4NCBGAGCNUG7F7IODQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:38.016326982 (538h55m37.887410027s) [seg_pos={0 0}, upload=472631]: Segment reached optimal threshold of 14 pieces. Dec 28 02:24:38.016585353 (538h55m37.887668398s) [noise=true , address=145.79.227.30:4766 , piece_id=67M2DMPAVWIQKQSWUAG73D4E4NRRSG4XEVPOUVIJPEQOUV7A6VVA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1k6zdamuKPkBdzyNZNWx4FCmaTkv33PchcwvTRR8bgcETBs4K7): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016594450 (538h55m37.887677495s) [noise=true , address=145.79.227.30:4766 , piece_id=67M2DMPAVWIQKQSWUAG73D4E4NRRSG4XEVPOUVIJPEQOUV7A6VVA, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016627093 (538h55m37.887710148s) [noise=true , address=45.91.116.9:4281 , piece_id=JZLDW777CBX2BP7MLOQV4NFP3H4XJVCNQXTHNSUWPVDOJLPZK36A, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gxf7DpJZthyR6f1PExX56tJavjhPw9Gxb4h8mtmeYKW3t3tS1): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016635258 (538h55m37.887718293s) [noise=true , address=45.91.116.9:4281 , piece_id=JZLDW777CBX2BP7MLOQV4NFP3H4XJVCNQXTHNSUWPVDOJLPZK36A, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016871298 (538h55m37.887954363s) [noise=true , address=145.79.227.44:4222 , piece_id=CZ2CS62CTHQFAIE3S3JO2N3VJ4XIR5YJ2LMRVLLX7I4VM2VZVQVA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZKW35KvrsoYRefbYeUFZo8bh4XZYycNKGEvTFjzibezjXgj3e): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016880124 (538h55m37.887963169s) [noise=true , address=145.79.227.44:4222 , piece_id=CZ2CS62CTHQFAIE3S3JO2N3VJ4XIR5YJ2LMRVLLX7I4VM2VZVQVA, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016934839 (538h55m37.888017884s) [noise=true , address=145.79.227.40:4062 , piece_id=PW6QWIPQWVUETLC6DJ624G6IB4P6EM4SYB72NAMQ4V6NUKNLUYLQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016943425 (538h55m37.88802649s) [noise=true , address=145.79.227.40:4062 , piece_id=PW6QWIPQWVUETLC6DJ624G6IB4P6EM4SYB72NAMQ4V6NUKNLUYLQ, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016968252 (538h55m37.888051297s) [seg_pos={0 0}, upload=472631]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 02:24:38.017067822 (538h55m37.888150867s) [upload=472631]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 02:24:38.017074244 (538h55m37.888157289s) [upload=472631]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 02:24:38.304306667 (538h55m38.175389712s) [upload=472631]: Done uploading segments... Dec 28 02:24:38.304329641 (538h55m38.175412686s) [upload=472631]: Upload finished. err: <nil> Duration: 44.730223221s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [............X....X............X........X....X...........X...] duration: 5.001512026s trace: https://jaeger.dev.storj.io/trace/235003f053bdab FAIL (2025-12-28 03:11:11 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00156156s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [.............X....X............X........X....X...........X..] duration: 3.452047068s trace: https://jaeger.dev.storj.io/trace/235003f053bdab FAIL (2025-12-28 03:10:46 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.452063228s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 36.140282592s trace: https://jaeger.dev.storj.io/trace/1822be52d200e5b5 FAIL (2025-12-28 03:10:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1ZAfYZ4wgpZVvatj6wtHona3hTaPcLxAmXC1DT2uKQyo8nLQQ7, Piece ID: 5MMQ44MXRT4XEL4X6WO54WRG2XYMX3AN3FNQ2H2XIUTKBXI3EEPA) context deadline exceeded; piecestore: (Node ID: 124guAM82oibYZsMho67Em6Ke6yEZXwT6LPMLLSZZBLqwoFGUkn, Piece ID: 22WJIPX4SM5NTIAW5ZRM2UKX32CTDBFCXPCCHRRWRCUCOF5CRE5Q) context deadline exceeded; piecestore: (Node ID: 1e6p1TD5CqUyWdZjkrxh2kRqMuyDiMUr9eimfeayDAKwhtBrZ8, Piece ID: XRI5EJFN7YYUG4QKLDGEFT6YAUHCUTTSXF735UDR6LZZBGMAZDJQ) context deadline exceeded ======LOGS====== 5.234848204 (538h55m35.105931239s) [noise=true , address=77.90.31.11:4572 , piece_id=5MMQ44MXRT4XEL4X6WO54WRG2XYMX3AN3FNQ2H2XIUTKBXI3EEPA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.235412057 (538h55m35.106495102s) [noise=true , address=145.79.227.22:4512 , piece_id=D46YLTIKGHG36DQ4YBKBSUDV4GGB4WVDWNOJS27IYYLN6AYZ5QFQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.235820946 (538h55m35.106903981s) [noise=true , address=77.90.31.19:4874 , piece_id=RHZUXUYMJU6RKQGSDZM5WDWIN5SQ7YIRSRATNNKT5GEMU2QH444A, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.236192192 (538h55m35.107275237s) [noise=true , address=77.90.31.16:4760 , piece_id=XRI5EJFN7YYUG4QKLDGEFT6YAUHCUTTSXF735UDR6LZZBGMAZDJQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.236564772 (538h55m35.107647817s) [noise=true , address=145.79.227.40:4062 , piece_id=PW6QWIPQWVUETLC6DJ624G6IB4P6EM4SYB72NAMQ4V6NUKNLUYLQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.236929766 (538h55m35.108012812s) [noise=true , address=64.49.13.139:4480 , piece_id=JAJY3F7IDFFMFKCK2QJV6W74RIESLRKBLXMN27Q3C45JWEPLBZLQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.237916846 (538h55m35.108999881s) [noise=true , address=145.79.227.34:4932 , piece_id=UPXSCQ45QDGPKNJXRFJ33EAA3QENVHRVQARU5TFJTEWICZNR253Q, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.238852948 (538h55m35.109935983s) [noise=true , address=45.91.116.12:4257 , piece_id=ZI67LIAHZW4G357GWKDPTB5SMXYGNXZTTTDMNAETDS2B6SWVDAEQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239152910 (538h55m35.110235945s) [noise=true , address=145.79.227.44:4222 , piece_id=CZ2CS62CTHQFAIE3S3JO2N3VJ4XIR5YJ2LMRVLLX7I4VM2VZVQVA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239409969 (538h55m35.110493004s) [noise=true , address=145.79.227.30:4773 , piece_id=KUGJXZCNMUE7WSK3VLY4AUBSOT4H6IY4XB4NCBGAGCNUG7F7IODQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239681136 (538h55m35.110764171s) [noise=true , address=77.90.31.13:4635 , piece_id=IIOY5X2W3XW4C2OFEBRCABMYFBI24RRTYNUSKN6KJU7CTUW4KONQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239954676 (538h55m35.111037711s) [noise=true , address=45.91.116.9:4281 , piece_id=JZLDW777CBX2BP7MLOQV4NFP3H4XJVCNQXTHNSUWPVDOJLPZK36A, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.240189914 (538h55m35.111272949s) [noise=true , address=145.79.227.35:4512 , piece_id=3E3VCCC45XVT5FHFDKXSTGGAGYBEML3BKU6JJTT33LCD2YMOMBZQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.240449147 (538h55m35.111532182s) [noise=true , address=45.91.116.8:4064 , piece_id=22WJIPX4SM5NTIAW5ZRM2UKX32CTDBFCXPCCHRRWRCUCOF5CRE5Q, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.240639640 (538h55m35.111722675s) [noise=true , address=145.79.227.32:4853 , piece_id=MBNCFDBQ4SZZXV4F7J5TRMOUYUQ5IPKQCGYGGM5SOIFJ6ZYCAUYA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.241137798 (538h55m35.112220833s) [noise=true , address=77.90.31.16:4751 , piece_id=N3Z2LLMCIA2F42VXZA7IDRU4XEKGOICH6TRVERODU76MFFE4QSHA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.241383797 (538h55m35.112466842s) [noise=true , address=77.90.31.18:4836 , piece_id=BSVEFI52EKN44CMINFUSZBNOH53IW5TXLVTF5IH6ZUTNII6RRLVA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:36.592880816 (538h55m36.463963871s) [upload=472631]: Next returned nil segment Dec 28 02:24:36.592901625 (538h55m36.46398467s) [upload=472631]: Waiting for error group managing segments... Dec 28 02:24:36.946933741 (538h55m36.818016786s) [noise=true , address=64.49.13.139:4480 , piece_id=JAJY3F7IDFFMFKCK2QJV6W74RIESLRKBLXMN27Q3C45JWEPLBZLQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.128189692 (538h55m36.999272737s) [noise=true , address=145.79.227.35:4512 , piece_id=3E3VCCC45XVT5FHFDKXSTGGAGYBEML3BKU6JJTT33LCD2YMOMBZQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.160760084 (538h55m37.031843119s) [noise=true , address=145.79.227.34:4932 , piece_id=UPXSCQ45QDGPKNJXRFJ33EAA3QENVHRVQARU5TFJTEWICZNR253Q, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.288838207 (538h55m37.159921252s) [noise=true , address=77.90.31.11:4572 , piece_id=5MMQ44MXRT4XEL4X6WO54WRG2XYMX3AN3FNQ2H2XIUTKBXI3EEPA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.319996310 (538h55m37.191079345s) [noise=true , address=145.79.227.22:4512 , piece_id=D46YLTIKGHG36DQ4YBKBSUDV4GGB4WVDWNOJS27IYYLN6AYZ5QFQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.320513273 (538h55m37.191596319s) [noise=true , address=77.90.31.18:4836 , piece_id=BSVEFI52EKN44CMINFUSZBNOH53IW5TXLVTF5IH6ZUTNII6RRLVA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.336522926 (538h55m37.207605971s) [noise=true , address=145.79.227.32:4853 , piece_id=MBNCFDBQ4SZZXV4F7J5TRMOUYUQ5IPKQCGYGGM5SOIFJ6ZYCAUYA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.384868031 (538h55m37.255951066s) [noise=true , address=77.90.31.16:4751 , piece_id=N3Z2LLMCIA2F42VXZA7IDRU4XEKGOICH6TRVERODU76MFFE4QSHA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.453563554 (538h55m37.324646599s) [noise=true , address=45.91.116.8:4064 , piece_id=22WJIPX4SM5NTIAW5ZRM2UKX32CTDBFCXPCCHRRWRCUCOF5CRE5Q, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.479454839 (538h55m37.350537884s) [noise=true , address=45.91.116.12:4257 , piece_id=ZI67LIAHZW4G357GWKDPTB5SMXYGNXZTTTDMNAETDS2B6SWVDAEQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.659710174 (538h55m37.530793219s) [noise=true , address=77.90.31.13:4635 , piece_id=IIOY5X2W3XW4C2OFEBRCABMYFBI24RRTYNUSKN6KJU7CTUW4KONQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.715039128 (538h55m37.586122163s) [noise=true , address=77.90.31.16:4760 , piece_id=XRI5EJFN7YYUG4QKLDGEFT6YAUHCUTTSXF735UDR6LZZBGMAZDJQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.977473406 (538h55m37.848556441s) [noise=true , address=77.90.31.19:4874 , piece_id=RHZUXUYMJU6RKQGSDZM5WDWIN5SQ7YIRSRATNNKT5GEMU2QH444A, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:38.016302745 (538h55m37.88738578s) [noise=true , address=145.79.227.30:4773 , piece_id=KUGJXZCNMUE7WSK3VLY4AUBSOT4H6IY4XB4NCBGAGCNUG7F7IODQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:38.016326982 (538h55m37.887410027s) [seg_pos={0 0}, upload=472631]: Segment reached optimal threshold of 14 pieces. Dec 28 02:24:38.016585353 (538h55m37.887668398s) [noise=true , address=145.79.227.30:4766 , piece_id=67M2DMPAVWIQKQSWUAG73D4E4NRRSG4XEVPOUVIJPEQOUV7A6VVA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1k6zdamuKPkBdzyNZNWx4FCmaTkv33PchcwvTRR8bgcETBs4K7): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016594450 (538h55m37.887677495s) [noise=true , address=145.79.227.30:4766 , piece_id=67M2DMPAVWIQKQSWUAG73D4E4NRRSG4XEVPOUVIJPEQOUV7A6VVA, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016627093 (538h55m37.887710148s) [noise=true , address=45.91.116.9:4281 , piece_id=JZLDW777CBX2BP7MLOQV4NFP3H4XJVCNQXTHNSUWPVDOJLPZK36A, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gxf7DpJZthyR6f1PExX56tJavjhPw9Gxb4h8mtmeYKW3t3tS1): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016635258 (538h55m37.887718293s) [noise=true , address=45.91.116.9:4281 , piece_id=JZLDW777CBX2BP7MLOQV4NFP3H4XJVCNQXTHNSUWPVDOJLPZK36A, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016871298 (538h55m37.887954363s) [noise=true , address=145.79.227.44:4222 , piece_id=CZ2CS62CTHQFAIE3S3JO2N3VJ4XIR5YJ2LMRVLLX7I4VM2VZVQVA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZKW35KvrsoYRefbYeUFZo8bh4XZYycNKGEvTFjzibezjXgj3e): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016880124 (538h55m37.887963169s) [noise=true , address=145.79.227.44:4222 , piece_id=CZ2CS62CTHQFAIE3S3JO2N3VJ4XIR5YJ2LMRVLLX7I4VM2VZVQVA, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016934839 (538h55m37.888017884s) [noise=true , address=145.79.227.40:4062 , piece_id=PW6QWIPQWVUETLC6DJ624G6IB4P6EM4SYB72NAMQ4V6NUKNLUYLQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016943425 (538h55m37.88802649s) [noise=true , address=145.79.227.40:4062 , piece_id=PW6QWIPQWVUETLC6DJ624G6IB4P6EM4SYB72NAMQ4V6NUKNLUYLQ, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016968252 (538h55m37.888051297s) [seg_pos={0 0}, upload=472631]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 02:24:38.017067822 (538h55m37.888150867s) [upload=472631]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 02:24:38.017074244 (538h55m37.888157289s) [upload=472631]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 02:24:38.304306667 (538h55m38.175389712s) [upload=472631]: Done uploading segments... Dec 28 02:24:38.304329641 (538h55m38.175412686s) [upload=472631]: Upload finished. err: <nil> Duration: 36.140305065s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 47 out of 60 is failed (threshold=3) [XX.XXXXXX.XXXX.XXXX..X..XXXXX.XXXXXXXXXXXXX.XXXXXXXXX.X..XX.] duration: 14.071858725s trace: https://jaeger.dev.storj.io/trace/30fe8cdffa81d60d FAIL (2025-12-28 03:10:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: QE4BN77KQCTWG6O4VIEN4JMWBRJ27DLOWX4BCJDJMKRLJITIPK5Q) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: J3A5ANOJLZBBBX5EOFPWRTVLJ4MOPJWO2DAUHSQRYJSJNKOKUBPQ) context deadline exceeded ======LOGS====== 28 03:09:31.842900787 (539h40m31.713983822s) [upload=473276]: Starting upload Dec 28 03:09:31.842985860 (539h40m31.714068915s) [upload=473276]: Done starting upload Dec 28 03:09:31.843136727 (539h40m31.714219782s) [upload=473276]: Uploading segments... Dec 28 03:09:31.843263529 (539h40m31.714346574s) [upload=473276]: Got next segment. Inline: false Dec 28 03:09:31.843309576 (539h40m31.714392641s) [upload=473276]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 03:09:31.843316148 (539h40m31.714399193s) [upload=473276]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 03:09:32.091638863 (539h40m31.962721908s) [seg_pos={0 0}, upload=473276]: Begin upload segment... Dec 28 03:09:32.091684179 (539h40m31.962767214s) [seg_pos={0 0}, upload=473276]: Done begin upload segment. Dec 28 03:09:32.091709558 (539h40m31.962792593s) [noise=true , address=45.91.116.9:4261 , piece_id=QE4BN77KQCTWG6O4VIEN4JMWBRJ27DLOWX4BCJDJMKRLJITIPK5Q, seg_pos={0 0}, upload=473276]: Uploading piece... Dec 28 03:09:32.091732532 (539h40m31.962815577s) [noise=true , address=145.79.227.13:4802 , piece_id=IDDNJMKDGWBL53RTXHZ27W7M6WANVSC76T46MDMNOSVBKYLER34Q, seg_pos={0 0}, upload=473276]: Uploading piece... Dec 28 03:09:32.092360778 (539h40m31.963443813s) [noise=true , address=45.91.116.11:4117 , piece_id=J3A5ANOJLZBBBX5EOFPWRTVLJ4MOPJWO2DAUHSQRYJSJNKOKUBPQ, seg_pos={0 0}, upload=473276]: Uploading piece... Dec 28 03:09:32.092821214 (539h40m31.96390426s) [noise=true , address=145.79.227.7:4295 , piece_id=ZS55RLW2XXOXUK64T4ABWCZJ5J6WX32Z3PYG7NZW7XDH2DQGQVEA, seg_pos={0 0}, upload=473276]: Uploading piece... Dec 28 03:09:32.093210165 (539h40m31.9642932s) [noise=true , address=145.79.227.36:4517 , piece_id=W2DV45TQYAZL3V4ZKGFXUXYVMG7DHHU4QTZ4WWAMMDYQ5OLEFKDQ, seg_pos={0 0}, upload=473276]: Uploading piece... Dec 28 03:09:32.093484267 (539h40m31.964567302s) [noise=true , address=77.90.31.21:4880 , piece_id=QFJVHJIC3UTM67CEQHYMHKEG6AOGBAM5P6T475S4BXIZ2O752SLQ, seg_pos={0 0}, upload=473276]: Uploading piece... Dec 28 03:09:32.093886552 (539h40m31.964969598s) [noise=true , address=145.79.227.41:4085 , piece_id=X72JKDXT6S7CZOE3MDNRVDHF5TIO2V2MGS6GPQ2EEW3DSX7EFZQQ, seg_pos={0 0}, upload=473276]: Uploading piece... Dec 28 03:09:32.095140079 (539h40m31.966223114s) [noise=true , address=77.90.31.10:4519 , piece_id=BTQVCUIINZPJQGB7EM4ZTEYACDOFN7IWKEI3L6ONYGH2IEOTZSNA, seg_pos={0 0}, upload=473276]: Uploading piece... Dec 28 03:09:32.095358536 (539h40m31.966441571s) [noise=true , address=145.79.227.45:4231 , piece_id=3VCTP37YAVDZTCLW7R62D6JHDOS62YJOW4FNL2RS36LERGCMXOJA, seg_pos={0 0}, upload=473276]: Uploading piece... Dec 28 03:09:32.096249953 (539h40m31.967332988s) [noise=true , address=45.91.116.5:4298 , piece_id=4T7UVHYLPRJLUUHLUPSQUXAEUXH2Y2ZWENWV7VFEWZDM76HOHK3Q, seg_pos={0 0}, upload=473276]: Uploading piece... Dec 28 03:09:32.097530952 (539h40m31.968613987s) [noise=true , address=145.79.227.42:4121 , piece_id=7QF7YSXAF6F4AGXTTXYBFZ4UXXBAQZCEKI2TAUMQJDXOQ3MDEILQ, seg_pos={0 0}, upload=473276]: Uploading piece... Dec 28 03:09:32.101540176 (539h40m31.972623222s) [noise=true , address=145.79.227.38:4978 , piece_id=VN6FCKACQRZ7MWPUIAUURPHCNML6DBRHUQ4LXNCTBFSVSNXMKLWA, seg_pos={0 0}, upload=473276]: Uploading piece... Dec 28 03:09:32.103185900 (539h40m31.974268935s) [noise=true , address=77.90.31.13:4629 , piece_id=ZAQCKXGM45ODLTTLH4GARCW3RRMY3PN3DYUKIPYKGLFY2AWJLYVA, seg_pos={0 0}, upload=473276]: Uploading piece... Dec 28 03:09:32.104285855 (539h40m31.97536889s) [noise=true , address=45.91.116.14:4189 , piece_id=E6GZY3XPIL4ZPOBA273UAM2O45OB5A2ALGK2ODWNCE7WDJU3HYCA, seg_pos={0 0}, upload=473276]: Uploading piece... Dec 28 03:09:32.105979329 (539h40m31.977062364s) [noise=true , address=145.79.227.15:4874 , piece_id=M5G7FSNIBFQ7HMPFUEQ5JD6FL53G6LQADAWGK5Q2CWPZISZUT3TA, seg_pos={0 0}, upload=473276]: Uploading piece... Dec 28 03:09:32.106435507 (539h40m31.977518542s) [noise=true , address=145.79.227.37:4566 , piece_id=56OEILXQJTPRZMGD6J5C3FJAJHHATMPBGSXZSXEO5GHRXOSDHMLA, seg_pos={0 0}, upload=473276]: Uploading piece... Dec 28 03:09:33.820927223 (539h40m33.692010258s) [upload=473276]: Next returned nil segment Dec 28 03:09:33.820942673 (539h40m33.692025708s) [upload=473276]: Waiting for error group managing segments... Dec 28 03:09:34.116226980 (539h40m33.987310025s) [noise=true , address=145.79.227.45:4231 , piece_id=3VCTP37YAVDZTCLW7R62D6JHDOS62YJOW4FNL2RS36LERGCMXOJA, seg_pos={0 0}, upload=473276]: Done uploading piece. err: <nil> Dec 28 03:09:34.118383466 (539h40m33.989466511s) [noise=true , address=145.79.227.36:4517 , piece_id=W2DV45TQYAZL3V4ZKGFXUXYVMG7DHHU4QTZ4WWAMMDYQ5OLEFKDQ, seg_pos={0 0}, upload=473276]: Done uploading piece. err: <nil> Dec 28 03:09:34.120008690 (539h40m33.991091725s) [noise=true , address=145.79.227.15:4874 , piece_id=M5G7FSNIBFQ7HMPFUEQ5JD6FL53G6LQADAWGK5Q2CWPZISZUT3TA, seg_pos={0 0}, upload=473276]: Done uploading piece. err: <nil> Dec 28 03:09:34.149025624 (539h40m34.020108659s) [noise=true , address=145.79.227.13:4802 , piece_id=IDDNJMKDGWBL53RTXHZ27W7M6WANVSC76T46MDMNOSVBKYLER34Q, seg_pos={0 0}, upload=473276]: Done uploading piece. err: <nil> Dec 28 03:09:34.301762367 (539h40m34.172845402s) [noise=true , address=77.90.31.13:4629 , piece_id=ZAQCKXGM45ODLTTLH4GARCW3RRMY3PN3DYUKIPYKGLFY2AWJLYVA, seg_pos={0 0}, upload=473276]: Done uploading piece. err: <nil> Dec 28 03:09:34.342051454 (539h40m34.213134489s) [noise=true , address=45.91.116.9:4261 , piece_id=QE4BN77KQCTWG6O4VIEN4JMWBRJ27DLOWX4BCJDJMKRLJITIPK5Q, seg_pos={0 0}, upload=473276]: Done uploading piece. err: <nil> Dec 28 03:09:34.377776209 (539h40m34.248859244s) [noise=true , address=45.91.116.5:4298 , piece_id=4T7UVHYLPRJLUUHLUPSQUXAEUXH2Y2ZWENWV7VFEWZDM76HOHK3Q, seg_pos={0 0}, upload=473276]: Done uploading piece. err: <nil> Dec 28 03:09:34.408185212 (539h40m34.279268247s) [noise=true , address=77.90.31.21:4880 , piece_id=QFJVHJIC3UTM67CEQHYMHKEG6AOGBAM5P6T475S4BXIZ2O752SLQ, seg_pos={0 0}, upload=473276]: Done uploading piece. err: <nil> Dec 28 03:09:34.636060612 (539h40m34.507143667s) [noise=true , address=45.91.116.11:4117 , piece_id=J3A5ANOJLZBBBX5EOFPWRTVLJ4MOPJWO2DAUHSQRYJSJNKOKUBPQ, seg_pos={0 0}, upload=473276]: Done uploading piece. err: <nil> Dec 28 03:09:34.751811039 (539h40m34.622894074s) [noise=true , address=145.79.227.38:4978 , piece_id=VN6FCKACQRZ7MWPUIAUURPHCNML6DBRHUQ4LXNCTBFSVSNXMKLWA, seg_pos={0 0}, upload=473276]: Done uploading piece. err: <nil> Dec 28 03:09:34.756647709 (539h40m34.627730744s) [noise=true , address=145.79.227.37:4566 , piece_id=56OEILXQJTPRZMGD6J5C3FJAJHHATMPBGSXZSXEO5GHRXOSDHMLA, seg_pos={0 0}, upload=473276]: Done uploading piece. err: <nil> Dec 28 03:09:34.759230737 (539h40m34.630313772s) [noise=true , address=145.79.227.42:4121 , piece_id=7QF7YSXAF6F4AGXTTXYBFZ4UXXBAQZCEKI2TAUMQJDXOQ3MDEILQ, seg_pos={0 0}, upload=473276]: Done uploading piece. err: <nil> Dec 28 03:09:34.967953718 (539h40m34.839036753s) [noise=true , address=145.79.227.41:4085 , piece_id=X72JKDXT6S7CZOE3MDNRVDHF5TIO2V2MGS6GPQ2EEW3DSX7EFZQQ, seg_pos={0 0}, upload=473276]: Done uploading piece. err: <nil> Dec 28 03:09:34.967973807 (539h40m34.839056842s) [seg_pos={0 0}, upload=473276]: Segment reached optimal threshold of 13 pieces. Dec 28 03:09:34.968177494 (539h40m34.839260539s) [noise=true , address=45.91.116.14:4189 , piece_id=E6GZY3XPIL4ZPOBA273UAM2O45OB5A2ALGK2ODWNCE7WDJU3HYCA, seg_pos={0 0}, upload=473276]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 28 03:09:34.968188125 (539h40m34.83927117s) [noise=true , address=45.91.116.14:4189 , piece_id=E6GZY3XPIL4ZPOBA273UAM2O45OB5A2ALGK2ODWNCE7WDJU3HYCA, seg_pos={0 0}, upload=473276]: Long tail cancellation due to optimal threshold Dec 28 03:09:34.968193855 (539h40m34.8392769s) [noise=true , address=77.90.31.10:4519 , piece_id=BTQVCUIINZPJQGB7EM4ZTEYACDOFN7IWKEI3L6ONYGH2IEOTZSNA, seg_pos={0 0}, upload=473276]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Dec 28 03:09:34.968202542 (539h40m34.839285577s) [noise=true , address=77.90.31.10:4519 , piece_id=BTQVCUIINZPJQGB7EM4ZTEYACDOFN7IWKEI3L6ONYGH2IEOTZSNA, seg_pos={0 0}, upload=473276]: Long tail cancellation due to optimal threshold Dec 28 03:09:34.968349261 (539h40m34.839432306s) [noise=true , address=145.79.227.7:4295 , piece_id=ZS55RLW2XXOXUK64T4ABWCZJ5J6WX32Z3PYG7NZW7XDH2DQGQVEA, seg_pos={0 0}, upload=473276]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT): protocol: write tcp 188.166.225.170:38156->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:38156->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38156->145.79.227.7:4295: use of closed network connection Dec 28 03:09:34.968355733 (539h40m34.839438768s) [noise=true , address=145.79.227.7:4295 , piece_id=ZS55RLW2XXOXUK64T4ABWCZJ5J6WX32Z3PYG7NZW7XDH2DQGQVEA, seg_pos={0 0}, upload=473276]: Long tail cancellation due to optimal threshold Dec 28 03:09:34.968378056 (539h40m34.839461121s) [seg_pos={0 0}, upload=473276]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 03:09:34.968481032 (539h40m34.839564077s) [upload=473276]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 03:09:34.968487644 (539h40m34.839570679s) [upload=473276]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 03:09:35.262134002 (539h40m35.133217037s) [upload=473276]: Done uploading segments... Dec 28 03:09:35.262152909 (539h40m35.133235944s) [upload=473276]: Upload finished. err: <nil> Duration: 14.07191832s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 48 out of 60 is failed (threshold=3) [XXX.XXXXXX.XXXX.XXXX..X..XXXXX.XXXXXXXXXXXXX.XXXXXXXXX.X..XX] duration: 30.004704583s trace: https://jaeger.dev.storj.io/trace/30fe8cdffa81d60d FAIL (2025-12-28 03:10:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: QE4BN77KQCTWG6O4VIEN4JMWBRJ27DLOWX4BCJDJMKRLJITIPK5Q) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: J3A5ANOJLZBBBX5EOFPWRTVLJ4MOPJWO2DAUHSQRYJSJNKOKUBPQ) context deadline exceeded ======LOGS====== 28 03:09:31.842900787 (539h40m31.713983822s) [upload=473276]: Starting upload Dec 28 03:09:31.842985860 (539h40m31.714068915s) [upload=473276]: Done starting upload Dec 28 03:09:31.843136727 (539h40m31.714219782s) [upload=473276]: Uploading segments... Dec 28 03:09:31.843263529 (539h40m31.714346574s) [upload=473276]: Got next segment. Inline: false Dec 28 03:09:31.843309576 (539h40m31.714392641s) [upload=473276]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 03:09:31.843316148 (539h40m31.714399193s) [upload=473276]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 03:09:32.091638863 (539h40m31.962721908s) [seg_pos={0 0}, upload=473276]: Begin upload segment... Dec 28 03:09:32.091684179 (539h40m31.962767214s) [seg_pos={0 0}, upload=473276]: Done begin upload segment. Dec 28 03:09:32.091709558 (539h40m31.962792593s) [noise=true , address=45.91.116.9:4261 , piece_id=QE4BN77KQCTWG6O4VIEN4JMWBRJ27DLOWX4BCJDJMKRLJITIPK5Q, seg_pos={0 0}, upload=473276]: Uploading piece... Dec 28 03:09:32.091732532 (539h40m31.962815577s) [noise=true , address=145.79.227.13:4802 , piece_id=IDDNJMKDGWBL53RTXHZ27W7M6WANVSC76T46MDMNOSVBKYLER34Q, seg_pos={0 0}, upload=473276]: Uploading piece... Dec 28 03:09:32.092360778 (539h40m31.963443813s) [noise=true , address=45.91.116.11:4117 , piece_id=J3A5ANOJLZBBBX5EOFPWRTVLJ4MOPJWO2DAUHSQRYJSJNKOKUBPQ, seg_pos={0 0}, upload=473276]: Uploading piece... Dec 28 03:09:32.092821214 (539h40m31.96390426s) [noise=true , address=145.79.227.7:4295 , piece_id=ZS55RLW2XXOXUK64T4ABWCZJ5J6WX32Z3PYG7NZW7XDH2DQGQVEA, seg_pos={0 0}, upload=473276]: Uploading piece... Dec 28 03:09:32.093210165 (539h40m31.9642932s) [noise=true , address=145.79.227.36:4517 , piece_id=W2DV45TQYAZL3V4ZKGFXUXYVMG7DHHU4QTZ4WWAMMDYQ5OLEFKDQ, seg_pos={0 0}, upload=473276]: Uploading piece... Dec 28 03:09:32.093484267 (539h40m31.964567302s) [noise=true , address=77.90.31.21:4880 , piece_id=QFJVHJIC3UTM67CEQHYMHKEG6AOGBAM5P6T475S4BXIZ2O752SLQ, seg_pos={0 0}, upload=473276]: Uploading piece... Dec 28 03:09:32.093886552 (539h40m31.964969598s) [noise=true , address=145.79.227.41:4085 , piece_id=X72JKDXT6S7CZOE3MDNRVDHF5TIO2V2MGS6GPQ2EEW3DSX7EFZQQ, seg_pos={0 0}, upload=473276]: Uploading piece... Dec 28 03:09:32.095140079 (539h40m31.966223114s) [noise=true , address=77.90.31.10:4519 , piece_id=BTQVCUIINZPJQGB7EM4ZTEYACDOFN7IWKEI3L6ONYGH2IEOTZSNA, seg_pos={0 0}, upload=473276]: Uploading piece... Dec 28 03:09:32.095358536 (539h40m31.966441571s) [noise=true , address=145.79.227.45:4231 , piece_id=3VCTP37YAVDZTCLW7R62D6JHDOS62YJOW4FNL2RS36LERGCMXOJA, seg_pos={0 0}, upload=473276]: Uploading piece... Dec 28 03:09:32.096249953 (539h40m31.967332988s) [noise=true , address=45.91.116.5:4298 , piece_id=4T7UVHYLPRJLUUHLUPSQUXAEUXH2Y2ZWENWV7VFEWZDM76HOHK3Q, seg_pos={0 0}, upload=473276]: Uploading piece... Dec 28 03:09:32.097530952 (539h40m31.968613987s) [noise=true , address=145.79.227.42:4121 , piece_id=7QF7YSXAF6F4AGXTTXYBFZ4UXXBAQZCEKI2TAUMQJDXOQ3MDEILQ, seg_pos={0 0}, upload=473276]: Uploading piece... Dec 28 03:09:32.101540176 (539h40m31.972623222s) [noise=true , address=145.79.227.38:4978 , piece_id=VN6FCKACQRZ7MWPUIAUURPHCNML6DBRHUQ4LXNCTBFSVSNXMKLWA, seg_pos={0 0}, upload=473276]: Uploading piece... Dec 28 03:09:32.103185900 (539h40m31.974268935s) [noise=true , address=77.90.31.13:4629 , piece_id=ZAQCKXGM45ODLTTLH4GARCW3RRMY3PN3DYUKIPYKGLFY2AWJLYVA, seg_pos={0 0}, upload=473276]: Uploading piece... Dec 28 03:09:32.104285855 (539h40m31.97536889s) [noise=true , address=45.91.116.14:4189 , piece_id=E6GZY3XPIL4ZPOBA273UAM2O45OB5A2ALGK2ODWNCE7WDJU3HYCA, seg_pos={0 0}, upload=473276]: Uploading piece... Dec 28 03:09:32.105979329 (539h40m31.977062364s) [noise=true , address=145.79.227.15:4874 , piece_id=M5G7FSNIBFQ7HMPFUEQ5JD6FL53G6LQADAWGK5Q2CWPZISZUT3TA, seg_pos={0 0}, upload=473276]: Uploading piece... Dec 28 03:09:32.106435507 (539h40m31.977518542s) [noise=true , address=145.79.227.37:4566 , piece_id=56OEILXQJTPRZMGD6J5C3FJAJHHATMPBGSXZSXEO5GHRXOSDHMLA, seg_pos={0 0}, upload=473276]: Uploading piece... Dec 28 03:09:33.820927223 (539h40m33.692010258s) [upload=473276]: Next returned nil segment Dec 28 03:09:33.820942673 (539h40m33.692025708s) [upload=473276]: Waiting for error group managing segments... Dec 28 03:09:34.116226980 (539h40m33.987310025s) [noise=true , address=145.79.227.45:4231 , piece_id=3VCTP37YAVDZTCLW7R62D6JHDOS62YJOW4FNL2RS36LERGCMXOJA, seg_pos={0 0}, upload=473276]: Done uploading piece. err: <nil> Dec 28 03:09:34.118383466 (539h40m33.989466511s) [noise=true , address=145.79.227.36:4517 , piece_id=W2DV45TQYAZL3V4ZKGFXUXYVMG7DHHU4QTZ4WWAMMDYQ5OLEFKDQ, seg_pos={0 0}, upload=473276]: Done uploading piece. err: <nil> Dec 28 03:09:34.120008690 (539h40m33.991091725s) [noise=true , address=145.79.227.15:4874 , piece_id=M5G7FSNIBFQ7HMPFUEQ5JD6FL53G6LQADAWGK5Q2CWPZISZUT3TA, seg_pos={0 0}, upload=473276]: Done uploading piece. err: <nil> Dec 28 03:09:34.149025624 (539h40m34.020108659s) [noise=true , address=145.79.227.13:4802 , piece_id=IDDNJMKDGWBL53RTXHZ27W7M6WANVSC76T46MDMNOSVBKYLER34Q, seg_pos={0 0}, upload=473276]: Done uploading piece. err: <nil> Dec 28 03:09:34.301762367 (539h40m34.172845402s) [noise=true , address=77.90.31.13:4629 , piece_id=ZAQCKXGM45ODLTTLH4GARCW3RRMY3PN3DYUKIPYKGLFY2AWJLYVA, seg_pos={0 0}, upload=473276]: Done uploading piece. err: <nil> Dec 28 03:09:34.342051454 (539h40m34.213134489s) [noise=true , address=45.91.116.9:4261 , piece_id=QE4BN77KQCTWG6O4VIEN4JMWBRJ27DLOWX4BCJDJMKRLJITIPK5Q, seg_pos={0 0}, upload=473276]: Done uploading piece. err: <nil> Dec 28 03:09:34.377776209 (539h40m34.248859244s) [noise=true , address=45.91.116.5:4298 , piece_id=4T7UVHYLPRJLUUHLUPSQUXAEUXH2Y2ZWENWV7VFEWZDM76HOHK3Q, seg_pos={0 0}, upload=473276]: Done uploading piece. err: <nil> Dec 28 03:09:34.408185212 (539h40m34.279268247s) [noise=true , address=77.90.31.21:4880 , piece_id=QFJVHJIC3UTM67CEQHYMHKEG6AOGBAM5P6T475S4BXIZ2O752SLQ, seg_pos={0 0}, upload=473276]: Done uploading piece. err: <nil> Dec 28 03:09:34.636060612 (539h40m34.507143667s) [noise=true , address=45.91.116.11:4117 , piece_id=J3A5ANOJLZBBBX5EOFPWRTVLJ4MOPJWO2DAUHSQRYJSJNKOKUBPQ, seg_pos={0 0}, upload=473276]: Done uploading piece. err: <nil> Dec 28 03:09:34.751811039 (539h40m34.622894074s) [noise=true , address=145.79.227.38:4978 , piece_id=VN6FCKACQRZ7MWPUIAUURPHCNML6DBRHUQ4LXNCTBFSVSNXMKLWA, seg_pos={0 0}, upload=473276]: Done uploading piece. err: <nil> Dec 28 03:09:34.756647709 (539h40m34.627730744s) [noise=true , address=145.79.227.37:4566 , piece_id=56OEILXQJTPRZMGD6J5C3FJAJHHATMPBGSXZSXEO5GHRXOSDHMLA, seg_pos={0 0}, upload=473276]: Done uploading piece. err: <nil> Dec 28 03:09:34.759230737 (539h40m34.630313772s) [noise=true , address=145.79.227.42:4121 , piece_id=7QF7YSXAF6F4AGXTTXYBFZ4UXXBAQZCEKI2TAUMQJDXOQ3MDEILQ, seg_pos={0 0}, upload=473276]: Done uploading piece. err: <nil> Dec 28 03:09:34.967953718 (539h40m34.839036753s) [noise=true , address=145.79.227.41:4085 , piece_id=X72JKDXT6S7CZOE3MDNRVDHF5TIO2V2MGS6GPQ2EEW3DSX7EFZQQ, seg_pos={0 0}, upload=473276]: Done uploading piece. err: <nil> Dec 28 03:09:34.967973807 (539h40m34.839056842s) [seg_pos={0 0}, upload=473276]: Segment reached optimal threshold of 13 pieces. Dec 28 03:09:34.968177494 (539h40m34.839260539s) [noise=true , address=45.91.116.14:4189 , piece_id=E6GZY3XPIL4ZPOBA273UAM2O45OB5A2ALGK2ODWNCE7WDJU3HYCA, seg_pos={0 0}, upload=473276]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 28 03:09:34.968188125 (539h40m34.83927117s) [noise=true , address=45.91.116.14:4189 , piece_id=E6GZY3XPIL4ZPOBA273UAM2O45OB5A2ALGK2ODWNCE7WDJU3HYCA, seg_pos={0 0}, upload=473276]: Long tail cancellation due to optimal threshold Dec 28 03:09:34.968193855 (539h40m34.8392769s) [noise=true , address=77.90.31.10:4519 , piece_id=BTQVCUIINZPJQGB7EM4ZTEYACDOFN7IWKEI3L6ONYGH2IEOTZSNA, seg_pos={0 0}, upload=473276]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Dec 28 03:09:34.968202542 (539h40m34.839285577s) [noise=true , address=77.90.31.10:4519 , piece_id=BTQVCUIINZPJQGB7EM4ZTEYACDOFN7IWKEI3L6ONYGH2IEOTZSNA, seg_pos={0 0}, upload=473276]: Long tail cancellation due to optimal threshold Dec 28 03:09:34.968349261 (539h40m34.839432306s) [noise=true , address=145.79.227.7:4295 , piece_id=ZS55RLW2XXOXUK64T4ABWCZJ5J6WX32Z3PYG7NZW7XDH2DQGQVEA, seg_pos={0 0}, upload=473276]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT): protocol: write tcp 188.166.225.170:38156->145.79.227.7:4295: use of closed network connection; write tcp 188.166.225.170:38156->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38156->145.79.227.7:4295: use of closed network connection Dec 28 03:09:34.968355733 (539h40m34.839438768s) [noise=true , address=145.79.227.7:4295 , piece_id=ZS55RLW2XXOXUK64T4ABWCZJ5J6WX32Z3PYG7NZW7XDH2DQGQVEA, seg_pos={0 0}, upload=473276]: Long tail cancellation due to optimal threshold Dec 28 03:09:34.968378056 (539h40m34.839461121s) [seg_pos={0 0}, upload=473276]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 03:09:34.968481032 (539h40m34.839564077s) [upload=473276]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 03:09:34.968487644 (539h40m34.839570679s) [upload=473276]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 03:09:35.262134002 (539h40m35.133217037s) [upload=473276]: Done uploading segments... Dec 28 03:09:35.262152909 (539h40m35.133235944s) [upload=473276]: Upload finished. err: <nil> Duration: 30.004728388s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [..............X....X............X........X....X...........X.] duration: 5.000659692s trace: https://jaeger.dev.storj.io/trace/235003f053bdab FAIL (2025-12-28 03:09:58 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000683567s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [X.....X.....................................................] duration: 36.480449673s trace: https://jaeger.dev.storj.io/trace/1822be52d200e5b5 FAIL (2025-12-28 03:09:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1ZAfYZ4wgpZVvatj6wtHona3hTaPcLxAmXC1DT2uKQyo8nLQQ7, Piece ID: 5MMQ44MXRT4XEL4X6WO54WRG2XYMX3AN3FNQ2H2XIUTKBXI3EEPA) context deadline exceeded; piecestore: (Node ID: 124guAM82oibYZsMho67Em6Ke6yEZXwT6LPMLLSZZBLqwoFGUkn, Piece ID: 22WJIPX4SM5NTIAW5ZRM2UKX32CTDBFCXPCCHRRWRCUCOF5CRE5Q) context deadline exceeded; piecestore: (Node ID: 1e6p1TD5CqUyWdZjkrxh2kRqMuyDiMUr9eimfeayDAKwhtBrZ8, Piece ID: XRI5EJFN7YYUG4QKLDGEFT6YAUHCUTTSXF735UDR6LZZBGMAZDJQ) context deadline exceeded ======LOGS====== 5.234848204 (538h55m35.105931239s) [noise=true , address=77.90.31.11:4572 , piece_id=5MMQ44MXRT4XEL4X6WO54WRG2XYMX3AN3FNQ2H2XIUTKBXI3EEPA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.235412057 (538h55m35.106495102s) [noise=true , address=145.79.227.22:4512 , piece_id=D46YLTIKGHG36DQ4YBKBSUDV4GGB4WVDWNOJS27IYYLN6AYZ5QFQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.235820946 (538h55m35.106903981s) [noise=true , address=77.90.31.19:4874 , piece_id=RHZUXUYMJU6RKQGSDZM5WDWIN5SQ7YIRSRATNNKT5GEMU2QH444A, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.236192192 (538h55m35.107275237s) [noise=true , address=77.90.31.16:4760 , piece_id=XRI5EJFN7YYUG4QKLDGEFT6YAUHCUTTSXF735UDR6LZZBGMAZDJQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.236564772 (538h55m35.107647817s) [noise=true , address=145.79.227.40:4062 , piece_id=PW6QWIPQWVUETLC6DJ624G6IB4P6EM4SYB72NAMQ4V6NUKNLUYLQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.236929766 (538h55m35.108012812s) [noise=true , address=64.49.13.139:4480 , piece_id=JAJY3F7IDFFMFKCK2QJV6W74RIESLRKBLXMN27Q3C45JWEPLBZLQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.237916846 (538h55m35.108999881s) [noise=true , address=145.79.227.34:4932 , piece_id=UPXSCQ45QDGPKNJXRFJ33EAA3QENVHRVQARU5TFJTEWICZNR253Q, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.238852948 (538h55m35.109935983s) [noise=true , address=45.91.116.12:4257 , piece_id=ZI67LIAHZW4G357GWKDPTB5SMXYGNXZTTTDMNAETDS2B6SWVDAEQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239152910 (538h55m35.110235945s) [noise=true , address=145.79.227.44:4222 , piece_id=CZ2CS62CTHQFAIE3S3JO2N3VJ4XIR5YJ2LMRVLLX7I4VM2VZVQVA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239409969 (538h55m35.110493004s) [noise=true , address=145.79.227.30:4773 , piece_id=KUGJXZCNMUE7WSK3VLY4AUBSOT4H6IY4XB4NCBGAGCNUG7F7IODQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239681136 (538h55m35.110764171s) [noise=true , address=77.90.31.13:4635 , piece_id=IIOY5X2W3XW4C2OFEBRCABMYFBI24RRTYNUSKN6KJU7CTUW4KONQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239954676 (538h55m35.111037711s) [noise=true , address=45.91.116.9:4281 , piece_id=JZLDW777CBX2BP7MLOQV4NFP3H4XJVCNQXTHNSUWPVDOJLPZK36A, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.240189914 (538h55m35.111272949s) [noise=true , address=145.79.227.35:4512 , piece_id=3E3VCCC45XVT5FHFDKXSTGGAGYBEML3BKU6JJTT33LCD2YMOMBZQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.240449147 (538h55m35.111532182s) [noise=true , address=45.91.116.8:4064 , piece_id=22WJIPX4SM5NTIAW5ZRM2UKX32CTDBFCXPCCHRRWRCUCOF5CRE5Q, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.240639640 (538h55m35.111722675s) [noise=true , address=145.79.227.32:4853 , piece_id=MBNCFDBQ4SZZXV4F7J5TRMOUYUQ5IPKQCGYGGM5SOIFJ6ZYCAUYA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.241137798 (538h55m35.112220833s) [noise=true , address=77.90.31.16:4751 , piece_id=N3Z2LLMCIA2F42VXZA7IDRU4XEKGOICH6TRVERODU76MFFE4QSHA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.241383797 (538h55m35.112466842s) [noise=true , address=77.90.31.18:4836 , piece_id=BSVEFI52EKN44CMINFUSZBNOH53IW5TXLVTF5IH6ZUTNII6RRLVA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:36.592880816 (538h55m36.463963871s) [upload=472631]: Next returned nil segment Dec 28 02:24:36.592901625 (538h55m36.46398467s) [upload=472631]: Waiting for error group managing segments... Dec 28 02:24:36.946933741 (538h55m36.818016786s) [noise=true , address=64.49.13.139:4480 , piece_id=JAJY3F7IDFFMFKCK2QJV6W74RIESLRKBLXMN27Q3C45JWEPLBZLQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.128189692 (538h55m36.999272737s) [noise=true , address=145.79.227.35:4512 , piece_id=3E3VCCC45XVT5FHFDKXSTGGAGYBEML3BKU6JJTT33LCD2YMOMBZQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.160760084 (538h55m37.031843119s) [noise=true , address=145.79.227.34:4932 , piece_id=UPXSCQ45QDGPKNJXRFJ33EAA3QENVHRVQARU5TFJTEWICZNR253Q, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.288838207 (538h55m37.159921252s) [noise=true , address=77.90.31.11:4572 , piece_id=5MMQ44MXRT4XEL4X6WO54WRG2XYMX3AN3FNQ2H2XIUTKBXI3EEPA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.319996310 (538h55m37.191079345s) [noise=true , address=145.79.227.22:4512 , piece_id=D46YLTIKGHG36DQ4YBKBSUDV4GGB4WVDWNOJS27IYYLN6AYZ5QFQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.320513273 (538h55m37.191596319s) [noise=true , address=77.90.31.18:4836 , piece_id=BSVEFI52EKN44CMINFUSZBNOH53IW5TXLVTF5IH6ZUTNII6RRLVA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.336522926 (538h55m37.207605971s) [noise=true , address=145.79.227.32:4853 , piece_id=MBNCFDBQ4SZZXV4F7J5TRMOUYUQ5IPKQCGYGGM5SOIFJ6ZYCAUYA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.384868031 (538h55m37.255951066s) [noise=true , address=77.90.31.16:4751 , piece_id=N3Z2LLMCIA2F42VXZA7IDRU4XEKGOICH6TRVERODU76MFFE4QSHA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.453563554 (538h55m37.324646599s) [noise=true , address=45.91.116.8:4064 , piece_id=22WJIPX4SM5NTIAW5ZRM2UKX32CTDBFCXPCCHRRWRCUCOF5CRE5Q, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.479454839 (538h55m37.350537884s) [noise=true , address=45.91.116.12:4257 , piece_id=ZI67LIAHZW4G357GWKDPTB5SMXYGNXZTTTDMNAETDS2B6SWVDAEQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.659710174 (538h55m37.530793219s) [noise=true , address=77.90.31.13:4635 , piece_id=IIOY5X2W3XW4C2OFEBRCABMYFBI24RRTYNUSKN6KJU7CTUW4KONQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.715039128 (538h55m37.586122163s) [noise=true , address=77.90.31.16:4760 , piece_id=XRI5EJFN7YYUG4QKLDGEFT6YAUHCUTTSXF735UDR6LZZBGMAZDJQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.977473406 (538h55m37.848556441s) [noise=true , address=77.90.31.19:4874 , piece_id=RHZUXUYMJU6RKQGSDZM5WDWIN5SQ7YIRSRATNNKT5GEMU2QH444A, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:38.016302745 (538h55m37.88738578s) [noise=true , address=145.79.227.30:4773 , piece_id=KUGJXZCNMUE7WSK3VLY4AUBSOT4H6IY4XB4NCBGAGCNUG7F7IODQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:38.016326982 (538h55m37.887410027s) [seg_pos={0 0}, upload=472631]: Segment reached optimal threshold of 14 pieces. Dec 28 02:24:38.016585353 (538h55m37.887668398s) [noise=true , address=145.79.227.30:4766 , piece_id=67M2DMPAVWIQKQSWUAG73D4E4NRRSG4XEVPOUVIJPEQOUV7A6VVA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1k6zdamuKPkBdzyNZNWx4FCmaTkv33PchcwvTRR8bgcETBs4K7): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016594450 (538h55m37.887677495s) [noise=true , address=145.79.227.30:4766 , piece_id=67M2DMPAVWIQKQSWUAG73D4E4NRRSG4XEVPOUVIJPEQOUV7A6VVA, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016627093 (538h55m37.887710148s) [noise=true , address=45.91.116.9:4281 , piece_id=JZLDW777CBX2BP7MLOQV4NFP3H4XJVCNQXTHNSUWPVDOJLPZK36A, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gxf7DpJZthyR6f1PExX56tJavjhPw9Gxb4h8mtmeYKW3t3tS1): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016635258 (538h55m37.887718293s) [noise=true , address=45.91.116.9:4281 , piece_id=JZLDW777CBX2BP7MLOQV4NFP3H4XJVCNQXTHNSUWPVDOJLPZK36A, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016871298 (538h55m37.887954363s) [noise=true , address=145.79.227.44:4222 , piece_id=CZ2CS62CTHQFAIE3S3JO2N3VJ4XIR5YJ2LMRVLLX7I4VM2VZVQVA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZKW35KvrsoYRefbYeUFZo8bh4XZYycNKGEvTFjzibezjXgj3e): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016880124 (538h55m37.887963169s) [noise=true , address=145.79.227.44:4222 , piece_id=CZ2CS62CTHQFAIE3S3JO2N3VJ4XIR5YJ2LMRVLLX7I4VM2VZVQVA, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016934839 (538h55m37.888017884s) [noise=true , address=145.79.227.40:4062 , piece_id=PW6QWIPQWVUETLC6DJ624G6IB4P6EM4SYB72NAMQ4V6NUKNLUYLQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016943425 (538h55m37.88802649s) [noise=true , address=145.79.227.40:4062 , piece_id=PW6QWIPQWVUETLC6DJ624G6IB4P6EM4SYB72NAMQ4V6NUKNLUYLQ, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016968252 (538h55m37.888051297s) [seg_pos={0 0}, upload=472631]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 02:24:38.017067822 (538h55m37.888150867s) [upload=472631]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 02:24:38.017074244 (538h55m37.888157289s) [upload=472631]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 02:24:38.304306667 (538h55m38.175389712s) [upload=472631]: Done uploading segments... Dec 28 02:24:38.304329641 (538h55m38.175412686s) [upload=472631]: Upload finished. err: <nil> Duration: 36.480476574s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [...............X....X............X........X....X...........X] duration: 5.00115042s trace: https://jaeger.dev.storj.io/trace/235003f053bdab FAIL (2025-12-28 03:09:42 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001213249s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [................X....X............X........X....X...........] duration: 3.379661433s trace: https://jaeger.dev.storj.io/trace/25c8812909e078aa FAIL (2025-12-28 03:09:08 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.379693846s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 47 out of 60 is failed (threshold=3) [.XXX.XXXXXX.XXXX.XXXX..X..XXXXX.XXXXXXXXXXXXX.XXXXXXXXX.X..X] duration: 30.011790304s trace: https://jaeger.dev.storj.io/trace/1a1827a20bcd6019 FAIL (2025-12-28 03:09:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: 5LCFB64DHJH2CCGD2XYSFCIQWEMB5ZFTXYRL5GF5HRPG7LG5GYHA) context deadline exceeded; piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: WMUZFK2B2NYP7MS4X4VPT7F5RW3MD342E4PIWZHHSQO3BIIZKNIA) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: 4YANGW7GT6DSRW747TB25QULR7F2JECOG3AF2E5E6CJLIYDHKUTQ) context deadline exceeded ======LOGS====== ec 28 03:08:33.441843475 (539h39m33.31292652s) [upload=473260]: Got next segment. Inline: false Dec 28 03:08:33.441896957 (539h39m33.312979992s) [upload=473260]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 03:08:33.441902637 (539h39m33.312985683s) [upload=473260]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 03:08:33.703309476 (539h39m33.574392521s) [seg_pos={0 0}, upload=473260]: Begin upload segment... Dec 28 03:08:33.703361515 (539h39m33.574444551s) [seg_pos={0 0}, upload=473260]: Done begin upload segment. Dec 28 03:08:33.703427040 (539h39m33.574510075s) [noise=true , address=145.79.227.38:4979 , piece_id=77NVWT4TQ6O3QIWXYP6LLSVKCEWQFK6FRZFNM2PAW2WKXQ6SDJCQ, seg_pos={0 0}, upload=473260]: Uploading piece... Dec 28 03:08:33.703459302 (539h39m33.574542337s) [noise=true , address=77.90.31.21:4880 , piece_id=J4MDLUPQLXRLDVRXKEXQYMVYCDY72N6CK5V7GLMZN7KTYCBV5XEA, seg_pos={0 0}, upload=473260]: Uploading piece... Dec 28 03:08:33.704053453 (539h39m33.575136488s) [noise=true , address=145.79.227.13:4804 , piece_id=QRJMYZHHOTGCLD5Z77DTXN4LTDTZUEIYB3TDCDARUWM6HPZ5GIHA, seg_pos={0 0}, upload=473260]: Uploading piece... Dec 28 03:08:33.704290243 (539h39m33.575373278s) [noise=true , address=45.91.116.12:4227 , piece_id=XCV5B7ZS5UVYFAPDITJC5PDX4HPN62X7XJGZNCUSFGXE3UFHEHEQ, seg_pos={0 0}, upload=473260]: Uploading piece... Dec 28 03:08:33.704629550 (539h39m33.575712585s) [noise=true , address=45.91.116.6:4973 , piece_id=JU4VY4V7N3AA5GPFL7ZHMO2ETTLRABQZDGMC2IMMEJOOCOIAUALA, seg_pos={0 0}, upload=473260]: Uploading piece... Dec 28 03:08:33.704900806 (539h39m33.575983841s) [noise=true , address=45.91.116.8:4046 , piece_id=4YANGW7GT6DSRW747TB25QULR7F2JECOG3AF2E5E6CJLIYDHKUTQ, seg_pos={0 0}, upload=473260]: Uploading piece... Dec 28 03:08:33.705235504 (539h39m33.576318539s) [noise=true , address=77.90.31.17:4770 , piece_id=6CA3S52ID2FD2UPUGOTOHYWEJBBTGAKAOB26MG3WCOVVQV6GKGOQ, seg_pos={0 0}, upload=473260]: Uploading piece... Dec 28 03:08:33.705553760 (539h39m33.576636795s) [noise=true , address=45.91.116.10:4081 , piece_id=SA64SBJ2JGKZVUJXLWSQNM7YI6UFXVEKCIEE23OEUOTBW3MLA3NQ, seg_pos={0 0}, upload=473260]: Uploading piece... Dec 28 03:08:33.705972617 (539h39m33.577055652s) [noise=true , address=45.91.116.14:4189 , piece_id=ZTFNX2UC6HZ66DDWBDJFGXDKCAP6JZHYF3XYSWXR24UYN5L55X6A, seg_pos={0 0}, upload=473260]: Uploading piece... Dec 28 03:08:33.706368260 (539h39m33.577451306s) [noise=true , address=77.90.31.11:4556 , piece_id=6IYZ53WUPYR6UAHAL5MNTPUJSWZVDT2HTTA6R3NPE24DZRHWMK2Q, seg_pos={0 0}, upload=473260]: Uploading piece... Dec 28 03:08:33.706773753 (539h39m33.577856798s) [noise=true , address=145.79.227.39:4013 , piece_id=MTRCBGLS3VN3IKNP7XD2XFSBI2LOFIIK32LJWANEQDDKTRAZKK4Q, seg_pos={0 0}, upload=473260]: Uploading piece... Dec 28 03:08:33.707114933 (539h39m33.578197968s) [noise=true , address=45.91.116.9:4263 , piece_id=5LCFB64DHJH2CCGD2XYSFCIQWEMB5ZFTXYRL5GF5HRPG7LG5GYHA, seg_pos={0 0}, upload=473260]: Uploading piece... Dec 28 03:08:33.707505897 (539h39m33.578588932s) [noise=true , address=145.79.227.52:4445 , piece_id=CKWPHRXTNLH2BWE37H7MCJN2FUUTOQ3GXBZ3XPGDZ6XGYNRKV4TQ, seg_pos={0 0}, upload=473260]: Uploading piece... Dec 28 03:08:33.707806328 (539h39m33.578889364s) [noise=true , address=145.79.227.17:4972 , piece_id=XXMKVBCRHSFVTBUUFMTJHY4RWSKAP2TQ5MJWEHW5R76PJE7KDTJQ, seg_pos={0 0}, upload=473260]: Uploading piece... Dec 28 03:08:33.708180922 (539h39m33.579263957s) [noise=true , address=145.79.227.27:4691 , piece_id=FD33PGNVKMLM7ZGKKYAHYEEJLCMEKMCOXDPO5OP5SNHA5T6M5A7A, seg_pos={0 0}, upload=473260]: Uploading piece... Dec 28 03:08:33.708530898 (539h39m33.579613943s) [noise=true , address=77.90.31.18:4808 , piece_id=WMUZFK2B2NYP7MS4X4VPT7F5RW3MD342E4PIWZHHSQO3BIIZKNIA, seg_pos={0 0}, upload=473260]: Uploading piece... Dec 28 03:08:35.576057586 (539h39m35.447140621s) [upload=473260]: Next returned nil segment Dec 28 03:08:35.576071934 (539h39m35.447154979s) [upload=473260]: Waiting for error group managing segments... Dec 28 03:08:35.945979925 (539h39m35.81706297s) [noise=true , address=45.91.116.8:4046 , piece_id=4YANGW7GT6DSRW747TB25QULR7F2JECOG3AF2E5E6CJLIYDHKUTQ, seg_pos={0 0}, upload=473260]: Done uploading piece. err: <nil> Dec 28 03:08:35.954801572 (539h39m35.825884608s) [noise=true , address=45.91.116.9:4263 , piece_id=5LCFB64DHJH2CCGD2XYSFCIQWEMB5ZFTXYRL5GF5HRPG7LG5GYHA, seg_pos={0 0}, upload=473260]: Done uploading piece. err: <nil> Dec 28 03:08:35.979496696 (539h39m35.850579731s) [noise=true , address=77.90.31.17:4770 , piece_id=6CA3S52ID2FD2UPUGOTOHYWEJBBTGAKAOB26MG3WCOVVQV6GKGOQ, seg_pos={0 0}, upload=473260]: Done uploading piece. err: <nil> Dec 28 03:08:36.004021996 (539h39m35.875105041s) [noise=true , address=77.90.31.11:4556 , piece_id=6IYZ53WUPYR6UAHAL5MNTPUJSWZVDT2HTTA6R3NPE24DZRHWMK2Q, seg_pos={0 0}, upload=473260]: Done uploading piece. err: <nil> Dec 28 03:08:36.015043403 (539h39m35.886126438s) [noise=true , address=77.90.31.21:4880 , piece_id=J4MDLUPQLXRLDVRXKEXQYMVYCDY72N6CK5V7GLMZN7KTYCBV5XEA, seg_pos={0 0}, upload=473260]: Done uploading piece. err: <nil> Dec 28 03:08:36.017938245 (539h39m35.88902128s) [noise=true , address=77.90.31.18:4808 , piece_id=WMUZFK2B2NYP7MS4X4VPT7F5RW3MD342E4PIWZHHSQO3BIIZKNIA, seg_pos={0 0}, upload=473260]: Done uploading piece. err: <nil> Dec 28 03:08:36.063760697 (539h39m35.934843743s) [noise=true , address=45.91.116.10:4081 , piece_id=SA64SBJ2JGKZVUJXLWSQNM7YI6UFXVEKCIEE23OEUOTBW3MLA3NQ, seg_pos={0 0}, upload=473260]: Done uploading piece. err: <nil> Dec 28 03:08:36.072117129 (539h39m35.943200175s) [noise=true , address=45.91.116.14:4189 , piece_id=ZTFNX2UC6HZ66DDWBDJFGXDKCAP6JZHYF3XYSWXR24UYN5L55X6A, seg_pos={0 0}, upload=473260]: Done uploading piece. err: <nil> Dec 28 03:08:36.187691631 (539h39m36.058774666s) [noise=true , address=145.79.227.52:4445 , piece_id=CKWPHRXTNLH2BWE37H7MCJN2FUUTOQ3GXBZ3XPGDZ6XGYNRKV4TQ, seg_pos={0 0}, upload=473260]: Done uploading piece. err: <nil> Dec 28 03:08:36.194148886 (539h39m36.065231921s) [noise=true , address=145.79.227.38:4979 , piece_id=77NVWT4TQ6O3QIWXYP6LLSVKCEWQFK6FRZFNM2PAW2WKXQ6SDJCQ, seg_pos={0 0}, upload=473260]: Done uploading piece. err: <nil> Dec 28 03:08:36.282961720 (539h39m36.154044765s) [noise=true , address=45.91.116.6:4973 , piece_id=JU4VY4V7N3AA5GPFL7ZHMO2ETTLRABQZDGMC2IMMEJOOCOIAUALA, seg_pos={0 0}, upload=473260]: Done uploading piece. err: <nil> Dec 28 03:08:36.327610337 (539h39m36.198693372s) [noise=true , address=145.79.227.17:4972 , piece_id=XXMKVBCRHSFVTBUUFMTJHY4RWSKAP2TQ5MJWEHW5R76PJE7KDTJQ, seg_pos={0 0}, upload=473260]: Done uploading piece. err: <nil> Dec 28 03:08:36.348377151 (539h39m36.219460186s) [noise=true , address=145.79.227.39:4013 , piece_id=MTRCBGLS3VN3IKNP7XD2XFSBI2LOFIIK32LJWANEQDDKTRAZKK4Q, seg_pos={0 0}, upload=473260]: Done uploading piece. err: <nil> Dec 28 03:08:36.348394123 (539h39m36.219477168s) [seg_pos={0 0}, upload=473260]: Segment reached optimal threshold of 13 pieces. Dec 28 03:08:36.348576310 (539h39m36.219659345s) [noise=true , address=45.91.116.12:4227 , piece_id=XCV5B7ZS5UVYFAPDITJC5PDX4HPN62X7XJGZNCUSFGXE3UFHEHEQ, seg_pos={0 0}, upload=473260]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Dec 28 03:08:36.348583063 (539h39m36.219666098s) [noise=true , address=45.91.116.12:4227 , piece_id=XCV5B7ZS5UVYFAPDITJC5PDX4HPN62X7XJGZNCUSFGXE3UFHEHEQ, seg_pos={0 0}, upload=473260]: Long tail cancellation due to optimal threshold Dec 28 03:08:36.348762074 (539h39m36.219845109s) [noise=true , address=145.79.227.13:4804 , piece_id=QRJMYZHHOTGCLD5Z77DTXN4LTDTZUEIYB3TDCDARUWM6HPZ5GIHA, seg_pos={0 0}, upload=473260]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g): protocol: write tcp 188.166.225.170:57088->145.79.227.13:4804: use of closed network connection; write tcp 188.166.225.170:57088->145.79.227.13:4804: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57088->145.79.227.13:4804: use of closed network connection Dec 28 03:08:36.348777964 (539h39m36.219860999s) [noise=true , address=145.79.227.13:4804 , piece_id=QRJMYZHHOTGCLD5Z77DTXN4LTDTZUEIYB3TDCDARUWM6HPZ5GIHA, seg_pos={0 0}, upload=473260]: Long tail cancellation due to optimal threshold Dec 28 03:08:36.348791450 (539h39m36.219874485s) [noise=true , address=145.79.227.27:4691 , piece_id=FD33PGNVKMLM7ZGKKYAHYEEJLCMEKMCOXDPO5OP5SNHA5T6M5A7A, seg_pos={0 0}, upload=473260]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg): protocol: write tcp 188.166.225.170:57036->145.79.227.27:4691: use of closed network connection; write tcp 188.166.225.170:57036->145.79.227.27:4691: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57036->145.79.227.27:4691: use of closed network connection Dec 28 03:08:36.348857736 (539h39m36.219940791s) [noise=true , address=145.79.227.27:4691 , piece_id=FD33PGNVKMLM7ZGKKYAHYEEJLCMEKMCOXDPO5OP5SNHA5T6M5A7A, seg_pos={0 0}, upload=473260]: Long tail cancellation due to optimal threshold Dec 28 03:08:36.348877974 (539h39m36.21996101s) [seg_pos={0 0}, upload=473260]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 03:08:36.348969199 (539h39m36.220052244s) [upload=473260]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 03:08:36.348973697 (539h39m36.220056732s) [upload=473260]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 03:08:36.648724569 (539h39m36.519807614s) [upload=473260]: Done uploading segments... Dec 28 03:08:36.648745669 (539h39m36.519828714s) [upload=473260]: Upload finished. err: <nil> Duration: 30.011805844s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 46 out of 60 is failed (threshold=3) [..XXX.XXXXXX.XXXX.XXXX..X..XXXXX.XXXXXXXXXXXXX.XXXXXXXXX.X..] duration: 27.137811809s trace: https://jaeger.dev.storj.io/trace/43402463fe7eab1a FAIL (2025-12-28 03:08:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 03:06:25.539781194 (539h37m25.410864239s) [upload=473228]: Starting upload Dec 28 03:06:25.539870084 (539h37m25.410953139s) [upload=473228]: Done starting upload Dec 28 03:06:25.539945457 (539h37m25.411028522s) [upload=473228]: Uploading segments... Dec 28 03:06:25.540158973 (539h37m25.411242019s) [upload=473228]: Got next segment. Inline: false Dec 28 03:06:25.540204249 (539h37m25.411287304s) [upload=473228]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 03:06:25.540210432 (539h37m25.411293477s) [upload=473228]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 03:06:25.787931021 (539h37m25.659014067s) [seg_pos={0 0}, upload=473228]: Begin upload segment... Dec 28 03:06:25.787978081 (539h37m25.659061116s) [seg_pos={0 0}, upload=473228]: Done begin upload segment. Dec 28 03:06:25.788004822 (539h37m25.659087857s) [noise=true , address=45.91.116.12:4227 , piece_id=N36GRWHIBMUZ4EW2EO7DJMZAG2Y57KPNHPKJUF7VW3DJQJ2WIB2Q, seg_pos={0 0}, upload=473228]: Uploading piece... Dec 28 03:06:25.788110774 (539h37m25.659193809s) [noise=true , address=145.79.227.38:4979 , piece_id=7G4QR36QC5SAR75VSG6VFODMPYY4GQXCVME4W7NH4GPE5XTT4LGQ, seg_pos={0 0}, upload=473228]: Uploading piece... Dec 28 03:06:25.788688915 (539h37m25.65977196s) [noise=true , address=45.91.116.7:4012 , piece_id=2DCO4OLJHODU3A6B7UBW5ZWQARGPMHOASUUIVBWNBCNGZE4AVALA, seg_pos={0 0}, upload=473228]: Uploading piece... Dec 28 03:06:25.789108784 (539h37m25.660191819s) [noise=true , address=145.79.227.30:4761 , piece_id=6NKI7A6EEK7NIPILA6OS5PCOJYSU2OJKX6KYSUV2HJGN4BINA4GA, seg_pos={0 0}, upload=473228]: Uploading piece... Dec 28 03:06:25.789616070 (539h37m25.660699115s) [noise=true , address=145.79.227.5:4222 , piece_id=NZ3XWPYNSDXX77W6PVO6TWJIQ5SPXHNG5GRCWQPF2POUBXVDWAAQ, seg_pos={0 0}, upload=473228]: Uploading piece... Dec 28 03:06:25.790105602 (539h37m25.661188637s) [noise=true , address=45.91.116.6:4976 , piece_id=IUOJMI3HVMHFOVI6ISMJVNCYKJYH3ULWBQUZCAHIOM4RVBRUODEQ, seg_pos={0 0}, upload=473228]: Uploading piece... Dec 28 03:06:25.790748046 (539h37m25.661831091s) [noise=true , address=145.79.227.50:4373 , piece_id=IKHFUZEHJ2EBW3OSFQQQDW5GNZGLQ5I5ZV3CJUAH3XDQKHT7N3MA, seg_pos={0 0}, upload=473228]: Uploading piece... Dec 28 03:06:25.791965764 (539h37m25.663048799s) [noise=true , address=145.79.227.46:4602 , piece_id=4PPVGNTMLH4EOPATOXWJIJCIGRKEVSUQKUSTYSFAFMFXGGCRVSAQ, seg_pos={0 0}, upload=473228]: Uploading piece... Dec 28 03:06:25.792334296 (539h37m25.663417341s) [noise=true , address=45.91.116.11:4119 , piece_id=VOJGVUXON4ROJKWLNK73DJ2RAOFFVIGFHWN4DXP3TZIHTNXCJGGA, seg_pos={0 0}, upload=473228]: Uploading piece... Dec 28 03:06:25.793292421 (539h37m25.664375466s) [noise=true , address=145.79.227.43:4159 , piece_id=LLSYBUNVQF3632GGJLTQ5HLWV3OHJXOJ5SIDV4CQYYYFFNZVUBZA, seg_pos={0 0}, upload=473228]: Uploading piece... Dec 28 03:06:25.794032850 (539h37m25.665115895s) [noise=true , address=45.91.116.4:4937 , piece_id=QYLY6256QZH7I3EDPOQPXDEBF4KQVAQZMEKPKVSYCVFJI4MBLOCQ, seg_pos={0 0}, upload=473228]: Uploading piece... Dec 28 03:06:25.794677198 (539h37m25.665760243s) [noise=true , address=77.90.31.12:4589 , piece_id=FAZS5NBX6RBIN4GXPDDX7WKQGB7VSEIDN3CYK5RHIAJ45T64LFCQ, seg_pos={0 0}, upload=473228]: Uploading piece... Dec 28 03:06:25.796052135 (539h37m25.66713518s) [noise=true , address=145.79.227.42:4121 , piece_id=4VAENCAD4QYTFJB7XSCHHERHLV6OCAOMRLHQJTXSXA3ZSSCETX2Q, seg_pos={0 0}, upload=473228]: Uploading piece... Dec 28 03:06:25.797213988 (539h37m25.668297023s) [noise=true , address=145.79.227.14:4838 , piece_id=XMWMAECJHUXCZQHYCMMGODAQKGTCH2LMH5X36D26FURK5OFRJXAQ, seg_pos={0 0}, upload=473228]: Uploading piece... Dec 28 03:06:25.798132226 (539h37m25.669215271s) [noise=true , address=145.79.227.37:4567 , piece_id=UGSVC67ZBN5WCQCG4Q5X4DOTBESFJS4NUXE7MO7LK6WTVVHECXKA, seg_pos={0 0}, upload=473228]: Uploading piece... Dec 28 03:06:25.799968814 (539h37m25.671051849s) [noise=true , address=145.79.227.52:4446 , piece_id=I4ODPMLBSEVD7IGX7BNZEGRMMFXUDCU5PALXMEM2XVLBJF64RTSA, seg_pos={0 0}, upload=473228]: Uploading piece... Dec 28 03:06:27.535635102 (539h37m27.406718147s) [upload=473228]: Next returned nil segment Dec 28 03:06:27.535648488 (539h37m27.406731523s) [upload=473228]: Waiting for error group managing segments... Dec 28 03:06:27.823327777 (539h37m27.694410812s) [noise=true , address=145.79.227.5:4222 , piece_id=NZ3XWPYNSDXX77W6PVO6TWJIQ5SPXHNG5GRCWQPF2POUBXVDWAAQ, seg_pos={0 0}, upload=473228]: Done uploading piece. err: <nil> Dec 28 03:06:27.831463058 (539h37m27.702546093s) [noise=true , address=145.79.227.42:4121 , piece_id=4VAENCAD4QYTFJB7XSCHHERHLV6OCAOMRLHQJTXSXA3ZSSCETX2Q, seg_pos={0 0}, upload=473228]: Done uploading piece. err: <nil> Dec 28 03:06:27.858379592 (539h37m27.729462637s) [noise=true , address=145.79.227.52:4446 , piece_id=I4ODPMLBSEVD7IGX7BNZEGRMMFXUDCU5PALXMEM2XVLBJF64RTSA, seg_pos={0 0}, upload=473228]: Done uploading piece. err: <nil> Dec 28 03:06:27.896928034 (539h37m27.768011079s) [noise=true , address=145.79.227.14:4838 , piece_id=XMWMAECJHUXCZQHYCMMGODAQKGTCH2LMH5X36D26FURK5OFRJXAQ, seg_pos={0 0}, upload=473228]: Done uploading piece. err: <nil> Dec 28 03:06:27.907339489 (539h37m27.778422534s) [noise=true , address=145.79.227.50:4373 , piece_id=IKHFUZEHJ2EBW3OSFQQQDW5GNZGLQ5I5ZV3CJUAH3XDQKHT7N3MA, seg_pos={0 0}, upload=473228]: Done uploading piece. err: <nil> Dec 28 03:06:28.012220530 (539h37m27.883303565s) [noise=true , address=45.91.116.4:4937 , piece_id=QYLY6256QZH7I3EDPOQPXDEBF4KQVAQZMEKPKVSYCVFJI4MBLOCQ, seg_pos={0 0}, upload=473228]: Done uploading piece. err: <nil> Dec 28 03:06:28.027980019 (539h37m27.899063054s) [noise=true , address=77.90.31.12:4589 , piece_id=FAZS5NBX6RBIN4GXPDDX7WKQGB7VSEIDN3CYK5RHIAJ45T64LFCQ, seg_pos={0 0}, upload=473228]: Done uploading piece. err: <nil> Dec 28 03:06:28.152311838 (539h37m28.023394873s) [noise=true , address=45.91.116.11:4119 , piece_id=VOJGVUXON4ROJKWLNK73DJ2RAOFFVIGFHWN4DXP3TZIHTNXCJGGA, seg_pos={0 0}, upload=473228]: Done uploading piece. err: <nil> Dec 28 03:06:28.339080767 (539h37m28.210163812s) [noise=true , address=45.91.116.7:4012 , piece_id=2DCO4OLJHODU3A6B7UBW5ZWQARGPMHOASUUIVBWNBCNGZE4AVALA, seg_pos={0 0}, upload=473228]: Done uploading piece. err: <nil> Dec 28 03:06:28.787117704 (539h37m28.658200739s) [noise=true , address=45.91.116.12:4227 , piece_id=N36GRWHIBMUZ4EW2EO7DJMZAG2Y57KPNHPKJUF7VW3DJQJ2WIB2Q, seg_pos={0 0}, upload=473228]: Done uploading piece. err: <nil> Dec 28 03:06:28.924995937 (539h37m28.796078982s) [noise=true , address=145.79.227.37:4567 , piece_id=UGSVC67ZBN5WCQCG4Q5X4DOTBESFJS4NUXE7MO7LK6WTVVHECXKA, seg_pos={0 0}, upload=473228]: Done uploading piece. err: <nil> Dec 28 03:06:28.937379989 (539h37m28.808463024s) [noise=true , address=145.79.227.38:4979 , piece_id=7G4QR36QC5SAR75VSG6VFODMPYY4GQXCVME4W7NH4GPE5XTT4LGQ, seg_pos={0 0}, upload=473228]: Done uploading piece. err: <nil> Dec 28 03:06:29.067756979 (539h37m28.938840014s) [noise=true , address=45.91.116.6:4976 , piece_id=IUOJMI3HVMHFOVI6ISMJVNCYKJYH3ULWBQUZCAHIOM4RVBRUODEQ, seg_pos={0 0}, upload=473228]: Done uploading piece. err: <nil> Dec 28 03:06:29.067775363 (539h37m28.938858398s) [seg_pos={0 0}, upload=473228]: Segment reached optimal threshold of 13 pieces. Dec 28 03:06:29.067958341 (539h37m28.939041376s) [noise=true , address=145.79.227.43:4159 , piece_id=LLSYBUNVQF3632GGJLTQ5HLWV3OHJXOJ5SIDV4CQYYYFFNZVUBZA, seg_pos={0 0}, upload=473228]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Dec 28 03:06:29.067964633 (539h37m28.939047668s) [noise=true , address=145.79.227.43:4159 , piece_id=LLSYBUNVQF3632GGJLTQ5HLWV3OHJXOJ5SIDV4CQYYYFFNZVUBZA, seg_pos={0 0}, upload=473228]: Long tail cancellation due to optimal threshold Dec 28 03:06:29.068027944 (539h37m28.939110989s) [noise=true , address=145.79.227.46:4602 , piece_id=4PPVGNTMLH4EOPATOXWJIJCIGRKEVSUQKUSTYSFAFMFXGGCRVSAQ, seg_pos={0 0}, upload=473228]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kgUqvw1r3gSKw1bNjzys3AiAJXdkAnSafmvcPgNVPBbN9JS1d): protocol: expected piece hash; context canceled; EOF Dec 28 03:06:29.068038815 (539h37m28.93912185s) [noise=true , address=145.79.227.46:4602 , piece_id=4PPVGNTMLH4EOPATOXWJIJCIGRKEVSUQKUSTYSFAFMFXGGCRVSAQ, seg_pos={0 0}, upload=473228]: Long tail cancellation due to optimal threshold Dec 28 03:06:29.068057480 (539h37m28.939140525s) [noise=true , address=145.79.227.30:4761 , piece_id=6NKI7A6EEK7NIPILA6OS5PCOJYSU2OJKX6KYSUV2HJGN4BINA4GA, seg_pos={0 0}, upload=473228]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 28 03:06:29.068063552 (539h37m28.939146587s) [noise=true , address=145.79.227.30:4761 , piece_id=6NKI7A6EEK7NIPILA6OS5PCOJYSU2OJKX6KYSUV2HJGN4BINA4GA, seg_pos={0 0}, upload=473228]: Long tail cancellation due to optimal threshold Dec 28 03:06:29.068080845 (539h37m28.93916389s) [seg_pos={0 0}, upload=473228]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 03:06:29.068161328 (539h37m28.939244363s) [upload=473228]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 03:06:29.068167790 (539h37m28.939250825s) [upload=473228]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 03:06:29.360691865 (539h37m29.2317749s) [upload=473228]: Done uploading segments... Dec 28 03:06:29.360709899 (539h37m29.231792934s) [upload=473228]: Upload finished. err: <nil> Duration: 27.137840994s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X.....X....................................................] duration: 18.513413041s trace: https://jaeger.dev.storj.io/trace/1822be52d200e5b5 FAIL (2025-12-28 03:08:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1ZAfYZ4wgpZVvatj6wtHona3hTaPcLxAmXC1DT2uKQyo8nLQQ7, Piece ID: 5MMQ44MXRT4XEL4X6WO54WRG2XYMX3AN3FNQ2H2XIUTKBXI3EEPA) context deadline exceeded; piecestore: (Node ID: 124guAM82oibYZsMho67Em6Ke6yEZXwT6LPMLLSZZBLqwoFGUkn, Piece ID: 22WJIPX4SM5NTIAW5ZRM2UKX32CTDBFCXPCCHRRWRCUCOF5CRE5Q) context deadline exceeded; piecestore: (Node ID: 1e6p1TD5CqUyWdZjkrxh2kRqMuyDiMUr9eimfeayDAKwhtBrZ8, Piece ID: XRI5EJFN7YYUG4QKLDGEFT6YAUHCUTTSXF735UDR6LZZBGMAZDJQ) context deadline exceeded ======LOGS====== 5.234848204 (538h55m35.105931239s) [noise=true , address=77.90.31.11:4572 , piece_id=5MMQ44MXRT4XEL4X6WO54WRG2XYMX3AN3FNQ2H2XIUTKBXI3EEPA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.235412057 (538h55m35.106495102s) [noise=true , address=145.79.227.22:4512 , piece_id=D46YLTIKGHG36DQ4YBKBSUDV4GGB4WVDWNOJS27IYYLN6AYZ5QFQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.235820946 (538h55m35.106903981s) [noise=true , address=77.90.31.19:4874 , piece_id=RHZUXUYMJU6RKQGSDZM5WDWIN5SQ7YIRSRATNNKT5GEMU2QH444A, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.236192192 (538h55m35.107275237s) [noise=true , address=77.90.31.16:4760 , piece_id=XRI5EJFN7YYUG4QKLDGEFT6YAUHCUTTSXF735UDR6LZZBGMAZDJQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.236564772 (538h55m35.107647817s) [noise=true , address=145.79.227.40:4062 , piece_id=PW6QWIPQWVUETLC6DJ624G6IB4P6EM4SYB72NAMQ4V6NUKNLUYLQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.236929766 (538h55m35.108012812s) [noise=true , address=64.49.13.139:4480 , piece_id=JAJY3F7IDFFMFKCK2QJV6W74RIESLRKBLXMN27Q3C45JWEPLBZLQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.237916846 (538h55m35.108999881s) [noise=true , address=145.79.227.34:4932 , piece_id=UPXSCQ45QDGPKNJXRFJ33EAA3QENVHRVQARU5TFJTEWICZNR253Q, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.238852948 (538h55m35.109935983s) [noise=true , address=45.91.116.12:4257 , piece_id=ZI67LIAHZW4G357GWKDPTB5SMXYGNXZTTTDMNAETDS2B6SWVDAEQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239152910 (538h55m35.110235945s) [noise=true , address=145.79.227.44:4222 , piece_id=CZ2CS62CTHQFAIE3S3JO2N3VJ4XIR5YJ2LMRVLLX7I4VM2VZVQVA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239409969 (538h55m35.110493004s) [noise=true , address=145.79.227.30:4773 , piece_id=KUGJXZCNMUE7WSK3VLY4AUBSOT4H6IY4XB4NCBGAGCNUG7F7IODQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239681136 (538h55m35.110764171s) [noise=true , address=77.90.31.13:4635 , piece_id=IIOY5X2W3XW4C2OFEBRCABMYFBI24RRTYNUSKN6KJU7CTUW4KONQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239954676 (538h55m35.111037711s) [noise=true , address=45.91.116.9:4281 , piece_id=JZLDW777CBX2BP7MLOQV4NFP3H4XJVCNQXTHNSUWPVDOJLPZK36A, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.240189914 (538h55m35.111272949s) [noise=true , address=145.79.227.35:4512 , piece_id=3E3VCCC45XVT5FHFDKXSTGGAGYBEML3BKU6JJTT33LCD2YMOMBZQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.240449147 (538h55m35.111532182s) [noise=true , address=45.91.116.8:4064 , piece_id=22WJIPX4SM5NTIAW5ZRM2UKX32CTDBFCXPCCHRRWRCUCOF5CRE5Q, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.240639640 (538h55m35.111722675s) [noise=true , address=145.79.227.32:4853 , piece_id=MBNCFDBQ4SZZXV4F7J5TRMOUYUQ5IPKQCGYGGM5SOIFJ6ZYCAUYA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.241137798 (538h55m35.112220833s) [noise=true , address=77.90.31.16:4751 , piece_id=N3Z2LLMCIA2F42VXZA7IDRU4XEKGOICH6TRVERODU76MFFE4QSHA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.241383797 (538h55m35.112466842s) [noise=true , address=77.90.31.18:4836 , piece_id=BSVEFI52EKN44CMINFUSZBNOH53IW5TXLVTF5IH6ZUTNII6RRLVA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:36.592880816 (538h55m36.463963871s) [upload=472631]: Next returned nil segment Dec 28 02:24:36.592901625 (538h55m36.46398467s) [upload=472631]: Waiting for error group managing segments... Dec 28 02:24:36.946933741 (538h55m36.818016786s) [noise=true , address=64.49.13.139:4480 , piece_id=JAJY3F7IDFFMFKCK2QJV6W74RIESLRKBLXMN27Q3C45JWEPLBZLQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.128189692 (538h55m36.999272737s) [noise=true , address=145.79.227.35:4512 , piece_id=3E3VCCC45XVT5FHFDKXSTGGAGYBEML3BKU6JJTT33LCD2YMOMBZQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.160760084 (538h55m37.031843119s) [noise=true , address=145.79.227.34:4932 , piece_id=UPXSCQ45QDGPKNJXRFJ33EAA3QENVHRVQARU5TFJTEWICZNR253Q, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.288838207 (538h55m37.159921252s) [noise=true , address=77.90.31.11:4572 , piece_id=5MMQ44MXRT4XEL4X6WO54WRG2XYMX3AN3FNQ2H2XIUTKBXI3EEPA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.319996310 (538h55m37.191079345s) [noise=true , address=145.79.227.22:4512 , piece_id=D46YLTIKGHG36DQ4YBKBSUDV4GGB4WVDWNOJS27IYYLN6AYZ5QFQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.320513273 (538h55m37.191596319s) [noise=true , address=77.90.31.18:4836 , piece_id=BSVEFI52EKN44CMINFUSZBNOH53IW5TXLVTF5IH6ZUTNII6RRLVA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.336522926 (538h55m37.207605971s) [noise=true , address=145.79.227.32:4853 , piece_id=MBNCFDBQ4SZZXV4F7J5TRMOUYUQ5IPKQCGYGGM5SOIFJ6ZYCAUYA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.384868031 (538h55m37.255951066s) [noise=true , address=77.90.31.16:4751 , piece_id=N3Z2LLMCIA2F42VXZA7IDRU4XEKGOICH6TRVERODU76MFFE4QSHA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.453563554 (538h55m37.324646599s) [noise=true , address=45.91.116.8:4064 , piece_id=22WJIPX4SM5NTIAW5ZRM2UKX32CTDBFCXPCCHRRWRCUCOF5CRE5Q, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.479454839 (538h55m37.350537884s) [noise=true , address=45.91.116.12:4257 , piece_id=ZI67LIAHZW4G357GWKDPTB5SMXYGNXZTTTDMNAETDS2B6SWVDAEQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.659710174 (538h55m37.530793219s) [noise=true , address=77.90.31.13:4635 , piece_id=IIOY5X2W3XW4C2OFEBRCABMYFBI24RRTYNUSKN6KJU7CTUW4KONQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.715039128 (538h55m37.586122163s) [noise=true , address=77.90.31.16:4760 , piece_id=XRI5EJFN7YYUG4QKLDGEFT6YAUHCUTTSXF735UDR6LZZBGMAZDJQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.977473406 (538h55m37.848556441s) [noise=true , address=77.90.31.19:4874 , piece_id=RHZUXUYMJU6RKQGSDZM5WDWIN5SQ7YIRSRATNNKT5GEMU2QH444A, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:38.016302745 (538h55m37.88738578s) [noise=true , address=145.79.227.30:4773 , piece_id=KUGJXZCNMUE7WSK3VLY4AUBSOT4H6IY4XB4NCBGAGCNUG7F7IODQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:38.016326982 (538h55m37.887410027s) [seg_pos={0 0}, upload=472631]: Segment reached optimal threshold of 14 pieces. Dec 28 02:24:38.016585353 (538h55m37.887668398s) [noise=true , address=145.79.227.30:4766 , piece_id=67M2DMPAVWIQKQSWUAG73D4E4NRRSG4XEVPOUVIJPEQOUV7A6VVA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1k6zdamuKPkBdzyNZNWx4FCmaTkv33PchcwvTRR8bgcETBs4K7): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016594450 (538h55m37.887677495s) [noise=true , address=145.79.227.30:4766 , piece_id=67M2DMPAVWIQKQSWUAG73D4E4NRRSG4XEVPOUVIJPEQOUV7A6VVA, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016627093 (538h55m37.887710148s) [noise=true , address=45.91.116.9:4281 , piece_id=JZLDW777CBX2BP7MLOQV4NFP3H4XJVCNQXTHNSUWPVDOJLPZK36A, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gxf7DpJZthyR6f1PExX56tJavjhPw9Gxb4h8mtmeYKW3t3tS1): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016635258 (538h55m37.887718293s) [noise=true , address=45.91.116.9:4281 , piece_id=JZLDW777CBX2BP7MLOQV4NFP3H4XJVCNQXTHNSUWPVDOJLPZK36A, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016871298 (538h55m37.887954363s) [noise=true , address=145.79.227.44:4222 , piece_id=CZ2CS62CTHQFAIE3S3JO2N3VJ4XIR5YJ2LMRVLLX7I4VM2VZVQVA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZKW35KvrsoYRefbYeUFZo8bh4XZYycNKGEvTFjzibezjXgj3e): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016880124 (538h55m37.887963169s) [noise=true , address=145.79.227.44:4222 , piece_id=CZ2CS62CTHQFAIE3S3JO2N3VJ4XIR5YJ2LMRVLLX7I4VM2VZVQVA, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016934839 (538h55m37.888017884s) [noise=true , address=145.79.227.40:4062 , piece_id=PW6QWIPQWVUETLC6DJ624G6IB4P6EM4SYB72NAMQ4V6NUKNLUYLQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016943425 (538h55m37.88802649s) [noise=true , address=145.79.227.40:4062 , piece_id=PW6QWIPQWVUETLC6DJ624G6IB4P6EM4SYB72NAMQ4V6NUKNLUYLQ, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016968252 (538h55m37.888051297s) [seg_pos={0 0}, upload=472631]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 02:24:38.017067822 (538h55m37.888150867s) [upload=472631]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 02:24:38.017074244 (538h55m37.888157289s) [upload=472631]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 02:24:38.304306667 (538h55m38.175389712s) [upload=472631]: Done uploading segments... Dec 28 02:24:38.304329641 (538h55m38.175412686s) [upload=472631]: Upload finished. err: <nil> Duration: 18.513438298s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [X................X....X............X........X....X..........] duration: 5.00087384s trace: https://jaeger.dev.storj.io/trace/25c8812909e078aa FAIL (2025-12-28 03:08:18 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000914035s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [.X................X....X............X........X....X.........] duration: 3.572549118s trace: https://jaeger.dev.storj.io/trace/25c8812909e078aa FAIL (2025-12-28 03:07:53 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.572568604s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..X.....X...................................................] duration: 18.777649343s trace: https://jaeger.dev.storj.io/trace/1822be52d200e5b5 FAIL (2025-12-28 03:07:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1ZAfYZ4wgpZVvatj6wtHona3hTaPcLxAmXC1DT2uKQyo8nLQQ7, Piece ID: 5MMQ44MXRT4XEL4X6WO54WRG2XYMX3AN3FNQ2H2XIUTKBXI3EEPA) context deadline exceeded; piecestore: (Node ID: 124guAM82oibYZsMho67Em6Ke6yEZXwT6LPMLLSZZBLqwoFGUkn, Piece ID: 22WJIPX4SM5NTIAW5ZRM2UKX32CTDBFCXPCCHRRWRCUCOF5CRE5Q) context deadline exceeded; piecestore: (Node ID: 1e6p1TD5CqUyWdZjkrxh2kRqMuyDiMUr9eimfeayDAKwhtBrZ8, Piece ID: XRI5EJFN7YYUG4QKLDGEFT6YAUHCUTTSXF735UDR6LZZBGMAZDJQ) context deadline exceeded ======LOGS====== 5.234848204 (538h55m35.105931239s) [noise=true , address=77.90.31.11:4572 , piece_id=5MMQ44MXRT4XEL4X6WO54WRG2XYMX3AN3FNQ2H2XIUTKBXI3EEPA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.235412057 (538h55m35.106495102s) [noise=true , address=145.79.227.22:4512 , piece_id=D46YLTIKGHG36DQ4YBKBSUDV4GGB4WVDWNOJS27IYYLN6AYZ5QFQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.235820946 (538h55m35.106903981s) [noise=true , address=77.90.31.19:4874 , piece_id=RHZUXUYMJU6RKQGSDZM5WDWIN5SQ7YIRSRATNNKT5GEMU2QH444A, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.236192192 (538h55m35.107275237s) [noise=true , address=77.90.31.16:4760 , piece_id=XRI5EJFN7YYUG4QKLDGEFT6YAUHCUTTSXF735UDR6LZZBGMAZDJQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.236564772 (538h55m35.107647817s) [noise=true , address=145.79.227.40:4062 , piece_id=PW6QWIPQWVUETLC6DJ624G6IB4P6EM4SYB72NAMQ4V6NUKNLUYLQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.236929766 (538h55m35.108012812s) [noise=true , address=64.49.13.139:4480 , piece_id=JAJY3F7IDFFMFKCK2QJV6W74RIESLRKBLXMN27Q3C45JWEPLBZLQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.237916846 (538h55m35.108999881s) [noise=true , address=145.79.227.34:4932 , piece_id=UPXSCQ45QDGPKNJXRFJ33EAA3QENVHRVQARU5TFJTEWICZNR253Q, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.238852948 (538h55m35.109935983s) [noise=true , address=45.91.116.12:4257 , piece_id=ZI67LIAHZW4G357GWKDPTB5SMXYGNXZTTTDMNAETDS2B6SWVDAEQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239152910 (538h55m35.110235945s) [noise=true , address=145.79.227.44:4222 , piece_id=CZ2CS62CTHQFAIE3S3JO2N3VJ4XIR5YJ2LMRVLLX7I4VM2VZVQVA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239409969 (538h55m35.110493004s) [noise=true , address=145.79.227.30:4773 , piece_id=KUGJXZCNMUE7WSK3VLY4AUBSOT4H6IY4XB4NCBGAGCNUG7F7IODQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239681136 (538h55m35.110764171s) [noise=true , address=77.90.31.13:4635 , piece_id=IIOY5X2W3XW4C2OFEBRCABMYFBI24RRTYNUSKN6KJU7CTUW4KONQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239954676 (538h55m35.111037711s) [noise=true , address=45.91.116.9:4281 , piece_id=JZLDW777CBX2BP7MLOQV4NFP3H4XJVCNQXTHNSUWPVDOJLPZK36A, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.240189914 (538h55m35.111272949s) [noise=true , address=145.79.227.35:4512 , piece_id=3E3VCCC45XVT5FHFDKXSTGGAGYBEML3BKU6JJTT33LCD2YMOMBZQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.240449147 (538h55m35.111532182s) [noise=true , address=45.91.116.8:4064 , piece_id=22WJIPX4SM5NTIAW5ZRM2UKX32CTDBFCXPCCHRRWRCUCOF5CRE5Q, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.240639640 (538h55m35.111722675s) [noise=true , address=145.79.227.32:4853 , piece_id=MBNCFDBQ4SZZXV4F7J5TRMOUYUQ5IPKQCGYGGM5SOIFJ6ZYCAUYA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.241137798 (538h55m35.112220833s) [noise=true , address=77.90.31.16:4751 , piece_id=N3Z2LLMCIA2F42VXZA7IDRU4XEKGOICH6TRVERODU76MFFE4QSHA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.241383797 (538h55m35.112466842s) [noise=true , address=77.90.31.18:4836 , piece_id=BSVEFI52EKN44CMINFUSZBNOH53IW5TXLVTF5IH6ZUTNII6RRLVA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:36.592880816 (538h55m36.463963871s) [upload=472631]: Next returned nil segment Dec 28 02:24:36.592901625 (538h55m36.46398467s) [upload=472631]: Waiting for error group managing segments... Dec 28 02:24:36.946933741 (538h55m36.818016786s) [noise=true , address=64.49.13.139:4480 , piece_id=JAJY3F7IDFFMFKCK2QJV6W74RIESLRKBLXMN27Q3C45JWEPLBZLQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.128189692 (538h55m36.999272737s) [noise=true , address=145.79.227.35:4512 , piece_id=3E3VCCC45XVT5FHFDKXSTGGAGYBEML3BKU6JJTT33LCD2YMOMBZQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.160760084 (538h55m37.031843119s) [noise=true , address=145.79.227.34:4932 , piece_id=UPXSCQ45QDGPKNJXRFJ33EAA3QENVHRVQARU5TFJTEWICZNR253Q, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.288838207 (538h55m37.159921252s) [noise=true , address=77.90.31.11:4572 , piece_id=5MMQ44MXRT4XEL4X6WO54WRG2XYMX3AN3FNQ2H2XIUTKBXI3EEPA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.319996310 (538h55m37.191079345s) [noise=true , address=145.79.227.22:4512 , piece_id=D46YLTIKGHG36DQ4YBKBSUDV4GGB4WVDWNOJS27IYYLN6AYZ5QFQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.320513273 (538h55m37.191596319s) [noise=true , address=77.90.31.18:4836 , piece_id=BSVEFI52EKN44CMINFUSZBNOH53IW5TXLVTF5IH6ZUTNII6RRLVA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.336522926 (538h55m37.207605971s) [noise=true , address=145.79.227.32:4853 , piece_id=MBNCFDBQ4SZZXV4F7J5TRMOUYUQ5IPKQCGYGGM5SOIFJ6ZYCAUYA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.384868031 (538h55m37.255951066s) [noise=true , address=77.90.31.16:4751 , piece_id=N3Z2LLMCIA2F42VXZA7IDRU4XEKGOICH6TRVERODU76MFFE4QSHA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.453563554 (538h55m37.324646599s) [noise=true , address=45.91.116.8:4064 , piece_id=22WJIPX4SM5NTIAW5ZRM2UKX32CTDBFCXPCCHRRWRCUCOF5CRE5Q, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.479454839 (538h55m37.350537884s) [noise=true , address=45.91.116.12:4257 , piece_id=ZI67LIAHZW4G357GWKDPTB5SMXYGNXZTTTDMNAETDS2B6SWVDAEQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.659710174 (538h55m37.530793219s) [noise=true , address=77.90.31.13:4635 , piece_id=IIOY5X2W3XW4C2OFEBRCABMYFBI24RRTYNUSKN6KJU7CTUW4KONQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.715039128 (538h55m37.586122163s) [noise=true , address=77.90.31.16:4760 , piece_id=XRI5EJFN7YYUG4QKLDGEFT6YAUHCUTTSXF735UDR6LZZBGMAZDJQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.977473406 (538h55m37.848556441s) [noise=true , address=77.90.31.19:4874 , piece_id=RHZUXUYMJU6RKQGSDZM5WDWIN5SQ7YIRSRATNNKT5GEMU2QH444A, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:38.016302745 (538h55m37.88738578s) [noise=true , address=145.79.227.30:4773 , piece_id=KUGJXZCNMUE7WSK3VLY4AUBSOT4H6IY4XB4NCBGAGCNUG7F7IODQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:38.016326982 (538h55m37.887410027s) [seg_pos={0 0}, upload=472631]: Segment reached optimal threshold of 14 pieces. Dec 28 02:24:38.016585353 (538h55m37.887668398s) [noise=true , address=145.79.227.30:4766 , piece_id=67M2DMPAVWIQKQSWUAG73D4E4NRRSG4XEVPOUVIJPEQOUV7A6VVA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1k6zdamuKPkBdzyNZNWx4FCmaTkv33PchcwvTRR8bgcETBs4K7): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016594450 (538h55m37.887677495s) [noise=true , address=145.79.227.30:4766 , piece_id=67M2DMPAVWIQKQSWUAG73D4E4NRRSG4XEVPOUVIJPEQOUV7A6VVA, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016627093 (538h55m37.887710148s) [noise=true , address=45.91.116.9:4281 , piece_id=JZLDW777CBX2BP7MLOQV4NFP3H4XJVCNQXTHNSUWPVDOJLPZK36A, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gxf7DpJZthyR6f1PExX56tJavjhPw9Gxb4h8mtmeYKW3t3tS1): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016635258 (538h55m37.887718293s) [noise=true , address=45.91.116.9:4281 , piece_id=JZLDW777CBX2BP7MLOQV4NFP3H4XJVCNQXTHNSUWPVDOJLPZK36A, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016871298 (538h55m37.887954363s) [noise=true , address=145.79.227.44:4222 , piece_id=CZ2CS62CTHQFAIE3S3JO2N3VJ4XIR5YJ2LMRVLLX7I4VM2VZVQVA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZKW35KvrsoYRefbYeUFZo8bh4XZYycNKGEvTFjzibezjXgj3e): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016880124 (538h55m37.887963169s) [noise=true , address=145.79.227.44:4222 , piece_id=CZ2CS62CTHQFAIE3S3JO2N3VJ4XIR5YJ2LMRVLLX7I4VM2VZVQVA, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016934839 (538h55m37.888017884s) [noise=true , address=145.79.227.40:4062 , piece_id=PW6QWIPQWVUETLC6DJ624G6IB4P6EM4SYB72NAMQ4V6NUKNLUYLQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016943425 (538h55m37.88802649s) [noise=true , address=145.79.227.40:4062 , piece_id=PW6QWIPQWVUETLC6DJ624G6IB4P6EM4SYB72NAMQ4V6NUKNLUYLQ, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016968252 (538h55m37.888051297s) [seg_pos={0 0}, upload=472631]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 02:24:38.017067822 (538h55m37.888150867s) [upload=472631]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 02:24:38.017074244 (538h55m37.888157289s) [upload=472631]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 02:24:38.304306667 (538h55m38.175389712s) [upload=472631]: Done uploading segments... Dec 28 02:24:38.304329641 (538h55m38.175412686s) [upload=472631]: Upload finished. err: <nil> Duration: 18.777677807s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [..X................X....X............X........X....X........] duration: 5.00100682s trace: https://jaeger.dev.storj.io/trace/25c8812909e078aa FAIL (2025-12-28 03:07:14 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001037168s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 46 out of 60 is failed (threshold=3) [...XXX.XXXXXX.XXXX.XXXX..X..XXXXX.XXXXXXXXXXXXX.XXXXXXXXX.X.] duration: 8.030048899s trace: https://jaeger.dev.storj.io/trace/43402463fe7eab1a FAIL (2025-12-28 03:07:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 03:06:25.539781194 (539h37m25.410864239s) [upload=473228]: Starting upload Dec 28 03:06:25.539870084 (539h37m25.410953139s) [upload=473228]: Done starting upload Dec 28 03:06:25.539945457 (539h37m25.411028522s) [upload=473228]: Uploading segments... Dec 28 03:06:25.540158973 (539h37m25.411242019s) [upload=473228]: Got next segment. Inline: false Dec 28 03:06:25.540204249 (539h37m25.411287304s) [upload=473228]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 03:06:25.540210432 (539h37m25.411293477s) [upload=473228]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 03:06:25.787931021 (539h37m25.659014067s) [seg_pos={0 0}, upload=473228]: Begin upload segment... Dec 28 03:06:25.787978081 (539h37m25.659061116s) [seg_pos={0 0}, upload=473228]: Done begin upload segment. Dec 28 03:06:25.788004822 (539h37m25.659087857s) [noise=true , address=45.91.116.12:4227 , piece_id=N36GRWHIBMUZ4EW2EO7DJMZAG2Y57KPNHPKJUF7VW3DJQJ2WIB2Q, seg_pos={0 0}, upload=473228]: Uploading piece... Dec 28 03:06:25.788110774 (539h37m25.659193809s) [noise=true , address=145.79.227.38:4979 , piece_id=7G4QR36QC5SAR75VSG6VFODMPYY4GQXCVME4W7NH4GPE5XTT4LGQ, seg_pos={0 0}, upload=473228]: Uploading piece... Dec 28 03:06:25.788688915 (539h37m25.65977196s) [noise=true , address=45.91.116.7:4012 , piece_id=2DCO4OLJHODU3A6B7UBW5ZWQARGPMHOASUUIVBWNBCNGZE4AVALA, seg_pos={0 0}, upload=473228]: Uploading piece... Dec 28 03:06:25.789108784 (539h37m25.660191819s) [noise=true , address=145.79.227.30:4761 , piece_id=6NKI7A6EEK7NIPILA6OS5PCOJYSU2OJKX6KYSUV2HJGN4BINA4GA, seg_pos={0 0}, upload=473228]: Uploading piece... Dec 28 03:06:25.789616070 (539h37m25.660699115s) [noise=true , address=145.79.227.5:4222 , piece_id=NZ3XWPYNSDXX77W6PVO6TWJIQ5SPXHNG5GRCWQPF2POUBXVDWAAQ, seg_pos={0 0}, upload=473228]: Uploading piece... Dec 28 03:06:25.790105602 (539h37m25.661188637s) [noise=true , address=45.91.116.6:4976 , piece_id=IUOJMI3HVMHFOVI6ISMJVNCYKJYH3ULWBQUZCAHIOM4RVBRUODEQ, seg_pos={0 0}, upload=473228]: Uploading piece... Dec 28 03:06:25.790748046 (539h37m25.661831091s) [noise=true , address=145.79.227.50:4373 , piece_id=IKHFUZEHJ2EBW3OSFQQQDW5GNZGLQ5I5ZV3CJUAH3XDQKHT7N3MA, seg_pos={0 0}, upload=473228]: Uploading piece... Dec 28 03:06:25.791965764 (539h37m25.663048799s) [noise=true , address=145.79.227.46:4602 , piece_id=4PPVGNTMLH4EOPATOXWJIJCIGRKEVSUQKUSTYSFAFMFXGGCRVSAQ, seg_pos={0 0}, upload=473228]: Uploading piece... Dec 28 03:06:25.792334296 (539h37m25.663417341s) [noise=true , address=45.91.116.11:4119 , piece_id=VOJGVUXON4ROJKWLNK73DJ2RAOFFVIGFHWN4DXP3TZIHTNXCJGGA, seg_pos={0 0}, upload=473228]: Uploading piece... Dec 28 03:06:25.793292421 (539h37m25.664375466s) [noise=true , address=145.79.227.43:4159 , piece_id=LLSYBUNVQF3632GGJLTQ5HLWV3OHJXOJ5SIDV4CQYYYFFNZVUBZA, seg_pos={0 0}, upload=473228]: Uploading piece... Dec 28 03:06:25.794032850 (539h37m25.665115895s) [noise=true , address=45.91.116.4:4937 , piece_id=QYLY6256QZH7I3EDPOQPXDEBF4KQVAQZMEKPKVSYCVFJI4MBLOCQ, seg_pos={0 0}, upload=473228]: Uploading piece... Dec 28 03:06:25.794677198 (539h37m25.665760243s) [noise=true , address=77.90.31.12:4589 , piece_id=FAZS5NBX6RBIN4GXPDDX7WKQGB7VSEIDN3CYK5RHIAJ45T64LFCQ, seg_pos={0 0}, upload=473228]: Uploading piece... Dec 28 03:06:25.796052135 (539h37m25.66713518s) [noise=true , address=145.79.227.42:4121 , piece_id=4VAENCAD4QYTFJB7XSCHHERHLV6OCAOMRLHQJTXSXA3ZSSCETX2Q, seg_pos={0 0}, upload=473228]: Uploading piece... Dec 28 03:06:25.797213988 (539h37m25.668297023s) [noise=true , address=145.79.227.14:4838 , piece_id=XMWMAECJHUXCZQHYCMMGODAQKGTCH2LMH5X36D26FURK5OFRJXAQ, seg_pos={0 0}, upload=473228]: Uploading piece... Dec 28 03:06:25.798132226 (539h37m25.669215271s) [noise=true , address=145.79.227.37:4567 , piece_id=UGSVC67ZBN5WCQCG4Q5X4DOTBESFJS4NUXE7MO7LK6WTVVHECXKA, seg_pos={0 0}, upload=473228]: Uploading piece... Dec 28 03:06:25.799968814 (539h37m25.671051849s) [noise=true , address=145.79.227.52:4446 , piece_id=I4ODPMLBSEVD7IGX7BNZEGRMMFXUDCU5PALXMEM2XVLBJF64RTSA, seg_pos={0 0}, upload=473228]: Uploading piece... Dec 28 03:06:27.535635102 (539h37m27.406718147s) [upload=473228]: Next returned nil segment Dec 28 03:06:27.535648488 (539h37m27.406731523s) [upload=473228]: Waiting for error group managing segments... Dec 28 03:06:27.823327777 (539h37m27.694410812s) [noise=true , address=145.79.227.5:4222 , piece_id=NZ3XWPYNSDXX77W6PVO6TWJIQ5SPXHNG5GRCWQPF2POUBXVDWAAQ, seg_pos={0 0}, upload=473228]: Done uploading piece. err: <nil> Dec 28 03:06:27.831463058 (539h37m27.702546093s) [noise=true , address=145.79.227.42:4121 , piece_id=4VAENCAD4QYTFJB7XSCHHERHLV6OCAOMRLHQJTXSXA3ZSSCETX2Q, seg_pos={0 0}, upload=473228]: Done uploading piece. err: <nil> Dec 28 03:06:27.858379592 (539h37m27.729462637s) [noise=true , address=145.79.227.52:4446 , piece_id=I4ODPMLBSEVD7IGX7BNZEGRMMFXUDCU5PALXMEM2XVLBJF64RTSA, seg_pos={0 0}, upload=473228]: Done uploading piece. err: <nil> Dec 28 03:06:27.896928034 (539h37m27.768011079s) [noise=true , address=145.79.227.14:4838 , piece_id=XMWMAECJHUXCZQHYCMMGODAQKGTCH2LMH5X36D26FURK5OFRJXAQ, seg_pos={0 0}, upload=473228]: Done uploading piece. err: <nil> Dec 28 03:06:27.907339489 (539h37m27.778422534s) [noise=true , address=145.79.227.50:4373 , piece_id=IKHFUZEHJ2EBW3OSFQQQDW5GNZGLQ5I5ZV3CJUAH3XDQKHT7N3MA, seg_pos={0 0}, upload=473228]: Done uploading piece. err: <nil> Dec 28 03:06:28.012220530 (539h37m27.883303565s) [noise=true , address=45.91.116.4:4937 , piece_id=QYLY6256QZH7I3EDPOQPXDEBF4KQVAQZMEKPKVSYCVFJI4MBLOCQ, seg_pos={0 0}, upload=473228]: Done uploading piece. err: <nil> Dec 28 03:06:28.027980019 (539h37m27.899063054s) [noise=true , address=77.90.31.12:4589 , piece_id=FAZS5NBX6RBIN4GXPDDX7WKQGB7VSEIDN3CYK5RHIAJ45T64LFCQ, seg_pos={0 0}, upload=473228]: Done uploading piece. err: <nil> Dec 28 03:06:28.152311838 (539h37m28.023394873s) [noise=true , address=45.91.116.11:4119 , piece_id=VOJGVUXON4ROJKWLNK73DJ2RAOFFVIGFHWN4DXP3TZIHTNXCJGGA, seg_pos={0 0}, upload=473228]: Done uploading piece. err: <nil> Dec 28 03:06:28.339080767 (539h37m28.210163812s) [noise=true , address=45.91.116.7:4012 , piece_id=2DCO4OLJHODU3A6B7UBW5ZWQARGPMHOASUUIVBWNBCNGZE4AVALA, seg_pos={0 0}, upload=473228]: Done uploading piece. err: <nil> Dec 28 03:06:28.787117704 (539h37m28.658200739s) [noise=true , address=45.91.116.12:4227 , piece_id=N36GRWHIBMUZ4EW2EO7DJMZAG2Y57KPNHPKJUF7VW3DJQJ2WIB2Q, seg_pos={0 0}, upload=473228]: Done uploading piece. err: <nil> Dec 28 03:06:28.924995937 (539h37m28.796078982s) [noise=true , address=145.79.227.37:4567 , piece_id=UGSVC67ZBN5WCQCG4Q5X4DOTBESFJS4NUXE7MO7LK6WTVVHECXKA, seg_pos={0 0}, upload=473228]: Done uploading piece. err: <nil> Dec 28 03:06:28.937379989 (539h37m28.808463024s) [noise=true , address=145.79.227.38:4979 , piece_id=7G4QR36QC5SAR75VSG6VFODMPYY4GQXCVME4W7NH4GPE5XTT4LGQ, seg_pos={0 0}, upload=473228]: Done uploading piece. err: <nil> Dec 28 03:06:29.067756979 (539h37m28.938840014s) [noise=true , address=45.91.116.6:4976 , piece_id=IUOJMI3HVMHFOVI6ISMJVNCYKJYH3ULWBQUZCAHIOM4RVBRUODEQ, seg_pos={0 0}, upload=473228]: Done uploading piece. err: <nil> Dec 28 03:06:29.067775363 (539h37m28.938858398s) [seg_pos={0 0}, upload=473228]: Segment reached optimal threshold of 13 pieces. Dec 28 03:06:29.067958341 (539h37m28.939041376s) [noise=true , address=145.79.227.43:4159 , piece_id=LLSYBUNVQF3632GGJLTQ5HLWV3OHJXOJ5SIDV4CQYYYFFNZVUBZA, seg_pos={0 0}, upload=473228]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Dec 28 03:06:29.067964633 (539h37m28.939047668s) [noise=true , address=145.79.227.43:4159 , piece_id=LLSYBUNVQF3632GGJLTQ5HLWV3OHJXOJ5SIDV4CQYYYFFNZVUBZA, seg_pos={0 0}, upload=473228]: Long tail cancellation due to optimal threshold Dec 28 03:06:29.068027944 (539h37m28.939110989s) [noise=true , address=145.79.227.46:4602 , piece_id=4PPVGNTMLH4EOPATOXWJIJCIGRKEVSUQKUSTYSFAFMFXGGCRVSAQ, seg_pos={0 0}, upload=473228]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kgUqvw1r3gSKw1bNjzys3AiAJXdkAnSafmvcPgNVPBbN9JS1d): protocol: expected piece hash; context canceled; EOF Dec 28 03:06:29.068038815 (539h37m28.93912185s) [noise=true , address=145.79.227.46:4602 , piece_id=4PPVGNTMLH4EOPATOXWJIJCIGRKEVSUQKUSTYSFAFMFXGGCRVSAQ, seg_pos={0 0}, upload=473228]: Long tail cancellation due to optimal threshold Dec 28 03:06:29.068057480 (539h37m28.939140525s) [noise=true , address=145.79.227.30:4761 , piece_id=6NKI7A6EEK7NIPILA6OS5PCOJYSU2OJKX6KYSUV2HJGN4BINA4GA, seg_pos={0 0}, upload=473228]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 28 03:06:29.068063552 (539h37m28.939146587s) [noise=true , address=145.79.227.30:4761 , piece_id=6NKI7A6EEK7NIPILA6OS5PCOJYSU2OJKX6KYSUV2HJGN4BINA4GA, seg_pos={0 0}, upload=473228]: Long tail cancellation due to optimal threshold Dec 28 03:06:29.068080845 (539h37m28.93916389s) [seg_pos={0 0}, upload=473228]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 03:06:29.068161328 (539h37m28.939244363s) [upload=473228]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 03:06:29.068167790 (539h37m28.939250825s) [upload=473228]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 03:06:29.360691865 (539h37m29.2317749s) [upload=473228]: Done uploading segments... Dec 28 03:06:29.360709899 (539h37m29.231792934s) [upload=473228]: Upload finished. err: <nil> Duration: 8.03006523s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...X.....X..................................................] duration: 33.17170089s trace: https://jaeger.dev.storj.io/trace/1822be52d200e5b5 FAIL (2025-12-28 03:07:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1ZAfYZ4wgpZVvatj6wtHona3hTaPcLxAmXC1DT2uKQyo8nLQQ7, Piece ID: 5MMQ44MXRT4XEL4X6WO54WRG2XYMX3AN3FNQ2H2XIUTKBXI3EEPA) context deadline exceeded; piecestore: (Node ID: 124guAM82oibYZsMho67Em6Ke6yEZXwT6LPMLLSZZBLqwoFGUkn, Piece ID: 22WJIPX4SM5NTIAW5ZRM2UKX32CTDBFCXPCCHRRWRCUCOF5CRE5Q) context deadline exceeded; piecestore: (Node ID: 1e6p1TD5CqUyWdZjkrxh2kRqMuyDiMUr9eimfeayDAKwhtBrZ8, Piece ID: XRI5EJFN7YYUG4QKLDGEFT6YAUHCUTTSXF735UDR6LZZBGMAZDJQ) context deadline exceeded ======LOGS====== 5.234848204 (538h55m35.105931239s) [noise=true , address=77.90.31.11:4572 , piece_id=5MMQ44MXRT4XEL4X6WO54WRG2XYMX3AN3FNQ2H2XIUTKBXI3EEPA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.235412057 (538h55m35.106495102s) [noise=true , address=145.79.227.22:4512 , piece_id=D46YLTIKGHG36DQ4YBKBSUDV4GGB4WVDWNOJS27IYYLN6AYZ5QFQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.235820946 (538h55m35.106903981s) [noise=true , address=77.90.31.19:4874 , piece_id=RHZUXUYMJU6RKQGSDZM5WDWIN5SQ7YIRSRATNNKT5GEMU2QH444A, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.236192192 (538h55m35.107275237s) [noise=true , address=77.90.31.16:4760 , piece_id=XRI5EJFN7YYUG4QKLDGEFT6YAUHCUTTSXF735UDR6LZZBGMAZDJQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.236564772 (538h55m35.107647817s) [noise=true , address=145.79.227.40:4062 , piece_id=PW6QWIPQWVUETLC6DJ624G6IB4P6EM4SYB72NAMQ4V6NUKNLUYLQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.236929766 (538h55m35.108012812s) [noise=true , address=64.49.13.139:4480 , piece_id=JAJY3F7IDFFMFKCK2QJV6W74RIESLRKBLXMN27Q3C45JWEPLBZLQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.237916846 (538h55m35.108999881s) [noise=true , address=145.79.227.34:4932 , piece_id=UPXSCQ45QDGPKNJXRFJ33EAA3QENVHRVQARU5TFJTEWICZNR253Q, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.238852948 (538h55m35.109935983s) [noise=true , address=45.91.116.12:4257 , piece_id=ZI67LIAHZW4G357GWKDPTB5SMXYGNXZTTTDMNAETDS2B6SWVDAEQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239152910 (538h55m35.110235945s) [noise=true , address=145.79.227.44:4222 , piece_id=CZ2CS62CTHQFAIE3S3JO2N3VJ4XIR5YJ2LMRVLLX7I4VM2VZVQVA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239409969 (538h55m35.110493004s) [noise=true , address=145.79.227.30:4773 , piece_id=KUGJXZCNMUE7WSK3VLY4AUBSOT4H6IY4XB4NCBGAGCNUG7F7IODQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239681136 (538h55m35.110764171s) [noise=true , address=77.90.31.13:4635 , piece_id=IIOY5X2W3XW4C2OFEBRCABMYFBI24RRTYNUSKN6KJU7CTUW4KONQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239954676 (538h55m35.111037711s) [noise=true , address=45.91.116.9:4281 , piece_id=JZLDW777CBX2BP7MLOQV4NFP3H4XJVCNQXTHNSUWPVDOJLPZK36A, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.240189914 (538h55m35.111272949s) [noise=true , address=145.79.227.35:4512 , piece_id=3E3VCCC45XVT5FHFDKXSTGGAGYBEML3BKU6JJTT33LCD2YMOMBZQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.240449147 (538h55m35.111532182s) [noise=true , address=45.91.116.8:4064 , piece_id=22WJIPX4SM5NTIAW5ZRM2UKX32CTDBFCXPCCHRRWRCUCOF5CRE5Q, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.240639640 (538h55m35.111722675s) [noise=true , address=145.79.227.32:4853 , piece_id=MBNCFDBQ4SZZXV4F7J5TRMOUYUQ5IPKQCGYGGM5SOIFJ6ZYCAUYA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.241137798 (538h55m35.112220833s) [noise=true , address=77.90.31.16:4751 , piece_id=N3Z2LLMCIA2F42VXZA7IDRU4XEKGOICH6TRVERODU76MFFE4QSHA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.241383797 (538h55m35.112466842s) [noise=true , address=77.90.31.18:4836 , piece_id=BSVEFI52EKN44CMINFUSZBNOH53IW5TXLVTF5IH6ZUTNII6RRLVA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:36.592880816 (538h55m36.463963871s) [upload=472631]: Next returned nil segment Dec 28 02:24:36.592901625 (538h55m36.46398467s) [upload=472631]: Waiting for error group managing segments... Dec 28 02:24:36.946933741 (538h55m36.818016786s) [noise=true , address=64.49.13.139:4480 , piece_id=JAJY3F7IDFFMFKCK2QJV6W74RIESLRKBLXMN27Q3C45JWEPLBZLQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.128189692 (538h55m36.999272737s) [noise=true , address=145.79.227.35:4512 , piece_id=3E3VCCC45XVT5FHFDKXSTGGAGYBEML3BKU6JJTT33LCD2YMOMBZQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.160760084 (538h55m37.031843119s) [noise=true , address=145.79.227.34:4932 , piece_id=UPXSCQ45QDGPKNJXRFJ33EAA3QENVHRVQARU5TFJTEWICZNR253Q, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.288838207 (538h55m37.159921252s) [noise=true , address=77.90.31.11:4572 , piece_id=5MMQ44MXRT4XEL4X6WO54WRG2XYMX3AN3FNQ2H2XIUTKBXI3EEPA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.319996310 (538h55m37.191079345s) [noise=true , address=145.79.227.22:4512 , piece_id=D46YLTIKGHG36DQ4YBKBSUDV4GGB4WVDWNOJS27IYYLN6AYZ5QFQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.320513273 (538h55m37.191596319s) [noise=true , address=77.90.31.18:4836 , piece_id=BSVEFI52EKN44CMINFUSZBNOH53IW5TXLVTF5IH6ZUTNII6RRLVA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.336522926 (538h55m37.207605971s) [noise=true , address=145.79.227.32:4853 , piece_id=MBNCFDBQ4SZZXV4F7J5TRMOUYUQ5IPKQCGYGGM5SOIFJ6ZYCAUYA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.384868031 (538h55m37.255951066s) [noise=true , address=77.90.31.16:4751 , piece_id=N3Z2LLMCIA2F42VXZA7IDRU4XEKGOICH6TRVERODU76MFFE4QSHA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.453563554 (538h55m37.324646599s) [noise=true , address=45.91.116.8:4064 , piece_id=22WJIPX4SM5NTIAW5ZRM2UKX32CTDBFCXPCCHRRWRCUCOF5CRE5Q, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.479454839 (538h55m37.350537884s) [noise=true , address=45.91.116.12:4257 , piece_id=ZI67LIAHZW4G357GWKDPTB5SMXYGNXZTTTDMNAETDS2B6SWVDAEQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.659710174 (538h55m37.530793219s) [noise=true , address=77.90.31.13:4635 , piece_id=IIOY5X2W3XW4C2OFEBRCABMYFBI24RRTYNUSKN6KJU7CTUW4KONQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.715039128 (538h55m37.586122163s) [noise=true , address=77.90.31.16:4760 , piece_id=XRI5EJFN7YYUG4QKLDGEFT6YAUHCUTTSXF735UDR6LZZBGMAZDJQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.977473406 (538h55m37.848556441s) [noise=true , address=77.90.31.19:4874 , piece_id=RHZUXUYMJU6RKQGSDZM5WDWIN5SQ7YIRSRATNNKT5GEMU2QH444A, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:38.016302745 (538h55m37.88738578s) [noise=true , address=145.79.227.30:4773 , piece_id=KUGJXZCNMUE7WSK3VLY4AUBSOT4H6IY4XB4NCBGAGCNUG7F7IODQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:38.016326982 (538h55m37.887410027s) [seg_pos={0 0}, upload=472631]: Segment reached optimal threshold of 14 pieces. Dec 28 02:24:38.016585353 (538h55m37.887668398s) [noise=true , address=145.79.227.30:4766 , piece_id=67M2DMPAVWIQKQSWUAG73D4E4NRRSG4XEVPOUVIJPEQOUV7A6VVA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1k6zdamuKPkBdzyNZNWx4FCmaTkv33PchcwvTRR8bgcETBs4K7): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016594450 (538h55m37.887677495s) [noise=true , address=145.79.227.30:4766 , piece_id=67M2DMPAVWIQKQSWUAG73D4E4NRRSG4XEVPOUVIJPEQOUV7A6VVA, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016627093 (538h55m37.887710148s) [noise=true , address=45.91.116.9:4281 , piece_id=JZLDW777CBX2BP7MLOQV4NFP3H4XJVCNQXTHNSUWPVDOJLPZK36A, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gxf7DpJZthyR6f1PExX56tJavjhPw9Gxb4h8mtmeYKW3t3tS1): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016635258 (538h55m37.887718293s) [noise=true , address=45.91.116.9:4281 , piece_id=JZLDW777CBX2BP7MLOQV4NFP3H4XJVCNQXTHNSUWPVDOJLPZK36A, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016871298 (538h55m37.887954363s) [noise=true , address=145.79.227.44:4222 , piece_id=CZ2CS62CTHQFAIE3S3JO2N3VJ4XIR5YJ2LMRVLLX7I4VM2VZVQVA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZKW35KvrsoYRefbYeUFZo8bh4XZYycNKGEvTFjzibezjXgj3e): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016880124 (538h55m37.887963169s) [noise=true , address=145.79.227.44:4222 , piece_id=CZ2CS62CTHQFAIE3S3JO2N3VJ4XIR5YJ2LMRVLLX7I4VM2VZVQVA, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016934839 (538h55m37.888017884s) [noise=true , address=145.79.227.40:4062 , piece_id=PW6QWIPQWVUETLC6DJ624G6IB4P6EM4SYB72NAMQ4V6NUKNLUYLQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016943425 (538h55m37.88802649s) [noise=true , address=145.79.227.40:4062 , piece_id=PW6QWIPQWVUETLC6DJ624G6IB4P6EM4SYB72NAMQ4V6NUKNLUYLQ, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016968252 (538h55m37.888051297s) [seg_pos={0 0}, upload=472631]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 02:24:38.017067822 (538h55m37.888150867s) [upload=472631]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 02:24:38.017074244 (538h55m37.888157289s) [upload=472631]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 02:24:38.304306667 (538h55m38.175389712s) [upload=472631]: Done uploading segments... Dec 28 02:24:38.304329641 (538h55m38.175412686s) [upload=472631]: Upload finished. err: <nil> Duration: 33.171872315s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 46 out of 60 is failed (threshold=3) [....XXX.XXXXXX.XXXX.XXXX..X..XXXXX.XXXXXXXXXXXXX.XXXXXXXXX.X] duration: 30.00274983s trace: https://jaeger.dev.storj.io/trace/43402463fe7eab1a FAIL (2025-12-28 03:06:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 03:06:25.539781194 (539h37m25.410864239s) [upload=473228]: Starting upload Dec 28 03:06:25.539870084 (539h37m25.410953139s) [upload=473228]: Done starting upload Dec 28 03:06:25.539945457 (539h37m25.411028522s) [upload=473228]: Uploading segments... Dec 28 03:06:25.540158973 (539h37m25.411242019s) [upload=473228]: Got next segment. Inline: false Dec 28 03:06:25.540204249 (539h37m25.411287304s) [upload=473228]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 03:06:25.540210432 (539h37m25.411293477s) [upload=473228]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 03:06:25.787931021 (539h37m25.659014067s) [seg_pos={0 0}, upload=473228]: Begin upload segment... Dec 28 03:06:25.787978081 (539h37m25.659061116s) [seg_pos={0 0}, upload=473228]: Done begin upload segment. Dec 28 03:06:25.788004822 (539h37m25.659087857s) [noise=true , address=45.91.116.12:4227 , piece_id=N36GRWHIBMUZ4EW2EO7DJMZAG2Y57KPNHPKJUF7VW3DJQJ2WIB2Q, seg_pos={0 0}, upload=473228]: Uploading piece... Dec 28 03:06:25.788110774 (539h37m25.659193809s) [noise=true , address=145.79.227.38:4979 , piece_id=7G4QR36QC5SAR75VSG6VFODMPYY4GQXCVME4W7NH4GPE5XTT4LGQ, seg_pos={0 0}, upload=473228]: Uploading piece... Dec 28 03:06:25.788688915 (539h37m25.65977196s) [noise=true , address=45.91.116.7:4012 , piece_id=2DCO4OLJHODU3A6B7UBW5ZWQARGPMHOASUUIVBWNBCNGZE4AVALA, seg_pos={0 0}, upload=473228]: Uploading piece... Dec 28 03:06:25.789108784 (539h37m25.660191819s) [noise=true , address=145.79.227.30:4761 , piece_id=6NKI7A6EEK7NIPILA6OS5PCOJYSU2OJKX6KYSUV2HJGN4BINA4GA, seg_pos={0 0}, upload=473228]: Uploading piece... Dec 28 03:06:25.789616070 (539h37m25.660699115s) [noise=true , address=145.79.227.5:4222 , piece_id=NZ3XWPYNSDXX77W6PVO6TWJIQ5SPXHNG5GRCWQPF2POUBXVDWAAQ, seg_pos={0 0}, upload=473228]: Uploading piece... Dec 28 03:06:25.790105602 (539h37m25.661188637s) [noise=true , address=45.91.116.6:4976 , piece_id=IUOJMI3HVMHFOVI6ISMJVNCYKJYH3ULWBQUZCAHIOM4RVBRUODEQ, seg_pos={0 0}, upload=473228]: Uploading piece... Dec 28 03:06:25.790748046 (539h37m25.661831091s) [noise=true , address=145.79.227.50:4373 , piece_id=IKHFUZEHJ2EBW3OSFQQQDW5GNZGLQ5I5ZV3CJUAH3XDQKHT7N3MA, seg_pos={0 0}, upload=473228]: Uploading piece... Dec 28 03:06:25.791965764 (539h37m25.663048799s) [noise=true , address=145.79.227.46:4602 , piece_id=4PPVGNTMLH4EOPATOXWJIJCIGRKEVSUQKUSTYSFAFMFXGGCRVSAQ, seg_pos={0 0}, upload=473228]: Uploading piece... Dec 28 03:06:25.792334296 (539h37m25.663417341s) [noise=true , address=45.91.116.11:4119 , piece_id=VOJGVUXON4ROJKWLNK73DJ2RAOFFVIGFHWN4DXP3TZIHTNXCJGGA, seg_pos={0 0}, upload=473228]: Uploading piece... Dec 28 03:06:25.793292421 (539h37m25.664375466s) [noise=true , address=145.79.227.43:4159 , piece_id=LLSYBUNVQF3632GGJLTQ5HLWV3OHJXOJ5SIDV4CQYYYFFNZVUBZA, seg_pos={0 0}, upload=473228]: Uploading piece... Dec 28 03:06:25.794032850 (539h37m25.665115895s) [noise=true , address=45.91.116.4:4937 , piece_id=QYLY6256QZH7I3EDPOQPXDEBF4KQVAQZMEKPKVSYCVFJI4MBLOCQ, seg_pos={0 0}, upload=473228]: Uploading piece... Dec 28 03:06:25.794677198 (539h37m25.665760243s) [noise=true , address=77.90.31.12:4589 , piece_id=FAZS5NBX6RBIN4GXPDDX7WKQGB7VSEIDN3CYK5RHIAJ45T64LFCQ, seg_pos={0 0}, upload=473228]: Uploading piece... Dec 28 03:06:25.796052135 (539h37m25.66713518s) [noise=true , address=145.79.227.42:4121 , piece_id=4VAENCAD4QYTFJB7XSCHHERHLV6OCAOMRLHQJTXSXA3ZSSCETX2Q, seg_pos={0 0}, upload=473228]: Uploading piece... Dec 28 03:06:25.797213988 (539h37m25.668297023s) [noise=true , address=145.79.227.14:4838 , piece_id=XMWMAECJHUXCZQHYCMMGODAQKGTCH2LMH5X36D26FURK5OFRJXAQ, seg_pos={0 0}, upload=473228]: Uploading piece... Dec 28 03:06:25.798132226 (539h37m25.669215271s) [noise=true , address=145.79.227.37:4567 , piece_id=UGSVC67ZBN5WCQCG4Q5X4DOTBESFJS4NUXE7MO7LK6WTVVHECXKA, seg_pos={0 0}, upload=473228]: Uploading piece... Dec 28 03:06:25.799968814 (539h37m25.671051849s) [noise=true , address=145.79.227.52:4446 , piece_id=I4ODPMLBSEVD7IGX7BNZEGRMMFXUDCU5PALXMEM2XVLBJF64RTSA, seg_pos={0 0}, upload=473228]: Uploading piece... Dec 28 03:06:27.535635102 (539h37m27.406718147s) [upload=473228]: Next returned nil segment Dec 28 03:06:27.535648488 (539h37m27.406731523s) [upload=473228]: Waiting for error group managing segments... Dec 28 03:06:27.823327777 (539h37m27.694410812s) [noise=true , address=145.79.227.5:4222 , piece_id=NZ3XWPYNSDXX77W6PVO6TWJIQ5SPXHNG5GRCWQPF2POUBXVDWAAQ, seg_pos={0 0}, upload=473228]: Done uploading piece. err: <nil> Dec 28 03:06:27.831463058 (539h37m27.702546093s) [noise=true , address=145.79.227.42:4121 , piece_id=4VAENCAD4QYTFJB7XSCHHERHLV6OCAOMRLHQJTXSXA3ZSSCETX2Q, seg_pos={0 0}, upload=473228]: Done uploading piece. err: <nil> Dec 28 03:06:27.858379592 (539h37m27.729462637s) [noise=true , address=145.79.227.52:4446 , piece_id=I4ODPMLBSEVD7IGX7BNZEGRMMFXUDCU5PALXMEM2XVLBJF64RTSA, seg_pos={0 0}, upload=473228]: Done uploading piece. err: <nil> Dec 28 03:06:27.896928034 (539h37m27.768011079s) [noise=true , address=145.79.227.14:4838 , piece_id=XMWMAECJHUXCZQHYCMMGODAQKGTCH2LMH5X36D26FURK5OFRJXAQ, seg_pos={0 0}, upload=473228]: Done uploading piece. err: <nil> Dec 28 03:06:27.907339489 (539h37m27.778422534s) [noise=true , address=145.79.227.50:4373 , piece_id=IKHFUZEHJ2EBW3OSFQQQDW5GNZGLQ5I5ZV3CJUAH3XDQKHT7N3MA, seg_pos={0 0}, upload=473228]: Done uploading piece. err: <nil> Dec 28 03:06:28.012220530 (539h37m27.883303565s) [noise=true , address=45.91.116.4:4937 , piece_id=QYLY6256QZH7I3EDPOQPXDEBF4KQVAQZMEKPKVSYCVFJI4MBLOCQ, seg_pos={0 0}, upload=473228]: Done uploading piece. err: <nil> Dec 28 03:06:28.027980019 (539h37m27.899063054s) [noise=true , address=77.90.31.12:4589 , piece_id=FAZS5NBX6RBIN4GXPDDX7WKQGB7VSEIDN3CYK5RHIAJ45T64LFCQ, seg_pos={0 0}, upload=473228]: Done uploading piece. err: <nil> Dec 28 03:06:28.152311838 (539h37m28.023394873s) [noise=true , address=45.91.116.11:4119 , piece_id=VOJGVUXON4ROJKWLNK73DJ2RAOFFVIGFHWN4DXP3TZIHTNXCJGGA, seg_pos={0 0}, upload=473228]: Done uploading piece. err: <nil> Dec 28 03:06:28.339080767 (539h37m28.210163812s) [noise=true , address=45.91.116.7:4012 , piece_id=2DCO4OLJHODU3A6B7UBW5ZWQARGPMHOASUUIVBWNBCNGZE4AVALA, seg_pos={0 0}, upload=473228]: Done uploading piece. err: <nil> Dec 28 03:06:28.787117704 (539h37m28.658200739s) [noise=true , address=45.91.116.12:4227 , piece_id=N36GRWHIBMUZ4EW2EO7DJMZAG2Y57KPNHPKJUF7VW3DJQJ2WIB2Q, seg_pos={0 0}, upload=473228]: Done uploading piece. err: <nil> Dec 28 03:06:28.924995937 (539h37m28.796078982s) [noise=true , address=145.79.227.37:4567 , piece_id=UGSVC67ZBN5WCQCG4Q5X4DOTBESFJS4NUXE7MO7LK6WTVVHECXKA, seg_pos={0 0}, upload=473228]: Done uploading piece. err: <nil> Dec 28 03:06:28.937379989 (539h37m28.808463024s) [noise=true , address=145.79.227.38:4979 , piece_id=7G4QR36QC5SAR75VSG6VFODMPYY4GQXCVME4W7NH4GPE5XTT4LGQ, seg_pos={0 0}, upload=473228]: Done uploading piece. err: <nil> Dec 28 03:06:29.067756979 (539h37m28.938840014s) [noise=true , address=45.91.116.6:4976 , piece_id=IUOJMI3HVMHFOVI6ISMJVNCYKJYH3ULWBQUZCAHIOM4RVBRUODEQ, seg_pos={0 0}, upload=473228]: Done uploading piece. err: <nil> Dec 28 03:06:29.067775363 (539h37m28.938858398s) [seg_pos={0 0}, upload=473228]: Segment reached optimal threshold of 13 pieces. Dec 28 03:06:29.067958341 (539h37m28.939041376s) [noise=true , address=145.79.227.43:4159 , piece_id=LLSYBUNVQF3632GGJLTQ5HLWV3OHJXOJ5SIDV4CQYYYFFNZVUBZA, seg_pos={0 0}, upload=473228]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Dec 28 03:06:29.067964633 (539h37m28.939047668s) [noise=true , address=145.79.227.43:4159 , piece_id=LLSYBUNVQF3632GGJLTQ5HLWV3OHJXOJ5SIDV4CQYYYFFNZVUBZA, seg_pos={0 0}, upload=473228]: Long tail cancellation due to optimal threshold Dec 28 03:06:29.068027944 (539h37m28.939110989s) [noise=true , address=145.79.227.46:4602 , piece_id=4PPVGNTMLH4EOPATOXWJIJCIGRKEVSUQKUSTYSFAFMFXGGCRVSAQ, seg_pos={0 0}, upload=473228]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kgUqvw1r3gSKw1bNjzys3AiAJXdkAnSafmvcPgNVPBbN9JS1d): protocol: expected piece hash; context canceled; EOF Dec 28 03:06:29.068038815 (539h37m28.93912185s) [noise=true , address=145.79.227.46:4602 , piece_id=4PPVGNTMLH4EOPATOXWJIJCIGRKEVSUQKUSTYSFAFMFXGGCRVSAQ, seg_pos={0 0}, upload=473228]: Long tail cancellation due to optimal threshold Dec 28 03:06:29.068057480 (539h37m28.939140525s) [noise=true , address=145.79.227.30:4761 , piece_id=6NKI7A6EEK7NIPILA6OS5PCOJYSU2OJKX6KYSUV2HJGN4BINA4GA, seg_pos={0 0}, upload=473228]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 28 03:06:29.068063552 (539h37m28.939146587s) [noise=true , address=145.79.227.30:4761 , piece_id=6NKI7A6EEK7NIPILA6OS5PCOJYSU2OJKX6KYSUV2HJGN4BINA4GA, seg_pos={0 0}, upload=473228]: Long tail cancellation due to optimal threshold Dec 28 03:06:29.068080845 (539h37m28.93916389s) [seg_pos={0 0}, upload=473228]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 03:06:29.068161328 (539h37m28.939244363s) [upload=473228]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 03:06:29.068167790 (539h37m28.939250825s) [upload=473228]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 03:06:29.360691865 (539h37m29.2317749s) [upload=473228]: Done uploading segments... Dec 28 03:06:29.360709899 (539h37m29.231792934s) [upload=473228]: Upload finished. err: <nil> Duration: 30.002775939s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [...X................X....X............X........X....X.......] duration: 4.886413725s trace: https://jaeger.dev.storj.io/trace/25c8812909e078aa FAIL (2025-12-28 03:06:30 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.88648945s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [....X.....X.................................................] duration: 35.227022381s trace: https://jaeger.dev.storj.io/trace/1822be52d200e5b5 FAIL (2025-12-28 03:06:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1ZAfYZ4wgpZVvatj6wtHona3hTaPcLxAmXC1DT2uKQyo8nLQQ7, Piece ID: 5MMQ44MXRT4XEL4X6WO54WRG2XYMX3AN3FNQ2H2XIUTKBXI3EEPA) context deadline exceeded; piecestore: (Node ID: 124guAM82oibYZsMho67Em6Ke6yEZXwT6LPMLLSZZBLqwoFGUkn, Piece ID: 22WJIPX4SM5NTIAW5ZRM2UKX32CTDBFCXPCCHRRWRCUCOF5CRE5Q) context deadline exceeded; piecestore: (Node ID: 1e6p1TD5CqUyWdZjkrxh2kRqMuyDiMUr9eimfeayDAKwhtBrZ8, Piece ID: XRI5EJFN7YYUG4QKLDGEFT6YAUHCUTTSXF735UDR6LZZBGMAZDJQ) context deadline exceeded ======LOGS====== 5.234848204 (538h55m35.105931239s) [noise=true , address=77.90.31.11:4572 , piece_id=5MMQ44MXRT4XEL4X6WO54WRG2XYMX3AN3FNQ2H2XIUTKBXI3EEPA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.235412057 (538h55m35.106495102s) [noise=true , address=145.79.227.22:4512 , piece_id=D46YLTIKGHG36DQ4YBKBSUDV4GGB4WVDWNOJS27IYYLN6AYZ5QFQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.235820946 (538h55m35.106903981s) [noise=true , address=77.90.31.19:4874 , piece_id=RHZUXUYMJU6RKQGSDZM5WDWIN5SQ7YIRSRATNNKT5GEMU2QH444A, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.236192192 (538h55m35.107275237s) [noise=true , address=77.90.31.16:4760 , piece_id=XRI5EJFN7YYUG4QKLDGEFT6YAUHCUTTSXF735UDR6LZZBGMAZDJQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.236564772 (538h55m35.107647817s) [noise=true , address=145.79.227.40:4062 , piece_id=PW6QWIPQWVUETLC6DJ624G6IB4P6EM4SYB72NAMQ4V6NUKNLUYLQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.236929766 (538h55m35.108012812s) [noise=true , address=64.49.13.139:4480 , piece_id=JAJY3F7IDFFMFKCK2QJV6W74RIESLRKBLXMN27Q3C45JWEPLBZLQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.237916846 (538h55m35.108999881s) [noise=true , address=145.79.227.34:4932 , piece_id=UPXSCQ45QDGPKNJXRFJ33EAA3QENVHRVQARU5TFJTEWICZNR253Q, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.238852948 (538h55m35.109935983s) [noise=true , address=45.91.116.12:4257 , piece_id=ZI67LIAHZW4G357GWKDPTB5SMXYGNXZTTTDMNAETDS2B6SWVDAEQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239152910 (538h55m35.110235945s) [noise=true , address=145.79.227.44:4222 , piece_id=CZ2CS62CTHQFAIE3S3JO2N3VJ4XIR5YJ2LMRVLLX7I4VM2VZVQVA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239409969 (538h55m35.110493004s) [noise=true , address=145.79.227.30:4773 , piece_id=KUGJXZCNMUE7WSK3VLY4AUBSOT4H6IY4XB4NCBGAGCNUG7F7IODQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239681136 (538h55m35.110764171s) [noise=true , address=77.90.31.13:4635 , piece_id=IIOY5X2W3XW4C2OFEBRCABMYFBI24RRTYNUSKN6KJU7CTUW4KONQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239954676 (538h55m35.111037711s) [noise=true , address=45.91.116.9:4281 , piece_id=JZLDW777CBX2BP7MLOQV4NFP3H4XJVCNQXTHNSUWPVDOJLPZK36A, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.240189914 (538h55m35.111272949s) [noise=true , address=145.79.227.35:4512 , piece_id=3E3VCCC45XVT5FHFDKXSTGGAGYBEML3BKU6JJTT33LCD2YMOMBZQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.240449147 (538h55m35.111532182s) [noise=true , address=45.91.116.8:4064 , piece_id=22WJIPX4SM5NTIAW5ZRM2UKX32CTDBFCXPCCHRRWRCUCOF5CRE5Q, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.240639640 (538h55m35.111722675s) [noise=true , address=145.79.227.32:4853 , piece_id=MBNCFDBQ4SZZXV4F7J5TRMOUYUQ5IPKQCGYGGM5SOIFJ6ZYCAUYA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.241137798 (538h55m35.112220833s) [noise=true , address=77.90.31.16:4751 , piece_id=N3Z2LLMCIA2F42VXZA7IDRU4XEKGOICH6TRVERODU76MFFE4QSHA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.241383797 (538h55m35.112466842s) [noise=true , address=77.90.31.18:4836 , piece_id=BSVEFI52EKN44CMINFUSZBNOH53IW5TXLVTF5IH6ZUTNII6RRLVA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:36.592880816 (538h55m36.463963871s) [upload=472631]: Next returned nil segment Dec 28 02:24:36.592901625 (538h55m36.46398467s) [upload=472631]: Waiting for error group managing segments... Dec 28 02:24:36.946933741 (538h55m36.818016786s) [noise=true , address=64.49.13.139:4480 , piece_id=JAJY3F7IDFFMFKCK2QJV6W74RIESLRKBLXMN27Q3C45JWEPLBZLQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.128189692 (538h55m36.999272737s) [noise=true , address=145.79.227.35:4512 , piece_id=3E3VCCC45XVT5FHFDKXSTGGAGYBEML3BKU6JJTT33LCD2YMOMBZQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.160760084 (538h55m37.031843119s) [noise=true , address=145.79.227.34:4932 , piece_id=UPXSCQ45QDGPKNJXRFJ33EAA3QENVHRVQARU5TFJTEWICZNR253Q, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.288838207 (538h55m37.159921252s) [noise=true , address=77.90.31.11:4572 , piece_id=5MMQ44MXRT4XEL4X6WO54WRG2XYMX3AN3FNQ2H2XIUTKBXI3EEPA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.319996310 (538h55m37.191079345s) [noise=true , address=145.79.227.22:4512 , piece_id=D46YLTIKGHG36DQ4YBKBSUDV4GGB4WVDWNOJS27IYYLN6AYZ5QFQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.320513273 (538h55m37.191596319s) [noise=true , address=77.90.31.18:4836 , piece_id=BSVEFI52EKN44CMINFUSZBNOH53IW5TXLVTF5IH6ZUTNII6RRLVA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.336522926 (538h55m37.207605971s) [noise=true , address=145.79.227.32:4853 , piece_id=MBNCFDBQ4SZZXV4F7J5TRMOUYUQ5IPKQCGYGGM5SOIFJ6ZYCAUYA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.384868031 (538h55m37.255951066s) [noise=true , address=77.90.31.16:4751 , piece_id=N3Z2LLMCIA2F42VXZA7IDRU4XEKGOICH6TRVERODU76MFFE4QSHA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.453563554 (538h55m37.324646599s) [noise=true , address=45.91.116.8:4064 , piece_id=22WJIPX4SM5NTIAW5ZRM2UKX32CTDBFCXPCCHRRWRCUCOF5CRE5Q, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.479454839 (538h55m37.350537884s) [noise=true , address=45.91.116.12:4257 , piece_id=ZI67LIAHZW4G357GWKDPTB5SMXYGNXZTTTDMNAETDS2B6SWVDAEQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.659710174 (538h55m37.530793219s) [noise=true , address=77.90.31.13:4635 , piece_id=IIOY5X2W3XW4C2OFEBRCABMYFBI24RRTYNUSKN6KJU7CTUW4KONQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.715039128 (538h55m37.586122163s) [noise=true , address=77.90.31.16:4760 , piece_id=XRI5EJFN7YYUG4QKLDGEFT6YAUHCUTTSXF735UDR6LZZBGMAZDJQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.977473406 (538h55m37.848556441s) [noise=true , address=77.90.31.19:4874 , piece_id=RHZUXUYMJU6RKQGSDZM5WDWIN5SQ7YIRSRATNNKT5GEMU2QH444A, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:38.016302745 (538h55m37.88738578s) [noise=true , address=145.79.227.30:4773 , piece_id=KUGJXZCNMUE7WSK3VLY4AUBSOT4H6IY4XB4NCBGAGCNUG7F7IODQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:38.016326982 (538h55m37.887410027s) [seg_pos={0 0}, upload=472631]: Segment reached optimal threshold of 14 pieces. Dec 28 02:24:38.016585353 (538h55m37.887668398s) [noise=true , address=145.79.227.30:4766 , piece_id=67M2DMPAVWIQKQSWUAG73D4E4NRRSG4XEVPOUVIJPEQOUV7A6VVA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1k6zdamuKPkBdzyNZNWx4FCmaTkv33PchcwvTRR8bgcETBs4K7): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016594450 (538h55m37.887677495s) [noise=true , address=145.79.227.30:4766 , piece_id=67M2DMPAVWIQKQSWUAG73D4E4NRRSG4XEVPOUVIJPEQOUV7A6VVA, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016627093 (538h55m37.887710148s) [noise=true , address=45.91.116.9:4281 , piece_id=JZLDW777CBX2BP7MLOQV4NFP3H4XJVCNQXTHNSUWPVDOJLPZK36A, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gxf7DpJZthyR6f1PExX56tJavjhPw9Gxb4h8mtmeYKW3t3tS1): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016635258 (538h55m37.887718293s) [noise=true , address=45.91.116.9:4281 , piece_id=JZLDW777CBX2BP7MLOQV4NFP3H4XJVCNQXTHNSUWPVDOJLPZK36A, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016871298 (538h55m37.887954363s) [noise=true , address=145.79.227.44:4222 , piece_id=CZ2CS62CTHQFAIE3S3JO2N3VJ4XIR5YJ2LMRVLLX7I4VM2VZVQVA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZKW35KvrsoYRefbYeUFZo8bh4XZYycNKGEvTFjzibezjXgj3e): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016880124 (538h55m37.887963169s) [noise=true , address=145.79.227.44:4222 , piece_id=CZ2CS62CTHQFAIE3S3JO2N3VJ4XIR5YJ2LMRVLLX7I4VM2VZVQVA, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016934839 (538h55m37.888017884s) [noise=true , address=145.79.227.40:4062 , piece_id=PW6QWIPQWVUETLC6DJ624G6IB4P6EM4SYB72NAMQ4V6NUKNLUYLQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016943425 (538h55m37.88802649s) [noise=true , address=145.79.227.40:4062 , piece_id=PW6QWIPQWVUETLC6DJ624G6IB4P6EM4SYB72NAMQ4V6NUKNLUYLQ, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016968252 (538h55m37.888051297s) [seg_pos={0 0}, upload=472631]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 02:24:38.017067822 (538h55m37.888150867s) [upload=472631]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 02:24:38.017074244 (538h55m37.888157289s) [upload=472631]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 02:24:38.304306667 (538h55m38.175389712s) [upload=472631]: Done uploading segments... Dec 28 02:24:38.304329641 (538h55m38.175412686s) [upload=472631]: Upload finished. err: <nil> Duration: 35.227049212s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [....X................X....X............X........X....X......] duration: 3.522623341s trace: https://jaeger.dev.storj.io/trace/25c8812909e078aa FAIL (2025-12-28 03:06:02 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.522647477s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 45 out of 60 is failed (threshold=3) [.....XXX.XXXXXX.XXXX.XXXX..X..XXXXX.XXXXXXXXXXXXX.XXXXXXXXX.] duration: 14.867080865s trace: https://jaeger.dev.storj.io/trace/761c0fba24d52cf0 FAIL (2025-12-28 03:05:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 03:04:25.257955017 (539h35m25.129038062s) [upload=473201]: Starting upload Dec 28 03:04:25.258010052 (539h35m25.129093087s) [upload=473201]: Done starting upload Dec 28 03:04:25.258203399 (539h35m25.129286435s) [upload=473201]: Uploading segments... Dec 28 03:04:25.258235050 (539h35m25.129318095s) [upload=473201]: Got next segment. Inline: false Dec 28 03:04:25.258268544 (539h35m25.129351579s) [upload=473201]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 03:04:25.258273042 (539h35m25.129356087s) [upload=473201]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 03:04:25.503840594 (539h35m25.374923629s) [seg_pos={0 0}, upload=473201]: Begin upload segment... Dec 28 03:04:25.503879828 (539h35m25.374962863s) [seg_pos={0 0}, upload=473201]: Done begin upload segment. Dec 28 03:04:25.503912491 (539h35m25.374995526s) [noise=true , address=145.79.227.45:4229 , piece_id=K7MNVHJM6URPEDG3E67JZUIWNPIJNL6BE4EFE47DW4HC7IZWJ3AQ, seg_pos={0 0}, upload=473201]: Uploading piece... Dec 28 03:04:25.503922641 (539h35m25.375005686s) [noise=true , address=145.79.227.17:4970 , piece_id=ID6TEOSJH7NHJ573UD4PGPQJ4AHV76A6QJDQNLOPEKHTJBLZTTOQ, seg_pos={0 0}, upload=473201]: Uploading piece... Dec 28 03:04:25.504315999 (539h35m25.375399044s) [noise=true , address=145.79.227.50:4374 , piece_id=GLC7OKYVH2TGR53DT77HOXG4G6OJKAGLP6DLVHWEKIPCYYNMEMDQ, seg_pos={0 0}, upload=473201]: Uploading piece... Dec 28 03:04:25.504755405 (539h35m25.37583844s) [noise=true , address=45.91.116.6:4973 , piece_id=RNAPEX6JF2U2U4QERQWUWJGVFK32FPNBZIQU2FO3DT2G6E6ATRBA, seg_pos={0 0}, upload=473201]: Uploading piece... Dec 28 03:04:25.505189733 (539h35m25.376272768s) [noise=true , address=45.91.116.9:4264 , piece_id=D77NGSC32NLEWLMY5NTPTWDHHLLWOUIZ2EKE5UGNZA6HEDIVJBXA, seg_pos={0 0}, upload=473201]: Uploading piece... Dec 28 03:04:25.505594995 (539h35m25.37667803s) [noise=true , address=145.79.227.11:4731 , piece_id=4EO3HS25FMKDX4VSN7OH3ZXJMSLJXWGO6S2UCKYHKT3QDDQ4ZOMA, seg_pos={0 0}, upload=473201]: Uploading piece... Dec 28 03:04:25.505964328 (539h35m25.377047363s) [noise=true , address=45.91.116.8:4045 , piece_id=57TEGS37MFACFAB5ZXERXDS64LSC2E46E7MJ6D4JNERWZOUJUWUA, seg_pos={0 0}, upload=473201]: Uploading piece... Dec 28 03:04:25.506373397 (539h35m25.377456432s) [noise=true , address=145.79.227.41:4087 , piece_id=O4WFB5XMWBUIEJVXE3N2QMVZ4OQFLP5HSAWN6WU4XLKWJ3S5FVVQ, seg_pos={0 0}, upload=473201]: Uploading piece... Dec 28 03:04:25.506475251 (539h35m25.377558296s) [noise=true , address=145.79.227.46:4603 , piece_id=BBURO6MDL5HZKM4ZYRYI4VS3M352VAA5MPJWKUXL7PXZUAHQE6EQ, seg_pos={0 0}, upload=473201]: Uploading piece... Dec 28 03:04:25.506896894 (539h35m25.377979939s) [noise=true , address=145.79.227.31:4799 , piece_id=43HXEF2GLUBRVOODC2H5UT6HB4ABPCFMPNSM2I63TE4RGF6D7HPQ, seg_pos={0 0}, upload=473201]: Uploading piece... Dec 28 03:04:25.507189340 (539h35m25.378272385s) [noise=true , address=45.91.116.5:4299 , piece_id=OY2DU6RUKIU7EOHAIRFFHSHTRKTJJMFC4PXCZ3NVN2V6AFEUC73Q, seg_pos={0 0}, upload=473201]: Uploading piece... Dec 28 03:04:25.507456249 (539h35m25.378539294s) [noise=true , address=145.79.227.38:4978 , piece_id=JGRS7ME4HLRQHHVEZ5CWP2ONR2ZI6UYAEYPO2XAN5RTHHT2WWFAA, seg_pos={0 0}, upload=473201]: Uploading piece... Dec 28 03:04:25.507798260 (539h35m25.378881305s) [noise=true , address=145.79.227.36:4518 , piece_id=GUK4FU6LNGH6WRLDGUXALFR7WYAUKHGXRHYQ5BTOW3UFHJODHVFA, seg_pos={0 0}, upload=473201]: Uploading piece... Dec 28 03:04:25.508135463 (539h35m25.379218498s) [noise=true , address=145.79.227.4:4187 , piece_id=EF3D5NZUOAQKVUITJKKOUWMCEZL57MZ4UWKU6VKDMTRTF6HVFZEQ, seg_pos={0 0}, upload=473201]: Uploading piece... Dec 28 03:04:25.508444099 (539h35m25.379527144s) [noise=true , address=145.79.227.48:4302 , piece_id=EEWFUOKV6EOSQ675DKL3QLY43FEZVUAS5MBEULHEWMCVYVZBA6BA, seg_pos={0 0}, upload=473201]: Uploading piece... Dec 28 03:04:25.508579277 (539h35m25.379662312s) [noise=true , address=145.79.227.12:4767 , piece_id=MQ3PGHP2DXDODVY2MEB3OTARYB4NZ4XTRW7L7RE5TEYYHDA2OIXA, seg_pos={0 0}, upload=473201]: Uploading piece... Dec 28 03:04:27.230910342 (539h35m27.101993377s) [upload=473201]: Next returned nil segment Dec 28 03:04:27.230923095 (539h35m27.102006131s) [upload=473201]: Waiting for error group managing segments... Dec 28 03:04:27.511612325 (539h35m27.38269537s) [noise=true , address=145.79.227.46:4603 , piece_id=BBURO6MDL5HZKM4ZYRYI4VS3M352VAA5MPJWKUXL7PXZUAHQE6EQ, seg_pos={0 0}, upload=473201]: Done uploading piece. err: <nil> Dec 28 03:04:27.538793362 (539h35m27.409876408s) [noise=true , address=145.79.227.4:4187 , piece_id=EF3D5NZUOAQKVUITJKKOUWMCEZL57MZ4UWKU6VKDMTRTF6HVFZEQ, seg_pos={0 0}, upload=473201]: Done uploading piece. err: <nil> Dec 28 03:04:27.586397048 (539h35m27.457480083s) [noise=true , address=145.79.227.48:4302 , piece_id=EEWFUOKV6EOSQ675DKL3QLY43FEZVUAS5MBEULHEWMCVYVZBA6BA, seg_pos={0 0}, upload=473201]: Done uploading piece. err: <nil> Dec 28 03:04:27.614590192 (539h35m27.485673227s) [noise=true , address=145.79.227.12:4767 , piece_id=MQ3PGHP2DXDODVY2MEB3OTARYB4NZ4XTRW7L7RE5TEYYHDA2OIXA, seg_pos={0 0}, upload=473201]: Done uploading piece. err: <nil> Dec 28 03:04:27.739578470 (539h35m27.610661525s) [noise=true , address=45.91.116.9:4264 , piece_id=D77NGSC32NLEWLMY5NTPTWDHHLLWOUIZ2EKE5UGNZA6HEDIVJBXA, seg_pos={0 0}, upload=473201]: Done uploading piece. err: <nil> Dec 28 03:04:27.745173835 (539h35m27.61625688s) [noise=true , address=45.91.116.5:4299 , piece_id=OY2DU6RUKIU7EOHAIRFFHSHTRKTJJMFC4PXCZ3NVN2V6AFEUC73Q, seg_pos={0 0}, upload=473201]: Done uploading piece. err: <nil> Dec 28 03:04:27.773320460 (539h35m27.644403495s) [noise=true , address=45.91.116.8:4045 , piece_id=57TEGS37MFACFAB5ZXERXDS64LSC2E46E7MJ6D4JNERWZOUJUWUA, seg_pos={0 0}, upload=473201]: Done uploading piece. err: <nil> Dec 28 03:04:27.779318893 (539h35m27.650401928s) [noise=true , address=145.79.227.41:4087 , piece_id=O4WFB5XMWBUIEJVXE3N2QMVZ4OQFLP5HSAWN6WU4XLKWJ3S5FVVQ, seg_pos={0 0}, upload=473201]: Done uploading piece. err: <nil> Dec 28 03:04:27.847756299 (539h35m27.718839334s) [noise=true , address=45.91.116.6:4973 , piece_id=RNAPEX6JF2U2U4QERQWUWJGVFK32FPNBZIQU2FO3DT2G6E6ATRBA, seg_pos={0 0}, upload=473201]: Done uploading piece. err: <nil> Dec 28 03:04:28.623260618 (539h35m28.494343653s) [noise=true , address=145.79.227.17:4970 , piece_id=ID6TEOSJH7NHJ573UD4PGPQJ4AHV76A6QJDQNLOPEKHTJBLZTTOQ, seg_pos={0 0}, upload=473201]: Done uploading piece. err: <nil> Dec 28 03:04:28.646834597 (539h35m28.517917632s) [noise=true , address=145.79.227.45:4229 , piece_id=K7MNVHJM6URPEDG3E67JZUIWNPIJNL6BE4EFE47DW4HC7IZWJ3AQ, seg_pos={0 0}, upload=473201]: Done uploading piece. err: <nil> Dec 28 03:04:28.651818427 (539h35m28.522901462s) [noise=true , address=145.79.227.38:4978 , piece_id=JGRS7ME4HLRQHHVEZ5CWP2ONR2ZI6UYAEYPO2XAN5RTHHT2WWFAA, seg_pos={0 0}, upload=473201]: Done uploading piece. err: <nil> Dec 28 03:04:28.655582684 (539h35m28.526665719s) [noise=true , address=145.79.227.31:4799 , piece_id=43HXEF2GLUBRVOODC2H5UT6HB4ABPCFMPNSM2I63TE4RGF6D7HPQ, seg_pos={0 0}, upload=473201]: Done uploading piece. err: <nil> Dec 28 03:04:28.655595680 (539h35m28.526678715s) [seg_pos={0 0}, upload=473201]: Segment reached optimal threshold of 13 pieces. Dec 28 03:04:28.655799739 (539h35m28.526882784s) [noise=true , address=145.79.227.36:4518 , piece_id=GUK4FU6LNGH6WRLDGUXALFR7WYAUKHGXRHYQ5BTOW3UFHJODHVFA, seg_pos={0 0}, upload=473201]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17TZ9JLdCivFF46guhSTW6hcoUmTz5uo5wvLkDqC4BGmpm9eLb): protocol: expected piece hash; context canceled; EOF Dec 28 03:04:28.655805980 (539h35m28.526889015s) [noise=true , address=145.79.227.36:4518 , piece_id=GUK4FU6LNGH6WRLDGUXALFR7WYAUKHGXRHYQ5BTOW3UFHJODHVFA, seg_pos={0 0}, upload=473201]: Long tail cancellation due to optimal threshold Dec 28 03:04:28.655916320 (539h35m28.526999355s) [noise=true , address=145.79.227.11:4731 , piece_id=4EO3HS25FMKDX4VSN7OH3ZXJMSLJXWGO6S2UCKYHKT3QDDQ4ZOMA, seg_pos={0 0}, upload=473201]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9): protocol: expected piece hash; context canceled; EOF Dec 28 03:04:28.655921530 (539h35m28.527004565s) [noise=true , address=145.79.227.11:4731 , piece_id=4EO3HS25FMKDX4VSN7OH3ZXJMSLJXWGO6S2UCKYHKT3QDDQ4ZOMA, seg_pos={0 0}, upload=473201]: Long tail cancellation due to optimal threshold Dec 28 03:04:28.655943713 (539h35m28.527026778s) [noise=true , address=145.79.227.50:4374 , piece_id=GLC7OKYVH2TGR53DT77HOXG4G6OJKAGLP6DLVHWEKIPCYYNMEMDQ, seg_pos={0 0}, upload=473201]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K): protocol: expected piece hash; context canceled; EOF Dec 28 03:04:28.655961255 (539h35m28.52704429s) [noise=true , address=145.79.227.50:4374 , piece_id=GLC7OKYVH2TGR53DT77HOXG4G6OJKAGLP6DLVHWEKIPCYYNMEMDQ, seg_pos={0 0}, upload=473201]: Long tail cancellation due to optimal threshold Dec 28 03:04:28.655984730 (539h35m28.527067766s) [seg_pos={0 0}, upload=473201]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 03:04:28.656087396 (539h35m28.527170431s) [upload=473201]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 03:04:28.656098537 (539h35m28.527181582s) [upload=473201]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 03:04:28.953337520 (539h35m28.824420555s) [upload=473201]: Done uploading segments... Dec 28 03:04:28.953356145 (539h35m28.82443918s) [upload=473201]: Upload finished. err: <nil> Duration: 14.867109209s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....X.....X................................................] duration: 52.297731785s trace: https://jaeger.dev.storj.io/trace/1822be52d200e5b5 FAIL (2025-12-28 03:05:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1ZAfYZ4wgpZVvatj6wtHona3hTaPcLxAmXC1DT2uKQyo8nLQQ7, Piece ID: 5MMQ44MXRT4XEL4X6WO54WRG2XYMX3AN3FNQ2H2XIUTKBXI3EEPA) context deadline exceeded; piecestore: (Node ID: 124guAM82oibYZsMho67Em6Ke6yEZXwT6LPMLLSZZBLqwoFGUkn, Piece ID: 22WJIPX4SM5NTIAW5ZRM2UKX32CTDBFCXPCCHRRWRCUCOF5CRE5Q) context deadline exceeded; piecestore: (Node ID: 1e6p1TD5CqUyWdZjkrxh2kRqMuyDiMUr9eimfeayDAKwhtBrZ8, Piece ID: XRI5EJFN7YYUG4QKLDGEFT6YAUHCUTTSXF735UDR6LZZBGMAZDJQ) context deadline exceeded ======LOGS====== 5.234848204 (538h55m35.105931239s) [noise=true , address=77.90.31.11:4572 , piece_id=5MMQ44MXRT4XEL4X6WO54WRG2XYMX3AN3FNQ2H2XIUTKBXI3EEPA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.235412057 (538h55m35.106495102s) [noise=true , address=145.79.227.22:4512 , piece_id=D46YLTIKGHG36DQ4YBKBSUDV4GGB4WVDWNOJS27IYYLN6AYZ5QFQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.235820946 (538h55m35.106903981s) [noise=true , address=77.90.31.19:4874 , piece_id=RHZUXUYMJU6RKQGSDZM5WDWIN5SQ7YIRSRATNNKT5GEMU2QH444A, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.236192192 (538h55m35.107275237s) [noise=true , address=77.90.31.16:4760 , piece_id=XRI5EJFN7YYUG4QKLDGEFT6YAUHCUTTSXF735UDR6LZZBGMAZDJQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.236564772 (538h55m35.107647817s) [noise=true , address=145.79.227.40:4062 , piece_id=PW6QWIPQWVUETLC6DJ624G6IB4P6EM4SYB72NAMQ4V6NUKNLUYLQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.236929766 (538h55m35.108012812s) [noise=true , address=64.49.13.139:4480 , piece_id=JAJY3F7IDFFMFKCK2QJV6W74RIESLRKBLXMN27Q3C45JWEPLBZLQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.237916846 (538h55m35.108999881s) [noise=true , address=145.79.227.34:4932 , piece_id=UPXSCQ45QDGPKNJXRFJ33EAA3QENVHRVQARU5TFJTEWICZNR253Q, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.238852948 (538h55m35.109935983s) [noise=true , address=45.91.116.12:4257 , piece_id=ZI67LIAHZW4G357GWKDPTB5SMXYGNXZTTTDMNAETDS2B6SWVDAEQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239152910 (538h55m35.110235945s) [noise=true , address=145.79.227.44:4222 , piece_id=CZ2CS62CTHQFAIE3S3JO2N3VJ4XIR5YJ2LMRVLLX7I4VM2VZVQVA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239409969 (538h55m35.110493004s) [noise=true , address=145.79.227.30:4773 , piece_id=KUGJXZCNMUE7WSK3VLY4AUBSOT4H6IY4XB4NCBGAGCNUG7F7IODQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239681136 (538h55m35.110764171s) [noise=true , address=77.90.31.13:4635 , piece_id=IIOY5X2W3XW4C2OFEBRCABMYFBI24RRTYNUSKN6KJU7CTUW4KONQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239954676 (538h55m35.111037711s) [noise=true , address=45.91.116.9:4281 , piece_id=JZLDW777CBX2BP7MLOQV4NFP3H4XJVCNQXTHNSUWPVDOJLPZK36A, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.240189914 (538h55m35.111272949s) [noise=true , address=145.79.227.35:4512 , piece_id=3E3VCCC45XVT5FHFDKXSTGGAGYBEML3BKU6JJTT33LCD2YMOMBZQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.240449147 (538h55m35.111532182s) [noise=true , address=45.91.116.8:4064 , piece_id=22WJIPX4SM5NTIAW5ZRM2UKX32CTDBFCXPCCHRRWRCUCOF5CRE5Q, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.240639640 (538h55m35.111722675s) [noise=true , address=145.79.227.32:4853 , piece_id=MBNCFDBQ4SZZXV4F7J5TRMOUYUQ5IPKQCGYGGM5SOIFJ6ZYCAUYA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.241137798 (538h55m35.112220833s) [noise=true , address=77.90.31.16:4751 , piece_id=N3Z2LLMCIA2F42VXZA7IDRU4XEKGOICH6TRVERODU76MFFE4QSHA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.241383797 (538h55m35.112466842s) [noise=true , address=77.90.31.18:4836 , piece_id=BSVEFI52EKN44CMINFUSZBNOH53IW5TXLVTF5IH6ZUTNII6RRLVA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:36.592880816 (538h55m36.463963871s) [upload=472631]: Next returned nil segment Dec 28 02:24:36.592901625 (538h55m36.46398467s) [upload=472631]: Waiting for error group managing segments... Dec 28 02:24:36.946933741 (538h55m36.818016786s) [noise=true , address=64.49.13.139:4480 , piece_id=JAJY3F7IDFFMFKCK2QJV6W74RIESLRKBLXMN27Q3C45JWEPLBZLQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.128189692 (538h55m36.999272737s) [noise=true , address=145.79.227.35:4512 , piece_id=3E3VCCC45XVT5FHFDKXSTGGAGYBEML3BKU6JJTT33LCD2YMOMBZQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.160760084 (538h55m37.031843119s) [noise=true , address=145.79.227.34:4932 , piece_id=UPXSCQ45QDGPKNJXRFJ33EAA3QENVHRVQARU5TFJTEWICZNR253Q, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.288838207 (538h55m37.159921252s) [noise=true , address=77.90.31.11:4572 , piece_id=5MMQ44MXRT4XEL4X6WO54WRG2XYMX3AN3FNQ2H2XIUTKBXI3EEPA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.319996310 (538h55m37.191079345s) [noise=true , address=145.79.227.22:4512 , piece_id=D46YLTIKGHG36DQ4YBKBSUDV4GGB4WVDWNOJS27IYYLN6AYZ5QFQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.320513273 (538h55m37.191596319s) [noise=true , address=77.90.31.18:4836 , piece_id=BSVEFI52EKN44CMINFUSZBNOH53IW5TXLVTF5IH6ZUTNII6RRLVA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.336522926 (538h55m37.207605971s) [noise=true , address=145.79.227.32:4853 , piece_id=MBNCFDBQ4SZZXV4F7J5TRMOUYUQ5IPKQCGYGGM5SOIFJ6ZYCAUYA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.384868031 (538h55m37.255951066s) [noise=true , address=77.90.31.16:4751 , piece_id=N3Z2LLMCIA2F42VXZA7IDRU4XEKGOICH6TRVERODU76MFFE4QSHA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.453563554 (538h55m37.324646599s) [noise=true , address=45.91.116.8:4064 , piece_id=22WJIPX4SM5NTIAW5ZRM2UKX32CTDBFCXPCCHRRWRCUCOF5CRE5Q, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.479454839 (538h55m37.350537884s) [noise=true , address=45.91.116.12:4257 , piece_id=ZI67LIAHZW4G357GWKDPTB5SMXYGNXZTTTDMNAETDS2B6SWVDAEQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.659710174 (538h55m37.530793219s) [noise=true , address=77.90.31.13:4635 , piece_id=IIOY5X2W3XW4C2OFEBRCABMYFBI24RRTYNUSKN6KJU7CTUW4KONQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.715039128 (538h55m37.586122163s) [noise=true , address=77.90.31.16:4760 , piece_id=XRI5EJFN7YYUG4QKLDGEFT6YAUHCUTTSXF735UDR6LZZBGMAZDJQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.977473406 (538h55m37.848556441s) [noise=true , address=77.90.31.19:4874 , piece_id=RHZUXUYMJU6RKQGSDZM5WDWIN5SQ7YIRSRATNNKT5GEMU2QH444A, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:38.016302745 (538h55m37.88738578s) [noise=true , address=145.79.227.30:4773 , piece_id=KUGJXZCNMUE7WSK3VLY4AUBSOT4H6IY4XB4NCBGAGCNUG7F7IODQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:38.016326982 (538h55m37.887410027s) [seg_pos={0 0}, upload=472631]: Segment reached optimal threshold of 14 pieces. Dec 28 02:24:38.016585353 (538h55m37.887668398s) [noise=true , address=145.79.227.30:4766 , piece_id=67M2DMPAVWIQKQSWUAG73D4E4NRRSG4XEVPOUVIJPEQOUV7A6VVA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1k6zdamuKPkBdzyNZNWx4FCmaTkv33PchcwvTRR8bgcETBs4K7): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016594450 (538h55m37.887677495s) [noise=true , address=145.79.227.30:4766 , piece_id=67M2DMPAVWIQKQSWUAG73D4E4NRRSG4XEVPOUVIJPEQOUV7A6VVA, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016627093 (538h55m37.887710148s) [noise=true , address=45.91.116.9:4281 , piece_id=JZLDW777CBX2BP7MLOQV4NFP3H4XJVCNQXTHNSUWPVDOJLPZK36A, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gxf7DpJZthyR6f1PExX56tJavjhPw9Gxb4h8mtmeYKW3t3tS1): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016635258 (538h55m37.887718293s) [noise=true , address=45.91.116.9:4281 , piece_id=JZLDW777CBX2BP7MLOQV4NFP3H4XJVCNQXTHNSUWPVDOJLPZK36A, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016871298 (538h55m37.887954363s) [noise=true , address=145.79.227.44:4222 , piece_id=CZ2CS62CTHQFAIE3S3JO2N3VJ4XIR5YJ2LMRVLLX7I4VM2VZVQVA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZKW35KvrsoYRefbYeUFZo8bh4XZYycNKGEvTFjzibezjXgj3e): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016880124 (538h55m37.887963169s) [noise=true , address=145.79.227.44:4222 , piece_id=CZ2CS62CTHQFAIE3S3JO2N3VJ4XIR5YJ2LMRVLLX7I4VM2VZVQVA, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016934839 (538h55m37.888017884s) [noise=true , address=145.79.227.40:4062 , piece_id=PW6QWIPQWVUETLC6DJ624G6IB4P6EM4SYB72NAMQ4V6NUKNLUYLQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016943425 (538h55m37.88802649s) [noise=true , address=145.79.227.40:4062 , piece_id=PW6QWIPQWVUETLC6DJ624G6IB4P6EM4SYB72NAMQ4V6NUKNLUYLQ, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016968252 (538h55m37.888051297s) [seg_pos={0 0}, upload=472631]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 02:24:38.017067822 (538h55m37.888150867s) [upload=472631]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 02:24:38.017074244 (538h55m37.888157289s) [upload=472631]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 02:24:38.304306667 (538h55m38.175389712s) [upload=472631]: Done uploading segments... Dec 28 02:24:38.304329641 (538h55m38.175412686s) [upload=472631]: Upload finished. err: <nil> Duration: 52.297759176s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [.....X................X....X............X........X....X.....] duration: 3.578607513s trace: https://jaeger.dev.storj.io/trace/25c8812909e078aa FAIL (2025-12-28 03:05:13 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.578646127s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [......X................X....X............X........X....X....] duration: 3.522495639s trace: https://jaeger.dev.storj.io/trace/25c8812909e078aa FAIL (2025-12-28 03:04:56 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.522520987s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 46 out of 60 is failed (threshold=3) [X.....XXX.XXXXXX.XXXX.XXXX..X..XXXXX.XXXXXXXXXXXXX.XXXXXXXXX] duration: 30.00270882s trace: https://jaeger.dev.storj.io/trace/761c0fba24d52cf0 FAIL (2025-12-28 03:04:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 03:04:25.257955017 (539h35m25.129038062s) [upload=473201]: Starting upload Dec 28 03:04:25.258010052 (539h35m25.129093087s) [upload=473201]: Done starting upload Dec 28 03:04:25.258203399 (539h35m25.129286435s) [upload=473201]: Uploading segments... Dec 28 03:04:25.258235050 (539h35m25.129318095s) [upload=473201]: Got next segment. Inline: false Dec 28 03:04:25.258268544 (539h35m25.129351579s) [upload=473201]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 03:04:25.258273042 (539h35m25.129356087s) [upload=473201]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 03:04:25.503840594 (539h35m25.374923629s) [seg_pos={0 0}, upload=473201]: Begin upload segment... Dec 28 03:04:25.503879828 (539h35m25.374962863s) [seg_pos={0 0}, upload=473201]: Done begin upload segment. Dec 28 03:04:25.503912491 (539h35m25.374995526s) [noise=true , address=145.79.227.45:4229 , piece_id=K7MNVHJM6URPEDG3E67JZUIWNPIJNL6BE4EFE47DW4HC7IZWJ3AQ, seg_pos={0 0}, upload=473201]: Uploading piece... Dec 28 03:04:25.503922641 (539h35m25.375005686s) [noise=true , address=145.79.227.17:4970 , piece_id=ID6TEOSJH7NHJ573UD4PGPQJ4AHV76A6QJDQNLOPEKHTJBLZTTOQ, seg_pos={0 0}, upload=473201]: Uploading piece... Dec 28 03:04:25.504315999 (539h35m25.375399044s) [noise=true , address=145.79.227.50:4374 , piece_id=GLC7OKYVH2TGR53DT77HOXG4G6OJKAGLP6DLVHWEKIPCYYNMEMDQ, seg_pos={0 0}, upload=473201]: Uploading piece... Dec 28 03:04:25.504755405 (539h35m25.37583844s) [noise=true , address=45.91.116.6:4973 , piece_id=RNAPEX6JF2U2U4QERQWUWJGVFK32FPNBZIQU2FO3DT2G6E6ATRBA, seg_pos={0 0}, upload=473201]: Uploading piece... Dec 28 03:04:25.505189733 (539h35m25.376272768s) [noise=true , address=45.91.116.9:4264 , piece_id=D77NGSC32NLEWLMY5NTPTWDHHLLWOUIZ2EKE5UGNZA6HEDIVJBXA, seg_pos={0 0}, upload=473201]: Uploading piece... Dec 28 03:04:25.505594995 (539h35m25.37667803s) [noise=true , address=145.79.227.11:4731 , piece_id=4EO3HS25FMKDX4VSN7OH3ZXJMSLJXWGO6S2UCKYHKT3QDDQ4ZOMA, seg_pos={0 0}, upload=473201]: Uploading piece... Dec 28 03:04:25.505964328 (539h35m25.377047363s) [noise=true , address=45.91.116.8:4045 , piece_id=57TEGS37MFACFAB5ZXERXDS64LSC2E46E7MJ6D4JNERWZOUJUWUA, seg_pos={0 0}, upload=473201]: Uploading piece... Dec 28 03:04:25.506373397 (539h35m25.377456432s) [noise=true , address=145.79.227.41:4087 , piece_id=O4WFB5XMWBUIEJVXE3N2QMVZ4OQFLP5HSAWN6WU4XLKWJ3S5FVVQ, seg_pos={0 0}, upload=473201]: Uploading piece... Dec 28 03:04:25.506475251 (539h35m25.377558296s) [noise=true , address=145.79.227.46:4603 , piece_id=BBURO6MDL5HZKM4ZYRYI4VS3M352VAA5MPJWKUXL7PXZUAHQE6EQ, seg_pos={0 0}, upload=473201]: Uploading piece... Dec 28 03:04:25.506896894 (539h35m25.377979939s) [noise=true , address=145.79.227.31:4799 , piece_id=43HXEF2GLUBRVOODC2H5UT6HB4ABPCFMPNSM2I63TE4RGF6D7HPQ, seg_pos={0 0}, upload=473201]: Uploading piece... Dec 28 03:04:25.507189340 (539h35m25.378272385s) [noise=true , address=45.91.116.5:4299 , piece_id=OY2DU6RUKIU7EOHAIRFFHSHTRKTJJMFC4PXCZ3NVN2V6AFEUC73Q, seg_pos={0 0}, upload=473201]: Uploading piece... Dec 28 03:04:25.507456249 (539h35m25.378539294s) [noise=true , address=145.79.227.38:4978 , piece_id=JGRS7ME4HLRQHHVEZ5CWP2ONR2ZI6UYAEYPO2XAN5RTHHT2WWFAA, seg_pos={0 0}, upload=473201]: Uploading piece... Dec 28 03:04:25.507798260 (539h35m25.378881305s) [noise=true , address=145.79.227.36:4518 , piece_id=GUK4FU6LNGH6WRLDGUXALFR7WYAUKHGXRHYQ5BTOW3UFHJODHVFA, seg_pos={0 0}, upload=473201]: Uploading piece... Dec 28 03:04:25.508135463 (539h35m25.379218498s) [noise=true , address=145.79.227.4:4187 , piece_id=EF3D5NZUOAQKVUITJKKOUWMCEZL57MZ4UWKU6VKDMTRTF6HVFZEQ, seg_pos={0 0}, upload=473201]: Uploading piece... Dec 28 03:04:25.508444099 (539h35m25.379527144s) [noise=true , address=145.79.227.48:4302 , piece_id=EEWFUOKV6EOSQ675DKL3QLY43FEZVUAS5MBEULHEWMCVYVZBA6BA, seg_pos={0 0}, upload=473201]: Uploading piece... Dec 28 03:04:25.508579277 (539h35m25.379662312s) [noise=true , address=145.79.227.12:4767 , piece_id=MQ3PGHP2DXDODVY2MEB3OTARYB4NZ4XTRW7L7RE5TEYYHDA2OIXA, seg_pos={0 0}, upload=473201]: Uploading piece... Dec 28 03:04:27.230910342 (539h35m27.101993377s) [upload=473201]: Next returned nil segment Dec 28 03:04:27.230923095 (539h35m27.102006131s) [upload=473201]: Waiting for error group managing segments... Dec 28 03:04:27.511612325 (539h35m27.38269537s) [noise=true , address=145.79.227.46:4603 , piece_id=BBURO6MDL5HZKM4ZYRYI4VS3M352VAA5MPJWKUXL7PXZUAHQE6EQ, seg_pos={0 0}, upload=473201]: Done uploading piece. err: <nil> Dec 28 03:04:27.538793362 (539h35m27.409876408s) [noise=true , address=145.79.227.4:4187 , piece_id=EF3D5NZUOAQKVUITJKKOUWMCEZL57MZ4UWKU6VKDMTRTF6HVFZEQ, seg_pos={0 0}, upload=473201]: Done uploading piece. err: <nil> Dec 28 03:04:27.586397048 (539h35m27.457480083s) [noise=true , address=145.79.227.48:4302 , piece_id=EEWFUOKV6EOSQ675DKL3QLY43FEZVUAS5MBEULHEWMCVYVZBA6BA, seg_pos={0 0}, upload=473201]: Done uploading piece. err: <nil> Dec 28 03:04:27.614590192 (539h35m27.485673227s) [noise=true , address=145.79.227.12:4767 , piece_id=MQ3PGHP2DXDODVY2MEB3OTARYB4NZ4XTRW7L7RE5TEYYHDA2OIXA, seg_pos={0 0}, upload=473201]: Done uploading piece. err: <nil> Dec 28 03:04:27.739578470 (539h35m27.610661525s) [noise=true , address=45.91.116.9:4264 , piece_id=D77NGSC32NLEWLMY5NTPTWDHHLLWOUIZ2EKE5UGNZA6HEDIVJBXA, seg_pos={0 0}, upload=473201]: Done uploading piece. err: <nil> Dec 28 03:04:27.745173835 (539h35m27.61625688s) [noise=true , address=45.91.116.5:4299 , piece_id=OY2DU6RUKIU7EOHAIRFFHSHTRKTJJMFC4PXCZ3NVN2V6AFEUC73Q, seg_pos={0 0}, upload=473201]: Done uploading piece. err: <nil> Dec 28 03:04:27.773320460 (539h35m27.644403495s) [noise=true , address=45.91.116.8:4045 , piece_id=57TEGS37MFACFAB5ZXERXDS64LSC2E46E7MJ6D4JNERWZOUJUWUA, seg_pos={0 0}, upload=473201]: Done uploading piece. err: <nil> Dec 28 03:04:27.779318893 (539h35m27.650401928s) [noise=true , address=145.79.227.41:4087 , piece_id=O4WFB5XMWBUIEJVXE3N2QMVZ4OQFLP5HSAWN6WU4XLKWJ3S5FVVQ, seg_pos={0 0}, upload=473201]: Done uploading piece. err: <nil> Dec 28 03:04:27.847756299 (539h35m27.718839334s) [noise=true , address=45.91.116.6:4973 , piece_id=RNAPEX6JF2U2U4QERQWUWJGVFK32FPNBZIQU2FO3DT2G6E6ATRBA, seg_pos={0 0}, upload=473201]: Done uploading piece. err: <nil> Dec 28 03:04:28.623260618 (539h35m28.494343653s) [noise=true , address=145.79.227.17:4970 , piece_id=ID6TEOSJH7NHJ573UD4PGPQJ4AHV76A6QJDQNLOPEKHTJBLZTTOQ, seg_pos={0 0}, upload=473201]: Done uploading piece. err: <nil> Dec 28 03:04:28.646834597 (539h35m28.517917632s) [noise=true , address=145.79.227.45:4229 , piece_id=K7MNVHJM6URPEDG3E67JZUIWNPIJNL6BE4EFE47DW4HC7IZWJ3AQ, seg_pos={0 0}, upload=473201]: Done uploading piece. err: <nil> Dec 28 03:04:28.651818427 (539h35m28.522901462s) [noise=true , address=145.79.227.38:4978 , piece_id=JGRS7ME4HLRQHHVEZ5CWP2ONR2ZI6UYAEYPO2XAN5RTHHT2WWFAA, seg_pos={0 0}, upload=473201]: Done uploading piece. err: <nil> Dec 28 03:04:28.655582684 (539h35m28.526665719s) [noise=true , address=145.79.227.31:4799 , piece_id=43HXEF2GLUBRVOODC2H5UT6HB4ABPCFMPNSM2I63TE4RGF6D7HPQ, seg_pos={0 0}, upload=473201]: Done uploading piece. err: <nil> Dec 28 03:04:28.655595680 (539h35m28.526678715s) [seg_pos={0 0}, upload=473201]: Segment reached optimal threshold of 13 pieces. Dec 28 03:04:28.655799739 (539h35m28.526882784s) [noise=true , address=145.79.227.36:4518 , piece_id=GUK4FU6LNGH6WRLDGUXALFR7WYAUKHGXRHYQ5BTOW3UFHJODHVFA, seg_pos={0 0}, upload=473201]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17TZ9JLdCivFF46guhSTW6hcoUmTz5uo5wvLkDqC4BGmpm9eLb): protocol: expected piece hash; context canceled; EOF Dec 28 03:04:28.655805980 (539h35m28.526889015s) [noise=true , address=145.79.227.36:4518 , piece_id=GUK4FU6LNGH6WRLDGUXALFR7WYAUKHGXRHYQ5BTOW3UFHJODHVFA, seg_pos={0 0}, upload=473201]: Long tail cancellation due to optimal threshold Dec 28 03:04:28.655916320 (539h35m28.526999355s) [noise=true , address=145.79.227.11:4731 , piece_id=4EO3HS25FMKDX4VSN7OH3ZXJMSLJXWGO6S2UCKYHKT3QDDQ4ZOMA, seg_pos={0 0}, upload=473201]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9): protocol: expected piece hash; context canceled; EOF Dec 28 03:04:28.655921530 (539h35m28.527004565s) [noise=true , address=145.79.227.11:4731 , piece_id=4EO3HS25FMKDX4VSN7OH3ZXJMSLJXWGO6S2UCKYHKT3QDDQ4ZOMA, seg_pos={0 0}, upload=473201]: Long tail cancellation due to optimal threshold Dec 28 03:04:28.655943713 (539h35m28.527026778s) [noise=true , address=145.79.227.50:4374 , piece_id=GLC7OKYVH2TGR53DT77HOXG4G6OJKAGLP6DLVHWEKIPCYYNMEMDQ, seg_pos={0 0}, upload=473201]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K): protocol: expected piece hash; context canceled; EOF Dec 28 03:04:28.655961255 (539h35m28.52704429s) [noise=true , address=145.79.227.50:4374 , piece_id=GLC7OKYVH2TGR53DT77HOXG4G6OJKAGLP6DLVHWEKIPCYYNMEMDQ, seg_pos={0 0}, upload=473201]: Long tail cancellation due to optimal threshold Dec 28 03:04:28.655984730 (539h35m28.527067766s) [seg_pos={0 0}, upload=473201]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 03:04:28.656087396 (539h35m28.527170431s) [upload=473201]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 03:04:28.656098537 (539h35m28.527181582s) [upload=473201]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 03:04:28.953337520 (539h35m28.824420555s) [upload=473201]: Done uploading segments... Dec 28 03:04:28.953356145 (539h35m28.82443918s) [upload=473201]: Upload finished. err: <nil> Duration: 30.002720282s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [......X.....X...............................................] duration: 30.275298105s trace: https://jaeger.dev.storj.io/trace/1822be52d200e5b5 FAIL (2025-12-28 03:04:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1ZAfYZ4wgpZVvatj6wtHona3hTaPcLxAmXC1DT2uKQyo8nLQQ7, Piece ID: 5MMQ44MXRT4XEL4X6WO54WRG2XYMX3AN3FNQ2H2XIUTKBXI3EEPA) context deadline exceeded; piecestore: (Node ID: 124guAM82oibYZsMho67Em6Ke6yEZXwT6LPMLLSZZBLqwoFGUkn, Piece ID: 22WJIPX4SM5NTIAW5ZRM2UKX32CTDBFCXPCCHRRWRCUCOF5CRE5Q) context deadline exceeded; piecestore: (Node ID: 1e6p1TD5CqUyWdZjkrxh2kRqMuyDiMUr9eimfeayDAKwhtBrZ8, Piece ID: XRI5EJFN7YYUG4QKLDGEFT6YAUHCUTTSXF735UDR6LZZBGMAZDJQ) context deadline exceeded ======LOGS====== 5.234848204 (538h55m35.105931239s) [noise=true , address=77.90.31.11:4572 , piece_id=5MMQ44MXRT4XEL4X6WO54WRG2XYMX3AN3FNQ2H2XIUTKBXI3EEPA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.235412057 (538h55m35.106495102s) [noise=true , address=145.79.227.22:4512 , piece_id=D46YLTIKGHG36DQ4YBKBSUDV4GGB4WVDWNOJS27IYYLN6AYZ5QFQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.235820946 (538h55m35.106903981s) [noise=true , address=77.90.31.19:4874 , piece_id=RHZUXUYMJU6RKQGSDZM5WDWIN5SQ7YIRSRATNNKT5GEMU2QH444A, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.236192192 (538h55m35.107275237s) [noise=true , address=77.90.31.16:4760 , piece_id=XRI5EJFN7YYUG4QKLDGEFT6YAUHCUTTSXF735UDR6LZZBGMAZDJQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.236564772 (538h55m35.107647817s) [noise=true , address=145.79.227.40:4062 , piece_id=PW6QWIPQWVUETLC6DJ624G6IB4P6EM4SYB72NAMQ4V6NUKNLUYLQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.236929766 (538h55m35.108012812s) [noise=true , address=64.49.13.139:4480 , piece_id=JAJY3F7IDFFMFKCK2QJV6W74RIESLRKBLXMN27Q3C45JWEPLBZLQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.237916846 (538h55m35.108999881s) [noise=true , address=145.79.227.34:4932 , piece_id=UPXSCQ45QDGPKNJXRFJ33EAA3QENVHRVQARU5TFJTEWICZNR253Q, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.238852948 (538h55m35.109935983s) [noise=true , address=45.91.116.12:4257 , piece_id=ZI67LIAHZW4G357GWKDPTB5SMXYGNXZTTTDMNAETDS2B6SWVDAEQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239152910 (538h55m35.110235945s) [noise=true , address=145.79.227.44:4222 , piece_id=CZ2CS62CTHQFAIE3S3JO2N3VJ4XIR5YJ2LMRVLLX7I4VM2VZVQVA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239409969 (538h55m35.110493004s) [noise=true , address=145.79.227.30:4773 , piece_id=KUGJXZCNMUE7WSK3VLY4AUBSOT4H6IY4XB4NCBGAGCNUG7F7IODQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239681136 (538h55m35.110764171s) [noise=true , address=77.90.31.13:4635 , piece_id=IIOY5X2W3XW4C2OFEBRCABMYFBI24RRTYNUSKN6KJU7CTUW4KONQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239954676 (538h55m35.111037711s) [noise=true , address=45.91.116.9:4281 , piece_id=JZLDW777CBX2BP7MLOQV4NFP3H4XJVCNQXTHNSUWPVDOJLPZK36A, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.240189914 (538h55m35.111272949s) [noise=true , address=145.79.227.35:4512 , piece_id=3E3VCCC45XVT5FHFDKXSTGGAGYBEML3BKU6JJTT33LCD2YMOMBZQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.240449147 (538h55m35.111532182s) [noise=true , address=45.91.116.8:4064 , piece_id=22WJIPX4SM5NTIAW5ZRM2UKX32CTDBFCXPCCHRRWRCUCOF5CRE5Q, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.240639640 (538h55m35.111722675s) [noise=true , address=145.79.227.32:4853 , piece_id=MBNCFDBQ4SZZXV4F7J5TRMOUYUQ5IPKQCGYGGM5SOIFJ6ZYCAUYA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.241137798 (538h55m35.112220833s) [noise=true , address=77.90.31.16:4751 , piece_id=N3Z2LLMCIA2F42VXZA7IDRU4XEKGOICH6TRVERODU76MFFE4QSHA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.241383797 (538h55m35.112466842s) [noise=true , address=77.90.31.18:4836 , piece_id=BSVEFI52EKN44CMINFUSZBNOH53IW5TXLVTF5IH6ZUTNII6RRLVA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:36.592880816 (538h55m36.463963871s) [upload=472631]: Next returned nil segment Dec 28 02:24:36.592901625 (538h55m36.46398467s) [upload=472631]: Waiting for error group managing segments... Dec 28 02:24:36.946933741 (538h55m36.818016786s) [noise=true , address=64.49.13.139:4480 , piece_id=JAJY3F7IDFFMFKCK2QJV6W74RIESLRKBLXMN27Q3C45JWEPLBZLQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.128189692 (538h55m36.999272737s) [noise=true , address=145.79.227.35:4512 , piece_id=3E3VCCC45XVT5FHFDKXSTGGAGYBEML3BKU6JJTT33LCD2YMOMBZQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.160760084 (538h55m37.031843119s) [noise=true , address=145.79.227.34:4932 , piece_id=UPXSCQ45QDGPKNJXRFJ33EAA3QENVHRVQARU5TFJTEWICZNR253Q, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.288838207 (538h55m37.159921252s) [noise=true , address=77.90.31.11:4572 , piece_id=5MMQ44MXRT4XEL4X6WO54WRG2XYMX3AN3FNQ2H2XIUTKBXI3EEPA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.319996310 (538h55m37.191079345s) [noise=true , address=145.79.227.22:4512 , piece_id=D46YLTIKGHG36DQ4YBKBSUDV4GGB4WVDWNOJS27IYYLN6AYZ5QFQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.320513273 (538h55m37.191596319s) [noise=true , address=77.90.31.18:4836 , piece_id=BSVEFI52EKN44CMINFUSZBNOH53IW5TXLVTF5IH6ZUTNII6RRLVA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.336522926 (538h55m37.207605971s) [noise=true , address=145.79.227.32:4853 , piece_id=MBNCFDBQ4SZZXV4F7J5TRMOUYUQ5IPKQCGYGGM5SOIFJ6ZYCAUYA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.384868031 (538h55m37.255951066s) [noise=true , address=77.90.31.16:4751 , piece_id=N3Z2LLMCIA2F42VXZA7IDRU4XEKGOICH6TRVERODU76MFFE4QSHA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.453563554 (538h55m37.324646599s) [noise=true , address=45.91.116.8:4064 , piece_id=22WJIPX4SM5NTIAW5ZRM2UKX32CTDBFCXPCCHRRWRCUCOF5CRE5Q, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.479454839 (538h55m37.350537884s) [noise=true , address=45.91.116.12:4257 , piece_id=ZI67LIAHZW4G357GWKDPTB5SMXYGNXZTTTDMNAETDS2B6SWVDAEQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.659710174 (538h55m37.530793219s) [noise=true , address=77.90.31.13:4635 , piece_id=IIOY5X2W3XW4C2OFEBRCABMYFBI24RRTYNUSKN6KJU7CTUW4KONQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.715039128 (538h55m37.586122163s) [noise=true , address=77.90.31.16:4760 , piece_id=XRI5EJFN7YYUG4QKLDGEFT6YAUHCUTTSXF735UDR6LZZBGMAZDJQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.977473406 (538h55m37.848556441s) [noise=true , address=77.90.31.19:4874 , piece_id=RHZUXUYMJU6RKQGSDZM5WDWIN5SQ7YIRSRATNNKT5GEMU2QH444A, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:38.016302745 (538h55m37.88738578s) [noise=true , address=145.79.227.30:4773 , piece_id=KUGJXZCNMUE7WSK3VLY4AUBSOT4H6IY4XB4NCBGAGCNUG7F7IODQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:38.016326982 (538h55m37.887410027s) [seg_pos={0 0}, upload=472631]: Segment reached optimal threshold of 14 pieces. Dec 28 02:24:38.016585353 (538h55m37.887668398s) [noise=true , address=145.79.227.30:4766 , piece_id=67M2DMPAVWIQKQSWUAG73D4E4NRRSG4XEVPOUVIJPEQOUV7A6VVA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1k6zdamuKPkBdzyNZNWx4FCmaTkv33PchcwvTRR8bgcETBs4K7): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016594450 (538h55m37.887677495s) [noise=true , address=145.79.227.30:4766 , piece_id=67M2DMPAVWIQKQSWUAG73D4E4NRRSG4XEVPOUVIJPEQOUV7A6VVA, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016627093 (538h55m37.887710148s) [noise=true , address=45.91.116.9:4281 , piece_id=JZLDW777CBX2BP7MLOQV4NFP3H4XJVCNQXTHNSUWPVDOJLPZK36A, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gxf7DpJZthyR6f1PExX56tJavjhPw9Gxb4h8mtmeYKW3t3tS1): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016635258 (538h55m37.887718293s) [noise=true , address=45.91.116.9:4281 , piece_id=JZLDW777CBX2BP7MLOQV4NFP3H4XJVCNQXTHNSUWPVDOJLPZK36A, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016871298 (538h55m37.887954363s) [noise=true , address=145.79.227.44:4222 , piece_id=CZ2CS62CTHQFAIE3S3JO2N3VJ4XIR5YJ2LMRVLLX7I4VM2VZVQVA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZKW35KvrsoYRefbYeUFZo8bh4XZYycNKGEvTFjzibezjXgj3e): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016880124 (538h55m37.887963169s) [noise=true , address=145.79.227.44:4222 , piece_id=CZ2CS62CTHQFAIE3S3JO2N3VJ4XIR5YJ2LMRVLLX7I4VM2VZVQVA, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016934839 (538h55m37.888017884s) [noise=true , address=145.79.227.40:4062 , piece_id=PW6QWIPQWVUETLC6DJ624G6IB4P6EM4SYB72NAMQ4V6NUKNLUYLQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016943425 (538h55m37.88802649s) [noise=true , address=145.79.227.40:4062 , piece_id=PW6QWIPQWVUETLC6DJ624G6IB4P6EM4SYB72NAMQ4V6NUKNLUYLQ, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016968252 (538h55m37.888051297s) [seg_pos={0 0}, upload=472631]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 02:24:38.017067822 (538h55m37.888150867s) [upload=472631]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 02:24:38.017074244 (538h55m37.888157289s) [upload=472631]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 02:24:38.304306667 (538h55m38.175389712s) [upload=472631]: Done uploading segments... Dec 28 02:24:38.304329641 (538h55m38.175412686s) [upload=472631]: Upload finished. err: <nil> Duration: 30.275314766s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 45 out of 60 is failed (threshold=3) [.X.....XXX.XXXXXX.XXXX.XXXX..X..XXXXX.XXXXXXXXXXXXX.XXXXXXXX] duration: 30.002507163s trace: https://jaeger.dev.storj.io/trace/666fe2323c7d92d8 FAIL (2025-12-28 03:04:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: JMDETOABHWM6CKPW523QFQIJADWVU73OGICXT3KZXP2NHAN5YKHQ) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: ADTEOG5T7TBWLZKHDUBDKPOX6MJVU375N727V55HTGQ734GVO4IA) context deadline exceeded ======LOGS====== Dec 28 03:03:55.037381146 (539h34m54.908464181s) [upload=473194]: Starting upload Dec 28 03:03:55.037467690 (539h34m54.908550725s) [upload=473194]: Done starting upload Dec 28 03:03:55.037698049 (539h34m54.908781094s) [upload=473194]: Uploading segments... Dec 28 03:03:55.037764294 (539h34m54.90884735s) [upload=473194]: Got next segment. Inline: false Dec 28 03:03:55.037828898 (539h34m54.908911933s) [upload=473194]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 03:03:55.037835421 (539h34m54.908918466s) [upload=473194]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 03:03:55.286224494 (539h34m55.157307549s) [seg_pos={0 0}, upload=473194]: Begin upload segment... Dec 28 03:03:55.286285631 (539h34m55.157368676s) [seg_pos={0 0}, upload=473194]: Done begin upload segment. Dec 28 03:03:55.286321028 (539h34m55.157404073s) [noise=true , address=45.91.116.4:4937 , piece_id=CHWFHVYH2HRMAAWTIVF5SJESULLCPYJ62LY52JVX7POVJ4FAZWKA, seg_pos={0 0}, upload=473194]: Uploading piece... Dec 28 03:03:55.286443151 (539h34m55.157526206s) [noise=true , address=145.79.227.42:4122 , piece_id=QCUU2PIYRY2Z426HXTTLUPWHPN3TOCXXYP6GL7HD6XL6GI5AHJ6Q, seg_pos={0 0}, upload=473194]: Uploading piece... Dec 28 03:03:55.287097496 (539h34m55.158180542s) [noise=true , address=77.90.31.10:4520 , piece_id=XTLQ4R6KY4E445LP5CKT2ERUB57F7H35SX7I7XOHIP2R7MZC2C5Q, seg_pos={0 0}, upload=473194]: Uploading piece... Dec 28 03:03:55.287517317 (539h34m55.158600352s) [noise=true , address=145.79.227.46:4603 , piece_id=ESGLQ2RY2IEZHLN4GXR2NTJUSCOJ6VLVGENCOKSFN6FZYHVISQMA, seg_pos={0 0}, upload=473194]: Uploading piece... Dec 28 03:03:55.287937477 (539h34m55.159020512s) [noise=true , address=145.79.227.11:4729 , piece_id=3ABXMT2XCKI5IHQTLMTZQWSVVD4G4EYQS6P64FWE2YXLYP237QKQ, seg_pos={0 0}, upload=473194]: Uploading piece... Dec 28 03:03:55.288389107 (539h34m55.159472152s) [noise=true , address=145.79.227.12:4767 , piece_id=ET4DHMUOB2N26QT6O46L6N6HX2XKSNPLVYI65AHLC7RYZMLOWZVA, seg_pos={0 0}, upload=473194]: Uploading piece... Dec 28 03:03:55.288850595 (539h34m55.15993364s) [noise=true , address=45.91.116.14:4190 , piece_id=JMDETOABHWM6CKPW523QFQIJADWVU73OGICXT3KZXP2NHAN5YKHQ, seg_pos={0 0}, upload=473194]: Uploading piece... Dec 28 03:03:55.289300452 (539h34m55.160383497s) [noise=true , address=45.91.116.7:4009 , piece_id=BRJD5MYF7EQUHZLHLE2KZDPBH55GWKWJ4JYM6UN47GVLO5LVKXAQ, seg_pos={0 0}, upload=473194]: Uploading piece... Dec 28 03:03:55.289886758 (539h34m55.160969824s) [noise=true , address=145.79.227.16:4912 , piece_id=I3KJF6Q6XSP3K4QJLODA5CSV72KHPMQIUC4SOTMI6BOQJ43JACUA, seg_pos={0 0}, upload=473194]: Uploading piece... Dec 28 03:03:55.291325197 (539h34m55.162408252s) [noise=true , address=145.79.227.14:4837 , piece_id=3YOX65MBH5IE7GH2MESV4VGB4N5GEMHU45YGSRUMUV2RULAHERTA, seg_pos={0 0}, upload=473194]: Uploading piece... Dec 28 03:03:55.292903983 (539h34m55.163987028s) [noise=true , address=45.91.116.8:4046 , piece_id=KSGKWSG53IH2G7IU7SXWOX73YW2NMMELZBFZMMFKXKLC7U7VGAAQ, seg_pos={0 0}, upload=473194]: Uploading piece... Dec 28 03:03:55.294586948 (539h34m55.165669994s) [noise=true , address=145.79.227.52:4446 , piece_id=4FHZZ5ILZMC6IYD3EBQH6G3TMIP3F4YQHUW67MC3MKC7I4DEJBIA, seg_pos={0 0}, upload=473194]: Uploading piece... Dec 28 03:03:55.296439526 (539h34m55.167522571s) [noise=true , address=145.79.227.40:4050 , piece_id=6L5TZR4RV2HZ6LJLQWTAITIQLFZJ2OOMCTDIE6LGDW72ORCVEINA, seg_pos={0 0}, upload=473194]: Uploading piece... Dec 28 03:03:55.298445074 (539h34m55.16952812s) [noise=true , address=45.91.116.9:4261 , piece_id=ADTEOG5T7TBWLZKHDUBDKPOX6MJVU375N727V55HTGQ734GVO4IA, seg_pos={0 0}, upload=473194]: Uploading piece... Dec 28 03:03:55.300920839 (539h34m55.172003884s) [noise=true , address=145.79.227.22:4511 , piece_id=7NFWKXQDALNUB53PFFOSEP4DRYXMUN6XPBH6DQOIG2GNMODRBAZA, seg_pos={0 0}, upload=473194]: Uploading piece... Dec 28 03:03:55.302581801 (539h34m55.173664866s) [noise=true , address=145.79.227.31:4799 , piece_id=PEEXADWFOBLIS3TSC6FZQLXDAXIW765NJZK6EDW5O3EVR4IKBVDQ, seg_pos={0 0}, upload=473194]: Uploading piece... Dec 28 03:03:56.924650961 (539h34m56.795734006s) [upload=473194]: Next returned nil segment Dec 28 03:03:56.924684355 (539h34m56.79576739s) [upload=473194]: Waiting for error group managing segments... Dec 28 03:03:57.312818967 (539h34m57.183902002s) [noise=true , address=145.79.227.12:4767 , piece_id=ET4DHMUOB2N26QT6O46L6N6HX2XKSNPLVYI65AHLC7RYZMLOWZVA, seg_pos={0 0}, upload=473194]: Done uploading piece. err: <nil> Dec 28 03:03:57.314603004 (539h34m57.185686099s) [noise=true , address=145.79.227.40:4050 , piece_id=6L5TZR4RV2HZ6LJLQWTAITIQLFZJ2OOMCTDIE6LGDW72ORCVEINA, seg_pos={0 0}, upload=473194]: Done uploading piece. err: <nil> Dec 28 03:03:57.314906753 (539h34m57.185989788s) [noise=true , address=145.79.227.46:4603 , piece_id=ESGLQ2RY2IEZHLN4GXR2NTJUSCOJ6VLVGENCOKSFN6FZYHVISQMA, seg_pos={0 0}, upload=473194]: Done uploading piece. err: <nil> Dec 28 03:03:57.382169112 (539h34m57.253252157s) [noise=true , address=145.79.227.52:4446 , piece_id=4FHZZ5ILZMC6IYD3EBQH6G3TMIP3F4YQHUW67MC3MKC7I4DEJBIA, seg_pos={0 0}, upload=473194]: Done uploading piece. err: <nil> Dec 28 03:03:57.392622296 (539h34m57.263705341s) [noise=true , address=145.79.227.42:4122 , piece_id=QCUU2PIYRY2Z426HXTTLUPWHPN3TOCXXYP6GL7HD6XL6GI5AHJ6Q, seg_pos={0 0}, upload=473194]: Done uploading piece. err: <nil> Dec 28 03:03:57.469038265 (539h34m57.3401213s) [noise=true , address=77.90.31.10:4520 , piece_id=XTLQ4R6KY4E445LP5CKT2ERUB57F7H35SX7I7XOHIP2R7MZC2C5Q, seg_pos={0 0}, upload=473194]: Done uploading piece. err: <nil> Dec 28 03:03:57.615811115 (539h34m57.48689416s) [noise=true , address=45.91.116.7:4009 , piece_id=BRJD5MYF7EQUHZLHLE2KZDPBH55GWKWJ4JYM6UN47GVLO5LVKXAQ, seg_pos={0 0}, upload=473194]: Done uploading piece. err: <nil> Dec 28 03:03:57.679141086 (539h34m57.550224131s) [noise=true , address=45.91.116.8:4046 , piece_id=KSGKWSG53IH2G7IU7SXWOX73YW2NMMELZBFZMMFKXKLC7U7VGAAQ, seg_pos={0 0}, upload=473194]: Done uploading piece. err: <nil> Dec 28 03:03:57.683866294 (539h34m57.554949329s) [noise=true , address=45.91.116.9:4261 , piece_id=ADTEOG5T7TBWLZKHDUBDKPOX6MJVU375N727V55HTGQ734GVO4IA, seg_pos={0 0}, upload=473194]: Done uploading piece. err: <nil> Dec 28 03:03:57.703230105 (539h34m57.57431314s) [noise=true , address=45.91.116.14:4190 , piece_id=JMDETOABHWM6CKPW523QFQIJADWVU73OGICXT3KZXP2NHAN5YKHQ, seg_pos={0 0}, upload=473194]: Done uploading piece. err: <nil> Dec 28 03:03:58.268306454 (539h34m58.139389489s) [noise=true , address=45.91.116.4:4937 , piece_id=CHWFHVYH2HRMAAWTIVF5SJESULLCPYJ62LY52JVX7POVJ4FAZWKA, seg_pos={0 0}, upload=473194]: Done uploading piece. err: <nil> Dec 28 03:03:58.484716736 (539h34m58.355799771s) [noise=true , address=145.79.227.11:4729 , piece_id=3ABXMT2XCKI5IHQTLMTZQWSVVD4G4EYQS6P64FWE2YXLYP237QKQ, seg_pos={0 0}, upload=473194]: Done uploading piece. err: <nil> Dec 28 03:03:58.507479820 (539h34m58.378562865s) [noise=true , address=145.79.227.22:4511 , piece_id=7NFWKXQDALNUB53PFFOSEP4DRYXMUN6XPBH6DQOIG2GNMODRBAZA, seg_pos={0 0}, upload=473194]: Done uploading piece. err: <nil> Dec 28 03:03:58.507498656 (539h34m58.378581692s) [seg_pos={0 0}, upload=473194]: Segment reached optimal threshold of 13 pieces. Dec 28 03:03:58.507696042 (539h34m58.378779077s) [noise=true , address=145.79.227.31:4799 , piece_id=PEEXADWFOBLIS3TSC6FZQLXDAXIW765NJZK6EDW5O3EVR4IKBVDQ, seg_pos={0 0}, upload=473194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X): protocol: expected piece hash; context canceled; EOF Dec 28 03:03:58.507702805 (539h34m58.37878584s) [noise=true , address=145.79.227.31:4799 , piece_id=PEEXADWFOBLIS3TSC6FZQLXDAXIW765NJZK6EDW5O3EVR4IKBVDQ, seg_pos={0 0}, upload=473194]: Long tail cancellation due to optimal threshold Dec 28 03:03:58.507806373 (539h34m58.378889408s) [noise=true , address=145.79.227.14:4837 , piece_id=3YOX65MBH5IE7GH2MESV4VGB4N5GEMHU45YGSRUMUV2RULAHERTA, seg_pos={0 0}, upload=473194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV): protocol: expected piece hash; context canceled; EOF Dec 28 03:03:58.507819006 (539h34m58.378902041s) [noise=true , address=145.79.227.14:4837 , piece_id=3YOX65MBH5IE7GH2MESV4VGB4N5GEMHU45YGSRUMUV2RULAHERTA, seg_pos={0 0}, upload=473194]: Long tail cancellation due to optimal threshold Dec 28 03:03:58.508034567 (539h34m58.379117612s) [noise=true , address=145.79.227.16:4912 , piece_id=I3KJF6Q6XSP3K4QJLODA5CSV72KHPMQIUC4SOTMI6BOQJ43JACUA, seg_pos={0 0}, upload=473194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR): protocol: expected piece hash; context canceled; EOF Dec 28 03:03:58.508042361 (539h34m58.379125396s) [noise=true , address=145.79.227.16:4912 , piece_id=I3KJF6Q6XSP3K4QJLODA5CSV72KHPMQIUC4SOTMI6BOQJ43JACUA, seg_pos={0 0}, upload=473194]: Long tail cancellation due to optimal threshold Dec 28 03:03:58.508070856 (539h34m58.379153921s) [seg_pos={0 0}, upload=473194]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 03:03:58.508178661 (539h34m58.379261696s) [upload=473194]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 03:03:58.508185404 (539h34m58.379268439s) [upload=473194]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 03:03:58.807031858 (539h34m58.678114903s) [upload=473194]: Done uploading segments... Dec 28 03:03:58.807054280 (539h34m58.678137315s) [upload=473194]: Upload finished. err: <nil> Duration: 30.002527053s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......X.....X..............................................] duration: 35.870344827s trace: https://jaeger.dev.storj.io/trace/1822be52d200e5b5 FAIL (2025-12-28 03:04:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1ZAfYZ4wgpZVvatj6wtHona3hTaPcLxAmXC1DT2uKQyo8nLQQ7, Piece ID: 5MMQ44MXRT4XEL4X6WO54WRG2XYMX3AN3FNQ2H2XIUTKBXI3EEPA) context deadline exceeded; piecestore: (Node ID: 124guAM82oibYZsMho67Em6Ke6yEZXwT6LPMLLSZZBLqwoFGUkn, Piece ID: 22WJIPX4SM5NTIAW5ZRM2UKX32CTDBFCXPCCHRRWRCUCOF5CRE5Q) context deadline exceeded; piecestore: (Node ID: 1e6p1TD5CqUyWdZjkrxh2kRqMuyDiMUr9eimfeayDAKwhtBrZ8, Piece ID: XRI5EJFN7YYUG4QKLDGEFT6YAUHCUTTSXF735UDR6LZZBGMAZDJQ) context deadline exceeded ======LOGS====== 5.234848204 (538h55m35.105931239s) [noise=true , address=77.90.31.11:4572 , piece_id=5MMQ44MXRT4XEL4X6WO54WRG2XYMX3AN3FNQ2H2XIUTKBXI3EEPA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.235412057 (538h55m35.106495102s) [noise=true , address=145.79.227.22:4512 , piece_id=D46YLTIKGHG36DQ4YBKBSUDV4GGB4WVDWNOJS27IYYLN6AYZ5QFQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.235820946 (538h55m35.106903981s) [noise=true , address=77.90.31.19:4874 , piece_id=RHZUXUYMJU6RKQGSDZM5WDWIN5SQ7YIRSRATNNKT5GEMU2QH444A, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.236192192 (538h55m35.107275237s) [noise=true , address=77.90.31.16:4760 , piece_id=XRI5EJFN7YYUG4QKLDGEFT6YAUHCUTTSXF735UDR6LZZBGMAZDJQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.236564772 (538h55m35.107647817s) [noise=true , address=145.79.227.40:4062 , piece_id=PW6QWIPQWVUETLC6DJ624G6IB4P6EM4SYB72NAMQ4V6NUKNLUYLQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.236929766 (538h55m35.108012812s) [noise=true , address=64.49.13.139:4480 , piece_id=JAJY3F7IDFFMFKCK2QJV6W74RIESLRKBLXMN27Q3C45JWEPLBZLQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.237916846 (538h55m35.108999881s) [noise=true , address=145.79.227.34:4932 , piece_id=UPXSCQ45QDGPKNJXRFJ33EAA3QENVHRVQARU5TFJTEWICZNR253Q, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.238852948 (538h55m35.109935983s) [noise=true , address=45.91.116.12:4257 , piece_id=ZI67LIAHZW4G357GWKDPTB5SMXYGNXZTTTDMNAETDS2B6SWVDAEQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239152910 (538h55m35.110235945s) [noise=true , address=145.79.227.44:4222 , piece_id=CZ2CS62CTHQFAIE3S3JO2N3VJ4XIR5YJ2LMRVLLX7I4VM2VZVQVA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239409969 (538h55m35.110493004s) [noise=true , address=145.79.227.30:4773 , piece_id=KUGJXZCNMUE7WSK3VLY4AUBSOT4H6IY4XB4NCBGAGCNUG7F7IODQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239681136 (538h55m35.110764171s) [noise=true , address=77.90.31.13:4635 , piece_id=IIOY5X2W3XW4C2OFEBRCABMYFBI24RRTYNUSKN6KJU7CTUW4KONQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239954676 (538h55m35.111037711s) [noise=true , address=45.91.116.9:4281 , piece_id=JZLDW777CBX2BP7MLOQV4NFP3H4XJVCNQXTHNSUWPVDOJLPZK36A, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.240189914 (538h55m35.111272949s) [noise=true , address=145.79.227.35:4512 , piece_id=3E3VCCC45XVT5FHFDKXSTGGAGYBEML3BKU6JJTT33LCD2YMOMBZQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.240449147 (538h55m35.111532182s) [noise=true , address=45.91.116.8:4064 , piece_id=22WJIPX4SM5NTIAW5ZRM2UKX32CTDBFCXPCCHRRWRCUCOF5CRE5Q, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.240639640 (538h55m35.111722675s) [noise=true , address=145.79.227.32:4853 , piece_id=MBNCFDBQ4SZZXV4F7J5TRMOUYUQ5IPKQCGYGGM5SOIFJ6ZYCAUYA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.241137798 (538h55m35.112220833s) [noise=true , address=77.90.31.16:4751 , piece_id=N3Z2LLMCIA2F42VXZA7IDRU4XEKGOICH6TRVERODU76MFFE4QSHA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.241383797 (538h55m35.112466842s) [noise=true , address=77.90.31.18:4836 , piece_id=BSVEFI52EKN44CMINFUSZBNOH53IW5TXLVTF5IH6ZUTNII6RRLVA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:36.592880816 (538h55m36.463963871s) [upload=472631]: Next returned nil segment Dec 28 02:24:36.592901625 (538h55m36.46398467s) [upload=472631]: Waiting for error group managing segments... Dec 28 02:24:36.946933741 (538h55m36.818016786s) [noise=true , address=64.49.13.139:4480 , piece_id=JAJY3F7IDFFMFKCK2QJV6W74RIESLRKBLXMN27Q3C45JWEPLBZLQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.128189692 (538h55m36.999272737s) [noise=true , address=145.79.227.35:4512 , piece_id=3E3VCCC45XVT5FHFDKXSTGGAGYBEML3BKU6JJTT33LCD2YMOMBZQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.160760084 (538h55m37.031843119s) [noise=true , address=145.79.227.34:4932 , piece_id=UPXSCQ45QDGPKNJXRFJ33EAA3QENVHRVQARU5TFJTEWICZNR253Q, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.288838207 (538h55m37.159921252s) [noise=true , address=77.90.31.11:4572 , piece_id=5MMQ44MXRT4XEL4X6WO54WRG2XYMX3AN3FNQ2H2XIUTKBXI3EEPA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.319996310 (538h55m37.191079345s) [noise=true , address=145.79.227.22:4512 , piece_id=D46YLTIKGHG36DQ4YBKBSUDV4GGB4WVDWNOJS27IYYLN6AYZ5QFQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.320513273 (538h55m37.191596319s) [noise=true , address=77.90.31.18:4836 , piece_id=BSVEFI52EKN44CMINFUSZBNOH53IW5TXLVTF5IH6ZUTNII6RRLVA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.336522926 (538h55m37.207605971s) [noise=true , address=145.79.227.32:4853 , piece_id=MBNCFDBQ4SZZXV4F7J5TRMOUYUQ5IPKQCGYGGM5SOIFJ6ZYCAUYA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.384868031 (538h55m37.255951066s) [noise=true , address=77.90.31.16:4751 , piece_id=N3Z2LLMCIA2F42VXZA7IDRU4XEKGOICH6TRVERODU76MFFE4QSHA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.453563554 (538h55m37.324646599s) [noise=true , address=45.91.116.8:4064 , piece_id=22WJIPX4SM5NTIAW5ZRM2UKX32CTDBFCXPCCHRRWRCUCOF5CRE5Q, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.479454839 (538h55m37.350537884s) [noise=true , address=45.91.116.12:4257 , piece_id=ZI67LIAHZW4G357GWKDPTB5SMXYGNXZTTTDMNAETDS2B6SWVDAEQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.659710174 (538h55m37.530793219s) [noise=true , address=77.90.31.13:4635 , piece_id=IIOY5X2W3XW4C2OFEBRCABMYFBI24RRTYNUSKN6KJU7CTUW4KONQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.715039128 (538h55m37.586122163s) [noise=true , address=77.90.31.16:4760 , piece_id=XRI5EJFN7YYUG4QKLDGEFT6YAUHCUTTSXF735UDR6LZZBGMAZDJQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.977473406 (538h55m37.848556441s) [noise=true , address=77.90.31.19:4874 , piece_id=RHZUXUYMJU6RKQGSDZM5WDWIN5SQ7YIRSRATNNKT5GEMU2QH444A, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:38.016302745 (538h55m37.88738578s) [noise=true , address=145.79.227.30:4773 , piece_id=KUGJXZCNMUE7WSK3VLY4AUBSOT4H6IY4XB4NCBGAGCNUG7F7IODQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:38.016326982 (538h55m37.887410027s) [seg_pos={0 0}, upload=472631]: Segment reached optimal threshold of 14 pieces. Dec 28 02:24:38.016585353 (538h55m37.887668398s) [noise=true , address=145.79.227.30:4766 , piece_id=67M2DMPAVWIQKQSWUAG73D4E4NRRSG4XEVPOUVIJPEQOUV7A6VVA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1k6zdamuKPkBdzyNZNWx4FCmaTkv33PchcwvTRR8bgcETBs4K7): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016594450 (538h55m37.887677495s) [noise=true , address=145.79.227.30:4766 , piece_id=67M2DMPAVWIQKQSWUAG73D4E4NRRSG4XEVPOUVIJPEQOUV7A6VVA, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016627093 (538h55m37.887710148s) [noise=true , address=45.91.116.9:4281 , piece_id=JZLDW777CBX2BP7MLOQV4NFP3H4XJVCNQXTHNSUWPVDOJLPZK36A, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gxf7DpJZthyR6f1PExX56tJavjhPw9Gxb4h8mtmeYKW3t3tS1): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016635258 (538h55m37.887718293s) [noise=true , address=45.91.116.9:4281 , piece_id=JZLDW777CBX2BP7MLOQV4NFP3H4XJVCNQXTHNSUWPVDOJLPZK36A, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016871298 (538h55m37.887954363s) [noise=true , address=145.79.227.44:4222 , piece_id=CZ2CS62CTHQFAIE3S3JO2N3VJ4XIR5YJ2LMRVLLX7I4VM2VZVQVA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZKW35KvrsoYRefbYeUFZo8bh4XZYycNKGEvTFjzibezjXgj3e): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016880124 (538h55m37.887963169s) [noise=true , address=145.79.227.44:4222 , piece_id=CZ2CS62CTHQFAIE3S3JO2N3VJ4XIR5YJ2LMRVLLX7I4VM2VZVQVA, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016934839 (538h55m37.888017884s) [noise=true , address=145.79.227.40:4062 , piece_id=PW6QWIPQWVUETLC6DJ624G6IB4P6EM4SYB72NAMQ4V6NUKNLUYLQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016943425 (538h55m37.88802649s) [noise=true , address=145.79.227.40:4062 , piece_id=PW6QWIPQWVUETLC6DJ624G6IB4P6EM4SYB72NAMQ4V6NUKNLUYLQ, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016968252 (538h55m37.888051297s) [seg_pos={0 0}, upload=472631]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 02:24:38.017067822 (538h55m37.888150867s) [upload=472631]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 02:24:38.017074244 (538h55m37.888157289s) [upload=472631]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 02:24:38.304306667 (538h55m38.175389712s) [upload=472631]: Done uploading segments... Dec 28 02:24:38.304329641 (538h55m38.175412686s) [upload=472631]: Upload finished. err: <nil> Duration: 35.87037307s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [.......X................X....X............X........X....X...] duration: 5.000732595s trace: https://jaeger.dev.storj.io/trace/25c8812909e078aa FAIL (2025-12-28 03:03:57 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000754779s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 45 out of 60 is failed (threshold=3) [X.X.....XXX.XXXXXX.XXXX.XXXX..X..XXXXX.XXXXXXXXXXXXX.XXXXXXX] duration: 30.003271949s trace: https://jaeger.dev.storj.io/trace/18982c42f7ae7d69 FAIL (2025-12-28 03:03:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: IRVK2KSSZN5LTLUAG6SYIFJSIFDZFO2CVVCQX5UMCZ7KOLAYDR2A) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: JTJCTS3B73M4FDKNXGCMDIP2LS2SJ55SCQGLUCD5ERPNGRO7SLVQ) context deadline exceeded; piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: 5T3KM6L5GZR5GZVZJJ3LLUEERBCLY4JN4XOVMSF5LWUURQ3W7IRA) context deadline exceeded ======LOGS====== 03:03:13.785205711 (539h34m13.656288757s) [upload=473182]: Starting upload Dec 28 03:03:13.785296074 (539h34m13.656379109s) [upload=473182]: Done starting upload Dec 28 03:03:13.785381115 (539h34m13.65646415s) [upload=473182]: Uploading segments... Dec 28 03:03:13.785627394 (539h34m13.656710429s) [upload=473182]: Got next segment. Inline: false Dec 28 03:03:13.785671449 (539h34m13.656754484s) [upload=473182]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 03:03:13.785676619 (539h34m13.656759654s) [upload=473182]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 03:03:14.027908167 (539h34m13.898991212s) [seg_pos={0 0}, upload=473182]: Begin upload segment... Dec 28 03:03:14.027958313 (539h34m13.899041358s) [seg_pos={0 0}, upload=473182]: Done begin upload segment. Dec 28 03:03:14.028010071 (539h34m13.899093136s) [noise=true , address=145.79.227.20:4438 , piece_id=77N6MPOL4KI6WYQMHZQB43CVMNQNFE5JYXPN4KRUAFW36PLNBVAA, seg_pos={0 0}, upload=473182]: Uploading piece... Dec 28 03:03:14.028054566 (539h34m13.899137601s) [noise=true , address=145.79.227.39:4015 , piece_id=6FBZWOD2GUWSTL7OMYLTJYNKROG5HDTTTYNJJNCSTUKQB2D2TO5Q, seg_pos={0 0}, upload=473182]: Uploading piece... Dec 28 03:03:14.028569356 (539h34m13.899652391s) [noise=true , address=77.90.31.12:4590 , piece_id=HZLWL3B2H6EZC4PI4UZ7LF7EUE7LWRQO75CL6BESW2N6UVVM3XVQ, seg_pos={0 0}, upload=473182]: Uploading piece... Dec 28 03:03:14.029322200 (539h34m13.900405245s) [noise=true , address=145.79.227.41:4086 , piece_id=2AVS4WW6OY3F4RF4C7XGSMKKBANTNLENG54NOO6TAOO67EM4R7TQ, seg_pos={0 0}, upload=473182]: Uploading piece... Dec 28 03:03:14.029670944 (539h34m13.900753979s) [noise=true , address=45.91.116.9:4263 , piece_id=3WJ2V75SLYFAVWCGXJGUEYODFNIW4LAUCBW5T5QBN6S54I7SKAPA, seg_pos={0 0}, upload=473182]: Uploading piece... Dec 28 03:03:14.030040398 (539h34m13.901123433s) [noise=true , address=145.79.227.32:4836 , piece_id=IHH62G5XJK2H4RKV4LP3FDOMLMEHIH7IY44PFNUVWSLZGXLUXZQA, seg_pos={0 0}, upload=473182]: Uploading piece... Dec 28 03:03:14.030421143 (539h34m13.901504178s) [noise=true , address=145.79.227.42:4123 , piece_id=DWHWZXC5TIWHYXWDEGXODBRXOGGUAYNYR4KLALWV4W2TNEPATE3A, seg_pos={0 0}, upload=473182]: Uploading piece... Dec 28 03:03:14.031041924 (539h34m13.902124959s) [noise=true , address=45.91.116.6:4973 , piece_id=JTJCTS3B73M4FDKNXGCMDIP2LS2SJ55SCQGLUCD5ERPNGRO7SLVQ, seg_pos={0 0}, upload=473182]: Uploading piece... Dec 28 03:03:14.032019727 (539h34m13.903102762s) [noise=true , address=145.79.227.14:4838 , piece_id=FR42ESNX5Q2VHGUVXNMIMYQVXADUS42BBXLQCCVPU2BB7OQFEMPA, seg_pos={0 0}, upload=473182]: Uploading piece... Dec 28 03:03:14.032403377 (539h34m13.903486422s) [noise=true , address=145.79.227.11:4732 , piece_id=J2QIQF4M37WAQV4I66QV55YHEQGCYFR3P25QZUK7F3VDGSARDTMQ, seg_pos={0 0}, upload=473182]: Uploading piece... Dec 28 03:03:14.033351162 (539h34m13.904434207s) [noise=true , address=77.90.31.13:4627 , piece_id=IRVK2KSSZN5LTLUAG6SYIFJSIFDZFO2CVVCQX5UMCZ7KOLAYDR2A, seg_pos={0 0}, upload=473182]: Uploading piece... Dec 28 03:03:14.033669949 (539h34m13.904752984s) [noise=true , address=77.90.31.11:4556 , piece_id=FCOFTDZI2F4HITY2T5MULXOBMJHUULIW5RRLDDMRGKPQ2YBE6FDQ, seg_pos={0 0}, upload=473182]: Uploading piece... Dec 28 03:03:14.035375316 (539h34m13.906458361s) [noise=true , address=145.79.227.38:4977 , piece_id=RBHFUHMTGN36M5KLW3MM7FMDADFVZ3GHGWNM3GCNKIZAK55CVTFQ, seg_pos={0 0}, upload=473182]: Uploading piece... Dec 28 03:03:14.036848591 (539h34m13.907931626s) [noise=true , address=145.79.227.51:4411 , piece_id=L5GI3KA23ZYBYM2U6G3PAYRTDRVY4R3OL6BI3LUJOSZHJBXGMSJQ, seg_pos={0 0}, upload=473182]: Uploading piece... Dec 28 03:03:14.038263215 (539h34m13.90934626s) [noise=true , address=77.90.31.10:4518 , piece_id=5T3KM6L5GZR5GZVZJJ3LLUEERBCLY4JN4XOVMSF5LWUURQ3W7IRA, seg_pos={0 0}, upload=473182]: Uploading piece... Dec 28 03:03:14.041518133 (539h34m13.912601178s) [noise=true , address=145.79.227.13:4804 , piece_id=IEJN7ALBAKCZJUYPZKQWZYDDE6FFRPBPR477AQZXVAHMOMSI6XNQ, seg_pos={0 0}, upload=473182]: Uploading piece... Dec 28 03:03:15.757833602 (539h34m15.628916637s) [upload=473182]: Next returned nil segment Dec 28 03:03:15.757848050 (539h34m15.628931085s) [upload=473182]: Waiting for error group managing segments... Dec 28 03:03:16.040334431 (539h34m15.911417466s) [noise=true , address=145.79.227.20:4438 , piece_id=77N6MPOL4KI6WYQMHZQB43CVMNQNFE5JYXPN4KRUAFW36PLNBVAA, seg_pos={0 0}, upload=473182]: Done uploading piece. err: <nil> Dec 28 03:03:16.045774991 (539h34m15.916858026s) [noise=true , address=145.79.227.32:4836 , piece_id=IHH62G5XJK2H4RKV4LP3FDOMLMEHIH7IY44PFNUVWSLZGXLUXZQA, seg_pos={0 0}, upload=473182]: Done uploading piece. err: <nil> Dec 28 03:03:16.204620940 (539h34m16.075703985s) [noise=true , address=77.90.31.11:4556 , piece_id=FCOFTDZI2F4HITY2T5MULXOBMJHUULIW5RRLDDMRGKPQ2YBE6FDQ, seg_pos={0 0}, upload=473182]: Done uploading piece. err: <nil> Dec 28 03:03:16.278830096 (539h34m16.149913141s) [noise=true , address=45.91.116.6:4973 , piece_id=JTJCTS3B73M4FDKNXGCMDIP2LS2SJ55SCQGLUCD5ERPNGRO7SLVQ, seg_pos={0 0}, upload=473182]: Done uploading piece. err: <nil> Dec 28 03:03:16.287483965 (539h34m16.158567s) [noise=true , address=77.90.31.10:4518 , piece_id=5T3KM6L5GZR5GZVZJJ3LLUEERBCLY4JN4XOVMSF5LWUURQ3W7IRA, seg_pos={0 0}, upload=473182]: Done uploading piece. err: <nil> Dec 28 03:03:16.334036198 (539h34m16.205119233s) [noise=true , address=77.90.31.13:4627 , piece_id=IRVK2KSSZN5LTLUAG6SYIFJSIFDZFO2CVVCQX5UMCZ7KOLAYDR2A, seg_pos={0 0}, upload=473182]: Done uploading piece. err: <nil> Dec 28 03:03:16.429571093 (539h34m16.300654128s) [noise=true , address=45.91.116.9:4263 , piece_id=3WJ2V75SLYFAVWCGXJGUEYODFNIW4LAUCBW5T5QBN6S54I7SKAPA, seg_pos={0 0}, upload=473182]: Done uploading piece. err: <nil> Dec 28 03:03:17.292122553 (539h34m17.163205598s) [noise=true , address=145.79.227.39:4015 , piece_id=6FBZWOD2GUWSTL7OMYLTJYNKROG5HDTTTYNJJNCSTUKQB2D2TO5Q, seg_pos={0 0}, upload=473182]: Done uploading piece. err: <nil> Dec 28 03:03:17.301014224 (539h34m17.172097279s) [noise=true , address=145.79.227.13:4804 , piece_id=IEJN7ALBAKCZJUYPZKQWZYDDE6FFRPBPR477AQZXVAHMOMSI6XNQ, seg_pos={0 0}, upload=473182]: Done uploading piece. err: <nil> Dec 28 03:03:17.309060566 (539h34m17.180143612s) [noise=true , address=145.79.227.42:4123 , piece_id=DWHWZXC5TIWHYXWDEGXODBRXOGGUAYNYR4KLALWV4W2TNEPATE3A, seg_pos={0 0}, upload=473182]: Done uploading piece. err: <nil> Dec 28 03:03:17.317535033 (539h34m17.188618078s) [noise=true , address=145.79.227.11:4732 , piece_id=J2QIQF4M37WAQV4I66QV55YHEQGCYFR3P25QZUK7F3VDGSARDTMQ, seg_pos={0 0}, upload=473182]: Done uploading piece. err: <nil> Dec 28 03:03:17.323047700 (539h34m17.194130755s) [noise=true , address=145.79.227.38:4977 , piece_id=RBHFUHMTGN36M5KLW3MM7FMDADFVZ3GHGWNM3GCNKIZAK55CVTFQ, seg_pos={0 0}, upload=473182]: Done uploading piece. err: <nil> Dec 28 03:03:17.332247177 (539h34m17.203330222s) [noise=true , address=145.79.227.41:4086 , piece_id=2AVS4WW6OY3F4RF4C7XGSMKKBANTNLENG54NOO6TAOO67EM4R7TQ, seg_pos={0 0}, upload=473182]: Done uploading piece. err: <nil> Dec 28 03:03:17.332259010 (539h34m17.203342055s) [seg_pos={0 0}, upload=473182]: Segment reached optimal threshold of 13 pieces. Dec 28 03:03:17.332433162 (539h34m17.203516207s) [noise=true , address=145.79.227.51:4411 , piece_id=L5GI3KA23ZYBYM2U6G3PAYRTDRVY4R3OL6BI3LUJOSZHJBXGMSJQ, seg_pos={0 0}, upload=473182]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR): protocol: write tcp 188.166.225.170:45020->145.79.227.51:4411: use of closed network connection; write tcp 188.166.225.170:45020->145.79.227.51:4411: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45020->145.79.227.51:4411: use of closed network connection Dec 28 03:03:17.332439113 (539h34m17.203522148s) [noise=true , address=145.79.227.51:4411 , piece_id=L5GI3KA23ZYBYM2U6G3PAYRTDRVY4R3OL6BI3LUJOSZHJBXGMSJQ, seg_pos={0 0}, upload=473182]: Long tail cancellation due to optimal threshold Dec 28 03:03:17.332534845 (539h34m17.20361789s) [noise=true , address=145.79.227.14:4838 , piece_id=FR42ESNX5Q2VHGUVXNMIMYQVXADUS42BBXLQCCVPU2BB7OQFEMPA, seg_pos={0 0}, upload=473182]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: expected piece hash; context canceled; EOF Dec 28 03:03:17.332546697 (539h34m17.203629743s) [noise=true , address=145.79.227.14:4838 , piece_id=FR42ESNX5Q2VHGUVXNMIMYQVXADUS42BBXLQCCVPU2BB7OQFEMPA, seg_pos={0 0}, upload=473182]: Long tail cancellation due to optimal threshold Dec 28 03:03:17.332570092 (539h34m17.203653127s) [noise=true , address=77.90.31.12:4590 , piece_id=HZLWL3B2H6EZC4PI4UZ7LF7EUE7LWRQO75CL6BESW2N6UVVM3XVQ, seg_pos={0 0}, upload=473182]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 28 03:03:17.332575943 (539h34m17.203658978s) [noise=true , address=77.90.31.12:4590 , piece_id=HZLWL3B2H6EZC4PI4UZ7LF7EUE7LWRQO75CL6BESW2N6UVVM3XVQ, seg_pos={0 0}, upload=473182]: Long tail cancellation due to optimal threshold Dec 28 03:03:17.332656036 (539h34m17.203739071s) [seg_pos={0 0}, upload=473182]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 03:03:17.332722282 (539h34m17.203805317s) [upload=473182]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 03:03:17.332727101 (539h34m17.203810146s) [upload=473182]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 03:03:17.629574026 (539h34m17.500657091s) [upload=473182]: Done uploading segments... Dec 28 03:03:17.629599194 (539h34m17.500682229s) [upload=473182]: Upload finished. err: <nil> Duration: 30.003296917s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [........X.....X.............................................] duration: 36.361242374s trace: https://jaeger.dev.storj.io/trace/1822be52d200e5b5 FAIL (2025-12-28 03:03:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1ZAfYZ4wgpZVvatj6wtHona3hTaPcLxAmXC1DT2uKQyo8nLQQ7, Piece ID: 5MMQ44MXRT4XEL4X6WO54WRG2XYMX3AN3FNQ2H2XIUTKBXI3EEPA) context deadline exceeded; piecestore: (Node ID: 124guAM82oibYZsMho67Em6Ke6yEZXwT6LPMLLSZZBLqwoFGUkn, Piece ID: 22WJIPX4SM5NTIAW5ZRM2UKX32CTDBFCXPCCHRRWRCUCOF5CRE5Q) context deadline exceeded; piecestore: (Node ID: 1e6p1TD5CqUyWdZjkrxh2kRqMuyDiMUr9eimfeayDAKwhtBrZ8, Piece ID: XRI5EJFN7YYUG4QKLDGEFT6YAUHCUTTSXF735UDR6LZZBGMAZDJQ) context deadline exceeded ======LOGS====== 5.234848204 (538h55m35.105931239s) [noise=true , address=77.90.31.11:4572 , piece_id=5MMQ44MXRT4XEL4X6WO54WRG2XYMX3AN3FNQ2H2XIUTKBXI3EEPA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.235412057 (538h55m35.106495102s) [noise=true , address=145.79.227.22:4512 , piece_id=D46YLTIKGHG36DQ4YBKBSUDV4GGB4WVDWNOJS27IYYLN6AYZ5QFQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.235820946 (538h55m35.106903981s) [noise=true , address=77.90.31.19:4874 , piece_id=RHZUXUYMJU6RKQGSDZM5WDWIN5SQ7YIRSRATNNKT5GEMU2QH444A, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.236192192 (538h55m35.107275237s) [noise=true , address=77.90.31.16:4760 , piece_id=XRI5EJFN7YYUG4QKLDGEFT6YAUHCUTTSXF735UDR6LZZBGMAZDJQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.236564772 (538h55m35.107647817s) [noise=true , address=145.79.227.40:4062 , piece_id=PW6QWIPQWVUETLC6DJ624G6IB4P6EM4SYB72NAMQ4V6NUKNLUYLQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.236929766 (538h55m35.108012812s) [noise=true , address=64.49.13.139:4480 , piece_id=JAJY3F7IDFFMFKCK2QJV6W74RIESLRKBLXMN27Q3C45JWEPLBZLQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.237916846 (538h55m35.108999881s) [noise=true , address=145.79.227.34:4932 , piece_id=UPXSCQ45QDGPKNJXRFJ33EAA3QENVHRVQARU5TFJTEWICZNR253Q, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.238852948 (538h55m35.109935983s) [noise=true , address=45.91.116.12:4257 , piece_id=ZI67LIAHZW4G357GWKDPTB5SMXYGNXZTTTDMNAETDS2B6SWVDAEQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239152910 (538h55m35.110235945s) [noise=true , address=145.79.227.44:4222 , piece_id=CZ2CS62CTHQFAIE3S3JO2N3VJ4XIR5YJ2LMRVLLX7I4VM2VZVQVA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239409969 (538h55m35.110493004s) [noise=true , address=145.79.227.30:4773 , piece_id=KUGJXZCNMUE7WSK3VLY4AUBSOT4H6IY4XB4NCBGAGCNUG7F7IODQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239681136 (538h55m35.110764171s) [noise=true , address=77.90.31.13:4635 , piece_id=IIOY5X2W3XW4C2OFEBRCABMYFBI24RRTYNUSKN6KJU7CTUW4KONQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.239954676 (538h55m35.111037711s) [noise=true , address=45.91.116.9:4281 , piece_id=JZLDW777CBX2BP7MLOQV4NFP3H4XJVCNQXTHNSUWPVDOJLPZK36A, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.240189914 (538h55m35.111272949s) [noise=true , address=145.79.227.35:4512 , piece_id=3E3VCCC45XVT5FHFDKXSTGGAGYBEML3BKU6JJTT33LCD2YMOMBZQ, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.240449147 (538h55m35.111532182s) [noise=true , address=45.91.116.8:4064 , piece_id=22WJIPX4SM5NTIAW5ZRM2UKX32CTDBFCXPCCHRRWRCUCOF5CRE5Q, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.240639640 (538h55m35.111722675s) [noise=true , address=145.79.227.32:4853 , piece_id=MBNCFDBQ4SZZXV4F7J5TRMOUYUQ5IPKQCGYGGM5SOIFJ6ZYCAUYA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.241137798 (538h55m35.112220833s) [noise=true , address=77.90.31.16:4751 , piece_id=N3Z2LLMCIA2F42VXZA7IDRU4XEKGOICH6TRVERODU76MFFE4QSHA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:35.241383797 (538h55m35.112466842s) [noise=true , address=77.90.31.18:4836 , piece_id=BSVEFI52EKN44CMINFUSZBNOH53IW5TXLVTF5IH6ZUTNII6RRLVA, seg_pos={0 0}, upload=472631]: Uploading piece... Dec 28 02:24:36.592880816 (538h55m36.463963871s) [upload=472631]: Next returned nil segment Dec 28 02:24:36.592901625 (538h55m36.46398467s) [upload=472631]: Waiting for error group managing segments... Dec 28 02:24:36.946933741 (538h55m36.818016786s) [noise=true , address=64.49.13.139:4480 , piece_id=JAJY3F7IDFFMFKCK2QJV6W74RIESLRKBLXMN27Q3C45JWEPLBZLQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.128189692 (538h55m36.999272737s) [noise=true , address=145.79.227.35:4512 , piece_id=3E3VCCC45XVT5FHFDKXSTGGAGYBEML3BKU6JJTT33LCD2YMOMBZQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.160760084 (538h55m37.031843119s) [noise=true , address=145.79.227.34:4932 , piece_id=UPXSCQ45QDGPKNJXRFJ33EAA3QENVHRVQARU5TFJTEWICZNR253Q, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.288838207 (538h55m37.159921252s) [noise=true , address=77.90.31.11:4572 , piece_id=5MMQ44MXRT4XEL4X6WO54WRG2XYMX3AN3FNQ2H2XIUTKBXI3EEPA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.319996310 (538h55m37.191079345s) [noise=true , address=145.79.227.22:4512 , piece_id=D46YLTIKGHG36DQ4YBKBSUDV4GGB4WVDWNOJS27IYYLN6AYZ5QFQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.320513273 (538h55m37.191596319s) [noise=true , address=77.90.31.18:4836 , piece_id=BSVEFI52EKN44CMINFUSZBNOH53IW5TXLVTF5IH6ZUTNII6RRLVA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.336522926 (538h55m37.207605971s) [noise=true , address=145.79.227.32:4853 , piece_id=MBNCFDBQ4SZZXV4F7J5TRMOUYUQ5IPKQCGYGGM5SOIFJ6ZYCAUYA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.384868031 (538h55m37.255951066s) [noise=true , address=77.90.31.16:4751 , piece_id=N3Z2LLMCIA2F42VXZA7IDRU4XEKGOICH6TRVERODU76MFFE4QSHA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.453563554 (538h55m37.324646599s) [noise=true , address=45.91.116.8:4064 , piece_id=22WJIPX4SM5NTIAW5ZRM2UKX32CTDBFCXPCCHRRWRCUCOF5CRE5Q, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.479454839 (538h55m37.350537884s) [noise=true , address=45.91.116.12:4257 , piece_id=ZI67LIAHZW4G357GWKDPTB5SMXYGNXZTTTDMNAETDS2B6SWVDAEQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.659710174 (538h55m37.530793219s) [noise=true , address=77.90.31.13:4635 , piece_id=IIOY5X2W3XW4C2OFEBRCABMYFBI24RRTYNUSKN6KJU7CTUW4KONQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.715039128 (538h55m37.586122163s) [noise=true , address=77.90.31.16:4760 , piece_id=XRI5EJFN7YYUG4QKLDGEFT6YAUHCUTTSXF735UDR6LZZBGMAZDJQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:37.977473406 (538h55m37.848556441s) [noise=true , address=77.90.31.19:4874 , piece_id=RHZUXUYMJU6RKQGSDZM5WDWIN5SQ7YIRSRATNNKT5GEMU2QH444A, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:38.016302745 (538h55m37.88738578s) [noise=true , address=145.79.227.30:4773 , piece_id=KUGJXZCNMUE7WSK3VLY4AUBSOT4H6IY4XB4NCBGAGCNUG7F7IODQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: <nil> Dec 28 02:24:38.016326982 (538h55m37.887410027s) [seg_pos={0 0}, upload=472631]: Segment reached optimal threshold of 14 pieces. Dec 28 02:24:38.016585353 (538h55m37.887668398s) [noise=true , address=145.79.227.30:4766 , piece_id=67M2DMPAVWIQKQSWUAG73D4E4NRRSG4XEVPOUVIJPEQOUV7A6VVA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1k6zdamuKPkBdzyNZNWx4FCmaTkv33PchcwvTRR8bgcETBs4K7): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016594450 (538h55m37.887677495s) [noise=true , address=145.79.227.30:4766 , piece_id=67M2DMPAVWIQKQSWUAG73D4E4NRRSG4XEVPOUVIJPEQOUV7A6VVA, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016627093 (538h55m37.887710148s) [noise=true , address=45.91.116.9:4281 , piece_id=JZLDW777CBX2BP7MLOQV4NFP3H4XJVCNQXTHNSUWPVDOJLPZK36A, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gxf7DpJZthyR6f1PExX56tJavjhPw9Gxb4h8mtmeYKW3t3tS1): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016635258 (538h55m37.887718293s) [noise=true , address=45.91.116.9:4281 , piece_id=JZLDW777CBX2BP7MLOQV4NFP3H4XJVCNQXTHNSUWPVDOJLPZK36A, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016871298 (538h55m37.887954363s) [noise=true , address=145.79.227.44:4222 , piece_id=CZ2CS62CTHQFAIE3S3JO2N3VJ4XIR5YJ2LMRVLLX7I4VM2VZVQVA, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZKW35KvrsoYRefbYeUFZo8bh4XZYycNKGEvTFjzibezjXgj3e): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016880124 (538h55m37.887963169s) [noise=true , address=145.79.227.44:4222 , piece_id=CZ2CS62CTHQFAIE3S3JO2N3VJ4XIR5YJ2LMRVLLX7I4VM2VZVQVA, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016934839 (538h55m37.888017884s) [noise=true , address=145.79.227.40:4062 , piece_id=PW6QWIPQWVUETLC6DJ624G6IB4P6EM4SYB72NAMQ4V6NUKNLUYLQ, seg_pos={0 0}, upload=472631]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb): protocol: expected piece hash; context canceled; EOF Dec 28 02:24:38.016943425 (538h55m37.88802649s) [noise=true , address=145.79.227.40:4062 , piece_id=PW6QWIPQWVUETLC6DJ624G6IB4P6EM4SYB72NAMQ4V6NUKNLUYLQ, seg_pos={0 0}, upload=472631]: Long tail cancellation due to optimal threshold Dec 28 02:24:38.016968252 (538h55m37.888051297s) [seg_pos={0 0}, upload=472631]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 02:24:38.017067822 (538h55m37.888150867s) [upload=472631]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 02:24:38.017074244 (538h55m37.888157289s) [upload=472631]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 02:24:38.304306667 (538h55m38.175389712s) [upload=472631]: Done uploading segments... Dec 28 02:24:38.304329641 (538h55m38.175412686s) [upload=472631]: Upload finished. err: <nil> Duration: 36.361284715s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [........X................X....X............X........X....X..] duration: 5.000587852s trace: https://jaeger.dev.storj.io/trace/25c8812909e078aa FAIL (2025-12-28 03:02:54 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000613421s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)