Orbiter History

Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure

Orbiter Check History

Showing 50 of 903537 reports (Page 11288 of 18071)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 31 out of 60 is failed (threshold=3) [.XX..X...XX.X.XX...X.X.X.XXX.X.X.X.XX..X.XXX.XX..X.XX..XXX..] duration: 5.681363478s trace: https://jaeger.dev.storj.io/trace/41dd262f22c0a8cd FAIL (2025-12-29 13:23:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: 2ZXKJFGCZ3JWYPXA5ZJXRJCOKBVK7KVO2TYQDHXBRAGAYMY6LZEA) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: DYK5ZC5IHVQKR4M6KOMBAXELT2BFID35YRH2LPPN5URMLPG435QQ) context deadline exceeded ======LOGS====== ts... Dec 29 13:22:40.582068626 (573h53m40.453151661s) [upload=503124]: Got next segment. Inline: false Dec 29 13:22:40.582101098 (573h53m40.453184143s) [upload=503124]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 13:22:40.582105416 (573h53m40.453188451s) [upload=503124]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 13:22:40.830399264 (573h53m40.701482309s) [seg_pos={0 0}, upload=503124]: Begin upload segment... Dec 29 13:22:40.830455933 (573h53m40.701538978s) [seg_pos={0 0}, upload=503124]: Done begin upload segment. Dec 29 13:22:40.830469127 (573h53m40.701552162s) [noise=true , address=45.91.116.10:4082 , piece_id=M4UR7KUJZJT53WYAROF6EOPHAXU6WWXE247ZZMYWHMMD3M3SVOSQ, seg_pos={0 0}, upload=503124]: Uploading piece... Dec 29 13:22:40.830542828 (573h53m40.701625863s) [noise=true , address=145.79.227.52:4447 , piece_id=RACK4VLSCFEX52APOHS5QKEIQKIZTGGHXVPQTAE5VQMSGGKMWC5Q, seg_pos={0 0}, upload=503124]: Uploading piece... Dec 29 13:22:40.830964520 (573h53m40.702047556s) [noise=true , address=145.79.227.14:4840 , piece_id=5TLCNUWDIQ72WIFPVWAZPIVNV6JCJRHSCOEPJW6D3JH2SXBSQ7KQ, seg_pos={0 0}, upload=503124]: Uploading piece... Dec 29 13:22:40.831322132 (573h53m40.702405177s) [noise=true , address=145.79.227.41:4087 , piece_id=MEV3CUDWUR5W3CIKX4D5SMKDBZ6RNC6B2HVOHC2MMUODASM7ZJBQ, seg_pos={0 0}, upload=503124]: Uploading piece... Dec 29 13:22:40.831727844 (573h53m40.702810879s) [noise=true , address=45.91.116.6:4975 , piece_id=PNL7J5CHPNE2RQRHV6R5XTBHHJT3OJC24YATO4DXAZVX5533HNPQ, seg_pos={0 0}, upload=503124]: Uploading piece... Dec 29 13:22:40.832141382 (573h53m40.703224427s) [noise=true , address=145.79.227.3:4150 , piece_id=DZ6XDLLNJSXWZ2O2LBNWR7M2S2GJ3CXXJK7YWCYT7Y5PDIFJYF7A, seg_pos={0 0}, upload=503124]: Uploading piece... Dec 29 13:22:40.832533829 (573h53m40.703616874s) [noise=true , address=145.79.227.11:4731 , piece_id=SCEAXXRIPST2OUFABUKFNQ5VITBMNQSBO3B2ZLIRRJOVPGK5Q45Q, seg_pos={0 0}, upload=503124]: Uploading piece... Dec 29 13:22:40.832979267 (573h53m40.704062312s) [noise=true , address=145.79.227.26:4655 , piece_id=Z7UZCYH2CHRIXZ4IFLIBAAKH4QZKMI43RQW72ITJRAB5HLQ2HYWA, seg_pos={0 0}, upload=503124]: Uploading piece... Dec 29 13:22:40.834762453 (573h53m40.705845488s) [noise=true , address=45.91.116.9:4261 , piece_id=DYK5ZC5IHVQKR4M6KOMBAXELT2BFID35YRH2LPPN5URMLPG435QQ, seg_pos={0 0}, upload=503124]: Uploading piece... Dec 29 13:22:40.836347671 (573h53m40.707430716s) [noise=true , address=45.91.116.14:4189 , piece_id=5SB546DUVS4JMPZKVWAGRLORU6UBXALX5YQPRW62TQUN5LEOKGTA, seg_pos={0 0}, upload=503124]: Uploading piece... Dec 29 13:22:40.837968648 (573h53m40.709051683s) [noise=true , address=145.79.227.2:4113 , piece_id=X4ZJCYAQUGPEOECG2S42B7W5NP4KUFAZZQ7A4YVUNU5RY7GRP7WA, seg_pos={0 0}, upload=503124]: Uploading piece... Dec 29 13:22:40.839659628 (573h53m40.710742663s) [noise=true , address=77.90.31.19:4844 , piece_id=2ZXKJFGCZ3JWYPXA5ZJXRJCOKBVK7KVO2TYQDHXBRAGAYMY6LZEA, seg_pos={0 0}, upload=503124]: Uploading piece... Dec 29 13:22:40.839922018 (573h53m40.711005063s) [noise=true , address=145.79.227.47:4265 , piece_id=XUHDCM4WYBFQ4T27DSSSR27OBVIGPO4KEYGXXFI3FEZV57UGDZDQ, seg_pos={0 0}, upload=503124]: Uploading piece... Dec 29 13:22:40.841141490 (573h53m40.712224525s) [noise=true , address=145.79.227.48:4303 , piece_id=DX3IG4Y4NKDQAB232LPH2EW64DNGNAAKZLGHH6BDPDYL4CVXY3VQ, seg_pos={0 0}, upload=503124]: Uploading piece... Dec 29 13:22:40.842461994 (573h53m40.713545029s) [noise=true , address=145.79.227.15:4874 , piece_id=M32DVD2PIAMPYWYVTVXTH55DGT7MWCTSC4QZFF2DSL3SF5IBQOXA, seg_pos={0 0}, upload=503124]: Uploading piece... Dec 29 13:22:40.842989569 (573h53m40.714072604s) [noise=true , address=145.79.227.39:4013 , piece_id=OWX27MIHCB3JBS4PSULL3P2PQEDY54V7PPEFPPN3N52TUZSYTX4A, seg_pos={0 0}, upload=503124]: Uploading piece... Dec 29 13:22:42.550561936 (573h53m42.421644971s) [upload=503124]: Next returned nil segment Dec 29 13:22:42.550578698 (573h53m42.421661743s) [upload=503124]: Waiting for error group managing segments... Dec 29 13:22:42.827721168 (573h53m42.698804203s) [noise=true , address=145.79.227.26:4655 , piece_id=Z7UZCYH2CHRIXZ4IFLIBAAKH4QZKMI43RQW72ITJRAB5HLQ2HYWA, seg_pos={0 0}, upload=503124]: Done uploading piece. err: <nil> Dec 29 13:22:42.830747090 (573h53m42.701830145s) [noise=true , address=145.79.227.52:4447 , piece_id=RACK4VLSCFEX52APOHS5QKEIQKIZTGGHXVPQTAE5VQMSGGKMWC5Q, seg_pos={0 0}, upload=503124]: Done uploading piece. err: <nil> Dec 29 13:22:42.831647725 (573h53m42.70273076s) [noise=true , address=145.79.227.41:4087 , piece_id=MEV3CUDWUR5W3CIKX4D5SMKDBZ6RNC6B2HVOHC2MMUODASM7ZJBQ, seg_pos={0 0}, upload=503124]: Done uploading piece. err: <nil> Dec 29 13:22:42.833222834 (573h53m42.704305869s) [noise=true , address=145.79.227.39:4013 , piece_id=OWX27MIHCB3JBS4PSULL3P2PQEDY54V7PPEFPPN3N52TUZSYTX4A, seg_pos={0 0}, upload=503124]: Done uploading piece. err: <nil> Dec 29 13:22:42.836561621 (573h53m42.707644666s) [noise=true , address=145.79.227.48:4303 , piece_id=DX3IG4Y4NKDQAB232LPH2EW64DNGNAAKZLGHH6BDPDYL4CVXY3VQ, seg_pos={0 0}, upload=503124]: Done uploading piece. err: <nil> Dec 29 13:22:42.841783204 (573h53m42.712866239s) [noise=true , address=145.79.227.47:4265 , piece_id=XUHDCM4WYBFQ4T27DSSSR27OBVIGPO4KEYGXXFI3FEZV57UGDZDQ, seg_pos={0 0}, upload=503124]: Done uploading piece. err: <nil> Dec 29 13:22:42.843113217 (573h53m42.714196252s) [noise=true , address=145.79.227.3:4150 , piece_id=DZ6XDLLNJSXWZ2O2LBNWR7M2S2GJ3CXXJK7YWCYT7Y5PDIFJYF7A, seg_pos={0 0}, upload=503124]: Done uploading piece. err: <nil> Dec 29 13:22:42.845625099 (573h53m42.716708134s) [noise=true , address=145.79.227.15:4874 , piece_id=M32DVD2PIAMPYWYVTVXTH55DGT7MWCTSC4QZFF2DSL3SF5IBQOXA, seg_pos={0 0}, upload=503124]: Done uploading piece. err: <nil> Dec 29 13:22:42.846786341 (573h53m42.717869376s) [noise=true , address=145.79.227.2:4113 , piece_id=X4ZJCYAQUGPEOECG2S42B7W5NP4KUFAZZQ7A4YVUNU5RY7GRP7WA, seg_pos={0 0}, upload=503124]: Done uploading piece. err: <nil> Dec 29 13:22:42.847257728 (573h53m42.718340773s) [noise=true , address=145.79.227.14:4840 , piece_id=5TLCNUWDIQ72WIFPVWAZPIVNV6JCJRHSCOEPJW6D3JH2SXBSQ7KQ, seg_pos={0 0}, upload=503124]: Done uploading piece. err: <nil> Dec 29 13:22:42.847691455 (573h53m42.7187745s) [noise=true , address=145.79.227.11:4731 , piece_id=SCEAXXRIPST2OUFABUKFNQ5VITBMNQSBO3B2ZLIRRJOVPGK5Q45Q, seg_pos={0 0}, upload=503124]: Done uploading piece. err: <nil> Dec 29 13:22:43.115180433 (573h53m42.986263478s) [noise=true , address=45.91.116.9:4261 , piece_id=DYK5ZC5IHVQKR4M6KOMBAXELT2BFID35YRH2LPPN5URMLPG435QQ, seg_pos={0 0}, upload=503124]: Done uploading piece. err: <nil> Dec 29 13:22:43.181275907 (573h53m43.052358942s) [noise=true , address=77.90.31.19:4844 , piece_id=2ZXKJFGCZ3JWYPXA5ZJXRJCOKBVK7KVO2TYQDHXBRAGAYMY6LZEA, seg_pos={0 0}, upload=503124]: Done uploading piece. err: <nil> Dec 29 13:22:43.181295685 (573h53m43.052378731s) [seg_pos={0 0}, upload=503124]: Segment reached optimal threshold of 13 pieces. Dec 29 13:22:43.181488944 (573h53m43.052571989s) [noise=true , address=45.91.116.6:4975 , piece_id=PNL7J5CHPNE2RQRHV6R5XTBHHJT3OJC24YATO4DXAZVX5533HNPQ, seg_pos={0 0}, upload=503124]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 29 13:22:43.181496568 (573h53m43.052579623s) [noise=true , address=45.91.116.6:4975 , piece_id=PNL7J5CHPNE2RQRHV6R5XTBHHJT3OJC24YATO4DXAZVX5533HNPQ, seg_pos={0 0}, upload=503124]: Long tail cancellation due to optimal threshold Dec 29 13:22:43.181740974 (573h53m43.052824019s) [noise=true , address=45.91.116.14:4189 , piece_id=5SB546DUVS4JMPZKVWAGRLORU6UBXALX5YQPRW62TQUN5LEOKGTA, seg_pos={0 0}, upload=503124]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: write tcp 188.166.225.170:36668->45.91.116.14:4189: use of closed network connection; write tcp 188.166.225.170:36668->45.91.116.14:4189: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36668->45.91.116.14:4189: use of closed network connection Dec 29 13:22:43.181747666 (573h53m43.052830701s) [noise=true , address=45.91.116.14:4189 , piece_id=5SB546DUVS4JMPZKVWAGRLORU6UBXALX5YQPRW62TQUN5LEOKGTA, seg_pos={0 0}, upload=503124]: Long tail cancellation due to optimal threshold Dec 29 13:22:43.181896720 (573h53m43.052979785s) [noise=true , address=45.91.116.10:4082 , piece_id=M4UR7KUJZJT53WYAROF6EOPHAXU6WWXE247ZZMYWHMMD3M3SVOSQ, seg_pos={0 0}, upload=503124]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: write tcp 188.166.225.170:42702->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:42702->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42702->45.91.116.10:4082: use of closed network connection Dec 29 13:22:43.181944431 (573h53m43.053027476s) [noise=true , address=45.91.116.10:4082 , piece_id=M4UR7KUJZJT53WYAROF6EOPHAXU6WWXE247ZZMYWHMMD3M3SVOSQ, seg_pos={0 0}, upload=503124]: Long tail cancellation due to optimal threshold Dec 29 13:22:43.181983956 (573h53m43.053066991s) [seg_pos={0 0}, upload=503124]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 13:22:43.182260503 (573h53m43.053343548s) [upload=503124]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 13:22:43.182267436 (573h53m43.053350471s) [upload=503124]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 13:22:43.485049817 (573h53m43.356132862s) [upload=503124]: Done uploading segments... Dec 29 13:22:43.485074533 (573h53m43.356157578s) [upload=503124]: Upload finished. err: <nil> Duration: 5.681387985s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [..................X.....X.....................XX............] duration: 48.952755682s trace: https://jaeger.dev.storj.io/trace/77d26141ea11d228 FAIL (2025-12-29 13:23:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12J2weU7Bmsr3RdQQ9QMuadT96XReBkyxTeaLJ2SEqVGGFX1PWu, Piece ID: JITX25FNZ4JIKYXQY345UX3SQUCPDWJ25NJQZ2TXGFRLVCRYXUOQ) context deadline exceeded; piecestore: (Node ID: 125sm6iHV94mJFLWrKeyRMH4b1qSMo32nDuW8uQr71xUL7Cema2, Piece ID: INTVU2EYTU3DIM6WQFQEGO6IEYH3YFP5U2VEN7A5KPTOKU2DWOOA) context deadline exceeded; piecestore: (Node ID: 12i57FF97oEddYLZpNxSq5or9WLEquL8ZW1cVUuW4AThBbDqcCL, Piece ID: Q35LXGPTOAG3R2UL3JLL7DNHHD4PSMEULYW6G3SPXKNVFZ5VTNRA) context deadline exceeded ======LOGS====== Q2LO7MUXLHE5YET7BHLLA, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.692912334 (573h43m3.563995379s) [noise=true , address=77.90.31.16:4759 , piece_id=ARC4HRP57U5VQXZTT63QYLXBPZURI5VMJBIQZOFKGLJBQ3K3KGLQ, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.693322595 (573h43m3.56440564s) [noise=true , address=77.90.31.15:4711 , piece_id=JITX25FNZ4JIKYXQY345UX3SQUCPDWJ25NJQZ2TXGFRLVCRYXUOQ, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.693739649 (573h43m3.564822694s) [noise=true , address=45.91.116.6:4981 , piece_id=INTVU2EYTU3DIM6WQFQEGO6IEYH3YFP5U2VEN7A5KPTOKU2DWOOA, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.694315887 (573h43m3.565398942s) [noise=true , address=145.79.227.4:4220 , piece_id=SHHJ7EW7DTNOZXVBP6A64VIX7P6AQN62TBCUT3KTH4MIB6UXY3BA, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.694765222 (573h43m3.565848267s) [noise=true , address=145.79.227.29:4952 , piece_id=DKYDB2R4DNCGH6T3SLRR2MOZZUPVZZVRL2BKGYOBYXJJVPSZEJ4Q, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.695202545 (573h43m3.56628559s) [noise=true , address=145.79.227.40:4058 , piece_id=KRHH2QZKSDPWTSSKYDVDWGKFJD3LPGP6U7KI2PPJ5T3TXYIUGJWA, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.695687528 (573h43m3.566770584s) [noise=true , address=77.90.31.19:4857 , piece_id=Q35LXGPTOAG3R2UL3JLL7DNHHD4PSMEULYW6G3SPXKNVFZ5VTNRA, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.696135221 (573h43m3.567218276s) [noise=true , address=45.91.116.8:4051 , piece_id=GDJFLQCHUF7BFQ7BVSNTVJ3TYZHYVOO6YYMO3KUIUSIEPAF4SFDQ, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.696703733 (573h43m3.567786778s) [noise=true , address=64.49.13.170:4720 , piece_id=HF6X3OT2EPPDE3DWAPWABVZUR3ERDBWXHG3NGNKEWZAD7G5AN4TA, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.698418298 (573h43m3.569501343s) [noise=true , address=145.79.227.36:4520 , piece_id=MYJSXUH563JFUW5Q7YUIWL7466G6ASY5WXR3T2PGUFFCTRTOBYJA, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.700198829 (573h43m3.571281874s) [noise=true , address=77.90.31.18:4821 , piece_id=LO5W47FHYOVGY5Q6BH5L6KX77HES3XRAFUTPI45HYPJZDO3YHLCQ, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.701993145 (573h43m3.57307619s) [noise=true , address=193.149.164.143:4699 , piece_id=UDFDCYO25MOJBTMDE6F2WILXE3HLOB75UJZ4LW7ZMNSCOBJI6UJA, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.703884687 (573h43m3.574967732s) [noise=true , address=145.79.227.33:4893 , piece_id=XGLX3DFNAY4F4GPGCGZ4ONDJ7NK62SAKX3PBRWHWI74EF3ZFMUSA, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.705687109 (573h43m3.576770154s) [noise=true , address=45.91.116.10:4094 , piece_id=6UUIZNTESBBWJMDQ5ZTCHU7S6YKM4GDKO4ZAC6T2FUTPVGFFDS3A, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.707557651 (573h43m3.578640696s) [noise=true , address=145.79.227.43:4164 , piece_id=J6Q4ZDOIIIBA2FXGZZTD6AYS7KGAWSRXXUBAPDPQWYKNU5GNXBBQ, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:05.168247719 (573h43m5.039330754s) [upload=502975]: Next returned nil segment Dec 29 13:12:05.168259962 (573h43m5.039343007s) [upload=502975]: Waiting for error group managing segments... Dec 29 13:12:05.558873127 (573h43m5.429956162s) [noise=true , address=145.79.227.4:4220 , piece_id=SHHJ7EW7DTNOZXVBP6A64VIX7P6AQN62TBCUT3KTH4MIB6UXY3BA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:05.563024312 (573h43m5.434107347s) [noise=true , address=64.49.13.170:4720 , piece_id=HF6X3OT2EPPDE3DWAPWABVZUR3ERDBWXHG3NGNKEWZAD7G5AN4TA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:05.564115719 (573h43m5.435198755s) [noise=true , address=145.79.227.36:4520 , piece_id=MYJSXUH563JFUW5Q7YUIWL7466G6ASY5WXR3T2PGUFFCTRTOBYJA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:05.593873402 (573h43m5.464956447s) [noise=true , address=145.79.227.29:4952 , piece_id=DKYDB2R4DNCGH6T3SLRR2MOZZUPVZZVRL2BKGYOBYXJJVPSZEJ4Q, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:05.898194604 (573h43m5.769277639s) [noise=true , address=45.91.116.10:4094 , piece_id=6UUIZNTESBBWJMDQ5ZTCHU7S6YKM4GDKO4ZAC6T2FUTPVGFFDS3A, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:05.912845292 (573h43m5.783928337s) [noise=true , address=77.90.31.19:4857 , piece_id=Q35LXGPTOAG3R2UL3JLL7DNHHD4PSMEULYW6G3SPXKNVFZ5VTNRA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:06.426532207 (573h43m6.297615252s) [noise=true , address=193.149.164.143:4699 , piece_id=UDFDCYO25MOJBTMDE6F2WILXE3HLOB75UJZ4LW7ZMNSCOBJI6UJA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:06.449589562 (573h43m6.320672607s) [noise=true , address=77.90.31.15:4711 , piece_id=JITX25FNZ4JIKYXQY345UX3SQUCPDWJ25NJQZ2TXGFRLVCRYXUOQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:06.453186841 (573h43m6.324269876s) [noise=true , address=145.79.227.40:4058 , piece_id=KRHH2QZKSDPWTSSKYDVDWGKFJD3LPGP6U7KI2PPJ5T3TXYIUGJWA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:06.510769276 (573h43m6.381852321s) [noise=true , address=145.79.227.43:4164 , piece_id=J6Q4ZDOIIIBA2FXGZZTD6AYS7KGAWSRXXUBAPDPQWYKNU5GNXBBQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:06.518868879 (573h43m6.389951924s) [noise=true , address=145.79.227.33:4893 , piece_id=XGLX3DFNAY4F4GPGCGZ4ONDJ7NK62SAKX3PBRWHWI74EF3ZFMUSA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:06.519238043 (573h43m6.390321098s) [noise=true , address=45.91.116.6:4981 , piece_id=INTVU2EYTU3DIM6WQFQEGO6IEYH3YFP5U2VEN7A5KPTOKU2DWOOA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:06.523211328 (573h43m6.394294373s) [noise=true , address=77.90.31.16:4759 , piece_id=ARC4HRP57U5VQXZTT63QYLXBPZURI5VMJBIQZOFKGLJBQ3K3KGLQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:06.767869783 (573h43m6.638952818s) [noise=true , address=145.79.227.34:4924 , piece_id=MAQXQTAGXQ7AEPUCESUSUGMHSDMMXS7HHKAEA7B3HW5X66PDUF5A, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:06.767889431 (573h43m6.638972476s) [seg_pos={0 0}, upload=502975]: Segment reached optimal threshold of 14 pieces. Dec 29 13:12:06.768102116 (573h43m6.639185161s) [noise=true , address=45.91.116.8:4051 , piece_id=GDJFLQCHUF7BFQ7BVSNTVJ3TYZHYVOO6YYMO3KUIUSIEPAF4SFDQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1preT33ei4yVXXYVLHhaUbHbiyL14oHFEiof6cCWPig83W6CT3): protocol: expected piece hash; context canceled; EOF Dec 29 13:12:06.768108257 (573h43m6.639191302s) [noise=true , address=45.91.116.8:4051 , piece_id=GDJFLQCHUF7BFQ7BVSNTVJ3TYZHYVOO6YYMO3KUIUSIEPAF4SFDQ, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold Dec 29 13:12:06.768219208 (573h43m6.639302254s) [noise=true , address=77.90.31.18:4821 , piece_id=LO5W47FHYOVGY5Q6BH5L6KX77HES3XRAFUTPI45HYPJZDO3YHLCQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH): protocol: expected piece hash; context canceled; EOF Dec 29 13:12:06.768224789 (573h43m6.639307824s) [noise=true , address=77.90.31.18:4821 , piece_id=LO5W47FHYOVGY5Q6BH5L6KX77HES3XRAFUTPI45HYPJZDO3YHLCQ, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold Dec 29 13:12:06.768384213 (573h43m6.639467258s) [noise=true , address=64.49.13.169:4711 , piece_id=4F4T4YKXMT7NF2C5KBUGJHQJGZL4VCKNKYB266PCVI3UACHJXZ5Q, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:48802->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:48802->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48802->64.49.13.169:4711: use of closed network connection Dec 29 13:12:06.768399261 (573h43m6.639482296s) [noise=true , address=64.49.13.169:4711 , piece_id=4F4T4YKXMT7NF2C5KBUGJHQJGZL4VCKNKYB266PCVI3UACHJXZ5Q, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold Dec 29 13:12:06.768732636 (573h43m6.639815671s) [noise=true , address=64.49.13.169:4709 , piece_id=TBHHX5ES2A6NQRGP25HIC7J7IBWLOJGQ2LO7MUXLHE5YET7BHLLA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:35310->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:35310->64.49.13.169:4709: use of closed network connection Dec 29 13:12:06.768745460 (573h43m6.639828505s) [noise=true , address=64.49.13.169:4709 , piece_id=TBHHX5ES2A6NQRGP25HIC7J7IBWLOJGQ2LO7MUXLHE5YET7BHLLA, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold Dec 29 13:12:06.768784586 (573h43m6.639867631s) [seg_pos={0 0}, upload=502975]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 13:12:06.768880217 (573h43m6.639963252s) [upload=502975]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 13:12:06.768884906 (573h43m6.639967941s) [upload=502975]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 13:12:07.067188330 (573h43m6.938271365s) [upload=502975]: Done uploading segments... Dec 29 13:12:07.067251099 (573h43m6.938334134s) [upload=502975]: Upload finished. err: <nil> Duration: 48.952781031s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [......X......................X....X.X....X.X................] duration: 15.439987295s trace: https://jaeger.dev.storj.io/trace/312ddb07b2dee82a FAIL (2025-12-29 13:23:45 UTC) orbiter warning: context deadline exceeded Duration: 15.440015918s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 31 out of 60 is failed (threshold=3) [..XX..X...XX.X.XX...X.X.X.XXX.X.X.X.XX..X.XXX.XX..X.XX..XXX.] duration: 6.101235217s trace: https://jaeger.dev.storj.io/trace/41dd262f22c0a8cd FAIL (2025-12-29 13:23:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: 2ZXKJFGCZ3JWYPXA5ZJXRJCOKBVK7KVO2TYQDHXBRAGAYMY6LZEA) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: DYK5ZC5IHVQKR4M6KOMBAXELT2BFID35YRH2LPPN5URMLPG435QQ) context deadline exceeded ======LOGS====== ts... Dec 29 13:22:40.582068626 (573h53m40.453151661s) [upload=503124]: Got next segment. Inline: false Dec 29 13:22:40.582101098 (573h53m40.453184143s) [upload=503124]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 13:22:40.582105416 (573h53m40.453188451s) [upload=503124]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 13:22:40.830399264 (573h53m40.701482309s) [seg_pos={0 0}, upload=503124]: Begin upload segment... Dec 29 13:22:40.830455933 (573h53m40.701538978s) [seg_pos={0 0}, upload=503124]: Done begin upload segment. Dec 29 13:22:40.830469127 (573h53m40.701552162s) [noise=true , address=45.91.116.10:4082 , piece_id=M4UR7KUJZJT53WYAROF6EOPHAXU6WWXE247ZZMYWHMMD3M3SVOSQ, seg_pos={0 0}, upload=503124]: Uploading piece... Dec 29 13:22:40.830542828 (573h53m40.701625863s) [noise=true , address=145.79.227.52:4447 , piece_id=RACK4VLSCFEX52APOHS5QKEIQKIZTGGHXVPQTAE5VQMSGGKMWC5Q, seg_pos={0 0}, upload=503124]: Uploading piece... Dec 29 13:22:40.830964520 (573h53m40.702047556s) [noise=true , address=145.79.227.14:4840 , piece_id=5TLCNUWDIQ72WIFPVWAZPIVNV6JCJRHSCOEPJW6D3JH2SXBSQ7KQ, seg_pos={0 0}, upload=503124]: Uploading piece... Dec 29 13:22:40.831322132 (573h53m40.702405177s) [noise=true , address=145.79.227.41:4087 , piece_id=MEV3CUDWUR5W3CIKX4D5SMKDBZ6RNC6B2HVOHC2MMUODASM7ZJBQ, seg_pos={0 0}, upload=503124]: Uploading piece... Dec 29 13:22:40.831727844 (573h53m40.702810879s) [noise=true , address=45.91.116.6:4975 , piece_id=PNL7J5CHPNE2RQRHV6R5XTBHHJT3OJC24YATO4DXAZVX5533HNPQ, seg_pos={0 0}, upload=503124]: Uploading piece... Dec 29 13:22:40.832141382 (573h53m40.703224427s) [noise=true , address=145.79.227.3:4150 , piece_id=DZ6XDLLNJSXWZ2O2LBNWR7M2S2GJ3CXXJK7YWCYT7Y5PDIFJYF7A, seg_pos={0 0}, upload=503124]: Uploading piece... Dec 29 13:22:40.832533829 (573h53m40.703616874s) [noise=true , address=145.79.227.11:4731 , piece_id=SCEAXXRIPST2OUFABUKFNQ5VITBMNQSBO3B2ZLIRRJOVPGK5Q45Q, seg_pos={0 0}, upload=503124]: Uploading piece... Dec 29 13:22:40.832979267 (573h53m40.704062312s) [noise=true , address=145.79.227.26:4655 , piece_id=Z7UZCYH2CHRIXZ4IFLIBAAKH4QZKMI43RQW72ITJRAB5HLQ2HYWA, seg_pos={0 0}, upload=503124]: Uploading piece... Dec 29 13:22:40.834762453 (573h53m40.705845488s) [noise=true , address=45.91.116.9:4261 , piece_id=DYK5ZC5IHVQKR4M6KOMBAXELT2BFID35YRH2LPPN5URMLPG435QQ, seg_pos={0 0}, upload=503124]: Uploading piece... Dec 29 13:22:40.836347671 (573h53m40.707430716s) [noise=true , address=45.91.116.14:4189 , piece_id=5SB546DUVS4JMPZKVWAGRLORU6UBXALX5YQPRW62TQUN5LEOKGTA, seg_pos={0 0}, upload=503124]: Uploading piece... Dec 29 13:22:40.837968648 (573h53m40.709051683s) [noise=true , address=145.79.227.2:4113 , piece_id=X4ZJCYAQUGPEOECG2S42B7W5NP4KUFAZZQ7A4YVUNU5RY7GRP7WA, seg_pos={0 0}, upload=503124]: Uploading piece... Dec 29 13:22:40.839659628 (573h53m40.710742663s) [noise=true , address=77.90.31.19:4844 , piece_id=2ZXKJFGCZ3JWYPXA5ZJXRJCOKBVK7KVO2TYQDHXBRAGAYMY6LZEA, seg_pos={0 0}, upload=503124]: Uploading piece... Dec 29 13:22:40.839922018 (573h53m40.711005063s) [noise=true , address=145.79.227.47:4265 , piece_id=XUHDCM4WYBFQ4T27DSSSR27OBVIGPO4KEYGXXFI3FEZV57UGDZDQ, seg_pos={0 0}, upload=503124]: Uploading piece... Dec 29 13:22:40.841141490 (573h53m40.712224525s) [noise=true , address=145.79.227.48:4303 , piece_id=DX3IG4Y4NKDQAB232LPH2EW64DNGNAAKZLGHH6BDPDYL4CVXY3VQ, seg_pos={0 0}, upload=503124]: Uploading piece... Dec 29 13:22:40.842461994 (573h53m40.713545029s) [noise=true , address=145.79.227.15:4874 , piece_id=M32DVD2PIAMPYWYVTVXTH55DGT7MWCTSC4QZFF2DSL3SF5IBQOXA, seg_pos={0 0}, upload=503124]: Uploading piece... Dec 29 13:22:40.842989569 (573h53m40.714072604s) [noise=true , address=145.79.227.39:4013 , piece_id=OWX27MIHCB3JBS4PSULL3P2PQEDY54V7PPEFPPN3N52TUZSYTX4A, seg_pos={0 0}, upload=503124]: Uploading piece... Dec 29 13:22:42.550561936 (573h53m42.421644971s) [upload=503124]: Next returned nil segment Dec 29 13:22:42.550578698 (573h53m42.421661743s) [upload=503124]: Waiting for error group managing segments... Dec 29 13:22:42.827721168 (573h53m42.698804203s) [noise=true , address=145.79.227.26:4655 , piece_id=Z7UZCYH2CHRIXZ4IFLIBAAKH4QZKMI43RQW72ITJRAB5HLQ2HYWA, seg_pos={0 0}, upload=503124]: Done uploading piece. err: <nil> Dec 29 13:22:42.830747090 (573h53m42.701830145s) [noise=true , address=145.79.227.52:4447 , piece_id=RACK4VLSCFEX52APOHS5QKEIQKIZTGGHXVPQTAE5VQMSGGKMWC5Q, seg_pos={0 0}, upload=503124]: Done uploading piece. err: <nil> Dec 29 13:22:42.831647725 (573h53m42.70273076s) [noise=true , address=145.79.227.41:4087 , piece_id=MEV3CUDWUR5W3CIKX4D5SMKDBZ6RNC6B2HVOHC2MMUODASM7ZJBQ, seg_pos={0 0}, upload=503124]: Done uploading piece. err: <nil> Dec 29 13:22:42.833222834 (573h53m42.704305869s) [noise=true , address=145.79.227.39:4013 , piece_id=OWX27MIHCB3JBS4PSULL3P2PQEDY54V7PPEFPPN3N52TUZSYTX4A, seg_pos={0 0}, upload=503124]: Done uploading piece. err: <nil> Dec 29 13:22:42.836561621 (573h53m42.707644666s) [noise=true , address=145.79.227.48:4303 , piece_id=DX3IG4Y4NKDQAB232LPH2EW64DNGNAAKZLGHH6BDPDYL4CVXY3VQ, seg_pos={0 0}, upload=503124]: Done uploading piece. err: <nil> Dec 29 13:22:42.841783204 (573h53m42.712866239s) [noise=true , address=145.79.227.47:4265 , piece_id=XUHDCM4WYBFQ4T27DSSSR27OBVIGPO4KEYGXXFI3FEZV57UGDZDQ, seg_pos={0 0}, upload=503124]: Done uploading piece. err: <nil> Dec 29 13:22:42.843113217 (573h53m42.714196252s) [noise=true , address=145.79.227.3:4150 , piece_id=DZ6XDLLNJSXWZ2O2LBNWR7M2S2GJ3CXXJK7YWCYT7Y5PDIFJYF7A, seg_pos={0 0}, upload=503124]: Done uploading piece. err: <nil> Dec 29 13:22:42.845625099 (573h53m42.716708134s) [noise=true , address=145.79.227.15:4874 , piece_id=M32DVD2PIAMPYWYVTVXTH55DGT7MWCTSC4QZFF2DSL3SF5IBQOXA, seg_pos={0 0}, upload=503124]: Done uploading piece. err: <nil> Dec 29 13:22:42.846786341 (573h53m42.717869376s) [noise=true , address=145.79.227.2:4113 , piece_id=X4ZJCYAQUGPEOECG2S42B7W5NP4KUFAZZQ7A4YVUNU5RY7GRP7WA, seg_pos={0 0}, upload=503124]: Done uploading piece. err: <nil> Dec 29 13:22:42.847257728 (573h53m42.718340773s) [noise=true , address=145.79.227.14:4840 , piece_id=5TLCNUWDIQ72WIFPVWAZPIVNV6JCJRHSCOEPJW6D3JH2SXBSQ7KQ, seg_pos={0 0}, upload=503124]: Done uploading piece. err: <nil> Dec 29 13:22:42.847691455 (573h53m42.7187745s) [noise=true , address=145.79.227.11:4731 , piece_id=SCEAXXRIPST2OUFABUKFNQ5VITBMNQSBO3B2ZLIRRJOVPGK5Q45Q, seg_pos={0 0}, upload=503124]: Done uploading piece. err: <nil> Dec 29 13:22:43.115180433 (573h53m42.986263478s) [noise=true , address=45.91.116.9:4261 , piece_id=DYK5ZC5IHVQKR4M6KOMBAXELT2BFID35YRH2LPPN5URMLPG435QQ, seg_pos={0 0}, upload=503124]: Done uploading piece. err: <nil> Dec 29 13:22:43.181275907 (573h53m43.052358942s) [noise=true , address=77.90.31.19:4844 , piece_id=2ZXKJFGCZ3JWYPXA5ZJXRJCOKBVK7KVO2TYQDHXBRAGAYMY6LZEA, seg_pos={0 0}, upload=503124]: Done uploading piece. err: <nil> Dec 29 13:22:43.181295685 (573h53m43.052378731s) [seg_pos={0 0}, upload=503124]: Segment reached optimal threshold of 13 pieces. Dec 29 13:22:43.181488944 (573h53m43.052571989s) [noise=true , address=45.91.116.6:4975 , piece_id=PNL7J5CHPNE2RQRHV6R5XTBHHJT3OJC24YATO4DXAZVX5533HNPQ, seg_pos={0 0}, upload=503124]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 29 13:22:43.181496568 (573h53m43.052579623s) [noise=true , address=45.91.116.6:4975 , piece_id=PNL7J5CHPNE2RQRHV6R5XTBHHJT3OJC24YATO4DXAZVX5533HNPQ, seg_pos={0 0}, upload=503124]: Long tail cancellation due to optimal threshold Dec 29 13:22:43.181740974 (573h53m43.052824019s) [noise=true , address=45.91.116.14:4189 , piece_id=5SB546DUVS4JMPZKVWAGRLORU6UBXALX5YQPRW62TQUN5LEOKGTA, seg_pos={0 0}, upload=503124]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: write tcp 188.166.225.170:36668->45.91.116.14:4189: use of closed network connection; write tcp 188.166.225.170:36668->45.91.116.14:4189: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36668->45.91.116.14:4189: use of closed network connection Dec 29 13:22:43.181747666 (573h53m43.052830701s) [noise=true , address=45.91.116.14:4189 , piece_id=5SB546DUVS4JMPZKVWAGRLORU6UBXALX5YQPRW62TQUN5LEOKGTA, seg_pos={0 0}, upload=503124]: Long tail cancellation due to optimal threshold Dec 29 13:22:43.181896720 (573h53m43.052979785s) [noise=true , address=45.91.116.10:4082 , piece_id=M4UR7KUJZJT53WYAROF6EOPHAXU6WWXE247ZZMYWHMMD3M3SVOSQ, seg_pos={0 0}, upload=503124]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: write tcp 188.166.225.170:42702->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:42702->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42702->45.91.116.10:4082: use of closed network connection Dec 29 13:22:43.181944431 (573h53m43.053027476s) [noise=true , address=45.91.116.10:4082 , piece_id=M4UR7KUJZJT53WYAROF6EOPHAXU6WWXE247ZZMYWHMMD3M3SVOSQ, seg_pos={0 0}, upload=503124]: Long tail cancellation due to optimal threshold Dec 29 13:22:43.181983956 (573h53m43.053066991s) [seg_pos={0 0}, upload=503124]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 13:22:43.182260503 (573h53m43.053343548s) [upload=503124]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 13:22:43.182267436 (573h53m43.053350471s) [upload=503124]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 13:22:43.485049817 (573h53m43.356132862s) [upload=503124]: Done uploading segments... Dec 29 13:22:43.485074533 (573h53m43.356157578s) [upload=503124]: Upload finished. err: <nil> Duration: 6.101264303s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 51 out of 60 is failed (threshold=6) [XXXXXXX.XXXXXXX.X.XX.XXX.XX.XXXXXXX.XX.XXXXXXXXXXX.XXXXXXXXX] duration: 5.00082331s trace: https://jaeger.dev.storj.io/trace/1f9b35a1b5471c62 FAIL (2025-12-29 13:23:16 UTC) orbiter warning: context deadline exceeded Duration: 5.000843838s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 32 out of 60 is failed (threshold=3) [X..XX..X...XX.X.XX...X.X.X.XXX.X.X.X.XX..X.XXX.XX..X.XX..XXX] duration: 30.004724722s trace: https://jaeger.dev.storj.io/trace/41dd262f22c0a8cd FAIL (2025-12-29 13:23:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: 2ZXKJFGCZ3JWYPXA5ZJXRJCOKBVK7KVO2TYQDHXBRAGAYMY6LZEA) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: DYK5ZC5IHVQKR4M6KOMBAXELT2BFID35YRH2LPPN5URMLPG435QQ) context deadline exceeded ======LOGS====== ts... Dec 29 13:22:40.582068626 (573h53m40.453151661s) [upload=503124]: Got next segment. Inline: false Dec 29 13:22:40.582101098 (573h53m40.453184143s) [upload=503124]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 13:22:40.582105416 (573h53m40.453188451s) [upload=503124]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 13:22:40.830399264 (573h53m40.701482309s) [seg_pos={0 0}, upload=503124]: Begin upload segment... Dec 29 13:22:40.830455933 (573h53m40.701538978s) [seg_pos={0 0}, upload=503124]: Done begin upload segment. Dec 29 13:22:40.830469127 (573h53m40.701552162s) [noise=true , address=45.91.116.10:4082 , piece_id=M4UR7KUJZJT53WYAROF6EOPHAXU6WWXE247ZZMYWHMMD3M3SVOSQ, seg_pos={0 0}, upload=503124]: Uploading piece... Dec 29 13:22:40.830542828 (573h53m40.701625863s) [noise=true , address=145.79.227.52:4447 , piece_id=RACK4VLSCFEX52APOHS5QKEIQKIZTGGHXVPQTAE5VQMSGGKMWC5Q, seg_pos={0 0}, upload=503124]: Uploading piece... Dec 29 13:22:40.830964520 (573h53m40.702047556s) [noise=true , address=145.79.227.14:4840 , piece_id=5TLCNUWDIQ72WIFPVWAZPIVNV6JCJRHSCOEPJW6D3JH2SXBSQ7KQ, seg_pos={0 0}, upload=503124]: Uploading piece... Dec 29 13:22:40.831322132 (573h53m40.702405177s) [noise=true , address=145.79.227.41:4087 , piece_id=MEV3CUDWUR5W3CIKX4D5SMKDBZ6RNC6B2HVOHC2MMUODASM7ZJBQ, seg_pos={0 0}, upload=503124]: Uploading piece... Dec 29 13:22:40.831727844 (573h53m40.702810879s) [noise=true , address=45.91.116.6:4975 , piece_id=PNL7J5CHPNE2RQRHV6R5XTBHHJT3OJC24YATO4DXAZVX5533HNPQ, seg_pos={0 0}, upload=503124]: Uploading piece... Dec 29 13:22:40.832141382 (573h53m40.703224427s) [noise=true , address=145.79.227.3:4150 , piece_id=DZ6XDLLNJSXWZ2O2LBNWR7M2S2GJ3CXXJK7YWCYT7Y5PDIFJYF7A, seg_pos={0 0}, upload=503124]: Uploading piece... Dec 29 13:22:40.832533829 (573h53m40.703616874s) [noise=true , address=145.79.227.11:4731 , piece_id=SCEAXXRIPST2OUFABUKFNQ5VITBMNQSBO3B2ZLIRRJOVPGK5Q45Q, seg_pos={0 0}, upload=503124]: Uploading piece... Dec 29 13:22:40.832979267 (573h53m40.704062312s) [noise=true , address=145.79.227.26:4655 , piece_id=Z7UZCYH2CHRIXZ4IFLIBAAKH4QZKMI43RQW72ITJRAB5HLQ2HYWA, seg_pos={0 0}, upload=503124]: Uploading piece... Dec 29 13:22:40.834762453 (573h53m40.705845488s) [noise=true , address=45.91.116.9:4261 , piece_id=DYK5ZC5IHVQKR4M6KOMBAXELT2BFID35YRH2LPPN5URMLPG435QQ, seg_pos={0 0}, upload=503124]: Uploading piece... Dec 29 13:22:40.836347671 (573h53m40.707430716s) [noise=true , address=45.91.116.14:4189 , piece_id=5SB546DUVS4JMPZKVWAGRLORU6UBXALX5YQPRW62TQUN5LEOKGTA, seg_pos={0 0}, upload=503124]: Uploading piece... Dec 29 13:22:40.837968648 (573h53m40.709051683s) [noise=true , address=145.79.227.2:4113 , piece_id=X4ZJCYAQUGPEOECG2S42B7W5NP4KUFAZZQ7A4YVUNU5RY7GRP7WA, seg_pos={0 0}, upload=503124]: Uploading piece... Dec 29 13:22:40.839659628 (573h53m40.710742663s) [noise=true , address=77.90.31.19:4844 , piece_id=2ZXKJFGCZ3JWYPXA5ZJXRJCOKBVK7KVO2TYQDHXBRAGAYMY6LZEA, seg_pos={0 0}, upload=503124]: Uploading piece... Dec 29 13:22:40.839922018 (573h53m40.711005063s) [noise=true , address=145.79.227.47:4265 , piece_id=XUHDCM4WYBFQ4T27DSSSR27OBVIGPO4KEYGXXFI3FEZV57UGDZDQ, seg_pos={0 0}, upload=503124]: Uploading piece... Dec 29 13:22:40.841141490 (573h53m40.712224525s) [noise=true , address=145.79.227.48:4303 , piece_id=DX3IG4Y4NKDQAB232LPH2EW64DNGNAAKZLGHH6BDPDYL4CVXY3VQ, seg_pos={0 0}, upload=503124]: Uploading piece... Dec 29 13:22:40.842461994 (573h53m40.713545029s) [noise=true , address=145.79.227.15:4874 , piece_id=M32DVD2PIAMPYWYVTVXTH55DGT7MWCTSC4QZFF2DSL3SF5IBQOXA, seg_pos={0 0}, upload=503124]: Uploading piece... Dec 29 13:22:40.842989569 (573h53m40.714072604s) [noise=true , address=145.79.227.39:4013 , piece_id=OWX27MIHCB3JBS4PSULL3P2PQEDY54V7PPEFPPN3N52TUZSYTX4A, seg_pos={0 0}, upload=503124]: Uploading piece... Dec 29 13:22:42.550561936 (573h53m42.421644971s) [upload=503124]: Next returned nil segment Dec 29 13:22:42.550578698 (573h53m42.421661743s) [upload=503124]: Waiting for error group managing segments... Dec 29 13:22:42.827721168 (573h53m42.698804203s) [noise=true , address=145.79.227.26:4655 , piece_id=Z7UZCYH2CHRIXZ4IFLIBAAKH4QZKMI43RQW72ITJRAB5HLQ2HYWA, seg_pos={0 0}, upload=503124]: Done uploading piece. err: <nil> Dec 29 13:22:42.830747090 (573h53m42.701830145s) [noise=true , address=145.79.227.52:4447 , piece_id=RACK4VLSCFEX52APOHS5QKEIQKIZTGGHXVPQTAE5VQMSGGKMWC5Q, seg_pos={0 0}, upload=503124]: Done uploading piece. err: <nil> Dec 29 13:22:42.831647725 (573h53m42.70273076s) [noise=true , address=145.79.227.41:4087 , piece_id=MEV3CUDWUR5W3CIKX4D5SMKDBZ6RNC6B2HVOHC2MMUODASM7ZJBQ, seg_pos={0 0}, upload=503124]: Done uploading piece. err: <nil> Dec 29 13:22:42.833222834 (573h53m42.704305869s) [noise=true , address=145.79.227.39:4013 , piece_id=OWX27MIHCB3JBS4PSULL3P2PQEDY54V7PPEFPPN3N52TUZSYTX4A, seg_pos={0 0}, upload=503124]: Done uploading piece. err: <nil> Dec 29 13:22:42.836561621 (573h53m42.707644666s) [noise=true , address=145.79.227.48:4303 , piece_id=DX3IG4Y4NKDQAB232LPH2EW64DNGNAAKZLGHH6BDPDYL4CVXY3VQ, seg_pos={0 0}, upload=503124]: Done uploading piece. err: <nil> Dec 29 13:22:42.841783204 (573h53m42.712866239s) [noise=true , address=145.79.227.47:4265 , piece_id=XUHDCM4WYBFQ4T27DSSSR27OBVIGPO4KEYGXXFI3FEZV57UGDZDQ, seg_pos={0 0}, upload=503124]: Done uploading piece. err: <nil> Dec 29 13:22:42.843113217 (573h53m42.714196252s) [noise=true , address=145.79.227.3:4150 , piece_id=DZ6XDLLNJSXWZ2O2LBNWR7M2S2GJ3CXXJK7YWCYT7Y5PDIFJYF7A, seg_pos={0 0}, upload=503124]: Done uploading piece. err: <nil> Dec 29 13:22:42.845625099 (573h53m42.716708134s) [noise=true , address=145.79.227.15:4874 , piece_id=M32DVD2PIAMPYWYVTVXTH55DGT7MWCTSC4QZFF2DSL3SF5IBQOXA, seg_pos={0 0}, upload=503124]: Done uploading piece. err: <nil> Dec 29 13:22:42.846786341 (573h53m42.717869376s) [noise=true , address=145.79.227.2:4113 , piece_id=X4ZJCYAQUGPEOECG2S42B7W5NP4KUFAZZQ7A4YVUNU5RY7GRP7WA, seg_pos={0 0}, upload=503124]: Done uploading piece. err: <nil> Dec 29 13:22:42.847257728 (573h53m42.718340773s) [noise=true , address=145.79.227.14:4840 , piece_id=5TLCNUWDIQ72WIFPVWAZPIVNV6JCJRHSCOEPJW6D3JH2SXBSQ7KQ, seg_pos={0 0}, upload=503124]: Done uploading piece. err: <nil> Dec 29 13:22:42.847691455 (573h53m42.7187745s) [noise=true , address=145.79.227.11:4731 , piece_id=SCEAXXRIPST2OUFABUKFNQ5VITBMNQSBO3B2ZLIRRJOVPGK5Q45Q, seg_pos={0 0}, upload=503124]: Done uploading piece. err: <nil> Dec 29 13:22:43.115180433 (573h53m42.986263478s) [noise=true , address=45.91.116.9:4261 , piece_id=DYK5ZC5IHVQKR4M6KOMBAXELT2BFID35YRH2LPPN5URMLPG435QQ, seg_pos={0 0}, upload=503124]: Done uploading piece. err: <nil> Dec 29 13:22:43.181275907 (573h53m43.052358942s) [noise=true , address=77.90.31.19:4844 , piece_id=2ZXKJFGCZ3JWYPXA5ZJXRJCOKBVK7KVO2TYQDHXBRAGAYMY6LZEA, seg_pos={0 0}, upload=503124]: Done uploading piece. err: <nil> Dec 29 13:22:43.181295685 (573h53m43.052378731s) [seg_pos={0 0}, upload=503124]: Segment reached optimal threshold of 13 pieces. Dec 29 13:22:43.181488944 (573h53m43.052571989s) [noise=true , address=45.91.116.6:4975 , piece_id=PNL7J5CHPNE2RQRHV6R5XTBHHJT3OJC24YATO4DXAZVX5533HNPQ, seg_pos={0 0}, upload=503124]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 29 13:22:43.181496568 (573h53m43.052579623s) [noise=true , address=45.91.116.6:4975 , piece_id=PNL7J5CHPNE2RQRHV6R5XTBHHJT3OJC24YATO4DXAZVX5533HNPQ, seg_pos={0 0}, upload=503124]: Long tail cancellation due to optimal threshold Dec 29 13:22:43.181740974 (573h53m43.052824019s) [noise=true , address=45.91.116.14:4189 , piece_id=5SB546DUVS4JMPZKVWAGRLORU6UBXALX5YQPRW62TQUN5LEOKGTA, seg_pos={0 0}, upload=503124]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: write tcp 188.166.225.170:36668->45.91.116.14:4189: use of closed network connection; write tcp 188.166.225.170:36668->45.91.116.14:4189: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36668->45.91.116.14:4189: use of closed network connection Dec 29 13:22:43.181747666 (573h53m43.052830701s) [noise=true , address=45.91.116.14:4189 , piece_id=5SB546DUVS4JMPZKVWAGRLORU6UBXALX5YQPRW62TQUN5LEOKGTA, seg_pos={0 0}, upload=503124]: Long tail cancellation due to optimal threshold Dec 29 13:22:43.181896720 (573h53m43.052979785s) [noise=true , address=45.91.116.10:4082 , piece_id=M4UR7KUJZJT53WYAROF6EOPHAXU6WWXE247ZZMYWHMMD3M3SVOSQ, seg_pos={0 0}, upload=503124]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: write tcp 188.166.225.170:42702->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:42702->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42702->45.91.116.10:4082: use of closed network connection Dec 29 13:22:43.181944431 (573h53m43.053027476s) [noise=true , address=45.91.116.10:4082 , piece_id=M4UR7KUJZJT53WYAROF6EOPHAXU6WWXE247ZZMYWHMMD3M3SVOSQ, seg_pos={0 0}, upload=503124]: Long tail cancellation due to optimal threshold Dec 29 13:22:43.181983956 (573h53m43.053066991s) [seg_pos={0 0}, upload=503124]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 13:22:43.182260503 (573h53m43.053343548s) [upload=503124]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 13:22:43.182267436 (573h53m43.053350471s) [upload=503124]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 13:22:43.485049817 (573h53m43.356132862s) [upload=503124]: Done uploading segments... Dec 29 13:22:43.485074533 (573h53m43.356157578s) [upload=503124]: Upload finished. err: <nil> Duration: 30.004752316s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 51 out of 60 is failed (threshold=6) [XXXXXXXX.XXXXXXX.X.XX.XXX.XX.XXXXXXX.XX.XXXXXXXXXXX.XXXXXXXX] duration: 5.000733516s trace: https://jaeger.dev.storj.io/trace/7d66e83044a90f1b FAIL (2025-12-29 13:22:59 UTC) orbiter warning: context deadline exceeded Duration: 5.00077842s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.......X......................X....X.X....X.X...............] duration: 21.2892869s trace: https://jaeger.dev.storj.io/trace/312ddb07b2dee82a FAIL (2025-12-29 13:22:53 UTC) orbiter warning: context deadline exceeded Duration: 21.289336775s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [...................X.....X.....................XX...........] duration: 32.962053257s trace: https://jaeger.dev.storj.io/trace/77d26141ea11d228 FAIL (2025-12-29 13:22:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12J2weU7Bmsr3RdQQ9QMuadT96XReBkyxTeaLJ2SEqVGGFX1PWu, Piece ID: JITX25FNZ4JIKYXQY345UX3SQUCPDWJ25NJQZ2TXGFRLVCRYXUOQ) context deadline exceeded; piecestore: (Node ID: 125sm6iHV94mJFLWrKeyRMH4b1qSMo32nDuW8uQr71xUL7Cema2, Piece ID: INTVU2EYTU3DIM6WQFQEGO6IEYH3YFP5U2VEN7A5KPTOKU2DWOOA) context deadline exceeded; piecestore: (Node ID: 12i57FF97oEddYLZpNxSq5or9WLEquL8ZW1cVUuW4AThBbDqcCL, Piece ID: Q35LXGPTOAG3R2UL3JLL7DNHHD4PSMEULYW6G3SPXKNVFZ5VTNRA) context deadline exceeded ======LOGS====== Q2LO7MUXLHE5YET7BHLLA, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.692912334 (573h43m3.563995379s) [noise=true , address=77.90.31.16:4759 , piece_id=ARC4HRP57U5VQXZTT63QYLXBPZURI5VMJBIQZOFKGLJBQ3K3KGLQ, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.693322595 (573h43m3.56440564s) [noise=true , address=77.90.31.15:4711 , piece_id=JITX25FNZ4JIKYXQY345UX3SQUCPDWJ25NJQZ2TXGFRLVCRYXUOQ, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.693739649 (573h43m3.564822694s) [noise=true , address=45.91.116.6:4981 , piece_id=INTVU2EYTU3DIM6WQFQEGO6IEYH3YFP5U2VEN7A5KPTOKU2DWOOA, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.694315887 (573h43m3.565398942s) [noise=true , address=145.79.227.4:4220 , piece_id=SHHJ7EW7DTNOZXVBP6A64VIX7P6AQN62TBCUT3KTH4MIB6UXY3BA, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.694765222 (573h43m3.565848267s) [noise=true , address=145.79.227.29:4952 , piece_id=DKYDB2R4DNCGH6T3SLRR2MOZZUPVZZVRL2BKGYOBYXJJVPSZEJ4Q, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.695202545 (573h43m3.56628559s) [noise=true , address=145.79.227.40:4058 , piece_id=KRHH2QZKSDPWTSSKYDVDWGKFJD3LPGP6U7KI2PPJ5T3TXYIUGJWA, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.695687528 (573h43m3.566770584s) [noise=true , address=77.90.31.19:4857 , piece_id=Q35LXGPTOAG3R2UL3JLL7DNHHD4PSMEULYW6G3SPXKNVFZ5VTNRA, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.696135221 (573h43m3.567218276s) [noise=true , address=45.91.116.8:4051 , piece_id=GDJFLQCHUF7BFQ7BVSNTVJ3TYZHYVOO6YYMO3KUIUSIEPAF4SFDQ, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.696703733 (573h43m3.567786778s) [noise=true , address=64.49.13.170:4720 , piece_id=HF6X3OT2EPPDE3DWAPWABVZUR3ERDBWXHG3NGNKEWZAD7G5AN4TA, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.698418298 (573h43m3.569501343s) [noise=true , address=145.79.227.36:4520 , piece_id=MYJSXUH563JFUW5Q7YUIWL7466G6ASY5WXR3T2PGUFFCTRTOBYJA, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.700198829 (573h43m3.571281874s) [noise=true , address=77.90.31.18:4821 , piece_id=LO5W47FHYOVGY5Q6BH5L6KX77HES3XRAFUTPI45HYPJZDO3YHLCQ, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.701993145 (573h43m3.57307619s) [noise=true , address=193.149.164.143:4699 , piece_id=UDFDCYO25MOJBTMDE6F2WILXE3HLOB75UJZ4LW7ZMNSCOBJI6UJA, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.703884687 (573h43m3.574967732s) [noise=true , address=145.79.227.33:4893 , piece_id=XGLX3DFNAY4F4GPGCGZ4ONDJ7NK62SAKX3PBRWHWI74EF3ZFMUSA, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.705687109 (573h43m3.576770154s) [noise=true , address=45.91.116.10:4094 , piece_id=6UUIZNTESBBWJMDQ5ZTCHU7S6YKM4GDKO4ZAC6T2FUTPVGFFDS3A, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.707557651 (573h43m3.578640696s) [noise=true , address=145.79.227.43:4164 , piece_id=J6Q4ZDOIIIBA2FXGZZTD6AYS7KGAWSRXXUBAPDPQWYKNU5GNXBBQ, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:05.168247719 (573h43m5.039330754s) [upload=502975]: Next returned nil segment Dec 29 13:12:05.168259962 (573h43m5.039343007s) [upload=502975]: Waiting for error group managing segments... Dec 29 13:12:05.558873127 (573h43m5.429956162s) [noise=true , address=145.79.227.4:4220 , piece_id=SHHJ7EW7DTNOZXVBP6A64VIX7P6AQN62TBCUT3KTH4MIB6UXY3BA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:05.563024312 (573h43m5.434107347s) [noise=true , address=64.49.13.170:4720 , piece_id=HF6X3OT2EPPDE3DWAPWABVZUR3ERDBWXHG3NGNKEWZAD7G5AN4TA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:05.564115719 (573h43m5.435198755s) [noise=true , address=145.79.227.36:4520 , piece_id=MYJSXUH563JFUW5Q7YUIWL7466G6ASY5WXR3T2PGUFFCTRTOBYJA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:05.593873402 (573h43m5.464956447s) [noise=true , address=145.79.227.29:4952 , piece_id=DKYDB2R4DNCGH6T3SLRR2MOZZUPVZZVRL2BKGYOBYXJJVPSZEJ4Q, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:05.898194604 (573h43m5.769277639s) [noise=true , address=45.91.116.10:4094 , piece_id=6UUIZNTESBBWJMDQ5ZTCHU7S6YKM4GDKO4ZAC6T2FUTPVGFFDS3A, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:05.912845292 (573h43m5.783928337s) [noise=true , address=77.90.31.19:4857 , piece_id=Q35LXGPTOAG3R2UL3JLL7DNHHD4PSMEULYW6G3SPXKNVFZ5VTNRA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:06.426532207 (573h43m6.297615252s) [noise=true , address=193.149.164.143:4699 , piece_id=UDFDCYO25MOJBTMDE6F2WILXE3HLOB75UJZ4LW7ZMNSCOBJI6UJA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:06.449589562 (573h43m6.320672607s) [noise=true , address=77.90.31.15:4711 , piece_id=JITX25FNZ4JIKYXQY345UX3SQUCPDWJ25NJQZ2TXGFRLVCRYXUOQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:06.453186841 (573h43m6.324269876s) [noise=true , address=145.79.227.40:4058 , piece_id=KRHH2QZKSDPWTSSKYDVDWGKFJD3LPGP6U7KI2PPJ5T3TXYIUGJWA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:06.510769276 (573h43m6.381852321s) [noise=true , address=145.79.227.43:4164 , piece_id=J6Q4ZDOIIIBA2FXGZZTD6AYS7KGAWSRXXUBAPDPQWYKNU5GNXBBQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:06.518868879 (573h43m6.389951924s) [noise=true , address=145.79.227.33:4893 , piece_id=XGLX3DFNAY4F4GPGCGZ4ONDJ7NK62SAKX3PBRWHWI74EF3ZFMUSA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:06.519238043 (573h43m6.390321098s) [noise=true , address=45.91.116.6:4981 , piece_id=INTVU2EYTU3DIM6WQFQEGO6IEYH3YFP5U2VEN7A5KPTOKU2DWOOA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:06.523211328 (573h43m6.394294373s) [noise=true , address=77.90.31.16:4759 , piece_id=ARC4HRP57U5VQXZTT63QYLXBPZURI5VMJBIQZOFKGLJBQ3K3KGLQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:06.767869783 (573h43m6.638952818s) [noise=true , address=145.79.227.34:4924 , piece_id=MAQXQTAGXQ7AEPUCESUSUGMHSDMMXS7HHKAEA7B3HW5X66PDUF5A, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:06.767889431 (573h43m6.638972476s) [seg_pos={0 0}, upload=502975]: Segment reached optimal threshold of 14 pieces. Dec 29 13:12:06.768102116 (573h43m6.639185161s) [noise=true , address=45.91.116.8:4051 , piece_id=GDJFLQCHUF7BFQ7BVSNTVJ3TYZHYVOO6YYMO3KUIUSIEPAF4SFDQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1preT33ei4yVXXYVLHhaUbHbiyL14oHFEiof6cCWPig83W6CT3): protocol: expected piece hash; context canceled; EOF Dec 29 13:12:06.768108257 (573h43m6.639191302s) [noise=true , address=45.91.116.8:4051 , piece_id=GDJFLQCHUF7BFQ7BVSNTVJ3TYZHYVOO6YYMO3KUIUSIEPAF4SFDQ, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold Dec 29 13:12:06.768219208 (573h43m6.639302254s) [noise=true , address=77.90.31.18:4821 , piece_id=LO5W47FHYOVGY5Q6BH5L6KX77HES3XRAFUTPI45HYPJZDO3YHLCQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH): protocol: expected piece hash; context canceled; EOF Dec 29 13:12:06.768224789 (573h43m6.639307824s) [noise=true , address=77.90.31.18:4821 , piece_id=LO5W47FHYOVGY5Q6BH5L6KX77HES3XRAFUTPI45HYPJZDO3YHLCQ, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold Dec 29 13:12:06.768384213 (573h43m6.639467258s) [noise=true , address=64.49.13.169:4711 , piece_id=4F4T4YKXMT7NF2C5KBUGJHQJGZL4VCKNKYB266PCVI3UACHJXZ5Q, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:48802->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:48802->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48802->64.49.13.169:4711: use of closed network connection Dec 29 13:12:06.768399261 (573h43m6.639482296s) [noise=true , address=64.49.13.169:4711 , piece_id=4F4T4YKXMT7NF2C5KBUGJHQJGZL4VCKNKYB266PCVI3UACHJXZ5Q, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold Dec 29 13:12:06.768732636 (573h43m6.639815671s) [noise=true , address=64.49.13.169:4709 , piece_id=TBHHX5ES2A6NQRGP25HIC7J7IBWLOJGQ2LO7MUXLHE5YET7BHLLA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:35310->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:35310->64.49.13.169:4709: use of closed network connection Dec 29 13:12:06.768745460 (573h43m6.639828505s) [noise=true , address=64.49.13.169:4709 , piece_id=TBHHX5ES2A6NQRGP25HIC7J7IBWLOJGQ2LO7MUXLHE5YET7BHLLA, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold Dec 29 13:12:06.768784586 (573h43m6.639867631s) [seg_pos={0 0}, upload=502975]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 13:12:06.768880217 (573h43m6.639963252s) [upload=502975]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 13:12:06.768884906 (573h43m6.639967941s) [upload=502975]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 13:12:07.067188330 (573h43m6.938271365s) [upload=502975]: Done uploading segments... Dec 29 13:12:07.067251099 (573h43m6.938334134s) [upload=502975]: Upload finished. err: <nil> Duration: 32.962072563s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [........X......................X....X.X....X.X..............] duration: 18.539777597s trace: https://jaeger.dev.storj.io/trace/312ddb07b2dee82a FAIL (2025-12-29 13:22:27 UTC) orbiter warning: context deadline exceeded Duration: 18.539804728s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [....................X.....X.....................XX..........] duration: 51.304686506s trace: https://jaeger.dev.storj.io/trace/77d26141ea11d228 FAIL (2025-12-29 13:22:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12J2weU7Bmsr3RdQQ9QMuadT96XReBkyxTeaLJ2SEqVGGFX1PWu, Piece ID: JITX25FNZ4JIKYXQY345UX3SQUCPDWJ25NJQZ2TXGFRLVCRYXUOQ) context deadline exceeded; piecestore: (Node ID: 125sm6iHV94mJFLWrKeyRMH4b1qSMo32nDuW8uQr71xUL7Cema2, Piece ID: INTVU2EYTU3DIM6WQFQEGO6IEYH3YFP5U2VEN7A5KPTOKU2DWOOA) context deadline exceeded; piecestore: (Node ID: 12i57FF97oEddYLZpNxSq5or9WLEquL8ZW1cVUuW4AThBbDqcCL, Piece ID: Q35LXGPTOAG3R2UL3JLL7DNHHD4PSMEULYW6G3SPXKNVFZ5VTNRA) context deadline exceeded ======LOGS====== Q2LO7MUXLHE5YET7BHLLA, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.692912334 (573h43m3.563995379s) [noise=true , address=77.90.31.16:4759 , piece_id=ARC4HRP57U5VQXZTT63QYLXBPZURI5VMJBIQZOFKGLJBQ3K3KGLQ, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.693322595 (573h43m3.56440564s) [noise=true , address=77.90.31.15:4711 , piece_id=JITX25FNZ4JIKYXQY345UX3SQUCPDWJ25NJQZ2TXGFRLVCRYXUOQ, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.693739649 (573h43m3.564822694s) [noise=true , address=45.91.116.6:4981 , piece_id=INTVU2EYTU3DIM6WQFQEGO6IEYH3YFP5U2VEN7A5KPTOKU2DWOOA, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.694315887 (573h43m3.565398942s) [noise=true , address=145.79.227.4:4220 , piece_id=SHHJ7EW7DTNOZXVBP6A64VIX7P6AQN62TBCUT3KTH4MIB6UXY3BA, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.694765222 (573h43m3.565848267s) [noise=true , address=145.79.227.29:4952 , piece_id=DKYDB2R4DNCGH6T3SLRR2MOZZUPVZZVRL2BKGYOBYXJJVPSZEJ4Q, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.695202545 (573h43m3.56628559s) [noise=true , address=145.79.227.40:4058 , piece_id=KRHH2QZKSDPWTSSKYDVDWGKFJD3LPGP6U7KI2PPJ5T3TXYIUGJWA, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.695687528 (573h43m3.566770584s) [noise=true , address=77.90.31.19:4857 , piece_id=Q35LXGPTOAG3R2UL3JLL7DNHHD4PSMEULYW6G3SPXKNVFZ5VTNRA, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.696135221 (573h43m3.567218276s) [noise=true , address=45.91.116.8:4051 , piece_id=GDJFLQCHUF7BFQ7BVSNTVJ3TYZHYVOO6YYMO3KUIUSIEPAF4SFDQ, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.696703733 (573h43m3.567786778s) [noise=true , address=64.49.13.170:4720 , piece_id=HF6X3OT2EPPDE3DWAPWABVZUR3ERDBWXHG3NGNKEWZAD7G5AN4TA, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.698418298 (573h43m3.569501343s) [noise=true , address=145.79.227.36:4520 , piece_id=MYJSXUH563JFUW5Q7YUIWL7466G6ASY5WXR3T2PGUFFCTRTOBYJA, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.700198829 (573h43m3.571281874s) [noise=true , address=77.90.31.18:4821 , piece_id=LO5W47FHYOVGY5Q6BH5L6KX77HES3XRAFUTPI45HYPJZDO3YHLCQ, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.701993145 (573h43m3.57307619s) [noise=true , address=193.149.164.143:4699 , piece_id=UDFDCYO25MOJBTMDE6F2WILXE3HLOB75UJZ4LW7ZMNSCOBJI6UJA, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.703884687 (573h43m3.574967732s) [noise=true , address=145.79.227.33:4893 , piece_id=XGLX3DFNAY4F4GPGCGZ4ONDJ7NK62SAKX3PBRWHWI74EF3ZFMUSA, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.705687109 (573h43m3.576770154s) [noise=true , address=45.91.116.10:4094 , piece_id=6UUIZNTESBBWJMDQ5ZTCHU7S6YKM4GDKO4ZAC6T2FUTPVGFFDS3A, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.707557651 (573h43m3.578640696s) [noise=true , address=145.79.227.43:4164 , piece_id=J6Q4ZDOIIIBA2FXGZZTD6AYS7KGAWSRXXUBAPDPQWYKNU5GNXBBQ, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:05.168247719 (573h43m5.039330754s) [upload=502975]: Next returned nil segment Dec 29 13:12:05.168259962 (573h43m5.039343007s) [upload=502975]: Waiting for error group managing segments... Dec 29 13:12:05.558873127 (573h43m5.429956162s) [noise=true , address=145.79.227.4:4220 , piece_id=SHHJ7EW7DTNOZXVBP6A64VIX7P6AQN62TBCUT3KTH4MIB6UXY3BA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:05.563024312 (573h43m5.434107347s) [noise=true , address=64.49.13.170:4720 , piece_id=HF6X3OT2EPPDE3DWAPWABVZUR3ERDBWXHG3NGNKEWZAD7G5AN4TA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:05.564115719 (573h43m5.435198755s) [noise=true , address=145.79.227.36:4520 , piece_id=MYJSXUH563JFUW5Q7YUIWL7466G6ASY5WXR3T2PGUFFCTRTOBYJA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:05.593873402 (573h43m5.464956447s) [noise=true , address=145.79.227.29:4952 , piece_id=DKYDB2R4DNCGH6T3SLRR2MOZZUPVZZVRL2BKGYOBYXJJVPSZEJ4Q, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:05.898194604 (573h43m5.769277639s) [noise=true , address=45.91.116.10:4094 , piece_id=6UUIZNTESBBWJMDQ5ZTCHU7S6YKM4GDKO4ZAC6T2FUTPVGFFDS3A, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:05.912845292 (573h43m5.783928337s) [noise=true , address=77.90.31.19:4857 , piece_id=Q35LXGPTOAG3R2UL3JLL7DNHHD4PSMEULYW6G3SPXKNVFZ5VTNRA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:06.426532207 (573h43m6.297615252s) [noise=true , address=193.149.164.143:4699 , piece_id=UDFDCYO25MOJBTMDE6F2WILXE3HLOB75UJZ4LW7ZMNSCOBJI6UJA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:06.449589562 (573h43m6.320672607s) [noise=true , address=77.90.31.15:4711 , piece_id=JITX25FNZ4JIKYXQY345UX3SQUCPDWJ25NJQZ2TXGFRLVCRYXUOQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:06.453186841 (573h43m6.324269876s) [noise=true , address=145.79.227.40:4058 , piece_id=KRHH2QZKSDPWTSSKYDVDWGKFJD3LPGP6U7KI2PPJ5T3TXYIUGJWA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:06.510769276 (573h43m6.381852321s) [noise=true , address=145.79.227.43:4164 , piece_id=J6Q4ZDOIIIBA2FXGZZTD6AYS7KGAWSRXXUBAPDPQWYKNU5GNXBBQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:06.518868879 (573h43m6.389951924s) [noise=true , address=145.79.227.33:4893 , piece_id=XGLX3DFNAY4F4GPGCGZ4ONDJ7NK62SAKX3PBRWHWI74EF3ZFMUSA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:06.519238043 (573h43m6.390321098s) [noise=true , address=45.91.116.6:4981 , piece_id=INTVU2EYTU3DIM6WQFQEGO6IEYH3YFP5U2VEN7A5KPTOKU2DWOOA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:06.523211328 (573h43m6.394294373s) [noise=true , address=77.90.31.16:4759 , piece_id=ARC4HRP57U5VQXZTT63QYLXBPZURI5VMJBIQZOFKGLJBQ3K3KGLQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:06.767869783 (573h43m6.638952818s) [noise=true , address=145.79.227.34:4924 , piece_id=MAQXQTAGXQ7AEPUCESUSUGMHSDMMXS7HHKAEA7B3HW5X66PDUF5A, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:06.767889431 (573h43m6.638972476s) [seg_pos={0 0}, upload=502975]: Segment reached optimal threshold of 14 pieces. Dec 29 13:12:06.768102116 (573h43m6.639185161s) [noise=true , address=45.91.116.8:4051 , piece_id=GDJFLQCHUF7BFQ7BVSNTVJ3TYZHYVOO6YYMO3KUIUSIEPAF4SFDQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1preT33ei4yVXXYVLHhaUbHbiyL14oHFEiof6cCWPig83W6CT3): protocol: expected piece hash; context canceled; EOF Dec 29 13:12:06.768108257 (573h43m6.639191302s) [noise=true , address=45.91.116.8:4051 , piece_id=GDJFLQCHUF7BFQ7BVSNTVJ3TYZHYVOO6YYMO3KUIUSIEPAF4SFDQ, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold Dec 29 13:12:06.768219208 (573h43m6.639302254s) [noise=true , address=77.90.31.18:4821 , piece_id=LO5W47FHYOVGY5Q6BH5L6KX77HES3XRAFUTPI45HYPJZDO3YHLCQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH): protocol: expected piece hash; context canceled; EOF Dec 29 13:12:06.768224789 (573h43m6.639307824s) [noise=true , address=77.90.31.18:4821 , piece_id=LO5W47FHYOVGY5Q6BH5L6KX77HES3XRAFUTPI45HYPJZDO3YHLCQ, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold Dec 29 13:12:06.768384213 (573h43m6.639467258s) [noise=true , address=64.49.13.169:4711 , piece_id=4F4T4YKXMT7NF2C5KBUGJHQJGZL4VCKNKYB266PCVI3UACHJXZ5Q, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:48802->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:48802->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48802->64.49.13.169:4711: use of closed network connection Dec 29 13:12:06.768399261 (573h43m6.639482296s) [noise=true , address=64.49.13.169:4711 , piece_id=4F4T4YKXMT7NF2C5KBUGJHQJGZL4VCKNKYB266PCVI3UACHJXZ5Q, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold Dec 29 13:12:06.768732636 (573h43m6.639815671s) [noise=true , address=64.49.13.169:4709 , piece_id=TBHHX5ES2A6NQRGP25HIC7J7IBWLOJGQ2LO7MUXLHE5YET7BHLLA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:35310->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:35310->64.49.13.169:4709: use of closed network connection Dec 29 13:12:06.768745460 (573h43m6.639828505s) [noise=true , address=64.49.13.169:4709 , piece_id=TBHHX5ES2A6NQRGP25HIC7J7IBWLOJGQ2LO7MUXLHE5YET7BHLLA, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold Dec 29 13:12:06.768784586 (573h43m6.639867631s) [seg_pos={0 0}, upload=502975]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 13:12:06.768880217 (573h43m6.639963252s) [upload=502975]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 13:12:06.768884906 (573h43m6.639967941s) [upload=502975]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 13:12:07.067188330 (573h43m6.938271365s) [upload=502975]: Done uploading segments... Dec 29 13:12:07.067251099 (573h43m6.938334134s) [upload=502975]: Upload finished. err: <nil> Duration: 51.304713056s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 32 out of 60 is failed (threshold=3) [XX..XX..X...XX.X.XX...X.X.X.XXX.X.X.X.XX..X.XXX.XX..X.XX..XX] duration: 30.003526685s trace: https://jaeger.dev.storj.io/trace/604b89599e5f4c4a FAIL (2025-12-29 13:21:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: PHRAU5QH2ISIZTWAVELY4S5P4NMZSF5RVIAB4YVWLRUREVOUX2QQ) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: 65VG2LVJQTXGX6WZEG7RAODJHUENUCRPIAJPGY5DY7AT6IBC75QQ) context deadline exceeded ======LOGS====== Dec 29 13:21:29.305485399 (573h52m29.176568434s) [upload=503106]: Starting upload Dec 29 13:21:29.305576383 (573h52m29.176659448s) [upload=503106]: Done starting upload Dec 29 13:21:29.305859522 (573h52m29.176942557s) [upload=503106]: Uploading segments... Dec 29 13:21:29.305897624 (573h52m29.176980659s) [upload=503106]: Got next segment. Inline: false Dec 29 13:21:29.305967126 (573h52m29.177050161s) [upload=503106]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 13:21:29.305972557 (573h52m29.177055592s) [upload=503106]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 13:21:29.554458631 (573h52m29.425541676s) [seg_pos={0 0}, upload=503106]: Begin upload segment... Dec 29 13:21:29.554513977 (573h52m29.425597042s) [seg_pos={0 0}, upload=503106]: Done begin upload segment. Dec 29 13:21:29.554547450 (573h52m29.425630495s) [noise=true , address=145.79.227.45:4230 , piece_id=47I6Z44MCO6VXZ5RXZF4KI2HKI7W3LQE5U2BGAL7SU3DUHEC54JQ, seg_pos={0 0}, upload=503106]: Uploading piece... Dec 29 13:21:29.554600070 (573h52m29.425683105s) [noise=true , address=145.79.227.50:4375 , piece_id=5443AIVVRUNVDCWWVBYC35JROW3LL7BGCIKGL757KAXAO263C2NQ, seg_pos={0 0}, upload=503106]: Uploading piece... Dec 29 13:21:29.555459617 (573h52m29.426542662s) [noise=true , address=145.79.227.39:4013 , piece_id=HB3AN6EPRVZXZ2YXJ5HQ6W53W5VBH2274ZYI5GIJXOUS65EQVEMA, seg_pos={0 0}, upload=503106]: Uploading piece... Dec 29 13:21:29.555927968 (573h52m29.427011004s) [noise=true , address=145.79.227.36:4518 , piece_id=BKCFM7DWK5PTYJP7EVJJZVKDAUSHMVQ65M6FZCMR6MU76LIE4UGQ, seg_pos={0 0}, upload=503106]: Uploading piece... Dec 29 13:21:29.556239281 (573h52m29.427322316s) [noise=true , address=145.79.227.40:4050 , piece_id=UKN4E2FLKIGTZY23H5TQG4AKEMIMPVQQAJMR3WBQHZ4RVBVL5NCA, seg_pos={0 0}, upload=503106]: Uploading piece... Dec 29 13:21:29.556534613 (573h52m29.427617648s) [noise=true , address=145.79.227.44:4193 , piece_id=WXI6KDZBHRFFM3RP75KJNO5VUGYVQZ2A25WKETS7JTCDAJVLLS3Q, seg_pos={0 0}, upload=503106]: Uploading piece... Dec 29 13:21:29.556802363 (573h52m29.427885408s) [noise=true , address=145.79.227.52:4447 , piece_id=KCHLJG7ZMKI4XHADDRRS472U52ZO2W4NOVK66EPGHLFM2PJNM5WA, seg_pos={0 0}, upload=503106]: Uploading piece... Dec 29 13:21:29.557074712 (573h52m29.428157757s) [noise=true , address=145.79.227.7:4294 , piece_id=VNR3GZHFPNXTYKDIZO532E2C3ZDDNP3427E7U4FNJ2SCAF2UYBSA, seg_pos={0 0}, upload=503106]: Uploading piece... Dec 29 13:21:29.557352862 (573h52m29.428435907s) [noise=true , address=145.79.227.5:4223 , piece_id=2MLJRSKNCDCLVEUKKZO4WCCYDFKPFHNN2A5XUHT7G7VVQPAQPDSQ, seg_pos={0 0}, upload=503106]: Uploading piece... Dec 29 13:21:29.557622896 (573h52m29.428705941s) [noise=true , address=145.79.227.30:4762 , piece_id=TVCPADCVI5J7YUVMMYPH52ARV6LGQ5M6264JV3XQCBBT6HH5NWUA, seg_pos={0 0}, upload=503106]: Uploading piece... Dec 29 13:21:29.558235091 (573h52m29.429318136s) [noise=true , address=145.79.227.13:4803 , piece_id=DROGBVHQ7QEPKWGKCTMUVXXKQSTHFR45NEHFYZN3TGZD4Y5U5CCA, seg_pos={0 0}, upload=503106]: Uploading piece... Dec 29 13:21:29.559193887 (573h52m29.430276933s) [noise=true , address=145.79.227.32:4834 , piece_id=VURF6RUNSBOTHAXJFOSRUO5VA7SVXU5VBUJS5IUV6XPUP6J4M5BA, seg_pos={0 0}, upload=503106]: Uploading piece... Dec 29 13:21:29.560182039 (573h52m29.431265084s) [noise=true , address=145.79.227.51:4409 , piece_id=4VGS5GIFVDXIQ7VRAVD3WXGS6G6GYIKXUUGDN2AOHCATP4GXWF4A, seg_pos={0 0}, upload=503106]: Uploading piece... Dec 29 13:21:29.561726229 (573h52m29.432809274s) [noise=true , address=145.79.227.26:4655 , piece_id=FAUVFPXA2HX2YB34FRRX3UYRFKNFIH2LQZJZPIDLBCSFTMCU6NKA, seg_pos={0 0}, upload=503106]: Uploading piece... Dec 29 13:21:29.563210716 (573h52m29.434293761s) [noise=true , address=45.91.116.13:4155 , piece_id=65VG2LVJQTXGX6WZEG7RAODJHUENUCRPIAJPGY5DY7AT6IBC75QQ, seg_pos={0 0}, upload=503106]: Uploading piece... Dec 29 13:21:29.564395672 (573h52m29.435478717s) [noise=true , address=45.91.116.7:4012 , piece_id=PHRAU5QH2ISIZTWAVELY4S5P4NMZSF5RVIAB4YVWLRUREVOUX2QQ, seg_pos={0 0}, upload=503106]: Uploading piece... Dec 29 13:21:31.247316584 (573h52m31.118399619s) [upload=503106]: Next returned nil segment Dec 29 13:21:31.247333277 (573h52m31.118416312s) [upload=503106]: Waiting for error group managing segments... Dec 29 13:21:31.538306352 (573h52m31.409389397s) [noise=true , address=145.79.227.45:4230 , piece_id=47I6Z44MCO6VXZ5RXZF4KI2HKI7W3LQE5U2BGAL7SU3DUHEC54JQ, seg_pos={0 0}, upload=503106]: Done uploading piece. err: <nil> Dec 29 13:21:31.541560920 (573h52m31.412643965s) [noise=true , address=145.79.227.13:4803 , piece_id=DROGBVHQ7QEPKWGKCTMUVXXKQSTHFR45NEHFYZN3TGZD4Y5U5CCA, seg_pos={0 0}, upload=503106]: Done uploading piece. err: <nil> Dec 29 13:21:31.546424059 (573h52m31.417507094s) [noise=true , address=145.79.227.7:4294 , piece_id=VNR3GZHFPNXTYKDIZO532E2C3ZDDNP3427E7U4FNJ2SCAF2UYBSA, seg_pos={0 0}, upload=503106]: Done uploading piece. err: <nil> Dec 29 13:21:31.547943653 (573h52m31.419026688s) [noise=true , address=145.79.227.52:4447 , piece_id=KCHLJG7ZMKI4XHADDRRS472U52ZO2W4NOVK66EPGHLFM2PJNM5WA, seg_pos={0 0}, upload=503106]: Done uploading piece. err: <nil> Dec 29 13:21:31.558389824 (573h52m31.429472869s) [noise=true , address=145.79.227.5:4223 , piece_id=2MLJRSKNCDCLVEUKKZO4WCCYDFKPFHNN2A5XUHT7G7VVQPAQPDSQ, seg_pos={0 0}, upload=503106]: Done uploading piece. err: <nil> Dec 29 13:21:31.558676831 (573h52m31.429759876s) [noise=true , address=145.79.227.36:4518 , piece_id=BKCFM7DWK5PTYJP7EVJJZVKDAUSHMVQ65M6FZCMR6MU76LIE4UGQ, seg_pos={0 0}, upload=503106]: Done uploading piece. err: <nil> Dec 29 13:21:31.565693180 (573h52m31.436776215s) [noise=true , address=145.79.227.32:4834 , piece_id=VURF6RUNSBOTHAXJFOSRUO5VA7SVXU5VBUJS5IUV6XPUP6J4M5BA, seg_pos={0 0}, upload=503106]: Done uploading piece. err: <nil> Dec 29 13:21:31.571255953 (573h52m31.442338988s) [noise=true , address=145.79.227.51:4409 , piece_id=4VGS5GIFVDXIQ7VRAVD3WXGS6G6GYIKXUUGDN2AOHCATP4GXWF4A, seg_pos={0 0}, upload=503106]: Done uploading piece. err: <nil> Dec 29 13:21:31.579025778 (573h52m31.450108823s) [noise=true , address=145.79.227.26:4655 , piece_id=FAUVFPXA2HX2YB34FRRX3UYRFKNFIH2LQZJZPIDLBCSFTMCU6NKA, seg_pos={0 0}, upload=503106]: Done uploading piece. err: <nil> Dec 29 13:21:31.586710400 (573h52m31.457793445s) [noise=true , address=145.79.227.50:4375 , piece_id=5443AIVVRUNVDCWWVBYC35JROW3LL7BGCIKGL757KAXAO263C2NQ, seg_pos={0 0}, upload=503106]: Done uploading piece. err: <nil> Dec 29 13:21:31.644744676 (573h52m31.515827731s) [noise=true , address=145.79.227.39:4013 , piece_id=HB3AN6EPRVZXZ2YXJ5HQ6W53W5VBH2274ZYI5GIJXOUS65EQVEMA, seg_pos={0 0}, upload=503106]: Done uploading piece. err: <nil> Dec 29 13:21:31.974641629 (573h52m31.845724664s) [noise=true , address=45.91.116.13:4155 , piece_id=65VG2LVJQTXGX6WZEG7RAODJHUENUCRPIAJPGY5DY7AT6IBC75QQ, seg_pos={0 0}, upload=503106]: Done uploading piece. err: <nil> Dec 29 13:21:32.666016528 (573h52m32.537099573s) [noise=true , address=45.91.116.7:4012 , piece_id=PHRAU5QH2ISIZTWAVELY4S5P4NMZSF5RVIAB4YVWLRUREVOUX2QQ, seg_pos={0 0}, upload=503106]: Done uploading piece. err: <nil> Dec 29 13:21:32.666039142 (573h52m32.537122177s) [seg_pos={0 0}, upload=503106]: Segment reached optimal threshold of 13 pieces. Dec 29 13:21:32.666320467 (573h52m32.537403502s) [noise=true , address=145.79.227.44:4193 , piece_id=WXI6KDZBHRFFM3RP75KJNO5VUGYVQZ2A25WKETS7JTCDAJVLLS3Q, seg_pos={0 0}, upload=503106]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 29 13:21:32.666330417 (573h52m32.537413462s) [noise=true , address=145.79.227.44:4193 , piece_id=WXI6KDZBHRFFM3RP75KJNO5VUGYVQZ2A25WKETS7JTCDAJVLLS3Q, seg_pos={0 0}, upload=503106]: Long tail cancellation due to optimal threshold Dec 29 13:21:32.666477617 (573h52m32.537560652s) [noise=true , address=145.79.227.30:4762 , piece_id=TVCPADCVI5J7YUVMMYPH52ARV6LGQ5M6264JV3XQCBBT6HH5NWUA, seg_pos={0 0}, upload=503106]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 29 13:21:32.666498176 (573h52m32.537581221s) [noise=true , address=145.79.227.30:4762 , piece_id=TVCPADCVI5J7YUVMMYPH52ARV6LGQ5M6264JV3XQCBBT6HH5NWUA, seg_pos={0 0}, upload=503106]: Long tail cancellation due to optimal threshold Dec 29 13:21:32.666514698 (573h52m32.537597733s) [noise=true , address=145.79.227.40:4050 , piece_id=UKN4E2FLKIGTZY23H5TQG4AKEMIMPVQQAJMR3WBQHZ4RVBVL5NCA, seg_pos={0 0}, upload=503106]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 29 13:21:32.666522653 (573h52m32.537605698s) [noise=true , address=145.79.227.40:4050 , piece_id=UKN4E2FLKIGTZY23H5TQG4AKEMIMPVQQAJMR3WBQHZ4RVBVL5NCA, seg_pos={0 0}, upload=503106]: Long tail cancellation due to optimal threshold Dec 29 13:21:32.666544575 (573h52m32.53762761s) [seg_pos={0 0}, upload=503106]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 13:21:32.666629255 (573h52m32.537712301s) [upload=503106]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 13:21:32.666636138 (573h52m32.537719183s) [upload=503106]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 13:21:32.964772775 (573h52m32.83585581s) [upload=503106]: Done uploading segments... Dec 29 13:21:32.964800187 (573h52m32.835883222s) [upload=503106]: Upload finished. err: <nil> Duration: 30.003550931s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 51 out of 60 is failed (threshold=6) [XXXXXXXXX.XXXXXXX.X.XX.XXX.XX.XXXXXXX.XX.XXXXXXXXXXX.XXXXXXX] duration: 5.004690204s trace: https://jaeger.dev.storj.io/trace/6f980a07b1938dd5 FAIL (2025-12-29 13:21:50 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.004705833s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.........X......................X....X.X....X.X.............] duration: 18.416542237s trace: https://jaeger.dev.storj.io/trace/312ddb07b2dee82a FAIL (2025-12-29 13:21:47 UTC) orbiter warning: context deadline exceeded Duration: 18.41657064s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 51 out of 60 is failed (threshold=6) [XXXXXXXXXX.XXXXXXX.X.XX.XXX.XX.XXXXXXX.XX.XXXXXXXXXXX.XXXXXX] duration: 5.001494947s trace: https://jaeger.dev.storj.io/trace/7db2c3b90d7bfb7a FAIL (2025-12-29 13:21:26 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001519724s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..........X......................X....X.X....X.X............] duration: 19.324380477s trace: https://jaeger.dev.storj.io/trace/312ddb07b2dee82a FAIL (2025-12-29 13:21:21 UTC) orbiter warning: context deadline exceeded Duration: 19.324407438s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 32 out of 60 is failed (threshold=3) [XXX..XX..X...XX.X.XX...X.X.X.XXX.X.X.X.XX..X.XXX.XX..X.XX..X] duration: 30.002569632s trace: https://jaeger.dev.storj.io/trace/6c448d09c99946d8 FAIL (2025-12-29 13:21:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: KVZWNODD4B4VBAP5U44UFDLYJCQQ55IKRAPU5P5TU676Q2RKD6RQ) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: FXJGAW4CRT3TJD7CCL4CBCT2ZUZP33YBWFEJATPXQGQ5DAAB5TYQ) context deadline exceeded ======LOGS====== Dec 29 13:20:39.207849003 (573h51m39.078932049s) [upload=503093]: Starting upload Dec 29 13:20:39.207901424 (573h51m39.078984459s) [upload=503093]: Done starting upload Dec 29 13:20:39.207936040 (573h51m39.079019075s) [upload=503093]: Uploading segments... Dec 29 13:20:39.208085825 (573h51m39.07916886s) [upload=503093]: Got next segment. Inline: false Dec 29 13:20:39.208126682 (573h51m39.079209727s) [upload=503093]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 13:20:39.208131632 (573h51m39.079214667s) [upload=503093]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 13:20:39.461378020 (573h51m39.332461066s) [seg_pos={0 0}, upload=503093]: Begin upload segment... Dec 29 13:20:39.461432785 (573h51m39.33251583s) [seg_pos={0 0}, upload=503093]: Done begin upload segment. Dec 29 13:20:39.461466088 (573h51m39.332549123s) [noise=true , address=145.79.227.20:4437 , piece_id=3CV3ZFPJOXLNOW2ZFA7RWUS2CWWEREBLOYTUNISWOSGJFB7YCYGA, seg_pos={0 0}, upload=503093]: Uploading piece... Dec 29 13:20:39.461475035 (573h51m39.33255808s) [noise=true , address=145.79.227.50:4374 , piece_id=NJXI3NIGZ4KGGO72FNFTCPUW7F44CEI6BQ53TH5BQVRPYWEA2FXQ, seg_pos={0 0}, upload=503093]: Uploading piece... Dec 29 13:20:39.462127046 (573h51m39.333210081s) [noise=true , address=45.91.116.6:4974 , piece_id=R7VFMXFKQAQ7E7NJERUE6HRDV7RHDKNHRV3EWQWHTN7QIWSDGDFQ, seg_pos={0 0}, upload=503093]: Uploading piece... Dec 29 13:20:39.462678486 (573h51m39.333761521s) [noise=true , address=145.79.227.14:4840 , piece_id=R62JOSJUCA64XCPPDO5KW6NWFLWVPZ54ABXSZN3OH5MD666YX4UA, seg_pos={0 0}, upload=503093]: Uploading piece... Dec 29 13:20:39.463211312 (573h51m39.334294357s) [noise=true , address=145.79.227.45:4231 , piece_id=RG33GQCOSFV6N5BBRGXYVU7TV6IAHFJ6FZMNTHDSZLBYAVLRME4A, seg_pos={0 0}, upload=503093]: Uploading piece... Dec 29 13:20:39.463683891 (573h51m39.334766936s) [noise=true , address=145.79.227.39:4014 , piece_id=PICDFVOCBDBNCER4HTJMOKLZYT5GFEQT5VCVSDTOEUDMVWXTY2QA, seg_pos={0 0}, upload=503093]: Uploading piece... Dec 29 13:20:39.464151332 (573h51m39.335234367s) [noise=true , address=145.79.227.43:4159 , piece_id=NTPA3SAEA7FPF22NBDQZDOZWJGQX25NI5HXVPSXU65QEBV7E2VZA, seg_pos={0 0}, upload=503093]: Uploading piece... Dec 29 13:20:39.464646785 (573h51m39.33572983s) [noise=true , address=145.79.227.15:4873 , piece_id=JXGHYEZVWBVPAVZM4TH3WSK6RHJO7MPA5JI7PWFFKH7RADVT6TMA, seg_pos={0 0}, upload=503093]: Uploading piece... Dec 29 13:20:39.465137319 (573h51m39.336220364s) [noise=true , address=145.79.227.3:4151 , piece_id=AHVOHOSHIBJF3QEMRHUXTPJHNZWHBBYNSOUTRZQR7Z4AQV64EKCQ, seg_pos={0 0}, upload=503093]: Uploading piece... Dec 29 13:20:39.466547624 (573h51m39.337630669s) [noise=true , address=145.79.227.13:4802 , piece_id=DUCGO7IMF2LR3GDPEW5TPHBCTEU57YXZQCJVSFNGQIGXG5I2A3JA, seg_pos={0 0}, upload=503093]: Uploading piece... Dec 29 13:20:39.468150696 (573h51m39.339233741s) [noise=true , address=145.79.227.4:4185 , piece_id=GVRNCIWBXCJUFEO5RTCPZDI2WAAQJ4LJPVMPJD767RZH6DC3WEGA, seg_pos={0 0}, upload=503093]: Uploading piece... Dec 29 13:20:39.470262889 (573h51m39.341345944s) [noise=true , address=77.90.31.10:4519 , piece_id=KVZWNODD4B4VBAP5U44UFDLYJCQQ55IKRAPU5P5TU676Q2RKD6RQ, seg_pos={0 0}, upload=503093]: Uploading piece... Dec 29 13:20:39.471984486 (573h51m39.343067531s) [noise=true , address=145.79.227.46:4601 , piece_id=KPK3E6GJE7NA47QXXEDYI7C7C76N4ZSKK5LRMWIRFI4RZJLS3YLQ, seg_pos={0 0}, upload=503093]: Uploading piece... Dec 29 13:20:39.473666971 (573h51m39.344750006s) [noise=true , address=45.91.116.13:4155 , piece_id=FXJGAW4CRT3TJD7CCL4CBCT2ZUZP33YBWFEJATPXQGQ5DAAB5TYQ, seg_pos={0 0}, upload=503093]: Uploading piece... Dec 29 13:20:39.475298907 (573h51m39.346381952s) [noise=true , address=145.79.227.48:4303 , piece_id=EIK2KSEPBNZ65G2KTD7PE3DARDWIQ4W22NHTDZ4A2BKGWGLCBQWQ, seg_pos={0 0}, upload=503093]: Uploading piece... Dec 29 13:20:39.477637862 (573h51m39.348720907s) [noise=true , address=77.90.31.16:4736 , piece_id=6Z2R4KCW3OJ53N6NNNUAV5S6XEFZWD4MLY3LZ7N2QYDHNZ4F7A4Q, seg_pos={0 0}, upload=503093]: Uploading piece... Dec 29 13:20:41.164407380 (573h51m41.035490415s) [upload=503093]: Next returned nil segment Dec 29 13:20:41.164420926 (573h51m41.035503971s) [upload=503093]: Waiting for error group managing segments... Dec 29 13:20:41.449718224 (573h51m41.320801259s) [noise=true , address=145.79.227.20:4437 , piece_id=3CV3ZFPJOXLNOW2ZFA7RWUS2CWWEREBLOYTUNISWOSGJFB7YCYGA, seg_pos={0 0}, upload=503093]: Done uploading piece. err: <nil> Dec 29 13:20:41.452201523 (573h51m41.323284558s) [noise=true , address=145.79.227.3:4151 , piece_id=AHVOHOSHIBJF3QEMRHUXTPJHNZWHBBYNSOUTRZQR7Z4AQV64EKCQ, seg_pos={0 0}, upload=503093]: Done uploading piece. err: <nil> Dec 29 13:20:41.452512085 (573h51m41.32359512s) [noise=true , address=145.79.227.45:4231 , piece_id=RG33GQCOSFV6N5BBRGXYVU7TV6IAHFJ6FZMNTHDSZLBYAVLRME4A, seg_pos={0 0}, upload=503093]: Done uploading piece. err: <nil> Dec 29 13:20:41.467217185 (573h51m41.33830022s) [noise=true , address=145.79.227.50:4374 , piece_id=NJXI3NIGZ4KGGO72FNFTCPUW7F44CEI6BQ53TH5BQVRPYWEA2FXQ, seg_pos={0 0}, upload=503093]: Done uploading piece. err: <nil> Dec 29 13:20:41.468366664 (573h51m41.339449699s) [noise=true , address=145.79.227.48:4303 , piece_id=EIK2KSEPBNZ65G2KTD7PE3DARDWIQ4W22NHTDZ4A2BKGWGLCBQWQ, seg_pos={0 0}, upload=503093]: Done uploading piece. err: <nil> Dec 29 13:20:41.469643274 (573h51m41.340726319s) [noise=true , address=145.79.227.4:4185 , piece_id=GVRNCIWBXCJUFEO5RTCPZDI2WAAQJ4LJPVMPJD767RZH6DC3WEGA, seg_pos={0 0}, upload=503093]: Done uploading piece. err: <nil> Dec 29 13:20:41.472638618 (573h51m41.343721653s) [noise=true , address=145.79.227.14:4840 , piece_id=R62JOSJUCA64XCPPDO5KW6NWFLWVPZ54ABXSZN3OH5MD666YX4UA, seg_pos={0 0}, upload=503093]: Done uploading piece. err: <nil> Dec 29 13:20:41.478536899 (573h51m41.349619934s) [noise=true , address=145.79.227.39:4014 , piece_id=PICDFVOCBDBNCER4HTJMOKLZYT5GFEQT5VCVSDTOEUDMVWXTY2QA, seg_pos={0 0}, upload=503093]: Done uploading piece. err: <nil> Dec 29 13:20:41.481285433 (573h51m41.352368468s) [noise=true , address=145.79.227.46:4601 , piece_id=KPK3E6GJE7NA47QXXEDYI7C7C76N4ZSKK5LRMWIRFI4RZJLS3YLQ, seg_pos={0 0}, upload=503093]: Done uploading piece. err: <nil> Dec 29 13:20:41.488632162 (573h51m41.359715207s) [noise=true , address=145.79.227.15:4873 , piece_id=JXGHYEZVWBVPAVZM4TH3WSK6RHJO7MPA5JI7PWFFKH7RADVT6TMA, seg_pos={0 0}, upload=503093]: Done uploading piece. err: <nil> Dec 29 13:20:41.494954481 (573h51m41.366037516s) [noise=true , address=145.79.227.13:4802 , piece_id=DUCGO7IMF2LR3GDPEW5TPHBCTEU57YXZQCJVSFNGQIGXG5I2A3JA, seg_pos={0 0}, upload=503093]: Done uploading piece. err: <nil> Dec 29 13:20:41.819654888 (573h51m41.690737923s) [noise=true , address=45.91.116.13:4155 , piece_id=FXJGAW4CRT3TJD7CCL4CBCT2ZUZP33YBWFEJATPXQGQ5DAAB5TYQ, seg_pos={0 0}, upload=503093]: Done uploading piece. err: <nil> Dec 29 13:20:42.501243663 (573h51m42.372326698s) [noise=true , address=77.90.31.10:4519 , piece_id=KVZWNODD4B4VBAP5U44UFDLYJCQQ55IKRAPU5P5TU676Q2RKD6RQ, seg_pos={0 0}, upload=503093]: Done uploading piece. err: <nil> Dec 29 13:20:42.501274792 (573h51m42.372357827s) [seg_pos={0 0}, upload=503093]: Segment reached optimal threshold of 13 pieces. Dec 29 13:20:42.501471597 (573h51m42.372554632s) [noise=true , address=145.79.227.43:4159 , piece_id=NTPA3SAEA7FPF22NBDQZDOZWJGQX25NI5HXVPSXU65QEBV7E2VZA, seg_pos={0 0}, upload=503093]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Dec 29 13:20:42.501478040 (573h51m42.372561085s) [noise=true , address=145.79.227.43:4159 , piece_id=NTPA3SAEA7FPF22NBDQZDOZWJGQX25NI5HXVPSXU65QEBV7E2VZA, seg_pos={0 0}, upload=503093]: Long tail cancellation due to optimal threshold Dec 29 13:20:42.501639937 (573h51m42.372722972s) [noise=true , address=77.90.31.16:4736 , piece_id=6Z2R4KCW3OJ53N6NNNUAV5S6XEFZWD4MLY3LZ7N2QYDHNZ4F7A4Q, seg_pos={0 0}, upload=503093]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 29 13:20:42.501658333 (573h51m42.372741368s) [noise=true , address=77.90.31.16:4736 , piece_id=6Z2R4KCW3OJ53N6NNNUAV5S6XEFZWD4MLY3LZ7N2QYDHNZ4F7A4Q, seg_pos={0 0}, upload=503093]: Long tail cancellation due to optimal threshold Dec 29 13:20:42.501680204 (573h51m42.372763249s) [noise=true , address=45.91.116.6:4974 , piece_id=R7VFMXFKQAQ7E7NJERUE6HRDV7RHDKNHRV3EWQWHTN7QIWSDGDFQ, seg_pos={0 0}, upload=503093]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 29 13:20:42.501687178 (573h51m42.372770223s) [noise=true , address=45.91.116.6:4974 , piece_id=R7VFMXFKQAQ7E7NJERUE6HRDV7RHDKNHRV3EWQWHTN7QIWSDGDFQ, seg_pos={0 0}, upload=503093]: Long tail cancellation due to optimal threshold Dec 29 13:20:42.501732494 (573h51m42.372815529s) [seg_pos={0 0}, upload=503093]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 13:20:42.501829308 (573h51m42.372912343s) [upload=503093]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 13:20:42.501834648 (573h51m42.372917693s) [upload=503093]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 13:20:42.820941104 (573h51m42.692024159s) [upload=503093]: Done uploading segments... Dec 29 13:20:42.820969999 (573h51m42.692053045s) [upload=503093]: Upload finished. err: <nil> Duration: 30.002591934s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....................X.....X.....................XX.........] duration: 51.877989757s trace: https://jaeger.dev.storj.io/trace/77d26141ea11d228 FAIL (2025-12-29 13:21:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12J2weU7Bmsr3RdQQ9QMuadT96XReBkyxTeaLJ2SEqVGGFX1PWu, Piece ID: JITX25FNZ4JIKYXQY345UX3SQUCPDWJ25NJQZ2TXGFRLVCRYXUOQ) context deadline exceeded; piecestore: (Node ID: 125sm6iHV94mJFLWrKeyRMH4b1qSMo32nDuW8uQr71xUL7Cema2, Piece ID: INTVU2EYTU3DIM6WQFQEGO6IEYH3YFP5U2VEN7A5KPTOKU2DWOOA) context deadline exceeded; piecestore: (Node ID: 12i57FF97oEddYLZpNxSq5or9WLEquL8ZW1cVUuW4AThBbDqcCL, Piece ID: Q35LXGPTOAG3R2UL3JLL7DNHHD4PSMEULYW6G3SPXKNVFZ5VTNRA) context deadline exceeded ======LOGS====== Q2LO7MUXLHE5YET7BHLLA, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.692912334 (573h43m3.563995379s) [noise=true , address=77.90.31.16:4759 , piece_id=ARC4HRP57U5VQXZTT63QYLXBPZURI5VMJBIQZOFKGLJBQ3K3KGLQ, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.693322595 (573h43m3.56440564s) [noise=true , address=77.90.31.15:4711 , piece_id=JITX25FNZ4JIKYXQY345UX3SQUCPDWJ25NJQZ2TXGFRLVCRYXUOQ, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.693739649 (573h43m3.564822694s) [noise=true , address=45.91.116.6:4981 , piece_id=INTVU2EYTU3DIM6WQFQEGO6IEYH3YFP5U2VEN7A5KPTOKU2DWOOA, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.694315887 (573h43m3.565398942s) [noise=true , address=145.79.227.4:4220 , piece_id=SHHJ7EW7DTNOZXVBP6A64VIX7P6AQN62TBCUT3KTH4MIB6UXY3BA, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.694765222 (573h43m3.565848267s) [noise=true , address=145.79.227.29:4952 , piece_id=DKYDB2R4DNCGH6T3SLRR2MOZZUPVZZVRL2BKGYOBYXJJVPSZEJ4Q, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.695202545 (573h43m3.56628559s) [noise=true , address=145.79.227.40:4058 , piece_id=KRHH2QZKSDPWTSSKYDVDWGKFJD3LPGP6U7KI2PPJ5T3TXYIUGJWA, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.695687528 (573h43m3.566770584s) [noise=true , address=77.90.31.19:4857 , piece_id=Q35LXGPTOAG3R2UL3JLL7DNHHD4PSMEULYW6G3SPXKNVFZ5VTNRA, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.696135221 (573h43m3.567218276s) [noise=true , address=45.91.116.8:4051 , piece_id=GDJFLQCHUF7BFQ7BVSNTVJ3TYZHYVOO6YYMO3KUIUSIEPAF4SFDQ, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.696703733 (573h43m3.567786778s) [noise=true , address=64.49.13.170:4720 , piece_id=HF6X3OT2EPPDE3DWAPWABVZUR3ERDBWXHG3NGNKEWZAD7G5AN4TA, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.698418298 (573h43m3.569501343s) [noise=true , address=145.79.227.36:4520 , piece_id=MYJSXUH563JFUW5Q7YUIWL7466G6ASY5WXR3T2PGUFFCTRTOBYJA, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.700198829 (573h43m3.571281874s) [noise=true , address=77.90.31.18:4821 , piece_id=LO5W47FHYOVGY5Q6BH5L6KX77HES3XRAFUTPI45HYPJZDO3YHLCQ, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.701993145 (573h43m3.57307619s) [noise=true , address=193.149.164.143:4699 , piece_id=UDFDCYO25MOJBTMDE6F2WILXE3HLOB75UJZ4LW7ZMNSCOBJI6UJA, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.703884687 (573h43m3.574967732s) [noise=true , address=145.79.227.33:4893 , piece_id=XGLX3DFNAY4F4GPGCGZ4ONDJ7NK62SAKX3PBRWHWI74EF3ZFMUSA, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.705687109 (573h43m3.576770154s) [noise=true , address=45.91.116.10:4094 , piece_id=6UUIZNTESBBWJMDQ5ZTCHU7S6YKM4GDKO4ZAC6T2FUTPVGFFDS3A, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.707557651 (573h43m3.578640696s) [noise=true , address=145.79.227.43:4164 , piece_id=J6Q4ZDOIIIBA2FXGZZTD6AYS7KGAWSRXXUBAPDPQWYKNU5GNXBBQ, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:05.168247719 (573h43m5.039330754s) [upload=502975]: Next returned nil segment Dec 29 13:12:05.168259962 (573h43m5.039343007s) [upload=502975]: Waiting for error group managing segments... Dec 29 13:12:05.558873127 (573h43m5.429956162s) [noise=true , address=145.79.227.4:4220 , piece_id=SHHJ7EW7DTNOZXVBP6A64VIX7P6AQN62TBCUT3KTH4MIB6UXY3BA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:05.563024312 (573h43m5.434107347s) [noise=true , address=64.49.13.170:4720 , piece_id=HF6X3OT2EPPDE3DWAPWABVZUR3ERDBWXHG3NGNKEWZAD7G5AN4TA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:05.564115719 (573h43m5.435198755s) [noise=true , address=145.79.227.36:4520 , piece_id=MYJSXUH563JFUW5Q7YUIWL7466G6ASY5WXR3T2PGUFFCTRTOBYJA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:05.593873402 (573h43m5.464956447s) [noise=true , address=145.79.227.29:4952 , piece_id=DKYDB2R4DNCGH6T3SLRR2MOZZUPVZZVRL2BKGYOBYXJJVPSZEJ4Q, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:05.898194604 (573h43m5.769277639s) [noise=true , address=45.91.116.10:4094 , piece_id=6UUIZNTESBBWJMDQ5ZTCHU7S6YKM4GDKO4ZAC6T2FUTPVGFFDS3A, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:05.912845292 (573h43m5.783928337s) [noise=true , address=77.90.31.19:4857 , piece_id=Q35LXGPTOAG3R2UL3JLL7DNHHD4PSMEULYW6G3SPXKNVFZ5VTNRA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:06.426532207 (573h43m6.297615252s) [noise=true , address=193.149.164.143:4699 , piece_id=UDFDCYO25MOJBTMDE6F2WILXE3HLOB75UJZ4LW7ZMNSCOBJI6UJA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:06.449589562 (573h43m6.320672607s) [noise=true , address=77.90.31.15:4711 , piece_id=JITX25FNZ4JIKYXQY345UX3SQUCPDWJ25NJQZ2TXGFRLVCRYXUOQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:06.453186841 (573h43m6.324269876s) [noise=true , address=145.79.227.40:4058 , piece_id=KRHH2QZKSDPWTSSKYDVDWGKFJD3LPGP6U7KI2PPJ5T3TXYIUGJWA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:06.510769276 (573h43m6.381852321s) [noise=true , address=145.79.227.43:4164 , piece_id=J6Q4ZDOIIIBA2FXGZZTD6AYS7KGAWSRXXUBAPDPQWYKNU5GNXBBQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:06.518868879 (573h43m6.389951924s) [noise=true , address=145.79.227.33:4893 , piece_id=XGLX3DFNAY4F4GPGCGZ4ONDJ7NK62SAKX3PBRWHWI74EF3ZFMUSA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:06.519238043 (573h43m6.390321098s) [noise=true , address=45.91.116.6:4981 , piece_id=INTVU2EYTU3DIM6WQFQEGO6IEYH3YFP5U2VEN7A5KPTOKU2DWOOA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:06.523211328 (573h43m6.394294373s) [noise=true , address=77.90.31.16:4759 , piece_id=ARC4HRP57U5VQXZTT63QYLXBPZURI5VMJBIQZOFKGLJBQ3K3KGLQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:06.767869783 (573h43m6.638952818s) [noise=true , address=145.79.227.34:4924 , piece_id=MAQXQTAGXQ7AEPUCESUSUGMHSDMMXS7HHKAEA7B3HW5X66PDUF5A, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:06.767889431 (573h43m6.638972476s) [seg_pos={0 0}, upload=502975]: Segment reached optimal threshold of 14 pieces. Dec 29 13:12:06.768102116 (573h43m6.639185161s) [noise=true , address=45.91.116.8:4051 , piece_id=GDJFLQCHUF7BFQ7BVSNTVJ3TYZHYVOO6YYMO3KUIUSIEPAF4SFDQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1preT33ei4yVXXYVLHhaUbHbiyL14oHFEiof6cCWPig83W6CT3): protocol: expected piece hash; context canceled; EOF Dec 29 13:12:06.768108257 (573h43m6.639191302s) [noise=true , address=45.91.116.8:4051 , piece_id=GDJFLQCHUF7BFQ7BVSNTVJ3TYZHYVOO6YYMO3KUIUSIEPAF4SFDQ, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold Dec 29 13:12:06.768219208 (573h43m6.639302254s) [noise=true , address=77.90.31.18:4821 , piece_id=LO5W47FHYOVGY5Q6BH5L6KX77HES3XRAFUTPI45HYPJZDO3YHLCQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH): protocol: expected piece hash; context canceled; EOF Dec 29 13:12:06.768224789 (573h43m6.639307824s) [noise=true , address=77.90.31.18:4821 , piece_id=LO5W47FHYOVGY5Q6BH5L6KX77HES3XRAFUTPI45HYPJZDO3YHLCQ, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold Dec 29 13:12:06.768384213 (573h43m6.639467258s) [noise=true , address=64.49.13.169:4711 , piece_id=4F4T4YKXMT7NF2C5KBUGJHQJGZL4VCKNKYB266PCVI3UACHJXZ5Q, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:48802->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:48802->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48802->64.49.13.169:4711: use of closed network connection Dec 29 13:12:06.768399261 (573h43m6.639482296s) [noise=true , address=64.49.13.169:4711 , piece_id=4F4T4YKXMT7NF2C5KBUGJHQJGZL4VCKNKYB266PCVI3UACHJXZ5Q, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold Dec 29 13:12:06.768732636 (573h43m6.639815671s) [noise=true , address=64.49.13.169:4709 , piece_id=TBHHX5ES2A6NQRGP25HIC7J7IBWLOJGQ2LO7MUXLHE5YET7BHLLA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:35310->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:35310->64.49.13.169:4709: use of closed network connection Dec 29 13:12:06.768745460 (573h43m6.639828505s) [noise=true , address=64.49.13.169:4709 , piece_id=TBHHX5ES2A6NQRGP25HIC7J7IBWLOJGQ2LO7MUXLHE5YET7BHLLA, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold Dec 29 13:12:06.768784586 (573h43m6.639867631s) [seg_pos={0 0}, upload=502975]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 13:12:06.768880217 (573h43m6.639963252s) [upload=502975]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 13:12:06.768884906 (573h43m6.639967941s) [upload=502975]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 13:12:07.067188330 (573h43m6.938271365s) [upload=502975]: Done uploading segments... Dec 29 13:12:07.067251099 (573h43m6.938334134s) [upload=502975]: Upload finished. err: <nil> Duration: 51.878008183s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 51 out of 60 is failed (threshold=6) [XXXXXXXXXXX.XXXXXXX.X.XX.XXX.XX.XXXXXXX.XX.XXXXXXXXXXX.XXXXX] duration: 5.001214882s trace: https://jaeger.dev.storj.io/trace/7bc5af7a12811fa6 FAIL (2025-12-29 13:20:47 UTC) orbiter warning: context deadline exceeded Duration: 5.001237795s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...........X......................X....X.X....X.X...........] duration: 23.509175471s trace: https://jaeger.dev.storj.io/trace/312ddb07b2dee82a FAIL (2025-12-29 13:20:24 UTC) orbiter warning: context deadline exceeded Duration: 23.509202813s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 51 out of 60 is failed (threshold=6) [XXXXXXXXXXXX.XXXXXXX.X.XX.XXX.XX.XXXXXXX.XX.XXXXXXXXXXX.XXXX] duration: 5.001948635s trace: https://jaeger.dev.storj.io/trace/a55d2040d97b756 FAIL (2025-12-29 13:20:03 UTC) orbiter warning: context deadline exceeded Duration: 5.001974895s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 32 out of 60 is failed (threshold=3) [XXXX..XX..X...XX.X.XX...X.X.X.XXX.X.X.X.XX..X.XXX.XX..X.XX..] duration: 5.637049539s trace: https://jaeger.dev.storj.io/trace/7dfc7d7e2d71a3fc FAIL (2025-12-29 13:19:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, Piece ID: 5F7YBUUFPMX6WQRAB7GVZ3GROFYM6FB2CPORE7PTVJTFLGDS4VXA) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: HVDX2NNTTOS5IO44Z7BICXLYG3USC47LHJJNX6EGJ4QV2FMX5VXQ) context deadline exceeded ======LOGS====== 29 13:17:39.540175545 (573h48m39.41125858s) [upload=503053]: Starting upload Dec 29 13:17:39.540229057 (573h48m39.411312092s) [upload=503053]: Done starting upload Dec 29 13:17:39.540441481 (573h48m39.411524526s) [upload=503053]: Uploading segments... Dec 29 13:17:39.540482360 (573h48m39.411565395s) [upload=503053]: Got next segment. Inline: false Dec 29 13:17:39.540511384 (573h48m39.411594419s) [upload=503053]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 13:17:39.540515733 (573h48m39.411598768s) [upload=503053]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 13:17:39.790547090 (573h48m39.661630125s) [seg_pos={0 0}, upload=503053]: Begin upload segment... Dec 29 13:17:39.790596244 (573h48m39.661679279s) [seg_pos={0 0}, upload=503053]: Done begin upload segment. Dec 29 13:17:39.790639095 (573h48m39.66172213s) [noise=true , address=45.91.116.11:4117 , piece_id=XBP7IUQXU5IKSX2JV7SS6XUJPFY5R2BQWYGM4ZJMO3NQ5SRG4I5A, seg_pos={0 0}, upload=503053]: Uploading piece... Dec 29 13:17:39.790757290 (573h48m39.661840325s) [noise=true , address=145.79.227.11:4732 , piece_id=N3SNFHQOB5ZJVTBESQBHHIDFJWYYHMGKL6V72TE2GPZPP32RGLXA, seg_pos={0 0}, upload=503053]: Uploading piece... Dec 29 13:17:39.791333458 (573h48m39.662416503s) [noise=true , address=145.79.227.20:4437 , piece_id=VKQI5YRGFKVELOMSFCAMAUUBW4KIQK3OV4JZPSNTM2CK6E6QOLBQ, seg_pos={0 0}, upload=503053]: Uploading piece... Dec 29 13:17:39.791640912 (573h48m39.662723947s) [noise=true , address=45.91.116.7:4009 , piece_id=W7DI7QVHOCHPWJYAYJOAYY6UIM3R67573GNZIDELY5F33IUXEWVA, seg_pos={0 0}, upload=503053]: Uploading piece... Dec 29 13:17:39.791929422 (573h48m39.663012457s) [noise=true , address=45.91.116.12:4225 , piece_id=QMVQJN5K4OTICS5P6XIZ7S4AOS4YGMLVH3FQ5U6FNZVSVS5FTDMQ, seg_pos={0 0}, upload=503053]: Uploading piece... Dec 29 13:17:39.792206600 (573h48m39.663289635s) [noise=true , address=45.91.116.6:4974 , piece_id=ZDDCURSLZ4246UHAL7AWSCPWTQ5YD7VAZ6VHAFST6MGFQCQGSOGQ, seg_pos={0 0}, upload=503053]: Uploading piece... Dec 29 13:17:39.792601952 (573h48m39.663684998s) [noise=true , address=145.79.227.38:4978 , piece_id=IZCZJWZ4QLTCH6BUSKY6CULNP2ULDMQIOFMCMM5SR3CHF6AS5ULA, seg_pos={0 0}, upload=503053]: Uploading piece... Dec 29 13:17:39.792950867 (573h48m39.664033912s) [noise=true , address=77.90.31.11:4553 , piece_id=5F7YBUUFPMX6WQRAB7GVZ3GROFYM6FB2CPORE7PTVJTFLGDS4VXA, seg_pos={0 0}, upload=503053]: Uploading piece... Dec 29 13:17:39.793476237 (573h48m39.664559282s) [noise=true , address=145.79.227.18:4006 , piece_id=O3WIX5QNXOMB5BRDCCNU5ZL62V4AT4MVWE3MU2GE6GIJ7TJYYDJA, seg_pos={0 0}, upload=503053]: Uploading piece... Dec 29 13:17:39.794762296 (573h48m39.665845341s) [noise=true , address=77.90.31.12:4589 , piece_id=3HJXH4YFU427ZWGKR3AQVGIMKWI2FLOCUBLG3P3ZG2BGR3WQ632A, seg_pos={0 0}, upload=503053]: Uploading piece... Dec 29 13:17:39.796190105 (573h48m39.66727315s) [noise=true , address=45.91.116.5:4299 , piece_id=HVDX2NNTTOS5IO44Z7BICXLYG3USC47LHJJNX6EGJ4QV2FMX5VXQ, seg_pos={0 0}, upload=503053]: Uploading piece... Dec 29 13:17:39.797256144 (573h48m39.668339189s) [noise=true , address=145.79.227.19:4077 , piece_id=G72N37Z4FKLBI46XHLVCJBNGKOT7AKZO5Y4GASBIPRCBNVP342OA, seg_pos={0 0}, upload=503053]: Uploading piece... Dec 29 13:17:39.798662533 (573h48m39.669745578s) [noise=true , address=145.79.227.16:4909 , piece_id=CRN2XGVACJY2E44PIS4OGG3WDIPXVTNLHN4TMYJJYEDS3XEBWPCQ, seg_pos={0 0}, upload=503053]: Uploading piece... Dec 29 13:17:39.799977255 (573h48m39.671060291s) [noise=true , address=145.79.227.4:4185 , piece_id=S4QJUU4G3DEFP2SYWCSJJPIAMRQQHZBEP6OKSPB4PNPTV6CZHTCQ, seg_pos={0 0}, upload=503053]: Uploading piece... Dec 29 13:17:39.804730256 (573h48m39.675813292s) [noise=true , address=145.79.227.51:4411 , piece_id=GQ3HRCSQWIHBBXLCDFO3BEP566L57JENKH7AZUO62J5VZ6YFCC7Q, seg_pos={0 0}, upload=503053]: Uploading piece... Dec 29 13:17:39.806081600 (573h48m39.677164645s) [noise=true , address=145.79.227.3:4149 , piece_id=VIKUB5FR3ZOV2B27BC3ZW3Z7PQSHP75TPB5NIJNOMWLX7FCUDRVQ, seg_pos={0 0}, upload=503053]: Uploading piece... Dec 29 13:17:41.493481509 (573h48m41.364564554s) [upload=503053]: Next returned nil segment Dec 29 13:17:41.493502690 (573h48m41.364585735s) [upload=503053]: Waiting for error group managing segments... Dec 29 13:17:41.772207224 (573h48m41.643290269s) [noise=true , address=145.79.227.19:4077 , piece_id=G72N37Z4FKLBI46XHLVCJBNGKOT7AKZO5Y4GASBIPRCBNVP342OA, seg_pos={0 0}, upload=503053]: Done uploading piece. err: <nil> Dec 29 13:17:41.782029908 (573h48m41.653112943s) [noise=true , address=145.79.227.11:4732 , piece_id=N3SNFHQOB5ZJVTBESQBHHIDFJWYYHMGKL6V72TE2GPZPP32RGLXA, seg_pos={0 0}, upload=503053]: Done uploading piece. err: <nil> Dec 29 13:17:41.787864948 (573h48m41.658947993s) [noise=true , address=145.79.227.16:4909 , piece_id=CRN2XGVACJY2E44PIS4OGG3WDIPXVTNLHN4TMYJJYEDS3XEBWPCQ, seg_pos={0 0}, upload=503053]: Done uploading piece. err: <nil> Dec 29 13:17:41.793494689 (573h48m41.664577724s) [noise=true , address=145.79.227.4:4185 , piece_id=S4QJUU4G3DEFP2SYWCSJJPIAMRQQHZBEP6OKSPB4PNPTV6CZHTCQ, seg_pos={0 0}, upload=503053]: Done uploading piece. err: <nil> Dec 29 13:17:41.800966757 (573h48m41.672049802s) [noise=true , address=145.79.227.18:4006 , piece_id=O3WIX5QNXOMB5BRDCCNU5ZL62V4AT4MVWE3MU2GE6GIJ7TJYYDJA, seg_pos={0 0}, upload=503053]: Done uploading piece. err: <nil> Dec 29 13:17:41.803150414 (573h48m41.674233459s) [noise=true , address=145.79.227.3:4149 , piece_id=VIKUB5FR3ZOV2B27BC3ZW3Z7PQSHP75TPB5NIJNOMWLX7FCUDRVQ, seg_pos={0 0}, upload=503053]: Done uploading piece. err: <nil> Dec 29 13:17:41.805106228 (573h48m41.676189263s) [noise=true , address=145.79.227.38:4978 , piece_id=IZCZJWZ4QLTCH6BUSKY6CULNP2ULDMQIOFMCMM5SR3CHF6AS5ULA, seg_pos={0 0}, upload=503053]: Done uploading piece. err: <nil> Dec 29 13:17:41.808591064 (573h48m41.679674099s) [noise=true , address=145.79.227.20:4437 , piece_id=VKQI5YRGFKVELOMSFCAMAUUBW4KIQK3OV4JZPSNTM2CK6E6QOLBQ, seg_pos={0 0}, upload=503053]: Done uploading piece. err: <nil> Dec 29 13:17:41.822363249 (573h48m41.693446294s) [noise=true , address=145.79.227.51:4411 , piece_id=GQ3HRCSQWIHBBXLCDFO3BEP566L57JENKH7AZUO62J5VZ6YFCC7Q, seg_pos={0 0}, upload=503053]: Done uploading piece. err: <nil> Dec 29 13:17:42.047382889 (573h48m41.918465934s) [noise=true , address=77.90.31.11:4553 , piece_id=5F7YBUUFPMX6WQRAB7GVZ3GROFYM6FB2CPORE7PTVJTFLGDS4VXA, seg_pos={0 0}, upload=503053]: Done uploading piece. err: <nil> Dec 29 13:17:42.147248348 (573h48m42.018331384s) [noise=true , address=45.91.116.5:4299 , piece_id=HVDX2NNTTOS5IO44Z7BICXLYG3USC47LHJJNX6EGJ4QV2FMX5VXQ, seg_pos={0 0}, upload=503053]: Done uploading piece. err: <nil> Dec 29 13:17:42.196027221 (573h48m42.067110256s) [noise=true , address=77.90.31.12:4589 , piece_id=3HJXH4YFU427ZWGKR3AQVGIMKWI2FLOCUBLG3P3ZG2BGR3WQ632A, seg_pos={0 0}, upload=503053]: Done uploading piece. err: <nil> Dec 29 13:17:42.216749780 (573h48m42.087832815s) [noise=true , address=45.91.116.12:4225 , piece_id=QMVQJN5K4OTICS5P6XIZ7S4AOS4YGMLVH3FQ5U6FNZVSVS5FTDMQ, seg_pos={0 0}, upload=503053]: Done uploading piece. err: <nil> Dec 29 13:17:42.216771340 (573h48m42.087854375s) [seg_pos={0 0}, upload=503053]: Segment reached optimal threshold of 13 pieces. Dec 29 13:17:42.216982773 (573h48m42.088065828s) [noise=true , address=45.91.116.7:4009 , piece_id=W7DI7QVHOCHPWJYAYJOAYY6UIM3R67573GNZIDELY5F33IUXEWVA, seg_pos={0 0}, upload=503053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 29 13:17:42.216994435 (573h48m42.088077481s) [noise=true , address=45.91.116.7:4009 , piece_id=W7DI7QVHOCHPWJYAYJOAYY6UIM3R67573GNZIDELY5F33IUXEWVA, seg_pos={0 0}, upload=503053]: Long tail cancellation due to optimal threshold Dec 29 13:17:42.217255532 (573h48m42.088338567s) [noise=true , address=45.91.116.6:4974 , piece_id=ZDDCURSLZ4246UHAL7AWSCPWTQ5YD7VAZ6VHAFST6MGFQCQGSOGQ, seg_pos={0 0}, upload=503053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:50160->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:50160->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50160->45.91.116.6:4974: use of closed network connection Dec 29 13:17:42.217262256 (573h48m42.088345301s) [noise=true , address=45.91.116.6:4974 , piece_id=ZDDCURSLZ4246UHAL7AWSCPWTQ5YD7VAZ6VHAFST6MGFQCQGSOGQ, seg_pos={0 0}, upload=503053]: Long tail cancellation due to optimal threshold Dec 29 13:17:42.217422140 (573h48m42.088505205s) [noise=true , address=45.91.116.11:4117 , piece_id=XBP7IUQXU5IKSX2JV7SS6XUJPFY5R2BQWYGM4ZJMO3NQ5SRG4I5A, seg_pos={0 0}, upload=503053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF Dec 29 13:17:42.217442519 (573h48m42.088525564s) [noise=true , address=45.91.116.11:4117 , piece_id=XBP7IUQXU5IKSX2JV7SS6XUJPFY5R2BQWYGM4ZJMO3NQ5SRG4I5A, seg_pos={0 0}, upload=503053]: Long tail cancellation due to optimal threshold Dec 29 13:17:42.217487414 (573h48m42.088570449s) [seg_pos={0 0}, upload=503053]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 13:17:42.217601521 (573h48m42.088684556s) [upload=503053]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 13:17:42.217608554 (573h48m42.088691599s) [upload=503053]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 13:17:42.510405140 (573h48m42.381488185s) [upload=503053]: Done uploading segments... Dec 29 13:17:42.510430329 (573h48m42.381513364s) [upload=503053]: Upload finished. err: <nil> Duration: 5.637078203s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [......................X.....X.....................XX........] duration: 40.28776165s trace: https://jaeger.dev.storj.io/trace/77d26141ea11d228 FAIL (2025-12-29 13:19:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12J2weU7Bmsr3RdQQ9QMuadT96XReBkyxTeaLJ2SEqVGGFX1PWu, Piece ID: JITX25FNZ4JIKYXQY345UX3SQUCPDWJ25NJQZ2TXGFRLVCRYXUOQ) context deadline exceeded; piecestore: (Node ID: 125sm6iHV94mJFLWrKeyRMH4b1qSMo32nDuW8uQr71xUL7Cema2, Piece ID: INTVU2EYTU3DIM6WQFQEGO6IEYH3YFP5U2VEN7A5KPTOKU2DWOOA) context deadline exceeded; piecestore: (Node ID: 12i57FF97oEddYLZpNxSq5or9WLEquL8ZW1cVUuW4AThBbDqcCL, Piece ID: Q35LXGPTOAG3R2UL3JLL7DNHHD4PSMEULYW6G3SPXKNVFZ5VTNRA) context deadline exceeded ======LOGS====== Q2LO7MUXLHE5YET7BHLLA, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.692912334 (573h43m3.563995379s) [noise=true , address=77.90.31.16:4759 , piece_id=ARC4HRP57U5VQXZTT63QYLXBPZURI5VMJBIQZOFKGLJBQ3K3KGLQ, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.693322595 (573h43m3.56440564s) [noise=true , address=77.90.31.15:4711 , piece_id=JITX25FNZ4JIKYXQY345UX3SQUCPDWJ25NJQZ2TXGFRLVCRYXUOQ, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.693739649 (573h43m3.564822694s) [noise=true , address=45.91.116.6:4981 , piece_id=INTVU2EYTU3DIM6WQFQEGO6IEYH3YFP5U2VEN7A5KPTOKU2DWOOA, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.694315887 (573h43m3.565398942s) [noise=true , address=145.79.227.4:4220 , piece_id=SHHJ7EW7DTNOZXVBP6A64VIX7P6AQN62TBCUT3KTH4MIB6UXY3BA, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.694765222 (573h43m3.565848267s) [noise=true , address=145.79.227.29:4952 , piece_id=DKYDB2R4DNCGH6T3SLRR2MOZZUPVZZVRL2BKGYOBYXJJVPSZEJ4Q, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.695202545 (573h43m3.56628559s) [noise=true , address=145.79.227.40:4058 , piece_id=KRHH2QZKSDPWTSSKYDVDWGKFJD3LPGP6U7KI2PPJ5T3TXYIUGJWA, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.695687528 (573h43m3.566770584s) [noise=true , address=77.90.31.19:4857 , piece_id=Q35LXGPTOAG3R2UL3JLL7DNHHD4PSMEULYW6G3SPXKNVFZ5VTNRA, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.696135221 (573h43m3.567218276s) [noise=true , address=45.91.116.8:4051 , piece_id=GDJFLQCHUF7BFQ7BVSNTVJ3TYZHYVOO6YYMO3KUIUSIEPAF4SFDQ, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.696703733 (573h43m3.567786778s) [noise=true , address=64.49.13.170:4720 , piece_id=HF6X3OT2EPPDE3DWAPWABVZUR3ERDBWXHG3NGNKEWZAD7G5AN4TA, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.698418298 (573h43m3.569501343s) [noise=true , address=145.79.227.36:4520 , piece_id=MYJSXUH563JFUW5Q7YUIWL7466G6ASY5WXR3T2PGUFFCTRTOBYJA, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.700198829 (573h43m3.571281874s) [noise=true , address=77.90.31.18:4821 , piece_id=LO5W47FHYOVGY5Q6BH5L6KX77HES3XRAFUTPI45HYPJZDO3YHLCQ, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.701993145 (573h43m3.57307619s) [noise=true , address=193.149.164.143:4699 , piece_id=UDFDCYO25MOJBTMDE6F2WILXE3HLOB75UJZ4LW7ZMNSCOBJI6UJA, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.703884687 (573h43m3.574967732s) [noise=true , address=145.79.227.33:4893 , piece_id=XGLX3DFNAY4F4GPGCGZ4ONDJ7NK62SAKX3PBRWHWI74EF3ZFMUSA, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.705687109 (573h43m3.576770154s) [noise=true , address=45.91.116.10:4094 , piece_id=6UUIZNTESBBWJMDQ5ZTCHU7S6YKM4GDKO4ZAC6T2FUTPVGFFDS3A, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.707557651 (573h43m3.578640696s) [noise=true , address=145.79.227.43:4164 , piece_id=J6Q4ZDOIIIBA2FXGZZTD6AYS7KGAWSRXXUBAPDPQWYKNU5GNXBBQ, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:05.168247719 (573h43m5.039330754s) [upload=502975]: Next returned nil segment Dec 29 13:12:05.168259962 (573h43m5.039343007s) [upload=502975]: Waiting for error group managing segments... Dec 29 13:12:05.558873127 (573h43m5.429956162s) [noise=true , address=145.79.227.4:4220 , piece_id=SHHJ7EW7DTNOZXVBP6A64VIX7P6AQN62TBCUT3KTH4MIB6UXY3BA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:05.563024312 (573h43m5.434107347s) [noise=true , address=64.49.13.170:4720 , piece_id=HF6X3OT2EPPDE3DWAPWABVZUR3ERDBWXHG3NGNKEWZAD7G5AN4TA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:05.564115719 (573h43m5.435198755s) [noise=true , address=145.79.227.36:4520 , piece_id=MYJSXUH563JFUW5Q7YUIWL7466G6ASY5WXR3T2PGUFFCTRTOBYJA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:05.593873402 (573h43m5.464956447s) [noise=true , address=145.79.227.29:4952 , piece_id=DKYDB2R4DNCGH6T3SLRR2MOZZUPVZZVRL2BKGYOBYXJJVPSZEJ4Q, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:05.898194604 (573h43m5.769277639s) [noise=true , address=45.91.116.10:4094 , piece_id=6UUIZNTESBBWJMDQ5ZTCHU7S6YKM4GDKO4ZAC6T2FUTPVGFFDS3A, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:05.912845292 (573h43m5.783928337s) [noise=true , address=77.90.31.19:4857 , piece_id=Q35LXGPTOAG3R2UL3JLL7DNHHD4PSMEULYW6G3SPXKNVFZ5VTNRA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:06.426532207 (573h43m6.297615252s) [noise=true , address=193.149.164.143:4699 , piece_id=UDFDCYO25MOJBTMDE6F2WILXE3HLOB75UJZ4LW7ZMNSCOBJI6UJA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:06.449589562 (573h43m6.320672607s) [noise=true , address=77.90.31.15:4711 , piece_id=JITX25FNZ4JIKYXQY345UX3SQUCPDWJ25NJQZ2TXGFRLVCRYXUOQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:06.453186841 (573h43m6.324269876s) [noise=true , address=145.79.227.40:4058 , piece_id=KRHH2QZKSDPWTSSKYDVDWGKFJD3LPGP6U7KI2PPJ5T3TXYIUGJWA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:06.510769276 (573h43m6.381852321s) [noise=true , address=145.79.227.43:4164 , piece_id=J6Q4ZDOIIIBA2FXGZZTD6AYS7KGAWSRXXUBAPDPQWYKNU5GNXBBQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:06.518868879 (573h43m6.389951924s) [noise=true , address=145.79.227.33:4893 , piece_id=XGLX3DFNAY4F4GPGCGZ4ONDJ7NK62SAKX3PBRWHWI74EF3ZFMUSA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:06.519238043 (573h43m6.390321098s) [noise=true , address=45.91.116.6:4981 , piece_id=INTVU2EYTU3DIM6WQFQEGO6IEYH3YFP5U2VEN7A5KPTOKU2DWOOA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:06.523211328 (573h43m6.394294373s) [noise=true , address=77.90.31.16:4759 , piece_id=ARC4HRP57U5VQXZTT63QYLXBPZURI5VMJBIQZOFKGLJBQ3K3KGLQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:06.767869783 (573h43m6.638952818s) [noise=true , address=145.79.227.34:4924 , piece_id=MAQXQTAGXQ7AEPUCESUSUGMHSDMMXS7HHKAEA7B3HW5X66PDUF5A, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:06.767889431 (573h43m6.638972476s) [seg_pos={0 0}, upload=502975]: Segment reached optimal threshold of 14 pieces. Dec 29 13:12:06.768102116 (573h43m6.639185161s) [noise=true , address=45.91.116.8:4051 , piece_id=GDJFLQCHUF7BFQ7BVSNTVJ3TYZHYVOO6YYMO3KUIUSIEPAF4SFDQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1preT33ei4yVXXYVLHhaUbHbiyL14oHFEiof6cCWPig83W6CT3): protocol: expected piece hash; context canceled; EOF Dec 29 13:12:06.768108257 (573h43m6.639191302s) [noise=true , address=45.91.116.8:4051 , piece_id=GDJFLQCHUF7BFQ7BVSNTVJ3TYZHYVOO6YYMO3KUIUSIEPAF4SFDQ, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold Dec 29 13:12:06.768219208 (573h43m6.639302254s) [noise=true , address=77.90.31.18:4821 , piece_id=LO5W47FHYOVGY5Q6BH5L6KX77HES3XRAFUTPI45HYPJZDO3YHLCQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH): protocol: expected piece hash; context canceled; EOF Dec 29 13:12:06.768224789 (573h43m6.639307824s) [noise=true , address=77.90.31.18:4821 , piece_id=LO5W47FHYOVGY5Q6BH5L6KX77HES3XRAFUTPI45HYPJZDO3YHLCQ, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold Dec 29 13:12:06.768384213 (573h43m6.639467258s) [noise=true , address=64.49.13.169:4711 , piece_id=4F4T4YKXMT7NF2C5KBUGJHQJGZL4VCKNKYB266PCVI3UACHJXZ5Q, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:48802->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:48802->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48802->64.49.13.169:4711: use of closed network connection Dec 29 13:12:06.768399261 (573h43m6.639482296s) [noise=true , address=64.49.13.169:4711 , piece_id=4F4T4YKXMT7NF2C5KBUGJHQJGZL4VCKNKYB266PCVI3UACHJXZ5Q, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold Dec 29 13:12:06.768732636 (573h43m6.639815671s) [noise=true , address=64.49.13.169:4709 , piece_id=TBHHX5ES2A6NQRGP25HIC7J7IBWLOJGQ2LO7MUXLHE5YET7BHLLA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:35310->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:35310->64.49.13.169:4709: use of closed network connection Dec 29 13:12:06.768745460 (573h43m6.639828505s) [noise=true , address=64.49.13.169:4709 , piece_id=TBHHX5ES2A6NQRGP25HIC7J7IBWLOJGQ2LO7MUXLHE5YET7BHLLA, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold Dec 29 13:12:06.768784586 (573h43m6.639867631s) [seg_pos={0 0}, upload=502975]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 13:12:06.768880217 (573h43m6.639963252s) [upload=502975]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 13:12:06.768884906 (573h43m6.639967941s) [upload=502975]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 13:12:07.067188330 (573h43m6.938271365s) [upload=502975]: Done uploading segments... Dec 29 13:12:07.067251099 (573h43m6.938334134s) [upload=502975]: Upload finished. err: <nil> Duration: 40.287776718s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [............X......................X....X.X....X.X..........] duration: 18.816443371s trace: https://jaeger.dev.storj.io/trace/312ddb07b2dee82a FAIL (2025-12-29 13:19:36 UTC) orbiter warning: context deadline exceeded Duration: 18.816469139s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 51 out of 60 is failed (threshold=6) [XXXXXXXXXXXXX.XXXXXXX.X.XX.XXX.XX.XXXXXXX.XX.XXXXXXXXXXX.XXX] duration: 5.001042176s trace: https://jaeger.dev.storj.io/trace/519fe981be6ed7c4 FAIL (2025-12-29 13:19:25 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001063035s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.............X......................X....X.X....X.X.........] duration: 26.070165376s trace: https://jaeger.dev.storj.io/trace/312ddb07b2dee82a FAIL (2025-12-29 13:19:17 UTC) orbiter warning: context deadline exceeded Duration: 26.070194111s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [X......................X.....X.....................XX.......] duration: 46.695736661s trace: https://jaeger.dev.storj.io/trace/77d26141ea11d228 FAIL (2025-12-29 13:18:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12J2weU7Bmsr3RdQQ9QMuadT96XReBkyxTeaLJ2SEqVGGFX1PWu, Piece ID: JITX25FNZ4JIKYXQY345UX3SQUCPDWJ25NJQZ2TXGFRLVCRYXUOQ) context deadline exceeded; piecestore: (Node ID: 125sm6iHV94mJFLWrKeyRMH4b1qSMo32nDuW8uQr71xUL7Cema2, Piece ID: INTVU2EYTU3DIM6WQFQEGO6IEYH3YFP5U2VEN7A5KPTOKU2DWOOA) context deadline exceeded; piecestore: (Node ID: 12i57FF97oEddYLZpNxSq5or9WLEquL8ZW1cVUuW4AThBbDqcCL, Piece ID: Q35LXGPTOAG3R2UL3JLL7DNHHD4PSMEULYW6G3SPXKNVFZ5VTNRA) context deadline exceeded ======LOGS====== Q2LO7MUXLHE5YET7BHLLA, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.692912334 (573h43m3.563995379s) [noise=true , address=77.90.31.16:4759 , piece_id=ARC4HRP57U5VQXZTT63QYLXBPZURI5VMJBIQZOFKGLJBQ3K3KGLQ, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.693322595 (573h43m3.56440564s) [noise=true , address=77.90.31.15:4711 , piece_id=JITX25FNZ4JIKYXQY345UX3SQUCPDWJ25NJQZ2TXGFRLVCRYXUOQ, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.693739649 (573h43m3.564822694s) [noise=true , address=45.91.116.6:4981 , piece_id=INTVU2EYTU3DIM6WQFQEGO6IEYH3YFP5U2VEN7A5KPTOKU2DWOOA, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.694315887 (573h43m3.565398942s) [noise=true , address=145.79.227.4:4220 , piece_id=SHHJ7EW7DTNOZXVBP6A64VIX7P6AQN62TBCUT3KTH4MIB6UXY3BA, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.694765222 (573h43m3.565848267s) [noise=true , address=145.79.227.29:4952 , piece_id=DKYDB2R4DNCGH6T3SLRR2MOZZUPVZZVRL2BKGYOBYXJJVPSZEJ4Q, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.695202545 (573h43m3.56628559s) [noise=true , address=145.79.227.40:4058 , piece_id=KRHH2QZKSDPWTSSKYDVDWGKFJD3LPGP6U7KI2PPJ5T3TXYIUGJWA, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.695687528 (573h43m3.566770584s) [noise=true , address=77.90.31.19:4857 , piece_id=Q35LXGPTOAG3R2UL3JLL7DNHHD4PSMEULYW6G3SPXKNVFZ5VTNRA, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.696135221 (573h43m3.567218276s) [noise=true , address=45.91.116.8:4051 , piece_id=GDJFLQCHUF7BFQ7BVSNTVJ3TYZHYVOO6YYMO3KUIUSIEPAF4SFDQ, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.696703733 (573h43m3.567786778s) [noise=true , address=64.49.13.170:4720 , piece_id=HF6X3OT2EPPDE3DWAPWABVZUR3ERDBWXHG3NGNKEWZAD7G5AN4TA, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.698418298 (573h43m3.569501343s) [noise=true , address=145.79.227.36:4520 , piece_id=MYJSXUH563JFUW5Q7YUIWL7466G6ASY5WXR3T2PGUFFCTRTOBYJA, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.700198829 (573h43m3.571281874s) [noise=true , address=77.90.31.18:4821 , piece_id=LO5W47FHYOVGY5Q6BH5L6KX77HES3XRAFUTPI45HYPJZDO3YHLCQ, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.701993145 (573h43m3.57307619s) [noise=true , address=193.149.164.143:4699 , piece_id=UDFDCYO25MOJBTMDE6F2WILXE3HLOB75UJZ4LW7ZMNSCOBJI6UJA, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.703884687 (573h43m3.574967732s) [noise=true , address=145.79.227.33:4893 , piece_id=XGLX3DFNAY4F4GPGCGZ4ONDJ7NK62SAKX3PBRWHWI74EF3ZFMUSA, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.705687109 (573h43m3.576770154s) [noise=true , address=45.91.116.10:4094 , piece_id=6UUIZNTESBBWJMDQ5ZTCHU7S6YKM4GDKO4ZAC6T2FUTPVGFFDS3A, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.707557651 (573h43m3.578640696s) [noise=true , address=145.79.227.43:4164 , piece_id=J6Q4ZDOIIIBA2FXGZZTD6AYS7KGAWSRXXUBAPDPQWYKNU5GNXBBQ, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:05.168247719 (573h43m5.039330754s) [upload=502975]: Next returned nil segment Dec 29 13:12:05.168259962 (573h43m5.039343007s) [upload=502975]: Waiting for error group managing segments... Dec 29 13:12:05.558873127 (573h43m5.429956162s) [noise=true , address=145.79.227.4:4220 , piece_id=SHHJ7EW7DTNOZXVBP6A64VIX7P6AQN62TBCUT3KTH4MIB6UXY3BA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:05.563024312 (573h43m5.434107347s) [noise=true , address=64.49.13.170:4720 , piece_id=HF6X3OT2EPPDE3DWAPWABVZUR3ERDBWXHG3NGNKEWZAD7G5AN4TA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:05.564115719 (573h43m5.435198755s) [noise=true , address=145.79.227.36:4520 , piece_id=MYJSXUH563JFUW5Q7YUIWL7466G6ASY5WXR3T2PGUFFCTRTOBYJA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:05.593873402 (573h43m5.464956447s) [noise=true , address=145.79.227.29:4952 , piece_id=DKYDB2R4DNCGH6T3SLRR2MOZZUPVZZVRL2BKGYOBYXJJVPSZEJ4Q, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:05.898194604 (573h43m5.769277639s) [noise=true , address=45.91.116.10:4094 , piece_id=6UUIZNTESBBWJMDQ5ZTCHU7S6YKM4GDKO4ZAC6T2FUTPVGFFDS3A, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:05.912845292 (573h43m5.783928337s) [noise=true , address=77.90.31.19:4857 , piece_id=Q35LXGPTOAG3R2UL3JLL7DNHHD4PSMEULYW6G3SPXKNVFZ5VTNRA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:06.426532207 (573h43m6.297615252s) [noise=true , address=193.149.164.143:4699 , piece_id=UDFDCYO25MOJBTMDE6F2WILXE3HLOB75UJZ4LW7ZMNSCOBJI6UJA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:06.449589562 (573h43m6.320672607s) [noise=true , address=77.90.31.15:4711 , piece_id=JITX25FNZ4JIKYXQY345UX3SQUCPDWJ25NJQZ2TXGFRLVCRYXUOQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:06.453186841 (573h43m6.324269876s) [noise=true , address=145.79.227.40:4058 , piece_id=KRHH2QZKSDPWTSSKYDVDWGKFJD3LPGP6U7KI2PPJ5T3TXYIUGJWA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:06.510769276 (573h43m6.381852321s) [noise=true , address=145.79.227.43:4164 , piece_id=J6Q4ZDOIIIBA2FXGZZTD6AYS7KGAWSRXXUBAPDPQWYKNU5GNXBBQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:06.518868879 (573h43m6.389951924s) [noise=true , address=145.79.227.33:4893 , piece_id=XGLX3DFNAY4F4GPGCGZ4ONDJ7NK62SAKX3PBRWHWI74EF3ZFMUSA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:06.519238043 (573h43m6.390321098s) [noise=true , address=45.91.116.6:4981 , piece_id=INTVU2EYTU3DIM6WQFQEGO6IEYH3YFP5U2VEN7A5KPTOKU2DWOOA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:06.523211328 (573h43m6.394294373s) [noise=true , address=77.90.31.16:4759 , piece_id=ARC4HRP57U5VQXZTT63QYLXBPZURI5VMJBIQZOFKGLJBQ3K3KGLQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:06.767869783 (573h43m6.638952818s) [noise=true , address=145.79.227.34:4924 , piece_id=MAQXQTAGXQ7AEPUCESUSUGMHSDMMXS7HHKAEA7B3HW5X66PDUF5A, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:06.767889431 (573h43m6.638972476s) [seg_pos={0 0}, upload=502975]: Segment reached optimal threshold of 14 pieces. Dec 29 13:12:06.768102116 (573h43m6.639185161s) [noise=true , address=45.91.116.8:4051 , piece_id=GDJFLQCHUF7BFQ7BVSNTVJ3TYZHYVOO6YYMO3KUIUSIEPAF4SFDQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1preT33ei4yVXXYVLHhaUbHbiyL14oHFEiof6cCWPig83W6CT3): protocol: expected piece hash; context canceled; EOF Dec 29 13:12:06.768108257 (573h43m6.639191302s) [noise=true , address=45.91.116.8:4051 , piece_id=GDJFLQCHUF7BFQ7BVSNTVJ3TYZHYVOO6YYMO3KUIUSIEPAF4SFDQ, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold Dec 29 13:12:06.768219208 (573h43m6.639302254s) [noise=true , address=77.90.31.18:4821 , piece_id=LO5W47FHYOVGY5Q6BH5L6KX77HES3XRAFUTPI45HYPJZDO3YHLCQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH): protocol: expected piece hash; context canceled; EOF Dec 29 13:12:06.768224789 (573h43m6.639307824s) [noise=true , address=77.90.31.18:4821 , piece_id=LO5W47FHYOVGY5Q6BH5L6KX77HES3XRAFUTPI45HYPJZDO3YHLCQ, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold Dec 29 13:12:06.768384213 (573h43m6.639467258s) [noise=true , address=64.49.13.169:4711 , piece_id=4F4T4YKXMT7NF2C5KBUGJHQJGZL4VCKNKYB266PCVI3UACHJXZ5Q, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:48802->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:48802->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48802->64.49.13.169:4711: use of closed network connection Dec 29 13:12:06.768399261 (573h43m6.639482296s) [noise=true , address=64.49.13.169:4711 , piece_id=4F4T4YKXMT7NF2C5KBUGJHQJGZL4VCKNKYB266PCVI3UACHJXZ5Q, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold Dec 29 13:12:06.768732636 (573h43m6.639815671s) [noise=true , address=64.49.13.169:4709 , piece_id=TBHHX5ES2A6NQRGP25HIC7J7IBWLOJGQ2LO7MUXLHE5YET7BHLLA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:35310->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:35310->64.49.13.169:4709: use of closed network connection Dec 29 13:12:06.768745460 (573h43m6.639828505s) [noise=true , address=64.49.13.169:4709 , piece_id=TBHHX5ES2A6NQRGP25HIC7J7IBWLOJGQ2LO7MUXLHE5YET7BHLLA, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold Dec 29 13:12:06.768784586 (573h43m6.639867631s) [seg_pos={0 0}, upload=502975]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 13:12:06.768880217 (573h43m6.639963252s) [upload=502975]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 13:12:06.768884906 (573h43m6.639967941s) [upload=502975]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 13:12:07.067188330 (573h43m6.938271365s) [upload=502975]: Done uploading segments... Dec 29 13:12:07.067251099 (573h43m6.938334134s) [upload=502975]: Upload finished. err: <nil> Duration: 46.695773272s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..............X......................X....X.X....X.X........] duration: 17.812136512s trace: https://jaeger.dev.storj.io/trace/312ddb07b2dee82a FAIL (2025-12-29 13:18:46 UTC) orbiter warning: context deadline exceeded Duration: 17.81215681s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 32 out of 60 is failed (threshold=3) [.XXXX..XX..X...XX.X.XX...X.X.X.XXX.X.X.X.XX..X.XXX.XX..X.XX.] duration: 5.905247576s trace: https://jaeger.dev.storj.io/trace/7dfc7d7e2d71a3fc FAIL (2025-12-29 13:18:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, Piece ID: 5F7YBUUFPMX6WQRAB7GVZ3GROFYM6FB2CPORE7PTVJTFLGDS4VXA) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: HVDX2NNTTOS5IO44Z7BICXLYG3USC47LHJJNX6EGJ4QV2FMX5VXQ) context deadline exceeded ======LOGS====== 29 13:17:39.540175545 (573h48m39.41125858s) [upload=503053]: Starting upload Dec 29 13:17:39.540229057 (573h48m39.411312092s) [upload=503053]: Done starting upload Dec 29 13:17:39.540441481 (573h48m39.411524526s) [upload=503053]: Uploading segments... Dec 29 13:17:39.540482360 (573h48m39.411565395s) [upload=503053]: Got next segment. Inline: false Dec 29 13:17:39.540511384 (573h48m39.411594419s) [upload=503053]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 13:17:39.540515733 (573h48m39.411598768s) [upload=503053]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 13:17:39.790547090 (573h48m39.661630125s) [seg_pos={0 0}, upload=503053]: Begin upload segment... Dec 29 13:17:39.790596244 (573h48m39.661679279s) [seg_pos={0 0}, upload=503053]: Done begin upload segment. Dec 29 13:17:39.790639095 (573h48m39.66172213s) [noise=true , address=45.91.116.11:4117 , piece_id=XBP7IUQXU5IKSX2JV7SS6XUJPFY5R2BQWYGM4ZJMO3NQ5SRG4I5A, seg_pos={0 0}, upload=503053]: Uploading piece... Dec 29 13:17:39.790757290 (573h48m39.661840325s) [noise=true , address=145.79.227.11:4732 , piece_id=N3SNFHQOB5ZJVTBESQBHHIDFJWYYHMGKL6V72TE2GPZPP32RGLXA, seg_pos={0 0}, upload=503053]: Uploading piece... Dec 29 13:17:39.791333458 (573h48m39.662416503s) [noise=true , address=145.79.227.20:4437 , piece_id=VKQI5YRGFKVELOMSFCAMAUUBW4KIQK3OV4JZPSNTM2CK6E6QOLBQ, seg_pos={0 0}, upload=503053]: Uploading piece... Dec 29 13:17:39.791640912 (573h48m39.662723947s) [noise=true , address=45.91.116.7:4009 , piece_id=W7DI7QVHOCHPWJYAYJOAYY6UIM3R67573GNZIDELY5F33IUXEWVA, seg_pos={0 0}, upload=503053]: Uploading piece... Dec 29 13:17:39.791929422 (573h48m39.663012457s) [noise=true , address=45.91.116.12:4225 , piece_id=QMVQJN5K4OTICS5P6XIZ7S4AOS4YGMLVH3FQ5U6FNZVSVS5FTDMQ, seg_pos={0 0}, upload=503053]: Uploading piece... Dec 29 13:17:39.792206600 (573h48m39.663289635s) [noise=true , address=45.91.116.6:4974 , piece_id=ZDDCURSLZ4246UHAL7AWSCPWTQ5YD7VAZ6VHAFST6MGFQCQGSOGQ, seg_pos={0 0}, upload=503053]: Uploading piece... Dec 29 13:17:39.792601952 (573h48m39.663684998s) [noise=true , address=145.79.227.38:4978 , piece_id=IZCZJWZ4QLTCH6BUSKY6CULNP2ULDMQIOFMCMM5SR3CHF6AS5ULA, seg_pos={0 0}, upload=503053]: Uploading piece... Dec 29 13:17:39.792950867 (573h48m39.664033912s) [noise=true , address=77.90.31.11:4553 , piece_id=5F7YBUUFPMX6WQRAB7GVZ3GROFYM6FB2CPORE7PTVJTFLGDS4VXA, seg_pos={0 0}, upload=503053]: Uploading piece... Dec 29 13:17:39.793476237 (573h48m39.664559282s) [noise=true , address=145.79.227.18:4006 , piece_id=O3WIX5QNXOMB5BRDCCNU5ZL62V4AT4MVWE3MU2GE6GIJ7TJYYDJA, seg_pos={0 0}, upload=503053]: Uploading piece... Dec 29 13:17:39.794762296 (573h48m39.665845341s) [noise=true , address=77.90.31.12:4589 , piece_id=3HJXH4YFU427ZWGKR3AQVGIMKWI2FLOCUBLG3P3ZG2BGR3WQ632A, seg_pos={0 0}, upload=503053]: Uploading piece... Dec 29 13:17:39.796190105 (573h48m39.66727315s) [noise=true , address=45.91.116.5:4299 , piece_id=HVDX2NNTTOS5IO44Z7BICXLYG3USC47LHJJNX6EGJ4QV2FMX5VXQ, seg_pos={0 0}, upload=503053]: Uploading piece... Dec 29 13:17:39.797256144 (573h48m39.668339189s) [noise=true , address=145.79.227.19:4077 , piece_id=G72N37Z4FKLBI46XHLVCJBNGKOT7AKZO5Y4GASBIPRCBNVP342OA, seg_pos={0 0}, upload=503053]: Uploading piece... Dec 29 13:17:39.798662533 (573h48m39.669745578s) [noise=true , address=145.79.227.16:4909 , piece_id=CRN2XGVACJY2E44PIS4OGG3WDIPXVTNLHN4TMYJJYEDS3XEBWPCQ, seg_pos={0 0}, upload=503053]: Uploading piece... Dec 29 13:17:39.799977255 (573h48m39.671060291s) [noise=true , address=145.79.227.4:4185 , piece_id=S4QJUU4G3DEFP2SYWCSJJPIAMRQQHZBEP6OKSPB4PNPTV6CZHTCQ, seg_pos={0 0}, upload=503053]: Uploading piece... Dec 29 13:17:39.804730256 (573h48m39.675813292s) [noise=true , address=145.79.227.51:4411 , piece_id=GQ3HRCSQWIHBBXLCDFO3BEP566L57JENKH7AZUO62J5VZ6YFCC7Q, seg_pos={0 0}, upload=503053]: Uploading piece... Dec 29 13:17:39.806081600 (573h48m39.677164645s) [noise=true , address=145.79.227.3:4149 , piece_id=VIKUB5FR3ZOV2B27BC3ZW3Z7PQSHP75TPB5NIJNOMWLX7FCUDRVQ, seg_pos={0 0}, upload=503053]: Uploading piece... Dec 29 13:17:41.493481509 (573h48m41.364564554s) [upload=503053]: Next returned nil segment Dec 29 13:17:41.493502690 (573h48m41.364585735s) [upload=503053]: Waiting for error group managing segments... Dec 29 13:17:41.772207224 (573h48m41.643290269s) [noise=true , address=145.79.227.19:4077 , piece_id=G72N37Z4FKLBI46XHLVCJBNGKOT7AKZO5Y4GASBIPRCBNVP342OA, seg_pos={0 0}, upload=503053]: Done uploading piece. err: <nil> Dec 29 13:17:41.782029908 (573h48m41.653112943s) [noise=true , address=145.79.227.11:4732 , piece_id=N3SNFHQOB5ZJVTBESQBHHIDFJWYYHMGKL6V72TE2GPZPP32RGLXA, seg_pos={0 0}, upload=503053]: Done uploading piece. err: <nil> Dec 29 13:17:41.787864948 (573h48m41.658947993s) [noise=true , address=145.79.227.16:4909 , piece_id=CRN2XGVACJY2E44PIS4OGG3WDIPXVTNLHN4TMYJJYEDS3XEBWPCQ, seg_pos={0 0}, upload=503053]: Done uploading piece. err: <nil> Dec 29 13:17:41.793494689 (573h48m41.664577724s) [noise=true , address=145.79.227.4:4185 , piece_id=S4QJUU4G3DEFP2SYWCSJJPIAMRQQHZBEP6OKSPB4PNPTV6CZHTCQ, seg_pos={0 0}, upload=503053]: Done uploading piece. err: <nil> Dec 29 13:17:41.800966757 (573h48m41.672049802s) [noise=true , address=145.79.227.18:4006 , piece_id=O3WIX5QNXOMB5BRDCCNU5ZL62V4AT4MVWE3MU2GE6GIJ7TJYYDJA, seg_pos={0 0}, upload=503053]: Done uploading piece. err: <nil> Dec 29 13:17:41.803150414 (573h48m41.674233459s) [noise=true , address=145.79.227.3:4149 , piece_id=VIKUB5FR3ZOV2B27BC3ZW3Z7PQSHP75TPB5NIJNOMWLX7FCUDRVQ, seg_pos={0 0}, upload=503053]: Done uploading piece. err: <nil> Dec 29 13:17:41.805106228 (573h48m41.676189263s) [noise=true , address=145.79.227.38:4978 , piece_id=IZCZJWZ4QLTCH6BUSKY6CULNP2ULDMQIOFMCMM5SR3CHF6AS5ULA, seg_pos={0 0}, upload=503053]: Done uploading piece. err: <nil> Dec 29 13:17:41.808591064 (573h48m41.679674099s) [noise=true , address=145.79.227.20:4437 , piece_id=VKQI5YRGFKVELOMSFCAMAUUBW4KIQK3OV4JZPSNTM2CK6E6QOLBQ, seg_pos={0 0}, upload=503053]: Done uploading piece. err: <nil> Dec 29 13:17:41.822363249 (573h48m41.693446294s) [noise=true , address=145.79.227.51:4411 , piece_id=GQ3HRCSQWIHBBXLCDFO3BEP566L57JENKH7AZUO62J5VZ6YFCC7Q, seg_pos={0 0}, upload=503053]: Done uploading piece. err: <nil> Dec 29 13:17:42.047382889 (573h48m41.918465934s) [noise=true , address=77.90.31.11:4553 , piece_id=5F7YBUUFPMX6WQRAB7GVZ3GROFYM6FB2CPORE7PTVJTFLGDS4VXA, seg_pos={0 0}, upload=503053]: Done uploading piece. err: <nil> Dec 29 13:17:42.147248348 (573h48m42.018331384s) [noise=true , address=45.91.116.5:4299 , piece_id=HVDX2NNTTOS5IO44Z7BICXLYG3USC47LHJJNX6EGJ4QV2FMX5VXQ, seg_pos={0 0}, upload=503053]: Done uploading piece. err: <nil> Dec 29 13:17:42.196027221 (573h48m42.067110256s) [noise=true , address=77.90.31.12:4589 , piece_id=3HJXH4YFU427ZWGKR3AQVGIMKWI2FLOCUBLG3P3ZG2BGR3WQ632A, seg_pos={0 0}, upload=503053]: Done uploading piece. err: <nil> Dec 29 13:17:42.216749780 (573h48m42.087832815s) [noise=true , address=45.91.116.12:4225 , piece_id=QMVQJN5K4OTICS5P6XIZ7S4AOS4YGMLVH3FQ5U6FNZVSVS5FTDMQ, seg_pos={0 0}, upload=503053]: Done uploading piece. err: <nil> Dec 29 13:17:42.216771340 (573h48m42.087854375s) [seg_pos={0 0}, upload=503053]: Segment reached optimal threshold of 13 pieces. Dec 29 13:17:42.216982773 (573h48m42.088065828s) [noise=true , address=45.91.116.7:4009 , piece_id=W7DI7QVHOCHPWJYAYJOAYY6UIM3R67573GNZIDELY5F33IUXEWVA, seg_pos={0 0}, upload=503053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 29 13:17:42.216994435 (573h48m42.088077481s) [noise=true , address=45.91.116.7:4009 , piece_id=W7DI7QVHOCHPWJYAYJOAYY6UIM3R67573GNZIDELY5F33IUXEWVA, seg_pos={0 0}, upload=503053]: Long tail cancellation due to optimal threshold Dec 29 13:17:42.217255532 (573h48m42.088338567s) [noise=true , address=45.91.116.6:4974 , piece_id=ZDDCURSLZ4246UHAL7AWSCPWTQ5YD7VAZ6VHAFST6MGFQCQGSOGQ, seg_pos={0 0}, upload=503053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:50160->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:50160->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50160->45.91.116.6:4974: use of closed network connection Dec 29 13:17:42.217262256 (573h48m42.088345301s) [noise=true , address=45.91.116.6:4974 , piece_id=ZDDCURSLZ4246UHAL7AWSCPWTQ5YD7VAZ6VHAFST6MGFQCQGSOGQ, seg_pos={0 0}, upload=503053]: Long tail cancellation due to optimal threshold Dec 29 13:17:42.217422140 (573h48m42.088505205s) [noise=true , address=45.91.116.11:4117 , piece_id=XBP7IUQXU5IKSX2JV7SS6XUJPFY5R2BQWYGM4ZJMO3NQ5SRG4I5A, seg_pos={0 0}, upload=503053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF Dec 29 13:17:42.217442519 (573h48m42.088525564s) [noise=true , address=45.91.116.11:4117 , piece_id=XBP7IUQXU5IKSX2JV7SS6XUJPFY5R2BQWYGM4ZJMO3NQ5SRG4I5A, seg_pos={0 0}, upload=503053]: Long tail cancellation due to optimal threshold Dec 29 13:17:42.217487414 (573h48m42.088570449s) [seg_pos={0 0}, upload=503053]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 13:17:42.217601521 (573h48m42.088684556s) [upload=503053]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 13:17:42.217608554 (573h48m42.088691599s) [upload=503053]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 13:17:42.510405140 (573h48m42.381488185s) [upload=503053]: Done uploading segments... Dec 29 13:17:42.510430329 (573h48m42.381513364s) [upload=503053]: Upload finished. err: <nil> Duration: 5.905272102s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 51 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXX.XXXXXXX.X.XX.XXX.XX.XXXXXXX.XX.XXXXXXXXXXX.XX] duration: 5.001169687s trace: https://jaeger.dev.storj.io/trace/152fcf26b967a83d FAIL (2025-12-29 13:18:32 UTC) orbiter warning: context deadline exceeded Duration: 5.001180858s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 32 out of 60 is failed (threshold=3) [..XXXX..XX..X...XX.X.XX...X.X.X.XXX.X.X.X.XX..X.XXX.XX..X.XX] duration: 30.003652164s trace: https://jaeger.dev.storj.io/trace/7dfc7d7e2d71a3fc FAIL (2025-12-29 13:18:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, Piece ID: 5F7YBUUFPMX6WQRAB7GVZ3GROFYM6FB2CPORE7PTVJTFLGDS4VXA) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: HVDX2NNTTOS5IO44Z7BICXLYG3USC47LHJJNX6EGJ4QV2FMX5VXQ) context deadline exceeded ======LOGS====== 29 13:17:39.540175545 (573h48m39.41125858s) [upload=503053]: Starting upload Dec 29 13:17:39.540229057 (573h48m39.411312092s) [upload=503053]: Done starting upload Dec 29 13:17:39.540441481 (573h48m39.411524526s) [upload=503053]: Uploading segments... Dec 29 13:17:39.540482360 (573h48m39.411565395s) [upload=503053]: Got next segment. Inline: false Dec 29 13:17:39.540511384 (573h48m39.411594419s) [upload=503053]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 13:17:39.540515733 (573h48m39.411598768s) [upload=503053]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 13:17:39.790547090 (573h48m39.661630125s) [seg_pos={0 0}, upload=503053]: Begin upload segment... Dec 29 13:17:39.790596244 (573h48m39.661679279s) [seg_pos={0 0}, upload=503053]: Done begin upload segment. Dec 29 13:17:39.790639095 (573h48m39.66172213s) [noise=true , address=45.91.116.11:4117 , piece_id=XBP7IUQXU5IKSX2JV7SS6XUJPFY5R2BQWYGM4ZJMO3NQ5SRG4I5A, seg_pos={0 0}, upload=503053]: Uploading piece... Dec 29 13:17:39.790757290 (573h48m39.661840325s) [noise=true , address=145.79.227.11:4732 , piece_id=N3SNFHQOB5ZJVTBESQBHHIDFJWYYHMGKL6V72TE2GPZPP32RGLXA, seg_pos={0 0}, upload=503053]: Uploading piece... Dec 29 13:17:39.791333458 (573h48m39.662416503s) [noise=true , address=145.79.227.20:4437 , piece_id=VKQI5YRGFKVELOMSFCAMAUUBW4KIQK3OV4JZPSNTM2CK6E6QOLBQ, seg_pos={0 0}, upload=503053]: Uploading piece... Dec 29 13:17:39.791640912 (573h48m39.662723947s) [noise=true , address=45.91.116.7:4009 , piece_id=W7DI7QVHOCHPWJYAYJOAYY6UIM3R67573GNZIDELY5F33IUXEWVA, seg_pos={0 0}, upload=503053]: Uploading piece... Dec 29 13:17:39.791929422 (573h48m39.663012457s) [noise=true , address=45.91.116.12:4225 , piece_id=QMVQJN5K4OTICS5P6XIZ7S4AOS4YGMLVH3FQ5U6FNZVSVS5FTDMQ, seg_pos={0 0}, upload=503053]: Uploading piece... Dec 29 13:17:39.792206600 (573h48m39.663289635s) [noise=true , address=45.91.116.6:4974 , piece_id=ZDDCURSLZ4246UHAL7AWSCPWTQ5YD7VAZ6VHAFST6MGFQCQGSOGQ, seg_pos={0 0}, upload=503053]: Uploading piece... Dec 29 13:17:39.792601952 (573h48m39.663684998s) [noise=true , address=145.79.227.38:4978 , piece_id=IZCZJWZ4QLTCH6BUSKY6CULNP2ULDMQIOFMCMM5SR3CHF6AS5ULA, seg_pos={0 0}, upload=503053]: Uploading piece... Dec 29 13:17:39.792950867 (573h48m39.664033912s) [noise=true , address=77.90.31.11:4553 , piece_id=5F7YBUUFPMX6WQRAB7GVZ3GROFYM6FB2CPORE7PTVJTFLGDS4VXA, seg_pos={0 0}, upload=503053]: Uploading piece... Dec 29 13:17:39.793476237 (573h48m39.664559282s) [noise=true , address=145.79.227.18:4006 , piece_id=O3WIX5QNXOMB5BRDCCNU5ZL62V4AT4MVWE3MU2GE6GIJ7TJYYDJA, seg_pos={0 0}, upload=503053]: Uploading piece... Dec 29 13:17:39.794762296 (573h48m39.665845341s) [noise=true , address=77.90.31.12:4589 , piece_id=3HJXH4YFU427ZWGKR3AQVGIMKWI2FLOCUBLG3P3ZG2BGR3WQ632A, seg_pos={0 0}, upload=503053]: Uploading piece... Dec 29 13:17:39.796190105 (573h48m39.66727315s) [noise=true , address=45.91.116.5:4299 , piece_id=HVDX2NNTTOS5IO44Z7BICXLYG3USC47LHJJNX6EGJ4QV2FMX5VXQ, seg_pos={0 0}, upload=503053]: Uploading piece... Dec 29 13:17:39.797256144 (573h48m39.668339189s) [noise=true , address=145.79.227.19:4077 , piece_id=G72N37Z4FKLBI46XHLVCJBNGKOT7AKZO5Y4GASBIPRCBNVP342OA, seg_pos={0 0}, upload=503053]: Uploading piece... Dec 29 13:17:39.798662533 (573h48m39.669745578s) [noise=true , address=145.79.227.16:4909 , piece_id=CRN2XGVACJY2E44PIS4OGG3WDIPXVTNLHN4TMYJJYEDS3XEBWPCQ, seg_pos={0 0}, upload=503053]: Uploading piece... Dec 29 13:17:39.799977255 (573h48m39.671060291s) [noise=true , address=145.79.227.4:4185 , piece_id=S4QJUU4G3DEFP2SYWCSJJPIAMRQQHZBEP6OKSPB4PNPTV6CZHTCQ, seg_pos={0 0}, upload=503053]: Uploading piece... Dec 29 13:17:39.804730256 (573h48m39.675813292s) [noise=true , address=145.79.227.51:4411 , piece_id=GQ3HRCSQWIHBBXLCDFO3BEP566L57JENKH7AZUO62J5VZ6YFCC7Q, seg_pos={0 0}, upload=503053]: Uploading piece... Dec 29 13:17:39.806081600 (573h48m39.677164645s) [noise=true , address=145.79.227.3:4149 , piece_id=VIKUB5FR3ZOV2B27BC3ZW3Z7PQSHP75TPB5NIJNOMWLX7FCUDRVQ, seg_pos={0 0}, upload=503053]: Uploading piece... Dec 29 13:17:41.493481509 (573h48m41.364564554s) [upload=503053]: Next returned nil segment Dec 29 13:17:41.493502690 (573h48m41.364585735s) [upload=503053]: Waiting for error group managing segments... Dec 29 13:17:41.772207224 (573h48m41.643290269s) [noise=true , address=145.79.227.19:4077 , piece_id=G72N37Z4FKLBI46XHLVCJBNGKOT7AKZO5Y4GASBIPRCBNVP342OA, seg_pos={0 0}, upload=503053]: Done uploading piece. err: <nil> Dec 29 13:17:41.782029908 (573h48m41.653112943s) [noise=true , address=145.79.227.11:4732 , piece_id=N3SNFHQOB5ZJVTBESQBHHIDFJWYYHMGKL6V72TE2GPZPP32RGLXA, seg_pos={0 0}, upload=503053]: Done uploading piece. err: <nil> Dec 29 13:17:41.787864948 (573h48m41.658947993s) [noise=true , address=145.79.227.16:4909 , piece_id=CRN2XGVACJY2E44PIS4OGG3WDIPXVTNLHN4TMYJJYEDS3XEBWPCQ, seg_pos={0 0}, upload=503053]: Done uploading piece. err: <nil> Dec 29 13:17:41.793494689 (573h48m41.664577724s) [noise=true , address=145.79.227.4:4185 , piece_id=S4QJUU4G3DEFP2SYWCSJJPIAMRQQHZBEP6OKSPB4PNPTV6CZHTCQ, seg_pos={0 0}, upload=503053]: Done uploading piece. err: <nil> Dec 29 13:17:41.800966757 (573h48m41.672049802s) [noise=true , address=145.79.227.18:4006 , piece_id=O3WIX5QNXOMB5BRDCCNU5ZL62V4AT4MVWE3MU2GE6GIJ7TJYYDJA, seg_pos={0 0}, upload=503053]: Done uploading piece. err: <nil> Dec 29 13:17:41.803150414 (573h48m41.674233459s) [noise=true , address=145.79.227.3:4149 , piece_id=VIKUB5FR3ZOV2B27BC3ZW3Z7PQSHP75TPB5NIJNOMWLX7FCUDRVQ, seg_pos={0 0}, upload=503053]: Done uploading piece. err: <nil> Dec 29 13:17:41.805106228 (573h48m41.676189263s) [noise=true , address=145.79.227.38:4978 , piece_id=IZCZJWZ4QLTCH6BUSKY6CULNP2ULDMQIOFMCMM5SR3CHF6AS5ULA, seg_pos={0 0}, upload=503053]: Done uploading piece. err: <nil> Dec 29 13:17:41.808591064 (573h48m41.679674099s) [noise=true , address=145.79.227.20:4437 , piece_id=VKQI5YRGFKVELOMSFCAMAUUBW4KIQK3OV4JZPSNTM2CK6E6QOLBQ, seg_pos={0 0}, upload=503053]: Done uploading piece. err: <nil> Dec 29 13:17:41.822363249 (573h48m41.693446294s) [noise=true , address=145.79.227.51:4411 , piece_id=GQ3HRCSQWIHBBXLCDFO3BEP566L57JENKH7AZUO62J5VZ6YFCC7Q, seg_pos={0 0}, upload=503053]: Done uploading piece. err: <nil> Dec 29 13:17:42.047382889 (573h48m41.918465934s) [noise=true , address=77.90.31.11:4553 , piece_id=5F7YBUUFPMX6WQRAB7GVZ3GROFYM6FB2CPORE7PTVJTFLGDS4VXA, seg_pos={0 0}, upload=503053]: Done uploading piece. err: <nil> Dec 29 13:17:42.147248348 (573h48m42.018331384s) [noise=true , address=45.91.116.5:4299 , piece_id=HVDX2NNTTOS5IO44Z7BICXLYG3USC47LHJJNX6EGJ4QV2FMX5VXQ, seg_pos={0 0}, upload=503053]: Done uploading piece. err: <nil> Dec 29 13:17:42.196027221 (573h48m42.067110256s) [noise=true , address=77.90.31.12:4589 , piece_id=3HJXH4YFU427ZWGKR3AQVGIMKWI2FLOCUBLG3P3ZG2BGR3WQ632A, seg_pos={0 0}, upload=503053]: Done uploading piece. err: <nil> Dec 29 13:17:42.216749780 (573h48m42.087832815s) [noise=true , address=45.91.116.12:4225 , piece_id=QMVQJN5K4OTICS5P6XIZ7S4AOS4YGMLVH3FQ5U6FNZVSVS5FTDMQ, seg_pos={0 0}, upload=503053]: Done uploading piece. err: <nil> Dec 29 13:17:42.216771340 (573h48m42.087854375s) [seg_pos={0 0}, upload=503053]: Segment reached optimal threshold of 13 pieces. Dec 29 13:17:42.216982773 (573h48m42.088065828s) [noise=true , address=45.91.116.7:4009 , piece_id=W7DI7QVHOCHPWJYAYJOAYY6UIM3R67573GNZIDELY5F33IUXEWVA, seg_pos={0 0}, upload=503053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 29 13:17:42.216994435 (573h48m42.088077481s) [noise=true , address=45.91.116.7:4009 , piece_id=W7DI7QVHOCHPWJYAYJOAYY6UIM3R67573GNZIDELY5F33IUXEWVA, seg_pos={0 0}, upload=503053]: Long tail cancellation due to optimal threshold Dec 29 13:17:42.217255532 (573h48m42.088338567s) [noise=true , address=45.91.116.6:4974 , piece_id=ZDDCURSLZ4246UHAL7AWSCPWTQ5YD7VAZ6VHAFST6MGFQCQGSOGQ, seg_pos={0 0}, upload=503053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:50160->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:50160->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50160->45.91.116.6:4974: use of closed network connection Dec 29 13:17:42.217262256 (573h48m42.088345301s) [noise=true , address=45.91.116.6:4974 , piece_id=ZDDCURSLZ4246UHAL7AWSCPWTQ5YD7VAZ6VHAFST6MGFQCQGSOGQ, seg_pos={0 0}, upload=503053]: Long tail cancellation due to optimal threshold Dec 29 13:17:42.217422140 (573h48m42.088505205s) [noise=true , address=45.91.116.11:4117 , piece_id=XBP7IUQXU5IKSX2JV7SS6XUJPFY5R2BQWYGM4ZJMO3NQ5SRG4I5A, seg_pos={0 0}, upload=503053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF Dec 29 13:17:42.217442519 (573h48m42.088525564s) [noise=true , address=45.91.116.11:4117 , piece_id=XBP7IUQXU5IKSX2JV7SS6XUJPFY5R2BQWYGM4ZJMO3NQ5SRG4I5A, seg_pos={0 0}, upload=503053]: Long tail cancellation due to optimal threshold Dec 29 13:17:42.217487414 (573h48m42.088570449s) [seg_pos={0 0}, upload=503053]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 13:17:42.217601521 (573h48m42.088684556s) [upload=503053]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 13:17:42.217608554 (573h48m42.088691599s) [upload=503053]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 13:17:42.510405140 (573h48m42.381488185s) [upload=503053]: Done uploading segments... Dec 29 13:17:42.510430329 (573h48m42.381513364s) [upload=503053]: Upload finished. err: <nil> Duration: 30.003676251s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.X......................X.....X.....................XX......] duration: 48.07919318s trace: https://jaeger.dev.storj.io/trace/77d26141ea11d228 FAIL (2025-12-29 13:18:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12J2weU7Bmsr3RdQQ9QMuadT96XReBkyxTeaLJ2SEqVGGFX1PWu, Piece ID: JITX25FNZ4JIKYXQY345UX3SQUCPDWJ25NJQZ2TXGFRLVCRYXUOQ) context deadline exceeded; piecestore: (Node ID: 125sm6iHV94mJFLWrKeyRMH4b1qSMo32nDuW8uQr71xUL7Cema2, Piece ID: INTVU2EYTU3DIM6WQFQEGO6IEYH3YFP5U2VEN7A5KPTOKU2DWOOA) context deadline exceeded; piecestore: (Node ID: 12i57FF97oEddYLZpNxSq5or9WLEquL8ZW1cVUuW4AThBbDqcCL, Piece ID: Q35LXGPTOAG3R2UL3JLL7DNHHD4PSMEULYW6G3SPXKNVFZ5VTNRA) context deadline exceeded ======LOGS====== Q2LO7MUXLHE5YET7BHLLA, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.692912334 (573h43m3.563995379s) [noise=true , address=77.90.31.16:4759 , piece_id=ARC4HRP57U5VQXZTT63QYLXBPZURI5VMJBIQZOFKGLJBQ3K3KGLQ, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.693322595 (573h43m3.56440564s) [noise=true , address=77.90.31.15:4711 , piece_id=JITX25FNZ4JIKYXQY345UX3SQUCPDWJ25NJQZ2TXGFRLVCRYXUOQ, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.693739649 (573h43m3.564822694s) [noise=true , address=45.91.116.6:4981 , piece_id=INTVU2EYTU3DIM6WQFQEGO6IEYH3YFP5U2VEN7A5KPTOKU2DWOOA, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.694315887 (573h43m3.565398942s) [noise=true , address=145.79.227.4:4220 , piece_id=SHHJ7EW7DTNOZXVBP6A64VIX7P6AQN62TBCUT3KTH4MIB6UXY3BA, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.694765222 (573h43m3.565848267s) [noise=true , address=145.79.227.29:4952 , piece_id=DKYDB2R4DNCGH6T3SLRR2MOZZUPVZZVRL2BKGYOBYXJJVPSZEJ4Q, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.695202545 (573h43m3.56628559s) [noise=true , address=145.79.227.40:4058 , piece_id=KRHH2QZKSDPWTSSKYDVDWGKFJD3LPGP6U7KI2PPJ5T3TXYIUGJWA, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.695687528 (573h43m3.566770584s) [noise=true , address=77.90.31.19:4857 , piece_id=Q35LXGPTOAG3R2UL3JLL7DNHHD4PSMEULYW6G3SPXKNVFZ5VTNRA, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.696135221 (573h43m3.567218276s) [noise=true , address=45.91.116.8:4051 , piece_id=GDJFLQCHUF7BFQ7BVSNTVJ3TYZHYVOO6YYMO3KUIUSIEPAF4SFDQ, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.696703733 (573h43m3.567786778s) [noise=true , address=64.49.13.170:4720 , piece_id=HF6X3OT2EPPDE3DWAPWABVZUR3ERDBWXHG3NGNKEWZAD7G5AN4TA, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.698418298 (573h43m3.569501343s) [noise=true , address=145.79.227.36:4520 , piece_id=MYJSXUH563JFUW5Q7YUIWL7466G6ASY5WXR3T2PGUFFCTRTOBYJA, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.700198829 (573h43m3.571281874s) [noise=true , address=77.90.31.18:4821 , piece_id=LO5W47FHYOVGY5Q6BH5L6KX77HES3XRAFUTPI45HYPJZDO3YHLCQ, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.701993145 (573h43m3.57307619s) [noise=true , address=193.149.164.143:4699 , piece_id=UDFDCYO25MOJBTMDE6F2WILXE3HLOB75UJZ4LW7ZMNSCOBJI6UJA, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.703884687 (573h43m3.574967732s) [noise=true , address=145.79.227.33:4893 , piece_id=XGLX3DFNAY4F4GPGCGZ4ONDJ7NK62SAKX3PBRWHWI74EF3ZFMUSA, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.705687109 (573h43m3.576770154s) [noise=true , address=45.91.116.10:4094 , piece_id=6UUIZNTESBBWJMDQ5ZTCHU7S6YKM4GDKO4ZAC6T2FUTPVGFFDS3A, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.707557651 (573h43m3.578640696s) [noise=true , address=145.79.227.43:4164 , piece_id=J6Q4ZDOIIIBA2FXGZZTD6AYS7KGAWSRXXUBAPDPQWYKNU5GNXBBQ, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:05.168247719 (573h43m5.039330754s) [upload=502975]: Next returned nil segment Dec 29 13:12:05.168259962 (573h43m5.039343007s) [upload=502975]: Waiting for error group managing segments... Dec 29 13:12:05.558873127 (573h43m5.429956162s) [noise=true , address=145.79.227.4:4220 , piece_id=SHHJ7EW7DTNOZXVBP6A64VIX7P6AQN62TBCUT3KTH4MIB6UXY3BA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:05.563024312 (573h43m5.434107347s) [noise=true , address=64.49.13.170:4720 , piece_id=HF6X3OT2EPPDE3DWAPWABVZUR3ERDBWXHG3NGNKEWZAD7G5AN4TA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:05.564115719 (573h43m5.435198755s) [noise=true , address=145.79.227.36:4520 , piece_id=MYJSXUH563JFUW5Q7YUIWL7466G6ASY5WXR3T2PGUFFCTRTOBYJA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:05.593873402 (573h43m5.464956447s) [noise=true , address=145.79.227.29:4952 , piece_id=DKYDB2R4DNCGH6T3SLRR2MOZZUPVZZVRL2BKGYOBYXJJVPSZEJ4Q, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:05.898194604 (573h43m5.769277639s) [noise=true , address=45.91.116.10:4094 , piece_id=6UUIZNTESBBWJMDQ5ZTCHU7S6YKM4GDKO4ZAC6T2FUTPVGFFDS3A, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:05.912845292 (573h43m5.783928337s) [noise=true , address=77.90.31.19:4857 , piece_id=Q35LXGPTOAG3R2UL3JLL7DNHHD4PSMEULYW6G3SPXKNVFZ5VTNRA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:06.426532207 (573h43m6.297615252s) [noise=true , address=193.149.164.143:4699 , piece_id=UDFDCYO25MOJBTMDE6F2WILXE3HLOB75UJZ4LW7ZMNSCOBJI6UJA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:06.449589562 (573h43m6.320672607s) [noise=true , address=77.90.31.15:4711 , piece_id=JITX25FNZ4JIKYXQY345UX3SQUCPDWJ25NJQZ2TXGFRLVCRYXUOQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:06.453186841 (573h43m6.324269876s) [noise=true , address=145.79.227.40:4058 , piece_id=KRHH2QZKSDPWTSSKYDVDWGKFJD3LPGP6U7KI2PPJ5T3TXYIUGJWA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:06.510769276 (573h43m6.381852321s) [noise=true , address=145.79.227.43:4164 , piece_id=J6Q4ZDOIIIBA2FXGZZTD6AYS7KGAWSRXXUBAPDPQWYKNU5GNXBBQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:06.518868879 (573h43m6.389951924s) [noise=true , address=145.79.227.33:4893 , piece_id=XGLX3DFNAY4F4GPGCGZ4ONDJ7NK62SAKX3PBRWHWI74EF3ZFMUSA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:06.519238043 (573h43m6.390321098s) [noise=true , address=45.91.116.6:4981 , piece_id=INTVU2EYTU3DIM6WQFQEGO6IEYH3YFP5U2VEN7A5KPTOKU2DWOOA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:06.523211328 (573h43m6.394294373s) [noise=true , address=77.90.31.16:4759 , piece_id=ARC4HRP57U5VQXZTT63QYLXBPZURI5VMJBIQZOFKGLJBQ3K3KGLQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:06.767869783 (573h43m6.638952818s) [noise=true , address=145.79.227.34:4924 , piece_id=MAQXQTAGXQ7AEPUCESUSUGMHSDMMXS7HHKAEA7B3HW5X66PDUF5A, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:06.767889431 (573h43m6.638972476s) [seg_pos={0 0}, upload=502975]: Segment reached optimal threshold of 14 pieces. Dec 29 13:12:06.768102116 (573h43m6.639185161s) [noise=true , address=45.91.116.8:4051 , piece_id=GDJFLQCHUF7BFQ7BVSNTVJ3TYZHYVOO6YYMO3KUIUSIEPAF4SFDQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1preT33ei4yVXXYVLHhaUbHbiyL14oHFEiof6cCWPig83W6CT3): protocol: expected piece hash; context canceled; EOF Dec 29 13:12:06.768108257 (573h43m6.639191302s) [noise=true , address=45.91.116.8:4051 , piece_id=GDJFLQCHUF7BFQ7BVSNTVJ3TYZHYVOO6YYMO3KUIUSIEPAF4SFDQ, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold Dec 29 13:12:06.768219208 (573h43m6.639302254s) [noise=true , address=77.90.31.18:4821 , piece_id=LO5W47FHYOVGY5Q6BH5L6KX77HES3XRAFUTPI45HYPJZDO3YHLCQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH): protocol: expected piece hash; context canceled; EOF Dec 29 13:12:06.768224789 (573h43m6.639307824s) [noise=true , address=77.90.31.18:4821 , piece_id=LO5W47FHYOVGY5Q6BH5L6KX77HES3XRAFUTPI45HYPJZDO3YHLCQ, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold Dec 29 13:12:06.768384213 (573h43m6.639467258s) [noise=true , address=64.49.13.169:4711 , piece_id=4F4T4YKXMT7NF2C5KBUGJHQJGZL4VCKNKYB266PCVI3UACHJXZ5Q, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:48802->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:48802->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48802->64.49.13.169:4711: use of closed network connection Dec 29 13:12:06.768399261 (573h43m6.639482296s) [noise=true , address=64.49.13.169:4711 , piece_id=4F4T4YKXMT7NF2C5KBUGJHQJGZL4VCKNKYB266PCVI3UACHJXZ5Q, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold Dec 29 13:12:06.768732636 (573h43m6.639815671s) [noise=true , address=64.49.13.169:4709 , piece_id=TBHHX5ES2A6NQRGP25HIC7J7IBWLOJGQ2LO7MUXLHE5YET7BHLLA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:35310->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:35310->64.49.13.169:4709: use of closed network connection Dec 29 13:12:06.768745460 (573h43m6.639828505s) [noise=true , address=64.49.13.169:4709 , piece_id=TBHHX5ES2A6NQRGP25HIC7J7IBWLOJGQ2LO7MUXLHE5YET7BHLLA, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold Dec 29 13:12:06.768784586 (573h43m6.639867631s) [seg_pos={0 0}, upload=502975]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 13:12:06.768880217 (573h43m6.639963252s) [upload=502975]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 13:12:06.768884906 (573h43m6.639967941s) [upload=502975]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 13:12:07.067188330 (573h43m6.938271365s) [upload=502975]: Done uploading segments... Dec 29 13:12:07.067251099 (573h43m6.938334134s) [upload=502975]: Upload finished. err: <nil> Duration: 48.079222726s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...............X......................X....X.X....X.X.......] duration: 15.830790231s trace: https://jaeger.dev.storj.io/trace/312ddb07b2dee82a FAIL (2025-12-29 13:17:58 UTC) orbiter warning: context deadline exceeded Duration: 15.830809767s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 51 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXX.XXXXXXX.X.XX.XXX.XX.XXXXXXX.XX.XXXXXXXXXXX.X] duration: 5.001350345s trace: https://jaeger.dev.storj.io/trace/2851c7e57c313806 FAIL (2025-12-29 13:17:51 UTC) orbiter warning: context deadline exceeded Duration: 5.001375182s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [................X......................X....X.X....X.X......] duration: 16.693111985s trace: https://jaeger.dev.storj.io/trace/312ddb07b2dee82a FAIL (2025-12-29 13:17:05 UTC) orbiter warning: context deadline exceeded Duration: 16.693138967s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [..X......................X.....X.....................XX.....] duration: 1m0.003065055s trace: https://jaeger.dev.storj.io/trace/77d26141ea11d228 FAIL (2025-12-29 13:17:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12J2weU7Bmsr3RdQQ9QMuadT96XReBkyxTeaLJ2SEqVGGFX1PWu, Piece ID: JITX25FNZ4JIKYXQY345UX3SQUCPDWJ25NJQZ2TXGFRLVCRYXUOQ) context deadline exceeded; piecestore: (Node ID: 125sm6iHV94mJFLWrKeyRMH4b1qSMo32nDuW8uQr71xUL7Cema2, Piece ID: INTVU2EYTU3DIM6WQFQEGO6IEYH3YFP5U2VEN7A5KPTOKU2DWOOA) context deadline exceeded; piecestore: (Node ID: 12i57FF97oEddYLZpNxSq5or9WLEquL8ZW1cVUuW4AThBbDqcCL, Piece ID: Q35LXGPTOAG3R2UL3JLL7DNHHD4PSMEULYW6G3SPXKNVFZ5VTNRA) context deadline exceeded ======LOGS====== Q2LO7MUXLHE5YET7BHLLA, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.692912334 (573h43m3.563995379s) [noise=true , address=77.90.31.16:4759 , piece_id=ARC4HRP57U5VQXZTT63QYLXBPZURI5VMJBIQZOFKGLJBQ3K3KGLQ, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.693322595 (573h43m3.56440564s) [noise=true , address=77.90.31.15:4711 , piece_id=JITX25FNZ4JIKYXQY345UX3SQUCPDWJ25NJQZ2TXGFRLVCRYXUOQ, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.693739649 (573h43m3.564822694s) [noise=true , address=45.91.116.6:4981 , piece_id=INTVU2EYTU3DIM6WQFQEGO6IEYH3YFP5U2VEN7A5KPTOKU2DWOOA, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.694315887 (573h43m3.565398942s) [noise=true , address=145.79.227.4:4220 , piece_id=SHHJ7EW7DTNOZXVBP6A64VIX7P6AQN62TBCUT3KTH4MIB6UXY3BA, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.694765222 (573h43m3.565848267s) [noise=true , address=145.79.227.29:4952 , piece_id=DKYDB2R4DNCGH6T3SLRR2MOZZUPVZZVRL2BKGYOBYXJJVPSZEJ4Q, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.695202545 (573h43m3.56628559s) [noise=true , address=145.79.227.40:4058 , piece_id=KRHH2QZKSDPWTSSKYDVDWGKFJD3LPGP6U7KI2PPJ5T3TXYIUGJWA, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.695687528 (573h43m3.566770584s) [noise=true , address=77.90.31.19:4857 , piece_id=Q35LXGPTOAG3R2UL3JLL7DNHHD4PSMEULYW6G3SPXKNVFZ5VTNRA, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.696135221 (573h43m3.567218276s) [noise=true , address=45.91.116.8:4051 , piece_id=GDJFLQCHUF7BFQ7BVSNTVJ3TYZHYVOO6YYMO3KUIUSIEPAF4SFDQ, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.696703733 (573h43m3.567786778s) [noise=true , address=64.49.13.170:4720 , piece_id=HF6X3OT2EPPDE3DWAPWABVZUR3ERDBWXHG3NGNKEWZAD7G5AN4TA, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.698418298 (573h43m3.569501343s) [noise=true , address=145.79.227.36:4520 , piece_id=MYJSXUH563JFUW5Q7YUIWL7466G6ASY5WXR3T2PGUFFCTRTOBYJA, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.700198829 (573h43m3.571281874s) [noise=true , address=77.90.31.18:4821 , piece_id=LO5W47FHYOVGY5Q6BH5L6KX77HES3XRAFUTPI45HYPJZDO3YHLCQ, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.701993145 (573h43m3.57307619s) [noise=true , address=193.149.164.143:4699 , piece_id=UDFDCYO25MOJBTMDE6F2WILXE3HLOB75UJZ4LW7ZMNSCOBJI6UJA, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.703884687 (573h43m3.574967732s) [noise=true , address=145.79.227.33:4893 , piece_id=XGLX3DFNAY4F4GPGCGZ4ONDJ7NK62SAKX3PBRWHWI74EF3ZFMUSA, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.705687109 (573h43m3.576770154s) [noise=true , address=45.91.116.10:4094 , piece_id=6UUIZNTESBBWJMDQ5ZTCHU7S6YKM4GDKO4ZAC6T2FUTPVGFFDS3A, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.707557651 (573h43m3.578640696s) [noise=true , address=145.79.227.43:4164 , piece_id=J6Q4ZDOIIIBA2FXGZZTD6AYS7KGAWSRXXUBAPDPQWYKNU5GNXBBQ, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:05.168247719 (573h43m5.039330754s) [upload=502975]: Next returned nil segment Dec 29 13:12:05.168259962 (573h43m5.039343007s) [upload=502975]: Waiting for error group managing segments... Dec 29 13:12:05.558873127 (573h43m5.429956162s) [noise=true , address=145.79.227.4:4220 , piece_id=SHHJ7EW7DTNOZXVBP6A64VIX7P6AQN62TBCUT3KTH4MIB6UXY3BA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:05.563024312 (573h43m5.434107347s) [noise=true , address=64.49.13.170:4720 , piece_id=HF6X3OT2EPPDE3DWAPWABVZUR3ERDBWXHG3NGNKEWZAD7G5AN4TA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:05.564115719 (573h43m5.435198755s) [noise=true , address=145.79.227.36:4520 , piece_id=MYJSXUH563JFUW5Q7YUIWL7466G6ASY5WXR3T2PGUFFCTRTOBYJA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:05.593873402 (573h43m5.464956447s) [noise=true , address=145.79.227.29:4952 , piece_id=DKYDB2R4DNCGH6T3SLRR2MOZZUPVZZVRL2BKGYOBYXJJVPSZEJ4Q, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:05.898194604 (573h43m5.769277639s) [noise=true , address=45.91.116.10:4094 , piece_id=6UUIZNTESBBWJMDQ5ZTCHU7S6YKM4GDKO4ZAC6T2FUTPVGFFDS3A, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:05.912845292 (573h43m5.783928337s) [noise=true , address=77.90.31.19:4857 , piece_id=Q35LXGPTOAG3R2UL3JLL7DNHHD4PSMEULYW6G3SPXKNVFZ5VTNRA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:06.426532207 (573h43m6.297615252s) [noise=true , address=193.149.164.143:4699 , piece_id=UDFDCYO25MOJBTMDE6F2WILXE3HLOB75UJZ4LW7ZMNSCOBJI6UJA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:06.449589562 (573h43m6.320672607s) [noise=true , address=77.90.31.15:4711 , piece_id=JITX25FNZ4JIKYXQY345UX3SQUCPDWJ25NJQZ2TXGFRLVCRYXUOQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:06.453186841 (573h43m6.324269876s) [noise=true , address=145.79.227.40:4058 , piece_id=KRHH2QZKSDPWTSSKYDVDWGKFJD3LPGP6U7KI2PPJ5T3TXYIUGJWA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:06.510769276 (573h43m6.381852321s) [noise=true , address=145.79.227.43:4164 , piece_id=J6Q4ZDOIIIBA2FXGZZTD6AYS7KGAWSRXXUBAPDPQWYKNU5GNXBBQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:06.518868879 (573h43m6.389951924s) [noise=true , address=145.79.227.33:4893 , piece_id=XGLX3DFNAY4F4GPGCGZ4ONDJ7NK62SAKX3PBRWHWI74EF3ZFMUSA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:06.519238043 (573h43m6.390321098s) [noise=true , address=45.91.116.6:4981 , piece_id=INTVU2EYTU3DIM6WQFQEGO6IEYH3YFP5U2VEN7A5KPTOKU2DWOOA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:06.523211328 (573h43m6.394294373s) [noise=true , address=77.90.31.16:4759 , piece_id=ARC4HRP57U5VQXZTT63QYLXBPZURI5VMJBIQZOFKGLJBQ3K3KGLQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:06.767869783 (573h43m6.638952818s) [noise=true , address=145.79.227.34:4924 , piece_id=MAQXQTAGXQ7AEPUCESUSUGMHSDMMXS7HHKAEA7B3HW5X66PDUF5A, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:06.767889431 (573h43m6.638972476s) [seg_pos={0 0}, upload=502975]: Segment reached optimal threshold of 14 pieces. Dec 29 13:12:06.768102116 (573h43m6.639185161s) [noise=true , address=45.91.116.8:4051 , piece_id=GDJFLQCHUF7BFQ7BVSNTVJ3TYZHYVOO6YYMO3KUIUSIEPAF4SFDQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1preT33ei4yVXXYVLHhaUbHbiyL14oHFEiof6cCWPig83W6CT3): protocol: expected piece hash; context canceled; EOF Dec 29 13:12:06.768108257 (573h43m6.639191302s) [noise=true , address=45.91.116.8:4051 , piece_id=GDJFLQCHUF7BFQ7BVSNTVJ3TYZHYVOO6YYMO3KUIUSIEPAF4SFDQ, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold Dec 29 13:12:06.768219208 (573h43m6.639302254s) [noise=true , address=77.90.31.18:4821 , piece_id=LO5W47FHYOVGY5Q6BH5L6KX77HES3XRAFUTPI45HYPJZDO3YHLCQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH): protocol: expected piece hash; context canceled; EOF Dec 29 13:12:06.768224789 (573h43m6.639307824s) [noise=true , address=77.90.31.18:4821 , piece_id=LO5W47FHYOVGY5Q6BH5L6KX77HES3XRAFUTPI45HYPJZDO3YHLCQ, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold Dec 29 13:12:06.768384213 (573h43m6.639467258s) [noise=true , address=64.49.13.169:4711 , piece_id=4F4T4YKXMT7NF2C5KBUGJHQJGZL4VCKNKYB266PCVI3UACHJXZ5Q, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:48802->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:48802->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48802->64.49.13.169:4711: use of closed network connection Dec 29 13:12:06.768399261 (573h43m6.639482296s) [noise=true , address=64.49.13.169:4711 , piece_id=4F4T4YKXMT7NF2C5KBUGJHQJGZL4VCKNKYB266PCVI3UACHJXZ5Q, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold Dec 29 13:12:06.768732636 (573h43m6.639815671s) [noise=true , address=64.49.13.169:4709 , piece_id=TBHHX5ES2A6NQRGP25HIC7J7IBWLOJGQ2LO7MUXLHE5YET7BHLLA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:35310->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:35310->64.49.13.169:4709: use of closed network connection Dec 29 13:12:06.768745460 (573h43m6.639828505s) [noise=true , address=64.49.13.169:4709 , piece_id=TBHHX5ES2A6NQRGP25HIC7J7IBWLOJGQ2LO7MUXLHE5YET7BHLLA, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold Dec 29 13:12:06.768784586 (573h43m6.639867631s) [seg_pos={0 0}, upload=502975]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 13:12:06.768880217 (573h43m6.639963252s) [upload=502975]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 13:12:06.768884906 (573h43m6.639967941s) [upload=502975]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 13:12:07.067188330 (573h43m6.938271365s) [upload=502975]: Done uploading segments... Dec 29 13:12:07.067251099 (573h43m6.938334134s) [upload=502975]: Upload finished. err: <nil> Duration: 1m0.003081858s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 50 out of 60 is failed (threshold=6) [.XXXXXXXXXXXXXXX.XXXXXXX.X.XX.XXX.XX.XXXXXXX.XX.XXXXXXXXXXX.] duration: 3.546565357s trace: https://jaeger.dev.storj.io/trace/3b4cc1fe0622afb1 FAIL (2025-12-29 13:17:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.546587741s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 31 out of 60 is failed (threshold=3) [...XXXX..XX..X...XX.X.XX...X.X.X.XXX.X.X.X.XX..X.XXX.XX..X.X] duration: 30.005002922s trace: https://jaeger.dev.storj.io/trace/2c2f58d8f8f5d89d FAIL (2025-12-29 13:16:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 13:16:28.200614904 (573h47m28.071697949s) [upload=503036]: Starting upload Dec 29 13:16:28.200668196 (573h47m28.071751231s) [upload=503036]: Done starting upload Dec 29 13:16:28.200769267 (573h47m28.071852312s) [upload=503036]: Uploading segments... Dec 29 13:16:28.200935133 (573h47m28.072018168s) [upload=503036]: Got next segment. Inline: false Dec 29 13:16:28.200980740 (573h47m28.072063775s) [upload=503036]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 13:16:28.200985279 (573h47m28.072068314s) [upload=503036]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 13:16:28.448026662 (573h47m28.319109697s) [seg_pos={0 0}, upload=503036]: Begin upload segment... Dec 29 13:16:28.448070365 (573h47m28.31915341s) [seg_pos={0 0}, upload=503036]: Done begin upload segment. Dec 29 13:16:28.448095453 (573h47m28.319178498s) [noise=true , address=145.79.227.38:4978 , piece_id=4B4LRE24UDXHOU7YUF22MXMJ42Y4DY4TAN4ST2KSJELOMTOVISAA, seg_pos={0 0}, upload=503036]: Uploading piece... Dec 29 13:16:28.449538911 (573h47m28.320622427s) [noise=true , address=77.90.31.12:4590 , piece_id=ZWS5BGCEQ74FJIM2EGZRM7PIV65WZPOTTGIOSUVR4XJU7GO3LTBQ, seg_pos={0 0}, upload=503036]: Uploading piece... Dec 29 13:16:28.450077097 (573h47m28.321160132s) [noise=true , address=145.79.227.36:4519 , piece_id=RKEC6V4BPQHUL2U7EQPRVPZFD7W4JKOGKLQLJU7KLVUUA3UFEHDA, seg_pos={0 0}, upload=503036]: Uploading piece... Dec 29 13:16:28.450502957 (573h47m28.321585992s) [noise=true , address=145.79.227.5:4223 , piece_id=232OAVTIDNYJLXBSQ4GQXSF3JOFCTQ2T4AEF44FBDXOYMHKY5JAA, seg_pos={0 0}, upload=503036]: Uploading piece... Dec 29 13:16:28.450830450 (573h47m28.321913496s) [noise=true , address=77.90.31.14:4662 , piece_id=HOMGJL4MX7PCFC2H6UGGRYRLU36S2GNUNJMOFQDUQQ625RZQKGPA, seg_pos={0 0}, upload=503036]: Uploading piece... Dec 29 13:16:28.451138517 (573h47m28.322221562s) [noise=true , address=145.79.227.34:4905 , piece_id=TN67EMFO6IPXQERT6CSTUWBPMVB24OGHK4O5D4Z5K5JLE2JZ75GA, seg_pos={0 0}, upload=503036]: Uploading piece... Dec 29 13:16:28.451435123 (573h47m28.322518168s) [noise=true , address=145.79.227.15:4873 , piece_id=PZ5RA3QF7BKL7PQGGEMSHMY6QAIPOPUFA4CTOUFIOYORVF3OXANA, seg_pos={0 0}, upload=503036]: Uploading piece... Dec 29 13:16:28.452517283 (573h47m28.323600318s) [noise=true , address=145.79.227.11:4732 , piece_id=K7VNACJQ6SPUXZJGGCPWLYS2MTGZCHOFWDILKNUHCS3RQC62W6KQ, seg_pos={0 0}, upload=503036]: Uploading piece... Dec 29 13:16:28.453648376 (573h47m28.324731421s) [noise=true , address=77.90.31.13:4629 , piece_id=DDVPKSA2DOGZA6PRYKAMJEGNNFERARZANJGUL6M3KNRT77KQN74A, seg_pos={0 0}, upload=503036]: Uploading piece... Dec 29 13:16:28.455018224 (573h47m28.32610127s) [noise=true , address=77.90.31.11:4554 , piece_id=IXFFSZ6Y3H6CAHCCMYHOYDCW2R7OWX6X6YMX5ASJYAUIOJB46J7Q, seg_pos={0 0}, upload=503036]: Uploading piece... Dec 29 13:16:28.456598685 (573h47m28.32768173s) [noise=true , address=145.79.227.42:4121 , piece_id=NZBRJUABDGME323XTIZRGIL3LWDO5J3CLVEQTN4EGG6GNZRRA64A, seg_pos={0 0}, upload=503036]: Uploading piece... Dec 29 13:16:28.457941451 (573h47m28.329024496s) [noise=true , address=145.79.227.51:4409 , piece_id=3ZXHFX75K7WNKO6GC5OHK523GU32FC2KJENO2PJQQKQGUB5T77YQ, seg_pos={0 0}, upload=503036]: Uploading piece... Dec 29 13:16:28.459153279 (573h47m28.330236314s) [noise=true , address=145.79.227.45:4229 , piece_id=S75QGRFW24EMSTAXVIBPSUYLRSYSTAVSCMHHSIOFUIXJ2IYLVN7Q, seg_pos={0 0}, upload=503036]: Uploading piece... Dec 29 13:16:28.460108889 (573h47m28.331191924s) [noise=true , address=45.91.116.4:4940 , piece_id=NIT6B2IQZZ3NRWOY5UHNE4RLD4FED7XAXOJSHPBTZTYNIZZXPKJQ, seg_pos={0 0}, upload=503036]: Uploading piece... Dec 29 13:16:28.460531423 (573h47m28.331614468s) [noise=true , address=45.91.116.11:4117 , piece_id=6L72KYW5N3UEULAELVXF7RX34OS3U4JTNC3U74CTNJYMBU5QUYQA, seg_pos={0 0}, upload=503036]: Uploading piece... Dec 29 13:16:28.462735500 (573h47m28.333818535s) [noise=true , address=45.91.116.9:4261 , piece_id=ETCOYVYUJNI6LTF2ETALFEDETEYX6VKQWJ2LRZN6RMF2HAUFZPUA, seg_pos={0 0}, upload=503036]: Uploading piece... Dec 29 13:16:30.164010988 (573h47m30.035094023s) [upload=503036]: Next returned nil segment Dec 29 13:16:30.164024012 (573h47m30.035107057s) [upload=503036]: Waiting for error group managing segments... Dec 29 13:16:30.445792321 (573h47m30.316875366s) [noise=true , address=145.79.227.36:4519 , piece_id=RKEC6V4BPQHUL2U7EQPRVPZFD7W4JKOGKLQLJU7KLVUUA3UFEHDA, seg_pos={0 0}, upload=503036]: Done uploading piece. err: <nil> Dec 29 13:16:30.457988935 (573h47m30.32907198s) [noise=true , address=145.79.227.38:4978 , piece_id=4B4LRE24UDXHOU7YUF22MXMJ42Y4DY4TAN4ST2KSJELOMTOVISAA, seg_pos={0 0}, upload=503036]: Done uploading piece. err: <nil> Dec 29 13:16:30.459344606 (573h47m30.330427651s) [noise=true , address=145.79.227.42:4121 , piece_id=NZBRJUABDGME323XTIZRGIL3LWDO5J3CLVEQTN4EGG6GNZRRA64A, seg_pos={0 0}, upload=503036]: Done uploading piece. err: <nil> Dec 29 13:16:30.459574563 (573h47m30.330657609s) [noise=true , address=145.79.227.15:4873 , piece_id=PZ5RA3QF7BKL7PQGGEMSHMY6QAIPOPUFA4CTOUFIOYORVF3OXANA, seg_pos={0 0}, upload=503036]: Done uploading piece. err: <nil> Dec 29 13:16:30.460826417 (573h47m30.331909462s) [noise=true , address=145.79.227.45:4229 , piece_id=S75QGRFW24EMSTAXVIBPSUYLRSYSTAVSCMHHSIOFUIXJ2IYLVN7Q, seg_pos={0 0}, upload=503036]: Done uploading piece. err: <nil> Dec 29 13:16:30.461062587 (573h47m30.332145632s) [noise=true , address=145.79.227.5:4223 , piece_id=232OAVTIDNYJLXBSQ4GQXSF3JOFCTQ2T4AEF44FBDXOYMHKY5JAA, seg_pos={0 0}, upload=503036]: Done uploading piece. err: <nil> Dec 29 13:16:30.470744513 (573h47m30.341827548s) [noise=true , address=145.79.227.51:4409 , piece_id=3ZXHFX75K7WNKO6GC5OHK523GU32FC2KJENO2PJQQKQGUB5T77YQ, seg_pos={0 0}, upload=503036]: Done uploading piece. err: <nil> Dec 29 13:16:30.471098787 (573h47m30.342181823s) [noise=true , address=145.79.227.11:4732 , piece_id=K7VNACJQ6SPUXZJGGCPWLYS2MTGZCHOFWDILKNUHCS3RQC62W6KQ, seg_pos={0 0}, upload=503036]: Done uploading piece. err: <nil> Dec 29 13:16:30.656370792 (573h47m30.527453827s) [noise=true , address=77.90.31.11:4554 , piece_id=IXFFSZ6Y3H6CAHCCMYHOYDCW2R7OWX6X6YMX5ASJYAUIOJB46J7Q, seg_pos={0 0}, upload=503036]: Done uploading piece. err: <nil> Dec 29 13:16:30.667724421 (573h47m30.538807456s) [noise=true , address=77.90.31.13:4629 , piece_id=DDVPKSA2DOGZA6PRYKAMJEGNNFERARZANJGUL6M3KNRT77KQN74A, seg_pos={0 0}, upload=503036]: Done uploading piece. err: <nil> Dec 29 13:16:30.803570542 (573h47m30.674653587s) [noise=true , address=77.90.31.12:4590 , piece_id=ZWS5BGCEQ74FJIM2EGZRM7PIV65WZPOTTGIOSUVR4XJU7GO3LTBQ, seg_pos={0 0}, upload=503036]: Done uploading piece. err: <nil> Dec 29 13:16:30.839511810 (573h47m30.710594855s) [noise=true , address=77.90.31.14:4662 , piece_id=HOMGJL4MX7PCFC2H6UGGRYRLU36S2GNUNJMOFQDUQQ625RZQKGPA, seg_pos={0 0}, upload=503036]: Done uploading piece. err: <nil> Dec 29 13:16:31.074238845 (573h47m30.9453219s) [noise=true , address=45.91.116.4:4940 , piece_id=NIT6B2IQZZ3NRWOY5UHNE4RLD4FED7XAXOJSHPBTZTYNIZZXPKJQ, seg_pos={0 0}, upload=503036]: Done uploading piece. err: <nil> Dec 29 13:16:31.074266748 (573h47m30.945349793s) [seg_pos={0 0}, upload=503036]: Segment reached optimal threshold of 13 pieces. Dec 29 13:16:31.074530430 (573h47m30.945613475s) [noise=true , address=145.79.227.34:4905 , piece_id=TN67EMFO6IPXQERT6CSTUWBPMVB24OGHK4O5D4Z5K5JLE2JZ75GA, seg_pos={0 0}, upload=503036]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 29 13:16:31.074538445 (573h47m30.94562151s) [noise=true , address=145.79.227.34:4905 , piece_id=TN67EMFO6IPXQERT6CSTUWBPMVB24OGHK4O5D4Z5K5JLE2JZ75GA, seg_pos={0 0}, upload=503036]: Long tail cancellation due to optimal threshold Dec 29 13:16:31.074619430 (573h47m30.945702465s) [noise=true , address=45.91.116.9:4261 , piece_id=ETCOYVYUJNI6LTF2ETALFEDETEYX6VKQWJ2LRZN6RMF2HAUFZPUA, seg_pos={0 0}, upload=503036]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF Dec 29 13:16:31.074636702 (573h47m30.945719747s) [noise=true , address=45.91.116.9:4261 , piece_id=ETCOYVYUJNI6LTF2ETALFEDETEYX6VKQWJ2LRZN6RMF2HAUFZPUA, seg_pos={0 0}, upload=503036]: Long tail cancellation due to optimal threshold Dec 29 13:16:31.074790486 (573h47m30.945873521s) [noise=true , address=45.91.116.11:4117 , piece_id=6L72KYW5N3UEULAELVXF7RX34OS3U4JTNC3U74CTNJYMBU5QUYQA, seg_pos={0 0}, upload=503036]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF Dec 29 13:16:31.074796457 (573h47m30.945879502s) [noise=true , address=45.91.116.11:4117 , piece_id=6L72KYW5N3UEULAELVXF7RX34OS3U4JTNC3U74CTNJYMBU5QUYQA, seg_pos={0 0}, upload=503036]: Long tail cancellation due to optimal threshold Dec 29 13:16:31.074812968 (573h47m30.945896003s) [seg_pos={0 0}, upload=503036]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 13:16:31.074897038 (573h47m30.945980084s) [upload=503036]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 13:16:31.074902118 (573h47m30.945985163s) [upload=503036]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 13:16:31.361425380 (573h47m31.232508425s) [upload=503036]: Done uploading segments... Dec 29 13:16:31.361445108 (573h47m31.232528143s) [upload=503036]: Upload finished. err: <nil> Duration: 30.005021607s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.................X......................X....X.X....X.X.....] duration: 13.042748993s trace: https://jaeger.dev.storj.io/trace/312ddb07b2dee82a FAIL (2025-12-29 13:16:44 UTC) orbiter warning: context deadline exceeded Duration: 13.042760083s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 51 out of 60 is failed (threshold=6) [X.XXXXXXXXXXXXXXX.XXXXXXX.X.XX.XXX.XX.XXXXXXX.XX.XXXXXXXXXXX] duration: 5.001292813s trace: https://jaeger.dev.storj.io/trace/3b4cc1fe0622afb1 FAIL (2025-12-29 13:16:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001320335s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..................X......................X....X.X....X.X....] duration: 19.738260043s trace: https://jaeger.dev.storj.io/trace/312ddb07b2dee82a FAIL (2025-12-29 13:16:31 UTC) orbiter warning: context deadline exceeded Duration: 19.73836346s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [...X......................X.....X.....................XX....] duration: 52.697793234s trace: https://jaeger.dev.storj.io/trace/77d26141ea11d228 FAIL (2025-12-29 13:16:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12J2weU7Bmsr3RdQQ9QMuadT96XReBkyxTeaLJ2SEqVGGFX1PWu, Piece ID: JITX25FNZ4JIKYXQY345UX3SQUCPDWJ25NJQZ2TXGFRLVCRYXUOQ) context deadline exceeded; piecestore: (Node ID: 125sm6iHV94mJFLWrKeyRMH4b1qSMo32nDuW8uQr71xUL7Cema2, Piece ID: INTVU2EYTU3DIM6WQFQEGO6IEYH3YFP5U2VEN7A5KPTOKU2DWOOA) context deadline exceeded; piecestore: (Node ID: 12i57FF97oEddYLZpNxSq5or9WLEquL8ZW1cVUuW4AThBbDqcCL, Piece ID: Q35LXGPTOAG3R2UL3JLL7DNHHD4PSMEULYW6G3SPXKNVFZ5VTNRA) context deadline exceeded ======LOGS====== Q2LO7MUXLHE5YET7BHLLA, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.692912334 (573h43m3.563995379s) [noise=true , address=77.90.31.16:4759 , piece_id=ARC4HRP57U5VQXZTT63QYLXBPZURI5VMJBIQZOFKGLJBQ3K3KGLQ, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.693322595 (573h43m3.56440564s) [noise=true , address=77.90.31.15:4711 , piece_id=JITX25FNZ4JIKYXQY345UX3SQUCPDWJ25NJQZ2TXGFRLVCRYXUOQ, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.693739649 (573h43m3.564822694s) [noise=true , address=45.91.116.6:4981 , piece_id=INTVU2EYTU3DIM6WQFQEGO6IEYH3YFP5U2VEN7A5KPTOKU2DWOOA, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.694315887 (573h43m3.565398942s) [noise=true , address=145.79.227.4:4220 , piece_id=SHHJ7EW7DTNOZXVBP6A64VIX7P6AQN62TBCUT3KTH4MIB6UXY3BA, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.694765222 (573h43m3.565848267s) [noise=true , address=145.79.227.29:4952 , piece_id=DKYDB2R4DNCGH6T3SLRR2MOZZUPVZZVRL2BKGYOBYXJJVPSZEJ4Q, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.695202545 (573h43m3.56628559s) [noise=true , address=145.79.227.40:4058 , piece_id=KRHH2QZKSDPWTSSKYDVDWGKFJD3LPGP6U7KI2PPJ5T3TXYIUGJWA, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.695687528 (573h43m3.566770584s) [noise=true , address=77.90.31.19:4857 , piece_id=Q35LXGPTOAG3R2UL3JLL7DNHHD4PSMEULYW6G3SPXKNVFZ5VTNRA, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.696135221 (573h43m3.567218276s) [noise=true , address=45.91.116.8:4051 , piece_id=GDJFLQCHUF7BFQ7BVSNTVJ3TYZHYVOO6YYMO3KUIUSIEPAF4SFDQ, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.696703733 (573h43m3.567786778s) [noise=true , address=64.49.13.170:4720 , piece_id=HF6X3OT2EPPDE3DWAPWABVZUR3ERDBWXHG3NGNKEWZAD7G5AN4TA, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.698418298 (573h43m3.569501343s) [noise=true , address=145.79.227.36:4520 , piece_id=MYJSXUH563JFUW5Q7YUIWL7466G6ASY5WXR3T2PGUFFCTRTOBYJA, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.700198829 (573h43m3.571281874s) [noise=true , address=77.90.31.18:4821 , piece_id=LO5W47FHYOVGY5Q6BH5L6KX77HES3XRAFUTPI45HYPJZDO3YHLCQ, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.701993145 (573h43m3.57307619s) [noise=true , address=193.149.164.143:4699 , piece_id=UDFDCYO25MOJBTMDE6F2WILXE3HLOB75UJZ4LW7ZMNSCOBJI6UJA, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.703884687 (573h43m3.574967732s) [noise=true , address=145.79.227.33:4893 , piece_id=XGLX3DFNAY4F4GPGCGZ4ONDJ7NK62SAKX3PBRWHWI74EF3ZFMUSA, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.705687109 (573h43m3.576770154s) [noise=true , address=45.91.116.10:4094 , piece_id=6UUIZNTESBBWJMDQ5ZTCHU7S6YKM4GDKO4ZAC6T2FUTPVGFFDS3A, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.707557651 (573h43m3.578640696s) [noise=true , address=145.79.227.43:4164 , piece_id=J6Q4ZDOIIIBA2FXGZZTD6AYS7KGAWSRXXUBAPDPQWYKNU5GNXBBQ, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:05.168247719 (573h43m5.039330754s) [upload=502975]: Next returned nil segment Dec 29 13:12:05.168259962 (573h43m5.039343007s) [upload=502975]: Waiting for error group managing segments... Dec 29 13:12:05.558873127 (573h43m5.429956162s) [noise=true , address=145.79.227.4:4220 , piece_id=SHHJ7EW7DTNOZXVBP6A64VIX7P6AQN62TBCUT3KTH4MIB6UXY3BA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:05.563024312 (573h43m5.434107347s) [noise=true , address=64.49.13.170:4720 , piece_id=HF6X3OT2EPPDE3DWAPWABVZUR3ERDBWXHG3NGNKEWZAD7G5AN4TA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:05.564115719 (573h43m5.435198755s) [noise=true , address=145.79.227.36:4520 , piece_id=MYJSXUH563JFUW5Q7YUIWL7466G6ASY5WXR3T2PGUFFCTRTOBYJA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:05.593873402 (573h43m5.464956447s) [noise=true , address=145.79.227.29:4952 , piece_id=DKYDB2R4DNCGH6T3SLRR2MOZZUPVZZVRL2BKGYOBYXJJVPSZEJ4Q, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:05.898194604 (573h43m5.769277639s) [noise=true , address=45.91.116.10:4094 , piece_id=6UUIZNTESBBWJMDQ5ZTCHU7S6YKM4GDKO4ZAC6T2FUTPVGFFDS3A, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:05.912845292 (573h43m5.783928337s) [noise=true , address=77.90.31.19:4857 , piece_id=Q35LXGPTOAG3R2UL3JLL7DNHHD4PSMEULYW6G3SPXKNVFZ5VTNRA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:06.426532207 (573h43m6.297615252s) [noise=true , address=193.149.164.143:4699 , piece_id=UDFDCYO25MOJBTMDE6F2WILXE3HLOB75UJZ4LW7ZMNSCOBJI6UJA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:06.449589562 (573h43m6.320672607s) [noise=true , address=77.90.31.15:4711 , piece_id=JITX25FNZ4JIKYXQY345UX3SQUCPDWJ25NJQZ2TXGFRLVCRYXUOQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:06.453186841 (573h43m6.324269876s) [noise=true , address=145.79.227.40:4058 , piece_id=KRHH2QZKSDPWTSSKYDVDWGKFJD3LPGP6U7KI2PPJ5T3TXYIUGJWA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:06.510769276 (573h43m6.381852321s) [noise=true , address=145.79.227.43:4164 , piece_id=J6Q4ZDOIIIBA2FXGZZTD6AYS7KGAWSRXXUBAPDPQWYKNU5GNXBBQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:06.518868879 (573h43m6.389951924s) [noise=true , address=145.79.227.33:4893 , piece_id=XGLX3DFNAY4F4GPGCGZ4ONDJ7NK62SAKX3PBRWHWI74EF3ZFMUSA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:06.519238043 (573h43m6.390321098s) [noise=true , address=45.91.116.6:4981 , piece_id=INTVU2EYTU3DIM6WQFQEGO6IEYH3YFP5U2VEN7A5KPTOKU2DWOOA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:06.523211328 (573h43m6.394294373s) [noise=true , address=77.90.31.16:4759 , piece_id=ARC4HRP57U5VQXZTT63QYLXBPZURI5VMJBIQZOFKGLJBQ3K3KGLQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:06.767869783 (573h43m6.638952818s) [noise=true , address=145.79.227.34:4924 , piece_id=MAQXQTAGXQ7AEPUCESUSUGMHSDMMXS7HHKAEA7B3HW5X66PDUF5A, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:06.767889431 (573h43m6.638972476s) [seg_pos={0 0}, upload=502975]: Segment reached optimal threshold of 14 pieces. Dec 29 13:12:06.768102116 (573h43m6.639185161s) [noise=true , address=45.91.116.8:4051 , piece_id=GDJFLQCHUF7BFQ7BVSNTVJ3TYZHYVOO6YYMO3KUIUSIEPAF4SFDQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1preT33ei4yVXXYVLHhaUbHbiyL14oHFEiof6cCWPig83W6CT3): protocol: expected piece hash; context canceled; EOF Dec 29 13:12:06.768108257 (573h43m6.639191302s) [noise=true , address=45.91.116.8:4051 , piece_id=GDJFLQCHUF7BFQ7BVSNTVJ3TYZHYVOO6YYMO3KUIUSIEPAF4SFDQ, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold Dec 29 13:12:06.768219208 (573h43m6.639302254s) [noise=true , address=77.90.31.18:4821 , piece_id=LO5W47FHYOVGY5Q6BH5L6KX77HES3XRAFUTPI45HYPJZDO3YHLCQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH): protocol: expected piece hash; context canceled; EOF Dec 29 13:12:06.768224789 (573h43m6.639307824s) [noise=true , address=77.90.31.18:4821 , piece_id=LO5W47FHYOVGY5Q6BH5L6KX77HES3XRAFUTPI45HYPJZDO3YHLCQ, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold Dec 29 13:12:06.768384213 (573h43m6.639467258s) [noise=true , address=64.49.13.169:4711 , piece_id=4F4T4YKXMT7NF2C5KBUGJHQJGZL4VCKNKYB266PCVI3UACHJXZ5Q, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:48802->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:48802->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48802->64.49.13.169:4711: use of closed network connection Dec 29 13:12:06.768399261 (573h43m6.639482296s) [noise=true , address=64.49.13.169:4711 , piece_id=4F4T4YKXMT7NF2C5KBUGJHQJGZL4VCKNKYB266PCVI3UACHJXZ5Q, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold Dec 29 13:12:06.768732636 (573h43m6.639815671s) [noise=true , address=64.49.13.169:4709 , piece_id=TBHHX5ES2A6NQRGP25HIC7J7IBWLOJGQ2LO7MUXLHE5YET7BHLLA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:35310->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:35310->64.49.13.169:4709: use of closed network connection Dec 29 13:12:06.768745460 (573h43m6.639828505s) [noise=true , address=64.49.13.169:4709 , piece_id=TBHHX5ES2A6NQRGP25HIC7J7IBWLOJGQ2LO7MUXLHE5YET7BHLLA, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold Dec 29 13:12:06.768784586 (573h43m6.639867631s) [seg_pos={0 0}, upload=502975]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 13:12:06.768880217 (573h43m6.639963252s) [upload=502975]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 13:12:06.768884906 (573h43m6.639967941s) [upload=502975]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 13:12:07.067188330 (573h43m6.938271365s) [upload=502975]: Done uploading segments... Dec 29 13:12:07.067251099 (573h43m6.938334134s) [upload=502975]: Upload finished. err: <nil> Duration: 52.697814786s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...................X......................X....X.X....X.X...] duration: 18.201055011s trace: https://jaeger.dev.storj.io/trace/312ddb07b2dee82a FAIL (2025-12-29 13:15:38 UTC) orbiter warning: context deadline exceeded Duration: 18.201084428s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 30 out of 60 is failed (threshold=3) [....XXXX..XX..X...XX.X.XX...X.X.X.XXX.X.X.X.XX..X.XXX.XX..X.] duration: 5.707482704s trace: https://jaeger.dev.storj.io/trace/4e9cc163fa850803 FAIL (2025-12-29 13:15:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: 4TWHAVGGSGRKTKDQTEIQVC3VXOOFYYI3LKEGFNIWML4U2UCSHBDA) context deadline exceeded; piecestore: (Node ID: 1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, Piece ID: 32T4H5B3CRKUA25W7EOTIUEERZ3D6LSOJ5GASLULAVDKLEQEXYTQ) context deadline exceeded ======LOGS====== Dec 29 13:14:59.052691585 (573h45m58.92377462s) [upload=503016]: Starting upload Dec 29 13:14:59.052744516 (573h45m58.923827551s) [upload=503016]: Done starting upload Dec 29 13:14:59.052915451 (573h45m58.923998496s) [upload=503016]: Uploading segments... Dec 29 13:14:59.052954847 (573h45m58.924037882s) [upload=503016]: Got next segment. Inline: false Dec 29 13:14:59.053000473 (573h45m58.924083508s) [upload=503016]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 13:14:59.053007426 (573h45m58.924090461s) [upload=503016]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 13:14:59.334293867 (573h45m59.205376912s) [seg_pos={0 0}, upload=503016]: Begin upload segment... Dec 29 13:14:59.334347518 (573h45m59.205430563s) [seg_pos={0 0}, upload=503016]: Done begin upload segment. Dec 29 13:14:59.334378097 (573h45m59.205461132s) [noise=true , address=145.79.227.18:4007 , piece_id=7E3ITPKMZPM4Y5FHVCGOEGHERB34CMAL2ES3TZ6SB3S3CHETRNBA, seg_pos={0 0}, upload=503016]: Uploading piece... Dec 29 13:14:59.334400440 (573h45m59.205483485s) [noise=true , address=145.79.227.2:4114 , piece_id=NGIVG57VGFEFYWCRKPDNI42UFERFVBFZK6JOWPTUQNROMW2XUB2A, seg_pos={0 0}, upload=503016]: Uploading piece... Dec 29 13:14:59.335194562 (573h45m59.206277597s) [noise=true , address=145.79.227.52:4445 , piece_id=R2PDEEEOJOVC4YNYKCPI7YI6B6M6YTCDC43ZF3MCY67B47X23AGQ, seg_pos={0 0}, upload=503016]: Uploading piece... Dec 29 13:14:59.335784075 (573h45m59.20686712s) [noise=true , address=145.79.227.42:4122 , piece_id=2ZJBOZCTH3AYIPS7KOGGKQKNS436D3UNB55JSCDZ5PYC3RHHTGLQ, seg_pos={0 0}, upload=503016]: Uploading piece... Dec 29 13:14:59.336178325 (573h45m59.20726137s) [noise=true , address=45.91.116.10:4084 , piece_id=4TWHAVGGSGRKTKDQTEIQVC3VXOOFYYI3LKEGFNIWML4U2UCSHBDA, seg_pos={0 0}, upload=503016]: Uploading piece... Dec 29 13:14:59.336601731 (573h45m59.207684776s) [noise=true , address=145.79.227.12:4765 , piece_id=UPBPU2QVDRTTM5SWJMYIDL2W35GZFXAKEGFPY2YIDI7BQ55ARG6Q, seg_pos={0 0}, upload=503016]: Uploading piece... Dec 29 13:14:59.337009347 (573h45m59.208092383s) [noise=true , address=145.79.227.11:4732 , piece_id=IG6W4GXSF2MINKPLWLTSFDWYIR7HMUODH26H6OHY2GTK6AYDV5NQ, seg_pos={0 0}, upload=503016]: Uploading piece... Dec 29 13:14:59.337451069 (573h45m59.208534114s) [noise=true , address=77.90.31.16:4736 , piece_id=UV57UQIOHTCZJZTZMSM622PGJOPNBTE7L7MNOOCX355GHS2W6I2Q, seg_pos={0 0}, upload=503016]: Uploading piece... Dec 29 13:14:59.337986047 (573h45m59.209069082s) [noise=true , address=77.90.31.12:4589 , piece_id=KFFXI3ZILAJTJKSUZ3TLOLC2JXYWWFG7IYBUUQSNPQOFUIC74PSA, seg_pos={0 0}, upload=503016]: Uploading piece... Dec 29 13:14:59.338221566 (573h45m59.209304611s) [noise=true , address=145.79.227.13:4802 , piece_id=AZ7RK2VDHZYDYFRWHNHW6FMZJJZ3XRNOHN62EGVDNGBKBZ7J6SDQ, seg_pos={0 0}, upload=503016]: Uploading piece... Dec 29 13:14:59.338678055 (573h45m59.2097611s) [noise=true , address=145.79.227.19:4077 , piece_id=BI7V27VENV4XRS4RYUGNHUEDWCV4UW2ZHBG367MJXHRUEPYOL5MA, seg_pos={0 0}, upload=503016]: Uploading piece... Dec 29 13:14:59.338856685 (573h45m59.20993973s) [noise=true , address=45.91.116.11:4120 , piece_id=VKMMVQVYAGLKQGNVPH6G7G62HDMNIY3KDDHCDBDBY6MG4YFDJPWQ, seg_pos={0 0}, upload=503016]: Uploading piece... Dec 29 13:14:59.339166555 (573h45m59.21024959s) [noise=true , address=145.79.227.32:4834 , piece_id=V5TUJNYHCLJXQVEVOTZDXIXJRFEQO3CPOTUU5KDHPNPFSOLURF4Q, seg_pos={0 0}, upload=503016]: Uploading piece... Dec 29 13:14:59.339443873 (573h45m59.210526918s) [noise=true , address=145.79.227.38:4979 , piece_id=NK5USLTPICTEVKDWIBAL735G73RE2HBDGOTULXFG2NQRQDYKBVWA, seg_pos={0 0}, upload=503016]: Uploading piece... Dec 29 13:14:59.340070727 (573h45m59.211153782s) [noise=true , address=77.90.31.11:4556 , piece_id=32T4H5B3CRKUA25W7EOTIUEERZ3D6LSOJ5GASLULAVDKLEQEXYTQ, seg_pos={0 0}, upload=503016]: Uploading piece... Dec 29 13:14:59.340738428 (573h45m59.211821463s) [noise=true , address=145.79.227.16:4912 , piece_id=JMH2XIRXCBLL6BY7NL2KI7MZXWPEDC6MAPOPORWZXB55XJJAPXHA, seg_pos={0 0}, upload=503016]: Uploading piece... Dec 29 13:15:00.997623811 (573h46m0.868706856s) [upload=503016]: Next returned nil segment Dec 29 13:15:00.997646735 (573h46m0.86872979s) [upload=503016]: Waiting for error group managing segments... Dec 29 13:15:01.322861973 (573h46m1.193945028s) [noise=true , address=145.79.227.38:4979 , piece_id=NK5USLTPICTEVKDWIBAL735G73RE2HBDGOTULXFG2NQRQDYKBVWA, seg_pos={0 0}, upload=503016]: Done uploading piece. err: <nil> Dec 29 13:15:01.323060331 (573h46m1.194143376s) [noise=true , address=145.79.227.13:4802 , piece_id=AZ7RK2VDHZYDYFRWHNHW6FMZJJZ3XRNOHN62EGVDNGBKBZ7J6SDQ, seg_pos={0 0}, upload=503016]: Done uploading piece. err: <nil> Dec 29 13:15:01.329368823 (573h46m1.200451868s) [noise=true , address=145.79.227.18:4007 , piece_id=7E3ITPKMZPM4Y5FHVCGOEGHERB34CMAL2ES3TZ6SB3S3CHETRNBA, seg_pos={0 0}, upload=503016]: Done uploading piece. err: <nil> Dec 29 13:15:01.330298153 (573h46m1.201381188s) [noise=true , address=145.79.227.32:4834 , piece_id=V5TUJNYHCLJXQVEVOTZDXIXJRFEQO3CPOTUU5KDHPNPFSOLURF4Q, seg_pos={0 0}, upload=503016]: Done uploading piece. err: <nil> Dec 29 13:15:01.331235036 (573h46m1.202318091s) [noise=true , address=145.79.227.19:4077 , piece_id=BI7V27VENV4XRS4RYUGNHUEDWCV4UW2ZHBG367MJXHRUEPYOL5MA, seg_pos={0 0}, upload=503016]: Done uploading piece. err: <nil> Dec 29 13:15:01.333910471 (573h46m1.204993516s) [noise=true , address=145.79.227.52:4445 , piece_id=R2PDEEEOJOVC4YNYKCPI7YI6B6M6YTCDC43ZF3MCY67B47X23AGQ, seg_pos={0 0}, upload=503016]: Done uploading piece. err: <nil> Dec 29 13:15:01.334003288 (573h46m1.205086343s) [noise=true , address=145.79.227.12:4765 , piece_id=UPBPU2QVDRTTM5SWJMYIDL2W35GZFXAKEGFPY2YIDI7BQ55ARG6Q, seg_pos={0 0}, upload=503016]: Done uploading piece. err: <nil> Dec 29 13:15:01.344178402 (573h46m1.215261448s) [noise=true , address=145.79.227.16:4912 , piece_id=JMH2XIRXCBLL6BY7NL2KI7MZXWPEDC6MAPOPORWZXB55XJJAPXHA, seg_pos={0 0}, upload=503016]: Done uploading piece. err: <nil> Dec 29 13:15:01.347340464 (573h46m1.218423509s) [noise=true , address=145.79.227.11:4732 , piece_id=IG6W4GXSF2MINKPLWLTSFDWYIR7HMUODH26H6OHY2GTK6AYDV5NQ, seg_pos={0 0}, upload=503016]: Done uploading piece. err: <nil> Dec 29 13:15:01.347892945 (573h46m1.21897599s) [noise=true , address=145.79.227.2:4114 , piece_id=NGIVG57VGFEFYWCRKPDNI42UFERFVBFZK6JOWPTUQNROMW2XUB2A, seg_pos={0 0}, upload=503016]: Done uploading piece. err: <nil> Dec 29 13:15:01.349781852 (573h46m1.220864898s) [noise=true , address=145.79.227.42:4122 , piece_id=2ZJBOZCTH3AYIPS7KOGGKQKNS436D3UNB55JSCDZ5PYC3RHHTGLQ, seg_pos={0 0}, upload=503016]: Done uploading piece. err: <nil> Dec 29 13:15:01.572830599 (573h46m1.443913644s) [noise=true , address=45.91.116.10:4084 , piece_id=4TWHAVGGSGRKTKDQTEIQVC3VXOOFYYI3LKEGFNIWML4U2UCSHBDA, seg_pos={0 0}, upload=503016]: Done uploading piece. err: <nil> Dec 29 13:15:01.640741902 (573h46m1.511824957s) [noise=true , address=77.90.31.11:4556 , piece_id=32T4H5B3CRKUA25W7EOTIUEERZ3D6LSOJ5GASLULAVDKLEQEXYTQ, seg_pos={0 0}, upload=503016]: Done uploading piece. err: <nil> Dec 29 13:15:01.640778241 (573h46m1.511861287s) [seg_pos={0 0}, upload=503016]: Segment reached optimal threshold of 13 pieces. Dec 29 13:15:01.641000604 (573h46m1.51208365s) [noise=true , address=45.91.116.11:4120 , piece_id=VKMMVQVYAGLKQGNVPH6G7G62HDMNIY3KDDHCDBDBY6MG4YFDJPWQ, seg_pos={0 0}, upload=503016]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 29 13:15:01.641007858 (573h46m1.512090903s) [noise=true , address=45.91.116.11:4120 , piece_id=VKMMVQVYAGLKQGNVPH6G7G62HDMNIY3KDDHCDBDBY6MG4YFDJPWQ, seg_pos={0 0}, upload=503016]: Long tail cancellation due to optimal threshold Dec 29 13:15:01.641330322 (573h46m1.512413367s) [noise=true , address=77.90.31.12:4589 , piece_id=KFFXI3ZILAJTJKSUZ3TLOLC2JXYWWFG7IYBUUQSNPQOFUIC74PSA, seg_pos={0 0}, upload=503016]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 29 13:15:01.641339240 (573h46m1.512422285s) [noise=true , address=77.90.31.12:4589 , piece_id=KFFXI3ZILAJTJKSUZ3TLOLC2JXYWWFG7IYBUUQSNPQOFUIC74PSA, seg_pos={0 0}, upload=503016]: Long tail cancellation due to optimal threshold Dec 29 13:15:01.641560470 (573h46m1.512643516s) [noise=true , address=77.90.31.16:4736 , piece_id=UV57UQIOHTCZJZTZMSM622PGJOPNBTE7L7MNOOCX355GHS2W6I2Q, seg_pos={0 0}, upload=503016]: Done uploading piece. err: <nil> Dec 29 13:15:01.641636635 (573h46m1.51271968s) [seg_pos={0 0}, upload=503016]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Dec 29 13:15:01.641752587 (573h46m1.512835632s) [upload=503016]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 13:15:01.641759810 (573h46m1.512842855s) [upload=503016]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 13:15:01.938778637 (573h46m1.809861682s) [upload=503016]: Done uploading segments... Dec 29 13:15:01.938806460 (573h46m1.809889515s) [upload=503016]: Upload finished. err: <nil> Duration: 5.707496992s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 51 out of 60 is failed (threshold=6) [XX.XXXXXXXXXXXXXXX.XXXXXXX.X.XX.XXX.XX.XXXXXXX.XX.XXXXXXXXXX] duration: 5.001688388s trace: https://jaeger.dev.storj.io/trace/3844b3a2f2557363 FAIL (2025-12-29 13:15:34 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001744805s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 31 out of 60 is failed (threshold=3) [X....XXXX..XX..X...XX.X.XX...X.X.X.XXX.X.X.X.XX..X.XXX.XX..X] duration: 30.004092351s trace: https://jaeger.dev.storj.io/trace/4e9cc163fa850803 FAIL (2025-12-29 13:15:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: 4TWHAVGGSGRKTKDQTEIQVC3VXOOFYYI3LKEGFNIWML4U2UCSHBDA) context deadline exceeded; piecestore: (Node ID: 1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, Piece ID: 32T4H5B3CRKUA25W7EOTIUEERZ3D6LSOJ5GASLULAVDKLEQEXYTQ) context deadline exceeded ======LOGS====== Dec 29 13:14:59.052691585 (573h45m58.92377462s) [upload=503016]: Starting upload Dec 29 13:14:59.052744516 (573h45m58.923827551s) [upload=503016]: Done starting upload Dec 29 13:14:59.052915451 (573h45m58.923998496s) [upload=503016]: Uploading segments... Dec 29 13:14:59.052954847 (573h45m58.924037882s) [upload=503016]: Got next segment. Inline: false Dec 29 13:14:59.053000473 (573h45m58.924083508s) [upload=503016]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 13:14:59.053007426 (573h45m58.924090461s) [upload=503016]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 13:14:59.334293867 (573h45m59.205376912s) [seg_pos={0 0}, upload=503016]: Begin upload segment... Dec 29 13:14:59.334347518 (573h45m59.205430563s) [seg_pos={0 0}, upload=503016]: Done begin upload segment. Dec 29 13:14:59.334378097 (573h45m59.205461132s) [noise=true , address=145.79.227.18:4007 , piece_id=7E3ITPKMZPM4Y5FHVCGOEGHERB34CMAL2ES3TZ6SB3S3CHETRNBA, seg_pos={0 0}, upload=503016]: Uploading piece... Dec 29 13:14:59.334400440 (573h45m59.205483485s) [noise=true , address=145.79.227.2:4114 , piece_id=NGIVG57VGFEFYWCRKPDNI42UFERFVBFZK6JOWPTUQNROMW2XUB2A, seg_pos={0 0}, upload=503016]: Uploading piece... Dec 29 13:14:59.335194562 (573h45m59.206277597s) [noise=true , address=145.79.227.52:4445 , piece_id=R2PDEEEOJOVC4YNYKCPI7YI6B6M6YTCDC43ZF3MCY67B47X23AGQ, seg_pos={0 0}, upload=503016]: Uploading piece... Dec 29 13:14:59.335784075 (573h45m59.20686712s) [noise=true , address=145.79.227.42:4122 , piece_id=2ZJBOZCTH3AYIPS7KOGGKQKNS436D3UNB55JSCDZ5PYC3RHHTGLQ, seg_pos={0 0}, upload=503016]: Uploading piece... Dec 29 13:14:59.336178325 (573h45m59.20726137s) [noise=true , address=45.91.116.10:4084 , piece_id=4TWHAVGGSGRKTKDQTEIQVC3VXOOFYYI3LKEGFNIWML4U2UCSHBDA, seg_pos={0 0}, upload=503016]: Uploading piece... Dec 29 13:14:59.336601731 (573h45m59.207684776s) [noise=true , address=145.79.227.12:4765 , piece_id=UPBPU2QVDRTTM5SWJMYIDL2W35GZFXAKEGFPY2YIDI7BQ55ARG6Q, seg_pos={0 0}, upload=503016]: Uploading piece... Dec 29 13:14:59.337009347 (573h45m59.208092383s) [noise=true , address=145.79.227.11:4732 , piece_id=IG6W4GXSF2MINKPLWLTSFDWYIR7HMUODH26H6OHY2GTK6AYDV5NQ, seg_pos={0 0}, upload=503016]: Uploading piece... Dec 29 13:14:59.337451069 (573h45m59.208534114s) [noise=true , address=77.90.31.16:4736 , piece_id=UV57UQIOHTCZJZTZMSM622PGJOPNBTE7L7MNOOCX355GHS2W6I2Q, seg_pos={0 0}, upload=503016]: Uploading piece... Dec 29 13:14:59.337986047 (573h45m59.209069082s) [noise=true , address=77.90.31.12:4589 , piece_id=KFFXI3ZILAJTJKSUZ3TLOLC2JXYWWFG7IYBUUQSNPQOFUIC74PSA, seg_pos={0 0}, upload=503016]: Uploading piece... Dec 29 13:14:59.338221566 (573h45m59.209304611s) [noise=true , address=145.79.227.13:4802 , piece_id=AZ7RK2VDHZYDYFRWHNHW6FMZJJZ3XRNOHN62EGVDNGBKBZ7J6SDQ, seg_pos={0 0}, upload=503016]: Uploading piece... Dec 29 13:14:59.338678055 (573h45m59.2097611s) [noise=true , address=145.79.227.19:4077 , piece_id=BI7V27VENV4XRS4RYUGNHUEDWCV4UW2ZHBG367MJXHRUEPYOL5MA, seg_pos={0 0}, upload=503016]: Uploading piece... Dec 29 13:14:59.338856685 (573h45m59.20993973s) [noise=true , address=45.91.116.11:4120 , piece_id=VKMMVQVYAGLKQGNVPH6G7G62HDMNIY3KDDHCDBDBY6MG4YFDJPWQ, seg_pos={0 0}, upload=503016]: Uploading piece... Dec 29 13:14:59.339166555 (573h45m59.21024959s) [noise=true , address=145.79.227.32:4834 , piece_id=V5TUJNYHCLJXQVEVOTZDXIXJRFEQO3CPOTUU5KDHPNPFSOLURF4Q, seg_pos={0 0}, upload=503016]: Uploading piece... Dec 29 13:14:59.339443873 (573h45m59.210526918s) [noise=true , address=145.79.227.38:4979 , piece_id=NK5USLTPICTEVKDWIBAL735G73RE2HBDGOTULXFG2NQRQDYKBVWA, seg_pos={0 0}, upload=503016]: Uploading piece... Dec 29 13:14:59.340070727 (573h45m59.211153782s) [noise=true , address=77.90.31.11:4556 , piece_id=32T4H5B3CRKUA25W7EOTIUEERZ3D6LSOJ5GASLULAVDKLEQEXYTQ, seg_pos={0 0}, upload=503016]: Uploading piece... Dec 29 13:14:59.340738428 (573h45m59.211821463s) [noise=true , address=145.79.227.16:4912 , piece_id=JMH2XIRXCBLL6BY7NL2KI7MZXWPEDC6MAPOPORWZXB55XJJAPXHA, seg_pos={0 0}, upload=503016]: Uploading piece... Dec 29 13:15:00.997623811 (573h46m0.868706856s) [upload=503016]: Next returned nil segment Dec 29 13:15:00.997646735 (573h46m0.86872979s) [upload=503016]: Waiting for error group managing segments... Dec 29 13:15:01.322861973 (573h46m1.193945028s) [noise=true , address=145.79.227.38:4979 , piece_id=NK5USLTPICTEVKDWIBAL735G73RE2HBDGOTULXFG2NQRQDYKBVWA, seg_pos={0 0}, upload=503016]: Done uploading piece. err: <nil> Dec 29 13:15:01.323060331 (573h46m1.194143376s) [noise=true , address=145.79.227.13:4802 , piece_id=AZ7RK2VDHZYDYFRWHNHW6FMZJJZ3XRNOHN62EGVDNGBKBZ7J6SDQ, seg_pos={0 0}, upload=503016]: Done uploading piece. err: <nil> Dec 29 13:15:01.329368823 (573h46m1.200451868s) [noise=true , address=145.79.227.18:4007 , piece_id=7E3ITPKMZPM4Y5FHVCGOEGHERB34CMAL2ES3TZ6SB3S3CHETRNBA, seg_pos={0 0}, upload=503016]: Done uploading piece. err: <nil> Dec 29 13:15:01.330298153 (573h46m1.201381188s) [noise=true , address=145.79.227.32:4834 , piece_id=V5TUJNYHCLJXQVEVOTZDXIXJRFEQO3CPOTUU5KDHPNPFSOLURF4Q, seg_pos={0 0}, upload=503016]: Done uploading piece. err: <nil> Dec 29 13:15:01.331235036 (573h46m1.202318091s) [noise=true , address=145.79.227.19:4077 , piece_id=BI7V27VENV4XRS4RYUGNHUEDWCV4UW2ZHBG367MJXHRUEPYOL5MA, seg_pos={0 0}, upload=503016]: Done uploading piece. err: <nil> Dec 29 13:15:01.333910471 (573h46m1.204993516s) [noise=true , address=145.79.227.52:4445 , piece_id=R2PDEEEOJOVC4YNYKCPI7YI6B6M6YTCDC43ZF3MCY67B47X23AGQ, seg_pos={0 0}, upload=503016]: Done uploading piece. err: <nil> Dec 29 13:15:01.334003288 (573h46m1.205086343s) [noise=true , address=145.79.227.12:4765 , piece_id=UPBPU2QVDRTTM5SWJMYIDL2W35GZFXAKEGFPY2YIDI7BQ55ARG6Q, seg_pos={0 0}, upload=503016]: Done uploading piece. err: <nil> Dec 29 13:15:01.344178402 (573h46m1.215261448s) [noise=true , address=145.79.227.16:4912 , piece_id=JMH2XIRXCBLL6BY7NL2KI7MZXWPEDC6MAPOPORWZXB55XJJAPXHA, seg_pos={0 0}, upload=503016]: Done uploading piece. err: <nil> Dec 29 13:15:01.347340464 (573h46m1.218423509s) [noise=true , address=145.79.227.11:4732 , piece_id=IG6W4GXSF2MINKPLWLTSFDWYIR7HMUODH26H6OHY2GTK6AYDV5NQ, seg_pos={0 0}, upload=503016]: Done uploading piece. err: <nil> Dec 29 13:15:01.347892945 (573h46m1.21897599s) [noise=true , address=145.79.227.2:4114 , piece_id=NGIVG57VGFEFYWCRKPDNI42UFERFVBFZK6JOWPTUQNROMW2XUB2A, seg_pos={0 0}, upload=503016]: Done uploading piece. err: <nil> Dec 29 13:15:01.349781852 (573h46m1.220864898s) [noise=true , address=145.79.227.42:4122 , piece_id=2ZJBOZCTH3AYIPS7KOGGKQKNS436D3UNB55JSCDZ5PYC3RHHTGLQ, seg_pos={0 0}, upload=503016]: Done uploading piece. err: <nil> Dec 29 13:15:01.572830599 (573h46m1.443913644s) [noise=true , address=45.91.116.10:4084 , piece_id=4TWHAVGGSGRKTKDQTEIQVC3VXOOFYYI3LKEGFNIWML4U2UCSHBDA, seg_pos={0 0}, upload=503016]: Done uploading piece. err: <nil> Dec 29 13:15:01.640741902 (573h46m1.511824957s) [noise=true , address=77.90.31.11:4556 , piece_id=32T4H5B3CRKUA25W7EOTIUEERZ3D6LSOJ5GASLULAVDKLEQEXYTQ, seg_pos={0 0}, upload=503016]: Done uploading piece. err: <nil> Dec 29 13:15:01.640778241 (573h46m1.511861287s) [seg_pos={0 0}, upload=503016]: Segment reached optimal threshold of 13 pieces. Dec 29 13:15:01.641000604 (573h46m1.51208365s) [noise=true , address=45.91.116.11:4120 , piece_id=VKMMVQVYAGLKQGNVPH6G7G62HDMNIY3KDDHCDBDBY6MG4YFDJPWQ, seg_pos={0 0}, upload=503016]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 29 13:15:01.641007858 (573h46m1.512090903s) [noise=true , address=45.91.116.11:4120 , piece_id=VKMMVQVYAGLKQGNVPH6G7G62HDMNIY3KDDHCDBDBY6MG4YFDJPWQ, seg_pos={0 0}, upload=503016]: Long tail cancellation due to optimal threshold Dec 29 13:15:01.641330322 (573h46m1.512413367s) [noise=true , address=77.90.31.12:4589 , piece_id=KFFXI3ZILAJTJKSUZ3TLOLC2JXYWWFG7IYBUUQSNPQOFUIC74PSA, seg_pos={0 0}, upload=503016]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 29 13:15:01.641339240 (573h46m1.512422285s) [noise=true , address=77.90.31.12:4589 , piece_id=KFFXI3ZILAJTJKSUZ3TLOLC2JXYWWFG7IYBUUQSNPQOFUIC74PSA, seg_pos={0 0}, upload=503016]: Long tail cancellation due to optimal threshold Dec 29 13:15:01.641560470 (573h46m1.512643516s) [noise=true , address=77.90.31.16:4736 , piece_id=UV57UQIOHTCZJZTZMSM622PGJOPNBTE7L7MNOOCX355GHS2W6I2Q, seg_pos={0 0}, upload=503016]: Done uploading piece. err: <nil> Dec 29 13:15:01.641636635 (573h46m1.51271968s) [seg_pos={0 0}, upload=503016]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Dec 29 13:15:01.641752587 (573h46m1.512835632s) [upload=503016]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 13:15:01.641759810 (573h46m1.512842855s) [upload=503016]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 13:15:01.938778637 (573h46m1.809861682s) [upload=503016]: Done uploading segments... Dec 29 13:15:01.938806460 (573h46m1.809889515s) [upload=503016]: Upload finished. err: <nil> Duration: 30.004117038s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [....................X......................X....X.X....X.X..] duration: 14.371367612s trace: https://jaeger.dev.storj.io/trace/312ddb07b2dee82a FAIL (2025-12-29 13:15:15 UTC) orbiter warning: context deadline exceeded Duration: 14.371400396s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 51 out of 60 is failed (threshold=6) [XXX.XXXXXXXXXXXXXXX.XXXXXXX.X.XX.XXX.XX.XXXXXXX.XX.XXXXXXXXX] duration: 5.002713205s trace: https://jaeger.dev.storj.io/trace/2f8a4abdce56b1b5 FAIL (2025-12-29 13:15:05 UTC) orbiter warning: context deadline exceeded Duration: 5.002738764s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [....X......................X.....X.....................XX...] duration: 48.53116564s trace: https://jaeger.dev.storj.io/trace/77d26141ea11d228 FAIL (2025-12-29 13:15:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12J2weU7Bmsr3RdQQ9QMuadT96XReBkyxTeaLJ2SEqVGGFX1PWu, Piece ID: JITX25FNZ4JIKYXQY345UX3SQUCPDWJ25NJQZ2TXGFRLVCRYXUOQ) context deadline exceeded; piecestore: (Node ID: 125sm6iHV94mJFLWrKeyRMH4b1qSMo32nDuW8uQr71xUL7Cema2, Piece ID: INTVU2EYTU3DIM6WQFQEGO6IEYH3YFP5U2VEN7A5KPTOKU2DWOOA) context deadline exceeded; piecestore: (Node ID: 12i57FF97oEddYLZpNxSq5or9WLEquL8ZW1cVUuW4AThBbDqcCL, Piece ID: Q35LXGPTOAG3R2UL3JLL7DNHHD4PSMEULYW6G3SPXKNVFZ5VTNRA) context deadline exceeded ======LOGS====== Q2LO7MUXLHE5YET7BHLLA, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.692912334 (573h43m3.563995379s) [noise=true , address=77.90.31.16:4759 , piece_id=ARC4HRP57U5VQXZTT63QYLXBPZURI5VMJBIQZOFKGLJBQ3K3KGLQ, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.693322595 (573h43m3.56440564s) [noise=true , address=77.90.31.15:4711 , piece_id=JITX25FNZ4JIKYXQY345UX3SQUCPDWJ25NJQZ2TXGFRLVCRYXUOQ, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.693739649 (573h43m3.564822694s) [noise=true , address=45.91.116.6:4981 , piece_id=INTVU2EYTU3DIM6WQFQEGO6IEYH3YFP5U2VEN7A5KPTOKU2DWOOA, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.694315887 (573h43m3.565398942s) [noise=true , address=145.79.227.4:4220 , piece_id=SHHJ7EW7DTNOZXVBP6A64VIX7P6AQN62TBCUT3KTH4MIB6UXY3BA, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.694765222 (573h43m3.565848267s) [noise=true , address=145.79.227.29:4952 , piece_id=DKYDB2R4DNCGH6T3SLRR2MOZZUPVZZVRL2BKGYOBYXJJVPSZEJ4Q, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.695202545 (573h43m3.56628559s) [noise=true , address=145.79.227.40:4058 , piece_id=KRHH2QZKSDPWTSSKYDVDWGKFJD3LPGP6U7KI2PPJ5T3TXYIUGJWA, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.695687528 (573h43m3.566770584s) [noise=true , address=77.90.31.19:4857 , piece_id=Q35LXGPTOAG3R2UL3JLL7DNHHD4PSMEULYW6G3SPXKNVFZ5VTNRA, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.696135221 (573h43m3.567218276s) [noise=true , address=45.91.116.8:4051 , piece_id=GDJFLQCHUF7BFQ7BVSNTVJ3TYZHYVOO6YYMO3KUIUSIEPAF4SFDQ, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.696703733 (573h43m3.567786778s) [noise=true , address=64.49.13.170:4720 , piece_id=HF6X3OT2EPPDE3DWAPWABVZUR3ERDBWXHG3NGNKEWZAD7G5AN4TA, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.698418298 (573h43m3.569501343s) [noise=true , address=145.79.227.36:4520 , piece_id=MYJSXUH563JFUW5Q7YUIWL7466G6ASY5WXR3T2PGUFFCTRTOBYJA, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.700198829 (573h43m3.571281874s) [noise=true , address=77.90.31.18:4821 , piece_id=LO5W47FHYOVGY5Q6BH5L6KX77HES3XRAFUTPI45HYPJZDO3YHLCQ, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.701993145 (573h43m3.57307619s) [noise=true , address=193.149.164.143:4699 , piece_id=UDFDCYO25MOJBTMDE6F2WILXE3HLOB75UJZ4LW7ZMNSCOBJI6UJA, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.703884687 (573h43m3.574967732s) [noise=true , address=145.79.227.33:4893 , piece_id=XGLX3DFNAY4F4GPGCGZ4ONDJ7NK62SAKX3PBRWHWI74EF3ZFMUSA, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.705687109 (573h43m3.576770154s) [noise=true , address=45.91.116.10:4094 , piece_id=6UUIZNTESBBWJMDQ5ZTCHU7S6YKM4GDKO4ZAC6T2FUTPVGFFDS3A, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:03.707557651 (573h43m3.578640696s) [noise=true , address=145.79.227.43:4164 , piece_id=J6Q4ZDOIIIBA2FXGZZTD6AYS7KGAWSRXXUBAPDPQWYKNU5GNXBBQ, seg_pos={0 0}, upload=502975]: Uploading piece... Dec 29 13:12:05.168247719 (573h43m5.039330754s) [upload=502975]: Next returned nil segment Dec 29 13:12:05.168259962 (573h43m5.039343007s) [upload=502975]: Waiting for error group managing segments... Dec 29 13:12:05.558873127 (573h43m5.429956162s) [noise=true , address=145.79.227.4:4220 , piece_id=SHHJ7EW7DTNOZXVBP6A64VIX7P6AQN62TBCUT3KTH4MIB6UXY3BA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:05.563024312 (573h43m5.434107347s) [noise=true , address=64.49.13.170:4720 , piece_id=HF6X3OT2EPPDE3DWAPWABVZUR3ERDBWXHG3NGNKEWZAD7G5AN4TA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:05.564115719 (573h43m5.435198755s) [noise=true , address=145.79.227.36:4520 , piece_id=MYJSXUH563JFUW5Q7YUIWL7466G6ASY5WXR3T2PGUFFCTRTOBYJA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:05.593873402 (573h43m5.464956447s) [noise=true , address=145.79.227.29:4952 , piece_id=DKYDB2R4DNCGH6T3SLRR2MOZZUPVZZVRL2BKGYOBYXJJVPSZEJ4Q, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:05.898194604 (573h43m5.769277639s) [noise=true , address=45.91.116.10:4094 , piece_id=6UUIZNTESBBWJMDQ5ZTCHU7S6YKM4GDKO4ZAC6T2FUTPVGFFDS3A, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:05.912845292 (573h43m5.783928337s) [noise=true , address=77.90.31.19:4857 , piece_id=Q35LXGPTOAG3R2UL3JLL7DNHHD4PSMEULYW6G3SPXKNVFZ5VTNRA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:06.426532207 (573h43m6.297615252s) [noise=true , address=193.149.164.143:4699 , piece_id=UDFDCYO25MOJBTMDE6F2WILXE3HLOB75UJZ4LW7ZMNSCOBJI6UJA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:06.449589562 (573h43m6.320672607s) [noise=true , address=77.90.31.15:4711 , piece_id=JITX25FNZ4JIKYXQY345UX3SQUCPDWJ25NJQZ2TXGFRLVCRYXUOQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:06.453186841 (573h43m6.324269876s) [noise=true , address=145.79.227.40:4058 , piece_id=KRHH2QZKSDPWTSSKYDVDWGKFJD3LPGP6U7KI2PPJ5T3TXYIUGJWA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:06.510769276 (573h43m6.381852321s) [noise=true , address=145.79.227.43:4164 , piece_id=J6Q4ZDOIIIBA2FXGZZTD6AYS7KGAWSRXXUBAPDPQWYKNU5GNXBBQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:06.518868879 (573h43m6.389951924s) [noise=true , address=145.79.227.33:4893 , piece_id=XGLX3DFNAY4F4GPGCGZ4ONDJ7NK62SAKX3PBRWHWI74EF3ZFMUSA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:06.519238043 (573h43m6.390321098s) [noise=true , address=45.91.116.6:4981 , piece_id=INTVU2EYTU3DIM6WQFQEGO6IEYH3YFP5U2VEN7A5KPTOKU2DWOOA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:06.523211328 (573h43m6.394294373s) [noise=true , address=77.90.31.16:4759 , piece_id=ARC4HRP57U5VQXZTT63QYLXBPZURI5VMJBIQZOFKGLJBQ3K3KGLQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:06.767869783 (573h43m6.638952818s) [noise=true , address=145.79.227.34:4924 , piece_id=MAQXQTAGXQ7AEPUCESUSUGMHSDMMXS7HHKAEA7B3HW5X66PDUF5A, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil> Dec 29 13:12:06.767889431 (573h43m6.638972476s) [seg_pos={0 0}, upload=502975]: Segment reached optimal threshold of 14 pieces. Dec 29 13:12:06.768102116 (573h43m6.639185161s) [noise=true , address=45.91.116.8:4051 , piece_id=GDJFLQCHUF7BFQ7BVSNTVJ3TYZHYVOO6YYMO3KUIUSIEPAF4SFDQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1preT33ei4yVXXYVLHhaUbHbiyL14oHFEiof6cCWPig83W6CT3): protocol: expected piece hash; context canceled; EOF Dec 29 13:12:06.768108257 (573h43m6.639191302s) [noise=true , address=45.91.116.8:4051 , piece_id=GDJFLQCHUF7BFQ7BVSNTVJ3TYZHYVOO6YYMO3KUIUSIEPAF4SFDQ, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold Dec 29 13:12:06.768219208 (573h43m6.639302254s) [noise=true , address=77.90.31.18:4821 , piece_id=LO5W47FHYOVGY5Q6BH5L6KX77HES3XRAFUTPI45HYPJZDO3YHLCQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH): protocol: expected piece hash; context canceled; EOF Dec 29 13:12:06.768224789 (573h43m6.639307824s) [noise=true , address=77.90.31.18:4821 , piece_id=LO5W47FHYOVGY5Q6BH5L6KX77HES3XRAFUTPI45HYPJZDO3YHLCQ, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold Dec 29 13:12:06.768384213 (573h43m6.639467258s) [noise=true , address=64.49.13.169:4711 , piece_id=4F4T4YKXMT7NF2C5KBUGJHQJGZL4VCKNKYB266PCVI3UACHJXZ5Q, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:48802->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:48802->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48802->64.49.13.169:4711: use of closed network connection Dec 29 13:12:06.768399261 (573h43m6.639482296s) [noise=true , address=64.49.13.169:4711 , piece_id=4F4T4YKXMT7NF2C5KBUGJHQJGZL4VCKNKYB266PCVI3UACHJXZ5Q, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold Dec 29 13:12:06.768732636 (573h43m6.639815671s) [noise=true , address=64.49.13.169:4709 , piece_id=TBHHX5ES2A6NQRGP25HIC7J7IBWLOJGQ2LO7MUXLHE5YET7BHLLA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:35310->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:35310->64.49.13.169:4709: use of closed network connection Dec 29 13:12:06.768745460 (573h43m6.639828505s) [noise=true , address=64.49.13.169:4709 , piece_id=TBHHX5ES2A6NQRGP25HIC7J7IBWLOJGQ2LO7MUXLHE5YET7BHLLA, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold Dec 29 13:12:06.768784586 (573h43m6.639867631s) [seg_pos={0 0}, upload=502975]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 13:12:06.768880217 (573h43m6.639963252s) [upload=502975]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 13:12:06.768884906 (573h43m6.639967941s) [upload=502975]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 13:12:07.067188330 (573h43m6.938271365s) [upload=502975]: Done uploading segments... Dec 29 13:12:07.067251099 (573h43m6.938334134s) [upload=502975]: Upload finished. err: <nil> Duration: 48.531200095s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 51 out of 60 is failed (threshold=6) [XXXX.XXXXXXXXXXXXXXX.XXXXXXX.X.XX.XXX.XX.XXXXXXX.XX.XXXXXXXX] duration: 5.001349691s trace: https://jaeger.dev.storj.io/trace/2981da047f45a9c0 FAIL (2025-12-29 13:14:42 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001368786s core uplink us-select-1 us1-placement-20

1
...
11286
11287
11288
11289
11290
...
18071