Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................................X..............X....] duration: 27.184128208s trace: https://jaeger.dev.storj.io/trace/5bdd2de9914a125e FAIL (2025-12-26 20:32:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh, Piece ID: L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q) context deadline exceeded; piecestore: (Node ID: 127RBQsp1SwQU6yWh6Y48QF3QCzeJxEGjob8gzF3LRutRZXA6A9, Piece ID: XUHE2A4VE3U65LONDRNKMJILERSNLNDTJ7R2UPX5D67UI426UQAA) context deadline exceeded; piecestore: (Node ID: 12NFW5tt5R84RaTvY9Up75kNjZ7bmLAgUun41KLdW9Q7eMv27Sq, Piece ID: Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA) context deadline exceeded ======LOGS====== , piece_id=M4GW7XWWHESKY7CULLFIM7THSWLI7Z2FVWZL4YDCPNEXRJNU5TFA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.621775106 (509h0m8.492858151s) [noise=true , address=145.79.227.22:4533 , piece_id=WVQ7YABIOYCCHCMZIRFXIUDKLFFC2KC5WJOABJC2F57QLRJK4H3A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.621846552 (509h0m8.492929597s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.622555641 (509h0m8.493638687s) [noise=true , address=145.79.227.43:4164 , piece_id=XCWV4H42H7J4PQAYNAEODGLSOXULESDU3JW6TJFDWXGIDFBRDGSA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.622948811 (509h0m8.494031846s) [noise=true , address=145.79.227.16:4922 , piece_id=N5JA6JVCAXPKDGIURCNVAWJD6KRTGNRY6S3IZMD5EYVFA454GOKA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623044693 (509h0m8.494127728s) [noise=true , address=77.90.31.11:4570 , piece_id=Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623366476 (509h0m8.494449521s) [noise=true , address=45.91.116.13:4165 , piece_id=YIRLJS6I6YR7WJRG5RLODG5HGZQJQW4V4PKCRIF2BXMWXAFRXTCA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623672749 (509h0m8.494755784s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623792828 (509h0m8.494875863s) [noise=true , address=145.79.227.28:4737 , piece_id=VI474SQ2NRST5NR4YCNYHUNBYBIJSEDCSHQOM6PEF7NEK3NXNTNA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624190374 (509h0m8.495273419s) [noise=true , address=145.79.227.14:4868 , piece_id=3Q4CBS6OWB36IHRUBCRLUZIJQAS76QU5O2YKKQC6XZ5E7FBX4PGA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624328067 (509h0m8.495411112s) [noise=true , address=77.90.31.14:4673 , piece_id=L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624580047 (509h0m8.495663092s) [noise=true , address=145.79.227.30:4784 , piece_id=TIVKXCMDHBQZ5LRKW2MQ3E53LWHQQCYPAZ27NYSRXJWGVEZCXJ3A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624915285 (509h0m8.49599833s) [noise=true , address=145.79.227.34:4930 , piece_id=2BWAT4EMK4QB54A5DRYBSPEGAVDVJPELOLA6ZDROZA4YCK37RJXA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.625529695 (509h0m8.49661276s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.626165055 (509h0m8.49724809s) [noise=true , address=77.90.31.21:4890 , piece_id=NIGMBPZC3J6POFXVSNWZZEJUVKUQ77B7J7IENDZ2C2W3WMZAOD5A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.626720573 (509h0m8.497803618s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:09.940468211 (509h0m9.811551256s) [upload=446649]: Next returned nil segment Dec 26 20:29:09.940490244 (509h0m9.811573289s) [upload=446649]: Waiting for error group managing segments... Dec 26 20:29:10.283357488 (509h0m10.154440524s) [noise=true , address=64.49.13.139:4477 , piece_id=5QMCZNADOLXNYEBNY7EPOMHRPK5EANDLPEUJLU6SQ2QJQCRF66BQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.503143617 (509h0m10.374226653s) [noise=true , address=145.79.227.28:4737 , piece_id=VI474SQ2NRST5NR4YCNYHUNBYBIJSEDCSHQOM6PEF7NEK3NXNTNA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.657417840 (509h0m10.528500875s) [noise=true , address=145.79.227.16:4922 , piece_id=N5JA6JVCAXPKDGIURCNVAWJD6KRTGNRY6S3IZMD5EYVFA454GOKA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.679733744 (509h0m10.550816789s) [noise=true , address=77.90.31.21:4890 , piece_id=NIGMBPZC3J6POFXVSNWZZEJUVKUQ77B7J7IENDZ2C2W3WMZAOD5A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.683939000 (509h0m10.555022046s) [noise=true , address=77.90.31.14:4673 , piece_id=L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.685467862 (509h0m10.556550907s) [noise=true , address=145.79.227.22:4533 , piece_id=WVQ7YABIOYCCHCMZIRFXIUDKLFFC2KC5WJOABJC2F57QLRJK4H3A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.772110194 (509h0m10.643193229s) [noise=true , address=45.91.116.13:4165 , piece_id=YIRLJS6I6YR7WJRG5RLODG5HGZQJQW4V4PKCRIF2BXMWXAFRXTCA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.820042584 (509h0m10.691125629s) [noise=true , address=77.90.31.11:4570 , piece_id=Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.866890631 (509h0m10.737973666s) [noise=true , address=45.91.116.5:4314 , piece_id=XUHE2A4VE3U65LONDRNKMJILERSNLNDTJ7R2UPX5D67UI426UQAA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.999531381 (509h0m10.870614416s) [noise=true , address=145.79.227.43:4164 , piece_id=XCWV4H42H7J4PQAYNAEODGLSOXULESDU3JW6TJFDWXGIDFBRDGSA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.160014888 (509h0m11.031097933s) [noise=true , address=145.79.227.14:4868 , piece_id=3Q4CBS6OWB36IHRUBCRLUZIJQAS76QU5O2YKKQC6XZ5E7FBX4PGA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.463630931 (509h0m11.334713977s) [noise=true , address=145.79.227.34:4930 , piece_id=2BWAT4EMK4QB54A5DRYBSPEGAVDVJPELOLA6ZDROZA4YCK37RJXA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.508749725 (509h0m11.37983276s) [noise=true , address=145.79.227.40:4058 , piece_id=M4GW7XWWHESKY7CULLFIM7THSWLI7Z2FVWZL4YDCPNEXRJNU5TFA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.661052545 (509h0m11.53213558s) [noise=true , address=145.79.227.30:4784 , piece_id=TIVKXCMDHBQZ5LRKW2MQ3E53LWHQQCYPAZ27NYSRXJWGVEZCXJ3A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.661070500 (509h0m11.532153535s) [seg_pos={0 0}, upload=446649]: Segment reached optimal threshold of 14 pieces. Dec 26 20:29:11.661242827 (509h0m11.532325872s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji): protocol: expected piece hash; context canceled; EOF Dec 26 20:29:11.661250402 (509h0m11.532333447s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661295628 (509h0m11.532378663s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt): protocol: write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection; write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection Dec 26 20:29:11.661309314 (509h0m11.532392349s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661349922 (509h0m11.532432957s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1etuVthkhtZexvw7u88vxw2HGPaMwfYCbEgKWF4GPRh9FeWTwn): protocol: expected piece hash; context canceled; EOF Dec 26 20:29:11.661356895 (509h0m11.53243994s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661563188 (509h0m11.532646223s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125rPFZnyTxsa7RBQfAkXtkVCdUJUykq644EK4Ny6HQ1Q7NzrF3): protocol: read tcp 188.166.225.170:36656->45.91.116.6:4982: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 20:29:11.661570592 (509h0m11.532653637s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661611429 (509h0m11.532694474s) [seg_pos={0 0}, upload=446649]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:29:11.661671444 (509h0m11.532754479s) [upload=446649]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:29:11.661676032 (509h0m11.532759077s) [upload=446649]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:29:11.969641419 (509h0m11.840724454s) [upload=446649]: Done uploading segments... Dec 26 20:29:11.969660866 (509h0m11.840743911s) [upload=446649]: Upload finished. err: <nil> Duration: 27.184146904s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [..........XX..X..X...........XXX.X.X....X..XXXX.X...X.X...XX] duration: 30.00249461s trace: https://jaeger.dev.storj.io/trace/27e75959268fd1fb FAIL (2025-12-26 20:32:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: DNN4PR3PPXXD2RCXET56LOETGKQUDMD6CNKZ55KBDBJSJPOJNVLQ) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: GKCV2YUR5Y53HIKQQHQ5DF5NTVZV66FIOXBGQRYBX3RLNTYBC2EQ) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: 7EZOTMSXLYP66G3SENQ3RFOE6E7N3P3NNLIGUXN4BCRUXWFYZUYA) context deadline exceeded ======LOGS====== Dec 26 20:32:01.948942244 (509h3m1.820025289s) [upload=446692]: Starting upload Dec 26 20:32:01.949000535 (509h3m1.82008357s) [upload=446692]: Done starting upload Dec 26 20:32:01.949200505 (509h3m1.820283551s) [upload=446692]: Uploading segments... Dec 26 20:32:01.949404915 (509h3m1.82048796s) [upload=446692]: Got next segment. Inline: false Dec 26 20:32:01.949469889 (509h3m1.820552934s) [upload=446692]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 20:32:01.949476802 (509h3m1.820559847s) [upload=446692]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 20:32:02.197003954 (509h3m2.068086999s) [seg_pos={0 0}, upload=446692]: Begin upload segment... Dec 26 20:32:02.197067095 (509h3m2.06815014s) [seg_pos={0 0}, upload=446692]: Done begin upload segment. Dec 26 20:32:02.197094536 (509h3m2.068177581s) [noise=true , address=45.91.116.5:4299 , piece_id=7EZOTMSXLYP66G3SENQ3RFOE6E7N3P3NNLIGUXN4BCRUXWFYZUYA, seg_pos={0 0}, upload=446692]: Uploading piece... Dec 26 20:32:02.197147898 (509h3m2.068230943s) [noise=true , address=145.79.227.51:4410 , piece_id=3E3IKO5CNT4DG6FNNEHBAZZP4HNNBGLXQAUAQ2ICIYFJQMXY3IAQ, seg_pos={0 0}, upload=446692]: Uploading piece... Dec 26 20:32:02.197605409 (509h3m2.068688454s) [noise=true , address=145.79.227.12:4767 , piece_id=4AWYUKKCLI2LTMPPVYQHI7QH4ZCL77E7IV62JOKXR65D7HURSITQ, seg_pos={0 0}, upload=446692]: Uploading piece... Dec 26 20:32:02.198013797 (509h3m2.069096842s) [noise=true , address=145.79.227.48:4301 , piece_id=G7Y7M3FFO7SQO7ABERSS6YCPNELFKRSK6TD6RXJ6OFSSPADO5BTA, seg_pos={0 0}, upload=446692]: Uploading piece... Dec 26 20:32:02.198553675 (509h3m2.06963672s) [noise=true , address=145.79.227.50:4374 , piece_id=6QE4AJFUNMTP5R35YII7AC4T5CRFXOLJRQOPFR6FBYWXPELHLJGQ, seg_pos={0 0}, upload=446692]: Uploading piece... Dec 26 20:32:02.198957294 (509h3m2.070040339s) [noise=true , address=145.79.227.29:4942 , piece_id=SYUAKGRXIF3G32L4BTYXOLEACBIYZWT3Y7TWFILTKI7VEGCGD26Q, seg_pos={0 0}, upload=446692]: Uploading piece... Dec 26 20:32:02.199323571 (509h3m2.070406616s) [noise=true , address=145.79.227.39:4015 , piece_id=NIAXSR6TS5S7VMQIEAXDEKURUB4QNCRH5OWWJR7GYCHYRUKJWJGA, seg_pos={0 0}, upload=446692]: Uploading piece... Dec 26 20:32:02.199825597 (509h3m2.070908632s) [noise=true , address=145.79.227.40:4049 , piece_id=MXTOJNJDT7P2FT2ALKPHQREOZ4BAIPBFHTPQZNVA7JCYOKR3QZKA, seg_pos={0 0}, upload=446692]: Uploading piece... Dec 26 20:32:02.200342211 (509h3m2.071425256s) [noise=true , address=145.79.227.36:4518 , piece_id=JU4OFTL7PDW425BKPYZS3KRWH7F5IVAQUBZBI5FK5HPUW6WRVWLQ, seg_pos={0 0}, upload=446692]: Uploading piece... Dec 26 20:32:02.201290877 (509h3m2.072373922s) [noise=true , address=145.79.227.49:4338 , piece_id=EOQRQNUOPZVZSSJKGWNPM4UYHLFCVD2CCMP2VNS6JEVVMBCZDJIQ, seg_pos={0 0}, upload=446692]: Uploading piece... Dec 26 20:32:02.201875891 (509h3m2.072958927s) [noise=true , address=145.79.227.11:4732 , piece_id=MTYQCHNCAOH6YKTSQ2Y3DRJ4JV673JHWAAJFP3NWBPMTIUYYUXOA, seg_pos={0 0}, upload=446692]: Uploading piece... Dec 26 20:32:02.203353184 (509h3m2.074436229s) [noise=true , address=145.79.227.47:4265 , piece_id=7KGEMYTWVRSN4BUQBOQK2Z4VO4EC73IGARPJKERFQUU47LG37PKQ, seg_pos={0 0}, upload=446692]: Uploading piece... Dec 26 20:32:02.203712769 (509h3m2.074795814s) [noise=true , address=45.91.116.4:4939 , piece_id=C3WWKTAMVO3PRSBRUGC2IS2W2FXA5SXUZKCAAQNPFDZVSGXKMOCQ, seg_pos={0 0}, upload=446692]: Uploading piece... Dec 26 20:32:02.205162319 (509h3m2.076245354s) [noise=true , address=45.91.116.12:4225 , piece_id=DNN4PR3PPXXD2RCXET56LOETGKQUDMD6CNKZ55KBDBJSJPOJNVLQ, seg_pos={0 0}, upload=446692]: Uploading piece... Dec 26 20:32:02.206627960 (509h3m2.077711005s) [noise=true , address=45.91.116.14:4189 , piece_id=GKCV2YUR5Y53HIKQQHQ5DF5NTVZV66FIOXBGQRYBX3RLNTYBC2EQ, seg_pos={0 0}, upload=446692]: Uploading piece... Dec 26 20:32:02.208047723 (509h3m2.079130768s) [noise=true , address=145.79.227.52:4447 , piece_id=B4OKPZ6XRZF52ZQSJEJSCKUB2V7BAJ7PUDV7ZDX2GFAQ2GE6IZHA, seg_pos={0 0}, upload=446692]: Uploading piece... Dec 26 20:32:03.826302155 (509h3m3.6973853s) [upload=446692]: Next returned nil segment Dec 26 20:32:03.826324998 (509h3m3.697408033s) [upload=446692]: Waiting for error group managing segments... Dec 26 20:32:04.213978995 (509h3m4.08506203s) [noise=true , address=145.79.227.39:4015 , piece_id=NIAXSR6TS5S7VMQIEAXDEKURUB4QNCRH5OWWJR7GYCHYRUKJWJGA, seg_pos={0 0}, upload=446692]: Done uploading piece. err: <nil> Dec 26 20:32:04.214499897 (509h3m4.085582952s) [noise=true , address=145.79.227.11:4732 , piece_id=MTYQCHNCAOH6YKTSQ2Y3DRJ4JV673JHWAAJFP3NWBPMTIUYYUXOA, seg_pos={0 0}, upload=446692]: Done uploading piece. err: <nil> Dec 26 20:32:04.232641842 (509h3m4.103724887s) [noise=true , address=145.79.227.36:4518 , piece_id=JU4OFTL7PDW425BKPYZS3KRWH7F5IVAQUBZBI5FK5HPUW6WRVWLQ, seg_pos={0 0}, upload=446692]: Done uploading piece. err: <nil> Dec 26 20:32:04.232961731 (509h3m4.104044776s) [noise=true , address=145.79.227.48:4301 , piece_id=G7Y7M3FFO7SQO7ABERSS6YCPNELFKRSK6TD6RXJ6OFSSPADO5BTA, seg_pos={0 0}, upload=446692]: Done uploading piece. err: <nil> Dec 26 20:32:04.235278292 (509h3m4.106361337s) [noise=true , address=145.79.227.47:4265 , piece_id=7KGEMYTWVRSN4BUQBOQK2Z4VO4EC73IGARPJKERFQUU47LG37PKQ, seg_pos={0 0}, upload=446692]: Done uploading piece. err: <nil> Dec 26 20:32:04.239793530 (509h3m4.110876575s) [noise=true , address=145.79.227.50:4374 , piece_id=6QE4AJFUNMTP5R35YII7AC4T5CRFXOLJRQOPFR6FBYWXPELHLJGQ, seg_pos={0 0}, upload=446692]: Done uploading piece. err: <nil> Dec 26 20:32:04.242143724 (509h3m4.113226769s) [noise=true , address=145.79.227.12:4767 , piece_id=4AWYUKKCLI2LTMPPVYQHI7QH4ZCL77E7IV62JOKXR65D7HURSITQ, seg_pos={0 0}, upload=446692]: Done uploading piece. err: <nil> Dec 26 20:32:04.248153939 (509h3m4.119236984s) [noise=true , address=145.79.227.29:4942 , piece_id=SYUAKGRXIF3G32L4BTYXOLEACBIYZWT3Y7TWFILTKI7VEGCGD26Q, seg_pos={0 0}, upload=446692]: Done uploading piece. err: <nil> Dec 26 20:32:04.249425882 (509h3m4.120508927s) [noise=true , address=145.79.227.49:4338 , piece_id=EOQRQNUOPZVZSSJKGWNPM4UYHLFCVD2CCMP2VNS6JEVVMBCZDJIQ, seg_pos={0 0}, upload=446692]: Done uploading piece. err: <nil> Dec 26 20:32:04.267334552 (509h3m4.138417597s) [noise=true , address=145.79.227.52:4447 , piece_id=B4OKPZ6XRZF52ZQSJEJSCKUB2V7BAJ7PUDV7ZDX2GFAQ2GE6IZHA, seg_pos={0 0}, upload=446692]: Done uploading piece. err: <nil> Dec 26 20:32:04.471340683 (509h3m4.342423718s) [noise=true , address=45.91.116.14:4189 , piece_id=GKCV2YUR5Y53HIKQQHQ5DF5NTVZV66FIOXBGQRYBX3RLNTYBC2EQ, seg_pos={0 0}, upload=446692]: Done uploading piece. err: <nil> Dec 26 20:32:04.515932554 (509h3m4.387015599s) [noise=true , address=45.91.116.12:4225 , piece_id=DNN4PR3PPXXD2RCXET56LOETGKQUDMD6CNKZ55KBDBJSJPOJNVLQ, seg_pos={0 0}, upload=446692]: Done uploading piece. err: <nil> Dec 26 20:32:04.893872974 (509h3m4.764956019s) [noise=true , address=45.91.116.5:4299 , piece_id=7EZOTMSXLYP66G3SENQ3RFOE6E7N3P3NNLIGUXN4BCRUXWFYZUYA, seg_pos={0 0}, upload=446692]: Done uploading piece. err: <nil> Dec 26 20:32:04.893898913 (509h3m4.764981968s) [seg_pos={0 0}, upload=446692]: Segment reached optimal threshold of 13 pieces. Dec 26 20:32:04.894142377 (509h3m4.765225422s) [noise=true , address=145.79.227.40:4049 , piece_id=MXTOJNJDT7P2FT2ALKPHQREOZ4BAIPBFHTPQZNVA7JCYOKR3QZKA, seg_pos={0 0}, upload=446692]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 26 20:32:04.894151966 (509h3m4.765235001s) [noise=true , address=145.79.227.40:4049 , piece_id=MXTOJNJDT7P2FT2ALKPHQREOZ4BAIPBFHTPQZNVA7JCYOKR3QZKA, seg_pos={0 0}, upload=446692]: Long tail cancellation due to optimal threshold Dec 26 20:32:04.894175169 (509h3m4.765258224s) [noise=true , address=45.91.116.4:4939 , piece_id=C3WWKTAMVO3PRSBRUGC2IS2W2FXA5SXUZKCAAQNPFDZVSGXKMOCQ, seg_pos={0 0}, upload=446692]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 26 20:32:04.894188415 (509h3m4.76527146s) [noise=true , address=45.91.116.4:4939 , piece_id=C3WWKTAMVO3PRSBRUGC2IS2W2FXA5SXUZKCAAQNPFDZVSGXKMOCQ, seg_pos={0 0}, upload=446692]: Long tail cancellation due to optimal threshold Dec 26 20:32:04.894515277 (509h3m4.765598322s) [noise=true , address=145.79.227.51:4410 , piece_id=3E3IKO5CNT4DG6FNNEHBAZZP4HNNBGLXQAUAQ2ICIYFJQMXY3IAQ, seg_pos={0 0}, upload=446692]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX): protocol: write tcp 188.166.225.170:54560->145.79.227.51:4410: use of closed network connection; write tcp 188.166.225.170:54560->145.79.227.51:4410: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54560->145.79.227.51:4410: use of closed network connection Dec 26 20:32:04.894526107 (509h3m4.765609152s) [noise=true , address=145.79.227.51:4410 , piece_id=3E3IKO5CNT4DG6FNNEHBAZZP4HNNBGLXQAUAQ2ICIYFJQMXY3IAQ, seg_pos={0 0}, upload=446692]: Long tail cancellation due to optimal threshold Dec 26 20:32:04.894549141 (509h3m4.765632187s) [seg_pos={0 0}, upload=446692]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 20:32:04.894657357 (509h3m4.765740402s) [upload=446692]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:32:04.894664441 (509h3m4.765747486s) [upload=446692]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:32:05.189650562 (509h3m5.060733617s) [upload=446692]: Done uploading segments... Dec 26 20:32:05.189677313 (509h3m5.060760358s) [upload=446692]: Upload finished. err: <nil> Duration: 30.002520568s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................................X..............X...] duration: 41.4004358s trace: https://jaeger.dev.storj.io/trace/5bdd2de9914a125e FAIL (2025-12-26 20:32:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh, Piece ID: L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q) context deadline exceeded; piecestore: (Node ID: 127RBQsp1SwQU6yWh6Y48QF3QCzeJxEGjob8gzF3LRutRZXA6A9, Piece ID: XUHE2A4VE3U65LONDRNKMJILERSNLNDTJ7R2UPX5D67UI426UQAA) context deadline exceeded; piecestore: (Node ID: 12NFW5tt5R84RaTvY9Up75kNjZ7bmLAgUun41KLdW9Q7eMv27Sq, Piece ID: Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA) context deadline exceeded ======LOGS====== , piece_id=M4GW7XWWHESKY7CULLFIM7THSWLI7Z2FVWZL4YDCPNEXRJNU5TFA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.621775106 (509h0m8.492858151s) [noise=true , address=145.79.227.22:4533 , piece_id=WVQ7YABIOYCCHCMZIRFXIUDKLFFC2KC5WJOABJC2F57QLRJK4H3A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.621846552 (509h0m8.492929597s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.622555641 (509h0m8.493638687s) [noise=true , address=145.79.227.43:4164 , piece_id=XCWV4H42H7J4PQAYNAEODGLSOXULESDU3JW6TJFDWXGIDFBRDGSA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.622948811 (509h0m8.494031846s) [noise=true , address=145.79.227.16:4922 , piece_id=N5JA6JVCAXPKDGIURCNVAWJD6KRTGNRY6S3IZMD5EYVFA454GOKA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623044693 (509h0m8.494127728s) [noise=true , address=77.90.31.11:4570 , piece_id=Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623366476 (509h0m8.494449521s) [noise=true , address=45.91.116.13:4165 , piece_id=YIRLJS6I6YR7WJRG5RLODG5HGZQJQW4V4PKCRIF2BXMWXAFRXTCA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623672749 (509h0m8.494755784s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623792828 (509h0m8.494875863s) [noise=true , address=145.79.227.28:4737 , piece_id=VI474SQ2NRST5NR4YCNYHUNBYBIJSEDCSHQOM6PEF7NEK3NXNTNA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624190374 (509h0m8.495273419s) [noise=true , address=145.79.227.14:4868 , piece_id=3Q4CBS6OWB36IHRUBCRLUZIJQAS76QU5O2YKKQC6XZ5E7FBX4PGA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624328067 (509h0m8.495411112s) [noise=true , address=77.90.31.14:4673 , piece_id=L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624580047 (509h0m8.495663092s) [noise=true , address=145.79.227.30:4784 , piece_id=TIVKXCMDHBQZ5LRKW2MQ3E53LWHQQCYPAZ27NYSRXJWGVEZCXJ3A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624915285 (509h0m8.49599833s) [noise=true , address=145.79.227.34:4930 , piece_id=2BWAT4EMK4QB54A5DRYBSPEGAVDVJPELOLA6ZDROZA4YCK37RJXA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.625529695 (509h0m8.49661276s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.626165055 (509h0m8.49724809s) [noise=true , address=77.90.31.21:4890 , piece_id=NIGMBPZC3J6POFXVSNWZZEJUVKUQ77B7J7IENDZ2C2W3WMZAOD5A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.626720573 (509h0m8.497803618s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:09.940468211 (509h0m9.811551256s) [upload=446649]: Next returned nil segment Dec 26 20:29:09.940490244 (509h0m9.811573289s) [upload=446649]: Waiting for error group managing segments... Dec 26 20:29:10.283357488 (509h0m10.154440524s) [noise=true , address=64.49.13.139:4477 , piece_id=5QMCZNADOLXNYEBNY7EPOMHRPK5EANDLPEUJLU6SQ2QJQCRF66BQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.503143617 (509h0m10.374226653s) [noise=true , address=145.79.227.28:4737 , piece_id=VI474SQ2NRST5NR4YCNYHUNBYBIJSEDCSHQOM6PEF7NEK3NXNTNA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.657417840 (509h0m10.528500875s) [noise=true , address=145.79.227.16:4922 , piece_id=N5JA6JVCAXPKDGIURCNVAWJD6KRTGNRY6S3IZMD5EYVFA454GOKA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.679733744 (509h0m10.550816789s) [noise=true , address=77.90.31.21:4890 , piece_id=NIGMBPZC3J6POFXVSNWZZEJUVKUQ77B7J7IENDZ2C2W3WMZAOD5A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.683939000 (509h0m10.555022046s) [noise=true , address=77.90.31.14:4673 , piece_id=L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.685467862 (509h0m10.556550907s) [noise=true , address=145.79.227.22:4533 , piece_id=WVQ7YABIOYCCHCMZIRFXIUDKLFFC2KC5WJOABJC2F57QLRJK4H3A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.772110194 (509h0m10.643193229s) [noise=true , address=45.91.116.13:4165 , piece_id=YIRLJS6I6YR7WJRG5RLODG5HGZQJQW4V4PKCRIF2BXMWXAFRXTCA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.820042584 (509h0m10.691125629s) [noise=true , address=77.90.31.11:4570 , piece_id=Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.866890631 (509h0m10.737973666s) [noise=true , address=45.91.116.5:4314 , piece_id=XUHE2A4VE3U65LONDRNKMJILERSNLNDTJ7R2UPX5D67UI426UQAA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.999531381 (509h0m10.870614416s) [noise=true , address=145.79.227.43:4164 , piece_id=XCWV4H42H7J4PQAYNAEODGLSOXULESDU3JW6TJFDWXGIDFBRDGSA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.160014888 (509h0m11.031097933s) [noise=true , address=145.79.227.14:4868 , piece_id=3Q4CBS6OWB36IHRUBCRLUZIJQAS76QU5O2YKKQC6XZ5E7FBX4PGA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.463630931 (509h0m11.334713977s) [noise=true , address=145.79.227.34:4930 , piece_id=2BWAT4EMK4QB54A5DRYBSPEGAVDVJPELOLA6ZDROZA4YCK37RJXA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.508749725 (509h0m11.37983276s) [noise=true , address=145.79.227.40:4058 , piece_id=M4GW7XWWHESKY7CULLFIM7THSWLI7Z2FVWZL4YDCPNEXRJNU5TFA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.661052545 (509h0m11.53213558s) [noise=true , address=145.79.227.30:4784 , piece_id=TIVKXCMDHBQZ5LRKW2MQ3E53LWHQQCYPAZ27NYSRXJWGVEZCXJ3A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.661070500 (509h0m11.532153535s) [seg_pos={0 0}, upload=446649]: Segment reached optimal threshold of 14 pieces. Dec 26 20:29:11.661242827 (509h0m11.532325872s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji): protocol: expected piece hash; context canceled; EOF Dec 26 20:29:11.661250402 (509h0m11.532333447s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661295628 (509h0m11.532378663s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt): protocol: write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection; write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection Dec 26 20:29:11.661309314 (509h0m11.532392349s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661349922 (509h0m11.532432957s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1etuVthkhtZexvw7u88vxw2HGPaMwfYCbEgKWF4GPRh9FeWTwn): protocol: expected piece hash; context canceled; EOF Dec 26 20:29:11.661356895 (509h0m11.53243994s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661563188 (509h0m11.532646223s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125rPFZnyTxsa7RBQfAkXtkVCdUJUykq644EK4Ny6HQ1Q7NzrF3): protocol: read tcp 188.166.225.170:36656->45.91.116.6:4982: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 20:29:11.661570592 (509h0m11.532653637s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661611429 (509h0m11.532694474s) [seg_pos={0 0}, upload=446649]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:29:11.661671444 (509h0m11.532754479s) [upload=446649]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:29:11.661676032 (509h0m11.532759077s) [upload=446649]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:29:11.969641419 (509h0m11.840724454s) [upload=446649]: Done uploading segments... Dec 26 20:29:11.969660866 (509h0m11.840743911s) [upload=446649]: Upload finished. err: <nil> Duration: 41.400462862s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [...........XX..X..X...........XXX.X.X....X..XXXX.X...X.X...X] duration: 30.002459797s trace: https://jaeger.dev.storj.io/trace/18a87bc710370f94 FAIL (2025-12-26 20:31:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: UZD2BNFOITUTRDWNQOYGEOWT4RZHFRPJ4HAZRIQKDZ6U7R6YC6AQ) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: XP5UVAKBBPHHKJDDPQI3ID45VTLXBHDH2UZ2XTTYF5NB3SVKVXPA) context deadline exceeded ======LOGS====== Dec 26 20:31:15.249705285 (509h2m15.12078833s) [upload=446680]: Starting upload Dec 26 20:31:15.249763476 (509h2m15.120846552s) [upload=446680]: Done starting upload Dec 26 20:31:15.249962275 (509h2m15.12104533s) [upload=446680]: Uploading segments... Dec 26 20:31:15.250052777 (509h2m15.121135822s) [upload=446680]: Got next segment. Inline: false Dec 26 20:31:15.250118021 (509h2m15.121201066s) [upload=446680]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 20:31:15.250124534 (509h2m15.121207569s) [upload=446680]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 20:31:15.498830742 (509h2m15.369913787s) [seg_pos={0 0}, upload=446680]: Begin upload segment... Dec 26 20:31:15.498875247 (509h2m15.369958292s) [seg_pos={0 0}, upload=446680]: Done begin upload segment. Dec 26 20:31:15.498914281 (509h2m15.369997316s) [noise=true , address=45.91.116.8:4046 , piece_id=6BMPHYK4HGFAO7D4GA6ZZLIIVDWMYDIV7EGLTQUXEMXDKQQSFT6Q, seg_pos={0 0}, upload=446680]: Uploading piece... Dec 26 20:31:15.498966099 (509h2m15.370049145s) [noise=true , address=45.91.116.9:4263 , piece_id=UZD2BNFOITUTRDWNQOYGEOWT4RZHFRPJ4HAZRIQKDZ6U7R6YC6AQ, seg_pos={0 0}, upload=446680]: Uploading piece... Dec 26 20:31:15.499765943 (509h2m15.370848978s) [noise=true , address=145.79.227.11:4731 , piece_id=YP7ZNN22TQR6IGVJMUPNZCY5P2TX5X7NRBTVVX6RYDDVEANJQCLQ, seg_pos={0 0}, upload=446680]: Uploading piece... Dec 26 20:31:15.500290481 (509h2m15.371373516s) [noise=true , address=145.79.227.38:4978 , piece_id=XKGGUL2JRC7AG3VHJOWHGAAALHIF3BCOVHAFC4ERZ5UWNYKDQYQA, seg_pos={0 0}, upload=446680]: Uploading piece... Dec 26 20:31:15.500743824 (509h2m15.37182687s) [noise=true , address=77.90.31.19:4844 , piece_id=ES2S3TKDQBQGYX4SI7PXU4WGSPTFPQMEMCBVYL7OJBRKVV4WR7YQ, seg_pos={0 0}, upload=446680]: Uploading piece... Dec 26 20:31:15.501134149 (509h2m15.372217194s) [noise=true , address=145.79.227.24:4582 , piece_id=ZGDRIKWXSQWWZUU3WRD63K26IXX36B7NT4I2XX65JQ3CUH74SMHQ, seg_pos={0 0}, upload=446680]: Uploading piece... Dec 26 20:31:15.501542786 (509h2m15.372625821s) [noise=true , address=45.91.116.11:4120 , piece_id=INMTX3FWZQA7LDOLYXY2AK6S53IWTLE7VA2GEBWLFNTZVNP3ZGTA, seg_pos={0 0}, upload=446680]: Uploading piece... Dec 26 20:31:15.501972675 (509h2m15.37305572s) [noise=true , address=145.79.227.49:4339 , piece_id=BYDGS4ESN7RWHZQIKD65LKO2EYLVH3UW5PUDHQ6S3KRVOKFVWKRA, seg_pos={0 0}, upload=446680]: Uploading piece... Dec 26 20:31:15.502215688 (509h2m15.373298723s) [noise=true , address=45.91.116.6:4975 , piece_id=HRF4HV5NEETDBSWCYKTH7KPIB7CZGNNSL3ZDNVFVUJNS3WCLV74A, seg_pos={0 0}, upload=446680]: Uploading piece... Dec 26 20:31:15.502506591 (509h2m15.373589627s) [noise=true , address=145.79.227.4:4185 , piece_id=HMMMNABR2YZ3NAWS6DWKHTMFBIZL6EY67HQGTHRRFLAKTOGDUQGA, seg_pos={0 0}, upload=446680]: Uploading piece... Dec 26 20:31:15.502731760 (509h2m15.373814795s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRCHKPDT3YUTVOWOIRHGTOXILHFUMKEQRDY6E4BHCQK2633K27Q, seg_pos={0 0}, upload=446680]: Uploading piece... Dec 26 20:31:15.503009218 (509h2m15.374092263s) [noise=true , address=145.79.227.6:4259 , piece_id=2BYGF5SFX2B6RVSOIDM3S5VMEUMMZPU3I2MHAVVTT7PIUTIEMCNA, seg_pos={0 0}, upload=446680]: Uploading piece... Dec 26 20:31:15.503100222 (509h2m15.374183257s) [noise=true , address=45.91.116.14:4192 , piece_id=POR6XYYMDJF5RJB2Y6GEDCM2BQLDIMM4WBOIQLR6BLE3NR2LCVJA, seg_pos={0 0}, upload=446680]: Uploading piece... Dec 26 20:31:15.503429940 (509h2m15.374512975s) [noise=true , address=45.91.116.5:4300 , piece_id=XP5UVAKBBPHHKJDDPQI3ID45VTLXBHDH2UZ2XTTYF5NB3SVKVXPA, seg_pos={0 0}, upload=446680]: Uploading piece... Dec 26 20:31:15.503691448 (509h2m15.374774483s) [noise=true , address=145.79.227.50:4375 , piece_id=SK56D73GB2GK67UCPXVH3PGVBFPWNO3INMAJQ73LNUUYHK6THPKQ, seg_pos={0 0}, upload=446680]: Uploading piece... Dec 26 20:31:15.504010986 (509h2m15.375094021s) [noise=true , address=145.79.227.15:4876 , piece_id=YVLMIVCGBXY7FMPQ7HSHTNPWP574Z6GXAJGEOPFEWP2MOYRJ5W2Q, seg_pos={0 0}, upload=446680]: Uploading piece... Dec 26 20:31:17.223207878 (509h2m17.094290913s) [upload=446680]: Next returned nil segment Dec 26 20:31:17.223247693 (509h2m17.094330728s) [upload=446680]: Waiting for error group managing segments... Dec 26 20:31:17.506070284 (509h2m17.377153319s) [noise=true , address=145.79.227.49:4339 , piece_id=BYDGS4ESN7RWHZQIKD65LKO2EYLVH3UW5PUDHQ6S3KRVOKFVWKRA, seg_pos={0 0}, upload=446680]: Done uploading piece. err: <nil> Dec 26 20:31:17.508235476 (509h2m17.379318511s) [noise=true , address=145.79.227.24:4582 , piece_id=ZGDRIKWXSQWWZUU3WRD63K26IXX36B7NT4I2XX65JQ3CUH74SMHQ, seg_pos={0 0}, upload=446680]: Done uploading piece. err: <nil> Dec 26 20:31:17.513199970 (509h2m17.384283005s) [noise=true , address=145.79.227.4:4185 , piece_id=HMMMNABR2YZ3NAWS6DWKHTMFBIZL6EY67HQGTHRRFLAKTOGDUQGA, seg_pos={0 0}, upload=446680]: Done uploading piece. err: <nil> Dec 26 20:31:17.519998766 (509h2m17.391081801s) [noise=true , address=145.79.227.11:4731 , piece_id=YP7ZNN22TQR6IGVJMUPNZCY5P2TX5X7NRBTVVX6RYDDVEANJQCLQ, seg_pos={0 0}, upload=446680]: Done uploading piece. err: <nil> Dec 26 20:31:17.524304123 (509h2m17.395387158s) [noise=true , address=145.79.227.50:4375 , piece_id=SK56D73GB2GK67UCPXVH3PGVBFPWNO3INMAJQ73LNUUYHK6THPKQ, seg_pos={0 0}, upload=446680]: Done uploading piece. err: <nil> Dec 26 20:31:17.527331688 (509h2m17.398414723s) [noise=true , address=145.79.227.15:4876 , piece_id=YVLMIVCGBXY7FMPQ7HSHTNPWP574Z6GXAJGEOPFEWP2MOYRJ5W2Q, seg_pos={0 0}, upload=446680]: Done uploading piece. err: <nil> Dec 26 20:31:17.533964359 (509h2m17.405047404s) [noise=true , address=145.79.227.38:4978 , piece_id=XKGGUL2JRC7AG3VHJOWHGAAALHIF3BCOVHAFC4ERZ5UWNYKDQYQA, seg_pos={0 0}, upload=446680]: Done uploading piece. err: <nil> Dec 26 20:31:17.730047793 (509h2m17.601130828s) [noise=true , address=45.91.116.9:4263 , piece_id=UZD2BNFOITUTRDWNQOYGEOWT4RZHFRPJ4HAZRIQKDZ6U7R6YC6AQ, seg_pos={0 0}, upload=446680]: Done uploading piece. err: <nil> Dec 26 20:31:17.731052847 (509h2m17.602135882s) [noise=true , address=45.91.116.14:4192 , piece_id=POR6XYYMDJF5RJB2Y6GEDCM2BQLDIMM4WBOIQLR6BLE3NR2LCVJA, seg_pos={0 0}, upload=446680]: Done uploading piece. err: <nil> Dec 26 20:31:17.812217318 (509h2m17.683300353s) [noise=true , address=145.79.227.47:4266 , piece_id=HNRCHKPDT3YUTVOWOIRHGTOXILHFUMKEQRDY6E4BHCQK2633K27Q, seg_pos={0 0}, upload=446680]: Done uploading piece. err: <nil> Dec 26 20:31:17.812894567 (509h2m17.683977602s) [noise=true , address=45.91.116.5:4300 , piece_id=XP5UVAKBBPHHKJDDPQI3ID45VTLXBHDH2UZ2XTTYF5NB3SVKVXPA, seg_pos={0 0}, upload=446680]: Done uploading piece. err: <nil> Dec 26 20:31:17.840125339 (509h2m17.711208374s) [noise=true , address=77.90.31.19:4844 , piece_id=ES2S3TKDQBQGYX4SI7PXU4WGSPTFPQMEMCBVYL7OJBRKVV4WR7YQ, seg_pos={0 0}, upload=446680]: Done uploading piece. err: <nil> Dec 26 20:31:17.863788638 (509h2m17.734871683s) [noise=true , address=45.91.116.8:4046 , piece_id=6BMPHYK4HGFAO7D4GA6ZZLIIVDWMYDIV7EGLTQUXEMXDKQQSFT6Q, seg_pos={0 0}, upload=446680]: Done uploading piece. err: <nil> Dec 26 20:31:17.863806011 (509h2m17.734889046s) [seg_pos={0 0}, upload=446680]: Segment reached optimal threshold of 13 pieces. Dec 26 20:31:17.864027612 (509h2m17.735110647s) [noise=true , address=45.91.116.11:4120 , piece_id=INMTX3FWZQA7LDOLYXY2AK6S53IWTLE7VA2GEBWLFNTZVNP3ZGTA, seg_pos={0 0}, upload=446680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 26 20:31:17.864034957 (509h2m17.735117992s) [noise=true , address=45.91.116.11:4120 , piece_id=INMTX3FWZQA7LDOLYXY2AK6S53IWTLE7VA2GEBWLFNTZVNP3ZGTA, seg_pos={0 0}, upload=446680]: Long tail cancellation due to optimal threshold Dec 26 20:31:17.864135178 (509h2m17.735218223s) [noise=true , address=45.91.116.6:4975 , piece_id=HRF4HV5NEETDBSWCYKTH7KPIB7CZGNNSL3ZDNVFVUJNS3WCLV74A, seg_pos={0 0}, upload=446680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 26 20:31:17.864172379 (509h2m17.735255414s) [noise=true , address=45.91.116.6:4975 , piece_id=HRF4HV5NEETDBSWCYKTH7KPIB7CZGNNSL3ZDNVFVUJNS3WCLV74A, seg_pos={0 0}, upload=446680]: Long tail cancellation due to optimal threshold Dec 26 20:31:17.864282548 (509h2m17.735365584s) [noise=true , address=145.79.227.6:4259 , piece_id=2BYGF5SFX2B6RVSOIDM3S5VMEUMMZPU3I2MHAVVTT7PIUTIEMCNA, seg_pos={0 0}, upload=446680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y): protocol: write tcp 188.166.225.170:48662->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:48662->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48662->145.79.227.6:4259: use of closed network connection Dec 26 20:31:17.864289411 (509h2m17.735372446s) [noise=true , address=145.79.227.6:4259 , piece_id=2BYGF5SFX2B6RVSOIDM3S5VMEUMMZPU3I2MHAVVTT7PIUTIEMCNA, seg_pos={0 0}, upload=446680]: Long tail cancellation due to optimal threshold Dec 26 20:31:17.864307856 (509h2m17.735390901s) [seg_pos={0 0}, upload=446680]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 20:31:17.864397607 (509h2m17.735480652s) [upload=446680]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:31:17.864402236 (509h2m17.735485271s) [upload=446680]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:31:18.161172324 (509h2m18.032255359s) [upload=446680]: Done uploading segments... Dec 26 20:31:18.161187844 (509h2m18.032270879s) [upload=446680]: Upload finished. err: <nil> Duration: 30.002483223s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................................X..............X..] duration: 31.24303585s trace: https://jaeger.dev.storj.io/trace/5bdd2de9914a125e FAIL (2025-12-26 20:31:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh, Piece ID: L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q) context deadline exceeded; piecestore: (Node ID: 127RBQsp1SwQU6yWh6Y48QF3QCzeJxEGjob8gzF3LRutRZXA6A9, Piece ID: XUHE2A4VE3U65LONDRNKMJILERSNLNDTJ7R2UPX5D67UI426UQAA) context deadline exceeded; piecestore: (Node ID: 12NFW5tt5R84RaTvY9Up75kNjZ7bmLAgUun41KLdW9Q7eMv27Sq, Piece ID: Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA) context deadline exceeded ======LOGS====== , piece_id=M4GW7XWWHESKY7CULLFIM7THSWLI7Z2FVWZL4YDCPNEXRJNU5TFA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.621775106 (509h0m8.492858151s) [noise=true , address=145.79.227.22:4533 , piece_id=WVQ7YABIOYCCHCMZIRFXIUDKLFFC2KC5WJOABJC2F57QLRJK4H3A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.621846552 (509h0m8.492929597s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.622555641 (509h0m8.493638687s) [noise=true , address=145.79.227.43:4164 , piece_id=XCWV4H42H7J4PQAYNAEODGLSOXULESDU3JW6TJFDWXGIDFBRDGSA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.622948811 (509h0m8.494031846s) [noise=true , address=145.79.227.16:4922 , piece_id=N5JA6JVCAXPKDGIURCNVAWJD6KRTGNRY6S3IZMD5EYVFA454GOKA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623044693 (509h0m8.494127728s) [noise=true , address=77.90.31.11:4570 , piece_id=Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623366476 (509h0m8.494449521s) [noise=true , address=45.91.116.13:4165 , piece_id=YIRLJS6I6YR7WJRG5RLODG5HGZQJQW4V4PKCRIF2BXMWXAFRXTCA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623672749 (509h0m8.494755784s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623792828 (509h0m8.494875863s) [noise=true , address=145.79.227.28:4737 , piece_id=VI474SQ2NRST5NR4YCNYHUNBYBIJSEDCSHQOM6PEF7NEK3NXNTNA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624190374 (509h0m8.495273419s) [noise=true , address=145.79.227.14:4868 , piece_id=3Q4CBS6OWB36IHRUBCRLUZIJQAS76QU5O2YKKQC6XZ5E7FBX4PGA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624328067 (509h0m8.495411112s) [noise=true , address=77.90.31.14:4673 , piece_id=L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624580047 (509h0m8.495663092s) [noise=true , address=145.79.227.30:4784 , piece_id=TIVKXCMDHBQZ5LRKW2MQ3E53LWHQQCYPAZ27NYSRXJWGVEZCXJ3A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624915285 (509h0m8.49599833s) [noise=true , address=145.79.227.34:4930 , piece_id=2BWAT4EMK4QB54A5DRYBSPEGAVDVJPELOLA6ZDROZA4YCK37RJXA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.625529695 (509h0m8.49661276s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.626165055 (509h0m8.49724809s) [noise=true , address=77.90.31.21:4890 , piece_id=NIGMBPZC3J6POFXVSNWZZEJUVKUQ77B7J7IENDZ2C2W3WMZAOD5A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.626720573 (509h0m8.497803618s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:09.940468211 (509h0m9.811551256s) [upload=446649]: Next returned nil segment Dec 26 20:29:09.940490244 (509h0m9.811573289s) [upload=446649]: Waiting for error group managing segments... Dec 26 20:29:10.283357488 (509h0m10.154440524s) [noise=true , address=64.49.13.139:4477 , piece_id=5QMCZNADOLXNYEBNY7EPOMHRPK5EANDLPEUJLU6SQ2QJQCRF66BQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.503143617 (509h0m10.374226653s) [noise=true , address=145.79.227.28:4737 , piece_id=VI474SQ2NRST5NR4YCNYHUNBYBIJSEDCSHQOM6PEF7NEK3NXNTNA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.657417840 (509h0m10.528500875s) [noise=true , address=145.79.227.16:4922 , piece_id=N5JA6JVCAXPKDGIURCNVAWJD6KRTGNRY6S3IZMD5EYVFA454GOKA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.679733744 (509h0m10.550816789s) [noise=true , address=77.90.31.21:4890 , piece_id=NIGMBPZC3J6POFXVSNWZZEJUVKUQ77B7J7IENDZ2C2W3WMZAOD5A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.683939000 (509h0m10.555022046s) [noise=true , address=77.90.31.14:4673 , piece_id=L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.685467862 (509h0m10.556550907s) [noise=true , address=145.79.227.22:4533 , piece_id=WVQ7YABIOYCCHCMZIRFXIUDKLFFC2KC5WJOABJC2F57QLRJK4H3A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.772110194 (509h0m10.643193229s) [noise=true , address=45.91.116.13:4165 , piece_id=YIRLJS6I6YR7WJRG5RLODG5HGZQJQW4V4PKCRIF2BXMWXAFRXTCA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.820042584 (509h0m10.691125629s) [noise=true , address=77.90.31.11:4570 , piece_id=Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.866890631 (509h0m10.737973666s) [noise=true , address=45.91.116.5:4314 , piece_id=XUHE2A4VE3U65LONDRNKMJILERSNLNDTJ7R2UPX5D67UI426UQAA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.999531381 (509h0m10.870614416s) [noise=true , address=145.79.227.43:4164 , piece_id=XCWV4H42H7J4PQAYNAEODGLSOXULESDU3JW6TJFDWXGIDFBRDGSA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.160014888 (509h0m11.031097933s) [noise=true , address=145.79.227.14:4868 , piece_id=3Q4CBS6OWB36IHRUBCRLUZIJQAS76QU5O2YKKQC6XZ5E7FBX4PGA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.463630931 (509h0m11.334713977s) [noise=true , address=145.79.227.34:4930 , piece_id=2BWAT4EMK4QB54A5DRYBSPEGAVDVJPELOLA6ZDROZA4YCK37RJXA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.508749725 (509h0m11.37983276s) [noise=true , address=145.79.227.40:4058 , piece_id=M4GW7XWWHESKY7CULLFIM7THSWLI7Z2FVWZL4YDCPNEXRJNU5TFA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.661052545 (509h0m11.53213558s) [noise=true , address=145.79.227.30:4784 , piece_id=TIVKXCMDHBQZ5LRKW2MQ3E53LWHQQCYPAZ27NYSRXJWGVEZCXJ3A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.661070500 (509h0m11.532153535s) [seg_pos={0 0}, upload=446649]: Segment reached optimal threshold of 14 pieces. Dec 26 20:29:11.661242827 (509h0m11.532325872s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji): protocol: expected piece hash; context canceled; EOF Dec 26 20:29:11.661250402 (509h0m11.532333447s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661295628 (509h0m11.532378663s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt): protocol: write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection; write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection Dec 26 20:29:11.661309314 (509h0m11.532392349s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661349922 (509h0m11.532432957s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1etuVthkhtZexvw7u88vxw2HGPaMwfYCbEgKWF4GPRh9FeWTwn): protocol: expected piece hash; context canceled; EOF Dec 26 20:29:11.661356895 (509h0m11.53243994s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661563188 (509h0m11.532646223s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125rPFZnyTxsa7RBQfAkXtkVCdUJUykq644EK4Ny6HQ1Q7NzrF3): protocol: read tcp 188.166.225.170:36656->45.91.116.6:4982: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 20:29:11.661570592 (509h0m11.532653637s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661611429 (509h0m11.532694474s) [seg_pos={0 0}, upload=446649]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:29:11.661671444 (509h0m11.532754479s) [upload=446649]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:29:11.661676032 (509h0m11.532759077s) [upload=446649]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:29:11.969641419 (509h0m11.840724454s) [upload=446649]: Done uploading segments... Dec 26 20:29:11.969660866 (509h0m11.840743911s) [upload=446649]: Upload finished. err: <nil> Duration: 31.243065657s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [............XX..X..X...........XXX.X.X....X..XXXX.X...X.X...] duration: 8.818674406s trace: https://jaeger.dev.storj.io/trace/13a2168592a1b0d1 FAIL (2025-12-26 20:30:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: LRDECELM2HNBG73DCMNXH4JGRGJY7GVTRGRIPPIVZFTPAVWDIXYA) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: MJ65ZTQBDAM6HODPAZ3FBBHQHA5XXVV43RELWC5O77EUSBLIUJ6Q) context deadline exceeded; piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: 2AMJYSPO5R42RNXQQOMJN7FE52LR555ORB3VBKM3CLNW5NMDLLYA) context deadline exceeded; piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: 7DEGRLZFNZUDMWOKBGQAUZ5EII7INYT3ATHBAFODRRIAAPTVNG4Q) context deadline exceeded ======LOGS====== 20:28:29.677935962 (508h59m29.549018997s) [upload=446641]: Starting upload Dec 26 20:28:29.678013801 (508h59m29.549096836s) [upload=446641]: Done starting upload Dec 26 20:28:29.678251913 (508h59m29.549334948s) [upload=446641]: Uploading segments... Dec 26 20:28:29.678448809 (508h59m29.549531844s) [upload=446641]: Got next segment. Inline: false Dec 26 20:28:29.678508312 (508h59m29.549591357s) [upload=446641]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 20:28:29.678513151 (508h59m29.549596186s) [upload=446641]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 20:28:29.925664959 (508h59m29.796748024s) [seg_pos={0 0}, upload=446641]: Begin upload segment... Dec 26 20:28:29.925733329 (508h59m29.796816364s) [seg_pos={0 0}, upload=446641]: Done begin upload segment. Dec 26 20:28:29.925739741 (508h59m29.796822776s) [noise=true , address=145.79.227.51:4409 , piece_id=3L6W4XDZBCO34CRQBE47E6NXVW4OEVMAD3CE2IONC322GKM6HCRQ, seg_pos={0 0}, upload=446641]: Uploading piece... Dec 26 20:28:29.925763677 (508h59m29.796846722s) [noise=true , address=45.91.116.4:4939 , piece_id=RGCRAOILK2NIAUEFZ3HHTF5L3MIMI65GHKHUMYL34AESHDZPOCEQ, seg_pos={0 0}, upload=446641]: Uploading piece... Dec 26 20:28:29.926292754 (508h59m29.79737579s) [noise=true , address=45.91.116.6:4976 , piece_id=7DEGRLZFNZUDMWOKBGQAUZ5EII7INYT3ATHBAFODRRIAAPTVNG4Q, seg_pos={0 0}, upload=446641]: Uploading piece... Dec 26 20:28:29.926671025 (508h59m29.79775406s) [noise=true , address=145.79.227.37:4567 , piece_id=KEKXRVZ6TL6UJBO7AYYIQSI5RDJDMVQXTNUINQGQQDWCXWNSPTNQ, seg_pos={0 0}, upload=446641]: Uploading piece... Dec 26 20:28:29.926962490 (508h59m29.798045525s) [noise=true , address=145.79.227.36:4517 , piece_id=ZTHEHA7KGDI3YZ3NKV65ZMLRYSZ3VJCHOUB75RZG2GQCNQHNXD5A, seg_pos={0 0}, upload=446641]: Uploading piece... Dec 26 20:28:29.927218868 (508h59m29.798301903s) [noise=true , address=145.79.227.46:4603 , piece_id=D4NLBH33S4MBTMYV4ULTHLL5DUT5ZLPYU2OFIRMZ2JBIXH6RFYXA, seg_pos={0 0}, upload=446641]: Uploading piece... Dec 26 20:28:29.927418408 (508h59m29.798501443s) [noise=true , address=45.91.116.10:4083 , piece_id=LRDECELM2HNBG73DCMNXH4JGRGJY7GVTRGRIPPIVZFTPAVWDIXYA, seg_pos={0 0}, upload=446641]: Uploading piece... Dec 26 20:28:29.927874998 (508h59m29.798958043s) [noise=true , address=77.90.31.15:4697 , piece_id=74W7KZ2D6H2AUKETVRYMCXVJ6HEBX66SCPI54SZLVSFINWVMJQJA, seg_pos={0 0}, upload=446641]: Uploading piece... Dec 26 20:28:29.928147736 (508h59m29.799230771s) [noise=true , address=145.79.227.2:4114 , piece_id=KPLGTMQAJN3JZHOZR2V2QL76VNUNZZRWNIQ7WPRX5EUPGJ2XOGSA, seg_pos={0 0}, upload=446641]: Uploading piece... Dec 26 20:28:29.928229733 (508h59m29.799312768s) [noise=true , address=45.91.116.8:4046 , piece_id=MJ65ZTQBDAM6HODPAZ3FBBHQHA5XXVV43RELWC5O77EUSBLIUJ6Q, seg_pos={0 0}, upload=446641]: Uploading piece... Dec 26 20:28:29.928491942 (508h59m29.799574977s) [noise=true , address=145.79.227.40:4051 , piece_id=MD7IDP6E37DEMNDSMAXTJDGNRRFZQNLLKHEHMYOJRTVWNNRLQ2BQ, seg_pos={0 0}, upload=446641]: Uploading piece... Dec 26 20:28:29.928799157 (508h59m29.799882192s) [noise=true , address=145.79.227.3:4151 , piece_id=TPR72DFEKFADGYHCAGQWJ4RJTJDNXXB5KOPKPC2HA3EFRJQH5OWQ, seg_pos={0 0}, upload=446641]: Uploading piece... Dec 26 20:28:29.929092376 (508h59m29.800175421s) [noise=true , address=145.79.227.45:4229 , piece_id=3T35SGG2KV4DYKBBIW6F5JEQMB6NL7YOMRB6R5ADIQJ3WHYQN5VQ, seg_pos={0 0}, upload=446641]: Uploading piece... Dec 26 20:28:29.929469754 (508h59m29.800552789s) [noise=true , address=145.79.227.21:4475 , piece_id=FM2HSCM6RJUESMMXEO4VMSRTTA2X4ER7JSWQ63O5BNDMTM2L6PEA, seg_pos={0 0}, upload=446641]: Uploading piece... Dec 26 20:28:29.929936122 (508h59m29.801019157s) [noise=true , address=77.90.31.13:4629 , piece_id=2AMJYSPO5R42RNXQQOMJN7FE52LR555ORB3VBKM3CLNW5NMDLLYA, seg_pos={0 0}, upload=446641]: Uploading piece... Dec 26 20:28:29.930013139 (508h59m29.801096174s) [noise=true , address=145.79.227.25:4618 , piece_id=UU6KFIQCYNZEWWMHSZ4NQ7TOZYA7N4AZMI4VN7RZAQCE4VAAMRFA, seg_pos={0 0}, upload=446641]: Uploading piece... Dec 26 20:28:31.349080054 (508h59m31.220163089s) [noise=true , address=77.90.31.15:4697 , piece_id=74W7KZ2D6H2AUKETVRYMCXVJ6HEBX66SCPI54SZLVSFINWVMJQJA, seg_pos={0 0}, upload=446641]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: storage node overloaded, request limit: 2000 Dec 26 20:28:31.603377739 (508h59m31.474460774s) [upload=446641]: Next returned nil segment Dec 26 20:28:31.603400412 (508h59m31.474483447s) [upload=446641]: Waiting for error group managing segments... Dec 26 20:28:31.928179705 (508h59m31.79926274s) [noise=true , address=145.79.227.25:4618 , piece_id=UU6KFIQCYNZEWWMHSZ4NQ7TOZYA7N4AZMI4VN7RZAQCE4VAAMRFA, seg_pos={0 0}, upload=446641]: Done uploading piece. err: <nil> Dec 26 20:28:31.930119068 (508h59m31.801202103s) [noise=true , address=145.79.227.21:4475 , piece_id=FM2HSCM6RJUESMMXEO4VMSRTTA2X4ER7JSWQ63O5BNDMTM2L6PEA, seg_pos={0 0}, upload=446641]: Done uploading piece. err: <nil> Dec 26 20:28:31.938003371 (508h59m31.809086416s) [noise=true , address=145.79.227.46:4603 , piece_id=D4NLBH33S4MBTMYV4ULTHLL5DUT5ZLPYU2OFIRMZ2JBIXH6RFYXA, seg_pos={0 0}, upload=446641]: Done uploading piece. err: <nil> Dec 26 20:28:31.940924854 (508h59m31.812007889s) [noise=true , address=145.79.227.45:4229 , piece_id=3T35SGG2KV4DYKBBIW6F5JEQMB6NL7YOMRB6R5ADIQJ3WHYQN5VQ, seg_pos={0 0}, upload=446641]: Done uploading piece. err: <nil> Dec 26 20:28:31.942105522 (508h59m31.813188557s) [noise=true , address=145.79.227.36:4517 , piece_id=ZTHEHA7KGDI3YZ3NKV65ZMLRYSZ3VJCHOUB75RZG2GQCNQHNXD5A, seg_pos={0 0}, upload=446641]: Done uploading piece. err: <nil> Dec 26 20:28:31.964426635 (508h59m31.83550967s) [noise=true , address=145.79.227.37:4567 , piece_id=KEKXRVZ6TL6UJBO7AYYIQSI5RDJDMVQXTNUINQGQQDWCXWNSPTNQ, seg_pos={0 0}, upload=446641]: Done uploading piece. err: <nil> Dec 26 20:28:32.158955319 (508h59m32.030038364s) [noise=true , address=45.91.116.4:4939 , piece_id=RGCRAOILK2NIAUEFZ3HHTF5L3MIMI65GHKHUMYL34AESHDZPOCEQ, seg_pos={0 0}, upload=446641]: Done uploading piece. err: <nil> Dec 26 20:28:32.161331084 (508h59m32.032414129s) [noise=true , address=45.91.116.8:4046 , piece_id=MJ65ZTQBDAM6HODPAZ3FBBHQHA5XXVV43RELWC5O77EUSBLIUJ6Q, seg_pos={0 0}, upload=446641]: Done uploading piece. err: <nil> Dec 26 20:28:32.235774115 (508h59m32.10685715s) [noise=true , address=45.91.116.6:4976 , piece_id=7DEGRLZFNZUDMWOKBGQAUZ5EII7INYT3ATHBAFODRRIAAPTVNG4Q, seg_pos={0 0}, upload=446641]: Done uploading piece. err: <nil> Dec 26 20:28:32.246433393 (508h59m32.117516428s) [noise=true , address=77.90.31.13:4629 , piece_id=2AMJYSPO5R42RNXQQOMJN7FE52LR555ORB3VBKM3CLNW5NMDLLYA, seg_pos={0 0}, upload=446641]: Done uploading piece. err: <nil> Dec 26 20:28:32.394498402 (508h59m32.265581447s) [noise=true , address=45.91.116.10:4083 , piece_id=LRDECELM2HNBG73DCMNXH4JGRGJY7GVTRGRIPPIVZFTPAVWDIXYA, seg_pos={0 0}, upload=446641]: Done uploading piece. err: <nil> Dec 26 20:28:32.471701779 (508h59m32.342784834s) [noise=true , address=145.79.227.3:4151 , piece_id=TPR72DFEKFADGYHCAGQWJ4RJTJDNXXB5KOPKPC2HA3EFRJQH5OWQ, seg_pos={0 0}, upload=446641]: Done uploading piece. err: <nil> Dec 26 20:28:33.272979594 (508h59m33.144062629s) [noise=true , address=145.79.227.40:4051 , piece_id=MD7IDP6E37DEMNDSMAXTJDGNRRFZQNLLKHEHMYOJRTVWNNRLQ2BQ, seg_pos={0 0}, upload=446641]: Done uploading piece. err: <nil> Dec 26 20:28:33.273003128 (508h59m33.144086173s) [seg_pos={0 0}, upload=446641]: Segment reached optimal threshold of 13 pieces. Dec 26 20:28:33.273210062 (508h59m33.144293097s) [noise=true , address=145.79.227.2:4114 , piece_id=KPLGTMQAJN3JZHOZR2V2QL76VNUNZZRWNIQ7WPRX5EUPGJ2XOGSA, seg_pos={0 0}, upload=446641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN): protocol: write tcp 188.166.225.170:35140->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:35140->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35140->145.79.227.2:4114: use of closed network connection Dec 26 20:28:33.273218258 (508h59m33.144301323s) [noise=true , address=145.79.227.2:4114 , piece_id=KPLGTMQAJN3JZHOZR2V2QL76VNUNZZRWNIQ7WPRX5EUPGJ2XOGSA, seg_pos={0 0}, upload=446641]: Long tail cancellation due to optimal threshold Dec 26 20:28:33.273418729 (508h59m33.144501764s) [noise=true , address=145.79.227.51:4409 , piece_id=3L6W4XDZBCO34CRQBE47E6NXVW4OEVMAD3CE2IONC322GKM6HCRQ, seg_pos={0 0}, upload=446641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UoCGA4FjE4yADPXL1oifSPJssPgSGh81REZHwGv2eu1Y1kEaj): protocol: write tcp 188.166.225.170:44904->145.79.227.51:4409: use of closed network connection; write tcp 188.166.225.170:44904->145.79.227.51:4409: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44904->145.79.227.51:4409: use of closed network connection Dec 26 20:28:33.273454016 (508h59m33.144537051s) [noise=true , address=145.79.227.51:4409 , piece_id=3L6W4XDZBCO34CRQBE47E6NXVW4OEVMAD3CE2IONC322GKM6HCRQ, seg_pos={0 0}, upload=446641]: Long tail cancellation due to optimal threshold Dec 26 20:28:33.273476710 (508h59m33.144559745s) [seg_pos={0 0}, upload=446641]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 20:28:33.273587150 (508h59m33.144670185s) [upload=446641]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:28:33.273604383 (508h59m33.144687418s) [upload=446641]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:28:33.579545726 (508h59m33.450628771s) [upload=446641]: Done uploading segments... Dec 26 20:28:33.579573469 (508h59m33.450656514s) [upload=446641]: Upload finished. err: <nil> Duration: 8.818695516s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [X..........................................X..............X.] duration: 29.841774228s trace: https://jaeger.dev.storj.io/trace/5bdd2de9914a125e FAIL (2025-12-26 20:30:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh, Piece ID: L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q) context deadline exceeded; piecestore: (Node ID: 127RBQsp1SwQU6yWh6Y48QF3QCzeJxEGjob8gzF3LRutRZXA6A9, Piece ID: XUHE2A4VE3U65LONDRNKMJILERSNLNDTJ7R2UPX5D67UI426UQAA) context deadline exceeded; piecestore: (Node ID: 12NFW5tt5R84RaTvY9Up75kNjZ7bmLAgUun41KLdW9Q7eMv27Sq, Piece ID: Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA) context deadline exceeded ======LOGS====== , piece_id=M4GW7XWWHESKY7CULLFIM7THSWLI7Z2FVWZL4YDCPNEXRJNU5TFA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.621775106 (509h0m8.492858151s) [noise=true , address=145.79.227.22:4533 , piece_id=WVQ7YABIOYCCHCMZIRFXIUDKLFFC2KC5WJOABJC2F57QLRJK4H3A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.621846552 (509h0m8.492929597s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.622555641 (509h0m8.493638687s) [noise=true , address=145.79.227.43:4164 , piece_id=XCWV4H42H7J4PQAYNAEODGLSOXULESDU3JW6TJFDWXGIDFBRDGSA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.622948811 (509h0m8.494031846s) [noise=true , address=145.79.227.16:4922 , piece_id=N5JA6JVCAXPKDGIURCNVAWJD6KRTGNRY6S3IZMD5EYVFA454GOKA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623044693 (509h0m8.494127728s) [noise=true , address=77.90.31.11:4570 , piece_id=Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623366476 (509h0m8.494449521s) [noise=true , address=45.91.116.13:4165 , piece_id=YIRLJS6I6YR7WJRG5RLODG5HGZQJQW4V4PKCRIF2BXMWXAFRXTCA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623672749 (509h0m8.494755784s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623792828 (509h0m8.494875863s) [noise=true , address=145.79.227.28:4737 , piece_id=VI474SQ2NRST5NR4YCNYHUNBYBIJSEDCSHQOM6PEF7NEK3NXNTNA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624190374 (509h0m8.495273419s) [noise=true , address=145.79.227.14:4868 , piece_id=3Q4CBS6OWB36IHRUBCRLUZIJQAS76QU5O2YKKQC6XZ5E7FBX4PGA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624328067 (509h0m8.495411112s) [noise=true , address=77.90.31.14:4673 , piece_id=L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624580047 (509h0m8.495663092s) [noise=true , address=145.79.227.30:4784 , piece_id=TIVKXCMDHBQZ5LRKW2MQ3E53LWHQQCYPAZ27NYSRXJWGVEZCXJ3A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624915285 (509h0m8.49599833s) [noise=true , address=145.79.227.34:4930 , piece_id=2BWAT4EMK4QB54A5DRYBSPEGAVDVJPELOLA6ZDROZA4YCK37RJXA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.625529695 (509h0m8.49661276s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.626165055 (509h0m8.49724809s) [noise=true , address=77.90.31.21:4890 , piece_id=NIGMBPZC3J6POFXVSNWZZEJUVKUQ77B7J7IENDZ2C2W3WMZAOD5A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.626720573 (509h0m8.497803618s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:09.940468211 (509h0m9.811551256s) [upload=446649]: Next returned nil segment Dec 26 20:29:09.940490244 (509h0m9.811573289s) [upload=446649]: Waiting for error group managing segments... Dec 26 20:29:10.283357488 (509h0m10.154440524s) [noise=true , address=64.49.13.139:4477 , piece_id=5QMCZNADOLXNYEBNY7EPOMHRPK5EANDLPEUJLU6SQ2QJQCRF66BQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.503143617 (509h0m10.374226653s) [noise=true , address=145.79.227.28:4737 , piece_id=VI474SQ2NRST5NR4YCNYHUNBYBIJSEDCSHQOM6PEF7NEK3NXNTNA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.657417840 (509h0m10.528500875s) [noise=true , address=145.79.227.16:4922 , piece_id=N5JA6JVCAXPKDGIURCNVAWJD6KRTGNRY6S3IZMD5EYVFA454GOKA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.679733744 (509h0m10.550816789s) [noise=true , address=77.90.31.21:4890 , piece_id=NIGMBPZC3J6POFXVSNWZZEJUVKUQ77B7J7IENDZ2C2W3WMZAOD5A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.683939000 (509h0m10.555022046s) [noise=true , address=77.90.31.14:4673 , piece_id=L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.685467862 (509h0m10.556550907s) [noise=true , address=145.79.227.22:4533 , piece_id=WVQ7YABIOYCCHCMZIRFXIUDKLFFC2KC5WJOABJC2F57QLRJK4H3A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.772110194 (509h0m10.643193229s) [noise=true , address=45.91.116.13:4165 , piece_id=YIRLJS6I6YR7WJRG5RLODG5HGZQJQW4V4PKCRIF2BXMWXAFRXTCA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.820042584 (509h0m10.691125629s) [noise=true , address=77.90.31.11:4570 , piece_id=Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.866890631 (509h0m10.737973666s) [noise=true , address=45.91.116.5:4314 , piece_id=XUHE2A4VE3U65LONDRNKMJILERSNLNDTJ7R2UPX5D67UI426UQAA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.999531381 (509h0m10.870614416s) [noise=true , address=145.79.227.43:4164 , piece_id=XCWV4H42H7J4PQAYNAEODGLSOXULESDU3JW6TJFDWXGIDFBRDGSA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.160014888 (509h0m11.031097933s) [noise=true , address=145.79.227.14:4868 , piece_id=3Q4CBS6OWB36IHRUBCRLUZIJQAS76QU5O2YKKQC6XZ5E7FBX4PGA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.463630931 (509h0m11.334713977s) [noise=true , address=145.79.227.34:4930 , piece_id=2BWAT4EMK4QB54A5DRYBSPEGAVDVJPELOLA6ZDROZA4YCK37RJXA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.508749725 (509h0m11.37983276s) [noise=true , address=145.79.227.40:4058 , piece_id=M4GW7XWWHESKY7CULLFIM7THSWLI7Z2FVWZL4YDCPNEXRJNU5TFA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.661052545 (509h0m11.53213558s) [noise=true , address=145.79.227.30:4784 , piece_id=TIVKXCMDHBQZ5LRKW2MQ3E53LWHQQCYPAZ27NYSRXJWGVEZCXJ3A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.661070500 (509h0m11.532153535s) [seg_pos={0 0}, upload=446649]: Segment reached optimal threshold of 14 pieces. Dec 26 20:29:11.661242827 (509h0m11.532325872s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji): protocol: expected piece hash; context canceled; EOF Dec 26 20:29:11.661250402 (509h0m11.532333447s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661295628 (509h0m11.532378663s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt): protocol: write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection; write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection Dec 26 20:29:11.661309314 (509h0m11.532392349s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661349922 (509h0m11.532432957s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1etuVthkhtZexvw7u88vxw2HGPaMwfYCbEgKWF4GPRh9FeWTwn): protocol: expected piece hash; context canceled; EOF Dec 26 20:29:11.661356895 (509h0m11.53243994s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661563188 (509h0m11.532646223s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125rPFZnyTxsa7RBQfAkXtkVCdUJUykq644EK4Ny6HQ1Q7NzrF3): protocol: read tcp 188.166.225.170:36656->45.91.116.6:4982: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 20:29:11.661570592 (509h0m11.532653637s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661611429 (509h0m11.532694474s) [seg_pos={0 0}, upload=446649]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:29:11.661671444 (509h0m11.532754479s) [upload=446649]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:29:11.661676032 (509h0m11.532759077s) [upload=446649]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:29:11.969641419 (509h0m11.840724454s) [upload=446649]: Done uploading segments... Dec 26 20:29:11.969660866 (509h0m11.840743911s) [upload=446649]: Upload finished. err: <nil> Duration: 29.841793053s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.............XX..X..X...........XXX.X.X....X..XXXX.X...X.X..] duration: 8.092562853s trace: https://jaeger.dev.storj.io/trace/13a2168592a1b0d1 FAIL (2025-12-26 20:30:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: LRDECELM2HNBG73DCMNXH4JGRGJY7GVTRGRIPPIVZFTPAVWDIXYA) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: MJ65ZTQBDAM6HODPAZ3FBBHQHA5XXVV43RELWC5O77EUSBLIUJ6Q) context deadline exceeded; piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: 2AMJYSPO5R42RNXQQOMJN7FE52LR555ORB3VBKM3CLNW5NMDLLYA) context deadline exceeded; piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: 7DEGRLZFNZUDMWOKBGQAUZ5EII7INYT3ATHBAFODRRIAAPTVNG4Q) context deadline exceeded ======LOGS====== 20:28:29.677935962 (508h59m29.549018997s) [upload=446641]: Starting upload Dec 26 20:28:29.678013801 (508h59m29.549096836s) [upload=446641]: Done starting upload Dec 26 20:28:29.678251913 (508h59m29.549334948s) [upload=446641]: Uploading segments... Dec 26 20:28:29.678448809 (508h59m29.549531844s) [upload=446641]: Got next segment. Inline: false Dec 26 20:28:29.678508312 (508h59m29.549591357s) [upload=446641]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 20:28:29.678513151 (508h59m29.549596186s) [upload=446641]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 20:28:29.925664959 (508h59m29.796748024s) [seg_pos={0 0}, upload=446641]: Begin upload segment... Dec 26 20:28:29.925733329 (508h59m29.796816364s) [seg_pos={0 0}, upload=446641]: Done begin upload segment. Dec 26 20:28:29.925739741 (508h59m29.796822776s) [noise=true , address=145.79.227.51:4409 , piece_id=3L6W4XDZBCO34CRQBE47E6NXVW4OEVMAD3CE2IONC322GKM6HCRQ, seg_pos={0 0}, upload=446641]: Uploading piece... Dec 26 20:28:29.925763677 (508h59m29.796846722s) [noise=true , address=45.91.116.4:4939 , piece_id=RGCRAOILK2NIAUEFZ3HHTF5L3MIMI65GHKHUMYL34AESHDZPOCEQ, seg_pos={0 0}, upload=446641]: Uploading piece... Dec 26 20:28:29.926292754 (508h59m29.79737579s) [noise=true , address=45.91.116.6:4976 , piece_id=7DEGRLZFNZUDMWOKBGQAUZ5EII7INYT3ATHBAFODRRIAAPTVNG4Q, seg_pos={0 0}, upload=446641]: Uploading piece... Dec 26 20:28:29.926671025 (508h59m29.79775406s) [noise=true , address=145.79.227.37:4567 , piece_id=KEKXRVZ6TL6UJBO7AYYIQSI5RDJDMVQXTNUINQGQQDWCXWNSPTNQ, seg_pos={0 0}, upload=446641]: Uploading piece... Dec 26 20:28:29.926962490 (508h59m29.798045525s) [noise=true , address=145.79.227.36:4517 , piece_id=ZTHEHA7KGDI3YZ3NKV65ZMLRYSZ3VJCHOUB75RZG2GQCNQHNXD5A, seg_pos={0 0}, upload=446641]: Uploading piece... Dec 26 20:28:29.927218868 (508h59m29.798301903s) [noise=true , address=145.79.227.46:4603 , piece_id=D4NLBH33S4MBTMYV4ULTHLL5DUT5ZLPYU2OFIRMZ2JBIXH6RFYXA, seg_pos={0 0}, upload=446641]: Uploading piece... Dec 26 20:28:29.927418408 (508h59m29.798501443s) [noise=true , address=45.91.116.10:4083 , piece_id=LRDECELM2HNBG73DCMNXH4JGRGJY7GVTRGRIPPIVZFTPAVWDIXYA, seg_pos={0 0}, upload=446641]: Uploading piece... Dec 26 20:28:29.927874998 (508h59m29.798958043s) [noise=true , address=77.90.31.15:4697 , piece_id=74W7KZ2D6H2AUKETVRYMCXVJ6HEBX66SCPI54SZLVSFINWVMJQJA, seg_pos={0 0}, upload=446641]: Uploading piece... Dec 26 20:28:29.928147736 (508h59m29.799230771s) [noise=true , address=145.79.227.2:4114 , piece_id=KPLGTMQAJN3JZHOZR2V2QL76VNUNZZRWNIQ7WPRX5EUPGJ2XOGSA, seg_pos={0 0}, upload=446641]: Uploading piece... Dec 26 20:28:29.928229733 (508h59m29.799312768s) [noise=true , address=45.91.116.8:4046 , piece_id=MJ65ZTQBDAM6HODPAZ3FBBHQHA5XXVV43RELWC5O77EUSBLIUJ6Q, seg_pos={0 0}, upload=446641]: Uploading piece... Dec 26 20:28:29.928491942 (508h59m29.799574977s) [noise=true , address=145.79.227.40:4051 , piece_id=MD7IDP6E37DEMNDSMAXTJDGNRRFZQNLLKHEHMYOJRTVWNNRLQ2BQ, seg_pos={0 0}, upload=446641]: Uploading piece... Dec 26 20:28:29.928799157 (508h59m29.799882192s) [noise=true , address=145.79.227.3:4151 , piece_id=TPR72DFEKFADGYHCAGQWJ4RJTJDNXXB5KOPKPC2HA3EFRJQH5OWQ, seg_pos={0 0}, upload=446641]: Uploading piece... Dec 26 20:28:29.929092376 (508h59m29.800175421s) [noise=true , address=145.79.227.45:4229 , piece_id=3T35SGG2KV4DYKBBIW6F5JEQMB6NL7YOMRB6R5ADIQJ3WHYQN5VQ, seg_pos={0 0}, upload=446641]: Uploading piece... Dec 26 20:28:29.929469754 (508h59m29.800552789s) [noise=true , address=145.79.227.21:4475 , piece_id=FM2HSCM6RJUESMMXEO4VMSRTTA2X4ER7JSWQ63O5BNDMTM2L6PEA, seg_pos={0 0}, upload=446641]: Uploading piece... Dec 26 20:28:29.929936122 (508h59m29.801019157s) [noise=true , address=77.90.31.13:4629 , piece_id=2AMJYSPO5R42RNXQQOMJN7FE52LR555ORB3VBKM3CLNW5NMDLLYA, seg_pos={0 0}, upload=446641]: Uploading piece... Dec 26 20:28:29.930013139 (508h59m29.801096174s) [noise=true , address=145.79.227.25:4618 , piece_id=UU6KFIQCYNZEWWMHSZ4NQ7TOZYA7N4AZMI4VN7RZAQCE4VAAMRFA, seg_pos={0 0}, upload=446641]: Uploading piece... Dec 26 20:28:31.349080054 (508h59m31.220163089s) [noise=true , address=77.90.31.15:4697 , piece_id=74W7KZ2D6H2AUKETVRYMCXVJ6HEBX66SCPI54SZLVSFINWVMJQJA, seg_pos={0 0}, upload=446641]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: storage node overloaded, request limit: 2000 Dec 26 20:28:31.603377739 (508h59m31.474460774s) [upload=446641]: Next returned nil segment Dec 26 20:28:31.603400412 (508h59m31.474483447s) [upload=446641]: Waiting for error group managing segments... Dec 26 20:28:31.928179705 (508h59m31.79926274s) [noise=true , address=145.79.227.25:4618 , piece_id=UU6KFIQCYNZEWWMHSZ4NQ7TOZYA7N4AZMI4VN7RZAQCE4VAAMRFA, seg_pos={0 0}, upload=446641]: Done uploading piece. err: <nil> Dec 26 20:28:31.930119068 (508h59m31.801202103s) [noise=true , address=145.79.227.21:4475 , piece_id=FM2HSCM6RJUESMMXEO4VMSRTTA2X4ER7JSWQ63O5BNDMTM2L6PEA, seg_pos={0 0}, upload=446641]: Done uploading piece. err: <nil> Dec 26 20:28:31.938003371 (508h59m31.809086416s) [noise=true , address=145.79.227.46:4603 , piece_id=D4NLBH33S4MBTMYV4ULTHLL5DUT5ZLPYU2OFIRMZ2JBIXH6RFYXA, seg_pos={0 0}, upload=446641]: Done uploading piece. err: <nil> Dec 26 20:28:31.940924854 (508h59m31.812007889s) [noise=true , address=145.79.227.45:4229 , piece_id=3T35SGG2KV4DYKBBIW6F5JEQMB6NL7YOMRB6R5ADIQJ3WHYQN5VQ, seg_pos={0 0}, upload=446641]: Done uploading piece. err: <nil> Dec 26 20:28:31.942105522 (508h59m31.813188557s) [noise=true , address=145.79.227.36:4517 , piece_id=ZTHEHA7KGDI3YZ3NKV65ZMLRYSZ3VJCHOUB75RZG2GQCNQHNXD5A, seg_pos={0 0}, upload=446641]: Done uploading piece. err: <nil> Dec 26 20:28:31.964426635 (508h59m31.83550967s) [noise=true , address=145.79.227.37:4567 , piece_id=KEKXRVZ6TL6UJBO7AYYIQSI5RDJDMVQXTNUINQGQQDWCXWNSPTNQ, seg_pos={0 0}, upload=446641]: Done uploading piece. err: <nil> Dec 26 20:28:32.158955319 (508h59m32.030038364s) [noise=true , address=45.91.116.4:4939 , piece_id=RGCRAOILK2NIAUEFZ3HHTF5L3MIMI65GHKHUMYL34AESHDZPOCEQ, seg_pos={0 0}, upload=446641]: Done uploading piece. err: <nil> Dec 26 20:28:32.161331084 (508h59m32.032414129s) [noise=true , address=45.91.116.8:4046 , piece_id=MJ65ZTQBDAM6HODPAZ3FBBHQHA5XXVV43RELWC5O77EUSBLIUJ6Q, seg_pos={0 0}, upload=446641]: Done uploading piece. err: <nil> Dec 26 20:28:32.235774115 (508h59m32.10685715s) [noise=true , address=45.91.116.6:4976 , piece_id=7DEGRLZFNZUDMWOKBGQAUZ5EII7INYT3ATHBAFODRRIAAPTVNG4Q, seg_pos={0 0}, upload=446641]: Done uploading piece. err: <nil> Dec 26 20:28:32.246433393 (508h59m32.117516428s) [noise=true , address=77.90.31.13:4629 , piece_id=2AMJYSPO5R42RNXQQOMJN7FE52LR555ORB3VBKM3CLNW5NMDLLYA, seg_pos={0 0}, upload=446641]: Done uploading piece. err: <nil> Dec 26 20:28:32.394498402 (508h59m32.265581447s) [noise=true , address=45.91.116.10:4083 , piece_id=LRDECELM2HNBG73DCMNXH4JGRGJY7GVTRGRIPPIVZFTPAVWDIXYA, seg_pos={0 0}, upload=446641]: Done uploading piece. err: <nil> Dec 26 20:28:32.471701779 (508h59m32.342784834s) [noise=true , address=145.79.227.3:4151 , piece_id=TPR72DFEKFADGYHCAGQWJ4RJTJDNXXB5KOPKPC2HA3EFRJQH5OWQ, seg_pos={0 0}, upload=446641]: Done uploading piece. err: <nil> Dec 26 20:28:33.272979594 (508h59m33.144062629s) [noise=true , address=145.79.227.40:4051 , piece_id=MD7IDP6E37DEMNDSMAXTJDGNRRFZQNLLKHEHMYOJRTVWNNRLQ2BQ, seg_pos={0 0}, upload=446641]: Done uploading piece. err: <nil> Dec 26 20:28:33.273003128 (508h59m33.144086173s) [seg_pos={0 0}, upload=446641]: Segment reached optimal threshold of 13 pieces. Dec 26 20:28:33.273210062 (508h59m33.144293097s) [noise=true , address=145.79.227.2:4114 , piece_id=KPLGTMQAJN3JZHOZR2V2QL76VNUNZZRWNIQ7WPRX5EUPGJ2XOGSA, seg_pos={0 0}, upload=446641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN): protocol: write tcp 188.166.225.170:35140->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:35140->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35140->145.79.227.2:4114: use of closed network connection Dec 26 20:28:33.273218258 (508h59m33.144301323s) [noise=true , address=145.79.227.2:4114 , piece_id=KPLGTMQAJN3JZHOZR2V2QL76VNUNZZRWNIQ7WPRX5EUPGJ2XOGSA, seg_pos={0 0}, upload=446641]: Long tail cancellation due to optimal threshold Dec 26 20:28:33.273418729 (508h59m33.144501764s) [noise=true , address=145.79.227.51:4409 , piece_id=3L6W4XDZBCO34CRQBE47E6NXVW4OEVMAD3CE2IONC322GKM6HCRQ, seg_pos={0 0}, upload=446641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UoCGA4FjE4yADPXL1oifSPJssPgSGh81REZHwGv2eu1Y1kEaj): protocol: write tcp 188.166.225.170:44904->145.79.227.51:4409: use of closed network connection; write tcp 188.166.225.170:44904->145.79.227.51:4409: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44904->145.79.227.51:4409: use of closed network connection Dec 26 20:28:33.273454016 (508h59m33.144537051s) [noise=true , address=145.79.227.51:4409 , piece_id=3L6W4XDZBCO34CRQBE47E6NXVW4OEVMAD3CE2IONC322GKM6HCRQ, seg_pos={0 0}, upload=446641]: Long tail cancellation due to optimal threshold Dec 26 20:28:33.273476710 (508h59m33.144559745s) [seg_pos={0 0}, upload=446641]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 20:28:33.273587150 (508h59m33.144670185s) [upload=446641]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:28:33.273604383 (508h59m33.144687418s) [upload=446641]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:28:33.579545726 (508h59m33.450628771s) [upload=446641]: Done uploading segments... Dec 26 20:28:33.579573469 (508h59m33.450656514s) [upload=446641]: Upload finished. err: <nil> Duration: 8.09259314s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.X..........................................X..............X] duration: 1m0.003526032s trace: https://jaeger.dev.storj.io/trace/5bdd2de9914a125e FAIL (2025-12-26 20:30:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh, Piece ID: L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q) context deadline exceeded; piecestore: (Node ID: 127RBQsp1SwQU6yWh6Y48QF3QCzeJxEGjob8gzF3LRutRZXA6A9, Piece ID: XUHE2A4VE3U65LONDRNKMJILERSNLNDTJ7R2UPX5D67UI426UQAA) context deadline exceeded; piecestore: (Node ID: 12NFW5tt5R84RaTvY9Up75kNjZ7bmLAgUun41KLdW9Q7eMv27Sq, Piece ID: Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA) context deadline exceeded ======LOGS====== , piece_id=M4GW7XWWHESKY7CULLFIM7THSWLI7Z2FVWZL4YDCPNEXRJNU5TFA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.621775106 (509h0m8.492858151s) [noise=true , address=145.79.227.22:4533 , piece_id=WVQ7YABIOYCCHCMZIRFXIUDKLFFC2KC5WJOABJC2F57QLRJK4H3A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.621846552 (509h0m8.492929597s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.622555641 (509h0m8.493638687s) [noise=true , address=145.79.227.43:4164 , piece_id=XCWV4H42H7J4PQAYNAEODGLSOXULESDU3JW6TJFDWXGIDFBRDGSA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.622948811 (509h0m8.494031846s) [noise=true , address=145.79.227.16:4922 , piece_id=N5JA6JVCAXPKDGIURCNVAWJD6KRTGNRY6S3IZMD5EYVFA454GOKA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623044693 (509h0m8.494127728s) [noise=true , address=77.90.31.11:4570 , piece_id=Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623366476 (509h0m8.494449521s) [noise=true , address=45.91.116.13:4165 , piece_id=YIRLJS6I6YR7WJRG5RLODG5HGZQJQW4V4PKCRIF2BXMWXAFRXTCA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623672749 (509h0m8.494755784s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.623792828 (509h0m8.494875863s) [noise=true , address=145.79.227.28:4737 , piece_id=VI474SQ2NRST5NR4YCNYHUNBYBIJSEDCSHQOM6PEF7NEK3NXNTNA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624190374 (509h0m8.495273419s) [noise=true , address=145.79.227.14:4868 , piece_id=3Q4CBS6OWB36IHRUBCRLUZIJQAS76QU5O2YKKQC6XZ5E7FBX4PGA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624328067 (509h0m8.495411112s) [noise=true , address=77.90.31.14:4673 , piece_id=L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624580047 (509h0m8.495663092s) [noise=true , address=145.79.227.30:4784 , piece_id=TIVKXCMDHBQZ5LRKW2MQ3E53LWHQQCYPAZ27NYSRXJWGVEZCXJ3A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.624915285 (509h0m8.49599833s) [noise=true , address=145.79.227.34:4930 , piece_id=2BWAT4EMK4QB54A5DRYBSPEGAVDVJPELOLA6ZDROZA4YCK37RJXA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.625529695 (509h0m8.49661276s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.626165055 (509h0m8.49724809s) [noise=true , address=77.90.31.21:4890 , piece_id=NIGMBPZC3J6POFXVSNWZZEJUVKUQ77B7J7IENDZ2C2W3WMZAOD5A, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:08.626720573 (509h0m8.497803618s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Uploading piece... Dec 26 20:29:09.940468211 (509h0m9.811551256s) [upload=446649]: Next returned nil segment Dec 26 20:29:09.940490244 (509h0m9.811573289s) [upload=446649]: Waiting for error group managing segments... Dec 26 20:29:10.283357488 (509h0m10.154440524s) [noise=true , address=64.49.13.139:4477 , piece_id=5QMCZNADOLXNYEBNY7EPOMHRPK5EANDLPEUJLU6SQ2QJQCRF66BQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.503143617 (509h0m10.374226653s) [noise=true , address=145.79.227.28:4737 , piece_id=VI474SQ2NRST5NR4YCNYHUNBYBIJSEDCSHQOM6PEF7NEK3NXNTNA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.657417840 (509h0m10.528500875s) [noise=true , address=145.79.227.16:4922 , piece_id=N5JA6JVCAXPKDGIURCNVAWJD6KRTGNRY6S3IZMD5EYVFA454GOKA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.679733744 (509h0m10.550816789s) [noise=true , address=77.90.31.21:4890 , piece_id=NIGMBPZC3J6POFXVSNWZZEJUVKUQ77B7J7IENDZ2C2W3WMZAOD5A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.683939000 (509h0m10.555022046s) [noise=true , address=77.90.31.14:4673 , piece_id=L5GDJHAB4CC5UKOPAZQNV4ZGQRQBHZTPWYDHZ4QGMVROQDCY365Q, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.685467862 (509h0m10.556550907s) [noise=true , address=145.79.227.22:4533 , piece_id=WVQ7YABIOYCCHCMZIRFXIUDKLFFC2KC5WJOABJC2F57QLRJK4H3A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.772110194 (509h0m10.643193229s) [noise=true , address=45.91.116.13:4165 , piece_id=YIRLJS6I6YR7WJRG5RLODG5HGZQJQW4V4PKCRIF2BXMWXAFRXTCA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.820042584 (509h0m10.691125629s) [noise=true , address=77.90.31.11:4570 , piece_id=Q2A276MRXPVHIGKCSRCHWIM325QI32YGQMU24XZONU2SOVCZIXLA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.866890631 (509h0m10.737973666s) [noise=true , address=45.91.116.5:4314 , piece_id=XUHE2A4VE3U65LONDRNKMJILERSNLNDTJ7R2UPX5D67UI426UQAA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:10.999531381 (509h0m10.870614416s) [noise=true , address=145.79.227.43:4164 , piece_id=XCWV4H42H7J4PQAYNAEODGLSOXULESDU3JW6TJFDWXGIDFBRDGSA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.160014888 (509h0m11.031097933s) [noise=true , address=145.79.227.14:4868 , piece_id=3Q4CBS6OWB36IHRUBCRLUZIJQAS76QU5O2YKKQC6XZ5E7FBX4PGA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.463630931 (509h0m11.334713977s) [noise=true , address=145.79.227.34:4930 , piece_id=2BWAT4EMK4QB54A5DRYBSPEGAVDVJPELOLA6ZDROZA4YCK37RJXA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.508749725 (509h0m11.37983276s) [noise=true , address=145.79.227.40:4058 , piece_id=M4GW7XWWHESKY7CULLFIM7THSWLI7Z2FVWZL4YDCPNEXRJNU5TFA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.661052545 (509h0m11.53213558s) [noise=true , address=145.79.227.30:4784 , piece_id=TIVKXCMDHBQZ5LRKW2MQ3E53LWHQQCYPAZ27NYSRXJWGVEZCXJ3A, seg_pos={0 0}, upload=446649]: Done uploading piece. err: <nil> Dec 26 20:29:11.661070500 (509h0m11.532153535s) [seg_pos={0 0}, upload=446649]: Segment reached optimal threshold of 14 pieces. Dec 26 20:29:11.661242827 (509h0m11.532325872s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji): protocol: expected piece hash; context canceled; EOF Dec 26 20:29:11.661250402 (509h0m11.532333447s) [noise=true , address=145.79.227.35:4490 , piece_id=6WNSOTMXYK2X3WAVUZMR6RMDN6G7ZS5NMTGKEETAQS63ECFSNVYQ, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661295628 (509h0m11.532378663s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt): protocol: write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection; write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52126->145.79.227.3:4180: use of closed network connection Dec 26 20:29:11.661309314 (509h0m11.532392349s) [noise=true , address=145.79.227.3:4180 , piece_id=AVJVKM4SPXJT5WJXYUMRKVOK3EZL6LII2CIN6YPTFQHEV5DAD5GA, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661349922 (509h0m11.532432957s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1etuVthkhtZexvw7u88vxw2HGPaMwfYCbEgKWF4GPRh9FeWTwn): protocol: expected piece hash; context canceled; EOF Dec 26 20:29:11.661356895 (509h0m11.53243994s) [noise=true , address=77.90.31.15:4721 , piece_id=C3HWITDNPUPHD3X3IGTSCZE4NQ3BM2GXJR6BXZEWPBXBJJD2WQLQ, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661563188 (509h0m11.532646223s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125rPFZnyTxsa7RBQfAkXtkVCdUJUykq644EK4Ny6HQ1Q7NzrF3): protocol: read tcp 188.166.225.170:36656->45.91.116.6:4982: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 20:29:11.661570592 (509h0m11.532653637s) [noise=true , address=45.91.116.6:4982 , piece_id=55SWRPB7AM6UOKC6LF6WWZZCBXEML5EPQM4UMIHMFLRPQZLJ7F5Q, seg_pos={0 0}, upload=446649]: Long tail cancellation due to optimal threshold Dec 26 20:29:11.661611429 (509h0m11.532694474s) [seg_pos={0 0}, upload=446649]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:29:11.661671444 (509h0m11.532754479s) [upload=446649]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:29:11.661676032 (509h0m11.532759077s) [upload=446649]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:29:11.969641419 (509h0m11.840724454s) [upload=446649]: Done uploading segments... Dec 26 20:29:11.969660866 (509h0m11.840743911s) [upload=446649]: Upload finished. err: <nil> Duration: 1m0.003553595s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [..............XX..X..X...........XXX.X.X....X..XXXX.X...X.X.] duration: 6.210313522s trace: https://jaeger.dev.storj.io/trace/13a2168592a1b0d1 FAIL (2025-12-26 20:29:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: LRDECELM2HNBG73DCMNXH4JGRGJY7GVTRGRIPPIVZFTPAVWDIXYA) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: MJ65ZTQBDAM6HODPAZ3FBBHQHA5XXVV43RELWC5O77EUSBLIUJ6Q) context deadline exceeded; piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: 2AMJYSPO5R42RNXQQOMJN7FE52LR555ORB3VBKM3CLNW5NMDLLYA) context deadline exceeded; piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: 7DEGRLZFNZUDMWOKBGQAUZ5EII7INYT3ATHBAFODRRIAAPTVNG4Q) context deadline exceeded ======LOGS====== 20:28:29.677935962 (508h59m29.549018997s) [upload=446641]: Starting upload Dec 26 20:28:29.678013801 (508h59m29.549096836s) [upload=446641]: Done starting upload Dec 26 20:28:29.678251913 (508h59m29.549334948s) [upload=446641]: Uploading segments... Dec 26 20:28:29.678448809 (508h59m29.549531844s) [upload=446641]: Got next segment. Inline: false Dec 26 20:28:29.678508312 (508h59m29.549591357s) [upload=446641]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 20:28:29.678513151 (508h59m29.549596186s) [upload=446641]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 20:28:29.925664959 (508h59m29.796748024s) [seg_pos={0 0}, upload=446641]: Begin upload segment... Dec 26 20:28:29.925733329 (508h59m29.796816364s) [seg_pos={0 0}, upload=446641]: Done begin upload segment. Dec 26 20:28:29.925739741 (508h59m29.796822776s) [noise=true , address=145.79.227.51:4409 , piece_id=3L6W4XDZBCO34CRQBE47E6NXVW4OEVMAD3CE2IONC322GKM6HCRQ, seg_pos={0 0}, upload=446641]: Uploading piece... Dec 26 20:28:29.925763677 (508h59m29.796846722s) [noise=true , address=45.91.116.4:4939 , piece_id=RGCRAOILK2NIAUEFZ3HHTF5L3MIMI65GHKHUMYL34AESHDZPOCEQ, seg_pos={0 0}, upload=446641]: Uploading piece... Dec 26 20:28:29.926292754 (508h59m29.79737579s) [noise=true , address=45.91.116.6:4976 , piece_id=7DEGRLZFNZUDMWOKBGQAUZ5EII7INYT3ATHBAFODRRIAAPTVNG4Q, seg_pos={0 0}, upload=446641]: Uploading piece... Dec 26 20:28:29.926671025 (508h59m29.79775406s) [noise=true , address=145.79.227.37:4567 , piece_id=KEKXRVZ6TL6UJBO7AYYIQSI5RDJDMVQXTNUINQGQQDWCXWNSPTNQ, seg_pos={0 0}, upload=446641]: Uploading piece... Dec 26 20:28:29.926962490 (508h59m29.798045525s) [noise=true , address=145.79.227.36:4517 , piece_id=ZTHEHA7KGDI3YZ3NKV65ZMLRYSZ3VJCHOUB75RZG2GQCNQHNXD5A, seg_pos={0 0}, upload=446641]: Uploading piece... Dec 26 20:28:29.927218868 (508h59m29.798301903s) [noise=true , address=145.79.227.46:4603 , piece_id=D4NLBH33S4MBTMYV4ULTHLL5DUT5ZLPYU2OFIRMZ2JBIXH6RFYXA, seg_pos={0 0}, upload=446641]: Uploading piece... Dec 26 20:28:29.927418408 (508h59m29.798501443s) [noise=true , address=45.91.116.10:4083 , piece_id=LRDECELM2HNBG73DCMNXH4JGRGJY7GVTRGRIPPIVZFTPAVWDIXYA, seg_pos={0 0}, upload=446641]: Uploading piece... Dec 26 20:28:29.927874998 (508h59m29.798958043s) [noise=true , address=77.90.31.15:4697 , piece_id=74W7KZ2D6H2AUKETVRYMCXVJ6HEBX66SCPI54SZLVSFINWVMJQJA, seg_pos={0 0}, upload=446641]: Uploading piece... Dec 26 20:28:29.928147736 (508h59m29.799230771s) [noise=true , address=145.79.227.2:4114 , piece_id=KPLGTMQAJN3JZHOZR2V2QL76VNUNZZRWNIQ7WPRX5EUPGJ2XOGSA, seg_pos={0 0}, upload=446641]: Uploading piece... Dec 26 20:28:29.928229733 (508h59m29.799312768s) [noise=true , address=45.91.116.8:4046 , piece_id=MJ65ZTQBDAM6HODPAZ3FBBHQHA5XXVV43RELWC5O77EUSBLIUJ6Q, seg_pos={0 0}, upload=446641]: Uploading piece... Dec 26 20:28:29.928491942 (508h59m29.799574977s) [noise=true , address=145.79.227.40:4051 , piece_id=MD7IDP6E37DEMNDSMAXTJDGNRRFZQNLLKHEHMYOJRTVWNNRLQ2BQ, seg_pos={0 0}, upload=446641]: Uploading piece... Dec 26 20:28:29.928799157 (508h59m29.799882192s) [noise=true , address=145.79.227.3:4151 , piece_id=TPR72DFEKFADGYHCAGQWJ4RJTJDNXXB5KOPKPC2HA3EFRJQH5OWQ, seg_pos={0 0}, upload=446641]: Uploading piece... Dec 26 20:28:29.929092376 (508h59m29.800175421s) [noise=true , address=145.79.227.45:4229 , piece_id=3T35SGG2KV4DYKBBIW6F5JEQMB6NL7YOMRB6R5ADIQJ3WHYQN5VQ, seg_pos={0 0}, upload=446641]: Uploading piece... Dec 26 20:28:29.929469754 (508h59m29.800552789s) [noise=true , address=145.79.227.21:4475 , piece_id=FM2HSCM6RJUESMMXEO4VMSRTTA2X4ER7JSWQ63O5BNDMTM2L6PEA, seg_pos={0 0}, upload=446641]: Uploading piece... Dec 26 20:28:29.929936122 (508h59m29.801019157s) [noise=true , address=77.90.31.13:4629 , piece_id=2AMJYSPO5R42RNXQQOMJN7FE52LR555ORB3VBKM3CLNW5NMDLLYA, seg_pos={0 0}, upload=446641]: Uploading piece... Dec 26 20:28:29.930013139 (508h59m29.801096174s) [noise=true , address=145.79.227.25:4618 , piece_id=UU6KFIQCYNZEWWMHSZ4NQ7TOZYA7N4AZMI4VN7RZAQCE4VAAMRFA, seg_pos={0 0}, upload=446641]: Uploading piece... Dec 26 20:28:31.349080054 (508h59m31.220163089s) [noise=true , address=77.90.31.15:4697 , piece_id=74W7KZ2D6H2AUKETVRYMCXVJ6HEBX66SCPI54SZLVSFINWVMJQJA, seg_pos={0 0}, upload=446641]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: storage node overloaded, request limit: 2000 Dec 26 20:28:31.603377739 (508h59m31.474460774s) [upload=446641]: Next returned nil segment Dec 26 20:28:31.603400412 (508h59m31.474483447s) [upload=446641]: Waiting for error group managing segments... Dec 26 20:28:31.928179705 (508h59m31.79926274s) [noise=true , address=145.79.227.25:4618 , piece_id=UU6KFIQCYNZEWWMHSZ4NQ7TOZYA7N4AZMI4VN7RZAQCE4VAAMRFA, seg_pos={0 0}, upload=446641]: Done uploading piece. err: <nil> Dec 26 20:28:31.930119068 (508h59m31.801202103s) [noise=true , address=145.79.227.21:4475 , piece_id=FM2HSCM6RJUESMMXEO4VMSRTTA2X4ER7JSWQ63O5BNDMTM2L6PEA, seg_pos={0 0}, upload=446641]: Done uploading piece. err: <nil> Dec 26 20:28:31.938003371 (508h59m31.809086416s) [noise=true , address=145.79.227.46:4603 , piece_id=D4NLBH33S4MBTMYV4ULTHLL5DUT5ZLPYU2OFIRMZ2JBIXH6RFYXA, seg_pos={0 0}, upload=446641]: Done uploading piece. err: <nil> Dec 26 20:28:31.940924854 (508h59m31.812007889s) [noise=true , address=145.79.227.45:4229 , piece_id=3T35SGG2KV4DYKBBIW6F5JEQMB6NL7YOMRB6R5ADIQJ3WHYQN5VQ, seg_pos={0 0}, upload=446641]: Done uploading piece. err: <nil> Dec 26 20:28:31.942105522 (508h59m31.813188557s) [noise=true , address=145.79.227.36:4517 , piece_id=ZTHEHA7KGDI3YZ3NKV65ZMLRYSZ3VJCHOUB75RZG2GQCNQHNXD5A, seg_pos={0 0}, upload=446641]: Done uploading piece. err: <nil> Dec 26 20:28:31.964426635 (508h59m31.83550967s) [noise=true , address=145.79.227.37:4567 , piece_id=KEKXRVZ6TL6UJBO7AYYIQSI5RDJDMVQXTNUINQGQQDWCXWNSPTNQ, seg_pos={0 0}, upload=446641]: Done uploading piece. err: <nil> Dec 26 20:28:32.158955319 (508h59m32.030038364s) [noise=true , address=45.91.116.4:4939 , piece_id=RGCRAOILK2NIAUEFZ3HHTF5L3MIMI65GHKHUMYL34AESHDZPOCEQ, seg_pos={0 0}, upload=446641]: Done uploading piece. err: <nil> Dec 26 20:28:32.161331084 (508h59m32.032414129s) [noise=true , address=45.91.116.8:4046 , piece_id=MJ65ZTQBDAM6HODPAZ3FBBHQHA5XXVV43RELWC5O77EUSBLIUJ6Q, seg_pos={0 0}, upload=446641]: Done uploading piece. err: <nil> Dec 26 20:28:32.235774115 (508h59m32.10685715s) [noise=true , address=45.91.116.6:4976 , piece_id=7DEGRLZFNZUDMWOKBGQAUZ5EII7INYT3ATHBAFODRRIAAPTVNG4Q, seg_pos={0 0}, upload=446641]: Done uploading piece. err: <nil> Dec 26 20:28:32.246433393 (508h59m32.117516428s) [noise=true , address=77.90.31.13:4629 , piece_id=2AMJYSPO5R42RNXQQOMJN7FE52LR555ORB3VBKM3CLNW5NMDLLYA, seg_pos={0 0}, upload=446641]: Done uploading piece. err: <nil> Dec 26 20:28:32.394498402 (508h59m32.265581447s) [noise=true , address=45.91.116.10:4083 , piece_id=LRDECELM2HNBG73DCMNXH4JGRGJY7GVTRGRIPPIVZFTPAVWDIXYA, seg_pos={0 0}, upload=446641]: Done uploading piece. err: <nil> Dec 26 20:28:32.471701779 (508h59m32.342784834s) [noise=true , address=145.79.227.3:4151 , piece_id=TPR72DFEKFADGYHCAGQWJ4RJTJDNXXB5KOPKPC2HA3EFRJQH5OWQ, seg_pos={0 0}, upload=446641]: Done uploading piece. err: <nil> Dec 26 20:28:33.272979594 (508h59m33.144062629s) [noise=true , address=145.79.227.40:4051 , piece_id=MD7IDP6E37DEMNDSMAXTJDGNRRFZQNLLKHEHMYOJRTVWNNRLQ2BQ, seg_pos={0 0}, upload=446641]: Done uploading piece. err: <nil> Dec 26 20:28:33.273003128 (508h59m33.144086173s) [seg_pos={0 0}, upload=446641]: Segment reached optimal threshold of 13 pieces. Dec 26 20:28:33.273210062 (508h59m33.144293097s) [noise=true , address=145.79.227.2:4114 , piece_id=KPLGTMQAJN3JZHOZR2V2QL76VNUNZZRWNIQ7WPRX5EUPGJ2XOGSA, seg_pos={0 0}, upload=446641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN): protocol: write tcp 188.166.225.170:35140->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:35140->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35140->145.79.227.2:4114: use of closed network connection Dec 26 20:28:33.273218258 (508h59m33.144301323s) [noise=true , address=145.79.227.2:4114 , piece_id=KPLGTMQAJN3JZHOZR2V2QL76VNUNZZRWNIQ7WPRX5EUPGJ2XOGSA, seg_pos={0 0}, upload=446641]: Long tail cancellation due to optimal threshold Dec 26 20:28:33.273418729 (508h59m33.144501764s) [noise=true , address=145.79.227.51:4409 , piece_id=3L6W4XDZBCO34CRQBE47E6NXVW4OEVMAD3CE2IONC322GKM6HCRQ, seg_pos={0 0}, upload=446641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UoCGA4FjE4yADPXL1oifSPJssPgSGh81REZHwGv2eu1Y1kEaj): protocol: write tcp 188.166.225.170:44904->145.79.227.51:4409: use of closed network connection; write tcp 188.166.225.170:44904->145.79.227.51:4409: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44904->145.79.227.51:4409: use of closed network connection Dec 26 20:28:33.273454016 (508h59m33.144537051s) [noise=true , address=145.79.227.51:4409 , piece_id=3L6W4XDZBCO34CRQBE47E6NXVW4OEVMAD3CE2IONC322GKM6HCRQ, seg_pos={0 0}, upload=446641]: Long tail cancellation due to optimal threshold Dec 26 20:28:33.273476710 (508h59m33.144559745s) [seg_pos={0 0}, upload=446641]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 20:28:33.273587150 (508h59m33.144670185s) [upload=446641]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:28:33.273604383 (508h59m33.144687418s) [upload=446641]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:28:33.579545726 (508h59m33.450628771s) [upload=446641]: Done uploading segments... Dec 26 20:28:33.579573469 (508h59m33.450656514s) [upload=446641]: Upload finished. err: <nil> Duration: 6.21034358s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [X..............XX..X..X...........XXX.X.X....X..XXXX.X...X.X] duration: 30.001935707s trace: https://jaeger.dev.storj.io/trace/13a2168592a1b0d1 FAIL (2025-12-26 20:28:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: LRDECELM2HNBG73DCMNXH4JGRGJY7GVTRGRIPPIVZFTPAVWDIXYA) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: MJ65ZTQBDAM6HODPAZ3FBBHQHA5XXVV43RELWC5O77EUSBLIUJ6Q) context deadline exceeded; piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: 2AMJYSPO5R42RNXQQOMJN7FE52LR555ORB3VBKM3CLNW5NMDLLYA) context deadline exceeded; piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: 7DEGRLZFNZUDMWOKBGQAUZ5EII7INYT3ATHBAFODRRIAAPTVNG4Q) context deadline exceeded ======LOGS====== 20:28:29.677935962 (508h59m29.549018997s) [upload=446641]: Starting upload Dec 26 20:28:29.678013801 (508h59m29.549096836s) [upload=446641]: Done starting upload Dec 26 20:28:29.678251913 (508h59m29.549334948s) [upload=446641]: Uploading segments... Dec 26 20:28:29.678448809 (508h59m29.549531844s) [upload=446641]: Got next segment. Inline: false Dec 26 20:28:29.678508312 (508h59m29.549591357s) [upload=446641]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 20:28:29.678513151 (508h59m29.549596186s) [upload=446641]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 20:28:29.925664959 (508h59m29.796748024s) [seg_pos={0 0}, upload=446641]: Begin upload segment... Dec 26 20:28:29.925733329 (508h59m29.796816364s) [seg_pos={0 0}, upload=446641]: Done begin upload segment. Dec 26 20:28:29.925739741 (508h59m29.796822776s) [noise=true , address=145.79.227.51:4409 , piece_id=3L6W4XDZBCO34CRQBE47E6NXVW4OEVMAD3CE2IONC322GKM6HCRQ, seg_pos={0 0}, upload=446641]: Uploading piece... Dec 26 20:28:29.925763677 (508h59m29.796846722s) [noise=true , address=45.91.116.4:4939 , piece_id=RGCRAOILK2NIAUEFZ3HHTF5L3MIMI65GHKHUMYL34AESHDZPOCEQ, seg_pos={0 0}, upload=446641]: Uploading piece... Dec 26 20:28:29.926292754 (508h59m29.79737579s) [noise=true , address=45.91.116.6:4976 , piece_id=7DEGRLZFNZUDMWOKBGQAUZ5EII7INYT3ATHBAFODRRIAAPTVNG4Q, seg_pos={0 0}, upload=446641]: Uploading piece... Dec 26 20:28:29.926671025 (508h59m29.79775406s) [noise=true , address=145.79.227.37:4567 , piece_id=KEKXRVZ6TL6UJBO7AYYIQSI5RDJDMVQXTNUINQGQQDWCXWNSPTNQ, seg_pos={0 0}, upload=446641]: Uploading piece... Dec 26 20:28:29.926962490 (508h59m29.798045525s) [noise=true , address=145.79.227.36:4517 , piece_id=ZTHEHA7KGDI3YZ3NKV65ZMLRYSZ3VJCHOUB75RZG2GQCNQHNXD5A, seg_pos={0 0}, upload=446641]: Uploading piece... Dec 26 20:28:29.927218868 (508h59m29.798301903s) [noise=true , address=145.79.227.46:4603 , piece_id=D4NLBH33S4MBTMYV4ULTHLL5DUT5ZLPYU2OFIRMZ2JBIXH6RFYXA, seg_pos={0 0}, upload=446641]: Uploading piece... Dec 26 20:28:29.927418408 (508h59m29.798501443s) [noise=true , address=45.91.116.10:4083 , piece_id=LRDECELM2HNBG73DCMNXH4JGRGJY7GVTRGRIPPIVZFTPAVWDIXYA, seg_pos={0 0}, upload=446641]: Uploading piece... Dec 26 20:28:29.927874998 (508h59m29.798958043s) [noise=true , address=77.90.31.15:4697 , piece_id=74W7KZ2D6H2AUKETVRYMCXVJ6HEBX66SCPI54SZLVSFINWVMJQJA, seg_pos={0 0}, upload=446641]: Uploading piece... Dec 26 20:28:29.928147736 (508h59m29.799230771s) [noise=true , address=145.79.227.2:4114 , piece_id=KPLGTMQAJN3JZHOZR2V2QL76VNUNZZRWNIQ7WPRX5EUPGJ2XOGSA, seg_pos={0 0}, upload=446641]: Uploading piece... Dec 26 20:28:29.928229733 (508h59m29.799312768s) [noise=true , address=45.91.116.8:4046 , piece_id=MJ65ZTQBDAM6HODPAZ3FBBHQHA5XXVV43RELWC5O77EUSBLIUJ6Q, seg_pos={0 0}, upload=446641]: Uploading piece... Dec 26 20:28:29.928491942 (508h59m29.799574977s) [noise=true , address=145.79.227.40:4051 , piece_id=MD7IDP6E37DEMNDSMAXTJDGNRRFZQNLLKHEHMYOJRTVWNNRLQ2BQ, seg_pos={0 0}, upload=446641]: Uploading piece... Dec 26 20:28:29.928799157 (508h59m29.799882192s) [noise=true , address=145.79.227.3:4151 , piece_id=TPR72DFEKFADGYHCAGQWJ4RJTJDNXXB5KOPKPC2HA3EFRJQH5OWQ, seg_pos={0 0}, upload=446641]: Uploading piece... Dec 26 20:28:29.929092376 (508h59m29.800175421s) [noise=true , address=145.79.227.45:4229 , piece_id=3T35SGG2KV4DYKBBIW6F5JEQMB6NL7YOMRB6R5ADIQJ3WHYQN5VQ, seg_pos={0 0}, upload=446641]: Uploading piece... Dec 26 20:28:29.929469754 (508h59m29.800552789s) [noise=true , address=145.79.227.21:4475 , piece_id=FM2HSCM6RJUESMMXEO4VMSRTTA2X4ER7JSWQ63O5BNDMTM2L6PEA, seg_pos={0 0}, upload=446641]: Uploading piece... Dec 26 20:28:29.929936122 (508h59m29.801019157s) [noise=true , address=77.90.31.13:4629 , piece_id=2AMJYSPO5R42RNXQQOMJN7FE52LR555ORB3VBKM3CLNW5NMDLLYA, seg_pos={0 0}, upload=446641]: Uploading piece... Dec 26 20:28:29.930013139 (508h59m29.801096174s) [noise=true , address=145.79.227.25:4618 , piece_id=UU6KFIQCYNZEWWMHSZ4NQ7TOZYA7N4AZMI4VN7RZAQCE4VAAMRFA, seg_pos={0 0}, upload=446641]: Uploading piece... Dec 26 20:28:31.349080054 (508h59m31.220163089s) [noise=true , address=77.90.31.15:4697 , piece_id=74W7KZ2D6H2AUKETVRYMCXVJ6HEBX66SCPI54SZLVSFINWVMJQJA, seg_pos={0 0}, upload=446641]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: storage node overloaded, request limit: 2000 Dec 26 20:28:31.603377739 (508h59m31.474460774s) [upload=446641]: Next returned nil segment Dec 26 20:28:31.603400412 (508h59m31.474483447s) [upload=446641]: Waiting for error group managing segments... Dec 26 20:28:31.928179705 (508h59m31.79926274s) [noise=true , address=145.79.227.25:4618 , piece_id=UU6KFIQCYNZEWWMHSZ4NQ7TOZYA7N4AZMI4VN7RZAQCE4VAAMRFA, seg_pos={0 0}, upload=446641]: Done uploading piece. err: <nil> Dec 26 20:28:31.930119068 (508h59m31.801202103s) [noise=true , address=145.79.227.21:4475 , piece_id=FM2HSCM6RJUESMMXEO4VMSRTTA2X4ER7JSWQ63O5BNDMTM2L6PEA, seg_pos={0 0}, upload=446641]: Done uploading piece. err: <nil> Dec 26 20:28:31.938003371 (508h59m31.809086416s) [noise=true , address=145.79.227.46:4603 , piece_id=D4NLBH33S4MBTMYV4ULTHLL5DUT5ZLPYU2OFIRMZ2JBIXH6RFYXA, seg_pos={0 0}, upload=446641]: Done uploading piece. err: <nil> Dec 26 20:28:31.940924854 (508h59m31.812007889s) [noise=true , address=145.79.227.45:4229 , piece_id=3T35SGG2KV4DYKBBIW6F5JEQMB6NL7YOMRB6R5ADIQJ3WHYQN5VQ, seg_pos={0 0}, upload=446641]: Done uploading piece. err: <nil> Dec 26 20:28:31.942105522 (508h59m31.813188557s) [noise=true , address=145.79.227.36:4517 , piece_id=ZTHEHA7KGDI3YZ3NKV65ZMLRYSZ3VJCHOUB75RZG2GQCNQHNXD5A, seg_pos={0 0}, upload=446641]: Done uploading piece. err: <nil> Dec 26 20:28:31.964426635 (508h59m31.83550967s) [noise=true , address=145.79.227.37:4567 , piece_id=KEKXRVZ6TL6UJBO7AYYIQSI5RDJDMVQXTNUINQGQQDWCXWNSPTNQ, seg_pos={0 0}, upload=446641]: Done uploading piece. err: <nil> Dec 26 20:28:32.158955319 (508h59m32.030038364s) [noise=true , address=45.91.116.4:4939 , piece_id=RGCRAOILK2NIAUEFZ3HHTF5L3MIMI65GHKHUMYL34AESHDZPOCEQ, seg_pos={0 0}, upload=446641]: Done uploading piece. err: <nil> Dec 26 20:28:32.161331084 (508h59m32.032414129s) [noise=true , address=45.91.116.8:4046 , piece_id=MJ65ZTQBDAM6HODPAZ3FBBHQHA5XXVV43RELWC5O77EUSBLIUJ6Q, seg_pos={0 0}, upload=446641]: Done uploading piece. err: <nil> Dec 26 20:28:32.235774115 (508h59m32.10685715s) [noise=true , address=45.91.116.6:4976 , piece_id=7DEGRLZFNZUDMWOKBGQAUZ5EII7INYT3ATHBAFODRRIAAPTVNG4Q, seg_pos={0 0}, upload=446641]: Done uploading piece. err: <nil> Dec 26 20:28:32.246433393 (508h59m32.117516428s) [noise=true , address=77.90.31.13:4629 , piece_id=2AMJYSPO5R42RNXQQOMJN7FE52LR555ORB3VBKM3CLNW5NMDLLYA, seg_pos={0 0}, upload=446641]: Done uploading piece. err: <nil> Dec 26 20:28:32.394498402 (508h59m32.265581447s) [noise=true , address=45.91.116.10:4083 , piece_id=LRDECELM2HNBG73DCMNXH4JGRGJY7GVTRGRIPPIVZFTPAVWDIXYA, seg_pos={0 0}, upload=446641]: Done uploading piece. err: <nil> Dec 26 20:28:32.471701779 (508h59m32.342784834s) [noise=true , address=145.79.227.3:4151 , piece_id=TPR72DFEKFADGYHCAGQWJ4RJTJDNXXB5KOPKPC2HA3EFRJQH5OWQ, seg_pos={0 0}, upload=446641]: Done uploading piece. err: <nil> Dec 26 20:28:33.272979594 (508h59m33.144062629s) [noise=true , address=145.79.227.40:4051 , piece_id=MD7IDP6E37DEMNDSMAXTJDGNRRFZQNLLKHEHMYOJRTVWNNRLQ2BQ, seg_pos={0 0}, upload=446641]: Done uploading piece. err: <nil> Dec 26 20:28:33.273003128 (508h59m33.144086173s) [seg_pos={0 0}, upload=446641]: Segment reached optimal threshold of 13 pieces. Dec 26 20:28:33.273210062 (508h59m33.144293097s) [noise=true , address=145.79.227.2:4114 , piece_id=KPLGTMQAJN3JZHOZR2V2QL76VNUNZZRWNIQ7WPRX5EUPGJ2XOGSA, seg_pos={0 0}, upload=446641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN): protocol: write tcp 188.166.225.170:35140->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:35140->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35140->145.79.227.2:4114: use of closed network connection Dec 26 20:28:33.273218258 (508h59m33.144301323s) [noise=true , address=145.79.227.2:4114 , piece_id=KPLGTMQAJN3JZHOZR2V2QL76VNUNZZRWNIQ7WPRX5EUPGJ2XOGSA, seg_pos={0 0}, upload=446641]: Long tail cancellation due to optimal threshold Dec 26 20:28:33.273418729 (508h59m33.144501764s) [noise=true , address=145.79.227.51:4409 , piece_id=3L6W4XDZBCO34CRQBE47E6NXVW4OEVMAD3CE2IONC322GKM6HCRQ, seg_pos={0 0}, upload=446641]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UoCGA4FjE4yADPXL1oifSPJssPgSGh81REZHwGv2eu1Y1kEaj): protocol: write tcp 188.166.225.170:44904->145.79.227.51:4409: use of closed network connection; write tcp 188.166.225.170:44904->145.79.227.51:4409: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44904->145.79.227.51:4409: use of closed network connection Dec 26 20:28:33.273454016 (508h59m33.144537051s) [noise=true , address=145.79.227.51:4409 , piece_id=3L6W4XDZBCO34CRQBE47E6NXVW4OEVMAD3CE2IONC322GKM6HCRQ, seg_pos={0 0}, upload=446641]: Long tail cancellation due to optimal threshold Dec 26 20:28:33.273476710 (508h59m33.144559745s) [seg_pos={0 0}, upload=446641]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 20:28:33.273587150 (508h59m33.144670185s) [upload=446641]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:28:33.273604383 (508h59m33.144687418s) [upload=446641]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:28:33.579545726 (508h59m33.450628771s) [upload=446641]: Done uploading segments... Dec 26 20:28:33.579573469 (508h59m33.450656514s) [upload=446641]: Upload finished. err: <nil> Duration: 30.00196338s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.X..............XX..X..X...........XXX.X.X....X..XXXX.X...X.] duration: 14.163976247s trace: https://jaeger.dev.storj.io/trace/282bf5cb0b7f2f62 FAIL (2025-12-26 20:28:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: XXGP2M7OY552L7DICY26WDEPS2UHXSCICEMNH4QV34ZQMDGJ5QLQ) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: AVJBGJM36G3J4DCEIO2CPVXMLJN73KQ2PVRXGZZTDRJGFMSUMQBQ) context deadline exceeded; piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: 5TU5FJ4CS3TTPXCTJX7DWOQ26EWMCW5CXVZ3TXX5GLR2PJJ2OBUA) context deadline exceeded ======LOGS====== 7:43.314860197 (508h58m43.185943242s) [upload=446629]: Starting upload Dec 26 20:27:43.314970056 (508h58m43.186053091s) [upload=446629]: Done starting upload Dec 26 20:27:43.315065658 (508h58m43.186148704s) [upload=446629]: Uploading segments... Dec 26 20:27:43.315347836 (508h58m43.186430881s) [upload=446629]: Got next segment. Inline: false Dec 26 20:27:43.315387090 (508h58m43.186470125s) [upload=446629]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 20:27:43.315391408 (508h58m43.186474453s) [upload=446629]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 20:27:43.558173556 (508h58m43.429256591s) [seg_pos={0 0}, upload=446629]: Begin upload segment... Dec 26 20:27:43.558211949 (508h58m43.429294984s) [seg_pos={0 0}, upload=446629]: Done begin upload segment. Dec 26 20:27:43.558243409 (508h58m43.429326454s) [noise=true , address=145.79.227.37:4565 , piece_id=FGE27STQAU2IHPCZW6UW5BMSX5YUR6X25R2Q7TMC4EGCQZ3C4XPA, seg_pos={0 0}, upload=446629]: Uploading piece... Dec 26 20:27:43.558250342 (508h58m43.429333377s) [noise=true , address=145.79.227.40:4050 , piece_id=A7XOXHY54JWHSR56IIVHTI5YBLZO7XYPI4PC3UVMEPUPTB6ES77Q, seg_pos={0 0}, upload=446629]: Uploading piece... Dec 26 20:27:43.558703665 (508h58m43.42978671s) [noise=true , address=77.90.31.13:4627 , piece_id=5TU5FJ4CS3TTPXCTJX7DWOQ26EWMCW5CXVZ3TXX5GLR2PJJ2OBUA, seg_pos={0 0}, upload=446629]: Uploading piece... Dec 26 20:27:43.561856889 (508h58m43.432939924s) [noise=true , address=145.79.227.45:4231 , piece_id=EPKSYGPZZGFOHP6RSND6THUESAWUBPV6Q5FAMI5BCKFXCDIM7X2A, seg_pos={0 0}, upload=446629]: Uploading piece... Dec 26 20:27:43.563252757 (508h58m43.434335792s) [noise=true , address=145.79.227.11:4729 , piece_id=UJF4VU4WVYO5NHBC7S67YY5P4M7M2BX5X35DIAEKZKCSPTBN4UKA, seg_pos={0 0}, upload=446629]: Uploading piece... Dec 26 20:27:43.564404851 (508h58m43.435487886s) [noise=true , address=45.91.116.11:4117 , piece_id=XXGP2M7OY552L7DICY26WDEPS2UHXSCICEMNH4QV34ZQMDGJ5QLQ, seg_pos={0 0}, upload=446629]: Uploading piece... Dec 26 20:27:43.566148401 (508h58m43.437231436s) [noise=true , address=145.79.227.35:4481 , piece_id=XC2MZW26P3FO5DNXN47DCKWIUT4DTQEYBED234VDXTKYM4V7CQLQ, seg_pos={0 0}, upload=446629]: Uploading piece... Dec 26 20:27:43.566447010 (508h58m43.437530045s) [noise=true , address=145.79.227.36:4519 , piece_id=SERLZSX34ACY4HHI2SQ6JPMQOMPZ6ZALIA3OWNE3ODNCJOBECYYQ, seg_pos={0 0}, upload=446629]: Uploading piece... Dec 26 20:27:43.568304947 (508h58m43.439387992s) [noise=true , address=145.79.227.42:4121 , piece_id=EGMF6M3WHBFJCN46PB2TJK2SNVQH3YPDGOUMS64FTAZLBBP5TNNA, seg_pos={0 0}, upload=446629]: Uploading piece... Dec 26 20:27:43.568585822 (508h58m43.439668857s) [noise=true , address=45.91.116.8:4045 , piece_id=FX2U5DIIOYBCQIFQGKHYC55UUPEU2PHBLZTLIT6WHUGXHSX22JIQ, seg_pos={0 0}, upload=446629]: Uploading piece... Dec 26 20:27:43.569786198 (508h58m43.440869233s) [noise=true , address=145.79.227.41:4085 , piece_id=RLNT6LMGRGHWBC75ASB34AQR75I3PZV6XK3ZRYMAUQB6PFOPRMZQ, seg_pos={0 0}, upload=446629]: Uploading piece... Dec 26 20:27:43.571225568 (508h58m43.442308674s) [noise=true , address=45.91.116.4:4939 , piece_id=2LW3WKL6D73Z7KNWT4F35KGGXAUBFAFDFUFWNVIQBAB3JGIE77ZQ, seg_pos={0 0}, upload=446629]: Uploading piece... Dec 26 20:27:43.574036672 (508h58m43.445119707s) [noise=true , address=145.79.227.12:4767 , piece_id=MCM4LWSLTZXNJKCXTMGJJTVIM5WJQRPJQFQVN4LC4GWXXCO44GQQ, seg_pos={0 0}, upload=446629]: Uploading piece... Dec 26 20:27:43.575216378 (508h58m43.446299413s) [noise=true , address=145.79.227.21:4475 , piece_id=FUOLSYL3VOBLXLMVORYWKU2W44FEM7U6SDAAAV7KBJIDHH5NTCYQ, seg_pos={0 0}, upload=446629]: Uploading piece... Dec 26 20:27:43.575302632 (508h58m43.446385677s) [noise=true , address=45.91.116.5:4300 , piece_id=AVJBGJM36G3J4DCEIO2CPVXMLJN73KQ2PVRXGZZTDRJGFMSUMQBQ, seg_pos={0 0}, upload=446629]: Uploading piece... Dec 26 20:27:43.576258863 (508h58m43.447341898s) [noise=true , address=145.79.227.22:4509 , piece_id=HBDOGP4U4YRZCDZNE4UKTYDQXJWE3SKJAHKFHV7RDALKVXZWZIPQ, seg_pos={0 0}, upload=446629]: Uploading piece... Dec 26 20:27:45.178797799 (508h58m45.049880874s) [upload=446629]: Next returned nil segment Dec 26 20:27:45.178812797 (508h58m45.049895842s) [upload=446629]: Waiting for error group managing segments... Dec 26 20:27:45.562299389 (508h58m45.433382424s) [noise=true , address=145.79.227.11:4729 , piece_id=UJF4VU4WVYO5NHBC7S67YY5P4M7M2BX5X35DIAEKZKCSPTBN4UKA, seg_pos={0 0}, upload=446629]: Done uploading piece. err: <nil> Dec 26 20:27:45.570168383 (508h58m45.441251428s) [noise=true , address=145.79.227.45:4231 , piece_id=EPKSYGPZZGFOHP6RSND6THUESAWUBPV6Q5FAMI5BCKFXCDIM7X2A, seg_pos={0 0}, upload=446629]: Done uploading piece. err: <nil> Dec 26 20:27:45.570279624 (508h58m45.441362659s) [noise=true , address=145.79.227.37:4565 , piece_id=FGE27STQAU2IHPCZW6UW5BMSX5YUR6X25R2Q7TMC4EGCQZ3C4XPA, seg_pos={0 0}, upload=446629]: Done uploading piece. err: <nil> Dec 26 20:27:45.571925940 (508h58m45.443008985s) [noise=true , address=145.79.227.12:4767 , piece_id=MCM4LWSLTZXNJKCXTMGJJTVIM5WJQRPJQFQVN4LC4GWXXCO44GQQ, seg_pos={0 0}, upload=446629]: Done uploading piece. err: <nil> Dec 26 20:27:45.575411326 (508h58m45.446494361s) [noise=true , address=145.79.227.36:4519 , piece_id=SERLZSX34ACY4HHI2SQ6JPMQOMPZ6ZALIA3OWNE3ODNCJOBECYYQ, seg_pos={0 0}, upload=446629]: Done uploading piece. err: <nil> Dec 26 20:27:45.576905371 (508h58m45.447988406s) [noise=true , address=145.79.227.21:4475 , piece_id=FUOLSYL3VOBLXLMVORYWKU2W44FEM7U6SDAAAV7KBJIDHH5NTCYQ, seg_pos={0 0}, upload=446629]: Done uploading piece. err: <nil> Dec 26 20:27:45.583690200 (508h58m45.454773235s) [noise=true , address=145.79.227.42:4121 , piece_id=EGMF6M3WHBFJCN46PB2TJK2SNVQH3YPDGOUMS64FTAZLBBP5TNNA, seg_pos={0 0}, upload=446629]: Done uploading piece. err: <nil> Dec 26 20:27:45.583861667 (508h58m45.454944712s) [noise=true , address=145.79.227.35:4481 , piece_id=XC2MZW26P3FO5DNXN47DCKWIUT4DTQEYBED234VDXTKYM4V7CQLQ, seg_pos={0 0}, upload=446629]: Done uploading piece. err: <nil> Dec 26 20:27:45.594278281 (508h58m45.465361316s) [noise=true , address=145.79.227.22:4509 , piece_id=HBDOGP4U4YRZCDZNE4UKTYDQXJWE3SKJAHKFHV7RDALKVXZWZIPQ, seg_pos={0 0}, upload=446629]: Done uploading piece. err: <nil> Dec 26 20:27:45.782835034 (508h58m45.653918079s) [noise=true , address=77.90.31.13:4627 , piece_id=5TU5FJ4CS3TTPXCTJX7DWOQ26EWMCW5CXVZ3TXX5GLR2PJJ2OBUA, seg_pos={0 0}, upload=446629]: Done uploading piece. err: <nil> Dec 26 20:27:45.817302586 (508h58m45.688385621s) [noise=true , address=45.91.116.11:4117 , piece_id=XXGP2M7OY552L7DICY26WDEPS2UHXSCICEMNH4QV34ZQMDGJ5QLQ, seg_pos={0 0}, upload=446629]: Done uploading piece. err: <nil> Dec 26 20:27:45.853334978 (508h58m45.724418013s) [noise=true , address=45.91.116.5:4300 , piece_id=AVJBGJM36G3J4DCEIO2CPVXMLJN73KQ2PVRXGZZTDRJGFMSUMQBQ, seg_pos={0 0}, upload=446629]: Done uploading piece. err: <nil> Dec 26 20:27:45.886998289 (508h58m45.758081324s) [noise=true , address=45.91.116.4:4939 , piece_id=2LW3WKL6D73Z7KNWT4F35KGGXAUBFAFDFUFWNVIQBAB3JGIE77ZQ, seg_pos={0 0}, upload=446629]: Done uploading piece. err: <nil> Dec 26 20:27:45.887017685 (508h58m45.75810072s) [seg_pos={0 0}, upload=446629]: Segment reached optimal threshold of 13 pieces. Dec 26 20:27:45.887238686 (508h58m45.758321721s) [noise=true , address=45.91.116.8:4045 , piece_id=FX2U5DIIOYBCQIFQGKHYC55UUPEU2PHBLZTLIT6WHUGXHSX22JIQ, seg_pos={0 0}, upload=446629]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 26 20:27:45.887246592 (508h58m45.758329627s) [noise=true , address=45.91.116.8:4045 , piece_id=FX2U5DIIOYBCQIFQGKHYC55UUPEU2PHBLZTLIT6WHUGXHSX22JIQ, seg_pos={0 0}, upload=446629]: Long tail cancellation due to optimal threshold Dec 26 20:27:45.887459286 (508h58m45.758542322s) [noise=true , address=145.79.227.41:4085 , piece_id=RLNT6LMGRGHWBC75ASB34AQR75I3PZV6XK3ZRYMAUQB6PFOPRMZQ, seg_pos={0 0}, upload=446629]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C): protocol: write tcp 188.166.225.170:40082->145.79.227.41:4085: use of closed network connection; write tcp 188.166.225.170:40082->145.79.227.41:4085: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40082->145.79.227.41:4085: use of closed network connection Dec 26 20:27:45.887466069 (508h58m45.758549114s) [noise=true , address=145.79.227.41:4085 , piece_id=RLNT6LMGRGHWBC75ASB34AQR75I3PZV6XK3ZRYMAUQB6PFOPRMZQ, seg_pos={0 0}, upload=446629]: Long tail cancellation due to optimal threshold Dec 26 20:27:45.887560339 (508h58m45.758643374s) [noise=true , address=145.79.227.40:4050 , piece_id=A7XOXHY54JWHSR56IIVHTI5YBLZO7XYPI4PC3UVMEPUPTB6ES77Q, seg_pos={0 0}, upload=446629]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 26 20:27:45.887565238 (508h58m45.758648273s) [noise=true , address=145.79.227.40:4050 , piece_id=A7XOXHY54JWHSR56IIVHTI5YBLZO7XYPI4PC3UVMEPUPTB6ES77Q, seg_pos={0 0}, upload=446629]: Long tail cancellation due to optimal threshold Dec 26 20:27:45.887580857 (508h58m45.758663903s) [seg_pos={0 0}, upload=446629]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 20:27:45.887721836 (508h58m45.758804871s) [upload=446629]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:27:45.887727266 (508h58m45.758810301s) [upload=446629]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:27:46.180208158 (508h58m46.051291203s) [upload=446629]: Done uploading segments... Dec 26 20:27:46.180225993 (508h58m46.051309038s) [upload=446629]: Upload finished. err: <nil> Duration: 14.163998008s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..X..........................................X..............] duration: 19.623780453s trace: https://jaeger.dev.storj.io/trace/663e21624a5a3654 FAIL (2025-12-26 20:28:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, Piece ID: PWVMOEKW5A4ZAVSVJAAHFTYYTQZGFHDF642GWR6EABEGWP2CDRGQ) context deadline exceeded; piecestore: (Node ID: 12whTD1Kvu4T7ANNtkSvDB66uTjVgfPTK9TAzExgvwJYgZ3zSFG, Piece ID: ME2LDK2QJI7LLG67ODDIJDKB5CAT47YDGMJ3CTVK4F6UDHJ4TPOQ) context deadline exceeded; piecestore: (Node ID: 12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH, Piece ID: HJGRPPEJK75YTIKBKNVK7J4TDRKBCNL7HA4JNMOKD2ROGC7C4P3Q) context deadline exceeded ======LOGS====== ) [noise=true , address=145.79.227.7:4296 , piece_id=PVH6VN4CG4O2DC22G74US7IC5GDIGM2KYYYQQPFO5MG22JQCAU2A, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.248055070 (508h48m1.119138115s) [noise=true , address=145.79.227.40:4078 , piece_id=J6TGA3SFQOLGL2C2TS352TSLZBMI6MDXU5BDTTGQYMN4A5QG462A, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.248175840 (508h48m1.119258875s) [noise=true , address=77.90.31.12:4600 , piece_id=JESM5JF4GWUDLEW4YT5SKHJOXD54TWRQBNGGQCBMDSS54XR4AEKQ, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.248604496 (508h48m1.119687531s) [noise=true , address=45.91.116.6:4985 , piece_id=WV5SXZIBMXB4FTNZFE4PKBR5ZWZTUUPADPRMXAVE4Z3HKDGLBOSQ, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.248640474 (508h48m1.119723509s) [noise=true , address=145.79.227.18:4021 , piece_id=ER7GXC5OHAPNGU37QTKPROKB4YTCDQBUHVEIEUH2K5FQ4X5AGM2Q, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.249180202 (508h48m1.120263238s) [noise=true , address=77.90.31.18:4821 , piece_id=HJGRPPEJK75YTIKBKNVK7J4TDRKBCNL7HA4JNMOKD2ROGC7C4P3Q, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.249851130 (508h48m1.120934175s) [noise=true , address=64.49.13.169:4709 , piece_id=ZGWBPAYFOQWU2LSTILYMVRNKGSP53UXEQHRW4NJEXW5OLNVCCWRQ, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.250638910 (508h48m1.121721955s) [noise=true , address=145.79.227.38:4001 , piece_id=TWPCSZEWSFC2ZT7QNWHRS3RY42LLBPIEF3EIQQ65OXFZSHWAV2GA, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.251000769 (508h48m1.122083814s) [noise=true , address=145.79.227.30:4792 , piece_id=RLEUVLBAOMJSDAYHMQJGKYQUC6NYYGSGUVIU5Z53XGBINIQV43GQ, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.251314737 (508h48m1.122397772s) [noise=true , address=145.79.227.5:4252 , piece_id=FI5RNNRKSIYHR3JKCOW3THJQKHAPYLUDDHT3Y4TRYU3OSMCSWQQQ, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.251635527 (508h48m1.122718572s) [noise=true , address=77.90.31.11:4588 , piece_id=ZWDOVL5PGZ4YPY6NZVVSBSWXDDD3DAJ4RUK2BWMOGTHDGWJ2G4DA, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.251919499 (508h48m1.123002544s) [noise=true , address=77.90.31.10:4536 , piece_id=V43VRDKXTE2PHOBPSQUIJPKMKY3ROAEQAI2D4XIJHOLQ3UIMN4RA, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.252236542 (508h48m1.123319588s) [noise=true , address=77.90.31.21:4894 , piece_id=AGL6GVJX2YLGHSZX33SUURUU5S22CF3YRMJTP3QMHHP7OKX2MFSA, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.252528137 (508h48m1.123611182s) [noise=true , address=77.90.31.15:4702 , piece_id=KJEPR7NMX4UHQZHTMDDJKBRLLQ3O5BLJSLYBXGCOFGP3LNC3FXGA, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.252861402 (508h48m1.123944447s) [noise=true , address=64.49.13.27:4961 , piece_id=XCVH62CEVRYGR53U5VG4L4HJMBJSQ562L4QVKTXZSC7XGAEB4LDQ, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:02.570977410 (508h48m2.442060455s) [upload=446477]: Next returned nil segment Dec 26 20:17:02.571003019 (508h48m2.442086054s) [upload=446477]: Waiting for error group managing segments... Dec 26 20:17:02.949353997 (508h48m2.820437042s) [noise=true , address=64.49.13.27:4961 , piece_id=XCVH62CEVRYGR53U5VG4L4HJMBJSQ562L4QVKTXZSC7XGAEB4LDQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.136546190 (508h48m3.007629235s) [noise=true , address=145.79.227.5:4252 , piece_id=FI5RNNRKSIYHR3JKCOW3THJQKHAPYLUDDHT3Y4TRYU3OSMCSWQQQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.155760468 (508h48m3.026843513s) [noise=true , address=145.79.227.38:4001 , piece_id=TWPCSZEWSFC2ZT7QNWHRS3RY42LLBPIEF3EIQQ65OXFZSHWAV2GA, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.179239646 (508h48m3.050322691s) [noise=true , address=145.79.227.18:4021 , piece_id=ER7GXC5OHAPNGU37QTKPROKB4YTCDQBUHVEIEUH2K5FQ4X5AGM2Q, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.372384563 (508h48m3.243467608s) [noise=true , address=77.90.31.15:4702 , piece_id=KJEPR7NMX4UHQZHTMDDJKBRLLQ3O5BLJSLYBXGCOFGP3LNC3FXGA, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.447635803 (508h48m3.318718848s) [noise=true , address=77.90.31.18:4821 , piece_id=HJGRPPEJK75YTIKBKNVK7J4TDRKBCNL7HA4JNMOKD2ROGC7C4P3Q, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.452148166 (508h48m3.323231201s) [noise=true , address=77.90.31.10:4536 , piece_id=V43VRDKXTE2PHOBPSQUIJPKMKY3ROAEQAI2D4XIJHOLQ3UIMN4RA, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.644263002 (508h48m3.515346047s) [noise=true , address=77.90.31.14:4689 , piece_id=PWVMOEKW5A4ZAVSVJAAHFTYYTQZGFHDF642GWR6EABEGWP2CDRGQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.710620497 (508h48m3.581703542s) [noise=true , address=45.91.116.8:4074 , piece_id=ME2LDK2QJI7LLG67ODDIJDKB5CAT47YDGMJ3CTVK4F6UDHJ4TPOQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:04.066910008 (508h48m3.937993053s) [noise=true , address=77.90.31.12:4600 , piece_id=JESM5JF4GWUDLEW4YT5SKHJOXD54TWRQBNGGQCBMDSS54XR4AEKQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:04.082282349 (508h48m3.953365394s) [noise=true , address=145.79.227.44:4219 , piece_id=KZVV2NJBIU22C4URLBZMQVQN4IZ2ZMQ5YODRJMCS6XYXULHCZWLQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:04.089048724 (508h48m3.960131759s) [noise=true , address=145.79.227.30:4792 , piece_id=RLEUVLBAOMJSDAYHMQJGKYQUC6NYYGSGUVIU5Z53XGBINIQV43GQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:04.288772749 (508h48m4.159855794s) [noise=true , address=77.90.31.21:4894 , piece_id=AGL6GVJX2YLGHSZX33SUURUU5S22CF3YRMJTP3QMHHP7OKX2MFSA, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:04.344183239 (508h48m4.215266294s) [noise=true , address=77.90.31.11:4588 , piece_id=ZWDOVL5PGZ4YPY6NZVVSBSWXDDD3DAJ4RUK2BWMOGTHDGWJ2G4DA, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:04.344208728 (508h48m4.215291773s) [seg_pos={0 0}, upload=446477]: Segment reached optimal threshold of 14 pieces. Dec 26 20:17:04.344470828 (508h48m4.215553873s) [noise=true , address=64.49.13.169:4709 , piece_id=ZGWBPAYFOQWU2LSTILYMVRNKGSP53UXEQHRW4NJEXW5OLNVCCWRQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:47062->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:47062->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47062->64.49.13.169:4709: use of closed network connection Dec 26 20:17:04.344481918 (508h48m4.215564953s) [noise=true , address=64.49.13.169:4709 , piece_id=ZGWBPAYFOQWU2LSTILYMVRNKGSP53UXEQHRW4NJEXW5OLNVCCWRQ, seg_pos={0 0}, upload=446477]: Long tail cancellation due to optimal threshold Dec 26 20:17:04.344541813 (508h48m4.215624848s) [noise=true , address=145.79.227.40:4078 , piece_id=J6TGA3SFQOLGL2C2TS352TSLZBMI6MDXU5BDTTGQYMN4A5QG462A, seg_pos={0 0}, upload=446477]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tzgtenV6us5ssD7xWGpZmVUoWEfbFhBNewbK9irHtZPkyDtRd): protocol: expected piece hash; context canceled; EOF Dec 26 20:17:04.344559476 (508h48m4.215642511s) [noise=true , address=145.79.227.40:4078 , piece_id=J6TGA3SFQOLGL2C2TS352TSLZBMI6MDXU5BDTTGQYMN4A5QG462A, seg_pos={0 0}, upload=446477]: Long tail cancellation due to optimal threshold Dec 26 20:17:04.344755479 (508h48m4.215838524s) [noise=true , address=145.79.227.7:4296 , piece_id=PVH6VN4CG4O2DC22G74US7IC5GDIGM2KYYYQQPFO5MG22JQCAU2A, seg_pos={0 0}, upload=446477]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cKhDbA4ptwux2LYNPNAJ7Qh8jECsGWd1KWaK6T6pD38bJWdt1): protocol: write tcp 188.166.225.170:48104->145.79.227.7:4296: use of closed network connection; write tcp 188.166.225.170:48104->145.79.227.7:4296: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48104->145.79.227.7:4296: use of closed network connection Dec 26 20:17:04.344763044 (508h48m4.215846099s) [noise=true , address=145.79.227.7:4296 , piece_id=PVH6VN4CG4O2DC22G74US7IC5GDIGM2KYYYQQPFO5MG22JQCAU2A, seg_pos={0 0}, upload=446477]: Long tail cancellation due to optimal threshold Dec 26 20:17:04.344928509 (508h48m4.216011544s) [noise=true , address=45.91.116.6:4985 , piece_id=WV5SXZIBMXB4FTNZFE4PKBR5ZWZTUUPADPRMXAVE4Z3HKDGLBOSQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RNs9MM9jDtNC2gZC63SYsePZTd5mE7dqUm9A5Z5QWdPX8PVnp): protocol: expected piece hash; context canceled; EOF Dec 26 20:17:04.344938277 (508h48m4.216021312s) [noise=true , address=45.91.116.6:4985 , piece_id=WV5SXZIBMXB4FTNZFE4PKBR5ZWZTUUPADPRMXAVE4Z3HKDGLBOSQ, seg_pos={0 0}, upload=446477]: Long tail cancellation due to optimal threshold Dec 26 20:17:04.344966722 (508h48m4.216049767s) [seg_pos={0 0}, upload=446477]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:17:04.345054548 (508h48m4.216137593s) [upload=446477]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:17:04.345059839 (508h48m4.216142874s) [upload=446477]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:17:04.653613492 (508h48m4.524696537s) [upload=446477]: Done uploading segments... Dec 26 20:17:04.653639822 (508h48m4.524722857s) [upload=446477]: Upload finished. err: <nil> Duration: 19.623810551s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [X.X..............XX..X..X...........XXX.X.X....X..XXXX.X...X] duration: 30.002426451s trace: https://jaeger.dev.storj.io/trace/282bf5cb0b7f2f62 FAIL (2025-12-26 20:28:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: XXGP2M7OY552L7DICY26WDEPS2UHXSCICEMNH4QV34ZQMDGJ5QLQ) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: AVJBGJM36G3J4DCEIO2CPVXMLJN73KQ2PVRXGZZTDRJGFMSUMQBQ) context deadline exceeded; piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: 5TU5FJ4CS3TTPXCTJX7DWOQ26EWMCW5CXVZ3TXX5GLR2PJJ2OBUA) context deadline exceeded ======LOGS====== 7:43.314860197 (508h58m43.185943242s) [upload=446629]: Starting upload Dec 26 20:27:43.314970056 (508h58m43.186053091s) [upload=446629]: Done starting upload Dec 26 20:27:43.315065658 (508h58m43.186148704s) [upload=446629]: Uploading segments... Dec 26 20:27:43.315347836 (508h58m43.186430881s) [upload=446629]: Got next segment. Inline: false Dec 26 20:27:43.315387090 (508h58m43.186470125s) [upload=446629]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 20:27:43.315391408 (508h58m43.186474453s) [upload=446629]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 20:27:43.558173556 (508h58m43.429256591s) [seg_pos={0 0}, upload=446629]: Begin upload segment... Dec 26 20:27:43.558211949 (508h58m43.429294984s) [seg_pos={0 0}, upload=446629]: Done begin upload segment. Dec 26 20:27:43.558243409 (508h58m43.429326454s) [noise=true , address=145.79.227.37:4565 , piece_id=FGE27STQAU2IHPCZW6UW5BMSX5YUR6X25R2Q7TMC4EGCQZ3C4XPA, seg_pos={0 0}, upload=446629]: Uploading piece... Dec 26 20:27:43.558250342 (508h58m43.429333377s) [noise=true , address=145.79.227.40:4050 , piece_id=A7XOXHY54JWHSR56IIVHTI5YBLZO7XYPI4PC3UVMEPUPTB6ES77Q, seg_pos={0 0}, upload=446629]: Uploading piece... Dec 26 20:27:43.558703665 (508h58m43.42978671s) [noise=true , address=77.90.31.13:4627 , piece_id=5TU5FJ4CS3TTPXCTJX7DWOQ26EWMCW5CXVZ3TXX5GLR2PJJ2OBUA, seg_pos={0 0}, upload=446629]: Uploading piece... Dec 26 20:27:43.561856889 (508h58m43.432939924s) [noise=true , address=145.79.227.45:4231 , piece_id=EPKSYGPZZGFOHP6RSND6THUESAWUBPV6Q5FAMI5BCKFXCDIM7X2A, seg_pos={0 0}, upload=446629]: Uploading piece... Dec 26 20:27:43.563252757 (508h58m43.434335792s) [noise=true , address=145.79.227.11:4729 , piece_id=UJF4VU4WVYO5NHBC7S67YY5P4M7M2BX5X35DIAEKZKCSPTBN4UKA, seg_pos={0 0}, upload=446629]: Uploading piece... Dec 26 20:27:43.564404851 (508h58m43.435487886s) [noise=true , address=45.91.116.11:4117 , piece_id=XXGP2M7OY552L7DICY26WDEPS2UHXSCICEMNH4QV34ZQMDGJ5QLQ, seg_pos={0 0}, upload=446629]: Uploading piece... Dec 26 20:27:43.566148401 (508h58m43.437231436s) [noise=true , address=145.79.227.35:4481 , piece_id=XC2MZW26P3FO5DNXN47DCKWIUT4DTQEYBED234VDXTKYM4V7CQLQ, seg_pos={0 0}, upload=446629]: Uploading piece... Dec 26 20:27:43.566447010 (508h58m43.437530045s) [noise=true , address=145.79.227.36:4519 , piece_id=SERLZSX34ACY4HHI2SQ6JPMQOMPZ6ZALIA3OWNE3ODNCJOBECYYQ, seg_pos={0 0}, upload=446629]: Uploading piece... Dec 26 20:27:43.568304947 (508h58m43.439387992s) [noise=true , address=145.79.227.42:4121 , piece_id=EGMF6M3WHBFJCN46PB2TJK2SNVQH3YPDGOUMS64FTAZLBBP5TNNA, seg_pos={0 0}, upload=446629]: Uploading piece... Dec 26 20:27:43.568585822 (508h58m43.439668857s) [noise=true , address=45.91.116.8:4045 , piece_id=FX2U5DIIOYBCQIFQGKHYC55UUPEU2PHBLZTLIT6WHUGXHSX22JIQ, seg_pos={0 0}, upload=446629]: Uploading piece... Dec 26 20:27:43.569786198 (508h58m43.440869233s) [noise=true , address=145.79.227.41:4085 , piece_id=RLNT6LMGRGHWBC75ASB34AQR75I3PZV6XK3ZRYMAUQB6PFOPRMZQ, seg_pos={0 0}, upload=446629]: Uploading piece... Dec 26 20:27:43.571225568 (508h58m43.442308674s) [noise=true , address=45.91.116.4:4939 , piece_id=2LW3WKL6D73Z7KNWT4F35KGGXAUBFAFDFUFWNVIQBAB3JGIE77ZQ, seg_pos={0 0}, upload=446629]: Uploading piece... Dec 26 20:27:43.574036672 (508h58m43.445119707s) [noise=true , address=145.79.227.12:4767 , piece_id=MCM4LWSLTZXNJKCXTMGJJTVIM5WJQRPJQFQVN4LC4GWXXCO44GQQ, seg_pos={0 0}, upload=446629]: Uploading piece... Dec 26 20:27:43.575216378 (508h58m43.446299413s) [noise=true , address=145.79.227.21:4475 , piece_id=FUOLSYL3VOBLXLMVORYWKU2W44FEM7U6SDAAAV7KBJIDHH5NTCYQ, seg_pos={0 0}, upload=446629]: Uploading piece... Dec 26 20:27:43.575302632 (508h58m43.446385677s) [noise=true , address=45.91.116.5:4300 , piece_id=AVJBGJM36G3J4DCEIO2CPVXMLJN73KQ2PVRXGZZTDRJGFMSUMQBQ, seg_pos={0 0}, upload=446629]: Uploading piece... Dec 26 20:27:43.576258863 (508h58m43.447341898s) [noise=true , address=145.79.227.22:4509 , piece_id=HBDOGP4U4YRZCDZNE4UKTYDQXJWE3SKJAHKFHV7RDALKVXZWZIPQ, seg_pos={0 0}, upload=446629]: Uploading piece... Dec 26 20:27:45.178797799 (508h58m45.049880874s) [upload=446629]: Next returned nil segment Dec 26 20:27:45.178812797 (508h58m45.049895842s) [upload=446629]: Waiting for error group managing segments... Dec 26 20:27:45.562299389 (508h58m45.433382424s) [noise=true , address=145.79.227.11:4729 , piece_id=UJF4VU4WVYO5NHBC7S67YY5P4M7M2BX5X35DIAEKZKCSPTBN4UKA, seg_pos={0 0}, upload=446629]: Done uploading piece. err: <nil> Dec 26 20:27:45.570168383 (508h58m45.441251428s) [noise=true , address=145.79.227.45:4231 , piece_id=EPKSYGPZZGFOHP6RSND6THUESAWUBPV6Q5FAMI5BCKFXCDIM7X2A, seg_pos={0 0}, upload=446629]: Done uploading piece. err: <nil> Dec 26 20:27:45.570279624 (508h58m45.441362659s) [noise=true , address=145.79.227.37:4565 , piece_id=FGE27STQAU2IHPCZW6UW5BMSX5YUR6X25R2Q7TMC4EGCQZ3C4XPA, seg_pos={0 0}, upload=446629]: Done uploading piece. err: <nil> Dec 26 20:27:45.571925940 (508h58m45.443008985s) [noise=true , address=145.79.227.12:4767 , piece_id=MCM4LWSLTZXNJKCXTMGJJTVIM5WJQRPJQFQVN4LC4GWXXCO44GQQ, seg_pos={0 0}, upload=446629]: Done uploading piece. err: <nil> Dec 26 20:27:45.575411326 (508h58m45.446494361s) [noise=true , address=145.79.227.36:4519 , piece_id=SERLZSX34ACY4HHI2SQ6JPMQOMPZ6ZALIA3OWNE3ODNCJOBECYYQ, seg_pos={0 0}, upload=446629]: Done uploading piece. err: <nil> Dec 26 20:27:45.576905371 (508h58m45.447988406s) [noise=true , address=145.79.227.21:4475 , piece_id=FUOLSYL3VOBLXLMVORYWKU2W44FEM7U6SDAAAV7KBJIDHH5NTCYQ, seg_pos={0 0}, upload=446629]: Done uploading piece. err: <nil> Dec 26 20:27:45.583690200 (508h58m45.454773235s) [noise=true , address=145.79.227.42:4121 , piece_id=EGMF6M3WHBFJCN46PB2TJK2SNVQH3YPDGOUMS64FTAZLBBP5TNNA, seg_pos={0 0}, upload=446629]: Done uploading piece. err: <nil> Dec 26 20:27:45.583861667 (508h58m45.454944712s) [noise=true , address=145.79.227.35:4481 , piece_id=XC2MZW26P3FO5DNXN47DCKWIUT4DTQEYBED234VDXTKYM4V7CQLQ, seg_pos={0 0}, upload=446629]: Done uploading piece. err: <nil> Dec 26 20:27:45.594278281 (508h58m45.465361316s) [noise=true , address=145.79.227.22:4509 , piece_id=HBDOGP4U4YRZCDZNE4UKTYDQXJWE3SKJAHKFHV7RDALKVXZWZIPQ, seg_pos={0 0}, upload=446629]: Done uploading piece. err: <nil> Dec 26 20:27:45.782835034 (508h58m45.653918079s) [noise=true , address=77.90.31.13:4627 , piece_id=5TU5FJ4CS3TTPXCTJX7DWOQ26EWMCW5CXVZ3TXX5GLR2PJJ2OBUA, seg_pos={0 0}, upload=446629]: Done uploading piece. err: <nil> Dec 26 20:27:45.817302586 (508h58m45.688385621s) [noise=true , address=45.91.116.11:4117 , piece_id=XXGP2M7OY552L7DICY26WDEPS2UHXSCICEMNH4QV34ZQMDGJ5QLQ, seg_pos={0 0}, upload=446629]: Done uploading piece. err: <nil> Dec 26 20:27:45.853334978 (508h58m45.724418013s) [noise=true , address=45.91.116.5:4300 , piece_id=AVJBGJM36G3J4DCEIO2CPVXMLJN73KQ2PVRXGZZTDRJGFMSUMQBQ, seg_pos={0 0}, upload=446629]: Done uploading piece. err: <nil> Dec 26 20:27:45.886998289 (508h58m45.758081324s) [noise=true , address=45.91.116.4:4939 , piece_id=2LW3WKL6D73Z7KNWT4F35KGGXAUBFAFDFUFWNVIQBAB3JGIE77ZQ, seg_pos={0 0}, upload=446629]: Done uploading piece. err: <nil> Dec 26 20:27:45.887017685 (508h58m45.75810072s) [seg_pos={0 0}, upload=446629]: Segment reached optimal threshold of 13 pieces. Dec 26 20:27:45.887238686 (508h58m45.758321721s) [noise=true , address=45.91.116.8:4045 , piece_id=FX2U5DIIOYBCQIFQGKHYC55UUPEU2PHBLZTLIT6WHUGXHSX22JIQ, seg_pos={0 0}, upload=446629]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 26 20:27:45.887246592 (508h58m45.758329627s) [noise=true , address=45.91.116.8:4045 , piece_id=FX2U5DIIOYBCQIFQGKHYC55UUPEU2PHBLZTLIT6WHUGXHSX22JIQ, seg_pos={0 0}, upload=446629]: Long tail cancellation due to optimal threshold Dec 26 20:27:45.887459286 (508h58m45.758542322s) [noise=true , address=145.79.227.41:4085 , piece_id=RLNT6LMGRGHWBC75ASB34AQR75I3PZV6XK3ZRYMAUQB6PFOPRMZQ, seg_pos={0 0}, upload=446629]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C): protocol: write tcp 188.166.225.170:40082->145.79.227.41:4085: use of closed network connection; write tcp 188.166.225.170:40082->145.79.227.41:4085: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40082->145.79.227.41:4085: use of closed network connection Dec 26 20:27:45.887466069 (508h58m45.758549114s) [noise=true , address=145.79.227.41:4085 , piece_id=RLNT6LMGRGHWBC75ASB34AQR75I3PZV6XK3ZRYMAUQB6PFOPRMZQ, seg_pos={0 0}, upload=446629]: Long tail cancellation due to optimal threshold Dec 26 20:27:45.887560339 (508h58m45.758643374s) [noise=true , address=145.79.227.40:4050 , piece_id=A7XOXHY54JWHSR56IIVHTI5YBLZO7XYPI4PC3UVMEPUPTB6ES77Q, seg_pos={0 0}, upload=446629]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 26 20:27:45.887565238 (508h58m45.758648273s) [noise=true , address=145.79.227.40:4050 , piece_id=A7XOXHY54JWHSR56IIVHTI5YBLZO7XYPI4PC3UVMEPUPTB6ES77Q, seg_pos={0 0}, upload=446629]: Long tail cancellation due to optimal threshold Dec 26 20:27:45.887580857 (508h58m45.758663903s) [seg_pos={0 0}, upload=446629]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 20:27:45.887721836 (508h58m45.758804871s) [upload=446629]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:27:45.887727266 (508h58m45.758810301s) [upload=446629]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:27:46.180208158 (508h58m46.051291203s) [upload=446629]: Done uploading segments... Dec 26 20:27:46.180225993 (508h58m46.051309038s) [upload=446629]: Upload finished. err: <nil> Duration: 30.002448734s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...X..........................................X.............] duration: 34.04524313s trace: https://jaeger.dev.storj.io/trace/663e21624a5a3654 FAIL (2025-12-26 20:27:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, Piece ID: PWVMOEKW5A4ZAVSVJAAHFTYYTQZGFHDF642GWR6EABEGWP2CDRGQ) context deadline exceeded; piecestore: (Node ID: 12whTD1Kvu4T7ANNtkSvDB66uTjVgfPTK9TAzExgvwJYgZ3zSFG, Piece ID: ME2LDK2QJI7LLG67ODDIJDKB5CAT47YDGMJ3CTVK4F6UDHJ4TPOQ) context deadline exceeded; piecestore: (Node ID: 12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH, Piece ID: HJGRPPEJK75YTIKBKNVK7J4TDRKBCNL7HA4JNMOKD2ROGC7C4P3Q) context deadline exceeded ======LOGS====== ) [noise=true , address=145.79.227.7:4296 , piece_id=PVH6VN4CG4O2DC22G74US7IC5GDIGM2KYYYQQPFO5MG22JQCAU2A, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.248055070 (508h48m1.119138115s) [noise=true , address=145.79.227.40:4078 , piece_id=J6TGA3SFQOLGL2C2TS352TSLZBMI6MDXU5BDTTGQYMN4A5QG462A, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.248175840 (508h48m1.119258875s) [noise=true , address=77.90.31.12:4600 , piece_id=JESM5JF4GWUDLEW4YT5SKHJOXD54TWRQBNGGQCBMDSS54XR4AEKQ, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.248604496 (508h48m1.119687531s) [noise=true , address=45.91.116.6:4985 , piece_id=WV5SXZIBMXB4FTNZFE4PKBR5ZWZTUUPADPRMXAVE4Z3HKDGLBOSQ, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.248640474 (508h48m1.119723509s) [noise=true , address=145.79.227.18:4021 , piece_id=ER7GXC5OHAPNGU37QTKPROKB4YTCDQBUHVEIEUH2K5FQ4X5AGM2Q, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.249180202 (508h48m1.120263238s) [noise=true , address=77.90.31.18:4821 , piece_id=HJGRPPEJK75YTIKBKNVK7J4TDRKBCNL7HA4JNMOKD2ROGC7C4P3Q, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.249851130 (508h48m1.120934175s) [noise=true , address=64.49.13.169:4709 , piece_id=ZGWBPAYFOQWU2LSTILYMVRNKGSP53UXEQHRW4NJEXW5OLNVCCWRQ, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.250638910 (508h48m1.121721955s) [noise=true , address=145.79.227.38:4001 , piece_id=TWPCSZEWSFC2ZT7QNWHRS3RY42LLBPIEF3EIQQ65OXFZSHWAV2GA, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.251000769 (508h48m1.122083814s) [noise=true , address=145.79.227.30:4792 , piece_id=RLEUVLBAOMJSDAYHMQJGKYQUC6NYYGSGUVIU5Z53XGBINIQV43GQ, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.251314737 (508h48m1.122397772s) [noise=true , address=145.79.227.5:4252 , piece_id=FI5RNNRKSIYHR3JKCOW3THJQKHAPYLUDDHT3Y4TRYU3OSMCSWQQQ, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.251635527 (508h48m1.122718572s) [noise=true , address=77.90.31.11:4588 , piece_id=ZWDOVL5PGZ4YPY6NZVVSBSWXDDD3DAJ4RUK2BWMOGTHDGWJ2G4DA, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.251919499 (508h48m1.123002544s) [noise=true , address=77.90.31.10:4536 , piece_id=V43VRDKXTE2PHOBPSQUIJPKMKY3ROAEQAI2D4XIJHOLQ3UIMN4RA, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.252236542 (508h48m1.123319588s) [noise=true , address=77.90.31.21:4894 , piece_id=AGL6GVJX2YLGHSZX33SUURUU5S22CF3YRMJTP3QMHHP7OKX2MFSA, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.252528137 (508h48m1.123611182s) [noise=true , address=77.90.31.15:4702 , piece_id=KJEPR7NMX4UHQZHTMDDJKBRLLQ3O5BLJSLYBXGCOFGP3LNC3FXGA, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.252861402 (508h48m1.123944447s) [noise=true , address=64.49.13.27:4961 , piece_id=XCVH62CEVRYGR53U5VG4L4HJMBJSQ562L4QVKTXZSC7XGAEB4LDQ, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:02.570977410 (508h48m2.442060455s) [upload=446477]: Next returned nil segment Dec 26 20:17:02.571003019 (508h48m2.442086054s) [upload=446477]: Waiting for error group managing segments... Dec 26 20:17:02.949353997 (508h48m2.820437042s) [noise=true , address=64.49.13.27:4961 , piece_id=XCVH62CEVRYGR53U5VG4L4HJMBJSQ562L4QVKTXZSC7XGAEB4LDQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.136546190 (508h48m3.007629235s) [noise=true , address=145.79.227.5:4252 , piece_id=FI5RNNRKSIYHR3JKCOW3THJQKHAPYLUDDHT3Y4TRYU3OSMCSWQQQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.155760468 (508h48m3.026843513s) [noise=true , address=145.79.227.38:4001 , piece_id=TWPCSZEWSFC2ZT7QNWHRS3RY42LLBPIEF3EIQQ65OXFZSHWAV2GA, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.179239646 (508h48m3.050322691s) [noise=true , address=145.79.227.18:4021 , piece_id=ER7GXC5OHAPNGU37QTKPROKB4YTCDQBUHVEIEUH2K5FQ4X5AGM2Q, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.372384563 (508h48m3.243467608s) [noise=true , address=77.90.31.15:4702 , piece_id=KJEPR7NMX4UHQZHTMDDJKBRLLQ3O5BLJSLYBXGCOFGP3LNC3FXGA, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.447635803 (508h48m3.318718848s) [noise=true , address=77.90.31.18:4821 , piece_id=HJGRPPEJK75YTIKBKNVK7J4TDRKBCNL7HA4JNMOKD2ROGC7C4P3Q, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.452148166 (508h48m3.323231201s) [noise=true , address=77.90.31.10:4536 , piece_id=V43VRDKXTE2PHOBPSQUIJPKMKY3ROAEQAI2D4XIJHOLQ3UIMN4RA, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.644263002 (508h48m3.515346047s) [noise=true , address=77.90.31.14:4689 , piece_id=PWVMOEKW5A4ZAVSVJAAHFTYYTQZGFHDF642GWR6EABEGWP2CDRGQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.710620497 (508h48m3.581703542s) [noise=true , address=45.91.116.8:4074 , piece_id=ME2LDK2QJI7LLG67ODDIJDKB5CAT47YDGMJ3CTVK4F6UDHJ4TPOQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:04.066910008 (508h48m3.937993053s) [noise=true , address=77.90.31.12:4600 , piece_id=JESM5JF4GWUDLEW4YT5SKHJOXD54TWRQBNGGQCBMDSS54XR4AEKQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:04.082282349 (508h48m3.953365394s) [noise=true , address=145.79.227.44:4219 , piece_id=KZVV2NJBIU22C4URLBZMQVQN4IZ2ZMQ5YODRJMCS6XYXULHCZWLQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:04.089048724 (508h48m3.960131759s) [noise=true , address=145.79.227.30:4792 , piece_id=RLEUVLBAOMJSDAYHMQJGKYQUC6NYYGSGUVIU5Z53XGBINIQV43GQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:04.288772749 (508h48m4.159855794s) [noise=true , address=77.90.31.21:4894 , piece_id=AGL6GVJX2YLGHSZX33SUURUU5S22CF3YRMJTP3QMHHP7OKX2MFSA, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:04.344183239 (508h48m4.215266294s) [noise=true , address=77.90.31.11:4588 , piece_id=ZWDOVL5PGZ4YPY6NZVVSBSWXDDD3DAJ4RUK2BWMOGTHDGWJ2G4DA, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:04.344208728 (508h48m4.215291773s) [seg_pos={0 0}, upload=446477]: Segment reached optimal threshold of 14 pieces. Dec 26 20:17:04.344470828 (508h48m4.215553873s) [noise=true , address=64.49.13.169:4709 , piece_id=ZGWBPAYFOQWU2LSTILYMVRNKGSP53UXEQHRW4NJEXW5OLNVCCWRQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:47062->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:47062->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47062->64.49.13.169:4709: use of closed network connection Dec 26 20:17:04.344481918 (508h48m4.215564953s) [noise=true , address=64.49.13.169:4709 , piece_id=ZGWBPAYFOQWU2LSTILYMVRNKGSP53UXEQHRW4NJEXW5OLNVCCWRQ, seg_pos={0 0}, upload=446477]: Long tail cancellation due to optimal threshold Dec 26 20:17:04.344541813 (508h48m4.215624848s) [noise=true , address=145.79.227.40:4078 , piece_id=J6TGA3SFQOLGL2C2TS352TSLZBMI6MDXU5BDTTGQYMN4A5QG462A, seg_pos={0 0}, upload=446477]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tzgtenV6us5ssD7xWGpZmVUoWEfbFhBNewbK9irHtZPkyDtRd): protocol: expected piece hash; context canceled; EOF Dec 26 20:17:04.344559476 (508h48m4.215642511s) [noise=true , address=145.79.227.40:4078 , piece_id=J6TGA3SFQOLGL2C2TS352TSLZBMI6MDXU5BDTTGQYMN4A5QG462A, seg_pos={0 0}, upload=446477]: Long tail cancellation due to optimal threshold Dec 26 20:17:04.344755479 (508h48m4.215838524s) [noise=true , address=145.79.227.7:4296 , piece_id=PVH6VN4CG4O2DC22G74US7IC5GDIGM2KYYYQQPFO5MG22JQCAU2A, seg_pos={0 0}, upload=446477]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cKhDbA4ptwux2LYNPNAJ7Qh8jECsGWd1KWaK6T6pD38bJWdt1): protocol: write tcp 188.166.225.170:48104->145.79.227.7:4296: use of closed network connection; write tcp 188.166.225.170:48104->145.79.227.7:4296: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48104->145.79.227.7:4296: use of closed network connection Dec 26 20:17:04.344763044 (508h48m4.215846099s) [noise=true , address=145.79.227.7:4296 , piece_id=PVH6VN4CG4O2DC22G74US7IC5GDIGM2KYYYQQPFO5MG22JQCAU2A, seg_pos={0 0}, upload=446477]: Long tail cancellation due to optimal threshold Dec 26 20:17:04.344928509 (508h48m4.216011544s) [noise=true , address=45.91.116.6:4985 , piece_id=WV5SXZIBMXB4FTNZFE4PKBR5ZWZTUUPADPRMXAVE4Z3HKDGLBOSQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RNs9MM9jDtNC2gZC63SYsePZTd5mE7dqUm9A5Z5QWdPX8PVnp): protocol: expected piece hash; context canceled; EOF Dec 26 20:17:04.344938277 (508h48m4.216021312s) [noise=true , address=45.91.116.6:4985 , piece_id=WV5SXZIBMXB4FTNZFE4PKBR5ZWZTUUPADPRMXAVE4Z3HKDGLBOSQ, seg_pos={0 0}, upload=446477]: Long tail cancellation due to optimal threshold Dec 26 20:17:04.344966722 (508h48m4.216049767s) [seg_pos={0 0}, upload=446477]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:17:04.345054548 (508h48m4.216137593s) [upload=446477]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:17:04.345059839 (508h48m4.216142874s) [upload=446477]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:17:04.653613492 (508h48m4.524696537s) [upload=446477]: Done uploading segments... Dec 26 20:17:04.653639822 (508h48m4.524722857s) [upload=446477]: Upload finished. err: <nil> Duration: 34.045350044s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.X.X..............XX..X..X...........XXX.X.X....X..XXXX.X...] duration: 7.554792065s trace: https://jaeger.dev.storj.io/trace/67714c2c4f903cf FAIL (2025-12-26 20:26:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 20:24:55.494029472 (508h55m55.365112517s) [upload=446586]: Starting upload Dec 26 20:24:55.494087212 (508h55m55.365170247s) [upload=446586]: Done starting upload Dec 26 20:24:55.494119003 (508h55m55.365202038s) [upload=446586]: Uploading segments... Dec 26 20:24:55.494252457 (508h55m55.365335492s) [upload=446586]: Got next segment. Inline: false Dec 26 20:24:55.494307782 (508h55m55.365390817s) [upload=446586]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 20:24:55.494315337 (508h55m55.365398382s) [upload=446586]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 20:24:55.746810137 (508h55m55.617893182s) [seg_pos={0 0}, upload=446586]: Begin upload segment... Dec 26 20:24:55.746856535 (508h55m55.61793958s) [seg_pos={0 0}, upload=446586]: Done begin upload segment. Dec 26 20:24:55.746885150 (508h55m55.617968185s) [noise=true , address=45.91.116.4:4939 , piece_id=Y4QBV6V2BBYLRA4LGSPN7JVCKWPIOB3U6M5IFZHU4DZQDWMDW5MA, seg_pos={0 0}, upload=446586]: Uploading piece... Dec 26 20:24:55.746896232 (508h55m55.617979267s) [noise=true , address=145.79.227.41:4085 , piece_id=DWGRFX2BYT2EHMHMR2K4OOJWTL6YYTVUQPSCLLNCNNYPDEZHJM2A, seg_pos={0 0}, upload=446586]: Uploading piece... Dec 26 20:24:55.747327733 (508h55m55.618410768s) [noise=true , address=77.90.31.19:4841 , piece_id=GMMTZZSSVPWM7DIJM6KZZMPV36JFPLZBZ72BE6XU55UTCFIPGQFA, seg_pos={0 0}, upload=446586]: Uploading piece... Dec 26 20:24:55.747716052 (508h55m55.618799087s) [noise=true , address=145.79.227.45:4229 , piece_id=E4OQU7EFYLK2R53DK56TETV72RM3LYT2ZPRJ5N646XKB2TZ2HOEQ, seg_pos={0 0}, upload=446586]: Uploading piece... Dec 26 20:24:55.747929068 (508h55m55.619012113s) [noise=true , address=145.79.227.50:4374 , piece_id=5YJ4C3TD3F2QIGIKODNYETR5MWDVKVWLGOCO6V3JT5HLPEPX5IWQ, seg_pos={0 0}, upload=446586]: Uploading piece... Dec 26 20:24:55.748359348 (508h55m55.619442383s) [noise=true , address=145.79.227.25:4618 , piece_id=G3ANK36VSN3ZKEG73LH5NLQGIQF5UJV6AN2TIFXPBNBK6RKWKMQQ, seg_pos={0 0}, upload=446586]: Uploading piece... Dec 26 20:24:55.748668495 (508h55m55.61975153s) [noise=true , address=145.79.227.39:4015 , piece_id=32BQH7QZLOHZ7AAZNVNG66WG7RQ774CGADSOJ7CKF3YJ36CDKZAA, seg_pos={0 0}, upload=446586]: Uploading piece... Dec 26 20:24:55.749022230 (508h55m55.620105265s) [noise=true , address=145.79.227.34:4905 , piece_id=O6LX6GU3ANZUY2V4PSTLUHDYXHWKGAS6SO4C4TNBUOTS7HQEWDJQ, seg_pos={0 0}, upload=446586]: Uploading piece... Dec 26 20:24:55.749295589 (508h55m55.620378624s) [noise=true , address=45.91.116.10:4081 , piece_id=HIOEKRJX6O575VMHJ7FW7D5RMOLJIA4X7IOJTLXN75BVIHE2SU6Q, seg_pos={0 0}, upload=446586]: Uploading piece... Dec 26 20:24:55.749557589 (508h55m55.620640624s) [noise=true , address=145.79.227.22:4509 , piece_id=35ECUPNQRNUEBKYMWS4LM2UXIHOCQ42TMBLMXVYDQJN4NOIW326Q, seg_pos={0 0}, upload=446586]: Uploading piece... Dec 26 20:24:55.749869994 (508h55m55.620953029s) [noise=true , address=145.79.227.21:4475 , piece_id=TP4LY24LYGV7Q65B7HNGNL6FETPT7U2D2HALRVUZIVJOELZ3NAIQ, seg_pos={0 0}, upload=446586]: Uploading piece... Dec 26 20:24:55.750031462 (508h55m55.621114497s) [noise=true , address=77.90.31.11:4554 , piece_id=TUQGNI4GBUW65O6L65JCF7YEKMQEJSTXZ5RB3Q3IEW36P3BFOA3Q, seg_pos={0 0}, upload=446586]: Uploading piece... Dec 26 20:24:55.750160987 (508h55m55.621244023s) [noise=true , address=45.91.116.13:4154 , piece_id=ZVBTPMKY2IYPRZ3PWETRSX6UYRZWJJX7CFYBGFHNZOGRG7SLRQKA, seg_pos={0 0}, upload=446586]: Uploading piece... Dec 26 20:24:55.750520222 (508h55m55.621603267s) [noise=true , address=145.79.227.11:4729 , piece_id=RF6IIANE5JBJUQ3OJOEHOKZHJRCSFWVFVXN5DYD64CUVAS6ZBOFA, seg_pos={0 0}, upload=446586]: Uploading piece... Dec 26 20:24:55.750621134 (508h55m55.621704169s) [noise=true , address=145.79.227.38:4977 , piece_id=4MHHY3I6ETVPLRMFPHDRC7GISU2SUOB2V5FIVCDDVSHEODEZIO4A, seg_pos={0 0}, upload=446586]: Uploading piece... Dec 26 20:24:55.750986900 (508h55m55.622069935s) [noise=true , address=145.79.227.36:4518 , piece_id=QUMDUDYNZM6LKOIPDZFT2463AOPS6XNV2T4LVXKFNQS6I23ADB3Q, seg_pos={0 0}, upload=446586]: Uploading piece... Dec 26 20:24:56.872741053 (508h55m56.743824108s) [noise=true , address=77.90.31.19:4841 , piece_id=GMMTZZSSVPWM7DIJM6KZZMPV36JFPLZBZ72BE6XU55UTCFIPGQFA, seg_pos={0 0}, upload=446586]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 26 20:24:57.460574714 (508h55m57.331657749s) [upload=446586]: Next returned nil segment Dec 26 20:24:57.460586987 (508h55m57.331670032s) [upload=446586]: Waiting for error group managing segments... Dec 26 20:24:57.749972475 (508h55m57.62105551s) [noise=true , address=145.79.227.45:4229 , piece_id=E4OQU7EFYLK2R53DK56TETV72RM3LYT2ZPRJ5N646XKB2TZ2HOEQ, seg_pos={0 0}, upload=446586]: Done uploading piece. err: <nil> Dec 26 20:24:57.750073327 (508h55m57.621156372s) [noise=true , address=145.79.227.25:4618 , piece_id=G3ANK36VSN3ZKEG73LH5NLQGIQF5UJV6AN2TIFXPBNBK6RKWKMQQ, seg_pos={0 0}, upload=446586]: Done uploading piece. err: <nil> Dec 26 20:24:57.751036380 (508h55m57.622119415s) [noise=true , address=145.79.227.38:4977 , piece_id=4MHHY3I6ETVPLRMFPHDRC7GISU2SUOB2V5FIVCDDVSHEODEZIO4A, seg_pos={0 0}, upload=446586]: Done uploading piece. err: <nil> Dec 26 20:24:57.752579279 (508h55m57.623662314s) [noise=true , address=145.79.227.50:4374 , piece_id=5YJ4C3TD3F2QIGIKODNYETR5MWDVKVWLGOCO6V3JT5HLPEPX5IWQ, seg_pos={0 0}, upload=446586]: Done uploading piece. err: <nil> Dec 26 20:24:57.753179982 (508h55m57.624263027s) [noise=true , address=145.79.227.36:4518 , piece_id=QUMDUDYNZM6LKOIPDZFT2463AOPS6XNV2T4LVXKFNQS6I23ADB3Q, seg_pos={0 0}, upload=446586]: Done uploading piece. err: <nil> Dec 26 20:24:57.754838511 (508h55m57.625921546s) [noise=true , address=145.79.227.22:4509 , piece_id=35ECUPNQRNUEBKYMWS4LM2UXIHOCQ42TMBLMXVYDQJN4NOIW326Q, seg_pos={0 0}, upload=446586]: Done uploading piece. err: <nil> Dec 26 20:24:57.774271073 (508h55m57.645354118s) [noise=true , address=145.79.227.21:4475 , piece_id=TP4LY24LYGV7Q65B7HNGNL6FETPT7U2D2HALRVUZIVJOELZ3NAIQ, seg_pos={0 0}, upload=446586]: Done uploading piece. err: <nil> Dec 26 20:24:57.774868932 (508h55m57.645951967s) [noise=true , address=145.79.227.11:4729 , piece_id=RF6IIANE5JBJUQ3OJOEHOKZHJRCSFWVFVXN5DYD64CUVAS6ZBOFA, seg_pos={0 0}, upload=446586]: Done uploading piece. err: <nil> Dec 26 20:24:57.788847749 (508h55m57.659930784s) [noise=true , address=145.79.227.34:4905 , piece_id=O6LX6GU3ANZUY2V4PSTLUHDYXHWKGAS6SO4C4TNBUOTS7HQEWDJQ, seg_pos={0 0}, upload=446586]: Done uploading piece. err: <nil> Dec 26 20:24:57.994242624 (508h55m57.865325659s) [noise=true , address=45.91.116.4:4939 , piece_id=Y4QBV6V2BBYLRA4LGSPN7JVCKWPIOB3U6M5IFZHU4DZQDWMDW5MA, seg_pos={0 0}, upload=446586]: Done uploading piece. err: <nil> Dec 26 20:24:58.202877610 (508h55m58.073960645s) [noise=true , address=45.91.116.13:4154 , piece_id=ZVBTPMKY2IYPRZ3PWETRSX6UYRZWJJX7CFYBGFHNZOGRG7SLRQKA, seg_pos={0 0}, upload=446586]: Done uploading piece. err: <nil> Dec 26 20:24:58.484650482 (508h55m58.355733527s) [noise=true , address=45.91.116.10:4081 , piece_id=HIOEKRJX6O575VMHJ7FW7D5RMOLJIA4X7IOJTLXN75BVIHE2SU6Q, seg_pos={0 0}, upload=446586]: Done uploading piece. err: <nil> Dec 26 20:24:59.192956101 (508h55m59.064039146s) [noise=true , address=145.79.227.41:4085 , piece_id=DWGRFX2BYT2EHMHMR2K4OOJWTL6YYTVUQPSCLLNCNNYPDEZHJM2A, seg_pos={0 0}, upload=446586]: Done uploading piece. err: <nil> Dec 26 20:24:59.192979436 (508h55m59.064062481s) [seg_pos={0 0}, upload=446586]: Segment reached optimal threshold of 13 pieces. Dec 26 20:24:59.193178264 (508h55m59.064261309s) [noise=true , address=77.90.31.11:4554 , piece_id=TUQGNI4GBUW65O6L65JCF7YEKMQEJSTXZ5RB3Q3IEW36P3BFOA3Q, seg_pos={0 0}, upload=446586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Dec 26 20:24:59.193187893 (508h55m59.064270938s) [noise=true , address=77.90.31.11:4554 , piece_id=TUQGNI4GBUW65O6L65JCF7YEKMQEJSTXZ5RB3Q3IEW36P3BFOA3Q, seg_pos={0 0}, upload=446586]: Long tail cancellation due to optimal threshold Dec 26 20:24:59.193500819 (508h55m59.064583864s) [noise=true , address=145.79.227.39:4015 , piece_id=32BQH7QZLOHZ7AAZNVNG66WG7RQ774CGADSOJ7CKF3YJ36CDKZAA, seg_pos={0 0}, upload=446586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE): protocol: expected piece hash; context canceled; EOF Dec 26 20:24:59.193509996 (508h55m59.064593041s) [noise=true , address=145.79.227.39:4015 , piece_id=32BQH7QZLOHZ7AAZNVNG66WG7RQ774CGADSOJ7CKF3YJ36CDKZAA, seg_pos={0 0}, upload=446586]: Long tail cancellation due to optimal threshold Dec 26 20:24:59.193528431 (508h55m59.064611466s) [seg_pos={0 0}, upload=446586]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 20:24:59.193607642 (508h55m59.064690687s) [upload=446586]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:24:59.193615136 (508h55m59.064698171s) [upload=446586]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:24:59.497679804 (508h55m59.368762869s) [upload=446586]: Done uploading segments... Dec 26 20:24:59.497700765 (508h55m59.36878382s) [upload=446586]: Upload finished. err: <nil> Duration: 7.554827993s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [....X..........................................X............] duration: 31.023325425s trace: https://jaeger.dev.storj.io/trace/663e21624a5a3654 FAIL (2025-12-26 20:26:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, Piece ID: PWVMOEKW5A4ZAVSVJAAHFTYYTQZGFHDF642GWR6EABEGWP2CDRGQ) context deadline exceeded; piecestore: (Node ID: 12whTD1Kvu4T7ANNtkSvDB66uTjVgfPTK9TAzExgvwJYgZ3zSFG, Piece ID: ME2LDK2QJI7LLG67ODDIJDKB5CAT47YDGMJ3CTVK4F6UDHJ4TPOQ) context deadline exceeded; piecestore: (Node ID: 12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH, Piece ID: HJGRPPEJK75YTIKBKNVK7J4TDRKBCNL7HA4JNMOKD2ROGC7C4P3Q) context deadline exceeded ======LOGS====== ) [noise=true , address=145.79.227.7:4296 , piece_id=PVH6VN4CG4O2DC22G74US7IC5GDIGM2KYYYQQPFO5MG22JQCAU2A, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.248055070 (508h48m1.119138115s) [noise=true , address=145.79.227.40:4078 , piece_id=J6TGA3SFQOLGL2C2TS352TSLZBMI6MDXU5BDTTGQYMN4A5QG462A, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.248175840 (508h48m1.119258875s) [noise=true , address=77.90.31.12:4600 , piece_id=JESM5JF4GWUDLEW4YT5SKHJOXD54TWRQBNGGQCBMDSS54XR4AEKQ, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.248604496 (508h48m1.119687531s) [noise=true , address=45.91.116.6:4985 , piece_id=WV5SXZIBMXB4FTNZFE4PKBR5ZWZTUUPADPRMXAVE4Z3HKDGLBOSQ, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.248640474 (508h48m1.119723509s) [noise=true , address=145.79.227.18:4021 , piece_id=ER7GXC5OHAPNGU37QTKPROKB4YTCDQBUHVEIEUH2K5FQ4X5AGM2Q, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.249180202 (508h48m1.120263238s) [noise=true , address=77.90.31.18:4821 , piece_id=HJGRPPEJK75YTIKBKNVK7J4TDRKBCNL7HA4JNMOKD2ROGC7C4P3Q, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.249851130 (508h48m1.120934175s) [noise=true , address=64.49.13.169:4709 , piece_id=ZGWBPAYFOQWU2LSTILYMVRNKGSP53UXEQHRW4NJEXW5OLNVCCWRQ, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.250638910 (508h48m1.121721955s) [noise=true , address=145.79.227.38:4001 , piece_id=TWPCSZEWSFC2ZT7QNWHRS3RY42LLBPIEF3EIQQ65OXFZSHWAV2GA, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.251000769 (508h48m1.122083814s) [noise=true , address=145.79.227.30:4792 , piece_id=RLEUVLBAOMJSDAYHMQJGKYQUC6NYYGSGUVIU5Z53XGBINIQV43GQ, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.251314737 (508h48m1.122397772s) [noise=true , address=145.79.227.5:4252 , piece_id=FI5RNNRKSIYHR3JKCOW3THJQKHAPYLUDDHT3Y4TRYU3OSMCSWQQQ, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.251635527 (508h48m1.122718572s) [noise=true , address=77.90.31.11:4588 , piece_id=ZWDOVL5PGZ4YPY6NZVVSBSWXDDD3DAJ4RUK2BWMOGTHDGWJ2G4DA, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.251919499 (508h48m1.123002544s) [noise=true , address=77.90.31.10:4536 , piece_id=V43VRDKXTE2PHOBPSQUIJPKMKY3ROAEQAI2D4XIJHOLQ3UIMN4RA, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.252236542 (508h48m1.123319588s) [noise=true , address=77.90.31.21:4894 , piece_id=AGL6GVJX2YLGHSZX33SUURUU5S22CF3YRMJTP3QMHHP7OKX2MFSA, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.252528137 (508h48m1.123611182s) [noise=true , address=77.90.31.15:4702 , piece_id=KJEPR7NMX4UHQZHTMDDJKBRLLQ3O5BLJSLYBXGCOFGP3LNC3FXGA, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.252861402 (508h48m1.123944447s) [noise=true , address=64.49.13.27:4961 , piece_id=XCVH62CEVRYGR53U5VG4L4HJMBJSQ562L4QVKTXZSC7XGAEB4LDQ, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:02.570977410 (508h48m2.442060455s) [upload=446477]: Next returned nil segment Dec 26 20:17:02.571003019 (508h48m2.442086054s) [upload=446477]: Waiting for error group managing segments... Dec 26 20:17:02.949353997 (508h48m2.820437042s) [noise=true , address=64.49.13.27:4961 , piece_id=XCVH62CEVRYGR53U5VG4L4HJMBJSQ562L4QVKTXZSC7XGAEB4LDQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.136546190 (508h48m3.007629235s) [noise=true , address=145.79.227.5:4252 , piece_id=FI5RNNRKSIYHR3JKCOW3THJQKHAPYLUDDHT3Y4TRYU3OSMCSWQQQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.155760468 (508h48m3.026843513s) [noise=true , address=145.79.227.38:4001 , piece_id=TWPCSZEWSFC2ZT7QNWHRS3RY42LLBPIEF3EIQQ65OXFZSHWAV2GA, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.179239646 (508h48m3.050322691s) [noise=true , address=145.79.227.18:4021 , piece_id=ER7GXC5OHAPNGU37QTKPROKB4YTCDQBUHVEIEUH2K5FQ4X5AGM2Q, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.372384563 (508h48m3.243467608s) [noise=true , address=77.90.31.15:4702 , piece_id=KJEPR7NMX4UHQZHTMDDJKBRLLQ3O5BLJSLYBXGCOFGP3LNC3FXGA, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.447635803 (508h48m3.318718848s) [noise=true , address=77.90.31.18:4821 , piece_id=HJGRPPEJK75YTIKBKNVK7J4TDRKBCNL7HA4JNMOKD2ROGC7C4P3Q, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.452148166 (508h48m3.323231201s) [noise=true , address=77.90.31.10:4536 , piece_id=V43VRDKXTE2PHOBPSQUIJPKMKY3ROAEQAI2D4XIJHOLQ3UIMN4RA, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.644263002 (508h48m3.515346047s) [noise=true , address=77.90.31.14:4689 , piece_id=PWVMOEKW5A4ZAVSVJAAHFTYYTQZGFHDF642GWR6EABEGWP2CDRGQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.710620497 (508h48m3.581703542s) [noise=true , address=45.91.116.8:4074 , piece_id=ME2LDK2QJI7LLG67ODDIJDKB5CAT47YDGMJ3CTVK4F6UDHJ4TPOQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:04.066910008 (508h48m3.937993053s) [noise=true , address=77.90.31.12:4600 , piece_id=JESM5JF4GWUDLEW4YT5SKHJOXD54TWRQBNGGQCBMDSS54XR4AEKQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:04.082282349 (508h48m3.953365394s) [noise=true , address=145.79.227.44:4219 , piece_id=KZVV2NJBIU22C4URLBZMQVQN4IZ2ZMQ5YODRJMCS6XYXULHCZWLQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:04.089048724 (508h48m3.960131759s) [noise=true , address=145.79.227.30:4792 , piece_id=RLEUVLBAOMJSDAYHMQJGKYQUC6NYYGSGUVIU5Z53XGBINIQV43GQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:04.288772749 (508h48m4.159855794s) [noise=true , address=77.90.31.21:4894 , piece_id=AGL6GVJX2YLGHSZX33SUURUU5S22CF3YRMJTP3QMHHP7OKX2MFSA, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:04.344183239 (508h48m4.215266294s) [noise=true , address=77.90.31.11:4588 , piece_id=ZWDOVL5PGZ4YPY6NZVVSBSWXDDD3DAJ4RUK2BWMOGTHDGWJ2G4DA, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:04.344208728 (508h48m4.215291773s) [seg_pos={0 0}, upload=446477]: Segment reached optimal threshold of 14 pieces. Dec 26 20:17:04.344470828 (508h48m4.215553873s) [noise=true , address=64.49.13.169:4709 , piece_id=ZGWBPAYFOQWU2LSTILYMVRNKGSP53UXEQHRW4NJEXW5OLNVCCWRQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:47062->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:47062->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47062->64.49.13.169:4709: use of closed network connection Dec 26 20:17:04.344481918 (508h48m4.215564953s) [noise=true , address=64.49.13.169:4709 , piece_id=ZGWBPAYFOQWU2LSTILYMVRNKGSP53UXEQHRW4NJEXW5OLNVCCWRQ, seg_pos={0 0}, upload=446477]: Long tail cancellation due to optimal threshold Dec 26 20:17:04.344541813 (508h48m4.215624848s) [noise=true , address=145.79.227.40:4078 , piece_id=J6TGA3SFQOLGL2C2TS352TSLZBMI6MDXU5BDTTGQYMN4A5QG462A, seg_pos={0 0}, upload=446477]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tzgtenV6us5ssD7xWGpZmVUoWEfbFhBNewbK9irHtZPkyDtRd): protocol: expected piece hash; context canceled; EOF Dec 26 20:17:04.344559476 (508h48m4.215642511s) [noise=true , address=145.79.227.40:4078 , piece_id=J6TGA3SFQOLGL2C2TS352TSLZBMI6MDXU5BDTTGQYMN4A5QG462A, seg_pos={0 0}, upload=446477]: Long tail cancellation due to optimal threshold Dec 26 20:17:04.344755479 (508h48m4.215838524s) [noise=true , address=145.79.227.7:4296 , piece_id=PVH6VN4CG4O2DC22G74US7IC5GDIGM2KYYYQQPFO5MG22JQCAU2A, seg_pos={0 0}, upload=446477]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cKhDbA4ptwux2LYNPNAJ7Qh8jECsGWd1KWaK6T6pD38bJWdt1): protocol: write tcp 188.166.225.170:48104->145.79.227.7:4296: use of closed network connection; write tcp 188.166.225.170:48104->145.79.227.7:4296: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48104->145.79.227.7:4296: use of closed network connection Dec 26 20:17:04.344763044 (508h48m4.215846099s) [noise=true , address=145.79.227.7:4296 , piece_id=PVH6VN4CG4O2DC22G74US7IC5GDIGM2KYYYQQPFO5MG22JQCAU2A, seg_pos={0 0}, upload=446477]: Long tail cancellation due to optimal threshold Dec 26 20:17:04.344928509 (508h48m4.216011544s) [noise=true , address=45.91.116.6:4985 , piece_id=WV5SXZIBMXB4FTNZFE4PKBR5ZWZTUUPADPRMXAVE4Z3HKDGLBOSQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RNs9MM9jDtNC2gZC63SYsePZTd5mE7dqUm9A5Z5QWdPX8PVnp): protocol: expected piece hash; context canceled; EOF Dec 26 20:17:04.344938277 (508h48m4.216021312s) [noise=true , address=45.91.116.6:4985 , piece_id=WV5SXZIBMXB4FTNZFE4PKBR5ZWZTUUPADPRMXAVE4Z3HKDGLBOSQ, seg_pos={0 0}, upload=446477]: Long tail cancellation due to optimal threshold Dec 26 20:17:04.344966722 (508h48m4.216049767s) [seg_pos={0 0}, upload=446477]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:17:04.345054548 (508h48m4.216137593s) [upload=446477]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:17:04.345059839 (508h48m4.216142874s) [upload=446477]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:17:04.653613492 (508h48m4.524696537s) [upload=446477]: Done uploading segments... Dec 26 20:17:04.653639822 (508h48m4.524722857s) [upload=446477]: Upload finished. err: <nil> Duration: 31.023354321s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [X.X.X..............XX..X..X...........XXX.X.X....X..XXXX.X..] duration: 5.974081999s trace: https://jaeger.dev.storj.io/trace/67714c2c4f903cf FAIL (2025-12-26 20:26:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 20:24:55.494029472 (508h55m55.365112517s) [upload=446586]: Starting upload Dec 26 20:24:55.494087212 (508h55m55.365170247s) [upload=446586]: Done starting upload Dec 26 20:24:55.494119003 (508h55m55.365202038s) [upload=446586]: Uploading segments... Dec 26 20:24:55.494252457 (508h55m55.365335492s) [upload=446586]: Got next segment. Inline: false Dec 26 20:24:55.494307782 (508h55m55.365390817s) [upload=446586]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 20:24:55.494315337 (508h55m55.365398382s) [upload=446586]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 20:24:55.746810137 (508h55m55.617893182s) [seg_pos={0 0}, upload=446586]: Begin upload segment... Dec 26 20:24:55.746856535 (508h55m55.61793958s) [seg_pos={0 0}, upload=446586]: Done begin upload segment. Dec 26 20:24:55.746885150 (508h55m55.617968185s) [noise=true , address=45.91.116.4:4939 , piece_id=Y4QBV6V2BBYLRA4LGSPN7JVCKWPIOB3U6M5IFZHU4DZQDWMDW5MA, seg_pos={0 0}, upload=446586]: Uploading piece... Dec 26 20:24:55.746896232 (508h55m55.617979267s) [noise=true , address=145.79.227.41:4085 , piece_id=DWGRFX2BYT2EHMHMR2K4OOJWTL6YYTVUQPSCLLNCNNYPDEZHJM2A, seg_pos={0 0}, upload=446586]: Uploading piece... Dec 26 20:24:55.747327733 (508h55m55.618410768s) [noise=true , address=77.90.31.19:4841 , piece_id=GMMTZZSSVPWM7DIJM6KZZMPV36JFPLZBZ72BE6XU55UTCFIPGQFA, seg_pos={0 0}, upload=446586]: Uploading piece... Dec 26 20:24:55.747716052 (508h55m55.618799087s) [noise=true , address=145.79.227.45:4229 , piece_id=E4OQU7EFYLK2R53DK56TETV72RM3LYT2ZPRJ5N646XKB2TZ2HOEQ, seg_pos={0 0}, upload=446586]: Uploading piece... Dec 26 20:24:55.747929068 (508h55m55.619012113s) [noise=true , address=145.79.227.50:4374 , piece_id=5YJ4C3TD3F2QIGIKODNYETR5MWDVKVWLGOCO6V3JT5HLPEPX5IWQ, seg_pos={0 0}, upload=446586]: Uploading piece... Dec 26 20:24:55.748359348 (508h55m55.619442383s) [noise=true , address=145.79.227.25:4618 , piece_id=G3ANK36VSN3ZKEG73LH5NLQGIQF5UJV6AN2TIFXPBNBK6RKWKMQQ, seg_pos={0 0}, upload=446586]: Uploading piece... Dec 26 20:24:55.748668495 (508h55m55.61975153s) [noise=true , address=145.79.227.39:4015 , piece_id=32BQH7QZLOHZ7AAZNVNG66WG7RQ774CGADSOJ7CKF3YJ36CDKZAA, seg_pos={0 0}, upload=446586]: Uploading piece... Dec 26 20:24:55.749022230 (508h55m55.620105265s) [noise=true , address=145.79.227.34:4905 , piece_id=O6LX6GU3ANZUY2V4PSTLUHDYXHWKGAS6SO4C4TNBUOTS7HQEWDJQ, seg_pos={0 0}, upload=446586]: Uploading piece... Dec 26 20:24:55.749295589 (508h55m55.620378624s) [noise=true , address=45.91.116.10:4081 , piece_id=HIOEKRJX6O575VMHJ7FW7D5RMOLJIA4X7IOJTLXN75BVIHE2SU6Q, seg_pos={0 0}, upload=446586]: Uploading piece... Dec 26 20:24:55.749557589 (508h55m55.620640624s) [noise=true , address=145.79.227.22:4509 , piece_id=35ECUPNQRNUEBKYMWS4LM2UXIHOCQ42TMBLMXVYDQJN4NOIW326Q, seg_pos={0 0}, upload=446586]: Uploading piece... Dec 26 20:24:55.749869994 (508h55m55.620953029s) [noise=true , address=145.79.227.21:4475 , piece_id=TP4LY24LYGV7Q65B7HNGNL6FETPT7U2D2HALRVUZIVJOELZ3NAIQ, seg_pos={0 0}, upload=446586]: Uploading piece... Dec 26 20:24:55.750031462 (508h55m55.621114497s) [noise=true , address=77.90.31.11:4554 , piece_id=TUQGNI4GBUW65O6L65JCF7YEKMQEJSTXZ5RB3Q3IEW36P3BFOA3Q, seg_pos={0 0}, upload=446586]: Uploading piece... Dec 26 20:24:55.750160987 (508h55m55.621244023s) [noise=true , address=45.91.116.13:4154 , piece_id=ZVBTPMKY2IYPRZ3PWETRSX6UYRZWJJX7CFYBGFHNZOGRG7SLRQKA, seg_pos={0 0}, upload=446586]: Uploading piece... Dec 26 20:24:55.750520222 (508h55m55.621603267s) [noise=true , address=145.79.227.11:4729 , piece_id=RF6IIANE5JBJUQ3OJOEHOKZHJRCSFWVFVXN5DYD64CUVAS6ZBOFA, seg_pos={0 0}, upload=446586]: Uploading piece... Dec 26 20:24:55.750621134 (508h55m55.621704169s) [noise=true , address=145.79.227.38:4977 , piece_id=4MHHY3I6ETVPLRMFPHDRC7GISU2SUOB2V5FIVCDDVSHEODEZIO4A, seg_pos={0 0}, upload=446586]: Uploading piece... Dec 26 20:24:55.750986900 (508h55m55.622069935s) [noise=true , address=145.79.227.36:4518 , piece_id=QUMDUDYNZM6LKOIPDZFT2463AOPS6XNV2T4LVXKFNQS6I23ADB3Q, seg_pos={0 0}, upload=446586]: Uploading piece... Dec 26 20:24:56.872741053 (508h55m56.743824108s) [noise=true , address=77.90.31.19:4841 , piece_id=GMMTZZSSVPWM7DIJM6KZZMPV36JFPLZBZ72BE6XU55UTCFIPGQFA, seg_pos={0 0}, upload=446586]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 26 20:24:57.460574714 (508h55m57.331657749s) [upload=446586]: Next returned nil segment Dec 26 20:24:57.460586987 (508h55m57.331670032s) [upload=446586]: Waiting for error group managing segments... Dec 26 20:24:57.749972475 (508h55m57.62105551s) [noise=true , address=145.79.227.45:4229 , piece_id=E4OQU7EFYLK2R53DK56TETV72RM3LYT2ZPRJ5N646XKB2TZ2HOEQ, seg_pos={0 0}, upload=446586]: Done uploading piece. err: <nil> Dec 26 20:24:57.750073327 (508h55m57.621156372s) [noise=true , address=145.79.227.25:4618 , piece_id=G3ANK36VSN3ZKEG73LH5NLQGIQF5UJV6AN2TIFXPBNBK6RKWKMQQ, seg_pos={0 0}, upload=446586]: Done uploading piece. err: <nil> Dec 26 20:24:57.751036380 (508h55m57.622119415s) [noise=true , address=145.79.227.38:4977 , piece_id=4MHHY3I6ETVPLRMFPHDRC7GISU2SUOB2V5FIVCDDVSHEODEZIO4A, seg_pos={0 0}, upload=446586]: Done uploading piece. err: <nil> Dec 26 20:24:57.752579279 (508h55m57.623662314s) [noise=true , address=145.79.227.50:4374 , piece_id=5YJ4C3TD3F2QIGIKODNYETR5MWDVKVWLGOCO6V3JT5HLPEPX5IWQ, seg_pos={0 0}, upload=446586]: Done uploading piece. err: <nil> Dec 26 20:24:57.753179982 (508h55m57.624263027s) [noise=true , address=145.79.227.36:4518 , piece_id=QUMDUDYNZM6LKOIPDZFT2463AOPS6XNV2T4LVXKFNQS6I23ADB3Q, seg_pos={0 0}, upload=446586]: Done uploading piece. err: <nil> Dec 26 20:24:57.754838511 (508h55m57.625921546s) [noise=true , address=145.79.227.22:4509 , piece_id=35ECUPNQRNUEBKYMWS4LM2UXIHOCQ42TMBLMXVYDQJN4NOIW326Q, seg_pos={0 0}, upload=446586]: Done uploading piece. err: <nil> Dec 26 20:24:57.774271073 (508h55m57.645354118s) [noise=true , address=145.79.227.21:4475 , piece_id=TP4LY24LYGV7Q65B7HNGNL6FETPT7U2D2HALRVUZIVJOELZ3NAIQ, seg_pos={0 0}, upload=446586]: Done uploading piece. err: <nil> Dec 26 20:24:57.774868932 (508h55m57.645951967s) [noise=true , address=145.79.227.11:4729 , piece_id=RF6IIANE5JBJUQ3OJOEHOKZHJRCSFWVFVXN5DYD64CUVAS6ZBOFA, seg_pos={0 0}, upload=446586]: Done uploading piece. err: <nil> Dec 26 20:24:57.788847749 (508h55m57.659930784s) [noise=true , address=145.79.227.34:4905 , piece_id=O6LX6GU3ANZUY2V4PSTLUHDYXHWKGAS6SO4C4TNBUOTS7HQEWDJQ, seg_pos={0 0}, upload=446586]: Done uploading piece. err: <nil> Dec 26 20:24:57.994242624 (508h55m57.865325659s) [noise=true , address=45.91.116.4:4939 , piece_id=Y4QBV6V2BBYLRA4LGSPN7JVCKWPIOB3U6M5IFZHU4DZQDWMDW5MA, seg_pos={0 0}, upload=446586]: Done uploading piece. err: <nil> Dec 26 20:24:58.202877610 (508h55m58.073960645s) [noise=true , address=45.91.116.13:4154 , piece_id=ZVBTPMKY2IYPRZ3PWETRSX6UYRZWJJX7CFYBGFHNZOGRG7SLRQKA, seg_pos={0 0}, upload=446586]: Done uploading piece. err: <nil> Dec 26 20:24:58.484650482 (508h55m58.355733527s) [noise=true , address=45.91.116.10:4081 , piece_id=HIOEKRJX6O575VMHJ7FW7D5RMOLJIA4X7IOJTLXN75BVIHE2SU6Q, seg_pos={0 0}, upload=446586]: Done uploading piece. err: <nil> Dec 26 20:24:59.192956101 (508h55m59.064039146s) [noise=true , address=145.79.227.41:4085 , piece_id=DWGRFX2BYT2EHMHMR2K4OOJWTL6YYTVUQPSCLLNCNNYPDEZHJM2A, seg_pos={0 0}, upload=446586]: Done uploading piece. err: <nil> Dec 26 20:24:59.192979436 (508h55m59.064062481s) [seg_pos={0 0}, upload=446586]: Segment reached optimal threshold of 13 pieces. Dec 26 20:24:59.193178264 (508h55m59.064261309s) [noise=true , address=77.90.31.11:4554 , piece_id=TUQGNI4GBUW65O6L65JCF7YEKMQEJSTXZ5RB3Q3IEW36P3BFOA3Q, seg_pos={0 0}, upload=446586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Dec 26 20:24:59.193187893 (508h55m59.064270938s) [noise=true , address=77.90.31.11:4554 , piece_id=TUQGNI4GBUW65O6L65JCF7YEKMQEJSTXZ5RB3Q3IEW36P3BFOA3Q, seg_pos={0 0}, upload=446586]: Long tail cancellation due to optimal threshold Dec 26 20:24:59.193500819 (508h55m59.064583864s) [noise=true , address=145.79.227.39:4015 , piece_id=32BQH7QZLOHZ7AAZNVNG66WG7RQ774CGADSOJ7CKF3YJ36CDKZAA, seg_pos={0 0}, upload=446586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE): protocol: expected piece hash; context canceled; EOF Dec 26 20:24:59.193509996 (508h55m59.064593041s) [noise=true , address=145.79.227.39:4015 , piece_id=32BQH7QZLOHZ7AAZNVNG66WG7RQ774CGADSOJ7CKF3YJ36CDKZAA, seg_pos={0 0}, upload=446586]: Long tail cancellation due to optimal threshold Dec 26 20:24:59.193528431 (508h55m59.064611466s) [seg_pos={0 0}, upload=446586]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 20:24:59.193607642 (508h55m59.064690687s) [upload=446586]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:24:59.193615136 (508h55m59.064698171s) [upload=446586]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:24:59.497679804 (508h55m59.368762869s) [upload=446586]: Done uploading segments... Dec 26 20:24:59.497700765 (508h55m59.36878382s) [upload=446586]: Upload finished. err: <nil> Duration: 5.974116755s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [.X.X.X..............XX..X..X...........XXX.X.X....X..XXXX.X.] duration: 6.439990536s trace: https://jaeger.dev.storj.io/trace/67714c2c4f903cf FAIL (2025-12-26 20:26:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 20:24:55.494029472 (508h55m55.365112517s) [upload=446586]: Starting upload Dec 26 20:24:55.494087212 (508h55m55.365170247s) [upload=446586]: Done starting upload Dec 26 20:24:55.494119003 (508h55m55.365202038s) [upload=446586]: Uploading segments... Dec 26 20:24:55.494252457 (508h55m55.365335492s) [upload=446586]: Got next segment. Inline: false Dec 26 20:24:55.494307782 (508h55m55.365390817s) [upload=446586]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 20:24:55.494315337 (508h55m55.365398382s) [upload=446586]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 20:24:55.746810137 (508h55m55.617893182s) [seg_pos={0 0}, upload=446586]: Begin upload segment... Dec 26 20:24:55.746856535 (508h55m55.61793958s) [seg_pos={0 0}, upload=446586]: Done begin upload segment. Dec 26 20:24:55.746885150 (508h55m55.617968185s) [noise=true , address=45.91.116.4:4939 , piece_id=Y4QBV6V2BBYLRA4LGSPN7JVCKWPIOB3U6M5IFZHU4DZQDWMDW5MA, seg_pos={0 0}, upload=446586]: Uploading piece... Dec 26 20:24:55.746896232 (508h55m55.617979267s) [noise=true , address=145.79.227.41:4085 , piece_id=DWGRFX2BYT2EHMHMR2K4OOJWTL6YYTVUQPSCLLNCNNYPDEZHJM2A, seg_pos={0 0}, upload=446586]: Uploading piece... Dec 26 20:24:55.747327733 (508h55m55.618410768s) [noise=true , address=77.90.31.19:4841 , piece_id=GMMTZZSSVPWM7DIJM6KZZMPV36JFPLZBZ72BE6XU55UTCFIPGQFA, seg_pos={0 0}, upload=446586]: Uploading piece... Dec 26 20:24:55.747716052 (508h55m55.618799087s) [noise=true , address=145.79.227.45:4229 , piece_id=E4OQU7EFYLK2R53DK56TETV72RM3LYT2ZPRJ5N646XKB2TZ2HOEQ, seg_pos={0 0}, upload=446586]: Uploading piece... Dec 26 20:24:55.747929068 (508h55m55.619012113s) [noise=true , address=145.79.227.50:4374 , piece_id=5YJ4C3TD3F2QIGIKODNYETR5MWDVKVWLGOCO6V3JT5HLPEPX5IWQ, seg_pos={0 0}, upload=446586]: Uploading piece... Dec 26 20:24:55.748359348 (508h55m55.619442383s) [noise=true , address=145.79.227.25:4618 , piece_id=G3ANK36VSN3ZKEG73LH5NLQGIQF5UJV6AN2TIFXPBNBK6RKWKMQQ, seg_pos={0 0}, upload=446586]: Uploading piece... Dec 26 20:24:55.748668495 (508h55m55.61975153s) [noise=true , address=145.79.227.39:4015 , piece_id=32BQH7QZLOHZ7AAZNVNG66WG7RQ774CGADSOJ7CKF3YJ36CDKZAA, seg_pos={0 0}, upload=446586]: Uploading piece... Dec 26 20:24:55.749022230 (508h55m55.620105265s) [noise=true , address=145.79.227.34:4905 , piece_id=O6LX6GU3ANZUY2V4PSTLUHDYXHWKGAS6SO4C4TNBUOTS7HQEWDJQ, seg_pos={0 0}, upload=446586]: Uploading piece... Dec 26 20:24:55.749295589 (508h55m55.620378624s) [noise=true , address=45.91.116.10:4081 , piece_id=HIOEKRJX6O575VMHJ7FW7D5RMOLJIA4X7IOJTLXN75BVIHE2SU6Q, seg_pos={0 0}, upload=446586]: Uploading piece... Dec 26 20:24:55.749557589 (508h55m55.620640624s) [noise=true , address=145.79.227.22:4509 , piece_id=35ECUPNQRNUEBKYMWS4LM2UXIHOCQ42TMBLMXVYDQJN4NOIW326Q, seg_pos={0 0}, upload=446586]: Uploading piece... Dec 26 20:24:55.749869994 (508h55m55.620953029s) [noise=true , address=145.79.227.21:4475 , piece_id=TP4LY24LYGV7Q65B7HNGNL6FETPT7U2D2HALRVUZIVJOELZ3NAIQ, seg_pos={0 0}, upload=446586]: Uploading piece... Dec 26 20:24:55.750031462 (508h55m55.621114497s) [noise=true , address=77.90.31.11:4554 , piece_id=TUQGNI4GBUW65O6L65JCF7YEKMQEJSTXZ5RB3Q3IEW36P3BFOA3Q, seg_pos={0 0}, upload=446586]: Uploading piece... Dec 26 20:24:55.750160987 (508h55m55.621244023s) [noise=true , address=45.91.116.13:4154 , piece_id=ZVBTPMKY2IYPRZ3PWETRSX6UYRZWJJX7CFYBGFHNZOGRG7SLRQKA, seg_pos={0 0}, upload=446586]: Uploading piece... Dec 26 20:24:55.750520222 (508h55m55.621603267s) [noise=true , address=145.79.227.11:4729 , piece_id=RF6IIANE5JBJUQ3OJOEHOKZHJRCSFWVFVXN5DYD64CUVAS6ZBOFA, seg_pos={0 0}, upload=446586]: Uploading piece... Dec 26 20:24:55.750621134 (508h55m55.621704169s) [noise=true , address=145.79.227.38:4977 , piece_id=4MHHY3I6ETVPLRMFPHDRC7GISU2SUOB2V5FIVCDDVSHEODEZIO4A, seg_pos={0 0}, upload=446586]: Uploading piece... Dec 26 20:24:55.750986900 (508h55m55.622069935s) [noise=true , address=145.79.227.36:4518 , piece_id=QUMDUDYNZM6LKOIPDZFT2463AOPS6XNV2T4LVXKFNQS6I23ADB3Q, seg_pos={0 0}, upload=446586]: Uploading piece... Dec 26 20:24:56.872741053 (508h55m56.743824108s) [noise=true , address=77.90.31.19:4841 , piece_id=GMMTZZSSVPWM7DIJM6KZZMPV36JFPLZBZ72BE6XU55UTCFIPGQFA, seg_pos={0 0}, upload=446586]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 26 20:24:57.460574714 (508h55m57.331657749s) [upload=446586]: Next returned nil segment Dec 26 20:24:57.460586987 (508h55m57.331670032s) [upload=446586]: Waiting for error group managing segments... Dec 26 20:24:57.749972475 (508h55m57.62105551s) [noise=true , address=145.79.227.45:4229 , piece_id=E4OQU7EFYLK2R53DK56TETV72RM3LYT2ZPRJ5N646XKB2TZ2HOEQ, seg_pos={0 0}, upload=446586]: Done uploading piece. err: <nil> Dec 26 20:24:57.750073327 (508h55m57.621156372s) [noise=true , address=145.79.227.25:4618 , piece_id=G3ANK36VSN3ZKEG73LH5NLQGIQF5UJV6AN2TIFXPBNBK6RKWKMQQ, seg_pos={0 0}, upload=446586]: Done uploading piece. err: <nil> Dec 26 20:24:57.751036380 (508h55m57.622119415s) [noise=true , address=145.79.227.38:4977 , piece_id=4MHHY3I6ETVPLRMFPHDRC7GISU2SUOB2V5FIVCDDVSHEODEZIO4A, seg_pos={0 0}, upload=446586]: Done uploading piece. err: <nil> Dec 26 20:24:57.752579279 (508h55m57.623662314s) [noise=true , address=145.79.227.50:4374 , piece_id=5YJ4C3TD3F2QIGIKODNYETR5MWDVKVWLGOCO6V3JT5HLPEPX5IWQ, seg_pos={0 0}, upload=446586]: Done uploading piece. err: <nil> Dec 26 20:24:57.753179982 (508h55m57.624263027s) [noise=true , address=145.79.227.36:4518 , piece_id=QUMDUDYNZM6LKOIPDZFT2463AOPS6XNV2T4LVXKFNQS6I23ADB3Q, seg_pos={0 0}, upload=446586]: Done uploading piece. err: <nil> Dec 26 20:24:57.754838511 (508h55m57.625921546s) [noise=true , address=145.79.227.22:4509 , piece_id=35ECUPNQRNUEBKYMWS4LM2UXIHOCQ42TMBLMXVYDQJN4NOIW326Q, seg_pos={0 0}, upload=446586]: Done uploading piece. err: <nil> Dec 26 20:24:57.774271073 (508h55m57.645354118s) [noise=true , address=145.79.227.21:4475 , piece_id=TP4LY24LYGV7Q65B7HNGNL6FETPT7U2D2HALRVUZIVJOELZ3NAIQ, seg_pos={0 0}, upload=446586]: Done uploading piece. err: <nil> Dec 26 20:24:57.774868932 (508h55m57.645951967s) [noise=true , address=145.79.227.11:4729 , piece_id=RF6IIANE5JBJUQ3OJOEHOKZHJRCSFWVFVXN5DYD64CUVAS6ZBOFA, seg_pos={0 0}, upload=446586]: Done uploading piece. err: <nil> Dec 26 20:24:57.788847749 (508h55m57.659930784s) [noise=true , address=145.79.227.34:4905 , piece_id=O6LX6GU3ANZUY2V4PSTLUHDYXHWKGAS6SO4C4TNBUOTS7HQEWDJQ, seg_pos={0 0}, upload=446586]: Done uploading piece. err: <nil> Dec 26 20:24:57.994242624 (508h55m57.865325659s) [noise=true , address=45.91.116.4:4939 , piece_id=Y4QBV6V2BBYLRA4LGSPN7JVCKWPIOB3U6M5IFZHU4DZQDWMDW5MA, seg_pos={0 0}, upload=446586]: Done uploading piece. err: <nil> Dec 26 20:24:58.202877610 (508h55m58.073960645s) [noise=true , address=45.91.116.13:4154 , piece_id=ZVBTPMKY2IYPRZ3PWETRSX6UYRZWJJX7CFYBGFHNZOGRG7SLRQKA, seg_pos={0 0}, upload=446586]: Done uploading piece. err: <nil> Dec 26 20:24:58.484650482 (508h55m58.355733527s) [noise=true , address=45.91.116.10:4081 , piece_id=HIOEKRJX6O575VMHJ7FW7D5RMOLJIA4X7IOJTLXN75BVIHE2SU6Q, seg_pos={0 0}, upload=446586]: Done uploading piece. err: <nil> Dec 26 20:24:59.192956101 (508h55m59.064039146s) [noise=true , address=145.79.227.41:4085 , piece_id=DWGRFX2BYT2EHMHMR2K4OOJWTL6YYTVUQPSCLLNCNNYPDEZHJM2A, seg_pos={0 0}, upload=446586]: Done uploading piece. err: <nil> Dec 26 20:24:59.192979436 (508h55m59.064062481s) [seg_pos={0 0}, upload=446586]: Segment reached optimal threshold of 13 pieces. Dec 26 20:24:59.193178264 (508h55m59.064261309s) [noise=true , address=77.90.31.11:4554 , piece_id=TUQGNI4GBUW65O6L65JCF7YEKMQEJSTXZ5RB3Q3IEW36P3BFOA3Q, seg_pos={0 0}, upload=446586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Dec 26 20:24:59.193187893 (508h55m59.064270938s) [noise=true , address=77.90.31.11:4554 , piece_id=TUQGNI4GBUW65O6L65JCF7YEKMQEJSTXZ5RB3Q3IEW36P3BFOA3Q, seg_pos={0 0}, upload=446586]: Long tail cancellation due to optimal threshold Dec 26 20:24:59.193500819 (508h55m59.064583864s) [noise=true , address=145.79.227.39:4015 , piece_id=32BQH7QZLOHZ7AAZNVNG66WG7RQ774CGADSOJ7CKF3YJ36CDKZAA, seg_pos={0 0}, upload=446586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE): protocol: expected piece hash; context canceled; EOF Dec 26 20:24:59.193509996 (508h55m59.064593041s) [noise=true , address=145.79.227.39:4015 , piece_id=32BQH7QZLOHZ7AAZNVNG66WG7RQ774CGADSOJ7CKF3YJ36CDKZAA, seg_pos={0 0}, upload=446586]: Long tail cancellation due to optimal threshold Dec 26 20:24:59.193528431 (508h55m59.064611466s) [seg_pos={0 0}, upload=446586]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 20:24:59.193607642 (508h55m59.064690687s) [upload=446586]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:24:59.193615136 (508h55m59.064698171s) [upload=446586]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:24:59.497679804 (508h55m59.368762869s) [upload=446586]: Done uploading segments... Dec 26 20:24:59.497700765 (508h55m59.36878382s) [upload=446586]: Upload finished. err: <nil> Duration: 6.440019841s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....X..........................................X...........] duration: 32.479901392s trace: https://jaeger.dev.storj.io/trace/663e21624a5a3654 FAIL (2025-12-26 20:26:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, Piece ID: PWVMOEKW5A4ZAVSVJAAHFTYYTQZGFHDF642GWR6EABEGWP2CDRGQ) context deadline exceeded; piecestore: (Node ID: 12whTD1Kvu4T7ANNtkSvDB66uTjVgfPTK9TAzExgvwJYgZ3zSFG, Piece ID: ME2LDK2QJI7LLG67ODDIJDKB5CAT47YDGMJ3CTVK4F6UDHJ4TPOQ) context deadline exceeded; piecestore: (Node ID: 12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH, Piece ID: HJGRPPEJK75YTIKBKNVK7J4TDRKBCNL7HA4JNMOKD2ROGC7C4P3Q) context deadline exceeded ======LOGS====== ) [noise=true , address=145.79.227.7:4296 , piece_id=PVH6VN4CG4O2DC22G74US7IC5GDIGM2KYYYQQPFO5MG22JQCAU2A, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.248055070 (508h48m1.119138115s) [noise=true , address=145.79.227.40:4078 , piece_id=J6TGA3SFQOLGL2C2TS352TSLZBMI6MDXU5BDTTGQYMN4A5QG462A, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.248175840 (508h48m1.119258875s) [noise=true , address=77.90.31.12:4600 , piece_id=JESM5JF4GWUDLEW4YT5SKHJOXD54TWRQBNGGQCBMDSS54XR4AEKQ, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.248604496 (508h48m1.119687531s) [noise=true , address=45.91.116.6:4985 , piece_id=WV5SXZIBMXB4FTNZFE4PKBR5ZWZTUUPADPRMXAVE4Z3HKDGLBOSQ, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.248640474 (508h48m1.119723509s) [noise=true , address=145.79.227.18:4021 , piece_id=ER7GXC5OHAPNGU37QTKPROKB4YTCDQBUHVEIEUH2K5FQ4X5AGM2Q, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.249180202 (508h48m1.120263238s) [noise=true , address=77.90.31.18:4821 , piece_id=HJGRPPEJK75YTIKBKNVK7J4TDRKBCNL7HA4JNMOKD2ROGC7C4P3Q, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.249851130 (508h48m1.120934175s) [noise=true , address=64.49.13.169:4709 , piece_id=ZGWBPAYFOQWU2LSTILYMVRNKGSP53UXEQHRW4NJEXW5OLNVCCWRQ, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.250638910 (508h48m1.121721955s) [noise=true , address=145.79.227.38:4001 , piece_id=TWPCSZEWSFC2ZT7QNWHRS3RY42LLBPIEF3EIQQ65OXFZSHWAV2GA, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.251000769 (508h48m1.122083814s) [noise=true , address=145.79.227.30:4792 , piece_id=RLEUVLBAOMJSDAYHMQJGKYQUC6NYYGSGUVIU5Z53XGBINIQV43GQ, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.251314737 (508h48m1.122397772s) [noise=true , address=145.79.227.5:4252 , piece_id=FI5RNNRKSIYHR3JKCOW3THJQKHAPYLUDDHT3Y4TRYU3OSMCSWQQQ, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.251635527 (508h48m1.122718572s) [noise=true , address=77.90.31.11:4588 , piece_id=ZWDOVL5PGZ4YPY6NZVVSBSWXDDD3DAJ4RUK2BWMOGTHDGWJ2G4DA, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.251919499 (508h48m1.123002544s) [noise=true , address=77.90.31.10:4536 , piece_id=V43VRDKXTE2PHOBPSQUIJPKMKY3ROAEQAI2D4XIJHOLQ3UIMN4RA, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.252236542 (508h48m1.123319588s) [noise=true , address=77.90.31.21:4894 , piece_id=AGL6GVJX2YLGHSZX33SUURUU5S22CF3YRMJTP3QMHHP7OKX2MFSA, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.252528137 (508h48m1.123611182s) [noise=true , address=77.90.31.15:4702 , piece_id=KJEPR7NMX4UHQZHTMDDJKBRLLQ3O5BLJSLYBXGCOFGP3LNC3FXGA, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.252861402 (508h48m1.123944447s) [noise=true , address=64.49.13.27:4961 , piece_id=XCVH62CEVRYGR53U5VG4L4HJMBJSQ562L4QVKTXZSC7XGAEB4LDQ, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:02.570977410 (508h48m2.442060455s) [upload=446477]: Next returned nil segment Dec 26 20:17:02.571003019 (508h48m2.442086054s) [upload=446477]: Waiting for error group managing segments... Dec 26 20:17:02.949353997 (508h48m2.820437042s) [noise=true , address=64.49.13.27:4961 , piece_id=XCVH62CEVRYGR53U5VG4L4HJMBJSQ562L4QVKTXZSC7XGAEB4LDQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.136546190 (508h48m3.007629235s) [noise=true , address=145.79.227.5:4252 , piece_id=FI5RNNRKSIYHR3JKCOW3THJQKHAPYLUDDHT3Y4TRYU3OSMCSWQQQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.155760468 (508h48m3.026843513s) [noise=true , address=145.79.227.38:4001 , piece_id=TWPCSZEWSFC2ZT7QNWHRS3RY42LLBPIEF3EIQQ65OXFZSHWAV2GA, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.179239646 (508h48m3.050322691s) [noise=true , address=145.79.227.18:4021 , piece_id=ER7GXC5OHAPNGU37QTKPROKB4YTCDQBUHVEIEUH2K5FQ4X5AGM2Q, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.372384563 (508h48m3.243467608s) [noise=true , address=77.90.31.15:4702 , piece_id=KJEPR7NMX4UHQZHTMDDJKBRLLQ3O5BLJSLYBXGCOFGP3LNC3FXGA, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.447635803 (508h48m3.318718848s) [noise=true , address=77.90.31.18:4821 , piece_id=HJGRPPEJK75YTIKBKNVK7J4TDRKBCNL7HA4JNMOKD2ROGC7C4P3Q, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.452148166 (508h48m3.323231201s) [noise=true , address=77.90.31.10:4536 , piece_id=V43VRDKXTE2PHOBPSQUIJPKMKY3ROAEQAI2D4XIJHOLQ3UIMN4RA, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.644263002 (508h48m3.515346047s) [noise=true , address=77.90.31.14:4689 , piece_id=PWVMOEKW5A4ZAVSVJAAHFTYYTQZGFHDF642GWR6EABEGWP2CDRGQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.710620497 (508h48m3.581703542s) [noise=true , address=45.91.116.8:4074 , piece_id=ME2LDK2QJI7LLG67ODDIJDKB5CAT47YDGMJ3CTVK4F6UDHJ4TPOQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:04.066910008 (508h48m3.937993053s) [noise=true , address=77.90.31.12:4600 , piece_id=JESM5JF4GWUDLEW4YT5SKHJOXD54TWRQBNGGQCBMDSS54XR4AEKQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:04.082282349 (508h48m3.953365394s) [noise=true , address=145.79.227.44:4219 , piece_id=KZVV2NJBIU22C4URLBZMQVQN4IZ2ZMQ5YODRJMCS6XYXULHCZWLQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:04.089048724 (508h48m3.960131759s) [noise=true , address=145.79.227.30:4792 , piece_id=RLEUVLBAOMJSDAYHMQJGKYQUC6NYYGSGUVIU5Z53XGBINIQV43GQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:04.288772749 (508h48m4.159855794s) [noise=true , address=77.90.31.21:4894 , piece_id=AGL6GVJX2YLGHSZX33SUURUU5S22CF3YRMJTP3QMHHP7OKX2MFSA, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:04.344183239 (508h48m4.215266294s) [noise=true , address=77.90.31.11:4588 , piece_id=ZWDOVL5PGZ4YPY6NZVVSBSWXDDD3DAJ4RUK2BWMOGTHDGWJ2G4DA, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:04.344208728 (508h48m4.215291773s) [seg_pos={0 0}, upload=446477]: Segment reached optimal threshold of 14 pieces. Dec 26 20:17:04.344470828 (508h48m4.215553873s) [noise=true , address=64.49.13.169:4709 , piece_id=ZGWBPAYFOQWU2LSTILYMVRNKGSP53UXEQHRW4NJEXW5OLNVCCWRQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:47062->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:47062->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47062->64.49.13.169:4709: use of closed network connection Dec 26 20:17:04.344481918 (508h48m4.215564953s) [noise=true , address=64.49.13.169:4709 , piece_id=ZGWBPAYFOQWU2LSTILYMVRNKGSP53UXEQHRW4NJEXW5OLNVCCWRQ, seg_pos={0 0}, upload=446477]: Long tail cancellation due to optimal threshold Dec 26 20:17:04.344541813 (508h48m4.215624848s) [noise=true , address=145.79.227.40:4078 , piece_id=J6TGA3SFQOLGL2C2TS352TSLZBMI6MDXU5BDTTGQYMN4A5QG462A, seg_pos={0 0}, upload=446477]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tzgtenV6us5ssD7xWGpZmVUoWEfbFhBNewbK9irHtZPkyDtRd): protocol: expected piece hash; context canceled; EOF Dec 26 20:17:04.344559476 (508h48m4.215642511s) [noise=true , address=145.79.227.40:4078 , piece_id=J6TGA3SFQOLGL2C2TS352TSLZBMI6MDXU5BDTTGQYMN4A5QG462A, seg_pos={0 0}, upload=446477]: Long tail cancellation due to optimal threshold Dec 26 20:17:04.344755479 (508h48m4.215838524s) [noise=true , address=145.79.227.7:4296 , piece_id=PVH6VN4CG4O2DC22G74US7IC5GDIGM2KYYYQQPFO5MG22JQCAU2A, seg_pos={0 0}, upload=446477]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cKhDbA4ptwux2LYNPNAJ7Qh8jECsGWd1KWaK6T6pD38bJWdt1): protocol: write tcp 188.166.225.170:48104->145.79.227.7:4296: use of closed network connection; write tcp 188.166.225.170:48104->145.79.227.7:4296: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48104->145.79.227.7:4296: use of closed network connection Dec 26 20:17:04.344763044 (508h48m4.215846099s) [noise=true , address=145.79.227.7:4296 , piece_id=PVH6VN4CG4O2DC22G74US7IC5GDIGM2KYYYQQPFO5MG22JQCAU2A, seg_pos={0 0}, upload=446477]: Long tail cancellation due to optimal threshold Dec 26 20:17:04.344928509 (508h48m4.216011544s) [noise=true , address=45.91.116.6:4985 , piece_id=WV5SXZIBMXB4FTNZFE4PKBR5ZWZTUUPADPRMXAVE4Z3HKDGLBOSQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RNs9MM9jDtNC2gZC63SYsePZTd5mE7dqUm9A5Z5QWdPX8PVnp): protocol: expected piece hash; context canceled; EOF Dec 26 20:17:04.344938277 (508h48m4.216021312s) [noise=true , address=45.91.116.6:4985 , piece_id=WV5SXZIBMXB4FTNZFE4PKBR5ZWZTUUPADPRMXAVE4Z3HKDGLBOSQ, seg_pos={0 0}, upload=446477]: Long tail cancellation due to optimal threshold Dec 26 20:17:04.344966722 (508h48m4.216049767s) [seg_pos={0 0}, upload=446477]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:17:04.345054548 (508h48m4.216137593s) [upload=446477]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:17:04.345059839 (508h48m4.216142874s) [upload=446477]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:17:04.653613492 (508h48m4.524696537s) [upload=446477]: Done uploading segments... Dec 26 20:17:04.653639822 (508h48m4.524722857s) [upload=446477]: Upload finished. err: <nil> Duration: 32.479917873s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [......X..........................................X..........] duration: 32.464484756s trace: https://jaeger.dev.storj.io/trace/663e21624a5a3654 FAIL (2025-12-26 20:25:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, Piece ID: PWVMOEKW5A4ZAVSVJAAHFTYYTQZGFHDF642GWR6EABEGWP2CDRGQ) context deadline exceeded; piecestore: (Node ID: 12whTD1Kvu4T7ANNtkSvDB66uTjVgfPTK9TAzExgvwJYgZ3zSFG, Piece ID: ME2LDK2QJI7LLG67ODDIJDKB5CAT47YDGMJ3CTVK4F6UDHJ4TPOQ) context deadline exceeded; piecestore: (Node ID: 12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH, Piece ID: HJGRPPEJK75YTIKBKNVK7J4TDRKBCNL7HA4JNMOKD2ROGC7C4P3Q) context deadline exceeded ======LOGS====== ) [noise=true , address=145.79.227.7:4296 , piece_id=PVH6VN4CG4O2DC22G74US7IC5GDIGM2KYYYQQPFO5MG22JQCAU2A, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.248055070 (508h48m1.119138115s) [noise=true , address=145.79.227.40:4078 , piece_id=J6TGA3SFQOLGL2C2TS352TSLZBMI6MDXU5BDTTGQYMN4A5QG462A, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.248175840 (508h48m1.119258875s) [noise=true , address=77.90.31.12:4600 , piece_id=JESM5JF4GWUDLEW4YT5SKHJOXD54TWRQBNGGQCBMDSS54XR4AEKQ, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.248604496 (508h48m1.119687531s) [noise=true , address=45.91.116.6:4985 , piece_id=WV5SXZIBMXB4FTNZFE4PKBR5ZWZTUUPADPRMXAVE4Z3HKDGLBOSQ, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.248640474 (508h48m1.119723509s) [noise=true , address=145.79.227.18:4021 , piece_id=ER7GXC5OHAPNGU37QTKPROKB4YTCDQBUHVEIEUH2K5FQ4X5AGM2Q, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.249180202 (508h48m1.120263238s) [noise=true , address=77.90.31.18:4821 , piece_id=HJGRPPEJK75YTIKBKNVK7J4TDRKBCNL7HA4JNMOKD2ROGC7C4P3Q, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.249851130 (508h48m1.120934175s) [noise=true , address=64.49.13.169:4709 , piece_id=ZGWBPAYFOQWU2LSTILYMVRNKGSP53UXEQHRW4NJEXW5OLNVCCWRQ, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.250638910 (508h48m1.121721955s) [noise=true , address=145.79.227.38:4001 , piece_id=TWPCSZEWSFC2ZT7QNWHRS3RY42LLBPIEF3EIQQ65OXFZSHWAV2GA, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.251000769 (508h48m1.122083814s) [noise=true , address=145.79.227.30:4792 , piece_id=RLEUVLBAOMJSDAYHMQJGKYQUC6NYYGSGUVIU5Z53XGBINIQV43GQ, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.251314737 (508h48m1.122397772s) [noise=true , address=145.79.227.5:4252 , piece_id=FI5RNNRKSIYHR3JKCOW3THJQKHAPYLUDDHT3Y4TRYU3OSMCSWQQQ, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.251635527 (508h48m1.122718572s) [noise=true , address=77.90.31.11:4588 , piece_id=ZWDOVL5PGZ4YPY6NZVVSBSWXDDD3DAJ4RUK2BWMOGTHDGWJ2G4DA, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.251919499 (508h48m1.123002544s) [noise=true , address=77.90.31.10:4536 , piece_id=V43VRDKXTE2PHOBPSQUIJPKMKY3ROAEQAI2D4XIJHOLQ3UIMN4RA, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.252236542 (508h48m1.123319588s) [noise=true , address=77.90.31.21:4894 , piece_id=AGL6GVJX2YLGHSZX33SUURUU5S22CF3YRMJTP3QMHHP7OKX2MFSA, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.252528137 (508h48m1.123611182s) [noise=true , address=77.90.31.15:4702 , piece_id=KJEPR7NMX4UHQZHTMDDJKBRLLQ3O5BLJSLYBXGCOFGP3LNC3FXGA, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.252861402 (508h48m1.123944447s) [noise=true , address=64.49.13.27:4961 , piece_id=XCVH62CEVRYGR53U5VG4L4HJMBJSQ562L4QVKTXZSC7XGAEB4LDQ, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:02.570977410 (508h48m2.442060455s) [upload=446477]: Next returned nil segment Dec 26 20:17:02.571003019 (508h48m2.442086054s) [upload=446477]: Waiting for error group managing segments... Dec 26 20:17:02.949353997 (508h48m2.820437042s) [noise=true , address=64.49.13.27:4961 , piece_id=XCVH62CEVRYGR53U5VG4L4HJMBJSQ562L4QVKTXZSC7XGAEB4LDQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.136546190 (508h48m3.007629235s) [noise=true , address=145.79.227.5:4252 , piece_id=FI5RNNRKSIYHR3JKCOW3THJQKHAPYLUDDHT3Y4TRYU3OSMCSWQQQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.155760468 (508h48m3.026843513s) [noise=true , address=145.79.227.38:4001 , piece_id=TWPCSZEWSFC2ZT7QNWHRS3RY42LLBPIEF3EIQQ65OXFZSHWAV2GA, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.179239646 (508h48m3.050322691s) [noise=true , address=145.79.227.18:4021 , piece_id=ER7GXC5OHAPNGU37QTKPROKB4YTCDQBUHVEIEUH2K5FQ4X5AGM2Q, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.372384563 (508h48m3.243467608s) [noise=true , address=77.90.31.15:4702 , piece_id=KJEPR7NMX4UHQZHTMDDJKBRLLQ3O5BLJSLYBXGCOFGP3LNC3FXGA, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.447635803 (508h48m3.318718848s) [noise=true , address=77.90.31.18:4821 , piece_id=HJGRPPEJK75YTIKBKNVK7J4TDRKBCNL7HA4JNMOKD2ROGC7C4P3Q, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.452148166 (508h48m3.323231201s) [noise=true , address=77.90.31.10:4536 , piece_id=V43VRDKXTE2PHOBPSQUIJPKMKY3ROAEQAI2D4XIJHOLQ3UIMN4RA, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.644263002 (508h48m3.515346047s) [noise=true , address=77.90.31.14:4689 , piece_id=PWVMOEKW5A4ZAVSVJAAHFTYYTQZGFHDF642GWR6EABEGWP2CDRGQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.710620497 (508h48m3.581703542s) [noise=true , address=45.91.116.8:4074 , piece_id=ME2LDK2QJI7LLG67ODDIJDKB5CAT47YDGMJ3CTVK4F6UDHJ4TPOQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:04.066910008 (508h48m3.937993053s) [noise=true , address=77.90.31.12:4600 , piece_id=JESM5JF4GWUDLEW4YT5SKHJOXD54TWRQBNGGQCBMDSS54XR4AEKQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:04.082282349 (508h48m3.953365394s) [noise=true , address=145.79.227.44:4219 , piece_id=KZVV2NJBIU22C4URLBZMQVQN4IZ2ZMQ5YODRJMCS6XYXULHCZWLQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:04.089048724 (508h48m3.960131759s) [noise=true , address=145.79.227.30:4792 , piece_id=RLEUVLBAOMJSDAYHMQJGKYQUC6NYYGSGUVIU5Z53XGBINIQV43GQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:04.288772749 (508h48m4.159855794s) [noise=true , address=77.90.31.21:4894 , piece_id=AGL6GVJX2YLGHSZX33SUURUU5S22CF3YRMJTP3QMHHP7OKX2MFSA, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:04.344183239 (508h48m4.215266294s) [noise=true , address=77.90.31.11:4588 , piece_id=ZWDOVL5PGZ4YPY6NZVVSBSWXDDD3DAJ4RUK2BWMOGTHDGWJ2G4DA, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:04.344208728 (508h48m4.215291773s) [seg_pos={0 0}, upload=446477]: Segment reached optimal threshold of 14 pieces. Dec 26 20:17:04.344470828 (508h48m4.215553873s) [noise=true , address=64.49.13.169:4709 , piece_id=ZGWBPAYFOQWU2LSTILYMVRNKGSP53UXEQHRW4NJEXW5OLNVCCWRQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:47062->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:47062->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47062->64.49.13.169:4709: use of closed network connection Dec 26 20:17:04.344481918 (508h48m4.215564953s) [noise=true , address=64.49.13.169:4709 , piece_id=ZGWBPAYFOQWU2LSTILYMVRNKGSP53UXEQHRW4NJEXW5OLNVCCWRQ, seg_pos={0 0}, upload=446477]: Long tail cancellation due to optimal threshold Dec 26 20:17:04.344541813 (508h48m4.215624848s) [noise=true , address=145.79.227.40:4078 , piece_id=J6TGA3SFQOLGL2C2TS352TSLZBMI6MDXU5BDTTGQYMN4A5QG462A, seg_pos={0 0}, upload=446477]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tzgtenV6us5ssD7xWGpZmVUoWEfbFhBNewbK9irHtZPkyDtRd): protocol: expected piece hash; context canceled; EOF Dec 26 20:17:04.344559476 (508h48m4.215642511s) [noise=true , address=145.79.227.40:4078 , piece_id=J6TGA3SFQOLGL2C2TS352TSLZBMI6MDXU5BDTTGQYMN4A5QG462A, seg_pos={0 0}, upload=446477]: Long tail cancellation due to optimal threshold Dec 26 20:17:04.344755479 (508h48m4.215838524s) [noise=true , address=145.79.227.7:4296 , piece_id=PVH6VN4CG4O2DC22G74US7IC5GDIGM2KYYYQQPFO5MG22JQCAU2A, seg_pos={0 0}, upload=446477]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cKhDbA4ptwux2LYNPNAJ7Qh8jECsGWd1KWaK6T6pD38bJWdt1): protocol: write tcp 188.166.225.170:48104->145.79.227.7:4296: use of closed network connection; write tcp 188.166.225.170:48104->145.79.227.7:4296: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48104->145.79.227.7:4296: use of closed network connection Dec 26 20:17:04.344763044 (508h48m4.215846099s) [noise=true , address=145.79.227.7:4296 , piece_id=PVH6VN4CG4O2DC22G74US7IC5GDIGM2KYYYQQPFO5MG22JQCAU2A, seg_pos={0 0}, upload=446477]: Long tail cancellation due to optimal threshold Dec 26 20:17:04.344928509 (508h48m4.216011544s) [noise=true , address=45.91.116.6:4985 , piece_id=WV5SXZIBMXB4FTNZFE4PKBR5ZWZTUUPADPRMXAVE4Z3HKDGLBOSQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RNs9MM9jDtNC2gZC63SYsePZTd5mE7dqUm9A5Z5QWdPX8PVnp): protocol: expected piece hash; context canceled; EOF Dec 26 20:17:04.344938277 (508h48m4.216021312s) [noise=true , address=45.91.116.6:4985 , piece_id=WV5SXZIBMXB4FTNZFE4PKBR5ZWZTUUPADPRMXAVE4Z3HKDGLBOSQ, seg_pos={0 0}, upload=446477]: Long tail cancellation due to optimal threshold Dec 26 20:17:04.344966722 (508h48m4.216049767s) [seg_pos={0 0}, upload=446477]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:17:04.345054548 (508h48m4.216137593s) [upload=446477]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:17:04.345059839 (508h48m4.216142874s) [upload=446477]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:17:04.653613492 (508h48m4.524696537s) [upload=446477]: Done uploading segments... Dec 26 20:17:04.653639822 (508h48m4.524722857s) [upload=446477]: Upload finished. err: <nil> Duration: 32.464509573s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [X.X.X.X..............XX..X..X...........XXX.X.X....X..XXXX.X] duration: 30.003136345s trace: https://jaeger.dev.storj.io/trace/67714c2c4f903cf FAIL (2025-12-26 20:25:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 20:24:55.494029472 (508h55m55.365112517s) [upload=446586]: Starting upload Dec 26 20:24:55.494087212 (508h55m55.365170247s) [upload=446586]: Done starting upload Dec 26 20:24:55.494119003 (508h55m55.365202038s) [upload=446586]: Uploading segments... Dec 26 20:24:55.494252457 (508h55m55.365335492s) [upload=446586]: Got next segment. Inline: false Dec 26 20:24:55.494307782 (508h55m55.365390817s) [upload=446586]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 20:24:55.494315337 (508h55m55.365398382s) [upload=446586]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 20:24:55.746810137 (508h55m55.617893182s) [seg_pos={0 0}, upload=446586]: Begin upload segment... Dec 26 20:24:55.746856535 (508h55m55.61793958s) [seg_pos={0 0}, upload=446586]: Done begin upload segment. Dec 26 20:24:55.746885150 (508h55m55.617968185s) [noise=true , address=45.91.116.4:4939 , piece_id=Y4QBV6V2BBYLRA4LGSPN7JVCKWPIOB3U6M5IFZHU4DZQDWMDW5MA, seg_pos={0 0}, upload=446586]: Uploading piece... Dec 26 20:24:55.746896232 (508h55m55.617979267s) [noise=true , address=145.79.227.41:4085 , piece_id=DWGRFX2BYT2EHMHMR2K4OOJWTL6YYTVUQPSCLLNCNNYPDEZHJM2A, seg_pos={0 0}, upload=446586]: Uploading piece... Dec 26 20:24:55.747327733 (508h55m55.618410768s) [noise=true , address=77.90.31.19:4841 , piece_id=GMMTZZSSVPWM7DIJM6KZZMPV36JFPLZBZ72BE6XU55UTCFIPGQFA, seg_pos={0 0}, upload=446586]: Uploading piece... Dec 26 20:24:55.747716052 (508h55m55.618799087s) [noise=true , address=145.79.227.45:4229 , piece_id=E4OQU7EFYLK2R53DK56TETV72RM3LYT2ZPRJ5N646XKB2TZ2HOEQ, seg_pos={0 0}, upload=446586]: Uploading piece... Dec 26 20:24:55.747929068 (508h55m55.619012113s) [noise=true , address=145.79.227.50:4374 , piece_id=5YJ4C3TD3F2QIGIKODNYETR5MWDVKVWLGOCO6V3JT5HLPEPX5IWQ, seg_pos={0 0}, upload=446586]: Uploading piece... Dec 26 20:24:55.748359348 (508h55m55.619442383s) [noise=true , address=145.79.227.25:4618 , piece_id=G3ANK36VSN3ZKEG73LH5NLQGIQF5UJV6AN2TIFXPBNBK6RKWKMQQ, seg_pos={0 0}, upload=446586]: Uploading piece... Dec 26 20:24:55.748668495 (508h55m55.61975153s) [noise=true , address=145.79.227.39:4015 , piece_id=32BQH7QZLOHZ7AAZNVNG66WG7RQ774CGADSOJ7CKF3YJ36CDKZAA, seg_pos={0 0}, upload=446586]: Uploading piece... Dec 26 20:24:55.749022230 (508h55m55.620105265s) [noise=true , address=145.79.227.34:4905 , piece_id=O6LX6GU3ANZUY2V4PSTLUHDYXHWKGAS6SO4C4TNBUOTS7HQEWDJQ, seg_pos={0 0}, upload=446586]: Uploading piece... Dec 26 20:24:55.749295589 (508h55m55.620378624s) [noise=true , address=45.91.116.10:4081 , piece_id=HIOEKRJX6O575VMHJ7FW7D5RMOLJIA4X7IOJTLXN75BVIHE2SU6Q, seg_pos={0 0}, upload=446586]: Uploading piece... Dec 26 20:24:55.749557589 (508h55m55.620640624s) [noise=true , address=145.79.227.22:4509 , piece_id=35ECUPNQRNUEBKYMWS4LM2UXIHOCQ42TMBLMXVYDQJN4NOIW326Q, seg_pos={0 0}, upload=446586]: Uploading piece... Dec 26 20:24:55.749869994 (508h55m55.620953029s) [noise=true , address=145.79.227.21:4475 , piece_id=TP4LY24LYGV7Q65B7HNGNL6FETPT7U2D2HALRVUZIVJOELZ3NAIQ, seg_pos={0 0}, upload=446586]: Uploading piece... Dec 26 20:24:55.750031462 (508h55m55.621114497s) [noise=true , address=77.90.31.11:4554 , piece_id=TUQGNI4GBUW65O6L65JCF7YEKMQEJSTXZ5RB3Q3IEW36P3BFOA3Q, seg_pos={0 0}, upload=446586]: Uploading piece... Dec 26 20:24:55.750160987 (508h55m55.621244023s) [noise=true , address=45.91.116.13:4154 , piece_id=ZVBTPMKY2IYPRZ3PWETRSX6UYRZWJJX7CFYBGFHNZOGRG7SLRQKA, seg_pos={0 0}, upload=446586]: Uploading piece... Dec 26 20:24:55.750520222 (508h55m55.621603267s) [noise=true , address=145.79.227.11:4729 , piece_id=RF6IIANE5JBJUQ3OJOEHOKZHJRCSFWVFVXN5DYD64CUVAS6ZBOFA, seg_pos={0 0}, upload=446586]: Uploading piece... Dec 26 20:24:55.750621134 (508h55m55.621704169s) [noise=true , address=145.79.227.38:4977 , piece_id=4MHHY3I6ETVPLRMFPHDRC7GISU2SUOB2V5FIVCDDVSHEODEZIO4A, seg_pos={0 0}, upload=446586]: Uploading piece... Dec 26 20:24:55.750986900 (508h55m55.622069935s) [noise=true , address=145.79.227.36:4518 , piece_id=QUMDUDYNZM6LKOIPDZFT2463AOPS6XNV2T4LVXKFNQS6I23ADB3Q, seg_pos={0 0}, upload=446586]: Uploading piece... Dec 26 20:24:56.872741053 (508h55m56.743824108s) [noise=true , address=77.90.31.19:4841 , piece_id=GMMTZZSSVPWM7DIJM6KZZMPV36JFPLZBZ72BE6XU55UTCFIPGQFA, seg_pos={0 0}, upload=446586]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 26 20:24:57.460574714 (508h55m57.331657749s) [upload=446586]: Next returned nil segment Dec 26 20:24:57.460586987 (508h55m57.331670032s) [upload=446586]: Waiting for error group managing segments... Dec 26 20:24:57.749972475 (508h55m57.62105551s) [noise=true , address=145.79.227.45:4229 , piece_id=E4OQU7EFYLK2R53DK56TETV72RM3LYT2ZPRJ5N646XKB2TZ2HOEQ, seg_pos={0 0}, upload=446586]: Done uploading piece. err: <nil> Dec 26 20:24:57.750073327 (508h55m57.621156372s) [noise=true , address=145.79.227.25:4618 , piece_id=G3ANK36VSN3ZKEG73LH5NLQGIQF5UJV6AN2TIFXPBNBK6RKWKMQQ, seg_pos={0 0}, upload=446586]: Done uploading piece. err: <nil> Dec 26 20:24:57.751036380 (508h55m57.622119415s) [noise=true , address=145.79.227.38:4977 , piece_id=4MHHY3I6ETVPLRMFPHDRC7GISU2SUOB2V5FIVCDDVSHEODEZIO4A, seg_pos={0 0}, upload=446586]: Done uploading piece. err: <nil> Dec 26 20:24:57.752579279 (508h55m57.623662314s) [noise=true , address=145.79.227.50:4374 , piece_id=5YJ4C3TD3F2QIGIKODNYETR5MWDVKVWLGOCO6V3JT5HLPEPX5IWQ, seg_pos={0 0}, upload=446586]: Done uploading piece. err: <nil> Dec 26 20:24:57.753179982 (508h55m57.624263027s) [noise=true , address=145.79.227.36:4518 , piece_id=QUMDUDYNZM6LKOIPDZFT2463AOPS6XNV2T4LVXKFNQS6I23ADB3Q, seg_pos={0 0}, upload=446586]: Done uploading piece. err: <nil> Dec 26 20:24:57.754838511 (508h55m57.625921546s) [noise=true , address=145.79.227.22:4509 , piece_id=35ECUPNQRNUEBKYMWS4LM2UXIHOCQ42TMBLMXVYDQJN4NOIW326Q, seg_pos={0 0}, upload=446586]: Done uploading piece. err: <nil> Dec 26 20:24:57.774271073 (508h55m57.645354118s) [noise=true , address=145.79.227.21:4475 , piece_id=TP4LY24LYGV7Q65B7HNGNL6FETPT7U2D2HALRVUZIVJOELZ3NAIQ, seg_pos={0 0}, upload=446586]: Done uploading piece. err: <nil> Dec 26 20:24:57.774868932 (508h55m57.645951967s) [noise=true , address=145.79.227.11:4729 , piece_id=RF6IIANE5JBJUQ3OJOEHOKZHJRCSFWVFVXN5DYD64CUVAS6ZBOFA, seg_pos={0 0}, upload=446586]: Done uploading piece. err: <nil> Dec 26 20:24:57.788847749 (508h55m57.659930784s) [noise=true , address=145.79.227.34:4905 , piece_id=O6LX6GU3ANZUY2V4PSTLUHDYXHWKGAS6SO4C4TNBUOTS7HQEWDJQ, seg_pos={0 0}, upload=446586]: Done uploading piece. err: <nil> Dec 26 20:24:57.994242624 (508h55m57.865325659s) [noise=true , address=45.91.116.4:4939 , piece_id=Y4QBV6V2BBYLRA4LGSPN7JVCKWPIOB3U6M5IFZHU4DZQDWMDW5MA, seg_pos={0 0}, upload=446586]: Done uploading piece. err: <nil> Dec 26 20:24:58.202877610 (508h55m58.073960645s) [noise=true , address=45.91.116.13:4154 , piece_id=ZVBTPMKY2IYPRZ3PWETRSX6UYRZWJJX7CFYBGFHNZOGRG7SLRQKA, seg_pos={0 0}, upload=446586]: Done uploading piece. err: <nil> Dec 26 20:24:58.484650482 (508h55m58.355733527s) [noise=true , address=45.91.116.10:4081 , piece_id=HIOEKRJX6O575VMHJ7FW7D5RMOLJIA4X7IOJTLXN75BVIHE2SU6Q, seg_pos={0 0}, upload=446586]: Done uploading piece. err: <nil> Dec 26 20:24:59.192956101 (508h55m59.064039146s) [noise=true , address=145.79.227.41:4085 , piece_id=DWGRFX2BYT2EHMHMR2K4OOJWTL6YYTVUQPSCLLNCNNYPDEZHJM2A, seg_pos={0 0}, upload=446586]: Done uploading piece. err: <nil> Dec 26 20:24:59.192979436 (508h55m59.064062481s) [seg_pos={0 0}, upload=446586]: Segment reached optimal threshold of 13 pieces. Dec 26 20:24:59.193178264 (508h55m59.064261309s) [noise=true , address=77.90.31.11:4554 , piece_id=TUQGNI4GBUW65O6L65JCF7YEKMQEJSTXZ5RB3Q3IEW36P3BFOA3Q, seg_pos={0 0}, upload=446586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Dec 26 20:24:59.193187893 (508h55m59.064270938s) [noise=true , address=77.90.31.11:4554 , piece_id=TUQGNI4GBUW65O6L65JCF7YEKMQEJSTXZ5RB3Q3IEW36P3BFOA3Q, seg_pos={0 0}, upload=446586]: Long tail cancellation due to optimal threshold Dec 26 20:24:59.193500819 (508h55m59.064583864s) [noise=true , address=145.79.227.39:4015 , piece_id=32BQH7QZLOHZ7AAZNVNG66WG7RQ774CGADSOJ7CKF3YJ36CDKZAA, seg_pos={0 0}, upload=446586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE): protocol: expected piece hash; context canceled; EOF Dec 26 20:24:59.193509996 (508h55m59.064593041s) [noise=true , address=145.79.227.39:4015 , piece_id=32BQH7QZLOHZ7AAZNVNG66WG7RQ774CGADSOJ7CKF3YJ36CDKZAA, seg_pos={0 0}, upload=446586]: Long tail cancellation due to optimal threshold Dec 26 20:24:59.193528431 (508h55m59.064611466s) [seg_pos={0 0}, upload=446586]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 20:24:59.193607642 (508h55m59.064690687s) [upload=446586]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:24:59.193615136 (508h55m59.064698171s) [upload=446586]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:24:59.497679804 (508h55m59.368762869s) [upload=446586]: Done uploading segments... Dec 26 20:24:59.497700765 (508h55m59.36878382s) [upload=446586]: Upload finished. err: <nil> Duration: 30.003157194s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......X..........................................X.........] duration: 28.664345121s trace: https://jaeger.dev.storj.io/trace/663e21624a5a3654 FAIL (2025-12-26 20:24:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, Piece ID: PWVMOEKW5A4ZAVSVJAAHFTYYTQZGFHDF642GWR6EABEGWP2CDRGQ) context deadline exceeded; piecestore: (Node ID: 12whTD1Kvu4T7ANNtkSvDB66uTjVgfPTK9TAzExgvwJYgZ3zSFG, Piece ID: ME2LDK2QJI7LLG67ODDIJDKB5CAT47YDGMJ3CTVK4F6UDHJ4TPOQ) context deadline exceeded; piecestore: (Node ID: 12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH, Piece ID: HJGRPPEJK75YTIKBKNVK7J4TDRKBCNL7HA4JNMOKD2ROGC7C4P3Q) context deadline exceeded ======LOGS====== ) [noise=true , address=145.79.227.7:4296 , piece_id=PVH6VN4CG4O2DC22G74US7IC5GDIGM2KYYYQQPFO5MG22JQCAU2A, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.248055070 (508h48m1.119138115s) [noise=true , address=145.79.227.40:4078 , piece_id=J6TGA3SFQOLGL2C2TS352TSLZBMI6MDXU5BDTTGQYMN4A5QG462A, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.248175840 (508h48m1.119258875s) [noise=true , address=77.90.31.12:4600 , piece_id=JESM5JF4GWUDLEW4YT5SKHJOXD54TWRQBNGGQCBMDSS54XR4AEKQ, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.248604496 (508h48m1.119687531s) [noise=true , address=45.91.116.6:4985 , piece_id=WV5SXZIBMXB4FTNZFE4PKBR5ZWZTUUPADPRMXAVE4Z3HKDGLBOSQ, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.248640474 (508h48m1.119723509s) [noise=true , address=145.79.227.18:4021 , piece_id=ER7GXC5OHAPNGU37QTKPROKB4YTCDQBUHVEIEUH2K5FQ4X5AGM2Q, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.249180202 (508h48m1.120263238s) [noise=true , address=77.90.31.18:4821 , piece_id=HJGRPPEJK75YTIKBKNVK7J4TDRKBCNL7HA4JNMOKD2ROGC7C4P3Q, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.249851130 (508h48m1.120934175s) [noise=true , address=64.49.13.169:4709 , piece_id=ZGWBPAYFOQWU2LSTILYMVRNKGSP53UXEQHRW4NJEXW5OLNVCCWRQ, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.250638910 (508h48m1.121721955s) [noise=true , address=145.79.227.38:4001 , piece_id=TWPCSZEWSFC2ZT7QNWHRS3RY42LLBPIEF3EIQQ65OXFZSHWAV2GA, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.251000769 (508h48m1.122083814s) [noise=true , address=145.79.227.30:4792 , piece_id=RLEUVLBAOMJSDAYHMQJGKYQUC6NYYGSGUVIU5Z53XGBINIQV43GQ, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.251314737 (508h48m1.122397772s) [noise=true , address=145.79.227.5:4252 , piece_id=FI5RNNRKSIYHR3JKCOW3THJQKHAPYLUDDHT3Y4TRYU3OSMCSWQQQ, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.251635527 (508h48m1.122718572s) [noise=true , address=77.90.31.11:4588 , piece_id=ZWDOVL5PGZ4YPY6NZVVSBSWXDDD3DAJ4RUK2BWMOGTHDGWJ2G4DA, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.251919499 (508h48m1.123002544s) [noise=true , address=77.90.31.10:4536 , piece_id=V43VRDKXTE2PHOBPSQUIJPKMKY3ROAEQAI2D4XIJHOLQ3UIMN4RA, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.252236542 (508h48m1.123319588s) [noise=true , address=77.90.31.21:4894 , piece_id=AGL6GVJX2YLGHSZX33SUURUU5S22CF3YRMJTP3QMHHP7OKX2MFSA, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.252528137 (508h48m1.123611182s) [noise=true , address=77.90.31.15:4702 , piece_id=KJEPR7NMX4UHQZHTMDDJKBRLLQ3O5BLJSLYBXGCOFGP3LNC3FXGA, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.252861402 (508h48m1.123944447s) [noise=true , address=64.49.13.27:4961 , piece_id=XCVH62CEVRYGR53U5VG4L4HJMBJSQ562L4QVKTXZSC7XGAEB4LDQ, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:02.570977410 (508h48m2.442060455s) [upload=446477]: Next returned nil segment Dec 26 20:17:02.571003019 (508h48m2.442086054s) [upload=446477]: Waiting for error group managing segments... Dec 26 20:17:02.949353997 (508h48m2.820437042s) [noise=true , address=64.49.13.27:4961 , piece_id=XCVH62CEVRYGR53U5VG4L4HJMBJSQ562L4QVKTXZSC7XGAEB4LDQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.136546190 (508h48m3.007629235s) [noise=true , address=145.79.227.5:4252 , piece_id=FI5RNNRKSIYHR3JKCOW3THJQKHAPYLUDDHT3Y4TRYU3OSMCSWQQQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.155760468 (508h48m3.026843513s) [noise=true , address=145.79.227.38:4001 , piece_id=TWPCSZEWSFC2ZT7QNWHRS3RY42LLBPIEF3EIQQ65OXFZSHWAV2GA, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.179239646 (508h48m3.050322691s) [noise=true , address=145.79.227.18:4021 , piece_id=ER7GXC5OHAPNGU37QTKPROKB4YTCDQBUHVEIEUH2K5FQ4X5AGM2Q, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.372384563 (508h48m3.243467608s) [noise=true , address=77.90.31.15:4702 , piece_id=KJEPR7NMX4UHQZHTMDDJKBRLLQ3O5BLJSLYBXGCOFGP3LNC3FXGA, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.447635803 (508h48m3.318718848s) [noise=true , address=77.90.31.18:4821 , piece_id=HJGRPPEJK75YTIKBKNVK7J4TDRKBCNL7HA4JNMOKD2ROGC7C4P3Q, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.452148166 (508h48m3.323231201s) [noise=true , address=77.90.31.10:4536 , piece_id=V43VRDKXTE2PHOBPSQUIJPKMKY3ROAEQAI2D4XIJHOLQ3UIMN4RA, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.644263002 (508h48m3.515346047s) [noise=true , address=77.90.31.14:4689 , piece_id=PWVMOEKW5A4ZAVSVJAAHFTYYTQZGFHDF642GWR6EABEGWP2CDRGQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.710620497 (508h48m3.581703542s) [noise=true , address=45.91.116.8:4074 , piece_id=ME2LDK2QJI7LLG67ODDIJDKB5CAT47YDGMJ3CTVK4F6UDHJ4TPOQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:04.066910008 (508h48m3.937993053s) [noise=true , address=77.90.31.12:4600 , piece_id=JESM5JF4GWUDLEW4YT5SKHJOXD54TWRQBNGGQCBMDSS54XR4AEKQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:04.082282349 (508h48m3.953365394s) [noise=true , address=145.79.227.44:4219 , piece_id=KZVV2NJBIU22C4URLBZMQVQN4IZ2ZMQ5YODRJMCS6XYXULHCZWLQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:04.089048724 (508h48m3.960131759s) [noise=true , address=145.79.227.30:4792 , piece_id=RLEUVLBAOMJSDAYHMQJGKYQUC6NYYGSGUVIU5Z53XGBINIQV43GQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:04.288772749 (508h48m4.159855794s) [noise=true , address=77.90.31.21:4894 , piece_id=AGL6GVJX2YLGHSZX33SUURUU5S22CF3YRMJTP3QMHHP7OKX2MFSA, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:04.344183239 (508h48m4.215266294s) [noise=true , address=77.90.31.11:4588 , piece_id=ZWDOVL5PGZ4YPY6NZVVSBSWXDDD3DAJ4RUK2BWMOGTHDGWJ2G4DA, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:04.344208728 (508h48m4.215291773s) [seg_pos={0 0}, upload=446477]: Segment reached optimal threshold of 14 pieces. Dec 26 20:17:04.344470828 (508h48m4.215553873s) [noise=true , address=64.49.13.169:4709 , piece_id=ZGWBPAYFOQWU2LSTILYMVRNKGSP53UXEQHRW4NJEXW5OLNVCCWRQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:47062->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:47062->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47062->64.49.13.169:4709: use of closed network connection Dec 26 20:17:04.344481918 (508h48m4.215564953s) [noise=true , address=64.49.13.169:4709 , piece_id=ZGWBPAYFOQWU2LSTILYMVRNKGSP53UXEQHRW4NJEXW5OLNVCCWRQ, seg_pos={0 0}, upload=446477]: Long tail cancellation due to optimal threshold Dec 26 20:17:04.344541813 (508h48m4.215624848s) [noise=true , address=145.79.227.40:4078 , piece_id=J6TGA3SFQOLGL2C2TS352TSLZBMI6MDXU5BDTTGQYMN4A5QG462A, seg_pos={0 0}, upload=446477]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tzgtenV6us5ssD7xWGpZmVUoWEfbFhBNewbK9irHtZPkyDtRd): protocol: expected piece hash; context canceled; EOF Dec 26 20:17:04.344559476 (508h48m4.215642511s) [noise=true , address=145.79.227.40:4078 , piece_id=J6TGA3SFQOLGL2C2TS352TSLZBMI6MDXU5BDTTGQYMN4A5QG462A, seg_pos={0 0}, upload=446477]: Long tail cancellation due to optimal threshold Dec 26 20:17:04.344755479 (508h48m4.215838524s) [noise=true , address=145.79.227.7:4296 , piece_id=PVH6VN4CG4O2DC22G74US7IC5GDIGM2KYYYQQPFO5MG22JQCAU2A, seg_pos={0 0}, upload=446477]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cKhDbA4ptwux2LYNPNAJ7Qh8jECsGWd1KWaK6T6pD38bJWdt1): protocol: write tcp 188.166.225.170:48104->145.79.227.7:4296: use of closed network connection; write tcp 188.166.225.170:48104->145.79.227.7:4296: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48104->145.79.227.7:4296: use of closed network connection Dec 26 20:17:04.344763044 (508h48m4.215846099s) [noise=true , address=145.79.227.7:4296 , piece_id=PVH6VN4CG4O2DC22G74US7IC5GDIGM2KYYYQQPFO5MG22JQCAU2A, seg_pos={0 0}, upload=446477]: Long tail cancellation due to optimal threshold Dec 26 20:17:04.344928509 (508h48m4.216011544s) [noise=true , address=45.91.116.6:4985 , piece_id=WV5SXZIBMXB4FTNZFE4PKBR5ZWZTUUPADPRMXAVE4Z3HKDGLBOSQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RNs9MM9jDtNC2gZC63SYsePZTd5mE7dqUm9A5Z5QWdPX8PVnp): protocol: expected piece hash; context canceled; EOF Dec 26 20:17:04.344938277 (508h48m4.216021312s) [noise=true , address=45.91.116.6:4985 , piece_id=WV5SXZIBMXB4FTNZFE4PKBR5ZWZTUUPADPRMXAVE4Z3HKDGLBOSQ, seg_pos={0 0}, upload=446477]: Long tail cancellation due to optimal threshold Dec 26 20:17:04.344966722 (508h48m4.216049767s) [seg_pos={0 0}, upload=446477]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:17:04.345054548 (508h48m4.216137593s) [upload=446477]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:17:04.345059839 (508h48m4.216142874s) [upload=446477]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:17:04.653613492 (508h48m4.524696537s) [upload=446477]: Done uploading segments... Dec 26 20:17:04.653639822 (508h48m4.524722857s) [upload=446477]: Upload finished. err: <nil> Duration: 28.664366132s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [.X.X.X.X..............XX..X..X...........XXX.X.X....X..XXXX.] duration: 6.613572812s trace: https://jaeger.dev.storj.io/trace/38d6fc2548068ce1 FAIL (2025-12-26 20:24:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== BatchRequestItem_ObjectBegin Dec 26 20:23:03.589900904 (508h54m3.460983939s) [upload=446559]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 20:23:03.839763603 (508h54m3.710846648s) [seg_pos={0 0}, upload=446559]: Begin upload segment... Dec 26 20:23:03.839814761 (508h54m3.710897806s) [seg_pos={0 0}, upload=446559]: Done begin upload segment. Dec 26 20:23:03.839842353 (508h54m3.710925388s) [noise=true , address=77.90.31.19:4844 , piece_id=P7HYROFKCFIOG7SGWFGEGZW6ZALEHXDL7JQP7G23JEYSIMW4LY7A, seg_pos={0 0}, upload=446559]: Uploading piece... Dec 26 20:23:03.840018929 (508h54m3.711101974s) [noise=true , address=45.91.116.5:4298 , piece_id=YWBVQX37SM57E5NBOUPSQB3IHSP6J42EK2ORULGAC5CABUVBEXYQ, seg_pos={0 0}, upload=446559]: Uploading piece... Dec 26 20:23:03.840890459 (508h54m3.711973505s) [noise=true , address=45.91.116.11:4118 , piece_id=JTDLKRGYANDXVIFTNFKEW4K7MIHCJYAGMBEYDK6OPGT37EPDDKLA, seg_pos={0 0}, upload=446559]: Uploading piece... Dec 26 20:23:03.841143541 (508h54m3.712226586s) [noise=true , address=145.79.227.21:4475 , piece_id=RHDIL6I6YMOXGGQBASRUE63W6X452O4EBNDXGWSBOBB3UNOPF2HQ, seg_pos={0 0}, upload=446559]: Uploading piece... Dec 26 20:23:03.841716412 (508h54m3.712799467s) [noise=true , address=145.79.227.11:4730 , piece_id=HEK7KDSHBAXR24SQ4ZK4JGOG7YH7LL6TV2JOGXGS7NSZE323STEA, seg_pos={0 0}, upload=446559]: Uploading piece... Dec 26 20:23:03.842282319 (508h54m3.713365364s) [noise=true , address=145.79.227.49:4337 , piece_id=6MO6YVADQNKSJWKVDYF6THUVWU7T5TWBY4JJDUZ6NIKK6N45CJBQ, seg_pos={0 0}, upload=446559]: Uploading piece... Dec 26 20:23:03.842913472 (508h54m3.713996517s) [noise=true , address=145.79.227.52:4446 , piece_id=RQKS4XE7RT7Q54E4WW3W7X7K6XF6SR37UM354VGHYMTYF6UQ6CXA, seg_pos={0 0}, upload=446559]: Uploading piece... Dec 26 20:23:03.843372215 (508h54m3.71445526s) [noise=true , address=77.90.31.13:4627 , piece_id=DRGW6ZDTAHUMTZLUHJBVXOGDEJHSRKWB7M3Q6TERL755ZLXXCAGA, seg_pos={0 0}, upload=446559]: Uploading piece... Dec 26 20:23:03.843875773 (508h54m3.714958818s) [noise=true , address=145.79.227.37:4565 , piece_id=E42J2H5FOJYU3QJ6QCAJGZ6R33OHALNAWE3LE47X7Y33A56RIGUQ, seg_pos={0 0}, upload=446559]: Uploading piece... Dec 26 20:23:03.844433996 (508h54m3.715517041s) [noise=true , address=45.91.116.13:4153 , piece_id=YHNRYXDNOSU2RMTRVQU46H2D73V5KJBGKWG6JLL5AMUCUIN5K3YQ, seg_pos={0 0}, upload=446559]: Uploading piece... Dec 26 20:23:03.846328695 (508h54m3.71741173s) [noise=true , address=145.79.227.48:4302 , piece_id=KBAWJG5R57OK35DU7KSB4ORIEX5WDTODHBUEZY4R3KJHCFMCN4JA, seg_pos={0 0}, upload=446559]: Uploading piece... Dec 26 20:23:03.846965006 (508h54m3.718048051s) [noise=true , address=45.91.116.14:4191 , piece_id=DBNRTQ7MSGVAHGPF5XEPPQBECW6IOD22VELUBZEDOF6SCRLQYKNA, seg_pos={0 0}, upload=446559]: Uploading piece... Dec 26 20:23:03.847407088 (508h54m3.718490133s) [noise=true , address=145.79.227.25:4618 , piece_id=HKVEEFMYIQQ3PZECXVWLS24FZC2JBLCZKS6OVJD3VWYFEZ6445ZA, seg_pos={0 0}, upload=446559]: Uploading piece... Dec 26 20:23:03.847776832 (508h54m3.718859867s) [noise=true , address=145.79.227.38:4978 , piece_id=YRQP2UJSII5EX47YQCRBBCJZNSUYOUVNJ6EVCW5KVOJWQORAM52Q, seg_pos={0 0}, upload=446559]: Uploading piece... Dec 26 20:23:03.848155603 (508h54m3.719238638s) [noise=true , address=145.79.227.39:4013 , piece_id=ZLVWOLLWB6RE3SL5DBPDE5VBFPWG6AT45KMXAFRNBO65I7IOZLYA, seg_pos={0 0}, upload=446559]: Uploading piece... Dec 26 20:23:03.848605089 (508h54m3.719688134s) [noise=true , address=77.90.31.17:4770 , piece_id=OCUUL35OXTDELCUL44KI75ZRJMRQ2BEZXSRMAHQGEWDOOUM6WLKA, seg_pos={0 0}, upload=446559]: Uploading piece... Dec 26 20:23:05.557671430 (508h54m5.428754475s) [upload=446559]: Next returned nil segment Dec 26 20:23:05.557736103 (508h54m5.428819138s) [upload=446559]: Waiting for error group managing segments... Dec 26 20:23:05.849417051 (508h54m5.720500086s) [noise=true , address=145.79.227.38:4978 , piece_id=YRQP2UJSII5EX47YQCRBBCJZNSUYOUVNJ6EVCW5KVOJWQORAM52Q, seg_pos={0 0}, upload=446559]: Done uploading piece. err: <nil> Dec 26 20:23:05.857270585 (508h54m5.72835363s) [noise=true , address=145.79.227.37:4565 , piece_id=E42J2H5FOJYU3QJ6QCAJGZ6R33OHALNAWE3LE47X7Y33A56RIGUQ, seg_pos={0 0}, upload=446559]: Done uploading piece. err: <nil> Dec 26 20:23:05.863831258 (508h54m5.734914294s) [noise=true , address=145.79.227.11:4730 , piece_id=HEK7KDSHBAXR24SQ4ZK4JGOG7YH7LL6TV2JOGXGS7NSZE323STEA, seg_pos={0 0}, upload=446559]: Done uploading piece. err: <nil> Dec 26 20:23:05.865686220 (508h54m5.736769255s) [noise=true , address=145.79.227.48:4302 , piece_id=KBAWJG5R57OK35DU7KSB4ORIEX5WDTODHBUEZY4R3KJHCFMCN4JA, seg_pos={0 0}, upload=446559]: Done uploading piece. err: <nil> Dec 26 20:23:05.867986480 (508h54m5.739069515s) [noise=true , address=145.79.227.52:4446 , piece_id=RQKS4XE7RT7Q54E4WW3W7X7K6XF6SR37UM354VGHYMTYF6UQ6CXA, seg_pos={0 0}, upload=446559]: Done uploading piece. err: <nil> Dec 26 20:23:05.874968957 (508h54m5.746051992s) [noise=true , address=145.79.227.49:4337 , piece_id=6MO6YVADQNKSJWKVDYF6THUVWU7T5TWBY4JJDUZ6NIKK6N45CJBQ, seg_pos={0 0}, upload=446559]: Done uploading piece. err: <nil> Dec 26 20:23:05.890746710 (508h54m5.761829745s) [noise=true , address=145.79.227.39:4013 , piece_id=ZLVWOLLWB6RE3SL5DBPDE5VBFPWG6AT45KMXAFRNBO65I7IOZLYA, seg_pos={0 0}, upload=446559]: Done uploading piece. err: <nil> Dec 26 20:23:05.978539391 (508h54m5.849622436s) [noise=true , address=145.79.227.21:4475 , piece_id=RHDIL6I6YMOXGGQBASRUE63W6X452O4EBNDXGWSBOBB3UNOPF2HQ, seg_pos={0 0}, upload=446559]: Done uploading piece. err: <nil> Dec 26 20:23:06.104062388 (508h54m5.975145423s) [noise=true , address=45.91.116.11:4118 , piece_id=JTDLKRGYANDXVIFTNFKEW4K7MIHCJYAGMBEYDK6OPGT37EPDDKLA, seg_pos={0 0}, upload=446559]: Done uploading piece. err: <nil> Dec 26 20:23:06.139385348 (508h54m6.010468383s) [noise=true , address=45.91.116.14:4191 , piece_id=DBNRTQ7MSGVAHGPF5XEPPQBECW6IOD22VELUBZEDOF6SCRLQYKNA, seg_pos={0 0}, upload=446559]: Done uploading piece. err: <nil> Dec 26 20:23:06.148938228 (508h54m6.020021263s) [noise=true , address=77.90.31.13:4627 , piece_id=DRGW6ZDTAHUMTZLUHJBVXOGDEJHSRKWB7M3Q6TERL755ZLXXCAGA, seg_pos={0 0}, upload=446559]: Done uploading piece. err: <nil> Dec 26 20:23:06.158201787 (508h54m6.029284823s) [noise=true , address=45.91.116.5:4298 , piece_id=YWBVQX37SM57E5NBOUPSQB3IHSP6J42EK2ORULGAC5CABUVBEXYQ, seg_pos={0 0}, upload=446559]: Done uploading piece. err: <nil> Dec 26 20:23:06.211138317 (508h54m6.082221362s) [noise=true , address=45.91.116.13:4153 , piece_id=YHNRYXDNOSU2RMTRVQU46H2D73V5KJBGKWG6JLL5AMUCUIN5K3YQ, seg_pos={0 0}, upload=446559]: Done uploading piece. err: <nil> Dec 26 20:23:06.211158235 (508h54m6.082241271s) [seg_pos={0 0}, upload=446559]: Segment reached optimal threshold of 13 pieces. Dec 26 20:23:06.211398393 (508h54m6.082481438s) [noise=true , address=77.90.31.19:4844 , piece_id=P7HYROFKCFIOG7SGWFGEGZW6ZALEHXDL7JQP7G23JEYSIMW4LY7A, seg_pos={0 0}, upload=446559]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: write tcp 188.166.225.170:38476->77.90.31.19:4844: use of closed network connection; write tcp 188.166.225.170:38476->77.90.31.19:4844: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38476->77.90.31.19:4844: use of closed network connection Dec 26 20:23:06.211407931 (508h54m6.082490986s) [noise=true , address=77.90.31.19:4844 , piece_id=P7HYROFKCFIOG7SGWFGEGZW6ZALEHXDL7JQP7G23JEYSIMW4LY7A, seg_pos={0 0}, upload=446559]: Long tail cancellation due to optimal threshold Dec 26 20:23:06.211621698 (508h54m6.082704743s) [noise=true , address=77.90.31.17:4770 , piece_id=OCUUL35OXTDELCUL44KI75ZRJMRQ2BEZXSRMAHQGEWDOOUM6WLKA, seg_pos={0 0}, upload=446559]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: write tcp 188.166.225.170:39932->77.90.31.17:4770: use of closed network connection; write tcp 188.166.225.170:39932->77.90.31.17:4770: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39932->77.90.31.17:4770: use of closed network connection Dec 26 20:23:06.211645834 (508h54m6.082728889s) [noise=true , address=77.90.31.17:4770 , piece_id=OCUUL35OXTDELCUL44KI75ZRJMRQ2BEZXSRMAHQGEWDOOUM6WLKA, seg_pos={0 0}, upload=446559]: Long tail cancellation due to optimal threshold Dec 26 20:23:06.211767345 (508h54m6.08285038s) [noise=true , address=145.79.227.25:4618 , piece_id=HKVEEFMYIQQ3PZECXVWLS24FZC2JBLCZKS6OVJD3VWYFEZ6445ZA, seg_pos={0 0}, upload=446559]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8): protocol: write tcp 188.166.225.170:55502->145.79.227.25:4618: use of closed network connection; write tcp 188.166.225.170:55502->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55502->145.79.227.25:4618: use of closed network connection Dec 26 20:23:06.211772566 (508h54m6.082855601s) [noise=true , address=145.79.227.25:4618 , piece_id=HKVEEFMYIQQ3PZECXVWLS24FZC2JBLCZKS6OVJD3VWYFEZ6445ZA, seg_pos={0 0}, upload=446559]: Long tail cancellation due to optimal threshold Dec 26 20:23:06.211787854 (508h54m6.082870889s) [seg_pos={0 0}, upload=446559]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 20:23:06.211882755 (508h54m6.08296579s) [upload=446559]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:23:06.211887805 (508h54m6.08297084s) [upload=446559]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:23:06.517193327 (508h54m6.388276372s) [upload=446559]: Done uploading segments... Dec 26 20:23:06.517216170 (508h54m6.388299205s) [upload=446559]: Upload finished. err: <nil> Duration:* 6.613606988s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [........X..........................................X........] duration: 37.584519856s trace: https://jaeger.dev.storj.io/trace/663e21624a5a3654 FAIL (2025-12-26 20:23:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, Piece ID: PWVMOEKW5A4ZAVSVJAAHFTYYTQZGFHDF642GWR6EABEGWP2CDRGQ) context deadline exceeded; piecestore: (Node ID: 12whTD1Kvu4T7ANNtkSvDB66uTjVgfPTK9TAzExgvwJYgZ3zSFG, Piece ID: ME2LDK2QJI7LLG67ODDIJDKB5CAT47YDGMJ3CTVK4F6UDHJ4TPOQ) context deadline exceeded; piecestore: (Node ID: 12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH, Piece ID: HJGRPPEJK75YTIKBKNVK7J4TDRKBCNL7HA4JNMOKD2ROGC7C4P3Q) context deadline exceeded ======LOGS====== ) [noise=true , address=145.79.227.7:4296 , piece_id=PVH6VN4CG4O2DC22G74US7IC5GDIGM2KYYYQQPFO5MG22JQCAU2A, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.248055070 (508h48m1.119138115s) [noise=true , address=145.79.227.40:4078 , piece_id=J6TGA3SFQOLGL2C2TS352TSLZBMI6MDXU5BDTTGQYMN4A5QG462A, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.248175840 (508h48m1.119258875s) [noise=true , address=77.90.31.12:4600 , piece_id=JESM5JF4GWUDLEW4YT5SKHJOXD54TWRQBNGGQCBMDSS54XR4AEKQ, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.248604496 (508h48m1.119687531s) [noise=true , address=45.91.116.6:4985 , piece_id=WV5SXZIBMXB4FTNZFE4PKBR5ZWZTUUPADPRMXAVE4Z3HKDGLBOSQ, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.248640474 (508h48m1.119723509s) [noise=true , address=145.79.227.18:4021 , piece_id=ER7GXC5OHAPNGU37QTKPROKB4YTCDQBUHVEIEUH2K5FQ4X5AGM2Q, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.249180202 (508h48m1.120263238s) [noise=true , address=77.90.31.18:4821 , piece_id=HJGRPPEJK75YTIKBKNVK7J4TDRKBCNL7HA4JNMOKD2ROGC7C4P3Q, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.249851130 (508h48m1.120934175s) [noise=true , address=64.49.13.169:4709 , piece_id=ZGWBPAYFOQWU2LSTILYMVRNKGSP53UXEQHRW4NJEXW5OLNVCCWRQ, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.250638910 (508h48m1.121721955s) [noise=true , address=145.79.227.38:4001 , piece_id=TWPCSZEWSFC2ZT7QNWHRS3RY42LLBPIEF3EIQQ65OXFZSHWAV2GA, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.251000769 (508h48m1.122083814s) [noise=true , address=145.79.227.30:4792 , piece_id=RLEUVLBAOMJSDAYHMQJGKYQUC6NYYGSGUVIU5Z53XGBINIQV43GQ, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.251314737 (508h48m1.122397772s) [noise=true , address=145.79.227.5:4252 , piece_id=FI5RNNRKSIYHR3JKCOW3THJQKHAPYLUDDHT3Y4TRYU3OSMCSWQQQ, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.251635527 (508h48m1.122718572s) [noise=true , address=77.90.31.11:4588 , piece_id=ZWDOVL5PGZ4YPY6NZVVSBSWXDDD3DAJ4RUK2BWMOGTHDGWJ2G4DA, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.251919499 (508h48m1.123002544s) [noise=true , address=77.90.31.10:4536 , piece_id=V43VRDKXTE2PHOBPSQUIJPKMKY3ROAEQAI2D4XIJHOLQ3UIMN4RA, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.252236542 (508h48m1.123319588s) [noise=true , address=77.90.31.21:4894 , piece_id=AGL6GVJX2YLGHSZX33SUURUU5S22CF3YRMJTP3QMHHP7OKX2MFSA, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.252528137 (508h48m1.123611182s) [noise=true , address=77.90.31.15:4702 , piece_id=KJEPR7NMX4UHQZHTMDDJKBRLLQ3O5BLJSLYBXGCOFGP3LNC3FXGA, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.252861402 (508h48m1.123944447s) [noise=true , address=64.49.13.27:4961 , piece_id=XCVH62CEVRYGR53U5VG4L4HJMBJSQ562L4QVKTXZSC7XGAEB4LDQ, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:02.570977410 (508h48m2.442060455s) [upload=446477]: Next returned nil segment Dec 26 20:17:02.571003019 (508h48m2.442086054s) [upload=446477]: Waiting for error group managing segments... Dec 26 20:17:02.949353997 (508h48m2.820437042s) [noise=true , address=64.49.13.27:4961 , piece_id=XCVH62CEVRYGR53U5VG4L4HJMBJSQ562L4QVKTXZSC7XGAEB4LDQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.136546190 (508h48m3.007629235s) [noise=true , address=145.79.227.5:4252 , piece_id=FI5RNNRKSIYHR3JKCOW3THJQKHAPYLUDDHT3Y4TRYU3OSMCSWQQQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.155760468 (508h48m3.026843513s) [noise=true , address=145.79.227.38:4001 , piece_id=TWPCSZEWSFC2ZT7QNWHRS3RY42LLBPIEF3EIQQ65OXFZSHWAV2GA, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.179239646 (508h48m3.050322691s) [noise=true , address=145.79.227.18:4021 , piece_id=ER7GXC5OHAPNGU37QTKPROKB4YTCDQBUHVEIEUH2K5FQ4X5AGM2Q, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.372384563 (508h48m3.243467608s) [noise=true , address=77.90.31.15:4702 , piece_id=KJEPR7NMX4UHQZHTMDDJKBRLLQ3O5BLJSLYBXGCOFGP3LNC3FXGA, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.447635803 (508h48m3.318718848s) [noise=true , address=77.90.31.18:4821 , piece_id=HJGRPPEJK75YTIKBKNVK7J4TDRKBCNL7HA4JNMOKD2ROGC7C4P3Q, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.452148166 (508h48m3.323231201s) [noise=true , address=77.90.31.10:4536 , piece_id=V43VRDKXTE2PHOBPSQUIJPKMKY3ROAEQAI2D4XIJHOLQ3UIMN4RA, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.644263002 (508h48m3.515346047s) [noise=true , address=77.90.31.14:4689 , piece_id=PWVMOEKW5A4ZAVSVJAAHFTYYTQZGFHDF642GWR6EABEGWP2CDRGQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.710620497 (508h48m3.581703542s) [noise=true , address=45.91.116.8:4074 , piece_id=ME2LDK2QJI7LLG67ODDIJDKB5CAT47YDGMJ3CTVK4F6UDHJ4TPOQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:04.066910008 (508h48m3.937993053s) [noise=true , address=77.90.31.12:4600 , piece_id=JESM5JF4GWUDLEW4YT5SKHJOXD54TWRQBNGGQCBMDSS54XR4AEKQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:04.082282349 (508h48m3.953365394s) [noise=true , address=145.79.227.44:4219 , piece_id=KZVV2NJBIU22C4URLBZMQVQN4IZ2ZMQ5YODRJMCS6XYXULHCZWLQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:04.089048724 (508h48m3.960131759s) [noise=true , address=145.79.227.30:4792 , piece_id=RLEUVLBAOMJSDAYHMQJGKYQUC6NYYGSGUVIU5Z53XGBINIQV43GQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:04.288772749 (508h48m4.159855794s) [noise=true , address=77.90.31.21:4894 , piece_id=AGL6GVJX2YLGHSZX33SUURUU5S22CF3YRMJTP3QMHHP7OKX2MFSA, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:04.344183239 (508h48m4.215266294s) [noise=true , address=77.90.31.11:4588 , piece_id=ZWDOVL5PGZ4YPY6NZVVSBSWXDDD3DAJ4RUK2BWMOGTHDGWJ2G4DA, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:04.344208728 (508h48m4.215291773s) [seg_pos={0 0}, upload=446477]: Segment reached optimal threshold of 14 pieces. Dec 26 20:17:04.344470828 (508h48m4.215553873s) [noise=true , address=64.49.13.169:4709 , piece_id=ZGWBPAYFOQWU2LSTILYMVRNKGSP53UXEQHRW4NJEXW5OLNVCCWRQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:47062->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:47062->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47062->64.49.13.169:4709: use of closed network connection Dec 26 20:17:04.344481918 (508h48m4.215564953s) [noise=true , address=64.49.13.169:4709 , piece_id=ZGWBPAYFOQWU2LSTILYMVRNKGSP53UXEQHRW4NJEXW5OLNVCCWRQ, seg_pos={0 0}, upload=446477]: Long tail cancellation due to optimal threshold Dec 26 20:17:04.344541813 (508h48m4.215624848s) [noise=true , address=145.79.227.40:4078 , piece_id=J6TGA3SFQOLGL2C2TS352TSLZBMI6MDXU5BDTTGQYMN4A5QG462A, seg_pos={0 0}, upload=446477]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tzgtenV6us5ssD7xWGpZmVUoWEfbFhBNewbK9irHtZPkyDtRd): protocol: expected piece hash; context canceled; EOF Dec 26 20:17:04.344559476 (508h48m4.215642511s) [noise=true , address=145.79.227.40:4078 , piece_id=J6TGA3SFQOLGL2C2TS352TSLZBMI6MDXU5BDTTGQYMN4A5QG462A, seg_pos={0 0}, upload=446477]: Long tail cancellation due to optimal threshold Dec 26 20:17:04.344755479 (508h48m4.215838524s) [noise=true , address=145.79.227.7:4296 , piece_id=PVH6VN4CG4O2DC22G74US7IC5GDIGM2KYYYQQPFO5MG22JQCAU2A, seg_pos={0 0}, upload=446477]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cKhDbA4ptwux2LYNPNAJ7Qh8jECsGWd1KWaK6T6pD38bJWdt1): protocol: write tcp 188.166.225.170:48104->145.79.227.7:4296: use of closed network connection; write tcp 188.166.225.170:48104->145.79.227.7:4296: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48104->145.79.227.7:4296: use of closed network connection Dec 26 20:17:04.344763044 (508h48m4.215846099s) [noise=true , address=145.79.227.7:4296 , piece_id=PVH6VN4CG4O2DC22G74US7IC5GDIGM2KYYYQQPFO5MG22JQCAU2A, seg_pos={0 0}, upload=446477]: Long tail cancellation due to optimal threshold Dec 26 20:17:04.344928509 (508h48m4.216011544s) [noise=true , address=45.91.116.6:4985 , piece_id=WV5SXZIBMXB4FTNZFE4PKBR5ZWZTUUPADPRMXAVE4Z3HKDGLBOSQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RNs9MM9jDtNC2gZC63SYsePZTd5mE7dqUm9A5Z5QWdPX8PVnp): protocol: expected piece hash; context canceled; EOF Dec 26 20:17:04.344938277 (508h48m4.216021312s) [noise=true , address=45.91.116.6:4985 , piece_id=WV5SXZIBMXB4FTNZFE4PKBR5ZWZTUUPADPRMXAVE4Z3HKDGLBOSQ, seg_pos={0 0}, upload=446477]: Long tail cancellation due to optimal threshold Dec 26 20:17:04.344966722 (508h48m4.216049767s) [seg_pos={0 0}, upload=446477]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:17:04.345054548 (508h48m4.216137593s) [upload=446477]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:17:04.345059839 (508h48m4.216142874s) [upload=446477]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:17:04.653613492 (508h48m4.524696537s) [upload=446477]: Done uploading segments... Dec 26 20:17:04.653639822 (508h48m4.524722857s) [upload=446477]: Upload finished. err: <nil> Duration: 37.584547409s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [X.X.X.X.X..............XX..X..X...........XXX.X.X....X..XXXX] duration: 30.002309803s trace: https://jaeger.dev.storj.io/trace/38d6fc2548068ce1 FAIL (2025-12-26 20:23:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== BatchRequestItem_ObjectBegin Dec 26 20:23:03.589900904 (508h54m3.460983939s) [upload=446559]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 20:23:03.839763603 (508h54m3.710846648s) [seg_pos={0 0}, upload=446559]: Begin upload segment... Dec 26 20:23:03.839814761 (508h54m3.710897806s) [seg_pos={0 0}, upload=446559]: Done begin upload segment. Dec 26 20:23:03.839842353 (508h54m3.710925388s) [noise=true , address=77.90.31.19:4844 , piece_id=P7HYROFKCFIOG7SGWFGEGZW6ZALEHXDL7JQP7G23JEYSIMW4LY7A, seg_pos={0 0}, upload=446559]: Uploading piece... Dec 26 20:23:03.840018929 (508h54m3.711101974s) [noise=true , address=45.91.116.5:4298 , piece_id=YWBVQX37SM57E5NBOUPSQB3IHSP6J42EK2ORULGAC5CABUVBEXYQ, seg_pos={0 0}, upload=446559]: Uploading piece... Dec 26 20:23:03.840890459 (508h54m3.711973505s) [noise=true , address=45.91.116.11:4118 , piece_id=JTDLKRGYANDXVIFTNFKEW4K7MIHCJYAGMBEYDK6OPGT37EPDDKLA, seg_pos={0 0}, upload=446559]: Uploading piece... Dec 26 20:23:03.841143541 (508h54m3.712226586s) [noise=true , address=145.79.227.21:4475 , piece_id=RHDIL6I6YMOXGGQBASRUE63W6X452O4EBNDXGWSBOBB3UNOPF2HQ, seg_pos={0 0}, upload=446559]: Uploading piece... Dec 26 20:23:03.841716412 (508h54m3.712799467s) [noise=true , address=145.79.227.11:4730 , piece_id=HEK7KDSHBAXR24SQ4ZK4JGOG7YH7LL6TV2JOGXGS7NSZE323STEA, seg_pos={0 0}, upload=446559]: Uploading piece... Dec 26 20:23:03.842282319 (508h54m3.713365364s) [noise=true , address=145.79.227.49:4337 , piece_id=6MO6YVADQNKSJWKVDYF6THUVWU7T5TWBY4JJDUZ6NIKK6N45CJBQ, seg_pos={0 0}, upload=446559]: Uploading piece... Dec 26 20:23:03.842913472 (508h54m3.713996517s) [noise=true , address=145.79.227.52:4446 , piece_id=RQKS4XE7RT7Q54E4WW3W7X7K6XF6SR37UM354VGHYMTYF6UQ6CXA, seg_pos={0 0}, upload=446559]: Uploading piece... Dec 26 20:23:03.843372215 (508h54m3.71445526s) [noise=true , address=77.90.31.13:4627 , piece_id=DRGW6ZDTAHUMTZLUHJBVXOGDEJHSRKWB7M3Q6TERL755ZLXXCAGA, seg_pos={0 0}, upload=446559]: Uploading piece... Dec 26 20:23:03.843875773 (508h54m3.714958818s) [noise=true , address=145.79.227.37:4565 , piece_id=E42J2H5FOJYU3QJ6QCAJGZ6R33OHALNAWE3LE47X7Y33A56RIGUQ, seg_pos={0 0}, upload=446559]: Uploading piece... Dec 26 20:23:03.844433996 (508h54m3.715517041s) [noise=true , address=45.91.116.13:4153 , piece_id=YHNRYXDNOSU2RMTRVQU46H2D73V5KJBGKWG6JLL5AMUCUIN5K3YQ, seg_pos={0 0}, upload=446559]: Uploading piece... Dec 26 20:23:03.846328695 (508h54m3.71741173s) [noise=true , address=145.79.227.48:4302 , piece_id=KBAWJG5R57OK35DU7KSB4ORIEX5WDTODHBUEZY4R3KJHCFMCN4JA, seg_pos={0 0}, upload=446559]: Uploading piece... Dec 26 20:23:03.846965006 (508h54m3.718048051s) [noise=true , address=45.91.116.14:4191 , piece_id=DBNRTQ7MSGVAHGPF5XEPPQBECW6IOD22VELUBZEDOF6SCRLQYKNA, seg_pos={0 0}, upload=446559]: Uploading piece... Dec 26 20:23:03.847407088 (508h54m3.718490133s) [noise=true , address=145.79.227.25:4618 , piece_id=HKVEEFMYIQQ3PZECXVWLS24FZC2JBLCZKS6OVJD3VWYFEZ6445ZA, seg_pos={0 0}, upload=446559]: Uploading piece... Dec 26 20:23:03.847776832 (508h54m3.718859867s) [noise=true , address=145.79.227.38:4978 , piece_id=YRQP2UJSII5EX47YQCRBBCJZNSUYOUVNJ6EVCW5KVOJWQORAM52Q, seg_pos={0 0}, upload=446559]: Uploading piece... Dec 26 20:23:03.848155603 (508h54m3.719238638s) [noise=true , address=145.79.227.39:4013 , piece_id=ZLVWOLLWB6RE3SL5DBPDE5VBFPWG6AT45KMXAFRNBO65I7IOZLYA, seg_pos={0 0}, upload=446559]: Uploading piece... Dec 26 20:23:03.848605089 (508h54m3.719688134s) [noise=true , address=77.90.31.17:4770 , piece_id=OCUUL35OXTDELCUL44KI75ZRJMRQ2BEZXSRMAHQGEWDOOUM6WLKA, seg_pos={0 0}, upload=446559]: Uploading piece... Dec 26 20:23:05.557671430 (508h54m5.428754475s) [upload=446559]: Next returned nil segment Dec 26 20:23:05.557736103 (508h54m5.428819138s) [upload=446559]: Waiting for error group managing segments... Dec 26 20:23:05.849417051 (508h54m5.720500086s) [noise=true , address=145.79.227.38:4978 , piece_id=YRQP2UJSII5EX47YQCRBBCJZNSUYOUVNJ6EVCW5KVOJWQORAM52Q, seg_pos={0 0}, upload=446559]: Done uploading piece. err: <nil> Dec 26 20:23:05.857270585 (508h54m5.72835363s) [noise=true , address=145.79.227.37:4565 , piece_id=E42J2H5FOJYU3QJ6QCAJGZ6R33OHALNAWE3LE47X7Y33A56RIGUQ, seg_pos={0 0}, upload=446559]: Done uploading piece. err: <nil> Dec 26 20:23:05.863831258 (508h54m5.734914294s) [noise=true , address=145.79.227.11:4730 , piece_id=HEK7KDSHBAXR24SQ4ZK4JGOG7YH7LL6TV2JOGXGS7NSZE323STEA, seg_pos={0 0}, upload=446559]: Done uploading piece. err: <nil> Dec 26 20:23:05.865686220 (508h54m5.736769255s) [noise=true , address=145.79.227.48:4302 , piece_id=KBAWJG5R57OK35DU7KSB4ORIEX5WDTODHBUEZY4R3KJHCFMCN4JA, seg_pos={0 0}, upload=446559]: Done uploading piece. err: <nil> Dec 26 20:23:05.867986480 (508h54m5.739069515s) [noise=true , address=145.79.227.52:4446 , piece_id=RQKS4XE7RT7Q54E4WW3W7X7K6XF6SR37UM354VGHYMTYF6UQ6CXA, seg_pos={0 0}, upload=446559]: Done uploading piece. err: <nil> Dec 26 20:23:05.874968957 (508h54m5.746051992s) [noise=true , address=145.79.227.49:4337 , piece_id=6MO6YVADQNKSJWKVDYF6THUVWU7T5TWBY4JJDUZ6NIKK6N45CJBQ, seg_pos={0 0}, upload=446559]: Done uploading piece. err: <nil> Dec 26 20:23:05.890746710 (508h54m5.761829745s) [noise=true , address=145.79.227.39:4013 , piece_id=ZLVWOLLWB6RE3SL5DBPDE5VBFPWG6AT45KMXAFRNBO65I7IOZLYA, seg_pos={0 0}, upload=446559]: Done uploading piece. err: <nil> Dec 26 20:23:05.978539391 (508h54m5.849622436s) [noise=true , address=145.79.227.21:4475 , piece_id=RHDIL6I6YMOXGGQBASRUE63W6X452O4EBNDXGWSBOBB3UNOPF2HQ, seg_pos={0 0}, upload=446559]: Done uploading piece. err: <nil> Dec 26 20:23:06.104062388 (508h54m5.975145423s) [noise=true , address=45.91.116.11:4118 , piece_id=JTDLKRGYANDXVIFTNFKEW4K7MIHCJYAGMBEYDK6OPGT37EPDDKLA, seg_pos={0 0}, upload=446559]: Done uploading piece. err: <nil> Dec 26 20:23:06.139385348 (508h54m6.010468383s) [noise=true , address=45.91.116.14:4191 , piece_id=DBNRTQ7MSGVAHGPF5XEPPQBECW6IOD22VELUBZEDOF6SCRLQYKNA, seg_pos={0 0}, upload=446559]: Done uploading piece. err: <nil> Dec 26 20:23:06.148938228 (508h54m6.020021263s) [noise=true , address=77.90.31.13:4627 , piece_id=DRGW6ZDTAHUMTZLUHJBVXOGDEJHSRKWB7M3Q6TERL755ZLXXCAGA, seg_pos={0 0}, upload=446559]: Done uploading piece. err: <nil> Dec 26 20:23:06.158201787 (508h54m6.029284823s) [noise=true , address=45.91.116.5:4298 , piece_id=YWBVQX37SM57E5NBOUPSQB3IHSP6J42EK2ORULGAC5CABUVBEXYQ, seg_pos={0 0}, upload=446559]: Done uploading piece. err: <nil> Dec 26 20:23:06.211138317 (508h54m6.082221362s) [noise=true , address=45.91.116.13:4153 , piece_id=YHNRYXDNOSU2RMTRVQU46H2D73V5KJBGKWG6JLL5AMUCUIN5K3YQ, seg_pos={0 0}, upload=446559]: Done uploading piece. err: <nil> Dec 26 20:23:06.211158235 (508h54m6.082241271s) [seg_pos={0 0}, upload=446559]: Segment reached optimal threshold of 13 pieces. Dec 26 20:23:06.211398393 (508h54m6.082481438s) [noise=true , address=77.90.31.19:4844 , piece_id=P7HYROFKCFIOG7SGWFGEGZW6ZALEHXDL7JQP7G23JEYSIMW4LY7A, seg_pos={0 0}, upload=446559]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: write tcp 188.166.225.170:38476->77.90.31.19:4844: use of closed network connection; write tcp 188.166.225.170:38476->77.90.31.19:4844: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38476->77.90.31.19:4844: use of closed network connection Dec 26 20:23:06.211407931 (508h54m6.082490986s) [noise=true , address=77.90.31.19:4844 , piece_id=P7HYROFKCFIOG7SGWFGEGZW6ZALEHXDL7JQP7G23JEYSIMW4LY7A, seg_pos={0 0}, upload=446559]: Long tail cancellation due to optimal threshold Dec 26 20:23:06.211621698 (508h54m6.082704743s) [noise=true , address=77.90.31.17:4770 , piece_id=OCUUL35OXTDELCUL44KI75ZRJMRQ2BEZXSRMAHQGEWDOOUM6WLKA, seg_pos={0 0}, upload=446559]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: write tcp 188.166.225.170:39932->77.90.31.17:4770: use of closed network connection; write tcp 188.166.225.170:39932->77.90.31.17:4770: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39932->77.90.31.17:4770: use of closed network connection Dec 26 20:23:06.211645834 (508h54m6.082728889s) [noise=true , address=77.90.31.17:4770 , piece_id=OCUUL35OXTDELCUL44KI75ZRJMRQ2BEZXSRMAHQGEWDOOUM6WLKA, seg_pos={0 0}, upload=446559]: Long tail cancellation due to optimal threshold Dec 26 20:23:06.211767345 (508h54m6.08285038s) [noise=true , address=145.79.227.25:4618 , piece_id=HKVEEFMYIQQ3PZECXVWLS24FZC2JBLCZKS6OVJD3VWYFEZ6445ZA, seg_pos={0 0}, upload=446559]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8): protocol: write tcp 188.166.225.170:55502->145.79.227.25:4618: use of closed network connection; write tcp 188.166.225.170:55502->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55502->145.79.227.25:4618: use of closed network connection Dec 26 20:23:06.211772566 (508h54m6.082855601s) [noise=true , address=145.79.227.25:4618 , piece_id=HKVEEFMYIQQ3PZECXVWLS24FZC2JBLCZKS6OVJD3VWYFEZ6445ZA, seg_pos={0 0}, upload=446559]: Long tail cancellation due to optimal threshold Dec 26 20:23:06.211787854 (508h54m6.082870889s) [seg_pos={0 0}, upload=446559]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 20:23:06.211882755 (508h54m6.08296579s) [upload=446559]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:23:06.211887805 (508h54m6.08297084s) [upload=446559]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:23:06.517193327 (508h54m6.388276372s) [upload=446559]: Done uploading segments... Dec 26 20:23:06.517216170 (508h54m6.388299205s) [upload=446559]: Upload finished. err: <nil> Duration:* 30.002335871s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........X..........................................X.......] duration: 32.528693589s trace: https://jaeger.dev.storj.io/trace/663e21624a5a3654 FAIL (2025-12-26 20:22:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, Piece ID: PWVMOEKW5A4ZAVSVJAAHFTYYTQZGFHDF642GWR6EABEGWP2CDRGQ) context deadline exceeded; piecestore: (Node ID: 12whTD1Kvu4T7ANNtkSvDB66uTjVgfPTK9TAzExgvwJYgZ3zSFG, Piece ID: ME2LDK2QJI7LLG67ODDIJDKB5CAT47YDGMJ3CTVK4F6UDHJ4TPOQ) context deadline exceeded; piecestore: (Node ID: 12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH, Piece ID: HJGRPPEJK75YTIKBKNVK7J4TDRKBCNL7HA4JNMOKD2ROGC7C4P3Q) context deadline exceeded ======LOGS====== ) [noise=true , address=145.79.227.7:4296 , piece_id=PVH6VN4CG4O2DC22G74US7IC5GDIGM2KYYYQQPFO5MG22JQCAU2A, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.248055070 (508h48m1.119138115s) [noise=true , address=145.79.227.40:4078 , piece_id=J6TGA3SFQOLGL2C2TS352TSLZBMI6MDXU5BDTTGQYMN4A5QG462A, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.248175840 (508h48m1.119258875s) [noise=true , address=77.90.31.12:4600 , piece_id=JESM5JF4GWUDLEW4YT5SKHJOXD54TWRQBNGGQCBMDSS54XR4AEKQ, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.248604496 (508h48m1.119687531s) [noise=true , address=45.91.116.6:4985 , piece_id=WV5SXZIBMXB4FTNZFE4PKBR5ZWZTUUPADPRMXAVE4Z3HKDGLBOSQ, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.248640474 (508h48m1.119723509s) [noise=true , address=145.79.227.18:4021 , piece_id=ER7GXC5OHAPNGU37QTKPROKB4YTCDQBUHVEIEUH2K5FQ4X5AGM2Q, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.249180202 (508h48m1.120263238s) [noise=true , address=77.90.31.18:4821 , piece_id=HJGRPPEJK75YTIKBKNVK7J4TDRKBCNL7HA4JNMOKD2ROGC7C4P3Q, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.249851130 (508h48m1.120934175s) [noise=true , address=64.49.13.169:4709 , piece_id=ZGWBPAYFOQWU2LSTILYMVRNKGSP53UXEQHRW4NJEXW5OLNVCCWRQ, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.250638910 (508h48m1.121721955s) [noise=true , address=145.79.227.38:4001 , piece_id=TWPCSZEWSFC2ZT7QNWHRS3RY42LLBPIEF3EIQQ65OXFZSHWAV2GA, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.251000769 (508h48m1.122083814s) [noise=true , address=145.79.227.30:4792 , piece_id=RLEUVLBAOMJSDAYHMQJGKYQUC6NYYGSGUVIU5Z53XGBINIQV43GQ, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.251314737 (508h48m1.122397772s) [noise=true , address=145.79.227.5:4252 , piece_id=FI5RNNRKSIYHR3JKCOW3THJQKHAPYLUDDHT3Y4TRYU3OSMCSWQQQ, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.251635527 (508h48m1.122718572s) [noise=true , address=77.90.31.11:4588 , piece_id=ZWDOVL5PGZ4YPY6NZVVSBSWXDDD3DAJ4RUK2BWMOGTHDGWJ2G4DA, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.251919499 (508h48m1.123002544s) [noise=true , address=77.90.31.10:4536 , piece_id=V43VRDKXTE2PHOBPSQUIJPKMKY3ROAEQAI2D4XIJHOLQ3UIMN4RA, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.252236542 (508h48m1.123319588s) [noise=true , address=77.90.31.21:4894 , piece_id=AGL6GVJX2YLGHSZX33SUURUU5S22CF3YRMJTP3QMHHP7OKX2MFSA, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.252528137 (508h48m1.123611182s) [noise=true , address=77.90.31.15:4702 , piece_id=KJEPR7NMX4UHQZHTMDDJKBRLLQ3O5BLJSLYBXGCOFGP3LNC3FXGA, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.252861402 (508h48m1.123944447s) [noise=true , address=64.49.13.27:4961 , piece_id=XCVH62CEVRYGR53U5VG4L4HJMBJSQ562L4QVKTXZSC7XGAEB4LDQ, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:02.570977410 (508h48m2.442060455s) [upload=446477]: Next returned nil segment Dec 26 20:17:02.571003019 (508h48m2.442086054s) [upload=446477]: Waiting for error group managing segments... Dec 26 20:17:02.949353997 (508h48m2.820437042s) [noise=true , address=64.49.13.27:4961 , piece_id=XCVH62CEVRYGR53U5VG4L4HJMBJSQ562L4QVKTXZSC7XGAEB4LDQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.136546190 (508h48m3.007629235s) [noise=true , address=145.79.227.5:4252 , piece_id=FI5RNNRKSIYHR3JKCOW3THJQKHAPYLUDDHT3Y4TRYU3OSMCSWQQQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.155760468 (508h48m3.026843513s) [noise=true , address=145.79.227.38:4001 , piece_id=TWPCSZEWSFC2ZT7QNWHRS3RY42LLBPIEF3EIQQ65OXFZSHWAV2GA, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.179239646 (508h48m3.050322691s) [noise=true , address=145.79.227.18:4021 , piece_id=ER7GXC5OHAPNGU37QTKPROKB4YTCDQBUHVEIEUH2K5FQ4X5AGM2Q, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.372384563 (508h48m3.243467608s) [noise=true , address=77.90.31.15:4702 , piece_id=KJEPR7NMX4UHQZHTMDDJKBRLLQ3O5BLJSLYBXGCOFGP3LNC3FXGA, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.447635803 (508h48m3.318718848s) [noise=true , address=77.90.31.18:4821 , piece_id=HJGRPPEJK75YTIKBKNVK7J4TDRKBCNL7HA4JNMOKD2ROGC7C4P3Q, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.452148166 (508h48m3.323231201s) [noise=true , address=77.90.31.10:4536 , piece_id=V43VRDKXTE2PHOBPSQUIJPKMKY3ROAEQAI2D4XIJHOLQ3UIMN4RA, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.644263002 (508h48m3.515346047s) [noise=true , address=77.90.31.14:4689 , piece_id=PWVMOEKW5A4ZAVSVJAAHFTYYTQZGFHDF642GWR6EABEGWP2CDRGQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.710620497 (508h48m3.581703542s) [noise=true , address=45.91.116.8:4074 , piece_id=ME2LDK2QJI7LLG67ODDIJDKB5CAT47YDGMJ3CTVK4F6UDHJ4TPOQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:04.066910008 (508h48m3.937993053s) [noise=true , address=77.90.31.12:4600 , piece_id=JESM5JF4GWUDLEW4YT5SKHJOXD54TWRQBNGGQCBMDSS54XR4AEKQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:04.082282349 (508h48m3.953365394s) [noise=true , address=145.79.227.44:4219 , piece_id=KZVV2NJBIU22C4URLBZMQVQN4IZ2ZMQ5YODRJMCS6XYXULHCZWLQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:04.089048724 (508h48m3.960131759s) [noise=true , address=145.79.227.30:4792 , piece_id=RLEUVLBAOMJSDAYHMQJGKYQUC6NYYGSGUVIU5Z53XGBINIQV43GQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:04.288772749 (508h48m4.159855794s) [noise=true , address=77.90.31.21:4894 , piece_id=AGL6GVJX2YLGHSZX33SUURUU5S22CF3YRMJTP3QMHHP7OKX2MFSA, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:04.344183239 (508h48m4.215266294s) [noise=true , address=77.90.31.11:4588 , piece_id=ZWDOVL5PGZ4YPY6NZVVSBSWXDDD3DAJ4RUK2BWMOGTHDGWJ2G4DA, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:04.344208728 (508h48m4.215291773s) [seg_pos={0 0}, upload=446477]: Segment reached optimal threshold of 14 pieces. Dec 26 20:17:04.344470828 (508h48m4.215553873s) [noise=true , address=64.49.13.169:4709 , piece_id=ZGWBPAYFOQWU2LSTILYMVRNKGSP53UXEQHRW4NJEXW5OLNVCCWRQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:47062->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:47062->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47062->64.49.13.169:4709: use of closed network connection Dec 26 20:17:04.344481918 (508h48m4.215564953s) [noise=true , address=64.49.13.169:4709 , piece_id=ZGWBPAYFOQWU2LSTILYMVRNKGSP53UXEQHRW4NJEXW5OLNVCCWRQ, seg_pos={0 0}, upload=446477]: Long tail cancellation due to optimal threshold Dec 26 20:17:04.344541813 (508h48m4.215624848s) [noise=true , address=145.79.227.40:4078 , piece_id=J6TGA3SFQOLGL2C2TS352TSLZBMI6MDXU5BDTTGQYMN4A5QG462A, seg_pos={0 0}, upload=446477]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tzgtenV6us5ssD7xWGpZmVUoWEfbFhBNewbK9irHtZPkyDtRd): protocol: expected piece hash; context canceled; EOF Dec 26 20:17:04.344559476 (508h48m4.215642511s) [noise=true , address=145.79.227.40:4078 , piece_id=J6TGA3SFQOLGL2C2TS352TSLZBMI6MDXU5BDTTGQYMN4A5QG462A, seg_pos={0 0}, upload=446477]: Long tail cancellation due to optimal threshold Dec 26 20:17:04.344755479 (508h48m4.215838524s) [noise=true , address=145.79.227.7:4296 , piece_id=PVH6VN4CG4O2DC22G74US7IC5GDIGM2KYYYQQPFO5MG22JQCAU2A, seg_pos={0 0}, upload=446477]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cKhDbA4ptwux2LYNPNAJ7Qh8jECsGWd1KWaK6T6pD38bJWdt1): protocol: write tcp 188.166.225.170:48104->145.79.227.7:4296: use of closed network connection; write tcp 188.166.225.170:48104->145.79.227.7:4296: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48104->145.79.227.7:4296: use of closed network connection Dec 26 20:17:04.344763044 (508h48m4.215846099s) [noise=true , address=145.79.227.7:4296 , piece_id=PVH6VN4CG4O2DC22G74US7IC5GDIGM2KYYYQQPFO5MG22JQCAU2A, seg_pos={0 0}, upload=446477]: Long tail cancellation due to optimal threshold Dec 26 20:17:04.344928509 (508h48m4.216011544s) [noise=true , address=45.91.116.6:4985 , piece_id=WV5SXZIBMXB4FTNZFE4PKBR5ZWZTUUPADPRMXAVE4Z3HKDGLBOSQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RNs9MM9jDtNC2gZC63SYsePZTd5mE7dqUm9A5Z5QWdPX8PVnp): protocol: expected piece hash; context canceled; EOF Dec 26 20:17:04.344938277 (508h48m4.216021312s) [noise=true , address=45.91.116.6:4985 , piece_id=WV5SXZIBMXB4FTNZFE4PKBR5ZWZTUUPADPRMXAVE4Z3HKDGLBOSQ, seg_pos={0 0}, upload=446477]: Long tail cancellation due to optimal threshold Dec 26 20:17:04.344966722 (508h48m4.216049767s) [seg_pos={0 0}, upload=446477]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:17:04.345054548 (508h48m4.216137593s) [upload=446477]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:17:04.345059839 (508h48m4.216142874s) [upload=446477]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:17:04.653613492 (508h48m4.524696537s) [upload=446477]: Done uploading segments... Dec 26 20:17:04.653639822 (508h48m4.524722857s) [upload=446477]: Upload finished. err: <nil> Duration: 32.528718816s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [.X.X.X.X.X..............XX..X..X...........XXX.X.X....X..XXX] duration: 30.003966782s trace: https://jaeger.dev.storj.io/trace/35923325315666f7 FAIL (2025-12-26 20:22:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: HFCTQC4PSTY4AXYGU6QPEAAWMDFJ35HCCZ6JPJTVOVKSD6OLFCYQ) context deadline exceeded; piecestore: (Node ID: 12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, Piece ID: GVXG76OZI563QRKTNVA5W65AYIIJR3H2O2RPSYM5MH4GLIEJQVBA) context deadline exceeded ======LOGS====== 20:22:11.366825253 (508h53m11.237908298s) [upload=446548]: Starting upload Dec 26 20:22:11.366915074 (508h53m11.23799812s) [upload=446548]: Done starting upload Dec 26 20:22:11.366972523 (508h53m11.238055568s) [upload=446548]: Uploading segments... Dec 26 20:22:11.367243309 (508h53m11.238326344s) [upload=446548]: Got next segment. Inline: false Dec 26 20:22:11.367288365 (508h53m11.2383714s) [upload=446548]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 20:22:11.367294486 (508h53m11.238377521s) [upload=446548]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 20:22:11.614698121 (508h53m11.485781156s) [seg_pos={0 0}, upload=446548]: Begin upload segment... Dec 26 20:22:11.614752915 (508h53m11.48583596s) [seg_pos={0 0}, upload=446548]: Done begin upload segment. Dec 26 20:22:11.614784265 (508h53m11.48586731s) [noise=true , address=77.90.31.21:4877 , piece_id=S7GR27SF2KEGO6263IHQCXRHDZ3PRTAORF5ZAPFNL3764EOH4WVQ, seg_pos={0 0}, upload=446548]: Uploading piece... Dec 26 20:22:11.614902160 (508h53m11.485985205s) [noise=true , address=145.79.227.41:4086 , piece_id=4DGW3D2ZBCPJRWVEHBTOMQJ2NLKCSDEZC6PB2OORHWREB2LP47EA, seg_pos={0 0}, upload=446548]: Uploading piece... Dec 26 20:22:11.615397152 (508h53m11.486480197s) [noise=true , address=145.79.227.22:4509 , piece_id=QWK6MIMXHRGCPR3657GZ4OOL6WNEY3UZRRJ62EX5JLTF3XR2IMHA, seg_pos={0 0}, upload=446548]: Uploading piece... Dec 26 20:22:11.615896513 (508h53m11.486979558s) [noise=true , address=145.79.227.51:4410 , piece_id=M4QI5O7UMDDX3ZX6OI5UYMBX2QE5WEGT32YCAC2VU47DIHRS4FWQ, seg_pos={0 0}, upload=446548]: Uploading piece... Dec 26 20:22:11.616177738 (508h53m11.487260783s) [noise=true , address=145.79.227.47:4265 , piece_id=FEWPBCQ7DPHIQKH6JBQQ3E3MVXP6QWTNH3PZXCSPES6ROWU5G7GA, seg_pos={0 0}, upload=446548]: Uploading piece... Dec 26 20:22:11.616489652 (508h53m11.487572697s) [noise=true , address=145.79.227.11:4731 , piece_id=2INDLVZUH7AE6J6HFMHGO5O7CG3NHCUQXYUAP57NQJRH6ZUH3OSA, seg_pos={0 0}, upload=446548]: Uploading piece... Dec 26 20:22:11.616836152 (508h53m11.487919187s) [noise=true , address=145.79.227.8:4330 , piece_id=PVMOQCFLSNM6A3R7WETUFJRJT37LUNEX3HISXRYYHHWPFOQDOIMQ, seg_pos={0 0}, upload=446548]: Uploading piece... Dec 26 20:22:11.617165289 (508h53m11.488248334s) [noise=true , address=45.91.116.9:4261 , piece_id=KBNJNT22YQG6FQ73JODZAPQC43MPAXPSF2BPLNC6ZJD3Z2Y5R7KA, seg_pos={0 0}, upload=446548]: Uploading piece... Dec 26 20:22:11.617438439 (508h53m11.488521475s) [noise=true , address=145.79.227.49:4339 , piece_id=XG77JJAAFJPPSVTRLZJDOEV3CNW66D2BOZLUCWETCEJWRRUNC6WQ, seg_pos={0 0}, upload=446548]: Uploading piece... Dec 26 20:22:11.617763117 (508h53m11.488846162s) [noise=true , address=145.79.227.15:4876 , piece_id=7OKTYMB7F2H6C3LRCUI6PWGT3NHWECFAUHN34IQNTKPLZGMPC5EA, seg_pos={0 0}, upload=446548]: Uploading piece... Dec 26 20:22:11.618084810 (508h53m11.489167855s) [noise=true , address=145.79.227.24:4582 , piece_id=P7F7ZW2WSQZRYZDCRWY3HPM4LMLE5IJTOP34MOUZAKK2XCG4S5YQ, seg_pos={0 0}, upload=446548]: Uploading piece... Dec 26 20:22:11.619492741 (508h53m11.490575776s) [noise=true , address=145.79.227.37:4566 , piece_id=RNQ4AAIC7P4GWLYSECNRH24XBNOIKGSL3YG2C2HRD2NU7IVCTIJA, seg_pos={0 0}, upload=446548]: Uploading piece... Dec 26 20:22:11.619818140 (508h53m11.490901176s) [noise=true , address=45.91.116.8:4046 , piece_id=HFCTQC4PSTY4AXYGU6QPEAAWMDFJ35HCCZ6JPJTVOVKSD6OLFCYQ, seg_pos={0 0}, upload=446548]: Uploading piece... Dec 26 20:22:11.620114966 (508h53m11.491198001s) [noise=true , address=145.79.227.40:4050 , piece_id=7LOB6KRAY356K5XV6BXLH33OZV6FCAVL6CWPYJZEGUG3VQ2RWG5Q, seg_pos={0 0}, upload=446548]: Uploading piece... Dec 26 20:22:11.620480492 (508h53m11.491563537s) [noise=true , address=45.91.116.10:4084 , piece_id=EMKO35AQZRDZQGIFU7FSWEPJQLLXSFE2T7R67XEQIEIGCBZNAN7A, seg_pos={0 0}, upload=446548]: Uploading piece... Dec 26 20:22:11.620859794 (508h53m11.491942829s) [noise=true , address=45.91.116.12:4228 , piece_id=GVXG76OZI563QRKTNVA5W65AYIIJR3H2O2RPSYM5MH4GLIEJQVBA, seg_pos={0 0}, upload=446548]: Uploading piece... Dec 26 20:22:13.239980740 (508h53m13.111063775s) [upload=446548]: Next returned nil segment Dec 26 20:22:13.239999025 (508h53m13.11108206s) [upload=446548]: Waiting for error group managing segments... Dec 26 20:22:13.625877748 (508h53m13.496960793s) [noise=true , address=145.79.227.11:4731 , piece_id=2INDLVZUH7AE6J6HFMHGO5O7CG3NHCUQXYUAP57NQJRH6ZUH3OSA, seg_pos={0 0}, upload=446548]: Done uploading piece. err: <nil> Dec 26 20:22:13.632743681 (508h53m13.503826727s) [noise=true , address=145.79.227.49:4339 , piece_id=XG77JJAAFJPPSVTRLZJDOEV3CNW66D2BOZLUCWETCEJWRRUNC6WQ, seg_pos={0 0}, upload=446548]: Done uploading piece. err: <nil> Dec 26 20:22:13.632806010 (508h53m13.503889045s) [noise=true , address=145.79.227.47:4265 , piece_id=FEWPBCQ7DPHIQKH6JBQQ3E3MVXP6QWTNH3PZXCSPES6ROWU5G7GA, seg_pos={0 0}, upload=446548]: Done uploading piece. err: <nil> Dec 26 20:22:13.635458711 (508h53m13.506541756s) [noise=true , address=145.79.227.8:4330 , piece_id=PVMOQCFLSNM6A3R7WETUFJRJT37LUNEX3HISXRYYHHWPFOQDOIMQ, seg_pos={0 0}, upload=446548]: Done uploading piece. err: <nil> Dec 26 20:22:13.637967679 (508h53m13.509050714s) [noise=true , address=145.79.227.37:4566 , piece_id=RNQ4AAIC7P4GWLYSECNRH24XBNOIKGSL3YG2C2HRD2NU7IVCTIJA, seg_pos={0 0}, upload=446548]: Done uploading piece. err: <nil> Dec 26 20:22:13.641444339 (508h53m13.512527374s) [noise=true , address=145.79.227.15:4876 , piece_id=7OKTYMB7F2H6C3LRCUI6PWGT3NHWECFAUHN34IQNTKPLZGMPC5EA, seg_pos={0 0}, upload=446548]: Done uploading piece. err: <nil> Dec 26 20:22:13.646718481 (508h53m13.517801527s) [noise=true , address=145.79.227.22:4509 , piece_id=QWK6MIMXHRGCPR3657GZ4OOL6WNEY3UZRRJ62EX5JLTF3XR2IMHA, seg_pos={0 0}, upload=446548]: Done uploading piece. err: <nil> Dec 26 20:22:13.909711180 (508h53m13.780794215s) [noise=true , address=45.91.116.10:4084 , piece_id=EMKO35AQZRDZQGIFU7FSWEPJQLLXSFE2T7R67XEQIEIGCBZNAN7A, seg_pos={0 0}, upload=446548]: Done uploading piece. err: <nil> Dec 26 20:22:13.934673262 (508h53m13.805756307s) [noise=true , address=77.90.31.21:4877 , piece_id=S7GR27SF2KEGO6263IHQCXRHDZ3PRTAORF5ZAPFNL3764EOH4WVQ, seg_pos={0 0}, upload=446548]: Done uploading piece. err: <nil> Dec 26 20:22:13.960508836 (508h53m13.831591881s) [noise=true , address=45.91.116.9:4261 , piece_id=KBNJNT22YQG6FQ73JODZAPQC43MPAXPSF2BPLNC6ZJD3Z2Y5R7KA, seg_pos={0 0}, upload=446548]: Done uploading piece. err: <nil> Dec 26 20:22:14.059238764 (508h53m13.930321799s) [noise=true , address=45.91.116.8:4046 , piece_id=HFCTQC4PSTY4AXYGU6QPEAAWMDFJ35HCCZ6JPJTVOVKSD6OLFCYQ, seg_pos={0 0}, upload=446548]: Done uploading piece. err: <nil> Dec 26 20:22:14.573429212 (508h53m14.444512247s) [noise=true , address=45.91.116.12:4228 , piece_id=GVXG76OZI563QRKTNVA5W65AYIIJR3H2O2RPSYM5MH4GLIEJQVBA, seg_pos={0 0}, upload=446548]: Done uploading piece. err: <nil> Dec 26 20:22:16.072176717 (508h53m15.943259762s) [noise=true , address=145.79.227.41:4086 , piece_id=4DGW3D2ZBCPJRWVEHBTOMQJ2NLKCSDEZC6PB2OORHWREB2LP47EA, seg_pos={0 0}, upload=446548]: Done uploading piece. err: <nil> Dec 26 20:22:16.072334799 (508h53m15.943417834s) [seg_pos={0 0}, upload=446548]: Segment reached optimal threshold of 13 pieces. Dec 26 20:22:16.072597188 (508h53m15.943680233s) [noise=true , address=145.79.227.40:4050 , piece_id=7LOB6KRAY356K5XV6BXLH33OZV6FCAVL6CWPYJZEGUG3VQ2RWG5Q, seg_pos={0 0}, upload=446548]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 26 20:22:16.072607067 (508h53m15.943690102s) [noise=true , address=145.79.227.40:4050 , piece_id=7LOB6KRAY356K5XV6BXLH33OZV6FCAVL6CWPYJZEGUG3VQ2RWG5Q, seg_pos={0 0}, upload=446548]: Long tail cancellation due to optimal threshold Dec 26 20:22:16.072692119 (508h53m15.943775154s) [noise=true , address=145.79.227.51:4410 , piece_id=M4QI5O7UMDDX3ZX6OI5UYMBX2QE5WEGT32YCAC2VU47DIHRS4FWQ, seg_pos={0 0}, upload=446548]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX): protocol: expected piece hash; context canceled; EOF Dec 26 20:22:16.072716596 (508h53m15.943799631s) [noise=true , address=145.79.227.51:4410 , piece_id=M4QI5O7UMDDX3ZX6OI5UYMBX2QE5WEGT32YCAC2VU47DIHRS4FWQ, seg_pos={0 0}, upload=446548]: Long tail cancellation due to optimal threshold Dec 26 20:22:16.072988764 (508h53m15.944071809s) [noise=true , address=145.79.227.24:4582 , piece_id=P7F7ZW2WSQZRYZDCRWY3HPM4LMLE5IJTOP34MOUZAKK2XCG4S5YQ, seg_pos={0 0}, upload=446548]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa): protocol: write tcp 188.166.225.170:36026->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:36026->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36026->145.79.227.24:4582: use of closed network connection Dec 26 20:22:16.072996689 (508h53m15.944079724s) [noise=true , address=145.79.227.24:4582 , piece_id=P7F7ZW2WSQZRYZDCRWY3HPM4LMLE5IJTOP34MOUZAKK2XCG4S5YQ, seg_pos={0 0}, upload=446548]: Long tail cancellation due to optimal threshold Dec 26 20:22:16.073011857 (508h53m15.944094892s) [seg_pos={0 0}, upload=446548]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 20:22:16.073241986 (508h53m15.944325021s) [upload=446548]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:22:16.073248188 (508h53m15.944331223s) [upload=446548]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:22:16.381229432 (508h53m16.252312467s) [upload=446548]: Done uploading segments... Dec 26 20:22:16.381257835 (508h53m16.25234088s) [upload=446548]: Upload finished. err: <nil> Duration: 30.003989115s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........X..........................................X......] duration: 27.944145144s trace: https://jaeger.dev.storj.io/trace/663e21624a5a3654 FAIL (2025-12-26 20:21:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, Piece ID: PWVMOEKW5A4ZAVSVJAAHFTYYTQZGFHDF642GWR6EABEGWP2CDRGQ) context deadline exceeded; piecestore: (Node ID: 12whTD1Kvu4T7ANNtkSvDB66uTjVgfPTK9TAzExgvwJYgZ3zSFG, Piece ID: ME2LDK2QJI7LLG67ODDIJDKB5CAT47YDGMJ3CTVK4F6UDHJ4TPOQ) context deadline exceeded; piecestore: (Node ID: 12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH, Piece ID: HJGRPPEJK75YTIKBKNVK7J4TDRKBCNL7HA4JNMOKD2ROGC7C4P3Q) context deadline exceeded ======LOGS====== ) [noise=true , address=145.79.227.7:4296 , piece_id=PVH6VN4CG4O2DC22G74US7IC5GDIGM2KYYYQQPFO5MG22JQCAU2A, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.248055070 (508h48m1.119138115s) [noise=true , address=145.79.227.40:4078 , piece_id=J6TGA3SFQOLGL2C2TS352TSLZBMI6MDXU5BDTTGQYMN4A5QG462A, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.248175840 (508h48m1.119258875s) [noise=true , address=77.90.31.12:4600 , piece_id=JESM5JF4GWUDLEW4YT5SKHJOXD54TWRQBNGGQCBMDSS54XR4AEKQ, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.248604496 (508h48m1.119687531s) [noise=true , address=45.91.116.6:4985 , piece_id=WV5SXZIBMXB4FTNZFE4PKBR5ZWZTUUPADPRMXAVE4Z3HKDGLBOSQ, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.248640474 (508h48m1.119723509s) [noise=true , address=145.79.227.18:4021 , piece_id=ER7GXC5OHAPNGU37QTKPROKB4YTCDQBUHVEIEUH2K5FQ4X5AGM2Q, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.249180202 (508h48m1.120263238s) [noise=true , address=77.90.31.18:4821 , piece_id=HJGRPPEJK75YTIKBKNVK7J4TDRKBCNL7HA4JNMOKD2ROGC7C4P3Q, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.249851130 (508h48m1.120934175s) [noise=true , address=64.49.13.169:4709 , piece_id=ZGWBPAYFOQWU2LSTILYMVRNKGSP53UXEQHRW4NJEXW5OLNVCCWRQ, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.250638910 (508h48m1.121721955s) [noise=true , address=145.79.227.38:4001 , piece_id=TWPCSZEWSFC2ZT7QNWHRS3RY42LLBPIEF3EIQQ65OXFZSHWAV2GA, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.251000769 (508h48m1.122083814s) [noise=true , address=145.79.227.30:4792 , piece_id=RLEUVLBAOMJSDAYHMQJGKYQUC6NYYGSGUVIU5Z53XGBINIQV43GQ, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.251314737 (508h48m1.122397772s) [noise=true , address=145.79.227.5:4252 , piece_id=FI5RNNRKSIYHR3JKCOW3THJQKHAPYLUDDHT3Y4TRYU3OSMCSWQQQ, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.251635527 (508h48m1.122718572s) [noise=true , address=77.90.31.11:4588 , piece_id=ZWDOVL5PGZ4YPY6NZVVSBSWXDDD3DAJ4RUK2BWMOGTHDGWJ2G4DA, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.251919499 (508h48m1.123002544s) [noise=true , address=77.90.31.10:4536 , piece_id=V43VRDKXTE2PHOBPSQUIJPKMKY3ROAEQAI2D4XIJHOLQ3UIMN4RA, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.252236542 (508h48m1.123319588s) [noise=true , address=77.90.31.21:4894 , piece_id=AGL6GVJX2YLGHSZX33SUURUU5S22CF3YRMJTP3QMHHP7OKX2MFSA, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.252528137 (508h48m1.123611182s) [noise=true , address=77.90.31.15:4702 , piece_id=KJEPR7NMX4UHQZHTMDDJKBRLLQ3O5BLJSLYBXGCOFGP3LNC3FXGA, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.252861402 (508h48m1.123944447s) [noise=true , address=64.49.13.27:4961 , piece_id=XCVH62CEVRYGR53U5VG4L4HJMBJSQ562L4QVKTXZSC7XGAEB4LDQ, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:02.570977410 (508h48m2.442060455s) [upload=446477]: Next returned nil segment Dec 26 20:17:02.571003019 (508h48m2.442086054s) [upload=446477]: Waiting for error group managing segments... Dec 26 20:17:02.949353997 (508h48m2.820437042s) [noise=true , address=64.49.13.27:4961 , piece_id=XCVH62CEVRYGR53U5VG4L4HJMBJSQ562L4QVKTXZSC7XGAEB4LDQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.136546190 (508h48m3.007629235s) [noise=true , address=145.79.227.5:4252 , piece_id=FI5RNNRKSIYHR3JKCOW3THJQKHAPYLUDDHT3Y4TRYU3OSMCSWQQQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.155760468 (508h48m3.026843513s) [noise=true , address=145.79.227.38:4001 , piece_id=TWPCSZEWSFC2ZT7QNWHRS3RY42LLBPIEF3EIQQ65OXFZSHWAV2GA, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.179239646 (508h48m3.050322691s) [noise=true , address=145.79.227.18:4021 , piece_id=ER7GXC5OHAPNGU37QTKPROKB4YTCDQBUHVEIEUH2K5FQ4X5AGM2Q, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.372384563 (508h48m3.243467608s) [noise=true , address=77.90.31.15:4702 , piece_id=KJEPR7NMX4UHQZHTMDDJKBRLLQ3O5BLJSLYBXGCOFGP3LNC3FXGA, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.447635803 (508h48m3.318718848s) [noise=true , address=77.90.31.18:4821 , piece_id=HJGRPPEJK75YTIKBKNVK7J4TDRKBCNL7HA4JNMOKD2ROGC7C4P3Q, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.452148166 (508h48m3.323231201s) [noise=true , address=77.90.31.10:4536 , piece_id=V43VRDKXTE2PHOBPSQUIJPKMKY3ROAEQAI2D4XIJHOLQ3UIMN4RA, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.644263002 (508h48m3.515346047s) [noise=true , address=77.90.31.14:4689 , piece_id=PWVMOEKW5A4ZAVSVJAAHFTYYTQZGFHDF642GWR6EABEGWP2CDRGQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.710620497 (508h48m3.581703542s) [noise=true , address=45.91.116.8:4074 , piece_id=ME2LDK2QJI7LLG67ODDIJDKB5CAT47YDGMJ3CTVK4F6UDHJ4TPOQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:04.066910008 (508h48m3.937993053s) [noise=true , address=77.90.31.12:4600 , piece_id=JESM5JF4GWUDLEW4YT5SKHJOXD54TWRQBNGGQCBMDSS54XR4AEKQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:04.082282349 (508h48m3.953365394s) [noise=true , address=145.79.227.44:4219 , piece_id=KZVV2NJBIU22C4URLBZMQVQN4IZ2ZMQ5YODRJMCS6XYXULHCZWLQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:04.089048724 (508h48m3.960131759s) [noise=true , address=145.79.227.30:4792 , piece_id=RLEUVLBAOMJSDAYHMQJGKYQUC6NYYGSGUVIU5Z53XGBINIQV43GQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:04.288772749 (508h48m4.159855794s) [noise=true , address=77.90.31.21:4894 , piece_id=AGL6GVJX2YLGHSZX33SUURUU5S22CF3YRMJTP3QMHHP7OKX2MFSA, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:04.344183239 (508h48m4.215266294s) [noise=true , address=77.90.31.11:4588 , piece_id=ZWDOVL5PGZ4YPY6NZVVSBSWXDDD3DAJ4RUK2BWMOGTHDGWJ2G4DA, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:04.344208728 (508h48m4.215291773s) [seg_pos={0 0}, upload=446477]: Segment reached optimal threshold of 14 pieces. Dec 26 20:17:04.344470828 (508h48m4.215553873s) [noise=true , address=64.49.13.169:4709 , piece_id=ZGWBPAYFOQWU2LSTILYMVRNKGSP53UXEQHRW4NJEXW5OLNVCCWRQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:47062->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:47062->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47062->64.49.13.169:4709: use of closed network connection Dec 26 20:17:04.344481918 (508h48m4.215564953s) [noise=true , address=64.49.13.169:4709 , piece_id=ZGWBPAYFOQWU2LSTILYMVRNKGSP53UXEQHRW4NJEXW5OLNVCCWRQ, seg_pos={0 0}, upload=446477]: Long tail cancellation due to optimal threshold Dec 26 20:17:04.344541813 (508h48m4.215624848s) [noise=true , address=145.79.227.40:4078 , piece_id=J6TGA3SFQOLGL2C2TS352TSLZBMI6MDXU5BDTTGQYMN4A5QG462A, seg_pos={0 0}, upload=446477]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tzgtenV6us5ssD7xWGpZmVUoWEfbFhBNewbK9irHtZPkyDtRd): protocol: expected piece hash; context canceled; EOF Dec 26 20:17:04.344559476 (508h48m4.215642511s) [noise=true , address=145.79.227.40:4078 , piece_id=J6TGA3SFQOLGL2C2TS352TSLZBMI6MDXU5BDTTGQYMN4A5QG462A, seg_pos={0 0}, upload=446477]: Long tail cancellation due to optimal threshold Dec 26 20:17:04.344755479 (508h48m4.215838524s) [noise=true , address=145.79.227.7:4296 , piece_id=PVH6VN4CG4O2DC22G74US7IC5GDIGM2KYYYQQPFO5MG22JQCAU2A, seg_pos={0 0}, upload=446477]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cKhDbA4ptwux2LYNPNAJ7Qh8jECsGWd1KWaK6T6pD38bJWdt1): protocol: write tcp 188.166.225.170:48104->145.79.227.7:4296: use of closed network connection; write tcp 188.166.225.170:48104->145.79.227.7:4296: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48104->145.79.227.7:4296: use of closed network connection Dec 26 20:17:04.344763044 (508h48m4.215846099s) [noise=true , address=145.79.227.7:4296 , piece_id=PVH6VN4CG4O2DC22G74US7IC5GDIGM2KYYYQQPFO5MG22JQCAU2A, seg_pos={0 0}, upload=446477]: Long tail cancellation due to optimal threshold Dec 26 20:17:04.344928509 (508h48m4.216011544s) [noise=true , address=45.91.116.6:4985 , piece_id=WV5SXZIBMXB4FTNZFE4PKBR5ZWZTUUPADPRMXAVE4Z3HKDGLBOSQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RNs9MM9jDtNC2gZC63SYsePZTd5mE7dqUm9A5Z5QWdPX8PVnp): protocol: expected piece hash; context canceled; EOF Dec 26 20:17:04.344938277 (508h48m4.216021312s) [noise=true , address=45.91.116.6:4985 , piece_id=WV5SXZIBMXB4FTNZFE4PKBR5ZWZTUUPADPRMXAVE4Z3HKDGLBOSQ, seg_pos={0 0}, upload=446477]: Long tail cancellation due to optimal threshold Dec 26 20:17:04.344966722 (508h48m4.216049767s) [seg_pos={0 0}, upload=446477]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:17:04.345054548 (508h48m4.216137593s) [upload=446477]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:17:04.345059839 (508h48m4.216142874s) [upload=446477]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:17:04.653613492 (508h48m4.524696537s) [upload=446477]: Done uploading segments... Dec 26 20:17:04.653639822 (508h48m4.524722857s) [upload=446477]: Upload finished. err: <nil> Duration: 27.944205961s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [..X.X.X.X.X..............XX..X..X...........XXX.X.X....X..XX] duration: 30.00386104s trace: https://jaeger.dev.storj.io/trace/563ea36aabc91dc2 FAIL (2025-12-26 20:21:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 20:21:09.793184611 (508h52m9.664267646s) [upload=446533]: Starting upload Dec 26 20:21:09.793240558 (508h52m9.664323593s) [upload=446533]: Done starting upload Dec 26 20:21:09.793438785 (508h52m9.66452182s) [upload=446533]: Uploading segments... Dec 26 20:21:09.793471086 (508h52m9.664554121s) [upload=446533]: Got next segment. Inline: false Dec 26 20:21:09.793497587 (508h52m9.664580632s) [upload=446533]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 20:21:09.793502055 (508h52m9.6645851s) [upload=446533]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 20:21:10.039091466 (508h52m9.910174501s) [seg_pos={0 0}, upload=446533]: Begin upload segment... Dec 26 20:21:10.039136792 (508h52m9.910219827s) [seg_pos={0 0}, upload=446533]: Done begin upload segment. Dec 26 20:21:10.039161610 (508h52m9.910244645s) [noise=true , address=145.79.227.38:4977 , piece_id=ABAW7UMGR7GYR7L3D4GK2MHUAHEIFUNHAC2OQPWFDJE7RCXJCUNA, seg_pos={0 0}, upload=446533]: Uploading piece... Dec 26 20:21:10.039265608 (508h52m9.910348643s) [noise=true , address=45.91.116.7:4010 , piece_id=BZIKLTTMDRJOZO4GDDO2DUI5MVXPZL5JBAPMC7VO567XJ6J4RC6A, seg_pos={0 0}, upload=446533]: Uploading piece... Dec 26 20:21:10.039711176 (508h52m9.910794221s) [noise=true , address=145.79.227.25:4618 , piece_id=PZAGHFR6IXOKZG2YIAQD6X5TNYCY6ZHGMCBI45NTI4L7CENXFQCQ, seg_pos={0 0}, upload=446533]: Uploading piece... Dec 26 20:21:10.040000337 (508h52m9.911083382s) [noise=true , address=145.79.227.21:4475 , piece_id=QZWXKCJEGZLOAMRK3J5GMITZXONODY3CFUTN2C6DW33BT2IOBDSA, seg_pos={0 0}, upload=446533]: Uploading piece... Dec 26 20:21:10.040280119 (508h52m9.911363164s) [noise=true , address=45.91.116.12:4228 , piece_id=ULGKA4JAUZDZUNDQUMINS7AMDAKLSOB74OOKASTJF3QN32DNYBJQ, seg_pos={0 0}, upload=446533]: Uploading piece... Dec 26 20:21:10.040730046 (508h52m9.911813081s) [noise=true , address=145.79.227.49:4339 , piece_id=7STQQIHH63Q4SBE7AR3E2HRHQA2YD2XTZCQ3G6PGAWMZZUR6RP2Q, seg_pos={0 0}, upload=446533]: Uploading piece... Dec 26 20:21:10.041062699 (508h52m9.912145734s) [noise=true , address=45.91.116.14:4190 , piece_id=AXKNTAXBTFEG3L3QVJCUFA3FAV7EJBZKNUXO6WSACQWFUH6TMFNA, seg_pos={0 0}, upload=446533]: Uploading piece... Dec 26 20:21:10.041071296 (508h52m9.912154331s) [noise=true , address=45.91.116.8:4046 , piece_id=L3RSHRXNGZXXDXIH64G2RPYZGML2Z7CRBANIS3J6KNJGFGUIH5IQ, seg_pos={0 0}, upload=446533]: Uploading piece... Dec 26 20:21:10.041365426 (508h52m9.912448471s) [noise=true , address=77.90.31.11:4556 , piece_id=T6Y6INIDR45LUDOG5GGR6Y3UYVGFHWEXPIOGISM6T7V7U6IA2A4Q, seg_pos={0 0}, upload=446533]: Uploading piece... Dec 26 20:21:10.041534929 (508h52m9.912617964s) [noise=true , address=145.79.227.34:4906 , piece_id=ZRTNMZODSZEUMEFMATWRDPX6KBBFGPBRZ5HJTQLOWODYEHG6JRNA, seg_pos={0 0}, upload=446533]: Uploading piece... Dec 26 20:21:10.041733357 (508h52m9.912816402s) [noise=true , address=45.91.116.13:4155 , piece_id=4D723FN73VGPGZACVF6RJZPRXAHBRBHK22JGC74H6R5QMDVFDJ4Q, seg_pos={0 0}, upload=446533]: Uploading piece... Dec 26 20:21:10.041913449 (508h52m9.912996485s) [noise=true , address=45.91.116.10:4083 , piece_id=LAMYNN6BA3INVBAIJDMFOK7M3Q4BUPFTB7YMXC4IYVIOVQRRCENQ, seg_pos={0 0}, upload=446533]: Uploading piece... Dec 26 20:21:10.042242085 (508h52m9.91332512s) [noise=true , address=145.79.227.41:4086 , piece_id=MJ2G2XXAS7IYLPC2FQNKEEUOAVMV4K3QMXXBBZPTOTZZSPBC5M4Q, seg_pos={0 0}, upload=446533]: Uploading piece... Dec 26 20:21:10.042742729 (508h52m9.913825774s) [noise=true , address=145.79.227.47:4266 , piece_id=EC422RNH7M37QS6BTUHLNIUEBR6MUZFND6XFHAORUD4ZPXSVZCZQ, seg_pos={0 0}, upload=446533]: Uploading piece... Dec 26 20:21:10.043085090 (508h52m9.914168125s) [noise=true , address=145.79.227.45:4229 , piece_id=5OPV6DNGESB3GV3JQEFXM4SVQWS436RAJ34LGCJBVRKRGBDKR2KQ, seg_pos={0 0}, upload=446533]: Uploading piece... Dec 26 20:21:10.043420920 (508h52m9.914503955s) [noise=true , address=45.91.116.5:4299 , piece_id=BQQS2K3J37EFMESRQ2PHLUN7S2WD5QSZZ6BJG37MSVOZFBB72ATQ, seg_pos={0 0}, upload=446533]: Uploading piece... Dec 26 20:21:11.644752574 (508h52m11.515835609s) [upload=446533]: Next returned nil segment Dec 26 20:21:11.644789854 (508h52m11.51587289s) [upload=446533]: Waiting for error group managing segments... Dec 26 20:21:12.030792392 (508h52m11.901875427s) [noise=true , address=145.79.227.49:4339 , piece_id=7STQQIHH63Q4SBE7AR3E2HRHQA2YD2XTZCQ3G6PGAWMZZUR6RP2Q, seg_pos={0 0}, upload=446533]: Done uploading piece. err: <nil> Dec 26 20:21:12.031471616 (508h52m11.902554661s) [noise=true , address=145.79.227.25:4618 , piece_id=PZAGHFR6IXOKZG2YIAQD6X5TNYCY6ZHGMCBI45NTI4L7CENXFQCQ, seg_pos={0 0}, upload=446533]: Done uploading piece. err: <nil> Dec 26 20:21:12.047736076 (508h52m11.918819121s) [noise=true , address=145.79.227.38:4977 , piece_id=ABAW7UMGR7GYR7L3D4GK2MHUAHEIFUNHAC2OQPWFDJE7RCXJCUNA, seg_pos={0 0}, upload=446533]: Done uploading piece. err: <nil> Dec 26 20:21:12.053357690 (508h52m11.924440735s) [noise=true , address=145.79.227.45:4229 , piece_id=5OPV6DNGESB3GV3JQEFXM4SVQWS436RAJ34LGCJBVRKRGBDKR2KQ, seg_pos={0 0}, upload=446533]: Done uploading piece. err: <nil> Dec 26 20:21:12.055719678 (508h52m11.926802723s) [noise=true , address=145.79.227.21:4475 , piece_id=QZWXKCJEGZLOAMRK3J5GMITZXONODY3CFUTN2C6DW33BT2IOBDSA, seg_pos={0 0}, upload=446533]: Done uploading piece. err: <nil> Dec 26 20:21:12.066014061 (508h52m11.937097106s) [noise=true , address=145.79.227.41:4086 , piece_id=MJ2G2XXAS7IYLPC2FQNKEEUOAVMV4K3QMXXBBZPTOTZZSPBC5M4Q, seg_pos={0 0}, upload=446533]: Done uploading piece. err: <nil> Dec 26 20:21:12.268818712 (508h52m12.139901757s) [noise=true , address=45.91.116.8:4046 , piece_id=L3RSHRXNGZXXDXIH64G2RPYZGML2Z7CRBANIS3J6KNJGFGUIH5IQ, seg_pos={0 0}, upload=446533]: Done uploading piece. err: <nil> Dec 26 20:21:12.305180470 (508h52m12.176263515s) [noise=true , address=45.91.116.7:4010 , piece_id=BZIKLTTMDRJOZO4GDDO2DUI5MVXPZL5JBAPMC7VO567XJ6J4RC6A, seg_pos={0 0}, upload=446533]: Done uploading piece. err: <nil> Dec 26 20:21:12.378257150 (508h52m12.249340195s) [noise=true , address=45.91.116.5:4299 , piece_id=BQQS2K3J37EFMESRQ2PHLUN7S2WD5QSZZ6BJG37MSVOZFBB72ATQ, seg_pos={0 0}, upload=446533]: Done uploading piece. err: <nil> Dec 26 20:21:12.380048210 (508h52m12.251131245s) [noise=true , address=45.91.116.12:4228 , piece_id=ULGKA4JAUZDZUNDQUMINS7AMDAKLSOB74OOKASTJF3QN32DNYBJQ, seg_pos={0 0}, upload=446533]: Done uploading piece. err: <nil> Dec 26 20:21:12.430550173 (508h52m12.301633208s) [noise=true , address=45.91.116.13:4155 , piece_id=4D723FN73VGPGZACVF6RJZPRXAHBRBHK22JGC74H6R5QMDVFDJ4Q, seg_pos={0 0}, upload=446533]: Done uploading piece. err: <nil> Dec 26 20:21:12.521696120 (508h52m12.392779165s) [noise=true , address=77.90.31.11:4556 , piece_id=T6Y6INIDR45LUDOG5GGR6Y3UYVGFHWEXPIOGISM6T7V7U6IA2A4Q, seg_pos={0 0}, upload=446533]: Done uploading piece. err: <nil> Dec 26 20:21:13.172576606 (508h52m13.043659641s) [noise=true , address=45.91.116.10:4083 , piece_id=LAMYNN6BA3INVBAIJDMFOK7M3Q4BUPFTB7YMXC4IYVIOVQRRCENQ, seg_pos={0 0}, upload=446533]: Done uploading piece. err: <nil> Dec 26 20:21:13.172657110 (508h52m13.043740155s) [seg_pos={0 0}, upload=446533]: Segment reached optimal threshold of 13 pieces. Dec 26 20:21:13.172929719 (508h52m13.044012764s) [noise=true , address=145.79.227.34:4906 , piece_id=ZRTNMZODSZEUMEFMATWRDPX6KBBFGPBRZ5HJTQLOWODYEHG6JRNA, seg_pos={0 0}, upload=446533]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 26 20:21:13.172940439 (508h52m13.044023474s) [noise=true , address=145.79.227.34:4906 , piece_id=ZRTNMZODSZEUMEFMATWRDPX6KBBFGPBRZ5HJTQLOWODYEHG6JRNA, seg_pos={0 0}, upload=446533]: Long tail cancellation due to optimal threshold Dec 26 20:21:13.172951510 (508h52m13.044034545s) [noise=true , address=45.91.116.14:4190 , piece_id=AXKNTAXBTFEG3L3QVJCUFA3FAV7EJBZKNUXO6WSACQWFUH6TMFNA, seg_pos={0 0}, upload=446533]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Dec 26 20:21:13.172967541 (508h52m13.044050586s) [noise=true , address=45.91.116.14:4190 , piece_id=AXKNTAXBTFEG3L3QVJCUFA3FAV7EJBZKNUXO6WSACQWFUH6TMFNA, seg_pos={0 0}, upload=446533]: Long tail cancellation due to optimal threshold Dec 26 20:21:13.173076028 (508h52m13.044159073s) [noise=true , address=145.79.227.47:4266 , piece_id=EC422RNH7M37QS6BTUHLNIUEBR6MUZFND6XFHAORUD4ZPXSVZCZQ, seg_pos={0 0}, upload=446533]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uzGTks5tffchQBGQK23fnWZXMfrsTHh4Zqs9UGwnWUmnQghEW): protocol: expected piece hash; context canceled; EOF Dec 26 20:21:13.173085095 (508h52m13.04416814s) [noise=true , address=145.79.227.47:4266 , piece_id=EC422RNH7M37QS6BTUHLNIUEBR6MUZFND6XFHAORUD4ZPXSVZCZQ, seg_pos={0 0}, upload=446533]: Long tail cancellation due to optimal threshold Dec 26 20:21:13.173107027 (508h52m13.044190062s) [seg_pos={0 0}, upload=446533]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 20:21:13.173191758 (508h52m13.044274803s) [upload=446533]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:21:13.173198560 (508h52m13.044281605s) [upload=446533]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:21:13.476324870 (508h52m13.347407915s) [upload=446533]: Done uploading segments... Dec 26 20:21:13.476343555 (508h52m13.34742659s) [upload=446533]: Upload finished. err: <nil> Duration: 30.003887401s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........X..........................................X.....] duration: 34.414080175s trace: https://jaeger.dev.storj.io/trace/663e21624a5a3654 FAIL (2025-12-26 20:21:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, Piece ID: PWVMOEKW5A4ZAVSVJAAHFTYYTQZGFHDF642GWR6EABEGWP2CDRGQ) context deadline exceeded; piecestore: (Node ID: 12whTD1Kvu4T7ANNtkSvDB66uTjVgfPTK9TAzExgvwJYgZ3zSFG, Piece ID: ME2LDK2QJI7LLG67ODDIJDKB5CAT47YDGMJ3CTVK4F6UDHJ4TPOQ) context deadline exceeded; piecestore: (Node ID: 12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH, Piece ID: HJGRPPEJK75YTIKBKNVK7J4TDRKBCNL7HA4JNMOKD2ROGC7C4P3Q) context deadline exceeded ======LOGS====== ) [noise=true , address=145.79.227.7:4296 , piece_id=PVH6VN4CG4O2DC22G74US7IC5GDIGM2KYYYQQPFO5MG22JQCAU2A, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.248055070 (508h48m1.119138115s) [noise=true , address=145.79.227.40:4078 , piece_id=J6TGA3SFQOLGL2C2TS352TSLZBMI6MDXU5BDTTGQYMN4A5QG462A, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.248175840 (508h48m1.119258875s) [noise=true , address=77.90.31.12:4600 , piece_id=JESM5JF4GWUDLEW4YT5SKHJOXD54TWRQBNGGQCBMDSS54XR4AEKQ, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.248604496 (508h48m1.119687531s) [noise=true , address=45.91.116.6:4985 , piece_id=WV5SXZIBMXB4FTNZFE4PKBR5ZWZTUUPADPRMXAVE4Z3HKDGLBOSQ, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.248640474 (508h48m1.119723509s) [noise=true , address=145.79.227.18:4021 , piece_id=ER7GXC5OHAPNGU37QTKPROKB4YTCDQBUHVEIEUH2K5FQ4X5AGM2Q, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.249180202 (508h48m1.120263238s) [noise=true , address=77.90.31.18:4821 , piece_id=HJGRPPEJK75YTIKBKNVK7J4TDRKBCNL7HA4JNMOKD2ROGC7C4P3Q, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.249851130 (508h48m1.120934175s) [noise=true , address=64.49.13.169:4709 , piece_id=ZGWBPAYFOQWU2LSTILYMVRNKGSP53UXEQHRW4NJEXW5OLNVCCWRQ, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.250638910 (508h48m1.121721955s) [noise=true , address=145.79.227.38:4001 , piece_id=TWPCSZEWSFC2ZT7QNWHRS3RY42LLBPIEF3EIQQ65OXFZSHWAV2GA, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.251000769 (508h48m1.122083814s) [noise=true , address=145.79.227.30:4792 , piece_id=RLEUVLBAOMJSDAYHMQJGKYQUC6NYYGSGUVIU5Z53XGBINIQV43GQ, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.251314737 (508h48m1.122397772s) [noise=true , address=145.79.227.5:4252 , piece_id=FI5RNNRKSIYHR3JKCOW3THJQKHAPYLUDDHT3Y4TRYU3OSMCSWQQQ, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.251635527 (508h48m1.122718572s) [noise=true , address=77.90.31.11:4588 , piece_id=ZWDOVL5PGZ4YPY6NZVVSBSWXDDD3DAJ4RUK2BWMOGTHDGWJ2G4DA, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.251919499 (508h48m1.123002544s) [noise=true , address=77.90.31.10:4536 , piece_id=V43VRDKXTE2PHOBPSQUIJPKMKY3ROAEQAI2D4XIJHOLQ3UIMN4RA, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.252236542 (508h48m1.123319588s) [noise=true , address=77.90.31.21:4894 , piece_id=AGL6GVJX2YLGHSZX33SUURUU5S22CF3YRMJTP3QMHHP7OKX2MFSA, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.252528137 (508h48m1.123611182s) [noise=true , address=77.90.31.15:4702 , piece_id=KJEPR7NMX4UHQZHTMDDJKBRLLQ3O5BLJSLYBXGCOFGP3LNC3FXGA, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.252861402 (508h48m1.123944447s) [noise=true , address=64.49.13.27:4961 , piece_id=XCVH62CEVRYGR53U5VG4L4HJMBJSQ562L4QVKTXZSC7XGAEB4LDQ, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:02.570977410 (508h48m2.442060455s) [upload=446477]: Next returned nil segment Dec 26 20:17:02.571003019 (508h48m2.442086054s) [upload=446477]: Waiting for error group managing segments... Dec 26 20:17:02.949353997 (508h48m2.820437042s) [noise=true , address=64.49.13.27:4961 , piece_id=XCVH62CEVRYGR53U5VG4L4HJMBJSQ562L4QVKTXZSC7XGAEB4LDQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.136546190 (508h48m3.007629235s) [noise=true , address=145.79.227.5:4252 , piece_id=FI5RNNRKSIYHR3JKCOW3THJQKHAPYLUDDHT3Y4TRYU3OSMCSWQQQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.155760468 (508h48m3.026843513s) [noise=true , address=145.79.227.38:4001 , piece_id=TWPCSZEWSFC2ZT7QNWHRS3RY42LLBPIEF3EIQQ65OXFZSHWAV2GA, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.179239646 (508h48m3.050322691s) [noise=true , address=145.79.227.18:4021 , piece_id=ER7GXC5OHAPNGU37QTKPROKB4YTCDQBUHVEIEUH2K5FQ4X5AGM2Q, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.372384563 (508h48m3.243467608s) [noise=true , address=77.90.31.15:4702 , piece_id=KJEPR7NMX4UHQZHTMDDJKBRLLQ3O5BLJSLYBXGCOFGP3LNC3FXGA, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.447635803 (508h48m3.318718848s) [noise=true , address=77.90.31.18:4821 , piece_id=HJGRPPEJK75YTIKBKNVK7J4TDRKBCNL7HA4JNMOKD2ROGC7C4P3Q, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.452148166 (508h48m3.323231201s) [noise=true , address=77.90.31.10:4536 , piece_id=V43VRDKXTE2PHOBPSQUIJPKMKY3ROAEQAI2D4XIJHOLQ3UIMN4RA, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.644263002 (508h48m3.515346047s) [noise=true , address=77.90.31.14:4689 , piece_id=PWVMOEKW5A4ZAVSVJAAHFTYYTQZGFHDF642GWR6EABEGWP2CDRGQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.710620497 (508h48m3.581703542s) [noise=true , address=45.91.116.8:4074 , piece_id=ME2LDK2QJI7LLG67ODDIJDKB5CAT47YDGMJ3CTVK4F6UDHJ4TPOQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:04.066910008 (508h48m3.937993053s) [noise=true , address=77.90.31.12:4600 , piece_id=JESM5JF4GWUDLEW4YT5SKHJOXD54TWRQBNGGQCBMDSS54XR4AEKQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:04.082282349 (508h48m3.953365394s) [noise=true , address=145.79.227.44:4219 , piece_id=KZVV2NJBIU22C4URLBZMQVQN4IZ2ZMQ5YODRJMCS6XYXULHCZWLQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:04.089048724 (508h48m3.960131759s) [noise=true , address=145.79.227.30:4792 , piece_id=RLEUVLBAOMJSDAYHMQJGKYQUC6NYYGSGUVIU5Z53XGBINIQV43GQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:04.288772749 (508h48m4.159855794s) [noise=true , address=77.90.31.21:4894 , piece_id=AGL6GVJX2YLGHSZX33SUURUU5S22CF3YRMJTP3QMHHP7OKX2MFSA, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:04.344183239 (508h48m4.215266294s) [noise=true , address=77.90.31.11:4588 , piece_id=ZWDOVL5PGZ4YPY6NZVVSBSWXDDD3DAJ4RUK2BWMOGTHDGWJ2G4DA, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:04.344208728 (508h48m4.215291773s) [seg_pos={0 0}, upload=446477]: Segment reached optimal threshold of 14 pieces. Dec 26 20:17:04.344470828 (508h48m4.215553873s) [noise=true , address=64.49.13.169:4709 , piece_id=ZGWBPAYFOQWU2LSTILYMVRNKGSP53UXEQHRW4NJEXW5OLNVCCWRQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:47062->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:47062->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47062->64.49.13.169:4709: use of closed network connection Dec 26 20:17:04.344481918 (508h48m4.215564953s) [noise=true , address=64.49.13.169:4709 , piece_id=ZGWBPAYFOQWU2LSTILYMVRNKGSP53UXEQHRW4NJEXW5OLNVCCWRQ, seg_pos={0 0}, upload=446477]: Long tail cancellation due to optimal threshold Dec 26 20:17:04.344541813 (508h48m4.215624848s) [noise=true , address=145.79.227.40:4078 , piece_id=J6TGA3SFQOLGL2C2TS352TSLZBMI6MDXU5BDTTGQYMN4A5QG462A, seg_pos={0 0}, upload=446477]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tzgtenV6us5ssD7xWGpZmVUoWEfbFhBNewbK9irHtZPkyDtRd): protocol: expected piece hash; context canceled; EOF Dec 26 20:17:04.344559476 (508h48m4.215642511s) [noise=true , address=145.79.227.40:4078 , piece_id=J6TGA3SFQOLGL2C2TS352TSLZBMI6MDXU5BDTTGQYMN4A5QG462A, seg_pos={0 0}, upload=446477]: Long tail cancellation due to optimal threshold Dec 26 20:17:04.344755479 (508h48m4.215838524s) [noise=true , address=145.79.227.7:4296 , piece_id=PVH6VN4CG4O2DC22G74US7IC5GDIGM2KYYYQQPFO5MG22JQCAU2A, seg_pos={0 0}, upload=446477]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cKhDbA4ptwux2LYNPNAJ7Qh8jECsGWd1KWaK6T6pD38bJWdt1): protocol: write tcp 188.166.225.170:48104->145.79.227.7:4296: use of closed network connection; write tcp 188.166.225.170:48104->145.79.227.7:4296: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48104->145.79.227.7:4296: use of closed network connection Dec 26 20:17:04.344763044 (508h48m4.215846099s) [noise=true , address=145.79.227.7:4296 , piece_id=PVH6VN4CG4O2DC22G74US7IC5GDIGM2KYYYQQPFO5MG22JQCAU2A, seg_pos={0 0}, upload=446477]: Long tail cancellation due to optimal threshold Dec 26 20:17:04.344928509 (508h48m4.216011544s) [noise=true , address=45.91.116.6:4985 , piece_id=WV5SXZIBMXB4FTNZFE4PKBR5ZWZTUUPADPRMXAVE4Z3HKDGLBOSQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RNs9MM9jDtNC2gZC63SYsePZTd5mE7dqUm9A5Z5QWdPX8PVnp): protocol: expected piece hash; context canceled; EOF Dec 26 20:17:04.344938277 (508h48m4.216021312s) [noise=true , address=45.91.116.6:4985 , piece_id=WV5SXZIBMXB4FTNZFE4PKBR5ZWZTUUPADPRMXAVE4Z3HKDGLBOSQ, seg_pos={0 0}, upload=446477]: Long tail cancellation due to optimal threshold Dec 26 20:17:04.344966722 (508h48m4.216049767s) [seg_pos={0 0}, upload=446477]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:17:04.345054548 (508h48m4.216137593s) [upload=446477]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:17:04.345059839 (508h48m4.216142874s) [upload=446477]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:17:04.653613492 (508h48m4.524696537s) [upload=446477]: Done uploading segments... Dec 26 20:17:04.653639822 (508h48m4.524722857s) [upload=446477]: Upload finished. err: <nil> Duration: 34.414096716s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [...X.X.X.X.X..............XX..X..X...........XXX.X.X....X..X] duration: 30.002024885s trace: https://jaeger.dev.storj.io/trace/2ba7810b1f205501 FAIL (2025-12-26 20:20:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: STXFINMWEFSVEKQHSWJ3WRMW5T4A7B3AAH6V5RILLTFNC5BVPYFA) context deadline exceeded; piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: 5DM2CVTIGHJOKLK3KAMM2Z6VWQC27MA75ZNWRD5IGQX3WA4MZJVA) context deadline exceeded ======LOGS====== ts... Dec 26 20:20:26.057370857 (508h51m25.928453892s) [upload=446522]: Got next segment. Inline: false Dec 26 20:20:26.057481599 (508h51m25.928564634s) [upload=446522]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 20:20:26.057488262 (508h51m25.928571297s) [upload=446522]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 20:20:26.307819292 (508h51m26.178902337s) [seg_pos={0 0}, upload=446522]: Begin upload segment... Dec 26 20:20:26.307880489 (508h51m26.178963524s) [seg_pos={0 0}, upload=446522]: Done begin upload segment. Dec 26 20:20:26.307899675 (508h51m26.17898271s) [noise=true , address=145.79.227.5:4223 , piece_id=62FF7YPAM2NBY5DJUEVMN6YYHKHGKQKPPVUJKKOVH7KFFT4RUL7A, seg_pos={0 0}, upload=446522]: Uploading piece... Dec 26 20:20:26.307917439 (508h51m26.179000474s) [noise=true , address=77.90.31.16:4735 , piece_id=SIFXCIJA76ZUUXTPUC6EASKPTUXEB43D2ZHFHK7GZBCJJPLZ4KIQ, seg_pos={0 0}, upload=446522]: Uploading piece... Dec 26 20:20:26.308047696 (508h51m26.179130731s) [noise=true , address=45.91.116.9:4264 , piece_id=6ORHC753BLKQURTCVUMJPPNNBS7R2VHHJLWTMR6USTFSELRE6J2Q, seg_pos={0 0}, upload=446522]: Uploading piece... Dec 26 20:20:26.308533511 (508h51m26.179616546s) [noise=true , address=145.79.227.47:4267 , piece_id=ZHBEUWVDVY5QC6EGY7YJD5TLNVAXEQL33RNIQOHQUGZFQTWM6FKQ, seg_pos={0 0}, upload=446522]: Uploading piece... Dec 26 20:20:26.308974101 (508h51m26.180057156s) [noise=true , address=145.79.227.36:4519 , piece_id=OY5NQVUB5PJUG6QMLEUHJDG2VOI2X4VPAATUZVBSLZPC3DEGTB3Q, seg_pos={0 0}, upload=446522]: Uploading piece... Dec 26 20:20:26.309423026 (508h51m26.180506071s) [noise=true , address=77.90.31.10:4519 , piece_id=STXFINMWEFSVEKQHSWJ3WRMW5T4A7B3AAH6V5RILLTFNC5BVPYFA, seg_pos={0 0}, upload=446522]: Uploading piece... Dec 26 20:20:26.309882009 (508h51m26.180965054s) [noise=true , address=145.79.227.42:4121 , piece_id=W2MCHGYX22ZTZIAAU5GNSKJFLDEZBV25WHF7MILPCLIIXPN6CXAQ, seg_pos={0 0}, upload=446522]: Uploading piece... Dec 26 20:20:26.310338839 (508h51m26.181421874s) [noise=true , address=145.79.227.22:4509 , piece_id=6ZCFQHNB2KHI4BOY74UXJAHTEYZVLXP7XHKYMAIPOGKTJNXKXXIQ, seg_pos={0 0}, upload=446522]: Uploading piece... Dec 26 20:20:26.310358777 (508h51m26.181441813s) [noise=true , address=45.91.116.13:4156 , piece_id=5DM2CVTIGHJOKLK3KAMM2Z6VWQC27MA75ZNWRD5IGQX3WA4MZJVA, seg_pos={0 0}, upload=446522]: Uploading piece... Dec 26 20:20:26.310921298 (508h51m26.182004334s) [noise=true , address=145.79.227.11:4729 , piece_id=OEERNCJ4XU7RPNW7JDGRJK33FDW5SRUAZHODMDDCI7BIT2YDVJ3Q, seg_pos={0 0}, upload=446522]: Uploading piece... Dec 26 20:20:26.311391734 (508h51m26.182474779s) [noise=true , address=145.79.227.8:4330 , piece_id=C4Z64TGZFS72OHHCKLWUU3QEL7FGAQEUIX4IZVIJ4DUA27IGMCFA, seg_pos={0 0}, upload=446522]: Uploading piece... Dec 26 20:20:26.311647852 (508h51m26.182730897s) [noise=true , address=145.79.227.50:4374 , piece_id=NDOPB6CVEGJCT3EKXPH46KHNFZAN2U7L2FD4347HAUO6TK3Z6QPA, seg_pos={0 0}, upload=446522]: Uploading piece... Dec 26 20:20:26.311837142 (508h51m26.182920177s) [noise=true , address=45.91.116.10:4083 , piece_id=WIGTGILA2O5TYZFKGKTR4JALP5546XGILBLR32ANMFOJJQ3FJINQ, seg_pos={0 0}, upload=446522]: Uploading piece... Dec 26 20:20:26.312303561 (508h51m26.183386606s) [noise=true , address=45.91.116.5:4299 , piece_id=U2GDM6QG4I2TVR4HTZUZ7DIB4PDE33XX6LZ46YA66CGH3HYTWCFQ, seg_pos={0 0}, upload=446522]: Uploading piece... Dec 26 20:20:26.312504633 (508h51m26.183587668s) [noise=true , address=45.91.116.12:4227 , piece_id=TPPZ5KQWDTJ4QOGPXV2QYG4DZNTIRFBUQVPCNIUM6OKIA56N7VTQ, seg_pos={0 0}, upload=446522]: Uploading piece... Dec 26 20:20:26.312852886 (508h51m26.183935931s) [noise=true , address=145.79.227.45:4230 , piece_id=335AEV37T76WHFEKCD3GUV47C3JNHIT5YKHAFZ2UXEAIDJ5HCNKQ, seg_pos={0 0}, upload=446522]: Uploading piece... Dec 26 20:20:27.957328020 (508h51m27.828411065s) [upload=446522]: Next returned nil segment Dec 26 20:20:27.957344342 (508h51m27.828427387s) [upload=446522]: Waiting for error group managing segments... Dec 26 20:20:28.307085323 (508h51m28.178168358s) [noise=true , address=145.79.227.22:4509 , piece_id=6ZCFQHNB2KHI4BOY74UXJAHTEYZVLXP7XHKYMAIPOGKTJNXKXXIQ, seg_pos={0 0}, upload=446522]: Done uploading piece. err: <nil> Dec 26 20:20:28.309453692 (508h51m28.180536737s) [noise=true , address=145.79.227.50:4374 , piece_id=NDOPB6CVEGJCT3EKXPH46KHNFZAN2U7L2FD4347HAUO6TK3Z6QPA, seg_pos={0 0}, upload=446522]: Done uploading piece. err: <nil> Dec 26 20:20:28.310125201 (508h51m28.181208246s) [noise=true , address=145.79.227.11:4729 , piece_id=OEERNCJ4XU7RPNW7JDGRJK33FDW5SRUAZHODMDDCI7BIT2YDVJ3Q, seg_pos={0 0}, upload=446522]: Done uploading piece. err: <nil> Dec 26 20:20:28.313544272 (508h51m28.184627317s) [noise=true , address=145.79.227.45:4230 , piece_id=335AEV37T76WHFEKCD3GUV47C3JNHIT5YKHAFZ2UXEAIDJ5HCNKQ, seg_pos={0 0}, upload=446522]: Done uploading piece. err: <nil> Dec 26 20:20:28.318883778 (508h51m28.189966814s) [noise=true , address=145.79.227.42:4121 , piece_id=W2MCHGYX22ZTZIAAU5GNSKJFLDEZBV25WHF7MILPCLIIXPN6CXAQ, seg_pos={0 0}, upload=446522]: Done uploading piece. err: <nil> Dec 26 20:20:28.320931899 (508h51m28.192014934s) [noise=true , address=145.79.227.5:4223 , piece_id=62FF7YPAM2NBY5DJUEVMN6YYHKHGKQKPPVUJKKOVH7KFFT4RUL7A, seg_pos={0 0}, upload=446522]: Done uploading piece. err: <nil> Dec 26 20:20:28.327524321 (508h51m28.198607366s) [noise=true , address=145.79.227.36:4519 , piece_id=OY5NQVUB5PJUG6QMLEUHJDG2VOI2X4VPAATUZVBSLZPC3DEGTB3Q, seg_pos={0 0}, upload=446522]: Done uploading piece. err: <nil> Dec 26 20:20:28.335890492 (508h51m28.206973537s) [noise=true , address=145.79.227.8:4330 , piece_id=C4Z64TGZFS72OHHCKLWUU3QEL7FGAQEUIX4IZVIJ4DUA27IGMCFA, seg_pos={0 0}, upload=446522]: Done uploading piece. err: <nil> Dec 26 20:20:28.517125051 (508h51m28.388208086s) [noise=true , address=145.79.227.47:4267 , piece_id=ZHBEUWVDVY5QC6EGY7YJD5TLNVAXEQL33RNIQOHQUGZFQTWM6FKQ, seg_pos={0 0}, upload=446522]: Done uploading piece. err: <nil> Dec 26 20:20:28.582738549 (508h51m28.453821584s) [noise=true , address=77.90.31.16:4735 , piece_id=SIFXCIJA76ZUUXTPUC6EASKPTUXEB43D2ZHFHK7GZBCJJPLZ4KIQ, seg_pos={0 0}, upload=446522]: Done uploading piece. err: <nil> Dec 26 20:20:28.585943792 (508h51m28.457026837s) [noise=true , address=77.90.31.10:4519 , piece_id=STXFINMWEFSVEKQHSWJ3WRMW5T4A7B3AAH6V5RILLTFNC5BVPYFA, seg_pos={0 0}, upload=446522]: Done uploading piece. err: <nil> Dec 26 20:20:28.657877715 (508h51m28.52896076s) [noise=true , address=45.91.116.13:4156 , piece_id=5DM2CVTIGHJOKLK3KAMM2Z6VWQC27MA75ZNWRD5IGQX3WA4MZJVA, seg_pos={0 0}, upload=446522]: Done uploading piece. err: <nil> Dec 26 20:20:28.715590730 (508h51m28.586673765s) [noise=true , address=45.91.116.10:4083 , piece_id=WIGTGILA2O5TYZFKGKTR4JALP5546XGILBLR32ANMFOJJQ3FJINQ, seg_pos={0 0}, upload=446522]: Done uploading piece. err: <nil> Dec 26 20:20:28.715613874 (508h51m28.586696909s) [seg_pos={0 0}, upload=446522]: Segment reached optimal threshold of 13 pieces. Dec 26 20:20:28.715918924 (508h51m28.587001969s) [noise=true , address=45.91.116.12:4227 , piece_id=TPPZ5KQWDTJ4QOGPXV2QYG4DZNTIRFBUQVPCNIUM6OKIA56N7VTQ, seg_pos={0 0}, upload=446522]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: write tcp 188.166.225.170:44992->45.91.116.12:4227: use of closed network connection; write tcp 188.166.225.170:44992->45.91.116.12:4227: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44992->45.91.116.12:4227: use of closed network connection Dec 26 20:20:28.715926620 (508h51m28.587009655s) [noise=true , address=45.91.116.12:4227 , piece_id=TPPZ5KQWDTJ4QOGPXV2QYG4DZNTIRFBUQVPCNIUM6OKIA56N7VTQ, seg_pos={0 0}, upload=446522]: Long tail cancellation due to optimal threshold Dec 26 20:20:28.716023183 (508h51m28.587106218s) [noise=true , address=45.91.116.9:4264 , piece_id=6ORHC753BLKQURTCVUMJPPNNBS7R2VHHJLWTMR6USTFSELRE6J2Q, seg_pos={0 0}, upload=446522]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 26 20:20:28.716032281 (508h51m28.587115326s) [noise=true , address=45.91.116.9:4264 , piece_id=6ORHC753BLKQURTCVUMJPPNNBS7R2VHHJLWTMR6USTFSELRE6J2Q, seg_pos={0 0}, upload=446522]: Long tail cancellation due to optimal threshold Dec 26 20:20:28.716113736 (508h51m28.587196771s) [noise=true , address=45.91.116.5:4299 , piece_id=U2GDM6QG4I2TVR4HTZUZ7DIB4PDE33XX6LZ46YA66CGH3HYTWCFQ, seg_pos={0 0}, upload=446522]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: write tcp 188.166.225.170:44362->45.91.116.5:4299: use of closed network connection; write tcp 188.166.225.170:44362->45.91.116.5:4299: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44362->45.91.116.5:4299: use of closed network connection Dec 26 20:20:28.716126450 (508h51m28.587209495s) [noise=true , address=45.91.116.5:4299 , piece_id=U2GDM6QG4I2TVR4HTZUZ7DIB4PDE33XX6LZ46YA66CGH3HYTWCFQ, seg_pos={0 0}, upload=446522]: Long tail cancellation due to optimal threshold Dec 26 20:20:28.716223976 (508h51m28.587307021s) [seg_pos={0 0}, upload=446522]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 20:20:28.716366787 (508h51m28.587449832s) [upload=446522]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:20:28.716372949 (508h51m28.587455994s) [upload=446522]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:20:29.019780404 (508h51m28.890863439s) [upload=446522]: Done uploading segments... Dec 26 20:20:29.019800383 (508h51m28.890883428s) [upload=446522]: Upload finished. err: <nil> Duration: 30.002045023s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [............X..........................................X....] duration: 30.935212959s trace: https://jaeger.dev.storj.io/trace/663e21624a5a3654 FAIL (2025-12-26 20:20:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, Piece ID: PWVMOEKW5A4ZAVSVJAAHFTYYTQZGFHDF642GWR6EABEGWP2CDRGQ) context deadline exceeded; piecestore: (Node ID: 12whTD1Kvu4T7ANNtkSvDB66uTjVgfPTK9TAzExgvwJYgZ3zSFG, Piece ID: ME2LDK2QJI7LLG67ODDIJDKB5CAT47YDGMJ3CTVK4F6UDHJ4TPOQ) context deadline exceeded; piecestore: (Node ID: 12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH, Piece ID: HJGRPPEJK75YTIKBKNVK7J4TDRKBCNL7HA4JNMOKD2ROGC7C4P3Q) context deadline exceeded ======LOGS====== ) [noise=true , address=145.79.227.7:4296 , piece_id=PVH6VN4CG4O2DC22G74US7IC5GDIGM2KYYYQQPFO5MG22JQCAU2A, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.248055070 (508h48m1.119138115s) [noise=true , address=145.79.227.40:4078 , piece_id=J6TGA3SFQOLGL2C2TS352TSLZBMI6MDXU5BDTTGQYMN4A5QG462A, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.248175840 (508h48m1.119258875s) [noise=true , address=77.90.31.12:4600 , piece_id=JESM5JF4GWUDLEW4YT5SKHJOXD54TWRQBNGGQCBMDSS54XR4AEKQ, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.248604496 (508h48m1.119687531s) [noise=true , address=45.91.116.6:4985 , piece_id=WV5SXZIBMXB4FTNZFE4PKBR5ZWZTUUPADPRMXAVE4Z3HKDGLBOSQ, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.248640474 (508h48m1.119723509s) [noise=true , address=145.79.227.18:4021 , piece_id=ER7GXC5OHAPNGU37QTKPROKB4YTCDQBUHVEIEUH2K5FQ4X5AGM2Q, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.249180202 (508h48m1.120263238s) [noise=true , address=77.90.31.18:4821 , piece_id=HJGRPPEJK75YTIKBKNVK7J4TDRKBCNL7HA4JNMOKD2ROGC7C4P3Q, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.249851130 (508h48m1.120934175s) [noise=true , address=64.49.13.169:4709 , piece_id=ZGWBPAYFOQWU2LSTILYMVRNKGSP53UXEQHRW4NJEXW5OLNVCCWRQ, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.250638910 (508h48m1.121721955s) [noise=true , address=145.79.227.38:4001 , piece_id=TWPCSZEWSFC2ZT7QNWHRS3RY42LLBPIEF3EIQQ65OXFZSHWAV2GA, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.251000769 (508h48m1.122083814s) [noise=true , address=145.79.227.30:4792 , piece_id=RLEUVLBAOMJSDAYHMQJGKYQUC6NYYGSGUVIU5Z53XGBINIQV43GQ, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.251314737 (508h48m1.122397772s) [noise=true , address=145.79.227.5:4252 , piece_id=FI5RNNRKSIYHR3JKCOW3THJQKHAPYLUDDHT3Y4TRYU3OSMCSWQQQ, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.251635527 (508h48m1.122718572s) [noise=true , address=77.90.31.11:4588 , piece_id=ZWDOVL5PGZ4YPY6NZVVSBSWXDDD3DAJ4RUK2BWMOGTHDGWJ2G4DA, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.251919499 (508h48m1.123002544s) [noise=true , address=77.90.31.10:4536 , piece_id=V43VRDKXTE2PHOBPSQUIJPKMKY3ROAEQAI2D4XIJHOLQ3UIMN4RA, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.252236542 (508h48m1.123319588s) [noise=true , address=77.90.31.21:4894 , piece_id=AGL6GVJX2YLGHSZX33SUURUU5S22CF3YRMJTP3QMHHP7OKX2MFSA, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.252528137 (508h48m1.123611182s) [noise=true , address=77.90.31.15:4702 , piece_id=KJEPR7NMX4UHQZHTMDDJKBRLLQ3O5BLJSLYBXGCOFGP3LNC3FXGA, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.252861402 (508h48m1.123944447s) [noise=true , address=64.49.13.27:4961 , piece_id=XCVH62CEVRYGR53U5VG4L4HJMBJSQ562L4QVKTXZSC7XGAEB4LDQ, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:02.570977410 (508h48m2.442060455s) [upload=446477]: Next returned nil segment Dec 26 20:17:02.571003019 (508h48m2.442086054s) [upload=446477]: Waiting for error group managing segments... Dec 26 20:17:02.949353997 (508h48m2.820437042s) [noise=true , address=64.49.13.27:4961 , piece_id=XCVH62CEVRYGR53U5VG4L4HJMBJSQ562L4QVKTXZSC7XGAEB4LDQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.136546190 (508h48m3.007629235s) [noise=true , address=145.79.227.5:4252 , piece_id=FI5RNNRKSIYHR3JKCOW3THJQKHAPYLUDDHT3Y4TRYU3OSMCSWQQQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.155760468 (508h48m3.026843513s) [noise=true , address=145.79.227.38:4001 , piece_id=TWPCSZEWSFC2ZT7QNWHRS3RY42LLBPIEF3EIQQ65OXFZSHWAV2GA, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.179239646 (508h48m3.050322691s) [noise=true , address=145.79.227.18:4021 , piece_id=ER7GXC5OHAPNGU37QTKPROKB4YTCDQBUHVEIEUH2K5FQ4X5AGM2Q, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.372384563 (508h48m3.243467608s) [noise=true , address=77.90.31.15:4702 , piece_id=KJEPR7NMX4UHQZHTMDDJKBRLLQ3O5BLJSLYBXGCOFGP3LNC3FXGA, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.447635803 (508h48m3.318718848s) [noise=true , address=77.90.31.18:4821 , piece_id=HJGRPPEJK75YTIKBKNVK7J4TDRKBCNL7HA4JNMOKD2ROGC7C4P3Q, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.452148166 (508h48m3.323231201s) [noise=true , address=77.90.31.10:4536 , piece_id=V43VRDKXTE2PHOBPSQUIJPKMKY3ROAEQAI2D4XIJHOLQ3UIMN4RA, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.644263002 (508h48m3.515346047s) [noise=true , address=77.90.31.14:4689 , piece_id=PWVMOEKW5A4ZAVSVJAAHFTYYTQZGFHDF642GWR6EABEGWP2CDRGQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.710620497 (508h48m3.581703542s) [noise=true , address=45.91.116.8:4074 , piece_id=ME2LDK2QJI7LLG67ODDIJDKB5CAT47YDGMJ3CTVK4F6UDHJ4TPOQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:04.066910008 (508h48m3.937993053s) [noise=true , address=77.90.31.12:4600 , piece_id=JESM5JF4GWUDLEW4YT5SKHJOXD54TWRQBNGGQCBMDSS54XR4AEKQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:04.082282349 (508h48m3.953365394s) [noise=true , address=145.79.227.44:4219 , piece_id=KZVV2NJBIU22C4URLBZMQVQN4IZ2ZMQ5YODRJMCS6XYXULHCZWLQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:04.089048724 (508h48m3.960131759s) [noise=true , address=145.79.227.30:4792 , piece_id=RLEUVLBAOMJSDAYHMQJGKYQUC6NYYGSGUVIU5Z53XGBINIQV43GQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:04.288772749 (508h48m4.159855794s) [noise=true , address=77.90.31.21:4894 , piece_id=AGL6GVJX2YLGHSZX33SUURUU5S22CF3YRMJTP3QMHHP7OKX2MFSA, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:04.344183239 (508h48m4.215266294s) [noise=true , address=77.90.31.11:4588 , piece_id=ZWDOVL5PGZ4YPY6NZVVSBSWXDDD3DAJ4RUK2BWMOGTHDGWJ2G4DA, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:04.344208728 (508h48m4.215291773s) [seg_pos={0 0}, upload=446477]: Segment reached optimal threshold of 14 pieces. Dec 26 20:17:04.344470828 (508h48m4.215553873s) [noise=true , address=64.49.13.169:4709 , piece_id=ZGWBPAYFOQWU2LSTILYMVRNKGSP53UXEQHRW4NJEXW5OLNVCCWRQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:47062->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:47062->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47062->64.49.13.169:4709: use of closed network connection Dec 26 20:17:04.344481918 (508h48m4.215564953s) [noise=true , address=64.49.13.169:4709 , piece_id=ZGWBPAYFOQWU2LSTILYMVRNKGSP53UXEQHRW4NJEXW5OLNVCCWRQ, seg_pos={0 0}, upload=446477]: Long tail cancellation due to optimal threshold Dec 26 20:17:04.344541813 (508h48m4.215624848s) [noise=true , address=145.79.227.40:4078 , piece_id=J6TGA3SFQOLGL2C2TS352TSLZBMI6MDXU5BDTTGQYMN4A5QG462A, seg_pos={0 0}, upload=446477]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tzgtenV6us5ssD7xWGpZmVUoWEfbFhBNewbK9irHtZPkyDtRd): protocol: expected piece hash; context canceled; EOF Dec 26 20:17:04.344559476 (508h48m4.215642511s) [noise=true , address=145.79.227.40:4078 , piece_id=J6TGA3SFQOLGL2C2TS352TSLZBMI6MDXU5BDTTGQYMN4A5QG462A, seg_pos={0 0}, upload=446477]: Long tail cancellation due to optimal threshold Dec 26 20:17:04.344755479 (508h48m4.215838524s) [noise=true , address=145.79.227.7:4296 , piece_id=PVH6VN4CG4O2DC22G74US7IC5GDIGM2KYYYQQPFO5MG22JQCAU2A, seg_pos={0 0}, upload=446477]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cKhDbA4ptwux2LYNPNAJ7Qh8jECsGWd1KWaK6T6pD38bJWdt1): protocol: write tcp 188.166.225.170:48104->145.79.227.7:4296: use of closed network connection; write tcp 188.166.225.170:48104->145.79.227.7:4296: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48104->145.79.227.7:4296: use of closed network connection Dec 26 20:17:04.344763044 (508h48m4.215846099s) [noise=true , address=145.79.227.7:4296 , piece_id=PVH6VN4CG4O2DC22G74US7IC5GDIGM2KYYYQQPFO5MG22JQCAU2A, seg_pos={0 0}, upload=446477]: Long tail cancellation due to optimal threshold Dec 26 20:17:04.344928509 (508h48m4.216011544s) [noise=true , address=45.91.116.6:4985 , piece_id=WV5SXZIBMXB4FTNZFE4PKBR5ZWZTUUPADPRMXAVE4Z3HKDGLBOSQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RNs9MM9jDtNC2gZC63SYsePZTd5mE7dqUm9A5Z5QWdPX8PVnp): protocol: expected piece hash; context canceled; EOF Dec 26 20:17:04.344938277 (508h48m4.216021312s) [noise=true , address=45.91.116.6:4985 , piece_id=WV5SXZIBMXB4FTNZFE4PKBR5ZWZTUUPADPRMXAVE4Z3HKDGLBOSQ, seg_pos={0 0}, upload=446477]: Long tail cancellation due to optimal threshold Dec 26 20:17:04.344966722 (508h48m4.216049767s) [seg_pos={0 0}, upload=446477]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:17:04.345054548 (508h48m4.216137593s) [upload=446477]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:17:04.345059839 (508h48m4.216142874s) [upload=446477]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:17:04.653613492 (508h48m4.524696537s) [upload=446477]: Done uploading segments... Dec 26 20:17:04.653639822 (508h48m4.524722857s) [upload=446477]: Upload finished. err: <nil> Duration: 30.935232536s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............X..........................................X...] duration: 42.697594193s trace: https://jaeger.dev.storj.io/trace/663e21624a5a3654 FAIL (2025-12-26 20:20:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, Piece ID: PWVMOEKW5A4ZAVSVJAAHFTYYTQZGFHDF642GWR6EABEGWP2CDRGQ) context deadline exceeded; piecestore: (Node ID: 12whTD1Kvu4T7ANNtkSvDB66uTjVgfPTK9TAzExgvwJYgZ3zSFG, Piece ID: ME2LDK2QJI7LLG67ODDIJDKB5CAT47YDGMJ3CTVK4F6UDHJ4TPOQ) context deadline exceeded; piecestore: (Node ID: 12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH, Piece ID: HJGRPPEJK75YTIKBKNVK7J4TDRKBCNL7HA4JNMOKD2ROGC7C4P3Q) context deadline exceeded ======LOGS====== ) [noise=true , address=145.79.227.7:4296 , piece_id=PVH6VN4CG4O2DC22G74US7IC5GDIGM2KYYYQQPFO5MG22JQCAU2A, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.248055070 (508h48m1.119138115s) [noise=true , address=145.79.227.40:4078 , piece_id=J6TGA3SFQOLGL2C2TS352TSLZBMI6MDXU5BDTTGQYMN4A5QG462A, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.248175840 (508h48m1.119258875s) [noise=true , address=77.90.31.12:4600 , piece_id=JESM5JF4GWUDLEW4YT5SKHJOXD54TWRQBNGGQCBMDSS54XR4AEKQ, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.248604496 (508h48m1.119687531s) [noise=true , address=45.91.116.6:4985 , piece_id=WV5SXZIBMXB4FTNZFE4PKBR5ZWZTUUPADPRMXAVE4Z3HKDGLBOSQ, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.248640474 (508h48m1.119723509s) [noise=true , address=145.79.227.18:4021 , piece_id=ER7GXC5OHAPNGU37QTKPROKB4YTCDQBUHVEIEUH2K5FQ4X5AGM2Q, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.249180202 (508h48m1.120263238s) [noise=true , address=77.90.31.18:4821 , piece_id=HJGRPPEJK75YTIKBKNVK7J4TDRKBCNL7HA4JNMOKD2ROGC7C4P3Q, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.249851130 (508h48m1.120934175s) [noise=true , address=64.49.13.169:4709 , piece_id=ZGWBPAYFOQWU2LSTILYMVRNKGSP53UXEQHRW4NJEXW5OLNVCCWRQ, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.250638910 (508h48m1.121721955s) [noise=true , address=145.79.227.38:4001 , piece_id=TWPCSZEWSFC2ZT7QNWHRS3RY42LLBPIEF3EIQQ65OXFZSHWAV2GA, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.251000769 (508h48m1.122083814s) [noise=true , address=145.79.227.30:4792 , piece_id=RLEUVLBAOMJSDAYHMQJGKYQUC6NYYGSGUVIU5Z53XGBINIQV43GQ, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.251314737 (508h48m1.122397772s) [noise=true , address=145.79.227.5:4252 , piece_id=FI5RNNRKSIYHR3JKCOW3THJQKHAPYLUDDHT3Y4TRYU3OSMCSWQQQ, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.251635527 (508h48m1.122718572s) [noise=true , address=77.90.31.11:4588 , piece_id=ZWDOVL5PGZ4YPY6NZVVSBSWXDDD3DAJ4RUK2BWMOGTHDGWJ2G4DA, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.251919499 (508h48m1.123002544s) [noise=true , address=77.90.31.10:4536 , piece_id=V43VRDKXTE2PHOBPSQUIJPKMKY3ROAEQAI2D4XIJHOLQ3UIMN4RA, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.252236542 (508h48m1.123319588s) [noise=true , address=77.90.31.21:4894 , piece_id=AGL6GVJX2YLGHSZX33SUURUU5S22CF3YRMJTP3QMHHP7OKX2MFSA, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.252528137 (508h48m1.123611182s) [noise=true , address=77.90.31.15:4702 , piece_id=KJEPR7NMX4UHQZHTMDDJKBRLLQ3O5BLJSLYBXGCOFGP3LNC3FXGA, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.252861402 (508h48m1.123944447s) [noise=true , address=64.49.13.27:4961 , piece_id=XCVH62CEVRYGR53U5VG4L4HJMBJSQ562L4QVKTXZSC7XGAEB4LDQ, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:02.570977410 (508h48m2.442060455s) [upload=446477]: Next returned nil segment Dec 26 20:17:02.571003019 (508h48m2.442086054s) [upload=446477]: Waiting for error group managing segments... Dec 26 20:17:02.949353997 (508h48m2.820437042s) [noise=true , address=64.49.13.27:4961 , piece_id=XCVH62CEVRYGR53U5VG4L4HJMBJSQ562L4QVKTXZSC7XGAEB4LDQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.136546190 (508h48m3.007629235s) [noise=true , address=145.79.227.5:4252 , piece_id=FI5RNNRKSIYHR3JKCOW3THJQKHAPYLUDDHT3Y4TRYU3OSMCSWQQQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.155760468 (508h48m3.026843513s) [noise=true , address=145.79.227.38:4001 , piece_id=TWPCSZEWSFC2ZT7QNWHRS3RY42LLBPIEF3EIQQ65OXFZSHWAV2GA, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.179239646 (508h48m3.050322691s) [noise=true , address=145.79.227.18:4021 , piece_id=ER7GXC5OHAPNGU37QTKPROKB4YTCDQBUHVEIEUH2K5FQ4X5AGM2Q, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.372384563 (508h48m3.243467608s) [noise=true , address=77.90.31.15:4702 , piece_id=KJEPR7NMX4UHQZHTMDDJKBRLLQ3O5BLJSLYBXGCOFGP3LNC3FXGA, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.447635803 (508h48m3.318718848s) [noise=true , address=77.90.31.18:4821 , piece_id=HJGRPPEJK75YTIKBKNVK7J4TDRKBCNL7HA4JNMOKD2ROGC7C4P3Q, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.452148166 (508h48m3.323231201s) [noise=true , address=77.90.31.10:4536 , piece_id=V43VRDKXTE2PHOBPSQUIJPKMKY3ROAEQAI2D4XIJHOLQ3UIMN4RA, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.644263002 (508h48m3.515346047s) [noise=true , address=77.90.31.14:4689 , piece_id=PWVMOEKW5A4ZAVSVJAAHFTYYTQZGFHDF642GWR6EABEGWP2CDRGQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.710620497 (508h48m3.581703542s) [noise=true , address=45.91.116.8:4074 , piece_id=ME2LDK2QJI7LLG67ODDIJDKB5CAT47YDGMJ3CTVK4F6UDHJ4TPOQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:04.066910008 (508h48m3.937993053s) [noise=true , address=77.90.31.12:4600 , piece_id=JESM5JF4GWUDLEW4YT5SKHJOXD54TWRQBNGGQCBMDSS54XR4AEKQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:04.082282349 (508h48m3.953365394s) [noise=true , address=145.79.227.44:4219 , piece_id=KZVV2NJBIU22C4URLBZMQVQN4IZ2ZMQ5YODRJMCS6XYXULHCZWLQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:04.089048724 (508h48m3.960131759s) [noise=true , address=145.79.227.30:4792 , piece_id=RLEUVLBAOMJSDAYHMQJGKYQUC6NYYGSGUVIU5Z53XGBINIQV43GQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:04.288772749 (508h48m4.159855794s) [noise=true , address=77.90.31.21:4894 , piece_id=AGL6GVJX2YLGHSZX33SUURUU5S22CF3YRMJTP3QMHHP7OKX2MFSA, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:04.344183239 (508h48m4.215266294s) [noise=true , address=77.90.31.11:4588 , piece_id=ZWDOVL5PGZ4YPY6NZVVSBSWXDDD3DAJ4RUK2BWMOGTHDGWJ2G4DA, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:04.344208728 (508h48m4.215291773s) [seg_pos={0 0}, upload=446477]: Segment reached optimal threshold of 14 pieces. Dec 26 20:17:04.344470828 (508h48m4.215553873s) [noise=true , address=64.49.13.169:4709 , piece_id=ZGWBPAYFOQWU2LSTILYMVRNKGSP53UXEQHRW4NJEXW5OLNVCCWRQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:47062->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:47062->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47062->64.49.13.169:4709: use of closed network connection Dec 26 20:17:04.344481918 (508h48m4.215564953s) [noise=true , address=64.49.13.169:4709 , piece_id=ZGWBPAYFOQWU2LSTILYMVRNKGSP53UXEQHRW4NJEXW5OLNVCCWRQ, seg_pos={0 0}, upload=446477]: Long tail cancellation due to optimal threshold Dec 26 20:17:04.344541813 (508h48m4.215624848s) [noise=true , address=145.79.227.40:4078 , piece_id=J6TGA3SFQOLGL2C2TS352TSLZBMI6MDXU5BDTTGQYMN4A5QG462A, seg_pos={0 0}, upload=446477]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tzgtenV6us5ssD7xWGpZmVUoWEfbFhBNewbK9irHtZPkyDtRd): protocol: expected piece hash; context canceled; EOF Dec 26 20:17:04.344559476 (508h48m4.215642511s) [noise=true , address=145.79.227.40:4078 , piece_id=J6TGA3SFQOLGL2C2TS352TSLZBMI6MDXU5BDTTGQYMN4A5QG462A, seg_pos={0 0}, upload=446477]: Long tail cancellation due to optimal threshold Dec 26 20:17:04.344755479 (508h48m4.215838524s) [noise=true , address=145.79.227.7:4296 , piece_id=PVH6VN4CG4O2DC22G74US7IC5GDIGM2KYYYQQPFO5MG22JQCAU2A, seg_pos={0 0}, upload=446477]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cKhDbA4ptwux2LYNPNAJ7Qh8jECsGWd1KWaK6T6pD38bJWdt1): protocol: write tcp 188.166.225.170:48104->145.79.227.7:4296: use of closed network connection; write tcp 188.166.225.170:48104->145.79.227.7:4296: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48104->145.79.227.7:4296: use of closed network connection Dec 26 20:17:04.344763044 (508h48m4.215846099s) [noise=true , address=145.79.227.7:4296 , piece_id=PVH6VN4CG4O2DC22G74US7IC5GDIGM2KYYYQQPFO5MG22JQCAU2A, seg_pos={0 0}, upload=446477]: Long tail cancellation due to optimal threshold Dec 26 20:17:04.344928509 (508h48m4.216011544s) [noise=true , address=45.91.116.6:4985 , piece_id=WV5SXZIBMXB4FTNZFE4PKBR5ZWZTUUPADPRMXAVE4Z3HKDGLBOSQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RNs9MM9jDtNC2gZC63SYsePZTd5mE7dqUm9A5Z5QWdPX8PVnp): protocol: expected piece hash; context canceled; EOF Dec 26 20:17:04.344938277 (508h48m4.216021312s) [noise=true , address=45.91.116.6:4985 , piece_id=WV5SXZIBMXB4FTNZFE4PKBR5ZWZTUUPADPRMXAVE4Z3HKDGLBOSQ, seg_pos={0 0}, upload=446477]: Long tail cancellation due to optimal threshold Dec 26 20:17:04.344966722 (508h48m4.216049767s) [seg_pos={0 0}, upload=446477]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:17:04.345054548 (508h48m4.216137593s) [upload=446477]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:17:04.345059839 (508h48m4.216142874s) [upload=446477]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:17:04.653613492 (508h48m4.524696537s) [upload=446477]: Done uploading segments... Dec 26 20:17:04.653639822 (508h48m4.524722857s) [upload=446477]: Upload finished. err: <nil> Duration: 42.697621475s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [....X.X.X.X.X..............XX..X..X...........XXX.X.X....X..] duration: 6.809462219s trace: https://jaeger.dev.storj.io/trace/40f126ea36ea665d FAIL (2025-12-26 20:19:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: VCI3H7OAZEUZQXGUNWNYRXQSJFL2M3XUZDSR5F65KLF4L35JPHRA) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: BUPCFZDXK4JZHO5467ZXKGBMAWOJ4I4X4I26UTUDW4SINITHBPDA) context deadline exceeded ======LOGS====== 0:17:41.014584898 (508h48m40.885667933s) [upload=446487]: Starting upload Dec 26 20:17:41.014638820 (508h48m40.885721855s) [upload=446487]: Done starting upload Dec 26 20:17:41.014812481 (508h48m40.885895516s) [upload=446487]: Uploading segments... Dec 26 20:17:41.014846826 (508h48m40.885929871s) [upload=446487]: Got next segment. Inline: false Dec 26 20:17:41.014871173 (508h48m40.885954208s) [upload=446487]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 20:17:41.014875401 (508h48m40.885958446s) [upload=446487]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 20:17:41.257281529 (508h48m41.128364574s) [seg_pos={0 0}, upload=446487]: Begin upload segment... Dec 26 20:17:41.257426135 (508h48m41.12850918s) [seg_pos={0 0}, upload=446487]: Done begin upload segment. Dec 26 20:17:41.257456963 (508h48m41.128540018s) [noise=true , address=145.79.227.46:4601 , piece_id=ZEVL7ISFPLK7332BK65UE6NH6DB5VRSECKF55I665UGXWBY2J4AQ, seg_pos={0 0}, upload=446487]: Uploading piece... Dec 26 20:17:41.257512720 (508h48m41.128595755s) [noise=true , address=145.79.227.42:4121 , piece_id=CL72TZYQF6DU6RURRBSCGMHFYUDWKQFOPYPNUF3Y3AODOGFONE4A, seg_pos={0 0}, upload=446487]: Uploading piece... Dec 26 20:17:41.258325507 (508h48m41.129408552s) [noise=true , address=145.79.227.36:4517 , piece_id=YGZHYLCPW7LDS7E4P3MQ7KMB57SBRWNDMFFAFBZQZGZBXKI7ZAAA, seg_pos={0 0}, upload=446487]: Uploading piece... Dec 26 20:17:41.259001555 (508h48m41.13008459s) [noise=true , address=145.79.227.49:4338 , piece_id=SNSNPRIF45ZYR4625WUVQ3DJEEXSWUY3FG26IRXQUTSRU3EVK4JA, seg_pos={0 0}, upload=446487]: Uploading piece... Dec 26 20:17:41.259741914 (508h48m41.13082495s) [noise=true , address=145.79.227.21:4475 , piece_id=2IC7FCDV4V4K3EIBYFBBJJEDLT26ZUQFSINCMOXVFWVUYYYXPMKQ, seg_pos={0 0}, upload=446487]: Uploading piece... Dec 26 20:17:41.260312020 (508h48m41.131395065s) [noise=true , address=145.79.227.37:4566 , piece_id=KXEGB7VCYIEH7WSSSJIY3XABIBGNRWHQL4VIMIWOCF57MJ5SGIYQ, seg_pos={0 0}, upload=446487]: Uploading piece... Dec 26 20:17:41.261640609 (508h48m41.132723654s) [noise=true , address=45.91.116.5:4300 , piece_id=LORS4EXAA6GA5JQTTKZC6D7QCETROW2CFDODUG7YYNQTHTIIBLHA, seg_pos={0 0}, upload=446487]: Uploading piece... Dec 26 20:17:41.262376821 (508h48m41.133459856s) [noise=true , address=145.79.227.30:4762 , piece_id=7PEOUJEEMX4KP63ZFN3EQUGU24DQPZIUFBG3VTQ4LTUCRXE5DCLA, seg_pos={0 0}, upload=446487]: Uploading piece... Dec 26 20:17:41.263036998 (508h48m41.134120043s) [noise=true , address=145.79.227.52:4445 , piece_id=KQUT6BOUIER2AHKWLLIIVN6EDMP7AU2KQ5BVQ7JWHNT5G3J5RDNQ, seg_pos={0 0}, upload=446487]: Uploading piece... Dec 26 20:17:41.264209501 (508h48m41.135292546s) [noise=true , address=77.90.31.10:4518 , piece_id=VDLRHPKBP3FY5YIKLBGUUEU23Y6XRW7FYTU22UC5OKFRDYNSXWJQ, seg_pos={0 0}, upload=446487]: Uploading piece... Dec 26 20:17:41.264383082 (508h48m41.135466127s) [noise=true , address=145.79.227.25:4618 , piece_id=3Q7CAENVOQIJ3PUBM24GCENTRSXR2O3RSZ5ST3WFYALLMVRVPL3A, seg_pos={0 0}, upload=446487]: Uploading piece... Dec 26 20:17:41.265461195 (508h48m41.13654423s) [noise=true , address=77.90.31.16:4736 , piece_id=VCI3H7OAZEUZQXGUNWNYRXQSJFL2M3XUZDSR5F65KLF4L35JPHRA, seg_pos={0 0}, upload=446487]: Uploading piece... Dec 26 20:17:41.266725352 (508h48m41.137808397s) [noise=true , address=145.79.227.2:4114 , piece_id=PMB6A4DRRZJZ22JQTEJNQA57PHBYYVABILPMIMZCTLJ3HMETUWTA, seg_pos={0 0}, upload=446487]: Uploading piece... Dec 26 20:17:41.267072442 (508h48m41.138155488s) [noise=true , address=145.79.227.39:4013 , piece_id=KCHNWV6IZRTADNLURJYYNI66XEJ7STVIXCKZ25BPYRHKC3SY3K2A, seg_pos={0 0}, upload=446487]: Uploading piece... Dec 26 20:17:41.268750017 (508h48m41.139833062s) [noise=true , address=45.91.116.10:4082 , piece_id=INALJ2OJEDH6KNPN422S67RP4XBT5UDTXHCO35LLDZJDJMDSWLPQ, seg_pos={0 0}, upload=446487]: Uploading piece... Dec 26 20:17:41.270307242 (508h48m41.141390277s) [noise=true , address=45.91.116.7:4012 , piece_id=BUPCFZDXK4JZHO5467ZXKGBMAWOJ4I4X4I26UTUDW4SINITHBPDA, seg_pos={0 0}, upload=446487]: Uploading piece... Dec 26 20:17:42.971252485 (508h48m42.84233552s) [upload=446487]: Next returned nil segment Dec 26 20:17:42.971265711 (508h48m42.842348746s) [upload=446487]: Waiting for error group managing segments... Dec 26 20:17:43.254289712 (508h48m43.125372747s) [noise=true , address=145.79.227.37:4566 , piece_id=KXEGB7VCYIEH7WSSSJIY3XABIBGNRWHQL4VIMIWOCF57MJ5SGIYQ, seg_pos={0 0}, upload=446487]: Done uploading piece. err: <nil> Dec 26 20:17:43.266691186 (508h48m43.137774231s) [noise=true , address=145.79.227.21:4475 , piece_id=2IC7FCDV4V4K3EIBYFBBJJEDLT26ZUQFSINCMOXVFWVUYYYXPMKQ, seg_pos={0 0}, upload=446487]: Done uploading piece. err: <nil> Dec 26 20:17:43.267959582 (508h48m43.139042617s) [noise=true , address=145.79.227.49:4338 , piece_id=SNSNPRIF45ZYR4625WUVQ3DJEEXSWUY3FG26IRXQUTSRU3EVK4JA, seg_pos={0 0}, upload=446487]: Done uploading piece. err: <nil> Dec 26 20:17:43.289294236 (508h48m43.160377271s) [noise=true , address=145.79.227.2:4114 , piece_id=PMB6A4DRRZJZ22JQTEJNQA57PHBYYVABILPMIMZCTLJ3HMETUWTA, seg_pos={0 0}, upload=446487]: Done uploading piece. err: <nil> Dec 26 20:17:43.290403348 (508h48m43.161486403s) [noise=true , address=145.79.227.36:4517 , piece_id=YGZHYLCPW7LDS7E4P3MQ7KMB57SBRWNDMFFAFBZQZGZBXKI7ZAAA, seg_pos={0 0}, upload=446487]: Done uploading piece. err: <nil> Dec 26 20:17:43.294942792 (508h48m43.166025827s) [noise=true , address=145.79.227.42:4121 , piece_id=CL72TZYQF6DU6RURRBSCGMHFYUDWKQFOPYPNUF3Y3AODOGFONE4A, seg_pos={0 0}, upload=446487]: Done uploading piece. err: <nil> Dec 26 20:17:43.301057566 (508h48m43.172140601s) [noise=true , address=145.79.227.25:4618 , piece_id=3Q7CAENVOQIJ3PUBM24GCENTRSXR2O3RSZ5ST3WFYALLMVRVPL3A, seg_pos={0 0}, upload=446487]: Done uploading piece. err: <nil> Dec 26 20:17:43.301501650 (508h48m43.172584685s) [noise=true , address=145.79.227.46:4601 , piece_id=ZEVL7ISFPLK7332BK65UE6NH6DB5VRSECKF55I665UGXWBY2J4AQ, seg_pos={0 0}, upload=446487]: Done uploading piece. err: <nil> Dec 26 20:17:43.319872011 (508h48m43.190955046s) [noise=true , address=145.79.227.52:4445 , piece_id=KQUT6BOUIER2AHKWLLIIVN6EDMP7AU2KQ5BVQ7JWHNT5G3J5RDNQ, seg_pos={0 0}, upload=446487]: Done uploading piece. err: <nil> Dec 26 20:17:43.512314291 (508h48m43.383397326s) [noise=true , address=45.91.116.7:4012 , piece_id=BUPCFZDXK4JZHO5467ZXKGBMAWOJ4I4X4I26UTUDW4SINITHBPDA, seg_pos={0 0}, upload=446487]: Done uploading piece. err: <nil> Dec 26 20:17:43.530414296 (508h48m43.401497331s) [noise=true , address=77.90.31.10:4518 , piece_id=VDLRHPKBP3FY5YIKLBGUUEU23Y6XRW7FYTU22UC5OKFRDYNSXWJQ, seg_pos={0 0}, upload=446487]: Done uploading piece. err: <nil> Dec 26 20:17:43.571567899 (508h48m43.442650934s) [noise=true , address=45.91.116.5:4300 , piece_id=LORS4EXAA6GA5JQTTKZC6D7QCETROW2CFDODUG7YYNQTHTIIBLHA, seg_pos={0 0}, upload=446487]: Done uploading piece. err: <nil> Dec 26 20:17:44.330593469 (508h48m44.201676504s) [noise=true , address=77.90.31.16:4736 , piece_id=VCI3H7OAZEUZQXGUNWNYRXQSJFL2M3XUZDSR5F65KLF4L35JPHRA, seg_pos={0 0}, upload=446487]: Done uploading piece. err: <nil> Dec 26 20:17:44.330614650 (508h48m44.201697685s) [seg_pos={0 0}, upload=446487]: Segment reached optimal threshold of 13 pieces. Dec 26 20:17:44.330824420 (508h48m44.201907465s) [noise=true , address=145.79.227.30:4762 , piece_id=7PEOUJEEMX4KP63ZFN3EQUGU24DQPZIUFBG3VTQ4LTUCRXE5DCLA, seg_pos={0 0}, upload=446487]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 26 20:17:44.330833887 (508h48m44.201916932s) [noise=true , address=145.79.227.30:4762 , piece_id=7PEOUJEEMX4KP63ZFN3EQUGU24DQPZIUFBG3VTQ4LTUCRXE5DCLA, seg_pos={0 0}, upload=446487]: Long tail cancellation due to optimal threshold Dec 26 20:17:44.330858304 (508h48m44.201941349s) [noise=true , address=145.79.227.39:4013 , piece_id=KCHNWV6IZRTADNLURJYYNI66XEJ7STVIXCKZ25BPYRHKC3SY3K2A, seg_pos={0 0}, upload=446487]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b): protocol: write tcp 188.166.225.170:44622->145.79.227.39:4013: use of closed network connection; write tcp 188.166.225.170:44622->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44622->145.79.227.39:4013: use of closed network connection Dec 26 20:17:44.330868163 (508h48m44.201951208s) [noise=true , address=145.79.227.39:4013 , piece_id=KCHNWV6IZRTADNLURJYYNI66XEJ7STVIXCKZ25BPYRHKC3SY3K2A, seg_pos={0 0}, upload=446487]: Long tail cancellation due to optimal threshold Dec 26 20:17:44.331061250 (508h48m44.202144295s) [noise=true , address=45.91.116.10:4082 , piece_id=INALJ2OJEDH6KNPN422S67RP4XBT5UDTXHCO35LLDZJDJMDSWLPQ, seg_pos={0 0}, upload=446487]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 26 20:17:44.331069176 (508h48m44.202152211s) [noise=true , address=45.91.116.10:4082 , piece_id=INALJ2OJEDH6KNPN422S67RP4XBT5UDTXHCO35LLDZJDJMDSWLPQ, seg_pos={0 0}, upload=446487]: Long tail cancellation due to optimal threshold Dec 26 20:17:44.331087610 (508h48m44.202170655s) [seg_pos={0 0}, upload=446487]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 20:17:44.331159136 (508h48m44.202242181s) [upload=446487]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:17:44.331172072 (508h48m44.202255117s) [upload=446487]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:17:44.617935152 (508h48m44.489018207s) [upload=446487]: Done uploading segments... Dec 26 20:17:44.617959108 (508h48m44.489042143s) [upload=446487]: Upload finished. err: <nil> Duration: 6.809488208s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............X..........................................X..] duration: 30.338945072s trace: https://jaeger.dev.storj.io/trace/663e21624a5a3654 FAIL (2025-12-26 20:19:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, Piece ID: PWVMOEKW5A4ZAVSVJAAHFTYYTQZGFHDF642GWR6EABEGWP2CDRGQ) context deadline exceeded; piecestore: (Node ID: 12whTD1Kvu4T7ANNtkSvDB66uTjVgfPTK9TAzExgvwJYgZ3zSFG, Piece ID: ME2LDK2QJI7LLG67ODDIJDKB5CAT47YDGMJ3CTVK4F6UDHJ4TPOQ) context deadline exceeded; piecestore: (Node ID: 12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH, Piece ID: HJGRPPEJK75YTIKBKNVK7J4TDRKBCNL7HA4JNMOKD2ROGC7C4P3Q) context deadline exceeded ======LOGS====== ) [noise=true , address=145.79.227.7:4296 , piece_id=PVH6VN4CG4O2DC22G74US7IC5GDIGM2KYYYQQPFO5MG22JQCAU2A, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.248055070 (508h48m1.119138115s) [noise=true , address=145.79.227.40:4078 , piece_id=J6TGA3SFQOLGL2C2TS352TSLZBMI6MDXU5BDTTGQYMN4A5QG462A, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.248175840 (508h48m1.119258875s) [noise=true , address=77.90.31.12:4600 , piece_id=JESM5JF4GWUDLEW4YT5SKHJOXD54TWRQBNGGQCBMDSS54XR4AEKQ, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.248604496 (508h48m1.119687531s) [noise=true , address=45.91.116.6:4985 , piece_id=WV5SXZIBMXB4FTNZFE4PKBR5ZWZTUUPADPRMXAVE4Z3HKDGLBOSQ, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.248640474 (508h48m1.119723509s) [noise=true , address=145.79.227.18:4021 , piece_id=ER7GXC5OHAPNGU37QTKPROKB4YTCDQBUHVEIEUH2K5FQ4X5AGM2Q, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.249180202 (508h48m1.120263238s) [noise=true , address=77.90.31.18:4821 , piece_id=HJGRPPEJK75YTIKBKNVK7J4TDRKBCNL7HA4JNMOKD2ROGC7C4P3Q, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.249851130 (508h48m1.120934175s) [noise=true , address=64.49.13.169:4709 , piece_id=ZGWBPAYFOQWU2LSTILYMVRNKGSP53UXEQHRW4NJEXW5OLNVCCWRQ, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.250638910 (508h48m1.121721955s) [noise=true , address=145.79.227.38:4001 , piece_id=TWPCSZEWSFC2ZT7QNWHRS3RY42LLBPIEF3EIQQ65OXFZSHWAV2GA, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.251000769 (508h48m1.122083814s) [noise=true , address=145.79.227.30:4792 , piece_id=RLEUVLBAOMJSDAYHMQJGKYQUC6NYYGSGUVIU5Z53XGBINIQV43GQ, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.251314737 (508h48m1.122397772s) [noise=true , address=145.79.227.5:4252 , piece_id=FI5RNNRKSIYHR3JKCOW3THJQKHAPYLUDDHT3Y4TRYU3OSMCSWQQQ, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.251635527 (508h48m1.122718572s) [noise=true , address=77.90.31.11:4588 , piece_id=ZWDOVL5PGZ4YPY6NZVVSBSWXDDD3DAJ4RUK2BWMOGTHDGWJ2G4DA, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.251919499 (508h48m1.123002544s) [noise=true , address=77.90.31.10:4536 , piece_id=V43VRDKXTE2PHOBPSQUIJPKMKY3ROAEQAI2D4XIJHOLQ3UIMN4RA, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.252236542 (508h48m1.123319588s) [noise=true , address=77.90.31.21:4894 , piece_id=AGL6GVJX2YLGHSZX33SUURUU5S22CF3YRMJTP3QMHHP7OKX2MFSA, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.252528137 (508h48m1.123611182s) [noise=true , address=77.90.31.15:4702 , piece_id=KJEPR7NMX4UHQZHTMDDJKBRLLQ3O5BLJSLYBXGCOFGP3LNC3FXGA, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.252861402 (508h48m1.123944447s) [noise=true , address=64.49.13.27:4961 , piece_id=XCVH62CEVRYGR53U5VG4L4HJMBJSQ562L4QVKTXZSC7XGAEB4LDQ, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:02.570977410 (508h48m2.442060455s) [upload=446477]: Next returned nil segment Dec 26 20:17:02.571003019 (508h48m2.442086054s) [upload=446477]: Waiting for error group managing segments... Dec 26 20:17:02.949353997 (508h48m2.820437042s) [noise=true , address=64.49.13.27:4961 , piece_id=XCVH62CEVRYGR53U5VG4L4HJMBJSQ562L4QVKTXZSC7XGAEB4LDQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.136546190 (508h48m3.007629235s) [noise=true , address=145.79.227.5:4252 , piece_id=FI5RNNRKSIYHR3JKCOW3THJQKHAPYLUDDHT3Y4TRYU3OSMCSWQQQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.155760468 (508h48m3.026843513s) [noise=true , address=145.79.227.38:4001 , piece_id=TWPCSZEWSFC2ZT7QNWHRS3RY42LLBPIEF3EIQQ65OXFZSHWAV2GA, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.179239646 (508h48m3.050322691s) [noise=true , address=145.79.227.18:4021 , piece_id=ER7GXC5OHAPNGU37QTKPROKB4YTCDQBUHVEIEUH2K5FQ4X5AGM2Q, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.372384563 (508h48m3.243467608s) [noise=true , address=77.90.31.15:4702 , piece_id=KJEPR7NMX4UHQZHTMDDJKBRLLQ3O5BLJSLYBXGCOFGP3LNC3FXGA, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.447635803 (508h48m3.318718848s) [noise=true , address=77.90.31.18:4821 , piece_id=HJGRPPEJK75YTIKBKNVK7J4TDRKBCNL7HA4JNMOKD2ROGC7C4P3Q, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.452148166 (508h48m3.323231201s) [noise=true , address=77.90.31.10:4536 , piece_id=V43VRDKXTE2PHOBPSQUIJPKMKY3ROAEQAI2D4XIJHOLQ3UIMN4RA, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.644263002 (508h48m3.515346047s) [noise=true , address=77.90.31.14:4689 , piece_id=PWVMOEKW5A4ZAVSVJAAHFTYYTQZGFHDF642GWR6EABEGWP2CDRGQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.710620497 (508h48m3.581703542s) [noise=true , address=45.91.116.8:4074 , piece_id=ME2LDK2QJI7LLG67ODDIJDKB5CAT47YDGMJ3CTVK4F6UDHJ4TPOQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:04.066910008 (508h48m3.937993053s) [noise=true , address=77.90.31.12:4600 , piece_id=JESM5JF4GWUDLEW4YT5SKHJOXD54TWRQBNGGQCBMDSS54XR4AEKQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:04.082282349 (508h48m3.953365394s) [noise=true , address=145.79.227.44:4219 , piece_id=KZVV2NJBIU22C4URLBZMQVQN4IZ2ZMQ5YODRJMCS6XYXULHCZWLQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:04.089048724 (508h48m3.960131759s) [noise=true , address=145.79.227.30:4792 , piece_id=RLEUVLBAOMJSDAYHMQJGKYQUC6NYYGSGUVIU5Z53XGBINIQV43GQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:04.288772749 (508h48m4.159855794s) [noise=true , address=77.90.31.21:4894 , piece_id=AGL6GVJX2YLGHSZX33SUURUU5S22CF3YRMJTP3QMHHP7OKX2MFSA, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:04.344183239 (508h48m4.215266294s) [noise=true , address=77.90.31.11:4588 , piece_id=ZWDOVL5PGZ4YPY6NZVVSBSWXDDD3DAJ4RUK2BWMOGTHDGWJ2G4DA, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:04.344208728 (508h48m4.215291773s) [seg_pos={0 0}, upload=446477]: Segment reached optimal threshold of 14 pieces. Dec 26 20:17:04.344470828 (508h48m4.215553873s) [noise=true , address=64.49.13.169:4709 , piece_id=ZGWBPAYFOQWU2LSTILYMVRNKGSP53UXEQHRW4NJEXW5OLNVCCWRQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:47062->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:47062->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47062->64.49.13.169:4709: use of closed network connection Dec 26 20:17:04.344481918 (508h48m4.215564953s) [noise=true , address=64.49.13.169:4709 , piece_id=ZGWBPAYFOQWU2LSTILYMVRNKGSP53UXEQHRW4NJEXW5OLNVCCWRQ, seg_pos={0 0}, upload=446477]: Long tail cancellation due to optimal threshold Dec 26 20:17:04.344541813 (508h48m4.215624848s) [noise=true , address=145.79.227.40:4078 , piece_id=J6TGA3SFQOLGL2C2TS352TSLZBMI6MDXU5BDTTGQYMN4A5QG462A, seg_pos={0 0}, upload=446477]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tzgtenV6us5ssD7xWGpZmVUoWEfbFhBNewbK9irHtZPkyDtRd): protocol: expected piece hash; context canceled; EOF Dec 26 20:17:04.344559476 (508h48m4.215642511s) [noise=true , address=145.79.227.40:4078 , piece_id=J6TGA3SFQOLGL2C2TS352TSLZBMI6MDXU5BDTTGQYMN4A5QG462A, seg_pos={0 0}, upload=446477]: Long tail cancellation due to optimal threshold Dec 26 20:17:04.344755479 (508h48m4.215838524s) [noise=true , address=145.79.227.7:4296 , piece_id=PVH6VN4CG4O2DC22G74US7IC5GDIGM2KYYYQQPFO5MG22JQCAU2A, seg_pos={0 0}, upload=446477]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cKhDbA4ptwux2LYNPNAJ7Qh8jECsGWd1KWaK6T6pD38bJWdt1): protocol: write tcp 188.166.225.170:48104->145.79.227.7:4296: use of closed network connection; write tcp 188.166.225.170:48104->145.79.227.7:4296: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48104->145.79.227.7:4296: use of closed network connection Dec 26 20:17:04.344763044 (508h48m4.215846099s) [noise=true , address=145.79.227.7:4296 , piece_id=PVH6VN4CG4O2DC22G74US7IC5GDIGM2KYYYQQPFO5MG22JQCAU2A, seg_pos={0 0}, upload=446477]: Long tail cancellation due to optimal threshold Dec 26 20:17:04.344928509 (508h48m4.216011544s) [noise=true , address=45.91.116.6:4985 , piece_id=WV5SXZIBMXB4FTNZFE4PKBR5ZWZTUUPADPRMXAVE4Z3HKDGLBOSQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RNs9MM9jDtNC2gZC63SYsePZTd5mE7dqUm9A5Z5QWdPX8PVnp): protocol: expected piece hash; context canceled; EOF Dec 26 20:17:04.344938277 (508h48m4.216021312s) [noise=true , address=45.91.116.6:4985 , piece_id=WV5SXZIBMXB4FTNZFE4PKBR5ZWZTUUPADPRMXAVE4Z3HKDGLBOSQ, seg_pos={0 0}, upload=446477]: Long tail cancellation due to optimal threshold Dec 26 20:17:04.344966722 (508h48m4.216049767s) [seg_pos={0 0}, upload=446477]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:17:04.345054548 (508h48m4.216137593s) [upload=446477]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:17:04.345059839 (508h48m4.216142874s) [upload=446477]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:17:04.653613492 (508h48m4.524696537s) [upload=446477]: Done uploading segments... Dec 26 20:17:04.653639822 (508h48m4.524722857s) [upload=446477]: Upload finished. err: <nil> Duration: 30.338974377s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.....X.X.X.X.X..............XX..X..X...........XXX.X.X....X.] duration: 23.810778663s trace: https://jaeger.dev.storj.io/trace/40f126ea36ea665d FAIL (2025-12-26 20:19:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: VCI3H7OAZEUZQXGUNWNYRXQSJFL2M3XUZDSR5F65KLF4L35JPHRA) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: BUPCFZDXK4JZHO5467ZXKGBMAWOJ4I4X4I26UTUDW4SINITHBPDA) context deadline exceeded ======LOGS====== 0:17:41.014584898 (508h48m40.885667933s) [upload=446487]: Starting upload Dec 26 20:17:41.014638820 (508h48m40.885721855s) [upload=446487]: Done starting upload Dec 26 20:17:41.014812481 (508h48m40.885895516s) [upload=446487]: Uploading segments... Dec 26 20:17:41.014846826 (508h48m40.885929871s) [upload=446487]: Got next segment. Inline: false Dec 26 20:17:41.014871173 (508h48m40.885954208s) [upload=446487]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 20:17:41.014875401 (508h48m40.885958446s) [upload=446487]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 20:17:41.257281529 (508h48m41.128364574s) [seg_pos={0 0}, upload=446487]: Begin upload segment... Dec 26 20:17:41.257426135 (508h48m41.12850918s) [seg_pos={0 0}, upload=446487]: Done begin upload segment. Dec 26 20:17:41.257456963 (508h48m41.128540018s) [noise=true , address=145.79.227.46:4601 , piece_id=ZEVL7ISFPLK7332BK65UE6NH6DB5VRSECKF55I665UGXWBY2J4AQ, seg_pos={0 0}, upload=446487]: Uploading piece... Dec 26 20:17:41.257512720 (508h48m41.128595755s) [noise=true , address=145.79.227.42:4121 , piece_id=CL72TZYQF6DU6RURRBSCGMHFYUDWKQFOPYPNUF3Y3AODOGFONE4A, seg_pos={0 0}, upload=446487]: Uploading piece... Dec 26 20:17:41.258325507 (508h48m41.129408552s) [noise=true , address=145.79.227.36:4517 , piece_id=YGZHYLCPW7LDS7E4P3MQ7KMB57SBRWNDMFFAFBZQZGZBXKI7ZAAA, seg_pos={0 0}, upload=446487]: Uploading piece... Dec 26 20:17:41.259001555 (508h48m41.13008459s) [noise=true , address=145.79.227.49:4338 , piece_id=SNSNPRIF45ZYR4625WUVQ3DJEEXSWUY3FG26IRXQUTSRU3EVK4JA, seg_pos={0 0}, upload=446487]: Uploading piece... Dec 26 20:17:41.259741914 (508h48m41.13082495s) [noise=true , address=145.79.227.21:4475 , piece_id=2IC7FCDV4V4K3EIBYFBBJJEDLT26ZUQFSINCMOXVFWVUYYYXPMKQ, seg_pos={0 0}, upload=446487]: Uploading piece... Dec 26 20:17:41.260312020 (508h48m41.131395065s) [noise=true , address=145.79.227.37:4566 , piece_id=KXEGB7VCYIEH7WSSSJIY3XABIBGNRWHQL4VIMIWOCF57MJ5SGIYQ, seg_pos={0 0}, upload=446487]: Uploading piece... Dec 26 20:17:41.261640609 (508h48m41.132723654s) [noise=true , address=45.91.116.5:4300 , piece_id=LORS4EXAA6GA5JQTTKZC6D7QCETROW2CFDODUG7YYNQTHTIIBLHA, seg_pos={0 0}, upload=446487]: Uploading piece... Dec 26 20:17:41.262376821 (508h48m41.133459856s) [noise=true , address=145.79.227.30:4762 , piece_id=7PEOUJEEMX4KP63ZFN3EQUGU24DQPZIUFBG3VTQ4LTUCRXE5DCLA, seg_pos={0 0}, upload=446487]: Uploading piece... Dec 26 20:17:41.263036998 (508h48m41.134120043s) [noise=true , address=145.79.227.52:4445 , piece_id=KQUT6BOUIER2AHKWLLIIVN6EDMP7AU2KQ5BVQ7JWHNT5G3J5RDNQ, seg_pos={0 0}, upload=446487]: Uploading piece... Dec 26 20:17:41.264209501 (508h48m41.135292546s) [noise=true , address=77.90.31.10:4518 , piece_id=VDLRHPKBP3FY5YIKLBGUUEU23Y6XRW7FYTU22UC5OKFRDYNSXWJQ, seg_pos={0 0}, upload=446487]: Uploading piece... Dec 26 20:17:41.264383082 (508h48m41.135466127s) [noise=true , address=145.79.227.25:4618 , piece_id=3Q7CAENVOQIJ3PUBM24GCENTRSXR2O3RSZ5ST3WFYALLMVRVPL3A, seg_pos={0 0}, upload=446487]: Uploading piece... Dec 26 20:17:41.265461195 (508h48m41.13654423s) [noise=true , address=77.90.31.16:4736 , piece_id=VCI3H7OAZEUZQXGUNWNYRXQSJFL2M3XUZDSR5F65KLF4L35JPHRA, seg_pos={0 0}, upload=446487]: Uploading piece... Dec 26 20:17:41.266725352 (508h48m41.137808397s) [noise=true , address=145.79.227.2:4114 , piece_id=PMB6A4DRRZJZ22JQTEJNQA57PHBYYVABILPMIMZCTLJ3HMETUWTA, seg_pos={0 0}, upload=446487]: Uploading piece... Dec 26 20:17:41.267072442 (508h48m41.138155488s) [noise=true , address=145.79.227.39:4013 , piece_id=KCHNWV6IZRTADNLURJYYNI66XEJ7STVIXCKZ25BPYRHKC3SY3K2A, seg_pos={0 0}, upload=446487]: Uploading piece... Dec 26 20:17:41.268750017 (508h48m41.139833062s) [noise=true , address=45.91.116.10:4082 , piece_id=INALJ2OJEDH6KNPN422S67RP4XBT5UDTXHCO35LLDZJDJMDSWLPQ, seg_pos={0 0}, upload=446487]: Uploading piece... Dec 26 20:17:41.270307242 (508h48m41.141390277s) [noise=true , address=45.91.116.7:4012 , piece_id=BUPCFZDXK4JZHO5467ZXKGBMAWOJ4I4X4I26UTUDW4SINITHBPDA, seg_pos={0 0}, upload=446487]: Uploading piece... Dec 26 20:17:42.971252485 (508h48m42.84233552s) [upload=446487]: Next returned nil segment Dec 26 20:17:42.971265711 (508h48m42.842348746s) [upload=446487]: Waiting for error group managing segments... Dec 26 20:17:43.254289712 (508h48m43.125372747s) [noise=true , address=145.79.227.37:4566 , piece_id=KXEGB7VCYIEH7WSSSJIY3XABIBGNRWHQL4VIMIWOCF57MJ5SGIYQ, seg_pos={0 0}, upload=446487]: Done uploading piece. err: <nil> Dec 26 20:17:43.266691186 (508h48m43.137774231s) [noise=true , address=145.79.227.21:4475 , piece_id=2IC7FCDV4V4K3EIBYFBBJJEDLT26ZUQFSINCMOXVFWVUYYYXPMKQ, seg_pos={0 0}, upload=446487]: Done uploading piece. err: <nil> Dec 26 20:17:43.267959582 (508h48m43.139042617s) [noise=true , address=145.79.227.49:4338 , piece_id=SNSNPRIF45ZYR4625WUVQ3DJEEXSWUY3FG26IRXQUTSRU3EVK4JA, seg_pos={0 0}, upload=446487]: Done uploading piece. err: <nil> Dec 26 20:17:43.289294236 (508h48m43.160377271s) [noise=true , address=145.79.227.2:4114 , piece_id=PMB6A4DRRZJZ22JQTEJNQA57PHBYYVABILPMIMZCTLJ3HMETUWTA, seg_pos={0 0}, upload=446487]: Done uploading piece. err: <nil> Dec 26 20:17:43.290403348 (508h48m43.161486403s) [noise=true , address=145.79.227.36:4517 , piece_id=YGZHYLCPW7LDS7E4P3MQ7KMB57SBRWNDMFFAFBZQZGZBXKI7ZAAA, seg_pos={0 0}, upload=446487]: Done uploading piece. err: <nil> Dec 26 20:17:43.294942792 (508h48m43.166025827s) [noise=true , address=145.79.227.42:4121 , piece_id=CL72TZYQF6DU6RURRBSCGMHFYUDWKQFOPYPNUF3Y3AODOGFONE4A, seg_pos={0 0}, upload=446487]: Done uploading piece. err: <nil> Dec 26 20:17:43.301057566 (508h48m43.172140601s) [noise=true , address=145.79.227.25:4618 , piece_id=3Q7CAENVOQIJ3PUBM24GCENTRSXR2O3RSZ5ST3WFYALLMVRVPL3A, seg_pos={0 0}, upload=446487]: Done uploading piece. err: <nil> Dec 26 20:17:43.301501650 (508h48m43.172584685s) [noise=true , address=145.79.227.46:4601 , piece_id=ZEVL7ISFPLK7332BK65UE6NH6DB5VRSECKF55I665UGXWBY2J4AQ, seg_pos={0 0}, upload=446487]: Done uploading piece. err: <nil> Dec 26 20:17:43.319872011 (508h48m43.190955046s) [noise=true , address=145.79.227.52:4445 , piece_id=KQUT6BOUIER2AHKWLLIIVN6EDMP7AU2KQ5BVQ7JWHNT5G3J5RDNQ, seg_pos={0 0}, upload=446487]: Done uploading piece. err: <nil> Dec 26 20:17:43.512314291 (508h48m43.383397326s) [noise=true , address=45.91.116.7:4012 , piece_id=BUPCFZDXK4JZHO5467ZXKGBMAWOJ4I4X4I26UTUDW4SINITHBPDA, seg_pos={0 0}, upload=446487]: Done uploading piece. err: <nil> Dec 26 20:17:43.530414296 (508h48m43.401497331s) [noise=true , address=77.90.31.10:4518 , piece_id=VDLRHPKBP3FY5YIKLBGUUEU23Y6XRW7FYTU22UC5OKFRDYNSXWJQ, seg_pos={0 0}, upload=446487]: Done uploading piece. err: <nil> Dec 26 20:17:43.571567899 (508h48m43.442650934s) [noise=true , address=45.91.116.5:4300 , piece_id=LORS4EXAA6GA5JQTTKZC6D7QCETROW2CFDODUG7YYNQTHTIIBLHA, seg_pos={0 0}, upload=446487]: Done uploading piece. err: <nil> Dec 26 20:17:44.330593469 (508h48m44.201676504s) [noise=true , address=77.90.31.16:4736 , piece_id=VCI3H7OAZEUZQXGUNWNYRXQSJFL2M3XUZDSR5F65KLF4L35JPHRA, seg_pos={0 0}, upload=446487]: Done uploading piece. err: <nil> Dec 26 20:17:44.330614650 (508h48m44.201697685s) [seg_pos={0 0}, upload=446487]: Segment reached optimal threshold of 13 pieces. Dec 26 20:17:44.330824420 (508h48m44.201907465s) [noise=true , address=145.79.227.30:4762 , piece_id=7PEOUJEEMX4KP63ZFN3EQUGU24DQPZIUFBG3VTQ4LTUCRXE5DCLA, seg_pos={0 0}, upload=446487]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 26 20:17:44.330833887 (508h48m44.201916932s) [noise=true , address=145.79.227.30:4762 , piece_id=7PEOUJEEMX4KP63ZFN3EQUGU24DQPZIUFBG3VTQ4LTUCRXE5DCLA, seg_pos={0 0}, upload=446487]: Long tail cancellation due to optimal threshold Dec 26 20:17:44.330858304 (508h48m44.201941349s) [noise=true , address=145.79.227.39:4013 , piece_id=KCHNWV6IZRTADNLURJYYNI66XEJ7STVIXCKZ25BPYRHKC3SY3K2A, seg_pos={0 0}, upload=446487]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b): protocol: write tcp 188.166.225.170:44622->145.79.227.39:4013: use of closed network connection; write tcp 188.166.225.170:44622->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44622->145.79.227.39:4013: use of closed network connection Dec 26 20:17:44.330868163 (508h48m44.201951208s) [noise=true , address=145.79.227.39:4013 , piece_id=KCHNWV6IZRTADNLURJYYNI66XEJ7STVIXCKZ25BPYRHKC3SY3K2A, seg_pos={0 0}, upload=446487]: Long tail cancellation due to optimal threshold Dec 26 20:17:44.331061250 (508h48m44.202144295s) [noise=true , address=45.91.116.10:4082 , piece_id=INALJ2OJEDH6KNPN422S67RP4XBT5UDTXHCO35LLDZJDJMDSWLPQ, seg_pos={0 0}, upload=446487]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 26 20:17:44.331069176 (508h48m44.202152211s) [noise=true , address=45.91.116.10:4082 , piece_id=INALJ2OJEDH6KNPN422S67RP4XBT5UDTXHCO35LLDZJDJMDSWLPQ, seg_pos={0 0}, upload=446487]: Long tail cancellation due to optimal threshold Dec 26 20:17:44.331087610 (508h48m44.202170655s) [seg_pos={0 0}, upload=446487]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 20:17:44.331159136 (508h48m44.202242181s) [upload=446487]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:17:44.331172072 (508h48m44.202255117s) [upload=446487]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:17:44.617935152 (508h48m44.489018207s) [upload=446487]: Done uploading segments... Dec 26 20:17:44.617959108 (508h48m44.489042143s) [upload=446487]: Upload finished. err: <nil> Duration: 23.810807528s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............X..........................................X.] duration: 36.086186778s trace: https://jaeger.dev.storj.io/trace/663e21624a5a3654 FAIL (2025-12-26 20:18:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, Piece ID: PWVMOEKW5A4ZAVSVJAAHFTYYTQZGFHDF642GWR6EABEGWP2CDRGQ) context deadline exceeded; piecestore: (Node ID: 12whTD1Kvu4T7ANNtkSvDB66uTjVgfPTK9TAzExgvwJYgZ3zSFG, Piece ID: ME2LDK2QJI7LLG67ODDIJDKB5CAT47YDGMJ3CTVK4F6UDHJ4TPOQ) context deadline exceeded; piecestore: (Node ID: 12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH, Piece ID: HJGRPPEJK75YTIKBKNVK7J4TDRKBCNL7HA4JNMOKD2ROGC7C4P3Q) context deadline exceeded ======LOGS====== ) [noise=true , address=145.79.227.7:4296 , piece_id=PVH6VN4CG4O2DC22G74US7IC5GDIGM2KYYYQQPFO5MG22JQCAU2A, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.248055070 (508h48m1.119138115s) [noise=true , address=145.79.227.40:4078 , piece_id=J6TGA3SFQOLGL2C2TS352TSLZBMI6MDXU5BDTTGQYMN4A5QG462A, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.248175840 (508h48m1.119258875s) [noise=true , address=77.90.31.12:4600 , piece_id=JESM5JF4GWUDLEW4YT5SKHJOXD54TWRQBNGGQCBMDSS54XR4AEKQ, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.248604496 (508h48m1.119687531s) [noise=true , address=45.91.116.6:4985 , piece_id=WV5SXZIBMXB4FTNZFE4PKBR5ZWZTUUPADPRMXAVE4Z3HKDGLBOSQ, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.248640474 (508h48m1.119723509s) [noise=true , address=145.79.227.18:4021 , piece_id=ER7GXC5OHAPNGU37QTKPROKB4YTCDQBUHVEIEUH2K5FQ4X5AGM2Q, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.249180202 (508h48m1.120263238s) [noise=true , address=77.90.31.18:4821 , piece_id=HJGRPPEJK75YTIKBKNVK7J4TDRKBCNL7HA4JNMOKD2ROGC7C4P3Q, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.249851130 (508h48m1.120934175s) [noise=true , address=64.49.13.169:4709 , piece_id=ZGWBPAYFOQWU2LSTILYMVRNKGSP53UXEQHRW4NJEXW5OLNVCCWRQ, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.250638910 (508h48m1.121721955s) [noise=true , address=145.79.227.38:4001 , piece_id=TWPCSZEWSFC2ZT7QNWHRS3RY42LLBPIEF3EIQQ65OXFZSHWAV2GA, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.251000769 (508h48m1.122083814s) [noise=true , address=145.79.227.30:4792 , piece_id=RLEUVLBAOMJSDAYHMQJGKYQUC6NYYGSGUVIU5Z53XGBINIQV43GQ, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.251314737 (508h48m1.122397772s) [noise=true , address=145.79.227.5:4252 , piece_id=FI5RNNRKSIYHR3JKCOW3THJQKHAPYLUDDHT3Y4TRYU3OSMCSWQQQ, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.251635527 (508h48m1.122718572s) [noise=true , address=77.90.31.11:4588 , piece_id=ZWDOVL5PGZ4YPY6NZVVSBSWXDDD3DAJ4RUK2BWMOGTHDGWJ2G4DA, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.251919499 (508h48m1.123002544s) [noise=true , address=77.90.31.10:4536 , piece_id=V43VRDKXTE2PHOBPSQUIJPKMKY3ROAEQAI2D4XIJHOLQ3UIMN4RA, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.252236542 (508h48m1.123319588s) [noise=true , address=77.90.31.21:4894 , piece_id=AGL6GVJX2YLGHSZX33SUURUU5S22CF3YRMJTP3QMHHP7OKX2MFSA, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.252528137 (508h48m1.123611182s) [noise=true , address=77.90.31.15:4702 , piece_id=KJEPR7NMX4UHQZHTMDDJKBRLLQ3O5BLJSLYBXGCOFGP3LNC3FXGA, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.252861402 (508h48m1.123944447s) [noise=true , address=64.49.13.27:4961 , piece_id=XCVH62CEVRYGR53U5VG4L4HJMBJSQ562L4QVKTXZSC7XGAEB4LDQ, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:02.570977410 (508h48m2.442060455s) [upload=446477]: Next returned nil segment Dec 26 20:17:02.571003019 (508h48m2.442086054s) [upload=446477]: Waiting for error group managing segments... Dec 26 20:17:02.949353997 (508h48m2.820437042s) [noise=true , address=64.49.13.27:4961 , piece_id=XCVH62CEVRYGR53U5VG4L4HJMBJSQ562L4QVKTXZSC7XGAEB4LDQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.136546190 (508h48m3.007629235s) [noise=true , address=145.79.227.5:4252 , piece_id=FI5RNNRKSIYHR3JKCOW3THJQKHAPYLUDDHT3Y4TRYU3OSMCSWQQQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.155760468 (508h48m3.026843513s) [noise=true , address=145.79.227.38:4001 , piece_id=TWPCSZEWSFC2ZT7QNWHRS3RY42LLBPIEF3EIQQ65OXFZSHWAV2GA, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.179239646 (508h48m3.050322691s) [noise=true , address=145.79.227.18:4021 , piece_id=ER7GXC5OHAPNGU37QTKPROKB4YTCDQBUHVEIEUH2K5FQ4X5AGM2Q, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.372384563 (508h48m3.243467608s) [noise=true , address=77.90.31.15:4702 , piece_id=KJEPR7NMX4UHQZHTMDDJKBRLLQ3O5BLJSLYBXGCOFGP3LNC3FXGA, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.447635803 (508h48m3.318718848s) [noise=true , address=77.90.31.18:4821 , piece_id=HJGRPPEJK75YTIKBKNVK7J4TDRKBCNL7HA4JNMOKD2ROGC7C4P3Q, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.452148166 (508h48m3.323231201s) [noise=true , address=77.90.31.10:4536 , piece_id=V43VRDKXTE2PHOBPSQUIJPKMKY3ROAEQAI2D4XIJHOLQ3UIMN4RA, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.644263002 (508h48m3.515346047s) [noise=true , address=77.90.31.14:4689 , piece_id=PWVMOEKW5A4ZAVSVJAAHFTYYTQZGFHDF642GWR6EABEGWP2CDRGQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.710620497 (508h48m3.581703542s) [noise=true , address=45.91.116.8:4074 , piece_id=ME2LDK2QJI7LLG67ODDIJDKB5CAT47YDGMJ3CTVK4F6UDHJ4TPOQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:04.066910008 (508h48m3.937993053s) [noise=true , address=77.90.31.12:4600 , piece_id=JESM5JF4GWUDLEW4YT5SKHJOXD54TWRQBNGGQCBMDSS54XR4AEKQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:04.082282349 (508h48m3.953365394s) [noise=true , address=145.79.227.44:4219 , piece_id=KZVV2NJBIU22C4URLBZMQVQN4IZ2ZMQ5YODRJMCS6XYXULHCZWLQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:04.089048724 (508h48m3.960131759s) [noise=true , address=145.79.227.30:4792 , piece_id=RLEUVLBAOMJSDAYHMQJGKYQUC6NYYGSGUVIU5Z53XGBINIQV43GQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:04.288772749 (508h48m4.159855794s) [noise=true , address=77.90.31.21:4894 , piece_id=AGL6GVJX2YLGHSZX33SUURUU5S22CF3YRMJTP3QMHHP7OKX2MFSA, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:04.344183239 (508h48m4.215266294s) [noise=true , address=77.90.31.11:4588 , piece_id=ZWDOVL5PGZ4YPY6NZVVSBSWXDDD3DAJ4RUK2BWMOGTHDGWJ2G4DA, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:04.344208728 (508h48m4.215291773s) [seg_pos={0 0}, upload=446477]: Segment reached optimal threshold of 14 pieces. Dec 26 20:17:04.344470828 (508h48m4.215553873s) [noise=true , address=64.49.13.169:4709 , piece_id=ZGWBPAYFOQWU2LSTILYMVRNKGSP53UXEQHRW4NJEXW5OLNVCCWRQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:47062->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:47062->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47062->64.49.13.169:4709: use of closed network connection Dec 26 20:17:04.344481918 (508h48m4.215564953s) [noise=true , address=64.49.13.169:4709 , piece_id=ZGWBPAYFOQWU2LSTILYMVRNKGSP53UXEQHRW4NJEXW5OLNVCCWRQ, seg_pos={0 0}, upload=446477]: Long tail cancellation due to optimal threshold Dec 26 20:17:04.344541813 (508h48m4.215624848s) [noise=true , address=145.79.227.40:4078 , piece_id=J6TGA3SFQOLGL2C2TS352TSLZBMI6MDXU5BDTTGQYMN4A5QG462A, seg_pos={0 0}, upload=446477]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tzgtenV6us5ssD7xWGpZmVUoWEfbFhBNewbK9irHtZPkyDtRd): protocol: expected piece hash; context canceled; EOF Dec 26 20:17:04.344559476 (508h48m4.215642511s) [noise=true , address=145.79.227.40:4078 , piece_id=J6TGA3SFQOLGL2C2TS352TSLZBMI6MDXU5BDTTGQYMN4A5QG462A, seg_pos={0 0}, upload=446477]: Long tail cancellation due to optimal threshold Dec 26 20:17:04.344755479 (508h48m4.215838524s) [noise=true , address=145.79.227.7:4296 , piece_id=PVH6VN4CG4O2DC22G74US7IC5GDIGM2KYYYQQPFO5MG22JQCAU2A, seg_pos={0 0}, upload=446477]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cKhDbA4ptwux2LYNPNAJ7Qh8jECsGWd1KWaK6T6pD38bJWdt1): protocol: write tcp 188.166.225.170:48104->145.79.227.7:4296: use of closed network connection; write tcp 188.166.225.170:48104->145.79.227.7:4296: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48104->145.79.227.7:4296: use of closed network connection Dec 26 20:17:04.344763044 (508h48m4.215846099s) [noise=true , address=145.79.227.7:4296 , piece_id=PVH6VN4CG4O2DC22G74US7IC5GDIGM2KYYYQQPFO5MG22JQCAU2A, seg_pos={0 0}, upload=446477]: Long tail cancellation due to optimal threshold Dec 26 20:17:04.344928509 (508h48m4.216011544s) [noise=true , address=45.91.116.6:4985 , piece_id=WV5SXZIBMXB4FTNZFE4PKBR5ZWZTUUPADPRMXAVE4Z3HKDGLBOSQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RNs9MM9jDtNC2gZC63SYsePZTd5mE7dqUm9A5Z5QWdPX8PVnp): protocol: expected piece hash; context canceled; EOF Dec 26 20:17:04.344938277 (508h48m4.216021312s) [noise=true , address=45.91.116.6:4985 , piece_id=WV5SXZIBMXB4FTNZFE4PKBR5ZWZTUUPADPRMXAVE4Z3HKDGLBOSQ, seg_pos={0 0}, upload=446477]: Long tail cancellation due to optimal threshold Dec 26 20:17:04.344966722 (508h48m4.216049767s) [seg_pos={0 0}, upload=446477]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:17:04.345054548 (508h48m4.216137593s) [upload=446477]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:17:04.345059839 (508h48m4.216142874s) [upload=446477]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:17:04.653613492 (508h48m4.524696537s) [upload=446477]: Done uploading segments... Dec 26 20:17:04.653639822 (508h48m4.524722857s) [upload=446477]: Upload finished. err: <nil> Duration: 36.086200855s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [......X.X.X.X.X..............XX..X..X...........XXX.X.X....X] duration: 30.001590969s trace: https://jaeger.dev.storj.io/trace/40f126ea36ea665d FAIL (2025-12-26 20:18:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: VCI3H7OAZEUZQXGUNWNYRXQSJFL2M3XUZDSR5F65KLF4L35JPHRA) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: BUPCFZDXK4JZHO5467ZXKGBMAWOJ4I4X4I26UTUDW4SINITHBPDA) context deadline exceeded ======LOGS====== 0:17:41.014584898 (508h48m40.885667933s) [upload=446487]: Starting upload Dec 26 20:17:41.014638820 (508h48m40.885721855s) [upload=446487]: Done starting upload Dec 26 20:17:41.014812481 (508h48m40.885895516s) [upload=446487]: Uploading segments... Dec 26 20:17:41.014846826 (508h48m40.885929871s) [upload=446487]: Got next segment. Inline: false Dec 26 20:17:41.014871173 (508h48m40.885954208s) [upload=446487]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 20:17:41.014875401 (508h48m40.885958446s) [upload=446487]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 20:17:41.257281529 (508h48m41.128364574s) [seg_pos={0 0}, upload=446487]: Begin upload segment... Dec 26 20:17:41.257426135 (508h48m41.12850918s) [seg_pos={0 0}, upload=446487]: Done begin upload segment. Dec 26 20:17:41.257456963 (508h48m41.128540018s) [noise=true , address=145.79.227.46:4601 , piece_id=ZEVL7ISFPLK7332BK65UE6NH6DB5VRSECKF55I665UGXWBY2J4AQ, seg_pos={0 0}, upload=446487]: Uploading piece... Dec 26 20:17:41.257512720 (508h48m41.128595755s) [noise=true , address=145.79.227.42:4121 , piece_id=CL72TZYQF6DU6RURRBSCGMHFYUDWKQFOPYPNUF3Y3AODOGFONE4A, seg_pos={0 0}, upload=446487]: Uploading piece... Dec 26 20:17:41.258325507 (508h48m41.129408552s) [noise=true , address=145.79.227.36:4517 , piece_id=YGZHYLCPW7LDS7E4P3MQ7KMB57SBRWNDMFFAFBZQZGZBXKI7ZAAA, seg_pos={0 0}, upload=446487]: Uploading piece... Dec 26 20:17:41.259001555 (508h48m41.13008459s) [noise=true , address=145.79.227.49:4338 , piece_id=SNSNPRIF45ZYR4625WUVQ3DJEEXSWUY3FG26IRXQUTSRU3EVK4JA, seg_pos={0 0}, upload=446487]: Uploading piece... Dec 26 20:17:41.259741914 (508h48m41.13082495s) [noise=true , address=145.79.227.21:4475 , piece_id=2IC7FCDV4V4K3EIBYFBBJJEDLT26ZUQFSINCMOXVFWVUYYYXPMKQ, seg_pos={0 0}, upload=446487]: Uploading piece... Dec 26 20:17:41.260312020 (508h48m41.131395065s) [noise=true , address=145.79.227.37:4566 , piece_id=KXEGB7VCYIEH7WSSSJIY3XABIBGNRWHQL4VIMIWOCF57MJ5SGIYQ, seg_pos={0 0}, upload=446487]: Uploading piece... Dec 26 20:17:41.261640609 (508h48m41.132723654s) [noise=true , address=45.91.116.5:4300 , piece_id=LORS4EXAA6GA5JQTTKZC6D7QCETROW2CFDODUG7YYNQTHTIIBLHA, seg_pos={0 0}, upload=446487]: Uploading piece... Dec 26 20:17:41.262376821 (508h48m41.133459856s) [noise=true , address=145.79.227.30:4762 , piece_id=7PEOUJEEMX4KP63ZFN3EQUGU24DQPZIUFBG3VTQ4LTUCRXE5DCLA, seg_pos={0 0}, upload=446487]: Uploading piece... Dec 26 20:17:41.263036998 (508h48m41.134120043s) [noise=true , address=145.79.227.52:4445 , piece_id=KQUT6BOUIER2AHKWLLIIVN6EDMP7AU2KQ5BVQ7JWHNT5G3J5RDNQ, seg_pos={0 0}, upload=446487]: Uploading piece... Dec 26 20:17:41.264209501 (508h48m41.135292546s) [noise=true , address=77.90.31.10:4518 , piece_id=VDLRHPKBP3FY5YIKLBGUUEU23Y6XRW7FYTU22UC5OKFRDYNSXWJQ, seg_pos={0 0}, upload=446487]: Uploading piece... Dec 26 20:17:41.264383082 (508h48m41.135466127s) [noise=true , address=145.79.227.25:4618 , piece_id=3Q7CAENVOQIJ3PUBM24GCENTRSXR2O3RSZ5ST3WFYALLMVRVPL3A, seg_pos={0 0}, upload=446487]: Uploading piece... Dec 26 20:17:41.265461195 (508h48m41.13654423s) [noise=true , address=77.90.31.16:4736 , piece_id=VCI3H7OAZEUZQXGUNWNYRXQSJFL2M3XUZDSR5F65KLF4L35JPHRA, seg_pos={0 0}, upload=446487]: Uploading piece... Dec 26 20:17:41.266725352 (508h48m41.137808397s) [noise=true , address=145.79.227.2:4114 , piece_id=PMB6A4DRRZJZ22JQTEJNQA57PHBYYVABILPMIMZCTLJ3HMETUWTA, seg_pos={0 0}, upload=446487]: Uploading piece... Dec 26 20:17:41.267072442 (508h48m41.138155488s) [noise=true , address=145.79.227.39:4013 , piece_id=KCHNWV6IZRTADNLURJYYNI66XEJ7STVIXCKZ25BPYRHKC3SY3K2A, seg_pos={0 0}, upload=446487]: Uploading piece... Dec 26 20:17:41.268750017 (508h48m41.139833062s) [noise=true , address=45.91.116.10:4082 , piece_id=INALJ2OJEDH6KNPN422S67RP4XBT5UDTXHCO35LLDZJDJMDSWLPQ, seg_pos={0 0}, upload=446487]: Uploading piece... Dec 26 20:17:41.270307242 (508h48m41.141390277s) [noise=true , address=45.91.116.7:4012 , piece_id=BUPCFZDXK4JZHO5467ZXKGBMAWOJ4I4X4I26UTUDW4SINITHBPDA, seg_pos={0 0}, upload=446487]: Uploading piece... Dec 26 20:17:42.971252485 (508h48m42.84233552s) [upload=446487]: Next returned nil segment Dec 26 20:17:42.971265711 (508h48m42.842348746s) [upload=446487]: Waiting for error group managing segments... Dec 26 20:17:43.254289712 (508h48m43.125372747s) [noise=true , address=145.79.227.37:4566 , piece_id=KXEGB7VCYIEH7WSSSJIY3XABIBGNRWHQL4VIMIWOCF57MJ5SGIYQ, seg_pos={0 0}, upload=446487]: Done uploading piece. err: <nil> Dec 26 20:17:43.266691186 (508h48m43.137774231s) [noise=true , address=145.79.227.21:4475 , piece_id=2IC7FCDV4V4K3EIBYFBBJJEDLT26ZUQFSINCMOXVFWVUYYYXPMKQ, seg_pos={0 0}, upload=446487]: Done uploading piece. err: <nil> Dec 26 20:17:43.267959582 (508h48m43.139042617s) [noise=true , address=145.79.227.49:4338 , piece_id=SNSNPRIF45ZYR4625WUVQ3DJEEXSWUY3FG26IRXQUTSRU3EVK4JA, seg_pos={0 0}, upload=446487]: Done uploading piece. err: <nil> Dec 26 20:17:43.289294236 (508h48m43.160377271s) [noise=true , address=145.79.227.2:4114 , piece_id=PMB6A4DRRZJZ22JQTEJNQA57PHBYYVABILPMIMZCTLJ3HMETUWTA, seg_pos={0 0}, upload=446487]: Done uploading piece. err: <nil> Dec 26 20:17:43.290403348 (508h48m43.161486403s) [noise=true , address=145.79.227.36:4517 , piece_id=YGZHYLCPW7LDS7E4P3MQ7KMB57SBRWNDMFFAFBZQZGZBXKI7ZAAA, seg_pos={0 0}, upload=446487]: Done uploading piece. err: <nil> Dec 26 20:17:43.294942792 (508h48m43.166025827s) [noise=true , address=145.79.227.42:4121 , piece_id=CL72TZYQF6DU6RURRBSCGMHFYUDWKQFOPYPNUF3Y3AODOGFONE4A, seg_pos={0 0}, upload=446487]: Done uploading piece. err: <nil> Dec 26 20:17:43.301057566 (508h48m43.172140601s) [noise=true , address=145.79.227.25:4618 , piece_id=3Q7CAENVOQIJ3PUBM24GCENTRSXR2O3RSZ5ST3WFYALLMVRVPL3A, seg_pos={0 0}, upload=446487]: Done uploading piece. err: <nil> Dec 26 20:17:43.301501650 (508h48m43.172584685s) [noise=true , address=145.79.227.46:4601 , piece_id=ZEVL7ISFPLK7332BK65UE6NH6DB5VRSECKF55I665UGXWBY2J4AQ, seg_pos={0 0}, upload=446487]: Done uploading piece. err: <nil> Dec 26 20:17:43.319872011 (508h48m43.190955046s) [noise=true , address=145.79.227.52:4445 , piece_id=KQUT6BOUIER2AHKWLLIIVN6EDMP7AU2KQ5BVQ7JWHNT5G3J5RDNQ, seg_pos={0 0}, upload=446487]: Done uploading piece. err: <nil> Dec 26 20:17:43.512314291 (508h48m43.383397326s) [noise=true , address=45.91.116.7:4012 , piece_id=BUPCFZDXK4JZHO5467ZXKGBMAWOJ4I4X4I26UTUDW4SINITHBPDA, seg_pos={0 0}, upload=446487]: Done uploading piece. err: <nil> Dec 26 20:17:43.530414296 (508h48m43.401497331s) [noise=true , address=77.90.31.10:4518 , piece_id=VDLRHPKBP3FY5YIKLBGUUEU23Y6XRW7FYTU22UC5OKFRDYNSXWJQ, seg_pos={0 0}, upload=446487]: Done uploading piece. err: <nil> Dec 26 20:17:43.571567899 (508h48m43.442650934s) [noise=true , address=45.91.116.5:4300 , piece_id=LORS4EXAA6GA5JQTTKZC6D7QCETROW2CFDODUG7YYNQTHTIIBLHA, seg_pos={0 0}, upload=446487]: Done uploading piece. err: <nil> Dec 26 20:17:44.330593469 (508h48m44.201676504s) [noise=true , address=77.90.31.16:4736 , piece_id=VCI3H7OAZEUZQXGUNWNYRXQSJFL2M3XUZDSR5F65KLF4L35JPHRA, seg_pos={0 0}, upload=446487]: Done uploading piece. err: <nil> Dec 26 20:17:44.330614650 (508h48m44.201697685s) [seg_pos={0 0}, upload=446487]: Segment reached optimal threshold of 13 pieces. Dec 26 20:17:44.330824420 (508h48m44.201907465s) [noise=true , address=145.79.227.30:4762 , piece_id=7PEOUJEEMX4KP63ZFN3EQUGU24DQPZIUFBG3VTQ4LTUCRXE5DCLA, seg_pos={0 0}, upload=446487]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 26 20:17:44.330833887 (508h48m44.201916932s) [noise=true , address=145.79.227.30:4762 , piece_id=7PEOUJEEMX4KP63ZFN3EQUGU24DQPZIUFBG3VTQ4LTUCRXE5DCLA, seg_pos={0 0}, upload=446487]: Long tail cancellation due to optimal threshold Dec 26 20:17:44.330858304 (508h48m44.201941349s) [noise=true , address=145.79.227.39:4013 , piece_id=KCHNWV6IZRTADNLURJYYNI66XEJ7STVIXCKZ25BPYRHKC3SY3K2A, seg_pos={0 0}, upload=446487]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b): protocol: write tcp 188.166.225.170:44622->145.79.227.39:4013: use of closed network connection; write tcp 188.166.225.170:44622->145.79.227.39:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44622->145.79.227.39:4013: use of closed network connection Dec 26 20:17:44.330868163 (508h48m44.201951208s) [noise=true , address=145.79.227.39:4013 , piece_id=KCHNWV6IZRTADNLURJYYNI66XEJ7STVIXCKZ25BPYRHKC3SY3K2A, seg_pos={0 0}, upload=446487]: Long tail cancellation due to optimal threshold Dec 26 20:17:44.331061250 (508h48m44.202144295s) [noise=true , address=45.91.116.10:4082 , piece_id=INALJ2OJEDH6KNPN422S67RP4XBT5UDTXHCO35LLDZJDJMDSWLPQ, seg_pos={0 0}, upload=446487]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 26 20:17:44.331069176 (508h48m44.202152211s) [noise=true , address=45.91.116.10:4082 , piece_id=INALJ2OJEDH6KNPN422S67RP4XBT5UDTXHCO35LLDZJDJMDSWLPQ, seg_pos={0 0}, upload=446487]: Long tail cancellation due to optimal threshold Dec 26 20:17:44.331087610 (508h48m44.202170655s) [seg_pos={0 0}, upload=446487]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 20:17:44.331159136 (508h48m44.202242181s) [upload=446487]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:17:44.331172072 (508h48m44.202255117s) [upload=446487]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:17:44.617935152 (508h48m44.489018207s) [upload=446487]: Done uploading segments... Dec 26 20:17:44.617959108 (508h48m44.489042143s) [upload=446487]: Upload finished. err: <nil> Duration: 30.00161196s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [................X..........................................X] duration: 1m0.004745109s trace: https://jaeger.dev.storj.io/trace/663e21624a5a3654 FAIL (2025-12-26 20:18:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, Piece ID: PWVMOEKW5A4ZAVSVJAAHFTYYTQZGFHDF642GWR6EABEGWP2CDRGQ) context deadline exceeded; piecestore: (Node ID: 12whTD1Kvu4T7ANNtkSvDB66uTjVgfPTK9TAzExgvwJYgZ3zSFG, Piece ID: ME2LDK2QJI7LLG67ODDIJDKB5CAT47YDGMJ3CTVK4F6UDHJ4TPOQ) context deadline exceeded; piecestore: (Node ID: 12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH, Piece ID: HJGRPPEJK75YTIKBKNVK7J4TDRKBCNL7HA4JNMOKD2ROGC7C4P3Q) context deadline exceeded ======LOGS====== ) [noise=true , address=145.79.227.7:4296 , piece_id=PVH6VN4CG4O2DC22G74US7IC5GDIGM2KYYYQQPFO5MG22JQCAU2A, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.248055070 (508h48m1.119138115s) [noise=true , address=145.79.227.40:4078 , piece_id=J6TGA3SFQOLGL2C2TS352TSLZBMI6MDXU5BDTTGQYMN4A5QG462A, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.248175840 (508h48m1.119258875s) [noise=true , address=77.90.31.12:4600 , piece_id=JESM5JF4GWUDLEW4YT5SKHJOXD54TWRQBNGGQCBMDSS54XR4AEKQ, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.248604496 (508h48m1.119687531s) [noise=true , address=45.91.116.6:4985 , piece_id=WV5SXZIBMXB4FTNZFE4PKBR5ZWZTUUPADPRMXAVE4Z3HKDGLBOSQ, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.248640474 (508h48m1.119723509s) [noise=true , address=145.79.227.18:4021 , piece_id=ER7GXC5OHAPNGU37QTKPROKB4YTCDQBUHVEIEUH2K5FQ4X5AGM2Q, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.249180202 (508h48m1.120263238s) [noise=true , address=77.90.31.18:4821 , piece_id=HJGRPPEJK75YTIKBKNVK7J4TDRKBCNL7HA4JNMOKD2ROGC7C4P3Q, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.249851130 (508h48m1.120934175s) [noise=true , address=64.49.13.169:4709 , piece_id=ZGWBPAYFOQWU2LSTILYMVRNKGSP53UXEQHRW4NJEXW5OLNVCCWRQ, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.250638910 (508h48m1.121721955s) [noise=true , address=145.79.227.38:4001 , piece_id=TWPCSZEWSFC2ZT7QNWHRS3RY42LLBPIEF3EIQQ65OXFZSHWAV2GA, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.251000769 (508h48m1.122083814s) [noise=true , address=145.79.227.30:4792 , piece_id=RLEUVLBAOMJSDAYHMQJGKYQUC6NYYGSGUVIU5Z53XGBINIQV43GQ, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.251314737 (508h48m1.122397772s) [noise=true , address=145.79.227.5:4252 , piece_id=FI5RNNRKSIYHR3JKCOW3THJQKHAPYLUDDHT3Y4TRYU3OSMCSWQQQ, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.251635527 (508h48m1.122718572s) [noise=true , address=77.90.31.11:4588 , piece_id=ZWDOVL5PGZ4YPY6NZVVSBSWXDDD3DAJ4RUK2BWMOGTHDGWJ2G4DA, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.251919499 (508h48m1.123002544s) [noise=true , address=77.90.31.10:4536 , piece_id=V43VRDKXTE2PHOBPSQUIJPKMKY3ROAEQAI2D4XIJHOLQ3UIMN4RA, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.252236542 (508h48m1.123319588s) [noise=true , address=77.90.31.21:4894 , piece_id=AGL6GVJX2YLGHSZX33SUURUU5S22CF3YRMJTP3QMHHP7OKX2MFSA, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.252528137 (508h48m1.123611182s) [noise=true , address=77.90.31.15:4702 , piece_id=KJEPR7NMX4UHQZHTMDDJKBRLLQ3O5BLJSLYBXGCOFGP3LNC3FXGA, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:01.252861402 (508h48m1.123944447s) [noise=true , address=64.49.13.27:4961 , piece_id=XCVH62CEVRYGR53U5VG4L4HJMBJSQ562L4QVKTXZSC7XGAEB4LDQ, seg_pos={0 0}, upload=446477]: Uploading piece... Dec 26 20:17:02.570977410 (508h48m2.442060455s) [upload=446477]: Next returned nil segment Dec 26 20:17:02.571003019 (508h48m2.442086054s) [upload=446477]: Waiting for error group managing segments... Dec 26 20:17:02.949353997 (508h48m2.820437042s) [noise=true , address=64.49.13.27:4961 , piece_id=XCVH62CEVRYGR53U5VG4L4HJMBJSQ562L4QVKTXZSC7XGAEB4LDQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.136546190 (508h48m3.007629235s) [noise=true , address=145.79.227.5:4252 , piece_id=FI5RNNRKSIYHR3JKCOW3THJQKHAPYLUDDHT3Y4TRYU3OSMCSWQQQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.155760468 (508h48m3.026843513s) [noise=true , address=145.79.227.38:4001 , piece_id=TWPCSZEWSFC2ZT7QNWHRS3RY42LLBPIEF3EIQQ65OXFZSHWAV2GA, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.179239646 (508h48m3.050322691s) [noise=true , address=145.79.227.18:4021 , piece_id=ER7GXC5OHAPNGU37QTKPROKB4YTCDQBUHVEIEUH2K5FQ4X5AGM2Q, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.372384563 (508h48m3.243467608s) [noise=true , address=77.90.31.15:4702 , piece_id=KJEPR7NMX4UHQZHTMDDJKBRLLQ3O5BLJSLYBXGCOFGP3LNC3FXGA, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.447635803 (508h48m3.318718848s) [noise=true , address=77.90.31.18:4821 , piece_id=HJGRPPEJK75YTIKBKNVK7J4TDRKBCNL7HA4JNMOKD2ROGC7C4P3Q, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.452148166 (508h48m3.323231201s) [noise=true , address=77.90.31.10:4536 , piece_id=V43VRDKXTE2PHOBPSQUIJPKMKY3ROAEQAI2D4XIJHOLQ3UIMN4RA, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.644263002 (508h48m3.515346047s) [noise=true , address=77.90.31.14:4689 , piece_id=PWVMOEKW5A4ZAVSVJAAHFTYYTQZGFHDF642GWR6EABEGWP2CDRGQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:03.710620497 (508h48m3.581703542s) [noise=true , address=45.91.116.8:4074 , piece_id=ME2LDK2QJI7LLG67ODDIJDKB5CAT47YDGMJ3CTVK4F6UDHJ4TPOQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:04.066910008 (508h48m3.937993053s) [noise=true , address=77.90.31.12:4600 , piece_id=JESM5JF4GWUDLEW4YT5SKHJOXD54TWRQBNGGQCBMDSS54XR4AEKQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:04.082282349 (508h48m3.953365394s) [noise=true , address=145.79.227.44:4219 , piece_id=KZVV2NJBIU22C4URLBZMQVQN4IZ2ZMQ5YODRJMCS6XYXULHCZWLQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:04.089048724 (508h48m3.960131759s) [noise=true , address=145.79.227.30:4792 , piece_id=RLEUVLBAOMJSDAYHMQJGKYQUC6NYYGSGUVIU5Z53XGBINIQV43GQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:04.288772749 (508h48m4.159855794s) [noise=true , address=77.90.31.21:4894 , piece_id=AGL6GVJX2YLGHSZX33SUURUU5S22CF3YRMJTP3QMHHP7OKX2MFSA, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:04.344183239 (508h48m4.215266294s) [noise=true , address=77.90.31.11:4588 , piece_id=ZWDOVL5PGZ4YPY6NZVVSBSWXDDD3DAJ4RUK2BWMOGTHDGWJ2G4DA, seg_pos={0 0}, upload=446477]: Done uploading piece. err: <nil> Dec 26 20:17:04.344208728 (508h48m4.215291773s) [seg_pos={0 0}, upload=446477]: Segment reached optimal threshold of 14 pieces. Dec 26 20:17:04.344470828 (508h48m4.215553873s) [noise=true , address=64.49.13.169:4709 , piece_id=ZGWBPAYFOQWU2LSTILYMVRNKGSP53UXEQHRW4NJEXW5OLNVCCWRQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:47062->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:47062->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47062->64.49.13.169:4709: use of closed network connection Dec 26 20:17:04.344481918 (508h48m4.215564953s) [noise=true , address=64.49.13.169:4709 , piece_id=ZGWBPAYFOQWU2LSTILYMVRNKGSP53UXEQHRW4NJEXW5OLNVCCWRQ, seg_pos={0 0}, upload=446477]: Long tail cancellation due to optimal threshold Dec 26 20:17:04.344541813 (508h48m4.215624848s) [noise=true , address=145.79.227.40:4078 , piece_id=J6TGA3SFQOLGL2C2TS352TSLZBMI6MDXU5BDTTGQYMN4A5QG462A, seg_pos={0 0}, upload=446477]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tzgtenV6us5ssD7xWGpZmVUoWEfbFhBNewbK9irHtZPkyDtRd): protocol: expected piece hash; context canceled; EOF Dec 26 20:17:04.344559476 (508h48m4.215642511s) [noise=true , address=145.79.227.40:4078 , piece_id=J6TGA3SFQOLGL2C2TS352TSLZBMI6MDXU5BDTTGQYMN4A5QG462A, seg_pos={0 0}, upload=446477]: Long tail cancellation due to optimal threshold Dec 26 20:17:04.344755479 (508h48m4.215838524s) [noise=true , address=145.79.227.7:4296 , piece_id=PVH6VN4CG4O2DC22G74US7IC5GDIGM2KYYYQQPFO5MG22JQCAU2A, seg_pos={0 0}, upload=446477]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cKhDbA4ptwux2LYNPNAJ7Qh8jECsGWd1KWaK6T6pD38bJWdt1): protocol: write tcp 188.166.225.170:48104->145.79.227.7:4296: use of closed network connection; write tcp 188.166.225.170:48104->145.79.227.7:4296: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48104->145.79.227.7:4296: use of closed network connection Dec 26 20:17:04.344763044 (508h48m4.215846099s) [noise=true , address=145.79.227.7:4296 , piece_id=PVH6VN4CG4O2DC22G74US7IC5GDIGM2KYYYQQPFO5MG22JQCAU2A, seg_pos={0 0}, upload=446477]: Long tail cancellation due to optimal threshold Dec 26 20:17:04.344928509 (508h48m4.216011544s) [noise=true , address=45.91.116.6:4985 , piece_id=WV5SXZIBMXB4FTNZFE4PKBR5ZWZTUUPADPRMXAVE4Z3HKDGLBOSQ, seg_pos={0 0}, upload=446477]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RNs9MM9jDtNC2gZC63SYsePZTd5mE7dqUm9A5Z5QWdPX8PVnp): protocol: expected piece hash; context canceled; EOF Dec 26 20:17:04.344938277 (508h48m4.216021312s) [noise=true , address=45.91.116.6:4985 , piece_id=WV5SXZIBMXB4FTNZFE4PKBR5ZWZTUUPADPRMXAVE4Z3HKDGLBOSQ, seg_pos={0 0}, upload=446477]: Long tail cancellation due to optimal threshold Dec 26 20:17:04.344966722 (508h48m4.216049767s) [seg_pos={0 0}, upload=446477]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:17:04.345054548 (508h48m4.216137593s) [upload=446477]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:17:04.345059839 (508h48m4.216142874s) [upload=446477]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:17:04.653613492 (508h48m4.524696537s) [upload=446477]: Done uploading segments... Dec 26 20:17:04.653639822 (508h48m4.524722857s) [upload=446477]: Upload finished. err: <nil> Duration: 1m0.004769025s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [X......X.X.X.X.X..............XX..X..X...........XXX.X.X....] duration: 7.632827816s trace: https://jaeger.dev.storj.io/trace/2deb4e2f5723983e FAIL (2025-12-26 20:16:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== c 26 20:14:34.361029973 (508h45m34.232113008s) [upload=446440]: Starting upload Dec 26 20:14:34.361150823 (508h45m34.232233928s) [upload=446440]: Done starting upload Dec 26 20:14:34.361343490 (508h45m34.232426525s) [upload=446440]: Uploading segments... Dec 26 20:14:34.361537168 (508h45m34.232620203s) [upload=446440]: Got next segment. Inline: false Dec 26 20:14:34.361747921 (508h45m34.232830956s) [upload=446440]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 20:14:34.361754172 (508h45m34.232837207s) [upload=446440]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 20:14:34.622967101 (508h45m34.494050136s) [seg_pos={0 0}, upload=446440]: Begin upload segment... Dec 26 20:14:34.623013139 (508h45m34.494096174s) [seg_pos={0 0}, upload=446440]: Done begin upload segment. Dec 26 20:14:34.623037004 (508h45m34.494120039s) [noise=true , address=145.79.227.45:4231 , piece_id=SXTYW3PLFDECYMRHYVTA6L5N6VI6KZEILG26K4ALKBEE3JDWJFPQ, seg_pos={0 0}, upload=446440]: Uploading piece... Dec 26 20:14:34.623139539 (508h45m34.494222584s) [noise=true , address=145.79.227.35:4482 , piece_id=R67ESFGWQNGDORGMN5OZXPFKYSWN4U2RDWOXDUZN5JYNW5DGBTSQ, seg_pos={0 0}, upload=446440]: Uploading piece... Dec 26 20:14:34.623677394 (508h45m34.494760439s) [noise=true , address=145.79.227.37:4566 , piece_id=PWFS52EYRFKEU6LERET3W2QXB6ZSFICUOV3GPDUMEHKY3IAY43UA, seg_pos={0 0}, upload=446440]: Uploading piece... Dec 26 20:14:34.624155955 (508h45m34.495239s) [noise=true , address=145.79.227.3:4150 , piece_id=2E222HDKHDP3JIRHBSZOPKKO74JIBJ6D43ZCV7PU6YLNJYETJHSQ, seg_pos={0 0}, upload=446440]: Uploading piece... Dec 26 20:14:34.624517764 (508h45m34.495600809s) [noise=true , address=145.79.227.42:4121 , piece_id=DNTY3PS7VF4DFKS75BEETF4IWLZW73ULTZ555US7OK4CPJJFULIQ, seg_pos={0 0}, upload=446440]: Uploading piece... Dec 26 20:14:34.624595833 (508h45m34.495678868s) [noise=true , address=145.79.227.15:4875 , piece_id=BT2HGGDEPGOVFUSERULJCM3ZNYK4QLH24ONVVJPI4BAXWAVUDWRA, seg_pos={0 0}, upload=446440]: Uploading piece... Dec 26 20:14:34.624958985 (508h45m34.49604203s) [noise=true , address=45.91.116.6:4975 , piece_id=7IRHTKTPFRL2JAPC3XCL3ZVHTUC22SBYDP2KVL6NTUVKMF6NWYRA, seg_pos={0 0}, upload=446440]: Uploading piece... Dec 26 20:14:34.625014510 (508h45m34.496097545s) [noise=true , address=145.79.227.41:4086 , piece_id=7SZPGLQGXXFNANI65M3BS756HJPRNNKDE3R2DSAMCVZXSH5YYXAQ, seg_pos={0 0}, upload=446440]: Uploading piece... Dec 26 20:14:34.625339408 (508h45m34.496422453s) [noise=true , address=145.79.227.25:4618 , piece_id=7H5GZ2TFX35CUSEH6LVYVILR34W2XDGPNWTJ5M3UIPP727L2TRHQ, seg_pos={0 0}, upload=446440]: Uploading piece... Dec 26 20:14:34.625348947 (508h45m34.496431982s) [noise=true , address=145.79.227.46:4602 , piece_id=QKSMTCTQV25F2DEQH2D2KADAINYCOJWFK33T6NVPIG2B37YF6TTQ, seg_pos={0 0}, upload=446440]: Uploading piece... Dec 26 20:14:34.625713531 (508h45m34.496796576s) [noise=true , address=45.91.116.7:4012 , piece_id=ROQL7PKQIQIKYVX74QAXLETAM4MRN5V4ELUK6CH2ZKKAOSSOGBBA, seg_pos={0 0}, upload=446440]: Uploading piece... Dec 26 20:14:34.625757023 (508h45m34.496840058s) [noise=true , address=45.91.116.14:4189 , piece_id=65A6IHIAES37HSTNBM5FUPZZAH35WFSLH6S3TBIII7O62VEQQ25A, seg_pos={0 0}, upload=446440]: Uploading piece... Dec 26 20:14:34.626124824 (508h45m34.497207859s) [noise=true , address=45.91.116.8:4045 , piece_id=3FYKMIDJTCLANIBL6QNXRIW4ELDG5OSSQUNMAYR5ECMBPSYW2CIQ, seg_pos={0 0}, upload=446440]: Uploading piece... Dec 26 20:14:34.626171092 (508h45m34.497254127s) [noise=true , address=77.90.31.21:4877 , piece_id=MRY3PPVMCVH7WP667T3ENAWH6TRZNMNGK4ZWGOZETOY5AGE7RUZA, seg_pos={0 0}, upload=446440]: Uploading piece... Dec 26 20:14:34.626655414 (508h45m34.497738449s) [noise=true , address=45.91.116.9:4263 , piece_id=GHLDY5TXPAJFLMLMPYPJ2MFBNDUNPKMOP7I23CATXKXWQOQT5WWQ, seg_pos={0 0}, upload=446440]: Uploading piece... Dec 26 20:14:34.628204384 (508h45m34.499287419s) [noise=true , address=145.79.227.49:4338 , piece_id=MZL5YI7C7MLDE2CB47SQS2XWPLJ5VUDVWIAO2HO67O236NLZO2FA, seg_pos={0 0}, upload=446440]: Uploading piece... Dec 26 20:14:36.339473084 (508h45m36.210556119s) [upload=446440]: Next returned nil segment Dec 26 20:14:36.339483454 (508h45m36.210566489s) [upload=446440]: Waiting for error group managing segments... Dec 26 20:14:36.614251004 (508h45m36.485334049s) [noise=true , address=145.79.227.41:4086 , piece_id=7SZPGLQGXXFNANI65M3BS756HJPRNNKDE3R2DSAMCVZXSH5YYXAQ, seg_pos={0 0}, upload=446440]: Done uploading piece. err: <nil> Dec 26 20:14:36.625627025 (508h45m36.49671007s) [noise=true , address=145.79.227.45:4231 , piece_id=SXTYW3PLFDECYMRHYVTA6L5N6VI6KZEILG26K4ALKBEE3JDWJFPQ, seg_pos={0 0}, upload=446440]: Done uploading piece. err: <nil> Dec 26 20:14:36.628714234 (508h45m36.499797269s) [noise=true , address=145.79.227.42:4121 , piece_id=DNTY3PS7VF4DFKS75BEETF4IWLZW73ULTZ555US7OK4CPJJFULIQ, seg_pos={0 0}, upload=446440]: Done uploading piece. err: <nil> Dec 26 20:14:36.632592198 (508h45m36.503675233s) [noise=true , address=145.79.227.46:4602 , piece_id=QKSMTCTQV25F2DEQH2D2KADAINYCOJWFK33T6NVPIG2B37YF6TTQ, seg_pos={0 0}, upload=446440]: Done uploading piece. err: <nil> Dec 26 20:14:36.634769304 (508h45m36.505852339s) [noise=true , address=145.79.227.25:4618 , piece_id=7H5GZ2TFX35CUSEH6LVYVILR34W2XDGPNWTJ5M3UIPP727L2TRHQ, seg_pos={0 0}, upload=446440]: Done uploading piece. err: <nil> Dec 26 20:14:36.635827759 (508h45m36.506910794s) [noise=true , address=145.79.227.49:4338 , piece_id=MZL5YI7C7MLDE2CB47SQS2XWPLJ5VUDVWIAO2HO67O236NLZO2FA, seg_pos={0 0}, upload=446440]: Done uploading piece. err: <nil> Dec 26 20:14:36.639557481 (508h45m36.510640516s) [noise=true , address=145.79.227.37:4566 , piece_id=PWFS52EYRFKEU6LERET3W2QXB6ZSFICUOV3GPDUMEHKY3IAY43UA, seg_pos={0 0}, upload=446440]: Done uploading piece. err: <nil> Dec 26 20:14:36.651231290 (508h45m36.522314325s) [noise=true , address=145.79.227.3:4150 , piece_id=2E222HDKHDP3JIRHBSZOPKKO74JIBJ6D43ZCV7PU6YLNJYETJHSQ, seg_pos={0 0}, upload=446440]: Done uploading piece. err: <nil> Dec 26 20:14:36.824853844 (508h45m36.695936879s) [noise=true , address=145.79.227.15:4875 , piece_id=BT2HGGDEPGOVFUSERULJCM3ZNYK4QLH24ONVVJPI4BAXWAVUDWRA, seg_pos={0 0}, upload=446440]: Done uploading piece. err: <nil> Dec 26 20:14:36.915641057 (508h45m36.786724112s) [noise=true , address=45.91.116.9:4263 , piece_id=GHLDY5TXPAJFLMLMPYPJ2MFBNDUNPKMOP7I23CATXKXWQOQT5WWQ, seg_pos={0 0}, upload=446440]: Done uploading piece. err: <nil> Dec 26 20:14:36.955746433 (508h45m36.826829468s) [noise=true , address=45.91.116.14:4189 , piece_id=65A6IHIAES37HSTNBM5FUPZZAH35WFSLH6S3TBIII7O62VEQQ25A, seg_pos={0 0}, upload=446440]: Done uploading piece. err: <nil> Dec 26 20:14:36.964794603 (508h45m36.835877638s) [noise=true , address=45.91.116.7:4012 , piece_id=ROQL7PKQIQIKYVX74QAXLETAM4MRN5V4ELUK6CH2ZKKAOSSOGBBA, seg_pos={0 0}, upload=446440]: Done uploading piece. err: <nil> Dec 26 20:14:36.973931722 (508h45m36.845014757s) [noise=true , address=77.90.31.21:4877 , piece_id=MRY3PPVMCVH7WP667T3ENAWH6TRZNMNGK4ZWGOZETOY5AGE7RUZA, seg_pos={0 0}, upload=446440]: Done uploading piece. err: <nil> Dec 26 20:14:36.973953122 (508h45m36.845036157s) [seg_pos={0 0}, upload=446440]: Segment reached optimal threshold of 13 pieces. Dec 26 20:14:36.974139407 (508h45m36.845222442s) [noise=true , address=45.91.116.6:4975 , piece_id=7IRHTKTPFRL2JAPC3XCL3ZVHTUC22SBYDP2KVL6NTUVKMF6NWYRA, seg_pos={0 0}, upload=446440]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: write tcp 188.166.225.170:60138->45.91.116.6:4975: use of closed network connection; write tcp 188.166.225.170:60138->45.91.116.6:4975: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60138->45.91.116.6:4975: use of closed network connection Dec 26 20:14:36.974145839 (508h45m36.845228875s) [noise=true , address=45.91.116.6:4975 , piece_id=7IRHTKTPFRL2JAPC3XCL3ZVHTUC22SBYDP2KVL6NTUVKMF6NWYRA, seg_pos={0 0}, upload=446440]: Long tail cancellation due to optimal threshold Dec 26 20:14:36.974286838 (508h45m36.845369873s) [noise=true , address=145.79.227.35:4482 , piece_id=R67ESFGWQNGDORGMN5OZXPFKYSWN4U2RDWOXDUZN5JYNW5DGBTSQ, seg_pos={0 0}, upload=446440]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: expected piece hash; context canceled; EOF Dec 26 20:14:36.974292198 (508h45m36.845375233s) [noise=true , address=145.79.227.35:4482 , piece_id=R67ESFGWQNGDORGMN5OZXPFKYSWN4U2RDWOXDUZN5JYNW5DGBTSQ, seg_pos={0 0}, upload=446440]: Long tail cancellation due to optimal threshold Dec 26 20:14:36.974609763 (508h45m36.845693409s) [noise=true , address=45.91.116.8:4045 , piece_id=3FYKMIDJTCLANIBL6QNXRIW4ELDG5OSSQUNMAYR5ECMBPSYW2CIQ, seg_pos={0 0}, upload=446440]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 26 20:14:36.974632897 (508h45m36.845715962s) [noise=true , address=45.91.116.8:4045 , piece_id=3FYKMIDJTCLANIBL6QNXRIW4ELDG5OSSQUNMAYR5ECMBPSYW2CIQ, seg_pos={0 0}, upload=446440]: Long tail cancellation due to optimal threshold Dec 26 20:14:36.974668805 (508h45m36.84575184s) [seg_pos={0 0}, upload=446440]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 20:14:36.974801618 (508h45m36.845884653s) [upload=446440]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:14:36.974810725 (508h45m36.845893761s) [upload=446440]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:14:37.282277490 (508h45m37.153360525s) [upload=446440]: Done uploading segments... Dec 26 20:14:37.282296936 (508h45m37.153379971s) [upload=446440]: Upload finished. err: <nil> Duration: 7.632855148s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [XX......X.X.X.X.X..............XX..X..X...........XXX.X.X...] duration: 7.250034587s trace: https://jaeger.dev.storj.io/trace/2deb4e2f5723983e FAIL (2025-12-26 20:16:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== c 26 20:14:34.361029973 (508h45m34.232113008s) [upload=446440]: Starting upload Dec 26 20:14:34.361150823 (508h45m34.232233928s) [upload=446440]: Done starting upload Dec 26 20:14:34.361343490 (508h45m34.232426525s) [upload=446440]: Uploading segments... Dec 26 20:14:34.361537168 (508h45m34.232620203s) [upload=446440]: Got next segment. Inline: false Dec 26 20:14:34.361747921 (508h45m34.232830956s) [upload=446440]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 20:14:34.361754172 (508h45m34.232837207s) [upload=446440]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 20:14:34.622967101 (508h45m34.494050136s) [seg_pos={0 0}, upload=446440]: Begin upload segment... Dec 26 20:14:34.623013139 (508h45m34.494096174s) [seg_pos={0 0}, upload=446440]: Done begin upload segment. Dec 26 20:14:34.623037004 (508h45m34.494120039s) [noise=true , address=145.79.227.45:4231 , piece_id=SXTYW3PLFDECYMRHYVTA6L5N6VI6KZEILG26K4ALKBEE3JDWJFPQ, seg_pos={0 0}, upload=446440]: Uploading piece... Dec 26 20:14:34.623139539 (508h45m34.494222584s) [noise=true , address=145.79.227.35:4482 , piece_id=R67ESFGWQNGDORGMN5OZXPFKYSWN4U2RDWOXDUZN5JYNW5DGBTSQ, seg_pos={0 0}, upload=446440]: Uploading piece... Dec 26 20:14:34.623677394 (508h45m34.494760439s) [noise=true , address=145.79.227.37:4566 , piece_id=PWFS52EYRFKEU6LERET3W2QXB6ZSFICUOV3GPDUMEHKY3IAY43UA, seg_pos={0 0}, upload=446440]: Uploading piece... Dec 26 20:14:34.624155955 (508h45m34.495239s) [noise=true , address=145.79.227.3:4150 , piece_id=2E222HDKHDP3JIRHBSZOPKKO74JIBJ6D43ZCV7PU6YLNJYETJHSQ, seg_pos={0 0}, upload=446440]: Uploading piece... Dec 26 20:14:34.624517764 (508h45m34.495600809s) [noise=true , address=145.79.227.42:4121 , piece_id=DNTY3PS7VF4DFKS75BEETF4IWLZW73ULTZ555US7OK4CPJJFULIQ, seg_pos={0 0}, upload=446440]: Uploading piece... Dec 26 20:14:34.624595833 (508h45m34.495678868s) [noise=true , address=145.79.227.15:4875 , piece_id=BT2HGGDEPGOVFUSERULJCM3ZNYK4QLH24ONVVJPI4BAXWAVUDWRA, seg_pos={0 0}, upload=446440]: Uploading piece... Dec 26 20:14:34.624958985 (508h45m34.49604203s) [noise=true , address=45.91.116.6:4975 , piece_id=7IRHTKTPFRL2JAPC3XCL3ZVHTUC22SBYDP2KVL6NTUVKMF6NWYRA, seg_pos={0 0}, upload=446440]: Uploading piece... Dec 26 20:14:34.625014510 (508h45m34.496097545s) [noise=true , address=145.79.227.41:4086 , piece_id=7SZPGLQGXXFNANI65M3BS756HJPRNNKDE3R2DSAMCVZXSH5YYXAQ, seg_pos={0 0}, upload=446440]: Uploading piece... Dec 26 20:14:34.625339408 (508h45m34.496422453s) [noise=true , address=145.79.227.25:4618 , piece_id=7H5GZ2TFX35CUSEH6LVYVILR34W2XDGPNWTJ5M3UIPP727L2TRHQ, seg_pos={0 0}, upload=446440]: Uploading piece... Dec 26 20:14:34.625348947 (508h45m34.496431982s) [noise=true , address=145.79.227.46:4602 , piece_id=QKSMTCTQV25F2DEQH2D2KADAINYCOJWFK33T6NVPIG2B37YF6TTQ, seg_pos={0 0}, upload=446440]: Uploading piece... Dec 26 20:14:34.625713531 (508h45m34.496796576s) [noise=true , address=45.91.116.7:4012 , piece_id=ROQL7PKQIQIKYVX74QAXLETAM4MRN5V4ELUK6CH2ZKKAOSSOGBBA, seg_pos={0 0}, upload=446440]: Uploading piece... Dec 26 20:14:34.625757023 (508h45m34.496840058s) [noise=true , address=45.91.116.14:4189 , piece_id=65A6IHIAES37HSTNBM5FUPZZAH35WFSLH6S3TBIII7O62VEQQ25A, seg_pos={0 0}, upload=446440]: Uploading piece... Dec 26 20:14:34.626124824 (508h45m34.497207859s) [noise=true , address=45.91.116.8:4045 , piece_id=3FYKMIDJTCLANIBL6QNXRIW4ELDG5OSSQUNMAYR5ECMBPSYW2CIQ, seg_pos={0 0}, upload=446440]: Uploading piece... Dec 26 20:14:34.626171092 (508h45m34.497254127s) [noise=true , address=77.90.31.21:4877 , piece_id=MRY3PPVMCVH7WP667T3ENAWH6TRZNMNGK4ZWGOZETOY5AGE7RUZA, seg_pos={0 0}, upload=446440]: Uploading piece... Dec 26 20:14:34.626655414 (508h45m34.497738449s) [noise=true , address=45.91.116.9:4263 , piece_id=GHLDY5TXPAJFLMLMPYPJ2MFBNDUNPKMOP7I23CATXKXWQOQT5WWQ, seg_pos={0 0}, upload=446440]: Uploading piece... Dec 26 20:14:34.628204384 (508h45m34.499287419s) [noise=true , address=145.79.227.49:4338 , piece_id=MZL5YI7C7MLDE2CB47SQS2XWPLJ5VUDVWIAO2HO67O236NLZO2FA, seg_pos={0 0}, upload=446440]: Uploading piece... Dec 26 20:14:36.339473084 (508h45m36.210556119s) [upload=446440]: Next returned nil segment Dec 26 20:14:36.339483454 (508h45m36.210566489s) [upload=446440]: Waiting for error group managing segments... Dec 26 20:14:36.614251004 (508h45m36.485334049s) [noise=true , address=145.79.227.41:4086 , piece_id=7SZPGLQGXXFNANI65M3BS756HJPRNNKDE3R2DSAMCVZXSH5YYXAQ, seg_pos={0 0}, upload=446440]: Done uploading piece. err: <nil> Dec 26 20:14:36.625627025 (508h45m36.49671007s) [noise=true , address=145.79.227.45:4231 , piece_id=SXTYW3PLFDECYMRHYVTA6L5N6VI6KZEILG26K4ALKBEE3JDWJFPQ, seg_pos={0 0}, upload=446440]: Done uploading piece. err: <nil> Dec 26 20:14:36.628714234 (508h45m36.499797269s) [noise=true , address=145.79.227.42:4121 , piece_id=DNTY3PS7VF4DFKS75BEETF4IWLZW73ULTZ555US7OK4CPJJFULIQ, seg_pos={0 0}, upload=446440]: Done uploading piece. err: <nil> Dec 26 20:14:36.632592198 (508h45m36.503675233s) [noise=true , address=145.79.227.46:4602 , piece_id=QKSMTCTQV25F2DEQH2D2KADAINYCOJWFK33T6NVPIG2B37YF6TTQ, seg_pos={0 0}, upload=446440]: Done uploading piece. err: <nil> Dec 26 20:14:36.634769304 (508h45m36.505852339s) [noise=true , address=145.79.227.25:4618 , piece_id=7H5GZ2TFX35CUSEH6LVYVILR34W2XDGPNWTJ5M3UIPP727L2TRHQ, seg_pos={0 0}, upload=446440]: Done uploading piece. err: <nil> Dec 26 20:14:36.635827759 (508h45m36.506910794s) [noise=true , address=145.79.227.49:4338 , piece_id=MZL5YI7C7MLDE2CB47SQS2XWPLJ5VUDVWIAO2HO67O236NLZO2FA, seg_pos={0 0}, upload=446440]: Done uploading piece. err: <nil> Dec 26 20:14:36.639557481 (508h45m36.510640516s) [noise=true , address=145.79.227.37:4566 , piece_id=PWFS52EYRFKEU6LERET3W2QXB6ZSFICUOV3GPDUMEHKY3IAY43UA, seg_pos={0 0}, upload=446440]: Done uploading piece. err: <nil> Dec 26 20:14:36.651231290 (508h45m36.522314325s) [noise=true , address=145.79.227.3:4150 , piece_id=2E222HDKHDP3JIRHBSZOPKKO74JIBJ6D43ZCV7PU6YLNJYETJHSQ, seg_pos={0 0}, upload=446440]: Done uploading piece. err: <nil> Dec 26 20:14:36.824853844 (508h45m36.695936879s) [noise=true , address=145.79.227.15:4875 , piece_id=BT2HGGDEPGOVFUSERULJCM3ZNYK4QLH24ONVVJPI4BAXWAVUDWRA, seg_pos={0 0}, upload=446440]: Done uploading piece. err: <nil> Dec 26 20:14:36.915641057 (508h45m36.786724112s) [noise=true , address=45.91.116.9:4263 , piece_id=GHLDY5TXPAJFLMLMPYPJ2MFBNDUNPKMOP7I23CATXKXWQOQT5WWQ, seg_pos={0 0}, upload=446440]: Done uploading piece. err: <nil> Dec 26 20:14:36.955746433 (508h45m36.826829468s) [noise=true , address=45.91.116.14:4189 , piece_id=65A6IHIAES37HSTNBM5FUPZZAH35WFSLH6S3TBIII7O62VEQQ25A, seg_pos={0 0}, upload=446440]: Done uploading piece. err: <nil> Dec 26 20:14:36.964794603 (508h45m36.835877638s) [noise=true , address=45.91.116.7:4012 , piece_id=ROQL7PKQIQIKYVX74QAXLETAM4MRN5V4ELUK6CH2ZKKAOSSOGBBA, seg_pos={0 0}, upload=446440]: Done uploading piece. err: <nil> Dec 26 20:14:36.973931722 (508h45m36.845014757s) [noise=true , address=77.90.31.21:4877 , piece_id=MRY3PPVMCVH7WP667T3ENAWH6TRZNMNGK4ZWGOZETOY5AGE7RUZA, seg_pos={0 0}, upload=446440]: Done uploading piece. err: <nil> Dec 26 20:14:36.973953122 (508h45m36.845036157s) [seg_pos={0 0}, upload=446440]: Segment reached optimal threshold of 13 pieces. Dec 26 20:14:36.974139407 (508h45m36.845222442s) [noise=true , address=45.91.116.6:4975 , piece_id=7IRHTKTPFRL2JAPC3XCL3ZVHTUC22SBYDP2KVL6NTUVKMF6NWYRA, seg_pos={0 0}, upload=446440]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: write tcp 188.166.225.170:60138->45.91.116.6:4975: use of closed network connection; write tcp 188.166.225.170:60138->45.91.116.6:4975: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60138->45.91.116.6:4975: use of closed network connection Dec 26 20:14:36.974145839 (508h45m36.845228875s) [noise=true , address=45.91.116.6:4975 , piece_id=7IRHTKTPFRL2JAPC3XCL3ZVHTUC22SBYDP2KVL6NTUVKMF6NWYRA, seg_pos={0 0}, upload=446440]: Long tail cancellation due to optimal threshold Dec 26 20:14:36.974286838 (508h45m36.845369873s) [noise=true , address=145.79.227.35:4482 , piece_id=R67ESFGWQNGDORGMN5OZXPFKYSWN4U2RDWOXDUZN5JYNW5DGBTSQ, seg_pos={0 0}, upload=446440]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: expected piece hash; context canceled; EOF Dec 26 20:14:36.974292198 (508h45m36.845375233s) [noise=true , address=145.79.227.35:4482 , piece_id=R67ESFGWQNGDORGMN5OZXPFKYSWN4U2RDWOXDUZN5JYNW5DGBTSQ, seg_pos={0 0}, upload=446440]: Long tail cancellation due to optimal threshold Dec 26 20:14:36.974609763 (508h45m36.845693409s) [noise=true , address=45.91.116.8:4045 , piece_id=3FYKMIDJTCLANIBL6QNXRIW4ELDG5OSSQUNMAYR5ECMBPSYW2CIQ, seg_pos={0 0}, upload=446440]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 26 20:14:36.974632897 (508h45m36.845715962s) [noise=true , address=45.91.116.8:4045 , piece_id=3FYKMIDJTCLANIBL6QNXRIW4ELDG5OSSQUNMAYR5ECMBPSYW2CIQ, seg_pos={0 0}, upload=446440]: Long tail cancellation due to optimal threshold Dec 26 20:14:36.974668805 (508h45m36.84575184s) [seg_pos={0 0}, upload=446440]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 20:14:36.974801618 (508h45m36.845884653s) [upload=446440]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:14:36.974810725 (508h45m36.845893761s) [upload=446440]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:14:37.282277490 (508h45m37.153360525s) [upload=446440]: Done uploading segments... Dec 26 20:14:37.282296936 (508h45m37.153379971s) [upload=446440]: Upload finished. err: <nil> Duration: 7.250059484s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 32.174711983s trace: https://jaeger.dev.storj.io/trace/269ebf1e68ab4194 FAIL (2025-12-26 20:16:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== (508h13m19.087063583s) [noise=true , address=145.79.227.40:4061 , piece_id=46F3KZE7JNV3GD7VNUF3AW73RXBUZQPNNOFZYXKSVUEZQOWVBWUA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.216300006 (508h13m19.087383051s) [noise=true , address=145.79.227.30:4768 , piece_id=OHGULVS5PFX3YLWJCTJHVEKUIWX5MJLUJ3436WW5ZGRN3MALSNXA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.216730495 (508h13m19.08781353s) [noise=true , address=145.79.227.18:4019 , piece_id=V2YUVXSEVPYFQYCOHRGWO74I7ADIXBT7G6RHMEY7GJLI7MD3BCOQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217064441 (508h13m19.088147486s) [noise=true , address=77.90.31.10:4538 , piece_id=2YHRM6ACRDJFZWAB4BMWDKNQZ3C5IJ2D3AYGSLVH5FLGM6BSCXKQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217290591 (508h13m19.088373626s) [noise=true , address=45.91.116.13:4173 , piece_id=OVWQGJIDNS3MXQQF7DNCSKLLNQXFSDQEDRVVYY6YOLKHIPN3HBWA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217611703 (508h13m19.088694738s) [noise=true , address=145.79.227.16:4920 , piece_id=MNTVSKV4PMJW2F3A57JHRD2EKZUMG4YLRR7RF2QQK226HUBVZFWQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217940660 (508h13m19.089023705s) [noise=true , address=77.90.31.13:4646 , piece_id=OTKXAUXVS6WAVBJ72G3FYCD7ZZZZZO7DLYLCEOFCTQR3UZU5SJNA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217996485 (508h13m19.08907952s) [noise=true , address=77.90.31.16:4745 , piece_id=MXGPB54M3YPAVE5UQDQN6ZQJKQTWYQJPOZE3VR5X5LJIIO33Y23Q, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.218363545 (508h13m19.08944659s) [noise=true , address=77.90.31.21:4904 , piece_id=EQVI6ALFENA3SP2NL3RDU4WMLGYP7M6ODPGTDQOQZTN7O5RF7QVA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.219020315 (508h13m19.090103361s) [noise=true , address=77.90.31.19:4862 , piece_id=GC7KS7DHD7LG6HALPCWDLX5CUDKIIPYCEBSDXFMMNCHRXM64KJBQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.219639084 (508h13m19.090722129s) [noise=true , address=145.79.227.29:4975 , piece_id=ZFMWT5IU7FM5MCABHYTXVNG2VUGPGMNCDCCOZKEVOLNC7DBCW36A, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.220085213 (508h13m19.091168248s) [noise=true , address=145.79.227.9:4372 , piece_id=SVA4YGNN2TNROPPY5VH455Q4KLBJTLQM2HIORTVGR5J7PL4XSTQA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.220586408 (508h13m19.091669443s) [noise=true , address=45.91.116.6:4987 , piece_id=LHYW5CZDAY6MSTOFWDWMUEY2LFBOTSTPL42UXHIYQFN4LA2LQGAA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.221122849 (508h13m19.092205894s) [noise=true , address=145.79.227.34:4915 , piece_id=AGQTKGU3FUJSCJ2NX4WB6OWIGBKCMK4D5BBL6UYVFWL3E3UZMGOA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:20.654199727 (508h13m20.525282772s) [upload=445976]: Next returned nil segment Dec 26 19:42:20.654213192 (508h13m20.525296237s) [upload=445976]: Waiting for error group managing segments... Dec 26 19:42:21.104984249 (508h13m20.976067294s) [noise=true , address=145.79.227.29:4975 , piece_id=ZFMWT5IU7FM5MCABHYTXVNG2VUGPGMNCDCCOZKEVOLNC7DBCW36A, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.129286805 (508h13m21.00036984s) [noise=true , address=145.79.227.18:4019 , piece_id=V2YUVXSEVPYFQYCOHRGWO74I7ADIXBT7G6RHMEY7GJLI7MD3BCOQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.153491254 (508h13m21.024574289s) [noise=true , address=145.79.227.9:4372 , piece_id=SVA4YGNN2TNROPPY5VH455Q4KLBJTLQM2HIORTVGR5J7PL4XSTQA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.285157470 (508h13m21.156240505s) [noise=true , address=77.90.31.18:4837 , piece_id=QDJSYSDH3AKR7WZ5PEAQZA5AKWF7KDKBHXVUVSER3EBZTI3MTUHQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.325205408 (508h13m21.196288443s) [noise=true , address=77.90.31.16:4745 , piece_id=MXGPB54M3YPAVE5UQDQN6ZQJKQTWYQJPOZE3VR5X5LJIIO33Y23Q, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.355521865 (508h13m21.2266049s) [noise=true , address=45.91.116.13:4173 , piece_id=OVWQGJIDNS3MXQQF7DNCSKLLNQXFSDQEDRVVYY6YOLKHIPN3HBWA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.386180926 (508h13m21.257263961s) [noise=true , address=77.90.31.19:4862 , piece_id=GC7KS7DHD7LG6HALPCWDLX5CUDKIIPYCEBSDXFMMNCHRXM64KJBQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.437176951 (508h13m21.308259996s) [noise=true , address=45.91.116.6:4987 , piece_id=LHYW5CZDAY6MSTOFWDWMUEY2LFBOTSTPL42UXHIYQFN4LA2LQGAA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.023103166 (508h13m21.894186211s) [noise=true , address=145.79.227.43:4177 , piece_id=UHJ2BQY3BDNPKNAMPXZ47OY45DRI6QLXTRSDTZSUYR6MWKVOL7OA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.050932298 (508h13m21.922015333s) [noise=true , address=77.90.31.21:4904 , piece_id=EQVI6ALFENA3SP2NL3RDU4WMLGYP7M6ODPGTDQOQZTN7O5RF7QVA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.098028337 (508h13m21.969111382s) [noise=true , address=145.79.227.44:4227 , piece_id=O5JCRQMK3YH2SEJD3JFU4Z2NO6JZZWT2DLGGZ4MTAOUUBEE7FBYQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.136785577 (508h13m22.007868612s) [noise=true , address=145.79.227.30:4768 , piece_id=OHGULVS5PFX3YLWJCTJHVEKUIWX5MJLUJ3436WW5ZGRN3MALSNXA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.236850869 (508h13m22.107933914s) [noise=true , address=145.79.227.40:4061 , piece_id=46F3KZE7JNV3GD7VNUF3AW73RXBUZQPNNOFZYXKSVUEZQOWVBWUA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.352167431 (508h13m22.223250536s) [noise=true , address=145.79.227.34:4915 , piece_id=AGQTKGU3FUJSCJ2NX4WB6OWIGBKCMK4D5BBL6UYVFWL3E3UZMGOA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.352194343 (508h13m22.223277398s) [seg_pos={0 0}, upload=445976]: Segment reached optimal threshold of 14 pieces. Dec 26 19:42:22.352561061 (508h13m22.223644137s) [noise=true , address=77.90.31.13:4646 , piece_id=OTKXAUXVS6WAVBJ72G3FYCD7ZZZZZO7DLYLCEOFCTQR3UZU5SJNA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5): protocol: write tcp 188.166.225.170:51910->77.90.31.13:4646: use of closed network connection; write tcp 188.166.225.170:51910->77.90.31.13:4646: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51910->77.90.31.13:4646: use of closed network connection Dec 26 19:42:22.352570299 (508h13m22.223653344s) [noise=true , address=77.90.31.13:4646 , piece_id=OTKXAUXVS6WAVBJ72G3FYCD7ZZZZZO7DLYLCEOFCTQR3UZU5SJNA, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352627187 (508h13m22.223710233s) [noise=true , address=77.90.31.10:4538 , piece_id=2YHRM6ACRDJFZWAB4BMWDKNQZ3C5IJ2D3AYGSLVH5FLGM6BSCXKQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N): protocol: expected piece hash; context canceled; EOF Dec 26 19:42:22.352648027 (508h13m22.223731062s) [noise=true , address=77.90.31.10:4538 , piece_id=2YHRM6ACRDJFZWAB4BMWDKNQZ3C5IJ2D3AYGSLVH5FLGM6BSCXKQ, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352717288 (508h13m22.223800334s) [noise=true , address=77.90.31.15:4702 , piece_id=FBW4Y6XYMCFEV35EFN4SEMC2PQRBCCERKFHI6WA5FO4ACSDLRCFA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW): protocol: write tcp 188.166.225.170:47524->77.90.31.15:4702: use of closed network connection; write tcp 188.166.225.170:47524->77.90.31.15:4702: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47524->77.90.31.15:4702: use of closed network connection Dec 26 19:42:22.352725615 (508h13m22.22380866s) [noise=true , address=77.90.31.15:4702 , piece_id=FBW4Y6XYMCFEV35EFN4SEMC2PQRBCCERKFHI6WA5FO4ACSDLRCFA, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352737828 (508h13m22.223820863s) [noise=true , address=145.79.227.16:4920 , piece_id=MNTVSKV4PMJW2F3A57JHRD2EKZUMG4YLRR7RF2QQK226HUBVZFWQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb): protocol: expected piece hash; write tcp 188.166.225.170:60084->145.79.227.16:4920: use of closed network connection; write tcp 188.166.225.170:60084->145.79.227.16:4920: use of closed network connection; EOF Dec 26 19:42:22.352745392 (508h13m22.223828437s) [noise=true , address=145.79.227.16:4920 , piece_id=MNTVSKV4PMJW2F3A57JHRD2EKZUMG4YLRR7RF2QQK226HUBVZFWQ, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352903484 (508h13m22.223986529s) [seg_pos={0 0}, upload=445976]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 19:42:22.353035194 (508h13m22.224118239s) [upload=445976]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 19:42:22.353042468 (508h13m22.224125513s) [upload=445976]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 19:42:22.651634339 (508h13m22.522717374s) [upload=445976]: Done uploading segments... Dec 26 19:42:22.651653245 (508h13m22.52273628s) [upload=445976]: Upload finished. err: <nil> Duration: 32.174737994s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [XXX......X.X.X.X.X..............XX..X..X...........XXX.X.X..] duration: 6.123240315s trace: https://jaeger.dev.storj.io/trace/2deb4e2f5723983e FAIL (2025-12-26 20:16:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== c 26 20:14:34.361029973 (508h45m34.232113008s) [upload=446440]: Starting upload Dec 26 20:14:34.361150823 (508h45m34.232233928s) [upload=446440]: Done starting upload Dec 26 20:14:34.361343490 (508h45m34.232426525s) [upload=446440]: Uploading segments... Dec 26 20:14:34.361537168 (508h45m34.232620203s) [upload=446440]: Got next segment. Inline: false Dec 26 20:14:34.361747921 (508h45m34.232830956s) [upload=446440]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 20:14:34.361754172 (508h45m34.232837207s) [upload=446440]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 20:14:34.622967101 (508h45m34.494050136s) [seg_pos={0 0}, upload=446440]: Begin upload segment... Dec 26 20:14:34.623013139 (508h45m34.494096174s) [seg_pos={0 0}, upload=446440]: Done begin upload segment. Dec 26 20:14:34.623037004 (508h45m34.494120039s) [noise=true , address=145.79.227.45:4231 , piece_id=SXTYW3PLFDECYMRHYVTA6L5N6VI6KZEILG26K4ALKBEE3JDWJFPQ, seg_pos={0 0}, upload=446440]: Uploading piece... Dec 26 20:14:34.623139539 (508h45m34.494222584s) [noise=true , address=145.79.227.35:4482 , piece_id=R67ESFGWQNGDORGMN5OZXPFKYSWN4U2RDWOXDUZN5JYNW5DGBTSQ, seg_pos={0 0}, upload=446440]: Uploading piece... Dec 26 20:14:34.623677394 (508h45m34.494760439s) [noise=true , address=145.79.227.37:4566 , piece_id=PWFS52EYRFKEU6LERET3W2QXB6ZSFICUOV3GPDUMEHKY3IAY43UA, seg_pos={0 0}, upload=446440]: Uploading piece... Dec 26 20:14:34.624155955 (508h45m34.495239s) [noise=true , address=145.79.227.3:4150 , piece_id=2E222HDKHDP3JIRHBSZOPKKO74JIBJ6D43ZCV7PU6YLNJYETJHSQ, seg_pos={0 0}, upload=446440]: Uploading piece... Dec 26 20:14:34.624517764 (508h45m34.495600809s) [noise=true , address=145.79.227.42:4121 , piece_id=DNTY3PS7VF4DFKS75BEETF4IWLZW73ULTZ555US7OK4CPJJFULIQ, seg_pos={0 0}, upload=446440]: Uploading piece... Dec 26 20:14:34.624595833 (508h45m34.495678868s) [noise=true , address=145.79.227.15:4875 , piece_id=BT2HGGDEPGOVFUSERULJCM3ZNYK4QLH24ONVVJPI4BAXWAVUDWRA, seg_pos={0 0}, upload=446440]: Uploading piece... Dec 26 20:14:34.624958985 (508h45m34.49604203s) [noise=true , address=45.91.116.6:4975 , piece_id=7IRHTKTPFRL2JAPC3XCL3ZVHTUC22SBYDP2KVL6NTUVKMF6NWYRA, seg_pos={0 0}, upload=446440]: Uploading piece... Dec 26 20:14:34.625014510 (508h45m34.496097545s) [noise=true , address=145.79.227.41:4086 , piece_id=7SZPGLQGXXFNANI65M3BS756HJPRNNKDE3R2DSAMCVZXSH5YYXAQ, seg_pos={0 0}, upload=446440]: Uploading piece... Dec 26 20:14:34.625339408 (508h45m34.496422453s) [noise=true , address=145.79.227.25:4618 , piece_id=7H5GZ2TFX35CUSEH6LVYVILR34W2XDGPNWTJ5M3UIPP727L2TRHQ, seg_pos={0 0}, upload=446440]: Uploading piece... Dec 26 20:14:34.625348947 (508h45m34.496431982s) [noise=true , address=145.79.227.46:4602 , piece_id=QKSMTCTQV25F2DEQH2D2KADAINYCOJWFK33T6NVPIG2B37YF6TTQ, seg_pos={0 0}, upload=446440]: Uploading piece... Dec 26 20:14:34.625713531 (508h45m34.496796576s) [noise=true , address=45.91.116.7:4012 , piece_id=ROQL7PKQIQIKYVX74QAXLETAM4MRN5V4ELUK6CH2ZKKAOSSOGBBA, seg_pos={0 0}, upload=446440]: Uploading piece... Dec 26 20:14:34.625757023 (508h45m34.496840058s) [noise=true , address=45.91.116.14:4189 , piece_id=65A6IHIAES37HSTNBM5FUPZZAH35WFSLH6S3TBIII7O62VEQQ25A, seg_pos={0 0}, upload=446440]: Uploading piece... Dec 26 20:14:34.626124824 (508h45m34.497207859s) [noise=true , address=45.91.116.8:4045 , piece_id=3FYKMIDJTCLANIBL6QNXRIW4ELDG5OSSQUNMAYR5ECMBPSYW2CIQ, seg_pos={0 0}, upload=446440]: Uploading piece... Dec 26 20:14:34.626171092 (508h45m34.497254127s) [noise=true , address=77.90.31.21:4877 , piece_id=MRY3PPVMCVH7WP667T3ENAWH6TRZNMNGK4ZWGOZETOY5AGE7RUZA, seg_pos={0 0}, upload=446440]: Uploading piece... Dec 26 20:14:34.626655414 (508h45m34.497738449s) [noise=true , address=45.91.116.9:4263 , piece_id=GHLDY5TXPAJFLMLMPYPJ2MFBNDUNPKMOP7I23CATXKXWQOQT5WWQ, seg_pos={0 0}, upload=446440]: Uploading piece... Dec 26 20:14:34.628204384 (508h45m34.499287419s) [noise=true , address=145.79.227.49:4338 , piece_id=MZL5YI7C7MLDE2CB47SQS2XWPLJ5VUDVWIAO2HO67O236NLZO2FA, seg_pos={0 0}, upload=446440]: Uploading piece... Dec 26 20:14:36.339473084 (508h45m36.210556119s) [upload=446440]: Next returned nil segment Dec 26 20:14:36.339483454 (508h45m36.210566489s) [upload=446440]: Waiting for error group managing segments... Dec 26 20:14:36.614251004 (508h45m36.485334049s) [noise=true , address=145.79.227.41:4086 , piece_id=7SZPGLQGXXFNANI65M3BS756HJPRNNKDE3R2DSAMCVZXSH5YYXAQ, seg_pos={0 0}, upload=446440]: Done uploading piece. err: <nil> Dec 26 20:14:36.625627025 (508h45m36.49671007s) [noise=true , address=145.79.227.45:4231 , piece_id=SXTYW3PLFDECYMRHYVTA6L5N6VI6KZEILG26K4ALKBEE3JDWJFPQ, seg_pos={0 0}, upload=446440]: Done uploading piece. err: <nil> Dec 26 20:14:36.628714234 (508h45m36.499797269s) [noise=true , address=145.79.227.42:4121 , piece_id=DNTY3PS7VF4DFKS75BEETF4IWLZW73ULTZ555US7OK4CPJJFULIQ, seg_pos={0 0}, upload=446440]: Done uploading piece. err: <nil> Dec 26 20:14:36.632592198 (508h45m36.503675233s) [noise=true , address=145.79.227.46:4602 , piece_id=QKSMTCTQV25F2DEQH2D2KADAINYCOJWFK33T6NVPIG2B37YF6TTQ, seg_pos={0 0}, upload=446440]: Done uploading piece. err: <nil> Dec 26 20:14:36.634769304 (508h45m36.505852339s) [noise=true , address=145.79.227.25:4618 , piece_id=7H5GZ2TFX35CUSEH6LVYVILR34W2XDGPNWTJ5M3UIPP727L2TRHQ, seg_pos={0 0}, upload=446440]: Done uploading piece. err: <nil> Dec 26 20:14:36.635827759 (508h45m36.506910794s) [noise=true , address=145.79.227.49:4338 , piece_id=MZL5YI7C7MLDE2CB47SQS2XWPLJ5VUDVWIAO2HO67O236NLZO2FA, seg_pos={0 0}, upload=446440]: Done uploading piece. err: <nil> Dec 26 20:14:36.639557481 (508h45m36.510640516s) [noise=true , address=145.79.227.37:4566 , piece_id=PWFS52EYRFKEU6LERET3W2QXB6ZSFICUOV3GPDUMEHKY3IAY43UA, seg_pos={0 0}, upload=446440]: Done uploading piece. err: <nil> Dec 26 20:14:36.651231290 (508h45m36.522314325s) [noise=true , address=145.79.227.3:4150 , piece_id=2E222HDKHDP3JIRHBSZOPKKO74JIBJ6D43ZCV7PU6YLNJYETJHSQ, seg_pos={0 0}, upload=446440]: Done uploading piece. err: <nil> Dec 26 20:14:36.824853844 (508h45m36.695936879s) [noise=true , address=145.79.227.15:4875 , piece_id=BT2HGGDEPGOVFUSERULJCM3ZNYK4QLH24ONVVJPI4BAXWAVUDWRA, seg_pos={0 0}, upload=446440]: Done uploading piece. err: <nil> Dec 26 20:14:36.915641057 (508h45m36.786724112s) [noise=true , address=45.91.116.9:4263 , piece_id=GHLDY5TXPAJFLMLMPYPJ2MFBNDUNPKMOP7I23CATXKXWQOQT5WWQ, seg_pos={0 0}, upload=446440]: Done uploading piece. err: <nil> Dec 26 20:14:36.955746433 (508h45m36.826829468s) [noise=true , address=45.91.116.14:4189 , piece_id=65A6IHIAES37HSTNBM5FUPZZAH35WFSLH6S3TBIII7O62VEQQ25A, seg_pos={0 0}, upload=446440]: Done uploading piece. err: <nil> Dec 26 20:14:36.964794603 (508h45m36.835877638s) [noise=true , address=45.91.116.7:4012 , piece_id=ROQL7PKQIQIKYVX74QAXLETAM4MRN5V4ELUK6CH2ZKKAOSSOGBBA, seg_pos={0 0}, upload=446440]: Done uploading piece. err: <nil> Dec 26 20:14:36.973931722 (508h45m36.845014757s) [noise=true , address=77.90.31.21:4877 , piece_id=MRY3PPVMCVH7WP667T3ENAWH6TRZNMNGK4ZWGOZETOY5AGE7RUZA, seg_pos={0 0}, upload=446440]: Done uploading piece. err: <nil> Dec 26 20:14:36.973953122 (508h45m36.845036157s) [seg_pos={0 0}, upload=446440]: Segment reached optimal threshold of 13 pieces. Dec 26 20:14:36.974139407 (508h45m36.845222442s) [noise=true , address=45.91.116.6:4975 , piece_id=7IRHTKTPFRL2JAPC3XCL3ZVHTUC22SBYDP2KVL6NTUVKMF6NWYRA, seg_pos={0 0}, upload=446440]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: write tcp 188.166.225.170:60138->45.91.116.6:4975: use of closed network connection; write tcp 188.166.225.170:60138->45.91.116.6:4975: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60138->45.91.116.6:4975: use of closed network connection Dec 26 20:14:36.974145839 (508h45m36.845228875s) [noise=true , address=45.91.116.6:4975 , piece_id=7IRHTKTPFRL2JAPC3XCL3ZVHTUC22SBYDP2KVL6NTUVKMF6NWYRA, seg_pos={0 0}, upload=446440]: Long tail cancellation due to optimal threshold Dec 26 20:14:36.974286838 (508h45m36.845369873s) [noise=true , address=145.79.227.35:4482 , piece_id=R67ESFGWQNGDORGMN5OZXPFKYSWN4U2RDWOXDUZN5JYNW5DGBTSQ, seg_pos={0 0}, upload=446440]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: expected piece hash; context canceled; EOF Dec 26 20:14:36.974292198 (508h45m36.845375233s) [noise=true , address=145.79.227.35:4482 , piece_id=R67ESFGWQNGDORGMN5OZXPFKYSWN4U2RDWOXDUZN5JYNW5DGBTSQ, seg_pos={0 0}, upload=446440]: Long tail cancellation due to optimal threshold Dec 26 20:14:36.974609763 (508h45m36.845693409s) [noise=true , address=45.91.116.8:4045 , piece_id=3FYKMIDJTCLANIBL6QNXRIW4ELDG5OSSQUNMAYR5ECMBPSYW2CIQ, seg_pos={0 0}, upload=446440]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 26 20:14:36.974632897 (508h45m36.845715962s) [noise=true , address=45.91.116.8:4045 , piece_id=3FYKMIDJTCLANIBL6QNXRIW4ELDG5OSSQUNMAYR5ECMBPSYW2CIQ, seg_pos={0 0}, upload=446440]: Long tail cancellation due to optimal threshold Dec 26 20:14:36.974668805 (508h45m36.84575184s) [seg_pos={0 0}, upload=446440]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 20:14:36.974801618 (508h45m36.845884653s) [upload=446440]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:14:36.974810725 (508h45m36.845893761s) [upload=446440]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:14:37.282277490 (508h45m37.153360525s) [upload=446440]: Done uploading segments... Dec 26 20:14:37.282296936 (508h45m37.153379971s) [upload=446440]: Upload finished. err: <nil> Duration: 6.123267588s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 29.390717191s trace: https://jaeger.dev.storj.io/trace/269ebf1e68ab4194 FAIL (2025-12-26 20:15:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== (508h13m19.087063583s) [noise=true , address=145.79.227.40:4061 , piece_id=46F3KZE7JNV3GD7VNUF3AW73RXBUZQPNNOFZYXKSVUEZQOWVBWUA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.216300006 (508h13m19.087383051s) [noise=true , address=145.79.227.30:4768 , piece_id=OHGULVS5PFX3YLWJCTJHVEKUIWX5MJLUJ3436WW5ZGRN3MALSNXA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.216730495 (508h13m19.08781353s) [noise=true , address=145.79.227.18:4019 , piece_id=V2YUVXSEVPYFQYCOHRGWO74I7ADIXBT7G6RHMEY7GJLI7MD3BCOQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217064441 (508h13m19.088147486s) [noise=true , address=77.90.31.10:4538 , piece_id=2YHRM6ACRDJFZWAB4BMWDKNQZ3C5IJ2D3AYGSLVH5FLGM6BSCXKQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217290591 (508h13m19.088373626s) [noise=true , address=45.91.116.13:4173 , piece_id=OVWQGJIDNS3MXQQF7DNCSKLLNQXFSDQEDRVVYY6YOLKHIPN3HBWA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217611703 (508h13m19.088694738s) [noise=true , address=145.79.227.16:4920 , piece_id=MNTVSKV4PMJW2F3A57JHRD2EKZUMG4YLRR7RF2QQK226HUBVZFWQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217940660 (508h13m19.089023705s) [noise=true , address=77.90.31.13:4646 , piece_id=OTKXAUXVS6WAVBJ72G3FYCD7ZZZZZO7DLYLCEOFCTQR3UZU5SJNA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217996485 (508h13m19.08907952s) [noise=true , address=77.90.31.16:4745 , piece_id=MXGPB54M3YPAVE5UQDQN6ZQJKQTWYQJPOZE3VR5X5LJIIO33Y23Q, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.218363545 (508h13m19.08944659s) [noise=true , address=77.90.31.21:4904 , piece_id=EQVI6ALFENA3SP2NL3RDU4WMLGYP7M6ODPGTDQOQZTN7O5RF7QVA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.219020315 (508h13m19.090103361s) [noise=true , address=77.90.31.19:4862 , piece_id=GC7KS7DHD7LG6HALPCWDLX5CUDKIIPYCEBSDXFMMNCHRXM64KJBQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.219639084 (508h13m19.090722129s) [noise=true , address=145.79.227.29:4975 , piece_id=ZFMWT5IU7FM5MCABHYTXVNG2VUGPGMNCDCCOZKEVOLNC7DBCW36A, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.220085213 (508h13m19.091168248s) [noise=true , address=145.79.227.9:4372 , piece_id=SVA4YGNN2TNROPPY5VH455Q4KLBJTLQM2HIORTVGR5J7PL4XSTQA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.220586408 (508h13m19.091669443s) [noise=true , address=45.91.116.6:4987 , piece_id=LHYW5CZDAY6MSTOFWDWMUEY2LFBOTSTPL42UXHIYQFN4LA2LQGAA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.221122849 (508h13m19.092205894s) [noise=true , address=145.79.227.34:4915 , piece_id=AGQTKGU3FUJSCJ2NX4WB6OWIGBKCMK4D5BBL6UYVFWL3E3UZMGOA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:20.654199727 (508h13m20.525282772s) [upload=445976]: Next returned nil segment Dec 26 19:42:20.654213192 (508h13m20.525296237s) [upload=445976]: Waiting for error group managing segments... Dec 26 19:42:21.104984249 (508h13m20.976067294s) [noise=true , address=145.79.227.29:4975 , piece_id=ZFMWT5IU7FM5MCABHYTXVNG2VUGPGMNCDCCOZKEVOLNC7DBCW36A, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.129286805 (508h13m21.00036984s) [noise=true , address=145.79.227.18:4019 , piece_id=V2YUVXSEVPYFQYCOHRGWO74I7ADIXBT7G6RHMEY7GJLI7MD3BCOQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.153491254 (508h13m21.024574289s) [noise=true , address=145.79.227.9:4372 , piece_id=SVA4YGNN2TNROPPY5VH455Q4KLBJTLQM2HIORTVGR5J7PL4XSTQA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.285157470 (508h13m21.156240505s) [noise=true , address=77.90.31.18:4837 , piece_id=QDJSYSDH3AKR7WZ5PEAQZA5AKWF7KDKBHXVUVSER3EBZTI3MTUHQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.325205408 (508h13m21.196288443s) [noise=true , address=77.90.31.16:4745 , piece_id=MXGPB54M3YPAVE5UQDQN6ZQJKQTWYQJPOZE3VR5X5LJIIO33Y23Q, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.355521865 (508h13m21.2266049s) [noise=true , address=45.91.116.13:4173 , piece_id=OVWQGJIDNS3MXQQF7DNCSKLLNQXFSDQEDRVVYY6YOLKHIPN3HBWA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.386180926 (508h13m21.257263961s) [noise=true , address=77.90.31.19:4862 , piece_id=GC7KS7DHD7LG6HALPCWDLX5CUDKIIPYCEBSDXFMMNCHRXM64KJBQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.437176951 (508h13m21.308259996s) [noise=true , address=45.91.116.6:4987 , piece_id=LHYW5CZDAY6MSTOFWDWMUEY2LFBOTSTPL42UXHIYQFN4LA2LQGAA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.023103166 (508h13m21.894186211s) [noise=true , address=145.79.227.43:4177 , piece_id=UHJ2BQY3BDNPKNAMPXZ47OY45DRI6QLXTRSDTZSUYR6MWKVOL7OA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.050932298 (508h13m21.922015333s) [noise=true , address=77.90.31.21:4904 , piece_id=EQVI6ALFENA3SP2NL3RDU4WMLGYP7M6ODPGTDQOQZTN7O5RF7QVA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.098028337 (508h13m21.969111382s) [noise=true , address=145.79.227.44:4227 , piece_id=O5JCRQMK3YH2SEJD3JFU4Z2NO6JZZWT2DLGGZ4MTAOUUBEE7FBYQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.136785577 (508h13m22.007868612s) [noise=true , address=145.79.227.30:4768 , piece_id=OHGULVS5PFX3YLWJCTJHVEKUIWX5MJLUJ3436WW5ZGRN3MALSNXA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.236850869 (508h13m22.107933914s) [noise=true , address=145.79.227.40:4061 , piece_id=46F3KZE7JNV3GD7VNUF3AW73RXBUZQPNNOFZYXKSVUEZQOWVBWUA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.352167431 (508h13m22.223250536s) [noise=true , address=145.79.227.34:4915 , piece_id=AGQTKGU3FUJSCJ2NX4WB6OWIGBKCMK4D5BBL6UYVFWL3E3UZMGOA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.352194343 (508h13m22.223277398s) [seg_pos={0 0}, upload=445976]: Segment reached optimal threshold of 14 pieces. Dec 26 19:42:22.352561061 (508h13m22.223644137s) [noise=true , address=77.90.31.13:4646 , piece_id=OTKXAUXVS6WAVBJ72G3FYCD7ZZZZZO7DLYLCEOFCTQR3UZU5SJNA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5): protocol: write tcp 188.166.225.170:51910->77.90.31.13:4646: use of closed network connection; write tcp 188.166.225.170:51910->77.90.31.13:4646: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51910->77.90.31.13:4646: use of closed network connection Dec 26 19:42:22.352570299 (508h13m22.223653344s) [noise=true , address=77.90.31.13:4646 , piece_id=OTKXAUXVS6WAVBJ72G3FYCD7ZZZZZO7DLYLCEOFCTQR3UZU5SJNA, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352627187 (508h13m22.223710233s) [noise=true , address=77.90.31.10:4538 , piece_id=2YHRM6ACRDJFZWAB4BMWDKNQZ3C5IJ2D3AYGSLVH5FLGM6BSCXKQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N): protocol: expected piece hash; context canceled; EOF Dec 26 19:42:22.352648027 (508h13m22.223731062s) [noise=true , address=77.90.31.10:4538 , piece_id=2YHRM6ACRDJFZWAB4BMWDKNQZ3C5IJ2D3AYGSLVH5FLGM6BSCXKQ, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352717288 (508h13m22.223800334s) [noise=true , address=77.90.31.15:4702 , piece_id=FBW4Y6XYMCFEV35EFN4SEMC2PQRBCCERKFHI6WA5FO4ACSDLRCFA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW): protocol: write tcp 188.166.225.170:47524->77.90.31.15:4702: use of closed network connection; write tcp 188.166.225.170:47524->77.90.31.15:4702: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47524->77.90.31.15:4702: use of closed network connection Dec 26 19:42:22.352725615 (508h13m22.22380866s) [noise=true , address=77.90.31.15:4702 , piece_id=FBW4Y6XYMCFEV35EFN4SEMC2PQRBCCERKFHI6WA5FO4ACSDLRCFA, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352737828 (508h13m22.223820863s) [noise=true , address=145.79.227.16:4920 , piece_id=MNTVSKV4PMJW2F3A57JHRD2EKZUMG4YLRR7RF2QQK226HUBVZFWQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb): protocol: expected piece hash; write tcp 188.166.225.170:60084->145.79.227.16:4920: use of closed network connection; write tcp 188.166.225.170:60084->145.79.227.16:4920: use of closed network connection; EOF Dec 26 19:42:22.352745392 (508h13m22.223828437s) [noise=true , address=145.79.227.16:4920 , piece_id=MNTVSKV4PMJW2F3A57JHRD2EKZUMG4YLRR7RF2QQK226HUBVZFWQ, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352903484 (508h13m22.223986529s) [seg_pos={0 0}, upload=445976]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 19:42:22.353035194 (508h13m22.224118239s) [upload=445976]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 19:42:22.353042468 (508h13m22.224125513s) [upload=445976]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 19:42:22.651634339 (508h13m22.522717374s) [upload=445976]: Done uploading segments... Dec 26 19:42:22.651653245 (508h13m22.52273628s) [upload=445976]: Upload finished. err: <nil> Duration: 29.390748058s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [XXXX......X.X.X.X.X..............XX..X..X...........XXX.X.X.] duration: 9.714814407s trace: https://jaeger.dev.storj.io/trace/2deb4e2f5723983e FAIL (2025-12-26 20:15:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== c 26 20:14:34.361029973 (508h45m34.232113008s) [upload=446440]: Starting upload Dec 26 20:14:34.361150823 (508h45m34.232233928s) [upload=446440]: Done starting upload Dec 26 20:14:34.361343490 (508h45m34.232426525s) [upload=446440]: Uploading segments... Dec 26 20:14:34.361537168 (508h45m34.232620203s) [upload=446440]: Got next segment. Inline: false Dec 26 20:14:34.361747921 (508h45m34.232830956s) [upload=446440]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 20:14:34.361754172 (508h45m34.232837207s) [upload=446440]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 20:14:34.622967101 (508h45m34.494050136s) [seg_pos={0 0}, upload=446440]: Begin upload segment... Dec 26 20:14:34.623013139 (508h45m34.494096174s) [seg_pos={0 0}, upload=446440]: Done begin upload segment. Dec 26 20:14:34.623037004 (508h45m34.494120039s) [noise=true , address=145.79.227.45:4231 , piece_id=SXTYW3PLFDECYMRHYVTA6L5N6VI6KZEILG26K4ALKBEE3JDWJFPQ, seg_pos={0 0}, upload=446440]: Uploading piece... Dec 26 20:14:34.623139539 (508h45m34.494222584s) [noise=true , address=145.79.227.35:4482 , piece_id=R67ESFGWQNGDORGMN5OZXPFKYSWN4U2RDWOXDUZN5JYNW5DGBTSQ, seg_pos={0 0}, upload=446440]: Uploading piece... Dec 26 20:14:34.623677394 (508h45m34.494760439s) [noise=true , address=145.79.227.37:4566 , piece_id=PWFS52EYRFKEU6LERET3W2QXB6ZSFICUOV3GPDUMEHKY3IAY43UA, seg_pos={0 0}, upload=446440]: Uploading piece... Dec 26 20:14:34.624155955 (508h45m34.495239s) [noise=true , address=145.79.227.3:4150 , piece_id=2E222HDKHDP3JIRHBSZOPKKO74JIBJ6D43ZCV7PU6YLNJYETJHSQ, seg_pos={0 0}, upload=446440]: Uploading piece... Dec 26 20:14:34.624517764 (508h45m34.495600809s) [noise=true , address=145.79.227.42:4121 , piece_id=DNTY3PS7VF4DFKS75BEETF4IWLZW73ULTZ555US7OK4CPJJFULIQ, seg_pos={0 0}, upload=446440]: Uploading piece... Dec 26 20:14:34.624595833 (508h45m34.495678868s) [noise=true , address=145.79.227.15:4875 , piece_id=BT2HGGDEPGOVFUSERULJCM3ZNYK4QLH24ONVVJPI4BAXWAVUDWRA, seg_pos={0 0}, upload=446440]: Uploading piece... Dec 26 20:14:34.624958985 (508h45m34.49604203s) [noise=true , address=45.91.116.6:4975 , piece_id=7IRHTKTPFRL2JAPC3XCL3ZVHTUC22SBYDP2KVL6NTUVKMF6NWYRA, seg_pos={0 0}, upload=446440]: Uploading piece... Dec 26 20:14:34.625014510 (508h45m34.496097545s) [noise=true , address=145.79.227.41:4086 , piece_id=7SZPGLQGXXFNANI65M3BS756HJPRNNKDE3R2DSAMCVZXSH5YYXAQ, seg_pos={0 0}, upload=446440]: Uploading piece... Dec 26 20:14:34.625339408 (508h45m34.496422453s) [noise=true , address=145.79.227.25:4618 , piece_id=7H5GZ2TFX35CUSEH6LVYVILR34W2XDGPNWTJ5M3UIPP727L2TRHQ, seg_pos={0 0}, upload=446440]: Uploading piece... Dec 26 20:14:34.625348947 (508h45m34.496431982s) [noise=true , address=145.79.227.46:4602 , piece_id=QKSMTCTQV25F2DEQH2D2KADAINYCOJWFK33T6NVPIG2B37YF6TTQ, seg_pos={0 0}, upload=446440]: Uploading piece... Dec 26 20:14:34.625713531 (508h45m34.496796576s) [noise=true , address=45.91.116.7:4012 , piece_id=ROQL7PKQIQIKYVX74QAXLETAM4MRN5V4ELUK6CH2ZKKAOSSOGBBA, seg_pos={0 0}, upload=446440]: Uploading piece... Dec 26 20:14:34.625757023 (508h45m34.496840058s) [noise=true , address=45.91.116.14:4189 , piece_id=65A6IHIAES37HSTNBM5FUPZZAH35WFSLH6S3TBIII7O62VEQQ25A, seg_pos={0 0}, upload=446440]: Uploading piece... Dec 26 20:14:34.626124824 (508h45m34.497207859s) [noise=true , address=45.91.116.8:4045 , piece_id=3FYKMIDJTCLANIBL6QNXRIW4ELDG5OSSQUNMAYR5ECMBPSYW2CIQ, seg_pos={0 0}, upload=446440]: Uploading piece... Dec 26 20:14:34.626171092 (508h45m34.497254127s) [noise=true , address=77.90.31.21:4877 , piece_id=MRY3PPVMCVH7WP667T3ENAWH6TRZNMNGK4ZWGOZETOY5AGE7RUZA, seg_pos={0 0}, upload=446440]: Uploading piece... Dec 26 20:14:34.626655414 (508h45m34.497738449s) [noise=true , address=45.91.116.9:4263 , piece_id=GHLDY5TXPAJFLMLMPYPJ2MFBNDUNPKMOP7I23CATXKXWQOQT5WWQ, seg_pos={0 0}, upload=446440]: Uploading piece... Dec 26 20:14:34.628204384 (508h45m34.499287419s) [noise=true , address=145.79.227.49:4338 , piece_id=MZL5YI7C7MLDE2CB47SQS2XWPLJ5VUDVWIAO2HO67O236NLZO2FA, seg_pos={0 0}, upload=446440]: Uploading piece... Dec 26 20:14:36.339473084 (508h45m36.210556119s) [upload=446440]: Next returned nil segment Dec 26 20:14:36.339483454 (508h45m36.210566489s) [upload=446440]: Waiting for error group managing segments... Dec 26 20:14:36.614251004 (508h45m36.485334049s) [noise=true , address=145.79.227.41:4086 , piece_id=7SZPGLQGXXFNANI65M3BS756HJPRNNKDE3R2DSAMCVZXSH5YYXAQ, seg_pos={0 0}, upload=446440]: Done uploading piece. err: <nil> Dec 26 20:14:36.625627025 (508h45m36.49671007s) [noise=true , address=145.79.227.45:4231 , piece_id=SXTYW3PLFDECYMRHYVTA6L5N6VI6KZEILG26K4ALKBEE3JDWJFPQ, seg_pos={0 0}, upload=446440]: Done uploading piece. err: <nil> Dec 26 20:14:36.628714234 (508h45m36.499797269s) [noise=true , address=145.79.227.42:4121 , piece_id=DNTY3PS7VF4DFKS75BEETF4IWLZW73ULTZ555US7OK4CPJJFULIQ, seg_pos={0 0}, upload=446440]: Done uploading piece. err: <nil> Dec 26 20:14:36.632592198 (508h45m36.503675233s) [noise=true , address=145.79.227.46:4602 , piece_id=QKSMTCTQV25F2DEQH2D2KADAINYCOJWFK33T6NVPIG2B37YF6TTQ, seg_pos={0 0}, upload=446440]: Done uploading piece. err: <nil> Dec 26 20:14:36.634769304 (508h45m36.505852339s) [noise=true , address=145.79.227.25:4618 , piece_id=7H5GZ2TFX35CUSEH6LVYVILR34W2XDGPNWTJ5M3UIPP727L2TRHQ, seg_pos={0 0}, upload=446440]: Done uploading piece. err: <nil> Dec 26 20:14:36.635827759 (508h45m36.506910794s) [noise=true , address=145.79.227.49:4338 , piece_id=MZL5YI7C7MLDE2CB47SQS2XWPLJ5VUDVWIAO2HO67O236NLZO2FA, seg_pos={0 0}, upload=446440]: Done uploading piece. err: <nil> Dec 26 20:14:36.639557481 (508h45m36.510640516s) [noise=true , address=145.79.227.37:4566 , piece_id=PWFS52EYRFKEU6LERET3W2QXB6ZSFICUOV3GPDUMEHKY3IAY43UA, seg_pos={0 0}, upload=446440]: Done uploading piece. err: <nil> Dec 26 20:14:36.651231290 (508h45m36.522314325s) [noise=true , address=145.79.227.3:4150 , piece_id=2E222HDKHDP3JIRHBSZOPKKO74JIBJ6D43ZCV7PU6YLNJYETJHSQ, seg_pos={0 0}, upload=446440]: Done uploading piece. err: <nil> Dec 26 20:14:36.824853844 (508h45m36.695936879s) [noise=true , address=145.79.227.15:4875 , piece_id=BT2HGGDEPGOVFUSERULJCM3ZNYK4QLH24ONVVJPI4BAXWAVUDWRA, seg_pos={0 0}, upload=446440]: Done uploading piece. err: <nil> Dec 26 20:14:36.915641057 (508h45m36.786724112s) [noise=true , address=45.91.116.9:4263 , piece_id=GHLDY5TXPAJFLMLMPYPJ2MFBNDUNPKMOP7I23CATXKXWQOQT5WWQ, seg_pos={0 0}, upload=446440]: Done uploading piece. err: <nil> Dec 26 20:14:36.955746433 (508h45m36.826829468s) [noise=true , address=45.91.116.14:4189 , piece_id=65A6IHIAES37HSTNBM5FUPZZAH35WFSLH6S3TBIII7O62VEQQ25A, seg_pos={0 0}, upload=446440]: Done uploading piece. err: <nil> Dec 26 20:14:36.964794603 (508h45m36.835877638s) [noise=true , address=45.91.116.7:4012 , piece_id=ROQL7PKQIQIKYVX74QAXLETAM4MRN5V4ELUK6CH2ZKKAOSSOGBBA, seg_pos={0 0}, upload=446440]: Done uploading piece. err: <nil> Dec 26 20:14:36.973931722 (508h45m36.845014757s) [noise=true , address=77.90.31.21:4877 , piece_id=MRY3PPVMCVH7WP667T3ENAWH6TRZNMNGK4ZWGOZETOY5AGE7RUZA, seg_pos={0 0}, upload=446440]: Done uploading piece. err: <nil> Dec 26 20:14:36.973953122 (508h45m36.845036157s) [seg_pos={0 0}, upload=446440]: Segment reached optimal threshold of 13 pieces. Dec 26 20:14:36.974139407 (508h45m36.845222442s) [noise=true , address=45.91.116.6:4975 , piece_id=7IRHTKTPFRL2JAPC3XCL3ZVHTUC22SBYDP2KVL6NTUVKMF6NWYRA, seg_pos={0 0}, upload=446440]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: write tcp 188.166.225.170:60138->45.91.116.6:4975: use of closed network connection; write tcp 188.166.225.170:60138->45.91.116.6:4975: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60138->45.91.116.6:4975: use of closed network connection Dec 26 20:14:36.974145839 (508h45m36.845228875s) [noise=true , address=45.91.116.6:4975 , piece_id=7IRHTKTPFRL2JAPC3XCL3ZVHTUC22SBYDP2KVL6NTUVKMF6NWYRA, seg_pos={0 0}, upload=446440]: Long tail cancellation due to optimal threshold Dec 26 20:14:36.974286838 (508h45m36.845369873s) [noise=true , address=145.79.227.35:4482 , piece_id=R67ESFGWQNGDORGMN5OZXPFKYSWN4U2RDWOXDUZN5JYNW5DGBTSQ, seg_pos={0 0}, upload=446440]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: expected piece hash; context canceled; EOF Dec 26 20:14:36.974292198 (508h45m36.845375233s) [noise=true , address=145.79.227.35:4482 , piece_id=R67ESFGWQNGDORGMN5OZXPFKYSWN4U2RDWOXDUZN5JYNW5DGBTSQ, seg_pos={0 0}, upload=446440]: Long tail cancellation due to optimal threshold Dec 26 20:14:36.974609763 (508h45m36.845693409s) [noise=true , address=45.91.116.8:4045 , piece_id=3FYKMIDJTCLANIBL6QNXRIW4ELDG5OSSQUNMAYR5ECMBPSYW2CIQ, seg_pos={0 0}, upload=446440]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 26 20:14:36.974632897 (508h45m36.845715962s) [noise=true , address=45.91.116.8:4045 , piece_id=3FYKMIDJTCLANIBL6QNXRIW4ELDG5OSSQUNMAYR5ECMBPSYW2CIQ, seg_pos={0 0}, upload=446440]: Long tail cancellation due to optimal threshold Dec 26 20:14:36.974668805 (508h45m36.84575184s) [seg_pos={0 0}, upload=446440]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 20:14:36.974801618 (508h45m36.845884653s) [upload=446440]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:14:36.974810725 (508h45m36.845893761s) [upload=446440]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:14:37.282277490 (508h45m37.153360525s) [upload=446440]: Done uploading segments... Dec 26 20:14:37.282296936 (508h45m37.153379971s) [upload=446440]: Upload finished. err: <nil> Duration: 9.71484669s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [XXXXX......X.X.X.X.X..............XX..X..X...........XXX.X.X] duration: 30.003657948s trace: https://jaeger.dev.storj.io/trace/2deb4e2f5723983e FAIL (2025-12-26 20:15:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== c 26 20:14:34.361029973 (508h45m34.232113008s) [upload=446440]: Starting upload Dec 26 20:14:34.361150823 (508h45m34.232233928s) [upload=446440]: Done starting upload Dec 26 20:14:34.361343490 (508h45m34.232426525s) [upload=446440]: Uploading segments... Dec 26 20:14:34.361537168 (508h45m34.232620203s) [upload=446440]: Got next segment. Inline: false Dec 26 20:14:34.361747921 (508h45m34.232830956s) [upload=446440]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 20:14:34.361754172 (508h45m34.232837207s) [upload=446440]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 20:14:34.622967101 (508h45m34.494050136s) [seg_pos={0 0}, upload=446440]: Begin upload segment... Dec 26 20:14:34.623013139 (508h45m34.494096174s) [seg_pos={0 0}, upload=446440]: Done begin upload segment. Dec 26 20:14:34.623037004 (508h45m34.494120039s) [noise=true , address=145.79.227.45:4231 , piece_id=SXTYW3PLFDECYMRHYVTA6L5N6VI6KZEILG26K4ALKBEE3JDWJFPQ, seg_pos={0 0}, upload=446440]: Uploading piece... Dec 26 20:14:34.623139539 (508h45m34.494222584s) [noise=true , address=145.79.227.35:4482 , piece_id=R67ESFGWQNGDORGMN5OZXPFKYSWN4U2RDWOXDUZN5JYNW5DGBTSQ, seg_pos={0 0}, upload=446440]: Uploading piece... Dec 26 20:14:34.623677394 (508h45m34.494760439s) [noise=true , address=145.79.227.37:4566 , piece_id=PWFS52EYRFKEU6LERET3W2QXB6ZSFICUOV3GPDUMEHKY3IAY43UA, seg_pos={0 0}, upload=446440]: Uploading piece... Dec 26 20:14:34.624155955 (508h45m34.495239s) [noise=true , address=145.79.227.3:4150 , piece_id=2E222HDKHDP3JIRHBSZOPKKO74JIBJ6D43ZCV7PU6YLNJYETJHSQ, seg_pos={0 0}, upload=446440]: Uploading piece... Dec 26 20:14:34.624517764 (508h45m34.495600809s) [noise=true , address=145.79.227.42:4121 , piece_id=DNTY3PS7VF4DFKS75BEETF4IWLZW73ULTZ555US7OK4CPJJFULIQ, seg_pos={0 0}, upload=446440]: Uploading piece... Dec 26 20:14:34.624595833 (508h45m34.495678868s) [noise=true , address=145.79.227.15:4875 , piece_id=BT2HGGDEPGOVFUSERULJCM3ZNYK4QLH24ONVVJPI4BAXWAVUDWRA, seg_pos={0 0}, upload=446440]: Uploading piece... Dec 26 20:14:34.624958985 (508h45m34.49604203s) [noise=true , address=45.91.116.6:4975 , piece_id=7IRHTKTPFRL2JAPC3XCL3ZVHTUC22SBYDP2KVL6NTUVKMF6NWYRA, seg_pos={0 0}, upload=446440]: Uploading piece... Dec 26 20:14:34.625014510 (508h45m34.496097545s) [noise=true , address=145.79.227.41:4086 , piece_id=7SZPGLQGXXFNANI65M3BS756HJPRNNKDE3R2DSAMCVZXSH5YYXAQ, seg_pos={0 0}, upload=446440]: Uploading piece... Dec 26 20:14:34.625339408 (508h45m34.496422453s) [noise=true , address=145.79.227.25:4618 , piece_id=7H5GZ2TFX35CUSEH6LVYVILR34W2XDGPNWTJ5M3UIPP727L2TRHQ, seg_pos={0 0}, upload=446440]: Uploading piece... Dec 26 20:14:34.625348947 (508h45m34.496431982s) [noise=true , address=145.79.227.46:4602 , piece_id=QKSMTCTQV25F2DEQH2D2KADAINYCOJWFK33T6NVPIG2B37YF6TTQ, seg_pos={0 0}, upload=446440]: Uploading piece... Dec 26 20:14:34.625713531 (508h45m34.496796576s) [noise=true , address=45.91.116.7:4012 , piece_id=ROQL7PKQIQIKYVX74QAXLETAM4MRN5V4ELUK6CH2ZKKAOSSOGBBA, seg_pos={0 0}, upload=446440]: Uploading piece... Dec 26 20:14:34.625757023 (508h45m34.496840058s) [noise=true , address=45.91.116.14:4189 , piece_id=65A6IHIAES37HSTNBM5FUPZZAH35WFSLH6S3TBIII7O62VEQQ25A, seg_pos={0 0}, upload=446440]: Uploading piece... Dec 26 20:14:34.626124824 (508h45m34.497207859s) [noise=true , address=45.91.116.8:4045 , piece_id=3FYKMIDJTCLANIBL6QNXRIW4ELDG5OSSQUNMAYR5ECMBPSYW2CIQ, seg_pos={0 0}, upload=446440]: Uploading piece... Dec 26 20:14:34.626171092 (508h45m34.497254127s) [noise=true , address=77.90.31.21:4877 , piece_id=MRY3PPVMCVH7WP667T3ENAWH6TRZNMNGK4ZWGOZETOY5AGE7RUZA, seg_pos={0 0}, upload=446440]: Uploading piece... Dec 26 20:14:34.626655414 (508h45m34.497738449s) [noise=true , address=45.91.116.9:4263 , piece_id=GHLDY5TXPAJFLMLMPYPJ2MFBNDUNPKMOP7I23CATXKXWQOQT5WWQ, seg_pos={0 0}, upload=446440]: Uploading piece... Dec 26 20:14:34.628204384 (508h45m34.499287419s) [noise=true , address=145.79.227.49:4338 , piece_id=MZL5YI7C7MLDE2CB47SQS2XWPLJ5VUDVWIAO2HO67O236NLZO2FA, seg_pos={0 0}, upload=446440]: Uploading piece... Dec 26 20:14:36.339473084 (508h45m36.210556119s) [upload=446440]: Next returned nil segment Dec 26 20:14:36.339483454 (508h45m36.210566489s) [upload=446440]: Waiting for error group managing segments... Dec 26 20:14:36.614251004 (508h45m36.485334049s) [noise=true , address=145.79.227.41:4086 , piece_id=7SZPGLQGXXFNANI65M3BS756HJPRNNKDE3R2DSAMCVZXSH5YYXAQ, seg_pos={0 0}, upload=446440]: Done uploading piece. err: <nil> Dec 26 20:14:36.625627025 (508h45m36.49671007s) [noise=true , address=145.79.227.45:4231 , piece_id=SXTYW3PLFDECYMRHYVTA6L5N6VI6KZEILG26K4ALKBEE3JDWJFPQ, seg_pos={0 0}, upload=446440]: Done uploading piece. err: <nil> Dec 26 20:14:36.628714234 (508h45m36.499797269s) [noise=true , address=145.79.227.42:4121 , piece_id=DNTY3PS7VF4DFKS75BEETF4IWLZW73ULTZ555US7OK4CPJJFULIQ, seg_pos={0 0}, upload=446440]: Done uploading piece. err: <nil> Dec 26 20:14:36.632592198 (508h45m36.503675233s) [noise=true , address=145.79.227.46:4602 , piece_id=QKSMTCTQV25F2DEQH2D2KADAINYCOJWFK33T6NVPIG2B37YF6TTQ, seg_pos={0 0}, upload=446440]: Done uploading piece. err: <nil> Dec 26 20:14:36.634769304 (508h45m36.505852339s) [noise=true , address=145.79.227.25:4618 , piece_id=7H5GZ2TFX35CUSEH6LVYVILR34W2XDGPNWTJ5M3UIPP727L2TRHQ, seg_pos={0 0}, upload=446440]: Done uploading piece. err: <nil> Dec 26 20:14:36.635827759 (508h45m36.506910794s) [noise=true , address=145.79.227.49:4338 , piece_id=MZL5YI7C7MLDE2CB47SQS2XWPLJ5VUDVWIAO2HO67O236NLZO2FA, seg_pos={0 0}, upload=446440]: Done uploading piece. err: <nil> Dec 26 20:14:36.639557481 (508h45m36.510640516s) [noise=true , address=145.79.227.37:4566 , piece_id=PWFS52EYRFKEU6LERET3W2QXB6ZSFICUOV3GPDUMEHKY3IAY43UA, seg_pos={0 0}, upload=446440]: Done uploading piece. err: <nil> Dec 26 20:14:36.651231290 (508h45m36.522314325s) [noise=true , address=145.79.227.3:4150 , piece_id=2E222HDKHDP3JIRHBSZOPKKO74JIBJ6D43ZCV7PU6YLNJYETJHSQ, seg_pos={0 0}, upload=446440]: Done uploading piece. err: <nil> Dec 26 20:14:36.824853844 (508h45m36.695936879s) [noise=true , address=145.79.227.15:4875 , piece_id=BT2HGGDEPGOVFUSERULJCM3ZNYK4QLH24ONVVJPI4BAXWAVUDWRA, seg_pos={0 0}, upload=446440]: Done uploading piece. err: <nil> Dec 26 20:14:36.915641057 (508h45m36.786724112s) [noise=true , address=45.91.116.9:4263 , piece_id=GHLDY5TXPAJFLMLMPYPJ2MFBNDUNPKMOP7I23CATXKXWQOQT5WWQ, seg_pos={0 0}, upload=446440]: Done uploading piece. err: <nil> Dec 26 20:14:36.955746433 (508h45m36.826829468s) [noise=true , address=45.91.116.14:4189 , piece_id=65A6IHIAES37HSTNBM5FUPZZAH35WFSLH6S3TBIII7O62VEQQ25A, seg_pos={0 0}, upload=446440]: Done uploading piece. err: <nil> Dec 26 20:14:36.964794603 (508h45m36.835877638s) [noise=true , address=45.91.116.7:4012 , piece_id=ROQL7PKQIQIKYVX74QAXLETAM4MRN5V4ELUK6CH2ZKKAOSSOGBBA, seg_pos={0 0}, upload=446440]: Done uploading piece. err: <nil> Dec 26 20:14:36.973931722 (508h45m36.845014757s) [noise=true , address=77.90.31.21:4877 , piece_id=MRY3PPVMCVH7WP667T3ENAWH6TRZNMNGK4ZWGOZETOY5AGE7RUZA, seg_pos={0 0}, upload=446440]: Done uploading piece. err: <nil> Dec 26 20:14:36.973953122 (508h45m36.845036157s) [seg_pos={0 0}, upload=446440]: Segment reached optimal threshold of 13 pieces. Dec 26 20:14:36.974139407 (508h45m36.845222442s) [noise=true , address=45.91.116.6:4975 , piece_id=7IRHTKTPFRL2JAPC3XCL3ZVHTUC22SBYDP2KVL6NTUVKMF6NWYRA, seg_pos={0 0}, upload=446440]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: write tcp 188.166.225.170:60138->45.91.116.6:4975: use of closed network connection; write tcp 188.166.225.170:60138->45.91.116.6:4975: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60138->45.91.116.6:4975: use of closed network connection Dec 26 20:14:36.974145839 (508h45m36.845228875s) [noise=true , address=45.91.116.6:4975 , piece_id=7IRHTKTPFRL2JAPC3XCL3ZVHTUC22SBYDP2KVL6NTUVKMF6NWYRA, seg_pos={0 0}, upload=446440]: Long tail cancellation due to optimal threshold Dec 26 20:14:36.974286838 (508h45m36.845369873s) [noise=true , address=145.79.227.35:4482 , piece_id=R67ESFGWQNGDORGMN5OZXPFKYSWN4U2RDWOXDUZN5JYNW5DGBTSQ, seg_pos={0 0}, upload=446440]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: expected piece hash; context canceled; EOF Dec 26 20:14:36.974292198 (508h45m36.845375233s) [noise=true , address=145.79.227.35:4482 , piece_id=R67ESFGWQNGDORGMN5OZXPFKYSWN4U2RDWOXDUZN5JYNW5DGBTSQ, seg_pos={0 0}, upload=446440]: Long tail cancellation due to optimal threshold Dec 26 20:14:36.974609763 (508h45m36.845693409s) [noise=true , address=45.91.116.8:4045 , piece_id=3FYKMIDJTCLANIBL6QNXRIW4ELDG5OSSQUNMAYR5ECMBPSYW2CIQ, seg_pos={0 0}, upload=446440]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 26 20:14:36.974632897 (508h45m36.845715962s) [noise=true , address=45.91.116.8:4045 , piece_id=3FYKMIDJTCLANIBL6QNXRIW4ELDG5OSSQUNMAYR5ECMBPSYW2CIQ, seg_pos={0 0}, upload=446440]: Long tail cancellation due to optimal threshold Dec 26 20:14:36.974668805 (508h45m36.84575184s) [seg_pos={0 0}, upload=446440]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 20:14:36.974801618 (508h45m36.845884653s) [upload=446440]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:14:36.974810725 (508h45m36.845893761s) [upload=446440]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:14:37.282277490 (508h45m37.153360525s) [upload=446440]: Done uploading segments... Dec 26 20:14:37.282296936 (508h45m37.153379971s) [upload=446440]: Upload finished. err: <nil> Duration: 30.003689168s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 29.628077965s trace: https://jaeger.dev.storj.io/trace/269ebf1e68ab4194 FAIL (2025-12-26 20:14:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== (508h13m19.087063583s) [noise=true , address=145.79.227.40:4061 , piece_id=46F3KZE7JNV3GD7VNUF3AW73RXBUZQPNNOFZYXKSVUEZQOWVBWUA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.216300006 (508h13m19.087383051s) [noise=true , address=145.79.227.30:4768 , piece_id=OHGULVS5PFX3YLWJCTJHVEKUIWX5MJLUJ3436WW5ZGRN3MALSNXA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.216730495 (508h13m19.08781353s) [noise=true , address=145.79.227.18:4019 , piece_id=V2YUVXSEVPYFQYCOHRGWO74I7ADIXBT7G6RHMEY7GJLI7MD3BCOQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217064441 (508h13m19.088147486s) [noise=true , address=77.90.31.10:4538 , piece_id=2YHRM6ACRDJFZWAB4BMWDKNQZ3C5IJ2D3AYGSLVH5FLGM6BSCXKQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217290591 (508h13m19.088373626s) [noise=true , address=45.91.116.13:4173 , piece_id=OVWQGJIDNS3MXQQF7DNCSKLLNQXFSDQEDRVVYY6YOLKHIPN3HBWA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217611703 (508h13m19.088694738s) [noise=true , address=145.79.227.16:4920 , piece_id=MNTVSKV4PMJW2F3A57JHRD2EKZUMG4YLRR7RF2QQK226HUBVZFWQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217940660 (508h13m19.089023705s) [noise=true , address=77.90.31.13:4646 , piece_id=OTKXAUXVS6WAVBJ72G3FYCD7ZZZZZO7DLYLCEOFCTQR3UZU5SJNA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.217996485 (508h13m19.08907952s) [noise=true , address=77.90.31.16:4745 , piece_id=MXGPB54M3YPAVE5UQDQN6ZQJKQTWYQJPOZE3VR5X5LJIIO33Y23Q, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.218363545 (508h13m19.08944659s) [noise=true , address=77.90.31.21:4904 , piece_id=EQVI6ALFENA3SP2NL3RDU4WMLGYP7M6ODPGTDQOQZTN7O5RF7QVA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.219020315 (508h13m19.090103361s) [noise=true , address=77.90.31.19:4862 , piece_id=GC7KS7DHD7LG6HALPCWDLX5CUDKIIPYCEBSDXFMMNCHRXM64KJBQ, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.219639084 (508h13m19.090722129s) [noise=true , address=145.79.227.29:4975 , piece_id=ZFMWT5IU7FM5MCABHYTXVNG2VUGPGMNCDCCOZKEVOLNC7DBCW36A, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.220085213 (508h13m19.091168248s) [noise=true , address=145.79.227.9:4372 , piece_id=SVA4YGNN2TNROPPY5VH455Q4KLBJTLQM2HIORTVGR5J7PL4XSTQA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.220586408 (508h13m19.091669443s) [noise=true , address=45.91.116.6:4987 , piece_id=LHYW5CZDAY6MSTOFWDWMUEY2LFBOTSTPL42UXHIYQFN4LA2LQGAA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:19.221122849 (508h13m19.092205894s) [noise=true , address=145.79.227.34:4915 , piece_id=AGQTKGU3FUJSCJ2NX4WB6OWIGBKCMK4D5BBL6UYVFWL3E3UZMGOA, seg_pos={0 0}, upload=445976]: Uploading piece... Dec 26 19:42:20.654199727 (508h13m20.525282772s) [upload=445976]: Next returned nil segment Dec 26 19:42:20.654213192 (508h13m20.525296237s) [upload=445976]: Waiting for error group managing segments... Dec 26 19:42:21.104984249 (508h13m20.976067294s) [noise=true , address=145.79.227.29:4975 , piece_id=ZFMWT5IU7FM5MCABHYTXVNG2VUGPGMNCDCCOZKEVOLNC7DBCW36A, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.129286805 (508h13m21.00036984s) [noise=true , address=145.79.227.18:4019 , piece_id=V2YUVXSEVPYFQYCOHRGWO74I7ADIXBT7G6RHMEY7GJLI7MD3BCOQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.153491254 (508h13m21.024574289s) [noise=true , address=145.79.227.9:4372 , piece_id=SVA4YGNN2TNROPPY5VH455Q4KLBJTLQM2HIORTVGR5J7PL4XSTQA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.285157470 (508h13m21.156240505s) [noise=true , address=77.90.31.18:4837 , piece_id=QDJSYSDH3AKR7WZ5PEAQZA5AKWF7KDKBHXVUVSER3EBZTI3MTUHQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.325205408 (508h13m21.196288443s) [noise=true , address=77.90.31.16:4745 , piece_id=MXGPB54M3YPAVE5UQDQN6ZQJKQTWYQJPOZE3VR5X5LJIIO33Y23Q, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.355521865 (508h13m21.2266049s) [noise=true , address=45.91.116.13:4173 , piece_id=OVWQGJIDNS3MXQQF7DNCSKLLNQXFSDQEDRVVYY6YOLKHIPN3HBWA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.386180926 (508h13m21.257263961s) [noise=true , address=77.90.31.19:4862 , piece_id=GC7KS7DHD7LG6HALPCWDLX5CUDKIIPYCEBSDXFMMNCHRXM64KJBQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:21.437176951 (508h13m21.308259996s) [noise=true , address=45.91.116.6:4987 , piece_id=LHYW5CZDAY6MSTOFWDWMUEY2LFBOTSTPL42UXHIYQFN4LA2LQGAA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.023103166 (508h13m21.894186211s) [noise=true , address=145.79.227.43:4177 , piece_id=UHJ2BQY3BDNPKNAMPXZ47OY45DRI6QLXTRSDTZSUYR6MWKVOL7OA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.050932298 (508h13m21.922015333s) [noise=true , address=77.90.31.21:4904 , piece_id=EQVI6ALFENA3SP2NL3RDU4WMLGYP7M6ODPGTDQOQZTN7O5RF7QVA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.098028337 (508h13m21.969111382s) [noise=true , address=145.79.227.44:4227 , piece_id=O5JCRQMK3YH2SEJD3JFU4Z2NO6JZZWT2DLGGZ4MTAOUUBEE7FBYQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.136785577 (508h13m22.007868612s) [noise=true , address=145.79.227.30:4768 , piece_id=OHGULVS5PFX3YLWJCTJHVEKUIWX5MJLUJ3436WW5ZGRN3MALSNXA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.236850869 (508h13m22.107933914s) [noise=true , address=145.79.227.40:4061 , piece_id=46F3KZE7JNV3GD7VNUF3AW73RXBUZQPNNOFZYXKSVUEZQOWVBWUA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.352167431 (508h13m22.223250536s) [noise=true , address=145.79.227.34:4915 , piece_id=AGQTKGU3FUJSCJ2NX4WB6OWIGBKCMK4D5BBL6UYVFWL3E3UZMGOA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: <nil> Dec 26 19:42:22.352194343 (508h13m22.223277398s) [seg_pos={0 0}, upload=445976]: Segment reached optimal threshold of 14 pieces. Dec 26 19:42:22.352561061 (508h13m22.223644137s) [noise=true , address=77.90.31.13:4646 , piece_id=OTKXAUXVS6WAVBJ72G3FYCD7ZZZZZO7DLYLCEOFCTQR3UZU5SJNA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5): protocol: write tcp 188.166.225.170:51910->77.90.31.13:4646: use of closed network connection; write tcp 188.166.225.170:51910->77.90.31.13:4646: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51910->77.90.31.13:4646: use of closed network connection Dec 26 19:42:22.352570299 (508h13m22.223653344s) [noise=true , address=77.90.31.13:4646 , piece_id=OTKXAUXVS6WAVBJ72G3FYCD7ZZZZZO7DLYLCEOFCTQR3UZU5SJNA, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352627187 (508h13m22.223710233s) [noise=true , address=77.90.31.10:4538 , piece_id=2YHRM6ACRDJFZWAB4BMWDKNQZ3C5IJ2D3AYGSLVH5FLGM6BSCXKQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N): protocol: expected piece hash; context canceled; EOF Dec 26 19:42:22.352648027 (508h13m22.223731062s) [noise=true , address=77.90.31.10:4538 , piece_id=2YHRM6ACRDJFZWAB4BMWDKNQZ3C5IJ2D3AYGSLVH5FLGM6BSCXKQ, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352717288 (508h13m22.223800334s) [noise=true , address=77.90.31.15:4702 , piece_id=FBW4Y6XYMCFEV35EFN4SEMC2PQRBCCERKFHI6WA5FO4ACSDLRCFA, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW): protocol: write tcp 188.166.225.170:47524->77.90.31.15:4702: use of closed network connection; write tcp 188.166.225.170:47524->77.90.31.15:4702: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47524->77.90.31.15:4702: use of closed network connection Dec 26 19:42:22.352725615 (508h13m22.22380866s) [noise=true , address=77.90.31.15:4702 , piece_id=FBW4Y6XYMCFEV35EFN4SEMC2PQRBCCERKFHI6WA5FO4ACSDLRCFA, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352737828 (508h13m22.223820863s) [noise=true , address=145.79.227.16:4920 , piece_id=MNTVSKV4PMJW2F3A57JHRD2EKZUMG4YLRR7RF2QQK226HUBVZFWQ, seg_pos={0 0}, upload=445976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb): protocol: expected piece hash; write tcp 188.166.225.170:60084->145.79.227.16:4920: use of closed network connection; write tcp 188.166.225.170:60084->145.79.227.16:4920: use of closed network connection; EOF Dec 26 19:42:22.352745392 (508h13m22.223828437s) [noise=true , address=145.79.227.16:4920 , piece_id=MNTVSKV4PMJW2F3A57JHRD2EKZUMG4YLRR7RF2QQK226HUBVZFWQ, seg_pos={0 0}, upload=445976]: Long tail cancellation due to optimal threshold Dec 26 19:42:22.352903484 (508h13m22.223986529s) [seg_pos={0 0}, upload=445976]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 19:42:22.353035194 (508h13m22.224118239s) [upload=445976]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 19:42:22.353042468 (508h13m22.224125513s) [upload=445976]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 19:42:22.651634339 (508h13m22.522717374s) [upload=445976]: Done uploading segments... Dec 26 19:42:22.651653245 (508h13m22.52273628s) [upload=445976]: Upload finished. err: <nil> Duration: 29.628101789s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [.XXXXX......X.X.X.X.X..............XX..X..X...........XXX.X.] duration: 6.144699186s trace: https://jaeger.dev.storj.io/trace/7eb8c3dc1d9f462 FAIL (2025-12-26 20:14:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: ULVBMQ3CY6NUCVSKXYI7HHJU7ML3OJGENS7HAA3BCHVQWHJAU3HA) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: 7YTDLKRI5VIQN6ZAHMPPNPA2EBNYWKUVEEJ2QET3PUG7U37AR2XA) context deadline exceeded; piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: HO5A4VS3E5N3MXVSEVW47S65MUKRA6HZIZ5LHQMXHSWTX2QPCTKA) context deadline exceeded; piecestore: (Node ID: 12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, Piece ID: VSX4KSUOXEX722TSSANNSCGFMG5TPTPNSK2SRX5Y4T5A5CXV5OHQ) context deadline exceeded ======LOGS====== c 26 20:13:30.255904555 (508h44m30.1269876s) [upload=446422]: Starting upload Dec 26 20:13:30.256022840 (508h44m30.127105885s) [upload=446422]: Done starting upload Dec 26 20:13:30.256105388 (508h44m30.127188423s) [upload=446422]: Uploading segments... Dec 26 20:13:30.256283427 (508h44m30.127366472s) [upload=446422]: Got next segment. Inline: false Dec 26 20:13:30.256353240 (508h44m30.127436275s) [upload=446422]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 20:13:30.256360033 (508h44m30.127443068s) [upload=446422]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 20:13:30.504911562 (508h44m30.375994618s) [seg_pos={0 0}, upload=446422]: Begin upload segment... Dec 26 20:13:30.504968291 (508h44m30.376051326s) [seg_pos={0 0}, upload=446422]: Done begin upload segment. Dec 26 20:13:30.505009940 (508h44m30.376093005s) [noise=true , address=145.79.227.5:4223 , piece_id=VSX4KSUOXEX722TSSANNSCGFMG5TPTPNSK2SRX5Y4T5A5CXV5OHQ, seg_pos={0 0}, upload=446422]: Uploading piece... Dec 26 20:13:30.505068542 (508h44m30.376151577s) [noise=true , address=145.79.227.37:4566 , piece_id=6THP2PXD54DRYVUOML6X3TOE74YS52OVQ5YAFRCXHJF2NVYTJTUA, seg_pos={0 0}, upload=446422]: Uploading piece... Dec 26 20:13:30.505857063 (508h44m30.376940118s) [noise=true , address=145.79.227.25:4618 , piece_id=XK62F6Y3YP3G62ZTZZHI53H37WRCGD5PJEK4FA43NSVCBO3EVLEQ, seg_pos={0 0}, upload=446422]: Uploading piece... Dec 26 20:13:30.505923760 (508h44m30.377006805s) [noise=true , address=45.91.116.6:4975 , piece_id=EEVL524JN6NZOM34Y7GXUPJB2JXAUGVAZW4IORF56QN5WDHJJUPQ, seg_pos={0 0}, upload=446422]: Uploading piece... Dec 26 20:13:30.506292352 (508h44m30.377375387s) [noise=true , address=145.79.227.6:4259 , piece_id=5D2YIQPAA34SA5NPPMT25AMCZSJB2DWP3OD4QVISWJ4UNU2DD7EQ, seg_pos={0 0}, upload=446422]: Uploading piece... Dec 26 20:13:30.506745395 (508h44m30.37782844s) [noise=true , address=145.79.227.29:4942 , piece_id=YWTUOG2JCJPM3YFNMR4OVVDT7D5YMHWSMMTXGPLJTEDIBDICDO3Q, seg_pos={0 0}, upload=446422]: Uploading piece... Dec 26 20:13:30.507126350 (508h44m30.378209816s) [noise=true , address=145.79.227.22:4509 , piece_id=BN7UBUYEBPJBGIAABJUCZ2HYIAIZCBY7V2N5XKLZR5E2RMB4OKOQ, seg_pos={0 0}, upload=446422]: Uploading piece... Dec 26 20:13:30.507493379 (508h44m30.378576414s) [noise=true , address=145.79.227.38:4977 , piece_id=NG6UWQV2RR2SNAXDRW56M3XMT6J7IQEYP7IKZ5RAOHVCQLDS62HA, seg_pos={0 0}, upload=446422]: Uploading piece... Dec 26 20:13:30.507837003 (508h44m30.378920058s) [noise=true , address=145.79.227.31:4799 , piece_id=SLZ6U4OQUCFPJA5CFALCPZO44EKSKYM3CJX2QVKFHMFVC2OGX4RQ, seg_pos={0 0}, upload=446422]: Uploading piece... Dec 26 20:13:30.508355170 (508h44m30.379438205s) [noise=true , address=77.90.31.21:4880 , piece_id=ULVBMQ3CY6NUCVSKXYI7HHJU7ML3OJGENS7HAA3BCHVQWHJAU3HA, seg_pos={0 0}, upload=446422]: Uploading piece... Dec 26 20:13:30.508734031 (508h44m30.379817076s) [noise=true , address=145.79.227.30:4761 , piece_id=L5XWDVIVYP7HRUGMXYQF2CKDRW36HNNHPJ5E5M4BP33T3BTZ3D6Q, seg_pos={0 0}, upload=446422]: Uploading piece... Dec 26 20:13:30.509153139 (508h44m30.380236174s) [noise=true , address=45.91.116.14:4189 , piece_id=7YTDLKRI5VIQN6ZAHMPPNPA2EBNYWKUVEEJ2QET3PUG7U37AR2XA, seg_pos={0 0}, upload=446422]: Uploading piece... Dec 26 20:13:30.509564182 (508h44m30.380647217s) [noise=true , address=145.79.227.11:4730 , piece_id=LQPR24JJVHUTWDOE6D67BQPF6FEEOL65LRVR3SETPUJHXK3W4HZA, seg_pos={0 0}, upload=446422]: Uploading piece... Dec 26 20:13:30.509940668 (508h44m30.381023703s) [noise=true , address=145.79.227.36:4517 , piece_id=7Z5VWPBXIIDRWXKHCXYY4K2RHVJGD3RJTIOBZ6Z2UDOO64YCHR7Q, seg_pos={0 0}, upload=446422]: Uploading piece... Dec 26 20:13:30.510990307 (508h44m30.382073342s) [noise=true , address=45.91.116.11:4118 , piece_id=HO5A4VS3E5N3MXVSEVW47S65MUKRA6HZIZ5LHQMXHSWTX2QPCTKA, seg_pos={0 0}, upload=446422]: Uploading piece... Dec 26 20:13:30.511425135 (508h44m30.38250817s) [noise=true , address=145.79.227.48:4302 , piece_id=JKOWG5NXYRYSZJWC5WAP4EOS4UKCNRV7YB7TCUCR3THAREKL2LSA, seg_pos={0 0}, upload=446422]: Uploading piece... Dec 26 20:13:32.214275616 (508h44m32.085358651s) [upload=446422]: Next returned nil segment Dec 26 20:13:32.214291817 (508h44m32.085374852s) [upload=446422]: Waiting for error group managing segments... Dec 26 20:13:32.508523589 (508h44m32.379606635s) [noise=true , address=145.79.227.38:4977 , piece_id=NG6UWQV2RR2SNAXDRW56M3XMT6J7IQEYP7IKZ5RAOHVCQLDS62HA, seg_pos={0 0}, upload=446422]: Done uploading piece. err: <nil> Dec 26 20:13:32.509628013 (508h44m32.380711048s) [noise=true , address=145.79.227.48:4302 , piece_id=JKOWG5NXYRYSZJWC5WAP4EOS4UKCNRV7YB7TCUCR3THAREKL2LSA, seg_pos={0 0}, upload=446422]: Done uploading piece. err: <nil> Dec 26 20:13:32.512947394 (508h44m32.384030429s) [noise=true , address=145.79.227.29:4942 , piece_id=YWTUOG2JCJPM3YFNMR4OVVDT7D5YMHWSMMTXGPLJTEDIBDICDO3Q, seg_pos={0 0}, upload=446422]: Done uploading piece. err: <nil> Dec 26 20:13:32.513373014 (508h44m32.384456049s) [noise=true , address=145.79.227.22:4509 , piece_id=BN7UBUYEBPJBGIAABJUCZ2HYIAIZCBY7V2N5XKLZR5E2RMB4OKOQ, seg_pos={0 0}, upload=446422]: Done uploading piece. err: <nil> Dec 26 20:13:32.518879399 (508h44m32.389962434s) [noise=true , address=145.79.227.31:4799 , piece_id=SLZ6U4OQUCFPJA5CFALCPZO44EKSKYM3CJX2QVKFHMFVC2OGX4RQ, seg_pos={0 0}, upload=446422]: Done uploading piece. err: <nil> Dec 26 20:13:32.524877440 (508h44m32.395960485s) [noise=true , address=145.79.227.11:4730 , piece_id=LQPR24JJVHUTWDOE6D67BQPF6FEEOL65LRVR3SETPUJHXK3W4HZA, seg_pos={0 0}, upload=446422]: Done uploading piece. err: <nil> Dec 26 20:13:32.528606681 (508h44m32.399689716s) [noise=true , address=145.79.227.6:4259 , piece_id=5D2YIQPAA34SA5NPPMT25AMCZSJB2DWP3OD4QVISWJ4UNU2DD7EQ, seg_pos={0 0}, upload=446422]: Done uploading piece. err: <nil> Dec 26 20:13:32.532434430 (508h44m32.403517475s) [noise=true , address=145.79.227.36:4517 , piece_id=7Z5VWPBXIIDRWXKHCXYY4K2RHVJGD3RJTIOBZ6Z2UDOO64YCHR7Q, seg_pos={0 0}, upload=446422]: Done uploading piece. err: <nil> Dec 26 20:13:32.540544673 (508h44m32.411627708s) [noise=true , address=145.79.227.37:4566 , piece_id=6THP2PXD54DRYVUOML6X3TOE74YS52OVQ5YAFRCXHJF2NVYTJTUA, seg_pos={0 0}, upload=446422]: Done uploading piece. err: <nil> Dec 26 20:13:33.000249101 (508h44m32.871332146s) [noise=true , address=45.91.116.14:4189 , piece_id=7YTDLKRI5VIQN6ZAHMPPNPA2EBNYWKUVEEJ2QET3PUG7U37AR2XA, seg_pos={0 0}, upload=446422]: Done uploading piece. err: <nil> Dec 26 20:13:33.775658741 (508h44m33.646741786s) [noise=true , address=77.90.31.21:4880 , piece_id=ULVBMQ3CY6NUCVSKXYI7HHJU7ML3OJGENS7HAA3BCHVQWHJAU3HA, seg_pos={0 0}, upload=446422]: Done uploading piece. err: <nil> Dec 26 20:13:33.803403391 (508h44m33.674486427s) [noise=true , address=45.91.116.11:4118 , piece_id=HO5A4VS3E5N3MXVSEVW47S65MUKRA6HZIZ5LHQMXHSWTX2QPCTKA, seg_pos={0 0}, upload=446422]: Done uploading piece. err: <nil> Dec 26 20:13:34.138384406 (508h44m34.009467441s) [noise=true , address=145.79.227.5:4223 , piece_id=VSX4KSUOXEX722TSSANNSCGFMG5TPTPNSK2SRX5Y4T5A5CXV5OHQ, seg_pos={0 0}, upload=446422]: Done uploading piece. err: <nil> Dec 26 20:13:34.138424493 (508h44m34.009507538s) [seg_pos={0 0}, upload=446422]: Segment reached optimal threshold of 13 pieces. Dec 26 20:13:34.138777035 (508h44m34.00986008s) [noise=true , address=145.79.227.25:4618 , piece_id=XK62F6Y3YP3G62ZTZZHI53H37WRCGD5PJEK4FA43NSVCBO3EVLEQ, seg_pos={0 0}, upload=446422]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8): protocol: expected piece hash; context canceled; EOF Dec 26 20:13:34.138787494 (508h44m34.009870529s) [noise=true , address=145.79.227.25:4618 , piece_id=XK62F6Y3YP3G62ZTZZHI53H37WRCGD5PJEK4FA43NSVCBO3EVLEQ, seg_pos={0 0}, upload=446422]: Long tail cancellation due to optimal threshold Dec 26 20:13:34.138891713 (508h44m34.009974748s) [noise=true , address=145.79.227.30:4761 , piece_id=L5XWDVIVYP7HRUGMXYQF2CKDRW36HNNHPJ5E5M4BP33T3BTZ3D6Q, seg_pos={0 0}, upload=446422]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 26 20:13:34.138902623 (508h44m34.009985668s) [noise=true , address=145.79.227.30:4761 , piece_id=L5XWDVIVYP7HRUGMXYQF2CKDRW36HNNHPJ5E5M4BP33T3BTZ3D6Q, seg_pos={0 0}, upload=446422]: Long tail cancellation due to optimal threshold Dec 26 20:13:34.139071996 (508h44m34.010155041s) [noise=true , address=45.91.116.6:4975 , piece_id=EEVL524JN6NZOM34Y7GXUPJB2JXAUGVAZW4IORF56QN5WDHJJUPQ, seg_pos={0 0}, upload=446422]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: write tcp 188.166.225.170:37800->45.91.116.6:4975: use of closed network connection; write tcp 188.166.225.170:37800->45.91.116.6:4975: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37800->45.91.116.6:4975: use of closed network connection Dec 26 20:13:34.139080772 (508h44m34.010163807s) [noise=true , address=45.91.116.6:4975 , piece_id=EEVL524JN6NZOM34Y7GXUPJB2JXAUGVAZW4IORF56QN5WDHJJUPQ, seg_pos={0 0}, upload=446422]: Long tail cancellation due to optimal threshold Dec 26 20:13:34.139103416 (508h44m34.010186451s) [seg_pos={0 0}, upload=446422]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 20:13:34.139181664 (508h44m34.01026471s) [upload=446422]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:13:34.139189811 (508h44m34.010272886s) [upload=446422]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:13:34.440108426 (508h44m34.311191471s) [upload=446422]: Done uploading segments... Dec 26 20:13:34.440126200 (508h44m34.311209235s) [upload=446422]: Upload finished. err: <nil> Duration: 6.144718041s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [X.XXXXX......X.X.X.X.X..............XX..X..X...........XXX.X] duration: 30.003124896s trace: https://jaeger.dev.storj.io/trace/7eb8c3dc1d9f462 FAIL (2025-12-26 20:13:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: ULVBMQ3CY6NUCVSKXYI7HHJU7ML3OJGENS7HAA3BCHVQWHJAU3HA) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: 7YTDLKRI5VIQN6ZAHMPPNPA2EBNYWKUVEEJ2QET3PUG7U37AR2XA) context deadline exceeded; piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: HO5A4VS3E5N3MXVSEVW47S65MUKRA6HZIZ5LHQMXHSWTX2QPCTKA) context deadline exceeded; piecestore: (Node ID: 12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, Piece ID: VSX4KSUOXEX722TSSANNSCGFMG5TPTPNSK2SRX5Y4T5A5CXV5OHQ) context deadline exceeded ======LOGS====== c 26 20:13:30.255904555 (508h44m30.1269876s) [upload=446422]: Starting upload Dec 26 20:13:30.256022840 (508h44m30.127105885s) [upload=446422]: Done starting upload Dec 26 20:13:30.256105388 (508h44m30.127188423s) [upload=446422]: Uploading segments... Dec 26 20:13:30.256283427 (508h44m30.127366472s) [upload=446422]: Got next segment. Inline: false Dec 26 20:13:30.256353240 (508h44m30.127436275s) [upload=446422]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 20:13:30.256360033 (508h44m30.127443068s) [upload=446422]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 20:13:30.504911562 (508h44m30.375994618s) [seg_pos={0 0}, upload=446422]: Begin upload segment... Dec 26 20:13:30.504968291 (508h44m30.376051326s) [seg_pos={0 0}, upload=446422]: Done begin upload segment. Dec 26 20:13:30.505009940 (508h44m30.376093005s) [noise=true , address=145.79.227.5:4223 , piece_id=VSX4KSUOXEX722TSSANNSCGFMG5TPTPNSK2SRX5Y4T5A5CXV5OHQ, seg_pos={0 0}, upload=446422]: Uploading piece... Dec 26 20:13:30.505068542 (508h44m30.376151577s) [noise=true , address=145.79.227.37:4566 , piece_id=6THP2PXD54DRYVUOML6X3TOE74YS52OVQ5YAFRCXHJF2NVYTJTUA, seg_pos={0 0}, upload=446422]: Uploading piece... Dec 26 20:13:30.505857063 (508h44m30.376940118s) [noise=true , address=145.79.227.25:4618 , piece_id=XK62F6Y3YP3G62ZTZZHI53H37WRCGD5PJEK4FA43NSVCBO3EVLEQ, seg_pos={0 0}, upload=446422]: Uploading piece... Dec 26 20:13:30.505923760 (508h44m30.377006805s) [noise=true , address=45.91.116.6:4975 , piece_id=EEVL524JN6NZOM34Y7GXUPJB2JXAUGVAZW4IORF56QN5WDHJJUPQ, seg_pos={0 0}, upload=446422]: Uploading piece... Dec 26 20:13:30.506292352 (508h44m30.377375387s) [noise=true , address=145.79.227.6:4259 , piece_id=5D2YIQPAA34SA5NPPMT25AMCZSJB2DWP3OD4QVISWJ4UNU2DD7EQ, seg_pos={0 0}, upload=446422]: Uploading piece... Dec 26 20:13:30.506745395 (508h44m30.37782844s) [noise=true , address=145.79.227.29:4942 , piece_id=YWTUOG2JCJPM3YFNMR4OVVDT7D5YMHWSMMTXGPLJTEDIBDICDO3Q, seg_pos={0 0}, upload=446422]: Uploading piece... Dec 26 20:13:30.507126350 (508h44m30.378209816s) [noise=true , address=145.79.227.22:4509 , piece_id=BN7UBUYEBPJBGIAABJUCZ2HYIAIZCBY7V2N5XKLZR5E2RMB4OKOQ, seg_pos={0 0}, upload=446422]: Uploading piece... Dec 26 20:13:30.507493379 (508h44m30.378576414s) [noise=true , address=145.79.227.38:4977 , piece_id=NG6UWQV2RR2SNAXDRW56M3XMT6J7IQEYP7IKZ5RAOHVCQLDS62HA, seg_pos={0 0}, upload=446422]: Uploading piece... Dec 26 20:13:30.507837003 (508h44m30.378920058s) [noise=true , address=145.79.227.31:4799 , piece_id=SLZ6U4OQUCFPJA5CFALCPZO44EKSKYM3CJX2QVKFHMFVC2OGX4RQ, seg_pos={0 0}, upload=446422]: Uploading piece... Dec 26 20:13:30.508355170 (508h44m30.379438205s) [noise=true , address=77.90.31.21:4880 , piece_id=ULVBMQ3CY6NUCVSKXYI7HHJU7ML3OJGENS7HAA3BCHVQWHJAU3HA, seg_pos={0 0}, upload=446422]: Uploading piece... Dec 26 20:13:30.508734031 (508h44m30.379817076s) [noise=true , address=145.79.227.30:4761 , piece_id=L5XWDVIVYP7HRUGMXYQF2CKDRW36HNNHPJ5E5M4BP33T3BTZ3D6Q, seg_pos={0 0}, upload=446422]: Uploading piece... Dec 26 20:13:30.509153139 (508h44m30.380236174s) [noise=true , address=45.91.116.14:4189 , piece_id=7YTDLKRI5VIQN6ZAHMPPNPA2EBNYWKUVEEJ2QET3PUG7U37AR2XA, seg_pos={0 0}, upload=446422]: Uploading piece... Dec 26 20:13:30.509564182 (508h44m30.380647217s) [noise=true , address=145.79.227.11:4730 , piece_id=LQPR24JJVHUTWDOE6D67BQPF6FEEOL65LRVR3SETPUJHXK3W4HZA, seg_pos={0 0}, upload=446422]: Uploading piece... Dec 26 20:13:30.509940668 (508h44m30.381023703s) [noise=true , address=145.79.227.36:4517 , piece_id=7Z5VWPBXIIDRWXKHCXYY4K2RHVJGD3RJTIOBZ6Z2UDOO64YCHR7Q, seg_pos={0 0}, upload=446422]: Uploading piece... Dec 26 20:13:30.510990307 (508h44m30.382073342s) [noise=true , address=45.91.116.11:4118 , piece_id=HO5A4VS3E5N3MXVSEVW47S65MUKRA6HZIZ5LHQMXHSWTX2QPCTKA, seg_pos={0 0}, upload=446422]: Uploading piece... Dec 26 20:13:30.511425135 (508h44m30.38250817s) [noise=true , address=145.79.227.48:4302 , piece_id=JKOWG5NXYRYSZJWC5WAP4EOS4UKCNRV7YB7TCUCR3THAREKL2LSA, seg_pos={0 0}, upload=446422]: Uploading piece... Dec 26 20:13:32.214275616 (508h44m32.085358651s) [upload=446422]: Next returned nil segment Dec 26 20:13:32.214291817 (508h44m32.085374852s) [upload=446422]: Waiting for error group managing segments... Dec 26 20:13:32.508523589 (508h44m32.379606635s) [noise=true , address=145.79.227.38:4977 , piece_id=NG6UWQV2RR2SNAXDRW56M3XMT6J7IQEYP7IKZ5RAOHVCQLDS62HA, seg_pos={0 0}, upload=446422]: Done uploading piece. err: <nil> Dec 26 20:13:32.509628013 (508h44m32.380711048s) [noise=true , address=145.79.227.48:4302 , piece_id=JKOWG5NXYRYSZJWC5WAP4EOS4UKCNRV7YB7TCUCR3THAREKL2LSA, seg_pos={0 0}, upload=446422]: Done uploading piece. err: <nil> Dec 26 20:13:32.512947394 (508h44m32.384030429s) [noise=true , address=145.79.227.29:4942 , piece_id=YWTUOG2JCJPM3YFNMR4OVVDT7D5YMHWSMMTXGPLJTEDIBDICDO3Q, seg_pos={0 0}, upload=446422]: Done uploading piece. err: <nil> Dec 26 20:13:32.513373014 (508h44m32.384456049s) [noise=true , address=145.79.227.22:4509 , piece_id=BN7UBUYEBPJBGIAABJUCZ2HYIAIZCBY7V2N5XKLZR5E2RMB4OKOQ, seg_pos={0 0}, upload=446422]: Done uploading piece. err: <nil> Dec 26 20:13:32.518879399 (508h44m32.389962434s) [noise=true , address=145.79.227.31:4799 , piece_id=SLZ6U4OQUCFPJA5CFALCPZO44EKSKYM3CJX2QVKFHMFVC2OGX4RQ, seg_pos={0 0}, upload=446422]: Done uploading piece. err: <nil> Dec 26 20:13:32.524877440 (508h44m32.395960485s) [noise=true , address=145.79.227.11:4730 , piece_id=LQPR24JJVHUTWDOE6D67BQPF6FEEOL65LRVR3SETPUJHXK3W4HZA, seg_pos={0 0}, upload=446422]: Done uploading piece. err: <nil> Dec 26 20:13:32.528606681 (508h44m32.399689716s) [noise=true , address=145.79.227.6:4259 , piece_id=5D2YIQPAA34SA5NPPMT25AMCZSJB2DWP3OD4QVISWJ4UNU2DD7EQ, seg_pos={0 0}, upload=446422]: Done uploading piece. err: <nil> Dec 26 20:13:32.532434430 (508h44m32.403517475s) [noise=true , address=145.79.227.36:4517 , piece_id=7Z5VWPBXIIDRWXKHCXYY4K2RHVJGD3RJTIOBZ6Z2UDOO64YCHR7Q, seg_pos={0 0}, upload=446422]: Done uploading piece. err: <nil> Dec 26 20:13:32.540544673 (508h44m32.411627708s) [noise=true , address=145.79.227.37:4566 , piece_id=6THP2PXD54DRYVUOML6X3TOE74YS52OVQ5YAFRCXHJF2NVYTJTUA, seg_pos={0 0}, upload=446422]: Done uploading piece. err: <nil> Dec 26 20:13:33.000249101 (508h44m32.871332146s) [noise=true , address=45.91.116.14:4189 , piece_id=7YTDLKRI5VIQN6ZAHMPPNPA2EBNYWKUVEEJ2QET3PUG7U37AR2XA, seg_pos={0 0}, upload=446422]: Done uploading piece. err: <nil> Dec 26 20:13:33.775658741 (508h44m33.646741786s) [noise=true , address=77.90.31.21:4880 , piece_id=ULVBMQ3CY6NUCVSKXYI7HHJU7ML3OJGENS7HAA3BCHVQWHJAU3HA, seg_pos={0 0}, upload=446422]: Done uploading piece. err: <nil> Dec 26 20:13:33.803403391 (508h44m33.674486427s) [noise=true , address=45.91.116.11:4118 , piece_id=HO5A4VS3E5N3MXVSEVW47S65MUKRA6HZIZ5LHQMXHSWTX2QPCTKA, seg_pos={0 0}, upload=446422]: Done uploading piece. err: <nil> Dec 26 20:13:34.138384406 (508h44m34.009467441s) [noise=true , address=145.79.227.5:4223 , piece_id=VSX4KSUOXEX722TSSANNSCGFMG5TPTPNSK2SRX5Y4T5A5CXV5OHQ, seg_pos={0 0}, upload=446422]: Done uploading piece. err: <nil> Dec 26 20:13:34.138424493 (508h44m34.009507538s) [seg_pos={0 0}, upload=446422]: Segment reached optimal threshold of 13 pieces. Dec 26 20:13:34.138777035 (508h44m34.00986008s) [noise=true , address=145.79.227.25:4618 , piece_id=XK62F6Y3YP3G62ZTZZHI53H37WRCGD5PJEK4FA43NSVCBO3EVLEQ, seg_pos={0 0}, upload=446422]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8): protocol: expected piece hash; context canceled; EOF Dec 26 20:13:34.138787494 (508h44m34.009870529s) [noise=true , address=145.79.227.25:4618 , piece_id=XK62F6Y3YP3G62ZTZZHI53H37WRCGD5PJEK4FA43NSVCBO3EVLEQ, seg_pos={0 0}, upload=446422]: Long tail cancellation due to optimal threshold Dec 26 20:13:34.138891713 (508h44m34.009974748s) [noise=true , address=145.79.227.30:4761 , piece_id=L5XWDVIVYP7HRUGMXYQF2CKDRW36HNNHPJ5E5M4BP33T3BTZ3D6Q, seg_pos={0 0}, upload=446422]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 26 20:13:34.138902623 (508h44m34.009985668s) [noise=true , address=145.79.227.30:4761 , piece_id=L5XWDVIVYP7HRUGMXYQF2CKDRW36HNNHPJ5E5M4BP33T3BTZ3D6Q, seg_pos={0 0}, upload=446422]: Long tail cancellation due to optimal threshold Dec 26 20:13:34.139071996 (508h44m34.010155041s) [noise=true , address=45.91.116.6:4975 , piece_id=EEVL524JN6NZOM34Y7GXUPJB2JXAUGVAZW4IORF56QN5WDHJJUPQ, seg_pos={0 0}, upload=446422]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: write tcp 188.166.225.170:37800->45.91.116.6:4975: use of closed network connection; write tcp 188.166.225.170:37800->45.91.116.6:4975: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37800->45.91.116.6:4975: use of closed network connection Dec 26 20:13:34.139080772 (508h44m34.010163807s) [noise=true , address=45.91.116.6:4975 , piece_id=EEVL524JN6NZOM34Y7GXUPJB2JXAUGVAZW4IORF56QN5WDHJJUPQ, seg_pos={0 0}, upload=446422]: Long tail cancellation due to optimal threshold Dec 26 20:13:34.139103416 (508h44m34.010186451s) [seg_pos={0 0}, upload=446422]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 20:13:34.139181664 (508h44m34.01026471s) [upload=446422]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:13:34.139189811 (508h44m34.010272886s) [upload=446422]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:13:34.440108426 (508h44m34.311191471s) [upload=446422]: Done uploading segments... Dec 26 20:13:34.440126200 (508h44m34.311209235s) [upload=446422]: Upload finished. err: <nil> Duration: 30.003156819s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)