Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [......X.X..X....XX.XX........X.X......X..X......XXX..X....XX] duration: 30.245447299s trace: https://jaeger.dev.storj.io/trace/5861a143d3414808 FAIL (2025-12-31 02:45:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 31 02:44:39.596223661 (611h15m39.467306696s) [upload=535685]: Starting upload Dec 31 02:44:39.596276562 (611h15m39.467359597s) [upload=535685]: Done starting upload Dec 31 02:44:39.596418161 (611h15m39.467501206s) [upload=535685]: Uploading segments... Dec 31 02:44:39.596479538 (611h15m39.467562573s) [upload=535685]: Got next segment. Inline: false Dec 31 02:44:39.596514685 (611h15m39.46759773s) [upload=535685]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 02:44:39.596520276 (611h15m39.467603321s) [upload=535685]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 02:44:39.852867672 (611h15m39.723950707s) [seg_pos={0 0}, upload=535685]: Begin upload segment... Dec 31 02:44:39.852906226 (611h15m39.723989261s) [seg_pos={0 0}, upload=535685]: Done begin upload segment. Dec 31 02:44:39.852926373 (611h15m39.724009408s) [noise=true , address=77.90.31.21:4878 , piece_id=X63TTVAEWQR2NR5CHWSOB4IUGZZUQCW5RVBXMRQSVTGXBFJXWGTQ, seg_pos={0 0}, upload=535685]: Uploading piece... Dec 31 02:44:39.856507673 (611h15m39.727590708s) [noise=true , address=145.79.227.31:4797 , piece_id=2YSF2TZRX53ZF6OSRTCP5RG6KXIADNLC3FMUDPOXMLD5CAZJIX2Q, seg_pos={0 0}, upload=535685]: Uploading piece... Dec 31 02:44:39.857681318 (611h15m39.728764353s) [noise=true , address=145.79.227.37:4567 , piece_id=NUQOWRV4PWZ2NIRXMTPEONXO4RCO4SICVFQPXSBZHLK5TCVL2KGQ, seg_pos={0 0}, upload=535685]: Uploading piece... Dec 31 02:44:39.859114166 (611h15m39.730197211s) [noise=true , address=145.79.227.25:4618 , piece_id=6FZQGOBVMLHNZWRCIJL6ESQBBCOMPN2GQSWGCEZOECKUFPRWPL7A, seg_pos={0 0}, upload=535685]: Uploading piece... Dec 31 02:44:39.860393041 (611h15m39.731476076s) [noise=true , address=145.79.227.48:4302 , piece_id=U6FDFK2SEA3C3YHEW6QDWYHEDV5CG2Z6LXEXQGIKQYZKS7INXVWA, seg_pos={0 0}, upload=535685]: Uploading piece... Dec 31 02:44:39.861406531 (611h15m39.732489566s) [noise=true , address=145.79.227.27:4689 , piece_id=VUE7TKCUJXXU3CAYJV3R5LIF2I5USMQIW5LGLD54ACA5CPMYLFKA, seg_pos={0 0}, upload=535685]: Uploading piece... Dec 31 02:44:39.862817277 (611h15m39.733900322s) [noise=true , address=145.79.227.5:4222 , piece_id=F2YFBUS3LP4P36FL7HMUEUWVFB3YVLUNM4IMK5NTA52XFFHMSUXQ, seg_pos={0 0}, upload=535685]: Uploading piece... Dec 31 02:44:39.862827426 (611h15m39.733910461s) [noise=true , address=145.79.227.8:4331 , piece_id=UMGABP4ODLBCNX4P2JO5MWEOV77ZLP2JAUAM7AWJUPH4KYPPVC6Q, seg_pos={0 0}, upload=535685]: Uploading piece... Dec 31 02:44:39.863889149 (611h15m39.734972194s) [noise=true , address=145.79.227.17:4971 , piece_id=IDVWBBZCNYSYZBILCNO4PRULO6HE7L23ZCPAQP5B3Z6WM5GC5ZEA, seg_pos={0 0}, upload=535685]: Uploading piece... Dec 31 02:44:39.863991744 (611h15m39.735074789s) [noise=true , address=145.79.227.51:4411 , piece_id=QBGSPZP3POLKHKUVRXNTY55KMEX2XCJO4JWF65MB3ZF6GLXB4YPQ, seg_pos={0 0}, upload=535685]: Uploading piece... Dec 31 02:44:39.865049949 (611h15m39.736132984s) [noise=true , address=145.79.227.11:4729 , piece_id=INHNZTTUJ7EFNGAUZGUMIZEJ6CJ3774M3RHYRU6XFWAKDXBBN4BQ, seg_pos={0 0}, upload=535685]: Uploading piece... Dec 31 02:44:39.865157774 (611h15m39.736240809s) [noise=true , address=45.91.116.8:4047 , piece_id=F3ULIDQHUQVXNQ6FBWNI2TF6RZXTHKRLHM5FA2F2OYPMDEZ4RHCQ, seg_pos={0 0}, upload=535685]: Uploading piece... Dec 31 02:44:39.866550796 (611h15m39.737633831s) [noise=true , address=145.79.227.13:4802 , piece_id=CJMWLTGYBO43VBBXJUL55CUKIVVQO6XGAVUROHHSKYSHZNYD4TJQ, seg_pos={0 0}, upload=535685]: Uploading piece... Dec 31 02:44:39.867652925 (611h15m39.73873597s) [noise=true , address=145.79.227.9:4366 , piece_id=XKYFNDBD67SOAU3ICJA4JYKMJYE4BPXXXYM37BC7HQ2PYDYGLHXQ, seg_pos={0 0}, upload=535685]: Uploading piece... Dec 31 02:44:39.868237698 (611h15m39.739320733s) [noise=true , address=145.79.227.28:4726 , piece_id=Y6ELOV3F2Z67DYUUF4AGRD722VYFIMEJM6NPJGMEPNFTSS4CKJVQ, seg_pos={0 0}, upload=535685]: Uploading piece... Dec 31 02:44:39.869440579 (611h15m39.740523614s) [noise=true , address=45.91.116.5:4300 , piece_id=LLJL3HGGPWADONVOP64Y3KRYTYXJ6KG6KXHZ7VDHVO2V6DVURRJA, seg_pos={0 0}, upload=535685]: Uploading piece... Dec 31 02:44:41.791794652 (611h15m41.662877687s) [upload=535685]: Next returned nil segment Dec 31 02:44:41.791813719 (611h15m41.662896764s) [upload=535685]: Waiting for error group managing segments... Dec 31 02:44:42.123587816 (611h15m41.994670851s) [noise=true , address=77.90.31.21:4878 , piece_id=X63TTVAEWQR2NR5CHWSOB4IUGZZUQCW5RVBXMRQSVTGXBFJXWGTQ, seg_pos={0 0}, upload=535685]: Done uploading piece. err: <nil> Dec 31 02:44:42.146086106 (611h15m42.017169141s) [noise=true , address=45.91.116.5:4300 , piece_id=LLJL3HGGPWADONVOP64Y3KRYTYXJ6KG6KXHZ7VDHVO2V6DVURRJA, seg_pos={0 0}, upload=535685]: Done uploading piece. err: <nil> Dec 31 02:44:42.165218396 (611h15m42.036301431s) [noise=true , address=45.91.116.8:4047 , piece_id=F3ULIDQHUQVXNQ6FBWNI2TF6RZXTHKRLHM5FA2F2OYPMDEZ4RHCQ, seg_pos={0 0}, upload=535685]: Done uploading piece. err: <nil> Dec 31 02:44:43.216603170 (611h15m43.087686205s) [noise=true , address=145.79.227.27:4689 , piece_id=VUE7TKCUJXXU3CAYJV3R5LIF2I5USMQIW5LGLD54ACA5CPMYLFKA, seg_pos={0 0}, upload=535685]: Done uploading piece. err: <nil> Dec 31 02:44:43.507652634 (611h15m43.378735679s) [noise=true , address=145.79.227.37:4567 , piece_id=NUQOWRV4PWZ2NIRXMTPEONXO4RCO4SICVFQPXSBZHLK5TCVL2KGQ, seg_pos={0 0}, upload=535685]: Done uploading piece. err: <nil> Dec 31 02:44:43.808499973 (611h15m43.679583008s) [noise=true , address=145.79.227.9:4366 , piece_id=XKYFNDBD67SOAU3ICJA4JYKMJYE4BPXXXYM37BC7HQ2PYDYGLHXQ, seg_pos={0 0}, upload=535685]: Done uploading piece. err: <nil> Dec 31 02:44:53.799691310 (611h15m53.670774366s) [noise=true , address=145.79.227.25:4618 , piece_id=6FZQGOBVMLHNZWRCIJL6ESQBBCOMPN2GQSWGCEZOECKUFPRWPL7A, seg_pos={0 0}, upload=535685]: Done uploading piece. err: <nil> Dec 31 02:44:54.993655723 (611h15m54.864738768s) [noise=true , address=145.79.227.17:4971 , piece_id=IDVWBBZCNYSYZBILCNO4PRULO6HE7L23ZCPAQP5B3Z6WM5GC5ZEA, seg_pos={0 0}, upload=535685]: Done uploading piece. err: <nil> Dec 31 02:44:56.232139659 (611h15m56.103222704s) [noise=true , address=145.79.227.11:4729 , piece_id=INHNZTTUJ7EFNGAUZGUMIZEJ6CJ3774M3RHYRU6XFWAKDXBBN4BQ, seg_pos={0 0}, upload=535685]: Done uploading piece. err: <nil> Dec 31 02:44:59.302479019 (611h15m59.173562054s) [noise=true , address=145.79.227.5:4222 , piece_id=F2YFBUS3LP4P36FL7HMUEUWVFB3YVLUNM4IMK5NTA52XFFHMSUXQ, seg_pos={0 0}, upload=535685]: Done uploading piece. err: <nil> Dec 31 02:45:08.239336734 (611h16m8.110419779s) [noise=true , address=145.79.227.28:4726 , piece_id=Y6ELOV3F2Z67DYUUF4AGRD722VYFIMEJM6NPJGMEPNFTSS4CKJVQ, seg_pos={0 0}, upload=535685]: Done uploading piece. err: <nil> Dec 31 02:45:08.590789552 (611h16m8.461872587s) [noise=true , address=145.79.227.31:4797 , piece_id=2YSF2TZRX53ZF6OSRTCP5RG6KXIADNLC3FMUDPOXMLD5CAZJIX2Q, seg_pos={0 0}, upload=535685]: Done uploading piece. err: <nil> Dec 31 02:45:08.904557925 (611h16m8.77564097s) [noise=true , address=145.79.227.51:4411 , piece_id=QBGSPZP3POLKHKUVRXNTY55KMEX2XCJO4JWF65MB3ZF6GLXB4YPQ, seg_pos={0 0}, upload=535685]: Done uploading piece. err: ecclient: upload failed (node:1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR, address:145.79.227.51:4411): protocol: write tcp 188.166.225.170:60440->145.79.227.51:4411: use of closed network connection; write tcp 188.166.225.170:60440->145.79.227.51:4411: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60440->145.79.227.51:4411: use of closed network connection Dec 31 02:45:08.904659438 (611h16m8.775742473s) [noise=true , address=145.79.227.13:4802 , piece_id=CJMWLTGYBO43VBBXJUL55CUKIVVQO6XGAVUROHHSKYSHZNYD4TJQ, seg_pos={0 0}, upload=535685]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: write tcp 188.166.225.170:58990->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:58990->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58990->145.79.227.13:4802: use of closed network connection Dec 31 02:45:08.904711658 (611h16m8.775794713s) [noise=true , address=145.79.227.8:4331 , piece_id=UMGABP4ODLBCNX4P2JO5MWEOV77ZLP2JAUAM7AWJUPH4KYPPVC6Q, seg_pos={0 0}, upload=535685]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: write tcp 188.166.225.170:44658->145.79.227.8:4331: use of closed network connection; write tcp 188.166.225.170:44658->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44658->145.79.227.8:4331: use of closed network connection Dec 31 02:45:08.904793383 (611h16m8.775876418s) [noise=true , address=145.79.227.48:4302 , piece_id=U6FDFK2SEA3C3YHEW6QDWYHEDV5CG2Z6LXEXQGIKQYZKS7INXVWA, seg_pos={0 0}, upload=535685]: Done uploading piece. err: ecclient: upload failed (node:1tGpuYdPLiy4ncDjxrZbQfp7Bq79NVBHDmr4vjaL4qMpudrvgR, address:145.79.227.48:4302): protocol: write tcp 188.166.225.170:40652->145.79.227.48:4302: use of closed network connection; write tcp 188.166.225.170:40652->145.79.227.48:4302: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40652->145.79.227.48:4302: use of closed network connection Dec 31 02:45:08.904817660 (611h16m8.775900695s) [seg_pos={0 0}, upload=535685]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 31 02:45:09.148806163 (611h16m9.019889198s) [upload=535685]: Done uploading segments... Dec 31 02:45:09.148862481 (611h16m9.019945516s) [upload=535685]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.245459242s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 58.945249735s trace: https://jaeger.dev.storj.io/trace/365693fea00df28 FAIL (2025-12-31 02:45:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [noise=true , address=145.79.227.49:4343 , piece_id=6L3HPTS7D2NRA4QEOH4RFM3D3B53CTCONRI6G6YVU4OOTUWTG7XA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.942060847 (610h25m24.813143892s) [noise=true , address=145.79.227.34:4937 , piece_id=5W7LBM3YJJTVUFYTELR57DVXB663OHO5N3Z2SPQPZ4G43IK47O3Q, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.942684104 (610h25m24.813767139s) [noise=true , address=64.49.13.139:4478 , piece_id=QN6RT2KWEA6APQVA66ZMKG4LH537UGYNOXOCVVA2YC2MKDOLT3YA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.943064007 (610h25m24.814147042s) [noise=true , address=145.79.227.40:4084 , piece_id=RUKTKDII6BRUWSYVS65CWUBT3QKG3ASUK3KA3O5VGE53XSG6IMOQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.943459560 (610h25m24.814542605s) [noise=true , address=145.79.227.14:4847 , piece_id=JX5NXTMVM3NSWIEVR4RX46ABSNPKTOFDLFBPKEXFQ3G3ANKV5OLA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.943865303 (610h25m24.814948348s) [noise=true , address=45.91.116.6:4981 , piece_id=CT474ZA2LVAZEBJHBDD4GB6NTHVHZFJLNE5OZV3DFS2NZKVW3EQQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.944259694 (610h25m24.815342749s) [noise=true , address=145.79.227.51:4443 , piece_id=73UUMRDY7BSZNCGBXCZYYRBVE5KLPMNPRNXOZEGWKQGLHJS32J2Q, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.946575182 (610h25m24.817658227s) [noise=true , address=77.90.31.10:4523 , piece_id=P6LIONXJI6AFZ2JTKCQ2W6RZCOZCK6MZ53EXAJKXYQXSX2PX4QEA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.948132909 (610h25m24.819215944s) [noise=true , address=145.79.227.20:4452 , piece_id=ZSDN2JZ3FQUVJYN5OURDXNKEDLDNULJOHKKRZVIPXZIWIYAHCRXQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.949605774 (610h25m24.820688819s) [noise=true , address=77.90.31.14:4680 , piece_id=S2E4WBA7MJVAZLT5JG3IMHK53RQC77CCHV2PDF6L2R2JEA2U3K4Q, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.951459002 (610h25m24.822542048s) [noise=true , address=193.149.164.143:4704 , piece_id=BHXHFBKTLTDMGXWY5QRTP2THZQE5OXFZ2LCMTO2KRYRWAHXWW3NQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.952986742 (610h25m24.824069777s) [noise=true , address=77.90.31.19:4848 , piece_id=VZYG32S6ODIPDKQYZRXECBHXOERZDDWD2OF35Z46W4XMUZ3OJLAA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.956916654 (610h25m24.827999689s) [noise=true , address=145.79.227.29:4971 , piece_id=MEPXHHOQ6SUZC3GIPCQEMOKKF5OJTKZEI3UYZBAYXAQZGSVEKSBA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.960665062 (610h25m24.831748097s) [noise=true , address=64.49.13.170:4720 , piece_id=HIATRSAPE4YLSODFZDPP43ZUMLDVM7GJEVMWE4ZGUQTP25LTHGTA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.962366001 (610h25m24.833449036s) [noise=true , address=145.79.227.2:4117 , piece_id=4SGFGB3IJ7LTXDMMWM2PGGOV64MC46IZ7OZQWH55J4NBQY7CHMZQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.963708216 (610h25m24.834791261s) [noise=true , address=145.79.227.35:4505 , piece_id=V7RS7A4QLZTH4G32ZUZNSF2Z5APVWZGJO77NEGP3SBWQT3WCMVLQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:26.352236138 (610h25m26.223319173s) [upload=534952]: Next returned nil segment Dec 31 01:54:26.352253821 (610h25m26.223336857s) [upload=534952]: Waiting for error group managing segments... Dec 31 01:54:26.812758374 (610h25m26.683841409s) [noise=true , address=145.79.227.51:4443 , piece_id=73UUMRDY7BSZNCGBXCZYYRBVE5KLPMNPRNXOZEGWKQGLHJS32J2Q, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.824371056 (610h25m26.695454101s) [noise=true , address=145.79.227.14:4847 , piece_id=JX5NXTMVM3NSWIEVR4RX46ABSNPKTOFDLFBPKEXFQ3G3ANKV5OLA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.838414046 (610h25m26.709497081s) [noise=true , address=64.49.13.138:4490 , piece_id=2XTUZE6PXRE3OARNJVT36FXXWUMYIBVNE6M2VVFAVBGKEE2H5YZQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.875499841 (610h25m26.746582876s) [noise=true , address=64.49.13.139:4478 , piece_id=QN6RT2KWEA6APQVA66ZMKG4LH537UGYNOXOCVVA2YC2MKDOLT3YA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.891607221 (610h25m26.762690256s) [noise=true , address=145.79.227.20:4452 , piece_id=ZSDN2JZ3FQUVJYN5OURDXNKEDLDNULJOHKKRZVIPXZIWIYAHCRXQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.988561223 (610h25m26.859644268s) [noise=true , address=64.49.13.170:4720 , piece_id=HIATRSAPE4YLSODFZDPP43ZUMLDVM7GJEVMWE4ZGUQTP25LTHGTA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.039035911 (610h25m26.910118946s) [noise=true , address=77.90.31.14:4680 , piece_id=S2E4WBA7MJVAZLT5JG3IMHK53RQC77CCHV2PDF6L2R2JEA2U3K4Q, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.098023090 (610h25m26.969106126s) [noise=true , address=45.91.116.6:4981 , piece_id=CT474ZA2LVAZEBJHBDD4GB6NTHVHZFJLNE5OZV3DFS2NZKVW3EQQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.653790754 (610h25m27.524873789s) [noise=true , address=77.90.31.21:4886 , piece_id=K34WPIBGGGEPHIKJ253XGLDCIQ3WLFN4AZ6EYNWVHHT2EIHS6YRQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.684192972 (610h25m27.555276017s) [noise=true , address=193.149.164.143:4704 , piece_id=BHXHFBKTLTDMGXWY5QRTP2THZQE5OXFZ2LCMTO2KRYRWAHXWW3NQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.717238425 (610h25m27.58832146s) [noise=true , address=145.79.227.40:4084 , piece_id=RUKTKDII6BRUWSYVS65CWUBT3QKG3ASUK3KA3O5VGE53XSG6IMOQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.718349742 (610h25m27.589432777s) [noise=true , address=77.90.31.10:4523 , piece_id=P6LIONXJI6AFZ2JTKCQ2W6RZCOZCK6MZ53EXAJKXYQXSX2PX4QEA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.803639562 (610h25m27.674722597s) [noise=true , address=145.79.227.29:4971 , piece_id=MEPXHHOQ6SUZC3GIPCQEMOKKF5OJTKZEI3UYZBAYXAQZGSVEKSBA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.812791279 (610h25m27.683874314s) [noise=true , address=145.79.227.49:4343 , piece_id=6L3HPTS7D2NRA4QEOH4RFM3D3B53CTCONRI6G6YVU4OOTUWTG7XA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.812812348 (610h25m27.683895393s) [seg_pos={0 0}, upload=534952]: Segment reached optimal threshold of 14 pieces. Dec 31 01:54:27.813018000 (610h25m27.684101035s) [noise=true , address=145.79.227.35:4505 , piece_id=V7RS7A4QLZTH4G32ZUZNSF2Z5APVWZGJO77NEGP3SBWQT3WCMVLQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i6itGoBhLsLiqC8et8Y5L62fLjcWxy18u39NAr4R3nXNMWcsf): protocol: expected piece hash; context canceled; EOF Dec 31 01:54:27.813024772 (610h25m27.684107808s) [noise=true , address=145.79.227.35:4505 , piece_id=V7RS7A4QLZTH4G32ZUZNSF2Z5APVWZGJO77NEGP3SBWQT3WCMVLQ, seg_pos={0 0}, upload=534952]: Long tail cancellation due to optimal threshold Dec 31 01:54:27.813175931 (610h25m27.684258976s) [noise=true , address=145.79.227.34:4937 , piece_id=5W7LBM3YJJTVUFYTELR57DVXB663OHO5N3Z2SPQPZ4G43IK47O3Q, seg_pos={0 0}, upload=534952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1LKrFThdW3jJpG6dRZLUaxLXTa5Wy27yQ8PGUEqjvBWfFoRB94): protocol: expected piece hash; context canceled; EOF Dec 31 01:54:27.813181702 (610h25m27.684264737s) [noise=true , address=145.79.227.34:4937 , piece_id=5W7LBM3YJJTVUFYTELR57DVXB663OHO5N3Z2SPQPZ4G43IK47O3Q, seg_pos={0 0}, upload=534952]: Long tail cancellation due to optimal threshold Dec 31 01:54:27.813407862 (610h25m27.684490907s) [noise=true , address=145.79.227.2:4117 , piece_id=4SGFGB3IJ7LTXDMMWM2PGGOV64MC46IZ7OZQWH55J4NBQY7CHMZQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122Htg72f5eUUutf1p3fYQpy1Wkf6AgdaQV83HHYye2n8zm3the): protocol: expected piece hash; context canceled; EOF Dec 31 01:54:27.813414264 (610h25m27.684497299s) [noise=true , address=145.79.227.2:4117 , piece_id=4SGFGB3IJ7LTXDMMWM2PGGOV64MC46IZ7OZQWH55J4NBQY7CHMZQ, seg_pos={0 0}, upload=534952]: Long tail cancellation due to optimal threshold Dec 31 01:54:27.813424614 (610h25m27.684507649s) [noise=true , address=77.90.31.19:4848 , piece_id=VZYG32S6ODIPDKQYZRXECBHXOERZDDWD2OF35Z46W4XMUZ3OJLAA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD): protocol: write tcp 188.166.225.170:42842->77.90.31.19:4848: use of closed network connection; write tcp 188.166.225.170:42842->77.90.31.19:4848: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42842->77.90.31.19:4848: use of closed network connection Dec 31 01:54:27.813432750 (610h25m27.684515785s) [noise=true , address=77.90.31.19:4848 , piece_id=VZYG32S6ODIPDKQYZRXECBHXOERZDDWD2OF35Z46W4XMUZ3OJLAA, seg_pos={0 0}, upload=534952]: Long tail cancellation due to optimal threshold Dec 31 01:54:27.813474088 (610h25m27.684557123s) [seg_pos={0 0}, upload=534952]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 01:54:27.813535325 (610h25m27.68461836s) [upload=534952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 01:54:27.813539613 (610h25m27.684622648s) [upload=534952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 01:54:28.099921034 (610h25m27.971004069s) [upload=534952]: Done uploading segments... Dec 31 01:54:28.099940451 (610h25m27.971023486s) [upload=534952]: Upload finished. err: <nil> Duration: 58.945277868s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [............X...............................X....X..........] duration: 3.497012217s trace: https://jaeger.dev.storj.io/trace/7bc7478624e56c1a FAIL (2025-12-31 02:44:47 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.497039499s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [.......X.X..X....XX.XX........X.X......X..X......XXX..X....X] duration: 30.244854853s trace: https://jaeger.dev.storj.io/trace/5733e6b8828d1d4e FAIL (2025-12-31 02:44:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== WHHCV5R2U46QIU2K63HNQ, seg_pos={0 0}, upload=535678]: Uploading piece... Dec 31 02:44:09.362159357 (611h15m9.233242392s) [noise=true , address=145.79.227.14:4837 , piece_id=KXIMCSLRWSS5XEWCYJFNXB5VYJ5Y3BNJGM6JNCFA6YUHLEB4RNJA, seg_pos={0 0}, upload=535678]: Uploading piece... Dec 31 02:44:09.362612690 (611h15m9.233695725s) [noise=true , address=45.91.116.9:4263 , piece_id=N2YZIF7QU7DNK7S4AVD4J74AEL662ODUXIYAV43AVZZPYOYPOJOA, seg_pos={0 0}, upload=535678]: Uploading piece... Dec 31 02:44:09.362832047 (611h15m9.233915082s) [noise=true , address=145.79.227.47:4265 , piece_id=GU2KU4DNI27YAYCPB63NMJUSG6NW2PXN24S22KQLQKBLGTLMYAAQ, seg_pos={0 0}, upload=535678]: Uploading piece... Dec 31 02:44:09.362987904 (611h15m9.234070949s) [noise=true , address=45.91.116.5:4300 , piece_id=SQBONRUXVZMCCFZNXEHNLDZBE4YCMKWCXSVFG6UKT7MQVCMWUA4A, seg_pos={0 0}, upload=535678]: Uploading piece... Dec 31 02:44:09.363208664 (611h15m9.234291699s) [noise=true , address=45.91.116.14:4192 , piece_id=UWVDTGQGCDAD4QSGVCSQJDVDLDYDRH2U5E6RT6KHTUMGFZNAIOTA, seg_pos={0 0}, upload=535678]: Uploading piece... Dec 31 02:44:09.363347118 (611h15m9.234430153s) [noise=true , address=145.79.227.37:4566 , piece_id=3XYK67GCTOOKZEDDMPQZBOGDBP4HVVWXQTMC65ANLN7J6MHPM2HQ, seg_pos={0 0}, upload=535678]: Uploading piece... Dec 31 02:44:09.363572187 (611h15m9.234655222s) [noise=true , address=145.79.227.9:4366 , piece_id=UV5QV4LVS5YJKKJUOLWIJRRD7BH3AZFTGLOEDSTD5H3JB5OLG26A, seg_pos={0 0}, upload=535678]: Uploading piece... Dec 31 02:44:09.363695942 (611h15m9.234778977s) [noise=true , address=145.79.227.28:4726 , piece_id=AU3WQM2FTOGNICPAUG4Z6OSEUD2F7TOGLVRSHIP7N6GPUHO3IZRA, seg_pos={0 0}, upload=535678]: Uploading piece... Dec 31 02:44:09.363922715 (611h15m9.23500576s) [noise=true , address=145.79.227.16:4909 , piece_id=KQIQ3SWFJUGHMYOKEMGCGTKTYLPTQUF7KZ5ET4BY6YZTLOKVZDOQ, seg_pos={0 0}, upload=535678]: Uploading piece... Dec 31 02:44:09.364057100 (611h15m9.235140135s) [noise=true , address=145.79.227.15:4874 , piece_id=P6P4V5NANMDJBZQABXJGLV734SMTAITWJV6NBDPLDBNJYH4G4ZLQ, seg_pos={0 0}, upload=535678]: Uploading piece... Dec 31 02:44:09.364416254 (611h15m9.235499289s) [noise=true , address=145.79.227.34:4905 , piece_id=EKOHHQ66GQLV7KTVQFU5KNBZDHGNJWGRSAJAHYSLWCFK53WVO52A, seg_pos={0 0}, upload=535678]: Uploading piece... Dec 31 02:44:09.364767513 (611h15m9.235850558s) [noise=true , address=45.91.116.11:4120 , piece_id=FMNF7HCSDVBXBWZ5K4O2XNG2TCGYJSXRDZ2KGGJMSUXLGBCPEMLQ, seg_pos={0 0}, upload=535678]: Uploading piece... Dec 31 02:44:09.365126757 (611h15m9.236209792s) [noise=true , address=145.79.227.25:4618 , piece_id=KIMH2VDLLQSPG3OBUNCO4YVX33ZGEFTX6MC3QW7US2PQOUA4RMXQ, seg_pos={0 0}, upload=535678]: Uploading piece... Dec 31 02:44:09.365507732 (611h15m9.236590777s) [noise=true , address=145.79.227.52:4447 , piece_id=2IS3OZX5BE4WKLEC46BEEWVVR5WUZ6DTJAINTLCEX4ODZMYTLOFA, seg_pos={0 0}, upload=535678]: Uploading piece... Dec 31 02:44:11.374258098 (611h15m11.245341143s) [upload=535678]: Next returned nil segment Dec 31 02:44:11.374273687 (611h15m11.245356722s) [upload=535678]: Waiting for error group managing segments... Dec 31 02:44:11.734001363 (611h15m11.605084398s) [noise=true , address=45.91.116.5:4300 , piece_id=SQBONRUXVZMCCFZNXEHNLDZBE4YCMKWCXSVFG6UKT7MQVCMWUA4A, seg_pos={0 0}, upload=535678]: Done uploading piece. err: <nil> Dec 31 02:44:12.034450315 (611h15m11.90553336s) [noise=true , address=45.91.116.9:4263 , piece_id=N2YZIF7QU7DNK7S4AVD4J74AEL662ODUXIYAV43AVZZPYOYPOJOA, seg_pos={0 0}, upload=535678]: Done uploading piece. err: <nil> Dec 31 02:44:12.446813955 (611h15m12.31789699s) [noise=true , address=45.91.116.14:4192 , piece_id=UWVDTGQGCDAD4QSGVCSQJDVDLDYDRH2U5E6RT6KHTUMGFZNAIOTA, seg_pos={0 0}, upload=535678]: Done uploading piece. err: <nil> Dec 31 02:44:12.717586789 (611h15m12.588669824s) [noise=true , address=45.91.116.11:4120 , piece_id=FMNF7HCSDVBXBWZ5K4O2XNG2TCGYJSXRDZ2KGGJMSUXLGBCPEMLQ, seg_pos={0 0}, upload=535678]: Done uploading piece. err: <nil> Dec 31 02:44:15.627815978 (611h15m15.498899013s) [noise=true , address=145.79.227.28:4726 , piece_id=AU3WQM2FTOGNICPAUG4Z6OSEUD2F7TOGLVRSHIP7N6GPUHO3IZRA, seg_pos={0 0}, upload=535678]: Done uploading piece. err: <nil> Dec 31 02:44:16.184796107 (611h15m16.055879152s) [noise=true , address=145.79.227.52:4447 , piece_id=2IS3OZX5BE4WKLEC46BEEWVVR5WUZ6DTJAINTLCEX4ODZMYTLOFA, seg_pos={0 0}, upload=535678]: Done uploading piece. err: <nil> Dec 31 02:44:20.094473709 (611h15m19.965556744s) [noise=true , address=145.79.227.14:4837 , piece_id=KXIMCSLRWSS5XEWCYJFNXB5VYJ5Y3BNJGM6JNCFA6YUHLEB4RNJA, seg_pos={0 0}, upload=535678]: Done uploading piece. err: <nil> Dec 31 02:44:26.790088215 (611h15m26.66117126s) [noise=true , address=145.79.227.37:4566 , piece_id=3XYK67GCTOOKZEDDMPQZBOGDBP4HVVWXQTMC65ANLN7J6MHPM2HQ, seg_pos={0 0}, upload=535678]: Done uploading piece. err: <nil> Dec 31 02:44:38.420799641 (611h15m38.291882686s) [noise=true , address=145.79.227.23:4547 , piece_id=42Q3YHUR3ZVEWNKVX6PIKGID5YDULLLWHHCV5R2U46QIU2K63HNQ, seg_pos={0 0}, upload=535678]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: write tcp 188.166.225.170:54306->145.79.227.23:4547: use of closed network connection; write tcp 188.166.225.170:54306->145.79.227.23:4547: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54306->145.79.227.23:4547: use of closed network connection Dec 31 02:44:38.420921403 (611h15m38.292004449s) [noise=true , address=145.79.227.16:4909 , piece_id=KQIQ3SWFJUGHMYOKEMGCGTKTYLPTQUF7KZ5ET4BY6YZTLOKVZDOQ, seg_pos={0 0}, upload=535678]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: write tcp 188.166.225.170:57758->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:57758->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57758->145.79.227.16:4909: use of closed network connection Dec 31 02:44:38.421022216 (611h15m38.292105251s) [noise=true , address=145.79.227.15:4874 , piece_id=P6P4V5NANMDJBZQABXJGLV734SMTAITWJV6NBDPLDBNJYH4G4ZLQ, seg_pos={0 0}, upload=535678]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:56854->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:56854->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56854->145.79.227.15:4874: use of closed network connection Dec 31 02:44:38.421131043 (611h15m38.292214098s) [noise=true , address=145.79.227.34:4905 , piece_id=EKOHHQ66GQLV7KTVQFU5KNBZDHGNJWGRSAJAHYSLWCFK53WVO52A, seg_pos={0 0}, upload=535678]: Done uploading piece. err: ecclient: upload failed (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9, address:145.79.227.34:4905): protocol: rpc: dial tcp 145.79.227.34:4905: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 31 02:44:38.421182300 (611h15m38.292265345s) [noise=true , address=145.79.227.17:4972 , piece_id=NPJG44CQAVFAROOMAOZXZRTM5QPCDD7VIW73NHFOKMTHX3OE5LAQ, seg_pos={0 0}, upload=535678]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: write tcp 188.166.225.170:36170->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:36170->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36170->145.79.227.17:4972: use of closed network connection Dec 31 02:44:38.421350260 (611h15m38.292433295s) [noise=true , address=145.79.227.9:4366 , piece_id=UV5QV4LVS5YJKKJUOLWIJRRD7BH3AZFTGLOEDSTD5H3JB5OLG26A, seg_pos={0 0}, upload=535678]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:46884->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:46884->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46884->145.79.227.9:4366: use of closed network connection Dec 31 02:44:38.421435502 (611h15m38.292518547s) [noise=true , address=145.79.227.25:4618 , piece_id=KIMH2VDLLQSPG3OBUNCO4YVX33ZGEFTX6MC3QW7US2PQOUA4RMXQ, seg_pos={0 0}, upload=535678]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 188.166.225.170:57768->145.79.227.25:4618: use of closed network connection; write tcp 188.166.225.170:57768->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57768->145.79.227.25:4618: use of closed network connection Dec 31 02:44:38.421587371 (611h15m38.292670407s) [noise=true , address=145.79.227.47:4265 , piece_id=GU2KU4DNI27YAYCPB63NMJUSG6NW2PXN24S22KQLQKBLGTLMYAAQ, seg_pos={0 0}, upload=535678]: Done uploading piece. err: ecclient: upload failed (node:12JvnJetuaEkstkPVNChKo8YQEYC68qRXPGQxx2AKZoHftkhajQ, address:145.79.227.47:4265): protocol: read tcp 188.166.225.170:54256->145.79.227.47:4265: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 31 02:44:38.421609353 (611h15m38.292692388s) [seg_pos={0 0}, upload=535678]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Dec 31 02:44:38.664156904 (611h15m38.535239949s) [upload=535678]: Done uploading segments... Dec 31 02:44:38.664198794 (611h15m38.535281829s) [upload=535678]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.244866194s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.............X...............................X....X.........] duration: 3.556996463s trace: https://jaeger.dev.storj.io/trace/7bc7478624e56c1a FAIL (2025-12-31 02:44:31 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.55701544s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 22.102012213s trace: https://jaeger.dev.storj.io/trace/365693fea00df28 FAIL (2025-12-31 02:44:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [noise=true , address=145.79.227.49:4343 , piece_id=6L3HPTS7D2NRA4QEOH4RFM3D3B53CTCONRI6G6YVU4OOTUWTG7XA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.942060847 (610h25m24.813143892s) [noise=true , address=145.79.227.34:4937 , piece_id=5W7LBM3YJJTVUFYTELR57DVXB663OHO5N3Z2SPQPZ4G43IK47O3Q, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.942684104 (610h25m24.813767139s) [noise=true , address=64.49.13.139:4478 , piece_id=QN6RT2KWEA6APQVA66ZMKG4LH537UGYNOXOCVVA2YC2MKDOLT3YA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.943064007 (610h25m24.814147042s) [noise=true , address=145.79.227.40:4084 , piece_id=RUKTKDII6BRUWSYVS65CWUBT3QKG3ASUK3KA3O5VGE53XSG6IMOQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.943459560 (610h25m24.814542605s) [noise=true , address=145.79.227.14:4847 , piece_id=JX5NXTMVM3NSWIEVR4RX46ABSNPKTOFDLFBPKEXFQ3G3ANKV5OLA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.943865303 (610h25m24.814948348s) [noise=true , address=45.91.116.6:4981 , piece_id=CT474ZA2LVAZEBJHBDD4GB6NTHVHZFJLNE5OZV3DFS2NZKVW3EQQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.944259694 (610h25m24.815342749s) [noise=true , address=145.79.227.51:4443 , piece_id=73UUMRDY7BSZNCGBXCZYYRBVE5KLPMNPRNXOZEGWKQGLHJS32J2Q, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.946575182 (610h25m24.817658227s) [noise=true , address=77.90.31.10:4523 , piece_id=P6LIONXJI6AFZ2JTKCQ2W6RZCOZCK6MZ53EXAJKXYQXSX2PX4QEA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.948132909 (610h25m24.819215944s) [noise=true , address=145.79.227.20:4452 , piece_id=ZSDN2JZ3FQUVJYN5OURDXNKEDLDNULJOHKKRZVIPXZIWIYAHCRXQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.949605774 (610h25m24.820688819s) [noise=true , address=77.90.31.14:4680 , piece_id=S2E4WBA7MJVAZLT5JG3IMHK53RQC77CCHV2PDF6L2R2JEA2U3K4Q, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.951459002 (610h25m24.822542048s) [noise=true , address=193.149.164.143:4704 , piece_id=BHXHFBKTLTDMGXWY5QRTP2THZQE5OXFZ2LCMTO2KRYRWAHXWW3NQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.952986742 (610h25m24.824069777s) [noise=true , address=77.90.31.19:4848 , piece_id=VZYG32S6ODIPDKQYZRXECBHXOERZDDWD2OF35Z46W4XMUZ3OJLAA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.956916654 (610h25m24.827999689s) [noise=true , address=145.79.227.29:4971 , piece_id=MEPXHHOQ6SUZC3GIPCQEMOKKF5OJTKZEI3UYZBAYXAQZGSVEKSBA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.960665062 (610h25m24.831748097s) [noise=true , address=64.49.13.170:4720 , piece_id=HIATRSAPE4YLSODFZDPP43ZUMLDVM7GJEVMWE4ZGUQTP25LTHGTA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.962366001 (610h25m24.833449036s) [noise=true , address=145.79.227.2:4117 , piece_id=4SGFGB3IJ7LTXDMMWM2PGGOV64MC46IZ7OZQWH55J4NBQY7CHMZQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.963708216 (610h25m24.834791261s) [noise=true , address=145.79.227.35:4505 , piece_id=V7RS7A4QLZTH4G32ZUZNSF2Z5APVWZGJO77NEGP3SBWQT3WCMVLQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:26.352236138 (610h25m26.223319173s) [upload=534952]: Next returned nil segment Dec 31 01:54:26.352253821 (610h25m26.223336857s) [upload=534952]: Waiting for error group managing segments... Dec 31 01:54:26.812758374 (610h25m26.683841409s) [noise=true , address=145.79.227.51:4443 , piece_id=73UUMRDY7BSZNCGBXCZYYRBVE5KLPMNPRNXOZEGWKQGLHJS32J2Q, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.824371056 (610h25m26.695454101s) [noise=true , address=145.79.227.14:4847 , piece_id=JX5NXTMVM3NSWIEVR4RX46ABSNPKTOFDLFBPKEXFQ3G3ANKV5OLA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.838414046 (610h25m26.709497081s) [noise=true , address=64.49.13.138:4490 , piece_id=2XTUZE6PXRE3OARNJVT36FXXWUMYIBVNE6M2VVFAVBGKEE2H5YZQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.875499841 (610h25m26.746582876s) [noise=true , address=64.49.13.139:4478 , piece_id=QN6RT2KWEA6APQVA66ZMKG4LH537UGYNOXOCVVA2YC2MKDOLT3YA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.891607221 (610h25m26.762690256s) [noise=true , address=145.79.227.20:4452 , piece_id=ZSDN2JZ3FQUVJYN5OURDXNKEDLDNULJOHKKRZVIPXZIWIYAHCRXQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.988561223 (610h25m26.859644268s) [noise=true , address=64.49.13.170:4720 , piece_id=HIATRSAPE4YLSODFZDPP43ZUMLDVM7GJEVMWE4ZGUQTP25LTHGTA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.039035911 (610h25m26.910118946s) [noise=true , address=77.90.31.14:4680 , piece_id=S2E4WBA7MJVAZLT5JG3IMHK53RQC77CCHV2PDF6L2R2JEA2U3K4Q, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.098023090 (610h25m26.969106126s) [noise=true , address=45.91.116.6:4981 , piece_id=CT474ZA2LVAZEBJHBDD4GB6NTHVHZFJLNE5OZV3DFS2NZKVW3EQQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.653790754 (610h25m27.524873789s) [noise=true , address=77.90.31.21:4886 , piece_id=K34WPIBGGGEPHIKJ253XGLDCIQ3WLFN4AZ6EYNWVHHT2EIHS6YRQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.684192972 (610h25m27.555276017s) [noise=true , address=193.149.164.143:4704 , piece_id=BHXHFBKTLTDMGXWY5QRTP2THZQE5OXFZ2LCMTO2KRYRWAHXWW3NQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.717238425 (610h25m27.58832146s) [noise=true , address=145.79.227.40:4084 , piece_id=RUKTKDII6BRUWSYVS65CWUBT3QKG3ASUK3KA3O5VGE53XSG6IMOQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.718349742 (610h25m27.589432777s) [noise=true , address=77.90.31.10:4523 , piece_id=P6LIONXJI6AFZ2JTKCQ2W6RZCOZCK6MZ53EXAJKXYQXSX2PX4QEA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.803639562 (610h25m27.674722597s) [noise=true , address=145.79.227.29:4971 , piece_id=MEPXHHOQ6SUZC3GIPCQEMOKKF5OJTKZEI3UYZBAYXAQZGSVEKSBA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.812791279 (610h25m27.683874314s) [noise=true , address=145.79.227.49:4343 , piece_id=6L3HPTS7D2NRA4QEOH4RFM3D3B53CTCONRI6G6YVU4OOTUWTG7XA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.812812348 (610h25m27.683895393s) [seg_pos={0 0}, upload=534952]: Segment reached optimal threshold of 14 pieces. Dec 31 01:54:27.813018000 (610h25m27.684101035s) [noise=true , address=145.79.227.35:4505 , piece_id=V7RS7A4QLZTH4G32ZUZNSF2Z5APVWZGJO77NEGP3SBWQT3WCMVLQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i6itGoBhLsLiqC8et8Y5L62fLjcWxy18u39NAr4R3nXNMWcsf): protocol: expected piece hash; context canceled; EOF Dec 31 01:54:27.813024772 (610h25m27.684107808s) [noise=true , address=145.79.227.35:4505 , piece_id=V7RS7A4QLZTH4G32ZUZNSF2Z5APVWZGJO77NEGP3SBWQT3WCMVLQ, seg_pos={0 0}, upload=534952]: Long tail cancellation due to optimal threshold Dec 31 01:54:27.813175931 (610h25m27.684258976s) [noise=true , address=145.79.227.34:4937 , piece_id=5W7LBM3YJJTVUFYTELR57DVXB663OHO5N3Z2SPQPZ4G43IK47O3Q, seg_pos={0 0}, upload=534952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1LKrFThdW3jJpG6dRZLUaxLXTa5Wy27yQ8PGUEqjvBWfFoRB94): protocol: expected piece hash; context canceled; EOF Dec 31 01:54:27.813181702 (610h25m27.684264737s) [noise=true , address=145.79.227.34:4937 , piece_id=5W7LBM3YJJTVUFYTELR57DVXB663OHO5N3Z2SPQPZ4G43IK47O3Q, seg_pos={0 0}, upload=534952]: Long tail cancellation due to optimal threshold Dec 31 01:54:27.813407862 (610h25m27.684490907s) [noise=true , address=145.79.227.2:4117 , piece_id=4SGFGB3IJ7LTXDMMWM2PGGOV64MC46IZ7OZQWH55J4NBQY7CHMZQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122Htg72f5eUUutf1p3fYQpy1Wkf6AgdaQV83HHYye2n8zm3the): protocol: expected piece hash; context canceled; EOF Dec 31 01:54:27.813414264 (610h25m27.684497299s) [noise=true , address=145.79.227.2:4117 , piece_id=4SGFGB3IJ7LTXDMMWM2PGGOV64MC46IZ7OZQWH55J4NBQY7CHMZQ, seg_pos={0 0}, upload=534952]: Long tail cancellation due to optimal threshold Dec 31 01:54:27.813424614 (610h25m27.684507649s) [noise=true , address=77.90.31.19:4848 , piece_id=VZYG32S6ODIPDKQYZRXECBHXOERZDDWD2OF35Z46W4XMUZ3OJLAA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD): protocol: write tcp 188.166.225.170:42842->77.90.31.19:4848: use of closed network connection; write tcp 188.166.225.170:42842->77.90.31.19:4848: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42842->77.90.31.19:4848: use of closed network connection Dec 31 01:54:27.813432750 (610h25m27.684515785s) [noise=true , address=77.90.31.19:4848 , piece_id=VZYG32S6ODIPDKQYZRXECBHXOERZDDWD2OF35Z46W4XMUZ3OJLAA, seg_pos={0 0}, upload=534952]: Long tail cancellation due to optimal threshold Dec 31 01:54:27.813474088 (610h25m27.684557123s) [seg_pos={0 0}, upload=534952]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 01:54:27.813535325 (610h25m27.68461836s) [upload=534952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 01:54:27.813539613 (610h25m27.684622648s) [upload=534952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 01:54:28.099921034 (610h25m27.971004069s) [upload=534952]: Done uploading segments... Dec 31 01:54:28.099940451 (610h25m27.971023486s) [upload=534952]: Upload finished. err: <nil> Duration: 22.102061638s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..............X...............................X....X........] duration: 3.411562319s trace: https://jaeger.dev.storj.io/trace/7bc7478624e56c1a FAIL (2025-12-31 02:43:48 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.411590272s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [X.......X.X..X....XX.XX........X.X......X..X......XXX..X....] duration: 5.798503568s trace: https://jaeger.dev.storj.io/trace/7796d308e68d2eba FAIL (2025-12-31 02:43:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, Piece ID: C5XVJ3QR25OBV3SQGCXATJHTMRQU76ZSSL3J4MCOYSMLMCZBJ3DQ) context deadline exceeded; piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: FTGU3JK3T27PZJ3MNQ735ICXBCFBSQRKVWSJCB4L2PY5QRMAJDIA) context deadline exceeded ======LOGS====== egments... Dec 31 02:41:10.854740410 (611h12m10.725823465s) [upload=535628]: Got next segment. Inline: false Dec 31 02:41:10.854780267 (611h12m10.725863312s) [upload=535628]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 02:41:10.854786138 (611h12m10.725869173s) [upload=535628]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 02:41:11.103813035 (611h12m10.974896091s) [seg_pos={0 0}, upload=535628]: Begin upload segment... Dec 31 02:41:11.103869233 (611h12m10.974952278s) [seg_pos={0 0}, upload=535628]: Done begin upload segment. Dec 31 02:41:11.103917354 (611h12m10.975000389s) [noise=true , address=145.79.227.39:4013 , piece_id=4EMEEV2RWPB5EE2Q4R4COOOLPC73M4ZE57BEU6G54ULD4PVHZAFQ, seg_pos={0 0}, upload=535628]: Uploading piece... Dec 31 02:41:11.103937894 (611h12m10.975020939s) [noise=true , address=77.90.31.10:4518 , piece_id=FTGU3JK3T27PZJ3MNQ735ICXBCFBSQRKVWSJCB4L2PY5QRMAJDIA, seg_pos={0 0}, upload=535628]: Uploading piece... Dec 31 02:41:11.104433838 (611h12m10.975516873s) [noise=true , address=145.79.227.6:4258 , piece_id=BG43YG3ZELUSXPNFUZGOQQYOLXZGLURSHDDGIFVSG7DBE3JHCGHA, seg_pos={0 0}, upload=535628]: Uploading piece... Dec 31 02:41:11.104787361 (611h12m10.975870396s) [noise=true , address=45.91.116.9:4263 , piece_id=QVIP6EJNGTD5FIMBDS2NCOK6R27OH5VAHK36T75HBI5F7KKH7YUQ, seg_pos={0 0}, upload=535628]: Uploading piece... Dec 31 02:41:11.105149000 (611h12m10.976232035s) [noise=true , address=145.79.227.4:4186 , piece_id=OFMISBUSLMHHEAELL2DDDTKND3UDZY2J7F7ASCTYTXYZOABZ4ZHQ, seg_pos={0 0}, upload=535628]: Uploading piece... Dec 31 02:41:11.105561805 (611h12m10.97664485s) [noise=true , address=145.79.227.32:4835 , piece_id=AVAICWG5CZ2FJEM3B72BGCZHTB234Q4TDRY635TDK2GCYGNNCMNQ, seg_pos={0 0}, upload=535628]: Uploading piece... Dec 31 02:41:11.106056818 (611h12m10.977139853s) [noise=true , address=145.79.227.51:4410 , piece_id=EOFZJEXLH6XYMUQCIWTU7KFZETRL3RI4FUN2OX7RQXKQLOD6JOOQ, seg_pos={0 0}, upload=535628]: Uploading piece... Dec 31 02:41:11.106636882 (611h12m10.977719948s) [noise=true , address=45.91.116.14:4190 , piece_id=UFR6AIVXP5ZPPQLVEZXKRSFB6CAZ7U7IL4CWCCKVRA5P3MC5MTLA, seg_pos={0 0}, upload=535628]: Uploading piece... Dec 31 02:41:11.107402160 (611h12m10.978485195s) [noise=true , address=77.90.31.15:4697 , piece_id=NMLIRYJXAH47RJP5ZSN5PCCPLKNZ2K2ANIMGBU42XRGP7NMHOOAQ, seg_pos={0 0}, upload=535628]: Uploading piece... Dec 31 02:41:11.107995509 (611h12m10.979078545s) [noise=true , address=145.79.227.50:4374 , piece_id=HRTWPEWMLYB24UEUFTHZHMRWAJJN6D3VKZ3YZ6B26LN54DQXVSAA, seg_pos={0 0}, upload=535628]: Uploading piece... Dec 31 02:41:11.108580824 (611h12m10.979663859s) [noise=true , address=145.79.227.25:4618 , piece_id=MZWHT24LGJXQVAE7X7K6BPVZMP7X6QETTMICMNCSYI4QBMTSUAMA, seg_pos={0 0}, upload=535628]: Uploading piece... Dec 31 02:41:11.109458766 (611h12m10.980541801s) [noise=true , address=45.91.116.11:4118 , piece_id=KLMGJP6XKFN23LRSEPCX52UBXRFJPHBQ3RWB66IIJXSKG2LIN7KA, seg_pos={0 0}, upload=535628]: Uploading piece... Dec 31 02:41:11.111091665 (611h12m10.9821747s) [noise=true , address=145.79.227.10:4403 , piece_id=D3XBXX6R5MHDLBWVSWZBQJJDYBK4VKXOMPAU5QVVG6FIVHKWKVNA, seg_pos={0 0}, upload=535628]: Uploading piece... Dec 31 02:41:11.112772295 (611h12m10.98385533s) [noise=true , address=77.90.31.11:4556 , piece_id=C5XVJ3QR25OBV3SQGCXATJHTMRQU76ZSSL3J4MCOYSMLMCZBJ3DQ, seg_pos={0 0}, upload=535628]: Uploading piece... Dec 31 02:41:11.115140324 (611h12m10.986223359s) [noise=true , address=145.79.227.13:4803 , piece_id=KE5H4FKU4PZROEEFQ5KABCDEOX2NUXLUD4IZBMCVSSVXSPKKSYRA, seg_pos={0 0}, upload=535628]: Uploading piece... Dec 31 02:41:11.117165320 (611h12m10.988248355s) [noise=true , address=77.90.31.14:4664 , piece_id=QYK7QKZ3KAIVPXVFPM45DJU4OB3KGDCPEOSZYIT44BD4P3EZCK3A, seg_pos={0 0}, upload=535628]: Uploading piece... Dec 31 02:41:12.815648949 (611h12m12.686731984s) [upload=535628]: Next returned nil segment Dec 31 02:41:12.815663367 (611h12m12.686746402s) [upload=535628]: Waiting for error group managing segments... Dec 31 02:41:13.093790374 (611h12m12.964873409s) [noise=true , address=145.79.227.13:4803 , piece_id=KE5H4FKU4PZROEEFQ5KABCDEOX2NUXLUD4IZBMCVSSVXSPKKSYRA, seg_pos={0 0}, upload=535628]: Done uploading piece. err: <nil> Dec 31 02:41:13.099111968 (611h12m12.970195003s) [noise=true , address=145.79.227.4:4186 , piece_id=OFMISBUSLMHHEAELL2DDDTKND3UDZY2J7F7ASCTYTXYZOABZ4ZHQ, seg_pos={0 0}, upload=535628]: Done uploading piece. err: <nil> Dec 31 02:41:13.100327292 (611h12m12.971410327s) [noise=true , address=145.79.227.25:4618 , piece_id=MZWHT24LGJXQVAE7X7K6BPVZMP7X6QETTMICMNCSYI4QBMTSUAMA, seg_pos={0 0}, upload=535628]: Done uploading piece. err: <nil> Dec 31 02:41:13.102226227 (611h12m12.973309262s) [noise=true , address=145.79.227.32:4835 , piece_id=AVAICWG5CZ2FJEM3B72BGCZHTB234Q4TDRY635TDK2GCYGNNCMNQ, seg_pos={0 0}, upload=535628]: Done uploading piece. err: <nil> Dec 31 02:41:13.126767006 (611h12m12.997850041s) [noise=true , address=145.79.227.51:4410 , piece_id=EOFZJEXLH6XYMUQCIWTU7KFZETRL3RI4FUN2OX7RQXKQLOD6JOOQ, seg_pos={0 0}, upload=535628]: Done uploading piece. err: <nil> Dec 31 02:41:13.142304611 (611h12m13.013387646s) [noise=true , address=145.79.227.50:4374 , piece_id=HRTWPEWMLYB24UEUFTHZHMRWAJJN6D3VKZ3YZ6B26LN54DQXVSAA, seg_pos={0 0}, upload=535628]: Done uploading piece. err: <nil> Dec 31 02:41:13.181506727 (611h12m13.052589762s) [noise=true , address=145.79.227.39:4013 , piece_id=4EMEEV2RWPB5EE2Q4R4COOOLPC73M4ZE57BEU6G54ULD4PVHZAFQ, seg_pos={0 0}, upload=535628]: Done uploading piece. err: <nil> Dec 31 02:41:13.182708215 (611h12m13.05379125s) [noise=true , address=145.79.227.6:4258 , piece_id=BG43YG3ZELUSXPNFUZGOQQYOLXZGLURSHDDGIFVSG7DBE3JHCGHA, seg_pos={0 0}, upload=535628]: Done uploading piece. err: <nil> Dec 31 02:41:13.185142701 (611h12m13.056225736s) [noise=true , address=145.79.227.10:4403 , piece_id=D3XBXX6R5MHDLBWVSWZBQJJDYBK4VKXOMPAU5QVVG6FIVHKWKVNA, seg_pos={0 0}, upload=535628]: Done uploading piece. err: <nil> Dec 31 02:41:13.297136831 (611h12m13.168219866s) [noise=true , address=77.90.31.10:4518 , piece_id=FTGU3JK3T27PZJ3MNQ735ICXBCFBSQRKVWSJCB4L2PY5QRMAJDIA, seg_pos={0 0}, upload=535628]: Done uploading piece. err: <nil> Dec 31 02:41:13.337206979 (611h12m13.208290014s) [noise=true , address=77.90.31.11:4556 , piece_id=C5XVJ3QR25OBV3SQGCXATJHTMRQU76ZSSL3J4MCOYSMLMCZBJ3DQ, seg_pos={0 0}, upload=535628]: Done uploading piece. err: <nil> Dec 31 02:41:13.344521868 (611h12m13.215604903s) [noise=true , address=77.90.31.15:4697 , piece_id=NMLIRYJXAH47RJP5ZSN5PCCPLKNZ2K2ANIMGBU42XRGP7NMHOOAQ, seg_pos={0 0}, upload=535628]: Done uploading piece. err: <nil> Dec 31 02:41:13.512080229 (611h12m13.383163264s) [noise=true , address=45.91.116.11:4118 , piece_id=KLMGJP6XKFN23LRSEPCX52UBXRFJPHBQ3RWB66IIJXSKG2LIN7KA, seg_pos={0 0}, upload=535628]: Done uploading piece. err: <nil> Dec 31 02:41:13.512104606 (611h12m13.383187641s) [seg_pos={0 0}, upload=535628]: Segment reached optimal threshold of 13 pieces. Dec 31 02:41:13.512325146 (611h12m13.383408181s) [noise=true , address=77.90.31.14:4664 , piece_id=QYK7QKZ3KAIVPXVFPM45DJU4OB3KGDCPEOSZYIT44BD4P3EZCK3A, seg_pos={0 0}, upload=535628]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: write tcp 188.166.225.170:50786->77.90.31.14:4664: use of closed network connection; write tcp 188.166.225.170:50786->77.90.31.14:4664: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50786->77.90.31.14:4664: use of closed network connection Dec 31 02:41:13.512331317 (611h12m13.383414362s) [noise=true , address=77.90.31.14:4664 , piece_id=QYK7QKZ3KAIVPXVFPM45DJU4OB3KGDCPEOSZYIT44BD4P3EZCK3A, seg_pos={0 0}, upload=535628]: Long tail cancellation due to optimal threshold Dec 31 02:41:13.512609918 (611h12m13.383692953s) [noise=true , address=45.91.116.14:4190 , piece_id=UFR6AIVXP5ZPPQLVEZXKRSFB6CAZ7U7IL4CWCCKVRA5P3MC5MTLA, seg_pos={0 0}, upload=535628]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: write tcp 188.166.225.170:42822->45.91.116.14:4190: use of closed network connection; write tcp 188.166.225.170:42822->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42822->45.91.116.14:4190: use of closed network connection Dec 31 02:41:13.512626390 (611h12m13.383709435s) [noise=true , address=45.91.116.14:4190 , piece_id=UFR6AIVXP5ZPPQLVEZXKRSFB6CAZ7U7IL4CWCCKVRA5P3MC5MTLA, seg_pos={0 0}, upload=535628]: Long tail cancellation due to optimal threshold Dec 31 02:41:13.512787786 (611h12m13.383870821s) [noise=true , address=45.91.116.9:4263 , piece_id=QVIP6EJNGTD5FIMBDS2NCOK6R27OH5VAHK36T75HBI5F7KKH7YUQ, seg_pos={0 0}, upload=535628]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 31 02:41:13.512793036 (611h12m13.383876071s) [noise=true , address=45.91.116.9:4263 , piece_id=QVIP6EJNGTD5FIMBDS2NCOK6R27OH5VAHK36T75HBI5F7KKH7YUQ, seg_pos={0 0}, upload=535628]: Long tail cancellation due to optimal threshold Dec 31 02:41:13.512814998 (611h12m13.383898034s) [seg_pos={0 0}, upload=535628]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 02:41:13.512893738 (611h12m13.383976773s) [upload=535628]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 02:41:13.512898317 (611h12m13.383981352s) [upload=535628]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 02:41:13.802131863 (611h12m13.673214918s) [upload=535628]: Done uploading segments... Dec 31 02:41:13.802155628 (611h12m13.673238663s) [upload=535628]: Upload finished. err: <nil> Duration: 5.79853129s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...............X...............................X....X.......] duration: 3.682769863s trace: https://jaeger.dev.storj.io/trace/7bc7478624e56c1a FAIL (2025-12-31 02:43:25 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.682783479s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 47.749927024s trace: https://jaeger.dev.storj.io/trace/365693fea00df28 FAIL (2025-12-31 02:43:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [noise=true , address=145.79.227.49:4343 , piece_id=6L3HPTS7D2NRA4QEOH4RFM3D3B53CTCONRI6G6YVU4OOTUWTG7XA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.942060847 (610h25m24.813143892s) [noise=true , address=145.79.227.34:4937 , piece_id=5W7LBM3YJJTVUFYTELR57DVXB663OHO5N3Z2SPQPZ4G43IK47O3Q, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.942684104 (610h25m24.813767139s) [noise=true , address=64.49.13.139:4478 , piece_id=QN6RT2KWEA6APQVA66ZMKG4LH537UGYNOXOCVVA2YC2MKDOLT3YA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.943064007 (610h25m24.814147042s) [noise=true , address=145.79.227.40:4084 , piece_id=RUKTKDII6BRUWSYVS65CWUBT3QKG3ASUK3KA3O5VGE53XSG6IMOQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.943459560 (610h25m24.814542605s) [noise=true , address=145.79.227.14:4847 , piece_id=JX5NXTMVM3NSWIEVR4RX46ABSNPKTOFDLFBPKEXFQ3G3ANKV5OLA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.943865303 (610h25m24.814948348s) [noise=true , address=45.91.116.6:4981 , piece_id=CT474ZA2LVAZEBJHBDD4GB6NTHVHZFJLNE5OZV3DFS2NZKVW3EQQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.944259694 (610h25m24.815342749s) [noise=true , address=145.79.227.51:4443 , piece_id=73UUMRDY7BSZNCGBXCZYYRBVE5KLPMNPRNXOZEGWKQGLHJS32J2Q, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.946575182 (610h25m24.817658227s) [noise=true , address=77.90.31.10:4523 , piece_id=P6LIONXJI6AFZ2JTKCQ2W6RZCOZCK6MZ53EXAJKXYQXSX2PX4QEA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.948132909 (610h25m24.819215944s) [noise=true , address=145.79.227.20:4452 , piece_id=ZSDN2JZ3FQUVJYN5OURDXNKEDLDNULJOHKKRZVIPXZIWIYAHCRXQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.949605774 (610h25m24.820688819s) [noise=true , address=77.90.31.14:4680 , piece_id=S2E4WBA7MJVAZLT5JG3IMHK53RQC77CCHV2PDF6L2R2JEA2U3K4Q, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.951459002 (610h25m24.822542048s) [noise=true , address=193.149.164.143:4704 , piece_id=BHXHFBKTLTDMGXWY5QRTP2THZQE5OXFZ2LCMTO2KRYRWAHXWW3NQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.952986742 (610h25m24.824069777s) [noise=true , address=77.90.31.19:4848 , piece_id=VZYG32S6ODIPDKQYZRXECBHXOERZDDWD2OF35Z46W4XMUZ3OJLAA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.956916654 (610h25m24.827999689s) [noise=true , address=145.79.227.29:4971 , piece_id=MEPXHHOQ6SUZC3GIPCQEMOKKF5OJTKZEI3UYZBAYXAQZGSVEKSBA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.960665062 (610h25m24.831748097s) [noise=true , address=64.49.13.170:4720 , piece_id=HIATRSAPE4YLSODFZDPP43ZUMLDVM7GJEVMWE4ZGUQTP25LTHGTA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.962366001 (610h25m24.833449036s) [noise=true , address=145.79.227.2:4117 , piece_id=4SGFGB3IJ7LTXDMMWM2PGGOV64MC46IZ7OZQWH55J4NBQY7CHMZQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.963708216 (610h25m24.834791261s) [noise=true , address=145.79.227.35:4505 , piece_id=V7RS7A4QLZTH4G32ZUZNSF2Z5APVWZGJO77NEGP3SBWQT3WCMVLQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:26.352236138 (610h25m26.223319173s) [upload=534952]: Next returned nil segment Dec 31 01:54:26.352253821 (610h25m26.223336857s) [upload=534952]: Waiting for error group managing segments... Dec 31 01:54:26.812758374 (610h25m26.683841409s) [noise=true , address=145.79.227.51:4443 , piece_id=73UUMRDY7BSZNCGBXCZYYRBVE5KLPMNPRNXOZEGWKQGLHJS32J2Q, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.824371056 (610h25m26.695454101s) [noise=true , address=145.79.227.14:4847 , piece_id=JX5NXTMVM3NSWIEVR4RX46ABSNPKTOFDLFBPKEXFQ3G3ANKV5OLA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.838414046 (610h25m26.709497081s) [noise=true , address=64.49.13.138:4490 , piece_id=2XTUZE6PXRE3OARNJVT36FXXWUMYIBVNE6M2VVFAVBGKEE2H5YZQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.875499841 (610h25m26.746582876s) [noise=true , address=64.49.13.139:4478 , piece_id=QN6RT2KWEA6APQVA66ZMKG4LH537UGYNOXOCVVA2YC2MKDOLT3YA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.891607221 (610h25m26.762690256s) [noise=true , address=145.79.227.20:4452 , piece_id=ZSDN2JZ3FQUVJYN5OURDXNKEDLDNULJOHKKRZVIPXZIWIYAHCRXQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.988561223 (610h25m26.859644268s) [noise=true , address=64.49.13.170:4720 , piece_id=HIATRSAPE4YLSODFZDPP43ZUMLDVM7GJEVMWE4ZGUQTP25LTHGTA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.039035911 (610h25m26.910118946s) [noise=true , address=77.90.31.14:4680 , piece_id=S2E4WBA7MJVAZLT5JG3IMHK53RQC77CCHV2PDF6L2R2JEA2U3K4Q, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.098023090 (610h25m26.969106126s) [noise=true , address=45.91.116.6:4981 , piece_id=CT474ZA2LVAZEBJHBDD4GB6NTHVHZFJLNE5OZV3DFS2NZKVW3EQQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.653790754 (610h25m27.524873789s) [noise=true , address=77.90.31.21:4886 , piece_id=K34WPIBGGGEPHIKJ253XGLDCIQ3WLFN4AZ6EYNWVHHT2EIHS6YRQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.684192972 (610h25m27.555276017s) [noise=true , address=193.149.164.143:4704 , piece_id=BHXHFBKTLTDMGXWY5QRTP2THZQE5OXFZ2LCMTO2KRYRWAHXWW3NQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.717238425 (610h25m27.58832146s) [noise=true , address=145.79.227.40:4084 , piece_id=RUKTKDII6BRUWSYVS65CWUBT3QKG3ASUK3KA3O5VGE53XSG6IMOQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.718349742 (610h25m27.589432777s) [noise=true , address=77.90.31.10:4523 , piece_id=P6LIONXJI6AFZ2JTKCQ2W6RZCOZCK6MZ53EXAJKXYQXSX2PX4QEA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.803639562 (610h25m27.674722597s) [noise=true , address=145.79.227.29:4971 , piece_id=MEPXHHOQ6SUZC3GIPCQEMOKKF5OJTKZEI3UYZBAYXAQZGSVEKSBA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.812791279 (610h25m27.683874314s) [noise=true , address=145.79.227.49:4343 , piece_id=6L3HPTS7D2NRA4QEOH4RFM3D3B53CTCONRI6G6YVU4OOTUWTG7XA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.812812348 (610h25m27.683895393s) [seg_pos={0 0}, upload=534952]: Segment reached optimal threshold of 14 pieces. Dec 31 01:54:27.813018000 (610h25m27.684101035s) [noise=true , address=145.79.227.35:4505 , piece_id=V7RS7A4QLZTH4G32ZUZNSF2Z5APVWZGJO77NEGP3SBWQT3WCMVLQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i6itGoBhLsLiqC8et8Y5L62fLjcWxy18u39NAr4R3nXNMWcsf): protocol: expected piece hash; context canceled; EOF Dec 31 01:54:27.813024772 (610h25m27.684107808s) [noise=true , address=145.79.227.35:4505 , piece_id=V7RS7A4QLZTH4G32ZUZNSF2Z5APVWZGJO77NEGP3SBWQT3WCMVLQ, seg_pos={0 0}, upload=534952]: Long tail cancellation due to optimal threshold Dec 31 01:54:27.813175931 (610h25m27.684258976s) [noise=true , address=145.79.227.34:4937 , piece_id=5W7LBM3YJJTVUFYTELR57DVXB663OHO5N3Z2SPQPZ4G43IK47O3Q, seg_pos={0 0}, upload=534952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1LKrFThdW3jJpG6dRZLUaxLXTa5Wy27yQ8PGUEqjvBWfFoRB94): protocol: expected piece hash; context canceled; EOF Dec 31 01:54:27.813181702 (610h25m27.684264737s) [noise=true , address=145.79.227.34:4937 , piece_id=5W7LBM3YJJTVUFYTELR57DVXB663OHO5N3Z2SPQPZ4G43IK47O3Q, seg_pos={0 0}, upload=534952]: Long tail cancellation due to optimal threshold Dec 31 01:54:27.813407862 (610h25m27.684490907s) [noise=true , address=145.79.227.2:4117 , piece_id=4SGFGB3IJ7LTXDMMWM2PGGOV64MC46IZ7OZQWH55J4NBQY7CHMZQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122Htg72f5eUUutf1p3fYQpy1Wkf6AgdaQV83HHYye2n8zm3the): protocol: expected piece hash; context canceled; EOF Dec 31 01:54:27.813414264 (610h25m27.684497299s) [noise=true , address=145.79.227.2:4117 , piece_id=4SGFGB3IJ7LTXDMMWM2PGGOV64MC46IZ7OZQWH55J4NBQY7CHMZQ, seg_pos={0 0}, upload=534952]: Long tail cancellation due to optimal threshold Dec 31 01:54:27.813424614 (610h25m27.684507649s) [noise=true , address=77.90.31.19:4848 , piece_id=VZYG32S6ODIPDKQYZRXECBHXOERZDDWD2OF35Z46W4XMUZ3OJLAA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD): protocol: write tcp 188.166.225.170:42842->77.90.31.19:4848: use of closed network connection; write tcp 188.166.225.170:42842->77.90.31.19:4848: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42842->77.90.31.19:4848: use of closed network connection Dec 31 01:54:27.813432750 (610h25m27.684515785s) [noise=true , address=77.90.31.19:4848 , piece_id=VZYG32S6ODIPDKQYZRXECBHXOERZDDWD2OF35Z46W4XMUZ3OJLAA, seg_pos={0 0}, upload=534952]: Long tail cancellation due to optimal threshold Dec 31 01:54:27.813474088 (610h25m27.684557123s) [seg_pos={0 0}, upload=534952]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 01:54:27.813535325 (610h25m27.68461836s) [upload=534952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 01:54:27.813539613 (610h25m27.684622648s) [upload=534952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 01:54:28.099921034 (610h25m27.971004069s) [upload=534952]: Done uploading segments... Dec 31 01:54:28.099940451 (610h25m27.971023486s) [upload=534952]: Upload finished. err: <nil> Duration: 47.749953253s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [.X.......X.X..X....XX.XX........X.X......X..X......XXX..X...] duration: 6.052624974s trace: https://jaeger.dev.storj.io/trace/7796d308e68d2eba FAIL (2025-12-31 02:43:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, Piece ID: C5XVJ3QR25OBV3SQGCXATJHTMRQU76ZSSL3J4MCOYSMLMCZBJ3DQ) context deadline exceeded; piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: FTGU3JK3T27PZJ3MNQ735ICXBCFBSQRKVWSJCB4L2PY5QRMAJDIA) context deadline exceeded ======LOGS====== egments... Dec 31 02:41:10.854740410 (611h12m10.725823465s) [upload=535628]: Got next segment. Inline: false Dec 31 02:41:10.854780267 (611h12m10.725863312s) [upload=535628]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 02:41:10.854786138 (611h12m10.725869173s) [upload=535628]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 02:41:11.103813035 (611h12m10.974896091s) [seg_pos={0 0}, upload=535628]: Begin upload segment... Dec 31 02:41:11.103869233 (611h12m10.974952278s) [seg_pos={0 0}, upload=535628]: Done begin upload segment. Dec 31 02:41:11.103917354 (611h12m10.975000389s) [noise=true , address=145.79.227.39:4013 , piece_id=4EMEEV2RWPB5EE2Q4R4COOOLPC73M4ZE57BEU6G54ULD4PVHZAFQ, seg_pos={0 0}, upload=535628]: Uploading piece... Dec 31 02:41:11.103937894 (611h12m10.975020939s) [noise=true , address=77.90.31.10:4518 , piece_id=FTGU3JK3T27PZJ3MNQ735ICXBCFBSQRKVWSJCB4L2PY5QRMAJDIA, seg_pos={0 0}, upload=535628]: Uploading piece... Dec 31 02:41:11.104433838 (611h12m10.975516873s) [noise=true , address=145.79.227.6:4258 , piece_id=BG43YG3ZELUSXPNFUZGOQQYOLXZGLURSHDDGIFVSG7DBE3JHCGHA, seg_pos={0 0}, upload=535628]: Uploading piece... Dec 31 02:41:11.104787361 (611h12m10.975870396s) [noise=true , address=45.91.116.9:4263 , piece_id=QVIP6EJNGTD5FIMBDS2NCOK6R27OH5VAHK36T75HBI5F7KKH7YUQ, seg_pos={0 0}, upload=535628]: Uploading piece... Dec 31 02:41:11.105149000 (611h12m10.976232035s) [noise=true , address=145.79.227.4:4186 , piece_id=OFMISBUSLMHHEAELL2DDDTKND3UDZY2J7F7ASCTYTXYZOABZ4ZHQ, seg_pos={0 0}, upload=535628]: Uploading piece... Dec 31 02:41:11.105561805 (611h12m10.97664485s) [noise=true , address=145.79.227.32:4835 , piece_id=AVAICWG5CZ2FJEM3B72BGCZHTB234Q4TDRY635TDK2GCYGNNCMNQ, seg_pos={0 0}, upload=535628]: Uploading piece... Dec 31 02:41:11.106056818 (611h12m10.977139853s) [noise=true , address=145.79.227.51:4410 , piece_id=EOFZJEXLH6XYMUQCIWTU7KFZETRL3RI4FUN2OX7RQXKQLOD6JOOQ, seg_pos={0 0}, upload=535628]: Uploading piece... Dec 31 02:41:11.106636882 (611h12m10.977719948s) [noise=true , address=45.91.116.14:4190 , piece_id=UFR6AIVXP5ZPPQLVEZXKRSFB6CAZ7U7IL4CWCCKVRA5P3MC5MTLA, seg_pos={0 0}, upload=535628]: Uploading piece... Dec 31 02:41:11.107402160 (611h12m10.978485195s) [noise=true , address=77.90.31.15:4697 , piece_id=NMLIRYJXAH47RJP5ZSN5PCCPLKNZ2K2ANIMGBU42XRGP7NMHOOAQ, seg_pos={0 0}, upload=535628]: Uploading piece... Dec 31 02:41:11.107995509 (611h12m10.979078545s) [noise=true , address=145.79.227.50:4374 , piece_id=HRTWPEWMLYB24UEUFTHZHMRWAJJN6D3VKZ3YZ6B26LN54DQXVSAA, seg_pos={0 0}, upload=535628]: Uploading piece... Dec 31 02:41:11.108580824 (611h12m10.979663859s) [noise=true , address=145.79.227.25:4618 , piece_id=MZWHT24LGJXQVAE7X7K6BPVZMP7X6QETTMICMNCSYI4QBMTSUAMA, seg_pos={0 0}, upload=535628]: Uploading piece... Dec 31 02:41:11.109458766 (611h12m10.980541801s) [noise=true , address=45.91.116.11:4118 , piece_id=KLMGJP6XKFN23LRSEPCX52UBXRFJPHBQ3RWB66IIJXSKG2LIN7KA, seg_pos={0 0}, upload=535628]: Uploading piece... Dec 31 02:41:11.111091665 (611h12m10.9821747s) [noise=true , address=145.79.227.10:4403 , piece_id=D3XBXX6R5MHDLBWVSWZBQJJDYBK4VKXOMPAU5QVVG6FIVHKWKVNA, seg_pos={0 0}, upload=535628]: Uploading piece... Dec 31 02:41:11.112772295 (611h12m10.98385533s) [noise=true , address=77.90.31.11:4556 , piece_id=C5XVJ3QR25OBV3SQGCXATJHTMRQU76ZSSL3J4MCOYSMLMCZBJ3DQ, seg_pos={0 0}, upload=535628]: Uploading piece... Dec 31 02:41:11.115140324 (611h12m10.986223359s) [noise=true , address=145.79.227.13:4803 , piece_id=KE5H4FKU4PZROEEFQ5KABCDEOX2NUXLUD4IZBMCVSSVXSPKKSYRA, seg_pos={0 0}, upload=535628]: Uploading piece... Dec 31 02:41:11.117165320 (611h12m10.988248355s) [noise=true , address=77.90.31.14:4664 , piece_id=QYK7QKZ3KAIVPXVFPM45DJU4OB3KGDCPEOSZYIT44BD4P3EZCK3A, seg_pos={0 0}, upload=535628]: Uploading piece... Dec 31 02:41:12.815648949 (611h12m12.686731984s) [upload=535628]: Next returned nil segment Dec 31 02:41:12.815663367 (611h12m12.686746402s) [upload=535628]: Waiting for error group managing segments... Dec 31 02:41:13.093790374 (611h12m12.964873409s) [noise=true , address=145.79.227.13:4803 , piece_id=KE5H4FKU4PZROEEFQ5KABCDEOX2NUXLUD4IZBMCVSSVXSPKKSYRA, seg_pos={0 0}, upload=535628]: Done uploading piece. err: <nil> Dec 31 02:41:13.099111968 (611h12m12.970195003s) [noise=true , address=145.79.227.4:4186 , piece_id=OFMISBUSLMHHEAELL2DDDTKND3UDZY2J7F7ASCTYTXYZOABZ4ZHQ, seg_pos={0 0}, upload=535628]: Done uploading piece. err: <nil> Dec 31 02:41:13.100327292 (611h12m12.971410327s) [noise=true , address=145.79.227.25:4618 , piece_id=MZWHT24LGJXQVAE7X7K6BPVZMP7X6QETTMICMNCSYI4QBMTSUAMA, seg_pos={0 0}, upload=535628]: Done uploading piece. err: <nil> Dec 31 02:41:13.102226227 (611h12m12.973309262s) [noise=true , address=145.79.227.32:4835 , piece_id=AVAICWG5CZ2FJEM3B72BGCZHTB234Q4TDRY635TDK2GCYGNNCMNQ, seg_pos={0 0}, upload=535628]: Done uploading piece. err: <nil> Dec 31 02:41:13.126767006 (611h12m12.997850041s) [noise=true , address=145.79.227.51:4410 , piece_id=EOFZJEXLH6XYMUQCIWTU7KFZETRL3RI4FUN2OX7RQXKQLOD6JOOQ, seg_pos={0 0}, upload=535628]: Done uploading piece. err: <nil> Dec 31 02:41:13.142304611 (611h12m13.013387646s) [noise=true , address=145.79.227.50:4374 , piece_id=HRTWPEWMLYB24UEUFTHZHMRWAJJN6D3VKZ3YZ6B26LN54DQXVSAA, seg_pos={0 0}, upload=535628]: Done uploading piece. err: <nil> Dec 31 02:41:13.181506727 (611h12m13.052589762s) [noise=true , address=145.79.227.39:4013 , piece_id=4EMEEV2RWPB5EE2Q4R4COOOLPC73M4ZE57BEU6G54ULD4PVHZAFQ, seg_pos={0 0}, upload=535628]: Done uploading piece. err: <nil> Dec 31 02:41:13.182708215 (611h12m13.05379125s) [noise=true , address=145.79.227.6:4258 , piece_id=BG43YG3ZELUSXPNFUZGOQQYOLXZGLURSHDDGIFVSG7DBE3JHCGHA, seg_pos={0 0}, upload=535628]: Done uploading piece. err: <nil> Dec 31 02:41:13.185142701 (611h12m13.056225736s) [noise=true , address=145.79.227.10:4403 , piece_id=D3XBXX6R5MHDLBWVSWZBQJJDYBK4VKXOMPAU5QVVG6FIVHKWKVNA, seg_pos={0 0}, upload=535628]: Done uploading piece. err: <nil> Dec 31 02:41:13.297136831 (611h12m13.168219866s) [noise=true , address=77.90.31.10:4518 , piece_id=FTGU3JK3T27PZJ3MNQ735ICXBCFBSQRKVWSJCB4L2PY5QRMAJDIA, seg_pos={0 0}, upload=535628]: Done uploading piece. err: <nil> Dec 31 02:41:13.337206979 (611h12m13.208290014s) [noise=true , address=77.90.31.11:4556 , piece_id=C5XVJ3QR25OBV3SQGCXATJHTMRQU76ZSSL3J4MCOYSMLMCZBJ3DQ, seg_pos={0 0}, upload=535628]: Done uploading piece. err: <nil> Dec 31 02:41:13.344521868 (611h12m13.215604903s) [noise=true , address=77.90.31.15:4697 , piece_id=NMLIRYJXAH47RJP5ZSN5PCCPLKNZ2K2ANIMGBU42XRGP7NMHOOAQ, seg_pos={0 0}, upload=535628]: Done uploading piece. err: <nil> Dec 31 02:41:13.512080229 (611h12m13.383163264s) [noise=true , address=45.91.116.11:4118 , piece_id=KLMGJP6XKFN23LRSEPCX52UBXRFJPHBQ3RWB66IIJXSKG2LIN7KA, seg_pos={0 0}, upload=535628]: Done uploading piece. err: <nil> Dec 31 02:41:13.512104606 (611h12m13.383187641s) [seg_pos={0 0}, upload=535628]: Segment reached optimal threshold of 13 pieces. Dec 31 02:41:13.512325146 (611h12m13.383408181s) [noise=true , address=77.90.31.14:4664 , piece_id=QYK7QKZ3KAIVPXVFPM45DJU4OB3KGDCPEOSZYIT44BD4P3EZCK3A, seg_pos={0 0}, upload=535628]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: write tcp 188.166.225.170:50786->77.90.31.14:4664: use of closed network connection; write tcp 188.166.225.170:50786->77.90.31.14:4664: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50786->77.90.31.14:4664: use of closed network connection Dec 31 02:41:13.512331317 (611h12m13.383414362s) [noise=true , address=77.90.31.14:4664 , piece_id=QYK7QKZ3KAIVPXVFPM45DJU4OB3KGDCPEOSZYIT44BD4P3EZCK3A, seg_pos={0 0}, upload=535628]: Long tail cancellation due to optimal threshold Dec 31 02:41:13.512609918 (611h12m13.383692953s) [noise=true , address=45.91.116.14:4190 , piece_id=UFR6AIVXP5ZPPQLVEZXKRSFB6CAZ7U7IL4CWCCKVRA5P3MC5MTLA, seg_pos={0 0}, upload=535628]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: write tcp 188.166.225.170:42822->45.91.116.14:4190: use of closed network connection; write tcp 188.166.225.170:42822->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42822->45.91.116.14:4190: use of closed network connection Dec 31 02:41:13.512626390 (611h12m13.383709435s) [noise=true , address=45.91.116.14:4190 , piece_id=UFR6AIVXP5ZPPQLVEZXKRSFB6CAZ7U7IL4CWCCKVRA5P3MC5MTLA, seg_pos={0 0}, upload=535628]: Long tail cancellation due to optimal threshold Dec 31 02:41:13.512787786 (611h12m13.383870821s) [noise=true , address=45.91.116.9:4263 , piece_id=QVIP6EJNGTD5FIMBDS2NCOK6R27OH5VAHK36T75HBI5F7KKH7YUQ, seg_pos={0 0}, upload=535628]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 31 02:41:13.512793036 (611h12m13.383876071s) [noise=true , address=45.91.116.9:4263 , piece_id=QVIP6EJNGTD5FIMBDS2NCOK6R27OH5VAHK36T75HBI5F7KKH7YUQ, seg_pos={0 0}, upload=535628]: Long tail cancellation due to optimal threshold Dec 31 02:41:13.512814998 (611h12m13.383898034s) [seg_pos={0 0}, upload=535628]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 02:41:13.512893738 (611h12m13.383976773s) [upload=535628]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 02:41:13.512898317 (611h12m13.383981352s) [upload=535628]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 02:41:13.802131863 (611h12m13.673214918s) [upload=535628]: Done uploading segments... Dec 31 02:41:13.802155628 (611h12m13.673238663s) [upload=535628]: Upload finished. err: <nil> Duration: 6.052654211s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [................X...............................X....X......] duration: 3.69056668s trace: https://jaeger.dev.storj.io/trace/7bc7478624e56c1a FAIL (2025-12-31 02:42:29 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.690590806s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [..X.......X.X..X....XX.XX........X.X......X..X......XXX..X..] duration: 6.168879856s trace: https://jaeger.dev.storj.io/trace/7796d308e68d2eba FAIL (2025-12-31 02:42:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, Piece ID: C5XVJ3QR25OBV3SQGCXATJHTMRQU76ZSSL3J4MCOYSMLMCZBJ3DQ) context deadline exceeded; piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: FTGU3JK3T27PZJ3MNQ735ICXBCFBSQRKVWSJCB4L2PY5QRMAJDIA) context deadline exceeded ======LOGS====== egments... Dec 31 02:41:10.854740410 (611h12m10.725823465s) [upload=535628]: Got next segment. Inline: false Dec 31 02:41:10.854780267 (611h12m10.725863312s) [upload=535628]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 02:41:10.854786138 (611h12m10.725869173s) [upload=535628]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 02:41:11.103813035 (611h12m10.974896091s) [seg_pos={0 0}, upload=535628]: Begin upload segment... Dec 31 02:41:11.103869233 (611h12m10.974952278s) [seg_pos={0 0}, upload=535628]: Done begin upload segment. Dec 31 02:41:11.103917354 (611h12m10.975000389s) [noise=true , address=145.79.227.39:4013 , piece_id=4EMEEV2RWPB5EE2Q4R4COOOLPC73M4ZE57BEU6G54ULD4PVHZAFQ, seg_pos={0 0}, upload=535628]: Uploading piece... Dec 31 02:41:11.103937894 (611h12m10.975020939s) [noise=true , address=77.90.31.10:4518 , piece_id=FTGU3JK3T27PZJ3MNQ735ICXBCFBSQRKVWSJCB4L2PY5QRMAJDIA, seg_pos={0 0}, upload=535628]: Uploading piece... Dec 31 02:41:11.104433838 (611h12m10.975516873s) [noise=true , address=145.79.227.6:4258 , piece_id=BG43YG3ZELUSXPNFUZGOQQYOLXZGLURSHDDGIFVSG7DBE3JHCGHA, seg_pos={0 0}, upload=535628]: Uploading piece... Dec 31 02:41:11.104787361 (611h12m10.975870396s) [noise=true , address=45.91.116.9:4263 , piece_id=QVIP6EJNGTD5FIMBDS2NCOK6R27OH5VAHK36T75HBI5F7KKH7YUQ, seg_pos={0 0}, upload=535628]: Uploading piece... Dec 31 02:41:11.105149000 (611h12m10.976232035s) [noise=true , address=145.79.227.4:4186 , piece_id=OFMISBUSLMHHEAELL2DDDTKND3UDZY2J7F7ASCTYTXYZOABZ4ZHQ, seg_pos={0 0}, upload=535628]: Uploading piece... Dec 31 02:41:11.105561805 (611h12m10.97664485s) [noise=true , address=145.79.227.32:4835 , piece_id=AVAICWG5CZ2FJEM3B72BGCZHTB234Q4TDRY635TDK2GCYGNNCMNQ, seg_pos={0 0}, upload=535628]: Uploading piece... Dec 31 02:41:11.106056818 (611h12m10.977139853s) [noise=true , address=145.79.227.51:4410 , piece_id=EOFZJEXLH6XYMUQCIWTU7KFZETRL3RI4FUN2OX7RQXKQLOD6JOOQ, seg_pos={0 0}, upload=535628]: Uploading piece... Dec 31 02:41:11.106636882 (611h12m10.977719948s) [noise=true , address=45.91.116.14:4190 , piece_id=UFR6AIVXP5ZPPQLVEZXKRSFB6CAZ7U7IL4CWCCKVRA5P3MC5MTLA, seg_pos={0 0}, upload=535628]: Uploading piece... Dec 31 02:41:11.107402160 (611h12m10.978485195s) [noise=true , address=77.90.31.15:4697 , piece_id=NMLIRYJXAH47RJP5ZSN5PCCPLKNZ2K2ANIMGBU42XRGP7NMHOOAQ, seg_pos={0 0}, upload=535628]: Uploading piece... Dec 31 02:41:11.107995509 (611h12m10.979078545s) [noise=true , address=145.79.227.50:4374 , piece_id=HRTWPEWMLYB24UEUFTHZHMRWAJJN6D3VKZ3YZ6B26LN54DQXVSAA, seg_pos={0 0}, upload=535628]: Uploading piece... Dec 31 02:41:11.108580824 (611h12m10.979663859s) [noise=true , address=145.79.227.25:4618 , piece_id=MZWHT24LGJXQVAE7X7K6BPVZMP7X6QETTMICMNCSYI4QBMTSUAMA, seg_pos={0 0}, upload=535628]: Uploading piece... Dec 31 02:41:11.109458766 (611h12m10.980541801s) [noise=true , address=45.91.116.11:4118 , piece_id=KLMGJP6XKFN23LRSEPCX52UBXRFJPHBQ3RWB66IIJXSKG2LIN7KA, seg_pos={0 0}, upload=535628]: Uploading piece... Dec 31 02:41:11.111091665 (611h12m10.9821747s) [noise=true , address=145.79.227.10:4403 , piece_id=D3XBXX6R5MHDLBWVSWZBQJJDYBK4VKXOMPAU5QVVG6FIVHKWKVNA, seg_pos={0 0}, upload=535628]: Uploading piece... Dec 31 02:41:11.112772295 (611h12m10.98385533s) [noise=true , address=77.90.31.11:4556 , piece_id=C5XVJ3QR25OBV3SQGCXATJHTMRQU76ZSSL3J4MCOYSMLMCZBJ3DQ, seg_pos={0 0}, upload=535628]: Uploading piece... Dec 31 02:41:11.115140324 (611h12m10.986223359s) [noise=true , address=145.79.227.13:4803 , piece_id=KE5H4FKU4PZROEEFQ5KABCDEOX2NUXLUD4IZBMCVSSVXSPKKSYRA, seg_pos={0 0}, upload=535628]: Uploading piece... Dec 31 02:41:11.117165320 (611h12m10.988248355s) [noise=true , address=77.90.31.14:4664 , piece_id=QYK7QKZ3KAIVPXVFPM45DJU4OB3KGDCPEOSZYIT44BD4P3EZCK3A, seg_pos={0 0}, upload=535628]: Uploading piece... Dec 31 02:41:12.815648949 (611h12m12.686731984s) [upload=535628]: Next returned nil segment Dec 31 02:41:12.815663367 (611h12m12.686746402s) [upload=535628]: Waiting for error group managing segments... Dec 31 02:41:13.093790374 (611h12m12.964873409s) [noise=true , address=145.79.227.13:4803 , piece_id=KE5H4FKU4PZROEEFQ5KABCDEOX2NUXLUD4IZBMCVSSVXSPKKSYRA, seg_pos={0 0}, upload=535628]: Done uploading piece. err: <nil> Dec 31 02:41:13.099111968 (611h12m12.970195003s) [noise=true , address=145.79.227.4:4186 , piece_id=OFMISBUSLMHHEAELL2DDDTKND3UDZY2J7F7ASCTYTXYZOABZ4ZHQ, seg_pos={0 0}, upload=535628]: Done uploading piece. err: <nil> Dec 31 02:41:13.100327292 (611h12m12.971410327s) [noise=true , address=145.79.227.25:4618 , piece_id=MZWHT24LGJXQVAE7X7K6BPVZMP7X6QETTMICMNCSYI4QBMTSUAMA, seg_pos={0 0}, upload=535628]: Done uploading piece. err: <nil> Dec 31 02:41:13.102226227 (611h12m12.973309262s) [noise=true , address=145.79.227.32:4835 , piece_id=AVAICWG5CZ2FJEM3B72BGCZHTB234Q4TDRY635TDK2GCYGNNCMNQ, seg_pos={0 0}, upload=535628]: Done uploading piece. err: <nil> Dec 31 02:41:13.126767006 (611h12m12.997850041s) [noise=true , address=145.79.227.51:4410 , piece_id=EOFZJEXLH6XYMUQCIWTU7KFZETRL3RI4FUN2OX7RQXKQLOD6JOOQ, seg_pos={0 0}, upload=535628]: Done uploading piece. err: <nil> Dec 31 02:41:13.142304611 (611h12m13.013387646s) [noise=true , address=145.79.227.50:4374 , piece_id=HRTWPEWMLYB24UEUFTHZHMRWAJJN6D3VKZ3YZ6B26LN54DQXVSAA, seg_pos={0 0}, upload=535628]: Done uploading piece. err: <nil> Dec 31 02:41:13.181506727 (611h12m13.052589762s) [noise=true , address=145.79.227.39:4013 , piece_id=4EMEEV2RWPB5EE2Q4R4COOOLPC73M4ZE57BEU6G54ULD4PVHZAFQ, seg_pos={0 0}, upload=535628]: Done uploading piece. err: <nil> Dec 31 02:41:13.182708215 (611h12m13.05379125s) [noise=true , address=145.79.227.6:4258 , piece_id=BG43YG3ZELUSXPNFUZGOQQYOLXZGLURSHDDGIFVSG7DBE3JHCGHA, seg_pos={0 0}, upload=535628]: Done uploading piece. err: <nil> Dec 31 02:41:13.185142701 (611h12m13.056225736s) [noise=true , address=145.79.227.10:4403 , piece_id=D3XBXX6R5MHDLBWVSWZBQJJDYBK4VKXOMPAU5QVVG6FIVHKWKVNA, seg_pos={0 0}, upload=535628]: Done uploading piece. err: <nil> Dec 31 02:41:13.297136831 (611h12m13.168219866s) [noise=true , address=77.90.31.10:4518 , piece_id=FTGU3JK3T27PZJ3MNQ735ICXBCFBSQRKVWSJCB4L2PY5QRMAJDIA, seg_pos={0 0}, upload=535628]: Done uploading piece. err: <nil> Dec 31 02:41:13.337206979 (611h12m13.208290014s) [noise=true , address=77.90.31.11:4556 , piece_id=C5XVJ3QR25OBV3SQGCXATJHTMRQU76ZSSL3J4MCOYSMLMCZBJ3DQ, seg_pos={0 0}, upload=535628]: Done uploading piece. err: <nil> Dec 31 02:41:13.344521868 (611h12m13.215604903s) [noise=true , address=77.90.31.15:4697 , piece_id=NMLIRYJXAH47RJP5ZSN5PCCPLKNZ2K2ANIMGBU42XRGP7NMHOOAQ, seg_pos={0 0}, upload=535628]: Done uploading piece. err: <nil> Dec 31 02:41:13.512080229 (611h12m13.383163264s) [noise=true , address=45.91.116.11:4118 , piece_id=KLMGJP6XKFN23LRSEPCX52UBXRFJPHBQ3RWB66IIJXSKG2LIN7KA, seg_pos={0 0}, upload=535628]: Done uploading piece. err: <nil> Dec 31 02:41:13.512104606 (611h12m13.383187641s) [seg_pos={0 0}, upload=535628]: Segment reached optimal threshold of 13 pieces. Dec 31 02:41:13.512325146 (611h12m13.383408181s) [noise=true , address=77.90.31.14:4664 , piece_id=QYK7QKZ3KAIVPXVFPM45DJU4OB3KGDCPEOSZYIT44BD4P3EZCK3A, seg_pos={0 0}, upload=535628]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: write tcp 188.166.225.170:50786->77.90.31.14:4664: use of closed network connection; write tcp 188.166.225.170:50786->77.90.31.14:4664: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50786->77.90.31.14:4664: use of closed network connection Dec 31 02:41:13.512331317 (611h12m13.383414362s) [noise=true , address=77.90.31.14:4664 , piece_id=QYK7QKZ3KAIVPXVFPM45DJU4OB3KGDCPEOSZYIT44BD4P3EZCK3A, seg_pos={0 0}, upload=535628]: Long tail cancellation due to optimal threshold Dec 31 02:41:13.512609918 (611h12m13.383692953s) [noise=true , address=45.91.116.14:4190 , piece_id=UFR6AIVXP5ZPPQLVEZXKRSFB6CAZ7U7IL4CWCCKVRA5P3MC5MTLA, seg_pos={0 0}, upload=535628]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: write tcp 188.166.225.170:42822->45.91.116.14:4190: use of closed network connection; write tcp 188.166.225.170:42822->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42822->45.91.116.14:4190: use of closed network connection Dec 31 02:41:13.512626390 (611h12m13.383709435s) [noise=true , address=45.91.116.14:4190 , piece_id=UFR6AIVXP5ZPPQLVEZXKRSFB6CAZ7U7IL4CWCCKVRA5P3MC5MTLA, seg_pos={0 0}, upload=535628]: Long tail cancellation due to optimal threshold Dec 31 02:41:13.512787786 (611h12m13.383870821s) [noise=true , address=45.91.116.9:4263 , piece_id=QVIP6EJNGTD5FIMBDS2NCOK6R27OH5VAHK36T75HBI5F7KKH7YUQ, seg_pos={0 0}, upload=535628]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 31 02:41:13.512793036 (611h12m13.383876071s) [noise=true , address=45.91.116.9:4263 , piece_id=QVIP6EJNGTD5FIMBDS2NCOK6R27OH5VAHK36T75HBI5F7KKH7YUQ, seg_pos={0 0}, upload=535628]: Long tail cancellation due to optimal threshold Dec 31 02:41:13.512814998 (611h12m13.383898034s) [seg_pos={0 0}, upload=535628]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 02:41:13.512893738 (611h12m13.383976773s) [upload=535628]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 02:41:13.512898317 (611h12m13.383981352s) [upload=535628]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 02:41:13.802131863 (611h12m13.673214918s) [upload=535628]: Done uploading segments... Dec 31 02:41:13.802155628 (611h12m13.673238663s) [upload=535628]: Upload finished. err: <nil> Duration: 6.168905727s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 34.524024573s trace: https://jaeger.dev.storj.io/trace/365693fea00df28 FAIL (2025-12-31 02:42:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [noise=true , address=145.79.227.49:4343 , piece_id=6L3HPTS7D2NRA4QEOH4RFM3D3B53CTCONRI6G6YVU4OOTUWTG7XA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.942060847 (610h25m24.813143892s) [noise=true , address=145.79.227.34:4937 , piece_id=5W7LBM3YJJTVUFYTELR57DVXB663OHO5N3Z2SPQPZ4G43IK47O3Q, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.942684104 (610h25m24.813767139s) [noise=true , address=64.49.13.139:4478 , piece_id=QN6RT2KWEA6APQVA66ZMKG4LH537UGYNOXOCVVA2YC2MKDOLT3YA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.943064007 (610h25m24.814147042s) [noise=true , address=145.79.227.40:4084 , piece_id=RUKTKDII6BRUWSYVS65CWUBT3QKG3ASUK3KA3O5VGE53XSG6IMOQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.943459560 (610h25m24.814542605s) [noise=true , address=145.79.227.14:4847 , piece_id=JX5NXTMVM3NSWIEVR4RX46ABSNPKTOFDLFBPKEXFQ3G3ANKV5OLA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.943865303 (610h25m24.814948348s) [noise=true , address=45.91.116.6:4981 , piece_id=CT474ZA2LVAZEBJHBDD4GB6NTHVHZFJLNE5OZV3DFS2NZKVW3EQQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.944259694 (610h25m24.815342749s) [noise=true , address=145.79.227.51:4443 , piece_id=73UUMRDY7BSZNCGBXCZYYRBVE5KLPMNPRNXOZEGWKQGLHJS32J2Q, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.946575182 (610h25m24.817658227s) [noise=true , address=77.90.31.10:4523 , piece_id=P6LIONXJI6AFZ2JTKCQ2W6RZCOZCK6MZ53EXAJKXYQXSX2PX4QEA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.948132909 (610h25m24.819215944s) [noise=true , address=145.79.227.20:4452 , piece_id=ZSDN2JZ3FQUVJYN5OURDXNKEDLDNULJOHKKRZVIPXZIWIYAHCRXQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.949605774 (610h25m24.820688819s) [noise=true , address=77.90.31.14:4680 , piece_id=S2E4WBA7MJVAZLT5JG3IMHK53RQC77CCHV2PDF6L2R2JEA2U3K4Q, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.951459002 (610h25m24.822542048s) [noise=true , address=193.149.164.143:4704 , piece_id=BHXHFBKTLTDMGXWY5QRTP2THZQE5OXFZ2LCMTO2KRYRWAHXWW3NQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.952986742 (610h25m24.824069777s) [noise=true , address=77.90.31.19:4848 , piece_id=VZYG32S6ODIPDKQYZRXECBHXOERZDDWD2OF35Z46W4XMUZ3OJLAA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.956916654 (610h25m24.827999689s) [noise=true , address=145.79.227.29:4971 , piece_id=MEPXHHOQ6SUZC3GIPCQEMOKKF5OJTKZEI3UYZBAYXAQZGSVEKSBA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.960665062 (610h25m24.831748097s) [noise=true , address=64.49.13.170:4720 , piece_id=HIATRSAPE4YLSODFZDPP43ZUMLDVM7GJEVMWE4ZGUQTP25LTHGTA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.962366001 (610h25m24.833449036s) [noise=true , address=145.79.227.2:4117 , piece_id=4SGFGB3IJ7LTXDMMWM2PGGOV64MC46IZ7OZQWH55J4NBQY7CHMZQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.963708216 (610h25m24.834791261s) [noise=true , address=145.79.227.35:4505 , piece_id=V7RS7A4QLZTH4G32ZUZNSF2Z5APVWZGJO77NEGP3SBWQT3WCMVLQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:26.352236138 (610h25m26.223319173s) [upload=534952]: Next returned nil segment Dec 31 01:54:26.352253821 (610h25m26.223336857s) [upload=534952]: Waiting for error group managing segments... Dec 31 01:54:26.812758374 (610h25m26.683841409s) [noise=true , address=145.79.227.51:4443 , piece_id=73UUMRDY7BSZNCGBXCZYYRBVE5KLPMNPRNXOZEGWKQGLHJS32J2Q, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.824371056 (610h25m26.695454101s) [noise=true , address=145.79.227.14:4847 , piece_id=JX5NXTMVM3NSWIEVR4RX46ABSNPKTOFDLFBPKEXFQ3G3ANKV5OLA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.838414046 (610h25m26.709497081s) [noise=true , address=64.49.13.138:4490 , piece_id=2XTUZE6PXRE3OARNJVT36FXXWUMYIBVNE6M2VVFAVBGKEE2H5YZQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.875499841 (610h25m26.746582876s) [noise=true , address=64.49.13.139:4478 , piece_id=QN6RT2KWEA6APQVA66ZMKG4LH537UGYNOXOCVVA2YC2MKDOLT3YA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.891607221 (610h25m26.762690256s) [noise=true , address=145.79.227.20:4452 , piece_id=ZSDN2JZ3FQUVJYN5OURDXNKEDLDNULJOHKKRZVIPXZIWIYAHCRXQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.988561223 (610h25m26.859644268s) [noise=true , address=64.49.13.170:4720 , piece_id=HIATRSAPE4YLSODFZDPP43ZUMLDVM7GJEVMWE4ZGUQTP25LTHGTA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.039035911 (610h25m26.910118946s) [noise=true , address=77.90.31.14:4680 , piece_id=S2E4WBA7MJVAZLT5JG3IMHK53RQC77CCHV2PDF6L2R2JEA2U3K4Q, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.098023090 (610h25m26.969106126s) [noise=true , address=45.91.116.6:4981 , piece_id=CT474ZA2LVAZEBJHBDD4GB6NTHVHZFJLNE5OZV3DFS2NZKVW3EQQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.653790754 (610h25m27.524873789s) [noise=true , address=77.90.31.21:4886 , piece_id=K34WPIBGGGEPHIKJ253XGLDCIQ3WLFN4AZ6EYNWVHHT2EIHS6YRQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.684192972 (610h25m27.555276017s) [noise=true , address=193.149.164.143:4704 , piece_id=BHXHFBKTLTDMGXWY5QRTP2THZQE5OXFZ2LCMTO2KRYRWAHXWW3NQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.717238425 (610h25m27.58832146s) [noise=true , address=145.79.227.40:4084 , piece_id=RUKTKDII6BRUWSYVS65CWUBT3QKG3ASUK3KA3O5VGE53XSG6IMOQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.718349742 (610h25m27.589432777s) [noise=true , address=77.90.31.10:4523 , piece_id=P6LIONXJI6AFZ2JTKCQ2W6RZCOZCK6MZ53EXAJKXYQXSX2PX4QEA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.803639562 (610h25m27.674722597s) [noise=true , address=145.79.227.29:4971 , piece_id=MEPXHHOQ6SUZC3GIPCQEMOKKF5OJTKZEI3UYZBAYXAQZGSVEKSBA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.812791279 (610h25m27.683874314s) [noise=true , address=145.79.227.49:4343 , piece_id=6L3HPTS7D2NRA4QEOH4RFM3D3B53CTCONRI6G6YVU4OOTUWTG7XA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.812812348 (610h25m27.683895393s) [seg_pos={0 0}, upload=534952]: Segment reached optimal threshold of 14 pieces. Dec 31 01:54:27.813018000 (610h25m27.684101035s) [noise=true , address=145.79.227.35:4505 , piece_id=V7RS7A4QLZTH4G32ZUZNSF2Z5APVWZGJO77NEGP3SBWQT3WCMVLQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i6itGoBhLsLiqC8et8Y5L62fLjcWxy18u39NAr4R3nXNMWcsf): protocol: expected piece hash; context canceled; EOF Dec 31 01:54:27.813024772 (610h25m27.684107808s) [noise=true , address=145.79.227.35:4505 , piece_id=V7RS7A4QLZTH4G32ZUZNSF2Z5APVWZGJO77NEGP3SBWQT3WCMVLQ, seg_pos={0 0}, upload=534952]: Long tail cancellation due to optimal threshold Dec 31 01:54:27.813175931 (610h25m27.684258976s) [noise=true , address=145.79.227.34:4937 , piece_id=5W7LBM3YJJTVUFYTELR57DVXB663OHO5N3Z2SPQPZ4G43IK47O3Q, seg_pos={0 0}, upload=534952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1LKrFThdW3jJpG6dRZLUaxLXTa5Wy27yQ8PGUEqjvBWfFoRB94): protocol: expected piece hash; context canceled; EOF Dec 31 01:54:27.813181702 (610h25m27.684264737s) [noise=true , address=145.79.227.34:4937 , piece_id=5W7LBM3YJJTVUFYTELR57DVXB663OHO5N3Z2SPQPZ4G43IK47O3Q, seg_pos={0 0}, upload=534952]: Long tail cancellation due to optimal threshold Dec 31 01:54:27.813407862 (610h25m27.684490907s) [noise=true , address=145.79.227.2:4117 , piece_id=4SGFGB3IJ7LTXDMMWM2PGGOV64MC46IZ7OZQWH55J4NBQY7CHMZQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122Htg72f5eUUutf1p3fYQpy1Wkf6AgdaQV83HHYye2n8zm3the): protocol: expected piece hash; context canceled; EOF Dec 31 01:54:27.813414264 (610h25m27.684497299s) [noise=true , address=145.79.227.2:4117 , piece_id=4SGFGB3IJ7LTXDMMWM2PGGOV64MC46IZ7OZQWH55J4NBQY7CHMZQ, seg_pos={0 0}, upload=534952]: Long tail cancellation due to optimal threshold Dec 31 01:54:27.813424614 (610h25m27.684507649s) [noise=true , address=77.90.31.19:4848 , piece_id=VZYG32S6ODIPDKQYZRXECBHXOERZDDWD2OF35Z46W4XMUZ3OJLAA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD): protocol: write tcp 188.166.225.170:42842->77.90.31.19:4848: use of closed network connection; write tcp 188.166.225.170:42842->77.90.31.19:4848: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42842->77.90.31.19:4848: use of closed network connection Dec 31 01:54:27.813432750 (610h25m27.684515785s) [noise=true , address=77.90.31.19:4848 , piece_id=VZYG32S6ODIPDKQYZRXECBHXOERZDDWD2OF35Z46W4XMUZ3OJLAA, seg_pos={0 0}, upload=534952]: Long tail cancellation due to optimal threshold Dec 31 01:54:27.813474088 (610h25m27.684557123s) [seg_pos={0 0}, upload=534952]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 01:54:27.813535325 (610h25m27.68461836s) [upload=534952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 01:54:27.813539613 (610h25m27.684622648s) [upload=534952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 01:54:28.099921034 (610h25m27.971004069s) [upload=534952]: Done uploading segments... Dec 31 01:54:28.099940451 (610h25m27.971023486s) [upload=534952]: Upload finished. err: <nil> Duration: 34.524041727s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.................X...............................X....X.....] duration: 3.501219096s trace: https://jaeger.dev.storj.io/trace/7bc7478624e56c1a FAIL (2025-12-31 02:42:11 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.501244605s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [X..X.......X.X..X....XX.XX........X.X......X..X......XXX..X.] duration: 7.937240197s trace: https://jaeger.dev.storj.io/trace/7796d308e68d2eba FAIL (2025-12-31 02:41:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, Piece ID: C5XVJ3QR25OBV3SQGCXATJHTMRQU76ZSSL3J4MCOYSMLMCZBJ3DQ) context deadline exceeded; piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: FTGU3JK3T27PZJ3MNQ735ICXBCFBSQRKVWSJCB4L2PY5QRMAJDIA) context deadline exceeded ======LOGS====== egments... Dec 31 02:41:10.854740410 (611h12m10.725823465s) [upload=535628]: Got next segment. Inline: false Dec 31 02:41:10.854780267 (611h12m10.725863312s) [upload=535628]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 02:41:10.854786138 (611h12m10.725869173s) [upload=535628]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 02:41:11.103813035 (611h12m10.974896091s) [seg_pos={0 0}, upload=535628]: Begin upload segment... Dec 31 02:41:11.103869233 (611h12m10.974952278s) [seg_pos={0 0}, upload=535628]: Done begin upload segment. Dec 31 02:41:11.103917354 (611h12m10.975000389s) [noise=true , address=145.79.227.39:4013 , piece_id=4EMEEV2RWPB5EE2Q4R4COOOLPC73M4ZE57BEU6G54ULD4PVHZAFQ, seg_pos={0 0}, upload=535628]: Uploading piece... Dec 31 02:41:11.103937894 (611h12m10.975020939s) [noise=true , address=77.90.31.10:4518 , piece_id=FTGU3JK3T27PZJ3MNQ735ICXBCFBSQRKVWSJCB4L2PY5QRMAJDIA, seg_pos={0 0}, upload=535628]: Uploading piece... Dec 31 02:41:11.104433838 (611h12m10.975516873s) [noise=true , address=145.79.227.6:4258 , piece_id=BG43YG3ZELUSXPNFUZGOQQYOLXZGLURSHDDGIFVSG7DBE3JHCGHA, seg_pos={0 0}, upload=535628]: Uploading piece... Dec 31 02:41:11.104787361 (611h12m10.975870396s) [noise=true , address=45.91.116.9:4263 , piece_id=QVIP6EJNGTD5FIMBDS2NCOK6R27OH5VAHK36T75HBI5F7KKH7YUQ, seg_pos={0 0}, upload=535628]: Uploading piece... Dec 31 02:41:11.105149000 (611h12m10.976232035s) [noise=true , address=145.79.227.4:4186 , piece_id=OFMISBUSLMHHEAELL2DDDTKND3UDZY2J7F7ASCTYTXYZOABZ4ZHQ, seg_pos={0 0}, upload=535628]: Uploading piece... Dec 31 02:41:11.105561805 (611h12m10.97664485s) [noise=true , address=145.79.227.32:4835 , piece_id=AVAICWG5CZ2FJEM3B72BGCZHTB234Q4TDRY635TDK2GCYGNNCMNQ, seg_pos={0 0}, upload=535628]: Uploading piece... Dec 31 02:41:11.106056818 (611h12m10.977139853s) [noise=true , address=145.79.227.51:4410 , piece_id=EOFZJEXLH6XYMUQCIWTU7KFZETRL3RI4FUN2OX7RQXKQLOD6JOOQ, seg_pos={0 0}, upload=535628]: Uploading piece... Dec 31 02:41:11.106636882 (611h12m10.977719948s) [noise=true , address=45.91.116.14:4190 , piece_id=UFR6AIVXP5ZPPQLVEZXKRSFB6CAZ7U7IL4CWCCKVRA5P3MC5MTLA, seg_pos={0 0}, upload=535628]: Uploading piece... Dec 31 02:41:11.107402160 (611h12m10.978485195s) [noise=true , address=77.90.31.15:4697 , piece_id=NMLIRYJXAH47RJP5ZSN5PCCPLKNZ2K2ANIMGBU42XRGP7NMHOOAQ, seg_pos={0 0}, upload=535628]: Uploading piece... Dec 31 02:41:11.107995509 (611h12m10.979078545s) [noise=true , address=145.79.227.50:4374 , piece_id=HRTWPEWMLYB24UEUFTHZHMRWAJJN6D3VKZ3YZ6B26LN54DQXVSAA, seg_pos={0 0}, upload=535628]: Uploading piece... Dec 31 02:41:11.108580824 (611h12m10.979663859s) [noise=true , address=145.79.227.25:4618 , piece_id=MZWHT24LGJXQVAE7X7K6BPVZMP7X6QETTMICMNCSYI4QBMTSUAMA, seg_pos={0 0}, upload=535628]: Uploading piece... Dec 31 02:41:11.109458766 (611h12m10.980541801s) [noise=true , address=45.91.116.11:4118 , piece_id=KLMGJP6XKFN23LRSEPCX52UBXRFJPHBQ3RWB66IIJXSKG2LIN7KA, seg_pos={0 0}, upload=535628]: Uploading piece... Dec 31 02:41:11.111091665 (611h12m10.9821747s) [noise=true , address=145.79.227.10:4403 , piece_id=D3XBXX6R5MHDLBWVSWZBQJJDYBK4VKXOMPAU5QVVG6FIVHKWKVNA, seg_pos={0 0}, upload=535628]: Uploading piece... Dec 31 02:41:11.112772295 (611h12m10.98385533s) [noise=true , address=77.90.31.11:4556 , piece_id=C5XVJ3QR25OBV3SQGCXATJHTMRQU76ZSSL3J4MCOYSMLMCZBJ3DQ, seg_pos={0 0}, upload=535628]: Uploading piece... Dec 31 02:41:11.115140324 (611h12m10.986223359s) [noise=true , address=145.79.227.13:4803 , piece_id=KE5H4FKU4PZROEEFQ5KABCDEOX2NUXLUD4IZBMCVSSVXSPKKSYRA, seg_pos={0 0}, upload=535628]: Uploading piece... Dec 31 02:41:11.117165320 (611h12m10.988248355s) [noise=true , address=77.90.31.14:4664 , piece_id=QYK7QKZ3KAIVPXVFPM45DJU4OB3KGDCPEOSZYIT44BD4P3EZCK3A, seg_pos={0 0}, upload=535628]: Uploading piece... Dec 31 02:41:12.815648949 (611h12m12.686731984s) [upload=535628]: Next returned nil segment Dec 31 02:41:12.815663367 (611h12m12.686746402s) [upload=535628]: Waiting for error group managing segments... Dec 31 02:41:13.093790374 (611h12m12.964873409s) [noise=true , address=145.79.227.13:4803 , piece_id=KE5H4FKU4PZROEEFQ5KABCDEOX2NUXLUD4IZBMCVSSVXSPKKSYRA, seg_pos={0 0}, upload=535628]: Done uploading piece. err: <nil> Dec 31 02:41:13.099111968 (611h12m12.970195003s) [noise=true , address=145.79.227.4:4186 , piece_id=OFMISBUSLMHHEAELL2DDDTKND3UDZY2J7F7ASCTYTXYZOABZ4ZHQ, seg_pos={0 0}, upload=535628]: Done uploading piece. err: <nil> Dec 31 02:41:13.100327292 (611h12m12.971410327s) [noise=true , address=145.79.227.25:4618 , piece_id=MZWHT24LGJXQVAE7X7K6BPVZMP7X6QETTMICMNCSYI4QBMTSUAMA, seg_pos={0 0}, upload=535628]: Done uploading piece. err: <nil> Dec 31 02:41:13.102226227 (611h12m12.973309262s) [noise=true , address=145.79.227.32:4835 , piece_id=AVAICWG5CZ2FJEM3B72BGCZHTB234Q4TDRY635TDK2GCYGNNCMNQ, seg_pos={0 0}, upload=535628]: Done uploading piece. err: <nil> Dec 31 02:41:13.126767006 (611h12m12.997850041s) [noise=true , address=145.79.227.51:4410 , piece_id=EOFZJEXLH6XYMUQCIWTU7KFZETRL3RI4FUN2OX7RQXKQLOD6JOOQ, seg_pos={0 0}, upload=535628]: Done uploading piece. err: <nil> Dec 31 02:41:13.142304611 (611h12m13.013387646s) [noise=true , address=145.79.227.50:4374 , piece_id=HRTWPEWMLYB24UEUFTHZHMRWAJJN6D3VKZ3YZ6B26LN54DQXVSAA, seg_pos={0 0}, upload=535628]: Done uploading piece. err: <nil> Dec 31 02:41:13.181506727 (611h12m13.052589762s) [noise=true , address=145.79.227.39:4013 , piece_id=4EMEEV2RWPB5EE2Q4R4COOOLPC73M4ZE57BEU6G54ULD4PVHZAFQ, seg_pos={0 0}, upload=535628]: Done uploading piece. err: <nil> Dec 31 02:41:13.182708215 (611h12m13.05379125s) [noise=true , address=145.79.227.6:4258 , piece_id=BG43YG3ZELUSXPNFUZGOQQYOLXZGLURSHDDGIFVSG7DBE3JHCGHA, seg_pos={0 0}, upload=535628]: Done uploading piece. err: <nil> Dec 31 02:41:13.185142701 (611h12m13.056225736s) [noise=true , address=145.79.227.10:4403 , piece_id=D3XBXX6R5MHDLBWVSWZBQJJDYBK4VKXOMPAU5QVVG6FIVHKWKVNA, seg_pos={0 0}, upload=535628]: Done uploading piece. err: <nil> Dec 31 02:41:13.297136831 (611h12m13.168219866s) [noise=true , address=77.90.31.10:4518 , piece_id=FTGU3JK3T27PZJ3MNQ735ICXBCFBSQRKVWSJCB4L2PY5QRMAJDIA, seg_pos={0 0}, upload=535628]: Done uploading piece. err: <nil> Dec 31 02:41:13.337206979 (611h12m13.208290014s) [noise=true , address=77.90.31.11:4556 , piece_id=C5XVJ3QR25OBV3SQGCXATJHTMRQU76ZSSL3J4MCOYSMLMCZBJ3DQ, seg_pos={0 0}, upload=535628]: Done uploading piece. err: <nil> Dec 31 02:41:13.344521868 (611h12m13.215604903s) [noise=true , address=77.90.31.15:4697 , piece_id=NMLIRYJXAH47RJP5ZSN5PCCPLKNZ2K2ANIMGBU42XRGP7NMHOOAQ, seg_pos={0 0}, upload=535628]: Done uploading piece. err: <nil> Dec 31 02:41:13.512080229 (611h12m13.383163264s) [noise=true , address=45.91.116.11:4118 , piece_id=KLMGJP6XKFN23LRSEPCX52UBXRFJPHBQ3RWB66IIJXSKG2LIN7KA, seg_pos={0 0}, upload=535628]: Done uploading piece. err: <nil> Dec 31 02:41:13.512104606 (611h12m13.383187641s) [seg_pos={0 0}, upload=535628]: Segment reached optimal threshold of 13 pieces. Dec 31 02:41:13.512325146 (611h12m13.383408181s) [noise=true , address=77.90.31.14:4664 , piece_id=QYK7QKZ3KAIVPXVFPM45DJU4OB3KGDCPEOSZYIT44BD4P3EZCK3A, seg_pos={0 0}, upload=535628]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: write tcp 188.166.225.170:50786->77.90.31.14:4664: use of closed network connection; write tcp 188.166.225.170:50786->77.90.31.14:4664: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50786->77.90.31.14:4664: use of closed network connection Dec 31 02:41:13.512331317 (611h12m13.383414362s) [noise=true , address=77.90.31.14:4664 , piece_id=QYK7QKZ3KAIVPXVFPM45DJU4OB3KGDCPEOSZYIT44BD4P3EZCK3A, seg_pos={0 0}, upload=535628]: Long tail cancellation due to optimal threshold Dec 31 02:41:13.512609918 (611h12m13.383692953s) [noise=true , address=45.91.116.14:4190 , piece_id=UFR6AIVXP5ZPPQLVEZXKRSFB6CAZ7U7IL4CWCCKVRA5P3MC5MTLA, seg_pos={0 0}, upload=535628]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: write tcp 188.166.225.170:42822->45.91.116.14:4190: use of closed network connection; write tcp 188.166.225.170:42822->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42822->45.91.116.14:4190: use of closed network connection Dec 31 02:41:13.512626390 (611h12m13.383709435s) [noise=true , address=45.91.116.14:4190 , piece_id=UFR6AIVXP5ZPPQLVEZXKRSFB6CAZ7U7IL4CWCCKVRA5P3MC5MTLA, seg_pos={0 0}, upload=535628]: Long tail cancellation due to optimal threshold Dec 31 02:41:13.512787786 (611h12m13.383870821s) [noise=true , address=45.91.116.9:4263 , piece_id=QVIP6EJNGTD5FIMBDS2NCOK6R27OH5VAHK36T75HBI5F7KKH7YUQ, seg_pos={0 0}, upload=535628]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 31 02:41:13.512793036 (611h12m13.383876071s) [noise=true , address=45.91.116.9:4263 , piece_id=QVIP6EJNGTD5FIMBDS2NCOK6R27OH5VAHK36T75HBI5F7KKH7YUQ, seg_pos={0 0}, upload=535628]: Long tail cancellation due to optimal threshold Dec 31 02:41:13.512814998 (611h12m13.383898034s) [seg_pos={0 0}, upload=535628]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 02:41:13.512893738 (611h12m13.383976773s) [upload=535628]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 02:41:13.512898317 (611h12m13.383981352s) [upload=535628]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 02:41:13.802131863 (611h12m13.673214918s) [upload=535628]: Done uploading segments... Dec 31 02:41:13.802155628 (611h12m13.673238663s) [upload=535628]: Upload finished. err: <nil> Duration: 7.937257039s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 24.498518458s trace: https://jaeger.dev.storj.io/trace/365693fea00df28 FAIL (2025-12-31 02:41:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [noise=true , address=145.79.227.49:4343 , piece_id=6L3HPTS7D2NRA4QEOH4RFM3D3B53CTCONRI6G6YVU4OOTUWTG7XA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.942060847 (610h25m24.813143892s) [noise=true , address=145.79.227.34:4937 , piece_id=5W7LBM3YJJTVUFYTELR57DVXB663OHO5N3Z2SPQPZ4G43IK47O3Q, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.942684104 (610h25m24.813767139s) [noise=true , address=64.49.13.139:4478 , piece_id=QN6RT2KWEA6APQVA66ZMKG4LH537UGYNOXOCVVA2YC2MKDOLT3YA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.943064007 (610h25m24.814147042s) [noise=true , address=145.79.227.40:4084 , piece_id=RUKTKDII6BRUWSYVS65CWUBT3QKG3ASUK3KA3O5VGE53XSG6IMOQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.943459560 (610h25m24.814542605s) [noise=true , address=145.79.227.14:4847 , piece_id=JX5NXTMVM3NSWIEVR4RX46ABSNPKTOFDLFBPKEXFQ3G3ANKV5OLA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.943865303 (610h25m24.814948348s) [noise=true , address=45.91.116.6:4981 , piece_id=CT474ZA2LVAZEBJHBDD4GB6NTHVHZFJLNE5OZV3DFS2NZKVW3EQQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.944259694 (610h25m24.815342749s) [noise=true , address=145.79.227.51:4443 , piece_id=73UUMRDY7BSZNCGBXCZYYRBVE5KLPMNPRNXOZEGWKQGLHJS32J2Q, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.946575182 (610h25m24.817658227s) [noise=true , address=77.90.31.10:4523 , piece_id=P6LIONXJI6AFZ2JTKCQ2W6RZCOZCK6MZ53EXAJKXYQXSX2PX4QEA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.948132909 (610h25m24.819215944s) [noise=true , address=145.79.227.20:4452 , piece_id=ZSDN2JZ3FQUVJYN5OURDXNKEDLDNULJOHKKRZVIPXZIWIYAHCRXQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.949605774 (610h25m24.820688819s) [noise=true , address=77.90.31.14:4680 , piece_id=S2E4WBA7MJVAZLT5JG3IMHK53RQC77CCHV2PDF6L2R2JEA2U3K4Q, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.951459002 (610h25m24.822542048s) [noise=true , address=193.149.164.143:4704 , piece_id=BHXHFBKTLTDMGXWY5QRTP2THZQE5OXFZ2LCMTO2KRYRWAHXWW3NQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.952986742 (610h25m24.824069777s) [noise=true , address=77.90.31.19:4848 , piece_id=VZYG32S6ODIPDKQYZRXECBHXOERZDDWD2OF35Z46W4XMUZ3OJLAA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.956916654 (610h25m24.827999689s) [noise=true , address=145.79.227.29:4971 , piece_id=MEPXHHOQ6SUZC3GIPCQEMOKKF5OJTKZEI3UYZBAYXAQZGSVEKSBA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.960665062 (610h25m24.831748097s) [noise=true , address=64.49.13.170:4720 , piece_id=HIATRSAPE4YLSODFZDPP43ZUMLDVM7GJEVMWE4ZGUQTP25LTHGTA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.962366001 (610h25m24.833449036s) [noise=true , address=145.79.227.2:4117 , piece_id=4SGFGB3IJ7LTXDMMWM2PGGOV64MC46IZ7OZQWH55J4NBQY7CHMZQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.963708216 (610h25m24.834791261s) [noise=true , address=145.79.227.35:4505 , piece_id=V7RS7A4QLZTH4G32ZUZNSF2Z5APVWZGJO77NEGP3SBWQT3WCMVLQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:26.352236138 (610h25m26.223319173s) [upload=534952]: Next returned nil segment Dec 31 01:54:26.352253821 (610h25m26.223336857s) [upload=534952]: Waiting for error group managing segments... Dec 31 01:54:26.812758374 (610h25m26.683841409s) [noise=true , address=145.79.227.51:4443 , piece_id=73UUMRDY7BSZNCGBXCZYYRBVE5KLPMNPRNXOZEGWKQGLHJS32J2Q, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.824371056 (610h25m26.695454101s) [noise=true , address=145.79.227.14:4847 , piece_id=JX5NXTMVM3NSWIEVR4RX46ABSNPKTOFDLFBPKEXFQ3G3ANKV5OLA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.838414046 (610h25m26.709497081s) [noise=true , address=64.49.13.138:4490 , piece_id=2XTUZE6PXRE3OARNJVT36FXXWUMYIBVNE6M2VVFAVBGKEE2H5YZQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.875499841 (610h25m26.746582876s) [noise=true , address=64.49.13.139:4478 , piece_id=QN6RT2KWEA6APQVA66ZMKG4LH537UGYNOXOCVVA2YC2MKDOLT3YA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.891607221 (610h25m26.762690256s) [noise=true , address=145.79.227.20:4452 , piece_id=ZSDN2JZ3FQUVJYN5OURDXNKEDLDNULJOHKKRZVIPXZIWIYAHCRXQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.988561223 (610h25m26.859644268s) [noise=true , address=64.49.13.170:4720 , piece_id=HIATRSAPE4YLSODFZDPP43ZUMLDVM7GJEVMWE4ZGUQTP25LTHGTA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.039035911 (610h25m26.910118946s) [noise=true , address=77.90.31.14:4680 , piece_id=S2E4WBA7MJVAZLT5JG3IMHK53RQC77CCHV2PDF6L2R2JEA2U3K4Q, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.098023090 (610h25m26.969106126s) [noise=true , address=45.91.116.6:4981 , piece_id=CT474ZA2LVAZEBJHBDD4GB6NTHVHZFJLNE5OZV3DFS2NZKVW3EQQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.653790754 (610h25m27.524873789s) [noise=true , address=77.90.31.21:4886 , piece_id=K34WPIBGGGEPHIKJ253XGLDCIQ3WLFN4AZ6EYNWVHHT2EIHS6YRQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.684192972 (610h25m27.555276017s) [noise=true , address=193.149.164.143:4704 , piece_id=BHXHFBKTLTDMGXWY5QRTP2THZQE5OXFZ2LCMTO2KRYRWAHXWW3NQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.717238425 (610h25m27.58832146s) [noise=true , address=145.79.227.40:4084 , piece_id=RUKTKDII6BRUWSYVS65CWUBT3QKG3ASUK3KA3O5VGE53XSG6IMOQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.718349742 (610h25m27.589432777s) [noise=true , address=77.90.31.10:4523 , piece_id=P6LIONXJI6AFZ2JTKCQ2W6RZCOZCK6MZ53EXAJKXYQXSX2PX4QEA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.803639562 (610h25m27.674722597s) [noise=true , address=145.79.227.29:4971 , piece_id=MEPXHHOQ6SUZC3GIPCQEMOKKF5OJTKZEI3UYZBAYXAQZGSVEKSBA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.812791279 (610h25m27.683874314s) [noise=true , address=145.79.227.49:4343 , piece_id=6L3HPTS7D2NRA4QEOH4RFM3D3B53CTCONRI6G6YVU4OOTUWTG7XA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.812812348 (610h25m27.683895393s) [seg_pos={0 0}, upload=534952]: Segment reached optimal threshold of 14 pieces. Dec 31 01:54:27.813018000 (610h25m27.684101035s) [noise=true , address=145.79.227.35:4505 , piece_id=V7RS7A4QLZTH4G32ZUZNSF2Z5APVWZGJO77NEGP3SBWQT3WCMVLQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i6itGoBhLsLiqC8et8Y5L62fLjcWxy18u39NAr4R3nXNMWcsf): protocol: expected piece hash; context canceled; EOF Dec 31 01:54:27.813024772 (610h25m27.684107808s) [noise=true , address=145.79.227.35:4505 , piece_id=V7RS7A4QLZTH4G32ZUZNSF2Z5APVWZGJO77NEGP3SBWQT3WCMVLQ, seg_pos={0 0}, upload=534952]: Long tail cancellation due to optimal threshold Dec 31 01:54:27.813175931 (610h25m27.684258976s) [noise=true , address=145.79.227.34:4937 , piece_id=5W7LBM3YJJTVUFYTELR57DVXB663OHO5N3Z2SPQPZ4G43IK47O3Q, seg_pos={0 0}, upload=534952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1LKrFThdW3jJpG6dRZLUaxLXTa5Wy27yQ8PGUEqjvBWfFoRB94): protocol: expected piece hash; context canceled; EOF Dec 31 01:54:27.813181702 (610h25m27.684264737s) [noise=true , address=145.79.227.34:4937 , piece_id=5W7LBM3YJJTVUFYTELR57DVXB663OHO5N3Z2SPQPZ4G43IK47O3Q, seg_pos={0 0}, upload=534952]: Long tail cancellation due to optimal threshold Dec 31 01:54:27.813407862 (610h25m27.684490907s) [noise=true , address=145.79.227.2:4117 , piece_id=4SGFGB3IJ7LTXDMMWM2PGGOV64MC46IZ7OZQWH55J4NBQY7CHMZQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122Htg72f5eUUutf1p3fYQpy1Wkf6AgdaQV83HHYye2n8zm3the): protocol: expected piece hash; context canceled; EOF Dec 31 01:54:27.813414264 (610h25m27.684497299s) [noise=true , address=145.79.227.2:4117 , piece_id=4SGFGB3IJ7LTXDMMWM2PGGOV64MC46IZ7OZQWH55J4NBQY7CHMZQ, seg_pos={0 0}, upload=534952]: Long tail cancellation due to optimal threshold Dec 31 01:54:27.813424614 (610h25m27.684507649s) [noise=true , address=77.90.31.19:4848 , piece_id=VZYG32S6ODIPDKQYZRXECBHXOERZDDWD2OF35Z46W4XMUZ3OJLAA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD): protocol: write tcp 188.166.225.170:42842->77.90.31.19:4848: use of closed network connection; write tcp 188.166.225.170:42842->77.90.31.19:4848: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42842->77.90.31.19:4848: use of closed network connection Dec 31 01:54:27.813432750 (610h25m27.684515785s) [noise=true , address=77.90.31.19:4848 , piece_id=VZYG32S6ODIPDKQYZRXECBHXOERZDDWD2OF35Z46W4XMUZ3OJLAA, seg_pos={0 0}, upload=534952]: Long tail cancellation due to optimal threshold Dec 31 01:54:27.813474088 (610h25m27.684557123s) [seg_pos={0 0}, upload=534952]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 01:54:27.813535325 (610h25m27.68461836s) [upload=534952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 01:54:27.813539613 (610h25m27.684622648s) [upload=534952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 01:54:28.099921034 (610h25m27.971004069s) [upload=534952]: Done uploading segments... Dec 31 01:54:28.099940451 (610h25m27.971023486s) [upload=534952]: Upload finished. err: <nil> Duration: 24.498536123s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.X..X.......X.X..X....XX.XX........X.X......X..X......XXX..X] duration: 30.002192388s trace: https://jaeger.dev.storj.io/trace/7796d308e68d2eba FAIL (2025-12-31 02:41:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, Piece ID: C5XVJ3QR25OBV3SQGCXATJHTMRQU76ZSSL3J4MCOYSMLMCZBJ3DQ) context deadline exceeded; piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: FTGU3JK3T27PZJ3MNQ735ICXBCFBSQRKVWSJCB4L2PY5QRMAJDIA) context deadline exceeded ======LOGS====== egments... Dec 31 02:41:10.854740410 (611h12m10.725823465s) [upload=535628]: Got next segment. Inline: false Dec 31 02:41:10.854780267 (611h12m10.725863312s) [upload=535628]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 02:41:10.854786138 (611h12m10.725869173s) [upload=535628]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 02:41:11.103813035 (611h12m10.974896091s) [seg_pos={0 0}, upload=535628]: Begin upload segment... Dec 31 02:41:11.103869233 (611h12m10.974952278s) [seg_pos={0 0}, upload=535628]: Done begin upload segment. Dec 31 02:41:11.103917354 (611h12m10.975000389s) [noise=true , address=145.79.227.39:4013 , piece_id=4EMEEV2RWPB5EE2Q4R4COOOLPC73M4ZE57BEU6G54ULD4PVHZAFQ, seg_pos={0 0}, upload=535628]: Uploading piece... Dec 31 02:41:11.103937894 (611h12m10.975020939s) [noise=true , address=77.90.31.10:4518 , piece_id=FTGU3JK3T27PZJ3MNQ735ICXBCFBSQRKVWSJCB4L2PY5QRMAJDIA, seg_pos={0 0}, upload=535628]: Uploading piece... Dec 31 02:41:11.104433838 (611h12m10.975516873s) [noise=true , address=145.79.227.6:4258 , piece_id=BG43YG3ZELUSXPNFUZGOQQYOLXZGLURSHDDGIFVSG7DBE3JHCGHA, seg_pos={0 0}, upload=535628]: Uploading piece... Dec 31 02:41:11.104787361 (611h12m10.975870396s) [noise=true , address=45.91.116.9:4263 , piece_id=QVIP6EJNGTD5FIMBDS2NCOK6R27OH5VAHK36T75HBI5F7KKH7YUQ, seg_pos={0 0}, upload=535628]: Uploading piece... Dec 31 02:41:11.105149000 (611h12m10.976232035s) [noise=true , address=145.79.227.4:4186 , piece_id=OFMISBUSLMHHEAELL2DDDTKND3UDZY2J7F7ASCTYTXYZOABZ4ZHQ, seg_pos={0 0}, upload=535628]: Uploading piece... Dec 31 02:41:11.105561805 (611h12m10.97664485s) [noise=true , address=145.79.227.32:4835 , piece_id=AVAICWG5CZ2FJEM3B72BGCZHTB234Q4TDRY635TDK2GCYGNNCMNQ, seg_pos={0 0}, upload=535628]: Uploading piece... Dec 31 02:41:11.106056818 (611h12m10.977139853s) [noise=true , address=145.79.227.51:4410 , piece_id=EOFZJEXLH6XYMUQCIWTU7KFZETRL3RI4FUN2OX7RQXKQLOD6JOOQ, seg_pos={0 0}, upload=535628]: Uploading piece... Dec 31 02:41:11.106636882 (611h12m10.977719948s) [noise=true , address=45.91.116.14:4190 , piece_id=UFR6AIVXP5ZPPQLVEZXKRSFB6CAZ7U7IL4CWCCKVRA5P3MC5MTLA, seg_pos={0 0}, upload=535628]: Uploading piece... Dec 31 02:41:11.107402160 (611h12m10.978485195s) [noise=true , address=77.90.31.15:4697 , piece_id=NMLIRYJXAH47RJP5ZSN5PCCPLKNZ2K2ANIMGBU42XRGP7NMHOOAQ, seg_pos={0 0}, upload=535628]: Uploading piece... Dec 31 02:41:11.107995509 (611h12m10.979078545s) [noise=true , address=145.79.227.50:4374 , piece_id=HRTWPEWMLYB24UEUFTHZHMRWAJJN6D3VKZ3YZ6B26LN54DQXVSAA, seg_pos={0 0}, upload=535628]: Uploading piece... Dec 31 02:41:11.108580824 (611h12m10.979663859s) [noise=true , address=145.79.227.25:4618 , piece_id=MZWHT24LGJXQVAE7X7K6BPVZMP7X6QETTMICMNCSYI4QBMTSUAMA, seg_pos={0 0}, upload=535628]: Uploading piece... Dec 31 02:41:11.109458766 (611h12m10.980541801s) [noise=true , address=45.91.116.11:4118 , piece_id=KLMGJP6XKFN23LRSEPCX52UBXRFJPHBQ3RWB66IIJXSKG2LIN7KA, seg_pos={0 0}, upload=535628]: Uploading piece... Dec 31 02:41:11.111091665 (611h12m10.9821747s) [noise=true , address=145.79.227.10:4403 , piece_id=D3XBXX6R5MHDLBWVSWZBQJJDYBK4VKXOMPAU5QVVG6FIVHKWKVNA, seg_pos={0 0}, upload=535628]: Uploading piece... Dec 31 02:41:11.112772295 (611h12m10.98385533s) [noise=true , address=77.90.31.11:4556 , piece_id=C5XVJ3QR25OBV3SQGCXATJHTMRQU76ZSSL3J4MCOYSMLMCZBJ3DQ, seg_pos={0 0}, upload=535628]: Uploading piece... Dec 31 02:41:11.115140324 (611h12m10.986223359s) [noise=true , address=145.79.227.13:4803 , piece_id=KE5H4FKU4PZROEEFQ5KABCDEOX2NUXLUD4IZBMCVSSVXSPKKSYRA, seg_pos={0 0}, upload=535628]: Uploading piece... Dec 31 02:41:11.117165320 (611h12m10.988248355s) [noise=true , address=77.90.31.14:4664 , piece_id=QYK7QKZ3KAIVPXVFPM45DJU4OB3KGDCPEOSZYIT44BD4P3EZCK3A, seg_pos={0 0}, upload=535628]: Uploading piece... Dec 31 02:41:12.815648949 (611h12m12.686731984s) [upload=535628]: Next returned nil segment Dec 31 02:41:12.815663367 (611h12m12.686746402s) [upload=535628]: Waiting for error group managing segments... Dec 31 02:41:13.093790374 (611h12m12.964873409s) [noise=true , address=145.79.227.13:4803 , piece_id=KE5H4FKU4PZROEEFQ5KABCDEOX2NUXLUD4IZBMCVSSVXSPKKSYRA, seg_pos={0 0}, upload=535628]: Done uploading piece. err: <nil> Dec 31 02:41:13.099111968 (611h12m12.970195003s) [noise=true , address=145.79.227.4:4186 , piece_id=OFMISBUSLMHHEAELL2DDDTKND3UDZY2J7F7ASCTYTXYZOABZ4ZHQ, seg_pos={0 0}, upload=535628]: Done uploading piece. err: <nil> Dec 31 02:41:13.100327292 (611h12m12.971410327s) [noise=true , address=145.79.227.25:4618 , piece_id=MZWHT24LGJXQVAE7X7K6BPVZMP7X6QETTMICMNCSYI4QBMTSUAMA, seg_pos={0 0}, upload=535628]: Done uploading piece. err: <nil> Dec 31 02:41:13.102226227 (611h12m12.973309262s) [noise=true , address=145.79.227.32:4835 , piece_id=AVAICWG5CZ2FJEM3B72BGCZHTB234Q4TDRY635TDK2GCYGNNCMNQ, seg_pos={0 0}, upload=535628]: Done uploading piece. err: <nil> Dec 31 02:41:13.126767006 (611h12m12.997850041s) [noise=true , address=145.79.227.51:4410 , piece_id=EOFZJEXLH6XYMUQCIWTU7KFZETRL3RI4FUN2OX7RQXKQLOD6JOOQ, seg_pos={0 0}, upload=535628]: Done uploading piece. err: <nil> Dec 31 02:41:13.142304611 (611h12m13.013387646s) [noise=true , address=145.79.227.50:4374 , piece_id=HRTWPEWMLYB24UEUFTHZHMRWAJJN6D3VKZ3YZ6B26LN54DQXVSAA, seg_pos={0 0}, upload=535628]: Done uploading piece. err: <nil> Dec 31 02:41:13.181506727 (611h12m13.052589762s) [noise=true , address=145.79.227.39:4013 , piece_id=4EMEEV2RWPB5EE2Q4R4COOOLPC73M4ZE57BEU6G54ULD4PVHZAFQ, seg_pos={0 0}, upload=535628]: Done uploading piece. err: <nil> Dec 31 02:41:13.182708215 (611h12m13.05379125s) [noise=true , address=145.79.227.6:4258 , piece_id=BG43YG3ZELUSXPNFUZGOQQYOLXZGLURSHDDGIFVSG7DBE3JHCGHA, seg_pos={0 0}, upload=535628]: Done uploading piece. err: <nil> Dec 31 02:41:13.185142701 (611h12m13.056225736s) [noise=true , address=145.79.227.10:4403 , piece_id=D3XBXX6R5MHDLBWVSWZBQJJDYBK4VKXOMPAU5QVVG6FIVHKWKVNA, seg_pos={0 0}, upload=535628]: Done uploading piece. err: <nil> Dec 31 02:41:13.297136831 (611h12m13.168219866s) [noise=true , address=77.90.31.10:4518 , piece_id=FTGU3JK3T27PZJ3MNQ735ICXBCFBSQRKVWSJCB4L2PY5QRMAJDIA, seg_pos={0 0}, upload=535628]: Done uploading piece. err: <nil> Dec 31 02:41:13.337206979 (611h12m13.208290014s) [noise=true , address=77.90.31.11:4556 , piece_id=C5XVJ3QR25OBV3SQGCXATJHTMRQU76ZSSL3J4MCOYSMLMCZBJ3DQ, seg_pos={0 0}, upload=535628]: Done uploading piece. err: <nil> Dec 31 02:41:13.344521868 (611h12m13.215604903s) [noise=true , address=77.90.31.15:4697 , piece_id=NMLIRYJXAH47RJP5ZSN5PCCPLKNZ2K2ANIMGBU42XRGP7NMHOOAQ, seg_pos={0 0}, upload=535628]: Done uploading piece. err: <nil> Dec 31 02:41:13.512080229 (611h12m13.383163264s) [noise=true , address=45.91.116.11:4118 , piece_id=KLMGJP6XKFN23LRSEPCX52UBXRFJPHBQ3RWB66IIJXSKG2LIN7KA, seg_pos={0 0}, upload=535628]: Done uploading piece. err: <nil> Dec 31 02:41:13.512104606 (611h12m13.383187641s) [seg_pos={0 0}, upload=535628]: Segment reached optimal threshold of 13 pieces. Dec 31 02:41:13.512325146 (611h12m13.383408181s) [noise=true , address=77.90.31.14:4664 , piece_id=QYK7QKZ3KAIVPXVFPM45DJU4OB3KGDCPEOSZYIT44BD4P3EZCK3A, seg_pos={0 0}, upload=535628]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: write tcp 188.166.225.170:50786->77.90.31.14:4664: use of closed network connection; write tcp 188.166.225.170:50786->77.90.31.14:4664: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50786->77.90.31.14:4664: use of closed network connection Dec 31 02:41:13.512331317 (611h12m13.383414362s) [noise=true , address=77.90.31.14:4664 , piece_id=QYK7QKZ3KAIVPXVFPM45DJU4OB3KGDCPEOSZYIT44BD4P3EZCK3A, seg_pos={0 0}, upload=535628]: Long tail cancellation due to optimal threshold Dec 31 02:41:13.512609918 (611h12m13.383692953s) [noise=true , address=45.91.116.14:4190 , piece_id=UFR6AIVXP5ZPPQLVEZXKRSFB6CAZ7U7IL4CWCCKVRA5P3MC5MTLA, seg_pos={0 0}, upload=535628]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: write tcp 188.166.225.170:42822->45.91.116.14:4190: use of closed network connection; write tcp 188.166.225.170:42822->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42822->45.91.116.14:4190: use of closed network connection Dec 31 02:41:13.512626390 (611h12m13.383709435s) [noise=true , address=45.91.116.14:4190 , piece_id=UFR6AIVXP5ZPPQLVEZXKRSFB6CAZ7U7IL4CWCCKVRA5P3MC5MTLA, seg_pos={0 0}, upload=535628]: Long tail cancellation due to optimal threshold Dec 31 02:41:13.512787786 (611h12m13.383870821s) [noise=true , address=45.91.116.9:4263 , piece_id=QVIP6EJNGTD5FIMBDS2NCOK6R27OH5VAHK36T75HBI5F7KKH7YUQ, seg_pos={0 0}, upload=535628]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 31 02:41:13.512793036 (611h12m13.383876071s) [noise=true , address=45.91.116.9:4263 , piece_id=QVIP6EJNGTD5FIMBDS2NCOK6R27OH5VAHK36T75HBI5F7KKH7YUQ, seg_pos={0 0}, upload=535628]: Long tail cancellation due to optimal threshold Dec 31 02:41:13.512814998 (611h12m13.383898034s) [seg_pos={0 0}, upload=535628]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 02:41:13.512893738 (611h12m13.383976773s) [upload=535628]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 02:41:13.512898317 (611h12m13.383981352s) [upload=535628]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 02:41:13.802131863 (611h12m13.673214918s) [upload=535628]: Done uploading segments... Dec 31 02:41:13.802155628 (611h12m13.673238663s) [upload=535628]: Upload finished. err: <nil> Duration: 30.002217868s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 37.960257488s trace: https://jaeger.dev.storj.io/trace/365693fea00df28 FAIL (2025-12-31 02:41:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [noise=true , address=145.79.227.49:4343 , piece_id=6L3HPTS7D2NRA4QEOH4RFM3D3B53CTCONRI6G6YVU4OOTUWTG7XA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.942060847 (610h25m24.813143892s) [noise=true , address=145.79.227.34:4937 , piece_id=5W7LBM3YJJTVUFYTELR57DVXB663OHO5N3Z2SPQPZ4G43IK47O3Q, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.942684104 (610h25m24.813767139s) [noise=true , address=64.49.13.139:4478 , piece_id=QN6RT2KWEA6APQVA66ZMKG4LH537UGYNOXOCVVA2YC2MKDOLT3YA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.943064007 (610h25m24.814147042s) [noise=true , address=145.79.227.40:4084 , piece_id=RUKTKDII6BRUWSYVS65CWUBT3QKG3ASUK3KA3O5VGE53XSG6IMOQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.943459560 (610h25m24.814542605s) [noise=true , address=145.79.227.14:4847 , piece_id=JX5NXTMVM3NSWIEVR4RX46ABSNPKTOFDLFBPKEXFQ3G3ANKV5OLA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.943865303 (610h25m24.814948348s) [noise=true , address=45.91.116.6:4981 , piece_id=CT474ZA2LVAZEBJHBDD4GB6NTHVHZFJLNE5OZV3DFS2NZKVW3EQQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.944259694 (610h25m24.815342749s) [noise=true , address=145.79.227.51:4443 , piece_id=73UUMRDY7BSZNCGBXCZYYRBVE5KLPMNPRNXOZEGWKQGLHJS32J2Q, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.946575182 (610h25m24.817658227s) [noise=true , address=77.90.31.10:4523 , piece_id=P6LIONXJI6AFZ2JTKCQ2W6RZCOZCK6MZ53EXAJKXYQXSX2PX4QEA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.948132909 (610h25m24.819215944s) [noise=true , address=145.79.227.20:4452 , piece_id=ZSDN2JZ3FQUVJYN5OURDXNKEDLDNULJOHKKRZVIPXZIWIYAHCRXQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.949605774 (610h25m24.820688819s) [noise=true , address=77.90.31.14:4680 , piece_id=S2E4WBA7MJVAZLT5JG3IMHK53RQC77CCHV2PDF6L2R2JEA2U3K4Q, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.951459002 (610h25m24.822542048s) [noise=true , address=193.149.164.143:4704 , piece_id=BHXHFBKTLTDMGXWY5QRTP2THZQE5OXFZ2LCMTO2KRYRWAHXWW3NQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.952986742 (610h25m24.824069777s) [noise=true , address=77.90.31.19:4848 , piece_id=VZYG32S6ODIPDKQYZRXECBHXOERZDDWD2OF35Z46W4XMUZ3OJLAA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.956916654 (610h25m24.827999689s) [noise=true , address=145.79.227.29:4971 , piece_id=MEPXHHOQ6SUZC3GIPCQEMOKKF5OJTKZEI3UYZBAYXAQZGSVEKSBA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.960665062 (610h25m24.831748097s) [noise=true , address=64.49.13.170:4720 , piece_id=HIATRSAPE4YLSODFZDPP43ZUMLDVM7GJEVMWE4ZGUQTP25LTHGTA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.962366001 (610h25m24.833449036s) [noise=true , address=145.79.227.2:4117 , piece_id=4SGFGB3IJ7LTXDMMWM2PGGOV64MC46IZ7OZQWH55J4NBQY7CHMZQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.963708216 (610h25m24.834791261s) [noise=true , address=145.79.227.35:4505 , piece_id=V7RS7A4QLZTH4G32ZUZNSF2Z5APVWZGJO77NEGP3SBWQT3WCMVLQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:26.352236138 (610h25m26.223319173s) [upload=534952]: Next returned nil segment Dec 31 01:54:26.352253821 (610h25m26.223336857s) [upload=534952]: Waiting for error group managing segments... Dec 31 01:54:26.812758374 (610h25m26.683841409s) [noise=true , address=145.79.227.51:4443 , piece_id=73UUMRDY7BSZNCGBXCZYYRBVE5KLPMNPRNXOZEGWKQGLHJS32J2Q, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.824371056 (610h25m26.695454101s) [noise=true , address=145.79.227.14:4847 , piece_id=JX5NXTMVM3NSWIEVR4RX46ABSNPKTOFDLFBPKEXFQ3G3ANKV5OLA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.838414046 (610h25m26.709497081s) [noise=true , address=64.49.13.138:4490 , piece_id=2XTUZE6PXRE3OARNJVT36FXXWUMYIBVNE6M2VVFAVBGKEE2H5YZQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.875499841 (610h25m26.746582876s) [noise=true , address=64.49.13.139:4478 , piece_id=QN6RT2KWEA6APQVA66ZMKG4LH537UGYNOXOCVVA2YC2MKDOLT3YA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.891607221 (610h25m26.762690256s) [noise=true , address=145.79.227.20:4452 , piece_id=ZSDN2JZ3FQUVJYN5OURDXNKEDLDNULJOHKKRZVIPXZIWIYAHCRXQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.988561223 (610h25m26.859644268s) [noise=true , address=64.49.13.170:4720 , piece_id=HIATRSAPE4YLSODFZDPP43ZUMLDVM7GJEVMWE4ZGUQTP25LTHGTA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.039035911 (610h25m26.910118946s) [noise=true , address=77.90.31.14:4680 , piece_id=S2E4WBA7MJVAZLT5JG3IMHK53RQC77CCHV2PDF6L2R2JEA2U3K4Q, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.098023090 (610h25m26.969106126s) [noise=true , address=45.91.116.6:4981 , piece_id=CT474ZA2LVAZEBJHBDD4GB6NTHVHZFJLNE5OZV3DFS2NZKVW3EQQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.653790754 (610h25m27.524873789s) [noise=true , address=77.90.31.21:4886 , piece_id=K34WPIBGGGEPHIKJ253XGLDCIQ3WLFN4AZ6EYNWVHHT2EIHS6YRQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.684192972 (610h25m27.555276017s) [noise=true , address=193.149.164.143:4704 , piece_id=BHXHFBKTLTDMGXWY5QRTP2THZQE5OXFZ2LCMTO2KRYRWAHXWW3NQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.717238425 (610h25m27.58832146s) [noise=true , address=145.79.227.40:4084 , piece_id=RUKTKDII6BRUWSYVS65CWUBT3QKG3ASUK3KA3O5VGE53XSG6IMOQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.718349742 (610h25m27.589432777s) [noise=true , address=77.90.31.10:4523 , piece_id=P6LIONXJI6AFZ2JTKCQ2W6RZCOZCK6MZ53EXAJKXYQXSX2PX4QEA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.803639562 (610h25m27.674722597s) [noise=true , address=145.79.227.29:4971 , piece_id=MEPXHHOQ6SUZC3GIPCQEMOKKF5OJTKZEI3UYZBAYXAQZGSVEKSBA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.812791279 (610h25m27.683874314s) [noise=true , address=145.79.227.49:4343 , piece_id=6L3HPTS7D2NRA4QEOH4RFM3D3B53CTCONRI6G6YVU4OOTUWTG7XA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.812812348 (610h25m27.683895393s) [seg_pos={0 0}, upload=534952]: Segment reached optimal threshold of 14 pieces. Dec 31 01:54:27.813018000 (610h25m27.684101035s) [noise=true , address=145.79.227.35:4505 , piece_id=V7RS7A4QLZTH4G32ZUZNSF2Z5APVWZGJO77NEGP3SBWQT3WCMVLQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i6itGoBhLsLiqC8et8Y5L62fLjcWxy18u39NAr4R3nXNMWcsf): protocol: expected piece hash; context canceled; EOF Dec 31 01:54:27.813024772 (610h25m27.684107808s) [noise=true , address=145.79.227.35:4505 , piece_id=V7RS7A4QLZTH4G32ZUZNSF2Z5APVWZGJO77NEGP3SBWQT3WCMVLQ, seg_pos={0 0}, upload=534952]: Long tail cancellation due to optimal threshold Dec 31 01:54:27.813175931 (610h25m27.684258976s) [noise=true , address=145.79.227.34:4937 , piece_id=5W7LBM3YJJTVUFYTELR57DVXB663OHO5N3Z2SPQPZ4G43IK47O3Q, seg_pos={0 0}, upload=534952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1LKrFThdW3jJpG6dRZLUaxLXTa5Wy27yQ8PGUEqjvBWfFoRB94): protocol: expected piece hash; context canceled; EOF Dec 31 01:54:27.813181702 (610h25m27.684264737s) [noise=true , address=145.79.227.34:4937 , piece_id=5W7LBM3YJJTVUFYTELR57DVXB663OHO5N3Z2SPQPZ4G43IK47O3Q, seg_pos={0 0}, upload=534952]: Long tail cancellation due to optimal threshold Dec 31 01:54:27.813407862 (610h25m27.684490907s) [noise=true , address=145.79.227.2:4117 , piece_id=4SGFGB3IJ7LTXDMMWM2PGGOV64MC46IZ7OZQWH55J4NBQY7CHMZQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122Htg72f5eUUutf1p3fYQpy1Wkf6AgdaQV83HHYye2n8zm3the): protocol: expected piece hash; context canceled; EOF Dec 31 01:54:27.813414264 (610h25m27.684497299s) [noise=true , address=145.79.227.2:4117 , piece_id=4SGFGB3IJ7LTXDMMWM2PGGOV64MC46IZ7OZQWH55J4NBQY7CHMZQ, seg_pos={0 0}, upload=534952]: Long tail cancellation due to optimal threshold Dec 31 01:54:27.813424614 (610h25m27.684507649s) [noise=true , address=77.90.31.19:4848 , piece_id=VZYG32S6ODIPDKQYZRXECBHXOERZDDWD2OF35Z46W4XMUZ3OJLAA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD): protocol: write tcp 188.166.225.170:42842->77.90.31.19:4848: use of closed network connection; write tcp 188.166.225.170:42842->77.90.31.19:4848: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42842->77.90.31.19:4848: use of closed network connection Dec 31 01:54:27.813432750 (610h25m27.684515785s) [noise=true , address=77.90.31.19:4848 , piece_id=VZYG32S6ODIPDKQYZRXECBHXOERZDDWD2OF35Z46W4XMUZ3OJLAA, seg_pos={0 0}, upload=534952]: Long tail cancellation due to optimal threshold Dec 31 01:54:27.813474088 (610h25m27.684557123s) [seg_pos={0 0}, upload=534952]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 01:54:27.813535325 (610h25m27.68461836s) [upload=534952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 01:54:27.813539613 (610h25m27.684622648s) [upload=534952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 01:54:28.099921034 (610h25m27.971004069s) [upload=534952]: Done uploading segments... Dec 31 01:54:28.099940451 (610h25m27.971023486s) [upload=534952]: Upload finished. err: <nil> Duration: 37.960275152s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..................X...............................X....X....] duration: 4.222032382s trace: https://jaeger.dev.storj.io/trace/7bc7478624e56c1a FAIL (2025-12-31 02:41:23 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.222062119s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 34.575130557s trace: https://jaeger.dev.storj.io/trace/365693fea00df28 FAIL (2025-12-31 02:40:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [noise=true , address=145.79.227.49:4343 , piece_id=6L3HPTS7D2NRA4QEOH4RFM3D3B53CTCONRI6G6YVU4OOTUWTG7XA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.942060847 (610h25m24.813143892s) [noise=true , address=145.79.227.34:4937 , piece_id=5W7LBM3YJJTVUFYTELR57DVXB663OHO5N3Z2SPQPZ4G43IK47O3Q, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.942684104 (610h25m24.813767139s) [noise=true , address=64.49.13.139:4478 , piece_id=QN6RT2KWEA6APQVA66ZMKG4LH537UGYNOXOCVVA2YC2MKDOLT3YA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.943064007 (610h25m24.814147042s) [noise=true , address=145.79.227.40:4084 , piece_id=RUKTKDII6BRUWSYVS65CWUBT3QKG3ASUK3KA3O5VGE53XSG6IMOQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.943459560 (610h25m24.814542605s) [noise=true , address=145.79.227.14:4847 , piece_id=JX5NXTMVM3NSWIEVR4RX46ABSNPKTOFDLFBPKEXFQ3G3ANKV5OLA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.943865303 (610h25m24.814948348s) [noise=true , address=45.91.116.6:4981 , piece_id=CT474ZA2LVAZEBJHBDD4GB6NTHVHZFJLNE5OZV3DFS2NZKVW3EQQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.944259694 (610h25m24.815342749s) [noise=true , address=145.79.227.51:4443 , piece_id=73UUMRDY7BSZNCGBXCZYYRBVE5KLPMNPRNXOZEGWKQGLHJS32J2Q, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.946575182 (610h25m24.817658227s) [noise=true , address=77.90.31.10:4523 , piece_id=P6LIONXJI6AFZ2JTKCQ2W6RZCOZCK6MZ53EXAJKXYQXSX2PX4QEA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.948132909 (610h25m24.819215944s) [noise=true , address=145.79.227.20:4452 , piece_id=ZSDN2JZ3FQUVJYN5OURDXNKEDLDNULJOHKKRZVIPXZIWIYAHCRXQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.949605774 (610h25m24.820688819s) [noise=true , address=77.90.31.14:4680 , piece_id=S2E4WBA7MJVAZLT5JG3IMHK53RQC77CCHV2PDF6L2R2JEA2U3K4Q, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.951459002 (610h25m24.822542048s) [noise=true , address=193.149.164.143:4704 , piece_id=BHXHFBKTLTDMGXWY5QRTP2THZQE5OXFZ2LCMTO2KRYRWAHXWW3NQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.952986742 (610h25m24.824069777s) [noise=true , address=77.90.31.19:4848 , piece_id=VZYG32S6ODIPDKQYZRXECBHXOERZDDWD2OF35Z46W4XMUZ3OJLAA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.956916654 (610h25m24.827999689s) [noise=true , address=145.79.227.29:4971 , piece_id=MEPXHHOQ6SUZC3GIPCQEMOKKF5OJTKZEI3UYZBAYXAQZGSVEKSBA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.960665062 (610h25m24.831748097s) [noise=true , address=64.49.13.170:4720 , piece_id=HIATRSAPE4YLSODFZDPP43ZUMLDVM7GJEVMWE4ZGUQTP25LTHGTA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.962366001 (610h25m24.833449036s) [noise=true , address=145.79.227.2:4117 , piece_id=4SGFGB3IJ7LTXDMMWM2PGGOV64MC46IZ7OZQWH55J4NBQY7CHMZQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.963708216 (610h25m24.834791261s) [noise=true , address=145.79.227.35:4505 , piece_id=V7RS7A4QLZTH4G32ZUZNSF2Z5APVWZGJO77NEGP3SBWQT3WCMVLQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:26.352236138 (610h25m26.223319173s) [upload=534952]: Next returned nil segment Dec 31 01:54:26.352253821 (610h25m26.223336857s) [upload=534952]: Waiting for error group managing segments... Dec 31 01:54:26.812758374 (610h25m26.683841409s) [noise=true , address=145.79.227.51:4443 , piece_id=73UUMRDY7BSZNCGBXCZYYRBVE5KLPMNPRNXOZEGWKQGLHJS32J2Q, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.824371056 (610h25m26.695454101s) [noise=true , address=145.79.227.14:4847 , piece_id=JX5NXTMVM3NSWIEVR4RX46ABSNPKTOFDLFBPKEXFQ3G3ANKV5OLA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.838414046 (610h25m26.709497081s) [noise=true , address=64.49.13.138:4490 , piece_id=2XTUZE6PXRE3OARNJVT36FXXWUMYIBVNE6M2VVFAVBGKEE2H5YZQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.875499841 (610h25m26.746582876s) [noise=true , address=64.49.13.139:4478 , piece_id=QN6RT2KWEA6APQVA66ZMKG4LH537UGYNOXOCVVA2YC2MKDOLT3YA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.891607221 (610h25m26.762690256s) [noise=true , address=145.79.227.20:4452 , piece_id=ZSDN2JZ3FQUVJYN5OURDXNKEDLDNULJOHKKRZVIPXZIWIYAHCRXQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.988561223 (610h25m26.859644268s) [noise=true , address=64.49.13.170:4720 , piece_id=HIATRSAPE4YLSODFZDPP43ZUMLDVM7GJEVMWE4ZGUQTP25LTHGTA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.039035911 (610h25m26.910118946s) [noise=true , address=77.90.31.14:4680 , piece_id=S2E4WBA7MJVAZLT5JG3IMHK53RQC77CCHV2PDF6L2R2JEA2U3K4Q, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.098023090 (610h25m26.969106126s) [noise=true , address=45.91.116.6:4981 , piece_id=CT474ZA2LVAZEBJHBDD4GB6NTHVHZFJLNE5OZV3DFS2NZKVW3EQQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.653790754 (610h25m27.524873789s) [noise=true , address=77.90.31.21:4886 , piece_id=K34WPIBGGGEPHIKJ253XGLDCIQ3WLFN4AZ6EYNWVHHT2EIHS6YRQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.684192972 (610h25m27.555276017s) [noise=true , address=193.149.164.143:4704 , piece_id=BHXHFBKTLTDMGXWY5QRTP2THZQE5OXFZ2LCMTO2KRYRWAHXWW3NQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.717238425 (610h25m27.58832146s) [noise=true , address=145.79.227.40:4084 , piece_id=RUKTKDII6BRUWSYVS65CWUBT3QKG3ASUK3KA3O5VGE53XSG6IMOQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.718349742 (610h25m27.589432777s) [noise=true , address=77.90.31.10:4523 , piece_id=P6LIONXJI6AFZ2JTKCQ2W6RZCOZCK6MZ53EXAJKXYQXSX2PX4QEA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.803639562 (610h25m27.674722597s) [noise=true , address=145.79.227.29:4971 , piece_id=MEPXHHOQ6SUZC3GIPCQEMOKKF5OJTKZEI3UYZBAYXAQZGSVEKSBA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.812791279 (610h25m27.683874314s) [noise=true , address=145.79.227.49:4343 , piece_id=6L3HPTS7D2NRA4QEOH4RFM3D3B53CTCONRI6G6YVU4OOTUWTG7XA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.812812348 (610h25m27.683895393s) [seg_pos={0 0}, upload=534952]: Segment reached optimal threshold of 14 pieces. Dec 31 01:54:27.813018000 (610h25m27.684101035s) [noise=true , address=145.79.227.35:4505 , piece_id=V7RS7A4QLZTH4G32ZUZNSF2Z5APVWZGJO77NEGP3SBWQT3WCMVLQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i6itGoBhLsLiqC8et8Y5L62fLjcWxy18u39NAr4R3nXNMWcsf): protocol: expected piece hash; context canceled; EOF Dec 31 01:54:27.813024772 (610h25m27.684107808s) [noise=true , address=145.79.227.35:4505 , piece_id=V7RS7A4QLZTH4G32ZUZNSF2Z5APVWZGJO77NEGP3SBWQT3WCMVLQ, seg_pos={0 0}, upload=534952]: Long tail cancellation due to optimal threshold Dec 31 01:54:27.813175931 (610h25m27.684258976s) [noise=true , address=145.79.227.34:4937 , piece_id=5W7LBM3YJJTVUFYTELR57DVXB663OHO5N3Z2SPQPZ4G43IK47O3Q, seg_pos={0 0}, upload=534952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1LKrFThdW3jJpG6dRZLUaxLXTa5Wy27yQ8PGUEqjvBWfFoRB94): protocol: expected piece hash; context canceled; EOF Dec 31 01:54:27.813181702 (610h25m27.684264737s) [noise=true , address=145.79.227.34:4937 , piece_id=5W7LBM3YJJTVUFYTELR57DVXB663OHO5N3Z2SPQPZ4G43IK47O3Q, seg_pos={0 0}, upload=534952]: Long tail cancellation due to optimal threshold Dec 31 01:54:27.813407862 (610h25m27.684490907s) [noise=true , address=145.79.227.2:4117 , piece_id=4SGFGB3IJ7LTXDMMWM2PGGOV64MC46IZ7OZQWH55J4NBQY7CHMZQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122Htg72f5eUUutf1p3fYQpy1Wkf6AgdaQV83HHYye2n8zm3the): protocol: expected piece hash; context canceled; EOF Dec 31 01:54:27.813414264 (610h25m27.684497299s) [noise=true , address=145.79.227.2:4117 , piece_id=4SGFGB3IJ7LTXDMMWM2PGGOV64MC46IZ7OZQWH55J4NBQY7CHMZQ, seg_pos={0 0}, upload=534952]: Long tail cancellation due to optimal threshold Dec 31 01:54:27.813424614 (610h25m27.684507649s) [noise=true , address=77.90.31.19:4848 , piece_id=VZYG32S6ODIPDKQYZRXECBHXOERZDDWD2OF35Z46W4XMUZ3OJLAA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD): protocol: write tcp 188.166.225.170:42842->77.90.31.19:4848: use of closed network connection; write tcp 188.166.225.170:42842->77.90.31.19:4848: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42842->77.90.31.19:4848: use of closed network connection Dec 31 01:54:27.813432750 (610h25m27.684515785s) [noise=true , address=77.90.31.19:4848 , piece_id=VZYG32S6ODIPDKQYZRXECBHXOERZDDWD2OF35Z46W4XMUZ3OJLAA, seg_pos={0 0}, upload=534952]: Long tail cancellation due to optimal threshold Dec 31 01:54:27.813474088 (610h25m27.684557123s) [seg_pos={0 0}, upload=534952]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 01:54:27.813535325 (610h25m27.68461836s) [upload=534952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 01:54:27.813539613 (610h25m27.684622648s) [upload=534952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 01:54:28.099921034 (610h25m27.971004069s) [upload=534952]: Done uploading segments... Dec 31 01:54:28.099940451 (610h25m27.971023486s) [upload=534952]: Upload finished. err: <nil> Duration: 34.575170463s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...................X...............................X....X...] duration: 3.354485199s trace: https://jaeger.dev.storj.io/trace/7bc7478624e56c1a FAIL (2025-12-31 02:40:31 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.354509604s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [..X..X.......X.X..X....XX.XX........X.X......X..X......XXX..] duration: 24.974162581s trace: https://jaeger.dev.storj.io/trace/6707b59ca1e5b458 FAIL (2025-12-31 02:40:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: 3Q7UQ3435OJP66RHTZTQPGPIJSUW5Y52F77WE3UT6YPEDIRKLOUQ) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: 5RYGOIN4YBYWIYUIQCUCLGJXIFWRIOTRLV6BOTEG7BE4VTFN4MVQ) context deadline exceeded ======LOGS====== (611h9m18.344144905s) [upload=535584]: Uploading segments... Dec 31 02:38:18.473115882 (611h9m18.344198917s) [upload=535584]: Got next segment. Inline: false Dec 31 02:38:18.473142283 (611h9m18.344225318s) [upload=535584]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 02:38:18.473146481 (611h9m18.344229516s) [upload=535584]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 02:38:18.716039759 (611h9m18.587122804s) [seg_pos={0 0}, upload=535584]: Begin upload segment... Dec 31 02:38:18.716109301 (611h9m18.587192336s) [seg_pos={0 0}, upload=535584]: Done begin upload segment. Dec 31 02:38:18.716137034 (611h9m18.587220069s) [noise=true , address=45.91.116.14:4189 , piece_id=ACDVMCUOLEM3E7GCVEJBPCS73OUJDMKEODTP2KPDFF7C2UX672QA, seg_pos={0 0}, upload=535584]: Uploading piece... Dec 31 02:38:18.716164727 (611h9m18.587247762s) [noise=true , address=45.91.116.6:4976 , piece_id=MEAAYINZGA6YTPNZCDUPZ3AODLPOIWNJIKQUB3Q56FIO4IN6R7FQ, seg_pos={0 0}, upload=535584]: Uploading piece... Dec 31 02:38:18.716631967 (611h9m18.587715002s) [noise=true , address=45.91.116.13:4153 , piece_id=3Q7UQ3435OJP66RHTZTQPGPIJSUW5Y52F77WE3UT6YPEDIRKLOUQ, seg_pos={0 0}, upload=535584]: Uploading piece... Dec 31 02:38:18.717028412 (611h9m18.588111457s) [noise=true , address=145.79.227.31:4798 , piece_id=5V4LMCUVVS7ZAEEH3W7JJCD3GU5WQSVYR3JJRUFPD3IHJJHPDP6A, seg_pos={0 0}, upload=535584]: Uploading piece... Dec 31 02:38:18.717627543 (611h9m18.588710588s) [noise=true , address=145.79.227.15:4873 , piece_id=OGUUFEOBIY7JUXGPHPAAEW7KP27NLO2X5QQDFMD4EGG5OJ4QTYMA, seg_pos={0 0}, upload=535584]: Uploading piece... Dec 31 02:38:18.718193149 (611h9m18.589276195s) [noise=true , address=45.91.116.8:4046 , piece_id=5RYGOIN4YBYWIYUIQCUCLGJXIFWRIOTRLV6BOTEG7BE4VTFN4MVQ, seg_pos={0 0}, upload=535584]: Uploading piece... Dec 31 02:38:18.718873404 (611h9m18.589956439s) [noise=true , address=77.90.31.16:4735 , piece_id=O7WO7ZQJJJUG7M4CYBZVDXWPILO6JPIXQOZUAXB5BZMLQLL77XQQ, seg_pos={0 0}, upload=535584]: Uploading piece... Dec 31 02:38:18.720029065 (611h9m18.5911121s) [noise=true , address=145.79.227.50:4375 , piece_id=POT3FF3VPATA3OAKHXHY3SUI2DI2UHT6GH7VXRRQBZIOXSMSVAEA, seg_pos={0 0}, upload=535584]: Uploading piece... Dec 31 02:38:18.721622589 (611h9m18.592705624s) [noise=true , address=45.91.116.7:4009 , piece_id=SVSZ5B3RAQYH4RGOV5UNTIHPEYCJTBB3EEYUVU2RW5UN62AAQJNA, seg_pos={0 0}, upload=535584]: Uploading piece... Dec 31 02:38:18.721912391 (611h9m18.592995436s) [noise=true , address=77.90.31.14:4664 , piece_id=P72CNBKIRCALEDSJUIYLZEGTQWWI454LSGO23GWIMBIMYMJZI5QQ, seg_pos={0 0}, upload=535584]: Uploading piece... Dec 31 02:38:18.725644548 (611h9m18.596727593s) [noise=true , address=145.79.227.41:4085 , piece_id=TQA2Z6FY6SSZME6TW3X4POZTJIYSWUMIMOAEXOZ7RM7JISRXABKQ, seg_pos={0 0}, upload=535584]: Uploading piece... Dec 31 02:38:18.726151974 (611h9m18.597235009s) [noise=true , address=77.90.31.11:4555 , piece_id=MEFFVXPYXERLBAATG5XSCEHUKHOA3DS6PIPHAEBHJ23Y3TGAFRVA, seg_pos={0 0}, upload=535584]: Uploading piece... Dec 31 02:38:18.728917180 (611h9m18.600000225s) [noise=true , address=145.79.227.51:4409 , piece_id=H5WXMPEJUQNNRKWNVJEIIUQ45BG5DLTQHGBGT2SRYAITH6JHUT3A, seg_pos={0 0}, upload=535584]: Uploading piece... Dec 31 02:38:18.728959831 (611h9m18.600042916s) [noise=true , address=145.79.227.9:4365 , piece_id=TTP6PH74NWNZ7PGOPU5DMJ6O47M2AGZX32KSGG72X5LJFVBJVV6A, seg_pos={0 0}, upload=535584]: Uploading piece... Dec 31 02:38:18.730985568 (611h9m18.602068603s) [noise=true , address=77.90.31.21:4880 , piece_id=J64Q3NU2YQFJOI3SZMV7KOFDOISWX32K4NTPAMLOZLM2BIG34TLA, seg_pos={0 0}, upload=535584]: Uploading piece... Dec 31 02:38:18.731444271 (611h9m18.602527306s) [noise=true , address=145.79.227.46:4603 , piece_id=PQK6LWX3DD3DQTZFOO36YCWCFPINDQITEZFAPMPEPUFNTP4ZPKFQ, seg_pos={0 0}, upload=535584]: Uploading piece... Dec 31 02:38:20.431446432 (611h9m20.302529508s) [upload=535584]: Next returned nil segment Dec 31 02:38:20.431463736 (611h9m20.302546801s) [upload=535584]: Waiting for error group managing segments... Dec 31 02:38:20.704147128 (611h9m20.575230183s) [noise=true , address=145.79.227.31:4798 , piece_id=5V4LMCUVVS7ZAEEH3W7JJCD3GU5WQSVYR3JJRUFPD3IHJJHPDP6A, seg_pos={0 0}, upload=535584]: Done uploading piece. err: <nil> Dec 31 02:38:20.718728553 (611h9m20.589811588s) [noise=true , address=145.79.227.51:4409 , piece_id=H5WXMPEJUQNNRKWNVJEIIUQ45BG5DLTQHGBGT2SRYAITH6JHUT3A, seg_pos={0 0}, upload=535584]: Done uploading piece. err: <nil> Dec 31 02:38:20.736431231 (611h9m20.607514266s) [noise=true , address=145.79.227.50:4375 , piece_id=POT3FF3VPATA3OAKHXHY3SUI2DI2UHT6GH7VXRRQBZIOXSMSVAEA, seg_pos={0 0}, upload=535584]: Done uploading piece. err: <nil> Dec 31 02:38:20.740008923 (611h9m20.611091968s) [noise=true , address=145.79.227.9:4365 , piece_id=TTP6PH74NWNZ7PGOPU5DMJ6O47M2AGZX32KSGG72X5LJFVBJVV6A, seg_pos={0 0}, upload=535584]: Done uploading piece. err: <nil> Dec 31 02:38:20.741909282 (611h9m20.612992337s) [noise=true , address=145.79.227.15:4873 , piece_id=OGUUFEOBIY7JUXGPHPAAEW7KP27NLO2X5QQDFMD4EGG5OJ4QTYMA, seg_pos={0 0}, upload=535584]: Done uploading piece. err: <nil> Dec 31 02:38:20.744063354 (611h9m20.615146399s) [noise=true , address=145.79.227.41:4085 , piece_id=TQA2Z6FY6SSZME6TW3X4POZTJIYSWUMIMOAEXOZ7RM7JISRXABKQ, seg_pos={0 0}, upload=535584]: Done uploading piece. err: <nil> Dec 31 02:38:20.817838119 (611h9m20.688921154s) [noise=true , address=145.79.227.46:4603 , piece_id=PQK6LWX3DD3DQTZFOO36YCWCFPINDQITEZFAPMPEPUFNTP4ZPKFQ, seg_pos={0 0}, upload=535584]: Done uploading piece. err: <nil> Dec 31 02:38:20.927769442 (611h9m20.798852477s) [noise=true , address=77.90.31.16:4735 , piece_id=O7WO7ZQJJJUG7M4CYBZVDXWPILO6JPIXQOZUAXB5BZMLQLL77XQQ, seg_pos={0 0}, upload=535584]: Done uploading piece. err: <nil> Dec 31 02:38:20.931851806 (611h9m20.802934851s) [noise=true , address=77.90.31.21:4880 , piece_id=J64Q3NU2YQFJOI3SZMV7KOFDOISWX32K4NTPAMLOZLM2BIG34TLA, seg_pos={0 0}, upload=535584]: Done uploading piece. err: <nil> Dec 31 02:38:20.972975961 (611h9m20.844059006s) [noise=true , address=77.90.31.11:4555 , piece_id=MEFFVXPYXERLBAATG5XSCEHUKHOA3DS6PIPHAEBHJ23Y3TGAFRVA, seg_pos={0 0}, upload=535584]: Done uploading piece. err: <nil> Dec 31 02:38:21.035055207 (611h9m20.906138242s) [noise=true , address=45.91.116.8:4046 , piece_id=5RYGOIN4YBYWIYUIQCUCLGJXIFWRIOTRLV6BOTEG7BE4VTFN4MVQ, seg_pos={0 0}, upload=535584]: Done uploading piece. err: <nil> Dec 31 02:38:21.059374474 (611h9m20.930457509s) [noise=true , address=45.91.116.13:4153 , piece_id=3Q7UQ3435OJP66RHTZTQPGPIJSUW5Y52F77WE3UT6YPEDIRKLOUQ, seg_pos={0 0}, upload=535584]: Done uploading piece. err: <nil> Dec 31 02:38:21.067861705 (611h9m20.93894474s) [noise=true , address=45.91.116.6:4976 , piece_id=MEAAYINZGA6YTPNZCDUPZ3AODLPOIWNJIKQUB3Q56FIO4IN6R7FQ, seg_pos={0 0}, upload=535584]: Done uploading piece. err: <nil> Dec 31 02:38:21.067888396 (611h9m20.938971451s) [seg_pos={0 0}, upload=535584]: Segment reached optimal threshold of 13 pieces. Dec 31 02:38:21.068160553 (611h9m20.939243588s) [noise=true , address=77.90.31.14:4664 , piece_id=P72CNBKIRCALEDSJUIYLZEGTQWWI454LSGO23GWIMBIMYMJZI5QQ, seg_pos={0 0}, upload=535584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: write tcp 188.166.225.170:49990->77.90.31.14:4664: use of closed network connection; write tcp 188.166.225.170:49990->77.90.31.14:4664: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49990->77.90.31.14:4664: use of closed network connection Dec 31 02:38:21.068209036 (611h9m20.939292081s) [noise=true , address=77.90.31.14:4664 , piece_id=P72CNBKIRCALEDSJUIYLZEGTQWWI454LSGO23GWIMBIMYMJZI5QQ, seg_pos={0 0}, upload=535584]: Long tail cancellation due to optimal threshold Dec 31 02:38:21.068222862 (611h9m20.939305917s) [noise=true , address=45.91.116.14:4189 , piece_id=ACDVMCUOLEM3E7GCVEJBPCS73OUJDMKEODTP2KPDFF7C2UX672QA, seg_pos={0 0}, upload=535584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 31 02:38:21.068232911 (611h9m20.939315946s) [noise=true , address=45.91.116.14:4189 , piece_id=ACDVMCUOLEM3E7GCVEJBPCS73OUJDMKEODTP2KPDFF7C2UX672QA, seg_pos={0 0}, upload=535584]: Long tail cancellation due to optimal threshold Dec 31 02:38:21.068372507 (611h9m20.939455562s) [noise=true , address=45.91.116.7:4009 , piece_id=SVSZ5B3RAQYH4RGOV5UNTIHPEYCJTBB3EEYUVU2RW5UN62AAQJNA, seg_pos={0 0}, upload=535584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:39450->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:39450->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39450->45.91.116.7:4009: use of closed network connection Dec 31 02:38:21.068380613 (611h9m20.939463648s) [noise=true , address=45.91.116.7:4009 , piece_id=SVSZ5B3RAQYH4RGOV5UNTIHPEYCJTBB3EEYUVU2RW5UN62AAQJNA, seg_pos={0 0}, upload=535584]: Long tail cancellation due to optimal threshold Dec 31 02:38:21.068401542 (611h9m20.939484587s) [seg_pos={0 0}, upload=535584]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 02:38:21.068492606 (611h9m20.939575651s) [upload=535584]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 02:38:21.068502314 (611h9m20.939585359s) [upload=535584]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 02:38:21.352975415 (611h9m21.22405849s) [upload=535584]: Done uploading segments... Dec 31 02:38:21.353002297 (611h9m21.224085342s) [upload=535584]: Upload finished. err: <nil> Duration: 24.97418348s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [....................X...............................X....X..] duration: 3.471756986s trace: https://jaeger.dev.storj.io/trace/7bc7478624e56c1a FAIL (2025-12-31 02:40:11 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.47178579s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 47.334205564s trace: https://jaeger.dev.storj.io/trace/365693fea00df28 FAIL (2025-12-31 02:40:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [noise=true , address=145.79.227.49:4343 , piece_id=6L3HPTS7D2NRA4QEOH4RFM3D3B53CTCONRI6G6YVU4OOTUWTG7XA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.942060847 (610h25m24.813143892s) [noise=true , address=145.79.227.34:4937 , piece_id=5W7LBM3YJJTVUFYTELR57DVXB663OHO5N3Z2SPQPZ4G43IK47O3Q, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.942684104 (610h25m24.813767139s) [noise=true , address=64.49.13.139:4478 , piece_id=QN6RT2KWEA6APQVA66ZMKG4LH537UGYNOXOCVVA2YC2MKDOLT3YA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.943064007 (610h25m24.814147042s) [noise=true , address=145.79.227.40:4084 , piece_id=RUKTKDII6BRUWSYVS65CWUBT3QKG3ASUK3KA3O5VGE53XSG6IMOQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.943459560 (610h25m24.814542605s) [noise=true , address=145.79.227.14:4847 , piece_id=JX5NXTMVM3NSWIEVR4RX46ABSNPKTOFDLFBPKEXFQ3G3ANKV5OLA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.943865303 (610h25m24.814948348s) [noise=true , address=45.91.116.6:4981 , piece_id=CT474ZA2LVAZEBJHBDD4GB6NTHVHZFJLNE5OZV3DFS2NZKVW3EQQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.944259694 (610h25m24.815342749s) [noise=true , address=145.79.227.51:4443 , piece_id=73UUMRDY7BSZNCGBXCZYYRBVE5KLPMNPRNXOZEGWKQGLHJS32J2Q, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.946575182 (610h25m24.817658227s) [noise=true , address=77.90.31.10:4523 , piece_id=P6LIONXJI6AFZ2JTKCQ2W6RZCOZCK6MZ53EXAJKXYQXSX2PX4QEA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.948132909 (610h25m24.819215944s) [noise=true , address=145.79.227.20:4452 , piece_id=ZSDN2JZ3FQUVJYN5OURDXNKEDLDNULJOHKKRZVIPXZIWIYAHCRXQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.949605774 (610h25m24.820688819s) [noise=true , address=77.90.31.14:4680 , piece_id=S2E4WBA7MJVAZLT5JG3IMHK53RQC77CCHV2PDF6L2R2JEA2U3K4Q, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.951459002 (610h25m24.822542048s) [noise=true , address=193.149.164.143:4704 , piece_id=BHXHFBKTLTDMGXWY5QRTP2THZQE5OXFZ2LCMTO2KRYRWAHXWW3NQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.952986742 (610h25m24.824069777s) [noise=true , address=77.90.31.19:4848 , piece_id=VZYG32S6ODIPDKQYZRXECBHXOERZDDWD2OF35Z46W4XMUZ3OJLAA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.956916654 (610h25m24.827999689s) [noise=true , address=145.79.227.29:4971 , piece_id=MEPXHHOQ6SUZC3GIPCQEMOKKF5OJTKZEI3UYZBAYXAQZGSVEKSBA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.960665062 (610h25m24.831748097s) [noise=true , address=64.49.13.170:4720 , piece_id=HIATRSAPE4YLSODFZDPP43ZUMLDVM7GJEVMWE4ZGUQTP25LTHGTA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.962366001 (610h25m24.833449036s) [noise=true , address=145.79.227.2:4117 , piece_id=4SGFGB3IJ7LTXDMMWM2PGGOV64MC46IZ7OZQWH55J4NBQY7CHMZQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.963708216 (610h25m24.834791261s) [noise=true , address=145.79.227.35:4505 , piece_id=V7RS7A4QLZTH4G32ZUZNSF2Z5APVWZGJO77NEGP3SBWQT3WCMVLQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:26.352236138 (610h25m26.223319173s) [upload=534952]: Next returned nil segment Dec 31 01:54:26.352253821 (610h25m26.223336857s) [upload=534952]: Waiting for error group managing segments... Dec 31 01:54:26.812758374 (610h25m26.683841409s) [noise=true , address=145.79.227.51:4443 , piece_id=73UUMRDY7BSZNCGBXCZYYRBVE5KLPMNPRNXOZEGWKQGLHJS32J2Q, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.824371056 (610h25m26.695454101s) [noise=true , address=145.79.227.14:4847 , piece_id=JX5NXTMVM3NSWIEVR4RX46ABSNPKTOFDLFBPKEXFQ3G3ANKV5OLA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.838414046 (610h25m26.709497081s) [noise=true , address=64.49.13.138:4490 , piece_id=2XTUZE6PXRE3OARNJVT36FXXWUMYIBVNE6M2VVFAVBGKEE2H5YZQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.875499841 (610h25m26.746582876s) [noise=true , address=64.49.13.139:4478 , piece_id=QN6RT2KWEA6APQVA66ZMKG4LH537UGYNOXOCVVA2YC2MKDOLT3YA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.891607221 (610h25m26.762690256s) [noise=true , address=145.79.227.20:4452 , piece_id=ZSDN2JZ3FQUVJYN5OURDXNKEDLDNULJOHKKRZVIPXZIWIYAHCRXQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.988561223 (610h25m26.859644268s) [noise=true , address=64.49.13.170:4720 , piece_id=HIATRSAPE4YLSODFZDPP43ZUMLDVM7GJEVMWE4ZGUQTP25LTHGTA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.039035911 (610h25m26.910118946s) [noise=true , address=77.90.31.14:4680 , piece_id=S2E4WBA7MJVAZLT5JG3IMHK53RQC77CCHV2PDF6L2R2JEA2U3K4Q, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.098023090 (610h25m26.969106126s) [noise=true , address=45.91.116.6:4981 , piece_id=CT474ZA2LVAZEBJHBDD4GB6NTHVHZFJLNE5OZV3DFS2NZKVW3EQQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.653790754 (610h25m27.524873789s) [noise=true , address=77.90.31.21:4886 , piece_id=K34WPIBGGGEPHIKJ253XGLDCIQ3WLFN4AZ6EYNWVHHT2EIHS6YRQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.684192972 (610h25m27.555276017s) [noise=true , address=193.149.164.143:4704 , piece_id=BHXHFBKTLTDMGXWY5QRTP2THZQE5OXFZ2LCMTO2KRYRWAHXWW3NQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.717238425 (610h25m27.58832146s) [noise=true , address=145.79.227.40:4084 , piece_id=RUKTKDII6BRUWSYVS65CWUBT3QKG3ASUK3KA3O5VGE53XSG6IMOQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.718349742 (610h25m27.589432777s) [noise=true , address=77.90.31.10:4523 , piece_id=P6LIONXJI6AFZ2JTKCQ2W6RZCOZCK6MZ53EXAJKXYQXSX2PX4QEA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.803639562 (610h25m27.674722597s) [noise=true , address=145.79.227.29:4971 , piece_id=MEPXHHOQ6SUZC3GIPCQEMOKKF5OJTKZEI3UYZBAYXAQZGSVEKSBA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.812791279 (610h25m27.683874314s) [noise=true , address=145.79.227.49:4343 , piece_id=6L3HPTS7D2NRA4QEOH4RFM3D3B53CTCONRI6G6YVU4OOTUWTG7XA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.812812348 (610h25m27.683895393s) [seg_pos={0 0}, upload=534952]: Segment reached optimal threshold of 14 pieces. Dec 31 01:54:27.813018000 (610h25m27.684101035s) [noise=true , address=145.79.227.35:4505 , piece_id=V7RS7A4QLZTH4G32ZUZNSF2Z5APVWZGJO77NEGP3SBWQT3WCMVLQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i6itGoBhLsLiqC8et8Y5L62fLjcWxy18u39NAr4R3nXNMWcsf): protocol: expected piece hash; context canceled; EOF Dec 31 01:54:27.813024772 (610h25m27.684107808s) [noise=true , address=145.79.227.35:4505 , piece_id=V7RS7A4QLZTH4G32ZUZNSF2Z5APVWZGJO77NEGP3SBWQT3WCMVLQ, seg_pos={0 0}, upload=534952]: Long tail cancellation due to optimal threshold Dec 31 01:54:27.813175931 (610h25m27.684258976s) [noise=true , address=145.79.227.34:4937 , piece_id=5W7LBM3YJJTVUFYTELR57DVXB663OHO5N3Z2SPQPZ4G43IK47O3Q, seg_pos={0 0}, upload=534952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1LKrFThdW3jJpG6dRZLUaxLXTa5Wy27yQ8PGUEqjvBWfFoRB94): protocol: expected piece hash; context canceled; EOF Dec 31 01:54:27.813181702 (610h25m27.684264737s) [noise=true , address=145.79.227.34:4937 , piece_id=5W7LBM3YJJTVUFYTELR57DVXB663OHO5N3Z2SPQPZ4G43IK47O3Q, seg_pos={0 0}, upload=534952]: Long tail cancellation due to optimal threshold Dec 31 01:54:27.813407862 (610h25m27.684490907s) [noise=true , address=145.79.227.2:4117 , piece_id=4SGFGB3IJ7LTXDMMWM2PGGOV64MC46IZ7OZQWH55J4NBQY7CHMZQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122Htg72f5eUUutf1p3fYQpy1Wkf6AgdaQV83HHYye2n8zm3the): protocol: expected piece hash; context canceled; EOF Dec 31 01:54:27.813414264 (610h25m27.684497299s) [noise=true , address=145.79.227.2:4117 , piece_id=4SGFGB3IJ7LTXDMMWM2PGGOV64MC46IZ7OZQWH55J4NBQY7CHMZQ, seg_pos={0 0}, upload=534952]: Long tail cancellation due to optimal threshold Dec 31 01:54:27.813424614 (610h25m27.684507649s) [noise=true , address=77.90.31.19:4848 , piece_id=VZYG32S6ODIPDKQYZRXECBHXOERZDDWD2OF35Z46W4XMUZ3OJLAA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD): protocol: write tcp 188.166.225.170:42842->77.90.31.19:4848: use of closed network connection; write tcp 188.166.225.170:42842->77.90.31.19:4848: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42842->77.90.31.19:4848: use of closed network connection Dec 31 01:54:27.813432750 (610h25m27.684515785s) [noise=true , address=77.90.31.19:4848 , piece_id=VZYG32S6ODIPDKQYZRXECBHXOERZDDWD2OF35Z46W4XMUZ3OJLAA, seg_pos={0 0}, upload=534952]: Long tail cancellation due to optimal threshold Dec 31 01:54:27.813474088 (610h25m27.684557123s) [seg_pos={0 0}, upload=534952]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 01:54:27.813535325 (610h25m27.68461836s) [upload=534952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 01:54:27.813539613 (610h25m27.684622648s) [upload=534952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 01:54:28.099921034 (610h25m27.971004069s) [upload=534952]: Done uploading segments... Dec 31 01:54:28.099940451 (610h25m27.971023486s) [upload=534952]: Upload finished. err: <nil> Duration: 47.334230052s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 51.210947492s trace: https://jaeger.dev.storj.io/trace/365693fea00df28 FAIL (2025-12-31 02:39:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [noise=true , address=145.79.227.49:4343 , piece_id=6L3HPTS7D2NRA4QEOH4RFM3D3B53CTCONRI6G6YVU4OOTUWTG7XA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.942060847 (610h25m24.813143892s) [noise=true , address=145.79.227.34:4937 , piece_id=5W7LBM3YJJTVUFYTELR57DVXB663OHO5N3Z2SPQPZ4G43IK47O3Q, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.942684104 (610h25m24.813767139s) [noise=true , address=64.49.13.139:4478 , piece_id=QN6RT2KWEA6APQVA66ZMKG4LH537UGYNOXOCVVA2YC2MKDOLT3YA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.943064007 (610h25m24.814147042s) [noise=true , address=145.79.227.40:4084 , piece_id=RUKTKDII6BRUWSYVS65CWUBT3QKG3ASUK3KA3O5VGE53XSG6IMOQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.943459560 (610h25m24.814542605s) [noise=true , address=145.79.227.14:4847 , piece_id=JX5NXTMVM3NSWIEVR4RX46ABSNPKTOFDLFBPKEXFQ3G3ANKV5OLA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.943865303 (610h25m24.814948348s) [noise=true , address=45.91.116.6:4981 , piece_id=CT474ZA2LVAZEBJHBDD4GB6NTHVHZFJLNE5OZV3DFS2NZKVW3EQQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.944259694 (610h25m24.815342749s) [noise=true , address=145.79.227.51:4443 , piece_id=73UUMRDY7BSZNCGBXCZYYRBVE5KLPMNPRNXOZEGWKQGLHJS32J2Q, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.946575182 (610h25m24.817658227s) [noise=true , address=77.90.31.10:4523 , piece_id=P6LIONXJI6AFZ2JTKCQ2W6RZCOZCK6MZ53EXAJKXYQXSX2PX4QEA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.948132909 (610h25m24.819215944s) [noise=true , address=145.79.227.20:4452 , piece_id=ZSDN2JZ3FQUVJYN5OURDXNKEDLDNULJOHKKRZVIPXZIWIYAHCRXQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.949605774 (610h25m24.820688819s) [noise=true , address=77.90.31.14:4680 , piece_id=S2E4WBA7MJVAZLT5JG3IMHK53RQC77CCHV2PDF6L2R2JEA2U3K4Q, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.951459002 (610h25m24.822542048s) [noise=true , address=193.149.164.143:4704 , piece_id=BHXHFBKTLTDMGXWY5QRTP2THZQE5OXFZ2LCMTO2KRYRWAHXWW3NQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.952986742 (610h25m24.824069777s) [noise=true , address=77.90.31.19:4848 , piece_id=VZYG32S6ODIPDKQYZRXECBHXOERZDDWD2OF35Z46W4XMUZ3OJLAA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.956916654 (610h25m24.827999689s) [noise=true , address=145.79.227.29:4971 , piece_id=MEPXHHOQ6SUZC3GIPCQEMOKKF5OJTKZEI3UYZBAYXAQZGSVEKSBA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.960665062 (610h25m24.831748097s) [noise=true , address=64.49.13.170:4720 , piece_id=HIATRSAPE4YLSODFZDPP43ZUMLDVM7GJEVMWE4ZGUQTP25LTHGTA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.962366001 (610h25m24.833449036s) [noise=true , address=145.79.227.2:4117 , piece_id=4SGFGB3IJ7LTXDMMWM2PGGOV64MC46IZ7OZQWH55J4NBQY7CHMZQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.963708216 (610h25m24.834791261s) [noise=true , address=145.79.227.35:4505 , piece_id=V7RS7A4QLZTH4G32ZUZNSF2Z5APVWZGJO77NEGP3SBWQT3WCMVLQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:26.352236138 (610h25m26.223319173s) [upload=534952]: Next returned nil segment Dec 31 01:54:26.352253821 (610h25m26.223336857s) [upload=534952]: Waiting for error group managing segments... Dec 31 01:54:26.812758374 (610h25m26.683841409s) [noise=true , address=145.79.227.51:4443 , piece_id=73UUMRDY7BSZNCGBXCZYYRBVE5KLPMNPRNXOZEGWKQGLHJS32J2Q, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.824371056 (610h25m26.695454101s) [noise=true , address=145.79.227.14:4847 , piece_id=JX5NXTMVM3NSWIEVR4RX46ABSNPKTOFDLFBPKEXFQ3G3ANKV5OLA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.838414046 (610h25m26.709497081s) [noise=true , address=64.49.13.138:4490 , piece_id=2XTUZE6PXRE3OARNJVT36FXXWUMYIBVNE6M2VVFAVBGKEE2H5YZQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.875499841 (610h25m26.746582876s) [noise=true , address=64.49.13.139:4478 , piece_id=QN6RT2KWEA6APQVA66ZMKG4LH537UGYNOXOCVVA2YC2MKDOLT3YA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.891607221 (610h25m26.762690256s) [noise=true , address=145.79.227.20:4452 , piece_id=ZSDN2JZ3FQUVJYN5OURDXNKEDLDNULJOHKKRZVIPXZIWIYAHCRXQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.988561223 (610h25m26.859644268s) [noise=true , address=64.49.13.170:4720 , piece_id=HIATRSAPE4YLSODFZDPP43ZUMLDVM7GJEVMWE4ZGUQTP25LTHGTA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.039035911 (610h25m26.910118946s) [noise=true , address=77.90.31.14:4680 , piece_id=S2E4WBA7MJVAZLT5JG3IMHK53RQC77CCHV2PDF6L2R2JEA2U3K4Q, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.098023090 (610h25m26.969106126s) [noise=true , address=45.91.116.6:4981 , piece_id=CT474ZA2LVAZEBJHBDD4GB6NTHVHZFJLNE5OZV3DFS2NZKVW3EQQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.653790754 (610h25m27.524873789s) [noise=true , address=77.90.31.21:4886 , piece_id=K34WPIBGGGEPHIKJ253XGLDCIQ3WLFN4AZ6EYNWVHHT2EIHS6YRQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.684192972 (610h25m27.555276017s) [noise=true , address=193.149.164.143:4704 , piece_id=BHXHFBKTLTDMGXWY5QRTP2THZQE5OXFZ2LCMTO2KRYRWAHXWW3NQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.717238425 (610h25m27.58832146s) [noise=true , address=145.79.227.40:4084 , piece_id=RUKTKDII6BRUWSYVS65CWUBT3QKG3ASUK3KA3O5VGE53XSG6IMOQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.718349742 (610h25m27.589432777s) [noise=true , address=77.90.31.10:4523 , piece_id=P6LIONXJI6AFZ2JTKCQ2W6RZCOZCK6MZ53EXAJKXYQXSX2PX4QEA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.803639562 (610h25m27.674722597s) [noise=true , address=145.79.227.29:4971 , piece_id=MEPXHHOQ6SUZC3GIPCQEMOKKF5OJTKZEI3UYZBAYXAQZGSVEKSBA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.812791279 (610h25m27.683874314s) [noise=true , address=145.79.227.49:4343 , piece_id=6L3HPTS7D2NRA4QEOH4RFM3D3B53CTCONRI6G6YVU4OOTUWTG7XA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.812812348 (610h25m27.683895393s) [seg_pos={0 0}, upload=534952]: Segment reached optimal threshold of 14 pieces. Dec 31 01:54:27.813018000 (610h25m27.684101035s) [noise=true , address=145.79.227.35:4505 , piece_id=V7RS7A4QLZTH4G32ZUZNSF2Z5APVWZGJO77NEGP3SBWQT3WCMVLQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i6itGoBhLsLiqC8et8Y5L62fLjcWxy18u39NAr4R3nXNMWcsf): protocol: expected piece hash; context canceled; EOF Dec 31 01:54:27.813024772 (610h25m27.684107808s) [noise=true , address=145.79.227.35:4505 , piece_id=V7RS7A4QLZTH4G32ZUZNSF2Z5APVWZGJO77NEGP3SBWQT3WCMVLQ, seg_pos={0 0}, upload=534952]: Long tail cancellation due to optimal threshold Dec 31 01:54:27.813175931 (610h25m27.684258976s) [noise=true , address=145.79.227.34:4937 , piece_id=5W7LBM3YJJTVUFYTELR57DVXB663OHO5N3Z2SPQPZ4G43IK47O3Q, seg_pos={0 0}, upload=534952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1LKrFThdW3jJpG6dRZLUaxLXTa5Wy27yQ8PGUEqjvBWfFoRB94): protocol: expected piece hash; context canceled; EOF Dec 31 01:54:27.813181702 (610h25m27.684264737s) [noise=true , address=145.79.227.34:4937 , piece_id=5W7LBM3YJJTVUFYTELR57DVXB663OHO5N3Z2SPQPZ4G43IK47O3Q, seg_pos={0 0}, upload=534952]: Long tail cancellation due to optimal threshold Dec 31 01:54:27.813407862 (610h25m27.684490907s) [noise=true , address=145.79.227.2:4117 , piece_id=4SGFGB3IJ7LTXDMMWM2PGGOV64MC46IZ7OZQWH55J4NBQY7CHMZQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122Htg72f5eUUutf1p3fYQpy1Wkf6AgdaQV83HHYye2n8zm3the): protocol: expected piece hash; context canceled; EOF Dec 31 01:54:27.813414264 (610h25m27.684497299s) [noise=true , address=145.79.227.2:4117 , piece_id=4SGFGB3IJ7LTXDMMWM2PGGOV64MC46IZ7OZQWH55J4NBQY7CHMZQ, seg_pos={0 0}, upload=534952]: Long tail cancellation due to optimal threshold Dec 31 01:54:27.813424614 (610h25m27.684507649s) [noise=true , address=77.90.31.19:4848 , piece_id=VZYG32S6ODIPDKQYZRXECBHXOERZDDWD2OF35Z46W4XMUZ3OJLAA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD): protocol: write tcp 188.166.225.170:42842->77.90.31.19:4848: use of closed network connection; write tcp 188.166.225.170:42842->77.90.31.19:4848: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42842->77.90.31.19:4848: use of closed network connection Dec 31 01:54:27.813432750 (610h25m27.684515785s) [noise=true , address=77.90.31.19:4848 , piece_id=VZYG32S6ODIPDKQYZRXECBHXOERZDDWD2OF35Z46W4XMUZ3OJLAA, seg_pos={0 0}, upload=534952]: Long tail cancellation due to optimal threshold Dec 31 01:54:27.813474088 (610h25m27.684557123s) [seg_pos={0 0}, upload=534952]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 01:54:27.813535325 (610h25m27.68461836s) [upload=534952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 01:54:27.813539613 (610h25m27.684622648s) [upload=534952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 01:54:28.099921034 (610h25m27.971004069s) [upload=534952]: Done uploading segments... Dec 31 01:54:28.099940451 (610h25m27.971023486s) [upload=534952]: Upload finished. err: <nil> Duration: 51.210980505s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.....................X...............................X....X.] duration: 3.439170891s trace: https://jaeger.dev.storj.io/trace/7bc7478624e56c1a FAIL (2025-12-31 02:39:01 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.43919672s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [...X..X.......X.X..X....XX.XX........X.X......X..X......XXX.] duration: 11.981456104s trace: https://jaeger.dev.storj.io/trace/6707b59ca1e5b458 FAIL (2025-12-31 02:38:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: 3Q7UQ3435OJP66RHTZTQPGPIJSUW5Y52F77WE3UT6YPEDIRKLOUQ) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: 5RYGOIN4YBYWIYUIQCUCLGJXIFWRIOTRLV6BOTEG7BE4VTFN4MVQ) context deadline exceeded ======LOGS====== (611h9m18.344144905s) [upload=535584]: Uploading segments... Dec 31 02:38:18.473115882 (611h9m18.344198917s) [upload=535584]: Got next segment. Inline: false Dec 31 02:38:18.473142283 (611h9m18.344225318s) [upload=535584]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 02:38:18.473146481 (611h9m18.344229516s) [upload=535584]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 02:38:18.716039759 (611h9m18.587122804s) [seg_pos={0 0}, upload=535584]: Begin upload segment... Dec 31 02:38:18.716109301 (611h9m18.587192336s) [seg_pos={0 0}, upload=535584]: Done begin upload segment. Dec 31 02:38:18.716137034 (611h9m18.587220069s) [noise=true , address=45.91.116.14:4189 , piece_id=ACDVMCUOLEM3E7GCVEJBPCS73OUJDMKEODTP2KPDFF7C2UX672QA, seg_pos={0 0}, upload=535584]: Uploading piece... Dec 31 02:38:18.716164727 (611h9m18.587247762s) [noise=true , address=45.91.116.6:4976 , piece_id=MEAAYINZGA6YTPNZCDUPZ3AODLPOIWNJIKQUB3Q56FIO4IN6R7FQ, seg_pos={0 0}, upload=535584]: Uploading piece... Dec 31 02:38:18.716631967 (611h9m18.587715002s) [noise=true , address=45.91.116.13:4153 , piece_id=3Q7UQ3435OJP66RHTZTQPGPIJSUW5Y52F77WE3UT6YPEDIRKLOUQ, seg_pos={0 0}, upload=535584]: Uploading piece... Dec 31 02:38:18.717028412 (611h9m18.588111457s) [noise=true , address=145.79.227.31:4798 , piece_id=5V4LMCUVVS7ZAEEH3W7JJCD3GU5WQSVYR3JJRUFPD3IHJJHPDP6A, seg_pos={0 0}, upload=535584]: Uploading piece... Dec 31 02:38:18.717627543 (611h9m18.588710588s) [noise=true , address=145.79.227.15:4873 , piece_id=OGUUFEOBIY7JUXGPHPAAEW7KP27NLO2X5QQDFMD4EGG5OJ4QTYMA, seg_pos={0 0}, upload=535584]: Uploading piece... Dec 31 02:38:18.718193149 (611h9m18.589276195s) [noise=true , address=45.91.116.8:4046 , piece_id=5RYGOIN4YBYWIYUIQCUCLGJXIFWRIOTRLV6BOTEG7BE4VTFN4MVQ, seg_pos={0 0}, upload=535584]: Uploading piece... Dec 31 02:38:18.718873404 (611h9m18.589956439s) [noise=true , address=77.90.31.16:4735 , piece_id=O7WO7ZQJJJUG7M4CYBZVDXWPILO6JPIXQOZUAXB5BZMLQLL77XQQ, seg_pos={0 0}, upload=535584]: Uploading piece... Dec 31 02:38:18.720029065 (611h9m18.5911121s) [noise=true , address=145.79.227.50:4375 , piece_id=POT3FF3VPATA3OAKHXHY3SUI2DI2UHT6GH7VXRRQBZIOXSMSVAEA, seg_pos={0 0}, upload=535584]: Uploading piece... Dec 31 02:38:18.721622589 (611h9m18.592705624s) [noise=true , address=45.91.116.7:4009 , piece_id=SVSZ5B3RAQYH4RGOV5UNTIHPEYCJTBB3EEYUVU2RW5UN62AAQJNA, seg_pos={0 0}, upload=535584]: Uploading piece... Dec 31 02:38:18.721912391 (611h9m18.592995436s) [noise=true , address=77.90.31.14:4664 , piece_id=P72CNBKIRCALEDSJUIYLZEGTQWWI454LSGO23GWIMBIMYMJZI5QQ, seg_pos={0 0}, upload=535584]: Uploading piece... Dec 31 02:38:18.725644548 (611h9m18.596727593s) [noise=true , address=145.79.227.41:4085 , piece_id=TQA2Z6FY6SSZME6TW3X4POZTJIYSWUMIMOAEXOZ7RM7JISRXABKQ, seg_pos={0 0}, upload=535584]: Uploading piece... Dec 31 02:38:18.726151974 (611h9m18.597235009s) [noise=true , address=77.90.31.11:4555 , piece_id=MEFFVXPYXERLBAATG5XSCEHUKHOA3DS6PIPHAEBHJ23Y3TGAFRVA, seg_pos={0 0}, upload=535584]: Uploading piece... Dec 31 02:38:18.728917180 (611h9m18.600000225s) [noise=true , address=145.79.227.51:4409 , piece_id=H5WXMPEJUQNNRKWNVJEIIUQ45BG5DLTQHGBGT2SRYAITH6JHUT3A, seg_pos={0 0}, upload=535584]: Uploading piece... Dec 31 02:38:18.728959831 (611h9m18.600042916s) [noise=true , address=145.79.227.9:4365 , piece_id=TTP6PH74NWNZ7PGOPU5DMJ6O47M2AGZX32KSGG72X5LJFVBJVV6A, seg_pos={0 0}, upload=535584]: Uploading piece... Dec 31 02:38:18.730985568 (611h9m18.602068603s) [noise=true , address=77.90.31.21:4880 , piece_id=J64Q3NU2YQFJOI3SZMV7KOFDOISWX32K4NTPAMLOZLM2BIG34TLA, seg_pos={0 0}, upload=535584]: Uploading piece... Dec 31 02:38:18.731444271 (611h9m18.602527306s) [noise=true , address=145.79.227.46:4603 , piece_id=PQK6LWX3DD3DQTZFOO36YCWCFPINDQITEZFAPMPEPUFNTP4ZPKFQ, seg_pos={0 0}, upload=535584]: Uploading piece... Dec 31 02:38:20.431446432 (611h9m20.302529508s) [upload=535584]: Next returned nil segment Dec 31 02:38:20.431463736 (611h9m20.302546801s) [upload=535584]: Waiting for error group managing segments... Dec 31 02:38:20.704147128 (611h9m20.575230183s) [noise=true , address=145.79.227.31:4798 , piece_id=5V4LMCUVVS7ZAEEH3W7JJCD3GU5WQSVYR3JJRUFPD3IHJJHPDP6A, seg_pos={0 0}, upload=535584]: Done uploading piece. err: <nil> Dec 31 02:38:20.718728553 (611h9m20.589811588s) [noise=true , address=145.79.227.51:4409 , piece_id=H5WXMPEJUQNNRKWNVJEIIUQ45BG5DLTQHGBGT2SRYAITH6JHUT3A, seg_pos={0 0}, upload=535584]: Done uploading piece. err: <nil> Dec 31 02:38:20.736431231 (611h9m20.607514266s) [noise=true , address=145.79.227.50:4375 , piece_id=POT3FF3VPATA3OAKHXHY3SUI2DI2UHT6GH7VXRRQBZIOXSMSVAEA, seg_pos={0 0}, upload=535584]: Done uploading piece. err: <nil> Dec 31 02:38:20.740008923 (611h9m20.611091968s) [noise=true , address=145.79.227.9:4365 , piece_id=TTP6PH74NWNZ7PGOPU5DMJ6O47M2AGZX32KSGG72X5LJFVBJVV6A, seg_pos={0 0}, upload=535584]: Done uploading piece. err: <nil> Dec 31 02:38:20.741909282 (611h9m20.612992337s) [noise=true , address=145.79.227.15:4873 , piece_id=OGUUFEOBIY7JUXGPHPAAEW7KP27NLO2X5QQDFMD4EGG5OJ4QTYMA, seg_pos={0 0}, upload=535584]: Done uploading piece. err: <nil> Dec 31 02:38:20.744063354 (611h9m20.615146399s) [noise=true , address=145.79.227.41:4085 , piece_id=TQA2Z6FY6SSZME6TW3X4POZTJIYSWUMIMOAEXOZ7RM7JISRXABKQ, seg_pos={0 0}, upload=535584]: Done uploading piece. err: <nil> Dec 31 02:38:20.817838119 (611h9m20.688921154s) [noise=true , address=145.79.227.46:4603 , piece_id=PQK6LWX3DD3DQTZFOO36YCWCFPINDQITEZFAPMPEPUFNTP4ZPKFQ, seg_pos={0 0}, upload=535584]: Done uploading piece. err: <nil> Dec 31 02:38:20.927769442 (611h9m20.798852477s) [noise=true , address=77.90.31.16:4735 , piece_id=O7WO7ZQJJJUG7M4CYBZVDXWPILO6JPIXQOZUAXB5BZMLQLL77XQQ, seg_pos={0 0}, upload=535584]: Done uploading piece. err: <nil> Dec 31 02:38:20.931851806 (611h9m20.802934851s) [noise=true , address=77.90.31.21:4880 , piece_id=J64Q3NU2YQFJOI3SZMV7KOFDOISWX32K4NTPAMLOZLM2BIG34TLA, seg_pos={0 0}, upload=535584]: Done uploading piece. err: <nil> Dec 31 02:38:20.972975961 (611h9m20.844059006s) [noise=true , address=77.90.31.11:4555 , piece_id=MEFFVXPYXERLBAATG5XSCEHUKHOA3DS6PIPHAEBHJ23Y3TGAFRVA, seg_pos={0 0}, upload=535584]: Done uploading piece. err: <nil> Dec 31 02:38:21.035055207 (611h9m20.906138242s) [noise=true , address=45.91.116.8:4046 , piece_id=5RYGOIN4YBYWIYUIQCUCLGJXIFWRIOTRLV6BOTEG7BE4VTFN4MVQ, seg_pos={0 0}, upload=535584]: Done uploading piece. err: <nil> Dec 31 02:38:21.059374474 (611h9m20.930457509s) [noise=true , address=45.91.116.13:4153 , piece_id=3Q7UQ3435OJP66RHTZTQPGPIJSUW5Y52F77WE3UT6YPEDIRKLOUQ, seg_pos={0 0}, upload=535584]: Done uploading piece. err: <nil> Dec 31 02:38:21.067861705 (611h9m20.93894474s) [noise=true , address=45.91.116.6:4976 , piece_id=MEAAYINZGA6YTPNZCDUPZ3AODLPOIWNJIKQUB3Q56FIO4IN6R7FQ, seg_pos={0 0}, upload=535584]: Done uploading piece. err: <nil> Dec 31 02:38:21.067888396 (611h9m20.938971451s) [seg_pos={0 0}, upload=535584]: Segment reached optimal threshold of 13 pieces. Dec 31 02:38:21.068160553 (611h9m20.939243588s) [noise=true , address=77.90.31.14:4664 , piece_id=P72CNBKIRCALEDSJUIYLZEGTQWWI454LSGO23GWIMBIMYMJZI5QQ, seg_pos={0 0}, upload=535584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: write tcp 188.166.225.170:49990->77.90.31.14:4664: use of closed network connection; write tcp 188.166.225.170:49990->77.90.31.14:4664: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49990->77.90.31.14:4664: use of closed network connection Dec 31 02:38:21.068209036 (611h9m20.939292081s) [noise=true , address=77.90.31.14:4664 , piece_id=P72CNBKIRCALEDSJUIYLZEGTQWWI454LSGO23GWIMBIMYMJZI5QQ, seg_pos={0 0}, upload=535584]: Long tail cancellation due to optimal threshold Dec 31 02:38:21.068222862 (611h9m20.939305917s) [noise=true , address=45.91.116.14:4189 , piece_id=ACDVMCUOLEM3E7GCVEJBPCS73OUJDMKEODTP2KPDFF7C2UX672QA, seg_pos={0 0}, upload=535584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 31 02:38:21.068232911 (611h9m20.939315946s) [noise=true , address=45.91.116.14:4189 , piece_id=ACDVMCUOLEM3E7GCVEJBPCS73OUJDMKEODTP2KPDFF7C2UX672QA, seg_pos={0 0}, upload=535584]: Long tail cancellation due to optimal threshold Dec 31 02:38:21.068372507 (611h9m20.939455562s) [noise=true , address=45.91.116.7:4009 , piece_id=SVSZ5B3RAQYH4RGOV5UNTIHPEYCJTBB3EEYUVU2RW5UN62AAQJNA, seg_pos={0 0}, upload=535584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:39450->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:39450->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39450->45.91.116.7:4009: use of closed network connection Dec 31 02:38:21.068380613 (611h9m20.939463648s) [noise=true , address=45.91.116.7:4009 , piece_id=SVSZ5B3RAQYH4RGOV5UNTIHPEYCJTBB3EEYUVU2RW5UN62AAQJNA, seg_pos={0 0}, upload=535584]: Long tail cancellation due to optimal threshold Dec 31 02:38:21.068401542 (611h9m20.939484587s) [seg_pos={0 0}, upload=535584]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 02:38:21.068492606 (611h9m20.939575651s) [upload=535584]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 02:38:21.068502314 (611h9m20.939585359s) [upload=535584]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 02:38:21.352975415 (611h9m21.22405849s) [upload=535584]: Done uploading segments... Dec 31 02:38:21.353002297 (611h9m21.224085342s) [upload=535584]: Upload finished. err: <nil> Duration: 11.981471283s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [....X..X.......X.X..X....XX.XX........X.X......X..X......XXX] duration: 30.002730633s trace: https://jaeger.dev.storj.io/trace/6707b59ca1e5b458 FAIL (2025-12-31 02:38:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: 3Q7UQ3435OJP66RHTZTQPGPIJSUW5Y52F77WE3UT6YPEDIRKLOUQ) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: 5RYGOIN4YBYWIYUIQCUCLGJXIFWRIOTRLV6BOTEG7BE4VTFN4MVQ) context deadline exceeded ======LOGS====== (611h9m18.344144905s) [upload=535584]: Uploading segments... Dec 31 02:38:18.473115882 (611h9m18.344198917s) [upload=535584]: Got next segment. Inline: false Dec 31 02:38:18.473142283 (611h9m18.344225318s) [upload=535584]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 02:38:18.473146481 (611h9m18.344229516s) [upload=535584]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 02:38:18.716039759 (611h9m18.587122804s) [seg_pos={0 0}, upload=535584]: Begin upload segment... Dec 31 02:38:18.716109301 (611h9m18.587192336s) [seg_pos={0 0}, upload=535584]: Done begin upload segment. Dec 31 02:38:18.716137034 (611h9m18.587220069s) [noise=true , address=45.91.116.14:4189 , piece_id=ACDVMCUOLEM3E7GCVEJBPCS73OUJDMKEODTP2KPDFF7C2UX672QA, seg_pos={0 0}, upload=535584]: Uploading piece... Dec 31 02:38:18.716164727 (611h9m18.587247762s) [noise=true , address=45.91.116.6:4976 , piece_id=MEAAYINZGA6YTPNZCDUPZ3AODLPOIWNJIKQUB3Q56FIO4IN6R7FQ, seg_pos={0 0}, upload=535584]: Uploading piece... Dec 31 02:38:18.716631967 (611h9m18.587715002s) [noise=true , address=45.91.116.13:4153 , piece_id=3Q7UQ3435OJP66RHTZTQPGPIJSUW5Y52F77WE3UT6YPEDIRKLOUQ, seg_pos={0 0}, upload=535584]: Uploading piece... Dec 31 02:38:18.717028412 (611h9m18.588111457s) [noise=true , address=145.79.227.31:4798 , piece_id=5V4LMCUVVS7ZAEEH3W7JJCD3GU5WQSVYR3JJRUFPD3IHJJHPDP6A, seg_pos={0 0}, upload=535584]: Uploading piece... Dec 31 02:38:18.717627543 (611h9m18.588710588s) [noise=true , address=145.79.227.15:4873 , piece_id=OGUUFEOBIY7JUXGPHPAAEW7KP27NLO2X5QQDFMD4EGG5OJ4QTYMA, seg_pos={0 0}, upload=535584]: Uploading piece... Dec 31 02:38:18.718193149 (611h9m18.589276195s) [noise=true , address=45.91.116.8:4046 , piece_id=5RYGOIN4YBYWIYUIQCUCLGJXIFWRIOTRLV6BOTEG7BE4VTFN4MVQ, seg_pos={0 0}, upload=535584]: Uploading piece... Dec 31 02:38:18.718873404 (611h9m18.589956439s) [noise=true , address=77.90.31.16:4735 , piece_id=O7WO7ZQJJJUG7M4CYBZVDXWPILO6JPIXQOZUAXB5BZMLQLL77XQQ, seg_pos={0 0}, upload=535584]: Uploading piece... Dec 31 02:38:18.720029065 (611h9m18.5911121s) [noise=true , address=145.79.227.50:4375 , piece_id=POT3FF3VPATA3OAKHXHY3SUI2DI2UHT6GH7VXRRQBZIOXSMSVAEA, seg_pos={0 0}, upload=535584]: Uploading piece... Dec 31 02:38:18.721622589 (611h9m18.592705624s) [noise=true , address=45.91.116.7:4009 , piece_id=SVSZ5B3RAQYH4RGOV5UNTIHPEYCJTBB3EEYUVU2RW5UN62AAQJNA, seg_pos={0 0}, upload=535584]: Uploading piece... Dec 31 02:38:18.721912391 (611h9m18.592995436s) [noise=true , address=77.90.31.14:4664 , piece_id=P72CNBKIRCALEDSJUIYLZEGTQWWI454LSGO23GWIMBIMYMJZI5QQ, seg_pos={0 0}, upload=535584]: Uploading piece... Dec 31 02:38:18.725644548 (611h9m18.596727593s) [noise=true , address=145.79.227.41:4085 , piece_id=TQA2Z6FY6SSZME6TW3X4POZTJIYSWUMIMOAEXOZ7RM7JISRXABKQ, seg_pos={0 0}, upload=535584]: Uploading piece... Dec 31 02:38:18.726151974 (611h9m18.597235009s) [noise=true , address=77.90.31.11:4555 , piece_id=MEFFVXPYXERLBAATG5XSCEHUKHOA3DS6PIPHAEBHJ23Y3TGAFRVA, seg_pos={0 0}, upload=535584]: Uploading piece... Dec 31 02:38:18.728917180 (611h9m18.600000225s) [noise=true , address=145.79.227.51:4409 , piece_id=H5WXMPEJUQNNRKWNVJEIIUQ45BG5DLTQHGBGT2SRYAITH6JHUT3A, seg_pos={0 0}, upload=535584]: Uploading piece... Dec 31 02:38:18.728959831 (611h9m18.600042916s) [noise=true , address=145.79.227.9:4365 , piece_id=TTP6PH74NWNZ7PGOPU5DMJ6O47M2AGZX32KSGG72X5LJFVBJVV6A, seg_pos={0 0}, upload=535584]: Uploading piece... Dec 31 02:38:18.730985568 (611h9m18.602068603s) [noise=true , address=77.90.31.21:4880 , piece_id=J64Q3NU2YQFJOI3SZMV7KOFDOISWX32K4NTPAMLOZLM2BIG34TLA, seg_pos={0 0}, upload=535584]: Uploading piece... Dec 31 02:38:18.731444271 (611h9m18.602527306s) [noise=true , address=145.79.227.46:4603 , piece_id=PQK6LWX3DD3DQTZFOO36YCWCFPINDQITEZFAPMPEPUFNTP4ZPKFQ, seg_pos={0 0}, upload=535584]: Uploading piece... Dec 31 02:38:20.431446432 (611h9m20.302529508s) [upload=535584]: Next returned nil segment Dec 31 02:38:20.431463736 (611h9m20.302546801s) [upload=535584]: Waiting for error group managing segments... Dec 31 02:38:20.704147128 (611h9m20.575230183s) [noise=true , address=145.79.227.31:4798 , piece_id=5V4LMCUVVS7ZAEEH3W7JJCD3GU5WQSVYR3JJRUFPD3IHJJHPDP6A, seg_pos={0 0}, upload=535584]: Done uploading piece. err: <nil> Dec 31 02:38:20.718728553 (611h9m20.589811588s) [noise=true , address=145.79.227.51:4409 , piece_id=H5WXMPEJUQNNRKWNVJEIIUQ45BG5DLTQHGBGT2SRYAITH6JHUT3A, seg_pos={0 0}, upload=535584]: Done uploading piece. err: <nil> Dec 31 02:38:20.736431231 (611h9m20.607514266s) [noise=true , address=145.79.227.50:4375 , piece_id=POT3FF3VPATA3OAKHXHY3SUI2DI2UHT6GH7VXRRQBZIOXSMSVAEA, seg_pos={0 0}, upload=535584]: Done uploading piece. err: <nil> Dec 31 02:38:20.740008923 (611h9m20.611091968s) [noise=true , address=145.79.227.9:4365 , piece_id=TTP6PH74NWNZ7PGOPU5DMJ6O47M2AGZX32KSGG72X5LJFVBJVV6A, seg_pos={0 0}, upload=535584]: Done uploading piece. err: <nil> Dec 31 02:38:20.741909282 (611h9m20.612992337s) [noise=true , address=145.79.227.15:4873 , piece_id=OGUUFEOBIY7JUXGPHPAAEW7KP27NLO2X5QQDFMD4EGG5OJ4QTYMA, seg_pos={0 0}, upload=535584]: Done uploading piece. err: <nil> Dec 31 02:38:20.744063354 (611h9m20.615146399s) [noise=true , address=145.79.227.41:4085 , piece_id=TQA2Z6FY6SSZME6TW3X4POZTJIYSWUMIMOAEXOZ7RM7JISRXABKQ, seg_pos={0 0}, upload=535584]: Done uploading piece. err: <nil> Dec 31 02:38:20.817838119 (611h9m20.688921154s) [noise=true , address=145.79.227.46:4603 , piece_id=PQK6LWX3DD3DQTZFOO36YCWCFPINDQITEZFAPMPEPUFNTP4ZPKFQ, seg_pos={0 0}, upload=535584]: Done uploading piece. err: <nil> Dec 31 02:38:20.927769442 (611h9m20.798852477s) [noise=true , address=77.90.31.16:4735 , piece_id=O7WO7ZQJJJUG7M4CYBZVDXWPILO6JPIXQOZUAXB5BZMLQLL77XQQ, seg_pos={0 0}, upload=535584]: Done uploading piece. err: <nil> Dec 31 02:38:20.931851806 (611h9m20.802934851s) [noise=true , address=77.90.31.21:4880 , piece_id=J64Q3NU2YQFJOI3SZMV7KOFDOISWX32K4NTPAMLOZLM2BIG34TLA, seg_pos={0 0}, upload=535584]: Done uploading piece. err: <nil> Dec 31 02:38:20.972975961 (611h9m20.844059006s) [noise=true , address=77.90.31.11:4555 , piece_id=MEFFVXPYXERLBAATG5XSCEHUKHOA3DS6PIPHAEBHJ23Y3TGAFRVA, seg_pos={0 0}, upload=535584]: Done uploading piece. err: <nil> Dec 31 02:38:21.035055207 (611h9m20.906138242s) [noise=true , address=45.91.116.8:4046 , piece_id=5RYGOIN4YBYWIYUIQCUCLGJXIFWRIOTRLV6BOTEG7BE4VTFN4MVQ, seg_pos={0 0}, upload=535584]: Done uploading piece. err: <nil> Dec 31 02:38:21.059374474 (611h9m20.930457509s) [noise=true , address=45.91.116.13:4153 , piece_id=3Q7UQ3435OJP66RHTZTQPGPIJSUW5Y52F77WE3UT6YPEDIRKLOUQ, seg_pos={0 0}, upload=535584]: Done uploading piece. err: <nil> Dec 31 02:38:21.067861705 (611h9m20.93894474s) [noise=true , address=45.91.116.6:4976 , piece_id=MEAAYINZGA6YTPNZCDUPZ3AODLPOIWNJIKQUB3Q56FIO4IN6R7FQ, seg_pos={0 0}, upload=535584]: Done uploading piece. err: <nil> Dec 31 02:38:21.067888396 (611h9m20.938971451s) [seg_pos={0 0}, upload=535584]: Segment reached optimal threshold of 13 pieces. Dec 31 02:38:21.068160553 (611h9m20.939243588s) [noise=true , address=77.90.31.14:4664 , piece_id=P72CNBKIRCALEDSJUIYLZEGTQWWI454LSGO23GWIMBIMYMJZI5QQ, seg_pos={0 0}, upload=535584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: write tcp 188.166.225.170:49990->77.90.31.14:4664: use of closed network connection; write tcp 188.166.225.170:49990->77.90.31.14:4664: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49990->77.90.31.14:4664: use of closed network connection Dec 31 02:38:21.068209036 (611h9m20.939292081s) [noise=true , address=77.90.31.14:4664 , piece_id=P72CNBKIRCALEDSJUIYLZEGTQWWI454LSGO23GWIMBIMYMJZI5QQ, seg_pos={0 0}, upload=535584]: Long tail cancellation due to optimal threshold Dec 31 02:38:21.068222862 (611h9m20.939305917s) [noise=true , address=45.91.116.14:4189 , piece_id=ACDVMCUOLEM3E7GCVEJBPCS73OUJDMKEODTP2KPDFF7C2UX672QA, seg_pos={0 0}, upload=535584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 31 02:38:21.068232911 (611h9m20.939315946s) [noise=true , address=45.91.116.14:4189 , piece_id=ACDVMCUOLEM3E7GCVEJBPCS73OUJDMKEODTP2KPDFF7C2UX672QA, seg_pos={0 0}, upload=535584]: Long tail cancellation due to optimal threshold Dec 31 02:38:21.068372507 (611h9m20.939455562s) [noise=true , address=45.91.116.7:4009 , piece_id=SVSZ5B3RAQYH4RGOV5UNTIHPEYCJTBB3EEYUVU2RW5UN62AAQJNA, seg_pos={0 0}, upload=535584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:39450->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:39450->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39450->45.91.116.7:4009: use of closed network connection Dec 31 02:38:21.068380613 (611h9m20.939463648s) [noise=true , address=45.91.116.7:4009 , piece_id=SVSZ5B3RAQYH4RGOV5UNTIHPEYCJTBB3EEYUVU2RW5UN62AAQJNA, seg_pos={0 0}, upload=535584]: Long tail cancellation due to optimal threshold Dec 31 02:38:21.068401542 (611h9m20.939484587s) [seg_pos={0 0}, upload=535584]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 02:38:21.068492606 (611h9m20.939575651s) [upload=535584]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 02:38:21.068502314 (611h9m20.939585359s) [upload=535584]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 02:38:21.352975415 (611h9m21.22405849s) [upload=535584]: Done uploading segments... Dec 31 02:38:21.353002297 (611h9m21.224085342s) [upload=535584]: Upload finished. err: <nil> Duration: 30.002742777s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.....X..X.......X.X..X....XX.XX........X.X......X..X......XX] duration: 30.003710168s trace: https://jaeger.dev.storj.io/trace/e245c580dbc1dfe FAIL (2025-12-31 02:38:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: HU3WYZYADXZMLHUWBVPZZXCTTNNLBYT52NKRQYSAOX3BSPKOFIBQ) context deadline exceeded; piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: BQJE2HHQLCZCIGGUTPNPUPMQ5R4FK6Z4CW4HPF6CLEKTQSPUPIFQ) context deadline exceeded ======LOGS====== 611h8m48.118548145s) [upload=535575]: Uploading segments... Dec 31 02:37:48.247497912 (611h8m48.118580947s) [upload=535575]: Got next segment. Inline: false Dec 31 02:37:48.247519473 (611h8m48.118602509s) [upload=535575]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 02:37:48.247523741 (611h8m48.118606777s) [upload=535575]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 02:37:48.499698473 (611h8m48.370781508s) [seg_pos={0 0}, upload=535575]: Begin upload segment... Dec 31 02:37:48.499740873 (611h8m48.370823908s) [seg_pos={0 0}, upload=535575]: Done begin upload segment. Dec 31 02:37:48.499761202 (611h8m48.370844237s) [noise=true , address=145.79.227.21:4474 , piece_id=HGP2VTBOZ3CZKUYC3RZLAJZCFGDHE6UK4QURCGW7PVLDJNQN24OQ, seg_pos={0 0}, upload=535575]: Uploading piece... Dec 31 02:37:48.499770710 (611h8m48.370853745s) [noise=true , address=145.79.227.41:4086 , piece_id=ZBOPXCE47GYTSNZRRY6FXBU425GYHMUEMM2LO6DYODH2MCVKGGTA, seg_pos={0 0}, upload=535575]: Uploading piece... Dec 31 02:37:48.500593086 (611h8m48.371676121s) [noise=true , address=145.79.227.2:4115 , piece_id=RQLXJOQAL5JSULQENFAVGYLDMBUEYZU6SZ23DWYW2MZFEBYGESSQ, seg_pos={0 0}, upload=535575]: Uploading piece... Dec 31 02:37:48.500886404 (611h8m48.37196945s) [noise=true , address=145.79.227.11:4729 , piece_id=JVOMGWKUQLTLGM7BHBJUC6Q2SZ4X74BBL4VUPJCYCX75RKS7JPBA, seg_pos={0 0}, upload=535575]: Uploading piece... Dec 31 02:37:48.501246059 (611h8m48.372329094s) [noise=true , address=145.79.227.7:4293 , piece_id=ZWBZUU43YH4G3HE3GRLPWHYWZRSA2LPYBKQEPFKTCXMARJCY64YQ, seg_pos={0 0}, upload=535575]: Uploading piece... Dec 31 02:37:48.501509852 (611h8m48.372592887s) [noise=true , address=145.79.227.34:4907 , piece_id=JQZ6OUDMO2KDJ6M3FANWJI6LNOJ3Q36CFM4FC45HWP3TATNZQ5UA, seg_pos={0 0}, upload=535575]: Uploading piece... Dec 31 02:37:48.501767502 (611h8m48.372850537s) [noise=true , address=145.79.227.51:4411 , piece_id=IMAZPT7VUBR3T2W2A7ZNRYFU2LXOCSOBTCAMWUQVKJSMHYGZGOAA, seg_pos={0 0}, upload=535575]: Uploading piece... Dec 31 02:37:48.501772853 (611h8m48.372855888s) [noise=true , address=77.90.31.13:4627 , piece_id=HU3WYZYADXZMLHUWBVPZZXCTTNNLBYT52NKRQYSAOX3BSPKOFIBQ, seg_pos={0 0}, upload=535575]: Uploading piece... Dec 31 02:37:48.502062915 (611h8m48.37314595s) [noise=true , address=45.91.116.9:4262 , piece_id=42EOUVITCXQKYK5S73IF2XK22HWWSWEDQ6UT3Z33F6ETKQHZGFFQ, seg_pos={0 0}, upload=535575]: Uploading piece... Dec 31 02:37:48.502171993 (611h8m48.373255028s) [noise=true , address=145.79.227.20:4437 , piece_id=WCPTJFYPMNW3RGQMK4BSLK5S5YCBUQG5OYNEJVC6VEXQDUMQRGJA, seg_pos={0 0}, upload=535575]: Uploading piece... Dec 31 02:37:48.502372054 (611h8m48.373455189s) [noise=true , address=145.79.227.50:4374 , piece_id=HO244SW4SOZORMMCEYTPGXKZXTI3UAGAF2G6HQ6AG5CWQT6W6FBQ, seg_pos={0 0}, upload=535575]: Uploading piece... Dec 31 02:37:48.502547498 (611h8m48.373630533s) [noise=true , address=45.91.116.6:4976 , piece_id=BQJE2HHQLCZCIGGUTPNPUPMQ5R4FK6Z4CW4HPF6CLEKTQSPUPIFQ, seg_pos={0 0}, upload=535575]: Uploading piece... Dec 31 02:37:48.502704797 (611h8m48.373787832s) [noise=true , address=145.79.227.42:4123 , piece_id=VYO7SPUB2M5R4DQQUD5DAQULL5GR75RS4AJI4NEL3A4BFPDPAHAQ, seg_pos={0 0}, upload=535575]: Uploading piece... Dec 31 02:37:48.502852439 (611h8m48.373935474s) [noise=true , address=77.90.31.18:4805 , piece_id=ATIFFJ4HGITKWKKJZR3FCD43GQNGTNYPLJEXAOFH74D3URLE5TGA, seg_pos={0 0}, upload=535575]: Uploading piece... Dec 31 02:37:48.502988879 (611h8m48.374071914s) [noise=true , address=77.90.31.12:4590 , piece_id=ZDOZNLYO4WBYJDXGZHCMSDELXFP3YCTUSTTUIDJCKGGWWOEZR5SA, seg_pos={0 0}, upload=535575]: Uploading piece... Dec 31 02:37:48.503270144 (611h8m48.374353189s) [noise=true , address=145.79.227.16:4912 , piece_id=NMT2UHGC5NVTQBJ4TY62RCQJIJME26U4BY6N4UO4YVXKTKGCF3YA, seg_pos={0 0}, upload=535575]: Uploading piece... Dec 31 02:37:50.104117573 (611h8m49.975200608s) [upload=535575]: Next returned nil segment Dec 31 02:37:50.104130627 (611h8m49.975213672s) [upload=535575]: Waiting for error group managing segments... Dec 31 02:37:50.486795518 (611h8m50.357878553s) [noise=true , address=145.79.227.2:4115 , piece_id=RQLXJOQAL5JSULQENFAVGYLDMBUEYZU6SZ23DWYW2MZFEBYGESSQ, seg_pos={0 0}, upload=535575]: Done uploading piece. err: <nil> Dec 31 02:37:50.495766359 (611h8m50.366849394s) [noise=true , address=145.79.227.20:4437 , piece_id=WCPTJFYPMNW3RGQMK4BSLK5S5YCBUQG5OYNEJVC6VEXQDUMQRGJA, seg_pos={0 0}, upload=535575]: Done uploading piece. err: <nil> Dec 31 02:37:50.497844637 (611h8m50.368927672s) [noise=true , address=145.79.227.41:4086 , piece_id=ZBOPXCE47GYTSNZRRY6FXBU425GYHMUEMM2LO6DYODH2MCVKGGTA, seg_pos={0 0}, upload=535575]: Done uploading piece. err: <nil> Dec 31 02:37:50.502023354 (611h8m50.373106399s) [noise=true , address=145.79.227.7:4293 , piece_id=ZWBZUU43YH4G3HE3GRLPWHYWZRSA2LPYBKQEPFKTCXMARJCY64YQ, seg_pos={0 0}, upload=535575]: Done uploading piece. err: <nil> Dec 31 02:37:50.504636680 (611h8m50.375719715s) [noise=true , address=145.79.227.11:4729 , piece_id=JVOMGWKUQLTLGM7BHBJUC6Q2SZ4X74BBL4VUPJCYCX75RKS7JPBA, seg_pos={0 0}, upload=535575]: Done uploading piece. err: <nil> Dec 31 02:37:50.554273063 (611h8m50.425356098s) [noise=true , address=145.79.227.51:4411 , piece_id=IMAZPT7VUBR3T2W2A7ZNRYFU2LXOCSOBTCAMWUQVKJSMHYGZGOAA, seg_pos={0 0}, upload=535575]: Done uploading piece. err: <nil> Dec 31 02:37:50.571499304 (611h8m50.442582339s) [noise=true , address=145.79.227.50:4374 , piece_id=HO244SW4SOZORMMCEYTPGXKZXTI3UAGAF2G6HQ6AG5CWQT6W6FBQ, seg_pos={0 0}, upload=535575]: Done uploading piece. err: <nil> Dec 31 02:37:50.572465444 (611h8m50.443548479s) [noise=true , address=145.79.227.21:4474 , piece_id=HGP2VTBOZ3CZKUYC3RZLAJZCFGDHE6UK4QURCGW7PVLDJNQN24OQ, seg_pos={0 0}, upload=535575]: Done uploading piece. err: <nil> Dec 31 02:37:50.572999320 (611h8m50.444082355s) [noise=true , address=145.79.227.42:4123 , piece_id=VYO7SPUB2M5R4DQQUD5DAQULL5GR75RS4AJI4NEL3A4BFPDPAHAQ, seg_pos={0 0}, upload=535575]: Done uploading piece. err: <nil> Dec 31 02:37:50.674358730 (611h8m50.545441775s) [noise=true , address=77.90.31.13:4627 , piece_id=HU3WYZYADXZMLHUWBVPZZXCTTNNLBYT52NKRQYSAOX3BSPKOFIBQ, seg_pos={0 0}, upload=535575]: Done uploading piece. err: <nil> Dec 31 02:37:50.722960404 (611h8m50.594043439s) [noise=true , address=145.79.227.16:4912 , piece_id=NMT2UHGC5NVTQBJ4TY62RCQJIJME26U4BY6N4UO4YVXKTKGCF3YA, seg_pos={0 0}, upload=535575]: Done uploading piece. err: <nil> Dec 31 02:37:50.835385065 (611h8m50.7064681s) [noise=true , address=45.91.116.6:4976 , piece_id=BQJE2HHQLCZCIGGUTPNPUPMQ5R4FK6Z4CW4HPF6CLEKTQSPUPIFQ, seg_pos={0 0}, upload=535575]: Done uploading piece. err: <nil> Dec 31 02:37:51.020974468 (611h8m50.892057504s) [noise=true , address=45.91.116.9:4262 , piece_id=42EOUVITCXQKYK5S73IF2XK22HWWSWEDQ6UT3Z33F6ETKQHZGFFQ, seg_pos={0 0}, upload=535575]: Done uploading piece. err: <nil> Dec 31 02:37:51.020992613 (611h8m50.892075659s) [seg_pos={0 0}, upload=535575]: Segment reached optimal threshold of 13 pieces. Dec 31 02:37:51.021213103 (611h8m50.892296138s) [noise=true , address=145.79.227.34:4907 , piece_id=JQZ6OUDMO2KDJ6M3FANWJI6LNOJ3Q36CFM4FC45HWP3TATNZQ5UA, seg_pos={0 0}, upload=535575]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 31 02:37:51.021221048 (611h8m50.892304083s) [noise=true , address=145.79.227.34:4907 , piece_id=JQZ6OUDMO2KDJ6M3FANWJI6LNOJ3Q36CFM4FC45HWP3TATNZQ5UA, seg_pos={0 0}, upload=535575]: Long tail cancellation due to optimal threshold Dec 31 02:37:51.021237941 (611h8m50.892320976s) [noise=true , address=77.90.31.12:4590 , piece_id=ZDOZNLYO4WBYJDXGZHCMSDELXFP3YCTUSTTUIDJCKGGWWOEZR5SA, seg_pos={0 0}, upload=535575]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: write tcp 188.166.225.170:53632->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:53632->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53632->77.90.31.12:4590: use of closed network connection Dec 31 02:37:51.021253630 (611h8m50.892336665s) [noise=true , address=77.90.31.12:4590 , piece_id=ZDOZNLYO4WBYJDXGZHCMSDELXFP3YCTUSTTUIDJCKGGWWOEZR5SA, seg_pos={0 0}, upload=535575]: Long tail cancellation due to optimal threshold Dec 31 02:37:51.021457048 (611h8m50.892540093s) [noise=true , address=77.90.31.18:4805 , piece_id=ATIFFJ4HGITKWKKJZR3FCD43GQNGTNYPLJEXAOFH74D3URLE5TGA, seg_pos={0 0}, upload=535575]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: write tcp 188.166.225.170:35108->77.90.31.18:4805: use of closed network connection; write tcp 188.166.225.170:35108->77.90.31.18:4805: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35108->77.90.31.18:4805: use of closed network connection Dec 31 02:37:51.021464251 (611h8m50.892547286s) [noise=true , address=77.90.31.18:4805 , piece_id=ATIFFJ4HGITKWKKJZR3FCD43GQNGTNYPLJEXAOFH74D3URLE5TGA, seg_pos={0 0}, upload=535575]: Long tail cancellation due to optimal threshold Dec 31 02:37:51.021481194 (611h8m50.892564229s) [seg_pos={0 0}, upload=535575]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 02:37:51.021540536 (611h8m50.892623571s) [upload=535575]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 02:37:51.021544423 (611h8m50.892627458s) [upload=535575]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 02:37:51.315085251 (611h8m51.186168316s) [upload=535575]: Done uploading segments... Dec 31 02:37:51.315116230 (611h8m51.186199276s) [upload=535575]: Upload finished. err: <nil> Duration: 30.003771145s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [......................X...............................X....X] duration: 5.001921423s trace: https://jaeger.dev.storj.io/trace/7bc7478624e56c1a FAIL (2025-12-31 02:38:08 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001947012s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.......................X...............................X....] duration: 3.42488854s trace: https://jaeger.dev.storj.io/trace/25cdf0a5222c0faa FAIL (2025-12-31 02:37:46 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.424915091s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 27.478735405s trace: https://jaeger.dev.storj.io/trace/365693fea00df28 FAIL (2025-12-31 02:37:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [noise=true , address=145.79.227.49:4343 , piece_id=6L3HPTS7D2NRA4QEOH4RFM3D3B53CTCONRI6G6YVU4OOTUWTG7XA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.942060847 (610h25m24.813143892s) [noise=true , address=145.79.227.34:4937 , piece_id=5W7LBM3YJJTVUFYTELR57DVXB663OHO5N3Z2SPQPZ4G43IK47O3Q, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.942684104 (610h25m24.813767139s) [noise=true , address=64.49.13.139:4478 , piece_id=QN6RT2KWEA6APQVA66ZMKG4LH537UGYNOXOCVVA2YC2MKDOLT3YA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.943064007 (610h25m24.814147042s) [noise=true , address=145.79.227.40:4084 , piece_id=RUKTKDII6BRUWSYVS65CWUBT3QKG3ASUK3KA3O5VGE53XSG6IMOQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.943459560 (610h25m24.814542605s) [noise=true , address=145.79.227.14:4847 , piece_id=JX5NXTMVM3NSWIEVR4RX46ABSNPKTOFDLFBPKEXFQ3G3ANKV5OLA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.943865303 (610h25m24.814948348s) [noise=true , address=45.91.116.6:4981 , piece_id=CT474ZA2LVAZEBJHBDD4GB6NTHVHZFJLNE5OZV3DFS2NZKVW3EQQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.944259694 (610h25m24.815342749s) [noise=true , address=145.79.227.51:4443 , piece_id=73UUMRDY7BSZNCGBXCZYYRBVE5KLPMNPRNXOZEGWKQGLHJS32J2Q, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.946575182 (610h25m24.817658227s) [noise=true , address=77.90.31.10:4523 , piece_id=P6LIONXJI6AFZ2JTKCQ2W6RZCOZCK6MZ53EXAJKXYQXSX2PX4QEA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.948132909 (610h25m24.819215944s) [noise=true , address=145.79.227.20:4452 , piece_id=ZSDN2JZ3FQUVJYN5OURDXNKEDLDNULJOHKKRZVIPXZIWIYAHCRXQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.949605774 (610h25m24.820688819s) [noise=true , address=77.90.31.14:4680 , piece_id=S2E4WBA7MJVAZLT5JG3IMHK53RQC77CCHV2PDF6L2R2JEA2U3K4Q, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.951459002 (610h25m24.822542048s) [noise=true , address=193.149.164.143:4704 , piece_id=BHXHFBKTLTDMGXWY5QRTP2THZQE5OXFZ2LCMTO2KRYRWAHXWW3NQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.952986742 (610h25m24.824069777s) [noise=true , address=77.90.31.19:4848 , piece_id=VZYG32S6ODIPDKQYZRXECBHXOERZDDWD2OF35Z46W4XMUZ3OJLAA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.956916654 (610h25m24.827999689s) [noise=true , address=145.79.227.29:4971 , piece_id=MEPXHHOQ6SUZC3GIPCQEMOKKF5OJTKZEI3UYZBAYXAQZGSVEKSBA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.960665062 (610h25m24.831748097s) [noise=true , address=64.49.13.170:4720 , piece_id=HIATRSAPE4YLSODFZDPP43ZUMLDVM7GJEVMWE4ZGUQTP25LTHGTA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.962366001 (610h25m24.833449036s) [noise=true , address=145.79.227.2:4117 , piece_id=4SGFGB3IJ7LTXDMMWM2PGGOV64MC46IZ7OZQWH55J4NBQY7CHMZQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.963708216 (610h25m24.834791261s) [noise=true , address=145.79.227.35:4505 , piece_id=V7RS7A4QLZTH4G32ZUZNSF2Z5APVWZGJO77NEGP3SBWQT3WCMVLQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:26.352236138 (610h25m26.223319173s) [upload=534952]: Next returned nil segment Dec 31 01:54:26.352253821 (610h25m26.223336857s) [upload=534952]: Waiting for error group managing segments... Dec 31 01:54:26.812758374 (610h25m26.683841409s) [noise=true , address=145.79.227.51:4443 , piece_id=73UUMRDY7BSZNCGBXCZYYRBVE5KLPMNPRNXOZEGWKQGLHJS32J2Q, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.824371056 (610h25m26.695454101s) [noise=true , address=145.79.227.14:4847 , piece_id=JX5NXTMVM3NSWIEVR4RX46ABSNPKTOFDLFBPKEXFQ3G3ANKV5OLA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.838414046 (610h25m26.709497081s) [noise=true , address=64.49.13.138:4490 , piece_id=2XTUZE6PXRE3OARNJVT36FXXWUMYIBVNE6M2VVFAVBGKEE2H5YZQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.875499841 (610h25m26.746582876s) [noise=true , address=64.49.13.139:4478 , piece_id=QN6RT2KWEA6APQVA66ZMKG4LH537UGYNOXOCVVA2YC2MKDOLT3YA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.891607221 (610h25m26.762690256s) [noise=true , address=145.79.227.20:4452 , piece_id=ZSDN2JZ3FQUVJYN5OURDXNKEDLDNULJOHKKRZVIPXZIWIYAHCRXQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.988561223 (610h25m26.859644268s) [noise=true , address=64.49.13.170:4720 , piece_id=HIATRSAPE4YLSODFZDPP43ZUMLDVM7GJEVMWE4ZGUQTP25LTHGTA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.039035911 (610h25m26.910118946s) [noise=true , address=77.90.31.14:4680 , piece_id=S2E4WBA7MJVAZLT5JG3IMHK53RQC77CCHV2PDF6L2R2JEA2U3K4Q, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.098023090 (610h25m26.969106126s) [noise=true , address=45.91.116.6:4981 , piece_id=CT474ZA2LVAZEBJHBDD4GB6NTHVHZFJLNE5OZV3DFS2NZKVW3EQQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.653790754 (610h25m27.524873789s) [noise=true , address=77.90.31.21:4886 , piece_id=K34WPIBGGGEPHIKJ253XGLDCIQ3WLFN4AZ6EYNWVHHT2EIHS6YRQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.684192972 (610h25m27.555276017s) [noise=true , address=193.149.164.143:4704 , piece_id=BHXHFBKTLTDMGXWY5QRTP2THZQE5OXFZ2LCMTO2KRYRWAHXWW3NQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.717238425 (610h25m27.58832146s) [noise=true , address=145.79.227.40:4084 , piece_id=RUKTKDII6BRUWSYVS65CWUBT3QKG3ASUK3KA3O5VGE53XSG6IMOQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.718349742 (610h25m27.589432777s) [noise=true , address=77.90.31.10:4523 , piece_id=P6LIONXJI6AFZ2JTKCQ2W6RZCOZCK6MZ53EXAJKXYQXSX2PX4QEA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.803639562 (610h25m27.674722597s) [noise=true , address=145.79.227.29:4971 , piece_id=MEPXHHOQ6SUZC3GIPCQEMOKKF5OJTKZEI3UYZBAYXAQZGSVEKSBA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.812791279 (610h25m27.683874314s) [noise=true , address=145.79.227.49:4343 , piece_id=6L3HPTS7D2NRA4QEOH4RFM3D3B53CTCONRI6G6YVU4OOTUWTG7XA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.812812348 (610h25m27.683895393s) [seg_pos={0 0}, upload=534952]: Segment reached optimal threshold of 14 pieces. Dec 31 01:54:27.813018000 (610h25m27.684101035s) [noise=true , address=145.79.227.35:4505 , piece_id=V7RS7A4QLZTH4G32ZUZNSF2Z5APVWZGJO77NEGP3SBWQT3WCMVLQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i6itGoBhLsLiqC8et8Y5L62fLjcWxy18u39NAr4R3nXNMWcsf): protocol: expected piece hash; context canceled; EOF Dec 31 01:54:27.813024772 (610h25m27.684107808s) [noise=true , address=145.79.227.35:4505 , piece_id=V7RS7A4QLZTH4G32ZUZNSF2Z5APVWZGJO77NEGP3SBWQT3WCMVLQ, seg_pos={0 0}, upload=534952]: Long tail cancellation due to optimal threshold Dec 31 01:54:27.813175931 (610h25m27.684258976s) [noise=true , address=145.79.227.34:4937 , piece_id=5W7LBM3YJJTVUFYTELR57DVXB663OHO5N3Z2SPQPZ4G43IK47O3Q, seg_pos={0 0}, upload=534952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1LKrFThdW3jJpG6dRZLUaxLXTa5Wy27yQ8PGUEqjvBWfFoRB94): protocol: expected piece hash; context canceled; EOF Dec 31 01:54:27.813181702 (610h25m27.684264737s) [noise=true , address=145.79.227.34:4937 , piece_id=5W7LBM3YJJTVUFYTELR57DVXB663OHO5N3Z2SPQPZ4G43IK47O3Q, seg_pos={0 0}, upload=534952]: Long tail cancellation due to optimal threshold Dec 31 01:54:27.813407862 (610h25m27.684490907s) [noise=true , address=145.79.227.2:4117 , piece_id=4SGFGB3IJ7LTXDMMWM2PGGOV64MC46IZ7OZQWH55J4NBQY7CHMZQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122Htg72f5eUUutf1p3fYQpy1Wkf6AgdaQV83HHYye2n8zm3the): protocol: expected piece hash; context canceled; EOF Dec 31 01:54:27.813414264 (610h25m27.684497299s) [noise=true , address=145.79.227.2:4117 , piece_id=4SGFGB3IJ7LTXDMMWM2PGGOV64MC46IZ7OZQWH55J4NBQY7CHMZQ, seg_pos={0 0}, upload=534952]: Long tail cancellation due to optimal threshold Dec 31 01:54:27.813424614 (610h25m27.684507649s) [noise=true , address=77.90.31.19:4848 , piece_id=VZYG32S6ODIPDKQYZRXECBHXOERZDDWD2OF35Z46W4XMUZ3OJLAA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD): protocol: write tcp 188.166.225.170:42842->77.90.31.19:4848: use of closed network connection; write tcp 188.166.225.170:42842->77.90.31.19:4848: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42842->77.90.31.19:4848: use of closed network connection Dec 31 01:54:27.813432750 (610h25m27.684515785s) [noise=true , address=77.90.31.19:4848 , piece_id=VZYG32S6ODIPDKQYZRXECBHXOERZDDWD2OF35Z46W4XMUZ3OJLAA, seg_pos={0 0}, upload=534952]: Long tail cancellation due to optimal threshold Dec 31 01:54:27.813474088 (610h25m27.684557123s) [seg_pos={0 0}, upload=534952]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 01:54:27.813535325 (610h25m27.68461836s) [upload=534952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 01:54:27.813539613 (610h25m27.684622648s) [upload=534952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 01:54:28.099921034 (610h25m27.971004069s) [upload=534952]: Done uploading segments... Dec 31 01:54:28.099940451 (610h25m27.971023486s) [upload=534952]: Upload finished. err: <nil> Duration: 27.478764671s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [......X..X.......X.X..X....XX.XX........X.X......X..X......X] duration: 30.002319374s trace: https://jaeger.dev.storj.io/trace/7c13230eedb85879 FAIL (2025-12-31 02:37:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: JAFYDUW6T5KQZFMW3LGBBHHJG6IO5QWN57ZX6EZN633BZPBE25NQ) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: J6XFQSZLVSFLXNCMDHR7OHXRP4SCWENMM4D5WF2IOMMUDF5BZIBA) context deadline exceeded ======LOGS====== Dec 31 02:36:54.474386358 (611h7m54.345469393s) [upload=535561]: Starting upload Dec 31 02:36:54.474457724 (611h7m54.345540769s) [upload=535561]: Done starting upload Dec 31 02:36:54.474717939 (611h7m54.345800974s) [upload=535561]: Uploading segments... Dec 31 02:36:54.474761973 (611h7m54.345845008s) [upload=535561]: Got next segment. Inline: false Dec 31 02:36:54.474795778 (611h7m54.345878813s) [upload=535561]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 02:36:54.474800627 (611h7m54.345883672s) [upload=535561]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 02:36:54.722846897 (611h7m54.593929932s) [seg_pos={0 0}, upload=535561]: Begin upload segment... Dec 31 02:36:54.722895820 (611h7m54.593978855s) [seg_pos={0 0}, upload=535561]: Done begin upload segment. Dec 31 02:36:54.722957177 (611h7m54.594040222s) [noise=true , address=145.79.227.11:4731 , piece_id=EHYPEMBIKBZ6U6XNHALGEMYYVG2RLXGXEC4P7IM3RAZHQJAFCUEA, seg_pos={0 0}, upload=535561]: Uploading piece... Dec 31 02:36:54.722966174 (611h7m54.594049219s) [noise=true , address=145.79.227.27:4691 , piece_id=FXZ5KNHGTP7R3FP7D46J4LUI24GGKTR6QMHWSAZY472VRZIDASSA, seg_pos={0 0}, upload=535561]: Uploading piece... Dec 31 02:36:54.723670927 (611h7m54.594753962s) [noise=true , address=145.79.227.35:4482 , piece_id=SVSWNUY4G4UJJVRIJZLCCQLJVNQJF4HHRMFZZIQ5WESGHGHVFQVQ, seg_pos={0 0}, upload=535561]: Uploading piece... Dec 31 02:36:54.724618721 (611h7m54.595701776s) [noise=true , address=77.90.31.19:4842 , piece_id=ESF2XEX63YE5Q55DW5YIWOXTKXZT5L36WG2JEZPQLYK2SF2IJTRA, seg_pos={0 0}, upload=535561]: Uploading piece... Dec 31 02:36:54.725119695 (611h7m54.59620274s) [noise=true , address=145.79.227.3:4149 , piece_id=A4MKUQDBUXHT2CLQNFMQGOH3CEJS6GCY7CWNID32MRLANRSZPORA, seg_pos={0 0}, upload=535561]: Uploading piece... Dec 31 02:36:54.726610283 (611h7m54.597693318s) [noise=true , address=77.90.31.14:4663 , piece_id=QUZAFFV7I3VWFJ5LXNBU2FJ3WYGKYU7GF6Z4ZWA6GP62S2SJN6PQ, seg_pos={0 0}, upload=535561]: Uploading piece... Dec 31 02:36:54.727173646 (611h7m54.598256691s) [noise=true , address=145.79.227.8:4330 , piece_id=5TDZ3FAK3VAVQFHCYK4Q63RRI3LXFXPDD3CNJKKLLEMJX6K37LUA, seg_pos={0 0}, upload=535561]: Uploading piece... Dec 31 02:36:54.727967217 (611h7m54.599050262s) [noise=true , address=145.79.227.17:4970 , piece_id=BOSJUUQ2OWBVC7JC7QYFIMJA72YZHWDHDVMCF6CDVXE5UNHPHCLQ, seg_pos={0 0}, upload=535561]: Uploading piece... Dec 31 02:36:54.728941132 (611h7m54.600024177s) [noise=true , address=45.91.116.11:4117 , piece_id=J6XFQSZLVSFLXNCMDHR7OHXRP4SCWENMM4D5WF2IOMMUDF5BZIBA, seg_pos={0 0}, upload=535561]: Uploading piece... Dec 31 02:36:54.731058773 (611h7m54.602141838s) [noise=true , address=145.79.227.45:4230 , piece_id=MDQBQH7ODVAJQZVGHRC5LMYA3PG36QWA7LYWDN4TAKWEX5HENTAQ, seg_pos={0 0}, upload=535561]: Uploading piece... Dec 31 02:36:54.734295837 (611h7m54.605378882s) [noise=true , address=145.79.227.28:4726 , piece_id=MTPY6PFNO2ENVYZAWBQL3TDQQNG4RDOVK4UZFFMJHR564ZNER2TA, seg_pos={0 0}, upload=535561]: Uploading piece... Dec 31 02:36:54.738705504 (611h7m54.609788539s) [noise=true , address=145.79.227.7:4295 , piece_id=D3NH2KVPLLGLM2FKB3SYFYVB7JDDUBUBR6LWHDIQC3ERJ45QBGDQ, seg_pos={0 0}, upload=535561]: Uploading piece... Dec 31 02:36:54.741680539 (611h7m54.612763574s) [noise=true , address=145.79.227.19:4078 , piece_id=PXRAS444MVUKJHWTG4QNCQ6GCUNK3SYGOSYTBEPRM7QHVKXMWKTA, seg_pos={0 0}, upload=535561]: Uploading piece... Dec 31 02:36:54.744327609 (611h7m54.615410644s) [noise=true , address=145.79.227.47:4265 , piece_id=G4I5LTQRZOANZVGTPNWO3RBFJYIEMVZNHVV3QVEHECVPCJXP3SFA, seg_pos={0 0}, upload=535561]: Uploading piece... Dec 31 02:36:54.745708318 (611h7m54.616791363s) [noise=true , address=145.79.227.41:4085 , piece_id=ZDEXH2MXTKHH2WNLXFGZLDHPIJAI47F4VIDTFIYNVUOTMOPZ6HUA, seg_pos={0 0}, upload=535561]: Uploading piece... Dec 31 02:36:54.747317933 (611h7m54.618400978s) [noise=true , address=45.91.116.8:4047 , piece_id=JAFYDUW6T5KQZFMW3LGBBHHJG6IO5QWN57ZX6EZN633BZPBE25NQ, seg_pos={0 0}, upload=535561]: Uploading piece... Dec 31 02:36:56.432441173 (611h7m56.303524208s) [upload=535561]: Next returned nil segment Dec 31 02:36:56.432454648 (611h7m56.303537683s) [upload=535561]: Waiting for error group managing segments... Dec 31 02:36:56.712091020 (611h7m56.583174055s) [noise=true , address=145.79.227.11:4731 , piece_id=EHYPEMBIKBZ6U6XNHALGEMYYVG2RLXGXEC4P7IM3RAZHQJAFCUEA, seg_pos={0 0}, upload=535561]: Done uploading piece. err: <nil> Dec 31 02:36:56.714771655 (611h7m56.58585469s) [noise=true , address=145.79.227.8:4330 , piece_id=5TDZ3FAK3VAVQFHCYK4Q63RRI3LXFXPDD3CNJKKLLEMJX6K37LUA, seg_pos={0 0}, upload=535561]: Done uploading piece. err: <nil> Dec 31 02:36:56.732856279 (611h7m56.603939324s) [noise=true , address=145.79.227.7:4295 , piece_id=D3NH2KVPLLGLM2FKB3SYFYVB7JDDUBUBR6LWHDIQC3ERJ45QBGDQ, seg_pos={0 0}, upload=535561]: Done uploading piece. err: <nil> Dec 31 02:36:56.734017240 (611h7m56.605100275s) [noise=true , address=145.79.227.19:4078 , piece_id=PXRAS444MVUKJHWTG4QNCQ6GCUNK3SYGOSYTBEPRM7QHVKXMWKTA, seg_pos={0 0}, upload=535561]: Done uploading piece. err: <nil> Dec 31 02:36:56.735258944 (611h7m56.606341979s) [noise=true , address=145.79.227.41:4085 , piece_id=ZDEXH2MXTKHH2WNLXFGZLDHPIJAI47F4VIDTFIYNVUOTMOPZ6HUA, seg_pos={0 0}, upload=535561]: Done uploading piece. err: <nil> Dec 31 02:36:56.735855711 (611h7m56.606938746s) [noise=true , address=145.79.227.45:4230 , piece_id=MDQBQH7ODVAJQZVGHRC5LMYA3PG36QWA7LYWDN4TAKWEX5HENTAQ, seg_pos={0 0}, upload=535561]: Done uploading piece. err: <nil> Dec 31 02:36:56.743693815 (611h7m56.614776851s) [noise=true , address=145.79.227.27:4691 , piece_id=FXZ5KNHGTP7R3FP7D46J4LUI24GGKTR6QMHWSAZY472VRZIDASSA, seg_pos={0 0}, upload=535561]: Done uploading piece. err: <nil> Dec 31 02:36:56.747174012 (611h7m56.618257057s) [noise=true , address=145.79.227.3:4149 , piece_id=A4MKUQDBUXHT2CLQNFMQGOH3CEJS6GCY7CWNID32MRLANRSZPORA, seg_pos={0 0}, upload=535561]: Done uploading piece. err: <nil> Dec 31 02:36:56.800716794 (611h7m56.671799839s) [noise=true , address=145.79.227.28:4726 , piece_id=MTPY6PFNO2ENVYZAWBQL3TDQQNG4RDOVK4UZFFMJHR564ZNER2TA, seg_pos={0 0}, upload=535561]: Done uploading piece. err: <nil> Dec 31 02:36:56.815362601 (611h7m56.686445636s) [noise=true , address=145.79.227.17:4970 , piece_id=BOSJUUQ2OWBVC7JC7QYFIMJA72YZHWDHDVMCF6CDVXE5UNHPHCLQ, seg_pos={0 0}, upload=535561]: Done uploading piece. err: <nil> Dec 31 02:36:57.054942240 (611h7m56.926025275s) [noise=true , address=45.91.116.11:4117 , piece_id=J6XFQSZLVSFLXNCMDHR7OHXRP4SCWENMM4D5WF2IOMMUDF5BZIBA, seg_pos={0 0}, upload=535561]: Done uploading piece. err: <nil> Dec 31 02:36:57.069210358 (611h7m56.940293403s) [noise=true , address=145.79.227.47:4265 , piece_id=G4I5LTQRZOANZVGTPNWO3RBFJYIEMVZNHVV3QVEHECVPCJXP3SFA, seg_pos={0 0}, upload=535561]: Done uploading piece. err: <nil> Dec 31 02:36:57.478755602 (611h7m57.349838667s) [noise=true , address=45.91.116.8:4047 , piece_id=JAFYDUW6T5KQZFMW3LGBBHHJG6IO5QWN57ZX6EZN633BZPBE25NQ, seg_pos={0 0}, upload=535561]: Done uploading piece. err: <nil> Dec 31 02:36:57.478781511 (611h7m57.349864546s) [seg_pos={0 0}, upload=535561]: Segment reached optimal threshold of 13 pieces. Dec 31 02:36:57.479137730 (611h7m57.350220785s) [noise=true , address=145.79.227.35:4482 , piece_id=SVSWNUY4G4UJJVRIJZLCCQLJVNQJF4HHRMFZZIQ5WESGHGHVFQVQ, seg_pos={0 0}, upload=535561]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: expected piece hash; context canceled; EOF Dec 31 02:36:57.479148680 (611h7m57.350231725s) [noise=true , address=145.79.227.35:4482 , piece_id=SVSWNUY4G4UJJVRIJZLCCQLJVNQJF4HHRMFZZIQ5WESGHGHVFQVQ, seg_pos={0 0}, upload=535561]: Long tail cancellation due to optimal threshold Dec 31 02:36:57.479310127 (611h7m57.350393182s) [noise=true , address=77.90.31.19:4842 , piece_id=ESF2XEX63YE5Q55DW5YIWOXTKXZT5L36WG2JEZPQLYK2SF2IJTRA, seg_pos={0 0}, upload=535561]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Dec 31 02:36:57.479318213 (611h7m57.350401248s) [noise=true , address=77.90.31.19:4842 , piece_id=ESF2XEX63YE5Q55DW5YIWOXTKXZT5L36WG2JEZPQLYK2SF2IJTRA, seg_pos={0 0}, upload=535561]: Long tail cancellation due to optimal threshold Dec 31 02:36:57.479567718 (611h7m57.350650773s) [noise=true , address=77.90.31.14:4663 , piece_id=QUZAFFV7I3VWFJ5LXNBU2FJ3WYGKYU7GF6Z4ZWA6GP62S2SJN6PQ, seg_pos={0 0}, upload=535561]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: read tcp 188.166.225.170:37154->77.90.31.14:4663: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 31 02:36:57.479577406 (611h7m57.350660451s) [noise=true , address=77.90.31.14:4663 , piece_id=QUZAFFV7I3VWFJ5LXNBU2FJ3WYGKYU7GF6Z4ZWA6GP62S2SJN6PQ, seg_pos={0 0}, upload=535561]: Long tail cancellation due to optimal threshold Dec 31 02:36:57.479601382 (611h7m57.350684417s) [seg_pos={0 0}, upload=535561]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 02:36:57.479691032 (611h7m57.350774088s) [upload=535561]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 02:36:57.479697536 (611h7m57.350780601s) [upload=535561]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 02:36:57.775960295 (611h7m57.647043331s) [upload=535561]: Done uploading segments... Dec 31 02:36:57.775977398 (611h7m57.647060433s) [upload=535561]: Upload finished. err: <nil> Duration: 30.002346836s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [........................X...............................X...] duration: 3.558586901s trace: https://jaeger.dev.storj.io/trace/25cdf0a5222c0faa FAIL (2025-12-31 02:37:06 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.558611948s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.......X..X.......X.X..X....XX.XX........X.X......X..X......] duration: 5.841912396s trace: https://jaeger.dev.storj.io/trace/54231a38e027f2fe FAIL (2025-12-31 02:36:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: DCML4BWFWKJ2AEQVRGRYOQ7Q4YY3BF2PF4GFQSOREMAG67M4BJ5Q) context deadline exceeded; piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: YBK3PD3XVPSHJZTXOEGCXAGKJFRBLUSTFH6ZHYEFYK5YRIMLGISQ) context deadline exceeded ======LOGS====== Dec 31 02:31:42.009467393 (611h2m41.880550439s) [upload=535486]: Starting upload Dec 31 02:31:42.009568907 (611h2m41.880651942s) [upload=535486]: Done starting upload Dec 31 02:31:42.009677694 (611h2m41.880760729s) [upload=535486]: Uploading segments... Dec 31 02:31:42.009860923 (611h2m41.880943969s) [upload=535486]: Got next segment. Inline: false Dec 31 02:31:42.009921478 (611h2m41.881004513s) [upload=535486]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 02:31:42.009926968 (611h2m41.881010003s) [upload=535486]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 02:31:42.257755154 (611h2m42.128838189s) [seg_pos={0 0}, upload=535486]: Begin upload segment... Dec 31 02:31:42.257856827 (611h2m42.128939862s) [seg_pos={0 0}, upload=535486]: Done begin upload segment. Dec 31 02:31:42.257895601 (611h2m42.128978646s) [noise=true , address=45.91.116.6:4973 , piece_id=BVEV2ZTEUP3UUZ7H6WBZ4MHY2JESHTR6Z2HY3AAD5XCRWE7NLMMQ, seg_pos={0 0}, upload=535486]: Uploading piece... Dec 31 02:31:42.257927502 (611h2m42.129010537s) [noise=true , address=77.90.31.19:4844 , piece_id=DJPEVJBHVIZGFMBSUBTEWEIMU3JCSWPSKUXWPDQGXJKXLFGXQQGA, seg_pos={0 0}, upload=535486]: Uploading piece... Dec 31 02:31:42.258733707 (611h2m42.129816752s) [noise=true , address=145.79.227.52:4447 , piece_id=65XJCKQL2VIDCHMSHGUSCFNPU7JWFPKRF2QXWNAHQT7T5GELABFQ, seg_pos={0 0}, upload=535486]: Uploading piece... Dec 31 02:31:42.259561023 (611h2m42.130644058s) [noise=true , address=45.91.116.9:4262 , piece_id=DCML4BWFWKJ2AEQVRGRYOQ7Q4YY3BF2PF4GFQSOREMAG67M4BJ5Q, seg_pos={0 0}, upload=535486]: Uploading piece... Dec 31 02:31:42.259700759 (611h2m42.130783794s) [noise=true , address=45.91.116.11:4118 , piece_id=JAUSAGDUAQ4BBYWNA7OSW6XATK2JLMRUM6K6QNHD5EB4DPZNPQZA, seg_pos={0 0}, upload=535486]: Uploading piece... Dec 31 02:31:42.260044383 (611h2m42.131127428s) [noise=true , address=45.91.116.12:4225 , piece_id=YBK3PD3XVPSHJZTXOEGCXAGKJFRBLUSTFH6ZHYEFYK5YRIMLGISQ, seg_pos={0 0}, upload=535486]: Uploading piece... Dec 31 02:31:42.260541770 (611h2m42.131624815s) [noise=true , address=45.91.116.5:4299 , piece_id=GZH33CCTCZPZX5ZPHEXDKBI2YSUP4GZ7YUQ6FIIHLCBB24SRSFGA, seg_pos={0 0}, upload=535486]: Uploading piece... Dec 31 02:31:42.261116324 (611h2m42.132199369s) [noise=true , address=145.79.227.18:4007 , piece_id=HA2C5LN5ZB6TVJBQ6GVELU4JQG5KII7CFBTZAHDT75KJP5J7FOYA, seg_pos={0 0}, upload=535486]: Uploading piece... Dec 31 02:31:42.262541628 (611h2m42.133624674s) [noise=true , address=145.79.227.15:4876 , piece_id=WCIBKRCJCRS4JGFTWDQHVUZ2TLV3OYKTSMIBSBSU4WBFZ7WZNHOA, seg_pos={0 0}, upload=535486]: Uploading piece... Dec 31 02:31:42.262838263 (611h2m42.133921308s) [noise=true , address=145.79.227.3:4150 , piece_id=GKSGSQWFKTLK4B4ARKBXK2FR5IDBCOJMZZUOFEI2KASMXPJUHI4A, seg_pos={0 0}, upload=535486]: Uploading piece... Dec 31 02:31:42.263683141 (611h2m42.134766186s) [noise=true , address=77.90.31.10:4519 , piece_id=CZ5AVVMI74GHKCHLXL47OQED5DAR66LRFIVH25A3JL6K7NXNAOSA, seg_pos={0 0}, upload=535486]: Uploading piece... Dec 31 02:31:42.263989505 (611h2m42.13507254s) [noise=true , address=145.79.227.14:4837 , piece_id=HIU57ZYKPL6JYTQB27K2XG67GGYG7XM63OCJ4YLMMQXMLKYDCFJA, seg_pos={0 0}, upload=535486]: Uploading piece... Dec 31 02:31:42.264792113 (611h2m42.135875158s) [noise=true , address=145.79.227.20:4438 , piece_id=MNT67N5BTYQB2JUNXV7TVSRL3DJMMTBXYQO4BSWEFALCWMK2ZP5Q, seg_pos={0 0}, upload=535486]: Uploading piece... Dec 31 02:31:42.265209829 (611h2m42.136292864s) [noise=true , address=145.79.227.39:4014 , piece_id=R3DDMHHRZSHJUDSEXG3BUD2SOIVBXYAJE6RRMTPKW6XAPT7CNLTQ, seg_pos={0 0}, upload=535486]: Uploading piece... Dec 31 02:31:42.266296698 (611h2m42.137379733s) [noise=true , address=145.79.227.24:4581 , piece_id=OPPGQUC2NXAGBLYXH5VM4RGY6ATR5DAUJAGRRKAQCZ3ENMNTMLBA, seg_pos={0 0}, upload=535486]: Uploading piece... Dec 31 02:31:42.270084160 (611h2m42.141167195s) [noise=true , address=145.79.227.21:4474 , piece_id=HBUEUQTYMITUOKLQEQHAOSCBHCVXCWGCUM4WDPC4ZMFKRQCW3GRA, seg_pos={0 0}, upload=535486]: Uploading piece... Dec 31 02:31:43.955549810 (611h2m43.826632865s) [upload=535486]: Next returned nil segment Dec 31 02:31:43.955567964 (611h2m43.826651009s) [upload=535486]: Waiting for error group managing segments... Dec 31 02:31:44.239121474 (611h2m44.110204519s) [noise=true , address=145.79.227.18:4007 , piece_id=HA2C5LN5ZB6TVJBQ6GVELU4JQG5KII7CFBTZAHDT75KJP5J7FOYA, seg_pos={0 0}, upload=535486]: Done uploading piece. err: <nil> Dec 31 02:31:44.247474961 (611h2m44.118557996s) [noise=true , address=145.79.227.24:4581 , piece_id=OPPGQUC2NXAGBLYXH5VM4RGY6ATR5DAUJAGRRKAQCZ3ENMNTMLBA, seg_pos={0 0}, upload=535486]: Done uploading piece. err: <nil> Dec 31 02:31:44.249933482 (611h2m44.121016527s) [noise=true , address=145.79.227.39:4014 , piece_id=R3DDMHHRZSHJUDSEXG3BUD2SOIVBXYAJE6RRMTPKW6XAPT7CNLTQ, seg_pos={0 0}, upload=535486]: Done uploading piece. err: <nil> Dec 31 02:31:44.250394820 (611h2m44.121477855s) [noise=true , address=145.79.227.52:4447 , piece_id=65XJCKQL2VIDCHMSHGUSCFNPU7JWFPKRF2QXWNAHQT7T5GELABFQ, seg_pos={0 0}, upload=535486]: Done uploading piece. err: <nil> Dec 31 02:31:44.260864926 (611h2m44.131947971s) [noise=true , address=145.79.227.3:4150 , piece_id=GKSGSQWFKTLK4B4ARKBXK2FR5IDBCOJMZZUOFEI2KASMXPJUHI4A, seg_pos={0 0}, upload=535486]: Done uploading piece. err: <nil> Dec 31 02:31:44.265136891 (611h2m44.136219936s) [noise=true , address=145.79.227.15:4876 , piece_id=WCIBKRCJCRS4JGFTWDQHVUZ2TLV3OYKTSMIBSBSU4WBFZ7WZNHOA, seg_pos={0 0}, upload=535486]: Done uploading piece. err: <nil> Dec 31 02:31:44.274014796 (611h2m44.145097831s) [noise=true , address=145.79.227.20:4438 , piece_id=MNT67N5BTYQB2JUNXV7TVSRL3DJMMTBXYQO4BSWEFALCWMK2ZP5Q, seg_pos={0 0}, upload=535486]: Done uploading piece. err: <nil> Dec 31 02:31:44.275430611 (611h2m44.146513646s) [noise=true , address=145.79.227.21:4474 , piece_id=HBUEUQTYMITUOKLQEQHAOSCBHCVXCWGCUM4WDPC4ZMFKRQCW3GRA, seg_pos={0 0}, upload=535486]: Done uploading piece. err: <nil> Dec 31 02:31:44.276265651 (611h2m44.147348686s) [noise=true , address=145.79.227.14:4837 , piece_id=HIU57ZYKPL6JYTQB27K2XG67GGYG7XM63OCJ4YLMMQXMLKYDCFJA, seg_pos={0 0}, upload=535486]: Done uploading piece. err: <nil> Dec 31 02:31:44.555968520 (611h2m44.427051555s) [noise=true , address=45.91.116.12:4225 , piece_id=YBK3PD3XVPSHJZTXOEGCXAGKJFRBLUSTFH6ZHYEFYK5YRIMLGISQ, seg_pos={0 0}, upload=535486]: Done uploading piece. err: <nil> Dec 31 02:31:44.563093667 (611h2m44.434176702s) [noise=true , address=77.90.31.10:4519 , piece_id=CZ5AVVMI74GHKCHLXL47OQED5DAR66LRFIVH25A3JL6K7NXNAOSA, seg_pos={0 0}, upload=535486]: Done uploading piece. err: <nil> Dec 31 02:31:44.625636847 (611h2m44.496719882s) [noise=true , address=45.91.116.9:4262 , piece_id=DCML4BWFWKJ2AEQVRGRYOQ7Q4YY3BF2PF4GFQSOREMAG67M4BJ5Q, seg_pos={0 0}, upload=535486]: Done uploading piece. err: <nil> Dec 31 02:31:44.717693032 (611h2m44.588776077s) [noise=true , address=45.91.116.6:4973 , piece_id=BVEV2ZTEUP3UUZ7H6WBZ4MHY2JESHTR6Z2HY3AAD5XCRWE7NLMMQ, seg_pos={0 0}, upload=535486]: Done uploading piece. err: <nil> Dec 31 02:31:44.717715566 (611h2m44.588798611s) [seg_pos={0 0}, upload=535486]: Segment reached optimal threshold of 13 pieces. Dec 31 02:31:44.718014545 (611h2m44.58909759s) [noise=true , address=45.91.116.11:4118 , piece_id=JAUSAGDUAQ4BBYWNA7OSW6XATK2JLMRUM6K6QNHD5EB4DPZNPQZA, seg_pos={0 0}, upload=535486]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF Dec 31 02:31:44.718024604 (611h2m44.589107639s) [noise=true , address=45.91.116.11:4118 , piece_id=JAUSAGDUAQ4BBYWNA7OSW6XATK2JLMRUM6K6QNHD5EB4DPZNPQZA, seg_pos={0 0}, upload=535486]: Long tail cancellation due to optimal threshold Dec 31 02:31:44.718138050 (611h2m44.589221125s) [noise=true , address=77.90.31.19:4844 , piece_id=DJPEVJBHVIZGFMBSUBTEWEIMU3JCSWPSKUXWPDQGXJKXLFGXQQGA, seg_pos={0 0}, upload=535486]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF Dec 31 02:31:44.718145925 (611h2m44.58922897s) [noise=true , address=77.90.31.19:4844 , piece_id=DJPEVJBHVIZGFMBSUBTEWEIMU3JCSWPSKUXWPDQGXJKXLFGXQQGA, seg_pos={0 0}, upload=535486]: Long tail cancellation due to optimal threshold Dec 31 02:31:44.718182945 (611h2m44.58926598s) [noise=true , address=45.91.116.5:4299 , piece_id=GZH33CCTCZPZX5ZPHEXDKBI2YSUP4GZ7YUQ6FIIHLCBB24SRSFGA, seg_pos={0 0}, upload=535486]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Dec 31 02:31:44.718191292 (611h2m44.589274327s) [noise=true , address=45.91.116.5:4299 , piece_id=GZH33CCTCZPZX5ZPHEXDKBI2YSUP4GZ7YUQ6FIIHLCBB24SRSFGA, seg_pos={0 0}, upload=535486]: Long tail cancellation due to optimal threshold Dec 31 02:31:44.718248981 (611h2m44.589332016s) [seg_pos={0 0}, upload=535486]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 02:31:44.718376454 (611h2m44.589459519s) [upload=535486]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 02:31:44.718437000 (611h2m44.589520045s) [upload=535486]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 02:31:45.017559155 (611h2m44.8886422s) [upload=535486]: Done uploading segments... Dec 31 02:31:45.017585174 (611h2m44.888668209s) [upload=535486]: Upload finished. err: <nil> Duration: 5.841944307s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 15.370704401s trace: https://jaeger.dev.storj.io/trace/365693fea00df28 FAIL (2025-12-31 02:36:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [noise=true , address=145.79.227.49:4343 , piece_id=6L3HPTS7D2NRA4QEOH4RFM3D3B53CTCONRI6G6YVU4OOTUWTG7XA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.942060847 (610h25m24.813143892s) [noise=true , address=145.79.227.34:4937 , piece_id=5W7LBM3YJJTVUFYTELR57DVXB663OHO5N3Z2SPQPZ4G43IK47O3Q, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.942684104 (610h25m24.813767139s) [noise=true , address=64.49.13.139:4478 , piece_id=QN6RT2KWEA6APQVA66ZMKG4LH537UGYNOXOCVVA2YC2MKDOLT3YA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.943064007 (610h25m24.814147042s) [noise=true , address=145.79.227.40:4084 , piece_id=RUKTKDII6BRUWSYVS65CWUBT3QKG3ASUK3KA3O5VGE53XSG6IMOQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.943459560 (610h25m24.814542605s) [noise=true , address=145.79.227.14:4847 , piece_id=JX5NXTMVM3NSWIEVR4RX46ABSNPKTOFDLFBPKEXFQ3G3ANKV5OLA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.943865303 (610h25m24.814948348s) [noise=true , address=45.91.116.6:4981 , piece_id=CT474ZA2LVAZEBJHBDD4GB6NTHVHZFJLNE5OZV3DFS2NZKVW3EQQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.944259694 (610h25m24.815342749s) [noise=true , address=145.79.227.51:4443 , piece_id=73UUMRDY7BSZNCGBXCZYYRBVE5KLPMNPRNXOZEGWKQGLHJS32J2Q, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.946575182 (610h25m24.817658227s) [noise=true , address=77.90.31.10:4523 , piece_id=P6LIONXJI6AFZ2JTKCQ2W6RZCOZCK6MZ53EXAJKXYQXSX2PX4QEA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.948132909 (610h25m24.819215944s) [noise=true , address=145.79.227.20:4452 , piece_id=ZSDN2JZ3FQUVJYN5OURDXNKEDLDNULJOHKKRZVIPXZIWIYAHCRXQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.949605774 (610h25m24.820688819s) [noise=true , address=77.90.31.14:4680 , piece_id=S2E4WBA7MJVAZLT5JG3IMHK53RQC77CCHV2PDF6L2R2JEA2U3K4Q, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.951459002 (610h25m24.822542048s) [noise=true , address=193.149.164.143:4704 , piece_id=BHXHFBKTLTDMGXWY5QRTP2THZQE5OXFZ2LCMTO2KRYRWAHXWW3NQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.952986742 (610h25m24.824069777s) [noise=true , address=77.90.31.19:4848 , piece_id=VZYG32S6ODIPDKQYZRXECBHXOERZDDWD2OF35Z46W4XMUZ3OJLAA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.956916654 (610h25m24.827999689s) [noise=true , address=145.79.227.29:4971 , piece_id=MEPXHHOQ6SUZC3GIPCQEMOKKF5OJTKZEI3UYZBAYXAQZGSVEKSBA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.960665062 (610h25m24.831748097s) [noise=true , address=64.49.13.170:4720 , piece_id=HIATRSAPE4YLSODFZDPP43ZUMLDVM7GJEVMWE4ZGUQTP25LTHGTA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.962366001 (610h25m24.833449036s) [noise=true , address=145.79.227.2:4117 , piece_id=4SGFGB3IJ7LTXDMMWM2PGGOV64MC46IZ7OZQWH55J4NBQY7CHMZQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.963708216 (610h25m24.834791261s) [noise=true , address=145.79.227.35:4505 , piece_id=V7RS7A4QLZTH4G32ZUZNSF2Z5APVWZGJO77NEGP3SBWQT3WCMVLQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:26.352236138 (610h25m26.223319173s) [upload=534952]: Next returned nil segment Dec 31 01:54:26.352253821 (610h25m26.223336857s) [upload=534952]: Waiting for error group managing segments... Dec 31 01:54:26.812758374 (610h25m26.683841409s) [noise=true , address=145.79.227.51:4443 , piece_id=73UUMRDY7BSZNCGBXCZYYRBVE5KLPMNPRNXOZEGWKQGLHJS32J2Q, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.824371056 (610h25m26.695454101s) [noise=true , address=145.79.227.14:4847 , piece_id=JX5NXTMVM3NSWIEVR4RX46ABSNPKTOFDLFBPKEXFQ3G3ANKV5OLA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.838414046 (610h25m26.709497081s) [noise=true , address=64.49.13.138:4490 , piece_id=2XTUZE6PXRE3OARNJVT36FXXWUMYIBVNE6M2VVFAVBGKEE2H5YZQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.875499841 (610h25m26.746582876s) [noise=true , address=64.49.13.139:4478 , piece_id=QN6RT2KWEA6APQVA66ZMKG4LH537UGYNOXOCVVA2YC2MKDOLT3YA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.891607221 (610h25m26.762690256s) [noise=true , address=145.79.227.20:4452 , piece_id=ZSDN2JZ3FQUVJYN5OURDXNKEDLDNULJOHKKRZVIPXZIWIYAHCRXQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.988561223 (610h25m26.859644268s) [noise=true , address=64.49.13.170:4720 , piece_id=HIATRSAPE4YLSODFZDPP43ZUMLDVM7GJEVMWE4ZGUQTP25LTHGTA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.039035911 (610h25m26.910118946s) [noise=true , address=77.90.31.14:4680 , piece_id=S2E4WBA7MJVAZLT5JG3IMHK53RQC77CCHV2PDF6L2R2JEA2U3K4Q, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.098023090 (610h25m26.969106126s) [noise=true , address=45.91.116.6:4981 , piece_id=CT474ZA2LVAZEBJHBDD4GB6NTHVHZFJLNE5OZV3DFS2NZKVW3EQQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.653790754 (610h25m27.524873789s) [noise=true , address=77.90.31.21:4886 , piece_id=K34WPIBGGGEPHIKJ253XGLDCIQ3WLFN4AZ6EYNWVHHT2EIHS6YRQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.684192972 (610h25m27.555276017s) [noise=true , address=193.149.164.143:4704 , piece_id=BHXHFBKTLTDMGXWY5QRTP2THZQE5OXFZ2LCMTO2KRYRWAHXWW3NQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.717238425 (610h25m27.58832146s) [noise=true , address=145.79.227.40:4084 , piece_id=RUKTKDII6BRUWSYVS65CWUBT3QKG3ASUK3KA3O5VGE53XSG6IMOQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.718349742 (610h25m27.589432777s) [noise=true , address=77.90.31.10:4523 , piece_id=P6LIONXJI6AFZ2JTKCQ2W6RZCOZCK6MZ53EXAJKXYQXSX2PX4QEA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.803639562 (610h25m27.674722597s) [noise=true , address=145.79.227.29:4971 , piece_id=MEPXHHOQ6SUZC3GIPCQEMOKKF5OJTKZEI3UYZBAYXAQZGSVEKSBA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.812791279 (610h25m27.683874314s) [noise=true , address=145.79.227.49:4343 , piece_id=6L3HPTS7D2NRA4QEOH4RFM3D3B53CTCONRI6G6YVU4OOTUWTG7XA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.812812348 (610h25m27.683895393s) [seg_pos={0 0}, upload=534952]: Segment reached optimal threshold of 14 pieces. Dec 31 01:54:27.813018000 (610h25m27.684101035s) [noise=true , address=145.79.227.35:4505 , piece_id=V7RS7A4QLZTH4G32ZUZNSF2Z5APVWZGJO77NEGP3SBWQT3WCMVLQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i6itGoBhLsLiqC8et8Y5L62fLjcWxy18u39NAr4R3nXNMWcsf): protocol: expected piece hash; context canceled; EOF Dec 31 01:54:27.813024772 (610h25m27.684107808s) [noise=true , address=145.79.227.35:4505 , piece_id=V7RS7A4QLZTH4G32ZUZNSF2Z5APVWZGJO77NEGP3SBWQT3WCMVLQ, seg_pos={0 0}, upload=534952]: Long tail cancellation due to optimal threshold Dec 31 01:54:27.813175931 (610h25m27.684258976s) [noise=true , address=145.79.227.34:4937 , piece_id=5W7LBM3YJJTVUFYTELR57DVXB663OHO5N3Z2SPQPZ4G43IK47O3Q, seg_pos={0 0}, upload=534952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1LKrFThdW3jJpG6dRZLUaxLXTa5Wy27yQ8PGUEqjvBWfFoRB94): protocol: expected piece hash; context canceled; EOF Dec 31 01:54:27.813181702 (610h25m27.684264737s) [noise=true , address=145.79.227.34:4937 , piece_id=5W7LBM3YJJTVUFYTELR57DVXB663OHO5N3Z2SPQPZ4G43IK47O3Q, seg_pos={0 0}, upload=534952]: Long tail cancellation due to optimal threshold Dec 31 01:54:27.813407862 (610h25m27.684490907s) [noise=true , address=145.79.227.2:4117 , piece_id=4SGFGB3IJ7LTXDMMWM2PGGOV64MC46IZ7OZQWH55J4NBQY7CHMZQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122Htg72f5eUUutf1p3fYQpy1Wkf6AgdaQV83HHYye2n8zm3the): protocol: expected piece hash; context canceled; EOF Dec 31 01:54:27.813414264 (610h25m27.684497299s) [noise=true , address=145.79.227.2:4117 , piece_id=4SGFGB3IJ7LTXDMMWM2PGGOV64MC46IZ7OZQWH55J4NBQY7CHMZQ, seg_pos={0 0}, upload=534952]: Long tail cancellation due to optimal threshold Dec 31 01:54:27.813424614 (610h25m27.684507649s) [noise=true , address=77.90.31.19:4848 , piece_id=VZYG32S6ODIPDKQYZRXECBHXOERZDDWD2OF35Z46W4XMUZ3OJLAA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD): protocol: write tcp 188.166.225.170:42842->77.90.31.19:4848: use of closed network connection; write tcp 188.166.225.170:42842->77.90.31.19:4848: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42842->77.90.31.19:4848: use of closed network connection Dec 31 01:54:27.813432750 (610h25m27.684515785s) [noise=true , address=77.90.31.19:4848 , piece_id=VZYG32S6ODIPDKQYZRXECBHXOERZDDWD2OF35Z46W4XMUZ3OJLAA, seg_pos={0 0}, upload=534952]: Long tail cancellation due to optimal threshold Dec 31 01:54:27.813474088 (610h25m27.684557123s) [seg_pos={0 0}, upload=534952]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 01:54:27.813535325 (610h25m27.68461836s) [upload=534952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 01:54:27.813539613 (610h25m27.684622648s) [upload=534952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 01:54:28.099921034 (610h25m27.971004069s) [upload=534952]: Done uploading segments... Dec 31 01:54:28.099940451 (610h25m27.971023486s) [upload=534952]: Upload finished. err: <nil> Duration: 15.370733616s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.........................X...............................X..] duration: 3.533248732s trace: https://jaeger.dev.storj.io/trace/25cdf0a5222c0faa FAIL (2025-12-31 02:36:07 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.533279711s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [........X..X.......X.X..X....XX.XX........X.X......X..X.....] duration: 6.273090051s trace: https://jaeger.dev.storj.io/trace/54231a38e027f2fe FAIL (2025-12-31 02:35:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: DCML4BWFWKJ2AEQVRGRYOQ7Q4YY3BF2PF4GFQSOREMAG67M4BJ5Q) context deadline exceeded; piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: YBK3PD3XVPSHJZTXOEGCXAGKJFRBLUSTFH6ZHYEFYK5YRIMLGISQ) context deadline exceeded ======LOGS====== Dec 31 02:31:42.009467393 (611h2m41.880550439s) [upload=535486]: Starting upload Dec 31 02:31:42.009568907 (611h2m41.880651942s) [upload=535486]: Done starting upload Dec 31 02:31:42.009677694 (611h2m41.880760729s) [upload=535486]: Uploading segments... Dec 31 02:31:42.009860923 (611h2m41.880943969s) [upload=535486]: Got next segment. Inline: false Dec 31 02:31:42.009921478 (611h2m41.881004513s) [upload=535486]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 02:31:42.009926968 (611h2m41.881010003s) [upload=535486]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 02:31:42.257755154 (611h2m42.128838189s) [seg_pos={0 0}, upload=535486]: Begin upload segment... Dec 31 02:31:42.257856827 (611h2m42.128939862s) [seg_pos={0 0}, upload=535486]: Done begin upload segment. Dec 31 02:31:42.257895601 (611h2m42.128978646s) [noise=true , address=45.91.116.6:4973 , piece_id=BVEV2ZTEUP3UUZ7H6WBZ4MHY2JESHTR6Z2HY3AAD5XCRWE7NLMMQ, seg_pos={0 0}, upload=535486]: Uploading piece... Dec 31 02:31:42.257927502 (611h2m42.129010537s) [noise=true , address=77.90.31.19:4844 , piece_id=DJPEVJBHVIZGFMBSUBTEWEIMU3JCSWPSKUXWPDQGXJKXLFGXQQGA, seg_pos={0 0}, upload=535486]: Uploading piece... Dec 31 02:31:42.258733707 (611h2m42.129816752s) [noise=true , address=145.79.227.52:4447 , piece_id=65XJCKQL2VIDCHMSHGUSCFNPU7JWFPKRF2QXWNAHQT7T5GELABFQ, seg_pos={0 0}, upload=535486]: Uploading piece... Dec 31 02:31:42.259561023 (611h2m42.130644058s) [noise=true , address=45.91.116.9:4262 , piece_id=DCML4BWFWKJ2AEQVRGRYOQ7Q4YY3BF2PF4GFQSOREMAG67M4BJ5Q, seg_pos={0 0}, upload=535486]: Uploading piece... Dec 31 02:31:42.259700759 (611h2m42.130783794s) [noise=true , address=45.91.116.11:4118 , piece_id=JAUSAGDUAQ4BBYWNA7OSW6XATK2JLMRUM6K6QNHD5EB4DPZNPQZA, seg_pos={0 0}, upload=535486]: Uploading piece... Dec 31 02:31:42.260044383 (611h2m42.131127428s) [noise=true , address=45.91.116.12:4225 , piece_id=YBK3PD3XVPSHJZTXOEGCXAGKJFRBLUSTFH6ZHYEFYK5YRIMLGISQ, seg_pos={0 0}, upload=535486]: Uploading piece... Dec 31 02:31:42.260541770 (611h2m42.131624815s) [noise=true , address=45.91.116.5:4299 , piece_id=GZH33CCTCZPZX5ZPHEXDKBI2YSUP4GZ7YUQ6FIIHLCBB24SRSFGA, seg_pos={0 0}, upload=535486]: Uploading piece... Dec 31 02:31:42.261116324 (611h2m42.132199369s) [noise=true , address=145.79.227.18:4007 , piece_id=HA2C5LN5ZB6TVJBQ6GVELU4JQG5KII7CFBTZAHDT75KJP5J7FOYA, seg_pos={0 0}, upload=535486]: Uploading piece... Dec 31 02:31:42.262541628 (611h2m42.133624674s) [noise=true , address=145.79.227.15:4876 , piece_id=WCIBKRCJCRS4JGFTWDQHVUZ2TLV3OYKTSMIBSBSU4WBFZ7WZNHOA, seg_pos={0 0}, upload=535486]: Uploading piece... Dec 31 02:31:42.262838263 (611h2m42.133921308s) [noise=true , address=145.79.227.3:4150 , piece_id=GKSGSQWFKTLK4B4ARKBXK2FR5IDBCOJMZZUOFEI2KASMXPJUHI4A, seg_pos={0 0}, upload=535486]: Uploading piece... Dec 31 02:31:42.263683141 (611h2m42.134766186s) [noise=true , address=77.90.31.10:4519 , piece_id=CZ5AVVMI74GHKCHLXL47OQED5DAR66LRFIVH25A3JL6K7NXNAOSA, seg_pos={0 0}, upload=535486]: Uploading piece... Dec 31 02:31:42.263989505 (611h2m42.13507254s) [noise=true , address=145.79.227.14:4837 , piece_id=HIU57ZYKPL6JYTQB27K2XG67GGYG7XM63OCJ4YLMMQXMLKYDCFJA, seg_pos={0 0}, upload=535486]: Uploading piece... Dec 31 02:31:42.264792113 (611h2m42.135875158s) [noise=true , address=145.79.227.20:4438 , piece_id=MNT67N5BTYQB2JUNXV7TVSRL3DJMMTBXYQO4BSWEFALCWMK2ZP5Q, seg_pos={0 0}, upload=535486]: Uploading piece... Dec 31 02:31:42.265209829 (611h2m42.136292864s) [noise=true , address=145.79.227.39:4014 , piece_id=R3DDMHHRZSHJUDSEXG3BUD2SOIVBXYAJE6RRMTPKW6XAPT7CNLTQ, seg_pos={0 0}, upload=535486]: Uploading piece... Dec 31 02:31:42.266296698 (611h2m42.137379733s) [noise=true , address=145.79.227.24:4581 , piece_id=OPPGQUC2NXAGBLYXH5VM4RGY6ATR5DAUJAGRRKAQCZ3ENMNTMLBA, seg_pos={0 0}, upload=535486]: Uploading piece... Dec 31 02:31:42.270084160 (611h2m42.141167195s) [noise=true , address=145.79.227.21:4474 , piece_id=HBUEUQTYMITUOKLQEQHAOSCBHCVXCWGCUM4WDPC4ZMFKRQCW3GRA, seg_pos={0 0}, upload=535486]: Uploading piece... Dec 31 02:31:43.955549810 (611h2m43.826632865s) [upload=535486]: Next returned nil segment Dec 31 02:31:43.955567964 (611h2m43.826651009s) [upload=535486]: Waiting for error group managing segments... Dec 31 02:31:44.239121474 (611h2m44.110204519s) [noise=true , address=145.79.227.18:4007 , piece_id=HA2C5LN5ZB6TVJBQ6GVELU4JQG5KII7CFBTZAHDT75KJP5J7FOYA, seg_pos={0 0}, upload=535486]: Done uploading piece. err: <nil> Dec 31 02:31:44.247474961 (611h2m44.118557996s) [noise=true , address=145.79.227.24:4581 , piece_id=OPPGQUC2NXAGBLYXH5VM4RGY6ATR5DAUJAGRRKAQCZ3ENMNTMLBA, seg_pos={0 0}, upload=535486]: Done uploading piece. err: <nil> Dec 31 02:31:44.249933482 (611h2m44.121016527s) [noise=true , address=145.79.227.39:4014 , piece_id=R3DDMHHRZSHJUDSEXG3BUD2SOIVBXYAJE6RRMTPKW6XAPT7CNLTQ, seg_pos={0 0}, upload=535486]: Done uploading piece. err: <nil> Dec 31 02:31:44.250394820 (611h2m44.121477855s) [noise=true , address=145.79.227.52:4447 , piece_id=65XJCKQL2VIDCHMSHGUSCFNPU7JWFPKRF2QXWNAHQT7T5GELABFQ, seg_pos={0 0}, upload=535486]: Done uploading piece. err: <nil> Dec 31 02:31:44.260864926 (611h2m44.131947971s) [noise=true , address=145.79.227.3:4150 , piece_id=GKSGSQWFKTLK4B4ARKBXK2FR5IDBCOJMZZUOFEI2KASMXPJUHI4A, seg_pos={0 0}, upload=535486]: Done uploading piece. err: <nil> Dec 31 02:31:44.265136891 (611h2m44.136219936s) [noise=true , address=145.79.227.15:4876 , piece_id=WCIBKRCJCRS4JGFTWDQHVUZ2TLV3OYKTSMIBSBSU4WBFZ7WZNHOA, seg_pos={0 0}, upload=535486]: Done uploading piece. err: <nil> Dec 31 02:31:44.274014796 (611h2m44.145097831s) [noise=true , address=145.79.227.20:4438 , piece_id=MNT67N5BTYQB2JUNXV7TVSRL3DJMMTBXYQO4BSWEFALCWMK2ZP5Q, seg_pos={0 0}, upload=535486]: Done uploading piece. err: <nil> Dec 31 02:31:44.275430611 (611h2m44.146513646s) [noise=true , address=145.79.227.21:4474 , piece_id=HBUEUQTYMITUOKLQEQHAOSCBHCVXCWGCUM4WDPC4ZMFKRQCW3GRA, seg_pos={0 0}, upload=535486]: Done uploading piece. err: <nil> Dec 31 02:31:44.276265651 (611h2m44.147348686s) [noise=true , address=145.79.227.14:4837 , piece_id=HIU57ZYKPL6JYTQB27K2XG67GGYG7XM63OCJ4YLMMQXMLKYDCFJA, seg_pos={0 0}, upload=535486]: Done uploading piece. err: <nil> Dec 31 02:31:44.555968520 (611h2m44.427051555s) [noise=true , address=45.91.116.12:4225 , piece_id=YBK3PD3XVPSHJZTXOEGCXAGKJFRBLUSTFH6ZHYEFYK5YRIMLGISQ, seg_pos={0 0}, upload=535486]: Done uploading piece. err: <nil> Dec 31 02:31:44.563093667 (611h2m44.434176702s) [noise=true , address=77.90.31.10:4519 , piece_id=CZ5AVVMI74GHKCHLXL47OQED5DAR66LRFIVH25A3JL6K7NXNAOSA, seg_pos={0 0}, upload=535486]: Done uploading piece. err: <nil> Dec 31 02:31:44.625636847 (611h2m44.496719882s) [noise=true , address=45.91.116.9:4262 , piece_id=DCML4BWFWKJ2AEQVRGRYOQ7Q4YY3BF2PF4GFQSOREMAG67M4BJ5Q, seg_pos={0 0}, upload=535486]: Done uploading piece. err: <nil> Dec 31 02:31:44.717693032 (611h2m44.588776077s) [noise=true , address=45.91.116.6:4973 , piece_id=BVEV2ZTEUP3UUZ7H6WBZ4MHY2JESHTR6Z2HY3AAD5XCRWE7NLMMQ, seg_pos={0 0}, upload=535486]: Done uploading piece. err: <nil> Dec 31 02:31:44.717715566 (611h2m44.588798611s) [seg_pos={0 0}, upload=535486]: Segment reached optimal threshold of 13 pieces. Dec 31 02:31:44.718014545 (611h2m44.58909759s) [noise=true , address=45.91.116.11:4118 , piece_id=JAUSAGDUAQ4BBYWNA7OSW6XATK2JLMRUM6K6QNHD5EB4DPZNPQZA, seg_pos={0 0}, upload=535486]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF Dec 31 02:31:44.718024604 (611h2m44.589107639s) [noise=true , address=45.91.116.11:4118 , piece_id=JAUSAGDUAQ4BBYWNA7OSW6XATK2JLMRUM6K6QNHD5EB4DPZNPQZA, seg_pos={0 0}, upload=535486]: Long tail cancellation due to optimal threshold Dec 31 02:31:44.718138050 (611h2m44.589221125s) [noise=true , address=77.90.31.19:4844 , piece_id=DJPEVJBHVIZGFMBSUBTEWEIMU3JCSWPSKUXWPDQGXJKXLFGXQQGA, seg_pos={0 0}, upload=535486]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF Dec 31 02:31:44.718145925 (611h2m44.58922897s) [noise=true , address=77.90.31.19:4844 , piece_id=DJPEVJBHVIZGFMBSUBTEWEIMU3JCSWPSKUXWPDQGXJKXLFGXQQGA, seg_pos={0 0}, upload=535486]: Long tail cancellation due to optimal threshold Dec 31 02:31:44.718182945 (611h2m44.58926598s) [noise=true , address=45.91.116.5:4299 , piece_id=GZH33CCTCZPZX5ZPHEXDKBI2YSUP4GZ7YUQ6FIIHLCBB24SRSFGA, seg_pos={0 0}, upload=535486]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Dec 31 02:31:44.718191292 (611h2m44.589274327s) [noise=true , address=45.91.116.5:4299 , piece_id=GZH33CCTCZPZX5ZPHEXDKBI2YSUP4GZ7YUQ6FIIHLCBB24SRSFGA, seg_pos={0 0}, upload=535486]: Long tail cancellation due to optimal threshold Dec 31 02:31:44.718248981 (611h2m44.589332016s) [seg_pos={0 0}, upload=535486]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 02:31:44.718376454 (611h2m44.589459519s) [upload=535486]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 02:31:44.718437000 (611h2m44.589520045s) [upload=535486]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 02:31:45.017559155 (611h2m44.8886422s) [upload=535486]: Done uploading segments... Dec 31 02:31:45.017585174 (611h2m44.888668209s) [upload=535486]: Upload finished. err: <nil> Duration: 6.273114947s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.........X..X.......X.X..X....XX.XX........X.X......X..X....] duration: 19.370086346s trace: https://jaeger.dev.storj.io/trace/54231a38e027f2fe FAIL (2025-12-31 02:35:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: DCML4BWFWKJ2AEQVRGRYOQ7Q4YY3BF2PF4GFQSOREMAG67M4BJ5Q) context deadline exceeded; piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: YBK3PD3XVPSHJZTXOEGCXAGKJFRBLUSTFH6ZHYEFYK5YRIMLGISQ) context deadline exceeded ======LOGS====== Dec 31 02:31:42.009467393 (611h2m41.880550439s) [upload=535486]: Starting upload Dec 31 02:31:42.009568907 (611h2m41.880651942s) [upload=535486]: Done starting upload Dec 31 02:31:42.009677694 (611h2m41.880760729s) [upload=535486]: Uploading segments... Dec 31 02:31:42.009860923 (611h2m41.880943969s) [upload=535486]: Got next segment. Inline: false Dec 31 02:31:42.009921478 (611h2m41.881004513s) [upload=535486]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 02:31:42.009926968 (611h2m41.881010003s) [upload=535486]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 02:31:42.257755154 (611h2m42.128838189s) [seg_pos={0 0}, upload=535486]: Begin upload segment... Dec 31 02:31:42.257856827 (611h2m42.128939862s) [seg_pos={0 0}, upload=535486]: Done begin upload segment. Dec 31 02:31:42.257895601 (611h2m42.128978646s) [noise=true , address=45.91.116.6:4973 , piece_id=BVEV2ZTEUP3UUZ7H6WBZ4MHY2JESHTR6Z2HY3AAD5XCRWE7NLMMQ, seg_pos={0 0}, upload=535486]: Uploading piece... Dec 31 02:31:42.257927502 (611h2m42.129010537s) [noise=true , address=77.90.31.19:4844 , piece_id=DJPEVJBHVIZGFMBSUBTEWEIMU3JCSWPSKUXWPDQGXJKXLFGXQQGA, seg_pos={0 0}, upload=535486]: Uploading piece... Dec 31 02:31:42.258733707 (611h2m42.129816752s) [noise=true , address=145.79.227.52:4447 , piece_id=65XJCKQL2VIDCHMSHGUSCFNPU7JWFPKRF2QXWNAHQT7T5GELABFQ, seg_pos={0 0}, upload=535486]: Uploading piece... Dec 31 02:31:42.259561023 (611h2m42.130644058s) [noise=true , address=45.91.116.9:4262 , piece_id=DCML4BWFWKJ2AEQVRGRYOQ7Q4YY3BF2PF4GFQSOREMAG67M4BJ5Q, seg_pos={0 0}, upload=535486]: Uploading piece... Dec 31 02:31:42.259700759 (611h2m42.130783794s) [noise=true , address=45.91.116.11:4118 , piece_id=JAUSAGDUAQ4BBYWNA7OSW6XATK2JLMRUM6K6QNHD5EB4DPZNPQZA, seg_pos={0 0}, upload=535486]: Uploading piece... Dec 31 02:31:42.260044383 (611h2m42.131127428s) [noise=true , address=45.91.116.12:4225 , piece_id=YBK3PD3XVPSHJZTXOEGCXAGKJFRBLUSTFH6ZHYEFYK5YRIMLGISQ, seg_pos={0 0}, upload=535486]: Uploading piece... Dec 31 02:31:42.260541770 (611h2m42.131624815s) [noise=true , address=45.91.116.5:4299 , piece_id=GZH33CCTCZPZX5ZPHEXDKBI2YSUP4GZ7YUQ6FIIHLCBB24SRSFGA, seg_pos={0 0}, upload=535486]: Uploading piece... Dec 31 02:31:42.261116324 (611h2m42.132199369s) [noise=true , address=145.79.227.18:4007 , piece_id=HA2C5LN5ZB6TVJBQ6GVELU4JQG5KII7CFBTZAHDT75KJP5J7FOYA, seg_pos={0 0}, upload=535486]: Uploading piece... Dec 31 02:31:42.262541628 (611h2m42.133624674s) [noise=true , address=145.79.227.15:4876 , piece_id=WCIBKRCJCRS4JGFTWDQHVUZ2TLV3OYKTSMIBSBSU4WBFZ7WZNHOA, seg_pos={0 0}, upload=535486]: Uploading piece... Dec 31 02:31:42.262838263 (611h2m42.133921308s) [noise=true , address=145.79.227.3:4150 , piece_id=GKSGSQWFKTLK4B4ARKBXK2FR5IDBCOJMZZUOFEI2KASMXPJUHI4A, seg_pos={0 0}, upload=535486]: Uploading piece... Dec 31 02:31:42.263683141 (611h2m42.134766186s) [noise=true , address=77.90.31.10:4519 , piece_id=CZ5AVVMI74GHKCHLXL47OQED5DAR66LRFIVH25A3JL6K7NXNAOSA, seg_pos={0 0}, upload=535486]: Uploading piece... Dec 31 02:31:42.263989505 (611h2m42.13507254s) [noise=true , address=145.79.227.14:4837 , piece_id=HIU57ZYKPL6JYTQB27K2XG67GGYG7XM63OCJ4YLMMQXMLKYDCFJA, seg_pos={0 0}, upload=535486]: Uploading piece... Dec 31 02:31:42.264792113 (611h2m42.135875158s) [noise=true , address=145.79.227.20:4438 , piece_id=MNT67N5BTYQB2JUNXV7TVSRL3DJMMTBXYQO4BSWEFALCWMK2ZP5Q, seg_pos={0 0}, upload=535486]: Uploading piece... Dec 31 02:31:42.265209829 (611h2m42.136292864s) [noise=true , address=145.79.227.39:4014 , piece_id=R3DDMHHRZSHJUDSEXG3BUD2SOIVBXYAJE6RRMTPKW6XAPT7CNLTQ, seg_pos={0 0}, upload=535486]: Uploading piece... Dec 31 02:31:42.266296698 (611h2m42.137379733s) [noise=true , address=145.79.227.24:4581 , piece_id=OPPGQUC2NXAGBLYXH5VM4RGY6ATR5DAUJAGRRKAQCZ3ENMNTMLBA, seg_pos={0 0}, upload=535486]: Uploading piece... Dec 31 02:31:42.270084160 (611h2m42.141167195s) [noise=true , address=145.79.227.21:4474 , piece_id=HBUEUQTYMITUOKLQEQHAOSCBHCVXCWGCUM4WDPC4ZMFKRQCW3GRA, seg_pos={0 0}, upload=535486]: Uploading piece... Dec 31 02:31:43.955549810 (611h2m43.826632865s) [upload=535486]: Next returned nil segment Dec 31 02:31:43.955567964 (611h2m43.826651009s) [upload=535486]: Waiting for error group managing segments... Dec 31 02:31:44.239121474 (611h2m44.110204519s) [noise=true , address=145.79.227.18:4007 , piece_id=HA2C5LN5ZB6TVJBQ6GVELU4JQG5KII7CFBTZAHDT75KJP5J7FOYA, seg_pos={0 0}, upload=535486]: Done uploading piece. err: <nil> Dec 31 02:31:44.247474961 (611h2m44.118557996s) [noise=true , address=145.79.227.24:4581 , piece_id=OPPGQUC2NXAGBLYXH5VM4RGY6ATR5DAUJAGRRKAQCZ3ENMNTMLBA, seg_pos={0 0}, upload=535486]: Done uploading piece. err: <nil> Dec 31 02:31:44.249933482 (611h2m44.121016527s) [noise=true , address=145.79.227.39:4014 , piece_id=R3DDMHHRZSHJUDSEXG3BUD2SOIVBXYAJE6RRMTPKW6XAPT7CNLTQ, seg_pos={0 0}, upload=535486]: Done uploading piece. err: <nil> Dec 31 02:31:44.250394820 (611h2m44.121477855s) [noise=true , address=145.79.227.52:4447 , piece_id=65XJCKQL2VIDCHMSHGUSCFNPU7JWFPKRF2QXWNAHQT7T5GELABFQ, seg_pos={0 0}, upload=535486]: Done uploading piece. err: <nil> Dec 31 02:31:44.260864926 (611h2m44.131947971s) [noise=true , address=145.79.227.3:4150 , piece_id=GKSGSQWFKTLK4B4ARKBXK2FR5IDBCOJMZZUOFEI2KASMXPJUHI4A, seg_pos={0 0}, upload=535486]: Done uploading piece. err: <nil> Dec 31 02:31:44.265136891 (611h2m44.136219936s) [noise=true , address=145.79.227.15:4876 , piece_id=WCIBKRCJCRS4JGFTWDQHVUZ2TLV3OYKTSMIBSBSU4WBFZ7WZNHOA, seg_pos={0 0}, upload=535486]: Done uploading piece. err: <nil> Dec 31 02:31:44.274014796 (611h2m44.145097831s) [noise=true , address=145.79.227.20:4438 , piece_id=MNT67N5BTYQB2JUNXV7TVSRL3DJMMTBXYQO4BSWEFALCWMK2ZP5Q, seg_pos={0 0}, upload=535486]: Done uploading piece. err: <nil> Dec 31 02:31:44.275430611 (611h2m44.146513646s) [noise=true , address=145.79.227.21:4474 , piece_id=HBUEUQTYMITUOKLQEQHAOSCBHCVXCWGCUM4WDPC4ZMFKRQCW3GRA, seg_pos={0 0}, upload=535486]: Done uploading piece. err: <nil> Dec 31 02:31:44.276265651 (611h2m44.147348686s) [noise=true , address=145.79.227.14:4837 , piece_id=HIU57ZYKPL6JYTQB27K2XG67GGYG7XM63OCJ4YLMMQXMLKYDCFJA, seg_pos={0 0}, upload=535486]: Done uploading piece. err: <nil> Dec 31 02:31:44.555968520 (611h2m44.427051555s) [noise=true , address=45.91.116.12:4225 , piece_id=YBK3PD3XVPSHJZTXOEGCXAGKJFRBLUSTFH6ZHYEFYK5YRIMLGISQ, seg_pos={0 0}, upload=535486]: Done uploading piece. err: <nil> Dec 31 02:31:44.563093667 (611h2m44.434176702s) [noise=true , address=77.90.31.10:4519 , piece_id=CZ5AVVMI74GHKCHLXL47OQED5DAR66LRFIVH25A3JL6K7NXNAOSA, seg_pos={0 0}, upload=535486]: Done uploading piece. err: <nil> Dec 31 02:31:44.625636847 (611h2m44.496719882s) [noise=true , address=45.91.116.9:4262 , piece_id=DCML4BWFWKJ2AEQVRGRYOQ7Q4YY3BF2PF4GFQSOREMAG67M4BJ5Q, seg_pos={0 0}, upload=535486]: Done uploading piece. err: <nil> Dec 31 02:31:44.717693032 (611h2m44.588776077s) [noise=true , address=45.91.116.6:4973 , piece_id=BVEV2ZTEUP3UUZ7H6WBZ4MHY2JESHTR6Z2HY3AAD5XCRWE7NLMMQ, seg_pos={0 0}, upload=535486]: Done uploading piece. err: <nil> Dec 31 02:31:44.717715566 (611h2m44.588798611s) [seg_pos={0 0}, upload=535486]: Segment reached optimal threshold of 13 pieces. Dec 31 02:31:44.718014545 (611h2m44.58909759s) [noise=true , address=45.91.116.11:4118 , piece_id=JAUSAGDUAQ4BBYWNA7OSW6XATK2JLMRUM6K6QNHD5EB4DPZNPQZA, seg_pos={0 0}, upload=535486]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF Dec 31 02:31:44.718024604 (611h2m44.589107639s) [noise=true , address=45.91.116.11:4118 , piece_id=JAUSAGDUAQ4BBYWNA7OSW6XATK2JLMRUM6K6QNHD5EB4DPZNPQZA, seg_pos={0 0}, upload=535486]: Long tail cancellation due to optimal threshold Dec 31 02:31:44.718138050 (611h2m44.589221125s) [noise=true , address=77.90.31.19:4844 , piece_id=DJPEVJBHVIZGFMBSUBTEWEIMU3JCSWPSKUXWPDQGXJKXLFGXQQGA, seg_pos={0 0}, upload=535486]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF Dec 31 02:31:44.718145925 (611h2m44.58922897s) [noise=true , address=77.90.31.19:4844 , piece_id=DJPEVJBHVIZGFMBSUBTEWEIMU3JCSWPSKUXWPDQGXJKXLFGXQQGA, seg_pos={0 0}, upload=535486]: Long tail cancellation due to optimal threshold Dec 31 02:31:44.718182945 (611h2m44.58926598s) [noise=true , address=45.91.116.5:4299 , piece_id=GZH33CCTCZPZX5ZPHEXDKBI2YSUP4GZ7YUQ6FIIHLCBB24SRSFGA, seg_pos={0 0}, upload=535486]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Dec 31 02:31:44.718191292 (611h2m44.589274327s) [noise=true , address=45.91.116.5:4299 , piece_id=GZH33CCTCZPZX5ZPHEXDKBI2YSUP4GZ7YUQ6FIIHLCBB24SRSFGA, seg_pos={0 0}, upload=535486]: Long tail cancellation due to optimal threshold Dec 31 02:31:44.718248981 (611h2m44.589332016s) [seg_pos={0 0}, upload=535486]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 02:31:44.718376454 (611h2m44.589459519s) [upload=535486]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 02:31:44.718437000 (611h2m44.589520045s) [upload=535486]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 02:31:45.017559155 (611h2m44.8886422s) [upload=535486]: Done uploading segments... Dec 31 02:31:45.017585174 (611h2m44.888668209s) [upload=535486]: Upload finished. err: <nil> Duration: 19.370114751s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..........................X...............................X.] duration: 3.781469253s trace: https://jaeger.dev.storj.io/trace/25cdf0a5222c0faa FAIL (2025-12-31 02:35:27 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.781496756s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 15.485477077s trace: https://jaeger.dev.storj.io/trace/365693fea00df28 FAIL (2025-12-31 02:35:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [noise=true , address=145.79.227.49:4343 , piece_id=6L3HPTS7D2NRA4QEOH4RFM3D3B53CTCONRI6G6YVU4OOTUWTG7XA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.942060847 (610h25m24.813143892s) [noise=true , address=145.79.227.34:4937 , piece_id=5W7LBM3YJJTVUFYTELR57DVXB663OHO5N3Z2SPQPZ4G43IK47O3Q, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.942684104 (610h25m24.813767139s) [noise=true , address=64.49.13.139:4478 , piece_id=QN6RT2KWEA6APQVA66ZMKG4LH537UGYNOXOCVVA2YC2MKDOLT3YA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.943064007 (610h25m24.814147042s) [noise=true , address=145.79.227.40:4084 , piece_id=RUKTKDII6BRUWSYVS65CWUBT3QKG3ASUK3KA3O5VGE53XSG6IMOQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.943459560 (610h25m24.814542605s) [noise=true , address=145.79.227.14:4847 , piece_id=JX5NXTMVM3NSWIEVR4RX46ABSNPKTOFDLFBPKEXFQ3G3ANKV5OLA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.943865303 (610h25m24.814948348s) [noise=true , address=45.91.116.6:4981 , piece_id=CT474ZA2LVAZEBJHBDD4GB6NTHVHZFJLNE5OZV3DFS2NZKVW3EQQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.944259694 (610h25m24.815342749s) [noise=true , address=145.79.227.51:4443 , piece_id=73UUMRDY7BSZNCGBXCZYYRBVE5KLPMNPRNXOZEGWKQGLHJS32J2Q, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.946575182 (610h25m24.817658227s) [noise=true , address=77.90.31.10:4523 , piece_id=P6LIONXJI6AFZ2JTKCQ2W6RZCOZCK6MZ53EXAJKXYQXSX2PX4QEA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.948132909 (610h25m24.819215944s) [noise=true , address=145.79.227.20:4452 , piece_id=ZSDN2JZ3FQUVJYN5OURDXNKEDLDNULJOHKKRZVIPXZIWIYAHCRXQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.949605774 (610h25m24.820688819s) [noise=true , address=77.90.31.14:4680 , piece_id=S2E4WBA7MJVAZLT5JG3IMHK53RQC77CCHV2PDF6L2R2JEA2U3K4Q, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.951459002 (610h25m24.822542048s) [noise=true , address=193.149.164.143:4704 , piece_id=BHXHFBKTLTDMGXWY5QRTP2THZQE5OXFZ2LCMTO2KRYRWAHXWW3NQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.952986742 (610h25m24.824069777s) [noise=true , address=77.90.31.19:4848 , piece_id=VZYG32S6ODIPDKQYZRXECBHXOERZDDWD2OF35Z46W4XMUZ3OJLAA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.956916654 (610h25m24.827999689s) [noise=true , address=145.79.227.29:4971 , piece_id=MEPXHHOQ6SUZC3GIPCQEMOKKF5OJTKZEI3UYZBAYXAQZGSVEKSBA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.960665062 (610h25m24.831748097s) [noise=true , address=64.49.13.170:4720 , piece_id=HIATRSAPE4YLSODFZDPP43ZUMLDVM7GJEVMWE4ZGUQTP25LTHGTA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.962366001 (610h25m24.833449036s) [noise=true , address=145.79.227.2:4117 , piece_id=4SGFGB3IJ7LTXDMMWM2PGGOV64MC46IZ7OZQWH55J4NBQY7CHMZQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.963708216 (610h25m24.834791261s) [noise=true , address=145.79.227.35:4505 , piece_id=V7RS7A4QLZTH4G32ZUZNSF2Z5APVWZGJO77NEGP3SBWQT3WCMVLQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:26.352236138 (610h25m26.223319173s) [upload=534952]: Next returned nil segment Dec 31 01:54:26.352253821 (610h25m26.223336857s) [upload=534952]: Waiting for error group managing segments... Dec 31 01:54:26.812758374 (610h25m26.683841409s) [noise=true , address=145.79.227.51:4443 , piece_id=73UUMRDY7BSZNCGBXCZYYRBVE5KLPMNPRNXOZEGWKQGLHJS32J2Q, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.824371056 (610h25m26.695454101s) [noise=true , address=145.79.227.14:4847 , piece_id=JX5NXTMVM3NSWIEVR4RX46ABSNPKTOFDLFBPKEXFQ3G3ANKV5OLA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.838414046 (610h25m26.709497081s) [noise=true , address=64.49.13.138:4490 , piece_id=2XTUZE6PXRE3OARNJVT36FXXWUMYIBVNE6M2VVFAVBGKEE2H5YZQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.875499841 (610h25m26.746582876s) [noise=true , address=64.49.13.139:4478 , piece_id=QN6RT2KWEA6APQVA66ZMKG4LH537UGYNOXOCVVA2YC2MKDOLT3YA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.891607221 (610h25m26.762690256s) [noise=true , address=145.79.227.20:4452 , piece_id=ZSDN2JZ3FQUVJYN5OURDXNKEDLDNULJOHKKRZVIPXZIWIYAHCRXQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.988561223 (610h25m26.859644268s) [noise=true , address=64.49.13.170:4720 , piece_id=HIATRSAPE4YLSODFZDPP43ZUMLDVM7GJEVMWE4ZGUQTP25LTHGTA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.039035911 (610h25m26.910118946s) [noise=true , address=77.90.31.14:4680 , piece_id=S2E4WBA7MJVAZLT5JG3IMHK53RQC77CCHV2PDF6L2R2JEA2U3K4Q, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.098023090 (610h25m26.969106126s) [noise=true , address=45.91.116.6:4981 , piece_id=CT474ZA2LVAZEBJHBDD4GB6NTHVHZFJLNE5OZV3DFS2NZKVW3EQQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.653790754 (610h25m27.524873789s) [noise=true , address=77.90.31.21:4886 , piece_id=K34WPIBGGGEPHIKJ253XGLDCIQ3WLFN4AZ6EYNWVHHT2EIHS6YRQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.684192972 (610h25m27.555276017s) [noise=true , address=193.149.164.143:4704 , piece_id=BHXHFBKTLTDMGXWY5QRTP2THZQE5OXFZ2LCMTO2KRYRWAHXWW3NQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.717238425 (610h25m27.58832146s) [noise=true , address=145.79.227.40:4084 , piece_id=RUKTKDII6BRUWSYVS65CWUBT3QKG3ASUK3KA3O5VGE53XSG6IMOQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.718349742 (610h25m27.589432777s) [noise=true , address=77.90.31.10:4523 , piece_id=P6LIONXJI6AFZ2JTKCQ2W6RZCOZCK6MZ53EXAJKXYQXSX2PX4QEA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.803639562 (610h25m27.674722597s) [noise=true , address=145.79.227.29:4971 , piece_id=MEPXHHOQ6SUZC3GIPCQEMOKKF5OJTKZEI3UYZBAYXAQZGSVEKSBA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.812791279 (610h25m27.683874314s) [noise=true , address=145.79.227.49:4343 , piece_id=6L3HPTS7D2NRA4QEOH4RFM3D3B53CTCONRI6G6YVU4OOTUWTG7XA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.812812348 (610h25m27.683895393s) [seg_pos={0 0}, upload=534952]: Segment reached optimal threshold of 14 pieces. Dec 31 01:54:27.813018000 (610h25m27.684101035s) [noise=true , address=145.79.227.35:4505 , piece_id=V7RS7A4QLZTH4G32ZUZNSF2Z5APVWZGJO77NEGP3SBWQT3WCMVLQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i6itGoBhLsLiqC8et8Y5L62fLjcWxy18u39NAr4R3nXNMWcsf): protocol: expected piece hash; context canceled; EOF Dec 31 01:54:27.813024772 (610h25m27.684107808s) [noise=true , address=145.79.227.35:4505 , piece_id=V7RS7A4QLZTH4G32ZUZNSF2Z5APVWZGJO77NEGP3SBWQT3WCMVLQ, seg_pos={0 0}, upload=534952]: Long tail cancellation due to optimal threshold Dec 31 01:54:27.813175931 (610h25m27.684258976s) [noise=true , address=145.79.227.34:4937 , piece_id=5W7LBM3YJJTVUFYTELR57DVXB663OHO5N3Z2SPQPZ4G43IK47O3Q, seg_pos={0 0}, upload=534952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1LKrFThdW3jJpG6dRZLUaxLXTa5Wy27yQ8PGUEqjvBWfFoRB94): protocol: expected piece hash; context canceled; EOF Dec 31 01:54:27.813181702 (610h25m27.684264737s) [noise=true , address=145.79.227.34:4937 , piece_id=5W7LBM3YJJTVUFYTELR57DVXB663OHO5N3Z2SPQPZ4G43IK47O3Q, seg_pos={0 0}, upload=534952]: Long tail cancellation due to optimal threshold Dec 31 01:54:27.813407862 (610h25m27.684490907s) [noise=true , address=145.79.227.2:4117 , piece_id=4SGFGB3IJ7LTXDMMWM2PGGOV64MC46IZ7OZQWH55J4NBQY7CHMZQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122Htg72f5eUUutf1p3fYQpy1Wkf6AgdaQV83HHYye2n8zm3the): protocol: expected piece hash; context canceled; EOF Dec 31 01:54:27.813414264 (610h25m27.684497299s) [noise=true , address=145.79.227.2:4117 , piece_id=4SGFGB3IJ7LTXDMMWM2PGGOV64MC46IZ7OZQWH55J4NBQY7CHMZQ, seg_pos={0 0}, upload=534952]: Long tail cancellation due to optimal threshold Dec 31 01:54:27.813424614 (610h25m27.684507649s) [noise=true , address=77.90.31.19:4848 , piece_id=VZYG32S6ODIPDKQYZRXECBHXOERZDDWD2OF35Z46W4XMUZ3OJLAA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD): protocol: write tcp 188.166.225.170:42842->77.90.31.19:4848: use of closed network connection; write tcp 188.166.225.170:42842->77.90.31.19:4848: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42842->77.90.31.19:4848: use of closed network connection Dec 31 01:54:27.813432750 (610h25m27.684515785s) [noise=true , address=77.90.31.19:4848 , piece_id=VZYG32S6ODIPDKQYZRXECBHXOERZDDWD2OF35Z46W4XMUZ3OJLAA, seg_pos={0 0}, upload=534952]: Long tail cancellation due to optimal threshold Dec 31 01:54:27.813474088 (610h25m27.684557123s) [seg_pos={0 0}, upload=534952]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 01:54:27.813535325 (610h25m27.68461836s) [upload=534952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 01:54:27.813539613 (610h25m27.684622648s) [upload=534952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 01:54:28.099921034 (610h25m27.971004069s) [upload=534952]: Done uploading segments... Dec 31 01:54:28.099940451 (610h25m27.971023486s) [upload=534952]: Upload finished. err: <nil> Duration: 15.485507144s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...........................X...............................X] duration: 5.001454693s trace: https://jaeger.dev.storj.io/trace/25cdf0a5222c0faa FAIL (2025-12-31 02:35:13 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001471395s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [..........X..X.......X.X..X....XX.XX........X.X......X..X...] duration: 5.762786412s trace: https://jaeger.dev.storj.io/trace/54231a38e027f2fe FAIL (2025-12-31 02:34:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: DCML4BWFWKJ2AEQVRGRYOQ7Q4YY3BF2PF4GFQSOREMAG67M4BJ5Q) context deadline exceeded; piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: YBK3PD3XVPSHJZTXOEGCXAGKJFRBLUSTFH6ZHYEFYK5YRIMLGISQ) context deadline exceeded ======LOGS====== Dec 31 02:31:42.009467393 (611h2m41.880550439s) [upload=535486]: Starting upload Dec 31 02:31:42.009568907 (611h2m41.880651942s) [upload=535486]: Done starting upload Dec 31 02:31:42.009677694 (611h2m41.880760729s) [upload=535486]: Uploading segments... Dec 31 02:31:42.009860923 (611h2m41.880943969s) [upload=535486]: Got next segment. Inline: false Dec 31 02:31:42.009921478 (611h2m41.881004513s) [upload=535486]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 02:31:42.009926968 (611h2m41.881010003s) [upload=535486]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 02:31:42.257755154 (611h2m42.128838189s) [seg_pos={0 0}, upload=535486]: Begin upload segment... Dec 31 02:31:42.257856827 (611h2m42.128939862s) [seg_pos={0 0}, upload=535486]: Done begin upload segment. Dec 31 02:31:42.257895601 (611h2m42.128978646s) [noise=true , address=45.91.116.6:4973 , piece_id=BVEV2ZTEUP3UUZ7H6WBZ4MHY2JESHTR6Z2HY3AAD5XCRWE7NLMMQ, seg_pos={0 0}, upload=535486]: Uploading piece... Dec 31 02:31:42.257927502 (611h2m42.129010537s) [noise=true , address=77.90.31.19:4844 , piece_id=DJPEVJBHVIZGFMBSUBTEWEIMU3JCSWPSKUXWPDQGXJKXLFGXQQGA, seg_pos={0 0}, upload=535486]: Uploading piece... Dec 31 02:31:42.258733707 (611h2m42.129816752s) [noise=true , address=145.79.227.52:4447 , piece_id=65XJCKQL2VIDCHMSHGUSCFNPU7JWFPKRF2QXWNAHQT7T5GELABFQ, seg_pos={0 0}, upload=535486]: Uploading piece... Dec 31 02:31:42.259561023 (611h2m42.130644058s) [noise=true , address=45.91.116.9:4262 , piece_id=DCML4BWFWKJ2AEQVRGRYOQ7Q4YY3BF2PF4GFQSOREMAG67M4BJ5Q, seg_pos={0 0}, upload=535486]: Uploading piece... Dec 31 02:31:42.259700759 (611h2m42.130783794s) [noise=true , address=45.91.116.11:4118 , piece_id=JAUSAGDUAQ4BBYWNA7OSW6XATK2JLMRUM6K6QNHD5EB4DPZNPQZA, seg_pos={0 0}, upload=535486]: Uploading piece... Dec 31 02:31:42.260044383 (611h2m42.131127428s) [noise=true , address=45.91.116.12:4225 , piece_id=YBK3PD3XVPSHJZTXOEGCXAGKJFRBLUSTFH6ZHYEFYK5YRIMLGISQ, seg_pos={0 0}, upload=535486]: Uploading piece... Dec 31 02:31:42.260541770 (611h2m42.131624815s) [noise=true , address=45.91.116.5:4299 , piece_id=GZH33CCTCZPZX5ZPHEXDKBI2YSUP4GZ7YUQ6FIIHLCBB24SRSFGA, seg_pos={0 0}, upload=535486]: Uploading piece... Dec 31 02:31:42.261116324 (611h2m42.132199369s) [noise=true , address=145.79.227.18:4007 , piece_id=HA2C5LN5ZB6TVJBQ6GVELU4JQG5KII7CFBTZAHDT75KJP5J7FOYA, seg_pos={0 0}, upload=535486]: Uploading piece... Dec 31 02:31:42.262541628 (611h2m42.133624674s) [noise=true , address=145.79.227.15:4876 , piece_id=WCIBKRCJCRS4JGFTWDQHVUZ2TLV3OYKTSMIBSBSU4WBFZ7WZNHOA, seg_pos={0 0}, upload=535486]: Uploading piece... Dec 31 02:31:42.262838263 (611h2m42.133921308s) [noise=true , address=145.79.227.3:4150 , piece_id=GKSGSQWFKTLK4B4ARKBXK2FR5IDBCOJMZZUOFEI2KASMXPJUHI4A, seg_pos={0 0}, upload=535486]: Uploading piece... Dec 31 02:31:42.263683141 (611h2m42.134766186s) [noise=true , address=77.90.31.10:4519 , piece_id=CZ5AVVMI74GHKCHLXL47OQED5DAR66LRFIVH25A3JL6K7NXNAOSA, seg_pos={0 0}, upload=535486]: Uploading piece... Dec 31 02:31:42.263989505 (611h2m42.13507254s) [noise=true , address=145.79.227.14:4837 , piece_id=HIU57ZYKPL6JYTQB27K2XG67GGYG7XM63OCJ4YLMMQXMLKYDCFJA, seg_pos={0 0}, upload=535486]: Uploading piece... Dec 31 02:31:42.264792113 (611h2m42.135875158s) [noise=true , address=145.79.227.20:4438 , piece_id=MNT67N5BTYQB2JUNXV7TVSRL3DJMMTBXYQO4BSWEFALCWMK2ZP5Q, seg_pos={0 0}, upload=535486]: Uploading piece... Dec 31 02:31:42.265209829 (611h2m42.136292864s) [noise=true , address=145.79.227.39:4014 , piece_id=R3DDMHHRZSHJUDSEXG3BUD2SOIVBXYAJE6RRMTPKW6XAPT7CNLTQ, seg_pos={0 0}, upload=535486]: Uploading piece... Dec 31 02:31:42.266296698 (611h2m42.137379733s) [noise=true , address=145.79.227.24:4581 , piece_id=OPPGQUC2NXAGBLYXH5VM4RGY6ATR5DAUJAGRRKAQCZ3ENMNTMLBA, seg_pos={0 0}, upload=535486]: Uploading piece... Dec 31 02:31:42.270084160 (611h2m42.141167195s) [noise=true , address=145.79.227.21:4474 , piece_id=HBUEUQTYMITUOKLQEQHAOSCBHCVXCWGCUM4WDPC4ZMFKRQCW3GRA, seg_pos={0 0}, upload=535486]: Uploading piece... Dec 31 02:31:43.955549810 (611h2m43.826632865s) [upload=535486]: Next returned nil segment Dec 31 02:31:43.955567964 (611h2m43.826651009s) [upload=535486]: Waiting for error group managing segments... Dec 31 02:31:44.239121474 (611h2m44.110204519s) [noise=true , address=145.79.227.18:4007 , piece_id=HA2C5LN5ZB6TVJBQ6GVELU4JQG5KII7CFBTZAHDT75KJP5J7FOYA, seg_pos={0 0}, upload=535486]: Done uploading piece. err: <nil> Dec 31 02:31:44.247474961 (611h2m44.118557996s) [noise=true , address=145.79.227.24:4581 , piece_id=OPPGQUC2NXAGBLYXH5VM4RGY6ATR5DAUJAGRRKAQCZ3ENMNTMLBA, seg_pos={0 0}, upload=535486]: Done uploading piece. err: <nil> Dec 31 02:31:44.249933482 (611h2m44.121016527s) [noise=true , address=145.79.227.39:4014 , piece_id=R3DDMHHRZSHJUDSEXG3BUD2SOIVBXYAJE6RRMTPKW6XAPT7CNLTQ, seg_pos={0 0}, upload=535486]: Done uploading piece. err: <nil> Dec 31 02:31:44.250394820 (611h2m44.121477855s) [noise=true , address=145.79.227.52:4447 , piece_id=65XJCKQL2VIDCHMSHGUSCFNPU7JWFPKRF2QXWNAHQT7T5GELABFQ, seg_pos={0 0}, upload=535486]: Done uploading piece. err: <nil> Dec 31 02:31:44.260864926 (611h2m44.131947971s) [noise=true , address=145.79.227.3:4150 , piece_id=GKSGSQWFKTLK4B4ARKBXK2FR5IDBCOJMZZUOFEI2KASMXPJUHI4A, seg_pos={0 0}, upload=535486]: Done uploading piece. err: <nil> Dec 31 02:31:44.265136891 (611h2m44.136219936s) [noise=true , address=145.79.227.15:4876 , piece_id=WCIBKRCJCRS4JGFTWDQHVUZ2TLV3OYKTSMIBSBSU4WBFZ7WZNHOA, seg_pos={0 0}, upload=535486]: Done uploading piece. err: <nil> Dec 31 02:31:44.274014796 (611h2m44.145097831s) [noise=true , address=145.79.227.20:4438 , piece_id=MNT67N5BTYQB2JUNXV7TVSRL3DJMMTBXYQO4BSWEFALCWMK2ZP5Q, seg_pos={0 0}, upload=535486]: Done uploading piece. err: <nil> Dec 31 02:31:44.275430611 (611h2m44.146513646s) [noise=true , address=145.79.227.21:4474 , piece_id=HBUEUQTYMITUOKLQEQHAOSCBHCVXCWGCUM4WDPC4ZMFKRQCW3GRA, seg_pos={0 0}, upload=535486]: Done uploading piece. err: <nil> Dec 31 02:31:44.276265651 (611h2m44.147348686s) [noise=true , address=145.79.227.14:4837 , piece_id=HIU57ZYKPL6JYTQB27K2XG67GGYG7XM63OCJ4YLMMQXMLKYDCFJA, seg_pos={0 0}, upload=535486]: Done uploading piece. err: <nil> Dec 31 02:31:44.555968520 (611h2m44.427051555s) [noise=true , address=45.91.116.12:4225 , piece_id=YBK3PD3XVPSHJZTXOEGCXAGKJFRBLUSTFH6ZHYEFYK5YRIMLGISQ, seg_pos={0 0}, upload=535486]: Done uploading piece. err: <nil> Dec 31 02:31:44.563093667 (611h2m44.434176702s) [noise=true , address=77.90.31.10:4519 , piece_id=CZ5AVVMI74GHKCHLXL47OQED5DAR66LRFIVH25A3JL6K7NXNAOSA, seg_pos={0 0}, upload=535486]: Done uploading piece. err: <nil> Dec 31 02:31:44.625636847 (611h2m44.496719882s) [noise=true , address=45.91.116.9:4262 , piece_id=DCML4BWFWKJ2AEQVRGRYOQ7Q4YY3BF2PF4GFQSOREMAG67M4BJ5Q, seg_pos={0 0}, upload=535486]: Done uploading piece. err: <nil> Dec 31 02:31:44.717693032 (611h2m44.588776077s) [noise=true , address=45.91.116.6:4973 , piece_id=BVEV2ZTEUP3UUZ7H6WBZ4MHY2JESHTR6Z2HY3AAD5XCRWE7NLMMQ, seg_pos={0 0}, upload=535486]: Done uploading piece. err: <nil> Dec 31 02:31:44.717715566 (611h2m44.588798611s) [seg_pos={0 0}, upload=535486]: Segment reached optimal threshold of 13 pieces. Dec 31 02:31:44.718014545 (611h2m44.58909759s) [noise=true , address=45.91.116.11:4118 , piece_id=JAUSAGDUAQ4BBYWNA7OSW6XATK2JLMRUM6K6QNHD5EB4DPZNPQZA, seg_pos={0 0}, upload=535486]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF Dec 31 02:31:44.718024604 (611h2m44.589107639s) [noise=true , address=45.91.116.11:4118 , piece_id=JAUSAGDUAQ4BBYWNA7OSW6XATK2JLMRUM6K6QNHD5EB4DPZNPQZA, seg_pos={0 0}, upload=535486]: Long tail cancellation due to optimal threshold Dec 31 02:31:44.718138050 (611h2m44.589221125s) [noise=true , address=77.90.31.19:4844 , piece_id=DJPEVJBHVIZGFMBSUBTEWEIMU3JCSWPSKUXWPDQGXJKXLFGXQQGA, seg_pos={0 0}, upload=535486]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF Dec 31 02:31:44.718145925 (611h2m44.58922897s) [noise=true , address=77.90.31.19:4844 , piece_id=DJPEVJBHVIZGFMBSUBTEWEIMU3JCSWPSKUXWPDQGXJKXLFGXQQGA, seg_pos={0 0}, upload=535486]: Long tail cancellation due to optimal threshold Dec 31 02:31:44.718182945 (611h2m44.58926598s) [noise=true , address=45.91.116.5:4299 , piece_id=GZH33CCTCZPZX5ZPHEXDKBI2YSUP4GZ7YUQ6FIIHLCBB24SRSFGA, seg_pos={0 0}, upload=535486]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Dec 31 02:31:44.718191292 (611h2m44.589274327s) [noise=true , address=45.91.116.5:4299 , piece_id=GZH33CCTCZPZX5ZPHEXDKBI2YSUP4GZ7YUQ6FIIHLCBB24SRSFGA, seg_pos={0 0}, upload=535486]: Long tail cancellation due to optimal threshold Dec 31 02:31:44.718248981 (611h2m44.589332016s) [seg_pos={0 0}, upload=535486]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 02:31:44.718376454 (611h2m44.589459519s) [upload=535486]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 02:31:44.718437000 (611h2m44.589520045s) [upload=535486]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 02:31:45.017559155 (611h2m44.8886422s) [upload=535486]: Done uploading segments... Dec 31 02:31:45.017585174 (611h2m44.888668209s) [upload=535486]: Upload finished. err: <nil> Duration: 5.762816428s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 15.582074555s trace: https://jaeger.dev.storj.io/trace/365693fea00df28 FAIL (2025-12-31 02:34:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [noise=true , address=145.79.227.49:4343 , piece_id=6L3HPTS7D2NRA4QEOH4RFM3D3B53CTCONRI6G6YVU4OOTUWTG7XA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.942060847 (610h25m24.813143892s) [noise=true , address=145.79.227.34:4937 , piece_id=5W7LBM3YJJTVUFYTELR57DVXB663OHO5N3Z2SPQPZ4G43IK47O3Q, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.942684104 (610h25m24.813767139s) [noise=true , address=64.49.13.139:4478 , piece_id=QN6RT2KWEA6APQVA66ZMKG4LH537UGYNOXOCVVA2YC2MKDOLT3YA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.943064007 (610h25m24.814147042s) [noise=true , address=145.79.227.40:4084 , piece_id=RUKTKDII6BRUWSYVS65CWUBT3QKG3ASUK3KA3O5VGE53XSG6IMOQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.943459560 (610h25m24.814542605s) [noise=true , address=145.79.227.14:4847 , piece_id=JX5NXTMVM3NSWIEVR4RX46ABSNPKTOFDLFBPKEXFQ3G3ANKV5OLA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.943865303 (610h25m24.814948348s) [noise=true , address=45.91.116.6:4981 , piece_id=CT474ZA2LVAZEBJHBDD4GB6NTHVHZFJLNE5OZV3DFS2NZKVW3EQQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.944259694 (610h25m24.815342749s) [noise=true , address=145.79.227.51:4443 , piece_id=73UUMRDY7BSZNCGBXCZYYRBVE5KLPMNPRNXOZEGWKQGLHJS32J2Q, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.946575182 (610h25m24.817658227s) [noise=true , address=77.90.31.10:4523 , piece_id=P6LIONXJI6AFZ2JTKCQ2W6RZCOZCK6MZ53EXAJKXYQXSX2PX4QEA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.948132909 (610h25m24.819215944s) [noise=true , address=145.79.227.20:4452 , piece_id=ZSDN2JZ3FQUVJYN5OURDXNKEDLDNULJOHKKRZVIPXZIWIYAHCRXQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.949605774 (610h25m24.820688819s) [noise=true , address=77.90.31.14:4680 , piece_id=S2E4WBA7MJVAZLT5JG3IMHK53RQC77CCHV2PDF6L2R2JEA2U3K4Q, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.951459002 (610h25m24.822542048s) [noise=true , address=193.149.164.143:4704 , piece_id=BHXHFBKTLTDMGXWY5QRTP2THZQE5OXFZ2LCMTO2KRYRWAHXWW3NQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.952986742 (610h25m24.824069777s) [noise=true , address=77.90.31.19:4848 , piece_id=VZYG32S6ODIPDKQYZRXECBHXOERZDDWD2OF35Z46W4XMUZ3OJLAA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.956916654 (610h25m24.827999689s) [noise=true , address=145.79.227.29:4971 , piece_id=MEPXHHOQ6SUZC3GIPCQEMOKKF5OJTKZEI3UYZBAYXAQZGSVEKSBA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.960665062 (610h25m24.831748097s) [noise=true , address=64.49.13.170:4720 , piece_id=HIATRSAPE4YLSODFZDPP43ZUMLDVM7GJEVMWE4ZGUQTP25LTHGTA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.962366001 (610h25m24.833449036s) [noise=true , address=145.79.227.2:4117 , piece_id=4SGFGB3IJ7LTXDMMWM2PGGOV64MC46IZ7OZQWH55J4NBQY7CHMZQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.963708216 (610h25m24.834791261s) [noise=true , address=145.79.227.35:4505 , piece_id=V7RS7A4QLZTH4G32ZUZNSF2Z5APVWZGJO77NEGP3SBWQT3WCMVLQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:26.352236138 (610h25m26.223319173s) [upload=534952]: Next returned nil segment Dec 31 01:54:26.352253821 (610h25m26.223336857s) [upload=534952]: Waiting for error group managing segments... Dec 31 01:54:26.812758374 (610h25m26.683841409s) [noise=true , address=145.79.227.51:4443 , piece_id=73UUMRDY7BSZNCGBXCZYYRBVE5KLPMNPRNXOZEGWKQGLHJS32J2Q, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.824371056 (610h25m26.695454101s) [noise=true , address=145.79.227.14:4847 , piece_id=JX5NXTMVM3NSWIEVR4RX46ABSNPKTOFDLFBPKEXFQ3G3ANKV5OLA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.838414046 (610h25m26.709497081s) [noise=true , address=64.49.13.138:4490 , piece_id=2XTUZE6PXRE3OARNJVT36FXXWUMYIBVNE6M2VVFAVBGKEE2H5YZQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.875499841 (610h25m26.746582876s) [noise=true , address=64.49.13.139:4478 , piece_id=QN6RT2KWEA6APQVA66ZMKG4LH537UGYNOXOCVVA2YC2MKDOLT3YA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.891607221 (610h25m26.762690256s) [noise=true , address=145.79.227.20:4452 , piece_id=ZSDN2JZ3FQUVJYN5OURDXNKEDLDNULJOHKKRZVIPXZIWIYAHCRXQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.988561223 (610h25m26.859644268s) [noise=true , address=64.49.13.170:4720 , piece_id=HIATRSAPE4YLSODFZDPP43ZUMLDVM7GJEVMWE4ZGUQTP25LTHGTA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.039035911 (610h25m26.910118946s) [noise=true , address=77.90.31.14:4680 , piece_id=S2E4WBA7MJVAZLT5JG3IMHK53RQC77CCHV2PDF6L2R2JEA2U3K4Q, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.098023090 (610h25m26.969106126s) [noise=true , address=45.91.116.6:4981 , piece_id=CT474ZA2LVAZEBJHBDD4GB6NTHVHZFJLNE5OZV3DFS2NZKVW3EQQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.653790754 (610h25m27.524873789s) [noise=true , address=77.90.31.21:4886 , piece_id=K34WPIBGGGEPHIKJ253XGLDCIQ3WLFN4AZ6EYNWVHHT2EIHS6YRQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.684192972 (610h25m27.555276017s) [noise=true , address=193.149.164.143:4704 , piece_id=BHXHFBKTLTDMGXWY5QRTP2THZQE5OXFZ2LCMTO2KRYRWAHXWW3NQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.717238425 (610h25m27.58832146s) [noise=true , address=145.79.227.40:4084 , piece_id=RUKTKDII6BRUWSYVS65CWUBT3QKG3ASUK3KA3O5VGE53XSG6IMOQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.718349742 (610h25m27.589432777s) [noise=true , address=77.90.31.10:4523 , piece_id=P6LIONXJI6AFZ2JTKCQ2W6RZCOZCK6MZ53EXAJKXYQXSX2PX4QEA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.803639562 (610h25m27.674722597s) [noise=true , address=145.79.227.29:4971 , piece_id=MEPXHHOQ6SUZC3GIPCQEMOKKF5OJTKZEI3UYZBAYXAQZGSVEKSBA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.812791279 (610h25m27.683874314s) [noise=true , address=145.79.227.49:4343 , piece_id=6L3HPTS7D2NRA4QEOH4RFM3D3B53CTCONRI6G6YVU4OOTUWTG7XA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.812812348 (610h25m27.683895393s) [seg_pos={0 0}, upload=534952]: Segment reached optimal threshold of 14 pieces. Dec 31 01:54:27.813018000 (610h25m27.684101035s) [noise=true , address=145.79.227.35:4505 , piece_id=V7RS7A4QLZTH4G32ZUZNSF2Z5APVWZGJO77NEGP3SBWQT3WCMVLQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i6itGoBhLsLiqC8et8Y5L62fLjcWxy18u39NAr4R3nXNMWcsf): protocol: expected piece hash; context canceled; EOF Dec 31 01:54:27.813024772 (610h25m27.684107808s) [noise=true , address=145.79.227.35:4505 , piece_id=V7RS7A4QLZTH4G32ZUZNSF2Z5APVWZGJO77NEGP3SBWQT3WCMVLQ, seg_pos={0 0}, upload=534952]: Long tail cancellation due to optimal threshold Dec 31 01:54:27.813175931 (610h25m27.684258976s) [noise=true , address=145.79.227.34:4937 , piece_id=5W7LBM3YJJTVUFYTELR57DVXB663OHO5N3Z2SPQPZ4G43IK47O3Q, seg_pos={0 0}, upload=534952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1LKrFThdW3jJpG6dRZLUaxLXTa5Wy27yQ8PGUEqjvBWfFoRB94): protocol: expected piece hash; context canceled; EOF Dec 31 01:54:27.813181702 (610h25m27.684264737s) [noise=true , address=145.79.227.34:4937 , piece_id=5W7LBM3YJJTVUFYTELR57DVXB663OHO5N3Z2SPQPZ4G43IK47O3Q, seg_pos={0 0}, upload=534952]: Long tail cancellation due to optimal threshold Dec 31 01:54:27.813407862 (610h25m27.684490907s) [noise=true , address=145.79.227.2:4117 , piece_id=4SGFGB3IJ7LTXDMMWM2PGGOV64MC46IZ7OZQWH55J4NBQY7CHMZQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122Htg72f5eUUutf1p3fYQpy1Wkf6AgdaQV83HHYye2n8zm3the): protocol: expected piece hash; context canceled; EOF Dec 31 01:54:27.813414264 (610h25m27.684497299s) [noise=true , address=145.79.227.2:4117 , piece_id=4SGFGB3IJ7LTXDMMWM2PGGOV64MC46IZ7OZQWH55J4NBQY7CHMZQ, seg_pos={0 0}, upload=534952]: Long tail cancellation due to optimal threshold Dec 31 01:54:27.813424614 (610h25m27.684507649s) [noise=true , address=77.90.31.19:4848 , piece_id=VZYG32S6ODIPDKQYZRXECBHXOERZDDWD2OF35Z46W4XMUZ3OJLAA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD): protocol: write tcp 188.166.225.170:42842->77.90.31.19:4848: use of closed network connection; write tcp 188.166.225.170:42842->77.90.31.19:4848: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42842->77.90.31.19:4848: use of closed network connection Dec 31 01:54:27.813432750 (610h25m27.684515785s) [noise=true , address=77.90.31.19:4848 , piece_id=VZYG32S6ODIPDKQYZRXECBHXOERZDDWD2OF35Z46W4XMUZ3OJLAA, seg_pos={0 0}, upload=534952]: Long tail cancellation due to optimal threshold Dec 31 01:54:27.813474088 (610h25m27.684557123s) [seg_pos={0 0}, upload=534952]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 01:54:27.813535325 (610h25m27.68461836s) [upload=534952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 01:54:27.813539613 (610h25m27.684622648s) [upload=534952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 01:54:28.099921034 (610h25m27.971004069s) [upload=534952]: Done uploading segments... Dec 31 01:54:28.099940451 (610h25m27.971023486s) [upload=534952]: Upload finished. err: <nil> Duration: 15.58208803s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............................X...............................] duration: 3.588562278s trace: https://jaeger.dev.storj.io/trace/740aa405bd68150f FAIL (2025-12-31 02:34:21 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.588590211s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 59.720180012s trace: https://jaeger.dev.storj.io/trace/365693fea00df28 FAIL (2025-12-31 02:34:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [noise=true , address=145.79.227.49:4343 , piece_id=6L3HPTS7D2NRA4QEOH4RFM3D3B53CTCONRI6G6YVU4OOTUWTG7XA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.942060847 (610h25m24.813143892s) [noise=true , address=145.79.227.34:4937 , piece_id=5W7LBM3YJJTVUFYTELR57DVXB663OHO5N3Z2SPQPZ4G43IK47O3Q, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.942684104 (610h25m24.813767139s) [noise=true , address=64.49.13.139:4478 , piece_id=QN6RT2KWEA6APQVA66ZMKG4LH537UGYNOXOCVVA2YC2MKDOLT3YA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.943064007 (610h25m24.814147042s) [noise=true , address=145.79.227.40:4084 , piece_id=RUKTKDII6BRUWSYVS65CWUBT3QKG3ASUK3KA3O5VGE53XSG6IMOQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.943459560 (610h25m24.814542605s) [noise=true , address=145.79.227.14:4847 , piece_id=JX5NXTMVM3NSWIEVR4RX46ABSNPKTOFDLFBPKEXFQ3G3ANKV5OLA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.943865303 (610h25m24.814948348s) [noise=true , address=45.91.116.6:4981 , piece_id=CT474ZA2LVAZEBJHBDD4GB6NTHVHZFJLNE5OZV3DFS2NZKVW3EQQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.944259694 (610h25m24.815342749s) [noise=true , address=145.79.227.51:4443 , piece_id=73UUMRDY7BSZNCGBXCZYYRBVE5KLPMNPRNXOZEGWKQGLHJS32J2Q, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.946575182 (610h25m24.817658227s) [noise=true , address=77.90.31.10:4523 , piece_id=P6LIONXJI6AFZ2JTKCQ2W6RZCOZCK6MZ53EXAJKXYQXSX2PX4QEA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.948132909 (610h25m24.819215944s) [noise=true , address=145.79.227.20:4452 , piece_id=ZSDN2JZ3FQUVJYN5OURDXNKEDLDNULJOHKKRZVIPXZIWIYAHCRXQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.949605774 (610h25m24.820688819s) [noise=true , address=77.90.31.14:4680 , piece_id=S2E4WBA7MJVAZLT5JG3IMHK53RQC77CCHV2PDF6L2R2JEA2U3K4Q, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.951459002 (610h25m24.822542048s) [noise=true , address=193.149.164.143:4704 , piece_id=BHXHFBKTLTDMGXWY5QRTP2THZQE5OXFZ2LCMTO2KRYRWAHXWW3NQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.952986742 (610h25m24.824069777s) [noise=true , address=77.90.31.19:4848 , piece_id=VZYG32S6ODIPDKQYZRXECBHXOERZDDWD2OF35Z46W4XMUZ3OJLAA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.956916654 (610h25m24.827999689s) [noise=true , address=145.79.227.29:4971 , piece_id=MEPXHHOQ6SUZC3GIPCQEMOKKF5OJTKZEI3UYZBAYXAQZGSVEKSBA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.960665062 (610h25m24.831748097s) [noise=true , address=64.49.13.170:4720 , piece_id=HIATRSAPE4YLSODFZDPP43ZUMLDVM7GJEVMWE4ZGUQTP25LTHGTA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.962366001 (610h25m24.833449036s) [noise=true , address=145.79.227.2:4117 , piece_id=4SGFGB3IJ7LTXDMMWM2PGGOV64MC46IZ7OZQWH55J4NBQY7CHMZQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.963708216 (610h25m24.834791261s) [noise=true , address=145.79.227.35:4505 , piece_id=V7RS7A4QLZTH4G32ZUZNSF2Z5APVWZGJO77NEGP3SBWQT3WCMVLQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:26.352236138 (610h25m26.223319173s) [upload=534952]: Next returned nil segment Dec 31 01:54:26.352253821 (610h25m26.223336857s) [upload=534952]: Waiting for error group managing segments... Dec 31 01:54:26.812758374 (610h25m26.683841409s) [noise=true , address=145.79.227.51:4443 , piece_id=73UUMRDY7BSZNCGBXCZYYRBVE5KLPMNPRNXOZEGWKQGLHJS32J2Q, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.824371056 (610h25m26.695454101s) [noise=true , address=145.79.227.14:4847 , piece_id=JX5NXTMVM3NSWIEVR4RX46ABSNPKTOFDLFBPKEXFQ3G3ANKV5OLA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.838414046 (610h25m26.709497081s) [noise=true , address=64.49.13.138:4490 , piece_id=2XTUZE6PXRE3OARNJVT36FXXWUMYIBVNE6M2VVFAVBGKEE2H5YZQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.875499841 (610h25m26.746582876s) [noise=true , address=64.49.13.139:4478 , piece_id=QN6RT2KWEA6APQVA66ZMKG4LH537UGYNOXOCVVA2YC2MKDOLT3YA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.891607221 (610h25m26.762690256s) [noise=true , address=145.79.227.20:4452 , piece_id=ZSDN2JZ3FQUVJYN5OURDXNKEDLDNULJOHKKRZVIPXZIWIYAHCRXQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.988561223 (610h25m26.859644268s) [noise=true , address=64.49.13.170:4720 , piece_id=HIATRSAPE4YLSODFZDPP43ZUMLDVM7GJEVMWE4ZGUQTP25LTHGTA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.039035911 (610h25m26.910118946s) [noise=true , address=77.90.31.14:4680 , piece_id=S2E4WBA7MJVAZLT5JG3IMHK53RQC77CCHV2PDF6L2R2JEA2U3K4Q, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.098023090 (610h25m26.969106126s) [noise=true , address=45.91.116.6:4981 , piece_id=CT474ZA2LVAZEBJHBDD4GB6NTHVHZFJLNE5OZV3DFS2NZKVW3EQQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.653790754 (610h25m27.524873789s) [noise=true , address=77.90.31.21:4886 , piece_id=K34WPIBGGGEPHIKJ253XGLDCIQ3WLFN4AZ6EYNWVHHT2EIHS6YRQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.684192972 (610h25m27.555276017s) [noise=true , address=193.149.164.143:4704 , piece_id=BHXHFBKTLTDMGXWY5QRTP2THZQE5OXFZ2LCMTO2KRYRWAHXWW3NQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.717238425 (610h25m27.58832146s) [noise=true , address=145.79.227.40:4084 , piece_id=RUKTKDII6BRUWSYVS65CWUBT3QKG3ASUK3KA3O5VGE53XSG6IMOQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.718349742 (610h25m27.589432777s) [noise=true , address=77.90.31.10:4523 , piece_id=P6LIONXJI6AFZ2JTKCQ2W6RZCOZCK6MZ53EXAJKXYQXSX2PX4QEA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.803639562 (610h25m27.674722597s) [noise=true , address=145.79.227.29:4971 , piece_id=MEPXHHOQ6SUZC3GIPCQEMOKKF5OJTKZEI3UYZBAYXAQZGSVEKSBA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.812791279 (610h25m27.683874314s) [noise=true , address=145.79.227.49:4343 , piece_id=6L3HPTS7D2NRA4QEOH4RFM3D3B53CTCONRI6G6YVU4OOTUWTG7XA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.812812348 (610h25m27.683895393s) [seg_pos={0 0}, upload=534952]: Segment reached optimal threshold of 14 pieces. Dec 31 01:54:27.813018000 (610h25m27.684101035s) [noise=true , address=145.79.227.35:4505 , piece_id=V7RS7A4QLZTH4G32ZUZNSF2Z5APVWZGJO77NEGP3SBWQT3WCMVLQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i6itGoBhLsLiqC8et8Y5L62fLjcWxy18u39NAr4R3nXNMWcsf): protocol: expected piece hash; context canceled; EOF Dec 31 01:54:27.813024772 (610h25m27.684107808s) [noise=true , address=145.79.227.35:4505 , piece_id=V7RS7A4QLZTH4G32ZUZNSF2Z5APVWZGJO77NEGP3SBWQT3WCMVLQ, seg_pos={0 0}, upload=534952]: Long tail cancellation due to optimal threshold Dec 31 01:54:27.813175931 (610h25m27.684258976s) [noise=true , address=145.79.227.34:4937 , piece_id=5W7LBM3YJJTVUFYTELR57DVXB663OHO5N3Z2SPQPZ4G43IK47O3Q, seg_pos={0 0}, upload=534952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1LKrFThdW3jJpG6dRZLUaxLXTa5Wy27yQ8PGUEqjvBWfFoRB94): protocol: expected piece hash; context canceled; EOF Dec 31 01:54:27.813181702 (610h25m27.684264737s) [noise=true , address=145.79.227.34:4937 , piece_id=5W7LBM3YJJTVUFYTELR57DVXB663OHO5N3Z2SPQPZ4G43IK47O3Q, seg_pos={0 0}, upload=534952]: Long tail cancellation due to optimal threshold Dec 31 01:54:27.813407862 (610h25m27.684490907s) [noise=true , address=145.79.227.2:4117 , piece_id=4SGFGB3IJ7LTXDMMWM2PGGOV64MC46IZ7OZQWH55J4NBQY7CHMZQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122Htg72f5eUUutf1p3fYQpy1Wkf6AgdaQV83HHYye2n8zm3the): protocol: expected piece hash; context canceled; EOF Dec 31 01:54:27.813414264 (610h25m27.684497299s) [noise=true , address=145.79.227.2:4117 , piece_id=4SGFGB3IJ7LTXDMMWM2PGGOV64MC46IZ7OZQWH55J4NBQY7CHMZQ, seg_pos={0 0}, upload=534952]: Long tail cancellation due to optimal threshold Dec 31 01:54:27.813424614 (610h25m27.684507649s) [noise=true , address=77.90.31.19:4848 , piece_id=VZYG32S6ODIPDKQYZRXECBHXOERZDDWD2OF35Z46W4XMUZ3OJLAA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD): protocol: write tcp 188.166.225.170:42842->77.90.31.19:4848: use of closed network connection; write tcp 188.166.225.170:42842->77.90.31.19:4848: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42842->77.90.31.19:4848: use of closed network connection Dec 31 01:54:27.813432750 (610h25m27.684515785s) [noise=true , address=77.90.31.19:4848 , piece_id=VZYG32S6ODIPDKQYZRXECBHXOERZDDWD2OF35Z46W4XMUZ3OJLAA, seg_pos={0 0}, upload=534952]: Long tail cancellation due to optimal threshold Dec 31 01:54:27.813474088 (610h25m27.684557123s) [seg_pos={0 0}, upload=534952]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 01:54:27.813535325 (610h25m27.68461836s) [upload=534952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 01:54:27.813539613 (610h25m27.684622648s) [upload=534952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 01:54:28.099921034 (610h25m27.971004069s) [upload=534952]: Done uploading segments... Dec 31 01:54:28.099940451 (610h25m27.971023486s) [upload=534952]: Upload finished. err: <nil> Duration: 59.720201863s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............................X..............................] duration: 5.001506304s trace: https://jaeger.dev.storj.io/trace/740aa405bd68150f FAIL (2025-12-31 02:33:48 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001522995s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [...........X..X.......X.X..X....XX.XX........X.X......X..X..] duration: 5.706786551s trace: https://jaeger.dev.storj.io/trace/54231a38e027f2fe FAIL (2025-12-31 02:33:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: DCML4BWFWKJ2AEQVRGRYOQ7Q4YY3BF2PF4GFQSOREMAG67M4BJ5Q) context deadline exceeded; piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: YBK3PD3XVPSHJZTXOEGCXAGKJFRBLUSTFH6ZHYEFYK5YRIMLGISQ) context deadline exceeded ======LOGS====== Dec 31 02:31:42.009467393 (611h2m41.880550439s) [upload=535486]: Starting upload Dec 31 02:31:42.009568907 (611h2m41.880651942s) [upload=535486]: Done starting upload Dec 31 02:31:42.009677694 (611h2m41.880760729s) [upload=535486]: Uploading segments... Dec 31 02:31:42.009860923 (611h2m41.880943969s) [upload=535486]: Got next segment. Inline: false Dec 31 02:31:42.009921478 (611h2m41.881004513s) [upload=535486]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 02:31:42.009926968 (611h2m41.881010003s) [upload=535486]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 02:31:42.257755154 (611h2m42.128838189s) [seg_pos={0 0}, upload=535486]: Begin upload segment... Dec 31 02:31:42.257856827 (611h2m42.128939862s) [seg_pos={0 0}, upload=535486]: Done begin upload segment. Dec 31 02:31:42.257895601 (611h2m42.128978646s) [noise=true , address=45.91.116.6:4973 , piece_id=BVEV2ZTEUP3UUZ7H6WBZ4MHY2JESHTR6Z2HY3AAD5XCRWE7NLMMQ, seg_pos={0 0}, upload=535486]: Uploading piece... Dec 31 02:31:42.257927502 (611h2m42.129010537s) [noise=true , address=77.90.31.19:4844 , piece_id=DJPEVJBHVIZGFMBSUBTEWEIMU3JCSWPSKUXWPDQGXJKXLFGXQQGA, seg_pos={0 0}, upload=535486]: Uploading piece... Dec 31 02:31:42.258733707 (611h2m42.129816752s) [noise=true , address=145.79.227.52:4447 , piece_id=65XJCKQL2VIDCHMSHGUSCFNPU7JWFPKRF2QXWNAHQT7T5GELABFQ, seg_pos={0 0}, upload=535486]: Uploading piece... Dec 31 02:31:42.259561023 (611h2m42.130644058s) [noise=true , address=45.91.116.9:4262 , piece_id=DCML4BWFWKJ2AEQVRGRYOQ7Q4YY3BF2PF4GFQSOREMAG67M4BJ5Q, seg_pos={0 0}, upload=535486]: Uploading piece... Dec 31 02:31:42.259700759 (611h2m42.130783794s) [noise=true , address=45.91.116.11:4118 , piece_id=JAUSAGDUAQ4BBYWNA7OSW6XATK2JLMRUM6K6QNHD5EB4DPZNPQZA, seg_pos={0 0}, upload=535486]: Uploading piece... Dec 31 02:31:42.260044383 (611h2m42.131127428s) [noise=true , address=45.91.116.12:4225 , piece_id=YBK3PD3XVPSHJZTXOEGCXAGKJFRBLUSTFH6ZHYEFYK5YRIMLGISQ, seg_pos={0 0}, upload=535486]: Uploading piece... Dec 31 02:31:42.260541770 (611h2m42.131624815s) [noise=true , address=45.91.116.5:4299 , piece_id=GZH33CCTCZPZX5ZPHEXDKBI2YSUP4GZ7YUQ6FIIHLCBB24SRSFGA, seg_pos={0 0}, upload=535486]: Uploading piece... Dec 31 02:31:42.261116324 (611h2m42.132199369s) [noise=true , address=145.79.227.18:4007 , piece_id=HA2C5LN5ZB6TVJBQ6GVELU4JQG5KII7CFBTZAHDT75KJP5J7FOYA, seg_pos={0 0}, upload=535486]: Uploading piece... Dec 31 02:31:42.262541628 (611h2m42.133624674s) [noise=true , address=145.79.227.15:4876 , piece_id=WCIBKRCJCRS4JGFTWDQHVUZ2TLV3OYKTSMIBSBSU4WBFZ7WZNHOA, seg_pos={0 0}, upload=535486]: Uploading piece... Dec 31 02:31:42.262838263 (611h2m42.133921308s) [noise=true , address=145.79.227.3:4150 , piece_id=GKSGSQWFKTLK4B4ARKBXK2FR5IDBCOJMZZUOFEI2KASMXPJUHI4A, seg_pos={0 0}, upload=535486]: Uploading piece... Dec 31 02:31:42.263683141 (611h2m42.134766186s) [noise=true , address=77.90.31.10:4519 , piece_id=CZ5AVVMI74GHKCHLXL47OQED5DAR66LRFIVH25A3JL6K7NXNAOSA, seg_pos={0 0}, upload=535486]: Uploading piece... Dec 31 02:31:42.263989505 (611h2m42.13507254s) [noise=true , address=145.79.227.14:4837 , piece_id=HIU57ZYKPL6JYTQB27K2XG67GGYG7XM63OCJ4YLMMQXMLKYDCFJA, seg_pos={0 0}, upload=535486]: Uploading piece... Dec 31 02:31:42.264792113 (611h2m42.135875158s) [noise=true , address=145.79.227.20:4438 , piece_id=MNT67N5BTYQB2JUNXV7TVSRL3DJMMTBXYQO4BSWEFALCWMK2ZP5Q, seg_pos={0 0}, upload=535486]: Uploading piece... Dec 31 02:31:42.265209829 (611h2m42.136292864s) [noise=true , address=145.79.227.39:4014 , piece_id=R3DDMHHRZSHJUDSEXG3BUD2SOIVBXYAJE6RRMTPKW6XAPT7CNLTQ, seg_pos={0 0}, upload=535486]: Uploading piece... Dec 31 02:31:42.266296698 (611h2m42.137379733s) [noise=true , address=145.79.227.24:4581 , piece_id=OPPGQUC2NXAGBLYXH5VM4RGY6ATR5DAUJAGRRKAQCZ3ENMNTMLBA, seg_pos={0 0}, upload=535486]: Uploading piece... Dec 31 02:31:42.270084160 (611h2m42.141167195s) [noise=true , address=145.79.227.21:4474 , piece_id=HBUEUQTYMITUOKLQEQHAOSCBHCVXCWGCUM4WDPC4ZMFKRQCW3GRA, seg_pos={0 0}, upload=535486]: Uploading piece... Dec 31 02:31:43.955549810 (611h2m43.826632865s) [upload=535486]: Next returned nil segment Dec 31 02:31:43.955567964 (611h2m43.826651009s) [upload=535486]: Waiting for error group managing segments... Dec 31 02:31:44.239121474 (611h2m44.110204519s) [noise=true , address=145.79.227.18:4007 , piece_id=HA2C5LN5ZB6TVJBQ6GVELU4JQG5KII7CFBTZAHDT75KJP5J7FOYA, seg_pos={0 0}, upload=535486]: Done uploading piece. err: <nil> Dec 31 02:31:44.247474961 (611h2m44.118557996s) [noise=true , address=145.79.227.24:4581 , piece_id=OPPGQUC2NXAGBLYXH5VM4RGY6ATR5DAUJAGRRKAQCZ3ENMNTMLBA, seg_pos={0 0}, upload=535486]: Done uploading piece. err: <nil> Dec 31 02:31:44.249933482 (611h2m44.121016527s) [noise=true , address=145.79.227.39:4014 , piece_id=R3DDMHHRZSHJUDSEXG3BUD2SOIVBXYAJE6RRMTPKW6XAPT7CNLTQ, seg_pos={0 0}, upload=535486]: Done uploading piece. err: <nil> Dec 31 02:31:44.250394820 (611h2m44.121477855s) [noise=true , address=145.79.227.52:4447 , piece_id=65XJCKQL2VIDCHMSHGUSCFNPU7JWFPKRF2QXWNAHQT7T5GELABFQ, seg_pos={0 0}, upload=535486]: Done uploading piece. err: <nil> Dec 31 02:31:44.260864926 (611h2m44.131947971s) [noise=true , address=145.79.227.3:4150 , piece_id=GKSGSQWFKTLK4B4ARKBXK2FR5IDBCOJMZZUOFEI2KASMXPJUHI4A, seg_pos={0 0}, upload=535486]: Done uploading piece. err: <nil> Dec 31 02:31:44.265136891 (611h2m44.136219936s) [noise=true , address=145.79.227.15:4876 , piece_id=WCIBKRCJCRS4JGFTWDQHVUZ2TLV3OYKTSMIBSBSU4WBFZ7WZNHOA, seg_pos={0 0}, upload=535486]: Done uploading piece. err: <nil> Dec 31 02:31:44.274014796 (611h2m44.145097831s) [noise=true , address=145.79.227.20:4438 , piece_id=MNT67N5BTYQB2JUNXV7TVSRL3DJMMTBXYQO4BSWEFALCWMK2ZP5Q, seg_pos={0 0}, upload=535486]: Done uploading piece. err: <nil> Dec 31 02:31:44.275430611 (611h2m44.146513646s) [noise=true , address=145.79.227.21:4474 , piece_id=HBUEUQTYMITUOKLQEQHAOSCBHCVXCWGCUM4WDPC4ZMFKRQCW3GRA, seg_pos={0 0}, upload=535486]: Done uploading piece. err: <nil> Dec 31 02:31:44.276265651 (611h2m44.147348686s) [noise=true , address=145.79.227.14:4837 , piece_id=HIU57ZYKPL6JYTQB27K2XG67GGYG7XM63OCJ4YLMMQXMLKYDCFJA, seg_pos={0 0}, upload=535486]: Done uploading piece. err: <nil> Dec 31 02:31:44.555968520 (611h2m44.427051555s) [noise=true , address=45.91.116.12:4225 , piece_id=YBK3PD3XVPSHJZTXOEGCXAGKJFRBLUSTFH6ZHYEFYK5YRIMLGISQ, seg_pos={0 0}, upload=535486]: Done uploading piece. err: <nil> Dec 31 02:31:44.563093667 (611h2m44.434176702s) [noise=true , address=77.90.31.10:4519 , piece_id=CZ5AVVMI74GHKCHLXL47OQED5DAR66LRFIVH25A3JL6K7NXNAOSA, seg_pos={0 0}, upload=535486]: Done uploading piece. err: <nil> Dec 31 02:31:44.625636847 (611h2m44.496719882s) [noise=true , address=45.91.116.9:4262 , piece_id=DCML4BWFWKJ2AEQVRGRYOQ7Q4YY3BF2PF4GFQSOREMAG67M4BJ5Q, seg_pos={0 0}, upload=535486]: Done uploading piece. err: <nil> Dec 31 02:31:44.717693032 (611h2m44.588776077s) [noise=true , address=45.91.116.6:4973 , piece_id=BVEV2ZTEUP3UUZ7H6WBZ4MHY2JESHTR6Z2HY3AAD5XCRWE7NLMMQ, seg_pos={0 0}, upload=535486]: Done uploading piece. err: <nil> Dec 31 02:31:44.717715566 (611h2m44.588798611s) [seg_pos={0 0}, upload=535486]: Segment reached optimal threshold of 13 pieces. Dec 31 02:31:44.718014545 (611h2m44.58909759s) [noise=true , address=45.91.116.11:4118 , piece_id=JAUSAGDUAQ4BBYWNA7OSW6XATK2JLMRUM6K6QNHD5EB4DPZNPQZA, seg_pos={0 0}, upload=535486]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF Dec 31 02:31:44.718024604 (611h2m44.589107639s) [noise=true , address=45.91.116.11:4118 , piece_id=JAUSAGDUAQ4BBYWNA7OSW6XATK2JLMRUM6K6QNHD5EB4DPZNPQZA, seg_pos={0 0}, upload=535486]: Long tail cancellation due to optimal threshold Dec 31 02:31:44.718138050 (611h2m44.589221125s) [noise=true , address=77.90.31.19:4844 , piece_id=DJPEVJBHVIZGFMBSUBTEWEIMU3JCSWPSKUXWPDQGXJKXLFGXQQGA, seg_pos={0 0}, upload=535486]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF Dec 31 02:31:44.718145925 (611h2m44.58922897s) [noise=true , address=77.90.31.19:4844 , piece_id=DJPEVJBHVIZGFMBSUBTEWEIMU3JCSWPSKUXWPDQGXJKXLFGXQQGA, seg_pos={0 0}, upload=535486]: Long tail cancellation due to optimal threshold Dec 31 02:31:44.718182945 (611h2m44.58926598s) [noise=true , address=45.91.116.5:4299 , piece_id=GZH33CCTCZPZX5ZPHEXDKBI2YSUP4GZ7YUQ6FIIHLCBB24SRSFGA, seg_pos={0 0}, upload=535486]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Dec 31 02:31:44.718191292 (611h2m44.589274327s) [noise=true , address=45.91.116.5:4299 , piece_id=GZH33CCTCZPZX5ZPHEXDKBI2YSUP4GZ7YUQ6FIIHLCBB24SRSFGA, seg_pos={0 0}, upload=535486]: Long tail cancellation due to optimal threshold Dec 31 02:31:44.718248981 (611h2m44.589332016s) [seg_pos={0 0}, upload=535486]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 02:31:44.718376454 (611h2m44.589459519s) [upload=535486]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 02:31:44.718437000 (611h2m44.589520045s) [upload=535486]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 02:31:45.017559155 (611h2m44.8886422s) [upload=535486]: Done uploading segments... Dec 31 02:31:45.017585174 (611h2m44.888668209s) [upload=535486]: Upload finished. err: <nil> Duration: 5.706810207s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [............X..X.......X.X..X....XX.XX........X.X......X..X.] duration: 6.24935019s trace: https://jaeger.dev.storj.io/trace/54231a38e027f2fe FAIL (2025-12-31 02:32:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: DCML4BWFWKJ2AEQVRGRYOQ7Q4YY3BF2PF4GFQSOREMAG67M4BJ5Q) context deadline exceeded; piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: YBK3PD3XVPSHJZTXOEGCXAGKJFRBLUSTFH6ZHYEFYK5YRIMLGISQ) context deadline exceeded ======LOGS====== Dec 31 02:31:42.009467393 (611h2m41.880550439s) [upload=535486]: Starting upload Dec 31 02:31:42.009568907 (611h2m41.880651942s) [upload=535486]: Done starting upload Dec 31 02:31:42.009677694 (611h2m41.880760729s) [upload=535486]: Uploading segments... Dec 31 02:31:42.009860923 (611h2m41.880943969s) [upload=535486]: Got next segment. Inline: false Dec 31 02:31:42.009921478 (611h2m41.881004513s) [upload=535486]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 02:31:42.009926968 (611h2m41.881010003s) [upload=535486]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 02:31:42.257755154 (611h2m42.128838189s) [seg_pos={0 0}, upload=535486]: Begin upload segment... Dec 31 02:31:42.257856827 (611h2m42.128939862s) [seg_pos={0 0}, upload=535486]: Done begin upload segment. Dec 31 02:31:42.257895601 (611h2m42.128978646s) [noise=true , address=45.91.116.6:4973 , piece_id=BVEV2ZTEUP3UUZ7H6WBZ4MHY2JESHTR6Z2HY3AAD5XCRWE7NLMMQ, seg_pos={0 0}, upload=535486]: Uploading piece... Dec 31 02:31:42.257927502 (611h2m42.129010537s) [noise=true , address=77.90.31.19:4844 , piece_id=DJPEVJBHVIZGFMBSUBTEWEIMU3JCSWPSKUXWPDQGXJKXLFGXQQGA, seg_pos={0 0}, upload=535486]: Uploading piece... Dec 31 02:31:42.258733707 (611h2m42.129816752s) [noise=true , address=145.79.227.52:4447 , piece_id=65XJCKQL2VIDCHMSHGUSCFNPU7JWFPKRF2QXWNAHQT7T5GELABFQ, seg_pos={0 0}, upload=535486]: Uploading piece... Dec 31 02:31:42.259561023 (611h2m42.130644058s) [noise=true , address=45.91.116.9:4262 , piece_id=DCML4BWFWKJ2AEQVRGRYOQ7Q4YY3BF2PF4GFQSOREMAG67M4BJ5Q, seg_pos={0 0}, upload=535486]: Uploading piece... Dec 31 02:31:42.259700759 (611h2m42.130783794s) [noise=true , address=45.91.116.11:4118 , piece_id=JAUSAGDUAQ4BBYWNA7OSW6XATK2JLMRUM6K6QNHD5EB4DPZNPQZA, seg_pos={0 0}, upload=535486]: Uploading piece... Dec 31 02:31:42.260044383 (611h2m42.131127428s) [noise=true , address=45.91.116.12:4225 , piece_id=YBK3PD3XVPSHJZTXOEGCXAGKJFRBLUSTFH6ZHYEFYK5YRIMLGISQ, seg_pos={0 0}, upload=535486]: Uploading piece... Dec 31 02:31:42.260541770 (611h2m42.131624815s) [noise=true , address=45.91.116.5:4299 , piece_id=GZH33CCTCZPZX5ZPHEXDKBI2YSUP4GZ7YUQ6FIIHLCBB24SRSFGA, seg_pos={0 0}, upload=535486]: Uploading piece... Dec 31 02:31:42.261116324 (611h2m42.132199369s) [noise=true , address=145.79.227.18:4007 , piece_id=HA2C5LN5ZB6TVJBQ6GVELU4JQG5KII7CFBTZAHDT75KJP5J7FOYA, seg_pos={0 0}, upload=535486]: Uploading piece... Dec 31 02:31:42.262541628 (611h2m42.133624674s) [noise=true , address=145.79.227.15:4876 , piece_id=WCIBKRCJCRS4JGFTWDQHVUZ2TLV3OYKTSMIBSBSU4WBFZ7WZNHOA, seg_pos={0 0}, upload=535486]: Uploading piece... Dec 31 02:31:42.262838263 (611h2m42.133921308s) [noise=true , address=145.79.227.3:4150 , piece_id=GKSGSQWFKTLK4B4ARKBXK2FR5IDBCOJMZZUOFEI2KASMXPJUHI4A, seg_pos={0 0}, upload=535486]: Uploading piece... Dec 31 02:31:42.263683141 (611h2m42.134766186s) [noise=true , address=77.90.31.10:4519 , piece_id=CZ5AVVMI74GHKCHLXL47OQED5DAR66LRFIVH25A3JL6K7NXNAOSA, seg_pos={0 0}, upload=535486]: Uploading piece... Dec 31 02:31:42.263989505 (611h2m42.13507254s) [noise=true , address=145.79.227.14:4837 , piece_id=HIU57ZYKPL6JYTQB27K2XG67GGYG7XM63OCJ4YLMMQXMLKYDCFJA, seg_pos={0 0}, upload=535486]: Uploading piece... Dec 31 02:31:42.264792113 (611h2m42.135875158s) [noise=true , address=145.79.227.20:4438 , piece_id=MNT67N5BTYQB2JUNXV7TVSRL3DJMMTBXYQO4BSWEFALCWMK2ZP5Q, seg_pos={0 0}, upload=535486]: Uploading piece... Dec 31 02:31:42.265209829 (611h2m42.136292864s) [noise=true , address=145.79.227.39:4014 , piece_id=R3DDMHHRZSHJUDSEXG3BUD2SOIVBXYAJE6RRMTPKW6XAPT7CNLTQ, seg_pos={0 0}, upload=535486]: Uploading piece... Dec 31 02:31:42.266296698 (611h2m42.137379733s) [noise=true , address=145.79.227.24:4581 , piece_id=OPPGQUC2NXAGBLYXH5VM4RGY6ATR5DAUJAGRRKAQCZ3ENMNTMLBA, seg_pos={0 0}, upload=535486]: Uploading piece... Dec 31 02:31:42.270084160 (611h2m42.141167195s) [noise=true , address=145.79.227.21:4474 , piece_id=HBUEUQTYMITUOKLQEQHAOSCBHCVXCWGCUM4WDPC4ZMFKRQCW3GRA, seg_pos={0 0}, upload=535486]: Uploading piece... Dec 31 02:31:43.955549810 (611h2m43.826632865s) [upload=535486]: Next returned nil segment Dec 31 02:31:43.955567964 (611h2m43.826651009s) [upload=535486]: Waiting for error group managing segments... Dec 31 02:31:44.239121474 (611h2m44.110204519s) [noise=true , address=145.79.227.18:4007 , piece_id=HA2C5LN5ZB6TVJBQ6GVELU4JQG5KII7CFBTZAHDT75KJP5J7FOYA, seg_pos={0 0}, upload=535486]: Done uploading piece. err: <nil> Dec 31 02:31:44.247474961 (611h2m44.118557996s) [noise=true , address=145.79.227.24:4581 , piece_id=OPPGQUC2NXAGBLYXH5VM4RGY6ATR5DAUJAGRRKAQCZ3ENMNTMLBA, seg_pos={0 0}, upload=535486]: Done uploading piece. err: <nil> Dec 31 02:31:44.249933482 (611h2m44.121016527s) [noise=true , address=145.79.227.39:4014 , piece_id=R3DDMHHRZSHJUDSEXG3BUD2SOIVBXYAJE6RRMTPKW6XAPT7CNLTQ, seg_pos={0 0}, upload=535486]: Done uploading piece. err: <nil> Dec 31 02:31:44.250394820 (611h2m44.121477855s) [noise=true , address=145.79.227.52:4447 , piece_id=65XJCKQL2VIDCHMSHGUSCFNPU7JWFPKRF2QXWNAHQT7T5GELABFQ, seg_pos={0 0}, upload=535486]: Done uploading piece. err: <nil> Dec 31 02:31:44.260864926 (611h2m44.131947971s) [noise=true , address=145.79.227.3:4150 , piece_id=GKSGSQWFKTLK4B4ARKBXK2FR5IDBCOJMZZUOFEI2KASMXPJUHI4A, seg_pos={0 0}, upload=535486]: Done uploading piece. err: <nil> Dec 31 02:31:44.265136891 (611h2m44.136219936s) [noise=true , address=145.79.227.15:4876 , piece_id=WCIBKRCJCRS4JGFTWDQHVUZ2TLV3OYKTSMIBSBSU4WBFZ7WZNHOA, seg_pos={0 0}, upload=535486]: Done uploading piece. err: <nil> Dec 31 02:31:44.274014796 (611h2m44.145097831s) [noise=true , address=145.79.227.20:4438 , piece_id=MNT67N5BTYQB2JUNXV7TVSRL3DJMMTBXYQO4BSWEFALCWMK2ZP5Q, seg_pos={0 0}, upload=535486]: Done uploading piece. err: <nil> Dec 31 02:31:44.275430611 (611h2m44.146513646s) [noise=true , address=145.79.227.21:4474 , piece_id=HBUEUQTYMITUOKLQEQHAOSCBHCVXCWGCUM4WDPC4ZMFKRQCW3GRA, seg_pos={0 0}, upload=535486]: Done uploading piece. err: <nil> Dec 31 02:31:44.276265651 (611h2m44.147348686s) [noise=true , address=145.79.227.14:4837 , piece_id=HIU57ZYKPL6JYTQB27K2XG67GGYG7XM63OCJ4YLMMQXMLKYDCFJA, seg_pos={0 0}, upload=535486]: Done uploading piece. err: <nil> Dec 31 02:31:44.555968520 (611h2m44.427051555s) [noise=true , address=45.91.116.12:4225 , piece_id=YBK3PD3XVPSHJZTXOEGCXAGKJFRBLUSTFH6ZHYEFYK5YRIMLGISQ, seg_pos={0 0}, upload=535486]: Done uploading piece. err: <nil> Dec 31 02:31:44.563093667 (611h2m44.434176702s) [noise=true , address=77.90.31.10:4519 , piece_id=CZ5AVVMI74GHKCHLXL47OQED5DAR66LRFIVH25A3JL6K7NXNAOSA, seg_pos={0 0}, upload=535486]: Done uploading piece. err: <nil> Dec 31 02:31:44.625636847 (611h2m44.496719882s) [noise=true , address=45.91.116.9:4262 , piece_id=DCML4BWFWKJ2AEQVRGRYOQ7Q4YY3BF2PF4GFQSOREMAG67M4BJ5Q, seg_pos={0 0}, upload=535486]: Done uploading piece. err: <nil> Dec 31 02:31:44.717693032 (611h2m44.588776077s) [noise=true , address=45.91.116.6:4973 , piece_id=BVEV2ZTEUP3UUZ7H6WBZ4MHY2JESHTR6Z2HY3AAD5XCRWE7NLMMQ, seg_pos={0 0}, upload=535486]: Done uploading piece. err: <nil> Dec 31 02:31:44.717715566 (611h2m44.588798611s) [seg_pos={0 0}, upload=535486]: Segment reached optimal threshold of 13 pieces. Dec 31 02:31:44.718014545 (611h2m44.58909759s) [noise=true , address=45.91.116.11:4118 , piece_id=JAUSAGDUAQ4BBYWNA7OSW6XATK2JLMRUM6K6QNHD5EB4DPZNPQZA, seg_pos={0 0}, upload=535486]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF Dec 31 02:31:44.718024604 (611h2m44.589107639s) [noise=true , address=45.91.116.11:4118 , piece_id=JAUSAGDUAQ4BBYWNA7OSW6XATK2JLMRUM6K6QNHD5EB4DPZNPQZA, seg_pos={0 0}, upload=535486]: Long tail cancellation due to optimal threshold Dec 31 02:31:44.718138050 (611h2m44.589221125s) [noise=true , address=77.90.31.19:4844 , piece_id=DJPEVJBHVIZGFMBSUBTEWEIMU3JCSWPSKUXWPDQGXJKXLFGXQQGA, seg_pos={0 0}, upload=535486]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF Dec 31 02:31:44.718145925 (611h2m44.58922897s) [noise=true , address=77.90.31.19:4844 , piece_id=DJPEVJBHVIZGFMBSUBTEWEIMU3JCSWPSKUXWPDQGXJKXLFGXQQGA, seg_pos={0 0}, upload=535486]: Long tail cancellation due to optimal threshold Dec 31 02:31:44.718182945 (611h2m44.58926598s) [noise=true , address=45.91.116.5:4299 , piece_id=GZH33CCTCZPZX5ZPHEXDKBI2YSUP4GZ7YUQ6FIIHLCBB24SRSFGA, seg_pos={0 0}, upload=535486]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Dec 31 02:31:44.718191292 (611h2m44.589274327s) [noise=true , address=45.91.116.5:4299 , piece_id=GZH33CCTCZPZX5ZPHEXDKBI2YSUP4GZ7YUQ6FIIHLCBB24SRSFGA, seg_pos={0 0}, upload=535486]: Long tail cancellation due to optimal threshold Dec 31 02:31:44.718248981 (611h2m44.589332016s) [seg_pos={0 0}, upload=535486]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 02:31:44.718376454 (611h2m44.589459519s) [upload=535486]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 02:31:44.718437000 (611h2m44.589520045s) [upload=535486]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 02:31:45.017559155 (611h2m44.8886422s) [upload=535486]: Done uploading segments... Dec 31 02:31:45.017585174 (611h2m44.888668209s) [upload=535486]: Upload finished. err: <nil> Duration: 6.249383813s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)