Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 33.168272496s trace: https://jaeger.dev.storj.io/trace/365693fea00df28 FAIL (2025-12-31 02:27:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [noise=true , address=145.79.227.49:4343 , piece_id=6L3HPTS7D2NRA4QEOH4RFM3D3B53CTCONRI6G6YVU4OOTUWTG7XA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.942060847 (610h25m24.813143892s) [noise=true , address=145.79.227.34:4937 , piece_id=5W7LBM3YJJTVUFYTELR57DVXB663OHO5N3Z2SPQPZ4G43IK47O3Q, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.942684104 (610h25m24.813767139s) [noise=true , address=64.49.13.139:4478 , piece_id=QN6RT2KWEA6APQVA66ZMKG4LH537UGYNOXOCVVA2YC2MKDOLT3YA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.943064007 (610h25m24.814147042s) [noise=true , address=145.79.227.40:4084 , piece_id=RUKTKDII6BRUWSYVS65CWUBT3QKG3ASUK3KA3O5VGE53XSG6IMOQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.943459560 (610h25m24.814542605s) [noise=true , address=145.79.227.14:4847 , piece_id=JX5NXTMVM3NSWIEVR4RX46ABSNPKTOFDLFBPKEXFQ3G3ANKV5OLA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.943865303 (610h25m24.814948348s) [noise=true , address=45.91.116.6:4981 , piece_id=CT474ZA2LVAZEBJHBDD4GB6NTHVHZFJLNE5OZV3DFS2NZKVW3EQQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.944259694 (610h25m24.815342749s) [noise=true , address=145.79.227.51:4443 , piece_id=73UUMRDY7BSZNCGBXCZYYRBVE5KLPMNPRNXOZEGWKQGLHJS32J2Q, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.946575182 (610h25m24.817658227s) [noise=true , address=77.90.31.10:4523 , piece_id=P6LIONXJI6AFZ2JTKCQ2W6RZCOZCK6MZ53EXAJKXYQXSX2PX4QEA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.948132909 (610h25m24.819215944s) [noise=true , address=145.79.227.20:4452 , piece_id=ZSDN2JZ3FQUVJYN5OURDXNKEDLDNULJOHKKRZVIPXZIWIYAHCRXQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.949605774 (610h25m24.820688819s) [noise=true , address=77.90.31.14:4680 , piece_id=S2E4WBA7MJVAZLT5JG3IMHK53RQC77CCHV2PDF6L2R2JEA2U3K4Q, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.951459002 (610h25m24.822542048s) [noise=true , address=193.149.164.143:4704 , piece_id=BHXHFBKTLTDMGXWY5QRTP2THZQE5OXFZ2LCMTO2KRYRWAHXWW3NQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.952986742 (610h25m24.824069777s) [noise=true , address=77.90.31.19:4848 , piece_id=VZYG32S6ODIPDKQYZRXECBHXOERZDDWD2OF35Z46W4XMUZ3OJLAA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.956916654 (610h25m24.827999689s) [noise=true , address=145.79.227.29:4971 , piece_id=MEPXHHOQ6SUZC3GIPCQEMOKKF5OJTKZEI3UYZBAYXAQZGSVEKSBA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.960665062 (610h25m24.831748097s) [noise=true , address=64.49.13.170:4720 , piece_id=HIATRSAPE4YLSODFZDPP43ZUMLDVM7GJEVMWE4ZGUQTP25LTHGTA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.962366001 (610h25m24.833449036s) [noise=true , address=145.79.227.2:4117 , piece_id=4SGFGB3IJ7LTXDMMWM2PGGOV64MC46IZ7OZQWH55J4NBQY7CHMZQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.963708216 (610h25m24.834791261s) [noise=true , address=145.79.227.35:4505 , piece_id=V7RS7A4QLZTH4G32ZUZNSF2Z5APVWZGJO77NEGP3SBWQT3WCMVLQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:26.352236138 (610h25m26.223319173s) [upload=534952]: Next returned nil segment Dec 31 01:54:26.352253821 (610h25m26.223336857s) [upload=534952]: Waiting for error group managing segments... Dec 31 01:54:26.812758374 (610h25m26.683841409s) [noise=true , address=145.79.227.51:4443 , piece_id=73UUMRDY7BSZNCGBXCZYYRBVE5KLPMNPRNXOZEGWKQGLHJS32J2Q, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.824371056 (610h25m26.695454101s) [noise=true , address=145.79.227.14:4847 , piece_id=JX5NXTMVM3NSWIEVR4RX46ABSNPKTOFDLFBPKEXFQ3G3ANKV5OLA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.838414046 (610h25m26.709497081s) [noise=true , address=64.49.13.138:4490 , piece_id=2XTUZE6PXRE3OARNJVT36FXXWUMYIBVNE6M2VVFAVBGKEE2H5YZQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.875499841 (610h25m26.746582876s) [noise=true , address=64.49.13.139:4478 , piece_id=QN6RT2KWEA6APQVA66ZMKG4LH537UGYNOXOCVVA2YC2MKDOLT3YA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.891607221 (610h25m26.762690256s) [noise=true , address=145.79.227.20:4452 , piece_id=ZSDN2JZ3FQUVJYN5OURDXNKEDLDNULJOHKKRZVIPXZIWIYAHCRXQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.988561223 (610h25m26.859644268s) [noise=true , address=64.49.13.170:4720 , piece_id=HIATRSAPE4YLSODFZDPP43ZUMLDVM7GJEVMWE4ZGUQTP25LTHGTA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.039035911 (610h25m26.910118946s) [noise=true , address=77.90.31.14:4680 , piece_id=S2E4WBA7MJVAZLT5JG3IMHK53RQC77CCHV2PDF6L2R2JEA2U3K4Q, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.098023090 (610h25m26.969106126s) [noise=true , address=45.91.116.6:4981 , piece_id=CT474ZA2LVAZEBJHBDD4GB6NTHVHZFJLNE5OZV3DFS2NZKVW3EQQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.653790754 (610h25m27.524873789s) [noise=true , address=77.90.31.21:4886 , piece_id=K34WPIBGGGEPHIKJ253XGLDCIQ3WLFN4AZ6EYNWVHHT2EIHS6YRQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.684192972 (610h25m27.555276017s) [noise=true , address=193.149.164.143:4704 , piece_id=BHXHFBKTLTDMGXWY5QRTP2THZQE5OXFZ2LCMTO2KRYRWAHXWW3NQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.717238425 (610h25m27.58832146s) [noise=true , address=145.79.227.40:4084 , piece_id=RUKTKDII6BRUWSYVS65CWUBT3QKG3ASUK3KA3O5VGE53XSG6IMOQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.718349742 (610h25m27.589432777s) [noise=true , address=77.90.31.10:4523 , piece_id=P6LIONXJI6AFZ2JTKCQ2W6RZCOZCK6MZ53EXAJKXYQXSX2PX4QEA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.803639562 (610h25m27.674722597s) [noise=true , address=145.79.227.29:4971 , piece_id=MEPXHHOQ6SUZC3GIPCQEMOKKF5OJTKZEI3UYZBAYXAQZGSVEKSBA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.812791279 (610h25m27.683874314s) [noise=true , address=145.79.227.49:4343 , piece_id=6L3HPTS7D2NRA4QEOH4RFM3D3B53CTCONRI6G6YVU4OOTUWTG7XA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.812812348 (610h25m27.683895393s) [seg_pos={0 0}, upload=534952]: Segment reached optimal threshold of 14 pieces. Dec 31 01:54:27.813018000 (610h25m27.684101035s) [noise=true , address=145.79.227.35:4505 , piece_id=V7RS7A4QLZTH4G32ZUZNSF2Z5APVWZGJO77NEGP3SBWQT3WCMVLQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i6itGoBhLsLiqC8et8Y5L62fLjcWxy18u39NAr4R3nXNMWcsf): protocol: expected piece hash; context canceled; EOF Dec 31 01:54:27.813024772 (610h25m27.684107808s) [noise=true , address=145.79.227.35:4505 , piece_id=V7RS7A4QLZTH4G32ZUZNSF2Z5APVWZGJO77NEGP3SBWQT3WCMVLQ, seg_pos={0 0}, upload=534952]: Long tail cancellation due to optimal threshold Dec 31 01:54:27.813175931 (610h25m27.684258976s) [noise=true , address=145.79.227.34:4937 , piece_id=5W7LBM3YJJTVUFYTELR57DVXB663OHO5N3Z2SPQPZ4G43IK47O3Q, seg_pos={0 0}, upload=534952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1LKrFThdW3jJpG6dRZLUaxLXTa5Wy27yQ8PGUEqjvBWfFoRB94): protocol: expected piece hash; context canceled; EOF Dec 31 01:54:27.813181702 (610h25m27.684264737s) [noise=true , address=145.79.227.34:4937 , piece_id=5W7LBM3YJJTVUFYTELR57DVXB663OHO5N3Z2SPQPZ4G43IK47O3Q, seg_pos={0 0}, upload=534952]: Long tail cancellation due to optimal threshold Dec 31 01:54:27.813407862 (610h25m27.684490907s) [noise=true , address=145.79.227.2:4117 , piece_id=4SGFGB3IJ7LTXDMMWM2PGGOV64MC46IZ7OZQWH55J4NBQY7CHMZQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122Htg72f5eUUutf1p3fYQpy1Wkf6AgdaQV83HHYye2n8zm3the): protocol: expected piece hash; context canceled; EOF Dec 31 01:54:27.813414264 (610h25m27.684497299s) [noise=true , address=145.79.227.2:4117 , piece_id=4SGFGB3IJ7LTXDMMWM2PGGOV64MC46IZ7OZQWH55J4NBQY7CHMZQ, seg_pos={0 0}, upload=534952]: Long tail cancellation due to optimal threshold Dec 31 01:54:27.813424614 (610h25m27.684507649s) [noise=true , address=77.90.31.19:4848 , piece_id=VZYG32S6ODIPDKQYZRXECBHXOERZDDWD2OF35Z46W4XMUZ3OJLAA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD): protocol: write tcp 188.166.225.170:42842->77.90.31.19:4848: use of closed network connection; write tcp 188.166.225.170:42842->77.90.31.19:4848: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42842->77.90.31.19:4848: use of closed network connection Dec 31 01:54:27.813432750 (610h25m27.684515785s) [noise=true , address=77.90.31.19:4848 , piece_id=VZYG32S6ODIPDKQYZRXECBHXOERZDDWD2OF35Z46W4XMUZ3OJLAA, seg_pos={0 0}, upload=534952]: Long tail cancellation due to optimal threshold Dec 31 01:54:27.813474088 (610h25m27.684557123s) [seg_pos={0 0}, upload=534952]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 01:54:27.813535325 (610h25m27.68461836s) [upload=534952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 01:54:27.813539613 (610h25m27.684622648s) [upload=534952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 01:54:28.099921034 (610h25m27.971004069s) [upload=534952]: Done uploading segments... Dec 31 01:54:28.099940451 (610h25m27.971023486s) [upload=534952]: Upload finished. err: <nil> Duration: 33.168291862s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...X..................................X.....................] duration: 3.416583718s trace: https://jaeger.dev.storj.io/trace/740aa405bd68150f FAIL (2025-12-31 02:27:12 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.416610821s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.....X............X..X.......X.X..X....XX.XX........X.X.....] duration: 6.521411646s trace: https://jaeger.dev.storj.io/trace/3b4011ac5a42e158 FAIL (2025-12-31 02:27:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: LXE52NV3QIBRTZK27IRWWLAZ3OU2IUHB25H7V6L7XETJSR7ZN67A) context deadline exceeded; piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: OWH5GGYCNXNDSZVDKNMKLXSMO25MW4I4TAJPZJLFKMBQKUSSPG3Q) context deadline exceeded ======LOGS====== 31 02:22:12.475370476 (610h53m12.346453511s) [upload=535351]: Starting upload Dec 31 02:22:12.475429659 (610h53m12.346512694s) [upload=535351]: Done starting upload Dec 31 02:22:12.475607157 (610h53m12.346690202s) [upload=535351]: Uploading segments... Dec 31 02:22:12.475783423 (610h53m12.346866468s) [upload=535351]: Got next segment. Inline: false Dec 31 02:22:12.475838657 (610h53m12.346921702s) [upload=535351]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 02:22:12.475845551 (610h53m12.346928596s) [upload=535351]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 02:22:12.724117301 (610h53m12.595200346s) [seg_pos={0 0}, upload=535351]: Begin upload segment... Dec 31 02:22:12.724171443 (610h53m12.595254488s) [seg_pos={0 0}, upload=535351]: Done begin upload segment. Dec 31 02:22:12.724222331 (610h53m12.595305376s) [noise=true , address=145.79.227.52:4447 , piece_id=AOEU3DUQFDXJAIRV2U7V72IXUCTVVTDQ3O4SHYDRUZ6BNCTIHGIQ, seg_pos={0 0}, upload=535351]: Uploading piece... Dec 31 02:22:12.724287495 (610h53m12.59537054s) [noise=true , address=145.79.227.37:4565 , piece_id=XSVPD47ZDWNMFOJJXGBGUWNN3I4H5MXJHIMGSRSWHRN5OHK4EIGQ, seg_pos={0 0}, upload=535351]: Uploading piece... Dec 31 02:22:12.724817614 (610h53m12.595900659s) [noise=true , address=45.91.116.4:4939 , piece_id=RIQWQXXNI7SCIR7TJXKETDANJ7CLQNRSIV5RS67F4RJYL66XFTSA, seg_pos={0 0}, upload=535351]: Uploading piece... Dec 31 02:22:12.725168582 (610h53m12.596251627s) [noise=true , address=145.79.227.23:4547 , piece_id=JF4GEBAKZHAAFTR6WG7AGQ33LRL22ZTMK5EDPUZQKUWLAONLFGHA, seg_pos={0 0}, upload=535351]: Uploading piece... Dec 31 02:22:12.725489473 (610h53m12.596572508s) [noise=true , address=145.79.227.24:4581 , piece_id=LS6IDZDA6IQBOXLUQBBFUAWKYA2USQV7NNU52662OZXQLINGLGMQ, seg_pos={0 0}, upload=535351]: Uploading piece... Dec 31 02:22:12.725848035 (610h53m12.59693107s) [noise=true , address=45.91.116.8:4045 , piece_id=LXE52NV3QIBRTZK27IRWWLAZ3OU2IUHB25H7V6L7XETJSR7ZN67A, seg_pos={0 0}, upload=535351]: Uploading piece... Dec 31 02:22:12.726610608 (610h53m12.597693643s) [noise=true , address=45.91.116.7:4011 , piece_id=OWH5GGYCNXNDSZVDKNMKLXSMO25MW4I4TAJPZJLFKMBQKUSSPG3Q, seg_pos={0 0}, upload=535351]: Uploading piece... Dec 31 02:22:12.727067127 (610h53m12.598150172s) [noise=true , address=145.79.227.11:4732 , piece_id=WJTPQC5HV7CLFYOGUNNQED2LS7347DBJY7AMUMW4MJWRDV4AODUA, seg_pos={0 0}, upload=535351]: Uploading piece... Dec 31 02:22:12.727529488 (610h53m12.598612523s) [noise=true , address=145.79.227.13:4803 , piece_id=F546YDIU4YMWOPRTFXKH7YYFFFFRXDKLL5N434GL6TZ6GITFE2NA, seg_pos={0 0}, upload=535351]: Uploading piece... Dec 31 02:22:12.728202930 (610h53m12.599285975s) [noise=true , address=145.79.227.31:4798 , piece_id=XHAR65JKVRG3ZGFKFRBTBOF6ZY5AKHTUPDI5DRXAM6ZYY3CJWEJA, seg_pos={0 0}, upload=535351]: Uploading piece... Dec 31 02:22:12.729252208 (610h53m12.600335253s) [noise=true , address=145.79.227.21:4474 , piece_id=VQKDL2DP37I3XY7BG7NDXAF7FDSMUVDRGQ3FU23EVTFFCHAT4L6A, seg_pos={0 0}, upload=535351]: Uploading piece... Dec 31 02:22:12.729402474 (610h53m12.600485509s) [noise=true , address=77.90.31.11:4555 , piece_id=VMAFYHAEN5YKSBAN33PS7FUSNBDQ3DBO5JG7KRBNJWBNW7M5LCYA, seg_pos={0 0}, upload=535351]: Uploading piece... Dec 31 02:22:12.730436854 (610h53m12.601519889s) [noise=true , address=145.79.227.6:4258 , piece_id=ONT4R23H4HUNEPFXYI7XCUL55STXVZH3P5QGQKCDI3B7PEZCKFPQ, seg_pos={0 0}, upload=535351]: Uploading piece... Dec 31 02:22:12.730941896 (610h53m12.602024931s) [noise=true , address=145.79.227.3:4151 , piece_id=466E5IHSVX2ZQ3UPTTLSHH4EUPZJ4H3PUY2XTZHYOYIFYBMFGGHA, seg_pos={0 0}, upload=535351]: Uploading piece... Dec 31 02:22:12.731601811 (610h53m12.602684846s) [noise=true , address=77.90.31.14:4661 , piece_id=5MI4JBBRHTC5MR57TUBEHNZVX6OD7AODOPSBCYH4TSYUPUYDRGNQ, seg_pos={0 0}, upload=535351]: Uploading piece... Dec 31 02:22:12.732236310 (610h53m12.603319355s) [noise=true , address=145.79.227.9:4367 , piece_id=5FD3BXSD5MB3N4C76AQ7P6CM4YZ2WUCJAV4TMFCOXTCTCHSMNIRQ, seg_pos={0 0}, upload=535351]: Uploading piece... Dec 31 02:22:14.430774662 (610h53m14.301857707s) [upload=535351]: Next returned nil segment Dec 31 02:22:14.430792586 (610h53m14.301875631s) [upload=535351]: Waiting for error group managing segments... Dec 31 02:22:14.711554761 (610h53m14.582637806s) [noise=true , address=145.79.227.21:4474 , piece_id=VQKDL2DP37I3XY7BG7NDXAF7FDSMUVDRGQ3FU23EVTFFCHAT4L6A, seg_pos={0 0}, upload=535351]: Done uploading piece. err: <nil> Dec 31 02:22:14.714690121 (610h53m14.585773166s) [noise=true , address=145.79.227.9:4367 , piece_id=5FD3BXSD5MB3N4C76AQ7P6CM4YZ2WUCJAV4TMFCOXTCTCHSMNIRQ, seg_pos={0 0}, upload=535351]: Done uploading piece. err: <nil> Dec 31 02:22:14.718092480 (610h53m14.589175525s) [noise=true , address=145.79.227.6:4258 , piece_id=ONT4R23H4HUNEPFXYI7XCUL55STXVZH3P5QGQKCDI3B7PEZCKFPQ, seg_pos={0 0}, upload=535351]: Done uploading piece. err: <nil> Dec 31 02:22:14.719657861 (610h53m14.590740906s) [noise=true , address=145.79.227.3:4151 , piece_id=466E5IHSVX2ZQ3UPTTLSHH4EUPZJ4H3PUY2XTZHYOYIFYBMFGGHA, seg_pos={0 0}, upload=535351]: Done uploading piece. err: <nil> Dec 31 02:22:14.728292722 (610h53m14.599375757s) [noise=true , address=145.79.227.13:4803 , piece_id=F546YDIU4YMWOPRTFXKH7YYFFFFRXDKLL5N434GL6TZ6GITFE2NA, seg_pos={0 0}, upload=535351]: Done uploading piece. err: <nil> Dec 31 02:22:14.728532268 (610h53m14.599615314s) [noise=true , address=145.79.227.37:4565 , piece_id=XSVPD47ZDWNMFOJJXGBGUWNN3I4H5MXJHIMGSRSWHRN5OHK4EIGQ, seg_pos={0 0}, upload=535351]: Done uploading piece. err: <nil> Dec 31 02:22:14.734847814 (610h53m14.605930859s) [noise=true , address=145.79.227.52:4447 , piece_id=AOEU3DUQFDXJAIRV2U7V72IXUCTVVTDQ3O4SHYDRUZ6BNCTIHGIQ, seg_pos={0 0}, upload=535351]: Done uploading piece. err: <nil> Dec 31 02:22:14.741499390 (610h53m14.612582425s) [noise=true , address=145.79.227.11:4732 , piece_id=WJTPQC5HV7CLFYOGUNNQED2LS7347DBJY7AMUMW4MJWRDV4AODUA, seg_pos={0 0}, upload=535351]: Done uploading piece. err: <nil> Dec 31 02:22:14.744680406 (610h53m14.615763452s) [noise=true , address=145.79.227.23:4547 , piece_id=JF4GEBAKZHAAFTR6WG7AGQ33LRL22ZTMK5EDPUZQKUWLAONLFGHA, seg_pos={0 0}, upload=535351]: Done uploading piece. err: <nil> Dec 31 02:22:14.799469321 (610h53m14.670552356s) [noise=true , address=145.79.227.31:4798 , piece_id=XHAR65JKVRG3ZGFKFRBTBOF6ZY5AKHTUPDI5DRXAM6ZYY3CJWEJA, seg_pos={0 0}, upload=535351]: Done uploading piece. err: <nil> Dec 31 02:22:14.799917525 (610h53m14.67100057s) [noise=true , address=145.79.227.24:4581 , piece_id=LS6IDZDA6IQBOXLUQBBFUAWKYA2USQV7NNU52662OZXQLINGLGMQ, seg_pos={0 0}, upload=535351]: Done uploading piece. err: <nil> Dec 31 02:22:15.084260048 (610h53m14.955343103s) [noise=true , address=45.91.116.8:4045 , piece_id=LXE52NV3QIBRTZK27IRWWLAZ3OU2IUHB25H7V6L7XETJSR7ZN67A, seg_pos={0 0}, upload=535351]: Done uploading piece. err: <nil> Dec 31 02:22:15.144344775 (610h53m15.01542781s) [noise=true , address=45.91.116.7:4011 , piece_id=OWH5GGYCNXNDSZVDKNMKLXSMO25MW4I4TAJPZJLFKMBQKUSSPG3Q, seg_pos={0 0}, upload=535351]: Done uploading piece. err: <nil> Dec 31 02:22:15.144367880 (610h53m15.015450915s) [seg_pos={0 0}, upload=535351]: Segment reached optimal threshold of 13 pieces. Dec 31 02:22:15.144589732 (610h53m15.015672767s) [noise=true , address=45.91.116.4:4939 , piece_id=RIQWQXXNI7SCIR7TJXKETDANJ7CLQNRSIV5RS67F4RJYL66XFTSA, seg_pos={0 0}, upload=535351]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 31 02:22:15.144597046 (610h53m15.015680082s) [noise=true , address=45.91.116.4:4939 , piece_id=RIQWQXXNI7SCIR7TJXKETDANJ7CLQNRSIV5RS67F4RJYL66XFTSA, seg_pos={0 0}, upload=535351]: Long tail cancellation due to optimal threshold Dec 31 02:22:15.144618166 (610h53m15.015701201s) [noise=true , address=77.90.31.14:4661 , piece_id=5MI4JBBRHTC5MR57TUBEHNZVX6OD7AODOPSBCYH4TSYUPUYDRGNQ, seg_pos={0 0}, upload=535351]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Dec 31 02:22:15.144626191 (610h53m15.015709226s) [noise=true , address=77.90.31.14:4661 , piece_id=5MI4JBBRHTC5MR57TUBEHNZVX6OD7AODOPSBCYH4TSYUPUYDRGNQ, seg_pos={0 0}, upload=535351]: Long tail cancellation due to optimal threshold Dec 31 02:22:15.144800363 (610h53m15.015883398s) [noise=true , address=77.90.31.11:4555 , piece_id=VMAFYHAEN5YKSBAN33PS7FUSNBDQ3DBO5JG7KRBNJWBNW7M5LCYA, seg_pos={0 0}, upload=535351]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: write tcp 188.166.225.170:50294->77.90.31.11:4555: use of closed network connection; write tcp 188.166.225.170:50294->77.90.31.11:4555: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50294->77.90.31.11:4555: use of closed network connection Dec 31 02:22:15.144807157 (610h53m15.015890192s) [noise=true , address=77.90.31.11:4555 , piece_id=VMAFYHAEN5YKSBAN33PS7FUSNBDQ3DBO5JG7KRBNJWBNW7M5LCYA, seg_pos={0 0}, upload=535351]: Long tail cancellation due to optimal threshold Dec 31 02:22:15.144821844 (610h53m15.015904889s) [seg_pos={0 0}, upload=535351]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 02:22:15.144898269 (610h53m15.015981304s) [upload=535351]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 02:22:15.144903008 (610h53m15.015986043s) [upload=535351]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 02:22:15.439148899 (610h53m15.310231934s) [upload=535351]: Done uploading segments... Dec 31 02:22:15.439170178 (610h53m15.310253214s) [upload=535351]: Upload finished. err: <nil> Duration: 6.521443726s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [......X............X..X.......X.X..X....XX.XX........X.X....] duration: 24.40735944s trace: https://jaeger.dev.storj.io/trace/3b4011ac5a42e158 FAIL (2025-12-31 02:26:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: LXE52NV3QIBRTZK27IRWWLAZ3OU2IUHB25H7V6L7XETJSR7ZN67A) context deadline exceeded; piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: OWH5GGYCNXNDSZVDKNMKLXSMO25MW4I4TAJPZJLFKMBQKUSSPG3Q) context deadline exceeded ======LOGS====== 31 02:22:12.475370476 (610h53m12.346453511s) [upload=535351]: Starting upload Dec 31 02:22:12.475429659 (610h53m12.346512694s) [upload=535351]: Done starting upload Dec 31 02:22:12.475607157 (610h53m12.346690202s) [upload=535351]: Uploading segments... Dec 31 02:22:12.475783423 (610h53m12.346866468s) [upload=535351]: Got next segment. Inline: false Dec 31 02:22:12.475838657 (610h53m12.346921702s) [upload=535351]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 02:22:12.475845551 (610h53m12.346928596s) [upload=535351]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 02:22:12.724117301 (610h53m12.595200346s) [seg_pos={0 0}, upload=535351]: Begin upload segment... Dec 31 02:22:12.724171443 (610h53m12.595254488s) [seg_pos={0 0}, upload=535351]: Done begin upload segment. Dec 31 02:22:12.724222331 (610h53m12.595305376s) [noise=true , address=145.79.227.52:4447 , piece_id=AOEU3DUQFDXJAIRV2U7V72IXUCTVVTDQ3O4SHYDRUZ6BNCTIHGIQ, seg_pos={0 0}, upload=535351]: Uploading piece... Dec 31 02:22:12.724287495 (610h53m12.59537054s) [noise=true , address=145.79.227.37:4565 , piece_id=XSVPD47ZDWNMFOJJXGBGUWNN3I4H5MXJHIMGSRSWHRN5OHK4EIGQ, seg_pos={0 0}, upload=535351]: Uploading piece... Dec 31 02:22:12.724817614 (610h53m12.595900659s) [noise=true , address=45.91.116.4:4939 , piece_id=RIQWQXXNI7SCIR7TJXKETDANJ7CLQNRSIV5RS67F4RJYL66XFTSA, seg_pos={0 0}, upload=535351]: Uploading piece... Dec 31 02:22:12.725168582 (610h53m12.596251627s) [noise=true , address=145.79.227.23:4547 , piece_id=JF4GEBAKZHAAFTR6WG7AGQ33LRL22ZTMK5EDPUZQKUWLAONLFGHA, seg_pos={0 0}, upload=535351]: Uploading piece... Dec 31 02:22:12.725489473 (610h53m12.596572508s) [noise=true , address=145.79.227.24:4581 , piece_id=LS6IDZDA6IQBOXLUQBBFUAWKYA2USQV7NNU52662OZXQLINGLGMQ, seg_pos={0 0}, upload=535351]: Uploading piece... Dec 31 02:22:12.725848035 (610h53m12.59693107s) [noise=true , address=45.91.116.8:4045 , piece_id=LXE52NV3QIBRTZK27IRWWLAZ3OU2IUHB25H7V6L7XETJSR7ZN67A, seg_pos={0 0}, upload=535351]: Uploading piece... Dec 31 02:22:12.726610608 (610h53m12.597693643s) [noise=true , address=45.91.116.7:4011 , piece_id=OWH5GGYCNXNDSZVDKNMKLXSMO25MW4I4TAJPZJLFKMBQKUSSPG3Q, seg_pos={0 0}, upload=535351]: Uploading piece... Dec 31 02:22:12.727067127 (610h53m12.598150172s) [noise=true , address=145.79.227.11:4732 , piece_id=WJTPQC5HV7CLFYOGUNNQED2LS7347DBJY7AMUMW4MJWRDV4AODUA, seg_pos={0 0}, upload=535351]: Uploading piece... Dec 31 02:22:12.727529488 (610h53m12.598612523s) [noise=true , address=145.79.227.13:4803 , piece_id=F546YDIU4YMWOPRTFXKH7YYFFFFRXDKLL5N434GL6TZ6GITFE2NA, seg_pos={0 0}, upload=535351]: Uploading piece... Dec 31 02:22:12.728202930 (610h53m12.599285975s) [noise=true , address=145.79.227.31:4798 , piece_id=XHAR65JKVRG3ZGFKFRBTBOF6ZY5AKHTUPDI5DRXAM6ZYY3CJWEJA, seg_pos={0 0}, upload=535351]: Uploading piece... Dec 31 02:22:12.729252208 (610h53m12.600335253s) [noise=true , address=145.79.227.21:4474 , piece_id=VQKDL2DP37I3XY7BG7NDXAF7FDSMUVDRGQ3FU23EVTFFCHAT4L6A, seg_pos={0 0}, upload=535351]: Uploading piece... Dec 31 02:22:12.729402474 (610h53m12.600485509s) [noise=true , address=77.90.31.11:4555 , piece_id=VMAFYHAEN5YKSBAN33PS7FUSNBDQ3DBO5JG7KRBNJWBNW7M5LCYA, seg_pos={0 0}, upload=535351]: Uploading piece... Dec 31 02:22:12.730436854 (610h53m12.601519889s) [noise=true , address=145.79.227.6:4258 , piece_id=ONT4R23H4HUNEPFXYI7XCUL55STXVZH3P5QGQKCDI3B7PEZCKFPQ, seg_pos={0 0}, upload=535351]: Uploading piece... Dec 31 02:22:12.730941896 (610h53m12.602024931s) [noise=true , address=145.79.227.3:4151 , piece_id=466E5IHSVX2ZQ3UPTTLSHH4EUPZJ4H3PUY2XTZHYOYIFYBMFGGHA, seg_pos={0 0}, upload=535351]: Uploading piece... Dec 31 02:22:12.731601811 (610h53m12.602684846s) [noise=true , address=77.90.31.14:4661 , piece_id=5MI4JBBRHTC5MR57TUBEHNZVX6OD7AODOPSBCYH4TSYUPUYDRGNQ, seg_pos={0 0}, upload=535351]: Uploading piece... Dec 31 02:22:12.732236310 (610h53m12.603319355s) [noise=true , address=145.79.227.9:4367 , piece_id=5FD3BXSD5MB3N4C76AQ7P6CM4YZ2WUCJAV4TMFCOXTCTCHSMNIRQ, seg_pos={0 0}, upload=535351]: Uploading piece... Dec 31 02:22:14.430774662 (610h53m14.301857707s) [upload=535351]: Next returned nil segment Dec 31 02:22:14.430792586 (610h53m14.301875631s) [upload=535351]: Waiting for error group managing segments... Dec 31 02:22:14.711554761 (610h53m14.582637806s) [noise=true , address=145.79.227.21:4474 , piece_id=VQKDL2DP37I3XY7BG7NDXAF7FDSMUVDRGQ3FU23EVTFFCHAT4L6A, seg_pos={0 0}, upload=535351]: Done uploading piece. err: <nil> Dec 31 02:22:14.714690121 (610h53m14.585773166s) [noise=true , address=145.79.227.9:4367 , piece_id=5FD3BXSD5MB3N4C76AQ7P6CM4YZ2WUCJAV4TMFCOXTCTCHSMNIRQ, seg_pos={0 0}, upload=535351]: Done uploading piece. err: <nil> Dec 31 02:22:14.718092480 (610h53m14.589175525s) [noise=true , address=145.79.227.6:4258 , piece_id=ONT4R23H4HUNEPFXYI7XCUL55STXVZH3P5QGQKCDI3B7PEZCKFPQ, seg_pos={0 0}, upload=535351]: Done uploading piece. err: <nil> Dec 31 02:22:14.719657861 (610h53m14.590740906s) [noise=true , address=145.79.227.3:4151 , piece_id=466E5IHSVX2ZQ3UPTTLSHH4EUPZJ4H3PUY2XTZHYOYIFYBMFGGHA, seg_pos={0 0}, upload=535351]: Done uploading piece. err: <nil> Dec 31 02:22:14.728292722 (610h53m14.599375757s) [noise=true , address=145.79.227.13:4803 , piece_id=F546YDIU4YMWOPRTFXKH7YYFFFFRXDKLL5N434GL6TZ6GITFE2NA, seg_pos={0 0}, upload=535351]: Done uploading piece. err: <nil> Dec 31 02:22:14.728532268 (610h53m14.599615314s) [noise=true , address=145.79.227.37:4565 , piece_id=XSVPD47ZDWNMFOJJXGBGUWNN3I4H5MXJHIMGSRSWHRN5OHK4EIGQ, seg_pos={0 0}, upload=535351]: Done uploading piece. err: <nil> Dec 31 02:22:14.734847814 (610h53m14.605930859s) [noise=true , address=145.79.227.52:4447 , piece_id=AOEU3DUQFDXJAIRV2U7V72IXUCTVVTDQ3O4SHYDRUZ6BNCTIHGIQ, seg_pos={0 0}, upload=535351]: Done uploading piece. err: <nil> Dec 31 02:22:14.741499390 (610h53m14.612582425s) [noise=true , address=145.79.227.11:4732 , piece_id=WJTPQC5HV7CLFYOGUNNQED2LS7347DBJY7AMUMW4MJWRDV4AODUA, seg_pos={0 0}, upload=535351]: Done uploading piece. err: <nil> Dec 31 02:22:14.744680406 (610h53m14.615763452s) [noise=true , address=145.79.227.23:4547 , piece_id=JF4GEBAKZHAAFTR6WG7AGQ33LRL22ZTMK5EDPUZQKUWLAONLFGHA, seg_pos={0 0}, upload=535351]: Done uploading piece. err: <nil> Dec 31 02:22:14.799469321 (610h53m14.670552356s) [noise=true , address=145.79.227.31:4798 , piece_id=XHAR65JKVRG3ZGFKFRBTBOF6ZY5AKHTUPDI5DRXAM6ZYY3CJWEJA, seg_pos={0 0}, upload=535351]: Done uploading piece. err: <nil> Dec 31 02:22:14.799917525 (610h53m14.67100057s) [noise=true , address=145.79.227.24:4581 , piece_id=LS6IDZDA6IQBOXLUQBBFUAWKYA2USQV7NNU52662OZXQLINGLGMQ, seg_pos={0 0}, upload=535351]: Done uploading piece. err: <nil> Dec 31 02:22:15.084260048 (610h53m14.955343103s) [noise=true , address=45.91.116.8:4045 , piece_id=LXE52NV3QIBRTZK27IRWWLAZ3OU2IUHB25H7V6L7XETJSR7ZN67A, seg_pos={0 0}, upload=535351]: Done uploading piece. err: <nil> Dec 31 02:22:15.144344775 (610h53m15.01542781s) [noise=true , address=45.91.116.7:4011 , piece_id=OWH5GGYCNXNDSZVDKNMKLXSMO25MW4I4TAJPZJLFKMBQKUSSPG3Q, seg_pos={0 0}, upload=535351]: Done uploading piece. err: <nil> Dec 31 02:22:15.144367880 (610h53m15.015450915s) [seg_pos={0 0}, upload=535351]: Segment reached optimal threshold of 13 pieces. Dec 31 02:22:15.144589732 (610h53m15.015672767s) [noise=true , address=45.91.116.4:4939 , piece_id=RIQWQXXNI7SCIR7TJXKETDANJ7CLQNRSIV5RS67F4RJYL66XFTSA, seg_pos={0 0}, upload=535351]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 31 02:22:15.144597046 (610h53m15.015680082s) [noise=true , address=45.91.116.4:4939 , piece_id=RIQWQXXNI7SCIR7TJXKETDANJ7CLQNRSIV5RS67F4RJYL66XFTSA, seg_pos={0 0}, upload=535351]: Long tail cancellation due to optimal threshold Dec 31 02:22:15.144618166 (610h53m15.015701201s) [noise=true , address=77.90.31.14:4661 , piece_id=5MI4JBBRHTC5MR57TUBEHNZVX6OD7AODOPSBCYH4TSYUPUYDRGNQ, seg_pos={0 0}, upload=535351]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Dec 31 02:22:15.144626191 (610h53m15.015709226s) [noise=true , address=77.90.31.14:4661 , piece_id=5MI4JBBRHTC5MR57TUBEHNZVX6OD7AODOPSBCYH4TSYUPUYDRGNQ, seg_pos={0 0}, upload=535351]: Long tail cancellation due to optimal threshold Dec 31 02:22:15.144800363 (610h53m15.015883398s) [noise=true , address=77.90.31.11:4555 , piece_id=VMAFYHAEN5YKSBAN33PS7FUSNBDQ3DBO5JG7KRBNJWBNW7M5LCYA, seg_pos={0 0}, upload=535351]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: write tcp 188.166.225.170:50294->77.90.31.11:4555: use of closed network connection; write tcp 188.166.225.170:50294->77.90.31.11:4555: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50294->77.90.31.11:4555: use of closed network connection Dec 31 02:22:15.144807157 (610h53m15.015890192s) [noise=true , address=77.90.31.11:4555 , piece_id=VMAFYHAEN5YKSBAN33PS7FUSNBDQ3DBO5JG7KRBNJWBNW7M5LCYA, seg_pos={0 0}, upload=535351]: Long tail cancellation due to optimal threshold Dec 31 02:22:15.144821844 (610h53m15.015904889s) [seg_pos={0 0}, upload=535351]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 02:22:15.144898269 (610h53m15.015981304s) [upload=535351]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 02:22:15.144903008 (610h53m15.015986043s) [upload=535351]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 02:22:15.439148899 (610h53m15.310231934s) [upload=535351]: Done uploading segments... Dec 31 02:22:15.439170178 (610h53m15.310253214s) [upload=535351]: Upload finished. err: <nil> Duration: 24.407387504s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 51.617814766s trace: https://jaeger.dev.storj.io/trace/365693fea00df28 FAIL (2025-12-31 02:26:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [noise=true , address=145.79.227.49:4343 , piece_id=6L3HPTS7D2NRA4QEOH4RFM3D3B53CTCONRI6G6YVU4OOTUWTG7XA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.942060847 (610h25m24.813143892s) [noise=true , address=145.79.227.34:4937 , piece_id=5W7LBM3YJJTVUFYTELR57DVXB663OHO5N3Z2SPQPZ4G43IK47O3Q, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.942684104 (610h25m24.813767139s) [noise=true , address=64.49.13.139:4478 , piece_id=QN6RT2KWEA6APQVA66ZMKG4LH537UGYNOXOCVVA2YC2MKDOLT3YA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.943064007 (610h25m24.814147042s) [noise=true , address=145.79.227.40:4084 , piece_id=RUKTKDII6BRUWSYVS65CWUBT3QKG3ASUK3KA3O5VGE53XSG6IMOQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.943459560 (610h25m24.814542605s) [noise=true , address=145.79.227.14:4847 , piece_id=JX5NXTMVM3NSWIEVR4RX46ABSNPKTOFDLFBPKEXFQ3G3ANKV5OLA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.943865303 (610h25m24.814948348s) [noise=true , address=45.91.116.6:4981 , piece_id=CT474ZA2LVAZEBJHBDD4GB6NTHVHZFJLNE5OZV3DFS2NZKVW3EQQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.944259694 (610h25m24.815342749s) [noise=true , address=145.79.227.51:4443 , piece_id=73UUMRDY7BSZNCGBXCZYYRBVE5KLPMNPRNXOZEGWKQGLHJS32J2Q, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.946575182 (610h25m24.817658227s) [noise=true , address=77.90.31.10:4523 , piece_id=P6LIONXJI6AFZ2JTKCQ2W6RZCOZCK6MZ53EXAJKXYQXSX2PX4QEA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.948132909 (610h25m24.819215944s) [noise=true , address=145.79.227.20:4452 , piece_id=ZSDN2JZ3FQUVJYN5OURDXNKEDLDNULJOHKKRZVIPXZIWIYAHCRXQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.949605774 (610h25m24.820688819s) [noise=true , address=77.90.31.14:4680 , piece_id=S2E4WBA7MJVAZLT5JG3IMHK53RQC77CCHV2PDF6L2R2JEA2U3K4Q, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.951459002 (610h25m24.822542048s) [noise=true , address=193.149.164.143:4704 , piece_id=BHXHFBKTLTDMGXWY5QRTP2THZQE5OXFZ2LCMTO2KRYRWAHXWW3NQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.952986742 (610h25m24.824069777s) [noise=true , address=77.90.31.19:4848 , piece_id=VZYG32S6ODIPDKQYZRXECBHXOERZDDWD2OF35Z46W4XMUZ3OJLAA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.956916654 (610h25m24.827999689s) [noise=true , address=145.79.227.29:4971 , piece_id=MEPXHHOQ6SUZC3GIPCQEMOKKF5OJTKZEI3UYZBAYXAQZGSVEKSBA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.960665062 (610h25m24.831748097s) [noise=true , address=64.49.13.170:4720 , piece_id=HIATRSAPE4YLSODFZDPP43ZUMLDVM7GJEVMWE4ZGUQTP25LTHGTA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.962366001 (610h25m24.833449036s) [noise=true , address=145.79.227.2:4117 , piece_id=4SGFGB3IJ7LTXDMMWM2PGGOV64MC46IZ7OZQWH55J4NBQY7CHMZQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.963708216 (610h25m24.834791261s) [noise=true , address=145.79.227.35:4505 , piece_id=V7RS7A4QLZTH4G32ZUZNSF2Z5APVWZGJO77NEGP3SBWQT3WCMVLQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:26.352236138 (610h25m26.223319173s) [upload=534952]: Next returned nil segment Dec 31 01:54:26.352253821 (610h25m26.223336857s) [upload=534952]: Waiting for error group managing segments... Dec 31 01:54:26.812758374 (610h25m26.683841409s) [noise=true , address=145.79.227.51:4443 , piece_id=73UUMRDY7BSZNCGBXCZYYRBVE5KLPMNPRNXOZEGWKQGLHJS32J2Q, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.824371056 (610h25m26.695454101s) [noise=true , address=145.79.227.14:4847 , piece_id=JX5NXTMVM3NSWIEVR4RX46ABSNPKTOFDLFBPKEXFQ3G3ANKV5OLA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.838414046 (610h25m26.709497081s) [noise=true , address=64.49.13.138:4490 , piece_id=2XTUZE6PXRE3OARNJVT36FXXWUMYIBVNE6M2VVFAVBGKEE2H5YZQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.875499841 (610h25m26.746582876s) [noise=true , address=64.49.13.139:4478 , piece_id=QN6RT2KWEA6APQVA66ZMKG4LH537UGYNOXOCVVA2YC2MKDOLT3YA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.891607221 (610h25m26.762690256s) [noise=true , address=145.79.227.20:4452 , piece_id=ZSDN2JZ3FQUVJYN5OURDXNKEDLDNULJOHKKRZVIPXZIWIYAHCRXQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.988561223 (610h25m26.859644268s) [noise=true , address=64.49.13.170:4720 , piece_id=HIATRSAPE4YLSODFZDPP43ZUMLDVM7GJEVMWE4ZGUQTP25LTHGTA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.039035911 (610h25m26.910118946s) [noise=true , address=77.90.31.14:4680 , piece_id=S2E4WBA7MJVAZLT5JG3IMHK53RQC77CCHV2PDF6L2R2JEA2U3K4Q, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.098023090 (610h25m26.969106126s) [noise=true , address=45.91.116.6:4981 , piece_id=CT474ZA2LVAZEBJHBDD4GB6NTHVHZFJLNE5OZV3DFS2NZKVW3EQQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.653790754 (610h25m27.524873789s) [noise=true , address=77.90.31.21:4886 , piece_id=K34WPIBGGGEPHIKJ253XGLDCIQ3WLFN4AZ6EYNWVHHT2EIHS6YRQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.684192972 (610h25m27.555276017s) [noise=true , address=193.149.164.143:4704 , piece_id=BHXHFBKTLTDMGXWY5QRTP2THZQE5OXFZ2LCMTO2KRYRWAHXWW3NQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.717238425 (610h25m27.58832146s) [noise=true , address=145.79.227.40:4084 , piece_id=RUKTKDII6BRUWSYVS65CWUBT3QKG3ASUK3KA3O5VGE53XSG6IMOQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.718349742 (610h25m27.589432777s) [noise=true , address=77.90.31.10:4523 , piece_id=P6LIONXJI6AFZ2JTKCQ2W6RZCOZCK6MZ53EXAJKXYQXSX2PX4QEA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.803639562 (610h25m27.674722597s) [noise=true , address=145.79.227.29:4971 , piece_id=MEPXHHOQ6SUZC3GIPCQEMOKKF5OJTKZEI3UYZBAYXAQZGSVEKSBA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.812791279 (610h25m27.683874314s) [noise=true , address=145.79.227.49:4343 , piece_id=6L3HPTS7D2NRA4QEOH4RFM3D3B53CTCONRI6G6YVU4OOTUWTG7XA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.812812348 (610h25m27.683895393s) [seg_pos={0 0}, upload=534952]: Segment reached optimal threshold of 14 pieces. Dec 31 01:54:27.813018000 (610h25m27.684101035s) [noise=true , address=145.79.227.35:4505 , piece_id=V7RS7A4QLZTH4G32ZUZNSF2Z5APVWZGJO77NEGP3SBWQT3WCMVLQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i6itGoBhLsLiqC8et8Y5L62fLjcWxy18u39NAr4R3nXNMWcsf): protocol: expected piece hash; context canceled; EOF Dec 31 01:54:27.813024772 (610h25m27.684107808s) [noise=true , address=145.79.227.35:4505 , piece_id=V7RS7A4QLZTH4G32ZUZNSF2Z5APVWZGJO77NEGP3SBWQT3WCMVLQ, seg_pos={0 0}, upload=534952]: Long tail cancellation due to optimal threshold Dec 31 01:54:27.813175931 (610h25m27.684258976s) [noise=true , address=145.79.227.34:4937 , piece_id=5W7LBM3YJJTVUFYTELR57DVXB663OHO5N3Z2SPQPZ4G43IK47O3Q, seg_pos={0 0}, upload=534952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1LKrFThdW3jJpG6dRZLUaxLXTa5Wy27yQ8PGUEqjvBWfFoRB94): protocol: expected piece hash; context canceled; EOF Dec 31 01:54:27.813181702 (610h25m27.684264737s) [noise=true , address=145.79.227.34:4937 , piece_id=5W7LBM3YJJTVUFYTELR57DVXB663OHO5N3Z2SPQPZ4G43IK47O3Q, seg_pos={0 0}, upload=534952]: Long tail cancellation due to optimal threshold Dec 31 01:54:27.813407862 (610h25m27.684490907s) [noise=true , address=145.79.227.2:4117 , piece_id=4SGFGB3IJ7LTXDMMWM2PGGOV64MC46IZ7OZQWH55J4NBQY7CHMZQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122Htg72f5eUUutf1p3fYQpy1Wkf6AgdaQV83HHYye2n8zm3the): protocol: expected piece hash; context canceled; EOF Dec 31 01:54:27.813414264 (610h25m27.684497299s) [noise=true , address=145.79.227.2:4117 , piece_id=4SGFGB3IJ7LTXDMMWM2PGGOV64MC46IZ7OZQWH55J4NBQY7CHMZQ, seg_pos={0 0}, upload=534952]: Long tail cancellation due to optimal threshold Dec 31 01:54:27.813424614 (610h25m27.684507649s) [noise=true , address=77.90.31.19:4848 , piece_id=VZYG32S6ODIPDKQYZRXECBHXOERZDDWD2OF35Z46W4XMUZ3OJLAA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD): protocol: write tcp 188.166.225.170:42842->77.90.31.19:4848: use of closed network connection; write tcp 188.166.225.170:42842->77.90.31.19:4848: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42842->77.90.31.19:4848: use of closed network connection Dec 31 01:54:27.813432750 (610h25m27.684515785s) [noise=true , address=77.90.31.19:4848 , piece_id=VZYG32S6ODIPDKQYZRXECBHXOERZDDWD2OF35Z46W4XMUZ3OJLAA, seg_pos={0 0}, upload=534952]: Long tail cancellation due to optimal threshold Dec 31 01:54:27.813474088 (610h25m27.684557123s) [seg_pos={0 0}, upload=534952]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 01:54:27.813535325 (610h25m27.68461836s) [upload=534952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 01:54:27.813539613 (610h25m27.684622648s) [upload=534952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 01:54:28.099921034 (610h25m27.971004069s) [upload=534952]: Done uploading segments... Dec 31 01:54:28.099940451 (610h25m27.971023486s) [upload=534952]: Upload finished. err: <nil> Duration: 51.617908636s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [....X..................................X....................] duration: 3.593028458s trace: https://jaeger.dev.storj.io/trace/740aa405bd68150f FAIL (2025-12-31 02:25:57 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.593053537s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 47.545301882s trace: https://jaeger.dev.storj.io/trace/365693fea00df28 FAIL (2025-12-31 02:25:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [noise=true , address=145.79.227.49:4343 , piece_id=6L3HPTS7D2NRA4QEOH4RFM3D3B53CTCONRI6G6YVU4OOTUWTG7XA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.942060847 (610h25m24.813143892s) [noise=true , address=145.79.227.34:4937 , piece_id=5W7LBM3YJJTVUFYTELR57DVXB663OHO5N3Z2SPQPZ4G43IK47O3Q, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.942684104 (610h25m24.813767139s) [noise=true , address=64.49.13.139:4478 , piece_id=QN6RT2KWEA6APQVA66ZMKG4LH537UGYNOXOCVVA2YC2MKDOLT3YA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.943064007 (610h25m24.814147042s) [noise=true , address=145.79.227.40:4084 , piece_id=RUKTKDII6BRUWSYVS65CWUBT3QKG3ASUK3KA3O5VGE53XSG6IMOQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.943459560 (610h25m24.814542605s) [noise=true , address=145.79.227.14:4847 , piece_id=JX5NXTMVM3NSWIEVR4RX46ABSNPKTOFDLFBPKEXFQ3G3ANKV5OLA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.943865303 (610h25m24.814948348s) [noise=true , address=45.91.116.6:4981 , piece_id=CT474ZA2LVAZEBJHBDD4GB6NTHVHZFJLNE5OZV3DFS2NZKVW3EQQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.944259694 (610h25m24.815342749s) [noise=true , address=145.79.227.51:4443 , piece_id=73UUMRDY7BSZNCGBXCZYYRBVE5KLPMNPRNXOZEGWKQGLHJS32J2Q, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.946575182 (610h25m24.817658227s) [noise=true , address=77.90.31.10:4523 , piece_id=P6LIONXJI6AFZ2JTKCQ2W6RZCOZCK6MZ53EXAJKXYQXSX2PX4QEA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.948132909 (610h25m24.819215944s) [noise=true , address=145.79.227.20:4452 , piece_id=ZSDN2JZ3FQUVJYN5OURDXNKEDLDNULJOHKKRZVIPXZIWIYAHCRXQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.949605774 (610h25m24.820688819s) [noise=true , address=77.90.31.14:4680 , piece_id=S2E4WBA7MJVAZLT5JG3IMHK53RQC77CCHV2PDF6L2R2JEA2U3K4Q, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.951459002 (610h25m24.822542048s) [noise=true , address=193.149.164.143:4704 , piece_id=BHXHFBKTLTDMGXWY5QRTP2THZQE5OXFZ2LCMTO2KRYRWAHXWW3NQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.952986742 (610h25m24.824069777s) [noise=true , address=77.90.31.19:4848 , piece_id=VZYG32S6ODIPDKQYZRXECBHXOERZDDWD2OF35Z46W4XMUZ3OJLAA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.956916654 (610h25m24.827999689s) [noise=true , address=145.79.227.29:4971 , piece_id=MEPXHHOQ6SUZC3GIPCQEMOKKF5OJTKZEI3UYZBAYXAQZGSVEKSBA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.960665062 (610h25m24.831748097s) [noise=true , address=64.49.13.170:4720 , piece_id=HIATRSAPE4YLSODFZDPP43ZUMLDVM7GJEVMWE4ZGUQTP25LTHGTA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.962366001 (610h25m24.833449036s) [noise=true , address=145.79.227.2:4117 , piece_id=4SGFGB3IJ7LTXDMMWM2PGGOV64MC46IZ7OZQWH55J4NBQY7CHMZQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.963708216 (610h25m24.834791261s) [noise=true , address=145.79.227.35:4505 , piece_id=V7RS7A4QLZTH4G32ZUZNSF2Z5APVWZGJO77NEGP3SBWQT3WCMVLQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:26.352236138 (610h25m26.223319173s) [upload=534952]: Next returned nil segment Dec 31 01:54:26.352253821 (610h25m26.223336857s) [upload=534952]: Waiting for error group managing segments... Dec 31 01:54:26.812758374 (610h25m26.683841409s) [noise=true , address=145.79.227.51:4443 , piece_id=73UUMRDY7BSZNCGBXCZYYRBVE5KLPMNPRNXOZEGWKQGLHJS32J2Q, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.824371056 (610h25m26.695454101s) [noise=true , address=145.79.227.14:4847 , piece_id=JX5NXTMVM3NSWIEVR4RX46ABSNPKTOFDLFBPKEXFQ3G3ANKV5OLA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.838414046 (610h25m26.709497081s) [noise=true , address=64.49.13.138:4490 , piece_id=2XTUZE6PXRE3OARNJVT36FXXWUMYIBVNE6M2VVFAVBGKEE2H5YZQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.875499841 (610h25m26.746582876s) [noise=true , address=64.49.13.139:4478 , piece_id=QN6RT2KWEA6APQVA66ZMKG4LH537UGYNOXOCVVA2YC2MKDOLT3YA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.891607221 (610h25m26.762690256s) [noise=true , address=145.79.227.20:4452 , piece_id=ZSDN2JZ3FQUVJYN5OURDXNKEDLDNULJOHKKRZVIPXZIWIYAHCRXQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.988561223 (610h25m26.859644268s) [noise=true , address=64.49.13.170:4720 , piece_id=HIATRSAPE4YLSODFZDPP43ZUMLDVM7GJEVMWE4ZGUQTP25LTHGTA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.039035911 (610h25m26.910118946s) [noise=true , address=77.90.31.14:4680 , piece_id=S2E4WBA7MJVAZLT5JG3IMHK53RQC77CCHV2PDF6L2R2JEA2U3K4Q, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.098023090 (610h25m26.969106126s) [noise=true , address=45.91.116.6:4981 , piece_id=CT474ZA2LVAZEBJHBDD4GB6NTHVHZFJLNE5OZV3DFS2NZKVW3EQQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.653790754 (610h25m27.524873789s) [noise=true , address=77.90.31.21:4886 , piece_id=K34WPIBGGGEPHIKJ253XGLDCIQ3WLFN4AZ6EYNWVHHT2EIHS6YRQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.684192972 (610h25m27.555276017s) [noise=true , address=193.149.164.143:4704 , piece_id=BHXHFBKTLTDMGXWY5QRTP2THZQE5OXFZ2LCMTO2KRYRWAHXWW3NQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.717238425 (610h25m27.58832146s) [noise=true , address=145.79.227.40:4084 , piece_id=RUKTKDII6BRUWSYVS65CWUBT3QKG3ASUK3KA3O5VGE53XSG6IMOQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.718349742 (610h25m27.589432777s) [noise=true , address=77.90.31.10:4523 , piece_id=P6LIONXJI6AFZ2JTKCQ2W6RZCOZCK6MZ53EXAJKXYQXSX2PX4QEA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.803639562 (610h25m27.674722597s) [noise=true , address=145.79.227.29:4971 , piece_id=MEPXHHOQ6SUZC3GIPCQEMOKKF5OJTKZEI3UYZBAYXAQZGSVEKSBA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.812791279 (610h25m27.683874314s) [noise=true , address=145.79.227.49:4343 , piece_id=6L3HPTS7D2NRA4QEOH4RFM3D3B53CTCONRI6G6YVU4OOTUWTG7XA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.812812348 (610h25m27.683895393s) [seg_pos={0 0}, upload=534952]: Segment reached optimal threshold of 14 pieces. Dec 31 01:54:27.813018000 (610h25m27.684101035s) [noise=true , address=145.79.227.35:4505 , piece_id=V7RS7A4QLZTH4G32ZUZNSF2Z5APVWZGJO77NEGP3SBWQT3WCMVLQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i6itGoBhLsLiqC8et8Y5L62fLjcWxy18u39NAr4R3nXNMWcsf): protocol: expected piece hash; context canceled; EOF Dec 31 01:54:27.813024772 (610h25m27.684107808s) [noise=true , address=145.79.227.35:4505 , piece_id=V7RS7A4QLZTH4G32ZUZNSF2Z5APVWZGJO77NEGP3SBWQT3WCMVLQ, seg_pos={0 0}, upload=534952]: Long tail cancellation due to optimal threshold Dec 31 01:54:27.813175931 (610h25m27.684258976s) [noise=true , address=145.79.227.34:4937 , piece_id=5W7LBM3YJJTVUFYTELR57DVXB663OHO5N3Z2SPQPZ4G43IK47O3Q, seg_pos={0 0}, upload=534952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1LKrFThdW3jJpG6dRZLUaxLXTa5Wy27yQ8PGUEqjvBWfFoRB94): protocol: expected piece hash; context canceled; EOF Dec 31 01:54:27.813181702 (610h25m27.684264737s) [noise=true , address=145.79.227.34:4937 , piece_id=5W7LBM3YJJTVUFYTELR57DVXB663OHO5N3Z2SPQPZ4G43IK47O3Q, seg_pos={0 0}, upload=534952]: Long tail cancellation due to optimal threshold Dec 31 01:54:27.813407862 (610h25m27.684490907s) [noise=true , address=145.79.227.2:4117 , piece_id=4SGFGB3IJ7LTXDMMWM2PGGOV64MC46IZ7OZQWH55J4NBQY7CHMZQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122Htg72f5eUUutf1p3fYQpy1Wkf6AgdaQV83HHYye2n8zm3the): protocol: expected piece hash; context canceled; EOF Dec 31 01:54:27.813414264 (610h25m27.684497299s) [noise=true , address=145.79.227.2:4117 , piece_id=4SGFGB3IJ7LTXDMMWM2PGGOV64MC46IZ7OZQWH55J4NBQY7CHMZQ, seg_pos={0 0}, upload=534952]: Long tail cancellation due to optimal threshold Dec 31 01:54:27.813424614 (610h25m27.684507649s) [noise=true , address=77.90.31.19:4848 , piece_id=VZYG32S6ODIPDKQYZRXECBHXOERZDDWD2OF35Z46W4XMUZ3OJLAA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD): protocol: write tcp 188.166.225.170:42842->77.90.31.19:4848: use of closed network connection; write tcp 188.166.225.170:42842->77.90.31.19:4848: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42842->77.90.31.19:4848: use of closed network connection Dec 31 01:54:27.813432750 (610h25m27.684515785s) [noise=true , address=77.90.31.19:4848 , piece_id=VZYG32S6ODIPDKQYZRXECBHXOERZDDWD2OF35Z46W4XMUZ3OJLAA, seg_pos={0 0}, upload=534952]: Long tail cancellation due to optimal threshold Dec 31 01:54:27.813474088 (610h25m27.684557123s) [seg_pos={0 0}, upload=534952]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 01:54:27.813535325 (610h25m27.68461836s) [upload=534952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 01:54:27.813539613 (610h25m27.684622648s) [upload=534952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 01:54:28.099921034 (610h25m27.971004069s) [upload=534952]: Done uploading segments... Dec 31 01:54:28.099940451 (610h25m27.971023486s) [upload=534952]: Upload finished. err: <nil> Duration: 47.545318053s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.....X..................................X...................] duration: 3.497247101s trace: https://jaeger.dev.storj.io/trace/740aa405bd68150f FAIL (2025-12-31 02:25:31 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.49728868s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.......X............X..X.......X.X..X....XX.XX........X.X...] duration: 6.709840879s trace: https://jaeger.dev.storj.io/trace/3b4011ac5a42e158 FAIL (2025-12-31 02:25:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: LXE52NV3QIBRTZK27IRWWLAZ3OU2IUHB25H7V6L7XETJSR7ZN67A) context deadline exceeded; piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: OWH5GGYCNXNDSZVDKNMKLXSMO25MW4I4TAJPZJLFKMBQKUSSPG3Q) context deadline exceeded ======LOGS====== 31 02:22:12.475370476 (610h53m12.346453511s) [upload=535351]: Starting upload Dec 31 02:22:12.475429659 (610h53m12.346512694s) [upload=535351]: Done starting upload Dec 31 02:22:12.475607157 (610h53m12.346690202s) [upload=535351]: Uploading segments... Dec 31 02:22:12.475783423 (610h53m12.346866468s) [upload=535351]: Got next segment. Inline: false Dec 31 02:22:12.475838657 (610h53m12.346921702s) [upload=535351]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 02:22:12.475845551 (610h53m12.346928596s) [upload=535351]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 02:22:12.724117301 (610h53m12.595200346s) [seg_pos={0 0}, upload=535351]: Begin upload segment... Dec 31 02:22:12.724171443 (610h53m12.595254488s) [seg_pos={0 0}, upload=535351]: Done begin upload segment. Dec 31 02:22:12.724222331 (610h53m12.595305376s) [noise=true , address=145.79.227.52:4447 , piece_id=AOEU3DUQFDXJAIRV2U7V72IXUCTVVTDQ3O4SHYDRUZ6BNCTIHGIQ, seg_pos={0 0}, upload=535351]: Uploading piece... Dec 31 02:22:12.724287495 (610h53m12.59537054s) [noise=true , address=145.79.227.37:4565 , piece_id=XSVPD47ZDWNMFOJJXGBGUWNN3I4H5MXJHIMGSRSWHRN5OHK4EIGQ, seg_pos={0 0}, upload=535351]: Uploading piece... Dec 31 02:22:12.724817614 (610h53m12.595900659s) [noise=true , address=45.91.116.4:4939 , piece_id=RIQWQXXNI7SCIR7TJXKETDANJ7CLQNRSIV5RS67F4RJYL66XFTSA, seg_pos={0 0}, upload=535351]: Uploading piece... Dec 31 02:22:12.725168582 (610h53m12.596251627s) [noise=true , address=145.79.227.23:4547 , piece_id=JF4GEBAKZHAAFTR6WG7AGQ33LRL22ZTMK5EDPUZQKUWLAONLFGHA, seg_pos={0 0}, upload=535351]: Uploading piece... Dec 31 02:22:12.725489473 (610h53m12.596572508s) [noise=true , address=145.79.227.24:4581 , piece_id=LS6IDZDA6IQBOXLUQBBFUAWKYA2USQV7NNU52662OZXQLINGLGMQ, seg_pos={0 0}, upload=535351]: Uploading piece... Dec 31 02:22:12.725848035 (610h53m12.59693107s) [noise=true , address=45.91.116.8:4045 , piece_id=LXE52NV3QIBRTZK27IRWWLAZ3OU2IUHB25H7V6L7XETJSR7ZN67A, seg_pos={0 0}, upload=535351]: Uploading piece... Dec 31 02:22:12.726610608 (610h53m12.597693643s) [noise=true , address=45.91.116.7:4011 , piece_id=OWH5GGYCNXNDSZVDKNMKLXSMO25MW4I4TAJPZJLFKMBQKUSSPG3Q, seg_pos={0 0}, upload=535351]: Uploading piece... Dec 31 02:22:12.727067127 (610h53m12.598150172s) [noise=true , address=145.79.227.11:4732 , piece_id=WJTPQC5HV7CLFYOGUNNQED2LS7347DBJY7AMUMW4MJWRDV4AODUA, seg_pos={0 0}, upload=535351]: Uploading piece... Dec 31 02:22:12.727529488 (610h53m12.598612523s) [noise=true , address=145.79.227.13:4803 , piece_id=F546YDIU4YMWOPRTFXKH7YYFFFFRXDKLL5N434GL6TZ6GITFE2NA, seg_pos={0 0}, upload=535351]: Uploading piece... Dec 31 02:22:12.728202930 (610h53m12.599285975s) [noise=true , address=145.79.227.31:4798 , piece_id=XHAR65JKVRG3ZGFKFRBTBOF6ZY5AKHTUPDI5DRXAM6ZYY3CJWEJA, seg_pos={0 0}, upload=535351]: Uploading piece... Dec 31 02:22:12.729252208 (610h53m12.600335253s) [noise=true , address=145.79.227.21:4474 , piece_id=VQKDL2DP37I3XY7BG7NDXAF7FDSMUVDRGQ3FU23EVTFFCHAT4L6A, seg_pos={0 0}, upload=535351]: Uploading piece... Dec 31 02:22:12.729402474 (610h53m12.600485509s) [noise=true , address=77.90.31.11:4555 , piece_id=VMAFYHAEN5YKSBAN33PS7FUSNBDQ3DBO5JG7KRBNJWBNW7M5LCYA, seg_pos={0 0}, upload=535351]: Uploading piece... Dec 31 02:22:12.730436854 (610h53m12.601519889s) [noise=true , address=145.79.227.6:4258 , piece_id=ONT4R23H4HUNEPFXYI7XCUL55STXVZH3P5QGQKCDI3B7PEZCKFPQ, seg_pos={0 0}, upload=535351]: Uploading piece... Dec 31 02:22:12.730941896 (610h53m12.602024931s) [noise=true , address=145.79.227.3:4151 , piece_id=466E5IHSVX2ZQ3UPTTLSHH4EUPZJ4H3PUY2XTZHYOYIFYBMFGGHA, seg_pos={0 0}, upload=535351]: Uploading piece... Dec 31 02:22:12.731601811 (610h53m12.602684846s) [noise=true , address=77.90.31.14:4661 , piece_id=5MI4JBBRHTC5MR57TUBEHNZVX6OD7AODOPSBCYH4TSYUPUYDRGNQ, seg_pos={0 0}, upload=535351]: Uploading piece... Dec 31 02:22:12.732236310 (610h53m12.603319355s) [noise=true , address=145.79.227.9:4367 , piece_id=5FD3BXSD5MB3N4C76AQ7P6CM4YZ2WUCJAV4TMFCOXTCTCHSMNIRQ, seg_pos={0 0}, upload=535351]: Uploading piece... Dec 31 02:22:14.430774662 (610h53m14.301857707s) [upload=535351]: Next returned nil segment Dec 31 02:22:14.430792586 (610h53m14.301875631s) [upload=535351]: Waiting for error group managing segments... Dec 31 02:22:14.711554761 (610h53m14.582637806s) [noise=true , address=145.79.227.21:4474 , piece_id=VQKDL2DP37I3XY7BG7NDXAF7FDSMUVDRGQ3FU23EVTFFCHAT4L6A, seg_pos={0 0}, upload=535351]: Done uploading piece. err: <nil> Dec 31 02:22:14.714690121 (610h53m14.585773166s) [noise=true , address=145.79.227.9:4367 , piece_id=5FD3BXSD5MB3N4C76AQ7P6CM4YZ2WUCJAV4TMFCOXTCTCHSMNIRQ, seg_pos={0 0}, upload=535351]: Done uploading piece. err: <nil> Dec 31 02:22:14.718092480 (610h53m14.589175525s) [noise=true , address=145.79.227.6:4258 , piece_id=ONT4R23H4HUNEPFXYI7XCUL55STXVZH3P5QGQKCDI3B7PEZCKFPQ, seg_pos={0 0}, upload=535351]: Done uploading piece. err: <nil> Dec 31 02:22:14.719657861 (610h53m14.590740906s) [noise=true , address=145.79.227.3:4151 , piece_id=466E5IHSVX2ZQ3UPTTLSHH4EUPZJ4H3PUY2XTZHYOYIFYBMFGGHA, seg_pos={0 0}, upload=535351]: Done uploading piece. err: <nil> Dec 31 02:22:14.728292722 (610h53m14.599375757s) [noise=true , address=145.79.227.13:4803 , piece_id=F546YDIU4YMWOPRTFXKH7YYFFFFRXDKLL5N434GL6TZ6GITFE2NA, seg_pos={0 0}, upload=535351]: Done uploading piece. err: <nil> Dec 31 02:22:14.728532268 (610h53m14.599615314s) [noise=true , address=145.79.227.37:4565 , piece_id=XSVPD47ZDWNMFOJJXGBGUWNN3I4H5MXJHIMGSRSWHRN5OHK4EIGQ, seg_pos={0 0}, upload=535351]: Done uploading piece. err: <nil> Dec 31 02:22:14.734847814 (610h53m14.605930859s) [noise=true , address=145.79.227.52:4447 , piece_id=AOEU3DUQFDXJAIRV2U7V72IXUCTVVTDQ3O4SHYDRUZ6BNCTIHGIQ, seg_pos={0 0}, upload=535351]: Done uploading piece. err: <nil> Dec 31 02:22:14.741499390 (610h53m14.612582425s) [noise=true , address=145.79.227.11:4732 , piece_id=WJTPQC5HV7CLFYOGUNNQED2LS7347DBJY7AMUMW4MJWRDV4AODUA, seg_pos={0 0}, upload=535351]: Done uploading piece. err: <nil> Dec 31 02:22:14.744680406 (610h53m14.615763452s) [noise=true , address=145.79.227.23:4547 , piece_id=JF4GEBAKZHAAFTR6WG7AGQ33LRL22ZTMK5EDPUZQKUWLAONLFGHA, seg_pos={0 0}, upload=535351]: Done uploading piece. err: <nil> Dec 31 02:22:14.799469321 (610h53m14.670552356s) [noise=true , address=145.79.227.31:4798 , piece_id=XHAR65JKVRG3ZGFKFRBTBOF6ZY5AKHTUPDI5DRXAM6ZYY3CJWEJA, seg_pos={0 0}, upload=535351]: Done uploading piece. err: <nil> Dec 31 02:22:14.799917525 (610h53m14.67100057s) [noise=true , address=145.79.227.24:4581 , piece_id=LS6IDZDA6IQBOXLUQBBFUAWKYA2USQV7NNU52662OZXQLINGLGMQ, seg_pos={0 0}, upload=535351]: Done uploading piece. err: <nil> Dec 31 02:22:15.084260048 (610h53m14.955343103s) [noise=true , address=45.91.116.8:4045 , piece_id=LXE52NV3QIBRTZK27IRWWLAZ3OU2IUHB25H7V6L7XETJSR7ZN67A, seg_pos={0 0}, upload=535351]: Done uploading piece. err: <nil> Dec 31 02:22:15.144344775 (610h53m15.01542781s) [noise=true , address=45.91.116.7:4011 , piece_id=OWH5GGYCNXNDSZVDKNMKLXSMO25MW4I4TAJPZJLFKMBQKUSSPG3Q, seg_pos={0 0}, upload=535351]: Done uploading piece. err: <nil> Dec 31 02:22:15.144367880 (610h53m15.015450915s) [seg_pos={0 0}, upload=535351]: Segment reached optimal threshold of 13 pieces. Dec 31 02:22:15.144589732 (610h53m15.015672767s) [noise=true , address=45.91.116.4:4939 , piece_id=RIQWQXXNI7SCIR7TJXKETDANJ7CLQNRSIV5RS67F4RJYL66XFTSA, seg_pos={0 0}, upload=535351]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 31 02:22:15.144597046 (610h53m15.015680082s) [noise=true , address=45.91.116.4:4939 , piece_id=RIQWQXXNI7SCIR7TJXKETDANJ7CLQNRSIV5RS67F4RJYL66XFTSA, seg_pos={0 0}, upload=535351]: Long tail cancellation due to optimal threshold Dec 31 02:22:15.144618166 (610h53m15.015701201s) [noise=true , address=77.90.31.14:4661 , piece_id=5MI4JBBRHTC5MR57TUBEHNZVX6OD7AODOPSBCYH4TSYUPUYDRGNQ, seg_pos={0 0}, upload=535351]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Dec 31 02:22:15.144626191 (610h53m15.015709226s) [noise=true , address=77.90.31.14:4661 , piece_id=5MI4JBBRHTC5MR57TUBEHNZVX6OD7AODOPSBCYH4TSYUPUYDRGNQ, seg_pos={0 0}, upload=535351]: Long tail cancellation due to optimal threshold Dec 31 02:22:15.144800363 (610h53m15.015883398s) [noise=true , address=77.90.31.11:4555 , piece_id=VMAFYHAEN5YKSBAN33PS7FUSNBDQ3DBO5JG7KRBNJWBNW7M5LCYA, seg_pos={0 0}, upload=535351]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: write tcp 188.166.225.170:50294->77.90.31.11:4555: use of closed network connection; write tcp 188.166.225.170:50294->77.90.31.11:4555: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50294->77.90.31.11:4555: use of closed network connection Dec 31 02:22:15.144807157 (610h53m15.015890192s) [noise=true , address=77.90.31.11:4555 , piece_id=VMAFYHAEN5YKSBAN33PS7FUSNBDQ3DBO5JG7KRBNJWBNW7M5LCYA, seg_pos={0 0}, upload=535351]: Long tail cancellation due to optimal threshold Dec 31 02:22:15.144821844 (610h53m15.015904889s) [seg_pos={0 0}, upload=535351]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 02:22:15.144898269 (610h53m15.015981304s) [upload=535351]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 02:22:15.144903008 (610h53m15.015986043s) [upload=535351]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 02:22:15.439148899 (610h53m15.310231934s) [upload=535351]: Done uploading segments... Dec 31 02:22:15.439170178 (610h53m15.310253214s) [upload=535351]: Upload finished. err: <nil> Duration: 6.709874281s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 34.079099041s trace: https://jaeger.dev.storj.io/trace/365693fea00df28 FAIL (2025-12-31 02:25:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [noise=true , address=145.79.227.49:4343 , piece_id=6L3HPTS7D2NRA4QEOH4RFM3D3B53CTCONRI6G6YVU4OOTUWTG7XA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.942060847 (610h25m24.813143892s) [noise=true , address=145.79.227.34:4937 , piece_id=5W7LBM3YJJTVUFYTELR57DVXB663OHO5N3Z2SPQPZ4G43IK47O3Q, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.942684104 (610h25m24.813767139s) [noise=true , address=64.49.13.139:4478 , piece_id=QN6RT2KWEA6APQVA66ZMKG4LH537UGYNOXOCVVA2YC2MKDOLT3YA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.943064007 (610h25m24.814147042s) [noise=true , address=145.79.227.40:4084 , piece_id=RUKTKDII6BRUWSYVS65CWUBT3QKG3ASUK3KA3O5VGE53XSG6IMOQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.943459560 (610h25m24.814542605s) [noise=true , address=145.79.227.14:4847 , piece_id=JX5NXTMVM3NSWIEVR4RX46ABSNPKTOFDLFBPKEXFQ3G3ANKV5OLA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.943865303 (610h25m24.814948348s) [noise=true , address=45.91.116.6:4981 , piece_id=CT474ZA2LVAZEBJHBDD4GB6NTHVHZFJLNE5OZV3DFS2NZKVW3EQQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.944259694 (610h25m24.815342749s) [noise=true , address=145.79.227.51:4443 , piece_id=73UUMRDY7BSZNCGBXCZYYRBVE5KLPMNPRNXOZEGWKQGLHJS32J2Q, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.946575182 (610h25m24.817658227s) [noise=true , address=77.90.31.10:4523 , piece_id=P6LIONXJI6AFZ2JTKCQ2W6RZCOZCK6MZ53EXAJKXYQXSX2PX4QEA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.948132909 (610h25m24.819215944s) [noise=true , address=145.79.227.20:4452 , piece_id=ZSDN2JZ3FQUVJYN5OURDXNKEDLDNULJOHKKRZVIPXZIWIYAHCRXQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.949605774 (610h25m24.820688819s) [noise=true , address=77.90.31.14:4680 , piece_id=S2E4WBA7MJVAZLT5JG3IMHK53RQC77CCHV2PDF6L2R2JEA2U3K4Q, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.951459002 (610h25m24.822542048s) [noise=true , address=193.149.164.143:4704 , piece_id=BHXHFBKTLTDMGXWY5QRTP2THZQE5OXFZ2LCMTO2KRYRWAHXWW3NQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.952986742 (610h25m24.824069777s) [noise=true , address=77.90.31.19:4848 , piece_id=VZYG32S6ODIPDKQYZRXECBHXOERZDDWD2OF35Z46W4XMUZ3OJLAA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.956916654 (610h25m24.827999689s) [noise=true , address=145.79.227.29:4971 , piece_id=MEPXHHOQ6SUZC3GIPCQEMOKKF5OJTKZEI3UYZBAYXAQZGSVEKSBA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.960665062 (610h25m24.831748097s) [noise=true , address=64.49.13.170:4720 , piece_id=HIATRSAPE4YLSODFZDPP43ZUMLDVM7GJEVMWE4ZGUQTP25LTHGTA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.962366001 (610h25m24.833449036s) [noise=true , address=145.79.227.2:4117 , piece_id=4SGFGB3IJ7LTXDMMWM2PGGOV64MC46IZ7OZQWH55J4NBQY7CHMZQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.963708216 (610h25m24.834791261s) [noise=true , address=145.79.227.35:4505 , piece_id=V7RS7A4QLZTH4G32ZUZNSF2Z5APVWZGJO77NEGP3SBWQT3WCMVLQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:26.352236138 (610h25m26.223319173s) [upload=534952]: Next returned nil segment Dec 31 01:54:26.352253821 (610h25m26.223336857s) [upload=534952]: Waiting for error group managing segments... Dec 31 01:54:26.812758374 (610h25m26.683841409s) [noise=true , address=145.79.227.51:4443 , piece_id=73UUMRDY7BSZNCGBXCZYYRBVE5KLPMNPRNXOZEGWKQGLHJS32J2Q, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.824371056 (610h25m26.695454101s) [noise=true , address=145.79.227.14:4847 , piece_id=JX5NXTMVM3NSWIEVR4RX46ABSNPKTOFDLFBPKEXFQ3G3ANKV5OLA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.838414046 (610h25m26.709497081s) [noise=true , address=64.49.13.138:4490 , piece_id=2XTUZE6PXRE3OARNJVT36FXXWUMYIBVNE6M2VVFAVBGKEE2H5YZQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.875499841 (610h25m26.746582876s) [noise=true , address=64.49.13.139:4478 , piece_id=QN6RT2KWEA6APQVA66ZMKG4LH537UGYNOXOCVVA2YC2MKDOLT3YA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.891607221 (610h25m26.762690256s) [noise=true , address=145.79.227.20:4452 , piece_id=ZSDN2JZ3FQUVJYN5OURDXNKEDLDNULJOHKKRZVIPXZIWIYAHCRXQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.988561223 (610h25m26.859644268s) [noise=true , address=64.49.13.170:4720 , piece_id=HIATRSAPE4YLSODFZDPP43ZUMLDVM7GJEVMWE4ZGUQTP25LTHGTA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.039035911 (610h25m26.910118946s) [noise=true , address=77.90.31.14:4680 , piece_id=S2E4WBA7MJVAZLT5JG3IMHK53RQC77CCHV2PDF6L2R2JEA2U3K4Q, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.098023090 (610h25m26.969106126s) [noise=true , address=45.91.116.6:4981 , piece_id=CT474ZA2LVAZEBJHBDD4GB6NTHVHZFJLNE5OZV3DFS2NZKVW3EQQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.653790754 (610h25m27.524873789s) [noise=true , address=77.90.31.21:4886 , piece_id=K34WPIBGGGEPHIKJ253XGLDCIQ3WLFN4AZ6EYNWVHHT2EIHS6YRQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.684192972 (610h25m27.555276017s) [noise=true , address=193.149.164.143:4704 , piece_id=BHXHFBKTLTDMGXWY5QRTP2THZQE5OXFZ2LCMTO2KRYRWAHXWW3NQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.717238425 (610h25m27.58832146s) [noise=true , address=145.79.227.40:4084 , piece_id=RUKTKDII6BRUWSYVS65CWUBT3QKG3ASUK3KA3O5VGE53XSG6IMOQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.718349742 (610h25m27.589432777s) [noise=true , address=77.90.31.10:4523 , piece_id=P6LIONXJI6AFZ2JTKCQ2W6RZCOZCK6MZ53EXAJKXYQXSX2PX4QEA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.803639562 (610h25m27.674722597s) [noise=true , address=145.79.227.29:4971 , piece_id=MEPXHHOQ6SUZC3GIPCQEMOKKF5OJTKZEI3UYZBAYXAQZGSVEKSBA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.812791279 (610h25m27.683874314s) [noise=true , address=145.79.227.49:4343 , piece_id=6L3HPTS7D2NRA4QEOH4RFM3D3B53CTCONRI6G6YVU4OOTUWTG7XA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.812812348 (610h25m27.683895393s) [seg_pos={0 0}, upload=534952]: Segment reached optimal threshold of 14 pieces. Dec 31 01:54:27.813018000 (610h25m27.684101035s) [noise=true , address=145.79.227.35:4505 , piece_id=V7RS7A4QLZTH4G32ZUZNSF2Z5APVWZGJO77NEGP3SBWQT3WCMVLQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i6itGoBhLsLiqC8et8Y5L62fLjcWxy18u39NAr4R3nXNMWcsf): protocol: expected piece hash; context canceled; EOF Dec 31 01:54:27.813024772 (610h25m27.684107808s) [noise=true , address=145.79.227.35:4505 , piece_id=V7RS7A4QLZTH4G32ZUZNSF2Z5APVWZGJO77NEGP3SBWQT3WCMVLQ, seg_pos={0 0}, upload=534952]: Long tail cancellation due to optimal threshold Dec 31 01:54:27.813175931 (610h25m27.684258976s) [noise=true , address=145.79.227.34:4937 , piece_id=5W7LBM3YJJTVUFYTELR57DVXB663OHO5N3Z2SPQPZ4G43IK47O3Q, seg_pos={0 0}, upload=534952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1LKrFThdW3jJpG6dRZLUaxLXTa5Wy27yQ8PGUEqjvBWfFoRB94): protocol: expected piece hash; context canceled; EOF Dec 31 01:54:27.813181702 (610h25m27.684264737s) [noise=true , address=145.79.227.34:4937 , piece_id=5W7LBM3YJJTVUFYTELR57DVXB663OHO5N3Z2SPQPZ4G43IK47O3Q, seg_pos={0 0}, upload=534952]: Long tail cancellation due to optimal threshold Dec 31 01:54:27.813407862 (610h25m27.684490907s) [noise=true , address=145.79.227.2:4117 , piece_id=4SGFGB3IJ7LTXDMMWM2PGGOV64MC46IZ7OZQWH55J4NBQY7CHMZQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122Htg72f5eUUutf1p3fYQpy1Wkf6AgdaQV83HHYye2n8zm3the): protocol: expected piece hash; context canceled; EOF Dec 31 01:54:27.813414264 (610h25m27.684497299s) [noise=true , address=145.79.227.2:4117 , piece_id=4SGFGB3IJ7LTXDMMWM2PGGOV64MC46IZ7OZQWH55J4NBQY7CHMZQ, seg_pos={0 0}, upload=534952]: Long tail cancellation due to optimal threshold Dec 31 01:54:27.813424614 (610h25m27.684507649s) [noise=true , address=77.90.31.19:4848 , piece_id=VZYG32S6ODIPDKQYZRXECBHXOERZDDWD2OF35Z46W4XMUZ3OJLAA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD): protocol: write tcp 188.166.225.170:42842->77.90.31.19:4848: use of closed network connection; write tcp 188.166.225.170:42842->77.90.31.19:4848: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42842->77.90.31.19:4848: use of closed network connection Dec 31 01:54:27.813432750 (610h25m27.684515785s) [noise=true , address=77.90.31.19:4848 , piece_id=VZYG32S6ODIPDKQYZRXECBHXOERZDDWD2OF35Z46W4XMUZ3OJLAA, seg_pos={0 0}, upload=534952]: Long tail cancellation due to optimal threshold Dec 31 01:54:27.813474088 (610h25m27.684557123s) [seg_pos={0 0}, upload=534952]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 01:54:27.813535325 (610h25m27.68461836s) [upload=534952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 01:54:27.813539613 (610h25m27.684622648s) [upload=534952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 01:54:28.099921034 (610h25m27.971004069s) [upload=534952]: Done uploading segments... Dec 31 01:54:28.099940451 (610h25m27.971023486s) [upload=534952]: Upload finished. err: <nil> Duration: 34.07912995s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [........X............X..X.......X.X..X....XX.XX........X.X..] duration: 6.068119181s trace: https://jaeger.dev.storj.io/trace/3b4011ac5a42e158 FAIL (2025-12-31 02:24:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: LXE52NV3QIBRTZK27IRWWLAZ3OU2IUHB25H7V6L7XETJSR7ZN67A) context deadline exceeded; piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: OWH5GGYCNXNDSZVDKNMKLXSMO25MW4I4TAJPZJLFKMBQKUSSPG3Q) context deadline exceeded ======LOGS====== 31 02:22:12.475370476 (610h53m12.346453511s) [upload=535351]: Starting upload Dec 31 02:22:12.475429659 (610h53m12.346512694s) [upload=535351]: Done starting upload Dec 31 02:22:12.475607157 (610h53m12.346690202s) [upload=535351]: Uploading segments... Dec 31 02:22:12.475783423 (610h53m12.346866468s) [upload=535351]: Got next segment. Inline: false Dec 31 02:22:12.475838657 (610h53m12.346921702s) [upload=535351]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 02:22:12.475845551 (610h53m12.346928596s) [upload=535351]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 02:22:12.724117301 (610h53m12.595200346s) [seg_pos={0 0}, upload=535351]: Begin upload segment... Dec 31 02:22:12.724171443 (610h53m12.595254488s) [seg_pos={0 0}, upload=535351]: Done begin upload segment. Dec 31 02:22:12.724222331 (610h53m12.595305376s) [noise=true , address=145.79.227.52:4447 , piece_id=AOEU3DUQFDXJAIRV2U7V72IXUCTVVTDQ3O4SHYDRUZ6BNCTIHGIQ, seg_pos={0 0}, upload=535351]: Uploading piece... Dec 31 02:22:12.724287495 (610h53m12.59537054s) [noise=true , address=145.79.227.37:4565 , piece_id=XSVPD47ZDWNMFOJJXGBGUWNN3I4H5MXJHIMGSRSWHRN5OHK4EIGQ, seg_pos={0 0}, upload=535351]: Uploading piece... Dec 31 02:22:12.724817614 (610h53m12.595900659s) [noise=true , address=45.91.116.4:4939 , piece_id=RIQWQXXNI7SCIR7TJXKETDANJ7CLQNRSIV5RS67F4RJYL66XFTSA, seg_pos={0 0}, upload=535351]: Uploading piece... Dec 31 02:22:12.725168582 (610h53m12.596251627s) [noise=true , address=145.79.227.23:4547 , piece_id=JF4GEBAKZHAAFTR6WG7AGQ33LRL22ZTMK5EDPUZQKUWLAONLFGHA, seg_pos={0 0}, upload=535351]: Uploading piece... Dec 31 02:22:12.725489473 (610h53m12.596572508s) [noise=true , address=145.79.227.24:4581 , piece_id=LS6IDZDA6IQBOXLUQBBFUAWKYA2USQV7NNU52662OZXQLINGLGMQ, seg_pos={0 0}, upload=535351]: Uploading piece... Dec 31 02:22:12.725848035 (610h53m12.59693107s) [noise=true , address=45.91.116.8:4045 , piece_id=LXE52NV3QIBRTZK27IRWWLAZ3OU2IUHB25H7V6L7XETJSR7ZN67A, seg_pos={0 0}, upload=535351]: Uploading piece... Dec 31 02:22:12.726610608 (610h53m12.597693643s) [noise=true , address=45.91.116.7:4011 , piece_id=OWH5GGYCNXNDSZVDKNMKLXSMO25MW4I4TAJPZJLFKMBQKUSSPG3Q, seg_pos={0 0}, upload=535351]: Uploading piece... Dec 31 02:22:12.727067127 (610h53m12.598150172s) [noise=true , address=145.79.227.11:4732 , piece_id=WJTPQC5HV7CLFYOGUNNQED2LS7347DBJY7AMUMW4MJWRDV4AODUA, seg_pos={0 0}, upload=535351]: Uploading piece... Dec 31 02:22:12.727529488 (610h53m12.598612523s) [noise=true , address=145.79.227.13:4803 , piece_id=F546YDIU4YMWOPRTFXKH7YYFFFFRXDKLL5N434GL6TZ6GITFE2NA, seg_pos={0 0}, upload=535351]: Uploading piece... Dec 31 02:22:12.728202930 (610h53m12.599285975s) [noise=true , address=145.79.227.31:4798 , piece_id=XHAR65JKVRG3ZGFKFRBTBOF6ZY5AKHTUPDI5DRXAM6ZYY3CJWEJA, seg_pos={0 0}, upload=535351]: Uploading piece... Dec 31 02:22:12.729252208 (610h53m12.600335253s) [noise=true , address=145.79.227.21:4474 , piece_id=VQKDL2DP37I3XY7BG7NDXAF7FDSMUVDRGQ3FU23EVTFFCHAT4L6A, seg_pos={0 0}, upload=535351]: Uploading piece... Dec 31 02:22:12.729402474 (610h53m12.600485509s) [noise=true , address=77.90.31.11:4555 , piece_id=VMAFYHAEN5YKSBAN33PS7FUSNBDQ3DBO5JG7KRBNJWBNW7M5LCYA, seg_pos={0 0}, upload=535351]: Uploading piece... Dec 31 02:22:12.730436854 (610h53m12.601519889s) [noise=true , address=145.79.227.6:4258 , piece_id=ONT4R23H4HUNEPFXYI7XCUL55STXVZH3P5QGQKCDI3B7PEZCKFPQ, seg_pos={0 0}, upload=535351]: Uploading piece... Dec 31 02:22:12.730941896 (610h53m12.602024931s) [noise=true , address=145.79.227.3:4151 , piece_id=466E5IHSVX2ZQ3UPTTLSHH4EUPZJ4H3PUY2XTZHYOYIFYBMFGGHA, seg_pos={0 0}, upload=535351]: Uploading piece... Dec 31 02:22:12.731601811 (610h53m12.602684846s) [noise=true , address=77.90.31.14:4661 , piece_id=5MI4JBBRHTC5MR57TUBEHNZVX6OD7AODOPSBCYH4TSYUPUYDRGNQ, seg_pos={0 0}, upload=535351]: Uploading piece... Dec 31 02:22:12.732236310 (610h53m12.603319355s) [noise=true , address=145.79.227.9:4367 , piece_id=5FD3BXSD5MB3N4C76AQ7P6CM4YZ2WUCJAV4TMFCOXTCTCHSMNIRQ, seg_pos={0 0}, upload=535351]: Uploading piece... Dec 31 02:22:14.430774662 (610h53m14.301857707s) [upload=535351]: Next returned nil segment Dec 31 02:22:14.430792586 (610h53m14.301875631s) [upload=535351]: Waiting for error group managing segments... Dec 31 02:22:14.711554761 (610h53m14.582637806s) [noise=true , address=145.79.227.21:4474 , piece_id=VQKDL2DP37I3XY7BG7NDXAF7FDSMUVDRGQ3FU23EVTFFCHAT4L6A, seg_pos={0 0}, upload=535351]: Done uploading piece. err: <nil> Dec 31 02:22:14.714690121 (610h53m14.585773166s) [noise=true , address=145.79.227.9:4367 , piece_id=5FD3BXSD5MB3N4C76AQ7P6CM4YZ2WUCJAV4TMFCOXTCTCHSMNIRQ, seg_pos={0 0}, upload=535351]: Done uploading piece. err: <nil> Dec 31 02:22:14.718092480 (610h53m14.589175525s) [noise=true , address=145.79.227.6:4258 , piece_id=ONT4R23H4HUNEPFXYI7XCUL55STXVZH3P5QGQKCDI3B7PEZCKFPQ, seg_pos={0 0}, upload=535351]: Done uploading piece. err: <nil> Dec 31 02:22:14.719657861 (610h53m14.590740906s) [noise=true , address=145.79.227.3:4151 , piece_id=466E5IHSVX2ZQ3UPTTLSHH4EUPZJ4H3PUY2XTZHYOYIFYBMFGGHA, seg_pos={0 0}, upload=535351]: Done uploading piece. err: <nil> Dec 31 02:22:14.728292722 (610h53m14.599375757s) [noise=true , address=145.79.227.13:4803 , piece_id=F546YDIU4YMWOPRTFXKH7YYFFFFRXDKLL5N434GL6TZ6GITFE2NA, seg_pos={0 0}, upload=535351]: Done uploading piece. err: <nil> Dec 31 02:22:14.728532268 (610h53m14.599615314s) [noise=true , address=145.79.227.37:4565 , piece_id=XSVPD47ZDWNMFOJJXGBGUWNN3I4H5MXJHIMGSRSWHRN5OHK4EIGQ, seg_pos={0 0}, upload=535351]: Done uploading piece. err: <nil> Dec 31 02:22:14.734847814 (610h53m14.605930859s) [noise=true , address=145.79.227.52:4447 , piece_id=AOEU3DUQFDXJAIRV2U7V72IXUCTVVTDQ3O4SHYDRUZ6BNCTIHGIQ, seg_pos={0 0}, upload=535351]: Done uploading piece. err: <nil> Dec 31 02:22:14.741499390 (610h53m14.612582425s) [noise=true , address=145.79.227.11:4732 , piece_id=WJTPQC5HV7CLFYOGUNNQED2LS7347DBJY7AMUMW4MJWRDV4AODUA, seg_pos={0 0}, upload=535351]: Done uploading piece. err: <nil> Dec 31 02:22:14.744680406 (610h53m14.615763452s) [noise=true , address=145.79.227.23:4547 , piece_id=JF4GEBAKZHAAFTR6WG7AGQ33LRL22ZTMK5EDPUZQKUWLAONLFGHA, seg_pos={0 0}, upload=535351]: Done uploading piece. err: <nil> Dec 31 02:22:14.799469321 (610h53m14.670552356s) [noise=true , address=145.79.227.31:4798 , piece_id=XHAR65JKVRG3ZGFKFRBTBOF6ZY5AKHTUPDI5DRXAM6ZYY3CJWEJA, seg_pos={0 0}, upload=535351]: Done uploading piece. err: <nil> Dec 31 02:22:14.799917525 (610h53m14.67100057s) [noise=true , address=145.79.227.24:4581 , piece_id=LS6IDZDA6IQBOXLUQBBFUAWKYA2USQV7NNU52662OZXQLINGLGMQ, seg_pos={0 0}, upload=535351]: Done uploading piece. err: <nil> Dec 31 02:22:15.084260048 (610h53m14.955343103s) [noise=true , address=45.91.116.8:4045 , piece_id=LXE52NV3QIBRTZK27IRWWLAZ3OU2IUHB25H7V6L7XETJSR7ZN67A, seg_pos={0 0}, upload=535351]: Done uploading piece. err: <nil> Dec 31 02:22:15.144344775 (610h53m15.01542781s) [noise=true , address=45.91.116.7:4011 , piece_id=OWH5GGYCNXNDSZVDKNMKLXSMO25MW4I4TAJPZJLFKMBQKUSSPG3Q, seg_pos={0 0}, upload=535351]: Done uploading piece. err: <nil> Dec 31 02:22:15.144367880 (610h53m15.015450915s) [seg_pos={0 0}, upload=535351]: Segment reached optimal threshold of 13 pieces. Dec 31 02:22:15.144589732 (610h53m15.015672767s) [noise=true , address=45.91.116.4:4939 , piece_id=RIQWQXXNI7SCIR7TJXKETDANJ7CLQNRSIV5RS67F4RJYL66XFTSA, seg_pos={0 0}, upload=535351]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 31 02:22:15.144597046 (610h53m15.015680082s) [noise=true , address=45.91.116.4:4939 , piece_id=RIQWQXXNI7SCIR7TJXKETDANJ7CLQNRSIV5RS67F4RJYL66XFTSA, seg_pos={0 0}, upload=535351]: Long tail cancellation due to optimal threshold Dec 31 02:22:15.144618166 (610h53m15.015701201s) [noise=true , address=77.90.31.14:4661 , piece_id=5MI4JBBRHTC5MR57TUBEHNZVX6OD7AODOPSBCYH4TSYUPUYDRGNQ, seg_pos={0 0}, upload=535351]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Dec 31 02:22:15.144626191 (610h53m15.015709226s) [noise=true , address=77.90.31.14:4661 , piece_id=5MI4JBBRHTC5MR57TUBEHNZVX6OD7AODOPSBCYH4TSYUPUYDRGNQ, seg_pos={0 0}, upload=535351]: Long tail cancellation due to optimal threshold Dec 31 02:22:15.144800363 (610h53m15.015883398s) [noise=true , address=77.90.31.11:4555 , piece_id=VMAFYHAEN5YKSBAN33PS7FUSNBDQ3DBO5JG7KRBNJWBNW7M5LCYA, seg_pos={0 0}, upload=535351]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: write tcp 188.166.225.170:50294->77.90.31.11:4555: use of closed network connection; write tcp 188.166.225.170:50294->77.90.31.11:4555: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50294->77.90.31.11:4555: use of closed network connection Dec 31 02:22:15.144807157 (610h53m15.015890192s) [noise=true , address=77.90.31.11:4555 , piece_id=VMAFYHAEN5YKSBAN33PS7FUSNBDQ3DBO5JG7KRBNJWBNW7M5LCYA, seg_pos={0 0}, upload=535351]: Long tail cancellation due to optimal threshold Dec 31 02:22:15.144821844 (610h53m15.015904889s) [seg_pos={0 0}, upload=535351]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 02:22:15.144898269 (610h53m15.015981304s) [upload=535351]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 02:22:15.144903008 (610h53m15.015986043s) [upload=535351]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 02:22:15.439148899 (610h53m15.310231934s) [upload=535351]: Done uploading segments... Dec 31 02:22:15.439170178 (610h53m15.310253214s) [upload=535351]: Upload finished. err: <nil> Duration: 6.068148207s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [......X..................................X..................] duration: 3.895649185s trace: https://jaeger.dev.storj.io/trace/740aa405bd68150f FAIL (2025-12-31 02:24:18 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.895671137s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [X..........................X................................] duration: 50.685341455s trace: https://jaeger.dev.storj.io/trace/365693fea00df28 FAIL (2025-12-31 02:24:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [noise=true , address=145.79.227.49:4343 , piece_id=6L3HPTS7D2NRA4QEOH4RFM3D3B53CTCONRI6G6YVU4OOTUWTG7XA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.942060847 (610h25m24.813143892s) [noise=true , address=145.79.227.34:4937 , piece_id=5W7LBM3YJJTVUFYTELR57DVXB663OHO5N3Z2SPQPZ4G43IK47O3Q, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.942684104 (610h25m24.813767139s) [noise=true , address=64.49.13.139:4478 , piece_id=QN6RT2KWEA6APQVA66ZMKG4LH537UGYNOXOCVVA2YC2MKDOLT3YA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.943064007 (610h25m24.814147042s) [noise=true , address=145.79.227.40:4084 , piece_id=RUKTKDII6BRUWSYVS65CWUBT3QKG3ASUK3KA3O5VGE53XSG6IMOQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.943459560 (610h25m24.814542605s) [noise=true , address=145.79.227.14:4847 , piece_id=JX5NXTMVM3NSWIEVR4RX46ABSNPKTOFDLFBPKEXFQ3G3ANKV5OLA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.943865303 (610h25m24.814948348s) [noise=true , address=45.91.116.6:4981 , piece_id=CT474ZA2LVAZEBJHBDD4GB6NTHVHZFJLNE5OZV3DFS2NZKVW3EQQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.944259694 (610h25m24.815342749s) [noise=true , address=145.79.227.51:4443 , piece_id=73UUMRDY7BSZNCGBXCZYYRBVE5KLPMNPRNXOZEGWKQGLHJS32J2Q, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.946575182 (610h25m24.817658227s) [noise=true , address=77.90.31.10:4523 , piece_id=P6LIONXJI6AFZ2JTKCQ2W6RZCOZCK6MZ53EXAJKXYQXSX2PX4QEA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.948132909 (610h25m24.819215944s) [noise=true , address=145.79.227.20:4452 , piece_id=ZSDN2JZ3FQUVJYN5OURDXNKEDLDNULJOHKKRZVIPXZIWIYAHCRXQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.949605774 (610h25m24.820688819s) [noise=true , address=77.90.31.14:4680 , piece_id=S2E4WBA7MJVAZLT5JG3IMHK53RQC77CCHV2PDF6L2R2JEA2U3K4Q, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.951459002 (610h25m24.822542048s) [noise=true , address=193.149.164.143:4704 , piece_id=BHXHFBKTLTDMGXWY5QRTP2THZQE5OXFZ2LCMTO2KRYRWAHXWW3NQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.952986742 (610h25m24.824069777s) [noise=true , address=77.90.31.19:4848 , piece_id=VZYG32S6ODIPDKQYZRXECBHXOERZDDWD2OF35Z46W4XMUZ3OJLAA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.956916654 (610h25m24.827999689s) [noise=true , address=145.79.227.29:4971 , piece_id=MEPXHHOQ6SUZC3GIPCQEMOKKF5OJTKZEI3UYZBAYXAQZGSVEKSBA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.960665062 (610h25m24.831748097s) [noise=true , address=64.49.13.170:4720 , piece_id=HIATRSAPE4YLSODFZDPP43ZUMLDVM7GJEVMWE4ZGUQTP25LTHGTA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.962366001 (610h25m24.833449036s) [noise=true , address=145.79.227.2:4117 , piece_id=4SGFGB3IJ7LTXDMMWM2PGGOV64MC46IZ7OZQWH55J4NBQY7CHMZQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.963708216 (610h25m24.834791261s) [noise=true , address=145.79.227.35:4505 , piece_id=V7RS7A4QLZTH4G32ZUZNSF2Z5APVWZGJO77NEGP3SBWQT3WCMVLQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:26.352236138 (610h25m26.223319173s) [upload=534952]: Next returned nil segment Dec 31 01:54:26.352253821 (610h25m26.223336857s) [upload=534952]: Waiting for error group managing segments... Dec 31 01:54:26.812758374 (610h25m26.683841409s) [noise=true , address=145.79.227.51:4443 , piece_id=73UUMRDY7BSZNCGBXCZYYRBVE5KLPMNPRNXOZEGWKQGLHJS32J2Q, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.824371056 (610h25m26.695454101s) [noise=true , address=145.79.227.14:4847 , piece_id=JX5NXTMVM3NSWIEVR4RX46ABSNPKTOFDLFBPKEXFQ3G3ANKV5OLA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.838414046 (610h25m26.709497081s) [noise=true , address=64.49.13.138:4490 , piece_id=2XTUZE6PXRE3OARNJVT36FXXWUMYIBVNE6M2VVFAVBGKEE2H5YZQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.875499841 (610h25m26.746582876s) [noise=true , address=64.49.13.139:4478 , piece_id=QN6RT2KWEA6APQVA66ZMKG4LH537UGYNOXOCVVA2YC2MKDOLT3YA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.891607221 (610h25m26.762690256s) [noise=true , address=145.79.227.20:4452 , piece_id=ZSDN2JZ3FQUVJYN5OURDXNKEDLDNULJOHKKRZVIPXZIWIYAHCRXQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.988561223 (610h25m26.859644268s) [noise=true , address=64.49.13.170:4720 , piece_id=HIATRSAPE4YLSODFZDPP43ZUMLDVM7GJEVMWE4ZGUQTP25LTHGTA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.039035911 (610h25m26.910118946s) [noise=true , address=77.90.31.14:4680 , piece_id=S2E4WBA7MJVAZLT5JG3IMHK53RQC77CCHV2PDF6L2R2JEA2U3K4Q, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.098023090 (610h25m26.969106126s) [noise=true , address=45.91.116.6:4981 , piece_id=CT474ZA2LVAZEBJHBDD4GB6NTHVHZFJLNE5OZV3DFS2NZKVW3EQQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.653790754 (610h25m27.524873789s) [noise=true , address=77.90.31.21:4886 , piece_id=K34WPIBGGGEPHIKJ253XGLDCIQ3WLFN4AZ6EYNWVHHT2EIHS6YRQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.684192972 (610h25m27.555276017s) [noise=true , address=193.149.164.143:4704 , piece_id=BHXHFBKTLTDMGXWY5QRTP2THZQE5OXFZ2LCMTO2KRYRWAHXWW3NQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.717238425 (610h25m27.58832146s) [noise=true , address=145.79.227.40:4084 , piece_id=RUKTKDII6BRUWSYVS65CWUBT3QKG3ASUK3KA3O5VGE53XSG6IMOQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.718349742 (610h25m27.589432777s) [noise=true , address=77.90.31.10:4523 , piece_id=P6LIONXJI6AFZ2JTKCQ2W6RZCOZCK6MZ53EXAJKXYQXSX2PX4QEA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.803639562 (610h25m27.674722597s) [noise=true , address=145.79.227.29:4971 , piece_id=MEPXHHOQ6SUZC3GIPCQEMOKKF5OJTKZEI3UYZBAYXAQZGSVEKSBA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.812791279 (610h25m27.683874314s) [noise=true , address=145.79.227.49:4343 , piece_id=6L3HPTS7D2NRA4QEOH4RFM3D3B53CTCONRI6G6YVU4OOTUWTG7XA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.812812348 (610h25m27.683895393s) [seg_pos={0 0}, upload=534952]: Segment reached optimal threshold of 14 pieces. Dec 31 01:54:27.813018000 (610h25m27.684101035s) [noise=true , address=145.79.227.35:4505 , piece_id=V7RS7A4QLZTH4G32ZUZNSF2Z5APVWZGJO77NEGP3SBWQT3WCMVLQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i6itGoBhLsLiqC8et8Y5L62fLjcWxy18u39NAr4R3nXNMWcsf): protocol: expected piece hash; context canceled; EOF Dec 31 01:54:27.813024772 (610h25m27.684107808s) [noise=true , address=145.79.227.35:4505 , piece_id=V7RS7A4QLZTH4G32ZUZNSF2Z5APVWZGJO77NEGP3SBWQT3WCMVLQ, seg_pos={0 0}, upload=534952]: Long tail cancellation due to optimal threshold Dec 31 01:54:27.813175931 (610h25m27.684258976s) [noise=true , address=145.79.227.34:4937 , piece_id=5W7LBM3YJJTVUFYTELR57DVXB663OHO5N3Z2SPQPZ4G43IK47O3Q, seg_pos={0 0}, upload=534952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1LKrFThdW3jJpG6dRZLUaxLXTa5Wy27yQ8PGUEqjvBWfFoRB94): protocol: expected piece hash; context canceled; EOF Dec 31 01:54:27.813181702 (610h25m27.684264737s) [noise=true , address=145.79.227.34:4937 , piece_id=5W7LBM3YJJTVUFYTELR57DVXB663OHO5N3Z2SPQPZ4G43IK47O3Q, seg_pos={0 0}, upload=534952]: Long tail cancellation due to optimal threshold Dec 31 01:54:27.813407862 (610h25m27.684490907s) [noise=true , address=145.79.227.2:4117 , piece_id=4SGFGB3IJ7LTXDMMWM2PGGOV64MC46IZ7OZQWH55J4NBQY7CHMZQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122Htg72f5eUUutf1p3fYQpy1Wkf6AgdaQV83HHYye2n8zm3the): protocol: expected piece hash; context canceled; EOF Dec 31 01:54:27.813414264 (610h25m27.684497299s) [noise=true , address=145.79.227.2:4117 , piece_id=4SGFGB3IJ7LTXDMMWM2PGGOV64MC46IZ7OZQWH55J4NBQY7CHMZQ, seg_pos={0 0}, upload=534952]: Long tail cancellation due to optimal threshold Dec 31 01:54:27.813424614 (610h25m27.684507649s) [noise=true , address=77.90.31.19:4848 , piece_id=VZYG32S6ODIPDKQYZRXECBHXOERZDDWD2OF35Z46W4XMUZ3OJLAA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD): protocol: write tcp 188.166.225.170:42842->77.90.31.19:4848: use of closed network connection; write tcp 188.166.225.170:42842->77.90.31.19:4848: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42842->77.90.31.19:4848: use of closed network connection Dec 31 01:54:27.813432750 (610h25m27.684515785s) [noise=true , address=77.90.31.19:4848 , piece_id=VZYG32S6ODIPDKQYZRXECBHXOERZDDWD2OF35Z46W4XMUZ3OJLAA, seg_pos={0 0}, upload=534952]: Long tail cancellation due to optimal threshold Dec 31 01:54:27.813474088 (610h25m27.684557123s) [seg_pos={0 0}, upload=534952]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 01:54:27.813535325 (610h25m27.68461836s) [upload=534952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 01:54:27.813539613 (610h25m27.684622648s) [upload=534952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 01:54:28.099921034 (610h25m27.971004069s) [upload=534952]: Done uploading segments... Dec 31 01:54:28.099940451 (610h25m27.971023486s) [upload=534952]: Upload finished. err: <nil> Duration: 50.685366123s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.........X............X..X.......X.X..X....XX.XX........X.X.] duration: 24.975366633s trace: https://jaeger.dev.storj.io/trace/3b4011ac5a42e158 FAIL (2025-12-31 02:23:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: LXE52NV3QIBRTZK27IRWWLAZ3OU2IUHB25H7V6L7XETJSR7ZN67A) context deadline exceeded; piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: OWH5GGYCNXNDSZVDKNMKLXSMO25MW4I4TAJPZJLFKMBQKUSSPG3Q) context deadline exceeded ======LOGS====== 31 02:22:12.475370476 (610h53m12.346453511s) [upload=535351]: Starting upload Dec 31 02:22:12.475429659 (610h53m12.346512694s) [upload=535351]: Done starting upload Dec 31 02:22:12.475607157 (610h53m12.346690202s) [upload=535351]: Uploading segments... Dec 31 02:22:12.475783423 (610h53m12.346866468s) [upload=535351]: Got next segment. Inline: false Dec 31 02:22:12.475838657 (610h53m12.346921702s) [upload=535351]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 02:22:12.475845551 (610h53m12.346928596s) [upload=535351]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 02:22:12.724117301 (610h53m12.595200346s) [seg_pos={0 0}, upload=535351]: Begin upload segment... Dec 31 02:22:12.724171443 (610h53m12.595254488s) [seg_pos={0 0}, upload=535351]: Done begin upload segment. Dec 31 02:22:12.724222331 (610h53m12.595305376s) [noise=true , address=145.79.227.52:4447 , piece_id=AOEU3DUQFDXJAIRV2U7V72IXUCTVVTDQ3O4SHYDRUZ6BNCTIHGIQ, seg_pos={0 0}, upload=535351]: Uploading piece... Dec 31 02:22:12.724287495 (610h53m12.59537054s) [noise=true , address=145.79.227.37:4565 , piece_id=XSVPD47ZDWNMFOJJXGBGUWNN3I4H5MXJHIMGSRSWHRN5OHK4EIGQ, seg_pos={0 0}, upload=535351]: Uploading piece... Dec 31 02:22:12.724817614 (610h53m12.595900659s) [noise=true , address=45.91.116.4:4939 , piece_id=RIQWQXXNI7SCIR7TJXKETDANJ7CLQNRSIV5RS67F4RJYL66XFTSA, seg_pos={0 0}, upload=535351]: Uploading piece... Dec 31 02:22:12.725168582 (610h53m12.596251627s) [noise=true , address=145.79.227.23:4547 , piece_id=JF4GEBAKZHAAFTR6WG7AGQ33LRL22ZTMK5EDPUZQKUWLAONLFGHA, seg_pos={0 0}, upload=535351]: Uploading piece... Dec 31 02:22:12.725489473 (610h53m12.596572508s) [noise=true , address=145.79.227.24:4581 , piece_id=LS6IDZDA6IQBOXLUQBBFUAWKYA2USQV7NNU52662OZXQLINGLGMQ, seg_pos={0 0}, upload=535351]: Uploading piece... Dec 31 02:22:12.725848035 (610h53m12.59693107s) [noise=true , address=45.91.116.8:4045 , piece_id=LXE52NV3QIBRTZK27IRWWLAZ3OU2IUHB25H7V6L7XETJSR7ZN67A, seg_pos={0 0}, upload=535351]: Uploading piece... Dec 31 02:22:12.726610608 (610h53m12.597693643s) [noise=true , address=45.91.116.7:4011 , piece_id=OWH5GGYCNXNDSZVDKNMKLXSMO25MW4I4TAJPZJLFKMBQKUSSPG3Q, seg_pos={0 0}, upload=535351]: Uploading piece... Dec 31 02:22:12.727067127 (610h53m12.598150172s) [noise=true , address=145.79.227.11:4732 , piece_id=WJTPQC5HV7CLFYOGUNNQED2LS7347DBJY7AMUMW4MJWRDV4AODUA, seg_pos={0 0}, upload=535351]: Uploading piece... Dec 31 02:22:12.727529488 (610h53m12.598612523s) [noise=true , address=145.79.227.13:4803 , piece_id=F546YDIU4YMWOPRTFXKH7YYFFFFRXDKLL5N434GL6TZ6GITFE2NA, seg_pos={0 0}, upload=535351]: Uploading piece... Dec 31 02:22:12.728202930 (610h53m12.599285975s) [noise=true , address=145.79.227.31:4798 , piece_id=XHAR65JKVRG3ZGFKFRBTBOF6ZY5AKHTUPDI5DRXAM6ZYY3CJWEJA, seg_pos={0 0}, upload=535351]: Uploading piece... Dec 31 02:22:12.729252208 (610h53m12.600335253s) [noise=true , address=145.79.227.21:4474 , piece_id=VQKDL2DP37I3XY7BG7NDXAF7FDSMUVDRGQ3FU23EVTFFCHAT4L6A, seg_pos={0 0}, upload=535351]: Uploading piece... Dec 31 02:22:12.729402474 (610h53m12.600485509s) [noise=true , address=77.90.31.11:4555 , piece_id=VMAFYHAEN5YKSBAN33PS7FUSNBDQ3DBO5JG7KRBNJWBNW7M5LCYA, seg_pos={0 0}, upload=535351]: Uploading piece... Dec 31 02:22:12.730436854 (610h53m12.601519889s) [noise=true , address=145.79.227.6:4258 , piece_id=ONT4R23H4HUNEPFXYI7XCUL55STXVZH3P5QGQKCDI3B7PEZCKFPQ, seg_pos={0 0}, upload=535351]: Uploading piece... Dec 31 02:22:12.730941896 (610h53m12.602024931s) [noise=true , address=145.79.227.3:4151 , piece_id=466E5IHSVX2ZQ3UPTTLSHH4EUPZJ4H3PUY2XTZHYOYIFYBMFGGHA, seg_pos={0 0}, upload=535351]: Uploading piece... Dec 31 02:22:12.731601811 (610h53m12.602684846s) [noise=true , address=77.90.31.14:4661 , piece_id=5MI4JBBRHTC5MR57TUBEHNZVX6OD7AODOPSBCYH4TSYUPUYDRGNQ, seg_pos={0 0}, upload=535351]: Uploading piece... Dec 31 02:22:12.732236310 (610h53m12.603319355s) [noise=true , address=145.79.227.9:4367 , piece_id=5FD3BXSD5MB3N4C76AQ7P6CM4YZ2WUCJAV4TMFCOXTCTCHSMNIRQ, seg_pos={0 0}, upload=535351]: Uploading piece... Dec 31 02:22:14.430774662 (610h53m14.301857707s) [upload=535351]: Next returned nil segment Dec 31 02:22:14.430792586 (610h53m14.301875631s) [upload=535351]: Waiting for error group managing segments... Dec 31 02:22:14.711554761 (610h53m14.582637806s) [noise=true , address=145.79.227.21:4474 , piece_id=VQKDL2DP37I3XY7BG7NDXAF7FDSMUVDRGQ3FU23EVTFFCHAT4L6A, seg_pos={0 0}, upload=535351]: Done uploading piece. err: <nil> Dec 31 02:22:14.714690121 (610h53m14.585773166s) [noise=true , address=145.79.227.9:4367 , piece_id=5FD3BXSD5MB3N4C76AQ7P6CM4YZ2WUCJAV4TMFCOXTCTCHSMNIRQ, seg_pos={0 0}, upload=535351]: Done uploading piece. err: <nil> Dec 31 02:22:14.718092480 (610h53m14.589175525s) [noise=true , address=145.79.227.6:4258 , piece_id=ONT4R23H4HUNEPFXYI7XCUL55STXVZH3P5QGQKCDI3B7PEZCKFPQ, seg_pos={0 0}, upload=535351]: Done uploading piece. err: <nil> Dec 31 02:22:14.719657861 (610h53m14.590740906s) [noise=true , address=145.79.227.3:4151 , piece_id=466E5IHSVX2ZQ3UPTTLSHH4EUPZJ4H3PUY2XTZHYOYIFYBMFGGHA, seg_pos={0 0}, upload=535351]: Done uploading piece. err: <nil> Dec 31 02:22:14.728292722 (610h53m14.599375757s) [noise=true , address=145.79.227.13:4803 , piece_id=F546YDIU4YMWOPRTFXKH7YYFFFFRXDKLL5N434GL6TZ6GITFE2NA, seg_pos={0 0}, upload=535351]: Done uploading piece. err: <nil> Dec 31 02:22:14.728532268 (610h53m14.599615314s) [noise=true , address=145.79.227.37:4565 , piece_id=XSVPD47ZDWNMFOJJXGBGUWNN3I4H5MXJHIMGSRSWHRN5OHK4EIGQ, seg_pos={0 0}, upload=535351]: Done uploading piece. err: <nil> Dec 31 02:22:14.734847814 (610h53m14.605930859s) [noise=true , address=145.79.227.52:4447 , piece_id=AOEU3DUQFDXJAIRV2U7V72IXUCTVVTDQ3O4SHYDRUZ6BNCTIHGIQ, seg_pos={0 0}, upload=535351]: Done uploading piece. err: <nil> Dec 31 02:22:14.741499390 (610h53m14.612582425s) [noise=true , address=145.79.227.11:4732 , piece_id=WJTPQC5HV7CLFYOGUNNQED2LS7347DBJY7AMUMW4MJWRDV4AODUA, seg_pos={0 0}, upload=535351]: Done uploading piece. err: <nil> Dec 31 02:22:14.744680406 (610h53m14.615763452s) [noise=true , address=145.79.227.23:4547 , piece_id=JF4GEBAKZHAAFTR6WG7AGQ33LRL22ZTMK5EDPUZQKUWLAONLFGHA, seg_pos={0 0}, upload=535351]: Done uploading piece. err: <nil> Dec 31 02:22:14.799469321 (610h53m14.670552356s) [noise=true , address=145.79.227.31:4798 , piece_id=XHAR65JKVRG3ZGFKFRBTBOF6ZY5AKHTUPDI5DRXAM6ZYY3CJWEJA, seg_pos={0 0}, upload=535351]: Done uploading piece. err: <nil> Dec 31 02:22:14.799917525 (610h53m14.67100057s) [noise=true , address=145.79.227.24:4581 , piece_id=LS6IDZDA6IQBOXLUQBBFUAWKYA2USQV7NNU52662OZXQLINGLGMQ, seg_pos={0 0}, upload=535351]: Done uploading piece. err: <nil> Dec 31 02:22:15.084260048 (610h53m14.955343103s) [noise=true , address=45.91.116.8:4045 , piece_id=LXE52NV3QIBRTZK27IRWWLAZ3OU2IUHB25H7V6L7XETJSR7ZN67A, seg_pos={0 0}, upload=535351]: Done uploading piece. err: <nil> Dec 31 02:22:15.144344775 (610h53m15.01542781s) [noise=true , address=45.91.116.7:4011 , piece_id=OWH5GGYCNXNDSZVDKNMKLXSMO25MW4I4TAJPZJLFKMBQKUSSPG3Q, seg_pos={0 0}, upload=535351]: Done uploading piece. err: <nil> Dec 31 02:22:15.144367880 (610h53m15.015450915s) [seg_pos={0 0}, upload=535351]: Segment reached optimal threshold of 13 pieces. Dec 31 02:22:15.144589732 (610h53m15.015672767s) [noise=true , address=45.91.116.4:4939 , piece_id=RIQWQXXNI7SCIR7TJXKETDANJ7CLQNRSIV5RS67F4RJYL66XFTSA, seg_pos={0 0}, upload=535351]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 31 02:22:15.144597046 (610h53m15.015680082s) [noise=true , address=45.91.116.4:4939 , piece_id=RIQWQXXNI7SCIR7TJXKETDANJ7CLQNRSIV5RS67F4RJYL66XFTSA, seg_pos={0 0}, upload=535351]: Long tail cancellation due to optimal threshold Dec 31 02:22:15.144618166 (610h53m15.015701201s) [noise=true , address=77.90.31.14:4661 , piece_id=5MI4JBBRHTC5MR57TUBEHNZVX6OD7AODOPSBCYH4TSYUPUYDRGNQ, seg_pos={0 0}, upload=535351]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Dec 31 02:22:15.144626191 (610h53m15.015709226s) [noise=true , address=77.90.31.14:4661 , piece_id=5MI4JBBRHTC5MR57TUBEHNZVX6OD7AODOPSBCYH4TSYUPUYDRGNQ, seg_pos={0 0}, upload=535351]: Long tail cancellation due to optimal threshold Dec 31 02:22:15.144800363 (610h53m15.015883398s) [noise=true , address=77.90.31.11:4555 , piece_id=VMAFYHAEN5YKSBAN33PS7FUSNBDQ3DBO5JG7KRBNJWBNW7M5LCYA, seg_pos={0 0}, upload=535351]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: write tcp 188.166.225.170:50294->77.90.31.11:4555: use of closed network connection; write tcp 188.166.225.170:50294->77.90.31.11:4555: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50294->77.90.31.11:4555: use of closed network connection Dec 31 02:22:15.144807157 (610h53m15.015890192s) [noise=true , address=77.90.31.11:4555 , piece_id=VMAFYHAEN5YKSBAN33PS7FUSNBDQ3DBO5JG7KRBNJWBNW7M5LCYA, seg_pos={0 0}, upload=535351]: Long tail cancellation due to optimal threshold Dec 31 02:22:15.144821844 (610h53m15.015904889s) [seg_pos={0 0}, upload=535351]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 02:22:15.144898269 (610h53m15.015981304s) [upload=535351]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 02:22:15.144903008 (610h53m15.015986043s) [upload=535351]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 02:22:15.439148899 (610h53m15.310231934s) [upload=535351]: Done uploading segments... Dec 31 02:22:15.439170178 (610h53m15.310253214s) [upload=535351]: Upload finished. err: <nil> Duration: 24.975402011s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.......X..................................X.................] duration: 3.551060926s trace: https://jaeger.dev.storj.io/trace/740aa405bd68150f FAIL (2025-12-31 02:23:22 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.551091024s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [..........X............X..X.......X.X..X....XX.XX........X.X] duration: 30.003355784s trace: https://jaeger.dev.storj.io/trace/3b4011ac5a42e158 FAIL (2025-12-31 02:22:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: LXE52NV3QIBRTZK27IRWWLAZ3OU2IUHB25H7V6L7XETJSR7ZN67A) context deadline exceeded; piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: OWH5GGYCNXNDSZVDKNMKLXSMO25MW4I4TAJPZJLFKMBQKUSSPG3Q) context deadline exceeded ======LOGS====== 31 02:22:12.475370476 (610h53m12.346453511s) [upload=535351]: Starting upload Dec 31 02:22:12.475429659 (610h53m12.346512694s) [upload=535351]: Done starting upload Dec 31 02:22:12.475607157 (610h53m12.346690202s) [upload=535351]: Uploading segments... Dec 31 02:22:12.475783423 (610h53m12.346866468s) [upload=535351]: Got next segment. Inline: false Dec 31 02:22:12.475838657 (610h53m12.346921702s) [upload=535351]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 02:22:12.475845551 (610h53m12.346928596s) [upload=535351]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 02:22:12.724117301 (610h53m12.595200346s) [seg_pos={0 0}, upload=535351]: Begin upload segment... Dec 31 02:22:12.724171443 (610h53m12.595254488s) [seg_pos={0 0}, upload=535351]: Done begin upload segment. Dec 31 02:22:12.724222331 (610h53m12.595305376s) [noise=true , address=145.79.227.52:4447 , piece_id=AOEU3DUQFDXJAIRV2U7V72IXUCTVVTDQ3O4SHYDRUZ6BNCTIHGIQ, seg_pos={0 0}, upload=535351]: Uploading piece... Dec 31 02:22:12.724287495 (610h53m12.59537054s) [noise=true , address=145.79.227.37:4565 , piece_id=XSVPD47ZDWNMFOJJXGBGUWNN3I4H5MXJHIMGSRSWHRN5OHK4EIGQ, seg_pos={0 0}, upload=535351]: Uploading piece... Dec 31 02:22:12.724817614 (610h53m12.595900659s) [noise=true , address=45.91.116.4:4939 , piece_id=RIQWQXXNI7SCIR7TJXKETDANJ7CLQNRSIV5RS67F4RJYL66XFTSA, seg_pos={0 0}, upload=535351]: Uploading piece... Dec 31 02:22:12.725168582 (610h53m12.596251627s) [noise=true , address=145.79.227.23:4547 , piece_id=JF4GEBAKZHAAFTR6WG7AGQ33LRL22ZTMK5EDPUZQKUWLAONLFGHA, seg_pos={0 0}, upload=535351]: Uploading piece... Dec 31 02:22:12.725489473 (610h53m12.596572508s) [noise=true , address=145.79.227.24:4581 , piece_id=LS6IDZDA6IQBOXLUQBBFUAWKYA2USQV7NNU52662OZXQLINGLGMQ, seg_pos={0 0}, upload=535351]: Uploading piece... Dec 31 02:22:12.725848035 (610h53m12.59693107s) [noise=true , address=45.91.116.8:4045 , piece_id=LXE52NV3QIBRTZK27IRWWLAZ3OU2IUHB25H7V6L7XETJSR7ZN67A, seg_pos={0 0}, upload=535351]: Uploading piece... Dec 31 02:22:12.726610608 (610h53m12.597693643s) [noise=true , address=45.91.116.7:4011 , piece_id=OWH5GGYCNXNDSZVDKNMKLXSMO25MW4I4TAJPZJLFKMBQKUSSPG3Q, seg_pos={0 0}, upload=535351]: Uploading piece... Dec 31 02:22:12.727067127 (610h53m12.598150172s) [noise=true , address=145.79.227.11:4732 , piece_id=WJTPQC5HV7CLFYOGUNNQED2LS7347DBJY7AMUMW4MJWRDV4AODUA, seg_pos={0 0}, upload=535351]: Uploading piece... Dec 31 02:22:12.727529488 (610h53m12.598612523s) [noise=true , address=145.79.227.13:4803 , piece_id=F546YDIU4YMWOPRTFXKH7YYFFFFRXDKLL5N434GL6TZ6GITFE2NA, seg_pos={0 0}, upload=535351]: Uploading piece... Dec 31 02:22:12.728202930 (610h53m12.599285975s) [noise=true , address=145.79.227.31:4798 , piece_id=XHAR65JKVRG3ZGFKFRBTBOF6ZY5AKHTUPDI5DRXAM6ZYY3CJWEJA, seg_pos={0 0}, upload=535351]: Uploading piece... Dec 31 02:22:12.729252208 (610h53m12.600335253s) [noise=true , address=145.79.227.21:4474 , piece_id=VQKDL2DP37I3XY7BG7NDXAF7FDSMUVDRGQ3FU23EVTFFCHAT4L6A, seg_pos={0 0}, upload=535351]: Uploading piece... Dec 31 02:22:12.729402474 (610h53m12.600485509s) [noise=true , address=77.90.31.11:4555 , piece_id=VMAFYHAEN5YKSBAN33PS7FUSNBDQ3DBO5JG7KRBNJWBNW7M5LCYA, seg_pos={0 0}, upload=535351]: Uploading piece... Dec 31 02:22:12.730436854 (610h53m12.601519889s) [noise=true , address=145.79.227.6:4258 , piece_id=ONT4R23H4HUNEPFXYI7XCUL55STXVZH3P5QGQKCDI3B7PEZCKFPQ, seg_pos={0 0}, upload=535351]: Uploading piece... Dec 31 02:22:12.730941896 (610h53m12.602024931s) [noise=true , address=145.79.227.3:4151 , piece_id=466E5IHSVX2ZQ3UPTTLSHH4EUPZJ4H3PUY2XTZHYOYIFYBMFGGHA, seg_pos={0 0}, upload=535351]: Uploading piece... Dec 31 02:22:12.731601811 (610h53m12.602684846s) [noise=true , address=77.90.31.14:4661 , piece_id=5MI4JBBRHTC5MR57TUBEHNZVX6OD7AODOPSBCYH4TSYUPUYDRGNQ, seg_pos={0 0}, upload=535351]: Uploading piece... Dec 31 02:22:12.732236310 (610h53m12.603319355s) [noise=true , address=145.79.227.9:4367 , piece_id=5FD3BXSD5MB3N4C76AQ7P6CM4YZ2WUCJAV4TMFCOXTCTCHSMNIRQ, seg_pos={0 0}, upload=535351]: Uploading piece... Dec 31 02:22:14.430774662 (610h53m14.301857707s) [upload=535351]: Next returned nil segment Dec 31 02:22:14.430792586 (610h53m14.301875631s) [upload=535351]: Waiting for error group managing segments... Dec 31 02:22:14.711554761 (610h53m14.582637806s) [noise=true , address=145.79.227.21:4474 , piece_id=VQKDL2DP37I3XY7BG7NDXAF7FDSMUVDRGQ3FU23EVTFFCHAT4L6A, seg_pos={0 0}, upload=535351]: Done uploading piece. err: <nil> Dec 31 02:22:14.714690121 (610h53m14.585773166s) [noise=true , address=145.79.227.9:4367 , piece_id=5FD3BXSD5MB3N4C76AQ7P6CM4YZ2WUCJAV4TMFCOXTCTCHSMNIRQ, seg_pos={0 0}, upload=535351]: Done uploading piece. err: <nil> Dec 31 02:22:14.718092480 (610h53m14.589175525s) [noise=true , address=145.79.227.6:4258 , piece_id=ONT4R23H4HUNEPFXYI7XCUL55STXVZH3P5QGQKCDI3B7PEZCKFPQ, seg_pos={0 0}, upload=535351]: Done uploading piece. err: <nil> Dec 31 02:22:14.719657861 (610h53m14.590740906s) [noise=true , address=145.79.227.3:4151 , piece_id=466E5IHSVX2ZQ3UPTTLSHH4EUPZJ4H3PUY2XTZHYOYIFYBMFGGHA, seg_pos={0 0}, upload=535351]: Done uploading piece. err: <nil> Dec 31 02:22:14.728292722 (610h53m14.599375757s) [noise=true , address=145.79.227.13:4803 , piece_id=F546YDIU4YMWOPRTFXKH7YYFFFFRXDKLL5N434GL6TZ6GITFE2NA, seg_pos={0 0}, upload=535351]: Done uploading piece. err: <nil> Dec 31 02:22:14.728532268 (610h53m14.599615314s) [noise=true , address=145.79.227.37:4565 , piece_id=XSVPD47ZDWNMFOJJXGBGUWNN3I4H5MXJHIMGSRSWHRN5OHK4EIGQ, seg_pos={0 0}, upload=535351]: Done uploading piece. err: <nil> Dec 31 02:22:14.734847814 (610h53m14.605930859s) [noise=true , address=145.79.227.52:4447 , piece_id=AOEU3DUQFDXJAIRV2U7V72IXUCTVVTDQ3O4SHYDRUZ6BNCTIHGIQ, seg_pos={0 0}, upload=535351]: Done uploading piece. err: <nil> Dec 31 02:22:14.741499390 (610h53m14.612582425s) [noise=true , address=145.79.227.11:4732 , piece_id=WJTPQC5HV7CLFYOGUNNQED2LS7347DBJY7AMUMW4MJWRDV4AODUA, seg_pos={0 0}, upload=535351]: Done uploading piece. err: <nil> Dec 31 02:22:14.744680406 (610h53m14.615763452s) [noise=true , address=145.79.227.23:4547 , piece_id=JF4GEBAKZHAAFTR6WG7AGQ33LRL22ZTMK5EDPUZQKUWLAONLFGHA, seg_pos={0 0}, upload=535351]: Done uploading piece. err: <nil> Dec 31 02:22:14.799469321 (610h53m14.670552356s) [noise=true , address=145.79.227.31:4798 , piece_id=XHAR65JKVRG3ZGFKFRBTBOF6ZY5AKHTUPDI5DRXAM6ZYY3CJWEJA, seg_pos={0 0}, upload=535351]: Done uploading piece. err: <nil> Dec 31 02:22:14.799917525 (610h53m14.67100057s) [noise=true , address=145.79.227.24:4581 , piece_id=LS6IDZDA6IQBOXLUQBBFUAWKYA2USQV7NNU52662OZXQLINGLGMQ, seg_pos={0 0}, upload=535351]: Done uploading piece. err: <nil> Dec 31 02:22:15.084260048 (610h53m14.955343103s) [noise=true , address=45.91.116.8:4045 , piece_id=LXE52NV3QIBRTZK27IRWWLAZ3OU2IUHB25H7V6L7XETJSR7ZN67A, seg_pos={0 0}, upload=535351]: Done uploading piece. err: <nil> Dec 31 02:22:15.144344775 (610h53m15.01542781s) [noise=true , address=45.91.116.7:4011 , piece_id=OWH5GGYCNXNDSZVDKNMKLXSMO25MW4I4TAJPZJLFKMBQKUSSPG3Q, seg_pos={0 0}, upload=535351]: Done uploading piece. err: <nil> Dec 31 02:22:15.144367880 (610h53m15.015450915s) [seg_pos={0 0}, upload=535351]: Segment reached optimal threshold of 13 pieces. Dec 31 02:22:15.144589732 (610h53m15.015672767s) [noise=true , address=45.91.116.4:4939 , piece_id=RIQWQXXNI7SCIR7TJXKETDANJ7CLQNRSIV5RS67F4RJYL66XFTSA, seg_pos={0 0}, upload=535351]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 31 02:22:15.144597046 (610h53m15.015680082s) [noise=true , address=45.91.116.4:4939 , piece_id=RIQWQXXNI7SCIR7TJXKETDANJ7CLQNRSIV5RS67F4RJYL66XFTSA, seg_pos={0 0}, upload=535351]: Long tail cancellation due to optimal threshold Dec 31 02:22:15.144618166 (610h53m15.015701201s) [noise=true , address=77.90.31.14:4661 , piece_id=5MI4JBBRHTC5MR57TUBEHNZVX6OD7AODOPSBCYH4TSYUPUYDRGNQ, seg_pos={0 0}, upload=535351]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Dec 31 02:22:15.144626191 (610h53m15.015709226s) [noise=true , address=77.90.31.14:4661 , piece_id=5MI4JBBRHTC5MR57TUBEHNZVX6OD7AODOPSBCYH4TSYUPUYDRGNQ, seg_pos={0 0}, upload=535351]: Long tail cancellation due to optimal threshold Dec 31 02:22:15.144800363 (610h53m15.015883398s) [noise=true , address=77.90.31.11:4555 , piece_id=VMAFYHAEN5YKSBAN33PS7FUSNBDQ3DBO5JG7KRBNJWBNW7M5LCYA, seg_pos={0 0}, upload=535351]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: write tcp 188.166.225.170:50294->77.90.31.11:4555: use of closed network connection; write tcp 188.166.225.170:50294->77.90.31.11:4555: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50294->77.90.31.11:4555: use of closed network connection Dec 31 02:22:15.144807157 (610h53m15.015890192s) [noise=true , address=77.90.31.11:4555 , piece_id=VMAFYHAEN5YKSBAN33PS7FUSNBDQ3DBO5JG7KRBNJWBNW7M5LCYA, seg_pos={0 0}, upload=535351]: Long tail cancellation due to optimal threshold Dec 31 02:22:15.144821844 (610h53m15.015904889s) [seg_pos={0 0}, upload=535351]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 02:22:15.144898269 (610h53m15.015981304s) [upload=535351]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 02:22:15.144903008 (610h53m15.015986043s) [upload=535351]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 02:22:15.439148899 (610h53m15.310231934s) [upload=535351]: Done uploading segments... Dec 31 02:22:15.439170178 (610h53m15.310253214s) [upload=535351]: Upload finished. err: <nil> Duration: 30.003381833s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X..........................X...............................] duration: 26.961093207s trace: https://jaeger.dev.storj.io/trace/365693fea00df28 FAIL (2025-12-31 02:22:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [noise=true , address=145.79.227.49:4343 , piece_id=6L3HPTS7D2NRA4QEOH4RFM3D3B53CTCONRI6G6YVU4OOTUWTG7XA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.942060847 (610h25m24.813143892s) [noise=true , address=145.79.227.34:4937 , piece_id=5W7LBM3YJJTVUFYTELR57DVXB663OHO5N3Z2SPQPZ4G43IK47O3Q, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.942684104 (610h25m24.813767139s) [noise=true , address=64.49.13.139:4478 , piece_id=QN6RT2KWEA6APQVA66ZMKG4LH537UGYNOXOCVVA2YC2MKDOLT3YA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.943064007 (610h25m24.814147042s) [noise=true , address=145.79.227.40:4084 , piece_id=RUKTKDII6BRUWSYVS65CWUBT3QKG3ASUK3KA3O5VGE53XSG6IMOQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.943459560 (610h25m24.814542605s) [noise=true , address=145.79.227.14:4847 , piece_id=JX5NXTMVM3NSWIEVR4RX46ABSNPKTOFDLFBPKEXFQ3G3ANKV5OLA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.943865303 (610h25m24.814948348s) [noise=true , address=45.91.116.6:4981 , piece_id=CT474ZA2LVAZEBJHBDD4GB6NTHVHZFJLNE5OZV3DFS2NZKVW3EQQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.944259694 (610h25m24.815342749s) [noise=true , address=145.79.227.51:4443 , piece_id=73UUMRDY7BSZNCGBXCZYYRBVE5KLPMNPRNXOZEGWKQGLHJS32J2Q, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.946575182 (610h25m24.817658227s) [noise=true , address=77.90.31.10:4523 , piece_id=P6LIONXJI6AFZ2JTKCQ2W6RZCOZCK6MZ53EXAJKXYQXSX2PX4QEA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.948132909 (610h25m24.819215944s) [noise=true , address=145.79.227.20:4452 , piece_id=ZSDN2JZ3FQUVJYN5OURDXNKEDLDNULJOHKKRZVIPXZIWIYAHCRXQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.949605774 (610h25m24.820688819s) [noise=true , address=77.90.31.14:4680 , piece_id=S2E4WBA7MJVAZLT5JG3IMHK53RQC77CCHV2PDF6L2R2JEA2U3K4Q, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.951459002 (610h25m24.822542048s) [noise=true , address=193.149.164.143:4704 , piece_id=BHXHFBKTLTDMGXWY5QRTP2THZQE5OXFZ2LCMTO2KRYRWAHXWW3NQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.952986742 (610h25m24.824069777s) [noise=true , address=77.90.31.19:4848 , piece_id=VZYG32S6ODIPDKQYZRXECBHXOERZDDWD2OF35Z46W4XMUZ3OJLAA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.956916654 (610h25m24.827999689s) [noise=true , address=145.79.227.29:4971 , piece_id=MEPXHHOQ6SUZC3GIPCQEMOKKF5OJTKZEI3UYZBAYXAQZGSVEKSBA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.960665062 (610h25m24.831748097s) [noise=true , address=64.49.13.170:4720 , piece_id=HIATRSAPE4YLSODFZDPP43ZUMLDVM7GJEVMWE4ZGUQTP25LTHGTA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.962366001 (610h25m24.833449036s) [noise=true , address=145.79.227.2:4117 , piece_id=4SGFGB3IJ7LTXDMMWM2PGGOV64MC46IZ7OZQWH55J4NBQY7CHMZQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.963708216 (610h25m24.834791261s) [noise=true , address=145.79.227.35:4505 , piece_id=V7RS7A4QLZTH4G32ZUZNSF2Z5APVWZGJO77NEGP3SBWQT3WCMVLQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:26.352236138 (610h25m26.223319173s) [upload=534952]: Next returned nil segment Dec 31 01:54:26.352253821 (610h25m26.223336857s) [upload=534952]: Waiting for error group managing segments... Dec 31 01:54:26.812758374 (610h25m26.683841409s) [noise=true , address=145.79.227.51:4443 , piece_id=73UUMRDY7BSZNCGBXCZYYRBVE5KLPMNPRNXOZEGWKQGLHJS32J2Q, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.824371056 (610h25m26.695454101s) [noise=true , address=145.79.227.14:4847 , piece_id=JX5NXTMVM3NSWIEVR4RX46ABSNPKTOFDLFBPKEXFQ3G3ANKV5OLA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.838414046 (610h25m26.709497081s) [noise=true , address=64.49.13.138:4490 , piece_id=2XTUZE6PXRE3OARNJVT36FXXWUMYIBVNE6M2VVFAVBGKEE2H5YZQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.875499841 (610h25m26.746582876s) [noise=true , address=64.49.13.139:4478 , piece_id=QN6RT2KWEA6APQVA66ZMKG4LH537UGYNOXOCVVA2YC2MKDOLT3YA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.891607221 (610h25m26.762690256s) [noise=true , address=145.79.227.20:4452 , piece_id=ZSDN2JZ3FQUVJYN5OURDXNKEDLDNULJOHKKRZVIPXZIWIYAHCRXQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.988561223 (610h25m26.859644268s) [noise=true , address=64.49.13.170:4720 , piece_id=HIATRSAPE4YLSODFZDPP43ZUMLDVM7GJEVMWE4ZGUQTP25LTHGTA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.039035911 (610h25m26.910118946s) [noise=true , address=77.90.31.14:4680 , piece_id=S2E4WBA7MJVAZLT5JG3IMHK53RQC77CCHV2PDF6L2R2JEA2U3K4Q, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.098023090 (610h25m26.969106126s) [noise=true , address=45.91.116.6:4981 , piece_id=CT474ZA2LVAZEBJHBDD4GB6NTHVHZFJLNE5OZV3DFS2NZKVW3EQQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.653790754 (610h25m27.524873789s) [noise=true , address=77.90.31.21:4886 , piece_id=K34WPIBGGGEPHIKJ253XGLDCIQ3WLFN4AZ6EYNWVHHT2EIHS6YRQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.684192972 (610h25m27.555276017s) [noise=true , address=193.149.164.143:4704 , piece_id=BHXHFBKTLTDMGXWY5QRTP2THZQE5OXFZ2LCMTO2KRYRWAHXWW3NQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.717238425 (610h25m27.58832146s) [noise=true , address=145.79.227.40:4084 , piece_id=RUKTKDII6BRUWSYVS65CWUBT3QKG3ASUK3KA3O5VGE53XSG6IMOQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.718349742 (610h25m27.589432777s) [noise=true , address=77.90.31.10:4523 , piece_id=P6LIONXJI6AFZ2JTKCQ2W6RZCOZCK6MZ53EXAJKXYQXSX2PX4QEA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.803639562 (610h25m27.674722597s) [noise=true , address=145.79.227.29:4971 , piece_id=MEPXHHOQ6SUZC3GIPCQEMOKKF5OJTKZEI3UYZBAYXAQZGSVEKSBA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.812791279 (610h25m27.683874314s) [noise=true , address=145.79.227.49:4343 , piece_id=6L3HPTS7D2NRA4QEOH4RFM3D3B53CTCONRI6G6YVU4OOTUWTG7XA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.812812348 (610h25m27.683895393s) [seg_pos={0 0}, upload=534952]: Segment reached optimal threshold of 14 pieces. Dec 31 01:54:27.813018000 (610h25m27.684101035s) [noise=true , address=145.79.227.35:4505 , piece_id=V7RS7A4QLZTH4G32ZUZNSF2Z5APVWZGJO77NEGP3SBWQT3WCMVLQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i6itGoBhLsLiqC8et8Y5L62fLjcWxy18u39NAr4R3nXNMWcsf): protocol: expected piece hash; context canceled; EOF Dec 31 01:54:27.813024772 (610h25m27.684107808s) [noise=true , address=145.79.227.35:4505 , piece_id=V7RS7A4QLZTH4G32ZUZNSF2Z5APVWZGJO77NEGP3SBWQT3WCMVLQ, seg_pos={0 0}, upload=534952]: Long tail cancellation due to optimal threshold Dec 31 01:54:27.813175931 (610h25m27.684258976s) [noise=true , address=145.79.227.34:4937 , piece_id=5W7LBM3YJJTVUFYTELR57DVXB663OHO5N3Z2SPQPZ4G43IK47O3Q, seg_pos={0 0}, upload=534952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1LKrFThdW3jJpG6dRZLUaxLXTa5Wy27yQ8PGUEqjvBWfFoRB94): protocol: expected piece hash; context canceled; EOF Dec 31 01:54:27.813181702 (610h25m27.684264737s) [noise=true , address=145.79.227.34:4937 , piece_id=5W7LBM3YJJTVUFYTELR57DVXB663OHO5N3Z2SPQPZ4G43IK47O3Q, seg_pos={0 0}, upload=534952]: Long tail cancellation due to optimal threshold Dec 31 01:54:27.813407862 (610h25m27.684490907s) [noise=true , address=145.79.227.2:4117 , piece_id=4SGFGB3IJ7LTXDMMWM2PGGOV64MC46IZ7OZQWH55J4NBQY7CHMZQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122Htg72f5eUUutf1p3fYQpy1Wkf6AgdaQV83HHYye2n8zm3the): protocol: expected piece hash; context canceled; EOF Dec 31 01:54:27.813414264 (610h25m27.684497299s) [noise=true , address=145.79.227.2:4117 , piece_id=4SGFGB3IJ7LTXDMMWM2PGGOV64MC46IZ7OZQWH55J4NBQY7CHMZQ, seg_pos={0 0}, upload=534952]: Long tail cancellation due to optimal threshold Dec 31 01:54:27.813424614 (610h25m27.684507649s) [noise=true , address=77.90.31.19:4848 , piece_id=VZYG32S6ODIPDKQYZRXECBHXOERZDDWD2OF35Z46W4XMUZ3OJLAA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD): protocol: write tcp 188.166.225.170:42842->77.90.31.19:4848: use of closed network connection; write tcp 188.166.225.170:42842->77.90.31.19:4848: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42842->77.90.31.19:4848: use of closed network connection Dec 31 01:54:27.813432750 (610h25m27.684515785s) [noise=true , address=77.90.31.19:4848 , piece_id=VZYG32S6ODIPDKQYZRXECBHXOERZDDWD2OF35Z46W4XMUZ3OJLAA, seg_pos={0 0}, upload=534952]: Long tail cancellation due to optimal threshold Dec 31 01:54:27.813474088 (610h25m27.684557123s) [seg_pos={0 0}, upload=534952]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 01:54:27.813535325 (610h25m27.68461836s) [upload=534952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 01:54:27.813539613 (610h25m27.684622648s) [upload=534952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 01:54:28.099921034 (610h25m27.971004069s) [upload=534952]: Done uploading segments... Dec 31 01:54:28.099940451 (610h25m27.971023486s) [upload=534952]: Upload finished. err: <nil> Duration: 26.961123626s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [........X..................................X................] duration: 3.881391808s trace: https://jaeger.dev.storj.io/trace/740aa405bd68150f FAIL (2025-12-31 02:22:33 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.881420582s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.........X..................................X...............] duration: 3.492454635s trace: https://jaeger.dev.storj.io/trace/740aa405bd68150f FAIL (2025-12-31 02:22:06 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.492528355s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [X..........X............X..X.......X.X..X....XX.XX........X.] duration: 25.857763873s trace: https://jaeger.dev.storj.io/trace/a9c722217f16777 FAIL (2025-12-31 02:21:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 31 02:20:47.408700015 (610h51m47.27978305s) [upload=535333]: Starting upload Dec 31 02:20:47.408766592 (610h51m47.279849627s) [upload=535333]: Done starting upload Dec 31 02:20:47.408832859 (610h51m47.279915904s) [upload=535333]: Uploading segments... Dec 31 02:20:47.408924874 (610h51m47.280007909s) [upload=535333]: Got next segment. Inline: false Dec 31 02:20:47.409049701 (610h51m47.280132736s) [upload=535333]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 02:20:47.409059070 (610h51m47.280142115s) [upload=535333]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 02:20:47.654327030 (610h51m47.525410065s) [seg_pos={0 0}, upload=535333]: Begin upload segment... Dec 31 02:20:47.654407543 (610h51m47.525490578s) [seg_pos={0 0}, upload=535333]: Done begin upload segment. Dec 31 02:20:47.654607914 (610h51m47.525690959s) [noise=true , address=145.79.227.21:4475 , piece_id=2ZV7HFMVGJABACAR37P64LIQMMPQLIHLJINH26TAGXUPPOI6UVZA, seg_pos={0 0}, upload=535333]: Uploading piece... Dec 31 02:20:47.654653401 (610h51m47.525736436s) [noise=true , address=45.91.116.9:4264 , piece_id=YPANPRX7EIQYGARSDMDEUN3QWERVE6E3XHGPW7SGMVW5SYGVVV7A, seg_pos={0 0}, upload=535333]: Uploading piece... Dec 31 02:20:47.655449547 (610h51m47.526532582s) [noise=true , address=145.79.227.9:4365 , piece_id=TP7BPAEVFDRCNA7O6MUTWDNADASNXWEPFWB3PIZR44RTHCPMOK5Q, seg_pos={0 0}, upload=535333]: Uploading piece... Dec 31 02:20:47.655842144 (610h51m47.526925179s) [noise=true , address=45.91.116.13:4155 , piece_id=WIJOLMCXL7VEF2GLVKLXTFWBFN64HKWEPGJCCS6PYBPKTQ6TZ2DQ, seg_pos={0 0}, upload=535333]: Uploading piece... Dec 31 02:20:47.656165640 (610h51m47.527248675s) [noise=true , address=77.90.31.16:4736 , piece_id=ISKP5W7KHCAJZS6XQMILH72XRV3ZGT5NOUG4BMCSQ47J7K5WLO6Q, seg_pos={0 0}, upload=535333]: Uploading piece... Dec 31 02:20:47.656470851 (610h51m47.527553886s) [noise=true , address=77.90.31.18:4805 , piece_id=ZVTZCD2YNL2JGM65EGGV36NCHQGLV4ARU642RPXSZ5DL2624F5LA, seg_pos={0 0}, upload=535333]: Uploading piece... Dec 31 02:20:47.656824805 (610h51m47.52790784s) [noise=true , address=45.91.116.6:4975 , piece_id=MODKDYY23IMVG7LVPWIOFIJYQD5ONCRV4RT2XAU2ELUICWGBLYMQ, seg_pos={0 0}, upload=535333]: Uploading piece... Dec 31 02:20:47.657150376 (610h51m47.528233411s) [noise=true , address=145.79.227.30:4761 , piece_id=4BLJGIYHEH2HVUFMKXBI2JYV4QWVTS52PQCAWXXMTJFQMPIHFOYQ, seg_pos={0 0}, upload=535333]: Uploading piece... Dec 31 02:20:47.657865256 (610h51m47.528948301s) [noise=true , address=45.91.116.8:4047 , piece_id=3JY4CMKNJOJFGR4MABZYQTUBAMFEUGXQMCUMWS3RSMJMKQOZKSRA, seg_pos={0 0}, upload=535333]: Uploading piece... Dec 31 02:20:47.658176599 (610h51m47.529259634s) [noise=true , address=45.91.116.4:4939 , piece_id=SF3GCQ2FQ5SMTJLYAGEJ6SLDNC3QJX5FHGPPLESKX7RGCMUXCD2A, seg_pos={0 0}, upload=535333]: Uploading piece... Dec 31 02:20:47.658607079 (610h51m47.529690114s) [noise=true , address=145.79.227.25:4618 , piece_id=J4G6JVICMMZFHRDIFERBOQALDKMZTJVPCKRIFYNYESD7Q3RUHWWQ, seg_pos={0 0}, upload=535333]: Uploading piece... Dec 31 02:20:47.658961403 (610h51m47.530044438s) [noise=true , address=145.79.227.32:4836 , piece_id=JQ5Z36OBUJVRBUFTBHYDKVHYLZAYFIILRRCXJY6ADGNNLTT5JWYQ, seg_pos={0 0}, upload=535333]: Uploading piece... Dec 31 02:20:47.659277666 (610h51m47.530360701s) [noise=true , address=145.79.227.16:4910 , piece_id=NN3D5FLS2UE54XHWWKZF3CDRTUBHWFJCU7Z3IRJFGSLXNGFZ5HGQ, seg_pos={0 0}, upload=535333]: Uploading piece... Dec 31 02:20:47.659566776 (610h51m47.530649812s) [noise=true , address=145.79.227.2:4114 , piece_id=SWBZCYKTAYJMJNU4NERZJ5DADUD2MXVHNV2WCOJ3GRSZ4CRR3EYQ, seg_pos={0 0}, upload=535333]: Uploading piece... Dec 31 02:20:47.659843313 (610h51m47.530926348s) [noise=true , address=145.79.227.23:4547 , piece_id=ARX3FJKLPGE5DHNUNJGGFRJKP477MCAJAMIYT6H36BCJ5RFLPOBA, seg_pos={0 0}, upload=535333]: Uploading piece... Dec 31 02:20:47.660155498 (610h51m47.531238533s) [noise=true , address=145.79.227.3:4151 , piece_id=2MHAVROHBI6H2DJSXONIAJJGGRZLWSGY3QZQBLMPMRA2SJVRQA6Q, seg_pos={0 0}, upload=535333]: Uploading piece... Dec 31 02:20:49.357386800 (610h51m49.228469835s) [upload=535333]: Next returned nil segment Dec 31 02:20:49.357407339 (610h51m49.228490374s) [upload=535333]: Waiting for error group managing segments... Dec 31 02:20:49.630942934 (610h51m49.502025969s) [noise=true , address=145.79.227.9:4365 , piece_id=TP7BPAEVFDRCNA7O6MUTWDNADASNXWEPFWB3PIZR44RTHCPMOK5Q, seg_pos={0 0}, upload=535333]: Done uploading piece. err: <nil> Dec 31 02:20:49.640973212 (610h51m49.512056257s) [noise=true , address=145.79.227.3:4151 , piece_id=2MHAVROHBI6H2DJSXONIAJJGGRZLWSGY3QZQBLMPMRA2SJVRQA6Q, seg_pos={0 0}, upload=535333]: Done uploading piece. err: <nil> Dec 31 02:20:49.644304165 (610h51m49.5153872s) [noise=true , address=145.79.227.16:4910 , piece_id=NN3D5FLS2UE54XHWWKZF3CDRTUBHWFJCU7Z3IRJFGSLXNGFZ5HGQ, seg_pos={0 0}, upload=535333]: Done uploading piece. err: <nil> Dec 31 02:20:49.644380781 (610h51m49.515463816s) [noise=true , address=145.79.227.32:4836 , piece_id=JQ5Z36OBUJVRBUFTBHYDKVHYLZAYFIILRRCXJY6ADGNNLTT5JWYQ, seg_pos={0 0}, upload=535333]: Done uploading piece. err: <nil> Dec 31 02:20:49.645163131 (610h51m49.516246166s) [noise=true , address=145.79.227.23:4547 , piece_id=ARX3FJKLPGE5DHNUNJGGFRJKP477MCAJAMIYT6H36BCJ5RFLPOBA, seg_pos={0 0}, upload=535333]: Done uploading piece. err: <nil> Dec 31 02:20:49.662285945 (610h51m49.53336898s) [noise=true , address=145.79.227.21:4475 , piece_id=2ZV7HFMVGJABACAR37P64LIQMMPQLIHLJINH26TAGXUPPOI6UVZA, seg_pos={0 0}, upload=535333]: Done uploading piece. err: <nil> Dec 31 02:20:49.729578257 (610h51m49.600661292s) [noise=true , address=145.79.227.2:4114 , piece_id=SWBZCYKTAYJMJNU4NERZJ5DADUD2MXVHNV2WCOJ3GRSZ4CRR3EYQ, seg_pos={0 0}, upload=535333]: Done uploading piece. err: <nil> Dec 31 02:20:49.745322846 (610h51m49.616405881s) [noise=true , address=145.79.227.25:4618 , piece_id=J4G6JVICMMZFHRDIFERBOQALDKMZTJVPCKRIFYNYESD7Q3RUHWWQ, seg_pos={0 0}, upload=535333]: Done uploading piece. err: <nil> Dec 31 02:20:49.947496550 (610h51m49.818579585s) [noise=true , address=45.91.116.6:4975 , piece_id=MODKDYY23IMVG7LVPWIOFIJYQD5ONCRV4RT2XAU2ELUICWGBLYMQ, seg_pos={0 0}, upload=535333]: Done uploading piece. err: <nil> Dec 31 02:20:49.968148674 (610h51m49.839231719s) [noise=true , address=77.90.31.16:4736 , piece_id=ISKP5W7KHCAJZS6XQMILH72XRV3ZGT5NOUG4BMCSQ47J7K5WLO6Q, seg_pos={0 0}, upload=535333]: Done uploading piece. err: <nil> Dec 31 02:20:49.974232399 (610h51m49.845315434s) [noise=true , address=77.90.31.18:4805 , piece_id=ZVTZCD2YNL2JGM65EGGV36NCHQGLV4ARU642RPXSZ5DL2624F5LA, seg_pos={0 0}, upload=535333]: Done uploading piece. err: <nil> Dec 31 02:20:49.997918510 (610h51m49.869001555s) [noise=true , address=45.91.116.8:4047 , piece_id=3JY4CMKNJOJFGR4MABZYQTUBAMFEUGXQMCUMWS3RSMJMKQOZKSRA, seg_pos={0 0}, upload=535333]: Done uploading piece. err: <nil> Dec 31 02:20:50.075045498 (610h51m49.946128563s) [noise=true , address=45.91.116.4:4939 , piece_id=SF3GCQ2FQ5SMTJLYAGEJ6SLDNC3QJX5FHGPPLESKX7RGCMUXCD2A, seg_pos={0 0}, upload=535333]: Done uploading piece. err: <nil> Dec 31 02:20:50.075074864 (610h51m49.946157909s) [seg_pos={0 0}, upload=535333]: Segment reached optimal threshold of 13 pieces. Dec 31 02:20:50.075391197 (610h51m49.946474242s) [noise=true , address=45.91.116.13:4155 , piece_id=WIJOLMCXL7VEF2GLVKLXTFWBFN64HKWEPGJCCS6PYBPKTQ6TZ2DQ, seg_pos={0 0}, upload=535333]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 31 02:20:50.075409983 (610h51m49.946493028s) [noise=true , address=45.91.116.13:4155 , piece_id=WIJOLMCXL7VEF2GLVKLXTFWBFN64HKWEPGJCCS6PYBPKTQ6TZ2DQ, seg_pos={0 0}, upload=535333]: Long tail cancellation due to optimal threshold Dec 31 02:20:50.075460629 (610h51m49.946543674s) [noise=true , address=45.91.116.9:4264 , piece_id=YPANPRX7EIQYGARSDMDEUN3QWERVE6E3XHGPW7SGMVW5SYGVVV7A, seg_pos={0 0}, upload=535333]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 31 02:20:50.075469616 (610h51m49.946552651s) [noise=true , address=45.91.116.9:4264 , piece_id=YPANPRX7EIQYGARSDMDEUN3QWERVE6E3XHGPW7SGMVW5SYGVVV7A, seg_pos={0 0}, upload=535333]: Long tail cancellation due to optimal threshold Dec 31 02:20:50.075604102 (610h51m49.946687147s) [noise=true , address=145.79.227.30:4761 , piece_id=4BLJGIYHEH2HVUFMKXBI2JYV4QWVTS52PQCAWXXMTJFQMPIHFOYQ, seg_pos={0 0}, upload=535333]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 31 02:20:50.075609783 (610h51m49.946692828s) [noise=true , address=145.79.227.30:4761 , piece_id=4BLJGIYHEH2HVUFMKXBI2JYV4QWVTS52PQCAWXXMTJFQMPIHFOYQ, seg_pos={0 0}, upload=535333]: Long tail cancellation due to optimal threshold Dec 31 02:20:50.075657023 (610h51m49.946740058s) [seg_pos={0 0}, upload=535333]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 02:20:50.075747355 (610h51m49.94683039s) [upload=535333]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 02:20:50.075752594 (610h51m49.946835629s) [upload=535333]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 02:20:50.370372256 (610h51m50.241455291s) [upload=535333]: Done uploading segments... Dec 31 02:20:50.370398446 (610h51m50.241481491s) [upload=535333]: Upload finished. err: <nil> Duration: 25.857780585s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..X..........................X..............................] duration: 41.451269477s trace: https://jaeger.dev.storj.io/trace/365693fea00df28 FAIL (2025-12-31 02:21:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [noise=true , address=145.79.227.49:4343 , piece_id=6L3HPTS7D2NRA4QEOH4RFM3D3B53CTCONRI6G6YVU4OOTUWTG7XA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.942060847 (610h25m24.813143892s) [noise=true , address=145.79.227.34:4937 , piece_id=5W7LBM3YJJTVUFYTELR57DVXB663OHO5N3Z2SPQPZ4G43IK47O3Q, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.942684104 (610h25m24.813767139s) [noise=true , address=64.49.13.139:4478 , piece_id=QN6RT2KWEA6APQVA66ZMKG4LH537UGYNOXOCVVA2YC2MKDOLT3YA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.943064007 (610h25m24.814147042s) [noise=true , address=145.79.227.40:4084 , piece_id=RUKTKDII6BRUWSYVS65CWUBT3QKG3ASUK3KA3O5VGE53XSG6IMOQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.943459560 (610h25m24.814542605s) [noise=true , address=145.79.227.14:4847 , piece_id=JX5NXTMVM3NSWIEVR4RX46ABSNPKTOFDLFBPKEXFQ3G3ANKV5OLA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.943865303 (610h25m24.814948348s) [noise=true , address=45.91.116.6:4981 , piece_id=CT474ZA2LVAZEBJHBDD4GB6NTHVHZFJLNE5OZV3DFS2NZKVW3EQQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.944259694 (610h25m24.815342749s) [noise=true , address=145.79.227.51:4443 , piece_id=73UUMRDY7BSZNCGBXCZYYRBVE5KLPMNPRNXOZEGWKQGLHJS32J2Q, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.946575182 (610h25m24.817658227s) [noise=true , address=77.90.31.10:4523 , piece_id=P6LIONXJI6AFZ2JTKCQ2W6RZCOZCK6MZ53EXAJKXYQXSX2PX4QEA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.948132909 (610h25m24.819215944s) [noise=true , address=145.79.227.20:4452 , piece_id=ZSDN2JZ3FQUVJYN5OURDXNKEDLDNULJOHKKRZVIPXZIWIYAHCRXQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.949605774 (610h25m24.820688819s) [noise=true , address=77.90.31.14:4680 , piece_id=S2E4WBA7MJVAZLT5JG3IMHK53RQC77CCHV2PDF6L2R2JEA2U3K4Q, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.951459002 (610h25m24.822542048s) [noise=true , address=193.149.164.143:4704 , piece_id=BHXHFBKTLTDMGXWY5QRTP2THZQE5OXFZ2LCMTO2KRYRWAHXWW3NQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.952986742 (610h25m24.824069777s) [noise=true , address=77.90.31.19:4848 , piece_id=VZYG32S6ODIPDKQYZRXECBHXOERZDDWD2OF35Z46W4XMUZ3OJLAA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.956916654 (610h25m24.827999689s) [noise=true , address=145.79.227.29:4971 , piece_id=MEPXHHOQ6SUZC3GIPCQEMOKKF5OJTKZEI3UYZBAYXAQZGSVEKSBA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.960665062 (610h25m24.831748097s) [noise=true , address=64.49.13.170:4720 , piece_id=HIATRSAPE4YLSODFZDPP43ZUMLDVM7GJEVMWE4ZGUQTP25LTHGTA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.962366001 (610h25m24.833449036s) [noise=true , address=145.79.227.2:4117 , piece_id=4SGFGB3IJ7LTXDMMWM2PGGOV64MC46IZ7OZQWH55J4NBQY7CHMZQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.963708216 (610h25m24.834791261s) [noise=true , address=145.79.227.35:4505 , piece_id=V7RS7A4QLZTH4G32ZUZNSF2Z5APVWZGJO77NEGP3SBWQT3WCMVLQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:26.352236138 (610h25m26.223319173s) [upload=534952]: Next returned nil segment Dec 31 01:54:26.352253821 (610h25m26.223336857s) [upload=534952]: Waiting for error group managing segments... Dec 31 01:54:26.812758374 (610h25m26.683841409s) [noise=true , address=145.79.227.51:4443 , piece_id=73UUMRDY7BSZNCGBXCZYYRBVE5KLPMNPRNXOZEGWKQGLHJS32J2Q, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.824371056 (610h25m26.695454101s) [noise=true , address=145.79.227.14:4847 , piece_id=JX5NXTMVM3NSWIEVR4RX46ABSNPKTOFDLFBPKEXFQ3G3ANKV5OLA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.838414046 (610h25m26.709497081s) [noise=true , address=64.49.13.138:4490 , piece_id=2XTUZE6PXRE3OARNJVT36FXXWUMYIBVNE6M2VVFAVBGKEE2H5YZQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.875499841 (610h25m26.746582876s) [noise=true , address=64.49.13.139:4478 , piece_id=QN6RT2KWEA6APQVA66ZMKG4LH537UGYNOXOCVVA2YC2MKDOLT3YA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.891607221 (610h25m26.762690256s) [noise=true , address=145.79.227.20:4452 , piece_id=ZSDN2JZ3FQUVJYN5OURDXNKEDLDNULJOHKKRZVIPXZIWIYAHCRXQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.988561223 (610h25m26.859644268s) [noise=true , address=64.49.13.170:4720 , piece_id=HIATRSAPE4YLSODFZDPP43ZUMLDVM7GJEVMWE4ZGUQTP25LTHGTA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.039035911 (610h25m26.910118946s) [noise=true , address=77.90.31.14:4680 , piece_id=S2E4WBA7MJVAZLT5JG3IMHK53RQC77CCHV2PDF6L2R2JEA2U3K4Q, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.098023090 (610h25m26.969106126s) [noise=true , address=45.91.116.6:4981 , piece_id=CT474ZA2LVAZEBJHBDD4GB6NTHVHZFJLNE5OZV3DFS2NZKVW3EQQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.653790754 (610h25m27.524873789s) [noise=true , address=77.90.31.21:4886 , piece_id=K34WPIBGGGEPHIKJ253XGLDCIQ3WLFN4AZ6EYNWVHHT2EIHS6YRQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.684192972 (610h25m27.555276017s) [noise=true , address=193.149.164.143:4704 , piece_id=BHXHFBKTLTDMGXWY5QRTP2THZQE5OXFZ2LCMTO2KRYRWAHXWW3NQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.717238425 (610h25m27.58832146s) [noise=true , address=145.79.227.40:4084 , piece_id=RUKTKDII6BRUWSYVS65CWUBT3QKG3ASUK3KA3O5VGE53XSG6IMOQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.718349742 (610h25m27.589432777s) [noise=true , address=77.90.31.10:4523 , piece_id=P6LIONXJI6AFZ2JTKCQ2W6RZCOZCK6MZ53EXAJKXYQXSX2PX4QEA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.803639562 (610h25m27.674722597s) [noise=true , address=145.79.227.29:4971 , piece_id=MEPXHHOQ6SUZC3GIPCQEMOKKF5OJTKZEI3UYZBAYXAQZGSVEKSBA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.812791279 (610h25m27.683874314s) [noise=true , address=145.79.227.49:4343 , piece_id=6L3HPTS7D2NRA4QEOH4RFM3D3B53CTCONRI6G6YVU4OOTUWTG7XA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.812812348 (610h25m27.683895393s) [seg_pos={0 0}, upload=534952]: Segment reached optimal threshold of 14 pieces. Dec 31 01:54:27.813018000 (610h25m27.684101035s) [noise=true , address=145.79.227.35:4505 , piece_id=V7RS7A4QLZTH4G32ZUZNSF2Z5APVWZGJO77NEGP3SBWQT3WCMVLQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i6itGoBhLsLiqC8et8Y5L62fLjcWxy18u39NAr4R3nXNMWcsf): protocol: expected piece hash; context canceled; EOF Dec 31 01:54:27.813024772 (610h25m27.684107808s) [noise=true , address=145.79.227.35:4505 , piece_id=V7RS7A4QLZTH4G32ZUZNSF2Z5APVWZGJO77NEGP3SBWQT3WCMVLQ, seg_pos={0 0}, upload=534952]: Long tail cancellation due to optimal threshold Dec 31 01:54:27.813175931 (610h25m27.684258976s) [noise=true , address=145.79.227.34:4937 , piece_id=5W7LBM3YJJTVUFYTELR57DVXB663OHO5N3Z2SPQPZ4G43IK47O3Q, seg_pos={0 0}, upload=534952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1LKrFThdW3jJpG6dRZLUaxLXTa5Wy27yQ8PGUEqjvBWfFoRB94): protocol: expected piece hash; context canceled; EOF Dec 31 01:54:27.813181702 (610h25m27.684264737s) [noise=true , address=145.79.227.34:4937 , piece_id=5W7LBM3YJJTVUFYTELR57DVXB663OHO5N3Z2SPQPZ4G43IK47O3Q, seg_pos={0 0}, upload=534952]: Long tail cancellation due to optimal threshold Dec 31 01:54:27.813407862 (610h25m27.684490907s) [noise=true , address=145.79.227.2:4117 , piece_id=4SGFGB3IJ7LTXDMMWM2PGGOV64MC46IZ7OZQWH55J4NBQY7CHMZQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122Htg72f5eUUutf1p3fYQpy1Wkf6AgdaQV83HHYye2n8zm3the): protocol: expected piece hash; context canceled; EOF Dec 31 01:54:27.813414264 (610h25m27.684497299s) [noise=true , address=145.79.227.2:4117 , piece_id=4SGFGB3IJ7LTXDMMWM2PGGOV64MC46IZ7OZQWH55J4NBQY7CHMZQ, seg_pos={0 0}, upload=534952]: Long tail cancellation due to optimal threshold Dec 31 01:54:27.813424614 (610h25m27.684507649s) [noise=true , address=77.90.31.19:4848 , piece_id=VZYG32S6ODIPDKQYZRXECBHXOERZDDWD2OF35Z46W4XMUZ3OJLAA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD): protocol: write tcp 188.166.225.170:42842->77.90.31.19:4848: use of closed network connection; write tcp 188.166.225.170:42842->77.90.31.19:4848: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42842->77.90.31.19:4848: use of closed network connection Dec 31 01:54:27.813432750 (610h25m27.684515785s) [noise=true , address=77.90.31.19:4848 , piece_id=VZYG32S6ODIPDKQYZRXECBHXOERZDDWD2OF35Z46W4XMUZ3OJLAA, seg_pos={0 0}, upload=534952]: Long tail cancellation due to optimal threshold Dec 31 01:54:27.813474088 (610h25m27.684557123s) [seg_pos={0 0}, upload=534952]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 01:54:27.813535325 (610h25m27.68461836s) [upload=534952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 01:54:27.813539613 (610h25m27.684622648s) [upload=534952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 01:54:28.099921034 (610h25m27.971004069s) [upload=534952]: Done uploading segments... Dec 31 01:54:28.099940451 (610h25m27.971023486s) [upload=534952]: Upload finished. err: <nil> Duration: 41.451300667s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..........X..................................X..............] duration: 3.440408733s trace: https://jaeger.dev.storj.io/trace/740aa405bd68150f FAIL (2025-12-31 02:21:34 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.440427939s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.X..........X............X..X.......X.X..X....XX.XX........X] duration: 30.003691668s trace: https://jaeger.dev.storj.io/trace/a9c722217f16777 FAIL (2025-12-31 02:21:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 31 02:20:47.408700015 (610h51m47.27978305s) [upload=535333]: Starting upload Dec 31 02:20:47.408766592 (610h51m47.279849627s) [upload=535333]: Done starting upload Dec 31 02:20:47.408832859 (610h51m47.279915904s) [upload=535333]: Uploading segments... Dec 31 02:20:47.408924874 (610h51m47.280007909s) [upload=535333]: Got next segment. Inline: false Dec 31 02:20:47.409049701 (610h51m47.280132736s) [upload=535333]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 02:20:47.409059070 (610h51m47.280142115s) [upload=535333]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 02:20:47.654327030 (610h51m47.525410065s) [seg_pos={0 0}, upload=535333]: Begin upload segment... Dec 31 02:20:47.654407543 (610h51m47.525490578s) [seg_pos={0 0}, upload=535333]: Done begin upload segment. Dec 31 02:20:47.654607914 (610h51m47.525690959s) [noise=true , address=145.79.227.21:4475 , piece_id=2ZV7HFMVGJABACAR37P64LIQMMPQLIHLJINH26TAGXUPPOI6UVZA, seg_pos={0 0}, upload=535333]: Uploading piece... Dec 31 02:20:47.654653401 (610h51m47.525736436s) [noise=true , address=45.91.116.9:4264 , piece_id=YPANPRX7EIQYGARSDMDEUN3QWERVE6E3XHGPW7SGMVW5SYGVVV7A, seg_pos={0 0}, upload=535333]: Uploading piece... Dec 31 02:20:47.655449547 (610h51m47.526532582s) [noise=true , address=145.79.227.9:4365 , piece_id=TP7BPAEVFDRCNA7O6MUTWDNADASNXWEPFWB3PIZR44RTHCPMOK5Q, seg_pos={0 0}, upload=535333]: Uploading piece... Dec 31 02:20:47.655842144 (610h51m47.526925179s) [noise=true , address=45.91.116.13:4155 , piece_id=WIJOLMCXL7VEF2GLVKLXTFWBFN64HKWEPGJCCS6PYBPKTQ6TZ2DQ, seg_pos={0 0}, upload=535333]: Uploading piece... Dec 31 02:20:47.656165640 (610h51m47.527248675s) [noise=true , address=77.90.31.16:4736 , piece_id=ISKP5W7KHCAJZS6XQMILH72XRV3ZGT5NOUG4BMCSQ47J7K5WLO6Q, seg_pos={0 0}, upload=535333]: Uploading piece... Dec 31 02:20:47.656470851 (610h51m47.527553886s) [noise=true , address=77.90.31.18:4805 , piece_id=ZVTZCD2YNL2JGM65EGGV36NCHQGLV4ARU642RPXSZ5DL2624F5LA, seg_pos={0 0}, upload=535333]: Uploading piece... Dec 31 02:20:47.656824805 (610h51m47.52790784s) [noise=true , address=45.91.116.6:4975 , piece_id=MODKDYY23IMVG7LVPWIOFIJYQD5ONCRV4RT2XAU2ELUICWGBLYMQ, seg_pos={0 0}, upload=535333]: Uploading piece... Dec 31 02:20:47.657150376 (610h51m47.528233411s) [noise=true , address=145.79.227.30:4761 , piece_id=4BLJGIYHEH2HVUFMKXBI2JYV4QWVTS52PQCAWXXMTJFQMPIHFOYQ, seg_pos={0 0}, upload=535333]: Uploading piece... Dec 31 02:20:47.657865256 (610h51m47.528948301s) [noise=true , address=45.91.116.8:4047 , piece_id=3JY4CMKNJOJFGR4MABZYQTUBAMFEUGXQMCUMWS3RSMJMKQOZKSRA, seg_pos={0 0}, upload=535333]: Uploading piece... Dec 31 02:20:47.658176599 (610h51m47.529259634s) [noise=true , address=45.91.116.4:4939 , piece_id=SF3GCQ2FQ5SMTJLYAGEJ6SLDNC3QJX5FHGPPLESKX7RGCMUXCD2A, seg_pos={0 0}, upload=535333]: Uploading piece... Dec 31 02:20:47.658607079 (610h51m47.529690114s) [noise=true , address=145.79.227.25:4618 , piece_id=J4G6JVICMMZFHRDIFERBOQALDKMZTJVPCKRIFYNYESD7Q3RUHWWQ, seg_pos={0 0}, upload=535333]: Uploading piece... Dec 31 02:20:47.658961403 (610h51m47.530044438s) [noise=true , address=145.79.227.32:4836 , piece_id=JQ5Z36OBUJVRBUFTBHYDKVHYLZAYFIILRRCXJY6ADGNNLTT5JWYQ, seg_pos={0 0}, upload=535333]: Uploading piece... Dec 31 02:20:47.659277666 (610h51m47.530360701s) [noise=true , address=145.79.227.16:4910 , piece_id=NN3D5FLS2UE54XHWWKZF3CDRTUBHWFJCU7Z3IRJFGSLXNGFZ5HGQ, seg_pos={0 0}, upload=535333]: Uploading piece... Dec 31 02:20:47.659566776 (610h51m47.530649812s) [noise=true , address=145.79.227.2:4114 , piece_id=SWBZCYKTAYJMJNU4NERZJ5DADUD2MXVHNV2WCOJ3GRSZ4CRR3EYQ, seg_pos={0 0}, upload=535333]: Uploading piece... Dec 31 02:20:47.659843313 (610h51m47.530926348s) [noise=true , address=145.79.227.23:4547 , piece_id=ARX3FJKLPGE5DHNUNJGGFRJKP477MCAJAMIYT6H36BCJ5RFLPOBA, seg_pos={0 0}, upload=535333]: Uploading piece... Dec 31 02:20:47.660155498 (610h51m47.531238533s) [noise=true , address=145.79.227.3:4151 , piece_id=2MHAVROHBI6H2DJSXONIAJJGGRZLWSGY3QZQBLMPMRA2SJVRQA6Q, seg_pos={0 0}, upload=535333]: Uploading piece... Dec 31 02:20:49.357386800 (610h51m49.228469835s) [upload=535333]: Next returned nil segment Dec 31 02:20:49.357407339 (610h51m49.228490374s) [upload=535333]: Waiting for error group managing segments... Dec 31 02:20:49.630942934 (610h51m49.502025969s) [noise=true , address=145.79.227.9:4365 , piece_id=TP7BPAEVFDRCNA7O6MUTWDNADASNXWEPFWB3PIZR44RTHCPMOK5Q, seg_pos={0 0}, upload=535333]: Done uploading piece. err: <nil> Dec 31 02:20:49.640973212 (610h51m49.512056257s) [noise=true , address=145.79.227.3:4151 , piece_id=2MHAVROHBI6H2DJSXONIAJJGGRZLWSGY3QZQBLMPMRA2SJVRQA6Q, seg_pos={0 0}, upload=535333]: Done uploading piece. err: <nil> Dec 31 02:20:49.644304165 (610h51m49.5153872s) [noise=true , address=145.79.227.16:4910 , piece_id=NN3D5FLS2UE54XHWWKZF3CDRTUBHWFJCU7Z3IRJFGSLXNGFZ5HGQ, seg_pos={0 0}, upload=535333]: Done uploading piece. err: <nil> Dec 31 02:20:49.644380781 (610h51m49.515463816s) [noise=true , address=145.79.227.32:4836 , piece_id=JQ5Z36OBUJVRBUFTBHYDKVHYLZAYFIILRRCXJY6ADGNNLTT5JWYQ, seg_pos={0 0}, upload=535333]: Done uploading piece. err: <nil> Dec 31 02:20:49.645163131 (610h51m49.516246166s) [noise=true , address=145.79.227.23:4547 , piece_id=ARX3FJKLPGE5DHNUNJGGFRJKP477MCAJAMIYT6H36BCJ5RFLPOBA, seg_pos={0 0}, upload=535333]: Done uploading piece. err: <nil> Dec 31 02:20:49.662285945 (610h51m49.53336898s) [noise=true , address=145.79.227.21:4475 , piece_id=2ZV7HFMVGJABACAR37P64LIQMMPQLIHLJINH26TAGXUPPOI6UVZA, seg_pos={0 0}, upload=535333]: Done uploading piece. err: <nil> Dec 31 02:20:49.729578257 (610h51m49.600661292s) [noise=true , address=145.79.227.2:4114 , piece_id=SWBZCYKTAYJMJNU4NERZJ5DADUD2MXVHNV2WCOJ3GRSZ4CRR3EYQ, seg_pos={0 0}, upload=535333]: Done uploading piece. err: <nil> Dec 31 02:20:49.745322846 (610h51m49.616405881s) [noise=true , address=145.79.227.25:4618 , piece_id=J4G6JVICMMZFHRDIFERBOQALDKMZTJVPCKRIFYNYESD7Q3RUHWWQ, seg_pos={0 0}, upload=535333]: Done uploading piece. err: <nil> Dec 31 02:20:49.947496550 (610h51m49.818579585s) [noise=true , address=45.91.116.6:4975 , piece_id=MODKDYY23IMVG7LVPWIOFIJYQD5ONCRV4RT2XAU2ELUICWGBLYMQ, seg_pos={0 0}, upload=535333]: Done uploading piece. err: <nil> Dec 31 02:20:49.968148674 (610h51m49.839231719s) [noise=true , address=77.90.31.16:4736 , piece_id=ISKP5W7KHCAJZS6XQMILH72XRV3ZGT5NOUG4BMCSQ47J7K5WLO6Q, seg_pos={0 0}, upload=535333]: Done uploading piece. err: <nil> Dec 31 02:20:49.974232399 (610h51m49.845315434s) [noise=true , address=77.90.31.18:4805 , piece_id=ZVTZCD2YNL2JGM65EGGV36NCHQGLV4ARU642RPXSZ5DL2624F5LA, seg_pos={0 0}, upload=535333]: Done uploading piece. err: <nil> Dec 31 02:20:49.997918510 (610h51m49.869001555s) [noise=true , address=45.91.116.8:4047 , piece_id=3JY4CMKNJOJFGR4MABZYQTUBAMFEUGXQMCUMWS3RSMJMKQOZKSRA, seg_pos={0 0}, upload=535333]: Done uploading piece. err: <nil> Dec 31 02:20:50.075045498 (610h51m49.946128563s) [noise=true , address=45.91.116.4:4939 , piece_id=SF3GCQ2FQ5SMTJLYAGEJ6SLDNC3QJX5FHGPPLESKX7RGCMUXCD2A, seg_pos={0 0}, upload=535333]: Done uploading piece. err: <nil> Dec 31 02:20:50.075074864 (610h51m49.946157909s) [seg_pos={0 0}, upload=535333]: Segment reached optimal threshold of 13 pieces. Dec 31 02:20:50.075391197 (610h51m49.946474242s) [noise=true , address=45.91.116.13:4155 , piece_id=WIJOLMCXL7VEF2GLVKLXTFWBFN64HKWEPGJCCS6PYBPKTQ6TZ2DQ, seg_pos={0 0}, upload=535333]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 31 02:20:50.075409983 (610h51m49.946493028s) [noise=true , address=45.91.116.13:4155 , piece_id=WIJOLMCXL7VEF2GLVKLXTFWBFN64HKWEPGJCCS6PYBPKTQ6TZ2DQ, seg_pos={0 0}, upload=535333]: Long tail cancellation due to optimal threshold Dec 31 02:20:50.075460629 (610h51m49.946543674s) [noise=true , address=45.91.116.9:4264 , piece_id=YPANPRX7EIQYGARSDMDEUN3QWERVE6E3XHGPW7SGMVW5SYGVVV7A, seg_pos={0 0}, upload=535333]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 31 02:20:50.075469616 (610h51m49.946552651s) [noise=true , address=45.91.116.9:4264 , piece_id=YPANPRX7EIQYGARSDMDEUN3QWERVE6E3XHGPW7SGMVW5SYGVVV7A, seg_pos={0 0}, upload=535333]: Long tail cancellation due to optimal threshold Dec 31 02:20:50.075604102 (610h51m49.946687147s) [noise=true , address=145.79.227.30:4761 , piece_id=4BLJGIYHEH2HVUFMKXBI2JYV4QWVTS52PQCAWXXMTJFQMPIHFOYQ, seg_pos={0 0}, upload=535333]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 31 02:20:50.075609783 (610h51m49.946692828s) [noise=true , address=145.79.227.30:4761 , piece_id=4BLJGIYHEH2HVUFMKXBI2JYV4QWVTS52PQCAWXXMTJFQMPIHFOYQ, seg_pos={0 0}, upload=535333]: Long tail cancellation due to optimal threshold Dec 31 02:20:50.075657023 (610h51m49.946740058s) [seg_pos={0 0}, upload=535333]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 02:20:50.075747355 (610h51m49.94683039s) [upload=535333]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 02:20:50.075752594 (610h51m49.946835629s) [upload=535333]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 02:20:50.370372256 (610h51m50.241455291s) [upload=535333]: Done uploading segments... Dec 31 02:20:50.370398446 (610h51m50.241481491s) [upload=535333]: Upload finished. err: <nil> Duration: 30.003780788s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...........X..................................X.............] duration: 3.63906749s trace: https://jaeger.dev.storj.io/trace/740aa405bd68150f FAIL (2025-12-31 02:20:42 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.639085224s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [..X..........X............X..X.......X.X..X....XX.XX........] duration: 6.166243976s trace: https://jaeger.dev.storj.io/trace/2f9cb5ff97a2ac44 FAIL (2025-12-31 02:20:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: XGFONRYPHMDUPP7RJLNRRUZ7KXPRFIH22WJKTFRVM7DWZHCR3VSQ) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: H2FC3TA5VGNJTA4ZFS6TCFN2DNWDNNZFR5WRX7QT6RKX4DVY5MIA) context deadline exceeded; piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: T5XK6HTXQQWVPISPFJ4ILOXCMCDD3URLTQZOMX5QZGG5IG5K6GDA) context deadline exceeded; piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: 6RIQ7G6CJOJNGTKLYY7EX4OQ3YBFEJKQAUG5XIJ2NJGJB4ULH2KA) context deadline exceeded ======LOGS====== Dec 31 02:15:34.470943883 (610h46m34.342026918s) [upload=535255]: Starting upload Dec 31 02:15:34.471018445 (610h46m34.34210148s) [upload=535255]: Done starting upload Dec 31 02:15:34.471184521 (610h46m34.342267566s) [upload=535255]: Uploading segments... Dec 31 02:15:34.471391205 (610h46m34.34247424s) [upload=535255]: Got next segment. Inline: false Dec 31 02:15:34.471450537 (610h46m34.342533572s) [upload=535255]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 02:15:34.471455456 (610h46m34.342538501s) [upload=535255]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 02:15:34.726848804 (610h46m34.597931849s) [seg_pos={0 0}, upload=535255]: Begin upload segment... Dec 31 02:15:34.726898429 (610h46m34.597981464s) [seg_pos={0 0}, upload=535255]: Done begin upload segment. Dec 31 02:15:34.726934788 (610h46m34.598017833s) [noise=true , address=145.79.227.16:4909 , piece_id=IEZDHX7O5NVWGLAJ4EQOK4F7RNHJBSFHB6C2TQ42EB7W7JATRMUQ, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.726975706 (610h46m34.598058741s) [noise=true , address=45.91.116.9:4263 , piece_id=6RIQ7G6CJOJNGTKLYY7EX4OQ3YBFEJKQAUG5XIJ2NJGJB4ULH2KA, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.727672262 (610h46m34.598755307s) [noise=true , address=77.90.31.19:4842 , piece_id=TMFJQ7VEVKCVQWYE6LGSF3AXEKYISQPRXDBZGVJUE63FD2EPS4RQ, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.727687450 (610h46m34.598770485s) [noise=true , address=145.79.227.31:4797 , piece_id=VGZA5SE3IY3VSBN3MLHUGPPCD5XYNJXNTCN522BCTKBFLCW34UTQ, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.728136986 (610h46m34.599220022s) [noise=true , address=145.79.227.34:4906 , piece_id=65OFQCDQITDTF4AHGDUZB6TY43UTXH4WFFS3MRI3LQD4M53GL3QQ, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.728259861 (610h46m34.599342906s) [noise=true , address=145.79.227.50:4375 , piece_id=QRAPDLJ3LM6WNYHLZ2VRYO2OIAVSZ3IQ5R62TLBPCAKVDAF2A3FQ, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.728640225 (610h46m34.59972326s) [noise=true , address=145.79.227.32:4835 , piece_id=L2ZTK326XY3DCKHDJHZ25E6MM6E4XN24XLDUBSCQ7OWXWDE34E5Q, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.728737830 (610h46m34.599820865s) [noise=true , address=145.79.227.35:4483 , piece_id=VOUM3VD3PNVSCKK26HQ3EKVE3DFZFQDDXXO3ULEGJKWODTFQZEFA, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.729042310 (610h46m34.600125355s) [noise=true , address=77.90.31.16:4736 , piece_id=CAIXHGWBKNOZ2JGNSN32NRQK6Y4CN5XCGOF32FJG27E4LTFDK4LA, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.729097326 (610h46m34.600180361s) [noise=true , address=145.79.227.21:4475 , piece_id=JTOGYE76I4LQSOP2HJIRROP4IFJESLLV353XQSEXFXJCPYPJE2CA, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.729374553 (610h46m34.600457589s) [noise=true , address=145.79.227.10:4402 , piece_id=XZBGK6UOLSDRIK6ZGH2XIWP2BWSFQK674DKAPZBEWCVD3H5GEBNQ, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.729450758 (610h46m34.600533793s) [noise=true , address=145.79.227.51:4409 , piece_id=WPGRG5MOZR525QAE5HGGSUPRFJ6LKBPQQAJWTMFTK4PPNVKWM4SQ, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.729852623 (610h46m34.600935658s) [noise=true , address=77.90.31.18:4805 , piece_id=XGFONRYPHMDUPP7RJLNRRUZ7KXPRFIH22WJKTFRVM7DWZHCR3VSQ, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.729870658 (610h46m34.600953693s) [noise=true , address=45.91.116.8:4045 , piece_id=H2FC3TA5VGNJTA4ZFS6TCFN2DNWDNNZFR5WRX7QT6RKX4DVY5MIA, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.730183253 (610h46m34.601266298s) [noise=true , address=77.90.31.15:4697 , piece_id=T5XK6HTXQQWVPISPFJ4ILOXCMCDD3URLTQZOMX5QZGG5IG5K6GDA, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.730255150 (610h46m34.601338185s) [noise=true , address=145.79.227.12:4767 , piece_id=3FIFSPGGXTLELWMFF3DVTSETVRY4RTTJ75HERKN3O3PJ4IH3PTOQ, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:36.422393419 (610h46m36.293476464s) [upload=535255]: Next returned nil segment Dec 31 02:15:36.422406284 (610h46m36.293489329s) [upload=535255]: Waiting for error group managing segments... Dec 31 02:15:36.710670932 (610h46m36.581753977s) [noise=true , address=145.79.227.31:4797 , piece_id=VGZA5SE3IY3VSBN3MLHUGPPCD5XYNJXNTCN522BCTKBFLCW34UTQ, seg_pos={0 0}, upload=535255]: Done uploading piece. err: <nil> Dec 31 02:15:36.711270734 (610h46m36.582353769s) [noise=true , address=145.79.227.32:4835 , piece_id=L2ZTK326XY3DCKHDJHZ25E6MM6E4XN24XLDUBSCQ7OWXWDE34E5Q, seg_pos={0 0}, upload=535255]: Done uploading piece. err: <nil> Dec 31 02:15:36.712146010 (610h46m36.583229045s) [noise=true , address=145.79.227.50:4375 , piece_id=QRAPDLJ3LM6WNYHLZ2VRYO2OIAVSZ3IQ5R62TLBPCAKVDAF2A3FQ, seg_pos={0 0}, upload=535255]: Done uploading piece. err: <nil> Dec 31 02:15:36.732635736 (610h46m36.603718771s) [noise=true , address=145.79.227.16:4909 , piece_id=IEZDHX7O5NVWGLAJ4EQOK4F7RNHJBSFHB6C2TQ42EB7W7JATRMUQ, seg_pos={0 0}, upload=535255]: Done uploading piece. err: <nil> Dec 31 02:15:36.737942711 (610h46m36.609025756s) [noise=true , address=145.79.227.21:4475 , piece_id=JTOGYE76I4LQSOP2HJIRROP4IFJESLLV353XQSEXFXJCPYPJE2CA, seg_pos={0 0}, upload=535255]: Done uploading piece. err: <nil> Dec 31 02:15:36.741292659 (610h46m36.612375694s) [noise=true , address=145.79.227.51:4409 , piece_id=WPGRG5MOZR525QAE5HGGSUPRFJ6LKBPQQAJWTMFTK4PPNVKWM4SQ, seg_pos={0 0}, upload=535255]: Done uploading piece. err: <nil> Dec 31 02:15:36.744308642 (610h46m36.615391677s) [noise=true , address=145.79.227.12:4767 , piece_id=3FIFSPGGXTLELWMFF3DVTSETVRY4RTTJ75HERKN3O3PJ4IH3PTOQ, seg_pos={0 0}, upload=535255]: Done uploading piece. err: <nil> Dec 31 02:15:36.795374577 (610h46m36.666457622s) [noise=true , address=145.79.227.10:4402 , piece_id=XZBGK6UOLSDRIK6ZGH2XIWP2BWSFQK674DKAPZBEWCVD3H5GEBNQ, seg_pos={0 0}, upload=535255]: Done uploading piece. err: <nil> Dec 31 02:15:36.881602926 (610h46m36.752685961s) [noise=true , address=77.90.31.18:4805 , piece_id=XGFONRYPHMDUPP7RJLNRRUZ7KXPRFIH22WJKTFRVM7DWZHCR3VSQ, seg_pos={0 0}, upload=535255]: Done uploading piece. err: <nil> Dec 31 02:15:37.071727813 (610h46m36.942810859s) [noise=true , address=45.91.116.9:4263 , piece_id=6RIQ7G6CJOJNGTKLYY7EX4OQ3YBFEJKQAUG5XIJ2NJGJB4ULH2KA, seg_pos={0 0}, upload=535255]: Done uploading piece. err: <nil> Dec 31 02:15:37.579913516 (610h46m37.450996561s) [noise=true , address=77.90.31.16:4736 , piece_id=CAIXHGWBKNOZ2JGNSN32NRQK6Y4CN5XCGOF32FJG27E4LTFDK4LA, seg_pos={0 0}, upload=535255]: Done uploading piece. err: <nil> Dec 31 02:15:37.737075300 (610h46m37.608158335s) [noise=true , address=77.90.31.15:4697 , piece_id=T5XK6HTXQQWVPISPFJ4ILOXCMCDD3URLTQZOMX5QZGG5IG5K6GDA, seg_pos={0 0}, upload=535255]: Done uploading piece. err: <nil> Dec 31 02:15:37.770719692 (610h46m37.641802727s) [noise=true , address=45.91.116.8:4045 , piece_id=H2FC3TA5VGNJTA4ZFS6TCFN2DNWDNNZFR5WRX7QT6RKX4DVY5MIA, seg_pos={0 0}, upload=535255]: Done uploading piece. err: <nil> Dec 31 02:15:37.770744329 (610h46m37.641827364s) [seg_pos={0 0}, upload=535255]: Segment reached optimal threshold of 13 pieces. Dec 31 02:15:37.770987743 (610h46m37.642070788s) [noise=true , address=145.79.227.35:4483 , piece_id=VOUM3VD3PNVSCKK26HQ3EKVE3DFZFQDDXXO3ULEGJKWODTFQZEFA, seg_pos={0 0}, upload=535255]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF Dec 31 02:15:37.770996148 (610h46m37.642079194s) [noise=true , address=145.79.227.35:4483 , piece_id=VOUM3VD3PNVSCKK26HQ3EKVE3DFZFQDDXXO3ULEGJKWODTFQZEFA, seg_pos={0 0}, upload=535255]: Long tail cancellation due to optimal threshold Dec 31 02:15:37.771089406 (610h46m37.642172441s) [noise=true , address=145.79.227.34:4906 , piece_id=65OFQCDQITDTF4AHGDUZB6TY43UTXH4WFFS3MRI3LQD4M53GL3QQ, seg_pos={0 0}, upload=535255]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 31 02:15:37.771104735 (610h46m37.64218778s) [noise=true , address=145.79.227.34:4906 , piece_id=65OFQCDQITDTF4AHGDUZB6TY43UTXH4WFFS3MRI3LQD4M53GL3QQ, seg_pos={0 0}, upload=535255]: Long tail cancellation due to optimal threshold Dec 31 02:15:37.771250514 (610h46m37.642333559s) [noise=true , address=77.90.31.19:4842 , piece_id=TMFJQ7VEVKCVQWYE6LGSF3AXEKYISQPRXDBZGVJUE63FD2EPS4RQ, seg_pos={0 0}, upload=535255]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Dec 31 02:15:37.771256555 (610h46m37.6423396s) [noise=true , address=77.90.31.19:4842 , piece_id=TMFJQ7VEVKCVQWYE6LGSF3AXEKYISQPRXDBZGVJUE63FD2EPS4RQ, seg_pos={0 0}, upload=535255]: Long tail cancellation due to optimal threshold Dec 31 02:15:37.771271012 (610h46m37.642354047s) [seg_pos={0 0}, upload=535255]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 02:15:37.771347367 (610h46m37.642430402s) [upload=535255]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 02:15:37.771351817 (610h46m37.642434852s) [upload=535255]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 02:15:38.072543669 (610h46m37.943626724s) [upload=535255]: Done uploading segments... Dec 31 02:15:38.072571552 (610h46m37.943654587s) [upload=535255]: Upload finished. err: <nil> Duration: 6.166273181s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...X..........................X.............................] duration: 15.337632838s trace: https://jaeger.dev.storj.io/trace/365693fea00df28 FAIL (2025-12-31 02:20:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [noise=true , address=145.79.227.49:4343 , piece_id=6L3HPTS7D2NRA4QEOH4RFM3D3B53CTCONRI6G6YVU4OOTUWTG7XA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.942060847 (610h25m24.813143892s) [noise=true , address=145.79.227.34:4937 , piece_id=5W7LBM3YJJTVUFYTELR57DVXB663OHO5N3Z2SPQPZ4G43IK47O3Q, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.942684104 (610h25m24.813767139s) [noise=true , address=64.49.13.139:4478 , piece_id=QN6RT2KWEA6APQVA66ZMKG4LH537UGYNOXOCVVA2YC2MKDOLT3YA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.943064007 (610h25m24.814147042s) [noise=true , address=145.79.227.40:4084 , piece_id=RUKTKDII6BRUWSYVS65CWUBT3QKG3ASUK3KA3O5VGE53XSG6IMOQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.943459560 (610h25m24.814542605s) [noise=true , address=145.79.227.14:4847 , piece_id=JX5NXTMVM3NSWIEVR4RX46ABSNPKTOFDLFBPKEXFQ3G3ANKV5OLA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.943865303 (610h25m24.814948348s) [noise=true , address=45.91.116.6:4981 , piece_id=CT474ZA2LVAZEBJHBDD4GB6NTHVHZFJLNE5OZV3DFS2NZKVW3EQQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.944259694 (610h25m24.815342749s) [noise=true , address=145.79.227.51:4443 , piece_id=73UUMRDY7BSZNCGBXCZYYRBVE5KLPMNPRNXOZEGWKQGLHJS32J2Q, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.946575182 (610h25m24.817658227s) [noise=true , address=77.90.31.10:4523 , piece_id=P6LIONXJI6AFZ2JTKCQ2W6RZCOZCK6MZ53EXAJKXYQXSX2PX4QEA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.948132909 (610h25m24.819215944s) [noise=true , address=145.79.227.20:4452 , piece_id=ZSDN2JZ3FQUVJYN5OURDXNKEDLDNULJOHKKRZVIPXZIWIYAHCRXQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.949605774 (610h25m24.820688819s) [noise=true , address=77.90.31.14:4680 , piece_id=S2E4WBA7MJVAZLT5JG3IMHK53RQC77CCHV2PDF6L2R2JEA2U3K4Q, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.951459002 (610h25m24.822542048s) [noise=true , address=193.149.164.143:4704 , piece_id=BHXHFBKTLTDMGXWY5QRTP2THZQE5OXFZ2LCMTO2KRYRWAHXWW3NQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.952986742 (610h25m24.824069777s) [noise=true , address=77.90.31.19:4848 , piece_id=VZYG32S6ODIPDKQYZRXECBHXOERZDDWD2OF35Z46W4XMUZ3OJLAA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.956916654 (610h25m24.827999689s) [noise=true , address=145.79.227.29:4971 , piece_id=MEPXHHOQ6SUZC3GIPCQEMOKKF5OJTKZEI3UYZBAYXAQZGSVEKSBA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.960665062 (610h25m24.831748097s) [noise=true , address=64.49.13.170:4720 , piece_id=HIATRSAPE4YLSODFZDPP43ZUMLDVM7GJEVMWE4ZGUQTP25LTHGTA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.962366001 (610h25m24.833449036s) [noise=true , address=145.79.227.2:4117 , piece_id=4SGFGB3IJ7LTXDMMWM2PGGOV64MC46IZ7OZQWH55J4NBQY7CHMZQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.963708216 (610h25m24.834791261s) [noise=true , address=145.79.227.35:4505 , piece_id=V7RS7A4QLZTH4G32ZUZNSF2Z5APVWZGJO77NEGP3SBWQT3WCMVLQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:26.352236138 (610h25m26.223319173s) [upload=534952]: Next returned nil segment Dec 31 01:54:26.352253821 (610h25m26.223336857s) [upload=534952]: Waiting for error group managing segments... Dec 31 01:54:26.812758374 (610h25m26.683841409s) [noise=true , address=145.79.227.51:4443 , piece_id=73UUMRDY7BSZNCGBXCZYYRBVE5KLPMNPRNXOZEGWKQGLHJS32J2Q, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.824371056 (610h25m26.695454101s) [noise=true , address=145.79.227.14:4847 , piece_id=JX5NXTMVM3NSWIEVR4RX46ABSNPKTOFDLFBPKEXFQ3G3ANKV5OLA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.838414046 (610h25m26.709497081s) [noise=true , address=64.49.13.138:4490 , piece_id=2XTUZE6PXRE3OARNJVT36FXXWUMYIBVNE6M2VVFAVBGKEE2H5YZQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.875499841 (610h25m26.746582876s) [noise=true , address=64.49.13.139:4478 , piece_id=QN6RT2KWEA6APQVA66ZMKG4LH537UGYNOXOCVVA2YC2MKDOLT3YA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.891607221 (610h25m26.762690256s) [noise=true , address=145.79.227.20:4452 , piece_id=ZSDN2JZ3FQUVJYN5OURDXNKEDLDNULJOHKKRZVIPXZIWIYAHCRXQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.988561223 (610h25m26.859644268s) [noise=true , address=64.49.13.170:4720 , piece_id=HIATRSAPE4YLSODFZDPP43ZUMLDVM7GJEVMWE4ZGUQTP25LTHGTA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.039035911 (610h25m26.910118946s) [noise=true , address=77.90.31.14:4680 , piece_id=S2E4WBA7MJVAZLT5JG3IMHK53RQC77CCHV2PDF6L2R2JEA2U3K4Q, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.098023090 (610h25m26.969106126s) [noise=true , address=45.91.116.6:4981 , piece_id=CT474ZA2LVAZEBJHBDD4GB6NTHVHZFJLNE5OZV3DFS2NZKVW3EQQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.653790754 (610h25m27.524873789s) [noise=true , address=77.90.31.21:4886 , piece_id=K34WPIBGGGEPHIKJ253XGLDCIQ3WLFN4AZ6EYNWVHHT2EIHS6YRQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.684192972 (610h25m27.555276017s) [noise=true , address=193.149.164.143:4704 , piece_id=BHXHFBKTLTDMGXWY5QRTP2THZQE5OXFZ2LCMTO2KRYRWAHXWW3NQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.717238425 (610h25m27.58832146s) [noise=true , address=145.79.227.40:4084 , piece_id=RUKTKDII6BRUWSYVS65CWUBT3QKG3ASUK3KA3O5VGE53XSG6IMOQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.718349742 (610h25m27.589432777s) [noise=true , address=77.90.31.10:4523 , piece_id=P6LIONXJI6AFZ2JTKCQ2W6RZCOZCK6MZ53EXAJKXYQXSX2PX4QEA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.803639562 (610h25m27.674722597s) [noise=true , address=145.79.227.29:4971 , piece_id=MEPXHHOQ6SUZC3GIPCQEMOKKF5OJTKZEI3UYZBAYXAQZGSVEKSBA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.812791279 (610h25m27.683874314s) [noise=true , address=145.79.227.49:4343 , piece_id=6L3HPTS7D2NRA4QEOH4RFM3D3B53CTCONRI6G6YVU4OOTUWTG7XA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.812812348 (610h25m27.683895393s) [seg_pos={0 0}, upload=534952]: Segment reached optimal threshold of 14 pieces. Dec 31 01:54:27.813018000 (610h25m27.684101035s) [noise=true , address=145.79.227.35:4505 , piece_id=V7RS7A4QLZTH4G32ZUZNSF2Z5APVWZGJO77NEGP3SBWQT3WCMVLQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i6itGoBhLsLiqC8et8Y5L62fLjcWxy18u39NAr4R3nXNMWcsf): protocol: expected piece hash; context canceled; EOF Dec 31 01:54:27.813024772 (610h25m27.684107808s) [noise=true , address=145.79.227.35:4505 , piece_id=V7RS7A4QLZTH4G32ZUZNSF2Z5APVWZGJO77NEGP3SBWQT3WCMVLQ, seg_pos={0 0}, upload=534952]: Long tail cancellation due to optimal threshold Dec 31 01:54:27.813175931 (610h25m27.684258976s) [noise=true , address=145.79.227.34:4937 , piece_id=5W7LBM3YJJTVUFYTELR57DVXB663OHO5N3Z2SPQPZ4G43IK47O3Q, seg_pos={0 0}, upload=534952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1LKrFThdW3jJpG6dRZLUaxLXTa5Wy27yQ8PGUEqjvBWfFoRB94): protocol: expected piece hash; context canceled; EOF Dec 31 01:54:27.813181702 (610h25m27.684264737s) [noise=true , address=145.79.227.34:4937 , piece_id=5W7LBM3YJJTVUFYTELR57DVXB663OHO5N3Z2SPQPZ4G43IK47O3Q, seg_pos={0 0}, upload=534952]: Long tail cancellation due to optimal threshold Dec 31 01:54:27.813407862 (610h25m27.684490907s) [noise=true , address=145.79.227.2:4117 , piece_id=4SGFGB3IJ7LTXDMMWM2PGGOV64MC46IZ7OZQWH55J4NBQY7CHMZQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122Htg72f5eUUutf1p3fYQpy1Wkf6AgdaQV83HHYye2n8zm3the): protocol: expected piece hash; context canceled; EOF Dec 31 01:54:27.813414264 (610h25m27.684497299s) [noise=true , address=145.79.227.2:4117 , piece_id=4SGFGB3IJ7LTXDMMWM2PGGOV64MC46IZ7OZQWH55J4NBQY7CHMZQ, seg_pos={0 0}, upload=534952]: Long tail cancellation due to optimal threshold Dec 31 01:54:27.813424614 (610h25m27.684507649s) [noise=true , address=77.90.31.19:4848 , piece_id=VZYG32S6ODIPDKQYZRXECBHXOERZDDWD2OF35Z46W4XMUZ3OJLAA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD): protocol: write tcp 188.166.225.170:42842->77.90.31.19:4848: use of closed network connection; write tcp 188.166.225.170:42842->77.90.31.19:4848: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42842->77.90.31.19:4848: use of closed network connection Dec 31 01:54:27.813432750 (610h25m27.684515785s) [noise=true , address=77.90.31.19:4848 , piece_id=VZYG32S6ODIPDKQYZRXECBHXOERZDDWD2OF35Z46W4XMUZ3OJLAA, seg_pos={0 0}, upload=534952]: Long tail cancellation due to optimal threshold Dec 31 01:54:27.813474088 (610h25m27.684557123s) [seg_pos={0 0}, upload=534952]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 01:54:27.813535325 (610h25m27.68461836s) [upload=534952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 01:54:27.813539613 (610h25m27.684622648s) [upload=534952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 01:54:28.099921034 (610h25m27.971004069s) [upload=534952]: Done uploading segments... Dec 31 01:54:28.099940451 (610h25m27.971023486s) [upload=534952]: Upload finished. err: <nil> Duration: 15.337662925s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [...X..........X............X..X.......X.X..X....XX.XX.......] duration: 5.919976452s trace: https://jaeger.dev.storj.io/trace/2f9cb5ff97a2ac44 FAIL (2025-12-31 02:20:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: XGFONRYPHMDUPP7RJLNRRUZ7KXPRFIH22WJKTFRVM7DWZHCR3VSQ) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: H2FC3TA5VGNJTA4ZFS6TCFN2DNWDNNZFR5WRX7QT6RKX4DVY5MIA) context deadline exceeded; piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: T5XK6HTXQQWVPISPFJ4ILOXCMCDD3URLTQZOMX5QZGG5IG5K6GDA) context deadline exceeded; piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: 6RIQ7G6CJOJNGTKLYY7EX4OQ3YBFEJKQAUG5XIJ2NJGJB4ULH2KA) context deadline exceeded ======LOGS====== Dec 31 02:15:34.470943883 (610h46m34.342026918s) [upload=535255]: Starting upload Dec 31 02:15:34.471018445 (610h46m34.34210148s) [upload=535255]: Done starting upload Dec 31 02:15:34.471184521 (610h46m34.342267566s) [upload=535255]: Uploading segments... Dec 31 02:15:34.471391205 (610h46m34.34247424s) [upload=535255]: Got next segment. Inline: false Dec 31 02:15:34.471450537 (610h46m34.342533572s) [upload=535255]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 02:15:34.471455456 (610h46m34.342538501s) [upload=535255]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 02:15:34.726848804 (610h46m34.597931849s) [seg_pos={0 0}, upload=535255]: Begin upload segment... Dec 31 02:15:34.726898429 (610h46m34.597981464s) [seg_pos={0 0}, upload=535255]: Done begin upload segment. Dec 31 02:15:34.726934788 (610h46m34.598017833s) [noise=true , address=145.79.227.16:4909 , piece_id=IEZDHX7O5NVWGLAJ4EQOK4F7RNHJBSFHB6C2TQ42EB7W7JATRMUQ, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.726975706 (610h46m34.598058741s) [noise=true , address=45.91.116.9:4263 , piece_id=6RIQ7G6CJOJNGTKLYY7EX4OQ3YBFEJKQAUG5XIJ2NJGJB4ULH2KA, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.727672262 (610h46m34.598755307s) [noise=true , address=77.90.31.19:4842 , piece_id=TMFJQ7VEVKCVQWYE6LGSF3AXEKYISQPRXDBZGVJUE63FD2EPS4RQ, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.727687450 (610h46m34.598770485s) [noise=true , address=145.79.227.31:4797 , piece_id=VGZA5SE3IY3VSBN3MLHUGPPCD5XYNJXNTCN522BCTKBFLCW34UTQ, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.728136986 (610h46m34.599220022s) [noise=true , address=145.79.227.34:4906 , piece_id=65OFQCDQITDTF4AHGDUZB6TY43UTXH4WFFS3MRI3LQD4M53GL3QQ, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.728259861 (610h46m34.599342906s) [noise=true , address=145.79.227.50:4375 , piece_id=QRAPDLJ3LM6WNYHLZ2VRYO2OIAVSZ3IQ5R62TLBPCAKVDAF2A3FQ, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.728640225 (610h46m34.59972326s) [noise=true , address=145.79.227.32:4835 , piece_id=L2ZTK326XY3DCKHDJHZ25E6MM6E4XN24XLDUBSCQ7OWXWDE34E5Q, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.728737830 (610h46m34.599820865s) [noise=true , address=145.79.227.35:4483 , piece_id=VOUM3VD3PNVSCKK26HQ3EKVE3DFZFQDDXXO3ULEGJKWODTFQZEFA, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.729042310 (610h46m34.600125355s) [noise=true , address=77.90.31.16:4736 , piece_id=CAIXHGWBKNOZ2JGNSN32NRQK6Y4CN5XCGOF32FJG27E4LTFDK4LA, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.729097326 (610h46m34.600180361s) [noise=true , address=145.79.227.21:4475 , piece_id=JTOGYE76I4LQSOP2HJIRROP4IFJESLLV353XQSEXFXJCPYPJE2CA, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.729374553 (610h46m34.600457589s) [noise=true , address=145.79.227.10:4402 , piece_id=XZBGK6UOLSDRIK6ZGH2XIWP2BWSFQK674DKAPZBEWCVD3H5GEBNQ, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.729450758 (610h46m34.600533793s) [noise=true , address=145.79.227.51:4409 , piece_id=WPGRG5MOZR525QAE5HGGSUPRFJ6LKBPQQAJWTMFTK4PPNVKWM4SQ, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.729852623 (610h46m34.600935658s) [noise=true , address=77.90.31.18:4805 , piece_id=XGFONRYPHMDUPP7RJLNRRUZ7KXPRFIH22WJKTFRVM7DWZHCR3VSQ, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.729870658 (610h46m34.600953693s) [noise=true , address=45.91.116.8:4045 , piece_id=H2FC3TA5VGNJTA4ZFS6TCFN2DNWDNNZFR5WRX7QT6RKX4DVY5MIA, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.730183253 (610h46m34.601266298s) [noise=true , address=77.90.31.15:4697 , piece_id=T5XK6HTXQQWVPISPFJ4ILOXCMCDD3URLTQZOMX5QZGG5IG5K6GDA, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.730255150 (610h46m34.601338185s) [noise=true , address=145.79.227.12:4767 , piece_id=3FIFSPGGXTLELWMFF3DVTSETVRY4RTTJ75HERKN3O3PJ4IH3PTOQ, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:36.422393419 (610h46m36.293476464s) [upload=535255]: Next returned nil segment Dec 31 02:15:36.422406284 (610h46m36.293489329s) [upload=535255]: Waiting for error group managing segments... Dec 31 02:15:36.710670932 (610h46m36.581753977s) [noise=true , address=145.79.227.31:4797 , piece_id=VGZA5SE3IY3VSBN3MLHUGPPCD5XYNJXNTCN522BCTKBFLCW34UTQ, seg_pos={0 0}, upload=535255]: Done uploading piece. err: <nil> Dec 31 02:15:36.711270734 (610h46m36.582353769s) [noise=true , address=145.79.227.32:4835 , piece_id=L2ZTK326XY3DCKHDJHZ25E6MM6E4XN24XLDUBSCQ7OWXWDE34E5Q, seg_pos={0 0}, upload=535255]: Done uploading piece. err: <nil> Dec 31 02:15:36.712146010 (610h46m36.583229045s) [noise=true , address=145.79.227.50:4375 , piece_id=QRAPDLJ3LM6WNYHLZ2VRYO2OIAVSZ3IQ5R62TLBPCAKVDAF2A3FQ, seg_pos={0 0}, upload=535255]: Done uploading piece. err: <nil> Dec 31 02:15:36.732635736 (610h46m36.603718771s) [noise=true , address=145.79.227.16:4909 , piece_id=IEZDHX7O5NVWGLAJ4EQOK4F7RNHJBSFHB6C2TQ42EB7W7JATRMUQ, seg_pos={0 0}, upload=535255]: Done uploading piece. err: <nil> Dec 31 02:15:36.737942711 (610h46m36.609025756s) [noise=true , address=145.79.227.21:4475 , piece_id=JTOGYE76I4LQSOP2HJIRROP4IFJESLLV353XQSEXFXJCPYPJE2CA, seg_pos={0 0}, upload=535255]: Done uploading piece. err: <nil> Dec 31 02:15:36.741292659 (610h46m36.612375694s) [noise=true , address=145.79.227.51:4409 , piece_id=WPGRG5MOZR525QAE5HGGSUPRFJ6LKBPQQAJWTMFTK4PPNVKWM4SQ, seg_pos={0 0}, upload=535255]: Done uploading piece. err: <nil> Dec 31 02:15:36.744308642 (610h46m36.615391677s) [noise=true , address=145.79.227.12:4767 , piece_id=3FIFSPGGXTLELWMFF3DVTSETVRY4RTTJ75HERKN3O3PJ4IH3PTOQ, seg_pos={0 0}, upload=535255]: Done uploading piece. err: <nil> Dec 31 02:15:36.795374577 (610h46m36.666457622s) [noise=true , address=145.79.227.10:4402 , piece_id=XZBGK6UOLSDRIK6ZGH2XIWP2BWSFQK674DKAPZBEWCVD3H5GEBNQ, seg_pos={0 0}, upload=535255]: Done uploading piece. err: <nil> Dec 31 02:15:36.881602926 (610h46m36.752685961s) [noise=true , address=77.90.31.18:4805 , piece_id=XGFONRYPHMDUPP7RJLNRRUZ7KXPRFIH22WJKTFRVM7DWZHCR3VSQ, seg_pos={0 0}, upload=535255]: Done uploading piece. err: <nil> Dec 31 02:15:37.071727813 (610h46m36.942810859s) [noise=true , address=45.91.116.9:4263 , piece_id=6RIQ7G6CJOJNGTKLYY7EX4OQ3YBFEJKQAUG5XIJ2NJGJB4ULH2KA, seg_pos={0 0}, upload=535255]: Done uploading piece. err: <nil> Dec 31 02:15:37.579913516 (610h46m37.450996561s) [noise=true , address=77.90.31.16:4736 , piece_id=CAIXHGWBKNOZ2JGNSN32NRQK6Y4CN5XCGOF32FJG27E4LTFDK4LA, seg_pos={0 0}, upload=535255]: Done uploading piece. err: <nil> Dec 31 02:15:37.737075300 (610h46m37.608158335s) [noise=true , address=77.90.31.15:4697 , piece_id=T5XK6HTXQQWVPISPFJ4ILOXCMCDD3URLTQZOMX5QZGG5IG5K6GDA, seg_pos={0 0}, upload=535255]: Done uploading piece. err: <nil> Dec 31 02:15:37.770719692 (610h46m37.641802727s) [noise=true , address=45.91.116.8:4045 , piece_id=H2FC3TA5VGNJTA4ZFS6TCFN2DNWDNNZFR5WRX7QT6RKX4DVY5MIA, seg_pos={0 0}, upload=535255]: Done uploading piece. err: <nil> Dec 31 02:15:37.770744329 (610h46m37.641827364s) [seg_pos={0 0}, upload=535255]: Segment reached optimal threshold of 13 pieces. Dec 31 02:15:37.770987743 (610h46m37.642070788s) [noise=true , address=145.79.227.35:4483 , piece_id=VOUM3VD3PNVSCKK26HQ3EKVE3DFZFQDDXXO3ULEGJKWODTFQZEFA, seg_pos={0 0}, upload=535255]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF Dec 31 02:15:37.770996148 (610h46m37.642079194s) [noise=true , address=145.79.227.35:4483 , piece_id=VOUM3VD3PNVSCKK26HQ3EKVE3DFZFQDDXXO3ULEGJKWODTFQZEFA, seg_pos={0 0}, upload=535255]: Long tail cancellation due to optimal threshold Dec 31 02:15:37.771089406 (610h46m37.642172441s) [noise=true , address=145.79.227.34:4906 , piece_id=65OFQCDQITDTF4AHGDUZB6TY43UTXH4WFFS3MRI3LQD4M53GL3QQ, seg_pos={0 0}, upload=535255]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 31 02:15:37.771104735 (610h46m37.64218778s) [noise=true , address=145.79.227.34:4906 , piece_id=65OFQCDQITDTF4AHGDUZB6TY43UTXH4WFFS3MRI3LQD4M53GL3QQ, seg_pos={0 0}, upload=535255]: Long tail cancellation due to optimal threshold Dec 31 02:15:37.771250514 (610h46m37.642333559s) [noise=true , address=77.90.31.19:4842 , piece_id=TMFJQ7VEVKCVQWYE6LGSF3AXEKYISQPRXDBZGVJUE63FD2EPS4RQ, seg_pos={0 0}, upload=535255]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Dec 31 02:15:37.771256555 (610h46m37.6423396s) [noise=true , address=77.90.31.19:4842 , piece_id=TMFJQ7VEVKCVQWYE6LGSF3AXEKYISQPRXDBZGVJUE63FD2EPS4RQ, seg_pos={0 0}, upload=535255]: Long tail cancellation due to optimal threshold Dec 31 02:15:37.771271012 (610h46m37.642354047s) [seg_pos={0 0}, upload=535255]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 02:15:37.771347367 (610h46m37.642430402s) [upload=535255]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 02:15:37.771351817 (610h46m37.642434852s) [upload=535255]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 02:15:38.072543669 (610h46m37.943626724s) [upload=535255]: Done uploading segments... Dec 31 02:15:38.072571552 (610h46m37.943654587s) [upload=535255]: Upload finished. err: <nil> Duration: 5.920031878s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [............X..................................X............] duration: 3.565945497s trace: https://jaeger.dev.storj.io/trace/740aa405bd68150f FAIL (2025-12-31 02:19:56 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.566000493s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [....X..........................X............................] duration: 52.314363174s trace: https://jaeger.dev.storj.io/trace/365693fea00df28 FAIL (2025-12-31 02:19:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [noise=true , address=145.79.227.49:4343 , piece_id=6L3HPTS7D2NRA4QEOH4RFM3D3B53CTCONRI6G6YVU4OOTUWTG7XA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.942060847 (610h25m24.813143892s) [noise=true , address=145.79.227.34:4937 , piece_id=5W7LBM3YJJTVUFYTELR57DVXB663OHO5N3Z2SPQPZ4G43IK47O3Q, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.942684104 (610h25m24.813767139s) [noise=true , address=64.49.13.139:4478 , piece_id=QN6RT2KWEA6APQVA66ZMKG4LH537UGYNOXOCVVA2YC2MKDOLT3YA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.943064007 (610h25m24.814147042s) [noise=true , address=145.79.227.40:4084 , piece_id=RUKTKDII6BRUWSYVS65CWUBT3QKG3ASUK3KA3O5VGE53XSG6IMOQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.943459560 (610h25m24.814542605s) [noise=true , address=145.79.227.14:4847 , piece_id=JX5NXTMVM3NSWIEVR4RX46ABSNPKTOFDLFBPKEXFQ3G3ANKV5OLA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.943865303 (610h25m24.814948348s) [noise=true , address=45.91.116.6:4981 , piece_id=CT474ZA2LVAZEBJHBDD4GB6NTHVHZFJLNE5OZV3DFS2NZKVW3EQQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.944259694 (610h25m24.815342749s) [noise=true , address=145.79.227.51:4443 , piece_id=73UUMRDY7BSZNCGBXCZYYRBVE5KLPMNPRNXOZEGWKQGLHJS32J2Q, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.946575182 (610h25m24.817658227s) [noise=true , address=77.90.31.10:4523 , piece_id=P6LIONXJI6AFZ2JTKCQ2W6RZCOZCK6MZ53EXAJKXYQXSX2PX4QEA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.948132909 (610h25m24.819215944s) [noise=true , address=145.79.227.20:4452 , piece_id=ZSDN2JZ3FQUVJYN5OURDXNKEDLDNULJOHKKRZVIPXZIWIYAHCRXQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.949605774 (610h25m24.820688819s) [noise=true , address=77.90.31.14:4680 , piece_id=S2E4WBA7MJVAZLT5JG3IMHK53RQC77CCHV2PDF6L2R2JEA2U3K4Q, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.951459002 (610h25m24.822542048s) [noise=true , address=193.149.164.143:4704 , piece_id=BHXHFBKTLTDMGXWY5QRTP2THZQE5OXFZ2LCMTO2KRYRWAHXWW3NQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.952986742 (610h25m24.824069777s) [noise=true , address=77.90.31.19:4848 , piece_id=VZYG32S6ODIPDKQYZRXECBHXOERZDDWD2OF35Z46W4XMUZ3OJLAA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.956916654 (610h25m24.827999689s) [noise=true , address=145.79.227.29:4971 , piece_id=MEPXHHOQ6SUZC3GIPCQEMOKKF5OJTKZEI3UYZBAYXAQZGSVEKSBA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.960665062 (610h25m24.831748097s) [noise=true , address=64.49.13.170:4720 , piece_id=HIATRSAPE4YLSODFZDPP43ZUMLDVM7GJEVMWE4ZGUQTP25LTHGTA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.962366001 (610h25m24.833449036s) [noise=true , address=145.79.227.2:4117 , piece_id=4SGFGB3IJ7LTXDMMWM2PGGOV64MC46IZ7OZQWH55J4NBQY7CHMZQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.963708216 (610h25m24.834791261s) [noise=true , address=145.79.227.35:4505 , piece_id=V7RS7A4QLZTH4G32ZUZNSF2Z5APVWZGJO77NEGP3SBWQT3WCMVLQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:26.352236138 (610h25m26.223319173s) [upload=534952]: Next returned nil segment Dec 31 01:54:26.352253821 (610h25m26.223336857s) [upload=534952]: Waiting for error group managing segments... Dec 31 01:54:26.812758374 (610h25m26.683841409s) [noise=true , address=145.79.227.51:4443 , piece_id=73UUMRDY7BSZNCGBXCZYYRBVE5KLPMNPRNXOZEGWKQGLHJS32J2Q, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.824371056 (610h25m26.695454101s) [noise=true , address=145.79.227.14:4847 , piece_id=JX5NXTMVM3NSWIEVR4RX46ABSNPKTOFDLFBPKEXFQ3G3ANKV5OLA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.838414046 (610h25m26.709497081s) [noise=true , address=64.49.13.138:4490 , piece_id=2XTUZE6PXRE3OARNJVT36FXXWUMYIBVNE6M2VVFAVBGKEE2H5YZQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.875499841 (610h25m26.746582876s) [noise=true , address=64.49.13.139:4478 , piece_id=QN6RT2KWEA6APQVA66ZMKG4LH537UGYNOXOCVVA2YC2MKDOLT3YA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.891607221 (610h25m26.762690256s) [noise=true , address=145.79.227.20:4452 , piece_id=ZSDN2JZ3FQUVJYN5OURDXNKEDLDNULJOHKKRZVIPXZIWIYAHCRXQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.988561223 (610h25m26.859644268s) [noise=true , address=64.49.13.170:4720 , piece_id=HIATRSAPE4YLSODFZDPP43ZUMLDVM7GJEVMWE4ZGUQTP25LTHGTA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.039035911 (610h25m26.910118946s) [noise=true , address=77.90.31.14:4680 , piece_id=S2E4WBA7MJVAZLT5JG3IMHK53RQC77CCHV2PDF6L2R2JEA2U3K4Q, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.098023090 (610h25m26.969106126s) [noise=true , address=45.91.116.6:4981 , piece_id=CT474ZA2LVAZEBJHBDD4GB6NTHVHZFJLNE5OZV3DFS2NZKVW3EQQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.653790754 (610h25m27.524873789s) [noise=true , address=77.90.31.21:4886 , piece_id=K34WPIBGGGEPHIKJ253XGLDCIQ3WLFN4AZ6EYNWVHHT2EIHS6YRQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.684192972 (610h25m27.555276017s) [noise=true , address=193.149.164.143:4704 , piece_id=BHXHFBKTLTDMGXWY5QRTP2THZQE5OXFZ2LCMTO2KRYRWAHXWW3NQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.717238425 (610h25m27.58832146s) [noise=true , address=145.79.227.40:4084 , piece_id=RUKTKDII6BRUWSYVS65CWUBT3QKG3ASUK3KA3O5VGE53XSG6IMOQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.718349742 (610h25m27.589432777s) [noise=true , address=77.90.31.10:4523 , piece_id=P6LIONXJI6AFZ2JTKCQ2W6RZCOZCK6MZ53EXAJKXYQXSX2PX4QEA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.803639562 (610h25m27.674722597s) [noise=true , address=145.79.227.29:4971 , piece_id=MEPXHHOQ6SUZC3GIPCQEMOKKF5OJTKZEI3UYZBAYXAQZGSVEKSBA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.812791279 (610h25m27.683874314s) [noise=true , address=145.79.227.49:4343 , piece_id=6L3HPTS7D2NRA4QEOH4RFM3D3B53CTCONRI6G6YVU4OOTUWTG7XA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.812812348 (610h25m27.683895393s) [seg_pos={0 0}, upload=534952]: Segment reached optimal threshold of 14 pieces. Dec 31 01:54:27.813018000 (610h25m27.684101035s) [noise=true , address=145.79.227.35:4505 , piece_id=V7RS7A4QLZTH4G32ZUZNSF2Z5APVWZGJO77NEGP3SBWQT3WCMVLQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i6itGoBhLsLiqC8et8Y5L62fLjcWxy18u39NAr4R3nXNMWcsf): protocol: expected piece hash; context canceled; EOF Dec 31 01:54:27.813024772 (610h25m27.684107808s) [noise=true , address=145.79.227.35:4505 , piece_id=V7RS7A4QLZTH4G32ZUZNSF2Z5APVWZGJO77NEGP3SBWQT3WCMVLQ, seg_pos={0 0}, upload=534952]: Long tail cancellation due to optimal threshold Dec 31 01:54:27.813175931 (610h25m27.684258976s) [noise=true , address=145.79.227.34:4937 , piece_id=5W7LBM3YJJTVUFYTELR57DVXB663OHO5N3Z2SPQPZ4G43IK47O3Q, seg_pos={0 0}, upload=534952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1LKrFThdW3jJpG6dRZLUaxLXTa5Wy27yQ8PGUEqjvBWfFoRB94): protocol: expected piece hash; context canceled; EOF Dec 31 01:54:27.813181702 (610h25m27.684264737s) [noise=true , address=145.79.227.34:4937 , piece_id=5W7LBM3YJJTVUFYTELR57DVXB663OHO5N3Z2SPQPZ4G43IK47O3Q, seg_pos={0 0}, upload=534952]: Long tail cancellation due to optimal threshold Dec 31 01:54:27.813407862 (610h25m27.684490907s) [noise=true , address=145.79.227.2:4117 , piece_id=4SGFGB3IJ7LTXDMMWM2PGGOV64MC46IZ7OZQWH55J4NBQY7CHMZQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122Htg72f5eUUutf1p3fYQpy1Wkf6AgdaQV83HHYye2n8zm3the): protocol: expected piece hash; context canceled; EOF Dec 31 01:54:27.813414264 (610h25m27.684497299s) [noise=true , address=145.79.227.2:4117 , piece_id=4SGFGB3IJ7LTXDMMWM2PGGOV64MC46IZ7OZQWH55J4NBQY7CHMZQ, seg_pos={0 0}, upload=534952]: Long tail cancellation due to optimal threshold Dec 31 01:54:27.813424614 (610h25m27.684507649s) [noise=true , address=77.90.31.19:4848 , piece_id=VZYG32S6ODIPDKQYZRXECBHXOERZDDWD2OF35Z46W4XMUZ3OJLAA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD): protocol: write tcp 188.166.225.170:42842->77.90.31.19:4848: use of closed network connection; write tcp 188.166.225.170:42842->77.90.31.19:4848: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42842->77.90.31.19:4848: use of closed network connection Dec 31 01:54:27.813432750 (610h25m27.684515785s) [noise=true , address=77.90.31.19:4848 , piece_id=VZYG32S6ODIPDKQYZRXECBHXOERZDDWD2OF35Z46W4XMUZ3OJLAA, seg_pos={0 0}, upload=534952]: Long tail cancellation due to optimal threshold Dec 31 01:54:27.813474088 (610h25m27.684557123s) [seg_pos={0 0}, upload=534952]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 01:54:27.813535325 (610h25m27.68461836s) [upload=534952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 01:54:27.813539613 (610h25m27.684622648s) [upload=534952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 01:54:28.099921034 (610h25m27.971004069s) [upload=534952]: Done uploading segments... Dec 31 01:54:28.099940451 (610h25m27.971023486s) [upload=534952]: Upload finished. err: <nil> Duration: 52.314389483s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [....X..........X............X..X.......X.X..X....XX.XX......] duration: 6.077189985s trace: https://jaeger.dev.storj.io/trace/2f9cb5ff97a2ac44 FAIL (2025-12-31 02:19:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: XGFONRYPHMDUPP7RJLNRRUZ7KXPRFIH22WJKTFRVM7DWZHCR3VSQ) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: H2FC3TA5VGNJTA4ZFS6TCFN2DNWDNNZFR5WRX7QT6RKX4DVY5MIA) context deadline exceeded; piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: T5XK6HTXQQWVPISPFJ4ILOXCMCDD3URLTQZOMX5QZGG5IG5K6GDA) context deadline exceeded; piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: 6RIQ7G6CJOJNGTKLYY7EX4OQ3YBFEJKQAUG5XIJ2NJGJB4ULH2KA) context deadline exceeded ======LOGS====== Dec 31 02:15:34.470943883 (610h46m34.342026918s) [upload=535255]: Starting upload Dec 31 02:15:34.471018445 (610h46m34.34210148s) [upload=535255]: Done starting upload Dec 31 02:15:34.471184521 (610h46m34.342267566s) [upload=535255]: Uploading segments... Dec 31 02:15:34.471391205 (610h46m34.34247424s) [upload=535255]: Got next segment. Inline: false Dec 31 02:15:34.471450537 (610h46m34.342533572s) [upload=535255]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 02:15:34.471455456 (610h46m34.342538501s) [upload=535255]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 02:15:34.726848804 (610h46m34.597931849s) [seg_pos={0 0}, upload=535255]: Begin upload segment... Dec 31 02:15:34.726898429 (610h46m34.597981464s) [seg_pos={0 0}, upload=535255]: Done begin upload segment. Dec 31 02:15:34.726934788 (610h46m34.598017833s) [noise=true , address=145.79.227.16:4909 , piece_id=IEZDHX7O5NVWGLAJ4EQOK4F7RNHJBSFHB6C2TQ42EB7W7JATRMUQ, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.726975706 (610h46m34.598058741s) [noise=true , address=45.91.116.9:4263 , piece_id=6RIQ7G6CJOJNGTKLYY7EX4OQ3YBFEJKQAUG5XIJ2NJGJB4ULH2KA, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.727672262 (610h46m34.598755307s) [noise=true , address=77.90.31.19:4842 , piece_id=TMFJQ7VEVKCVQWYE6LGSF3AXEKYISQPRXDBZGVJUE63FD2EPS4RQ, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.727687450 (610h46m34.598770485s) [noise=true , address=145.79.227.31:4797 , piece_id=VGZA5SE3IY3VSBN3MLHUGPPCD5XYNJXNTCN522BCTKBFLCW34UTQ, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.728136986 (610h46m34.599220022s) [noise=true , address=145.79.227.34:4906 , piece_id=65OFQCDQITDTF4AHGDUZB6TY43UTXH4WFFS3MRI3LQD4M53GL3QQ, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.728259861 (610h46m34.599342906s) [noise=true , address=145.79.227.50:4375 , piece_id=QRAPDLJ3LM6WNYHLZ2VRYO2OIAVSZ3IQ5R62TLBPCAKVDAF2A3FQ, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.728640225 (610h46m34.59972326s) [noise=true , address=145.79.227.32:4835 , piece_id=L2ZTK326XY3DCKHDJHZ25E6MM6E4XN24XLDUBSCQ7OWXWDE34E5Q, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.728737830 (610h46m34.599820865s) [noise=true , address=145.79.227.35:4483 , piece_id=VOUM3VD3PNVSCKK26HQ3EKVE3DFZFQDDXXO3ULEGJKWODTFQZEFA, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.729042310 (610h46m34.600125355s) [noise=true , address=77.90.31.16:4736 , piece_id=CAIXHGWBKNOZ2JGNSN32NRQK6Y4CN5XCGOF32FJG27E4LTFDK4LA, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.729097326 (610h46m34.600180361s) [noise=true , address=145.79.227.21:4475 , piece_id=JTOGYE76I4LQSOP2HJIRROP4IFJESLLV353XQSEXFXJCPYPJE2CA, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.729374553 (610h46m34.600457589s) [noise=true , address=145.79.227.10:4402 , piece_id=XZBGK6UOLSDRIK6ZGH2XIWP2BWSFQK674DKAPZBEWCVD3H5GEBNQ, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.729450758 (610h46m34.600533793s) [noise=true , address=145.79.227.51:4409 , piece_id=WPGRG5MOZR525QAE5HGGSUPRFJ6LKBPQQAJWTMFTK4PPNVKWM4SQ, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.729852623 (610h46m34.600935658s) [noise=true , address=77.90.31.18:4805 , piece_id=XGFONRYPHMDUPP7RJLNRRUZ7KXPRFIH22WJKTFRVM7DWZHCR3VSQ, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.729870658 (610h46m34.600953693s) [noise=true , address=45.91.116.8:4045 , piece_id=H2FC3TA5VGNJTA4ZFS6TCFN2DNWDNNZFR5WRX7QT6RKX4DVY5MIA, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.730183253 (610h46m34.601266298s) [noise=true , address=77.90.31.15:4697 , piece_id=T5XK6HTXQQWVPISPFJ4ILOXCMCDD3URLTQZOMX5QZGG5IG5K6GDA, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.730255150 (610h46m34.601338185s) [noise=true , address=145.79.227.12:4767 , piece_id=3FIFSPGGXTLELWMFF3DVTSETVRY4RTTJ75HERKN3O3PJ4IH3PTOQ, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:36.422393419 (610h46m36.293476464s) [upload=535255]: Next returned nil segment Dec 31 02:15:36.422406284 (610h46m36.293489329s) [upload=535255]: Waiting for error group managing segments... Dec 31 02:15:36.710670932 (610h46m36.581753977s) [noise=true , address=145.79.227.31:4797 , piece_id=VGZA5SE3IY3VSBN3MLHUGPPCD5XYNJXNTCN522BCTKBFLCW34UTQ, seg_pos={0 0}, upload=535255]: Done uploading piece. err: <nil> Dec 31 02:15:36.711270734 (610h46m36.582353769s) [noise=true , address=145.79.227.32:4835 , piece_id=L2ZTK326XY3DCKHDJHZ25E6MM6E4XN24XLDUBSCQ7OWXWDE34E5Q, seg_pos={0 0}, upload=535255]: Done uploading piece. err: <nil> Dec 31 02:15:36.712146010 (610h46m36.583229045s) [noise=true , address=145.79.227.50:4375 , piece_id=QRAPDLJ3LM6WNYHLZ2VRYO2OIAVSZ3IQ5R62TLBPCAKVDAF2A3FQ, seg_pos={0 0}, upload=535255]: Done uploading piece. err: <nil> Dec 31 02:15:36.732635736 (610h46m36.603718771s) [noise=true , address=145.79.227.16:4909 , piece_id=IEZDHX7O5NVWGLAJ4EQOK4F7RNHJBSFHB6C2TQ42EB7W7JATRMUQ, seg_pos={0 0}, upload=535255]: Done uploading piece. err: <nil> Dec 31 02:15:36.737942711 (610h46m36.609025756s) [noise=true , address=145.79.227.21:4475 , piece_id=JTOGYE76I4LQSOP2HJIRROP4IFJESLLV353XQSEXFXJCPYPJE2CA, seg_pos={0 0}, upload=535255]: Done uploading piece. err: <nil> Dec 31 02:15:36.741292659 (610h46m36.612375694s) [noise=true , address=145.79.227.51:4409 , piece_id=WPGRG5MOZR525QAE5HGGSUPRFJ6LKBPQQAJWTMFTK4PPNVKWM4SQ, seg_pos={0 0}, upload=535255]: Done uploading piece. err: <nil> Dec 31 02:15:36.744308642 (610h46m36.615391677s) [noise=true , address=145.79.227.12:4767 , piece_id=3FIFSPGGXTLELWMFF3DVTSETVRY4RTTJ75HERKN3O3PJ4IH3PTOQ, seg_pos={0 0}, upload=535255]: Done uploading piece. err: <nil> Dec 31 02:15:36.795374577 (610h46m36.666457622s) [noise=true , address=145.79.227.10:4402 , piece_id=XZBGK6UOLSDRIK6ZGH2XIWP2BWSFQK674DKAPZBEWCVD3H5GEBNQ, seg_pos={0 0}, upload=535255]: Done uploading piece. err: <nil> Dec 31 02:15:36.881602926 (610h46m36.752685961s) [noise=true , address=77.90.31.18:4805 , piece_id=XGFONRYPHMDUPP7RJLNRRUZ7KXPRFIH22WJKTFRVM7DWZHCR3VSQ, seg_pos={0 0}, upload=535255]: Done uploading piece. err: <nil> Dec 31 02:15:37.071727813 (610h46m36.942810859s) [noise=true , address=45.91.116.9:4263 , piece_id=6RIQ7G6CJOJNGTKLYY7EX4OQ3YBFEJKQAUG5XIJ2NJGJB4ULH2KA, seg_pos={0 0}, upload=535255]: Done uploading piece. err: <nil> Dec 31 02:15:37.579913516 (610h46m37.450996561s) [noise=true , address=77.90.31.16:4736 , piece_id=CAIXHGWBKNOZ2JGNSN32NRQK6Y4CN5XCGOF32FJG27E4LTFDK4LA, seg_pos={0 0}, upload=535255]: Done uploading piece. err: <nil> Dec 31 02:15:37.737075300 (610h46m37.608158335s) [noise=true , address=77.90.31.15:4697 , piece_id=T5XK6HTXQQWVPISPFJ4ILOXCMCDD3URLTQZOMX5QZGG5IG5K6GDA, seg_pos={0 0}, upload=535255]: Done uploading piece. err: <nil> Dec 31 02:15:37.770719692 (610h46m37.641802727s) [noise=true , address=45.91.116.8:4045 , piece_id=H2FC3TA5VGNJTA4ZFS6TCFN2DNWDNNZFR5WRX7QT6RKX4DVY5MIA, seg_pos={0 0}, upload=535255]: Done uploading piece. err: <nil> Dec 31 02:15:37.770744329 (610h46m37.641827364s) [seg_pos={0 0}, upload=535255]: Segment reached optimal threshold of 13 pieces. Dec 31 02:15:37.770987743 (610h46m37.642070788s) [noise=true , address=145.79.227.35:4483 , piece_id=VOUM3VD3PNVSCKK26HQ3EKVE3DFZFQDDXXO3ULEGJKWODTFQZEFA, seg_pos={0 0}, upload=535255]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF Dec 31 02:15:37.770996148 (610h46m37.642079194s) [noise=true , address=145.79.227.35:4483 , piece_id=VOUM3VD3PNVSCKK26HQ3EKVE3DFZFQDDXXO3ULEGJKWODTFQZEFA, seg_pos={0 0}, upload=535255]: Long tail cancellation due to optimal threshold Dec 31 02:15:37.771089406 (610h46m37.642172441s) [noise=true , address=145.79.227.34:4906 , piece_id=65OFQCDQITDTF4AHGDUZB6TY43UTXH4WFFS3MRI3LQD4M53GL3QQ, seg_pos={0 0}, upload=535255]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 31 02:15:37.771104735 (610h46m37.64218778s) [noise=true , address=145.79.227.34:4906 , piece_id=65OFQCDQITDTF4AHGDUZB6TY43UTXH4WFFS3MRI3LQD4M53GL3QQ, seg_pos={0 0}, upload=535255]: Long tail cancellation due to optimal threshold Dec 31 02:15:37.771250514 (610h46m37.642333559s) [noise=true , address=77.90.31.19:4842 , piece_id=TMFJQ7VEVKCVQWYE6LGSF3AXEKYISQPRXDBZGVJUE63FD2EPS4RQ, seg_pos={0 0}, upload=535255]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Dec 31 02:15:37.771256555 (610h46m37.6423396s) [noise=true , address=77.90.31.19:4842 , piece_id=TMFJQ7VEVKCVQWYE6LGSF3AXEKYISQPRXDBZGVJUE63FD2EPS4RQ, seg_pos={0 0}, upload=535255]: Long tail cancellation due to optimal threshold Dec 31 02:15:37.771271012 (610h46m37.642354047s) [seg_pos={0 0}, upload=535255]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 02:15:37.771347367 (610h46m37.642430402s) [upload=535255]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 02:15:37.771351817 (610h46m37.642434852s) [upload=535255]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 02:15:38.072543669 (610h46m37.943626724s) [upload=535255]: Done uploading segments... Dec 31 02:15:38.072571552 (610h46m37.943654587s) [upload=535255]: Upload finished. err: <nil> Duration: 6.077217509s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.............X..................................X...........] duration: 3.766819172s trace: https://jaeger.dev.storj.io/trace/740aa405bd68150f FAIL (2025-12-31 02:19:02 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.76684484s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [X....X..........X............X..X.......X.X..X....XX.XX.....] duration: 5.878234489s trace: https://jaeger.dev.storj.io/trace/2f9cb5ff97a2ac44 FAIL (2025-12-31 02:18:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: XGFONRYPHMDUPP7RJLNRRUZ7KXPRFIH22WJKTFRVM7DWZHCR3VSQ) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: H2FC3TA5VGNJTA4ZFS6TCFN2DNWDNNZFR5WRX7QT6RKX4DVY5MIA) context deadline exceeded; piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: T5XK6HTXQQWVPISPFJ4ILOXCMCDD3URLTQZOMX5QZGG5IG5K6GDA) context deadline exceeded; piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: 6RIQ7G6CJOJNGTKLYY7EX4OQ3YBFEJKQAUG5XIJ2NJGJB4ULH2KA) context deadline exceeded ======LOGS====== Dec 31 02:15:34.470943883 (610h46m34.342026918s) [upload=535255]: Starting upload Dec 31 02:15:34.471018445 (610h46m34.34210148s) [upload=535255]: Done starting upload Dec 31 02:15:34.471184521 (610h46m34.342267566s) [upload=535255]: Uploading segments... Dec 31 02:15:34.471391205 (610h46m34.34247424s) [upload=535255]: Got next segment. Inline: false Dec 31 02:15:34.471450537 (610h46m34.342533572s) [upload=535255]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 02:15:34.471455456 (610h46m34.342538501s) [upload=535255]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 02:15:34.726848804 (610h46m34.597931849s) [seg_pos={0 0}, upload=535255]: Begin upload segment... Dec 31 02:15:34.726898429 (610h46m34.597981464s) [seg_pos={0 0}, upload=535255]: Done begin upload segment. Dec 31 02:15:34.726934788 (610h46m34.598017833s) [noise=true , address=145.79.227.16:4909 , piece_id=IEZDHX7O5NVWGLAJ4EQOK4F7RNHJBSFHB6C2TQ42EB7W7JATRMUQ, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.726975706 (610h46m34.598058741s) [noise=true , address=45.91.116.9:4263 , piece_id=6RIQ7G6CJOJNGTKLYY7EX4OQ3YBFEJKQAUG5XIJ2NJGJB4ULH2KA, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.727672262 (610h46m34.598755307s) [noise=true , address=77.90.31.19:4842 , piece_id=TMFJQ7VEVKCVQWYE6LGSF3AXEKYISQPRXDBZGVJUE63FD2EPS4RQ, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.727687450 (610h46m34.598770485s) [noise=true , address=145.79.227.31:4797 , piece_id=VGZA5SE3IY3VSBN3MLHUGPPCD5XYNJXNTCN522BCTKBFLCW34UTQ, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.728136986 (610h46m34.599220022s) [noise=true , address=145.79.227.34:4906 , piece_id=65OFQCDQITDTF4AHGDUZB6TY43UTXH4WFFS3MRI3LQD4M53GL3QQ, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.728259861 (610h46m34.599342906s) [noise=true , address=145.79.227.50:4375 , piece_id=QRAPDLJ3LM6WNYHLZ2VRYO2OIAVSZ3IQ5R62TLBPCAKVDAF2A3FQ, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.728640225 (610h46m34.59972326s) [noise=true , address=145.79.227.32:4835 , piece_id=L2ZTK326XY3DCKHDJHZ25E6MM6E4XN24XLDUBSCQ7OWXWDE34E5Q, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.728737830 (610h46m34.599820865s) [noise=true , address=145.79.227.35:4483 , piece_id=VOUM3VD3PNVSCKK26HQ3EKVE3DFZFQDDXXO3ULEGJKWODTFQZEFA, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.729042310 (610h46m34.600125355s) [noise=true , address=77.90.31.16:4736 , piece_id=CAIXHGWBKNOZ2JGNSN32NRQK6Y4CN5XCGOF32FJG27E4LTFDK4LA, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.729097326 (610h46m34.600180361s) [noise=true , address=145.79.227.21:4475 , piece_id=JTOGYE76I4LQSOP2HJIRROP4IFJESLLV353XQSEXFXJCPYPJE2CA, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.729374553 (610h46m34.600457589s) [noise=true , address=145.79.227.10:4402 , piece_id=XZBGK6UOLSDRIK6ZGH2XIWP2BWSFQK674DKAPZBEWCVD3H5GEBNQ, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.729450758 (610h46m34.600533793s) [noise=true , address=145.79.227.51:4409 , piece_id=WPGRG5MOZR525QAE5HGGSUPRFJ6LKBPQQAJWTMFTK4PPNVKWM4SQ, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.729852623 (610h46m34.600935658s) [noise=true , address=77.90.31.18:4805 , piece_id=XGFONRYPHMDUPP7RJLNRRUZ7KXPRFIH22WJKTFRVM7DWZHCR3VSQ, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.729870658 (610h46m34.600953693s) [noise=true , address=45.91.116.8:4045 , piece_id=H2FC3TA5VGNJTA4ZFS6TCFN2DNWDNNZFR5WRX7QT6RKX4DVY5MIA, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.730183253 (610h46m34.601266298s) [noise=true , address=77.90.31.15:4697 , piece_id=T5XK6HTXQQWVPISPFJ4ILOXCMCDD3URLTQZOMX5QZGG5IG5K6GDA, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.730255150 (610h46m34.601338185s) [noise=true , address=145.79.227.12:4767 , piece_id=3FIFSPGGXTLELWMFF3DVTSETVRY4RTTJ75HERKN3O3PJ4IH3PTOQ, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:36.422393419 (610h46m36.293476464s) [upload=535255]: Next returned nil segment Dec 31 02:15:36.422406284 (610h46m36.293489329s) [upload=535255]: Waiting for error group managing segments... Dec 31 02:15:36.710670932 (610h46m36.581753977s) [noise=true , address=145.79.227.31:4797 , piece_id=VGZA5SE3IY3VSBN3MLHUGPPCD5XYNJXNTCN522BCTKBFLCW34UTQ, seg_pos={0 0}, upload=535255]: Done uploading piece. err: <nil> Dec 31 02:15:36.711270734 (610h46m36.582353769s) [noise=true , address=145.79.227.32:4835 , piece_id=L2ZTK326XY3DCKHDJHZ25E6MM6E4XN24XLDUBSCQ7OWXWDE34E5Q, seg_pos={0 0}, upload=535255]: Done uploading piece. err: <nil> Dec 31 02:15:36.712146010 (610h46m36.583229045s) [noise=true , address=145.79.227.50:4375 , piece_id=QRAPDLJ3LM6WNYHLZ2VRYO2OIAVSZ3IQ5R62TLBPCAKVDAF2A3FQ, seg_pos={0 0}, upload=535255]: Done uploading piece. err: <nil> Dec 31 02:15:36.732635736 (610h46m36.603718771s) [noise=true , address=145.79.227.16:4909 , piece_id=IEZDHX7O5NVWGLAJ4EQOK4F7RNHJBSFHB6C2TQ42EB7W7JATRMUQ, seg_pos={0 0}, upload=535255]: Done uploading piece. err: <nil> Dec 31 02:15:36.737942711 (610h46m36.609025756s) [noise=true , address=145.79.227.21:4475 , piece_id=JTOGYE76I4LQSOP2HJIRROP4IFJESLLV353XQSEXFXJCPYPJE2CA, seg_pos={0 0}, upload=535255]: Done uploading piece. err: <nil> Dec 31 02:15:36.741292659 (610h46m36.612375694s) [noise=true , address=145.79.227.51:4409 , piece_id=WPGRG5MOZR525QAE5HGGSUPRFJ6LKBPQQAJWTMFTK4PPNVKWM4SQ, seg_pos={0 0}, upload=535255]: Done uploading piece. err: <nil> Dec 31 02:15:36.744308642 (610h46m36.615391677s) [noise=true , address=145.79.227.12:4767 , piece_id=3FIFSPGGXTLELWMFF3DVTSETVRY4RTTJ75HERKN3O3PJ4IH3PTOQ, seg_pos={0 0}, upload=535255]: Done uploading piece. err: <nil> Dec 31 02:15:36.795374577 (610h46m36.666457622s) [noise=true , address=145.79.227.10:4402 , piece_id=XZBGK6UOLSDRIK6ZGH2XIWP2BWSFQK674DKAPZBEWCVD3H5GEBNQ, seg_pos={0 0}, upload=535255]: Done uploading piece. err: <nil> Dec 31 02:15:36.881602926 (610h46m36.752685961s) [noise=true , address=77.90.31.18:4805 , piece_id=XGFONRYPHMDUPP7RJLNRRUZ7KXPRFIH22WJKTFRVM7DWZHCR3VSQ, seg_pos={0 0}, upload=535255]: Done uploading piece. err: <nil> Dec 31 02:15:37.071727813 (610h46m36.942810859s) [noise=true , address=45.91.116.9:4263 , piece_id=6RIQ7G6CJOJNGTKLYY7EX4OQ3YBFEJKQAUG5XIJ2NJGJB4ULH2KA, seg_pos={0 0}, upload=535255]: Done uploading piece. err: <nil> Dec 31 02:15:37.579913516 (610h46m37.450996561s) [noise=true , address=77.90.31.16:4736 , piece_id=CAIXHGWBKNOZ2JGNSN32NRQK6Y4CN5XCGOF32FJG27E4LTFDK4LA, seg_pos={0 0}, upload=535255]: Done uploading piece. err: <nil> Dec 31 02:15:37.737075300 (610h46m37.608158335s) [noise=true , address=77.90.31.15:4697 , piece_id=T5XK6HTXQQWVPISPFJ4ILOXCMCDD3URLTQZOMX5QZGG5IG5K6GDA, seg_pos={0 0}, upload=535255]: Done uploading piece. err: <nil> Dec 31 02:15:37.770719692 (610h46m37.641802727s) [noise=true , address=45.91.116.8:4045 , piece_id=H2FC3TA5VGNJTA4ZFS6TCFN2DNWDNNZFR5WRX7QT6RKX4DVY5MIA, seg_pos={0 0}, upload=535255]: Done uploading piece. err: <nil> Dec 31 02:15:37.770744329 (610h46m37.641827364s) [seg_pos={0 0}, upload=535255]: Segment reached optimal threshold of 13 pieces. Dec 31 02:15:37.770987743 (610h46m37.642070788s) [noise=true , address=145.79.227.35:4483 , piece_id=VOUM3VD3PNVSCKK26HQ3EKVE3DFZFQDDXXO3ULEGJKWODTFQZEFA, seg_pos={0 0}, upload=535255]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF Dec 31 02:15:37.770996148 (610h46m37.642079194s) [noise=true , address=145.79.227.35:4483 , piece_id=VOUM3VD3PNVSCKK26HQ3EKVE3DFZFQDDXXO3ULEGJKWODTFQZEFA, seg_pos={0 0}, upload=535255]: Long tail cancellation due to optimal threshold Dec 31 02:15:37.771089406 (610h46m37.642172441s) [noise=true , address=145.79.227.34:4906 , piece_id=65OFQCDQITDTF4AHGDUZB6TY43UTXH4WFFS3MRI3LQD4M53GL3QQ, seg_pos={0 0}, upload=535255]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 31 02:15:37.771104735 (610h46m37.64218778s) [noise=true , address=145.79.227.34:4906 , piece_id=65OFQCDQITDTF4AHGDUZB6TY43UTXH4WFFS3MRI3LQD4M53GL3QQ, seg_pos={0 0}, upload=535255]: Long tail cancellation due to optimal threshold Dec 31 02:15:37.771250514 (610h46m37.642333559s) [noise=true , address=77.90.31.19:4842 , piece_id=TMFJQ7VEVKCVQWYE6LGSF3AXEKYISQPRXDBZGVJUE63FD2EPS4RQ, seg_pos={0 0}, upload=535255]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Dec 31 02:15:37.771256555 (610h46m37.6423396s) [noise=true , address=77.90.31.19:4842 , piece_id=TMFJQ7VEVKCVQWYE6LGSF3AXEKYISQPRXDBZGVJUE63FD2EPS4RQ, seg_pos={0 0}, upload=535255]: Long tail cancellation due to optimal threshold Dec 31 02:15:37.771271012 (610h46m37.642354047s) [seg_pos={0 0}, upload=535255]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 02:15:37.771347367 (610h46m37.642430402s) [upload=535255]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 02:15:37.771351817 (610h46m37.642434852s) [upload=535255]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 02:15:38.072543669 (610h46m37.943626724s) [upload=535255]: Done uploading segments... Dec 31 02:15:38.072571552 (610h46m37.943654587s) [upload=535255]: Upload finished. err: <nil> Duration: 5.878259064s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....X..........................X...........................] duration: 34.95511271s trace: https://jaeger.dev.storj.io/trace/365693fea00df28 FAIL (2025-12-31 02:18:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [noise=true , address=145.79.227.49:4343 , piece_id=6L3HPTS7D2NRA4QEOH4RFM3D3B53CTCONRI6G6YVU4OOTUWTG7XA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.942060847 (610h25m24.813143892s) [noise=true , address=145.79.227.34:4937 , piece_id=5W7LBM3YJJTVUFYTELR57DVXB663OHO5N3Z2SPQPZ4G43IK47O3Q, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.942684104 (610h25m24.813767139s) [noise=true , address=64.49.13.139:4478 , piece_id=QN6RT2KWEA6APQVA66ZMKG4LH537UGYNOXOCVVA2YC2MKDOLT3YA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.943064007 (610h25m24.814147042s) [noise=true , address=145.79.227.40:4084 , piece_id=RUKTKDII6BRUWSYVS65CWUBT3QKG3ASUK3KA3O5VGE53XSG6IMOQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.943459560 (610h25m24.814542605s) [noise=true , address=145.79.227.14:4847 , piece_id=JX5NXTMVM3NSWIEVR4RX46ABSNPKTOFDLFBPKEXFQ3G3ANKV5OLA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.943865303 (610h25m24.814948348s) [noise=true , address=45.91.116.6:4981 , piece_id=CT474ZA2LVAZEBJHBDD4GB6NTHVHZFJLNE5OZV3DFS2NZKVW3EQQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.944259694 (610h25m24.815342749s) [noise=true , address=145.79.227.51:4443 , piece_id=73UUMRDY7BSZNCGBXCZYYRBVE5KLPMNPRNXOZEGWKQGLHJS32J2Q, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.946575182 (610h25m24.817658227s) [noise=true , address=77.90.31.10:4523 , piece_id=P6LIONXJI6AFZ2JTKCQ2W6RZCOZCK6MZ53EXAJKXYQXSX2PX4QEA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.948132909 (610h25m24.819215944s) [noise=true , address=145.79.227.20:4452 , piece_id=ZSDN2JZ3FQUVJYN5OURDXNKEDLDNULJOHKKRZVIPXZIWIYAHCRXQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.949605774 (610h25m24.820688819s) [noise=true , address=77.90.31.14:4680 , piece_id=S2E4WBA7MJVAZLT5JG3IMHK53RQC77CCHV2PDF6L2R2JEA2U3K4Q, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.951459002 (610h25m24.822542048s) [noise=true , address=193.149.164.143:4704 , piece_id=BHXHFBKTLTDMGXWY5QRTP2THZQE5OXFZ2LCMTO2KRYRWAHXWW3NQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.952986742 (610h25m24.824069777s) [noise=true , address=77.90.31.19:4848 , piece_id=VZYG32S6ODIPDKQYZRXECBHXOERZDDWD2OF35Z46W4XMUZ3OJLAA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.956916654 (610h25m24.827999689s) [noise=true , address=145.79.227.29:4971 , piece_id=MEPXHHOQ6SUZC3GIPCQEMOKKF5OJTKZEI3UYZBAYXAQZGSVEKSBA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.960665062 (610h25m24.831748097s) [noise=true , address=64.49.13.170:4720 , piece_id=HIATRSAPE4YLSODFZDPP43ZUMLDVM7GJEVMWE4ZGUQTP25LTHGTA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.962366001 (610h25m24.833449036s) [noise=true , address=145.79.227.2:4117 , piece_id=4SGFGB3IJ7LTXDMMWM2PGGOV64MC46IZ7OZQWH55J4NBQY7CHMZQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.963708216 (610h25m24.834791261s) [noise=true , address=145.79.227.35:4505 , piece_id=V7RS7A4QLZTH4G32ZUZNSF2Z5APVWZGJO77NEGP3SBWQT3WCMVLQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:26.352236138 (610h25m26.223319173s) [upload=534952]: Next returned nil segment Dec 31 01:54:26.352253821 (610h25m26.223336857s) [upload=534952]: Waiting for error group managing segments... Dec 31 01:54:26.812758374 (610h25m26.683841409s) [noise=true , address=145.79.227.51:4443 , piece_id=73UUMRDY7BSZNCGBXCZYYRBVE5KLPMNPRNXOZEGWKQGLHJS32J2Q, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.824371056 (610h25m26.695454101s) [noise=true , address=145.79.227.14:4847 , piece_id=JX5NXTMVM3NSWIEVR4RX46ABSNPKTOFDLFBPKEXFQ3G3ANKV5OLA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.838414046 (610h25m26.709497081s) [noise=true , address=64.49.13.138:4490 , piece_id=2XTUZE6PXRE3OARNJVT36FXXWUMYIBVNE6M2VVFAVBGKEE2H5YZQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.875499841 (610h25m26.746582876s) [noise=true , address=64.49.13.139:4478 , piece_id=QN6RT2KWEA6APQVA66ZMKG4LH537UGYNOXOCVVA2YC2MKDOLT3YA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.891607221 (610h25m26.762690256s) [noise=true , address=145.79.227.20:4452 , piece_id=ZSDN2JZ3FQUVJYN5OURDXNKEDLDNULJOHKKRZVIPXZIWIYAHCRXQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.988561223 (610h25m26.859644268s) [noise=true , address=64.49.13.170:4720 , piece_id=HIATRSAPE4YLSODFZDPP43ZUMLDVM7GJEVMWE4ZGUQTP25LTHGTA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.039035911 (610h25m26.910118946s) [noise=true , address=77.90.31.14:4680 , piece_id=S2E4WBA7MJVAZLT5JG3IMHK53RQC77CCHV2PDF6L2R2JEA2U3K4Q, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.098023090 (610h25m26.969106126s) [noise=true , address=45.91.116.6:4981 , piece_id=CT474ZA2LVAZEBJHBDD4GB6NTHVHZFJLNE5OZV3DFS2NZKVW3EQQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.653790754 (610h25m27.524873789s) [noise=true , address=77.90.31.21:4886 , piece_id=K34WPIBGGGEPHIKJ253XGLDCIQ3WLFN4AZ6EYNWVHHT2EIHS6YRQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.684192972 (610h25m27.555276017s) [noise=true , address=193.149.164.143:4704 , piece_id=BHXHFBKTLTDMGXWY5QRTP2THZQE5OXFZ2LCMTO2KRYRWAHXWW3NQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.717238425 (610h25m27.58832146s) [noise=true , address=145.79.227.40:4084 , piece_id=RUKTKDII6BRUWSYVS65CWUBT3QKG3ASUK3KA3O5VGE53XSG6IMOQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.718349742 (610h25m27.589432777s) [noise=true , address=77.90.31.10:4523 , piece_id=P6LIONXJI6AFZ2JTKCQ2W6RZCOZCK6MZ53EXAJKXYQXSX2PX4QEA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.803639562 (610h25m27.674722597s) [noise=true , address=145.79.227.29:4971 , piece_id=MEPXHHOQ6SUZC3GIPCQEMOKKF5OJTKZEI3UYZBAYXAQZGSVEKSBA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.812791279 (610h25m27.683874314s) [noise=true , address=145.79.227.49:4343 , piece_id=6L3HPTS7D2NRA4QEOH4RFM3D3B53CTCONRI6G6YVU4OOTUWTG7XA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.812812348 (610h25m27.683895393s) [seg_pos={0 0}, upload=534952]: Segment reached optimal threshold of 14 pieces. Dec 31 01:54:27.813018000 (610h25m27.684101035s) [noise=true , address=145.79.227.35:4505 , piece_id=V7RS7A4QLZTH4G32ZUZNSF2Z5APVWZGJO77NEGP3SBWQT3WCMVLQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i6itGoBhLsLiqC8et8Y5L62fLjcWxy18u39NAr4R3nXNMWcsf): protocol: expected piece hash; context canceled; EOF Dec 31 01:54:27.813024772 (610h25m27.684107808s) [noise=true , address=145.79.227.35:4505 , piece_id=V7RS7A4QLZTH4G32ZUZNSF2Z5APVWZGJO77NEGP3SBWQT3WCMVLQ, seg_pos={0 0}, upload=534952]: Long tail cancellation due to optimal threshold Dec 31 01:54:27.813175931 (610h25m27.684258976s) [noise=true , address=145.79.227.34:4937 , piece_id=5W7LBM3YJJTVUFYTELR57DVXB663OHO5N3Z2SPQPZ4G43IK47O3Q, seg_pos={0 0}, upload=534952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1LKrFThdW3jJpG6dRZLUaxLXTa5Wy27yQ8PGUEqjvBWfFoRB94): protocol: expected piece hash; context canceled; EOF Dec 31 01:54:27.813181702 (610h25m27.684264737s) [noise=true , address=145.79.227.34:4937 , piece_id=5W7LBM3YJJTVUFYTELR57DVXB663OHO5N3Z2SPQPZ4G43IK47O3Q, seg_pos={0 0}, upload=534952]: Long tail cancellation due to optimal threshold Dec 31 01:54:27.813407862 (610h25m27.684490907s) [noise=true , address=145.79.227.2:4117 , piece_id=4SGFGB3IJ7LTXDMMWM2PGGOV64MC46IZ7OZQWH55J4NBQY7CHMZQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122Htg72f5eUUutf1p3fYQpy1Wkf6AgdaQV83HHYye2n8zm3the): protocol: expected piece hash; context canceled; EOF Dec 31 01:54:27.813414264 (610h25m27.684497299s) [noise=true , address=145.79.227.2:4117 , piece_id=4SGFGB3IJ7LTXDMMWM2PGGOV64MC46IZ7OZQWH55J4NBQY7CHMZQ, seg_pos={0 0}, upload=534952]: Long tail cancellation due to optimal threshold Dec 31 01:54:27.813424614 (610h25m27.684507649s) [noise=true , address=77.90.31.19:4848 , piece_id=VZYG32S6ODIPDKQYZRXECBHXOERZDDWD2OF35Z46W4XMUZ3OJLAA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD): protocol: write tcp 188.166.225.170:42842->77.90.31.19:4848: use of closed network connection; write tcp 188.166.225.170:42842->77.90.31.19:4848: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42842->77.90.31.19:4848: use of closed network connection Dec 31 01:54:27.813432750 (610h25m27.684515785s) [noise=true , address=77.90.31.19:4848 , piece_id=VZYG32S6ODIPDKQYZRXECBHXOERZDDWD2OF35Z46W4XMUZ3OJLAA, seg_pos={0 0}, upload=534952]: Long tail cancellation due to optimal threshold Dec 31 01:54:27.813474088 (610h25m27.684557123s) [seg_pos={0 0}, upload=534952]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 01:54:27.813535325 (610h25m27.68461836s) [upload=534952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 01:54:27.813539613 (610h25m27.684622648s) [upload=534952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 01:54:28.099921034 (610h25m27.971004069s) [upload=534952]: Done uploading segments... Dec 31 01:54:28.099940451 (610h25m27.971023486s) [upload=534952]: Upload finished. err: <nil> Duration: 34.955146336s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.X....X..........X............X..X.......X.X..X....XX.XX....] duration: 5.745785782s trace: https://jaeger.dev.storj.io/trace/2f9cb5ff97a2ac44 FAIL (2025-12-31 02:18:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: XGFONRYPHMDUPP7RJLNRRUZ7KXPRFIH22WJKTFRVM7DWZHCR3VSQ) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: H2FC3TA5VGNJTA4ZFS6TCFN2DNWDNNZFR5WRX7QT6RKX4DVY5MIA) context deadline exceeded; piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: T5XK6HTXQQWVPISPFJ4ILOXCMCDD3URLTQZOMX5QZGG5IG5K6GDA) context deadline exceeded; piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: 6RIQ7G6CJOJNGTKLYY7EX4OQ3YBFEJKQAUG5XIJ2NJGJB4ULH2KA) context deadline exceeded ======LOGS====== Dec 31 02:15:34.470943883 (610h46m34.342026918s) [upload=535255]: Starting upload Dec 31 02:15:34.471018445 (610h46m34.34210148s) [upload=535255]: Done starting upload Dec 31 02:15:34.471184521 (610h46m34.342267566s) [upload=535255]: Uploading segments... Dec 31 02:15:34.471391205 (610h46m34.34247424s) [upload=535255]: Got next segment. Inline: false Dec 31 02:15:34.471450537 (610h46m34.342533572s) [upload=535255]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 02:15:34.471455456 (610h46m34.342538501s) [upload=535255]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 02:15:34.726848804 (610h46m34.597931849s) [seg_pos={0 0}, upload=535255]: Begin upload segment... Dec 31 02:15:34.726898429 (610h46m34.597981464s) [seg_pos={0 0}, upload=535255]: Done begin upload segment. Dec 31 02:15:34.726934788 (610h46m34.598017833s) [noise=true , address=145.79.227.16:4909 , piece_id=IEZDHX7O5NVWGLAJ4EQOK4F7RNHJBSFHB6C2TQ42EB7W7JATRMUQ, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.726975706 (610h46m34.598058741s) [noise=true , address=45.91.116.9:4263 , piece_id=6RIQ7G6CJOJNGTKLYY7EX4OQ3YBFEJKQAUG5XIJ2NJGJB4ULH2KA, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.727672262 (610h46m34.598755307s) [noise=true , address=77.90.31.19:4842 , piece_id=TMFJQ7VEVKCVQWYE6LGSF3AXEKYISQPRXDBZGVJUE63FD2EPS4RQ, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.727687450 (610h46m34.598770485s) [noise=true , address=145.79.227.31:4797 , piece_id=VGZA5SE3IY3VSBN3MLHUGPPCD5XYNJXNTCN522BCTKBFLCW34UTQ, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.728136986 (610h46m34.599220022s) [noise=true , address=145.79.227.34:4906 , piece_id=65OFQCDQITDTF4AHGDUZB6TY43UTXH4WFFS3MRI3LQD4M53GL3QQ, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.728259861 (610h46m34.599342906s) [noise=true , address=145.79.227.50:4375 , piece_id=QRAPDLJ3LM6WNYHLZ2VRYO2OIAVSZ3IQ5R62TLBPCAKVDAF2A3FQ, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.728640225 (610h46m34.59972326s) [noise=true , address=145.79.227.32:4835 , piece_id=L2ZTK326XY3DCKHDJHZ25E6MM6E4XN24XLDUBSCQ7OWXWDE34E5Q, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.728737830 (610h46m34.599820865s) [noise=true , address=145.79.227.35:4483 , piece_id=VOUM3VD3PNVSCKK26HQ3EKVE3DFZFQDDXXO3ULEGJKWODTFQZEFA, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.729042310 (610h46m34.600125355s) [noise=true , address=77.90.31.16:4736 , piece_id=CAIXHGWBKNOZ2JGNSN32NRQK6Y4CN5XCGOF32FJG27E4LTFDK4LA, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.729097326 (610h46m34.600180361s) [noise=true , address=145.79.227.21:4475 , piece_id=JTOGYE76I4LQSOP2HJIRROP4IFJESLLV353XQSEXFXJCPYPJE2CA, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.729374553 (610h46m34.600457589s) [noise=true , address=145.79.227.10:4402 , piece_id=XZBGK6UOLSDRIK6ZGH2XIWP2BWSFQK674DKAPZBEWCVD3H5GEBNQ, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.729450758 (610h46m34.600533793s) [noise=true , address=145.79.227.51:4409 , piece_id=WPGRG5MOZR525QAE5HGGSUPRFJ6LKBPQQAJWTMFTK4PPNVKWM4SQ, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.729852623 (610h46m34.600935658s) [noise=true , address=77.90.31.18:4805 , piece_id=XGFONRYPHMDUPP7RJLNRRUZ7KXPRFIH22WJKTFRVM7DWZHCR3VSQ, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.729870658 (610h46m34.600953693s) [noise=true , address=45.91.116.8:4045 , piece_id=H2FC3TA5VGNJTA4ZFS6TCFN2DNWDNNZFR5WRX7QT6RKX4DVY5MIA, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.730183253 (610h46m34.601266298s) [noise=true , address=77.90.31.15:4697 , piece_id=T5XK6HTXQQWVPISPFJ4ILOXCMCDD3URLTQZOMX5QZGG5IG5K6GDA, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.730255150 (610h46m34.601338185s) [noise=true , address=145.79.227.12:4767 , piece_id=3FIFSPGGXTLELWMFF3DVTSETVRY4RTTJ75HERKN3O3PJ4IH3PTOQ, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:36.422393419 (610h46m36.293476464s) [upload=535255]: Next returned nil segment Dec 31 02:15:36.422406284 (610h46m36.293489329s) [upload=535255]: Waiting for error group managing segments... Dec 31 02:15:36.710670932 (610h46m36.581753977s) [noise=true , address=145.79.227.31:4797 , piece_id=VGZA5SE3IY3VSBN3MLHUGPPCD5XYNJXNTCN522BCTKBFLCW34UTQ, seg_pos={0 0}, upload=535255]: Done uploading piece. err: <nil> Dec 31 02:15:36.711270734 (610h46m36.582353769s) [noise=true , address=145.79.227.32:4835 , piece_id=L2ZTK326XY3DCKHDJHZ25E6MM6E4XN24XLDUBSCQ7OWXWDE34E5Q, seg_pos={0 0}, upload=535255]: Done uploading piece. err: <nil> Dec 31 02:15:36.712146010 (610h46m36.583229045s) [noise=true , address=145.79.227.50:4375 , piece_id=QRAPDLJ3LM6WNYHLZ2VRYO2OIAVSZ3IQ5R62TLBPCAKVDAF2A3FQ, seg_pos={0 0}, upload=535255]: Done uploading piece. err: <nil> Dec 31 02:15:36.732635736 (610h46m36.603718771s) [noise=true , address=145.79.227.16:4909 , piece_id=IEZDHX7O5NVWGLAJ4EQOK4F7RNHJBSFHB6C2TQ42EB7W7JATRMUQ, seg_pos={0 0}, upload=535255]: Done uploading piece. err: <nil> Dec 31 02:15:36.737942711 (610h46m36.609025756s) [noise=true , address=145.79.227.21:4475 , piece_id=JTOGYE76I4LQSOP2HJIRROP4IFJESLLV353XQSEXFXJCPYPJE2CA, seg_pos={0 0}, upload=535255]: Done uploading piece. err: <nil> Dec 31 02:15:36.741292659 (610h46m36.612375694s) [noise=true , address=145.79.227.51:4409 , piece_id=WPGRG5MOZR525QAE5HGGSUPRFJ6LKBPQQAJWTMFTK4PPNVKWM4SQ, seg_pos={0 0}, upload=535255]: Done uploading piece. err: <nil> Dec 31 02:15:36.744308642 (610h46m36.615391677s) [noise=true , address=145.79.227.12:4767 , piece_id=3FIFSPGGXTLELWMFF3DVTSETVRY4RTTJ75HERKN3O3PJ4IH3PTOQ, seg_pos={0 0}, upload=535255]: Done uploading piece. err: <nil> Dec 31 02:15:36.795374577 (610h46m36.666457622s) [noise=true , address=145.79.227.10:4402 , piece_id=XZBGK6UOLSDRIK6ZGH2XIWP2BWSFQK674DKAPZBEWCVD3H5GEBNQ, seg_pos={0 0}, upload=535255]: Done uploading piece. err: <nil> Dec 31 02:15:36.881602926 (610h46m36.752685961s) [noise=true , address=77.90.31.18:4805 , piece_id=XGFONRYPHMDUPP7RJLNRRUZ7KXPRFIH22WJKTFRVM7DWZHCR3VSQ, seg_pos={0 0}, upload=535255]: Done uploading piece. err: <nil> Dec 31 02:15:37.071727813 (610h46m36.942810859s) [noise=true , address=45.91.116.9:4263 , piece_id=6RIQ7G6CJOJNGTKLYY7EX4OQ3YBFEJKQAUG5XIJ2NJGJB4ULH2KA, seg_pos={0 0}, upload=535255]: Done uploading piece. err: <nil> Dec 31 02:15:37.579913516 (610h46m37.450996561s) [noise=true , address=77.90.31.16:4736 , piece_id=CAIXHGWBKNOZ2JGNSN32NRQK6Y4CN5XCGOF32FJG27E4LTFDK4LA, seg_pos={0 0}, upload=535255]: Done uploading piece. err: <nil> Dec 31 02:15:37.737075300 (610h46m37.608158335s) [noise=true , address=77.90.31.15:4697 , piece_id=T5XK6HTXQQWVPISPFJ4ILOXCMCDD3URLTQZOMX5QZGG5IG5K6GDA, seg_pos={0 0}, upload=535255]: Done uploading piece. err: <nil> Dec 31 02:15:37.770719692 (610h46m37.641802727s) [noise=true , address=45.91.116.8:4045 , piece_id=H2FC3TA5VGNJTA4ZFS6TCFN2DNWDNNZFR5WRX7QT6RKX4DVY5MIA, seg_pos={0 0}, upload=535255]: Done uploading piece. err: <nil> Dec 31 02:15:37.770744329 (610h46m37.641827364s) [seg_pos={0 0}, upload=535255]: Segment reached optimal threshold of 13 pieces. Dec 31 02:15:37.770987743 (610h46m37.642070788s) [noise=true , address=145.79.227.35:4483 , piece_id=VOUM3VD3PNVSCKK26HQ3EKVE3DFZFQDDXXO3ULEGJKWODTFQZEFA, seg_pos={0 0}, upload=535255]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF Dec 31 02:15:37.770996148 (610h46m37.642079194s) [noise=true , address=145.79.227.35:4483 , piece_id=VOUM3VD3PNVSCKK26HQ3EKVE3DFZFQDDXXO3ULEGJKWODTFQZEFA, seg_pos={0 0}, upload=535255]: Long tail cancellation due to optimal threshold Dec 31 02:15:37.771089406 (610h46m37.642172441s) [noise=true , address=145.79.227.34:4906 , piece_id=65OFQCDQITDTF4AHGDUZB6TY43UTXH4WFFS3MRI3LQD4M53GL3QQ, seg_pos={0 0}, upload=535255]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 31 02:15:37.771104735 (610h46m37.64218778s) [noise=true , address=145.79.227.34:4906 , piece_id=65OFQCDQITDTF4AHGDUZB6TY43UTXH4WFFS3MRI3LQD4M53GL3QQ, seg_pos={0 0}, upload=535255]: Long tail cancellation due to optimal threshold Dec 31 02:15:37.771250514 (610h46m37.642333559s) [noise=true , address=77.90.31.19:4842 , piece_id=TMFJQ7VEVKCVQWYE6LGSF3AXEKYISQPRXDBZGVJUE63FD2EPS4RQ, seg_pos={0 0}, upload=535255]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Dec 31 02:15:37.771256555 (610h46m37.6423396s) [noise=true , address=77.90.31.19:4842 , piece_id=TMFJQ7VEVKCVQWYE6LGSF3AXEKYISQPRXDBZGVJUE63FD2EPS4RQ, seg_pos={0 0}, upload=535255]: Long tail cancellation due to optimal threshold Dec 31 02:15:37.771271012 (610h46m37.642354047s) [seg_pos={0 0}, upload=535255]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 02:15:37.771347367 (610h46m37.642430402s) [upload=535255]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 02:15:37.771351817 (610h46m37.642434852s) [upload=535255]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 02:15:38.072543669 (610h46m37.943626724s) [upload=535255]: Done uploading segments... Dec 31 02:15:38.072571552 (610h46m37.943654587s) [upload=535255]: Upload finished. err: <nil> Duration: 5.74581149s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..............X..................................X..........] duration: 3.428524199s trace: https://jaeger.dev.storj.io/trace/740aa405bd68150f FAIL (2025-12-31 02:18:04 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.428549807s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [......X..........................X..........................] duration: 31.841867024s trace: https://jaeger.dev.storj.io/trace/365693fea00df28 FAIL (2025-12-31 02:17:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [noise=true , address=145.79.227.49:4343 , piece_id=6L3HPTS7D2NRA4QEOH4RFM3D3B53CTCONRI6G6YVU4OOTUWTG7XA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.942060847 (610h25m24.813143892s) [noise=true , address=145.79.227.34:4937 , piece_id=5W7LBM3YJJTVUFYTELR57DVXB663OHO5N3Z2SPQPZ4G43IK47O3Q, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.942684104 (610h25m24.813767139s) [noise=true , address=64.49.13.139:4478 , piece_id=QN6RT2KWEA6APQVA66ZMKG4LH537UGYNOXOCVVA2YC2MKDOLT3YA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.943064007 (610h25m24.814147042s) [noise=true , address=145.79.227.40:4084 , piece_id=RUKTKDII6BRUWSYVS65CWUBT3QKG3ASUK3KA3O5VGE53XSG6IMOQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.943459560 (610h25m24.814542605s) [noise=true , address=145.79.227.14:4847 , piece_id=JX5NXTMVM3NSWIEVR4RX46ABSNPKTOFDLFBPKEXFQ3G3ANKV5OLA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.943865303 (610h25m24.814948348s) [noise=true , address=45.91.116.6:4981 , piece_id=CT474ZA2LVAZEBJHBDD4GB6NTHVHZFJLNE5OZV3DFS2NZKVW3EQQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.944259694 (610h25m24.815342749s) [noise=true , address=145.79.227.51:4443 , piece_id=73UUMRDY7BSZNCGBXCZYYRBVE5KLPMNPRNXOZEGWKQGLHJS32J2Q, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.946575182 (610h25m24.817658227s) [noise=true , address=77.90.31.10:4523 , piece_id=P6LIONXJI6AFZ2JTKCQ2W6RZCOZCK6MZ53EXAJKXYQXSX2PX4QEA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.948132909 (610h25m24.819215944s) [noise=true , address=145.79.227.20:4452 , piece_id=ZSDN2JZ3FQUVJYN5OURDXNKEDLDNULJOHKKRZVIPXZIWIYAHCRXQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.949605774 (610h25m24.820688819s) [noise=true , address=77.90.31.14:4680 , piece_id=S2E4WBA7MJVAZLT5JG3IMHK53RQC77CCHV2PDF6L2R2JEA2U3K4Q, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.951459002 (610h25m24.822542048s) [noise=true , address=193.149.164.143:4704 , piece_id=BHXHFBKTLTDMGXWY5QRTP2THZQE5OXFZ2LCMTO2KRYRWAHXWW3NQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.952986742 (610h25m24.824069777s) [noise=true , address=77.90.31.19:4848 , piece_id=VZYG32S6ODIPDKQYZRXECBHXOERZDDWD2OF35Z46W4XMUZ3OJLAA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.956916654 (610h25m24.827999689s) [noise=true , address=145.79.227.29:4971 , piece_id=MEPXHHOQ6SUZC3GIPCQEMOKKF5OJTKZEI3UYZBAYXAQZGSVEKSBA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.960665062 (610h25m24.831748097s) [noise=true , address=64.49.13.170:4720 , piece_id=HIATRSAPE4YLSODFZDPP43ZUMLDVM7GJEVMWE4ZGUQTP25LTHGTA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.962366001 (610h25m24.833449036s) [noise=true , address=145.79.227.2:4117 , piece_id=4SGFGB3IJ7LTXDMMWM2PGGOV64MC46IZ7OZQWH55J4NBQY7CHMZQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.963708216 (610h25m24.834791261s) [noise=true , address=145.79.227.35:4505 , piece_id=V7RS7A4QLZTH4G32ZUZNSF2Z5APVWZGJO77NEGP3SBWQT3WCMVLQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:26.352236138 (610h25m26.223319173s) [upload=534952]: Next returned nil segment Dec 31 01:54:26.352253821 (610h25m26.223336857s) [upload=534952]: Waiting for error group managing segments... Dec 31 01:54:26.812758374 (610h25m26.683841409s) [noise=true , address=145.79.227.51:4443 , piece_id=73UUMRDY7BSZNCGBXCZYYRBVE5KLPMNPRNXOZEGWKQGLHJS32J2Q, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.824371056 (610h25m26.695454101s) [noise=true , address=145.79.227.14:4847 , piece_id=JX5NXTMVM3NSWIEVR4RX46ABSNPKTOFDLFBPKEXFQ3G3ANKV5OLA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.838414046 (610h25m26.709497081s) [noise=true , address=64.49.13.138:4490 , piece_id=2XTUZE6PXRE3OARNJVT36FXXWUMYIBVNE6M2VVFAVBGKEE2H5YZQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.875499841 (610h25m26.746582876s) [noise=true , address=64.49.13.139:4478 , piece_id=QN6RT2KWEA6APQVA66ZMKG4LH537UGYNOXOCVVA2YC2MKDOLT3YA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.891607221 (610h25m26.762690256s) [noise=true , address=145.79.227.20:4452 , piece_id=ZSDN2JZ3FQUVJYN5OURDXNKEDLDNULJOHKKRZVIPXZIWIYAHCRXQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.988561223 (610h25m26.859644268s) [noise=true , address=64.49.13.170:4720 , piece_id=HIATRSAPE4YLSODFZDPP43ZUMLDVM7GJEVMWE4ZGUQTP25LTHGTA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.039035911 (610h25m26.910118946s) [noise=true , address=77.90.31.14:4680 , piece_id=S2E4WBA7MJVAZLT5JG3IMHK53RQC77CCHV2PDF6L2R2JEA2U3K4Q, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.098023090 (610h25m26.969106126s) [noise=true , address=45.91.116.6:4981 , piece_id=CT474ZA2LVAZEBJHBDD4GB6NTHVHZFJLNE5OZV3DFS2NZKVW3EQQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.653790754 (610h25m27.524873789s) [noise=true , address=77.90.31.21:4886 , piece_id=K34WPIBGGGEPHIKJ253XGLDCIQ3WLFN4AZ6EYNWVHHT2EIHS6YRQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.684192972 (610h25m27.555276017s) [noise=true , address=193.149.164.143:4704 , piece_id=BHXHFBKTLTDMGXWY5QRTP2THZQE5OXFZ2LCMTO2KRYRWAHXWW3NQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.717238425 (610h25m27.58832146s) [noise=true , address=145.79.227.40:4084 , piece_id=RUKTKDII6BRUWSYVS65CWUBT3QKG3ASUK3KA3O5VGE53XSG6IMOQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.718349742 (610h25m27.589432777s) [noise=true , address=77.90.31.10:4523 , piece_id=P6LIONXJI6AFZ2JTKCQ2W6RZCOZCK6MZ53EXAJKXYQXSX2PX4QEA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.803639562 (610h25m27.674722597s) [noise=true , address=145.79.227.29:4971 , piece_id=MEPXHHOQ6SUZC3GIPCQEMOKKF5OJTKZEI3UYZBAYXAQZGSVEKSBA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.812791279 (610h25m27.683874314s) [noise=true , address=145.79.227.49:4343 , piece_id=6L3HPTS7D2NRA4QEOH4RFM3D3B53CTCONRI6G6YVU4OOTUWTG7XA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.812812348 (610h25m27.683895393s) [seg_pos={0 0}, upload=534952]: Segment reached optimal threshold of 14 pieces. Dec 31 01:54:27.813018000 (610h25m27.684101035s) [noise=true , address=145.79.227.35:4505 , piece_id=V7RS7A4QLZTH4G32ZUZNSF2Z5APVWZGJO77NEGP3SBWQT3WCMVLQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i6itGoBhLsLiqC8et8Y5L62fLjcWxy18u39NAr4R3nXNMWcsf): protocol: expected piece hash; context canceled; EOF Dec 31 01:54:27.813024772 (610h25m27.684107808s) [noise=true , address=145.79.227.35:4505 , piece_id=V7RS7A4QLZTH4G32ZUZNSF2Z5APVWZGJO77NEGP3SBWQT3WCMVLQ, seg_pos={0 0}, upload=534952]: Long tail cancellation due to optimal threshold Dec 31 01:54:27.813175931 (610h25m27.684258976s) [noise=true , address=145.79.227.34:4937 , piece_id=5W7LBM3YJJTVUFYTELR57DVXB663OHO5N3Z2SPQPZ4G43IK47O3Q, seg_pos={0 0}, upload=534952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1LKrFThdW3jJpG6dRZLUaxLXTa5Wy27yQ8PGUEqjvBWfFoRB94): protocol: expected piece hash; context canceled; EOF Dec 31 01:54:27.813181702 (610h25m27.684264737s) [noise=true , address=145.79.227.34:4937 , piece_id=5W7LBM3YJJTVUFYTELR57DVXB663OHO5N3Z2SPQPZ4G43IK47O3Q, seg_pos={0 0}, upload=534952]: Long tail cancellation due to optimal threshold Dec 31 01:54:27.813407862 (610h25m27.684490907s) [noise=true , address=145.79.227.2:4117 , piece_id=4SGFGB3IJ7LTXDMMWM2PGGOV64MC46IZ7OZQWH55J4NBQY7CHMZQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122Htg72f5eUUutf1p3fYQpy1Wkf6AgdaQV83HHYye2n8zm3the): protocol: expected piece hash; context canceled; EOF Dec 31 01:54:27.813414264 (610h25m27.684497299s) [noise=true , address=145.79.227.2:4117 , piece_id=4SGFGB3IJ7LTXDMMWM2PGGOV64MC46IZ7OZQWH55J4NBQY7CHMZQ, seg_pos={0 0}, upload=534952]: Long tail cancellation due to optimal threshold Dec 31 01:54:27.813424614 (610h25m27.684507649s) [noise=true , address=77.90.31.19:4848 , piece_id=VZYG32S6ODIPDKQYZRXECBHXOERZDDWD2OF35Z46W4XMUZ3OJLAA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD): protocol: write tcp 188.166.225.170:42842->77.90.31.19:4848: use of closed network connection; write tcp 188.166.225.170:42842->77.90.31.19:4848: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42842->77.90.31.19:4848: use of closed network connection Dec 31 01:54:27.813432750 (610h25m27.684515785s) [noise=true , address=77.90.31.19:4848 , piece_id=VZYG32S6ODIPDKQYZRXECBHXOERZDDWD2OF35Z46W4XMUZ3OJLAA, seg_pos={0 0}, upload=534952]: Long tail cancellation due to optimal threshold Dec 31 01:54:27.813474088 (610h25m27.684557123s) [seg_pos={0 0}, upload=534952]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 01:54:27.813535325 (610h25m27.68461836s) [upload=534952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 01:54:27.813539613 (610h25m27.684622648s) [upload=534952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 01:54:28.099921034 (610h25m27.971004069s) [upload=534952]: Done uploading segments... Dec 31 01:54:28.099940451 (610h25m27.971023486s) [upload=534952]: Upload finished. err: <nil> Duration: 31.841884105s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...............X..................................X.........] duration: 3.635699885s trace: https://jaeger.dev.storj.io/trace/740aa405bd68150f FAIL (2025-12-31 02:17:27 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.635896017s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [X.X....X..........X............X..X.......X.X..X....XX.XX...] duration: 5.721381876s trace: https://jaeger.dev.storj.io/trace/2f9cb5ff97a2ac44 FAIL (2025-12-31 02:17:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: XGFONRYPHMDUPP7RJLNRRUZ7KXPRFIH22WJKTFRVM7DWZHCR3VSQ) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: H2FC3TA5VGNJTA4ZFS6TCFN2DNWDNNZFR5WRX7QT6RKX4DVY5MIA) context deadline exceeded; piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: T5XK6HTXQQWVPISPFJ4ILOXCMCDD3URLTQZOMX5QZGG5IG5K6GDA) context deadline exceeded; piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: 6RIQ7G6CJOJNGTKLYY7EX4OQ3YBFEJKQAUG5XIJ2NJGJB4ULH2KA) context deadline exceeded ======LOGS====== Dec 31 02:15:34.470943883 (610h46m34.342026918s) [upload=535255]: Starting upload Dec 31 02:15:34.471018445 (610h46m34.34210148s) [upload=535255]: Done starting upload Dec 31 02:15:34.471184521 (610h46m34.342267566s) [upload=535255]: Uploading segments... Dec 31 02:15:34.471391205 (610h46m34.34247424s) [upload=535255]: Got next segment. Inline: false Dec 31 02:15:34.471450537 (610h46m34.342533572s) [upload=535255]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 02:15:34.471455456 (610h46m34.342538501s) [upload=535255]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 02:15:34.726848804 (610h46m34.597931849s) [seg_pos={0 0}, upload=535255]: Begin upload segment... Dec 31 02:15:34.726898429 (610h46m34.597981464s) [seg_pos={0 0}, upload=535255]: Done begin upload segment. Dec 31 02:15:34.726934788 (610h46m34.598017833s) [noise=true , address=145.79.227.16:4909 , piece_id=IEZDHX7O5NVWGLAJ4EQOK4F7RNHJBSFHB6C2TQ42EB7W7JATRMUQ, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.726975706 (610h46m34.598058741s) [noise=true , address=45.91.116.9:4263 , piece_id=6RIQ7G6CJOJNGTKLYY7EX4OQ3YBFEJKQAUG5XIJ2NJGJB4ULH2KA, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.727672262 (610h46m34.598755307s) [noise=true , address=77.90.31.19:4842 , piece_id=TMFJQ7VEVKCVQWYE6LGSF3AXEKYISQPRXDBZGVJUE63FD2EPS4RQ, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.727687450 (610h46m34.598770485s) [noise=true , address=145.79.227.31:4797 , piece_id=VGZA5SE3IY3VSBN3MLHUGPPCD5XYNJXNTCN522BCTKBFLCW34UTQ, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.728136986 (610h46m34.599220022s) [noise=true , address=145.79.227.34:4906 , piece_id=65OFQCDQITDTF4AHGDUZB6TY43UTXH4WFFS3MRI3LQD4M53GL3QQ, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.728259861 (610h46m34.599342906s) [noise=true , address=145.79.227.50:4375 , piece_id=QRAPDLJ3LM6WNYHLZ2VRYO2OIAVSZ3IQ5R62TLBPCAKVDAF2A3FQ, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.728640225 (610h46m34.59972326s) [noise=true , address=145.79.227.32:4835 , piece_id=L2ZTK326XY3DCKHDJHZ25E6MM6E4XN24XLDUBSCQ7OWXWDE34E5Q, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.728737830 (610h46m34.599820865s) [noise=true , address=145.79.227.35:4483 , piece_id=VOUM3VD3PNVSCKK26HQ3EKVE3DFZFQDDXXO3ULEGJKWODTFQZEFA, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.729042310 (610h46m34.600125355s) [noise=true , address=77.90.31.16:4736 , piece_id=CAIXHGWBKNOZ2JGNSN32NRQK6Y4CN5XCGOF32FJG27E4LTFDK4LA, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.729097326 (610h46m34.600180361s) [noise=true , address=145.79.227.21:4475 , piece_id=JTOGYE76I4LQSOP2HJIRROP4IFJESLLV353XQSEXFXJCPYPJE2CA, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.729374553 (610h46m34.600457589s) [noise=true , address=145.79.227.10:4402 , piece_id=XZBGK6UOLSDRIK6ZGH2XIWP2BWSFQK674DKAPZBEWCVD3H5GEBNQ, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.729450758 (610h46m34.600533793s) [noise=true , address=145.79.227.51:4409 , piece_id=WPGRG5MOZR525QAE5HGGSUPRFJ6LKBPQQAJWTMFTK4PPNVKWM4SQ, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.729852623 (610h46m34.600935658s) [noise=true , address=77.90.31.18:4805 , piece_id=XGFONRYPHMDUPP7RJLNRRUZ7KXPRFIH22WJKTFRVM7DWZHCR3VSQ, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.729870658 (610h46m34.600953693s) [noise=true , address=45.91.116.8:4045 , piece_id=H2FC3TA5VGNJTA4ZFS6TCFN2DNWDNNZFR5WRX7QT6RKX4DVY5MIA, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.730183253 (610h46m34.601266298s) [noise=true , address=77.90.31.15:4697 , piece_id=T5XK6HTXQQWVPISPFJ4ILOXCMCDD3URLTQZOMX5QZGG5IG5K6GDA, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.730255150 (610h46m34.601338185s) [noise=true , address=145.79.227.12:4767 , piece_id=3FIFSPGGXTLELWMFF3DVTSETVRY4RTTJ75HERKN3O3PJ4IH3PTOQ, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:36.422393419 (610h46m36.293476464s) [upload=535255]: Next returned nil segment Dec 31 02:15:36.422406284 (610h46m36.293489329s) [upload=535255]: Waiting for error group managing segments... Dec 31 02:15:36.710670932 (610h46m36.581753977s) [noise=true , address=145.79.227.31:4797 , piece_id=VGZA5SE3IY3VSBN3MLHUGPPCD5XYNJXNTCN522BCTKBFLCW34UTQ, seg_pos={0 0}, upload=535255]: Done uploading piece. err: <nil> Dec 31 02:15:36.711270734 (610h46m36.582353769s) [noise=true , address=145.79.227.32:4835 , piece_id=L2ZTK326XY3DCKHDJHZ25E6MM6E4XN24XLDUBSCQ7OWXWDE34E5Q, seg_pos={0 0}, upload=535255]: Done uploading piece. err: <nil> Dec 31 02:15:36.712146010 (610h46m36.583229045s) [noise=true , address=145.79.227.50:4375 , piece_id=QRAPDLJ3LM6WNYHLZ2VRYO2OIAVSZ3IQ5R62TLBPCAKVDAF2A3FQ, seg_pos={0 0}, upload=535255]: Done uploading piece. err: <nil> Dec 31 02:15:36.732635736 (610h46m36.603718771s) [noise=true , address=145.79.227.16:4909 , piece_id=IEZDHX7O5NVWGLAJ4EQOK4F7RNHJBSFHB6C2TQ42EB7W7JATRMUQ, seg_pos={0 0}, upload=535255]: Done uploading piece. err: <nil> Dec 31 02:15:36.737942711 (610h46m36.609025756s) [noise=true , address=145.79.227.21:4475 , piece_id=JTOGYE76I4LQSOP2HJIRROP4IFJESLLV353XQSEXFXJCPYPJE2CA, seg_pos={0 0}, upload=535255]: Done uploading piece. err: <nil> Dec 31 02:15:36.741292659 (610h46m36.612375694s) [noise=true , address=145.79.227.51:4409 , piece_id=WPGRG5MOZR525QAE5HGGSUPRFJ6LKBPQQAJWTMFTK4PPNVKWM4SQ, seg_pos={0 0}, upload=535255]: Done uploading piece. err: <nil> Dec 31 02:15:36.744308642 (610h46m36.615391677s) [noise=true , address=145.79.227.12:4767 , piece_id=3FIFSPGGXTLELWMFF3DVTSETVRY4RTTJ75HERKN3O3PJ4IH3PTOQ, seg_pos={0 0}, upload=535255]: Done uploading piece. err: <nil> Dec 31 02:15:36.795374577 (610h46m36.666457622s) [noise=true , address=145.79.227.10:4402 , piece_id=XZBGK6UOLSDRIK6ZGH2XIWP2BWSFQK674DKAPZBEWCVD3H5GEBNQ, seg_pos={0 0}, upload=535255]: Done uploading piece. err: <nil> Dec 31 02:15:36.881602926 (610h46m36.752685961s) [noise=true , address=77.90.31.18:4805 , piece_id=XGFONRYPHMDUPP7RJLNRRUZ7KXPRFIH22WJKTFRVM7DWZHCR3VSQ, seg_pos={0 0}, upload=535255]: Done uploading piece. err: <nil> Dec 31 02:15:37.071727813 (610h46m36.942810859s) [noise=true , address=45.91.116.9:4263 , piece_id=6RIQ7G6CJOJNGTKLYY7EX4OQ3YBFEJKQAUG5XIJ2NJGJB4ULH2KA, seg_pos={0 0}, upload=535255]: Done uploading piece. err: <nil> Dec 31 02:15:37.579913516 (610h46m37.450996561s) [noise=true , address=77.90.31.16:4736 , piece_id=CAIXHGWBKNOZ2JGNSN32NRQK6Y4CN5XCGOF32FJG27E4LTFDK4LA, seg_pos={0 0}, upload=535255]: Done uploading piece. err: <nil> Dec 31 02:15:37.737075300 (610h46m37.608158335s) [noise=true , address=77.90.31.15:4697 , piece_id=T5XK6HTXQQWVPISPFJ4ILOXCMCDD3URLTQZOMX5QZGG5IG5K6GDA, seg_pos={0 0}, upload=535255]: Done uploading piece. err: <nil> Dec 31 02:15:37.770719692 (610h46m37.641802727s) [noise=true , address=45.91.116.8:4045 , piece_id=H2FC3TA5VGNJTA4ZFS6TCFN2DNWDNNZFR5WRX7QT6RKX4DVY5MIA, seg_pos={0 0}, upload=535255]: Done uploading piece. err: <nil> Dec 31 02:15:37.770744329 (610h46m37.641827364s) [seg_pos={0 0}, upload=535255]: Segment reached optimal threshold of 13 pieces. Dec 31 02:15:37.770987743 (610h46m37.642070788s) [noise=true , address=145.79.227.35:4483 , piece_id=VOUM3VD3PNVSCKK26HQ3EKVE3DFZFQDDXXO3ULEGJKWODTFQZEFA, seg_pos={0 0}, upload=535255]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF Dec 31 02:15:37.770996148 (610h46m37.642079194s) [noise=true , address=145.79.227.35:4483 , piece_id=VOUM3VD3PNVSCKK26HQ3EKVE3DFZFQDDXXO3ULEGJKWODTFQZEFA, seg_pos={0 0}, upload=535255]: Long tail cancellation due to optimal threshold Dec 31 02:15:37.771089406 (610h46m37.642172441s) [noise=true , address=145.79.227.34:4906 , piece_id=65OFQCDQITDTF4AHGDUZB6TY43UTXH4WFFS3MRI3LQD4M53GL3QQ, seg_pos={0 0}, upload=535255]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 31 02:15:37.771104735 (610h46m37.64218778s) [noise=true , address=145.79.227.34:4906 , piece_id=65OFQCDQITDTF4AHGDUZB6TY43UTXH4WFFS3MRI3LQD4M53GL3QQ, seg_pos={0 0}, upload=535255]: Long tail cancellation due to optimal threshold Dec 31 02:15:37.771250514 (610h46m37.642333559s) [noise=true , address=77.90.31.19:4842 , piece_id=TMFJQ7VEVKCVQWYE6LGSF3AXEKYISQPRXDBZGVJUE63FD2EPS4RQ, seg_pos={0 0}, upload=535255]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Dec 31 02:15:37.771256555 (610h46m37.6423396s) [noise=true , address=77.90.31.19:4842 , piece_id=TMFJQ7VEVKCVQWYE6LGSF3AXEKYISQPRXDBZGVJUE63FD2EPS4RQ, seg_pos={0 0}, upload=535255]: Long tail cancellation due to optimal threshold Dec 31 02:15:37.771271012 (610h46m37.642354047s) [seg_pos={0 0}, upload=535255]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 02:15:37.771347367 (610h46m37.642430402s) [upload=535255]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 02:15:37.771351817 (610h46m37.642434852s) [upload=535255]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 02:15:38.072543669 (610h46m37.943626724s) [upload=535255]: Done uploading segments... Dec 31 02:15:38.072571552 (610h46m37.943654587s) [upload=535255]: Upload finished. err: <nil> Duration: 5.721411462s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......X..........................X.........................] duration: 52.026294041s trace: https://jaeger.dev.storj.io/trace/365693fea00df28 FAIL (2025-12-31 02:17:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [noise=true , address=145.79.227.49:4343 , piece_id=6L3HPTS7D2NRA4QEOH4RFM3D3B53CTCONRI6G6YVU4OOTUWTG7XA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.942060847 (610h25m24.813143892s) [noise=true , address=145.79.227.34:4937 , piece_id=5W7LBM3YJJTVUFYTELR57DVXB663OHO5N3Z2SPQPZ4G43IK47O3Q, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.942684104 (610h25m24.813767139s) [noise=true , address=64.49.13.139:4478 , piece_id=QN6RT2KWEA6APQVA66ZMKG4LH537UGYNOXOCVVA2YC2MKDOLT3YA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.943064007 (610h25m24.814147042s) [noise=true , address=145.79.227.40:4084 , piece_id=RUKTKDII6BRUWSYVS65CWUBT3QKG3ASUK3KA3O5VGE53XSG6IMOQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.943459560 (610h25m24.814542605s) [noise=true , address=145.79.227.14:4847 , piece_id=JX5NXTMVM3NSWIEVR4RX46ABSNPKTOFDLFBPKEXFQ3G3ANKV5OLA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.943865303 (610h25m24.814948348s) [noise=true , address=45.91.116.6:4981 , piece_id=CT474ZA2LVAZEBJHBDD4GB6NTHVHZFJLNE5OZV3DFS2NZKVW3EQQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.944259694 (610h25m24.815342749s) [noise=true , address=145.79.227.51:4443 , piece_id=73UUMRDY7BSZNCGBXCZYYRBVE5KLPMNPRNXOZEGWKQGLHJS32J2Q, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.946575182 (610h25m24.817658227s) [noise=true , address=77.90.31.10:4523 , piece_id=P6LIONXJI6AFZ2JTKCQ2W6RZCOZCK6MZ53EXAJKXYQXSX2PX4QEA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.948132909 (610h25m24.819215944s) [noise=true , address=145.79.227.20:4452 , piece_id=ZSDN2JZ3FQUVJYN5OURDXNKEDLDNULJOHKKRZVIPXZIWIYAHCRXQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.949605774 (610h25m24.820688819s) [noise=true , address=77.90.31.14:4680 , piece_id=S2E4WBA7MJVAZLT5JG3IMHK53RQC77CCHV2PDF6L2R2JEA2U3K4Q, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.951459002 (610h25m24.822542048s) [noise=true , address=193.149.164.143:4704 , piece_id=BHXHFBKTLTDMGXWY5QRTP2THZQE5OXFZ2LCMTO2KRYRWAHXWW3NQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.952986742 (610h25m24.824069777s) [noise=true , address=77.90.31.19:4848 , piece_id=VZYG32S6ODIPDKQYZRXECBHXOERZDDWD2OF35Z46W4XMUZ3OJLAA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.956916654 (610h25m24.827999689s) [noise=true , address=145.79.227.29:4971 , piece_id=MEPXHHOQ6SUZC3GIPCQEMOKKF5OJTKZEI3UYZBAYXAQZGSVEKSBA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.960665062 (610h25m24.831748097s) [noise=true , address=64.49.13.170:4720 , piece_id=HIATRSAPE4YLSODFZDPP43ZUMLDVM7GJEVMWE4ZGUQTP25LTHGTA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.962366001 (610h25m24.833449036s) [noise=true , address=145.79.227.2:4117 , piece_id=4SGFGB3IJ7LTXDMMWM2PGGOV64MC46IZ7OZQWH55J4NBQY7CHMZQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.963708216 (610h25m24.834791261s) [noise=true , address=145.79.227.35:4505 , piece_id=V7RS7A4QLZTH4G32ZUZNSF2Z5APVWZGJO77NEGP3SBWQT3WCMVLQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:26.352236138 (610h25m26.223319173s) [upload=534952]: Next returned nil segment Dec 31 01:54:26.352253821 (610h25m26.223336857s) [upload=534952]: Waiting for error group managing segments... Dec 31 01:54:26.812758374 (610h25m26.683841409s) [noise=true , address=145.79.227.51:4443 , piece_id=73UUMRDY7BSZNCGBXCZYYRBVE5KLPMNPRNXOZEGWKQGLHJS32J2Q, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.824371056 (610h25m26.695454101s) [noise=true , address=145.79.227.14:4847 , piece_id=JX5NXTMVM3NSWIEVR4RX46ABSNPKTOFDLFBPKEXFQ3G3ANKV5OLA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.838414046 (610h25m26.709497081s) [noise=true , address=64.49.13.138:4490 , piece_id=2XTUZE6PXRE3OARNJVT36FXXWUMYIBVNE6M2VVFAVBGKEE2H5YZQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.875499841 (610h25m26.746582876s) [noise=true , address=64.49.13.139:4478 , piece_id=QN6RT2KWEA6APQVA66ZMKG4LH537UGYNOXOCVVA2YC2MKDOLT3YA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.891607221 (610h25m26.762690256s) [noise=true , address=145.79.227.20:4452 , piece_id=ZSDN2JZ3FQUVJYN5OURDXNKEDLDNULJOHKKRZVIPXZIWIYAHCRXQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.988561223 (610h25m26.859644268s) [noise=true , address=64.49.13.170:4720 , piece_id=HIATRSAPE4YLSODFZDPP43ZUMLDVM7GJEVMWE4ZGUQTP25LTHGTA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.039035911 (610h25m26.910118946s) [noise=true , address=77.90.31.14:4680 , piece_id=S2E4WBA7MJVAZLT5JG3IMHK53RQC77CCHV2PDF6L2R2JEA2U3K4Q, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.098023090 (610h25m26.969106126s) [noise=true , address=45.91.116.6:4981 , piece_id=CT474ZA2LVAZEBJHBDD4GB6NTHVHZFJLNE5OZV3DFS2NZKVW3EQQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.653790754 (610h25m27.524873789s) [noise=true , address=77.90.31.21:4886 , piece_id=K34WPIBGGGEPHIKJ253XGLDCIQ3WLFN4AZ6EYNWVHHT2EIHS6YRQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.684192972 (610h25m27.555276017s) [noise=true , address=193.149.164.143:4704 , piece_id=BHXHFBKTLTDMGXWY5QRTP2THZQE5OXFZ2LCMTO2KRYRWAHXWW3NQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.717238425 (610h25m27.58832146s) [noise=true , address=145.79.227.40:4084 , piece_id=RUKTKDII6BRUWSYVS65CWUBT3QKG3ASUK3KA3O5VGE53XSG6IMOQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.718349742 (610h25m27.589432777s) [noise=true , address=77.90.31.10:4523 , piece_id=P6LIONXJI6AFZ2JTKCQ2W6RZCOZCK6MZ53EXAJKXYQXSX2PX4QEA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.803639562 (610h25m27.674722597s) [noise=true , address=145.79.227.29:4971 , piece_id=MEPXHHOQ6SUZC3GIPCQEMOKKF5OJTKZEI3UYZBAYXAQZGSVEKSBA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.812791279 (610h25m27.683874314s) [noise=true , address=145.79.227.49:4343 , piece_id=6L3HPTS7D2NRA4QEOH4RFM3D3B53CTCONRI6G6YVU4OOTUWTG7XA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.812812348 (610h25m27.683895393s) [seg_pos={0 0}, upload=534952]: Segment reached optimal threshold of 14 pieces. Dec 31 01:54:27.813018000 (610h25m27.684101035s) [noise=true , address=145.79.227.35:4505 , piece_id=V7RS7A4QLZTH4G32ZUZNSF2Z5APVWZGJO77NEGP3SBWQT3WCMVLQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i6itGoBhLsLiqC8et8Y5L62fLjcWxy18u39NAr4R3nXNMWcsf): protocol: expected piece hash; context canceled; EOF Dec 31 01:54:27.813024772 (610h25m27.684107808s) [noise=true , address=145.79.227.35:4505 , piece_id=V7RS7A4QLZTH4G32ZUZNSF2Z5APVWZGJO77NEGP3SBWQT3WCMVLQ, seg_pos={0 0}, upload=534952]: Long tail cancellation due to optimal threshold Dec 31 01:54:27.813175931 (610h25m27.684258976s) [noise=true , address=145.79.227.34:4937 , piece_id=5W7LBM3YJJTVUFYTELR57DVXB663OHO5N3Z2SPQPZ4G43IK47O3Q, seg_pos={0 0}, upload=534952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1LKrFThdW3jJpG6dRZLUaxLXTa5Wy27yQ8PGUEqjvBWfFoRB94): protocol: expected piece hash; context canceled; EOF Dec 31 01:54:27.813181702 (610h25m27.684264737s) [noise=true , address=145.79.227.34:4937 , piece_id=5W7LBM3YJJTVUFYTELR57DVXB663OHO5N3Z2SPQPZ4G43IK47O3Q, seg_pos={0 0}, upload=534952]: Long tail cancellation due to optimal threshold Dec 31 01:54:27.813407862 (610h25m27.684490907s) [noise=true , address=145.79.227.2:4117 , piece_id=4SGFGB3IJ7LTXDMMWM2PGGOV64MC46IZ7OZQWH55J4NBQY7CHMZQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122Htg72f5eUUutf1p3fYQpy1Wkf6AgdaQV83HHYye2n8zm3the): protocol: expected piece hash; context canceled; EOF Dec 31 01:54:27.813414264 (610h25m27.684497299s) [noise=true , address=145.79.227.2:4117 , piece_id=4SGFGB3IJ7LTXDMMWM2PGGOV64MC46IZ7OZQWH55J4NBQY7CHMZQ, seg_pos={0 0}, upload=534952]: Long tail cancellation due to optimal threshold Dec 31 01:54:27.813424614 (610h25m27.684507649s) [noise=true , address=77.90.31.19:4848 , piece_id=VZYG32S6ODIPDKQYZRXECBHXOERZDDWD2OF35Z46W4XMUZ3OJLAA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD): protocol: write tcp 188.166.225.170:42842->77.90.31.19:4848: use of closed network connection; write tcp 188.166.225.170:42842->77.90.31.19:4848: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42842->77.90.31.19:4848: use of closed network connection Dec 31 01:54:27.813432750 (610h25m27.684515785s) [noise=true , address=77.90.31.19:4848 , piece_id=VZYG32S6ODIPDKQYZRXECBHXOERZDDWD2OF35Z46W4XMUZ3OJLAA, seg_pos={0 0}, upload=534952]: Long tail cancellation due to optimal threshold Dec 31 01:54:27.813474088 (610h25m27.684557123s) [seg_pos={0 0}, upload=534952]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 01:54:27.813535325 (610h25m27.68461836s) [upload=534952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 01:54:27.813539613 (610h25m27.684622648s) [upload=534952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 01:54:28.099921034 (610h25m27.971004069s) [upload=534952]: Done uploading segments... Dec 31 01:54:28.099940451 (610h25m27.971023486s) [upload=534952]: Upload finished. err: <nil> Duration: 52.026311705s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [XX.X....X..........X............X..X.......X.X..X....XX.XX..] duration: 5.708135306s trace: https://jaeger.dev.storj.io/trace/2f9cb5ff97a2ac44 FAIL (2025-12-31 02:16:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: XGFONRYPHMDUPP7RJLNRRUZ7KXPRFIH22WJKTFRVM7DWZHCR3VSQ) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: H2FC3TA5VGNJTA4ZFS6TCFN2DNWDNNZFR5WRX7QT6RKX4DVY5MIA) context deadline exceeded; piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: T5XK6HTXQQWVPISPFJ4ILOXCMCDD3URLTQZOMX5QZGG5IG5K6GDA) context deadline exceeded; piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: 6RIQ7G6CJOJNGTKLYY7EX4OQ3YBFEJKQAUG5XIJ2NJGJB4ULH2KA) context deadline exceeded ======LOGS====== Dec 31 02:15:34.470943883 (610h46m34.342026918s) [upload=535255]: Starting upload Dec 31 02:15:34.471018445 (610h46m34.34210148s) [upload=535255]: Done starting upload Dec 31 02:15:34.471184521 (610h46m34.342267566s) [upload=535255]: Uploading segments... Dec 31 02:15:34.471391205 (610h46m34.34247424s) [upload=535255]: Got next segment. Inline: false Dec 31 02:15:34.471450537 (610h46m34.342533572s) [upload=535255]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 02:15:34.471455456 (610h46m34.342538501s) [upload=535255]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 02:15:34.726848804 (610h46m34.597931849s) [seg_pos={0 0}, upload=535255]: Begin upload segment... Dec 31 02:15:34.726898429 (610h46m34.597981464s) [seg_pos={0 0}, upload=535255]: Done begin upload segment. Dec 31 02:15:34.726934788 (610h46m34.598017833s) [noise=true , address=145.79.227.16:4909 , piece_id=IEZDHX7O5NVWGLAJ4EQOK4F7RNHJBSFHB6C2TQ42EB7W7JATRMUQ, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.726975706 (610h46m34.598058741s) [noise=true , address=45.91.116.9:4263 , piece_id=6RIQ7G6CJOJNGTKLYY7EX4OQ3YBFEJKQAUG5XIJ2NJGJB4ULH2KA, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.727672262 (610h46m34.598755307s) [noise=true , address=77.90.31.19:4842 , piece_id=TMFJQ7VEVKCVQWYE6LGSF3AXEKYISQPRXDBZGVJUE63FD2EPS4RQ, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.727687450 (610h46m34.598770485s) [noise=true , address=145.79.227.31:4797 , piece_id=VGZA5SE3IY3VSBN3MLHUGPPCD5XYNJXNTCN522BCTKBFLCW34UTQ, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.728136986 (610h46m34.599220022s) [noise=true , address=145.79.227.34:4906 , piece_id=65OFQCDQITDTF4AHGDUZB6TY43UTXH4WFFS3MRI3LQD4M53GL3QQ, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.728259861 (610h46m34.599342906s) [noise=true , address=145.79.227.50:4375 , piece_id=QRAPDLJ3LM6WNYHLZ2VRYO2OIAVSZ3IQ5R62TLBPCAKVDAF2A3FQ, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.728640225 (610h46m34.59972326s) [noise=true , address=145.79.227.32:4835 , piece_id=L2ZTK326XY3DCKHDJHZ25E6MM6E4XN24XLDUBSCQ7OWXWDE34E5Q, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.728737830 (610h46m34.599820865s) [noise=true , address=145.79.227.35:4483 , piece_id=VOUM3VD3PNVSCKK26HQ3EKVE3DFZFQDDXXO3ULEGJKWODTFQZEFA, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.729042310 (610h46m34.600125355s) [noise=true , address=77.90.31.16:4736 , piece_id=CAIXHGWBKNOZ2JGNSN32NRQK6Y4CN5XCGOF32FJG27E4LTFDK4LA, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.729097326 (610h46m34.600180361s) [noise=true , address=145.79.227.21:4475 , piece_id=JTOGYE76I4LQSOP2HJIRROP4IFJESLLV353XQSEXFXJCPYPJE2CA, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.729374553 (610h46m34.600457589s) [noise=true , address=145.79.227.10:4402 , piece_id=XZBGK6UOLSDRIK6ZGH2XIWP2BWSFQK674DKAPZBEWCVD3H5GEBNQ, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.729450758 (610h46m34.600533793s) [noise=true , address=145.79.227.51:4409 , piece_id=WPGRG5MOZR525QAE5HGGSUPRFJ6LKBPQQAJWTMFTK4PPNVKWM4SQ, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.729852623 (610h46m34.600935658s) [noise=true , address=77.90.31.18:4805 , piece_id=XGFONRYPHMDUPP7RJLNRRUZ7KXPRFIH22WJKTFRVM7DWZHCR3VSQ, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.729870658 (610h46m34.600953693s) [noise=true , address=45.91.116.8:4045 , piece_id=H2FC3TA5VGNJTA4ZFS6TCFN2DNWDNNZFR5WRX7QT6RKX4DVY5MIA, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.730183253 (610h46m34.601266298s) [noise=true , address=77.90.31.15:4697 , piece_id=T5XK6HTXQQWVPISPFJ4ILOXCMCDD3URLTQZOMX5QZGG5IG5K6GDA, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.730255150 (610h46m34.601338185s) [noise=true , address=145.79.227.12:4767 , piece_id=3FIFSPGGXTLELWMFF3DVTSETVRY4RTTJ75HERKN3O3PJ4IH3PTOQ, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:36.422393419 (610h46m36.293476464s) [upload=535255]: Next returned nil segment Dec 31 02:15:36.422406284 (610h46m36.293489329s) [upload=535255]: Waiting for error group managing segments... Dec 31 02:15:36.710670932 (610h46m36.581753977s) [noise=true , address=145.79.227.31:4797 , piece_id=VGZA5SE3IY3VSBN3MLHUGPPCD5XYNJXNTCN522BCTKBFLCW34UTQ, seg_pos={0 0}, upload=535255]: Done uploading piece. err: <nil> Dec 31 02:15:36.711270734 (610h46m36.582353769s) [noise=true , address=145.79.227.32:4835 , piece_id=L2ZTK326XY3DCKHDJHZ25E6MM6E4XN24XLDUBSCQ7OWXWDE34E5Q, seg_pos={0 0}, upload=535255]: Done uploading piece. err: <nil> Dec 31 02:15:36.712146010 (610h46m36.583229045s) [noise=true , address=145.79.227.50:4375 , piece_id=QRAPDLJ3LM6WNYHLZ2VRYO2OIAVSZ3IQ5R62TLBPCAKVDAF2A3FQ, seg_pos={0 0}, upload=535255]: Done uploading piece. err: <nil> Dec 31 02:15:36.732635736 (610h46m36.603718771s) [noise=true , address=145.79.227.16:4909 , piece_id=IEZDHX7O5NVWGLAJ4EQOK4F7RNHJBSFHB6C2TQ42EB7W7JATRMUQ, seg_pos={0 0}, upload=535255]: Done uploading piece. err: <nil> Dec 31 02:15:36.737942711 (610h46m36.609025756s) [noise=true , address=145.79.227.21:4475 , piece_id=JTOGYE76I4LQSOP2HJIRROP4IFJESLLV353XQSEXFXJCPYPJE2CA, seg_pos={0 0}, upload=535255]: Done uploading piece. err: <nil> Dec 31 02:15:36.741292659 (610h46m36.612375694s) [noise=true , address=145.79.227.51:4409 , piece_id=WPGRG5MOZR525QAE5HGGSUPRFJ6LKBPQQAJWTMFTK4PPNVKWM4SQ, seg_pos={0 0}, upload=535255]: Done uploading piece. err: <nil> Dec 31 02:15:36.744308642 (610h46m36.615391677s) [noise=true , address=145.79.227.12:4767 , piece_id=3FIFSPGGXTLELWMFF3DVTSETVRY4RTTJ75HERKN3O3PJ4IH3PTOQ, seg_pos={0 0}, upload=535255]: Done uploading piece. err: <nil> Dec 31 02:15:36.795374577 (610h46m36.666457622s) [noise=true , address=145.79.227.10:4402 , piece_id=XZBGK6UOLSDRIK6ZGH2XIWP2BWSFQK674DKAPZBEWCVD3H5GEBNQ, seg_pos={0 0}, upload=535255]: Done uploading piece. err: <nil> Dec 31 02:15:36.881602926 (610h46m36.752685961s) [noise=true , address=77.90.31.18:4805 , piece_id=XGFONRYPHMDUPP7RJLNRRUZ7KXPRFIH22WJKTFRVM7DWZHCR3VSQ, seg_pos={0 0}, upload=535255]: Done uploading piece. err: <nil> Dec 31 02:15:37.071727813 (610h46m36.942810859s) [noise=true , address=45.91.116.9:4263 , piece_id=6RIQ7G6CJOJNGTKLYY7EX4OQ3YBFEJKQAUG5XIJ2NJGJB4ULH2KA, seg_pos={0 0}, upload=535255]: Done uploading piece. err: <nil> Dec 31 02:15:37.579913516 (610h46m37.450996561s) [noise=true , address=77.90.31.16:4736 , piece_id=CAIXHGWBKNOZ2JGNSN32NRQK6Y4CN5XCGOF32FJG27E4LTFDK4LA, seg_pos={0 0}, upload=535255]: Done uploading piece. err: <nil> Dec 31 02:15:37.737075300 (610h46m37.608158335s) [noise=true , address=77.90.31.15:4697 , piece_id=T5XK6HTXQQWVPISPFJ4ILOXCMCDD3URLTQZOMX5QZGG5IG5K6GDA, seg_pos={0 0}, upload=535255]: Done uploading piece. err: <nil> Dec 31 02:15:37.770719692 (610h46m37.641802727s) [noise=true , address=45.91.116.8:4045 , piece_id=H2FC3TA5VGNJTA4ZFS6TCFN2DNWDNNZFR5WRX7QT6RKX4DVY5MIA, seg_pos={0 0}, upload=535255]: Done uploading piece. err: <nil> Dec 31 02:15:37.770744329 (610h46m37.641827364s) [seg_pos={0 0}, upload=535255]: Segment reached optimal threshold of 13 pieces. Dec 31 02:15:37.770987743 (610h46m37.642070788s) [noise=true , address=145.79.227.35:4483 , piece_id=VOUM3VD3PNVSCKK26HQ3EKVE3DFZFQDDXXO3ULEGJKWODTFQZEFA, seg_pos={0 0}, upload=535255]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF Dec 31 02:15:37.770996148 (610h46m37.642079194s) [noise=true , address=145.79.227.35:4483 , piece_id=VOUM3VD3PNVSCKK26HQ3EKVE3DFZFQDDXXO3ULEGJKWODTFQZEFA, seg_pos={0 0}, upload=535255]: Long tail cancellation due to optimal threshold Dec 31 02:15:37.771089406 (610h46m37.642172441s) [noise=true , address=145.79.227.34:4906 , piece_id=65OFQCDQITDTF4AHGDUZB6TY43UTXH4WFFS3MRI3LQD4M53GL3QQ, seg_pos={0 0}, upload=535255]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 31 02:15:37.771104735 (610h46m37.64218778s) [noise=true , address=145.79.227.34:4906 , piece_id=65OFQCDQITDTF4AHGDUZB6TY43UTXH4WFFS3MRI3LQD4M53GL3QQ, seg_pos={0 0}, upload=535255]: Long tail cancellation due to optimal threshold Dec 31 02:15:37.771250514 (610h46m37.642333559s) [noise=true , address=77.90.31.19:4842 , piece_id=TMFJQ7VEVKCVQWYE6LGSF3AXEKYISQPRXDBZGVJUE63FD2EPS4RQ, seg_pos={0 0}, upload=535255]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Dec 31 02:15:37.771256555 (610h46m37.6423396s) [noise=true , address=77.90.31.19:4842 , piece_id=TMFJQ7VEVKCVQWYE6LGSF3AXEKYISQPRXDBZGVJUE63FD2EPS4RQ, seg_pos={0 0}, upload=535255]: Long tail cancellation due to optimal threshold Dec 31 02:15:37.771271012 (610h46m37.642354047s) [seg_pos={0 0}, upload=535255]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 02:15:37.771347367 (610h46m37.642430402s) [upload=535255]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 02:15:37.771351817 (610h46m37.642434852s) [upload=535255]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 02:15:38.072543669 (610h46m37.943626724s) [upload=535255]: Done uploading segments... Dec 31 02:15:38.072571552 (610h46m37.943654587s) [upload=535255]: Upload finished. err: <nil> Duration: 5.708231207s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [XXX.X....X..........X............X..X.......X.X..X....XX.XX.] duration: 6.26986983s trace: https://jaeger.dev.storj.io/trace/2f9cb5ff97a2ac44 FAIL (2025-12-31 02:16:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: XGFONRYPHMDUPP7RJLNRRUZ7KXPRFIH22WJKTFRVM7DWZHCR3VSQ) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: H2FC3TA5VGNJTA4ZFS6TCFN2DNWDNNZFR5WRX7QT6RKX4DVY5MIA) context deadline exceeded; piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: T5XK6HTXQQWVPISPFJ4ILOXCMCDD3URLTQZOMX5QZGG5IG5K6GDA) context deadline exceeded; piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: 6RIQ7G6CJOJNGTKLYY7EX4OQ3YBFEJKQAUG5XIJ2NJGJB4ULH2KA) context deadline exceeded ======LOGS====== Dec 31 02:15:34.470943883 (610h46m34.342026918s) [upload=535255]: Starting upload Dec 31 02:15:34.471018445 (610h46m34.34210148s) [upload=535255]: Done starting upload Dec 31 02:15:34.471184521 (610h46m34.342267566s) [upload=535255]: Uploading segments... Dec 31 02:15:34.471391205 (610h46m34.34247424s) [upload=535255]: Got next segment. Inline: false Dec 31 02:15:34.471450537 (610h46m34.342533572s) [upload=535255]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 02:15:34.471455456 (610h46m34.342538501s) [upload=535255]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 02:15:34.726848804 (610h46m34.597931849s) [seg_pos={0 0}, upload=535255]: Begin upload segment... Dec 31 02:15:34.726898429 (610h46m34.597981464s) [seg_pos={0 0}, upload=535255]: Done begin upload segment. Dec 31 02:15:34.726934788 (610h46m34.598017833s) [noise=true , address=145.79.227.16:4909 , piece_id=IEZDHX7O5NVWGLAJ4EQOK4F7RNHJBSFHB6C2TQ42EB7W7JATRMUQ, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.726975706 (610h46m34.598058741s) [noise=true , address=45.91.116.9:4263 , piece_id=6RIQ7G6CJOJNGTKLYY7EX4OQ3YBFEJKQAUG5XIJ2NJGJB4ULH2KA, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.727672262 (610h46m34.598755307s) [noise=true , address=77.90.31.19:4842 , piece_id=TMFJQ7VEVKCVQWYE6LGSF3AXEKYISQPRXDBZGVJUE63FD2EPS4RQ, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.727687450 (610h46m34.598770485s) [noise=true , address=145.79.227.31:4797 , piece_id=VGZA5SE3IY3VSBN3MLHUGPPCD5XYNJXNTCN522BCTKBFLCW34UTQ, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.728136986 (610h46m34.599220022s) [noise=true , address=145.79.227.34:4906 , piece_id=65OFQCDQITDTF4AHGDUZB6TY43UTXH4WFFS3MRI3LQD4M53GL3QQ, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.728259861 (610h46m34.599342906s) [noise=true , address=145.79.227.50:4375 , piece_id=QRAPDLJ3LM6WNYHLZ2VRYO2OIAVSZ3IQ5R62TLBPCAKVDAF2A3FQ, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.728640225 (610h46m34.59972326s) [noise=true , address=145.79.227.32:4835 , piece_id=L2ZTK326XY3DCKHDJHZ25E6MM6E4XN24XLDUBSCQ7OWXWDE34E5Q, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.728737830 (610h46m34.599820865s) [noise=true , address=145.79.227.35:4483 , piece_id=VOUM3VD3PNVSCKK26HQ3EKVE3DFZFQDDXXO3ULEGJKWODTFQZEFA, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.729042310 (610h46m34.600125355s) [noise=true , address=77.90.31.16:4736 , piece_id=CAIXHGWBKNOZ2JGNSN32NRQK6Y4CN5XCGOF32FJG27E4LTFDK4LA, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.729097326 (610h46m34.600180361s) [noise=true , address=145.79.227.21:4475 , piece_id=JTOGYE76I4LQSOP2HJIRROP4IFJESLLV353XQSEXFXJCPYPJE2CA, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.729374553 (610h46m34.600457589s) [noise=true , address=145.79.227.10:4402 , piece_id=XZBGK6UOLSDRIK6ZGH2XIWP2BWSFQK674DKAPZBEWCVD3H5GEBNQ, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.729450758 (610h46m34.600533793s) [noise=true , address=145.79.227.51:4409 , piece_id=WPGRG5MOZR525QAE5HGGSUPRFJ6LKBPQQAJWTMFTK4PPNVKWM4SQ, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.729852623 (610h46m34.600935658s) [noise=true , address=77.90.31.18:4805 , piece_id=XGFONRYPHMDUPP7RJLNRRUZ7KXPRFIH22WJKTFRVM7DWZHCR3VSQ, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.729870658 (610h46m34.600953693s) [noise=true , address=45.91.116.8:4045 , piece_id=H2FC3TA5VGNJTA4ZFS6TCFN2DNWDNNZFR5WRX7QT6RKX4DVY5MIA, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.730183253 (610h46m34.601266298s) [noise=true , address=77.90.31.15:4697 , piece_id=T5XK6HTXQQWVPISPFJ4ILOXCMCDD3URLTQZOMX5QZGG5IG5K6GDA, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.730255150 (610h46m34.601338185s) [noise=true , address=145.79.227.12:4767 , piece_id=3FIFSPGGXTLELWMFF3DVTSETVRY4RTTJ75HERKN3O3PJ4IH3PTOQ, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:36.422393419 (610h46m36.293476464s) [upload=535255]: Next returned nil segment Dec 31 02:15:36.422406284 (610h46m36.293489329s) [upload=535255]: Waiting for error group managing segments... Dec 31 02:15:36.710670932 (610h46m36.581753977s) [noise=true , address=145.79.227.31:4797 , piece_id=VGZA5SE3IY3VSBN3MLHUGPPCD5XYNJXNTCN522BCTKBFLCW34UTQ, seg_pos={0 0}, upload=535255]: Done uploading piece. err: <nil> Dec 31 02:15:36.711270734 (610h46m36.582353769s) [noise=true , address=145.79.227.32:4835 , piece_id=L2ZTK326XY3DCKHDJHZ25E6MM6E4XN24XLDUBSCQ7OWXWDE34E5Q, seg_pos={0 0}, upload=535255]: Done uploading piece. err: <nil> Dec 31 02:15:36.712146010 (610h46m36.583229045s) [noise=true , address=145.79.227.50:4375 , piece_id=QRAPDLJ3LM6WNYHLZ2VRYO2OIAVSZ3IQ5R62TLBPCAKVDAF2A3FQ, seg_pos={0 0}, upload=535255]: Done uploading piece. err: <nil> Dec 31 02:15:36.732635736 (610h46m36.603718771s) [noise=true , address=145.79.227.16:4909 , piece_id=IEZDHX7O5NVWGLAJ4EQOK4F7RNHJBSFHB6C2TQ42EB7W7JATRMUQ, seg_pos={0 0}, upload=535255]: Done uploading piece. err: <nil> Dec 31 02:15:36.737942711 (610h46m36.609025756s) [noise=true , address=145.79.227.21:4475 , piece_id=JTOGYE76I4LQSOP2HJIRROP4IFJESLLV353XQSEXFXJCPYPJE2CA, seg_pos={0 0}, upload=535255]: Done uploading piece. err: <nil> Dec 31 02:15:36.741292659 (610h46m36.612375694s) [noise=true , address=145.79.227.51:4409 , piece_id=WPGRG5MOZR525QAE5HGGSUPRFJ6LKBPQQAJWTMFTK4PPNVKWM4SQ, seg_pos={0 0}, upload=535255]: Done uploading piece. err: <nil> Dec 31 02:15:36.744308642 (610h46m36.615391677s) [noise=true , address=145.79.227.12:4767 , piece_id=3FIFSPGGXTLELWMFF3DVTSETVRY4RTTJ75HERKN3O3PJ4IH3PTOQ, seg_pos={0 0}, upload=535255]: Done uploading piece. err: <nil> Dec 31 02:15:36.795374577 (610h46m36.666457622s) [noise=true , address=145.79.227.10:4402 , piece_id=XZBGK6UOLSDRIK6ZGH2XIWP2BWSFQK674DKAPZBEWCVD3H5GEBNQ, seg_pos={0 0}, upload=535255]: Done uploading piece. err: <nil> Dec 31 02:15:36.881602926 (610h46m36.752685961s) [noise=true , address=77.90.31.18:4805 , piece_id=XGFONRYPHMDUPP7RJLNRRUZ7KXPRFIH22WJKTFRVM7DWZHCR3VSQ, seg_pos={0 0}, upload=535255]: Done uploading piece. err: <nil> Dec 31 02:15:37.071727813 (610h46m36.942810859s) [noise=true , address=45.91.116.9:4263 , piece_id=6RIQ7G6CJOJNGTKLYY7EX4OQ3YBFEJKQAUG5XIJ2NJGJB4ULH2KA, seg_pos={0 0}, upload=535255]: Done uploading piece. err: <nil> Dec 31 02:15:37.579913516 (610h46m37.450996561s) [noise=true , address=77.90.31.16:4736 , piece_id=CAIXHGWBKNOZ2JGNSN32NRQK6Y4CN5XCGOF32FJG27E4LTFDK4LA, seg_pos={0 0}, upload=535255]: Done uploading piece. err: <nil> Dec 31 02:15:37.737075300 (610h46m37.608158335s) [noise=true , address=77.90.31.15:4697 , piece_id=T5XK6HTXQQWVPISPFJ4ILOXCMCDD3URLTQZOMX5QZGG5IG5K6GDA, seg_pos={0 0}, upload=535255]: Done uploading piece. err: <nil> Dec 31 02:15:37.770719692 (610h46m37.641802727s) [noise=true , address=45.91.116.8:4045 , piece_id=H2FC3TA5VGNJTA4ZFS6TCFN2DNWDNNZFR5WRX7QT6RKX4DVY5MIA, seg_pos={0 0}, upload=535255]: Done uploading piece. err: <nil> Dec 31 02:15:37.770744329 (610h46m37.641827364s) [seg_pos={0 0}, upload=535255]: Segment reached optimal threshold of 13 pieces. Dec 31 02:15:37.770987743 (610h46m37.642070788s) [noise=true , address=145.79.227.35:4483 , piece_id=VOUM3VD3PNVSCKK26HQ3EKVE3DFZFQDDXXO3ULEGJKWODTFQZEFA, seg_pos={0 0}, upload=535255]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF Dec 31 02:15:37.770996148 (610h46m37.642079194s) [noise=true , address=145.79.227.35:4483 , piece_id=VOUM3VD3PNVSCKK26HQ3EKVE3DFZFQDDXXO3ULEGJKWODTFQZEFA, seg_pos={0 0}, upload=535255]: Long tail cancellation due to optimal threshold Dec 31 02:15:37.771089406 (610h46m37.642172441s) [noise=true , address=145.79.227.34:4906 , piece_id=65OFQCDQITDTF4AHGDUZB6TY43UTXH4WFFS3MRI3LQD4M53GL3QQ, seg_pos={0 0}, upload=535255]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 31 02:15:37.771104735 (610h46m37.64218778s) [noise=true , address=145.79.227.34:4906 , piece_id=65OFQCDQITDTF4AHGDUZB6TY43UTXH4WFFS3MRI3LQD4M53GL3QQ, seg_pos={0 0}, upload=535255]: Long tail cancellation due to optimal threshold Dec 31 02:15:37.771250514 (610h46m37.642333559s) [noise=true , address=77.90.31.19:4842 , piece_id=TMFJQ7VEVKCVQWYE6LGSF3AXEKYISQPRXDBZGVJUE63FD2EPS4RQ, seg_pos={0 0}, upload=535255]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Dec 31 02:15:37.771256555 (610h46m37.6423396s) [noise=true , address=77.90.31.19:4842 , piece_id=TMFJQ7VEVKCVQWYE6LGSF3AXEKYISQPRXDBZGVJUE63FD2EPS4RQ, seg_pos={0 0}, upload=535255]: Long tail cancellation due to optimal threshold Dec 31 02:15:37.771271012 (610h46m37.642354047s) [seg_pos={0 0}, upload=535255]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 02:15:37.771347367 (610h46m37.642430402s) [upload=535255]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 02:15:37.771351817 (610h46m37.642434852s) [upload=535255]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 02:15:38.072543669 (610h46m37.943626724s) [upload=535255]: Done uploading segments... Dec 31 02:15:38.072571552 (610h46m37.943654587s) [upload=535255]: Upload finished. err: <nil> Duration: 6.269959771s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [................X..................................X........] duration: 3.670689743s trace: https://jaeger.dev.storj.io/trace/740aa405bd68150f FAIL (2025-12-31 02:16:27 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.670764716s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [........X..........................X........................] duration: 37.922341872s trace: https://jaeger.dev.storj.io/trace/365693fea00df28 FAIL (2025-12-31 02:16:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [noise=true , address=145.79.227.49:4343 , piece_id=6L3HPTS7D2NRA4QEOH4RFM3D3B53CTCONRI6G6YVU4OOTUWTG7XA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.942060847 (610h25m24.813143892s) [noise=true , address=145.79.227.34:4937 , piece_id=5W7LBM3YJJTVUFYTELR57DVXB663OHO5N3Z2SPQPZ4G43IK47O3Q, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.942684104 (610h25m24.813767139s) [noise=true , address=64.49.13.139:4478 , piece_id=QN6RT2KWEA6APQVA66ZMKG4LH537UGYNOXOCVVA2YC2MKDOLT3YA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.943064007 (610h25m24.814147042s) [noise=true , address=145.79.227.40:4084 , piece_id=RUKTKDII6BRUWSYVS65CWUBT3QKG3ASUK3KA3O5VGE53XSG6IMOQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.943459560 (610h25m24.814542605s) [noise=true , address=145.79.227.14:4847 , piece_id=JX5NXTMVM3NSWIEVR4RX46ABSNPKTOFDLFBPKEXFQ3G3ANKV5OLA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.943865303 (610h25m24.814948348s) [noise=true , address=45.91.116.6:4981 , piece_id=CT474ZA2LVAZEBJHBDD4GB6NTHVHZFJLNE5OZV3DFS2NZKVW3EQQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.944259694 (610h25m24.815342749s) [noise=true , address=145.79.227.51:4443 , piece_id=73UUMRDY7BSZNCGBXCZYYRBVE5KLPMNPRNXOZEGWKQGLHJS32J2Q, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.946575182 (610h25m24.817658227s) [noise=true , address=77.90.31.10:4523 , piece_id=P6LIONXJI6AFZ2JTKCQ2W6RZCOZCK6MZ53EXAJKXYQXSX2PX4QEA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.948132909 (610h25m24.819215944s) [noise=true , address=145.79.227.20:4452 , piece_id=ZSDN2JZ3FQUVJYN5OURDXNKEDLDNULJOHKKRZVIPXZIWIYAHCRXQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.949605774 (610h25m24.820688819s) [noise=true , address=77.90.31.14:4680 , piece_id=S2E4WBA7MJVAZLT5JG3IMHK53RQC77CCHV2PDF6L2R2JEA2U3K4Q, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.951459002 (610h25m24.822542048s) [noise=true , address=193.149.164.143:4704 , piece_id=BHXHFBKTLTDMGXWY5QRTP2THZQE5OXFZ2LCMTO2KRYRWAHXWW3NQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.952986742 (610h25m24.824069777s) [noise=true , address=77.90.31.19:4848 , piece_id=VZYG32S6ODIPDKQYZRXECBHXOERZDDWD2OF35Z46W4XMUZ3OJLAA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.956916654 (610h25m24.827999689s) [noise=true , address=145.79.227.29:4971 , piece_id=MEPXHHOQ6SUZC3GIPCQEMOKKF5OJTKZEI3UYZBAYXAQZGSVEKSBA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.960665062 (610h25m24.831748097s) [noise=true , address=64.49.13.170:4720 , piece_id=HIATRSAPE4YLSODFZDPP43ZUMLDVM7GJEVMWE4ZGUQTP25LTHGTA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.962366001 (610h25m24.833449036s) [noise=true , address=145.79.227.2:4117 , piece_id=4SGFGB3IJ7LTXDMMWM2PGGOV64MC46IZ7OZQWH55J4NBQY7CHMZQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.963708216 (610h25m24.834791261s) [noise=true , address=145.79.227.35:4505 , piece_id=V7RS7A4QLZTH4G32ZUZNSF2Z5APVWZGJO77NEGP3SBWQT3WCMVLQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:26.352236138 (610h25m26.223319173s) [upload=534952]: Next returned nil segment Dec 31 01:54:26.352253821 (610h25m26.223336857s) [upload=534952]: Waiting for error group managing segments... Dec 31 01:54:26.812758374 (610h25m26.683841409s) [noise=true , address=145.79.227.51:4443 , piece_id=73UUMRDY7BSZNCGBXCZYYRBVE5KLPMNPRNXOZEGWKQGLHJS32J2Q, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.824371056 (610h25m26.695454101s) [noise=true , address=145.79.227.14:4847 , piece_id=JX5NXTMVM3NSWIEVR4RX46ABSNPKTOFDLFBPKEXFQ3G3ANKV5OLA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.838414046 (610h25m26.709497081s) [noise=true , address=64.49.13.138:4490 , piece_id=2XTUZE6PXRE3OARNJVT36FXXWUMYIBVNE6M2VVFAVBGKEE2H5YZQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.875499841 (610h25m26.746582876s) [noise=true , address=64.49.13.139:4478 , piece_id=QN6RT2KWEA6APQVA66ZMKG4LH537UGYNOXOCVVA2YC2MKDOLT3YA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.891607221 (610h25m26.762690256s) [noise=true , address=145.79.227.20:4452 , piece_id=ZSDN2JZ3FQUVJYN5OURDXNKEDLDNULJOHKKRZVIPXZIWIYAHCRXQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.988561223 (610h25m26.859644268s) [noise=true , address=64.49.13.170:4720 , piece_id=HIATRSAPE4YLSODFZDPP43ZUMLDVM7GJEVMWE4ZGUQTP25LTHGTA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.039035911 (610h25m26.910118946s) [noise=true , address=77.90.31.14:4680 , piece_id=S2E4WBA7MJVAZLT5JG3IMHK53RQC77CCHV2PDF6L2R2JEA2U3K4Q, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.098023090 (610h25m26.969106126s) [noise=true , address=45.91.116.6:4981 , piece_id=CT474ZA2LVAZEBJHBDD4GB6NTHVHZFJLNE5OZV3DFS2NZKVW3EQQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.653790754 (610h25m27.524873789s) [noise=true , address=77.90.31.21:4886 , piece_id=K34WPIBGGGEPHIKJ253XGLDCIQ3WLFN4AZ6EYNWVHHT2EIHS6YRQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.684192972 (610h25m27.555276017s) [noise=true , address=193.149.164.143:4704 , piece_id=BHXHFBKTLTDMGXWY5QRTP2THZQE5OXFZ2LCMTO2KRYRWAHXWW3NQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.717238425 (610h25m27.58832146s) [noise=true , address=145.79.227.40:4084 , piece_id=RUKTKDII6BRUWSYVS65CWUBT3QKG3ASUK3KA3O5VGE53XSG6IMOQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.718349742 (610h25m27.589432777s) [noise=true , address=77.90.31.10:4523 , piece_id=P6LIONXJI6AFZ2JTKCQ2W6RZCOZCK6MZ53EXAJKXYQXSX2PX4QEA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.803639562 (610h25m27.674722597s) [noise=true , address=145.79.227.29:4971 , piece_id=MEPXHHOQ6SUZC3GIPCQEMOKKF5OJTKZEI3UYZBAYXAQZGSVEKSBA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.812791279 (610h25m27.683874314s) [noise=true , address=145.79.227.49:4343 , piece_id=6L3HPTS7D2NRA4QEOH4RFM3D3B53CTCONRI6G6YVU4OOTUWTG7XA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.812812348 (610h25m27.683895393s) [seg_pos={0 0}, upload=534952]: Segment reached optimal threshold of 14 pieces. Dec 31 01:54:27.813018000 (610h25m27.684101035s) [noise=true , address=145.79.227.35:4505 , piece_id=V7RS7A4QLZTH4G32ZUZNSF2Z5APVWZGJO77NEGP3SBWQT3WCMVLQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i6itGoBhLsLiqC8et8Y5L62fLjcWxy18u39NAr4R3nXNMWcsf): protocol: expected piece hash; context canceled; EOF Dec 31 01:54:27.813024772 (610h25m27.684107808s) [noise=true , address=145.79.227.35:4505 , piece_id=V7RS7A4QLZTH4G32ZUZNSF2Z5APVWZGJO77NEGP3SBWQT3WCMVLQ, seg_pos={0 0}, upload=534952]: Long tail cancellation due to optimal threshold Dec 31 01:54:27.813175931 (610h25m27.684258976s) [noise=true , address=145.79.227.34:4937 , piece_id=5W7LBM3YJJTVUFYTELR57DVXB663OHO5N3Z2SPQPZ4G43IK47O3Q, seg_pos={0 0}, upload=534952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1LKrFThdW3jJpG6dRZLUaxLXTa5Wy27yQ8PGUEqjvBWfFoRB94): protocol: expected piece hash; context canceled; EOF Dec 31 01:54:27.813181702 (610h25m27.684264737s) [noise=true , address=145.79.227.34:4937 , piece_id=5W7LBM3YJJTVUFYTELR57DVXB663OHO5N3Z2SPQPZ4G43IK47O3Q, seg_pos={0 0}, upload=534952]: Long tail cancellation due to optimal threshold Dec 31 01:54:27.813407862 (610h25m27.684490907s) [noise=true , address=145.79.227.2:4117 , piece_id=4SGFGB3IJ7LTXDMMWM2PGGOV64MC46IZ7OZQWH55J4NBQY7CHMZQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122Htg72f5eUUutf1p3fYQpy1Wkf6AgdaQV83HHYye2n8zm3the): protocol: expected piece hash; context canceled; EOF Dec 31 01:54:27.813414264 (610h25m27.684497299s) [noise=true , address=145.79.227.2:4117 , piece_id=4SGFGB3IJ7LTXDMMWM2PGGOV64MC46IZ7OZQWH55J4NBQY7CHMZQ, seg_pos={0 0}, upload=534952]: Long tail cancellation due to optimal threshold Dec 31 01:54:27.813424614 (610h25m27.684507649s) [noise=true , address=77.90.31.19:4848 , piece_id=VZYG32S6ODIPDKQYZRXECBHXOERZDDWD2OF35Z46W4XMUZ3OJLAA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD): protocol: write tcp 188.166.225.170:42842->77.90.31.19:4848: use of closed network connection; write tcp 188.166.225.170:42842->77.90.31.19:4848: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42842->77.90.31.19:4848: use of closed network connection Dec 31 01:54:27.813432750 (610h25m27.684515785s) [noise=true , address=77.90.31.19:4848 , piece_id=VZYG32S6ODIPDKQYZRXECBHXOERZDDWD2OF35Z46W4XMUZ3OJLAA, seg_pos={0 0}, upload=534952]: Long tail cancellation due to optimal threshold Dec 31 01:54:27.813474088 (610h25m27.684557123s) [seg_pos={0 0}, upload=534952]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 01:54:27.813535325 (610h25m27.68461836s) [upload=534952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 01:54:27.813539613 (610h25m27.684622648s) [upload=534952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 01:54:28.099921034 (610h25m27.971004069s) [upload=534952]: Done uploading segments... Dec 31 01:54:28.099940451 (610h25m27.971023486s) [upload=534952]: Upload finished. err: <nil> Duration: 37.922377128s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.................X..................................X.......] duration: 3.529867902s trace: https://jaeger.dev.storj.io/trace/740aa405bd68150f FAIL (2025-12-31 02:16:04 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.529921834s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.XXX.X....X..........X............X..X.......X.X..X....XX.XX] duration: 30.00321952s trace: https://jaeger.dev.storj.io/trace/2f9cb5ff97a2ac44 FAIL (2025-12-31 02:16:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: XGFONRYPHMDUPP7RJLNRRUZ7KXPRFIH22WJKTFRVM7DWZHCR3VSQ) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: H2FC3TA5VGNJTA4ZFS6TCFN2DNWDNNZFR5WRX7QT6RKX4DVY5MIA) context deadline exceeded; piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: T5XK6HTXQQWVPISPFJ4ILOXCMCDD3URLTQZOMX5QZGG5IG5K6GDA) context deadline exceeded; piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: 6RIQ7G6CJOJNGTKLYY7EX4OQ3YBFEJKQAUG5XIJ2NJGJB4ULH2KA) context deadline exceeded ======LOGS====== Dec 31 02:15:34.470943883 (610h46m34.342026918s) [upload=535255]: Starting upload Dec 31 02:15:34.471018445 (610h46m34.34210148s) [upload=535255]: Done starting upload Dec 31 02:15:34.471184521 (610h46m34.342267566s) [upload=535255]: Uploading segments... Dec 31 02:15:34.471391205 (610h46m34.34247424s) [upload=535255]: Got next segment. Inline: false Dec 31 02:15:34.471450537 (610h46m34.342533572s) [upload=535255]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 02:15:34.471455456 (610h46m34.342538501s) [upload=535255]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 02:15:34.726848804 (610h46m34.597931849s) [seg_pos={0 0}, upload=535255]: Begin upload segment... Dec 31 02:15:34.726898429 (610h46m34.597981464s) [seg_pos={0 0}, upload=535255]: Done begin upload segment. Dec 31 02:15:34.726934788 (610h46m34.598017833s) [noise=true , address=145.79.227.16:4909 , piece_id=IEZDHX7O5NVWGLAJ4EQOK4F7RNHJBSFHB6C2TQ42EB7W7JATRMUQ, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.726975706 (610h46m34.598058741s) [noise=true , address=45.91.116.9:4263 , piece_id=6RIQ7G6CJOJNGTKLYY7EX4OQ3YBFEJKQAUG5XIJ2NJGJB4ULH2KA, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.727672262 (610h46m34.598755307s) [noise=true , address=77.90.31.19:4842 , piece_id=TMFJQ7VEVKCVQWYE6LGSF3AXEKYISQPRXDBZGVJUE63FD2EPS4RQ, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.727687450 (610h46m34.598770485s) [noise=true , address=145.79.227.31:4797 , piece_id=VGZA5SE3IY3VSBN3MLHUGPPCD5XYNJXNTCN522BCTKBFLCW34UTQ, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.728136986 (610h46m34.599220022s) [noise=true , address=145.79.227.34:4906 , piece_id=65OFQCDQITDTF4AHGDUZB6TY43UTXH4WFFS3MRI3LQD4M53GL3QQ, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.728259861 (610h46m34.599342906s) [noise=true , address=145.79.227.50:4375 , piece_id=QRAPDLJ3LM6WNYHLZ2VRYO2OIAVSZ3IQ5R62TLBPCAKVDAF2A3FQ, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.728640225 (610h46m34.59972326s) [noise=true , address=145.79.227.32:4835 , piece_id=L2ZTK326XY3DCKHDJHZ25E6MM6E4XN24XLDUBSCQ7OWXWDE34E5Q, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.728737830 (610h46m34.599820865s) [noise=true , address=145.79.227.35:4483 , piece_id=VOUM3VD3PNVSCKK26HQ3EKVE3DFZFQDDXXO3ULEGJKWODTFQZEFA, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.729042310 (610h46m34.600125355s) [noise=true , address=77.90.31.16:4736 , piece_id=CAIXHGWBKNOZ2JGNSN32NRQK6Y4CN5XCGOF32FJG27E4LTFDK4LA, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.729097326 (610h46m34.600180361s) [noise=true , address=145.79.227.21:4475 , piece_id=JTOGYE76I4LQSOP2HJIRROP4IFJESLLV353XQSEXFXJCPYPJE2CA, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.729374553 (610h46m34.600457589s) [noise=true , address=145.79.227.10:4402 , piece_id=XZBGK6UOLSDRIK6ZGH2XIWP2BWSFQK674DKAPZBEWCVD3H5GEBNQ, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.729450758 (610h46m34.600533793s) [noise=true , address=145.79.227.51:4409 , piece_id=WPGRG5MOZR525QAE5HGGSUPRFJ6LKBPQQAJWTMFTK4PPNVKWM4SQ, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.729852623 (610h46m34.600935658s) [noise=true , address=77.90.31.18:4805 , piece_id=XGFONRYPHMDUPP7RJLNRRUZ7KXPRFIH22WJKTFRVM7DWZHCR3VSQ, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.729870658 (610h46m34.600953693s) [noise=true , address=45.91.116.8:4045 , piece_id=H2FC3TA5VGNJTA4ZFS6TCFN2DNWDNNZFR5WRX7QT6RKX4DVY5MIA, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.730183253 (610h46m34.601266298s) [noise=true , address=77.90.31.15:4697 , piece_id=T5XK6HTXQQWVPISPFJ4ILOXCMCDD3URLTQZOMX5QZGG5IG5K6GDA, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:34.730255150 (610h46m34.601338185s) [noise=true , address=145.79.227.12:4767 , piece_id=3FIFSPGGXTLELWMFF3DVTSETVRY4RTTJ75HERKN3O3PJ4IH3PTOQ, seg_pos={0 0}, upload=535255]: Uploading piece... Dec 31 02:15:36.422393419 (610h46m36.293476464s) [upload=535255]: Next returned nil segment Dec 31 02:15:36.422406284 (610h46m36.293489329s) [upload=535255]: Waiting for error group managing segments... Dec 31 02:15:36.710670932 (610h46m36.581753977s) [noise=true , address=145.79.227.31:4797 , piece_id=VGZA5SE3IY3VSBN3MLHUGPPCD5XYNJXNTCN522BCTKBFLCW34UTQ, seg_pos={0 0}, upload=535255]: Done uploading piece. err: <nil> Dec 31 02:15:36.711270734 (610h46m36.582353769s) [noise=true , address=145.79.227.32:4835 , piece_id=L2ZTK326XY3DCKHDJHZ25E6MM6E4XN24XLDUBSCQ7OWXWDE34E5Q, seg_pos={0 0}, upload=535255]: Done uploading piece. err: <nil> Dec 31 02:15:36.712146010 (610h46m36.583229045s) [noise=true , address=145.79.227.50:4375 , piece_id=QRAPDLJ3LM6WNYHLZ2VRYO2OIAVSZ3IQ5R62TLBPCAKVDAF2A3FQ, seg_pos={0 0}, upload=535255]: Done uploading piece. err: <nil> Dec 31 02:15:36.732635736 (610h46m36.603718771s) [noise=true , address=145.79.227.16:4909 , piece_id=IEZDHX7O5NVWGLAJ4EQOK4F7RNHJBSFHB6C2TQ42EB7W7JATRMUQ, seg_pos={0 0}, upload=535255]: Done uploading piece. err: <nil> Dec 31 02:15:36.737942711 (610h46m36.609025756s) [noise=true , address=145.79.227.21:4475 , piece_id=JTOGYE76I4LQSOP2HJIRROP4IFJESLLV353XQSEXFXJCPYPJE2CA, seg_pos={0 0}, upload=535255]: Done uploading piece. err: <nil> Dec 31 02:15:36.741292659 (610h46m36.612375694s) [noise=true , address=145.79.227.51:4409 , piece_id=WPGRG5MOZR525QAE5HGGSUPRFJ6LKBPQQAJWTMFTK4PPNVKWM4SQ, seg_pos={0 0}, upload=535255]: Done uploading piece. err: <nil> Dec 31 02:15:36.744308642 (610h46m36.615391677s) [noise=true , address=145.79.227.12:4767 , piece_id=3FIFSPGGXTLELWMFF3DVTSETVRY4RTTJ75HERKN3O3PJ4IH3PTOQ, seg_pos={0 0}, upload=535255]: Done uploading piece. err: <nil> Dec 31 02:15:36.795374577 (610h46m36.666457622s) [noise=true , address=145.79.227.10:4402 , piece_id=XZBGK6UOLSDRIK6ZGH2XIWP2BWSFQK674DKAPZBEWCVD3H5GEBNQ, seg_pos={0 0}, upload=535255]: Done uploading piece. err: <nil> Dec 31 02:15:36.881602926 (610h46m36.752685961s) [noise=true , address=77.90.31.18:4805 , piece_id=XGFONRYPHMDUPP7RJLNRRUZ7KXPRFIH22WJKTFRVM7DWZHCR3VSQ, seg_pos={0 0}, upload=535255]: Done uploading piece. err: <nil> Dec 31 02:15:37.071727813 (610h46m36.942810859s) [noise=true , address=45.91.116.9:4263 , piece_id=6RIQ7G6CJOJNGTKLYY7EX4OQ3YBFEJKQAUG5XIJ2NJGJB4ULH2KA, seg_pos={0 0}, upload=535255]: Done uploading piece. err: <nil> Dec 31 02:15:37.579913516 (610h46m37.450996561s) [noise=true , address=77.90.31.16:4736 , piece_id=CAIXHGWBKNOZ2JGNSN32NRQK6Y4CN5XCGOF32FJG27E4LTFDK4LA, seg_pos={0 0}, upload=535255]: Done uploading piece. err: <nil> Dec 31 02:15:37.737075300 (610h46m37.608158335s) [noise=true , address=77.90.31.15:4697 , piece_id=T5XK6HTXQQWVPISPFJ4ILOXCMCDD3URLTQZOMX5QZGG5IG5K6GDA, seg_pos={0 0}, upload=535255]: Done uploading piece. err: <nil> Dec 31 02:15:37.770719692 (610h46m37.641802727s) [noise=true , address=45.91.116.8:4045 , piece_id=H2FC3TA5VGNJTA4ZFS6TCFN2DNWDNNZFR5WRX7QT6RKX4DVY5MIA, seg_pos={0 0}, upload=535255]: Done uploading piece. err: <nil> Dec 31 02:15:37.770744329 (610h46m37.641827364s) [seg_pos={0 0}, upload=535255]: Segment reached optimal threshold of 13 pieces. Dec 31 02:15:37.770987743 (610h46m37.642070788s) [noise=true , address=145.79.227.35:4483 , piece_id=VOUM3VD3PNVSCKK26HQ3EKVE3DFZFQDDXXO3ULEGJKWODTFQZEFA, seg_pos={0 0}, upload=535255]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF Dec 31 02:15:37.770996148 (610h46m37.642079194s) [noise=true , address=145.79.227.35:4483 , piece_id=VOUM3VD3PNVSCKK26HQ3EKVE3DFZFQDDXXO3ULEGJKWODTFQZEFA, seg_pos={0 0}, upload=535255]: Long tail cancellation due to optimal threshold Dec 31 02:15:37.771089406 (610h46m37.642172441s) [noise=true , address=145.79.227.34:4906 , piece_id=65OFQCDQITDTF4AHGDUZB6TY43UTXH4WFFS3MRI3LQD4M53GL3QQ, seg_pos={0 0}, upload=535255]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 31 02:15:37.771104735 (610h46m37.64218778s) [noise=true , address=145.79.227.34:4906 , piece_id=65OFQCDQITDTF4AHGDUZB6TY43UTXH4WFFS3MRI3LQD4M53GL3QQ, seg_pos={0 0}, upload=535255]: Long tail cancellation due to optimal threshold Dec 31 02:15:37.771250514 (610h46m37.642333559s) [noise=true , address=77.90.31.19:4842 , piece_id=TMFJQ7VEVKCVQWYE6LGSF3AXEKYISQPRXDBZGVJUE63FD2EPS4RQ, seg_pos={0 0}, upload=535255]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Dec 31 02:15:37.771256555 (610h46m37.6423396s) [noise=true , address=77.90.31.19:4842 , piece_id=TMFJQ7VEVKCVQWYE6LGSF3AXEKYISQPRXDBZGVJUE63FD2EPS4RQ, seg_pos={0 0}, upload=535255]: Long tail cancellation due to optimal threshold Dec 31 02:15:37.771271012 (610h46m37.642354047s) [seg_pos={0 0}, upload=535255]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 02:15:37.771347367 (610h46m37.642430402s) [upload=535255]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 02:15:37.771351817 (610h46m37.642434852s) [upload=535255]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 02:15:38.072543669 (610h46m37.943626724s) [upload=535255]: Done uploading segments... Dec 31 02:15:38.072571552 (610h46m37.943654587s) [upload=535255]: Upload finished. err: <nil> Duration: 30.003250969s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..................X..................................X......] duration: 3.519498659s trace: https://jaeger.dev.storj.io/trace/740aa405bd68150f FAIL (2025-12-31 02:15:25 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.51953147s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [..XXX.X....X..........X............X..X.......X.X..X....XX.X] duration: 30.003353882s trace: https://jaeger.dev.storj.io/trace/4f8dd902ce665b21 FAIL (2025-12-31 02:15:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, Piece ID: ZCI467B65CWICFCNVROWMOLRTSXPRXEGW7UAISQQXB754IJRSLLQ) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: LTEONU5EG4ZVDMTGJM5RGCQ4XAILCTAFLCWBXKPESDZRNUA2T75A) context deadline exceeded ======LOGS====== :37.786870284 (610h45m37.657953329s) [upload=535239]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 02:14:38.039161925 (610h45m37.91024498s) [seg_pos={0 0}, upload=535239]: Begin upload segment... Dec 31 02:14:38.039221138 (610h45m37.910304203s) [seg_pos={0 0}, upload=535239]: Done begin upload segment. Dec 31 02:14:38.039254552 (610h45m37.910337607s) [noise=true , address=145.79.227.5:4222 , piece_id=OHBW2VVVPY5MBVRNMNICGPBDBULGLWIXBZZFRCYLLIBHBDK65FQA, seg_pos={0 0}, upload=535239]: Uploading piece... Dec 31 02:14:38.039310277 (610h45m37.910393322s) [noise=true , address=45.91.116.4:4940 , piece_id=4IJOPYTYRZXNK6Y5MSM4FBR5A777UN2GSBWAZUAYFFS7ALAHUDIQ, seg_pos={0 0}, upload=535239]: Uploading piece... Dec 31 02:14:38.039834195 (610h45m37.91091724s) [noise=true , address=45.91.116.11:4117 , piece_id=LTEONU5EG4ZVDMTGJM5RGCQ4XAILCTAFLCWBXKPESDZRNUA2T75A, seg_pos={0 0}, upload=535239]: Uploading piece... Dec 31 02:14:38.039963241 (610h45m37.911046286s) [noise=true , address=145.79.227.47:4265 , piece_id=52RP3UK5N5IFWM7NLKPB5E4XI3FVOXQLZTM2ARXXJBVTU7WMNFZA, seg_pos={0 0}, upload=535239]: Uploading piece... Dec 31 02:14:38.040190644 (610h45m37.911273689s) [noise=true , address=145.79.227.13:4803 , piece_id=J6ZXXRT2C5LKRHSHFEH4ZPSE6EWODQ2PY5SS6M7PU3W4IBBJMZYA, seg_pos={0 0}, upload=535239]: Uploading piece... Dec 31 02:14:38.040326363 (610h45m37.911409408s) [noise=true , address=145.79.227.25:4617 , piece_id=VY6UV35MA6P5SMSEOBRVXKJTKI7XCRFPBP2NOOC3FHYIYEURAXCQ, seg_pos={0 0}, upload=535239]: Uploading piece... Dec 31 02:14:38.040530851 (610h45m37.911613896s) [noise=true , address=145.79.227.20:4437 , piece_id=MAYAWLZUZKK4D36V7NJEOQO6NUOVVYUXWU2HVFYISZBK636CJNPA, seg_pos={0 0}, upload=535239]: Uploading piece... Dec 31 02:14:38.040737385 (610h45m37.91182043s) [noise=true , address=45.91.116.14:4192 , piece_id=66FTF7LNKPLJGOKDAO2TKO2NDGMLZB4RWVQLFX6WJJYJL6QA6X5Q, seg_pos={0 0}, upload=535239]: Uploading piece... Dec 31 02:14:38.040885107 (610h45m37.911968142s) [noise=true , address=145.79.227.16:4912 , piece_id=NLZMXVBELJUSC3YOX4WH7CXZVSE6PW7BGAO5UOFABB64HVP32Q7Q, seg_pos={0 0}, upload=535239]: Uploading piece... Dec 31 02:14:38.041129762 (610h45m37.912212798s) [noise=true , address=45.91.116.6:4975 , piece_id=WYHBYOD2KNNNBKLYEABO3IJQZ7A5BCFY2VDUC42FSQ4BSJXZB7YQ, seg_pos={0 0}, upload=535239]: Uploading piece... Dec 31 02:14:38.041261242 (610h45m37.912344287s) [noise=true , address=145.79.227.2:4115 , piece_id=QOZ5LLFOXK64DDBFZNXYC57DPZTFOA3A52DZLVMJAXAQ6PMQ5EZQ, seg_pos={0 0}, upload=535239]: Uploading piece... Dec 31 02:14:38.041498614 (610h45m37.912581659s) [noise=true , address=77.90.31.17:4773 , piece_id=CTQHDW5OQQIU3HCRXTK3MBMAWVII26KJ6VTS7SKUOECK52EYJ73A, seg_pos={0 0}, upload=535239]: Uploading piece... Dec 31 02:14:38.041642067 (610h45m37.912725102s) [noise=true , address=145.79.227.11:4731 , piece_id=N6NQV3MQOKQ73XVGNALDUTVVKRN3NZMHN3LUSLUETQ7B4QULHHWA, seg_pos={0 0}, upload=535239]: Uploading piece... Dec 31 02:14:38.041874691 (610h45m37.912957736s) [noise=true , address=77.90.31.14:4661 , piece_id=ZCI467B65CWICFCNVROWMOLRTSXPRXEGW7UAISQQXB754IJRSLLQ, seg_pos={0 0}, upload=535239]: Uploading piece... Dec 31 02:14:38.042052519 (610h45m37.913135564s) [noise=true , address=145.79.227.7:4293 , piece_id=2JYV7IU6MP6IM5BDF5J4HPRPHJ6525XERBHINQLZGXXW3K6XJSRQ, seg_pos={0 0}, upload=535239]: Uploading piece... Dec 31 02:14:38.042250667 (610h45m37.913333702s) [noise=true , address=145.79.227.15:4875 , piece_id=TQALP5AZFPE4BONLLZBAMU47XM2TBBRO4K4OU3C3XFB5ZC7ZCGOQ, seg_pos={0 0}, upload=535239]: Uploading piece... Dec 31 02:14:39.634068194 (610h45m39.50515124s) [upload=535239]: Next returned nil segment Dec 31 02:14:39.634080849 (610h45m39.505163894s) [upload=535239]: Waiting for error group managing segments... Dec 31 02:14:40.013473986 (610h45m39.884557031s) [noise=true , address=145.79.227.15:4875 , piece_id=TQALP5AZFPE4BONLLZBAMU47XM2TBBRO4K4OU3C3XFB5ZC7ZCGOQ, seg_pos={0 0}, upload=535239]: Done uploading piece. err: <nil> Dec 31 02:14:40.014699309 (610h45m39.885782344s) [noise=true , address=145.79.227.25:4617 , piece_id=VY6UV35MA6P5SMSEOBRVXKJTKI7XCRFPBP2NOOC3FHYIYEURAXCQ, seg_pos={0 0}, upload=535239]: Done uploading piece. err: <nil> Dec 31 02:14:40.015237854 (610h45m39.886320889s) [noise=true , address=145.79.227.5:4222 , piece_id=OHBW2VVVPY5MBVRNMNICGPBDBULGLWIXBZZFRCYLLIBHBDK65FQA, seg_pos={0 0}, upload=535239]: Done uploading piece. err: <nil> Dec 31 02:14:40.028465390 (610h45m39.899548435s) [noise=true , address=145.79.227.11:4731 , piece_id=N6NQV3MQOKQ73XVGNALDUTVVKRN3NZMHN3LUSLUETQ7B4QULHHWA, seg_pos={0 0}, upload=535239]: Done uploading piece. err: <nil> Dec 31 02:14:40.039794142 (610h45m39.910877177s) [noise=true , address=145.79.227.13:4803 , piece_id=J6ZXXRT2C5LKRHSHFEH4ZPSE6EWODQ2PY5SS6M7PU3W4IBBJMZYA, seg_pos={0 0}, upload=535239]: Done uploading piece. err: <nil> Dec 31 02:14:40.040169597 (610h45m39.911252642s) [noise=true , address=145.79.227.7:4293 , piece_id=2JYV7IU6MP6IM5BDF5J4HPRPHJ6525XERBHINQLZGXXW3K6XJSRQ, seg_pos={0 0}, upload=535239]: Done uploading piece. err: <nil> Dec 31 02:14:40.046650708 (610h45m39.917733783s) [noise=true , address=145.79.227.2:4115 , piece_id=QOZ5LLFOXK64DDBFZNXYC57DPZTFOA3A52DZLVMJAXAQ6PMQ5EZQ, seg_pos={0 0}, upload=535239]: Done uploading piece. err: <nil> Dec 31 02:14:40.051922766 (610h45m39.923005801s) [noise=true , address=145.79.227.16:4912 , piece_id=NLZMXVBELJUSC3YOX4WH7CXZVSE6PW7BGAO5UOFABB64HVP32Q7Q, seg_pos={0 0}, upload=535239]: Done uploading piece. err: <nil> Dec 31 02:14:40.061258493 (610h45m39.932341528s) [noise=true , address=145.79.227.47:4265 , piece_id=52RP3UK5N5IFWM7NLKPB5E4XI3FVOXQLZTM2ARXXJBVTU7WMNFZA, seg_pos={0 0}, upload=535239]: Done uploading piece. err: <nil> Dec 31 02:14:40.104436648 (610h45m39.975519683s) [noise=true , address=145.79.227.20:4437 , piece_id=MAYAWLZUZKK4D36V7NJEOQO6NUOVVYUXWU2HVFYISZBK636CJNPA, seg_pos={0 0}, upload=535239]: Done uploading piece. err: <nil> Dec 31 02:14:40.224783723 (610h45m40.095866778s) [noise=true , address=77.90.31.14:4661 , piece_id=ZCI467B65CWICFCNVROWMOLRTSXPRXEGW7UAISQQXB754IJRSLLQ, seg_pos={0 0}, upload=535239]: Done uploading piece. err: <nil> Dec 31 02:14:40.474786135 (610h45m40.34586918s) [noise=true , address=45.91.116.14:4192 , piece_id=66FTF7LNKPLJGOKDAO2TKO2NDGMLZB4RWVQLFX6WJJYJL6QA6X5Q, seg_pos={0 0}, upload=535239]: Done uploading piece. err: <nil> Dec 31 02:14:40.566077013 (610h45m40.437160058s) [noise=true , address=45.91.116.11:4117 , piece_id=LTEONU5EG4ZVDMTGJM5RGCQ4XAILCTAFLCWBXKPESDZRNUA2T75A, seg_pos={0 0}, upload=535239]: Done uploading piece. err: <nil> Dec 31 02:14:40.566102300 (610h45m40.437185345s) [seg_pos={0 0}, upload=535239]: Segment reached optimal threshold of 13 pieces. Dec 31 02:14:40.566312751 (610h45m40.437395796s) [noise=true , address=77.90.31.17:4773 , piece_id=CTQHDW5OQQIU3HCRXTK3MBMAWVII26KJ6VTS7SKUOECK52EYJ73A, seg_pos={0 0}, upload=535239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: write tcp 188.166.225.170:33512->77.90.31.17:4773: use of closed network connection; write tcp 188.166.225.170:33512->77.90.31.17:4773: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33512->77.90.31.17:4773: use of closed network connection Dec 31 02:14:40.566323061 (610h45m40.437406106s) [noise=true , address=77.90.31.17:4773 , piece_id=CTQHDW5OQQIU3HCRXTK3MBMAWVII26KJ6VTS7SKUOECK52EYJ73A, seg_pos={0 0}, upload=535239]: Long tail cancellation due to optimal threshold Dec 31 02:14:40.566348479 (610h45m40.437431524s) [noise=true , address=45.91.116.6:4975 , piece_id=WYHBYOD2KNNNBKLYEABO3IJQZ7A5BCFY2VDUC42FSQ4BSJXZB7YQ, seg_pos={0 0}, upload=535239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: write tcp 188.166.225.170:44274->45.91.116.6:4975: use of closed network connection; write tcp 188.166.225.170:44274->45.91.116.6:4975: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44274->45.91.116.6:4975: use of closed network connection Dec 31 02:14:40.566358829 (610h45m40.437441864s) [noise=true , address=45.91.116.6:4975 , piece_id=WYHBYOD2KNNNBKLYEABO3IJQZ7A5BCFY2VDUC42FSQ4BSJXZB7YQ, seg_pos={0 0}, upload=535239]: Long tail cancellation due to optimal threshold Dec 31 02:14:40.566588596 (610h45m40.437671641s) [noise=true , address=45.91.116.4:4940 , piece_id=4IJOPYTYRZXNK6Y5MSM4FBR5A777UN2GSBWAZUAYFFS7ALAHUDIQ, seg_pos={0 0}, upload=535239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: write tcp 188.166.225.170:54278->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:54278->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54278->45.91.116.4:4940: use of closed network connection Dec 31 02:14:40.566599337 (610h45m40.437682372s) [noise=true , address=45.91.116.4:4940 , piece_id=4IJOPYTYRZXNK6Y5MSM4FBR5A777UN2GSBWAZUAYFFS7ALAHUDIQ, seg_pos={0 0}, upload=535239]: Long tail cancellation due to optimal threshold Dec 31 02:14:40.566711761 (610h45m40.437794796s) [seg_pos={0 0}, upload=535239]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 02:14:40.566871024 (610h45m40.437954059s) [upload=535239]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 02:14:40.566878328 (610h45m40.437961374s) [upload=535239]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 02:14:40.856098205 (610h45m40.72718124s) [upload=535239]: Done uploading segments... Dec 31 02:14:40.856116771 (610h45m40.727199806s) [upload=535239]: Upload finished. err: <nil> Duration:* 30.003376816s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........X..........................X.......................] duration: 23.85839924s trace: https://jaeger.dev.storj.io/trace/365693fea00df28 FAIL (2025-12-31 02:15:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [noise=true , address=145.79.227.49:4343 , piece_id=6L3HPTS7D2NRA4QEOH4RFM3D3B53CTCONRI6G6YVU4OOTUWTG7XA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.942060847 (610h25m24.813143892s) [noise=true , address=145.79.227.34:4937 , piece_id=5W7LBM3YJJTVUFYTELR57DVXB663OHO5N3Z2SPQPZ4G43IK47O3Q, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.942684104 (610h25m24.813767139s) [noise=true , address=64.49.13.139:4478 , piece_id=QN6RT2KWEA6APQVA66ZMKG4LH537UGYNOXOCVVA2YC2MKDOLT3YA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.943064007 (610h25m24.814147042s) [noise=true , address=145.79.227.40:4084 , piece_id=RUKTKDII6BRUWSYVS65CWUBT3QKG3ASUK3KA3O5VGE53XSG6IMOQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.943459560 (610h25m24.814542605s) [noise=true , address=145.79.227.14:4847 , piece_id=JX5NXTMVM3NSWIEVR4RX46ABSNPKTOFDLFBPKEXFQ3G3ANKV5OLA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.943865303 (610h25m24.814948348s) [noise=true , address=45.91.116.6:4981 , piece_id=CT474ZA2LVAZEBJHBDD4GB6NTHVHZFJLNE5OZV3DFS2NZKVW3EQQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.944259694 (610h25m24.815342749s) [noise=true , address=145.79.227.51:4443 , piece_id=73UUMRDY7BSZNCGBXCZYYRBVE5KLPMNPRNXOZEGWKQGLHJS32J2Q, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.946575182 (610h25m24.817658227s) [noise=true , address=77.90.31.10:4523 , piece_id=P6LIONXJI6AFZ2JTKCQ2W6RZCOZCK6MZ53EXAJKXYQXSX2PX4QEA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.948132909 (610h25m24.819215944s) [noise=true , address=145.79.227.20:4452 , piece_id=ZSDN2JZ3FQUVJYN5OURDXNKEDLDNULJOHKKRZVIPXZIWIYAHCRXQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.949605774 (610h25m24.820688819s) [noise=true , address=77.90.31.14:4680 , piece_id=S2E4WBA7MJVAZLT5JG3IMHK53RQC77CCHV2PDF6L2R2JEA2U3K4Q, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.951459002 (610h25m24.822542048s) [noise=true , address=193.149.164.143:4704 , piece_id=BHXHFBKTLTDMGXWY5QRTP2THZQE5OXFZ2LCMTO2KRYRWAHXWW3NQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.952986742 (610h25m24.824069777s) [noise=true , address=77.90.31.19:4848 , piece_id=VZYG32S6ODIPDKQYZRXECBHXOERZDDWD2OF35Z46W4XMUZ3OJLAA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.956916654 (610h25m24.827999689s) [noise=true , address=145.79.227.29:4971 , piece_id=MEPXHHOQ6SUZC3GIPCQEMOKKF5OJTKZEI3UYZBAYXAQZGSVEKSBA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.960665062 (610h25m24.831748097s) [noise=true , address=64.49.13.170:4720 , piece_id=HIATRSAPE4YLSODFZDPP43ZUMLDVM7GJEVMWE4ZGUQTP25LTHGTA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.962366001 (610h25m24.833449036s) [noise=true , address=145.79.227.2:4117 , piece_id=4SGFGB3IJ7LTXDMMWM2PGGOV64MC46IZ7OZQWH55J4NBQY7CHMZQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.963708216 (610h25m24.834791261s) [noise=true , address=145.79.227.35:4505 , piece_id=V7RS7A4QLZTH4G32ZUZNSF2Z5APVWZGJO77NEGP3SBWQT3WCMVLQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:26.352236138 (610h25m26.223319173s) [upload=534952]: Next returned nil segment Dec 31 01:54:26.352253821 (610h25m26.223336857s) [upload=534952]: Waiting for error group managing segments... Dec 31 01:54:26.812758374 (610h25m26.683841409s) [noise=true , address=145.79.227.51:4443 , piece_id=73UUMRDY7BSZNCGBXCZYYRBVE5KLPMNPRNXOZEGWKQGLHJS32J2Q, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.824371056 (610h25m26.695454101s) [noise=true , address=145.79.227.14:4847 , piece_id=JX5NXTMVM3NSWIEVR4RX46ABSNPKTOFDLFBPKEXFQ3G3ANKV5OLA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.838414046 (610h25m26.709497081s) [noise=true , address=64.49.13.138:4490 , piece_id=2XTUZE6PXRE3OARNJVT36FXXWUMYIBVNE6M2VVFAVBGKEE2H5YZQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.875499841 (610h25m26.746582876s) [noise=true , address=64.49.13.139:4478 , piece_id=QN6RT2KWEA6APQVA66ZMKG4LH537UGYNOXOCVVA2YC2MKDOLT3YA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.891607221 (610h25m26.762690256s) [noise=true , address=145.79.227.20:4452 , piece_id=ZSDN2JZ3FQUVJYN5OURDXNKEDLDNULJOHKKRZVIPXZIWIYAHCRXQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.988561223 (610h25m26.859644268s) [noise=true , address=64.49.13.170:4720 , piece_id=HIATRSAPE4YLSODFZDPP43ZUMLDVM7GJEVMWE4ZGUQTP25LTHGTA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.039035911 (610h25m26.910118946s) [noise=true , address=77.90.31.14:4680 , piece_id=S2E4WBA7MJVAZLT5JG3IMHK53RQC77CCHV2PDF6L2R2JEA2U3K4Q, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.098023090 (610h25m26.969106126s) [noise=true , address=45.91.116.6:4981 , piece_id=CT474ZA2LVAZEBJHBDD4GB6NTHVHZFJLNE5OZV3DFS2NZKVW3EQQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.653790754 (610h25m27.524873789s) [noise=true , address=77.90.31.21:4886 , piece_id=K34WPIBGGGEPHIKJ253XGLDCIQ3WLFN4AZ6EYNWVHHT2EIHS6YRQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.684192972 (610h25m27.555276017s) [noise=true , address=193.149.164.143:4704 , piece_id=BHXHFBKTLTDMGXWY5QRTP2THZQE5OXFZ2LCMTO2KRYRWAHXWW3NQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.717238425 (610h25m27.58832146s) [noise=true , address=145.79.227.40:4084 , piece_id=RUKTKDII6BRUWSYVS65CWUBT3QKG3ASUK3KA3O5VGE53XSG6IMOQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.718349742 (610h25m27.589432777s) [noise=true , address=77.90.31.10:4523 , piece_id=P6LIONXJI6AFZ2JTKCQ2W6RZCOZCK6MZ53EXAJKXYQXSX2PX4QEA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.803639562 (610h25m27.674722597s) [noise=true , address=145.79.227.29:4971 , piece_id=MEPXHHOQ6SUZC3GIPCQEMOKKF5OJTKZEI3UYZBAYXAQZGSVEKSBA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.812791279 (610h25m27.683874314s) [noise=true , address=145.79.227.49:4343 , piece_id=6L3HPTS7D2NRA4QEOH4RFM3D3B53CTCONRI6G6YVU4OOTUWTG7XA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.812812348 (610h25m27.683895393s) [seg_pos={0 0}, upload=534952]: Segment reached optimal threshold of 14 pieces. Dec 31 01:54:27.813018000 (610h25m27.684101035s) [noise=true , address=145.79.227.35:4505 , piece_id=V7RS7A4QLZTH4G32ZUZNSF2Z5APVWZGJO77NEGP3SBWQT3WCMVLQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i6itGoBhLsLiqC8et8Y5L62fLjcWxy18u39NAr4R3nXNMWcsf): protocol: expected piece hash; context canceled; EOF Dec 31 01:54:27.813024772 (610h25m27.684107808s) [noise=true , address=145.79.227.35:4505 , piece_id=V7RS7A4QLZTH4G32ZUZNSF2Z5APVWZGJO77NEGP3SBWQT3WCMVLQ, seg_pos={0 0}, upload=534952]: Long tail cancellation due to optimal threshold Dec 31 01:54:27.813175931 (610h25m27.684258976s) [noise=true , address=145.79.227.34:4937 , piece_id=5W7LBM3YJJTVUFYTELR57DVXB663OHO5N3Z2SPQPZ4G43IK47O3Q, seg_pos={0 0}, upload=534952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1LKrFThdW3jJpG6dRZLUaxLXTa5Wy27yQ8PGUEqjvBWfFoRB94): protocol: expected piece hash; context canceled; EOF Dec 31 01:54:27.813181702 (610h25m27.684264737s) [noise=true , address=145.79.227.34:4937 , piece_id=5W7LBM3YJJTVUFYTELR57DVXB663OHO5N3Z2SPQPZ4G43IK47O3Q, seg_pos={0 0}, upload=534952]: Long tail cancellation due to optimal threshold Dec 31 01:54:27.813407862 (610h25m27.684490907s) [noise=true , address=145.79.227.2:4117 , piece_id=4SGFGB3IJ7LTXDMMWM2PGGOV64MC46IZ7OZQWH55J4NBQY7CHMZQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122Htg72f5eUUutf1p3fYQpy1Wkf6AgdaQV83HHYye2n8zm3the): protocol: expected piece hash; context canceled; EOF Dec 31 01:54:27.813414264 (610h25m27.684497299s) [noise=true , address=145.79.227.2:4117 , piece_id=4SGFGB3IJ7LTXDMMWM2PGGOV64MC46IZ7OZQWH55J4NBQY7CHMZQ, seg_pos={0 0}, upload=534952]: Long tail cancellation due to optimal threshold Dec 31 01:54:27.813424614 (610h25m27.684507649s) [noise=true , address=77.90.31.19:4848 , piece_id=VZYG32S6ODIPDKQYZRXECBHXOERZDDWD2OF35Z46W4XMUZ3OJLAA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD): protocol: write tcp 188.166.225.170:42842->77.90.31.19:4848: use of closed network connection; write tcp 188.166.225.170:42842->77.90.31.19:4848: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42842->77.90.31.19:4848: use of closed network connection Dec 31 01:54:27.813432750 (610h25m27.684515785s) [noise=true , address=77.90.31.19:4848 , piece_id=VZYG32S6ODIPDKQYZRXECBHXOERZDDWD2OF35Z46W4XMUZ3OJLAA, seg_pos={0 0}, upload=534952]: Long tail cancellation due to optimal threshold Dec 31 01:54:27.813474088 (610h25m27.684557123s) [seg_pos={0 0}, upload=534952]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 01:54:27.813535325 (610h25m27.68461836s) [upload=534952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 01:54:27.813539613 (610h25m27.684622648s) [upload=534952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 01:54:28.099921034 (610h25m27.971004069s) [upload=534952]: Done uploading segments... Dec 31 01:54:28.099940451 (610h25m27.971023486s) [upload=534952]: Upload finished. err: <nil> Duration: 23.858431172s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...................X..................................X.....] duration: 3.620000383s trace: https://jaeger.dev.storj.io/trace/740aa405bd68150f FAIL (2025-12-31 02:14:59 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.620029329s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........X..........................X......................] duration: 35.835756414s trace: https://jaeger.dev.storj.io/trace/365693fea00df28 FAIL (2025-12-31 02:14:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [noise=true , address=145.79.227.49:4343 , piece_id=6L3HPTS7D2NRA4QEOH4RFM3D3B53CTCONRI6G6YVU4OOTUWTG7XA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.942060847 (610h25m24.813143892s) [noise=true , address=145.79.227.34:4937 , piece_id=5W7LBM3YJJTVUFYTELR57DVXB663OHO5N3Z2SPQPZ4G43IK47O3Q, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.942684104 (610h25m24.813767139s) [noise=true , address=64.49.13.139:4478 , piece_id=QN6RT2KWEA6APQVA66ZMKG4LH537UGYNOXOCVVA2YC2MKDOLT3YA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.943064007 (610h25m24.814147042s) [noise=true , address=145.79.227.40:4084 , piece_id=RUKTKDII6BRUWSYVS65CWUBT3QKG3ASUK3KA3O5VGE53XSG6IMOQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.943459560 (610h25m24.814542605s) [noise=true , address=145.79.227.14:4847 , piece_id=JX5NXTMVM3NSWIEVR4RX46ABSNPKTOFDLFBPKEXFQ3G3ANKV5OLA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.943865303 (610h25m24.814948348s) [noise=true , address=45.91.116.6:4981 , piece_id=CT474ZA2LVAZEBJHBDD4GB6NTHVHZFJLNE5OZV3DFS2NZKVW3EQQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.944259694 (610h25m24.815342749s) [noise=true , address=145.79.227.51:4443 , piece_id=73UUMRDY7BSZNCGBXCZYYRBVE5KLPMNPRNXOZEGWKQGLHJS32J2Q, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.946575182 (610h25m24.817658227s) [noise=true , address=77.90.31.10:4523 , piece_id=P6LIONXJI6AFZ2JTKCQ2W6RZCOZCK6MZ53EXAJKXYQXSX2PX4QEA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.948132909 (610h25m24.819215944s) [noise=true , address=145.79.227.20:4452 , piece_id=ZSDN2JZ3FQUVJYN5OURDXNKEDLDNULJOHKKRZVIPXZIWIYAHCRXQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.949605774 (610h25m24.820688819s) [noise=true , address=77.90.31.14:4680 , piece_id=S2E4WBA7MJVAZLT5JG3IMHK53RQC77CCHV2PDF6L2R2JEA2U3K4Q, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.951459002 (610h25m24.822542048s) [noise=true , address=193.149.164.143:4704 , piece_id=BHXHFBKTLTDMGXWY5QRTP2THZQE5OXFZ2LCMTO2KRYRWAHXWW3NQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.952986742 (610h25m24.824069777s) [noise=true , address=77.90.31.19:4848 , piece_id=VZYG32S6ODIPDKQYZRXECBHXOERZDDWD2OF35Z46W4XMUZ3OJLAA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.956916654 (610h25m24.827999689s) [noise=true , address=145.79.227.29:4971 , piece_id=MEPXHHOQ6SUZC3GIPCQEMOKKF5OJTKZEI3UYZBAYXAQZGSVEKSBA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.960665062 (610h25m24.831748097s) [noise=true , address=64.49.13.170:4720 , piece_id=HIATRSAPE4YLSODFZDPP43ZUMLDVM7GJEVMWE4ZGUQTP25LTHGTA, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.962366001 (610h25m24.833449036s) [noise=true , address=145.79.227.2:4117 , piece_id=4SGFGB3IJ7LTXDMMWM2PGGOV64MC46IZ7OZQWH55J4NBQY7CHMZQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:24.963708216 (610h25m24.834791261s) [noise=true , address=145.79.227.35:4505 , piece_id=V7RS7A4QLZTH4G32ZUZNSF2Z5APVWZGJO77NEGP3SBWQT3WCMVLQ, seg_pos={0 0}, upload=534952]: Uploading piece... Dec 31 01:54:26.352236138 (610h25m26.223319173s) [upload=534952]: Next returned nil segment Dec 31 01:54:26.352253821 (610h25m26.223336857s) [upload=534952]: Waiting for error group managing segments... Dec 31 01:54:26.812758374 (610h25m26.683841409s) [noise=true , address=145.79.227.51:4443 , piece_id=73UUMRDY7BSZNCGBXCZYYRBVE5KLPMNPRNXOZEGWKQGLHJS32J2Q, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.824371056 (610h25m26.695454101s) [noise=true , address=145.79.227.14:4847 , piece_id=JX5NXTMVM3NSWIEVR4RX46ABSNPKTOFDLFBPKEXFQ3G3ANKV5OLA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.838414046 (610h25m26.709497081s) [noise=true , address=64.49.13.138:4490 , piece_id=2XTUZE6PXRE3OARNJVT36FXXWUMYIBVNE6M2VVFAVBGKEE2H5YZQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.875499841 (610h25m26.746582876s) [noise=true , address=64.49.13.139:4478 , piece_id=QN6RT2KWEA6APQVA66ZMKG4LH537UGYNOXOCVVA2YC2MKDOLT3YA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.891607221 (610h25m26.762690256s) [noise=true , address=145.79.227.20:4452 , piece_id=ZSDN2JZ3FQUVJYN5OURDXNKEDLDNULJOHKKRZVIPXZIWIYAHCRXQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:26.988561223 (610h25m26.859644268s) [noise=true , address=64.49.13.170:4720 , piece_id=HIATRSAPE4YLSODFZDPP43ZUMLDVM7GJEVMWE4ZGUQTP25LTHGTA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.039035911 (610h25m26.910118946s) [noise=true , address=77.90.31.14:4680 , piece_id=S2E4WBA7MJVAZLT5JG3IMHK53RQC77CCHV2PDF6L2R2JEA2U3K4Q, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.098023090 (610h25m26.969106126s) [noise=true , address=45.91.116.6:4981 , piece_id=CT474ZA2LVAZEBJHBDD4GB6NTHVHZFJLNE5OZV3DFS2NZKVW3EQQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.653790754 (610h25m27.524873789s) [noise=true , address=77.90.31.21:4886 , piece_id=K34WPIBGGGEPHIKJ253XGLDCIQ3WLFN4AZ6EYNWVHHT2EIHS6YRQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.684192972 (610h25m27.555276017s) [noise=true , address=193.149.164.143:4704 , piece_id=BHXHFBKTLTDMGXWY5QRTP2THZQE5OXFZ2LCMTO2KRYRWAHXWW3NQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.717238425 (610h25m27.58832146s) [noise=true , address=145.79.227.40:4084 , piece_id=RUKTKDII6BRUWSYVS65CWUBT3QKG3ASUK3KA3O5VGE53XSG6IMOQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.718349742 (610h25m27.589432777s) [noise=true , address=77.90.31.10:4523 , piece_id=P6LIONXJI6AFZ2JTKCQ2W6RZCOZCK6MZ53EXAJKXYQXSX2PX4QEA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.803639562 (610h25m27.674722597s) [noise=true , address=145.79.227.29:4971 , piece_id=MEPXHHOQ6SUZC3GIPCQEMOKKF5OJTKZEI3UYZBAYXAQZGSVEKSBA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.812791279 (610h25m27.683874314s) [noise=true , address=145.79.227.49:4343 , piece_id=6L3HPTS7D2NRA4QEOH4RFM3D3B53CTCONRI6G6YVU4OOTUWTG7XA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: <nil> Dec 31 01:54:27.812812348 (610h25m27.683895393s) [seg_pos={0 0}, upload=534952]: Segment reached optimal threshold of 14 pieces. Dec 31 01:54:27.813018000 (610h25m27.684101035s) [noise=true , address=145.79.227.35:4505 , piece_id=V7RS7A4QLZTH4G32ZUZNSF2Z5APVWZGJO77NEGP3SBWQT3WCMVLQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i6itGoBhLsLiqC8et8Y5L62fLjcWxy18u39NAr4R3nXNMWcsf): protocol: expected piece hash; context canceled; EOF Dec 31 01:54:27.813024772 (610h25m27.684107808s) [noise=true , address=145.79.227.35:4505 , piece_id=V7RS7A4QLZTH4G32ZUZNSF2Z5APVWZGJO77NEGP3SBWQT3WCMVLQ, seg_pos={0 0}, upload=534952]: Long tail cancellation due to optimal threshold Dec 31 01:54:27.813175931 (610h25m27.684258976s) [noise=true , address=145.79.227.34:4937 , piece_id=5W7LBM3YJJTVUFYTELR57DVXB663OHO5N3Z2SPQPZ4G43IK47O3Q, seg_pos={0 0}, upload=534952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1LKrFThdW3jJpG6dRZLUaxLXTa5Wy27yQ8PGUEqjvBWfFoRB94): protocol: expected piece hash; context canceled; EOF Dec 31 01:54:27.813181702 (610h25m27.684264737s) [noise=true , address=145.79.227.34:4937 , piece_id=5W7LBM3YJJTVUFYTELR57DVXB663OHO5N3Z2SPQPZ4G43IK47O3Q, seg_pos={0 0}, upload=534952]: Long tail cancellation due to optimal threshold Dec 31 01:54:27.813407862 (610h25m27.684490907s) [noise=true , address=145.79.227.2:4117 , piece_id=4SGFGB3IJ7LTXDMMWM2PGGOV64MC46IZ7OZQWH55J4NBQY7CHMZQ, seg_pos={0 0}, upload=534952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122Htg72f5eUUutf1p3fYQpy1Wkf6AgdaQV83HHYye2n8zm3the): protocol: expected piece hash; context canceled; EOF Dec 31 01:54:27.813414264 (610h25m27.684497299s) [noise=true , address=145.79.227.2:4117 , piece_id=4SGFGB3IJ7LTXDMMWM2PGGOV64MC46IZ7OZQWH55J4NBQY7CHMZQ, seg_pos={0 0}, upload=534952]: Long tail cancellation due to optimal threshold Dec 31 01:54:27.813424614 (610h25m27.684507649s) [noise=true , address=77.90.31.19:4848 , piece_id=VZYG32S6ODIPDKQYZRXECBHXOERZDDWD2OF35Z46W4XMUZ3OJLAA, seg_pos={0 0}, upload=534952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD): protocol: write tcp 188.166.225.170:42842->77.90.31.19:4848: use of closed network connection; write tcp 188.166.225.170:42842->77.90.31.19:4848: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42842->77.90.31.19:4848: use of closed network connection Dec 31 01:54:27.813432750 (610h25m27.684515785s) [noise=true , address=77.90.31.19:4848 , piece_id=VZYG32S6ODIPDKQYZRXECBHXOERZDDWD2OF35Z46W4XMUZ3OJLAA, seg_pos={0 0}, upload=534952]: Long tail cancellation due to optimal threshold Dec 31 01:54:27.813474088 (610h25m27.684557123s) [seg_pos={0 0}, upload=534952]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 01:54:27.813535325 (610h25m27.68461836s) [upload=534952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 01:54:27.813539613 (610h25m27.684622648s) [upload=534952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 01:54:28.099921034 (610h25m27.971004069s) [upload=534952]: Done uploading segments... Dec 31 01:54:28.099940451 (610h25m27.971023486s) [upload=534952]: Upload finished. err: <nil> Duration: 35.835783897s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)